2024-04-26T11:04:55.800 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T11:04:55.801 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --verbose --name leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} --archive /home/teuthworker/archive/leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674685 -- /home/teuthworker/archive/leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674685/orig.config.yaml 2024-04-26T11:04:55.866 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T11:04:55.949 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674685 branch: wip-lusov-quiescer-fixes description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} email: null first_in_suite: false job_id: '7674685' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer-fixes ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer-fixes sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4375 sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer-fixes suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSrtqM7fDLQ8xmfTVkZ174ibHAnD2G+821ZsYrkJ5vEDeMQuFuSYKeLIrUvgg6NCnRYG5ZKct3THSVBpoilji4= smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHOV89Se2CUexWWKr/qC/qicY/QVYhDKpvxOFn+WMb48EIrSkHY1t4drUKAVpB0MO4WT/4CeWMOLMpOe6ZK3Bw= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEabkQ+HhABZTpFG2o7UxHK8hfxKnhaEnZH4JnjMAa3PsTkgRFoeIs+rtxjkB/uaR2ozc0IkQwW9/xHUwip4ABc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_10:54:14 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T11:04:55.949 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa; will attempt to use it 2024-04-26T11:04:55.950 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks 2024-04-26T11:04:55.950 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T11:04:55.952 INFO:teuthology.task.internal:Checking packages... 2024-04-26T11:04:55.974 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1' 2024-04-26T11:04:55.975 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T11:04:55.975 INFO:teuthology.packaging:ref: None 2024-04-26T11:04:55.975 INFO:teuthology.packaging:tag: None 2024-04-26T11:04:55.975 INFO:teuthology.packaging:branch: wip-lusov-quiescer-fixes 2024-04-26T11:04:55.975 INFO:teuthology.packaging:sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:04:55.975 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer-fixes 2024-04-26T11:04:56.123 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3267.gbc94a2a9 2024-04-26T11:04:56.125 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T11:04:56.136 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T11:04:56.136 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T11:04:56.176 INFO:teuthology.task.internal:Saving configuration 2024-04-26T11:04:56.197 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T11:04:56.211 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T11:04:56.232 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674685', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 10:59:49.186087', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSrtqM7fDLQ8xmfTVkZ174ibHAnD2G+821ZsYrkJ5vEDeMQuFuSYKeLIrUvgg6NCnRYG5ZKct3THSVBpoilji4='} 2024-04-26T11:04:56.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674685', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 10:59:49.187450', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHOV89Se2CUexWWKr/qC/qicY/QVYhDKpvxOFn+WMb48EIrSkHY1t4drUKAVpB0MO4WT/4CeWMOLMpOe6ZK3Bw='} 2024-04-26T11:04:56.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674685', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 10:59:49.188625', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEabkQ+HhABZTpFG2o7UxHK8hfxKnhaEnZH4JnjMAa3PsTkgRFoeIs+rtxjkB/uaR2ozc0IkQwW9/xHUwip4ABc='} 2024-04-26T11:04:56.266 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T11:04:56.281 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T11:04:56.281 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T11:04:56.281 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T11:04:56.282 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T11:04:56.407 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe36eb580d0>, signals=[15]) 2024-04-26T11:04:56.407 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T11:04:56.415 INFO:teuthology.task.internal:Opening connections... 2024-04-26T11:04:56.415 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:04:56.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:04:56.495 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-04-26T11:04:56.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:04:56.569 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T11:04:56.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:04:56.646 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T11:04:56.654 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-04-26T11:04:56.675 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-04-26T11:04:56.675 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-04-26T11:04:56.733 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-04-26T11:04:56.733 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-04-26T11:04:56.733 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-04-26T11:04:56.733 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-04-26T11:04:56.733 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-04-26T11:04:56.733 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-04-26T11:04:56.733 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T11:04:56.734 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-04-26T11:04:56.734 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-04-26T11:04:56.734 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T11:04:56.734 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-04-26T11:04:56.734 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T11:04:56.734 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T11:04:56.734 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T11:04:56.735 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-04-26T11:04:56.758 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-04-26T11:04:56.776 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-04-26T11:04:56.776 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-04-26T11:04:56.834 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T11:04:56.835 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-04-26T11:04:56.835 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T11:04:56.835 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T11:04:56.835 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T11:04:56.835 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-04-26T11:04:56.857 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-04-26T11:04:56.875 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-04-26T11:04:56.875 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-04-26T11:04:56.932 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-04-26T11:04:56.933 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T11:04:56.933 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-04-26T11:04:56.933 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T11:04:56.933 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T11:04:56.933 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T11:04:56.933 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-04-26T11:04:56.958 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T11:04:56.969 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T11:04:56.998 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T11:04:56.998 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-04-26T11:04:57.001 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-04-26T11:04:57.004 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-04-26T11:04:57.020 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T11:04:57.030 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T11:04:57.031 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-04-26T11:04:57.060 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-04-26T11:04:57.062 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-04-26T11:04:57.199 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T11:04:57.272 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T11:04:57.289 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-26T11:04:57.289 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T11:04:57.289 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T11:04:57.289 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T11:04:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T11:04:57.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T11:04:57.465 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T11:04:57.465 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T11:04:57.466 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-04-26T11:04:57.484 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-437.el9.x86_64 2024-04-26T11:04:57.484 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T11:04:57.484 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T11:04:57.484 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum upgrade -y linux-firmware 2024-04-26T11:04:59.816 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-26T11:05:01.692 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-26T11:05:05.842 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-26T11:05:07.542 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-26T11:05:09.236 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-26T11:05:14.409 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-04-26T11:05:16.461 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-04-26T11:05:16.462 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-04-26T11:05:16.462 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-04-26T11:05:16.556 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T11:05:16.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T11:05:16.775 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T11:05:16.775 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T11:05:16.778 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi033.front.sepia.ceph.com to complete... 2024-04-26T11:05:55.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26 11:05:55 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T11:05:55.043 INFO:teuthology.orchestra.run.smithi033.stderr:FINISHED --2024-04-26 11:05:55-- 2024-04-26T11:05:55.043 INFO:teuthology.orchestra.run.smithi033.stderr:Total wall clock time: 38s 2024-04-26T11:05:55.044 INFO:teuthology.orchestra.run.smithi033.stderr:Downloaded: 1 files, 341M in 38s (9.05 MB/s) 2024-04-26T11:05:55.088 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T11:05:55.088 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T11:05:57.564 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying... ######################################## 2024-04-26T11:05:57.701 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing... ######################################## 2024-04-26T11:05:57.831 INFO:teuthology.orchestra.run.smithi033.stdout:Updating / installing... 2024-04-26T11:06:06.139 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T11:06:43.889 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T11:06:43.889 DEBUG:teuthology.orchestra.run.smithi033:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T11:06:44.015 INFO:teuthology.orchestra.run.smithi033.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T11:06:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T11:06:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T11:06:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T11:06:44.019 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T11:06:44.020 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T11:06:44.021 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T11:06:44.022 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T11:06:44.023 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T11:06:44.024 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T11:06:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T11:06:44.026 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T11:06:44.027 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T11:06:44.028 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T11:06:44.029 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T11:06:44.030 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T11:06:44.031 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T11:06:44.032 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T11:06:44.033 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T11:06:44.034 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T11:06:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T11:06:44.036 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T11:06:44.037 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T11:06:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T11:06:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T11:06:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T11:06:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T11:06:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T11:06:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T11:06:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T11:06:44.039 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T11:06:44.040 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T11:06:44.041 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T11:06:44.042 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T11:06:44.043 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T11:06:44.044 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T11:06:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T11:06:44.046 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T11:06:44.047 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T11:06:44.048 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T11:06:44.049 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T11:06:44.050 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T11:06:44.051 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T11:06:44.052 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T11:06:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T11:06:44.054 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T11:06:44.055 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T11:06:44.056 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T11:06:44.057 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T11:06:44.058 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T11:06:44.059 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T11:06:44.060 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T11:06:44.061 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T11:06:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T11:06:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T11:06:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T11:06:44.065 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T11:06:44.066 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T11:06:44.067 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T11:06:44.068 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T11:06:44.069 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T11:06:44.070 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T11:06:44.071 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T11:06:44.072 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T11:06:44.073 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T11:06:44.074 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T11:06:44.075 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T11:06:44.076 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T11:06:44.077 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T11:06:44.078 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T11:06:44.079 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T11:06:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T11:06:44.081 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T11:06:44.082 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T11:06:44.083 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T11:06:44.084 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T11:06:44.085 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T11:06:44.086 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T11:06:44.087 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T11:06:44.088 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T11:06:44.089 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T11:06:44.090 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T11:06:44.091 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T11:06:44.092 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T11:06:44.093 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T11:06:44.094 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T11:06:44.095 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T11:06:44.096 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T11:06:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T11:06:44.098 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T11:06:44.099 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T11:06:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T11:06:44.101 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T11:06:44.102 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T11:06:44.103 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T11:06:44.104 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T11:06:44.105 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T11:06:44.107 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T11:06:44.107 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T11:06:44.107 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T11:06:44.107 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T11:06:44.107 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T11:06:44.107 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T11:06:44.108 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T11:06:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T11:06:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T11:06:44.111 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T11:06:44.112 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T11:06:44.113 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T11:06:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T11:06:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T11:06:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T11:06:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T11:06:44.118 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T11:06:44.119 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T11:06:44.120 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T11:06:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T11:06:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T11:06:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T11:06:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T11:06:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T11:06:44.126 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T11:06:44.127 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T11:06:44.128 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T11:06:44.129 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T11:06:44.130 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T11:06:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T11:06:44.132 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T11:06:44.133 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T11:06:44.134 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T11:06:44.135 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T11:06:44.136 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T11:06:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T11:06:44.138 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T11:06:44.138 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T11:06:44.138 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T11:06:44.138 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T11:06:44.138 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T11:06:44.138 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T11:06:44.138 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T11:06:44.139 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T11:06:44.140 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T11:06:44.141 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T11:06:44.142 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T11:06:44.143 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T11:06:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T11:06:44.145 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T11:06:44.146 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T11:06:44.147 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T11:06:44.148 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T11:06:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T11:06:44.150 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T11:06:44.150 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T11:06:44.150 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T11:06:44.150 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T11:06:44.150 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T11:06:44.150 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T11:06:44.151 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T11:06:44.152 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T11:06:44.152 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T11:06:44.153 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T11:06:44.154 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T11:06:44.155 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T11:06:44.156 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T11:06:44.157 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T11:06:44.158 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T11:06:44.159 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T11:06:44.160 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T11:06:44.161 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T11:06:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T11:06:44.163 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T11:06:44.164 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T11:06:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T11:06:44.166 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T11:06:44.167 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T11:06:44.168 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T11:06:44.169 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T11:06:44.170 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T11:06:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T11:06:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T11:06:44.177 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:44.177 DEBUG:teuthology.orchestra.run.smithi033:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout:fi 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout:for file in vmlinuz System.map config; do 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout: fi 2024-04-26T11:06:44.202 INFO:teuthology.orchestra.run.smithi033.stdout:done 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:fi 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:fi 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:44.203 INFO:teuthology.orchestra.run.smithi033.stdout:fi 2024-04-26T11:06:44.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-04-26T11:06:44.279 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-04-26T11:06:44.280 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:06:44.281 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T11:06:44.282 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T11:06:44.282 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:44.282 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T11:06:44.803 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-04-26T11:06:46.133 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T11:06:46.159 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-04-26T11:06:46.161 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-04-26T11:06:46.200 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.SSbSKfk94r 2024-04-26T11:06:46.200 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SSbSKfk94r 2024-04-26T11:06:46.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.SSbSKfk94r 2024-04-26T11:06:46.412 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.SSbSKfk94r is 6KB 2024-04-26T11:06:46.461 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.SSbSKfk94r 2024-04-26T11:06:46.476 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T11:06:46.539 INFO:teuthology.orchestra.run.smithi033.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T11:06:46.539 INFO:teuthology.orchestra.run.smithi033.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T11:06:46.539 INFO:teuthology.orchestra.run.smithi033.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T11:06:46.539 INFO:teuthology.orchestra.run.smithi033.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T11:06:46.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T11:06:46.697 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-04-26T11:07:16.701 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T11:07:16.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-26T11:07:16.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:07:35.074 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-04-26T11:07:44.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-26T11:07:44.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:08:02.466 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-04-26T11:08:14.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-26T11:08:14.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:08:16.659 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-04-26T11:08:17.482 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-26T11:08:17.482 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T11:08:17.482 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-04-26T11:08:17.522 INFO:teuthology.orchestra.run.smithi033.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T11:08:17.523 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T11:08:17.523 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T11:08:17.523 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T11:08:18.524 DEBUG:teuthology.parallel:result is None 2024-04-26T11:08:18.525 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T11:08:18.532 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T11:08:18.533 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T11:08:18.540 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T11:08:18.543 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T11:08:18.580 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T11:08:18.587 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T11:08:18.626 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T11:08:18.626 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T11:08:18.631 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T11:08:18.633 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T11:08:18.687 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T11:08:18.693 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T11:08:18.694 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T11:08:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T11:08:18.721 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T11:08:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T11:08:18.738 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T11:08:18.744 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T11:08:18.763 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T11:08:18.794 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T11:08:18.812 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T11:08:18.831 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T11:08:18.845 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T11:08:18.893 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T11:08:18.940 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T11:08:18.944 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T11:08:18.951 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T11:08:18.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T11:08:18.957 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T11:08:18.959 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T11:08:19.028 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T11:08:19.036 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T11:08:19.037 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T11:08:19.044 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T11:08:19.047 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T11:08:19.087 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T11:08:19.163 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T11:08:19.235 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T11:08:19.301 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T11:08:19.379 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:08:19.379 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T11:08:19.462 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T11:08:19.508 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T11:08:19.563 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T11:08:19.647 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T11:08:19.692 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:08:19.692 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T11:08:19.757 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T11:08:19.801 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T11:08:19.862 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T11:08:19.907 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T11:08:19.948 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:08:19.948 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T11:08:20.014 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-04-26T11:08:20.023 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-04-26T11:08:20.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-04-26T11:08:20.059 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T11:08:20.087 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T11:08:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T11:08:20.470 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T11:08:20.478 INFO:teuthology.task.internal:Starting timer... 2024-04-26T11:08:20.478 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T11:08:20.542 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T11:08:20.587 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T11:08:20.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-04-26T11:08:20.706 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-04-26T11:08:20.710 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-04-26T11:08:20.772 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-04-26T11:08:20.774 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-04-26T11:08:20.839 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-04-26T11:08:20.841 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T11:08:20.841 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-04-26T11:08:20.899 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-04-26T11:08:20.899 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-04-26T11:08:20.923 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-04-26T11:08:20.923 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-04-26T11:08:20.941 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-04-26T11:08:20.941 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-04-26T11:08:20.941 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T11:08:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T11:08:21.012 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T11:08:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T11:08:21.040 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T11:08:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T11:08:21.065 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T11:08:21.066 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-04-26T11:08:21.257 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-04-26T11:08:21.285 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-04-26T11:08:21.311 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T11:08:21.323 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T11:08:21.339 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T11:08:21.340 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi096.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T11:13:37.427 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-04-26T11:13:37.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-26T11:13:37.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:13:37.547 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-04-26T11:13:37.707 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-26T11:13:37.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-26T11:13:37.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:13:37.783 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-04-26T11:13:37.862 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-26T11:13:37.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T11:13:37.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T11:13:37.947 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-04-26T11:13:38.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T11:13:38.023 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T11:13:38.035 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T11:13:38.035 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T11:13:38.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T11:13:38.045 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T11:13:38.045 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T11:13:38.049 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T11:13:38.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T11:13:38.076 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T11:13:38.090 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T11:13:38.090 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T11:13:38.104 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T11:13:38.113 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-04-26T11:13:38.124 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-04-26T11:13:38.127 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-04-26T11:13:38.137 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T11:13:38.138 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-04-26T11:13:38.138 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T11:13:38.150 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T11:13:38.151 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T11:13:38.164 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T11:13:38.181 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T11:13:38.205 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-04-26T11:13:38.208 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T11:13:38.208 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-04-26T11:13:38.208 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.208 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.208 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.209 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.217 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-04-26T11:13:38.220 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T11:13:38.220 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-04-26T11:13:38.220 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.220 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.221 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.221 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.259 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-04-26T11:13:38.292 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-04-26T11:13:38.336 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T11:13:38.382 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T11:13:38.532 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-04-26T11:13:38.542 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T11:13:38.542 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-04-26T11:13:38.542 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.542 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.542 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.542 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T11:13:38.544 INFO:teuthology.run_tasks:Running task install... 2024-04-26T11:13:38.554 DEBUG:teuthology.task.install:project ceph 2024-04-26T11:13:38.554 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1'}} 2024-04-26T11:13:38.554 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1'} 2024-04-26T11:13:38.554 INFO:teuthology.task.install:Using flavor: default 2024-04-26T11:13:38.561 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T11:13:38.561 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T11:13:38.562 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'tag': None, 'wait_for_package': False} 2024-04-26T11:13:38.562 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:13:38.564 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'tag': None, 'wait_for_package': False} 2024-04-26T11:13:38.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:13:38.565 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'tag': None, 'wait_for_package': False} 2024-04-26T11:13:38.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:13:38.713 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/ 2024-04-26T11:13:38.713 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3267.gbc94a2a9 2024-04-26T11:13:38.786 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/ 2024-04-26T11:13:38.786 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3267.gbc94a2a9 2024-04-26T11:13:38.802 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/ 2024-04-26T11:13:38.802 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3267.gbc94a2a9 2024-04-26T11:13:38.935 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T11:13:38.935 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:13:38.935 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T11:13:38.960 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T11:13:38.960 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T11:13:38.994 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T11:13:38.994 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:13:38.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T11:13:39.008 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T11:13:39.009 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:13:39.009 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T11:13:39.024 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T11:13:39.036 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T11:13:39.036 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T11:13:39.082 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T11:13:39.083 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T11:13:39.098 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T11:13:39.102 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T11:13:39.161 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-04-26T11:13:39.162 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-04-26T11:13:39.177 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T11:13:39.211 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T11:13:39.241 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-04-26T11:13:39.243 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-04-26T11:13:39.327 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T11:13:39.386 INFO:teuthology.orchestra.run.smithi194.stdout:48 files removed 2024-04-26T11:13:39.410 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-04-26T11:13:39.413 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T11:13:39.414 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-04-26T11:13:39.466 INFO:teuthology.orchestra.run.smithi096.stdout:48 files removed 2024-04-26T11:13:39.493 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T11:13:39.752 INFO:teuthology.orchestra.run.smithi033.stdout:48 files removed 2024-04-26T11:13:39.792 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T11:13:40.073 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 211 kB/s | 84 kB 00:00 2024-04-26T11:13:40.134 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 218 kB/s | 84 kB 00:00 2024-04-26T11:13:40.543 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 32 kB/s | 12 kB 00:00 2024-04-26T11:13:40.557 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 26 kB/s | 12 kB 00:00 2024-04-26T11:13:40.833 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 141 kB/s | 84 kB 00:00 2024-04-26T11:13:40.907 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-26T11:13:40.985 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-04-26T11:13:41.417 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-26T11:13:41.818 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-04-26T11:13:41.976 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-04-26T11:13:43.944 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-26T11:13:48.103 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.2 MB 00:00 2024-04-26T11:14:01.310 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 428 kB/s | 8.0 MB 00:19 2024-04-26T11:14:04.153 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-26T11:14:05.875 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 332 kB/s | 8.0 MB 00:24 2024-04-26T11:14:06.624 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 960 B/s | 16 kB 00:17 2024-04-26T11:14:08.170 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-26T11:14:11.001 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-04-26T11:14:12.120 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.2 MB 00:03 2024-04-26T11:14:13.259 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-26T11:14:13.764 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-04-26T11:14:15.159 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-26T11:14:15.394 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T11:14:15.394 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T11:14:15.406 INFO:teuthology.orchestra.run.smithi194.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T11:14:15.406 INFO:teuthology.orchestra.run.smithi194.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T11:14:15.407 INFO:teuthology.orchestra.run.smithi194.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T11:14:15.408 INFO:teuthology.orchestra.run.smithi194.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T11:14:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T11:14:15.411 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T11:14:15.412 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T11:14:15.473 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T11:14:15.487 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 6.6 k 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 890 k 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 145 k 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 1.5 M 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 142 k 2024-04-26T11:14:15.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 3.5 M 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 7.4 M 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 49 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 11 M 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 49 M 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 273 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 762 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 224 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 27 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 18 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T11:14:15.489 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 32 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 746 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 127 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 162 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 320 k 2024-04-26T11:14:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 298 k 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 99 k 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 85 k 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 170 k 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.4 M 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T11:14:15.491 INFO:teuthology.orchestra.run.smithi194.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 21 M 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 34 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 2.4 M 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 249 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 4.7 M 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 17 M 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 16 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 25 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T11:14:15.492 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 163 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 501 k 2024-04-26T11:14:15.493 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T11:14:15.494 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 45 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 135 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T11:14:15.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T11:14:15.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T11:14:15.497 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T11:14:15.498 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T11:14:15.499 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout:Install 148 Packages 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 4 Packages 2024-04-26T11:14:15.500 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:14:15.501 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 222 M 2024-04-26T11:14:15.501 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T11:14:15.847 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-26T11:14:16.590 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-26T11:14:18.103 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-26T11:14:22.008 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-04-26T11:14:23.116 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-26T11:14:24.585 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T11:14:24.586 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T11:14:24.608 INFO:teuthology.orchestra.run.smithi033.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T11:14:24.609 INFO:teuthology.orchestra.run.smithi033.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T11:14:24.615 INFO:teuthology.orchestra.run.smithi033.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T11:14:24.615 INFO:teuthology.orchestra.run.smithi033.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T11:14:24.616 INFO:teuthology.orchestra.run.smithi033.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T11:14:24.619 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T11:14:24.620 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T11:14:24.697 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-04-26T11:14:24.740 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2024-04-26T11:14:24.740 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-04-26T11:14:24.740 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2024-04-26T11:14:24.740 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 6.6 k 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 890 k 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 145 k 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 1.5 M 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 142 k 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 3.5 M 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 7.4 M 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 49 k 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 11 M 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 49 M 2024-04-26T11:14:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 273 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 762 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 224 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 27 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 18 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 32 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 746 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 127 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T11:14:24.742 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 162 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 320 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 298 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 99 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 85 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 170 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T11:14:24.743 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.4 M 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 21 M 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 34 k 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 2.4 M 2024-04-26T11:14:24.744 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 249 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 4.7 M 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 17 M 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 16 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 25 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T11:14:24.745 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 163 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 501 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T11:14:24.746 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 45 k 2024-04-26T11:14:24.747 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 135 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T11:14:24.748 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T11:14:24.749 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T11:14:24.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T11:14:24.751 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2024-04-26T11:14:24.752 INFO:teuthology.orchestra.run.smithi033.stdout:Install 148 Packages 2024-04-26T11:14:24.753 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 4 Packages 2024-04-26T11:14:24.753 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:14:24.753 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 222 M 2024-04-26T11:14:24.753 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-04-26T11:14:25.234 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T11:14:25.234 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T11:14:25.245 INFO:teuthology.orchestra.run.smithi096.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T11:14:25.246 INFO:teuthology.orchestra.run.smithi096.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T11:14:25.246 INFO:teuthology.orchestra.run.smithi096.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T11:14:25.247 INFO:teuthology.orchestra.run.smithi096.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T11:14:25.249 INFO:teuthology.orchestra.run.smithi096.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T11:14:25.251 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T11:14:25.252 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T11:14:25.311 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 6.6 k 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 890 k 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 145 k 2024-04-26T11:14:25.330 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 1.5 M 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 142 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 3.5 M 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 7.4 M 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 49 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 11 M 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 49 M 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 273 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 762 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 224 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 27 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 18 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T11:14:25.331 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 32 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 746 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 127 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 162 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 320 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 298 k 2024-04-26T11:14:25.332 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 99 k 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 85 k 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 170 k 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.4 M 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T11:14:25.333 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 21 M 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 34 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 2.4 M 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 249 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 4.7 M 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 17 M 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 16 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 25 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T11:14:25.334 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 163 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 501 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T11:14:25.335 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T11:14:25.336 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 45 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 135 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T11:14:25.337 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T11:14:25.338 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T11:14:25.339 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T11:14:25.340 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T11:14:25.341 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout:Install 148 Packages 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 4 Packages 2024-04-26T11:14:25.342 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:14:25.343 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 222 M 2024-04-26T11:14:25.343 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-26T11:14:25.901 INFO:teuthology.orchestra.run.smithi033.stdout:(1/152): ceph-19.0.0-3267.gbc94a2a9.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-26T11:14:26.143 INFO:teuthology.orchestra.run.smithi033.stdout:(2/152): ceph-fuse-19.0.0-3267.gbc94a2a9.el9.x8 3.6 MB/s | 890 kB 00:00 2024-04-26T11:14:26.227 INFO:teuthology.orchestra.run.smithi033.stdout:(3/152): ceph-immutable-object-cache-19.0.0-326 1.7 MB/s | 145 kB 00:00 2024-04-26T11:14:26.343 INFO:teuthology.orchestra.run.smithi033.stdout:(4/152): ceph-base-19.0.0-3267.gbc94a2a9.el9.x8 7.4 MB/s | 5.4 MB 00:00 2024-04-26T11:14:26.510 INFO:teuthology.orchestra.run.smithi033.stdout:(5/152): ceph-mds-19.0.0-3267.gbc94a2a9.el9.x86 8.4 MB/s | 2.4 MB 00:00 2024-04-26T11:14:26.594 INFO:teuthology.orchestra.run.smithi033.stdout:(6/152): ceph-mgr-19.0.0-3267.gbc94a2a9.el9.x86 6.1 MB/s | 1.5 MB 00:00 2024-04-26T11:14:26.986 INFO:teuthology.orchestra.run.smithi033.stdout:(7/152): ceph-mon-19.0.0-3267.gbc94a2a9.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-04-26T11:14:28.144 INFO:teuthology.orchestra.run.smithi033.stdout:(8/152): ceph-common-19.0.0-3267.gbc94a2a9.el9. 8.4 MB/s | 21 MB 00:02 2024-04-26T11:14:28.219 INFO:teuthology.orchestra.run.smithi033.stdout:(9/152): ceph-selinux-19.0.0-3267.gbc94a2a9.el9 341 kB/s | 25 kB 00:00 2024-04-26T11:14:28.578 INFO:teuthology.orchestra.run.smithi033.stdout:(10/152): ceph-osd-19.0.0-3267.gbc94a2a9.el9.x8 8.5 MB/s | 17 MB 00:01 2024-04-26T11:14:28.637 INFO:teuthology.orchestra.run.smithi033.stdout:(11/152): cephfs-mirror-19.0.0-3267.gbc94a2a9.e 3.8 MB/s | 224 kB 00:00 2024-04-26T11:14:28.795 INFO:teuthology.orchestra.run.smithi033.stdout:(12/152): ceph-radosgw-19.0.0-3267.gbc94a2a9.el 5.9 MB/s | 11 MB 00:01 2024-04-26T11:14:28.837 INFO:teuthology.orchestra.run.smithi033.stdout:(13/152): cephfs-shell-19.0.0-3267.gbc94a2a9.el 135 kB/s | 27 kB 00:00 2024-04-26T11:14:28.879 INFO:teuthology.orchestra.run.smithi033.stdout:(14/152): libcephfs-devel-19.0.0-3267.gbc94a2a9 385 kB/s | 32 kB 00:00 2024-04-26T11:14:28.954 INFO:teuthology.orchestra.run.smithi033.stdout:(15/152): libcephfs2-19.0.0-3267.gbc94a2a9.el9. 6.3 MB/s | 746 kB 00:00 2024-04-26T11:14:28.979 INFO:teuthology.orchestra.run.smithi033.stdout:(16/152): libcephsqlite-19.0.0-3267.gbc94a2a9.e 1.6 MB/s | 163 kB 00:00 2024-04-26T11:14:29.038 INFO:teuthology.orchestra.run.smithi033.stdout:(17/152): librados-devel-19.0.0-3267.gbc94a2a9. 1.5 MB/s | 127 kB 00:00 2024-04-26T11:14:29.121 INFO:teuthology.orchestra.run.smithi033.stdout:(18/152): libradosstriper1-19.0.0-3267.gbc94a2a 3.5 MB/s | 501 kB 00:00 2024-04-26T11:14:29.222 INFO:teuthology.orchestra.run.smithi033.stdout:(19/152): python3-ceph-argparse-19.0.0-3267.gbc 455 kB/s | 45 kB 00:00 2024-04-26T11:14:29.306 INFO:teuthology.orchestra.run.smithi033.stdout:(20/152): python3-ceph-common-19.0.0-3267.gbc94 1.6 MB/s | 135 kB 00:00 2024-04-26T11:14:29.398 INFO:teuthology.orchestra.run.smithi033.stdout:(21/152): python3-cephfs-19.0.0-3267.gbc94a2a9. 1.7 MB/s | 162 kB 00:00 2024-04-26T11:14:29.548 INFO:teuthology.orchestra.run.smithi033.stdout:(22/152): python3-rados-19.0.0-3267.gbc94a2a9.e 2.1 MB/s | 320 kB 00:00 2024-04-26T11:14:29.640 INFO:teuthology.orchestra.run.smithi033.stdout:(23/152): python3-rbd-19.0.0-3267.gbc94a2a9.el9 3.2 MB/s | 298 kB 00:00 2024-04-26T11:14:29.815 INFO:teuthology.orchestra.run.smithi033.stdout:(24/152): librgw2-19.0.0-3267.gbc94a2a9.el9.x86 6.9 MB/s | 5.4 MB 00:00 2024-04-26T11:14:29.840 INFO:teuthology.orchestra.run.smithi033.stdout:(25/152): python3-rgw-19.0.0-3267.gbc94a2a9.el9 499 kB/s | 99 kB 00:00 2024-04-26T11:14:29.916 INFO:teuthology.orchestra.run.smithi033.stdout:(26/152): rbd-fuse-19.0.0-3267.gbc94a2a9.el9.x8 859 kB/s | 85 kB 00:00 2024-04-26T11:14:30.032 INFO:teuthology.orchestra.run.smithi033.stdout:(27/152): rbd-nbd-19.0.0-3267.gbc94a2a9.el9.x86 1.4 MB/s | 170 kB 00:00 2024-04-26T11:14:30.108 INFO:teuthology.orchestra.run.smithi033.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3267.g 462 kB/s | 34 kB 00:00 2024-04-26T11:14:30.208 INFO:teuthology.orchestra.run.smithi033.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3267.gbc94a2a 1.4 MB/s | 142 kB 00:00 2024-04-26T11:14:30.433 INFO:teuthology.orchestra.run.smithi033.stdout:(30/152): rbd-mirror-19.0.0-3267.gbc94a2a9.el9. 5.2 MB/s | 3.1 MB 00:00 2024-04-26T11:14:30.725 INFO:teuthology.orchestra.run.smithi033.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3267.gbc94a 6.9 MB/s | 3.5 MB 00:00 2024-04-26T11:14:30.809 INFO:teuthology.orchestra.run.smithi033.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3267.gbc 2.9 MB/s | 249 kB 00:00 2024-04-26T11:14:30.884 INFO:teuthology.orchestra.run.smithi033.stdout:(33/152): ceph-mgr-rook-19.0.0-3267.gbc94a2a9.e 664 kB/s | 49 kB 00:00 2024-04-26T11:14:30.959 INFO:teuthology.orchestra.run.smithi033.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3267.gb 223 kB/s | 16 kB 00:00 2024-04-26T11:14:31.051 INFO:teuthology.orchestra.run.smithi033.stdout:(35/152): ceph-volume-19.0.0-3267.gbc94a2a9.el9 2.9 MB/s | 273 kB 00:00 2024-04-26T11:14:31.218 INFO:teuthology.orchestra.run.smithi033.stdout:(36/152): cephadm-19.0.0-3267.gbc94a2a9.el9.noa 4.5 MB/s | 762 kB 00:00 2024-04-26T11:14:31.277 INFO:teuthology.orchestra.run.smithi033.stdout:(37/152): cephfs-top-19.0.0-3267.gbc94a2a9.el9. 318 kB/s | 18 kB 00:00 2024-04-26T11:14:31.811 INFO:teuthology.orchestra.run.smithi033.stdout:(38/152): ceph-test-19.0.0-3267.gbc94a2a9.el9.x 14 MB/s | 49 MB 00:03 2024-04-26T11:14:32.052 INFO:teuthology.orchestra.run.smithi033.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-04-26T11:14:32.086 INFO:teuthology.orchestra.run.smithi033.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 99 kB/s | 80 kB 00:00 2024-04-26T11:14:32.219 INFO:teuthology.orchestra.run.smithi033.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-26T11:14:32.270 INFO:teuthology.orchestra.run.smithi033.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-04-26T11:14:32.312 INFO:teuthology.orchestra.run.smithi033.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-04-26T11:14:32.450 INFO:teuthology.orchestra.run.smithi194.stdout:(1/152): ceph-19.0.0-3267.gbc94a2a9.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-26T11:14:32.587 INFO:teuthology.orchestra.run.smithi033.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 3.4 MB/s | 1.2 MB 00:00 2024-04-26T11:14:32.612 INFO:teuthology.orchestra.run.smithi033.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 313 kB/s | 106 kB 00:00 2024-04-26T11:14:32.637 INFO:teuthology.orchestra.run.smithi033.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 417 kB/s | 135 kB 00:00 2024-04-26T11:14:32.663 INFO:teuthology.orchestra.run.smithi033.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.7 MB/s | 125 kB 00:00 2024-04-26T11:14:32.696 INFO:teuthology.orchestra.run.smithi033.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 215 kB 00:00 2024-04-26T11:14:32.780 INFO:teuthology.orchestra.run.smithi033.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 689 kB 00:00 2024-04-26T11:14:32.783 INFO:teuthology.orchestra.run.smithi194.stdout:(2/152): ceph-base-19.0.0-3267.gbc94a2a9.el9.x8 9.4 MB/s | 5.4 MB 00:00 2024-04-26T11:14:32.833 INFO:teuthology.orchestra.run.smithi194.stdout:(3/152): ceph-immutable-object-cache-19.0.0-326 2.9 MB/s | 145 kB 00:00 2024-04-26T11:14:32.872 INFO:teuthology.orchestra.run.smithi033.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 943 kB 00:00 2024-04-26T11:14:32.897 INFO:teuthology.orchestra.run.smithi033.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 541 kB/s | 107 kB 00:00 2024-04-26T11:14:32.922 INFO:teuthology.orchestra.run.smithi033.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 157 kB/s | 21 kB 00:00 2024-04-26T11:14:32.956 INFO:teuthology.orchestra.run.smithi033.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-26T11:14:32.981 INFO:teuthology.orchestra.run.smithi033.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 600 kB/s | 33 kB 00:00 2024-04-26T11:14:33.000 INFO:teuthology.orchestra.run.smithi194.stdout:(4/152): ceph-mds-19.0.0-3267.gbc94a2a9.el9.x86 14 MB/s | 2.4 MB 00:00 2024-04-26T11:14:33.109 INFO:teuthology.orchestra.run.smithi194.stdout:(5/152): ceph-mgr-19.0.0-3267.gbc94a2a9.el9.x86 14 MB/s | 1.5 MB 00:00 2024-04-26T11:14:33.156 INFO:teuthology.orchestra.run.smithi033.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 93 kB/s | 18 kB 00:00 2024-04-26T11:14:33.176 INFO:teuthology.orchestra.run.smithi194.stdout:(6/152): ceph-fuse-19.0.0-3267.gbc94a2a9.el9.x8 1.2 MB/s | 890 kB 00:00 2024-04-26T11:14:33.232 INFO:teuthology.orchestra.run.smithi033.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.8 MB/s | 3.0 MB 00:00 2024-04-26T11:14:33.257 INFO:teuthology.orchestra.run.smithi033.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 41 kB/s | 11 kB 00:00 2024-04-26T11:14:33.290 INFO:teuthology.orchestra.run.smithi033.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-04-26T11:14:33.324 INFO:teuthology.orchestra.run.smithi033.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 707 kB/s | 45 kB 00:00 2024-04-26T11:14:33.366 INFO:teuthology.orchestra.run.smithi033.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.1 MB/s | 662 kB 00:00 2024-04-26T11:14:33.393 INFO:teuthology.orchestra.run.smithi194.stdout:(7/152): ceph-mon-19.0.0-3267.gbc94a2a9.el9.x86 17 MB/s | 4.7 MB 00:00 2024-04-26T11:14:33.423 INFO:teuthology.orchestra.run.smithi033.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 4.5 MB/s | 582 kB 00:00 2024-04-26T11:14:33.451 INFO:teuthology.orchestra.run.smithi033.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 215 kB/s | 25 kB 00:00 2024-04-26T11:14:33.475 INFO:teuthology.orchestra.run.smithi033.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 449 kB/s | 47 kB 00:00 2024-04-26T11:14:33.508 INFO:teuthology.orchestra.run.smithi033.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.0 MB/s | 292 kB 00:00 2024-04-26T11:14:33.542 INFO:teuthology.orchestra.run.smithi033.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-26T11:14:33.567 INFO:teuthology.orchestra.run.smithi033.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 402 kB/s | 35 kB 00:00 2024-04-26T11:14:33.850 INFO:teuthology.orchestra.run.smithi033.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-26T11:14:34.010 INFO:teuthology.orchestra.run.smithi194.stdout:(8/152): ceph-radosgw-19.0.0-3267.gbc94a2a9.el9 17 MB/s | 11 MB 00:00 2024-04-26T11:14:34.060 INFO:teuthology.orchestra.run.smithi194.stdout:(9/152): ceph-selinux-19.0.0-3267.gbc94a2a9.el9 506 kB/s | 25 kB 00:00 2024-04-26T11:14:34.143 INFO:teuthology.orchestra.run.smithi033.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-04-26T11:14:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-04-26T11:14:34.243 INFO:teuthology.orchestra.run.smithi033.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 2.4 MB/s | 911 kB 00:00 2024-04-26T11:14:34.268 INFO:teuthology.orchestra.run.smithi033.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-04-26T11:14:34.510 INFO:teuthology.orchestra.run.smithi033.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-04-26T11:14:34.569 INFO:teuthology.orchestra.run.smithi033.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 294 kB/s | 87 kB 00:00 2024-04-26T11:14:34.661 INFO:teuthology.orchestra.run.smithi033.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-04-26T11:14:34.794 INFO:teuthology.orchestra.run.smithi033.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-04-26T11:14:34.911 INFO:teuthology.orchestra.run.smithi033.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-26T11:14:34.936 INFO:teuthology.orchestra.run.smithi033.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 957 kB/s | 248 kB 00:00 2024-04-26T11:14:34.962 INFO:teuthology.orchestra.run.smithi033.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 167 kB/s | 26 kB 00:00 2024-04-26T11:14:34.987 INFO:teuthology.orchestra.run.smithi033.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 393 kB/s | 19 kB 00:00 2024-04-26T11:14:35.012 INFO:teuthology.orchestra.run.smithi033.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.3 MB/s | 172 kB 00:00 2024-04-26T11:14:35.037 INFO:teuthology.orchestra.run.smithi033.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 481 kB/s | 35 kB 00:00 2024-04-26T11:14:35.187 INFO:teuthology.orchestra.run.smithi033.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 250 kB 00:00 2024-04-26T11:14:35.313 INFO:teuthology.orchestra.run.smithi033.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-04-26T11:14:35.347 INFO:teuthology.orchestra.run.smithi033.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 725 kB/s | 222 kB 00:00 2024-04-26T11:14:35.372 INFO:teuthology.orchestra.run.smithi033.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 459 kB/s | 77 kB 00:00 2024-04-26T11:14:35.414 INFO:teuthology.orchestra.run.smithi033.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 434 kB/s | 42 kB 00:00 2024-04-26T11:14:35.447 INFO:teuthology.orchestra.run.smithi033.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.6 MB/s | 269 kB 00:00 2024-04-26T11:14:35.473 INFO:teuthology.orchestra.run.smithi033.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-04-26T11:14:35.506 INFO:teuthology.orchestra.run.smithi033.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 279 kB 00:00 2024-04-26T11:14:35.531 INFO:teuthology.orchestra.run.smithi033.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 931 kB/s | 51 kB 00:00 2024-04-26T11:14:35.557 INFO:teuthology.orchestra.run.smithi033.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 673 kB/s | 54 kB 00:00 2024-04-26T11:14:35.815 INFO:teuthology.orchestra.run.smithi033.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 169 kB/s | 42 kB 00:00 2024-04-26T11:14:35.840 INFO:teuthology.orchestra.run.smithi033.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 163 kB/s | 44 kB 00:00 2024-04-26T11:14:36.232 INFO:teuthology.orchestra.run.smithi033.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-26T11:14:36.266 INFO:teuthology.orchestra.run.smithi033.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 491 kB/s | 221 kB 00:00 2024-04-26T11:14:36.294 INFO:teuthology.orchestra.run.smithi033.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 83 kB/s | 36 kB 00:00 2024-04-26T11:14:36.316 INFO:teuthology.orchestra.run.smithi033.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 592 kB/s | 47 kB 00:00 2024-04-26T11:14:36.375 INFO:teuthology.orchestra.run.smithi033.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-26T11:14:36.400 INFO:teuthology.orchestra.run.smithi033.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 538 kB/s | 58 kB 00:00 2024-04-26T11:14:36.426 INFO:teuthology.orchestra.run.smithi033.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 896 kB/s | 94 kB 00:00 2024-04-26T11:14:36.451 INFO:teuthology.orchestra.run.smithi033.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-04-26T11:14:36.476 INFO:teuthology.orchestra.run.smithi033.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 612 kB/s | 45 kB 00:00 2024-04-26T11:14:36.501 INFO:teuthology.orchestra.run.smithi033.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 304 kB/s | 22 kB 00:00 2024-04-26T11:14:36.568 INFO:teuthology.orchestra.run.smithi033.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.3 MB/s | 863 kB 00:00 2024-04-26T11:14:36.861 INFO:teuthology.orchestra.run.smithi194.stdout:(10/152): ceph-test-19.0.0-3267.gbc94a2a9.el9.x 18 MB/s | 49 MB 00:02 2024-04-26T11:14:36.993 INFO:teuthology.orchestra.run.smithi033.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 64 kB/s | 19 kB 00:00 2024-04-26T11:14:37.061 INFO:teuthology.orchestra.run.smithi194.stdout:(11/152): ceph-common-19.0.0-3267.gbc94a2a9.el9 4.4 MB/s | 21 MB 00:04 2024-04-26T11:14:37.094 INFO:teuthology.orchestra.run.smithi194.stdout:(12/152): cephfs-mirror-19.0.0-3267.gbc94a2a9.e 962 kB/s | 224 kB 00:00 2024-04-26T11:14:37.102 INFO:teuthology.orchestra.run.smithi033.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 516 kB/s | 309 kB 00:00 2024-04-26T11:14:37.136 INFO:teuthology.orchestra.run.smithi194.stdout:(13/152): libcephfs-devel-19.0.0-3267.gbc94a2a9 764 kB/s | 32 kB 00:00 2024-04-26T11:14:37.144 INFO:teuthology.orchestra.run.smithi033.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 823 kB/s | 548 kB 00:00 2024-04-26T11:14:37.204 INFO:teuthology.orchestra.run.smithi194.stdout:(14/152): cephfs-shell-19.0.0-3267.gbc94a2a9.el 188 kB/s | 27 kB 00:00 2024-04-26T11:14:37.262 INFO:teuthology.orchestra.run.smithi194.stdout:(15/152): libcephfs2-19.0.0-3267.gbc94a2a9.el9. 5.8 MB/s | 746 kB 00:00 2024-04-26T11:14:37.304 INFO:teuthology.orchestra.run.smithi194.stdout:(16/152): libcephsqlite-19.0.0-3267.gbc94a2a9.e 1.6 MB/s | 163 kB 00:00 2024-04-26T11:14:37.329 INFO:teuthology.orchestra.run.smithi194.stdout:(17/152): librados-devel-19.0.0-3267.gbc94a2a9. 1.9 MB/s | 127 kB 00:00 2024-04-26T11:14:37.496 INFO:teuthology.orchestra.run.smithi194.stdout:(18/152): libradosstriper1-19.0.0-3267.gbc94a2a 2.5 MB/s | 501 kB 00:00 2024-04-26T11:14:37.663 INFO:teuthology.orchestra.run.smithi194.stdout:(19/152): ceph-osd-19.0.0-3267.gbc94a2a9.el9.x8 3.8 MB/s | 17 MB 00:04 2024-04-26T11:14:37.689 INFO:teuthology.orchestra.run.smithi194.stdout:(20/152): python3-ceph-argparse-19.0.0-3267.gbc 235 kB/s | 45 kB 00:00 2024-04-26T11:14:37.772 INFO:teuthology.orchestra.run.smithi194.stdout:(21/152): librgw2-19.0.0-3267.gbc94a2a9.el9.x86 12 MB/s | 5.4 MB 00:00 2024-04-26T11:14:37.797 INFO:teuthology.orchestra.run.smithi194.stdout:(22/152): python3-ceph-common-19.0.0-3267.gbc94 1.0 MB/s | 135 kB 00:00 2024-04-26T11:14:37.822 INFO:teuthology.orchestra.run.smithi194.stdout:(23/152): python3-cephfs-19.0.0-3267.gbc94a2a9. 1.2 MB/s | 162 kB 00:00 2024-04-26T11:14:37.848 INFO:teuthology.orchestra.run.smithi194.stdout:(24/152): python3-rados-19.0.0-3267.gbc94a2a9.e 4.1 MB/s | 320 kB 00:00 2024-04-26T11:14:37.876 INFO:teuthology.orchestra.run.smithi194.stdout:(25/152): python3-rbd-19.0.0-3267.gbc94a2a9.el9 3.7 MB/s | 298 kB 00:00 2024-04-26T11:14:37.898 INFO:teuthology.orchestra.run.smithi194.stdout:(26/152): rbd-fuse-19.0.0-3267.gbc94a2a9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-26T11:14:37.923 INFO:teuthology.orchestra.run.smithi194.stdout:(27/152): python3-rgw-19.0.0-3267.gbc94a2a9.el9 986 kB/s | 99 kB 00:00 2024-04-26T11:14:37.949 INFO:teuthology.orchestra.run.smithi194.stdout:(28/152): rbd-nbd-19.0.0-3267.gbc94a2a9.el9.x86 3.3 MB/s | 170 kB 00:00 2024-04-26T11:14:37.999 INFO:teuthology.orchestra.run.smithi194.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3267.g 455 kB/s | 34 kB 00:00 2024-04-26T11:14:38.024 INFO:teuthology.orchestra.run.smithi194.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3267.gbc94a2a 1.8 MB/s | 142 kB 00:00 2024-04-26T11:14:38.241 INFO:teuthology.orchestra.run.smithi194.stdout:(31/152): rbd-mirror-19.0.0-3267.gbc94a2a9.el9. 8.5 MB/s | 3.1 MB 00:00 2024-04-26T11:14:38.311 INFO:teuthology.orchestra.run.smithi033.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-04-26T11:14:38.358 INFO:teuthology.orchestra.run.smithi194.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3267.gbc 2.1 MB/s | 249 kB 00:00 2024-04-26T11:14:38.361 INFO:teuthology.orchestra.run.smithi033.stdout:(109/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 21 kB/s | 26 kB 00:01 2024-04-26T11:14:38.386 INFO:teuthology.orchestra.run.smithi033.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 39 kB/s | 49 kB 00:01 2024-04-26T11:14:38.412 INFO:teuthology.orchestra.run.smithi033.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 681 kB/s | 67 kB 00:00 2024-04-26T11:14:38.491 INFO:teuthology.orchestra.run.smithi194.stdout:(33/152): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-04-26T11:14:38.496 INFO:teuthology.orchestra.run.smithi033.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-04-26T11:14:38.541 INFO:teuthology.orchestra.run.smithi194.stdout:(34/152): ceph-mgr-rook-19.0.0-3267.gbc94a2a9.e 311 kB/s | 49 kB 00:00 2024-04-26T11:14:38.542 INFO:teuthology.orchestra.run.smithi194.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3267.gb 328 kB/s | 16 kB 00:00 2024-04-26T11:14:38.609 INFO:teuthology.orchestra.run.smithi194.stdout:(36/152): ceph-volume-19.0.0-3267.gbc94a2a9.el9 2.9 MB/s | 273 kB 00:00 2024-04-26T11:14:38.620 INFO:teuthology.orchestra.run.smithi033.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-04-26T11:14:38.644 INFO:teuthology.orchestra.run.smithi194.stdout:(37/152): cephadm-19.0.0-3267.gbc94a2a9.el9.noa 7.3 MB/s | 762 kB 00:00 2024-04-26T11:14:38.679 INFO:teuthology.orchestra.run.smithi033.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.0 MB/s | 548 kB 00:00 2024-04-26T11:14:38.704 INFO:teuthology.orchestra.run.smithi033.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 209 kB/s | 43 kB 00:00 2024-04-26T11:14:38.715 INFO:teuthology.orchestra.run.smithi194.stdout:(38/152): ceph-mgr-dashboard-19.0.0-3267.gbc94a 4.9 MB/s | 3.5 MB 00:00 2024-04-26T11:14:38.728 INFO:teuthology.orchestra.run.smithi194.stdout:(39/152): cephfs-top-19.0.0-3267.gbc94a2a9.el9. 153 kB/s | 18 kB 00:00 2024-04-26T11:14:38.729 INFO:teuthology.orchestra.run.smithi033.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 312 kB/s | 32 kB 00:00 2024-04-26T11:14:38.782 INFO:teuthology.orchestra.run.smithi033.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 197 kB/s | 15 kB 00:00 2024-04-26T11:14:38.815 INFO:teuthology.orchestra.run.smithi033.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-26T11:14:38.882 INFO:teuthology.orchestra.run.smithi033.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-26T11:14:38.916 INFO:teuthology.orchestra.run.smithi033.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.3 MB/s | 303 kB 00:00 2024-04-26T11:14:38.941 INFO:teuthology.orchestra.run.smithi033.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 288 kB/s | 35 kB 00:00 2024-04-26T11:14:38.975 INFO:teuthology.orchestra.run.smithi033.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-26T11:14:38.979 INFO:teuthology.orchestra.run.smithi194.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 241 kB/s | 80 kB 00:00 2024-04-26T11:14:39.012 INFO:teuthology.orchestra.run.smithi194.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 688 kB/s | 195 kB 00:00 2024-04-26T11:14:39.204 INFO:teuthology.orchestra.run.smithi194.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-04-26T11:14:39.229 INFO:teuthology.orchestra.run.smithi194.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-04-26T11:14:39.350 INFO:teuthology.orchestra.run.smithi033.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.7 MB/s | 2.0 MB 00:00 2024-04-26T11:14:39.375 INFO:teuthology.orchestra.run.smithi033.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 333 kB/s | 144 kB 00:00 2024-04-26T11:14:39.400 INFO:teuthology.orchestra.run.smithi033.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 25 kB/s | 11 kB 00:00 2024-04-26T11:14:39.592 INFO:teuthology.orchestra.run.smithi033.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 74 kB/s | 18 kB 00:00 2024-04-26T11:14:39.618 INFO:teuthology.orchestra.run.smithi033.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 107 kB/s | 23 kB 00:00 2024-04-26T11:14:39.643 INFO:teuthology.orchestra.run.smithi033.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 81 kB/s | 19 kB 00:00 2024-04-26T11:14:39.685 INFO:teuthology.orchestra.run.smithi033.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 216 kB/s | 20 kB 00:00 2024-04-26T11:14:39.838 INFO:teuthology.orchestra.run.smithi194.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 1.5 MB/s | 1.2 MB 00:00 2024-04-26T11:14:39.872 INFO:teuthology.orchestra.run.smithi194.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 160 kB/s | 106 kB 00:00 2024-04-26T11:14:40.005 INFO:teuthology.orchestra.run.smithi194.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 174 kB/s | 135 kB 00:00 2024-04-26T11:14:40.031 INFO:teuthology.orchestra.run.smithi194.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 650 kB/s | 125 kB 00:00 2024-04-26T11:14:40.131 INFO:teuthology.orchestra.run.smithi194.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 829 kB/s | 215 kB 00:00 2024-04-26T11:14:40.262 INFO:teuthology.orchestra.run.smithi033.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 1.7 MB/s | 1.0 MB 00:00 2024-04-26T11:14:40.293 INFO:teuthology.orchestra.run.smithi033.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 71 kB/s | 46 kB 00:00 2024-04-26T11:14:40.319 INFO:teuthology.orchestra.run.smithi033.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 124 kB/s | 79 kB 00:00 2024-04-26T11:14:40.411 INFO:teuthology.orchestra.run.smithi033.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 501 kB/s | 58 kB 00:00 2024-04-26T11:14:40.415 INFO:teuthology.orchestra.run.smithi194.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 689 kB 00:00 2024-04-26T11:14:40.602 INFO:teuthology.orchestra.run.smithi033.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 887 kB/s | 272 kB 00:00 2024-04-26T11:14:40.627 INFO:teuthology.orchestra.run.smithi033.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 53 kB/s | 16 kB 00:00 2024-04-26T11:14:40.654 INFO:teuthology.orchestra.run.smithi033.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 373 kB/s | 90 kB 00:00 2024-04-26T11:14:40.656 INFO:teuthology.orchestra.run.smithi194.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 203 kB/s | 107 kB 00:00 2024-04-26T11:14:40.703 INFO:teuthology.orchestra.run.smithi033.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 385 kB/s | 28 kB 00:00 2024-04-26T11:14:40.728 INFO:teuthology.orchestra.run.smithi033.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 313 kB/s | 31 kB 00:00 2024-04-26T11:14:40.799 INFO:teuthology.orchestra.run.smithi194.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-04-26T11:14:40.812 INFO:teuthology.orchestra.run.smithi033.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-26T11:14:40.815 INFO:teuthology.orchestra.run.smithi194.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 54 kB/s | 21 kB 00:00 2024-04-26T11:14:40.846 INFO:teuthology.orchestra.run.smithi033.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 515 kB/s | 59 kB 00:00 2024-04-26T11:14:40.871 INFO:teuthology.orchestra.run.smithi033.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 256 kB/s | 36 kB 00:00 2024-04-26T11:14:40.883 INFO:teuthology.orchestra.run.smithi194.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 1.4 MB/s | 313 kB 00:00 2024-04-26T11:14:40.896 INFO:teuthology.orchestra.run.smithi033.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 613 kB/s | 51 kB 00:00 2024-04-26T11:14:40.899 INFO:teuthology.orchestra.run.smithi194.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-04-26T11:14:41.013 INFO:teuthology.orchestra.run.smithi033.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-04-26T11:14:41.038 INFO:teuthology.orchestra.run.smithi033.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 540 kB/s | 90 kB 00:00 2024-04-26T11:14:41.158 INFO:teuthology.orchestra.run.smithi194.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 66 kB/s | 18 kB 00:00 2024-04-26T11:14:41.175 INFO:teuthology.orchestra.run.smithi194.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 40 kB/s | 11 kB 00:00 2024-04-26T11:14:41.197 INFO:teuthology.orchestra.run.smithi033.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.4 MB/s | 427 kB 00:00 2024-04-26T11:14:41.239 INFO:teuthology.orchestra.run.smithi033.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 89 kB/s | 20 kB 00:00 2024-04-26T11:14:41.264 INFO:teuthology.orchestra.run.smithi033.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 853 kB/s | 192 kB 00:00 2024-04-26T11:14:41.283 INFO:teuthology.orchestra.run.smithi194.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.4 MB/s | 3.0 MB 00:00 2024-04-26T11:14:41.309 INFO:teuthology.orchestra.run.smithi194.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-04-26T11:14:41.334 INFO:teuthology.orchestra.run.smithi194.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 286 kB/s | 45 kB 00:00 2024-04-26T11:14:41.359 INFO:teuthology.orchestra.run.smithi194.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-04-26T11:14:41.409 INFO:teuthology.orchestra.run.smithi194.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 7.7 MB/s | 582 kB 00:00 2024-04-26T11:14:41.435 INFO:teuthology.orchestra.run.smithi194.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 257 kB/s | 25 kB 00:00 2024-04-26T11:14:41.460 INFO:teuthology.orchestra.run.smithi194.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-04-26T11:14:41.485 INFO:teuthology.orchestra.run.smithi194.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-26T11:14:41.510 INFO:teuthology.orchestra.run.smithi194.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 709 kB/s | 35 kB 00:00 2024-04-26T11:14:41.535 INFO:teuthology.orchestra.run.smithi194.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-04-26T11:14:41.769 INFO:teuthology.orchestra.run.smithi194.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-04-26T11:14:41.831 INFO:teuthology.orchestra.run.smithi033.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-04-26T11:14:41.886 INFO:teuthology.orchestra.run.smithi194.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-26T11:14:41.914 INFO:teuthology.orchestra.run.smithi033.stdout:(149/152): librbd1-19.0.0-3267.gbc94a2a9.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-04-26T11:14:41.920 INFO:teuthology.orchestra.run.smithi194.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-04-26T11:14:41.945 INFO:teuthology.orchestra.run.smithi194.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-04-26T11:14:41.970 INFO:teuthology.orchestra.run.smithi194.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-26T11:14:41.981 INFO:teuthology.orchestra.run.smithi033.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 1.3 MB 00:00 2024-04-26T11:14:42.029 INFO:teuthology.orchestra.run.smithi194.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-26T11:14:42.054 INFO:teuthology.orchestra.run.smithi194.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-04-26T11:14:42.082 INFO:teuthology.orchestra.run.smithi033.stdout:(151/152): librados2-19.0.0-3267.gbc94a2a9.el9. 4.0 MB/s | 3.4 MB 00:00 2024-04-26T11:14:42.157 INFO:teuthology.orchestra.run.smithi033.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 10 MB/s | 2.4 MB 00:00 2024-04-26T11:14:42.174 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-04-26T11:14:42.174 INFO:teuthology.orchestra.run.smithi033.stdout:Total 13 MB/s | 222 MB 00:17 2024-04-26T11:14:42.196 INFO:teuthology.orchestra.run.smithi194.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-04-26T11:14:42.271 INFO:teuthology.orchestra.run.smithi194.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 6.0 MB 00:00 2024-04-26T11:14:42.305 INFO:teuthology.orchestra.run.smithi194.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 986 kB/s | 244 kB 00:00 2024-04-26T11:14:42.330 INFO:teuthology.orchestra.run.smithi194.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-04-26T11:14:42.355 INFO:teuthology.orchestra.run.smithi194.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 516 kB/s | 26 kB 00:00 2024-04-26T11:14:42.380 INFO:teuthology.orchestra.run.smithi194.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 256 kB/s | 19 kB 00:00 2024-04-26T11:14:42.406 INFO:teuthology.orchestra.run.smithi194.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-26T11:14:42.431 INFO:teuthology.orchestra.run.smithi194.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 698 kB/s | 35 kB 00:00 2024-04-26T11:14:42.523 INFO:teuthology.orchestra.run.smithi194.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-04-26T11:14:42.548 INFO:teuthology.orchestra.run.smithi194.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-04-26T11:14:42.573 INFO:teuthology.orchestra.run.smithi194.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-04-26T11:14:42.598 INFO:teuthology.orchestra.run.smithi194.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-04-26T11:14:42.624 INFO:teuthology.orchestra.run.smithi194.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 42 kB 00:00 2024-04-26T11:14:42.631 INFO:teuthology.orchestra.run.smithi096.stdout:(1/152): ceph-19.0.0-3267.gbc94a2a9.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-04-26T11:14:42.649 INFO:teuthology.orchestra.run.smithi194.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-26T11:14:42.674 INFO:teuthology.orchestra.run.smithi194.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-26T11:14:42.699 INFO:teuthology.orchestra.run.smithi194.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-26T11:14:42.716 INFO:teuthology.orchestra.run.smithi194.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 764 kB/s | 51 kB 00:00 2024-04-26T11:14:42.742 INFO:teuthology.orchestra.run.smithi194.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 805 kB/s | 54 kB 00:00 2024-04-26T11:14:42.808 INFO:teuthology.orchestra.run.smithi194.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 454 kB/s | 42 kB 00:00 2024-04-26T11:14:42.976 INFO:teuthology.orchestra.run.smithi194.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 190 kB/s | 44 kB 00:00 2024-04-26T11:14:43.126 INFO:teuthology.orchestra.run.smithi194.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-04-26T11:14:43.159 INFO:teuthology.orchestra.run.smithi194.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 634 kB/s | 221 kB 00:00 2024-04-26T11:14:43.176 INFO:teuthology.orchestra.run.smithi194.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 180 kB/s | 36 kB 00:00 2024-04-26T11:14:43.182 INFO:teuthology.orchestra.run.smithi096.stdout:(2/152): ceph-fuse-19.0.0-3267.gbc94a2a9.el9.x8 1.6 MB/s | 890 kB 00:00 2024-04-26T11:14:43.202 INFO:teuthology.orchestra.run.smithi194.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 305 kB 00:00 2024-04-26T11:14:43.218 INFO:teuthology.orchestra.run.smithi194.stdout:(98/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-04-26T11:14:43.235 INFO:teuthology.orchestra.run.smithi194.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 991 kB/s | 58 kB 00:00 2024-04-26T11:14:43.261 INFO:teuthology.orchestra.run.smithi194.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-04-26T11:14:43.278 INFO:teuthology.orchestra.run.smithi194.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-26T11:14:43.294 INFO:teuthology.orchestra.run.smithi194.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 762 kB/s | 45 kB 00:00 2024-04-26T11:14:43.307 INFO:teuthology.orchestra.run.smithi096.stdout:(3/152): ceph-immutable-object-cache-19.0.0-326 1.1 MB/s | 145 kB 00:00 2024-04-26T11:14:43.311 INFO:teuthology.orchestra.run.smithi194.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 442 kB/s | 22 kB 00:00 2024-04-26T11:14:43.362 INFO:teuthology.orchestra.run.smithi194.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-04-26T11:14:43.637 INFO:teuthology.orchestra.run.smithi194.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-04-26T11:14:43.757 INFO:teuthology.orchestra.run.smithi096.stdout:(4/152): ceph-base-19.0.0-3267.gbc94a2a9.el9.x8 3.7 MB/s | 5.4 MB 00:01 2024-04-26T11:14:43.807 INFO:teuthology.orchestra.run.smithi096.stdout:(5/152): ceph-mds-19.0.0-3267.gbc94a2a9.el9.x86 4.7 MB/s | 2.4 MB 00:00 2024-04-26T11:14:43.887 INFO:teuthology.orchestra.run.smithi194.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 537 kB/s | 309 kB 00:00 2024-04-26T11:14:44.057 INFO:teuthology.orchestra.run.smithi096.stdout:(6/152): ceph-mgr-19.0.0-3267.gbc94a2a9.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-04-26T11:14:44.079 INFO:teuthology.orchestra.run.smithi194.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 699 kB/s | 548 kB 00:00 2024-04-26T11:14:44.263 INFO:teuthology.orchestra.run.smithi194.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 69 kB/s | 26 kB 00:00 2024-04-26T11:14:44.430 INFO:teuthology.orchestra.run.smithi194.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-04-26T11:14:44.613 INFO:teuthology.orchestra.run.smithi194.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 192 kB/s | 67 kB 00:00 2024-04-26T11:14:44.783 INFO:teuthology.orchestra.run.smithi096.stdout:(7/152): ceph-common-19.0.0-3267.gbc94a2a9.el9. 8.6 MB/s | 21 MB 00:02 2024-04-26T11:14:44.858 INFO:teuthology.orchestra.run.smithi096.stdout:(8/152): ceph-mon-19.0.0-3267.gbc94a2a9.el9.x86 4.5 MB/s | 4.7 MB 00:01 2024-04-26T11:14:44.863 INFO:teuthology.orchestra.run.smithi194.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 348 kB/s | 151 kB 00:00 2024-04-26T11:14:44.925 INFO:teuthology.orchestra.run.smithi096.stdout:(9/152): ceph-selinux-19.0.0-3267.gbc94a2a9.el9 378 kB/s | 25 kB 00:00 2024-04-26T11:14:45.002 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-04-26T11:14:45.173 INFO:teuthology.orchestra.run.smithi194.stdout:(112/152): libarrow-9.0.0-11.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-04-26T11:14:45.258 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-04-26T11:14:45.258 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-04-26T11:14:45.289 INFO:teuthology.orchestra.run.smithi194.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-04-26T11:14:45.464 INFO:teuthology.orchestra.run.smithi194.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 913 kB/s | 548 kB 00:00 2024-04-26T11:14:45.515 INFO:teuthology.orchestra.run.smithi194.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 127 kB/s | 43 kB 00:00 2024-04-26T11:14:45.557 INFO:teuthology.orchestra.run.smithi194.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 120 kB/s | 32 kB 00:00 2024-04-26T11:14:45.600 INFO:teuthology.orchestra.run.smithi096.stdout:(10/152): ceph-osd-19.0.0-3267.gbc94a2a9.el9.x8 11 MB/s | 17 MB 00:01 2024-04-26T11:14:45.726 INFO:teuthology.orchestra.run.smithi096.stdout:(11/152): ceph-radosgw-19.0.0-3267.gbc94a2a9.el 11 MB/s | 11 MB 00:00 2024-04-26T11:14:45.748 INFO:teuthology.orchestra.run.smithi194.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 53 kB/s | 15 kB 00:00 2024-04-26T11:14:45.751 INFO:teuthology.orchestra.run.smithi096.stdout:(12/152): cephfs-mirror-19.0.0-3267.gbc94a2a9.e 1.5 MB/s | 224 kB 00:00 2024-04-26T11:14:45.776 INFO:teuthology.orchestra.run.smithi096.stdout:(13/152): cephfs-shell-19.0.0-3267.gbc94a2a9.el 542 kB/s | 27 kB 00:00 2024-04-26T11:14:45.809 INFO:teuthology.orchestra.run.smithi096.stdout:(14/152): libcephfs-devel-19.0.0-3267.gbc94a2a9 542 kB/s | 32 kB 00:00 2024-04-26T11:14:45.852 INFO:teuthology.orchestra.run.smithi096.stdout:(15/152): libcephfs2-19.0.0-3267.gbc94a2a9.el9. 9.7 MB/s | 746 kB 00:00 2024-04-26T11:14:45.877 INFO:teuthology.orchestra.run.smithi096.stdout:(16/152): libcephsqlite-19.0.0-3267.gbc94a2a9.e 2.4 MB/s | 163 kB 00:00 2024-04-26T11:14:45.902 INFO:teuthology.orchestra.run.smithi096.stdout:(17/152): librados-devel-19.0.0-3267.gbc94a2a9. 2.5 MB/s | 127 kB 00:00 2024-04-26T11:14:45.949 INFO:teuthology.orchestra.run.smithi194.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 396 kB/s | 172 kB 00:00 2024-04-26T11:14:45.952 INFO:teuthology.orchestra.run.smithi096.stdout:(18/152): libradosstriper1-19.0.0-3267.gbc94a2a 6.5 MB/s | 501 kB 00:00 2024-04-26T11:14:46.036 INFO:teuthology.orchestra.run.smithi096.stdout:(19/152): python3-ceph-argparse-19.0.0-3267.gbc 541 kB/s | 45 kB 00:00 2024-04-26T11:14:46.065 INFO:teuthology.orchestra.run.smithi194.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 705 kB/s | 358 kB 00:00 2024-04-26T11:14:46.120 INFO:teuthology.orchestra.run.smithi096.stdout:(20/152): python3-ceph-common-19.0.0-3267.gbc94 1.6 MB/s | 135 kB 00:00 2024-04-26T11:14:46.224 INFO:teuthology.orchestra.run.smithi194.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 128 kB/s | 35 kB 00:00 2024-04-26T11:14:46.245 INFO:teuthology.orchestra.run.smithi096.stdout:(21/152): librgw2-19.0.0-3267.gbc94a2a9.el9.x86 16 MB/s | 5.4 MB 00:00 2024-04-26T11:14:46.270 INFO:teuthology.orchestra.run.smithi096.stdout:(22/152): python3-cephfs-19.0.0-3267.gbc94a2a9. 1.1 MB/s | 162 kB 00:00 2024-04-26T11:14:46.274 INFO:teuthology.orchestra.run.smithi194.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 576 kB/s | 303 kB 00:00 2024-04-26T11:14:46.304 INFO:teuthology.orchestra.run.smithi096.stdout:(23/152): python3-rados-19.0.0-3267.gbc94a2a9.e 5.3 MB/s | 320 kB 00:00 2024-04-26T11:14:46.345 INFO:teuthology.orchestra.run.smithi096.stdout:(24/152): python3-rbd-19.0.0-3267.gbc94a2a9.el9 3.9 MB/s | 298 kB 00:00 2024-04-26T11:14:46.371 INFO:teuthology.orchestra.run.smithi096.stdout:(25/152): python3-rgw-19.0.0-3267.gbc94a2a9.el9 1.5 MB/s | 99 kB 00:00 2024-04-26T11:14:46.608 INFO:teuthology.orchestra.run.smithi194.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 379 kB/s | 206 kB 00:00 2024-04-26T11:14:46.629 INFO:teuthology.orchestra.run.smithi096.stdout:(26/152): rbd-fuse-19.0.0-3267.gbc94a2a9.el9.x8 301 kB/s | 85 kB 00:00 2024-04-26T11:14:46.697 INFO:teuthology.orchestra.run.smithi096.stdout:(27/152): rbd-mirror-19.0.0-3267.gbc94a2a9.el9. 9.5 MB/s | 3.1 MB 00:00 2024-04-26T11:14:46.717 INFO:teuthology.orchestra.run.smithi194.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 327 kB/s | 144 kB 00:00 2024-04-26T11:14:46.722 INFO:teuthology.orchestra.run.smithi096.stdout:(28/152): rbd-nbd-19.0.0-3267.gbc94a2a9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-04-26T11:14:46.747 INFO:teuthology.orchestra.run.smithi096.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3267.g 685 kB/s | 34 kB 00:00 2024-04-26T11:14:46.789 INFO:teuthology.orchestra.run.smithi096.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3267.gbc94a2a 2.1 MB/s | 142 kB 00:00 2024-04-26T11:14:46.825 INFO:teuthology.orchestra.run.smithi194.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 49 kB/s | 11 kB 00:00 2024-04-26T11:14:47.009 INFO:teuthology.orchestra.run.smithi194.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 61 kB/s | 18 kB 00:00 2024-04-26T11:14:47.014 INFO:teuthology.orchestra.run.smithi096.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3267.gbc94a 13 MB/s | 3.5 MB 00:00 2024-04-26T11:14:47.093 INFO:teuthology.orchestra.run.smithi194.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 87 kB/s | 23 kB 00:00 2024-04-26T11:14:47.098 INFO:teuthology.orchestra.run.smithi096.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3267.gbc 2.9 MB/s | 249 kB 00:00 2024-04-26T11:14:47.140 INFO:teuthology.orchestra.run.smithi096.stdout:(33/152): ceph-mgr-rook-19.0.0-3267.gbc94a2a9.e 1.2 MB/s | 49 kB 00:00 2024-04-26T11:14:47.190 INFO:teuthology.orchestra.run.smithi096.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3267.gb 329 kB/s | 16 kB 00:00 2024-04-26T11:14:47.192 INFO:teuthology.orchestra.run.smithi194.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.1 MB/s | 2.0 MB 00:00 2024-04-26T11:14:47.249 INFO:teuthology.orchestra.run.smithi096.stdout:(35/152): ceph-volume-19.0.0-3267.gbc94a2a9.el9 4.6 MB/s | 273 kB 00:00 2024-04-26T11:14:47.276 INFO:teuthology.orchestra.run.smithi194.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 73 kB/s | 19 kB 00:00 2024-04-26T11:14:47.349 INFO:teuthology.orchestra.run.smithi096.stdout:(36/152): cephadm-19.0.0-3267.gbc94a2a9.el9.noa 7.5 MB/s | 762 kB 00:00 2024-04-26T11:14:47.376 INFO:teuthology.orchestra.run.smithi194.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 69 kB/s | 20 kB 00:00 2024-04-26T11:14:47.449 INFO:teuthology.orchestra.run.smithi096.stdout:(37/152): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-04-26T11:14:47.475 INFO:teuthology.orchestra.run.smithi096.stdout:(38/152): cephfs-top-19.0.0-3267.gbc94a2a9.el9. 145 kB/s | 18 kB 00:00 2024-04-26T11:14:47.643 INFO:teuthology.orchestra.run.smithi194.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 126 kB/s | 46 kB 00:00 2024-04-26T11:14:47.658 INFO:teuthology.orchestra.run.smithi096.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 386 kB/s | 80 kB 00:00 2024-04-26T11:14:47.725 INFO:teuthology.orchestra.run.smithi096.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-04-26T11:14:47.750 INFO:teuthology.orchestra.run.smithi096.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-26T11:14:47.801 INFO:teuthology.orchestra.run.smithi096.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-26T11:14:47.802 INFO:teuthology.orchestra.run.smithi194.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 185 kB/s | 79 kB 00:00 2024-04-26T11:14:47.859 INFO:teuthology.orchestra.run.smithi096.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-26T11:14:47.876 INFO:teuthology.orchestra.run.smithi096.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-26T11:14:47.877 INFO:teuthology.orchestra.run.smithi194.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-04-26T11:14:47.910 INFO:teuthology.orchestra.run.smithi096.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi096.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-26T11:14:47.960 INFO:teuthology.orchestra.run.smithi096.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-26T11:14:47.994 INFO:teuthology.orchestra.run.smithi096.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-26T11:14:47.995 INFO:teuthology.orchestra.run.smithi194.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 165 kB/s | 58 kB 00:00 2024-04-26T11:14:48.036 INFO:teuthology.orchestra.run.smithi096.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-26T11:14:48.051 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-04-26T11:14:48.052 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-04-26T11:14:48.061 INFO:teuthology.orchestra.run.smithi096.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-26T11:14:48.078 INFO:teuthology.orchestra.run.smithi096.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 517 kB/s | 21 kB 00:00 2024-04-26T11:14:48.103 INFO:teuthology.orchestra.run.smithi096.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 313 kB 00:00 2024-04-26T11:14:48.121 INFO:teuthology.orchestra.run.smithi096.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 793 kB/s | 33 kB 00:00 2024-04-26T11:14:48.145 INFO:teuthology.orchestra.run.smithi194.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 61 kB/s | 16 kB 00:00 2024-04-26T11:14:48.179 INFO:teuthology.orchestra.run.smithi096.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-26T11:14:48.221 INFO:teuthology.orchestra.run.smithi096.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-04-26T11:14:48.238 INFO:teuthology.orchestra.run.smithi096.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 197 kB/s | 11 kB 00:00 2024-04-26T11:14:48.263 INFO:teuthology.orchestra.run.smithi096.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-26T11:14:48.280 INFO:teuthology.orchestra.run.smithi096.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-26T11:14:48.311 INFO:teuthology.orchestra.run.smithi194.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 535 kB/s | 272 kB 00:00 2024-04-26T11:14:48.313 INFO:teuthology.orchestra.run.smithi096.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-26T11:14:48.337 INFO:teuthology.orchestra.run.smithi194.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 263 kB/s | 90 kB 00:00 2024-04-26T11:14:48.339 INFO:teuthology.orchestra.run.smithi096.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 582 kB 00:00 2024-04-26T11:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 624 kB/s | 25 kB 00:00 2024-04-26T11:14:48.373 INFO:teuthology.orchestra.run.smithi096.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-26T11:14:48.398 INFO:teuthology.orchestra.run.smithi096.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 292 kB 00:00 2024-04-26T11:14:48.423 INFO:teuthology.orchestra.run.smithi096.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-26T11:14:48.445 INFO:teuthology.orchestra.run.smithi194.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 94 kB/s | 28 kB 00:00 2024-04-26T11:14:48.448 INFO:teuthology.orchestra.run.smithi096.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 710 kB/s | 35 kB 00:00 2024-04-26T11:14:48.557 INFO:teuthology.orchestra.run.smithi096.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-04-26T11:14:48.579 INFO:teuthology.orchestra.run.smithi194.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-04-26T11:14:48.641 INFO:teuthology.orchestra.run.smithi096.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-04-26T11:14:48.683 INFO:teuthology.orchestra.run.smithi096.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-26T11:14:48.717 INFO:teuthology.orchestra.run.smithi096.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-26T11:14:48.742 INFO:teuthology.orchestra.run.smithi096.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-26T11:14:48.787 INFO:teuthology.orchestra.run.smithi194.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-04-26T11:14:48.813 INFO:teuthology.orchestra.run.smithi194.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 396 kB/s | 188 kB 00:00 2024-04-26T11:14:48.842 INFO:teuthology.orchestra.run.smithi096.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-26T11:14:48.846 INFO:teuthology.orchestra.run.smithi194.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-04-26T11:14:48.917 INFO:teuthology.orchestra.run.smithi096.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 29 MB/s | 5.8 MB 00:00 2024-04-26T11:14:48.943 INFO:teuthology.orchestra.run.smithi096.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 893 kB/s | 87 kB 00:00 2024-04-26T11:14:49.085 INFO:teuthology.orchestra.run.smithi096.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 6.0 MB 00:00 2024-04-26T11:14:49.110 INFO:teuthology.orchestra.run.smithi096.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-04-26T11:14:49.135 INFO:teuthology.orchestra.run.smithi096.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-26T11:14:49.138 INFO:teuthology.orchestra.run.smithi194.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 145 kB/s | 51 kB 00:00 2024-04-26T11:14:49.160 INFO:teuthology.orchestra.run.smithi096.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 516 kB/s | 26 kB 00:00 2024-04-26T11:14:49.186 INFO:teuthology.orchestra.run.smithi096.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-26T11:14:49.211 INFO:teuthology.orchestra.run.smithi096.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-26T11:14:49.236 INFO:teuthology.orchestra.run.smithi096.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-26T11:14:49.280 INFO:teuthology.orchestra.run.smithi194.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 207 kB/s | 90 kB 00:00 2024-04-26T11:14:49.330 INFO:teuthology.orchestra.run.smithi194.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 444 kB/s | 230 kB 00:00 2024-04-26T11:14:49.387 INFO:teuthology.orchestra.run.smithi096.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-04-26T11:14:49.412 INFO:teuthology.orchestra.run.smithi096.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 250 kB 00:00 2024-04-26T11:14:49.437 INFO:teuthology.orchestra.run.smithi096.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-26T11:14:49.462 INFO:teuthology.orchestra.run.smithi096.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T11:14:49.488 INFO:teuthology.orchestra.run.smithi096.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-04-26T11:14:49.513 INFO:teuthology.orchestra.run.smithi096.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-26T11:14:49.530 INFO:teuthology.orchestra.run.smithi096.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.7 MB/s | 159 kB 00:00 2024-04-26T11:14:49.547 INFO:teuthology.orchestra.run.smithi194.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 75 kB/s | 20 kB 00:00 2024-04-26T11:14:49.566 INFO:teuthology.orchestra.run.smithi096.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-26T11:14:49.589 INFO:teuthology.orchestra.run.smithi096.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-04-26T11:14:49.606 INFO:teuthology.orchestra.run.smithi096.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-04-26T11:14:49.731 INFO:teuthology.orchestra.run.smithi194.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 721 kB/s | 427 kB 00:00 2024-04-26T11:14:49.748 INFO:teuthology.orchestra.run.smithi096.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 295 kB/s | 42 kB 00:00 2024-04-26T11:14:49.839 INFO:teuthology.orchestra.run.smithi194.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 377 kB/s | 192 kB 00:00 2024-04-26T11:14:50.023 INFO:teuthology.orchestra.run.smithi096.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-26T11:14:50.040 INFO:teuthology.orchestra.run.smithi096.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 152 kB/s | 44 kB 00:00 2024-04-26T11:14:50.065 INFO:teuthology.orchestra.run.smithi096.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-04-26T11:14:50.082 INFO:teuthology.orchestra.run.smithi096.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 862 kB/s | 36 kB 00:00 2024-04-26T11:14:50.099 INFO:teuthology.orchestra.run.smithi096.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-26T11:14:50.124 INFO:teuthology.orchestra.run.smithi096.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 305 kB 00:00 2024-04-26T11:14:50.142 INFO:teuthology.orchestra.run.smithi096.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-04-26T11:14:50.167 INFO:teuthology.orchestra.run.smithi096.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-26T11:14:50.192 INFO:teuthology.orchestra.run.smithi096.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-26T11:14:50.217 INFO:teuthology.orchestra.run.smithi096.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-04-26T11:14:50.242 INFO:teuthology.orchestra.run.smithi096.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-04-26T11:14:50.276 INFO:teuthology.orchestra.run.smithi096.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-26T11:14:50.340 INFO:teuthology.orchestra.run.smithi194.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-04-26T11:14:50.415 INFO:teuthology.orchestra.run.smithi194.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 18 MB/s | 1.3 MB 00:00 2024-04-26T11:14:50.515 INFO:teuthology.orchestra.run.smithi194.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-04-26T11:14:50.574 INFO:teuthology.orchestra.run.smithi194.stdout:(151/152): librbd1-19.0.0-3267.gbc94a2a9.el9.x8 4.3 MB/s | 3.1 MB 00:00 2024-04-26T11:14:50.674 INFO:teuthology.orchestra.run.smithi194.stdout:(152/152): librados2-19.0.0-3267.gbc94a2a9.el9. 3.6 MB/s | 3.4 MB 00:00 2024-04-26T11:14:50.679 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T11:14:50.679 INFO:teuthology.orchestra.run.smithi194.stdout:Total 6.3 MB/s | 222 MB 00:35 2024-04-26T11:14:51.226 INFO:teuthology.orchestra.run.smithi096.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 325 kB/s | 309 kB 00:00 2024-04-26T11:14:51.335 INFO:teuthology.orchestra.run.smithi096.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-04-26T11:14:51.360 INFO:teuthology.orchestra.run.smithi096.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 491 kB/s | 548 kB 00:01 2024-04-26T11:14:51.477 INFO:teuthology.orchestra.run.smithi096.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-26T11:14:51.669 INFO:teuthology.orchestra.run.smithi096.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-04-26T11:14:51.722 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T11:14:51.861 INFO:teuthology.orchestra.run.smithi096.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 352 kB/s | 67 kB 00:00 2024-04-26T11:14:51.866 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T11:14:51.866 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T11:14:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-04-26T11:14:53.286 INFO:teuthology.orchestra.run.smithi096.stdout:(111/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 740 kB/s | 839 kB 00:01 2024-04-26T11:14:53.623 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T11:14:53.623 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T11:14:54.053 INFO:teuthology.orchestra.run.smithi096.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 715 kB/s | 548 kB 00:00 2024-04-26T11:14:54.178 INFO:teuthology.orchestra.run.smithi096.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 348 kB/s | 43 kB 00:00 2024-04-26T11:14:54.287 INFO:teuthology.orchestra.run.smithi096.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-26T11:14:54.404 INFO:teuthology.orchestra.run.smithi096.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-04-26T11:14:54.640 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-04-26T11:14:54.671 INFO:teuthology.orchestra.run.smithi096.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 644 kB/s | 172 kB 00:00 2024-04-26T11:14:54.740 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T11:14:54.817 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T11:14:54.933 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T11:14:55.002 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T11:14:55.041 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T11:14:55.170 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T11:14:55.212 INFO:teuthology.orchestra.run.smithi096.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 662 kB/s | 358 kB 00:00 2024-04-26T11:14:55.628 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T11:14:55.663 INFO:teuthology.orchestra.run.smithi096.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 674 kB/s | 303 kB 00:00 2024-04-26T11:14:55.737 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:14:55.779 INFO:teuthology.orchestra.run.smithi096.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 303 kB/s | 35 kB 00:00 2024-04-26T11:14:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:14:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:14:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:14:56.105 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T11:14:56.122 INFO:teuthology.orchestra.run.smithi096.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 602 kB/s | 206 kB 00:00 2024-04-26T11:14:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T11:14:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T11:14:56.597 INFO:teuthology.orchestra.run.smithi096.stdout:(121/152): libarrow-9.0.0-11.el9.x86_64.rpm 859 kB/s | 4.4 MB 00:05 2024-04-26T11:14:56.609 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:14:56.730 INFO:teuthology.orchestra.run.smithi096.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-26T11:14:56.821 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:14:56.839 INFO:teuthology.orchestra.run.smithi096.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.8 MB/s | 2.0 MB 00:00 2024-04-26T11:14:56.852 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T11:14:56.856 INFO:teuthology.orchestra.run.smithi096.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-04-26T11:14:56.931 INFO:teuthology.orchestra.run.smithi096.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-04-26T11:14:56.956 INFO:teuthology.orchestra.run.smithi096.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-26T11:14:56.982 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T11:14:57.012 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T11:14:57.023 INFO:teuthology.orchestra.run.smithi096.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-26T11:14:57.049 INFO:teuthology.orchestra.run.smithi096.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-26T11:14:57.081 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T11:14:57.149 INFO:teuthology.orchestra.run.smithi096.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-04-26T11:14:57.224 INFO:teuthology.orchestra.run.smithi096.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-04-26T11:14:57.258 INFO:teuthology.orchestra.run.smithi096.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 728 kB/s | 79 kB 00:00 2024-04-26T11:14:57.333 INFO:teuthology.orchestra.run.smithi096.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-04-26T11:14:57.358 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T11:14:57.367 INFO:teuthology.orchestra.run.smithi096.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-26T11:14:57.429 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T11:14:57.433 INFO:teuthology.orchestra.run.smithi096.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-26T11:14:57.475 INFO:teuthology.orchestra.run.smithi096.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 830 kB/s | 90 kB 00:00 2024-04-26T11:14:57.542 INFO:teuthology.orchestra.run.smithi096.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-04-26T11:14:57.576 INFO:teuthology.orchestra.run.smithi096.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-26T11:14:57.634 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T11:14:57.651 INFO:teuthology.orchestra.run.smithi096.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-26T11:14:57.685 INFO:teuthology.orchestra.run.smithi096.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-04-26T11:14:57.752 INFO:teuthology.orchestra.run.smithi096.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-04-26T11:14:57.758 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T11:14:57.785 INFO:teuthology.orchestra.run.smithi096.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-04-26T11:14:57.803 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T11:14:57.853 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T11:14:57.860 INFO:teuthology.orchestra.run.smithi096.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-26T11:14:57.886 INFO:teuthology.orchestra.run.smithi096.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-04-26T11:14:57.962 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T11:14:57.969 INFO:teuthology.orchestra.run.smithi096.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-04-26T11:14:57.994 INFO:teuthology.orchestra.run.smithi096.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-26T11:14:58.016 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T11:14:58.078 INFO:teuthology.orchestra.run.smithi096.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-26T11:14:58.100 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T11:14:58.156 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T11:14:58.195 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T11:14:58.320 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T11:14:58.396 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T11:14:58.438 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T11:14:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T11:14:58.587 INFO:teuthology.orchestra.run.smithi096.stdout:(147/152): ceph-test-19.0.0-3267.gbc94a2a9.el9. 3.6 MB/s | 49 MB 00:13 2024-04-26T11:14:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T11:14:58.629 INFO:teuthology.orchestra.run.smithi096.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-04-26T11:14:58.665 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 31/156 2024-04-26T11:14:58.712 INFO:teuthology.orchestra.run.smithi096.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 16 MB/s | 1.3 MB 00:00 2024-04-26T11:14:58.717 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 32/156 2024-04-26T11:14:58.755 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 33/156 2024-04-26T11:14:58.846 INFO:teuthology.orchestra.run.smithi096.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2024-04-26T11:14:58.878 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T11:14:58.945 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T11:14:59.059 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T11:14:59.252 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T11:14:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T11:14:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T11:14:59.727 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T11:14:59.766 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T11:14:59.819 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T11:14:59.853 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T11:14:59.879 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T11:14:59.921 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T11:15:00.053 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T11:15:00.250 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T11:15:00.346 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:15:00.468 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:15:00.497 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:15:00.570 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:15:00.579 INFO:teuthology.orchestra.run.smithi096.stdout:(151/152): librados2-19.0.0-3267.gbc94a2a9.el9. 1.4 MB/s | 3.4 MB 00:02 2024-04-26T11:15:00.608 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T11:15:00.609 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T11:15:00.629 INFO:teuthology.orchestra.run.smithi096.stdout:(152/152): librbd1-19.0.0-3267.gbc94a2a9.el9.x8 1.5 MB/s | 3.1 MB 00:02 2024-04-26T11:15:00.636 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-26T11:15:00.636 INFO:teuthology.orchestra.run.smithi096.stdout:Total 6.3 MB/s | 222 MB 00:35 2024-04-26T11:15:00.647 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T11:15:00.705 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T11:15:00.757 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 43/156 2024-04-26T11:15:00.758 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T11:15:00.806 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T11:15:00.835 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:15:00.840 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T11:15:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:15:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T11:15:01.023 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T11:15:01.045 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T11:15:01.067 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T11:15:01.160 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T11:15:01.241 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T11:15:01.278 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T11:15:01.311 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T11:15:01.326 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T11:15:01.417 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T11:15:01.425 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T11:15:01.464 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T11:15:01.505 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T11:15:01.512 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T11:15:01.559 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T11:15:01.573 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T11:15:01.594 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T11:15:01.628 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T11:15:01.658 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T11:15:01.692 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T11:15:01.693 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T11:15:01.706 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-26T11:15:01.737 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T11:15:01.759 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T11:15:01.769 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T11:15:01.797 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T11:15:01.849 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-26T11:15:01.849 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-26T11:15:01.865 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T11:15:01.919 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T11:15:01.958 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T11:15:01.973 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T11:15:01.993 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T11:15:02.026 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T11:15:02.062 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T11:15:02.069 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T11:15:02.104 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 31/156 2024-04-26T11:15:02.109 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T11:15:02.148 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 32/156 2024-04-26T11:15:02.176 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 33/156 2024-04-26T11:15:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T11:15:02.267 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T11:15:02.291 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T11:15:02.305 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T11:15:02.352 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T11:15:02.364 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T11:15:02.386 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T11:15:02.419 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T11:15:02.505 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T11:15:02.560 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T11:15:02.619 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T11:15:02.799 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T11:15:02.842 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T11:15:02.887 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T11:15:02.937 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T11:15:03.006 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T11:15:03.064 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T11:15:03.078 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T11:15:03.114 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T11:15:03.159 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T11:15:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T11:15:03.201 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T11:15:03.219 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T11:15:03.243 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T11:15:03.282 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 43/156 2024-04-26T11:15:03.296 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T11:15:03.323 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T11:15:03.359 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T11:15:03.480 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T11:15:03.511 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T11:15:03.543 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T11:15:03.574 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T11:15:03.597 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T11:15:03.615 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T11:15:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T11:15:03.686 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T11:15:03.695 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-26T11:15:03.695 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-26T11:15:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T11:15:03.738 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T11:15:03.793 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T11:15:03.829 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T11:15:03.835 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T11:15:03.871 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T11:15:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T11:15:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T11:15:04.048 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T11:15:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T11:15:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T11:15:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T11:15:04.277 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T11:15:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T11:15:04.373 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T11:15:04.399 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T11:15:04.423 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T11:15:04.496 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T11:15:04.632 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T11:15:04.664 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T11:15:04.692 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T11:15:04.726 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T11:15:04.765 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T11:15:04.796 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T11:15:04.852 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T11:15:04.896 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T11:15:04.902 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T11:15:04.944 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T11:15:04.959 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T11:15:04.986 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T11:15:04.993 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T11:15:05.020 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T11:15:05.106 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T11:15:05.123 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T11:15:05.150 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T11:15:05.181 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T11:15:05.214 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T11:15:05.229 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T11:15:05.327 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T11:15:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T11:15:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T11:15:05.382 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T11:15:05.616 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T11:15:05.649 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T11:15:05.760 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T11:15:06.094 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T11:15:06.138 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T11:15:06.172 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T11:15:06.269 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T11:15:06.321 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T11:15:06.409 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T11:15:06.457 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T11:15:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T11:15:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T11:15:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T11:15:06.740 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T11:15:06.827 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T11:15:07.379 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T11:15:07.484 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T11:15:08.516 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T11:15:08.605 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T11:15:08.642 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T11:15:08.680 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T11:15:08.687 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T11:15:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T11:15:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T11:15:08.835 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T11:15:09.070 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T11:15:09.118 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T11:15:09.156 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T11:15:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T11:15:09.232 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T11:15:09.265 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 100/156 2024-04-26T11:15:09.297 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T11:15:09.323 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T11:15:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 100/156 2024-04-26T11:15:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T11:15:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-26T11:15:09.749 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T11:15:09.787 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T11:15:09.841 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T11:15:09.842 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T11:15:09.868 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T11:15:09.883 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T11:15:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T11:15:09.912 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T11:15:09.922 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T11:15:09.951 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T11:15:09.952 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 104/156 2024-04-26T11:15:10.045 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T11:15:10.183 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T11:15:10.214 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 104/156 2024-04-26T11:15:10.272 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T11:15:10.384 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:15:10.515 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:15:10.544 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:15:10.634 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:15:10.664 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T11:15:10.812 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T11:15:10.861 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T11:15:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:15:10.993 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:15:11.015 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T11:15:11.078 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T11:15:11.100 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T11:15:11.124 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T11:15:11.324 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T11:15:11.358 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T11:15:11.468 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T11:15:11.562 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T11:15:11.599 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T11:15:11.634 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T11:15:11.708 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T11:15:11.749 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T11:15:11.793 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T11:15:11.825 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T11:15:11.862 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T11:15:11.931 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T11:15:11.984 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T11:15:12.014 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T11:15:12.049 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T11:15:12.126 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T11:15:12.169 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 31/156 2024-04-26T11:15:12.212 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 32/156 2024-04-26T11:15:12.240 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 33/156 2024-04-26T11:15:12.284 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:12.302 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T11:15:12.325 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:12.356 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T11:15:12.430 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T11:15:12.562 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T11:15:12.615 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T11:15:12.869 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:12.920 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T11:15:13.011 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T11:15:13.011 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T11:15:13.011 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T11:15:13.011 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:13.038 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T11:15:13.129 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T11:15:13.210 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T11:15:13.240 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T11:15:13.275 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T11:15:13.312 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 43/156 2024-04-26T11:15:13.363 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T11:15:13.526 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T11:15:13.591 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T11:15:13.646 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T11:15:13.653 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:13.693 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:13.718 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T11:15:13.749 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T11:15:13.786 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T11:15:13.849 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T11:15:13.885 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T11:15:13.930 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T11:15:13.970 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T11:15:14.115 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T11:15:14.167 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T11:15:14.198 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T11:15:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T11:15:14.333 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T11:15:14.381 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T11:15:14.381 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:14.421 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T11:15:14.430 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T11:15:14.446 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T11:15:14.471 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T11:15:14.548 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T11:15:14.597 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T11:15:14.597 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T11:15:14.597 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T11:15:14.597 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:14.642 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T11:15:14.671 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T11:15:14.703 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T11:15:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T11:15:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T11:15:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T11:15:14.844 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T11:15:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T11:15:14.909 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T11:15:14.942 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T11:15:14.983 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T11:15:15.018 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T11:15:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T11:15:15.157 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T11:15:15.188 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T11:15:15.235 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T11:15:15.333 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T11:15:15.381 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T11:15:16.095 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T11:15:16.145 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T11:15:16.179 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T11:15:16.277 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T11:15:16.328 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T11:15:16.414 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T11:15:16.472 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T11:15:16.649 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T11:15:16.678 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T11:15:16.755 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T11:15:17.392 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T11:15:17.500 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T11:15:18.723 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T11:15:18.798 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T11:15:18.826 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T11:15:18.859 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T11:15:19.169 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T11:15:19.214 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T11:15:19.304 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T11:15:19.326 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 100/156 2024-04-26T11:15:19.363 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T11:15:19.841 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T11:15:20.050 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T11:15:20.120 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T11:15:20.143 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 104/156 2024-04-26T11:15:22.377 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:22.406 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:22.952 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T11:15:23.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T11:15:23.076 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T11:15:23.076 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T11:15:23.076 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T11:15:23.076 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-04-26T11:15:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:24.539 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T11:15:24.554 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T11:15:24.554 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:24.554 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T11:15:24.554 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:26.042 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T11:15:26.066 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T11:15:26.067 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:26.067 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T11:15:26.067 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T11:15:26.067 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T11:15:26.067 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:26.301 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T11:15:26.323 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T11:15:26.323 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:26.323 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T11:15:26.323 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T11:15:26.323 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T11:15:26.323 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:26.768 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T11:15:26.790 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T11:15:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T11:15:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T11:15:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T11:15:26.791 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:26.890 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T11:15:26.938 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T11:15:26.994 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T11:15:27.185 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T11:15:27.235 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T11:15:27.306 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T11:15:27.411 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T11:15:27.549 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T11:15:27.585 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T11:15:27.617 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T11:15:27.644 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T11:15:28.038 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T11:15:28.068 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T11:15:28.100 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 124/156 2024-04-26T11:15:28.170 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 125/156 2024-04-26T11:15:28.199 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T11:15:28.446 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T11:15:28.446 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-04-26T11:15:28.446 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-04-26T11:15:28.447 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-04-26T11:15:28.447 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-04-26T11:15:28.447 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-04-26T11:15:28.447 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-04-26T11:15:28.447 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-04-26T11:15:28.447 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:28.685 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T11:15:28.743 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T11:15:28.743 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:28.743 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T11:15:28.743 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T11:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T11:15:30.229 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T11:15:30.260 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T11:15:30.359 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T11:15:30.437 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T11:15:30.521 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 129/156 2024-04-26T11:15:30.523 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T11:15:30.523 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:30.523 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T11:15:30.523 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T11:15:30.523 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T11:15:30.523 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:30.546 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T11:15:30.568 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T11:15:30.568 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:30.568 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T11:15:30.568 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T11:15:30.568 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T11:15:30.568 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:30.629 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T11:15:30.640 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T11:15:30.738 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 132/156 2024-04-26T11:15:30.787 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T11:15:30.794 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 133/156 2024-04-26T11:15:30.863 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T11:15:30.863 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:30.863 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T11:15:30.864 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T11:15:30.864 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T11:15:30.864 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:31.371 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T11:15:31.445 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T11:15:31.445 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:31.445 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T11:15:31.445 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T11:15:31.445 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T11:15:31.445 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:31.581 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T11:15:31.644 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T11:15:31.720 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T11:15:31.731 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T11:15:31.773 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T11:15:31.795 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T11:15:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:31.796 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T11:15:31.796 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T11:15:31.796 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T11:15:31.796 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:31.841 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T11:15:31.861 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T11:15:31.862 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:31.862 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T11:15:31.862 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:31.908 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T11:15:31.930 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T11:15:31.930 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:31.930 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T11:15:31.930 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T11:15:31.931 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T11:15:31.931 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:31.977 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T11:15:32.035 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T11:15:32.125 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T11:15:32.234 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T11:15:32.234 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T11:15:32.256 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T11:15:32.256 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:32.256 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T11:15:32.256 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T11:15:32.256 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T11:15:32.257 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:32.410 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T11:15:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T11:15:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T11:15:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T11:15:33.400 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T11:15:33.503 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T11:15:33.574 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 124/156 2024-04-26T11:15:33.698 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 125/156 2024-04-26T11:15:33.729 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-04-26T11:15:34.345 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:34.521 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T11:15:34.536 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T11:15:34.536 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:34.536 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T11:15:34.536 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:35.097 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T11:15:35.142 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T11:15:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T11:15:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T11:15:36.049 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:36.049 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T11:15:36.049 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T11:15:36.049 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T11:15:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:36.283 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T11:15:36.305 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T11:15:36.305 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:36.305 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T11:15:36.305 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T11:15:36.305 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T11:15:36.305 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T11:15:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T11:15:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T11:15:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 129/156 2024-04-26T11:15:36.716 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T11:15:36.741 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T11:15:36.763 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T11:15:36.764 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:36.764 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T11:15:36.764 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T11:15:36.764 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T11:15:36.764 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T11:15:36.790 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:36.790 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T11:15:36.790 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T11:15:36.790 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T11:15:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T11:15:36.854 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 139/156 2024-04-26T11:15:36.862 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T11:15:36.889 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T11:15:36.895 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 140/156 2024-04-26T11:15:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T11:15:36.944 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 141/156 2024-04-26T11:15:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T11:15:36.984 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T11:15:36.999 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 132/156 2024-04-26T11:15:37.012 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T11:15:37.066 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 133/156 2024-04-26T11:15:37.149 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T11:15:37.191 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 144/156 2024-04-26T11:15:37.200 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T11:15:37.233 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T11:15:37.262 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 146/156 2024-04-26T11:15:37.276 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T11:15:37.365 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T11:15:37.502 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T11:15:37.542 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T11:15:37.573 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T11:15:37.601 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T11:15:37.848 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:37.902 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:37.921 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:38.071 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T11:15:38.110 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T11:15:38.137 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T11:15:38.149 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 124/156 2024-04-26T11:15:38.169 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T11:15:38.219 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 125/156 2024-04-26T11:15:38.239 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T11:15:38.240 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:38.240 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T11:15:38.240 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T11:15:38.240 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T11:15:38.240 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:38.248 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T11:15:38.286 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T11:15:38.360 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T11:15:38.360 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:38.360 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T11:15:38.360 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:38.421 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T11:15:38.500 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T11:15:38.500 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:38.500 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T11:15:38.500 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T11:15:38.500 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T11:15:38.501 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 148/156 2024-04-26T11:15:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T11:15:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T11:15:38.879 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T11:15:38.951 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T11:15:38.951 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:38.951 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T11:15:38.952 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T11:15:38.952 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T11:15:38.952 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:38.952 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T11:15:39.002 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T11:15:39.002 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T11:15:39.086 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T11:15:39.119 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T11:15:39.119 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T11:15:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T11:15:39.284 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T11:15:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:39.316 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T11:15:40.281 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T11:15:40.308 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T11:15:40.409 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T11:15:40.570 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 129/156 2024-04-26T11:15:40.610 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T11:15:40.633 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T11:15:40.634 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:40.634 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T11:15:40.634 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T11:15:40.634 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T11:15:40.634 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:40.686 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T11:15:40.697 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T11:15:40.794 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 132/156 2024-04-26T11:15:40.842 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 133/156 2024-04-26T11:15:41.766 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T11:15:41.797 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T11:15:41.819 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T11:15:41.819 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:41.820 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T11:15:41.820 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T11:15:41.820 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T11:15:41.820 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:41.881 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T11:15:41.903 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T11:15:41.903 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:41.903 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T11:15:41.903 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:41.948 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T11:15:41.970 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T11:15:41.970 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:41.970 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T11:15:41.970 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T11:15:41.970 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T11:15:41.970 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:42.282 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T11:15:42.304 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T11:15:42.304 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T11:15:42.304 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T11:15:42.304 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T11:15:42.304 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T11:15:42.305 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:44.270 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 139/156 2024-04-26T11:15:44.327 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 140/156 2024-04-26T11:15:44.414 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 141/156 2024-04-26T11:15:44.479 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T11:15:44.531 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T11:15:44.861 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 144/156 2024-04-26T11:15:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T11:15:44.951 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 146/156 2024-04-26T11:15:45.079 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:45.079 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 1/156 2024-04-26T11:15:45.079 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 3/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 4/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 5/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 6/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 7/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 11/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 12/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 14/156 2024-04-26T11:15:45.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 15/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 16/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 17/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 18/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 19/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 20/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 21/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 22/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 23/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 24/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 25/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 26/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 27/156 2024-04-26T11:15:45.081 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 28/156 2024-04-26T11:15:45.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 29/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 30/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 31/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 32/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 33/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 34/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 35/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 36/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 37/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 38/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 39/156 2024-04-26T11:15:45.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T11:15:45.095 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T11:15:45.096 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T11:15:45.096 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T11:15:45.096 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T11:15:45.096 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T11:15:45.096 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T11:15:45.096 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T11:15:45.096 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T11:15:45.097 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T11:15:45.098 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T11:15:45.099 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T11:15:45.100 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T11:15:45.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T11:15:45.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T11:15:45.103 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 149/156 2024-04-26T11:15:45.104 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T11:15:45.104 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 151/156 2024-04-26T11:15:45.104 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T11:15:45.104 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T11:15:45.104 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T11:15:45.104 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T11:15:45.960 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:46.007 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:46.078 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:46.986 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 139/156 2024-04-26T11:15:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 140/156 2024-04-26T11:15:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 141/156 2024-04-26T11:15:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T11:15:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T11:15:47.310 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 144/156 2024-04-26T11:15:47.340 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:47.340 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T11:15:47.341 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.342 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.343 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T11:15:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.345 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T11:15:47.346 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T11:15:47.347 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T11:15:47.348 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T11:15:47.349 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T11:15:47.350 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T11:15:47.351 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T11:15:47.352 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.353 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.354 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:47.354 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T11:15:47.354 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T11:15:47.354 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T11:15:47.354 INFO:teuthology.orchestra.run.smithi194.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T11:15:47.354 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T11:15:47.354 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T11:15:47.355 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T11:15:47.355 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:15:47.355 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T11:15:47.365 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T11:15:47.395 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 146/156 2024-04-26T11:15:47.557 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 148/156 2024-04-26T11:15:47.655 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T11:15:47.740 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T11:15:47.753 DEBUG:teuthology.parallel:result is None 2024-04-26T11:15:47.916 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T11:15:47.961 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T11:15:47.962 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T11:15:47.998 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:48.042 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:48.062 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T11:15:48.088 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T11:15:48.135 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T11:15:48.136 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T11:15:48.275 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T11:15:48.421 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:48.722 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:48.722 INFO:teuthology.orchestra.run.smithi033.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T11:15:48.722 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:48.893 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 148/156 2024-04-26T11:15:48.963 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T11:15:49.004 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T11:15:49.085 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T11:15:49.134 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T11:15:49.134 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T11:15:49.218 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T11:15:49.251 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T11:15:49.251 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T11:15:49.338 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T11:15:49.408 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 1/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 3/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 4/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 5/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 6/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 7/156 2024-04-26T11:15:54.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 11/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 12/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 14/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 15/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 16/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 17/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 18/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 19/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 20/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 21/156 2024-04-26T11:15:54.220 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 22/156 2024-04-26T11:15:54.221 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 23/156 2024-04-26T11:15:54.221 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 24/156 2024-04-26T11:15:54.221 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 25/156 2024-04-26T11:15:54.221 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 26/156 2024-04-26T11:15:54.221 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 27/156 2024-04-26T11:15:54.221 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 28/156 2024-04-26T11:15:54.221 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 29/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 30/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 31/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 32/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 33/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 34/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 35/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 36/156 2024-04-26T11:15:54.233 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 37/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 38/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 39/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T11:15:54.234 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T11:15:54.235 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T11:15:54.236 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T11:15:54.237 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T11:15:54.238 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T11:15:54.239 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T11:15:54.240 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T11:15:54.241 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 149/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 151/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T11:15:54.242 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T11:15:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 1/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 3/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 4/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 5/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 6/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 7/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 11/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 12/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 14/156 2024-04-26T11:15:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 15/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 16/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 17/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 18/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 19/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 20/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 21/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 22/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 23/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 24/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 25/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 26/156 2024-04-26T11:15:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 27/156 2024-04-26T11:15:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 28/156 2024-04-26T11:15:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 29/156 2024-04-26T11:15:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 30/156 2024-04-26T11:15:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 31/156 2024-04-26T11:15:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 32/156 2024-04-26T11:15:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 33/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 34/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 35/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 36/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 37/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 38/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 39/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T11:15:55.688 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T11:15:55.689 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T11:15:55.690 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T11:15:55.691 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T11:15:55.692 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T11:15:55.693 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T11:15:55.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 149/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T11:15:55.695 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 151/156 2024-04-26T11:15:55.696 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T11:15:55.696 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T11:15:55.696 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T11:15:55.696 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T11:15:56.673 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T11:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.677 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T11:15:56.678 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T11:15:56.679 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T11:15:56.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T11:15:56.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T11:15:56.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T11:15:56.683 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T11:15:56.684 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T11:15:56.685 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T11:15:56.686 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T11:15:56.686 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T11:15:56.686 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:15:56.686 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-04-26T11:15:57.254 DEBUG:teuthology.parallel:result is None 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T11:15:57.799 INFO:teuthology.orchestra.run.smithi096.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.801 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T11:15:57.802 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.803 INFO:teuthology.orchestra.run.smithi096.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T11:15:57.804 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.805 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T11:15:57.806 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T11:15:57.807 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T11:15:57.808 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T11:15:57.809 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T11:15:57.810 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T11:15:57.811 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T11:15:57.812 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T11:15:57.812 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-26T11:15:57.812 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-26T11:15:58.181 DEBUG:teuthology.parallel:result is None 2024-04-26T11:15:58.182 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:15:58.830 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T11:15:58.887 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-3267.gbc94a2a9.el9 2024-04-26T11:15:58.887 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3267.gbc94a2a9.el9 2024-04-26T11:15:58.888 INFO:teuthology.task.install:The correct ceph version 19.0.0-3267.gbc94a2a9 is installed. 2024-04-26T11:15:58.889 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:15:59.205 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T11:15:59.228 INFO:teuthology.orchestra.run.smithi096.stdout:19.0.0-3267.gbc94a2a9.el9 2024-04-26T11:15:59.228 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3267.gbc94a2a9.el9 2024-04-26T11:15:59.228 INFO:teuthology.task.install:The correct ceph version 19.0.0-3267.gbc94a2a9 is installed. 2024-04-26T11:15:59.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:15:59.372 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T11:15:59.395 INFO:teuthology.orchestra.run.smithi194.stdout:19.0.0-3267.gbc94a2a9.el9 2024-04-26T11:15:59.396 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3267.gbc94a2a9.el9 2024-04-26T11:15:59.396 INFO:teuthology.task.install:The correct ceph version 19.0.0-3267.gbc94a2a9 is installed. 2024-04-26T11:15:59.398 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T11:15:59.398 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:15:59.398 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T11:15:59.472 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:15:59.472 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T11:15:59.498 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:15:59.498 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T11:15:59.525 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T11:15:59.526 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:15:59.526 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T11:15:59.593 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T11:15:59.695 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:15:59.695 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T11:15:59.720 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T11:15:59.783 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:15:59.783 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T11:15:59.808 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T11:15:59.873 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T11:15:59.873 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:15:59.873 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T11:15:59.939 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T11:16:00.046 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:16:00.046 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T11:16:00.069 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T11:16:00.133 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:16:00.133 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T11:16:00.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T11:16:00.226 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T11:16:00.226 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:16:00.226 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T11:16:00.304 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T11:16:00.417 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:16:00.417 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T11:16:00.441 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T11:16:00.505 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:16:00.505 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T11:16:00.532 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T11:16:00.601 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T11:16:00.704 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'subvols': {'create': 2}} 2024-04-26T11:16:00.704 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:16:00.705 INFO:tasks.cephadm:Cluster fsid is 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:16:00.705 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T11:16:00.705 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.33', 'mon.b': '172.21.15.96', 'mon.c': '172.21.15.194'} 2024-04-26T11:16:00.705 INFO:tasks.cephadm:First mon is mon.a on smithi033 2024-04-26T11:16:00.705 INFO:tasks.cephadm:First mgr is x 2024-04-26T11:16:00.705 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T11:16:00.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-04-26T11:16:00.778 DEBUG:teuthology.orchestra.run.smithi096:> sudo hostname $(hostname -s) 2024-04-26T11:16:00.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-04-26T11:16:00.831 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T11:16:00.832 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:16:00.977 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78913/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3267-gbc94a2a9', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3267.gbc94a2a9'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 08:22:46.917786', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer-fixes', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-26T11:16:01.199 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-lusov-quiescer-fixes, sha1 bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:16:01.200 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm 2024-04-26T11:16:01.201 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:02.568 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-r--r--. 1 ubuntu ubuntu 781590 Apr 26 11:16 /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:02.568 DEBUG:teuthology.orchestra.run.smithi096:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:04.244 INFO:teuthology.orchestra.run.smithi096.stdout:-rw-r--r--. 1 ubuntu ubuntu 781590 Apr 26 11:16 /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:04.244 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:05.762 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-r--r--. 1 ubuntu ubuntu 781590 Apr 26 11:16 /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:05.762 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:05.829 DEBUG:teuthology.orchestra.run.smithi096:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:05.848 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T11:16:05.885 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 on all hosts... 2024-04-26T11:16:05.885 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 pull 2024-04-26T11:16:05.893 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 pull 2024-04-26T11:16:05.896 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 pull 2024-04-26T11:16:07.453 INFO:teuthology.orchestra.run.smithi096.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T11:16:07.595 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T11:16:09.151 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T11:16:35.918 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-26T11:16:35.918 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev)", 2024-04-26T11:16:35.918 INFO:teuthology.orchestra.run.smithi096.stdout: "image_id": "3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555", 2024-04-26T11:16:35.918 INFO:teuthology.orchestra.run.smithi096.stdout: "repo_digests": [ 2024-04-26T11:16:35.918 INFO:teuthology.orchestra.run.smithi096.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc" 2024-04-26T11:16:35.918 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-04-26T11:16:35.918 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-26T11:16:37.334 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-26T11:16:37.334 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev)", 2024-04-26T11:16:37.334 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555", 2024-04-26T11:16:37.334 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-04-26T11:16:37.334 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc" 2024-04-26T11:16:37.334 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-26T11:16:37.334 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-26T11:16:51.336 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:16:51.337 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev)", 2024-04-26T11:16:51.337 INFO:teuthology.orchestra.run.smithi033.stdout: "image_id": "3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555", 2024-04-26T11:16:51.337 INFO:teuthology.orchestra.run.smithi033.stdout: "repo_digests": [ 2024-04-26T11:16:51.337 INFO:teuthology.orchestra.run.smithi033.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc" 2024-04-26T11:16:51.337 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-26T11:16:51.337 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:16:51.391 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-04-26T11:16:51.482 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph 2024-04-26T11:16:51.514 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-04-26T11:16:51.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-04-26T11:16:51.627 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /etc/ceph 2024-04-26T11:16:51.655 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-04-26T11:16:51.683 INFO:tasks.cephadm:Writing seed config... 2024-04-26T11:16:51.684 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T11:16:51.685 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T11:16:51.686 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T11:16:51.687 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T11:16:51.688 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T11:16:51.688 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T11:16:51.688 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T11:16:51.688 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T11:16:51.688 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T11:16:51.689 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:16:51.689 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T11:16:51.741 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5cc7f146-03be-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T11:16:51.741 DEBUG:teuthology.orchestra.run.smithi033:mon.a> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T11:16:51.787 DEBUG:teuthology.orchestra.run.smithi033:mgr.x> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T11:16:51.834 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T11:16:51.834 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 -v bootstrap --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.33 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T11:16:52.108 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-04-26T11:16:52.108 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', '-v', 'bootstrap', '--fsid', '5cc7f146-03be-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.33', '--skip-admin-label'] 2024-04-26T11:16:52.109 INFO:teuthology.orchestra.run.smithi033.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T11:16:52.109 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying podman|docker is present... 2024-04-26T11:16:52.169 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T11:16:52.170 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying lvm2 is present... 2024-04-26T11:16:52.170 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying time synchronization is in place... 2024-04-26T11:16:52.187 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T11:16:52.187 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T11:16:52.201 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T11:16:52.201 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-04-26T11:16:52.217 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout enabled 2024-04-26T11:16:52.235 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout active 2024-04-26T11:16:52.236 INFO:teuthology.orchestra.run.smithi033.stdout:Unit chronyd.service is enabled and running 2024-04-26T11:16:52.236 INFO:teuthology.orchestra.run.smithi033.stdout:Repeating the final host check... 2024-04-26T11:16:52.291 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T11:16:52.292 INFO:teuthology.orchestra.run.smithi033.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T11:16:52.292 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl is present 2024-04-26T11:16:52.292 INFO:teuthology.orchestra.run.smithi033.stdout:lvcreate is present 2024-04-26T11:16:52.308 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T11:16:52.308 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T11:16:52.323 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T11:16:52.323 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-04-26T11:16:52.341 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout enabled 2024-04-26T11:16:52.357 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout active 2024-04-26T11:16:52.357 INFO:teuthology.orchestra.run.smithi033.stdout:Unit chronyd.service is enabled and running 2024-04-26T11:16:52.357 INFO:teuthology.orchestra.run.smithi033.stdout:Host looks OK 2024-04-26T11:16:52.357 INFO:teuthology.orchestra.run.smithi033.stdout:Cluster fsid: 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:16:52.357 INFO:teuthology.orchestra.run.smithi033.stdout:Acquiring lock 140477794532512 on /run/cephadm/5cc7f146-03be-11ef-bc93-c7b262605968.lock 2024-04-26T11:16:52.358 INFO:teuthology.orchestra.run.smithi033.stdout:Lock 140477794532512 acquired on /run/cephadm/5cc7f146-03be-11ef-bc93-c7b262605968.lock 2024-04-26T11:16:52.358 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying IP 172.21.15.33 port 3300 ... 2024-04-26T11:16:52.359 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying IP 172.21.15.33 port 6789 ... 2024-04-26T11:16:52.359 INFO:teuthology.orchestra.run.smithi033.stdout:Base mon IP(s) is [172.21.15.33:3300, 172.21.15.33:6789], mon addrv is [v2:172.21.15.33:3300,v1:172.21.15.33:6789] 2024-04-26T11:16:52.367 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.33 metric 100 2024-04-26T11:16:52.367 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.33 metric 100 2024-04-26T11:16:52.375 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T11:16:52.375 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T11:16:52.375 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T11:16:52.382 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T11:16:52.383 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T11:16:52.383 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T11:16:52.383 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T11:16:52.383 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:d294/64 scope link noprefixroute 2024-04-26T11:16:52.383 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T11:16:52.384 INFO:teuthology.orchestra.run.smithi033.stdout:Mon IP `172.21.15.33` is in CIDR network `172.21.0.0/20` 2024-04-26T11:16:52.384 INFO:teuthology.orchestra.run.smithi033.stdout:Mon IP `172.21.15.33` is in CIDR network `172.21.0.0/20` 2024-04-26T11:16:52.384 INFO:teuthology.orchestra.run.smithi033.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T11:16:52.385 INFO:teuthology.orchestra.run.smithi033.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T11:16:52.386 INFO:teuthology.orchestra.run.smithi033.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T11:16:58.492 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 2024-04-26T11:16:58.493 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T11:16:58.493 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T11:16:58.493 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying blob sha256:449452a284afe5840255d087488a224329a4679c5f5d200bb867a639c75ddfd2 2024-04-26T11:16:58.493 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T11:16:58.493 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying config sha256:3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 2024-04-26T11:16:58.493 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T11:16:59.643 INFO:teuthology.orchestra.run.smithi033.stdout:ceph: stdout ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:16:59.644 INFO:teuthology.orchestra.run.smithi033.stdout:Ceph version: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:16:59.644 INFO:teuthology.orchestra.run.smithi033.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T11:17:00.817 INFO:teuthology.orchestra.run.smithi033.stdout:stat: stdout 167 167 2024-04-26T11:17:00.817 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial keys... 2024-04-26T11:17:01.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQAtjStmK+6TEhAAJedOzkqo7ZoJ6zHazi8Jvw== 2024-04-26T11:17:03.155 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQAujStmQx+UGRAA8DgQufF+00m2VummI01Srg== 2024-04-26T11:17:04.281 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQAvjStmgcQvKBAAdM8A7s5Ee3BgDYOgm1VGWQ== 2024-04-26T11:17:04.282 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial monmap... 2024-04-26T11:17:05.342 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T11:17:05.343 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool for a [v2:172.21.15.33:3300,v1:172.21.15.33:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = quincy 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: set fsid to 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:17:05.346 INFO:teuthology.orchestra.run.smithi033.stdout:Creating mon... 2024-04-26T11:17:06.459 INFO:teuthology.orchestra.run.smithi033.stdout:create mon.a on 2024-04-26T11:17:06.838 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T11:17:07.092 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T11:17:07.421 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5cc7f146-03be-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-5cc7f146-03be-11ef-bc93-c7b262605968.target. 2024-04-26T11:17:07.422 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5cc7f146-03be-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-5cc7f146-03be-11ef-bc93-c7b262605968.target. 2024-04-26T11:17:07.812 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.a 2024-04-26T11:17:07.812 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T11:17:08.106 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5cc7f146-03be-11ef-bc93-c7b262605968.target.wants/ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-5cc7f146-03be-11ef-bc93-c7b262605968@.service. 2024-04-26T11:17:08.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:08 smithi033 podman[27462]: 2024-04-26 11:17:08.782427157 +0000 UTC m=+0.359894286 container start 79e3bf54db1efd4ecc430d8d4c6624991746a787e1319efb989aa07d29870778 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T11:17:08.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:08 smithi033 bash[27462]: 79e3bf54db1efd4ecc430d8d4c6624991746a787e1319efb989aa07d29870778 2024-04-26T11:17:08.887 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T11:17:08.888 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-04-26T11:17:08.902 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T11:17:08.903 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-04-26T11:17:08.903 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-04-26T11:17:08.903 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T11:17:08.903 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mon to start... 2024-04-26T11:17:08.903 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mon... 2024-04-26T11:17:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:08 smithi033 systemd[1]: Started Ceph mon.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:17:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:08 smithi033 ceph-mon[27480]: mkfs 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:09.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:08 smithi033 ceph-mon[27480]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T11:17:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: monmap epoch 1 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: last_changed 2024-04-26T11:17:04.749857+0000 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: min_mon_release 19 (squid) 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: election_strategy: 1 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: fsmap 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: mgrmap e1: no daemons active 2024-04-26T11:17:10.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:09 smithi033 ceph-mon[27480]: from='client.? 172.21.15.33:0/3157487919' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T11:17:10.283 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T11:17:10.284 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout id: 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:10.284 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T11:17:10.284 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:10.284 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout services: 2024-04-26T11:17:10.284 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.759337s) 2024-04-26T11:17:10.285 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout data: 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T11:17:10.286 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:10.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.530+0000 7fe962200700 1 Processor -- start 2024-04-26T11:17:10.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.530+0000 7fe962200700 1 -- start start 2024-04-26T11:17:10.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.530+0000 7fe962200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:10.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.530+0000 7fe962200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c112620 con 0x7fe95c10fcf0 2024-04-26T11:17:10.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.532+0000 7fe95b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:10.290 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.532+0000 7fe95b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1120e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54036/0 (socket says 172.21.15.33:54036) 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.532+0000 7fe95b400700 1 -- 172.21.15.33:0/3800394656 learned_addr learned my addr 172.21.15.33:0/3800394656 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.532+0000 7fe95b400700 1 -- 172.21.15.33:0/3800394656 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95c1127a0 con 0x7fe95c10fcf0 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.533+0000 7fe95b400700 1 --2- 172.21.15.33:0/3800394656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1120e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009e80 tx=0x7fe94c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc98362a67d085e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.533+0000 7fe95aa00700 1 -- 172.21.15.33:0/3800394656 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe94c017070 con 0x7fe95c10fcf0 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.533+0000 7fe95aa00700 1 -- 172.21.15.33:0/3800394656 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe94c003a20 con 0x7fe95c10fcf0 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.533+0000 7fe95aa00700 1 -- 172.21.15.33:0/3800394656 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe94c003d20 con 0x7fe95c10fcf0 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.533+0000 7fe962200700 1 -- 172.21.15.33:0/3800394656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 msgr2=0x7fe95c1120e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.534+0000 7fe962200700 1 --2- 172.21.15.33:0/3800394656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1120e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009e80 tx=0x7fe94c00b330 comp rx=0 tx=0).stop 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.534+0000 7fe962200700 1 -- 172.21.15.33:0/3800394656 shutdown_connections 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.534+0000 7fe962200700 1 --2- 172.21.15.33:0/3800394656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1120e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:10.291 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.534+0000 7fe962200700 1 -- 172.21.15.33:0/3800394656 >> 172.21.15.33:0/3800394656 conn(0x7fe95c07e300 msgr2=0x7fe95c080760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.534+0000 7fe962200700 1 -- 172.21.15.33:0/3800394656 shutdown_connections 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.534+0000 7fe962200700 1 -- 172.21.15.33:0/3800394656 wait complete. 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.535+0000 7fe962200700 1 Processor -- start 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.535+0000 7fe962200700 1 -- start start 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.535+0000 7fe962200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.535+0000 7fe962200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c112bb0 con 0x7fe95c10fcf0 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.536+0000 7fe95b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.536+0000 7fe95b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1a2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54040/0 (socket says 172.21.15.33:54040) 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.536+0000 7fe95b400700 1 -- 172.21.15.33:0/3157487919 learned_addr learned my addr 172.21.15.33:0/3157487919 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.537+0000 7fe95b400700 1 -- 172.21.15.33:0/3157487919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95c1a3010 con 0x7fe95c10fcf0 2024-04-26T11:17:10.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.537+0000 7fe95b400700 1 --2- 172.21.15.33:0/3157487919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1a2ad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe95c082a00 tx=0x7fe94c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:10.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.537+0000 7fe958c00700 1 -- 172.21.15.33:0/3157487919 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe94c022020 con 0x7fe95c10fcf0 2024-04-26T11:17:10.293 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.537+0000 7fe958c00700 1 -- 172.21.15.33:0/3157487919 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe94c018040 con 0x7fe95c10fcf0 2024-04-26T11:17:10.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.537+0000 7fe958c00700 1 -- 172.21.15.33:0/3157487919 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe94c017070 con 0x7fe95c10fcf0 2024-04-26T11:17:10.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.537+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95c1a32a0 con 0x7fe95c10fcf0 2024-04-26T11:17:10.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.537+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95c1a5f40 con 0x7fe95c10fcf0 2024-04-26T11:17:10.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.538+0000 7fe958c00700 1 -- 172.21.15.33:0/3157487919 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe94c025020 con 0x7fe95c10fcf0 2024-04-26T11:17:10.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.538+0000 7fe958c00700 1 -- 172.21.15.33:0/3157487919 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe94c00f050 con 0x7fe95c10fcf0 2024-04-26T11:17:10.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.539+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe95c064a80 con 0x7fe95c10fcf0 2024-04-26T11:17:10.295 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.542+0000 7fe958c00700 1 -- 172.21.15.33:0/3157487919 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe94c02c030 con 0x7fe95c10fcf0 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.623+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fe95c02e650 con 0x7fe95c10fcf0 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.623+0000 7fe958c00700 1 -- 172.21.15.33:0/3157487919 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fe94c01e480 con 0x7fe95c10fcf0 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.625+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 msgr2=0x7fe95c1a2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.625+0000 7fe962200700 1 --2- 172.21.15.33:0/3157487919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1a2ad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe95c082a00 tx=0x7fe94c004ad0 comp rx=0 tx=0).stop 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.625+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 shutdown_connections 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.625+0000 7fe962200700 1 --2- 172.21.15.33:0/3157487919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe95c10fcf0 0x7fe95c1a2ad0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.625+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 >> 172.21.15.33:0/3157487919 conn(0x7fe95c07e300 msgr2=0x7fe95c07f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.625+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 shutdown_connections 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:09.625+0000 7fe962200700 1 -- 172.21.15.33:0/3157487919 wait complete. 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:mon is available 2024-04-26T11:17:10.296 INFO:teuthology.orchestra.run.smithi033.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T11:17:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:11 smithi033 ceph-mon[27480]: from='client.? 172.21.15.33:0/3672686222' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T11:17:11.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:11 smithi033 ceph-mon[27480]: from='client.? 172.21.15.33:0/3672686222' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T11:17:11.604 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:11.604 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T11:17:11.604 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout fsid = 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.33:3300,v1:172.21.15.33:6789] 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T11:17:11.605 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.892+0000 7fe7f0800700 1 Processor -- start 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.892+0000 7fe7f0800700 1 -- start start 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.893+0000 7fe7f0800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e807d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.893+0000 7fe7f0800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e807d8a0 con 0x7fe7e807ee60 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.893+0000 7fe7ede00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e807d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.893+0000 7fe7ede00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e807d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48896/0 (socket says 172.21.15.33:48896) 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.894+0000 7fe7ede00700 1 -- 172.21.15.33:0/685419883 learned_addr learned my addr 172.21.15.33:0/685419883 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.894+0000 7fe7ede00700 1 -- 172.21.15.33:0/685419883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e807da20 con 0x7fe7e807ee60 2024-04-26T11:17:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.894+0000 7fe7ede00700 1 --2- 172.21.15.33:0/685419883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e807d360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0098a0 tx=0x7fe7dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe983e8980279de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.895+0000 7fe7eca00700 1 -- 172.21.15.33:0/685419883 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7dc004160 con 0x7fe7e807ee60 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.895+0000 7fe7eca00700 1 -- 172.21.15.33:0/685419883 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7dc004300 con 0x7fe7e807ee60 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.895+0000 7fe7eca00700 1 -- 172.21.15.33:0/685419883 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7dc0036a0 con 0x7fe7e807ee60 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.895+0000 7fe7f0800700 1 -- 172.21.15.33:0/685419883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 msgr2=0x7fe7e807d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.895+0000 7fe7f0800700 1 --2- 172.21.15.33:0/685419883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e807d360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0098a0 tx=0x7fe7dc009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.896+0000 7fe7f0800700 1 -- 172.21.15.33:0/685419883 shutdown_connections 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.896+0000 7fe7f0800700 1 --2- 172.21.15.33:0/685419883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e807d360 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.896+0000 7fe7f0800700 1 -- 172.21.15.33:0/685419883 >> 172.21.15.33:0/685419883 conn(0x7fe7e81043c0 msgr2=0x7fe7e81067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.896+0000 7fe7f0800700 1 -- 172.21.15.33:0/685419883 shutdown_connections 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.896+0000 7fe7f0800700 1 -- 172.21.15.33:0/685419883 wait complete. 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.897+0000 7fe7f0800700 1 Processor -- start 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.897+0000 7fe7f0800700 1 -- start start 2024-04-26T11:17:11.607 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.897+0000 7fe7f0800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e81a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.897+0000 7fe7f0800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e807df50 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.898+0000 7fe7ede00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e81a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.898+0000 7fe7ede00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e81a2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48902/0 (socket says 172.21.15.33:48902) 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.898+0000 7fe7ede00700 1 -- 172.21.15.33:0/3672686222 learned_addr learned my addr 172.21.15.33:0/3672686222 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.898+0000 7fe7ede00700 1 -- 172.21.15.33:0/3672686222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e81a2fd0 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.898+0000 7fe7ede00700 1 --2- 172.21.15.33:0/3672686222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e81a2a90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc000c00 tx=0x7fe7dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.898+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7dc003db0 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.899+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7dc011630 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.899+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e81a3260 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.899+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7dc011950 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.899+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e81a5f50 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.899+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe7dc01a460 con 0x7fe7e807ee60 2024-04-26T11:17:11.608 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.899+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe7dc01b780 con 0x7fe7e807ee60 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.900+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e8064a80 con 0x7fe7e807ee60 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.903+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe7dc01b980 con 0x7fe7e807ee60 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.981+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe7e802e650 con 0x7fe7e807ee60 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.991+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fe7dc01bb60 con 0x7fe7e807ee60 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.992+0000 7fe7e2a00700 1 -- 172.21.15.33:0/3672686222 <== mon.0 v2:172.21.15.33:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe7dc01ad70 con 0x7fe7e807ee60 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.992+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 msgr2=0x7fe7e81a2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.992+0000 7fe7f0800700 1 --2- 172.21.15.33:0/3672686222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e81a2a90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc000c00 tx=0x7fe7dc004750 comp rx=0 tx=0).stop 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.993+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 shutdown_connections 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.993+0000 7fe7f0800700 1 --2- 172.21.15.33:0/3672686222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7e807ee60 0x7fe7e81a2a90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.993+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 >> 172.21.15.33:0/3672686222 conn(0x7fe7e81043c0 msgr2=0x7fe7e81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.993+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 shutdown_connections 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:10.994+0000 7fe7f0800700 1 -- 172.21.15.33:0/3672686222 wait complete. 2024-04-26T11:17:11.609 INFO:teuthology.orchestra.run.smithi033.stdout:Generating new minimal ceph.conf... 2024-04-26T11:17:12.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:12 smithi033 ceph-mon[27480]: from='client.? 172.21.15.33:0/86858280' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:17:12.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.202+0000 7f64a1c00700 1 Processor -- start 2024-04-26T11:17:12.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.202+0000 7f64a1c00700 1 -- start start 2024-04-26T11:17:12.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.203+0000 7f64a1c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c10b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:12.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.203+0000 7f64a1c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f649c10b8e0 con 0x7f649c10af80 2024-04-26T11:17:12.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.203+0000 7f649aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c10b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:12.880 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.203+0000 7f649aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c10b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48912/0 (socket says 172.21.15.33:48912) 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.203+0000 7f649aa00700 1 -- 172.21.15.33:0/1120297475 learned_addr learned my addr 172.21.15.33:0/1120297475 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.204+0000 7f649aa00700 1 -- 172.21.15.33:0/1120297475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c10c090 con 0x7f649c10af80 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.204+0000 7f649aa00700 1 --2- 172.21.15.33:0/1120297475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c10b3a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f648c0098a0 tx=0x7f648c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810b265681d9a86a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f649a000700 1 -- 172.21.15.33:0/1120297475 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c004160 con 0x7f649c10af80 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f649a000700 1 -- 172.21.15.33:0/1120297475 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f648c004300 con 0x7f649c10af80 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f649a000700 1 -- 172.21.15.33:0/1120297475 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c011af0 con 0x7f649c10af80 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f64a1c00700 1 -- 172.21.15.33:0/1120297475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 msgr2=0x7f649c10b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f64a1c00700 1 --2- 172.21.15.33:0/1120297475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c10b3a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f648c0098a0 tx=0x7f648c009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f64a1c00700 1 -- 172.21.15.33:0/1120297475 shutdown_connections 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f64a1c00700 1 --2- 172.21.15.33:0/1120297475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c10b3a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.205+0000 7f64a1c00700 1 -- 172.21.15.33:0/1120297475 >> 172.21.15.33:0/1120297475 conn(0x7f649c07e350 msgr2=0x7f649c07e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.206+0000 7f64a1c00700 1 -- 172.21.15.33:0/1120297475 shutdown_connections 2024-04-26T11:17:12.883 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.206+0000 7f64a1c00700 1 -- 172.21.15.33:0/1120297475 wait complete. 2024-04-26T11:17:12.884 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.207+0000 7f64a1c00700 1 Processor -- start 2024-04-26T11:17:12.884 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.207+0000 7f64a1c00700 1 -- start start 2024-04-26T11:17:12.884 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.207+0000 7f64a1c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:12.884 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.207+0000 7f64a1c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f649c10f4d0 con 0x7f649c10af80 2024-04-26T11:17:12.884 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.207+0000 7f649aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:12.884 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.207+0000 7f649aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48914/0 (socket says 172.21.15.33:48914) 2024-04-26T11:17:12.886 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.207+0000 7f649aa00700 1 -- 172.21.15.33:0/86858280 learned_addr learned my addr 172.21.15.33:0/86858280 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:12.886 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.208+0000 7f649aa00700 1 -- 172.21.15.33:0/86858280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c19f3b0 con 0x7f649c10af80 2024-04-26T11:17:12.886 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.208+0000 7f649aa00700 1 --2- 172.21.15.33:0/86858280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c19ee70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f648c003f40 tx=0x7f648c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:12.886 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.208+0000 7f6493400700 1 -- 172.21.15.33:0/86858280 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c0038b0 con 0x7f649c10af80 2024-04-26T11:17:12.886 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.209+0000 7f6493400700 1 -- 172.21.15.33:0/86858280 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f648c003a50 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.209+0000 7f6493400700 1 -- 172.21.15.33:0/86858280 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c01a5c0 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.209+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f649c19f640 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.209+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f649c1a2330 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.209+0000 7f6493400700 1 -- 172.21.15.33:0/86858280 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f648c01ac00 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.209+0000 7f6493400700 1 -- 172.21.15.33:0/86858280 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f648c019700 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.210+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f649c064a80 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.213+0000 7f6493400700 1 -- 172.21.15.33:0/86858280 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f648c019900 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.291+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f649c1a2500 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.292+0000 7f6493400700 1 -- 172.21.15.33:0/86858280 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f648c019ae0 con 0x7f649c10af80 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.294+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 msgr2=0x7f649c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.294+0000 7f64a1c00700 1 --2- 172.21.15.33:0/86858280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c19ee70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f648c003f40 tx=0x7f648c011890 comp rx=0 tx=0).stop 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.294+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 shutdown_connections 2024-04-26T11:17:12.887 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.294+0000 7f64a1c00700 1 --2- 172.21.15.33:0/86858280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f649c10af80 0x7f649c19ee70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:12.888 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.294+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 >> 172.21.15.33:0/86858280 conn(0x7f649c07e350 msgr2=0x7f649c108480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:12.888 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.295+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 shutdown_connections 2024-04-26T11:17:12.888 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:12.295+0000 7f64a1c00700 1 -- 172.21.15.33:0/86858280 wait complete. 2024-04-26T11:17:12.888 INFO:teuthology.orchestra.run.smithi033.stdout:Restarting the monitor... 2024-04-26T11:17:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:12 smithi033 systemd[1]: Stopping Ceph mon.a for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:17:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:13 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a[27476]: 2024-04-26T11:17:13.397+0000 7f3994400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T11:17:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:13 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a[27476]: 2024-04-26T11:17:13.397+0000 7f3994400700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T11:17:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:13 smithi033 podman[27722]: 2024-04-26 11:17:13.573028582 +0000 UTC m=+0.309872806 container died 79e3bf54db1efd4ecc430d8d4c6624991746a787e1319efb989aa07d29870778 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T11:17:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:13 smithi033 podman[27722]: 2024-04-26 11:17:13.980421623 +0000 UTC m=+0.717265873 container cleanup 79e3bf54db1efd4ecc430d8d4c6624991746a787e1319efb989aa07d29870778 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:17:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:13 smithi033 bash[27722]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a 2024-04-26T11:17:14.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:14 smithi033 podman[27736]: 2024-04-26 11:17:14.506573663 +0000 UTC m=+0.912495209 container remove 79e3bf54db1efd4ecc430d8d4c6624991746a787e1319efb989aa07d29870778 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:17:15.393 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T11:17:15.393 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 systemd[1]: Stopped Ceph mon.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:17:15.393 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.a.service: Consumed 2.021s CPU time. 2024-04-26T11:17:15.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 systemd[1]: Starting Ceph mon.a for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:17:15.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 podman[27827]: 2024-04-26 11:17:15.507287699 +0000 UTC m=+0.125047666 container create fcffa444f119013bc22abe2feded81edc30c3f1805af8cd930560348d84bf743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-26T11:17:15.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 podman[27827]: 2024-04-26 11:17:15.418809987 +0000 UTC m=+0.036569913 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:17:15.823 INFO:teuthology.orchestra.run.smithi033.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 podman[27827]: 2024-04-26 11:17:15.735555925 +0000 UTC m=+0.353315894 container init fcffa444f119013bc22abe2feded81edc30c3f1805af8cd930560348d84bf743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 podman[27827]: 2024-04-26 11:17:15.746878492 +0000 UTC m=+0.364638462 container start fcffa444f119013bc22abe2feded81edc30c3f1805af8cd930560348d84bf743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev), process ceph-mon, pid 2 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: pidfile_write: ignore empty --pid-file 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: load: jerasure load: lrc 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: RocksDB version: 7.9.2 2024-04-26T11:17:16.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Git sha 0 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Compile date 2024-04-26 07:20:26 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: DB SUMMARY 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: DB Session ID: BDZN7WR3CHY91QDZMB2F 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: CURRENT file: CURRENT 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104532 ; 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.error_if_exists: 0 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.create_if_missing: 0 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.paranoid_checks: 1 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T11:17:16.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.env: 0x558cc30dccc0 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.info_log: 0x558ce3953dc0 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.statistics: (nil) 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.use_fsync: 0 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_log_file_size: 0 2024-04-26T11:17:16.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.allow_fallocate: 1 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.use_direct_reads: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.db_log_dir: 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.wal_dir: 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T11:17:16.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.write_buffer_manager: 0x558ce39250e0 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T11:17:16.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.unordered_write: 0 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.row_cache: None 2024-04-26T11:17:16.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.wal_filter: None 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.two_write_queues: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.wal_compression: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.atomic_flush: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T11:17:16.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.log_readahead_size: 0 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_background_jobs: 2 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_background_compactions: -1 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_subcompactions: 1 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T11:17:16.157 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_open_files: -1 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_background_flushes: -1 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Compression algorithms supported: 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T11:17:16.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kZSTD supported: 0 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kXpressCompression supported: 0 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kZlibCompression supported: 1 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kSnappyCompression supported: 1 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kLZ4Compression supported: 1 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: kBZip2Compression supported: 0 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.merge_operator: 2024-04-26T11:17:16.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_filter: None 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558ce3953f80) 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: cache_index_and_filter_blocks: 1 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: pin_top_level_index_and_filter: 1 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_type: 0 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: data_block_index_type: 0 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_shortening: 1 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T11:17:16.160 INFO:journalctl@ceph.mon.a.smithi033.stdout: checksum: 4 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: no_block_cache: 0 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache: 0x558ce39571f0 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_name: BinnedLRUCache 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_options: 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: capacity : 536870912 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: num_shard_bits : 4 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: strict_capacity_limit : 0 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: high_pri_pool_ratio: 0.000 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_compressed: (nil) 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: persistent_cache: (nil) 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_size: 4096 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_size_deviation: 10 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_restart_interval: 16 2024-04-26T11:17:16.161 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_block_restart_interval: 1 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: metadata_block_size: 4096 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: partition_filters: 0 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: use_delta_encoding: 1 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: filter_policy: bloomfilter 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: whole_key_filtering: 1 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: verify_compression: 0 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: read_amp_bytes_per_bit: 0 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: format_version: 5 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: enable_index_compression: 1 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_align: 0 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: max_auto_readahead_size: 262144 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: prepopulate_block_cache: 0 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: initial_auto_readahead_size: 8192 2024-04-26T11:17:16.162 INFO:journalctl@ceph.mon.a.smithi033.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression: NoCompression 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.num_levels: 7 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T11:17:16.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 bash[27827]: fcffa444f119013bc22abe2feded81edc30c3f1805af8cd930560348d84bf743 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T11:17:16.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T11:17:16.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T11:17:16.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T11:17:16.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T11:17:16.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T11:17:16.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T11:17:16.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T11:17:16.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T11:17:16.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T11:17:16.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.inplace_update_support: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.bloom_locality: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.max_successive_merges: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T11:17:16.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.ttl: 2592000 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.enable_blob_files: false 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.min_blob_size: 0 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T11:17:16.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b30ff202-71f7-4ac3-85a8-1617be98b122 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130235806656, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130235808713, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99759, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97641, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 89999, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714130235, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b30ff202-71f7-4ac3-85a8-1617be98b122", "db_session_id": "BDZN7WR3CHY91QDZMB2F", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130235808870, "job": 1, "event": "recovery_finished"} 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T11:17:16.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558ce3a74000 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: DB pointer 0x558ce3a60000 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** DB Stats ** 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T11:17:16.173 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** Compaction Stats [default] ** 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: L0 2/0 99.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: Sum 2/0 99.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** Compaction Stats [default] ** 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T11:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 70.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative compaction: 0.00 GB write, 13.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval compaction: 0.00 GB write, 13.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T11:17:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout: Block cache BinnedLRUCache@0x558ce39571f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: starting mon.a rank 0 at public addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] at bind addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???) e1 preinit fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???).mds e1 new map 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???).mds e1 print_map 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: e1 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: legacy client fscid: -1 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout: No filesystems configured 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:17:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 systemd[1]: Started Ceph mon.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: monmap epoch 1 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: last_changed 2024-04-26T11:17:04.749857+0000 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: min_mon_release 19 (squid) 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: election_strategy: 1 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: fsmap 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T11:17:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:15 smithi033 ceph-mon[27845]: mgrmap e1: no daemons active 2024-04-26T11:17:17.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.470+0000 7fa589600700 1 Processor -- start 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.470+0000 7fa589600700 1 -- start start 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.470+0000 7fa589600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa584107990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.471+0000 7fa589600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa58407ced0 con 0x7fa584107570 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.471+0000 7fa582a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa584107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.471+0000 7fa582a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa584107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48916/0 (socket says 172.21.15.33:48916) 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.471+0000 7fa582a00700 1 -- 172.21.15.33:0/126455691 learned_addr learned my addr 172.21.15.33:0/126455691 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.472+0000 7fa582a00700 1 -- 172.21.15.33:0/126455691 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa584107ed0 con 0x7fa584107570 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.472+0000 7fa582a00700 1 --2- 172.21.15.33:0/126455691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa584107990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa574009e80 tx=0x7fa57400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bd0f9a91a66163 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.473+0000 7fa581600700 1 -- 172.21.15.33:0/126455691 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa574017070 con 0x7fa584107570 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.473+0000 7fa581600700 1 -- 172.21.15.33:0/126455691 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa57400b8c0 con 0x7fa584107570 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.473+0000 7fa581600700 1 -- 172.21.15.33:0/126455691 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa574004490 con 0x7fa584107570 2024-04-26T11:17:17.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.474+0000 7fa589600700 1 -- 172.21.15.33:0/126455691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 msgr2=0x7fa584107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.474+0000 7fa589600700 1 --2- 172.21.15.33:0/126455691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa584107990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa574009e80 tx=0x7fa57400b330 comp rx=0 tx=0).stop 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.475+0000 7fa589600700 1 -- 172.21.15.33:0/126455691 shutdown_connections 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.475+0000 7fa589600700 1 --2- 172.21.15.33:0/126455691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa584107990 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.475+0000 7fa589600700 1 -- 172.21.15.33:0/126455691 >> 172.21.15.33:0/126455691 conn(0x7fa584103210 msgr2=0x7fa584105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.475+0000 7fa589600700 1 -- 172.21.15.33:0/126455691 shutdown_connections 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.475+0000 7fa589600700 1 -- 172.21.15.33:0/126455691 wait complete. 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.476+0000 7fa589600700 1 Processor -- start 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.477+0000 7fa589600700 1 -- start start 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.477+0000 7fa589600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa58419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.477+0000 7fa589600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa584108f00 con 0x7fa584107570 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.477+0000 7fa582a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa58419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.477+0000 7fa582a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa58419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48922/0 (socket says 172.21.15.33:48922) 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.477+0000 7fa582a00700 1 -- 172.21.15.33:0/1257131231 learned_addr learned my addr 172.21.15.33:0/1257131231 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.478+0000 7fa582a00700 1 -- 172.21.15.33:0/1257131231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa58419f380 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.478+0000 7fa582a00700 1 --2- 172.21.15.33:0/1257131231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa58419ee40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa57400b7b0 tx=0x7fa57400be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.479+0000 7fa57b400700 1 -- 172.21.15.33:0/1257131231 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa574020020 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.479+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa58419f610 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.479+0000 7fa57b400700 1 -- 172.21.15.33:0/1257131231 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa574003d10 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.479+0000 7fa57b400700 1 -- 172.21.15.33:0/1257131231 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa574017040 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.479+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5841a2300 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.479+0000 7fa57b400700 1 -- 172.21.15.33:0/1257131231 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa574023020 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.479+0000 7fa57b400700 1 -- 172.21.15.33:0/1257131231 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa57400f050 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.480+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa584064a80 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.484+0000 7fa57b400700 1 -- 172.21.15.33:0/1257131231 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa57402a020 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.563+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fa58402e770 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.568+0000 7fa57b400700 1 -- 172.21.15.33:0/1257131231 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fa574003ec0 con 0x7fa584107570 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.569+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 msgr2=0x7fa58419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:17.276 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.569+0000 7fa589600700 1 --2- 172.21.15.33:0/1257131231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa58419ee40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa57400b7b0 tx=0x7fa57400be90 comp rx=0 tx=0).stop 2024-04-26T11:17:17.277 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.570+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 shutdown_connections 2024-04-26T11:17:17.277 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.570+0000 7fa589600700 1 --2- 172.21.15.33:0/1257131231 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa584107570 0x7fa58419ee40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:17.277 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.570+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 >> 172.21.15.33:0/1257131231 conn(0x7fa584103210 msgr2=0x7fa584195a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:17.277 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.570+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 shutdown_connections 2024-04-26T11:17:17.277 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:16.570+0000 7fa589600700 1 -- 172.21.15.33:0/1257131231 wait complete. 2024-04-26T11:17:17.277 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T11:17:17.299 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T11:17:17.299 INFO:teuthology.orchestra.run.smithi033.stdout:Creating mgr... 2024-04-26T11:17:17.300 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T11:17:17.301 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T11:17:17.724 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mgr.x 2024-04-26T11:17:17.724 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T11:17:17.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:17 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1257131231' entity='client.admin' 2024-04-26T11:17:18.123 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5cc7f146-03be-11ef-bc93-c7b262605968.target.wants/ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-5cc7f146-03be-11ef-bc93-c7b262605968@.service. 2024-04-26T11:17:18.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:18 smithi033 systemd[1]: Starting Ceph mgr.x for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:17:18.855 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:18 smithi033 podman[28066]: 2024-04-26 11:17:18.534752516 +0000 UTC m=+0.130203588 container create afcc88cd545955558ca3a0fd35091c2075a81977eca8237fead6d4e7733ffcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T11:17:18.855 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:18 smithi033 podman[28066]: 2024-04-26 11:17:18.441027488 +0000 UTC m=+0.036478522 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T11:17:18.855 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:18 smithi033 podman[28066]: 2024-04-26 11:17:18.7631556 +0000 UTC m=+0.358606715 container init afcc88cd545955558ca3a0fd35091c2075a81977eca8237fead6d4e7733ffcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-26T11:17:18.855 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:18 smithi033 podman[28066]: 2024-04-26 11:17:18.773803177 +0000 UTC m=+0.369254252 container start afcc88cd545955558ca3a0fd35091c2075a81977eca8237fead6d4e7733ffcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:17:18.855 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:18 smithi033 bash[28066]: afcc88cd545955558ca3a0fd35091c2075a81977eca8237fead6d4e7733ffcdf 2024-04-26T11:17:18.889 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T11:17:18.889 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-04-26T11:17:18.909 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T11:17:18.909 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-04-26T11:17:18.909 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-04-26T11:17:18.909 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T11:17:18.935 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T11:17:18.935 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-04-26T11:17:18.953 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T11:17:18.953 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-04-26T11:17:18.953 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-04-26T11:17:18.953 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T11:17:18.954 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr to start... 2024-04-26T11:17:18.954 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr... 2024-04-26T11:17:19.107 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:18 smithi033 systemd[1]: Started Ceph mgr.x for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:17:19.107 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:19.018+0000 7fbc7fb77200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:17:19.410 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:19.290+0000 7fbc7fb77200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:17:19.669 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:19.437+0000 7fbc7fb77200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:17:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:19 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1241489560' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T11:17:19.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:19.881+0000 7fbc7fb77200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:17:20.302 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:19.999+0000 7fbc7fb77200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:17:20.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:20.312 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:17:20.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:17:20.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T11:17:20.313 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T11:17:20.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T11:17:20.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T11:17:20.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:20.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T11:17:20.314 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T11:17:20.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-04-26T11:17:20.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:20.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T11:17:20.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T11:17:20.315 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T11:17:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:20.320 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T11:17:20.321 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:20.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:20.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:20.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T11:17:20.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:20.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T11:17:08.866977+0000", 2024-04-26T11:17:20.327 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.574+0000 7f1558a00700 1 Processor -- start 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.574+0000 7f1558a00700 1 -- start start 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.575+0000 7f1558a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f1554108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.575+0000 7f1558a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1554109520 con 0x7f1554108b30 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.576+0000 7f1552800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f1554108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.576+0000 7f1552800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f1554108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48950/0 (socket says 172.21.15.33:48950) 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.576+0000 7f1552800700 1 -- 172.21.15.33:0/3395616512 learned_addr learned my addr 172.21.15.33:0/3395616512 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.576+0000 7f1552800700 1 -- 172.21.15.33:0/3395616512 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1554109d30 con 0x7f1554108b30 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.577+0000 7f1552800700 1 --2- 172.21.15.33:0/3395616512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f1554108f50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f15440098a0 tx=0x7f1544009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cde27ef66c31aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.577+0000 7f1551400700 1 -- 172.21.15.33:0/3395616512 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1544004160 con 0x7f1554108b30 2024-04-26T11:17:20.328 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.577+0000 7f1551400700 1 -- 172.21.15.33:0/3395616512 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1544004300 con 0x7f1554108b30 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.577+0000 7f1551400700 1 -- 172.21.15.33:0/3395616512 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1544011af0 con 0x7f1554108b30 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.578+0000 7f1558a00700 1 -- 172.21.15.33:0/3395616512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 msgr2=0x7f1554108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.578+0000 7f1558a00700 1 --2- 172.21.15.33:0/3395616512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f1554108f50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f15440098a0 tx=0x7f1544009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.578+0000 7f1558a00700 1 -- 172.21.15.33:0/3395616512 shutdown_connections 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.578+0000 7f1558a00700 1 --2- 172.21.15.33:0/3395616512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f1554108f50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.578+0000 7f1558a00700 1 -- 172.21.15.33:0/3395616512 >> 172.21.15.33:0/3395616512 conn(0x7f15541040f0 msgr2=0x7f1554106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.578+0000 7f1558a00700 1 -- 172.21.15.33:0/3395616512 shutdown_connections 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.578+0000 7f1558a00700 1 -- 172.21.15.33:0/3395616512 wait complete. 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.580+0000 7f1558a00700 1 Processor -- start 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.580+0000 7f1558a00700 1 -- start start 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.580+0000 7f1558a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f155419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.580+0000 7f1558a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f155410d0e0 con 0x7f1554108b30 2024-04-26T11:17:20.329 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.581+0000 7f1552800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f155419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.581+0000 7f1552800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f155419cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48962/0 (socket says 172.21.15.33:48962) 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.581+0000 7f1552800700 1 -- 172.21.15.33:0/1241489560 learned_addr learned my addr 172.21.15.33:0/1241489560 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.581+0000 7f1552800700 1 -- 172.21.15.33:0/1241489560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155419d080 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.581+0000 7f1552800700 1 --2- 172.21.15.33:0/1241489560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f155419cb40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f15440117b0 tx=0x7f1544011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.582+0000 7f154b400700 1 -- 172.21.15.33:0/1241489560 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1544003af0 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.582+0000 7f154b400700 1 -- 172.21.15.33:0/1241489560 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1544020070 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.582+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155419d310 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.582+0000 7f154b400700 1 -- 172.21.15.33:0/1241489560 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1544019630 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.582+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15541a0000 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.582+0000 7f154b400700 1 -- 172.21.15.33:0/1241489560 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1544019c30 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.582+0000 7f154b400700 1 -- 172.21.15.33:0/1241489560 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f154400ba70 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.583+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f155406a530 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.586+0000 7f154b400700 1 -- 172.21.15.33:0/1241489560 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f15440042c0 con 0x7f1554108b30 2024-04-26T11:17:20.330 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.667+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f15541a0550 con 0x7f1554108b30 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.668+0000 7f154b400700 1 -- 172.21.15.33:0/1241489560 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f154401a9c0 con 0x7f1554108b30 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.669+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 msgr2=0x7f155419cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.669+0000 7f1558a00700 1 --2- 172.21.15.33:0/1241489560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f155419cb40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f15440117b0 tx=0x7f1544011b70 comp rx=0 tx=0).stop 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.670+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 shutdown_connections 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.670+0000 7f1558a00700 1 --2- 172.21.15.33:0/1241489560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1554108b30 0x7f155419cb40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.670+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 >> 172.21.15.33:0/1241489560 conn(0x7f15541040f0 msgr2=0x7f1554104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.670+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 shutdown_connections 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:19.670+0000 7f1558a00700 1 -- 172.21.15.33:0/1241489560 wait complete. 2024-04-26T11:17:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:mgr not available, waiting (1/15)... 2024-04-26T11:17:21.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:21 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:21.633+0000 7fbc7fb77200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:17:21.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:21 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:21.865+0000 7fbc7fb77200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:17:22.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:22.084+0000 7fbc7fb77200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:17:22.763 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:22.480+0000 7fbc7fb77200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:17:22.809 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:22.610+0000 7fbc7fb77200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:17:23.014 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:22.876+0000 7fbc7fb77200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:17:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:23 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3077220964' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T11:17:23.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:23.143+0000 7fbc7fb77200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:17:24.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:24.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:17:24.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:17:24.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T11:17:24.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T11:17:24.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T11:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T11:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T11:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T11:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:24.027 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T11:17:24.028 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T11:17:24.028 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T11:17:24.036 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T11:17:24.037 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T11:17:08.866977+0000", 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.921+0000 7ff567a00700 1 Processor -- start 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.922+0000 7ff567a00700 1 -- start start 2024-04-26T11:17:24.038 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.922+0000 7ff567a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff56010bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.922+0000 7ff567a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff56007ce90 con 0x7ff5601097c0 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.922+0000 7ff565000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff56010bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.922+0000 7ff565000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff56010bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59044/0 (socket says 172.21.15.33:59044) 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.922+0000 7ff565000700 1 -- 172.21.15.33:0/1649932800 learned_addr learned my addr 172.21.15.33:0/1649932800 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.923+0000 7ff565000700 1 -- 172.21.15.33:0/1649932800 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56010c0f0 con 0x7ff5601097c0 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.923+0000 7ff565000700 1 --2- 172.21.15.33:0/1649932800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff56010bbb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5540098a0 tx=0x7ff554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9f4aedbf674709 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.923+0000 7ff55f400700 1 -- 172.21.15.33:0/1649932800 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff554004160 con 0x7ff5601097c0 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.924+0000 7ff55f400700 1 -- 172.21.15.33:0/1649932800 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff554004300 con 0x7ff5601097c0 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.924+0000 7ff55f400700 1 -- 172.21.15.33:0/1649932800 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff554011af0 con 0x7ff5601097c0 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.924+0000 7ff567a00700 1 -- 172.21.15.33:0/1649932800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 msgr2=0x7ff56010bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.924+0000 7ff567a00700 1 --2- 172.21.15.33:0/1649932800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff56010bbb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5540098a0 tx=0x7ff554009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.924+0000 7ff567a00700 1 -- 172.21.15.33:0/1649932800 shutdown_connections 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.924+0000 7ff567a00700 1 --2- 172.21.15.33:0/1649932800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff56010bbb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:24.039 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.924+0000 7ff567a00700 1 -- 172.21.15.33:0/1649932800 >> 172.21.15.33:0/1649932800 conn(0x7ff560103210 msgr2=0x7ff560105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.925+0000 7ff567a00700 1 -- 172.21.15.33:0/1649932800 shutdown_connections 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.925+0000 7ff567a00700 1 -- 172.21.15.33:0/1649932800 wait complete. 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.926+0000 7ff567a00700 1 Processor -- start 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.926+0000 7ff567a00700 1 -- start start 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.926+0000 7ff567a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff560109040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.926+0000 7ff567a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff56010d0b0 con 0x7ff5601097c0 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.926+0000 7ff565000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff560109040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.926+0000 7ff565000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff560109040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59052/0 (socket says 172.21.15.33:59052) 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.926+0000 7ff565000700 1 -- 172.21.15.33:0/3077220964 learned_addr learned my addr 172.21.15.33:0/3077220964 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.927+0000 7ff565000700 1 -- 172.21.15.33:0/3077220964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff560109580 con 0x7ff5601097c0 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.927+0000 7ff565000700 1 --2- 172.21.15.33:0/3077220964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff560109040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff554011ac0 tx=0x7ff554011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.927+0000 7ff55d600700 1 -- 172.21.15.33:0/3077220964 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff554003b20 con 0x7ff5601097c0 2024-04-26T11:17:24.040 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.927+0000 7ff55d600700 1 -- 172.21.15.33:0/3077220964 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff55401a8c0 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.927+0000 7ff55d600700 1 -- 172.21.15.33:0/3077220964 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5540248d0 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.927+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff560107770 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.928+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff560107cb0 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.929+0000 7ff55d600700 1 -- 172.21.15.33:0/3077220964 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff554022020 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.929+0000 7ff55d600700 1 -- 172.21.15.33:0/3077220964 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff55402d980 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.929+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff560064a80 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:22.932+0000 7ff55d600700 1 -- 172.21.15.33:0/3077220964 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff55402d3f0 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.011+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff5601081e0 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.012+0000 7ff55d600700 1 -- 172.21.15.33:0/3077220964 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff55401aa70 con 0x7ff5601097c0 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.014+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 msgr2=0x7ff560109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.014+0000 7ff567a00700 1 --2- 172.21.15.33:0/3077220964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff560109040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff554011ac0 tx=0x7ff554011ba0 comp rx=0 tx=0).stop 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.014+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 shutdown_connections 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.014+0000 7ff567a00700 1 --2- 172.21.15.33:0/3077220964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5601097c0 0x7ff560109040 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:24.041 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.014+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 >> 172.21.15.33:0/3077220964 conn(0x7ff560103210 msgr2=0x7ff560103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:24.042 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.015+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 shutdown_connections 2024-04-26T11:17:24.042 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:23.015+0000 7ff567a00700 1 -- 172.21.15.33:0/3077220964 wait complete. 2024-04-26T11:17:24.042 INFO:teuthology.orchestra.run.smithi033.stdout:mgr not available, waiting (2/15)... 2024-04-26T11:17:24.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:23.977+0000 7fbc7fb77200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:17:24.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:24 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:24.540+0000 7fbc7fb77200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:17:24.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:24 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:24.671+0000 7fbc7fb77200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:17:25.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:24 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:24.943+0000 7fbc7fb77200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:17:25.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:25.611+0000 7fbc7fb77200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:17:25.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:25.737+0000 7fbc7fb77200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:17:26.321 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:26.068+0000 7fbc7fb77200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:17:26.321 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:26.256+0000 7fbc7fb77200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:17:26.653 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:26.389+0000 7fbc7fb77200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:17:26.654 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:26.548+0000 7fbc7fb77200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:17:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:26 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3298567354' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T11:17:27.230 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:26.932+0000 7fbc7fb77200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:17:27.249 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:27.249 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T11:17:27.250 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T11:17:27.251 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:27.252 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T11:17:27.253 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T11:17:08.866977+0000", 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.561+0000 7fc5bea00700 1 Processor -- start 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.562+0000 7fc5bea00700 1 -- start start 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.562+0000 7fc5bea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b807d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.562+0000 7fc5bea00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b807d820 con 0x7fc5b807ee30 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.563+0000 7fc5b7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b807d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.563+0000 7fc5b7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b807d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59054/0 (socket says 172.21.15.33:59054) 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.563+0000 7fc5b7e00700 1 -- 172.21.15.33:0/2275651226 learned_addr learned my addr 172.21.15.33:0/2275651226 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.563+0000 7fc5b7e00700 1 -- 172.21.15.33:0/2275651226 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b807d9a0 con 0x7fc5b807ee30 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.563+0000 7fc5b7e00700 1 --2- 172.21.15.33:0/2275651226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b807d2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80098a0 tx=0x7fc5a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e927ed30d5c1f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.564+0000 7fc5b7400700 1 -- 172.21.15.33:0/2275651226 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a8004160 con 0x7fc5b807ee30 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.564+0000 7fc5b7400700 1 -- 172.21.15.33:0/2275651226 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc5a8004300 con 0x7fc5b807ee30 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.564+0000 7fc5b7400700 1 -- 172.21.15.33:0/2275651226 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a8011af0 con 0x7fc5b807ee30 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.564+0000 7fc5bea00700 1 -- 172.21.15.33:0/2275651226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 msgr2=0x7fc5b807d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.564+0000 7fc5bea00700 1 --2- 172.21.15.33:0/2275651226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b807d2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80098a0 tx=0x7fc5a8009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.564+0000 7fc5bea00700 1 -- 172.21.15.33:0/2275651226 shutdown_connections 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.565+0000 7fc5bea00700 1 --2- 172.21.15.33:0/2275651226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b807d2e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.565+0000 7fc5bea00700 1 -- 172.21.15.33:0/2275651226 >> 172.21.15.33:0/2275651226 conn(0x7fc5b8104300 msgr2=0x7fc5b8106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.565+0000 7fc5bea00700 1 -- 172.21.15.33:0/2275651226 shutdown_connections 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.565+0000 7fc5bea00700 1 -- 172.21.15.33:0/2275651226 wait complete. 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.566+0000 7fc5bea00700 1 Processor -- start 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.566+0000 7fc5bea00700 1 -- start start 2024-04-26T11:17:27.255 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.566+0000 7fc5bea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.566+0000 7fc5bea00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b810f2d0 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.566+0000 7fc5b7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.566+0000 7fc5b7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b819ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59070/0 (socket says 172.21.15.33:59070) 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.566+0000 7fc5b7e00700 1 -- 172.21.15.33:0/3298567354 learned_addr learned my addr 172.21.15.33:0/3298567354 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.567+0000 7fc5b7e00700 1 -- 172.21.15.33:0/3298567354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b819f3b0 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.567+0000 7fc5b7e00700 1 --2- 172.21.15.33:0/3298567354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b819ee70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8011ac0 tx=0x7fc5a8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.567+0000 7fc5b5600700 1 -- 172.21.15.33:0/3298567354 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a8003b20 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.567+0000 7fc5b5600700 1 -- 172.21.15.33:0/3298567354 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc5a801a8c0 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.567+0000 7fc5b5600700 1 -- 172.21.15.33:0/3298567354 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a80248d0 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.567+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b819f640 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.568+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b81a2330 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.568+0000 7fc5b5600700 1 -- 172.21.15.33:0/3298567354 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc5a8022020 con 0x7fc5b807ee30 2024-04-26T11:17:27.256 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.568+0000 7fc5b5600700 1 -- 172.21.15.33:0/3298567354 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc5a80193c0 con 0x7fc5b807ee30 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.569+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b8064a80 con 0x7fc5b807ee30 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.571+0000 7fc5b5600700 1 -- 172.21.15.33:0/3298567354 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc5a8042e50 con 0x7fc5b807ee30 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.652+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc5b81a2870 con 0x7fc5b807ee30 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.653+0000 7fc5b5600700 1 -- 172.21.15.33:0/3298567354 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fc5a8031020 con 0x7fc5b807ee30 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.654+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 msgr2=0x7fc5b819ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.654+0000 7fc5bea00700 1 --2- 172.21.15.33:0/3298567354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b819ee70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8011ac0 tx=0x7fc5a8011ba0 comp rx=0 tx=0).stop 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.655+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 shutdown_connections 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.655+0000 7fc5bea00700 1 --2- 172.21.15.33:0/3298567354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5b807ee30 0x7fc5b819ee70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.655+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 >> 172.21.15.33:0/3298567354 conn(0x7fc5b8104300 msgr2=0x7fc5b8104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.655+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 shutdown_connections 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:26.655+0000 7fc5bea00700 1 -- 172.21.15.33:0/3298567354 wait complete. 2024-04-26T11:17:27.257 INFO:teuthology.orchestra.run.smithi033.stdout:mgr not available, waiting (3/15)... 2024-04-26T11:17:28.049 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:27 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:27.754+0000 7fbc7fb77200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:28.047+0000 7fbc7fb77200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: Activating manager daemon x 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: mgrmap e2: x(active, starting, since 0.0125015s) 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: Manager daemon x is now available 2024-04-26T11:17:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:17:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' 2024-04-26T11:17:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:17:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:28 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' 2024-04-26T11:17:29.406 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:29 smithi033 ceph-mon[27845]: from='mgr.14100 172.21.15.33:0/4228536384' entity='mgr.x' 2024-04-26T11:17:29.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:29 smithi033 ceph-mon[27845]: mgrmap e3: x(active, since 1.02231s) 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T11:17:30.812 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:30.813 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T11:17:30.814 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T11:17:30.814 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T11:17:30.822 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T11:17:30.822 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T11:17:30.822 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T11:17:30.822 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:30.822 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T11:17:30.822 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:30.823 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T11:17:08.866977+0000", 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T11:17:30.824 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.849+0000 7f5903800700 1 Processor -- start 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.850+0000 7f5903800700 1 -- start start 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.850+0000 7f5903800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc07d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.850+0000 7f5903800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc07d840 con 0x7f58fc07ee50 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.851+0000 7f5900e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc07d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.851+0000 7f5900e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc07d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52768/0 (socket says 172.21.15.33:52768) 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.851+0000 7f5900e00700 1 -- 172.21.15.33:0/1074780987 learned_addr learned my addr 172.21.15.33:0/1074780987 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.851+0000 7f5900e00700 1 -- 172.21.15.33:0/1074780987 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc07d9c0 con 0x7f58fc07ee50 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.851+0000 7f5900e00700 1 --2- 172.21.15.33:0/1074780987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc07d300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f58f00098a0 tx=0x7f58f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36ebab707906324 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.852+0000 7f58fb400700 1 -- 172.21.15.33:0/1074780987 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f0004160 con 0x7f58fc07ee50 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.852+0000 7f58fb400700 1 -- 172.21.15.33:0/1074780987 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f58f0004300 con 0x7f58fc07ee50 2024-04-26T11:17:30.825 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.852+0000 7f58fb400700 1 -- 172.21.15.33:0/1074780987 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f0011af0 con 0x7f58fc07ee50 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.852+0000 7f5903800700 1 -- 172.21.15.33:0/1074780987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 msgr2=0x7f58fc07d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.852+0000 7f5903800700 1 --2- 172.21.15.33:0/1074780987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc07d300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f58f00098a0 tx=0x7f58f0009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.853+0000 7f5903800700 1 -- 172.21.15.33:0/1074780987 shutdown_connections 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.853+0000 7f5903800700 1 --2- 172.21.15.33:0/1074780987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc07d300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.853+0000 7f5903800700 1 -- 172.21.15.33:0/1074780987 >> 172.21.15.33:0/1074780987 conn(0x7f58fc104300 msgr2=0x7f58fc106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.853+0000 7f5903800700 1 -- 172.21.15.33:0/1074780987 shutdown_connections 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.853+0000 7f5903800700 1 -- 172.21.15.33:0/1074780987 wait complete. 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.854+0000 7f5903800700 1 Processor -- start 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.854+0000 7f5903800700 1 -- start start 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.854+0000 7f5903800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.854+0000 7f5903800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc10f2b0 con 0x7f58fc07ee50 2024-04-26T11:17:30.826 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.855+0000 7f5900e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.855+0000 7f5900e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc19ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52780/0 (socket says 172.21.15.33:52780) 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.855+0000 7f5900e00700 1 -- 172.21.15.33:0/843437759 learned_addr learned my addr 172.21.15.33:0/843437759 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.855+0000 7f5900e00700 1 -- 172.21.15.33:0/843437759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc19f3c0 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.855+0000 7f5900e00700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc19ee80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f58f0011ac0 tx=0x7f58f0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.856+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f0003b20 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.856+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f58f001a8c0 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.856+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f00248d0 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.856+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc19f650 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.856+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58fc1a2340 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.857+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7f58f0022020 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.857+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58fc064a80 con 0x7f58fc07ee50 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.858+0000 7f58f9600700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f58e4039490 0x7f58e403b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:30.827 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.858+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f58f004d280 con 0x7f58fc07ee50 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.858+0000 7f58fbe00700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f58e4039490 0x7f58e403b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.859+0000 7f58fbe00700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f58e4039490 0x7f58e403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f58e8006fd0 tx=0x7f58e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:29.864+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58f001ab20 con 0x7f58fc07ee50 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.112+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f58f002d3f0 con 0x7f58fc07ee50 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.181+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f58fc1a2510 con 0x7f58fc07ee50 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.182+0000 7f58f9600700 1 -- 172.21.15.33:0/843437759 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f58fc1a2510 con 0x7f58fc07ee50 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f58e4039490 msgr2=0x7f58e403b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f58e4039490 0x7f58e403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f58e8006fd0 tx=0x7f58e8006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 msgr2=0x7f58fc19ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc19ee80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f58f0011ac0 tx=0x7f58f0011ba0 comp rx=0 tx=0).stop 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 shutdown_connections 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f58e4039490 0x7f58e403b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 --2- 172.21.15.33:0/843437759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f58fc07ee50 0x7f58fc19ee80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:30.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 >> 172.21.15.33:0/843437759 conn(0x7f58fc104300 msgr2=0x7f58fc104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:30.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 shutdown_connections 2024-04-26T11:17:30.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:30.185+0000 7f5903800700 1 -- 172.21.15.33:0/843437759 wait complete. 2024-04-26T11:17:30.829 INFO:teuthology.orchestra.run.smithi033.stdout:mgr is available 2024-04-26T11:17:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:31 smithi033 ceph-mon[27845]: mgrmap e4: x(active, since 2s) 2024-04-26T11:17:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:31 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/843437759' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T11:17:32.343 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:32 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1516860140' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T11:17:32.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:32.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout fsid = 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T11:17:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.436+0000 7f0527800700 1 Processor -- start 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.436+0000 7f0527800700 1 -- start start 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.436+0000 7f0527800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052007d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.436+0000 7f0527800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f052007d840 con 0x7f052007ee50 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.437+0000 7f0524e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052007d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.437+0000 7f0524e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052007d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52786/0 (socket says 172.21.15.33:52786) 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.437+0000 7f0524e00700 1 -- 172.21.15.33:0/2006160395 learned_addr learned my addr 172.21.15.33:0/2006160395 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:32.363 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.437+0000 7f0524e00700 1 -- 172.21.15.33:0/2006160395 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052007d9c0 con 0x7f052007ee50 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.438+0000 7f0524e00700 1 --2- 172.21.15.33:0/2006160395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052007d300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0510009e80 tx=0x7f051000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2282667e3424c11e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.438+0000 7f051fe00700 1 -- 172.21.15.33:0/2006160395 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0510017070 con 0x7f052007ee50 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.438+0000 7f051fe00700 1 -- 172.21.15.33:0/2006160395 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f051000b8c0 con 0x7f052007ee50 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.438+0000 7f051fe00700 1 -- 172.21.15.33:0/2006160395 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0510004490 con 0x7f052007ee50 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.439+0000 7f0527800700 1 -- 172.21.15.33:0/2006160395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 msgr2=0x7f052007d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.439+0000 7f0527800700 1 --2- 172.21.15.33:0/2006160395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052007d300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0510009e80 tx=0x7f051000b330 comp rx=0 tx=0).stop 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.439+0000 7f0527800700 1 -- 172.21.15.33:0/2006160395 shutdown_connections 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.439+0000 7f0527800700 1 --2- 172.21.15.33:0/2006160395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052007d300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.439+0000 7f0527800700 1 -- 172.21.15.33:0/2006160395 >> 172.21.15.33:0/2006160395 conn(0x7f0520104300 msgr2=0x7f0520106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.439+0000 7f0527800700 1 -- 172.21.15.33:0/2006160395 shutdown_connections 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.439+0000 7f0527800700 1 -- 172.21.15.33:0/2006160395 wait complete. 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.440+0000 7f0527800700 1 Processor -- start 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.441+0000 7f0527800700 1 -- start start 2024-04-26T11:17:32.364 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.441+0000 7f0527800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.441+0000 7f0527800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f052010f2b0 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.441+0000 7f0524e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.441+0000 7f0524e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052019ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52800/0 (socket says 172.21.15.33:52800) 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.441+0000 7f0524e00700 1 -- 172.21.15.33:0/1516860140 learned_addr learned my addr 172.21.15.33:0/1516860140 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.442+0000 7f0524e00700 1 -- 172.21.15.33:0/1516860140 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052019f360 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.442+0000 7f0524e00700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052019ee20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f051000b7b0 tx=0x7f051000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.442+0000 7f051e000700 1 -- 172.21.15.33:0/1516860140 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0510020020 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.442+0000 7f051e000700 1 -- 172.21.15.33:0/1516860140 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0510003c60 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.442+0000 7f051e000700 1 -- 172.21.15.33:0/1516860140 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0510017040 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.442+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f052019f5f0 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.442+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05201a22e0 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.444+0000 7f051e000700 1 -- 172.21.15.33:0/1516860140 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f0510023020 con 0x7f052007ee50 2024-04-26T11:17:32.365 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.444+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0520064a80 con 0x7f052007ee50 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.444+0000 7f051e000700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f05040395b0 0x7f050403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.444+0000 7f051e000700 1 -- 172.21.15.33:0/1516860140 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f051000f050 con 0x7f052007ee50 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.445+0000 7f0517e00700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f05040395b0 0x7f050403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.445+0000 7f0517e00700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f05040395b0 0x7f050403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0508006fd0 tx=0x7f0508006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.450+0000 7f051e000700 1 -- 172.21.15.33:0/1516860140 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f051002a020 con 0x7f052007ee50 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.678+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f05201a3b50 con 0x7f052007ee50 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.680+0000 7f051e000700 1 -- 172.21.15.33:0/1516860140 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f051002c300 con 0x7f052007ee50 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.683+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f05040395b0 msgr2=0x7f050403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.683+0000 7f0527800700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f05040395b0 0x7f050403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0508006fd0 tx=0x7f0508006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 msgr2=0x7f052019ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052019ee20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f051000b7b0 tx=0x7f051000bda0 comp rx=0 tx=0).stop 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 shutdown_connections 2024-04-26T11:17:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f05040395b0 0x7f050403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:32.367 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 --2- 172.21.15.33:0/1516860140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f052007ee50 0x7f052019ee20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:32.367 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 >> 172.21.15.33:0/1516860140 conn(0x7f0520104300 msgr2=0x7f0520104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:32.367 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 shutdown_connections 2024-04-26T11:17:32.367 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:31.684+0000 7f0527800700 1 -- 172.21.15.33:0/1516860140 wait complete. 2024-04-26T11:17:32.367 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling cephadm module... 2024-04-26T11:17:33.611 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:33 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2535285948' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T11:17:33.611 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:33 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: ignoring --setuser ceph since I am not root 2024-04-26T11:17:33.612 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:33 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: ignoring --setgroup ceph since I am not root 2024-04-26T11:17:33.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:33 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:33.610+0000 7f5db0109200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:17:33.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:33 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:33.855+0000 7f5db0109200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:17:34.017 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.966+0000 7f7b93600700 1 Processor -- start 2024-04-26T11:17:34.017 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.967+0000 7f7b93600700 1 -- start start 2024-04-26T11:17:34.018 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.967+0000 7f7b93600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c10bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:34.018 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.967+0000 7f7b93600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c07ce90 con 0x7f7b8c1097c0 2024-04-26T11:17:34.018 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.968+0000 7f7b90c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c10bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:34.018 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.968+0000 7f7b90c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c10bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52810/0 (socket says 172.21.15.33:52810) 2024-04-26T11:17:34.018 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.968+0000 7f7b90c00700 1 -- 172.21.15.33:0/4093883091 learned_addr learned my addr 172.21.15.33:0/4093883091 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:34.018 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.968+0000 7f7b90c00700 1 -- 172.21.15.33:0/4093883091 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c10c0f0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.969+0000 7f7b90c00700 1 --2- 172.21.15.33:0/4093883091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c10bbb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7b800098a0 tx=0x7f7b80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46c46d78f908da5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.969+0000 7f7b8b400700 1 -- 172.21.15.33:0/4093883091 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b80004160 con 0x7f7b8c1097c0 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.969+0000 7f7b8b400700 1 -- 172.21.15.33:0/4093883091 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7b80004300 con 0x7f7b8c1097c0 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.969+0000 7f7b8b400700 1 -- 172.21.15.33:0/4093883091 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b80011af0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.970+0000 7f7b93600700 1 -- 172.21.15.33:0/4093883091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 msgr2=0x7f7b8c10bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.970+0000 7f7b93600700 1 --2- 172.21.15.33:0/4093883091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c10bbb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7b800098a0 tx=0x7f7b80009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.970+0000 7f7b93600700 1 -- 172.21.15.33:0/4093883091 shutdown_connections 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.970+0000 7f7b93600700 1 --2- 172.21.15.33:0/4093883091 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c10bbb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:34.023 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.970+0000 7f7b93600700 1 -- 172.21.15.33:0/4093883091 >> 172.21.15.33:0/4093883091 conn(0x7f7b8c103210 msgr2=0x7f7b8c105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.970+0000 7f7b93600700 1 -- 172.21.15.33:0/4093883091 shutdown_connections 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.970+0000 7f7b93600700 1 -- 172.21.15.33:0/4093883091 wait complete. 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.971+0000 7f7b93600700 1 Processor -- start 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.971+0000 7f7b93600700 1 -- start start 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.971+0000 7f7b93600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.972+0000 7f7b93600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c10d0b0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.972+0000 7f7b90c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.972+0000 7f7b90c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c19eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52820/0 (socket says 172.21.15.33:52820) 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.972+0000 7f7b90c00700 1 -- 172.21.15.33:0/2535285948 learned_addr learned my addr 172.21.15.33:0/2535285948 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.972+0000 7f7b90c00700 1 -- 172.21.15.33:0/2535285948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c19f420 con 0x7f7b8c1097c0 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.973+0000 7f7b90c00700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c19eee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b80011ac0 tx=0x7f7b80011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.973+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b80003b20 con 0x7f7b8c1097c0 2024-04-26T11:17:34.024 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.973+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7b8001a8c0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.973+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c19f6b0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.973+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b800248d0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.973+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c1a23a0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.974+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f7b80022020 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.974+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8c064a80 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.975+0000 7f7b89600700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f7b740395b0 0x7f7b7403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.975+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7b8004d490 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.975+0000 7f7b8be00700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f7b740395b0 0x7f7b7403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.976+0000 7f7b8be00700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f7b740395b0 0x7f7b7403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006fd0 tx=0x7f7b78006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:32.981+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b8000f2e0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.275+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f7b8c1a2570 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.329+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f7b8c1a2570 con 0x7f7b8c1097c0 2024-04-26T11:17:34.025 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.333+0000 7f7b89600700 1 -- 172.21.15.33:0/2535285948 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f7b8002d3f0 con 0x7f7b8c1097c0 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.333+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f7b740395b0 msgr2=0x7f7b7403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.333+0000 7f7b93600700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f7b740395b0 0x7f7b7403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006fd0 tx=0x7f7b78006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.334+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 msgr2=0x7f7b8c19eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.334+0000 7f7b93600700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c19eee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b80011ac0 tx=0x7f7b80011ba0 comp rx=0 tx=0).stop 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.335+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 shutdown_connections 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.335+0000 7f7b93600700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f7b740395b0 0x7f7b7403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.335+0000 7f7b93600700 1 --2- 172.21.15.33:0/2535285948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b8c1097c0 0x7f7b8c19eee0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.335+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 >> 172.21.15.33:0/2535285948 conn(0x7f7b8c103210 msgr2=0x7f7b8c103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.335+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 shutdown_connections 2024-04-26T11:17:34.026 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:33.335+0000 7f7b93600700 1 -- 172.21.15.33:0/2535285948 wait complete. 2024-04-26T11:17:34.327 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:33 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:33.990+0000 7f5db0109200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:17:34.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:34 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2535285948' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T11:17:34.593 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:34 smithi033 ceph-mon[27845]: mgrmap e5: x(active, since 5s) 2024-04-26T11:17:34.593 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:34.404+0000 7f5db0109200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:17:34.593 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:34.522+0000 7f5db0109200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:17:35.605 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:35 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/4063987101' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T11:17:35.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.590+0000 7f2fffe00700 1 Processor -- start 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.591+0000 7f2fffe00700 1 -- start start 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.592+0000 7f2fffe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001097e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.592+0000 7f2fffe00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f300007ced0 con 0x7f30001073b0 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.592+0000 7f2ffea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.592+0000 7f2ffea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001097e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52856/0 (socket says 172.21.15.33:52856) 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.592+0000 7f2ffea00700 1 -- 172.21.15.33:0/535111335 learned_addr learned my addr 172.21.15.33:0/535111335 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.593+0000 7f2ffea00700 1 -- 172.21.15.33:0/535111335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3000109d20 con 0x7f30001073b0 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.593+0000 7f2ffea00700 1 --2- 172.21.15.33:0/535111335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001097e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009e80 tx=0x7f2ff000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4473da8d1254a092 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.593+0000 7f2ffe000700 1 -- 172.21.15.33:0/535111335 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ff0017070 con 0x7f30001073b0 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.593+0000 7f2ffe000700 1 -- 172.21.15.33:0/535111335 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ff000b8c0 con 0x7f30001073b0 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.594+0000 7f2ffe000700 1 -- 172.21.15.33:0/535111335 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ff0004490 con 0x7f30001073b0 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.594+0000 7f2fffe00700 1 -- 172.21.15.33:0/535111335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 msgr2=0x7f30001097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.594+0000 7f2fffe00700 1 --2- 172.21.15.33:0/535111335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001097e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009e80 tx=0x7f2ff000b330 comp rx=0 tx=0).stop 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.594+0000 7f2fffe00700 1 -- 172.21.15.33:0/535111335 shutdown_connections 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.594+0000 7f2fffe00700 1 --2- 172.21.15.33:0/535111335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001097e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.594+0000 7f2fffe00700 1 -- 172.21.15.33:0/535111335 >> 172.21.15.33:0/535111335 conn(0x7f3000102fd0 msgr2=0x7f3000105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.595+0000 7f2fffe00700 1 -- 172.21.15.33:0/535111335 shutdown_connections 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.595+0000 7f2fffe00700 1 -- 172.21.15.33:0/535111335 wait complete. 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.596+0000 7f2fffe00700 1 Processor -- start 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.596+0000 7f2fffe00700 1 -- start start 2024-04-26T11:17:35.619 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.597+0000 7f2fffe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.597+0000 7f2fffe00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f300010ad50 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.597+0000 7f2ffea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.597+0000 7f2ffea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001a30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52858/0 (socket says 172.21.15.33:52858) 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.597+0000 7f2ffea00700 1 -- 172.21.15.33:0/4063987101 learned_addr learned my addr 172.21.15.33:0/4063987101 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.597+0000 7f2ffea00700 1 -- 172.21.15.33:0/4063987101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30001a3600 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.598+0000 7f2ffea00700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001a30c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b7b0 tx=0x7f2ff000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.598+0000 7f2ff7e00700 1 -- 172.21.15.33:0/4063987101 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ff0020020 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.598+0000 7f2ff7e00700 1 -- 172.21.15.33:0/4063987101 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ff0003d10 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.598+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30001a3890 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.598+0000 7f2ff7e00700 1 -- 172.21.15.33:0/4063987101 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ff0017040 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.598+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30001a6580 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.599+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f300006a530 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.599+0000 7f2ff7e00700 1 -- 172.21.15.33:0/4063987101 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f2ff0023020 con 0x7f30001073b0 2024-04-26T11:17:35.620 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.600+0000 7f2ff7e00700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 0x7f2fe403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.600+0000 7f2ff7e00700 1 -- 172.21.15.33:0/4063987101 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ff000f050 con 0x7f30001073b0 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.600+0000 7f2ff6000700 1 -- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 msgr2=0x7f2fe403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.600+0000 7f2ff6000700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 0x7f2fe403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.605+0000 7f2ff7e00700 1 -- 172.21.15.33:0/4063987101 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ff002a020 con 0x7f30001073b0 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.801+0000 7f2ff6000700 1 -- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 msgr2=0x7f2fe403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.801+0000 7f2ff6000700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 0x7f2fe403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.896+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f300002e7d0 con 0x7f30001073b0 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.897+0000 7f2ff7e00700 1 -- 172.21.15.33:0/4063987101 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f2ff002b260 con 0x7f30001073b0 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.899+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 msgr2=0x7f2fe403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:17:35.621 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.899+0000 7f2fffe00700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 0x7f2fe403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.899+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 msgr2=0x7f30001a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.899+0000 7f2fffe00700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001a30c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b7b0 tx=0x7f2ff000be90 comp rx=0 tx=0).stop 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.900+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 shutdown_connections 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.900+0000 7f2fffe00700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7f2fe4039600 0x7f2fe403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.900+0000 7f2fffe00700 1 --2- 172.21.15.33:0/4063987101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f30001073b0 0x7f30001a30c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.900+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 >> 172.21.15.33:0/4063987101 conn(0x7f3000102fd0 msgr2=0x7f3000105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.900+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 shutdown_connections 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:34.900+0000 7f2fffe00700 1 -- 172.21.15.33:0/4063987101 wait complete. 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for the mgr to restart... 2024-04-26T11:17:35.622 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr epoch 5... 2024-04-26T11:17:36.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:36 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:36.122+0000 7f5db0109200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:17:36.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:36 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:36.356+0000 7f5db0109200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:17:36.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:36 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:36.574+0000 7f5db0109200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:17:37.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:36 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:36.968+0000 7f5db0109200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:17:37.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:37 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:37.095+0000 7f5db0109200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:17:37.616 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:37 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:37.352+0000 7f5db0109200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:17:37.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:37 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:37.614+0000 7f5db0109200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:17:38.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:38 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:38.448+0000 7f5db0109200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:17:39.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:39 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:39.017+0000 7f5db0109200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:17:39.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:39 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:39.150+0000 7f5db0109200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:17:39.427 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:39 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:39.424+0000 7f5db0109200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:17:40.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:40 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:40.092+0000 7f5db0109200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:17:40.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:40 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:40.219+0000 7f5db0109200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:17:40.864 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:40 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:40.549+0000 7f5db0109200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:17:40.864 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:40 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:40.732+0000 7f5db0109200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:17:40.864 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:40 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:40.862+0000 7f5db0109200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:17:41.176 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:41 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:41.016+0000 7f5db0109200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:17:41.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:41 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:41.390+0000 7f5db0109200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:17:42.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:42.172+0000 7f5db0109200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:17:42.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:17:42.463+0000 7f5db0109200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: Active manager daemon x restarted 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: Activating manager daemon x 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: mgrmap e6: x(active, starting, since 0.0106948s) 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T11:17:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:42 smithi033 ceph-mon[27845]: Manager daemon x is now available 2024-04-26T11:17:43.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: Found migration_current of "None". Setting to last migration. 2024-04-26T11:17:43.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:43.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:43.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:17:43.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:17:43.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:17:43.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:17:43.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: mgrmap e7: x(active, since 1.01924s) 2024-04-26T11:17:43.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:43.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:43 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.151+0000 7eff5ac00700 1 Processor -- start 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.151+0000 7eff5ac00700 1 -- start start 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.152+0000 7eff5ac00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff54108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.152+0000 7eff5ac00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7eff54108640 con 0x7eff54107c50 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.152+0000 7eff53e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff54108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.152+0000 7eff53e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff54108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52872/0 (socket says 172.21.15.33:52872) 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.152+0000 7eff53e00700 1 -- 172.21.15.33:0/4019773978 learned_addr learned my addr 172.21.15.33:0/4019773978 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:44.115 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.153+0000 7eff53e00700 1 -- 172.21.15.33:0/4019773978 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff54108e50 con 0x7eff54107c50 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.153+0000 7eff53e00700 1 --2- 172.21.15.33:0/4019773978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff54108070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff440098a0 tx=0x7eff44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fe15c534992583 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.153+0000 7eff52a00700 1 -- 172.21.15.33:0/4019773978 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff44004160 con 0x7eff54107c50 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.153+0000 7eff52a00700 1 -- 172.21.15.33:0/4019773978 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff44004300 con 0x7eff54107c50 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.153+0000 7eff52a00700 1 -- 172.21.15.33:0/4019773978 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff44011af0 con 0x7eff54107c50 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.154+0000 7eff5ac00700 1 -- 172.21.15.33:0/4019773978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 msgr2=0x7eff54108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.154+0000 7eff5ac00700 1 --2- 172.21.15.33:0/4019773978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff54108070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff440098a0 tx=0x7eff44009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.154+0000 7eff5ac00700 1 -- 172.21.15.33:0/4019773978 shutdown_connections 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.154+0000 7eff5ac00700 1 --2- 172.21.15.33:0/4019773978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff54108070 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.154+0000 7eff5ac00700 1 -- 172.21.15.33:0/4019773978 >> 172.21.15.33:0/4019773978 conn(0x7eff541031d0 msgr2=0x7eff54105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.155+0000 7eff5ac00700 1 -- 172.21.15.33:0/4019773978 shutdown_connections 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.155+0000 7eff5ac00700 1 -- 172.21.15.33:0/4019773978 wait complete. 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.156+0000 7eff5ac00700 1 Processor -- start 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.156+0000 7eff5ac00700 1 -- start start 2024-04-26T11:17:44.116 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.156+0000 7eff5ac00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff5419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.156+0000 7eff5ac00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7eff5410c200 con 0x7eff54107c50 2024-04-26T11:17:44.117 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.156+0000 7eff53e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff5419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.157+0000 7eff53e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff5419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52882/0 (socket says 172.21.15.33:52882) 2024-04-26T11:17:44.121 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.157+0000 7eff53e00700 1 -- 172.21.15.33:0/1092345807 learned_addr learned my addr 172.21.15.33:0/1092345807 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.157+0000 7eff53e00700 1 -- 172.21.15.33:0/1092345807 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5419d220 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.157+0000 7eff53e00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff5419cce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7eff44000c00 tx=0x7eff44011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.158+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff44003b20 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.158+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff4401a8c0 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.158+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff44024940 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.158+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff5419d4b0 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.158+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff541a0190 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.159+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7eff44022020 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.160+0000 7eff50c00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.160+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 --> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7eff3803c1d0 con 0x7eff38039600 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.160+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7eff4404d490 con 0x7eff54107c50 2024-04-26T11:17:44.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.160+0000 7eff53400700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 msgr2=0x7eff3803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.160+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.361+0000 7eff53400700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 msgr2=0x7eff3803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.361+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.762+0000 7eff53400700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 msgr2=0x7eff3803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:36.762+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:37.563+0000 7eff53400700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 msgr2=0x7eff3803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:37.563+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:39.165+0000 7eff53400700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 msgr2=0x7eff3803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:39.165+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:42.369+0000 7eff53400700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 msgr2=0x7eff3803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/927259861 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:42.369+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:42.473+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7eff4402d3f0 con 0x7eff54107c50 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:42.474+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 msgr2=0x7eff3803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:17:44.123 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:42.474+0000 7eff50c00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.477+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7eff4404ef50 con 0x7eff54107c50 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.478+0000 7eff50c00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7eff3803cef0 0x7eff3803f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.478+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7eff4402be50 con 0x7eff3803cef0 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.480+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7eff3803cef0 0x7eff3803f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.483+0000 7eff53400700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7eff3803cef0 0x7eff3803f2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7eff40003a10 tx=0x7eff400092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.485+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7eff4402be50 con 0x7eff3803cef0 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.494+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7eff5410c050 con 0x7eff3803cef0 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.495+0000 7eff50c00700 1 -- 172.21.15.33:0/1092345807 <== mgr.14120 v2:172.21.15.33:6800/334919779 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7eff5410c050 con 0x7eff3803cef0 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7eff3803cef0 msgr2=0x7eff3803f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7eff3803cef0 0x7eff3803f2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7eff40003a10 tx=0x7eff400092b0 comp rx=0 tx=0).stop 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 msgr2=0x7eff5419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff5419cce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7eff44000c00 tx=0x7eff44011b70 comp rx=0 tx=0).stop 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 shutdown_connections 2024-04-26T11:17:44.124 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7eff3803cef0 0x7eff3803f2e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7eff54107c50 0x7eff5419cce0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 --2- 172.21.15.33:0/1092345807 >> [v2:172.21.15.33:6800/927259861,v1:172.21.15.33:6801/927259861] conn(0x7eff38039600 0x7eff3803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 >> 172.21.15.33:0/1092345807 conn(0x7eff541031d0 msgr2=0x7eff54104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 shutdown_connections 2024-04-26T11:17:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:43.496+0000 7eff5ac00700 1 -- 172.21.15.33:0/1092345807 wait complete. 2024-04-26T11:17:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:mgr epoch 5 is available 2024-04-26T11:17:44.125 INFO:teuthology.orchestra.run.smithi033.stdout:Setting orchestrator backend to cephadm... 2024-04-26T11:17:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:44 smithi033 ceph-mon[27845]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T11:17:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:44 smithi033 ceph-mon[27845]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T11:17:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:44 smithi033 ceph-mon[27845]: [26/Apr/2024:11:17:44] ENGINE Bus STARTING 2024-04-26T11:17:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:44 smithi033 ceph-mon[27845]: [26/Apr/2024:11:17:44] ENGINE Serving on https://172.21.15.33:7150 2024-04-26T11:17:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:44 smithi033 ceph-mon[27845]: [26/Apr/2024:11:17:44] ENGINE Serving on http://172.21.15.33:8765 2024-04-26T11:17:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:44 smithi033 ceph-mon[27845]: [26/Apr/2024:11:17:44] ENGINE Bus STARTED 2024-04-26T11:17:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:44 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:17:45.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:45 smithi033 ceph-mon[27845]: mgrmap e8: x(active, since 2s) 2024-04-26T11:17:45.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:45 smithi033 ceph-mon[27845]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:17:45.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:45 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:45.626 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:45 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:17:45.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.724+0000 7f5874a00700 1 Processor -- start 2024-04-26T11:17:45.635 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.725+0000 7f5874a00700 1 -- start start 2024-04-26T11:17:45.638 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.725+0000 7f5874a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.725+0000 7f5874a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5870112620 con 0x7f587010fcf0 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.725+0000 7f586de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.725+0000 7f586de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701120e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38542/0 (socket says 172.21.15.33:38542) 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.726+0000 7f586de00700 1 -- 172.21.15.33:0/602765281 learned_addr learned my addr 172.21.15.33:0/602765281 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.726+0000 7f586de00700 1 -- 172.21.15.33:0/602765281 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58701127a0 con 0x7f587010fcf0 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.726+0000 7f586de00700 1 --2- 172.21.15.33:0/602765281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701120e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f58600098a0 tx=0x7f5860009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa95e243021e6c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.726+0000 7f586ca00700 1 -- 172.21.15.33:0/602765281 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5860004160 con 0x7f587010fcf0 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f586ca00700 1 -- 172.21.15.33:0/602765281 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5860004300 con 0x7f587010fcf0 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f586ca00700 1 -- 172.21.15.33:0/602765281 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5860011af0 con 0x7f587010fcf0 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f5874a00700 1 -- 172.21.15.33:0/602765281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 msgr2=0x7f58701120e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f5874a00700 1 --2- 172.21.15.33:0/602765281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701120e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f58600098a0 tx=0x7f5860009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f5874a00700 1 -- 172.21.15.33:0/602765281 shutdown_connections 2024-04-26T11:17:45.639 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f5874a00700 1 --2- 172.21.15.33:0/602765281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701120e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f5874a00700 1 -- 172.21.15.33:0/602765281 >> 172.21.15.33:0/602765281 conn(0x7f587007e300 msgr2=0x7f5870080760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f5874a00700 1 -- 172.21.15.33:0/602765281 shutdown_connections 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.727+0000 7f5874a00700 1 -- 172.21.15.33:0/602765281 wait complete. 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.728+0000 7f5874a00700 1 Processor -- start 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.729+0000 7f5874a00700 1 -- start start 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.729+0000 7f5874a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701a3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.729+0000 7f5874a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5870113700 con 0x7f587010fcf0 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.729+0000 7f586de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701a3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:45.640 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.729+0000 7f586de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701a3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38552/0 (socket says 172.21.15.33:38552) 2024-04-26T11:17:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.729+0000 7f586de00700 1 -- 172.21.15.33:0/1722134824 learned_addr learned my addr 172.21.15.33:0/1722134824 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.729+0000 7f586de00700 1 -- 172.21.15.33:0/1722134824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58701a3890 con 0x7f587010fcf0 2024-04-26T11:17:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.730+0000 7f586de00700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701a3350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5860011ac0 tx=0x7f5860011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.730+0000 7f5866a00700 1 -- 172.21.15.33:0/1722134824 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5860003970 con 0x7f587010fcf0 2024-04-26T11:17:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.730+0000 7f5866a00700 1 -- 172.21.15.33:0/1722134824 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5860003b10 con 0x7f587010fcf0 2024-04-26T11:17:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.730+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58701a3b20 con 0x7f587010fcf0 2024-04-26T11:17:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.730+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58701a67c0 con 0x7f587010fcf0 2024-04-26T11:17:45.643 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.731+0000 7f5866a00700 1 -- 172.21.15.33:0/1722134824 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5860024610 con 0x7f587010fcf0 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.732+0000 7f5866a00700 1 -- 172.21.15.33:0/1722134824 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5860024860 con 0x7f587010fcf0 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.732+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5850005360 con 0x7f587010fcf0 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.732+0000 7f5866a00700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5854039600 0x7f585403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.732+0000 7f5866a00700 1 -- 172.21.15.33:0/1722134824 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f586004d940 con 0x7f587010fcf0 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.732+0000 7f586d400700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5854039600 0x7f585403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.733+0000 7f586d400700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5854039600 0x7f585403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f585c006fd0 tx=0x7f585c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.738+0000 7f5866a00700 1 -- 172.21.15.33:0/1722134824 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58600172b0 con 0x7f587010fcf0 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.983+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f5850000bf0 con 0x7f5854039600 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.992+0000 7f5866a00700 1 -- 172.21.15.33:0/1722134824 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5850000bf0 con 0x7f5854039600 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5854039600 msgr2=0x7f585403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:45.644 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5854039600 0x7f585403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f585c006fd0 tx=0x7f585c006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 msgr2=0x7f58701a3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701a3350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5860011ac0 tx=0x7f5860011ba0 comp rx=0 tx=0).stop 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 shutdown_connections 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5854039600 0x7f585403bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 --2- 172.21.15.33:0/1722134824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f587010fcf0 0x7f58701a3350 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 >> 172.21.15.33:0/1722134824 conn(0x7f587007e300 msgr2=0x7f587007ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.995+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 shutdown_connections 2024-04-26T11:17:45.645 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:44.996+0000 7f5874a00700 1 -- 172.21.15.33:0/1722134824 wait complete. 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.241+0000 7f5c1ba00700 1 Processor -- start 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.241+0000 7f5c1ba00700 1 -- start start 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.242+0000 7f5c1ba00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1410b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.242+0000 7f5c1ba00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1410b8e0 con 0x7f5c1410af80 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.242+0000 7f5c19000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1410b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.242+0000 7f5c19000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1410b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38560/0 (socket says 172.21.15.33:38560) 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.242+0000 7f5c19000700 1 -- 172.21.15.33:0/4276846669 learned_addr learned my addr 172.21.15.33:0/4276846669 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.243+0000 7f5c19000700 1 -- 172.21.15.33:0/4276846669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1410c090 con 0x7f5c1410af80 2024-04-26T11:17:47.106 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.243+0000 7f5c19000700 1 --2- 172.21.15.33:0/4276846669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1410b3a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c080098a0 tx=0x7f5c08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae3f560ac775757 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.243+0000 7f5c13400700 1 -- 172.21.15.33:0/4276846669 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c08004160 con 0x7f5c1410af80 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.243+0000 7f5c13400700 1 -- 172.21.15.33:0/4276846669 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5c08004300 con 0x7f5c1410af80 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.243+0000 7f5c13400700 1 -- 172.21.15.33:0/4276846669 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c08011af0 con 0x7f5c1410af80 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.244+0000 7f5c1ba00700 1 -- 172.21.15.33:0/4276846669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 msgr2=0x7f5c1410b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.244+0000 7f5c1ba00700 1 --2- 172.21.15.33:0/4276846669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1410b3a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c080098a0 tx=0x7f5c08009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.244+0000 7f5c1ba00700 1 -- 172.21.15.33:0/4276846669 shutdown_connections 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.244+0000 7f5c1ba00700 1 --2- 172.21.15.33:0/4276846669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1410b3a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.244+0000 7f5c1ba00700 1 -- 172.21.15.33:0/4276846669 >> 172.21.15.33:0/4276846669 conn(0x7f5c1407e320 msgr2=0x7f5c1407e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.244+0000 7f5c1ba00700 1 -- 172.21.15.33:0/4276846669 shutdown_connections 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.244+0000 7f5c1ba00700 1 -- 172.21.15.33:0/4276846669 wait complete. 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.245+0000 7f5c1ba00700 1 Processor -- start 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.246+0000 7f5c1ba00700 1 -- start start 2024-04-26T11:17:47.107 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.246+0000 7f5c1ba00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.246+0000 7f5c1ba00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1410f4d0 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.246+0000 7f5c19000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.246+0000 7f5c19000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38568/0 (socket says 172.21.15.33:38568) 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.246+0000 7f5c19000700 1 -- 172.21.15.33:0/2092550581 learned_addr learned my addr 172.21.15.33:0/2092550581 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.247+0000 7f5c19000700 1 -- 172.21.15.33:0/2092550581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1419f3b0 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.247+0000 7f5c19000700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1419ee70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5c08011ac0 tx=0x7f5c08011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.247+0000 7f5c11600700 1 -- 172.21.15.33:0/2092550581 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c08003b20 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.247+0000 7f5c11600700 1 -- 172.21.15.33:0/2092550581 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5c0801a8c0 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.247+0000 7f5c11600700 1 -- 172.21.15.33:0/2092550581 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c080248d0 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.247+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1419f640 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.248+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c141a2330 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.249+0000 7f5c11600700 1 -- 172.21.15.33:0/2092550581 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5c08022020 con 0x7f5c1410af80 2024-04-26T11:17:47.108 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.249+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c1406a530 con 0x7f5c1410af80 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.249+0000 7f5c11600700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5bfc039600 0x7f5bfc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.249+0000 7f5c11600700 1 -- 172.21.15.33:0/2092550581 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5c0804d900 con 0x7f5c1410af80 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.249+0000 7f5c13e00700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5bfc039600 0x7f5bfc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.250+0000 7f5c13e00700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5bfc039600 0x7f5bfc03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5c04006fd0 tx=0x7f5c04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.256+0000 7f5c11600700 1 -- 172.21.15.33:0/2092550581 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c0801ab20 con 0x7f5c1410af80 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.491+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f5c14108b90 con 0x7f5bfc039600 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.492+0000 7f5c11600700 1 -- 172.21.15.33:0/2092550581 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f5c14108b90 con 0x7f5bfc039600 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.495+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5bfc039600 msgr2=0x7f5bfc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.495+0000 7f5c1ba00700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5bfc039600 0x7f5bfc03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5c04006fd0 tx=0x7f5c04006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.495+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 msgr2=0x7f5c1419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.495+0000 7f5c1ba00700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1419ee70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5c08011ac0 tx=0x7f5c08011ba0 comp rx=0 tx=0).stop 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.496+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 shutdown_connections 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.496+0000 7f5c1ba00700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5bfc039600 0x7f5bfc03bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:47.109 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.496+0000 7f5c1ba00700 1 --2- 172.21.15.33:0/2092550581 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c1410af80 0x7f5c1419ee70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:47.110 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.496+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 >> 172.21.15.33:0/2092550581 conn(0x7f5c1407e320 msgr2=0x7f5c14108480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:47.110 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.496+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 shutdown_connections 2024-04-26T11:17:47.110 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:46.496+0000 7f5c1ba00700 1 -- 172.21.15.33:0/2092550581 wait complete. 2024-04-26T11:17:47.110 INFO:teuthology.orchestra.run.smithi033.stdout:Generating ssh key... 2024-04-26T11:17:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:47 smithi033 ceph-mon[27845]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:17:48.885 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-mon[27845]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:17:48.885 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-mon[27845]: Generating ssh key... 2024-04-26T11:17:49.177 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: Generating public/private rsa key pair. 2024-04-26T11:17:49.178 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: Your identification has been saved in /tmp/tmp4sb3rehy/key. 2024-04-26T11:17:49.178 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: Your public key has been saved in /tmp/tmp4sb3rehy/key.pub. 2024-04-26T11:17:49.178 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: The key fingerprint is: 2024-04-26T11:17:49.178 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: SHA256:iJ9cRA4l1DyvZKyK0RubHJNBIyjs/pOine8yver3vSg ceph-5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:49.178 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: The key's randomart image is: 2024-04-26T11:17:49.179 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: +---[RSA 3072]----+ 2024-04-26T11:17:49.179 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: |.. .++o | 2024-04-26T11:17:49.179 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: |o.. o =+ | 2024-04-26T11:17:49.179 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: |o o . .oo | 2024-04-26T11:17:49.179 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: | . .. o+ . | 2024-04-26T11:17:49.179 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: |. ..o.+S. | 2024-04-26T11:17:49.180 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: | .. *o.o. | 2024-04-26T11:17:49.180 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: | .=.O+ | 2024-04-26T11:17:49.180 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: | o+=E o | 2024-04-26T11:17:49.180 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: |o.*B=+o o. | 2024-04-26T11:17:49.180 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:17:48 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: +----[SHA256]-----+ 2024-04-26T11:17:49.586 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.686+0000 7f5e0fa00700 1 Processor -- start 2024-04-26T11:17:49.587 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.687+0000 7f5e0fa00700 1 -- start start 2024-04-26T11:17:49.587 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.687+0000 7f5e0fa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e08109160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:49.587 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.687+0000 7f5e0fa00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e08109730 con 0x7f5e08108d40 2024-04-26T11:17:49.587 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.688+0000 7f5e0d000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e08109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:49.587 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.688+0000 7f5e0d000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e08109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38570/0 (socket says 172.21.15.33:38570) 2024-04-26T11:17:49.587 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.688+0000 7f5e0d000700 1 -- 172.21.15.33:0/664740103 learned_addr learned my addr 172.21.15.33:0/664740103 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:49.587 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.688+0000 7f5e0d000700 1 -- 172.21.15.33:0/664740103 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e08109ef0 con 0x7f5e08108d40 2024-04-26T11:17:49.591 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.689+0000 7f5e0d000700 1 --2- 172.21.15.33:0/664740103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e08109160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098a0 tx=0x7f5dfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7dde44c51ffe3c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:49.591 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.689+0000 7f5e07400700 1 -- 172.21.15.33:0/664740103 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc004160 con 0x7f5e08108d40 2024-04-26T11:17:49.591 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.689+0000 7f5e07400700 1 -- 172.21.15.33:0/664740103 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5dfc004300 con 0x7f5e08108d40 2024-04-26T11:17:49.591 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.689+0000 7f5e07400700 1 -- 172.21.15.33:0/664740103 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc011af0 con 0x7f5e08108d40 2024-04-26T11:17:49.591 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.690+0000 7f5e0fa00700 1 -- 172.21.15.33:0/664740103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 msgr2=0x7f5e08109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:49.591 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.690+0000 7f5e0fa00700 1 --2- 172.21.15.33:0/664740103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e08109160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098a0 tx=0x7f5dfc009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.690+0000 7f5e0fa00700 1 -- 172.21.15.33:0/664740103 shutdown_connections 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.690+0000 7f5e0fa00700 1 --2- 172.21.15.33:0/664740103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e08109160 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.690+0000 7f5e0fa00700 1 -- 172.21.15.33:0/664740103 >> 172.21.15.33:0/664740103 conn(0x7f5e08104300 msgr2=0x7f5e08106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.690+0000 7f5e0fa00700 1 -- 172.21.15.33:0/664740103 shutdown_connections 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.690+0000 7f5e0fa00700 1 -- 172.21.15.33:0/664740103 wait complete. 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.691+0000 7f5e0fa00700 1 Processor -- start 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.692+0000 7f5e0fa00700 1 -- start start 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.692+0000 7f5e0fa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e0819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.692+0000 7f5e0fa00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0810d2a0 con 0x7f5e08108d40 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.692+0000 7f5e0d000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e0819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.692+0000 7f5e0d000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e0819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38586/0 (socket says 172.21.15.33:38586) 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.692+0000 7f5e0d000700 1 -- 172.21.15.33:0/2415489743 learned_addr learned my addr 172.21.15.33:0/2415489743 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.693+0000 7f5e0d000700 1 -- 172.21.15.33:0/2415489743 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0819d1e0 con 0x7f5e08108d40 2024-04-26T11:17:49.592 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.693+0000 7f5e0d000700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e0819cca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc003f40 tx=0x7f5dfc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.693+0000 7f5e05600700 1 -- 172.21.15.33:0/2415489743 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc0038b0 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.694+0000 7f5e05600700 1 -- 172.21.15.33:0/2415489743 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5dfc003a50 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.694+0000 7f5e05600700 1 -- 172.21.15.33:0/2415489743 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc01a5c0 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.694+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0819d470 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.694+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e081a0160 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.695+0000 7f5e05600700 1 -- 172.21.15.33:0/2415489743 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5dfc024430 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.695+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e08064a80 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.696+0000 7f5e05600700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5df00395b0 0x7f5df003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.696+0000 7f5e05600700 1 -- 172.21.15.33:0/2415489743 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5dfc04db50 con 0x7f5e08108d40 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.696+0000 7f5e07e00700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5df00395b0 0x7f5df003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:49.593 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.697+0000 7f5e07e00700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5df00395b0 0x7f5df003ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5df4006fd0 tx=0x7f5df4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:49.594 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.701+0000 7f5e05600700 1 -- 172.21.15.33:0/2415489743 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dfc02cc00 con 0x7f5e08108d40 2024-04-26T11:17:49.594 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:47.934+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f5e081055c0 con 0x7f5df00395b0 2024-04-26T11:17:49.594 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.896+0000 7f5e05600700 1 -- 172.21.15.33:0/2415489743 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5e081055c0 con 0x7f5df00395b0 2024-04-26T11:17:49.594 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5df00395b0 msgr2=0x7f5df003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5df00395b0 0x7f5df003ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5df4006fd0 tx=0x7f5df4006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 msgr2=0x7f5e0819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e0819cca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc003f40 tx=0x7f5dfc011890 comp rx=0 tx=0).stop 2024-04-26T11:17:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 shutdown_connections 2024-04-26T11:17:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f5df00395b0 0x7f5df003ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 --2- 172.21.15.33:0/2415489743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e08108d40 0x7f5e0819cca0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 >> 172.21.15.33:0/2415489743 conn(0x7f5e08104300 msgr2=0x7f5e08104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:49.596 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 shutdown_connections 2024-04-26T11:17:49.596 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:48.899+0000 7f5e0fa00700 1 -- 172.21.15.33:0/2415489743 wait complete. 2024-04-26T11:17:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:49 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:49 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:51.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweyW5HCAjZMykYNqx62yDDB1HRB+Ki/g0trwk11T6XN+wG9P0ZbNqJQXWSp1uVzjtFayvRJ4axyYSJqRcqur2dpjv/8+ACAPWrDIkSaFB3hKakn5xPUVcs9GZ7qMZer+WEPfcKYQ/+z68XOwdkFFuvYe07IPnNUNVM2/LsoA1b/HAZ4g7pE3lk57ICRaPyNFA9Qw3TmUQGWjU3pZYjmt0j0bDbqp0PdYzjFb4f+35Jfcbb1uSQ1cpnGCzuq3QEIaxjA7AlqkXV4DpHkvUIztv3Yi/oR2Eqb/Sb/SaDVsCLA44inepIv/KTKcpqi3GvBa8GqARuBa8RJUub0lLYpjbtTnmgci1C/NkIKrSCM7UL41bKE8ZWdD0+7Yr/MsRNqwg5GdnPqunyix6ZSGNt3kLV5wLJzEfRwE++n2cEJod3GjyUMWNvgnlUYbQQCdADpdPG+n/1L8nC+XgdaSItmN0hIV7U/s0d4bYhoMCN6DjJ5TdmuM6rVRkbRsEhN52V0c= ceph-5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:17:51.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.234+0000 7f3e2ac00700 1 Processor -- start 2024-04-26T11:17:51.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.235+0000 7f3e2ac00700 1 -- start start 2024-04-26T11:17:51.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.235+0000 7f3e2ac00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2410b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.235+0000 7f3e2ac00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410b8c0 con 0x7f3e2410af60 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.235+0000 7f3e23e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2410b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.235+0000 7f3e23e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2410b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50714/0 (socket says 172.21.15.33:50714) 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.235+0000 7f3e23e00700 1 -- 172.21.15.33:0/73564204 learned_addr learned my addr 172.21.15.33:0/73564204 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.236+0000 7f3e23e00700 1 -- 172.21.15.33:0/73564204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2410c070 con 0x7f3e2410af60 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.236+0000 7f3e23e00700 1 --2- 172.21.15.33:0/73564204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2410b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e140098a0 tx=0x7f3e14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e82f6e33655ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.237+0000 7f3e22a00700 1 -- 172.21.15.33:0/73564204 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e14004160 con 0x7f3e2410af60 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.237+0000 7f3e22a00700 1 -- 172.21.15.33:0/73564204 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3e14004300 con 0x7f3e2410af60 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.237+0000 7f3e22a00700 1 -- 172.21.15.33:0/73564204 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e14011af0 con 0x7f3e2410af60 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.237+0000 7f3e2ac00700 1 -- 172.21.15.33:0/73564204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 msgr2=0x7f3e2410b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.237+0000 7f3e2ac00700 1 --2- 172.21.15.33:0/73564204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2410b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e140098a0 tx=0x7f3e14009bb0 comp rx=0 tx=0).stop 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.237+0000 7f3e2ac00700 1 -- 172.21.15.33:0/73564204 shutdown_connections 2024-04-26T11:17:51.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.237+0000 7f3e2ac00700 1 --2- 172.21.15.33:0/73564204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2410b380 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.238+0000 7f3e2ac00700 1 -- 172.21.15.33:0/73564204 >> 172.21.15.33:0/73564204 conn(0x7f3e2407e350 msgr2=0x7f3e2407e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.238+0000 7f3e2ac00700 1 -- 172.21.15.33:0/73564204 shutdown_connections 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.238+0000 7f3e2ac00700 1 -- 172.21.15.33:0/73564204 wait complete. 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.239+0000 7f3e2ac00700 1 Processor -- start 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.239+0000 7f3e2ac00700 1 -- start start 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.240+0000 7f3e2ac00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.240+0000 7f3e2ac00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410f4b0 con 0x7f3e2410af60 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.240+0000 7f3e23e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.240+0000 7f3e23e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50726/0 (socket says 172.21.15.33:50726) 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.240+0000 7f3e23e00700 1 -- 172.21.15.33:0/3289535260 learned_addr learned my addr 172.21.15.33:0/3289535260 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.240+0000 7f3e23e00700 1 -- 172.21.15.33:0/3289535260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2419f3a0 con 0x7f3e2410af60 2024-04-26T11:17:51.190 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.241+0000 7f3e23e00700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2419ee60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e14011ac0 tx=0x7f3e14011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.241+0000 7f3e20c00700 1 -- 172.21.15.33:0/3289535260 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e14003b20 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.241+0000 7f3e20c00700 1 -- 172.21.15.33:0/3289535260 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3e1401a8c0 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.241+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e2419f630 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.241+0000 7f3e20c00700 1 -- 172.21.15.33:0/3289535260 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e140248d0 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.241+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241a2320 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.242+0000 7f3e20c00700 1 -- 172.21.15.33:0/3289535260 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3e14022020 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.243+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e24064a80 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.243+0000 7f3e20c00700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f3e08039600 0x7f3e0803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.243+0000 7f3e20c00700 1 -- 172.21.15.33:0/3289535260 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3e1404d900 con 0x7f3e2410af60 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.243+0000 7f3e23400700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f3e08039600 0x7f3e0803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.244+0000 7f3e23400700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f3e08039600 0x7f3e0803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3e10006fd0 tx=0x7f3e10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:51.191 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.248+0000 7f3e20c00700 1 -- 172.21.15.33:0/3289535260 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e1400f2e0 con 0x7f3e2410af60 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.480+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f3e24108b70 con 0x7f3e08039600 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.481+0000 7f3e20c00700 1 -- 172.21.15.33:0/3289535260 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f3e24108b70 con 0x7f3e08039600 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f3e08039600 msgr2=0x7f3e0803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f3e08039600 0x7f3e0803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3e10006fd0 tx=0x7f3e10006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 msgr2=0x7f3e2419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2419ee60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e14011ac0 tx=0x7f3e14011ba0 comp rx=0 tx=0).stop 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 shutdown_connections 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f3e08039600 0x7f3e0803bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 --2- 172.21.15.33:0/3289535260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e2410af60 0x7f3e2419ee60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 >> 172.21.15.33:0/3289535260 conn(0x7f3e2407e350 msgr2=0x7f3e24108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:51.192 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 shutdown_connections 2024-04-26T11:17:51.193 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:50.485+0000 7f3e2ac00700 1 -- 172.21.15.33:0/3289535260 wait complete. 2024-04-26T11:17:51.193 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T11:17:51.193 INFO:teuthology.orchestra.run.smithi033.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T11:17:51.198 INFO:teuthology.orchestra.run.smithi033.stdout:Adding host smithi033... 2024-04-26T11:17:51.796 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:51 smithi033 ceph-mon[27845]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:17:52.733 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:52 smithi033 ceph-mon[27845]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "addr": "172.21.15.33", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:17:53.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:53 smithi033 ceph-mon[27845]: Deploying cephadm binary to smithi033 2024-04-26T11:17:59.338 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Added host 'smithi033' with addr '172.21.15.33' 2024-04-26T11:17:59.338 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.792+0000 7f5300c00700 1 Processor -- start 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.793+0000 7f5300c00700 1 -- start start 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.793+0000 7f5300c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc109240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.793+0000 7f5300c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc109810 con 0x7f52fc108e20 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.794+0000 7f52f9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.794+0000 7f52f9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50742/0 (socket says 172.21.15.33:50742) 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.794+0000 7f52f9e00700 1 -- 172.21.15.33:0/1267492990 learned_addr learned my addr 172.21.15.33:0/1267492990 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.794+0000 7f52f9e00700 1 -- 172.21.15.33:0/1267492990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc109f80 con 0x7f52fc108e20 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.795+0000 7f52f9e00700 1 --2- 172.21.15.33:0/1267492990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc109240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009e80 tx=0x7f52ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639efce6a760bc8f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.795+0000 7f52f8a00700 1 -- 172.21.15.33:0/1267492990 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52ec017070 con 0x7f52fc108e20 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.795+0000 7f52f8a00700 1 -- 172.21.15.33:0/1267492990 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f52ec00b8c0 con 0x7f52fc108e20 2024-04-26T11:17:59.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.795+0000 7f52f8a00700 1 -- 172.21.15.33:0/1267492990 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52ec004490 con 0x7f52fc108e20 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.796+0000 7f5300c00700 1 -- 172.21.15.33:0/1267492990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 msgr2=0x7f52fc109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.796+0000 7f5300c00700 1 --2- 172.21.15.33:0/1267492990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc109240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009e80 tx=0x7f52ec00b330 comp rx=0 tx=0).stop 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.796+0000 7f5300c00700 1 -- 172.21.15.33:0/1267492990 shutdown_connections 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.796+0000 7f5300c00700 1 --2- 172.21.15.33:0/1267492990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc109240 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.796+0000 7f5300c00700 1 -- 172.21.15.33:0/1267492990 >> 172.21.15.33:0/1267492990 conn(0x7f52fc104400 msgr2=0x7f52fc106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.796+0000 7f5300c00700 1 -- 172.21.15.33:0/1267492990 shutdown_connections 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.796+0000 7f5300c00700 1 -- 172.21.15.33:0/1267492990 wait complete. 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.797+0000 7f5300c00700 1 Processor -- start 2024-04-26T11:17:59.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.797+0000 7f5300c00700 1 -- start start 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.798+0000 7f5300c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc07eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.798+0000 7f5300c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc10d330 con 0x7f52fc108e20 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.798+0000 7f52f9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc07eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.798+0000 7f52f9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc07eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50748/0 (socket says 172.21.15.33:50748) 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.798+0000 7f52f9e00700 1 -- 172.21.15.33:0/3243605513 learned_addr learned my addr 172.21.15.33:0/3243605513 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.798+0000 7f52f9e00700 1 -- 172.21.15.33:0/3243605513 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc07f0d0 con 0x7f52fc108e20 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.799+0000 7f52f9e00700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc07eb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00b7b0 tx=0x7f52ec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.799+0000 7f52f2a00700 1 -- 172.21.15.33:0/3243605513 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52ec020020 con 0x7f52fc108e20 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.799+0000 7f52f2a00700 1 -- 172.21.15.33:0/3243605513 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f52ec012430 con 0x7f52fc108e20 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.799+0000 7f52f2a00700 1 -- 172.21.15.33:0/3243605513 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52ec017040 con 0x7f52fc108e20 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.799+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc07d2c0 con 0x7f52fc108e20 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.799+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc07d800 con 0x7f52fc108e20 2024-04-26T11:17:59.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.800+0000 7f52f2a00700 1 -- 172.21.15.33:0/3243605513 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f52ec023020 con 0x7f52fc108e20 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.800+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc064a80 con 0x7f52fc108e20 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.801+0000 7f52f2a00700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f52e00395b0 0x7f52e003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.801+0000 7f52f2a00700 1 -- 172.21.15.33:0/3243605513 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f52ec00f050 con 0x7f52fc108e20 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.801+0000 7f52f9400700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f52e00395b0 0x7f52e003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.802+0000 7f52f9400700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f52e00395b0 0x7f52e003ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006fd0 tx=0x7f52e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:51.807+0000 7f52f2a00700 1 -- 172.21.15.33:0/3243605513 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f52ec02a020 con 0x7f52fc108e20 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:52.042+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi033", "addr": "172.21.15.33", "target": ["mon-mgr", ""]}) -- 0x7f52fc105600 con 0x7f52e00395b0 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.593+0000 7f52f2a00700 1 -- 172.21.15.33:0/3243605513 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f52fc105600 con 0x7f52e00395b0 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f52e00395b0 msgr2=0x7f52e003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f52e00395b0 0x7f52e003ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006fd0 tx=0x7f52e8006e40 comp rx=0 tx=0).stop 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 msgr2=0x7f52fc07eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc07eb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00b7b0 tx=0x7f52ec00be20 comp rx=0 tx=0).stop 2024-04-26T11:17:59.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 shutdown_connections 2024-04-26T11:17:59.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f52e00395b0 0x7f52e003ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:59.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 --2- 172.21.15.33:0/3243605513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52fc108e20 0x7f52fc07eb90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:17:59.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.597+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 >> 172.21.15.33:0/3243605513 conn(0x7f52fc104400 msgr2=0x7f52fc104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:17:59.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.598+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 shutdown_connections 2024-04-26T11:17:59.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:17:58.598+0000 7f5300c00700 1 -- 172.21.15.33:0/3243605513 wait complete. 2024-04-26T11:17:59.347 INFO:teuthology.orchestra.run.smithi033.stdout:Deploying unmanaged mon service... 2024-04-26T11:17:59.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:59 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:17:59.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:59 smithi033 ceph-mon[27845]: Added host smithi033 2024-04-26T11:17:59.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:17:59 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:18:01.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:01 smithi033 ceph-mon[27845]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:18:01.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:01 smithi033 ceph-mon[27845]: Saving service mon spec with placement count:5 2024-04-26T11:18:01.564 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:01 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.001+0000 7f8917200700 1 Processor -- start 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.001+0000 7f8917200700 1 -- start start 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.002+0000 7f8917200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f891807fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.002+0000 7f8917200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8918080270 con 0x7f891807f880 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.002+0000 7f8915e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f891807fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.002+0000 7f8915e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f891807fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41256/0 (socket says 172.21.15.33:41256) 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.002+0000 7f8915e00700 1 -- 172.21.15.33:0/1957692205 learned_addr learned my addr 172.21.15.33:0/1957692205 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:01.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.003+0000 7f8915e00700 1 -- 172.21.15.33:0/1957692205 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8918080ad0 con 0x7f891807f880 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.003+0000 7f8915e00700 1 --2- 172.21.15.33:0/1957692205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f891807fca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89080098a0 tx=0x7f8908009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ade14218880ac0c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.003+0000 7f8914a00700 1 -- 172.21.15.33:0/1957692205 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8908004160 con 0x7f891807f880 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.003+0000 7f8914a00700 1 -- 172.21.15.33:0/1957692205 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8908004300 con 0x7f891807f880 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.003+0000 7f8914a00700 1 -- 172.21.15.33:0/1957692205 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8908011af0 con 0x7f891807f880 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.004+0000 7f8917200700 1 -- 172.21.15.33:0/1957692205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 msgr2=0x7f891807fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.004+0000 7f8917200700 1 --2- 172.21.15.33:0/1957692205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f891807fca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89080098a0 tx=0x7f8908009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.004+0000 7f8917200700 1 -- 172.21.15.33:0/1957692205 shutdown_connections 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.004+0000 7f8917200700 1 --2- 172.21.15.33:0/1957692205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f891807fca0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.004+0000 7f8917200700 1 -- 172.21.15.33:0/1957692205 >> 172.21.15.33:0/1957692205 conn(0x7f891807e210 msgr2=0x7f891807e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.004+0000 7f8917200700 1 -- 172.21.15.33:0/1957692205 shutdown_connections 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.004+0000 7f8917200700 1 -- 172.21.15.33:0/1957692205 wait complete. 2024-04-26T11:18:01.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.005+0000 7f8917200700 1 Processor -- start 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.006+0000 7f8917200700 1 -- start start 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.006+0000 7f8917200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f89181a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.006+0000 7f8917200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8918081e90 con 0x7f891807f880 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.006+0000 7f8915e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f89181a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.006+0000 7f8915e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f89181a3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41258/0 (socket says 172.21.15.33:41258) 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.006+0000 7f8915e00700 1 -- 172.21.15.33:0/13038449 learned_addr learned my addr 172.21.15.33:0/13038449 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.007+0000 7f8915e00700 1 -- 172.21.15.33:0/13038449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89181a3750 con 0x7f891807f880 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.007+0000 7f8915e00700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f89181a3210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8908011ac0 tx=0x7f8908011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.007+0000 7f890ea00700 1 -- 172.21.15.33:0/13038449 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8908003970 con 0x7f891807f880 2024-04-26T11:18:01.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.007+0000 7f890ea00700 1 -- 172.21.15.33:0/13038449 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8908003b10 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.007+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89181a39e0 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.007+0000 7f890ea00700 1 -- 172.21.15.33:0/13038449 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8908024610 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.008+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89181a66d0 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.009+0000 7f890ea00700 1 -- 172.21.15.33:0/13038449 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f890801a9e0 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.009+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f891806a530 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.009+0000 7f890ea00700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88fc039600 0x7f88fc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.009+0000 7f890ea00700 1 -- 172.21.15.33:0/13038449 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f890804d980 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.009+0000 7f8915400700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88fc039600 0x7f88fc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.010+0000 7f8915400700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88fc039600 0x7f88fc03bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8904006fd0 tx=0x7f8904006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.015+0000 7f890ea00700 1 -- 172.21.15.33:0/13038449 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f890804d2d0 con 0x7f891807f880 2024-04-26T11:18:01.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.276+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f891802e740 con 0x7f88fc039600 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.286+0000 7f890ea00700 1 -- 172.21.15.33:0/13038449 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f891802e740 con 0x7f88fc039600 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88fc039600 msgr2=0x7f88fc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88fc039600 0x7f88fc03bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8904006fd0 tx=0x7f8904006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 msgr2=0x7f89181a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f89181a3210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8908011ac0 tx=0x7f8908011ba0 comp rx=0 tx=0).stop 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 shutdown_connections 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88fc039600 0x7f88fc03bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 --2- 172.21.15.33:0/13038449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f891807f880 0x7f89181a3210 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 >> 172.21.15.33:0/13038449 conn(0x7f891807e210 msgr2=0x7f8918199e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 shutdown_connections 2024-04-26T11:18:01.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:00.289+0000 7f8917200700 1 -- 172.21.15.33:0/13038449 wait complete. 2024-04-26T11:18:01.963 INFO:teuthology.orchestra.run.smithi033.stdout:Deploying unmanaged mgr service... 2024-04-26T11:18:02.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:02 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.533+0000 7f6eba600700 1 Processor -- start 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.533+0000 7f6eba600700 1 -- start start 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.533+0000 7f6eba600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb407d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.533+0000 7f6eba600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb407d8a0 con 0x7f6eb407ee60 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.534+0000 7f6eb3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb407d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.534+0000 7f6eb3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb407d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41260/0 (socket says 172.21.15.33:41260) 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.534+0000 7f6eb3400700 1 -- 172.21.15.33:0/2513187881 learned_addr learned my addr 172.21.15.33:0/2513187881 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:03.478 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.534+0000 7f6eb3400700 1 -- 172.21.15.33:0/2513187881 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb407da20 con 0x7f6eb407ee60 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.535+0000 7f6eb3400700 1 --2- 172.21.15.33:0/2513187881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb407d360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40098a0 tx=0x7f6ea4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83bef106cab5670 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.535+0000 7f6eb2000700 1 -- 172.21.15.33:0/2513187881 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea4004160 con 0x7f6eb407ee60 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.535+0000 7f6eb2000700 1 -- 172.21.15.33:0/2513187881 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea4004300 con 0x7f6eb407ee60 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.535+0000 7f6eb2000700 1 -- 172.21.15.33:0/2513187881 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea4011a20 con 0x7f6eb407ee60 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.536+0000 7f6eba600700 1 -- 172.21.15.33:0/2513187881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 msgr2=0x7f6eb407d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.536+0000 7f6eba600700 1 --2- 172.21.15.33:0/2513187881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb407d360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40098a0 tx=0x7f6ea4009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.536+0000 7f6eba600700 1 -- 172.21.15.33:0/2513187881 shutdown_connections 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.536+0000 7f6eba600700 1 --2- 172.21.15.33:0/2513187881 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb407d360 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.536+0000 7f6eba600700 1 -- 172.21.15.33:0/2513187881 >> 172.21.15.33:0/2513187881 conn(0x7f6eb41043c0 msgr2=0x7f6eb41067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.536+0000 7f6eba600700 1 -- 172.21.15.33:0/2513187881 shutdown_connections 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.536+0000 7f6eba600700 1 -- 172.21.15.33:0/2513187881 wait complete. 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.537+0000 7f6eba600700 1 Processor -- start 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.537+0000 7f6eba600700 1 -- start start 2024-04-26T11:18:03.479 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.538+0000 7f6eba600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.538+0000 7f6eba600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410f300 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.538+0000 7f6eb3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.538+0000 7f6eb3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41276/0 (socket says 172.21.15.33:41276) 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.538+0000 7f6eb3400700 1 -- 172.21.15.33:0/2776356763 learned_addr learned my addr 172.21.15.33:0/2776356763 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.539+0000 7f6eb3400700 1 -- 172.21.15.33:0/2776356763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb419f3a0 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.539+0000 7f6eb3400700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb419ee60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4004660 tx=0x7f6ea4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.539+0000 7f6eabe00700 1 -- 172.21.15.33:0/2776356763 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea4003aa0 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.539+0000 7f6eabe00700 1 -- 172.21.15.33:0/2776356763 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea401b8f0 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.539+0000 7f6eabe00700 1 -- 172.21.15.33:0/2776356763 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea4023900 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.539+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb419f630 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.540+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb41a2320 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.541+0000 7f6eabe00700 1 -- 172.21.15.33:0/2776356763 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6ea401baa0 con 0x7f6eb407ee60 2024-04-26T11:18:03.480 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.541+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb4064a80 con 0x7f6eb407ee60 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.541+0000 7f6eabe00700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f6e98039600 0x7f6e9803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.541+0000 7f6eabe00700 1 -- 172.21.15.33:0/2776356763 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6ea404ddf0 con 0x7f6eb407ee60 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.542+0000 7f6eb2a00700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f6e98039600 0x7f6e9803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.543+0000 7f6eb2a00700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f6e98039600 0x7f6e9803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0006fd0 tx=0x7f6ea0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.548+0000 7f6eabe00700 1 -- 172.21.15.33:0/2776356763 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ea401f030 con 0x7f6eb407ee60 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.797+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6eb41055c0 con 0x7f6e98039600 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.803+0000 7f6eabe00700 1 -- 172.21.15.33:0/2776356763 <== mgr.14120 v2:172.21.15.33:6800/334919779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6eb41055c0 con 0x7f6e98039600 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f6e98039600 msgr2=0x7f6e9803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f6e98039600 0x7f6e9803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0006fd0 tx=0x7f6ea0006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 msgr2=0x7f6eb419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb419ee60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4004660 tx=0x7f6ea4011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 shutdown_connections 2024-04-26T11:18:03.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f6e98039600 0x7f6e9803bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:03.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 --2- 172.21.15.33:0/2776356763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6eb407ee60 0x7f6eb419ee60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:03.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 >> 172.21.15.33:0/2776356763 conn(0x7f6eb41043c0 msgr2=0x7f6eb4104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:03.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 shutdown_connections 2024-04-26T11:18:03.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:02.806+0000 7f6eba600700 1 -- 172.21.15.33:0/2776356763 wait complete. 2024-04-26T11:18:03.767 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:03 smithi033 ceph-mon[27845]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:03.767 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:03 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:18:03.767 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:03 smithi033 ceph-mon[27845]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:18:03.767 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:03 smithi033 ceph-mon[27845]: Saving service mgr spec with placement count:2 2024-04-26T11:18:03.767 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:03 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:18:05.396 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:05 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/141726062' entity='client.admin' 2024-04-26T11:18:05.397 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:05 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:18:05.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.071+0000 7f59c5400700 1 Processor -- start 2024-04-26T11:18:05.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.071+0000 7f59c5400700 1 -- start start 2024-04-26T11:18:05.414 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.072+0000 7f59c5400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c010bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:05.414 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.072+0000 7f59c5400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f59c007ced0 con 0x7f59c0109780 2024-04-26T11:18:05.414 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.072+0000 7f59be800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c010bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:05.414 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.072+0000 7f59be800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c010bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41286/0 (socket says 172.21.15.33:41286) 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.072+0000 7f59be800700 1 -- 172.21.15.33:0/4077430666 learned_addr learned my addr 172.21.15.33:0/4077430666 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.073+0000 7f59be800700 1 -- 172.21.15.33:0/4077430666 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c010c0b0 con 0x7f59c0109780 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.073+0000 7f59be800700 1 --2- 172.21.15.33:0/4077430666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c010bb70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f59b00098a0 tx=0x7f59b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce4f6eba48b9a84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.073+0000 7f59bd400700 1 -- 172.21.15.33:0/4077430666 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59b0004160 con 0x7f59c0109780 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.074+0000 7f59bd400700 1 -- 172.21.15.33:0/4077430666 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b0004300 con 0x7f59c0109780 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.074+0000 7f59bd400700 1 -- 172.21.15.33:0/4077430666 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59b0011a20 con 0x7f59c0109780 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.074+0000 7f59c5400700 1 -- 172.21.15.33:0/4077430666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 msgr2=0x7f59c010bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.074+0000 7f59c5400700 1 --2- 172.21.15.33:0/4077430666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c010bb70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f59b00098a0 tx=0x7f59b0009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.075+0000 7f59c5400700 1 -- 172.21.15.33:0/4077430666 shutdown_connections 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.075+0000 7f59c5400700 1 --2- 172.21.15.33:0/4077430666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c010bb70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.075+0000 7f59c5400700 1 -- 172.21.15.33:0/4077430666 >> 172.21.15.33:0/4077430666 conn(0x7f59c0103210 msgr2=0x7f59c0105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.075+0000 7f59c5400700 1 -- 172.21.15.33:0/4077430666 shutdown_connections 2024-04-26T11:18:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.075+0000 7f59c5400700 1 -- 172.21.15.33:0/4077430666 wait complete. 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.076+0000 7f59c5400700 1 Processor -- start 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.076+0000 7f59c5400700 1 -- start start 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.076+0000 7f59c5400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.076+0000 7f59c5400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010d0c0 con 0x7f59c0109780 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.077+0000 7f59be800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.077+0000 7f59be800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c019cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41288/0 (socket says 172.21.15.33:41288) 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.077+0000 7f59be800700 1 -- 172.21.15.33:0/141726062 learned_addr learned my addr 172.21.15.33:0/141726062 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.077+0000 7f59be800700 1 -- 172.21.15.33:0/141726062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c019d160 con 0x7f59c0109780 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.078+0000 7f59be800700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c019cc20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f59b0000c00 tx=0x7f59b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.078+0000 7f59b7400700 1 -- 172.21.15.33:0/141726062 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59b0003aa0 con 0x7f59c0109780 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.078+0000 7f59b7400700 1 -- 172.21.15.33:0/141726062 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b001b8f0 con 0x7f59c0109780 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.078+0000 7f59c5400700 1 -- 172.21.15.33:0/141726062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c019d3f0 con 0x7f59c0109780 2024-04-26T11:18:05.416 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.078+0000 7f59b7400700 1 -- 172.21.15.33:0/141726062 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59b0023870 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.078+0000 7f59c5400700 1 -- 172.21.15.33:0/141726062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c01a00e0 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.079+0000 7f59b7400700 1 -- 172.21.15.33:0/141726062 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f59b0023b70 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.079+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5998005330 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.080+0000 7f59b7400700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f59a40395b0 0x7f59a403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.080+0000 7f59b7400700 1 -- 172.21.15.33:0/141726062 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f59b004df60 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.080+0000 7f59bde00700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f59a40395b0 0x7f59a403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.081+0000 7f59bde00700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f59a40395b0 0x7f59a403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006fd0 tx=0x7f59ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.086+0000 7f59b7400700 1 -- 172.21.15.33:0/141726062 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59b001f030 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.332+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f5998005620 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.337+0000 7f59b7400700 1 -- 172.21.15.33:0/141726062 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f59b004d8b0 con 0x7f59c0109780 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.340+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f59a40395b0 msgr2=0x7f59a403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.340+0000 7f59b4c00700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f59a40395b0 0x7f59a403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006fd0 tx=0x7f59ac006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:05.417 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.340+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 msgr2=0x7f59c019cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.340+0000 7f59b4c00700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c019cc20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f59b0000c00 tx=0x7f59b0011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.341+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 shutdown_connections 2024-04-26T11:18:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.341+0000 7f59b4c00700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f59a40395b0 0x7f59a403ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.341+0000 7f59b4c00700 1 --2- 172.21.15.33:0/141726062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59c0109780 0x7f59c019cc20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.341+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 >> 172.21.15.33:0/141726062 conn(0x7f59c0103210 msgr2=0x7f59c0103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.341+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 shutdown_connections 2024-04-26T11:18:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:04.341+0000 7f59b4c00700 1 -- 172.21.15.33:0/141726062 wait complete. 2024-04-26T11:18:06.596 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:06 smithi033 ceph-mon[27845]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:06.596 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:06 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1258408071' entity='client.admin' 2024-04-26T11:18:06.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.972+0000 7fa20c800700 1 Processor -- start 2024-04-26T11:18:06.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.973+0000 7fa20c800700 1 -- start start 2024-04-26T11:18:06.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.973+0000 7fa20c800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa2041091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:06.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.973+0000 7fa20c800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa2041097b0 con 0x7fa204108dc0 2024-04-26T11:18:06.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.973+0000 7fa209e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa2041091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.973+0000 7fa209e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa2041091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41292/0 (socket says 172.21.15.33:41292) 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.973+0000 7fa209e00700 1 -- 172.21.15.33:0/3637462500 learned_addr learned my addr 172.21.15.33:0/3637462500 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.974+0000 7fa209e00700 1 -- 172.21.15.33:0/3637462500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa204109f20 con 0x7fa204108dc0 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.974+0000 7fa209e00700 1 --2- 172.21.15.33:0/3637462500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa2041091e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80098a0 tx=0x7fa1f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55abbdc225c07e8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.974+0000 7fa208a00700 1 -- 172.21.15.33:0/3637462500 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f8004160 con 0x7fa204108dc0 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.974+0000 7fa208a00700 1 -- 172.21.15.33:0/3637462500 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1f8004300 con 0x7fa204108dc0 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.974+0000 7fa208a00700 1 -- 172.21.15.33:0/3637462500 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f8011a20 con 0x7fa204108dc0 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.975+0000 7fa20c800700 1 -- 172.21.15.33:0/3637462500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 msgr2=0x7fa2041091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.975+0000 7fa20c800700 1 --2- 172.21.15.33:0/3637462500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa2041091e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80098a0 tx=0x7fa1f8009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.975+0000 7fa20c800700 1 -- 172.21.15.33:0/3637462500 shutdown_connections 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.975+0000 7fa20c800700 1 --2- 172.21.15.33:0/3637462500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa2041091e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.975+0000 7fa20c800700 1 -- 172.21.15.33:0/3637462500 >> 172.21.15.33:0/3637462500 conn(0x7fa204104360 msgr2=0x7fa2041067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.975+0000 7fa20c800700 1 -- 172.21.15.33:0/3637462500 shutdown_connections 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.975+0000 7fa20c800700 1 -- 172.21.15.33:0/3637462500 wait complete. 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.976+0000 7fa20c800700 1 Processor -- start 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.976+0000 7fa20c800700 1 -- start start 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.977+0000 7fa20c800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa20419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.977+0000 7fa20c800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa20410d2d0 con 0x7fa204108dc0 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.977+0000 7fa209e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa20419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.977+0000 7fa209e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa20419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41300/0 (socket says 172.21.15.33:41300) 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.977+0000 7fa209e00700 1 -- 172.21.15.33:0/1258408071 learned_addr learned my addr 172.21.15.33:0/1258408071 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.977+0000 7fa209e00700 1 -- 172.21.15.33:0/1258408071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20419f440 con 0x7fa204108dc0 2024-04-26T11:18:06.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.977+0000 7fa209e00700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa20419ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8000c00 tx=0x7fa1f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.978+0000 7fa1fea00700 1 -- 172.21.15.33:0/1258408071 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f8003aa0 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.978+0000 7fa1fea00700 1 -- 172.21.15.33:0/1258408071 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1f801b8f0 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.978+0000 7fa1fea00700 1 -- 172.21.15.33:0/1258408071 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1f8023890 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.978+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20419f6d0 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.978+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2041a23c0 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.979+0000 7fa1fea00700 1 -- 172.21.15.33:0/1258408071 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa1f801b440 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.979+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa204064a80 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.980+0000 7fa1fea00700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7fa1ec039600 0x7fa1ec03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.980+0000 7fa1fea00700 1 -- 172.21.15.33:0/1258408071 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa1f804ddf0 con 0x7fa204108dc0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.980+0000 7fa209400700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7fa1ec039600 0x7fa1ec03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:06.962 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.981+0000 7fa209400700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7fa1ec039600 0x7fa1ec03bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4006fd0 tx=0x7fa1f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:05.985+0000 7fa1fea00700 1 -- 172.21.15.33:0/1258408071 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1f801f030 con 0x7fa204108dc0 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.220+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fa2041a2a20 con 0x7fa204108dc0 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.225+0000 7fa1fea00700 1 -- 172.21.15.33:0/1258408071 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fa1f801bda0 con 0x7fa204108dc0 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7fa1ec039600 msgr2=0x7fa1ec03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7fa1ec039600 0x7fa1ec03bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4006fd0 tx=0x7fa1f4006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 msgr2=0x7fa20419ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa20419ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8000c00 tx=0x7fa1f8011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 shutdown_connections 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7fa1ec039600 0x7fa1ec03bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 --2- 172.21.15.33:0/1258408071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa204108dc0 0x7fa20419ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.229+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 >> 172.21.15.33:0/1258408071 conn(0x7fa204104360 msgr2=0x7fa204104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:06.963 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.230+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 shutdown_connections 2024-04-26T11:18:06.964 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:06.230+0000 7fa20c800700 1 -- 172.21.15.33:0/1258408071 wait complete. 2024-04-26T11:18:06.964 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling the dashboard module... 2024-04-26T11:18:08.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:08 smithi033 ceph-mon[27845]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:08.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:08 smithi033 ceph-mon[27845]: from='mgr.14120 172.21.15.33:0/3030394715' entity='mgr.x' 2024-04-26T11:18:08.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:08 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/306392676' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T11:18:08.763 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:08 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: ignoring --setuser ceph since I am not root 2024-04-26T11:18:08.763 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:08 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: ignoring --setgroup ceph since I am not root 2024-04-26T11:18:08.764 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:08 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:08.661+0000 7f701266f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:18:09.071 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:08 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:08.924+0000 7f701266f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:18:09.071 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:09 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:09.068+0000 7f701266f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:18:09.183 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.459+0000 7f88a1e00700 1 Processor -- start 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.459+0000 7f88a1e00700 1 -- start start 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.459+0000 7f88a1e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c07d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.459+0000 7f88a1e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f889c07d8a0 con 0x7f889c07ee60 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.460+0000 7f889aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c07d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.460+0000 7f889aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c07d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41306/0 (socket says 172.21.15.33:41306) 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.460+0000 7f889aa00700 1 -- 172.21.15.33:0/2296361111 learned_addr learned my addr 172.21.15.33:0/2296361111 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.460+0000 7f889aa00700 1 -- 172.21.15.33:0/2296361111 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c07da20 con 0x7f889c07ee60 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.461+0000 7f889aa00700 1 --2- 172.21.15.33:0/2296361111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c07d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f888c0098a0 tx=0x7f888c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378a5f81dcdfddd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.461+0000 7f8899600700 1 -- 172.21.15.33:0/2296361111 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f888c004160 con 0x7f889c07ee60 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.461+0000 7f8899600700 1 -- 172.21.15.33:0/2296361111 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f888c004300 con 0x7f889c07ee60 2024-04-26T11:18:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.461+0000 7f8899600700 1 -- 172.21.15.33:0/2296361111 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f888c011a20 con 0x7f889c07ee60 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.462+0000 7f88a1e00700 1 -- 172.21.15.33:0/2296361111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 msgr2=0x7f889c07d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.462+0000 7f88a1e00700 1 --2- 172.21.15.33:0/2296361111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c07d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f888c0098a0 tx=0x7f888c009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.462+0000 7f88a1e00700 1 -- 172.21.15.33:0/2296361111 shutdown_connections 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.462+0000 7f88a1e00700 1 --2- 172.21.15.33:0/2296361111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c07d360 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.462+0000 7f88a1e00700 1 -- 172.21.15.33:0/2296361111 >> 172.21.15.33:0/2296361111 conn(0x7f889c1043c0 msgr2=0x7f889c1067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.462+0000 7f88a1e00700 1 -- 172.21.15.33:0/2296361111 shutdown_connections 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.462+0000 7f88a1e00700 1 -- 172.21.15.33:0/2296361111 wait complete. 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.464+0000 7f88a1e00700 1 Processor -- start 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.464+0000 7f88a1e00700 1 -- start start 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.464+0000 7f88a1e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c1a3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.464+0000 7f88a1e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f889c10aef0 con 0x7f889c07ee60 2024-04-26T11:18:09.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.464+0000 7f889aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c1a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.464+0000 7f889aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c1a3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41314/0 (socket says 172.21.15.33:41314) 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.464+0000 7f889aa00700 1 -- 172.21.15.33:0/306392676 learned_addr learned my addr 172.21.15.33:0/306392676 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.465+0000 7f889aa00700 1 -- 172.21.15.33:0/306392676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c1a37b0 con 0x7f889c07ee60 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.465+0000 7f889aa00700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c1a3270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f888c0116e0 tx=0x7f888c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.465+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f888c003b20 con 0x7f889c07ee60 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.466+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f888c01b8f0 con 0x7f889c07ee60 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.466+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f888c0238c0 con 0x7f889c07ee60 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.466+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889c1a3a40 con 0x7f889c07ee60 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.466+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889c1a6730 con 0x7f889c07ee60 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.467+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f889c064a80 con 0x7f889c07ee60 2024-04-26T11:18:09.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.468+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f888c01baa0 con 0x7f889c07ee60 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.468+0000 7f88a0a00700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88800395b0 0x7f888003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.468+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f888c04e0c0 con 0x7f889c07ee60 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.469+0000 7f889a000700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88800395b0 0x7f888003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.469+0000 7f889a000700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88800395b0 0x7f888003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.475+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f888c01f030 con 0x7f889c07ee60 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:07.769+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f889c02e7d0 con 0x7f889c07ee60 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.350+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f888c00be50 con 0x7f889c07ee60 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.353+0000 7f88a0a00700 1 -- 172.21.15.33:0/306392676 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f888c023a60 con 0x7f889c07ee60 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88800395b0 msgr2=0x7f888003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88800395b0 0x7f888003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:09.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 msgr2=0x7f889c1a3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:09.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c1a3270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f888c0116e0 tx=0x7f888c011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:09.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 shutdown_connections 2024-04-26T11:18:09.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f88800395b0 0x7f888003ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:09.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 --2- 172.21.15.33:0/306392676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f889c07ee60 0x7f889c1a3270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:09.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 >> 172.21.15.33:0/306392676 conn(0x7f889c1043c0 msgr2=0x7f889c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:09.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.356+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 shutdown_connections 2024-04-26T11:18:09.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:08.357+0000 7f88a1e00700 1 -- 172.21.15.33:0/306392676 wait complete. 2024-04-26T11:18:09.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:09 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/306392676' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T11:18:09.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:09 smithi033 ceph-mon[27845]: mgrmap e9: x(active, since 25s) 2024-04-26T11:18:09.629 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:09 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:09.509+0000 7f701266f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:18:09.629 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:09 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:09.625+0000 7f701266f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:18:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:10 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2146793713' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T11:18:11.505 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:11 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:11.273+0000 7f701266f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T11:18:11.526 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T11:18:11.527 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T11:18:11.527 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T11:18:11.527 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.381+0000 7f1901800700 1 Processor -- start 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.381+0000 7f1901800700 1 -- start start 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.381+0000 7f1901800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc07d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.381+0000 7f1901800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc07d840 con 0x7f18fc07ee00 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.382+0000 7f18faa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc07d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.382+0000 7f18faa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc07d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39944/0 (socket says 172.21.15.33:39944) 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.382+0000 7f18faa00700 1 -- 172.21.15.33:0/905195138 learned_addr learned my addr 172.21.15.33:0/905195138 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:11.528 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.382+0000 7f18faa00700 1 -- 172.21.15.33:0/905195138 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18fc07d9c0 con 0x7f18fc07ee00 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.383+0000 7f18faa00700 1 --2- 172.21.15.33:0/905195138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc07d300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0098a0 tx=0x7f18ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6675d8cfefeeb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.383+0000 7f18f9600700 1 -- 172.21.15.33:0/905195138 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec004160 con 0x7f18fc07ee00 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.383+0000 7f18f9600700 1 -- 172.21.15.33:0/905195138 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18ec004300 con 0x7f18fc07ee00 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.383+0000 7f18f9600700 1 -- 172.21.15.33:0/905195138 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec011a20 con 0x7f18fc07ee00 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.383+0000 7f1901800700 1 -- 172.21.15.33:0/905195138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 msgr2=0x7f18fc07d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.383+0000 7f1901800700 1 --2- 172.21.15.33:0/905195138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc07d300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0098a0 tx=0x7f18ec009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.384+0000 7f1901800700 1 -- 172.21.15.33:0/905195138 shutdown_connections 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.384+0000 7f1901800700 1 --2- 172.21.15.33:0/905195138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc07d300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:11.531 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.384+0000 7f1901800700 1 -- 172.21.15.33:0/905195138 >> 172.21.15.33:0/905195138 conn(0x7f18fc104310 msgr2=0x7f18fc106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.384+0000 7f1901800700 1 -- 172.21.15.33:0/905195138 shutdown_connections 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.384+0000 7f1901800700 1 -- 172.21.15.33:0/905195138 wait complete. 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.385+0000 7f1901800700 1 Processor -- start 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.385+0000 7f1901800700 1 -- start start 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.385+0000 7f1901800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.385+0000 7f1901800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc10f270 con 0x7f18fc07ee00 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.385+0000 7f18faa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.386+0000 7f18faa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39950/0 (socket says 172.21.15.33:39950) 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.386+0000 7f18faa00700 1 -- 172.21.15.33:0/2146793713 learned_addr learned my addr 172.21.15.33:0/2146793713 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.386+0000 7f18faa00700 1 -- 172.21.15.33:0/2146793713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18fc19f310 con 0x7f18fc07ee00 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.386+0000 7f18faa00700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc19edd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18ec000c00 tx=0x7f18ec011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.386+0000 7f18f3400700 1 -- 172.21.15.33:0/2146793713 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec003aa0 con 0x7f18fc07ee00 2024-04-26T11:18:11.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.386+0000 7f18f3400700 1 -- 172.21.15.33:0/2146793713 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18ec01b8f0 con 0x7f18fc07ee00 2024-04-26T11:18:11.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.386+0000 7f18f3400700 1 -- 172.21.15.33:0/2146793713 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18ec023870 con 0x7f18fc07ee00 2024-04-26T11:18:11.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.387+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18fc19f5a0 con 0x7f18fc07ee00 2024-04-26T11:18:11.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.387+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18fc1a2290 con 0x7f18fc07ee00 2024-04-26T11:18:11.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.388+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18fc064a80 con 0x7f18fc07ee00 2024-04-26T11:18:11.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.388+0000 7f18f3400700 1 -- 172.21.15.33:0/2146793713 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f18ec023b70 con 0x7f18fc07ee00 2024-04-26T11:18:11.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.389+0000 7f18f3400700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 0x7f18e003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:11.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.389+0000 7f18f3400700 1 -- 172.21.15.33:0/2146793713 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f18ec04dfb0 con 0x7f18fc07ee00 2024-04-26T11:18:11.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.389+0000 7f18fa000700 1 -- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 msgr2=0x7f18e003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/334919779 2024-04-26T11:18:11.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.389+0000 7f18fa000700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 0x7f18e003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.395+0000 7f18f3400700 1 -- 172.21.15.33:0/2146793713 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18ec01f030 con 0x7f18fc07ee00 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.590+0000 7f18fa000700 1 -- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 msgr2=0x7f18e003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/334919779 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.590+0000 7f18fa000700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 0x7f18e003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.684+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f18fc1a2460 con 0x7f18fc07ee00 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.685+0000 7f18f3400700 1 -- 172.21.15.33:0/2146793713 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f18ec01a3d0 con 0x7f18fc07ee00 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 msgr2=0x7f18e003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 0x7f18e003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 msgr2=0x7f18fc19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc19edd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18ec000c00 tx=0x7f18ec011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 shutdown_connections 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f18e0039600 0x7f18e003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:11.537 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 --2- 172.21.15.33:0/2146793713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18fc07ee00 0x7f18fc19edd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:11.538 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 >> 172.21.15.33:0/2146793713 conn(0x7f18fc104310 msgr2=0x7f18fc104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:11.538 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 shutdown_connections 2024-04-26T11:18:11.538 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:10.688+0000 7f1901800700 1 -- 172.21.15.33:0/2146793713 wait complete. 2024-04-26T11:18:11.538 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for the mgr to restart... 2024-04-26T11:18:11.538 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr epoch 9... 2024-04-26T11:18:11.810 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:11 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:11.509+0000 7f701266f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:18:11.811 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:11 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:11.739+0000 7f701266f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:18:12.415 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:12 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:12.141+0000 7f701266f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:18:12.415 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:12 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:12.269+0000 7f701266f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:18:12.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:12 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:12.529+0000 7f701266f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:18:13.176 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:12 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:12.791+0000 7f701266f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:18:13.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:13 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:13.609+0000 7f701266f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:18:14.587 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:14 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:14.178+0000 7f701266f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:18:14.588 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:14 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:14.311+0000 7f701266f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:18:14.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:14 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:14.586+0000 7f701266f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:18:15.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:15.255+0000 7f701266f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:18:15.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:15.383+0000 7f701266f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:18:16.033 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:15.716+0000 7f701266f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:18:16.033 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:15.900+0000 7f701266f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:18:16.033 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:16.032+0000 7f701266f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:18:16.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:16.187+0000 7f701266f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:18:16.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:16.563+0000 7f701266f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:18:17.659 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:17.358+0000 7f701266f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:18:17.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:18:17.658+0000 7f701266f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:18:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: Active manager daemon x restarted 2024-04-26T11:18:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: Activating manager daemon x 2024-04-26T11:18:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T11:18:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: mgrmap e10: x(active, starting, since 0.00686696s) 2024-04-26T11:18:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:18:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T11:18:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T11:18:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T11:18:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T11:18:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:17 smithi033 ceph-mon[27845]: Manager daemon x is now available 2024-04-26T11:18:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:18:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:18:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:18:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: [26/Apr/2024:11:18:18] ENGINE Bus STARTING 2024-04-26T11:18:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: [26/Apr/2024:11:18:18] ENGINE Serving on https://172.21.15.33:7150 2024-04-26T11:18:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: [26/Apr/2024:11:18:18] ENGINE Serving on http://172.21.15.33:8765 2024-04-26T11:18:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: [26/Apr/2024:11:18:18] ENGINE Bus STARTED 2024-04-26T11:18:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:18 smithi033 ceph-mon[27845]: mgrmap e11: x(active, since 1.00991s) 2024-04-26T11:18:19.481 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.204+0000 7f7b56400700 1 Processor -- start 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.204+0000 7f7b56400700 1 -- start start 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.204+0000 7f7b56400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5010b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.204+0000 7f7b56400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010b9b0 con 0x7f7b5010afc0 2024-04-26T11:18:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.205+0000 7f7b4f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5010b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.205+0000 7f7b4f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5010b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39962/0 (socket says 172.21.15.33:39962) 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.205+0000 7f7b4f400700 1 -- 172.21.15.33:0/3994009167 learned_addr learned my addr 172.21.15.33:0/3994009167 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.205+0000 7f7b4f400700 1 -- 172.21.15.33:0/3994009167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5010c170 con 0x7f7b5010afc0 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.206+0000 7f7b4f400700 1 --2- 172.21.15.33:0/3994009167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5010b3e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b400098a0 tx=0x7f7b40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c648732663e116dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.206+0000 7f7b4e000700 1 -- 172.21.15.33:0/3994009167 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b40004160 con 0x7f7b5010afc0 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.206+0000 7f7b4e000700 1 -- 172.21.15.33:0/3994009167 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40004300 con 0x7f7b5010afc0 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.206+0000 7f7b4e000700 1 -- 172.21.15.33:0/3994009167 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b40011a20 con 0x7f7b5010afc0 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.207+0000 7f7b56400700 1 -- 172.21.15.33:0/3994009167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 msgr2=0x7f7b5010b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.207+0000 7f7b56400700 1 --2- 172.21.15.33:0/3994009167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5010b3e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b400098a0 tx=0x7f7b40009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.207+0000 7f7b56400700 1 -- 172.21.15.33:0/3994009167 shutdown_connections 2024-04-26T11:18:19.485 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.207+0000 7f7b56400700 1 --2- 172.21.15.33:0/3994009167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5010b3e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.207+0000 7f7b56400700 1 -- 172.21.15.33:0/3994009167 >> 172.21.15.33:0/3994009167 conn(0x7f7b5007e3b0 msgr2=0x7f7b5007e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.207+0000 7f7b56400700 1 -- 172.21.15.33:0/3994009167 shutdown_connections 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.207+0000 7f7b56400700 1 -- 172.21.15.33:0/3994009167 wait complete. 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.208+0000 7f7b56400700 1 Processor -- start 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.208+0000 7f7b56400700 1 -- start start 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.209+0000 7f7b56400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.209+0000 7f7b56400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010f520 con 0x7f7b5010afc0 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.209+0000 7f7b4f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.209+0000 7f7b4f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5019ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39976/0 (socket says 172.21.15.33:39976) 2024-04-26T11:18:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.209+0000 7f7b4f400700 1 -- 172.21.15.33:0/888386951 learned_addr learned my addr 172.21.15.33:0/888386951 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.210+0000 7f7b4f400700 1 -- 172.21.15.33:0/888386951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5019f370 con 0x7f7b5010afc0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.210+0000 7f7b4f400700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5019ee30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009520 tx=0x7f7b40011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.210+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b400039e0 con 0x7f7b5010afc0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.210+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b4001b8f0 con 0x7f7b5010afc0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.210+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b40019770 con 0x7f7b5010afc0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.210+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5019f600 con 0x7f7b5010afc0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.210+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b501a22f0 con 0x7f7b5010afc0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.211+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f7b4001b440 con 0x7f7b5010afc0 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.212+0000 7f7b47e00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.212+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 --> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7b5006a530 con 0x7f7b34039650 2024-04-26T11:18:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.212+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7b4004dee0 con 0x7f7b5010afc0 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.212+0000 7f7b4ea00700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 msgr2=0x7f7b3403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/334919779 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.212+0000 7f7b4ea00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.413+0000 7f7b4ea00700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 msgr2=0x7f7b3403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/334919779 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.413+0000 7f7b4ea00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.814+0000 7f7b4ea00700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 msgr2=0x7f7b3403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/334919779 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:12.814+0000 7f7b4ea00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:13.615+0000 7f7b4ea00700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 msgr2=0x7f7b3403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/334919779 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:13.615+0000 7f7b4ea00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:15.217+0000 7f7b4ea00700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 msgr2=0x7f7b3403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/334919779 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:15.217+0000 7f7b4ea00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:17.665+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f7b4001a3d0 con 0x7f7b5010afc0 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:17.665+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 msgr2=0x7f7b3403bb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:17.665+0000 7f7b47e00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:19.492 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.667+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f7b4004fcb0 con 0x7f7b5010afc0 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.667+0000 7f7b47e00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7b3403cee0 0x7f7b3403f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.667+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7b40022ec0 con 0x7f7b3403cee0 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.669+0000 7f7b4ea00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7b3403cee0 0x7f7b3403f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.671+0000 7f7b4ea00700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7b3403cee0 0x7f7b3403f2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c003a10 tx=0x7f7b3c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.672+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f7b40022ec0 con 0x7f7b3403cee0 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.682+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f7b50108b70 con 0x7f7b3403cee0 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.683+0000 7f7b47e00700 1 -- 172.21.15.33:0/888386951 <== mgr.14152 v2:172.21.15.33:6800/657098496 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f7b50108b70 con 0x7f7b3403cee0 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.683+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7b3403cee0 msgr2=0x7f7b3403f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.683+0000 7f7b56400700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7b3403cee0 0x7f7b3403f2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c003a10 tx=0x7f7b3c009270 comp rx=0 tx=0).stop 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.683+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 msgr2=0x7f7b5019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.683+0000 7f7b56400700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5019ee30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009520 tx=0x7f7b40011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.684+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 shutdown_connections 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.684+0000 7f7b56400700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7b3403cee0 0x7f7b3403f2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.684+0000 7f7b56400700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b5010afc0 0x7f7b5019ee30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:19.493 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.684+0000 7f7b56400700 1 --2- 172.21.15.33:0/888386951 >> [v2:172.21.15.33:6800/334919779,v1:172.21.15.33:6801/334919779] conn(0x7f7b34039650 0x7f7b3403bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:19.494 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.684+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 >> 172.21.15.33:0/888386951 conn(0x7f7b5007e3b0 msgr2=0x7f7b50108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:19.494 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.684+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 shutdown_connections 2024-04-26T11:18:19.494 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:18.684+0000 7f7b56400700 1 -- 172.21.15.33:0/888386951 wait complete. 2024-04-26T11:18:19.494 INFO:teuthology.orchestra.run.smithi033.stdout:mgr epoch 9 is available 2024-04-26T11:18:19.494 INFO:teuthology.orchestra.run.smithi033.stdout:Generating a dashboard self-signed certificate... 2024-04-26T11:18:19.982 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:19 smithi033 ceph-mon[27845]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T11:18:19.982 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:19 smithi033 ceph-mon[27845]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T11:18:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:20 smithi033 ceph-mon[27845]: mgrmap e12: x(active, since 2s) 2024-04-26T11:18:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.487+0000 7f5048000700 1 Processor -- start 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.487+0000 7f5048000700 1 -- start start 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.488+0000 7f5048000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f5040109160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.488+0000 7f5048000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5040109730 con 0x7f5040108d40 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.489+0000 7f5045600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f5040109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.489+0000 7f5045600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f5040109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42348/0 (socket says 172.21.15.33:42348) 2024-04-26T11:18:21.539 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.489+0000 7f5045600700 1 -- 172.21.15.33:0/1644381685 learned_addr learned my addr 172.21.15.33:0/1644381685 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:21.544 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.489+0000 7f5045600700 1 -- 172.21.15.33:0/1644381685 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5040109f40 con 0x7f5040108d40 2024-04-26T11:18:21.544 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.489+0000 7f5045600700 1 --2- 172.21.15.33:0/1644381685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f5040109160 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f50340098a0 tx=0x7f5034009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883c9cf365d8502d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:21.544 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.490+0000 7f503fe00700 1 -- 172.21.15.33:0/1644381685 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5034004160 con 0x7f5040108d40 2024-04-26T11:18:21.544 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.490+0000 7f503fe00700 1 -- 172.21.15.33:0/1644381685 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5034004300 con 0x7f5040108d40 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.490+0000 7f503fe00700 1 -- 172.21.15.33:0/1644381685 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5034011a20 con 0x7f5040108d40 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.491+0000 7f5048000700 1 -- 172.21.15.33:0/1644381685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 msgr2=0x7f5040109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.491+0000 7f5048000700 1 --2- 172.21.15.33:0/1644381685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f5040109160 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f50340098a0 tx=0x7f5034009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.491+0000 7f5048000700 1 -- 172.21.15.33:0/1644381685 shutdown_connections 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.491+0000 7f5048000700 1 --2- 172.21.15.33:0/1644381685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f5040109160 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.491+0000 7f5048000700 1 -- 172.21.15.33:0/1644381685 >> 172.21.15.33:0/1644381685 conn(0x7f5040104300 msgr2=0x7f5040106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.491+0000 7f5048000700 1 -- 172.21.15.33:0/1644381685 shutdown_connections 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.491+0000 7f5048000700 1 -- 172.21.15.33:0/1644381685 wait complete. 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.492+0000 7f5048000700 1 Processor -- start 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.492+0000 7f5048000700 1 -- start start 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.493+0000 7f5048000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f504019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.493+0000 7f5048000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f504010d2f0 con 0x7f5040108d40 2024-04-26T11:18:21.545 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.493+0000 7f5045600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f504019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.493+0000 7f5045600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f504019eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42364/0 (socket says 172.21.15.33:42364) 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.493+0000 7f5045600700 1 -- 172.21.15.33:0/3520199599 learned_addr learned my addr 172.21.15.33:0/3520199599 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.493+0000 7f5045600700 1 -- 172.21.15.33:0/3520199599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504019f410 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.494+0000 7f5045600700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f504019eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5034000c00 tx=0x7f5034011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.494+0000 7f503e000700 1 -- 172.21.15.33:0/3520199599 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5034003aa0 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.494+0000 7f503e000700 1 -- 172.21.15.33:0/3520199599 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f503401b8f0 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.494+0000 7f503e000700 1 -- 172.21.15.33:0/3520199599 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5034023890 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.494+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504019f6a0 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.494+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50401a2390 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.496+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5040064a80 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.496+0000 7f503e000700 1 -- 172.21.15.33:0/3520199599 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f503401b440 con 0x7f5040108d40 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.496+0000 7f503e000700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5028039660 0x7f502803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:21.546 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.496+0000 7f503e000700 1 -- 172.21.15.33:0/3520199599 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f503404df30 con 0x7f5040108d40 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.497+0000 7f5044c00700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5028039660 0x7f502803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.498+0000 7f5044c00700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5028039660 0x7f502803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f502c006fd0 tx=0x7f502c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.502+0000 7f503e000700 1 -- 172.21.15.33:0/3520199599 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f503401f030 con 0x7f5040108d40 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.751+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f50401054b0 con 0x7f5028039660 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.864+0000 7f503e000700 1 -- 172.21.15.33:0/3520199599 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f50401054b0 con 0x7f5028039660 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5028039660 msgr2=0x7f502803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5028039660 0x7f502803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f502c006fd0 tx=0x7f502c006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 msgr2=0x7f504019eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f504019eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5034000c00 tx=0x7f5034011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:21.547 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 shutdown_connections 2024-04-26T11:18:21.548 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5028039660 0x7f502803bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:21.548 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 --2- 172.21.15.33:0/3520199599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5040108d40 0x7f504019eed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:21.548 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.866+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 >> 172.21.15.33:0/3520199599 conn(0x7f5040104300 msgr2=0x7f5040104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:21.548 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.867+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 shutdown_connections 2024-04-26T11:18:21.548 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:20.867+0000 7f5048000700 1 -- 172.21.15.33:0/3520199599 wait complete. 2024-04-26T11:18:21.549 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial admin user... 2024-04-26T11:18:22.040 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:21 smithi033 ceph-mon[27845]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:18:22.040 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:21 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:22.040 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:21 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:22 smithi033 ceph-mon[27845]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:18:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:22 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:22 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:23.338 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$UmAnIkpkdndM89AaHwVSkODS9vEghD8GvftjaMHty/klQIUyIJxRi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714130302, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.129+0000 7f5d13200700 1 Processor -- start 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.129+0000 7f5d13200700 1 -- start start 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.129+0000 7f5d13200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c107990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.129+0000 7f5d13200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c07ced0 con 0x7f5d0c107570 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.130+0000 7f5d0be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.130+0000 7f5d0be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42372/0 (socket says 172.21.15.33:42372) 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.130+0000 7f5d0be00700 1 -- 172.21.15.33:0/2869802202 learned_addr learned my addr 172.21.15.33:0/2869802202 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.130+0000 7f5d0be00700 1 -- 172.21.15.33:0/2869802202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c107ed0 con 0x7f5d0c107570 2024-04-26T11:18:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.131+0000 7f5d0be00700 1 --2- 172.21.15.33:0/2869802202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c107990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0098a0 tx=0x7f5cfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f117cc76df413e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:23.343 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.131+0000 7f5d0aa00700 1 -- 172.21.15.33:0/2869802202 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cfc004160 con 0x7f5d0c107570 2024-04-26T11:18:23.343 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.131+0000 7f5d0aa00700 1 -- 172.21.15.33:0/2869802202 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cfc004300 con 0x7f5d0c107570 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.131+0000 7f5d0aa00700 1 -- 172.21.15.33:0/2869802202 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cfc011a20 con 0x7f5d0c107570 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.132+0000 7f5d13200700 1 -- 172.21.15.33:0/2869802202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 msgr2=0x7f5d0c107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.132+0000 7f5d13200700 1 --2- 172.21.15.33:0/2869802202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c107990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0098a0 tx=0x7f5cfc009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.132+0000 7f5d13200700 1 -- 172.21.15.33:0/2869802202 shutdown_connections 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.132+0000 7f5d13200700 1 --2- 172.21.15.33:0/2869802202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c107990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.132+0000 7f5d13200700 1 -- 172.21.15.33:0/2869802202 >> 172.21.15.33:0/2869802202 conn(0x7f5d0c103210 msgr2=0x7f5d0c105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.132+0000 7f5d13200700 1 -- 172.21.15.33:0/2869802202 shutdown_connections 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.132+0000 7f5d13200700 1 -- 172.21.15.33:0/2869802202 wait complete. 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.134+0000 7f5d13200700 1 Processor -- start 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.134+0000 7f5d13200700 1 -- start start 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.135+0000 7f5d13200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.135+0000 7f5d13200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c108f40 con 0x7f5d0c107570 2024-04-26T11:18:23.344 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.135+0000 7f5d0be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.135+0000 7f5d0be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c19ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42380/0 (socket says 172.21.15.33:42380) 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.135+0000 7f5d0be00700 1 -- 172.21.15.33:0/453999223 learned_addr learned my addr 172.21.15.33:0/453999223 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.135+0000 7f5d0be00700 1 -- 172.21.15.33:0/453999223 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c19f360 con 0x7f5d0c107570 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.136+0000 7f5d0be00700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c19ee20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc000c00 tx=0x7f5cfc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.136+0000 7f5d08c00700 1 -- 172.21.15.33:0/453999223 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cfc003aa0 con 0x7f5d0c107570 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.136+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d0c19f5f0 con 0x7f5d0c107570 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.136+0000 7f5d08c00700 1 -- 172.21.15.33:0/453999223 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cfc01b8f0 con 0x7f5d0c107570 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.136+0000 7f5d08c00700 1 -- 172.21.15.33:0/453999223 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cfc023870 con 0x7f5d0c107570 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.136+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0c1a22e0 con 0x7f5d0c107570 2024-04-26T11:18:23.345 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.137+0000 7f5d08c00700 1 -- 172.21.15.33:0/453999223 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5cfc023b70 con 0x7f5d0c107570 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.137+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d0c064a80 con 0x7f5d0c107570 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.138+0000 7f5d08c00700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cf0039600 0x7f5cf003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.138+0000 7f5d08c00700 1 -- 172.21.15.33:0/453999223 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5cfc04e080 con 0x7f5d0c107570 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.138+0000 7f5d0b400700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cf0039600 0x7f5cf003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.139+0000 7f5d0b400700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cf0039600 0x7f5cf003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8006fd0 tx=0x7f5cf8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.144+0000 7f5d08c00700 1 -- 172.21.15.33:0/453999223 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5cfc01f030 con 0x7f5d0c107570 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.392+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f5d0c198940 con 0x7f5cf0039600 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.651+0000 7f5d08c00700 1 -- 172.21.15.33:0/453999223 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f5d0c198940 con 0x7f5cf0039600 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cf0039600 msgr2=0x7f5cf003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cf0039600 0x7f5cf003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8006fd0 tx=0x7f5cf8006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 msgr2=0x7f5d0c19ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c19ee20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc000c00 tx=0x7f5cfc011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:23.346 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 shutdown_connections 2024-04-26T11:18:23.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cf0039600 0x7f5cf003bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:23.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 --2- 172.21.15.33:0/453999223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d0c107570 0x7f5d0c19ee20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:23.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 >> 172.21.15.33:0/453999223 conn(0x7f5d0c103210 msgr2=0x7f5d0c195a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:23.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 shutdown_connections 2024-04-26T11:18:23.347 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:22.654+0000 7f5d13200700 1 -- 172.21.15.33:0/453999223 wait complete. 2024-04-26T11:18:23.347 INFO:teuthology.orchestra.run.smithi033.stdout:Fetching dashboard port number... 2024-04-26T11:18:24.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:24 smithi033 ceph-mon[27845]: mgrmap e13: x(active, since 6s) 2024-04-26T11:18:24.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:24 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3980741299' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T11:18:25.564 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T11:18:25.567 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.911+0000 7f4dec800700 1 Processor -- start 2024-04-26T11:18:25.567 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.911+0000 7f4dec800700 1 -- start start 2024-04-26T11:18:25.567 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.911+0000 7f4dec800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de410b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:25.567 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.911+0000 7f4dec800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4de410b8e0 con 0x7f4de410af80 2024-04-26T11:18:25.567 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.912+0000 7f4de9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de410b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:25.567 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.912+0000 7f4de9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de410b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42392/0 (socket says 172.21.15.33:42392) 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.912+0000 7f4de9e00700 1 -- 172.21.15.33:0/2628843785 learned_addr learned my addr 172.21.15.33:0/2628843785 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.913+0000 7f4de9e00700 1 -- 172.21.15.33:0/2628843785 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de410c090 con 0x7f4de410af80 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.913+0000 7f4de9e00700 1 --2- 172.21.15.33:0/2628843785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de410b3a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80098a0 tx=0x7f4dd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fba06845d547cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.913+0000 7f4de8a00700 1 -- 172.21.15.33:0/2628843785 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd8004160 con 0x7f4de410af80 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.914+0000 7f4de8a00700 1 -- 172.21.15.33:0/2628843785 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dd8004300 con 0x7f4de410af80 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.914+0000 7f4de8a00700 1 -- 172.21.15.33:0/2628843785 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd8011a20 con 0x7f4de410af80 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.914+0000 7f4dec800700 1 -- 172.21.15.33:0/2628843785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 msgr2=0x7f4de410b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.914+0000 7f4dec800700 1 --2- 172.21.15.33:0/2628843785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de410b3a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80098a0 tx=0x7f4dd8009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.915+0000 7f4dec800700 1 -- 172.21.15.33:0/2628843785 shutdown_connections 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.915+0000 7f4dec800700 1 --2- 172.21.15.33:0/2628843785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de410b3a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.915+0000 7f4dec800700 1 -- 172.21.15.33:0/2628843785 >> 172.21.15.33:0/2628843785 conn(0x7f4de407e350 msgr2=0x7f4de407e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.915+0000 7f4dec800700 1 -- 172.21.15.33:0/2628843785 shutdown_connections 2024-04-26T11:18:25.568 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.915+0000 7f4dec800700 1 -- 172.21.15.33:0/2628843785 wait complete. 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.916+0000 7f4dec800700 1 Processor -- start 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.916+0000 7f4dec800700 1 -- start start 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.916+0000 7f4dec800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.916+0000 7f4dec800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4de410f4d0 con 0x7f4de410af80 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.917+0000 7f4de9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.917+0000 7f4de9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de419ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42400/0 (socket says 172.21.15.33:42400) 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.917+0000 7f4de9e00700 1 -- 172.21.15.33:0/3980741299 learned_addr learned my addr 172.21.15.33:0/3980741299 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.917+0000 7f4de9e00700 1 -- 172.21.15.33:0/3980741299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de419f320 con 0x7f4de410af80 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.917+0000 7f4de9e00700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de419ede0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8000c00 tx=0x7f4dd8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.918+0000 7f4ddea00700 1 -- 172.21.15.33:0/3980741299 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd8003aa0 con 0x7f4de410af80 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.918+0000 7f4ddea00700 1 -- 172.21.15.33:0/3980741299 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dd801b8f0 con 0x7f4de410af80 2024-04-26T11:18:25.569 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.918+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de419f5b0 con 0x7f4de410af80 2024-04-26T11:18:25.571 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.918+0000 7f4ddea00700 1 -- 172.21.15.33:0/3980741299 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd8023870 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.918+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de41a22a0 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.919+0000 7f4ddea00700 1 -- 172.21.15.33:0/3980741299 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4dd8023b70 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.920+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de410f650 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.920+0000 7f4ddea00700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4dcc0396a0 0x7f4dcc03bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.920+0000 7f4ddea00700 1 -- 172.21.15.33:0/3980741299 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4dd804e160 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.920+0000 7f4de9400700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4dcc0396a0 0x7f4dcc03bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.921+0000 7f4de9400700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4dcc0396a0 0x7f4dcc03bb60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4006fd0 tx=0x7f4dd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:23.925+0000 7f4ddea00700 1 -- 172.21.15.33:0/3980741299 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4dd801f030 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.165+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f4de41a28a0 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.165+0000 7f4ddea00700 1 -- 172.21.15.33:0/3980741299 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f4dd801bd30 con 0x7f4de410af80 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.168+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4dcc0396a0 msgr2=0x7f4dcc03bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.168+0000 7f4dec800700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4dcc0396a0 0x7f4dcc03bb60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4006fd0 tx=0x7f4dd4006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.168+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 msgr2=0x7f4de419ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.168+0000 7f4dec800700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de419ede0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8000c00 tx=0x7f4dd8011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.169+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 shutdown_connections 2024-04-26T11:18:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.169+0000 7f4dec800700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4dcc0396a0 0x7f4dcc03bb60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.169+0000 7f4dec800700 1 --2- 172.21.15.33:0/3980741299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4de410af80 0x7f4de419ede0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.169+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 >> 172.21.15.33:0/3980741299 conn(0x7f4de407e350 msgr2=0x7f4de4108410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.169+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 shutdown_connections 2024-04-26T11:18:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:24.169+0000 7f4dec800700 1 -- 172.21.15.33:0/3980741299 wait complete. 2024-04-26T11:18:25.598 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T11:18:25.598 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-04-26T11:18:25.625 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T11:18:25.625 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-04-26T11:18:25.625 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-04-26T11:18:25.625 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T11:18:25.632 INFO:teuthology.orchestra.run.smithi033.stdout:Ceph Dashboard is now available at: 2024-04-26T11:18:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: URL: https://smithi033.front.sepia.ceph.com:8443/ 2024-04-26T11:18:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: User: admin 2024-04-26T11:18:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: Password: nzep7k7oy9 2024-04-26T11:18:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:25.634 INFO:teuthology.orchestra.run.smithi033.stdout:Saving cluster configuration to /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config directory 2024-04-26T11:18:25.636 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling autotune for osd_memory_target 2024-04-26T11:18:27.121 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.275+0000 7f2bae800700 1 Processor -- start 2024-04-26T11:18:27.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.275+0000 7f2bae800700 1 -- start start 2024-04-26T11:18:27.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.276+0000 7f2bae800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba810b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:27.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.276+0000 7f2bae800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba810b8c0 con 0x7f2ba810af60 2024-04-26T11:18:27.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.276+0000 7f2ba7400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba810b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:27.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.277+0000 7f2ba7400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba810b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42402/0 (socket says 172.21.15.33:42402) 2024-04-26T11:18:27.122 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.277+0000 7f2ba7400700 1 -- 172.21.15.33:0/3980810944 learned_addr learned my addr 172.21.15.33:0/3980810944 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:27.127 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.277+0000 7f2ba7400700 1 -- 172.21.15.33:0/3980810944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba810c070 con 0x7f2ba810af60 2024-04-26T11:18:27.127 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.277+0000 7f2ba7400700 1 --2- 172.21.15.33:0/3980810944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba810b380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b980098a0 tx=0x7f2b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639430b19dc29632 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:27.127 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.278+0000 7f2ba6a00700 1 -- 172.21.15.33:0/3980810944 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b98004160 con 0x7f2ba810af60 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.278+0000 7f2ba6a00700 1 -- 172.21.15.33:0/3980810944 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b98004300 con 0x7f2ba810af60 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.278+0000 7f2ba6a00700 1 -- 172.21.15.33:0/3980810944 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b98011a20 con 0x7f2ba810af60 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.278+0000 7f2bae800700 1 -- 172.21.15.33:0/3980810944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 msgr2=0x7f2ba810b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.278+0000 7f2bae800700 1 --2- 172.21.15.33:0/3980810944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba810b380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b980098a0 tx=0x7f2b98009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.279+0000 7f2bae800700 1 -- 172.21.15.33:0/3980810944 shutdown_connections 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.279+0000 7f2bae800700 1 --2- 172.21.15.33:0/3980810944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba810b380 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.279+0000 7f2bae800700 1 -- 172.21.15.33:0/3980810944 >> 172.21.15.33:0/3980810944 conn(0x7f2ba807e350 msgr2=0x7f2ba807e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.279+0000 7f2bae800700 1 -- 172.21.15.33:0/3980810944 shutdown_connections 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.279+0000 7f2bae800700 1 -- 172.21.15.33:0/3980810944 wait complete. 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.280+0000 7f2bae800700 1 Processor -- start 2024-04-26T11:18:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.280+0000 7f2bae800700 1 -- start start 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.280+0000 7f2bae800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.280+0000 7f2bae800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba810f4b0 con 0x7f2ba810af60 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.281+0000 7f2ba7400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.281+0000 7f2ba7400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba819ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42406/0 (socket says 172.21.15.33:42406) 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.281+0000 7f2ba7400700 1 -- 172.21.15.33:0/651763256 learned_addr learned my addr 172.21.15.33:0/651763256 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.281+0000 7f2ba7400700 1 -- 172.21.15.33:0/651763256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba819f340 con 0x7f2ba810af60 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.281+0000 7f2ba7400700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba819ee00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b98000c00 tx=0x7f2b98011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.282+0000 7f2ba4c00700 1 -- 172.21.15.33:0/651763256 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b98003aa0 con 0x7f2ba810af60 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.282+0000 7f2ba4c00700 1 -- 172.21.15.33:0/651763256 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b9801b8f0 con 0x7f2ba810af60 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.282+0000 7f2ba4c00700 1 -- 172.21.15.33:0/651763256 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b98023870 con 0x7f2ba810af60 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.282+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba819f5d0 con 0x7f2ba810af60 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.282+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba81a2270 con 0x7f2ba810af60 2024-04-26T11:18:27.129 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.283+0000 7f2ba4c00700 1 -- 172.21.15.33:0/651763256 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2b9801b440 con 0x7f2ba810af60 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.283+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba810f630 con 0x7f2ba810af60 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.283+0000 7f2ba4c00700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2b8c039700 0x7f2b8c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.283+0000 7f2ba4c00700 1 -- 172.21.15.33:0/651763256 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2b9804e0e0 con 0x7f2ba810af60 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.284+0000 7f2b9fe00700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2b8c039700 0x7f2b8c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.284+0000 7f2b9fe00700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2b8c039700 0x7f2b8c03bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006fd0 tx=0x7f2b90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.289+0000 7f2ba4c00700 1 -- 172.21.15.33:0/651763256 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b9801f030 con 0x7f2ba810af60 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.519+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f2ba81a28c0 con 0x7f2ba810af60 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.520+0000 7f2ba4c00700 1 -- 172.21.15.33:0/651763256 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f2b980292a0 con 0x7f2ba810af60 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2b8c039700 msgr2=0x7f2b8c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2b8c039700 0x7f2b8c03bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006fd0 tx=0x7f2b90006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:27.130 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 msgr2=0x7f2ba819ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:27.131 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba819ee00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b98000c00 tx=0x7f2b98011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:27.131 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 shutdown_connections 2024-04-26T11:18:27.132 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2b8c039700 0x7f2b8c03bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:27.132 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 --2- 172.21.15.33:0/651763256 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ba810af60 0x7f2ba819ee00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:27.132 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 >> 172.21.15.33:0/651763256 conn(0x7f2ba807e350 msgr2=0x7f2ba8108400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:27.132 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 shutdown_connections 2024-04-26T11:18:27.132 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:26.524+0000 7f2bae800700 1 -- 172.21.15.33:0/651763256 wait complete. 2024-04-26T11:18:28.696 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.695+0000 7f6699c00700 1 Processor -- start 2024-04-26T11:18:28.696 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.696+0000 7f6699c00700 1 -- start start 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.696+0000 7f6699c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669410bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.696+0000 7f6699c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f669407ce50 con 0x7f6694109850 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.697+0000 7f6692a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669410bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.697+0000 7f6692a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669410bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42414/0 (socket says 172.21.15.33:42414) 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.697+0000 7f6692a00700 1 -- 172.21.15.33:0/3905917542 learned_addr learned my addr 172.21.15.33:0/3905917542 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.697+0000 7f6692a00700 1 -- 172.21.15.33:0/3905917542 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669410c180 con 0x7f6694109850 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.697+0000 7f6692a00700 1 --2- 172.21.15.33:0/3905917542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669410bc40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66840098a0 tx=0x7f6684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2600e33a708948b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.698+0000 7f6691600700 1 -- 172.21.15.33:0/3905917542 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684004160 con 0x7f6694109850 2024-04-26T11:18:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.698+0000 7f6691600700 1 -- 172.21.15.33:0/3905917542 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6684004300 con 0x7f6694109850 2024-04-26T11:18:28.702 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.698+0000 7f6691600700 1 -- 172.21.15.33:0/3905917542 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684011a20 con 0x7f6694109850 2024-04-26T11:18:28.702 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.699+0000 7f6699c00700 1 -- 172.21.15.33:0/3905917542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 msgr2=0x7f669410bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:28.702 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.699+0000 7f6699c00700 1 --2- 172.21.15.33:0/3905917542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669410bc40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66840098a0 tx=0x7f6684009bb0 comp rx=0 tx=0).stop 2024-04-26T11:18:28.702 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.699+0000 7f6699c00700 1 -- 172.21.15.33:0/3905917542 shutdown_connections 2024-04-26T11:18:28.702 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.699+0000 7f6699c00700 1 --2- 172.21.15.33:0/3905917542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669410bc40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.699+0000 7f6699c00700 1 -- 172.21.15.33:0/3905917542 >> 172.21.15.33:0/3905917542 conn(0x7f66941031c0 msgr2=0x7f6694105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.699+0000 7f6699c00700 1 -- 172.21.15.33:0/3905917542 shutdown_connections 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.699+0000 7f6699c00700 1 -- 172.21.15.33:0/3905917542 wait complete. 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.700+0000 7f6699c00700 1 Processor -- start 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.700+0000 7f6699c00700 1 -- start start 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.701+0000 7f6699c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.701+0000 7f6699c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f669410d120 con 0x7f6694109850 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.701+0000 7f6692a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.701+0000 7f6692a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42422/0 (socket says 172.21.15.33:42422) 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.701+0000 7f6692a00700 1 -- 172.21.15.33:0/3175019548 learned_addr learned my addr 172.21.15.33:0/3175019548 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.701+0000 7f6692a00700 1 -- 172.21.15.33:0/3175019548 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669419d1e0 con 0x7f6694109850 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.702+0000 7f6692a00700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669419cca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6684004660 tx=0x7f6684011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:28.703 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.702+0000 7f668b400700 1 -- 172.21.15.33:0/3175019548 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684003aa0 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.702+0000 7f668b400700 1 -- 172.21.15.33:0/3175019548 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f668401b8f0 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.702+0000 7f668b400700 1 -- 172.21.15.33:0/3175019548 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684023900 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.702+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f669419d470 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.702+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66941a0160 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.703+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6694064a80 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.704+0000 7f668b400700 1 -- 172.21.15.33:0/3175019548 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f668401baa0 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.704+0000 7f668b400700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6678039700 0x7f667803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.704+0000 7f668b400700 1 -- 172.21.15.33:0/3175019548 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f668404e170 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.705+0000 7f6692000700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6678039700 0x7f667803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.705+0000 7f6692000700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6678039700 0x7f667803bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:27.710+0000 7f668b400700 1 -- 172.21.15.33:0/3175019548 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f668401f030 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.057+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f66941a03d0 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.063+0000 7f668b400700 1 -- 172.21.15.33:0/3175019548 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f66840202b0 con 0x7f6694109850 2024-04-26T11:18:28.704 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6678039700 msgr2=0x7f667803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6678039700 0x7f667803bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 msgr2=0x7f669419cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669419cca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6684004660 tx=0x7f6684011aa0 comp rx=0 tx=0).stop 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 shutdown_connections 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6678039700 0x7f667803bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 --2- 172.21.15.33:0/3175019548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6694109850 0x7f669419cca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 >> 172.21.15.33:0/3175019548 conn(0x7f66941031c0 msgr2=0x7f6694103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.066+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 shutdown_connections 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-04-26T11:18:28.067+0000 7f6699c00700 1 -- 172.21.15.33:0/3175019548 wait complete. 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T11:18:28.705 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: ceph telemetry on 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout:For more information see: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:28.706 INFO:teuthology.orchestra.run.smithi033.stdout:Bootstrap complete. 2024-04-26T11:18:28.799 INFO:tasks.cephadm:Fetching config... 2024-04-26T11:18:28.800 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:18:28.800 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T11:18:28.852 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T11:18:28.853 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:18:28.853 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T11:18:28.900 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T11:18:28.900 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:18:28.900 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T11:18:29.021 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T11:18:29.021 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:18:29.021 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T11:18:29.123 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T11:18:29.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweyW5HCAjZMykYNqx62yDDB1HRB+Ki/g0trwk11T6XN+wG9P0ZbNqJQXWSp1uVzjtFayvRJ4axyYSJqRcqur2dpjv/8+ACAPWrDIkSaFB3hKakn5xPUVcs9GZ7qMZer+WEPfcKYQ/+z68XOwdkFFuvYe07IPnNUNVM2/LsoA1b/HAZ4g7pE3lk57ICRaPyNFA9Qw3TmUQGWjU3pZYjmt0j0bDbqp0PdYzjFb4f+35Jfcbb1uSQ1cpnGCzuq3QEIaxjA7AlqkXV4DpHkvUIztv3Yi/oR2Eqb/Sb/SaDVsCLA44inepIv/KTKcpqi3GvBa8GqARuBa8RJUub0lLYpjbtTnmgci1C/NkIKrSCM7UL41bKE8ZWdD0+7Yr/MsRNqwg5GdnPqunyix6ZSGNt3kLV5wLJzEfRwE++n2cEJod3GjyUMWNvgnlUYbQQCdADpdPG+n/1L8nC+XgdaSItmN0hIV7U/s0d4bYhoMCN6DjJ5TdmuM6rVRkbRsEhN52V0c= ceph-5cc7f146-03be-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T11:18:29.230 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweyW5HCAjZMykYNqx62yDDB1HRB+Ki/g0trwk11T6XN+wG9P0ZbNqJQXWSp1uVzjtFayvRJ4axyYSJqRcqur2dpjv/8+ACAPWrDIkSaFB3hKakn5xPUVcs9GZ7qMZer+WEPfcKYQ/+z68XOwdkFFuvYe07IPnNUNVM2/LsoA1b/HAZ4g7pE3lk57ICRaPyNFA9Qw3TmUQGWjU3pZYjmt0j0bDbqp0PdYzjFb4f+35Jfcbb1uSQ1cpnGCzuq3QEIaxjA7AlqkXV4DpHkvUIztv3Yi/oR2Eqb/Sb/SaDVsCLA44inepIv/KTKcpqi3GvBa8GqARuBa8RJUub0lLYpjbtTnmgci1C/NkIKrSCM7UL41bKE8ZWdD0+7Yr/MsRNqwg5GdnPqunyix6ZSGNt3kLV5wLJzEfRwE++n2cEJod3GjyUMWNvgnlUYbQQCdADpdPG+n/1L8nC+XgdaSItmN0hIV7U/s0d4bYhoMCN6DjJ5TdmuM6rVRkbRsEhN52V0c= ceph-5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:18:29.260 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:29 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3175019548' entity='client.admin' 2024-04-26T11:18:29.268 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweyW5HCAjZMykYNqx62yDDB1HRB+Ki/g0trwk11T6XN+wG9P0ZbNqJQXWSp1uVzjtFayvRJ4axyYSJqRcqur2dpjv/8+ACAPWrDIkSaFB3hKakn5xPUVcs9GZ7qMZer+WEPfcKYQ/+z68XOwdkFFuvYe07IPnNUNVM2/LsoA1b/HAZ4g7pE3lk57ICRaPyNFA9Qw3TmUQGWjU3pZYjmt0j0bDbqp0PdYzjFb4f+35Jfcbb1uSQ1cpnGCzuq3QEIaxjA7AlqkXV4DpHkvUIztv3Yi/oR2Eqb/Sb/SaDVsCLA44inepIv/KTKcpqi3GvBa8GqARuBa8RJUub0lLYpjbtTnmgci1C/NkIKrSCM7UL41bKE8ZWdD0+7Yr/MsRNqwg5GdnPqunyix6ZSGNt3kLV5wLJzEfRwE++n2cEJod3GjyUMWNvgnlUYbQQCdADpdPG+n/1L8nC+XgdaSItmN0hIV7U/s0d4bYhoMCN6DjJ5TdmuM6rVRkbRsEhN52V0c= ceph-5cc7f146-03be-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T11:18:29.308 INFO:teuthology.orchestra.run.smithi096.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweyW5HCAjZMykYNqx62yDDB1HRB+Ki/g0trwk11T6XN+wG9P0ZbNqJQXWSp1uVzjtFayvRJ4axyYSJqRcqur2dpjv/8+ACAPWrDIkSaFB3hKakn5xPUVcs9GZ7qMZer+WEPfcKYQ/+z68XOwdkFFuvYe07IPnNUNVM2/LsoA1b/HAZ4g7pE3lk57ICRaPyNFA9Qw3TmUQGWjU3pZYjmt0j0bDbqp0PdYzjFb4f+35Jfcbb1uSQ1cpnGCzuq3QEIaxjA7AlqkXV4DpHkvUIztv3Yi/oR2Eqb/Sb/SaDVsCLA44inepIv/KTKcpqi3GvBa8GqARuBa8RJUub0lLYpjbtTnmgci1C/NkIKrSCM7UL41bKE8ZWdD0+7Yr/MsRNqwg5GdnPqunyix6ZSGNt3kLV5wLJzEfRwE++n2cEJod3GjyUMWNvgnlUYbQQCdADpdPG+n/1L8nC+XgdaSItmN0hIV7U/s0d4bYhoMCN6DjJ5TdmuM6rVRkbRsEhN52V0c= ceph-5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:18:29.320 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweyW5HCAjZMykYNqx62yDDB1HRB+Ki/g0trwk11T6XN+wG9P0ZbNqJQXWSp1uVzjtFayvRJ4axyYSJqRcqur2dpjv/8+ACAPWrDIkSaFB3hKakn5xPUVcs9GZ7qMZer+WEPfcKYQ/+z68XOwdkFFuvYe07IPnNUNVM2/LsoA1b/HAZ4g7pE3lk57ICRaPyNFA9Qw3TmUQGWjU3pZYjmt0j0bDbqp0PdYzjFb4f+35Jfcbb1uSQ1cpnGCzuq3QEIaxjA7AlqkXV4DpHkvUIztv3Yi/oR2Eqb/Sb/SaDVsCLA44inepIv/KTKcpqi3GvBa8GqARuBa8RJUub0lLYpjbtTnmgci1C/NkIKrSCM7UL41bKE8ZWdD0+7Yr/MsRNqwg5GdnPqunyix6ZSGNt3kLV5wLJzEfRwE++n2cEJod3GjyUMWNvgnlUYbQQCdADpdPG+n/1L8nC+XgdaSItmN0hIV7U/s0d4bYhoMCN6DjJ5TdmuM6rVRkbRsEhN52V0c= ceph-5cc7f146-03be-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T11:18:29.360 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCweyW5HCAjZMykYNqx62yDDB1HRB+Ki/g0trwk11T6XN+wG9P0ZbNqJQXWSp1uVzjtFayvRJ4axyYSJqRcqur2dpjv/8+ACAPWrDIkSaFB3hKakn5xPUVcs9GZ7qMZer+WEPfcKYQ/+z68XOwdkFFuvYe07IPnNUNVM2/LsoA1b/HAZ4g7pE3lk57ICRaPyNFA9Qw3TmUQGWjU3pZYjmt0j0bDbqp0PdYzjFb4f+35Jfcbb1uSQ1cpnGCzuq3QEIaxjA7AlqkXV4DpHkvUIztv3Yi/oR2Eqb/Sb/SaDVsCLA44inepIv/KTKcpqi3GvBa8GqARuBa8RJUub0lLYpjbtTnmgci1C/NkIKrSCM7UL41bKE8ZWdD0+7Yr/MsRNqwg5GdnPqunyix6ZSGNt3kLV5wLJzEfRwE++n2cEJod3GjyUMWNvgnlUYbQQCdADpdPG+n/1L8nC+XgdaSItmN0hIV7U/s0d4bYhoMCN6DjJ5TdmuM6rVRkbRsEhN52V0c= ceph-5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:18:29.372 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T11:18:29.802 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:18:32.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.267+0000 7f950d600700 1 -- 172.21.15.33:0/3267547817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 msgr2=0x7f95081032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:32.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.267+0000 7f950d600700 1 --2- 172.21.15.33:0/3267547817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f95081032a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009910 tx=0x7f94f8009c20 comp rx=0 tx=0).stop 2024-04-26T11:18:32.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.267+0000 7f950d600700 1 -- 172.21.15.33:0/3267547817 shutdown_connections 2024-04-26T11:18:32.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.267+0000 7f950d600700 1 --2- 172.21.15.33:0/3267547817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f95081032a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:32.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.267+0000 7f950d600700 1 -- 172.21.15.33:0/3267547817 >> 172.21.15.33:0/3267547817 conn(0x7f95080fe420 msgr2=0x7f9508100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:32.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.267+0000 7f950d600700 1 -- 172.21.15.33:0/3267547817 shutdown_connections 2024-04-26T11:18:32.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.267+0000 7f950d600700 1 -- 172.21.15.33:0/3267547817 wait complete. 2024-04-26T11:18:32.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.268+0000 7f950d600700 1 Processor -- start 2024-04-26T11:18:32.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.268+0000 7f950d600700 1 -- start start 2024-04-26T11:18:32.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.269+0000 7f950d600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f9508198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:32.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.269+0000 7f950d600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9508108d40 con 0x7f9508102e80 2024-04-26T11:18:32.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.269+0000 7f9506a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f9508198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:32.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.269+0000 7f9506a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f9508198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43670/0 (socket says 172.21.15.33:43670) 2024-04-26T11:18:32.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.269+0000 7f9506a00700 1 -- 172.21.15.33:0/2114624150 learned_addr learned my addr 172.21.15.33:0/2114624150 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:32.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.270+0000 7f9506a00700 1 -- 172.21.15.33:0/2114624150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9508198a50 con 0x7f9508102e80 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.270+0000 7f9506a00700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f9508198510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94f8005190 tx=0x7f94f8004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.270+0000 7f94ff400700 1 -- 172.21.15.33:0/2114624150 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94f8027070 con 0x7f9508102e80 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.270+0000 7f94ff400700 1 -- 172.21.15.33:0/2114624150 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94f8022470 con 0x7f9508102e80 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.270+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9508198ce0 con 0x7f9508102e80 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.270+0000 7f94ff400700 1 -- 172.21.15.33:0/2114624150 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94f800f5f0 con 0x7f9508102e80 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.270+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f950819b980 con 0x7f9508102e80 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.272+0000 7f94ff400700 1 -- 172.21.15.33:0/2114624150 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f94f802a070 con 0x7f9508102e80 2024-04-26T11:18:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.272+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95081094c0 con 0x7f9508102e80 2024-04-26T11:18:32.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.272+0000 7f94ff400700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f94ec0396b0 0x7f94ec03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:32.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.272+0000 7f94ff400700 1 -- 172.21.15.33:0/2114624150 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f94f801e030 con 0x7f9508102e80 2024-04-26T11:18:32.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.272+0000 7f9506000700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f94ec0396b0 0x7f94ec03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:32.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.273+0000 7f9506000700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f94ec0396b0 0x7f94ec03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f94f4006fd0 tx=0x7f94f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:32.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.278+0000 7f94ff400700 1 -- 172.21.15.33:0/2114624150 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94f8013070 con 0x7f9508102e80 2024-04-26T11:18:32.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.520+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f950802e5a0 con 0x7f9508102e80 2024-04-26T11:18:32.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.524+0000 7f94ff400700 1 -- 172.21.15.33:0/2114624150 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f94f80310c0 con 0x7f9508102e80 2024-04-26T11:18:32.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.527+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f94ec0396b0 msgr2=0x7f94ec03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.527+0000 7f950d600700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f94ec0396b0 0x7f94ec03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f94f4006fd0 tx=0x7f94f4006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.527+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 msgr2=0x7f9508198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.527+0000 7f950d600700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f9508198510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94f8005190 tx=0x7f94f8004f80 comp rx=0 tx=0).stop 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.528+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 shutdown_connections 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.528+0000 7f950d600700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f94ec0396b0 0x7f94ec03bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.528+0000 7f950d600700 1 --2- 172.21.15.33:0/2114624150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9508102e80 0x7f9508198510 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.528+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 >> 172.21.15.33:0/2114624150 conn(0x7f95080fe420 msgr2=0x7f95080fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.528+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 shutdown_connections 2024-04-26T11:18:32.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:32.528+0000 7f950d600700 1 -- 172.21.15.33:0/2114624150 wait complete. 2024-04-26T11:18:33.415 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T11:18:33.416 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T11:18:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:33 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2114624150' entity='client.admin' 2024-04-26T11:18:33.830 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:18:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.507+0000 7f5e00400700 1 -- 172.21.15.33:0/959752539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 msgr2=0x7f5df80758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.507+0000 7f5e00400700 1 --2- 172.21.15.33:0/959752539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df80758b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009910 tx=0x7f5dec009c20 comp rx=0 tx=0).stop 2024-04-26T11:18:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.507+0000 7f5e00400700 1 -- 172.21.15.33:0/959752539 shutdown_connections 2024-04-26T11:18:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.507+0000 7f5e00400700 1 --2- 172.21.15.33:0/959752539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df80758b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.507+0000 7f5e00400700 1 -- 172.21.15.33:0/959752539 >> 172.21.15.33:0/959752539 conn(0x7f5df80fe420 msgr2=0x7f5df8100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.507+0000 7f5e00400700 1 -- 172.21.15.33:0/959752539 shutdown_connections 2024-04-26T11:18:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.507+0000 7f5e00400700 1 -- 172.21.15.33:0/959752539 wait complete. 2024-04-26T11:18:36.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:36.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:36.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:18:36.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:36.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:18:36.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:36.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.508+0000 7f5e00400700 1 Processor -- start 2024-04-26T11:18:36.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.508+0000 7f5e00400700 1 -- start start 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.509+0000 7f5e00400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.509+0000 7f5e00400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5df810f460 con 0x7f5df8077450 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.509+0000 7f5dfda00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.509+0000 7f5dfda00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43688/0 (socket says 172.21.15.33:43688) 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.509+0000 7f5dfda00700 1 -- 172.21.15.33:0/315546199 learned_addr learned my addr 172.21.15.33:0/315546199 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.510+0000 7f5dfda00700 1 -- 172.21.15.33:0/315546199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df819f000 con 0x7f5df8077450 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.510+0000 7f5dfda00700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df819eac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5dec005950 tx=0x7f5dec004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.510+0000 7f5df6000700 1 -- 172.21.15.33:0/315546199 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dec026070 con 0x7f5df8077450 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.510+0000 7f5df6000700 1 -- 172.21.15.33:0/315546199 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dec00bc00 con 0x7f5df8077450 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.510+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df819f290 con 0x7f5df8077450 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.510+0000 7f5df6000700 1 -- 172.21.15.33:0/315546199 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dec003a10 con 0x7f5df8077450 2024-04-26T11:18:36.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.510+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df81a1f80 con 0x7f5df8077450 2024-04-26T11:18:36.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.512+0000 7f5df6000700 1 -- 172.21.15.33:0/315546199 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5dec029070 con 0x7f5df8077450 2024-04-26T11:18:36.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.512+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df810fb40 con 0x7f5df8077450 2024-04-26T11:18:36.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.512+0000 7f5df6000700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5de0039650 0x7f5de003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:36.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.512+0000 7f5df6000700 1 -- 172.21.15.33:0/315546199 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5dec01e070 con 0x7f5df8077450 2024-04-26T11:18:36.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.512+0000 7f5dfd000700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5de0039650 0x7f5de003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:36.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.513+0000 7f5dfd000700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5de0039650 0x7f5de003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5de4006fd0 tx=0x7f5de4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:36.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.518+0000 7f5df6000700 1 -- 172.21.15.33:0/315546199 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dec013070 con 0x7f5df8077450 2024-04-26T11:18:36.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.753+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f5df802e5a0 con 0x7f5de0039650 2024-04-26T11:18:36.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.760+0000 7f5df6000700 1 -- 172.21.15.33:0/315546199 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5df802e5a0 con 0x7f5de0039650 2024-04-26T11:18:36.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.763+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5de0039650 msgr2=0x7f5de003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:36.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.763+0000 7f5e00400700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5de0039650 0x7f5de003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5de4006fd0 tx=0x7f5de4006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.763+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 msgr2=0x7f5df819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.763+0000 7f5e00400700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df819eac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5dec005950 tx=0x7f5dec004fa0 comp rx=0 tx=0).stop 2024-04-26T11:18:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.764+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 shutdown_connections 2024-04-26T11:18:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.764+0000 7f5e00400700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5de0039650 0x7f5de003bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.764+0000 7f5e00400700 1 --2- 172.21.15.33:0/315546199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5df8077450 0x7f5df819eac0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.764+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 >> 172.21.15.33:0/315546199 conn(0x7f5df80fe420 msgr2=0x7f5df80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.764+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 shutdown_connections 2024-04-26T11:18:36.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:36.764+0000 7f5e00400700 1 -- 172.21.15.33:0/315546199 wait complete. 2024-04-26T11:18:37.451 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi096 2024-04-26T11:18:37.451 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:18:37.451 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.conf 2024-04-26T11:18:37.469 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:18:37.470 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:18:37.527 INFO:tasks.cephadm:Adding host smithi096 to orchestrator... 2024-04-26T11:18:37.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch host add smithi096 2024-04-26T11:18:37.957 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:18:38.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:37 smithi033 ceph-mon[27845]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:18:38.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:38.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:18:38.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:18:38.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:18:38.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:37 smithi033 ceph-mon[27845]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-26T11:18:38.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:37 smithi033 ceph-mon[27845]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:18:39.113 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:38 smithi033 ceph-mon[27845]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:39.114 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:38 smithi033 ceph-mon[27845]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:18:39.821 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:39 smithi033 ceph-mon[27845]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:18:39.821 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:39.821 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:39.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:39.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.997+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1795039514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 msgr2=0x7ff6f0105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:39.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.997+0000 7ff6f7a00700 1 --2- 172.21.15.33:0/1795039514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f0105690 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009910 tx=0x7ff6d8009c20 comp rx=0 tx=0).stop 2024-04-26T11:18:39.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.997+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1795039514 shutdown_connections 2024-04-26T11:18:39.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.997+0000 7ff6f7a00700 1 --2- 172.21.15.33:0/1795039514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f0105690 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:39.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.997+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1795039514 >> 172.21.15.33:0/1795039514 conn(0x7ff6f0100b40 msgr2=0x7ff6f0102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:40.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.997+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1795039514 shutdown_connections 2024-04-26T11:18:40.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.997+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1795039514 wait complete. 2024-04-26T11:18:40.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.999+0000 7ff6f7a00700 1 Processor -- start 2024-04-26T11:18:40.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.999+0000 7ff6f7a00700 1 -- start start 2024-04-26T11:18:40.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.999+0000 7ff6f7a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f01a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:40.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:39.999+0000 7ff6f7a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f010f530 con 0x7ff6f01052b0 2024-04-26T11:18:40.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.000+0000 7ff6f5000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f01a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:40.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.000+0000 7ff6f5000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f01a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44488/0 (socket says 172.21.15.33:44488) 2024-04-26T11:18:40.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.000+0000 7ff6f5000700 1 -- 172.21.15.33:0/1707580051 learned_addr learned my addr 172.21.15.33:0/1707580051 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:40.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.000+0000 7ff6f5000700 1 -- 172.21.15.33:0/1707580051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f01a0c50 con 0x7ff6f01052b0 2024-04-26T11:18:40.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.001+0000 7ff6f5000700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f01a0710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80056d0 tx=0x7ff6d80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:40.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.001+0000 7ff6ee000700 1 -- 172.21.15.33:0/1707580051 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d8026070 con 0x7ff6f01052b0 2024-04-26T11:18:40.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.001+0000 7ff6ee000700 1 -- 172.21.15.33:0/1707580051 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6d800bba0 con 0x7ff6f01052b0 2024-04-26T11:18:40.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.001+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f01a0f40 con 0x7ff6f01052b0 2024-04-26T11:18:40.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.001+0000 7ff6ee000700 1 -- 172.21.15.33:0/1707580051 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d8003920 con 0x7ff6f01052b0 2024-04-26T11:18:40.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.001+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f019a8d0 con 0x7ff6f01052b0 2024-04-26T11:18:40.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.002+0000 7ff6ee000700 1 -- 172.21.15.33:0/1707580051 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff6d8029070 con 0x7ff6f01052b0 2024-04-26T11:18:40.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.002+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f0106790 con 0x7ff6f01052b0 2024-04-26T11:18:40.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.003+0000 7ff6ee000700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff6d00396b0 0x7ff6d003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:40.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.003+0000 7ff6ee000700 1 -- 172.21.15.33:0/1707580051 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff6d801e070 con 0x7ff6f01052b0 2024-04-26T11:18:40.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.003+0000 7ff6e7e00700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff6d00396b0 0x7ff6d003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:40.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.004+0000 7ff6e7e00700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff6d00396b0 0x7ff6d003bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0006fd0 tx=0x7ff6e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:40.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.008+0000 7ff6ee000700 1 -- 172.21.15.33:0/1707580051 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6d8013070 con 0x7ff6f01052b0 2024-04-26T11:18:40.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:40.245+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}) -- 0x7ff6f019be00 con 0x7ff6d00396b0 2024-04-26T11:18:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:40 smithi033 ceph-mon[27845]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:40 smithi033 ceph-mon[27845]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:18:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:41 smithi033 ceph-mon[27845]: Deploying cephadm binary to smithi096 2024-04-26T11:18:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:42 smithi033 ceph-mon[27845]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:45.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.170+0000 7ff6ee000700 1 -- 172.21.15.33:0/1707580051 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7ff6f019be00 con 0x7ff6d00396b0 2024-04-26T11:18:45.172 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:44 smithi033 ceph-mon[27845]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:45.172 INFO:teuthology.orchestra.run.smithi033.stdout:Added host 'smithi096' with addr '172.21.15.96' 2024-04-26T11:18:45.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff6d00396b0 msgr2=0x7ff6d003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff6d00396b0 0x7ff6d003bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0006fd0 tx=0x7ff6e0006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 msgr2=0x7ff6f01a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f01a0710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80056d0 tx=0x7ff6d80057b0 comp rx=0 tx=0).stop 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 shutdown_connections 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff6d00396b0 0x7ff6d003bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 --2- 172.21.15.33:0/1707580051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6f01052b0 0x7ff6f01a0710 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 >> 172.21.15.33:0/1707580051 conn(0x7ff6f0100b40 msgr2=0x7ff6f0108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.174+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 shutdown_connections 2024-04-26T11:18:45.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:45.175+0000 7ff6f7a00700 1 -- 172.21.15.33:0/1707580051 wait complete. 2024-04-26T11:18:45.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T11:18:46.288 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:18:46.435 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:46 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:46.435 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:46 smithi033 ceph-mon[27845]: Added host smithi096 2024-04-26T11:18:46.435 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:46 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:18:46.435 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:46 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:47 smithi033 ceph-mon[27845]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:48.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.238+0000 7f3f79e00700 1 -- 172.21.15.33:0/2777838579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 msgr2=0x7f3f741032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:48.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.238+0000 7f3f79e00700 1 --2- 172.21.15.33:0/2777838579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f741032b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009910 tx=0x7f3f68009c20 comp rx=0 tx=0).stop 2024-04-26T11:18:48.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.238+0000 7f3f79e00700 1 -- 172.21.15.33:0/2777838579 shutdown_connections 2024-04-26T11:18:48.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.238+0000 7f3f79e00700 1 --2- 172.21.15.33:0/2777838579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f741032b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:48.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.238+0000 7f3f79e00700 1 -- 172.21.15.33:0/2777838579 >> 172.21.15.33:0/2777838579 conn(0x7f3f740fe780 msgr2=0x7f3f74100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:48.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.238+0000 7f3f79e00700 1 -- 172.21.15.33:0/2777838579 shutdown_connections 2024-04-26T11:18:48.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.238+0000 7f3f79e00700 1 -- 172.21.15.33:0/2777838579 wait complete. 2024-04-26T11:18:48.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.239+0000 7f3f79e00700 1 Processor -- start 2024-04-26T11:18:48.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.240+0000 7f3f79e00700 1 -- start start 2024-04-26T11:18:48.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.240+0000 7f3f79e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f74198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:48.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.240+0000 7f3f79e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7410d150 con 0x7f3f74102ed0 2024-04-26T11:18:48.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.240+0000 7f3f78a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f74198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:48.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.240+0000 7f3f78a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f74198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44504/0 (socket says 172.21.15.33:44504) 2024-04-26T11:18:48.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.240+0000 7f3f78a00700 1 -- 172.21.15.33:0/481621093 learned_addr learned my addr 172.21.15.33:0/481621093 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:48.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.241+0000 7f3f78a00700 1 -- 172.21.15.33:0/481621093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f74198c90 con 0x7f3f74102ed0 2024-04-26T11:18:48.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.241+0000 7f3f78a00700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f74198750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006010 tx=0x7f3f68005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:48.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.241+0000 7f3f71600700 1 -- 172.21.15.33:0/481621093 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f68026070 con 0x7f3f74102ed0 2024-04-26T11:18:48.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.241+0000 7f3f71600700 1 -- 172.21.15.33:0/481621093 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f6800bba0 con 0x7f3f74102ed0 2024-04-26T11:18:48.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.241+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f74198f80 con 0x7f3f74102ed0 2024-04-26T11:18:48.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.242+0000 7f3f71600700 1 -- 172.21.15.33:0/481621093 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f68003910 con 0x7f3f74102ed0 2024-04-26T11:18:48.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.242+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f7419d020 con 0x7f3f74102ed0 2024-04-26T11:18:48.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.243+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f741043b0 con 0x7f3f74102ed0 2024-04-26T11:18:48.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.243+0000 7f3f71600700 1 -- 172.21.15.33:0/481621093 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3f68029070 con 0x7f3f74102ed0 2024-04-26T11:18:48.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.244+0000 7f3f71600700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f3f5c0396b0 0x7f3f5c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:48.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.244+0000 7f3f71600700 1 -- 172.21.15.33:0/481621093 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3f6801e070 con 0x7f3f74102ed0 2024-04-26T11:18:48.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.244+0000 7f3f73e00700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f3f5c0396b0 0x7f3f5c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:48.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.245+0000 7f3f73e00700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f3f5c0396b0 0x7f3f5c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:48.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.254+0000 7f3f71600700 1 -- 172.21.15.33:0/481621093 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f68013070 con 0x7f3f74102ed0 2024-04-26T11:18:48.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.500+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f74199ab0 con 0x7f3f5c0396b0 2024-04-26T11:18:48.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.501+0000 7f3f71600700 1 -- 172.21.15.33:0/481621093 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f3f74199ab0 con 0x7f3f5c0396b0 2024-04-26T11:18:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:18:48.503 INFO:teuthology.orchestra.run.smithi033.stdout:[{"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}] 2024-04-26T11:18:48.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f3f5c0396b0 msgr2=0x7f3f5c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:48.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f3f5c0396b0 0x7f3f5c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:48.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 msgr2=0x7f3f74198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:48.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f74198750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006010 tx=0x7f3f68005780 comp rx=0 tx=0).stop 2024-04-26T11:18:48.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 shutdown_connections 2024-04-26T11:18:48.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f3f5c0396b0 0x7f3f5c03bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:48.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 --2- 172.21.15.33:0/481621093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3f74102ed0 0x7f3f74198750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:48.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.504+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 >> 172.21.15.33:0/481621093 conn(0x7f3f740fe780 msgr2=0x7f3f7410aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:48.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.505+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 shutdown_connections 2024-04-26T11:18:48.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:48.505+0000 7f3f79e00700 1 -- 172.21.15.33:0/481621093 wait complete. 2024-04-26T11:18:49.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:48 smithi033 ceph-mon[27845]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:49.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:48 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:49.223 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-04-26T11:18:49.223 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:18:49.223 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-04-26T11:18:49.241 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:18:49.241 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:18:49.299 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-04-26T11:18:49.299 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch host add smithi194 2024-04-26T11:18:49.738 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:18:50.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:49 smithi033 ceph-mon[27845]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:18:51.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:50 smithi033 ceph-mon[27845]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:51.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.561+0000 7f894f200700 1 -- 172.21.15.33:0/1934729880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 msgr2=0x7f895006ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:51.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.561+0000 7f894f200700 1 --2- 172.21.15.33:0/1934729880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f895006ac20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8940009910 tx=0x7f8940009c20 comp rx=0 tx=0).stop 2024-04-26T11:18:51.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.562+0000 7f894f200700 1 -- 172.21.15.33:0/1934729880 shutdown_connections 2024-04-26T11:18:51.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.562+0000 7f894f200700 1 --2- 172.21.15.33:0/1934729880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f895006ac20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:51.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.562+0000 7f894f200700 1 -- 172.21.15.33:0/1934729880 >> 172.21.15.33:0/1934729880 conn(0x7f89500776d0 msgr2=0x7f8950077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:51.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.562+0000 7f894f200700 1 -- 172.21.15.33:0/1934729880 shutdown_connections 2024-04-26T11:18:51.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.562+0000 7f894f200700 1 -- 172.21.15.33:0/1934729880 wait complete. 2024-04-26T11:18:51.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.563+0000 7f894f200700 1 Processor -- start 2024-04-26T11:18:51.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.563+0000 7f894f200700 1 -- start start 2024-04-26T11:18:51.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.564+0000 7f894f200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f89501a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:51.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.564+0000 7f894f200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f895010f3b0 con 0x7f895006a840 2024-04-26T11:18:51.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.564+0000 7f894de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f89501a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:51.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.564+0000 7f894de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f89501a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58754/0 (socket says 172.21.15.33:58754) 2024-04-26T11:18:51.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.564+0000 7f894de00700 1 -- 172.21.15.33:0/3826181830 learned_addr learned my addr 172.21.15.33:0/3826181830 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:51.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.564+0000 7f894de00700 1 -- 172.21.15.33:0/3826181830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89501a0be0 con 0x7f895006a840 2024-04-26T11:18:51.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.565+0000 7f894de00700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f89501a06a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8940006010 tx=0x7f8940005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:51.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.565+0000 7f8947400700 1 -- 172.21.15.33:0/3826181830 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8940026070 con 0x7f895006a840 2024-04-26T11:18:51.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.565+0000 7f8947400700 1 -- 172.21.15.33:0/3826181830 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f894000bc00 con 0x7f895006a840 2024-04-26T11:18:51.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.565+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89501a0e70 con 0x7f895006a840 2024-04-26T11:18:51.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.565+0000 7f8947400700 1 -- 172.21.15.33:0/3826181830 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8940003910 con 0x7f895006a840 2024-04-26T11:18:51.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.565+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895019a770 con 0x7f895006a840 2024-04-26T11:18:51.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.566+0000 7f8947400700 1 -- 172.21.15.33:0/3826181830 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8940029070 con 0x7f895006a840 2024-04-26T11:18:51.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.566+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89501021b0 con 0x7f895006a840 2024-04-26T11:18:51.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.567+0000 7f8947400700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f89340396b0 0x7f893403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:51.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.567+0000 7f8947400700 1 -- 172.21.15.33:0/3826181830 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f894001e070 con 0x7f895006a840 2024-04-26T11:18:51.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.567+0000 7f8945400700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f89340396b0 0x7f893403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:51.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.568+0000 7f8945400700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f89340396b0 0x7f893403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8938006fd0 tx=0x7f8938006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:51.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.573+0000 7f8947400700 1 -- 172.21.15.33:0/3826181830 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8940013070 con 0x7f895006a840 2024-04-26T11:18:51.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:51.813+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) -- 0x7f895019bca0 con 0x7f89340396b0 2024-04-26T11:18:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:18:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:52 smithi033 ceph-mon[27845]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:18:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:53 smithi033 ceph-mon[27845]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:53 smithi033 ceph-mon[27845]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:18:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:53 smithi033 ceph-mon[27845]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:18:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:53 smithi033 ceph-mon[27845]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:18:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:54 smithi033 ceph-mon[27845]: Deploying cephadm binary to smithi194 2024-04-26T11:18:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:55 smithi033 ceph-mon[27845]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:56.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.724+0000 7f8947400700 1 -- 172.21.15.33:0/3826181830 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f895019bca0 con 0x7f89340396b0 2024-04-26T11:18:56.727 INFO:teuthology.orchestra.run.smithi033.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-04-26T11:18:56.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f89340396b0 msgr2=0x7f893403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:56.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f89340396b0 0x7f893403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8938006fd0 tx=0x7f8938006e40 comp rx=0 tx=0).stop 2024-04-26T11:18:56.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 msgr2=0x7f89501a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:56.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f89501a06a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8940006010 tx=0x7f8940005780 comp rx=0 tx=0).stop 2024-04-26T11:18:56.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 shutdown_connections 2024-04-26T11:18:56.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f89340396b0 0x7f893403bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:56.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 --2- 172.21.15.33:0/3826181830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f895006a840 0x7f89501a06a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:56.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.728+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 >> 172.21.15.33:0/3826181830 conn(0x7f89500776d0 msgr2=0x7f89501002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:56.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.729+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 shutdown_connections 2024-04-26T11:18:56.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:56.729+0000 7f894f200700 1 -- 172.21.15.33:0/3826181830 wait complete. 2024-04-26T11:18:57.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T11:18:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:57 smithi033 ceph-mon[27845]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:18:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:57.883 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:18:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:58 smithi033 ceph-mon[27845]: Added host smithi194 2024-04-26T11:18:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:18:59.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:18:59 smithi033 ceph-mon[27845]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:18:59.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.839+0000 7fd2ec400700 1 -- 172.21.15.33:0/2354934204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 msgr2=0x7fd2e4105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:18:59.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.839+0000 7fd2ec400700 1 --2- 172.21.15.33:0/2354934204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e4105690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009910 tx=0x7fd2d8009c20 comp rx=0 tx=0).stop 2024-04-26T11:18:59.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.840+0000 7fd2ec400700 1 -- 172.21.15.33:0/2354934204 shutdown_connections 2024-04-26T11:18:59.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.840+0000 7fd2ec400700 1 --2- 172.21.15.33:0/2354934204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e4105690 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:18:59.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.840+0000 7fd2ec400700 1 -- 172.21.15.33:0/2354934204 >> 172.21.15.33:0/2354934204 conn(0x7fd2e4100b40 msgr2=0x7fd2e4102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:18:59.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.840+0000 7fd2ec400700 1 -- 172.21.15.33:0/2354934204 shutdown_connections 2024-04-26T11:18:59.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.840+0000 7fd2ec400700 1 -- 172.21.15.33:0/2354934204 wait complete. 2024-04-26T11:18:59.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.841+0000 7fd2ec400700 1 Processor -- start 2024-04-26T11:18:59.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.841+0000 7fd2ec400700 1 -- start start 2024-04-26T11:18:59.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.842+0000 7fd2ec400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e41a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:59.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.842+0000 7fd2ec400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e410f530 con 0x7fd2e41052b0 2024-04-26T11:18:59.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.842+0000 7fd2e9a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e41a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:59.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.842+0000 7fd2e9a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e41a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56700/0 (socket says 172.21.15.33:56700) 2024-04-26T11:18:59.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.842+0000 7fd2e9a00700 1 -- 172.21.15.33:0/870742099 learned_addr learned my addr 172.21.15.33:0/870742099 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:18:59.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.842+0000 7fd2e9a00700 1 -- 172.21.15.33:0/870742099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e41a0c50 con 0x7fd2e41052b0 2024-04-26T11:18:59.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.843+0000 7fd2e9a00700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e41a0710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80056d0 tx=0x7fd2d80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:59.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.843+0000 7fd2e2000700 1 -- 172.21.15.33:0/870742099 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2d8026070 con 0x7fd2e41052b0 2024-04-26T11:18:59.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.843+0000 7fd2e2000700 1 -- 172.21.15.33:0/870742099 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2d800bba0 con 0x7fd2e41052b0 2024-04-26T11:18:59.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.843+0000 7fd2e2000700 1 -- 172.21.15.33:0/870742099 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2d8003920 con 0x7fd2e41052b0 2024-04-26T11:18:59.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.843+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e41a0f40 con 0x7fd2e41052b0 2024-04-26T11:18:59.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.843+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e419a8d0 con 0x7fd2e41052b0 2024-04-26T11:18:59.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.844+0000 7fd2e2000700 1 -- 172.21.15.33:0/870742099 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd2d8029070 con 0x7fd2e41052b0 2024-04-26T11:18:59.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.844+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e4106790 con 0x7fd2e41052b0 2024-04-26T11:18:59.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.845+0000 7fd2e2000700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd2cc039650 0x7fd2cc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:18:59.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.845+0000 7fd2e2000700 1 -- 172.21.15.33:0/870742099 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd2d801e070 con 0x7fd2e41052b0 2024-04-26T11:18:59.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.845+0000 7fd2e9000700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd2cc039650 0x7fd2cc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:18:59.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.846+0000 7fd2e9000700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd2cc039650 0x7fd2cc03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0006fd0 tx=0x7fd2d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:18:59.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:18:59.850+0000 7fd2e2000700 1 -- 172.21.15.33:0/870742099 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2d8013070 con 0x7fd2e41052b0 2024-04-26T11:19:00.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.090+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2e419be00 con 0x7fd2cc039650 2024-04-26T11:19:00.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.092+0000 7fd2e2000700 1 -- 172.21.15.33:0/870742099 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fd2e419be00 con 0x7fd2cc039650 2024-04-26T11:19:00.094 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:19:00.094 INFO:teuthology.orchestra.run.smithi033.stdout:[{"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.095+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd2cc039650 msgr2=0x7fd2cc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd2cc039650 0x7fd2cc03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0006fd0 tx=0x7fd2d0006e40 comp rx=0 tx=0).stop 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 msgr2=0x7fd2e41a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e41a0710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80056d0 tx=0x7fd2d80057b0 comp rx=0 tx=0).stop 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 shutdown_connections 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd2cc039650 0x7fd2cc03bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 --2- 172.21.15.33:0/870742099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2e41052b0 0x7fd2e41a0710 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:00.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 >> 172.21.15.33:0/870742099 conn(0x7fd2e4100b40 msgr2=0x7fd2e4108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:00.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 shutdown_connections 2024-04-26T11:19:00.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:00.096+0000 7fd2ec400700 1 -- 172.21.15.33:0/870742099 wait complete. 2024-04-26T11:19:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:00 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:00.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:00 smithi033 ceph-mon[27845]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:00.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:00 smithi033 ceph-mon[27845]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:19:00.764 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T11:19:00.764 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T11:19:01.238 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:19:02.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:02.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:02.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:02.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:02.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:02.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:19:02.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:02.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:03.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.169+0000 7f2e3e600700 1 -- 172.21.15.33:0/1152991745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 msgr2=0x7f2e38103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:03.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.169+0000 7f2e3e600700 1 --2- 172.21.15.33:0/1152991745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e38103470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009910 tx=0x7f2e28009c20 comp rx=0 tx=0).stop 2024-04-26T11:19:03.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.170+0000 7f2e3e600700 1 -- 172.21.15.33:0/1152991745 shutdown_connections 2024-04-26T11:19:03.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.170+0000 7f2e3e600700 1 --2- 172.21.15.33:0/1152991745 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e38103470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:03.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.170+0000 7f2e3e600700 1 -- 172.21.15.33:0/1152991745 >> 172.21.15.33:0/1152991745 conn(0x7f2e380fe960 msgr2=0x7f2e38100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:03.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.170+0000 7f2e3e600700 1 -- 172.21.15.33:0/1152991745 shutdown_connections 2024-04-26T11:19:03.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.170+0000 7f2e3e600700 1 -- 172.21.15.33:0/1152991745 wait complete. 2024-04-26T11:19:03.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.171+0000 7f2e3e600700 1 Processor -- start 2024-04-26T11:19:03.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.171+0000 7f2e3e600700 1 -- start start 2024-04-26T11:19:03.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.172+0000 7f2e3e600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e3819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:03.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.172+0000 7f2e3e600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3810d310 con 0x7f2e38103090 2024-04-26T11:19:03.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.172+0000 7f2e37400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e3819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:03.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.172+0000 7f2e37400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e3819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56726/0 (socket says 172.21.15.33:56726) 2024-04-26T11:19:03.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.172+0000 7f2e37400700 1 -- 172.21.15.33:0/409980356 learned_addr learned my addr 172.21.15.33:0/409980356 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:19:03.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.173+0000 7f2e37400700 1 -- 172.21.15.33:0/409980356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3819eb00 con 0x7f2e38103090 2024-04-26T11:19:03.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.173+0000 7f2e37400700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e3819e5c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e28006010 tx=0x7f2e28005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:03.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.173+0000 7f2e2fe00700 1 -- 172.21.15.33:0/409980356 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e28026070 con 0x7f2e38103090 2024-04-26T11:19:03.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.173+0000 7f2e2fe00700 1 -- 172.21.15.33:0/409980356 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e2800bba0 con 0x7f2e38103090 2024-04-26T11:19:03.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.173+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e3819edf0 con 0x7f2e38103090 2024-04-26T11:19:03.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.174+0000 7f2e2fe00700 1 -- 172.21.15.33:0/409980356 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e28003910 con 0x7f2e38103090 2024-04-26T11:19:03.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.174+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e38198860 con 0x7f2e38103090 2024-04-26T11:19:03.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.175+0000 7f2e2fe00700 1 -- 172.21.15.33:0/409980356 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2e28029070 con 0x7f2e38103090 2024-04-26T11:19:03.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.175+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e38104570 con 0x7f2e38103090 2024-04-26T11:19:03.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.175+0000 7f2e2fe00700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2e1c0396b0 0x7f2e1c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:03.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.175+0000 7f2e2fe00700 1 -- 172.21.15.33:0/409980356 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2e2801e070 con 0x7f2e38103090 2024-04-26T11:19:03.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.176+0000 7f2e36a00700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2e1c0396b0 0x7f2e1c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:03.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.176+0000 7f2e36a00700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2e1c0396b0 0x7f2e1c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006fd0 tx=0x7f2e24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:03.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.181+0000 7f2e2fe00700 1 -- 172.21.15.33:0/409980356 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e28013070 con 0x7f2e38103090 2024-04-26T11:19:03.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.418+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2e38199bc0 con 0x7f2e38103090 2024-04-26T11:19:03.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.746+0000 7f2e2fe00700 1 -- 172.21.15.33:0/409980356 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f2e2800f840 con 0x7f2e38103090 2024-04-26T11:19:03.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2e1c0396b0 msgr2=0x7f2e1c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:03.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2e1c0396b0 0x7f2e1c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006fd0 tx=0x7f2e24006e40 comp rx=0 tx=0).stop 2024-04-26T11:19:03.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 msgr2=0x7f2e3819e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:03.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e3819e5c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e28006010 tx=0x7f2e28005780 comp rx=0 tx=0).stop 2024-04-26T11:19:03.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 shutdown_connections 2024-04-26T11:19:03.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2e1c0396b0 0x7f2e1c03bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:03.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 --2- 172.21.15.33:0/409980356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2e38103090 0x7f2e3819e5c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:03.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.750+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 >> 172.21.15.33:0/409980356 conn(0x7f2e380fe960 msgr2=0x7f2e38104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:03.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.751+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 shutdown_connections 2024-04-26T11:19:03.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:03.751+0000 7f2e3e600700 1 -- 172.21.15.33:0/409980356 wait complete. 2024-04-26T11:19:03.754 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2024-04-26T11:19:04.092 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:03 smithi033 ceph-mon[27845]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:19:04.092 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:03 smithi033 ceph-mon[27845]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:04.092 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:03 smithi033 ceph-mon[27845]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:04.092 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:03 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/409980356' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T11:19:04.485 INFO:tasks.cephadm:Adding mon.a on smithi033 2024-04-26T11:19:04.485 INFO:tasks.cephadm:Adding mon.b on smithi096 2024-04-26T11:19:04.486 INFO:tasks.cephadm:Adding mon.c on smithi194 2024-04-26T11:19:04.486 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c' 2024-04-26T11:19:04.723 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:04 smithi033 ceph-mon[27845]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:19:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:04 smithi033 ceph-mon[27845]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:04 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/409980356' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T11:19:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:04 smithi033 ceph-mon[27845]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:06.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.099+0000 7f4d63cbd700 1 -- 172.21.15.194:0/1209209456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 msgr2=0x7f4d5c109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.099+0000 7f4d63cbd700 1 --2- 172.21.15.194:0/1209209456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c109220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800b0a0 tx=0x7f4d5800b3b0 comp rx=0 tx=0).stop 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.099+0000 7f4d63cbd700 1 -- 172.21.15.194:0/1209209456 shutdown_connections 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.099+0000 7f4d63cbd700 1 --2- 172.21.15.194:0/1209209456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c109220 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.099+0000 7f4d63cbd700 1 -- 172.21.15.194:0/1209209456 >> 172.21.15.194:0/1209209456 conn(0x7f4d5c0792b0 msgr2=0x7f4d5c0796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.099+0000 7f4d63cbd700 1 -- 172.21.15.194:0/1209209456 shutdown_connections 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.099+0000 7f4d63cbd700 1 -- 172.21.15.194:0/1209209456 wait complete. 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.100+0000 7f4d63cbd700 1 Processor -- start 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.100+0000 7f4d63cbd700 1 -- start start 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.100+0000 7f4d63cbd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:06.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.100+0000 7f4d63cbd700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c10d310 con 0x7f4d5c108e40 2024-04-26T11:19:06.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.101+0000 7f4d61a59700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:06.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.101+0000 7f4d61a59700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.194:56860/0 (socket says 172.21.15.194:56860) 2024-04-26T11:19:06.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.101+0000 7f4d61a59700 1 -- 172.21.15.194:0/2819481969 learned_addr learned my addr 172.21.15.194:0/2819481969 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:19:06.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.101+0000 7f4d61a59700 1 -- 172.21.15.194:0/2819481969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5c19eae0 con 0x7f4d5c108e40 2024-04-26T11:19:06.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.102+0000 7f4d61a59700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c19e5a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d58006130 tx=0x7f4d580097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:06.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.102+0000 7f4d52ffd700 1 -- 172.21.15.194:0/2819481969 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d58021020 con 0x7f4d5c108e40 2024-04-26T11:19:06.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.102+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5c19ed70 con 0x7f4d5c108e40 2024-04-26T11:19:06.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.102+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5c198670 con 0x7f4d5c108e40 2024-04-26T11:19:06.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.102+0000 7f4d52ffd700 1 -- 172.21.15.194:0/2819481969 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d580079b0 con 0x7f4d5c108e40 2024-04-26T11:19:06.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.102+0000 7f4d52ffd700 1 -- 172.21.15.194:0/2819481969 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d58017040 con 0x7f4d5c108e40 2024-04-26T11:19:06.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.103+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d5c06b000 con 0x7f4d5c108e40 2024-04-26T11:19:06.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.104+0000 7f4d52ffd700 1 -- 172.21.15.194:0/2819481969 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4d58024020 con 0x7f4d5c108e40 2024-04-26T11:19:06.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.104+0000 7f4d52ffd700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4d480396b0 0x7f4d4803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:06.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.104+0000 7f4d52ffd700 1 -- 172.21.15.194:0/2819481969 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4d5800f050 con 0x7f4d5c108e40 2024-04-26T11:19:06.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.105+0000 7f4d61258700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4d480396b0 0x7f4d4803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:06.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.105+0000 7f4d61258700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4d480396b0 0x7f4d4803bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c006fd0 tx=0x7f4d4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:06.122 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.121+0000 7f4d52ffd700 1 -- 172.21.15.194:0/2819481969 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d5802b020 con 0x7f4d5c108e40 2024-04-26T11:19:06.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.327+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}) -- 0x7f4d5c199ba0 con 0x7f4d480396b0 2024-04-26T11:19:06.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.342+0000 7f4d52ffd700 1 -- 172.21.15.194:0/2819481969 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4d5c199ba0 con 0x7f4d480396b0 2024-04-26T11:19:06.344 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-04-26T11:19:06.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4d480396b0 msgr2=0x7f4d4803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:06.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4d480396b0 0x7f4d4803bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c006fd0 tx=0x7f4d4c006e40 comp rx=0 tx=0).stop 2024-04-26T11:19:06.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 msgr2=0x7f4d5c19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:06.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c19e5a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d58006130 tx=0x7f4d580097d0 comp rx=0 tx=0).stop 2024-04-26T11:19:06.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 shutdown_connections 2024-04-26T11:19:06.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4d480396b0 0x7f4d4803bb70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:06.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 --2- 172.21.15.194:0/2819481969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d5c108e40 0x7f4d5c19e5a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:06.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 >> 172.21.15.194:0/2819481969 conn(0x7f4d5c0792b0 msgr2=0x7f4d5c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:06.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 shutdown_connections 2024-04-26T11:19:06.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:06.345+0000 7f4d63cbd700 1 -- 172.21.15.194:0/2819481969 wait complete. 2024-04-26T11:19:07.076 DEBUG:teuthology.orchestra.run.smithi096:mon.b> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T11:19:07.079 DEBUG:teuthology.orchestra.run.smithi194:mon.c> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T11:19:07.081 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T11:19:07.081 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: Saving service mon spec with placement smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c;count:3 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:07 smithi033 ceph-mon[27845]: Deploying daemon mon.c on smithi194 2024-04-26T11:19:08.092 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:09 smithi033 ceph-mon[27845]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:10.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 -- 172.21.15.194:0/1711258580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 msgr2=0x7f227c102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:10.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 --2- 172.21.15.194:0/1711258580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c102350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f226c009910 tx=0x7f226c009c20 comp rx=0 tx=0).stop 2024-04-26T11:19:10.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 -- 172.21.15.194:0/1711258580 shutdown_connections 2024-04-26T11:19:10.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 --2- 172.21.15.194:0/1711258580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c102350 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:10.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 -- 172.21.15.194:0/1711258580 >> 172.21.15.194:0/1711258580 conn(0x7f227c0fd820 msgr2=0x7f227c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:10.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 -- 172.21.15.194:0/1711258580 shutdown_connections 2024-04-26T11:19:10.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 -- 172.21.15.194:0/1711258580 wait complete. 2024-04-26T11:19:10.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 Processor -- start 2024-04-26T11:19:10.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.658+0000 7f228275e700 1 -- start start 2024-04-26T11:19:10.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.659+0000 7f228275e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:10.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.659+0000 7f228275e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f227c10c1f0 con 0x7f227c101f70 2024-04-26T11:19:10.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.659+0000 7f227bfff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:10.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.659+0000 7f227bfff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.194:56866/0 (socket says 172.21.15.194:56866) 2024-04-26T11:19:10.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.659+0000 7f227bfff700 1 -- 172.21.15.194:0/2800387328 learned_addr learned my addr 172.21.15.194:0/2800387328 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:19:10.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.660+0000 7f227bfff700 1 -- 172.21.15.194:0/2800387328 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c19eb30 con 0x7f227c101f70 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.660+0000 7f227bfff700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c19e5f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f226c0056d0 tx=0x7f226c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.661+0000 7f22797fa700 1 -- 172.21.15.194:0/2800387328 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f226c026070 con 0x7f227c101f70 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.661+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c19ee20 con 0x7f227c101f70 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.661+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c198890 con 0x7f227c101f70 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.661+0000 7f22797fa700 1 -- 172.21.15.194:0/2800387328 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f226c00bba0 con 0x7f227c101f70 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.661+0000 7f22797fa700 1 -- 172.21.15.194:0/2800387328 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f226c030060 con 0x7f227c101f70 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.662+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227c103450 con 0x7f227c101f70 2024-04-26T11:19:10.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.662+0000 7f22797fa700 1 -- 172.21.15.194:0/2800387328 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f226c029070 con 0x7f227c101f70 2024-04-26T11:19:10.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.663+0000 7f22797fa700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2264039650 0x7f226403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:10.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.663+0000 7f22797fa700 1 -- 172.21.15.194:0/2800387328 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f226c01e070 con 0x7f227c101f70 2024-04-26T11:19:10.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.663+0000 7f227b7fe700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2264039650 0x7f226403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:10.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.664+0000 7f227b7fe700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2264039650 0x7f226403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2270006fd0 tx=0x7f2270006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:10.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.677+0000 7f22797fa700 1 -- 172.21.15.194:0/2800387328 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f226c013070 con 0x7f227c101f70 2024-04-26T11:19:10.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.964+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f227c0008d0 con 0x7f227c101f70 2024-04-26T11:19:10.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.965+0000 7f22797fa700 1 -- 172.21.15.194:0/2800387328 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f226c004390 con 0x7f227c101f70 2024-04-26T11:19:10.967 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:19:10.967 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","modified":"2024-04-26T11:17:04.749857Z","created":"2024-04-26T11:17:04.749857Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T11:19:10.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2264039650 msgr2=0x7f226403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:10.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2264039650 0x7f226403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2270006fd0 tx=0x7f2270006e40 comp rx=0 tx=0).stop 2024-04-26T11:19:10.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 msgr2=0x7f227c19e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:10.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c19e5f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f226c0056d0 tx=0x7f226c0057b0 comp rx=0 tx=0).stop 2024-04-26T11:19:10.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 shutdown_connections 2024-04-26T11:19:10.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2264039650 0x7f226403bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:10.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 --2- 172.21.15.194:0/2800387328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f227c101f70 0x7f227c19e5f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:10.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 >> 172.21.15.194:0/2800387328 conn(0x7f227c0fd820 msgr2=0x7f227c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:10.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 shutdown_connections 2024-04-26T11:19:10.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:10.968+0000 7f228275e700 1 -- 172.21.15.194:0/2800387328 wait complete. 2024-04-26T11:19:10.970 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-04-26T11:19:11.031 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 podman[32974]: 2024-04-26 11:19:11.005416088 +0000 UTC m=+0.340781170 container start b3fd41da0315bbce5446ec1b0c3e10b3062741a767524f8e180b0eba5cfd5db3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:19:11.031 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T11:19:11.031 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev), process ceph-mon, pid 2 2024-04-26T11:19:11.031 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pidfile_write: ignore empty --pid-file 2024-04-26T11:19:11.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: load: jerasure load: lrc 2024-04-26T11:19:11.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: RocksDB version: 7.9.2 2024-04-26T11:19:11.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Git sha 0 2024-04-26T11:19:11.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Compile date 2024-04-26 07:20:26 2024-04-26T11:19:11.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: DB SUMMARY 2024-04-26T11:19:11.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: DB Session ID: ZPIBDEJX88L898LX547V 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: CURRENT file: CURRENT 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.error_if_exists: 0 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.create_if_missing: 0 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.paranoid_checks: 1 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.env: 0x55a5a9e7ccc0 2024-04-26T11:19:11.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.info_log: 0x55a5ac44fd80 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.statistics: (nil) 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.use_fsync: 0 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_log_file_size: 0 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.allow_fallocate: 1 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T11:19:11.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.use_direct_reads: 0 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.db_log_dir: 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.wal_dir: 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T11:19:11.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.write_buffer_manager: 0x55a5ac4210e0 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.unordered_write: 0 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T11:19:11.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.row_cache: None 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.wal_filter: None 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.two_write_queues: 0 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T11:19:11.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.wal_compression: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.atomic_flush: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.log_readahead_size: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_background_jobs: 2 2024-04-26T11:19:11.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_background_compactions: -1 2024-04-26T11:19:11.362 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_subcompactions: 1 2024-04-26T11:19:11.363 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T11:19:11.363 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T11:19:11.363 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T11:19:11.363 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T11:19:11.363 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_open_files: -1 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_background_flushes: -1 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Compression algorithms supported: 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kZSTD supported: 0 2024-04-26T11:19:11.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kXpressCompression supported: 0 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kZlibCompression supported: 1 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kSnappyCompression supported: 1 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kLZ4Compression supported: 1 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: kBZip2Compression supported: 0 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.merge_operator: 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_filter: None 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T11:19:11.365 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a5ac44ff40) 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: cache_index_and_filter_blocks: 1 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: pin_top_level_index_and_filter: 1 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: index_type: 0 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: data_block_index_type: 0 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: index_shortening: 1 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: checksum: 4 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: no_block_cache: 0 2024-04-26T11:19:11.366 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache: 0x55a5ac4531f0 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache_name: BinnedLRUCache 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache_options: 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: capacity : 536870912 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: num_shard_bits : 4 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: strict_capacity_limit : 0 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: high_pri_pool_ratio: 0.000 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache_compressed: (nil) 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: persistent_cache: (nil) 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_size: 4096 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_size_deviation: 10 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_restart_interval: 16 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: index_block_restart_interval: 1 2024-04-26T11:19:11.367 INFO:journalctl@ceph.mon.c.smithi194.stdout: metadata_block_size: 4096 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: partition_filters: 0 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: use_delta_encoding: 1 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: filter_policy: bloomfilter 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: whole_key_filtering: 1 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: verify_compression: 0 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: read_amp_bytes_per_bit: 0 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: format_version: 5 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: enable_index_compression: 1 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_align: 0 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: max_auto_readahead_size: 262144 2024-04-26T11:19:11.368 INFO:journalctl@ceph.mon.c.smithi194.stdout: prepopulate_block_cache: 0 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout: initial_auto_readahead_size: 8192 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression: NoCompression 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.num_levels: 7 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T11:19:11.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T11:19:11.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T11:19:11.371 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T11:19:11.372 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T11:19:11.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.inplace_update_support: 0 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.bloom_locality: 0 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.max_successive_merges: 0 2024-04-26T11:19:11.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.ttl: 2592000 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.enable_blob_files: false 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.min_blob_size: 0 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T11:19:11.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d903e93a-2af7-4c48-9825-6e936e3d70e3 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130351035544, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130351035951, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714130351, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d903e93a-2af7-4c48-9825-6e936e3d70e3", "db_session_id": "ZPIBDEJX88L898LX547V", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130351036037, "job": 1, "event": "recovery_finished"} 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T11:19:11.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a5ac570000 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: DB pointer 0x55a5ac55c000 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** DB Stats ** 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-26T11:19:11.377 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** Compaction Stats [default] ** 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** Compaction Stats [default] ** 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-26T11:19:11.378 INFO:journalctl@ceph.mon.c.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: Block cache BinnedLRUCache@0x55a5ac4531f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T11:19:11.379 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: using public_addr v2:172.21.15.194:0/0 -> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: starting mon.c rank -1 at public addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] at bind addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(???) e0 preinit fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).mds e1 new map 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout: e1 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout: legacy client fscid: -1 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout: No filesystems configured 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T11:19:11.380 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Added host smithi096 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Deploying cephadm binary to smithi194 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.383 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Added host smithi194 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/409980356' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/409980356' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Saving service mon spec with placement smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c;count:3 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: Deploying daemon mon.c on smithi194 2024-04-26T11:19:11.386 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:11.387 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 ceph-mon[33017]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T11:19:11.387 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 bash[32974]: b3fd41da0315bbce5446ec1b0c3e10b3062741a767524f8e180b0eba5cfd5db3 2024-04-26T11:19:11.387 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:11 smithi194 systemd[1]: Started Ceph mon.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:19:12.676 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T11:19:12.677 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T11:19:12.920 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:19:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: mon.a calling monitor election 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: mon.c calling monitor election 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: monmap epoch 2 2024-04-26T11:19:16.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: last_changed 2024-04-26T11:19:11.068914+0000 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: min_mon_release 19 (squid) 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: election_strategy: 1 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: fsmap 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: mgrmap e13: x(active, since 58s) 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: overall HEALTH_OK 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:16.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: mon.a calling monitor election 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: mon.c calling monitor election 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T11:19:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: monmap epoch 2 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: last_changed 2024-04-26T11:19:11.068914+0000 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: min_mon_release 19 (squid) 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: election_strategy: 1 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: fsmap 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: mgrmap e13: x(active, since 58s) 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: overall HEALTH_OK 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:16.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:17.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.249+0000 7f439814f700 1 -- 172.21.15.194:0/325272816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c0035c0 msgr2=0x7f437c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:17.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.249+0000 7f439814f700 1 --2- 172.21.15.194:0/325272816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c0035c0 0x7f437c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4390102e70 tx=0x7f438c00feb0 comp rx=0 tx=0).stop 2024-04-26T11:19:17.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.249+0000 7f439814f700 1 -- 172.21.15.194:0/325272816 shutdown_connections 2024-04-26T11:19:17.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.249+0000 7f439814f700 1 --2- 172.21.15.194:0/325272816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c0035c0 0x7f437c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:17.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.249+0000 7f439814f700 1 -- 172.21.15.194:0/325272816 >> 172.21.15.194:0/325272816 conn(0x7f43900fea30 msgr2=0x7f4390100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:17.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.249+0000 7f439814f700 1 -- 172.21.15.194:0/325272816 shutdown_connections 2024-04-26T11:19:17.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.249+0000 7f439814f700 1 -- 172.21.15.194:0/325272816 wait complete. 2024-04-26T11:19:17.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.250+0000 7f439814f700 1 Processor -- start 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.250+0000 7f439814f700 1 -- start start 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.250+0000 7f439814f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 0x7f43901a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.250+0000 7f439814f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43901a0d50 0x7f439019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.250+0000 7f439814f700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f439010ae90 con 0x7f43901a0d50 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.250+0000 7f439814f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f439010ad10 con 0x7f437c0035c0 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f4395eeb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 0x7f43901a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f4395eeb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 0x7f43901a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:59688/0 (socket says 172.21.15.194:59688) 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f4395eeb700 1 -- 172.21.15.194:0/2532657255 learned_addr learned my addr 172.21.15.194:0/2532657255 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f4395eeb700 1 -- 172.21.15.194:0/2532657255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 msgr2=0x7f43901a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f4395eeb700 1 -- 172.21.15.194:0/2532657255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 msgr2=0x7f43901a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f4395eeb700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 0x7f43901a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f4395eeb700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 0x7f43901a0810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:19:17.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.251+0000 7f43956ea700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43901a0d50 0x7f439019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:17.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.252+0000 7f43956ea700 1 -- 172.21.15.194:0/2532657255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 msgr2=0x7f43901a0810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:19:17.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.252+0000 7f43956ea700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 0x7f43901a0810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.252+0000 7f43956ea700 1 -- 172.21.15.194:0/2532657255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43901a1390 con 0x7f43901a0d50 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.252+0000 7f43956ea700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43901a0d50 0x7f439019a9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f438000b4d0 tx=0x7f438000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.253+0000 7f4386ffd700 1 -- 172.21.15.194:0/2532657255 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4380011630 con 0x7f43901a0d50 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.253+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439019af10 con 0x7f43901a0d50 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.253+0000 7f4386ffd700 1 -- 172.21.15.194:0/2532657255 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4380011c80 con 0x7f43901a0d50 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.253+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f439019b4a0 con 0x7f43901a0d50 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.253+0000 7f4386ffd700 1 -- 172.21.15.194:0/2532657255 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4380010da0 con 0x7f43901a0d50 2024-04-26T11:19:17.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.254+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f439019b810 con 0x7f43901a0d50 2024-04-26T11:19:17.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.254+0000 7f4386ffd700 1 -- 172.21.15.194:0/2532657255 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4380018020 con 0x7f43901a0d50 2024-04-26T11:19:17.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.255+0000 7f4386ffd700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4374039440 0x7f437403b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:17.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.255+0000 7f4386ffd700 1 -- 172.21.15.194:0/2532657255 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f438004c6d0 con 0x7f43901a0d50 2024-04-26T11:19:17.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.255+0000 7f4395eeb700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4374039440 0x7f437403b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:17.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.256+0000 7f4395eeb700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4374039440 0x7f437403b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f438c00fa60 tx=0x7f438c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:17.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.270+0000 7f4386ffd700 1 -- 172.21.15.194:0/2532657255 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f438001f020 con 0x7f43901a0d50 2024-04-26T11:19:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:17 smithi033 ceph-mon[27845]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:17 smithi033 ceph-mon[27845]: Deploying daemon mon.b on smithi096 2024-04-26T11:19:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:17.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.553+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f43901a56d0 con 0x7f43901a0d50 2024-04-26T11:19:17.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:17 smithi194 ceph-mon[33017]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:17.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:17.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:17.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:17.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:17 smithi194 ceph-mon[33017]: Deploying daemon mon.b on smithi096 2024-04-26T11:19:17.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:17.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.554+0000 7f4386ffd700 1 -- 172.21.15.194:0/2532657255 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f4380010410 con 0x7f43901a0d50 2024-04-26T11:19:17.556 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:19:17.556 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","modified":"2024-04-26T11:19:11.068914Z","created":"2024-04-26T11:17:04.749857Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T11:19:17.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4374039440 msgr2=0x7f437403b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:17.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4374039440 0x7f437403b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f438c00fa60 tx=0x7f438c005960 comp rx=0 tx=0).stop 2024-04-26T11:19:17.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43901a0d50 msgr2=0x7f439019a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43901a0d50 0x7f439019a9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f438000b4d0 tx=0x7f438000b890 comp rx=0 tx=0).stop 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 shutdown_connections 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4374039440 0x7f437403b900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f437c0035c0 0x7f43901a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 --2- 172.21.15.194:0/2532657255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43901a0d50 0x7f439019a9d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 >> 172.21.15.194:0/2532657255 conn(0x7f43900fea30 msgr2=0x7f4390104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 shutdown_connections 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:17.556+0000 7f439814f700 1 -- 172.21.15.194:0/2532657255 wait complete. 2024-04-26T11:19:17.558 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-04-26T11:19:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:18 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/2532657255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T11:19:18.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:19:18 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:19:18.069+0000 7f6fd7600700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T11:19:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:18 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/2532657255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T11:19:19.183 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T11:19:19.183 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T11:19:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:19 smithi033 ceph-mon[27845]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:19.427 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:19:19.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:19 smithi194 ceph-mon[33017]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:19 smithi096 podman[32734]: 2024-04-26 11:19:19.767254576 +0000 UTC m=+0.018222632 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:19:20.379 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 podman[32734]: 2024-04-26 11:19:20.035329945 +0000 UTC m=+0.286297990 container init fa293eb99080116c890a42e19007fbaeefb7e3173c354d45925ff85324b18a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 podman[32734]: 2024-04-26 11:19:20.038258664 +0000 UTC m=+0.289226710 container start fa293eb99080116c890a42e19007fbaeefb7e3173c354d45925ff85324b18a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev), process ceph-mon, pid 2 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pidfile_write: ignore empty --pid-file 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: load: jerasure load: lrc 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: RocksDB version: 7.9.2 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Git sha 0 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Compile date 2024-04-26 07:20:26 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: DB SUMMARY 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: DB Session ID: ALUNDULTAA8YNE21SPRZ 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: CURRENT file: CURRENT 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T11:19:20.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.error_if_exists: 0 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.create_if_missing: 0 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.paranoid_checks: 1 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.env: 0x55af04d81cc0 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.info_log: 0x55af05d85d80 2024-04-26T11:19:20.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T11:19:20.382 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.statistics: (nil) 2024-04-26T11:19:20.382 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.use_fsync: 0 2024-04-26T11:19:20.382 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_log_file_size: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.allow_fallocate: 1 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.use_direct_reads: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.db_log_dir: 2024-04-26T11:19:20.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.wal_dir: 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.write_buffer_manager: 0x55af05d570e0 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T11:19:20.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.unordered_write: 0 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T11:19:20.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.row_cache: None 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.wal_filter: None 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.two_write_queues: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.wal_compression: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.atomic_flush: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T11:19:20.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.log_readahead_size: 0 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_background_jobs: 2 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_background_compactions: -1 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_subcompactions: 1 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T11:19:20.387 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_open_files: -1 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_background_flushes: -1 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Compression algorithms supported: 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kZSTD supported: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kXpressCompression supported: 0 2024-04-26T11:19:20.388 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kZlibCompression supported: 1 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kSnappyCompression supported: 1 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kLZ4Compression supported: 1 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: kBZip2Compression supported: 0 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.merge_operator: 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_filter: None 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T11:19:20.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55af05d85f40) 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: cache_index_and_filter_blocks: 1 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: pin_top_level_index_and_filter: 1 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_type: 0 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: data_block_index_type: 0 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_shortening: 1 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: checksum: 4 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: no_block_cache: 0 2024-04-26T11:19:20.390 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache: 0x55af05d891f0 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_name: BinnedLRUCache 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_options: 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: capacity : 536870912 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: num_shard_bits : 4 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: strict_capacity_limit : 0 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: high_pri_pool_ratio: 0.000 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_compressed: (nil) 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: persistent_cache: (nil) 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_size: 4096 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_size_deviation: 10 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_restart_interval: 16 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_block_restart_interval: 1 2024-04-26T11:19:20.391 INFO:journalctl@ceph.mon.b.smithi096.stdout: metadata_block_size: 4096 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: partition_filters: 0 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: use_delta_encoding: 1 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: filter_policy: bloomfilter 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: whole_key_filtering: 1 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: verify_compression: 0 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: read_amp_bytes_per_bit: 0 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: format_version: 5 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: enable_index_compression: 1 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_align: 0 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: max_auto_readahead_size: 262144 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: prepopulate_block_cache: 0 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: initial_auto_readahead_size: 8192 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T11:19:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression: NoCompression 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.num_levels: 7 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T11:19:20.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T11:19:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T11:19:20.395 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T11:19:20.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.inplace_update_support: 0 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.bloom_locality: 0 2024-04-26T11:19:20.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.max_successive_merges: 0 2024-04-26T11:19:20.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T11:19:20.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T11:19:20.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T11:19:20.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T11:19:20.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.ttl: 2592000 2024-04-26T11:19:20.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T11:19:20.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T11:19:20.399 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T11:19:20.399 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.enable_blob_files: false 2024-04-26T11:19:20.399 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.min_blob_size: 0 2024-04-26T11:19:20.399 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T11:19:20.399 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 33a55d1f-5369-4d30-9448-88d21e7e967d 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130360067838, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130360068245, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714130360, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "33a55d1f-5369-4d30-9448-88d21e7e967d", "db_session_id": "ALUNDULTAA8YNE21SPRZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714130360068323, "job": 1, "event": "recovery_finished"} 2024-04-26T11:19:20.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55af05ea6000 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: DB pointer 0x55af05e92000 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** DB Stats ** 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T11:19:20.401 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-26T11:19:20.402 INFO:journalctl@ceph.mon.b.smithi096.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: Block cache BinnedLRUCache@0x55af05d891f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-26T11:19:20.403 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: using public_addr v2:172.21.15.96:0/0 -> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: starting mon.b rank -1 at public addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] at bind addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(???) e0 preinit fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).mds e1 new map 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout: e1 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout: legacy client fscid: -1 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout: No filesystems configured 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T11:19:20.404 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Added host smithi096 2024-04-26T11:19:20.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:20.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Deploying cephadm binary to smithi194 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:20.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Added host smithi194 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:19:20.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/409980356' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/409980356' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Saving service mon spec with placement smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c;count:3 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Deploying daemon mon.c on smithi194 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.a calling monitor election 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:20.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.c calling monitor election 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: monmap epoch 2 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: last_changed 2024-04-26T11:19:11.068914+0000 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:20.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: min_mon_release 19 (squid) 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: election_strategy: 1 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: fsmap 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mgrmap e13: x(active, since 58s) 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:20.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: Deploying daemon mon.b on smithi096 2024-04-26T11:19:20.413 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:20.413 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/2532657255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T11:19:20.413 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:20.413 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 ceph-mon[32753]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T11:19:20.413 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 bash[32734]: fa293eb99080116c890a42e19007fbaeefb7e3173c354d45925ff85324b18a9d 2024-04-26T11:19:20.413 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:20 smithi096 systemd[1]: Started Ceph mon.b for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.392+0000 7f777de74700 1 -- 172.21.15.194:0/2194724611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77600035c0 msgr2=0x7f7760005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.392+0000 7f777de74700 1 --2- 172.21.15.194:0/2194724611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77600035c0 0x7f7760005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7778102e40 tx=0x7f776800fe60 comp rx=0 tx=0).stop 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.392+0000 7f777de74700 1 -- 172.21.15.194:0/2194724611 shutdown_connections 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.392+0000 7f777de74700 1 --2- 172.21.15.194:0/2194724611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77600035c0 0x7f7760005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.392+0000 7f777de74700 1 -- 172.21.15.194:0/2194724611 >> 172.21.15.194:0/2194724611 conn(0x7f77780fea20 msgr2=0x7f7778100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.392+0000 7f777de74700 1 -- 172.21.15.194:0/2194724611 shutdown_connections 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.392+0000 7f777de74700 1 -- 172.21.15.194:0/2194724611 wait complete. 2024-04-26T11:19:25.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 Processor -- start 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 -- start start 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 0x7f77781a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77781a0e70 0x7f777819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 0x7f777819b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f777810ae00 con 0x7f77781a0e70 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f777810ac80 con 0x7f77600035c0 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.393+0000 7f777de74700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f777810af80 con 0x7f777819b030 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f77777fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 0x7f77781a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f7777fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 0x7f777819b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:25.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f77777fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 0x7f77781a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.194:43472/0 (socket says 172.21.15.194:43472) 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f77777fe700 1 -- 172.21.15.194:0/3247780017 learned_addr learned my addr 172.21.15.194:0/3247780017 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f77777fe700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 msgr2=0x7f77781a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f77777fe700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 msgr2=0x7f77781a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f77777fe700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 0x7f77781a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f7777fff700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 msgr2=0x7f777819b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f7777fff700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 msgr2=0x7f777819b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f7777fff700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 0x7f777819b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f77777fe700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 0x7f77781a0930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f7777fff700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 0x7f777819b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.394+0000 7f7776ffd700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77781a0e70 0x7f777819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7776ffd700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 msgr2=0x7f777819b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7776ffd700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 0x7f777819b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7776ffd700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 msgr2=0x7f77781a0930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:19:25.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7776ffd700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 0x7f77781a0930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7776ffd700 1 -- 172.21.15.194:0/3247780017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77781a7800 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7776ffd700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77781a0e70 0x7f777819aaf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f776c00b550 tx=0x7f776c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7774ff9700 1 -- 172.21.15.194:0/3247780017 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c004230 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77781a7a30 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7774ff9700 1 -- 172.21.15.194:0/3247780017 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f776c0043d0 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f7774ff9700 1 -- 172.21.15.194:0/3247780017 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c011b30 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.395+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77781a7fc0 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.397+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7778064a80 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.397+0000 7f7774ff9700 1 -- 172.21.15.194:0/3247780017 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f776c0036a0 con 0x7f77781a0e70 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.397+0000 7f7774ff9700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7764039ae0 0x7f776403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:25.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.397+0000 7f7774ff9700 1 -- 172.21.15.194:0/3247780017 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f776c04c310 con 0x7f77781a0e70 2024-04-26T11:19:25.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.398+0000 7f77777fe700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7764039ae0 0x7f776403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:25.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.399+0000 7f77777fe700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7764039ae0 0x7f776403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7768005d10 tx=0x7f7768005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:25.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.413+0000 7f7774ff9700 1 -- 172.21.15.194:0/3247780017 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f776c020b20 con 0x7f77781a0e70 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: mon.c calling monitor election 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: mon.a calling monitor election 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: monmap epoch 3 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: last_changed 2024-04-26T11:19:20.097857+0000 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: min_mon_release 19 (squid) 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: election_strategy: 1 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-26T11:19:25.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: fsmap 2024-04-26T11:19:25.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:25.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: mgrmap e13: x(active, since 67s) 2024-04-26T11:19:25.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: overall HEALTH_OK 2024-04-26T11:19:25.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:25.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: mon.c calling monitor election 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: mon.a calling monitor election 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:25.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: monmap epoch 3 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: last_changed 2024-04-26T11:19:20.097857+0000 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: min_mon_release 19 (squid) 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: election_strategy: 1 2024-04-26T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: fsmap 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: mgrmap e13: x(active, since 67s) 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: overall HEALTH_OK 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:25.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:25.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.708+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f777802e650 con 0x7f77781a0e70 2024-04-26T11:19:25.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.709+0000 7f7774ff9700 1 -- 172.21.15.194:0/3247780017 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f776c010380 con 0x7f77781a0e70 2024-04-26T11:19:25.711 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T11:19:25.711 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","modified":"2024-04-26T11:19:20.097857Z","created":"2024-04-26T11:17:04.749857Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T11:19:25.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.712+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7764039ae0 msgr2=0x7f776403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:25.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.712+0000 7f777de74700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7764039ae0 0x7f776403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7768005d10 tx=0x7f7768005ca0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.712+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77781a0e70 msgr2=0x7f777819aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:25.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.712+0000 7f777de74700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77781a0e70 0x7f777819aaf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f776c00b550 tx=0x7f776c00b910 comp rx=0 tx=0).stop 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.712+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 shutdown_connections 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.712+0000 7f777de74700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f7764039ae0 0x7f776403bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.713+0000 7f777de74700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f777819b030 0x7f777819b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.713+0000 7f777de74700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77600035c0 0x7f77781a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.713+0000 7f777de74700 1 --2- 172.21.15.194:0/3247780017 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77781a0e70 0x7f777819aaf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.713+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 >> 172.21.15.194:0/3247780017 conn(0x7f77780fea20 msgr2=0x7f77780ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.713+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 shutdown_connections 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:25.713+0000 7f777de74700 1 -- 172.21.15.194:0/3247780017 wait complete. 2024-04-26T11:19:25.715 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-04-26T11:19:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:26 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:26.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:26 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:26.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:26 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:26.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:26 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/3247780017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T11:19:26.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:26 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:26.390 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T11:19:26.390 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T11:19:26.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:26 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:26.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:26 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:26.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:26 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:26.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:26 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/3247780017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T11:19:26.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:26 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:19:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.c calling monitor election 2024-04-26T11:19:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.a calling monitor election 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: monmap epoch 3 2024-04-26T11:19:27.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: last_changed 2024-04-26T11:19:20.097857+0000 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: min_mon_release 19 (squid) 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: election_strategy: 1 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: fsmap 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mgrmap e13: x(active, since 67s) 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:27.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:27.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:27.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:27.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:27.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/3247780017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T11:19:27.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: mon.b calling monitor election 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: mon.b calling monitor election 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: mon.c calling monitor election 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: mon.a calling monitor election 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: monmap epoch 3 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: last_changed 2024-04-26T11:19:20.097857+0000 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: min_mon_release 19 (squid) 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: election_strategy: 1 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: fsmap 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: mgrmap e13: x(active, since 69s) 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: overall HEALTH_OK 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:19:28.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:27 smithi194 ceph-mon[33017]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: mon.b calling monitor election 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: mon.b calling monitor election 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: mon.c calling monitor election 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: mon.a calling monitor election 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: monmap epoch 3 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:28.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: last_changed 2024-04-26T11:19:20.097857+0000 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: min_mon_release 19 (squid) 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: election_strategy: 1 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: fsmap 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: mgrmap e13: x(active, since 69s) 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: overall HEALTH_OK 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:28.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:28.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-26T11:19:28.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:19:28.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:27 smithi033 ceph-mon[27845]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:19:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.b calling monitor election 2024-04-26T11:19:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.b calling monitor election 2024-04-26T11:19:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.c calling monitor election 2024-04-26T11:19:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.a calling monitor election 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: monmap epoch 3 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: last_changed 2024-04-26T11:19:20.097857+0000 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: created 2024-04-26T11:17:04.749857+0000 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: min_mon_release 19 (squid) 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: election_strategy: 1 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: fsmap 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T11:19:28.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: mgrmap e13: x(active, since 69s) 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:19:28.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:27 smithi096 ceph-mon[32753]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:19:28.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.769+0000 7fd091000700 1 -- 172.21.15.33:0/1345944395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 msgr2=0x7fd08c075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:28.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.769+0000 7fd091000700 1 --2- 172.21.15.33:0/1345944395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c075ba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00b0a0 tx=0x7fd07c00b3b0 comp rx=0 tx=0).stop 2024-04-26T11:19:28.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.769+0000 7fd091000700 1 -- 172.21.15.33:0/1345944395 shutdown_connections 2024-04-26T11:19:28.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.769+0000 7fd091000700 1 --2- 172.21.15.33:0/1345944395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c075ba0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:28.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.769+0000 7fd091000700 1 -- 172.21.15.33:0/1345944395 >> 172.21.15.33:0/1345944395 conn(0x7fd08c0fe960 msgr2=0x7fd08c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:28.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.769+0000 7fd091000700 1 -- 172.21.15.33:0/1345944395 shutdown_connections 2024-04-26T11:19:28.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.769+0000 7fd091000700 1 -- 172.21.15.33:0/1345944395 wait complete. 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.770+0000 7fd091000700 1 Processor -- start 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd091000700 1 -- start start 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd091000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd091000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd08c19f660 0x7fd08c1a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd091000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd08c1a3f30 0x7fd08c1a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd091000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c113930 con 0x7fd08c0757c0 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd091000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c1137b0 con 0x7fd08c19f660 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd091000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c113ab0 con 0x7fd08c1a3f30 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd089e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd089e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c19f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45726/0 (socket says 172.21.15.33:45726) 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd089400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd08c19f660 0x7fd08c1a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:28.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd089e00700 1 -- 172.21.15.33:0/1469997340 learned_addr learned my addr 172.21.15.33:0/1469997340 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.771+0000 7fd08a800700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd08c1a3f30 0x7fd08c1a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.772+0000 7fd089400700 1 -- 172.21.15.33:0/1469997340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd08c1a3f30 msgr2=0x7fd08c1a43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.772+0000 7fd089400700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd08c1a3f30 0x7fd08c1a43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.772+0000 7fd089400700 1 -- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 msgr2=0x7fd08c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.772+0000 7fd089400700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.772+0000 7fd089400700 1 -- 172.21.15.33:0/1469997340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08c1abca0 con 0x7fd08c19f660 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.772+0000 7fd089e00700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.772+0000 7fd089400700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd08c19f660 0x7fd08c1a39f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd07800b550 tx=0x7fd07800b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.773+0000 7fd082a00700 1 -- 172.21.15.33:0/1469997340 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078004230 con 0x7fd08c19f660 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.773+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08c1abed0 con 0x7fd08c19f660 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.773+0000 7fd082a00700 1 -- 172.21.15.33:0/1469997340 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0780043d0 con 0x7fd08c19f660 2024-04-26T11:19:28.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.773+0000 7fd082a00700 1 -- 172.21.15.33:0/1469997340 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078011a50 con 0x7fd08c19f660 2024-04-26T11:19:28.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.773+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd08c1ac380 con 0x7fd08c19f660 2024-04-26T11:19:28.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.774+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd08c064a80 con 0x7fd08c19f660 2024-04-26T11:19:28.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.774+0000 7fd082a00700 1 -- 172.21.15.33:0/1469997340 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd078011c30 con 0x7fd08c19f660 2024-04-26T11:19:28.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.775+0000 7fd082a00700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd070039ae0 0x7fd07003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:28.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.775+0000 7fd082a00700 1 -- 172.21.15.33:0/1469997340 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd07804c3a0 con 0x7fd08c19f660 2024-04-26T11:19:28.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.775+0000 7fd089e00700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd070039ae0 0x7fd07003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:28.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.776+0000 7fd089e00700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd070039ae0 0x7fd07003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009e60 tx=0x7fd07c005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:28.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:28.780+0000 7fd082a00700 1 -- 172.21.15.33:0/1469997340 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd078021e00 con 0x7fd08c19f660 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:29.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.022+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd08c1ac710 con 0x7fd08c19f660 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.022+0000 7fd082a00700 1 -- 172.21.15.33:0/1469997340 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fd07801a780 con 0x7fd08c19f660 2024-04-26T11:19:29.025 INFO:teuthology.orchestra.run.smithi033.stdout:# minimal ceph.conf for 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:29.025 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-04-26T11:19:29.025 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:19:29.025 INFO:teuthology.orchestra.run.smithi033.stdout: mon_host = [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-04-26T11:19:29.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd070039ae0 msgr2=0x7fd07003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd070039ae0 0x7fd07003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009e60 tx=0x7fd07c005040 comp rx=0 tx=0).stop 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd08c19f660 msgr2=0x7fd08c1a39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd08c19f660 0x7fd08c1a39f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd07800b550 tx=0x7fd07800b860 comp rx=0 tx=0).stop 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 shutdown_connections 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd070039ae0 0x7fd07003bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd08c1a3f30 0x7fd08c1a43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd08c0757c0 0x7fd08c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 --2- 172.21.15.33:0/1469997340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd08c19f660 0x7fd08c1a39f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 >> 172.21.15.33:0/1469997340 conn(0x7fd08c0fe960 msgr2=0x7fd08c0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 shutdown_connections 2024-04-26T11:19:29.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:29.026+0000 7fd091000700 1 -- 172.21.15.33:0/1469997340 wait complete. 2024-04-26T11:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.349 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:19:29.097+0000 7f6fd7600700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T11:19:29.685 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T11:19:29.686 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:19:29.686 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T11:19:29.759 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:19:29.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:29.870 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:19:29.870 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T11:19:29.902 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:19:29.903 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T11:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: Reconfiguring daemon mon.a on smithi033 2024-04-26T11:19:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:29 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1469997340' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:29.970 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:19:29.970 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T11:19:29.997 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:19:29.997 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:19:30.061 INFO:tasks.cephadm:Adding mgr.x on smithi033 2024-04-26T11:19:30.061 INFO:tasks.cephadm:Adding mgr.y on smithi096 2024-04-26T11:19:30.061 INFO:tasks.cephadm:Adding mgr.z on smithi194 2024-04-26T11:19:30.061 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi033=x;smithi096=y;smithi194=z' 2024-04-26T11:19:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T11:19:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: Reconfiguring daemon mon.a on smithi033 2024-04-26T11:19:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:29 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1469997340' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T11:19:30.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:30.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:30.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:30.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: Reconfiguring daemon mon.a on smithi033 2024-04-26T11:19:30.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:29 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1469997340' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:30.334 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:19:30.985 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:30 smithi033 ceph-mon[27845]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:30 smithi194 ceph-mon[33017]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:30 smithi096 ceph-mon[32753]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.785+0000 7f5532abd700 1 -- 172.21.15.194:0/697276255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c102fb0 msgr2=0x7f552c1033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.785+0000 7f5532abd700 1 --2- 172.21.15.194:0/697276255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c102fb0 0x7f552c1033d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5528009910 tx=0x7f5528009c20 comp rx=0 tx=0).stop 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.785+0000 7f5532abd700 1 -- 172.21.15.194:0/697276255 shutdown_connections 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.785+0000 7f5532abd700 1 --2- 172.21.15.194:0/697276255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c102fb0 0x7f552c1033d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.785+0000 7f5532abd700 1 -- 172.21.15.194:0/697276255 >> 172.21.15.194:0/697276255 conn(0x7f552c0fe550 msgr2=0x7f552c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.785+0000 7f5532abd700 1 -- 172.21.15.194:0/697276255 shutdown_connections 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.785+0000 7f5532abd700 1 -- 172.21.15.194:0/697276255 wait complete. 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 Processor -- start 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 -- start start 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f552c102fb0 0x7f552c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:31.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c075810 0x7f552c075c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f552c0761d0 0x7f552c1a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f552c108ef0 con 0x7f552c075810 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f552c108d70 con 0x7f552c0761d0 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5532abd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f552c109070 con 0x7f552c102fb0 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5530859700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f552c102fb0 0x7f552c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5530859700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f552c102fb0 0x7f552c0771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:37170/0 (socket says 172.21.15.194:37170) 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f5530859700 1 -- 172.21.15.194:0/1588593817 learned_addr learned my addr 172.21.15.194:0/1588593817 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.786+0000 7f553105a700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f552c0761d0 0x7f552c1a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5530859700 1 -- 172.21.15.194:0/1588593817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f552c0761d0 msgr2=0x7f552c1a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5530859700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f552c0761d0 0x7f552c1a2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5530859700 1 -- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c075810 msgr2=0x7f552c075c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5523fff700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c075810 0x7f552c075c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:31.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5530859700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c075810 0x7f552c075c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5530859700 1 -- 172.21.15.194:0/1588593817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552c1a2d20 con 0x7f552c102fb0 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f553105a700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f552c0761d0 0x7f552c1a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5530859700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f552c102fb0 0x7f552c0771c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5528005e00 tx=0x7f5528009e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5521ffb700 1 -- 172.21.15.194:0/1588593817 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528027070 con 0x7f552c102fb0 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552c1a2fb0 con 0x7f552c102fb0 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5521ffb700 1 -- 172.21.15.194:0/1588593817 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f552800be10 con 0x7f552c102fb0 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5521ffb700 1 -- 172.21.15.194:0/1588593817 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552800f460 con 0x7f552c102fb0 2024-04-26T11:19:31.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.787+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f552c1a3430 con 0x7f552c102fb0 2024-04-26T11:19:31.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.788+0000 7f5521ffb700 1 -- 172.21.15.194:0/1588593817 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f552802a070 con 0x7f552c102fb0 2024-04-26T11:19:31.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.788+0000 7f5521ffb700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5514039ae0 0x7f551403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:31.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.788+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f552c064a80 con 0x7f552c102fb0 2024-04-26T11:19:31.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.788+0000 7f5521ffb700 1 -- 172.21.15.194:0/1588593817 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f552804dbb0 con 0x7f552c102fb0 2024-04-26T11:19:31.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.789+0000 7f5523fff700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5514039ae0 0x7f551403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:31.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.790+0000 7f5523fff700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5514039ae0 0x7f551403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f551c0078e0 tx=0x7f551c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:31.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:31.794+0000 7f5521ffb700 1 -- 172.21.15.194:0/1588593817 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5528013070 con 0x7f552c102fb0 2024-04-26T11:19:32.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.002+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi194=z", "target": ["mon-mgr", ""]}) -- 0x7f552c063860 con 0x7f5514039ae0 2024-04-26T11:19:32.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.016+0000 7f5521ffb700 1 -- 172.21.15.194:0/1588593817 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f552c063860 con 0x7f5514039ae0 2024-04-26T11:19:32.018 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5514039ae0 msgr2=0x7f551403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5514039ae0 0x7f551403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f551c0078e0 tx=0x7f551c008040 comp rx=0 tx=0).stop 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f552c102fb0 msgr2=0x7f552c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f552c102fb0 0x7f552c0771c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5528005e00 tx=0x7f5528009e60 comp rx=0 tx=0).stop 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 shutdown_connections 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5514039ae0 0x7f551403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f552c0761d0 0x7f552c1a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:32.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f552c102fb0 0x7f552c0771c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:32.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 --2- 172.21.15.194:0/1588593817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f552c075810 0x7f552c075c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:32.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 >> 172.21.15.194:0/1588593817 conn(0x7f552c0fe550 msgr2=0x7f552c0ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:32.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 shutdown_connections 2024-04-26T11:19:32.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:19:32.018+0000 7f5532abd700 1 -- 172.21.15.194:0/1588593817 wait complete. 2024-04-26T11:19:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: Reconfiguring mon.b (monmap changed)... 2024-04-26T11:19:32.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:32.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:32.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:32.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: Reconfiguring daemon mon.b on smithi096 2024-04-26T11:19:32.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: Reconfiguring mon.b (monmap changed)... 2024-04-26T11:19:32.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:32.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:32.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:32.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: Reconfiguring daemon mon.b on smithi096 2024-04-26T11:19:32.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: Reconfiguring mon.b (monmap changed)... 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: Reconfiguring daemon mon.b on smithi096 2024-04-26T11:19:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:32.667 DEBUG:teuthology.orchestra.run.smithi096:mgr.y> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T11:19:32.670 DEBUG:teuthology.orchestra.run.smithi194:mgr.z> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T11:19:32.709 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T11:19:32.710 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:19:32.710 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T11:19:32.775 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T11:19:32.775 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-04-26T11:19:32.825 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T11:19:32.825 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:32.825 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-04-26T11:19:32.826 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:32.826 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:32.826 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-26 11:18:33.699274672 +0000 2024-04-26T11:19:32.826 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-26 11:13:09.178976000 +0000 2024-04-26T11:19:32.826 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-26 11:13:09.178976000 +0000 2024-04-26T11:19:32.826 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-26 11:13:09.178976000 +0000 2024-04-26T11:19:32.826 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T11:19:32.910 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-26T11:19:32.910 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-26T11:19:32.910 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000397209 s, 1.3 MB/s 2024-04-26T11:19:32.915 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T11:19:32.975 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-04-26T11:19:33.029 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-26 11:18:33.756273803 +0000 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-26 11:13:09.797966946 +0000 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-26 11:13:09.797966946 +0000 2024-04-26T11:19:33.030 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-26 11:13:09.797966946 +0000 2024-04-26T11:19:33.030 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T11:19:33.114 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-26T11:19:33.114 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-26T11:19:33.114 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00039624 s, 1.3 MB/s 2024-04-26T11:19:33.118 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T11:19:33.177 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-04-26T11:19:33.229 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-26 11:18:33.809272995 +0000 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-26 11:13:10.429957702 +0000 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-26 11:13:10.429957702 +0000 2024-04-26T11:19:33.230 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-26 11:13:10.429957702 +0000 2024-04-26T11:19:33.230 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T11:19:33.315 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-26T11:19:33.315 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-26T11:19:33.315 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000547543 s, 935 kB/s 2024-04-26T11:19:33.319 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T11:19:33.377 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-04-26T11:19:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: Saving service mgr spec with placement smithi033=x;smithi096=y;smithi194=z;count:3 2024-04-26T11:19:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:33 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:33.428 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T11:19:33.428 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:33.428 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-26T11:19:33.428 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:33.428 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:33.429 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-26 11:18:33.861272203 +0000 2024-04-26T11:19:33.429 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-26 11:13:11.061948458 +0000 2024-04-26T11:19:33.429 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-26 11:13:11.061948458 +0000 2024-04-26T11:19:33.429 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-26 11:13:11.061948458 +0000 2024-04-26T11:19:33.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: Saving service mgr spec with placement smithi033=x;smithi096=y;smithi194=z;count:3 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:33 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:33.448 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:33.448 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:33.448 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: Saving service mgr spec with placement smithi033=x;smithi096=y;smithi194=z;count:3 2024-04-26T11:19:33.448 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:33.448 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:33.448 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T11:19:33.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T11:19:33.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:33 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:33.510 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-26T11:19:33.510 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-26T11:19:33.510 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000389425 s, 1.3 MB/s 2024-04-26T11:19:33.513 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T11:19:33.570 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-26T11:19:33.571 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T11:19:33.589 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T11:19:33.589 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-04-26T11:19:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-26 11:19:19.573964203 +0000 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-26 11:12:28.236418882 +0000 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-26 11:12:28.236418882 +0000 2024-04-26T11:19:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-26 11:12:28.236418882 +0000 2024-04-26T11:19:33.647 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T11:19:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-26T11:19:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-26T11:19:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000300896 s, 1.7 MB/s 2024-04-26T11:19:33.714 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T11:19:33.771 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-04-26T11:19:33.828 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T11:19:33.828 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:33.828 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T11:19:33.828 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:33.828 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:33.828 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-26 11:19:19.573964203 +0000 2024-04-26T11:19:33.829 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-26 11:12:28.575411705 +0000 2024-04-26T11:19:33.829 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-26 11:12:28.575411705 +0000 2024-04-26T11:19:33.829 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-26 11:12:28.575411705 +0000 2024-04-26T11:19:33.829 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T11:19:33.892 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-26T11:19:33.893 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-26T11:19:33.893 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000214445 s, 2.4 MB/s 2024-04-26T11:19:33.894 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T11:19:33.951 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-04-26T11:19:34.006 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-26 11:19:19.573964203 +0000 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-26 11:12:28.908404655 +0000 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-26 11:12:28.908404655 +0000 2024-04-26T11:19:34.007 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-26 11:12:28.908404655 +0000 2024-04-26T11:19:34.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T11:19:34.070 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-26T11:19:34.070 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-26T11:19:34.070 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000211718 s, 2.4 MB/s 2024-04-26T11:19:34.071 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T11:19:34.128 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-04-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-26 11:19:19.574964183 +0000 2024-04-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-26 11:12:29.239397647 +0000 2024-04-26T11:19:34.185 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-26 11:12:29.239397647 +0000 2024-04-26T11:19:34.185 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-26 11:12:29.239397647 +0000 2024-04-26T11:19:34.185 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T11:19:34.246 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-26T11:19:34.246 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-26T11:19:34.247 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.00021305 s, 2.4 MB/s 2024-04-26T11:19:34.248 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T11:19:34.306 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T11:19:34.306 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T11:19:34.321 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T11:19:34.322 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-04-26T11:19:34.376 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T11:19:34.376 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:34.376 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-26T11:19:34.376 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:34.376 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:34.377 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 11:19:10.168112680 +0000 2024-04-26T11:19:34.377 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 11:12:13.019407244 +0000 2024-04-26T11:19:34.377 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 11:12:13.019407244 +0000 2024-04-26T11:19:34.377 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-26 11:12:13.019407244 +0000 2024-04-26T11:19:34.377 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T11:19:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: Reconfiguring mon.c (monmap changed)... 2024-04-26T11:19:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: Reconfiguring daemon mon.c on smithi194 2024-04-26T11:19:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T11:19:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: Reconfiguring mon.c (monmap changed)... 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: Reconfiguring daemon mon.c on smithi194 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:34.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: Reconfiguring mon.c (monmap changed)... 2024-04-26T11:19:34.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: Reconfiguring daemon mon.c on smithi194 2024-04-26T11:19:34.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:34.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:34.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:34.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:34.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:34.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T11:19:34.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:34.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:34.440 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T11:19:34.440 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T11:19:34.440 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000195953 s, 2.6 MB/s 2024-04-26T11:19:34.441 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T11:19:34.496 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 11:19:10.169112660 +0000 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 11:12:13.368400151 +0000 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 11:12:13.368400151 +0000 2024-04-26T11:19:34.552 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-26 11:12:13.368400151 +0000 2024-04-26T11:19:34.553 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T11:19:34.617 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T11:19:34.617 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T11:19:34.617 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000210891 s, 2.4 MB/s 2024-04-26T11:19:34.619 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T11:19:34.675 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-04-26T11:19:34.730 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T11:19:34.730 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:34.730 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-26T11:19:34.731 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:34.731 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:34.731 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 11:19:10.169112660 +0000 2024-04-26T11:19:34.731 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 11:12:13.691393587 +0000 2024-04-26T11:19:34.731 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 11:12:13.691393587 +0000 2024-04-26T11:19:34.731 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-26 11:12:13.691393587 +0000 2024-04-26T11:19:34.731 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T11:19:34.792 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T11:19:34.792 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T11:19:34.792 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000210073 s, 2.4 MB/s 2024-04-26T11:19:34.793 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T11:19:34.848 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 11:19:10.169112660 +0000 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 11:12:14.045386392 +0000 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 11:12:14.045386392 +0000 2024-04-26T11:19:34.904 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-26 11:12:14.045386392 +0000 2024-04-26T11:19:34.904 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T11:19:34.967 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T11:19:34.967 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T11:19:34.967 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000194579 s, 2.6 MB/s 2024-04-26T11:19:34.968 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T11:19:35.023 INFO:tasks.cephadm:Deploying osd.0 on smithi033 with /dev/vg_nvme/lv_4... 2024-04-26T11:19:35.023 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T11:19:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:35 smithi194 ceph-mon[33017]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:35 smithi194 ceph-mon[33017]: Deploying daemon mgr.z on smithi194 2024-04-26T11:19:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:35 smithi033 ceph-mon[27845]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:35 smithi033 ceph-mon[27845]: Deploying daemon mgr.z on smithi194 2024-04-26T11:19:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:35 smithi096 ceph-mon[32753]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:35 smithi096 ceph-mon[32753]: Deploying daemon mgr.z on smithi194 2024-04-26T11:19:35.492 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:19:36.354 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 systemd[1]: Starting Ceph mgr.z for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:19:36.762 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 podman[34867]: 2024-04-26 11:19:36.370026535 +0000 UTC m=+0.018307357 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:19:36.762 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 podman[34867]: 2024-04-26 11:19:36.46734793 +0000 UTC m=+0.115628782 container create 53bfce5aaddf5a64540509eb733442856d041e7a4c42edf796aa76d6dcbeb6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T11:19:36.766 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 podman[34867]: 2024-04-26 11:19:36.653068473 +0000 UTC m=+0.301349284 container init 53bfce5aaddf5a64540509eb733442856d041e7a4c42edf796aa76d6dcbeb6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T11:19:36.766 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 podman[34867]: 2024-04-26 11:19:36.656036896 +0000 UTC m=+0.304317707 container start 53bfce5aaddf5a64540509eb733442856d041e7a4c42edf796aa76d6dcbeb6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-04-26T11:19:36.766 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 ceph-mgr[34886]: -- 172.21.15.194:0/2812187236 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56413833e5a0 con 0x564138313000 2024-04-26T11:19:36.766 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 bash[34867]: 53bfce5aaddf5a64540509eb733442856d041e7a4c42edf796aa76d6dcbeb6ba 2024-04-26T11:19:37.092 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 systemd[1]: Started Ceph mgr.z for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:19:37.092 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:36.803+0000 7fb2f4399200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:19:37.092 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:36 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:36.989+0000 7fb2f4399200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:19:37.092 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:37.090+0000 7fb2f4399200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:19:37.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:37.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:37.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T11:19:37.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:37.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:37 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T11:19:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T11:19:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:37 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:37.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:37.399+0000 7fb2f4399200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:19:37.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:37 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:37.482+0000 7fb2f4399200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:19:38.244 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T11:19:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T11:19:38.245 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-26T11:19:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-26T11:19:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0332199 s, 316 MB/s 2024-04-26T11:19:38.245 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-26T11:19:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:38 smithi096 ceph-mon[32753]: Deploying daemon mgr.y on smithi096 2024-04-26T11:19:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:38 smithi194 ceph-mon[33017]: Deploying daemon mgr.y on smithi096 2024-04-26T11:19:38.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:38 smithi033 ceph-mon[27845]: Deploying daemon mgr.y on smithi096 2024-04-26T11:19:38.998 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4 2024-04-26T11:19:39.015 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:38 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:38.671+0000 7fb2f4399200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:19:39.015 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:38 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:38.840+0000 7fb2f4399200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:19:39.015 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:39 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:39.013+0000 7fb2f4399200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:19:39.073 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:38 smithi096 systemd[1]: Starting Ceph mgr.y for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:19:39.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:39 smithi194 ceph-mon[33017]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:39.350 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:39 smithi096 ceph-mon[32753]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:39.351 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 podman[34139]: 2024-04-26 11:19:39.072677329 +0000 UTC m=+0.105168062 container create 013987178ca29328ae9198375e2c746740464bd680d7613439b4b880d274836e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-26T11:19:39.351 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 podman[34139]: 2024-04-26 11:19:38.985281953 +0000 UTC m=+0.017772656 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:19:39.351 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 podman[34139]: 2024-04-26 11:19:39.266600635 +0000 UTC m=+0.299091325 container init 013987178ca29328ae9198375e2c746740464bd680d7613439b4b880d274836e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T11:19:39.351 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 podman[34139]: 2024-04-26 11:19:39.269167804 +0000 UTC m=+0.301658503 container start 013987178ca29328ae9198375e2c746740464bd680d7613439b4b880d274836e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T11:19:39.351 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 ceph-mgr[34158]: -- 172.21.15.96:0/3665280252 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625830485a0 con 0x56258301d000 2024-04-26T11:19:39.351 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 bash[34139]: 013987178ca29328ae9198375e2c746740464bd680d7613439b4b880d274836e 2024-04-26T11:19:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:39 smithi033 ceph-mon[27845]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:39.474 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:19:39.593 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:39 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:39.300+0000 7fb2f4399200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:19:39.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:39 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:39.389+0000 7fb2f4399200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:19:39.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:39 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:39.592+0000 7fb2f4399200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:19:39.608 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 systemd[1]: Started Ceph mgr.y for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:19:39.608 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:39.420+0000 7f8ff734f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:19:39.608 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:39.607+0000 7f8ff734f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:19:39.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:39 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:39.785+0000 7fb2f4399200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:19:39.933 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:39 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:39.710+0000 7f8ff734f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:19:40.423 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:40.035+0000 7f8ff734f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:19:40.423 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:40.123+0000 7f8ff734f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:40 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:40 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:40 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:40 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:40 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:40 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:40 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:40 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:40 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:40 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:40.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:40.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:40.777 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:40.418+0000 7fb2f4399200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:19:41.104 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:40.845+0000 7fb2f4399200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:19:41.104 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:40 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:40.945+0000 7fb2f4399200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 -- 172.21.15.33:0/434813658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104690 msgr2=0x7ff518104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 --2- 172.21.15.33:0/434813658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104690 0x7ff518104af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5000099c0 tx=0x7ff500009cd0 comp rx=0 tx=0).stop 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 -- 172.21.15.33:0/434813658 shutdown_connections 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 --2- 172.21.15.33:0/434813658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518105030 0x7ff51810b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 --2- 172.21.15.33:0/434813658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff518103490 0x7ff5181038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 --2- 172.21.15.33:0/434813658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104690 0x7ff518104af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 -- 172.21.15.33:0/434813658 >> 172.21.15.33:0/434813658 conn(0x7ff5180fe9f0 msgr2=0x7ff518100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.445+0000 7ff51de00700 1 -- 172.21.15.33:0/434813658 shutdown_connections 2024-04-26T11:19:41.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.446+0000 7ff51de00700 1 -- 172.21.15.33:0/434813658 wait complete. 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.446+0000 7ff51de00700 1 Processor -- start 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff51de00700 1 -- start start 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff51de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518103490 0x7ff51819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff51de00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff518104690 0x7ff51819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff51de00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518105030 0x7ff5181a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff51de00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff51810ef50 con 0x7ff518103490 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff51de00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff51810edd0 con 0x7ff518105030 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff51de00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff51810f0d0 con 0x7ff518104690 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff517400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518105030 0x7ff5181a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff516a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518103490 0x7ff51819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff517400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518105030 0x7ff5181a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58888/0 (socket says 172.21.15.33:58888) 2024-04-26T11:19:41.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff517400700 1 -- 172.21.15.33:0/3370296225 learned_addr learned my addr 172.21.15.33:0/3370296225 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:19:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.447+0000 7ff50e000700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff518104690 0x7ff51819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff517400700 1 -- 172.21.15.33:0/3370296225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff518104690 msgr2=0x7ff51819bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff517400700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff518104690 0x7ff51819bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff517400700 1 -- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518103490 msgr2=0x7ff51819b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:19:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff517400700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518103490 0x7ff51819b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:19:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff517400700 1 -- 172.21.15.33:0/3370296225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5181a3460 con 0x7ff518105030 2024-04-26T11:19:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff517400700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518105030 0x7ff5181a2dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff51000d980 tx=0x7ff51000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510017070 con 0x7ff518105030 2024-04-26T11:19:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff510005020 con 0x7ff518105030 2024-04-26T11:19:41.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.448+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51000fa90 con 0x7ff518105030 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.449+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5181a3750 con 0x7ff518105030 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.449+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5181a3c40 con 0x7ff518105030 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.450+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff51000fc30 con 0x7ff518105030 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.450+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff51806a110 con 0x7ff518105030 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.450+0000 7ff51ca00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff4fc039890 0x7ff4fc03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.450+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff51004c810 con 0x7ff518105030 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.451+0000 7ff516a00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff4fc039890 0x7ff4fc03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:19:41.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.451+0000 7ff516a00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff4fc039890 0x7ff4fc03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5181044f0 tx=0x7ff50800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:19:41.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.456+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff510020020 con 0x7ff518105030 2024-04-26T11:19:41.470 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:41.144+0000 7fb2f4399200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:19:41.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:41.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:41.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:41.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.683 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:41.394+0000 7f8ff734f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:19:41.683 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:41.566+0000 7f8ff734f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:19:41.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:41.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:41.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:41.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:41.704+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff518076d70 con 0x7ff4fc039890 2024-04-26T11:19:41.764 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:41.674+0000 7fb2f4399200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:19:41.764 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:41.762+0000 7fb2f4399200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:41.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:42.014 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:41.733+0000 7f8ff734f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:19:42.014 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:42.012+0000 7f8ff734f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:19:42.104 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:41 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:41.996+0000 7fb2f4399200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:19:42.306 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:42.105+0000 7f8ff734f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:19:42.306 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:42.305+0000 7f8ff734f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:19:42.478 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:42.125+0000 7fb2f4399200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:19:42.478 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:42.216+0000 7fb2f4399200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:19:42.478 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:42.328+0000 7fb2f4399200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:19:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:19:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:19:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:42.683 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:42.495+0000 7f8ff734f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:19:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:19:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:19:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:42.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:42.628+0000 7fb2f4399200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:19:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T11:19:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T11:19:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:19:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:19:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:43.433 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:43.089+0000 7f8ff734f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:19:43.478 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:43.211+0000 7fb2f4399200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:19:43.478 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:19:43.420+0000 7fb2f4399200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:19:43.659 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T11:19:43.659 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: Reconfiguring daemon mgr.x on smithi033 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: Standby manager daemon z started 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T11:19:43.660 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:43 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:19:43.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:43.753+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7ff510010460 con 0x7ff518105030 2024-04-26T11:19:43.809 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:43.511+0000 7f8ff734f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:19:43.809 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:43.607+0000 7f8ff734f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:19:43.809 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:43.808+0000 7f8ff734f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:19:43.809 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T11:19:43.809 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: Reconfiguring daemon mgr.x on smithi033 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: Standby manager daemon z started 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T11:19:43.810 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:43 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:19:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T11:19:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: Reconfiguring daemon mgr.x on smithi033 2024-04-26T11:19:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:19:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: Standby manager daemon z started 2024-04-26T11:19:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T11:19:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:19:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T11:19:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:43 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/2713826582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:19:44.658 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:44.326+0000 7f8ff734f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:19:44.658 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:44.417+0000 7f8ff734f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-26T11:19:44.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-26T11:19:44.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T11:19:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:19:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:19:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:19:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-26T11:19:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T11:19:44.933 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:44.657+0000 7f8ff734f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:19:44.933 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:44.787+0000 7f8ff734f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:19:44.933 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:44.881+0000 7f8ff734f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:19:45.294 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:44 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:44.995+0000 7f8ff734f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:19:45.683 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:45 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:45.292+0000 7f8ff734f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:19:46.183 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:45 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:45.889+0000 7f8ff734f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:19:46.183 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:19:46.104+0000 7f8ff734f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:19:46.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:19:46.767+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mon.2 v2:172.21.15.96:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7ff510004a30 con 0x7ff518105030 2024-04-26T11:19:47.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: Standby manager daemon y started 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2695046360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]: dispatch 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]: dispatch 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]': finished 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T11:19:47.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:46 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:19:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: Standby manager daemon y started 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2695046360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]: dispatch 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]: dispatch 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]': finished 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T11:19:47.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:46 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:19:47.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:47.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: Standby manager daemon y started 2024-04-26T11:19:47.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/1032374022' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2695046360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]: dispatch 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]: dispatch 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d1df23-f689-46b9-a67c-a8846bd7c120"}]': finished 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T11:19:47.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:46 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:19:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:47 smithi194 ceph-mon[33017]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T11:19:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T11:19:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:47 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1171941214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:19:48.114 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:47 smithi033 ceph-mon[27845]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T11:19:48.114 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T11:19:48.114 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:47 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1171941214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:19:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:47 smithi096 ceph-mon[32753]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T11:19:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T11:19:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:47 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1171941214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:19:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:48 smithi096 ceph-mon[32753]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:48 smithi194 ceph-mon[33017]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:48 smithi033 ceph-mon[27845]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:50 smithi194 ceph-mon[33017]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:50 smithi033 ceph-mon[27845]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:50 smithi096 ceph-mon[32753]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:52.876 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:52 smithi033 ceph-mon[27845]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:52 smithi194 ceph-mon[33017]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:52 smithi096 ceph-mon[32753]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:54 smithi194 ceph-mon[33017]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:55.129 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:54 smithi033 ceph-mon[27845]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:54 smithi096 ceph-mon[32753]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T11:19:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T11:19:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T11:19:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:19:56.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:56 smithi033 ceph-mon[27845]: Deploying daemon osd.0 on smithi033 2024-04-26T11:19:56.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:56 smithi033 ceph-mon[27845]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:56 smithi194 ceph-mon[33017]: Deploying daemon osd.0 on smithi033 2024-04-26T11:19:57.133 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:56 smithi194 ceph-mon[33017]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:56 smithi096 ceph-mon[32753]: Deploying daemon osd.0 on smithi033 2024-04-26T11:19:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:56 smithi096 ceph-mon[32753]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:19:58 smithi194 ceph-mon[33017]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:59.142 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:19:58 smithi033 ceph-mon[27845]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:19:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:19:58 smithi096 ceph-mon[32753]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:01.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:00 smithi194 ceph-mon[33017]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:01.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:00 smithi194 ceph-mon[33017]: overall HEALTH_OK 2024-04-26T11:20:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:00 smithi033 ceph-mon[27845]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:01.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:00 smithi033 ceph-mon[27845]: overall HEALTH_OK 2024-04-26T11:20:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:00 smithi096 ceph-mon[32753]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:00 smithi096 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T11:20:03.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:02 smithi033 ceph-mon[27845]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:02 smithi194 ceph-mon[33017]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:02 smithi096 ceph-mon[32753]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:04 smithi033 ceph-mon[27845]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:04 smithi194 ceph-mon[33017]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:05.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:04 smithi096 ceph-mon[32753]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:07.063 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:06 smithi033 ceph-mon[27845]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:07.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:06 smithi096 ceph-mon[32753]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:06 smithi194 ceph-mon[33017]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:08 smithi033 ceph-mon[27845]: from='osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T11:20:08.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:08 smithi033 ceph-mon[27845]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T11:20:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:08 smithi096 ceph-mon[32753]: from='osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T11:20:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:08 smithi096 ceph-mon[32753]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T11:20:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:08.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:08.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:08.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:08 smithi194 ceph-mon[33017]: from='osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T11:20:08.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:08 smithi194 ceph-mon[33017]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T11:20:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.862+0000 7ff51ca00700 1 -- 172.21.15.33:0/3370296225 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff518076d70 con 0x7ff4fc039890 2024-04-26T11:20:08.865 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 0 on host 'smithi033' 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.867+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff4fc039890 msgr2=0x7ff4fc03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.867+0000 7ff51de00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff4fc039890 0x7ff4fc03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5181044f0 tx=0x7ff50800b040 comp rx=0 tx=0).stop 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.867+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518105030 msgr2=0x7ff5181a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.867+0000 7ff51de00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518105030 0x7ff5181a2dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff51000d980 tx=0x7ff51000dc90 comp rx=0 tx=0).stop 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 shutdown_connections 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff4fc039890 0x7ff4fc03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518105030 0x7ff5181a2dc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518103490 0x7ff51819b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 --2- 172.21.15.33:0/3370296225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff518104690 0x7ff51819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 >> 172.21.15.33:0/3370296225 conn(0x7ff5180fe9f0 msgr2=0x7ff518075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:20:08.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 shutdown_connections 2024-04-26T11:20:08.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:20:08.868+0000 7ff51de00700 1 -- 172.21.15.33:0/3370296225 wait complete. 2024-04-26T11:20:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T11:20:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T11:20:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: from='osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:20:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:20:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T11:20:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T11:20:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: from='osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:20:09.671 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:20:09.671 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:09.671 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:09.671 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: from='osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:09.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:09.727 DEBUG:teuthology.orchestra.run.smithi033:osd.0> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T11:20:09.732 INFO:tasks.cephadm:Deploying osd.1 on smithi096 with /dev/vg_nvme/lv_4... 2024-04-26T11:20:09.732 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T11:20:09.927 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-0[42184]: 2024-04-26T11:20:09.823+0000 7f8ed6000700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T11:20:09.927 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:20:09 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-0[42184]: 2024-04-26T11:20:09.839+0000 7f8ecbe00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:20:09.978 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:20:10.258 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:10 smithi033 ceph-mon[27845]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:20:10.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:10 smithi033 ceph-mon[27845]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T11:20:10.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:10 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:10.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:10 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:10.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:10 smithi033 ceph-mon[27845]: from='osd.0 ' entity='osd.0' 2024-04-26T11:20:10.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:10 smithi096 ceph-mon[32753]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:20:10.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:10 smithi096 ceph-mon[32753]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T11:20:10.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:10 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:10.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:10 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:10.389 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:10 smithi096 ceph-mon[32753]: from='osd.0 ' entity='osd.0' 2024-04-26T11:20:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:10 smithi194 ceph-mon[33017]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:20:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:10 smithi194 ceph-mon[33017]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T11:20:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:10 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:10 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:10 smithi194 ceph-mon[33017]: from='osd.0 ' entity='osd.0' 2024-04-26T11:20:11.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:11 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:20:11.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:11 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:20:11.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:11 smithi096 ceph-mon[32753]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:11.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:11 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:11.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:11 smithi096 ceph-mon[32753]: osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] boot 2024-04-26T11:20:11.259 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:11 smithi096 ceph-mon[32753]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T11:20:11.259 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:11 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:11.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:11 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:20:11.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:11 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:20:11.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:11 smithi033 ceph-mon[27845]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:11.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:11 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:11.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:11 smithi033 ceph-mon[27845]: osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] boot 2024-04-26T11:20:11.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:11 smithi033 ceph-mon[27845]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T11:20:11.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:11 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:11 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:20:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:11 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:20:11.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:11 smithi194 ceph-mon[33017]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T11:20:11.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:11.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:11 smithi194 ceph-mon[33017]: osd.0 [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] boot 2024-04-26T11:20:11.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:11 smithi194 ceph-mon[33017]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T11:20:11.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:20:11.802 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T11:20:11.802 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T11:20:11.803 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-26T11:20:11.803 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-26T11:20:11.803 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111249 s, 943 MB/s 2024-04-26T11:20:11.803 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-26T11:20:12.424 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_4 2024-04-26T11:20:12.655 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:20:13.043 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:12 smithi096 ceph-mon[32753]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:13.043 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:12 smithi096 ceph-mon[32753]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T11:20:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:12 smithi194 ceph-mon[33017]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:12 smithi194 ceph-mon[33017]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T11:20:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:12 smithi033 ceph-mon[27845]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:12 smithi033 ceph-mon[27845]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T11:20:14.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 -- 172.21.15.96:0/2974639070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 msgr2=0x7fb27c103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:14.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 --2- 172.21.15.96:0/2974639070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c103220 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00aad0 tx=0x7fb26c00ade0 comp rx=0 tx=0).stop 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 -- 172.21.15.96:0/2974639070 shutdown_connections 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 --2- 172.21.15.96:0/2974639070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c103760 0x7fb27c10a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 --2- 172.21.15.96:0/2974639070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c101bc0 0x7fb27c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 --2- 172.21.15.96:0/2974639070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c103220 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 -- 172.21.15.96:0/2974639070 >> 172.21.15.96:0/2974639070 conn(0x7fb27c0fd160 msgr2=0x7fb27c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 -- 172.21.15.96:0/2974639070 shutdown_connections 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 -- 172.21.15.96:0/2974639070 wait complete. 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.041+0000 7fb283e84700 1 Processor -- start 2024-04-26T11:20:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 -- start start 2024-04-26T11:20:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c101bc0 0x7fb27c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c103760 0x7fb27c1a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c10cea0 con 0x7fb27c103760 2024-04-26T11:20:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c10cd20 con 0x7fb27c102dc0 2024-04-26T11:20:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c10d020 con 0x7fb27c101bc0 2024-04-26T11:20:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56846/0 (socket says 172.21.15.96:56846) 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 -- 172.21.15.96:0/632502416 learned_addr learned my addr 172.21.15.96:0/632502416 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb281c20700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c101bc0 0x7fb27c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 -- 172.21.15.96:0/632502416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c101bc0 msgr2=0x7fb27c199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb282421700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c103760 0x7fb27c1a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c101bc0 0x7fb27c199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 -- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c103760 msgr2=0x7fb27c1a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c103760 0x7fb27c1a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 -- 172.21.15.96:0/632502416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c1a1360 con 0x7fb27c102dc0 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb281c20700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c101bc0 0x7fb27c199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb282421700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c103760 0x7fb27c1a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb28141f700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c199c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00c690 tx=0x7fb26c005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:20:14.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb272ffd700 1 -- 172.21.15.96:0/632502416 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0040f0 con 0x7fb27c102dc0 2024-04-26T11:20:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c1a15f0 con 0x7fb27c102dc0 2024-04-26T11:20:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb272ffd700 1 -- 172.21.15.96:0/632502416 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb26c004290 con 0x7fb27c102dc0 2024-04-26T11:20:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.042+0000 7fb272ffd700 1 -- 172.21.15.96:0/632502416 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c00e450 con 0x7fb27c102dc0 2024-04-26T11:20:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.043+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c1a1aa0 con 0x7fb27c102dc0 2024-04-26T11:20:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.043+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27c06a110 con 0x7fb27c102dc0 2024-04-26T11:20:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.045+0000 7fb272ffd700 1 -- 172.21.15.96:0/632502416 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb26c01e070 con 0x7fb27c102dc0 2024-04-26T11:20:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.046+0000 7fb272ffd700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb2680a3bb0 0x7fb2680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.046+0000 7fb272ffd700 1 -- 172.21.15.96:0/632502416 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fb26c0d0540 con 0x7fb27c102dc0 2024-04-26T11:20:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.046+0000 7fb281c20700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb2680a3bb0 0x7fb2680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:20:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.047+0000 7fb281c20700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb2680a3bb0 0x7fb2680a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb27c102c20 tx=0x7fb278009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:20:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.049+0000 7fb272ffd700 1 -- 172.21.15.96:0/632502416 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb26c09ea20 con 0x7fb27c102dc0 2024-04-26T11:20:14.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:14.258+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb27c063860 con 0x7fb2680a3bb0 2024-04-26T11:20:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:14 smithi194 ceph-mon[33017]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:14 smithi194 ceph-mon[33017]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:20:15.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:20:15.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:20:15.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:15.169 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:14 smithi096 ceph-mon[32753]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:15.169 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:14 smithi096 ceph-mon[32753]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:20:15.169 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:20:15.169 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:20:15.169 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:14 smithi033 ceph-mon[27845]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:14 smithi033 ceph-mon[27845]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:20:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:20:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:20:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi033 2024-04-26T11:20:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:16.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 17094M 2024-04-26T11:20:16.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:16.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:16.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi033 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 17094M 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi033 2024-04-26T11:20:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 17094M 2024-04-26T11:20:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:16.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:16.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:16.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:17.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:16 smithi033 ceph-mon[27845]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:17.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:16 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/2088590848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]: dispatch 2024-04-26T11:20:17.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:16 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]: dispatch 2024-04-26T11:20:17.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:16 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]': finished 2024-04-26T11:20:17.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:16 smithi033 ceph-mon[27845]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T11:20:17.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:16 smithi194 ceph-mon[33017]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:16 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/2088590848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]: dispatch 2024-04-26T11:20:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:16 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]: dispatch 2024-04-26T11:20:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:16 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]': finished 2024-04-26T11:20:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:16 smithi194 ceph-mon[33017]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T11:20:17.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:16 smithi096 ceph-mon[32753]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:16 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/2088590848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]: dispatch 2024-04-26T11:20:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:16 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]: dispatch 2024-04-26T11:20:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:16 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d3789d6-b7e0-41b5-b81f-5f2f2492d663"}]': finished 2024-04-26T11:20:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:16 smithi096 ceph-mon[32753]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T11:20:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:16 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:18.354 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:17 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/1353250829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:20:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:17 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/1353250829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:20:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:17 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/1353250829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:20:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:19 smithi194 ceph-mon[33017]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:19 smithi033 ceph-mon[27845]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:19 smithi096 ceph-mon[32753]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:21 smithi096 ceph-mon[32753]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:21 smithi194 ceph-mon[33017]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:21 smithi033 ceph-mon[27845]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:23.292 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:23 smithi096 ceph-mon[32753]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:23 smithi194 ceph-mon[33017]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:23 smithi033 ceph-mon[27845]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:24.342 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T11:20:24.342 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:24.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T11:20:24.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T11:20:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:25.330 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:25 smithi096 ceph-mon[32753]: Deploying daemon osd.1 on smithi096 2024-04-26T11:20:25.331 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:25 smithi096 ceph-mon[32753]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:25 smithi194 ceph-mon[33017]: Deploying daemon osd.1 on smithi096 2024-04-26T11:20:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:25 smithi194 ceph-mon[33017]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:25 smithi033 ceph-mon[27845]: Deploying daemon osd.1 on smithi096 2024-04-26T11:20:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:25 smithi033 ceph-mon[27845]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:27.252 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:27 smithi096 ceph-mon[32753]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:27 smithi194 ceph-mon[33017]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:27 smithi033 ceph-mon[27845]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:29 smithi194 ceph-mon[33017]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:29.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:29 smithi096 ceph-mon[32753]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:29.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:29.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:29.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:29 smithi033 ceph-mon[27845]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:31 smithi096 ceph-mon[32753]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:31 smithi194 ceph-mon[33017]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:31 smithi033 ceph-mon[27845]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:31.998 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 1 on host 'smithi096' 2024-04-26T11:20:31.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:31.997+0000 7fb272ffd700 1 -- 172.21.15.96:0/632502416 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb27c063860 con 0x7fb2680a3bb0 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:31.999+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb2680a3bb0 msgr2=0x7fb2680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:31.999+0000 7fb283e84700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb2680a3bb0 0x7fb2680a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb27c102c20 tx=0x7fb278009040 comp rx=0 tx=0).stop 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:31.999+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 msgr2=0x7fb27c199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:31.999+0000 7fb283e84700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c199c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00c690 tx=0x7fb26c005f10 comp rx=0 tx=0).stop 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 shutdown_connections 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb2680a3bb0 0x7fb2680a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb27c103760 0x7fb27c1a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c101bc0 0x7fb27c199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 --2- 172.21.15.96:0/632502416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb27c102dc0 0x7fb27c199c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 >> 172.21.15.96:0/632502416 conn(0x7fb27c0fd160 msgr2=0x7fb27c1060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:20:32.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 shutdown_connections 2024-04-26T11:20:32.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:20:32.000+0000 7fb283e84700 1 -- 172.21.15.96:0/632502416 wait complete. 2024-04-26T11:20:32.598 DEBUG:teuthology.orchestra.run.smithi096:osd.1> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T11:20:32.601 INFO:tasks.cephadm:Deploying osd.2 on smithi194 with /dev/vg_nvme/lv_4... 2024-04-26T11:20:32.601 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T11:20:32.849 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:20:32.955 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-1[37334]: 2024-04-26T11:20:32.691+0000 7fd9a06067c0 -1 osd.1 0 log_to_monitors true 2024-04-26T11:20:33.279 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:32 smithi194 ceph-mon[33017]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:33.279 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.279 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.280 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.280 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:32 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.280 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:32 smithi194 ceph-mon[33017]: from='osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T11:20:33.280 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:32 smithi194 ceph-mon[33017]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T11:20:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:32 smithi033 ceph-mon[27845]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:32 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:32 smithi033 ceph-mon[27845]: from='osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T11:20:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:32 smithi033 ceph-mon[27845]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T11:20:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-mon[32753]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-mon[32753]: from='osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T11:20:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:32 smithi096 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T11:20:34.218 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:34 smithi194 ceph-mon[33017]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T11:20:34.218 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:34 smithi194 ceph-mon[33017]: from='osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:20:34.218 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:34 smithi194 ceph-mon[33017]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T11:20:34.219 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:34.219 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:34 smithi194 ceph-mon[33017]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:20:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:34 smithi033 ceph-mon[27845]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T11:20:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:34 smithi033 ceph-mon[27845]: from='osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:20:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:34 smithi033 ceph-mon[27845]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T11:20:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:34 smithi033 ceph-mon[27845]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:20:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:34 smithi096 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T11:20:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:34 smithi096 ceph-mon[32753]: from='osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:20:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:34 smithi096 ceph-mon[32753]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T11:20:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:34 smithi096 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:20:34.434 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:20:34 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-1[37334]: 2024-04-26T11:20:34.104+0000 7fd995c6b700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T11:20:34.434 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:20:34 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-1[37334]: 2024-04-26T11:20:34.108+0000 7fd991a5e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:20:34.558 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T11:20:34.558 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T11:20:34.558 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-26T11:20:34.558 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-26T11:20:34.558 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113416 s, 925 MB/s 2024-04-26T11:20:34.559 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-26T11:20:35.141 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-04-26T11:20:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:35 smithi194 ceph-mon[33017]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:35 smithi194 ceph-mon[33017]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:20:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:35 smithi194 ceph-mon[33017]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T11:20:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:35.392 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:20:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:35 smithi033 ceph-mon[27845]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:35 smithi033 ceph-mon[27845]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:20:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:35 smithi033 ceph-mon[27845]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T11:20:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:35 smithi096 ceph-mon[32753]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:35 smithi096 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:20:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:35 smithi096 ceph-mon[32753]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T11:20:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:36.394 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:20:36.394 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] boot 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi096 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:36.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:36.396 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:36.396 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] boot 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi096 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:36.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:36.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:36.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: osd.1 [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] boot 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi096 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:36.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:36.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 -- 172.21.15.194:0/2106585871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c109180 msgr2=0x7fdc3c10b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2106585871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c109180 0x7fdc3c10b5f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400b0a0 tx=0x7fdc3400b3b0 comp rx=0 tx=0).stop 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 -- 172.21.15.194:0/2106585871 shutdown_connections 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2106585871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c109180 0x7fdc3c10b5f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2106585871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdc3c103f20 0x7fdc3c106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2106585871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc3c106850 0x7fdc3c108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 -- 172.21.15.194:0/2106585871 >> 172.21.15.194:0/2106585871 conn(0x7fdc3c0fd950 msgr2=0x7fdc3c0ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 -- 172.21.15.194:0/2106585871 shutdown_connections 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.948+0000 7fdc43a41700 1 -- 172.21.15.194:0/2106585871 wait complete. 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 Processor -- start 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 -- start start 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdc3c103f20 0x7fdc3c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c106850 0x7fdc3c199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:36.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc3c109180 0x7fdc3c1a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10ef90 con 0x7fdc3c109180 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10ee10 con 0x7fdc3c103f20 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc43a41700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10f110 con 0x7fdc3c106850 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c106850 0x7fdc3c199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c106850 0x7fdc3c199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:35042/0 (socket says 172.21.15.194:35042) 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 -- 172.21.15.194:0/2598423423 learned_addr learned my addr 172.21.15.194:0/2598423423 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc417dd700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdc3c103f20 0x7fdc3c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 -- 172.21.15.194:0/2598423423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdc3c103f20 msgr2=0x7fdc3c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdc3c103f20 0x7fdc3c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 -- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc3c109180 msgr2=0x7fdc3c1a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc3c109180 0x7fdc3c1a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc40fdc700 1 -- 172.21.15.194:0/2598423423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c1a12d0 con 0x7fdc3c106850 2024-04-26T11:20:36.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.949+0000 7fdc417dd700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdc3c103f20 0x7fdc3c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:20:36.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.950+0000 7fdc40fdc700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c106850 0x7fdc3c199bb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00b560 tx=0x7fdc2c00b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:20:36.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.950+0000 7fdc327fc700 1 -- 172.21.15.194:0/2598423423 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c015020 con 0x7fdc3c106850 2024-04-26T11:20:36.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.950+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c1a15c0 con 0x7fdc3c106850 2024-04-26T11:20:36.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.950+0000 7fdc327fc700 1 -- 172.21.15.194:0/2598423423 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc2c011ce0 con 0x7fdc3c106850 2024-04-26T11:20:36.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.950+0000 7fdc327fc700 1 -- 172.21.15.194:0/2598423423 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c010da0 con 0x7fdc3c106850 2024-04-26T11:20:36.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.950+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c1a1b80 con 0x7fdc3c106850 2024-04-26T11:20:36.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.951+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c06a110 con 0x7fdc3c106850 2024-04-26T11:20:36.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.952+0000 7fdc327fc700 1 -- 172.21.15.194:0/2598423423 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fdc2c018020 con 0x7fdc3c106850 2024-04-26T11:20:36.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.952+0000 7fdc327fc700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdc280a3bb0 0x7fdc280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:20:36.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.952+0000 7fdc327fc700 1 -- 172.21.15.194:0/2598423423 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fdc2c0ce3a0 con 0x7fdc3c106850 2024-04-26T11:20:36.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.953+0000 7fdc417dd700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdc280a3bb0 0x7fdc280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:20:36.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.954+0000 7fdc417dd700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdc280a3bb0 0x7fdc280a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc38005fd0 tx=0x7fdc38005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:20:36.958 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:36.957+0000 7fdc327fc700 1 -- 172.21.15.194:0/2598423423 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc2c01f020 con 0x7fdc3c106850 2024-04-26T11:20:37.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:37.164+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fdc3c063860 con 0x7fdc280a3bb0 2024-04-26T11:20:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:37 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 17088M 2024-04-26T11:20:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:37 smithi194 ceph-mon[33017]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:37 smithi194 ceph-mon[33017]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T11:20:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:37 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 17088M 2024-04-26T11:20:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:37 smithi033 ceph-mon[27845]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:37 smithi033 ceph-mon[27845]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T11:20:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:37 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 17088M 2024-04-26T11:20:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:37 smithi096 ceph-mon[32753]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T11:20:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:37 smithi096 ceph-mon[32753]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T11:20:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:38 smithi033 ceph-mon[27845]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:20:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:38 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:20:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:38 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:20:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:38 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:38 smithi096 ceph-mon[32753]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:20:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:38 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:20:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:38 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:20:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:38 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:38.461 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:38 smithi194 ceph-mon[33017]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:20:38.461 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:38 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:20:38.461 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:38 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:20:38.461 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:38 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:39 smithi033 ceph-mon[27845]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:39 smithi096 ceph-mon[32753]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:39 smithi194 ceph-mon[33017]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:40 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/36299559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]: dispatch 2024-04-26T11:20:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:40 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]: dispatch 2024-04-26T11:20:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:40 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]': finished 2024-04-26T11:20:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:40 smithi033 ceph-mon[27845]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T11:20:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:40 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:40 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/36299559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]: dispatch 2024-04-26T11:20:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:40 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]: dispatch 2024-04-26T11:20:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:40 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]': finished 2024-04-26T11:20:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:40 smithi096 ceph-mon[32753]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T11:20:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:40 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:40 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/36299559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]: dispatch 2024-04-26T11:20:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:40 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]: dispatch 2024-04-26T11:20:40.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:40 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05e220c5-ccfa-45ed-97a6-d2281a055fa6"}]': finished 2024-04-26T11:20:40.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:40 smithi194 ceph-mon[33017]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T11:20:40.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:40 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:41 smithi033 ceph-mon[27845]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:41 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/144986656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:20:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:41 smithi096 ceph-mon[32753]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:41 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/144986656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:20:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:41 smithi194 ceph-mon[33017]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:41 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/144986656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:20:43.423 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:43 smithi194 ceph-mon[33017]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:43 smithi033 ceph-mon[27845]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:43 smithi096 ceph-mon[32753]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:45 smithi033 ceph-mon[27845]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:45 smithi096 ceph-mon[32753]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:45.531 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:45 smithi194 ceph-mon[33017]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:47.422 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:47 smithi194 ceph-mon[33017]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:47.422 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T11:20:47.422 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:47 smithi033 ceph-mon[27845]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T11:20:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:47 smithi096 ceph-mon[32753]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T11:20:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:48 smithi096 ceph-mon[32753]: Deploying daemon osd.2 on smithi194 2024-04-26T11:20:48.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:48 smithi194 ceph-mon[33017]: Deploying daemon osd.2 on smithi194 2024-04-26T11:20:48.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:48 smithi033 ceph-mon[27845]: Deploying daemon osd.2 on smithi194 2024-04-26T11:20:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:49 smithi033 ceph-mon[27845]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:49 smithi096 ceph-mon[32753]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:49.490 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:49 smithi194 ceph-mon[33017]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:51 smithi194 ceph-mon[33017]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:51 smithi033 ceph-mon[27845]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:51 smithi096 ceph-mon[32753]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:52 smithi096 ceph-mon[32753]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:52 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:52 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:52 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:53.282 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:52 smithi194 ceph-mon[33017]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:53.282 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:52 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:53.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:52 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:53.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:52 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:52 smithi033 ceph-mon[27845]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:53.444 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:53.444 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:53.445 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:52 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:55.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:55 smithi194 ceph-mon[33017]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:55.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:55.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:55.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.599+0000 7fdc327fc700 1 -- 172.21.15.194:0/2598423423 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdc3c063860 con 0x7fdc280a3bb0 2024-04-26T11:20:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 2 on host 'smithi194' 2024-04-26T11:20:55.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdc280a3bb0 msgr2=0x7fdc280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:55.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdc280a3bb0 0x7fdc280a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc38005fd0 tx=0x7fdc38005ee0 comp rx=0 tx=0).stop 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c106850 msgr2=0x7fdc3c199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c106850 0x7fdc3c199bb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00b560 tx=0x7fdc2c00b920 comp rx=0 tx=0).stop 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 shutdown_connections 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdc280a3bb0 0x7fdc280a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdc3c109180 0x7fdc3c1a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdc3c103f20 0x7fdc3c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 --2- 172.21.15.194:0/2598423423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc3c106850 0x7fdc3c199bb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 >> 172.21.15.194:0/2598423423 conn(0x7fdc3c0fd950 msgr2=0x7fdc3c104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 shutdown_connections 2024-04-26T11:20:55.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:20:55.601+0000 7fdc43a41700 1 -- 172.21.15.194:0/2598423423 wait complete. 2024-04-26T11:20:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:55 smithi033 ceph-mon[27845]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:55 smithi096 ceph-mon[32753]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:56.230 DEBUG:teuthology.orchestra.run.smithi194:osd.2> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T11:20:56.233 INFO:tasks.cephadm:Deploying osd.3 on smithi033 with /dev/vg_nvme/lv_3... 2024-04-26T11:20:56.233 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T11:20:56.699 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:20:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:56.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:56.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:56 smithi194 ceph-mon[33017]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:56.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:56 smithi194 ceph-mon[33017]: from='osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T11:20:56.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:56 smithi194 ceph-mon[33017]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T11:20:56.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:56.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:56.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:56.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:56 smithi033 ceph-mon[27845]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:56.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:56 smithi033 ceph-mon[27845]: from='osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T11:20:56.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:56 smithi033 ceph-mon[27845]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T11:20:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:20:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:56 smithi096 ceph-mon[32753]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:56 smithi096 ceph-mon[32753]: from='osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T11:20:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:56 smithi096 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T11:20:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:57 smithi194 ceph-mon[33017]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T11:20:57.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:57 smithi194 ceph-mon[33017]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T11:20:57.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:57 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:57.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:57 smithi194 ceph-mon[33017]: from='osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:20:57.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:57 smithi194 ceph-mon[33017]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:20:57.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:57 smithi033 ceph-mon[27845]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T11:20:57.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:57 smithi033 ceph-mon[27845]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T11:20:57.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:57.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:57 smithi033 ceph-mon[27845]: from='osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:20:57.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:57 smithi033 ceph-mon[27845]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:20:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:57 smithi096 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T11:20:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:57 smithi096 ceph-mon[32753]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T11:20:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:57 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:57 smithi096 ceph-mon[32753]: from='osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:20:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:57 smithi096 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:20:58.354 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-2[38148]: 2024-04-26T11:20:58.014+0000 7f0b7c5c2700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T11:20:58.355 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-2[38148]: 2024-04-26T11:20:58.019+0000 7f0b743ad700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:20:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-mon[33017]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:20:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-mon[33017]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T11:20:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:58.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:58.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-mon[33017]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:58.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:58 smithi194 ceph-mon[33017]: from='osd.2 ' entity='osd.2' 2024-04-26T11:20:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:58 smithi033 ceph-mon[27845]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:20:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:58 smithi033 ceph-mon[27845]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T11:20:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:58 smithi033 ceph-mon[27845]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:58 smithi033 ceph-mon[27845]: from='osd.2 ' entity='osd.2' 2024-04-26T11:20:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:58 smithi096 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:20:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:58 smithi096 ceph-mon[32753]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T11:20:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:58 smithi096 ceph-mon[32753]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T11:20:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:58 smithi096 ceph-mon[32753]: from='osd.2 ' entity='osd.2' 2024-04-26T11:20:59.522 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T11:20:59.522 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T11:20:59.522 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-26T11:20:59.522 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-26T11:20:59.522 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0335731 s, 312 MB/s 2024-04-26T11:20:59.522 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-26T11:20:59.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:20:59.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:20:59.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:59.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] boot 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi194 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 17088M 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:59.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:20:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:20:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:20:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] boot 2024-04-26T11:20:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T11:20:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi194 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 17088M 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:20:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:20:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: osd.2 [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] boot 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi194 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 17088M 2024-04-26T11:21:00.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:00.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:00.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:00.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:20:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:00.334 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_3 2024-04-26T11:21:00.806 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:21:00.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:00 smithi033 ceph-mon[27845]: pgmap v89: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:00.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:00 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T11:21:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:00 smithi096 ceph-mon[32753]: pgmap v89: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:00 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T11:21:01.008 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:00 smithi194 ceph-mon[33017]: pgmap v89: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:01.008 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:00 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T11:21:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T11:21:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:01 smithi033 ceph-mon[27845]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T11:21:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T11:21:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T11:21:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:01 smithi096 ceph-mon[32753]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T11:21:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T11:21:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T11:21:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:01 smithi194 ceph-mon[33017]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T11:21:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:01 smithi033 sudo[46364]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:01 smithi033 sudo[46364]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:01 smithi033 sudo[46364]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46364]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46367]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46367]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46367]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.266 INFO:journalctl@ceph.osd.0.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46367]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41159]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41159]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41162]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41162]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41162]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.373 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41162]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46388]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T11:21:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46388]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46388]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46388]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46393]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T11:21:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46393]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46393]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 sudo[46393]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42226]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T11:21:02.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42226]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42226]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42226]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42229]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T11:21:02.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42229]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42229]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42229]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42219]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42219]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42219]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42222]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42222]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42222]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.606 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 26 11:21:02 smithi194 sudo[42222]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.646 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41165]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T11:21:02.646 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41165]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.646 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41165]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41165]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41168]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41168]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41168]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 sudo[41168]: pam_unix(sudo:session): session closed for user root 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:02.647 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:02.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.860+0000 7f91f1000700 1 -- 172.21.15.33:0/3550299787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec105070 msgr2=0x7f91ec10b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.860+0000 7f91f1000700 1 --2- 172.21.15.33:0/3550299787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec105070 0x7f91ec10b920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009fc0 tx=0x7f91e400b330 comp rx=0 tx=0).stop 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.861+0000 7f91f1000700 1 -- 172.21.15.33:0/3550299787 shutdown_connections 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.861+0000 7f91f1000700 1 --2- 172.21.15.33:0/3550299787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec105070 0x7f91ec10b920 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.861+0000 7f91f1000700 1 --2- 172.21.15.33:0/3550299787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f91ec1034d0 0x7f91ec1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.861+0000 7f91f1000700 1 --2- 172.21.15.33:0/3550299787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91ec1046d0 0x7f91ec104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.861+0000 7f91f1000700 1 -- 172.21.15.33:0/3550299787 >> 172.21.15.33:0/3550299787 conn(0x7f91ec0fea70 msgr2=0x7f91ec100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.861+0000 7f91f1000700 1 -- 172.21.15.33:0/3550299787 shutdown_connections 2024-04-26T11:21:02.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.861+0000 7f91f1000700 1 -- 172.21.15.33:0/3550299787 wait complete. 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:02.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:02.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:02 smithi033 ceph-mon[27845]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:02.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.862+0000 7f91f1000700 1 Processor -- start 2024-04-26T11:21:02.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.862+0000 7f91f1000700 1 -- start start 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91f1000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec1034d0 0x7f91ec0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91f1000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91ec1046d0 0x7f91ec075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91f1000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f91ec105070 0x7f91ec075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91f1000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec10efa0 con 0x7f91ec1046d0 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91f1000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec10ee20 con 0x7f91ec1034d0 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91f1000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec10f120 con 0x7f91ec105070 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91e9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec1034d0 0x7f91ec0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91e9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec1034d0 0x7f91ec0771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37216/0 (socket says 172.21.15.33:37216) 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.863+0000 7f91e9e00700 1 -- 172.21.15.33:0/286547976 learned_addr learned my addr 172.21.15.33:0/286547976 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:21:02.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91ea800700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f91ec105070 0x7f91ec075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e1400700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91ec1046d0 0x7f91ec075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e9e00700 1 -- 172.21.15.33:0/286547976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f91ec105070 msgr2=0x7f91ec075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e9e00700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f91ec105070 0x7f91ec075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e9e00700 1 -- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91ec1046d0 msgr2=0x7f91ec075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e9e00700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91ec1046d0 0x7f91ec075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e9e00700 1 -- 172.21.15.33:0/286547976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec076600 con 0x7f91ec1034d0 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e1400700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91ec1046d0 0x7f91ec075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e9e00700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec1034d0 0x7f91ec0771c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00e970 tx=0x7f91dc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.864+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00cbd0 con 0x7f91ec1034d0 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.865+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91dc0044d0 con 0x7f91ec1034d0 2024-04-26T11:21:02.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.865+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc003df0 con 0x7f91ec1034d0 2024-04-26T11:21:02.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.865+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec1a3710 con 0x7f91ec1034d0 2024-04-26T11:21:02.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.865+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec1a3bd0 con 0x7f91ec1034d0 2024-04-26T11:21:02.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.866+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ec06a110 con 0x7f91ec1034d0 2024-04-26T11:21:02.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.868+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f91dc003f90 con 0x7f91ec1034d0 2024-04-26T11:21:02.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.869+0000 7f91e3400700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f91d00a3c30 0x7f91d00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:02.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.869+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f91dc0cf1f0 con 0x7f91ec1034d0 2024-04-26T11:21:02.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.869+0000 7f91e1400700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f91d00a3c30 0x7f91d00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:21:02.872 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.870+0000 7f91e1400700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f91d00a3c30 0x7f91d00a60f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f91d4005fd0 tx=0x7f91d4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:21:02.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:02.873+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91dc016330 con 0x7f91ec1034d0 2024-04-26T11:21:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:02.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:02 smithi096 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:21:03.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:02 smithi194 ceph-mon[33017]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T11:21:03.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:03.115+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f91ec063860 con 0x7f91d00a3c30 2024-04-26T11:21:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:03 smithi033 ceph-mon[27845]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T11:21:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:03 smithi033 ceph-mon[27845]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:21:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:03 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:21:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:03 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:21:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:03 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:03 smithi096 ceph-mon[32753]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T11:21:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:03 smithi096 ceph-mon[32753]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:21:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:03 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:21:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:03 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:21:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:03 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:03 smithi194 ceph-mon[33017]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T11:21:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:03 smithi194 ceph-mon[33017]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:21:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:03 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:21:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:03 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:21:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:03 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:04.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:04.687+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f91dc098f50 con 0x7f91ec1034d0 2024-04-26T11:21:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:04 smithi033 ceph-mon[27845]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:04 smithi096 ceph-mon[32753]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:04 smithi194 ceph-mon[33017]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:05.895 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:05 smithi033 ceph-mon[27845]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T11:21:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:05 smithi194 ceph-mon[33017]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T11:21:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:05 smithi096 ceph-mon[32753]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T11:21:07.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:06 smithi096 ceph-mon[32753]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:06 smithi194 ceph-mon[33017]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:06 smithi033 ceph-mon[27845]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:07 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3310246414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f814ef05-a751-446c-b5c9-319aa93738c1"}]: dispatch 2024-04-26T11:21:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:07 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3310246414' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f814ef05-a751-446c-b5c9-319aa93738c1"}]': finished 2024-04-26T11:21:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:07 smithi194 ceph-mon[33017]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T11:21:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:07 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:07 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3310246414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f814ef05-a751-446c-b5c9-319aa93738c1"}]: dispatch 2024-04-26T11:21:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:07 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3310246414' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f814ef05-a751-446c-b5c9-319aa93738c1"}]': finished 2024-04-26T11:21:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:07 smithi033 ceph-mon[27845]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T11:21:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:07 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:07 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3310246414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f814ef05-a751-446c-b5c9-319aa93738c1"}]: dispatch 2024-04-26T11:21:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:07 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3310246414' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f814ef05-a751-446c-b5c9-319aa93738c1"}]': finished 2024-04-26T11:21:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:07 smithi096 ceph-mon[32753]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T11:21:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:07 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:08 smithi194 ceph-mon[33017]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:08 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/491165770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:21:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:08 smithi033 ceph-mon[27845]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:08 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/491165770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:21:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:08 smithi096 ceph-mon[32753]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:08 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/491165770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:21:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:10 smithi194 ceph-mon[33017]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:10 smithi033 ceph-mon[27845]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:11.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:10 smithi096 ceph-mon[32753]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:12 smithi194 ceph-mon[33017]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:12 smithi033 ceph-mon[27845]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:12 smithi096 ceph-mon[32753]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:14.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:14 smithi033 ceph-mon[27845]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:14 smithi194 ceph-mon[33017]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:14 smithi096 ceph-mon[32753]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:16 smithi194 ceph-mon[33017]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T11:21:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:16 smithi194 ceph-mon[33017]: Deploying daemon osd.3 on smithi033 2024-04-26T11:21:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:16 smithi033 ceph-mon[27845]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T11:21:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:16 smithi033 ceph-mon[27845]: Deploying daemon osd.3 on smithi033 2024-04-26T11:21:17.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:16 smithi096 ceph-mon[32753]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:17.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:16 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T11:21:17.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:16 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:17.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:16 smithi096 ceph-mon[32753]: Deploying daemon osd.3 on smithi033 2024-04-26T11:21:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:18 smithi194 ceph-mon[33017]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:19.123 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:18 smithi033 ceph-mon[27845]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:18 smithi096 ceph-mon[32753]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:20 smithi194 ceph-mon[33017]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:20 smithi033 ceph-mon[27845]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:20 smithi096 ceph-mon[32753]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:22 smithi194 ceph-mon[33017]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:22 smithi033 ceph-mon[27845]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:22 smithi096 ceph-mon[32753]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:24 smithi033 ceph-mon[27845]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:24 smithi194 ceph-mon[33017]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:24 smithi096 ceph-mon[32753]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:26.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:26.146 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:26.146 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:25 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:25 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:25 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:26 smithi033 ceph-mon[27845]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:26 smithi096 ceph-mon[32753]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:26 smithi194 ceph-mon[33017]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:29 smithi033 ceph-mon[27845]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:29 smithi033 ceph-mon[27845]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T11:21:29.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.565+0000 7f91e3400700 1 -- 172.21.15.33:0/286547976 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f91ec063860 con 0x7f91d00a3c30 2024-04-26T11:21:29.567 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 3 on host 'smithi033' 2024-04-26T11:21:29.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.568+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f91d00a3c30 msgr2=0x7f91d00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:29.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.568+0000 7f91f1000700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f91d00a3c30 0x7f91d00a60f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f91d4005fd0 tx=0x7f91d4009310 comp rx=0 tx=0).stop 2024-04-26T11:21:29.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.568+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec1034d0 msgr2=0x7f91ec0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:29.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.568+0000 7f91f1000700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec1034d0 0x7f91ec0771c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00e970 tx=0x7f91dc00ec80 comp rx=0 tx=0).stop 2024-04-26T11:21:29.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 shutdown_connections 2024-04-26T11:21:29.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f91d00a3c30 0x7f91d00a60f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:29.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f91ec105070 0x7f91ec075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:29.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91ec1034d0 0x7f91ec0771c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:29.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 --2- 172.21.15.33:0/286547976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91ec1046d0 0x7f91ec075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:29.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 >> 172.21.15.33:0/286547976 conn(0x7f91ec0fea70 msgr2=0x7f91ec1079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:21:29.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 shutdown_connections 2024-04-26T11:21:29.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:21:29.569+0000 7f91f1000700 1 -- 172.21.15.33:0/286547976 wait complete. 2024-04-26T11:21:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:29 smithi194 ceph-mon[33017]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:29 smithi194 ceph-mon[33017]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T11:21:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:29 smithi096 ceph-mon[32753]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:29 smithi096 ceph-mon[32753]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T11:21:30.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:30 smithi033 ceph-mon[27845]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T11:21:30.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:30 smithi033 ceph-mon[27845]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T11:21:30.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:30 smithi033 ceph-mon[27845]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:21:30.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:30 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:30.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:30 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:30.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:30 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:30.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:30 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:30.520 DEBUG:teuthology.orchestra.run.smithi033:osd.3> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T11:21:30.528 INFO:tasks.cephadm:Deploying osd.4 on smithi096 with /dev/vg_nvme/lv_3... 2024-04-26T11:21:30.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T11:21:30.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:30 smithi096 ceph-mon[32753]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T11:21:30.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:30 smithi096 ceph-mon[32753]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T11:21:30.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:30 smithi096 ceph-mon[32753]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:21:30.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:30 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:30.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:30 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:30.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:30 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:30.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:30 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:30 smithi194 ceph-mon[33017]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T11:21:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:30 smithi194 ceph-mon[33017]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T11:21:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:30 smithi194 ceph-mon[33017]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:21:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:30 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:30 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:30 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:30 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:30.774 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] boot 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T11:21:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:31.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:21:31.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T11:21:31.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] boot 2024-04-26T11:21:31.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T11:21:31.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T11:21:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: from='osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:21:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T11:21:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: osd.3 [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] boot 2024-04-26T11:21:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T11:21:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:21:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:32 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:21:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:32 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:21:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:32 smithi096 ceph-mon[32753]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T11:21:32.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:32 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:21:32.554 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:32 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:21:32.554 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:32 smithi033 ceph-mon[27845]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T11:21:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:32 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:21:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:32 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:21:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:32 smithi194 ceph-mon[33017]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T11:21:32.681 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T11:21:32.682 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T11:21:32.682 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-26T11:21:32.682 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-26T11:21:32.682 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112733 s, 930 MB/s 2024-04-26T11:21:32.682 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-26T11:21:33.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_3 2024-04-26T11:21:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:33 smithi194 ceph-mon[33017]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:33 smithi194 ceph-mon[33017]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T11:21:33.627 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:21:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:33 smithi033 ceph-mon[27845]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:33 smithi033 ceph-mon[27845]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T11:21:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:33 smithi096 ceph-mon[32753]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:33 smithi096 ceph-mon[32753]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T11:21:35.000 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:34 smithi096 ceph-mon[32753]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:35.000 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:34 smithi096 ceph-mon[32753]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T11:21:35.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:34 smithi033 ceph-mon[27845]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:35.134 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:34 smithi033 ceph-mon[27845]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T11:21:35.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/568030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c8102d10 msgr2=0x7fa0c8103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:35.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/568030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c8102d10 0x7fa0c8103170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00099c0 tx=0x7fa0b0009cd0 comp rx=0 tx=0).stop 2024-04-26T11:21:35.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/568030 shutdown_connections 2024-04-26T11:21:35.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/568030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0c81036b0 0x7fa0c8109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:35.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/568030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c8101b10 0x7fa0c8101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:35.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/568030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c8102d10 0x7fa0c8103170 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:35.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/568030 >> 172.21.15.96:0/568030 conn(0x7fa0c80fd090 msgr2=0x7fa0c80ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:21:35.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/568030 shutdown_connections 2024-04-26T11:21:35.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.203+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/568030 wait complete. 2024-04-26T11:21:35.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 Processor -- start 2024-04-26T11:21:35.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 -- start start 2024-04-26T11:21:35.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c8101b10 0x7fa0c81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:35.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0c8102d10 0x7fa0c8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:35.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c81036b0 0x7fa0c819fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:35.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810df20 con 0x7fa0c8102d10 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810dda0 con 0x7fa0c81036b0 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0cdfc6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810e0a0 con 0x7fa0c8101b10 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0c7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c81036b0 0x7fa0c819fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0c7fff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c81036b0 0x7fa0c819fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46288/0 (socket says 172.21.15.96:46288) 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.204+0000 7fa0c7fff700 1 -- 172.21.15.96:0/1435157443 learned_addr learned my addr 172.21.15.96:0/1435157443 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c77fe700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c8101b10 0x7fa0c81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c7fff700 1 -- 172.21.15.96:0/1435157443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c8101b10 msgr2=0x7fa0c81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c7fff700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c8101b10 0x7fa0c81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c7fff700 1 -- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0c8102d10 msgr2=0x7fa0c8198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c7fff700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0c8102d10 0x7fa0c8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c7fff700 1 -- 172.21.15.96:0/1435157443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c81a01d0 con 0x7fa0c81036b0 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c77fe700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c8101b10 0x7fa0c81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:21:35.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c7fff700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c81036b0 0x7fa0c819fb30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000b970 tx=0x7fa0c000bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:21:35.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c57fa700 1 -- 172.21.15.96:0/1435157443 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0019070 con 0x7fa0c81036b0 2024-04-26T11:21:35.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c57fa700 1 -- 172.21.15.96:0/1435157443 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c0014440 con 0x7fa0c81036b0 2024-04-26T11:21:35.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c81a04c0 con 0x7fa0c81036b0 2024-04-26T11:21:35.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0c57fa700 1 -- 172.21.15.96:0/1435157443 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c000ce60 con 0x7fa0c81036b0 2024-04-26T11:21:35.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.205+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c81a0a30 con 0x7fa0c81036b0 2024-04-26T11:21:35.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.206+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c806a110 con 0x7fa0c81036b0 2024-04-26T11:21:35.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.207+0000 7fa0c57fa700 1 -- 172.21.15.96:0/1435157443 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa0c001c070 con 0x7fa0c81036b0 2024-04-26T11:21:35.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.208+0000 7fa0c57fa700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa0b40a3c40 0x7fa0b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:21:35.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.208+0000 7fa0c57fa700 1 -- 172.21.15.96:0/1435157443 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7fa0c00ce5e0 con 0x7fa0c81036b0 2024-04-26T11:21:35.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.208+0000 7fa0c77fe700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa0b40a3c40 0x7fa0b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:21:35.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.209+0000 7fa0c77fe700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa0b40a3c40 0x7fa0b40a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8102b70 tx=0x7fa0b8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:21:35.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.212+0000 7fa0c57fa700 1 -- 172.21.15.96:0/1435157443 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0c009c1e0 con 0x7fa0c81036b0 2024-04-26T11:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:34 smithi194 ceph-mon[33017]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:34 smithi194 ceph-mon[33017]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T11:21:35.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:35.417+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa0c8063860 con 0x7fa0b40a3c40 2024-04-26T11:21:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:35 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:36.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:36.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:35 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T11:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:36.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:35 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:36 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi033 2024-04-26T11:21:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:36 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 8547M 2024-04-26T11:21:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:36 smithi033 ceph-mon[27845]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:36 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi033 2024-04-26T11:21:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:36 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 8547M 2024-04-26T11:21:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:36 smithi096 ceph-mon[32753]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:36 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi033 2024-04-26T11:21:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:36 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 8547M 2024-04-26T11:21:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:36 smithi194 ceph-mon[33017]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:38 smithi096 ceph-mon[32753]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T11:21:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:38 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/3719665358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]: dispatch 2024-04-26T11:21:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:38 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]: dispatch 2024-04-26T11:21:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:38 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]': finished 2024-04-26T11:21:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:38 smithi096 ceph-mon[32753]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T11:21:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:38 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:39.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:38 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/2473440282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:21:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:38 smithi194 ceph-mon[33017]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T11:21:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:38 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/3719665358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]: dispatch 2024-04-26T11:21:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:38 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]: dispatch 2024-04-26T11:21:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:38 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]': finished 2024-04-26T11:21:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:38 smithi194 ceph-mon[33017]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T11:21:39.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:38 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:39.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:38 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/2473440282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:38 smithi033 ceph-mon[27845]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T11:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:38 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/3719665358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]: dispatch 2024-04-26T11:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:38 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]: dispatch 2024-04-26T11:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:38 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5f7754c-be64-4277-8ce2-6d3117800d71"}]': finished 2024-04-26T11:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:38 smithi033 ceph-mon[27845]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T11:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:38 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:39.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:38 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/2473440282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:21:41.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:40 smithi096 ceph-mon[32753]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:21:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:40 smithi194 ceph-mon[33017]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:21:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:40 smithi033 ceph-mon[27845]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:21:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:42 smithi194 ceph-mon[33017]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:21:43.373 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:42 smithi096 ceph-mon[32753]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:21:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:42 smithi033 ceph-mon[27845]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:21:45.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:44 smithi096 ceph-mon[32753]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T11:21:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:44 smithi194 ceph-mon[33017]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T11:21:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:44 smithi033 ceph-mon[27845]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T11:21:46.238 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:45 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T11:21:46.239 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:45 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:45 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T11:21:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:45 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:45 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T11:21:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:45 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:47.244 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:46 smithi096 ceph-mon[32753]: Deploying daemon osd.4 on smithi096 2024-04-26T11:21:47.244 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:46 smithi096 ceph-mon[32753]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T11:21:47.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:46 smithi033 ceph-mon[27845]: Deploying daemon osd.4 on smithi096 2024-04-26T11:21:47.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:46 smithi033 ceph-mon[27845]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T11:21:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:46 smithi194 ceph-mon[33017]: Deploying daemon osd.4 on smithi096 2024-04-26T11:21:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:46 smithi194 ceph-mon[33017]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T11:21:49.045 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:48 smithi033 ceph-mon[27845]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:49.318 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:48 smithi096 ceph-mon[32753]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:48 smithi194 ceph-mon[33017]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:50 smithi096 ceph-mon[32753]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:50 smithi194 ceph-mon[33017]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:50 smithi033 ceph-mon[27845]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:52.070 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:52.070 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:52.070 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:53 smithi194 ceph-mon[33017]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:53 smithi033 ceph-mon[27845]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:53 smithi096 ceph-mon[32753]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:54.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:54.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:54.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:54.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:54.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:54.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:54 smithi096 ceph-mon[32753]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:54.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.868+0000 7fa0c57fa700 1 -- 172.21.15.96:0/1435157443 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa0c8063860 con 0x7fa0b40a3c40 2024-04-26T11:21:54.870 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 4 on host 'smithi096' 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.870+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa0b40a3c40 msgr2=0x7fa0b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.870+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa0b40a3c40 0x7fa0b40a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8102b70 tx=0x7fa0b8008040 comp rx=0 tx=0).stop 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.870+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c81036b0 msgr2=0x7fa0c819fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.870+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c81036b0 0x7fa0c819fb30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000b970 tx=0x7fa0c000bd30 comp rx=0 tx=0).stop 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 shutdown_connections 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa0b40a3c40 0x7fa0b40a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0c81036b0 0x7fa0c819fb30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c8101b10 0x7fa0c81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 --2- 172.21.15.96:0/1435157443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0c8102d10 0x7fa0c8198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 >> 172.21.15.96:0/1435157443 conn(0x7fa0c80fd090 msgr2=0x7fa0c8106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 shutdown_connections 2024-04-26T11:21:54.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:21:54.871+0000 7fa0cdfc6700 1 -- 172.21.15.96:0/1435157443 wait complete. 2024-04-26T11:21:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:54 smithi033 ceph-mon[27845]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:55.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:55.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:55.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:54 smithi194 ceph-mon[33017]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:55.514 DEBUG:teuthology.orchestra.run.smithi096:osd.4> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T11:21:55.517 INFO:tasks.cephadm:Deploying osd.5 on smithi194 with /dev/vg_nvme/lv_3... 2024-04-26T11:21:55.518 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T11:21:55.762 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:21:55.975 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:55.975 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:55.975 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:55.975 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:55 smithi096 ceph-mon[32753]: from='osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T11:21:55.976 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:55 smithi096 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T11:21:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:55 smithi194 ceph-mon[33017]: from='osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T11:21:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:55 smithi194 ceph-mon[33017]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T11:21:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:21:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:55 smithi033 ceph-mon[27845]: from='osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T11:21:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:55 smithi033 ceph-mon[27845]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T11:21:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:56 smithi194 ceph-mon[33017]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:56 smithi194 ceph-mon[33017]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T11:21:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:56 smithi194 ceph-mon[33017]: from='osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:21:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:56 smithi194 ceph-mon[33017]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T11:21:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:56 smithi194 ceph-mon[33017]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:21:57.123 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-mon[32753]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:57.124 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T11:21:57.124 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-mon[32753]: from='osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:21:57.124 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-mon[32753]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T11:21:57.124 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:57.124 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:21:57.124 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-4[43994]: 2024-04-26T11:21:56.842+0000 7fe2b3b6d700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T11:21:57.124 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 26 11:21:56 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-4[43994]: 2024-04-26T11:21:56.847+0000 7fe2af960700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:21:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:56 smithi033 ceph-mon[27845]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:56 smithi033 ceph-mon[27845]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T11:21:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:56 smithi033 ceph-mon[27845]: from='osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:21:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:56 smithi033 ceph-mon[27845]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T11:21:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:56 smithi033 ceph-mon[27845]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:21:57.774 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T11:21:57.774 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T11:21:57.774 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-26T11:21:57.774 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-26T11:21:57.774 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115342 s, 909 MB/s 2024-04-26T11:21:57.774 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-26T11:21:58.001 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:57 smithi096 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:21:58.001 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:57 smithi096 ceph-mon[32753]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T11:21:58.002 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:57 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:58.002 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:57 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:57 smithi033 ceph-mon[27845]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:21:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:57 smithi033 ceph-mon[27845]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T11:21:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:57 smithi194 ceph-mon[33017]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:21:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:57 smithi194 ceph-mon[33017]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T11:21:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:57 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:57 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:58.416 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-04-26T11:21:58.662 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] boot 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi096 2024-04-26T11:21:58.954 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 8544M 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:58.955 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] boot 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi096 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 8544M 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:59.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:59.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: osd.4 [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] boot 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi096 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:59.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:21:59.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 8544M 2024-04-26T11:21:59.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:21:59.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:21:59.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:21:59.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:21:59 smithi033 ceph-mon[27845]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T11:22:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:21:59 smithi096 ceph-mon[32753]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T11:22:00.210 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:21:59 smithi194 ceph-mon[33017]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 -- 172.21.15.194:0/1254776912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730102d30 msgr2=0x7f1730103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 --2- 172.21.15.194:0/1254776912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730102d30 0x7f1730103190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f172000aaa0 tx=0x7f172000adb0 comp rx=0 tx=0).stop 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 -- 172.21.15.194:0/1254776912 shutdown_connections 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 --2- 172.21.15.194:0/1254776912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17301036d0 0x7f1730109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 --2- 172.21.15.194:0/1254776912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1730101b30 0x7f1730101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 --2- 172.21.15.194:0/1254776912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730102d30 0x7f1730103190 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 -- 172.21.15.194:0/1254776912 >> 172.21.15.194:0/1254776912 conn(0x7f17300fd0d0 msgr2=0x7f17300ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:22:00.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 -- 172.21.15.194:0/1254776912 shutdown_connections 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 -- 172.21.15.194:0/1254776912 wait complete. 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 Processor -- start 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.312+0000 7f17377cd700 1 -- start start 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17377cd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730101b30 0x7f1730197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17377cd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1730102d30 0x7f1730197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17377cd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17301036d0 0x7f173019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17377cd700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f173010cde0 con 0x7f1730102d30 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17377cd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f173010cc60 con 0x7f17301036d0 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17377cd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f173010cf60 con 0x7f1730101b30 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730101b30 0x7f1730197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730101b30 0x7f1730197420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:33836/0 (socket says 172.21.15.194:33836) 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 -- 172.21.15.194:0/3356496142 learned_addr learned my addr 172.21.15.194:0/3356496142 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:22:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735d6a700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17301036d0 0x7f173019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 -- 172.21.15.194:0/3356496142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17301036d0 msgr2=0x7f173019e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17301036d0 0x7f173019e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 -- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1730102d30 msgr2=0x7f1730197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1730102d30 0x7f1730197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 -- 172.21.15.194:0/3356496142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173019f080 con 0x7f1730101b30 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f1735569700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730101b30 0x7f1730197420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f172c00d9e0 tx=0x7f172c00dda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17267fc700 1 -- 172.21.15.194:0/3356496142 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c01b070 con 0x7f1730101b30 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173019f370 con 0x7f1730101b30 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17267fc700 1 -- 172.21.15.194:0/3356496142 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f172c00ecd0 con 0x7f1730101b30 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.313+0000 7f17267fc700 1 -- 172.21.15.194:0/3356496142 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c0145a0 con 0x7f1730101b30 2024-04-26T11:22:00.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.314+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f173019f8b0 con 0x7f1730101b30 2024-04-26T11:22:00.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.314+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f173006a110 con 0x7f1730101b30 2024-04-26T11:22:00.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.316+0000 7f17267fc700 1 -- 172.21.15.194:0/3356496142 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f172c01e070 con 0x7f1730101b30 2024-04-26T11:22:00.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.316+0000 7f17267fc700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f171c0a3c40 0x7f171c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:00.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.316+0000 7f17267fc700 1 -- 172.21.15.194:0/3356496142 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f172c0d0e10 con 0x7f1730101b30 2024-04-26T11:22:00.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.317+0000 7f1734d68700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f171c0a3c40 0x7f171c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:22:00.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.318+0000 7f1734d68700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f171c0a3c40 0x7f171c0a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f172000a9f0 tx=0x7f1720005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:22:00.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.320+0000 7f17267fc700 1 -- 172.21.15.194:0/3356496142 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f172c09e860 con 0x7f1730101b30 2024-04-26T11:22:00.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:00.525+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1730063860 con 0x7f171c0a3c40 2024-04-26T11:22:01.168 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:00 smithi194 ceph-mon[33017]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:01.169 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:00 smithi194 ceph-mon[33017]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:22:01.169 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:00 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:22:01.169 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:00 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:22:01.169 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:00 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:00 smithi033 ceph-mon[27845]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:00 smithi033 ceph-mon[27845]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:22:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:00 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:22:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:00 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:22:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:00 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:00 smithi096 ceph-mon[32753]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:00 smithi096 ceph-mon[32753]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:22:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:00 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:22:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:00 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:22:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:00 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:02 smithi194 ceph-mon[33017]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:02 smithi033 ceph-mon[27845]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:02 smithi096 ceph-mon[32753]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:03 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/3118704052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]: dispatch 2024-04-26T11:22:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:03 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]: dispatch 2024-04-26T11:22:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:03 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]': finished 2024-04-26T11:22:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:03 smithi194 ceph-mon[33017]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T11:22:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:03 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:03 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/3118704052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]: dispatch 2024-04-26T11:22:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:03 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]: dispatch 2024-04-26T11:22:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:03 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]': finished 2024-04-26T11:22:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:03 smithi033 ceph-mon[27845]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T11:22:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:03 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:03 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/3118704052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]: dispatch 2024-04-26T11:22:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:03 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]: dispatch 2024-04-26T11:22:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:03 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a138a4e3-39a6-4646-9eb8-27184d6dda61"}]': finished 2024-04-26T11:22:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:03 smithi096 ceph-mon[32753]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T11:22:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:03 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:04 smithi033 ceph-mon[27845]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:04 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/3253636856' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:22:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:04 smithi096 ceph-mon[32753]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:04 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/3253636856' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:22:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:04 smithi194 ceph-mon[33017]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:04 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/3253636856' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:22:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:06 smithi033 ceph-mon[27845]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:07.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:06 smithi096 ceph-mon[32753]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:06 smithi194 ceph-mon[33017]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:08 smithi033 ceph-mon[27845]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:08 smithi096 ceph-mon[32753]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:09.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:08 smithi194 ceph-mon[33017]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:10 smithi033 ceph-mon[27845]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:10 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T11:22:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:10 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:10 smithi033 ceph-mon[27845]: Deploying daemon osd.5 on smithi194 2024-04-26T11:22:11.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:10 smithi096 ceph-mon[32753]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:11.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:10 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T11:22:11.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:10 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:11.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:10 smithi096 ceph-mon[32753]: Deploying daemon osd.5 on smithi194 2024-04-26T11:22:11.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:10 smithi194 ceph-mon[33017]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:11.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:10 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T11:22:11.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:10 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:11.209 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:10 smithi194 ceph-mon[33017]: Deploying daemon osd.5 on smithi194 2024-04-26T11:22:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:12 smithi194 ceph-mon[33017]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:12 smithi033 ceph-mon[27845]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:12 smithi096 ceph-mon[32753]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:14 smithi033 ceph-mon[27845]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:14 smithi096 ceph-mon[32753]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:15.269 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:14 smithi194 ceph-mon[33017]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:17 smithi033 ceph-mon[27845]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:17 smithi096 ceph-mon[32753]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:17 smithi194 ceph-mon[33017]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:19.290 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:19 smithi194 ceph-mon[33017]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:19 smithi033 ceph-mon[27845]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:19 smithi096 ceph-mon[32753]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:19.663 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 5 on host 'smithi194' 2024-04-26T11:22:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.661+0000 7f17267fc700 1 -- 172.21.15.194:0/3356496142 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1730063860 con 0x7f171c0a3c40 2024-04-26T11:22:19.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f171c0a3c40 msgr2=0x7f171c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f171c0a3c40 0x7f171c0a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f172000a9f0 tx=0x7f1720005fb0 comp rx=0 tx=0).stop 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730101b30 msgr2=0x7f1730197420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730101b30 0x7f1730197420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f172c00d9e0 tx=0x7f172c00dda0 comp rx=0 tx=0).stop 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 shutdown_connections 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f171c0a3c40 0x7f171c0a6100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17301036d0 0x7f173019e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1730101b30 0x7f1730197420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 --2- 172.21.15.194:0/3356496142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1730102d30 0x7f1730197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 >> 172.21.15.194:0/3356496142 conn(0x7f17300fd0d0 msgr2=0x7f1730106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 shutdown_connections 2024-04-26T11:22:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:22:19.664+0000 7f17377cd700 1 -- 172.21.15.194:0/3356496142 wait complete. 2024-04-26T11:22:20.359 DEBUG:teuthology.orchestra.run.smithi194:osd.5> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T11:22:20.362 INFO:tasks.cephadm:Deploying osd.6 on smithi033 with /dev/vg_nvme/lv_2... 2024-04-26T11:22:20.362 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T11:22:20.828 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:22:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:20 smithi033 ceph-mon[27845]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:20 smithi033 ceph-mon[27845]: from='osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T11:22:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:20 smithi033 ceph-mon[27845]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T11:22:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:20.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:20 smithi096 ceph-mon[32753]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:20 smithi096 ceph-mon[32753]: from='osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T11:22:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:20 smithi096 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T11:22:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:20 smithi194 ceph-mon[33017]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:20 smithi194 ceph-mon[33017]: from='osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T11:22:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:20 smithi194 ceph-mon[33017]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T11:22:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:21 smithi033 ceph-mon[27845]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T11:22:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:21 smithi033 ceph-mon[27845]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T11:22:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:21 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:21 smithi033 ceph-mon[27845]: from='osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:22:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:21 smithi033 ceph-mon[27845]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:22:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:21 smithi096 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T11:22:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:21 smithi096 ceph-mon[32753]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T11:22:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:21 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:21 smithi096 ceph-mon[32753]: from='osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:22:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:21 smithi096 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:22:22.026 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:21 smithi194 ceph-mon[33017]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T11:22:22.026 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:21 smithi194 ceph-mon[33017]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T11:22:22.026 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:21 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:22.026 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:21 smithi194 ceph-mon[33017]: from='osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:22:22.026 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:21 smithi194 ceph-mon[33017]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:22:22.027 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 26 11:22:21 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-5[45087]: 2024-04-26T11:22:21.661+0000 7f41609b0700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T11:22:22.027 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 26 11:22:21 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-5[45087]: 2024-04-26T11:22:21.667+0000 7f415c7a3700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:22:22.857 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:22 smithi033 ceph-mon[27845]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:22:22.857 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:22 smithi033 ceph-mon[27845]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T11:22:22.857 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:22 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:22.857 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:22 smithi033 ceph-mon[27845]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:22 smithi096 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:22:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:22 smithi096 ceph-mon[32753]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T11:22:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:22 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:22 smithi096 ceph-mon[32753]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:22 smithi194 ceph-mon[33017]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:22:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:22 smithi194 ceph-mon[33017]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T11:22:23.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:22 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:23.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:22 smithi194 ceph-mon[33017]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T11:22:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:23 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:22:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:23 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:22:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:23 smithi194 ceph-mon[33017]: osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] boot 2024-04-26T11:22:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:23 smithi194 ceph-mon[33017]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T11:22:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:23 smithi194 ceph-mon[33017]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T11:22:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:23 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:22:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:23 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:22:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:23 smithi096 ceph-mon[32753]: osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] boot 2024-04-26T11:22:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:23 smithi096 ceph-mon[32753]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T11:22:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:23 smithi096 ceph-mon[32753]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T11:22:24.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:23 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:22:24.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:23 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:22:24.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:23 smithi033 ceph-mon[27845]: osd.5 [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] boot 2024-04-26T11:22:24.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:23 smithi033 ceph-mon[27845]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T11:22:24.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:22:24.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:23 smithi033 ceph-mon[27845]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T11:22:24.002 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T11:22:24.002 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T11:22:24.002 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-26T11:22:24.002 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-26T11:22:24.002 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0391841 s, 268 MB/s 2024-04-26T11:22:24.002 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 2024-04-26T11:22:24.002 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-26T11:22:25.018 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_2 2024-04-26T11:22:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi194 2024-04-26T11:22:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 8544M 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:24 smithi033 ceph-mon[27845]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi194 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 8544M 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:22:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:22:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:24 smithi096 ceph-mon[32753]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T11:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi194 2024-04-26T11:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 8544M 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:24 smithi194 ceph-mon[33017]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T11:22:25.474 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:22:27.138 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:26 smithi033 ceph-mon[27845]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:27.139 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:26 smithi033 ceph-mon[27845]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T11:22:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:26 smithi096 ceph-mon[32753]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:26 smithi096 ceph-mon[32753]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T11:22:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:26 smithi194 ceph-mon[33017]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:26 smithi194 ceph-mon[33017]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T11:22:27.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 -- 172.21.15.33:0/820548635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 msgr2=0x7fe3e0106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:27.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 --2- 172.21.15.33:0/820548635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 0x7fe3e0106cf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0099c0 tx=0x7fe3cc009cd0 comp rx=0 tx=0).stop 2024-04-26T11:22:27.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 -- 172.21.15.33:0/820548635 shutdown_connections 2024-04-26T11:22:27.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 --2- 172.21.15.33:0/820548635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0107230 0x7fe3e010dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:27.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 --2- 172.21.15.33:0/820548635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3e0105690 0x7fe3e0105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:27.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 --2- 172.21.15.33:0/820548635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 0x7fe3e0106cf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:27.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 -- 172.21.15.33:0/820548635 >> 172.21.15.33:0/820548635 conn(0x7fe3e0100c30 msgr2=0x7fe3e0103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:22:27.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 -- 172.21.15.33:0/820548635 shutdown_connections 2024-04-26T11:22:27.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.631+0000 7fe3e8400700 1 -- 172.21.15.33:0/820548635 wait complete. 2024-04-26T11:22:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.633+0000 7fe3e8400700 1 Processor -- start 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.633+0000 7fe3e8400700 1 -- start start 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.633+0000 7fe3e8400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0105690 0x7fe3e019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.633+0000 7fe3e8400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 0x7fe3e019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e8400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3e0107230 0x7fe3e01a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e8400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0111170 con 0x7fe3e0105690 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e8400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0110ff0 con 0x7fe3e0106890 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e8400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01112f0 con 0x7fe3e0107230 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0105690 0x7fe3e019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0105690 0x7fe3e019b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33648/0 (socket says 172.21.15.33:33648) 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 -- 172.21.15.33:0/2608460505 learned_addr learned my addr 172.21.15.33:0/2608460505 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5000700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 0x7fe3e019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:22:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e6400700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3e0107230 0x7fe3e01a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:22:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 -- 172.21.15.33:0/2608460505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3e0107230 msgr2=0x7fe3e01a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3e0107230 0x7fe3e01a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 -- 172.21.15.33:0/2608460505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 msgr2=0x7fe3e019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 0x7fe3e019bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5a00700 1 -- 172.21.15.33:0/2608460505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e01a3470 con 0x7fe3e0105690 2024-04-26T11:22:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.634+0000 7fe3e5000700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 0x7fe3e019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:22:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.635+0000 7fe3e5a00700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0105690 0x7fe3e019b810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400e980 tx=0x7fe3d400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:22:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.635+0000 7fe3de000700 1 -- 172.21.15.33:0/2608460505 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d400cbd0 con 0x7fe3e0105690 2024-04-26T11:22:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.635+0000 7fe3de000700 1 -- 172.21.15.33:0/2608460505 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3d40044e0 con 0x7fe3e0105690 2024-04-26T11:22:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.635+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e01a3760 con 0x7fe3e0105690 2024-04-26T11:22:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.635+0000 7fe3de000700 1 -- 172.21.15.33:0/2608460505 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4003e00 con 0x7fe3e0105690 2024-04-26T11:22:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.635+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e01a3c50 con 0x7fe3e0105690 2024-04-26T11:22:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.637+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e006a110 con 0x7fe3e0105690 2024-04-26T11:22:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.639+0000 7fe3de000700 1 -- 172.21.15.33:0/2608460505 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe3d4004000 con 0x7fe3e0105690 2024-04-26T11:22:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.640+0000 7fe3de000700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3c80a3c40 0x7fe3c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:22:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.640+0000 7fe3de000700 1 -- 172.21.15.33:0/2608460505 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7fe3d40cf940 con 0x7fe3e0105690 2024-04-26T11:22:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.641+0000 7fe3e5000700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3c80a3c40 0x7fe3c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:22:27.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.641+0000 7fe3e5000700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3c80a3c40 0x7fe3c80a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc000c00 tx=0x7fe3cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:22:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.645+0000 7fe3de000700 1 -- 172.21.15.33:0/2608460505 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3d400b2c0 con 0x7fe3e0105690 2024-04-26T11:22:27.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:27.886+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe3e0063860 con 0x7fe3c80a3c40 2024-04-26T11:22:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:27 smithi033 ceph-mon[27845]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T11:22:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:22:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:22:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:27 smithi194 ceph-mon[33017]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T11:22:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:22:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:22:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:27 smithi096 ceph-mon[32753]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T11:22:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:22:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:22:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:28 smithi033 ceph-mon[27845]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-04-26T11:22:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:28 smithi033 ceph-mon[27845]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:22:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:28 smithi194 ceph-mon[33017]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-04-26T11:22:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:28 smithi194 ceph-mon[33017]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:22:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:28 smithi096 ceph-mon[32753]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-04-26T11:22:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:28 smithi096 ceph-mon[32753]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:22:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:30 smithi194 ceph-mon[33017]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:22:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:30 smithi033 ceph-mon[27845]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:22:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:30 smithi096 ceph-mon[32753]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:22:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:31 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2727797351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]: dispatch 2024-04-26T11:22:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:31 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]: dispatch 2024-04-26T11:22:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:31 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]': finished 2024-04-26T11:22:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:31 smithi194 ceph-mon[33017]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T11:22:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:31 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2727797351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]: dispatch 2024-04-26T11:22:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:31 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]: dispatch 2024-04-26T11:22:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:31 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]': finished 2024-04-26T11:22:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:31 smithi033 ceph-mon[27845]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T11:22:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:31 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2727797351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]: dispatch 2024-04-26T11:22:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:31 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]: dispatch 2024-04-26T11:22:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:31 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79871b86-d637-4952-b923-615080fad108"}]': finished 2024-04-26T11:22:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:31 smithi096 ceph-mon[32753]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T11:22:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:32 smithi194 ceph-mon[33017]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T11:22:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:32 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2222890247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:22:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:32 smithi033 ceph-mon[27845]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T11:22:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:32 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2222890247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:22:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:32 smithi096 ceph-mon[32753]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T11:22:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:32 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2222890247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:22:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:34 smithi194 ceph-mon[33017]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:22:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:34 smithi033 ceph-mon[27845]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:22:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:34 smithi096 ceph-mon[32753]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:22:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:36 smithi194 ceph-mon[33017]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:36 smithi033 ceph-mon[27845]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:36 smithi096 ceph-mon[32753]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:39.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:38 smithi096 ceph-mon[32753]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:39.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:38 smithi033 ceph-mon[27845]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:38 smithi194 ceph-mon[33017]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:40 smithi033 ceph-mon[27845]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:40 smithi194 ceph-mon[33017]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:40 smithi096 ceph-mon[32753]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T11:22:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:41 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:41 smithi194 ceph-mon[33017]: Deploying daemon osd.6 on smithi033 2024-04-26T11:22:42.404 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T11:22:42.404 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:41 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:42.404 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:41 smithi033 ceph-mon[27845]: Deploying daemon osd.6 on smithi033 2024-04-26T11:22:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T11:22:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:41 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:22:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:41 smithi096 ceph-mon[32753]: Deploying daemon osd.6 on smithi033 2024-04-26T11:22:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:42 smithi033 ceph-mon[27845]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:42 smithi194 ceph-mon[33017]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:42 smithi096 ceph-mon[32753]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:44 smithi194 ceph-mon[33017]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:45.371 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:44 smithi033 ceph-mon[27845]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:44 smithi096 ceph-mon[32753]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:46 smithi194 ceph-mon[33017]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:47.367 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:46 smithi033 ceph-mon[27845]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:46 smithi096 ceph-mon[32753]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:49.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:49 smithi033 ceph-mon[27845]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:48 smithi194 ceph-mon[33017]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:49 smithi096 ceph-mon[32753]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:51.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:51 smithi033 ceph-mon[27845]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:51.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:51.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:51.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:51 smithi096 ceph-mon[32753]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:51 smithi194 ceph-mon[33017]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:22:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:53 smithi033 ceph-mon[27845]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:53 smithi096 ceph-mon[32753]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:53 smithi194 ceph-mon[33017]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:55 smithi033 ceph-mon[27845]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:55 smithi033 ceph-mon[27845]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T11:22:55.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.309+0000 7fe3de000700 1 -- 172.21.15.33:0/2608460505 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe3e0063860 con 0x7fe3c80a3c40 2024-04-26T11:22:55.311 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 6 on host 'smithi033' 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.313+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3c80a3c40 msgr2=0x7fe3c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.313+0000 7fe3e8400700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3c80a3c40 0x7fe3c80a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc000c00 tx=0x7fe3cc005fb0 comp rx=0 tx=0).stop 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0105690 msgr2=0x7fe3e019b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0105690 0x7fe3e019b810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400e980 tx=0x7fe3d400ec90 comp rx=0 tx=0).stop 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 shutdown_connections 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3c80a3c40 0x7fe3c80a6100 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3e0107230 0x7fe3e01a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3e0105690 0x7fe3e019b810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 --2- 172.21.15.33:0/2608460505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e0106890 0x7fe3e019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:22:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 >> 172.21.15.33:0/2608460505 conn(0x7fe3e0100c30 msgr2=0x7fe3e0109be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:22:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 shutdown_connections 2024-04-26T11:22:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:22:55.314+0000 7fe3e8400700 1 -- 172.21.15.33:0/2608460505 wait complete. 2024-04-26T11:22:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:55 smithi194 ceph-mon[33017]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:55 smithi194 ceph-mon[33017]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T11:22:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:55 smithi096 ceph-mon[32753]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:55 smithi096 ceph-mon[32753]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T11:22:56.103 DEBUG:teuthology.orchestra.run.smithi033:osd.6> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T11:22:56.109 INFO:tasks.cephadm:Deploying osd.7 on smithi096 with /dev/vg_nvme/lv_2... 2024-04-26T11:22:56.109 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T11:22:56.357 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:22:56.363 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:56.363 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:56.363 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-mon[27845]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T11:22:56.363 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-mon[27845]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T11:22:56.363 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-mon[27845]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:22:56.363 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:56 smithi096 ceph-mon[32753]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T11:22:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:56 smithi096 ceph-mon[32753]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T11:22:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:56 smithi096 ceph-mon[32753]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:22:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:22:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:56 smithi194 ceph-mon[33017]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T11:22:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:56 smithi194 ceph-mon[33017]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T11:22:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:56 smithi194 ceph-mon[33017]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:22:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:56.625 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-6[55629]: 2024-04-26T11:22:56.400+0000 7f9917400700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T11:22:56.626 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 26 11:22:56 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-6[55629]: 2024-04-26T11:22:56.419+0000 7f9912200700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:22:57.661 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:57 smithi096 ceph-mon[32753]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:57.661 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:57 smithi096 ceph-mon[32753]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:22:57.661 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:57 smithi096 ceph-mon[32753]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T11:22:57.661 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:57 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:57.662 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:57 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:57 smithi033 ceph-mon[27845]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:57 smithi033 ceph-mon[27845]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:22:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:57 smithi033 ceph-mon[27845]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T11:22:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:57 smithi194 ceph-mon[33017]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T11:22:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:57 smithi194 ceph-mon[33017]: from='osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:22:57.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:57 smithi194 ceph-mon[33017]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T11:22:57.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:57 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:57.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:57 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:58.345 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T11:22:58.345 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T11:22:58.345 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-26T11:22:58.345 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-26T11:22:58.346 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131841 s, 795 MB/s 2024-04-26T11:22:58.346 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-26T11:22:58.650 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:58 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:22:58.651 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:58 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:22:58.651 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:58 smithi096 ceph-mon[32753]: osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] boot 2024-04-26T11:22:58.651 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:58 smithi096 ceph-mon[32753]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T11:22:58.651 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:58 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:58 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:22:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:58 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:22:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:58 smithi033 ceph-mon[27845]: osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] boot 2024-04-26T11:22:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:58 smithi033 ceph-mon[27845]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T11:22:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:58 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:58 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:58 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:58 smithi194 ceph-mon[33017]: osd.6 [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] boot 2024-04-26T11:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:58 smithi194 ceph-mon[33017]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T11:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:58 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:22:59.004 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_2 2024-04-26T11:22:59.250 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:22:59.622 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:59 smithi096 ceph-mon[32753]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:22:59.623 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:22:59 smithi096 ceph-mon[32753]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T11:22:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:59 smithi033 ceph-mon[27845]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:22:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:22:59 smithi033 ceph-mon[27845]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T11:22:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:59 smithi194 ceph-mon[33017]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:22:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:22:59 smithi194 ceph-mon[33017]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 -- 172.21.15.96:0/1190432429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 msgr2=0x7f35601039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 --2- 172.21.15.96:0/1190432429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f35601039f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35540099c0 tx=0x7f3554009cd0 comp rx=0 tx=0).stop 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 -- 172.21.15.96:0/1190432429 shutdown_connections 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 --2- 172.21.15.96:0/1190432429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3560103f30 0x7f356010a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 --2- 172.21.15.96:0/1190432429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560102390 0x7f35601027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 --2- 172.21.15.96:0/1190432429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f35601039f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 -- 172.21.15.96:0/1190432429 >> 172.21.15.96:0/1190432429 conn(0x7f35600fd950 msgr2=0x7f35600ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:23:00.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 -- 172.21.15.96:0/1190432429 shutdown_connections 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 -- 172.21.15.96:0/1190432429 wait complete. 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 Processor -- start 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.896+0000 7f3566383700 1 -- start start 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f3566383700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560102390 0x7f356006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f3566383700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f356019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f3566383700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3560103f30 0x7f35601a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f3566383700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f356010dea0 con 0x7f3560102390 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f3566383700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f356010dd20 con 0x7f3560103590 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f3566383700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f356010e020 con 0x7f3560103f30 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f356019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f356019e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39244/0 (socket says 172.21.15.96:39244) 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 -- 172.21.15.96:0/1143168002 learned_addr learned my addr 172.21.15.96:0/1143168002 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f3564920700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3560103f30 0x7f35601a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:00.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 -- 172.21.15.96:0/1143168002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3560103f30 msgr2=0x7f35601a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3560103f30 0x7f35601a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 -- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560102390 msgr2=0x7f356006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560102390 0x7f356006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 -- 172.21.15.96:0/1143168002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35601a1350 con 0x7f3560103590 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.897+0000 7f355f7fe700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f356019e440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f355400b580 tx=0x7f3554009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.898+0000 7f355d7fa700 1 -- 172.21.15.96:0/1143168002 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554027070 con 0x7f3560103590 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.898+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35601a15e0 con 0x7f3560103590 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.898+0000 7f355d7fa700 1 -- 172.21.15.96:0/1143168002 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3554004120 con 0x7f3560103590 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.898+0000 7f355d7fa700 1 -- 172.21.15.96:0/1143168002 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554022d30 con 0x7f3560103590 2024-04-26T11:23:00.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.898+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35601a1b10 con 0x7f3560103590 2024-04-26T11:23:00.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.899+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f356006a110 con 0x7f3560103590 2024-04-26T11:23:00.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.900+0000 7f355d7fa700 1 -- 172.21.15.96:0/1143168002 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f355402a070 con 0x7f3560103590 2024-04-26T11:23:00.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.900+0000 7f355d7fa700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f35480a3c40 0x7f35480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:00.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.900+0000 7f355d7fa700 1 -- 172.21.15.96:0/1143168002 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f35540d0560 con 0x7f3560103590 2024-04-26T11:23:00.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.901+0000 7f355ffff700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f35480a3c40 0x7f35480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:00.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.902+0000 7f355ffff700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f35480a3c40 0x7f35480a6100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f35601033f0 tx=0x7f3550006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:23:00.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:00.905+0000 7f355d7fa700 1 -- 172.21.15.96:0/1143168002 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3554013070 con 0x7f3560103590 2024-04-26T11:23:01.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:01.112+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3560063860 con 0x7f35480a3c40 2024-04-26T11:23:01.657 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:01.657 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.657 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.657 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:01.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:01 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:01 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:01 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:02 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi033 2024-04-26T11:23:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:02 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 5698M 2024-04-26T11:23:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:02 smithi033 ceph-mon[27845]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:02 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi033 2024-04-26T11:23:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:02 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 5698M 2024-04-26T11:23:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:02 smithi096 ceph-mon[32753]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:02 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi033 2024-04-26T11:23:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:02 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 5698M 2024-04-26T11:23:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:02 smithi194 ceph-mon[33017]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:03 smithi033 ceph-mon[27845]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:03 smithi096 ceph-mon[32753]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:03 smithi194 ceph-mon[33017]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:04 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/996510523' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]: dispatch 2024-04-26T11:23:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:04 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]: dispatch 2024-04-26T11:23:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:04 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]': finished 2024-04-26T11:23:04.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:04 smithi096 ceph-mon[32753]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T11:23:04.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:04 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/996510523' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]: dispatch 2024-04-26T11:23:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:04 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]: dispatch 2024-04-26T11:23:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:04 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]': finished 2024-04-26T11:23:04.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:04 smithi194 ceph-mon[33017]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T11:23:04.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:04 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/996510523' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]: dispatch 2024-04-26T11:23:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:04 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]: dispatch 2024-04-26T11:23:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:04 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "514a0fd5-1191-4c6c-8724-d13762ffb9bd"}]': finished 2024-04-26T11:23:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:04 smithi033 ceph-mon[27845]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T11:23:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:05 smithi194 ceph-mon[33017]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:05 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/433578919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:05 smithi033 ceph-mon[27845]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:05 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/433578919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:05 smithi096 ceph-mon[32753]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:05 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/433578919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:07 smithi194 ceph-mon[33017]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:07.869 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:07 smithi096 ceph-mon[32753]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:07 smithi033 ceph-mon[27845]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:09.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:09 smithi096 ceph-mon[32753]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:09 smithi194 ceph-mon[33017]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:09 smithi033 ceph-mon[27845]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:11 smithi096 ceph-mon[32753]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:11 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T11:23:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:11 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:11 smithi194 ceph-mon[33017]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T11:23:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:11 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:11 smithi033 ceph-mon[27845]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:11 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T11:23:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:11 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:12 smithi194 ceph-mon[33017]: Deploying daemon osd.7 on smithi096 2024-04-26T11:23:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:12 smithi096 ceph-mon[32753]: Deploying daemon osd.7 on smithi096 2024-04-26T11:23:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:12 smithi033 ceph-mon[27845]: Deploying daemon osd.7 on smithi096 2024-04-26T11:23:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:13 smithi096 ceph-mon[32753]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:13 smithi194 ceph-mon[33017]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:13 smithi033 ceph-mon[27845]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:15 smithi096 ceph-mon[32753]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:15 smithi194 ceph-mon[33017]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:15 smithi033 ceph-mon[27845]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:17 smithi096 ceph-mon[32753]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:17.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:17 smithi033 ceph-mon[27845]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:17.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:17.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:17.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:17 smithi194 ceph-mon[33017]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:19 smithi096 ceph-mon[32753]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:19 smithi194 ceph-mon[33017]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:19 smithi033 ceph-mon[27845]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:20.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.594+0000 7f355d7fa700 1 -- 172.21.15.96:0/1143168002 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3560063860 con 0x7f35480a3c40 2024-04-26T11:23:20.597 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 7 on host 'smithi096' 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.596+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f35480a3c40 msgr2=0x7f35480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.596+0000 7f3566383700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f35480a3c40 0x7f35480a6100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f35601033f0 tx=0x7f3550006d20 comp rx=0 tx=0).stop 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.596+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 msgr2=0x7f356019e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.596+0000 7f3566383700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f356019e440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f355400b580 tx=0x7f3554009580 comp rx=0 tx=0).stop 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 shutdown_connections 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f35480a3c40 0x7f35480a6100 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3560103f30 0x7f35601a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560102390 0x7f356006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 --2- 172.21.15.96:0/1143168002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560103590 0x7f356019e440 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 >> 172.21.15.96:0/1143168002 conn(0x7f35600fd950 msgr2=0x7f3560106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 shutdown_connections 2024-04-26T11:23:20.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:23:20.597+0000 7f3566383700 1 -- 172.21.15.96:0/1143168002 wait complete. 2024-04-26T11:23:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:20.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T11:23:20.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T11:23:20.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:20.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:20.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T11:23:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T11:23:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:21.292 DEBUG:teuthology.orchestra.run.smithi096:osd.7> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T11:23:21.294 INFO:tasks.cephadm:Deploying osd.8 on smithi194 with /dev/vg_nvme/lv_2... 2024-04-26T11:23:21.294 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T11:23:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:23:21.802 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:21 smithi096 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T11:23:21.802 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:21 smithi096 ceph-mon[32753]: from='osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:23:21.802 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:21 smithi096 ceph-mon[32753]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T11:23:21.802 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:21 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:21.803 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:21 smithi096 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:23:21.803 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 26 11:23:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-7[50560]: 2024-04-26T11:23:21.768+0000 7fe3f1aed700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T11:23:21.803 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 26 11:23:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-7[50560]: 2024-04-26T11:23:21.780+0000 7fe3ed8e0700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:23:22.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:21 smithi194 ceph-mon[33017]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T11:23:22.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:21 smithi194 ceph-mon[33017]: from='osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:23:22.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:21 smithi194 ceph-mon[33017]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T11:23:22.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:21 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:22.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:21 smithi194 ceph-mon[33017]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:23:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:21 smithi033 ceph-mon[27845]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T11:23:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:21 smithi033 ceph-mon[27845]: from='osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:23:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:21 smithi033 ceph-mon[27845]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T11:23:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:21 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:21 smithi033 ceph-mon[27845]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:23:22.897 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:22 smithi194 ceph-mon[33017]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:22.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:22 smithi194 ceph-mon[33017]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:23:22.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:22 smithi194 ceph-mon[33017]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T11:23:22.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:22 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:22.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:22 smithi194 ceph-mon[33017]: osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] boot 2024-04-26T11:23:22.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:22 smithi194 ceph-mon[33017]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T11:23:22.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:22 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:23.044 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:22 smithi096 ceph-mon[32753]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:23.044 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:22 smithi096 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:23:23.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:22 smithi096 ceph-mon[32753]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T11:23:23.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:22 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:23.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:22 smithi096 ceph-mon[32753]: osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] boot 2024-04-26T11:23:23.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:22 smithi096 ceph-mon[32753]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T11:23:23.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:22 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:22 smithi033 ceph-mon[27845]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-26T11:23:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:22 smithi033 ceph-mon[27845]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:23:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:22 smithi033 ceph-mon[27845]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T11:23:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:22 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:22 smithi033 ceph-mon[27845]: osd.7 [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] boot 2024-04-26T11:23:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:22 smithi033 ceph-mon[27845]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T11:23:23.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:22 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:23:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T11:23:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T11:23:23.498 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-26T11:23:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-26T11:23:23.498 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131235 s, 799 MB/s 2024-04-26T11:23:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-26T11:23:23.910 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:23 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:23:23.910 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:23 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:23:23.911 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:23 smithi194 ceph-mon[33017]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T11:23:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:23 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:23:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:23 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:23:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:23 smithi096 ceph-mon[32753]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T11:23:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:23 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:23:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:23 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:23:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:23 smithi033 ceph-mon[27845]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T11:23:24.232 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-04-26T11:23:24.475 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:23:24.848 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: pgmap v194: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:24.848 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi096 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 5696M 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:24.849 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:24 smithi194 ceph-mon[33017]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T11:23:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: pgmap v194: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi096 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 5696M 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:24 smithi033 ceph-mon[27845]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: pgmap v194: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi096 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 5696M 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:25.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:24 smithi096 ceph-mon[32753]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T11:23:26.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 -- 172.21.15.194:0/901227447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710103480 msgr2=0x7f47101038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:26.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 --2- 172.21.15.194:0/901227447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710103480 0x7f47101038a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4700009990 tx=0x7f4700009ca0 comp rx=0 tx=0).stop 2024-04-26T11:23:26.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 -- 172.21.15.194:0/901227447 shutdown_connections 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 --2- 172.21.15.194:0/901227447 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4710105020 0x7f471010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 --2- 172.21.15.194:0/901227447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710103480 0x7f47101038a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 --2- 172.21.15.194:0/901227447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4710104680 0x7f4710104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 -- 172.21.15.194:0/901227447 >> 172.21.15.194:0/901227447 conn(0x7f47100fea20 msgr2=0x7f4710100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 -- 172.21.15.194:0/901227447 shutdown_connections 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 -- 172.21.15.194:0/901227447 wait complete. 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 Processor -- start 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.131+0000 7f471670c700 1 -- start start 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4710103480 0x7f4710199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4710104680 0x7f4710199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710105020 0x7f47101a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:26.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f471010eff0 con 0x7f4710103480 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f471010ee70 con 0x7f4710104680 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f471010f170 con 0x7f4710105020 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710105020 0x7f47101a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710105020 0x7f47101a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:47046/0 (socket says 172.21.15.194:47046) 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 -- 172.21.15.194:0/3518808817 learned_addr learned my addr 172.21.15.194:0/3518808817 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f470f7fe700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4710104680 0x7f4710199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 -- 172.21.15.194:0/3518808817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4710104680 msgr2=0x7f4710199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4710104680 0x7f4710199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 -- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4710103480 msgr2=0x7f4710199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4710103480 0x7f4710199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 -- 172.21.15.194:0/3518808817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47101a1320 con 0x7f4710105020 2024-04-26T11:23:26.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f4714ca9700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710105020 0x7f47101a0c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f470800b9a0 tx=0x7f470800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:23:26.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f470d7fa700 1 -- 172.21.15.194:0/3518808817 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708019070 con 0x7f4710105020 2024-04-26T11:23:26.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47101a1610 con 0x7f4710105020 2024-04-26T11:23:26.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f470d7fa700 1 -- 172.21.15.194:0/3518808817 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4708014440 con 0x7f4710105020 2024-04-26T11:23:26.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f470d7fa700 1 -- 172.21.15.194:0/3518808817 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470800ce60 con 0x7f4710105020 2024-04-26T11:23:26.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47101a1b80 con 0x7f4710105020 2024-04-26T11:23:26.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.133+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f471006a110 con 0x7f4710105020 2024-04-26T11:23:26.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.135+0000 7f470d7fa700 1 -- 172.21.15.194:0/3518808817 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f470801c070 con 0x7f4710105020 2024-04-26T11:23:26.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.135+0000 7f470d7fa700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46f80a3c40 0x7f46f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:26.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.135+0000 7f470d7fa700 1 -- 172.21.15.194:0/3518808817 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f47080cf1e0 con 0x7f4710105020 2024-04-26T11:23:26.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.136+0000 7f470ffff700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46f80a3c40 0x7f46f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:26.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.137+0000 7f470ffff700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46f80a3c40 0x7f46f80a6100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f47000098e0 tx=0x7f4700003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:23:26.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.139+0000 7f470d7fa700 1 -- 172.21.15.194:0/3518808817 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f470809c7b0 con 0x7f4710105020 2024-04-26T11:23:26.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:26.345+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4710063860 con 0x7f46f80a3c40 2024-04-26T11:23:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:26 smithi194 ceph-mon[33017]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:26 smithi194 ceph-mon[33017]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T11:23:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:26 smithi194 ceph-mon[33017]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:26 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:26 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:26 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:26 smithi033 ceph-mon[27845]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:26 smithi033 ceph-mon[27845]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T11:23:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:26 smithi033 ceph-mon[27845]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:26 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:26 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:26 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:26 smithi096 ceph-mon[32753]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:26 smithi096 ceph-mon[32753]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T11:23:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:26 smithi096 ceph-mon[32753]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:26 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:26 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:26 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:27 smithi194 ceph-mon[33017]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T11:23:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:27 smithi033 ceph-mon[27845]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T11:23:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:27 smithi096 ceph-mon[32753]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T11:23:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:28 smithi194 ceph-mon[33017]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:28 smithi033 ceph-mon[27845]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:28 smithi096 ceph-mon[32753]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:29 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/3176502526' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]: dispatch 2024-04-26T11:23:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:29 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]: dispatch 2024-04-26T11:23:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:29 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]': finished 2024-04-26T11:23:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:29 smithi194 ceph-mon[33017]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T11:23:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:29 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/1604811267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:29 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/3176502526' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]: dispatch 2024-04-26T11:23:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:29 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]: dispatch 2024-04-26T11:23:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:29 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]': finished 2024-04-26T11:23:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:29 smithi033 ceph-mon[27845]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T11:23:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:30.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:29 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/1604811267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:29 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/3176502526' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]: dispatch 2024-04-26T11:23:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:29 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]: dispatch 2024-04-26T11:23:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:29 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b44bf32-22e5-4c02-8be5-79db01d9f8a6"}]': finished 2024-04-26T11:23:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:29 smithi096 ceph-mon[32753]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T11:23:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:29 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/1604811267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:30 smithi194 ceph-mon[33017]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:30 smithi194 ceph-mon[33017]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T11:23:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:30 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:23:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:30 smithi033 ceph-mon[27845]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:30 smithi033 ceph-mon[27845]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T11:23:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:30 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:23:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:30 smithi096 ceph-mon[32753]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:30 smithi096 ceph-mon[32753]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T11:23:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:30 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:23:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:32 smithi194 ceph-mon[33017]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:32 smithi033 ceph-mon[27845]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:32 smithi096 ceph-mon[32753]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:34 smithi194 ceph-mon[33017]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:34 smithi033 ceph-mon[27845]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:34 smithi096 ceph-mon[32753]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:36 smithi194 ceph-mon[33017]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T11:23:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:36 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:36 smithi194 ceph-mon[33017]: Deploying daemon osd.8 on smithi194 2024-04-26T11:23:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:36 smithi033 ceph-mon[27845]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T11:23:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:36 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:36 smithi033 ceph-mon[27845]: Deploying daemon osd.8 on smithi194 2024-04-26T11:23:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:36 smithi096 ceph-mon[32753]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T11:23:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:36 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:36 smithi096 ceph-mon[32753]: Deploying daemon osd.8 on smithi194 2024-04-26T11:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:38 smithi194 ceph-mon[33017]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:38 smithi033 ceph-mon[27845]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:38 smithi096 ceph-mon[32753]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:40 smithi033 ceph-mon[27845]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:41.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:40 smithi096 ceph-mon[32753]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:41.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:40 smithi194 ceph-mon[33017]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:42.945 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:42 smithi194 ceph-mon[33017]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:42.946 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:42.946 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:42.946 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:42 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:42 smithi033 ceph-mon[27845]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:42 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:42 smithi096 ceph-mon[32753]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:42 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:44 smithi194 ceph-mon[33017]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:44 smithi033 ceph-mon[27845]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:45.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:44 smithi096 ceph-mon[32753]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:45.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.635+0000 7f470d7fa700 1 -- 172.21.15.194:0/3518808817 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4710063860 con 0x7f46f80a3c40 2024-04-26T11:23:45.637 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 8 on host 'smithi194' 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46f80a3c40 msgr2=0x7f46f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46f80a3c40 0x7f46f80a6100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f47000098e0 tx=0x7f4700003860 comp rx=0 tx=0).stop 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710105020 msgr2=0x7f47101a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710105020 0x7f47101a0c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f470800b9a0 tx=0x7f470800bd60 comp rx=0 tx=0).stop 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 shutdown_connections 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46f80a3c40 0x7f46f80a6100 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4710105020 0x7f47101a0c80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4710103480 0x7f4710199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 --2- 172.21.15.194:0/3518808817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4710104680 0x7f4710199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.637+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 >> 172.21.15.194:0/3518808817 conn(0x7f47100fea20 msgr2=0x7f4710107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:23:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.638+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 shutdown_connections 2024-04-26T11:23:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:23:45.638+0000 7f471670c700 1 -- 172.21.15.194:0/3518808817 wait complete. 2024-04-26T11:23:46.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:46 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:46.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:46 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:46.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:46 smithi194 ceph-mon[33017]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:46.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:46 smithi194 ceph-mon[33017]: from='osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T11:23:46.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:46 smithi194 ceph-mon[33017]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T11:23:46.740 DEBUG:teuthology.orchestra.run.smithi194:osd.8> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T11:23:46.742 INFO:tasks.cephadm:Deploying osd.9 on smithi033 with /dev/vg_nvme/lv_1... 2024-04-26T11:23:46.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T11:23:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:46 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:46 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:46 smithi033 ceph-mon[27845]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:46 smithi033 ceph-mon[27845]: from='osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T11:23:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:46 smithi033 ceph-mon[27845]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T11:23:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:46 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:46 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:46 smithi096 ceph-mon[32753]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:46 smithi096 ceph-mon[32753]: from='osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T11:23:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:46 smithi096 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T11:23:47.232 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:23:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:47 smithi033 ceph-mon[27845]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T11:23:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:47 smithi033 ceph-mon[27845]: from='osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:23:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:47 smithi033 ceph-mon[27845]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T11:23:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:47 smithi033 ceph-mon[27845]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:23:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:47.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:47 smithi194 ceph-mon[33017]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T11:23:47.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:47 smithi194 ceph-mon[33017]: from='osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:23:47.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:47 smithi194 ceph-mon[33017]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T11:23:47.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:47.743 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:47 smithi194 ceph-mon[33017]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:23:47.743 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:47.743 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:47 smithi096 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T11:23:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:47 smithi096 ceph-mon[32753]: from='osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:23:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:47 smithi096 ceph-mon[32753]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T11:23:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:47 smithi096 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:23:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:48.354 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-8[52081]: 2024-04-26T11:23:48.020+0000 7f43c7bc3700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T11:23:48.354 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-8[52081]: 2024-04-26T11:23:48.031+0000 7f43bf9ae700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:23:48.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:23:48.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T11:23:48.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:48.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:48.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: from='osd.8 ' entity='osd.8' 2024-04-26T11:23:48.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] boot 2024-04-26T11:23:48.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T11:23:48.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:48 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:48.913 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:23:48.913 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T11:23:48.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:48.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:48.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: from='osd.8 ' entity='osd.8' 2024-04-26T11:23:48.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] boot 2024-04-26T11:23:48.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T11:23:48.915 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:48 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: from='osd.8 ' entity='osd.8' 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: osd.8 [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] boot 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T11:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:48 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:23:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:49 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:23:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:49 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:23:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:49 smithi033 ceph-mon[27845]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T11:23:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:49 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:23:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:49 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:23:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:49 smithi096 ceph-mon[32753]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T11:23:50.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:49 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:23:50.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:49 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:23:50.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:49 smithi194 ceph-mon[33017]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T11:23:50.153 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T11:23:50.154 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T11:23:50.154 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-26T11:23:50.154 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-26T11:23:50.154 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0382803 s, 274 MB/s 2024-04-26T11:23:50.154 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-26T11:23:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:23:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi194 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T11:23:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi194 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T11:23:50.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi194 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T11:23:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:51.098 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_1 2024-04-26T11:23:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:23:51.905 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:51 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 5696M 2024-04-26T11:23:51.905 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:51.905 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:51.905 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:51.905 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:51 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 5696M 2024-04-26T11:23:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:51 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 5696M 2024-04-26T11:23:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:23:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:52.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:23:52.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:23:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:52 smithi033 ceph-mon[27845]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:23:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:52 smithi033 ceph-mon[27845]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T11:23:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:52 smithi194 ceph-mon[33017]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:23:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:52 smithi194 ceph-mon[33017]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T11:23:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:52 smithi096 ceph-mon[32753]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:23:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:52 smithi096 ceph-mon[32753]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T11:23:53.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.722+0000 7f1386200700 1 -- 172.21.15.33:0/2469040891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 msgr2=0x7f138010b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:53.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.722+0000 7f1386200700 1 --2- 172.21.15.33:0/2469040891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 0x7f138010b120 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f13680099c0 tx=0x7f1368009cd0 comp rx=0 tx=0).stop 2024-04-26T11:23:53.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.722+0000 7f1386200700 1 -- 172.21.15.33:0/2469040891 shutdown_connections 2024-04-26T11:23:53.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.722+0000 7f1386200700 1 --2- 172.21.15.33:0/2469040891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f138010b660 0x7f1380111f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:53.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.722+0000 7f1386200700 1 --2- 172.21.15.33:0/2469040891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 0x7f1380109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:53.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.722+0000 7f1386200700 1 --2- 172.21.15.33:0/2469040891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 0x7f138010b120 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:53.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.722+0000 7f1386200700 1 -- 172.21.15.33:0/2469040891 >> 172.21.15.33:0/2469040891 conn(0x7f1380078a90 msgr2=0x7f138007aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:23:53.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.723+0000 7f1386200700 1 -- 172.21.15.33:0/2469040891 shutdown_connections 2024-04-26T11:23:53.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.723+0000 7f1386200700 1 -- 172.21.15.33:0/2469040891 wait complete. 2024-04-26T11:23:53.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.724+0000 7f1386200700 1 Processor -- start 2024-04-26T11:23:53.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.724+0000 7f1386200700 1 -- start start 2024-04-26T11:23:53.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f1386200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 0x7f138019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:53.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f1386200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 0x7f13801a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:53.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f1386200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f138010b660 0x7f13801a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:53.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f1386200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1380115580 con 0x7f138010b660 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f1386200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1380115400 con 0x7f138010acc0 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f1386200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1380115700 con 0x7f1380109ac0 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 0x7f138019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 0x7f138019fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:44540/0 (socket says 172.21.15.33:44540) 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 -- 172.21.15.33:0/4105826264 learned_addr learned my addr 172.21.15.33:0/4105826264 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137fe00700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f138010b660 0x7f13801a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f1377e00700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 0x7f13801a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 -- 172.21.15.33:0/4105826264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 msgr2=0x7f13801a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 0x7f13801a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 -- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f138010b660 msgr2=0x7f13801a7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f138010b660 0x7f13801a7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:23:53.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.725+0000 7f137f400700 1 -- 172.21.15.33:0/4105826264 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13801a7860 con 0x7f1380109ac0 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.726+0000 7f1377e00700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 0x7f13801a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.726+0000 7f137f400700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 0x7f138019fc10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f137000e970 tx=0x7f137000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.726+0000 7f137cc00700 1 -- 172.21.15.33:0/4105826264 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370004220 con 0x7f1380109ac0 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.726+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13801a7b50 con 0x7f1380109ac0 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.726+0000 7f137cc00700 1 -- 172.21.15.33:0/4105826264 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13700043c0 con 0x7f1380109ac0 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.726+0000 7f137cc00700 1 -- 172.21.15.33:0/4105826264 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370003ce0 con 0x7f1380109ac0 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.726+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13801a80e0 con 0x7f1380109ac0 2024-04-26T11:23:53.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.727+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138006a110 con 0x7f1380109ac0 2024-04-26T11:23:53.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.730+0000 7f137cc00700 1 -- 172.21.15.33:0/4105826264 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f137000c7e0 con 0x7f1380109ac0 2024-04-26T11:23:53.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.731+0000 7f137cc00700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13640a3c40 0x7f13640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:23:53.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.731+0000 7f137cc00700 1 -- 172.21.15.33:0/4105826264 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f13700cfed0 con 0x7f1380109ac0 2024-04-26T11:23:53.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.731+0000 7f1377e00700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13640a3c40 0x7f13640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:23:53.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.732+0000 7f1377e00700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13640a3c40 0x7f13640a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1368000c00 tx=0x7f1368005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:23:53.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.733+0000 7f137cc00700 1 -- 172.21.15.33:0/4105826264 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f137009d1c0 con 0x7f1380109ac0 2024-04-26T11:23:53.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:23:53.983+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1380063860 con 0x7f13640a3c40 2024-04-26T11:23:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:53 smithi194 ceph-mon[33017]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T11:23:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:53 smithi033 ceph-mon[27845]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T11:23:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:53 smithi096 ceph-mon[32753]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T11:23:55.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:54 smithi033 ceph-mon[27845]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-04-26T11:23:55.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:54 smithi033 ceph-mon[27845]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:55.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:55.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:55.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:54 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:54 smithi194 ceph-mon[33017]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-04-26T11:23:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:54 smithi194 ceph-mon[33017]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:54 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:54 smithi096 ceph-mon[32753]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-04-26T11:23:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:54 smithi096 ceph-mon[32753]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:23:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:23:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:23:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:54 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:23:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:56 smithi194 ceph-mon[33017]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:23:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:56 smithi033 ceph-mon[27845]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:23:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:56 smithi096 ceph-mon[32753]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T11:23:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:57 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3858886616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]: dispatch 2024-04-26T11:23:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:57 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]: dispatch 2024-04-26T11:23:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:57 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]': finished 2024-04-26T11:23:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:57 smithi194 ceph-mon[33017]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T11:23:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:57 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:23:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:57 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3858886616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]: dispatch 2024-04-26T11:23:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:57 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]: dispatch 2024-04-26T11:23:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:57 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]': finished 2024-04-26T11:23:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:57 smithi033 ceph-mon[27845]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T11:23:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:57 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:23:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:57 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3858886616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]: dispatch 2024-04-26T11:23:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:57 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]: dispatch 2024-04-26T11:23:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:57 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a0a3c0f-0405-46c7-bff5-886d3232f60c"}]': finished 2024-04-26T11:23:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:57 smithi096 ceph-mon[32753]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T11:23:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:57 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:23:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:58 smithi033 ceph-mon[27845]: pgmap v223: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T11:23:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:23:58 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3630178004' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:58 smithi194 ceph-mon[33017]: pgmap v223: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T11:23:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:23:58 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3630178004' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:23:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:58 smithi096 ceph-mon[32753]: pgmap v223: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T11:23:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:23:58 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3630178004' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:24:01.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:00 smithi194 ceph-mon[33017]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:24:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:00 smithi033 ceph-mon[27845]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:24:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:00 smithi096 ceph-mon[32753]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T11:24:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:03 smithi194 ceph-mon[33017]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:03 smithi033 ceph-mon[27845]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:03 smithi096 ceph-mon[32753]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:05 smithi194 ceph-mon[33017]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:05 smithi033 ceph-mon[27845]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:05 smithi096 ceph-mon[32753]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:07 smithi194 ceph-mon[33017]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:07 smithi033 ceph-mon[27845]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:07 smithi096 ceph-mon[32753]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T11:24:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:08 smithi194 ceph-mon[33017]: Deploying daemon osd.9 on smithi033 2024-04-26T11:24:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T11:24:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:08 smithi033 ceph-mon[27845]: Deploying daemon osd.9 on smithi033 2024-04-26T11:24:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T11:24:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:08 smithi096 ceph-mon[32753]: Deploying daemon osd.9 on smithi033 2024-04-26T11:24:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:09 smithi033 ceph-mon[27845]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:09 smithi194 ceph-mon[33017]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:09 smithi096 ceph-mon[32753]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:11 smithi033 ceph-mon[27845]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:11 smithi194 ceph-mon[33017]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:11 smithi096 ceph-mon[32753]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:13.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:13 smithi033 ceph-mon[27845]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:13 smithi194 ceph-mon[33017]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:13 smithi096 ceph-mon[32753]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:15 smithi033 ceph-mon[27845]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:15 smithi194 ceph-mon[33017]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:15 smithi096 ceph-mon[32753]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:17 smithi033 ceph-mon[27845]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:17 smithi096 ceph-mon[32753]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:17 smithi194 ceph-mon[33017]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:19.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:19 smithi033 ceph-mon[27845]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:19 smithi096 ceph-mon[32753]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:19 smithi194 ceph-mon[33017]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:21.042 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 9 on host 'smithi033' 2024-04-26T11:24:21.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.039+0000 7f137cc00700 1 -- 172.21.15.33:0/4105826264 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1380063860 con 0x7f13640a3c40 2024-04-26T11:24:21.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.042+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13640a3c40 msgr2=0x7f13640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:21.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.042+0000 7f1386200700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13640a3c40 0x7f13640a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1368000c00 tx=0x7f1368005fb0 comp rx=0 tx=0).stop 2024-04-26T11:24:21.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 msgr2=0x7f138019fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:21.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 0x7f138019fc10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f137000e970 tx=0x7f137000ec80 comp rx=0 tx=0).stop 2024-04-26T11:24:21.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 shutdown_connections 2024-04-26T11:24:21.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13640a3c40 0x7f13640a6100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:21.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f138010b660 0x7f13801a7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:21.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1380109ac0 0x7f138019fc10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:21.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 --2- 172.21.15.33:0/4105826264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f138010acc0 0x7f13801a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:21.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 >> 172.21.15.33:0/4105826264 conn(0x7f1380078a90 msgr2=0x7f138010e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:24:21.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 shutdown_connections 2024-04-26T11:24:21.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:24:21.043+0000 7f1386200700 1 -- 172.21.15.33:0/4105826264 wait complete. 2024-04-26T11:24:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:20 smithi033 ceph-mon[27845]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:20 smithi033 ceph-mon[27845]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T11:24:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:20 smithi194 ceph-mon[33017]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:20 smithi194 ceph-mon[33017]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T11:24:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:20 smithi096 ceph-mon[32753]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:20 smithi096 ceph-mon[32753]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T11:24:21.825 DEBUG:teuthology.orchestra.run.smithi033:osd.9> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T11:24:21.832 INFO:tasks.cephadm:Deploying osd.10 on smithi096 with /dev/vg_nvme/lv_1... 2024-04-26T11:24:21.832 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T11:24:22.081 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:24:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:21 smithi033 ceph-mon[27845]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T11:24:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:21 smithi033 ceph-mon[27845]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T11:24:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:21 smithi033 ceph-mon[27845]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:24:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:21 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:21 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:21 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:21 smithi096 ceph-mon[32753]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T11:24:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:21 smithi096 ceph-mon[32753]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T11:24:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:21 smithi096 ceph-mon[32753]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:24:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:21 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:21 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:21 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:21 smithi194 ceph-mon[33017]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T11:24:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:21 smithi194 ceph-mon[33017]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T11:24:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:21 smithi194 ceph-mon[33017]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-26T11:24:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:21 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:21 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:21 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:22.843 INFO:journalctl@ceph.osd.9.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-9[62198]: 2024-04-26T11:24:22.623+0000 7f2d79e00700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T11:24:22.843 INFO:journalctl@ceph.osd.9.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-9[62198]: 2024-04-26T11:24:22.662+0000 7f2d6fc00700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:24:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:22 smithi096 ceph-mon[32753]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:22 smithi096 ceph-mon[32753]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:24:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:22 smithi096 ceph-mon[32753]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T11:24:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:22 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:22 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:22 smithi096 ceph-mon[32753]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' 2024-04-26T11:24:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:22 smithi194 ceph-mon[33017]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:22 smithi194 ceph-mon[33017]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:24:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:22 smithi194 ceph-mon[33017]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T11:24:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:22 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:22 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:22 smithi194 ceph-mon[33017]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' 2024-04-26T11:24:23.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-mon[27845]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:23.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-mon[27845]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-26T11:24:23.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-mon[27845]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T11:24:23.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:23.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:23.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:22 smithi033 ceph-mon[27845]: from='osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107]' entity='osd.9' 2024-04-26T11:24:24.130 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:23 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:24:24.130 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:23 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:24:24.130 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:24.130 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:23 smithi033 ceph-mon[27845]: osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] boot 2024-04-26T11:24:24.130 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:23 smithi033 ceph-mon[27845]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T11:24:24.130 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:24.203 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:23 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:24:24.203 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:23 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:24:24.203 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:24.203 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:23 smithi096 ceph-mon[32753]: osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] boot 2024-04-26T11:24:24.203 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:23 smithi096 ceph-mon[32753]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T11:24:24.203 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:24.204 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T11:24:24.204 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T11:24:24.204 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-26T11:24:24.204 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-26T11:24:24.204 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0175671 s, 597 MB/s 2024-04-26T11:24:24.204 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-26T11:24:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:23 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:24:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:23 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:24:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:23 smithi194 ceph-mon[33017]: osd.9 [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] boot 2024-04-26T11:24:24.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:23 smithi194 ceph-mon[33017]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T11:24:24.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:24:24.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_1 2024-04-26T11:24:25.146 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T11:24:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:24 smithi096 ceph-mon[32753]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:24 smithi194 ceph-mon[33017]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:24 smithi033 ceph-mon[27845]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-26T11:24:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:25 smithi033 ceph-mon[27845]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T11:24:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:25 smithi194 ceph-mon[33017]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T11:24:26.357 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:25 smithi096 ceph-mon[32753]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 -- 172.21.15.96:0/3703380530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 msgr2=0x7f6b68104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3703380530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68104af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b500099c0 tx=0x7f6b50009cd0 comp rx=0 tx=0).stop 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 -- 172.21.15.96:0/3703380530 shutdown_connections 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3703380530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b68105030 0x7f6b6810b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3703380530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b68103490 0x7f6b681038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3703380530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68104af0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 -- 172.21.15.96:0/3703380530 >> 172.21.15.96:0/3703380530 conn(0x7f6b680fea30 msgr2=0x7f6b68100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 -- 172.21.15.96:0/3703380530 shutdown_connections 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.887+0000 7f6b6db96700 1 -- 172.21.15.96:0/3703380530 wait complete. 2024-04-26T11:24:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 Processor -- start 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 -- start start 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b68103490 0x7f6b680771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b68105030 0x7f6b68075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6810f020 con 0x7f6b68103490 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6810eea0 con 0x7f6b68104690 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b6db96700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6810f1a0 con 0x7f6b68105030 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68075810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:37566/0 (socket says 172.21.15.96:37566) 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 -- 172.21.15.96:0/3299867060 learned_addr learned my addr 172.21.15.96:0/3299867060 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b67fff700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b68105030 0x7f6b68075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 -- 172.21.15.96:0/3299867060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b68105030 msgr2=0x7f6b68075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:26.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b68105030 0x7f6b68075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 -- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b68103490 msgr2=0x7f6b680771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b68103490 0x7f6b680771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 -- 172.21.15.96:0/3299867060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b68076600 con 0x7f6b68104690 2024-04-26T11:24:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.888+0000 7f6b66ffd700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68075810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b50006010 tx=0x7f6b500047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:24:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.889+0000 7f6b64ff9700 1 -- 172.21.15.96:0/3299867060 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50003d70 con 0x7f6b68104690 2024-04-26T11:24:26.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.889+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b681a37f0 con 0x7f6b68104690 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.889+0000 7f6b64ff9700 1 -- 172.21.15.96:0/3299867060 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b50004830 con 0x7f6b68104690 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.889+0000 7f6b64ff9700 1 -- 172.21.15.96:0/3299867060 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50005550 con 0x7f6b68104690 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.889+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b681a3c60 con 0x7f6b68104690 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.890+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6806a110 con 0x7f6b68104690 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.891+0000 7f6b64ff9700 1 -- 172.21.15.96:0/3299867060 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6b50029060 con 0x7f6b68104690 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.891+0000 7f6b64ff9700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b540a3c40 0x7f6b540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.891+0000 7f6b64ff9700 1 -- 172.21.15.96:0/3299867060 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f6b500d0c80 con 0x7f6b68104690 2024-04-26T11:24:26.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.892+0000 7f6b677fe700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b540a3c40 0x7f6b540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:24:26.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.893+0000 7f6b677fe700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b540a3c40 0x7f6b540a6100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6b681044f0 tx=0x7f6b60009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:24:26.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:26.895+0000 7f6b64ff9700 1 -- 172.21.15.96:0/3299867060 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b5005dff0 con 0x7f6b68104690 2024-04-26T11:24:27.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:27.105+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6b68063860 con 0x7f6b540a3c40 2024-04-26T11:24:27.116 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:27 smithi033 ceph-mon[27845]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:27 smithi194 ceph-mon[33017]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:27 smithi096 ceph-mon[32753]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:28.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:24:28.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:24:28.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:24:28.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi033 2024-04-26T11:24:28.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 4273M 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:28.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:24:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:24:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:24:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi033 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 4273M 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi033 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 4273M 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:28.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:28.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:29.281 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:29 smithi096 ceph-mon[32753]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:29 smithi194 ceph-mon[33017]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:29 smithi033 ceph-mon[27845]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:30 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/2894482317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]: dispatch 2024-04-26T11:24:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:30 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]: dispatch 2024-04-26T11:24:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:30 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]': finished 2024-04-26T11:24:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:30 smithi096 ceph-mon[32753]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T11:24:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:30 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:30 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/2894482317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]: dispatch 2024-04-26T11:24:30.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:30 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]: dispatch 2024-04-26T11:24:30.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:30 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]': finished 2024-04-26T11:24:30.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:30 smithi194 ceph-mon[33017]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T11:24:30.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:30 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:30 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/2894482317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]: dispatch 2024-04-26T11:24:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:30 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]: dispatch 2024-04-26T11:24:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:30 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a7b1104-145f-4393-bf38-a75dcaf06b8d"}]': finished 2024-04-26T11:24:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:30 smithi033 ceph-mon[27845]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T11:24:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:30 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:31 smithi194 ceph-mon[33017]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:31 smithi194 ceph-mon[33017]: from='client.? 172.21.15.96:0/3192994790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:24:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:31 smithi033 ceph-mon[27845]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:31 smithi033 ceph-mon[27845]: from='client.? 172.21.15.96:0/3192994790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:24:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:31 smithi096 ceph-mon[32753]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:31 smithi096 ceph-mon[32753]: from='client.? 172.21.15.96:0/3192994790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:24:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:33 smithi194 ceph-mon[33017]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:33 smithi033 ceph-mon[27845]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:33 smithi096 ceph-mon[32753]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:35 smithi096 ceph-mon[32753]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:35 smithi194 ceph-mon[33017]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:35 smithi033 ceph-mon[27845]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:37 smithi194 ceph-mon[33017]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:37 smithi033 ceph-mon[27845]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:37 smithi096 ceph-mon[32753]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:38.313 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:38 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T11:24:38.313 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:38 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:38.313 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:38 smithi096 ceph-mon[32753]: Deploying daemon osd.10 on smithi096 2024-04-26T11:24:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:38 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T11:24:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:38 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:38 smithi194 ceph-mon[33017]: Deploying daemon osd.10 on smithi096 2024-04-26T11:24:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:38 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T11:24:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:38 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:38 smithi033 ceph-mon[27845]: Deploying daemon osd.10 on smithi096 2024-04-26T11:24:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:39 smithi194 ceph-mon[33017]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:39.360 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:39 smithi096 ceph-mon[32753]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:39 smithi033 ceph-mon[27845]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:41.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:41 smithi096 ceph-mon[32753]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:41 smithi194 ceph-mon[33017]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:41 smithi033 ceph-mon[27845]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:43 smithi096 ceph-mon[32753]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:43 smithi194 ceph-mon[33017]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:43 smithi033 ceph-mon[27845]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:45 smithi194 ceph-mon[33017]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:45.611 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:45 smithi096 ceph-mon[32753]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:45 smithi033 ceph-mon[27845]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:47.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.251+0000 7f6b64ff9700 1 -- 172.21.15.96:0/3299867060 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6b68063860 con 0x7f6b540a3c40 2024-04-26T11:24:47.253 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 10 on host 'smithi096' 2024-04-26T11:24:47.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b540a3c40 msgr2=0x7f6b540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:47.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b540a3c40 0x7f6b540a6100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6b681044f0 tx=0x7f6b60009250 comp rx=0 tx=0).stop 2024-04-26T11:24:47.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 msgr2=0x7f6b68075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68075810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b50006010 tx=0x7f6b500047b0 comp rx=0 tx=0).stop 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 shutdown_connections 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b540a3c40 0x7f6b540a6100 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b68105030 0x7f6b68075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b68103490 0x7f6b680771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 --2- 172.21.15.96:0/3299867060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b68104690 0x7f6b68075810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 >> 172.21.15.96:0/3299867060 conn(0x7f6b680fea30 msgr2=0x7f6b68107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 shutdown_connections 2024-04-26T11:24:47.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-26T11:24:47.253+0000 7f6b6db96700 1 -- 172.21.15.96:0/3299867060 wait complete. 2024-04-26T11:24:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T11:24:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.919 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T11:24:47.920 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T11:24:47.920 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T11:24:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T11:24:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:24:47.948 DEBUG:teuthology.orchestra.run.smithi096:osd.10> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T11:24:47.950 INFO:tasks.cephadm:Deploying osd.11 on smithi194 with /dev/vg_nvme/lv_1... 2024-04-26T11:24:47.951 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T11:24:48.198 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:24:49.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-mon[32753]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:49.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T11:24:49.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-mon[32753]: from='osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:24:49.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-mon[32753]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T11:24:49.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:49.403 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:24:49.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:49 smithi194 ceph-mon[33017]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:49.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:49 smithi194 ceph-mon[33017]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T11:24:49.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:49 smithi194 ceph-mon[33017]: from='osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:24:49.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:49 smithi194 ceph-mon[33017]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T11:24:49.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:49.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:49 smithi194 ceph-mon[33017]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:24:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:49 smithi033 ceph-mon[27845]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:49 smithi033 ceph-mon[27845]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T11:24:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:49 smithi033 ceph-mon[27845]: from='osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:24:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:49 smithi033 ceph-mon[27845]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T11:24:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:49 smithi033 ceph-mon[27845]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-26T11:24:49.933 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-10[57159]: 2024-04-26T11:24:49.667+0000 7f6a701f8700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T11:24:49.933 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 26 11:24:49 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-10[57159]: 2024-04-26T11:24:49.683+0000 7f6a67fe3700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:24:50.257 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T11:24:50.257 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T11:24:50.257 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-26T11:24:50.257 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-26T11:24:50.257 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117698 s, 891 MB/s 2024-04-26T11:24:50.257 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-26T11:24:50.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:50 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:24:50.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:50 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:24:50.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:50 smithi194 ceph-mon[33017]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:24:50.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:50 smithi194 ceph-mon[33017]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T11:24:50.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:50.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:50.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:50 smithi194 ceph-mon[33017]: from='osd.10 ' entity='osd.10' 2024-04-26T11:24:50.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:50 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:24:50.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:50 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:24:50.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:50 smithi096 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:24:50.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:50 smithi096 ceph-mon[32753]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T11:24:50.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:50.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:50.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:50 smithi096 ceph-mon[32753]: from='osd.10 ' entity='osd.10' 2024-04-26T11:24:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:50 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:24:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:50 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:24:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:50 smithi033 ceph-mon[27845]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-26T11:24:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:50 smithi033 ceph-mon[27845]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T11:24:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:50 smithi033 ceph-mon[27845]: from='osd.10 ' entity='osd.10' 2024-04-26T11:24:50.912 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-04-26T11:24:51.159 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:24:51.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] boot 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T11:24:51.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:51.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:51.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:51 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] boot 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T11:24:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:51.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:51.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:51 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: osd.10 [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] boot 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:24:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:51.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T11:24:51.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:51.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:24:51.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:51 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:24:52.334 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:52 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi096 2024-04-26T11:24:52.334 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:52 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 4272M 2024-04-26T11:24:52.334 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:52 smithi194 ceph-mon[33017]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T11:24:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:52 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi096 2024-04-26T11:24:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:52 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 4272M 2024-04-26T11:24:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:52 smithi033 ceph-mon[27845]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T11:24:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:52 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi096 2024-04-26T11:24:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:52 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 4272M 2024-04-26T11:24:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:52 smithi096 ceph-mon[32753]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T11:24:52.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 -- 172.21.15.194:0/837897942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408109180 msgr2=0x7f240810b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 --2- 172.21.15.194:0/837897942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408109180 0x7f240810b5f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009990 tx=0x7f23f8009ca0 comp rx=0 tx=0).stop 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 -- 172.21.15.194:0/837897942 shutdown_connections 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 --2- 172.21.15.194:0/837897942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408109180 0x7f240810b5f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 --2- 172.21.15.194:0/837897942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2408103f20 0x7f2408106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 --2- 172.21.15.194:0/837897942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2408106850 0x7f2408108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 -- 172.21.15.194:0/837897942 >> 172.21.15.194:0/837897942 conn(0x7f24080fd950 msgr2=0x7f24080ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 -- 172.21.15.194:0/837897942 shutdown_connections 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 -- 172.21.15.194:0/837897942 wait complete. 2024-04-26T11:24:52.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 Processor -- start 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.815+0000 7f240db54700 1 -- start start 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408103f20 0x7f2408199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2408106850 0x7f2408199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2408109180 0x7f24081a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f240810ef90 con 0x7f2408106850 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f240810ee10 con 0x7f2408109180 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f240810f110 con 0x7f2408103f20 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408103f20 0x7f2408199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408103f20 0x7f2408199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:41530/0 (socket says 172.21.15.194:41530) 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 -- 172.21.15.194:0/1928082405 learned_addr learned my addr 172.21.15.194:0/1928082405 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f2407fff700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2408109180 0x7f24081a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 -- 172.21.15.194:0/1928082405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2408109180 msgr2=0x7f24081a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:52.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2408109180 0x7f24081a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 -- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2408106850 msgr2=0x7f2408199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2408106850 0x7f2408199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 -- 172.21.15.194:0/1928082405 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24081a1290 con 0x7f2408103f20 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f2407fff700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2408109180 0x7f24081a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f24077fe700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408103f20 0x7f2408199630 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f240000add0 tx=0x7f240000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f2404ff9700 1 -- 172.21.15.194:0/1928082405 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400017020 con 0x7f2408103f20 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24081a1580 con 0x7f2408103f20 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f2404ff9700 1 -- 172.21.15.194:0/1928082405 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f240000cdf0 con 0x7f2408103f20 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f2404ff9700 1 -- 172.21.15.194:0/1928082405 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400005520 con 0x7f2408103f20 2024-04-26T11:24:52.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.816+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24081a1b40 con 0x7f2408103f20 2024-04-26T11:24:52.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.817+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f240806a110 con 0x7f2408103f20 2024-04-26T11:24:52.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.818+0000 7f2404ff9700 1 -- 172.21.15.194:0/1928082405 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2400004ad0 con 0x7f2408103f20 2024-04-26T11:24:52.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.819+0000 7f2404ff9700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23f00a3c40 0x7f23f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:24:52.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.819+0000 7f2404ff9700 1 -- 172.21.15.194:0/1928082405 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f24000cfb80 con 0x7f2408103f20 2024-04-26T11:24:52.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.820+0000 7f2406ffd700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23f00a3c40 0x7f23f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:24:52.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.820+0000 7f2406ffd700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23f00a3c40 0x7f23f00a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0097b0 tx=0x7f23fc006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:24:52.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:52.823+0000 7f2404ff9700 1 -- 172.21.15.194:0/1928082405 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2400021020 con 0x7f2408103f20 2024-04-26T11:24:53.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:24:53.031+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2408063860 con 0x7f23f00a3c40 2024-04-26T11:24:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:53 smithi194 ceph-mon[33017]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:24:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:24:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:24:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:53 smithi033 ceph-mon[27845]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:24:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:24:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:24:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:53 smithi096 ceph-mon[32753]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:24:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T11:24:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T11:24:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:24:54.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:54 smithi194 ceph-mon[33017]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:24:54.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:54 smithi194 ceph-mon[33017]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T11:24:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:54 smithi033 ceph-mon[27845]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:24:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:54 smithi033 ceph-mon[27845]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T11:24:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:54 smithi096 ceph-mon[32753]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:24:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:54 smithi096 ceph-mon[32753]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T11:24:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:55 smithi194 ceph-mon[33017]: pgmap v263: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T11:24:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:55 smithi194 ceph-mon[33017]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T11:24:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:55 smithi033 ceph-mon[27845]: pgmap v263: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T11:24:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:55 smithi033 ceph-mon[27845]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T11:24:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:55 smithi096 ceph-mon[32753]: pgmap v263: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T11:24:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:55 smithi096 ceph-mon[32753]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T11:24:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:56 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/3595158010' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]: dispatch 2024-04-26T11:24:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:56 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]: dispatch 2024-04-26T11:24:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:56 smithi194 ceph-mon[33017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]': finished 2024-04-26T11:24:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:56 smithi194 ceph-mon[33017]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T11:24:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:24:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:56 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/3595158010' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]: dispatch 2024-04-26T11:24:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:56 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]: dispatch 2024-04-26T11:24:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:56 smithi033 ceph-mon[27845]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]': finished 2024-04-26T11:24:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:56 smithi033 ceph-mon[27845]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T11:24:56.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:24:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:56 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/3595158010' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]: dispatch 2024-04-26T11:24:56.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:56 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]: dispatch 2024-04-26T11:24:56.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:56 smithi096 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58375031-d243-4316-9c22-1edf99890668"}]': finished 2024-04-26T11:24:56.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:56 smithi096 ceph-mon[32753]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T11:24:56.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:24:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:57 smithi194 ceph-mon[33017]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T11:24:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:57 smithi194 ceph-mon[33017]: from='client.? 172.21.15.194:0/347561414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:24:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:57 smithi033 ceph-mon[27845]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T11:24:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:57 smithi033 ceph-mon[27845]: from='client.? 172.21.15.194:0/347561414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:24:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:57 smithi096 ceph-mon[32753]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T11:24:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:57 smithi096 ceph-mon[32753]: from='client.? 172.21.15.194:0/347561414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T11:24:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:24:59 smithi194 ceph-mon[33017]: pgmap v267: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T11:24:59.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:24:59 smithi033 ceph-mon[27845]: pgmap v267: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T11:24:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:24:59 smithi096 ceph-mon[32753]: pgmap v267: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T11:25:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:01 smithi194 ceph-mon[33017]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:01 smithi033 ceph-mon[27845]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:01 smithi096 ceph-mon[32753]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:03 smithi194 ceph-mon[33017]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:03 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T11:25:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:03 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:03 smithi033 ceph-mon[27845]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:03 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T11:25:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:03 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:03.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:03 smithi096 ceph-mon[32753]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:03.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:03 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T11:25:03.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:03 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:04 smithi194 ceph-mon[33017]: Deploying daemon osd.11 on smithi194 2024-04-26T11:25:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:04 smithi033 ceph-mon[27845]: Deploying daemon osd.11 on smithi194 2024-04-26T11:25:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:04 smithi096 ceph-mon[32753]: Deploying daemon osd.11 on smithi194 2024-04-26T11:25:05.579 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:05 smithi194 ceph-mon[33017]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:05 smithi033 ceph-mon[27845]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:05 smithi096 ceph-mon[32753]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:07 smithi194 ceph-mon[33017]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:07 smithi033 ceph-mon[27845]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:07 smithi096 ceph-mon[32753]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:09 smithi194 ceph-mon[33017]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:09 smithi033 ceph-mon[27845]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:09 smithi096 ceph-mon[32753]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:11 smithi194 ceph-mon[33017]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:11 smithi033 ceph-mon[27845]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:11 smithi096 ceph-mon[32753]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:12.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.268+0000 7f2404ff9700 1 -- 172.21.15.194:0/1928082405 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2408063860 con 0x7f23f00a3c40 2024-04-26T11:25:12.270 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 11 on host 'smithi194' 2024-04-26T11:25:12.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23f00a3c40 msgr2=0x7f23f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:12.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23f00a3c40 0x7f23f00a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0097b0 tx=0x7f23fc006d20 comp rx=0 tx=0).stop 2024-04-26T11:25:12.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408103f20 msgr2=0x7f2408199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:12.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408103f20 0x7f2408199630 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f240000add0 tx=0x7f240000c330 comp rx=0 tx=0).stop 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 shutdown_connections 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23f00a3c40 0x7f23f00a6100 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2408109180 0x7f24081a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2408103f20 0x7f2408199630 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 --2- 172.21.15.194:0/1928082405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2408106850 0x7f2408199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.270+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 >> 172.21.15.194:0/1928082405 conn(0x7f24080fd950 msgr2=0x7f2408104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.271+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 shutdown_connections 2024-04-26T11:25:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:12.271+0000 7f240db54700 1 -- 172.21.15.194:0/1928082405 wait complete. 2024-04-26T11:25:13.143 DEBUG:teuthology.orchestra.run.smithi194:osd.11> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T11:25:13.146 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T11:25:13.146 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T11:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:13 smithi194 ceph-mon[33017]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:13 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:13 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:13 smithi194 ceph-mon[33017]: from='osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T11:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:13 smithi194 ceph-mon[33017]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T11:25:13.629 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:13 smithi033 ceph-mon[27845]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:13 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:13 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:13 smithi033 ceph-mon[27845]: from='osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T11:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:13 smithi033 ceph-mon[27845]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T11:25:13.678 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:13 smithi096 ceph-mon[32753]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:13.678 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:13 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:13.679 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:13 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:13.679 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:13 smithi096 ceph-mon[32753]: from='osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T11:25:13.679 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:13 smithi096 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T11:25:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-mon[33017]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T11:25:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-mon[33017]: from='osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:25:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-mon[33017]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T11:25:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-mon[33017]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:25:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:14.605 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-11[59151]: 2024-04-26T11:25:14.354+0000 7f159dd4a700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T11:25:14.606 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 26 11:25:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-osd-11[59151]: 2024-04-26T11:25:14.371+0000 7f1599b3d700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T11:25:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:14 smithi033 ceph-mon[27845]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T11:25:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:14 smithi033 ceph-mon[27845]: from='osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:25:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:14 smithi033 ceph-mon[27845]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T11:25:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:14 smithi033 ceph-mon[27845]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:25:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:14 smithi096 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T11:25:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:14 smithi096 ceph-mon[32753]: from='osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:25:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:14 smithi096 ceph-mon[32753]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T11:25:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:14 smithi096 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-26T11:25:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:15 smithi033 ceph-mon[27845]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:15.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:15 smithi033 ceph-mon[27845]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:25:15.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:15 smithi033 ceph-mon[27845]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T11:25:15.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:15.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:15 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:15.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:15 smithi194 ceph-mon[33017]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:15.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:15 smithi194 ceph-mon[33017]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:25:15.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:15 smithi194 ceph-mon[33017]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T11:25:15.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:15.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:15 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:15 smithi096 ceph-mon[32753]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:15 smithi096 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-26T11:25:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:15 smithi096 ceph-mon[32753]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T11:25:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:15 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:15.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.987+0000 7f2f1ca00700 1 -- 172.21.15.33:0/1399240702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181034b0 msgr2=0x7f2f181038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:15.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.987+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/1399240702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181034b0 0x7f2f181038d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800b0a0 tx=0x7f2f0800b3b0 comp rx=0 tx=0).stop 2024-04-26T11:25:15.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.987+0000 7f2f1ca00700 1 -- 172.21.15.33:0/1399240702 shutdown_connections 2024-04-26T11:25:15.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.987+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/1399240702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f18105050 0x7f2f1810b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:15.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.987+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/1399240702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181034b0 0x7f2f181038d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:15.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.987+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/1399240702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f181046b0 0x7f2f18104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:15.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.987+0000 7f2f1ca00700 1 -- 172.21.15.33:0/1399240702 >> 172.21.15.33:0/1399240702 conn(0x7f2f180fea70 msgr2=0x7f2f18100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:15.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.988+0000 7f2f1ca00700 1 -- 172.21.15.33:0/1399240702 shutdown_connections 2024-04-26T11:25:15.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.988+0000 7f2f1ca00700 1 -- 172.21.15.33:0/1399240702 wait complete. 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.989+0000 7f2f1ca00700 1 Processor -- start 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.989+0000 7f2f1ca00700 1 -- start start 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f1ca00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f181034b0 0x7f2f1819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f1ca00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181046b0 0x7f2f1819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f1ca00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f18105050 0x7f2f181a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f1ca00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810ef40 con 0x7f2f181034b0 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f1ca00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810edc0 con 0x7f2f18105050 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f1ca00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810f0c0 con 0x7f2f181046b0 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f16800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f18105050 0x7f2f181a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f16800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f18105050 0x7f2f181a2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33762/0 (socket says 172.21.15.33:33762) 2024-04-26T11:25:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f16800700 1 -- 172.21.15.33:0/2044197921 learned_addr learned my addr 172.21.15.33:0/2044197921 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:15.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f15400700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181046b0 0x7f2f1819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.990+0000 7f2f15e00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f181034b0 0x7f2f1819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f16800700 1 -- 172.21.15.33:0/2044197921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181046b0 msgr2=0x7f2f1819bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f16800700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181046b0 0x7f2f1819bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f16800700 1 -- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f181034b0 msgr2=0x7f2f1819b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f16800700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f181034b0 0x7f2f1819b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f16800700 1 -- 172.21.15.33:0/2044197921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f181a3450 con 0x7f2f18105050 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f16800700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f18105050 0x7f2f181a2db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000d980 tx=0x7f2f1000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f0ea00700 1 -- 172.21.15.33:0/2044197921 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10016070 con 0x7f2f18105050 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f0ea00700 1 -- 172.21.15.33:0/2044197921 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f10004830 con 0x7f2f18105050 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f0ea00700 1 -- 172.21.15.33:0/2044197921 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f100054e0 con 0x7f2f18105050 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f181a3740 con 0x7f2f18105050 2024-04-26T11:25:15.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.991+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f181a3c30 con 0x7f2f18105050 2024-04-26T11:25:15.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.993+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1806a110 con 0x7f2f18105050 2024-04-26T11:25:15.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.995+0000 7f2f0ea00700 1 -- 172.21.15.33:0/2044197921 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2f100049e0 con 0x7f2f18105050 2024-04-26T11:25:15.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.996+0000 7f2f0ea00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2efc0a3cc0 0x7f2efc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:15.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.996+0000 7f2f0ea00700 1 -- 172.21.15.33:0/2044197921 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f2f100cfcc0 con 0x7f2f18105050 2024-04-26T11:25:15.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.996+0000 7f2f15e00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2efc0a3cc0 0x7f2efc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:15.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:15.997+0000 7f2f15e00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2efc0a3cc0 0x7f2efc0a6180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009fd0 tx=0x7f2f0800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:16.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.001+0000 7f2f0ea00700 1 -- 172.21.15.33:0/2044197921 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f100d6080 con 0x7f2f18105050 2024-04-26T11:25:16.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.231+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2f181a45c0 con 0x7f2f18105050 2024-04-26T11:25:16.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.232+0000 7f2f0ea00700 1 -- 172.21.15.33:0/2044197921 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f2f1001f020 con 0x7f2f18105050 2024-04-26T11:25:16.234 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.234+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2efc0a3cc0 msgr2=0x7f2efc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.234+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2efc0a3cc0 0x7f2efc0a6180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009fd0 tx=0x7f2f0800b7b0 comp rx=0 tx=0).stop 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f18105050 msgr2=0x7f2f181a2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f18105050 0x7f2f181a2db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000d980 tx=0x7f2f1000dc90 comp rx=0 tx=0).stop 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 shutdown_connections 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2efc0a3cc0 0x7f2efc0a6180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f18105050 0x7f2f181a2db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f181034b0 0x7f2f1819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 --2- 172.21.15.33:0/2044197921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f181046b0 0x7f2f1819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 >> 172.21.15.33:0/2044197921 conn(0x7f2f180fea70 msgr2=0x7f2f18075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 shutdown_connections 2024-04-26T11:25:16.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:16.235+0000 7f2f1ca00700 1 -- 172.21.15.33:0/2044197921 wait complete. 2024-04-26T11:25:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:16 smithi194 ceph-mon[33017]: purged_snaps scrub starts 2024-04-26T11:25:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:16 smithi194 ceph-mon[33017]: purged_snaps scrub ok 2024-04-26T11:25:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:16 smithi194 ceph-mon[33017]: osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] boot 2024-04-26T11:25:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:16 smithi194 ceph-mon[33017]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T11:25:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:16 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:16 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2044197921' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T11:25:16.597 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:16 smithi033 ceph-mon[27845]: purged_snaps scrub starts 2024-04-26T11:25:16.597 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:16 smithi033 ceph-mon[27845]: purged_snaps scrub ok 2024-04-26T11:25:16.597 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:16.597 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:16 smithi033 ceph-mon[27845]: osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] boot 2024-04-26T11:25:16.598 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:16 smithi033 ceph-mon[27845]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T11:25:16.598 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:16 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:16.598 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:16 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2044197921' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T11:25:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:16 smithi096 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T11:25:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:16 smithi096 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T11:25:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:16 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:16 smithi096 ceph-mon[32753]: osd.11 [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] boot 2024-04-26T11:25:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:16 smithi096 ceph-mon[32753]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T11:25:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:16 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:25:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:16 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2044197921' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T11:25:17.035 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1714130715,"num_in_osds":12,"osd_in_since":1714130696,"num_remapped_pgs":0} 2024-04-26T11:25:17.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:25:17.450 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T11:25:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:17 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:17.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:17.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:17 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T11:25:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T11:25:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:17.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:17 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:18 smithi033 ceph-mon[27845]: Detected new or changed devices on smithi194 2024-04-26T11:25:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:18 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 4272M 2024-04-26T11:25:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:18 smithi096 ceph-mon[32753]: Detected new or changed devices on smithi194 2024-04-26T11:25:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:18 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 4272M 2024-04-26T11:25:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:18 smithi194 ceph-mon[33017]: Detected new or changed devices on smithi194 2024-04-26T11:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:18 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 4272M 2024-04-26T11:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:18.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 -- 172.21.15.33:0/1065537166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4101eb0 msgr2=0x7fa2f4102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 --2- 172.21.15.33:0/1065537166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4101eb0 0x7fa2f4102290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009990 tx=0x7fa2e4009ca0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 -- 172.21.15.33:0/1065537166 shutdown_connections 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 --2- 172.21.15.33:0/1065537166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2f4108720 0x7fa2f41102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 --2- 172.21.15.33:0/1065537166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4101eb0 0x7fa2f4102290 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 --2- 172.21.15.33:0/1065537166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2f4102860 0x7fa2f4108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 -- 172.21.15.33:0/1065537166 >> 172.21.15.33:0/1065537166 conn(0x7fa2f40775d0 msgr2=0x7fa2f40779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.615+0000 7fa2f3200700 1 -- 172.21.15.33:0/1065537166 shutdown_connections 2024-04-26T11:25:19.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.616+0000 7fa2f3200700 1 -- 172.21.15.33:0/1065537166 wait complete. 2024-04-26T11:25:19.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 Processor -- start 2024-04-26T11:25:19.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 -- start start 2024-04-26T11:25:19.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2f4101eb0 0x7fa2f41a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4102860 0x7fa2f41a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2f4108720 0x7fa2f419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f41135f0 con 0x7fa2f4102860 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f4113470 con 0x7fa2f4101eb0 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.617+0000 7fa2f3200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f4113770 con 0x7fa2f4108720 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4102860 0x7fa2f41a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2f4101eb0 0x7fa2f41a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4102860 0x7fa2f41a1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52824/0 (socket says 172.21.15.33:52824) 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1400700 1 -- 172.21.15.33:0/2629115894 learned_addr learned my addr 172.21.15.33:0/2629115894 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f2800700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2f4108720 0x7fa2f419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1e00700 1 -- 172.21.15.33:0/2629115894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2f4108720 msgr2=0x7fa2f419ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1e00700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2f4108720 0x7fa2f419ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1e00700 1 -- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4102860 msgr2=0x7fa2f41a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1e00700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4102860 0x7fa2f41a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1e00700 1 -- 172.21.15.33:0/2629115894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f419c1a0 con 0x7fa2f4101eb0 2024-04-26T11:25:19.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.618+0000 7fa2f1400700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4102860 0x7fa2f41a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:25:19.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.619+0000 7fa2f1e00700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2f4101eb0 0x7fa2f41a1880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4000c00 tx=0x7fa2e40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:19.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.619+0000 7fa2eaa00700 1 -- 172.21.15.33:0/2629115894 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4026070 con 0x7fa2f4101eb0 2024-04-26T11:25:19.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.619+0000 7fa2eaa00700 1 -- 172.21.15.33:0/2629115894 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e4021410 con 0x7fa2f4101eb0 2024-04-26T11:25:19.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.619+0000 7fa2eaa00700 1 -- 172.21.15.33:0/2629115894 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4017650 con 0x7fa2f4101eb0 2024-04-26T11:25:19.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.619+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f419c430 con 0x7fa2f4101eb0 2024-04-26T11:25:19.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.619+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f41a8750 con 0x7fa2f4101eb0 2024-04-26T11:25:19.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.620+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f406a110 con 0x7fa2f4101eb0 2024-04-26T11:25:19.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.622+0000 7fa2eaa00700 1 -- 172.21.15.33:0/2629115894 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa2e4029070 con 0x7fa2f4101eb0 2024-04-26T11:25:19.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.623+0000 7fa2eaa00700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa2d80a3cc0 0x7fa2d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:19.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.624+0000 7fa2eaa00700 1 -- 172.21.15.33:0/2629115894 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa2e40d0eb0 con 0x7fa2f4101eb0 2024-04-26T11:25:19.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.624+0000 7fa2f1400700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa2d80a3cc0 0x7fa2d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:19.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.624+0000 7fa2f1400700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa2d80a3cc0 0x7fa2d80a6180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4104e50 tx=0x7fa2e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:19.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.626+0000 7fa2eaa00700 1 -- 172.21.15.33:0/2629115894 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2e4013070 con 0x7fa2f4101eb0 2024-04-26T11:25:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:19 smithi033 ceph-mon[27845]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:19 smithi096 ceph-mon[32753]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:19 smithi194 ceph-mon[33017]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:19.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.863+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2f41a8920 con 0x7fa2f4101eb0 2024-04-26T11:25:19.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.865+0000 7fa2eaa00700 1 -- 172.21.15.33:0/2629115894 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18405 (secure 0 0 0) 0x7fa2e409dd20 con 0x7fa2f4101eb0 2024-04-26T11:25:19.867 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:25:19.868 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":81,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:25:16.365135+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:25:19.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa2d80a3cc0 msgr2=0x7fa2d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:19.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa2d80a3cc0 0x7fa2d80a6180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4104e50 tx=0x7fa2e0009260 comp rx=0 tx=0).stop 2024-04-26T11:25:19.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2f4101eb0 msgr2=0x7fa2f41a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:19.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2f4101eb0 0x7fa2f41a1880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4000c00 tx=0x7fa2e40047b0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 shutdown_connections 2024-04-26T11:25:19.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa2d80a3cc0 0x7fa2d80a6180 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2f4108720 0x7fa2f419ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2f4101eb0 0x7fa2f41a1880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 --2- 172.21.15.33:0/2629115894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2f4102860 0x7fa2f41a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:19.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 >> 172.21.15.33:0/2629115894 conn(0x7fa2f40775d0 msgr2=0x7fa2f40ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:19.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 shutdown_connections 2024-04-26T11:25:19.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:19.868+0000 7fa2f3200700 1 -- 172.21.15.33:0/2629115894 wait complete. 2024-04-26T11:25:20.593 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:20 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2629115894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:25:20.666 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T11:20:59.833195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T11:25:20.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T11:25:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:20 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2629115894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:25:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:20 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2629115894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:25:21.148 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:21.598 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:21 smithi033 ceph-mon[27845]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:21 smithi096 ceph-mon[32753]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:21 smithi194 ceph-mon[33017]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.279+0000 7f4976000700 1 -- 172.21.15.33:0/1993531710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010fa50 msgr2=0x7f497010fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.279+0000 7f4976000700 1 --2- 172.21.15.33:0/1993531710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010fa50 0x7f497010fe30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49600098f0 tx=0x7f4960009c00 comp rx=0 tx=0).stop 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.280+0000 7f4976000700 1 -- 172.21.15.33:0/1993531710 shutdown_connections 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.280+0000 7f4976000700 1 --2- 172.21.15.33:0/1993531710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f497010a410 0x7f4970116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.280+0000 7f4976000700 1 --2- 172.21.15.33:0/1993531710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010fa50 0x7f497010fe30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.280+0000 7f4976000700 1 --2- 172.21.15.33:0/1993531710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4970109a50 0x7f4970109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.280+0000 7f4976000700 1 -- 172.21.15.33:0/1993531710 >> 172.21.15.33:0/1993531710 conn(0x7f4970076bf0 msgr2=0x7f4970108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.280+0000 7f4976000700 1 -- 172.21.15.33:0/1993531710 shutdown_connections 2024-04-26T11:25:23.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.280+0000 7f4976000700 1 -- 172.21.15.33:0/1993531710 wait complete. 2024-04-26T11:25:23.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 Processor -- start 2024-04-26T11:25:23.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 -- start start 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4970109a50 0x7f49701a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010a410 0x7f49701a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f497010fa50 0x7f49701a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4970119c50 con 0x7f497010a410 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4970119ad0 con 0x7f4970109a50 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.281+0000 7f4976000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4970119dd0 con 0x7f497010fa50 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010a410 0x7f49701a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010a410 0x7f49701a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46510/0 (socket says 172.21.15.33:46510) 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 -- 172.21.15.33:0/1565315870 learned_addr learned my addr 172.21.15.33:0/1565315870 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f496fe00700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f497010fa50 0x7f49701a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:23.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f496f400700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4970109a50 0x7f49701a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:23.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 -- 172.21.15.33:0/1565315870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f497010fa50 msgr2=0x7f49701a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:23.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f497010fa50 0x7f49701a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 -- 172.21.15.33:0/1565315870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4970109a50 msgr2=0x7f49701a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:23.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4970109a50 0x7f49701a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.282+0000 7f4967e00700 1 -- 172.21.15.33:0/1565315870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49701a5220 con 0x7f497010a410 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.283+0000 7f4967e00700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010a410 0x7f49701a0750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f495800e9c0 tx=0x7f495800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.283+0000 7f496cc00700 1 -- 172.21.15.33:0/1565315870 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495800cbd0 con 0x7f497010a410 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.283+0000 7f496cc00700 1 -- 172.21.15.33:0/1565315870 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f495800ee60 con 0x7f497010a410 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.283+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49701a5510 con 0x7f497010a410 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.283+0000 7f496cc00700 1 -- 172.21.15.33:0/1565315870 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49580055c0 con 0x7f497010a410 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.283+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49701acd90 con 0x7f497010a410 2024-04-26T11:25:23.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.284+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497006a110 con 0x7f497010a410 2024-04-26T11:25:23.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.289+0000 7f496cc00700 1 -- 172.21.15.33:0/1565315870 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4958004750 con 0x7f497010a410 2024-04-26T11:25:23.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.290+0000 7f496cc00700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f49540a3c40 0x7f49540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:23.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.291+0000 7f496cc00700 1 -- 172.21.15.33:0/1565315870 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f49580d0470 con 0x7f497010a410 2024-04-26T11:25:23.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.291+0000 7f496f400700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f49540a3c40 0x7f49540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:23.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.292+0000 7f496f400700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f49540a3c40 0x7f49540a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4960009fd0 tx=0x7f4960019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:23.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.293+0000 7f496cc00700 1 -- 172.21.15.33:0/1565315870 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4958053ff0 con 0x7f497010a410 2024-04-26T11:25:23.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.526+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f49701a15a0 con 0x7f497010a410 2024-04-26T11:25:23.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.527+0000 7f496cc00700 1 -- 172.21.15.33:0/1565315870 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f49701a15a0 con 0x7f497010a410 2024-04-26T11:25:23.529 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-04-26T11:25:23.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f49540a3c40 msgr2=0x7f49540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:23.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f49540a3c40 0x7f49540a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4960009fd0 tx=0x7f4960019040 comp rx=0 tx=0).stop 2024-04-26T11:25:23.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010a410 msgr2=0x7f49701a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010a410 0x7f49701a0750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f495800e9c0 tx=0x7f495800ecd0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 shutdown_connections 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f49540a3c40 0x7f49540a6100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f497010fa50 0x7f49701a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4970109a50 0x7f49701a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 --2- 172.21.15.33:0/1565315870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f497010a410 0x7f49701a0750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.530+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 >> 172.21.15.33:0/1565315870 conn(0x7f4970076bf0 msgr2=0x7f49700772d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.531+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 shutdown_connections 2024-04-26T11:25:23.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:23.531+0000 7f4976000700 1 -- 172.21.15.33:0/1565315870 wait complete. 2024-04-26T11:25:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:23 smithi033 ceph-mon[27845]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:23 smithi096 ceph-mon[32753]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:23 smithi194 ceph-mon[33017]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.a on smithi033 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.d on smithi033 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.g on smithi033 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.j on smithi033 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.b on smithi096 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.e on smithi096 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.h on smithi096 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.k on smithi096 2024-04-26T11:25:24.377 INFO:tasks.cephadm:Adding mds.c on smithi194 2024-04-26T11:25:24.378 INFO:tasks.cephadm:Adding mds.f on smithi194 2024-04-26T11:25:24.378 INFO:tasks.cephadm:Adding mds.i on smithi194 2024-04-26T11:25:24.378 INFO:tasks.cephadm:Adding mds.l on smithi194 2024-04-26T11:25:24.378 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l' 2024-04-26T11:25:24.626 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T11:25:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:24 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1565315870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T11:25:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:24 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1565315870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T11:25:24.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:24 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1565315870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T11:25:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:25 smithi033 ceph-mon[27845]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:25 smithi096 ceph-mon[32753]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:25.785 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:25 smithi194 ceph-mon[33017]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:26.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 -- 172.21.15.194:0/1915710531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc101790 msgr2=0x7f5ccc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:26.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 --2- 172.21.15.194:0/1915710531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc101790 0x7f5ccc101bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009990 tx=0x7f5cbc009ca0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 -- 172.21.15.194:0/1915710531 shutdown_connections 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 --2- 172.21.15.194:0/1915710531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ccc1051b0 0x7f5ccc10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 --2- 172.21.15.194:0/1915710531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc101790 0x7f5ccc101bb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 --2- 172.21.15.194:0/1915710531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ccc104810 0x7f5ccc104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 -- 172.21.15.194:0/1915710531 >> 172.21.15.194:0/1915710531 conn(0x7f5ccc0fd0d0 msgr2=0x7f5ccc0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 -- 172.21.15.194:0/1915710531 shutdown_connections 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 -- 172.21.15.194:0/1915710531 wait complete. 2024-04-26T11:25:26.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.304+0000 7f5cd1212700 1 Processor -- start 2024-04-26T11:25:26.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cd1212700 1 -- start start 2024-04-26T11:25:26.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cd1212700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ccc101790 0x7f5ccc199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:26.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cd1212700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ccc104810 0x7f5ccc199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:26.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cd1212700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc1051b0 0x7f5ccc1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:26.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cd1212700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc10efd0 con 0x7f5ccc101790 2024-04-26T11:25:26.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cd1212700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc10ee50 con 0x7f5ccc104810 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cd1212700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc10f150 con 0x7f5ccc1051b0 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc1051b0 0x7f5ccc1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc1051b0 0x7f5ccc1a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48168/0 (socket says 172.21.15.194:48168) 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 -- 172.21.15.194:0/971993988 learned_addr learned my addr 172.21.15.194:0/971993988 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5cc259c700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ccc104810 0x7f5ccc199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 -- 172.21.15.194:0/971993988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ccc104810 msgr2=0x7f5ccc199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ccc104810 0x7f5ccc199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 -- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ccc101790 msgr2=0x7f5ccc199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:26.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ccc101790 0x7f5ccc199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.305+0000 7f5ccb59e700 1 -- 172.21.15.194:0/971993988 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc1a12a0 con 0x7f5ccc1051b0 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.306+0000 7f5ccb59e700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc1051b0 0x7f5ccc1a0c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007f10 tx=0x7f5cc400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.306+0000 7f5cc8d99700 1 -- 172.21.15.194:0/971993988 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4004bb0 con 0x7f5ccc1051b0 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.306+0000 7f5cc8d99700 1 -- 172.21.15.194:0/971993988 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cc4004d50 con 0x7f5ccc1051b0 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.306+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ccc1a1590 con 0x7f5ccc1051b0 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.306+0000 7f5cc8d99700 1 -- 172.21.15.194:0/971993988 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4005a30 con 0x7f5ccc1051b0 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.306+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ccc1a1ad0 con 0x7f5ccc1051b0 2024-04-26T11:25:26.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.307+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ccc06a110 con 0x7f5ccc1051b0 2024-04-26T11:25:26.311 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.308+0000 7f5cc8d99700 1 -- 172.21.15.194:0/971993988 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5cc401a020 con 0x7f5ccc1051b0 2024-04-26T11:25:26.311 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.309+0000 7f5cc8d99700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cb80a3cc0 0x7f5cb80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:26.311 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.309+0000 7f5cc8d99700 1 -- 172.21.15.194:0/971993988 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5cc40cfd70 con 0x7f5ccc1051b0 2024-04-26T11:25:26.311 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.310+0000 7f5ccad9d700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cb80a3cc0 0x7f5cb80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:26.311 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.311+0000 7f5ccad9d700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cb80a3cc0 0x7f5cb80a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00b5c0 tx=0x7f5cbc003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:26.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.313+0000 7f5cc8d99700 1 -- 172.21.15.194:0/971993988 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5cc4010070 con 0x7f5ccc1051b0 2024-04-26T11:25:26.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.521+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}) -- 0x7f5ccc063b20 con 0x7f5cb80a3cc0 2024-04-26T11:25:26.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.538+0000 7f5cc8d99700 1 -- 172.21.15.194:0/971993988 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f5ccc063b20 con 0x7f5cb80a3cc0 2024-04-26T11:25:26.540 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mds.all update... 2024-04-26T11:25:26.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cb80a3cc0 msgr2=0x7f5cb80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:26.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cb80a3cc0 0x7f5cb80a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00b5c0 tx=0x7f5cbc003860 comp rx=0 tx=0).stop 2024-04-26T11:25:26.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc1051b0 msgr2=0x7f5ccc1a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:26.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc1051b0 0x7f5ccc1a0c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007f10 tx=0x7f5cc400d3b0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 shutdown_connections 2024-04-26T11:25:26.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f5cb80a3cc0 0x7f5cb80a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ccc1051b0 0x7f5ccc1a0c00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5ccc101790 0x7f5ccc199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 --2- 172.21.15.194:0/971993988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ccc104810 0x7f5ccc199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:26.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 >> 172.21.15.194:0/971993988 conn(0x7f5ccc0fd0d0 msgr2=0x7f5ccc107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:26.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 shutdown_connections 2024-04-26T11:25:26.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T11:25:26.540+0000 7f5cd1212700 1 -- 172.21.15.194:0/971993988 wait complete. 2024-04-26T11:25:27.210 DEBUG:teuthology.orchestra.run.smithi033:mds.a> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T11:25:27.219 DEBUG:teuthology.orchestra.run.smithi033:mds.d> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T11:25:27.271 DEBUG:teuthology.orchestra.run.smithi033:mds.g> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T11:25:27.319 DEBUG:teuthology.orchestra.run.smithi033:mds.j> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T11:25:27.365 DEBUG:teuthology.orchestra.run.smithi096:mds.b> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T11:25:27.368 DEBUG:teuthology.orchestra.run.smithi096:mds.e> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T11:25:27.415 DEBUG:teuthology.orchestra.run.smithi096:mds.h> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T11:25:27.462 DEBUG:teuthology.orchestra.run.smithi096:mds.k> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T11:25:27.507 DEBUG:teuthology.orchestra.run.smithi194:mds.c> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T11:25:27.509 DEBUG:teuthology.orchestra.run.smithi194:mds.f> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T11:25:27.552 DEBUG:teuthology.orchestra.run.smithi194:mds.i> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T11:25:27.595 DEBUG:teuthology.orchestra.run.smithi194:mds.l> sudo journalctl -f -n 0 -u ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T11:25:27.638 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T11:25:27.638 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:25:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:27 smithi096 ceph-mon[32753]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:27 smithi096 ceph-mon[32753]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:25:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:27 smithi096 ceph-mon[32753]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-26T11:25:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:27 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:27 smithi194 ceph-mon[33017]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:27 smithi194 ceph-mon[33017]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:25:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:27 smithi194 ceph-mon[33017]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-26T11:25:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:27 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:27 smithi033 ceph-mon[27845]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:27 smithi033 ceph-mon[27845]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:25:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:27 smithi033 ceph-mon[27845]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-26T11:25:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:27 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:25:28.109 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:28.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:28.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:28.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:28.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:28.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:28.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:28.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:28.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:28.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:28.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:28.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:28.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:25:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:29.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:29 smithi194 ceph-mon[33017]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:29.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:29 smithi194 ceph-mon[33017]: Deploying daemon mds.l on smithi194 2024-04-26T11:25:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:29 smithi033 ceph-mon[27845]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:29 smithi033 ceph-mon[27845]: Deploying daemon mds.l on smithi194 2024-04-26T11:25:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:29 smithi096 ceph-mon[32753]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:29 smithi096 ceph-mon[32753]: Deploying daemon mds.l on smithi194 2024-04-26T11:25:30.104 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:29 smithi194 systemd[1]: Starting Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:25:30.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 -- 172.21.15.33:0/1326601492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406b130 msgr2=0x7f5a0406b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:30.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 --2- 172.21.15.33:0/1326601492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406b130 0x7f5a0406b590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0098f0 tx=0x7f59ec009c00 comp rx=0 tx=0).stop 2024-04-26T11:25:30.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 -- 172.21.15.33:0/1326601492 shutdown_connections 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 --2- 172.21.15.33:0/1326601492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0406bad0 0x7f5a041102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 --2- 172.21.15.33:0/1326601492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 0x7f5a04109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 --2- 172.21.15.33:0/1326601492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406b130 0x7f5a0406b590 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 -- 172.21.15.33:0/1326601492 >> 172.21.15.33:0/1326601492 conn(0x7f5a040792d0 msgr2=0x7f5a040796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 -- 172.21.15.33:0/1326601492 shutdown_connections 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.335+0000 7f5a0a200700 1 -- 172.21.15.33:0/1326601492 wait complete. 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 Processor -- start 2024-04-26T11:25:30.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 -- start start 2024-04-26T11:25:30.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0406b130 0x7f5a0419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:30.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406bad0 0x7f5a0419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:30.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 0x7f5a041999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:30.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04113620 con 0x7f5a041094a0 2024-04-26T11:25:30.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041134a0 con 0x7f5a0406b130 2024-04-26T11:25:30.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.337+0000 7f5a0a200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5a041137a0 con 0x7f5a0406bad0 2024-04-26T11:25:30.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 0x7f5a041999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:30.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f59faa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406bad0 0x7f5a0419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:30.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 0x7f5a041999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39328/0 (socket says 172.21.15.33:39328) 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0406b130 0x7f5a0419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 -- 172.21.15.33:0/544257964 learned_addr learned my addr 172.21.15.33:0/544257964 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 -- 172.21.15.33:0/544257964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406bad0 msgr2=0x7f5a0419fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406bad0 0x7f5a0419fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 -- 172.21.15.33:0/544257964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0406b130 msgr2=0x7f5a0419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0406b130 0x7f5a0419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03e00700 1 -- 172.21.15.33:0/544257964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0419a120 con 0x7f5a041094a0 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.338+0000 7f5a03400700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0406b130 0x7f5a0419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:25:30.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.339+0000 7f5a03e00700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 0x7f5a041999c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00ef30 tx=0x7f59fc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:30.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.339+0000 7f5a00c00700 1 -- 172.21.15.33:0/544257964 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0092e0 con 0x7f5a041094a0 2024-04-26T11:25:30.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.339+0000 7f5a00c00700 1 -- 172.21.15.33:0/544257964 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59fc004bb0 con 0x7f5a041094a0 2024-04-26T11:25:30.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.339+0000 7f5a00c00700 1 -- 172.21.15.33:0/544257964 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc004020 con 0x7f5a041094a0 2024-04-26T11:25:30.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.339+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0419a3b0 con 0x7f5a041094a0 2024-04-26T11:25:30.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.339+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a041a66d0 con 0x7f5a041094a0 2024-04-26T11:25:30.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.340+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0406a110 con 0x7f5a041094a0 2024-04-26T11:25:30.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.343+0000 7f5a00c00700 1 -- 172.21.15.33:0/544257964 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f59fc01b030 con 0x7f5a041094a0 2024-04-26T11:25:30.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.344+0000 7f5a00c00700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f59e80a3cc0 0x7f59e80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:30.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.344+0000 7f5a00c00700 1 -- 172.21.15.33:0/544257964 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59fc0d0370 con 0x7f5a041094a0 2024-04-26T11:25:30.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.345+0000 7f5a03400700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f59e80a3cc0 0x7f59e80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:30.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.345+0000 7f5a03400700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f59e80a3cc0 0x7f59e80a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5a0419ab00 tx=0x7f59f4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:30.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.347+0000 7f5a00c00700 1 -- 172.21.15.33:0/544257964 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59fc022020 con 0x7f5a041094a0 2024-04-26T11:25:30.489 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 podman[63495]: 2024-04-26 11:25:30.120552735 +0000 UTC m=+0.018669186 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:30.489 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 podman[63495]: 2024-04-26 11:25:30.235024009 +0000 UTC m=+0.133140466 container create e1dea43708931cbcdb48f9e9a27e5c523b175729ee602345e9995f87e84497b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T11:25:30.489 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 podman[63495]: 2024-04-26 11:25:30.462735416 +0000 UTC m=+0.360851868 container init e1dea43708931cbcdb48f9e9a27e5c523b175729ee602345e9995f87e84497b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T11:25:30.489 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 podman[63495]: 2024-04-26 11:25:30.465177274 +0000 UTC m=+0.363293726 container start e1dea43708931cbcdb48f9e9a27e5c523b175729ee602345e9995f87e84497b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T11:25:30.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.632+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a0402e490 con 0x7f5a041094a0 2024-04-26T11:25:30.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.633+0000 7f5a00c00700 1 -- 172.21.15.33:0/544257964 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f59fc010040 con 0x7f5a041094a0 2024-04-26T11:25:30.635 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:25:30.636 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.636+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f59e80a3cc0 msgr2=0x7f59e80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.636+0000 7f5a0a200700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f59e80a3cc0 0x7f59e80a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5a0419ab00 tx=0x7f59f4009310 comp rx=0 tx=0).stop 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.636+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 msgr2=0x7f5a041999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.636+0000 7f5a0a200700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 0x7f5a041999c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00ef30 tx=0x7f59fc00c3a0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 shutdown_connections 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f59e80a3cc0 0x7f59e80a6180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a041094a0 0x7f5a041999c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0406b130 0x7f5a0419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 --2- 172.21.15.33:0/544257964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a0406bad0 0x7f5a0419fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:30.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 >> 172.21.15.33:0/544257964 conn(0x7f5a040792d0 msgr2=0x7f5a041004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:30.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 shutdown_connections 2024-04-26T11:25:30.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:30.637+0000 7f5a0a200700 1 -- 172.21.15.33:0/544257964 wait complete. 2024-04-26T11:25:30.640 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 2 2024-04-26T11:25:30.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 ceph-mds[63514]: -- 172.21.15.194:0/1135040682 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560f9a45a40 con 0x5560f9a18800 2024-04-26T11:25:30.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: starting mds.l at 2024-04-26T11:25:30.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 bash[63495]: e1dea43708931cbcdb48f9e9a27e5c523b175729ee602345e9995f87e84497b2 2024-04-26T11:25:30.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:25:30 smithi194 systemd[1]: Started Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:31.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:31.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:boot 2024-04-26T11:25:31.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: fsmap 1 up:standby 2024-04-26T11:25:31.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:25:31.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/544257964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:31.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:31.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:31.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:boot 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: fsmap 1 up:standby 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/544257964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:31.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:31.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:boot 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: fsmap 1 up:standby 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/544257964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:31.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:32 smithi096 ceph-mon[32753]: Deploying daemon mds.k on smithi096 2024-04-26T11:25:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:32 smithi194 ceph-mon[33017]: Deploying daemon mds.k on smithi096 2024-04-26T11:25:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:32 smithi033 ceph-mon[27845]: Deploying daemon mds.k on smithi096 2024-04-26T11:25:33.111 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:32 smithi096 systemd[1]: Starting Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:25:33.366 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 podman[61436]: 2024-04-26 11:25:33.014336799 +0000 UTC m=+0.018203752 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:33.366 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 podman[61436]: 2024-04-26 11:25:33.121164744 +0000 UTC m=+0.125031758 container create e770095a5864e754c4ae734bb51cfe9dac95d6d0220c100a5964aa403b152613 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T11:25:33.366 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 podman[61436]: 2024-04-26 11:25:33.365976297 +0000 UTC m=+0.369843240 container init e770095a5864e754c4ae734bb51cfe9dac95d6d0220c100a5964aa403b152613 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T11:25:33.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:25:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:33 smithi096 ceph-mon[32753]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:33.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 podman[61436]: 2024-04-26 11:25:33.368706799 +0000 UTC m=+0.372573749 container start e770095a5864e754c4ae734bb51cfe9dac95d6d0220c100a5964aa403b152613 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T11:25:33.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 ceph-mds[61455]: -- 172.21.15.96:0/2775653798 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d6d29da40 con 0x563d6d271000 2024-04-26T11:25:33.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: starting mds.k at 2024-04-26T11:25:33.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 bash[61436]: e770095a5864e754c4ae734bb51cfe9dac95d6d0220c100a5964aa403b152613 2024-04-26T11:25:33.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:25:33 smithi096 systemd[1]: Started Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:33 smithi194 ceph-mon[33017]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:33.923 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:33 smithi033 ceph-mon[27845]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:boot 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: fsmap 2 up:standby 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:34.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:34.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:34.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:34 smithi096 ceph-mon[32753]: Deploying daemon mds.j on smithi033 2024-04-26T11:25:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:boot 2024-04-26T11:25:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: fsmap 2 up:standby 2024-04-26T11:25:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:25:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:34 smithi194 ceph-mon[33017]: Deploying daemon mds.j on smithi033 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:boot 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: fsmap 2 up:standby 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:34 smithi033 ceph-mon[27845]: Deploying daemon mds.j on smithi033 2024-04-26T11:25:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:35 smithi033 ceph-mon[27845]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:35 smithi194 ceph-mon[33017]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:35 smithi096 ceph-mon[32753]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:36.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.662+0000 7f9ed2400700 1 -- 172.21.15.33:0/963050308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc10fa50 msgr2=0x7f9ecc10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:36.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.662+0000 7f9ed2400700 1 --2- 172.21.15.33:0/963050308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc10fa50 0x7f9ecc10fe30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009990 tx=0x7f9ebc009ca0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.663+0000 7f9ed2400700 1 -- 172.21.15.33:0/963050308 shutdown_connections 2024-04-26T11:25:36.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.663+0000 7f9ed2400700 1 --2- 172.21.15.33:0/963050308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ecc10a410 0x7f9ecc116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.663+0000 7f9ed2400700 1 --2- 172.21.15.33:0/963050308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc10fa50 0x7f9ecc10fe30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.663+0000 7f9ed2400700 1 --2- 172.21.15.33:0/963050308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ecc109a50 0x7f9ecc109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.663+0000 7f9ed2400700 1 -- 172.21.15.33:0/963050308 >> 172.21.15.33:0/963050308 conn(0x7f9ecc076bf0 msgr2=0x7f9ecc108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:36.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.663+0000 7f9ed2400700 1 -- 172.21.15.33:0/963050308 shutdown_connections 2024-04-26T11:25:36.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.663+0000 7f9ed2400700 1 -- 172.21.15.33:0/963050308 wait complete. 2024-04-26T11:25:36.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.664+0000 7f9ed2400700 1 Processor -- start 2024-04-26T11:25:36.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.664+0000 7f9ed2400700 1 -- start start 2024-04-26T11:25:36.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ed2400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc109a50 0x7f9ecc1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:36.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ed2400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ecc10a410 0x7f9ecc1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:36.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ed2400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ecc10fa50 0x7f9ecc1a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:36.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ed2400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc119c90 con 0x7f9ecc10a410 2024-04-26T11:25:36.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ed2400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc119b10 con 0x7f9ecc109a50 2024-04-26T11:25:36.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ed2400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc119e10 con 0x7f9ecc10fa50 2024-04-26T11:25:36.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ecb400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc109a50 0x7f9ecc1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:36.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ecb400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc109a50 0x7f9ecc1a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:59042/0 (socket says 172.21.15.33:59042) 2024-04-26T11:25:36.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ecb400700 1 -- 172.21.15.33:0/1096850174 learned_addr learned my addr 172.21.15.33:0/1096850174 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:36.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.665+0000 7f9ecaa00700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ecc10a410 0x7f9ecc1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:36.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ecbe00700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ecc10fa50 0x7f9ecc1a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:36.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ecb400700 1 -- 172.21.15.33:0/1096850174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ecc10fa50 msgr2=0x7f9ecc1a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:36.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ecb400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ecc10fa50 0x7f9ecc1a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ecb400700 1 -- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ecc10a410 msgr2=0x7f9ecc1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:36.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ecb400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ecc10a410 0x7f9ecc1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ecb400700 1 -- 172.21.15.33:0/1096850174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ecc1a5120 con 0x7f9ecc109a50 2024-04-26T11:25:36.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ecb400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc109a50 0x7f9ecc1a0160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc000c00 tx=0x7f9ebc009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:36.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.666+0000 7f9ec3e00700 1 -- 172.21.15.33:0/1096850174 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc027070 con 0x7f9ecc109a50 2024-04-26T11:25:36.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.667+0000 7f9ec3e00700 1 -- 172.21.15.33:0/1096850174 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ebc022470 con 0x7f9ecc109a50 2024-04-26T11:25:36.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.667+0000 7f9ec3e00700 1 -- 172.21.15.33:0/1096850174 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc017650 con 0x7f9ecc109a50 2024-04-26T11:25:36.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.667+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ecc1a53b0 con 0x7f9ecc109a50 2024-04-26T11:25:36.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.667+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ecc1acc90 con 0x7f9ecc109a50 2024-04-26T11:25:36.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.668+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ecc111fe0 con 0x7f9ecc109a50 2024-04-26T11:25:36.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.670+0000 7f9ec3e00700 1 -- 172.21.15.33:0/1096850174 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9ebc02a070 con 0x7f9ecc109a50 2024-04-26T11:25:36.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.671+0000 7f9ec3e00700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9eb00a3cc0 0x7f9eb00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:36.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.671+0000 7f9ec3e00700 1 -- 172.21.15.33:0/1096850174 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9ebc0d0f20 con 0x7f9ecc109a50 2024-04-26T11:25:36.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.672+0000 7f9ecaa00700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9eb00a3cc0 0x7f9eb00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:36.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.672+0000 7f9ecaa00700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9eb00a3cc0 0x7f9eb00a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc078ba0 tx=0x7f9eb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:36.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.676+0000 7f9ec3e00700 1 -- 172.21.15.33:0/1096850174 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ebc05eff0 con 0x7f9ecc109a50 2024-04-26T11:25:36.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.970+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ecc10b190 con 0x7f9ecc109a50 2024-04-26T11:25:36.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.971+0000 7f9ec3e00700 1 -- 172.21.15.33:0/1096850174 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f9ebc013070 con 0x7f9ecc109a50 2024-04-26T11:25:36.972 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:25:36.972 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T11:25:36.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.973+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9eb00a3cc0 msgr2=0x7f9eb00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:36.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.973+0000 7f9ed2400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9eb00a3cc0 0x7f9eb00a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc078ba0 tx=0x7f9eb8009260 comp rx=0 tx=0).stop 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.973+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc109a50 msgr2=0x7f9ecc1a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.973+0000 7f9ed2400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc109a50 0x7f9ecc1a0160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc000c00 tx=0x7f9ebc009510 comp rx=0 tx=0).stop 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 shutdown_connections 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9eb00a3cc0 0x7f9eb00a6180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ecc10fa50 0x7f9ecc1a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc109a50 0x7f9ecc1a0160 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 --2- 172.21.15.33:0/1096850174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ecc10a410 0x7f9ecc1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:36.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 >> 172.21.15.33:0/1096850174 conn(0x7f9ecc076bf0 msgr2=0x7f9ecc077250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:36.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 shutdown_connections 2024-04-26T11:25:36.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:36.974+0000 7f9ed2400700 1 -- 172.21.15.33:0/1096850174 wait complete. 2024-04-26T11:25:36.977 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 3 2024-04-26T11:25:37.832 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:37 smithi033 ceph-mon[27845]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:37.832 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:37 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1096850174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:37 smithi194 ceph-mon[33017]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:37 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1096850174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:37 smithi096 ceph-mon[32753]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:37 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1096850174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:38.607 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:25:38 smithi033 podman[67393]: 2024-04-26 11:25:38.318399469 +0000 UTC m=+0.166451339 container create 09e1ce08f377bb1f5748908391a920a0fed9730559d97ad8f60dfa01c74be379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T11:25:38.607 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:25:38 smithi033 podman[67393]: 2024-04-26 11:25:38.605670977 +0000 UTC m=+0.453722850 container init 09e1ce08f377bb1f5748908391a920a0fed9730559d97ad8f60dfa01c74be379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T11:25:38.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:25:38 smithi033 podman[67393]: 2024-04-26 11:25:38.617153843 +0000 UTC m=+0.465205669 container start 09e1ce08f377bb1f5748908391a920a0fed9730559d97ad8f60dfa01c74be379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T11:25:38.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:25:38 smithi033 ceph-mds[67411]: -- 172.21.15.33:0/2337649953 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563ba307fa40 con 0x563ba3052800 2024-04-26T11:25:38.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:25:38 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: starting mds.j at 2024-04-26T11:25:38.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:25:38 smithi033 bash[67393]: 09e1ce08f377bb1f5748908391a920a0fed9730559d97ad8f60dfa01c74be379 2024-04-26T11:25:38.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:25:38 smithi033 systemd[1]: Started Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:39 smithi194 ceph-mon[33017]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:39 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:39 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:39 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:39 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:39 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:39 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:39 smithi033 ceph-mon[27845]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:39 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:39 smithi096 ceph-mon[32753]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:39 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:39 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:39 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:39 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:39 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:39 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:40.024 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:25:40.854 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:40 smithi033 ceph-mon[27845]: Deploying daemon mds.d on smithi033 2024-04-26T11:25:40.854 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:40 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:boot 2024-04-26T11:25:40.854 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:40 smithi033 ceph-mon[27845]: fsmap 3 up:standby 2024-04-26T11:25:40.854 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:40 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:25:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:40 smithi194 ceph-mon[33017]: Deploying daemon mds.d on smithi033 2024-04-26T11:25:40.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:40 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:boot 2024-04-26T11:25:40.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:40 smithi194 ceph-mon[33017]: fsmap 3 up:standby 2024-04-26T11:25:40.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:40 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:25:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:40 smithi096 ceph-mon[32753]: Deploying daemon mds.d on smithi033 2024-04-26T11:25:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:40 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:boot 2024-04-26T11:25:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:40 smithi096 ceph-mon[32753]: fsmap 3 up:standby 2024-04-26T11:25:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:40 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:25:41.303 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:41.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:41 smithi033 ceph-mon[27845]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:41 smithi194 ceph-mon[33017]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:41.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:41 smithi096 ceph-mon[32753]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:42.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:42 smithi033 systemd[1]: Starting Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:25:42.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:42 smithi033 ceph-mon[27845]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:42 smithi096 ceph-mon[32753]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:42 smithi194 ceph-mon[33017]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:43.142 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:42 smithi033 podman[67793]: 2024-04-26 11:25:42.73475716 +0000 UTC m=+0.043165961 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:43.143 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:42 smithi033 podman[67793]: 2024-04-26 11:25:42.848860838 +0000 UTC m=+0.157269661 container create c177a1841c35c10c29b13a9ed17d5941c255b19be4b08f394e9c25be7ad75bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-26T11:25:43.143 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:43 smithi033 podman[67793]: 2024-04-26 11:25:43.138918354 +0000 UTC m=+0.447327231 container init c177a1841c35c10c29b13a9ed17d5941c255b19be4b08f394e9c25be7ad75bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T11:25:43.414 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:43 smithi033 podman[67793]: 2024-04-26 11:25:43.152509467 +0000 UTC m=+0.460918270 container start c177a1841c35c10c29b13a9ed17d5941c255b19be4b08f394e9c25be7ad75bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T11:25:43.414 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:43 smithi033 ceph-mds[67827]: -- 172.21.15.33:0/2719352861 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55814a505a40 con 0x55814a4d9000 2024-04-26T11:25:43.415 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:43 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: starting mds.d at 2024-04-26T11:25:43.415 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:43 smithi033 bash[67793]: c177a1841c35c10c29b13a9ed17d5941c255b19be4b08f394e9c25be7ad75bf5 2024-04-26T11:25:43.415 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:25:43 smithi033 systemd[1]: Started Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.739+0000 7f1153800700 1 -- 172.21.15.33:0/2336817280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c106000 msgr2=0x7f114c112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.739+0000 7f1153800700 1 --2- 172.21.15.33:0/2336817280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c106000 0x7f114c112510 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f114400b0a0 tx=0x7f114400b3b0 comp rx=0 tx=0).stop 2024-04-26T11:25:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.739+0000 7f1153800700 1 -- 172.21.15.33:0/2336817280 shutdown_connections 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.739+0000 7f1153800700 1 --2- 172.21.15.33:0/2336817280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c106000 0x7f114c112510 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.739+0000 7f1153800700 1 --2- 172.21.15.33:0/2336817280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f114c10b640 0x7f114c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.739+0000 7f1153800700 1 --2- 172.21.15.33:0/2336817280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f114c105640 0x7f114c105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.739+0000 7f1153800700 1 -- 172.21.15.33:0/2336817280 >> 172.21.15.33:0/2336817280 conn(0x7f114c101130 msgr2=0x7f114c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.740+0000 7f1153800700 1 -- 172.21.15.33:0/2336817280 shutdown_connections 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.740+0000 7f1153800700 1 -- 172.21.15.33:0/2336817280 wait complete. 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 Processor -- start 2024-04-26T11:25:43.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 -- start start 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c105640 0x7f114c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f114c106000 0x7f114c1a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f114c10b640 0x7f114c19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1158d0 con 0x7f114c10b640 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f114c115750 con 0x7f114c105640 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1153800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f114c115a50 con 0x7f114c106000 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.741+0000 7f1150e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c105640 0x7f114c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c105640 0x7f114c1a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:54406/0 (socket says 172.21.15.33:54406) 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 -- 172.21.15.33:0/1022382448 learned_addr learned my addr 172.21.15.33:0/1022382448 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1151800700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f114c10b640 0x7f114c19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f114be00700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f114c106000 0x7f114c1a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 -- 172.21.15.33:0/1022382448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f114c106000 msgr2=0x7f114c1a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:43.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f114c106000 0x7f114c1a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 -- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f114c10b640 msgr2=0x7f114c19bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f114c10b640 0x7f114c19bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 -- 172.21.15.33:0/1022382448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c19c2b0 con 0x7f114c105640 2024-04-26T11:25:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1150e00700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c105640 0x7f114c1a1940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f114000eb30 tx=0x7f114000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.742+0000 7f1149600700 1 -- 172.21.15.33:0/1022382448 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114000cbd0 con 0x7f114c105640 2024-04-26T11:25:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.743+0000 7f1149600700 1 -- 172.21.15.33:0/1022382448 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11400044d0 con 0x7f114c105640 2024-04-26T11:25:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.743+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114c19c5a0 con 0x7f114c105640 2024-04-26T11:25:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.743+0000 7f1149600700 1 -- 172.21.15.33:0/1022382448 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140003e30 con 0x7f114c105640 2024-04-26T11:25:43.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.743+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f114c1a8810 con 0x7f114c105640 2024-04-26T11:25:43.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.744+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f114c06a110 con 0x7f114c105640 2024-04-26T11:25:43.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.746+0000 7f1149600700 1 -- 172.21.15.33:0/1022382448 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1140004000 con 0x7f114c105640 2024-04-26T11:25:43.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.747+0000 7f1149600700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f11340a3cc0 0x7f11340a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:43.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.747+0000 7f1149600700 1 -- 172.21.15.33:0/1022382448 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f11400d0480 con 0x7f114c105640 2024-04-26T11:25:43.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.748+0000 7f114be00700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f11340a3cc0 0x7f11340a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:43.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.748+0000 7f114be00700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f11340a3cc0 0x7f11340a6180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f114c1037a0 tx=0x7f1138009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:43.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:43.749+0000 7f1149600700 1 -- 172.21.15.33:0/1022382448 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1140053ff0 con 0x7f114c105640 2024-04-26T11:25:44.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.033+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f114c02e490 con 0x7f114c105640 2024-04-26T11:25:44.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.034+0000 7f1149600700 1 -- 172.21.15.33:0/1022382448 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f11400042f0 con 0x7f114c105640 2024-04-26T11:25:44.035 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:25:44.035 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T11:25:44.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f11340a3cc0 msgr2=0x7f11340a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:44.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f11340a3cc0 0x7f11340a6180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f114c1037a0 tx=0x7f1138009260 comp rx=0 tx=0).stop 2024-04-26T11:25:44.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c105640 msgr2=0x7f114c1a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:44.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c105640 0x7f114c1a1940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f114000eb30 tx=0x7f114000ee40 comp rx=0 tx=0).stop 2024-04-26T11:25:44.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 shutdown_connections 2024-04-26T11:25:44.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f11340a3cc0 0x7f11340a6180 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:44.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f114c10b640 0x7f114c19bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:44.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f114c105640 0x7f114c1a1940 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:44.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 --2- 172.21.15.33:0/1022382448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f114c106000 0x7f114c1a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:44.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 >> 172.21.15.33:0/1022382448 conn(0x7f114c101130 msgr2=0x7f114c101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:44.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.037+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 shutdown_connections 2024-04-26T11:25:44.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:44.038+0000 7f1153800700 1 -- 172.21.15.33:0/1022382448 wait complete. 2024-04-26T11:25:44.040 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 5 2024-04-26T11:25:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: Deploying daemon mds.e on smithi096 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:boot 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: fsmap 4 up:standby 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:25:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:44 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1022382448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: Deploying daemon mds.e on smithi096 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:boot 2024-04-26T11:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: fsmap 4 up:standby 2024-04-26T11:25:44.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:25:44.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:44 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1022382448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: Deploying daemon mds.e on smithi096 2024-04-26T11:25:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:boot 2024-04-26T11:25:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: fsmap 4 up:standby 2024-04-26T11:25:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:25:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:44 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1022382448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:45 smithi033 ceph-mon[27845]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:45.677 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:45 smithi096 ceph-mon[32753]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:45 smithi194 ceph-mon[33017]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:45.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:45 smithi096 podman[61759]: 2024-04-26 11:25:45.802630269 +0000 UTC m=+0.018450159 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:46.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:45 smithi096 podman[61759]: 2024-04-26 11:25:45.934632964 +0000 UTC m=+0.150452845 container create c88e058eb2a1057dfffb0124e6bdbbb208a41a2c57fc37e99447983e205eb666 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:25:46.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:46 smithi096 podman[61759]: 2024-04-26 11:25:46.179064578 +0000 UTC m=+0.394884462 container init c88e058eb2a1057dfffb0124e6bdbbb208a41a2c57fc37e99447983e205eb666 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:25:46.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:46 smithi096 podman[61759]: 2024-04-26 11:25:46.18179864 +0000 UTC m=+0.397618524 container start c88e058eb2a1057dfffb0124e6bdbbb208a41a2c57fc37e99447983e205eb666 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T11:25:46.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:46 smithi096 ceph-mds[61780]: -- 172.21.15.96:0/1747452632 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56368db5da40 con 0x56368db31000 2024-04-26T11:25:46.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:46 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: starting mds.e at 2024-04-26T11:25:46.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:46 smithi096 bash[61759]: c88e058eb2a1057dfffb0124e6bdbbb208a41a2c57fc37e99447983e205eb666 2024-04-26T11:25:46.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:25:46 smithi096 systemd[1]: Started Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:46.906 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:25:47.393 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: Deploying daemon mds.f on smithi194 2024-04-26T11:25:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:boot 2024-04-26T11:25:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: fsmap 5 up:standby 2024-04-26T11:25:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:25:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: Deploying daemon mds.f on smithi194 2024-04-26T11:25:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:boot 2024-04-26T11:25:47.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: fsmap 5 up:standby 2024-04-26T11:25:47.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:25:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:47.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:47.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:47.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: Deploying daemon mds.f on smithi194 2024-04-26T11:25:47.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:boot 2024-04-26T11:25:47.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: fsmap 5 up:standby 2024-04-26T11:25:47.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:25:48.839 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:48 smithi194 systemd[1]: Starting Ceph mds.f for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:25:49.100 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:48 smithi194 podman[63806]: 2024-04-26 11:25:48.74147544 +0000 UTC m=+0.018353586 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:49.100 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:48 smithi194 podman[63806]: 2024-04-26 11:25:48.864397885 +0000 UTC m=+0.141276021 container create 18d645bf5b4ba670f276723dd1708d6ebc9d2da50d30ff841183ddeee46f3ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T11:25:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mon[33017]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:49.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:49.355 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:49 smithi194 podman[63806]: 2024-04-26 11:25:49.100485323 +0000 UTC m=+0.377363456 container init 18d645bf5b4ba670f276723dd1708d6ebc9d2da50d30ff841183ddeee46f3ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T11:25:49.355 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:49 smithi194 podman[63806]: 2024-04-26 11:25:49.103053722 +0000 UTC m=+0.379931862 container start 18d645bf5b4ba670f276723dd1708d6ebc9d2da50d30ff841183ddeee46f3ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T11:25:49.355 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-mds[63825]: -- 172.21.15.194:0/585001830 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556bcae07a40 con 0x556bcaddac00 2024-04-26T11:25:49.355 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:49 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-f[63821]: starting mds.f at 2024-04-26T11:25:49.355 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:49 smithi194 bash[63806]: 18d645bf5b4ba670f276723dd1708d6ebc9d2da50d30ff841183ddeee46f3ec6 2024-04-26T11:25:49.355 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 26 11:25:49 smithi194 systemd[1]: Started Ceph mds.f for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:49.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 -- 172.21.15.33:0/3601828631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f42381145f0 msgr2=0x7f42381169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:49.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 --2- 172.21.15.33:0/3601828631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f42381145f0 0x7f42381169e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f423000b0a0 tx=0x7f423000b3b0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 -- 172.21.15.33:0/3601828631 shutdown_connections 2024-04-26T11:25:49.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 --2- 172.21.15.33:0/3601828631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f42381145f0 0x7f42381169e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 --2- 172.21.15.33:0/3601828631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4238075650 0x7f4238075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 --2- 172.21.15.33:0/3601828631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4238075f70 0x7f42381140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 -- 172.21.15.33:0/3601828631 >> 172.21.15.33:0/3601828631 conn(0x7f423807a580 msgr2=0x7f423807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:49.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 -- 172.21.15.33:0/3601828631 shutdown_connections 2024-04-26T11:25:49.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.502+0000 7f423e400700 1 -- 172.21.15.33:0/3601828631 wait complete. 2024-04-26T11:25:49.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.503+0000 7f423e400700 1 Processor -- start 2024-04-26T11:25:49.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.503+0000 7f423e400700 1 -- start start 2024-04-26T11:25:49.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f423e400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4238075650 0x7f42381a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:49.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f423e400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4238075f70 0x7f42381a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:49.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f423e400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42381145f0 0x7f42381a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f423e400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4238119c20 con 0x7f42381145f0 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f423e400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4238119aa0 con 0x7f4238075650 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f423e400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4238119da0 con 0x7f4238075f70 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f422fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4238075f70 0x7f42381a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f422fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4238075f70 0x7f42381a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:35758/0 (socket says 172.21.15.33:35758) 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f422fe00700 1 -- 172.21.15.33:0/275662360 learned_addr learned my addr 172.21.15.33:0/275662360 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f4237e00700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42381145f0 0x7f42381a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.504+0000 7f4237400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4238075650 0x7f42381a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f422fe00700 1 -- 172.21.15.33:0/275662360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4238075650 msgr2=0x7f42381a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f422fe00700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4238075650 0x7f42381a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f422fe00700 1 -- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42381145f0 msgr2=0x7f42381a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f422fe00700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42381145f0 0x7f42381a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f422fe00700 1 -- 172.21.15.33:0/275662360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42381a51c0 con 0x7f4238075f70 2024-04-26T11:25:49.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f4237e00700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42381145f0 0x7f42381a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:25:49.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f422fe00700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4238075f70 0x7f42381a06f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f422000e9c0 tx=0x7f422000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:49.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f4234c00700 1 -- 172.21.15.33:0/275662360 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422000cbd0 con 0x7f4238075f70 2024-04-26T11:25:49.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f4234c00700 1 -- 172.21.15.33:0/275662360 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f422000cd70 con 0x7f4238075f70 2024-04-26T11:25:49.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f4234c00700 1 -- 172.21.15.33:0/275662360 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220005610 con 0x7f4238075f70 2024-04-26T11:25:49.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42381a54b0 con 0x7f4238075f70 2024-04-26T11:25:49.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.505+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42381acd30 con 0x7f4238075f70 2024-04-26T11:25:49.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.507+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f423806a110 con 0x7f4238075f70 2024-04-26T11:25:49.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.508+0000 7f4234c00700 1 -- 172.21.15.33:0/275662360 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4220004750 con 0x7f4238075f70 2024-04-26T11:25:49.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.510+0000 7f4234c00700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f421c0a3cc0 0x7f421c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:49.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.510+0000 7f4234c00700 1 -- 172.21.15.33:0/275662360 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f42200d0200 con 0x7f4238075f70 2024-04-26T11:25:49.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.510+0000 7f4237400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f421c0a3cc0 0x7f421c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:49.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.511+0000 7f4237400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f421c0a3cc0 0x7f421c0a6180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4228005950 tx=0x7f422800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:49.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.514+0000 7f4234c00700 1 -- 172.21.15.33:0/275662360 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f422009cfe0 con 0x7f4238075f70 2024-04-26T11:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:49 smithi033 ceph-mon[27845]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:49 smithi096 ceph-mon[32753]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:49.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.795+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42381acf00 con 0x7f4238075f70 2024-04-26T11:25:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.796+0000 7f4234c00700 1 -- 172.21.15.33:0/275662360 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f422009c8e0 con 0x7f4238075f70 2024-04-26T11:25:49.797 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:25:49.798 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T11:25:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f421c0a3cc0 msgr2=0x7f421c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f421c0a3cc0 0x7f421c0a6180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4228005950 tx=0x7f422800b410 comp rx=0 tx=0).stop 2024-04-26T11:25:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4238075f70 msgr2=0x7f42381a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4238075f70 0x7f42381a06f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f422000e9c0 tx=0x7f422000ecd0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 shutdown_connections 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f421c0a3cc0 0x7f421c0a6180 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42381145f0 0x7f42381a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4238075650 0x7f42381a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 --2- 172.21.15.33:0/275662360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4238075f70 0x7f42381a06f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 >> 172.21.15.33:0/275662360 conn(0x7f423807a580 msgr2=0x7f4238113420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.799+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 shutdown_connections 2024-04-26T11:25:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:49.800+0000 7f423e400700 1 -- 172.21.15.33:0/275662360 wait complete. 2024-04-26T11:25:49.802 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 7 2024-04-26T11:25:50.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:50 smithi033 ceph-mon[27845]: Deploying daemon mds.c on smithi194 2024-04-26T11:25:50.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:50 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6834/1403509498,v1:172.21.15.194:6835/1403509498] up:boot 2024-04-26T11:25:50.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:50 smithi033 ceph-mon[27845]: fsmap 6 up:standby 2024-04-26T11:25:50.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T11:25:50.540 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:50 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/275662360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:50 smithi194 ceph-mon[33017]: Deploying daemon mds.c on smithi194 2024-04-26T11:25:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:50 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6834/1403509498,v1:172.21.15.194:6835/1403509498] up:boot 2024-04-26T11:25:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:50 smithi194 ceph-mon[33017]: fsmap 6 up:standby 2024-04-26T11:25:50.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T11:25:50.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:50 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/275662360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:50 smithi096 ceph-mon[32753]: Deploying daemon mds.c on smithi194 2024-04-26T11:25:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:50 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6834/1403509498,v1:172.21.15.194:6835/1403509498] up:boot 2024-04-26T11:25:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:50 smithi096 ceph-mon[32753]: fsmap 6 up:standby 2024-04-26T11:25:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T11:25:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:50 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/275662360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:51.456 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:51 smithi194 ceph-mon[33017]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:51 smithi033 ceph-mon[27845]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:51.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:51 smithi096 ceph-mon[32753]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:51.854 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:51 smithi194 systemd[1]: Starting Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:25:52.148 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:51 smithi194 podman[64112]: 2024-04-26 11:25:51.762375314 +0000 UTC m=+0.017723546 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:52.148 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:51 smithi194 podman[64112]: 2024-04-26 11:25:51.884847215 +0000 UTC m=+0.140195433 container create 6d29997d79c4238c9474081edc60fe900af09f1f442008b8ead04052a404bc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T11:25:52.148 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:52 smithi194 podman[64112]: 2024-04-26 11:25:52.147890614 +0000 UTC m=+0.403238832 container init 6d29997d79c4238c9474081edc60fe900af09f1f442008b8ead04052a404bc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:25:52.599 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:25:52.604 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:52 smithi194 podman[64112]: 2024-04-26 11:25:52.150554181 +0000 UTC m=+0.405902410 container start 6d29997d79c4238c9474081edc60fe900af09f1f442008b8ead04052a404bc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T11:25:52.604 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:52 smithi194 ceph-mds[64131]: -- 172.21.15.194:0/403414608 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55719d9bba40 con 0x55719d98e800 2024-04-26T11:25:52.604 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:52 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: starting mds.c at 2024-04-26T11:25:52.604 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:52 smithi194 bash[64112]: 6d29997d79c4238c9474081edc60fe900af09f1f442008b8ead04052a404bc83 2024-04-26T11:25:52.605 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:25:52 smithi194 systemd[1]: Started Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:53.066 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: Deploying daemon mds.h on smithi096 2024-04-26T11:25:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:boot 2024-04-26T11:25:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: fsmap 7 up:standby 2024-04-26T11:25:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:53 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: Deploying daemon mds.h on smithi096 2024-04-26T11:25:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:boot 2024-04-26T11:25:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: fsmap 7 up:standby 2024-04-26T11:25:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:53 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:25:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:53.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:53.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:53.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: Deploying daemon mds.h on smithi096 2024-04-26T11:25:53.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:boot 2024-04-26T11:25:53.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: fsmap 7 up:standby 2024-04-26T11:25:53.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:53 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:25:54.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:54 smithi096 systemd[1]: Starting Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:25:55.025 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:54 smithi096 podman[62065]: 2024-04-26 11:25:54.651628018 +0000 UTC m=+0.018033965 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:55.026 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:54 smithi096 podman[62065]: 2024-04-26 11:25:54.772654354 +0000 UTC m=+0.139060297 container create 292a5264335ecf7fd36f0a7d2e9795878cafb321dd9dfa06cfca425d9f97a973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:25:55.026 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:55 smithi096 podman[62065]: 2024-04-26 11:25:55.025556763 +0000 UTC m=+0.391962697 container init 292a5264335ecf7fd36f0a7d2e9795878cafb321dd9dfa06cfca425d9f97a973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 -- 172.21.15.33:0/609342751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806b130 msgr2=0x7fb3e806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 --2- 172.21.15.33:0/609342751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806b130 0x7fb3e806b590 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009990 tx=0x7fb3d4009ca0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 -- 172.21.15.33:0/609342751 shutdown_connections 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 --2- 172.21.15.33:0/609342751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3e806bad0 0x7fb3e81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 --2- 172.21.15.33:0/609342751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3e81094a0 0x7fb3e8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 --2- 172.21.15.33:0/609342751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806b130 0x7fb3e806b590 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 -- 172.21.15.33:0/609342751 >> 172.21.15.33:0/609342751 conn(0x7fb3e80792d0 msgr2=0x7fb3e80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 -- 172.21.15.33:0/609342751 shutdown_connections 2024-04-26T11:25:55.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.145+0000 7fb3f0000700 1 -- 172.21.15.33:0/609342751 wait complete. 2024-04-26T11:25:55.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.146+0000 7fb3f0000700 1 Processor -- start 2024-04-26T11:25:55.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3f0000700 1 -- start start 2024-04-26T11:25:55.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3f0000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3e806b130 0x7fb3e81a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:55.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3f0000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806bad0 0x7fb3e81a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3f0000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3e81094a0 0x7fb3e819bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3f0000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8113600 con 0x7fb3e81094a0 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3f0000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8113480 con 0x7fb3e806bad0 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3f0000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8113780 con 0x7fb3e806b130 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3ecc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806bad0 0x7fb3e81a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3ecc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806bad0 0x7fb3e81a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51632/0 (socket says 172.21.15.33:51632) 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3ecc00700 1 -- 172.21.15.33:0/2449225897 learned_addr learned my addr 172.21.15.33:0/2449225897 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3ed600700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3e806b130 0x7fb3e81a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.147+0000 7fb3ee000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3e81094a0 0x7fb3e819bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3ecc00700 1 -- 172.21.15.33:0/2449225897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3e806b130 msgr2=0x7fb3e81a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3ecc00700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3e806b130 0x7fb3e81a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3ecc00700 1 -- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3e81094a0 msgr2=0x7fb3e819bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:55.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3ecc00700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3e81094a0 0x7fb3e819bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3ecc00700 1 -- 172.21.15.33:0/2449225897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e819c460 con 0x7fb3e806bad0 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3ecc00700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806bad0 0x7fb3e81a1fa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4005910 tx=0x7fb3d400b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3e6000700 1 -- 172.21.15.33:0/2449225897 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4027070 con 0x7fb3e806bad0 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3e6000700 1 -- 172.21.15.33:0/2449225897 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3d4022470 con 0x7fb3e806bad0 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e819c6f0 con 0x7fb3e806bad0 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.148+0000 7fb3e6000700 1 -- 172.21.15.33:0/2449225897 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4017650 con 0x7fb3e806bad0 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.149+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e81a89e0 con 0x7fb3e806bad0 2024-04-26T11:25:55.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.150+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e819ca50 con 0x7fb3e806bad0 2024-04-26T11:25:55.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.152+0000 7fb3e6000700 1 -- 172.21.15.33:0/2449225897 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb3d402a070 con 0x7fb3e806bad0 2024-04-26T11:25:55.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.153+0000 7fb3e6000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb3d00a3cc0 0x7fb3d00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:25:55.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.153+0000 7fb3e6000700 1 -- 172.21.15.33:0/2449225897 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb3d40d0ca0 con 0x7fb3e806bad0 2024-04-26T11:25:55.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.153+0000 7fb3ed600700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb3d00a3cc0 0x7fb3d00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:25:55.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.154+0000 7fb3ed600700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb3d00a3cc0 0x7fb3d00a6180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc007b60 tx=0x7fb3dc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:25:55.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.157+0000 7fb3e6000700 1 -- 172.21.15.33:0/2449225897 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3d4013070 con 0x7fb3e806bad0 2024-04-26T11:25:55.310 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:55 smithi096 podman[62065]: 2024-04-26 11:25:55.028233942 +0000 UTC m=+0.394639878 container start 292a5264335ecf7fd36f0a7d2e9795878cafb321dd9dfa06cfca425d9f97a973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:25:55.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mds[62084]: -- 172.21.15.96:0/1442746225 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e89c93da40 con 0x55e89c911000 2024-04-26T11:25:55.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: starting mds.h at 2024-04-26T11:25:55.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:55 smithi096 bash[62065]: 292a5264335ecf7fd36f0a7d2e9795878cafb321dd9dfa06cfca425d9f97a973 2024-04-26T11:25:55.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:25:55 smithi096 systemd[1]: Started Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.447+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3e806a140 con 0x7fb3e806bad0 2024-04-26T11:25:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.448+0000 7fb3e6000700 1 -- 172.21.15.33:0/2449225897 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7fb3e806a140 con 0x7fb3e806bad0 2024-04-26T11:25:55.450 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:25:55.450 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.451+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb3d00a3cc0 msgr2=0x7fb3d00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.451+0000 7fb3f0000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb3d00a3cc0 0x7fb3d00a6180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc007b60 tx=0x7fb3dc0073d0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.451+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806bad0 msgr2=0x7fb3e81a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.451+0000 7fb3f0000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806bad0 0x7fb3e81a1fa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4005910 tx=0x7fb3d400b670 comp rx=0 tx=0).stop 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 shutdown_connections 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb3d00a3cc0 0x7fb3d00a6180 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3e81094a0 0x7fb3e819bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3e806b130 0x7fb3e81a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 --2- 172.21.15.33:0/2449225897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3e806bad0 0x7fb3e81a1fa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:25:55.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 >> 172.21.15.33:0/2449225897 conn(0x7fb3e80792d0 msgr2=0x7fb3e810e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:25:55.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 shutdown_connections 2024-04-26T11:25:55.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:25:55.452+0000 7fb3f0000700 1 -- 172.21.15.33:0/2449225897 wait complete. 2024-04-26T11:25:55.454 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 9 2024-04-26T11:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:55 smithi194 ceph-mon[33017]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:55 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:55 smithi033 ceph-mon[27845]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:55 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mon[32753]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:55 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:56.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:56 smithi194 ceph-mon[33017]: Deploying daemon mds.i on smithi194 2024-04-26T11:25:56.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:56 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] up:boot 2024-04-26T11:25:56.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:56 smithi194 ceph-mon[33017]: fsmap 8 up:standby 2024-04-26T11:25:56.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:56 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:25:56.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:56 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2449225897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:56 smithi033 ceph-mon[27845]: Deploying daemon mds.i on smithi194 2024-04-26T11:25:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:56 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] up:boot 2024-04-26T11:25:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:56 smithi033 ceph-mon[27845]: fsmap 8 up:standby 2024-04-26T11:25:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:56 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:25:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:56 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2449225897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:56 smithi096 ceph-mon[32753]: Deploying daemon mds.i on smithi194 2024-04-26T11:25:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:56 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] up:boot 2024-04-26T11:25:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:56 smithi096 ceph-mon[32753]: fsmap 8 up:standby 2024-04-26T11:25:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:56 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:25:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:56 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2449225897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:25:57.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:57 smithi194 ceph-mon[33017]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:57.605 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 systemd[1]: Starting Ceph mds.i for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:25:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:57 smithi033 ceph-mon[27845]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:57 smithi096 ceph-mon[32753]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:57.891 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 podman[64420]: 2024-04-26 11:25:57.530331572 +0000 UTC m=+0.017975302 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:25:57.891 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 podman[64420]: 2024-04-26 11:25:57.654283337 +0000 UTC m=+0.141927038 container create 40d633ff90a85865b57a2bf0eaa1a4798ec0a4d307fdfa7ea0a30a52604973f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-26T11:25:57.892 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 podman[64420]: 2024-04-26 11:25:57.890452639 +0000 UTC m=+0.378096340 container init 40d633ff90a85865b57a2bf0eaa1a4798ec0a4d307fdfa7ea0a30a52604973f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T11:25:58.239 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:25:58.354 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 podman[64420]: 2024-04-26 11:25:57.893541061 +0000 UTC m=+0.381184772 container start 40d633ff90a85865b57a2bf0eaa1a4798ec0a4d307fdfa7ea0a30a52604973f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T11:25:58.354 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 ceph-mds[64439]: -- 172.21.15.194:0/3600757424 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560a5bb9da40 con 0x560a5bb70800 2024-04-26T11:25:58.354 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-i[64435]: starting mds.i at 2024-04-26T11:25:58.354 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 bash[64420]: 40d633ff90a85865b57a2bf0eaa1a4798ec0a4d307fdfa7ea0a30a52604973f9 2024-04-26T11:25:58.354 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 26 11:25:57 smithi194 systemd[1]: Started Ceph mds.i for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:25:58.686 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:25:59.102 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: Deploying daemon mds.g on smithi033 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:boot 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: fsmap 9 up:standby 2024-04-26T11:25:59.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:25:59 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T11:25:59.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:59.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: Deploying daemon mds.g on smithi033 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:boot 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: fsmap 9 up:standby 2024-04-26T11:25:59.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:25:59 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:25:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: Deploying daemon mds.g on smithi033 2024-04-26T11:25:59.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:boot 2024-04-26T11:25:59.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: fsmap 9 up:standby 2024-04-26T11:25:59.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:25:59 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T11:26:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:01 smithi033 ceph-mon[27845]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:01 smithi194 ceph-mon[33017]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.401+0000 7fd217e00700 1 -- 172.21.15.33:0/1488740496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd21006bad0 msgr2=0x7fd2101102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 --2- 172.21.15.33:0/1488740496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd21006bad0 0x7fd2101102e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd20800b0a0 tx=0x7fd20800b3b0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 -- 172.21.15.33:0/1488740496 shutdown_connections 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 --2- 172.21.15.33:0/1488740496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd21006bad0 0x7fd2101102e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 --2- 172.21.15.33:0/1488740496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2101094a0 0x7fd210109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 --2- 172.21.15.33:0/1488740496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 0x7fd21006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 -- 172.21.15.33:0/1488740496 >> 172.21.15.33:0/1488740496 conn(0x7fd2100792d0 msgr2=0x7fd2100796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 -- 172.21.15.33:0/1488740496 shutdown_connections 2024-04-26T11:26:01.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.402+0000 7fd217e00700 1 -- 172.21.15.33:0/1488740496 wait complete. 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 Processor -- start 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 -- start start 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 0x7fd21019bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd21006bad0 0x7fd21019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2101094a0 0x7fd2101a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2101135f0 con 0x7fd2101094a0 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd210113470 con 0x7fd21006b130 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.403+0000 7fd217e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd210113770 con 0x7fd21006bad0 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 0x7fd21019bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 0x7fd21019bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:44364/0 (socket says 172.21.15.33:44364) 2024-04-26T11:26:01.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215400700 1 -- 172.21.15.33:0/1789315664 learned_addr learned my addr 172.21.15.33:0/1789315664 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2101094a0 0x7fd2101a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd214a00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd21006bad0 0x7fd21019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215e00700 1 -- 172.21.15.33:0/1789315664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd21006bad0 msgr2=0x7fd21019c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd21006bad0 0x7fd21019c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215e00700 1 -- 172.21.15.33:0/1789315664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 msgr2=0x7fd21019bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 0x7fd21019bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215e00700 1 -- 172.21.15.33:0/1789315664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2101a0d30 con 0x7fd2101094a0 2024-04-26T11:26:01.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.404+0000 7fd215400700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 0x7fd21019bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:01.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.405+0000 7fd215e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2101094a0 0x7fd2101a05f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd20800b070 tx=0x7fd208004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:01.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.405+0000 7fd20e000700 1 -- 172.21.15.33:0/1789315664 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208021070 con 0x7fd2101094a0 2024-04-26T11:26:01.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.405+0000 7fd20e000700 1 -- 172.21.15.33:0/1789315664 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd208007e30 con 0x7fd2101094a0 2024-04-26T11:26:01.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.405+0000 7fd20e000700 1 -- 172.21.15.33:0/1789315664 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2080045f0 con 0x7fd2101094a0 2024-04-26T11:26:01.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.405+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2101a0fc0 con 0x7fd2101094a0 2024-04-26T11:26:01.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.405+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2101a88a0 con 0x7fd2101094a0 2024-04-26T11:26:01.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.406+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21006a110 con 0x7fd2101094a0 2024-04-26T11:26:01.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.408+0000 7fd20e000700 1 -- 172.21.15.33:0/1789315664 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd208007980 con 0x7fd2101094a0 2024-04-26T11:26:01.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.410+0000 7fd20e000700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd1f80a3cc0 0x7fd1f80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:01.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.410+0000 7fd20e000700 1 -- 172.21.15.33:0/1789315664 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd2080d0d50 con 0x7fd2101094a0 2024-04-26T11:26:01.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.410+0000 7fd215400700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd1f80a3cc0 0x7fd1f80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:01.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.411+0000 7fd215400700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd1f80a3cc0 0x7fd1f80a6180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd204011550 tx=0x7fd204009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:01.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.415+0000 7fd20e000700 1 -- 172.21.15.33:0/1789315664 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd20809dbc0 con 0x7fd2101094a0 2024-04-26T11:26:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:01 smithi096 ceph-mon[32753]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:01.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.697+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd21002e490 con 0x7fd2101094a0 2024-04-26T11:26:01.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.698+0000 7fd20e000700 1 -- 172.21.15.33:0/1789315664 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7fd20809d4c0 con 0x7fd2101094a0 2024-04-26T11:26:01.699 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:26:01.699 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T11:26:01.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd1f80a3cc0 msgr2=0x7fd1f80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:01.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd1f80a3cc0 0x7fd1f80a6180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd204011550 tx=0x7fd204009260 comp rx=0 tx=0).stop 2024-04-26T11:26:01.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2101094a0 msgr2=0x7fd2101a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:01.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2101094a0 0x7fd2101a05f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd20800b070 tx=0x7fd208004c80 comp rx=0 tx=0).stop 2024-04-26T11:26:01.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 shutdown_connections 2024-04-26T11:26:01.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd1f80a3cc0 0x7fd1f80a6180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2101094a0 0x7fd2101a05f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21006b130 0x7fd21019bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 --2- 172.21.15.33:0/1789315664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd21006bad0 0x7fd21019c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:01.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 >> 172.21.15.33:0/1789315664 conn(0x7fd2100792d0 msgr2=0x7fd21010e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:01.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 shutdown_connections 2024-04-26T11:26:01.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:01.701+0000 7fd217e00700 1 -- 172.21.15.33:0/1789315664 wait complete. 2024-04-26T11:26:01.704 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 10 2024-04-26T11:26:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:02 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1789315664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:02.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:02 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1789315664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:02.453 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:02 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1789315664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:03 smithi194 ceph-mon[33017]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:03.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:03 smithi033 ceph-mon[27845]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:03.361 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:26:03 smithi033 podman[68636]: 2024-04-26 11:26:03.16801993 +0000 UTC m=+0.133529135 container create cb67d2abbe02999b4decde9512b00dacf0f0812f0c8ea0fc15e562aacf930589 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True) 2024-04-26T11:26:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:03 smithi096 ceph-mon[32753]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:03.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:26:03 smithi033 podman[68636]: 2024-04-26 11:26:03.455828055 +0000 UTC m=+0.421337267 container init cb67d2abbe02999b4decde9512b00dacf0f0812f0c8ea0fc15e562aacf930589 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T11:26:03.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:26:03 smithi033 podman[68636]: 2024-04-26 11:26:03.466324632 +0000 UTC m=+0.431833840 container start cb67d2abbe02999b4decde9512b00dacf0f0812f0c8ea0fc15e562aacf930589 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T11:26:03.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:26:03 smithi033 ceph-mds[68654]: -- 172.21.15.33:0/4131209892 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e58b0ba40 con 0x555e58ade800 2024-04-26T11:26:03.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:26:03 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: starting mds.g at 2024-04-26T11:26:03.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:26:03 smithi033 bash[68636]: cb67d2abbe02999b4decde9512b00dacf0f0812f0c8ea0fc15e562aacf930589 2024-04-26T11:26:03.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:26:03 smithi033 systemd[1]: Started Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:26:04.903 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:26:04.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: Deploying daemon mds.a on smithi033 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:boot 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: fsmap 10 up:standby 2024-04-26T11:26:04.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:04 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:26:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: Deploying daemon mds.a on smithi033 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:boot 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: fsmap 10 up:standby 2024-04-26T11:26:05.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:04 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:26:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:26:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:26:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: Deploying daemon mds.a on smithi033 2024-04-26T11:26:05.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:05.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:boot 2024-04-26T11:26:05.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: fsmap 10 up:standby 2024-04-26T11:26:05.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:04 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:26:06.167 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:06 smithi033 ceph-mon[27845]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:06.927 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:06 smithi033 systemd[1]: Starting Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:26:07.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:06 smithi096 ceph-mon[32753]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:06 smithi194 ceph-mon[33017]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:08.014 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:07 smithi033 podman[69030]: 2024-04-26 11:26:07.66640066 +0000 UTC m=+0.037149005 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:26:08.014 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:07 smithi033 podman[69030]: 2024-04-26 11:26:07.786420247 +0000 UTC m=+0.157168633 container create 72590b12b61c3be371a043d086898a0d923cdbb6d1823f6c75260a431ec6ee0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T11:26:08.294 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:08 smithi033 podman[69030]: 2024-04-26 11:26:08.233699299 +0000 UTC m=+0.604447687 container init 72590b12b61c3be371a043d086898a0d923cdbb6d1823f6c75260a431ec6ee0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:26:08.294 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:08 smithi033 podman[69030]: 2024-04-26 11:26:08.244797959 +0000 UTC m=+0.615546327 container start 72590b12b61c3be371a043d086898a0d923cdbb6d1823f6c75260a431ec6ee0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 -- 172.21.15.33:0/444712100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400760e0 msgr2=0x7fa740113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 --2- 172.21.15.33:0/444712100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400760e0 0x7fa740113e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009990 tx=0x7fa72c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 -- 172.21.15.33:0/444712100 shutdown_connections 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 --2- 172.21.15.33:0/444712100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa740114340 0x7fa740116730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 --2- 172.21.15.33:0/444712100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7400757c0 0x7fa740075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 --2- 172.21.15.33:0/444712100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400760e0 0x7fa740113e00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 -- 172.21.15.33:0/444712100 >> 172.21.15.33:0/444712100 conn(0x7fa7400fee40 msgr2=0x7fa740101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:08.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 -- 172.21.15.33:0/444712100 shutdown_connections 2024-04-26T11:26:08.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.524+0000 7fa73fe00700 1 -- 172.21.15.33:0/444712100 wait complete. 2024-04-26T11:26:08.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.525+0000 7fa73fe00700 1 Processor -- start 2024-04-26T11:26:08.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.525+0000 7fa73fe00700 1 -- start start 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400757c0 0x7fa7401a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73fe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400760e0 0x7fa7401a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73fe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa740114340 0x7fa7401a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73fe00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa740119b80 con 0x7fa740114340 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73fe00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa740119a00 con 0x7fa7400760e0 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73fe00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa740119d00 con 0x7fa7400757c0 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73ea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400757c0 0x7fa7401a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:08.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73e000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400760e0 0x7fa7401a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:08.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73ea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400757c0 0x7fa7401a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:45790/0 (socket says 172.21.15.33:45790) 2024-04-26T11:26:08.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73ea00700 1 -- 172.21.15.33:0/1823051518 learned_addr learned my addr 172.21.15.33:0/1823051518 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:08.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.526+0000 7fa73f400700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa740114340 0x7fa7401a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa73ea00700 1 -- 172.21.15.33:0/1823051518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400760e0 msgr2=0x7fa7401a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa73ea00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400760e0 0x7fa7401a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa73ea00700 1 -- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa740114340 msgr2=0x7fa7401a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa73ea00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa740114340 0x7fa7401a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa73ea00700 1 -- 172.21.15.33:0/1823051518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7401a5090 con 0x7fa7400757c0 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa73e000700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400760e0 0x7fa7401a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa73ea00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400757c0 0x7fa7401a00d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e9a0 tx=0x7fa73000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.527+0000 7fa737400700 1 -- 172.21.15.33:0/1823051518 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73000ca10 con 0x7fa7400757c0 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.528+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7401a5380 con 0x7fa7400757c0 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.528+0000 7fa737400700 1 -- 172.21.15.33:0/1823051518 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa73000cbb0 con 0x7fa7400757c0 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.528+0000 7fa737400700 1 -- 172.21.15.33:0/1823051518 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730018680 con 0x7fa7400757c0 2024-04-26T11:26:08.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.528+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7401acc00 con 0x7fa7400757c0 2024-04-26T11:26:08.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.529+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa74006a110 con 0x7fa7400757c0 2024-04-26T11:26:08.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.531+0000 7fa737400700 1 -- 172.21.15.33:0/1823051518 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa730010430 con 0x7fa7400757c0 2024-04-26T11:26:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.532+0000 7fa737400700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7240a3cc0 0x7fa7240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.532+0000 7fa737400700 1 -- 172.21.15.33:0/1823051518 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa7300d0130 con 0x7fa7400757c0 2024-04-26T11:26:08.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.533+0000 7fa73e000700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7240a3cc0 0x7fa7240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:08.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.533+0000 7fa73e000700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7240a3cc0 0x7fa7240a6180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa7401015a0 tx=0x7fa72c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:08.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.537+0000 7fa737400700 1 -- 172.21.15.33:0/1823051518 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa73009cf10 con 0x7fa7400757c0 2024-04-26T11:26:08.551 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mds[69072]: -- 172.21.15.33:0/2440444478 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a78877a40 con 0x563a7884ac00 2024-04-26T11:26:08.551 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: starting mds.a at 2024-04-26T11:26:08.870 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:08 smithi033 bash[69030]: 72590b12b61c3be371a043d086898a0d923cdbb6d1823f6c75260a431ec6ee0a 2024-04-26T11:26:08.870 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:26:08 smithi033 systemd[1]: Started Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:26:08.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.881+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7401a13e0 con 0x7fa7400757c0 2024-04-26T11:26:08.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.883+0000 7fa737400700 1 -- 172.21.15.33:0/1823051518 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7fa73009c810 con 0x7fa7400757c0 2024-04-26T11:26:08.884 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:26:08.884 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T11:26:08.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7240a3cc0 msgr2=0x7fa7240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7240a3cc0 0x7fa7240a6180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa7401015a0 tx=0x7fa72c00b260 comp rx=0 tx=0).stop 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400757c0 msgr2=0x7fa7401a00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400757c0 0x7fa7401a00d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e9a0 tx=0x7fa73000ecb0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 shutdown_connections 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7240a3cc0 0x7fa7240a6180 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa740114340 0x7fa7401a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7400757c0 0x7fa7401a00d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.885+0000 7fa73fe00700 1 --2- 172.21.15.33:0/1823051518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7400760e0 0x7fa7401a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.886+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 >> 172.21.15.33:0/1823051518 conn(0x7fa7400fee40 msgr2=0x7fa740100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.886+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 shutdown_connections 2024-04-26T11:26:08.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:08.886+0000 7fa73fe00700 1 -- 172.21.15.33:0/1823051518 wait complete. 2024-04-26T11:26:08.888 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 12 2024-04-26T11:26:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mon[27845]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:26:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:26:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:08 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:08 smithi096 ceph-mon[32753]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:26:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:26:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:08 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:08 smithi194 ceph-mon[33017]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:09.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T11:26:09.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T11:26:09.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:08 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:09 smithi096 ceph-mon[32753]: Deploying daemon mds.b on smithi096 2024-04-26T11:26:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:09 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:boot 2024-04-26T11:26:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:09 smithi096 ceph-mon[32753]: fsmap 11 up:standby 2024-04-26T11:26:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:09 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:26:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:09 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1823051518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:10.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:09 smithi194 ceph-mon[33017]: Deploying daemon mds.b on smithi096 2024-04-26T11:26:10.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:09 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:boot 2024-04-26T11:26:10.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:09 smithi194 ceph-mon[33017]: fsmap 11 up:standby 2024-04-26T11:26:10.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:09 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:26:10.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:09 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1823051518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:09 smithi033 ceph-mon[27845]: Deploying daemon mds.b on smithi096 2024-04-26T11:26:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:09 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:boot 2024-04-26T11:26:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:09 smithi033 ceph-mon[27845]: fsmap 11 up:standby 2024-04-26T11:26:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:09 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:26:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:09 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1823051518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:10.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:10 smithi096 ceph-mon[32753]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:10 smithi033 ceph-mon[27845]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:10 smithi194 ceph-mon[33017]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:11.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:26:11.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:26:11 smithi096 systemd[1]: Started Ceph mds.b for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:26:12.124 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] up:boot 2024-04-26T11:26:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: fsmap 12 up:standby 2024-04-26T11:26:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:26:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:12 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:26:12.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] up:boot 2024-04-26T11:26:12.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: fsmap 12 up:standby 2024-04-26T11:26:12.873 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:26:12.873 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:12 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] up:boot 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: fsmap 12 up:standby 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:26:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:12 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:26:13.843 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:13 smithi033 ceph-mon[27845]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:13.843 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:13 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:13 smithi194 ceph-mon[33017]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:13 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:13 smithi096 ceph-mon[32753]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:13 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.330+0000 7f84ace00700 1 -- 172.21.15.33:0/3106889485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 msgr2=0x7f84a8109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:14.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.330+0000 7f84ace00700 1 --2- 172.21.15.33:0/3106889485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 0x7f84a8109ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8494009990 tx=0x7f8494009ca0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.331+0000 7f84ace00700 1 -- 172.21.15.33:0/3106889485 shutdown_connections 2024-04-26T11:26:14.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.331+0000 7f84ace00700 1 --2- 172.21.15.33:0/3106889485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810a410 0x7f84a8116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.331+0000 7f84ace00700 1 --2- 172.21.15.33:0/3106889485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a810fa50 0x7f84a810fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.331+0000 7f84ace00700 1 --2- 172.21.15.33:0/3106889485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 0x7f84a8109ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.331+0000 7f84ace00700 1 -- 172.21.15.33:0/3106889485 >> 172.21.15.33:0/3106889485 conn(0x7f84a8076bf0 msgr2=0x7f84a8108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:14.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.331+0000 7f84ace00700 1 -- 172.21.15.33:0/3106889485 shutdown_connections 2024-04-26T11:26:14.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.331+0000 7f84ace00700 1 -- 172.21.15.33:0/3106889485 wait complete. 2024-04-26T11:26:14.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.332+0000 7f84ace00700 1 Processor -- start 2024-04-26T11:26:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.332+0000 7f84ace00700 1 -- start start 2024-04-26T11:26:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84ace00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 0x7f84a81a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84ace00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a810a410 0x7f84a81a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84ace00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810fa50 0x7f84a81a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84ace00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84a8119c30 con 0x7f84a810a410 2024-04-26T11:26:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84ace00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84a8119ab0 con 0x7f84a8109a50 2024-04-26T11:26:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84ace00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f84a8119db0 con 0x7f84a810fa50 2024-04-26T11:26:14.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84a6800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810fa50 0x7f84a81a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:14.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84a5e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 0x7f84a81a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:14.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84a6800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810fa50 0x7f84a81a49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:36336/0 (socket says 172.21.15.33:36336) 2024-04-26T11:26:14.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84a6800700 1 -- 172.21.15.33:0/3922540402 learned_addr learned my addr 172.21.15.33:0/3922540402 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.333+0000 7f84a5400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a810a410 0x7f84a81a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f84a6800700 1 -- 172.21.15.33:0/3922540402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 msgr2=0x7f84a81a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f84a6800700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 0x7f84a81a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f84a6800700 1 -- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a810a410 msgr2=0x7f84a81a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f84a6800700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a810a410 0x7f84a81a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f84a6800700 1 -- 172.21.15.33:0/3922540402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a81a5100 con 0x7f84a810fa50 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f84a5e00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 0x7f84a81a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f84a6800700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810fa50 0x7f84a81a49c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f84a000c370 tx=0x7f84a000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.334+0000 7f849ea00700 1 -- 172.21.15.33:0/3922540402 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a00092e0 con 0x7f84a810fa50 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.335+0000 7f849ea00700 1 -- 172.21.15.33:0/3922540402 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84a0003a00 con 0x7f84a810fa50 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.335+0000 7f849ea00700 1 -- 172.21.15.33:0/3922540402 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a00046f0 con 0x7f84a810fa50 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.335+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a81a5390 con 0x7f84a810fa50 2024-04-26T11:26:14.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.335+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a81acc70 con 0x7f84a810fa50 2024-04-26T11:26:14.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.336+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a806a110 con 0x7f84a810fa50 2024-04-26T11:26:14.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.338+0000 7f849ea00700 1 -- 172.21.15.33:0/3922540402 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f84a001a020 con 0x7f84a810fa50 2024-04-26T11:26:14.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.339+0000 7f849ea00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f848c0a3cc0 0x7f848c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:14.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.340+0000 7f849ea00700 1 -- 172.21.15.33:0/3922540402 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f84a00cf850 con 0x7f84a810fa50 2024-04-26T11:26:14.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.340+0000 7f84a5e00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f848c0a3cc0 0x7f848c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:14.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.340+0000 7f84a5e00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f848c0a3cc0 0x7f848c0a6180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8498009fd0 tx=0x7f8498009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:14.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.344+0000 7f849ea00700 1 -- 172.21.15.33:0/3922540402 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84a0010040 con 0x7f84a810fa50 2024-04-26T11:26:14.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.633+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84a802e490 con 0x7f84a810fa50 2024-04-26T11:26:14.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.634+0000 7f849ea00700 1 -- 172.21.15.33:0/3922540402 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7f84a009c630 con 0x7f84a810fa50 2024-04-26T11:26:14.636 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:26:14.636 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24527,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T11:26:14.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f848c0a3cc0 msgr2=0x7f848c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:14.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f848c0a3cc0 0x7f848c0a6180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8498009fd0 tx=0x7f8498009260 comp rx=0 tx=0).stop 2024-04-26T11:26:14.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810fa50 msgr2=0x7f84a81a49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810fa50 0x7f84a81a49c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f84a000c370 tx=0x7f84a000c680 comp rx=0 tx=0).stop 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 shutdown_connections 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f848c0a3cc0 0x7f848c0a6180 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84a810fa50 0x7f84a81a49c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84a8109a50 0x7f84a81a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 --2- 172.21.15.33:0/3922540402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84a810a410 0x7f84a81a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.638+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 >> 172.21.15.33:0/3922540402 conn(0x7f84a8076bf0 msgr2=0x7f84a80771f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.639+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 shutdown_connections 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:14.639+0000 7f84ace00700 1 -- 172.21.15.33:0/3922540402 wait complete. 2024-04-26T11:26:14.640 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 13 2024-04-26T11:26:14.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:14 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:14 smithi033 ceph-mon[27845]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:14 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:14 smithi096 ceph-mon[32753]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:14 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:14 smithi194 ceph-mon[33017]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:15.899 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:15 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3922540402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:15 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3922540402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:15.960 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-04-26T11:26:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:15 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3922540402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:26:16.422 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:16.858 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:16 smithi033 ceph-mon[27845]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:16 smithi096 ceph-mon[32753]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:16 smithi194 ceph-mon[33017]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.526+0000 7f72e0000700 1 -- 172.21.15.33:0/807176723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d81094a0 msgr2=0x7f72d8109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.526+0000 7f72e0000700 1 --2- 172.21.15.33:0/807176723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d81094a0 0x7f72d8109880 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009990 tx=0x7f72cc009ca0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.527+0000 7f72e0000700 1 -- 172.21.15.33:0/807176723 shutdown_connections 2024-04-26T11:26:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.527+0000 7f72e0000700 1 --2- 172.21.15.33:0/807176723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d806c4f0 0x7f72d81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.527+0000 7f72e0000700 1 --2- 172.21.15.33:0/807176723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d81094a0 0x7f72d8109880 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.527+0000 7f72e0000700 1 --2- 172.21.15.33:0/807176723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 0x7f72d806bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.527+0000 7f72e0000700 1 -- 172.21.15.33:0/807176723 >> 172.21.15.33:0/807176723 conn(0x7f72d80792d0 msgr2=0x7f72d80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:18.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.527+0000 7f72e0000700 1 -- 172.21.15.33:0/807176723 shutdown_connections 2024-04-26T11:26:18.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.527+0000 7f72e0000700 1 -- 172.21.15.33:0/807176723 wait complete. 2024-04-26T11:26:18.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.528+0000 7f72e0000700 1 Processor -- start 2024-04-26T11:26:18.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.528+0000 7f72e0000700 1 -- start start 2024-04-26T11:26:18.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.528+0000 7f72e0000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 0x7f72d819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:18.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.528+0000 7f72e0000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d806c4f0 0x7f72d819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:18.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72e0000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d81094a0 0x7f72d81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:18.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72e0000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f72d81135e0 con 0x7f72d806bb50 2024-04-26T11:26:18.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72e0000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72d8113460 con 0x7f72d81094a0 2024-04-26T11:26:18.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72e0000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f72d8113760 con 0x7f72d806c4f0 2024-04-26T11:26:18.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72dd600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 0x7f72d819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:18.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72dcc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d806c4f0 0x7f72d819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:18.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72dcc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d806c4f0 0x7f72d819c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:36360/0 (socket says 172.21.15.33:36360) 2024-04-26T11:26:18.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72dd600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 0x7f72d819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33550/0 (socket says 172.21.15.33:33550) 2024-04-26T11:26:18.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72dcc00700 1 -- 172.21.15.33:0/2193037393 learned_addr learned my addr 172.21.15.33:0/2193037393 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:18.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72de000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d81094a0 0x7f72d81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:18.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72de000700 1 -- 172.21.15.33:0/2193037393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d806c4f0 msgr2=0x7f72d819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:18.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72de000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d806c4f0 0x7f72d819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72de000700 1 -- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 msgr2=0x7f72d819bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:18.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72de000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 0x7f72d819bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.529+0000 7f72de000700 1 -- 172.21.15.33:0/2193037393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d81a0d50 con 0x7f72d81094a0 2024-04-26T11:26:18.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.530+0000 7f72dd600700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 0x7f72d819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:18.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.530+0000 7f72de000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d81094a0 0x7f72d81a0610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f72d000efb0 tx=0x7f72d000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:18.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.530+0000 7f72d6000700 1 -- 172.21.15.33:0/2193037393 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0003a00 con 0x7f72d81094a0 2024-04-26T11:26:18.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.530+0000 7f72d6000700 1 -- 172.21.15.33:0/2193037393 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d000ce50 con 0x7f72d81094a0 2024-04-26T11:26:18.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.530+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d81a1040 con 0x7f72d81094a0 2024-04-26T11:26:18.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.530+0000 7f72d6000700 1 -- 172.21.15.33:0/2193037393 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0004710 con 0x7f72d81094a0 2024-04-26T11:26:18.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.530+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d81a88c0 con 0x7f72d81094a0 2024-04-26T11:26:18.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.532+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d806a110 con 0x7f72d81094a0 2024-04-26T11:26:18.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.533+0000 7f72d6000700 1 -- 172.21.15.33:0/2193037393 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f72d0012050 con 0x7f72d81094a0 2024-04-26T11:26:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.535+0000 7f72d6000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72c00a3cc0 0x7f72c00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.535+0000 7f72d6000700 1 -- 172.21.15.33:0/2193037393 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f72d00cf800 con 0x7f72d81094a0 2024-04-26T11:26:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.535+0000 7f72dd600700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72c00a3cc0 0x7f72c00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.536+0000 7f72dd600700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72c00a3cc0 0x7f72c00a6180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00b2e0 tx=0x7f72cc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.539+0000 7f72d6000700 1 -- 172.21.15.33:0/2193037393 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72d0022060 con 0x7f72d81094a0 2024-04-26T11:26:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.781+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f72d802e490 con 0x7f72d81094a0 2024-04-26T11:26:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.782+0000 7f72d6000700 1 -- 172.21.15.33:0/2193037393 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f72d0010040 con 0x7f72d81094a0 2024-04-26T11:26:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:26:18.784 INFO:teuthology.orchestra.run.smithi033.stdout:[ 2024-04-26T11:26:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-26T11:26:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: "poolnum": 1, 2024-04-26T11:26:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: "poolname": ".mgr" 2024-04-26T11:26:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:26:18.784 INFO:teuthology.orchestra.run.smithi033.stdout:] 2024-04-26T11:26:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72c00a3cc0 msgr2=0x7f72c00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:18.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72c00a3cc0 0x7f72c00a6180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00b2e0 tx=0x7f72cc01a040 comp rx=0 tx=0).stop 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d81094a0 msgr2=0x7f72d81a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d81094a0 0x7f72d81a0610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f72d000efb0 tx=0x7f72d000c3a0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 shutdown_connections 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72c00a3cc0 0x7f72c00a6180 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d81094a0 0x7f72d81a0610 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d806bb50 0x7f72d819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 --2- 172.21.15.33:0/2193037393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d806c4f0 0x7f72d819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.785+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 >> 172.21.15.33:0/2193037393 conn(0x7f72d80792d0 msgr2=0x7f72d810e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.786+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 shutdown_connections 2024-04-26T11:26:18.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:18.786+0000 7f72e0000700 1 -- 172.21.15.33:0/2193037393 wait complete. 2024-04-26T11:26:19.171 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:19.171 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.172 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.172 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:19.172 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:26:19.172 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.172 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:26:19.172 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:18 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2193037393' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:26:19.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:18 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2193037393' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T11:26:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:26:19.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:19.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:26:19.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:18 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2193037393' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T11:26:19.666 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T11:26:19.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T11:26:20.750 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:21.097 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:20 smithi033 ceph-mon[27845]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:21.098 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.098 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.098 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.098 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:20 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:20 smithi096 ceph-mon[32753]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:20 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:20 smithi194 ceph-mon[33017]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:21.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:21.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:20 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:22.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 -- 172.21.15.33:0/2238327811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba01083b0 msgr2=0x7f6ba0108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:22.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/2238327811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba01083b0 0x7f6ba0108790 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009990 tx=0x7f6b90009ca0 comp rx=0 tx=0).stop 2024-04-26T11:26:22.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 -- 172.21.15.33:0/2238327811 shutdown_connections 2024-04-26T11:26:22.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/2238327811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ba0103e90 0x7f6ba0110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:22.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/2238327811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba01083b0 0x7f6ba0108790 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:22.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/2238327811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ba01034d0 0x7f6ba0103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:22.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 -- 172.21.15.33:0/2238327811 >> 172.21.15.33:0/2238327811 conn(0x7f6ba007a580 msgr2=0x7f6ba007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:22.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 -- 172.21.15.33:0/2238327811 shutdown_connections 2024-04-26T11:26:22.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.742+0000 7f6ba6a00700 1 -- 172.21.15.33:0/2238327811 wait complete. 2024-04-26T11:26:22.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.743+0000 7f6ba6a00700 1 Processor -- start 2024-04-26T11:26:22.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.743+0000 7f6ba6a00700 1 -- start start 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6ba6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ba01034d0 0x7f6ba01a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6ba6a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba0103e90 0x7f6ba01a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6ba6a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ba01083b0 0x7f6ba019bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6ba6a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0113690 con 0x7f6ba01034d0 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6ba6a00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0113510 con 0x7f6ba01083b0 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6ba6a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0113810 con 0x7f6ba0103e90 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6b9f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba0103e90 0x7f6ba01a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6b9f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba0103e90 0x7f6ba01a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:41014/0 (socket says 172.21.15.33:41014) 2024-04-26T11:26:22.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6b9f400700 1 -- 172.21.15.33:0/590829685 learned_addr learned my addr 172.21.15.33:0/590829685 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6b9fe00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ba01034d0 0x7f6ba01a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.744+0000 7f6ba4a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ba01083b0 0x7f6ba019bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9f400700 1 -- 172.21.15.33:0/590829685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ba01083b0 msgr2=0x7f6ba019bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9f400700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ba01083b0 0x7f6ba019bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9f400700 1 -- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ba01034d0 msgr2=0x7f6ba01a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9f400700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ba01034d0 0x7f6ba01a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9f400700 1 -- 172.21.15.33:0/590829685 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba019c260 con 0x7f6ba0103e90 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6ba4a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ba01083b0 0x7f6ba019bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9f400700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba0103e90 0x7f6ba01a1e80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00e9a0 tx=0x7f6b8c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9cc00700 1 -- 172.21.15.33:0/590829685 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00cbd0 con 0x7f6ba0103e90 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9cc00700 1 -- 172.21.15.33:0/590829685 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b8c00cd70 con 0x7f6ba0103e90 2024-04-26T11:26:22.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6b9cc00700 1 -- 172.21.15.33:0/590829685 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c018680 con 0x7f6ba0103e90 2024-04-26T11:26:22.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.745+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba019c550 con 0x7f6ba0103e90 2024-04-26T11:26:22.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.746+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba01a8810 con 0x7f6ba0103e90 2024-04-26T11:26:22.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.747+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba006a110 con 0x7f6ba0103e90 2024-04-26T11:26:22.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.750+0000 7f6b9cc00700 1 -- 172.21.15.33:0/590829685 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6b8c010430 con 0x7f6ba0103e90 2024-04-26T11:26:22.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.751+0000 7f6b9cc00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b840a3cc0 0x7f6b840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:22.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.752+0000 7f6b9cc00700 1 -- 172.21.15.33:0/590829685 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6b8c0d0ba0 con 0x7f6ba0103e90 2024-04-26T11:26:22.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.752+0000 7f6b9fe00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b840a3cc0 0x7f6b840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:22.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.752+0000 7f6b9cc00700 1 -- 172.21.15.33:0/590829685 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b8c026070 con 0x7f6ba0103e90 2024-04-26T11:26:22.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.753+0000 7f6b9fe00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b840a3cc0 0x7f6b840a6180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b790 tx=0x7f6b9000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:22.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:22.994+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f6ba019cff0 con 0x7f6ba0103e90 2024-04-26T11:26:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:22 smithi033 ceph-mon[27845]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:22 smithi096 ceph-mon[32753]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:22 smithi194 ceph-mon[33017]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:23.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.937+0000 7f6b9cc00700 1 -- 172.21.15.33:0/590829685 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f6b8c09d980 con 0x7f6ba0103e90 2024-04-26T11:26:23.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b840a3cc0 msgr2=0x7f6b840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:23.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b840a3cc0 0x7f6b840a6180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b790 tx=0x7f6b9000b260 comp rx=0 tx=0).stop 2024-04-26T11:26:23.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba0103e90 msgr2=0x7f6ba01a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:23.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba0103e90 0x7f6ba01a1e80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00e9a0 tx=0x7f6b8c00ecb0 comp rx=0 tx=0).stop 2024-04-26T11:26:23.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 shutdown_connections 2024-04-26T11:26:23.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6b840a3cc0 0x7f6b840a6180 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:23.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ba01083b0 0x7f6ba019bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:23.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ba01034d0 0x7f6ba01a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:23.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 --2- 172.21.15.33:0/590829685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ba0103e90 0x7f6ba01a1e80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:23.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.945+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 >> 172.21.15.33:0/590829685 conn(0x7f6ba007a580 msgr2=0x7f6ba01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:23.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.947+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 shutdown_connections 2024-04-26T11:26:23.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:23.947+0000 7f6ba6a00700 1 -- 172.21.15.33:0/590829685 wait complete. 2024-04-26T11:26:23.954 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_metadata' created 2024-04-26T11:26:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:26:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:23 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/590829685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T11:26:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:23 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T11:26:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:26:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:23 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/590829685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T11:26:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:23 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T11:26:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:26:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:26:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:26:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:23 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/590829685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T11:26:24.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:23 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T11:26:24.824 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T11:26:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:24 smithi033 ceph-mon[27845]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:24 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T11:26:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:24 smithi033 ceph-mon[27845]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T11:26:25.317 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:24 smithi194 ceph-mon[33017]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:24 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T11:26:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:24 smithi194 ceph-mon[33017]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T11:26:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:24 smithi096 ceph-mon[32753]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:24 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T11:26:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:24 smithi096 ceph-mon[32753]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T11:26:26.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:25 smithi033 ceph-mon[27845]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T11:26:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:25 smithi194 ceph-mon[33017]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T11:26:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:25 smithi096 ceph-mon[32753]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T11:26:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:26 smithi033 ceph-mon[27845]: pgmap v317: 33 pgs: 6 creating+peering, 4 active+clean, 23 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:26 smithi033 ceph-mon[27845]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T11:26:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:26 smithi033 ceph-mon[27845]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T11:26:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:26 smithi194 ceph-mon[33017]: pgmap v317: 33 pgs: 6 creating+peering, 4 active+clean, 23 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:26 smithi194 ceph-mon[33017]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T11:26:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:26 smithi194 ceph-mon[33017]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T11:26:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:26 smithi096 ceph-mon[32753]: pgmap v317: 33 pgs: 6 creating+peering, 4 active+clean, 23 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:26 smithi096 ceph-mon[32753]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T11:26:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:26 smithi096 ceph-mon[32753]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T11:26:27.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 -- 172.21.15.33:0/2398864345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406bad0 msgr2=0x7f75f41102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 --2- 172.21.15.33:0/2398864345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406bad0 0x7f75f41102e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009f90 tx=0x7f75ec00b330 comp rx=0 tx=0).stop 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 -- 172.21.15.33:0/2398864345 shutdown_connections 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 --2- 172.21.15.33:0/2398864345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406bad0 0x7f75f41102e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 --2- 172.21.15.33:0/2398864345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f41094a0 0x7f75f4109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 --2- 172.21.15.33:0/2398864345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f406b130 0x7f75f406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 -- 172.21.15.33:0/2398864345 >> 172.21.15.33:0/2398864345 conn(0x7f75f40792d0 msgr2=0x7f75f40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.684+0000 7f75f9e00700 1 -- 172.21.15.33:0/2398864345 shutdown_connections 2024-04-26T11:26:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.685+0000 7f75f9e00700 1 -- 172.21.15.33:0/2398864345 wait complete. 2024-04-26T11:26:27.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 Processor -- start 2024-04-26T11:26:27.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 -- start start 2024-04-26T11:26:27.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406b130 0x7f75f419bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f406bad0 0x7f75f419c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f41094a0 0x7f75f41a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4113620 con 0x7f75f406bad0 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41134a0 con 0x7f75f406b130 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.686+0000 7f75f9e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41137a0 con 0x7f75f41094a0 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f2000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f406bad0 0x7f75f419c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f3400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f41094a0 0x7f75f41a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f2000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f406bad0 0x7f75f419c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38892/0 (socket says 172.21.15.33:38892) 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f2000700 1 -- 172.21.15.33:0/3805921738 learned_addr learned my addr 172.21.15.33:0/3805921738 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:27.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f2a00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406b130 0x7f75f419bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:27.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f3400700 1 -- 172.21.15.33:0/3805921738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406b130 msgr2=0x7f75f419bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:27.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f3400700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406b130 0x7f75f419bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f3400700 1 -- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f406bad0 msgr2=0x7f75f419c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f3400700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f406bad0 0x7f75f419c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f3400700 1 -- 172.21.15.33:0/3805921738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f41a0cf0 con 0x7f75f41094a0 2024-04-26T11:26:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.687+0000 7f75f3400700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f41094a0 0x7f75f41a05b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00b6c0 tx=0x7f75ec00cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.688+0000 7f75f8a00700 1 -- 172.21.15.33:0/3805921738 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec021070 con 0x7f75f41094a0 2024-04-26T11:26:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.688+0000 7f75f8a00700 1 -- 172.21.15.33:0/3805921738 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75ec007ac0 con 0x7f75f41094a0 2024-04-26T11:26:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.688+0000 7f75f8a00700 1 -- 172.21.15.33:0/3805921738 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec005040 con 0x7f75f41094a0 2024-04-26T11:26:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.688+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f41a0f80 con 0x7f75f41094a0 2024-04-26T11:26:27.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.688+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41a8860 con 0x7f75f41094a0 2024-04-26T11:26:27.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.689+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f406a110 con 0x7f75f41094a0 2024-04-26T11:26:27.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.692+0000 7f75f8a00700 1 -- 172.21.15.33:0/3805921738 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f75ec024070 con 0x7f75f41094a0 2024-04-26T11:26:27.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.693+0000 7f75f8a00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f75d80a3cc0 0x7f75d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:27.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.693+0000 7f75f8a00700 1 -- 172.21.15.33:0/3805921738 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f75ec00f020 con 0x7f75f41094a0 2024-04-26T11:26:27.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.694+0000 7f75f2a00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f75d80a3cc0 0x7f75d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:27.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.694+0000 7f75f2a00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f75d80a3cc0 0x7f75d80a6180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f75e4005fd0 tx=0x7f75e4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:27.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.695+0000 7f75f8a00700 1 -- 172.21.15.33:0/3805921738 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75ec02b020 con 0x7f75f41094a0 2024-04-26T11:26:27.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.929+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f75f402e490 con 0x7f75f41094a0 2024-04-26T11:26:27.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.985+0000 7f75f8a00700 1 -- 172.21.15.33:0/3805921738 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) ==== 170+0+0 (secure 0 0 0) 0x7f75ec09e520 con 0x7f75f41094a0 2024-04-26T11:26:27.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f75d80a3cc0 msgr2=0x7f75d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:27.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f75d80a3cc0 0x7f75d80a6180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f75e4005fd0 tx=0x7f75e4009310 comp rx=0 tx=0).stop 2024-04-26T11:26:27.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f41094a0 msgr2=0x7f75f41a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:27.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f41094a0 0x7f75f41a05b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00b6c0 tx=0x7f75ec00cc30 comp rx=0 tx=0).stop 2024-04-26T11:26:27.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 shutdown_connections 2024-04-26T11:26:27.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f75d80a3cc0 0x7f75d80a6180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f41094a0 0x7f75f41a05b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75f406b130 0x7f75f419bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.992+0000 7f75f9e00700 1 --2- 172.21.15.33:0/3805921738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75f406bad0 0x7f75f419c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:27.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.993+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 >> 172.21.15.33:0/3805921738 conn(0x7f75f40792d0 msgr2=0x7f75f410e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:27.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.993+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 shutdown_connections 2024-04-26T11:26:27.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:27.993+0000 7f75f9e00700 1 -- 172.21.15.33:0/3805921738 wait complete. 2024-04-26T11:26:28.001 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data' created 2024-04-26T11:26:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:27 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3805921738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T11:26:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:27 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T11:26:28.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:27 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3805921738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T11:26:28.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:27 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T11:26:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:27 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3805921738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T11:26:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:27 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T11:26:28.948 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T11:26:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:28 smithi033 ceph-mon[27845]: pgmap v319: 33 pgs: 6 creating+peering, 18 active+clean, 9 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:28 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T11:26:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:28 smithi033 ceph-mon[27845]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T11:26:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:28 smithi194 ceph-mon[33017]: pgmap v319: 33 pgs: 6 creating+peering, 18 active+clean, 9 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:28 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T11:26:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:28 smithi194 ceph-mon[33017]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T11:26:29.425 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:28 smithi096 ceph-mon[32753]: pgmap v319: 33 pgs: 6 creating+peering, 18 active+clean, 9 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:28 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T11:26:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:28 smithi096 ceph-mon[32753]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T11:26:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:30 smithi033 ceph-mon[27845]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T11:26:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:30 smithi194 ceph-mon[33017]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T11:26:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:30 smithi096 ceph-mon[32753]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T11:26:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:31 smithi033 ceph-mon[27845]: pgmap v322: 97 pgs: 1 active, 1 creating+activating, 6 creating+peering, 37 active+clean, 52 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:31 smithi033 ceph-mon[27845]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T11:26:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:31 smithi033 ceph-mon[27845]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T11:26:31.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:31 smithi194 ceph-mon[33017]: pgmap v322: 97 pgs: 1 active, 1 creating+activating, 6 creating+peering, 37 active+clean, 52 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:31.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:31 smithi194 ceph-mon[33017]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T11:26:31.256 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:31 smithi194 ceph-mon[33017]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T11:26:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:31 smithi096 ceph-mon[32753]: pgmap v322: 97 pgs: 1 active, 1 creating+activating, 6 creating+peering, 37 active+clean, 52 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:31 smithi096 ceph-mon[32753]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T11:26:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:31 smithi096 ceph-mon[32753]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T11:26:31.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 -- 172.21.15.33:0/2118043775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 msgr2=0x7f87cc109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:31.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 --2- 172.21.15.33:0/2118043775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 0x7f87cc109880 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f87c000b0a0 tx=0x7f87c000b3b0 comp rx=0 tx=0).stop 2024-04-26T11:26:31.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 -- 172.21.15.33:0/2118043775 shutdown_connections 2024-04-26T11:26:31.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 --2- 172.21.15.33:0/2118043775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 0x7f87cc1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:31.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 --2- 172.21.15.33:0/2118043775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 0x7f87cc109880 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:31.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 --2- 172.21.15.33:0/2118043775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87cc06b130 0x7f87cc06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:31.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 -- 172.21.15.33:0/2118043775 >> 172.21.15.33:0/2118043775 conn(0x7f87cc0792d0 msgr2=0x7f87cc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:31.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 -- 172.21.15.33:0/2118043775 shutdown_connections 2024-04-26T11:26:31.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.775+0000 7f87d3e00700 1 -- 172.21.15.33:0/2118043775 wait complete. 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.776+0000 7f87d3e00700 1 Processor -- start 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.776+0000 7f87d3e00700 1 -- start start 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87cc06b130 0x7f87cc19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d3e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 0x7f87cc19c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 0x7f87cc1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d3e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc1135b0 con 0x7f87cc1094a0 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d3e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc113430 con 0x7f87cc06bad0 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d3e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc113730 con 0x7f87cc06b130 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d0a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 0x7f87cc19c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d1e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 0x7f87cc1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:31.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d1400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87cc06b130 0x7f87cc19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:31.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d0a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 0x7f87cc19c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:52630/0 (socket says 172.21.15.33:52630) 2024-04-26T11:26:31.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.777+0000 7f87d0a00700 1 -- 172.21.15.33:0/3790831551 learned_addr learned my addr 172.21.15.33:0/3790831551 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:31.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d0a00700 1 -- 172.21.15.33:0/3790831551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87cc06b130 msgr2=0x7f87cc19bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d0a00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87cc06b130 0x7f87cc19bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d0a00700 1 -- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 msgr2=0x7f87cc1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d0a00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 0x7f87cc1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d0a00700 1 -- 172.21.15.33:0/3790831551 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc1a0d30 con 0x7f87cc06bad0 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d1e00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 0x7f87cc1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d0a00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 0x7f87cc19c260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e9c0 tx=0x7f87b800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87ca000700 1 -- 172.21.15.33:0/3790831551 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b800cbd0 con 0x7f87cc06bad0 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87ca000700 1 -- 172.21.15.33:0/3790831551 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b800ee50 con 0x7f87cc06bad0 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87cc1a1020 con 0x7f87cc06bad0 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87ca000700 1 -- 172.21.15.33:0/3790831551 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b8005550 con 0x7f87cc06bad0 2024-04-26T11:26:31.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.778+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87cc1a88a0 con 0x7f87cc06bad0 2024-04-26T11:26:31.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.780+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87cc06a110 con 0x7f87cc06bad0 2024-04-26T11:26:31.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.782+0000 7f87ca000700 1 -- 172.21.15.33:0/3790831551 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f87b8004750 con 0x7f87cc06bad0 2024-04-26T11:26:31.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.783+0000 7f87ca000700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f87b40a3cc0 0x7f87b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:31.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.783+0000 7f87ca000700 1 -- 172.21.15.33:0/3790831551 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7879+0+0 (secure 0 0 0) 0x7f87b80d04f0 con 0x7f87cc06bad0 2024-04-26T11:26:31.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.783+0000 7f87d1400700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f87b40a3cc0 0x7f87b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:31.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.784+0000 7f87d1400700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f87b40a3cc0 0x7f87b40a6180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009e60 tx=0x7f87c000b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:31.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:31.787+0000 7f87ca000700 1 -- 172.21.15.33:0/3790831551 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87b809cfe0 con 0x7f87cc06bad0 2024-04-26T11:26:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:32.070+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f87cc02e490 con 0x7f87cc06bad0 2024-04-26T11:26:33.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.028+0000 7f87ca000700 1 -- 172.21.15.33:0/3790831551 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f87b809c8e0 con 0x7f87cc06bad0 2024-04-26T11:26:33.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.035+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f87b40a3cc0 msgr2=0x7f87b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:33.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.035+0000 7f87d3e00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f87b40a3cc0 0x7f87b40a6180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009e60 tx=0x7f87c000b700 comp rx=0 tx=0).stop 2024-04-26T11:26:33.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.035+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 msgr2=0x7f87cc19c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:33.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.035+0000 7f87d3e00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 0x7f87cc19c260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e9c0 tx=0x7f87b800ecd0 comp rx=0 tx=0).stop 2024-04-26T11:26:33.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.036+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 shutdown_connections 2024-04-26T11:26:33.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.036+0000 7f87d3e00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f87b40a3cc0 0x7f87b40a6180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:33.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.036+0000 7f87d3e00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87cc1094a0 0x7f87cc1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:33.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.036+0000 7f87d3e00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87cc06b130 0x7f87cc19bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:33.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.036+0000 7f87d3e00700 1 --2- 172.21.15.33:0/3790831551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87cc06bad0 0x7f87cc19c260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:33.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.036+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 >> 172.21.15.33:0/3790831551 conn(0x7f87cc0792d0 msgr2=0x7f87cc10e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:33.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.037+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 shutdown_connections 2024-04-26T11:26:33.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:33.037+0000 7f87d3e00700 1 -- 172.21.15.33:0/3790831551 wait complete. 2024-04-26T11:26:33.046 INFO:teuthology.orchestra.run.smithi033.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T11:26:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:33 smithi194 ceph-mon[33017]: pgmap v324: 97 pgs: 1 active, 1 creating+activating, 6 creating+peering, 70 active+clean, 19 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:33 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3790831551' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T11:26:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:33 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T11:26:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:33 smithi194 ceph-mon[33017]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T11:26:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:33 smithi033 ceph-mon[27845]: pgmap v324: 97 pgs: 1 active, 1 creating+activating, 6 creating+peering, 70 active+clean, 19 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:33 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3790831551' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T11:26:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:33 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T11:26:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:33 smithi033 ceph-mon[27845]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T11:26:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:33 smithi096 ceph-mon[32753]: pgmap v324: 97 pgs: 1 active, 1 creating+activating, 6 creating+peering, 70 active+clean, 19 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:26:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:33 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3790831551' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T11:26:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:33 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T11:26:33.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:33 smithi096 ceph-mon[32753]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T11:26:33.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:26:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:34 smithi033 ceph-mon[27845]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T11:26:34.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:34 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T11:26:34.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:34 smithi033 ceph-mon[27845]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T11:26:34.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:34 smithi033 ceph-mon[27845]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T11:26:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:34 smithi194 ceph-mon[33017]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T11:26:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:34 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T11:26:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:34 smithi194 ceph-mon[33017]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T11:26:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:34 smithi194 ceph-mon[33017]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T11:26:34.421 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:34 smithi096 ceph-mon[32753]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T11:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:34 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T11:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:34 smithi096 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T11:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:34 smithi096 ceph-mon[32753]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T11:26:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:35 smithi033 ceph-mon[27845]: pgmap v326: 97 pgs: 1 active, 1 creating+activating, 2 creating+peering, 93 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 175 B/s wr, 3 op/s 2024-04-26T11:26:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:35 smithi033 ceph-mon[27845]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T11:26:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:35 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:26:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:35 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] up:active 2024-04-26T11:26:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:35 smithi033 ceph-mon[27845]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:35 smithi033 ceph-mon[27845]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:35 smithi194 ceph-mon[33017]: pgmap v326: 97 pgs: 1 active, 1 creating+activating, 2 creating+peering, 93 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 175 B/s wr, 3 op/s 2024-04-26T11:26:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:35 smithi194 ceph-mon[33017]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T11:26:35.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:35 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:26:35.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:35 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] up:active 2024-04-26T11:26:35.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:35 smithi194 ceph-mon[33017]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:35.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:35 smithi194 ceph-mon[33017]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:35 smithi096 ceph-mon[32753]: pgmap v326: 97 pgs: 1 active, 1 creating+activating, 2 creating+peering, 93 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 175 B/s wr, 3 op/s 2024-04-26T11:26:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:35 smithi096 ceph-mon[32753]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T11:26:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:35 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:26:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:35 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] up:active 2024-04-26T11:26:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:35 smithi096 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:35 smithi096 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:36.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 -- 172.21.15.33:0/189797600 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c105f30 msgr2=0x7f382c1123f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:36.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 --2- 172.21.15.33:0/189797600 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c105f30 0x7f382c1123f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38240099c0 tx=0x7f3824009cd0 comp rx=0 tx=0).stop 2024-04-26T11:26:36.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 -- 172.21.15.33:0/189797600 shutdown_connections 2024-04-26T11:26:36.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 --2- 172.21.15.33:0/189797600 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c105f30 0x7f382c1123f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:36.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 --2- 172.21.15.33:0/189797600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f382c10b620 0x7f382c10ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:36.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 --2- 172.21.15.33:0/189797600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105570 0x7f382c1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:36.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 -- 172.21.15.33:0/189797600 >> 172.21.15.33:0/189797600 conn(0x7f382c101040 msgr2=0x7f382c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:36.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 -- 172.21.15.33:0/189797600 shutdown_connections 2024-04-26T11:26:36.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.780+0000 7f3831400700 1 -- 172.21.15.33:0/189797600 wait complete. 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.781+0000 7f3831400700 1 Processor -- start 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.781+0000 7f3831400700 1 -- start start 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3831400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f382c105570 0x7f382c1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3831400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105f30 0x7f382c1a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3831400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c10b620 0x7f382c19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3831400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f382c115780 con 0x7f382c10b620 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3831400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f382c115600 con 0x7f382c105f30 2024-04-26T11:26:36.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3831400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f382c115900 con 0x7f382c105570 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105f30 0x7f382c1a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3830a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c10b620 0x7f382c19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105f30 0x7f382c1a1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:52654/0 (socket says 172.21.15.33:52654) 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 -- 172.21.15.33:0/3351617697 learned_addr learned my addr 172.21.15.33:0/3351617697 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f382be00700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f382c105570 0x7f382c1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 -- 172.21.15.33:0/3351617697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f382c105570 msgr2=0x7f382c1a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f382c105570 0x7f382c1a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 -- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c10b620 msgr2=0x7f382c19b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c10b620 0x7f382c19b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:36.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.782+0000 7f3823400700 1 -- 172.21.15.33:0/3351617697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382c19bfe0 con 0x7f382c105f30 2024-04-26T11:26:36.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.783+0000 7f3823400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105f30 0x7f382c1a1c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f381400e9a0 tx=0x7f381400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:36.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.783+0000 7f3829600700 1 -- 172.21.15.33:0/3351617697 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381400ca10 con 0x7f382c105f30 2024-04-26T11:26:36.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.783+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382c19c2d0 con 0x7f382c105f30 2024-04-26T11:26:36.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.783+0000 7f3829600700 1 -- 172.21.15.33:0/3351617697 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f381400cbb0 con 0x7f382c105f30 2024-04-26T11:26:36.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.783+0000 7f3829600700 1 -- 172.21.15.33:0/3351617697 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814018680 con 0x7f382c105f30 2024-04-26T11:26:36.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.783+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f382c1a8590 con 0x7f382c105f30 2024-04-26T11:26:36.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.784+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382c06a110 con 0x7f382c105f30 2024-04-26T11:26:36.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.786+0000 7f3829600700 1 -- 172.21.15.33:0/3351617697 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3814010430 con 0x7f382c105f30 2024-04-26T11:26:36.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.788+0000 7f3829600700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f38100a3cc0 0x7f38100a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:36.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.788+0000 7f3829600700 1 -- 172.21.15.33:0/3351617697 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f38140d04f0 con 0x7f382c105f30 2024-04-26T11:26:36.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.788+0000 7f382be00700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f38100a3cc0 0x7f38100a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:36.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.789+0000 7f382be00700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f38100a3cc0 0x7f38100a6180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f382c19c9c0 tx=0x7f381c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:36.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:36.790+0000 7f3829600700 1 -- 172.21.15.33:0/3351617697 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f381409cf80 con 0x7f382c105f30 2024-04-26T11:26:37.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.033+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f382c19cdb0 con 0x7f382c105f30 2024-04-26T11:26:37.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.035+0000 7f3829600700 1 -- 172.21.15.33:0/3351617697 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22168 (secure 0 0 0) 0x7f381409c880 con 0x7f382c105f30 2024-04-26T11:26:37.037 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:26:37.038 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":88,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:26:32.073273+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:26:37.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f38100a3cc0 msgr2=0x7f38100a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:37.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f38100a3cc0 0x7f38100a6180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f382c19c9c0 tx=0x7f381c00b040 comp rx=0 tx=0).stop 2024-04-26T11:26:37.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105f30 msgr2=0x7f382c1a1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:37.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105f30 0x7f382c1a1c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f381400e9a0 tx=0x7f381400ecb0 comp rx=0 tx=0).stop 2024-04-26T11:26:37.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 shutdown_connections 2024-04-26T11:26:37.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f38100a3cc0 0x7f38100a6180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:37.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f382c10b620 0x7f382c19b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:37.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f382c105570 0x7f382c1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:37.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 --2- 172.21.15.33:0/3351617697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f382c105f30 0x7f382c1a1c00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:37.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 >> 172.21.15.33:0/3351617697 conn(0x7f382c101040 msgr2=0x7f382c1026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:37.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.038+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 shutdown_connections 2024-04-26T11:26:37.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:37.039+0000 7f3831400700 1 -- 172.21.15.33:0/3351617697 wait complete. 2024-04-26T11:26:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:37 smithi096 ceph-mon[32753]: pgmap v327: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T11:26:37.183 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:26:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:26:36.892+0000 7f31c45fd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T11:26:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:37 smithi194 ceph-mon[33017]: pgmap v327: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T11:26:37.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:37 smithi033 ceph-mon[27845]: pgmap v327: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T11:26:37.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:26:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:38 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3351617697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:26:38.306 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:38 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3351617697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:26:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:38 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3351617697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:26:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:39 smithi033 ceph-mon[27845]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T11:26:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:39 smithi194 ceph-mon[33017]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T11:26:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:39 smithi096 ceph-mon[32753]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.479+0000 7ff580c00700 1 -- 172.21.15.33:0/2839760300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06b130 msgr2=0x7ff57c06b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.479+0000 7ff580c00700 1 --2- 172.21.15.33:0/2839760300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06b130 0x7ff57c06b510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009990 tx=0x7ff56c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.479+0000 7ff580c00700 1 -- 172.21.15.33:0/2839760300 shutdown_connections 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.479+0000 7ff580c00700 1 --2- 172.21.15.33:0/2839760300 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c10fed0 0x7ff57c1122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.479+0000 7ff580c00700 1 --2- 172.21.15.33:0/2839760300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06b130 0x7ff57c06b510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.479+0000 7ff580c00700 1 --2- 172.21.15.33:0/2839760300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06ba50 0x7ff57c10f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.479+0000 7ff580c00700 1 -- 172.21.15.33:0/2839760300 >> 172.21.15.33:0/2839760300 conn(0x7ff57c0792d0 msgr2=0x7ff57c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.480+0000 7ff580c00700 1 -- 172.21.15.33:0/2839760300 shutdown_connections 2024-04-26T11:26:40.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.480+0000 7ff580c00700 1 -- 172.21.15.33:0/2839760300 wait complete. 2024-04-26T11:26:40.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.481+0000 7ff580c00700 1 Processor -- start 2024-04-26T11:26:40.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.481+0000 7ff580c00700 1 -- start start 2024-04-26T11:26:40.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.481+0000 7ff580c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06b130 0x7ff57c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:40.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.481+0000 7ff580c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06ba50 0x7ff57c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:40.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.481+0000 7ff580c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c10fed0 0x7ff57c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:40.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff580c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c115830 con 0x7ff57c06ba50 2024-04-26T11:26:40.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff580c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c1156b0 con 0x7ff57c10fed0 2024-04-26T11:26:40.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff580c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c1159b0 con 0x7ff57c06b130 2024-04-26T11:26:40.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff579e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06b130 0x7ff57c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff579e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06b130 0x7ff57c19bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:45756/0 (socket says 172.21.15.33:45756) 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff579e00700 1 -- 172.21.15.33:0/319818213 learned_addr learned my addr 172.21.15.33:0/319818213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff571400700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06ba50 0x7ff57c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.482+0000 7ff57a800700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c10fed0 0x7ff57c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff571400700 1 -- 172.21.15.33:0/319818213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06b130 msgr2=0x7ff57c19bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff571400700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06b130 0x7ff57c19bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff571400700 1 -- 172.21.15.33:0/319818213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c10fed0 msgr2=0x7ff57c1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff571400700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c10fed0 0x7ff57c1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff571400700 1 -- 172.21.15.33:0/319818213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c1a0d50 con 0x7ff57c06ba50 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff579e00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06b130 0x7ff57c19bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff571400700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06ba50 0x7ff57c19c280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff56400e9f0 tx=0x7ff56400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff573400700 1 -- 172.21.15.33:0/319818213 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56400c9c0 con 0x7ff57c06ba50 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57c1a1040 con 0x7ff57c06ba50 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff573400700 1 -- 172.21.15.33:0/319818213 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff56400cb60 con 0x7ff57c06ba50 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff573400700 1 -- 172.21.15.33:0/319818213 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564018680 con 0x7ff57c06ba50 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.483+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57c1a88c0 con 0x7ff57c06ba50 2024-04-26T11:26:40.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.485+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57c06a110 con 0x7ff57c06ba50 2024-04-26T11:26:40.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.489+0000 7ff573400700 1 -- 172.21.15.33:0/319818213 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff564010430 con 0x7ff57c06ba50 2024-04-26T11:26:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.490+0000 7ff573400700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff5600a3c40 0x7ff5600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.490+0000 7ff573400700 1 -- 172.21.15.33:0/319818213 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff5640d07c0 con 0x7ff57c06ba50 2024-04-26T11:26:40.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.491+0000 7ff579e00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff5600a3c40 0x7ff5600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:40.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.491+0000 7ff579e00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff5600a3c40 0x7ff5600a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff56c006010 tx=0x7ff56c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:40.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.494+0000 7ff573400700 1 -- 172.21.15.33:0/319818213 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff56409d250 con 0x7ff57c06ba50 2024-04-26T11:26:40.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.726+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff57c02e490 con 0x7ff57c06ba50 2024-04-26T11:26:40.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.730+0000 7ff573400700 1 -- 172.21.15.33:0/319818213 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22168 (secure 0 0 0) 0x7ff56409cb50 con 0x7ff57c06ba50 2024-04-26T11:26:40.732 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:26:40.732 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":88,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:26:32.073273+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:26:40.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff5600a3c40 msgr2=0x7ff5600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:40.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff5600a3c40 0x7ff5600a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff56c006010 tx=0x7ff56c01a040 comp rx=0 tx=0).stop 2024-04-26T11:26:40.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06ba50 msgr2=0x7ff57c19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:40.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06ba50 0x7ff57c19c280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff56400e9f0 tx=0x7ff56400ed00 comp rx=0 tx=0).stop 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 shutdown_connections 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff5600a3c40 0x7ff5600a6100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c10fed0 0x7ff57c1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff57c06b130 0x7ff57c19bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 --2- 172.21.15.33:0/319818213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff57c06ba50 0x7ff57c19c280 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.733+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 >> 172.21.15.33:0/319818213 conn(0x7ff57c0792d0 msgr2=0x7ff57c111380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.734+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 shutdown_connections 2024-04-26T11:26:40.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:40.734+0000 7ff580c00700 1 -- 172.21.15.33:0/319818213 wait complete. 2024-04-26T11:26:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:41 smithi194 ceph-mon[33017]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T11:26:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:41 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/319818213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:26:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:41 smithi033 ceph-mon[27845]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T11:26:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:41 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/319818213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:26:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:41 smithi096 ceph-mon[32753]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T11:26:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:41 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/319818213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:26:41.516 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-04-26T11:26:41.984 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:43 smithi033 ceph-mon[27845]: pgmap v330: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T11:26:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:43 smithi194 ceph-mon[33017]: pgmap v330: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T11:26:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:43 smithi096 ceph-mon[32753]: pgmap v330: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T11:26:44.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 -- 172.21.15.33:0/2800621545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e8109a50 msgr2=0x7fb8e8109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:44.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 --2- 172.21.15.33:0/2800621545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e8109a50 0x7fb8e8109ed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009990 tx=0x7fb8d4009ca0 comp rx=0 tx=0).stop 2024-04-26T11:26:44.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 -- 172.21.15.33:0/2800621545 shutdown_connections 2024-04-26T11:26:44.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 --2- 172.21.15.33:0/2800621545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e810a410 0x7fb8e8116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:44.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 --2- 172.21.15.33:0/2800621545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e810fa50 0x7fb8e810fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:44.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 --2- 172.21.15.33:0/2800621545 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e8109a50 0x7fb8e8109ed0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:44.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 -- 172.21.15.33:0/2800621545 >> 172.21.15.33:0/2800621545 conn(0x7fb8e8076bf0 msgr2=0x7fb8e8108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:44.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 -- 172.21.15.33:0/2800621545 shutdown_connections 2024-04-26T11:26:44.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.067+0000 7fb8ed800700 1 -- 172.21.15.33:0/2800621545 wait complete. 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 Processor -- start 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 -- start start 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e8109a50 0x7fb8e81a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e810a410 0x7fb8e81a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e810fa50 0x7fb8e81a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e8119ca0 con 0x7fb8e810fa50 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e8119b20 con 0x7fb8e810a410 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.069+0000 7fb8ed800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e8119e20 con 0x7fb8e8109a50 2024-04-26T11:26:44.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e8109a50 0x7fb8e81a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e7400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e810fa50 0x7fb8e81a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e8109a50 0x7fb8e81a0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:45780/0 (socket says 172.21.15.33:45780) 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 -- 172.21.15.33:0/1265427906 learned_addr learned my addr 172.21.15.33:0/1265427906 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6000700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e810a410 0x7fb8e81a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 -- 172.21.15.33:0/1265427906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e810a410 msgr2=0x7fb8e81a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e810a410 0x7fb8e81a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 -- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e810fa50 msgr2=0x7fb8e81a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e810fa50 0x7fb8e81a49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:44.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 -- 172.21.15.33:0/1265427906 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e81a5130 con 0x7fb8e8109a50 2024-04-26T11:26:44.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.070+0000 7fb8e6a00700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e8109a50 0x7fb8e81a0170 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800e9c0 tx=0x7fb8d800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:44.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.071+0000 7fb8df400700 1 -- 172.21.15.33:0/1265427906 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d800cbd0 con 0x7fb8e8109a50 2024-04-26T11:26:44.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.071+0000 7fb8df400700 1 -- 172.21.15.33:0/1265427906 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d800ee50 con 0x7fb8e8109a50 2024-04-26T11:26:44.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.071+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e81a5420 con 0x7fb8e8109a50 2024-04-26T11:26:44.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.071+0000 7fb8df400700 1 -- 172.21.15.33:0/1265427906 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8005550 con 0x7fb8e8109a50 2024-04-26T11:26:44.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.071+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e81acca0 con 0x7fb8e8109a50 2024-04-26T11:26:44.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.072+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e806a110 con 0x7fb8e8109a50 2024-04-26T11:26:44.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.074+0000 7fb8df400700 1 -- 172.21.15.33:0/1265427906 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb8d8004750 con 0x7fb8e8109a50 2024-04-26T11:26:44.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.076+0000 7fb8df400700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb8cc0a3c40 0x7fb8cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:44.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.076+0000 7fb8df400700 1 -- 172.21.15.33:0/1265427906 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fb8d80d0510 con 0x7fb8e8109a50 2024-04-26T11:26:44.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.076+0000 7fb8e6000700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb8cc0a3c40 0x7fb8cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:44.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.077+0000 7fb8e6000700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb8cc0a3c40 0x7fb8cc0a6100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8078b00 tx=0x7fb8d400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:44.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.080+0000 7fb8df400700 1 -- 172.21.15.33:0/1265427906 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8d809cfa0 con 0x7fb8e8109a50 2024-04-26T11:26:44.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:44.362+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb8e802e490 con 0x7fb8e8109a50 2024-04-26T11:26:45.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.062+0000 7fb8df400700 1 -- 172.21.15.33:0/1265427906 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fb8d809c8a0 con 0x7fb8e8109a50 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.065+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb8cc0a3c40 msgr2=0x7fb8cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.065+0000 7fb8ed800700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb8cc0a3c40 0x7fb8cc0a6100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8078b00 tx=0x7fb8d400b260 comp rx=0 tx=0).stop 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.065+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e8109a50 msgr2=0x7fb8e81a0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.065+0000 7fb8ed800700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e8109a50 0x7fb8e81a0170 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800e9c0 tx=0x7fb8d800ecd0 comp rx=0 tx=0).stop 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 shutdown_connections 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb8cc0a3c40 0x7fb8cc0a6100 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb8e810fa50 0x7fb8e81a49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8e8109a50 0x7fb8e81a0170 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 --2- 172.21.15.33:0/1265427906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e810a410 0x7fb8e81a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 >> 172.21.15.33:0/1265427906 conn(0x7fb8e8076bf0 msgr2=0x7fb8e80771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 shutdown_connections 2024-04-26T11:26:45.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:45.066+0000 7fb8ed800700 1 -- 172.21.15.33:0/1265427906 wait complete. 2024-04-26T11:26:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:45 smithi194 ceph-mon[33017]: pgmap v331: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T11:26:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:45 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1265427906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T11:26:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:45 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T11:26:45.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:45 smithi033 ceph-mon[27845]: pgmap v331: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T11:26:45.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:45 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1265427906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T11:26:45.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:45 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T11:26:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:45 smithi096 ceph-mon[32753]: pgmap v331: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T11:26:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:45 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1265427906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T11:26:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:45 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T11:26:45.837 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-26T11:26:45.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-04-26T11:26:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:46 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T11:26:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:46 smithi194 ceph-mon[33017]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:46 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T11:26:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:46 smithi033 ceph-mon[27845]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:46 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T11:26:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:46 smithi096 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:47 smithi194 ceph-mon[33017]: pgmap v332: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-26T11:26:47.374 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:47 smithi033 ceph-mon[27845]: pgmap v332: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-26T11:26:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:47 smithi096 ceph-mon[32753]: pgmap v332: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-26T11:26:48.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 -- 172.21.15.33:0/2042467079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79681035b0 msgr2=0x7f7968103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:48.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 --2- 172.21.15.33:0/2042467079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79681035b0 0x7f7968103990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7958009fc0 tx=0x7f795800b330 comp rx=0 tx=0).stop 2024-04-26T11:26:48.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 -- 172.21.15.33:0/2042467079 shutdown_connections 2024-04-26T11:26:48.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 --2- 172.21.15.33:0/2042467079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7968108690 0x7f7968110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:48.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 --2- 172.21.15.33:0/2042467079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79681035b0 0x7f7968103990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:48.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 --2- 172.21.15.33:0/2042467079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7968103f60 0x7f7968107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:48.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 -- 172.21.15.33:0/2042467079 >> 172.21.15.33:0/2042467079 conn(0x7f79680feea0 msgr2=0x7f79681012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:48.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 -- 172.21.15.33:0/2042467079 shutdown_connections 2024-04-26T11:26:48.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.522+0000 7f7967200700 1 -- 172.21.15.33:0/2042467079 wait complete. 2024-04-26T11:26:48.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.524+0000 7f7967200700 1 Processor -- start 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.524+0000 7f7967200700 1 -- start start 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.524+0000 7f7967200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79681035b0 0x7f79681a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7967200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7968103f60 0x7f79681a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7967200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7968108690 0x7f796819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7967200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7968113550 con 0x7f7968103f60 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7967200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79681133d0 con 0x7f79681035b0 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7967200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f79681136d0 con 0x7f7968108690 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79681035b0 0x7f79681a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79681035b0 0x7f79681a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36120/0 (socket says 172.21.15.33:36120) 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 -- 172.21.15.33:0/1557766316 learned_addr learned my addr 172.21.15.33:0/1557766316 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965400700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7968103f60 0x7f79681a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7966800700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7968108690 0x7f796819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:48.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 -- 172.21.15.33:0/1557766316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7968108690 msgr2=0x7f796819baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:48.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7968108690 0x7f796819baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:48.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 -- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7968103f60 msgr2=0x7f79681a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:48.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7968103f60 0x7f79681a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:48.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7965e00700 1 -- 172.21.15.33:0/1557766316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796819c250 con 0x7f79681035b0 2024-04-26T11:26:48.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.525+0000 7f7966800700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7968108690 0x7f796819baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:26:48.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.526+0000 7f7965e00700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79681035b0 0x7f79681a18e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f795800ccb0 tx=0x7f795800cd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:48.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.526+0000 7f795ea00700 1 -- 172.21.15.33:0/1557766316 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958021070 con 0x7f79681035b0 2024-04-26T11:26:48.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.526+0000 7f795ea00700 1 -- 172.21.15.33:0/1557766316 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7958007b80 con 0x7f79681035b0 2024-04-26T11:26:48.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.526+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796819c4e0 con 0x7f79681035b0 2024-04-26T11:26:48.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.526+0000 7f795ea00700 1 -- 172.21.15.33:0/1557766316 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580052a0 con 0x7f79681035b0 2024-04-26T11:26:48.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.526+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79681a87b0 con 0x7f79681035b0 2024-04-26T11:26:48.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.527+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796806a110 con 0x7f79681035b0 2024-04-26T11:26:48.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.531+0000 7f795ea00700 1 -- 172.21.15.33:0/1557766316 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7958024070 con 0x7f79681035b0 2024-04-26T11:26:48.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.533+0000 7f795ea00700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f794c0a3cc0 0x7f794c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:48.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.533+0000 7f795ea00700 1 -- 172.21.15.33:0/1557766316 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f795800f020 con 0x7f79681035b0 2024-04-26T11:26:48.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.533+0000 7f7965400700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f794c0a3cc0 0x7f794c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:48.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.534+0000 7f7965400700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f794c0a3cc0 0x7f794c0a6180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7954009750 tx=0x7f7954009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:48.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.534+0000 7f795ea00700 1 -- 172.21.15.33:0/1557766316 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f795802b020 con 0x7f79681035b0 2024-04-26T11:26:48.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:48.819+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f79681a8980 con 0x7f79681035b0 2024-04-26T11:26:49.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.079+0000 7f795ea00700 1 -- 172.21.15.33:0/1557766316 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f7958007d30 con 0x7f79681035b0 2024-04-26T11:26:49.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.085+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f794c0a3cc0 msgr2=0x7f794c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:49.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.085+0000 7f7967200700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f794c0a3cc0 0x7f794c0a6180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7954009750 tx=0x7f7954009310 comp rx=0 tx=0).stop 2024-04-26T11:26:49.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.085+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79681035b0 msgr2=0x7f79681a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:49.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.085+0000 7f7967200700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79681035b0 0x7f79681a18e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f795800ccb0 tx=0x7f795800cd90 comp rx=0 tx=0).stop 2024-04-26T11:26:49.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.086+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 shutdown_connections 2024-04-26T11:26:49.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.086+0000 7f7967200700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f794c0a3cc0 0x7f794c0a6180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:49.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.086+0000 7f7967200700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7968108690 0x7f796819baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:49.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.086+0000 7f7967200700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79681035b0 0x7f79681a18e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:49.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.086+0000 7f7967200700 1 --2- 172.21.15.33:0/1557766316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7968103f60 0x7f79681a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:49.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.086+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 >> 172.21.15.33:0/1557766316 conn(0x7f79680feea0 msgr2=0x7f79681001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:49.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.087+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 shutdown_connections 2024-04-26T11:26:49.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:49.087+0000 7f7967200700 1 -- 172.21.15.33:0/1557766316 wait complete. 2024-04-26T11:26:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:49 smithi194 ceph-mon[33017]: pgmap v333: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T11:26:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:49 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1557766316' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T11:26:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:49 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T11:26:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:49 smithi033 ceph-mon[27845]: pgmap v333: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T11:26:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:49 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1557766316' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T11:26:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:49 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T11:26:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:49 smithi096 ceph-mon[32753]: pgmap v333: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T11:26:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:49 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1557766316' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T11:26:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:49 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T11:26:49.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-04-26T11:26:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:50 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T11:26:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:50 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T11:26:50.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:50 smithi194 ceph-mon[33017]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T11:26:50.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:50 smithi194 ceph-mon[33017]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:50.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:50 smithi194 ceph-mon[33017]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-26T11:26:50.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:50 smithi194 ceph-mon[33017]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:26:50.358 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:50 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T11:26:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:50 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T11:26:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:50 smithi033 ceph-mon[27845]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T11:26:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:50 smithi033 ceph-mon[27845]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:50 smithi033 ceph-mon[27845]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-26T11:26:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:50 smithi033 ceph-mon[27845]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:26:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:50 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T11:26:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:50 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T11:26:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:50 smithi096 ceph-mon[32753]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T11:26:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:50 smithi096 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T11:26:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:50 smithi096 ceph-mon[32753]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-26T11:26:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:50 smithi096 ceph-mon[32753]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:26:51.349 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: pgmap v334: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-26T11:26:51.349 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] up:active 2024-04-26T11:26:51.349 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T11:26:51.349 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T11:26:51.350 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:26:51.350 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-26T11:26:51.350 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-26T11:26:51.350 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:51 smithi033 ceph-mon[27845]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: pgmap v334: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] up:active 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-26T11:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:51 smithi194 ceph-mon[33017]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T11:26:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: pgmap v334: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-26T11:26:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] up:active 2024-04-26T11:26:51.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T11:26:51.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T11:26:51.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:26:51.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-26T11:26:51.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-26T11:26:51.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-mon[32753]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T11:26:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:52 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:26:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:52 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-26T11:26:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:52 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:26:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:52 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-26T11:26:52.183 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:26:51.894+0000 7f31c45fd700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:26:52.183 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:26:51 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:26:51.894+0000 7f31c45fd700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T11:26:52.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 -- 172.21.15.33:0/2728991110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 msgr2=0x7ff358109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 --2- 172.21.15.33:0/2728991110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 0x7ff358109880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff348009990 tx=0x7ff348009ca0 comp rx=0 tx=0).stop 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 -- 172.21.15.33:0/2728991110 shutdown_connections 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 --2- 172.21.15.33:0/2728991110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 0x7ff3581102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 --2- 172.21.15.33:0/2728991110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 0x7ff358109880 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 --2- 172.21.15.33:0/2728991110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 0x7ff35806bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 -- 172.21.15.33:0/2728991110 >> 172.21.15.33:0/2728991110 conn(0x7ff358079290 msgr2=0x7ff3580796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 -- 172.21.15.33:0/2728991110 shutdown_connections 2024-04-26T11:26:52.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.520+0000 7ff35d200700 1 -- 172.21.15.33:0/2728991110 wait complete. 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 Processor -- start 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 -- start start 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 0x7ff35819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 0x7ff35819c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 0x7ff3581a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff358113600 con 0x7ff35806bb00 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff358113480 con 0x7ff3581094a0 2024-04-26T11:26:52.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff35d200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff358113780 con 0x7ff35806c4a0 2024-04-26T11:26:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.522+0000 7ff355e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 0x7ff35819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 0x7ff35819c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 0x7ff35819bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38928/0 (socket says 172.21.15.33:38928) 2024-04-26T11:26:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 0x7ff35819c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:51240/0 (socket says 172.21.15.33:51240) 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355e00700 1 -- 172.21.15.33:0/2969708201 learned_addr learned my addr 172.21.15.33:0/2969708201 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff356800700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 0x7ff3581a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355e00700 1 -- 172.21.15.33:0/2969708201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 msgr2=0x7ff35819c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355e00700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 0x7ff35819c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355e00700 1 -- 172.21.15.33:0/2969708201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 msgr2=0x7ff3581a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355e00700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 0x7ff3581a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355e00700 1 -- 172.21.15.33:0/2969708201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3581a0dc0 con 0x7ff35806bb00 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff355400700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 0x7ff35819c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.523+0000 7ff356800700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 0x7ff3581a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:26:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.524+0000 7ff355e00700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 0x7ff35819bdb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff348000c00 tx=0x7ff3480057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.524+0000 7ff34ea00700 1 -- 172.21.15.33:0/2969708201 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348027070 con 0x7ff35806bb00 2024-04-26T11:26:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.524+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3581a1050 con 0x7ff35806bb00 2024-04-26T11:26:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.524+0000 7ff34ea00700 1 -- 172.21.15.33:0/2969708201 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff348022470 con 0x7ff35806bb00 2024-04-26T11:26:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.524+0000 7ff34ea00700 1 -- 172.21.15.33:0/2969708201 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348017650 con 0x7ff35806bb00 2024-04-26T11:26:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.524+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3581a8930 con 0x7ff35806bb00 2024-04-26T11:26:52.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.526+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff358104b50 con 0x7ff35806bb00 2024-04-26T11:26:52.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.528+0000 7ff34ea00700 1 -- 172.21.15.33:0/2969708201 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff34802a070 con 0x7ff35806bb00 2024-04-26T11:26:52.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.529+0000 7ff34ea00700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff33c0a3c40 0x7ff33c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:52.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.530+0000 7ff34ea00700 1 -- 172.21.15.33:0/2969708201 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff3480d10c0 con 0x7ff35806bb00 2024-04-26T11:26:52.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.530+0000 7ff355400700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff33c0a3c40 0x7ff33c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:52.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.531+0000 7ff355400700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff33c0a3c40 0x7ff33c0a6100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff3580799b0 tx=0x7ff344009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:52.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.533+0000 7ff34ea00700 1 -- 172.21.15.33:0/2969708201 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff348013070 con 0x7ff35806bb00 2024-04-26T11:26:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:52 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:26:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:52 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-26T11:26:52.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:52.814+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7ff35819d110 con 0x7ff35806bb00 2024-04-26T11:26:53.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.123+0000 7ff34ea00700 1 -- 172.21.15.33:0/2969708201 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7ff34809dd00 con 0x7ff35806bb00 2024-04-26T11:26:53.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff33c0a3c40 msgr2=0x7ff33c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:53.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff33c0a3c40 0x7ff33c0a6100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff3580799b0 tx=0x7ff344009260 comp rx=0 tx=0).stop 2024-04-26T11:26:53.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 msgr2=0x7ff35819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:53.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 0x7ff35819bdb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff348000c00 tx=0x7ff3480057c0 comp rx=0 tx=0).stop 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 shutdown_connections 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff33c0a3c40 0x7ff33c0a6100 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3581094a0 0x7ff3581a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff35806bb00 0x7ff35819bdb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 --2- 172.21.15.33:0/2969708201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff35806c4a0 0x7ff35819c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.129+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 >> 172.21.15.33:0/2969708201 conn(0x7ff358079290 msgr2=0x7ff35810e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.130+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 shutdown_connections 2024-04-26T11:26:53.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:53.130+0000 7ff35d200700 1 -- 172.21.15.33:0/2969708201 wait complete. 2024-04-26T11:26:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:53 smithi033 ceph-mon[27845]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-26T11:26:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:53 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2969708201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T11:26:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:53 smithi096 ceph-mon[32753]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-26T11:26:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:53 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2969708201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T11:26:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:53 smithi194 ceph-mon[33017]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-26T11:26:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:53 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2969708201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T11:26:53.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-04-26T11:26:54.304 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:54 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2969708201' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T11:26:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:54 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-26T11:26:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:54 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:26:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:54 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2969708201' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T11:26:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:54 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-26T11:26:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:54 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:26:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:54 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2969708201' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T11:26:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:54 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-26T11:26:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:54 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:26:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:55 smithi033 ceph-mon[27845]: pgmap v336: 97 pgs: 97 active+clean; 580 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T11:26:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:55 smithi096 ceph-mon[32753]: pgmap v336: 97 pgs: 97 active+clean; 580 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T11:26:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:55 smithi194 ceph-mon[33017]: pgmap v336: 97 pgs: 97 active+clean; 580 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T11:26:56.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 -- 172.21.15.33:0/2205188420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924102e60 msgr2=0x7f8924106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:56.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 --2- 172.21.15.33:0/2205188420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924102e60 0x7f8924106eb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f89100099c0 tx=0x7f8910009cd0 comp rx=0 tx=0).stop 2024-04-26T11:26:56.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 -- 172.21.15.33:0/2205188420 shutdown_connections 2024-04-26T11:26:56.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 --2- 172.21.15.33:0/2205188420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8924107590 0x7f892410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:56.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 --2- 172.21.15.33:0/2205188420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 0x7f8924102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:56.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 --2- 172.21.15.33:0/2205188420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924102e60 0x7f8924106eb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:56.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 -- 172.21.15.33:0/2205188420 >> 172.21.15.33:0/2205188420 conn(0x7f89240fdd40 msgr2=0x7f8924100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:56.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 -- 172.21.15.33:0/2205188420 shutdown_connections 2024-04-26T11:26:56.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.435+0000 7f892ac00700 1 -- 172.21.15.33:0/2205188420 wait complete. 2024-04-26T11:26:56.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.436+0000 7f892ac00700 1 Processor -- start 2024-04-26T11:26:56.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.436+0000 7f892ac00700 1 -- start start 2024-04-26T11:26:56.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f892ac00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 0x7f89241a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:56.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f892ac00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8924102e60 0x7f89241a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:56.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f892ac00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924107590 0x7f892419ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:56.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f892ac00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8924112420 con 0x7f8924107590 2024-04-26T11:26:56.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f892ac00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f89241122a0 con 0x7f89241024b0 2024-04-26T11:26:56.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f892ac00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f89241125a0 con 0x7f8924102e60 2024-04-26T11:26:56.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f8929800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 0x7f89241a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:56.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f8929800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 0x7f89241a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:38494/0 (socket says 172.21.15.33:38494) 2024-04-26T11:26:56.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f8929800700 1 -- 172.21.15.33:0/2182663176 learned_addr learned my addr 172.21.15.33:0/2182663176 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f892a200700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924107590 0x7f892419ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f8928e00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8924102e60 0x7f89241a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.437+0000 7f8929800700 1 -- 172.21.15.33:0/2182663176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8924102e60 msgr2=0x7f89241a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8929800700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8924102e60 0x7f89241a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8929800700 1 -- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924107590 msgr2=0x7f892419ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8929800700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924107590 0x7f892419ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8929800700 1 -- 172.21.15.33:0/2182663176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892419c180 con 0x7f89241024b0 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8929800700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 0x7f89241a1860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f891800ef10 tx=0x7f891800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8922000700 1 -- 172.21.15.33:0/2182663176 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891800c9c0 con 0x7f89241024b0 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8922000700 1 -- 172.21.15.33:0/2182663176 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8918019070 con 0x7f89241024b0 2024-04-26T11:26:56.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892419c410 con 0x7f89241024b0 2024-04-26T11:26:56.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f8922000700 1 -- 172.21.15.33:0/2182663176 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8918014480 con 0x7f89241024b0 2024-04-26T11:26:56.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.438+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89241a8730 con 0x7f89241024b0 2024-04-26T11:26:56.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.440+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f892406a110 con 0x7f89241024b0 2024-04-26T11:26:56.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.441+0000 7f8922000700 1 -- 172.21.15.33:0/2182663176 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8918012050 con 0x7f89241024b0 2024-04-26T11:26:56.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.443+0000 7f8922000700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f890c0a3c40 0x7f890c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:26:56.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.443+0000 7f8922000700 1 -- 172.21.15.33:0/2182663176 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f89180d0260 con 0x7f89241024b0 2024-04-26T11:26:56.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.443+0000 7f8928e00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f890c0a3c40 0x7f890c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:26:56.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.444+0000 7f8928e00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f890c0a3c40 0x7f890c0a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8910000c00 tx=0x7f8910005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:26:56.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.447+0000 7f8922000700 1 -- 172.21.15.33:0/2182663176 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8918010040 con 0x7f89241024b0 2024-04-26T11:26:56.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:56.730+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f892402e490 con 0x7f89241024b0 2024-04-26T11:26:57.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.159+0000 7f8922000700 1 -- 172.21.15.33:0/2182663176 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f891809ccf0 con 0x7f89241024b0 2024-04-26T11:26:57.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f890c0a3c40 msgr2=0x7f890c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:57.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f890c0a3c40 0x7f890c0a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8910000c00 tx=0x7f8910005fb0 comp rx=0 tx=0).stop 2024-04-26T11:26:57.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 msgr2=0x7f89241a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:26:57.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 0x7f89241a1860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f891800ef10 tx=0x7f891800c3a0 comp rx=0 tx=0).stop 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 shutdown_connections 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f890c0a3c40 0x7f890c0a6100 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8924107590 0x7f892419ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89241024b0 0x7f89241a1860 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 --2- 172.21.15.33:0/2182663176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8924102e60 0x7f89241a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.163+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 >> 172.21.15.33:0/2182663176 conn(0x7f89240fdd40 msgr2=0x7f892410dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.164+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 shutdown_connections 2024-04-26T11:26:57.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:26:57.164+0000 7f892ac00700 1 -- 172.21.15.33:0/2182663176 wait complete. 2024-04-26T11:26:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:57 smithi033 ceph-mon[27845]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T11:26:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:57 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2182663176' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T11:26:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:57 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T11:26:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:57 smithi096 ceph-mon[32753]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T11:26:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:57 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2182663176' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T11:26:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:57 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T11:26:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:57 smithi194 ceph-mon[33017]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T11:26:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:57 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2182663176' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T11:26:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:57 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T11:26:57.967 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-26T11:26:57.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-04-26T11:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:58 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T11:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:58 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:58 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T11:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:58 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:26:58.458 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:26:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:58 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T11:26:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:58 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:26:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:26:59 smithi033 ceph-mon[27845]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T11:26:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:26:59 smithi096 ceph-mon[32753]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T11:26:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:26:59 smithi194 ceph-mon[33017]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T11:27:00.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 -- 172.21.15.33:0/1905326309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 msgr2=0x7f9e5006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:00.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 --2- 172.21.15.33:0/1905326309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e5006bf50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e380098f0 tx=0x7f9e38009c00 comp rx=0 tx=0).stop 2024-04-26T11:27:00.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 -- 172.21.15.33:0/1905326309 shutdown_connections 2024-04-26T11:27:00.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 --2- 172.21.15.33:0/1905326309 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e5006c490 0x7f9e50110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 --2- 172.21.15.33:0/1905326309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e501094c0 0x7f9e501098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 --2- 172.21.15.33:0/1905326309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e5006bf50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 -- 172.21.15.33:0/1905326309 >> 172.21.15.33:0/1905326309 conn(0x7f9e500792d0 msgr2=0x7f9e500796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:00.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 -- 172.21.15.33:0/1905326309 shutdown_connections 2024-04-26T11:27:00.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.511+0000 7f9e56c00700 1 -- 172.21.15.33:0/1905326309 wait complete. 2024-04-26T11:27:00.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.512+0000 7f9e56c00700 1 Processor -- start 2024-04-26T11:27:00.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.512+0000 7f9e56c00700 1 -- start start 2024-04-26T11:27:00.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e56c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e501027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:00.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e56c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e5006c490 0x7f9e50102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:00.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e56c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e501094c0 0x7f9e50103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:00.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e56c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50113600 con 0x7f9e501094c0 2024-04-26T11:27:00.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e56c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50113480 con 0x7f9e5006baf0 2024-04-26T11:27:00.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e56c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50113780 con 0x7f9e5006c490 2024-04-26T11:27:00.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e4fe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e501027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:00.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e54c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e501094c0 0x7f9e50103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:00.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e4fe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e501027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41968/0 (socket says 172.21.15.33:41968) 2024-04-26T11:27:00.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e4fe00700 1 -- 172.21.15.33:0/2860024028 learned_addr learned my addr 172.21.15.33:0/2860024028 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:00.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.513+0000 7f9e47e00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e5006c490 0x7f9e50102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:00.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4fe00700 1 -- 172.21.15.33:0/2860024028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e5006c490 msgr2=0x7f9e50102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4fe00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e5006c490 0x7f9e50102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4fe00700 1 -- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e501094c0 msgr2=0x7f9e50103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4fe00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e501094c0 0x7f9e50103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4fe00700 1 -- 172.21.15.33:0/2860024028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e501a80a0 con 0x7f9e5006baf0 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4fe00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e501027d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000ec10 tx=0x7f9e4000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4d600700 1 -- 172.21.15.33:0/2860024028 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4000cc90 con 0x7f9e5006baf0 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e4d600700 1 -- 172.21.15.33:0/2860024028 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e4000ce30 con 0x7f9e5006baf0 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.514+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e501a8390 con 0x7f9e5006baf0 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.515+0000 7f9e4d600700 1 -- 172.21.15.33:0/2860024028 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40010430 con 0x7f9e5006baf0 2024-04-26T11:27:00.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.515+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e501a88f0 con 0x7f9e5006baf0 2024-04-26T11:27:00.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.516+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e50104b70 con 0x7f9e5006baf0 2024-04-26T11:27:00.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.518+0000 7f9e4d600700 1 -- 172.21.15.33:0/2860024028 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9e40004000 con 0x7f9e5006baf0 2024-04-26T11:27:00.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.519+0000 7f9e4d600700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9e340a3cc0 0x7f9e340a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:00.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.519+0000 7f9e4d600700 1 -- 172.21.15.33:0/2860024028 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f9e400d04b0 con 0x7f9e5006baf0 2024-04-26T11:27:00.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.519+0000 7f9e47e00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9e340a3cc0 0x7f9e340a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:00.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.520+0000 7f9e47e00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9e340a3cc0 0x7f9e340a6180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e380098c0 tx=0x7f9e38019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:00.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.523+0000 7f9e4d600700 1 -- 172.21.15.33:0/2860024028 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e4009cf40 con 0x7f9e5006baf0 2024-04-26T11:27:00.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.792+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9e50063860 con 0x7f9e340a3cc0 2024-04-26T11:27:00.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.827+0000 7f9e4d600700 1 -- 172.21.15.33:0/2860024028 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9e50063860 con 0x7f9e340a3cc0 2024-04-26T11:27:00.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9e340a3cc0 msgr2=0x7f9e340a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:00.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9e340a3cc0 0x7f9e340a6180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e380098c0 tx=0x7f9e38019040 comp rx=0 tx=0).stop 2024-04-26T11:27:00.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 msgr2=0x7f9e501027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:00.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e501027d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000ec10 tx=0x7f9e4000ef20 comp rx=0 tx=0).stop 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 shutdown_connections 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9e340a3cc0 0x7f9e340a6180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e501094c0 0x7f9e50103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e5006baf0 0x7f9e501027d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 --2- 172.21.15.33:0/2860024028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e5006c490 0x7f9e50102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 >> 172.21.15.33:0/2860024028 conn(0x7f9e500792d0 msgr2=0x7f9e501008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.830+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 shutdown_connections 2024-04-26T11:27:00.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:00.831+0000 7f9e56c00700 1 -- 172.21.15.33:0/2860024028 wait complete. 2024-04-26T11:27:01.175 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:27:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:27:00.801+0000 7f6f67600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:27:01.175 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:27:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:27:00.801+0000 7f6f67600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:27:01.175 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:27:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:27:00.801+0000 7f6f67600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:27:01.175 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:27:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:27:00.801+0000 7f6f67600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:27:01.175 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:27:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:27:00.801+0000 7f6f67600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:27:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:01 smithi033 ceph-mon[27845]: pgmap v339: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T11:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:01 smithi096 ceph-mon[32753]: pgmap v339: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T11:27:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:01 smithi194 ceph-mon[33017]: pgmap v339: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T11:27:01.659 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T11:27:02.142 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:02.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:02 smithi033 ceph-mon[27845]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:02 smithi194 ceph-mon[33017]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:02 smithi096 ceph-mon[32753]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:03.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:03 smithi033 ceph-mon[27845]: pgmap v340: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T11:27:03.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:03 smithi033 ceph-mon[27845]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T11:27:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:03 smithi194 ceph-mon[33017]: pgmap v340: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T11:27:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:03 smithi194 ceph-mon[33017]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T11:27:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:03 smithi096 ceph-mon[32753]: pgmap v340: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T11:27:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:03 smithi096 ceph-mon[32753]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 -- 172.21.15.33:0/3624133894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075650 msgr2=0x7fc4cc075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 --2- 172.21.15.33:0/3624133894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075650 0x7fc4cc075a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009990 tx=0x7fc4bc009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 -- 172.21.15.33:0/3624133894 shutdown_connections 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 --2- 172.21.15.33:0/3624133894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4cc10fe50 0x7fc4cc112240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 --2- 172.21.15.33:0/3624133894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075650 0x7fc4cc075a30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 --2- 172.21.15.33:0/3624133894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4cc075f70 0x7fc4cc10f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 -- 172.21.15.33:0/3624133894 >> 172.21.15.33:0/3624133894 conn(0x7fc4cc07a5d0 msgr2=0x7fc4cc07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:04.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.256+0000 7fc4d3800700 1 -- 172.21.15.33:0/3624133894 shutdown_connections 2024-04-26T11:27:04.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.257+0000 7fc4d3800700 1 -- 172.21.15.33:0/3624133894 wait complete. 2024-04-26T11:27:04.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.258+0000 7fc4d3800700 1 Processor -- start 2024-04-26T11:27:04.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.258+0000 7fc4d3800700 1 -- start start 2024-04-26T11:27:04.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.258+0000 7fc4d3800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4cc075650 0x7fc4cc1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:04.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.258+0000 7fc4d3800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075f70 0x7fc4cc1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:04.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4d3800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4cc10fe50 0x7fc4cc19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4d3800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc115850 con 0x7fc4cc075f70 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4d3800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1156d0 con 0x7fc4cc10fe50 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4d3800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1159d0 con 0x7fc4cc075650 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075f70 0x7fc4cc1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075f70 0x7fc4cc1a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39940/0 (socket says 172.21.15.33:39940) 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 -- 172.21.15.33:0/782914862 learned_addr learned my addr 172.21.15.33:0/782914862 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4d0e00700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4cc075650 0x7fc4cc1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4d1800700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4cc10fe50 0x7fc4cc19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 -- 172.21.15.33:0/782914862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4cc075650 msgr2=0x7fc4cc1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4cc075650 0x7fc4cc1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 -- 172.21.15.33:0/782914862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4cc10fe50 msgr2=0x7fc4cc19bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4cc10fe50 0x7fc4cc19bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.259+0000 7fc4c3e00700 1 -- 172.21.15.33:0/782914862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc19c290 con 0x7fc4cc075f70 2024-04-26T11:27:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.260+0000 7fc4c3e00700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075f70 0x7fc4cc1a1e60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400e970 tx=0x7fc4b400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.260+0000 7fc4ca000700 1 -- 172.21.15.33:0/782914862 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4004220 con 0x7fc4cc075f70 2024-04-26T11:27:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.260+0000 7fc4ca000700 1 -- 172.21.15.33:0/782914862 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b40043c0 con 0x7fc4cc075f70 2024-04-26T11:27:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.260+0000 7fc4ca000700 1 -- 172.21.15.33:0/782914862 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4003ce0 con 0x7fc4cc075f70 2024-04-26T11:27:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.260+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc19c580 con 0x7fc4cc075f70 2024-04-26T11:27:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.260+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc1a87f0 con 0x7fc4cc075f70 2024-04-26T11:27:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.261+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc06a110 con 0x7fc4cc075f70 2024-04-26T11:27:04.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.263+0000 7fc4ca000700 1 -- 172.21.15.33:0/782914862 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc4b400c7e0 con 0x7fc4cc075f70 2024-04-26T11:27:04.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.265+0000 7fc4ca000700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4b00a3cd0 0x7fc4b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:04.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.265+0000 7fc4ca000700 1 -- 172.21.15.33:0/782914862 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fc4b40d06f0 con 0x7fc4cc075f70 2024-04-26T11:27:04.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.265+0000 7fc4d0e00700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4b00a3cd0 0x7fc4b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:04.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.266+0000 7fc4d0e00700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4b00a3cd0 0x7fc4b00a6190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc006010 tx=0x7fc4bc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:04.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.270+0000 7fc4ca000700 1 -- 172.21.15.33:0/782914862 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4b409d020 con 0x7fc4cc075f70 2024-04-26T11:27:04.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.540+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc4cc063860 con 0x7fc4b00a3cd0 2024-04-26T11:27:04.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.581+0000 7fc4ca000700 1 -- 172.21.15.33:0/782914862 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc4cc063860 con 0x7fc4b00a3cd0 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.584+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4b00a3cd0 msgr2=0x7fc4b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.584+0000 7fc4d3800700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4b00a3cd0 0x7fc4b00a6190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc006010 tx=0x7fc4bc00b260 comp rx=0 tx=0).stop 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.584+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075f70 msgr2=0x7fc4cc1a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.584+0000 7fc4d3800700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075f70 0x7fc4cc1a1e60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400e970 tx=0x7fc4b400ec80 comp rx=0 tx=0).stop 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 shutdown_connections 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4b00a3cd0 0x7fc4b00a6190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4cc10fe50 0x7fc4cc19bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4cc075650 0x7fc4cc1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 --2- 172.21.15.33:0/782914862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc4cc075f70 0x7fc4cc1a1e60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:04.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 >> 172.21.15.33:0/782914862 conn(0x7fc4cc07a5d0 msgr2=0x7fc4cc100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:04.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 shutdown_connections 2024-04-26T11:27:04.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:04.585+0000 7fc4d3800700 1 -- 172.21.15.33:0/782914862 wait complete. 2024-04-26T11:27:05.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:05 smithi033 ceph-mon[27845]: pgmap v341: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-26T11:27:05.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:05 smithi033 ceph-mon[27845]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:05.425 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T11:27:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:05 smithi194 ceph-mon[33017]: pgmap v341: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-26T11:27:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:05 smithi194 ceph-mon[33017]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:05 smithi096 ceph-mon[32753]: pgmap v341: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-26T11:27:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:05 smithi096 ceph-mon[32753]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:07.584 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:07 smithi033 ceph-mon[27845]: pgmap v342: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-04-26T11:27:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:07 smithi194 ceph-mon[33017]: pgmap v342: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-04-26T11:27:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:07 smithi096 ceph-mon[32753]: pgmap v342: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.135+0000 7f79dc800700 1 -- 172.21.15.33:0/2167517499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 msgr2=0x7f79d4109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.135+0000 7f79dc800700 1 --2- 172.21.15.33:0/2167517499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 0x7f79d4109860 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009990 tx=0x7f79c8009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.135+0000 7f79dc800700 1 -- 172.21.15.33:0/2167517499 shutdown_connections 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.135+0000 7f79dc800700 1 --2- 172.21.15.33:0/2167517499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79d4103ea0 0x7f79d41102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.135+0000 7f79dc800700 1 --2- 172.21.15.33:0/2167517499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 0x7f79d4109860 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.135+0000 7f79dc800700 1 --2- 172.21.15.33:0/2167517499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 0x7f79d4103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.135+0000 7f79dc800700 1 -- 172.21.15.33:0/2167517499 >> 172.21.15.33:0/2167517499 conn(0x7f79d407a5e0 msgr2=0x7f79d407a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:08.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.136+0000 7f79dc800700 1 -- 172.21.15.33:0/2167517499 shutdown_connections 2024-04-26T11:27:08.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.136+0000 7f79dc800700 1 -- 172.21.15.33:0/2167517499 wait complete. 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 Processor -- start 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 -- start start 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 0x7f79d419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79d4103ea0 0x7f79d419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 0x7f79d4199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4113600 con 0x7f79d41034e0 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4113480 con 0x7f79d4109480 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.137+0000 7f79dc800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4113780 con 0x7f79d4103ea0 2024-04-26T11:27:08.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 0x7f79d419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 0x7f79d419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39966/0 (socket says 172.21.15.33:39966) 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9e00700 1 -- 172.21.15.33:0/1897536023 learned_addr learned my addr 172.21.15.33:0/1897536023 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9400700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79d4103ea0 0x7f79d419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79da800700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 0x7f79d4199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9400700 1 -- 172.21.15.33:0/1897536023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 msgr2=0x7f79d4199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9400700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 0x7f79d4199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9400700 1 -- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 msgr2=0x7f79d419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9400700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 0x7f79d419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9400700 1 -- 172.21.15.33:0/1897536023 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d419a080 con 0x7f79d4103ea0 2024-04-26T11:27:08.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.138+0000 7f79d9e00700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 0x7f79d419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:27:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.139+0000 7f79d9400700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79d4103ea0 0x7f79d419fca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e9f0 tx=0x7f79c400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.139+0000 7f79cea00700 1 -- 172.21.15.33:0/1897536023 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c400c9c0 con 0x7f79d4103ea0 2024-04-26T11:27:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.139+0000 7f79cea00700 1 -- 172.21.15.33:0/1897536023 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c400cb60 con 0x7f79d4103ea0 2024-04-26T11:27:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.139+0000 7f79cea00700 1 -- 172.21.15.33:0/1897536023 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4018680 con 0x7f79d4103ea0 2024-04-26T11:27:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.139+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d419a370 con 0x7f79d4103ea0 2024-04-26T11:27:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.139+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d41a6630 con 0x7f79d4103ea0 2024-04-26T11:27:08.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.141+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d410b980 con 0x7f79d4103ea0 2024-04-26T11:27:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.143+0000 7f79cea00700 1 -- 172.21.15.33:0/1897536023 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f79c4010430 con 0x7f79d4103ea0 2024-04-26T11:27:08.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.144+0000 7f79cea00700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f79bc0a3d50 0x7f79bc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:08.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.144+0000 7f79cea00700 1 -- 172.21.15.33:0/1897536023 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f79c40d05c0 con 0x7f79d4103ea0 2024-04-26T11:27:08.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.144+0000 7f79d9e00700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f79bc0a3d50 0x7f79bc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:08.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.145+0000 7f79d9e00700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f79bc0a3d50 0x7f79bc0a6210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098e0 tx=0x7f79c800b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:08.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.148+0000 7f79cea00700 1 -- 172.21.15.33:0/1897536023 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79c409cef0 con 0x7f79d4103ea0 2024-04-26T11:27:08.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.418+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f79d4063860 con 0x7f79bc0a3d50 2024-04-26T11:27:08.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.428+0000 7f79cea00700 1 -- 172.21.15.33:0/1897536023 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f79d4063860 con 0x7f79bc0a3d50 2024-04-26T11:27:08.429 INFO:teuthology.orchestra.run.smithi033.stdout:/volumes/qa/sv_0/5be4735c-7bab-447a-a7c0-3005b9c27d35 2024-04-26T11:27:08.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.431+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f79bc0a3d50 msgr2=0x7f79bc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:08.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.431+0000 7f79dc800700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f79bc0a3d50 0x7f79bc0a6210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098e0 tx=0x7f79c800b2a0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79d4103ea0 msgr2=0x7f79d419fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:08.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79d4103ea0 0x7f79d419fca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e9f0 tx=0x7f79c400ed00 comp rx=0 tx=0).stop 2024-04-26T11:27:08.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 shutdown_connections 2024-04-26T11:27:08.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f79bc0a3d50 0x7f79bc0a6210 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79d4109480 0x7f79d4199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79d41034e0 0x7f79d419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 --2- 172.21.15.33:0/1897536023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79d4103ea0 0x7f79d419fca0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:08.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 >> 172.21.15.33:0/1897536023 conn(0x7f79d407a5e0 msgr2=0x7f79d41015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:08.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 shutdown_connections 2024-04-26T11:27:08.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:08.432+0000 7f79dc800700 1 -- 172.21.15.33:0/1897536023 wait complete. 2024-04-26T11:27:09.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T11:27:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:09 smithi033 ceph-mon[27845]: pgmap v343: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T11:27:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:09 smithi033 ceph-mon[27845]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:09.598 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:09 smithi194 ceph-mon[33017]: pgmap v343: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T11:27:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:09 smithi194 ceph-mon[33017]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:09 smithi096 ceph-mon[32753]: pgmap v343: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T11:27:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:09 smithi096 ceph-mon[32753]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:11.422 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:11 smithi033 ceph-mon[27845]: pgmap v344: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-26T11:27:11.485 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:11 smithi096 ceph-mon[32753]: pgmap v344: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-26T11:27:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:11 smithi194 ceph-mon[33017]: pgmap v344: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-26T11:27:11.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 -- 172.21.15.33:0/2018606863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006bad0 msgr2=0x7f9c701102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:11.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 --2- 172.21.15.33:0/2018606863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006bad0 0x7f9c701102e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800b0a0 tx=0x7f9c6800b3b0 comp rx=0 tx=0).stop 2024-04-26T11:27:11.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 -- 172.21.15.33:0/2018606863 shutdown_connections 2024-04-26T11:27:11.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 --2- 172.21.15.33:0/2018606863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006bad0 0x7f9c701102e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:11.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 --2- 172.21.15.33:0/2018606863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c701094a0 0x7f9c70109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:11.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 --2- 172.21.15.33:0/2018606863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c7006b130 0x7f9c7006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:11.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 -- 172.21.15.33:0/2018606863 >> 172.21.15.33:0/2018606863 conn(0x7f9c700792d0 msgr2=0x7f9c700796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:11.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 -- 172.21.15.33:0/2018606863 shutdown_connections 2024-04-26T11:27:11.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.722+0000 7f9c75000700 1 -- 172.21.15.33:0/2018606863 wait complete. 2024-04-26T11:27:11.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.723+0000 7f9c75000700 1 Processor -- start 2024-04-26T11:27:11.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c75000700 1 -- start start 2024-04-26T11:27:11.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c75000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006b130 0x7f9c7019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:11.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c75000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006bad0 0x7f9c7019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:11.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c75000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c701094a0 0x7f9c701998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:11.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c75000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9c701135f0 con 0x7f9c7006b130 2024-04-26T11:27:11.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c75000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9c70113470 con 0x7f9c7006bad0 2024-04-26T11:27:11.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c75000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9c70113770 con 0x7f9c701094a0 2024-04-26T11:27:11.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c6de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006b130 0x7f9c7019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:11.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c6e800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c701094a0 0x7f9c701998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:11.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c6de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006b130 0x7f9c7019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47464/0 (socket says 172.21.15.33:47464) 2024-04-26T11:27:11.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.724+0000 7f9c6de00700 1 -- 172.21.15.33:0/1172884868 learned_addr learned my addr 172.21.15.33:0/1172884868 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:11.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.725+0000 7f9c65400700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006bad0 0x7f9c7019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.725+0000 7f9c6e800700 1 -- 172.21.15.33:0/1172884868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006bad0 msgr2=0x7f9c7019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.725+0000 7f9c6e800700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006bad0 0x7f9c7019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.725+0000 7f9c6e800700 1 -- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006b130 msgr2=0x7f9c7019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.725+0000 7f9c6e800700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006b130 0x7f9c7019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.725+0000 7f9c6e800700 1 -- 172.21.15.33:0/1172884868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c7019a010 con 0x7f9c701094a0 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.725+0000 7f9c6e800700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c701094a0 0x7f9c701998b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800b070 tx=0x7f9c68004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.726+0000 7f9c67400700 1 -- 172.21.15.33:0/1172884868 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c68021070 con 0x7f9c701094a0 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.726+0000 7f9c67400700 1 -- 172.21.15.33:0/1172884868 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c68007e30 con 0x7f9c701094a0 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.726+0000 7f9c67400700 1 -- 172.21.15.33:0/1172884868 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c680045f0 con 0x7f9c701094a0 2024-04-26T11:27:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.726+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c7019a2a0 con 0x7f9c701094a0 2024-04-26T11:27:11.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.726+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c701a65c0 con 0x7f9c701094a0 2024-04-26T11:27:11.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.727+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c7006a110 con 0x7f9c701094a0 2024-04-26T11:27:11.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.729+0000 7f9c67400700 1 -- 172.21.15.33:0/1172884868 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9c68007980 con 0x7f9c701094a0 2024-04-26T11:27:11.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.730+0000 7f9c67400700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9c500a3d50 0x7f9c500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:11.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.731+0000 7f9c67400700 1 -- 172.21.15.33:0/1172884868 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f9c6800f020 con 0x7f9c701094a0 2024-04-26T11:27:11.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.731+0000 7f9c6de00700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9c500a3d50 0x7f9c500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:11.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.731+0000 7f9c6de00700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9c500a3d50 0x7f9c500a6210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9c7019a9f0 tx=0x7f9c60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:11.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:11.735+0000 7f9c67400700 1 -- 172.21.15.33:0/1172884868 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c6802a020 con 0x7f9c701094a0 2024-04-26T11:27:12.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.003+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9c70063860 con 0x7f9c500a3d50 2024-04-26T11:27:12.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.046+0000 7f9c67400700 1 -- 172.21.15.33:0/1172884868 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9c70063860 con 0x7f9c500a3d50 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9c500a3d50 msgr2=0x7f9c500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9c500a3d50 0x7f9c500a6210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9c7019a9f0 tx=0x7f9c60009260 comp rx=0 tx=0).stop 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c701094a0 msgr2=0x7f9c701998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c701094a0 0x7f9c701998b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800b070 tx=0x7f9c68004c80 comp rx=0 tx=0).stop 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 shutdown_connections 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f9c500a3d50 0x7f9c500a6210 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c701094a0 0x7f9c701998b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:12.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c7006b130 0x7f9c7019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:12.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 --2- 172.21.15.33:0/1172884868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006bad0 0x7f9c7019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:12.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 >> 172.21.15.33:0/1172884868 conn(0x7f9c700792d0 msgr2=0x7f9c70100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:12.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.049+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 shutdown_connections 2024-04-26T11:27:12.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:12.050+0000 7f9c75000700 1 -- 172.21.15.33:0/1172884868 wait complete. 2024-04-26T11:27:12.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T11:27:13.312 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:13.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:13 smithi033 ceph-mon[27845]: pgmap v345: 97 pgs: 97 active+clean; 608 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-04-26T11:27:13.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:13 smithi033 ceph-mon[27845]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:13 smithi194 ceph-mon[33017]: pgmap v345: 97 pgs: 97 active+clean; 608 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-04-26T11:27:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:13 smithi194 ceph-mon[33017]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:13 smithi096 ceph-mon[32753]: pgmap v345: 97 pgs: 97 active+clean; 608 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-04-26T11:27:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:13 smithi096 ceph-mon[32753]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:15.231 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:15 smithi033 ceph-mon[27845]: pgmap v346: 97 pgs: 97 active+clean; 608 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T11:27:15.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 -- 172.21.15.33:0/2163791559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 msgr2=0x7f0b9c1167c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/2163791559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 0x7f0b9c1167c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400b0a0 tx=0x7f0b9400b3b0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 -- 172.21.15.33:0/2163791559 shutdown_connections 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/2163791559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 0x7f0b9c1167c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/2163791559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b9c10e830 0x7f0b9c10ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/2163791559 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 0x7f0b9c109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 -- 172.21.15.33:0/2163791559 >> 172.21.15.33:0/2163791559 conn(0x7f0b9c0791b0 msgr2=0x7f0b9c108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 -- 172.21.15.33:0/2163791559 shutdown_connections 2024-04-26T11:27:15.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.516+0000 7f0ba0e00700 1 -- 172.21.15.33:0/2163791559 wait complete. 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.517+0000 7f0ba0e00700 1 Processor -- start 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0ba0e00700 1 -- start start 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0ba0e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 0x7f0b9c1a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0ba0e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 0x7f0b9c1a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0ba0e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b9c10e830 0x7f0b9c1a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0ba0e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9c119b80 con 0x7f0b9c10e830 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0ba0e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9c119a00 con 0x7f0b9c10a320 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0ba0e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9c119d00 con 0x7f0b9c109960 2024-04-26T11:27:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0b9be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b9c10e830 0x7f0b9c1a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0b9b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 0x7f0b9c1a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0b9be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b9c10e830 0x7f0b9c1a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47474/0 (socket says 172.21.15.33:47474) 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0b9b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 0x7f0b9c1a0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:39024/0 (socket says 172.21.15.33:39024) 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0b9be00700 1 -- 172.21.15.33:0/3890920383 learned_addr learned my addr 172.21.15.33:0/3890920383 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.518+0000 7f0b9aa00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 0x7f0b9c1a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b9b400700 1 -- 172.21.15.33:0/3890920383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 msgr2=0x7f0b9c1a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b9b400700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 0x7f0b9c1a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b9b400700 1 -- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b9c10e830 msgr2=0x7f0b9c1a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b9b400700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b9c10e830 0x7f0b9c1a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b9b400700 1 -- 172.21.15.33:0/3890920383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9c1a4ff0 con 0x7f0b9c109960 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b9aa00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 0x7f0b9c1a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b9b400700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 0x7f0b9c1a0030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00e9f0 tx=0x7f0b8c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:15.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b93e00700 1 -- 172.21.15.33:0/3890920383 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c00cbd0 con 0x7f0b9c109960 2024-04-26T11:27:15.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.519+0000 7f0b93e00700 1 -- 172.21.15.33:0/3890920383 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8c00cd70 con 0x7f0b9c109960 2024-04-26T11:27:15.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.520+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b9c1a52e0 con 0x7f0b9c109960 2024-04-26T11:27:15.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.520+0000 7f0b93e00700 1 -- 172.21.15.33:0/3890920383 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c018680 con 0x7f0b9c109960 2024-04-26T11:27:15.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.520+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b9c1acb60 con 0x7f0b9c109960 2024-04-26T11:27:15.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.521+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b9c06a110 con 0x7f0b9c109960 2024-04-26T11:27:15.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.523+0000 7f0b93e00700 1 -- 172.21.15.33:0/3890920383 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0b8c010430 con 0x7f0b9c109960 2024-04-26T11:27:15.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.524+0000 7f0b93e00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0b800a3cd0 0x7f0b800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:15.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.524+0000 7f0b93e00700 1 -- 172.21.15.33:0/3890920383 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f0b8c0d05f0 con 0x7f0b9c109960 2024-04-26T11:27:15.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.524+0000 7f0b9aa00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0b800a3cd0 0x7f0b800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:15.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.525+0000 7f0b9aa00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0b800a3cd0 0x7f0b800a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c1094a0 tx=0x7f0b88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:15.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.527+0000 7f0b93e00700 1 -- 172.21.15.33:0/3890920383 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b8c09cf20 con 0x7f0b9c109960 2024-04-26T11:27:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:15 smithi194 ceph-mon[33017]: pgmap v346: 97 pgs: 97 active+clean; 608 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T11:27:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:15 smithi096 ceph-mon[32753]: pgmap v346: 97 pgs: 97 active+clean; 608 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T11:27:15.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.796+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f0b9c063860 con 0x7f0b800a3cd0 2024-04-26T11:27:15.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.809+0000 7f0b93e00700 1 -- 172.21.15.33:0/3890920383 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0b9c063860 con 0x7f0b800a3cd0 2024-04-26T11:27:15.810 INFO:teuthology.orchestra.run.smithi033.stdout:/volumes/qa/sv_1/2b1eefe1-b994-4781-af0e-f418abad6198 2024-04-26T11:27:15.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0b800a3cd0 msgr2=0x7f0b800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:15.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0b800a3cd0 0x7f0b800a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c1094a0 tx=0x7f0b88009260 comp rx=0 tx=0).stop 2024-04-26T11:27:15.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 msgr2=0x7f0b9c1a0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:15.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 0x7f0b9c1a0030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00e9f0 tx=0x7f0b8c00ed00 comp rx=0 tx=0).stop 2024-04-26T11:27:15.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 shutdown_connections 2024-04-26T11:27:15.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0b800a3cd0 0x7f0b800a6190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0b9c10e830 0x7f0b9c1a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b9c109960 0x7f0b9c1a0030 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 --2- 172.21.15.33:0/3890920383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b9c10a320 0x7f0b9c1a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:15.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 >> 172.21.15.33:0/3890920383 conn(0x7f0b9c0791b0 msgr2=0x7f0b9c079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:15.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 shutdown_connections 2024-04-26T11:27:15.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:15.812+0000 7f0ba0e00700 1 -- 172.21.15.33:0/3890920383 wait complete. 2024-04-26T11:27:16.574 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:27:17.046 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:17.535 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:17 smithi033 ceph-mon[27845]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:17.535 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:17 smithi033 ceph-mon[27845]: pgmap v347: 97 pgs: 97 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-26T11:27:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:17 smithi194 ceph-mon[33017]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:17 smithi194 ceph-mon[33017]: pgmap v347: 97 pgs: 97 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-26T11:27:17.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:17 smithi096 ceph-mon[32753]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:27:17.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:17 smithi096 ceph-mon[32753]: pgmap v347: 97 pgs: 97 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-26T11:27:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T11:27:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-26T11:27:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-26T11:27:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-26T11:27:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-04-26T11:27:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-26T11:27:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T11:27:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-26T11:27:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:18 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-04-26T11:27:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T11:27:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-26T11:27:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-26T11:27:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-26T11:27:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-04-26T11:27:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-26T11:27:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T11:27:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-26T11:27:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:18 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-26T11:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:18 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-04-26T11:27:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 -- 172.21.15.33:0/2012317706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806b130 msgr2=0x7fbdf806b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 --2- 172.21.15.33:0/2012317706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806b130 0x7fbdf806b510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009990 tx=0x7fbdec009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 -- 172.21.15.33:0/2012317706 shutdown_connections 2024-04-26T11:27:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 --2- 172.21.15.33:0/2012317706 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf810fed0 0x7fbdf81122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 --2- 172.21.15.33:0/2012317706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806b130 0x7fbdf806b510 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 --2- 172.21.15.33:0/2012317706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806ba50 0x7fbdf810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 -- 172.21.15.33:0/2012317706 >> 172.21.15.33:0/2012317706 conn(0x7fbdf80792d0 msgr2=0x7fbdf80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:19.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 -- 172.21.15.33:0/2012317706 shutdown_connections 2024-04-26T11:27:19.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.200+0000 7fbe00800700 1 -- 172.21.15.33:0/2012317706 wait complete. 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.201+0000 7fbe00800700 1 Processor -- start 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.201+0000 7fbe00800700 1 -- start start 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbe00800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806b130 0x7fbdf81a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbe00800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806ba50 0x7fbdf81a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbe00800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf810fed0 0x7fbdf819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbe00800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf81157f0 con 0x7fbdf810fed0 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbe00800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf8115670 con 0x7fbdf806ba50 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbe00800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf8115970 con 0x7fbdf806b130 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806b130 0x7fbdf81a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806b130 0x7fbdf81a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:39046/0 (socket says 172.21.15.33:39046) 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 -- 172.21.15.33:0/717440322 learned_addr learned my addr 172.21.15.33:0/717440322 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfe800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf810fed0 0x7fbdf819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:19.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfd400700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806ba50 0x7fbdf81a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 -- 172.21.15.33:0/717440322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806ba50 msgr2=0x7fbdf81a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806ba50 0x7fbdf81a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 -- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf810fed0 msgr2=0x7fbdf819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf810fed0 0x7fbdf819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfde00700 1 -- 172.21.15.33:0/717440322 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf819c410 con 0x7fbdf806b130 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.202+0000 7fbdfe800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf810fed0 0x7fbdf819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.203+0000 7fbdfde00700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806b130 0x7fbdf81a1a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbdec000c00 tx=0x7fbdec00bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.203+0000 7fbdf2a00700 1 -- 172.21.15.33:0/717440322 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec027070 con 0x7fbdf806b130 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.203+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf819c6a0 con 0x7fbdf806b130 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.203+0000 7fbdf2a00700 1 -- 172.21.15.33:0/717440322 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdec022470 con 0x7fbdf806b130 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.203+0000 7fbdf2a00700 1 -- 172.21.15.33:0/717440322 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec017650 con 0x7fbdf806b130 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.203+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf81a8810 con 0x7fbdf806b130 2024-04-26T11:27:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.204+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf8102940 con 0x7fbdf806b130 2024-04-26T11:27:19.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.207+0000 7fbdf2a00700 1 -- 172.21.15.33:0/717440322 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbdec02a070 con 0x7fbdf806b130 2024-04-26T11:27:19.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.208+0000 7fbdf2a00700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbde00a3cd0 0x7fbde00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.208+0000 7fbdf2a00700 1 -- 172.21.15.33:0/717440322 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fbdec0d17b0 con 0x7fbdf806b130 2024-04-26T11:27:19.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.208+0000 7fbdfd400700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbde00a3cd0 0x7fbde00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:19.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.209+0000 7fbdfd400700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbde00a3cd0 0x7fbde00a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbdf819d280 tx=0x7fbde8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.210+0000 7fbdf2a00700 1 -- 172.21.15.33:0/717440322 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbdec013070 con 0x7fbdf806b130 2024-04-26T11:27:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: pgmap v348: 97 pgs: 97 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-26T11:27:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-04-26T11:27:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:19 smithi033 ceph-mon[27845]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T11:27:19.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.498+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdf802e490 con 0x7fbdf806b130 2024-04-26T11:27:19.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.499+0000 7fbdf2a00700 1 -- 172.21.15.33:0/717440322 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fbdec09e080 con 0x7fbdf806b130 2024-04-26T11:27:19.502 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:27:19.502 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:26:56.732811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:27:19.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.502+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbde00a3cd0 msgr2=0x7fbde00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:19.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.502+0000 7fbe00800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbde00a3cd0 0x7fbde00a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbdf819d280 tx=0x7fbde8009260 comp rx=0 tx=0).stop 2024-04-26T11:27:19.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.502+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806b130 msgr2=0x7fbdf81a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:19.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.502+0000 7fbe00800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806b130 0x7fbdf81a1a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbdec000c00 tx=0x7fbdec00bfd0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 shutdown_connections 2024-04-26T11:27:19.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbde00a3cd0 0x7fbde00a6190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf810fed0 0x7fbdf819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf806b130 0x7fbdf81a1a60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 --2- 172.21.15.33:0/717440322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf806ba50 0x7fbdf81a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:19.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 >> 172.21.15.33:0/717440322 conn(0x7fbdf80792d0 msgr2=0x7fbdf81113a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:19.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 shutdown_connections 2024-04-26T11:27:19.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:19.503+0000 7fbe00800700 1 -- 172.21.15.33:0/717440322 wait complete. 2024-04-26T11:27:19.506 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-04-26T11:27:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: pgmap v348: 97 pgs: 97 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-26T11:27:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T11:27:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-26T11:27:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-26T11:27:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-26T11:27:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-04-26T11:27:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-26T11:27:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T11:27:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-26T11:27:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-04-26T11:27:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:19 smithi194 ceph-mon[33017]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T11:27:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: pgmap v348: 97 pgs: 97 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-26T11:27:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T11:27:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-26T11:27:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-26T11:27:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-26T11:27:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-04-26T11:27:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-26T11:27:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T11:27:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-26T11:27:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-04-26T11:27:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:19 smithi096 ceph-mon[32753]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T11:27:20.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:27:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:20 smithi194 ceph-mon[33017]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T11:27:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:20 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/717440322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:27:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:20 smithi033 ceph-mon[27845]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T11:27:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:20 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/717440322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:27:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:20 smithi096 ceph-mon[32753]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T11:27:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:20 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/717440322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:27:20.772 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:21 smithi194 ceph-mon[33017]: pgmap v351: 97 pgs: 5 peering, 92 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-26T11:27:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:21 smithi033 ceph-mon[27845]: pgmap v351: 97 pgs: 5 peering, 92 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-26T11:27:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:21 smithi096 ceph-mon[32753]: pgmap v351: 97 pgs: 5 peering, 92 active+clean; 625 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-26T11:27:23.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 -- 172.21.15.33:0/1732722182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075650 msgr2=0x7fb424075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:23.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 --2- 172.21.15.33:0/1732722182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075650 0x7fb424075a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb414009990 tx=0x7fb414009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 -- 172.21.15.33:0/1732722182 shutdown_connections 2024-04-26T11:27:23.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 --2- 172.21.15.33:0/1732722182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb42410fe20 0x7fb424112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 --2- 172.21.15.33:0/1732722182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075650 0x7fb424075a30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 --2- 172.21.15.33:0/1732722182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb424075f70 0x7fb42410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 -- 172.21.15.33:0/1732722182 >> 172.21.15.33:0/1732722182 conn(0x7fb42407a580 msgr2=0x7fb42407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:23.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 -- 172.21.15.33:0/1732722182 shutdown_connections 2024-04-26T11:27:23.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:22.998+0000 7fb42b000700 1 -- 172.21.15.33:0/1732722182 wait complete. 2024-04-26T11:27:23.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 Processor -- start 2024-04-26T11:27:23.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 -- start start 2024-04-26T11:27:23.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb424075650 0x7fb424103540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:23.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075f70 0x7fb424103a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:23.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb42410fe20 0x7fb424103fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:23.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb424115810 con 0x7fb424075f70 2024-04-26T11:27:23.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb424115690 con 0x7fb424075650 2024-04-26T11:27:23.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.000+0000 7fb42b000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb424115990 con 0x7fb42410fe20 2024-04-26T11:27:23.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075f70 0x7fb424103a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:23.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075f70 0x7fb424103a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42098/0 (socket says 172.21.15.33:42098) 2024-04-26T11:27:23.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423400700 1 -- 172.21.15.33:0/580918316 learned_addr learned my addr 172.21.15.33:0/580918316 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:23.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423e00700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb424075650 0x7fb424103540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:23.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb429000700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb42410fe20 0x7fb424103fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:23.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423e00700 1 -- 172.21.15.33:0/580918316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb42410fe20 msgr2=0x7fb424103fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:23.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423e00700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb42410fe20 0x7fb424103fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423e00700 1 -- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075f70 msgr2=0x7fb424103a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:23.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423e00700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075f70 0x7fb424103a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423e00700 1 -- 172.21.15.33:0/580918316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4241ac430 con 0x7fb424075650 2024-04-26T11:27:23.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.001+0000 7fb423e00700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb424075650 0x7fb424103540 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb4140047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:23.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.002+0000 7fb420c00700 1 -- 172.21.15.33:0/580918316 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414026070 con 0x7fb424075650 2024-04-26T11:27:23.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.002+0000 7fb420c00700 1 -- 172.21.15.33:0/580918316 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb414021410 con 0x7fb424075650 2024-04-26T11:27:23.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.002+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4241ac6c0 con 0x7fb424075650 2024-04-26T11:27:23.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.002+0000 7fb420c00700 1 -- 172.21.15.33:0/580918316 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414017650 con 0x7fb424075650 2024-04-26T11:27:23.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.002+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4241acb70 con 0x7fb424075650 2024-04-26T11:27:23.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.003+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb424076d40 con 0x7fb424075650 2024-04-26T11:27:23.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.005+0000 7fb420c00700 1 -- 172.21.15.33:0/580918316 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb414029070 con 0x7fb424075650 2024-04-26T11:27:23.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.006+0000 7fb420c00700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb4080a3d80 0x7fb4080a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:23.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.007+0000 7fb420c00700 1 -- 172.21.15.33:0/580918316 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb4140d1170 con 0x7fb424075650 2024-04-26T11:27:23.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.007+0000 7fb423400700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb4080a3d80 0x7fb4080a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:23.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.007+0000 7fb423400700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb4080a3d80 0x7fb4080a6240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb410009680 tx=0x7fb410009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:23.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.011+0000 7fb420c00700 1 -- 172.21.15.33:0/580918316 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb41405dff0 con 0x7fb424075650 2024-04-26T11:27:23.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.269+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb42402e5b0 con 0x7fb424075650 2024-04-26T11:27:23.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.271+0000 7fb420c00700 1 -- 172.21.15.33:0/580918316 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22592 (secure 0 0 0) 0x7fb414013070 con 0x7fb424075650 2024-04-26T11:27:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:27:23.274 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:27:19.230568+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.274+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb4080a3d80 msgr2=0x7fb4080a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb4080a3d80 0x7fb4080a6240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb410009680 tx=0x7fb410009260 comp rx=0 tx=0).stop 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb424075650 msgr2=0x7fb424103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb424075650 0x7fb424103540 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb4140047b0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 shutdown_connections 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb4080a3d80 0x7fb4080a6240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb42410fe20 0x7fb424103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb424075650 0x7fb424103540 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 --2- 172.21.15.33:0/580918316 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb424075f70 0x7fb424103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:23.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 >> 172.21.15.33:0/580918316 conn(0x7fb42407a580 msgr2=0x7fb4241006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:23.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 shutdown_connections 2024-04-26T11:27:23.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:23.275+0000 7fb42b000700 1 -- 172.21.15.33:0/580918316 wait complete. 2024-04-26T11:27:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:23 smithi194 ceph-mon[33017]: pgmap v352: 97 pgs: 5 peering, 92 active+clean; 634 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-04-26T11:27:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:23 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:27:23.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:23 smithi033 ceph-mon[27845]: pgmap v352: 97 pgs: 5 peering, 92 active+clean; 634 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-04-26T11:27:23.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:23 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:27:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:23 smithi096 ceph-mon[32753]: pgmap v352: 97 pgs: 5 peering, 92 active+clean; 634 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-04-26T11:27:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:23 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:27:24.114 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T11:27:24.114 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T11:27:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:24 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/580918316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:24 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/580918316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:24 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/580918316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:25.215 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:25.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:25 smithi033 ceph-mon[27845]: pgmap v353: 97 pgs: 5 peering, 92 active+clean; 634 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T11:27:25.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:25 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:25 smithi194 ceph-mon[33017]: pgmap v353: 97 pgs: 5 peering, 92 active+clean; 634 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T11:27:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:25 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:25 smithi096 ceph-mon[32753]: pgmap v353: 97 pgs: 5 peering, 92 active+clean; 634 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T11:27:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:25 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:25 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:25 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:25.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:25 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:27.453 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:27 smithi033 ceph-mon[27845]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.7 KiB/s wr, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:27.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.450+0000 7f2358600700 1 -- 172.21.15.33:0/3913641253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f235010fe20 msgr2=0x7f2350112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:27.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.450+0000 7f2358600700 1 --2- 172.21.15.33:0/3913641253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f235010fe20 0x7f2350112210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f234800b0a0 tx=0x7f234800b3b0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.451+0000 7f2358600700 1 -- 172.21.15.33:0/3913641253 shutdown_connections 2024-04-26T11:27:27.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.451+0000 7f2358600700 1 --2- 172.21.15.33:0/3913641253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f235010fe20 0x7f2350112210 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.451+0000 7f2358600700 1 --2- 172.21.15.33:0/3913641253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2350075650 0x7f2350075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.451+0000 7f2358600700 1 --2- 172.21.15.33:0/3913641253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075f70 0x7f235010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.451+0000 7f2358600700 1 -- 172.21.15.33:0/3913641253 >> 172.21.15.33:0/3913641253 conn(0x7f235007a580 msgr2=0x7f235007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:27.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.451+0000 7f2358600700 1 -- 172.21.15.33:0/3913641253 shutdown_connections 2024-04-26T11:27:27.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.451+0000 7f2358600700 1 -- 172.21.15.33:0/3913641253 wait complete. 2024-04-26T11:27:27.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.452+0000 7f2358600700 1 Processor -- start 2024-04-26T11:27:27.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.452+0000 7f2358600700 1 -- start start 2024-04-26T11:27:27.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2358600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075650 0x7f23501a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:27.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2358600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350075f70 0x7f23501a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:27.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2358600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f235010fe20 0x7f235019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:27.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2358600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23501157d0 con 0x7f2350075f70 2024-04-26T11:27:27.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2358600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2350115650 con 0x7f2350075650 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2358600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2350115950 con 0x7f235010fe20 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2355c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075650 0x7f23501a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2355c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075650 0x7f23501a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56074/0 (socket says 172.21.15.33:56074) 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2355c00700 1 -- 172.21.15.33:0/3044332755 learned_addr learned my addr 172.21.15.33:0/3044332755 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2355200700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350075f70 0x7f23501a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.453+0000 7f2356600700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f235010fe20 0x7f235019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2355c00700 1 -- 172.21.15.33:0/3044332755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f235010fe20 msgr2=0x7f235019bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2355c00700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f235010fe20 0x7f235019bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2355c00700 1 -- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350075f70 msgr2=0x7f23501a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2355c00700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350075f70 0x7f23501a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2355c00700 1 -- 172.21.15.33:0/3044332755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235019c210 con 0x7f2350075650 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2355200700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350075f70 0x7f23501a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2355c00700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075650 0x7f23501a18a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f234400e9c0 tx=0x7f234400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:27.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f234e000700 1 -- 172.21.15.33:0/3044332755 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234400cbd0 con 0x7f2350075650 2024-04-26T11:27:27.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235019c500 con 0x7f2350075650 2024-04-26T11:27:27.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f234e000700 1 -- 172.21.15.33:0/3044332755 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f234400cd70 con 0x7f2350075650 2024-04-26T11:27:27.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.454+0000 7f234e000700 1 -- 172.21.15.33:0/3044332755 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344005610 con 0x7f2350075650 2024-04-26T11:27:27.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.455+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23501a8770 con 0x7f2350075650 2024-04-26T11:27:27.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.456+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235006a110 con 0x7f2350075650 2024-04-26T11:27:27.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.458+0000 7f234e000700 1 -- 172.21.15.33:0/3044332755 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2344004750 con 0x7f2350075650 2024-04-26T11:27:27.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.459+0000 7f234e000700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23380a3cd0 0x7f23380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:27.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.459+0000 7f234e000700 1 -- 172.21.15.33:0/3044332755 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f23440d0900 con 0x7f2350075650 2024-04-26T11:27:27.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.459+0000 7f2355200700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23380a3cd0 0x7f23380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:27.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.460+0000 7f2355200700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23380a3cd0 0x7f23380a6190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f233c005950 tx=0x7f233c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.464+0000 7f234e000700 1 -- 172.21.15.33:0/3044332755 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f234409d140 con 0x7f2350075650 2024-04-26T11:27:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:27 smithi194 ceph-mon[33017]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.7 KiB/s wr, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:27 smithi096 ceph-mon[32753]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.7 KiB/s wr, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:27.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.707+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2350063860 con 0x7f23380a3cd0 2024-04-26T11:27:27.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.724+0000 7f234e000700 1 -- 172.21.15.33:0/3044332755 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259073 (secure 0 0 0) 0x7f2350063860 con 0x7f23380a3cd0 2024-04-26T11:27:27.726 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:27:27.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23380a3cd0 msgr2=0x7f23380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:27.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23380a3cd0 0x7f23380a6190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f233c005950 tx=0x7f233c0058e0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075650 msgr2=0x7f23501a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:27.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075650 0x7f23501a18a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f234400e9c0 tx=0x7f234400ecd0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 shutdown_connections 2024-04-26T11:27:27.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23380a3cd0 0x7f23380a6190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f235010fe20 0x7f235019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350075650 0x7f23501a18a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 --2- 172.21.15.33:0/3044332755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350075f70 0x7f23501a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:27.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 >> 172.21.15.33:0/3044332755 conn(0x7f235007a580 msgr2=0x7f2350100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:27.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 shutdown_connections 2024-04-26T11:27:27.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:27.729+0000 7f2358600700 1 -- 172.21.15.33:0/3044332755 wait complete. 2024-04-26T11:27:27.734 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-26T11:27:28.535 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-04-26T11:27:25.827947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657756,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":353,"num_read_kb":299,"num_write":351,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":294,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357044,"kb_used_data":11992,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584780,"statfs":{"total":1151940427776,"available":1151574814720,"internally_reserved":0,"allocated":12279808,"data_stored":7717190,"data_compressed":182166,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17566,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":43,"num_write":14,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003254"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.839113+0000","last_change":"2024-04-26T11:26:29.018818+0000","last_active":"2024-04-26T11:27:19.839113+0000","last_peered":"2024-04-26T11:27:19.839113+0000","last_clean":"2024-04-26T11:27:19.839113+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:27:19.839113+0000","last_undegraded":"2024-04-26T11:27:19.839113+0000","last_fullsized":"2024-04-26T11:27:19.839113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:05.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890260+0000","last_change":"2024-04-26T11:26:29.021338+0000","last_active":"2024-04-26T11:27:19.890260+0000","last_peered":"2024-04-26T11:27:19.890260+0000","last_clean":"2024-04-26T11:27:19.890260+0000","last_became_active":"2024-04-26T11:26:29.021019+0000","last_became_peered":"2024-04-26T11:26:29.021019+0000","last_unstale":"2024-04-26T11:27:19.890260+0000","last_undegraded":"2024-04-26T11:27:19.890260+0000","last_fullsized":"2024-04-26T11:27:19.890260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:52:44.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890073+0000","last_change":"2024-04-26T11:26:29.014166+0000","last_active":"2024-04-26T11:27:19.890073+0000","last_peered":"2024-04-26T11:27:19.890073+0000","last_clean":"2024-04-26T11:27:19.890073+0000","last_became_active":"2024-04-26T11:26:29.013812+0000","last_became_peered":"2024-04-26T11:26:29.013812+0000","last_unstale":"2024-04-26T11:27:19.890073+0000","last_undegraded":"2024-04-26T11:27:19.890073+0000","last_fullsized":"2024-04-26T11:27:19.890073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:08.683633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240873+0000","last_change":"2024-04-26T11:26:28.999249+0000","last_active":"2024-04-26T11:27:19.240873+0000","last_peered":"2024-04-26T11:27:19.240873+0000","last_clean":"2024-04-26T11:27:19.240873+0000","last_became_active":"2024-04-26T11:26:28.998927+0000","last_became_peered":"2024-04-26T11:26:28.998927+0000","last_unstale":"2024-04-26T11:27:19.240873+0000","last_undegraded":"2024-04-26T11:27:19.240873+0000","last_fullsized":"2024-04-26T11:27:19.240873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:56.674537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245154+0000","last_change":"2024-04-26T11:26:29.017202+0000","last_active":"2024-04-26T11:27:19.245154+0000","last_peered":"2024-04-26T11:27:19.245154+0000","last_clean":"2024-04-26T11:27:19.245154+0000","last_became_active":"2024-04-26T11:26:29.016849+0000","last_became_peered":"2024-04-26T11:26:29.016849+0000","last_unstale":"2024-04-26T11:27:19.245154+0000","last_undegraded":"2024-04-26T11:27:19.245154+0000","last_fullsized":"2024-04-26T11:27:19.245154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:58:22.164118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.278294+0000","last_change":"2024-04-26T11:27:19.278294+0000","last_active":"2024-04-26T11:27:19.278294+0000","last_peered":"2024-04-26T11:27:19.278294+0000","last_clean":"2024-04-26T11:27:19.278294+0000","last_became_active":"2024-04-26T11:27:19.277853+0000","last_became_peered":"2024-04-26T11:27:19.277853+0000","last_unstale":"2024-04-26T11:27:19.278294+0000","last_undegraded":"2024-04-26T11:27:19.278294+0000","last_fullsized":"2024-04-26T11:27:19.278294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:12:18.733166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276702+0000","last_change":"2024-04-26T11:26:28.991406+0000","last_active":"2024-04-26T11:27:19.276702+0000","last_peered":"2024-04-26T11:27:19.276702+0000","last_clean":"2024-04-26T11:27:19.276702+0000","last_became_active":"2024-04-26T11:26:28.989303+0000","last_became_peered":"2024-04-26T11:26:28.989303+0000","last_unstale":"2024-04-26T11:27:19.276702+0000","last_undegraded":"2024-04-26T11:27:19.276702+0000","last_fullsized":"2024-04-26T11:27:19.276702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:42:59.449078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245298+0000","last_change":"2024-04-26T11:26:29.020922+0000","last_active":"2024-04-26T11:27:19.245298+0000","last_peered":"2024-04-26T11:27:19.245298+0000","last_clean":"2024-04-26T11:27:19.245298+0000","last_became_active":"2024-04-26T11:26:29.020576+0000","last_became_peered":"2024-04-26T11:26:29.020576+0000","last_unstale":"2024-04-26T11:27:19.245298+0000","last_undegraded":"2024-04-26T11:27:19.245298+0000","last_fullsized":"2024-04-26T11:27:19.245298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:48.770319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245000+0000","last_change":"2024-04-26T11:26:29.004171+0000","last_active":"2024-04-26T11:27:19.245000+0000","last_peered":"2024-04-26T11:27:19.245000+0000","last_clean":"2024-04-26T11:27:19.245000+0000","last_became_active":"2024-04-26T11:26:29.003816+0000","last_became_peered":"2024-04-26T11:26:29.003816+0000","last_unstale":"2024-04-26T11:27:19.245000+0000","last_undegraded":"2024-04-26T11:27:19.245000+0000","last_fullsized":"2024-04-26T11:27:19.245000+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:45:14.807634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242873+0000","last_change":"2024-04-26T11:26:28.999453+0000","last_active":"2024-04-26T11:27:19.242873+0000","last_peered":"2024-04-26T11:27:19.242873+0000","last_clean":"2024-04-26T11:27:19.242873+0000","last_became_active":"2024-04-26T11:26:28.999104+0000","last_became_peered":"2024-04-26T11:26:28.999104+0000","last_unstale":"2024-04-26T11:27:19.242873+0000","last_undegraded":"2024-04-26T11:27:19.242873+0000","last_fullsized":"2024-04-26T11:27:19.242873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:55.408949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277046+0000","last_change":"2024-04-26T11:26:28.999218+0000","last_active":"2024-04-26T11:27:19.277046+0000","last_peered":"2024-04-26T11:27:19.277046+0000","last_clean":"2024-04-26T11:27:19.277046+0000","last_became_active":"2024-04-26T11:26:28.998584+0000","last_became_peered":"2024-04-26T11:26:28.998584+0000","last_unstale":"2024-04-26T11:27:19.277046+0000","last_undegraded":"2024-04-26T11:27:19.277046+0000","last_fullsized":"2024-04-26T11:27:19.277046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:18.040520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240742+0000","last_change":"2024-04-26T11:26:28.997121+0000","last_active":"2024-04-26T11:27:19.240742+0000","last_peered":"2024-04-26T11:27:19.240742+0000","last_clean":"2024-04-26T11:27:19.240742+0000","last_became_active":"2024-04-26T11:26:28.996556+0000","last_became_peered":"2024-04-26T11:26:28.996556+0000","last_unstale":"2024-04-26T11:27:19.240742+0000","last_undegraded":"2024-04-26T11:27:19.240742+0000","last_fullsized":"2024-04-26T11:27:19.240742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:18:55.251371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412020+0000","last_change":"2024-04-26T11:26:28.999525+0000","last_active":"2024-04-26T11:27:19.412020+0000","last_peered":"2024-04-26T11:27:19.412020+0000","last_clean":"2024-04-26T11:27:19.412020+0000","last_became_active":"2024-04-26T11:26:28.998578+0000","last_became_peered":"2024-04-26T11:26:28.998578+0000","last_unstale":"2024-04-26T11:27:19.412020+0000","last_undegraded":"2024-04-26T11:27:19.412020+0000","last_fullsized":"2024-04-26T11:27:19.412020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:32:35.523804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.889408+0000","last_change":"2024-04-26T11:26:29.008427+0000","last_active":"2024-04-26T11:27:19.889408+0000","last_peered":"2024-04-26T11:27:19.889408+0000","last_clean":"2024-04-26T11:27:19.889408+0000","last_became_active":"2024-04-26T11:26:29.008073+0000","last_became_peered":"2024-04-26T11:26:29.008073+0000","last_unstale":"2024-04-26T11:27:19.889408+0000","last_undegraded":"2024-04-26T11:27:19.889408+0000","last_fullsized":"2024-04-26T11:27:19.889408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:03.185941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.889360+0000","last_change":"2024-04-26T11:26:29.015136+0000","last_active":"2024-04-26T11:27:19.889360+0000","last_peered":"2024-04-26T11:27:19.889360+0000","last_clean":"2024-04-26T11:27:19.889360+0000","last_became_active":"2024-04-26T11:26:29.014800+0000","last_became_peered":"2024-04-26T11:26:29.014800+0000","last_unstale":"2024-04-26T11:27:19.889360+0000","last_undegraded":"2024-04-26T11:27:19.889360+0000","last_fullsized":"2024-04-26T11:27:19.889360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:28:35.871046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244404+0000","last_change":"2024-04-26T11:26:29.024795+0000","last_active":"2024-04-26T11:27:19.244404+0000","last_peered":"2024-04-26T11:27:19.244404+0000","last_clean":"2024-04-26T11:27:19.244404+0000","last_became_active":"2024-04-26T11:26:29.024430+0000","last_became_peered":"2024-04-26T11:26:29.024430+0000","last_unstale":"2024-04-26T11:27:19.244404+0000","last_undegraded":"2024-04-26T11:27:19.244404+0000","last_fullsized":"2024-04-26T11:27:19.244404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:05:20.172869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.839207+0000","last_change":"2024-04-26T11:26:29.018822+0000","last_active":"2024-04-26T11:27:19.839207+0000","last_peered":"2024-04-26T11:27:19.839207+0000","last_clean":"2024-04-26T11:27:19.839207+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:27:19.839207+0000","last_undegraded":"2024-04-26T11:27:19.839207+0000","last_fullsized":"2024-04-26T11:27:19.839207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:04.475400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242166+0000","last_change":"2024-04-26T11:26:29.001984+0000","last_active":"2024-04-26T11:27:19.242166+0000","last_peered":"2024-04-26T11:27:19.242166+0000","last_clean":"2024-04-26T11:27:19.242166+0000","last_became_active":"2024-04-26T11:26:28.999492+0000","last_became_peered":"2024-04-26T11:26:28.999492+0000","last_unstale":"2024-04-26T11:27:19.242166+0000","last_undegraded":"2024-04-26T11:27:19.242166+0000","last_fullsized":"2024-04-26T11:27:19.242166+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:21.398823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242387+0000","last_change":"2024-04-26T11:26:29.019677+0000","last_active":"2024-04-26T11:27:19.242387+0000","last_peered":"2024-04-26T11:27:19.242387+0000","last_clean":"2024-04-26T11:27:19.242387+0000","last_became_active":"2024-04-26T11:26:29.019315+0000","last_became_peered":"2024-04-26T11:26:29.019315+0000","last_unstale":"2024-04-26T11:27:19.242387+0000","last_undegraded":"2024-04-26T11:27:19.242387+0000","last_fullsized":"2024-04-26T11:27:19.242387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:20.239140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242574+0000","last_change":"2024-04-26T11:26:24.964887+0000","last_active":"2024-04-26T11:27:19.242574+0000","last_peered":"2024-04-26T11:27:19.242574+0000","last_clean":"2024-04-26T11:27:19.242574+0000","last_became_active":"2024-04-26T11:26:24.964380+0000","last_became_peered":"2024-04-26T11:26:24.964380+0000","last_unstale":"2024-04-26T11:27:19.242574+0000","last_undegraded":"2024-04-26T11:27:19.242574+0000","last_fullsized":"2024-04-26T11:27:19.242574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:59:41.347542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244889+0000","last_change":"2024-04-26T11:27:19.244889+0000","last_active":"2024-04-26T11:27:19.244889+0000","last_peered":"2024-04-26T11:27:19.244889+0000","last_clean":"2024-04-26T11:27:19.244889+0000","last_became_active":"2024-04-26T11:27:19.244428+0000","last_became_peered":"2024-04-26T11:27:19.244428+0000","last_unstale":"2024-04-26T11:27:19.244889+0000","last_undegraded":"2024-04-26T11:27:19.244889+0000","last_fullsized":"2024-04-26T11:27:19.244889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:09:39.366242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246839+0000","last_change":"2024-04-26T11:26:25.082741+0000","last_active":"2024-04-26T11:27:19.246839+0000","last_peered":"2024-04-26T11:27:19.246839+0000","last_clean":"2024-04-26T11:27:19.246839+0000","last_became_active":"2024-04-26T11:26:25.082361+0000","last_became_peered":"2024-04-26T11:26:25.082361+0000","last_unstale":"2024-04-26T11:27:19.246839+0000","last_undegraded":"2024-04-26T11:27:19.246839+0000","last_fullsized":"2024-04-26T11:27:19.246839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:23:17.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.891043+0000","last_change":"2024-04-26T11:26:29.014817+0000","last_active":"2024-04-26T11:27:19.891043+0000","last_peered":"2024-04-26T11:27:19.891043+0000","last_clean":"2024-04-26T11:27:19.891043+0000","last_became_active":"2024-04-26T11:26:29.014419+0000","last_became_peered":"2024-04-26T11:26:29.014419+0000","last_unstale":"2024-04-26T11:27:19.891043+0000","last_undegraded":"2024-04-26T11:27:19.891043+0000","last_fullsized":"2024-04-26T11:27:19.891043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:33:28.355075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245880+0000","last_change":"2024-04-26T11:26:24.960224+0000","last_active":"2024-04-26T11:27:19.245880+0000","last_peered":"2024-04-26T11:27:19.245880+0000","last_clean":"2024-04-26T11:27:19.245880+0000","last_became_active":"2024-04-26T11:26:24.959848+0000","last_became_peered":"2024-04-26T11:26:24.959848+0000","last_unstale":"2024-04-26T11:27:19.245880+0000","last_undegraded":"2024-04-26T11:27:19.245880+0000","last_fullsized":"2024-04-26T11:27:19.245880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:49:10.932187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244563+0000","last_change":"2024-04-26T11:26:29.001946+0000","last_active":"2024-04-26T11:27:19.244563+0000","last_peered":"2024-04-26T11:27:19.244563+0000","last_clean":"2024-04-26T11:27:19.244563+0000","last_became_active":"2024-04-26T11:26:29.001579+0000","last_became_peered":"2024-04-26T11:26:29.001579+0000","last_unstale":"2024-04-26T11:27:19.244563+0000","last_undegraded":"2024-04-26T11:27:19.244563+0000","last_fullsized":"2024-04-26T11:27:19.244563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:14:43.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245485+0000","last_change":"2024-04-26T11:26:24.956148+0000","last_active":"2024-04-26T11:27:19.245485+0000","last_peered":"2024-04-26T11:27:19.245485+0000","last_clean":"2024-04-26T11:27:19.245485+0000","last_became_active":"2024-04-26T11:26:24.955736+0000","last_became_peered":"2024-04-26T11:26:24.955736+0000","last_unstale":"2024-04-26T11:27:19.245485+0000","last_undegraded":"2024-04-26T11:27:19.245485+0000","last_fullsized":"2024-04-26T11:27:19.245485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:43.705470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.415134+0000","last_change":"2024-04-26T11:27:19.415134+0000","last_active":"2024-04-26T11:27:19.415134+0000","last_peered":"2024-04-26T11:27:19.415134+0000","last_clean":"2024-04-26T11:27:19.415134+0000","last_became_active":"2024-04-26T11:27:19.414703+0000","last_became_peered":"2024-04-26T11:27:19.414703+0000","last_unstale":"2024-04-26T11:27:19.415134+0000","last_undegraded":"2024-04-26T11:27:19.415134+0000","last_fullsized":"2024-04-26T11:27:19.415134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:41:11.268296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:21.183058+0000","last_change":"2024-04-26T11:26:24.958350+0000","last_active":"2024-04-26T11:27:21.183058+0000","last_peered":"2024-04-26T11:27:21.183058+0000","last_clean":"2024-04-26T11:27:21.183058+0000","last_became_active":"2024-04-26T11:26:24.957382+0000","last_became_peered":"2024-04-26T11:26:24.957382+0000","last_unstale":"2024-04-26T11:27:21.183058+0000","last_undegraded":"2024-04-26T11:27:21.183058+0000","last_fullsized":"2024-04-26T11:27:21.183058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:03.854227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245438+0000","last_change":"2024-04-26T11:26:28.997568+0000","last_active":"2024-04-26T11:27:19.245438+0000","last_peered":"2024-04-26T11:27:19.245438+0000","last_clean":"2024-04-26T11:27:19.245438+0000","last_became_active":"2024-04-26T11:26:28.997161+0000","last_became_peered":"2024-04-26T11:26:28.997161+0000","last_unstale":"2024-04-26T11:27:19.245438+0000","last_undegraded":"2024-04-26T11:27:19.245438+0000","last_fullsized":"2024-04-26T11:27:19.245438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:04.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.243026+0000","last_change":"2024-04-26T11:26:24.958093+0000","last_active":"2024-04-26T11:27:19.243026+0000","last_peered":"2024-04-26T11:27:19.243026+0000","last_clean":"2024-04-26T11:27:19.243026+0000","last_became_active":"2024-04-26T11:26:24.957730+0000","last_became_peered":"2024-04-26T11:26:24.957730+0000","last_unstale":"2024-04-26T11:27:19.243026+0000","last_undegraded":"2024-04-26T11:27:19.243026+0000","last_fullsized":"2024-04-26T11:27:19.243026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:09:29.827063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.248406+0000","last_change":"2024-04-26T11:27:19.248406+0000","last_active":"2024-04-26T11:27:19.248406+0000","last_peered":"2024-04-26T11:27:19.248406+0000","last_clean":"2024-04-26T11:27:19.248406+0000","last_became_active":"2024-04-26T11:27:19.247980+0000","last_became_peered":"2024-04-26T11:27:19.247980+0000","last_unstale":"2024-04-26T11:27:19.248406+0000","last_undegraded":"2024-04-26T11:27:19.248406+0000","last_fullsized":"2024-04-26T11:27:19.248406+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:51:34.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.411996+0000","last_change":"2024-04-26T11:26:24.963463+0000","last_active":"2024-04-26T11:27:19.411996+0000","last_peered":"2024-04-26T11:27:19.411996+0000","last_clean":"2024-04-26T11:27:19.411996+0000","last_became_active":"2024-04-26T11:26:24.963030+0000","last_became_peered":"2024-04-26T11:26:24.963030+0000","last_unstale":"2024-04-26T11:27:19.411996+0000","last_undegraded":"2024-04-26T11:27:19.411996+0000","last_fullsized":"2024-04-26T11:27:19.411996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:04:17.844500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245456+0000","last_change":"2024-04-26T11:26:29.013899+0000","last_active":"2024-04-26T11:27:19.245456+0000","last_peered":"2024-04-26T11:27:19.245456+0000","last_clean":"2024-04-26T11:27:19.245456+0000","last_became_active":"2024-04-26T11:26:29.012674+0000","last_became_peered":"2024-04-26T11:26:29.012674+0000","last_unstale":"2024-04-26T11:27:19.245456+0000","last_undegraded":"2024-04-26T11:27:19.245456+0000","last_fullsized":"2024-04-26T11:27:19.245456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:33:46.259841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240050+0000","last_change":"2024-04-26T11:26:24.957803+0000","last_active":"2024-04-26T11:27:19.240050+0000","last_peered":"2024-04-26T11:27:19.240050+0000","last_clean":"2024-04-26T11:27:19.240050+0000","last_became_active":"2024-04-26T11:26:24.957496+0000","last_became_peered":"2024-04-26T11:26:24.957496+0000","last_unstale":"2024-04-26T11:27:19.240050+0000","last_undegraded":"2024-04-26T11:27:19.240050+0000","last_fullsized":"2024-04-26T11:27:19.240050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:15.497106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240308+0000","last_change":"2024-04-26T11:26:29.021648+0000","last_active":"2024-04-26T11:27:19.240308+0000","last_peered":"2024-04-26T11:27:19.240308+0000","last_clean":"2024-04-26T11:27:19.240308+0000","last_became_active":"2024-04-26T11:26:29.020689+0000","last_became_peered":"2024-04-26T11:26:29.020689+0000","last_unstale":"2024-04-26T11:27:19.240308+0000","last_undegraded":"2024-04-26T11:27:19.240308+0000","last_fullsized":"2024-04-26T11:27:19.240308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:15:42.321851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:24.186866+0000","last_change":"2024-04-26T11:26:24.966184+0000","last_active":"2024-04-26T11:27:24.186866+0000","last_peered":"2024-04-26T11:27:24.186866+0000","last_clean":"2024-04-26T11:27:24.186866+0000","last_became_active":"2024-04-26T11:26:24.965850+0000","last_became_peered":"2024-04-26T11:26:24.965850+0000","last_unstale":"2024-04-26T11:27:24.186866+0000","last_undegraded":"2024-04-26T11:27:24.186866+0000","last_fullsized":"2024-04-26T11:27:24.186866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:49.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244856+0000","last_change":"2024-04-26T11:26:29.003570+0000","last_active":"2024-04-26T11:27:19.244856+0000","last_peered":"2024-04-26T11:27:19.244856+0000","last_clean":"2024-04-26T11:27:19.244856+0000","last_became_active":"2024-04-26T11:26:29.003039+0000","last_became_peered":"2024-04-26T11:26:29.003039+0000","last_unstale":"2024-04-26T11:27:19.244856+0000","last_undegraded":"2024-04-26T11:27:19.244856+0000","last_fullsized":"2024-04-26T11:27:19.244856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:29:16.166432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277109+0000","last_change":"2024-04-26T11:26:24.954968+0000","last_active":"2024-04-26T11:27:19.277109+0000","last_peered":"2024-04-26T11:27:19.277109+0000","last_clean":"2024-04-26T11:27:19.277109+0000","last_became_active":"2024-04-26T11:26:24.954595+0000","last_became_peered":"2024-04-26T11:26:24.954595+0000","last_unstale":"2024-04-26T11:27:19.277109+0000","last_undegraded":"2024-04-26T11:27:19.277109+0000","last_fullsized":"2024-04-26T11:27:19.277109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:52:21.988942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890791+0000","last_change":"2024-04-26T11:26:29.015601+0000","last_active":"2024-04-26T11:27:19.890791+0000","last_peered":"2024-04-26T11:27:19.890791+0000","last_clean":"2024-04-26T11:27:19.890791+0000","last_became_active":"2024-04-26T11:26:29.015199+0000","last_became_peered":"2024-04-26T11:26:29.015199+0000","last_unstale":"2024-04-26T11:27:19.890791+0000","last_undegraded":"2024-04-26T11:27:19.890791+0000","last_fullsized":"2024-04-26T11:27:19.890791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:15:04.573214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240544+0000","last_change":"2024-04-26T11:26:25.085542+0000","last_active":"2024-04-26T11:27:19.240544+0000","last_peered":"2024-04-26T11:27:19.240544+0000","last_clean":"2024-04-26T11:27:19.240544+0000","last_became_active":"2024-04-26T11:26:25.085230+0000","last_became_peered":"2024-04-26T11:26:25.085230+0000","last_unstale":"2024-04-26T11:27:19.240544+0000","last_undegraded":"2024-04-26T11:27:19.240544+0000","last_fullsized":"2024-04-26T11:27:19.240544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:51.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244661+0000","last_change":"2024-04-26T11:26:29.003954+0000","last_active":"2024-04-26T11:27:19.244661+0000","last_peered":"2024-04-26T11:27:19.244661+0000","last_clean":"2024-04-26T11:27:19.244661+0000","last_became_active":"2024-04-26T11:26:29.003576+0000","last_became_peered":"2024-04-26T11:26:29.003576+0000","last_unstale":"2024-04-26T11:27:19.244661+0000","last_undegraded":"2024-04-26T11:27:19.244661+0000","last_fullsized":"2024-04-26T11:27:19.244661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:24.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":130,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:25.188093+0000","last_change":"2024-04-26T11:26:24.957823+0000","last_active":"2024-04-26T11:27:25.188093+0000","last_peered":"2024-04-26T11:27:25.188093+0000","last_clean":"2024-04-26T11:27:25.188093+0000","last_became_active":"2024-04-26T11:26:24.957434+0000","last_became_peered":"2024-04-26T11:26:24.957434+0000","last_unstale":"2024-04-26T11:27:25.188093+0000","last_undegraded":"2024-04-26T11:27:25.188093+0000","last_fullsized":"2024-04-26T11:27:25.188093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:38:43.247282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63153,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":74,"num_read_kb":98,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245570+0000","last_change":"2024-04-26T11:26:28.994764+0000","last_active":"2024-04-26T11:27:19.245570+0000","last_peered":"2024-04-26T11:27:19.245570+0000","last_clean":"2024-04-26T11:27:19.245570+0000","last_became_active":"2024-04-26T11:26:28.994421+0000","last_became_peered":"2024-04-26T11:26:28.994421+0000","last_unstale":"2024-04-26T11:27:19.245570+0000","last_undegraded":"2024-04-26T11:27:19.245570+0000","last_fullsized":"2024-04-26T11:27:19.245570+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:06:06.499434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242227+0000","last_change":"2024-04-26T11:26:24.968036+0000","last_active":"2024-04-26T11:27:19.242227+0000","last_peered":"2024-04-26T11:27:19.242227+0000","last_clean":"2024-04-26T11:27:19.242227+0000","last_became_active":"2024-04-26T11:26:24.967684+0000","last_became_peered":"2024-04-26T11:26:24.967684+0000","last_unstale":"2024-04-26T11:27:19.242227+0000","last_undegraded":"2024-04-26T11:27:19.242227+0000","last_fullsized":"2024-04-26T11:27:19.242227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:00:27.467411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412101+0000","last_change":"2024-04-26T11:26:29.020642+0000","last_active":"2024-04-26T11:27:19.412101+0000","last_peered":"2024-04-26T11:27:19.412101+0000","last_clean":"2024-04-26T11:27:19.412101+0000","last_became_active":"2024-04-26T11:26:29.020278+0000","last_became_peered":"2024-04-26T11:26:29.020278+0000","last_unstale":"2024-04-26T11:27:19.412101+0000","last_undegraded":"2024-04-26T11:27:19.412101+0000","last_fullsized":"2024-04-26T11:27:19.412101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:48.932064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240421+0000","last_change":"2024-04-26T11:26:24.970111+0000","last_active":"2024-04-26T11:27:19.240421+0000","last_peered":"2024-04-26T11:27:19.240421+0000","last_clean":"2024-04-26T11:27:19.240421+0000","last_became_active":"2024-04-26T11:26:24.969629+0000","last_became_peered":"2024-04-26T11:26:24.969629+0000","last_unstale":"2024-04-26T11:27:19.240421+0000","last_undegraded":"2024-04-26T11:27:19.240421+0000","last_fullsized":"2024-04-26T11:27:19.240421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:52:07.859419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276856+0000","last_change":"2024-04-26T11:26:29.002986+0000","last_active":"2024-04-26T11:27:19.276856+0000","last_peered":"2024-04-26T11:27:19.276856+0000","last_clean":"2024-04-26T11:27:19.276856+0000","last_became_active":"2024-04-26T11:26:29.002610+0000","last_became_peered":"2024-04-26T11:26:29.002610+0000","last_unstale":"2024-04-26T11:27:19.276856+0000","last_undegraded":"2024-04-26T11:27:19.276856+0000","last_fullsized":"2024-04-26T11:27:19.276856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:04.877356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:22.204951+0000","last_change":"2024-04-26T11:26:24.957825+0000","last_active":"2024-04-26T11:27:22.204951+0000","last_peered":"2024-04-26T11:27:22.204951+0000","last_clean":"2024-04-26T11:27:22.204951+0000","last_became_active":"2024-04-26T11:26:24.957464+0000","last_became_peered":"2024-04-26T11:26:24.957464+0000","last_unstale":"2024-04-26T11:27:22.204951+0000","last_undegraded":"2024-04-26T11:27:22.204951+0000","last_fullsized":"2024-04-26T11:27:22.204951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:06:49.350268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276605+0000","last_change":"2024-04-26T11:26:28.994223+0000","last_active":"2024-04-26T11:27:19.276605+0000","last_peered":"2024-04-26T11:27:19.276605+0000","last_clean":"2024-04-26T11:27:19.276605+0000","last_became_active":"2024-04-26T11:26:28.993388+0000","last_became_peered":"2024-04-26T11:26:28.993388+0000","last_unstale":"2024-04-26T11:27:19.276605+0000","last_undegraded":"2024-04-26T11:27:19.276605+0000","last_fullsized":"2024-04-26T11:27:19.276605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:07:24.208028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245056+0000","last_change":"2024-04-26T11:26:25.078473+0000","last_active":"2024-04-26T11:27:19.245056+0000","last_peered":"2024-04-26T11:27:19.245056+0000","last_clean":"2024-04-26T11:27:19.245056+0000","last_became_active":"2024-04-26T11:26:25.078138+0000","last_became_peered":"2024-04-26T11:26:25.078138+0000","last_unstale":"2024-04-26T11:27:19.245056+0000","last_undegraded":"2024-04-26T11:27:19.245056+0000","last_fullsized":"2024-04-26T11:27:19.245056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:19.692345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240164+0000","last_change":"2024-04-26T11:26:28.998687+0000","last_active":"2024-04-26T11:27:19.240164+0000","last_peered":"2024-04-26T11:27:19.240164+0000","last_clean":"2024-04-26T11:27:19.240164+0000","last_became_active":"2024-04-26T11:26:28.998358+0000","last_became_peered":"2024-04-26T11:26:28.998358+0000","last_unstale":"2024-04-26T11:27:19.240164+0000","last_undegraded":"2024-04-26T11:27:19.240164+0000","last_fullsized":"2024-04-26T11:27:19.240164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:40:51.358188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246101+0000","last_change":"2024-04-26T11:26:24.956213+0000","last_active":"2024-04-26T11:27:19.246101+0000","last_peered":"2024-04-26T11:27:19.246101+0000","last_clean":"2024-04-26T11:27:19.246101+0000","last_became_active":"2024-04-26T11:26:24.955773+0000","last_became_peered":"2024-04-26T11:26:24.955773+0000","last_unstale":"2024-04-26T11:27:19.246101+0000","last_undegraded":"2024-04-26T11:27:19.246101+0000","last_fullsized":"2024-04-26T11:27:19.246101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:38.253367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.839222+0000","last_change":"2024-04-26T11:26:29.001389+0000","last_active":"2024-04-26T11:27:19.839222+0000","last_peered":"2024-04-26T11:27:19.839222+0000","last_clean":"2024-04-26T11:27:19.839222+0000","last_became_active":"2024-04-26T11:26:29.001023+0000","last_became_peered":"2024-04-26T11:26:29.001023+0000","last_unstale":"2024-04-26T11:27:19.839222+0000","last_undegraded":"2024-04-26T11:27:19.839222+0000","last_fullsized":"2024-04-26T11:27:19.839222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:57:23.606518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.893959+0000","last_change":"2024-04-26T11:27:19.893959+0000","last_active":"2024-04-26T11:27:19.893959+0000","last_peered":"2024-04-26T11:27:19.893959+0000","last_clean":"2024-04-26T11:27:19.893959+0000","last_became_active":"2024-04-26T11:27:19.892819+0000","last_became_peered":"2024-04-26T11:27:19.892819+0000","last_unstale":"2024-04-26T11:27:19.893959+0000","last_undegraded":"2024-04-26T11:27:19.893959+0000","last_fullsized":"2024-04-26T11:27:19.893959+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:53:39.829381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242593+0000","last_change":"2024-04-26T11:26:29.019906+0000","last_active":"2024-04-26T11:27:19.242593+0000","last_peered":"2024-04-26T11:27:19.242593+0000","last_clean":"2024-04-26T11:27:19.242593+0000","last_became_active":"2024-04-26T11:26:29.019545+0000","last_became_peered":"2024-04-26T11:26:29.019545+0000","last_unstale":"2024-04-26T11:27:19.242593+0000","last_undegraded":"2024-04-26T11:27:19.242593+0000","last_fullsized":"2024-04-26T11:27:19.242593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:48.617269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276976+0000","last_change":"2024-04-26T11:26:24.956400+0000","last_active":"2024-04-26T11:27:19.276976+0000","last_peered":"2024-04-26T11:27:19.276976+0000","last_clean":"2024-04-26T11:27:19.276976+0000","last_became_active":"2024-04-26T11:26:24.956030+0000","last_became_peered":"2024-04-26T11:26:24.956030+0000","last_unstale":"2024-04-26T11:27:19.276976+0000","last_undegraded":"2024-04-26T11:27:19.276976+0000","last_fullsized":"2024-04-26T11:27:19.276976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:47.841006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242474+0000","last_change":"2024-04-26T11:26:29.013087+0000","last_active":"2024-04-26T11:27:19.242474+0000","last_peered":"2024-04-26T11:27:19.242474+0000","last_clean":"2024-04-26T11:27:19.242474+0000","last_became_active":"2024-04-26T11:26:29.012330+0000","last_became_peered":"2024-04-26T11:26:29.012330+0000","last_unstale":"2024-04-26T11:27:19.242474+0000","last_undegraded":"2024-04-26T11:27:19.242474+0000","last_fullsized":"2024-04-26T11:27:19.242474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:12:50.752329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245410+0000","last_change":"2024-04-26T11:27:19.245410+0000","last_active":"2024-04-26T11:27:19.245410+0000","last_peered":"2024-04-26T11:27:19.245410+0000","last_clean":"2024-04-26T11:27:19.245410+0000","last_became_active":"2024-04-26T11:27:19.244578+0000","last_became_peered":"2024-04-26T11:27:19.244578+0000","last_unstale":"2024-04-26T11:27:19.245410+0000","last_undegraded":"2024-04-26T11:27:19.245410+0000","last_fullsized":"2024-04-26T11:27:19.245410+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:07:50.576496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:21.202051+0000","last_change":"2024-04-26T11:26:25.082517+0000","last_active":"2024-04-26T11:27:21.202051+0000","last_peered":"2024-04-26T11:27:21.202051+0000","last_clean":"2024-04-26T11:27:21.202051+0000","last_became_active":"2024-04-26T11:26:25.081724+0000","last_became_peered":"2024-04-26T11:26:25.081724+0000","last_unstale":"2024-04-26T11:27:21.202051+0000","last_undegraded":"2024-04-26T11:27:21.202051+0000","last_fullsized":"2024-04-26T11:27:21.202051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:59:40.006615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246084+0000","last_change":"2024-04-26T11:26:29.021425+0000","last_active":"2024-04-26T11:27:19.246084+0000","last_peered":"2024-04-26T11:27:19.246084+0000","last_clean":"2024-04-26T11:27:19.246084+0000","last_became_active":"2024-04-26T11:26:29.020881+0000","last_became_peered":"2024-04-26T11:26:29.020881+0000","last_unstale":"2024-04-26T11:27:19.246084+0000","last_undegraded":"2024-04-26T11:27:19.246084+0000","last_fullsized":"2024-04-26T11:27:19.246084+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:18.350323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240518+0000","last_change":"2024-04-26T11:26:24.963614+0000","last_active":"2024-04-26T11:27:19.240518+0000","last_peered":"2024-04-26T11:27:19.240518+0000","last_clean":"2024-04-26T11:27:19.240518+0000","last_became_active":"2024-04-26T11:26:24.963312+0000","last_became_peered":"2024-04-26T11:26:24.963312+0000","last_unstale":"2024-04-26T11:27:19.240518+0000","last_undegraded":"2024-04-26T11:27:19.240518+0000","last_fullsized":"2024-04-26T11:27:19.240518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:55.886637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276763+0000","last_change":"2024-04-26T11:24:54.292436+0000","last_active":"2024-04-26T11:27:19.276763+0000","last_peered":"2024-04-26T11:27:19.276763+0000","last_clean":"2024-04-26T11:27:19.276763+0000","last_became_active":"2024-04-26T11:24:54.291921+0000","last_became_peered":"2024-04-26T11:24:54.291921+0000","last_unstale":"2024-04-26T11:27:19.276763+0000","last_undegraded":"2024-04-26T11:27:19.276763+0000","last_fullsized":"2024-04-26T11:27:19.276763+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_clean_scrub_stamp":"2024-04-26T11:21:00.617277+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:11:01.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.411585+0000","last_change":"2024-04-26T11:26:29.000905+0000","last_active":"2024-04-26T11:27:19.411585+0000","last_peered":"2024-04-26T11:27:19.411585+0000","last_clean":"2024-04-26T11:27:19.411585+0000","last_became_active":"2024-04-26T11:26:29.000517+0000","last_became_peered":"2024-04-26T11:26:29.000517+0000","last_unstale":"2024-04-26T11:27:19.411585+0000","last_undegraded":"2024-04-26T11:27:19.411585+0000","last_fullsized":"2024-04-26T11:27:19.411585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:38:55.394316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277067+0000","last_change":"2024-04-26T11:26:24.957489+0000","last_active":"2024-04-26T11:27:19.277067+0000","last_peered":"2024-04-26T11:27:19.277067+0000","last_clean":"2024-04-26T11:27:19.277067+0000","last_became_active":"2024-04-26T11:26:24.957151+0000","last_became_peered":"2024-04-26T11:26:24.957151+0000","last_unstale":"2024-04-26T11:27:19.277067+0000","last_undegraded":"2024-04-26T11:27:19.277067+0000","last_fullsized":"2024-04-26T11:27:19.277067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:42:25.343105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245684+0000","last_change":"2024-04-26T11:26:25.086767+0000","last_active":"2024-04-26T11:27:19.245684+0000","last_peered":"2024-04-26T11:27:19.245684+0000","last_clean":"2024-04-26T11:27:19.245684+0000","last_became_active":"2024-04-26T11:26:25.086440+0000","last_became_peered":"2024-04-26T11:26:25.086440+0000","last_unstale":"2024-04-26T11:27:19.245684+0000","last_undegraded":"2024-04-26T11:27:19.245684+0000","last_fullsized":"2024-04-26T11:27:19.245684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:02.294644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412245+0000","last_change":"2024-04-26T11:26:29.003683+0000","last_active":"2024-04-26T11:27:19.412245+0000","last_peered":"2024-04-26T11:27:19.412245+0000","last_clean":"2024-04-26T11:27:19.412245+0000","last_became_active":"2024-04-26T11:26:29.003307+0000","last_became_peered":"2024-04-26T11:26:29.003307+0000","last_unstale":"2024-04-26T11:27:19.412245+0000","last_undegraded":"2024-04-26T11:27:19.412245+0000","last_fullsized":"2024-04-26T11:27:19.412245+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:41:00.124540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.851358+0000","last_change":"2024-04-26T11:27:19.849965+0000","last_active":"2024-04-26T11:27:19.851358+0000","last_peered":"2024-04-26T11:27:19.851358+0000","last_clean":"2024-04-26T11:27:19.851358+0000","last_became_active":"2024-04-26T11:27:19.840545+0000","last_became_peered":"2024-04-26T11:27:19.840545+0000","last_unstale":"2024-04-26T11:27:19.851358+0000","last_undegraded":"2024-04-26T11:27:19.851358+0000","last_fullsized":"2024-04-26T11:27:19.851358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:08.003704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246440+0000","last_change":"2024-04-26T11:26:29.016783+0000","last_active":"2024-04-26T11:27:19.246440+0000","last_peered":"2024-04-26T11:27:19.246440+0000","last_clean":"2024-04-26T11:27:19.246440+0000","last_became_active":"2024-04-26T11:26:29.013998+0000","last_became_peered":"2024-04-26T11:26:29.013998+0000","last_unstale":"2024-04-26T11:27:19.246440+0000","last_undegraded":"2024-04-26T11:27:19.246440+0000","last_fullsized":"2024-04-26T11:27:19.246440+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:24:44.453094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240597+0000","last_change":"2024-04-26T11:26:25.902330+0000","last_active":"2024-04-26T11:27:19.240597+0000","last_peered":"2024-04-26T11:27:19.240597+0000","last_clean":"2024-04-26T11:27:19.240597+0000","last_became_active":"2024-04-26T11:26:25.901948+0000","last_became_peered":"2024-04-26T11:26:25.901948+0000","last_unstale":"2024-04-26T11:27:19.240597+0000","last_undegraded":"2024-04-26T11:27:19.240597+0000","last_fullsized":"2024-04-26T11:27:19.240597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:00:23.541393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890462+0000","last_change":"2024-04-26T11:26:29.025692+0000","last_active":"2024-04-26T11:27:19.890462+0000","last_peered":"2024-04-26T11:27:19.890462+0000","last_clean":"2024-04-26T11:27:19.890462+0000","last_became_active":"2024-04-26T11:26:29.025112+0000","last_became_peered":"2024-04-26T11:26:29.025112+0000","last_unstale":"2024-04-26T11:27:19.890462+0000","last_undegraded":"2024-04-26T11:27:19.890462+0000","last_fullsized":"2024-04-26T11:27:19.890462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:39:22.637865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244415+0000","last_change":"2024-04-26T11:26:25.080270+0000","last_active":"2024-04-26T11:27:19.244415+0000","last_peered":"2024-04-26T11:27:19.244415+0000","last_clean":"2024-04-26T11:27:19.244415+0000","last_became_active":"2024-04-26T11:26:25.079895+0000","last_became_peered":"2024-04-26T11:26:25.079895+0000","last_unstale":"2024-04-26T11:27:19.244415+0000","last_undegraded":"2024-04-26T11:27:19.244415+0000","last_fullsized":"2024-04-26T11:27:19.244415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:47:08.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245194+0000","last_change":"2024-04-26T11:26:28.996540+0000","last_active":"2024-04-26T11:27:19.245194+0000","last_peered":"2024-04-26T11:27:19.245194+0000","last_clean":"2024-04-26T11:27:19.245194+0000","last_became_active":"2024-04-26T11:26:28.994724+0000","last_became_peered":"2024-04-26T11:26:28.994724+0000","last_unstale":"2024-04-26T11:27:19.245194+0000","last_undegraded":"2024-04-26T11:27:19.245194+0000","last_fullsized":"2024-04-26T11:27:19.245194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:13.382332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245828+0000","last_change":"2024-04-26T11:26:24.967935+0000","last_active":"2024-04-26T11:27:19.245828+0000","last_peered":"2024-04-26T11:27:19.245828+0000","last_clean":"2024-04-26T11:27:19.245828+0000","last_became_active":"2024-04-26T11:26:24.966465+0000","last_became_peered":"2024-04-26T11:26:24.966465+0000","last_unstale":"2024-04-26T11:27:19.245828+0000","last_undegraded":"2024-04-26T11:27:19.245828+0000","last_fullsized":"2024-04-26T11:27:19.245828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:27:38.817316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838591+0000","last_change":"2024-04-26T11:26:29.024243+0000","last_active":"2024-04-26T11:27:19.838591+0000","last_peered":"2024-04-26T11:27:19.838591+0000","last_clean":"2024-04-26T11:27:19.838591+0000","last_became_active":"2024-04-26T11:26:29.023900+0000","last_became_peered":"2024-04-26T11:26:29.023900+0000","last_unstale":"2024-04-26T11:27:19.838591+0000","last_undegraded":"2024-04-26T11:27:19.838591+0000","last_fullsized":"2024-04-26T11:27:19.838591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:28:23.446568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242062+0000","last_change":"2024-04-26T11:26:24.969953+0000","last_active":"2024-04-26T11:27:19.242062+0000","last_peered":"2024-04-26T11:27:19.242062+0000","last_clean":"2024-04-26T11:27:19.242062+0000","last_became_active":"2024-04-26T11:26:24.969371+0000","last_became_peered":"2024-04-26T11:26:24.969371+0000","last_unstale":"2024-04-26T11:27:19.242062+0000","last_undegraded":"2024-04-26T11:27:19.242062+0000","last_fullsized":"2024-04-26T11:27:19.242062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:07:24.323833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245829+0000","last_change":"2024-04-26T11:26:29.029771+0000","last_active":"2024-04-26T11:27:19.245829+0000","last_peered":"2024-04-26T11:27:19.245829+0000","last_clean":"2024-04-26T11:27:19.245829+0000","last_became_active":"2024-04-26T11:26:29.029387+0000","last_became_peered":"2024-04-26T11:26:29.029387+0000","last_unstale":"2024-04-26T11:27:19.245829+0000","last_undegraded":"2024-04-26T11:27:19.245829+0000","last_fullsized":"2024-04-26T11:27:19.245829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:40.856935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245304+0000","last_change":"2024-04-26T11:26:24.958608+0000","last_active":"2024-04-26T11:27:19.245304+0000","last_peered":"2024-04-26T11:27:19.245304+0000","last_clean":"2024-04-26T11:27:19.245304+0000","last_became_active":"2024-04-26T11:26:24.958188+0000","last_became_peered":"2024-04-26T11:26:24.958188+0000","last_unstale":"2024-04-26T11:27:19.245304+0000","last_undegraded":"2024-04-26T11:27:19.245304+0000","last_fullsized":"2024-04-26T11:27:19.245304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:31:41.442551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277133+0000","last_change":"2024-04-26T11:26:29.000207+0000","last_active":"2024-04-26T11:27:19.277133+0000","last_peered":"2024-04-26T11:27:19.277133+0000","last_clean":"2024-04-26T11:27:19.277133+0000","last_became_active":"2024-04-26T11:26:28.999894+0000","last_became_peered":"2024-04-26T11:26:28.999894+0000","last_unstale":"2024-04-26T11:27:19.277133+0000","last_undegraded":"2024-04-26T11:27:19.277133+0000","last_fullsized":"2024-04-26T11:27:19.277133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:52:10.349511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242602+0000","last_change":"2024-04-26T11:26:25.902036+0000","last_active":"2024-04-26T11:27:19.242602+0000","last_peered":"2024-04-26T11:27:19.242602+0000","last_clean":"2024-04-26T11:27:19.242602+0000","last_became_active":"2024-04-26T11:26:25.901567+0000","last_became_peered":"2024-04-26T11:26:25.901567+0000","last_unstale":"2024-04-26T11:27:19.242602+0000","last_undegraded":"2024-04-26T11:27:19.242602+0000","last_fullsized":"2024-04-26T11:27:19.242602+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:58.757780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277220+0000","last_change":"2024-04-26T11:26:28.998915+0000","last_active":"2024-04-26T11:27:19.277220+0000","last_peered":"2024-04-26T11:27:19.277220+0000","last_clean":"2024-04-26T11:27:19.277220+0000","last_became_active":"2024-04-26T11:26:28.998180+0000","last_became_peered":"2024-04-26T11:26:28.998180+0000","last_unstale":"2024-04-26T11:27:19.277220+0000","last_undegraded":"2024-04-26T11:27:19.277220+0000","last_fullsized":"2024-04-26T11:27:19.277220+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:19.972697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240356+0000","last_change":"2024-04-26T11:26:24.966029+0000","last_active":"2024-04-26T11:27:19.240356+0000","last_peered":"2024-04-26T11:27:19.240356+0000","last_clean":"2024-04-26T11:27:19.240356+0000","last_became_active":"2024-04-26T11:26:24.965676+0000","last_became_peered":"2024-04-26T11:26:24.965676+0000","last_unstale":"2024-04-26T11:27:19.240356+0000","last_undegraded":"2024-04-26T11:27:19.240356+0000","last_fullsized":"2024-04-26T11:27:19.240356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:18:06.904563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276954+0000","last_change":"2024-04-26T11:26:29.020093+0000","last_active":"2024-04-26T11:27:19.276954+0000","last_peered":"2024-04-26T11:27:19.276954+0000","last_clean":"2024-04-26T11:27:19.276954+0000","last_became_active":"2024-04-26T11:26:29.019690+0000","last_became_peered":"2024-04-26T11:26:29.019690+0000","last_unstale":"2024-04-26T11:27:19.276954+0000","last_undegraded":"2024-04-26T11:27:19.276954+0000","last_fullsized":"2024-04-26T11:27:19.276954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:35:44.501217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412931+0000","last_change":"2024-04-26T11:26:24.966757+0000","last_active":"2024-04-26T11:27:19.412931+0000","last_peered":"2024-04-26T11:27:19.412931+0000","last_clean":"2024-04-26T11:27:19.412931+0000","last_became_active":"2024-04-26T11:26:24.966298+0000","last_became_peered":"2024-04-26T11:26:24.966298+0000","last_unstale":"2024-04-26T11:27:19.412931+0000","last_undegraded":"2024-04-26T11:27:19.412931+0000","last_fullsized":"2024-04-26T11:27:19.412931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:17:40.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277300+0000","last_change":"2024-04-26T11:26:28.997774+0000","last_active":"2024-04-26T11:27:19.277300+0000","last_peered":"2024-04-26T11:27:19.277300+0000","last_clean":"2024-04-26T11:27:19.277300+0000","last_became_active":"2024-04-26T11:26:28.997455+0000","last_became_peered":"2024-04-26T11:26:28.997455+0000","last_unstale":"2024-04-26T11:27:19.277300+0000","last_undegraded":"2024-04-26T11:27:19.277300+0000","last_fullsized":"2024-04-26T11:27:19.277300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:56.097022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838639+0000","last_change":"2024-04-26T11:26:28.999714+0000","last_active":"2024-04-26T11:27:19.838639+0000","last_peered":"2024-04-26T11:27:19.838639+0000","last_clean":"2024-04-26T11:27:19.838639+0000","last_became_active":"2024-04-26T11:26:28.999377+0000","last_became_peered":"2024-04-26T11:26:28.999377+0000","last_unstale":"2024-04-26T11:27:19.838639+0000","last_undegraded":"2024-04-26T11:27:19.838639+0000","last_fullsized":"2024-04-26T11:27:19.838639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:39:59.141556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.247154+0000","last_change":"2024-04-26T11:27:19.247154+0000","last_active":"2024-04-26T11:27:19.247154+0000","last_peered":"2024-04-26T11:27:19.247154+0000","last_clean":"2024-04-26T11:27:19.247154+0000","last_became_active":"2024-04-26T11:27:19.246590+0000","last_became_peered":"2024-04-26T11:27:19.246590+0000","last_unstale":"2024-04-26T11:27:19.247154+0000","last_undegraded":"2024-04-26T11:27:19.247154+0000","last_fullsized":"2024-04-26T11:27:19.247154+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:12:58.401699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242036+0000","last_change":"2024-04-26T11:26:29.019983+0000","last_active":"2024-04-26T11:27:19.242036+0000","last_peered":"2024-04-26T11:27:19.242036+0000","last_clean":"2024-04-26T11:27:19.242036+0000","last_became_active":"2024-04-26T11:26:29.019536+0000","last_became_peered":"2024-04-26T11:26:29.019536+0000","last_unstale":"2024-04-26T11:27:19.242036+0000","last_undegraded":"2024-04-26T11:27:19.242036+0000","last_fullsized":"2024-04-26T11:27:19.242036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:15:06.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.889701+0000","last_change":"2024-04-26T11:26:29.027089+0000","last_active":"2024-04-26T11:27:19.889701+0000","last_peered":"2024-04-26T11:27:19.889701+0000","last_clean":"2024-04-26T11:27:19.889701+0000","last_became_active":"2024-04-26T11:26:29.026030+0000","last_became_peered":"2024-04-26T11:26:29.026030+0000","last_unstale":"2024-04-26T11:27:19.889701+0000","last_undegraded":"2024-04-26T11:27:19.889701+0000","last_fullsized":"2024-04-26T11:27:19.889701+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:24:06.105600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412095+0000","last_change":"2024-04-26T11:26:29.018472+0000","last_active":"2024-04-26T11:27:19.412095+0000","last_peered":"2024-04-26T11:27:19.412095+0000","last_clean":"2024-04-26T11:27:19.412095+0000","last_became_active":"2024-04-26T11:26:29.015947+0000","last_became_peered":"2024-04-26T11:26:29.015947+0000","last_unstale":"2024-04-26T11:27:19.412095+0000","last_undegraded":"2024-04-26T11:27:19.412095+0000","last_fullsized":"2024-04-26T11:27:19.412095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:35:48.082709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838871+0000","last_change":"2024-04-26T11:26:28.997582+0000","last_active":"2024-04-26T11:27:19.838871+0000","last_peered":"2024-04-26T11:27:19.838871+0000","last_clean":"2024-04-26T11:27:19.838871+0000","last_became_active":"2024-04-26T11:26:28.997202+0000","last_became_peered":"2024-04-26T11:26:28.997202+0000","last_unstale":"2024-04-26T11:27:19.838871+0000","last_undegraded":"2024-04-26T11:27:19.838871+0000","last_fullsized":"2024-04-26T11:27:19.838871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:59.776455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242635+0000","last_change":"2024-04-26T11:26:28.999428+0000","last_active":"2024-04-26T11:27:19.242635+0000","last_peered":"2024-04-26T11:27:19.242635+0000","last_clean":"2024-04-26T11:27:19.242635+0000","last_became_active":"2024-04-26T11:26:28.999093+0000","last_became_peered":"2024-04-26T11:26:28.999093+0000","last_unstale":"2024-04-26T11:27:19.242635+0000","last_undegraded":"2024-04-26T11:27:19.242635+0000","last_fullsized":"2024-04-26T11:27:19.242635+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:07.222231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246478+0000","last_change":"2024-04-26T11:26:29.019673+0000","last_active":"2024-04-26T11:27:19.246478+0000","last_peered":"2024-04-26T11:27:19.246478+0000","last_clean":"2024-04-26T11:27:19.246478+0000","last_became_active":"2024-04-26T11:26:29.017289+0000","last_became_peered":"2024-04-26T11:26:29.017289+0000","last_unstale":"2024-04-26T11:27:19.246478+0000","last_undegraded":"2024-04-26T11:27:19.246478+0000","last_fullsized":"2024-04-26T11:27:19.246478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:35.848004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240127+0000","last_change":"2024-04-26T11:26:29.017234+0000","last_active":"2024-04-26T11:27:19.240127+0000","last_peered":"2024-04-26T11:27:19.240127+0000","last_clean":"2024-04-26T11:27:19.240127+0000","last_became_active":"2024-04-26T11:26:29.016863+0000","last_became_peered":"2024-04-26T11:26:29.016863+0000","last_unstale":"2024-04-26T11:27:19.240127+0000","last_undegraded":"2024-04-26T11:27:19.240127+0000","last_fullsized":"2024-04-26T11:27:19.240127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:14:31.136286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245593+0000","last_change":"2024-04-26T11:26:29.018122+0000","last_active":"2024-04-26T11:27:19.245593+0000","last_peered":"2024-04-26T11:27:19.245593+0000","last_clean":"2024-04-26T11:27:19.245593+0000","last_became_active":"2024-04-26T11:26:29.016269+0000","last_became_peered":"2024-04-26T11:26:29.016269+0000","last_unstale":"2024-04-26T11:27:19.245593+0000","last_undegraded":"2024-04-26T11:27:19.245593+0000","last_fullsized":"2024-04-26T11:27:19.245593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:22.135086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.278596+0000","last_change":"2024-04-26T11:27:19.278596+0000","last_active":"2024-04-26T11:27:19.278596+0000","last_peered":"2024-04-26T11:27:19.278596+0000","last_clean":"2024-04-26T11:27:19.278596+0000","last_became_active":"2024-04-26T11:27:19.278138+0000","last_became_peered":"2024-04-26T11:27:19.278138+0000","last_unstale":"2024-04-26T11:27:19.278596+0000","last_undegraded":"2024-04-26T11:27:19.278596+0000","last_fullsized":"2024-04-26T11:27:19.278596+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:17.038527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412133+0000","last_change":"2024-04-26T11:26:29.026905+0000","last_active":"2024-04-26T11:27:19.412133+0000","last_peered":"2024-04-26T11:27:19.412133+0000","last_clean":"2024-04-26T11:27:19.412133+0000","last_became_active":"2024-04-26T11:26:29.026539+0000","last_became_peered":"2024-04-26T11:26:29.026539+0000","last_unstale":"2024-04-26T11:27:19.412133+0000","last_undegraded":"2024-04-26T11:27:19.412133+0000","last_fullsized":"2024-04-26T11:27:19.412133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:41:12.169466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838812+0000","last_change":"2024-04-26T11:26:28.999953+0000","last_active":"2024-04-26T11:27:19.838812+0000","last_peered":"2024-04-26T11:27:19.838812+0000","last_clean":"2024-04-26T11:27:19.838812+0000","last_became_active":"2024-04-26T11:26:28.998344+0000","last_became_peered":"2024-04-26T11:26:28.998344+0000","last_unstale":"2024-04-26T11:27:19.838812+0000","last_undegraded":"2024-04-26T11:27:19.838812+0000","last_fullsized":"2024-04-26T11:27:19.838812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:30:53.269171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281068,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67154,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":217,"num_read_kb":180,"num_write":168,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":192752,"data_compressed":5472,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383706,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":479968,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32280,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712872,"statfs":{"total":95995035648,"available":95961980928,"internally_reserved":0,"allocated":1130496,"data_stored":1134993,"data_compressed":43361,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480204,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070509,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":480124,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332145,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":1072054,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495671,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":479593,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":533829,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":480864,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":543889,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":479904,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":480421,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757309,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":1071347,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":54443,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64503,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65122,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T11:27:28.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T11:27:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:28 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:28 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:28 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:29.034 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:29 smithi033 ceph-mon[27845]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:27:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:29 smithi033 ceph-mon[27845]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:27:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:27:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:29 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:29 smithi096 ceph-mon[32753]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:27:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:29 smithi096 ceph-mon[32753]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:27:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:27:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:29 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:29 smithi194 ceph-mon[33017]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:27:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:29 smithi194 ceph-mon[33017]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:27:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:27:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:29 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 -- 172.21.15.33:0/799155901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 msgr2=0x7f23a006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 --2- 172.21.15.33:0/799155901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 0x7f23a006b590 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2388009990 tx=0x7f2388009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 -- 172.21.15.33:0/799155901 shutdown_connections 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 --2- 172.21.15.33:0/799155901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 0x7f23a01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 --2- 172.21.15.33:0/799155901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a01094a0 0x7f23a0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 --2- 172.21.15.33:0/799155901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 0x7f23a006b590 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 -- 172.21.15.33:0/799155901 >> 172.21.15.33:0/799155901 conn(0x7f23a00792d0 msgr2=0x7f23a00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:31.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 -- 172.21.15.33:0/799155901 shutdown_connections 2024-04-26T11:27:31.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.232+0000 7f23a7200700 1 -- 172.21.15.33:0/799155901 wait complete. 2024-04-26T11:27:31.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 Processor -- start 2024-04-26T11:27:31.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 -- start start 2024-04-26T11:27:31.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 0x7f23a01023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:31.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 0x7f23a0102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:31.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a01094a0 0x7f23a0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:31.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f23a0113600 con 0x7f23a006b130 2024-04-26T11:27:31.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f23a0113480 con 0x7f23a01094a0 2024-04-26T11:27:31.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.234+0000 7f23a7200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23a0113780 con 0x7f23a006bad0 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f239fe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 0x7f23a01023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 0x7f23a0102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 0x7f23a0102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:33444/0 (socket says 172.21.15.33:33444) 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 -- 172.21.15.33:0/1128578132 learned_addr learned my addr 172.21.15.33:0/1128578132 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f23a5200700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a01094a0 0x7f23a0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 -- 172.21.15.33:0/1128578132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a01094a0 msgr2=0x7f23a0102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a01094a0 0x7f23a0102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 -- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 msgr2=0x7f23a01023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 0x7f23a01023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.235+0000 7f2397400700 1 -- 172.21.15.33:0/1128578132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a01a8050 con 0x7f23a006bad0 2024-04-26T11:27:31.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.236+0000 7f2397400700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 0x7f23a0102930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2388000c00 tx=0x7f23880057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:31.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.236+0000 7f239d600700 1 -- 172.21.15.33:0/1128578132 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388027070 con 0x7f23a006bad0 2024-04-26T11:27:31.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.236+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23a01a82e0 con 0x7f23a006bad0 2024-04-26T11:27:31.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.236+0000 7f239d600700 1 -- 172.21.15.33:0/1128578132 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2388022470 con 0x7f23a006bad0 2024-04-26T11:27:31.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.236+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23a01a8790 con 0x7f23a006bad0 2024-04-26T11:27:31.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.236+0000 7f239d600700 1 -- 172.21.15.33:0/1128578132 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388017650 con 0x7f23a006bad0 2024-04-26T11:27:31.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.239+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23a006a110 con 0x7f23a006bad0 2024-04-26T11:27:31.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.240+0000 7f239d600700 1 -- 172.21.15.33:0/1128578132 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f238802a070 con 0x7f23a006bad0 2024-04-26T11:27:31.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.241+0000 7f239d600700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23840a3d50 0x7f23840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:31.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.241+0000 7f239d600700 1 -- 172.21.15.33:0/1128578132 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f23880d12f0 con 0x7f23a006bad0 2024-04-26T11:27:31.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.242+0000 7f239fe00700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23840a3d50 0x7f23840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:31.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.242+0000 7f239fe00700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23840a3d50 0x7f23840a6210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2390009680 tx=0x7f2390009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:31.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.245+0000 7f239d600700 1 -- 172.21.15.33:0/1128578132 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2388013070 con 0x7f23a006bad0 2024-04-26T11:27:31.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.478+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f23a0063860 con 0x7f23840a3d50 2024-04-26T11:27:31.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.495+0000 7f239d600700 1 -- 172.21.15.33:0/1128578132 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259011 (secure 0 0 0) 0x7f23a0063860 con 0x7f23840a3d50 2024-04-26T11:27:31.499 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:27:31.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23840a3d50 msgr2=0x7f23840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:31.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23840a3d50 0x7f23840a6210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2390009680 tx=0x7f2390009260 comp rx=0 tx=0).stop 2024-04-26T11:27:31.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 msgr2=0x7f23a0102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:31.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 0x7f23a0102930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2388000c00 tx=0x7f23880057c0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 shutdown_connections 2024-04-26T11:27:31.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f23840a3d50 0x7f23840a6210 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a01094a0 0x7f23a0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23a006b130 0x7f23a01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 --2- 172.21.15.33:0/1128578132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23a006bad0 0x7f23a0102930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:31.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 >> 172.21.15.33:0/1128578132 conn(0x7f23a00792d0 msgr2=0x7f23a0100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:31.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.500+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 shutdown_connections 2024-04-26T11:27:31.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:31.501+0000 7f23a7200700 1 -- 172.21.15.33:0/1128578132 wait complete. 2024-04-26T11:27:31.505 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-26T11:27:31.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:31 smithi033 ceph-mon[27845]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:31.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:31 smithi194 ceph-mon[33017]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:31 smithi096 ceph-mon[32753]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:32.312 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-04-26T11:27:29.829279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657756,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":368,"num_read_kb":304,"num_write":351,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357052,"kb_used_data":12000,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584772,"statfs":{"total":1151940427776,"available":1151574806528,"internally_reserved":0,"allocated":12288000,"data_stored":7725900,"data_compressed":182166,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17566,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":48,"num_write":8,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.590774"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.839113+0000","last_change":"2024-04-26T11:26:29.018818+0000","last_active":"2024-04-26T11:27:19.839113+0000","last_peered":"2024-04-26T11:27:19.839113+0000","last_clean":"2024-04-26T11:27:19.839113+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:27:19.839113+0000","last_undegraded":"2024-04-26T11:27:19.839113+0000","last_fullsized":"2024-04-26T11:27:19.839113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:05.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890260+0000","last_change":"2024-04-26T11:26:29.021338+0000","last_active":"2024-04-26T11:27:19.890260+0000","last_peered":"2024-04-26T11:27:19.890260+0000","last_clean":"2024-04-26T11:27:19.890260+0000","last_became_active":"2024-04-26T11:26:29.021019+0000","last_became_peered":"2024-04-26T11:26:29.021019+0000","last_unstale":"2024-04-26T11:27:19.890260+0000","last_undegraded":"2024-04-26T11:27:19.890260+0000","last_fullsized":"2024-04-26T11:27:19.890260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:52:44.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890073+0000","last_change":"2024-04-26T11:26:29.014166+0000","last_active":"2024-04-26T11:27:19.890073+0000","last_peered":"2024-04-26T11:27:19.890073+0000","last_clean":"2024-04-26T11:27:19.890073+0000","last_became_active":"2024-04-26T11:26:29.013812+0000","last_became_peered":"2024-04-26T11:26:29.013812+0000","last_unstale":"2024-04-26T11:27:19.890073+0000","last_undegraded":"2024-04-26T11:27:19.890073+0000","last_fullsized":"2024-04-26T11:27:19.890073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:08.683633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240873+0000","last_change":"2024-04-26T11:26:28.999249+0000","last_active":"2024-04-26T11:27:19.240873+0000","last_peered":"2024-04-26T11:27:19.240873+0000","last_clean":"2024-04-26T11:27:19.240873+0000","last_became_active":"2024-04-26T11:26:28.998927+0000","last_became_peered":"2024-04-26T11:26:28.998927+0000","last_unstale":"2024-04-26T11:27:19.240873+0000","last_undegraded":"2024-04-26T11:27:19.240873+0000","last_fullsized":"2024-04-26T11:27:19.240873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:56.674537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245154+0000","last_change":"2024-04-26T11:26:29.017202+0000","last_active":"2024-04-26T11:27:19.245154+0000","last_peered":"2024-04-26T11:27:19.245154+0000","last_clean":"2024-04-26T11:27:19.245154+0000","last_became_active":"2024-04-26T11:26:29.016849+0000","last_became_peered":"2024-04-26T11:26:29.016849+0000","last_unstale":"2024-04-26T11:27:19.245154+0000","last_undegraded":"2024-04-26T11:27:19.245154+0000","last_fullsized":"2024-04-26T11:27:19.245154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:58:22.164118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.278294+0000","last_change":"2024-04-26T11:27:19.278294+0000","last_active":"2024-04-26T11:27:19.278294+0000","last_peered":"2024-04-26T11:27:19.278294+0000","last_clean":"2024-04-26T11:27:19.278294+0000","last_became_active":"2024-04-26T11:27:19.277853+0000","last_became_peered":"2024-04-26T11:27:19.277853+0000","last_unstale":"2024-04-26T11:27:19.278294+0000","last_undegraded":"2024-04-26T11:27:19.278294+0000","last_fullsized":"2024-04-26T11:27:19.278294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:12:18.733166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276702+0000","last_change":"2024-04-26T11:26:28.991406+0000","last_active":"2024-04-26T11:27:19.276702+0000","last_peered":"2024-04-26T11:27:19.276702+0000","last_clean":"2024-04-26T11:27:19.276702+0000","last_became_active":"2024-04-26T11:26:28.989303+0000","last_became_peered":"2024-04-26T11:26:28.989303+0000","last_unstale":"2024-04-26T11:27:19.276702+0000","last_undegraded":"2024-04-26T11:27:19.276702+0000","last_fullsized":"2024-04-26T11:27:19.276702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:42:59.449078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245298+0000","last_change":"2024-04-26T11:26:29.020922+0000","last_active":"2024-04-26T11:27:19.245298+0000","last_peered":"2024-04-26T11:27:19.245298+0000","last_clean":"2024-04-26T11:27:19.245298+0000","last_became_active":"2024-04-26T11:26:29.020576+0000","last_became_peered":"2024-04-26T11:26:29.020576+0000","last_unstale":"2024-04-26T11:27:19.245298+0000","last_undegraded":"2024-04-26T11:27:19.245298+0000","last_fullsized":"2024-04-26T11:27:19.245298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:48.770319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245000+0000","last_change":"2024-04-26T11:26:29.004171+0000","last_active":"2024-04-26T11:27:19.245000+0000","last_peered":"2024-04-26T11:27:19.245000+0000","last_clean":"2024-04-26T11:27:19.245000+0000","last_became_active":"2024-04-26T11:26:29.003816+0000","last_became_peered":"2024-04-26T11:26:29.003816+0000","last_unstale":"2024-04-26T11:27:19.245000+0000","last_undegraded":"2024-04-26T11:27:19.245000+0000","last_fullsized":"2024-04-26T11:27:19.245000+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:45:14.807634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242873+0000","last_change":"2024-04-26T11:26:28.999453+0000","last_active":"2024-04-26T11:27:19.242873+0000","last_peered":"2024-04-26T11:27:19.242873+0000","last_clean":"2024-04-26T11:27:19.242873+0000","last_became_active":"2024-04-26T11:26:28.999104+0000","last_became_peered":"2024-04-26T11:26:28.999104+0000","last_unstale":"2024-04-26T11:27:19.242873+0000","last_undegraded":"2024-04-26T11:27:19.242873+0000","last_fullsized":"2024-04-26T11:27:19.242873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:55.408949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277046+0000","last_change":"2024-04-26T11:26:28.999218+0000","last_active":"2024-04-26T11:27:19.277046+0000","last_peered":"2024-04-26T11:27:19.277046+0000","last_clean":"2024-04-26T11:27:19.277046+0000","last_became_active":"2024-04-26T11:26:28.998584+0000","last_became_peered":"2024-04-26T11:26:28.998584+0000","last_unstale":"2024-04-26T11:27:19.277046+0000","last_undegraded":"2024-04-26T11:27:19.277046+0000","last_fullsized":"2024-04-26T11:27:19.277046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:18.040520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240742+0000","last_change":"2024-04-26T11:26:28.997121+0000","last_active":"2024-04-26T11:27:19.240742+0000","last_peered":"2024-04-26T11:27:19.240742+0000","last_clean":"2024-04-26T11:27:19.240742+0000","last_became_active":"2024-04-26T11:26:28.996556+0000","last_became_peered":"2024-04-26T11:26:28.996556+0000","last_unstale":"2024-04-26T11:27:19.240742+0000","last_undegraded":"2024-04-26T11:27:19.240742+0000","last_fullsized":"2024-04-26T11:27:19.240742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:18:55.251371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412020+0000","last_change":"2024-04-26T11:26:28.999525+0000","last_active":"2024-04-26T11:27:19.412020+0000","last_peered":"2024-04-26T11:27:19.412020+0000","last_clean":"2024-04-26T11:27:19.412020+0000","last_became_active":"2024-04-26T11:26:28.998578+0000","last_became_peered":"2024-04-26T11:26:28.998578+0000","last_unstale":"2024-04-26T11:27:19.412020+0000","last_undegraded":"2024-04-26T11:27:19.412020+0000","last_fullsized":"2024-04-26T11:27:19.412020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:32:35.523804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.889408+0000","last_change":"2024-04-26T11:26:29.008427+0000","last_active":"2024-04-26T11:27:19.889408+0000","last_peered":"2024-04-26T11:27:19.889408+0000","last_clean":"2024-04-26T11:27:19.889408+0000","last_became_active":"2024-04-26T11:26:29.008073+0000","last_became_peered":"2024-04-26T11:26:29.008073+0000","last_unstale":"2024-04-26T11:27:19.889408+0000","last_undegraded":"2024-04-26T11:27:19.889408+0000","last_fullsized":"2024-04-26T11:27:19.889408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:03.185941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.889360+0000","last_change":"2024-04-26T11:26:29.015136+0000","last_active":"2024-04-26T11:27:19.889360+0000","last_peered":"2024-04-26T11:27:19.889360+0000","last_clean":"2024-04-26T11:27:19.889360+0000","last_became_active":"2024-04-26T11:26:29.014800+0000","last_became_peered":"2024-04-26T11:26:29.014800+0000","last_unstale":"2024-04-26T11:27:19.889360+0000","last_undegraded":"2024-04-26T11:27:19.889360+0000","last_fullsized":"2024-04-26T11:27:19.889360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:28:35.871046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244404+0000","last_change":"2024-04-26T11:26:29.024795+0000","last_active":"2024-04-26T11:27:19.244404+0000","last_peered":"2024-04-26T11:27:19.244404+0000","last_clean":"2024-04-26T11:27:19.244404+0000","last_became_active":"2024-04-26T11:26:29.024430+0000","last_became_peered":"2024-04-26T11:26:29.024430+0000","last_unstale":"2024-04-26T11:27:19.244404+0000","last_undegraded":"2024-04-26T11:27:19.244404+0000","last_fullsized":"2024-04-26T11:27:19.244404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:05:20.172869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.839207+0000","last_change":"2024-04-26T11:26:29.018822+0000","last_active":"2024-04-26T11:27:19.839207+0000","last_peered":"2024-04-26T11:27:19.839207+0000","last_clean":"2024-04-26T11:27:19.839207+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:27:19.839207+0000","last_undegraded":"2024-04-26T11:27:19.839207+0000","last_fullsized":"2024-04-26T11:27:19.839207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:04.475400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242166+0000","last_change":"2024-04-26T11:26:29.001984+0000","last_active":"2024-04-26T11:27:19.242166+0000","last_peered":"2024-04-26T11:27:19.242166+0000","last_clean":"2024-04-26T11:27:19.242166+0000","last_became_active":"2024-04-26T11:26:28.999492+0000","last_became_peered":"2024-04-26T11:26:28.999492+0000","last_unstale":"2024-04-26T11:27:19.242166+0000","last_undegraded":"2024-04-26T11:27:19.242166+0000","last_fullsized":"2024-04-26T11:27:19.242166+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:21.398823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242387+0000","last_change":"2024-04-26T11:26:29.019677+0000","last_active":"2024-04-26T11:27:19.242387+0000","last_peered":"2024-04-26T11:27:19.242387+0000","last_clean":"2024-04-26T11:27:19.242387+0000","last_became_active":"2024-04-26T11:26:29.019315+0000","last_became_peered":"2024-04-26T11:26:29.019315+0000","last_unstale":"2024-04-26T11:27:19.242387+0000","last_undegraded":"2024-04-26T11:27:19.242387+0000","last_fullsized":"2024-04-26T11:27:19.242387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:20.239140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242574+0000","last_change":"2024-04-26T11:26:24.964887+0000","last_active":"2024-04-26T11:27:19.242574+0000","last_peered":"2024-04-26T11:27:19.242574+0000","last_clean":"2024-04-26T11:27:19.242574+0000","last_became_active":"2024-04-26T11:26:24.964380+0000","last_became_peered":"2024-04-26T11:26:24.964380+0000","last_unstale":"2024-04-26T11:27:19.242574+0000","last_undegraded":"2024-04-26T11:27:19.242574+0000","last_fullsized":"2024-04-26T11:27:19.242574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:59:41.347542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244889+0000","last_change":"2024-04-26T11:27:19.244889+0000","last_active":"2024-04-26T11:27:19.244889+0000","last_peered":"2024-04-26T11:27:19.244889+0000","last_clean":"2024-04-26T11:27:19.244889+0000","last_became_active":"2024-04-26T11:27:19.244428+0000","last_became_peered":"2024-04-26T11:27:19.244428+0000","last_unstale":"2024-04-26T11:27:19.244889+0000","last_undegraded":"2024-04-26T11:27:19.244889+0000","last_fullsized":"2024-04-26T11:27:19.244889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:09:39.366242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246839+0000","last_change":"2024-04-26T11:26:25.082741+0000","last_active":"2024-04-26T11:27:19.246839+0000","last_peered":"2024-04-26T11:27:19.246839+0000","last_clean":"2024-04-26T11:27:19.246839+0000","last_became_active":"2024-04-26T11:26:25.082361+0000","last_became_peered":"2024-04-26T11:26:25.082361+0000","last_unstale":"2024-04-26T11:27:19.246839+0000","last_undegraded":"2024-04-26T11:27:19.246839+0000","last_fullsized":"2024-04-26T11:27:19.246839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:23:17.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.891043+0000","last_change":"2024-04-26T11:26:29.014817+0000","last_active":"2024-04-26T11:27:19.891043+0000","last_peered":"2024-04-26T11:27:19.891043+0000","last_clean":"2024-04-26T11:27:19.891043+0000","last_became_active":"2024-04-26T11:26:29.014419+0000","last_became_peered":"2024-04-26T11:26:29.014419+0000","last_unstale":"2024-04-26T11:27:19.891043+0000","last_undegraded":"2024-04-26T11:27:19.891043+0000","last_fullsized":"2024-04-26T11:27:19.891043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:33:28.355075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245880+0000","last_change":"2024-04-26T11:26:24.960224+0000","last_active":"2024-04-26T11:27:19.245880+0000","last_peered":"2024-04-26T11:27:19.245880+0000","last_clean":"2024-04-26T11:27:19.245880+0000","last_became_active":"2024-04-26T11:26:24.959848+0000","last_became_peered":"2024-04-26T11:26:24.959848+0000","last_unstale":"2024-04-26T11:27:19.245880+0000","last_undegraded":"2024-04-26T11:27:19.245880+0000","last_fullsized":"2024-04-26T11:27:19.245880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:49:10.932187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244563+0000","last_change":"2024-04-26T11:26:29.001946+0000","last_active":"2024-04-26T11:27:19.244563+0000","last_peered":"2024-04-26T11:27:19.244563+0000","last_clean":"2024-04-26T11:27:19.244563+0000","last_became_active":"2024-04-26T11:26:29.001579+0000","last_became_peered":"2024-04-26T11:26:29.001579+0000","last_unstale":"2024-04-26T11:27:19.244563+0000","last_undegraded":"2024-04-26T11:27:19.244563+0000","last_fullsized":"2024-04-26T11:27:19.244563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:14:43.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245485+0000","last_change":"2024-04-26T11:26:24.956148+0000","last_active":"2024-04-26T11:27:19.245485+0000","last_peered":"2024-04-26T11:27:19.245485+0000","last_clean":"2024-04-26T11:27:19.245485+0000","last_became_active":"2024-04-26T11:26:24.955736+0000","last_became_peered":"2024-04-26T11:26:24.955736+0000","last_unstale":"2024-04-26T11:27:19.245485+0000","last_undegraded":"2024-04-26T11:27:19.245485+0000","last_fullsized":"2024-04-26T11:27:19.245485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:43.705470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.415134+0000","last_change":"2024-04-26T11:27:19.415134+0000","last_active":"2024-04-26T11:27:19.415134+0000","last_peered":"2024-04-26T11:27:19.415134+0000","last_clean":"2024-04-26T11:27:19.415134+0000","last_became_active":"2024-04-26T11:27:19.414703+0000","last_became_peered":"2024-04-26T11:27:19.414703+0000","last_unstale":"2024-04-26T11:27:19.415134+0000","last_undegraded":"2024-04-26T11:27:19.415134+0000","last_fullsized":"2024-04-26T11:27:19.415134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:41:11.268296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:26.189647+0000","last_change":"2024-04-26T11:26:24.958350+0000","last_active":"2024-04-26T11:27:26.189647+0000","last_peered":"2024-04-26T11:27:26.189647+0000","last_clean":"2024-04-26T11:27:26.189647+0000","last_became_active":"2024-04-26T11:26:24.957382+0000","last_became_peered":"2024-04-26T11:26:24.957382+0000","last_unstale":"2024-04-26T11:27:26.189647+0000","last_undegraded":"2024-04-26T11:27:26.189647+0000","last_fullsized":"2024-04-26T11:27:26.189647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:03.854227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245438+0000","last_change":"2024-04-26T11:26:28.997568+0000","last_active":"2024-04-26T11:27:19.245438+0000","last_peered":"2024-04-26T11:27:19.245438+0000","last_clean":"2024-04-26T11:27:19.245438+0000","last_became_active":"2024-04-26T11:26:28.997161+0000","last_became_peered":"2024-04-26T11:26:28.997161+0000","last_unstale":"2024-04-26T11:27:19.245438+0000","last_undegraded":"2024-04-26T11:27:19.245438+0000","last_fullsized":"2024-04-26T11:27:19.245438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:04.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.243026+0000","last_change":"2024-04-26T11:26:24.958093+0000","last_active":"2024-04-26T11:27:19.243026+0000","last_peered":"2024-04-26T11:27:19.243026+0000","last_clean":"2024-04-26T11:27:19.243026+0000","last_became_active":"2024-04-26T11:26:24.957730+0000","last_became_peered":"2024-04-26T11:26:24.957730+0000","last_unstale":"2024-04-26T11:27:19.243026+0000","last_undegraded":"2024-04-26T11:27:19.243026+0000","last_fullsized":"2024-04-26T11:27:19.243026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:09:29.827063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.248406+0000","last_change":"2024-04-26T11:27:19.248406+0000","last_active":"2024-04-26T11:27:19.248406+0000","last_peered":"2024-04-26T11:27:19.248406+0000","last_clean":"2024-04-26T11:27:19.248406+0000","last_became_active":"2024-04-26T11:27:19.247980+0000","last_became_peered":"2024-04-26T11:27:19.247980+0000","last_unstale":"2024-04-26T11:27:19.248406+0000","last_undegraded":"2024-04-26T11:27:19.248406+0000","last_fullsized":"2024-04-26T11:27:19.248406+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:51:34.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.411996+0000","last_change":"2024-04-26T11:26:24.963463+0000","last_active":"2024-04-26T11:27:19.411996+0000","last_peered":"2024-04-26T11:27:19.411996+0000","last_clean":"2024-04-26T11:27:19.411996+0000","last_became_active":"2024-04-26T11:26:24.963030+0000","last_became_peered":"2024-04-26T11:26:24.963030+0000","last_unstale":"2024-04-26T11:27:19.411996+0000","last_undegraded":"2024-04-26T11:27:19.411996+0000","last_fullsized":"2024-04-26T11:27:19.411996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:04:17.844500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245456+0000","last_change":"2024-04-26T11:26:29.013899+0000","last_active":"2024-04-26T11:27:19.245456+0000","last_peered":"2024-04-26T11:27:19.245456+0000","last_clean":"2024-04-26T11:27:19.245456+0000","last_became_active":"2024-04-26T11:26:29.012674+0000","last_became_peered":"2024-04-26T11:26:29.012674+0000","last_unstale":"2024-04-26T11:27:19.245456+0000","last_undegraded":"2024-04-26T11:27:19.245456+0000","last_fullsized":"2024-04-26T11:27:19.245456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:33:46.259841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240050+0000","last_change":"2024-04-26T11:26:24.957803+0000","last_active":"2024-04-26T11:27:19.240050+0000","last_peered":"2024-04-26T11:27:19.240050+0000","last_clean":"2024-04-26T11:27:19.240050+0000","last_became_active":"2024-04-26T11:26:24.957496+0000","last_became_peered":"2024-04-26T11:26:24.957496+0000","last_unstale":"2024-04-26T11:27:19.240050+0000","last_undegraded":"2024-04-26T11:27:19.240050+0000","last_fullsized":"2024-04-26T11:27:19.240050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:15.497106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240308+0000","last_change":"2024-04-26T11:26:29.021648+0000","last_active":"2024-04-26T11:27:19.240308+0000","last_peered":"2024-04-26T11:27:19.240308+0000","last_clean":"2024-04-26T11:27:19.240308+0000","last_became_active":"2024-04-26T11:26:29.020689+0000","last_became_peered":"2024-04-26T11:26:29.020689+0000","last_unstale":"2024-04-26T11:27:19.240308+0000","last_undegraded":"2024-04-26T11:27:19.240308+0000","last_fullsized":"2024-04-26T11:27:19.240308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:15:42.321851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:24.186866+0000","last_change":"2024-04-26T11:26:24.966184+0000","last_active":"2024-04-26T11:27:24.186866+0000","last_peered":"2024-04-26T11:27:24.186866+0000","last_clean":"2024-04-26T11:27:24.186866+0000","last_became_active":"2024-04-26T11:26:24.965850+0000","last_became_peered":"2024-04-26T11:26:24.965850+0000","last_unstale":"2024-04-26T11:27:24.186866+0000","last_undegraded":"2024-04-26T11:27:24.186866+0000","last_fullsized":"2024-04-26T11:27:24.186866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:49.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244856+0000","last_change":"2024-04-26T11:26:29.003570+0000","last_active":"2024-04-26T11:27:19.244856+0000","last_peered":"2024-04-26T11:27:19.244856+0000","last_clean":"2024-04-26T11:27:19.244856+0000","last_became_active":"2024-04-26T11:26:29.003039+0000","last_became_peered":"2024-04-26T11:26:29.003039+0000","last_unstale":"2024-04-26T11:27:19.244856+0000","last_undegraded":"2024-04-26T11:27:19.244856+0000","last_fullsized":"2024-04-26T11:27:19.244856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:29:16.166432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277109+0000","last_change":"2024-04-26T11:26:24.954968+0000","last_active":"2024-04-26T11:27:19.277109+0000","last_peered":"2024-04-26T11:27:19.277109+0000","last_clean":"2024-04-26T11:27:19.277109+0000","last_became_active":"2024-04-26T11:26:24.954595+0000","last_became_peered":"2024-04-26T11:26:24.954595+0000","last_unstale":"2024-04-26T11:27:19.277109+0000","last_undegraded":"2024-04-26T11:27:19.277109+0000","last_fullsized":"2024-04-26T11:27:19.277109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:52:21.988942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890791+0000","last_change":"2024-04-26T11:26:29.015601+0000","last_active":"2024-04-26T11:27:19.890791+0000","last_peered":"2024-04-26T11:27:19.890791+0000","last_clean":"2024-04-26T11:27:19.890791+0000","last_became_active":"2024-04-26T11:26:29.015199+0000","last_became_peered":"2024-04-26T11:26:29.015199+0000","last_unstale":"2024-04-26T11:27:19.890791+0000","last_undegraded":"2024-04-26T11:27:19.890791+0000","last_fullsized":"2024-04-26T11:27:19.890791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:15:04.573214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240544+0000","last_change":"2024-04-26T11:26:25.085542+0000","last_active":"2024-04-26T11:27:19.240544+0000","last_peered":"2024-04-26T11:27:19.240544+0000","last_clean":"2024-04-26T11:27:19.240544+0000","last_became_active":"2024-04-26T11:26:25.085230+0000","last_became_peered":"2024-04-26T11:26:25.085230+0000","last_unstale":"2024-04-26T11:27:19.240544+0000","last_undegraded":"2024-04-26T11:27:19.240544+0000","last_fullsized":"2024-04-26T11:27:19.240544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:51.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244661+0000","last_change":"2024-04-26T11:26:29.003954+0000","last_active":"2024-04-26T11:27:19.244661+0000","last_peered":"2024-04-26T11:27:19.244661+0000","last_clean":"2024-04-26T11:27:19.244661+0000","last_became_active":"2024-04-26T11:26:29.003576+0000","last_became_peered":"2024-04-26T11:26:29.003576+0000","last_unstale":"2024-04-26T11:27:19.244661+0000","last_undegraded":"2024-04-26T11:27:19.244661+0000","last_fullsized":"2024-04-26T11:27:19.244661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:24.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":130,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:25.188093+0000","last_change":"2024-04-26T11:26:24.957823+0000","last_active":"2024-04-26T11:27:25.188093+0000","last_peered":"2024-04-26T11:27:25.188093+0000","last_clean":"2024-04-26T11:27:25.188093+0000","last_became_active":"2024-04-26T11:26:24.957434+0000","last_became_peered":"2024-04-26T11:26:24.957434+0000","last_unstale":"2024-04-26T11:27:25.188093+0000","last_undegraded":"2024-04-26T11:27:25.188093+0000","last_fullsized":"2024-04-26T11:27:25.188093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:38:43.247282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63153,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":74,"num_read_kb":98,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245570+0000","last_change":"2024-04-26T11:26:28.994764+0000","last_active":"2024-04-26T11:27:19.245570+0000","last_peered":"2024-04-26T11:27:19.245570+0000","last_clean":"2024-04-26T11:27:19.245570+0000","last_became_active":"2024-04-26T11:26:28.994421+0000","last_became_peered":"2024-04-26T11:26:28.994421+0000","last_unstale":"2024-04-26T11:27:19.245570+0000","last_undegraded":"2024-04-26T11:27:19.245570+0000","last_fullsized":"2024-04-26T11:27:19.245570+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:06:06.499434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242227+0000","last_change":"2024-04-26T11:26:24.968036+0000","last_active":"2024-04-26T11:27:19.242227+0000","last_peered":"2024-04-26T11:27:19.242227+0000","last_clean":"2024-04-26T11:27:19.242227+0000","last_became_active":"2024-04-26T11:26:24.967684+0000","last_became_peered":"2024-04-26T11:26:24.967684+0000","last_unstale":"2024-04-26T11:27:19.242227+0000","last_undegraded":"2024-04-26T11:27:19.242227+0000","last_fullsized":"2024-04-26T11:27:19.242227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:00:27.467411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412101+0000","last_change":"2024-04-26T11:26:29.020642+0000","last_active":"2024-04-26T11:27:19.412101+0000","last_peered":"2024-04-26T11:27:19.412101+0000","last_clean":"2024-04-26T11:27:19.412101+0000","last_became_active":"2024-04-26T11:26:29.020278+0000","last_became_peered":"2024-04-26T11:26:29.020278+0000","last_unstale":"2024-04-26T11:27:19.412101+0000","last_undegraded":"2024-04-26T11:27:19.412101+0000","last_fullsized":"2024-04-26T11:27:19.412101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:48.932064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240421+0000","last_change":"2024-04-26T11:26:24.970111+0000","last_active":"2024-04-26T11:27:19.240421+0000","last_peered":"2024-04-26T11:27:19.240421+0000","last_clean":"2024-04-26T11:27:19.240421+0000","last_became_active":"2024-04-26T11:26:24.969629+0000","last_became_peered":"2024-04-26T11:26:24.969629+0000","last_unstale":"2024-04-26T11:27:19.240421+0000","last_undegraded":"2024-04-26T11:27:19.240421+0000","last_fullsized":"2024-04-26T11:27:19.240421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:52:07.859419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276856+0000","last_change":"2024-04-26T11:26:29.002986+0000","last_active":"2024-04-26T11:27:19.276856+0000","last_peered":"2024-04-26T11:27:19.276856+0000","last_clean":"2024-04-26T11:27:19.276856+0000","last_became_active":"2024-04-26T11:26:29.002610+0000","last_became_peered":"2024-04-26T11:26:29.002610+0000","last_unstale":"2024-04-26T11:27:19.276856+0000","last_undegraded":"2024-04-26T11:27:19.276856+0000","last_fullsized":"2024-04-26T11:27:19.276856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:04.877356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:27.213931+0000","last_change":"2024-04-26T11:26:24.957825+0000","last_active":"2024-04-26T11:27:27.213931+0000","last_peered":"2024-04-26T11:27:27.213931+0000","last_clean":"2024-04-26T11:27:27.213931+0000","last_became_active":"2024-04-26T11:26:24.957464+0000","last_became_peered":"2024-04-26T11:26:24.957464+0000","last_unstale":"2024-04-26T11:27:27.213931+0000","last_undegraded":"2024-04-26T11:27:27.213931+0000","last_fullsized":"2024-04-26T11:27:27.213931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:06:49.350268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276605+0000","last_change":"2024-04-26T11:26:28.994223+0000","last_active":"2024-04-26T11:27:19.276605+0000","last_peered":"2024-04-26T11:27:19.276605+0000","last_clean":"2024-04-26T11:27:19.276605+0000","last_became_active":"2024-04-26T11:26:28.993388+0000","last_became_peered":"2024-04-26T11:26:28.993388+0000","last_unstale":"2024-04-26T11:27:19.276605+0000","last_undegraded":"2024-04-26T11:27:19.276605+0000","last_fullsized":"2024-04-26T11:27:19.276605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:07:24.208028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245056+0000","last_change":"2024-04-26T11:26:25.078473+0000","last_active":"2024-04-26T11:27:19.245056+0000","last_peered":"2024-04-26T11:27:19.245056+0000","last_clean":"2024-04-26T11:27:19.245056+0000","last_became_active":"2024-04-26T11:26:25.078138+0000","last_became_peered":"2024-04-26T11:26:25.078138+0000","last_unstale":"2024-04-26T11:27:19.245056+0000","last_undegraded":"2024-04-26T11:27:19.245056+0000","last_fullsized":"2024-04-26T11:27:19.245056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:19.692345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240164+0000","last_change":"2024-04-26T11:26:28.998687+0000","last_active":"2024-04-26T11:27:19.240164+0000","last_peered":"2024-04-26T11:27:19.240164+0000","last_clean":"2024-04-26T11:27:19.240164+0000","last_became_active":"2024-04-26T11:26:28.998358+0000","last_became_peered":"2024-04-26T11:26:28.998358+0000","last_unstale":"2024-04-26T11:27:19.240164+0000","last_undegraded":"2024-04-26T11:27:19.240164+0000","last_fullsized":"2024-04-26T11:27:19.240164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:40:51.358188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246101+0000","last_change":"2024-04-26T11:26:24.956213+0000","last_active":"2024-04-26T11:27:19.246101+0000","last_peered":"2024-04-26T11:27:19.246101+0000","last_clean":"2024-04-26T11:27:19.246101+0000","last_became_active":"2024-04-26T11:26:24.955773+0000","last_became_peered":"2024-04-26T11:26:24.955773+0000","last_unstale":"2024-04-26T11:27:19.246101+0000","last_undegraded":"2024-04-26T11:27:19.246101+0000","last_fullsized":"2024-04-26T11:27:19.246101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:38.253367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.839222+0000","last_change":"2024-04-26T11:26:29.001389+0000","last_active":"2024-04-26T11:27:19.839222+0000","last_peered":"2024-04-26T11:27:19.839222+0000","last_clean":"2024-04-26T11:27:19.839222+0000","last_became_active":"2024-04-26T11:26:29.001023+0000","last_became_peered":"2024-04-26T11:26:29.001023+0000","last_unstale":"2024-04-26T11:27:19.839222+0000","last_undegraded":"2024-04-26T11:27:19.839222+0000","last_fullsized":"2024-04-26T11:27:19.839222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:57:23.606518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.893959+0000","last_change":"2024-04-26T11:27:19.893959+0000","last_active":"2024-04-26T11:27:19.893959+0000","last_peered":"2024-04-26T11:27:19.893959+0000","last_clean":"2024-04-26T11:27:19.893959+0000","last_became_active":"2024-04-26T11:27:19.892819+0000","last_became_peered":"2024-04-26T11:27:19.892819+0000","last_unstale":"2024-04-26T11:27:19.893959+0000","last_undegraded":"2024-04-26T11:27:19.893959+0000","last_fullsized":"2024-04-26T11:27:19.893959+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:53:39.829381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242593+0000","last_change":"2024-04-26T11:26:29.019906+0000","last_active":"2024-04-26T11:27:19.242593+0000","last_peered":"2024-04-26T11:27:19.242593+0000","last_clean":"2024-04-26T11:27:19.242593+0000","last_became_active":"2024-04-26T11:26:29.019545+0000","last_became_peered":"2024-04-26T11:26:29.019545+0000","last_unstale":"2024-04-26T11:27:19.242593+0000","last_undegraded":"2024-04-26T11:27:19.242593+0000","last_fullsized":"2024-04-26T11:27:19.242593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:48.617269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276976+0000","last_change":"2024-04-26T11:26:24.956400+0000","last_active":"2024-04-26T11:27:19.276976+0000","last_peered":"2024-04-26T11:27:19.276976+0000","last_clean":"2024-04-26T11:27:19.276976+0000","last_became_active":"2024-04-26T11:26:24.956030+0000","last_became_peered":"2024-04-26T11:26:24.956030+0000","last_unstale":"2024-04-26T11:27:19.276976+0000","last_undegraded":"2024-04-26T11:27:19.276976+0000","last_fullsized":"2024-04-26T11:27:19.276976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:47.841006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242474+0000","last_change":"2024-04-26T11:26:29.013087+0000","last_active":"2024-04-26T11:27:19.242474+0000","last_peered":"2024-04-26T11:27:19.242474+0000","last_clean":"2024-04-26T11:27:19.242474+0000","last_became_active":"2024-04-26T11:26:29.012330+0000","last_became_peered":"2024-04-26T11:26:29.012330+0000","last_unstale":"2024-04-26T11:27:19.242474+0000","last_undegraded":"2024-04-26T11:27:19.242474+0000","last_fullsized":"2024-04-26T11:27:19.242474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:12:50.752329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245410+0000","last_change":"2024-04-26T11:27:19.245410+0000","last_active":"2024-04-26T11:27:19.245410+0000","last_peered":"2024-04-26T11:27:19.245410+0000","last_clean":"2024-04-26T11:27:19.245410+0000","last_became_active":"2024-04-26T11:27:19.244578+0000","last_became_peered":"2024-04-26T11:27:19.244578+0000","last_unstale":"2024-04-26T11:27:19.245410+0000","last_undegraded":"2024-04-26T11:27:19.245410+0000","last_fullsized":"2024-04-26T11:27:19.245410+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:07:50.576496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:26.210759+0000","last_change":"2024-04-26T11:26:25.082517+0000","last_active":"2024-04-26T11:27:26.210759+0000","last_peered":"2024-04-26T11:27:26.210759+0000","last_clean":"2024-04-26T11:27:26.210759+0000","last_became_active":"2024-04-26T11:26:25.081724+0000","last_became_peered":"2024-04-26T11:26:25.081724+0000","last_unstale":"2024-04-26T11:27:26.210759+0000","last_undegraded":"2024-04-26T11:27:26.210759+0000","last_fullsized":"2024-04-26T11:27:26.210759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:59:40.006615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246084+0000","last_change":"2024-04-26T11:26:29.021425+0000","last_active":"2024-04-26T11:27:19.246084+0000","last_peered":"2024-04-26T11:27:19.246084+0000","last_clean":"2024-04-26T11:27:19.246084+0000","last_became_active":"2024-04-26T11:26:29.020881+0000","last_became_peered":"2024-04-26T11:26:29.020881+0000","last_unstale":"2024-04-26T11:27:19.246084+0000","last_undegraded":"2024-04-26T11:27:19.246084+0000","last_fullsized":"2024-04-26T11:27:19.246084+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:18.350323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240518+0000","last_change":"2024-04-26T11:26:24.963614+0000","last_active":"2024-04-26T11:27:19.240518+0000","last_peered":"2024-04-26T11:27:19.240518+0000","last_clean":"2024-04-26T11:27:19.240518+0000","last_became_active":"2024-04-26T11:26:24.963312+0000","last_became_peered":"2024-04-26T11:26:24.963312+0000","last_unstale":"2024-04-26T11:27:19.240518+0000","last_undegraded":"2024-04-26T11:27:19.240518+0000","last_fullsized":"2024-04-26T11:27:19.240518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:55.886637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":360,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276824+0000","last_change":"2024-04-26T11:24:54.292436+0000","last_active":"2024-04-26T11:27:19.276824+0000","last_peered":"2024-04-26T11:27:19.276824+0000","last_clean":"2024-04-26T11:27:19.276824+0000","last_became_active":"2024-04-26T11:24:54.291921+0000","last_became_peered":"2024-04-26T11:24:54.291921+0000","last_unstale":"2024-04-26T11:27:19.276824+0000","last_undegraded":"2024-04-26T11:27:19.276824+0000","last_fullsized":"2024-04-26T11:27:19.276824+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_clean_scrub_stamp":"2024-04-26T11:21:00.617277+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:11:01.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.411585+0000","last_change":"2024-04-26T11:26:29.000905+0000","last_active":"2024-04-26T11:27:19.411585+0000","last_peered":"2024-04-26T11:27:19.411585+0000","last_clean":"2024-04-26T11:27:19.411585+0000","last_became_active":"2024-04-26T11:26:29.000517+0000","last_became_peered":"2024-04-26T11:26:29.000517+0000","last_unstale":"2024-04-26T11:27:19.411585+0000","last_undegraded":"2024-04-26T11:27:19.411585+0000","last_fullsized":"2024-04-26T11:27:19.411585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:38:55.394316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277067+0000","last_change":"2024-04-26T11:26:24.957489+0000","last_active":"2024-04-26T11:27:19.277067+0000","last_peered":"2024-04-26T11:27:19.277067+0000","last_clean":"2024-04-26T11:27:19.277067+0000","last_became_active":"2024-04-26T11:26:24.957151+0000","last_became_peered":"2024-04-26T11:26:24.957151+0000","last_unstale":"2024-04-26T11:27:19.277067+0000","last_undegraded":"2024-04-26T11:27:19.277067+0000","last_fullsized":"2024-04-26T11:27:19.277067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:42:25.343105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245684+0000","last_change":"2024-04-26T11:26:25.086767+0000","last_active":"2024-04-26T11:27:19.245684+0000","last_peered":"2024-04-26T11:27:19.245684+0000","last_clean":"2024-04-26T11:27:19.245684+0000","last_became_active":"2024-04-26T11:26:25.086440+0000","last_became_peered":"2024-04-26T11:26:25.086440+0000","last_unstale":"2024-04-26T11:27:19.245684+0000","last_undegraded":"2024-04-26T11:27:19.245684+0000","last_fullsized":"2024-04-26T11:27:19.245684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:02.294644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412245+0000","last_change":"2024-04-26T11:26:29.003683+0000","last_active":"2024-04-26T11:27:19.412245+0000","last_peered":"2024-04-26T11:27:19.412245+0000","last_clean":"2024-04-26T11:27:19.412245+0000","last_became_active":"2024-04-26T11:26:29.003307+0000","last_became_peered":"2024-04-26T11:26:29.003307+0000","last_unstale":"2024-04-26T11:27:19.412245+0000","last_undegraded":"2024-04-26T11:27:19.412245+0000","last_fullsized":"2024-04-26T11:27:19.412245+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:41:00.124540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.851358+0000","last_change":"2024-04-26T11:27:19.849965+0000","last_active":"2024-04-26T11:27:19.851358+0000","last_peered":"2024-04-26T11:27:19.851358+0000","last_clean":"2024-04-26T11:27:19.851358+0000","last_became_active":"2024-04-26T11:27:19.840545+0000","last_became_peered":"2024-04-26T11:27:19.840545+0000","last_unstale":"2024-04-26T11:27:19.851358+0000","last_undegraded":"2024-04-26T11:27:19.851358+0000","last_fullsized":"2024-04-26T11:27:19.851358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:08.003704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246440+0000","last_change":"2024-04-26T11:26:29.016783+0000","last_active":"2024-04-26T11:27:19.246440+0000","last_peered":"2024-04-26T11:27:19.246440+0000","last_clean":"2024-04-26T11:27:19.246440+0000","last_became_active":"2024-04-26T11:26:29.013998+0000","last_became_peered":"2024-04-26T11:26:29.013998+0000","last_unstale":"2024-04-26T11:27:19.246440+0000","last_undegraded":"2024-04-26T11:27:19.246440+0000","last_fullsized":"2024-04-26T11:27:19.246440+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:24:44.453094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240597+0000","last_change":"2024-04-26T11:26:25.902330+0000","last_active":"2024-04-26T11:27:19.240597+0000","last_peered":"2024-04-26T11:27:19.240597+0000","last_clean":"2024-04-26T11:27:19.240597+0000","last_became_active":"2024-04-26T11:26:25.901948+0000","last_became_peered":"2024-04-26T11:26:25.901948+0000","last_unstale":"2024-04-26T11:27:19.240597+0000","last_undegraded":"2024-04-26T11:27:19.240597+0000","last_fullsized":"2024-04-26T11:27:19.240597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:00:23.541393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.890462+0000","last_change":"2024-04-26T11:26:29.025692+0000","last_active":"2024-04-26T11:27:19.890462+0000","last_peered":"2024-04-26T11:27:19.890462+0000","last_clean":"2024-04-26T11:27:19.890462+0000","last_became_active":"2024-04-26T11:26:29.025112+0000","last_became_peered":"2024-04-26T11:26:29.025112+0000","last_unstale":"2024-04-26T11:27:19.890462+0000","last_undegraded":"2024-04-26T11:27:19.890462+0000","last_fullsized":"2024-04-26T11:27:19.890462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:39:22.637865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244415+0000","last_change":"2024-04-26T11:26:25.080270+0000","last_active":"2024-04-26T11:27:19.244415+0000","last_peered":"2024-04-26T11:27:19.244415+0000","last_clean":"2024-04-26T11:27:19.244415+0000","last_became_active":"2024-04-26T11:26:25.079895+0000","last_became_peered":"2024-04-26T11:26:25.079895+0000","last_unstale":"2024-04-26T11:27:19.244415+0000","last_undegraded":"2024-04-26T11:27:19.244415+0000","last_fullsized":"2024-04-26T11:27:19.244415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:47:08.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245194+0000","last_change":"2024-04-26T11:26:28.996540+0000","last_active":"2024-04-26T11:27:19.245194+0000","last_peered":"2024-04-26T11:27:19.245194+0000","last_clean":"2024-04-26T11:27:19.245194+0000","last_became_active":"2024-04-26T11:26:28.994724+0000","last_became_peered":"2024-04-26T11:26:28.994724+0000","last_unstale":"2024-04-26T11:27:19.245194+0000","last_undegraded":"2024-04-26T11:27:19.245194+0000","last_fullsized":"2024-04-26T11:27:19.245194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:13.382332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245828+0000","last_change":"2024-04-26T11:26:24.967935+0000","last_active":"2024-04-26T11:27:19.245828+0000","last_peered":"2024-04-26T11:27:19.245828+0000","last_clean":"2024-04-26T11:27:19.245828+0000","last_became_active":"2024-04-26T11:26:24.966465+0000","last_became_peered":"2024-04-26T11:26:24.966465+0000","last_unstale":"2024-04-26T11:27:19.245828+0000","last_undegraded":"2024-04-26T11:27:19.245828+0000","last_fullsized":"2024-04-26T11:27:19.245828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:27:38.817316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838591+0000","last_change":"2024-04-26T11:26:29.024243+0000","last_active":"2024-04-26T11:27:19.838591+0000","last_peered":"2024-04-26T11:27:19.838591+0000","last_clean":"2024-04-26T11:27:19.838591+0000","last_became_active":"2024-04-26T11:26:29.023900+0000","last_became_peered":"2024-04-26T11:26:29.023900+0000","last_unstale":"2024-04-26T11:27:19.838591+0000","last_undegraded":"2024-04-26T11:27:19.838591+0000","last_fullsized":"2024-04-26T11:27:19.838591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:28:23.446568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242062+0000","last_change":"2024-04-26T11:26:24.969953+0000","last_active":"2024-04-26T11:27:19.242062+0000","last_peered":"2024-04-26T11:27:19.242062+0000","last_clean":"2024-04-26T11:27:19.242062+0000","last_became_active":"2024-04-26T11:26:24.969371+0000","last_became_peered":"2024-04-26T11:26:24.969371+0000","last_unstale":"2024-04-26T11:27:19.242062+0000","last_undegraded":"2024-04-26T11:27:19.242062+0000","last_fullsized":"2024-04-26T11:27:19.242062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:07:24.323833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245829+0000","last_change":"2024-04-26T11:26:29.029771+0000","last_active":"2024-04-26T11:27:19.245829+0000","last_peered":"2024-04-26T11:27:19.245829+0000","last_clean":"2024-04-26T11:27:19.245829+0000","last_became_active":"2024-04-26T11:26:29.029387+0000","last_became_peered":"2024-04-26T11:26:29.029387+0000","last_unstale":"2024-04-26T11:27:19.245829+0000","last_undegraded":"2024-04-26T11:27:19.245829+0000","last_fullsized":"2024-04-26T11:27:19.245829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:40.856935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245304+0000","last_change":"2024-04-26T11:26:24.958608+0000","last_active":"2024-04-26T11:27:19.245304+0000","last_peered":"2024-04-26T11:27:19.245304+0000","last_clean":"2024-04-26T11:27:19.245304+0000","last_became_active":"2024-04-26T11:26:24.958188+0000","last_became_peered":"2024-04-26T11:26:24.958188+0000","last_unstale":"2024-04-26T11:27:19.245304+0000","last_undegraded":"2024-04-26T11:27:19.245304+0000","last_fullsized":"2024-04-26T11:27:19.245304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:31:41.442551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277133+0000","last_change":"2024-04-26T11:26:29.000207+0000","last_active":"2024-04-26T11:27:19.277133+0000","last_peered":"2024-04-26T11:27:19.277133+0000","last_clean":"2024-04-26T11:27:19.277133+0000","last_became_active":"2024-04-26T11:26:28.999894+0000","last_became_peered":"2024-04-26T11:26:28.999894+0000","last_unstale":"2024-04-26T11:27:19.277133+0000","last_undegraded":"2024-04-26T11:27:19.277133+0000","last_fullsized":"2024-04-26T11:27:19.277133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:52:10.349511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242602+0000","last_change":"2024-04-26T11:26:25.902036+0000","last_active":"2024-04-26T11:27:19.242602+0000","last_peered":"2024-04-26T11:27:19.242602+0000","last_clean":"2024-04-26T11:27:19.242602+0000","last_became_active":"2024-04-26T11:26:25.901567+0000","last_became_peered":"2024-04-26T11:26:25.901567+0000","last_unstale":"2024-04-26T11:27:19.242602+0000","last_undegraded":"2024-04-26T11:27:19.242602+0000","last_fullsized":"2024-04-26T11:27:19.242602+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:58.757780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277220+0000","last_change":"2024-04-26T11:26:28.998915+0000","last_active":"2024-04-26T11:27:19.277220+0000","last_peered":"2024-04-26T11:27:19.277220+0000","last_clean":"2024-04-26T11:27:19.277220+0000","last_became_active":"2024-04-26T11:26:28.998180+0000","last_became_peered":"2024-04-26T11:26:28.998180+0000","last_unstale":"2024-04-26T11:27:19.277220+0000","last_undegraded":"2024-04-26T11:27:19.277220+0000","last_fullsized":"2024-04-26T11:27:19.277220+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:19.972697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240356+0000","last_change":"2024-04-26T11:26:24.966029+0000","last_active":"2024-04-26T11:27:19.240356+0000","last_peered":"2024-04-26T11:27:19.240356+0000","last_clean":"2024-04-26T11:27:19.240356+0000","last_became_active":"2024-04-26T11:26:24.965676+0000","last_became_peered":"2024-04-26T11:26:24.965676+0000","last_unstale":"2024-04-26T11:27:19.240356+0000","last_undegraded":"2024-04-26T11:27:19.240356+0000","last_fullsized":"2024-04-26T11:27:19.240356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:18:06.904563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276954+0000","last_change":"2024-04-26T11:26:29.020093+0000","last_active":"2024-04-26T11:27:19.276954+0000","last_peered":"2024-04-26T11:27:19.276954+0000","last_clean":"2024-04-26T11:27:19.276954+0000","last_became_active":"2024-04-26T11:26:29.019690+0000","last_became_peered":"2024-04-26T11:26:29.019690+0000","last_unstale":"2024-04-26T11:27:19.276954+0000","last_undegraded":"2024-04-26T11:27:19.276954+0000","last_fullsized":"2024-04-26T11:27:19.276954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:35:44.501217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412931+0000","last_change":"2024-04-26T11:26:24.966757+0000","last_active":"2024-04-26T11:27:19.412931+0000","last_peered":"2024-04-26T11:27:19.412931+0000","last_clean":"2024-04-26T11:27:19.412931+0000","last_became_active":"2024-04-26T11:26:24.966298+0000","last_became_peered":"2024-04-26T11:26:24.966298+0000","last_unstale":"2024-04-26T11:27:19.412931+0000","last_undegraded":"2024-04-26T11:27:19.412931+0000","last_fullsized":"2024-04-26T11:27:19.412931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:17:40.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277300+0000","last_change":"2024-04-26T11:26:28.997774+0000","last_active":"2024-04-26T11:27:19.277300+0000","last_peered":"2024-04-26T11:27:19.277300+0000","last_clean":"2024-04-26T11:27:19.277300+0000","last_became_active":"2024-04-26T11:26:28.997455+0000","last_became_peered":"2024-04-26T11:26:28.997455+0000","last_unstale":"2024-04-26T11:27:19.277300+0000","last_undegraded":"2024-04-26T11:27:19.277300+0000","last_fullsized":"2024-04-26T11:27:19.277300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:56.097022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838639+0000","last_change":"2024-04-26T11:26:28.999714+0000","last_active":"2024-04-26T11:27:19.838639+0000","last_peered":"2024-04-26T11:27:19.838639+0000","last_clean":"2024-04-26T11:27:19.838639+0000","last_became_active":"2024-04-26T11:26:28.999377+0000","last_became_peered":"2024-04-26T11:26:28.999377+0000","last_unstale":"2024-04-26T11:27:19.838639+0000","last_undegraded":"2024-04-26T11:27:19.838639+0000","last_fullsized":"2024-04-26T11:27:19.838639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:39:59.141556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.247154+0000","last_change":"2024-04-26T11:27:19.247154+0000","last_active":"2024-04-26T11:27:19.247154+0000","last_peered":"2024-04-26T11:27:19.247154+0000","last_clean":"2024-04-26T11:27:19.247154+0000","last_became_active":"2024-04-26T11:27:19.246590+0000","last_became_peered":"2024-04-26T11:27:19.246590+0000","last_unstale":"2024-04-26T11:27:19.247154+0000","last_undegraded":"2024-04-26T11:27:19.247154+0000","last_fullsized":"2024-04-26T11:27:19.247154+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:12:58.401699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242036+0000","last_change":"2024-04-26T11:26:29.019983+0000","last_active":"2024-04-26T11:27:19.242036+0000","last_peered":"2024-04-26T11:27:19.242036+0000","last_clean":"2024-04-26T11:27:19.242036+0000","last_became_active":"2024-04-26T11:26:29.019536+0000","last_became_peered":"2024-04-26T11:26:29.019536+0000","last_unstale":"2024-04-26T11:27:19.242036+0000","last_undegraded":"2024-04-26T11:27:19.242036+0000","last_fullsized":"2024-04-26T11:27:19.242036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:15:06.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.889701+0000","last_change":"2024-04-26T11:26:29.027089+0000","last_active":"2024-04-26T11:27:19.889701+0000","last_peered":"2024-04-26T11:27:19.889701+0000","last_clean":"2024-04-26T11:27:19.889701+0000","last_became_active":"2024-04-26T11:26:29.026030+0000","last_became_peered":"2024-04-26T11:26:29.026030+0000","last_unstale":"2024-04-26T11:27:19.889701+0000","last_undegraded":"2024-04-26T11:27:19.889701+0000","last_fullsized":"2024-04-26T11:27:19.889701+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:24:06.105600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412095+0000","last_change":"2024-04-26T11:26:29.018472+0000","last_active":"2024-04-26T11:27:19.412095+0000","last_peered":"2024-04-26T11:27:19.412095+0000","last_clean":"2024-04-26T11:27:19.412095+0000","last_became_active":"2024-04-26T11:26:29.015947+0000","last_became_peered":"2024-04-26T11:26:29.015947+0000","last_unstale":"2024-04-26T11:27:19.412095+0000","last_undegraded":"2024-04-26T11:27:19.412095+0000","last_fullsized":"2024-04-26T11:27:19.412095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:35:48.082709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838871+0000","last_change":"2024-04-26T11:26:28.997582+0000","last_active":"2024-04-26T11:27:19.838871+0000","last_peered":"2024-04-26T11:27:19.838871+0000","last_clean":"2024-04-26T11:27:19.838871+0000","last_became_active":"2024-04-26T11:26:28.997202+0000","last_became_peered":"2024-04-26T11:26:28.997202+0000","last_unstale":"2024-04-26T11:27:19.838871+0000","last_undegraded":"2024-04-26T11:27:19.838871+0000","last_fullsized":"2024-04-26T11:27:19.838871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:59.776455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242635+0000","last_change":"2024-04-26T11:26:28.999428+0000","last_active":"2024-04-26T11:27:19.242635+0000","last_peered":"2024-04-26T11:27:19.242635+0000","last_clean":"2024-04-26T11:27:19.242635+0000","last_became_active":"2024-04-26T11:26:28.999093+0000","last_became_peered":"2024-04-26T11:26:28.999093+0000","last_unstale":"2024-04-26T11:27:19.242635+0000","last_undegraded":"2024-04-26T11:27:19.242635+0000","last_fullsized":"2024-04-26T11:27:19.242635+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:07.222231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246478+0000","last_change":"2024-04-26T11:26:29.019673+0000","last_active":"2024-04-26T11:27:19.246478+0000","last_peered":"2024-04-26T11:27:19.246478+0000","last_clean":"2024-04-26T11:27:19.246478+0000","last_became_active":"2024-04-26T11:26:29.017289+0000","last_became_peered":"2024-04-26T11:26:29.017289+0000","last_unstale":"2024-04-26T11:27:19.246478+0000","last_undegraded":"2024-04-26T11:27:19.246478+0000","last_fullsized":"2024-04-26T11:27:19.246478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:35.848004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240127+0000","last_change":"2024-04-26T11:26:29.017234+0000","last_active":"2024-04-26T11:27:19.240127+0000","last_peered":"2024-04-26T11:27:19.240127+0000","last_clean":"2024-04-26T11:27:19.240127+0000","last_became_active":"2024-04-26T11:26:29.016863+0000","last_became_peered":"2024-04-26T11:26:29.016863+0000","last_unstale":"2024-04-26T11:27:19.240127+0000","last_undegraded":"2024-04-26T11:27:19.240127+0000","last_fullsized":"2024-04-26T11:27:19.240127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:14:31.136286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245593+0000","last_change":"2024-04-26T11:26:29.018122+0000","last_active":"2024-04-26T11:27:19.245593+0000","last_peered":"2024-04-26T11:27:19.245593+0000","last_clean":"2024-04-26T11:27:19.245593+0000","last_became_active":"2024-04-26T11:26:29.016269+0000","last_became_peered":"2024-04-26T11:26:29.016269+0000","last_unstale":"2024-04-26T11:27:19.245593+0000","last_undegraded":"2024-04-26T11:27:19.245593+0000","last_fullsized":"2024-04-26T11:27:19.245593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:22.135086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.278596+0000","last_change":"2024-04-26T11:27:19.278596+0000","last_active":"2024-04-26T11:27:19.278596+0000","last_peered":"2024-04-26T11:27:19.278596+0000","last_clean":"2024-04-26T11:27:19.278596+0000","last_became_active":"2024-04-26T11:27:19.278138+0000","last_became_peered":"2024-04-26T11:27:19.278138+0000","last_unstale":"2024-04-26T11:27:19.278596+0000","last_undegraded":"2024-04-26T11:27:19.278596+0000","last_fullsized":"2024-04-26T11:27:19.278596+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:17.038527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.412133+0000","last_change":"2024-04-26T11:26:29.026905+0000","last_active":"2024-04-26T11:27:19.412133+0000","last_peered":"2024-04-26T11:27:19.412133+0000","last_clean":"2024-04-26T11:27:19.412133+0000","last_became_active":"2024-04-26T11:26:29.026539+0000","last_became_peered":"2024-04-26T11:26:29.026539+0000","last_unstale":"2024-04-26T11:27:19.412133+0000","last_undegraded":"2024-04-26T11:27:19.412133+0000","last_fullsized":"2024-04-26T11:27:19.412133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:41:12.169466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.838812+0000","last_change":"2024-04-26T11:26:28.999953+0000","last_active":"2024-04-26T11:27:19.838812+0000","last_peered":"2024-04-26T11:27:19.838812+0000","last_clean":"2024-04-26T11:27:19.838812+0000","last_became_active":"2024-04-26T11:26:28.998344+0000","last_became_peered":"2024-04-26T11:26:28.998344+0000","last_unstale":"2024-04-26T11:27:19.838812+0000","last_undegraded":"2024-04-26T11:27:19.838812+0000","last_fullsized":"2024-04-26T11:27:19.838812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:30:53.269171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281068,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67154,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":232,"num_read_kb":185,"num_write":168,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":201462,"data_compressed":5472,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383707,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":479968,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32280,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712872,"statfs":{"total":95995035648,"available":95961980928,"internally_reserved":0,"allocated":1130496,"data_stored":1134993,"data_compressed":43361,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808871,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480204,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":480124,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":1072054,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":479593,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":542539,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":480864,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":543889,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":479904,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":480421,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":1071347,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63153,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64503,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65122,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T11:27:32.318 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T11:27:32.319 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T11:27:32.319 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T11:27:32.823 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:32 smithi033 ceph-mon[27845]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:27:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:32 smithi096 ceph-mon[32753]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:27:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:32 smithi194 ceph-mon[33017]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:27:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:33 smithi033 ceph-mon[27845]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:33 smithi096 ceph-mon[32753]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:33 smithi194 ceph-mon[33017]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:34.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:34 smithi033 ceph-mon[27845]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:34 smithi096 ceph-mon[32753]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:34.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 -- 172.21.15.33:0/1397060022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13541091a0 msgr2=0x7f135410b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 --2- 172.21.15.33:0/1397060022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13541091a0 0x7f135410b610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f134c009f90 tx=0x7f134c00b330 comp rx=0 tx=0).stop 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 -- 172.21.15.33:0/1397060022 shutdown_connections 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 --2- 172.21.15.33:0/1397060022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13541091a0 0x7f135410b610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 --2- 172.21.15.33:0/1397060022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1354103f40 0x7f1354106330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 --2- 172.21.15.33:0/1397060022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1354106870 0x7f1354108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 -- 172.21.15.33:0/1397060022 >> 172.21.15.33:0/1397060022 conn(0x7f13540fd970 msgr2=0x7f13540ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 -- 172.21.15.33:0/1397060022 shutdown_connections 2024-04-26T11:27:34.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.971+0000 7f135aa00700 1 -- 172.21.15.33:0/1397060022 wait complete. 2024-04-26T11:27:34.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 Processor -- start 2024-04-26T11:27:34.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 -- start start 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1354103f40 0x7f13541994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1354106870 0x7f1354199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13541091a0 0x7f13541a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f135410efc0 con 0x7f13541091a0 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f135410ee40 con 0x7f1354103f40 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.973+0000 7f135aa00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f135410f140 con 0x7f1354106870 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1354106870 0x7f1354199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1354106870 0x7f1354199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:33460/0 (socket says 172.21.15.33:33460) 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 -- 172.21.15.33:0/2064124854 learned_addr learned my addr 172.21.15.33:0/2064124854 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353e00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1354103f40 0x7f13541994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1358a00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13541091a0 0x7f13541a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 -- 172.21.15.33:0/2064124854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1354103f40 msgr2=0x7f13541994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:34.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1354103f40 0x7f13541994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:34.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 -- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13541091a0 msgr2=0x7f13541a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:34.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13541091a0 0x7f13541a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:34.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 -- 172.21.15.33:0/2064124854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13541a1140 con 0x7f1354106870 2024-04-26T11:27:34.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1358a00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13541091a0 0x7f13541a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:27:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.974+0000 7f1353400700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1354106870 0x7f1354199a20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f134000e9d0 tx=0x7f134000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.975+0000 7f1350c00700 1 -- 172.21.15.33:0/2064124854 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134000cbd0 con 0x7f1354106870 2024-04-26T11:27:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.975+0000 7f1350c00700 1 -- 172.21.15.33:0/2064124854 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f134000cd70 con 0x7f1354106870 2024-04-26T11:27:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.975+0000 7f1350c00700 1 -- 172.21.15.33:0/2064124854 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340018680 con 0x7f1354106870 2024-04-26T11:27:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.975+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13541a1430 con 0x7f1354106870 2024-04-26T11:27:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.975+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13541a19c0 con 0x7f1354106870 2024-04-26T11:27:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.976+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1354051960 con 0x7f1354106870 2024-04-26T11:27:34.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.979+0000 7f1350c00700 1 -- 172.21.15.33:0/2064124854 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1340018820 con 0x7f1354106870 2024-04-26T11:27:34.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.980+0000 7f1350c00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13380a3e00 0x7f13380a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:34.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.980+0000 7f1350c00700 1 -- 172.21.15.33:0/2064124854 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f13400d07a0 con 0x7f1354106870 2024-04-26T11:27:34.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.980+0000 7f1353e00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13380a3e00 0x7f13380a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:34.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.981+0000 7f1353e00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13380a3e00 0x7f13380a62c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1344005fd0 tx=0x7f1344009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:34.984+0000 7f1350c00700 1 -- 172.21.15.33:0/2064124854 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f134009cfe0 con 0x7f1354106870 2024-04-26T11:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:34 smithi194 ceph-mon[33017]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:35.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.311+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f1354064030 con 0x7f1354106870 2024-04-26T11:27:35.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.319+0000 7f1350c00700 1 -- 172.21.15.33:0/2064124854 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f134009c8e0 con 0x7f1354106870 2024-04-26T11:27:35.321 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-04-26T11:27:35.321 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQCnjytmfiDFEhAATVyM7IW+X7ssiHjLfqFIdA== 2024-04-26T11:27:35.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.322+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13380a3e00 msgr2=0x7f13380a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:35.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.322+0000 7f135aa00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13380a3e00 0x7f13380a62c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1344005fd0 tx=0x7f1344009310 comp rx=0 tx=0).stop 2024-04-26T11:27:35.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.322+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1354106870 msgr2=0x7f1354199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:35.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.322+0000 7f135aa00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1354106870 0x7f1354199a20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f134000e9d0 tx=0x7f134000ed90 comp rx=0 tx=0).stop 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 shutdown_connections 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f13380a3e00 0x7f13380a62c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13541091a0 0x7f13541a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1354103f40 0x7f13541994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 --2- 172.21.15.33:0/2064124854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1354106870 0x7f1354199a20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 >> 172.21.15.33:0/2064124854 conn(0x7f13540fd970 msgr2=0x7f13541048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 shutdown_connections 2024-04-26T11:27:35.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:35.323+0000 7f135aa00700 1 -- 172.21.15.33:0/2064124854 wait complete. 2024-04-26T11:27:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:35 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2064124854' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T11:27:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:35 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T11:27:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:35 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T11:27:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:35 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2064124854' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T11:27:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:35 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T11:27:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:35 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T11:27:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:35 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2064124854' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T11:27:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:35 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T11:27:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:35 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T11:27:36.134 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:27:36.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T11:27:36.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T11:27:36.252 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T11:27:36.252 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T11:27:36.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T11:27:36.765 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:36.915 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:36 smithi033 ceph-mon[27845]: pgmap v359: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:36 smithi096 ceph-mon[32753]: pgmap v359: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:36 smithi194 ceph-mon[33017]: pgmap v359: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T11:27:39.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 -- 172.21.15.33:0/2059889569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee006bb00 msgr2=0x7f6ee006bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:39.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2059889569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee006bb00 0x7f6ee006bf60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009990 tx=0x7f6ecc009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 -- 172.21.15.33:0/2059889569 shutdown_connections 2024-04-26T11:27:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2059889569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ee006c4a0 0x7f6ee01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2059889569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee01094a0 0x7f6ee0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2059889569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee006bb00 0x7f6ee006bf60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 -- 172.21.15.33:0/2059889569 >> 172.21.15.33:0/2059889569 conn(0x7f6ee0079290 msgr2=0x7f6ee00796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 -- 172.21.15.33:0/2059889569 shutdown_connections 2024-04-26T11:27:39.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.012+0000 7f6ee6000700 1 -- 172.21.15.33:0/2059889569 wait complete. 2024-04-26T11:27:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.014+0000 7f6ee6000700 1 Processor -- start 2024-04-26T11:27:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.015+0000 7f6ee6000700 1 -- start start 2024-04-26T11:27:39.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.015+0000 7f6ee6000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee006bb00 0x7f6ee01023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:39.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.015+0000 7f6ee6000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ee006c4a0 0x7f6ee0102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:39.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.015+0000 7f6ee6000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee01094a0 0x7f6ee0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:39.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.015+0000 7f6ee6000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee0113600 con 0x7f6ee006bb00 2024-04-26T11:27:39.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.015+0000 7f6ee6000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee0113480 con 0x7f6ee006c4a0 2024-04-26T11:27:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.015+0000 7f6ee6000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee0113780 con 0x7f6ee01094a0 2024-04-26T11:27:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee006bb00 0x7f6ee01023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee006bb00 0x7f6ee01023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43726/0 (socket says 172.21.15.33:43726) 2024-04-26T11:27:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 -- 172.21.15.33:0/2536598831 learned_addr learned my addr 172.21.15.33:0/2536598831 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edfe00700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee01094a0 0x7f6ee0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:39.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edea00700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ee006c4a0 0x7f6ee0102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 -- 172.21.15.33:0/2536598831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee01094a0 msgr2=0x7f6ee0102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:39.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee01094a0 0x7f6ee0102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 -- 172.21.15.33:0/2536598831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ee006c4a0 msgr2=0x7f6ee0102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:39.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ee006c4a0 0x7f6ee0102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.016+0000 7f6edf400700 1 -- 172.21.15.33:0/2536598831 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee01a8040 con 0x7f6ee006bb00 2024-04-26T11:27:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.017+0000 7f6edf400700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee006bb00 0x7f6ee01023f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000e9a0 tx=0x7f6ed000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.017+0000 7f6ed7e00700 1 -- 172.21.15.33:0/2536598831 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000ca10 con 0x7f6ee006bb00 2024-04-26T11:27:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.017+0000 7f6ed7e00700 1 -- 172.21.15.33:0/2536598831 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ed000cbb0 con 0x7f6ee006bb00 2024-04-26T11:27:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.017+0000 7f6ed7e00700 1 -- 172.21.15.33:0/2536598831 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0018680 con 0x7f6ee006bb00 2024-04-26T11:27:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.017+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ee01a8330 con 0x7f6ee006bb00 2024-04-26T11:27:39.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.017+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ee01a88c0 con 0x7f6ee006bb00 2024-04-26T11:27:39.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.018+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ee0104b50 con 0x7f6ee006bb00 2024-04-26T11:27:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.020+0000 7f6ed7e00700 1 -- 172.21.15.33:0/2536598831 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6ed0010430 con 0x7f6ee006bb00 2024-04-26T11:27:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.023+0000 7f6ed7e00700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6ec40a3d50 0x7f6ec40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.023+0000 7f6ed7e00700 1 -- 172.21.15.33:0/2536598831 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f6ed00d0660 con 0x7f6ee006bb00 2024-04-26T11:27:39.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.023+0000 7f6edea00700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6ec40a3d50 0x7f6ec40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:39.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.024+0000 7f6edea00700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6ec40a3d50 0x7f6ec40a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00798e0 tx=0x7f6ecc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:39.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.028+0000 7f6ed7e00700 1 -- 172.21.15.33:0/2536598831 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ed009cea0 con 0x7f6ee006bb00 2024-04-26T11:27:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:38 smithi033 ceph-mon[27845]: pgmap v360: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:38 smithi096 ceph-mon[32753]: pgmap v360: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:39.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.327+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f6ee002e490 con 0x7f6ee006bb00 2024-04-26T11:27:39.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.336+0000 7f6ed7e00700 1 -- 172.21.15.33:0/2536598831 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264995 (secure 0 0 0) 0x7f6ed009c7a0 con 0x7f6ee006bb00 2024-04-26T11:27:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.343+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6ec40a3d50 msgr2=0x7f6ec40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.343+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6ec40a3d50 0x7f6ec40a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00798e0 tx=0x7f6ecc005fd0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.343+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee006bb00 msgr2=0x7f6ee01023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.343+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee006bb00 0x7f6ee01023f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000e9a0 tx=0x7f6ed000ecb0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 shutdown_connections 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f6ec40a3d50 0x7f6ec40a6210 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ee01094a0 0x7f6ee0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ee006bb00 0x7f6ee01023f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 --2- 172.21.15.33:0/2536598831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ee006c4a0 0x7f6ee0102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:39.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 >> 172.21.15.33:0/2536598831 conn(0x7f6ee0079290 msgr2=0x7f6ee0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:39.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 shutdown_connections 2024-04-26T11:27:39.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:39.345+0000 7f6ee6000700 1 -- 172.21.15.33:0/2536598831 wait complete. 2024-04-26T11:27:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:38 smithi194 ceph-mon[33017]: pgmap v360: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:39 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2536598831' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T11:27:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:39 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2536598831' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T11:27:40.310 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":657098496},{"type":"v1","addr":"172.21.15.33:6801","nonce":657098496}]},"active_addr":"172.21.15.33:6801/657098496","active_change":"2024-04-26T11:18:17.661028+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.33:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1763622610}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1152006795}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1149083719}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3707727076}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":2763871516}]}]} 2024-04-26T11:27:40.315 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T11:27:40.315 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T11:27:40.316 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:27:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:39 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2536598831' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T11:27:40.821 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:41.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:40 smithi033 ceph-mon[27845]: pgmap v361: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T11:27:41.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:40 smithi096 ceph-mon[32753]: pgmap v361: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T11:27:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:40 smithi194 ceph-mon[33017]: pgmap v361: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 -- 172.21.15.33:0/4210038446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06b130 msgr2=0x7f0e9c06b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 --2- 172.21.15.33:0/4210038446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06b130 0x7f0e9c06b510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009990 tx=0x7f0e8c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 -- 172.21.15.33:0/4210038446 shutdown_connections 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 --2- 172.21.15.33:0/4210038446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e9c10fed0 0x7f0e9c1122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 --2- 172.21.15.33:0/4210038446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06b130 0x7f0e9c06b510 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 --2- 172.21.15.33:0/4210038446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e9c06ba50 0x7f0e9c10f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 -- 172.21.15.33:0/4210038446 >> 172.21.15.33:0/4210038446 conn(0x7f0e9c0792d0 msgr2=0x7f0e9c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 -- 172.21.15.33:0/4210038446 shutdown_connections 2024-04-26T11:27:43.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.058+0000 7f0ea3000700 1 -- 172.21.15.33:0/4210038446 wait complete. 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 Processor -- start 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 -- start start 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e9c06b130 0x7f0e9c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06ba50 0x7f0e9c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e9c10fed0 0x7f0e9c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c115830 con 0x7f0e9c06ba50 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c1156b0 con 0x7f0e9c10fed0 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0ea3000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c1159b0 con 0x7f0e9c06b130 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0e9b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06ba50 0x7f0e9c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0e9b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06ba50 0x7f0e9c19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54128/0 (socket says 172.21.15.33:54128) 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.060+0000 7f0e9b400700 1 -- 172.21.15.33:0/2906880779 learned_addr learned my addr 172.21.15.33:0/2906880779 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0e9be00700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e9c06b130 0x7f0e9c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0ea1000700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e9c10fed0 0x7f0e9c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0e9b400700 1 -- 172.21.15.33:0/2906880779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e9c06b130 msgr2=0x7f0e9c19bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0e9b400700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e9c06b130 0x7f0e9c19bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0e9b400700 1 -- 172.21.15.33:0/2906880779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e9c10fed0 msgr2=0x7f0e9c1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0e9b400700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e9c10fed0 0x7f0e9c1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0e9b400700 1 -- 172.21.15.33:0/2906880779 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9c1a0d50 con 0x7f0e9c06ba50 2024-04-26T11:27:43.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.061+0000 7f0e9b400700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06ba50 0x7f0e9c19c280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800e9a0 tx=0x7f0e8800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:43.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.062+0000 7f0e98c00700 1 -- 172.21.15.33:0/2906880779 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8800c9c0 con 0x7f0e9c06ba50 2024-04-26T11:27:43.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.062+0000 7f0e98c00700 1 -- 172.21.15.33:0/2906880779 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e8800cb60 con 0x7f0e9c06ba50 2024-04-26T11:27:43.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.062+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9c1a1040 con 0x7f0e9c06ba50 2024-04-26T11:27:43.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.062+0000 7f0e98c00700 1 -- 172.21.15.33:0/2906880779 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88018680 con 0x7f0e9c06ba50 2024-04-26T11:27:43.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.062+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e9c1a88c0 con 0x7f0e9c06ba50 2024-04-26T11:27:43.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.063+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e9c06a110 con 0x7f0e9c06ba50 2024-04-26T11:27:43.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.066+0000 7f0e98c00700 1 -- 172.21.15.33:0/2906880779 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0e88010430 con 0x7f0e9c06ba50 2024-04-26T11:27:43.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.068+0000 7f0e98c00700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0e800a3cd0 0x7f0e800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:43.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.068+0000 7f0e98c00700 1 -- 172.21.15.33:0/2906880779 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0e880d0880 con 0x7f0e9c06ba50 2024-04-26T11:27:43.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.068+0000 7f0e9be00700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0e800a3cd0 0x7f0e800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:43.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.069+0000 7f0e9be00700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0e800a3cd0 0x7f0e800a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c006010 tx=0x7f0e8c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:43.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.073+0000 7f0e98c00700 1 -- 172.21.15.33:0/2906880779 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e8809d0c0 con 0x7f0e9c06ba50 2024-04-26T11:27:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:42 smithi033 ceph-mon[27845]: pgmap v362: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:43.179 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:42 smithi194 ceph-mon[33017]: pgmap v362: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:42 smithi096 ceph-mon[32753]: pgmap v362: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:43.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.306+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e9c02e490 con 0x7f0e9c06ba50 2024-04-26T11:27:43.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.310+0000 7f0e98c00700 1 -- 172.21.15.33:0/2906880779 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22592 (secure 0 0 0) 0x7f0e8809c9c0 con 0x7f0e9c06ba50 2024-04-26T11:27:43.312 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:27:43.312 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:27:19.230568+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:27:43.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.313+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0e800a3cd0 msgr2=0x7f0e800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:43.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.313+0000 7f0ea3000700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0e800a3cd0 0x7f0e800a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c006010 tx=0x7f0e8c00b260 comp rx=0 tx=0).stop 2024-04-26T11:27:43.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.313+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06ba50 msgr2=0x7f0e9c19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.313+0000 7f0ea3000700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06ba50 0x7f0e9c19c280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800e9a0 tx=0x7f0e8800ecb0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 shutdown_connections 2024-04-26T11:27:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0e800a3cd0 0x7f0e800a6190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e9c10fed0 0x7f0e9c1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e9c06b130 0x7f0e9c19bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 --2- 172.21.15.33:0/2906880779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e9c06ba50 0x7f0e9c19c280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 >> 172.21.15.33:0/2906880779 conn(0x7f0e9c0792d0 msgr2=0x7f0e9c1113e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 shutdown_connections 2024-04-26T11:27:43.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:43.314+0000 7f0ea3000700 1 -- 172.21.15.33:0/2906880779 wait complete. 2024-04-26T11:27:44.171 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:43 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2906880779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:43 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2906880779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:44.256 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T11:27:44.257 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:27:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:43 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2906880779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:44.730 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:44 smithi033 ceph-mon[27845]: pgmap v363: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:44 smithi096 ceph-mon[32753]: pgmap v363: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:44 smithi194 ceph-mon[33017]: pgmap v363: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 -- 172.21.15.33:0/2529109128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 msgr2=0x7f77e006bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 --2- 172.21.15.33:0/2529109128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 0x7f77e006bad0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009990 tx=0x7f77d0009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 -- 172.21.15.33:0/2529109128 shutdown_connections 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 --2- 172.21.15.33:0/2529109128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 0x7f77e01062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 --2- 172.21.15.33:0/2529109128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e0108350 0x7f77e006b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 --2- 172.21.15.33:0/2529109128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 0x7f77e006bad0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 -- 172.21.15.33:0/2529109128 >> 172.21.15.33:0/2529109128 conn(0x7f77e00792d0 msgr2=0x7f77e00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:47.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 -- 172.21.15.33:0/2529109128 shutdown_connections 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.025+0000 7f77e7600700 1 -- 172.21.15.33:0/2529109128 wait complete. 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.026+0000 7f77e7600700 1 Processor -- start 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e7600700 1 -- start start 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e7600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 0x7f77e01a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e7600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 0x7f77e01a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e7600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e0108350 0x7f77e01a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e7600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f77e0119c40 con 0x7f77e0105e60 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e7600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f77e0119ac0 con 0x7f77e0108350 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e7600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f77e0119dc0 con 0x7f77e006b670 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e4c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 0x7f77e01a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77d7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 0x7f77e01a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77d7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 0x7f77e01a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54148/0 (socket says 172.21.15.33:54148) 2024-04-26T11:27:47.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77d7e00700 1 -- 172.21.15.33:0/2089042694 learned_addr learned my addr 172.21.15.33:0/2089042694 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.027+0000 7f77e4c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 0x7f77e01a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:33820/0 (socket says 172.21.15.33:33820) 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.028+0000 7f77e5600700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e0108350 0x7f77e01a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.028+0000 7f77d7e00700 1 -- 172.21.15.33:0/2089042694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 msgr2=0x7f77e01a00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.028+0000 7f77d7e00700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 0x7f77e01a00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.028+0000 7f77d7e00700 1 -- 172.21.15.33:0/2089042694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e0108350 msgr2=0x7f77e01a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.028+0000 7f77d7e00700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e0108350 0x7f77e01a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.028+0000 7f77d7e00700 1 -- 172.21.15.33:0/2089042694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e01a5100 con 0x7f77e0105e60 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.028+0000 7f77d7e00700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 0x7f77e01a0630 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f77d0000c00 tx=0x7f77d00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.029+0000 7f77de000700 1 -- 172.21.15.33:0/2089042694 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0027070 con 0x7f77e0105e60 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.029+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e01a5390 con 0x7f77e0105e60 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.029+0000 7f77de000700 1 -- 172.21.15.33:0/2089042694 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d0022470 con 0x7f77e0105e60 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.029+0000 7f77de000700 1 -- 172.21.15.33:0/2089042694 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0017650 con 0x7f77e0105e60 2024-04-26T11:27:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.029+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e01acc70 con 0x7f77e0105e60 2024-04-26T11:27:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.030+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e006a110 con 0x7f77e0105e60 2024-04-26T11:27:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.033+0000 7f77de000700 1 -- 172.21.15.33:0/2089042694 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f77d002a070 con 0x7f77e0105e60 2024-04-26T11:27:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.035+0000 7f77de000700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f77c00a3cd0 0x7f77c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.035+0000 7f77de000700 1 -- 172.21.15.33:0/2089042694 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f77d00d1300 con 0x7f77e0105e60 2024-04-26T11:27:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.035+0000 7f77e4c00700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f77c00a3cd0 0x7f77c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.036+0000 7f77e4c00700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f77c00a3cd0 0x7f77c00a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009be0 tx=0x7f77c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:47.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.038+0000 7f77de000700 1 -- 172.21.15.33:0/2089042694 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77d0013070 con 0x7f77e0105e60 2024-04-26T11:27:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:46 smithi033 ceph-mon[27845]: pgmap v364: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:46 smithi096 ceph-mon[32753]: pgmap v364: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:47.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.269+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77e01ace40 con 0x7f77e0105e60 2024-04-26T11:27:47.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.273+0000 7f77de000700 1 -- 172.21.15.33:0/2089042694 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22592 (secure 0 0 0) 0x7f77d009dbd0 con 0x7f77e0105e60 2024-04-26T11:27:47.275 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:27:47.275 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:27:19.230568+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.276+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f77c00a3cd0 msgr2=0x7f77c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.276+0000 7f77e7600700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f77c00a3cd0 0x7f77c00a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009be0 tx=0x7f77c8009260 comp rx=0 tx=0).stop 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.276+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 msgr2=0x7f77e01a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.276+0000 7f77e7600700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 0x7f77e01a0630 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f77d0000c00 tx=0x7f77d00057c0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 shutdown_connections 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f77c00a3cd0 0x7f77c00a6190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e0108350 0x7f77e01a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77e006b670 0x7f77e01a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 --2- 172.21.15.33:0/2089042694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e0105e60 0x7f77e01a0630 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 >> 172.21.15.33:0/2089042694 conn(0x7f77e00792d0 msgr2=0x7f77e0104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 shutdown_connections 2024-04-26T11:27:47.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:47.277+0000 7f77e7600700 1 -- 172.21.15.33:0/2089042694 wait complete. 2024-04-26T11:27:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:46 smithi194 ceph-mon[33017]: pgmap v364: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:27:48.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T11:27:48.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T11:27:48.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T11:27:48.125 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T11:27:48.125 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T11:27:48.126 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T11:27:48.126 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T11:27:48.126 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T11:27:48.127 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T11:27:48.127 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T11:27:48.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T11:27:48.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T11:27:48.240 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:47 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2089042694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:47 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2089042694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:47 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2089042694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:27:49.239 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:49.277 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:49.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:48 smithi033 ceph-mon[27845]: pgmap v365: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:48 smithi194 ceph-mon[33017]: pgmap v365: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:49.431 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:48 smithi096 ceph-mon[32753]: pgmap v365: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:49.437 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:49.440 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:50.880 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:51.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:50 smithi033 ceph-mon[27845]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:27:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:50 smithi194 ceph-mon[33017]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:27:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:50 smithi096 ceph-mon[32753]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:27:53.124 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:53.125 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:53.134 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:53.139 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:53.140 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:27:53.265 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:52 smithi033 ceph-mon[27845]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:27:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:52 smithi194 ceph-mon[33017]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:27:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:52 smithi096 ceph-mon[32753]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:27:54.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.692+0000 7fba14400700 1 -- 172.21.15.33:0/2748266528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075650 msgr2=0x7fba0c075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:54.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.692+0000 7fba14400700 1 --2- 172.21.15.33:0/2748266528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075650 0x7fba0c075a30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba00009990 tx=0x7fba00009ca0 comp rx=0 tx=0).stop 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.693+0000 7fba14400700 1 -- 172.21.15.33:0/2748266528 shutdown_connections 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.693+0000 7fba14400700 1 --2- 172.21.15.33:0/2748266528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba0c10fe20 0x7fba0c112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.693+0000 7fba14400700 1 --2- 172.21.15.33:0/2748266528 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075650 0x7fba0c075a30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.693+0000 7fba14400700 1 --2- 172.21.15.33:0/2748266528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba0c075f70 0x7fba0c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.693+0000 7fba14400700 1 -- 172.21.15.33:0/2748266528 >> 172.21.15.33:0/2748266528 conn(0x7fba0c07a580 msgr2=0x7fba0c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.693+0000 7fba14400700 1 -- 172.21.15.33:0/2748266528 shutdown_connections 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.693+0000 7fba14400700 1 -- 172.21.15.33:0/2748266528 wait complete. 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 Processor -- start 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 -- start start 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba0c075650 0x7fba0c1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075f70 0x7fba0c1a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba0c10fe20 0x7fba0c19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c115850 con 0x7fba0c075f70 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1156d0 con 0x7fba0c10fe20 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.694+0000 7fba14400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1159d0 con 0x7fba0c075650 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075f70 0x7fba0c1a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075f70 0x7fba0c1a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55356/0 (socket says 172.21.15.33:55356) 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 -- 172.21.15.33:0/167563965 learned_addr learned my addr 172.21.15.33:0/167563965 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba12400700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba0c10fe20 0x7fba0c19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11a00700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba0c075650 0x7fba0c1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 -- 172.21.15.33:0/167563965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba0c075650 msgr2=0x7fba0c1a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba0c075650 0x7fba0c1a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 -- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba0c10fe20 msgr2=0x7fba0c19bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba0c10fe20 0x7fba0c19bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11000700 1 -- 172.21.15.33:0/167563965 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c19c410 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba11a00700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba0c075650 0x7fba0c1a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.695+0000 7fba12400700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba0c10fe20 0x7fba0c19bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.696+0000 7fba11000700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075f70 0x7fba0c1a1fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800d980 tx=0x7fb9f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.696+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8016070 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.696+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f8004830 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.696+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8005540 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.696+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c19c700 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.696+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c1a89e0 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.697+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba0c063b70 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.699+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb9f8004220 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.701+0000 7fba0a000700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.701+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb9f80d09d0 con 0x7fba0c075f70 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.701+0000 7fba11a00700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:54.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.701+0000 7fba0a000700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] conn(0x7fb9f40ad890 0x7fb9f40afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:54.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.701+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 --> [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9f40b0400 con 0x7fb9f40ad890 2024-04-26T11:27:54.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.701+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb9f80d0d80 con 0x7fba0c075f70 2024-04-26T11:27:54.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.702+0000 7fba11a00700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba00006010 tx=0x7fba0000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:54.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.702+0000 7fba12400700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] conn(0x7fb9f40ad890 0x7fb9f40afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:54.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.702+0000 7fba12400700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] conn(0x7fb9f40ad890 0x7fb9f40afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:54.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.703+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== osd.7 v2:172.21.15.96:6816/1631656381 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9f40b0400 con 0x7fb9f40ad890 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.722+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 --> [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fba0c063ff0 con 0x7fb9f40ad890 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.722+0000 7fba0a000700 1 -- 172.21.15.33:0/167563965 <== osd.7 v2:172.21.15.96:6816/1631656381 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fba0c063ff0 con 0x7fb9f40ad890 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] conn(0x7fb9f40ad890 msgr2=0x7fb9f40afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] conn(0x7fb9f40ad890 0x7fb9f40afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb9f40a3cd0 msgr2=0x7fb9f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba00006010 tx=0x7fba0000b260 comp rx=0 tx=0).stop 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075f70 msgr2=0x7fba0c1a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 --2- 172.21.15.33:0/167563965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba0c075f70 0x7fba0c1a1fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800d980 tx=0x7fb9f800dc90 comp rx=0 tx=0).stop 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba12400700 1 -- 172.21.15.33:0/167563965 reap_dead start 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 shutdown_connections 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.723+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 >> 172.21.15.33:0/167563965 conn(0x7fba0c07a580 msgr2=0x7fba0c100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:54.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.724+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 shutdown_connections 2024-04-26T11:27:54.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:53.724+0000 7fba14400700 1 -- 172.21.15.33:0/167563965 wait complete. 2024-04-26T11:27:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:54 smithi033 ceph-mon[27845]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:54 smithi194 ceph-mon[33017]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:54 smithi096 ceph-mon[32753]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:57.226 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:56 smithi033 ceph-mon[27845]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:27:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:56 smithi194 ceph-mon[33017]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:27:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:56 smithi096 ceph-mon[32753]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 -- 172.21.15.33:0/1779938798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075f70 msgr2=0x7f630410f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 --2- 172.21.15.33:0/1779938798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075f70 0x7f630410f8e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f62f00098f0 tx=0x7f62f0009c00 comp rx=0 tx=0).stop 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 -- 172.21.15.33:0/1779938798 shutdown_connections 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 --2- 172.21.15.33:0/1779938798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f630410fe20 0x7f6304112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 --2- 172.21.15.33:0/1779938798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6304075650 0x7f6304075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 --2- 172.21.15.33:0/1779938798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075f70 0x7f630410f8e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 -- 172.21.15.33:0/1779938798 >> 172.21.15.33:0/1779938798 conn(0x7f630407a580 msgr2=0x7f630407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.745+0000 7f6309c00700 1 -- 172.21.15.33:0/1779938798 shutdown_connections 2024-04-26T11:27:57.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.746+0000 7f6309c00700 1 -- 172.21.15.33:0/1779938798 wait complete. 2024-04-26T11:27:57.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 Processor -- start 2024-04-26T11:27:57.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 -- start start 2024-04-26T11:27:57.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075650 0x7f630419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:57.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304075f70 0x7f630419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:57.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410fe20 0x7f63041a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:57.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6304115850 con 0x7f6304075650 2024-04-26T11:27:57.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f63041156d0 con 0x7f630410fe20 2024-04-26T11:27:57.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.747+0000 7f6309c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f63041159d0 con 0x7f6304075f70 2024-04-26T11:27:57.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304075f70 0x7f630419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:57.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075650 0x7f630419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304075f70 0x7f630419c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52844/0 (socket says 172.21.15.33:52844) 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302000700 1 -- 172.21.15.33:0/1371322120 learned_addr learned my addr 172.21.15.33:0/1371322120 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6303400700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410fe20 0x7f63041a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302a00700 1 -- 172.21.15.33:0/1371322120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304075f70 msgr2=0x7f630419c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302a00700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304075f70 0x7f630419c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302a00700 1 -- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410fe20 msgr2=0x7f63041a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302a00700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410fe20 0x7f63041a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.748+0000 7f6302a00700 1 -- 172.21.15.33:0/1371322120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63041a0d30 con 0x7f6304075650 2024-04-26T11:27:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.749+0000 7f6302a00700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075650 0x7f630419bd20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62f400e9c0 tx=0x7f62f400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:57.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.749+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f400c9c0 con 0x7f6304075650 2024-04-26T11:27:57.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.749+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62f400cb60 con 0x7f6304075650 2024-04-26T11:27:57.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.749+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63041a1020 con 0x7f6304075650 2024-04-26T11:27:57.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.749+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f4018420 con 0x7f6304075650 2024-04-26T11:27:57.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.749+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63041a88a0 con 0x7f6304075650 2024-04-26T11:27:57.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.750+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6304063bb0 con 0x7f6304075650 2024-04-26T11:27:57.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.752+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f62f4010430 con 0x7f6304075650 2024-04-26T11:27:57.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.754+0000 7f62fb400700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3d50 0x7f62e80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:57.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.754+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f62f40d0e30 con 0x7f6304075650 2024-04-26T11:27:57.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.754+0000 7f6302000700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3d50 0x7f62e80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:57.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.754+0000 7f62fb400700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] conn(0x7f62e80ad910 0x7f62e80afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:27:57.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.754+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 --> [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62e80b0480 con 0x7f62e80ad910 2024-04-26T11:27:57.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.754+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f62f40d11e0 con 0x7f6304075650 2024-04-26T11:27:57.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.755+0000 7f6303400700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] conn(0x7f62e80ad910 0x7f62e80afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:27:57.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.755+0000 7f6302000700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3d50 0x7f62e80a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6304102fe0 tx=0x7f62f0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:57.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.755+0000 7f6303400700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] conn(0x7f62e80ad910 0x7f62e80afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:27:57.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.756+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== osd.1 v2:172.21.15.96:6800/2891673076 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62e80b0480 con 0x7f62e80ad910 2024-04-26T11:27:57.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.774+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 --> [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6304063fe0 con 0x7f62e80ad910 2024-04-26T11:27:57.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.775+0000 7f62fb400700 1 -- 172.21.15.33:0/1371322120 <== osd.1 v2:172.21.15.96:6800/2891673076 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6304063fe0 con 0x7f62e80ad910 2024-04-26T11:27:57.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] conn(0x7f62e80ad910 msgr2=0x7f62e80afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:57.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] conn(0x7f62e80ad910 0x7f62e80afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:27:57.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3d50 msgr2=0x7f62e80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:57.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3d50 0x7f62e80a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6304102fe0 tx=0x7f62f0019040 comp rx=0 tx=0).stop 2024-04-26T11:27:57.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075650 msgr2=0x7f630419bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:27:57.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 --2- 172.21.15.33:0/1371322120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304075650 0x7f630419bd20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62f400e9c0 tx=0x7f62f400ecd0 comp rx=0 tx=0).stop 2024-04-26T11:27:57.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6303400700 1 -- 172.21.15.33:0/1371322120 reap_dead start 2024-04-26T11:27:57.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 shutdown_connections 2024-04-26T11:27:57.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 >> 172.21.15.33:0/1371322120 conn(0x7f630407a580 msgr2=0x7f6304100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:27:57.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 shutdown_connections 2024-04-26T11:27:57.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:27:57.776+0000 7f6309c00700 1 -- 172.21.15.33:0/1371322120 wait complete. 2024-04-26T11:27:59.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:27:58 smithi033 ceph-mon[27845]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:27:58 smithi194 ceph-mon[33017]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:27:58 smithi096 ceph-mon[32753]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:27:59.481 INFO:teuthology.orchestra.run.smithi033.stdout:55834574939 2024-04-26T11:27:59.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T11:28:00.665 INFO:teuthology.orchestra.run.smithi033.stdout:219043332152 2024-04-26T11:28:00.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T11:28:01.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:00 smithi033 ceph-mon[27845]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:00 smithi194 ceph-mon[33017]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:00 smithi096 ceph-mon[32753]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:02.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 -- 172.21.15.33:0/2360250891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 msgr2=0x7fcdb810ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:02.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 --2- 172.21.15.33:0/2360250891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 0x7fcdb810ba00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009990 tx=0x7fcda8009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 -- 172.21.15.33:0/2360250891 shutdown_connections 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 --2- 172.21.15.33:0/2360250891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcdb8105fe0 0x7fcdb81124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 --2- 172.21.15.33:0/2360250891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 0x7fcdb810ba00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 --2- 172.21.15.33:0/2360250891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdb8105620 0x7fcdb8105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 -- 172.21.15.33:0/2360250891 >> 172.21.15.33:0/2360250891 conn(0x7fcdb8101130 msgr2=0x7fcdb8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 -- 172.21.15.33:0/2360250891 shutdown_connections 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.395+0000 7fcdbea00700 1 -- 172.21.15.33:0/2360250891 wait complete. 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.396+0000 7fcdbea00700 1 Processor -- start 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.396+0000 7fcdbea00700 1 -- start start 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcdb8105620 0x7fcdb81a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdb8105fe0 0x7fcdb81a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 0x7fcdb819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:02.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbea00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8115800 con 0x7fcdb810b620 2024-04-26T11:28:02.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbea00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8115680 con 0x7fcdb8105fe0 2024-04-26T11:28:02.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbea00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8115980 con 0x7fcdb8105620 2024-04-26T11:28:02.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbca00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 0x7fcdb819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:02.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbca00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 0x7fcdb819bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36500/0 (socket says 172.21.15.33:36500) 2024-04-26T11:28:02.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdbca00700 1 -- 172.21.15.33:0/1381928733 learned_addr learned my addr 172.21.15.33:0/1381928733 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:02.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdb7e00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcdb8105620 0x7fcdb81a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:02.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.397+0000 7fcdb7400700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdb8105fe0 0x7fcdb81a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbca00700 1 -- 172.21.15.33:0/1381928733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcdb8105620 msgr2=0x7fcdb81a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbca00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcdb8105620 0x7fcdb81a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbca00700 1 -- 172.21.15.33:0/1381928733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdb8105fe0 msgr2=0x7fcdb81a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbca00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdb8105fe0 0x7fcdb81a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbca00700 1 -- 172.21.15.33:0/1381928733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb819c260 con 0x7fcdb810b620 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbca00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 0x7fcdb819bb00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000ef30 tx=0x7fcdb000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb00092e0 con 0x7fcdb810b620 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb819c4f0 con 0x7fcdb810b620 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb0004bb0 con 0x7fcdb810b620 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0004020 con 0x7fcdb810b620 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.398+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81a87c0 con 0x7fcdb810b620 2024-04-26T11:28:02.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.399+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcdb8063bb0 con 0x7fcdb810b620 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.401+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcdb001b030 con 0x7fcdb810b620 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.403+0000 7fcdb4c00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcd9c0a3cd0 0x7fcd9c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.403+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcdb00d0990 con 0x7fcdb810b620 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.403+0000 7fcdb7e00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcd9c0a3cd0 0x7fcd9c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.403+0000 7fcdb4c00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] conn(0x7fcd9c0ad890 0x7fcd9c0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.403+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 --> [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd9c0b0400 con 0x7fcd9c0ad890 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.403+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fcdb00d0d40 con 0x7fcdb810b620 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.404+0000 7fcdb7400700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] conn(0x7fcd9c0ad890 0x7fcd9c0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.404+0000 7fcdb7e00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcd9c0a3cd0 0x7fcd9c0a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcda8006010 tx=0x7fcda801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.404+0000 7fcdb7400700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] conn(0x7fcd9c0ad890 0x7fcd9c0afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.405+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== osd.3 v2:172.21.15.33:6810/2572626709 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcd9c0b0400 con 0x7fcd9c0ad890 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.424+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 --> [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcdb8064030 con 0x7fcd9c0ad890 2024-04-26T11:28:02.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.424+0000 7fcdb4c00700 1 -- 172.21.15.33:0/1381928733 <== osd.3 v2:172.21.15.33:6810/2572626709 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcdb8064030 con 0x7fcd9c0ad890 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] conn(0x7fcd9c0ad890 msgr2=0x7fcd9c0afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] conn(0x7fcd9c0ad890 0x7fcd9c0afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcd9c0a3cd0 msgr2=0x7fcd9c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcd9c0a3cd0 0x7fcd9c0a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcda8006010 tx=0x7fcda801a040 comp rx=0 tx=0).stop 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 msgr2=0x7fcdb819bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 --2- 172.21.15.33:0/1381928733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcdb810b620 0x7fcdb819bb00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000ef30 tx=0x7fcdb000c3a0 comp rx=0 tx=0).stop 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbca00700 1 -- 172.21.15.33:0/1381928733 reap_dead start 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 shutdown_connections 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 >> 172.21.15.33:0/1381928733 conn(0x7fcdb8101130 msgr2=0x7fcdb8101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 shutdown_connections 2024-04-26T11:28:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:01.425+0000 7fcdbea00700 1 -- 172.21.15.33:0/1381928733 wait complete. 2024-04-26T11:28:02.498 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:03.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:02 smithi033 ceph-mon[27845]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:02 smithi194 ceph-mon[33017]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:02 smithi096 ceph-mon[32753]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:05.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:04 smithi033 ceph-mon[27845]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:04 smithi194 ceph-mon[33017]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:05.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 -- 172.21.15.33:0/4213898343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e41094c0 msgr2=0x7f90e41098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 --2- 172.21.15.33:0/4213898343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e41094c0 0x7f90e41098a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009990 tx=0x7f90d4009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 -- 172.21.15.33:0/4213898343 shutdown_connections 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 --2- 172.21.15.33:0/4213898343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90e4103e30 0x7f90e4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 --2- 172.21.15.33:0/4213898343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e41094c0 0x7f90e41098a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 --2- 172.21.15.33:0/4213898343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90e4103470 0x7f90e41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 -- 172.21.15.33:0/4213898343 >> 172.21.15.33:0/4213898343 conn(0x7f90e407a580 msgr2=0x7f90e407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 -- 172.21.15.33:0/4213898343 shutdown_connections 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.337+0000 7f90e8e00700 1 -- 172.21.15.33:0/4213898343 wait complete. 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.338+0000 7f90e8e00700 1 Processor -- start 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e8e00700 1 -- start start 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e8e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e4103470 0x7f90e419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e8e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90e4103e30 0x7f90e419c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:05.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e8e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90e41094c0 0x7f90e41a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e8e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f90e4113610 con 0x7f90e4103470 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e8e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f90e4113490 con 0x7f90e41094c0 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e8e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f90e4113790 con 0x7f90e4103e30 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e1e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e4103470 0x7f90e419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e1400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90e4103e30 0x7f90e419c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e1e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e4103470 0x7f90e419be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36516/0 (socket says 172.21.15.33:36516) 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e1e00700 1 -- 172.21.15.33:0/4016641356 learned_addr learned my addr 172.21.15.33:0/4016641356 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.339+0000 7f90e1400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90e4103e30 0x7f90e419c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:59280/0 (socket says 172.21.15.33:59280) 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.340+0000 7f90e2800700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90e41094c0 0x7f90e41a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.340+0000 7f90e1e00700 1 -- 172.21.15.33:0/4016641356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90e4103e30 msgr2=0x7f90e419c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.340+0000 7f90e1e00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90e4103e30 0x7f90e419c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.340+0000 7f90e1e00700 1 -- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90e41094c0 msgr2=0x7f90e41a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:05.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.340+0000 7f90e1e00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90e41094c0 0x7f90e41a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.340+0000 7f90e1e00700 1 -- 172.21.15.33:0/4016641356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e41a0e10 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.340+0000 7f90e1e00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e4103470 0x7f90e419be00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f90d4000c00 tx=0x7f90d40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.341+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4027070 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.341+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90d4022470 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.341+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4017650 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.341+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e41a10a0 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.341+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e41a8980 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.342+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f90e4076930 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.347+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f90d402a070 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.349+0000 7f90daa00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f90c80a3cd0 0x7f90c80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.349+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f90d40d18c0 con 0x7f90e4103470 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.349+0000 7f90e1400700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f90c80a3cd0 0x7f90c80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.349+0000 7f90daa00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] conn(0x7f90c80ad760 0x7f90c80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.349+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 --> [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90c80b02d0 con 0x7f90c80ad760 2024-04-26T11:28:05.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.349+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f90d40d1c70 con 0x7f90e4103470 2024-04-26T11:28:05.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.350+0000 7f90e2800700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] conn(0x7f90c80ad760 0x7f90c80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:05.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.350+0000 7f90e1400700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f90c80a3cd0 0x7f90c80a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009680 tx=0x7f90d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:05.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.350+0000 7f90e2800700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] conn(0x7f90c80ad760 0x7f90c80afc00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:05.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.351+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== osd.10 v2:172.21.15.96:6824/2356422092 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f90c80b02d0 con 0x7f90c80ad760 2024-04-26T11:28:05.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.370+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 --> [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f90e4104b60 con 0x7f90c80ad760 2024-04-26T11:28:05.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90daa00700 1 -- 172.21.15.33:0/4016641356 <== osd.10 v2:172.21.15.96:6824/2356422092 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f90e4104b60 con 0x7f90c80ad760 2024-04-26T11:28:05.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] conn(0x7f90c80ad760 msgr2=0x7f90c80afc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:05.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] conn(0x7f90c80ad760 0x7f90c80afc00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f90c80a3cd0 msgr2=0x7f90c80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f90c80a3cd0 0x7f90c80a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009680 tx=0x7f90d0009260 comp rx=0 tx=0).stop 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e4103470 msgr2=0x7f90e419be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 --2- 172.21.15.33:0/4016641356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f90e4103470 0x7f90e419be00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f90d4000c00 tx=0x7f90d40057c0 comp rx=0 tx=0).stop 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e2800700 1 -- 172.21.15.33:0/4016641356 reap_dead start 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 shutdown_connections 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.371+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 >> 172.21.15.33:0/4016641356 conn(0x7f90e407a580 msgr2=0x7f90e4102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.372+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 shutdown_connections 2024-04-26T11:28:05.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:05.372+0000 7f90e8e00700 1 -- 172.21.15.33:0/4016641356 wait complete. 2024-04-26T11:28:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:04 smithi096 ceph-mon[32753]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.338+0000 7fb7cdc00700 1 -- 172.21.15.33:0/774901062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7c8103470 msgr2=0x7fb7c81038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.338+0000 7fb7cdc00700 1 --2- 172.21.15.33:0/774901062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7c8103470 0x7fb7c81038f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009990 tx=0x7fb7b4009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.339+0000 7fb7cdc00700 1 -- 172.21.15.33:0/774901062 shutdown_connections 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.339+0000 7fb7cdc00700 1 --2- 172.21.15.33:0/774901062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7c8103e30 0x7fb7c8110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.339+0000 7fb7cdc00700 1 --2- 172.21.15.33:0/774901062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c81094c0 0x7fb7c81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.339+0000 7fb7cdc00700 1 --2- 172.21.15.33:0/774901062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7c8103470 0x7fb7c81038f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.339+0000 7fb7cdc00700 1 -- 172.21.15.33:0/774901062 >> 172.21.15.33:0/774901062 conn(0x7fb7c807a580 msgr2=0x7fb7c807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.339+0000 7fb7cdc00700 1 -- 172.21.15.33:0/774901062 shutdown_connections 2024-04-26T11:28:06.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.339+0000 7fb7cdc00700 1 -- 172.21.15.33:0/774901062 wait complete. 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 Processor -- start 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 -- start start 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103470 0x7fb7c819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7c8103e30 0x7fb7c819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7c81094c0 0x7fb7c81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8113640 con 0x7fb7c8103470 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81134c0 con 0x7fb7c81094c0 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.340+0000 7fb7cdc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81137c0 con 0x7fb7c8103e30 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103470 0x7fb7c819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103470 0x7fb7c819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36538/0 (socket says 172.21.15.33:36538) 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 -- 172.21.15.33:0/93005185 learned_addr learned my addr 172.21.15.33:0/93005185 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c7400700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7c81094c0 0x7fb7c81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6000700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7c8103e30 0x7fb7c819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 -- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7c8103e30 msgr2=0x7fb7c819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7c8103e30 0x7fb7c819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 -- 172.21.15.33:0/93005185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7c81094c0 msgr2=0x7fb7c81a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7c81094c0 0x7fb7c81a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c6a00700 1 -- 172.21.15.33:0/93005185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c81a0d50 con 0x7fb7c8103470 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.341+0000 7fb7c7400700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7c81094c0 0x7fb7c81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.342+0000 7fb7c6a00700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103470 0x7fb7c819bd90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800e9a0 tx=0x7fb7b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.342+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b800cbd0 con 0x7fb7c8103470 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.342+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c81a1040 con 0x7fb7c8103470 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.342+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b800cd70 con 0x7fb7c8103470 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.342+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8018680 con 0x7fb7c8103470 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.342+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c81a88c0 con 0x7fb7c8103470 2024-04-26T11:28:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.343+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb7c81029f0 con 0x7fb7c8103470 2024-04-26T11:28:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.346+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb7b8010430 con 0x7fb7c8103470 2024-04-26T11:28:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.347+0000 7fb7bf400700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb7ac0a3cd0 0x7fb7ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.347+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb7b80d0c90 con 0x7fb7c8103470 2024-04-26T11:28:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.347+0000 7fb7c6000700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb7ac0a3cd0 0x7fb7ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.347+0000 7fb7bf400700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] conn(0x7fb7ac0ad890 0x7fb7ac0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.347+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 --> [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7b8023e50 con 0x7fb7ac0ad890 2024-04-26T11:28:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.348+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb7b80d1040 con 0x7fb7c8103470 2024-04-26T11:28:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.348+0000 7fb7c7400700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] conn(0x7fb7ac0ad890 0x7fb7ac0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.348+0000 7fb7c6000700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb7ac0a3cd0 0x7fb7ac0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb7c81106d0 tx=0x7fb7b4005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.349+0000 7fb7c7400700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] conn(0x7fb7ac0ad890 0x7fb7ac0afd30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:06.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.350+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== osd.2 v2:172.21.15.194:6800/3454447938 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb7b8023e50 con 0x7fb7ac0ad890 2024-04-26T11:28:06.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.371+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 --> [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb7c8104b60 con 0x7fb7ac0ad890 2024-04-26T11:28:06.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.372+0000 7fb7bf400700 1 -- 172.21.15.33:0/93005185 <== osd.2 v2:172.21.15.194:6800/3454447938 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb7c8104b60 con 0x7fb7ac0ad890 2024-04-26T11:28:06.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.372+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] conn(0x7fb7ac0ad890 msgr2=0x7fb7ac0afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:06.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.372+0000 7fb7cdc00700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] conn(0x7fb7ac0ad890 0x7fb7ac0afd30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb7ac0a3cd0 msgr2=0x7fb7ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb7ac0a3cd0 0x7fb7ac0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb7c81106d0 tx=0x7fb7b4005a90 comp rx=0 tx=0).stop 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103470 msgr2=0x7fb7c819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 --2- 172.21.15.33:0/93005185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb7c8103470 0x7fb7c819bd90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800e9a0 tx=0x7fb7b800ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7c7400700 1 -- 172.21.15.33:0/93005185 reap_dead start 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 shutdown_connections 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 >> 172.21.15.33:0/93005185 conn(0x7fb7c807a580 msgr2=0x7fb7c8101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 shutdown_connections 2024-04-26T11:28:06.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:06.373+0000 7fb7cdc00700 1 -- 172.21.15.33:0/93005185 wait complete. 2024-04-26T11:28:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:06 smithi194 ceph-mon[33017]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:07.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:06 smithi033 ceph-mon[27845]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:06 smithi096 ceph-mon[32753]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:07.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 -- 172.21.15.33:0/2327583615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 msgr2=0x7fc47c109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:07.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 --2- 172.21.15.33:0/2327583615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 0x7fc47c109880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009990 tx=0x7fc46c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 -- 172.21.15.33:0/2327583615 shutdown_connections 2024-04-26T11:28:07.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 --2- 172.21.15.33:0/2327583615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc47c06bad0 0x7fc47c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 --2- 172.21.15.33:0/2327583615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 0x7fc47c109880 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 --2- 172.21.15.33:0/2327583615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc47c06b130 0x7fc47c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 -- 172.21.15.33:0/2327583615 >> 172.21.15.33:0/2327583615 conn(0x7fc47c0792d0 msgr2=0x7fc47c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 -- 172.21.15.33:0/2327583615 shutdown_connections 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.697+0000 7fc481200700 1 -- 172.21.15.33:0/2327583615 wait complete. 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.698+0000 7fc481200700 1 Processor -- start 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.698+0000 7fc481200700 1 -- start start 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc481200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc47c06b130 0x7fc47c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc481200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc47c06bad0 0x7fc47c19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc481200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 0x7fc47c199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc481200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c113600 con 0x7fc47c06b130 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc481200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c113480 con 0x7fc47c06bad0 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc481200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c113780 con 0x7fc47c1094a0 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc479e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc47c06b130 0x7fc47c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:07.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc479e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc47c06b130 0x7fc47c19f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36550/0 (socket says 172.21.15.33:36550) 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc479e00700 1 -- 172.21.15.33:0/2510691835 learned_addr learned my addr 172.21.15.33:0/2510691835 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc479400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc47c06bad0 0x7fc47c19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.699+0000 7fc47a800700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 0x7fc47c199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc479e00700 1 -- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 msgr2=0x7fc47c199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc479e00700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 0x7fc47c199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc479e00700 1 -- 172.21.15.33:0/2510691835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc47c06bad0 msgr2=0x7fc47c19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc479e00700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc47c06bad0 0x7fc47c19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc479e00700 1 -- 172.21.15.33:0/2510691835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47c19a0a0 con 0x7fc47c06b130 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc479400700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc47c06bad0 0x7fc47c19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc47a800700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc47c1094a0 0x7fc47c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc479e00700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc47c06b130 0x7fc47c19f780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc46c000c00 tx=0x7fc46c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c027070 con 0x7fc47c06b130 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc46c022470 con 0x7fc47c06b130 2024-04-26T11:28:07.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc47c19a330 con 0x7fc47c06b130 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.700+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c017650 con 0x7fc47c06b130 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.701+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc47c1a6650 con 0x7fc47c06b130 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.702+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc47c063bb0 con 0x7fc47c06b130 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.704+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc46c02a070 con 0x7fc47c06b130 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.706+0000 7fc472a00700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4600a3cd0 0x7fc4600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.707+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc46c0d1430 con 0x7fc47c06b130 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.707+0000 7fc479400700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4600a3cd0 0x7fc4600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.707+0000 7fc472a00700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] conn(0x7fc4600ad890 0x7fc4600afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.707+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 --> [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4600b0400 con 0x7fc4600ad890 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.707+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc46c0d17e0 con 0x7fc47c06b130 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.708+0000 7fc479400700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4600a3cd0 0x7fc4600a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0799b0 tx=0x7fc468009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.708+0000 7fc47a800700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] conn(0x7fc4600ad890 0x7fc4600afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.708+0000 7fc47a800700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] conn(0x7fc4600ad890 0x7fc4600afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:07.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.709+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== osd.8 v2:172.21.15.194:6816/529571686 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4600b0400 con 0x7fc4600ad890 2024-04-26T11:28:07.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.727+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 --> [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc47c064030 con 0x7fc4600ad890 2024-04-26T11:28:07.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.728+0000 7fc472a00700 1 -- 172.21.15.33:0/2510691835 <== osd.8 v2:172.21.15.194:6816/529571686 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc47c064030 con 0x7fc4600ad890 2024-04-26T11:28:07.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] conn(0x7fc4600ad890 msgr2=0x7fc4600afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:07.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] conn(0x7fc4600ad890 0x7fc4600afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4600a3cd0 msgr2=0x7fc4600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fc4600a3cd0 0x7fc4600a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0799b0 tx=0x7fc468009260 comp rx=0 tx=0).stop 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc47c06b130 msgr2=0x7fc47c19f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 --2- 172.21.15.33:0/2510691835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc47c06b130 0x7fc47c19f780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc46c000c00 tx=0x7fc46c0057c0 comp rx=0 tx=0).stop 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc47a800700 1 -- 172.21.15.33:0/2510691835 reap_dead start 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 shutdown_connections 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 >> 172.21.15.33:0/2510691835 conn(0x7fc47c0792d0 msgr2=0x7fc47c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.729+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 shutdown_connections 2024-04-26T11:28:07.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.730+0000 7fc481200700 1 -- 172.21.15.33:0/2510691835 wait complete. 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 -- 172.21.15.33:0/1355507711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 msgr2=0x7f00381038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 --2- 172.21.15.33:0/1355507711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 0x7f00381038f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f00240098f0 tx=0x7f0024009c00 comp rx=0 tx=0).stop 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 -- 172.21.15.33:0/1355507711 shutdown_connections 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 --2- 172.21.15.33:0/1355507711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0038103e30 0x7f0038110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 --2- 172.21.15.33:0/1355507711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00381094c0 0x7f00381098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 --2- 172.21.15.33:0/1355507711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 0x7f00381038f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 -- 172.21.15.33:0/1355507711 >> 172.21.15.33:0/1355507711 conn(0x7f003807a580 msgr2=0x7f003807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 -- 172.21.15.33:0/1355507711 shutdown_connections 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.821+0000 7f003d600700 1 -- 172.21.15.33:0/1355507711 wait complete. 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 Processor -- start 2024-04-26T11:28:08.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 -- start start 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 0x7f003819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0038103e30 0x7f003819fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00381094c0 0x7f0038199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0038113640 con 0x7f00381094c0 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00381134c0 con 0x7f0038103e30 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.823+0000 7f003d600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00381137c0 con 0x7f0038103470 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0037400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00381094c0 0x7f0038199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 0x7f003819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0037400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00381094c0 0x7f0038199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36562/0 (socket says 172.21.15.33:36562) 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0037400700 1 -- 172.21.15.33:0/851151801 learned_addr learned my addr 172.21.15.33:0/851151801 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036000700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0038103e30 0x7f003819fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036a00700 1 -- 172.21.15.33:0/851151801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0038103e30 msgr2=0x7f003819fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036a00700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0038103e30 0x7f003819fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036a00700 1 -- 172.21.15.33:0/851151801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00381094c0 msgr2=0x7f0038199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036a00700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00381094c0 0x7f0038199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036a00700 1 -- 172.21.15.33:0/851151801 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003819a180 con 0x7f0038103470 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.824+0000 7f0036a00700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 0x7f003819f860 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f002800eb90 tx=0x7f002800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.825+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002800cc90 con 0x7f0038103470 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.825+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002800ce30 con 0x7f0038103470 2024-04-26T11:28:08.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.825+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003819a470 con 0x7f0038103470 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.825+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028010430 con 0x7f0038103470 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.825+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00381a6730 con 0x7f0038103470 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.826+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0038063b70 con 0x7f0038103470 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.829+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0028003ea0 con 0x7f0038103470 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.830+0000 7f002f400700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f001c0a3d50 0x7f001c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.830+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f00280d0ab0 con 0x7f0038103470 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.830+0000 7f0036000700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f001c0a3d50 0x7f001c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.830+0000 7f002f400700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] conn(0x7f001c0ad840 0x7f001c0afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:08.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.831+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 --> [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f002801ee50 con 0x7f001c0ad840 2024-04-26T11:28:08.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.831+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f00280d0e60 con 0x7f0038103470 2024-04-26T11:28:08.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.831+0000 7f0037400700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] conn(0x7f001c0ad840 0x7f001c0afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:08.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.831+0000 7f0036000700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f001c0a3d50 0x7f001c0a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f00240098c0 tx=0x7f0024019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:08.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.831+0000 7f0037400700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] conn(0x7f001c0ad840 0x7f001c0afce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:08.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.832+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== osd.5 v2:172.21.15.194:6808/769884619 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f002801ee50 con 0x7f001c0ad840 2024-04-26T11:28:08.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.851+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 --> [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0038063ff0 con 0x7f001c0ad840 2024-04-26T11:28:08.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.852+0000 7f002f400700 1 -- 172.21.15.33:0/851151801 <== osd.5 v2:172.21.15.194:6808/769884619 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0038063ff0 con 0x7f001c0ad840 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] conn(0x7f001c0ad840 msgr2=0x7f001c0afce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] conn(0x7f001c0ad840 0x7f001c0afce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f001c0a3d50 msgr2=0x7f001c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f001c0a3d50 0x7f001c0a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f00240098c0 tx=0x7f0024019040 comp rx=0 tx=0).stop 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 msgr2=0x7f003819f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 --2- 172.21.15.33:0/851151801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0038103470 0x7f003819f860 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f002800eb90 tx=0x7f002800eea0 comp rx=0 tx=0).stop 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f0037400700 1 -- 172.21.15.33:0/851151801 reap_dead start 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 shutdown_connections 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 >> 172.21.15.33:0/851151801 conn(0x7f003807a580 msgr2=0x7f0038102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 shutdown_connections 2024-04-26T11:28:08.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:07.853+0000 7f003d600700 1 -- 172.21.15.33:0/851151801 wait complete. 2024-04-26T11:28:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:08 smithi194 ceph-mon[33017]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:09.361 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:08 smithi033 ceph-mon[27845]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:08 smithi096 ceph-mon[32753]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:09.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.865+0000 7f72da800700 1 -- 172.21.15.33:0/660536738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d41094a0 msgr2=0x7f72d4109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.865+0000 7f72da800700 1 --2- 172.21.15.33:0/660536738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d41094a0 0x7f72d4109880 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009990 tx=0x7f72c4009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.865+0000 7f72da800700 1 -- 172.21.15.33:0/660536738 shutdown_connections 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.865+0000 7f72da800700 1 --2- 172.21.15.33:0/660536738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d406bad0 0x7f72d41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.865+0000 7f72da800700 1 --2- 172.21.15.33:0/660536738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d41094a0 0x7f72d4109880 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.865+0000 7f72da800700 1 --2- 172.21.15.33:0/660536738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d406b130 0x7f72d406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.865+0000 7f72da800700 1 -- 172.21.15.33:0/660536738 >> 172.21.15.33:0/660536738 conn(0x7f72d40792d0 msgr2=0x7f72d40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.866+0000 7f72da800700 1 -- 172.21.15.33:0/660536738 shutdown_connections 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.866+0000 7f72da800700 1 -- 172.21.15.33:0/660536738 wait complete. 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.867+0000 7f72da800700 1 Processor -- start 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.867+0000 7f72da800700 1 -- start start 2024-04-26T11:28:09.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72da800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d406b130 0x7f72d41023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72da800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d406bad0 0x7f72d4102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72da800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d41094a0 0x7f72d4102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72da800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f72d41135e0 con 0x7f72d406b130 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72da800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4113460 con 0x7f72d41094a0 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72da800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4113760 con 0x7f72d406bad0 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72d3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d406b130 0x7f72d41023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72d3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d406b130 0x7f72d41023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36322/0 (socket says 172.21.15.33:36322) 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72d3400700 1 -- 172.21.15.33:0/487340504 learned_addr learned my addr 172.21.15.33:0/487340504 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72d2a00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d406bad0 0x7f72d4102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.868+0000 7f72d3e00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d41094a0 0x7f72d4102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72d2a00700 1 -- 172.21.15.33:0/487340504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d41094a0 msgr2=0x7f72d4102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72d2a00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72d41094a0 0x7f72d4102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72d2a00700 1 -- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d406b130 msgr2=0x7f72d41023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72d2a00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d406b130 0x7f72d41023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72d2a00700 1 -- 172.21.15.33:0/487340504 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d41ac520 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72d3400700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72d406b130 0x7f72d41023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72d2a00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d406bad0 0x7f72d4102930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f72c000e9f0 tx=0x7f72c000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c000c9c0 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c000cb60 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d41ac7b0 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d41accf0 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.869+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0018680 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.871+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72b4001030 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.875+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f72c0018820 con 0x7f72d406bad0 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.876+0000 7f72cbe00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72b80a3cd0 0x7f72b80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.877+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f72c00d05c0 con 0x7f72d406bad0 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.877+0000 7f72d3400700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72b80a3cd0 0x7f72b80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.877+0000 7f72cbe00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] conn(0x7f72b80ad890 0x7f72b80afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.877+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 --> [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72c0017e70 con 0x7f72b80ad890 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.877+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f72c00d7050 con 0x7f72d406bad0 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.877+0000 7f72d3e00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] conn(0x7f72b80ad890 0x7f72b80afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.877+0000 7f72d3400700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72b80a3cd0 0x7f72b80a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f72c4006010 tx=0x7f72c400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.878+0000 7f72d3e00700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] conn(0x7f72b80ad890 0x7f72b80afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.879+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== osd.6 v2:172.21.15.33:6818/3152193593 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72c0017e70 con 0x7f72b80ad890 2024-04-26T11:28:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.897+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 --> [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f72b4002e60 con 0x7f72b80ad890 2024-04-26T11:28:09.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.898+0000 7f72cbe00700 1 -- 172.21.15.33:0/487340504 <== osd.6 v2:172.21.15.33:6818/3152193593 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f72b4002e60 con 0x7f72b80ad890 2024-04-26T11:28:09.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.898+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] conn(0x7f72b80ad890 msgr2=0x7f72b80afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.898+0000 7f72da800700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] conn(0x7f72b80ad890 0x7f72b80afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72b80a3cd0 msgr2=0x7f72b80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f72b80a3cd0 0x7f72b80a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f72c4006010 tx=0x7f72c400b280 comp rx=0 tx=0).stop 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d406bad0 msgr2=0x7f72d4102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 --2- 172.21.15.33:0/487340504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72d406bad0 0x7f72d4102930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f72c000e9f0 tx=0x7f72c000ed00 comp rx=0 tx=0).stop 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72d3e00700 1 -- 172.21.15.33:0/487340504 reap_dead start 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 shutdown_connections 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 >> 172.21.15.33:0/487340504 conn(0x7f72d40792d0 msgr2=0x7f72d41004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 shutdown_connections 2024-04-26T11:28:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:09.899+0000 7f72da800700 1 -- 172.21.15.33:0/487340504 wait complete. 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.679+0000 7f2074400700 1 -- 172.21.15.33:0/3915599433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c105610 msgr2=0x7f206c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.679+0000 7f2074400700 1 --2- 172.21.15.33:0/3915599433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c105610 0x7f206c105a90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2058009990 tx=0x7f2058009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.680+0000 7f2074400700 1 -- 172.21.15.33:0/3915599433 shutdown_connections 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.680+0000 7f2074400700 1 --2- 172.21.15.33:0/3915599433 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f206c105fd0 0x7f206c1124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.680+0000 7f2074400700 1 --2- 172.21.15.33:0/3915599433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f206c10b6a0 0x7f206c10ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.680+0000 7f2074400700 1 --2- 172.21.15.33:0/3915599433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c105610 0x7f206c105a90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.680+0000 7f2074400700 1 -- 172.21.15.33:0/3915599433 >> 172.21.15.33:0/3915599433 conn(0x7f206c101100 msgr2=0x7f206c103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.680+0000 7f2074400700 1 -- 172.21.15.33:0/3915599433 shutdown_connections 2024-04-26T11:28:10.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.680+0000 7f2074400700 1 -- 172.21.15.33:0/3915599433 wait complete. 2024-04-26T11:28:10.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.681+0000 7f2074400700 1 Processor -- start 2024-04-26T11:28:10.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.681+0000 7f2074400700 1 -- start start 2024-04-26T11:28:10.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2074400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f206c105610 0x7f206c1a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:10.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2074400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f206c105fd0 0x7f206c1a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:10.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2074400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c10b6a0 0x7f206c19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:10.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2074400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f206c115810 con 0x7f206c105610 2024-04-26T11:28:10.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2074400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f206c115690 con 0x7f206c10b6a0 2024-04-26T11:28:10.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2074400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f206c115990 con 0x7f206c105fd0 2024-04-26T11:28:10.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2071a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f206c105610 0x7f206c1a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:10.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2071a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f206c105610 0x7f206c1a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36348/0 (socket says 172.21.15.33:36348) 2024-04-26T11:28:10.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2071a00700 1 -- 172.21.15.33:0/868429114 learned_addr learned my addr 172.21.15.33:0/868429114 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:10.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.682+0000 7f2071000700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f206c105fd0 0x7f206c1a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:10.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2072400700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c10b6a0 0x7f206c19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2071000700 1 -- 172.21.15.33:0/868429114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c10b6a0 msgr2=0x7f206c19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2071000700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c10b6a0 0x7f206c19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2071000700 1 -- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f206c105610 msgr2=0x7f206c1a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2071000700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f206c105610 0x7f206c1a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2071000700 1 -- 172.21.15.33:0/868429114 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c19c230 con 0x7f206c105fd0 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2071a00700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f206c105610 0x7f206c1a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2072400700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f206c10b6a0 0x7f206c19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.683+0000 7f2071000700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f206c105fd0 0x7f206c1a1e00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20580059d0 tx=0x7f20580047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.684+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058027070 con 0x7f206c105fd0 2024-04-26T11:28:10.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.684+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2058022470 con 0x7f206c105fd0 2024-04-26T11:28:10.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.684+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c19c4c0 con 0x7f206c105fd0 2024-04-26T11:28:10.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.684+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c1a8790 con 0x7f206c105fd0 2024-04-26T11:28:10.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.684+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058017650 con 0x7f206c105fd0 2024-04-26T11:28:10.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.685+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f206c063bb0 con 0x7f206c105fd0 2024-04-26T11:28:10.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.687+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f205802a070 con 0x7f206c105fd0 2024-04-26T11:28:10.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.688+0000 7f206a000700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20540a3cd0 0x7f20540a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:10.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.689+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f20580d1a60 con 0x7f206c105fd0 2024-04-26T11:28:10.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.689+0000 7f2071a00700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20540a3cd0 0x7f20540a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:10.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.689+0000 7f206a000700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] conn(0x7f20540ad890 0x7f20540afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:10.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.689+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 --> [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2058010ea0 con 0x7f20540ad890 2024-04-26T11:28:10.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.689+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f20580d1e10 con 0x7f206c105fd0 2024-04-26T11:28:10.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.689+0000 7f2072400700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] conn(0x7f20540ad890 0x7f20540afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:10.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.690+0000 7f2071a00700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20540a3cd0 0x7f20540a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f206c10a7d0 tx=0x7f2060009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:10.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.690+0000 7f2072400700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] conn(0x7f20540ad890 0x7f20540afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:10.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.691+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== osd.9 v2:172.21.15.33:6826/1639452107 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2058010ea0 con 0x7f20540ad890 2024-04-26T11:28:10.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.712+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 --> [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f206c063fb0 con 0x7f20540ad890 2024-04-26T11:28:10.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.712+0000 7f206a000700 1 -- 172.21.15.33:0/868429114 <== osd.9 v2:172.21.15.33:6826/1639452107 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f206c063fb0 con 0x7f20540ad890 2024-04-26T11:28:10.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.712+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] conn(0x7f20540ad890 msgr2=0x7f20540afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:10.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.712+0000 7f2074400700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] conn(0x7f20540ad890 0x7f20540afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.712+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20540a3cd0 msgr2=0x7f20540a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.712+0000 7f2074400700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20540a3cd0 0x7f20540a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f206c10a7d0 tx=0x7f2060009260 comp rx=0 tx=0).stop 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.714+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f206c105fd0 msgr2=0x7f206c1a1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.714+0000 7f2074400700 1 --2- 172.21.15.33:0/868429114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f206c105fd0 0x7f206c1a1e00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20580059d0 tx=0x7f20580047b0 comp rx=0 tx=0).stop 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.714+0000 7f2072400700 1 -- 172.21.15.33:0/868429114 reap_dead start 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.714+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 shutdown_connections 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.714+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 >> 172.21.15.33:0/868429114 conn(0x7f206c101100 msgr2=0x7f206c102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.714+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 shutdown_connections 2024-04-26T11:28:10.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:10.714+0000 7f2074400700 1 -- 172.21.15.33:0/868429114 wait complete. 2024-04-26T11:28:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:10 smithi033 ceph-mon[27845]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:11.243 INFO:teuthology.orchestra.run.smithi033.stdout:253403070518 2024-04-26T11:28:11.243 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T11:28:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:10 smithi194 ceph-mon[33017]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:10 smithi096 ceph-mon[32753]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:12.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 -- 172.21.15.33:0/1731437558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006bb50 msgr2=0x7f708006bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 --2- 172.21.15.33:0/1731437558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006bb50 0x7f708006bfb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f706c0098f0 tx=0x7f706c009c00 comp rx=0 tx=0).stop 2024-04-26T11:28:12.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 -- 172.21.15.33:0/1731437558 shutdown_connections 2024-04-26T11:28:12.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 --2- 172.21.15.33:0/1731437558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f708006c4f0 0x7f70801102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 --2- 172.21.15.33:0/1731437558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70801094a0 0x7f7080109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 --2- 172.21.15.33:0/1731437558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006bb50 0x7f708006bfb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 -- 172.21.15.33:0/1731437558 >> 172.21.15.33:0/1731437558 conn(0x7f70800792d0 msgr2=0x7f70800796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:12.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 -- 172.21.15.33:0/1731437558 shutdown_connections 2024-04-26T11:28:12.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.042+0000 7f7088600700 1 -- 172.21.15.33:0/1731437558 wait complete. 2024-04-26T11:28:12.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.043+0000 7f7088600700 1 Processor -- start 2024-04-26T11:28:12.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7088600700 1 -- start start 2024-04-26T11:28:12.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7088600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f708006bb50 0x7f708019bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7088600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006c4f0 0x7f708019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7088600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70801094a0 0x7f70801a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7088600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7080113620 con 0x7f708006bb50 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7088600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f70801134a0 con 0x7f70801094a0 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7088600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f70801137a0 con 0x7f708006c4f0 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7085c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f708006bb50 0x7f708019bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7085200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006c4f0 0x7f708019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7085c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f708006bb50 0x7f708019bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36376/0 (socket says 172.21.15.33:36376) 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7085c00700 1 -- 172.21.15.33:0/3772768937 learned_addr learned my addr 172.21.15.33:0/3772768937 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.044+0000 7f7086600700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70801094a0 0x7f70801a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.045+0000 7f7085200700 1 -- 172.21.15.33:0/3772768937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70801094a0 msgr2=0x7f70801a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.045+0000 7f7085200700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70801094a0 0x7f70801a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.045+0000 7f7085200700 1 -- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f708006bb50 msgr2=0x7f708019bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.045+0000 7f7085200700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f708006bb50 0x7f708019bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.045+0000 7f7085200700 1 -- 172.21.15.33:0/3772768937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70801a0da0 con 0x7f708006c4f0 2024-04-26T11:28:12.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.045+0000 7f7085c00700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f708006bb50 0x7f708019bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:12.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.045+0000 7f7085200700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006c4f0 0x7f708019c2d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f706c00c010 tx=0x7f706c00fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:12.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.046+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c0050c0 con 0x7f708006c4f0 2024-04-26T11:28:12.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.046+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f706c00b560 con 0x7f708006c4f0 2024-04-26T11:28:12.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.046+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c003a00 con 0x7f708006c4f0 2024-04-26T11:28:12.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.046+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70801a1030 con 0x7f708006c4f0 2024-04-26T11:28:12.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.046+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801a8910 con 0x7f708006c4f0 2024-04-26T11:28:12.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.047+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7080100c20 con 0x7f708006c4f0 2024-04-26T11:28:12.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.049+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f706c00b710 con 0x7f708006c4f0 2024-04-26T11:28:12.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.050+0000 7f707e000700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f70680a3d50 0x7f70680a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.050+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f706c0d1750 con 0x7f708006c4f0 2024-04-26T11:28:12.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.050+0000 7f7085c00700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f70680a3d50 0x7f70680a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.051+0000 7f707e000700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] conn(0x7f70680ad910 0x7f70680afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.051+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 --> [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f706c02ee50 con 0x7f70680ad910 2024-04-26T11:28:12.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.051+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f706c0d1b00 con 0x7f708006c4f0 2024-04-26T11:28:12.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.051+0000 7f7086600700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] conn(0x7f70680ad910 0x7f70680afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.051+0000 7f7085c00700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f70680a3d50 0x7f70680a6210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f7074009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:12.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.052+0000 7f7086600700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] conn(0x7f70680ad910 0x7f70680afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:12.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.053+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== osd.0 v2:172.21.15.33:6802/2032124728 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c02ee50 con 0x7f70680ad910 2024-04-26T11:28:12.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.071+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 --> [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7080104b50 con 0x7f70680ad910 2024-04-26T11:28:12.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.072+0000 7f707e000700 1 -- 172.21.15.33:0/3772768937 <== osd.0 v2:172.21.15.33:6802/2032124728 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7080104b50 con 0x7f70680ad910 2024-04-26T11:28:12.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] conn(0x7f70680ad910 msgr2=0x7f70680afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] conn(0x7f70680ad910 0x7f70680afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f70680a3d50 msgr2=0x7f70680a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f70680a3d50 0x7f70680a6210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f7074009310 comp rx=0 tx=0).stop 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006c4f0 msgr2=0x7f708019c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 --2- 172.21.15.33:0/3772768937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f708006c4f0 0x7f708019c2d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f706c00c010 tx=0x7f706c00fa00 comp rx=0 tx=0).stop 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7086600700 1 -- 172.21.15.33:0/3772768937 reap_dead start 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 shutdown_connections 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 >> 172.21.15.33:0/3772768937 conn(0x7f70800792d0 msgr2=0x7f70801001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 shutdown_connections 2024-04-26T11:28:12.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.073+0000 7f7088600700 1 -- 172.21.15.33:0/3772768937 wait complete. 2024-04-26T11:28:12.419 INFO:teuthology.orchestra.run.smithi033.stdout:77309411416 2024-04-26T11:28:12.419 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T11:28:12.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 -- 172.21.15.33:0/4178160110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 msgr2=0x7f83f41169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 --2- 172.21.15.33:0/4178160110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 0x7f83f41169e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00b0a0 tx=0x7f83ec00b3b0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 -- 172.21.15.33:0/4178160110 shutdown_connections 2024-04-26T11:28:12.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 --2- 172.21.15.33:0/4178160110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 0x7f83f41169e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 --2- 172.21.15.33:0/4178160110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f4075650 0x7f83f4075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 --2- 172.21.15.33:0/4178160110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f4075f70 0x7f83f41140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 -- 172.21.15.33:0/4178160110 >> 172.21.15.33:0/4178160110 conn(0x7f83f407a580 msgr2=0x7f83f407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:12.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 -- 172.21.15.33:0/4178160110 shutdown_connections 2024-04-26T11:28:12.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.847+0000 7f83fa200700 1 -- 172.21.15.33:0/4178160110 wait complete. 2024-04-26T11:28:12.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 Processor -- start 2024-04-26T11:28:12.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 -- start start 2024-04-26T11:28:12.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f4075650 0x7f83f4113930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f4075f70 0x7f83f4113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 0x7f83f410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f83f4107e90 con 0x7f83f4075650 2024-04-26T11:28:12.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83f4107d10 con 0x7f83f4075f70 2024-04-26T11:28:12.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.849+0000 7f83fa200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83f4108010 con 0x7f83f41145f0 2024-04-26T11:28:12.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f4075650 0x7f83f4113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f4075650 0x7f83f4113930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36382/0 (socket says 172.21.15.33:36382) 2024-04-26T11:28:12.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 -- 172.21.15.33:0/3590857566 learned_addr learned my addr 172.21.15.33:0/3590857566 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:12.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f2a00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f4075f70 0x7f83f4113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3e00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 0x7f83f410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 -- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 msgr2=0x7f83f410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 0x7f83f410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 -- 172.21.15.33:0/3590857566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f4075f70 msgr2=0x7f83f4113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f4075f70 0x7f83f4113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3400700 1 -- 172.21.15.33:0/3590857566 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f410e300 con 0x7f83f4075650 2024-04-26T11:28:12.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f3e00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f41145f0 0x7f83f410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:12.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.850+0000 7f83f2a00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f4075f70 0x7f83f4113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:12.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.851+0000 7f83f3400700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f4075650 0x7f83f4113930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f83e400e9a0 tx=0x7f83e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:12.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.851+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e400ca10 con 0x7f83f4075650 2024-04-26T11:28:12.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.851+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83e400cbb0 con 0x7f83f4075650 2024-04-26T11:28:12.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.851+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f410e590 con 0x7f83f4075650 2024-04-26T11:28:12.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.851+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4018680 con 0x7f83f4075650 2024-04-26T11:28:12.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.851+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f41b5520 con 0x7f83f4075650 2024-04-26T11:28:12.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.853+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83f4063b30 con 0x7f83f4075650 2024-04-26T11:28:12.855 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.855+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f83e4010430 con 0x7f83f4075650 2024-04-26T11:28:12.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.856+0000 7f83ebe00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f83d80a3cd0 0x7f83d80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.856+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f83e40d0730 con 0x7f83f4075650 2024-04-26T11:28:12.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.856+0000 7f83f2a00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f83d80a3cd0 0x7f83d80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.856+0000 7f83ebe00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] conn(0x7f83d80ad890 0x7f83d80afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:12.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.857+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 --> [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83d80b0400 con 0x7f83d80ad890 2024-04-26T11:28:12.857 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.857+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f83e40d7050 con 0x7f83f4075650 2024-04-26T11:28:12.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.857+0000 7f83f3e00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] conn(0x7f83d80ad890 0x7f83d80afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:12.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.857+0000 7f83f2a00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f83d80a3cd0 0x7f83d80a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f83f410f170 tx=0x7f83e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:12.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.858+0000 7f83f3e00700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] conn(0x7f83d80ad890 0x7f83d80afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:12.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.859+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== osd.11 v2:172.21.15.194:6824/2370303582 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f83d80b0400 con 0x7f83d80ad890 2024-04-26T11:28:12.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.877+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 --> [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f83f4063f30 con 0x7f83d80ad890 2024-04-26T11:28:12.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.878+0000 7f83ebe00700 1 -- 172.21.15.33:0/3590857566 <== osd.11 v2:172.21.15.194:6824/2370303582 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f83f4063f30 con 0x7f83d80ad890 2024-04-26T11:28:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.878+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] conn(0x7f83d80ad890 msgr2=0x7f83d80afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.878+0000 7f83fa200700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] conn(0x7f83d80ad890 0x7f83d80afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.878+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f83d80a3cd0 msgr2=0x7f83d80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.878+0000 7f83fa200700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f83d80a3cd0 0x7f83d80a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f83f410f170 tx=0x7f83e0009260 comp rx=0 tx=0).stop 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.878+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f4075650 msgr2=0x7f83f4113930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.878+0000 7f83fa200700 1 --2- 172.21.15.33:0/3590857566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f83f4075650 0x7f83f4113930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f83e400e9a0 tx=0x7f83e400ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.879+0000 7f83f3e00700 1 -- 172.21.15.33:0/3590857566 reap_dead start 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.879+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 shutdown_connections 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.879+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 >> 172.21.15.33:0/3590857566 conn(0x7f83f407a580 msgr2=0x7f83f4106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.879+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 shutdown_connections 2024-04-26T11:28:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:12.879+0000 7f83fa200700 1 -- 172.21.15.33:0/3590857566 wait complete. 2024-04-26T11:28:13.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:12 smithi033 ceph-mon[27845]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-26T11:28:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:12 smithi194 ceph-mon[33017]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-26T11:28:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:12 smithi096 ceph-mon[32753]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-26T11:28:13.764 INFO:teuthology.orchestra.run.smithi033.stdout:197568495681 2024-04-26T11:28:13.765 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T11:28:14.803 INFO:teuthology.orchestra.run.smithi033.stdout:287762808880 2024-04-26T11:28:14.804 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 -- 172.21.15.33:0/1524000013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd17810fbf0 msgr2=0x7fd178112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 --2- 172.21.15.33:0/1524000013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd17810fbf0 0x7fd178112000 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd17000b0a0 tx=0x7fd17000b3b0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 -- 172.21.15.33:0/1524000013 shutdown_connections 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 --2- 172.21.15.33:0/1524000013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd17810fbf0 0x7fd178112000 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 --2- 172.21.15.33:0/1524000013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1780757f0 0x7fd178075bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 --2- 172.21.15.33:0/1524000013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd178076110 0x7fd17810f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 -- 172.21.15.33:0/1524000013 >> 172.21.15.33:0/1524000013 conn(0x7fd1780feea0 msgr2=0x7fd1781012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:15.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 -- 172.21.15.33:0/1524000013 shutdown_connections 2024-04-26T11:28:15.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.008+0000 7fd177200700 1 -- 172.21.15.33:0/1524000013 wait complete. 2024-04-26T11:28:15.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.009+0000 7fd177200700 1 Processor -- start 2024-04-26T11:28:15.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.009+0000 7fd177200700 1 -- start start 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.009+0000 7fd177200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1780757f0 0x7fd1781a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd177200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178076110 0x7fd1781a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd177200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd17810fbf0 0x7fd17819b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd177200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd178115750 con 0x7fd1780757f0 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd177200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781155d0 con 0x7fd17810fbf0 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd177200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781158d0 con 0x7fd178076110 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1780757f0 0x7fd1781a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1780757f0 0x7fd1781a17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36400/0 (socket says 172.21.15.33:36400) 2024-04-26T11:28:15.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 -- 172.21.15.33:0/2870947663 learned_addr learned my addr 172.21.15.33:0/2870947663 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175400700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178076110 0x7fd1781a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd176800700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd17810fbf0 0x7fd17819b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 -- 172.21.15.33:0/2870947663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178076110 msgr2=0x7fd1781a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178076110 0x7fd1781a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 -- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd17810fbf0 msgr2=0x7fd17819b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd17810fbf0 0x7fd17819b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd175e00700 1 -- 172.21.15.33:0/2870947663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17819c110 con 0x7fd1780757f0 2024-04-26T11:28:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.010+0000 7fd176800700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd17810fbf0 0x7fd17819b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:28:15.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.011+0000 7fd175e00700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1780757f0 0x7fd1781a17f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd16800e9a0 tx=0x7fd16800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:15.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.011+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16800cbd0 con 0x7fd1780757f0 2024-04-26T11:28:15.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.011+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16800cd70 con 0x7fd1780757f0 2024-04-26T11:28:15.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.011+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168018680 con 0x7fd1780757f0 2024-04-26T11:28:15.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.011+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd17819c400 con 0x7fd1780757f0 2024-04-26T11:28:15.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.011+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1781a86c0 con 0x7fd1780757f0 2024-04-26T11:28:15.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.012+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd178063bb0 con 0x7fd1780757f0 2024-04-26T11:28:15.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.015+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd168010430 con 0x7fd1780757f0 2024-04-26T11:28:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.016+0000 7fd16ea00700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd15c0a3d50 0x7fd15c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.016+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd1680d0ff0 con 0x7fd1780757f0 2024-04-26T11:28:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.016+0000 7fd175400700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd15c0a3d50 0x7fd15c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.017+0000 7fd16ea00700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] conn(0x7fd15c0ad910 0x7fd15c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:15.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.017+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 --> [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd168027e50 con 0x7fd15c0ad910 2024-04-26T11:28:15.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.017+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd1680d1400 con 0x7fd1780757f0 2024-04-26T11:28:15.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.017+0000 7fd175400700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd15c0a3d50 0x7fd15c0a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd178112460 tx=0x7fd164009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:15.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.017+0000 7fd176800700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] conn(0x7fd15c0ad910 0x7fd15c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:15.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.018+0000 7fd176800700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] conn(0x7fd15c0ad910 0x7fd15c0afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:15.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.019+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== osd.4 v2:172.21.15.96:6808/237079136 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd168027e50 con 0x7fd15c0ad910 2024-04-26T11:28:15.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.037+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 --> [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd178063fb0 con 0x7fd15c0ad910 2024-04-26T11:28:15.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.038+0000 7fd16ea00700 1 -- 172.21.15.33:0/2870947663 <== osd.4 v2:172.21.15.96:6808/237079136 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd178063fb0 con 0x7fd15c0ad910 2024-04-26T11:28:15.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] conn(0x7fd15c0ad910 msgr2=0x7fd15c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:15.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] conn(0x7fd15c0ad910 0x7fd15c0afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd15c0a3d50 msgr2=0x7fd15c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:15.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd15c0a3d50 0x7fd15c0a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd178112460 tx=0x7fd164009260 comp rx=0 tx=0).stop 2024-04-26T11:28:15.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1780757f0 msgr2=0x7fd1781a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:15.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 --2- 172.21.15.33:0/2870947663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd1780757f0 0x7fd1781a17f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd16800e9a0 tx=0x7fd16800ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:15.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd176800700 1 -- 172.21.15.33:0/2870947663 reap_dead start 2024-04-26T11:28:15.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 shutdown_connections 2024-04-26T11:28:15.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 >> 172.21.15.33:0/2870947663 conn(0x7fd1780feea0 msgr2=0x7fd1780ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:15.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 shutdown_connections 2024-04-26T11:28:15.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:15.039+0000 7fd177200700 1 -- 172.21.15.33:0/2870947663 wait complete. 2024-04-26T11:28:15.317 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:15 smithi033 ceph-mon[27845]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:15 smithi194 ceph-mon[33017]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:15.379 INFO:teuthology.orchestra.run.smithi033.stdout:343597383717 2024-04-26T11:28:15.379 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T11:28:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:15 smithi096 ceph-mon[32753]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:15.712 INFO:teuthology.orchestra.run.smithi033.stdout:309237645354 2024-04-26T11:28:15.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T11:28:16.130 INFO:teuthology.orchestra.run.smithi033.stdout:34359738466 2024-04-26T11:28:16.131 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T11:28:16.463 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:16.486 INFO:teuthology.orchestra.run.smithi033.stdout:107374182481 2024-04-26T11:28:16.487 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T11:28:16.945 INFO:teuthology.orchestra.run.smithi033.stdout:141733920846 2024-04-26T11:28:16.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T11:28:17.068 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:17 smithi033 ceph-mon[27845]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:17 smithi194 ceph-mon[33017]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:17.379 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:17.431 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:17.432 INFO:teuthology.orchestra.run.smithi033.stdout:163208757319 2024-04-26T11:28:17.433 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T11:28:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:17 smithi096 ceph-mon[32753]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:17.447 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:19.122 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:19 smithi033 ceph-mon[27845]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:19 smithi194 ceph-mon[33017]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:19 smithi096 ceph-mon[32753]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:20.318 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:20 smithi033 ceph-mon[27845]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-26T11:28:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:20 smithi194 ceph-mon[33017]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-26T11:28:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:20 smithi096 ceph-mon[32753]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-26T11:28:21.274 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:21.275 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:21.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:21 smithi033 ceph-mon[27845]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T11:28:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:21 smithi194 ceph-mon[33017]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T11:28:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:21 smithi096 ceph-mon[32753]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T11:28:21.506 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:21.691 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:23.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:23 smithi033 ceph-mon[27845]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:28:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:23 smithi194 ceph-mon[33017]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:28:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:23 smithi096 ceph-mon[32753]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:28:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 -- 172.21.15.33:0/2624749397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd4581070d0 msgr2=0x7fd4581074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 --2- 172.21.15.33:0/2624749397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd4581070d0 0x7fd4581074b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd448009990 tx=0x7fd448009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 -- 172.21.15.33:0/2624749397 shutdown_connections 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 --2- 172.21.15.33:0/2624749397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd458102270 0x7fd4581026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 --2- 172.21.15.33:0/2624749397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd4581070d0 0x7fd4581074b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 --2- 172.21.15.33:0/2624749397 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd458107a80 0x7fd458101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 -- 172.21.15.33:0/2624749397 >> 172.21.15.33:0/2624749397 conn(0x7fd4580775d0 msgr2=0x7fd4580779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 -- 172.21.15.33:0/2624749397 shutdown_connections 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.032+0000 7fd45dc00700 1 -- 172.21.15.33:0/2624749397 wait complete. 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 Processor -- start 2024-04-26T11:28:25.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 -- start start 2024-04-26T11:28:25.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd458102270 0x7fd4581a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:25.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4581070d0 0x7fd4581a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:25.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd458107a80 0x7fd4581a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd458119cd0 con 0x7fd458102270 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd458119b50 con 0x7fd4581070d0 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.034+0000 7fd45dc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd458119e50 con 0x7fd458107a80 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd458102270 0x7fd4581a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd458102270 0x7fd4581a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41912/0 (socket says 172.21.15.33:41912) 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 -- 172.21.15.33:0/3199204401 learned_addr learned my addr 172.21.15.33:0/3199204401 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457400700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4581070d0 0x7fd4581a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd45d200700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd458107a80 0x7fd4581a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:25.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 -- 172.21.15.33:0/3199204401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd458107a80 msgr2=0x7fd4581a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:25.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd458107a80 0x7fd4581a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 -- 172.21.15.33:0/3199204401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4581070d0 msgr2=0x7fd4581a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:25.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4581070d0 0x7fd4581a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.035+0000 7fd457e00700 1 -- 172.21.15.33:0/3199204401 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4581a5270 con 0x7fd458102270 2024-04-26T11:28:25.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.036+0000 7fd457e00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd458102270 0x7fd4581a0260 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd448000c00 tx=0x7fd4480057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:25.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.036+0000 7fd454c00700 1 -- 172.21.15.33:0/3199204401 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448027070 con 0x7fd458102270 2024-04-26T11:28:25.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.036+0000 7fd454c00700 1 -- 172.21.15.33:0/3199204401 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd448022470 con 0x7fd458102270 2024-04-26T11:28:25.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.036+0000 7fd454c00700 1 -- 172.21.15.33:0/3199204401 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448017650 con 0x7fd458102270 2024-04-26T11:28:25.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.036+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4581a5500 con 0x7fd458102270 2024-04-26T11:28:25.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.036+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4581acde0 con 0x7fd458102270 2024-04-26T11:28:25.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.037+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd45806a110 con 0x7fd458102270 2024-04-26T11:28:25.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.039+0000 7fd454c00700 1 -- 172.21.15.33:0/3199204401 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd44802a070 con 0x7fd458102270 2024-04-26T11:28:25.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.041+0000 7fd454c00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd43c0a3c40 0x7fd43c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:25.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.041+0000 7fd454c00700 1 -- 172.21.15.33:0/3199204401 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd4480d1140 con 0x7fd458102270 2024-04-26T11:28:25.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.041+0000 7fd457400700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd43c0a3c40 0x7fd43c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:25.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.042+0000 7fd457400700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd43c0a3c40 0x7fd43c0a6100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd458104b30 tx=0x7fd444009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:25.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.046+0000 7fd454c00700 1 -- 172.21.15.33:0/3199204401 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd448013070 con 0x7fd458102270 2024-04-26T11:28:25.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:25 smithi033 ceph-mon[27845]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T11:28:25.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.294+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd4581acfb0 con 0x7fd458102270 2024-04-26T11:28:25.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.295+0000 7fd454c00700 1 -- 172.21.15.33:0/3199204401 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd44809db70 con 0x7fd458102270 2024-04-26T11:28:25.297 INFO:teuthology.orchestra.run.smithi033.stdout:219043332158 2024-04-26T11:28:25.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.297+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd43c0a3c40 msgr2=0x7fd43c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.297+0000 7fd45dc00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd43c0a3c40 0x7fd43c0a6100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd458104b30 tx=0x7fd444009260 comp rx=0 tx=0).stop 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.297+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd458102270 msgr2=0x7fd4581a0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.297+0000 7fd45dc00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd458102270 0x7fd4581a0260 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd448000c00 tx=0x7fd4480057c0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 shutdown_connections 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd43c0a3c40 0x7fd43c0a6100 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd458107a80 0x7fd4581a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd458102270 0x7fd4581a0260 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 --2- 172.21.15.33:0/3199204401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4581070d0 0x7fd4581a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 >> 172.21.15.33:0/3199204401 conn(0x7fd4580775d0 msgr2=0x7fd458102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 shutdown_connections 2024-04-26T11:28:25.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:25.298+0000 7fd45dc00700 1 -- 172.21.15.33:0/3199204401 wait complete. 2024-04-26T11:28:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:25 smithi194 ceph-mon[33017]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T11:28:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:25 smithi096 ceph-mon[32753]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T11:28:25.814 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:26.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:26 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3199204401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T11:28:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:26 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3199204401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T11:28:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:26 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3199204401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T11:28:26.468 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:27 smithi033 ceph-mon[27845]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T11:28:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:27 smithi194 ceph-mon[33017]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T11:28:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:27 smithi096 ceph-mon[32753]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 -- 172.21.15.33:0/2964940761 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01083b0 msgr2=0x7f09a0108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 --2- 172.21.15.33:0/2964940761 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01083b0 0x7f09a0108790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09900099c0 tx=0x7f0990009cd0 comp rx=0 tx=0).stop 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 -- 172.21.15.33:0/2964940761 shutdown_connections 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 --2- 172.21.15.33:0/2964940761 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a0103e90 0x7f09a0110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 --2- 172.21.15.33:0/2964940761 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01083b0 0x7f09a0108790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 --2- 172.21.15.33:0/2964940761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09a01034d0 0x7f09a0103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 -- 172.21.15.33:0/2964940761 >> 172.21.15.33:0/2964940761 conn(0x7f09a007a580 msgr2=0x7f09a007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 -- 172.21.15.33:0/2964940761 shutdown_connections 2024-04-26T11:28:27.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.914+0000 7f09a7000700 1 -- 172.21.15.33:0/2964940761 wait complete. 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 Processor -- start 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 -- start start 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01034d0 0x7f09a019bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a0103e90 0x7f09a019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09a01083b0 0x7f09a01a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f09a0113660 con 0x7f09a0103e90 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01134e0 con 0x7f09a01034d0 2024-04-26T11:28:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.916+0000 7f09a7000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01137e0 con 0x7f09a01083b0 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a0103e90 0x7f09a019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01034d0 0x7f09a019bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01034d0 0x7f09a019bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:60010/0 (socket says 172.21.15.33:60010) 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 -- 172.21.15.33:0/3434543664 learned_addr learned my addr 172.21.15.33:0/3434543664 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f09a5000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09a01083b0 0x7f09a01a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 -- 172.21.15.33:0/3434543664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09a01083b0 msgr2=0x7f09a01a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09a01083b0 0x7f09a01a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 -- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a0103e90 msgr2=0x7f09a019c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a0103e90 0x7f09a019c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f099fe00700 1 -- 172.21.15.33:0/3434543664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a01a0de0 con 0x7f09a01034d0 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.917+0000 7f09a5000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09a01083b0 0x7f09a01a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.918+0000 7f099fe00700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01034d0 0x7f09a019bdd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0990006010 tx=0x7f0990005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.918+0000 7f099cc00700 1 -- 172.21.15.33:0/3434543664 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990026070 con 0x7f09a01034d0 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.918+0000 7f099cc00700 1 -- 172.21.15.33:0/3434543664 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0990003c70 con 0x7f09a01034d0 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.918+0000 7f099cc00700 1 -- 172.21.15.33:0/3434543664 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099000fb40 con 0x7f09a01034d0 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.918+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a01a1070 con 0x7f09a01034d0 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.918+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a01a8950 con 0x7f09a01034d0 2024-04-26T11:28:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.919+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a0104b90 con 0x7f09a01034d0 2024-04-26T11:28:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.923+0000 7f099cc00700 1 -- 172.21.15.33:0/3434543664 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0990029070 con 0x7f09a01034d0 2024-04-26T11:28:27.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.924+0000 7f099cc00700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f09840a3cc0 0x7f09840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:27.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.924+0000 7f099cc00700 1 -- 172.21.15.33:0/3434543664 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f09900d12c0 con 0x7f09a01034d0 2024-04-26T11:28:27.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.924+0000 7f099f400700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f09840a3cc0 0x7f09840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:27.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.925+0000 7f099f400700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f09840a3cc0 0x7f09840a6180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09a0110670 tx=0x7f098c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:27.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.925+0000 7f099cc00700 1 -- 172.21.15.33:0/3434543664 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0990030020 con 0x7f09a01034d0 2024-04-26T11:28:28.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.176+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f09a019d130 con 0x7f09a01034d0 2024-04-26T11:28:28.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.176+0000 7f099cc00700 1 -- 172.21.15.33:0/3434543664 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0990013070 con 0x7f09a01034d0 2024-04-26T11:28:28.178 INFO:teuthology.orchestra.run.smithi033.stdout:287762808883 2024-04-26T11:28:28.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f09840a3cc0 msgr2=0x7f09840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:28.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f09840a3cc0 0x7f09840a6180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09a0110670 tx=0x7f098c009260 comp rx=0 tx=0).stop 2024-04-26T11:28:28.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01034d0 msgr2=0x7f09a019bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01034d0 0x7f09a019bdd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0990006010 tx=0x7f0990005280 comp rx=0 tx=0).stop 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 shutdown_connections 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f09840a3cc0 0x7f09840a6180 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09a01083b0 0x7f09a01a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09a01034d0 0x7f09a019bdd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 --2- 172.21.15.33:0/3434543664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09a0103e90 0x7f09a019c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.179+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 >> 172.21.15.33:0/3434543664 conn(0x7f09a007a580 msgr2=0x7f09a0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.180+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 shutdown_connections 2024-04-26T11:28:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.180+0000 7f09a7000700 1 -- 172.21.15.33:0/3434543664 wait complete. 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 -- 172.21.15.33:0/3105014216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 msgr2=0x7faf4806bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 --2- 172.21.15.33:0/3105014216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 0x7faf4806bf80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faf30009990 tx=0x7faf30009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 -- 172.21.15.33:0/3105014216 shutdown_connections 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 --2- 172.21.15.33:0/3105014216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf4806c4c0 0x7faf48110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 --2- 172.21.15.33:0/3105014216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf481094d0 0x7faf481098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 --2- 172.21.15.33:0/3105014216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 0x7faf4806bf80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 -- 172.21.15.33:0/3105014216 >> 172.21.15.33:0/3105014216 conn(0x7faf48079360 msgr2=0x7faf48079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:28.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 -- 172.21.15.33:0/3105014216 shutdown_connections 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.825+0000 7faf4d800700 1 -- 172.21.15.33:0/3105014216 wait complete. 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.826+0000 7faf4d800700 1 Processor -- start 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf4d800700 1 -- start start 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf4d800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 0x7faf4819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf4d800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf4806c4c0 0x7faf4819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf4d800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf481094d0 0x7faf481a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf4d800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf48113630 con 0x7faf481094d0 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf4d800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf481134b0 con 0x7faf4806bb20 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf4d800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf481137b0 con 0x7faf4806c4c0 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf47400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf481094d0 0x7faf481a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf3e000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf4806c4c0 0x7faf4819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:28.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf46a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 0x7faf4819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf3e000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf4806c4c0 0x7faf4819c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:48816/0 (socket says 172.21.15.33:48816) 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.827+0000 7faf3e000700 1 -- 172.21.15.33:0/3938533392 learned_addr learned my addr 172.21.15.33:0/3938533392 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf3e000700 1 -- 172.21.15.33:0/3938533392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 msgr2=0x7faf4819bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf3e000700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 0x7faf4819bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf3e000700 1 -- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf481094d0 msgr2=0x7faf481a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf3e000700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf481094d0 0x7faf481a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf3e000700 1 -- 172.21.15.33:0/3938533392 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf481a0db0 con 0x7faf4806c4c0 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf46a00700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 0x7faf4819bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf47400700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf481094d0 0x7faf481a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.828+0000 7faf3e000700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf4806c4c0 0x7faf4819c2e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faf30005910 tx=0x7faf3000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.829+0000 7faf3fe00700 1 -- 172.21.15.33:0/3938533392 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30027070 con 0x7faf4806c4c0 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.829+0000 7faf3fe00700 1 -- 172.21.15.33:0/3938533392 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf30022470 con 0x7faf4806c4c0 2024-04-26T11:28:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.829+0000 7faf3fe00700 1 -- 172.21.15.33:0/3938533392 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30017650 con 0x7faf4806c4c0 2024-04-26T11:28:28.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.829+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf481a1040 con 0x7faf4806c4c0 2024-04-26T11:28:28.700 INFO:teuthology.orchestra.run.smithi033.stdout:55834574945 2024-04-26T11:28:28.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.829+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf481a8920 con 0x7faf4806c4c0 2024-04-26T11:28:28.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.830+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf48104b80 con 0x7faf4806c4c0 2024-04-26T11:28:28.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.832+0000 7faf3fe00700 1 -- 172.21.15.33:0/3938533392 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faf3002a070 con 0x7faf4806c4c0 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.833+0000 7faf3fe00700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf2c0a3cc0 0x7faf2c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.833+0000 7faf3fe00700 1 -- 172.21.15.33:0/3938533392 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faf300d1260 con 0x7faf4806c4c0 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.834+0000 7faf46a00700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf2c0a3cc0 0x7faf2c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.834+0000 7faf46a00700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf2c0a3cc0 0x7faf2c0a6180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faf38009fd0 tx=0x7faf38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:27.836+0000 7faf3fe00700 1 -- 172.21.15.33:0/3938533392 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf30013070 con 0x7faf4806c4c0 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.074+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7faf481a8af0 con 0x7faf4806c4c0 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.075+0000 7faf3fe00700 1 -- 172.21.15.33:0/3938533392 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faf3009ddb0 con 0x7faf4806c4c0 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.078+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf2c0a3cc0 msgr2=0x7faf2c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.078+0000 7faf4d800700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf2c0a3cc0 0x7faf2c0a6180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faf38009fd0 tx=0x7faf38009260 comp rx=0 tx=0).stop 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.078+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf4806c4c0 msgr2=0x7faf4819c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.078+0000 7faf4d800700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf4806c4c0 0x7faf4819c2e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faf30005910 tx=0x7faf3000b670 comp rx=0 tx=0).stop 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 shutdown_connections 2024-04-26T11:28:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf2c0a3cc0 0x7faf2c0a6180 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf481094d0 0x7faf481a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf4806bb20 0x7faf4819bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 --2- 172.21.15.33:0/3938533392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf4806c4c0 0x7faf4819c2e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:28.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 >> 172.21.15.33:0/3938533392 conn(0x7faf48079360 msgr2=0x7faf481001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:28.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 shutdown_connections 2024-04-26T11:28:28.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:28.079+0000 7faf4d800700 1 -- 172.21.15.33:0/3938533392 wait complete. 2024-04-26T11:28:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:29 smithi033 ceph-mon[27845]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T11:28:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:29 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3938533392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T11:28:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:29 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3434543664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T11:28:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:29 smithi194 ceph-mon[33017]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T11:28:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:29 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3938533392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T11:28:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:29 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3434543664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T11:28:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:29 smithi096 ceph-mon[32753]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T11:28:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:29 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3938533392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T11:28:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:29 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3434543664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T11:28:29.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 -- 172.21.15.33:0/3143431449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 msgr2=0x7f0d28110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/3143431449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 0x7f0d28110320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009f90 tx=0x7f0d2000b330 comp rx=0 tx=0).stop 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 -- 172.21.15.33:0/3143431449 shutdown_connections 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/3143431449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 0x7f0d28110320 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/3143431449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281094e0 0x7f0d281098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/3143431449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d281034a0 0x7f0d28103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 -- 172.21.15.33:0/3143431449 >> 172.21.15.33:0/3143431449 conn(0x7f0d2807a4f0 msgr2=0x7f0d2807a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.637+0000 7f0d2dc00700 1 -- 172.21.15.33:0/3143431449 shutdown_connections 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.638+0000 7f0d2dc00700 1 -- 172.21.15.33:0/3143431449 wait complete. 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 Processor -- start 2024-04-26T11:28:29.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 -- start start 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281034a0 0x7f0d281a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 0x7f0d281a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d281094e0 0x7f0d2819bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d28113630 con 0x7f0d281094e0 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0d281134b0 con 0x7f0d28103e40 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d2dc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0d281137b0 con 0x7f0d281034a0 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d26a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281034a0 0x7f0d281a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d26a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281034a0 0x7f0d281a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:48850/0 (socket says 172.21.15.33:48850) 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d26a00700 1 -- 172.21.15.33:0/2713543035 learned_addr learned my addr 172.21.15.33:0/2713543035 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:29.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.639+0000 7f0d26000700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 0x7f0d281a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d27400700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d281094e0 0x7f0d2819bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d26a00700 1 -- 172.21.15.33:0/2713543035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 msgr2=0x7f0d281a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d26a00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 0x7f0d281a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d26a00700 1 -- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d281094e0 msgr2=0x7f0d2819bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d26a00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d281094e0 0x7f0d2819bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d26a00700 1 -- 172.21.15.33:0/2713543035 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2819c2a0 con 0x7f0d281034a0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d26000700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 0x7f0d281a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d26a00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281034a0 0x7f0d281a1930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800e9a0 tx=0x7f0d1800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.640+0000 7f0d1f400700 1 -- 172.21.15.33:0/2713543035 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1800c9c0 con 0x7f0d281034a0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.641+0000 7f0d1f400700 1 -- 172.21.15.33:0/2713543035 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d1800cb60 con 0x7f0d281034a0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.641+0000 7f0d1f400700 1 -- 172.21.15.33:0/2713543035 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18018680 con 0x7f0d281034a0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.641+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2819c590 con 0x7f0d281034a0 2024-04-26T11:28:29.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.641+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d281a8800 con 0x7f0d281034a0 2024-04-26T11:28:29.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.642+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2806a110 con 0x7f0d281034a0 2024-04-26T11:28:29.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.644+0000 7f0d1f400700 1 -- 172.21.15.33:0/2713543035 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0d18010430 con 0x7f0d281034a0 2024-04-26T11:28:29.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.645+0000 7f0d1f400700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0d0c0a3cc0 0x7f0d0c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:29.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.646+0000 7f0d1f400700 1 -- 172.21.15.33:0/2713543035 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0d180d0690 con 0x7f0d281034a0 2024-04-26T11:28:29.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.646+0000 7f0d26000700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0d0c0a3cc0 0x7f0d0c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:29.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.647+0000 7f0d26000700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0d0c0a3cc0 0x7f0d0c0a6180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0d28110760 tx=0x7f0d14009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:29.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.648+0000 7f0d1f400700 1 -- 172.21.15.33:0/2713543035 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d1809d070 con 0x7f0d281034a0 2024-04-26T11:28:29.916 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332152 got 219043332158 for osd.7 2024-04-26T11:28:29.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.913+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0d281a8a70 con 0x7f0d281034a0 2024-04-26T11:28:29.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.914+0000 7f0d1f400700 1 -- 172.21.15.33:0/2713543035 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0d1809c970 con 0x7f0d281034a0 2024-04-26T11:28:29.917 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:29.917 INFO:teuthology.orchestra.run.smithi033.stdout:197568495685 2024-04-26T11:28:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0d0c0a3cc0 msgr2=0x7f0d0c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0d0c0a3cc0 0x7f0d0c0a6180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0d28110760 tx=0x7f0d14009310 comp rx=0 tx=0).stop 2024-04-26T11:28:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281034a0 msgr2=0x7f0d281a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281034a0 0x7f0d281a1930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800e9a0 tx=0x7f0d1800ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 shutdown_connections 2024-04-26T11:28:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f0d0c0a3cc0 0x7f0d0c0a6180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d281094e0 0x7f0d2819bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d281034a0 0x7f0d281a1930 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 --2- 172.21.15.33:0/2713543035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d28103e40 0x7f0d281a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.918+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 >> 172.21.15.33:0/2713543035 conn(0x7f0d2807a4f0 msgr2=0x7f0d28101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.919+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 shutdown_connections 2024-04-26T11:28:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:29.919+0000 7f0d2dc00700 1 -- 172.21.15.33:0/2713543035 wait complete. 2024-04-26T11:28:30.080 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808880 got 287762808883 for osd.9 2024-04-26T11:28:30.080 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:30 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:28:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:30 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2713543035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T11:28:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:30 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:28:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:30 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2713543035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T11:28:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:30 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:28:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:30 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2713543035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T11:28:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:31 smithi096 ceph-mon[32753]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T11:28:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:31 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:31 smithi033 ceph-mon[27845]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T11:28:31.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:31 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:31 smithi194 ceph-mon[33017]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T11:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:31 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:33.073 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495681 got 197568495685 for osd.6 2024-04-26T11:28:33.073 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:33 smithi033 ceph-mon[27845]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:28:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:33 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2008479916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T11:28:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:33 smithi194 ceph-mon[33017]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:28:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:33 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2008479916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.608+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2812534189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 msgr2=0x7f8fc41166a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.608+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2812534189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 0x7f8fc41166a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00b0a0 tx=0x7f8fbc00b3b0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.609+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2812534189 shutdown_connections 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.609+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2812534189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 0x7f8fc41166a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.609+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2812534189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc4077950 0x7f8fc4077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.609+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2812534189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4078270 0x7f8fc4113d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.609+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2812534189 >> 172.21.15.33:0/2812534189 conn(0x7f8fc4101100 msgr2=0x7f8fc4103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.609+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2812534189 shutdown_connections 2024-04-26T11:28:33.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.609+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2812534189 wait complete. 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 Processor -- start 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 -- start start 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4077950 0x7f8fc41a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc4078270 0x7f8fc41a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 0x7f8fc41a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc4119c40 con 0x7f8fc4077950 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc4119ac0 con 0x7f8fc4078270 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.610+0000 7f8fcaa00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc4119dc0 con 0x7f8fc41142b0 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4077950 0x7f8fc41a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc8a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 0x7f8fc41a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4077950 0x7f8fc41a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58710/0 (socket says 172.21.15.33:58710) 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 -- 172.21.15.33:0/2008479916 learned_addr learned my addr 172.21.15.33:0/2008479916 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:33.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3400700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc4078270 0x7f8fc41a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 -- 172.21.15.33:0/2008479916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 msgr2=0x7f8fc41a49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 0x7f8fc41a49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 -- 172.21.15.33:0/2008479916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc4078270 msgr2=0x7f8fc41a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc4078270 0x7f8fc41a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 -- 172.21.15.33:0/2008479916 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc41a50e0 con 0x7f8fc4077950 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.611+0000 7f8fc3e00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4077950 0x7f8fc41a00d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400e9a0 tx=0x7f8fb400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.612+0000 7f8fc0c00700 1 -- 172.21.15.33:0/2008479916 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400ca10 con 0x7f8fc4077950 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.612+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc41a53d0 con 0x7f8fc4077950 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.612+0000 7f8fc0c00700 1 -- 172.21.15.33:0/2008479916 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fb400cbb0 con 0x7f8fc4077950 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.612+0000 7f8fc0c00700 1 -- 172.21.15.33:0/2008479916 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4018680 con 0x7f8fc4077950 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.612+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc41acc50 con 0x7f8fc4077950 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.613+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc406a110 con 0x7f8fc4077950 2024-04-26T11:28:33.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.615+0000 7f8fc0c00700 1 -- 172.21.15.33:0/2008479916 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8fb4010430 con 0x7f8fc4077950 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.617+0000 7f8fc0c00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8fa80a3c40 0x7f8fa80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.617+0000 7f8fc0c00700 1 -- 172.21.15.33:0/2008479916 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8fb40d05c0 con 0x7f8fc4077950 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.617+0000 7f8fc3400700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8fa80a3c40 0x7f8fa80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.618+0000 7f8fc3400700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8fa80a3c40 0x7f8fa80a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8fc41047c0 tx=0x7f8fb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.621+0000 7f8fc0c00700 1 -- 172.21.15.33:0/2008479916 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8fb409cf60 con 0x7f8fc4077950 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.853+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8fc402e490 con 0x7f8fc4077950 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.853+0000 7f8fc0c00700 1 -- 172.21.15.33:0/2008479916 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8fb409c860 con 0x7f8fc4077950 2024-04-26T11:28:33.682 INFO:teuthology.orchestra.run.smithi033.stdout:77309411421 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8fa80a3c40 msgr2=0x7f8fa80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8fa80a3c40 0x7f8fa80a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8fc41047c0 tx=0x7f8fb0009260 comp rx=0 tx=0).stop 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4077950 msgr2=0x7f8fc41a00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4077950 0x7f8fc41a00d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400e9a0 tx=0x7f8fb400ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 shutdown_connections 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8fa80a3c40 0x7f8fa80a6100 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fc41142b0 0x7f8fc41a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8fc4077950 0x7f8fc41a00d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 --2- 172.21.15.33:0/2008479916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc4078270 0x7f8fc41a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:33.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.856+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 >> 172.21.15.33:0/2008479916 conn(0x7f8fc4101100 msgr2=0x7f8fc41034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:33.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.857+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 shutdown_connections 2024-04-26T11:28:33.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:32.857+0000 7f8fcaa00700 1 -- 172.21.15.33:0/2008479916 wait complete. 2024-04-26T11:28:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:33 smithi096 ceph-mon[32753]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T11:28:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:33 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2008479916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T11:28:34.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.795+0000 7f8b69600700 1 -- 172.21.15.33:0/956372921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64109490 msgr2=0x7f8b64109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:34.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.795+0000 7f8b69600700 1 --2- 172.21.15.33:0/956372921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64109490 0x7f8b64109870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009990 tx=0x7f8b54009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:34.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.796+0000 7f8b69600700 1 -- 172.21.15.33:0/956372921 shutdown_connections 2024-04-26T11:28:34.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.796+0000 7f8b69600700 1 --2- 172.21.15.33:0/956372921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b6406c360 0x7f8b641102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:34.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.796+0000 7f8b69600700 1 --2- 172.21.15.33:0/956372921 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b64109490 0x7f8b64109870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:34.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.796+0000 7f8b69600700 1 --2- 172.21.15.33:0/956372921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b6406b9c0 0x7f8b6406be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:34.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.796+0000 7f8b69600700 1 -- 172.21.15.33:0/956372921 >> 172.21.15.33:0/956372921 conn(0x7f8b64079290 msgr2=0x7f8b640796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:34.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.796+0000 7f8b69600700 1 -- 172.21.15.33:0/956372921 shutdown_connections 2024-04-26T11:28:34.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.796+0000 7f8b69600700 1 -- 172.21.15.33:0/956372921 wait complete. 2024-04-26T11:28:34.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.797+0000 7f8b69600700 1 Processor -- start 2024-04-26T11:28:34.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b69600700 1 -- start start 2024-04-26T11:28:34.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b69600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b6406b9c0 0x7f8b6419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:34.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b69600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b6406c360 0x7f8b6419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:34.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b69600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b64109490 0x7f8b641a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:34.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b69600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641135d0 con 0x7f8b6406c360 2024-04-26T11:28:34.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b69600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64113450 con 0x7f8b64109490 2024-04-26T11:28:34.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b69600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64113750 con 0x7f8b6406b9c0 2024-04-26T11:28:34.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b62000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b6406c360 0x7f8b6419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:34.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b62000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b6406c360 0x7f8b6419c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58726/0 (socket says 172.21.15.33:58726) 2024-04-26T11:28:34.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b62000700 1 -- 172.21.15.33:0/1716164824 learned_addr learned my addr 172.21.15.33:0/1716164824 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:34.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.798+0000 7f8b62a00700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b6406b9c0 0x7f8b6419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:34.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.799+0000 7f8b63400700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b64109490 0x7f8b641a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.799+0000 7f8b62000700 1 -- 172.21.15.33:0/1716164824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b6406b9c0 msgr2=0x7f8b6419bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.799+0000 7f8b62000700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b6406b9c0 0x7f8b6419bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.799+0000 7f8b62000700 1 -- 172.21.15.33:0/1716164824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b64109490 msgr2=0x7f8b641a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.799+0000 7f8b62000700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b64109490 0x7f8b641a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.799+0000 7f8b62000700 1 -- 172.21.15.33:0/1716164824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b641a0d30 con 0x7f8b6406c360 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.799+0000 7f8b62000700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b6406c360 0x7f8b6419c260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000e9a0 tx=0x7f8b5000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.800+0000 7f8b5b400700 1 -- 172.21.15.33:0/1716164824 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5000c9c0 con 0x7f8b6406c360 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.800+0000 7f8b5b400700 1 -- 172.21.15.33:0/1716164824 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b5000cb60 con 0x7f8b6406c360 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.800+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b641a1020 con 0x7f8b6406c360 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.800+0000 7f8b5b400700 1 -- 172.21.15.33:0/1716164824 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50018680 con 0x7f8b6406c360 2024-04-26T11:28:34.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.800+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b641a88a0 con 0x7f8b6406c360 2024-04-26T11:28:34.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.801+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b6406a110 con 0x7f8b6406c360 2024-04-26T11:28:34.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.804+0000 7f8b5b400700 1 -- 172.21.15.33:0/1716164824 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8b50010430 con 0x7f8b6406c360 2024-04-26T11:28:34.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.805+0000 7f8b5b400700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8b480a3c40 0x7f8b480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:34.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.806+0000 7f8b5b400700 1 -- 172.21.15.33:0/1716164824 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8b500d06f0 con 0x7f8b6406c360 2024-04-26T11:28:34.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.806+0000 7f8b62a00700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8b480a3c40 0x7f8b480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:34.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.807+0000 7f8b62a00700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8b480a3c40 0x7f8b480a6100 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009750 tx=0x7f8b5400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:34.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.810+0000 7f8b5b400700 1 -- 172.21.15.33:0/1716164824 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b5009d090 con 0x7f8b6406c360 2024-04-26T11:28:35.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.058+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8b6402e490 con 0x7f8b6406c360 2024-04-26T11:28:35.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.059+0000 7f8b5b400700 1 -- 172.21.15.33:0/1716164824 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8b5009c990 con 0x7f8b6406c360 2024-04-26T11:28:35.061 INFO:teuthology.orchestra.run.smithi033.stdout:34359738470 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8b480a3c40 msgr2=0x7f8b480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8b480a3c40 0x7f8b480a6100 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009750 tx=0x7f8b5400b260 comp rx=0 tx=0).stop 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b6406c360 msgr2=0x7f8b6419c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b6406c360 0x7f8b6419c260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000e9a0 tx=0x7f8b5000ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 shutdown_connections 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8b480a3c40 0x7f8b480a6100 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b64109490 0x7f8b641a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b6406b9c0 0x7f8b6419bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 --2- 172.21.15.33:0/1716164824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b6406c360 0x7f8b6419c260 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:35.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 >> 172.21.15.33:0/1716164824 conn(0x7f8b64079290 msgr2=0x7f8b6410e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 shutdown_connections 2024-04-26T11:28:35.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:35.062+0000 7f8b69600700 1 -- 172.21.15.33:0/1716164824 wait complete. 2024-04-26T11:28:35.275 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411416 got 77309411421 for osd.2 2024-04-26T11:28:35.275 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:35 smithi033 ceph-mon[27845]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T11:28:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:35 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/824878553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T11:28:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:35 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1716164824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T11:28:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:35 smithi194 ceph-mon[33017]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T11:28:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:35 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/824878553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T11:28:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:35 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1716164824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T11:28:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:35 smithi096 ceph-mon[32753]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T11:28:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:35 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/824878553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T11:28:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:35 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1716164824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T11:28:36.488 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738466 got 34359738470 for osd.0 2024-04-26T11:28:36.489 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:36.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 -- 172.21.15.33:0/2002201357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 msgr2=0x7fab70075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:36.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 --2- 172.21.15.33:0/2002201357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 0x7fab70075a30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fab60009990 tx=0x7fab60009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 -- 172.21.15.33:0/2002201357 shutdown_connections 2024-04-26T11:28:36.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 --2- 172.21.15.33:0/2002201357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab7010fe20 0x7fab70112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 --2- 172.21.15.33:0/2002201357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 0x7fab70075a30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 --2- 172.21.15.33:0/2002201357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab70075f70 0x7fab7010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 -- 172.21.15.33:0/2002201357 >> 172.21.15.33:0/2002201357 conn(0x7fab7007a580 msgr2=0x7fab7007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.679+0000 7fab76e00700 1 -- 172.21.15.33:0/2002201357 shutdown_connections 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.680+0000 7fab76e00700 1 -- 172.21.15.33:0/2002201357 wait complete. 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.681+0000 7fab76e00700 1 Processor -- start 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.681+0000 7fab76e00700 1 -- start start 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab76e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 0x7fab7019bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab76e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab70075f70 0x7fab7019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab76e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab7010fe20 0x7fab701a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab76e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab70115810 con 0x7fab70075f70 2024-04-26T11:28:36.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab76e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fab70115690 con 0x7fab70075650 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab76e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fab70115990 con 0x7fab7010fe20 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab6f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab70075f70 0x7fab7019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab74e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab7010fe20 0x7fab701a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab6f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab70075f70 0x7fab7019c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58716/0 (socket says 172.21.15.33:58716) 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.683+0000 7fab6f400700 1 -- 172.21.15.33:0/824878553 learned_addr learned my addr 172.21.15.33:0/824878553 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.682+0000 7fab6fe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 0x7fab7019bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.683+0000 7fab6fe00700 1 -- 172.21.15.33:0/824878553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab7010fe20 msgr2=0x7fab701a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.683+0000 7fab6fe00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab7010fe20 0x7fab701a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.683+0000 7fab6fe00700 1 -- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab70075f70 msgr2=0x7fab7019c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.683+0000 7fab6fe00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab70075f70 0x7fab7019c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.683+0000 7fab6fe00700 1 -- 172.21.15.33:0/824878553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab701a0d50 con 0x7fab70075650 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.683+0000 7fab6fe00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 0x7fab7019bd40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fab600098e0 tx=0x7fab600057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:36.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.684+0000 7fab6cc00700 1 -- 172.21.15.33:0/824878553 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60027070 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.684+0000 7fab6cc00700 1 -- 172.21.15.33:0/824878553 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab60022470 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.684+0000 7fab6cc00700 1 -- 172.21.15.33:0/824878553 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60017650 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.684+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab701a0fe0 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.684+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab701a88c0 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.685+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7006a110 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.687+0000 7fab6cc00700 1 -- 172.21.15.33:0/824878553 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fab6002a070 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.688+0000 7fab6cc00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fab540a3cc0 0x7fab540a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.689+0000 7fab6cc00700 1 -- 172.21.15.33:0/824878553 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fab600d1060 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.689+0000 7fab6f400700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fab540a3cc0 0x7fab540a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.689+0000 7fab6f400700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fab540a3cc0 0x7fab540a6180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab70102fa0 tx=0x7fab5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.693+0000 7fab6cc00700 1 -- 172.21.15.33:0/824878553 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab6005eff0 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.961+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fab7002e490 con 0x7fab70075650 2024-04-26T11:28:36.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.962+0000 7fab6cc00700 1 -- 172.21.15.33:0/824878553 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fab60013070 con 0x7fab70075650 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stdout:107374182487 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fab540a3cc0 msgr2=0x7fab540a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fab540a3cc0 0x7fab540a6180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab70102fa0 tx=0x7fab5c009260 comp rx=0 tx=0).stop 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 msgr2=0x7fab7019bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 0x7fab7019bd40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fab600098e0 tx=0x7fab600057c0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 shutdown_connections 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fab540a3cc0 0x7fab540a6180 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab7010fe20 0x7fab701a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab70075650 0x7fab7019bd40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 --2- 172.21.15.33:0/824878553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab70075f70 0x7fab7019c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:36.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 >> 172.21.15.33:0/824878553 conn(0x7fab7007a580 msgr2=0x7fab70100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:36.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 shutdown_connections 2024-04-26T11:28:36.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:34.965+0000 7fab76e00700 1 -- 172.21.15.33:0/824878553 wait complete. 2024-04-26T11:28:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:37 smithi033 ceph-mon[27845]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:28:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:37 smithi194 ceph-mon[33017]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:28:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:37 smithi096 ceph-mon[32753]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:28:37.863 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182481 got 107374182487 for osd.3 2024-04-26T11:28:37.863 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:38.113 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:38.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574939 got 55834574945 for osd.1 2024-04-26T11:28:38.162 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:38.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.640+0000 7fd745400700 1 -- 172.21.15.33:0/2961772808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd740105640 msgr2=0x7fd740105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.640+0000 7fd745400700 1 --2- 172.21.15.33:0/2961772808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd740105640 0x7fd740105ac0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009990 tx=0x7fd72c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.640+0000 7fd745400700 1 -- 172.21.15.33:0/2961772808 shutdown_connections 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.640+0000 7fd745400700 1 --2- 172.21.15.33:0/2961772808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740106000 0x7fd740112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.640+0000 7fd745400700 1 --2- 172.21.15.33:0/2961772808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd74010b640 0x7fd74010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.640+0000 7fd745400700 1 --2- 172.21.15.33:0/2961772808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd740105640 0x7fd740105ac0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.640+0000 7fd745400700 1 -- 172.21.15.33:0/2961772808 >> 172.21.15.33:0/2961772808 conn(0x7fd740101130 msgr2=0x7fd740103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.641+0000 7fd745400700 1 -- 172.21.15.33:0/2961772808 shutdown_connections 2024-04-26T11:28:38.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.641+0000 7fd745400700 1 -- 172.21.15.33:0/2961772808 wait complete. 2024-04-26T11:28:38.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.642+0000 7fd745400700 1 Processor -- start 2024-04-26T11:28:38.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.642+0000 7fd745400700 1 -- start start 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd745400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740105640 0x7fd74019bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd745400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd740106000 0x7fd74019c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd745400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd74010b640 0x7fd7401a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd745400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd7401158d0 con 0x7fd740105640 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd745400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd740115750 con 0x7fd74010b640 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd745400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd740115a50 con 0x7fd740106000 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd73e800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740105640 0x7fd74019bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd73e800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740105640 0x7fd74019bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58744/0 (socket says 172.21.15.33:58744) 2024-04-26T11:28:38.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd73e800700 1 -- 172.21.15.33:0/3795984726 learned_addr learned my addr 172.21.15.33:0/3795984726 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:38.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.643+0000 7fd73de00700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd740106000 0x7fd74019c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:38.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73f200700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd74010b640 0x7fd7401a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73e800700 1 -- 172.21.15.33:0/3795984726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd740106000 msgr2=0x7fd74019c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73e800700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd740106000 0x7fd74019c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73e800700 1 -- 172.21.15.33:0/3795984726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd74010b640 msgr2=0x7fd7401a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73e800700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd74010b640 0x7fd7401a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73e800700 1 -- 172.21.15.33:0/3795984726 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7401a0d30 con 0x7fd740105640 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73f200700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd74010b640 0x7fd7401a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd73e800700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740105640 0x7fd74019bd20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e9a0 tx=0x7fd73000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.644+0000 7fd737400700 1 -- 172.21.15.33:0/3795984726 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73000cbd0 con 0x7fd740105640 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.645+0000 7fd737400700 1 -- 172.21.15.33:0/3795984726 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd73000cd70 con 0x7fd740105640 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.645+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7401a1020 con 0x7fd740105640 2024-04-26T11:28:38.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.645+0000 7fd737400700 1 -- 172.21.15.33:0/3795984726 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd730018680 con 0x7fd740105640 2024-04-26T11:28:38.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.645+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7401a88a0 con 0x7fd740105640 2024-04-26T11:28:38.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.646+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd74006a110 con 0x7fd740105640 2024-04-26T11:28:38.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.648+0000 7fd737400700 1 -- 172.21.15.33:0/3795984726 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd730010430 con 0x7fd740105640 2024-04-26T11:28:38.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.649+0000 7fd737400700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd7240a3c40 0x7fd7240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:38.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.649+0000 7fd737400700 1 -- 172.21.15.33:0/3795984726 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd7300d05f0 con 0x7fd740105640 2024-04-26T11:28:38.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.650+0000 7fd73de00700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd7240a3c40 0x7fd7240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:38.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.650+0000 7fd73de00700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd7240a3c40 0x7fd7240a6100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd740103770 tx=0x7fd72c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:38.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.654+0000 7fd737400700 1 -- 172.21.15.33:0/3795984726 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd73009cf90 con 0x7fd740105640 2024-04-26T11:28:38.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.927+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd74002e490 con 0x7fd740105640 2024-04-26T11:28:38.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.928+0000 7fd737400700 1 -- 172.21.15.33:0/3795984726 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd730085e20 con 0x7fd740105640 2024-04-26T11:28:38.930 INFO:teuthology.orchestra.run.smithi033.stdout:343597383722 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd7240a3c40 msgr2=0x7fd7240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd7240a3c40 0x7fd7240a6100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd740103770 tx=0x7fd72c00b260 comp rx=0 tx=0).stop 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740105640 msgr2=0x7fd74019bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740105640 0x7fd74019bd20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e9a0 tx=0x7fd73000ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 shutdown_connections 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fd7240a3c40 0x7fd7240a6100 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd74010b640 0x7fd7401a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd740105640 0x7fd74019bd20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 --2- 172.21.15.33:0/3795984726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd740106000 0x7fd74019c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:38.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 >> 172.21.15.33:0/3795984726 conn(0x7fd740101130 msgr2=0x7fd7401018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:38.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 shutdown_connections 2024-04-26T11:28:38.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:38.933+0000 7fd745400700 1 -- 172.21.15.33:0/3795984726 wait complete. 2024-04-26T11:28:39.204 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:39 smithi033 ceph-mon[27845]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:39.204 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:39 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3795984726' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T11:28:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:39 smithi194 ceph-mon[33017]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:39 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3795984726' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T11:28:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:39 smithi096 ceph-mon[32753]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:39 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3795984726' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T11:28:40.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 -- 172.21.15.33:0/2456711965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c24105640 msgr2=0x7f2c24105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:40.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 --2- 172.21.15.33:0/2456711965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c24105640 0x7f2c24105ac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2c100098f0 tx=0x7f2c10009c00 comp rx=0 tx=0).stop 2024-04-26T11:28:40.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 -- 172.21.15.33:0/2456711965 shutdown_connections 2024-04-26T11:28:40.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 --2- 172.21.15.33:0/2456711965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c24106000 0x7f2c24112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:40.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 --2- 172.21.15.33:0/2456711965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c2410b640 0x7f2c2410ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:40.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 --2- 172.21.15.33:0/2456711965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c24105640 0x7f2c24105ac0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:40.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 -- 172.21.15.33:0/2456711965 >> 172.21.15.33:0/2456711965 conn(0x7f2c24101130 msgr2=0x7f2c24103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:40.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 -- 172.21.15.33:0/2456711965 shutdown_connections 2024-04-26T11:28:40.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.936+0000 7f2c29000700 1 -- 172.21.15.33:0/2456711965 wait complete. 2024-04-26T11:28:40.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.937+0000 7f2c29000700 1 Processor -- start 2024-04-26T11:28:40.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c29000700 1 -- start start 2024-04-26T11:28:40.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c29000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c24105640 0x7f2c2419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:40.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c29000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c24106000 0x7f2c2419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:40.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c29000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c2410b640 0x7f2c241a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:40.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c29000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2c241158d0 con 0x7f2c24105640 2024-04-26T11:28:40.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c29000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2c24115750 con 0x7f2c24106000 2024-04-26T11:28:40.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c29000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2c24115a50 con 0x7f2c2410b640 2024-04-26T11:28:40.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c21e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c24105640 0x7f2c2419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:40.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c22800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c2410b640 0x7f2c241a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:40.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c22800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c2410b640 0x7f2c241a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50928/0 (socket says 172.21.15.33:50928) 2024-04-26T11:28:40.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.938+0000 7f2c22800700 1 -- 172.21.15.33:0/4254382225 learned_addr learned my addr 172.21.15.33:0/4254382225 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:40.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c21400700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c24106000 0x7f2c2419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:40.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c22800700 1 -- 172.21.15.33:0/4254382225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c24106000 msgr2=0x7f2c2419c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:40.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c22800700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c24106000 0x7f2c2419c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:40.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c22800700 1 -- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c24105640 msgr2=0x7f2c2419bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:40.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c22800700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c24105640 0x7f2c2419bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:40.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c22800700 1 -- 172.21.15.33:0/4254382225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c241a0d70 con 0x7f2c2410b640 2024-04-26T11:28:40.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c21e00700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c24105640 0x7f2c2419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:40.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.939+0000 7f2c22800700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c2410b640 0x7f2c241a0630 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00e3c0 tx=0x7f2c1c00e6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:40.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.940+0000 7f2c1aa00700 1 -- 172.21.15.33:0/4254382225 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c0090d0 con 0x7f2c2410b640 2024-04-26T11:28:40.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.940+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c241a1060 con 0x7f2c2410b640 2024-04-26T11:28:40.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.940+0000 7f2c1aa00700 1 -- 172.21.15.33:0/4254382225 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c1c009270 con 0x7f2c2410b640 2024-04-26T11:28:40.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.940+0000 7f2c1aa00700 1 -- 172.21.15.33:0/4254382225 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c007df0 con 0x7f2c2410b640 2024-04-26T11:28:40.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.940+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c241a88e0 con 0x7f2c2410b640 2024-04-26T11:28:40.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.941+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c2406a110 con 0x7f2c2410b640 2024-04-26T11:28:40.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.943+0000 7f2c1aa00700 1 -- 172.21.15.33:0/4254382225 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2c1c012050 con 0x7f2c2410b640 2024-04-26T11:28:40.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.944+0000 7f2c1aa00700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2c080a3c40 0x7f2c080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:40.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.945+0000 7f2c1aa00700 1 -- 172.21.15.33:0/4254382225 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f2c1c0cfe70 con 0x7f2c2410b640 2024-04-26T11:28:40.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.945+0000 7f2c21e00700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2c080a3c40 0x7f2c080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:40.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.945+0000 7f2c21e00700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2c080a3c40 0x7f2c080a6100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009fd0 tx=0x7f2c14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:40.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:40.949+0000 7f2c1aa00700 1 -- 172.21.15.33:0/4254382225 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c1c022060 con 0x7f2c2410b640 2024-04-26T11:28:41.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.203+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f2c2402e490 con 0x7f2c2410b640 2024-04-26T11:28:41.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.206+0000 7f2c1aa00700 1 -- 172.21.15.33:0/4254382225 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2c1c010040 con 0x7f2c2410b640 2024-04-26T11:28:41.208 INFO:teuthology.orchestra.run.smithi033.stdout:163208757325 2024-04-26T11:28:41.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2c080a3c40 msgr2=0x7f2c080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2c080a3c40 0x7f2c080a6100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009fd0 tx=0x7f2c14009260 comp rx=0 tx=0).stop 2024-04-26T11:28:41.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c2410b640 msgr2=0x7f2c241a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c2410b640 0x7f2c241a0630 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00e3c0 tx=0x7f2c1c00e6d0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 shutdown_connections 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2c080a3c40 0x7f2c080a6100 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c2410b640 0x7f2c241a0630 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c24105640 0x7f2c2419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 --2- 172.21.15.33:0/4254382225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c24106000 0x7f2c2419c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.209+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 >> 172.21.15.33:0/4254382225 conn(0x7f2c24101130 msgr2=0x7f2c24101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.210+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 shutdown_connections 2024-04-26T11:28:41.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.210+0000 7f2c29000700 1 -- 172.21.15.33:0/4254382225 wait complete. 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 -- 172.21.15.33:0/1283412777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c1094f0 msgr2=0x7fb69c1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 --2- 172.21.15.33:0/1283412777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c1094f0 0x7fb69c1098d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0098f0 tx=0x7fb68c009c00 comp rx=0 tx=0).stop 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 -- 172.21.15.33:0/1283412777 shutdown_connections 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 --2- 172.21.15.33:0/1283412777 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb69c103e30 0x7fb69c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 --2- 172.21.15.33:0/1283412777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c1094f0 0x7fb69c1098d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 --2- 172.21.15.33:0/1283412777 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103470 0x7fb69c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 -- 172.21.15.33:0/1283412777 >> 172.21.15.33:0/1283412777 conn(0x7fb69c07a5d0 msgr2=0x7fb69c07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.173+0000 7fb6a2600700 1 -- 172.21.15.33:0/1283412777 shutdown_connections 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.174+0000 7fb6a2600700 1 -- 172.21.15.33:0/1283412777 wait complete. 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.175+0000 7fb6a2600700 1 Processor -- start 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.175+0000 7fb6a2600700 1 -- start start 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb6a2600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c103470 0x7fb69c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb6a2600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103e30 0x7fb69c19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb6a2600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb69c1094f0 0x7fb69c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb6a2600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c113640 con 0x7fb69c1094f0 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb6a2600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c1134c0 con 0x7fb69c103470 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb6a2600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c1137c0 con 0x7fb69c103e30 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb69aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103e30 0x7fb69c19fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb69aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103e30 0x7fb69c19fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50950/0 (socket says 172.21.15.33:50950) 2024-04-26T11:28:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb69aa00700 1 -- 172.21.15.33:0/136297963 learned_addr learned my addr 172.21.15.33:0/136297963 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb69b400700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c103470 0x7fb69c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.176+0000 7fb69be00700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb69c1094f0 0x7fb69c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.177+0000 7fb69aa00700 1 -- 172.21.15.33:0/136297963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c103470 msgr2=0x7fb69c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.177+0000 7fb69aa00700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c103470 0x7fb69c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.177+0000 7fb69aa00700 1 -- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb69c1094f0 msgr2=0x7fb69c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.177+0000 7fb69aa00700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb69c1094f0 0x7fb69c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.177+0000 7fb69aa00700 1 -- 172.21.15.33:0/136297963 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69c19a120 con 0x7fb69c103e30 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.177+0000 7fb69aa00700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103e30 0x7fb69c19fd40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e9a0 tx=0x7fb68800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.177+0000 7fb693e00700 1 -- 172.21.15.33:0/136297963 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68800ca10 con 0x7fb69c103e30 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.178+0000 7fb693e00700 1 -- 172.21.15.33:0/136297963 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb68800cbb0 con 0x7fb69c103e30 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.178+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69c19a410 con 0x7fb69c103e30 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.178+0000 7fb693e00700 1 -- 172.21.15.33:0/136297963 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688018680 con 0x7fb69c103e30 2024-04-26T11:28:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.178+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69c1a66d0 con 0x7fb69c103e30 2024-04-26T11:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.179+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb69c06a110 con 0x7fb69c103e30 2024-04-26T11:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.182+0000 7fb693e00700 1 -- 172.21.15.33:0/136297963 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb688010430 con 0x7fb69c103e30 2024-04-26T11:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.184+0000 7fb693e00700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb6800a3cc0 0x7fb6800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.184+0000 7fb693e00700 1 -- 172.21.15.33:0/136297963 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb6880d05e0 con 0x7fb69c103e30 2024-04-26T11:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.185+0000 7fb69b400700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb6800a3cc0 0x7fb6800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.185+0000 7fb693e00700 1 -- 172.21.15.33:0/136297963 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb68809cf80 con 0x7fb69c103e30 2024-04-26T11:28:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.185+0000 7fb69b400700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb6800a3cc0 0x7fb6800a6180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb69c19ab00 tx=0x7fb68c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.448+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fb69c02e490 con 0x7fb69c103e30 2024-04-26T11:28:41.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.449+0000 7fb693e00700 1 -- 172.21.15.33:0/136297963 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb68809c880 con 0x7fb69c103e30 2024-04-26T11:28:41.450 INFO:teuthology.orchestra.run.smithi033.stdout:253403070525 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.451+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb6800a3cc0 msgr2=0x7fb6800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.451+0000 7fb6a2600700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb6800a3cc0 0x7fb6800a6180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb69c19ab00 tx=0x7fb68c019040 comp rx=0 tx=0).stop 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.451+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103e30 msgr2=0x7fb69c19fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.451+0000 7fb6a2600700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103e30 0x7fb69c19fd40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e9a0 tx=0x7fb68800ecb0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 shutdown_connections 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb6800a3cc0 0x7fb6800a6180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb69c1094f0 0x7fb69c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb69c103470 0x7fb69c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 --2- 172.21.15.33:0/136297963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb69c103e30 0x7fb69c19fd40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 >> 172.21.15.33:0/136297963 conn(0x7fb69c07a5d0 msgr2=0x7fb69c1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 shutdown_connections 2024-04-26T11:28:41.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.452+0000 7fb6a2600700 1 -- 172.21.15.33:0/136297963 wait complete. 2024-04-26T11:28:41.472 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:41 smithi033 ceph-mon[27845]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:41.485 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:41 smithi096 ceph-mon[32753]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:41 smithi194 ceph-mon[33017]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:41.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 -- 172.21.15.33:0/2080742890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806b130 msgr2=0x7f4e7806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/2080742890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806b130 0x7f4e7806b590 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4e640098f0 tx=0x7f4e64009c00 comp rx=0 tx=0).stop 2024-04-26T11:28:41.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 -- 172.21.15.33:0/2080742890 shutdown_connections 2024-04-26T11:28:41.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/2080742890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7806bad0 0x7f4e781102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/2080742890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e781094a0 0x7f4e78109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/2080742890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806b130 0x7f4e7806b590 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 -- 172.21.15.33:0/2080742890 >> 172.21.15.33:0/2080742890 conn(0x7f4e780792d0 msgr2=0x7f4e780796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:41.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 -- 172.21.15.33:0/2080742890 shutdown_connections 2024-04-26T11:28:41.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.287+0000 7f4e7ee00700 1 -- 172.21.15.33:0/2080742890 wait complete. 2024-04-26T11:28:41.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.288+0000 7f4e7ee00700 1 Processor -- start 2024-04-26T11:28:41.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.288+0000 7f4e7ee00700 1 -- start start 2024-04-26T11:28:41.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e7ee00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7806b130 0x7f4e781023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e7ee00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806bad0 0x7f4e78102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e7ee00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781094a0 0x7f4e78102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e7ee00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e78113600 con 0x7f4e7806bad0 2024-04-26T11:28:41.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e7ee00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e78113480 con 0x7f4e7806b130 2024-04-26T11:28:41.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e7ee00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e78113780 con 0x7f4e781094a0 2024-04-26T11:28:41.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e77e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7806b130 0x7f4e781023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e77400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806bad0 0x7f4e78102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e7ce00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781094a0 0x7f4e78102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e77e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7806b130 0x7f4e781023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56038/0 (socket says 172.21.15.33:56038) 2024-04-26T11:28:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e77400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806bad0 0x7f4e78102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36300/0 (socket says 172.21.15.33:36300) 2024-04-26T11:28:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.289+0000 7f4e77e00700 1 -- 172.21.15.33:0/3203650546 learned_addr learned my addr 172.21.15.33:0/3203650546 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e77e00700 1 -- 172.21.15.33:0/3203650546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781094a0 msgr2=0x7f4e78102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e77e00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781094a0 0x7f4e78102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e77e00700 1 -- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806bad0 msgr2=0x7f4e78102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e77e00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806bad0 0x7f4e78102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e77e00700 1 -- 172.21.15.33:0/3203650546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e781a8100 con 0x7f4e7806b130 2024-04-26T11:28:41.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e7ce00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781094a0 0x7f4e78102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:41.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e77e00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7806b130 0x7f4e781023f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800eb30 tx=0x7f4e6800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:41.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e74c00700 1 -- 172.21.15.33:0/3203650546 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800cbd0 con 0x7f4e7806b130 2024-04-26T11:28:41.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e74c00700 1 -- 172.21.15.33:0/3203650546 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e680044d0 con 0x7f4e7806b130 2024-04-26T11:28:41.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e781a83f0 con 0x7f4e7806b130 2024-04-26T11:28:41.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e74c00700 1 -- 172.21.15.33:0/3203650546 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68003e30 con 0x7f4e7806b130 2024-04-26T11:28:41.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.290+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e781a8900 con 0x7f4e7806b130 2024-04-26T11:28:41.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.292+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7806a110 con 0x7f4e7806b130 2024-04-26T11:28:41.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.293+0000 7f4e74c00700 1 -- 172.21.15.33:0/3203650546 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4e68004000 con 0x7f4e7806b130 2024-04-26T11:28:41.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.295+0000 7f4e74c00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4e5c0a3c40 0x7f4e5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:41.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.295+0000 7f4e74c00700 1 -- 172.21.15.33:0/3203650546 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f4e680d0730 con 0x7f4e7806b130 2024-04-26T11:28:41.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.295+0000 7f4e77400700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4e5c0a3c40 0x7f4e5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:41.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.296+0000 7f4e77400700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4e5c0a3c40 0x7f4e5c0a6100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4e640098c0 tx=0x7f4e64019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:41.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.299+0000 7f4e74c00700 1 -- 172.21.15.33:0/3203650546 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e680042f0 con 0x7f4e7806b130 2024-04-26T11:28:41.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.553+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4e78103950 con 0x7f4e7806b130 2024-04-26T11:28:41.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.554+0000 7f4e74c00700 1 -- 172.21.15.33:0/3203650546 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4e78103950 con 0x7f4e7806b130 2024-04-26T11:28:41.661 INFO:teuthology.orchestra.run.smithi033.stdout:141733920851 2024-04-26T11:28:41.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.557+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4e5c0a3c40 msgr2=0x7f4e5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.557+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4e5c0a3c40 0x7f4e5c0a6100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4e640098c0 tx=0x7f4e64019040 comp rx=0 tx=0).stop 2024-04-26T11:28:41.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.557+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7806b130 msgr2=0x7f4e781023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:41.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.557+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7806b130 0x7f4e781023f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800eb30 tx=0x7f4e6800ee40 comp rx=0 tx=0).stop 2024-04-26T11:28:41.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 shutdown_connections 2024-04-26T11:28:41.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f4e5c0a3c40 0x7f4e5c0a6100 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781094a0 0x7f4e78102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7806b130 0x7f4e781023f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 --2- 172.21.15.33:0/3203650546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7806bad0 0x7f4e78102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:41.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 >> 172.21.15.33:0/3203650546 conn(0x7f4e780792d0 msgr2=0x7f4e78100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:41.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 shutdown_connections 2024-04-26T11:28:41.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:41.558+0000 7f4e7ee00700 1 -- 172.21.15.33:0/3203650546 wait complete. 2024-04-26T11:28:42.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:42 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/4254382225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T11:28:42.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:42 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/136297963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T11:28:42.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:42 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3203650546' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T11:28:42.498 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383717 got 343597383722 for osd.11 2024-04-26T11:28:42.499 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:42 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/4254382225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T11:28:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:42 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/136297963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T11:28:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:42 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3203650546' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T11:28:42.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:42 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/4254382225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T11:28:42.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:42 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/136297963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T11:28:42.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:42 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3203650546' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T11:28:43.239 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920846 got 141733920851 for osd.4 2024-04-26T11:28:43.239 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:43 smithi033 ceph-mon[27845]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:43 smithi194 ceph-mon[33017]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:43 smithi096 ceph-mon[32753]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:44.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070518 got 253403070525 for osd.8 2024-04-26T11:28:44.258 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:44.570 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757319 got 163208757325 for osd.5 2024-04-26T11:28:44.570 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.079+0000 7f2572e00700 1 -- 172.21.15.33:0/4255969671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06bad0 msgr2=0x7f256c1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.079+0000 7f2572e00700 1 --2- 172.21.15.33:0/4255969671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06bad0 0x7f256c1102e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2564009f90 tx=0x7f256400b330 comp rx=0 tx=0).stop 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.080+0000 7f2572e00700 1 -- 172.21.15.33:0/4255969671 shutdown_connections 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.080+0000 7f2572e00700 1 --2- 172.21.15.33:0/4255969671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06bad0 0x7f256c1102e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.080+0000 7f2572e00700 1 --2- 172.21.15.33:0/4255969671 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256c1094a0 0x7f256c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.080+0000 7f2572e00700 1 --2- 172.21.15.33:0/4255969671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f256c06b130 0x7f256c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.080+0000 7f2572e00700 1 -- 172.21.15.33:0/4255969671 >> 172.21.15.33:0/4255969671 conn(0x7f256c0792d0 msgr2=0x7f256c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:45.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.080+0000 7f2572e00700 1 -- 172.21.15.33:0/4255969671 shutdown_connections 2024-04-26T11:28:45.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.080+0000 7f2572e00700 1 -- 172.21.15.33:0/4255969671 wait complete. 2024-04-26T11:28:45.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.081+0000 7f2572e00700 1 Processor -- start 2024-04-26T11:28:45.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.081+0000 7f2572e00700 1 -- start start 2024-04-26T11:28:45.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f2572e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06b130 0x7f256c1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:45.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f2572e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f256c06bad0 0x7f256c102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:45.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f2572e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256c1094a0 0x7f256c102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:45.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f2572e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f256c113620 con 0x7f256c1094a0 2024-04-26T11:28:45.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f2572e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1134a0 con 0x7f256c06b130 2024-04-26T11:28:45.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f2572e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1137a0 con 0x7f256c06bad0 2024-04-26T11:28:45.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f256be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06b130 0x7f256c1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:45.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f256be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06b130 0x7f256c1023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56050/0 (socket says 172.21.15.33:56050) 2024-04-26T11:28:45.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f256be00700 1 -- 172.21.15.33:0/2441989930 learned_addr learned my addr 172.21.15.33:0/2441989930 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:45.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f256b400700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f256c06bad0 0x7f256c102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:45.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.082+0000 7f2570e00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256c1094a0 0x7f256c102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:45.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f256be00700 1 -- 172.21.15.33:0/2441989930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f256c06bad0 msgr2=0x7f256c102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:45.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f256be00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f256c06bad0 0x7f256c102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f256be00700 1 -- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256c1094a0 msgr2=0x7f256c102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f256be00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256c1094a0 0x7f256c102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f256be00700 1 -- 172.21.15.33:0/2441989930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c1a80c0 con 0x7f256c06b130 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f256b400700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f256c06bad0 0x7f256c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f256be00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06b130 0x7f256c1023f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f255c00e9c0 tx=0x7f255c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f2568c00700 1 -- 172.21.15.33:0/2441989930 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c00cbd0 con 0x7f256c06b130 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f2568c00700 1 -- 172.21.15.33:0/2441989930 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f255c00cd70 con 0x7f256c06b130 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f2568c00700 1 -- 172.21.15.33:0/2441989930 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c005580 con 0x7f256c06b130 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256c1a83b0 con 0x7f256c06b130 2024-04-26T11:28:45.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.083+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f256c1a8940 con 0x7f256c06b130 2024-04-26T11:28:45.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.085+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f256c06a110 con 0x7f256c06b130 2024-04-26T11:28:45.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.087+0000 7f2568c00700 1 -- 172.21.15.33:0/2441989930 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f255c004750 con 0x7f256c06b130 2024-04-26T11:28:45.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.089+0000 7f2568c00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f25500a3cc0 0x7f25500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:45.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.089+0000 7f2568c00700 1 -- 172.21.15.33:0/2441989930 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f255c0d0850 con 0x7f256c06b130 2024-04-26T11:28:45.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.089+0000 7f256b400700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f25500a3cc0 0x7f25500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:45.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.090+0000 7f256b400700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f25500a3cc0 0x7f25500a6180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f256c0799d0 tx=0x7f2558009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:45.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.090+0000 7f2568c00700 1 -- 172.21.15.33:0/2441989930 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f255c09d280 con 0x7f256c06b130 2024-04-26T11:28:45.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.367+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f256c103920 con 0x7f256c06b130 2024-04-26T11:28:45.368 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:45 smithi033 ceph-mon[27845]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:45.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.367+0000 7f2568c00700 1 -- 172.21.15.33:0/2441989930 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f255c09cb80 con 0x7f256c06b130 2024-04-26T11:28:45.369 INFO:teuthology.orchestra.run.smithi033.stdout:309237645361 2024-04-26T11:28:45.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f25500a3cc0 msgr2=0x7f25500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:45.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f25500a3cc0 0x7f25500a6180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f256c0799d0 tx=0x7f2558009260 comp rx=0 tx=0).stop 2024-04-26T11:28:45.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06b130 msgr2=0x7f256c1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:45.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06b130 0x7f256c1023f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f255c00e9c0 tx=0x7f255c00ecd0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 shutdown_connections 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f25500a3cc0 0x7f25500a6180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f256c1094a0 0x7f256c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f256c06b130 0x7f256c1023f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 --2- 172.21.15.33:0/2441989930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f256c06bad0 0x7f256c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.370+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 >> 172.21.15.33:0/2441989930 conn(0x7f256c0792d0 msgr2=0x7f256c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.371+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 shutdown_connections 2024-04-26T11:28:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:45.371+0000 7f2572e00700 1 -- 172.21.15.33:0/2441989930 wait complete. 2024-04-26T11:28:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:45 smithi194 ceph-mon[33017]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:45 smithi096 ceph-mon[32753]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:46.158 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645354 got 309237645361 for osd.10 2024-04-26T11:28:46.159 DEBUG:teuthology.parallel:result is None 2024-04-26T11:28:46.159 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T11:28:46.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T11:28:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:46 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2441989930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T11:28:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:46 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2441989930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T11:28:46.607 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:46 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2441989930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T11:28:47.345 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:47 smithi033 ceph-mon[27845]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:47.346 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:47.346 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:47 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:47 smithi194 ceph-mon[33017]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:47 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:47 smithi096 ceph-mon[32753]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:47 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:48 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:28:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:48 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:28:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:48 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:48.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:48 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:28:48.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:48 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:28:48.657 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:48 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:48 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:28:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:48 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:28:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:48 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:28:48.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 -- 172.21.15.33:0/2481985051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f630410b640 msgr2=0x7f630410ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:48.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 --2- 172.21.15.33:0/2481985051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f630410b640 0x7f630410ba20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62f400b0a0 tx=0x7f62f400b3b0 comp rx=0 tx=0).stop 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 -- 172.21.15.33:0/2481985051 shutdown_connections 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 --2- 172.21.15.33:0/2481985051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6304106000 0x7f6304112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 --2- 172.21.15.33:0/2481985051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f630410b640 0x7f630410ba20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 --2- 172.21.15.33:0/2481985051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304105640 0x7f6304105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 -- 172.21.15.33:0/2481985051 >> 172.21.15.33:0/2481985051 conn(0x7f6304101130 msgr2=0x7f6304103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 -- 172.21.15.33:0/2481985051 shutdown_connections 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.819+0000 7f6309600700 1 -- 172.21.15.33:0/2481985051 wait complete. 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.821+0000 7f6309600700 1 Processor -- start 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.821+0000 7f6309600700 1 -- start start 2024-04-26T11:28:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.821+0000 7f6309600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304105640 0x7f630419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6309600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304106000 0x7f630419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6309600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410b640 0x7f63041a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6309600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6304115890 con 0x7f6304106000 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6309600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6304115710 con 0x7f630410b640 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6309600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6304115a10 con 0x7f6304105640 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6302a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304105640 0x7f630419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6302a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304105640 0x7f630419bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50984/0 (socket says 172.21.15.33:50984) 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6302a00700 1 -- 172.21.15.33:0/3506626771 learned_addr learned my addr 172.21.15.33:0/3506626771 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f6303400700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410b640 0x7f63041a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.822+0000 7f62fa000700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304106000 0x7f630419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f6303400700 1 -- 172.21.15.33:0/3506626771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304105640 msgr2=0x7f630419bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:48.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f6303400700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304105640 0x7f630419bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:48.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f6303400700 1 -- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304106000 msgr2=0x7f630419c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:48.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f6303400700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304106000 0x7f630419c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f6303400700 1 -- 172.21.15.33:0/3506626771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63041a0d50 con 0x7f630410b640 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f62fa000700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304106000 0x7f630419c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f6303400700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410b640 0x7f63041a0610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00c970 tx=0x7f62fc00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f62fbe00700 1 -- 172.21.15.33:0/3506626771 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc00e880 con 0x7f630410b640 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f62fbe00700 1 -- 172.21.15.33:0/3506626771 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62fc00f450 con 0x7f630410b640 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.823+0000 7f62fbe00700 1 -- 172.21.15.33:0/3506626771 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc018660 con 0x7f630410b640 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.824+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63041a1040 con 0x7f630410b640 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.824+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63041a88c0 con 0x7f630410b640 2024-04-26T11:28:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.825+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f630406a110 con 0x7f630410b640 2024-04-26T11:28:48.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.827+0000 7f62fbe00700 1 -- 172.21.15.33:0/3506626771 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f62fc00ea20 con 0x7f630410b640 2024-04-26T11:28:48.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.828+0000 7f62fbe00700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3cc0 0x7f62e80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:48.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.828+0000 7f62fbe00700 1 -- 172.21.15.33:0/3506626771 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f62fc0d0360 con 0x7f630410b640 2024-04-26T11:28:48.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.829+0000 7f6302a00700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3cc0 0x7f62e80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:48.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.829+0000 7f6302a00700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3cc0 0x7f62e80a6180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009e60 tx=0x7f62f4006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:48.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:48.830+0000 7f62fbe00700 1 -- 172.21.15.33:0/3506626771 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62fc014030 con 0x7f630410b640 2024-04-26T11:28:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.048+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6304063860 con 0x7f62e80a3cc0 2024-04-26T11:28:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.072+0000 7f62fbe00700 1 -- 172.21.15.33:0/3506626771 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259124 (secure 0 0 0) 0x7f6304063860 con 0x7f62e80a3cc0 2024-04-26T11:28:49.079 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:28:49.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3cc0 msgr2=0x7f62e80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:49.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3cc0 0x7f62e80a6180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009e60 tx=0x7f62f4006230 comp rx=0 tx=0).stop 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410b640 msgr2=0x7f63041a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410b640 0x7f63041a0610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00c970 tx=0x7f62fc00cc80 comp rx=0 tx=0).stop 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 shutdown_connections 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f62e80a3cc0 0x7f62e80a6180 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f630410b640 0x7f63041a0610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6304105640 0x7f630419bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 --2- 172.21.15.33:0/3506626771 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6304106000 0x7f630419c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 >> 172.21.15.33:0/3506626771 conn(0x7f6304101130 msgr2=0x7f63041016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 shutdown_connections 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:49.078+0000 7f6309600700 1 -- 172.21.15.33:0/3506626771 wait complete. 2024-04-26T11:28:49.082 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-26T11:28:49.834 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:49 smithi033 ceph-mon[27845]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:49 smithi194 ceph-mon[33017]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:49.902 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-04-26T11:28:47.863051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660643,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":843,"num_read_kb":547,"num_write":360,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356992,"kb_used_data":12000,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584832,"statfs":{"total":1151940427776,"available":1151574867968,"internally_reserved":0,"allocated":12288000,"data_stored":7734561,"data_compressed":182166,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005438"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825734+0000","last_change":"2024-04-26T11:26:29.018818+0000","last_active":"2024-04-26T11:28:17.825734+0000","last_peered":"2024-04-26T11:28:17.825734+0000","last_clean":"2024-04-26T11:28:17.825734+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:28:17.825734+0000","last_undegraded":"2024-04-26T11:28:17.825734+0000","last_fullsized":"2024-04-26T11:28:17.825734+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:05.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.827907+0000","last_change":"2024-04-26T11:26:29.021338+0000","last_active":"2024-04-26T11:28:17.827907+0000","last_peered":"2024-04-26T11:28:17.827907+0000","last_clean":"2024-04-26T11:28:17.827907+0000","last_became_active":"2024-04-26T11:26:29.021019+0000","last_became_peered":"2024-04-26T11:26:29.021019+0000","last_unstale":"2024-04-26T11:28:17.827907+0000","last_undegraded":"2024-04-26T11:28:17.827907+0000","last_fullsized":"2024-04-26T11:28:17.827907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:52:44.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.832078+0000","last_change":"2024-04-26T11:26:29.014166+0000","last_active":"2024-04-26T11:28:17.832078+0000","last_peered":"2024-04-26T11:28:17.832078+0000","last_clean":"2024-04-26T11:28:17.832078+0000","last_became_active":"2024-04-26T11:26:29.013812+0000","last_became_peered":"2024-04-26T11:26:29.013812+0000","last_unstale":"2024-04-26T11:28:17.832078+0000","last_undegraded":"2024-04-26T11:28:17.832078+0000","last_fullsized":"2024-04-26T11:28:17.832078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:08.683633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.800283+0000","last_change":"2024-04-26T11:26:28.999249+0000","last_active":"2024-04-26T11:28:17.800283+0000","last_peered":"2024-04-26T11:28:17.800283+0000","last_clean":"2024-04-26T11:28:17.800283+0000","last_became_active":"2024-04-26T11:26:28.998927+0000","last_became_peered":"2024-04-26T11:26:28.998927+0000","last_unstale":"2024-04-26T11:28:17.800283+0000","last_undegraded":"2024-04-26T11:28:17.800283+0000","last_fullsized":"2024-04-26T11:28:17.800283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:56.674537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.836010+0000","last_change":"2024-04-26T11:26:29.017202+0000","last_active":"2024-04-26T11:28:17.836010+0000","last_peered":"2024-04-26T11:28:17.836010+0000","last_clean":"2024-04-26T11:28:17.836010+0000","last_became_active":"2024-04-26T11:26:29.016849+0000","last_became_peered":"2024-04-26T11:26:29.016849+0000","last_unstale":"2024-04-26T11:28:17.836010+0000","last_undegraded":"2024-04-26T11:28:17.836010+0000","last_fullsized":"2024-04-26T11:28:17.836010+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:58:22.164118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.816779+0000","last_change":"2024-04-26T11:27:19.278294+0000","last_active":"2024-04-26T11:28:17.816779+0000","last_peered":"2024-04-26T11:28:17.816779+0000","last_clean":"2024-04-26T11:28:17.816779+0000","last_became_active":"2024-04-26T11:27:19.277853+0000","last_became_peered":"2024-04-26T11:27:19.277853+0000","last_unstale":"2024-04-26T11:28:17.816779+0000","last_undegraded":"2024-04-26T11:28:17.816779+0000","last_fullsized":"2024-04-26T11:28:17.816779+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:12:18.733166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.811178+0000","last_change":"2024-04-26T11:26:28.991406+0000","last_active":"2024-04-26T11:28:17.811178+0000","last_peered":"2024-04-26T11:28:17.811178+0000","last_clean":"2024-04-26T11:28:17.811178+0000","last_became_active":"2024-04-26T11:26:28.989303+0000","last_became_peered":"2024-04-26T11:26:28.989303+0000","last_unstale":"2024-04-26T11:28:17.811178+0000","last_undegraded":"2024-04-26T11:28:17.811178+0000","last_fullsized":"2024-04-26T11:28:17.811178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:42:59.449078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.836608+0000","last_change":"2024-04-26T11:26:29.020922+0000","last_active":"2024-04-26T11:28:17.836608+0000","last_peered":"2024-04-26T11:28:17.836608+0000","last_clean":"2024-04-26T11:28:17.836608+0000","last_became_active":"2024-04-26T11:26:29.020576+0000","last_became_peered":"2024-04-26T11:26:29.020576+0000","last_unstale":"2024-04-26T11:28:17.836608+0000","last_undegraded":"2024-04-26T11:28:17.836608+0000","last_fullsized":"2024-04-26T11:28:17.836608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:48.770319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.817013+0000","last_change":"2024-04-26T11:26:29.004171+0000","last_active":"2024-04-26T11:28:17.817013+0000","last_peered":"2024-04-26T11:28:17.817013+0000","last_clean":"2024-04-26T11:28:17.817013+0000","last_became_active":"2024-04-26T11:26:29.003816+0000","last_became_peered":"2024-04-26T11:26:29.003816+0000","last_unstale":"2024-04-26T11:28:17.817013+0000","last_undegraded":"2024-04-26T11:28:17.817013+0000","last_fullsized":"2024-04-26T11:28:17.817013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:45:14.807634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.809247+0000","last_change":"2024-04-26T11:26:28.999453+0000","last_active":"2024-04-26T11:28:17.809247+0000","last_peered":"2024-04-26T11:28:17.809247+0000","last_clean":"2024-04-26T11:28:17.809247+0000","last_became_active":"2024-04-26T11:26:28.999104+0000","last_became_peered":"2024-04-26T11:26:28.999104+0000","last_unstale":"2024-04-26T11:28:17.809247+0000","last_undegraded":"2024-04-26T11:28:17.809247+0000","last_fullsized":"2024-04-26T11:28:17.809247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:55.408949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825591+0000","last_change":"2024-04-26T11:26:28.999218+0000","last_active":"2024-04-26T11:28:17.825591+0000","last_peered":"2024-04-26T11:28:17.825591+0000","last_clean":"2024-04-26T11:28:17.825591+0000","last_became_active":"2024-04-26T11:26:28.998584+0000","last_became_peered":"2024-04-26T11:26:28.998584+0000","last_unstale":"2024-04-26T11:28:17.825591+0000","last_undegraded":"2024-04-26T11:28:17.825591+0000","last_fullsized":"2024-04-26T11:28:17.825591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:18.040520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.798069+0000","last_change":"2024-04-26T11:26:28.997121+0000","last_active":"2024-04-26T11:28:17.798069+0000","last_peered":"2024-04-26T11:28:17.798069+0000","last_clean":"2024-04-26T11:28:17.798069+0000","last_became_active":"2024-04-26T11:26:28.996556+0000","last_became_peered":"2024-04-26T11:26:28.996556+0000","last_unstale":"2024-04-26T11:28:17.798069+0000","last_undegraded":"2024-04-26T11:28:17.798069+0000","last_fullsized":"2024-04-26T11:28:17.798069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:18:55.251371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.826697+0000","last_change":"2024-04-26T11:26:28.999525+0000","last_active":"2024-04-26T11:28:17.826697+0000","last_peered":"2024-04-26T11:28:17.826697+0000","last_clean":"2024-04-26T11:28:17.826697+0000","last_became_active":"2024-04-26T11:26:28.998578+0000","last_became_peered":"2024-04-26T11:26:28.998578+0000","last_unstale":"2024-04-26T11:28:17.826697+0000","last_undegraded":"2024-04-26T11:28:17.826697+0000","last_fullsized":"2024-04-26T11:28:17.826697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:32:35.523804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.829477+0000","last_change":"2024-04-26T11:26:29.008427+0000","last_active":"2024-04-26T11:28:17.829477+0000","last_peered":"2024-04-26T11:28:17.829477+0000","last_clean":"2024-04-26T11:28:17.829477+0000","last_became_active":"2024-04-26T11:26:29.008073+0000","last_became_peered":"2024-04-26T11:26:29.008073+0000","last_unstale":"2024-04-26T11:28:17.829477+0000","last_undegraded":"2024-04-26T11:28:17.829477+0000","last_fullsized":"2024-04-26T11:28:17.829477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:03.185941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.828903+0000","last_change":"2024-04-26T11:26:29.015136+0000","last_active":"2024-04-26T11:28:17.828903+0000","last_peered":"2024-04-26T11:28:17.828903+0000","last_clean":"2024-04-26T11:28:17.828903+0000","last_became_active":"2024-04-26T11:26:29.014800+0000","last_became_peered":"2024-04-26T11:26:29.014800+0000","last_unstale":"2024-04-26T11:28:17.828903+0000","last_undegraded":"2024-04-26T11:28:17.828903+0000","last_fullsized":"2024-04-26T11:28:17.828903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:28:35.871046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.831711+0000","last_change":"2024-04-26T11:26:29.024795+0000","last_active":"2024-04-26T11:28:17.831711+0000","last_peered":"2024-04-26T11:28:17.831711+0000","last_clean":"2024-04-26T11:28:17.831711+0000","last_became_active":"2024-04-26T11:26:29.024430+0000","last_became_peered":"2024-04-26T11:26:29.024430+0000","last_unstale":"2024-04-26T11:28:17.831711+0000","last_undegraded":"2024-04-26T11:28:17.831711+0000","last_fullsized":"2024-04-26T11:28:17.831711+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:05:20.172869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825816+0000","last_change":"2024-04-26T11:26:29.018822+0000","last_active":"2024-04-26T11:28:17.825816+0000","last_peered":"2024-04-26T11:28:17.825816+0000","last_clean":"2024-04-26T11:28:17.825816+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:28:17.825816+0000","last_undegraded":"2024-04-26T11:28:17.825816+0000","last_fullsized":"2024-04-26T11:28:17.825816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:04.475400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.818128+0000","last_change":"2024-04-26T11:26:29.001984+0000","last_active":"2024-04-26T11:28:17.818128+0000","last_peered":"2024-04-26T11:28:17.818128+0000","last_clean":"2024-04-26T11:28:17.818128+0000","last_became_active":"2024-04-26T11:26:28.999492+0000","last_became_peered":"2024-04-26T11:26:28.999492+0000","last_unstale":"2024-04-26T11:28:17.818128+0000","last_undegraded":"2024-04-26T11:28:17.818128+0000","last_fullsized":"2024-04-26T11:28:17.818128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:21.398823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837202+0000","last_change":"2024-04-26T11:26:29.019677+0000","last_active":"2024-04-26T11:28:17.837202+0000","last_peered":"2024-04-26T11:28:17.837202+0000","last_clean":"2024-04-26T11:28:17.837202+0000","last_became_active":"2024-04-26T11:26:29.019315+0000","last_became_peered":"2024-04-26T11:26:29.019315+0000","last_unstale":"2024-04-26T11:28:17.837202+0000","last_undegraded":"2024-04-26T11:28:17.837202+0000","last_fullsized":"2024-04-26T11:28:17.837202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:20.239140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242662+0000","last_change":"2024-04-26T11:26:24.964887+0000","last_active":"2024-04-26T11:27:19.242662+0000","last_peered":"2024-04-26T11:27:19.242662+0000","last_clean":"2024-04-26T11:27:19.242662+0000","last_became_active":"2024-04-26T11:26:24.964380+0000","last_became_peered":"2024-04-26T11:26:24.964380+0000","last_unstale":"2024-04-26T11:27:19.242662+0000","last_undegraded":"2024-04-26T11:27:19.242662+0000","last_fullsized":"2024-04-26T11:27:19.242662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:59:41.347542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.817293+0000","last_change":"2024-04-26T11:27:19.244889+0000","last_active":"2024-04-26T11:28:17.817293+0000","last_peered":"2024-04-26T11:28:17.817293+0000","last_clean":"2024-04-26T11:28:17.817293+0000","last_became_active":"2024-04-26T11:27:19.244428+0000","last_became_peered":"2024-04-26T11:27:19.244428+0000","last_unstale":"2024-04-26T11:28:17.817293+0000","last_undegraded":"2024-04-26T11:28:17.817293+0000","last_fullsized":"2024-04-26T11:28:17.817293+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:09:39.366242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246899+0000","last_change":"2024-04-26T11:26:25.082741+0000","last_active":"2024-04-26T11:27:19.246899+0000","last_peered":"2024-04-26T11:27:19.246899+0000","last_clean":"2024-04-26T11:27:19.246899+0000","last_became_active":"2024-04-26T11:26:25.082361+0000","last_became_peered":"2024-04-26T11:26:25.082361+0000","last_unstale":"2024-04-26T11:27:19.246899+0000","last_undegraded":"2024-04-26T11:27:19.246899+0000","last_fullsized":"2024-04-26T11:27:19.246899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:23:17.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.833585+0000","last_change":"2024-04-26T11:26:29.014817+0000","last_active":"2024-04-26T11:28:17.833585+0000","last_peered":"2024-04-26T11:28:17.833585+0000","last_clean":"2024-04-26T11:28:17.833585+0000","last_became_active":"2024-04-26T11:26:29.014419+0000","last_became_peered":"2024-04-26T11:26:29.014419+0000","last_unstale":"2024-04-26T11:28:17.833585+0000","last_undegraded":"2024-04-26T11:28:17.833585+0000","last_fullsized":"2024-04-26T11:28:17.833585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:33:28.355075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245880+0000","last_change":"2024-04-26T11:26:24.960224+0000","last_active":"2024-04-26T11:27:19.245880+0000","last_peered":"2024-04-26T11:27:19.245880+0000","last_clean":"2024-04-26T11:27:19.245880+0000","last_became_active":"2024-04-26T11:26:24.959848+0000","last_became_peered":"2024-04-26T11:26:24.959848+0000","last_unstale":"2024-04-26T11:27:19.245880+0000","last_undegraded":"2024-04-26T11:27:19.245880+0000","last_fullsized":"2024-04-26T11:27:19.245880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:49:10.932187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.822732+0000","last_change":"2024-04-26T11:26:29.001946+0000","last_active":"2024-04-26T11:28:17.822732+0000","last_peered":"2024-04-26T11:28:17.822732+0000","last_clean":"2024-04-26T11:28:17.822732+0000","last_became_active":"2024-04-26T11:26:29.001579+0000","last_became_peered":"2024-04-26T11:26:29.001579+0000","last_unstale":"2024-04-26T11:28:17.822732+0000","last_undegraded":"2024-04-26T11:28:17.822732+0000","last_fullsized":"2024-04-26T11:28:17.822732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:14:43.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245485+0000","last_change":"2024-04-26T11:26:24.956148+0000","last_active":"2024-04-26T11:27:19.245485+0000","last_peered":"2024-04-26T11:27:19.245485+0000","last_clean":"2024-04-26T11:27:19.245485+0000","last_became_active":"2024-04-26T11:26:24.955736+0000","last_became_peered":"2024-04-26T11:26:24.955736+0000","last_unstale":"2024-04-26T11:27:19.245485+0000","last_undegraded":"2024-04-26T11:27:19.245485+0000","last_fullsized":"2024-04-26T11:27:19.245485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:43.705470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.821808+0000","last_change":"2024-04-26T11:27:19.415134+0000","last_active":"2024-04-26T11:28:17.821808+0000","last_peered":"2024-04-26T11:28:17.821808+0000","last_clean":"2024-04-26T11:28:17.821808+0000","last_became_active":"2024-04-26T11:27:19.414703+0000","last_became_peered":"2024-04-26T11:27:19.414703+0000","last_unstale":"2024-04-26T11:28:17.821808+0000","last_undegraded":"2024-04-26T11:28:17.821808+0000","last_fullsized":"2024-04-26T11:28:17.821808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:41:11.268296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":148,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:46.304255+0000","last_change":"2024-04-26T11:26:24.958350+0000","last_active":"2024-04-26T11:28:46.304255+0000","last_peered":"2024-04-26T11:28:46.304255+0000","last_clean":"2024-04-26T11:28:46.304255+0000","last_became_active":"2024-04-26T11:26:24.957382+0000","last_became_peered":"2024-04-26T11:26:24.957382+0000","last_unstale":"2024-04-26T11:28:46.304255+0000","last_undegraded":"2024-04-26T11:28:46.304255+0000","last_fullsized":"2024-04-26T11:28:46.304255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:03.854227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.816625+0000","last_change":"2024-04-26T11:26:28.997568+0000","last_active":"2024-04-26T11:28:17.816625+0000","last_peered":"2024-04-26T11:28:17.816625+0000","last_clean":"2024-04-26T11:28:17.816625+0000","last_became_active":"2024-04-26T11:26:28.997161+0000","last_became_peered":"2024-04-26T11:26:28.997161+0000","last_unstale":"2024-04-26T11:28:17.816625+0000","last_undegraded":"2024-04-26T11:28:17.816625+0000","last_fullsized":"2024-04-26T11:28:17.816625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:04.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.243026+0000","last_change":"2024-04-26T11:26:24.958093+0000","last_active":"2024-04-26T11:27:19.243026+0000","last_peered":"2024-04-26T11:27:19.243026+0000","last_clean":"2024-04-26T11:27:19.243026+0000","last_became_active":"2024-04-26T11:26:24.957730+0000","last_became_peered":"2024-04-26T11:26:24.957730+0000","last_unstale":"2024-04-26T11:27:19.243026+0000","last_undegraded":"2024-04-26T11:27:19.243026+0000","last_fullsized":"2024-04-26T11:27:19.243026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:09:29.827063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.843097+0000","last_change":"2024-04-26T11:27:19.248406+0000","last_active":"2024-04-26T11:28:17.843097+0000","last_peered":"2024-04-26T11:28:17.843097+0000","last_clean":"2024-04-26T11:28:17.843097+0000","last_became_active":"2024-04-26T11:27:19.247980+0000","last_became_peered":"2024-04-26T11:27:19.247980+0000","last_unstale":"2024-04-26T11:28:17.843097+0000","last_undegraded":"2024-04-26T11:28:17.843097+0000","last_fullsized":"2024-04-26T11:28:17.843097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:51:34.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.411996+0000","last_change":"2024-04-26T11:26:24.963463+0000","last_active":"2024-04-26T11:27:19.411996+0000","last_peered":"2024-04-26T11:27:19.411996+0000","last_clean":"2024-04-26T11:27:19.411996+0000","last_became_active":"2024-04-26T11:26:24.963030+0000","last_became_peered":"2024-04-26T11:26:24.963030+0000","last_unstale":"2024-04-26T11:27:19.411996+0000","last_undegraded":"2024-04-26T11:27:19.411996+0000","last_fullsized":"2024-04-26T11:27:19.411996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:04:17.844500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.841817+0000","last_change":"2024-04-26T11:26:29.013899+0000","last_active":"2024-04-26T11:28:17.841817+0000","last_peered":"2024-04-26T11:28:17.841817+0000","last_clean":"2024-04-26T11:28:17.841817+0000","last_became_active":"2024-04-26T11:26:29.012674+0000","last_became_peered":"2024-04-26T11:26:29.012674+0000","last_unstale":"2024-04-26T11:28:17.841817+0000","last_undegraded":"2024-04-26T11:28:17.841817+0000","last_fullsized":"2024-04-26T11:28:17.841817+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:33:46.259841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240050+0000","last_change":"2024-04-26T11:26:24.957803+0000","last_active":"2024-04-26T11:27:19.240050+0000","last_peered":"2024-04-26T11:27:19.240050+0000","last_clean":"2024-04-26T11:27:19.240050+0000","last_became_active":"2024-04-26T11:26:24.957496+0000","last_became_peered":"2024-04-26T11:26:24.957496+0000","last_unstale":"2024-04-26T11:27:19.240050+0000","last_undegraded":"2024-04-26T11:27:19.240050+0000","last_fullsized":"2024-04-26T11:27:19.240050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:15.497106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.843048+0000","last_change":"2024-04-26T11:26:29.021648+0000","last_active":"2024-04-26T11:28:17.843048+0000","last_peered":"2024-04-26T11:28:17.843048+0000","last_clean":"2024-04-26T11:28:17.843048+0000","last_became_active":"2024-04-26T11:26:29.020689+0000","last_became_peered":"2024-04-26T11:26:29.020689+0000","last_unstale":"2024-04-26T11:28:17.843048+0000","last_undegraded":"2024-04-26T11:28:17.843048+0000","last_fullsized":"2024-04-26T11:28:17.843048+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:15:42.321851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":141,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:43.299169+0000","last_change":"2024-04-26T11:26:24.966184+0000","last_active":"2024-04-26T11:28:43.299169+0000","last_peered":"2024-04-26T11:28:43.299169+0000","last_clean":"2024-04-26T11:28:43.299169+0000","last_became_active":"2024-04-26T11:26:24.965850+0000","last_became_peered":"2024-04-26T11:26:24.965850+0000","last_unstale":"2024-04-26T11:28:43.299169+0000","last_undegraded":"2024-04-26T11:28:43.299169+0000","last_fullsized":"2024-04-26T11:28:43.299169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:49.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.822515+0000","last_change":"2024-04-26T11:26:29.003570+0000","last_active":"2024-04-26T11:28:17.822515+0000","last_peered":"2024-04-26T11:28:17.822515+0000","last_clean":"2024-04-26T11:28:17.822515+0000","last_became_active":"2024-04-26T11:26:29.003039+0000","last_became_peered":"2024-04-26T11:26:29.003039+0000","last_unstale":"2024-04-26T11:28:17.822515+0000","last_undegraded":"2024-04-26T11:28:17.822515+0000","last_fullsized":"2024-04-26T11:28:17.822515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:29:16.166432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277109+0000","last_change":"2024-04-26T11:26:24.954968+0000","last_active":"2024-04-26T11:27:19.277109+0000","last_peered":"2024-04-26T11:27:19.277109+0000","last_clean":"2024-04-26T11:27:19.277109+0000","last_became_active":"2024-04-26T11:26:24.954595+0000","last_became_peered":"2024-04-26T11:26:24.954595+0000","last_unstale":"2024-04-26T11:27:19.277109+0000","last_undegraded":"2024-04-26T11:27:19.277109+0000","last_fullsized":"2024-04-26T11:27:19.277109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:52:21.988942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.832433+0000","last_change":"2024-04-26T11:26:29.015601+0000","last_active":"2024-04-26T11:28:17.832433+0000","last_peered":"2024-04-26T11:28:17.832433+0000","last_clean":"2024-04-26T11:28:17.832433+0000","last_became_active":"2024-04-26T11:26:29.015199+0000","last_became_peered":"2024-04-26T11:26:29.015199+0000","last_unstale":"2024-04-26T11:28:17.832433+0000","last_undegraded":"2024-04-26T11:28:17.832433+0000","last_fullsized":"2024-04-26T11:28:17.832433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:15:04.573214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240544+0000","last_change":"2024-04-26T11:26:25.085542+0000","last_active":"2024-04-26T11:27:19.240544+0000","last_peered":"2024-04-26T11:27:19.240544+0000","last_clean":"2024-04-26T11:27:19.240544+0000","last_became_active":"2024-04-26T11:26:25.085230+0000","last_became_peered":"2024-04-26T11:26:25.085230+0000","last_unstale":"2024-04-26T11:27:19.240544+0000","last_undegraded":"2024-04-26T11:27:19.240544+0000","last_fullsized":"2024-04-26T11:27:19.240544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:51.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.827384+0000","last_change":"2024-04-26T11:26:29.003954+0000","last_active":"2024-04-26T11:28:17.827384+0000","last_peered":"2024-04-26T11:28:17.827384+0000","last_clean":"2024-04-26T11:28:17.827384+0000","last_became_active":"2024-04-26T11:26:29.003576+0000","last_became_peered":"2024-04-26T11:26:29.003576+0000","last_unstale":"2024-04-26T11:28:17.827384+0000","last_undegraded":"2024-04-26T11:28:17.827384+0000","last_fullsized":"2024-04-26T11:28:17.827384+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:24.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:44.301421+0000","last_change":"2024-04-26T11:26:24.957823+0000","last_active":"2024-04-26T11:28:44.301421+0000","last_peered":"2024-04-26T11:28:44.301421+0000","last_clean":"2024-04-26T11:28:44.301421+0000","last_became_active":"2024-04-26T11:26:24.957434+0000","last_became_peered":"2024-04-26T11:26:24.957434+0000","last_unstale":"2024-04-26T11:28:44.301421+0000","last_undegraded":"2024-04-26T11:28:44.301421+0000","last_fullsized":"2024-04-26T11:28:44.301421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:38:43.247282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65899,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":234,"num_read_kb":181,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.803191+0000","last_change":"2024-04-26T11:26:28.994764+0000","last_active":"2024-04-26T11:28:17.803191+0000","last_peered":"2024-04-26T11:28:17.803191+0000","last_clean":"2024-04-26T11:28:17.803191+0000","last_became_active":"2024-04-26T11:26:28.994421+0000","last_became_peered":"2024-04-26T11:26:28.994421+0000","last_unstale":"2024-04-26T11:28:17.803191+0000","last_undegraded":"2024-04-26T11:28:17.803191+0000","last_fullsized":"2024-04-26T11:28:17.803191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:06:06.499434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242341+0000","last_change":"2024-04-26T11:26:24.968036+0000","last_active":"2024-04-26T11:27:19.242341+0000","last_peered":"2024-04-26T11:27:19.242341+0000","last_clean":"2024-04-26T11:27:19.242341+0000","last_became_active":"2024-04-26T11:26:24.967684+0000","last_became_peered":"2024-04-26T11:26:24.967684+0000","last_unstale":"2024-04-26T11:27:19.242341+0000","last_undegraded":"2024-04-26T11:27:19.242341+0000","last_fullsized":"2024-04-26T11:27:19.242341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:00:27.467411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.842567+0000","last_change":"2024-04-26T11:26:29.020642+0000","last_active":"2024-04-26T11:28:17.842567+0000","last_peered":"2024-04-26T11:28:17.842567+0000","last_clean":"2024-04-26T11:28:17.842567+0000","last_became_active":"2024-04-26T11:26:29.020278+0000","last_became_peered":"2024-04-26T11:26:29.020278+0000","last_unstale":"2024-04-26T11:28:17.842567+0000","last_undegraded":"2024-04-26T11:28:17.842567+0000","last_fullsized":"2024-04-26T11:28:17.842567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:48.932064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240545+0000","last_change":"2024-04-26T11:26:24.970111+0000","last_active":"2024-04-26T11:27:19.240545+0000","last_peered":"2024-04-26T11:27:19.240545+0000","last_clean":"2024-04-26T11:27:19.240545+0000","last_became_active":"2024-04-26T11:26:24.969629+0000","last_became_peered":"2024-04-26T11:26:24.969629+0000","last_unstale":"2024-04-26T11:27:19.240545+0000","last_undegraded":"2024-04-26T11:27:19.240545+0000","last_fullsized":"2024-04-26T11:27:19.240545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:52:07.859419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.823326+0000","last_change":"2024-04-26T11:26:29.002986+0000","last_active":"2024-04-26T11:28:17.823326+0000","last_peered":"2024-04-26T11:28:17.823326+0000","last_clean":"2024-04-26T11:28:17.823326+0000","last_became_active":"2024-04-26T11:26:29.002610+0000","last_became_peered":"2024-04-26T11:26:29.002610+0000","last_unstale":"2024-04-26T11:28:17.823326+0000","last_undegraded":"2024-04-26T11:28:17.823326+0000","last_fullsized":"2024-04-26T11:28:17.823326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:04.877356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:42.381834+0000","last_change":"2024-04-26T11:26:24.957825+0000","last_active":"2024-04-26T11:28:42.381834+0000","last_peered":"2024-04-26T11:28:42.381834+0000","last_clean":"2024-04-26T11:28:42.381834+0000","last_became_active":"2024-04-26T11:26:24.957464+0000","last_became_peered":"2024-04-26T11:26:24.957464+0000","last_unstale":"2024-04-26T11:28:42.381834+0000","last_undegraded":"2024-04-26T11:28:42.381834+0000","last_fullsized":"2024-04-26T11:28:42.381834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:06:49.350268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.811198+0000","last_change":"2024-04-26T11:26:28.994223+0000","last_active":"2024-04-26T11:28:17.811198+0000","last_peered":"2024-04-26T11:28:17.811198+0000","last_clean":"2024-04-26T11:28:17.811198+0000","last_became_active":"2024-04-26T11:26:28.993388+0000","last_became_peered":"2024-04-26T11:26:28.993388+0000","last_unstale":"2024-04-26T11:28:17.811198+0000","last_undegraded":"2024-04-26T11:28:17.811198+0000","last_fullsized":"2024-04-26T11:28:17.811198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:07:24.208028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245056+0000","last_change":"2024-04-26T11:26:25.078473+0000","last_active":"2024-04-26T11:27:19.245056+0000","last_peered":"2024-04-26T11:27:19.245056+0000","last_clean":"2024-04-26T11:27:19.245056+0000","last_became_active":"2024-04-26T11:26:25.078138+0000","last_became_peered":"2024-04-26T11:26:25.078138+0000","last_unstale":"2024-04-26T11:27:19.245056+0000","last_undegraded":"2024-04-26T11:27:19.245056+0000","last_fullsized":"2024-04-26T11:27:19.245056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:19.692345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.811891+0000","last_change":"2024-04-26T11:26:28.998687+0000","last_active":"2024-04-26T11:28:17.811891+0000","last_peered":"2024-04-26T11:28:17.811891+0000","last_clean":"2024-04-26T11:28:17.811891+0000","last_became_active":"2024-04-26T11:26:28.998358+0000","last_became_peered":"2024-04-26T11:26:28.998358+0000","last_unstale":"2024-04-26T11:28:17.811891+0000","last_undegraded":"2024-04-26T11:28:17.811891+0000","last_fullsized":"2024-04-26T11:28:17.811891+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:40:51.358188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246156+0000","last_change":"2024-04-26T11:26:24.956213+0000","last_active":"2024-04-26T11:27:19.246156+0000","last_peered":"2024-04-26T11:27:19.246156+0000","last_clean":"2024-04-26T11:27:19.246156+0000","last_became_active":"2024-04-26T11:26:24.955773+0000","last_became_peered":"2024-04-26T11:26:24.955773+0000","last_unstale":"2024-04-26T11:27:19.246156+0000","last_undegraded":"2024-04-26T11:27:19.246156+0000","last_fullsized":"2024-04-26T11:27:19.246156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:38.253367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.814609+0000","last_change":"2024-04-26T11:26:29.001389+0000","last_active":"2024-04-26T11:28:17.814609+0000","last_peered":"2024-04-26T11:28:17.814609+0000","last_clean":"2024-04-26T11:28:17.814609+0000","last_became_active":"2024-04-26T11:26:29.001023+0000","last_became_peered":"2024-04-26T11:26:29.001023+0000","last_unstale":"2024-04-26T11:28:17.814609+0000","last_undegraded":"2024-04-26T11:28:17.814609+0000","last_fullsized":"2024-04-26T11:28:17.814609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:57:23.606518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.895387+0000","last_change":"2024-04-26T11:27:19.893959+0000","last_active":"2024-04-26T11:27:19.895387+0000","last_peered":"2024-04-26T11:27:19.895387+0000","last_clean":"2024-04-26T11:27:19.895387+0000","last_became_active":"2024-04-26T11:27:19.892819+0000","last_became_peered":"2024-04-26T11:27:19.892819+0000","last_unstale":"2024-04-26T11:27:19.895387+0000","last_undegraded":"2024-04-26T11:27:19.895387+0000","last_fullsized":"2024-04-26T11:27:19.895387+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:53:39.829381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.836883+0000","last_change":"2024-04-26T11:26:29.019906+0000","last_active":"2024-04-26T11:28:17.836883+0000","last_peered":"2024-04-26T11:28:17.836883+0000","last_clean":"2024-04-26T11:28:17.836883+0000","last_became_active":"2024-04-26T11:26:29.019545+0000","last_became_peered":"2024-04-26T11:26:29.019545+0000","last_unstale":"2024-04-26T11:28:17.836883+0000","last_undegraded":"2024-04-26T11:28:17.836883+0000","last_fullsized":"2024-04-26T11:28:17.836883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:48.617269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277025+0000","last_change":"2024-04-26T11:26:24.956400+0000","last_active":"2024-04-26T11:27:19.277025+0000","last_peered":"2024-04-26T11:27:19.277025+0000","last_clean":"2024-04-26T11:27:19.277025+0000","last_became_active":"2024-04-26T11:26:24.956030+0000","last_became_peered":"2024-04-26T11:26:24.956030+0000","last_unstale":"2024-04-26T11:27:19.277025+0000","last_undegraded":"2024-04-26T11:27:19.277025+0000","last_fullsized":"2024-04-26T11:27:19.277025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:47.841006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.824606+0000","last_change":"2024-04-26T11:26:29.013087+0000","last_active":"2024-04-26T11:28:17.824606+0000","last_peered":"2024-04-26T11:28:17.824606+0000","last_clean":"2024-04-26T11:28:17.824606+0000","last_became_active":"2024-04-26T11:26:29.012330+0000","last_became_peered":"2024-04-26T11:26:29.012330+0000","last_unstale":"2024-04-26T11:28:17.824606+0000","last_undegraded":"2024-04-26T11:28:17.824606+0000","last_fullsized":"2024-04-26T11:28:17.824606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:12:50.752329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.812867+0000","last_change":"2024-04-26T11:27:19.245410+0000","last_active":"2024-04-26T11:28:17.812867+0000","last_peered":"2024-04-26T11:28:17.812867+0000","last_clean":"2024-04-26T11:28:17.812867+0000","last_became_active":"2024-04-26T11:27:19.244578+0000","last_became_peered":"2024-04-26T11:27:19.244578+0000","last_unstale":"2024-04-26T11:28:17.812867+0000","last_undegraded":"2024-04-26T11:28:17.812867+0000","last_fullsized":"2024-04-26T11:28:17.812867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:07:50.576496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:46.390943+0000","last_change":"2024-04-26T11:26:25.082517+0000","last_active":"2024-04-26T11:28:46.390943+0000","last_peered":"2024-04-26T11:28:46.390943+0000","last_clean":"2024-04-26T11:28:46.390943+0000","last_became_active":"2024-04-26T11:26:25.081724+0000","last_became_peered":"2024-04-26T11:26:25.081724+0000","last_unstale":"2024-04-26T11:28:46.390943+0000","last_undegraded":"2024-04-26T11:28:46.390943+0000","last_fullsized":"2024-04-26T11:28:46.390943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:59:40.006615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837462+0000","last_change":"2024-04-26T11:26:29.021425+0000","last_active":"2024-04-26T11:28:17.837462+0000","last_peered":"2024-04-26T11:28:17.837462+0000","last_clean":"2024-04-26T11:28:17.837462+0000","last_became_active":"2024-04-26T11:26:29.020881+0000","last_became_peered":"2024-04-26T11:26:29.020881+0000","last_unstale":"2024-04-26T11:28:17.837462+0000","last_undegraded":"2024-04-26T11:28:17.837462+0000","last_fullsized":"2024-04-26T11:28:17.837462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:18.350323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240589+0000","last_change":"2024-04-26T11:26:24.963614+0000","last_active":"2024-04-26T11:27:19.240589+0000","last_peered":"2024-04-26T11:27:19.240589+0000","last_clean":"2024-04-26T11:27:19.240589+0000","last_became_active":"2024-04-26T11:26:24.963312+0000","last_became_peered":"2024-04-26T11:26:24.963312+0000","last_unstale":"2024-04-26T11:27:19.240589+0000","last_undegraded":"2024-04-26T11:27:19.240589+0000","last_fullsized":"2024-04-26T11:27:19.240589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:55.886637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":360,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276824+0000","last_change":"2024-04-26T11:24:54.292436+0000","last_active":"2024-04-26T11:27:19.276824+0000","last_peered":"2024-04-26T11:27:19.276824+0000","last_clean":"2024-04-26T11:27:19.276824+0000","last_became_active":"2024-04-26T11:24:54.291921+0000","last_became_peered":"2024-04-26T11:24:54.291921+0000","last_unstale":"2024-04-26T11:27:19.276824+0000","last_undegraded":"2024-04-26T11:27:19.276824+0000","last_fullsized":"2024-04-26T11:27:19.276824+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_clean_scrub_stamp":"2024-04-26T11:21:00.617277+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:11:01.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.820865+0000","last_change":"2024-04-26T11:26:29.000905+0000","last_active":"2024-04-26T11:28:17.820865+0000","last_peered":"2024-04-26T11:28:17.820865+0000","last_clean":"2024-04-26T11:28:17.820865+0000","last_became_active":"2024-04-26T11:26:29.000517+0000","last_became_peered":"2024-04-26T11:26:29.000517+0000","last_unstale":"2024-04-26T11:28:17.820865+0000","last_undegraded":"2024-04-26T11:28:17.820865+0000","last_fullsized":"2024-04-26T11:28:17.820865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:38:55.394316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277140+0000","last_change":"2024-04-26T11:26:24.957489+0000","last_active":"2024-04-26T11:27:19.277140+0000","last_peered":"2024-04-26T11:27:19.277140+0000","last_clean":"2024-04-26T11:27:19.277140+0000","last_became_active":"2024-04-26T11:26:24.957151+0000","last_became_peered":"2024-04-26T11:26:24.957151+0000","last_unstale":"2024-04-26T11:27:19.277140+0000","last_undegraded":"2024-04-26T11:27:19.277140+0000","last_fullsized":"2024-04-26T11:27:19.277140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:42:25.343105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245739+0000","last_change":"2024-04-26T11:26:25.086767+0000","last_active":"2024-04-26T11:27:19.245739+0000","last_peered":"2024-04-26T11:27:19.245739+0000","last_clean":"2024-04-26T11:27:19.245739+0000","last_became_active":"2024-04-26T11:26:25.086440+0000","last_became_peered":"2024-04-26T11:26:25.086440+0000","last_unstale":"2024-04-26T11:27:19.245739+0000","last_undegraded":"2024-04-26T11:27:19.245739+0000","last_fullsized":"2024-04-26T11:27:19.245739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:02.294644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825487+0000","last_change":"2024-04-26T11:26:29.003683+0000","last_active":"2024-04-26T11:28:17.825487+0000","last_peered":"2024-04-26T11:28:17.825487+0000","last_clean":"2024-04-26T11:28:17.825487+0000","last_became_active":"2024-04-26T11:26:29.003307+0000","last_became_peered":"2024-04-26T11:26:29.003307+0000","last_unstale":"2024-04-26T11:28:17.825487+0000","last_undegraded":"2024-04-26T11:28:17.825487+0000","last_fullsized":"2024-04-26T11:28:17.825487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:41:00.124540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:18.048485+0000","last_change":"2024-04-26T11:27:19.849965+0000","last_active":"2024-04-26T11:28:18.048485+0000","last_peered":"2024-04-26T11:28:18.048485+0000","last_clean":"2024-04-26T11:28:18.048485+0000","last_became_active":"2024-04-26T11:27:19.840545+0000","last_became_peered":"2024-04-26T11:27:19.840545+0000","last_unstale":"2024-04-26T11:28:18.048485+0000","last_undegraded":"2024-04-26T11:28:18.048485+0000","last_fullsized":"2024-04-26T11:28:18.048485+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:08.003704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.839888+0000","last_change":"2024-04-26T11:26:29.016783+0000","last_active":"2024-04-26T11:28:17.839888+0000","last_peered":"2024-04-26T11:28:17.839888+0000","last_clean":"2024-04-26T11:28:17.839888+0000","last_became_active":"2024-04-26T11:26:29.013998+0000","last_became_peered":"2024-04-26T11:26:29.013998+0000","last_unstale":"2024-04-26T11:28:17.839888+0000","last_undegraded":"2024-04-26T11:28:17.839888+0000","last_fullsized":"2024-04-26T11:28:17.839888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:24:44.453094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240650+0000","last_change":"2024-04-26T11:26:25.902330+0000","last_active":"2024-04-26T11:27:19.240650+0000","last_peered":"2024-04-26T11:27:19.240650+0000","last_clean":"2024-04-26T11:27:19.240650+0000","last_became_active":"2024-04-26T11:26:25.901948+0000","last_became_peered":"2024-04-26T11:26:25.901948+0000","last_unstale":"2024-04-26T11:27:19.240650+0000","last_undegraded":"2024-04-26T11:27:19.240650+0000","last_fullsized":"2024-04-26T11:27:19.240650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:00:23.541393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.844428+0000","last_change":"2024-04-26T11:26:29.025692+0000","last_active":"2024-04-26T11:28:17.844428+0000","last_peered":"2024-04-26T11:28:17.844428+0000","last_clean":"2024-04-26T11:28:17.844428+0000","last_became_active":"2024-04-26T11:26:29.025112+0000","last_became_peered":"2024-04-26T11:26:29.025112+0000","last_unstale":"2024-04-26T11:28:17.844428+0000","last_undegraded":"2024-04-26T11:28:17.844428+0000","last_fullsized":"2024-04-26T11:28:17.844428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:39:22.637865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244481+0000","last_change":"2024-04-26T11:26:25.080270+0000","last_active":"2024-04-26T11:27:19.244481+0000","last_peered":"2024-04-26T11:27:19.244481+0000","last_clean":"2024-04-26T11:27:19.244481+0000","last_became_active":"2024-04-26T11:26:25.079895+0000","last_became_peered":"2024-04-26T11:26:25.079895+0000","last_unstale":"2024-04-26T11:27:19.244481+0000","last_undegraded":"2024-04-26T11:27:19.244481+0000","last_fullsized":"2024-04-26T11:27:19.244481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:47:08.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.790599+0000","last_change":"2024-04-26T11:26:28.996540+0000","last_active":"2024-04-26T11:28:17.790599+0000","last_peered":"2024-04-26T11:28:17.790599+0000","last_clean":"2024-04-26T11:28:17.790599+0000","last_became_active":"2024-04-26T11:26:28.994724+0000","last_became_peered":"2024-04-26T11:26:28.994724+0000","last_unstale":"2024-04-26T11:28:17.790599+0000","last_undegraded":"2024-04-26T11:28:17.790599+0000","last_fullsized":"2024-04-26T11:28:17.790599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:13.382332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245883+0000","last_change":"2024-04-26T11:26:24.967935+0000","last_active":"2024-04-26T11:27:19.245883+0000","last_peered":"2024-04-26T11:27:19.245883+0000","last_clean":"2024-04-26T11:27:19.245883+0000","last_became_active":"2024-04-26T11:26:24.966465+0000","last_became_peered":"2024-04-26T11:26:24.966465+0000","last_unstale":"2024-04-26T11:27:19.245883+0000","last_undegraded":"2024-04-26T11:27:19.245883+0000","last_fullsized":"2024-04-26T11:27:19.245883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:27:38.817316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.833429+0000","last_change":"2024-04-26T11:26:29.024243+0000","last_active":"2024-04-26T11:28:17.833429+0000","last_peered":"2024-04-26T11:28:17.833429+0000","last_clean":"2024-04-26T11:28:17.833429+0000","last_became_active":"2024-04-26T11:26:29.023900+0000","last_became_peered":"2024-04-26T11:26:29.023900+0000","last_unstale":"2024-04-26T11:28:17.833429+0000","last_undegraded":"2024-04-26T11:28:17.833429+0000","last_fullsized":"2024-04-26T11:28:17.833429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:28:23.446568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242131+0000","last_change":"2024-04-26T11:26:24.969953+0000","last_active":"2024-04-26T11:27:19.242131+0000","last_peered":"2024-04-26T11:27:19.242131+0000","last_clean":"2024-04-26T11:27:19.242131+0000","last_became_active":"2024-04-26T11:26:24.969371+0000","last_became_peered":"2024-04-26T11:26:24.969371+0000","last_unstale":"2024-04-26T11:27:19.242131+0000","last_undegraded":"2024-04-26T11:27:19.242131+0000","last_fullsized":"2024-04-26T11:27:19.242131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:07:24.323833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.841678+0000","last_change":"2024-04-26T11:26:29.029771+0000","last_active":"2024-04-26T11:28:17.841678+0000","last_peered":"2024-04-26T11:28:17.841678+0000","last_clean":"2024-04-26T11:28:17.841678+0000","last_became_active":"2024-04-26T11:26:29.029387+0000","last_became_peered":"2024-04-26T11:26:29.029387+0000","last_unstale":"2024-04-26T11:28:17.841678+0000","last_undegraded":"2024-04-26T11:28:17.841678+0000","last_fullsized":"2024-04-26T11:28:17.841678+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:40.856935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245363+0000","last_change":"2024-04-26T11:26:24.958608+0000","last_active":"2024-04-26T11:27:19.245363+0000","last_peered":"2024-04-26T11:27:19.245363+0000","last_clean":"2024-04-26T11:27:19.245363+0000","last_became_active":"2024-04-26T11:26:24.958188+0000","last_became_peered":"2024-04-26T11:26:24.958188+0000","last_unstale":"2024-04-26T11:27:19.245363+0000","last_undegraded":"2024-04-26T11:27:19.245363+0000","last_fullsized":"2024-04-26T11:27:19.245363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:31:41.442551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825185+0000","last_change":"2024-04-26T11:26:29.000207+0000","last_active":"2024-04-26T11:28:17.825185+0000","last_peered":"2024-04-26T11:28:17.825185+0000","last_clean":"2024-04-26T11:28:17.825185+0000","last_became_active":"2024-04-26T11:26:28.999894+0000","last_became_peered":"2024-04-26T11:26:28.999894+0000","last_unstale":"2024-04-26T11:28:17.825185+0000","last_undegraded":"2024-04-26T11:28:17.825185+0000","last_fullsized":"2024-04-26T11:28:17.825185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:52:10.349511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242688+0000","last_change":"2024-04-26T11:26:25.902036+0000","last_active":"2024-04-26T11:27:19.242688+0000","last_peered":"2024-04-26T11:27:19.242688+0000","last_clean":"2024-04-26T11:27:19.242688+0000","last_became_active":"2024-04-26T11:26:25.901567+0000","last_became_peered":"2024-04-26T11:26:25.901567+0000","last_unstale":"2024-04-26T11:27:19.242688+0000","last_undegraded":"2024-04-26T11:27:19.242688+0000","last_fullsized":"2024-04-26T11:27:19.242688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:58.757780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825941+0000","last_change":"2024-04-26T11:26:28.998915+0000","last_active":"2024-04-26T11:28:17.825941+0000","last_peered":"2024-04-26T11:28:17.825941+0000","last_clean":"2024-04-26T11:28:17.825941+0000","last_became_active":"2024-04-26T11:26:28.998180+0000","last_became_peered":"2024-04-26T11:26:28.998180+0000","last_unstale":"2024-04-26T11:28:17.825941+0000","last_undegraded":"2024-04-26T11:28:17.825941+0000","last_fullsized":"2024-04-26T11:28:17.825941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:19.972697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240438+0000","last_change":"2024-04-26T11:26:24.966029+0000","last_active":"2024-04-26T11:27:19.240438+0000","last_peered":"2024-04-26T11:27:19.240438+0000","last_clean":"2024-04-26T11:27:19.240438+0000","last_became_active":"2024-04-26T11:26:24.965676+0000","last_became_peered":"2024-04-26T11:26:24.965676+0000","last_unstale":"2024-04-26T11:27:19.240438+0000","last_undegraded":"2024-04-26T11:27:19.240438+0000","last_fullsized":"2024-04-26T11:27:19.240438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:18:06.904563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837486+0000","last_change":"2024-04-26T11:26:29.020093+0000","last_active":"2024-04-26T11:28:17.837486+0000","last_peered":"2024-04-26T11:28:17.837486+0000","last_clean":"2024-04-26T11:28:17.837486+0000","last_became_active":"2024-04-26T11:26:29.019690+0000","last_became_peered":"2024-04-26T11:26:29.019690+0000","last_unstale":"2024-04-26T11:28:17.837486+0000","last_undegraded":"2024-04-26T11:28:17.837486+0000","last_fullsized":"2024-04-26T11:28:17.837486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:35:44.501217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.413081+0000","last_change":"2024-04-26T11:26:24.966757+0000","last_active":"2024-04-26T11:27:19.413081+0000","last_peered":"2024-04-26T11:27:19.413081+0000","last_clean":"2024-04-26T11:27:19.413081+0000","last_became_active":"2024-04-26T11:26:24.966298+0000","last_became_peered":"2024-04-26T11:26:24.966298+0000","last_unstale":"2024-04-26T11:27:19.413081+0000","last_undegraded":"2024-04-26T11:27:19.413081+0000","last_fullsized":"2024-04-26T11:27:19.413081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:17:40.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.816124+0000","last_change":"2024-04-26T11:26:28.997774+0000","last_active":"2024-04-26T11:28:17.816124+0000","last_peered":"2024-04-26T11:28:17.816124+0000","last_clean":"2024-04-26T11:28:17.816124+0000","last_became_active":"2024-04-26T11:26:28.997455+0000","last_became_peered":"2024-04-26T11:26:28.997455+0000","last_unstale":"2024-04-26T11:28:17.816124+0000","last_undegraded":"2024-04-26T11:28:17.816124+0000","last_fullsized":"2024-04-26T11:28:17.816124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:56.097022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.815649+0000","last_change":"2024-04-26T11:26:28.999714+0000","last_active":"2024-04-26T11:28:17.815649+0000","last_peered":"2024-04-26T11:28:17.815649+0000","last_clean":"2024-04-26T11:28:17.815649+0000","last_became_active":"2024-04-26T11:26:28.999377+0000","last_became_peered":"2024-04-26T11:26:28.999377+0000","last_unstale":"2024-04-26T11:28:17.815649+0000","last_undegraded":"2024-04-26T11:28:17.815649+0000","last_fullsized":"2024-04-26T11:28:17.815649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:39:59.141556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.806463+0000","last_change":"2024-04-26T11:27:19.247154+0000","last_active":"2024-04-26T11:28:17.806463+0000","last_peered":"2024-04-26T11:28:17.806463+0000","last_clean":"2024-04-26T11:28:17.806463+0000","last_became_active":"2024-04-26T11:27:19.246590+0000","last_became_peered":"2024-04-26T11:27:19.246590+0000","last_unstale":"2024-04-26T11:28:17.806463+0000","last_undegraded":"2024-04-26T11:28:17.806463+0000","last_fullsized":"2024-04-26T11:28:17.806463+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:12:58.401699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837157+0000","last_change":"2024-04-26T11:26:29.019983+0000","last_active":"2024-04-26T11:28:17.837157+0000","last_peered":"2024-04-26T11:28:17.837157+0000","last_clean":"2024-04-26T11:28:17.837157+0000","last_became_active":"2024-04-26T11:26:29.019536+0000","last_became_peered":"2024-04-26T11:26:29.019536+0000","last_unstale":"2024-04-26T11:28:17.837157+0000","last_undegraded":"2024-04-26T11:28:17.837157+0000","last_fullsized":"2024-04-26T11:28:17.837157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:15:06.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.834360+0000","last_change":"2024-04-26T11:26:29.027089+0000","last_active":"2024-04-26T11:28:17.834360+0000","last_peered":"2024-04-26T11:28:17.834360+0000","last_clean":"2024-04-26T11:28:17.834360+0000","last_became_active":"2024-04-26T11:26:29.026030+0000","last_became_peered":"2024-04-26T11:26:29.026030+0000","last_unstale":"2024-04-26T11:28:17.834360+0000","last_undegraded":"2024-04-26T11:28:17.834360+0000","last_fullsized":"2024-04-26T11:28:17.834360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:24:06.105600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825719+0000","last_change":"2024-04-26T11:26:29.018472+0000","last_active":"2024-04-26T11:28:17.825719+0000","last_peered":"2024-04-26T11:28:17.825719+0000","last_clean":"2024-04-26T11:28:17.825719+0000","last_became_active":"2024-04-26T11:26:29.015947+0000","last_became_peered":"2024-04-26T11:26:29.015947+0000","last_unstale":"2024-04-26T11:28:17.825719+0000","last_undegraded":"2024-04-26T11:28:17.825719+0000","last_fullsized":"2024-04-26T11:28:17.825719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:35:48.082709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.814545+0000","last_change":"2024-04-26T11:26:28.997582+0000","last_active":"2024-04-26T11:28:17.814545+0000","last_peered":"2024-04-26T11:28:17.814545+0000","last_clean":"2024-04-26T11:28:17.814545+0000","last_became_active":"2024-04-26T11:26:28.997202+0000","last_became_peered":"2024-04-26T11:26:28.997202+0000","last_unstale":"2024-04-26T11:28:17.814545+0000","last_undegraded":"2024-04-26T11:28:17.814545+0000","last_fullsized":"2024-04-26T11:28:17.814545+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:59.776455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.814923+0000","last_change":"2024-04-26T11:26:28.999428+0000","last_active":"2024-04-26T11:28:17.814923+0000","last_peered":"2024-04-26T11:28:17.814923+0000","last_clean":"2024-04-26T11:28:17.814923+0000","last_became_active":"2024-04-26T11:26:28.999093+0000","last_became_peered":"2024-04-26T11:26:28.999093+0000","last_unstale":"2024-04-26T11:28:17.814923+0000","last_undegraded":"2024-04-26T11:28:17.814923+0000","last_fullsized":"2024-04-26T11:28:17.814923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:07.222231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.832441+0000","last_change":"2024-04-26T11:26:29.019673+0000","last_active":"2024-04-26T11:28:17.832441+0000","last_peered":"2024-04-26T11:28:17.832441+0000","last_clean":"2024-04-26T11:28:17.832441+0000","last_became_active":"2024-04-26T11:26:29.017289+0000","last_became_peered":"2024-04-26T11:26:29.017289+0000","last_unstale":"2024-04-26T11:28:17.832441+0000","last_undegraded":"2024-04-26T11:28:17.832441+0000","last_fullsized":"2024-04-26T11:28:17.832441+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:35.848004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837058+0000","last_change":"2024-04-26T11:26:29.017234+0000","last_active":"2024-04-26T11:28:17.837058+0000","last_peered":"2024-04-26T11:28:17.837058+0000","last_clean":"2024-04-26T11:28:17.837058+0000","last_became_active":"2024-04-26T11:26:29.016863+0000","last_became_peered":"2024-04-26T11:26:29.016863+0000","last_unstale":"2024-04-26T11:28:17.837058+0000","last_undegraded":"2024-04-26T11:28:17.837058+0000","last_fullsized":"2024-04-26T11:28:17.837058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:14:31.136286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.839115+0000","last_change":"2024-04-26T11:26:29.018122+0000","last_active":"2024-04-26T11:28:17.839115+0000","last_peered":"2024-04-26T11:28:17.839115+0000","last_clean":"2024-04-26T11:28:17.839115+0000","last_became_active":"2024-04-26T11:26:29.016269+0000","last_became_peered":"2024-04-26T11:26:29.016269+0000","last_unstale":"2024-04-26T11:28:17.839115+0000","last_undegraded":"2024-04-26T11:28:17.839115+0000","last_fullsized":"2024-04-26T11:28:17.839115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:22.135086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.826827+0000","last_change":"2024-04-26T11:27:19.278596+0000","last_active":"2024-04-26T11:28:17.826827+0000","last_peered":"2024-04-26T11:28:17.826827+0000","last_clean":"2024-04-26T11:28:17.826827+0000","last_became_active":"2024-04-26T11:27:19.278138+0000","last_became_peered":"2024-04-26T11:27:19.278138+0000","last_unstale":"2024-04-26T11:28:17.826827+0000","last_undegraded":"2024-04-26T11:28:17.826827+0000","last_fullsized":"2024-04-26T11:28:17.826827+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:17.038527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.829918+0000","last_change":"2024-04-26T11:26:29.026905+0000","last_active":"2024-04-26T11:28:17.829918+0000","last_peered":"2024-04-26T11:28:17.829918+0000","last_clean":"2024-04-26T11:28:17.829918+0000","last_became_active":"2024-04-26T11:26:29.026539+0000","last_became_peered":"2024-04-26T11:26:29.026539+0000","last_unstale":"2024-04-26T11:28:17.829918+0000","last_undegraded":"2024-04-26T11:28:17.829918+0000","last_fullsized":"2024-04-26T11:28:17.829918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:41:12.169466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.813412+0000","last_change":"2024-04-26T11:26:28.999953+0000","last_active":"2024-04-26T11:28:17.813412+0000","last_peered":"2024-04-26T11:28:17.813412+0000","last_clean":"2024-04-26T11:28:17.813412+0000","last_became_active":"2024-04-26T11:26:28.998344+0000","last_became_peered":"2024-04-26T11:26:28.998344+0000","last_unstale":"2024-04-26T11:28:17.813412+0000","last_undegraded":"2024-04-26T11:28:17.813412+0000","last_fullsized":"2024-04-26T11:28:17.813412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:30:53.269171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281068,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70041,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":707,"num_read_kb":428,"num_write":177,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":210123,"data_compressed":5472,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":479968,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32280,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712872,"statfs":{"total":95995035648,"available":95961980928,"internally_reserved":0,"allocated":1130496,"data_stored":1137880,"data_compressed":43361,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480204,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070526,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480124,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332163,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1072195,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495689,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":479593,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":545285,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480864,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":546635,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479904,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":480562,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1071347,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65899,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67249,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68009,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T11:28:49.907 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T11:28:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:49 smithi096 ceph-mon[32753]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:50.384 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:50.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:50 smithi033 ceph-mon[27845]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:28:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:50 smithi096 ceph-mon[32753]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:28:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:50 smithi194 ceph-mon[33017]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:28:51.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:51 smithi033 ceph-mon[27845]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:51 smithi096 ceph-mon[32753]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:51 smithi194 ceph-mon[33017]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:28:52.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 -- 172.21.15.33:0/1062578571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104580 msgr2=0x7f335c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 --2- 172.21.15.33:0/1062578571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104580 0x7f335c104960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f334c009990 tx=0x7f334c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 -- 172.21.15.33:0/1062578571 shutdown_connections 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 --2- 172.21.15.33:0/1062578571 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f335c10d9b0 0x7f335c10fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 --2- 172.21.15.33:0/1062578571 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104580 0x7f335c104960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 --2- 172.21.15.33:0/1062578571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104ea0 0x7f335c10d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 -- 172.21.15.33:0/1062578571 >> 172.21.15.33:0/1062578571 conn(0x7f335c0fddd0 msgr2=0x7f335c1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 -- 172.21.15.33:0/1062578571 shutdown_connections 2024-04-26T11:28:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.694+0000 7f335be00700 1 -- 172.21.15.33:0/1062578571 wait complete. 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.695+0000 7f335be00700 1 Processor -- start 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335be00700 1 -- start start 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104580 0x7f335c1021e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104ea0 0x7f335c102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f335c10d9b0 0x7f335c102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335be00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1135f0 con 0x7f335c104ea0 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335be00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f335c113470 con 0x7f335c104580 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335be00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f335c113770 con 0x7f335c10d9b0 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335aa00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104580 0x7f335c1021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335aa00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104580 0x7f335c1021e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33414/0 (socket says 172.21.15.33:33414) 2024-04-26T11:28:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335aa00700 1 -- 172.21.15.33:0/2022800263 learned_addr learned my addr 172.21.15.33:0/2022800263 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335a000700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104ea0 0x7f335c102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.696+0000 7f335b400700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f335c10d9b0 0x7f335c102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.697+0000 7f335aa00700 1 -- 172.21.15.33:0/2022800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f335c10d9b0 msgr2=0x7f335c102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.697+0000 7f335aa00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f335c10d9b0 0x7f335c102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.697+0000 7f335aa00700 1 -- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104ea0 msgr2=0x7f335c102720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.697+0000 7f335aa00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104ea0 0x7f335c102720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.697+0000 7f335aa00700 1 -- 172.21.15.33:0/2022800263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c1ac4b0 con 0x7f335c104580 2024-04-26T11:28:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.697+0000 7f335aa00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104580 0x7f335c1021e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f334c000c00 tx=0x7f334c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.697+0000 7f3353400700 1 -- 172.21.15.33:0/2022800263 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c027070 con 0x7f335c104580 2024-04-26T11:28:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.698+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c1ac6e0 con 0x7f335c104580 2024-04-26T11:28:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.698+0000 7f3353400700 1 -- 172.21.15.33:0/2022800263 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f334c022470 con 0x7f335c104580 2024-04-26T11:28:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.698+0000 7f3353400700 1 -- 172.21.15.33:0/2022800263 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c017650 con 0x7f335c104580 2024-04-26T11:28:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.698+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c1acc20 con 0x7f335c104580 2024-04-26T11:28:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.699+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f335c06c270 con 0x7f335c104580 2024-04-26T11:28:52.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.701+0000 7f3353400700 1 -- 172.21.15.33:0/2022800263 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f334c02a070 con 0x7f335c104580 2024-04-26T11:28:52.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.702+0000 7f3353400700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f33400a3c40 0x7f33400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:52.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.702+0000 7f3353400700 1 -- 172.21.15.33:0/2022800263 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f334c0d1170 con 0x7f335c104580 2024-04-26T11:28:52.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.703+0000 7f335a000700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f33400a3c40 0x7f33400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:52.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.703+0000 7f335a000700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f33400a3c40 0x7f33400a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f335c1039a0 tx=0x7f3348009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:52.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.705+0000 7f3353400700 1 -- 172.21.15.33:0/2022800263 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f334c013070 con 0x7f335c104580 2024-04-26T11:28:52.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.934+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f335c103710 con 0x7f33400a3c40 2024-04-26T11:28:52.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.956+0000 7f3353400700 1 -- 172.21.15.33:0/2022800263 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259124 (secure 0 0 0) 0x7f335c103710 con 0x7f33400a3c40 2024-04-26T11:28:52.958 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:28:52.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f33400a3c40 msgr2=0x7f33400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:52.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f33400a3c40 0x7f33400a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f335c1039a0 tx=0x7f3348009260 comp rx=0 tx=0).stop 2024-04-26T11:28:52.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104580 msgr2=0x7f335c1021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:52.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104580 0x7f335c1021e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f334c000c00 tx=0x7f334c0057c0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 shutdown_connections 2024-04-26T11:28:52.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f33400a3c40 0x7f33400a6100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f335c10d9b0 0x7f335c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c104580 0x7f335c1021e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.963+0000 7f335be00700 1 --2- 172.21.15.33:0/2022800263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f335c104ea0 0x7f335c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:52.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.964+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 >> 172.21.15.33:0/2022800263 conn(0x7f335c0fddd0 msgr2=0x7f335c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:52.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.964+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 shutdown_connections 2024-04-26T11:28:52.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:52.964+0000 7f335be00700 1 -- 172.21.15.33:0/2022800263 wait complete. 2024-04-26T11:28:52.968 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-26T11:28:53.877 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:53 smithi033 ceph-mon[27845]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:53.881 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-04-26T11:28:51.864793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660643,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":863,"num_read_kb":557,"num_write":360,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356992,"kb_used_data":12000,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584832,"statfs":{"total":1151940427776,"available":1151574867968,"internally_reserved":0,"allocated":12288000,"data_stored":7734561,"data_compressed":182166,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005244"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825734+0000","last_change":"2024-04-26T11:26:29.018818+0000","last_active":"2024-04-26T11:28:17.825734+0000","last_peered":"2024-04-26T11:28:17.825734+0000","last_clean":"2024-04-26T11:28:17.825734+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:28:17.825734+0000","last_undegraded":"2024-04-26T11:28:17.825734+0000","last_fullsized":"2024-04-26T11:28:17.825734+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:05.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.827907+0000","last_change":"2024-04-26T11:26:29.021338+0000","last_active":"2024-04-26T11:28:17.827907+0000","last_peered":"2024-04-26T11:28:17.827907+0000","last_clean":"2024-04-26T11:28:17.827907+0000","last_became_active":"2024-04-26T11:26:29.021019+0000","last_became_peered":"2024-04-26T11:26:29.021019+0000","last_unstale":"2024-04-26T11:28:17.827907+0000","last_undegraded":"2024-04-26T11:28:17.827907+0000","last_fullsized":"2024-04-26T11:28:17.827907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:52:44.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.832078+0000","last_change":"2024-04-26T11:26:29.014166+0000","last_active":"2024-04-26T11:28:17.832078+0000","last_peered":"2024-04-26T11:28:17.832078+0000","last_clean":"2024-04-26T11:28:17.832078+0000","last_became_active":"2024-04-26T11:26:29.013812+0000","last_became_peered":"2024-04-26T11:26:29.013812+0000","last_unstale":"2024-04-26T11:28:17.832078+0000","last_undegraded":"2024-04-26T11:28:17.832078+0000","last_fullsized":"2024-04-26T11:28:17.832078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:08.683633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.800283+0000","last_change":"2024-04-26T11:26:28.999249+0000","last_active":"2024-04-26T11:28:17.800283+0000","last_peered":"2024-04-26T11:28:17.800283+0000","last_clean":"2024-04-26T11:28:17.800283+0000","last_became_active":"2024-04-26T11:26:28.998927+0000","last_became_peered":"2024-04-26T11:26:28.998927+0000","last_unstale":"2024-04-26T11:28:17.800283+0000","last_undegraded":"2024-04-26T11:28:17.800283+0000","last_fullsized":"2024-04-26T11:28:17.800283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:56.674537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.836010+0000","last_change":"2024-04-26T11:26:29.017202+0000","last_active":"2024-04-26T11:28:17.836010+0000","last_peered":"2024-04-26T11:28:17.836010+0000","last_clean":"2024-04-26T11:28:17.836010+0000","last_became_active":"2024-04-26T11:26:29.016849+0000","last_became_peered":"2024-04-26T11:26:29.016849+0000","last_unstale":"2024-04-26T11:28:17.836010+0000","last_undegraded":"2024-04-26T11:28:17.836010+0000","last_fullsized":"2024-04-26T11:28:17.836010+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:58:22.164118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.816779+0000","last_change":"2024-04-26T11:27:19.278294+0000","last_active":"2024-04-26T11:28:17.816779+0000","last_peered":"2024-04-26T11:28:17.816779+0000","last_clean":"2024-04-26T11:28:17.816779+0000","last_became_active":"2024-04-26T11:27:19.277853+0000","last_became_peered":"2024-04-26T11:27:19.277853+0000","last_unstale":"2024-04-26T11:28:17.816779+0000","last_undegraded":"2024-04-26T11:28:17.816779+0000","last_fullsized":"2024-04-26T11:28:17.816779+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:12:18.733166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.811178+0000","last_change":"2024-04-26T11:26:28.991406+0000","last_active":"2024-04-26T11:28:17.811178+0000","last_peered":"2024-04-26T11:28:17.811178+0000","last_clean":"2024-04-26T11:28:17.811178+0000","last_became_active":"2024-04-26T11:26:28.989303+0000","last_became_peered":"2024-04-26T11:26:28.989303+0000","last_unstale":"2024-04-26T11:28:17.811178+0000","last_undegraded":"2024-04-26T11:28:17.811178+0000","last_fullsized":"2024-04-26T11:28:17.811178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:42:59.449078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.836608+0000","last_change":"2024-04-26T11:26:29.020922+0000","last_active":"2024-04-26T11:28:17.836608+0000","last_peered":"2024-04-26T11:28:17.836608+0000","last_clean":"2024-04-26T11:28:17.836608+0000","last_became_active":"2024-04-26T11:26:29.020576+0000","last_became_peered":"2024-04-26T11:26:29.020576+0000","last_unstale":"2024-04-26T11:28:17.836608+0000","last_undegraded":"2024-04-26T11:28:17.836608+0000","last_fullsized":"2024-04-26T11:28:17.836608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:48.770319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.817013+0000","last_change":"2024-04-26T11:26:29.004171+0000","last_active":"2024-04-26T11:28:17.817013+0000","last_peered":"2024-04-26T11:28:17.817013+0000","last_clean":"2024-04-26T11:28:17.817013+0000","last_became_active":"2024-04-26T11:26:29.003816+0000","last_became_peered":"2024-04-26T11:26:29.003816+0000","last_unstale":"2024-04-26T11:28:17.817013+0000","last_undegraded":"2024-04-26T11:28:17.817013+0000","last_fullsized":"2024-04-26T11:28:17.817013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:45:14.807634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.809247+0000","last_change":"2024-04-26T11:26:28.999453+0000","last_active":"2024-04-26T11:28:17.809247+0000","last_peered":"2024-04-26T11:28:17.809247+0000","last_clean":"2024-04-26T11:28:17.809247+0000","last_became_active":"2024-04-26T11:26:28.999104+0000","last_became_peered":"2024-04-26T11:26:28.999104+0000","last_unstale":"2024-04-26T11:28:17.809247+0000","last_undegraded":"2024-04-26T11:28:17.809247+0000","last_fullsized":"2024-04-26T11:28:17.809247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:55.408949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825591+0000","last_change":"2024-04-26T11:26:28.999218+0000","last_active":"2024-04-26T11:28:17.825591+0000","last_peered":"2024-04-26T11:28:17.825591+0000","last_clean":"2024-04-26T11:28:17.825591+0000","last_became_active":"2024-04-26T11:26:28.998584+0000","last_became_peered":"2024-04-26T11:26:28.998584+0000","last_unstale":"2024-04-26T11:28:17.825591+0000","last_undegraded":"2024-04-26T11:28:17.825591+0000","last_fullsized":"2024-04-26T11:28:17.825591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:18.040520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.798069+0000","last_change":"2024-04-26T11:26:28.997121+0000","last_active":"2024-04-26T11:28:17.798069+0000","last_peered":"2024-04-26T11:28:17.798069+0000","last_clean":"2024-04-26T11:28:17.798069+0000","last_became_active":"2024-04-26T11:26:28.996556+0000","last_became_peered":"2024-04-26T11:26:28.996556+0000","last_unstale":"2024-04-26T11:28:17.798069+0000","last_undegraded":"2024-04-26T11:28:17.798069+0000","last_fullsized":"2024-04-26T11:28:17.798069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:18:55.251371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.826697+0000","last_change":"2024-04-26T11:26:28.999525+0000","last_active":"2024-04-26T11:28:17.826697+0000","last_peered":"2024-04-26T11:28:17.826697+0000","last_clean":"2024-04-26T11:28:17.826697+0000","last_became_active":"2024-04-26T11:26:28.998578+0000","last_became_peered":"2024-04-26T11:26:28.998578+0000","last_unstale":"2024-04-26T11:28:17.826697+0000","last_undegraded":"2024-04-26T11:28:17.826697+0000","last_fullsized":"2024-04-26T11:28:17.826697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:32:35.523804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.829596+0000","last_change":"2024-04-26T11:26:29.008427+0000","last_active":"2024-04-26T11:28:17.829596+0000","last_peered":"2024-04-26T11:28:17.829596+0000","last_clean":"2024-04-26T11:28:17.829596+0000","last_became_active":"2024-04-26T11:26:29.008073+0000","last_became_peered":"2024-04-26T11:26:29.008073+0000","last_unstale":"2024-04-26T11:28:17.829596+0000","last_undegraded":"2024-04-26T11:28:17.829596+0000","last_fullsized":"2024-04-26T11:28:17.829596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:03.185941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.829020+0000","last_change":"2024-04-26T11:26:29.015136+0000","last_active":"2024-04-26T11:28:17.829020+0000","last_peered":"2024-04-26T11:28:17.829020+0000","last_clean":"2024-04-26T11:28:17.829020+0000","last_became_active":"2024-04-26T11:26:29.014800+0000","last_became_peered":"2024-04-26T11:26:29.014800+0000","last_unstale":"2024-04-26T11:28:17.829020+0000","last_undegraded":"2024-04-26T11:28:17.829020+0000","last_fullsized":"2024-04-26T11:28:17.829020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:28:35.871046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.831807+0000","last_change":"2024-04-26T11:26:29.024795+0000","last_active":"2024-04-26T11:28:17.831807+0000","last_peered":"2024-04-26T11:28:17.831807+0000","last_clean":"2024-04-26T11:28:17.831807+0000","last_became_active":"2024-04-26T11:26:29.024430+0000","last_became_peered":"2024-04-26T11:26:29.024430+0000","last_unstale":"2024-04-26T11:28:17.831807+0000","last_undegraded":"2024-04-26T11:28:17.831807+0000","last_fullsized":"2024-04-26T11:28:17.831807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:05:20.172869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825898+0000","last_change":"2024-04-26T11:26:29.018822+0000","last_active":"2024-04-26T11:28:17.825898+0000","last_peered":"2024-04-26T11:28:17.825898+0000","last_clean":"2024-04-26T11:28:17.825898+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:28:17.825898+0000","last_undegraded":"2024-04-26T11:28:17.825898+0000","last_fullsized":"2024-04-26T11:28:17.825898+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:04.475400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.818312+0000","last_change":"2024-04-26T11:26:29.001984+0000","last_active":"2024-04-26T11:28:17.818312+0000","last_peered":"2024-04-26T11:28:17.818312+0000","last_clean":"2024-04-26T11:28:17.818312+0000","last_became_active":"2024-04-26T11:26:28.999492+0000","last_became_peered":"2024-04-26T11:26:28.999492+0000","last_unstale":"2024-04-26T11:28:17.818312+0000","last_undegraded":"2024-04-26T11:28:17.818312+0000","last_fullsized":"2024-04-26T11:28:17.818312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:21.398823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837202+0000","last_change":"2024-04-26T11:26:29.019677+0000","last_active":"2024-04-26T11:28:17.837202+0000","last_peered":"2024-04-26T11:28:17.837202+0000","last_clean":"2024-04-26T11:28:17.837202+0000","last_became_active":"2024-04-26T11:26:29.019315+0000","last_became_peered":"2024-04-26T11:26:29.019315+0000","last_unstale":"2024-04-26T11:28:17.837202+0000","last_undegraded":"2024-04-26T11:28:17.837202+0000","last_fullsized":"2024-04-26T11:28:17.837202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:20.239140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242662+0000","last_change":"2024-04-26T11:26:24.964887+0000","last_active":"2024-04-26T11:27:19.242662+0000","last_peered":"2024-04-26T11:27:19.242662+0000","last_clean":"2024-04-26T11:27:19.242662+0000","last_became_active":"2024-04-26T11:26:24.964380+0000","last_became_peered":"2024-04-26T11:26:24.964380+0000","last_unstale":"2024-04-26T11:27:19.242662+0000","last_undegraded":"2024-04-26T11:27:19.242662+0000","last_fullsized":"2024-04-26T11:27:19.242662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:59:41.347542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.817293+0000","last_change":"2024-04-26T11:27:19.244889+0000","last_active":"2024-04-26T11:28:17.817293+0000","last_peered":"2024-04-26T11:28:17.817293+0000","last_clean":"2024-04-26T11:28:17.817293+0000","last_became_active":"2024-04-26T11:27:19.244428+0000","last_became_peered":"2024-04-26T11:27:19.244428+0000","last_unstale":"2024-04-26T11:28:17.817293+0000","last_undegraded":"2024-04-26T11:28:17.817293+0000","last_fullsized":"2024-04-26T11:28:17.817293+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:09:39.366242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246899+0000","last_change":"2024-04-26T11:26:25.082741+0000","last_active":"2024-04-26T11:27:19.246899+0000","last_peered":"2024-04-26T11:27:19.246899+0000","last_clean":"2024-04-26T11:27:19.246899+0000","last_became_active":"2024-04-26T11:26:25.082361+0000","last_became_peered":"2024-04-26T11:26:25.082361+0000","last_unstale":"2024-04-26T11:27:19.246899+0000","last_undegraded":"2024-04-26T11:27:19.246899+0000","last_fullsized":"2024-04-26T11:27:19.246899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:23:17.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.833585+0000","last_change":"2024-04-26T11:26:29.014817+0000","last_active":"2024-04-26T11:28:17.833585+0000","last_peered":"2024-04-26T11:28:17.833585+0000","last_clean":"2024-04-26T11:28:17.833585+0000","last_became_active":"2024-04-26T11:26:29.014419+0000","last_became_peered":"2024-04-26T11:26:29.014419+0000","last_unstale":"2024-04-26T11:28:17.833585+0000","last_undegraded":"2024-04-26T11:28:17.833585+0000","last_fullsized":"2024-04-26T11:28:17.833585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:33:28.355075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245880+0000","last_change":"2024-04-26T11:26:24.960224+0000","last_active":"2024-04-26T11:27:19.245880+0000","last_peered":"2024-04-26T11:27:19.245880+0000","last_clean":"2024-04-26T11:27:19.245880+0000","last_became_active":"2024-04-26T11:26:24.959848+0000","last_became_peered":"2024-04-26T11:26:24.959848+0000","last_unstale":"2024-04-26T11:27:19.245880+0000","last_undegraded":"2024-04-26T11:27:19.245880+0000","last_fullsized":"2024-04-26T11:27:19.245880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:49:10.932187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.822732+0000","last_change":"2024-04-26T11:26:29.001946+0000","last_active":"2024-04-26T11:28:17.822732+0000","last_peered":"2024-04-26T11:28:17.822732+0000","last_clean":"2024-04-26T11:28:17.822732+0000","last_became_active":"2024-04-26T11:26:29.001579+0000","last_became_peered":"2024-04-26T11:26:29.001579+0000","last_unstale":"2024-04-26T11:28:17.822732+0000","last_undegraded":"2024-04-26T11:28:17.822732+0000","last_fullsized":"2024-04-26T11:28:17.822732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:14:43.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245485+0000","last_change":"2024-04-26T11:26:24.956148+0000","last_active":"2024-04-26T11:27:19.245485+0000","last_peered":"2024-04-26T11:27:19.245485+0000","last_clean":"2024-04-26T11:27:19.245485+0000","last_became_active":"2024-04-26T11:26:24.955736+0000","last_became_peered":"2024-04-26T11:26:24.955736+0000","last_unstale":"2024-04-26T11:27:19.245485+0000","last_undegraded":"2024-04-26T11:27:19.245485+0000","last_fullsized":"2024-04-26T11:27:19.245485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:43.705470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.821808+0000","last_change":"2024-04-26T11:27:19.415134+0000","last_active":"2024-04-26T11:28:17.821808+0000","last_peered":"2024-04-26T11:28:17.821808+0000","last_clean":"2024-04-26T11:28:17.821808+0000","last_became_active":"2024-04-26T11:27:19.414703+0000","last_became_peered":"2024-04-26T11:27:19.414703+0000","last_unstale":"2024-04-26T11:28:17.821808+0000","last_undegraded":"2024-04-26T11:28:17.821808+0000","last_fullsized":"2024-04-26T11:28:17.821808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:41:11.268296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":148,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:46.304255+0000","last_change":"2024-04-26T11:26:24.958350+0000","last_active":"2024-04-26T11:28:46.304255+0000","last_peered":"2024-04-26T11:28:46.304255+0000","last_clean":"2024-04-26T11:28:46.304255+0000","last_became_active":"2024-04-26T11:26:24.957382+0000","last_became_peered":"2024-04-26T11:26:24.957382+0000","last_unstale":"2024-04-26T11:28:46.304255+0000","last_undegraded":"2024-04-26T11:28:46.304255+0000","last_fullsized":"2024-04-26T11:28:46.304255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:03.854227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.816625+0000","last_change":"2024-04-26T11:26:28.997568+0000","last_active":"2024-04-26T11:28:17.816625+0000","last_peered":"2024-04-26T11:28:17.816625+0000","last_clean":"2024-04-26T11:28:17.816625+0000","last_became_active":"2024-04-26T11:26:28.997161+0000","last_became_peered":"2024-04-26T11:26:28.997161+0000","last_unstale":"2024-04-26T11:28:17.816625+0000","last_undegraded":"2024-04-26T11:28:17.816625+0000","last_fullsized":"2024-04-26T11:28:17.816625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:04.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.243026+0000","last_change":"2024-04-26T11:26:24.958093+0000","last_active":"2024-04-26T11:27:19.243026+0000","last_peered":"2024-04-26T11:27:19.243026+0000","last_clean":"2024-04-26T11:27:19.243026+0000","last_became_active":"2024-04-26T11:26:24.957730+0000","last_became_peered":"2024-04-26T11:26:24.957730+0000","last_unstale":"2024-04-26T11:27:19.243026+0000","last_undegraded":"2024-04-26T11:27:19.243026+0000","last_fullsized":"2024-04-26T11:27:19.243026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:09:29.827063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.843097+0000","last_change":"2024-04-26T11:27:19.248406+0000","last_active":"2024-04-26T11:28:17.843097+0000","last_peered":"2024-04-26T11:28:17.843097+0000","last_clean":"2024-04-26T11:28:17.843097+0000","last_became_active":"2024-04-26T11:27:19.247980+0000","last_became_peered":"2024-04-26T11:27:19.247980+0000","last_unstale":"2024-04-26T11:28:17.843097+0000","last_undegraded":"2024-04-26T11:28:17.843097+0000","last_fullsized":"2024-04-26T11:28:17.843097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:51:34.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.411996+0000","last_change":"2024-04-26T11:26:24.963463+0000","last_active":"2024-04-26T11:27:19.411996+0000","last_peered":"2024-04-26T11:27:19.411996+0000","last_clean":"2024-04-26T11:27:19.411996+0000","last_became_active":"2024-04-26T11:26:24.963030+0000","last_became_peered":"2024-04-26T11:26:24.963030+0000","last_unstale":"2024-04-26T11:27:19.411996+0000","last_undegraded":"2024-04-26T11:27:19.411996+0000","last_fullsized":"2024-04-26T11:27:19.411996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:04:17.844500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.841817+0000","last_change":"2024-04-26T11:26:29.013899+0000","last_active":"2024-04-26T11:28:17.841817+0000","last_peered":"2024-04-26T11:28:17.841817+0000","last_clean":"2024-04-26T11:28:17.841817+0000","last_became_active":"2024-04-26T11:26:29.012674+0000","last_became_peered":"2024-04-26T11:26:29.012674+0000","last_unstale":"2024-04-26T11:28:17.841817+0000","last_undegraded":"2024-04-26T11:28:17.841817+0000","last_fullsized":"2024-04-26T11:28:17.841817+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:33:46.259841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240050+0000","last_change":"2024-04-26T11:26:24.957803+0000","last_active":"2024-04-26T11:27:19.240050+0000","last_peered":"2024-04-26T11:27:19.240050+0000","last_clean":"2024-04-26T11:27:19.240050+0000","last_became_active":"2024-04-26T11:26:24.957496+0000","last_became_peered":"2024-04-26T11:26:24.957496+0000","last_unstale":"2024-04-26T11:27:19.240050+0000","last_undegraded":"2024-04-26T11:27:19.240050+0000","last_fullsized":"2024-04-26T11:27:19.240050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:15.497106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.843048+0000","last_change":"2024-04-26T11:26:29.021648+0000","last_active":"2024-04-26T11:28:17.843048+0000","last_peered":"2024-04-26T11:28:17.843048+0000","last_clean":"2024-04-26T11:28:17.843048+0000","last_became_active":"2024-04-26T11:26:29.020689+0000","last_became_peered":"2024-04-26T11:26:29.020689+0000","last_unstale":"2024-04-26T11:28:17.843048+0000","last_undegraded":"2024-04-26T11:28:17.843048+0000","last_fullsized":"2024-04-26T11:28:17.843048+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:15:42.321851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:49.307845+0000","last_change":"2024-04-26T11:26:24.966184+0000","last_active":"2024-04-26T11:28:49.307845+0000","last_peered":"2024-04-26T11:28:49.307845+0000","last_clean":"2024-04-26T11:28:49.307845+0000","last_became_active":"2024-04-26T11:26:24.965850+0000","last_became_peered":"2024-04-26T11:26:24.965850+0000","last_unstale":"2024-04-26T11:28:49.307845+0000","last_undegraded":"2024-04-26T11:28:49.307845+0000","last_fullsized":"2024-04-26T11:28:49.307845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:49.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.822515+0000","last_change":"2024-04-26T11:26:29.003570+0000","last_active":"2024-04-26T11:28:17.822515+0000","last_peered":"2024-04-26T11:28:17.822515+0000","last_clean":"2024-04-26T11:28:17.822515+0000","last_became_active":"2024-04-26T11:26:29.003039+0000","last_became_peered":"2024-04-26T11:26:29.003039+0000","last_unstale":"2024-04-26T11:28:17.822515+0000","last_undegraded":"2024-04-26T11:28:17.822515+0000","last_fullsized":"2024-04-26T11:28:17.822515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:29:16.166432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277109+0000","last_change":"2024-04-26T11:26:24.954968+0000","last_active":"2024-04-26T11:27:19.277109+0000","last_peered":"2024-04-26T11:27:19.277109+0000","last_clean":"2024-04-26T11:27:19.277109+0000","last_became_active":"2024-04-26T11:26:24.954595+0000","last_became_peered":"2024-04-26T11:26:24.954595+0000","last_unstale":"2024-04-26T11:27:19.277109+0000","last_undegraded":"2024-04-26T11:27:19.277109+0000","last_fullsized":"2024-04-26T11:27:19.277109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:52:21.988942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.832433+0000","last_change":"2024-04-26T11:26:29.015601+0000","last_active":"2024-04-26T11:28:17.832433+0000","last_peered":"2024-04-26T11:28:17.832433+0000","last_clean":"2024-04-26T11:28:17.832433+0000","last_became_active":"2024-04-26T11:26:29.015199+0000","last_became_peered":"2024-04-26T11:26:29.015199+0000","last_unstale":"2024-04-26T11:28:17.832433+0000","last_undegraded":"2024-04-26T11:28:17.832433+0000","last_fullsized":"2024-04-26T11:28:17.832433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:15:04.573214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240544+0000","last_change":"2024-04-26T11:26:25.085542+0000","last_active":"2024-04-26T11:27:19.240544+0000","last_peered":"2024-04-26T11:27:19.240544+0000","last_clean":"2024-04-26T11:27:19.240544+0000","last_became_active":"2024-04-26T11:26:25.085230+0000","last_became_peered":"2024-04-26T11:26:25.085230+0000","last_unstale":"2024-04-26T11:27:19.240544+0000","last_undegraded":"2024-04-26T11:27:19.240544+0000","last_fullsized":"2024-04-26T11:27:19.240544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:51.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.827464+0000","last_change":"2024-04-26T11:26:29.003954+0000","last_active":"2024-04-26T11:28:17.827464+0000","last_peered":"2024-04-26T11:28:17.827464+0000","last_clean":"2024-04-26T11:28:17.827464+0000","last_became_active":"2024-04-26T11:26:29.003576+0000","last_became_peered":"2024-04-26T11:26:29.003576+0000","last_unstale":"2024-04-26T11:28:17.827464+0000","last_undegraded":"2024-04-26T11:28:17.827464+0000","last_fullsized":"2024-04-26T11:28:17.827464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:24.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":306,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:49.307870+0000","last_change":"2024-04-26T11:26:24.957823+0000","last_active":"2024-04-26T11:28:49.307870+0000","last_peered":"2024-04-26T11:28:49.307870+0000","last_clean":"2024-04-26T11:28:49.307870+0000","last_became_active":"2024-04-26T11:26:24.957434+0000","last_became_peered":"2024-04-26T11:26:24.957434+0000","last_unstale":"2024-04-26T11:28:49.307870+0000","last_undegraded":"2024-04-26T11:28:49.307870+0000","last_fullsized":"2024-04-26T11:28:49.307870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:38:43.247282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65899,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":244,"num_read_kb":186,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.803191+0000","last_change":"2024-04-26T11:26:28.994764+0000","last_active":"2024-04-26T11:28:17.803191+0000","last_peered":"2024-04-26T11:28:17.803191+0000","last_clean":"2024-04-26T11:28:17.803191+0000","last_became_active":"2024-04-26T11:26:28.994421+0000","last_became_peered":"2024-04-26T11:26:28.994421+0000","last_unstale":"2024-04-26T11:28:17.803191+0000","last_undegraded":"2024-04-26T11:28:17.803191+0000","last_fullsized":"2024-04-26T11:28:17.803191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:06:06.499434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242341+0000","last_change":"2024-04-26T11:26:24.968036+0000","last_active":"2024-04-26T11:27:19.242341+0000","last_peered":"2024-04-26T11:27:19.242341+0000","last_clean":"2024-04-26T11:27:19.242341+0000","last_became_active":"2024-04-26T11:26:24.967684+0000","last_became_peered":"2024-04-26T11:26:24.967684+0000","last_unstale":"2024-04-26T11:27:19.242341+0000","last_undegraded":"2024-04-26T11:27:19.242341+0000","last_fullsized":"2024-04-26T11:27:19.242341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:00:27.467411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.842567+0000","last_change":"2024-04-26T11:26:29.020642+0000","last_active":"2024-04-26T11:28:17.842567+0000","last_peered":"2024-04-26T11:28:17.842567+0000","last_clean":"2024-04-26T11:28:17.842567+0000","last_became_active":"2024-04-26T11:26:29.020278+0000","last_became_peered":"2024-04-26T11:26:29.020278+0000","last_unstale":"2024-04-26T11:28:17.842567+0000","last_undegraded":"2024-04-26T11:28:17.842567+0000","last_fullsized":"2024-04-26T11:28:17.842567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:48.932064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240545+0000","last_change":"2024-04-26T11:26:24.970111+0000","last_active":"2024-04-26T11:27:19.240545+0000","last_peered":"2024-04-26T11:27:19.240545+0000","last_clean":"2024-04-26T11:27:19.240545+0000","last_became_active":"2024-04-26T11:26:24.969629+0000","last_became_peered":"2024-04-26T11:26:24.969629+0000","last_unstale":"2024-04-26T11:27:19.240545+0000","last_undegraded":"2024-04-26T11:27:19.240545+0000","last_fullsized":"2024-04-26T11:27:19.240545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:52:07.859419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.823326+0000","last_change":"2024-04-26T11:26:29.002986+0000","last_active":"2024-04-26T11:28:17.823326+0000","last_peered":"2024-04-26T11:28:17.823326+0000","last_clean":"2024-04-26T11:28:17.823326+0000","last_became_active":"2024-04-26T11:26:29.002610+0000","last_became_peered":"2024-04-26T11:26:29.002610+0000","last_unstale":"2024-04-26T11:28:17.823326+0000","last_undegraded":"2024-04-26T11:28:17.823326+0000","last_fullsized":"2024-04-26T11:28:17.823326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:04.877356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:48.396254+0000","last_change":"2024-04-26T11:26:24.957825+0000","last_active":"2024-04-26T11:28:48.396254+0000","last_peered":"2024-04-26T11:28:48.396254+0000","last_clean":"2024-04-26T11:28:48.396254+0000","last_became_active":"2024-04-26T11:26:24.957464+0000","last_became_peered":"2024-04-26T11:26:24.957464+0000","last_unstale":"2024-04-26T11:28:48.396254+0000","last_undegraded":"2024-04-26T11:28:48.396254+0000","last_fullsized":"2024-04-26T11:28:48.396254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:06:49.350268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.811198+0000","last_change":"2024-04-26T11:26:28.994223+0000","last_active":"2024-04-26T11:28:17.811198+0000","last_peered":"2024-04-26T11:28:17.811198+0000","last_clean":"2024-04-26T11:28:17.811198+0000","last_became_active":"2024-04-26T11:26:28.993388+0000","last_became_peered":"2024-04-26T11:26:28.993388+0000","last_unstale":"2024-04-26T11:28:17.811198+0000","last_undegraded":"2024-04-26T11:28:17.811198+0000","last_fullsized":"2024-04-26T11:28:17.811198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:07:24.208028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245322+0000","last_change":"2024-04-26T11:26:25.078473+0000","last_active":"2024-04-26T11:27:19.245322+0000","last_peered":"2024-04-26T11:27:19.245322+0000","last_clean":"2024-04-26T11:27:19.245322+0000","last_became_active":"2024-04-26T11:26:25.078138+0000","last_became_peered":"2024-04-26T11:26:25.078138+0000","last_unstale":"2024-04-26T11:27:19.245322+0000","last_undegraded":"2024-04-26T11:27:19.245322+0000","last_fullsized":"2024-04-26T11:27:19.245322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:19.692345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.811891+0000","last_change":"2024-04-26T11:26:28.998687+0000","last_active":"2024-04-26T11:28:17.811891+0000","last_peered":"2024-04-26T11:28:17.811891+0000","last_clean":"2024-04-26T11:28:17.811891+0000","last_became_active":"2024-04-26T11:26:28.998358+0000","last_became_peered":"2024-04-26T11:26:28.998358+0000","last_unstale":"2024-04-26T11:28:17.811891+0000","last_undegraded":"2024-04-26T11:28:17.811891+0000","last_fullsized":"2024-04-26T11:28:17.811891+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:40:51.358188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.246156+0000","last_change":"2024-04-26T11:26:24.956213+0000","last_active":"2024-04-26T11:27:19.246156+0000","last_peered":"2024-04-26T11:27:19.246156+0000","last_clean":"2024-04-26T11:27:19.246156+0000","last_became_active":"2024-04-26T11:26:24.955773+0000","last_became_peered":"2024-04-26T11:26:24.955773+0000","last_unstale":"2024-04-26T11:27:19.246156+0000","last_undegraded":"2024-04-26T11:27:19.246156+0000","last_fullsized":"2024-04-26T11:27:19.246156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:38.253367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.814609+0000","last_change":"2024-04-26T11:26:29.001389+0000","last_active":"2024-04-26T11:28:17.814609+0000","last_peered":"2024-04-26T11:28:17.814609+0000","last_clean":"2024-04-26T11:28:17.814609+0000","last_became_active":"2024-04-26T11:26:29.001023+0000","last_became_peered":"2024-04-26T11:26:29.001023+0000","last_unstale":"2024-04-26T11:28:17.814609+0000","last_undegraded":"2024-04-26T11:28:17.814609+0000","last_fullsized":"2024-04-26T11:28:17.814609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:57:23.606518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.895387+0000","last_change":"2024-04-26T11:27:19.893959+0000","last_active":"2024-04-26T11:27:19.895387+0000","last_peered":"2024-04-26T11:27:19.895387+0000","last_clean":"2024-04-26T11:27:19.895387+0000","last_became_active":"2024-04-26T11:27:19.892819+0000","last_became_peered":"2024-04-26T11:27:19.892819+0000","last_unstale":"2024-04-26T11:27:19.895387+0000","last_undegraded":"2024-04-26T11:27:19.895387+0000","last_fullsized":"2024-04-26T11:27:19.895387+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:53:39.829381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.836883+0000","last_change":"2024-04-26T11:26:29.019906+0000","last_active":"2024-04-26T11:28:17.836883+0000","last_peered":"2024-04-26T11:28:17.836883+0000","last_clean":"2024-04-26T11:28:17.836883+0000","last_became_active":"2024-04-26T11:26:29.019545+0000","last_became_peered":"2024-04-26T11:26:29.019545+0000","last_unstale":"2024-04-26T11:28:17.836883+0000","last_undegraded":"2024-04-26T11:28:17.836883+0000","last_fullsized":"2024-04-26T11:28:17.836883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:48.617269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277025+0000","last_change":"2024-04-26T11:26:24.956400+0000","last_active":"2024-04-26T11:27:19.277025+0000","last_peered":"2024-04-26T11:27:19.277025+0000","last_clean":"2024-04-26T11:27:19.277025+0000","last_became_active":"2024-04-26T11:26:24.956030+0000","last_became_peered":"2024-04-26T11:26:24.956030+0000","last_unstale":"2024-04-26T11:27:19.277025+0000","last_undegraded":"2024-04-26T11:27:19.277025+0000","last_fullsized":"2024-04-26T11:27:19.277025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:47.841006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.824606+0000","last_change":"2024-04-26T11:26:29.013087+0000","last_active":"2024-04-26T11:28:17.824606+0000","last_peered":"2024-04-26T11:28:17.824606+0000","last_clean":"2024-04-26T11:28:17.824606+0000","last_became_active":"2024-04-26T11:26:29.012330+0000","last_became_peered":"2024-04-26T11:26:29.012330+0000","last_unstale":"2024-04-26T11:28:17.824606+0000","last_undegraded":"2024-04-26T11:28:17.824606+0000","last_fullsized":"2024-04-26T11:28:17.824606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:12:50.752329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.812867+0000","last_change":"2024-04-26T11:27:19.245410+0000","last_active":"2024-04-26T11:28:17.812867+0000","last_peered":"2024-04-26T11:28:17.812867+0000","last_clean":"2024-04-26T11:28:17.812867+0000","last_became_active":"2024-04-26T11:27:19.244578+0000","last_became_peered":"2024-04-26T11:27:19.244578+0000","last_unstale":"2024-04-26T11:28:17.812867+0000","last_undegraded":"2024-04-26T11:28:17.812867+0000","last_fullsized":"2024-04-26T11:28:17.812867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:07:50.576496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:46.390943+0000","last_change":"2024-04-26T11:26:25.082517+0000","last_active":"2024-04-26T11:28:46.390943+0000","last_peered":"2024-04-26T11:28:46.390943+0000","last_clean":"2024-04-26T11:28:46.390943+0000","last_became_active":"2024-04-26T11:26:25.081724+0000","last_became_peered":"2024-04-26T11:26:25.081724+0000","last_unstale":"2024-04-26T11:28:46.390943+0000","last_undegraded":"2024-04-26T11:28:46.390943+0000","last_fullsized":"2024-04-26T11:28:46.390943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:59:40.006615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837462+0000","last_change":"2024-04-26T11:26:29.021425+0000","last_active":"2024-04-26T11:28:17.837462+0000","last_peered":"2024-04-26T11:28:17.837462+0000","last_clean":"2024-04-26T11:28:17.837462+0000","last_became_active":"2024-04-26T11:26:29.020881+0000","last_became_peered":"2024-04-26T11:26:29.020881+0000","last_unstale":"2024-04-26T11:28:17.837462+0000","last_undegraded":"2024-04-26T11:28:17.837462+0000","last_fullsized":"2024-04-26T11:28:17.837462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:18.350323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240589+0000","last_change":"2024-04-26T11:26:24.963614+0000","last_active":"2024-04-26T11:27:19.240589+0000","last_peered":"2024-04-26T11:27:19.240589+0000","last_clean":"2024-04-26T11:27:19.240589+0000","last_became_active":"2024-04-26T11:26:24.963312+0000","last_became_peered":"2024-04-26T11:26:24.963312+0000","last_unstale":"2024-04-26T11:27:19.240589+0000","last_undegraded":"2024-04-26T11:27:19.240589+0000","last_fullsized":"2024-04-26T11:27:19.240589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:55.886637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":360,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.276824+0000","last_change":"2024-04-26T11:24:54.292436+0000","last_active":"2024-04-26T11:27:19.276824+0000","last_peered":"2024-04-26T11:27:19.276824+0000","last_clean":"2024-04-26T11:27:19.276824+0000","last_became_active":"2024-04-26T11:24:54.291921+0000","last_became_peered":"2024-04-26T11:24:54.291921+0000","last_unstale":"2024-04-26T11:27:19.276824+0000","last_undegraded":"2024-04-26T11:27:19.276824+0000","last_fullsized":"2024-04-26T11:27:19.276824+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_clean_scrub_stamp":"2024-04-26T11:21:00.617277+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:11:01.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.820865+0000","last_change":"2024-04-26T11:26:29.000905+0000","last_active":"2024-04-26T11:28:17.820865+0000","last_peered":"2024-04-26T11:28:17.820865+0000","last_clean":"2024-04-26T11:28:17.820865+0000","last_became_active":"2024-04-26T11:26:29.000517+0000","last_became_peered":"2024-04-26T11:26:29.000517+0000","last_unstale":"2024-04-26T11:28:17.820865+0000","last_undegraded":"2024-04-26T11:28:17.820865+0000","last_fullsized":"2024-04-26T11:28:17.820865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:38:55.394316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.277140+0000","last_change":"2024-04-26T11:26:24.957489+0000","last_active":"2024-04-26T11:27:19.277140+0000","last_peered":"2024-04-26T11:27:19.277140+0000","last_clean":"2024-04-26T11:27:19.277140+0000","last_became_active":"2024-04-26T11:26:24.957151+0000","last_became_peered":"2024-04-26T11:26:24.957151+0000","last_unstale":"2024-04-26T11:27:19.277140+0000","last_undegraded":"2024-04-26T11:27:19.277140+0000","last_fullsized":"2024-04-26T11:27:19.277140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:42:25.343105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245739+0000","last_change":"2024-04-26T11:26:25.086767+0000","last_active":"2024-04-26T11:27:19.245739+0000","last_peered":"2024-04-26T11:27:19.245739+0000","last_clean":"2024-04-26T11:27:19.245739+0000","last_became_active":"2024-04-26T11:26:25.086440+0000","last_became_peered":"2024-04-26T11:26:25.086440+0000","last_unstale":"2024-04-26T11:27:19.245739+0000","last_undegraded":"2024-04-26T11:27:19.245739+0000","last_fullsized":"2024-04-26T11:27:19.245739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:02.294644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825487+0000","last_change":"2024-04-26T11:26:29.003683+0000","last_active":"2024-04-26T11:28:17.825487+0000","last_peered":"2024-04-26T11:28:17.825487+0000","last_clean":"2024-04-26T11:28:17.825487+0000","last_became_active":"2024-04-26T11:26:29.003307+0000","last_became_peered":"2024-04-26T11:26:29.003307+0000","last_unstale":"2024-04-26T11:28:17.825487+0000","last_undegraded":"2024-04-26T11:28:17.825487+0000","last_fullsized":"2024-04-26T11:28:17.825487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:41:00.124540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:18.048485+0000","last_change":"2024-04-26T11:27:19.849965+0000","last_active":"2024-04-26T11:28:18.048485+0000","last_peered":"2024-04-26T11:28:18.048485+0000","last_clean":"2024-04-26T11:28:18.048485+0000","last_became_active":"2024-04-26T11:27:19.840545+0000","last_became_peered":"2024-04-26T11:27:19.840545+0000","last_unstale":"2024-04-26T11:28:18.048485+0000","last_undegraded":"2024-04-26T11:28:18.048485+0000","last_fullsized":"2024-04-26T11:28:18.048485+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:08.003704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.839888+0000","last_change":"2024-04-26T11:26:29.016783+0000","last_active":"2024-04-26T11:28:17.839888+0000","last_peered":"2024-04-26T11:28:17.839888+0000","last_clean":"2024-04-26T11:28:17.839888+0000","last_became_active":"2024-04-26T11:26:29.013998+0000","last_became_peered":"2024-04-26T11:26:29.013998+0000","last_unstale":"2024-04-26T11:28:17.839888+0000","last_undegraded":"2024-04-26T11:28:17.839888+0000","last_fullsized":"2024-04-26T11:28:17.839888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:24:44.453094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240650+0000","last_change":"2024-04-26T11:26:25.902330+0000","last_active":"2024-04-26T11:27:19.240650+0000","last_peered":"2024-04-26T11:27:19.240650+0000","last_clean":"2024-04-26T11:27:19.240650+0000","last_became_active":"2024-04-26T11:26:25.901948+0000","last_became_peered":"2024-04-26T11:26:25.901948+0000","last_unstale":"2024-04-26T11:27:19.240650+0000","last_undegraded":"2024-04-26T11:27:19.240650+0000","last_fullsized":"2024-04-26T11:27:19.240650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:00:23.541393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.844428+0000","last_change":"2024-04-26T11:26:29.025692+0000","last_active":"2024-04-26T11:28:17.844428+0000","last_peered":"2024-04-26T11:28:17.844428+0000","last_clean":"2024-04-26T11:28:17.844428+0000","last_became_active":"2024-04-26T11:26:29.025112+0000","last_became_peered":"2024-04-26T11:26:29.025112+0000","last_unstale":"2024-04-26T11:28:17.844428+0000","last_undegraded":"2024-04-26T11:28:17.844428+0000","last_fullsized":"2024-04-26T11:28:17.844428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:39:22.637865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.244481+0000","last_change":"2024-04-26T11:26:25.080270+0000","last_active":"2024-04-26T11:27:19.244481+0000","last_peered":"2024-04-26T11:27:19.244481+0000","last_clean":"2024-04-26T11:27:19.244481+0000","last_became_active":"2024-04-26T11:26:25.079895+0000","last_became_peered":"2024-04-26T11:26:25.079895+0000","last_unstale":"2024-04-26T11:27:19.244481+0000","last_undegraded":"2024-04-26T11:27:19.244481+0000","last_fullsized":"2024-04-26T11:27:19.244481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:47:08.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.790599+0000","last_change":"2024-04-26T11:26:28.996540+0000","last_active":"2024-04-26T11:28:17.790599+0000","last_peered":"2024-04-26T11:28:17.790599+0000","last_clean":"2024-04-26T11:28:17.790599+0000","last_became_active":"2024-04-26T11:26:28.994724+0000","last_became_peered":"2024-04-26T11:26:28.994724+0000","last_unstale":"2024-04-26T11:28:17.790599+0000","last_undegraded":"2024-04-26T11:28:17.790599+0000","last_fullsized":"2024-04-26T11:28:17.790599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:13.382332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245883+0000","last_change":"2024-04-26T11:26:24.967935+0000","last_active":"2024-04-26T11:27:19.245883+0000","last_peered":"2024-04-26T11:27:19.245883+0000","last_clean":"2024-04-26T11:27:19.245883+0000","last_became_active":"2024-04-26T11:26:24.966465+0000","last_became_peered":"2024-04-26T11:26:24.966465+0000","last_unstale":"2024-04-26T11:27:19.245883+0000","last_undegraded":"2024-04-26T11:27:19.245883+0000","last_fullsized":"2024-04-26T11:27:19.245883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:27:38.817316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.833429+0000","last_change":"2024-04-26T11:26:29.024243+0000","last_active":"2024-04-26T11:28:17.833429+0000","last_peered":"2024-04-26T11:28:17.833429+0000","last_clean":"2024-04-26T11:28:17.833429+0000","last_became_active":"2024-04-26T11:26:29.023900+0000","last_became_peered":"2024-04-26T11:26:29.023900+0000","last_unstale":"2024-04-26T11:28:17.833429+0000","last_undegraded":"2024-04-26T11:28:17.833429+0000","last_fullsized":"2024-04-26T11:28:17.833429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:28:23.446568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242131+0000","last_change":"2024-04-26T11:26:24.969953+0000","last_active":"2024-04-26T11:27:19.242131+0000","last_peered":"2024-04-26T11:27:19.242131+0000","last_clean":"2024-04-26T11:27:19.242131+0000","last_became_active":"2024-04-26T11:26:24.969371+0000","last_became_peered":"2024-04-26T11:26:24.969371+0000","last_unstale":"2024-04-26T11:27:19.242131+0000","last_undegraded":"2024-04-26T11:27:19.242131+0000","last_fullsized":"2024-04-26T11:27:19.242131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:07:24.323833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.841678+0000","last_change":"2024-04-26T11:26:29.029771+0000","last_active":"2024-04-26T11:28:17.841678+0000","last_peered":"2024-04-26T11:28:17.841678+0000","last_clean":"2024-04-26T11:28:17.841678+0000","last_became_active":"2024-04-26T11:26:29.029387+0000","last_became_peered":"2024-04-26T11:26:29.029387+0000","last_unstale":"2024-04-26T11:28:17.841678+0000","last_undegraded":"2024-04-26T11:28:17.841678+0000","last_fullsized":"2024-04-26T11:28:17.841678+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:40.856935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.245363+0000","last_change":"2024-04-26T11:26:24.958608+0000","last_active":"2024-04-26T11:27:19.245363+0000","last_peered":"2024-04-26T11:27:19.245363+0000","last_clean":"2024-04-26T11:27:19.245363+0000","last_became_active":"2024-04-26T11:26:24.958188+0000","last_became_peered":"2024-04-26T11:26:24.958188+0000","last_unstale":"2024-04-26T11:27:19.245363+0000","last_undegraded":"2024-04-26T11:27:19.245363+0000","last_fullsized":"2024-04-26T11:27:19.245363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:31:41.442551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825185+0000","last_change":"2024-04-26T11:26:29.000207+0000","last_active":"2024-04-26T11:28:17.825185+0000","last_peered":"2024-04-26T11:28:17.825185+0000","last_clean":"2024-04-26T11:28:17.825185+0000","last_became_active":"2024-04-26T11:26:28.999894+0000","last_became_peered":"2024-04-26T11:26:28.999894+0000","last_unstale":"2024-04-26T11:28:17.825185+0000","last_undegraded":"2024-04-26T11:28:17.825185+0000","last_fullsized":"2024-04-26T11:28:17.825185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:52:10.349511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.242688+0000","last_change":"2024-04-26T11:26:25.902036+0000","last_active":"2024-04-26T11:27:19.242688+0000","last_peered":"2024-04-26T11:27:19.242688+0000","last_clean":"2024-04-26T11:27:19.242688+0000","last_became_active":"2024-04-26T11:26:25.901567+0000","last_became_peered":"2024-04-26T11:26:25.901567+0000","last_unstale":"2024-04-26T11:27:19.242688+0000","last_undegraded":"2024-04-26T11:27:19.242688+0000","last_fullsized":"2024-04-26T11:27:19.242688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:58.757780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825941+0000","last_change":"2024-04-26T11:26:28.998915+0000","last_active":"2024-04-26T11:28:17.825941+0000","last_peered":"2024-04-26T11:28:17.825941+0000","last_clean":"2024-04-26T11:28:17.825941+0000","last_became_active":"2024-04-26T11:26:28.998180+0000","last_became_peered":"2024-04-26T11:26:28.998180+0000","last_unstale":"2024-04-26T11:28:17.825941+0000","last_undegraded":"2024-04-26T11:28:17.825941+0000","last_fullsized":"2024-04-26T11:28:17.825941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:19.972697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.240438+0000","last_change":"2024-04-26T11:26:24.966029+0000","last_active":"2024-04-26T11:27:19.240438+0000","last_peered":"2024-04-26T11:27:19.240438+0000","last_clean":"2024-04-26T11:27:19.240438+0000","last_became_active":"2024-04-26T11:26:24.965676+0000","last_became_peered":"2024-04-26T11:26:24.965676+0000","last_unstale":"2024-04-26T11:27:19.240438+0000","last_undegraded":"2024-04-26T11:27:19.240438+0000","last_fullsized":"2024-04-26T11:27:19.240438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:18:06.904563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837486+0000","last_change":"2024-04-26T11:26:29.020093+0000","last_active":"2024-04-26T11:28:17.837486+0000","last_peered":"2024-04-26T11:28:17.837486+0000","last_clean":"2024-04-26T11:28:17.837486+0000","last_became_active":"2024-04-26T11:26:29.019690+0000","last_became_peered":"2024-04-26T11:26:29.019690+0000","last_unstale":"2024-04-26T11:28:17.837486+0000","last_undegraded":"2024-04-26T11:28:17.837486+0000","last_fullsized":"2024-04-26T11:28:17.837486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:35:44.501217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:27:19.413081+0000","last_change":"2024-04-26T11:26:24.966757+0000","last_active":"2024-04-26T11:27:19.413081+0000","last_peered":"2024-04-26T11:27:19.413081+0000","last_clean":"2024-04-26T11:27:19.413081+0000","last_became_active":"2024-04-26T11:26:24.966298+0000","last_became_peered":"2024-04-26T11:26:24.966298+0000","last_unstale":"2024-04-26T11:27:19.413081+0000","last_undegraded":"2024-04-26T11:27:19.413081+0000","last_fullsized":"2024-04-26T11:27:19.413081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:17:40.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.816124+0000","last_change":"2024-04-26T11:26:28.997774+0000","last_active":"2024-04-26T11:28:17.816124+0000","last_peered":"2024-04-26T11:28:17.816124+0000","last_clean":"2024-04-26T11:28:17.816124+0000","last_became_active":"2024-04-26T11:26:28.997455+0000","last_became_peered":"2024-04-26T11:26:28.997455+0000","last_unstale":"2024-04-26T11:28:17.816124+0000","last_undegraded":"2024-04-26T11:28:17.816124+0000","last_fullsized":"2024-04-26T11:28:17.816124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:56.097022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.815649+0000","last_change":"2024-04-26T11:26:28.999714+0000","last_active":"2024-04-26T11:28:17.815649+0000","last_peered":"2024-04-26T11:28:17.815649+0000","last_clean":"2024-04-26T11:28:17.815649+0000","last_became_active":"2024-04-26T11:26:28.999377+0000","last_became_peered":"2024-04-26T11:26:28.999377+0000","last_unstale":"2024-04-26T11:28:17.815649+0000","last_undegraded":"2024-04-26T11:28:17.815649+0000","last_fullsized":"2024-04-26T11:28:17.815649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:39:59.141556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.806463+0000","last_change":"2024-04-26T11:27:19.247154+0000","last_active":"2024-04-26T11:28:17.806463+0000","last_peered":"2024-04-26T11:28:17.806463+0000","last_clean":"2024-04-26T11:28:17.806463+0000","last_became_active":"2024-04-26T11:27:19.246590+0000","last_became_peered":"2024-04-26T11:27:19.246590+0000","last_unstale":"2024-04-26T11:28:17.806463+0000","last_undegraded":"2024-04-26T11:28:17.806463+0000","last_fullsized":"2024-04-26T11:28:17.806463+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:12:58.401699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837157+0000","last_change":"2024-04-26T11:26:29.019983+0000","last_active":"2024-04-26T11:28:17.837157+0000","last_peered":"2024-04-26T11:28:17.837157+0000","last_clean":"2024-04-26T11:28:17.837157+0000","last_became_active":"2024-04-26T11:26:29.019536+0000","last_became_peered":"2024-04-26T11:26:29.019536+0000","last_unstale":"2024-04-26T11:28:17.837157+0000","last_undegraded":"2024-04-26T11:28:17.837157+0000","last_fullsized":"2024-04-26T11:28:17.837157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:15:06.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.834360+0000","last_change":"2024-04-26T11:26:29.027089+0000","last_active":"2024-04-26T11:28:17.834360+0000","last_peered":"2024-04-26T11:28:17.834360+0000","last_clean":"2024-04-26T11:28:17.834360+0000","last_became_active":"2024-04-26T11:26:29.026030+0000","last_became_peered":"2024-04-26T11:26:29.026030+0000","last_unstale":"2024-04-26T11:28:17.834360+0000","last_undegraded":"2024-04-26T11:28:17.834360+0000","last_fullsized":"2024-04-26T11:28:17.834360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:24:06.105600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.825719+0000","last_change":"2024-04-26T11:26:29.018472+0000","last_active":"2024-04-26T11:28:17.825719+0000","last_peered":"2024-04-26T11:28:17.825719+0000","last_clean":"2024-04-26T11:28:17.825719+0000","last_became_active":"2024-04-26T11:26:29.015947+0000","last_became_peered":"2024-04-26T11:26:29.015947+0000","last_unstale":"2024-04-26T11:28:17.825719+0000","last_undegraded":"2024-04-26T11:28:17.825719+0000","last_fullsized":"2024-04-26T11:28:17.825719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:35:48.082709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.814545+0000","last_change":"2024-04-26T11:26:28.997582+0000","last_active":"2024-04-26T11:28:17.814545+0000","last_peered":"2024-04-26T11:28:17.814545+0000","last_clean":"2024-04-26T11:28:17.814545+0000","last_became_active":"2024-04-26T11:26:28.997202+0000","last_became_peered":"2024-04-26T11:26:28.997202+0000","last_unstale":"2024-04-26T11:28:17.814545+0000","last_undegraded":"2024-04-26T11:28:17.814545+0000","last_fullsized":"2024-04-26T11:28:17.814545+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:59.776455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.814923+0000","last_change":"2024-04-26T11:26:28.999428+0000","last_active":"2024-04-26T11:28:17.814923+0000","last_peered":"2024-04-26T11:28:17.814923+0000","last_clean":"2024-04-26T11:28:17.814923+0000","last_became_active":"2024-04-26T11:26:28.999093+0000","last_became_peered":"2024-04-26T11:26:28.999093+0000","last_unstale":"2024-04-26T11:28:17.814923+0000","last_undegraded":"2024-04-26T11:28:17.814923+0000","last_fullsized":"2024-04-26T11:28:17.814923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:07.222231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.832441+0000","last_change":"2024-04-26T11:26:29.019673+0000","last_active":"2024-04-26T11:28:17.832441+0000","last_peered":"2024-04-26T11:28:17.832441+0000","last_clean":"2024-04-26T11:28:17.832441+0000","last_became_active":"2024-04-26T11:26:29.017289+0000","last_became_peered":"2024-04-26T11:26:29.017289+0000","last_unstale":"2024-04-26T11:28:17.832441+0000","last_undegraded":"2024-04-26T11:28:17.832441+0000","last_fullsized":"2024-04-26T11:28:17.832441+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:35.848004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.837058+0000","last_change":"2024-04-26T11:26:29.017234+0000","last_active":"2024-04-26T11:28:17.837058+0000","last_peered":"2024-04-26T11:28:17.837058+0000","last_clean":"2024-04-26T11:28:17.837058+0000","last_became_active":"2024-04-26T11:26:29.016863+0000","last_became_peered":"2024-04-26T11:26:29.016863+0000","last_unstale":"2024-04-26T11:28:17.837058+0000","last_undegraded":"2024-04-26T11:28:17.837058+0000","last_fullsized":"2024-04-26T11:28:17.837058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:14:31.136286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.839115+0000","last_change":"2024-04-26T11:26:29.018122+0000","last_active":"2024-04-26T11:28:17.839115+0000","last_peered":"2024-04-26T11:28:17.839115+0000","last_clean":"2024-04-26T11:28:17.839115+0000","last_became_active":"2024-04-26T11:26:29.016269+0000","last_became_peered":"2024-04-26T11:26:29.016269+0000","last_unstale":"2024-04-26T11:28:17.839115+0000","last_undegraded":"2024-04-26T11:28:17.839115+0000","last_fullsized":"2024-04-26T11:28:17.839115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:22.135086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.826827+0000","last_change":"2024-04-26T11:27:19.278596+0000","last_active":"2024-04-26T11:28:17.826827+0000","last_peered":"2024-04-26T11:28:17.826827+0000","last_clean":"2024-04-26T11:28:17.826827+0000","last_became_active":"2024-04-26T11:27:19.278138+0000","last_became_peered":"2024-04-26T11:27:19.278138+0000","last_unstale":"2024-04-26T11:28:17.826827+0000","last_undegraded":"2024-04-26T11:28:17.826827+0000","last_fullsized":"2024-04-26T11:28:17.826827+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:17.038527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.829918+0000","last_change":"2024-04-26T11:26:29.026905+0000","last_active":"2024-04-26T11:28:17.829918+0000","last_peered":"2024-04-26T11:28:17.829918+0000","last_clean":"2024-04-26T11:28:17.829918+0000","last_became_active":"2024-04-26T11:26:29.026539+0000","last_became_peered":"2024-04-26T11:26:29.026539+0000","last_unstale":"2024-04-26T11:28:17.829918+0000","last_undegraded":"2024-04-26T11:28:17.829918+0000","last_fullsized":"2024-04-26T11:28:17.829918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:41:12.169466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T11:28:17.813412+0000","last_change":"2024-04-26T11:26:28.999953+0000","last_active":"2024-04-26T11:28:17.813412+0000","last_peered":"2024-04-26T11:28:17.813412+0000","last_clean":"2024-04-26T11:28:17.813412+0000","last_became_active":"2024-04-26T11:26:28.998344+0000","last_became_peered":"2024-04-26T11:26:28.998344+0000","last_unstale":"2024-04-26T11:28:17.813412+0000","last_undegraded":"2024-04-26T11:28:17.813412+0000","last_fullsized":"2024-04-26T11:28:17.813412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:30:53.269171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281068,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70041,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":727,"num_read_kb":438,"num_write":177,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":210123,"data_compressed":5472,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":479968,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645363,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32280,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712872,"statfs":{"total":95995035648,"available":95961980928,"internally_reserved":0,"allocated":1130496,"data_stored":1137880,"data_compressed":43361,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480204,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070527,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480124,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332163,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1072195,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495689,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":479593,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":545285,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480864,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":546635,"data_compressed":7611,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182491,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479904,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":480562,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1071347,"data_compressed":41537,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65899,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67249,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68009,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T11:28:53.885 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T11:28:53.885 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T11:28:53.886 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T11:28:53.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T11:28:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:53 smithi096 ceph-mon[32753]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:53 smithi194 ceph-mon[33017]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:54.388 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:54.793 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:54 smithi033 ceph-mon[27845]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:28:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:54 smithi096 ceph-mon[32753]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:28:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:54 smithi194 ceph-mon[33017]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:28:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:55 smithi033 ceph-mon[27845]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:55 smithi096 ceph-mon[32753]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:55 smithi194 ceph-mon[33017]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:56.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 -- 172.21.15.33:0/1206387925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 msgr2=0x7f377c103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:56.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 --2- 172.21.15.33:0/1206387925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 0x7f377c103950 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3764009920 tx=0x7f3764009c30 comp rx=0 tx=0).stop 2024-04-26T11:28:56.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 -- 172.21.15.33:0/1206387925 shutdown_connections 2024-04-26T11:28:56.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 --2- 172.21.15.33:0/1206387925 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f377c103e90 0x7f377c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 --2- 172.21.15.33:0/1206387925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f377c1083b0 0x7f377c108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 --2- 172.21.15.33:0/1206387925 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 0x7f377c103950 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 -- 172.21.15.33:0/1206387925 >> 172.21.15.33:0/1206387925 conn(0x7f377c07a580 msgr2=0x7f377c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:56.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 -- 172.21.15.33:0/1206387925 shutdown_connections 2024-04-26T11:28:56.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.624+0000 7f3781c00700 1 -- 172.21.15.33:0/1206387925 wait complete. 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.625+0000 7f3781c00700 1 Processor -- start 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.625+0000 7f3781c00700 1 -- start start 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f3781c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 0x7f377c075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f3781c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f377c103e90 0x7f377c075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f3781c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f377c1083b0 0x7f377c0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f3781c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f377c113640 con 0x7f377c1034d0 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f3781c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f377c1134c0 con 0x7f377c1083b0 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f3781c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f377c1137c0 con 0x7f377c103e90 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f377aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 0x7f377c075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:56.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f377b400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f377c1083b0 0x7f377c0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f377aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 0x7f377c075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54036/0 (socket says 172.21.15.33:54036) 2024-04-26T11:28:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f377aa00700 1 -- 172.21.15.33:0/393736990 learned_addr learned my addr 172.21.15.33:0/393736990 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:28:56.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.626+0000 7f3772000700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f377c103e90 0x7f377c075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f377b400700 1 -- 172.21.15.33:0/393736990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f377c103e90 msgr2=0x7f377c075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f377b400700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f377c103e90 0x7f377c075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f377b400700 1 -- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 msgr2=0x7f377c075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f377b400700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 0x7f377c075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f377b400700 1 -- 172.21.15.33:0/393736990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377c1a80a0 con 0x7f377c1083b0 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f377b400700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f377c1083b0 0x7f377c0761a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f377400efc0 tx=0x7f377400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f3773e00700 1 -- 172.21.15.33:0/393736990 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37740093a0 con 0x7f377c1083b0 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f3773e00700 1 -- 172.21.15.33:0/393736990 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377400c940 con 0x7f377c1083b0 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f377c1a8330 con 0x7f377c1083b0 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f3773e00700 1 -- 172.21.15.33:0/393736990 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3774004420 con 0x7f377c1083b0 2024-04-26T11:28:56.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.627+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f377c1a87f0 con 0x7f377c1083b0 2024-04-26T11:28:56.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.629+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f377c06a110 con 0x7f377c1083b0 2024-04-26T11:28:56.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.631+0000 7f3773e00700 1 -- 172.21.15.33:0/393736990 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f377401b030 con 0x7f377c1083b0 2024-04-26T11:28:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.632+0000 7f3773e00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f37600a3c40 0x7f37600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:28:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.632+0000 7f3773e00700 1 -- 172.21.15.33:0/393736990 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f37740cff40 con 0x7f377c1083b0 2024-04-26T11:28:56.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.632+0000 7f377aa00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f37600a3c40 0x7f37600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:28:56.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.633+0000 7f377aa00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f37600a3c40 0x7f37600a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f376c009cb0 tx=0x7f376c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:28:56.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.634+0000 7f3773e00700 1 -- 172.21.15.33:0/393736990 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3774022060 con 0x7f377c1083b0 2024-04-26T11:28:56.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.915+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f377c02e650 con 0x7f377c1083b0 2024-04-26T11:28:56.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.915+0000 7f3773e00700 1 -- 172.21.15.33:0/393736990 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3774010040 con 0x7f377c1083b0 2024-04-26T11:28:56.916 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:28:56.916 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T11:28:56.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f37600a3c40 msgr2=0x7f37600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:56.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f37600a3c40 0x7f37600a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f376c009cb0 tx=0x7f376c009310 comp rx=0 tx=0).stop 2024-04-26T11:28:56.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f377c1083b0 msgr2=0x7f377c0761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:28:56.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f377c1083b0 0x7f377c0761a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f377400efc0 tx=0x7f377400c3a0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 shutdown_connections 2024-04-26T11:28:56.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f37600a3c40 0x7f37600a6100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f377c1083b0 0x7f377c0761a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f377c1034d0 0x7f377c075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 --2- 172.21.15.33:0/393736990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f377c103e90 0x7f377c075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:28:56.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 >> 172.21.15.33:0/393736990 conn(0x7f377c07a580 msgr2=0x7f377c101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:28:56.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.920+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 shutdown_connections 2024-04-26T11:28:56.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:28:56.921+0000 7f3781c00700 1 -- 172.21.15.33:0/393736990 wait complete. 2024-04-26T11:28:57.677 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T11:28:57.677 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:28:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:57 smithi033 ceph-mon[27845]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:57 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/393736990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T11:28:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:57 smithi096 ceph-mon[32753]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:57 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/393736990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T11:28:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:57 smithi194 ceph-mon[33017]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:28:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:57 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/393736990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T11:28:58.153 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:28:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:28:58 smithi033 ceph-mon[27845]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:28:58 smithi194 ceph-mon[33017]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:28:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:28:58 smithi096 ceph-mon[32753]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:00.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 -- 172.21.15.33:0/2648790220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa84106000 msgr2=0x7faa84112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 --2- 172.21.15.33:0/2648790220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa84106000 0x7faa84112510 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009f90 tx=0x7faa7c00b330 comp rx=0 tx=0).stop 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 -- 172.21.15.33:0/2648790220 shutdown_connections 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 --2- 172.21.15.33:0/2648790220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa84106000 0x7faa84112510 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 --2- 172.21.15.33:0/2648790220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa8410b640 0x7faa8410ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 --2- 172.21.15.33:0/2648790220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 0x7faa84105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 -- 172.21.15.33:0/2648790220 >> 172.21.15.33:0/2648790220 conn(0x7faa84101130 msgr2=0x7faa84103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 -- 172.21.15.33:0/2648790220 shutdown_connections 2024-04-26T11:29:00.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.435+0000 7faa8bc00700 1 -- 172.21.15.33:0/2648790220 wait complete. 2024-04-26T11:29:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.436+0000 7faa8bc00700 1 Processor -- start 2024-04-26T11:29:00.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa8bc00700 1 -- start start 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa8bc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 0x7faa841a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa8bc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa84106000 0x7faa841a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa8bc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa8410b640 0x7faa8419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa8bc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faa841158d0 con 0x7faa84105640 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa8bc00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa84115750 con 0x7faa84106000 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa8bc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faa84115a50 con 0x7faa8410b640 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa89200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 0x7faa841a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.437+0000 7faa89200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 0x7faa841a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34788/0 (socket says 172.21.15.33:34788) 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa89200700 1 -- 172.21.15.33:0/116178284 learned_addr learned my addr 172.21.15.33:0/116178284 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa83e00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa84106000 0x7faa841a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa89c00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa8410b640 0x7faa8419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:00.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa83e00700 1 -- 172.21.15.33:0/116178284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa8410b640 msgr2=0x7faa8419baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa83e00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa8410b640 0x7faa8419baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa83e00700 1 -- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 msgr2=0x7faa841a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa83e00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 0x7faa841a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa83e00700 1 -- 172.21.15.33:0/116178284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8419c250 con 0x7faa84106000 2024-04-26T11:29:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa89200700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 0x7faa841a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:00.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.438+0000 7faa83e00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa84106000 0x7faa841a1e20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa7000d950 tx=0x7faa7000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.439+0000 7faa81600700 1 -- 172.21.15.33:0/116178284 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70016070 con 0x7faa84106000 2024-04-26T11:29:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.439+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8419c540 con 0x7faa84106000 2024-04-26T11:29:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.439+0000 7faa81600700 1 -- 172.21.15.33:0/116178284 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa7000fd40 con 0x7faa84106000 2024-04-26T11:29:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.439+0000 7faa81600700 1 -- 172.21.15.33:0/116178284 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7000e3b0 con 0x7faa84106000 2024-04-26T11:29:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.439+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa841a87b0 con 0x7faa84106000 2024-04-26T11:29:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.440+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8406a110 con 0x7faa84106000 2024-04-26T11:29:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.442+0000 7faa81600700 1 -- 172.21.15.33:0/116178284 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faa70019070 con 0x7faa84106000 2024-04-26T11:29:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.444+0000 7faa81600700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faa6c0a3c40 0x7faa6c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.444+0000 7faa81600700 1 -- 172.21.15.33:0/116178284 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faa700d0100 con 0x7faa84106000 2024-04-26T11:29:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.444+0000 7faa89200700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faa6c0a3c40 0x7faa6c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.444+0000 7faa89200700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faa6c0a3c40 0x7faa6c0a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa84103a60 tx=0x7faa78009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:00.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.446+0000 7faa81600700 1 -- 172.21.15.33:0/116178284 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa7009caa0 con 0x7faa84106000 2024-04-26T11:29:00.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.728+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa8402e490 con 0x7faa84106000 2024-04-26T11:29:00.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.730+0000 7faa81600700 1 -- 172.21.15.33:0/116178284 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7faa7009c3a0 con 0x7faa84106000 2024-04-26T11:29:00.731 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:00.731 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:26:56.732811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:29:00.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faa6c0a3c40 msgr2=0x7faa6c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:00.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faa6c0a3c40 0x7faa6c0a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa84103a60 tx=0x7faa78009310 comp rx=0 tx=0).stop 2024-04-26T11:29:00.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa84106000 msgr2=0x7faa841a1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:00.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa84106000 0x7faa841a1e20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa7000d950 tx=0x7faa7000dc60 comp rx=0 tx=0).stop 2024-04-26T11:29:00.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 shutdown_connections 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faa6c0a3c40 0x7faa6c0a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa8410b640 0x7faa8419baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa84105640 0x7faa841a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 --2- 172.21.15.33:0/116178284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa84106000 0x7faa841a1e20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 >> 172.21.15.33:0/116178284 conn(0x7faa84101130 msgr2=0x7faa84101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 shutdown_connections 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:00.733+0000 7faa8bc00700 1 -- 172.21.15.33:0/116178284 wait complete. 2024-04-26T11:29:00.735 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-04-26T11:29:01.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:00 smithi033 ceph-mon[27845]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:01.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:00 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/116178284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:00 smithi096 ceph-mon[32753]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:00 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/116178284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:00 smithi194 ceph-mon[33017]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:00 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/116178284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:01.550 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:29:02.033 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:02 smithi096 ceph-mon[32753]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:03.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:02 smithi033 ceph-mon[27845]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:02 smithi194 ceph-mon[33017]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.366+0000 7fba4bc00700 1 -- 172.21.15.33:0/644693621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4410fed0 msgr2=0x7fba441122c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.366+0000 7fba4bc00700 1 --2- 172.21.15.33:0/644693621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4410fed0 0x7fba441122c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00b0a0 tx=0x7fba3c00b3b0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.367+0000 7fba4bc00700 1 -- 172.21.15.33:0/644693621 shutdown_connections 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.367+0000 7fba4bc00700 1 --2- 172.21.15.33:0/644693621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4410fed0 0x7fba441122c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.367+0000 7fba4bc00700 1 --2- 172.21.15.33:0/644693621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba4406b130 0x7fba4406b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.367+0000 7fba4bc00700 1 --2- 172.21.15.33:0/644693621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba4406ba50 0x7fba4410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.367+0000 7fba4bc00700 1 -- 172.21.15.33:0/644693621 >> 172.21.15.33:0/644693621 conn(0x7fba440792d0 msgr2=0x7fba440796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.367+0000 7fba4bc00700 1 -- 172.21.15.33:0/644693621 shutdown_connections 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.367+0000 7fba4bc00700 1 -- 172.21.15.33:0/644693621 wait complete. 2024-04-26T11:29:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.368+0000 7fba4bc00700 1 Processor -- start 2024-04-26T11:29:04.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.368+0000 7fba4bc00700 1 -- start start 2024-04-26T11:29:04.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.368+0000 7fba4bc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4406b130 0x7fba4419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:04.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.368+0000 7fba4bc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba4406ba50 0x7fba4419c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:04.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba4bc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba4410fed0 0x7fba441a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:04.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba4bc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba44115830 con 0x7fba4406b130 2024-04-26T11:29:04.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba4bc00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fba441156b0 con 0x7fba4410fed0 2024-04-26T11:29:04.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba4bc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fba441159b0 con 0x7fba4406ba50 2024-04-26T11:29:04.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4406b130 0x7fba4419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba43e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba4406ba50 0x7fba4419c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba4410fed0 0x7fba441a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba43e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba4406ba50 0x7fba4419c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:51748/0 (socket says 172.21.15.33:51748) 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba43e00700 1 -- 172.21.15.33:0/4178942349 learned_addr learned my addr 172.21.15.33:0/4178942349 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4406b130 0x7fba4419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34800/0 (socket says 172.21.15.33:34800) 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49c00700 1 -- 172.21.15.33:0/4178942349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba4406ba50 msgr2=0x7fba4419c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49c00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba4406ba50 0x7fba4419c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49c00700 1 -- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4406b130 msgr2=0x7fba4419bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49c00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4406b130 0x7fba4419bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.369+0000 7fba49c00700 1 -- 172.21.15.33:0/4178942349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba441a0d60 con 0x7fba4410fed0 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.370+0000 7fba49c00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba4410fed0 0x7fba441a0620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00b070 tx=0x7fba3c004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.370+0000 7fba41600700 1 -- 172.21.15.33:0/4178942349 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c021070 con 0x7fba4410fed0 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.370+0000 7fba41600700 1 -- 172.21.15.33:0/4178942349 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba3c007e30 con 0x7fba4410fed0 2024-04-26T11:29:04.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.370+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba441a0ff0 con 0x7fba4410fed0 2024-04-26T11:29:04.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.370+0000 7fba41600700 1 -- 172.21.15.33:0/4178942349 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0045f0 con 0x7fba4410fed0 2024-04-26T11:29:04.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.370+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba441a88d0 con 0x7fba4410fed0 2024-04-26T11:29:04.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.371+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba44102940 con 0x7fba4410fed0 2024-04-26T11:29:04.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.373+0000 7fba41600700 1 -- 172.21.15.33:0/4178942349 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fba3c007980 con 0x7fba4410fed0 2024-04-26T11:29:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.375+0000 7fba41600700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fba2c0a3c40 0x7fba2c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.375+0000 7fba41600700 1 -- 172.21.15.33:0/4178942349 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fba3c0d1130 con 0x7fba4410fed0 2024-04-26T11:29:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.375+0000 7fba49200700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fba2c0a3c40 0x7fba2c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.376+0000 7fba49200700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fba2c0a3c40 0x7fba2c0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba38009c60 tx=0x7fba38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.377+0000 7fba41600700 1 -- 172.21.15.33:0/4178942349 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba3c09db60 con 0x7fba4410fed0 2024-04-26T11:29:04.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.660+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba4402e490 con 0x7fba4410fed0 2024-04-26T11:29:04.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.662+0000 7fba41600700 1 -- 172.21.15.33:0/4178942349 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fba3c09d460 con 0x7fba4410fed0 2024-04-26T11:29:04.663 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:04.663 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:26:56.732811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:29:04.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fba2c0a3c40 msgr2=0x7fba2c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:04.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fba2c0a3c40 0x7fba2c0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba38009c60 tx=0x7fba38009260 comp rx=0 tx=0).stop 2024-04-26T11:29:04.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba4410fed0 msgr2=0x7fba441a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:04.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba4410fed0 0x7fba441a0620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00b070 tx=0x7fba3c004c80 comp rx=0 tx=0).stop 2024-04-26T11:29:04.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 shutdown_connections 2024-04-26T11:29:04.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fba2c0a3c40 0x7fba2c0a6100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba4410fed0 0x7fba441a0620 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4406b130 0x7fba4419bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 --2- 172.21.15.33:0/4178942349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba4406ba50 0x7fba4419c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:04.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 >> 172.21.15.33:0/4178942349 conn(0x7fba440792d0 msgr2=0x7fba441113c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:04.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 shutdown_connections 2024-04-26T11:29:04.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:04.665+0000 7fba4bc00700 1 -- 172.21.15.33:0/4178942349 wait complete. 2024-04-26T11:29:04.667 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-04-26T11:29:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:04 smithi194 ceph-mon[33017]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:04 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/4178942349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:04 smithi033 ceph-mon[27845]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:04 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/4178942349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:04 smithi096 ceph-mon[32753]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:04 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/4178942349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:05.648 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:29:06.105 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:06 smithi033 ceph-mon[27845]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:06 smithi194 ceph-mon[33017]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:06 smithi096 ceph-mon[32753]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:08.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 -- 172.21.15.33:0/955104483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc4109a50 msgr2=0x7fedc4109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:08.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 --2- 172.21.15.33:0/955104483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc4109a50 0x7fedc4109ed0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fedb00098f0 tx=0x7fedb0009c00 comp rx=0 tx=0).stop 2024-04-26T11:29:08.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 -- 172.21.15.33:0/955104483 shutdown_connections 2024-04-26T11:29:08.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 --2- 172.21.15.33:0/955104483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedc410a410 0x7fedc4116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 --2- 172.21.15.33:0/955104483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedc410fa50 0x7fedc410fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 --2- 172.21.15.33:0/955104483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc4109a50 0x7fedc4109ed0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 -- 172.21.15.33:0/955104483 >> 172.21.15.33:0/955104483 conn(0x7fedc4076bf0 msgr2=0x7fedc4108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:08.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 -- 172.21.15.33:0/955104483 shutdown_connections 2024-04-26T11:29:08.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.347+0000 7fedcaa00700 1 -- 172.21.15.33:0/955104483 wait complete. 2024-04-26T11:29:08.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.348+0000 7fedcaa00700 1 Processor -- start 2024-04-26T11:29:08.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.349+0000 7fedcaa00700 1 -- start start 2024-04-26T11:29:08.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.349+0000 7fedcaa00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedc4109a50 0x7fedc41a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:08.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.349+0000 7fedcaa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc410a410 0x7fedc41a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.349+0000 7fedcaa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedc410fa50 0x7fedc41a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.349+0000 7fedcaa00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fedc4119ce0 con 0x7fedc410fa50 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.349+0000 7fedcaa00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fedc4119b60 con 0x7fedc4109a50 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.349+0000 7fedcaa00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fedc4119e60 con 0x7fedc410a410 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc410a410 0x7fedc41a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc8a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedc410fa50 0x7fedc41a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedc4109a50 0x7fedc41a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc410a410 0x7fedc41a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:51766/0 (socket says 172.21.15.33:51766) 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 -- 172.21.15.33:0/634532232 learned_addr learned my addr 172.21.15.33:0/634532232 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 -- 172.21.15.33:0/634532232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedc4109a50 msgr2=0x7fedc41a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:08.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedc4109a50 0x7fedc41a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 -- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedc410fa50 msgr2=0x7fedc41a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedc410fa50 0x7fedc41a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 -- 172.21.15.33:0/634532232 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc41a51d0 con 0x7fedc410a410 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.350+0000 7fedc3400700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc410a410 0x7fedc41a0750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009230 tx=0x7fedb000fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.351+0000 7fedc0c00700 1 -- 172.21.15.33:0/634532232 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb00050c0 con 0x7fedc410a410 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.351+0000 7fedc0c00700 1 -- 172.21.15.33:0/634532232 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedb000b560 con 0x7fedc410a410 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.351+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc41a5460 con 0x7fedc410a410 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.351+0000 7fedc0c00700 1 -- 172.21.15.33:0/634532232 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0003a00 con 0x7fedc410a410 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.351+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc41acd40 con 0x7fedc410a410 2024-04-26T11:29:08.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.352+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc406a110 con 0x7fedc410a410 2024-04-26T11:29:08.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.354+0000 7fedc0c00700 1 -- 172.21.15.33:0/634532232 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fedb000b710 con 0x7fedc410a410 2024-04-26T11:29:08.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.356+0000 7fedc0c00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7feda80a3cc0 0x7feda80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:08.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.356+0000 7fedc0c00700 1 -- 172.21.15.33:0/634532232 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fedb00d12c0 con 0x7fedc410a410 2024-04-26T11:29:08.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.356+0000 7fedc3e00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7feda80a3cc0 0x7feda80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:08.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.357+0000 7fedc3e00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7feda80a3cc0 0x7feda80a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009be0 tx=0x7fedb4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:08.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.357+0000 7fedc0c00700 1 -- 172.21.15.33:0/634532232 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedb0013030 con 0x7fedc410a410 2024-04-26T11:29:08.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.593+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fedc41a1520 con 0x7fedc410a410 2024-04-26T11:29:08.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.595+0000 7fedc0c00700 1 -- 172.21.15.33:0/634532232 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22592 (secure 0 0 0) 0x7fedb009dcf0 con 0x7fedc410a410 2024-04-26T11:29:08.597 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:08.597 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:27:19.230568+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:29:08.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7feda80a3cc0 msgr2=0x7feda80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:08.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7feda80a3cc0 0x7feda80a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009be0 tx=0x7fedb4009260 comp rx=0 tx=0).stop 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc410a410 msgr2=0x7fedc41a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc410a410 0x7fedc41a0750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009230 tx=0x7fedb000fa70 comp rx=0 tx=0).stop 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 shutdown_connections 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7feda80a3cc0 0x7feda80a6180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fedc410fa50 0x7fedc41a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedc4109a50 0x7fedc41a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 --2- 172.21.15.33:0/634532232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedc410a410 0x7fedc41a0750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.598+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 >> 172.21.15.33:0/634532232 conn(0x7fedc4076bf0 msgr2=0x7fedc4077250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.599+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 shutdown_connections 2024-04-26T11:29:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:08.599+0000 7fedcaa00700 1 -- 172.21.15.33:0/634532232 wait complete. 2024-04-26T11:29:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:08 smithi194 ceph-mon[33017]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:08 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/634532232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:29:09.417 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:08 smithi033 ceph-mon[27845]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:09.417 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:08 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/634532232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:29:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:08 smithi096 ceph-mon[32753]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:08 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/634532232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:29:09.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:29:09.976 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:11.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:10 smithi096 ceph-mon[32753]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:10 smithi194 ceph-mon[33017]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:11.401 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:10 smithi033 ceph-mon[27845]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:12.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 -- 172.21.15.33:0/3009947714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 msgr2=0x7fdebc113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 --2- 172.21.15.33:0/3009947714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 0x7fdebc113e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009990 tx=0x7fdea8009ca0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 -- 172.21.15.33:0/3009947714 shutdown_connections 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 --2- 172.21.15.33:0/3009947714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdebc114340 0x7fdebc116730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 --2- 172.21.15.33:0/3009947714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc0757c0 0x7fdebc075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 --2- 172.21.15.33:0/3009947714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 0x7fdebc113e00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 -- 172.21.15.33:0/3009947714 >> 172.21.15.33:0/3009947714 conn(0x7fdebc0fee40 msgr2=0x7fdebc101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 -- 172.21.15.33:0/3009947714 shutdown_connections 2024-04-26T11:29:12.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.233+0000 7fdebbe00700 1 -- 172.21.15.33:0/3009947714 wait complete. 2024-04-26T11:29:12.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.234+0000 7fdebbe00700 1 Processor -- start 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.234+0000 7fdebbe00700 1 -- start start 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebbe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdebc0757c0 0x7fdebc1a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebbe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 0x7fdebc1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebbe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc114340 0x7fdebc1a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebbe00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc119b10 con 0x7fdebc0760e0 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebbe00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc119990 con 0x7fdebc0757c0 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebbe00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc119c90 con 0x7fdebc114340 2024-04-26T11:29:12.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebb400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc114340 0x7fdebc1a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebb400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc114340 0x7fdebc1a48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:40050/0 (socket says 172.21.15.33:40050) 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebb400700 1 -- 172.21.15.33:0/4065388186 learned_addr learned my addr 172.21.15.33:0/4065388186 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdebaa00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdebc0757c0 0x7fdebc1a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.235+0000 7fdeba000700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 0x7fdebc1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebb400700 1 -- 172.21.15.33:0/4065388186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdebc0757c0 msgr2=0x7fdebc1a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebb400700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdebc0757c0 0x7fdebc1a0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebb400700 1 -- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 msgr2=0x7fdebc1a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebb400700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 0x7fdebc1a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebb400700 1 -- 172.21.15.33:0/4065388186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc1a5020 con 0x7fdebc114340 2024-04-26T11:29:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebaa00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdebc0757c0 0x7fdebc1a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebb400700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc114340 0x7fdebc1a48e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400ef90 tx=0x7fdeb400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdeb3400700 1 -- 172.21.15.33:0/4065388186 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb40092e0 con 0x7fdebc114340 2024-04-26T11:29:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc1a52b0 con 0x7fdebc114340 2024-04-26T11:29:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdeb3400700 1 -- 172.21.15.33:0/4065388186 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdeb4004bb0 con 0x7fdebc114340 2024-04-26T11:29:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.236+0000 7fdeb3400700 1 -- 172.21.15.33:0/4065388186 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4004280 con 0x7fdebc114340 2024-04-26T11:29:12.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.237+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc1acb90 con 0x7fdebc114340 2024-04-26T11:29:12.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.238+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdebc06a110 con 0x7fdebc114340 2024-04-26T11:29:12.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.240+0000 7fdeb3400700 1 -- 172.21.15.33:0/4065388186 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdeb401b030 con 0x7fdebc114340 2024-04-26T11:29:12.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.241+0000 7fdeb3400700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdea00a3c40 0x7fdea00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:12.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.241+0000 7fdeb3400700 1 -- 172.21.15.33:0/4065388186 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdeb40cff00 con 0x7fdebc114340 2024-04-26T11:29:12.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.242+0000 7fdebaa00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdea00a3c40 0x7fdea00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:12.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.242+0000 7fdebaa00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdea00a3c40 0x7fdea00a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdeac005950 tx=0x7fdeac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:12.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.246+0000 7fdeb3400700 1 -- 172.21.15.33:0/4065388186 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdeb4022020 con 0x7fdebc114340 2024-04-26T11:29:12.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.528+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdebc02e490 con 0x7fdebc114340 2024-04-26T11:29:12.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.529+0000 7fdeb3400700 1 -- 172.21.15.33:0/4065388186 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fdeb4010040 con 0x7fdebc114340 2024-04-26T11:29:12.561 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:12.561 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:26:56.732811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:29:12.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.532+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdea00a3c40 msgr2=0x7fdea00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:12.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.532+0000 7fdebbe00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdea00a3c40 0x7fdea00a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdeac005950 tx=0x7fdeac00b410 comp rx=0 tx=0).stop 2024-04-26T11:29:12.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.532+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc114340 msgr2=0x7fdebc1a48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:12.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.532+0000 7fdebbe00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc114340 0x7fdebc1a48e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400ef90 tx=0x7fdeb400c3a0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 shutdown_connections 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fdea00a3c40 0x7fdea00a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdebc114340 0x7fdebc1a48e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdebc0757c0 0x7fdebc1a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 --2- 172.21.15.33:0/4065388186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdebc0760e0 0x7fdebc1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 >> 172.21.15.33:0/4065388186 conn(0x7fdebc0fee40 msgr2=0x7fdebc100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 shutdown_connections 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:12.533+0000 7fdebbe00700 1 -- 172.21.15.33:0/4065388186 wait complete. 2024-04-26T11:29:12.563 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-04-26T11:29:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:13 smithi033 ceph-mon[27845]: pgmap v407: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:13 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/4065388186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:13.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T11:26:32.073203+0000', 'modified': '2024-04-26T11:26:56.732811+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24527, 'mds_1': 24488, 'mds_2': 24484}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6835/1403509498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 1403509498}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 1403509498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6835/1002586260', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6834', 'nonce': 1002586260}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1002586260}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6837/1265312544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 1265312544}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 1265312544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.194:6839/517247009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 517247009}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 517247009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.96:6837/823522377', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 823522377}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 823522377}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.96:6839/1966519759', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 1966519759}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 1966519759}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24488, 24527, 24484]} 2024-04-26T11:29:13.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T11:29:13.330 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T11:29:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:13 smithi194 ceph-mon[33017]: pgmap v407: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:13 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/4065388186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:13 smithi096 ceph-mon[32753]: pgmap v407: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:13 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/4065388186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:15 smithi033 ceph-mon[27845]: pgmap v408: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:15 smithi194 ceph-mon[33017]: pgmap v408: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:15 smithi096 ceph-mon[32753]: pgmap v408: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.156+0000 7fe0b1000700 1 -- 172.21.15.33:0/2863276016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06bad0 msgr2=0x7fe0ac1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.156+0000 7fe0b1000700 1 --2- 172.21.15.33:0/2863276016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac1102e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400b0a0 tx=0x7fe0a400b3b0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.157+0000 7fe0b1000700 1 -- 172.21.15.33:0/2863276016 shutdown_connections 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.157+0000 7fe0b1000700 1 --2- 172.21.15.33:0/2863276016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac1102e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.157+0000 7fe0b1000700 1 --2- 172.21.15.33:0/2863276016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0ac1094a0 0x7fe0ac109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.157+0000 7fe0b1000700 1 --2- 172.21.15.33:0/2863276016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06b130 0x7fe0ac06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.157+0000 7fe0b1000700 1 -- 172.21.15.33:0/2863276016 >> 172.21.15.33:0/2863276016 conn(0x7fe0ac0792d0 msgr2=0x7fe0ac0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.157+0000 7fe0b1000700 1 -- 172.21.15.33:0/2863276016 shutdown_connections 2024-04-26T11:29:16.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.157+0000 7fe0b1000700 1 -- 172.21.15.33:0/2863276016 wait complete. 2024-04-26T11:29:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.158+0000 7fe0b1000700 1 Processor -- start 2024-04-26T11:29:16.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.158+0000 7fe0b1000700 1 -- start start 2024-04-26T11:29:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0b1000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06b130 0x7fe0ac1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0b1000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0b1000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0ac1094a0 0x7fe0ac102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0b1000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac1135a0 con 0x7fe0ac06b130 2024-04-26T11:29:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0b1000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac113420 con 0x7fe0ac06bad0 2024-04-26T11:29:16.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0b1000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac113720 con 0x7fe0ac1094a0 2024-04-26T11:29:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06b130 0x7fe0ac1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0a3e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06b130 0x7fe0ac1023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35040/0 (socket says 172.21.15.33:35040) 2024-04-26T11:29:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1428440984 learned_addr learned my addr 172.21.15.33:0/1428440984 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.159+0000 7fe0a3e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41270/0 (socket says 172.21.15.33:41270) 2024-04-26T11:29:16.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0aa800700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0ac1094a0 0x7fe0ac102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1428440984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0ac1094a0 msgr2=0x7fe0ac102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0ac1094a0 0x7fe0ac102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1428440984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06bad0 msgr2=0x7fe0ac102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1428440984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ac1a7fb0 con 0x7fe0ac06b130 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06b130 0x7fe0ac1023f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00e9a0 tx=0x7fe09c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:16.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.160+0000 7fe0a2a00700 1 -- 172.21.15.33:0/1428440984 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c00ca10 con 0x7fe0ac06b130 2024-04-26T11:29:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.161+0000 7fe0a2a00700 1 -- 172.21.15.33:0/1428440984 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe09c00cbb0 con 0x7fe0ac06b130 2024-04-26T11:29:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.161+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ac1a82a0 con 0x7fe0ac06b130 2024-04-26T11:29:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.161+0000 7fe0a2a00700 1 -- 172.21.15.33:0/1428440984 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c018680 con 0x7fe0ac06b130 2024-04-26T11:29:16.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.161+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0ac1a87e0 con 0x7fe0ac06b130 2024-04-26T11:29:16.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.164+0000 7fe0a2a00700 1 -- 172.21.15.33:0/1428440984 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe09c010430 con 0x7fe0ac06b130 2024-04-26T11:29:16.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.165+0000 7fe0a2a00700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0900a3c40 0x7fe0900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.165+0000 7fe0bb988b80 1 Processor -- start 2024-04-26T11:29:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0a2a00700 1 -- 172.21.15.33:0/1428440984 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe09c0d0520 con 0x7fe0ac06b130 2024-04-26T11:29:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0bb988b80 1 -- start start 2024-04-26T11:29:16.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0900a3c40 0x7fe0900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c3080 0x5586329c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558632915840 con 0x5586329c73b0 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5586328b4160 con 0x5586329c7cd0 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558632911800 con 0x5586329c3080 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.166+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35042/0 (socket says 172.21.15.33:35042) 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 -- 172.21.15.33:0/3277475456 learned_addr learned my addr 172.21.15.33:0/3277475456 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0900a3c40 0x7fe0900a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac108aa0 tx=0x7fe094009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0aa800700 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c3080 0x5586329c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 -- 172.21.15.33:0/3277475456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c3080 msgr2=0x5586329c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c3080 0x5586329c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 -- 172.21.15.33:0/3277475456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 msgr2=0x5586329c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 -- 172.21.15.33:0/3277475456 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586328bc5e0 con 0x5586329c73b0 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0aa800700 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c3080 0x5586329c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.167+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c7790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009230 tx=0x7fe09c09cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbf558ed61ac0ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.168+0000 7fe09b400700 1 -- 172.21.15.33:0/3277475456 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0a5430 con 0x5586329c73b0 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.168+0000 7fe09b400700 1 -- 172.21.15.33:0/3277475456 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe09c00cd60 con 0x5586329c73b0 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.168+0000 7fe09b400700 1 -- 172.21.15.33:0/3277475456 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0a4fa0 con 0x5586329c73b0 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.168+0000 7fe0bb988b80 1 -- 172.21.15.33:0/3277475456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 msgr2=0x5586329c7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.168+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c7790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009230 tx=0x7fe09c09cf90 comp rx=0 tx=0).stop 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 -- 172.21.15.33:0/3277475456 shutdown_connections 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c3080 0x5586329c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c7790 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/3277475456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 -- 172.21.15.33:0/3277475456 >> 172.21.15.33:0/3277475456 conn(0x5586328f02a0 msgr2=0x5586329bd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 -- 172.21.15.33:0/3277475456 shutdown_connections 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 -- 172.21.15.33:0/3277475456 wait complete. 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.169+0000 7fe0bb988b80 1 -- start start 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 1 Processor -- start 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5586329cf810 con 0x5586329c73b0 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5586328b03b0 con 0x5586329c3080 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5586329cf990 con 0x5586329c7cd0 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0bb988b80 10 client.0 _ll_register_callbacks cb 0x5586328a9330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41298/0 (socket says 172.21.15.33:41298) 2024-04-26T11:29:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1252226958 learned_addr learned my addr 172.21.15.33:0/1252226958 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.170+0000 7fe0aa800700 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 msgr2=0x5586329cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1252226958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 msgr2=0x5586329cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1252226958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558632862880 con 0x5586329c3080 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329cea60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0a4f00 tx=0x7fe09c09c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33d5ddd4ec56d70a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe08fe00700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe08fe00700 1 -- 172.21.15.33:0/1252226958 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0ba060 con 0x5586329c3080 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe08fe00700 1 -- 172.21.15.33:0/1252226958 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe09c0a4fa0 con 0x5586329c3080 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.171+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe09c00cd60 con 0x5586329c3080 2024-04-26T11:29:16.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe08fe00700 1 -- 172.21.15.33:0/1252226958 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe09c0a51a0 con 0x5586329c3080 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 10 client.24869 fetch_fsmap learned FSMap version 25 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5586329da030 con 0x5586329c3080 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe08fe00700 1 -- 172.21.15.33:0/1252226958 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe09c00cd60 con 0x5586329c3080 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe08fe00700 1 -- 172.21.15.33:0/1252226958 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fe09c09daf0 con 0x5586329c3080 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 10 client.24869 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 10 client.24869 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 20 client.24869 populate_metadata read hostname 'smithi033' 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x5586329db380 0x5586329dd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 4 client.24869 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.172+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 --> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558632923220 con 0x5586329db380 2024-04-26T11:29:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.173+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x5586329db380 0x5586329dd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.173+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x5586329db380 0x5586329dd560 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d97c77d40f4f8edb server_cookie=40c602d38cb054ad in_seq=0 out_seq=0 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.173+0000 7fe08fe00700 10 client.24869 ms_handle_connect on v2:172.21.15.194:6838/517247009 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe08fe00700 1 -- 172.21.15.33:0/1252226958 <== mds.2 v2:172.21.15.194:6838/517247009 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558632923220 con 0x5586329db380 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe08fe00700 10 client.24869 handle_command_reply: tid=0 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 client.24869 shutdown 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 msgr2=0x5586329cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329cea60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0a4f00 tx=0x7fe09c09c620 comp rx=0 tx=0).stop 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 shutdown_connections 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x5586329db380 0x5586329dd560 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329cea60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1252226958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe08fe00700 0 client.24869 ms_handle_reset on v2:172.21.15.194:6838/517247009 2024-04-26T11:29:16.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.174+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 >> 172.21.15.33:0/1252226958 conn(0x5586328f02a0 msgr2=0x5586329b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:16.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.175+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 shutdown_connections 2024-04-26T11:29:16.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.175+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1252226958 wait complete. 2024-04-26T11:29:16.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.175+0000 7fe0bb988b80 20 client.24869 trim_cache size 0 max 16384 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.197+0000 7fe0bb988b80 1 Processor -- start 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0bb988b80 1 -- start start 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c3080 0x5586329c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c73b0 0x5586329c9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5586329db8a0 con 0x5586329c3080 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5586329dbf70 con 0x5586329c7cd0 2024-04-26T11:29:16.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5586329dc640 con 0x5586329c73b0 2024-04-26T11:29:16.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c3080 0x5586329c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0aa800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0aa800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41302/0 (socket says 172.21.15.33:41302) 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c3080 0x5586329c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35068/0 (socket says 172.21.15.33:35068) 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0aa800700 1 -- 172.21.15.33:0/638519464 learned_addr learned my addr 172.21.15.33:0/638519464 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.198+0000 7fe0a3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c73b0 0x5586329c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0aa800700 1 -- 172.21.15.33:0/638519464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c73b0 msgr2=0x5586329c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0aa800700 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c73b0 0x5586329c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0aa800700 1 -- 172.21.15.33:0/638519464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c3080 msgr2=0x5586329c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0aa800700 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c3080 0x5586329c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0aa800700 1 -- 172.21.15.33:0/638519464 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586329be800 con 0x5586329c7cd0 2024-04-26T11:29:16.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c73b0 0x5586329c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:16.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c3080 0x5586329c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:16.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.199+0000 7fe0aa800700 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c1b30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4009520 tx=0x7fe0a4009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6e9399877e71c84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:16.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.200+0000 7fe09aa00700 1 -- 172.21.15.33:0/638519464 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400bb30 con 0x5586329c7cd0 2024-04-26T11:29:16.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.200+0000 7fe09aa00700 1 -- 172.21.15.33:0/638519464 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0a400bcd0 con 0x5586329c7cd0 2024-04-26T11:29:16.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.200+0000 7fe09aa00700 1 -- 172.21.15.33:0/638519464 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4007e20 con 0x5586329c7cd0 2024-04-26T11:29:16.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.200+0000 7fe0bb988b80 1 -- 172.21.15.33:0/638519464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 msgr2=0x5586329c1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.200+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c1b30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4009520 tx=0x7fe0a4009550 comp rx=0 tx=0).stop 2024-04-26T11:29:16.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 -- 172.21.15.33:0/638519464 shutdown_connections 2024-04-26T11:29:16.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c7cd0 0x5586329c1b30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c3080 0x5586329c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/638519464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c73b0 0x5586329c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 -- 172.21.15.33:0/638519464 >> 172.21.15.33:0/638519464 conn(0x5586328f02a0 msgr2=0x5586329bd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:16.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 -- 172.21.15.33:0/638519464 shutdown_connections 2024-04-26T11:29:16.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 -- 172.21.15.33:0/638519464 wait complete. 2024-04-26T11:29:16.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.201+0000 7fe0bb988b80 1 -- start start 2024-04-26T11:29:16.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 1 Processor -- start 2024-04-26T11:29:16.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329c9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5586329dd3e0 con 0x5586329c73b0 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5586329dcd10 con 0x5586329c3080 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5586329ddab0 con 0x5586329c7cd0 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0bb988b80 10 client.0 _ll_register_callbacks cb 0x5586328a9330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0a9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41308/0 (socket says 172.21.15.33:41308) 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1171437962 learned_addr learned my addr 172.21.15.33:0/1171437962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.202+0000 7fe0aa800700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 msgr2=0x5586329c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1171437962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 msgr2=0x5586329c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0a9e00700 1 -- 172.21.15.33:0/1171437962 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586329e4b80 con 0x5586329c3080 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0aa800700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0a9e00700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329c8a10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe09c000c00 tx=0x7fe09c0092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee0b852db47e8df6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe08fe00700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe08fe00700 1 -- 172.21.15.33:0/1171437962 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0ba040 con 0x5586329c3080 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe08fe00700 1 -- 172.21.15.33:0/1171437962 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe09c098510 con 0x5586329c3080 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a400bcd0 con 0x5586329c3080 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe08fe00700 1 -- 172.21.15.33:0/1171437962 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe09c098a60 con 0x5586329c3080 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0bb988b80 10 client.24875 fetch_fsmap learned FSMap version 25 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.203+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558632a43510 con 0x5586329c3080 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe08fe00700 1 -- 172.21.15.33:0/1171437962 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0a400bcd0 con 0x5586329c3080 2024-04-26T11:29:16.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe08fe00700 1 -- 172.21.15.33:0/1171437962 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fe09c09daf0 con 0x5586329c3080 2024-04-26T11:29:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe0bb988b80 10 client.24875 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:29:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe0bb988b80 10 client.24875 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T11:29:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe0bb988b80 20 client.24875 populate_metadata read hostname 'smithi033' 2024-04-26T11:29:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558632a47f10 0x558632a46c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe0bb988b80 4 client.24875 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:29:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 --> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] -- command(tid 0: {"prefix": "status"}) -- 0x55863291d2c0 con 0x558632a47f10 2024-04-26T11:29:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.204+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558632a47f10 0x558632a46c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.205+0000 7fe0a3e00700 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558632a47f10 0x558632a46c20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=721e0b60dc83aa57 server_cookie=794fcda6674f6a4f in_seq=0 out_seq=0 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.205+0000 7fe08fe00700 10 client.24875 ms_handle_connect on v2:172.21.15.194:6838/517247009 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.205+0000 7fe08fe00700 1 -- 172.21.15.33:0/1171437962 <== mds.2 v2:172.21.15.194:6838/517247009 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55863291d2c0 con 0x558632a47f10 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.205+0000 7fe08fe00700 10 client.24875 handle_command_reply: tid=0 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.205+0000 7fe0bb988b80 1 client.24875 shutdown 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.205+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 msgr2=0x5586329c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.205+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329c8a10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe09c000c00 tx=0x7fe09c0092d0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 shutdown_connections 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558632a47f10 0x558632a46c20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5586329c7cd0 0x5586329c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586329c3080 0x5586329c8a10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 --2- 172.21.15.33:0/1171437962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5586329c73b0 0x5586329c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe08fe00700 0 client.24875 ms_handle_reset on v2:172.21.15.194:6838/517247009 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 >> 172.21.15.33:0/1171437962 conn(0x5586328f02a0 msgr2=0x5586329c1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 shutdown_connections 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 1 -- 172.21.15.33:0/1171437962 wait complete. 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.206+0000 7fe0bb988b80 20 client.24875 trim_cache size 0 max 16384 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24484, 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:29:16.212 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:29:16.213 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:29:16.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 146.09705171900001, 2024-04-26T11:29:16.213 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-04-26T11:29:16.213 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-04-26T11:29:16.213 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T11:29:16.213 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 198.28710245299999 2024-04-26T11:29:16.213 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0900a3c40 msgr2=0x7fe0900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0900a3c40 0x7fe0900a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac108aa0 tx=0x7fe094009260 comp rx=0 tx=0).stop 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06b130 msgr2=0x7fe0ac1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06b130 0x7fe0ac1023f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00e9a0 tx=0x7fe09c00ecb0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 shutdown_connections 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0900a3c40 0x7fe0900a6100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0ac1094a0 0x7fe0ac102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0ac06b130 0x7fe0ac1023f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 --2- 172.21.15.33:0/1428440984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0ac06bad0 0x7fe0ac102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:16.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 >> 172.21.15.33:0/1428440984 conn(0x7fe0ac0792d0 msgr2=0x7fe0ac100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:16.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 shutdown_connections 2024-04-26T11:29:16.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:16.207+0000 7fe0b1000700 1 -- 172.21.15.33:0/1428440984 wait complete. 2024-04-26T11:29:17.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-26T11:29:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:17 smithi194 ceph-mon[33017]: pgmap v409: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:29:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:17 smithi033 ceph-mon[27845]: pgmap v409: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:29:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:17 smithi096 ceph-mon[32753]: pgmap v409: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:29:17.526 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:19 smithi194 ceph-mon[33017]: pgmap v410: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:19.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:19 smithi033 ceph-mon[27845]: pgmap v410: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:19 smithi096 ceph-mon[32753]: pgmap v410: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.696+0000 7f180e000700 1 -- 172.21.15.33:0/196870113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18081094a0 msgr2=0x7f1808109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.696+0000 7f180e000700 1 --2- 172.21.15.33:0/196870113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18081094a0 0x7f1808109880 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009990 tx=0x7f17f8009ca0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.696+0000 7f180e000700 1 -- 172.21.15.33:0/196870113 shutdown_connections 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.696+0000 7f180e000700 1 --2- 172.21.15.33:0/196870113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806bad0 0x7f18081102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.696+0000 7f180e000700 1 --2- 172.21.15.33:0/196870113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18081094a0 0x7f1808109880 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.696+0000 7f180e000700 1 --2- 172.21.15.33:0/196870113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f180806b130 0x7f180806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.696+0000 7f180e000700 1 -- 172.21.15.33:0/196870113 >> 172.21.15.33:0/196870113 conn(0x7f18080792d0 msgr2=0x7f18080796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.697+0000 7f180e000700 1 -- 172.21.15.33:0/196870113 shutdown_connections 2024-04-26T11:29:19.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.697+0000 7f180e000700 1 -- 172.21.15.33:0/196870113 wait complete. 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 Processor -- start 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 -- start start 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806b130 0x7f180819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f180806bad0 0x7f180819c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18081094a0 0x7f18081a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18081135b0 con 0x7f180806bad0 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1808113430 con 0x7f18081094a0 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.698+0000 7f180e000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1808113730 con 0x7f180806b130 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1806a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f180806bad0 0x7f180819c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806b130 0x7f180819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806b130 0x7f180819bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:40114/0 (socket says 172.21.15.33:40114) 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 -- 172.21.15.33:0/876886889 learned_addr learned my addr 172.21.15.33:0/876886889 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:19.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807e00700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18081094a0 0x7f18081a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 -- 172.21.15.33:0/876886889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18081094a0 msgr2=0x7f18081a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18081094a0 0x7f18081a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 -- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f180806bad0 msgr2=0x7f180819c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f180806bad0 0x7f180819c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 -- 172.21.15.33:0/876886889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18081a0d30 con 0x7f180806b130 2024-04-26T11:29:19.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807e00700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18081094a0 0x7f18081a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:29:19.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1806a00700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f180806bad0 0x7f180819c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:19.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.699+0000 7f1807400700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806b130 0x7f180819bd20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17f80057c0 tx=0x7f17f8009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:19.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.700+0000 7f17ffe00700 1 -- 172.21.15.33:0/876886889 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8026070 con 0x7f180806b130 2024-04-26T11:29:19.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.700+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18081a0fc0 con 0x7f180806b130 2024-04-26T11:29:19.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.700+0000 7f17ffe00700 1 -- 172.21.15.33:0/876886889 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f8021410 con 0x7f180806b130 2024-04-26T11:29:19.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.700+0000 7f17ffe00700 1 -- 172.21.15.33:0/876886889 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8017650 con 0x7f180806b130 2024-04-26T11:29:19.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.700+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18081a88a0 con 0x7f180806b130 2024-04-26T11:29:19.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.704+0000 7f18189afb80 1 Processor -- start 2024-04-26T11:29:19.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.704+0000 7f18189afb80 1 -- start start 2024-04-26T11:29:19.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f18189afb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f18189afb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e2cd0 0x557bfe1ddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f18189afb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1de080 0x557bfe1de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f18189afb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557bfe130840 con 0x557bfe1e23b0 2024-04-26T11:29:19.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f18189afb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557bfe0cf160 con 0x557bfe1de080 2024-04-26T11:29:19.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f18189afb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557bfe12c800 con 0x557bfe1e2cd0 2024-04-26T11:29:19.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f17ffe00700 1 -- 172.21.15.33:0/876886889 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f17f8029070 con 0x7f180806b130 2024-04-26T11:29:19.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f1806a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1de080 0x557bfe1de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f1806a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1de080 0x557bfe1de4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41336/0 (socket says 172.21.15.33:41336) 2024-04-26T11:29:19.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f1806a00700 1 -- 172.21.15.33:0/636345920 learned_addr learned my addr 172.21.15.33:0/636345920 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:19.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.705+0000 7f1807e00700 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.706+0000 7f1807400700 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e2cd0 0x557bfe1ddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.706+0000 7f1806a00700 1 -- 172.21.15.33:0/636345920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e2cd0 msgr2=0x557bfe1ddb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.706+0000 7f1806a00700 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e2cd0 0x557bfe1ddb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.706+0000 7f1806a00700 1 -- 172.21.15.33:0/636345920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 msgr2=0x557bfe1e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.706+0000 7f1806a00700 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.706+0000 7f1806a00700 1 -- 172.21.15.33:0/636345920 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bfe0d75e0 con 0x557bfe1de080 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.706+0000 7f1806a00700 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1de080 0x557bfe1de4c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17f40115f0 tx=0x7f17f4011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c7419b0a2b55f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.707+0000 7f17fcc00700 1 -- 172.21.15.33:0/636345920 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f4004220 con 0x557bfe1de080 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.707+0000 7f17fcc00700 1 -- 172.21.15.33:0/636345920 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f40043c0 con 0x557bfe1de080 2024-04-26T11:29:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.707+0000 7f17fcc00700 1 -- 172.21.15.33:0/636345920 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f40198d0 con 0x557bfe1de080 2024-04-26T11:29:19.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.707+0000 7f17ffe00700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f17ec0a3d40 0x7f17ec0a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f17ffe00700 1 -- 172.21.15.33:0/876886889 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f17f80d1100 con 0x7f180806b130 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f1807e00700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f17ec0a3d40 0x7f17ec0a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 -- 172.21.15.33:0/636345920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1de080 msgr2=0x557bfe1de4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1de080 0x557bfe1de4c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17f40115f0 tx=0x7f17f4011900 comp rx=0 tx=0).stop 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 -- 172.21.15.33:0/636345920 shutdown_connections 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1de080 0x557bfe1de4c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 --2- 172.21.15.33:0/636345920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e2cd0 0x557bfe1ddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 -- 172.21.15.33:0/636345920 >> 172.21.15.33:0/636345920 conn(0x557bfe10b2a0 msgr2=0x557bfe1d8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 -- 172.21.15.33:0/636345920 shutdown_connections 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.708+0000 7f18189afb80 1 -- 172.21.15.33:0/636345920 wait complete. 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.709+0000 7f1807e00700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f17ec0a3d40 0x7f17ec0a6200 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1800009d70 tx=0x7f1800009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:19.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.709+0000 7f18189afb80 1 -- start start 2024-04-26T11:29:19.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.709+0000 7f18189afb80 1 Processor -- start 2024-04-26T11:29:19.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f18189afb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1de080 0x557bfe1e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f18189afb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f18189afb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f18189afb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1ea810 con 0x557bfe1de080 2024-04-26T11:29:19.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f18189afb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557bfe0cb3b0 con 0x557bfe1e2cd0 2024-04-26T11:29:19.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f18189afb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1ea990 con 0x557bfe1e23b0 2024-04-26T11:29:19.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f18189afb80 10 client.0 _ll_register_callbacks cb 0x557bfe0c4330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41348/0 (socket says 172.21.15.33:41348) 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807e00700 1 -- 172.21.15.33:0/854449802 learned_addr learned my addr 172.21.15.33:0/854449802 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807400700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1de080 0x557bfe1e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1806a00700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807e00700 1 -- 172.21.15.33:0/854449802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e23b0 msgr2=0x557bfe1e6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807e00700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807e00700 1 -- 172.21.15.33:0/854449802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1de080 msgr2=0x557bfe1e9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.710+0000 7f1807e00700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1de080 0x557bfe1e9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f1807e00700 1 -- 172.21.15.33:0/854449802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bfe07d880 con 0x557bfe1e2cd0 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f1806a00700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f1807e00700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1800009550 tx=0x7f1800004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c89b2ffbe66d311b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f17f1600700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:29:19.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f17f1600700 1 -- 172.21.15.33:0/854449802 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180001d050 con 0x557bfe1e2cd0 2024-04-26T11:29:19.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17f40043c0 con 0x557bfe1e2cd0 2024-04-26T11:29:19.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f17f1600700 1 -- 172.21.15.33:0/854449802 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1800017890 con 0x557bfe1e2cd0 2024-04-26T11:29:19.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.711+0000 7f17f1600700 1 -- 172.21.15.33:0/854449802 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1800017a90 con 0x557bfe1e2cd0 2024-04-26T11:29:19.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 10 client.24887 fetch_fsmap learned FSMap version 25 2024-04-26T11:29:19.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557bfe1f5060 con 0x557bfe1e2cd0 2024-04-26T11:29:19.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f17f1600700 1 -- 172.21.15.33:0/854449802 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f40043c0 con 0x557bfe1e2cd0 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f17f1600700 1 -- 172.21.15.33:0/854449802 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f1800022c50 con 0x557bfe1e2cd0 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 10 client.24887 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 10 client.24887 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 20 client.24887 populate_metadata read hostname 'smithi033' 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe1f7750 0x557bfe1f8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 4 client.24887 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.712+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 --> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bfe13e220 con 0x557bfe1f7750 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.713+0000 7f1807400700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe1f7750 0x557bfe1f8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.713+0000 7f1807400700 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe1f7750 0x557bfe1f8bc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e536d867529b0006 server_cookie=a5f02529da72fa3 in_seq=0 out_seq=0 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.713+0000 7f17f1600700 10 client.24887 ms_handle_connect on v2:172.21.15.96:6836/823522377 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f17f1600700 1 -- 172.21.15.33:0/854449802 <== mds.1 v2:172.21.15.96:6836/823522377 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bfe13e220 con 0x557bfe1f7750 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f17f1600700 10 client.24887 handle_command_reply: tid=0 2024-04-26T11:29:19.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 client.24887 shutdown 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 msgr2=0x557bfe1e7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1800009550 tx=0x7f1800004b40 comp rx=0 tx=0).stop 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 shutdown_connections 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe1f7750 0x557bfe1f8bc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1de080 0x557bfe1e9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 --2- 172.21.15.33:0/854449802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 >> 172.21.15.33:0/854449802 conn(0x557bfe10b2a0 msgr2=0x557bfe1d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.714+0000 7f17f1600700 0 client.24887 ms_handle_reset on v2:172.21.15.96:6836/823522377 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.715+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 shutdown_connections 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.715+0000 7f18189afb80 1 -- 172.21.15.33:0/854449802 wait complete. 2024-04-26T11:29:19.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.715+0000 7f18189afb80 20 client.24887 trim_cache size 0 max 16384 2024-04-26T11:29:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 Processor -- start 2024-04-26T11:29:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 -- start start 2024-04-26T11:29:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1f6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1f6f00 con 0x557bfe1e23b0 2024-04-26T11:29:19.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1f75d0 con 0x557bfe1e2cd0 2024-04-26T11:29:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.737+0000 7f18189afb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1f7ca0 con 0x557bfe1de080 2024-04-26T11:29:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1806a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:41352/0 (socket says 172.21.15.33:41352) 2024-04-26T11:29:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 -- 172.21.15.33:0/3067641102 learned_addr learned my addr 172.21.15.33:0/3067641102 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:19.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807400700 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 -- 172.21.15.33:0/3067641102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 msgr2=0x557bfe1f6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1f6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 -- 172.21.15.33:0/3067641102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 msgr2=0x557bfe1e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 -- 172.21.15.33:0/3067641102 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bfe1d8b50 con 0x557bfe1e2cd0 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.738+0000 7f1807e00700 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7200 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1800009f20 tx=0x7f180002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8110b6a0dc7e859 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.739+0000 7f17f3e00700 1 -- 172.21.15.33:0/3067641102 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180001d070 con 0x557bfe1e2cd0 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.739+0000 7f17f3e00700 1 -- 172.21.15.33:0/3067641102 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f180003b070 con 0x557bfe1e2cd0 2024-04-26T11:29:19.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.739+0000 7f17f3e00700 1 -- 172.21.15.33:0/3067641102 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800005e60 con 0x557bfe1e2cd0 2024-04-26T11:29:19.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 -- 172.21.15.33:0/3067641102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 msgr2=0x557bfe1e7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7200 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1800009f20 tx=0x7f180002b8a0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 -- 172.21.15.33:0/3067641102 shutdown_connections 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e7200 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1f6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 --2- 172.21.15.33:0/3067641102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe1e6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 -- 172.21.15.33:0/3067641102 >> 172.21.15.33:0/3067641102 conn(0x557bfe10b2a0 msgr2=0x557bfe1e9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 -- 172.21.15.33:0/3067641102 shutdown_connections 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 -- 172.21.15.33:0/3067641102 wait complete. 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.740+0000 7f18189afb80 1 -- start start 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 1 Processor -- start 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1dc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe202030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1f8a40 con 0x557bfe1e23b0 2024-04-26T11:29:19.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1f8370 con 0x557bfe1e2cd0 2024-04-26T11:29:19.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557bfe1f9110 con 0x557bfe1de080 2024-04-26T11:29:19.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f18189afb80 10 client.0 _ll_register_callbacks cb 0x557bfe0c4330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:29:19.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f1806a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe202030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f1807400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1dc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f1807400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1dc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:40158/0 (socket says 172.21.15.33:40158) 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f1807400700 1 -- 172.21.15.33:0/4222381359 learned_addr learned my addr 172.21.15.33:0/4222381359 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.741+0000 7f1807e00700 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f1807400700 1 -- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 msgr2=0x557bfe1e8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f1807400700 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f1807400700 1 -- 172.21.15.33:0/4222381359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 msgr2=0x557bfe202030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f1807400700 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe202030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f1807400700 1 -- 172.21.15.33:0/4222381359 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bfe1f3c60 con 0x557bfe1de080 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f1807400700 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1dc310 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17f8005f50 tx=0x7f17f8009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc864103d216b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f17f1600700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f17f1600700 1 -- 172.21.15.33:0/4222381359 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f809d290 con 0x557bfe1de080 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f180003b070 con 0x557bfe1de080 2024-04-26T11:29:19.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.742+0000 7f17f1600700 1 -- 172.21.15.33:0/4222381359 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f17f80218c0 con 0x557bfe1de080 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f17f1600700 1 -- 172.21.15.33:0/4222381359 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17f809ce20 con 0x557bfe1de080 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f18189afb80 10 client.24883 fetch_fsmap learned FSMap version 25 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557bfe25ecc0 con 0x557bfe1de080 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f17f1600700 1 -- 172.21.15.33:0/4222381359 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f180003b070 con 0x557bfe1de080 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f17f1600700 1 -- 172.21.15.33:0/4222381359 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f17f809d720 con 0x557bfe1de080 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f18189afb80 10 client.24883 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f18189afb80 10 client.24883 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f18189afb80 20 client.24883 populate_metadata read hostname 'smithi033' 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.743+0000 7f18189afb80 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe263870 0x557bfe262580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.744+0000 7f18189afb80 4 client.24883 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.744+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 --> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] -- command(tid 0: {"prefix": "status"}) -- 0x557bfe1382c0 con 0x557bfe263870 2024-04-26T11:29:19.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.744+0000 7f1806a00700 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe263870 0x557bfe262580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:19.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f1806a00700 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe263870 0x557bfe262580 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c539d0e198b4768 server_cookie=f143dfe879367c52 in_seq=0 out_seq=0 2024-04-26T11:29:19.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f17f1600700 10 client.24883 ms_handle_connect on v2:172.21.15.96:6836/823522377 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f17f1600700 1 -- 172.21.15.33:0/4222381359 <== mds.1 v2:172.21.15.96:6836/823522377 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557bfe1382c0 con 0x557bfe263870 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f17f1600700 10 client.24883 handle_command_reply: tid=0 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 client.24883 shutdown 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 msgr2=0x557bfe1dc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1dc310 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17f8005f50 tx=0x7f17f8009ca0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 shutdown_connections 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x557bfe263870 0x557bfe262580 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557bfe1e2cd0 0x557bfe1e8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557bfe1de080 0x557bfe1dc310 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 --2- 172.21.15.33:0/4222381359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557bfe1e23b0 0x557bfe202030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f17f1600700 0 client.24883 ms_handle_reset on v2:172.21.15.96:6836/823522377 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.745+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 >> 172.21.15.33:0/4222381359 conn(0x557bfe10b2a0 msgr2=0x557bfe1e6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.746+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 shutdown_connections 2024-04-26T11:29:19.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.746+0000 7f18189afb80 1 -- 172.21.15.33:0/4222381359 wait complete. 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.746+0000 7f18189afb80 20 client.24883 trim_cache size 0 max 16384 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24488, 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 150.65965628500001, 2024-04-26T11:29:19.749 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-04-26T11:29:19.750 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-04-26T11:29:19.750 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T11:29:19.750 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 204.692301512 2024-04-26T11:29:19.750 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:29:19.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.746+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f17ec0a3d40 msgr2=0x7f17ec0a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.746+0000 7f180e000700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f17ec0a3d40 0x7f17ec0a6200 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1800009d70 tx=0x7f1800009040 comp rx=0 tx=0).stop 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.746+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806b130 msgr2=0x7f180819bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.746+0000 7f180e000700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806b130 0x7f180819bd20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17f80057c0 tx=0x7f17f8009580 comp rx=0 tx=0).stop 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 shutdown_connections 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f17ec0a3d40 0x7f17ec0a6200 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18081094a0 0x7f18081a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f180806b130 0x7f180819bd20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 --2- 172.21.15.33:0/876886889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f180806bad0 0x7f180819c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 >> 172.21.15.33:0/876886889 conn(0x7f18080792d0 msgr2=0x7f180810e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 shutdown_connections 2024-04-26T11:29:19.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:19.747+0000 7f180e000700 1 -- 172.21.15.33:0/876886889 wait complete. 2024-04-26T11:29:20.682 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T11:29:21.176 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:21 smithi033 ceph-mon[27845]: pgmap v411: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T11:29:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:21 smithi096 ceph-mon[32753]: pgmap v411: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T11:29:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:21 smithi194 ceph-mon[33017]: pgmap v411: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T11:29:23.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:23 smithi033 ceph-mon[27845]: pgmap v412: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.397+0000 7fe105600700 1 -- 172.21.15.33:0/968281906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1001094e0 msgr2=0x7fe1001098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.397+0000 7fe105600700 1 --2- 172.21.15.33:0/968281906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1001094e0 0x7fe1001098c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009990 tx=0x7fe0f0009ca0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.397+0000 7fe105600700 1 -- 172.21.15.33:0/968281906 shutdown_connections 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.397+0000 7fe105600700 1 --2- 172.21.15.33:0/968281906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe100103e40 0x7fe100110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.397+0000 7fe105600700 1 --2- 172.21.15.33:0/968281906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1001094e0 0x7fe1001098c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.397+0000 7fe105600700 1 --2- 172.21.15.33:0/968281906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1001034a0 0x7fe100103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.397+0000 7fe105600700 1 -- 172.21.15.33:0/968281906 >> 172.21.15.33:0/968281906 conn(0x7fe10007a4f0 msgr2=0x7fe10007a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.398+0000 7fe105600700 1 -- 172.21.15.33:0/968281906 shutdown_connections 2024-04-26T11:29:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.398+0000 7fe105600700 1 -- 172.21.15.33:0/968281906 wait complete. 2024-04-26T11:29:23.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.399+0000 7fe105600700 1 Processor -- start 2024-04-26T11:29:23.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.399+0000 7fe105600700 1 -- start start 2024-04-26T11:29:23.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe105600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1001034a0 0x7fe10019bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe105600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe100103e40 0x7fe10019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe105600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1001094e0 0x7fe1001a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe105600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe100113620 con 0x7fe1001034a0 2024-04-26T11:29:23.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe105600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe1001134a0 con 0x7fe1001094e0 2024-04-26T11:29:23.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe105600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe1001137a0 con 0x7fe100103e40 2024-04-26T11:29:23.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1001034a0 0x7fe10019bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe0fe000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe100103e40 0x7fe10019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe0fe000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe100103e40 0x7fe10019c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:35784/0 (socket says 172.21.15.33:35784) 2024-04-26T11:29:23.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1001034a0 0x7fe10019bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46766/0 (socket says 172.21.15.33:46766) 2024-04-26T11:29:23.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe0fe000700 1 -- 172.21.15.33:0/2678110397 learned_addr learned my addr 172.21.15.33:0/2678110397 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:23.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.400+0000 7fe0ff400700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1001094e0 0x7fe1001a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.401+0000 7fe0fe000700 1 -- 172.21.15.33:0/2678110397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1001094e0 msgr2=0x7fe1001a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.401+0000 7fe0fe000700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1001094e0 0x7fe1001a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.401+0000 7fe0fe000700 1 -- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1001034a0 msgr2=0x7fe10019bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.401+0000 7fe0fe000700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1001034a0 0x7fe10019bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.401+0000 7fe0fe000700 1 -- 172.21.15.33:0/2678110397 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1001a0d80 con 0x7fe100103e40 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.401+0000 7fe0fea00700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1001034a0 0x7fe10019bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.401+0000 7fe0fe000700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe100103e40 0x7fe10019c2b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00e9a0 tx=0x7fe0ec00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.402+0000 7fe0f7400700 1 -- 172.21.15.33:0/2678110397 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec00ca10 con 0x7fe100103e40 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.402+0000 7fe0f7400700 1 -- 172.21.15.33:0/2678110397 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec00cbb0 con 0x7fe100103e40 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.402+0000 7fe0f7400700 1 -- 172.21.15.33:0/2678110397 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec018680 con 0x7fe100103e40 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.402+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1001a1070 con 0x7fe100103e40 2024-04-26T11:29:23.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.402+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1001a88f0 con 0x7fe100103e40 2024-04-26T11:29:23.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.406+0000 7fe0f7400700 1 -- 172.21.15.33:0/2678110397 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe0ec010430 con 0x7fe100103e40 2024-04-26T11:29:23.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.406+0000 7fe10ff69b80 1 Processor -- start 2024-04-26T11:29:23.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe10ff69b80 1 -- start start 2024-04-26T11:29:23.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c22d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22dcd0 0x559e6c228b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c229080 0x559e6c2294c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e6c17b840 con 0x559e6c22dcd0 2024-04-26T11:29:23.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559e6c11a160 con 0x559e6c229080 2024-04-26T11:29:23.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559e6c177800 con 0x559e6c22d3b0 2024-04-26T11:29:23.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.407+0000 7fe0ff400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c22d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c22d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:35800/0 (socket says 172.21.15.33:35800) 2024-04-26T11:29:23.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22dcd0 0x559e6c228b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 -- 172.21.15.33:0/647433634 learned_addr learned my addr 172.21.15.33:0/647433634 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:23.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0f7400700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0e40a3cc0 0x7fe0e40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0fe000700 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c229080 0x559e6c2294c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 -- 172.21.15.33:0/647433634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c229080 msgr2=0x559e6c2294c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c229080 0x559e6c2294c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 -- 172.21.15.33:0/647433634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22dcd0 msgr2=0x559e6c228b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22dcd0 0x559e6c228b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 -- 172.21.15.33:0/647433634 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6c1225e0 con 0x559e6c22d3b0 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0fe000700 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c229080 0x559e6c2294c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0fea00700 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22dcd0 0x559e6c228b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0f7400700 1 -- 172.21.15.33:0/2678110397 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe0ec0d0510 con 0x7fe100103e40 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.408+0000 7fe0ff400700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0e40a3cc0 0x7fe0e40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.409+0000 7fe0ff400700 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c22d790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8011790 tx=0x7fe0f8011aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf3271d3b4d786a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.409+0000 7fe0ebe00700 1 -- 172.21.15.33:0/647433634 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8017670 con 0x559e6c22d3b0 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.409+0000 7fe0ff400700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0e40a3cc0 0x7fe0e40a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8020000 tx=0x7fe0f8021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.409+0000 7fe0ebe00700 1 -- 172.21.15.33:0/647433634 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0f80158a0 con 0x559e6c22d3b0 2024-04-26T11:29:23.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.409+0000 7fe0ebe00700 1 -- 172.21.15.33:0/647433634 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80167f0 con 0x559e6c22d3b0 2024-04-26T11:29:23.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 -- 172.21.15.33:0/647433634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 msgr2=0x559e6c22d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c22d790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8011790 tx=0x7fe0f8011aa0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 -- 172.21.15.33:0/647433634 shutdown_connections 2024-04-26T11:29:23.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c229080 0x559e6c2294c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c22d790 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/647433634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22dcd0 0x559e6c228b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 -- 172.21.15.33:0/647433634 >> 172.21.15.33:0/647433634 conn(0x559e6c1562a0 msgr2=0x559e6c223610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:23.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.410+0000 7fe10ff69b80 1 -- 172.21.15.33:0/647433634 shutdown_connections 2024-04-26T11:29:23.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.411+0000 7fe10ff69b80 1 -- 172.21.15.33:0/647433634 wait complete. 2024-04-26T11:29:23.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.411+0000 7fe10ff69b80 1 -- start start 2024-04-26T11:29:23.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.411+0000 7fe10ff69b80 1 Processor -- start 2024-04-26T11:29:23.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c229080 0x559e6c234b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22d3b0 0x559e6c231e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c232340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e6c235810 con 0x559e6c22d3b0 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559e6c1163b0 con 0x559e6c22dcd0 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559e6c235990 con 0x559e6c229080 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe10ff69b80 10 client.0 _ll_register_callbacks cb 0x559e6c10f330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe0fe000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22d3b0 0x559e6c231e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c229080 0x559e6c234b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe0fe000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22d3b0 0x559e6c231e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46776/0 (socket says 172.21.15.33:46776) 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.412+0000 7fe0fe000700 1 -- 172.21.15.33:0/3904175363 learned_addr learned my addr 172.21.15.33:0/3904175363 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0ff400700 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c232340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0fea00700 1 -- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 msgr2=0x559e6c232340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0fea00700 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c232340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0fea00700 1 -- 172.21.15.33:0/3904175363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22d3b0 msgr2=0x559e6c231e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0fea00700 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22d3b0 0x559e6c231e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0fea00700 1 -- 172.21.15.33:0/3904175363 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e6c0c8880 con 0x559e6c229080 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0fe000700 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22d3b0 0x559e6c231e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0fea00700 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c229080 0x559e6c234b00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0000c00 tx=0x7fe0f00052e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b63aacddb36e5a49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.413+0000 7fe0e8c00700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.414+0000 7fe0e8c00700 1 -- 172.21.15.33:0/3904175363 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0003a90 con 0x559e6c229080 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.414+0000 7fe0e8c00700 1 -- 172.21.15.33:0/3904175363 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe0f0003c30 con 0x559e6c229080 2024-04-26T11:29:23.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.414+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0f80158a0 con 0x559e6c229080 2024-04-26T11:29:23.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.414+0000 7fe0e8c00700 1 -- 172.21.15.33:0/3904175363 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe0f0003e30 con 0x559e6c229080 2024-04-26T11:29:23.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.414+0000 7fe10ff69b80 10 client.24895 fetch_fsmap learned FSMap version 25 2024-04-26T11:29:23.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.414+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559e6c2400c0 con 0x559e6c229080 2024-04-26T11:29:23.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe0e8c00700 1 -- 172.21.15.33:0/3904175363 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0f80158a0 con 0x559e6c229080 2024-04-26T11:29:23.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe0e8c00700 1 -- 172.21.15.33:0/3904175363 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fe0f002c030 con 0x559e6c229080 2024-04-26T11:29:23.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe10ff69b80 10 client.24895 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe10ff69b80 10 client.24895 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe10ff69b80 20 client.24895 populate_metadata read hostname 'smithi033' 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2426f0 0x559e6c243b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe10ff69b80 4 client.24895 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e6c189220 con 0x559e6c2426f0 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.415+0000 7fe0fe000700 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2426f0 0x559e6c243b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.416+0000 7fe0fe000700 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2426f0 0x559e6c243b60 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81a36007135fa78b server_cookie=69203bc661e98ef5 in_seq=0 out_seq=0 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.416+0000 7fe0e8c00700 10 client.24895 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe0e8c00700 1 -- 172.21.15.33:0/3904175363 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e6c189220 con 0x559e6c2426f0 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe0e8c00700 10 client.24895 handle_command_reply: tid=0 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 client.24895 shutdown 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c229080 msgr2=0x559e6c234b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c229080 0x559e6c234b00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0000c00 tx=0x7fe0f00052e0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 shutdown_connections 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2426f0 0x559e6c243b60 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c232340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c229080 0x559e6c234b00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/3904175363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c22d3b0 0x559e6c231e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe0e8c00700 0 client.24895 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 >> 172.21.15.33:0/3904175363 conn(0x559e6c1562a0 msgr2=0x559e6c21de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 shutdown_connections 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.417+0000 7fe10ff69b80 1 -- 172.21.15.33:0/3904175363 wait complete. 2024-04-26T11:29:23.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.418+0000 7fe10ff69b80 20 client.24895 trim_cache size 0 max 16384 2024-04-26T11:29:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:23 smithi096 ceph-mon[32753]: pgmap v412: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:29:23.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.440+0000 7fe10ff69b80 1 Processor -- start 2024-04-26T11:29:23.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe10ff69b80 1 -- start start 2024-04-26T11:29:23.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c2234e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22d3b0 0x559e6c22f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22dcd0 0x559e6c22f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e6c241ea0 con 0x559e6c229080 2024-04-26T11:29:23.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559e6c242570 con 0x559e6c22d3b0 2024-04-26T11:29:23.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559e6c242c40 con 0x559e6c22dcd0 2024-04-26T11:29:23.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c2234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c2234e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46778/0 (socket says 172.21.15.33:46778) 2024-04-26T11:29:23.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe0fea00700 1 -- 172.21.15.33:0/1570768273 learned_addr learned my addr 172.21.15.33:0/1570768273 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:23.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe0ff400700 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22dcd0 0x559e6c22f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.441+0000 7fe0fe000700 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22d3b0 0x559e6c22f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0fea00700 1 -- 172.21.15.33:0/1570768273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22dcd0 msgr2=0x559e6c22f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0fea00700 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22dcd0 0x559e6c22f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0fea00700 1 -- 172.21.15.33:0/1570768273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22d3b0 msgr2=0x559e6c22f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0fea00700 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22d3b0 0x559e6c22f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0fea00700 1 -- 172.21.15.33:0/1570768273 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6c230370 con 0x559e6c229080 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0fea00700 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c2234e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00098e0 tx=0x7fe0f000bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2af996c224d99b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0eb400700 1 -- 172.21.15.33:0/1570768273 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f003d070 con 0x559e6c229080 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0eb400700 1 -- 172.21.15.33:0/1570768273 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0f000f730 con 0x559e6c229080 2024-04-26T11:29:23.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.442+0000 7fe0eb400700 1 -- 172.21.15.33:0/1570768273 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f00246e0 con 0x559e6c229080 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 -- 172.21.15.33:0/1570768273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 msgr2=0x559e6c2234e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c2234e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00098e0 tx=0x7fe0f000bfc0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 -- 172.21.15.33:0/1570768273 shutdown_connections 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22dcd0 0x559e6c22f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c2234e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/1570768273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22d3b0 0x559e6c22f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 -- 172.21.15.33:0/1570768273 >> 172.21.15.33:0/1570768273 conn(0x559e6c1562a0 msgr2=0x559e6c234920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.443+0000 7fe10ff69b80 1 -- 172.21.15.33:0/1570768273 shutdown_connections 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.444+0000 7fe10ff69b80 1 -- 172.21.15.33:0/1570768273 wait complete. 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.444+0000 7fe10ff69b80 1 -- start start 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.444+0000 7fe10ff69b80 1 Processor -- start 2024-04-26T11:29:23.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c24cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c24d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe10ff69b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c226d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559e6c2439e0 con 0x559e6c229080 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559e6c243310 con 0x559e6c22dcd0 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe10ff69b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559e6c2440b0 con 0x559e6c22d3b0 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe10ff69b80 10 client.0 _ll_register_callbacks cb 0x559e6c10f330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c24cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c24cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46786/0 (socket says 172.21.15.33:46786) 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fea00700 1 -- 172.21.15.33:0/481660421 learned_addr learned my addr 172.21.15.33:0/481660421 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0ff400700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c226d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fe000700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c24d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fea00700 1 -- 172.21.15.33:0/481660421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 msgr2=0x559e6c24d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fea00700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c24d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fea00700 1 -- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 msgr2=0x559e6c226d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.445+0000 7fe0fea00700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c226d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe0fea00700 1 -- 172.21.15.33:0/481660421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e6c227550 con 0x559e6c229080 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe0ff400700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c226d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe0fea00700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c24cfd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005950 tx=0x7fe0f001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceb5fd74e155a1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe0e8c00700 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:29:23.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe0e8c00700 1 -- 172.21.15.33:0/481660421 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f003d040 con 0x559e6c229080 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0f000f730 con 0x559e6c229080 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe0e8c00700 1 -- 172.21.15.33:0/481660421 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe0f0003890 con 0x559e6c229080 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe0e8c00700 1 -- 172.21.15.33:0/481660421 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe0f0049050 con 0x559e6c229080 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe10ff69b80 10 client.14796 fetch_fsmap learned FSMap version 25 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.446+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559e6c2b2030 con 0x559e6c229080 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe0e8c00700 1 -- 172.21.15.33:0/481660421 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0f000f730 con 0x559e6c229080 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe0e8c00700 1 -- 172.21.15.33:0/481660421 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fe0f0040070 con 0x559e6c229080 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe10ff69b80 10 client.14796 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe10ff69b80 10 client.14796 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe10ff69b80 20 client.14796 populate_metadata read hostname 'smithi033' 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2b6690 0x559e6c2b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe10ff69b80 4 client.14796 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:29:23.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.447+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "status"}) -- 0x559e6c1832c0 con 0x559e6c2b6690 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.448+0000 7fe0fe000700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2b6690 0x559e6c2b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.448+0000 7fe0fe000700 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2b6690 0x559e6c2b53a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f56ba0c60465991c server_cookie=b09bc03dc0680e45 in_seq=0 out_seq=0 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.448+0000 7fe0e8c00700 10 client.14796 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.448+0000 7fe0e8c00700 1 -- 172.21.15.33:0/481660421 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559e6c1832c0 con 0x559e6c2b6690 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.448+0000 7fe0e8c00700 10 client.14796 handle_command_reply: tid=0 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.448+0000 7fe10ff69b80 1 client.14796 shutdown 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 msgr2=0x559e6c24cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c24cfd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005950 tx=0x7fe0f001a040 comp rx=0 tx=0).stop 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 shutdown_connections 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559e6c2b6690 0x559e6c2b53a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559e6c22dcd0 0x559e6c226d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559e6c229080 0x559e6c24cfd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 --2- 172.21.15.33:0/481660421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6c22d3b0 0x559e6c24d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 >> 172.21.15.33:0/481660421 conn(0x559e6c1562a0 msgr2=0x559e6c2234e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe0e8c00700 0 client.14796 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 shutdown_connections 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 1 -- 172.21.15.33:0/481660421 wait complete. 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.449+0000 7fe10ff69b80 20 client.14796 trim_cache size 0 max 16384 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24527, 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:29:23.454 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 170.42240461899999, 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 192.03441242299999 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:29:23.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0e40a3cc0 msgr2=0x7fe0e40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0e40a3cc0 0x7fe0e40a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8020000 tx=0x7fe0f8021000 comp rx=0 tx=0).stop 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe100103e40 msgr2=0x7fe10019c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe100103e40 0x7fe10019c2b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00e9a0 tx=0x7fe0ec00ecb0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 shutdown_connections 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe0e40a3cc0 0x7fe0e40a6180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1001094e0 0x7fe1001a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1001034a0 0x7fe10019bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 --2- 172.21.15.33:0/2678110397 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe100103e40 0x7fe10019c2b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 >> 172.21.15.33:0/2678110397 conn(0x7fe10007a4f0 msgr2=0x7fe100101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 shutdown_connections 2024-04-26T11:29:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:23.450+0000 7fe105600700 1 -- 172.21.15.33:0/2678110397 wait complete. 2024-04-26T11:29:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:23 smithi194 ceph-mon[33017]: pgmap v412: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T11:29:24.398 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T11:29:24.398 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T11:29:24.408 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:29:24.408 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T11:29:24.885 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:25.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:25 smithi033 ceph-mon[27845]: pgmap v413: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:25 smithi096 ceph-mon[32753]: pgmap v413: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:25 smithi194 ceph-mon[33017]: pgmap v413: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T11:29:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.203+0000 7fbdede00700 1 -- 172.21.15.33:0/1121217898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde81033b0 msgr2=0x7fbde81037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.203+0000 7fbdede00700 1 --2- 172.21.15.33:0/1121217898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde81033b0 0x7fbde81037d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800b0d0 tx=0x7fbdd800b3e0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.204+0000 7fbdede00700 1 -- 172.21.15.33:0/1121217898 shutdown_connections 2024-04-26T11:29:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.204+0000 7fbdede00700 1 --2- 172.21.15.33:0/1121217898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde8104f50 0x7fbde810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.204+0000 7fbdede00700 1 --2- 172.21.15.33:0/1121217898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde81033b0 0x7fbde81037d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.204+0000 7fbdede00700 1 --2- 172.21.15.33:0/1121217898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81045b0 0x7fbde8104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.204+0000 7fbdede00700 1 -- 172.21.15.33:0/1121217898 >> 172.21.15.33:0/1121217898 conn(0x7fbde80fe950 msgr2=0x7fbde8100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.204+0000 7fbdede00700 1 -- 172.21.15.33:0/1121217898 shutdown_connections 2024-04-26T11:29:27.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.204+0000 7fbdede00700 1 -- 172.21.15.33:0/1121217898 wait complete. 2024-04-26T11:29:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 Processor -- start 2024-04-26T11:29:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 -- start start 2024-04-26T11:29:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81033b0 0x7fbde819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde81045b0 0x7fbde819bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:27.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde8104f50 0x7fbde81a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbde810ee70 con 0x7fbde81045b0 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbde810ecf0 con 0x7fbde8104f50 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.205+0000 7fbdede00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbde810eff0 con 0x7fbde81033b0 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbddfe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde81045b0 0x7fbde819bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81033b0 0x7fbde819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81033b0 0x7fbde819b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:35860/0 (socket says 172.21.15.33:35860) 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 -- 172.21.15.33:0/2820768265 learned_addr learned my addr 172.21.15.33:0/2820768265 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbded400700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde8104f50 0x7fbde81a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 -- 172.21.15.33:0/2820768265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde8104f50 msgr2=0x7fbde81a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde8104f50 0x7fbde81a2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 -- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde81045b0 msgr2=0x7fbde819bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde81045b0 0x7fbde819bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 -- 172.21.15.33:0/2820768265 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde81a3330 con 0x7fbde81033b0 2024-04-26T11:29:27.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbded400700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde8104f50 0x7fbde81a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:29:27.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.206+0000 7fbdeca00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81033b0 0x7fbde819b7c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009580 tx=0x7fbdd80095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:27.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.207+0000 7fbde6000700 1 -- 172.21.15.33:0/2820768265 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd801f070 con 0x7fbde81033b0 2024-04-26T11:29:27.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.207+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde81a35c0 con 0x7fbde81033b0 2024-04-26T11:29:27.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.207+0000 7fbde6000700 1 -- 172.21.15.33:0/2820768265 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdd8007b80 con 0x7fbde81033b0 2024-04-26T11:29:27.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.207+0000 7fbde6000700 1 -- 172.21.15.33:0/2820768265 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8004890 con 0x7fbde81033b0 2024-04-26T11:29:27.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.207+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde81a3a70 con 0x7fbde81033b0 2024-04-26T11:29:27.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.208+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde806a110 con 0x7fbde81033b0 2024-04-26T11:29:27.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.210+0000 7fbde6000700 1 -- 172.21.15.33:0/2820768265 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbdd8004020 con 0x7fbde81033b0 2024-04-26T11:29:27.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.211+0000 7fbde6000700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbdcc0a3cc0 0x7fbdcc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:27.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.211+0000 7fbde6000700 1 -- 172.21.15.33:0/2820768265 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbdd800f020 con 0x7fbde81033b0 2024-04-26T11:29:27.213 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.212+0000 7fbddfe00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbdcc0a3cc0 0x7fbdcc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.212+0000 7fbddfe00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbdcc0a3cc0 0x7fbdcc0a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009c60 tx=0x7fbdd4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:27.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.216+0000 7fbde6000700 1 -- 172.21.15.33:0/2820768265 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbdd809dc60 con 0x7fbde81033b0 2024-04-26T11:29:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:27 smithi033 ceph-mon[27845]: pgmap v414: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:27 smithi096 ceph-mon[32753]: pgmap v414: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:27.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.470+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fbde81a3cd0 con 0x7fbdcc0a3cc0 2024-04-26T11:29:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.473+0000 7fbde6000700 1 -- 172.21.15.33:0/2820768265 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fbde81a3cd0 con 0x7fbdcc0a3cc0 2024-04-26T11:29:27.475 INFO:teuthology.orchestra.run.smithi033.stdout:Backend: cephadm 2024-04-26T11:29:27.475 INFO:teuthology.orchestra.run.smithi033.stdout:Available: Yes 2024-04-26T11:29:27.475 INFO:teuthology.orchestra.run.smithi033.stdout:Paused: No 2024-04-26T11:29:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.476+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbdcc0a3cc0 msgr2=0x7fbdcc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.476+0000 7fbdede00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbdcc0a3cc0 0x7fbdcc0a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009c60 tx=0x7fbdd4009260 comp rx=0 tx=0).stop 2024-04-26T11:29:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.476+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81033b0 msgr2=0x7fbde819b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:27.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.476+0000 7fbdede00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81033b0 0x7fbde819b7c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009580 tx=0x7fbdd80095b0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.476+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 shutdown_connections 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.476+0000 7fbdede00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbdcc0a3cc0 0x7fbdcc0a6180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.477+0000 7fbdede00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde8104f50 0x7fbde81a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.477+0000 7fbdede00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbde81033b0 0x7fbde819b7c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.477+0000 7fbdede00700 1 --2- 172.21.15.33:0/2820768265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbde81045b0 0x7fbde819bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.477+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 >> 172.21.15.33:0/2820768265 conn(0x7fbde80fe950 msgr2=0x7fbde8075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.477+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 shutdown_connections 2024-04-26T11:29:27.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:27.477+0000 7fbdede00700 1 -- 172.21.15.33:0/2820768265 wait complete. 2024-04-26T11:29:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:27 smithi194 ceph-mon[33017]: pgmap v414: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:28.208 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T11:29:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:28 smithi033 ceph-mon[27845]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:28 smithi096 ceph-mon[32753]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:28 smithi194 ceph-mon[33017]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:28.682 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:29 smithi033 ceph-mon[27845]: pgmap v415: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:29 smithi096 ceph-mon[32753]: pgmap v415: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:29 smithi194 ceph-mon[33017]: pgmap v415: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.012+0000 7fe3b3200700 1 -- 172.21.15.33:0/2753251808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b41035b0 msgr2=0x7fe3b4109e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.012+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2753251808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b41035b0 0x7fe3b4109e40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac00b0a0 tx=0x7fe3ac00b3b0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.012+0000 7fe3b3200700 1 -- 172.21.15.33:0/2753251808 shutdown_connections 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.012+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2753251808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b41035b0 0x7fe3b4109e40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.012+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2753251808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 0x7fe3b4102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.012+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2753251808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b4102bf0 0x7fe3b4103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.012+0000 7fe3b3200700 1 -- 172.21.15.33:0/2753251808 >> 172.21.15.33:0/2753251808 conn(0x7fe3b40fcba0 msgr2=0x7fe3b40fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.013+0000 7fe3b3200700 1 -- 172.21.15.33:0/2753251808 shutdown_connections 2024-04-26T11:29:31.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.013+0000 7fe3b3200700 1 -- 172.21.15.33:0/2753251808 wait complete. 2024-04-26T11:29:31.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.014+0000 7fe3b3200700 1 Processor -- start 2024-04-26T11:29:31.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.014+0000 7fe3b3200700 1 -- start start 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b3200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 0x7fe3b4199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b3200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b4102bf0 0x7fe3b41999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b3200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b41035b0 0x7fe3b41a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b3200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b410cc40 con 0x7fe3b4102bf0 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b3200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b410cac0 con 0x7fe3b4101d40 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b3200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b410cdc0 con 0x7fe3b41035b0 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 0x7fe3b4199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b4102bf0 0x7fe3b41999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 0x7fe3b4199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:59082/0 (socket says 172.21.15.33:59082) 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 -- 172.21.15.33:0/2531354691 learned_addr learned my addr 172.21.15.33:0/2531354691 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:31.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b2800700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b41035b0 0x7fe3b41a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:31.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 -- 172.21.15.33:0/2531354691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b41035b0 msgr2=0x7fe3b41a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:31.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b41035b0 0x7fe3b41a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 -- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b4102bf0 msgr2=0x7fe3b41999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b4102bf0 0x7fe3b41999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.015+0000 7fe3b1e00700 1 -- 172.21.15.33:0/2531354691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b41a10d0 con 0x7fe3b4101d40 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.016+0000 7fe3b2800700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b41035b0 0x7fe3b41a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.016+0000 7fe3b1e00700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 0x7fe3b4199470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400ea80 tx=0x7fe3a400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.016+0000 7fe3aaa00700 1 -- 172.21.15.33:0/2531354691 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a400ccd0 con 0x7fe3b4101d40 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.016+0000 7fe3aaa00700 1 -- 172.21.15.33:0/2531354691 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3a40044d0 con 0x7fe3b4101d40 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.016+0000 7fe3aaa00700 1 -- 172.21.15.33:0/2531354691 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4015390 con 0x7fe3b4101d40 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.016+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b41a13c0 con 0x7fe3b4101d40 2024-04-26T11:29:31.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.016+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b41a18b0 con 0x7fe3b4101d40 2024-04-26T11:29:31.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.018+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3b406a110 con 0x7fe3b4101d40 2024-04-26T11:29:31.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.020+0000 7fe3aaa00700 1 -- 172.21.15.33:0/2531354691 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe3a4004000 con 0x7fe3b4101d40 2024-04-26T11:29:31.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.021+0000 7fe3aaa00700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3980a3c40 0x7fe3980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:31.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.021+0000 7fe3aaa00700 1 -- 172.21.15.33:0/2531354691 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe3a40d0340 con 0x7fe3b4101d40 2024-04-26T11:29:31.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.021+0000 7fe3b1400700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3980a3c40 0x7fe3980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:31.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.022+0000 7fe3b1400700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3980a3c40 0x7fe3980a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe3b41019c0 tx=0x7fe3a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:31.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.025+0000 7fe3aaa00700 1 -- 172.21.15.33:0/2531354691 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3a40092c0 con 0x7fe3b4101d40 2024-04-26T11:29:31.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.267+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe3b41097c0 con 0x7fe3980a3c40 2024-04-26T11:29:31.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:31 smithi033 ceph-mon[27845]: pgmap v416: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:31.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.310+0000 7fe3aaa00700 1 -- 172.21.15.33:0/2531354691 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fe3b41097c0 con 0x7fe3980a3c40 2024-04-26T11:29:31.313 INFO:teuthology.orchestra.run.smithi033.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T11:29:31.313 INFO:teuthology.orchestra.run.smithi033.stdout:mds.a smithi033 running (3m) 44s ago 3m 12.9M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 72590b12b61c 2024-04-26T11:29:31.313 INFO:teuthology.orchestra.run.smithi033.stdout:mds.b smithi096 running (3m) 60s ago 3m 27.2M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 3d77cf91fa21 2024-04-26T11:29:31.313 INFO:teuthology.orchestra.run.smithi033.stdout:mds.c smithi194 running (3m) 60s ago 3m 22.5M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 6d29997d79c4 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.d smithi033 running (3m) 44s ago 3m 13.2M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b c177a1841c35 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.e smithi096 running (3m) 60s ago 3m 19.2M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b c88e058eb2a1 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.f smithi194 running (3m) 60s ago 3m 21.6M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 18d645bf5b4b 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.g smithi033 running (3m) 44s ago 3m 13.0M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b cb67d2abbe02 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.h smithi096 running (3m) 60s ago 3m 19.7M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 292a5264335e 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.i smithi194 running (3m) 60s ago 3m 19.7M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 40d633ff90a8 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.j smithi033 running (3m) 44s ago 3m 13.2M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 09e1ce08f377 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.k smithi096 running (3m) 60s ago 3m 16.1M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b e770095a5864 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mds.l smithi194 running (4m) 60s ago 4m 14.6M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b e1dea4370893 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.x smithi033 *:9283,8765 running (12m) 44s ago 12m 516M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b afcc88cd5459 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.y smithi096 *:8443,8765 running (9m) 60s ago 9m 453M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 013987178ca2 2024-04-26T11:29:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.z smithi194 *:8443,8765 running (9m) 60s ago 9m 453M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 53bfce5aaddf 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a smithi033 running (12m) 44s ago 12m 49.5M 2048M 19.0.0-3267-gbc94a2a9 3a6bde214c5b fcffa444f119 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b smithi096 running (10m) 60s ago 10m 43.2M 2048M 19.0.0-3267-gbc94a2a9 3a6bde214c5b fa293eb99080 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c smithi194 running (10m) 60s ago 10m 45.5M 2048M 19.0.0-3267-gbc94a2a9 3a6bde214c5b b3fd41da0315 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 smithi033 running (9m) 44s ago 9m 56.5M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 8d42484db9d5 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 smithi096 running (9m) 60s ago 9m 48.7M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b e2d5e8c73cfc 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 smithi194 running (8m) 60s ago 8m 59.5M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b b313852c5270 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 smithi033 running (8m) 44s ago 8m 43.5M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b a2dfc3a0083e 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 smithi096 running (7m) 60s ago 7m 44.0M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 072d519b3e6a 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 smithi194 running (7m) 60s ago 7m 47.9M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 7c05238e3f85 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 smithi033 running (6m) 44s ago 6m 42.5M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 1cafd84a933b 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 smithi096 running (6m) 60s ago 6m 47.3M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 9bc3372ae84c 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 smithi194 running (5m) 60s ago 5m 55.2M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 9d4bb6c9d170 2024-04-26T11:29:31.315 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 smithi033 running (5m) 44s ago 5m 53.6M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 63d49a59760e 2024-04-26T11:29:31.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 smithi096 running (4m) 60s ago 4m 54.2M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 400d01bf070e 2024-04-26T11:29:31.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 smithi194 running (4m) 60s ago 4m 42.2M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 930756ed8c94 2024-04-26T11:29:31.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3980a3c40 msgr2=0x7fe3980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:31.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3980a3c40 0x7fe3980a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe3b41019c0 tx=0x7fe3a0009260 comp rx=0 tx=0).stop 2024-04-26T11:29:31.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 msgr2=0x7fe3b4199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:31.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 0x7fe3b4199470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400ea80 tx=0x7fe3a400ed90 comp rx=0 tx=0).stop 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 shutdown_connections 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe3980a3c40 0x7fe3980a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b41035b0 0x7fe3b41a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3b4101d40 0x7fe3b4199470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 --2- 172.21.15.33:0/2531354691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3b4102bf0 0x7fe3b41999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 >> 172.21.15.33:0/2531354691 conn(0x7fe3b40fcba0 msgr2=0x7fe3b41080a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 shutdown_connections 2024-04-26T11:29:31.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:31.313+0000 7fe3b3200700 1 -- 172.21.15.33:0/2531354691 wait complete. 2024-04-26T11:29:31.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:31 smithi194 ceph-mon[33017]: pgmap v416: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:31 smithi096 ceph-mon[32753]: pgmap v416: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:32.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T11:29:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:32 smithi033 ceph-mon[27845]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:32 smithi096 ceph-mon[32753]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:32.599 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:32 smithi194 ceph-mon[33017]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:33 smithi033 ceph-mon[27845]: pgmap v417: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:33 smithi096 ceph-mon[32753]: pgmap v417: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:33 smithi194 ceph-mon[33017]: pgmap v417: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 -- 172.21.15.33:0/1927576847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c104660 msgr2=0x7f8c8c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 --2- 172.21.15.33:0/1927576847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c104660 0x7f8c8c106af0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c740098f0 tx=0x7f8c74009c00 comp rx=0 tx=0).stop 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 -- 172.21.15.33:0/1927576847 shutdown_connections 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 --2- 172.21.15.33:0/1927576847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c107030 0x7f8c8c1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 --2- 172.21.15.33:0/1927576847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c8c101cf0 0x7f8c8c104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 --2- 172.21.15.33:0/1927576847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c104660 0x7f8c8c106af0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 -- 172.21.15.33:0/1927576847 >> 172.21.15.33:0/1927576847 conn(0x7f8c8c0fd910 msgr2=0x7f8c8c0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 -- 172.21.15.33:0/1927576847 shutdown_connections 2024-04-26T11:29:34.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.720+0000 7f8c93a00700 1 -- 172.21.15.33:0/1927576847 wait complete. 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.721+0000 7f8c93a00700 1 Processor -- start 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.722+0000 7f8c93a00700 1 -- start start 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.722+0000 7f8c93a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c101cf0 0x7f8c8c19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.722+0000 7f8c93a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c8c104660 0x7f8c8c1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.722+0000 7f8c93a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c107030 0x7f8c8c1a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.722+0000 7f8c93a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c1154b0 con 0x7f8c8c101cf0 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.722+0000 7f8c93a00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c115330 con 0x7f8c8c107030 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.722+0000 7f8c93a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c115630 con 0x7f8c8c104660 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c91000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c101cf0 0x7f8c8c19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c91000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c101cf0 0x7f8c8c19fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42400/0 (socket says 172.21.15.33:42400) 2024-04-26T11:29:34.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c91000700 1 -- 172.21.15.33:0/475587586 learned_addr learned my addr 172.21.15.33:0/475587586 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c83e00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c8c104660 0x7f8c8c1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c91a00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c107030 0x7f8c8c1a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c83e00700 1 -- 172.21.15.33:0/475587586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c107030 msgr2=0x7f8c8c1a7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c83e00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c107030 0x7f8c8c1a7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c83e00700 1 -- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c101cf0 msgr2=0x7f8c8c19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c83e00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c101cf0 0x7f8c8c19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c83e00700 1 -- 172.21.15.33:0/475587586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8c1a7830 con 0x7f8c8c104660 2024-04-26T11:29:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.723+0000 7f8c91000700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c101cf0 0x7f8c8c19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:34.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.724+0000 7f8c83e00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c8c104660 0x7f8c8c1a0160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8c74000c00 tx=0x7f8c74005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:34.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.724+0000 7f8c8a000700 1 -- 172.21.15.33:0/475587586 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74025070 con 0x7f8c8c104660 2024-04-26T11:29:34.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.724+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8c1a7ac0 con 0x7f8c8c104660 2024-04-26T11:29:34.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.724+0000 7f8c8a000700 1 -- 172.21.15.33:0/475587586 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c74005670 con 0x7f8c8c104660 2024-04-26T11:29:34.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.724+0000 7f8c8a000700 1 -- 172.21.15.33:0/475587586 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74003ae0 con 0x7f8c8c104660 2024-04-26T11:29:34.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.724+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8c1a8020 con 0x7f8c8c104660 2024-04-26T11:29:34.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.725+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8c06a110 con 0x7f8c8c104660 2024-04-26T11:29:34.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.727+0000 7f8c8a000700 1 -- 172.21.15.33:0/475587586 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8c74028070 con 0x7f8c8c104660 2024-04-26T11:29:34.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.729+0000 7f8c8a000700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8c700a3c40 0x7f8c700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:34.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.729+0000 7f8c8a000700 1 -- 172.21.15.33:0/475587586 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8c740d0b50 con 0x7f8c8c104660 2024-04-26T11:29:34.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.729+0000 7f8c91000700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8c700a3c40 0x7f8c700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:34.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.730+0000 7f8c91000700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8c700a3c40 0x7f8c700a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c005950 tx=0x7f8c7c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:34.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.733+0000 7f8c8a000700 1 -- 172.21.15.33:0/475587586 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c7405cff0 con 0x7f8c8c104660 2024-04-26T11:29:34.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.971+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8c8c063860 con 0x7f8c700a3c40 2024-04-26T11:29:34.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.981+0000 7f8c8a000700 1 -- 172.21.15.33:0/475587586 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8c8c063860 con 0x7f8c700a3c40 2024-04-26T11:29:34.983 INFO:teuthology.orchestra.run.smithi033.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T11:29:34.983 INFO:teuthology.orchestra.run.smithi033.stdout:mds.all 12/12 63s ago 4m smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-26T11:29:34.983 INFO:teuthology.orchestra.run.smithi033.stdout:mgr 3/3 63s ago 10m smithi033=x;smithi096=y;smithi194=z;count:3 2024-04-26T11:29:34.983 INFO:teuthology.orchestra.run.smithi033.stdout:mon 3/3 63s ago 10m smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi194:172.21.15.194=c;count:3 2024-04-26T11:29:34.983 INFO:teuthology.orchestra.run.smithi033.stdout:osd 12 63s ago - 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8c700a3c40 msgr2=0x7f8c700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8c700a3c40 0x7f8c700a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c005950 tx=0x7f8c7c009310 comp rx=0 tx=0).stop 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c8c104660 msgr2=0x7f8c8c1a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c8c104660 0x7f8c8c1a0160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8c74000c00 tx=0x7f8c74005270 comp rx=0 tx=0).stop 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 shutdown_connections 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f8c700a3c40 0x7f8c700a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c8c107030 0x7f8c8c1a7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c8c101cf0 0x7f8c8c19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 --2- 172.21.15.33:0/475587586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c8c104660 0x7f8c8c1a0160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:34.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 >> 172.21.15.33:0/475587586 conn(0x7f8c8c0fd910 msgr2=0x7f8c8c1027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:34.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 shutdown_connections 2024-04-26T11:29:34.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:34.984+0000 7f8c93a00700 1 -- 172.21.15.33:0/475587586 wait complete. 2024-04-26T11:29:35.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:35 smithi033 ceph-mon[27845]: pgmap v418: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:35 smithi096 ceph-mon[32753]: pgmap v418: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:35 smithi194 ceph-mon[33017]: pgmap v418: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:35.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T11:29:36.242 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:36.422 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:36 smithi033 ceph-mon[27845]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:36 smithi096 ceph-mon[32753]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:36 smithi194 ceph-mon[33017]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:37 smithi033 ceph-mon[27845]: pgmap v419: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:37 smithi096 ceph-mon[32753]: pgmap v419: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:37 smithi194 ceph-mon[33017]: pgmap v419: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:38.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 -- 172.21.15.33:0/1705798223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c103f10 msgr2=0x7f829c106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 --2- 172.21.15.33:0/1705798223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c103f10 0x7f829c106300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f828c0098f0 tx=0x7f828c009c00 comp rx=0 tx=0).stop 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 -- 172.21.15.33:0/1705798223 shutdown_connections 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 --2- 172.21.15.33:0/1705798223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c109170 0x7f829c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 --2- 172.21.15.33:0/1705798223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c103f10 0x7f829c106300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 --2- 172.21.15.33:0/1705798223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829c106840 0x7f829c108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 -- 172.21.15.33:0/1705798223 >> 172.21.15.33:0/1705798223 conn(0x7f829c0fd920 msgr2=0x7f829c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 -- 172.21.15.33:0/1705798223 shutdown_connections 2024-04-26T11:29:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.469+0000 7f82a2600700 1 -- 172.21.15.33:0/1705798223 wait complete. 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.470+0000 7f82a2600700 1 Processor -- start 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f82a2600700 1 -- start start 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f82a2600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c103f10 0x7f829c19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f82a2600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829c106840 0x7f829c19bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f82a2600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c109170 0x7f829c1a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f82a2600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10ef00 con 0x7f829c106840 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f82a2600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10ed80 con 0x7f829c109170 2024-04-26T11:29:38.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f82a2600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10f080 con 0x7f829c103f10 2024-04-26T11:29:38.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f829b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c103f10 0x7f829c19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:38.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f8292a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829c106840 0x7f829c19bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:38.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f829b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c103f10 0x7f829c19b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:56426/0 (socket says 172.21.15.33:56426) 2024-04-26T11:29:38.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.471+0000 7f829b400700 1 -- 172.21.15.33:0/668201637 learned_addr learned my addr 172.21.15.33:0/668201637 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:38.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.472+0000 7f829be00700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c109170 0x7f829c1a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:38.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.472+0000 7f829b400700 1 -- 172.21.15.33:0/668201637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c109170 msgr2=0x7f829c1a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.472+0000 7f829b400700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c109170 0x7f829c1a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.472+0000 7f829b400700 1 -- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829c106840 msgr2=0x7f829c19bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.472+0000 7f829b400700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829c106840 0x7f829c19bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.472+0000 7f829b400700 1 -- 172.21.15.33:0/668201637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c1a34e0 con 0x7f829c103f10 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.472+0000 7f829b400700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c103f10 0x7f829c19b880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f828c0059d0 tx=0x7f828c010f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.473+0000 7f8298c00700 1 -- 172.21.15.33:0/668201637 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c0054b0 con 0x7f829c103f10 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.473+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c1a3770 con 0x7f829c103f10 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.473+0000 7f8298c00700 1 -- 172.21.15.33:0/668201637 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f828c005650 con 0x7f829c103f10 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.473+0000 7f8298c00700 1 -- 172.21.15.33:0/668201637 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c004d00 con 0x7f829c103f10 2024-04-26T11:29:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.473+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c1a3c20 con 0x7f829c103f10 2024-04-26T11:29:38.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.474+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f829c06a110 con 0x7f829c103f10 2024-04-26T11:29:38.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.476+0000 7f8298c00700 1 -- 172.21.15.33:0/668201637 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f828c00bb10 con 0x7f829c103f10 2024-04-26T11:29:38.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.477+0000 7f8298c00700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f82800a3c40 0x7f82800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:38.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.477+0000 7f8298c00700 1 -- 172.21.15.33:0/668201637 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f828c0d1540 con 0x7f829c103f10 2024-04-26T11:29:38.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.478+0000 7f8292a00700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f82800a3c40 0x7f82800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:38.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.478+0000 7f8292a00700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f82800a3c40 0x7f82800a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8284005fd0 tx=0x7f8284009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:38.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.480+0000 7f8298c00700 1 -- 172.21.15.33:0/668201637 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f828c013030 con 0x7f829c103f10 2024-04-26T11:29:38.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.721+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f829c102710 con 0x7f82800a3c40 2024-04-26T11:29:38.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.723+0000 7f8298c00700 1 -- 172.21.15.33:0/668201637 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f829c102710 con 0x7f82800a3c40 2024-04-26T11:29:38.725 INFO:teuthology.orchestra.run.smithi033.stdout:HOST ADDR LABELS STATUS 2024-04-26T11:29:38.725 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033 172.21.15.33 2024-04-26T11:29:38.726 INFO:teuthology.orchestra.run.smithi033.stdout:smithi096 172.21.15.96 2024-04-26T11:29:38.726 INFO:teuthology.orchestra.run.smithi033.stdout:smithi194 172.21.15.194 2024-04-26T11:29:38.726 INFO:teuthology.orchestra.run.smithi033.stdout:3 hosts in cluster 2024-04-26T11:29:38.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.726+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f82800a3c40 msgr2=0x7f82800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.726+0000 7f82a2600700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f82800a3c40 0x7f82800a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8284005fd0 tx=0x7f8284009310 comp rx=0 tx=0).stop 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.726+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c103f10 msgr2=0x7f829c19b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.726+0000 7f82a2600700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c103f10 0x7f829c19b880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f828c0059d0 tx=0x7f828c010f70 comp rx=0 tx=0).stop 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 shutdown_connections 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f82800a3c40 0x7f82800a6100 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f829c109170 0x7f829c1a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f829c103f10 0x7f829c19b880 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 --2- 172.21.15.33:0/668201637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f829c106840 0x7f829c19bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 >> 172.21.15.33:0/668201637 conn(0x7f829c0fd920 msgr2=0x7f829c102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 shutdown_connections 2024-04-26T11:29:38.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:38.727+0000 7f82a2600700 1 -- 172.21.15.33:0/668201637 wait complete. 2024-04-26T11:29:39.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:39 smithi033 ceph-mon[27845]: pgmap v420: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:39 smithi096 ceph-mon[32753]: pgmap v420: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:39.517 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T11:29:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:39 smithi194 ceph-mon[33017]: pgmap v420: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:39.988 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:40 smithi033 ceph-mon[27845]: from='client.24925 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:40 smithi194 ceph-mon[33017]: from='client.24925 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:40 smithi096 ceph-mon[32753]: from='client.24925 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:41.391 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:41 smithi033 ceph-mon[27845]: pgmap v421: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:41.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:41 smithi096 ceph-mon[32753]: pgmap v421: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:41 smithi194 ceph-mon[33017]: pgmap v421: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:42.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 -- 172.21.15.33:0/3433418792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590103f10 msgr2=0x7fb590106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:42.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 --2- 172.21.15.33:0/3433418792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590103f10 0x7fb590106300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb580009990 tx=0x7fb580009ca0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 -- 172.21.15.33:0/3433418792 shutdown_connections 2024-04-26T11:29:42.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 --2- 172.21.15.33:0/3433418792 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb590109170 0x7fb59010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 --2- 172.21.15.33:0/3433418792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590103f10 0x7fb590106300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 --2- 172.21.15.33:0/3433418792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590106840 0x7fb590108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 -- 172.21.15.33:0/3433418792 >> 172.21.15.33:0/3433418792 conn(0x7fb5900fd920 msgr2=0x7fb5900ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:42.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 -- 172.21.15.33:0/3433418792 shutdown_connections 2024-04-26T11:29:42.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.225+0000 7fb595e00700 1 -- 172.21.15.33:0/3433418792 wait complete. 2024-04-26T11:29:42.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.226+0000 7fb595e00700 1 Processor -- start 2024-04-26T11:29:42.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb595e00700 1 -- start start 2024-04-26T11:29:42.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb595e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590103f10 0x7fb5901996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:42.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb595e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb590106840 0x7fb590199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:42.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb595e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590109170 0x7fb5901a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:42.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb595e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb59010ef50 con 0x7fb590109170 2024-04-26T11:29:42.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb595e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb59010edd0 con 0x7fb590103f10 2024-04-26T11:29:42.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb595e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb59010f0d0 con 0x7fb590106840 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb58ea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590103f10 0x7fb5901996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb58ea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590103f10 0x7fb5901996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42324/0 (socket says 172.21.15.33:42324) 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb58ea00700 1 -- 172.21.15.33:0/1153672202 learned_addr learned my addr 172.21.15.33:0/1153672202 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb586000700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb590106840 0x7fb590199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.227+0000 7fb58f400700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590109170 0x7fb5901a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.228+0000 7fb58ea00700 1 -- 172.21.15.33:0/1153672202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb590106840 msgr2=0x7fb590199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.228+0000 7fb58ea00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb590106840 0x7fb590199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.228+0000 7fb58ea00700 1 -- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590109170 msgr2=0x7fb5901a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.228+0000 7fb58ea00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590109170 0x7fb5901a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.228+0000 7fb58ea00700 1 -- 172.21.15.33:0/1153672202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5901a12d0 con 0x7fb590103f10 2024-04-26T11:29:42.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.228+0000 7fb58ea00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590103f10 0x7fb5901996c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb5800098e0 tx=0x7fb5800057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.229+0000 7fb594a00700 1 -- 172.21.15.33:0/1153672202 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb580026070 con 0x7fb590103f10 2024-04-26T11:29:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.229+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5901a1560 con 0x7fb590103f10 2024-04-26T11:29:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.229+0000 7fb594a00700 1 -- 172.21.15.33:0/1153672202 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb580021410 con 0x7fb590103f10 2024-04-26T11:29:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.229+0000 7fb594a00700 1 -- 172.21.15.33:0/1153672202 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb580017650 con 0x7fb590103f10 2024-04-26T11:29:42.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.229+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5901a1a10 con 0x7fb590103f10 2024-04-26T11:29:42.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.230+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb59006a110 con 0x7fb590103f10 2024-04-26T11:29:42.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.232+0000 7fb594a00700 1 -- 172.21.15.33:0/1153672202 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb580029070 con 0x7fb590103f10 2024-04-26T11:29:42.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.233+0000 7fb594a00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb5740a3c40 0x7fb5740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:42.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.233+0000 7fb594a00700 1 -- 172.21.15.33:0/1153672202 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb5800d1340 con 0x7fb590103f10 2024-04-26T11:29:42.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.233+0000 7fb586000700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb5740a3c40 0x7fb5740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:42.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.234+0000 7fb586000700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb5740a3c40 0x7fb5740a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb578009680 tx=0x7fb578009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:42.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.238+0000 7fb594a00700 1 -- 172.21.15.33:0/1153672202 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb58005dff0 con 0x7fb590103f10 2024-04-26T11:29:42.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.478+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 --> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fb590063860 con 0x7fb5740a3c40 2024-04-26T11:29:42.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.481+0000 7fb594a00700 1 -- 172.21.15.33:0/1153672202 <== mgr.14152 v2:172.21.15.33:6800/657098496 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fb590063860 con 0x7fb5740a3c40 2024-04-26T11:29:42.483 INFO:teuthology.orchestra.run.smithi033.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T11:29:42.483 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-04-26T11:29:42.483 INFO:teuthology.orchestra.run.smithi033.stdout:smithi096 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NH400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T11:29:42.483 INFO:teuthology.orchestra.run.smithi033.stdout:smithi194 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6240002A400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T11:29:42.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.484+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb5740a3c40 msgr2=0x7fb5740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:42.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.484+0000 7fb595e00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb5740a3c40 0x7fb5740a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb578009680 tx=0x7fb578009260 comp rx=0 tx=0).stop 2024-04-26T11:29:42.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.484+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590103f10 msgr2=0x7fb5901996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:42.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.484+0000 7fb595e00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590103f10 0x7fb5901996c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb5800098e0 tx=0x7fb5800057a0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.484+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 shutdown_connections 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.484+0000 7fb595e00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fb5740a3c40 0x7fb5740a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.484+0000 7fb595e00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590109170 0x7fb5901a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.485+0000 7fb595e00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb590103f10 0x7fb5901996c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.485+0000 7fb595e00700 1 --2- 172.21.15.33:0/1153672202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb590106840 0x7fb590199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.485+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 >> 172.21.15.33:0/1153672202 conn(0x7fb5900fd920 msgr2=0x7fb590105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.485+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 shutdown_connections 2024-04-26T11:29:42.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:42.485+0000 7fb595e00700 1 -- 172.21.15.33:0/1153672202 wait complete. 2024-04-26T11:29:43.316 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T11:29:43.325 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:29:43.325 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T11:29:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:43 smithi194 ceph-mon[33017]: pgmap v422: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:43 smithi194 ceph-mon[33017]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:43 smithi033 ceph-mon[27845]: pgmap v422: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:43 smithi033 ceph-mon[27845]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:43 smithi096 ceph-mon[32753]: pgmap v422: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:43 smithi096 ceph-mon[32753]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:29:43.808 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:45.518 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:45 smithi033 ceph-mon[27845]: pgmap v423: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:29:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:45 smithi194 ceph-mon[33017]: pgmap v423: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:29:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:45 smithi096 ceph-mon[32753]: pgmap v423: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:29:46.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 -- 172.21.15.33:0/3564957568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe14010b280 msgr2=0x7fe14010d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:46.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 --2- 172.21.15.33:0/3564957568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe14010b280 0x7fe14010d710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe138009f90 tx=0x7fe13800b330 comp rx=0 tx=0).stop 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 -- 172.21.15.33:0/3564957568 shutdown_connections 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 --2- 172.21.15.33:0/3564957568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe14010b280 0x7fe14010d710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 --2- 172.21.15.33:0/3564957568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 0x7fe140075850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 --2- 172.21.15.33:0/3564957568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe140075e20 0x7fe1400762a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 -- 172.21.15.33:0/3564957568 >> 172.21.15.33:0/3564957568 conn(0x7fe1400fe960 msgr2=0x7fe140100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 -- 172.21.15.33:0/3564957568 shutdown_connections 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.016+0000 7fe13fe00700 1 -- 172.21.15.33:0/3564957568 wait complete. 2024-04-26T11:29:46.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 Processor -- start 2024-04-26T11:29:46.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 -- start start 2024-04-26T11:29:46.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe140075e20 0x7fe14019b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:46.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 0x7fe14019bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14010b280 0x7fe1401a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe140111020 con 0x7fe1400773f0 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe140110ea0 con 0x7fe14010b280 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.018+0000 7fe13fe00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401111a0 con 0x7fe140075e20 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 0x7fe14019bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 0x7fe14019bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53182/0 (socket says 172.21.15.33:53182) 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 -- 172.21.15.33:0/1627450710 learned_addr learned my addr 172.21.15.33:0/1627450710 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe13ea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe140075e20 0x7fe14019b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe13f400700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14010b280 0x7fe1401a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:46.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 -- 172.21.15.33:0/1627450710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe140075e20 msgr2=0x7fe14019b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:46.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe140075e20 0x7fe14019b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 -- 172.21.15.33:0/1627450710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14010b280 msgr2=0x7fe1401a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:46.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14010b280 0x7fe1401a2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe136000700 1 -- 172.21.15.33:0/1627450710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1401a3280 con 0x7fe1400773f0 2024-04-26T11:29:46.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.019+0000 7fe13ea00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe140075e20 0x7fe14019b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:46.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.020+0000 7fe136000700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 0x7fe14019bb60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe12800e9f0 tx=0x7fe12800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:46.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.020+0000 7fe137e00700 1 -- 172.21.15.33:0/1627450710 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12800cbd0 con 0x7fe1400773f0 2024-04-26T11:29:46.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.020+0000 7fe137e00700 1 -- 172.21.15.33:0/1627450710 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe12800ee90 con 0x7fe1400773f0 2024-04-26T11:29:46.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.020+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1401a3570 con 0x7fe1400773f0 2024-04-26T11:29:46.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.020+0000 7fe137e00700 1 -- 172.21.15.33:0/1627450710 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe128005590 con 0x7fe1400773f0 2024-04-26T11:29:46.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.020+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1401a3ab0 con 0x7fe1400773f0 2024-04-26T11:29:46.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.021+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe14006a110 con 0x7fe1400773f0 2024-04-26T11:29:46.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.023+0000 7fe137e00700 1 -- 172.21.15.33:0/1627450710 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe128004750 con 0x7fe1400773f0 2024-04-26T11:29:46.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.025+0000 7fe137e00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe1240a3cc0 0x7fe1240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:46.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.025+0000 7fe137e00700 1 -- 172.21.15.33:0/1627450710 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe1280d0850 con 0x7fe1400773f0 2024-04-26T11:29:46.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.025+0000 7fe13ea00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe1240a3cc0 0x7fe1240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:46.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.026+0000 7fe13ea00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe1240a3cc0 0x7fe1240a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe130009750 tx=0x7fe130009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:46.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.028+0000 7fe137e00700 1 -- 172.21.15.33:0/1627450710 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe128053ff0 con 0x7fe1400773f0 2024-04-26T11:29:46.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.308+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fe14002e490 con 0x7fe1400773f0 2024-04-26T11:29:46.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.310+0000 7fe137e00700 1 -- 172.21.15.33:0/1627450710 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3046 (secure 0 0 0) 0x7fe12809d1f0 con 0x7fe1400773f0 2024-04-26T11:29:46.312 INFO:teuthology.orchestra.run.smithi033.stdout:e25 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:legacy client fscid: 1 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:Filesystem 'cephfs' (1) 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:fs_name cephfs 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 25 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-04-26T11:26:32.073203+0000 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-04-26T11:26:56.732811+0000 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:tableserver 0 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:root 0 2024-04-26T11:29:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:session_timeout 300 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:session_autoclose 300 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:max_file_size 1099511627776 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:max_xattr_size 65536 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:required_client_features {} 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:last_failure 0 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:last_failure_osd_epoch 0 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:max_mds 3 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:in 0,1,2 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:up {0=24527,1=24488,2=24484} 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:failed 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:damaged 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:stopped 2024-04-26T11:29:46.314 INFO:teuthology.orchestra.run.smithi033.stdout:data_pools [3] 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:metadata_pool 2 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:inline_data disabled 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:balancer 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:bal_rank_mask -1 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:standby_count_wanted 0 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:qdb_cluster leader: 24527 members: 24488,24527,24484 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.b{0:24527} state up:active seq 7 addr [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.c{0:24472} state up:standby-replay seq 1 addr [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.h{1:24488} state up:active seq 15 addr [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.e{1:24470} state up:standby-replay seq 1 addr [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.i{2:24484} state up:active seq 15 addr [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.f{2:24460} state up:standby-replay seq 1 addr [v2:172.21.15.194:6834/1403509498,v1:172.21.15.194:6835/1403509498] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.315 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout:Standby daemons: 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.j{-1:14358} state up:standby seq 1 addr [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.g{-1:14394} state up:standby seq 1 addr [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.a{-1:14400} state up:standby seq 1 addr [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.l{-1:24424} state up:standby seq 1 addr [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.k{-1:24440} state up:standby seq 1 addr [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.316 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.d{-1:24458} state up:standby seq 1 addr [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T11:29:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.313+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe1240a3cc0 msgr2=0x7fe1240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.313+0000 7fe13fe00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe1240a3cc0 0x7fe1240a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe130009750 tx=0x7fe130009310 comp rx=0 tx=0).stop 2024-04-26T11:29:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.313+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 msgr2=0x7fe14019bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.313+0000 7fe13fe00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 0x7fe14019bb60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe12800e9f0 tx=0x7fe12800ed00 comp rx=0 tx=0).stop 2024-04-26T11:29:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 shutdown_connections 2024-04-26T11:29:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fe1240a3cc0 0x7fe1240a6180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe14010b280 0x7fe1401a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe140075e20 0x7fe14019b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 --2- 172.21.15.33:0/1627450710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe1400773f0 0x7fe14019bb60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:46.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 >> 172.21.15.33:0/1627450710 conn(0x7fe1400fe960 msgr2=0x7fe1401071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:46.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 shutdown_connections 2024-04-26T11:29:46.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:46.314+0000 7fe13fe00700 1 -- 172.21.15.33:0/1627450710 wait complete. 2024-04-26T11:29:46.318 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-04-26T11:29:47.155 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T11:29:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:47 smithi033 ceph-mon[27845]: pgmap v424: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:47 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1627450710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T11:29:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:47 smithi194 ceph-mon[33017]: pgmap v424: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:47 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1627450710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T11:29:47.625 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:47 smithi096 ceph-mon[32753]: pgmap v424: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:47 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1627450710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T11:29:48.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:48 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:48 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:29:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:48 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:29:49.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:49 smithi033 ceph-mon[27845]: pgmap v425: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:49.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:49 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:49 smithi096 ceph-mon[32753]: pgmap v425: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:49 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.723+0000 7f2ef0c00700 1 -- 172.21.15.33:0/374163466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec103f70 msgr2=0x7f2eec106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.723+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/374163466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec103f70 0x7f2eec106360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009990 tx=0x7f2edc009ca0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.724+0000 7f2ef0c00700 1 -- 172.21.15.33:0/374163466 shutdown_connections 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.724+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/374163466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2eec1091d0 0x7f2eec10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.724+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/374163466 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec103f70 0x7f2eec106360 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.724+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/374163466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec1068a0 0x7f2eec108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.724+0000 7f2ef0c00700 1 -- 172.21.15.33:0/374163466 >> 172.21.15.33:0/374163466 conn(0x7f2eec0fd920 msgr2=0x7f2eec0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.724+0000 7f2ef0c00700 1 -- 172.21.15.33:0/374163466 shutdown_connections 2024-04-26T11:29:49.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.724+0000 7f2ef0c00700 1 -- 172.21.15.33:0/374163466 wait complete. 2024-04-26T11:29:49.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.725+0000 7f2ef0c00700 1 Processor -- start 2024-04-26T11:29:49.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.725+0000 7f2ef0c00700 1 -- start start 2024-04-26T11:29:49.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ef0c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec103f70 0x7f2eec199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:49.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ef0c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec1068a0 0x7f2eec199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:49.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ef0c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2eec1091d0 0x7f2eec1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:49.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ef0c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10ef50 con 0x7f2eec1068a0 2024-04-26T11:29:49.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ef0c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10edd0 con 0x7f2eec103f70 2024-04-26T11:29:49.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ef0c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10f0d0 con 0x7f2eec1091d0 2024-04-26T11:29:49.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec103f70 0x7f2eec199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:49.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec1068a0 0x7f2eec199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec103f70 0x7f2eec199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42370/0 (socket says 172.21.15.33:42370) 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 -- 172.21.15.33:0/3130056030 learned_addr learned my addr 172.21.15.33:0/3130056030 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2eea800700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2eec1091d0 0x7f2eec1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 -- 172.21.15.33:0/3130056030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2eec1091d0 msgr2=0x7f2eec1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2eec1091d0 0x7f2eec1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 -- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec1068a0 msgr2=0x7f2eec199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec1068a0 0x7f2eec199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.726+0000 7f2ee9e00700 1 -- 172.21.15.33:0/3130056030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec1a12b0 con 0x7f2eec103f70 2024-04-26T11:29:49.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.727+0000 7f2ee9e00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec103f70 0x7f2eec199650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000c00 tx=0x7f2edc0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:49.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.727+0000 7f2ee2a00700 1 -- 172.21.15.33:0/3130056030 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc026070 con 0x7f2eec103f70 2024-04-26T11:29:49.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.727+0000 7f2ee2a00700 1 -- 172.21.15.33:0/3130056030 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2edc021410 con 0x7f2eec103f70 2024-04-26T11:29:49.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.727+0000 7f2ee2a00700 1 -- 172.21.15.33:0/3130056030 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc017650 con 0x7f2eec103f70 2024-04-26T11:29:49.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.727+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec1a1540 con 0x7f2eec103f70 2024-04-26T11:29:49.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.727+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec1a1940 con 0x7f2eec103f70 2024-04-26T11:29:49.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.728+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec06a110 con 0x7f2eec103f70 2024-04-26T11:29:49.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.732+0000 7f2ee2a00700 1 -- 172.21.15.33:0/3130056030 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2edc009d10 con 0x7f2eec103f70 2024-04-26T11:29:49.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.733+0000 7f2ee2a00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2ed00a3cc0 0x7f2ed00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:49.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.733+0000 7f2ee2a00700 1 -- 172.21.15.33:0/3130056030 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f2edc0d1100 con 0x7f2eec103f70 2024-04-26T11:29:49.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.734+0000 7f2ee9400700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2ed00a3cc0 0x7f2ed00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:49.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.734+0000 7f2ee2a00700 1 -- 172.21.15.33:0/3130056030 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2edc013070 con 0x7f2eec103f70 2024-04-26T11:29:49.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.734+0000 7f2ee9400700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2ed00a3cc0 0x7f2ed00a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009680 tx=0x7f2ed8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:49 smithi194 ceph-mon[33017]: pgmap v425: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:49 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:49.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.966+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f2eec064030 con 0x7f2eec103f70 2024-04-26T11:29:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.968+0000 7f2ee2a00700 1 -- 172.21.15.33:0/3130056030 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5290 (secure 0 0 0) 0x7f2edc09db30 con 0x7f2eec103f70 2024-04-26T11:29:49.970 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 90 2024-04-26T11:29:49.970 INFO:teuthology.orchestra.run.smithi033.stdout:fsid 5cc7f146-03be-11ef-bc93-c7b262605968 2024-04-26T11:29:49.970 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-04-26T11:17:08.866250+0000 2024-04-26T11:29:49.970 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-04-26T11:27:19.230568+0000 2024-04-26T11:29:49.970 INFO:teuthology.orchestra.run.smithi033.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T11:29:49.970 INFO:teuthology.orchestra.run.smithi033.stdout:crush_version 26 2024-04-26T11:29:49.970 INFO:teuthology.orchestra.run.smithi033.stdout:full_ratio 0.95 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:backfillfull_ratio 0.9 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:nearfull_ratio 0.85 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:require_min_compat_client luminous 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:min_compat_client luminous 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:require_osd_release squid 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:stretch_mode_enabled false 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:max_osd 12 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 up in weight 1 up_from 8 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6802/2032124728,v1:172.21.15.33:6803/2032124728] [v2:172.21.15.33:6804/2032124728,v1:172.21.15.33:6805/2032124728] exists,up 57d1df23-f689-46b9-a67c-a8846bd7c120 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6800/2891673076,v1:172.21.15.96:6801/2891673076] [v2:172.21.15.96:6802/2891673076,v1:172.21.15.96:6803/2891673076] exists,up 8d3789d6-b7e0-41b5-b81f-5f2f2492d663 2024-04-26T11:29:49.971 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6800/3454447938,v1:172.21.15.194:6801/3454447938] [v2:172.21.15.194:6802/3454447938,v1:172.21.15.194:6803/3454447938] exists,up 05e220c5-ccfa-45ed-97a6-d2281a055fa6 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6810/2572626709,v1:172.21.15.33:6811/2572626709] [v2:172.21.15.33:6812/2572626709,v1:172.21.15.33:6813/2572626709] exists,up f814ef05-a751-446c-b5c9-319aa93738c1 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6808/237079136,v1:172.21.15.96:6809/237079136] [v2:172.21.15.96:6810/237079136,v1:172.21.15.96:6811/237079136] exists,up f5f7754c-be64-4277-8ce2-6d3117800d71 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 up in weight 1 up_from 38 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6808/769884619,v1:172.21.15.194:6809/769884619] [v2:172.21.15.194:6810/769884619,v1:172.21.15.194:6811/769884619] exists,up a138a4e3-39a6-4646-9eb8-27184d6dda61 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 up in weight 1 up_from 46 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6818/3152193593,v1:172.21.15.33:6819/3152193593] [v2:172.21.15.33:6820/3152193593,v1:172.21.15.33:6821/3152193593] exists,up 79871b86-d637-4952-b923-615080fad108 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 up in weight 1 up_from 51 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6816/1631656381,v1:172.21.15.96:6817/1631656381] [v2:172.21.15.96:6818/1631656381,v1:172.21.15.96:6819/1631656381] exists,up 514a0fd5-1191-4c6c-8724-d13762ffb9bd 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 up in weight 1 up_from 59 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6816/529571686,v1:172.21.15.194:6817/529571686] [v2:172.21.15.194:6818/529571686,v1:172.21.15.194:6819/529571686] exists,up 6b44bf32-22e5-4c02-8be5-79db01d9f8a6 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 up in weight 1 up_from 67 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6826/1639452107,v1:172.21.15.33:6827/1639452107] [v2:172.21.15.33:6828/1639452107,v1:172.21.15.33:6829/1639452107] exists,up 9a0a3c0f-0405-46c7-bff5-886d3232f60c 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 up in weight 1 up_from 72 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6824/2356422092,v1:172.21.15.96:6825/2356422092] [v2:172.21.15.96:6826/2356422092,v1:172.21.15.96:6827/2356422092] exists,up 9a7b1104-145f-4393-bf38-a75dcaf06b8d 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 up in weight 1 up_from 80 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6824/2370303582,v1:172.21.15.194:6825/2370303582] [v2:172.21.15.194:6826/2370303582,v1:172.21.15.194:6827/2370303582] exists,up 58375031-d243-4316-9c22-1edf99890668 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.4 [2,11] 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.6 [8,11] 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.0 [5,9] 2024-04-26T11:29:49.972 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.1b [5,9] 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.1d [4,6] 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.1e [3,9] 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.21 [4,8] 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.2a [3,8] 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.3a [4,7] 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2640535669 expires 2024-04-27T11:18:17.660858+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/580146805 expires 2024-04-27T11:18:17.660858+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6801/334919779 expires 2024-04-27T11:18:17.660858+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/334919779 expires 2024-04-27T11:18:17.660858+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2936895268 expires 2024-04-27T11:17:42.466956+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2839525448 expires 2024-04-27T11:17:42.466956+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/1725605750 expires 2024-04-27T11:17:42.466956+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6801/927259861 expires 2024-04-27T11:17:42.466956+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/927259861 expires 2024-04-27T11:17:42.466956+0000 2024-04-26T11:29:49.973 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/1539238418 expires 2024-04-27T11:18:17.660858+0000 2024-04-26T11:29:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2ed00a3cc0 msgr2=0x7f2ed00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2ed00a3cc0 0x7f2ed00a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009680 tx=0x7f2ed8009260 comp rx=0 tx=0).stop 2024-04-26T11:29:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec103f70 msgr2=0x7f2eec199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec103f70 0x7f2eec199650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000c00 tx=0x7f2edc0047b0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 shutdown_connections 2024-04-26T11:29:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f2ed00a3cc0 0x7f2ed00a6180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2eec1091d0 0x7f2eec1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2eec103f70 0x7f2eec199650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 --2- 172.21.15.33:0/3130056030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2eec1068a0 0x7f2eec199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 >> 172.21.15.33:0/3130056030 conn(0x7f2eec0fd920 msgr2=0x7f2eec105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 shutdown_connections 2024-04-26T11:29:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:49.971+0000 7f2ef0c00700 1 -- 172.21.15.33:0/3130056030 wait complete. 2024-04-26T11:29:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:29:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:29:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:50 smithi033 ceph-mon[27845]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:50 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3130056030' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T11:29:50.770 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T11:29:50.781 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:29:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:29:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:29:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:50 smithi194 ceph-mon[33017]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:50 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3130056030' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T11:29:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:29:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:29:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:50 smithi096 ceph-mon[32753]: from='mgr.14152 172.21.15.33:0/704581522' entity='mgr.x' 2024-04-26T11:29:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:50 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3130056030' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T11:29:51.238 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:51 smithi033 ceph-mon[27845]: pgmap v426: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:51 smithi194 ceph-mon[33017]: pgmap v426: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:51 smithi096 ceph-mon[32753]: pgmap v426: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:29:53.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3000472215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48105640 msgr2=0x7fbe48105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:53.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3000472215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48105640 0x7fbe48105ac0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009990 tx=0x7fbe34009ca0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.480 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3000472215 shutdown_connections 2024-04-26T11:29:53.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3000472215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe48106000 0x7fbe48112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3000472215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4810b640 0x7fbe4810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3000472215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48105640 0x7fbe48105ac0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3000472215 >> 172.21.15.33:0/3000472215 conn(0x7fbe48101130 msgr2=0x7fbe48103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:53.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3000472215 shutdown_connections 2024-04-26T11:29:53.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.479+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3000472215 wait complete. 2024-04-26T11:29:53.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.480+0000 7fbe4dc00700 1 Processor -- start 2024-04-26T11:29:53.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.480+0000 7fbe4dc00700 1 -- start start 2024-04-26T11:29:53.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe4dc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe48105640 0x7fbe4819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:53.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe4dc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48106000 0x7fbe4819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:53.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe4dc00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4810b640 0x7fbe481a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe4dc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe48115890 con 0x7fbe48106000 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe4dc00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe48115710 con 0x7fbe4810b640 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe4dc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbe48115a10 con 0x7fbe48105640 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe47400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4810b640 0x7fbe481a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe48105640 0x7fbe4819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe46000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48106000 0x7fbe4819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe48105640 0x7fbe4819bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:36006/0 (socket says 172.21.15.33:36006) 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe47400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4810b640 0x7fbe481a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:38646/0 (socket says 172.21.15.33:38646) 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.481+0000 7fbe46a00700 1 -- 172.21.15.33:0/3045647590 learned_addr learned my addr 172.21.15.33:0/3045647590 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe46a00700 1 -- 172.21.15.33:0/3045647590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4810b640 msgr2=0x7fbe481a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe46a00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4810b640 0x7fbe481a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe46a00700 1 -- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48106000 msgr2=0x7fbe4819c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe46a00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48106000 0x7fbe4819c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe46a00700 1 -- 172.21.15.33:0/3045647590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe481a0db0 con 0x7fbe48105640 2024-04-26T11:29:53.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe46a00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe48105640 0x7fbe4819bda0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800e9a0 tx=0x7fbe3800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:53.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe3f400700 1 -- 172.21.15.33:0/3045647590 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800ca10 con 0x7fbe48105640 2024-04-26T11:29:53.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe3f400700 1 -- 172.21.15.33:0/3045647590 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3800cbb0 con 0x7fbe48105640 2024-04-26T11:29:53.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe481a10a0 con 0x7fbe48105640 2024-04-26T11:29:53.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe481a8920 con 0x7fbe48105640 2024-04-26T11:29:53.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.482+0000 7fbe3f400700 1 -- 172.21.15.33:0/3045647590 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe38018680 con 0x7fbe48105640 2024-04-26T11:29:53.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.483+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe28005330 con 0x7fbe48105640 2024-04-26T11:29:53.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.485+0000 7fbe3f400700 1 -- 172.21.15.33:0/3045647590 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbe38018820 con 0x7fbe48105640 2024-04-26T11:29:53.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.487+0000 7fbe3f400700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe2c0a3cc0 0x7fbe2c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:53.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.487+0000 7fbe3f400700 1 -- 172.21.15.33:0/3045647590 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbe380d03e0 con 0x7fbe48105640 2024-04-26T11:29:53.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.487+0000 7fbe46000700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe2c0a3cc0 0x7fbe2c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:53.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.488+0000 7fbe46000700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe2c0a3cc0 0x7fbe2c0a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe48103790 tx=0x7fbe3400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:53.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.491+0000 7fbe3f400700 1 -- 172.21.15.33:0/3045647590 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe3809cd80 con 0x7fbe48105640 2024-04-26T11:29:53.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.773+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe28005620 con 0x7fbe48105640 2024-04-26T11:29:53.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:53 smithi033 ceph-mon[27845]: pgmap v427: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:53.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.774+0000 7fbe3f400700 1 -- 172.21.15.33:0/3045647590 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fbe3809c680 con 0x7fbe48105640 2024-04-26T11:29:53.776 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:53.776 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:26:56.732811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:29:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe2c0a3cc0 msgr2=0x7fbe2c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe2c0a3cc0 0x7fbe2c0a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe48103790 tx=0x7fbe3400b260 comp rx=0 tx=0).stop 2024-04-26T11:29:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe48105640 msgr2=0x7fbe4819bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:53.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe48105640 0x7fbe4819bda0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800e9a0 tx=0x7fbe3800ecb0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 shutdown_connections 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe2c0a3cc0 0x7fbe2c0a6180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4810b640 0x7fbe481a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe48105640 0x7fbe4819bda0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 --2- 172.21.15.33:0/3045647590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe48106000 0x7fbe4819c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.777+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 >> 172.21.15.33:0/3045647590 conn(0x7fbe48101130 msgr2=0x7fbe48101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.778+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 shutdown_connections 2024-04-26T11:29:53.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:53.778+0000 7fbe4dc00700 1 -- 172.21.15.33:0/3045647590 wait complete. 2024-04-26T11:29:53.781 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-04-26T11:29:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:53 smithi194 ceph-mon[33017]: pgmap v427: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:53 smithi096 ceph-mon[32753]: pgmap v427: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:54.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:29:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:54 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3045647590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:54 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3045647590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:54 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3045647590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:55.082 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:55.556 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:55 smithi033 ceph-mon[27845]: pgmap v428: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:29:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:55 smithi194 ceph-mon[33017]: pgmap v428: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:29:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:55 smithi096 ceph-mon[32753]: pgmap v428: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:29:57.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.143+0000 7f4698c00700 1 -- 172.21.15.33:0/477344131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 msgr2=0x7f4694109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:57.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.143+0000 7f4698c00700 1 --2- 172.21.15.33:0/477344131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 0x7f4694109880 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f46840098f0 tx=0x7f4684009c00 comp rx=0 tx=0).stop 2024-04-26T11:29:57.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.144+0000 7f4698c00700 1 -- 172.21.15.33:0/477344131 shutdown_connections 2024-04-26T11:29:57.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.144+0000 7f4698c00700 1 --2- 172.21.15.33:0/477344131 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f469406bad0 0x7f46941102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.144+0000 7f4698c00700 1 --2- 172.21.15.33:0/477344131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 0x7f4694109880 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.144+0000 7f4698c00700 1 --2- 172.21.15.33:0/477344131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406b130 0x7f469406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.144+0000 7f4698c00700 1 -- 172.21.15.33:0/477344131 >> 172.21.15.33:0/477344131 conn(0x7f46940792d0 msgr2=0x7f46940796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:57.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.144+0000 7f4698c00700 1 -- 172.21.15.33:0/477344131 shutdown_connections 2024-04-26T11:29:57.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.144+0000 7f4698c00700 1 -- 172.21.15.33:0/477344131 wait complete. 2024-04-26T11:29:57.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.145+0000 7f4698c00700 1 Processor -- start 2024-04-26T11:29:57.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.145+0000 7f4698c00700 1 -- start start 2024-04-26T11:29:57.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f4698c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f469406b130 0x7f469419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:57.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f4698c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406bad0 0x7f469419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:57.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f4698c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 0x7f4694199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:57.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f4698c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46941135e0 con 0x7f469406b130 2024-04-26T11:29:57.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f4698c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4694113460 con 0x7f46941094a0 2024-04-26T11:29:57.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f4698c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4694113760 con 0x7f469406bad0 2024-04-26T11:29:57.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f4691e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f469406b130 0x7f469419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:57.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f468be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406bad0 0x7f469419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:57.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f468be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406bad0 0x7f469419fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:36018/0 (socket says 172.21.15.33:36018) 2024-04-26T11:29:57.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.146+0000 7f468be00700 1 -- 172.21.15.33:0/2317766135 learned_addr learned my addr 172.21.15.33:0/2317766135 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:29:57.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4692800700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 0x7f4694199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4691e00700 1 -- 172.21.15.33:0/2317766135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406bad0 msgr2=0x7f469419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4691e00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406bad0 0x7f469419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4691e00700 1 -- 172.21.15.33:0/2317766135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 msgr2=0x7f4694199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4691e00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 0x7f4694199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4691e00700 1 -- 172.21.15.33:0/2317766135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469419a0e0 con 0x7f469406b130 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4692800700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 0x7f4694199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f468be00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406bad0 0x7f469419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:29:57.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.147+0000 7f4691e00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f469406b130 0x7f469419f7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f468400c010 tx=0x7f468400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:57.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.148+0000 7f468aa00700 1 -- 172.21.15.33:0/2317766135 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840050c0 con 0x7f469406b130 2024-04-26T11:29:57.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.148+0000 7f468aa00700 1 -- 172.21.15.33:0/2317766135 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f468400b560 con 0x7f469406b130 2024-04-26T11:29:57.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.148+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469419a370 con 0x7f469406b130 2024-04-26T11:29:57.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.148+0000 7f468aa00700 1 -- 172.21.15.33:0/2317766135 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684003a00 con 0x7f469406b130 2024-04-26T11:29:57.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.148+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46941a6690 con 0x7f469406b130 2024-04-26T11:29:57.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.149+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4694104b50 con 0x7f469406b130 2024-04-26T11:29:57.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.152+0000 7f468aa00700 1 -- 172.21.15.33:0/2317766135 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f468400b710 con 0x7f469406b130 2024-04-26T11:29:57.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.153+0000 7f468aa00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46780a3c40 0x7f46780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:29:57.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.153+0000 7f468aa00700 1 -- 172.21.15.33:0/2317766135 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f46840d1480 con 0x7f469406b130 2024-04-26T11:29:57.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.153+0000 7f468be00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46780a3c40 0x7f46780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:29:57.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.154+0000 7f468be00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46780a3c40 0x7f46780a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4694079980 tx=0x7f467c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:29:57.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.156+0000 7f468aa00700 1 -- 172.21.15.33:0/2317766135 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4684013030 con 0x7f469406b130 2024-04-26T11:29:57.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.435+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f469419aeb0 con 0x7f469406b130 2024-04-26T11:29:57.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.437+0000 7f468aa00700 1 -- 172.21.15.33:0/2317766135 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7f4684021380 con 0x7f469406b130 2024-04-26T11:29:57.438 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:29:57.438 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:26:56.732811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:29:57.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46780a3c40 msgr2=0x7f46780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:57.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46780a3c40 0x7f46780a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4694079980 tx=0x7f467c009260 comp rx=0 tx=0).stop 2024-04-26T11:29:57.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f469406b130 msgr2=0x7f469419f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:29:57.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f469406b130 0x7f469419f7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f468400c010 tx=0x7f468400fa00 comp rx=0 tx=0).stop 2024-04-26T11:29:57.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 shutdown_connections 2024-04-26T11:29:57.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f46780a3c40 0x7f46780a6100 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4694079980 tx=0x7f467c009260 comp rx=0 tx=0).stop 2024-04-26T11:29:57.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46941094a0 0x7f4694199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f469406b130 0x7f469419f7c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 --2- 172.21.15.33:0/2317766135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f469406bad0 0x7f469419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:29:57.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 >> 172.21.15.33:0/2317766135 conn(0x7f46940792d0 msgr2=0x7f4694100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:29:57.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.440+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 shutdown_connections 2024-04-26T11:29:57.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:29:57.441+0000 7f4698c00700 1 -- 172.21.15.33:0/2317766135 wait complete. 2024-04-26T11:29:57.443 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-04-26T11:29:57.846 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:57 smithi033 ceph-mon[27845]: pgmap v429: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:57.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:57 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2317766135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:57 smithi194 ceph-mon[33017]: pgmap v429: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:57 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2317766135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:57 smithi096 ceph-mon[32753]: pgmap v429: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:29:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:57 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2317766135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:29:58.279 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:29:58.756 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:29:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:29:59 smithi194 ceph-mon[33017]: pgmap v430: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:29:59 smithi033 ceph-mon[27845]: pgmap v430: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:29:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:29:59 smithi096 ceph-mon[32753]: pgmap v430: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:30:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:00 smithi033 ceph-mon[27845]: overall HEALTH_OK 2024-04-26T11:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:00 smithi194 ceph-mon[33017]: overall HEALTH_OK 2024-04-26T11:30:00.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.906+0000 7f399e800700 1 -- 172.21.15.33:0/2395543434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 msgr2=0x7f399810f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:00.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.906+0000 7f399e800700 1 --2- 172.21.15.33:0/2395543434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399810f1d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f399000b0a0 tx=0x7f399000b3b0 comp rx=0 tx=0).stop 2024-04-26T11:30:00.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.906+0000 7f399e800700 1 -- 172.21.15.33:0/2395543434 shutdown_connections 2024-04-26T11:30:00.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.906+0000 7f399e800700 1 --2- 172.21.15.33:0/2395543434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399810f1d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:00.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.906+0000 7f399e800700 1 --2- 172.21.15.33:0/2395543434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3998108390 0x7f3998108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:00.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.906+0000 7f399e800700 1 --2- 172.21.15.33:0/2395543434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998102320 0x7f39981027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:00.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.906+0000 7f399e800700 1 -- 172.21.15.33:0/2395543434 >> 172.21.15.33:0/2395543434 conn(0x7f39980792d0 msgr2=0x7f39980796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:00.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.907+0000 7f399e800700 1 -- 172.21.15.33:0/2395543434 shutdown_connections 2024-04-26T11:30:00.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.907+0000 7f399e800700 1 -- 172.21.15.33:0/2395543434 wait complete. 2024-04-26T11:30:00.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.908+0000 7f399e800700 1 Processor -- start 2024-04-26T11:30:00.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.908+0000 7f399e800700 1 -- start start 2024-04-26T11:30:00.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f399e800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998102320 0x7f399819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f399e800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f399e800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3998108390 0x7f39981a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f399e800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39981124b0 con 0x7f3998102ce0 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f399e800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3998112330 con 0x7f3998108390 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f399e800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3998112630 con 0x7f3998102320 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f3996a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f3996a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399819adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50858/0 (socket says 172.21.15.33:50858) 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f3996a00700 1 -- 172.21.15.33:0/4226479257 learned_addr learned my addr 172.21.15.33:0/4226479257 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f3997400700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998102320 0x7f399819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.909+0000 7f3997e00700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3998108390 0x7f39981a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3997400700 1 -- 172.21.15.33:0/4226479257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3998108390 msgr2=0x7f39981a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3997400700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3998108390 0x7f39981a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3997400700 1 -- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 msgr2=0x7f399819adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:00.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3997400700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399819adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3997400700 1 -- 172.21.15.33:0/4226479257 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39981a5e70 con 0x7f3998102320 2024-04-26T11:30:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3997e00700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3998108390 0x7f39981a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:30:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3996a00700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:00.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f3997400700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998102320 0x7f399819a8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f398800e9f0 tx=0x7f398800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f398fe00700 1 -- 172.21.15.33:0/4226479257 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398800cbd0 con 0x7f3998102320 2024-04-26T11:30:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f398fe00700 1 -- 172.21.15.33:0/4226479257 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f398800cd70 con 0x7f3998102320 2024-04-26T11:30:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39981a60a0 con 0x7f3998102320 2024-04-26T11:30:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.910+0000 7f398fe00700 1 -- 172.21.15.33:0/4226479257 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988018680 con 0x7f3998102320 2024-04-26T11:30:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.911+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39981a6600 con 0x7f3998102320 2024-04-26T11:30:00.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.912+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399806a110 con 0x7f3998102320 2024-04-26T11:30:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.914+0000 7f398fe00700 1 -- 172.21.15.33:0/4226479257 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3988010430 con 0x7f3998102320 2024-04-26T11:30:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.915+0000 7f398fe00700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f397c0a3cc0 0x7f397c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.915+0000 7f398fe00700 1 -- 172.21.15.33:0/4226479257 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f39880d0520 con 0x7f3998102320 2024-04-26T11:30:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.915+0000 7f3996a00700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f397c0a3cc0 0x7f397c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:00.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.916+0000 7f3996a00700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f397c0a3cc0 0x7f397c0a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3998079840 tx=0x7f3984009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:00.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:00.919+0000 7f398fe00700 1 -- 172.21.15.33:0/4226479257 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f398809cec0 con 0x7f3998102320 2024-04-26T11:30:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:00 smithi096 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T11:30:01.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.156+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f399802e490 con 0x7f3998102320 2024-04-26T11:30:01.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.159+0000 7f398fe00700 1 -- 172.21.15.33:0/4226479257 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22592 (secure 0 0 0) 0x7f398809c7c0 con 0x7f3998102320 2024-04-26T11:30:01.160 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:01.160 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":90,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:27:19.230568+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:30:01.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f397c0a3cc0 msgr2=0x7f397c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:01.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f397c0a3cc0 0x7f397c0a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3998079840 tx=0x7f3984009260 comp rx=0 tx=0).stop 2024-04-26T11:30:01.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998102320 msgr2=0x7f399819a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:01.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998102320 0x7f399819a8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f398800e9f0 tx=0x7f398800ed00 comp rx=0 tx=0).stop 2024-04-26T11:30:01.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 shutdown_connections 2024-04-26T11:30:01.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f397c0a3cc0 0x7f397c0a6180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:01.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3998108390 0x7f39981a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:01.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998102320 0x7f399819a8b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:01.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 --2- 172.21.15.33:0/4226479257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3998102ce0 0x7f399819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:01.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 >> 172.21.15.33:0/4226479257 conn(0x7f39980792d0 msgr2=0x7f3998100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:01.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.162+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 shutdown_connections 2024-04-26T11:30:01.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:01.163+0000 7f399e800700 1 -- 172.21.15.33:0/4226479257 wait complete. 2024-04-26T11:30:01.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:01 smithi194 ceph-mon[33017]: pgmap v431: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:30:01.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:01 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/4226479257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:30:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:01 smithi033 ceph-mon[27845]: pgmap v431: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:30:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:01 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/4226479257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:30:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:01 smithi096 ceph-mon[32753]: pgmap v431: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:30:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:01 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/4226479257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:30:02.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T11:26:32.073203+0000', 'modified': '2024-04-26T11:26:56.732811+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24527, 'mds_1': 24488, 'mds_2': 24484}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6835/1403509498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 1403509498}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 1403509498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6835/1002586260', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6834', 'nonce': 1002586260}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1002586260}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6837/1265312544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 1265312544}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 1265312544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.194:6839/517247009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 517247009}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 517247009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.96:6837/823522377', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 823522377}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 823522377}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.96:6839/1966519759', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 1966519759}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 1966519759}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24488, 24527, 24484]} 2024-04-26T11:30:02.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T11:30:02.041 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T11:30:02.530 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:03 smithi033 ceph-mon[27845]: pgmap v432: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:03 smithi194 ceph-mon[33017]: pgmap v432: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:03 smithi096 ceph-mon[32753]: pgmap v432: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.719+0000 7ff2b5800700 1 -- 172.21.15.33:0/1997429261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b01094a0 msgr2=0x7ff2b0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.719+0000 7ff2b5800700 1 --2- 172.21.15.33:0/1997429261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b01094a0 0x7ff2b0109880 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009990 tx=0x7ff2a0009ca0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.719+0000 7ff2b5800700 1 -- 172.21.15.33:0/1997429261 shutdown_connections 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.719+0000 7ff2b5800700 1 --2- 172.21.15.33:0/1997429261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2b006bad0 0x7ff2b01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.719+0000 7ff2b5800700 1 --2- 172.21.15.33:0/1997429261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b01094a0 0x7ff2b0109880 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.719+0000 7ff2b5800700 1 --2- 172.21.15.33:0/1997429261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2b006b130 0x7ff2b006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.719+0000 7ff2b5800700 1 -- 172.21.15.33:0/1997429261 >> 172.21.15.33:0/1997429261 conn(0x7ff2b00792d0 msgr2=0x7ff2b00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.720+0000 7ff2b5800700 1 -- 172.21.15.33:0/1997429261 shutdown_connections 2024-04-26T11:30:04.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.720+0000 7ff2b5800700 1 -- 172.21.15.33:0/1997429261 wait complete. 2024-04-26T11:30:04.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 Processor -- start 2024-04-26T11:30:04.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 -- start start 2024-04-26T11:30:04.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2b006b130 0x7ff2b019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b006bad0 0x7ff2b019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2b01094a0 0x7ff2b0199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b0113590 con 0x7ff2b01094a0 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b0113410 con 0x7ff2b006bad0 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.721+0000 7ff2b5800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b0113710 con 0x7ff2b006b130 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2aea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2b006b130 0x7ff2b019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2a6000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b006bad0 0x7ff2b019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2aea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2b006b130 0x7ff2b019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:51388/0 (socket says 172.21.15.33:51388) 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2af400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2b01094a0 0x7ff2b0199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2aea00700 1 -- 172.21.15.33:0/3358850824 learned_addr learned my addr 172.21.15.33:0/3358850824 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:04.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2a6000700 1 -- 172.21.15.33:0/3358850824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2b006b130 msgr2=0x7ff2b019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2b006b130 0x7ff2b019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2a6000700 1 -- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2b01094a0 msgr2=0x7ff2b0199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2b01094a0 0x7ff2b0199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2a6000700 1 -- 172.21.15.33:0/3358850824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b019a0b0 con 0x7ff2b006bad0 2024-04-26T11:30:04.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.722+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b006bad0 0x7ff2b019fcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff29800e9f0 tx=0x7ff29800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:04.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.723+0000 7ff2a7e00700 1 -- 172.21.15.33:0/3358850824 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29800cb70 con 0x7ff2b006bad0 2024-04-26T11:30:04.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.723+0000 7ff2a7e00700 1 -- 172.21.15.33:0/3358850824 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff29800ee80 con 0x7ff2b006bad0 2024-04-26T11:30:04.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.723+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b019a3a0 con 0x7ff2b006bad0 2024-04-26T11:30:04.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.723+0000 7ff2a7e00700 1 -- 172.21.15.33:0/3358850824 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298003d40 con 0x7ff2b006bad0 2024-04-26T11:30:04.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.723+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b01a6660 con 0x7ff2b006bad0 2024-04-26T11:30:04.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.726+0000 7ff2a7e00700 1 -- 172.21.15.33:0/3358850824 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff298003ee0 con 0x7ff2b006bad0 2024-04-26T11:30:04.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.727+0000 7ff2a7e00700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff2900a3cc0 0x7ff2900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 Processor -- start 2024-04-26T11:30:04.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2a7e00700 1 -- 172.21.15.33:0/3358850824 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff2980d0460 con 0x7ff2b006bad0 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff2900a3cc0 0x7ff2900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 -- start start 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075de7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075de1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075de2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f075d30840 con 0x55f075de8250 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f075ccf160 con 0x55f075de7930 2024-04-26T11:30:04.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.728+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f075d2c800 con 0x55f075de24e0 2024-04-26T11:30:04.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff2900a3cc0 0x7ff2900a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009750 tx=0x7ff2a001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:04.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2a6000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075de1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2a6000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075de1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50880/0 (socket says 172.21.15.33:50880) 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2a6000700 1 -- 172.21.15.33:0/2147116446 learned_addr learned my addr 172.21.15.33:0/2147116446 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2aea00700 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075de7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2af400700 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075de2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2aea00700 1 -- 172.21.15.33:0/2147116446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 msgr2=0x55f075de2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2aea00700 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075de2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2aea00700 1 -- 172.21.15.33:0/2147116446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 msgr2=0x55f075de1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2aea00700 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075de1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2aea00700 1 -- 172.21.15.33:0/2147116446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f075cd75e0 con 0x55f075de7930 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.729+0000 7ff2a6000700 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075de1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.730+0000 7ff2aea00700 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075de7d10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000b2e0 tx=0x7ff2a001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d5830713c4b995 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.730+0000 7ff29fe00700 1 -- 172.21.15.33:0/2147116446 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a000bb00 con 0x55f075de7930 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.730+0000 7ff29fe00700 1 -- 172.21.15.33:0/2147116446 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2a000bca0 con 0x55f075de7930 2024-04-26T11:30:04.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.730+0000 7ff29fe00700 1 -- 172.21.15.33:0/2147116446 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a000f880 con 0x55f075de7930 2024-04-26T11:30:04.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2147116446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 msgr2=0x55f075de7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075de7d10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000b2e0 tx=0x7ff2a001f000 comp rx=0 tx=0).stop 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2147116446 shutdown_connections 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075de2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075de7d10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2147116446 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075de1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2147116446 >> 172.21.15.33:0/2147116446 conn(0x55f075d0b2a0 msgr2=0x55f075dd8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2147116446 shutdown_connections 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2147116446 wait complete. 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.731+0000 7ff2c00b2b80 1 -- start start 2024-04-26T11:30:04.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 1 Processor -- start 2024-04-26T11:30:04.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075dee060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075deb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075deb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f075deec30 con 0x55f075de8250 2024-04-26T11:30:04.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f075ccb3b0 con 0x55f075de7930 2024-04-26T11:30:04.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f075deedb0 con 0x55f075de24e0 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.732+0000 7ff2c00b2b80 10 client.0 _ll_register_callbacks cb 0x55f075cc4330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075deb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2af400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075deb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075deb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51560/0 (socket says 172.21.15.33:51560) 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 -- 172.21.15.33:0/3657741676 learned_addr learned my addr 172.21.15.33:0/3657741676 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075dee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 -- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 msgr2=0x55f075dee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075dee060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 -- 172.21.15.33:0/3657741676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 msgr2=0x55f075deb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075deb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 -- 172.21.15.33:0/3657741676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f075c7d880 con 0x55f075de7930 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075deb360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff298000c00 tx=0x7ff29809c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6ef3d826da4ebc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.733+0000 7ff29cc00700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:30:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff29cc00700 1 -- 172.21.15.33:0/3657741676 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298009400 con 0x55f075de7930 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2a000bca0 con 0x55f075de7930 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff29cc00700 1 -- 172.21.15.33:0/3657741676 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff29809c960 con 0x55f075de7930 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff29cc00700 1 -- 172.21.15.33:0/3657741676 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff29809cb60 con 0x55f075de7930 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 10 client.24983 fetch_fsmap learned FSMap version 25 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f075df96b0 con 0x55f075de7930 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff29cc00700 1 -- 172.21.15.33:0/3657741676 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2a000bca0 con 0x55f075de7930 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff29cc00700 1 -- 172.21.15.33:0/3657741676 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7ff2980042f0 con 0x55f075de7930 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 10 client.24983 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 10 client.24983 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 20 client.24983 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075dfbc00 0x55f075dfd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 4 client.24983 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:30:04.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.734+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 --> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f075d3e220 con 0x55f075dfbc00 2024-04-26T11:30:04.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.735+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075dfbc00 0x55f075dfd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.735+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075dfbc00 0x55f075dfd070 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad2e9c5a697e407e server_cookie=724bce65d48998f8 in_seq=0 out_seq=0 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.735+0000 7ff29cc00700 10 client.24983 ms_handle_connect on v2:172.21.15.194:6838/517247009 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.736+0000 7ff29cc00700 1 -- 172.21.15.33:0/3657741676 <== mds.2 v2:172.21.15.194:6838/517247009 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f075d3e220 con 0x55f075dfbc00 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.736+0000 7ff29cc00700 10 client.24983 handle_command_reply: tid=0 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.736+0000 7ff2c00b2b80 1 client.24983 shutdown 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.736+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 msgr2=0x55f075deb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075deb360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff298000c00 tx=0x7ff29809c7f0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 shutdown_connections 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075dfbc00 0x55f075dfd070 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de8250 0x55f075deb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de24e0 0x55f075dee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3657741676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de7930 0x55f075deb360 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 >> 172.21.15.33:0/3657741676 conn(0x55f075d0b2a0 msgr2=0x55f075de6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:04.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff29cc00700 0 client.24983 ms_handle_reset on v2:172.21.15.194:6838/517247009 2024-04-26T11:30:04.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 shutdown_connections 2024-04-26T11:30:04.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3657741676 wait complete. 2024-04-26T11:30:04.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.737+0000 7ff2c00b2b80 20 client.24983 trim_cache size 0 max 16384 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 Processor -- start 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 -- start start 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de24e0 0x55f075dedba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de7930 0x55f075de4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de8250 0x55f075de4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f075dfb3b0 con 0x55f075de7930 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f075dfba80 con 0x55f075de24e0 2024-04-26T11:30:04.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.760+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f075dfc150 con 0x55f075de8250 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de7930 0x55f075de4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de7930 0x55f075de4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50908/0 (socket says 172.21.15.33:50908) 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2aea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de24e0 0x55f075dedba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 -- 172.21.15.33:0/2815229906 learned_addr learned my addr 172.21.15.33:0/2815229906 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2af400700 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de8250 0x55f075de4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 -- 172.21.15.33:0/2815229906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de8250 msgr2=0x55f075de4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de8250 0x55f075de4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 -- 172.21.15.33:0/2815229906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de24e0 msgr2=0x55f075dedba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de24e0 0x55f075dedba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 -- 172.21.15.33:0/2815229906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f075dd9420 con 0x55f075de7930 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2aea00700 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de24e0 0x55f075dedba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.761+0000 7ff2a6000700 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de7930 0x55f075de4400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c5b0 tx=0x7ff298003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42af4258f5d8fee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.762+0000 7ff29f400700 1 -- 172.21.15.33:0/2815229906 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2980986f0 con 0x55f075de7930 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.762+0000 7ff29f400700 1 -- 172.21.15.33:0/2815229906 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff298082070 con 0x55f075de7930 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.762+0000 7ff29f400700 1 -- 172.21.15.33:0/2815229906 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298004430 con 0x55f075de7930 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.762+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2815229906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de7930 msgr2=0x55f075de4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.762+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de7930 0x55f075de4400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c5b0 tx=0x7ff298003680 comp rx=0 tx=0).stop 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2815229906 shutdown_connections 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de8250 0x55f075de4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de24e0 0x55f075dedba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/2815229906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de7930 0x55f075de4400 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2815229906 >> 172.21.15.33:0/2815229906 conn(0x55f075d0b2a0 msgr2=0x55f075dda070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2815229906 shutdown_connections 2024-04-26T11:30:04.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/2815229906 wait complete. 2024-04-26T11:30:04.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 -- start start 2024-04-26T11:30:04.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.763+0000 7ff2c00b2b80 1 Processor -- start 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de24e0 0x55f075de4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de7930 0x55f075de4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2c00b2b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de8250 0x55f075e04ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f075dfcef0 con 0x55f075de24e0 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f075dfc820 con 0x55f075de8250 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2c00b2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f075dfd5c0 con 0x55f075de7930 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2c00b2b80 10 client.0 _ll_register_callbacks cb 0x55f075cc4330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2aea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de24e0 0x55f075de4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2aea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de24e0 0x55f075de4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50924/0 (socket says 172.21.15.33:50924) 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2aea00700 1 -- 172.21.15.33:0/3527437697 learned_addr learned my addr 172.21.15.33:0/3527437697 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:04.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de7930 0x55f075de4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.764+0000 7ff2af400700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de8250 0x55f075e04ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2aea00700 1 -- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de7930 msgr2=0x55f075de4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de7930 0x55f075de4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2aea00700 1 -- 172.21.15.33:0/3527437697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de8250 msgr2=0x55f075e04ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de8250 0x55f075e04ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2aea00700 1 -- 172.21.15.33:0/3527437697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f075de21a0 con 0x55f075de24e0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2af400700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de8250 0x55f075e04ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2aea00700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de24e0 0x55f075de4040 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005d10 tx=0x7ff2a000f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44401db809bf0bcb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff29cc00700 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff29cc00700 1 -- 172.21.15.33:0/3527437697 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0039bc0 con 0x55f075de24e0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff29cc00700 1 -- 172.21.15.33:0/3527437697 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff2a0039d60 con 0x55f075de24e0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.765+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff298082070 con 0x55f075de24e0 2024-04-26T11:30:04.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff29cc00700 1 -- 172.21.15.33:0/3527437697 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff2a0039330 con 0x55f075de24e0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 10 client.14874 fetch_fsmap learned FSMap version 25 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f075e63370 con 0x55f075de24e0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff29cc00700 1 -- 172.21.15.33:0/3527437697 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff298082070 con 0x55f075de24e0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff29cc00700 1 -- 172.21.15.33:0/3527437697 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7ff2a0003be0 con 0x55f075de24e0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 10 client.14874 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 10 client.14874 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 20 client.14874 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075e679b0 0x55f075e666c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 4 client.14874 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.766+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 --> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] -- command(tid 0: {"prefix": "status"}) -- 0x55f075d382c0 con 0x55f075e679b0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.767+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075e679b0 0x55f075e666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.767+0000 7ff2a6000700 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075e679b0 0x55f075e666c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7f3d85674339662 server_cookie=e4008a152ed46514 in_seq=0 out_seq=0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.767+0000 7ff29cc00700 10 client.14874 ms_handle_connect on v2:172.21.15.194:6838/517247009 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff29cc00700 1 -- 172.21.15.33:0/3527437697 <== mds.2 v2:172.21.15.194:6838/517247009 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f075d382c0 con 0x55f075e679b0 2024-04-26T11:30:04.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff29cc00700 10 client.14874 handle_command_reply: tid=0 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 client.14874 shutdown 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de24e0 msgr2=0x55f075de4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de24e0 0x55f075de4040 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005d10 tx=0x7ff2a000f460 comp rx=0 tx=0).stop 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 shutdown_connections 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x55f075e679b0 0x55f075e666c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f075de8250 0x55f075e04ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f075de24e0 0x55f075de4040 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 --2- 172.21.15.33:0/3527437697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f075de7930 0x55f075de4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 >> 172.21.15.33:0/3527437697 conn(0x55f075d0b2a0 msgr2=0x55f075de8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff29cc00700 0 client.14874 ms_handle_reset on v2:172.21.15.194:6838/517247009 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 shutdown_connections 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 1 -- 172.21.15.33:0/3527437697 wait complete. 2024-04-26T11:30:04.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.768+0000 7ff2c00b2b80 20 client.14874 trim_cache size 0 max 16384 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24484, 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 194.65938707399999, 2024-04-26T11:30:04.773 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-04-26T11:30:04.774 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-04-26T11:30:04.774 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T11:30:04.774 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 246.849437647 2024-04-26T11:30:04.774 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:30:04.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff2900a3cc0 msgr2=0x7ff2900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff2900a3cc0 0x7ff2900a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009750 tx=0x7ff2a001a040 comp rx=0 tx=0).stop 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b006bad0 msgr2=0x7ff2b019fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b006bad0 0x7ff2b019fcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff29800e9f0 tx=0x7ff29800ed00 comp rx=0 tx=0).stop 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 shutdown_connections 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7ff2900a3cc0 0x7ff2900a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2b01094a0 0x7ff2b0199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2b006b130 0x7ff2b019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 --2- 172.21.15.33:0/3358850824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2b006bad0 0x7ff2b019fcd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.769+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 >> 172.21.15.33:0/3358850824 conn(0x7ff2b00792d0 msgr2=0x7ff2b01004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.770+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 shutdown_connections 2024-04-26T11:30:04.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:04.770+0000 7ff2b5800700 1 -- 172.21.15.33:0/3358850824 wait complete. 2024-04-26T11:30:05.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:05 smithi033 ceph-mon[27845]: pgmap v433: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:30:05.672 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-26T11:30:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:05 smithi194 ceph-mon[33017]: pgmap v433: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:30:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:05 smithi096 ceph-mon[32753]: pgmap v433: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T11:30:06.135 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:07 smithi033 ceph-mon[27845]: pgmap v434: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:07 smithi194 ceph-mon[33017]: pgmap v434: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:07 smithi096 ceph-mon[32753]: pgmap v434: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 -- 172.21.15.33:0/3599713874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 msgr2=0x7faf30112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 --2- 172.21.15.33:0/3599713874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf30112510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faf28009f90 tx=0x7faf2800b330 comp rx=0 tx=0).stop 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 -- 172.21.15.33:0/3599713874 shutdown_connections 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 --2- 172.21.15.33:0/3599713874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf30112510 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 --2- 172.21.15.33:0/3599713874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3010b640 0x7faf3010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 --2- 172.21.15.33:0/3599713874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf30105640 0x7faf30105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 -- 172.21.15.33:0/3599713874 >> 172.21.15.33:0/3599713874 conn(0x7faf30101130 msgr2=0x7faf30103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 -- 172.21.15.33:0/3599713874 shutdown_connections 2024-04-26T11:30:08.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.292+0000 7faf36000700 1 -- 172.21.15.33:0/3599713874 wait complete. 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 Processor -- start 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 -- start start 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf30105640 0x7faf301a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf301a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3010b640 0x7faf3019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf30115880 con 0x7faf30106000 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf30115700 con 0x7faf30105640 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.294+0000 7faf36000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf30115a00 con 0x7faf3010b640 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf301a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf301a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50940/0 (socket says 172.21.15.33:50940) 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 -- 172.21.15.33:0/2758365420 learned_addr learned my addr 172.21.15.33:0/2758365420 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2f400700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf30105640 0x7faf301a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 -- 172.21.15.33:0/2758365420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3010b640 msgr2=0x7faf3019ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3010b640 0x7faf3019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 -- 172.21.15.33:0/2758365420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf30105640 msgr2=0x7faf301a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf30105640 0x7faf301a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 -- 172.21.15.33:0/2758365420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3019c1e0 con 0x7faf30106000 2024-04-26T11:30:08.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.295+0000 7faf2ea00700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf301a1e00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00e9a0 tx=0x7faf1c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:08.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.296+0000 7faf27e00700 1 -- 172.21.15.33:0/2758365420 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c00cbd0 con 0x7faf30106000 2024-04-26T11:30:08.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.296+0000 7faf27e00700 1 -- 172.21.15.33:0/2758365420 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf1c00cd70 con 0x7faf30106000 2024-04-26T11:30:08.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.296+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3019c4d0 con 0x7faf30106000 2024-04-26T11:30:08.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.296+0000 7faf27e00700 1 -- 172.21.15.33:0/2758365420 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c018680 con 0x7faf30106000 2024-04-26T11:30:08.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.296+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf301a8790 con 0x7faf30106000 2024-04-26T11:30:08.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.300+0000 7faf27e00700 1 -- 172.21.15.33:0/2758365420 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faf1c010430 con 0x7faf30106000 2024-04-26T11:30:08.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 Processor -- start 2024-04-26T11:30:08.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 -- start start 2024-04-26T11:30:08.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784a3b0 0x563ec784a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784acd0 0x563ec7845b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec78464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563ec7798840 con 0x563ec784acd0 2024-04-26T11:30:08.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563ec7737160 con 0x563ec7846080 2024-04-26T11:30:08.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.301+0000 7faf4090cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563ec7794800 con 0x563ec784a3b0 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784a3b0 0x563ec784a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784acd0 0x563ec7845b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784a3b0 0x563ec784a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:51458/0 (socket says 172.21.15.33:51458) 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2fe00700 1 -- 172.21.15.33:0/3937774024 learned_addr learned my addr 172.21.15.33:0/3937774024 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf27e00700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf140a3cc0 0x7faf140a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2ea00700 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec78464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf27e00700 1 -- 172.21.15.33:0/2758365420 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faf1c0d05e0 con 0x7faf30106000 2024-04-26T11:30:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2fe00700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf140a3cc0 0x7faf140a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2f400700 1 -- 172.21.15.33:0/3937774024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784a3b0 msgr2=0x563ec784a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2f400700 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784a3b0 0x563ec784a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2f400700 1 -- 172.21.15.33:0/3937774024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 msgr2=0x563ec78464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2f400700 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec78464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2f400700 1 -- 172.21.15.33:0/3937774024 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ec773f5e0 con 0x563ec784acd0 2024-04-26T11:30:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.302+0000 7faf2ea00700 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec78464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:30:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.303+0000 7faf2f400700 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784acd0 0x563ec7845b40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf20005fd0 tx=0x7faf200058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb0f841fa354e00 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.303+0000 7faf24c00700 1 -- 172.21.15.33:0/3937774024 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20011c80 con 0x563ec784acd0 2024-04-26T11:30:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.303+0000 7faf24c00700 1 -- 172.21.15.33:0/3937774024 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf20011e20 con 0x563ec784acd0 2024-04-26T11:30:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.303+0000 7faf24c00700 1 -- 172.21.15.33:0/3937774024 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf200193e0 con 0x563ec784acd0 2024-04-26T11:30:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 -- 172.21.15.33:0/3937774024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784acd0 msgr2=0x563ec7845b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf2fe00700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf140a3cc0 0x7faf140a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faf28009550 tx=0x7faf2800b680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784acd0 0x563ec7845b40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf20005fd0 tx=0x7faf200058e0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 -- 172.21.15.33:0/3937774024 shutdown_connections 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec78464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784a3b0 0x563ec784a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3937774024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784acd0 0x563ec7845b40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 -- 172.21.15.33:0/3937774024 >> 172.21.15.33:0/3937774024 conn(0x563ec77732a0 msgr2=0x563ec7840610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 -- 172.21.15.33:0/3937774024 shutdown_connections 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.304+0000 7faf4090cb80 1 -- 172.21.15.33:0/3937774024 wait complete. 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 -- start start 2024-04-26T11:30:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 Processor -- start 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7851a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563ec7852810 con 0x563ec7846080 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563ec77333b0 con 0x563ec784a3b0 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563ec7852990 con 0x563ec784acd0 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.305+0000 7faf4090cb80 10 client.0 _ll_register_callbacks cb 0x563ec772c330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7851a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:51464/0 (socket says 172.21.15.33:51464) 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 -- 172.21.15.33:0/4013510653 learned_addr learned my addr 172.21.15.33:0/4013510653 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:08.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2ea00700 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 -- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 msgr2=0x563ec784ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 -- 172.21.15.33:0/4013510653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 msgr2=0x563ec7851a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7851a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2fe00700 1 -- 172.21.15.33:0/4013510653 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ec76e5880 con 0x563ec784acd0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.306+0000 7faf2ea00700 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf2fe00700 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f280 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faf28003e20 tx=0x7faf28009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=753842632c3ddbbf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf19600700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf19600700 1 -- 172.21.15.33:0/4013510653 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2802e020 con 0x563ec784acd0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf19600700 1 -- 172.21.15.33:0/4013510653 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faf28004240 con 0x563ec784acd0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf20011e20 con 0x563ec784acd0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf19600700 1 -- 172.21.15.33:0/4013510653 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf28003e50 con 0x563ec784acd0 2024-04-26T11:30:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf4090cb80 10 client.24988 fetch_fsmap learned FSMap version 25 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.307+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563ec785d060 con 0x563ec784acd0 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf19600700 1 -- 172.21.15.33:0/4013510653 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf20011e20 con 0x563ec784acd0 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf19600700 1 -- 172.21.15.33:0/4013510653 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7faf28011070 con 0x563ec784acd0 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf4090cb80 10 client.24988 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf4090cb80 10 client.24988 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf4090cb80 20 client.24988 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf4090cb80 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec785f700 0x563ec7860b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf4090cb80 4 client.24988 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:30:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.308+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 --> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ec77a6220 con 0x563ec785f700 2024-04-26T11:30:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.309+0000 7faf2f400700 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec785f700 0x563ec7860b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.309+0000 7faf2f400700 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec785f700 0x563ec7860b70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cdf4c851cb474174 server_cookie=e66bd581497078c7 in_seq=0 out_seq=0 2024-04-26T11:30:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.309+0000 7faf19600700 10 client.24988 ms_handle_connect on v2:172.21.15.96:6836/823522377 2024-04-26T11:30:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf19600700 1 -- 172.21.15.33:0/4013510653 <== mds.1 v2:172.21.15.96:6836/823522377 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ec77a6220 con 0x563ec785f700 2024-04-26T11:30:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf19600700 10 client.24988 handle_command_reply: tid=0 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 client.24988 shutdown 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 msgr2=0x563ec784f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f280 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faf28003e20 tx=0x7faf28009920 comp rx=0 tx=0).stop 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 shutdown_connections 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f280 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7851a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec785f700 0x563ec7860b70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 --2- 172.21.15.33:0/4013510653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf19600700 0 client.24988 ms_handle_reset on v2:172.21.15.96:6836/823522377 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.310+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 >> 172.21.15.33:0/4013510653 conn(0x563ec77732a0 msgr2=0x563ec783ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.311+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 shutdown_connections 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.311+0000 7faf4090cb80 1 -- 172.21.15.33:0/4013510653 wait complete. 2024-04-26T11:30:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.311+0000 7faf4090cb80 20 client.24988 trim_cache size 0 max 16384 2024-04-26T11:30:08.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.333+0000 7faf4090cb80 1 Processor -- start 2024-04-26T11:30:08.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf4090cb80 1 -- start start 2024-04-26T11:30:08.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7846460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf4090cb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563ec785eeb0 con 0x563ec7846080 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf4090cb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563ec785f580 con 0x563ec784a3b0 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf4090cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563ec785fc50 con 0x563ec784acd0 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf2f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7846460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf2f400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7846460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50956/0 (socket says 172.21.15.33:50956) 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf2f400700 1 -- 172.21.15.33:0/1249948348 learned_addr learned my addr 172.21.15.33:0/1249948348 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.334+0000 7faf2ea00700 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf2f400700 1 -- 172.21.15.33:0/1249948348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 msgr2=0x563ec784f240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf2f400700 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf2f400700 1 -- 172.21.15.33:0/1249948348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 msgr2=0x563ec784ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf2f400700 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf2f400700 1 -- 172.21.15.33:0/1249948348 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ec783cfd0 con 0x563ec7846080 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf2ea00700 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:30:08.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf2f400700 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7846460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faf20005950 tx=0x7faf20009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8dd3ee52730285 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf1be00700 1 -- 172.21.15.33:0/1249948348 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20031070 con 0x563ec7846080 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf1be00700 1 -- 172.21.15.33:0/1249948348 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf20005500 con 0x563ec7846080 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.335+0000 7faf1be00700 1 -- 172.21.15.33:0/1249948348 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20004970 con 0x563ec7846080 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 -- 172.21.15.33:0/1249948348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 msgr2=0x563ec7846460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7846460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faf20005950 tx=0x7faf20009290 comp rx=0 tx=0).stop 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 -- 172.21.15.33:0/1249948348 shutdown_connections 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec784f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec7846080 0x563ec7846460 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 --2- 172.21.15.33:0/1249948348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec784a3b0 0x563ec784ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 -- 172.21.15.33:0/1249948348 >> 172.21.15.33:0/1249948348 conn(0x563ec77732a0 msgr2=0x563ec78405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 -- 172.21.15.33:0/1249948348 shutdown_connections 2024-04-26T11:30:08.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.336+0000 7faf4090cb80 1 -- 172.21.15.33:0/1249948348 wait complete. 2024-04-26T11:30:08.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.337+0000 7faf4090cb80 1 -- start start 2024-04-26T11:30:08.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.337+0000 7faf4090cb80 1 Processor -- start 2024-04-26T11:30:08.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.337+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec7869fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.337+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784a3b0 0x563ec786a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.337+0000 7faf4090cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec7843b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.337+0000 7faf4090cb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563ec78609f0 con 0x563ec784a3b0 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf4090cb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563ec7860320 con 0x563ec7846080 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf4090cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563ec78610c0 con 0x563ec784acd0 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf4090cb80 10 client.0 _ll_register_callbacks cb 0x563ec772c330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784a3b0 0x563ec786a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2f400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec7869fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784a3b0 0x563ec786a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50960/0 (socket says 172.21.15.33:50960) 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 -- 172.21.15.33:0/3734683403 learned_addr learned my addr 172.21.15.33:0/3734683403 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2fe00700 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec7843b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 -- 172.21.15.33:0/3734683403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 msgr2=0x563ec7843b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec7843b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 -- 172.21.15.33:0/3734683403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 msgr2=0x563ec7869fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec7869fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.338+0000 7faf2ea00700 1 -- 172.21.15.33:0/3734683403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ec7844390 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf2ea00700 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784a3b0 0x563ec786a520 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faf1c098ce0 tx=0x7faf1c010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387b6a057aee3353 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf19600700 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf19600700 1 -- 172.21.15.33:0/3734683403 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c09ddb0 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf19600700 1 -- 172.21.15.33:0/3734683403 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7faf1c010720 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf20005500 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf19600700 1 -- 172.21.15.33:0/3734683403 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf1c09cf80 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf4090cb80 10 client.14892 fetch_fsmap learned FSMap version 25 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563ec785ba10 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.339+0000 7faf19600700 1 -- 172.21.15.33:0/3734683403 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf20005500 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf19600700 1 -- 172.21.15.33:0/3734683403 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7faf1c098040 con 0x563ec784a3b0 2024-04-26T11:30:08.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf4090cb80 10 client.14892 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:30:08.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf4090cb80 10 client.14892 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T11:30:08.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf4090cb80 20 client.14892 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:08.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec78cb230 0x563ec78c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:08.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf4090cb80 4 client.14892 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:30:08.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 --> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] -- command(tid 0: {"prefix": "status"}) -- 0x563ec77a02c0 con 0x563ec78cb230 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.340+0000 7faf2f400700 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec78cb230 0x563ec78c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf2f400700 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec78cb230 0x563ec78c9f40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=614e1863170f61ac server_cookie=861f62775477b084 in_seq=0 out_seq=0 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf19600700 10 client.14892 ms_handle_connect on v2:172.21.15.96:6836/823522377 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf19600700 1 -- 172.21.15.33:0/3734683403 <== mds.1 v2:172.21.15.96:6836/823522377 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x563ec77a02c0 con 0x563ec78cb230 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf19600700 10 client.14892 handle_command_reply: tid=0 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 client.14892 shutdown 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784a3b0 msgr2=0x563ec786a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784a3b0 0x563ec786a520 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faf1c098ce0 tx=0x7faf1c010cb0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 shutdown_connections 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ec784acd0 0x563ec7843b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x563ec78cb230 0x563ec78c9f40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ec7846080 0x563ec7869fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 --2- 172.21.15.33:0/3734683403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ec784a3b0 0x563ec786a520 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf19600700 0 client.14892 ms_handle_reset on v2:172.21.15.96:6836/823522377 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.341+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 >> 172.21.15.33:0/3734683403 conn(0x563ec77732a0 msgr2=0x563ec784ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.342+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 shutdown_connections 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.342+0000 7faf4090cb80 1 -- 172.21.15.33:0/3734683403 wait complete. 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.342+0000 7faf4090cb80 20 client.14892 trim_cache size 0 max 16384 2024-04-26T11:30:08.345 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24488, 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 199.255737729, 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 253.288371883 2024-04-26T11:30:08.346 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.342+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf140a3cc0 msgr2=0x7faf140a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.342+0000 7faf36000700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf140a3cc0 0x7faf140a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faf28009550 tx=0x7faf2800b680 comp rx=0 tx=0).stop 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.342+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 msgr2=0x7faf301a1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.342+0000 7faf36000700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf301a1e00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00e9a0 tx=0x7faf1c00ed60 comp rx=0 tx=0).stop 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 shutdown_connections 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7faf140a3cc0 0x7faf140a6180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3010b640 0x7faf3019ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf30105640 0x7faf301a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 --2- 172.21.15.33:0/2758365420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf30106000 0x7faf301a1e00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:08.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 >> 172.21.15.33:0/2758365420 conn(0x7faf30101130 msgr2=0x7faf301016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:08.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 shutdown_connections 2024-04-26T11:30:08.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:08.343+0000 7faf36000700 1 -- 172.21.15.33:0/2758365420 wait complete. 2024-04-26T11:30:09.266 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T11:30:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:09 smithi194 ceph-mon[33017]: pgmap v435: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:30:09.896 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:09 smithi033 ceph-mon[27845]: pgmap v435: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:30:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:09 smithi096 ceph-mon[32753]: pgmap v435: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T11:30:11.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:11 smithi033 ceph-mon[27845]: pgmap v436: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:30:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:11 smithi194 ceph-mon[33017]: pgmap v436: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:30:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:11 smithi096 ceph-mon[32753]: pgmap v436: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T11:30:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 -- 172.21.15.33:0/1563193202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb41145f0 msgr2=0x7fbeb41169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 --2- 172.21.15.33:0/1563193202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb41145f0 0x7fbeb41169e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009f90 tx=0x7fbeac00b330 comp rx=0 tx=0).stop 2024-04-26T11:30:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 -- 172.21.15.33:0/1563193202 shutdown_connections 2024-04-26T11:30:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 --2- 172.21.15.33:0/1563193202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb41145f0 0x7fbeb41169e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 --2- 172.21.15.33:0/1563193202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbeb4075650 0x7fbeb4075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 --2- 172.21.15.33:0/1563193202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb4075f70 0x7fbeb41140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 -- 172.21.15.33:0/1563193202 >> 172.21.15.33:0/1563193202 conn(0x7fbeb407a580 msgr2=0x7fbeb407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:11.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 -- 172.21.15.33:0/1563193202 shutdown_connections 2024-04-26T11:30:11.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.968+0000 7fbebba00700 1 -- 172.21.15.33:0/1563193202 wait complete. 2024-04-26T11:30:11.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.969+0000 7fbebba00700 1 Processor -- start 2024-04-26T11:30:11.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbebba00700 1 -- start start 2024-04-26T11:30:11.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbebba00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbeb4075650 0x7fbeb41a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbebba00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb4075f70 0x7fbeb41a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbebba00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb41145f0 0x7fbeb41a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbebba00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb4119c10 con 0x7fbeb4075f70 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbebba00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb4119a90 con 0x7fbeb4075650 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbebba00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb4119d90 con 0x7fbeb41145f0 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.970+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb4075f70 0x7fbeb41a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb4075f70 0x7fbeb41a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36992/0 (socket says 172.21.15.33:36992) 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 -- 172.21.15.33:0/1522428605 learned_addr learned my addr 172.21.15.33:0/1522428605 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb9000700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbeb4075650 0x7fbeb41a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb41145f0 0x7fbeb41a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 -- 172.21.15.33:0/1522428605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb41145f0 msgr2=0x7fbeb41a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:11.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb41145f0 0x7fbeb41a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 -- 172.21.15.33:0/1522428605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbeb4075650 msgr2=0x7fbeb41a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:11.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbeb4075650 0x7fbeb41a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 -- 172.21.15.33:0/1522428605 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb41a5170 con 0x7fbeb4075f70 2024-04-26T11:30:11.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb9000700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbeb4075650 0x7fbeb41a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:11.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.971+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb4075f70 0x7fbeb41a06a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbea000e9c0 tx=0x7fbea000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:11.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.972+0000 7fbeb1600700 1 -- 172.21.15.33:0/1522428605 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea000cbd0 con 0x7fbeb4075f70 2024-04-26T11:30:11.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.972+0000 7fbeb1600700 1 -- 172.21.15.33:0/1522428605 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbea000cd70 con 0x7fbeb4075f70 2024-04-26T11:30:11.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.972+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb41a5460 con 0x7fbeb4075f70 2024-04-26T11:30:11.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.972+0000 7fbeb1600700 1 -- 172.21.15.33:0/1522428605 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0005580 con 0x7fbeb4075f70 2024-04-26T11:30:11.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.972+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb41acce0 con 0x7fbeb4075f70 2024-04-26T11:30:11.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.975+0000 7fbeb1600700 1 -- 172.21.15.33:0/1522428605 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbea0004750 con 0x7fbeb4075f70 2024-04-26T11:30:11.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 Processor -- start 2024-04-26T11:30:11.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbeb1600700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe9c0a3cc0 0x7fbe9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 -- start start 2024-04-26T11:30:11.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbeb1600700 1 -- 172.21.15.33:0/1522428605 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbea00d0560 con 0x7fbeb4075f70 2024-04-26T11:30:11.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe9c0a3cc0 0x7fbe9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb023b0 0x5618beb02790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb02cd0 0x5618beafdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beafe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618bea50840 con 0x5618beafe080 2024-04-26T11:30:11.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5618be9ef160 con 0x5618beb023b0 2024-04-26T11:30:11.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.977+0000 7fbec6325b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5618bea4c800 con 0x5618beb02cd0 2024-04-26T11:30:11.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb02cd0 0x5618beafdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb9000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb023b0 0x5618beb02790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb02cd0 0x5618beafdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57284/0 (socket says 172.21.15.33:57284) 2024-04-26T11:30:11.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 -- 172.21.15.33:0/2493907727 learned_addr learned my addr 172.21.15.33:0/2493907727 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beafe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 -- 172.21.15.33:0/2493907727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb023b0 msgr2=0x5618beb02790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb023b0 0x5618beb02790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 -- 172.21.15.33:0/2493907727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 msgr2=0x5618beafe4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beafe4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe9c0a3cc0 0x7fbe9c0a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0060b0 tx=0x7fbeac009c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.978+0000 7fbeb3e00700 1 -- 172.21.15.33:0/2493907727 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618be9f75e0 con 0x5618beb02cd0 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.979+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb02cd0 0x5618beafdb40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbea00a4f40 tx=0x7fbea0009550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56148223e087694d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.979+0000 7fbea6000700 1 -- 172.21.15.33:0/2493907727 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0005720 con 0x5618beb02cd0 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.979+0000 7fbea6000700 1 -- 172.21.15.33:0/2493907727 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbea009cf00 con 0x5618beb02cd0 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.979+0000 7fbea6000700 1 -- 172.21.15.33:0/2493907727 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea009ca60 con 0x5618beb02cd0 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 -- 172.21.15.33:0/2493907727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb02cd0 msgr2=0x5618beafdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:11.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb02cd0 0x5618beafdb40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbea00a4f40 tx=0x7fbea0009550 comp rx=0 tx=0).stop 2024-04-26T11:30:11.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 -- 172.21.15.33:0/2493907727 shutdown_connections 2024-04-26T11:30:11.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beafe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb023b0 0x5618beb02790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 --2- 172.21.15.33:0/2493907727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb02cd0 0x5618beafdb40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 -- 172.21.15.33:0/2493907727 >> 172.21.15.33:0/2493907727 conn(0x5618bea2b2a0 msgr2=0x5618beaf8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:11.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 -- 172.21.15.33:0/2493907727 shutdown_connections 2024-04-26T11:30:11.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 -- 172.21.15.33:0/2493907727 wait complete. 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.980+0000 7fbec6325b80 1 -- start start 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 1 Processor -- start 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb09ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb06dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beb07310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618beb0a810 con 0x5618beafe080 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5618be9eb3b0 con 0x5618beb02cd0 2024-04-26T11:30:11.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5618beb0a990 con 0x5618beb023b0 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbec6325b80 10 client.0 _ll_register_callbacks cb 0x5618be9e4330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.981+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb06dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb9000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb09ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb06dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57294/0 (socket says 172.21.15.33:57294) 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 -- 172.21.15.33:0/3354577077 learned_addr learned my addr 172.21.15.33:0/3354577077 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beb07310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 -- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 msgr2=0x5618beb07310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beb07310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 -- 172.21.15.33:0/3354577077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 msgr2=0x5618beb09ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb09ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 -- 172.21.15.33:0/3354577077 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618be99d880 con 0x5618beb023b0 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb06dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbea000c5b0 tx=0x7fbea0005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e97223c6e36a515 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:11.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbe9aa00700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3354577077 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00b9030 con 0x5618beb023b0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.982+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3354577077 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbea00d0110 con 0x5618beb023b0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbea009cf00 con 0x5618beb023b0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3354577077 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbea00a5470 con 0x5618beb023b0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbec6325b80 10 client.25003 fetch_fsmap learned FSMap version 25 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5618beb0a810 con 0x5618beb023b0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3354577077 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbea009cf00 con 0x5618beb023b0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3354577077 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fbea0084050 con 0x5618beb023b0 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbec6325b80 10 client.25003 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbec6325b80 10 client.25003 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbec6325b80 20 client.25003 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.983+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb17090 0x5618beb18500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.984+0000 7fbec6325b80 4 client.25003 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.984+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618bea5e220 con 0x5618beb17090 2024-04-26T11:30:11.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.984+0000 7fbeb9000700 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb17090 0x5618beb18500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:11.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.984+0000 7fbeb9000700 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb17090 0x5618beb18500 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65739c3ffc619acf server_cookie=73ad9d5447b376d4 in_seq=0 out_seq=0 2024-04-26T11:30:11.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.984+0000 7fbe9aa00700 10 client.25003 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.985+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3354577077 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618bea5e220 con 0x5618beb17090 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.985+0000 7fbe9aa00700 10 client.25003 handle_command_reply: tid=0 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.985+0000 7fbec6325b80 1 client.25003 shutdown 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.985+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 msgr2=0x5618beb06dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.985+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb06dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbea000c5b0 tx=0x7fbea0005840 comp rx=0 tx=0).stop 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.985+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 shutdown_connections 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.985+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb17090 0x5618beb18500 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beb07310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb09ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3354577077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb06dd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 >> 172.21.15.33:0/3354577077 conn(0x5618bea2b2a0 msgr2=0x5618beaf2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbe9aa00700 0 client.25003 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:11.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 shutdown_connections 2024-04-26T11:30:11.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbec6325b80 1 -- 172.21.15.33:0/3354577077 wait complete. 2024-04-26T11:30:11.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:11.986+0000 7fbec6325b80 20 client.25003 trim_cache size 0 max 16384 2024-04-26T11:30:12.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.008+0000 7fbec6325b80 1 Processor -- start 2024-04-26T11:30:12.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.008+0000 7fbec6325b80 1 -- start start 2024-04-26T11:30:12.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beafe080 0x5618beb06d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:12.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb072d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:12.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beb02cd0 0x5618beb056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:12.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbec6325b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618beb16840 con 0x5618beb02cd0 2024-04-26T11:30:12.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbec6325b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5618beb16f10 con 0x5618beafe080 2024-04-26T11:30:12.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbec6325b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5618beb175e0 con 0x5618beb023b0 2024-04-26T11:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbeb9a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beb02cd0 0x5618beb056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbeb9000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beafe080 0x5618beb06d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbeb9a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beb02cd0 0x5618beb056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37030/0 (socket says 172.21.15.33:37030) 2024-04-26T11:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbeb9a00700 1 -- 172.21.15.33:0/1729627427 learned_addr learned my addr 172.21.15.33:0/1729627427 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.009+0000 7fbeb9a00700 1 -- 172.21.15.33:0/1729627427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 msgr2=0x5618beb072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:12.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb072d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbeb9a00700 1 -- 172.21.15.33:0/1729627427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beafe080 msgr2=0x5618beb06d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beafe080 0x5618beb06d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbeb9a00700 1 -- 172.21.15.33:0/1729627427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618beb05340 con 0x5618beb02cd0 2024-04-26T11:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beb02cd0 0x5618beb056c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbeac015000 tx=0x7fbeac020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fec07b6622df6d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbea5600700 1 -- 172.21.15.33:0/1729627427 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac030070 con 0x5618beb02cd0 2024-04-26T11:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbea5600700 1 -- 172.21.15.33:0/1729627427 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeac00be30 con 0x5618beb02cd0 2024-04-26T11:30:12.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.010+0000 7fbea5600700 1 -- 172.21.15.33:0/1729627427 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac025070 con 0x5618beb02cd0 2024-04-26T11:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 -- 172.21.15.33:0/1729627427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beb02cd0 msgr2=0x5618beb056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:12.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beb02cd0 0x5618beb056c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbeac015000 tx=0x7fbeac020040 comp rx=0 tx=0).stop 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 -- 172.21.15.33:0/1729627427 shutdown_connections 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beb02cd0 0x5618beb056c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beafe080 0x5618beb06d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 --2- 172.21.15.33:0/1729627427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 -- 172.21.15.33:0/1729627427 >> 172.21.15.33:0/1729627427 conn(0x5618bea2b2a0 msgr2=0x5618beb03a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 -- 172.21.15.33:0/1729627427 shutdown_connections 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.011+0000 7fbec6325b80 1 -- 172.21.15.33:0/1729627427 wait complete. 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 -- start start 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 Processor -- start 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb21970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb21eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beafcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:12.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618beb18380 con 0x5618beafe080 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5618beb17cb0 con 0x5618beb02cd0 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5618beb18a50 con 0x5618beb023b0 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.012+0000 7fbec6325b80 10 client.0 _ll_register_callbacks cb 0x5618beafa740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb9000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb21970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb21eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb9000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb21970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37046/0 (socket says 172.21.15.33:37046) 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb9000700 1 -- 172.21.15.33:0/3698115843 learned_addr learned my addr 172.21.15.33:0/3698115843 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb9a00700 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beafcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb3e00700 1 -- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 msgr2=0x5618beafcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beafcec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb3e00700 1 -- 172.21.15.33:0/3698115843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 msgr2=0x5618beb21970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb21970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb3e00700 1 -- 172.21.15.33:0/3698115843 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618beafd500 con 0x5618beb023b0 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb9000700 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb21970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbeb3e00700 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb21eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbea00058b0 tx=0x7fbea0005910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22a29427b4cfb0be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.013+0000 7fbe9aa00700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3698115843 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea009cf00 con 0x5618beb023b0 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3698115843 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbea009bc50 con 0x5618beb023b0 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeac00be30 con 0x5618beb023b0 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3698115843 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbea00aa050 con 0x5618beb023b0 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbec6325b80 10 client.25009 fetch_fsmap learned FSMap version 25 2024-04-26T11:30:12.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5618beb7eac0 con 0x5618beb023b0 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3698115843 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeac00be30 con 0x5618beb023b0 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.014+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3698115843 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fbea0084020 con 0x5618beb023b0 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.015+0000 7fbec6325b80 10 client.25009 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.015+0000 7fbec6325b80 10 client.25009 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.015+0000 7fbec6325b80 20 client.25009 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.015+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb83440 0x5618beb82150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.015+0000 7fbec6325b80 4 client.25009 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.015+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "status"}) -- 0x5618bea582c0 con 0x5618beb83440 2024-04-26T11:30:12.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.015+0000 7fbeb9000700 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb83440 0x5618beb82150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbeb9000700 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb83440 0x5618beb82150 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5fc06cf9258ee57 server_cookie=247a072bfcf23f6f in_seq=0 out_seq=0 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbe9aa00700 10 client.25009 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbe9aa00700 1 -- 172.21.15.33:0/3698115843 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5618bea582c0 con 0x5618beb83440 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbe9aa00700 10 client.25009 handle_command_reply: tid=0 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 client.25009 shutdown 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 msgr2=0x5618beb21eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb21eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbea00058b0 tx=0x7fbea0005910 comp rx=0 tx=0).stop 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 shutdown_connections 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618beb83440 0x5618beb82150 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618beb02cd0 0x5618beafcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618beafe080 0x5618beb21970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 --2- 172.21.15.33:0/3698115843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618beb023b0 0x5618beb21eb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbe9aa00700 0 client.25009 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:12.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.016+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 >> 172.21.15.33:0/3698115843 conn(0x5618bea2b2a0 msgr2=0x5618beb04220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.017+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 shutdown_connections 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.017+0000 7fbec6325b80 1 -- 172.21.15.33:0/3698115843 wait complete. 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.017+0000 7fbec6325b80 20 client.25009 trim_cache size 0 max 16384 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24527, 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:30:12.021 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:30:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 218.988999578, 2024-04-26T11:30:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 25, 2024-04-26T11:30:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 90, 2024-04-26T11:30:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T11:30:12.022 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 240.60100684599999 2024-04-26T11:30:12.022 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe9c0a3cc0 msgr2=0x7fbe9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe9c0a3cc0 0x7fbe9c0a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0060b0 tx=0x7fbeac009c80 comp rx=0 tx=0).stop 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb4075f70 msgr2=0x7fbeb41a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb4075f70 0x7fbeb41a06a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbea000e9c0 tx=0x7fbea000ecd0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 shutdown_connections 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe9c0a3cc0 0x7fbe9c0a6180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb41145f0 0x7fbeb41a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbeb4075650 0x7fbeb41a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 --2- 172.21.15.33:0/1522428605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbeb4075f70 0x7fbeb41a06a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 >> 172.21.15.33:0/1522428605 conn(0x7fbeb407a580 msgr2=0x7fbeb4113420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 shutdown_connections 2024-04-26T11:30:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:12.018+0000 7fbebba00700 1 -- 172.21.15.33:0/1522428605 wait complete. 2024-04-26T11:30:12.894 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T11:30:12.906 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T11:30:12.906 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T11:30:12.906 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T11:30:12.906 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-26T11:30:12.907 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T11:30:12.907 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:30:12.907 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-26T11:30:13.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 -- 172.21.15.33:0/2436013945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 msgr2=0x7fd008074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 --2- 172.21.15.33:0/2436013945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 0x7fd008074dc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009930 tx=0x7fcffc02a420 comp rx=0 tx=0).stop 2024-04-26T11:30:13.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 -- 172.21.15.33:0/2436013945 shutdown_connections 2024-04-26T11:30:13.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 --2- 172.21.15.33:0/2436013945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd00810eda0 0x7fd008111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 --2- 172.21.15.33:0/2436013945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd008075390 0x7fd008075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 --2- 172.21.15.33:0/2436013945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 0x7fd008074dc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 -- 172.21.15.33:0/2436013945 >> 172.21.15.33:0/2436013945 conn(0x7fd0080fdd40 msgr2=0x7fd008100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:13.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 -- 172.21.15.33:0/2436013945 shutdown_connections 2024-04-26T11:30:13.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.131+0000 7fd00e000640 1 -- 172.21.15.33:0/2436013945 wait complete. 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.132+0000 7fd00e000640 1 Processor -- start 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.132+0000 7fd00e000640 1 -- start start 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd008075390 0x7fd00819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 0x7fd00819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00e000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00810eda0 0x7fd0081a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00e000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0081148f0 con 0x7fd00810eda0 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00e000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd008114770 con 0x7fd008076970 2024-04-26T11:30:13.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00e000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd008114a70 con 0x7fd008075390 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00d600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00810eda0 0x7fd0081a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd008075390 0x7fd00819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd007e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 0x7fd00819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd008075390 0x7fd00819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57334/0 (socket says 172.21.15.33:57334) 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 -- 172.21.15.33:0/3576022310 learned_addr learned my addr 172.21.15.33:0/3576022310 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 -- 172.21.15.33:0/3576022310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 msgr2=0x7fd00819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 0x7fd00819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 -- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00810eda0 msgr2=0x7fd0081a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00810eda0 0x7fd0081a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00cc00640 1 -- 172.21.15.33:0/3576022310 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0081a6c70 con 0x7fd008075390 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.133+0000 7fd00d600640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00810eda0 0x7fd0081a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.134+0000 7fd00cc00640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd008075390 0x7fd00819f050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009a50 tx=0x7fcffc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:13.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.134+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc03f030 con 0x7fd008075390 2024-04-26T11:30:13.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.134+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcffc031910 con 0x7fd008075390 2024-04-26T11:30:13.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.134+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0081a6f00 con 0x7fd008075390 2024-04-26T11:30:13.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.134+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc035070 con 0x7fd008075390 2024-04-26T11:30:13.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.134+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0081a7300 con 0x7fd008075390 2024-04-26T11:30:13.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.135+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd008104f70 con 0x7fd008075390 2024-04-26T11:30:13.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.137+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcffc042030 con 0x7fd008075390 2024-04-26T11:30:13.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.139+0000 7fd005600640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcff00a3ba0 0x7fcff00a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.139+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcffc0f0d90 con 0x7fd008075390 2024-04-26T11:30:13.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.139+0000 7fd007e00640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcff00a3ba0 0x7fcff00a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:13.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.140+0000 7fd007e00640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcff00a3ba0 0x7fcff00a6060 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcff40095f0 tx=0x7fcff4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:13.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.141+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcffc0bd0e0 con 0x7fd008075390 2024-04-26T11:30:13.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.404+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fd00802dd20 con 0x7fd008075390 2024-04-26T11:30:13.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.563+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fcffc0b9750 con 0x7fd008075390 2024-04-26T11:30:13.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.564+0000 7fd005600640 1 -- 172.21.15.33:0/3576022310 <== mon.1 v2:172.21.15.194:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fcffc0f2af0 con 0x7fd008075390 2024-04-26T11:30:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcff00a3ba0 msgr2=0x7fcff00a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcff00a3ba0 0x7fcff00a6060 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcff40095f0 tx=0x7fcff4005f90 comp rx=0 tx=0).stop 2024-04-26T11:30:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd008075390 msgr2=0x7fd00819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd008075390 0x7fd00819f050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009a50 tx=0x7fcffc0042f0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 shutdown_connections 2024-04-26T11:30:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fcff00a3ba0 0x7fcff00a6060 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd00810eda0 0x7fd0081a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd008076970 0x7fd00819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 --2- 172.21.15.33:0/3576022310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd008075390 0x7fd00819f050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 >> 172.21.15.33:0/3576022310 conn(0x7fd0080fdd40 msgr2=0x7fd008102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 shutdown_connections 2024-04-26T11:30:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.568+0000 7fd00e000640 1 -- 172.21.15.33:0/3576022310 wait complete. 2024-04-26T11:30:13.590 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 -- 172.21.15.33:0/4115208740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40105db0 msgr2=0x7fbe40106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 --2- 172.21.15.33:0/4115208740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40105db0 0x7fbe40106230 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009900 tx=0x7fbe2c02a3f0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 -- 172.21.15.33:0/4115208740 shutdown_connections 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 --2- 172.21.15.33:0/4115208740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe40106770 0x7fbe4010d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 --2- 172.21.15.33:0/4115208740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40105db0 0x7fbe40106230 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 --2- 172.21.15.33:0/4115208740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe40104bc0 0x7fbe40104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 -- 172.21.15.33:0/4115208740 >> 172.21.15.33:0/4115208740 conn(0x7fbe40100150 msgr2=0x7fbe40102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 -- 172.21.15.33:0/4115208740 shutdown_connections 2024-04-26T11:30:13.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.801+0000 7fbe46400640 1 -- 172.21.15.33:0/4115208740 wait complete. 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 Processor -- start 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 -- start start 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40104bc0 0x7fbe4019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe40105db0 0x7fbe4019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe40106770 0x7fbe401a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe40110740 con 0x7fbe40106770 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe401105c0 con 0x7fbe40104bc0 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.802+0000 7fbe46400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbe401108c0 con 0x7fbe40105db0 2024-04-26T11:30:13.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe40105db0 0x7fbe4019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40104bc0 0x7fbe4019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe40105db0 0x7fbe4019b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57382/0 (socket says 172.21.15.33:57382) 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40104bc0 0x7fbe4019ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:60002/0 (socket says 172.21.15.33:60002) 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3ea00640 1 -- 172.21.15.33:0/1332944722 learned_addr learned my addr 172.21.15.33:0/1332944722 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3fe00640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe40106770 0x7fbe401a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3f400640 1 -- 172.21.15.33:0/1332944722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe40105db0 msgr2=0x7fbe4019b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3f400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe40105db0 0x7fbe4019b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3f400640 1 -- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe40106770 msgr2=0x7fbe401a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3f400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe40106770 0x7fbe401a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.803+0000 7fbe3f400640 1 -- 172.21.15.33:0/1332944722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe401a2960 con 0x7fbe40104bc0 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.804+0000 7fbe3f400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40104bc0 0x7fbe4019ad50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000e9c0 tx=0x7fbe3000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:13.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.804+0000 7fbe37e00640 1 -- 172.21.15.33:0/1332944722 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000ccc0 con 0x7fbe40104bc0 2024-04-26T11:30:13.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.804+0000 7fbe37e00640 1 -- 172.21.15.33:0/1332944722 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe300035f0 con 0x7fbe40104bc0 2024-04-26T11:30:13.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.804+0000 7fbe37e00640 1 -- 172.21.15.33:0/1332944722 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000ce60 con 0x7fbe40104bc0 2024-04-26T11:30:13.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.804+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe401a2c50 con 0x7fbe40104bc0 2024-04-26T11:30:13.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.804+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe401a3150 con 0x7fbe40104bc0 2024-04-26T11:30:13.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.805+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe4010bee0 con 0x7fbe40104bc0 2024-04-26T11:30:13.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.807+0000 7fbe37e00640 1 -- 172.21.15.33:0/1332944722 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fbe30004000 con 0x7fbe40104bc0 2024-04-26T11:30:13.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.808+0000 7fbe37e00640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 0x7fbe240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:13.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.809+0000 7fbe3ea00640 1 -- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 msgr2=0x7fbe240a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:13.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.809+0000 7fbe37e00640 1 -- 172.21.15.33:0/1332944722 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbe300d0870 con 0x7fbe40104bc0 2024-04-26T11:30:13.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.809+0000 7fbe3ea00640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 0x7fbe240a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:30:13.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:13.813+0000 7fbe37e00640 1 -- 172.21.15.33:0/1332944722 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe3009d210 con 0x7fbe40104bc0 2024-04-26T11:30:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-mon[33017]: pgmap v437: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3576022310' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T11:30:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T11:30:13.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: ignoring --setuser ceph since I am not root 2024-04-26T11:30:13.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: ignoring --setgroup ceph since I am not root 2024-04-26T11:30:13.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-mgr[34886]: -- 172.21.15.194:0/1615723305 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f2301705a0 con 0x55f230144800 2024-04-26T11:30:13.855 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:13.732+0000 7f337b4cf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:30:13.898 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:13 smithi033 ceph-mon[27845]: pgmap v437: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:13.899 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:13 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3576022310' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T11:30:13.899 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:13 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T11:30:13.899 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:13 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: ignoring --setuser ceph since I am not root 2024-04-26T11:30:13.899 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:13 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: ignoring --setgroup ceph since I am not root 2024-04-26T11:30:13.899 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:13 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:13.896+0000 7fb16fa63200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:30:13.912 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-mon[32753]: pgmap v437: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:30:13.913 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3576022310' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T11:30:13.913 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T11:30:13.913 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: ignoring --setuser ceph since I am not root 2024-04-26T11:30:13.913 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: ignoring --setgroup ceph since I am not root 2024-04-26T11:30:13.913 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-mgr[34158]: -- 172.21.15.96:0/3190716818 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628603325a0 con 0x562860307000 2024-04-26T11:30:13.913 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:13.727+0000 7f63b4498200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T11:30:13.913 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:13 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:13.912+0000 7f63b4498200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:30:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.010+0000 7fbe3ea00640 1 -- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 msgr2=0x7fbe240a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.010+0000 7fbe3ea00640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 0x7fbe240a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:30:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.024+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fbe4002dad0 con 0x7fbe40104bc0 2024-04-26T11:30:14.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.030+0000 7fbe37e00640 1 -- 172.21.15.33:0/1332944722 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7fbe3009cb10 con 0x7fbe40104bc0 2024-04-26T11:30:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 msgr2=0x7fbe240a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:30:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 0x7fbe240a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40104bc0 msgr2=0x7fbe4019ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40104bc0 0x7fbe4019ad50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000e9c0 tx=0x7fbe3000ee80 comp rx=0 tx=0).stop 2024-04-26T11:30:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 shutdown_connections 2024-04-26T11:30:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe40105db0 0x7fbe4019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fbe240a3d20 0x7fbe240a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe40106770 0x7fbe401a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 --2- 172.21.15.33:0/1332944722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe40104bc0 0x7fbe4019ad50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.033+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 >> 172.21.15.33:0/1332944722 conn(0x7fbe40100150 msgr2=0x7fbe401090b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.034+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 shutdown_connections 2024-04-26T11:30:14.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.034+0000 7fbe46400640 1 -- 172.21.15.33:0/1332944722 wait complete. 2024-04-26T11:30:14.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-26T11:30:14.182 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:14 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:14.015+0000 7f63b4498200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:30:14.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.241+0000 7f2030800640 1 -- 172.21.15.33:0/774506401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 msgr2=0x7f2028111400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:14.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.241+0000 7f2030800640 1 --2- 172.21.15.33:0/774506401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 0x7f2028111400 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2024009ca0 tx=0x7f202402a8b0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.241+0000 7f2030800640 1 -- 172.21.15.33:0/774506401 shutdown_connections 2024-04-26T11:30:14.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.241+0000 7f2030800640 1 --2- 172.21.15.33:0/774506401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f202810a1b0 0x7f202810a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.241+0000 7f2030800640 1 --2- 172.21.15.33:0/774506401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 0x7f2028111400 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.241+0000 7f2030800640 1 --2- 172.21.15.33:0/774506401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 0x7f20281093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.241+0000 7f2030800640 1 -- 172.21.15.33:0/774506401 >> 172.21.15.33:0/774506401 conn(0x7f2028077f80 msgr2=0x7f202807a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:14.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.242+0000 7f2030800640 1 -- 172.21.15.33:0/774506401 shutdown_connections 2024-04-26T11:30:14.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.242+0000 7f2030800640 1 -- 172.21.15.33:0/774506401 wait complete. 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 Processor -- start 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 -- start start 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 0x7f202819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f202810a1b0 0x7f202819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 0x7f20281a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2028114b00 con 0x7f202810ab70 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2028114980 con 0x7f2028108fc0 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f2030800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2028114c80 con 0x7f202810a1b0 2024-04-26T11:30:14.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f202de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 0x7f202819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f202de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 0x7f202819f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:60032/0 (socket says 172.21.15.33:60032) 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f202de00640 1 -- 172.21.15.33:0/4110531275 learned_addr learned my addr 172.21.15.33:0/4110531275 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.243+0000 7f202e800640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 0x7f20281a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.244+0000 7f202d400640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f202810a1b0 0x7f202819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.244+0000 7f202de00640 1 -- 172.21.15.33:0/4110531275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f202810a1b0 msgr2=0x7f202819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.244+0000 7f202de00640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f202810a1b0 0x7f202819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.244+0000 7f202de00640 1 -- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 msgr2=0x7f20281a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.244+0000 7f202de00640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 0x7f20281a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.244+0000 7f202de00640 1 -- 172.21.15.33:0/4110531275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20281a6d20 con 0x7f2028108fc0 2024-04-26T11:30:14.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.244+0000 7f202de00640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 0x7f202819f100 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9c0 tx=0x7f201c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:14.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.245+0000 7f2022a00640 1 -- 172.21.15.33:0/4110531275 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c00cde0 con 0x7f2028108fc0 2024-04-26T11:30:14.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.245+0000 7f2022a00640 1 -- 172.21.15.33:0/4110531275 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c0108e0 con 0x7f2028108fc0 2024-04-26T11:30:14.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.245+0000 7f2022a00640 1 -- 172.21.15.33:0/4110531275 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c018960 con 0x7f2028108fc0 2024-04-26T11:30:14.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.245+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20281a6fb0 con 0x7f2028108fc0 2024-04-26T11:30:14.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.245+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20281a7420 con 0x7f2028108fc0 2024-04-26T11:30:14.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.246+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20281102e0 con 0x7f2028108fc0 2024-04-26T11:30:14.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.250+0000 7f2022a00640 1 -- 172.21.15.33:0/4110531275 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f201c01f020 con 0x7f2028108fc0 2024-04-26T11:30:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.251+0000 7f2022a00640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 0x7f20100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.251+0000 7f202d400640 1 -- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 msgr2=0x7f20100a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.251+0000 7f2022a00640 1 -- 172.21.15.33:0/4110531275 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f201c0d0670 con 0x7f2028108fc0 2024-04-26T11:30:14.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.251+0000 7f202d400640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 0x7f20100a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:30:14.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.256+0000 7f2022a00640 1 -- 172.21.15.33:0/4110531275 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f201c053ff0 con 0x7f2028108fc0 2024-04-26T11:30:14.330 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:13 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:13.915+0000 7f337b4cf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:30:14.331 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:14.018+0000 7f337b4cf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:30:14.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:14.162+0000 7fb16fa63200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:30:14.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:14.312+0000 7fb16fa63200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T11:30:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.452+0000 7f202d400640 1 -- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 msgr2=0x7f20100a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:14.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.452+0000 7f202d400640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 0x7f20100a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.461+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f202802dad0 con 0x7f2028108fc0 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.469+0000 7f2022a00640 1 -- 172.21.15.33:0/4110531275 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f201c09d010 con 0x7f2028108fc0 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 msgr2=0x7f20100a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 0x7f20100a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 msgr2=0x7f202819f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 0x7f202819f100 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9c0 tx=0x7f201c00ee80 comp rx=0 tx=0).stop 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 shutdown_connections 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f202810a1b0 0x7f202819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f20100a3d20 0x7f20100a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f202810ab70 0x7f20281a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 --2- 172.21.15.33:0/4110531275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2028108fc0 0x7f202819f100 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:14.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 >> 172.21.15.33:0/4110531275 conn(0x7f2028077f80 msgr2=0x7f202810d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:14.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 shutdown_connections 2024-04-26T11:30:14.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:14.472+0000 7f2030800640 1 -- 172.21.15.33:0/4110531275 wait complete. 2024-04-26T11:30:14.526 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T11:30:14.550 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T11:30:14.551 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T11:30:14.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:14.551 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-04-26T11:30:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:14 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T11:30:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:14 smithi194 ceph-mon[33017]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-26T11:30:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:14 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' 2024-04-26T11:30:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:14 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' 2024-04-26T11:30:14.604 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:14.329+0000 7f337b4cf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:30:14.604 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:14.414+0000 7f337b4cf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:30:14.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:14.607 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link delete ceph-brx 2024-04-26T11:30:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:14 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T11:30:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:14 smithi096 ceph-mon[32753]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-26T11:30:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:14 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' 2024-04-26T11:30:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:14 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' 2024-04-26T11:30:14.683 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:14 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:14.325+0000 7f63b4498200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:30:14.683 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:14 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:14.409+0000 7f63b4498200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:30:14.684 INFO:teuthology.orchestra.run.smithi033.stderr:Cannot find device "ceph-brx" 2024-04-26T11:30:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T11:30:14.686 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v2', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-26T11:30:14.686 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T11:30:14.687 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-26T11:30:14.687 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-26T11:30:14.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T11:30:14.744 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:14.725+0000 7fb16fa63200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T11:30:14.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T11:30:14.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-mon[27845]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-26T11:30:14.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' 2024-04-26T11:30:14.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' 2024-04-26T11:30:15.115 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:15.176 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:14 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:14.842+0000 7fb16fa63200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T11:30:15.921 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:15.584+0000 7f63b4498200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:30:15.921 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:15.751+0000 7f63b4498200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:30:15.921 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:15.920+0000 7f63b4498200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:30:15.956 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:15 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:15.614+0000 7f337b4cf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:30:15.956 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:15 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:15.782+0000 7f337b4cf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:30:15.956 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:15 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:15.955+0000 7f337b4cf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:30:16.503 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:16 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:16.207+0000 7f63b4498200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:30:16.503 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:16 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:16.301+0000 7f63b4498200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:30:16.542 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:16 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:16.243+0000 7f337b4cf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:30:16.543 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:16 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:16.337+0000 7f337b4cf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:30:16.839 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:16.473+0000 7fb16fa63200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T11:30:16.839 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:16.711+0000 7fb16fa63200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T11:30:16.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:16 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:16.541+0000 7f337b4cf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:30:16.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:16 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:16.736+0000 7f337b4cf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:30:16.933 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:16 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:16.502+0000 7f63b4498200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:30:16.933 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:16 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:16.695+0000 7f63b4498200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:30:17.176 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:16.942+0000 7fb16fa63200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.346+0000 7f84e9400700 1 -- 172.21.15.33:0/986210079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 msgr2=0x7f84e41098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.346+0000 7f84e9400700 1 --2- 172.21.15.33:0/986210079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 0x7f84e41098a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009990 tx=0x7f84d4009ca0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.346+0000 7f84e9400700 1 -- 172.21.15.33:0/986210079 shutdown_connections 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.346+0000 7f84e9400700 1 --2- 172.21.15.33:0/986210079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84e4103e30 0x7f84e4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.346+0000 7f84e9400700 1 --2- 172.21.15.33:0/986210079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 0x7f84e41098a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.346+0000 7f84e9400700 1 --2- 172.21.15.33:0/986210079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 0x7f84e41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.346+0000 7f84e9400700 1 -- 172.21.15.33:0/986210079 >> 172.21.15.33:0/986210079 conn(0x7f84e407a580 msgr2=0x7f84e407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.347+0000 7f84e9400700 1 -- 172.21.15.33:0/986210079 shutdown_connections 2024-04-26T11:30:17.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.347+0000 7f84e9400700 1 -- 172.21.15.33:0/986210079 wait complete. 2024-04-26T11:30:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 Processor -- start 2024-04-26T11:30:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 -- start start 2024-04-26T11:30:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 0x7f84e419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:17.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84e4103e30 0x7f84e419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 0x7f84e4197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84e41114b0 con 0x7f84e4103e30 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84e4111330 con 0x7f84e4103470 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.348+0000 7f84e9400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f84e4111630 con 0x7f84e41094c0 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 0x7f84e419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e1e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84e4103e30 0x7f84e419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 0x7f84e419d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:60050/0 (socket says 172.21.15.33:60050) 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e1e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84e4103e30 0x7f84e419dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37128/0 (socket says 172.21.15.33:37128) 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 -- 172.21.15.33:0/2112208074 learned_addr learned my addr 172.21.15.33:0/2112208074 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:17.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e3200700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 0x7f84e4197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 -- 172.21.15.33:0/2112208074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 msgr2=0x7f84e4197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 0x7f84e4197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 -- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84e4103e30 msgr2=0x7f84e419dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84e4103e30 0x7f84e419dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e2800700 1 -- 172.21.15.33:0/2112208074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e4198010 con 0x7f84e4103470 2024-04-26T11:30:17.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.349+0000 7f84e3200700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 0x7f84e4197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.350+0000 7f84e2800700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 0x7f84e419d660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f84d4005910 tx=0x7f84d400b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.350+0000 7f84db400700 1 -- 172.21.15.33:0/2112208074 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4027070 con 0x7f84e4103470 2024-04-26T11:30:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.350+0000 7f84db400700 1 -- 172.21.15.33:0/2112208074 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d4022470 con 0x7f84e4103470 2024-04-26T11:30:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.350+0000 7f84db400700 1 -- 172.21.15.33:0/2112208074 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4017650 con 0x7f84e4103470 2024-04-26T11:30:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.350+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e41982a0 con 0x7f84e4103470 2024-04-26T11:30:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.350+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e41a4410 con 0x7f84e4103470 2024-04-26T11:30:17.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.351+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e4104b60 con 0x7f84e4103470 2024-04-26T11:30:17.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.353+0000 7f84db400700 1 -- 172.21.15.33:0/2112208074 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f84d402a070 con 0x7f84e4103470 2024-04-26T11:30:17.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.354+0000 7f84db400700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 0x7f84c80a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:17.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.354+0000 7f84e1e00700 1 -- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 msgr2=0x7f84c80a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:17.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.354+0000 7f84db400700 1 -- 172.21.15.33:0/2112208074 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f84d40d1150 con 0x7f84e4103470 2024-04-26T11:30:17.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.354+0000 7f84e1e00700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 0x7f84c80a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:30:17.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.358+0000 7f84db400700 1 -- 172.21.15.33:0/2112208074 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84d4013070 con 0x7f84e4103470 2024-04-26T11:30:17.520 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:17 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:17.384+0000 7fb16fa63200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T11:30:17.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.555+0000 7f84e1e00700 1 -- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 msgr2=0x7f84c80a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:17.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.555+0000 7f84e1e00700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 0x7f84c80a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:30:17.603 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:17 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:17.346+0000 7f337b4cf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:30:17.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.637+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f84e402e490 con 0x7f84e4103470 2024-04-26T11:30:17.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.638+0000 7f84db400700 1 -- 172.21.15.33:0/2112208074 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f84d409dca0 con 0x7f84e4103470 2024-04-26T11:30:17.639 INFO:teuthology.orchestra.run.smithi033.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 msgr2=0x7f84c80a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 0x7f84c80a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 msgr2=0x7f84e419d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 0x7f84e419d660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f84d4005910 tx=0x7f84d400b670 comp rx=0 tx=0).stop 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 shutdown_connections 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7f84c80a3c90 0x7f84c80a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84e41094c0 0x7f84e4197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84e4103470 0x7f84e419d660 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 --2- 172.21.15.33:0/2112208074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84e4103e30 0x7f84e419dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:17.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 >> 172.21.15.33:0/2112208074 conn(0x7f84e407a580 msgr2=0x7f84e4102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:17.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 shutdown_connections 2024-04-26T11:30:17.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:17.641+0000 7f84e9400700 1 -- 172.21.15.33:0/2112208074 wait complete. 2024-04-26T11:30:17.682 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:17 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:17.294+0000 7f63b4498200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:30:17.799 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:17 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:17.519+0000 7fb16fa63200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T11:30:17.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:17 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2112208074' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T11:30:18.015 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:17 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2112208074' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T11:30:18.015 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:17 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:17.717+0000 7f63b4498200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:30:18.015 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:17 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:17.814+0000 7f63b4498200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:30:18.058 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:17 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:17.797+0000 7fb16fa63200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T11:30:18.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:17 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2112208074' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T11:30:18.068 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:17 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:17.769+0000 7f337b4cf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:30:18.068 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:17 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:17.866+0000 7f337b4cf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:30:18.354 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:18 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:18.067+0000 7f337b4cf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:30:18.426 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:18 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:18.075+0000 7fb16fa63200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T11:30:18.432 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:18 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:18.014+0000 7f63b4498200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:30:18.524 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi033.front.sepia.ceph.com 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T11:30:18.525 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-26T11:30:18.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:18.525 DEBUG:teuthology.orchestra.run.smithi033:> ip addr 2024-04-26T11:30:18.570 INFO:teuthology.orchestra.run.smithi033.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T11:30:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T11:30:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T11:30:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-04-26T11:30:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 ::1/128 scope host 2024-04-26T11:30:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:14 brd ff:ff:ff:ff:ff:ff 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f0 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:15 brd ff:ff:ff:ff:ff:ff 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f1 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:94 brd ff:ff:ff:ff:ff:ff 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f0 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: inet 172.21.15.33/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft 41877sec preferred_lft 41877sec 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 fe80::ae1f:6bff:fea5:d294/64 scope link noprefixroute 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T11:30:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:95 brd ff:ff:ff:ff:ff:ff 2024-04-26T11:30:18.572 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f1 2024-04-26T11:30:18.572 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T11:30:18.572 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T11:30:18.572 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-04-26T11:30:18.572 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add name ceph-brx type bridge 2024-04-26T11:30:18.572 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr flush dev ceph-brx 2024-04-26T11:30:18.572 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set ceph-brx up 2024-04-26T11:30:18.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T11:30:18.573 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-04-26T11:30:18.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T11:30:18.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:18 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:18.594+0000 7f337b4cf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:30:18.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:18 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:18.687+0000 7f337b4cf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:30:18.867 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:18 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:18.533+0000 7f63b4498200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:30:18.867 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:18 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:18.624+0000 7f63b4498200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:30:18.997 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:18 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:18.936+0000 7fb16fa63200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T11:30:19.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:19.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:19.065 DEBUG:teuthology.orchestra.run.smithi033:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T11:30:19.137 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-04-26T11:30:19.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:19.141 DEBUG:teuthology.orchestra.run.smithi033:> route 2024-04-26T11:30:19.182 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:18 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:18.866+0000 7f63b4498200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:30:19.183 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:18 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:18.998+0000 7f63b4498200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:30:19.183 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:19 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:19.092+0000 7f63b4498200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:30:19.196 INFO:teuthology.orchestra.run.smithi033.stdout:Kernel IP routing table 2024-04-26T11:30:19.196 INFO:teuthology.orchestra.run.smithi033.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T11:30:19.196 INFO:teuthology.orchestra.run.smithi033.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T11:30:19.196 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T11:30:19.196 INFO:teuthology.orchestra.run.smithi033.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T11:30:19.197 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T11:30:19.197 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-04-26T11:30:19.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T11:30:19.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T11:30:19.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T11:30:19.197 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-04-26T11:30:19.273 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:18 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:18.930+0000 7f337b4cf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:30:19.273 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:19 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:19.062+0000 7f337b4cf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:30:19.273 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:19 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:19.157+0000 7f337b4cf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:30:19.273 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:19 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:19.272+0000 7f337b4cf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:30:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T11:30:19.509 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:19 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:19.208+0000 7f63b4498200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:30:19.614 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:19.532+0000 7fb16fa63200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T11:30:19.854 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:19 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:19.578+0000 7f337b4cf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:30:19.926 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:19.670+0000 7fb16fa63200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T11:30:19.933 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:19 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:19.508+0000 7f63b4498200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:30:20.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:20.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:20.021 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-04-26T11:30:20.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:20.071 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list-id 2024-04-26T11:30:20.169 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T11:30:20.170 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-04-26T11:30:20.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-26T11:30:20.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-26T11:30:20.170 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-04-26T11:30:20.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T11:30:20.301 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:19 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:19.952+0000 7fb16fa63200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T11:30:20.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:20.356 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-26T11:30:20.356 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-04-26T11:30:20.388 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:20.101+0000 7f63b4498200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:30:20.388 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-y[34154]: 2024-04-26T11:30:20.317+0000 7f63b4498200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:30:20.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T11:30:20.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-mon[27845]: Standby manager daemon y restarted 2024-04-26T11:30:20.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-mon[27845]: Standby manager daemon y started 2024-04-26T11:30:20.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T11:30:20.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:30:20.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T11:30:20.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:30:20.604 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:20.180+0000 7f337b4cf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:30:20.604 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-z[34882]: 2024-04-26T11:30:20.401+0000 7f337b4cf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:30:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-mon[33017]: Standby manager daemon y restarted 2024-04-26T11:30:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-mon[33017]: Standby manager daemon y started 2024-04-26T11:30:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T11:30:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:30:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T11:30:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:20 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:30:20.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:20.640 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T11:30:20.640 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-04-26T11:30:20.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set brx.0 up 2024-04-26T11:30:20.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T11:30:20.640 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-04-26T11:30:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-mon[32753]: Standby manager daemon y restarted 2024-04-26T11:30:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-mon[32753]: Standby manager daemon y started 2024-04-26T11:30:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T11:30:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:30:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T11:30:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:20 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.96:0/2214453416' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:30:20.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T11:30:20.816 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:20.658+0000 7fb16fa63200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T11:30:20.816 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:20 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:20.789+0000 7fb16fa63200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T11:30:20.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:20.829 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T11:30:20.829 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -v /tmp/cephfs 2024-04-26T11:30:20.871 INFO:teuthology.orchestra.run.smithi033.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-26T11:30:20.871 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T11:30:20.872 DEBUG:teuthology.orchestra.run.smithi033:> chmod 0000 /tmp/cephfs 2024-04-26T11:30:20.966 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T11:30:20.966 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T11:30:20.966 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/2b1eefe1-b994-4781-af0e-f418abad6198 2024-04-26T11:30:20.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:20.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/2b1eefe1-b994-4781-af0e-f418abad6198 /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context=system_u:object_r:postgresql_db_t:s0,nofallback 2024-04-26T11:30:21.126 INFO:teuthology.orchestra.run.smithi033.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-26T11:30:21.126 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-26T11:30:21.429 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:21.142+0000 7fb16fa63200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T11:30:21.429 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:21.337+0000 7fb16fa63200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T11:30:21.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-mon[27845]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-04-26T11:30:21.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-mon[27845]: Standby manager daemon z restarted 2024-04-26T11:30:21.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-mon[27845]: Standby manager daemon z started 2024-04-26T11:30:21.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T11:30:21.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:30:21.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T11:30:21.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-mon[27845]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:30:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:21 smithi096 ceph-mon[32753]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-04-26T11:30:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:21 smithi096 ceph-mon[32753]: Standby manager daemon z restarted 2024-04-26T11:30:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:21 smithi096 ceph-mon[32753]: Standby manager daemon z started 2024-04-26T11:30:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:21 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T11:30:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:21 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:30:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:21 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T11:30:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:21 smithi096 ceph-mon[32753]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:30:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:21 smithi194 ceph-mon[33017]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-04-26T11:30:21.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:21 smithi194 ceph-mon[33017]: Standby manager daemon z restarted 2024-04-26T11:30:21.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:21 smithi194 ceph-mon[33017]: Standby manager daemon z started 2024-04-26T11:30:21.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:21 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T11:30:21.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:21 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T11:30:21.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:21 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T11:30:21.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:21 smithi194 ceph-mon[33017]: from='mgr.? 172.21.15.194:0/643466263' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T11:30:21.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:21.477+0000 7fb16fa63200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T11:30:21.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:21 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:21.639+0000 7fb16fa63200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T11:30:21.949 INFO:teuthology.orchestra.run.smithi033.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-26T11:30:21.949 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-26T11:30:21.949 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: resolved to: "172.21.15.33:3300,172.21.15.96:3300,172.21.15.194:3300" 2024-04-26T11:30:21.950 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: trying mount with new device syntax: 0@5cc7f146-03be-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/2b1eefe1-b994-4781-af0e-f418abad6198 2024-04-26T11:30:21.950 INFO:teuthology.orchestra.run.smithi033.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",name=0,key=0,mon_addr=172.21.15.33:3300/172.21.15.96:3300/172.21.15.194:3300" will pass to kernel 2024-04-26T11:30:21.957 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T11:30:21.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T11:30:21.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 1777 /tmp/cephfs 2024-04-26T11:30:22.026 DEBUG:teuthology.orchestra.run.smithi033:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T11:30:22.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T11:30:22.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:22.131 INFO:teuthology.orchestra.run.smithi033.stdout:client14952 2024-04-26T11:30:22.132 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-26T11:30:22.132 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T11:30:22.176 DEBUG:teuthology.orchestra.run.smithi033:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T11:30:22.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T11:30:22.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:22.334 INFO:teuthology.orchestra.run.smithi033.stdout:client14952 2024-04-26T11:30:22.334 DEBUG:teuthology.orchestra.run.smithi033:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T11:30:22.403 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:22.027+0000 7fb16fa63200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T11:30:22.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T11:30:22.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:22.436 INFO:teuthology.orchestra.run.smithi033.stdout:client14952 2024-04-26T11:30:22.436 DEBUG:teuthology.orchestra.run.smithi033:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5cc7f146-03be-11ef-bc93-c7b262605968.client14952/status') 2024-04-26T11:30:22.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T11:30:22.562 INFO:teuthology.orchestra.run.smithi033.stdout:instance: client.14952 (3)192.168.0.1:0/3404657790 2024-04-26T11:30:22.562 INFO:teuthology.orchestra.run.smithi033.stdout:blocklisted: false 2024-04-26T11:30:22.562 INFO:teuthology.orchestra.run.smithi033.stderr:0+1 records in 2024-04-26T11:30:22.562 INFO:teuthology.orchestra.run.smithi033.stderr:0+1 records out 2024-04-26T11:30:22.562 INFO:teuthology.orchestra.run.smithi033.stderr:70 bytes copied, 0.000101504 s, 690 kB/s 2024-04-26T11:30:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T11:30:22.574 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T11:30:22.583 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T11:30:22.584 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:30:22.584 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-26T11:30:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:22 smithi033 ceph-mon[27845]: mgrmap e21: x(active, since 12m), standbys: z, y 2024-04-26T11:30:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:22 smithi096 ceph-mon[32753]: mgrmap e21: x(active, since 12m), standbys: z, y 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.773+0000 7fa807e00640 1 -- 172.21.15.33:0/4180422576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 msgr2=0x7fa808103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.773+0000 7fa807e00640 1 --2- 172.21.15.33:0/4180422576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa808103e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009900 tx=0x7fa7f002a410 comp rx=0 tx=0).stop 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.774+0000 7fa807e00640 1 -- 172.21.15.33:0/4180422576 shutdown_connections 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.774+0000 7fa807e00640 1 --2- 172.21.15.33:0/4180422576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8081043a0 0x7fa80810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.774+0000 7fa807e00640 1 --2- 172.21.15.33:0/4180422576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa808103e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.774+0000 7fa807e00640 1 --2- 172.21.15.33:0/4180422576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8081027f0 0x7fa808102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.774+0000 7fa807e00640 1 -- 172.21.15.33:0/4180422576 >> 172.21.15.33:0/4180422576 conn(0x7fa8080fdd60 msgr2=0x7fa8081001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:22.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.774+0000 7fa807e00640 1 -- 172.21.15.33:0/4180422576 shutdown_connections 2024-04-26T11:30:22.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.774+0000 7fa807e00640 1 -- 172.21.15.33:0/4180422576 wait complete. 2024-04-26T11:30:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 Processor -- start 2024-04-26T11:30:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 -- start start 2024-04-26T11:30:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8081027f0 0x7fa80819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa80819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8081043a0 0x7fa8081a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa80810e320 con 0x7fa8081039e0 2024-04-26T11:30:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa80810e1a0 con 0x7fa8081027f0 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa807e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa80810e4a0 con 0x7fa8081043a0 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa7ffe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa80819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.775+0000 7fa7ffe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa80819b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40740/0 (socket says 172.21.15.33:40740) 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa7ffe00640 1 -- 172.21.15.33:0/917940793 learned_addr learned my addr 172.21.15.33:0/917940793 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa806a00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8081027f0 0x7fa80819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa807400640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8081043a0 0x7fa8081a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa806a00640 1 -- 172.21.15.33:0/917940793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8081043a0 msgr2=0x7fa8081a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa806a00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8081043a0 0x7fa8081a21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa806a00640 1 -- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 msgr2=0x7fa80819b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa806a00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa80819b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa806a00640 1 -- 172.21.15.33:0/917940793 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8081a2890 con 0x7fa8081027f0 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa7ffe00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa80819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.776+0000 7fa806a00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8081027f0 0x7fa80819ac80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800e9c0 tx=0x7fa7f800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.777+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800ccc0 con 0x7fa8081027f0 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.777+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8081a2b80 con 0x7fa8081027f0 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.777+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7f80035f0 con 0x7fa8081027f0 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.777+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800ce60 con 0x7fa8081027f0 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.777+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8081a30d0 con 0x7fa8081027f0 2024-04-26T11:30:22.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.778+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8081a32a0 con 0x7fa8081027f0 2024-04-26T11:30:22.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.780+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 138663+0+0 (secure 0 0 0) 0x7fa7f8004000 con 0x7fa8081027f0 2024-04-26T11:30:22.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.781+0000 7fa7ff400640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 0x7fa7ec0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:22.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.781+0000 7fa7ffe00640 1 -- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 msgr2=0x7fa7ec0a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:22.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.781+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fa7f80d08a0 con 0x7fa8081027f0 2024-04-26T11:30:22.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.781+0000 7fa7ffe00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 0x7fa7ec0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:30:22.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.785+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7f809d240 con 0x7fa8081027f0 2024-04-26T11:30:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:22 smithi194 ceph-mon[33017]: mgrmap e21: x(active, since 12m), standbys: z, y 2024-04-26T11:30:22.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.982+0000 7fa7ffe00640 1 -- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 msgr2=0x7fa7ec0a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/657098496 2024-04-26T11:30:22.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:22.982+0000 7fa7ffe00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 0x7fa7ec0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:30:23.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.035+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fa8081a3470 con 0x7fa8081027f0 2024-04-26T11:30:23.150 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:22 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:22.848+0000 7fb16fa63200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T11:30:23.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.172+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7fa7f8099120 con 0x7fa8081027f0 2024-04-26T11:30:23.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.174+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 msgr2=0x7fa7ec0a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:30:23.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.174+0000 7fa7ff400640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 0x7fa7ec0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.407 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.149+0000 7fb16fa63200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T11:30:23.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.422+0000 7fa7ff400640 1 -- 172.21.15.33:0/917940793 <== mon.2 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7fa7f80586f0 con 0x7fa8081027f0 2024-04-26T11:30:23.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8081027f0 msgr2=0x7fa80819ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:23.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8081027f0 0x7fa80819ac80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800e9c0 tx=0x7fa7f800ee80 comp rx=0 tx=0).stop 2024-04-26T11:30:23.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 shutdown_connections 2024-04-26T11:30:23.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8081043a0 0x7fa8081a21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:6800/657098496,v1:172.21.15.33:6801/657098496] conn(0x7fa7ec0a3bf0 0x7fa7ec0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8081039e0 0x7fa80819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 --2- 172.21.15.33:0/917940793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8081027f0 0x7fa80819ac80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 >> 172.21.15.33:0/917940793 conn(0x7fa8080fdd60 msgr2=0x7fa808074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:23.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 shutdown_connections 2024-04-26T11:30:23.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.425+0000 7fa807e00640 1 -- 172.21.15.33:0/917940793 wait complete. 2024-04-26T11:30:23.449 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-26T11:30:23.459 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T11:30:23.483 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T11:30:23.484 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:30:23.484 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-26T11:30:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 -- 172.21.15.33:0/1914357788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56410a680 msgr2=0x7fc56410cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 --2- 172.21.15.33:0/1914357788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56410a680 0x7fc56410cb20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009ca0 tx=0x7fc55c02a950 comp rx=0 tx=0).stop 2024-04-26T11:30:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 -- 172.21.15.33:0/1914357788 shutdown_connections 2024-04-26T11:30:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 --2- 172.21.15.33:0/1914357788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc564075330 0x7fc564075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 --2- 172.21.15.33:0/1914357788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56410a680 0x7fc56410cb20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 --2- 172.21.15.33:0/1914357788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc564076910 0x7fc564074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 -- 172.21.15.33:0/1914357788 >> 172.21.15.33:0/1914357788 conn(0x7fc5640fdd60 msgr2=0x7fc5641001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:23.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 -- 172.21.15.33:0/1914357788 shutdown_connections 2024-04-26T11:30:23.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.668+0000 7fc569200640 1 -- 172.21.15.33:0/1914357788 wait complete. 2024-04-26T11:30:23.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 Processor -- start 2024-04-26T11:30:23.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 -- start start 2024-04-26T11:30:23.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc564075330 0x7fc56419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:23.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc564076910 0x7fc56419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc56410a680 0x7fc5641a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc564110570 con 0x7fc564076910 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5641103f0 con 0x7fc564075330 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc569200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc5641106f0 con 0x7fc56410a680 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.669+0000 7fc563400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc564075330 0x7fc56419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc564075330 0x7fc56419ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37234/0 (socket says 172.21.15.33:37234) 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc562a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc564076910 0x7fc56419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 -- 172.21.15.33:0/4051555547 learned_addr learned my addr 172.21.15.33:0/4051555547 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563e00640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc56410a680 0x7fc5641a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 -- 172.21.15.33:0/4051555547 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc56410a680 msgr2=0x7fc5641a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc56410a680 0x7fc5641a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 -- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc564076910 msgr2=0x7fc56419b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc564076910 0x7fc56419b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:23.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 -- 172.21.15.33:0/4051555547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5641a27a0 con 0x7fc564075330 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc563400640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc564075330 0x7fc56419ab80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc55400e9f0 tx=0x7fc55400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55400cd80 con 0x7fc564075330 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5540108e0 con 0x7fc564075330 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.670+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554018980 con 0x7fc564075330 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.671+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5641a2a90 con 0x7fc564075330 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.671+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5641a2f90 con 0x7fc564075330 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.672+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc564109440 con 0x7fc564075330 2024-04-26T11:30:23.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.674+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7fc554010430 con 0x7fc564075330 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.502+0000 7fb0bfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.502+0000 7fb0bfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.502+0000 7fb0bfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.502+0000 7fb0bfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.502+0000 7fb0bfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.503+0000 7fb0c5400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.504+0000 7fb0c5400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.504+0000 7fb0c5400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.504+0000 7fb0c5400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:30:23.504+0000 7fb0c5400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/917940793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: Active manager daemon x restarted 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: Activating manager daemon x 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: mgrmap e22: x(active, starting, since 0.0247826s), standbys: z, y 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:30:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T11:30:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T11:30:23.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:30:23.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:30:23.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:30:23.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.676+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc5540d0320 con 0x7fc564075330 2024-04-26T11:30:23.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:30:23.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:30:23.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:30:23.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: Manager daemon x is now available 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:30:23.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:30:23.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:23.679+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc55409cf90 con 0x7fc564075330 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/917940793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: Active manager daemon x restarted 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: Activating manager daemon x 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: mgrmap e22: x(active, starting, since 0.0247826s), standbys: z, y 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:30:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T11:30:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:30:23.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: Manager daemon x is now available 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:30:23.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:30:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/917940793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T11:30:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T11:30:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: Active manager daemon x restarted 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: Activating manager daemon x 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: mgrmap e22: x(active, starting, since 0.0247826s), standbys: z, y 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:30:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T11:30:23.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: Manager daemon x is now available 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T11:30:23.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:30:23.858 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T11:30:24.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.178+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7fc5540985e0 con 0x7fc564075330 2024-04-26T11:30:24.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.179+0000 7fc55be00640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc5480aa9a0 0x7fc5480acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:24.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.179+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 --> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc55401ee50 con 0x7fc5480aa9a0 2024-04-26T11:30:24.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.183+0000 7fc562a00640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc5480aa9a0 0x7fc5480acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:24.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.190+0000 7fc562a00640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc5480aa9a0 0x7fc5480acd90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc56419c090 tx=0x7fc550005f90 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:24.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.292+0000 7fc55be00640 1 -- 172.21.15.33:0/4051555547 <== mgr.14931 v2:172.21.15.33:6800/516906866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc55401ee50 con 0x7fc5480aa9a0 2024-04-26T11:30:24.293 INFO:teuthology.orchestra.run.smithi033.stdout:Schedule set for path / 2024-04-26T11:30:24.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.294+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc5480aa9a0 msgr2=0x7fc5480acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.294+0000 7fc569200640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc5480aa9a0 0x7fc5480acd90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc56419c090 tx=0x7fc550005f90 comp rx=0 tx=0).stop 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc564075330 msgr2=0x7fc56419ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc564075330 0x7fc56419ab80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc55400e9f0 tx=0x7fc55400eeb0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 shutdown_connections 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc5480aa9a0 0x7fc5480acd90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc56410a680 0x7fc5641a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc564076910 0x7fc56419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 --2- 172.21.15.33:0/4051555547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc564075330 0x7fc56419ab80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 >> 172.21.15.33:0/4051555547 conn(0x7fc5640fdd60 msgr2=0x7fc564106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 shutdown_connections 2024-04-26T11:30:24.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.295+0000 7fc569200640 1 -- 172.21.15.33:0/4051555547 wait complete. 2024-04-26T11:30:24.312 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-26T11:30:24.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 -- 172.21.15.33:0/660636067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18103400 msgr2=0x7f5d181057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:24.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 --2- 172.21.15.33:0/660636067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18103400 0x7f5d181057f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009900 tx=0x7f5d0c02a3f0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 -- 172.21.15.33:0/660636067 shutdown_connections 2024-04-26T11:30:24.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 --2- 172.21.15.33:0/660636067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d18105d30 0x7f5d18108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 --2- 172.21.15.33:0/660636067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d18108660 0x7f5d1810aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 --2- 172.21.15.33:0/660636067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18103400 0x7f5d181057f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 -- 172.21.15.33:0/660636067 >> 172.21.15.33:0/660636067 conn(0x7f5d180fcdf0 msgr2=0x7f5d180ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:24.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 -- 172.21.15.33:0/660636067 shutdown_connections 2024-04-26T11:30:24.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.485+0000 7f5d1f600640 1 -- 172.21.15.33:0/660636067 wait complete. 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.486+0000 7f5d1f600640 1 Processor -- start 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.486+0000 7f5d1f600640 1 -- start start 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1f600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d18103400 0x7f5d1819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1f600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d18105d30 0x7f5d1819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1f600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18108660 0x7f5d181a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1f600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1810e4e0 con 0x7f5d18105d30 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1f600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1810e360 con 0x7f5d18108660 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1f600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1810e660 con 0x7f5d18103400 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d17e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d18105d30 0x7f5d1819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:24.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d18103400 0x7f5d1819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18108660 0x7f5d181a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d18103400 0x7f5d1819ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:33478/0 (socket says 172.21.15.33:33478) 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18108660 0x7f5d181a2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37256/0 (socket says 172.21.15.33:37256) 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1cc00640 1 -- 172.21.15.33:0/1033300303 learned_addr learned my addr 172.21.15.33:0/1033300303 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 -- 172.21.15.33:0/1033300303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d18103400 msgr2=0x7f5d1819ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d18103400 0x7f5d1819ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 -- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d18105d30 msgr2=0x7f5d1819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d18105d30 0x7f5d1819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 -- 172.21.15.33:0/1033300303 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d181a2970 con 0x7f5d18108660 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.487+0000 7f5d1d600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18108660 0x7f5d181a2300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009fd0 tx=0x7f5d1000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.489+0000 7f5d15600640 1 -- 172.21.15.33:0/1033300303 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1000ce20 con 0x7f5d18108660 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.489+0000 7f5d15600640 1 -- 172.21.15.33:0/1033300303 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d10005020 con 0x7f5d18108660 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.489+0000 7f5d15600640 1 -- 172.21.15.33:0/1033300303 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10010690 con 0x7f5d18108660 2024-04-26T11:30:24.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.489+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d181a2c60 con 0x7f5d18108660 2024-04-26T11:30:24.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.489+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d181a31b0 con 0x7f5d18108660 2024-04-26T11:30:24.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.489+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d1810a1b0 con 0x7f5d18108660 2024-04-26T11:30:24.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.491+0000 7f5d15600640 1 -- 172.21.15.33:0/1033300303 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f5d100054d0 con 0x7f5d18108660 2024-04-26T11:30:24.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.493+0000 7f5d15600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5d000a3ae0 0x7f5d000a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:24.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.493+0000 7f5d15600640 1 -- 172.21.15.33:0/1033300303 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5d100d0570 con 0x7f5d18108660 2024-04-26T11:30:24.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.494+0000 7f5d1cc00640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5d000a3ae0 0x7f5d000a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:24.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.494+0000 7f5d1cc00640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5d000a3ae0 0x7f5d000a5fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009230 tx=0x7f5d0c0058d0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:24.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.496+0000 7f5d15600640 1 -- 172.21.15.33:0/1033300303 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d1009d0f0 con 0x7f5d18108660 2024-04-26T11:30:24.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:24 smithi194 ceph-mon[33017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T11:30:24.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:24 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:30:24.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:24 smithi194 ceph-mon[33017]: mgrmap e23: x(active, since 1.02655s), standbys: z, y 2024-04-26T11:30:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:24 smithi033 ceph-mon[27845]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T11:30:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:24 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:30:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:24 smithi033 ceph-mon[27845]: mgrmap e23: x(active, since 1.02655s), standbys: z, y 2024-04-26T11:30:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:24 smithi096 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T11:30:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:24 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:30:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:24 smithi096 ceph-mon[32753]: mgrmap e23: x(active, since 1.02655s), standbys: z, y 2024-04-26T11:30:24.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.753+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 --> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f5d18062e10 con 0x7f5d000a3ae0 2024-04-26T11:30:24.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.824+0000 7f5d15600640 1 -- 172.21.15.33:0/1033300303 <== mgr.14931 v2:172.21.15.33:6800/516906866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f5d18062e10 con 0x7f5d000a3ae0 2024-04-26T11:30:24.825 INFO:teuthology.orchestra.run.smithi033.stdout:Retention added to path / 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5d000a3ae0 msgr2=0x7f5d000a5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5d000a3ae0 0x7f5d000a5fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009230 tx=0x7f5d0c0058d0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18108660 msgr2=0x7f5d181a2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18108660 0x7f5d181a2300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009fd0 tx=0x7f5d1000ef10 comp rx=0 tx=0).stop 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 shutdown_connections 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5d000a3ae0 0x7f5d000a5fa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d18108660 0x7f5d181a2300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d18105d30 0x7f5d1819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 --2- 172.21.15.33:0/1033300303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d18103400 0x7f5d1819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 >> 172.21.15.33:0/1033300303 conn(0x7f5d180fcdf0 msgr2=0x7f5d18102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 shutdown_connections 2024-04-26T11:30:24.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:24.827+0000 7f5d1f600640 1 -- 172.21.15.33:0/1033300303 wait complete. 2024-04-26T11:30:24.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T11:30:25.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 -- 172.21.15.33:0/3262839721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770102a80 msgr2=0x7f5770102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 --2- 172.21.15.33:0/3262839721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770102a80 0x7f5770102f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f575c009900 tx=0x7f575c02a410 comp rx=0 tx=0).stop 2024-04-26T11:30:25.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 -- 172.21.15.33:0/3262839721 shutdown_connections 2024-04-26T11:30:25.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 --2- 172.21.15.33:0/3262839721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5770103440 0x7f5770109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 --2- 172.21.15.33:0/3262839721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770102a80 0x7f5770102f00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 --2- 172.21.15.33:0/3262839721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5770101890 0x7f5770101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 -- 172.21.15.33:0/3262839721 >> 172.21.15.33:0/3262839721 conn(0x7f57700fce00 msgr2=0x7f57700ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:25.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.023+0000 7f5776200640 1 -- 172.21.15.33:0/3262839721 shutdown_connections 2024-04-26T11:30:25.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.024+0000 7f5776200640 1 -- 172.21.15.33:0/3262839721 wait complete. 2024-04-26T11:30:25.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 Processor -- start 2024-04-26T11:30:25.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 -- start start 2024-04-26T11:30:25.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5770101890 0x7f577019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5770102a80 0x7f577019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770103440 0x7f57701a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f577010e4e0 con 0x7f5770101890 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f577010e360 con 0x7f5770103440 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f5776200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f577010e660 con 0x7f5770102a80 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f576f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5770101890 0x7f577019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f576fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770103440 0x7f57701a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f576fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770103440 0x7f57701a2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37270/0 (socket says 172.21.15.33:37270) 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.025+0000 7f576fe00640 1 -- 172.21.15.33:0/3461990067 learned_addr learned my addr 172.21.15.33:0/3461990067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576ea00640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5770102a80 0x7f577019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576fe00640 1 -- 172.21.15.33:0/3461990067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5770102a80 msgr2=0x7f577019b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576fe00640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5770102a80 0x7f577019b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576fe00640 1 -- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5770101890 msgr2=0x7f577019ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576fe00640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5770101890 0x7f577019ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576fe00640 1 -- 172.21.15.33:0/3461990067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57701a2930 con 0x7f5770103440 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576f400640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5770101890 0x7f577019ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f576fe00640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770103440 0x7f57701a2290 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f576800eb10 tx=0x7f576800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.026+0000 7f5767e00640 1 -- 172.21.15.33:0/3461990067 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768004b20 con 0x7f5770103440 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.027+0000 7f5767e00640 1 -- 172.21.15.33:0/3461990067 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5768004cc0 con 0x7f5770103440 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.027+0000 7f5767e00640 1 -- 172.21.15.33:0/3461990067 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768010670 con 0x7f5770103440 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.027+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57701a2c20 con 0x7f5770103440 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.027+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57701a3120 con 0x7f5770103440 2024-04-26T11:30:25.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.028+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57701a32f0 con 0x7f5770103440 2024-04-26T11:30:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.031+0000 7f5767e00640 1 -- 172.21.15.33:0/3461990067 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f5768005020 con 0x7f5770103440 2024-04-26T11:30:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.033+0000 7f5767e00640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57540a3a60 0x7f57540a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.033+0000 7f5767e00640 1 -- 172.21.15.33:0/3461990067 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f57680d0700 con 0x7f5770103440 2024-04-26T11:30:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.033+0000 7f576f400640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57540a3a60 0x7f57540a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.034+0000 7f576f400640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57540a3a60 0x7f57540a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f577019c1f0 tx=0x7f5760005f90 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:25.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.036+0000 7f5767e00640 1 -- 172.21.15.33:0/3461990067 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f576809d280 con 0x7f5770103440 2024-04-26T11:30:25.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.284+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 --> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f577010bdb0 con 0x7f57540a3a60 2024-04-26T11:30:25.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.299+0000 7f5767e00640 1 -- 172.21.15.33:0/3461990067 <== mgr.14931 v2:172.21.15.33:6800/516906866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f577010bdb0 con 0x7f57540a3a60 2024-04-26T11:30:25.299 INFO:teuthology.orchestra.run.smithi033.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T11:30:24", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.301+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57540a3a60 msgr2=0x7f57540a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.301+0000 7f5776200640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57540a3a60 0x7f57540a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f577019c1f0 tx=0x7f5760005f90 comp rx=0 tx=0).stop 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770103440 msgr2=0x7f57701a2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770103440 0x7f57701a2290 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f576800eb10 tx=0x7f576800c2d0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 shutdown_connections 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57540a3a60 0x7f57540a5f20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5770102a80 0x7f577019b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5770103440 0x7f57701a2290 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 --2- 172.21.15.33:0/3461990067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5770101890 0x7f577019ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 >> 172.21.15.33:0/3461990067 conn(0x7f57700fce00 msgr2=0x7f577010a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 shutdown_connections 2024-04-26T11:30:25.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.302+0000 7f5776200640 1 -- 172.21.15.33:0/3461990067 wait complete. 2024-04-26T11:30:25.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 -- 172.21.15.33:0/3621361068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 msgr2=0x7fad3810d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 --2- 172.21.15.33:0/3621361068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 0x7fad3810d000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fad34009be0 tx=0x7fad3402a6c0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 -- 172.21.15.33:0/3621361068 shutdown_connections 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 --2- 172.21.15.33:0/3621361068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 0x7fad3810d000 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 --2- 172.21.15.33:0/3621361068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 0x7fad38106230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 --2- 172.21.15.33:0/3621361068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad38104bc0 0x7fad38104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 -- 172.21.15.33:0/3621361068 >> 172.21.15.33:0/3621361068 conn(0x7fad38100150 msgr2=0x7fad38102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:25.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 -- 172.21.15.33:0/3621361068 shutdown_connections 2024-04-26T11:30:25.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.514+0000 7fad40800640 1 -- 172.21.15.33:0/3621361068 wait complete. 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.515+0000 7fad40800640 1 Processor -- start 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.515+0000 7fad40800640 1 -- start start 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad40800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad38104bc0 0x7fad3819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad40800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 0x7fad3819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad40800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 0x7fad381a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad40800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad381106f0 con 0x7fad38104bc0 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad40800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fad38110570 con 0x7fad38105db0 2024-04-26T11:30:25.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad40800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fad38110870 con 0x7fad38106770 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 0x7fad3819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3de00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad38104bc0 0x7fad3819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 0x7fad3819b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37302/0 (socket says 172.21.15.33:37302) 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 -- 172.21.15.33:0/1858491918 learned_addr learned my addr 172.21.15.33:0/1858491918 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3e800640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 0x7fad381a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 -- 172.21.15.33:0/1858491918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 msgr2=0x7fad381a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 0x7fad381a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 -- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad38104bc0 msgr2=0x7fad3819ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad38104bc0 0x7fad3819ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.516+0000 7fad3d400640 1 -- 172.21.15.33:0/1858491918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad381a2960 con 0x7fad38105db0 2024-04-26T11:30:25.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.517+0000 7fad3d400640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 0x7fad3819b280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad2400e9f0 tx=0x7fad2400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:25.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.518+0000 7fad32a00640 1 -- 172.21.15.33:0/1858491918 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2400ccc0 con 0x7fad38105db0 2024-04-26T11:30:25.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.518+0000 7fad32a00640 1 -- 172.21.15.33:0/1858491918 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2400ce60 con 0x7fad38105db0 2024-04-26T11:30:25.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.518+0000 7fad32a00640 1 -- 172.21.15.33:0/1858491918 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24010430 con 0x7fad38105db0 2024-04-26T11:30:25.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.518+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad381a2c50 con 0x7fad38105db0 2024-04-26T11:30:25.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.518+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad381a31a0 con 0x7fad38105db0 2024-04-26T11:30:25.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.520+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad3810be80 con 0x7fad38105db0 2024-04-26T11:30:25.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.522+0000 7fad32a00640 1 -- 172.21.15.33:0/1858491918 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7fad240046c0 con 0x7fad38105db0 2024-04-26T11:30:25.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.524+0000 7fad32a00640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fad200a3f20 0x7fad200a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:25.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.524+0000 7fad32a00640 1 -- 172.21.15.33:0/1858491918 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fad240d0af0 con 0x7fad38105db0 2024-04-26T11:30:25.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.524+0000 7fad3de00640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fad200a3f20 0x7fad200a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:25.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.525+0000 7fad3de00640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fad200a3f20 0x7fad200a63e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad38105c10 tx=0x7fad2c009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:25.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.528+0000 7fad32a00640 1 -- 172.21.15.33:0/1858491918 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad2409d140 con 0x7fad38105db0 2024-04-26T11:30:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:25 smithi033 ceph-mon[27845]: [26/Apr/2024:11:30:24] ENGINE Bus STARTING 2024-04-26T11:30:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:25 smithi033 ceph-mon[27845]: [26/Apr/2024:11:30:24] ENGINE Serving on http://172.21.15.33:8765 2024-04-26T11:30:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:25 smithi033 ceph-mon[27845]: from='client.25112 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:25 smithi033 ceph-mon[27845]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:25 smithi033 ceph-mon[27845]: [26/Apr/2024:11:30:24] ENGINE Serving on https://172.21.15.33:7150 2024-04-26T11:30:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:25 smithi033 ceph-mon[27845]: [26/Apr/2024:11:30:24] ENGINE Bus STARTED 2024-04-26T11:30:25.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.791+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 --> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fad38062d70 con 0x7fad200a3f20 2024-04-26T11:30:25.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.802+0000 7fad32a00640 1 -- 172.21.15.33:0/1858491918 <== mgr.14931 v2:172.21.15.33:6800/516906866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fad38062d70 con 0x7fad200a3f20 2024-04-26T11:30:25.803 INFO:teuthology.orchestra.run.smithi033.stdout:/ 1m 6m3h 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fad200a3f20 msgr2=0x7fad200a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fad200a3f20 0x7fad200a63e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad38105c10 tx=0x7fad2c009260 comp rx=0 tx=0).stop 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 msgr2=0x7fad3819b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 0x7fad3819b280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad2400e9f0 tx=0x7fad2400edf0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 shutdown_connections 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fad200a3f20 0x7fad200a63e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad38106770 0x7fad381a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad38105db0 0x7fad3819b280 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 --2- 172.21.15.33:0/1858491918 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad38104bc0 0x7fad3819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:25.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 >> 172.21.15.33:0/1858491918 conn(0x7fad38100150 msgr2=0x7fad38109050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:25.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 shutdown_connections 2024-04-26T11:30:25.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:25.805+0000 7fad40800640 1 -- 172.21.15.33:0/1858491918 wait complete. 2024-04-26T11:30:25.825 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-26T11:30:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:25 smithi194 ceph-mon[33017]: [26/Apr/2024:11:30:24] ENGINE Bus STARTING 2024-04-26T11:30:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:25 smithi194 ceph-mon[33017]: [26/Apr/2024:11:30:24] ENGINE Serving on http://172.21.15.33:8765 2024-04-26T11:30:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:25 smithi194 ceph-mon[33017]: from='client.25112 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:25 smithi194 ceph-mon[33017]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:25 smithi194 ceph-mon[33017]: [26/Apr/2024:11:30:24] ENGINE Serving on https://172.21.15.33:7150 2024-04-26T11:30:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:25 smithi194 ceph-mon[33017]: [26/Apr/2024:11:30:24] ENGINE Bus STARTED 2024-04-26T11:30:25.895 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-26T11:30:25.906 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-26T11:30:25.951 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:30:25.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-26T11:30:25.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:25 smithi096 ceph-mon[32753]: [26/Apr/2024:11:30:24] ENGINE Bus STARTING 2024-04-26T11:30:25.953 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:25 smithi096 ceph-mon[32753]: [26/Apr/2024:11:30:24] ENGINE Serving on http://172.21.15.33:8765 2024-04-26T11:30:25.953 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:25 smithi096 ceph-mon[32753]: from='client.25112 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:25.953 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:25 smithi096 ceph-mon[32753]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:25.953 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:25 smithi096 ceph-mon[32753]: [26/Apr/2024:11:30:24] ENGINE Serving on https://172.21.15.33:7150 2024-04-26T11:30:25.953 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:25 smithi096 ceph-mon[32753]: [26/Apr/2024:11:30:24] ENGINE Bus STARTED 2024-04-26T11:30:25.957 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-26T11:30:26.004 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T11:30:26.005 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T11:30:26.494 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:26.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:26.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:26.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:26.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-04-26T11:30:26.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.905 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:27.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:28.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.743+0000 7fdef8000700 1 -- 172.21.15.33:0/1075870658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 msgr2=0x7fdef0106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:28.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.743+0000 7fdef8000700 1 --2- 172.21.15.33:0/1075870658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef0106f80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0098f0 tx=0x7fdedc009c00 comp rx=0 tx=0).stop 2024-04-26T11:30:28.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.744+0000 7fdef8000700 1 -- 172.21.15.33:0/1075870658 shutdown_connections 2024-04-26T11:30:28.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.744+0000 7fdef8000700 1 --2- 172.21.15.33:0/1075870658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdef0107660 0x7fdef010f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:28.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.744+0000 7fdef8000700 1 --2- 172.21.15.33:0/1075870658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef0108280 0x7fdef0108660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:28.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.744+0000 7fdef8000700 1 --2- 172.21.15.33:0/1075870658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef0106f80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:28.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.744+0000 7fdef8000700 1 -- 172.21.15.33:0/1075870658 >> 172.21.15.33:0/1075870658 conn(0x7fdef0078520 msgr2=0x7fdef00ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:28.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:28.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.744+0000 7fdef8000700 1 -- 172.21.15.33:0/1075870658 shutdown_connections 2024-04-26T11:30:28.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.744+0000 7fdef8000700 1 -- 172.21.15.33:0/1075870658 wait complete. 2024-04-26T11:30:28.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 Processor -- start 2024-04-26T11:30:28.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 -- start start 2024-04-26T11:30:28.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdef0107660 0x7fdef0074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:28.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef0108280 0x7fdef006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:28.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef006e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:28.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0112500 con 0x7fdef0108280 2024-04-26T11:30:28.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0112380 con 0x7fdef0107660 2024-04-26T11:30:28.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.746+0000 7fdef8000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0112680 con 0x7fdef0108c30 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.747+0000 7fdef4c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef0108280 0x7fdef006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.747+0000 7fdef6000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.747+0000 7fdef6000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef006e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:33554/0 (socket says 172.21.15.33:33554) 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.747+0000 7fdef6000700 1 -- 172.21.15.33:0/1753917321 learned_addr learned my addr 172.21.15.33:0/1753917321 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.748+0000 7fdef6000700 1 -- 172.21.15.33:0/1753917321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdef0107660 msgr2=0x7fdef0074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.748+0000 7fdef6000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdef0107660 0x7fdef0074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.748+0000 7fdef6000700 1 -- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef0108280 msgr2=0x7fdef006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.748+0000 7fdef6000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef0108280 0x7fdef006dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.748+0000 7fdef6000700 1 -- 172.21.15.33:0/1753917321 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef006e910 con 0x7fdef0108c30 2024-04-26T11:30:28.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.748+0000 7fdef6000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef006e270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e3c0 tx=0x7fdee800e6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:28.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.748+0000 7fdeee000700 1 -- 172.21.15.33:0/1753917321 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80092d0 con 0x7fdef0108c30 2024-04-26T11:30:28.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.749+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef00727d0 con 0x7fdef0108c30 2024-04-26T11:30:28.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.749+0000 7fdeee000700 1 -- 172.21.15.33:0/1753917321 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee800ee70 con 0x7fdef0108c30 2024-04-26T11:30:28.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.749+0000 7fdeee000700 1 -- 172.21.15.33:0/1753917321 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee8004030 con 0x7fdef0108c30 2024-04-26T11:30:28.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.749+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef0072ce0 con 0x7fdef0108c30 2024-04-26T11:30:28.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.750+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef0101920 con 0x7fdef0108c30 2024-04-26T11:30:28.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.752+0000 7fdeee000700 1 -- 172.21.15.33:0/1753917321 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fdee8012050 con 0x7fdef0108c30 2024-04-26T11:30:28.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.754+0000 7fdeee000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fded80a3ec0 0x7fded80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:28.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.754+0000 7fdeee000700 1 -- 172.21.15.33:0/1753917321 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdee80d06b0 con 0x7fdef0108c30 2024-04-26T11:30:28.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.754+0000 7fdef5600700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fded80a3ec0 0x7fded80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:28.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.755+0000 7fdef5600700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fded80a3ec0 0x7fded80a6380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdee4005e90 tx=0x7fdee4005d00 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:28.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:28.759+0000 7fdeee000700 1 -- 172.21.15.33:0/1753917321 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdee8022020 con 0x7fdef0108c30 2024-04-26T11:30:29.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.015+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 --> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdef0072f10 con 0x7fded80a3ec0 2024-04-26T11:30:29.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.029+0000 7fdeee000700 1 -- 172.21.15.33:0/1753917321 <== mgr.14931 v2:172.21.15.33:6800/516906866 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258058 (secure 0 0 0) 0x7fdef0072f10 con 0x7fded80a3ec0 2024-04-26T11:30:29.036 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:29.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fded80a3ec0 msgr2=0x7fded80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:29.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fded80a3ec0 0x7fded80a6380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdee4005e90 tx=0x7fdee4005d00 comp rx=0 tx=0).stop 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 msgr2=0x7fdef006e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef006e270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e3c0 tx=0x7fdee800e6d0 comp rx=0 tx=0).stop 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 shutdown_connections 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fded80a3ec0 0x7fded80a6380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdef0108c30 0x7fdef006e270 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdef0107660 0x7fdef0074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 --2- 172.21.15.33:0/1753917321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdef0108280 0x7fdef006dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 >> 172.21.15.33:0/1753917321 conn(0x7fdef0078520 msgr2=0x7fdef01052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 shutdown_connections 2024-04-26T11:30:29.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:29.036+0000 7fdef8000700 1 -- 172.21.15.33:0/1753917321 wait complete. 2024-04-26T11:30:29.040 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:30:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:30:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:30:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:30:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T11:30:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-04-26T11:30:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:29.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:30:29.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:30:30.007 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-26T11:30:27.189510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1192706,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1526,"num_read_kb":922,"num_write":482,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357320,"kb_used_data":12288,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584504,"statfs":{"total":1151940427776,"available":1151574532096,"internally_reserved":0,"allocated":12582912,"data_stored":9428875,"data_compressed":290166,"data_compressed_allocated":389120,"data_compressed_original":3629056,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.988118"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308540+0000","last_change":"2024-04-26T11:26:29.018818+0000","last_active":"2024-04-26T11:30:23.308540+0000","last_peered":"2024-04-26T11:30:23.308540+0000","last_clean":"2024-04-26T11:30:23.308540+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:30:23.308540+0000","last_undegraded":"2024-04-26T11:30:23.308540+0000","last_fullsized":"2024-04-26T11:30:23.308540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:05.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518402+0000","last_change":"2024-04-26T11:26:29.021338+0000","last_active":"2024-04-26T11:30:23.518402+0000","last_peered":"2024-04-26T11:30:23.518402+0000","last_clean":"2024-04-26T11:30:23.518402+0000","last_became_active":"2024-04-26T11:26:29.021019+0000","last_became_peered":"2024-04-26T11:26:29.021019+0000","last_unstale":"2024-04-26T11:30:23.518402+0000","last_undegraded":"2024-04-26T11:30:23.518402+0000","last_fullsized":"2024-04-26T11:30:23.518402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:52:44.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518696+0000","last_change":"2024-04-26T11:26:29.014166+0000","last_active":"2024-04-26T11:30:23.518696+0000","last_peered":"2024-04-26T11:30:23.518696+0000","last_clean":"2024-04-26T11:30:23.518696+0000","last_became_active":"2024-04-26T11:26:29.013812+0000","last_became_peered":"2024-04-26T11:26:29.013812+0000","last_unstale":"2024-04-26T11:30:23.518696+0000","last_undegraded":"2024-04-26T11:30:23.518696+0000","last_fullsized":"2024-04-26T11:30:23.518696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:08.683633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513477+0000","last_change":"2024-04-26T11:26:28.999249+0000","last_active":"2024-04-26T11:30:23.513477+0000","last_peered":"2024-04-26T11:30:23.513477+0000","last_clean":"2024-04-26T11:30:23.513477+0000","last_became_active":"2024-04-26T11:26:28.998927+0000","last_became_peered":"2024-04-26T11:26:28.998927+0000","last_unstale":"2024-04-26T11:30:23.513477+0000","last_undegraded":"2024-04-26T11:30:23.513477+0000","last_fullsized":"2024-04-26T11:30:23.513477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:56.674537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698533+0000","last_change":"2024-04-26T11:26:29.017202+0000","last_active":"2024-04-26T11:30:23.698533+0000","last_peered":"2024-04-26T11:30:23.698533+0000","last_clean":"2024-04-26T11:30:23.698533+0000","last_became_active":"2024-04-26T11:26:29.016849+0000","last_became_peered":"2024-04-26T11:26:29.016849+0000","last_unstale":"2024-04-26T11:30:23.698533+0000","last_undegraded":"2024-04-26T11:30:23.698533+0000","last_fullsized":"2024-04-26T11:30:23.698533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:58:22.164118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165326+0000","last_change":"2024-04-26T11:27:19.278294+0000","last_active":"2024-04-26T11:30:23.165326+0000","last_peered":"2024-04-26T11:30:23.165326+0000","last_clean":"2024-04-26T11:30:23.165326+0000","last_became_active":"2024-04-26T11:27:19.277853+0000","last_became_peered":"2024-04-26T11:27:19.277853+0000","last_unstale":"2024-04-26T11:30:23.165326+0000","last_undegraded":"2024-04-26T11:30:23.165326+0000","last_fullsized":"2024-04-26T11:30:23.165326+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:12:18.733166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515577+0000","last_change":"2024-04-26T11:26:28.991406+0000","last_active":"2024-04-26T11:30:23.515577+0000","last_peered":"2024-04-26T11:30:23.515577+0000","last_clean":"2024-04-26T11:30:23.515577+0000","last_became_active":"2024-04-26T11:26:28.989303+0000","last_became_peered":"2024-04-26T11:26:28.989303+0000","last_unstale":"2024-04-26T11:30:23.515577+0000","last_undegraded":"2024-04-26T11:30:23.515577+0000","last_fullsized":"2024-04-26T11:30:23.515577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:42:59.449078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163073+0000","last_change":"2024-04-26T11:26:29.020922+0000","last_active":"2024-04-26T11:30:23.163073+0000","last_peered":"2024-04-26T11:30:23.163073+0000","last_clean":"2024-04-26T11:30:23.163073+0000","last_became_active":"2024-04-26T11:26:29.020576+0000","last_became_peered":"2024-04-26T11:26:29.020576+0000","last_unstale":"2024-04-26T11:30:23.163073+0000","last_undegraded":"2024-04-26T11:30:23.163073+0000","last_fullsized":"2024-04-26T11:30:23.163073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:48.770319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520362+0000","last_change":"2024-04-26T11:26:29.004171+0000","last_active":"2024-04-26T11:30:23.520362+0000","last_peered":"2024-04-26T11:30:23.520362+0000","last_clean":"2024-04-26T11:30:23.520362+0000","last_became_active":"2024-04-26T11:26:29.003816+0000","last_became_peered":"2024-04-26T11:26:29.003816+0000","last_unstale":"2024-04-26T11:30:23.520362+0000","last_undegraded":"2024-04-26T11:30:23.520362+0000","last_fullsized":"2024-04-26T11:30:23.520362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:45:14.807634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165819+0000","last_change":"2024-04-26T11:26:28.999453+0000","last_active":"2024-04-26T11:30:23.165819+0000","last_peered":"2024-04-26T11:30:23.165819+0000","last_clean":"2024-04-26T11:30:23.165819+0000","last_became_active":"2024-04-26T11:26:28.999104+0000","last_became_peered":"2024-04-26T11:26:28.999104+0000","last_unstale":"2024-04-26T11:30:23.165819+0000","last_undegraded":"2024-04-26T11:30:23.165819+0000","last_fullsized":"2024-04-26T11:30:23.165819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:55.408949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515355+0000","last_change":"2024-04-26T11:26:28.999218+0000","last_active":"2024-04-26T11:30:23.515355+0000","last_peered":"2024-04-26T11:30:23.515355+0000","last_clean":"2024-04-26T11:30:23.515355+0000","last_became_active":"2024-04-26T11:26:28.998584+0000","last_became_peered":"2024-04-26T11:26:28.998584+0000","last_unstale":"2024-04-26T11:30:23.515355+0000","last_undegraded":"2024-04-26T11:30:23.515355+0000","last_fullsized":"2024-04-26T11:30:23.515355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:18.040520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513411+0000","last_change":"2024-04-26T11:26:28.997121+0000","last_active":"2024-04-26T11:30:23.513411+0000","last_peered":"2024-04-26T11:30:23.513411+0000","last_clean":"2024-04-26T11:30:23.513411+0000","last_became_active":"2024-04-26T11:26:28.996556+0000","last_became_peered":"2024-04-26T11:26:28.996556+0000","last_unstale":"2024-04-26T11:30:23.513411+0000","last_undegraded":"2024-04-26T11:30:23.513411+0000","last_fullsized":"2024-04-26T11:30:23.513411+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:18:55.251371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514338+0000","last_change":"2024-04-26T11:26:28.999525+0000","last_active":"2024-04-26T11:30:23.514338+0000","last_peered":"2024-04-26T11:30:23.514338+0000","last_clean":"2024-04-26T11:30:23.514338+0000","last_became_active":"2024-04-26T11:26:28.998578+0000","last_became_peered":"2024-04-26T11:26:28.998578+0000","last_unstale":"2024-04-26T11:30:23.514338+0000","last_undegraded":"2024-04-26T11:30:23.514338+0000","last_fullsized":"2024-04-26T11:30:23.514338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:32:35.523804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518043+0000","last_change":"2024-04-26T11:26:29.008427+0000","last_active":"2024-04-26T11:30:23.518043+0000","last_peered":"2024-04-26T11:30:23.518043+0000","last_clean":"2024-04-26T11:30:23.518043+0000","last_became_active":"2024-04-26T11:26:29.008073+0000","last_became_peered":"2024-04-26T11:26:29.008073+0000","last_unstale":"2024-04-26T11:30:23.518043+0000","last_undegraded":"2024-04-26T11:30:23.518043+0000","last_fullsized":"2024-04-26T11:30:23.518043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:03.185941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518029+0000","last_change":"2024-04-26T11:26:29.015136+0000","last_active":"2024-04-26T11:30:23.518029+0000","last_peered":"2024-04-26T11:30:23.518029+0000","last_clean":"2024-04-26T11:30:23.518029+0000","last_became_active":"2024-04-26T11:26:29.014800+0000","last_became_peered":"2024-04-26T11:26:29.014800+0000","last_unstale":"2024-04-26T11:30:23.518029+0000","last_undegraded":"2024-04-26T11:30:23.518029+0000","last_fullsized":"2024-04-26T11:30:23.518029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:28:35.871046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519847+0000","last_change":"2024-04-26T11:26:29.024795+0000","last_active":"2024-04-26T11:30:23.519847+0000","last_peered":"2024-04-26T11:30:23.519847+0000","last_clean":"2024-04-26T11:30:23.519847+0000","last_became_active":"2024-04-26T11:26:29.024430+0000","last_became_peered":"2024-04-26T11:26:29.024430+0000","last_unstale":"2024-04-26T11:30:23.519847+0000","last_undegraded":"2024-04-26T11:30:23.519847+0000","last_fullsized":"2024-04-26T11:30:23.519847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:05:20.172869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308685+0000","last_change":"2024-04-26T11:26:29.018822+0000","last_active":"2024-04-26T11:30:23.308685+0000","last_peered":"2024-04-26T11:30:23.308685+0000","last_clean":"2024-04-26T11:30:23.308685+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:30:23.308685+0000","last_undegraded":"2024-04-26T11:30:23.308685+0000","last_fullsized":"2024-04-26T11:30:23.308685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:04.475400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306306+0000","last_change":"2024-04-26T11:26:29.001984+0000","last_active":"2024-04-26T11:30:23.306306+0000","last_peered":"2024-04-26T11:30:23.306306+0000","last_clean":"2024-04-26T11:30:23.306306+0000","last_became_active":"2024-04-26T11:26:28.999492+0000","last_became_peered":"2024-04-26T11:26:28.999492+0000","last_unstale":"2024-04-26T11:30:23.306306+0000","last_undegraded":"2024-04-26T11:30:23.306306+0000","last_fullsized":"2024-04-26T11:30:23.306306+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:21.398823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165815+0000","last_change":"2024-04-26T11:26:29.019677+0000","last_active":"2024-04-26T11:30:23.165815+0000","last_peered":"2024-04-26T11:30:23.165815+0000","last_clean":"2024-04-26T11:30:23.165815+0000","last_became_active":"2024-04-26T11:26:29.019315+0000","last_became_peered":"2024-04-26T11:26:29.019315+0000","last_unstale":"2024-04-26T11:30:23.165815+0000","last_undegraded":"2024-04-26T11:30:23.165815+0000","last_fullsized":"2024-04-26T11:30:23.165815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:20.239140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306242+0000","last_change":"2024-04-26T11:26:24.964887+0000","last_active":"2024-04-26T11:30:23.306242+0000","last_peered":"2024-04-26T11:30:23.306242+0000","last_clean":"2024-04-26T11:30:23.306242+0000","last_became_active":"2024-04-26T11:26:24.964380+0000","last_became_peered":"2024-04-26T11:26:24.964380+0000","last_unstale":"2024-04-26T11:30:23.306242+0000","last_undegraded":"2024-04-26T11:30:23.306242+0000","last_fullsized":"2024-04-26T11:30:23.306242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:59:41.347542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306031+0000","last_change":"2024-04-26T11:27:19.244889+0000","last_active":"2024-04-26T11:30:23.306031+0000","last_peered":"2024-04-26T11:30:23.306031+0000","last_clean":"2024-04-26T11:30:23.306031+0000","last_became_active":"2024-04-26T11:27:19.244428+0000","last_became_peered":"2024-04-26T11:27:19.244428+0000","last_unstale":"2024-04-26T11:30:23.306031+0000","last_undegraded":"2024-04-26T11:30:23.306031+0000","last_fullsized":"2024-04-26T11:30:23.306031+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:09:39.366242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166206+0000","last_change":"2024-04-26T11:26:25.082741+0000","last_active":"2024-04-26T11:30:23.166206+0000","last_peered":"2024-04-26T11:30:23.166206+0000","last_clean":"2024-04-26T11:30:23.166206+0000","last_became_active":"2024-04-26T11:26:25.082361+0000","last_became_peered":"2024-04-26T11:26:25.082361+0000","last_unstale":"2024-04-26T11:30:23.166206+0000","last_undegraded":"2024-04-26T11:30:23.166206+0000","last_fullsized":"2024-04-26T11:30:23.166206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:23:17.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519051+0000","last_change":"2024-04-26T11:26:29.014817+0000","last_active":"2024-04-26T11:30:23.519051+0000","last_peered":"2024-04-26T11:30:23.519051+0000","last_clean":"2024-04-26T11:30:23.519051+0000","last_became_active":"2024-04-26T11:26:29.014419+0000","last_became_peered":"2024-04-26T11:26:29.014419+0000","last_unstale":"2024-04-26T11:30:23.519051+0000","last_undegraded":"2024-04-26T11:30:23.519051+0000","last_fullsized":"2024-04-26T11:30:23.519051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:33:28.355075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698995+0000","last_change":"2024-04-26T11:26:24.960224+0000","last_active":"2024-04-26T11:30:23.698995+0000","last_peered":"2024-04-26T11:30:23.698995+0000","last_clean":"2024-04-26T11:30:23.698995+0000","last_became_active":"2024-04-26T11:26:24.959848+0000","last_became_peered":"2024-04-26T11:26:24.959848+0000","last_unstale":"2024-04-26T11:30:23.698995+0000","last_undegraded":"2024-04-26T11:30:23.698995+0000","last_fullsized":"2024-04-26T11:30:23.698995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:49:10.932187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520146+0000","last_change":"2024-04-26T11:26:29.001946+0000","last_active":"2024-04-26T11:30:23.520146+0000","last_peered":"2024-04-26T11:30:23.520146+0000","last_clean":"2024-04-26T11:30:23.520146+0000","last_became_active":"2024-04-26T11:26:29.001579+0000","last_became_peered":"2024-04-26T11:26:29.001579+0000","last_unstale":"2024-04-26T11:30:23.520146+0000","last_undegraded":"2024-04-26T11:30:23.520146+0000","last_fullsized":"2024-04-26T11:30:23.520146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:14:43.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698818+0000","last_change":"2024-04-26T11:26:24.956148+0000","last_active":"2024-04-26T11:30:23.698818+0000","last_peered":"2024-04-26T11:30:23.698818+0000","last_clean":"2024-04-26T11:30:23.698818+0000","last_became_active":"2024-04-26T11:26:24.955736+0000","last_became_peered":"2024-04-26T11:26:24.955736+0000","last_unstale":"2024-04-26T11:30:23.698818+0000","last_undegraded":"2024-04-26T11:30:23.698818+0000","last_fullsized":"2024-04-26T11:30:23.698818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:43.705470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317648+0000","last_change":"2024-04-26T11:27:19.415134+0000","last_active":"2024-04-26T11:30:23.317648+0000","last_peered":"2024-04-26T11:30:23.317648+0000","last_clean":"2024-04-26T11:30:23.317648+0000","last_became_active":"2024-04-26T11:27:19.414703+0000","last_became_peered":"2024-04-26T11:27:19.414703+0000","last_unstale":"2024-04-26T11:30:23.317648+0000","last_undegraded":"2024-04-26T11:30:23.317648+0000","last_fullsized":"2024-04-26T11:30:23.317648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:41:11.268296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":248,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.439833+0000","last_change":"2024-04-26T11:26:24.958350+0000","last_active":"2024-04-26T11:30:23.439833+0000","last_peered":"2024-04-26T11:30:23.439833+0000","last_clean":"2024-04-26T11:30:23.439833+0000","last_became_active":"2024-04-26T11:26:24.957382+0000","last_became_peered":"2024-04-26T11:26:24.957382+0000","last_unstale":"2024-04-26T11:30:23.439833+0000","last_undegraded":"2024-04-26T11:30:23.439833+0000","last_fullsized":"2024-04-26T11:30:23.439833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:03.854227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":213,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698584+0000","last_change":"2024-04-26T11:26:28.997568+0000","last_active":"2024-04-26T11:30:23.698584+0000","last_peered":"2024-04-26T11:30:23.698584+0000","last_clean":"2024-04-26T11:30:23.698584+0000","last_became_active":"2024-04-26T11:26:28.997161+0000","last_became_peered":"2024-04-26T11:26:28.997161+0000","last_unstale":"2024-04-26T11:30:23.698584+0000","last_undegraded":"2024-04-26T11:30:23.698584+0000","last_fullsized":"2024-04-26T11:30:23.698584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:04.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165161+0000","last_change":"2024-04-26T11:26:24.958093+0000","last_active":"2024-04-26T11:30:23.165161+0000","last_peered":"2024-04-26T11:30:23.165161+0000","last_clean":"2024-04-26T11:30:23.165161+0000","last_became_active":"2024-04-26T11:26:24.957730+0000","last_became_peered":"2024-04-26T11:26:24.957730+0000","last_unstale":"2024-04-26T11:30:23.165161+0000","last_undegraded":"2024-04-26T11:30:23.165161+0000","last_fullsized":"2024-04-26T11:30:23.165161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:09:29.827063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306012+0000","last_change":"2024-04-26T11:27:19.248406+0000","last_active":"2024-04-26T11:30:23.306012+0000","last_peered":"2024-04-26T11:30:23.306012+0000","last_clean":"2024-04-26T11:30:23.306012+0000","last_became_active":"2024-04-26T11:27:19.247980+0000","last_became_peered":"2024-04-26T11:27:19.247980+0000","last_unstale":"2024-04-26T11:30:23.306012+0000","last_undegraded":"2024-04-26T11:30:23.306012+0000","last_fullsized":"2024-04-26T11:30:23.306012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:51:34.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514048+0000","last_change":"2024-04-26T11:26:24.963463+0000","last_active":"2024-04-26T11:30:23.514048+0000","last_peered":"2024-04-26T11:30:23.514048+0000","last_clean":"2024-04-26T11:30:23.514048+0000","last_became_active":"2024-04-26T11:26:24.963030+0000","last_became_peered":"2024-04-26T11:26:24.963030+0000","last_unstale":"2024-04-26T11:30:23.514048+0000","last_undegraded":"2024-04-26T11:30:23.514048+0000","last_fullsized":"2024-04-26T11:30:23.514048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:04:17.844500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163342+0000","last_change":"2024-04-26T11:26:29.013899+0000","last_active":"2024-04-26T11:30:23.163342+0000","last_peered":"2024-04-26T11:30:23.163342+0000","last_clean":"2024-04-26T11:30:23.163342+0000","last_became_active":"2024-04-26T11:26:29.012674+0000","last_became_peered":"2024-04-26T11:26:29.012674+0000","last_unstale":"2024-04-26T11:30:23.163342+0000","last_undegraded":"2024-04-26T11:30:23.163342+0000","last_fullsized":"2024-04-26T11:30:23.163342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:33:46.259841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317199+0000","last_change":"2024-04-26T11:26:24.957803+0000","last_active":"2024-04-26T11:30:23.317199+0000","last_peered":"2024-04-26T11:30:23.317199+0000","last_clean":"2024-04-26T11:30:23.317199+0000","last_became_active":"2024-04-26T11:26:24.957496+0000","last_became_peered":"2024-04-26T11:26:24.957496+0000","last_unstale":"2024-04-26T11:30:23.317199+0000","last_undegraded":"2024-04-26T11:30:23.317199+0000","last_fullsized":"2024-04-26T11:30:23.317199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:15.497106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306343+0000","last_change":"2024-04-26T11:26:29.021648+0000","last_active":"2024-04-26T11:30:23.306343+0000","last_peered":"2024-04-26T11:30:23.306343+0000","last_clean":"2024-04-26T11:30:23.306343+0000","last_became_active":"2024-04-26T11:26:29.020689+0000","last_became_peered":"2024-04-26T11:26:29.020689+0000","last_unstale":"2024-04-26T11:30:23.306343+0000","last_undegraded":"2024-04-26T11:30:23.306343+0000","last_fullsized":"2024-04-26T11:30:23.306343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:15:42.321851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":244,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.439488+0000","last_change":"2024-04-26T11:26:24.966184+0000","last_active":"2024-04-26T11:30:23.439488+0000","last_peered":"2024-04-26T11:30:23.439488+0000","last_clean":"2024-04-26T11:30:23.439488+0000","last_became_active":"2024-04-26T11:26:24.965850+0000","last_became_peered":"2024-04-26T11:26:24.965850+0000","last_unstale":"2024-04-26T11:30:23.439488+0000","last_undegraded":"2024-04-26T11:30:23.439488+0000","last_fullsized":"2024-04-26T11:30:23.439488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:49.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":211,"num_read_kb":211,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520125+0000","last_change":"2024-04-26T11:26:29.003570+0000","last_active":"2024-04-26T11:30:23.520125+0000","last_peered":"2024-04-26T11:30:23.520125+0000","last_clean":"2024-04-26T11:30:23.520125+0000","last_became_active":"2024-04-26T11:26:29.003039+0000","last_became_peered":"2024-04-26T11:26:29.003039+0000","last_unstale":"2024-04-26T11:30:23.520125+0000","last_undegraded":"2024-04-26T11:30:23.520125+0000","last_fullsized":"2024-04-26T11:30:23.520125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:29:16.166432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515513+0000","last_change":"2024-04-26T11:26:24.954968+0000","last_active":"2024-04-26T11:30:23.515513+0000","last_peered":"2024-04-26T11:30:23.515513+0000","last_clean":"2024-04-26T11:30:23.515513+0000","last_became_active":"2024-04-26T11:26:24.954595+0000","last_became_peered":"2024-04-26T11:26:24.954595+0000","last_unstale":"2024-04-26T11:30:23.515513+0000","last_undegraded":"2024-04-26T11:30:23.515513+0000","last_fullsized":"2024-04-26T11:30:23.515513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:52:21.988942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518513+0000","last_change":"2024-04-26T11:26:29.015601+0000","last_active":"2024-04-26T11:30:23.518513+0000","last_peered":"2024-04-26T11:30:23.518513+0000","last_clean":"2024-04-26T11:30:23.518513+0000","last_became_active":"2024-04-26T11:26:29.015199+0000","last_became_peered":"2024-04-26T11:26:29.015199+0000","last_unstale":"2024-04-26T11:30:23.518513+0000","last_undegraded":"2024-04-26T11:30:23.518513+0000","last_fullsized":"2024-04-26T11:30:23.518513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:15:04.573214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317777+0000","last_change":"2024-04-26T11:26:25.085542+0000","last_active":"2024-04-26T11:30:23.317777+0000","last_peered":"2024-04-26T11:30:23.317777+0000","last_clean":"2024-04-26T11:30:23.317777+0000","last_became_active":"2024-04-26T11:26:25.085230+0000","last_became_peered":"2024-04-26T11:26:25.085230+0000","last_unstale":"2024-04-26T11:30:23.317777+0000","last_undegraded":"2024-04-26T11:30:23.317777+0000","last_fullsized":"2024-04-26T11:30:23.317777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:51.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520229+0000","last_change":"2024-04-26T11:26:29.003954+0000","last_active":"2024-04-26T11:30:23.520229+0000","last_peered":"2024-04-26T11:30:23.520229+0000","last_clean":"2024-04-26T11:30:23.520229+0000","last_became_active":"2024-04-26T11:26:29.003576+0000","last_became_peered":"2024-04-26T11:26:29.003576+0000","last_unstale":"2024-04-26T11:30:23.520229+0000","last_undegraded":"2024-04-26T11:30:23.520229+0000","last_fullsized":"2024-04-26T11:30:23.520229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:24.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":500,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.440677+0000","last_change":"2024-04-26T11:26:24.957823+0000","last_active":"2024-04-26T11:30:23.440677+0000","last_peered":"2024-04-26T11:30:23.440677+0000","last_clean":"2024-04-26T11:30:23.440677+0000","last_became_active":"2024-04-26T11:26:24.957434+0000","last_became_peered":"2024-04-26T11:26:24.957434+0000","last_unstale":"2024-04-26T11:30:23.440677+0000","last_undegraded":"2024-04-26T11:30:23.440677+0000","last_fullsized":"2024-04-26T11:30:23.440677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:38:43.247282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71973,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":433,"num_read_kb":281,"num_write":39,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698802+0000","last_change":"2024-04-26T11:26:28.994764+0000","last_active":"2024-04-26T11:30:23.698802+0000","last_peered":"2024-04-26T11:30:23.698802+0000","last_clean":"2024-04-26T11:30:23.698802+0000","last_became_active":"2024-04-26T11:26:28.994421+0000","last_became_peered":"2024-04-26T11:26:28.994421+0000","last_unstale":"2024-04-26T11:30:23.698802+0000","last_undegraded":"2024-04-26T11:30:23.698802+0000","last_fullsized":"2024-04-26T11:30:23.698802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:06:06.499434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165530+0000","last_change":"2024-04-26T11:26:24.968036+0000","last_active":"2024-04-26T11:30:23.165530+0000","last_peered":"2024-04-26T11:30:23.165530+0000","last_clean":"2024-04-26T11:30:23.165530+0000","last_became_active":"2024-04-26T11:26:24.967684+0000","last_became_peered":"2024-04-26T11:26:24.967684+0000","last_unstale":"2024-04-26T11:30:23.165530+0000","last_undegraded":"2024-04-26T11:30:23.165530+0000","last_fullsized":"2024-04-26T11:30:23.165530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:00:27.467411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514528+0000","last_change":"2024-04-26T11:26:29.020642+0000","last_active":"2024-04-26T11:30:23.514528+0000","last_peered":"2024-04-26T11:30:23.514528+0000","last_clean":"2024-04-26T11:30:23.514528+0000","last_became_active":"2024-04-26T11:26:29.020278+0000","last_became_peered":"2024-04-26T11:26:29.020278+0000","last_unstale":"2024-04-26T11:30:23.514528+0000","last_undegraded":"2024-04-26T11:30:23.514528+0000","last_fullsized":"2024-04-26T11:30:23.514528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:48.932064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317403+0000","last_change":"2024-04-26T11:26:24.970111+0000","last_active":"2024-04-26T11:30:23.317403+0000","last_peered":"2024-04-26T11:30:23.317403+0000","last_clean":"2024-04-26T11:30:23.317403+0000","last_became_active":"2024-04-26T11:26:24.969629+0000","last_became_peered":"2024-04-26T11:26:24.969629+0000","last_unstale":"2024-04-26T11:30:23.317403+0000","last_undegraded":"2024-04-26T11:30:23.317403+0000","last_fullsized":"2024-04-26T11:30:23.317403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:52:07.859419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515448+0000","last_change":"2024-04-26T11:26:29.002986+0000","last_active":"2024-04-26T11:30:23.515448+0000","last_peered":"2024-04-26T11:30:23.515448+0000","last_clean":"2024-04-26T11:30:23.515448+0000","last_became_active":"2024-04-26T11:26:29.002610+0000","last_became_peered":"2024-04-26T11:26:29.002610+0000","last_unstale":"2024-04-26T11:30:23.515448+0000","last_undegraded":"2024-04-26T11:30:23.515448+0000","last_fullsized":"2024-04-26T11:30:23.515448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:04.877356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":252,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.611991+0000","last_change":"2024-04-26T11:26:24.957825+0000","last_active":"2024-04-26T11:30:23.611991+0000","last_peered":"2024-04-26T11:30:23.611991+0000","last_clean":"2024-04-26T11:30:23.611991+0000","last_became_active":"2024-04-26T11:26:24.957464+0000","last_became_peered":"2024-04-26T11:26:24.957464+0000","last_unstale":"2024-04-26T11:30:23.611991+0000","last_undegraded":"2024-04-26T11:30:23.611991+0000","last_fullsized":"2024-04-26T11:30:23.611991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:06:49.350268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515219+0000","last_change":"2024-04-26T11:26:28.994223+0000","last_active":"2024-04-26T11:30:23.515219+0000","last_peered":"2024-04-26T11:30:23.515219+0000","last_clean":"2024-04-26T11:30:23.515219+0000","last_became_active":"2024-04-26T11:26:28.993388+0000","last_became_peered":"2024-04-26T11:26:28.993388+0000","last_unstale":"2024-04-26T11:30:23.515219+0000","last_undegraded":"2024-04-26T11:30:23.515219+0000","last_fullsized":"2024-04-26T11:30:23.515219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:07:24.208028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698494+0000","last_change":"2024-04-26T11:26:25.078473+0000","last_active":"2024-04-26T11:30:23.698494+0000","last_peered":"2024-04-26T11:30:23.698494+0000","last_clean":"2024-04-26T11:30:23.698494+0000","last_became_active":"2024-04-26T11:26:25.078138+0000","last_became_peered":"2024-04-26T11:26:25.078138+0000","last_unstale":"2024-04-26T11:30:23.698494+0000","last_undegraded":"2024-04-26T11:30:23.698494+0000","last_fullsized":"2024-04-26T11:30:23.698494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:19.692345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317226+0000","last_change":"2024-04-26T11:26:28.998687+0000","last_active":"2024-04-26T11:30:23.317226+0000","last_peered":"2024-04-26T11:30:23.317226+0000","last_clean":"2024-04-26T11:30:23.317226+0000","last_became_active":"2024-04-26T11:26:28.998358+0000","last_became_peered":"2024-04-26T11:26:28.998358+0000","last_unstale":"2024-04-26T11:30:23.317226+0000","last_undegraded":"2024-04-26T11:30:23.317226+0000","last_fullsized":"2024-04-26T11:30:23.317226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:40:51.358188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698939+0000","last_change":"2024-04-26T11:26:24.956213+0000","last_active":"2024-04-26T11:30:23.698939+0000","last_peered":"2024-04-26T11:30:23.698939+0000","last_clean":"2024-04-26T11:30:23.698939+0000","last_became_active":"2024-04-26T11:26:24.955773+0000","last_became_peered":"2024-04-26T11:26:24.955773+0000","last_unstale":"2024-04-26T11:30:23.698939+0000","last_undegraded":"2024-04-26T11:30:23.698939+0000","last_fullsized":"2024-04-26T11:30:23.698939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:38.253367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308369+0000","last_change":"2024-04-26T11:26:29.001389+0000","last_active":"2024-04-26T11:30:23.308369+0000","last_peered":"2024-04-26T11:30:23.308369+0000","last_clean":"2024-04-26T11:30:23.308369+0000","last_became_active":"2024-04-26T11:26:29.001023+0000","last_became_peered":"2024-04-26T11:26:29.001023+0000","last_unstale":"2024-04-26T11:30:23.308369+0000","last_undegraded":"2024-04-26T11:30:23.308369+0000","last_fullsized":"2024-04-26T11:30:23.308369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:57:23.606518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306250+0000","last_change":"2024-04-26T11:27:19.893959+0000","last_active":"2024-04-26T11:30:23.306250+0000","last_peered":"2024-04-26T11:30:23.306250+0000","last_clean":"2024-04-26T11:30:23.306250+0000","last_became_active":"2024-04-26T11:27:19.892819+0000","last_became_peered":"2024-04-26T11:27:19.892819+0000","last_unstale":"2024-04-26T11:30:23.306250+0000","last_undegraded":"2024-04-26T11:30:23.306250+0000","last_fullsized":"2024-04-26T11:30:23.306250+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:53:39.829381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165605+0000","last_change":"2024-04-26T11:26:29.019906+0000","last_active":"2024-04-26T11:30:23.165605+0000","last_peered":"2024-04-26T11:30:23.165605+0000","last_clean":"2024-04-26T11:30:23.165605+0000","last_became_active":"2024-04-26T11:26:29.019545+0000","last_became_peered":"2024-04-26T11:26:29.019545+0000","last_unstale":"2024-04-26T11:30:23.165605+0000","last_undegraded":"2024-04-26T11:30:23.165605+0000","last_fullsized":"2024-04-26T11:30:23.165605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:48.617269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515436+0000","last_change":"2024-04-26T11:26:24.956400+0000","last_active":"2024-04-26T11:30:23.515436+0000","last_peered":"2024-04-26T11:30:23.515436+0000","last_clean":"2024-04-26T11:30:23.515436+0000","last_became_active":"2024-04-26T11:26:24.956030+0000","last_became_peered":"2024-04-26T11:26:24.956030+0000","last_unstale":"2024-04-26T11:30:23.515436+0000","last_undegraded":"2024-04-26T11:30:23.515436+0000","last_fullsized":"2024-04-26T11:30:23.515436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:47.841006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165227+0000","last_change":"2024-04-26T11:26:29.013087+0000","last_active":"2024-04-26T11:30:23.165227+0000","last_peered":"2024-04-26T11:30:23.165227+0000","last_clean":"2024-04-26T11:30:23.165227+0000","last_became_active":"2024-04-26T11:26:29.012330+0000","last_became_peered":"2024-04-26T11:26:29.012330+0000","last_unstale":"2024-04-26T11:30:23.165227+0000","last_undegraded":"2024-04-26T11:30:23.165227+0000","last_fullsized":"2024-04-26T11:30:23.165227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:12:50.752329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306001+0000","last_change":"2024-04-26T11:27:19.245410+0000","last_active":"2024-04-26T11:30:23.306001+0000","last_peered":"2024-04-26T11:30:23.306001+0000","last_clean":"2024-04-26T11:30:23.306001+0000","last_became_active":"2024-04-26T11:27:19.244578+0000","last_became_peered":"2024-04-26T11:27:19.244578+0000","last_unstale":"2024-04-26T11:30:23.306001+0000","last_undegraded":"2024-04-26T11:30:23.306001+0000","last_fullsized":"2024-04-26T11:30:23.306001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:07:50.576496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":246,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.517472+0000","last_change":"2024-04-26T11:26:25.082517+0000","last_active":"2024-04-26T11:30:23.517472+0000","last_peered":"2024-04-26T11:30:23.517472+0000","last_clean":"2024-04-26T11:30:23.517472+0000","last_became_active":"2024-04-26T11:26:25.081724+0000","last_became_peered":"2024-04-26T11:26:25.081724+0000","last_unstale":"2024-04-26T11:30:23.517472+0000","last_undegraded":"2024-04-26T11:30:23.517472+0000","last_fullsized":"2024-04-26T11:30:23.517472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:59:40.006615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":211,"num_read_kb":211,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.164078+0000","last_change":"2024-04-26T11:26:29.021425+0000","last_active":"2024-04-26T11:30:23.164078+0000","last_peered":"2024-04-26T11:30:23.164078+0000","last_clean":"2024-04-26T11:30:23.164078+0000","last_became_active":"2024-04-26T11:26:29.020881+0000","last_became_peered":"2024-04-26T11:26:29.020881+0000","last_unstale":"2024-04-26T11:30:23.164078+0000","last_undegraded":"2024-04-26T11:30:23.164078+0000","last_fullsized":"2024-04-26T11:30:23.164078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:18.350323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513430+0000","last_change":"2024-04-26T11:26:24.963614+0000","last_active":"2024-04-26T11:30:23.513430+0000","last_peered":"2024-04-26T11:30:23.513430+0000","last_clean":"2024-04-26T11:30:23.513430+0000","last_became_active":"2024-04-26T11:26:24.963312+0000","last_became_peered":"2024-04-26T11:26:24.963312+0000","last_unstale":"2024-04-26T11:30:23.513430+0000","last_undegraded":"2024-04-26T11:30:23.513430+0000","last_fullsized":"2024-04-26T11:30:23.513430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:55.886637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":363,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515509+0000","last_change":"2024-04-26T11:24:54.292436+0000","last_active":"2024-04-26T11:30:23.515509+0000","last_peered":"2024-04-26T11:30:23.515509+0000","last_clean":"2024-04-26T11:30:23.515509+0000","last_became_active":"2024-04-26T11:24:54.291921+0000","last_became_peered":"2024-04-26T11:24:54.291921+0000","last_unstale":"2024-04-26T11:30:23.515509+0000","last_undegraded":"2024-04-26T11:30:23.515509+0000","last_fullsized":"2024-04-26T11:30:23.515509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_clean_scrub_stamp":"2024-04-26T11:21:00.617277+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:11:01.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513834+0000","last_change":"2024-04-26T11:26:29.000905+0000","last_active":"2024-04-26T11:30:23.513834+0000","last_peered":"2024-04-26T11:30:23.513834+0000","last_clean":"2024-04-26T11:30:23.513834+0000","last_became_active":"2024-04-26T11:26:29.000517+0000","last_became_peered":"2024-04-26T11:26:29.000517+0000","last_unstale":"2024-04-26T11:30:23.513834+0000","last_undegraded":"2024-04-26T11:30:23.513834+0000","last_fullsized":"2024-04-26T11:30:23.513834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:38:55.394316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515450+0000","last_change":"2024-04-26T11:26:24.957489+0000","last_active":"2024-04-26T11:30:23.515450+0000","last_peered":"2024-04-26T11:30:23.515450+0000","last_clean":"2024-04-26T11:30:23.515450+0000","last_became_active":"2024-04-26T11:26:24.957151+0000","last_became_peered":"2024-04-26T11:26:24.957151+0000","last_unstale":"2024-04-26T11:30:23.515450+0000","last_undegraded":"2024-04-26T11:30:23.515450+0000","last_fullsized":"2024-04-26T11:30:23.515450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:42:25.343105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698886+0000","last_change":"2024-04-26T11:26:25.086767+0000","last_active":"2024-04-26T11:30:23.698886+0000","last_peered":"2024-04-26T11:30:23.698886+0000","last_clean":"2024-04-26T11:30:23.698886+0000","last_became_active":"2024-04-26T11:26:25.086440+0000","last_became_peered":"2024-04-26T11:26:25.086440+0000","last_unstale":"2024-04-26T11:30:23.698886+0000","last_undegraded":"2024-04-26T11:30:23.698886+0000","last_fullsized":"2024-04-26T11:30:23.698886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:02.294644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514542+0000","last_change":"2024-04-26T11:26:29.003683+0000","last_active":"2024-04-26T11:30:23.514542+0000","last_peered":"2024-04-26T11:30:23.514542+0000","last_clean":"2024-04-26T11:30:23.514542+0000","last_became_active":"2024-04-26T11:26:29.003307+0000","last_became_peered":"2024-04-26T11:26:29.003307+0000","last_unstale":"2024-04-26T11:30:23.514542+0000","last_undegraded":"2024-04-26T11:30:23.514542+0000","last_fullsized":"2024-04-26T11:30:23.514542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:41:00.124540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'49","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.458693+0000","last_change":"2024-04-26T11:27:19.849965+0000","last_active":"2024-04-26T11:30:23.458693+0000","last_peered":"2024-04-26T11:30:23.458693+0000","last_clean":"2024-04-26T11:30:23.458693+0000","last_became_active":"2024-04-26T11:27:19.840545+0000","last_became_peered":"2024-04-26T11:27:19.840545+0000","last_unstale":"2024-04-26T11:30:23.458693+0000","last_undegraded":"2024-04-26T11:30:23.458693+0000","last_fullsized":"2024-04-26T11:30:23.458693+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:08.003704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166953+0000","last_change":"2024-04-26T11:26:29.016783+0000","last_active":"2024-04-26T11:30:23.166953+0000","last_peered":"2024-04-26T11:30:23.166953+0000","last_clean":"2024-04-26T11:30:23.166953+0000","last_became_active":"2024-04-26T11:26:29.013998+0000","last_became_peered":"2024-04-26T11:26:29.013998+0000","last_unstale":"2024-04-26T11:30:23.166953+0000","last_undegraded":"2024-04-26T11:30:23.166953+0000","last_fullsized":"2024-04-26T11:30:23.166953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:24:44.453094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513649+0000","last_change":"2024-04-26T11:26:25.902330+0000","last_active":"2024-04-26T11:30:23.513649+0000","last_peered":"2024-04-26T11:30:23.513649+0000","last_clean":"2024-04-26T11:30:23.513649+0000","last_became_active":"2024-04-26T11:26:25.901948+0000","last_became_peered":"2024-04-26T11:26:25.901948+0000","last_unstale":"2024-04-26T11:30:23.513649+0000","last_undegraded":"2024-04-26T11:30:23.513649+0000","last_fullsized":"2024-04-26T11:30:23.513649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:00:23.541393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518542+0000","last_change":"2024-04-26T11:26:29.025692+0000","last_active":"2024-04-26T11:30:23.518542+0000","last_peered":"2024-04-26T11:30:23.518542+0000","last_clean":"2024-04-26T11:30:23.518542+0000","last_became_active":"2024-04-26T11:26:29.025112+0000","last_became_peered":"2024-04-26T11:26:29.025112+0000","last_unstale":"2024-04-26T11:30:23.518542+0000","last_undegraded":"2024-04-26T11:30:23.518542+0000","last_fullsized":"2024-04-26T11:30:23.518542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:39:22.637865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519849+0000","last_change":"2024-04-26T11:26:25.080270+0000","last_active":"2024-04-26T11:30:23.519849+0000","last_peered":"2024-04-26T11:30:23.519849+0000","last_clean":"2024-04-26T11:30:23.519849+0000","last_became_active":"2024-04-26T11:26:25.079895+0000","last_became_peered":"2024-04-26T11:26:25.079895+0000","last_unstale":"2024-04-26T11:30:23.519849+0000","last_undegraded":"2024-04-26T11:30:23.519849+0000","last_fullsized":"2024-04-26T11:30:23.519849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:47:08.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698571+0000","last_change":"2024-04-26T11:26:28.996540+0000","last_active":"2024-04-26T11:30:23.698571+0000","last_peered":"2024-04-26T11:30:23.698571+0000","last_clean":"2024-04-26T11:30:23.698571+0000","last_became_active":"2024-04-26T11:26:28.994724+0000","last_became_peered":"2024-04-26T11:26:28.994724+0000","last_unstale":"2024-04-26T11:30:23.698571+0000","last_undegraded":"2024-04-26T11:30:23.698571+0000","last_fullsized":"2024-04-26T11:30:23.698571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:13.382332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163378+0000","last_change":"2024-04-26T11:26:24.967935+0000","last_active":"2024-04-26T11:30:23.163378+0000","last_peered":"2024-04-26T11:30:23.163378+0000","last_clean":"2024-04-26T11:30:23.163378+0000","last_became_active":"2024-04-26T11:26:24.966465+0000","last_became_peered":"2024-04-26T11:26:24.966465+0000","last_unstale":"2024-04-26T11:30:23.163378+0000","last_undegraded":"2024-04-26T11:30:23.163378+0000","last_fullsized":"2024-04-26T11:30:23.163378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:27:38.817316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308221+0000","last_change":"2024-04-26T11:26:29.024243+0000","last_active":"2024-04-26T11:30:23.308221+0000","last_peered":"2024-04-26T11:30:23.308221+0000","last_clean":"2024-04-26T11:30:23.308221+0000","last_became_active":"2024-04-26T11:26:29.023900+0000","last_became_peered":"2024-04-26T11:26:29.023900+0000","last_unstale":"2024-04-26T11:30:23.308221+0000","last_undegraded":"2024-04-26T11:30:23.308221+0000","last_fullsized":"2024-04-26T11:30:23.308221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:28:23.446568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165372+0000","last_change":"2024-04-26T11:26:24.969953+0000","last_active":"2024-04-26T11:30:23.165372+0000","last_peered":"2024-04-26T11:30:23.165372+0000","last_clean":"2024-04-26T11:30:23.165372+0000","last_became_active":"2024-04-26T11:26:24.969371+0000","last_became_peered":"2024-04-26T11:26:24.969371+0000","last_unstale":"2024-04-26T11:30:23.165372+0000","last_undegraded":"2024-04-26T11:30:23.165372+0000","last_fullsized":"2024-04-26T11:30:23.165372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:07:24.323833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163379+0000","last_change":"2024-04-26T11:26:29.029771+0000","last_active":"2024-04-26T11:30:23.163379+0000","last_peered":"2024-04-26T11:30:23.163379+0000","last_clean":"2024-04-26T11:30:23.163379+0000","last_became_active":"2024-04-26T11:26:29.029387+0000","last_became_peered":"2024-04-26T11:26:29.029387+0000","last_unstale":"2024-04-26T11:30:23.163379+0000","last_undegraded":"2024-04-26T11:30:23.163379+0000","last_fullsized":"2024-04-26T11:30:23.163379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:40.856935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698580+0000","last_change":"2024-04-26T11:26:24.958608+0000","last_active":"2024-04-26T11:30:23.698580+0000","last_peered":"2024-04-26T11:30:23.698580+0000","last_clean":"2024-04-26T11:30:23.698580+0000","last_became_active":"2024-04-26T11:26:24.958188+0000","last_became_peered":"2024-04-26T11:26:24.958188+0000","last_unstale":"2024-04-26T11:30:23.698580+0000","last_undegraded":"2024-04-26T11:30:23.698580+0000","last_fullsized":"2024-04-26T11:30:23.698580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:31:41.442551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515634+0000","last_change":"2024-04-26T11:26:29.000207+0000","last_active":"2024-04-26T11:30:23.515634+0000","last_peered":"2024-04-26T11:30:23.515634+0000","last_clean":"2024-04-26T11:30:23.515634+0000","last_became_active":"2024-04-26T11:26:28.999894+0000","last_became_peered":"2024-04-26T11:26:28.999894+0000","last_unstale":"2024-04-26T11:30:23.515634+0000","last_undegraded":"2024-04-26T11:30:23.515634+0000","last_fullsized":"2024-04-26T11:30:23.515634+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:52:10.349511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165748+0000","last_change":"2024-04-26T11:26:25.902036+0000","last_active":"2024-04-26T11:30:23.165748+0000","last_peered":"2024-04-26T11:30:23.165748+0000","last_clean":"2024-04-26T11:30:23.165748+0000","last_became_active":"2024-04-26T11:26:25.901567+0000","last_became_peered":"2024-04-26T11:26:25.901567+0000","last_unstale":"2024-04-26T11:30:23.165748+0000","last_undegraded":"2024-04-26T11:30:23.165748+0000","last_fullsized":"2024-04-26T11:30:23.165748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:58.757780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515510+0000","last_change":"2024-04-26T11:26:28.998915+0000","last_active":"2024-04-26T11:30:23.515510+0000","last_peered":"2024-04-26T11:30:23.515510+0000","last_clean":"2024-04-26T11:30:23.515510+0000","last_became_active":"2024-04-26T11:26:28.998180+0000","last_became_peered":"2024-04-26T11:26:28.998180+0000","last_unstale":"2024-04-26T11:30:23.515510+0000","last_undegraded":"2024-04-26T11:30:23.515510+0000","last_fullsized":"2024-04-26T11:30:23.515510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:19.972697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.455501+0000","last_change":"2024-04-26T11:26:24.966029+0000","last_active":"2024-04-26T11:30:23.455501+0000","last_peered":"2024-04-26T11:30:23.455501+0000","last_clean":"2024-04-26T11:30:23.455501+0000","last_became_active":"2024-04-26T11:26:24.965676+0000","last_became_peered":"2024-04-26T11:26:24.965676+0000","last_unstale":"2024-04-26T11:30:23.455501+0000","last_undegraded":"2024-04-26T11:30:23.455501+0000","last_fullsized":"2024-04-26T11:30:23.455501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:18:06.904563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515320+0000","last_change":"2024-04-26T11:26:29.020093+0000","last_active":"2024-04-26T11:30:23.515320+0000","last_peered":"2024-04-26T11:30:23.515320+0000","last_clean":"2024-04-26T11:30:23.515320+0000","last_became_active":"2024-04-26T11:26:29.019690+0000","last_became_peered":"2024-04-26T11:26:29.019690+0000","last_unstale":"2024-04-26T11:30:23.515320+0000","last_undegraded":"2024-04-26T11:30:23.515320+0000","last_fullsized":"2024-04-26T11:30:23.515320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:35:44.501217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515925+0000","last_change":"2024-04-26T11:26:24.966757+0000","last_active":"2024-04-26T11:30:23.515925+0000","last_peered":"2024-04-26T11:30:23.515925+0000","last_clean":"2024-04-26T11:30:23.515925+0000","last_became_active":"2024-04-26T11:26:24.966298+0000","last_became_peered":"2024-04-26T11:26:24.966298+0000","last_unstale":"2024-04-26T11:30:23.515925+0000","last_undegraded":"2024-04-26T11:30:23.515925+0000","last_fullsized":"2024-04-26T11:30:23.515925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:17:40.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515641+0000","last_change":"2024-04-26T11:26:28.997774+0000","last_active":"2024-04-26T11:30:23.515641+0000","last_peered":"2024-04-26T11:30:23.515641+0000","last_clean":"2024-04-26T11:30:23.515641+0000","last_became_active":"2024-04-26T11:26:28.997455+0000","last_became_peered":"2024-04-26T11:26:28.997455+0000","last_unstale":"2024-04-26T11:30:23.515641+0000","last_undegraded":"2024-04-26T11:30:23.515641+0000","last_fullsized":"2024-04-26T11:30:23.515641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:56.097022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308289+0000","last_change":"2024-04-26T11:26:28.999714+0000","last_active":"2024-04-26T11:30:23.308289+0000","last_peered":"2024-04-26T11:30:23.308289+0000","last_clean":"2024-04-26T11:30:23.308289+0000","last_became_active":"2024-04-26T11:26:28.999377+0000","last_became_peered":"2024-04-26T11:26:28.999377+0000","last_unstale":"2024-04-26T11:30:23.308289+0000","last_undegraded":"2024-04-26T11:30:23.308289+0000","last_fullsized":"2024-04-26T11:30:23.308289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:39:59.141556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165347+0000","last_change":"2024-04-26T11:27:19.247154+0000","last_active":"2024-04-26T11:30:23.165347+0000","last_peered":"2024-04-26T11:30:23.165347+0000","last_clean":"2024-04-26T11:30:23.165347+0000","last_became_active":"2024-04-26T11:27:19.246590+0000","last_became_peered":"2024-04-26T11:27:19.246590+0000","last_unstale":"2024-04-26T11:30:23.165347+0000","last_undegraded":"2024-04-26T11:30:23.165347+0000","last_fullsized":"2024-04-26T11:30:23.165347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:12:58.401699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165282+0000","last_change":"2024-04-26T11:26:29.019983+0000","last_active":"2024-04-26T11:30:23.165282+0000","last_peered":"2024-04-26T11:30:23.165282+0000","last_clean":"2024-04-26T11:30:23.165282+0000","last_became_active":"2024-04-26T11:26:29.019536+0000","last_became_peered":"2024-04-26T11:26:29.019536+0000","last_unstale":"2024-04-26T11:30:23.165282+0000","last_undegraded":"2024-04-26T11:30:23.165282+0000","last_fullsized":"2024-04-26T11:30:23.165282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:15:06.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518424+0000","last_change":"2024-04-26T11:26:29.027089+0000","last_active":"2024-04-26T11:30:23.518424+0000","last_peered":"2024-04-26T11:30:23.518424+0000","last_clean":"2024-04-26T11:30:23.518424+0000","last_became_active":"2024-04-26T11:26:29.026030+0000","last_became_peered":"2024-04-26T11:26:29.026030+0000","last_unstale":"2024-04-26T11:30:23.518424+0000","last_undegraded":"2024-04-26T11:30:23.518424+0000","last_fullsized":"2024-04-26T11:30:23.518424+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:24:06.105600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514398+0000","last_change":"2024-04-26T11:26:29.018472+0000","last_active":"2024-04-26T11:30:23.514398+0000","last_peered":"2024-04-26T11:30:23.514398+0000","last_clean":"2024-04-26T11:30:23.514398+0000","last_became_active":"2024-04-26T11:26:29.015947+0000","last_became_peered":"2024-04-26T11:26:29.015947+0000","last_unstale":"2024-04-26T11:30:23.514398+0000","last_undegraded":"2024-04-26T11:30:23.514398+0000","last_fullsized":"2024-04-26T11:30:23.514398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:35:48.082709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308522+0000","last_change":"2024-04-26T11:26:28.997582+0000","last_active":"2024-04-26T11:30:23.308522+0000","last_peered":"2024-04-26T11:30:23.308522+0000","last_clean":"2024-04-26T11:30:23.308522+0000","last_became_active":"2024-04-26T11:26:28.997202+0000","last_became_peered":"2024-04-26T11:26:28.997202+0000","last_unstale":"2024-04-26T11:30:23.308522+0000","last_undegraded":"2024-04-26T11:30:23.308522+0000","last_fullsized":"2024-04-26T11:30:23.308522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:59.776455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165909+0000","last_change":"2024-04-26T11:26:28.999428+0000","last_active":"2024-04-26T11:30:23.165909+0000","last_peered":"2024-04-26T11:30:23.165909+0000","last_clean":"2024-04-26T11:30:23.165909+0000","last_became_active":"2024-04-26T11:26:28.999093+0000","last_became_peered":"2024-04-26T11:26:28.999093+0000","last_unstale":"2024-04-26T11:30:23.165909+0000","last_undegraded":"2024-04-26T11:30:23.165909+0000","last_fullsized":"2024-04-26T11:30:23.165909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:07.222231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166613+0000","last_change":"2024-04-26T11:26:29.019673+0000","last_active":"2024-04-26T11:30:23.166613+0000","last_peered":"2024-04-26T11:30:23.166613+0000","last_clean":"2024-04-26T11:30:23.166613+0000","last_became_active":"2024-04-26T11:26:29.017289+0000","last_became_peered":"2024-04-26T11:26:29.017289+0000","last_unstale":"2024-04-26T11:30:23.166613+0000","last_undegraded":"2024-04-26T11:30:23.166613+0000","last_fullsized":"2024-04-26T11:30:23.166613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:35.848004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317338+0000","last_change":"2024-04-26T11:26:29.017234+0000","last_active":"2024-04-26T11:30:23.317338+0000","last_peered":"2024-04-26T11:30:23.317338+0000","last_clean":"2024-04-26T11:30:23.317338+0000","last_became_active":"2024-04-26T11:26:29.016863+0000","last_became_peered":"2024-04-26T11:26:29.016863+0000","last_unstale":"2024-04-26T11:30:23.317338+0000","last_undegraded":"2024-04-26T11:30:23.317338+0000","last_fullsized":"2024-04-26T11:30:23.317338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:14:31.136286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163192+0000","last_change":"2024-04-26T11:26:29.018122+0000","last_active":"2024-04-26T11:30:23.163192+0000","last_peered":"2024-04-26T11:30:23.163192+0000","last_clean":"2024-04-26T11:30:23.163192+0000","last_became_active":"2024-04-26T11:26:29.016269+0000","last_became_peered":"2024-04-26T11:26:29.016269+0000","last_unstale":"2024-04-26T11:30:23.163192+0000","last_undegraded":"2024-04-26T11:30:23.163192+0000","last_fullsized":"2024-04-26T11:30:23.163192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:22.135086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520050+0000","last_change":"2024-04-26T11:27:19.278596+0000","last_active":"2024-04-26T11:30:23.520050+0000","last_peered":"2024-04-26T11:30:23.520050+0000","last_clean":"2024-04-26T11:30:23.520050+0000","last_became_active":"2024-04-26T11:27:19.278138+0000","last_became_peered":"2024-04-26T11:27:19.278138+0000","last_unstale":"2024-04-26T11:30:23.520050+0000","last_undegraded":"2024-04-26T11:30:23.520050+0000","last_fullsized":"2024-04-26T11:30:23.520050+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:17.038527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514500+0000","last_change":"2024-04-26T11:26:29.026905+0000","last_active":"2024-04-26T11:30:23.514500+0000","last_peered":"2024-04-26T11:30:23.514500+0000","last_clean":"2024-04-26T11:30:23.514500+0000","last_became_active":"2024-04-26T11:26:29.026539+0000","last_became_peered":"2024-04-26T11:26:29.026539+0000","last_unstale":"2024-04-26T11:30:23.514500+0000","last_undegraded":"2024-04-26T11:30:23.514500+0000","last_fullsized":"2024-04-26T11:30:23.514500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:41:12.169466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308385+0000","last_change":"2024-04-26T11:26:28.999953+0000","last_active":"2024-04-26T11:30:23.308385+0000","last_peered":"2024-04-26T11:30:23.308385+0000","last_clean":"2024-04-26T11:30:23.308385+0000","last_became_active":"2024-04-26T11:26:28.998344+0000","last_became_peered":"2024-04-26T11:26:28.998344+0000","last_unstale":"2024-04-26T11:30:23.308385+0000","last_undegraded":"2024-04-26T11:30:23.308385+0000","last_fullsized":"2024-04-26T11:30:23.308385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:30:53.269171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107250,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602104,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1390,"num_read_kb":803,"num_write":299,"num_write_kb":1445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1806312,"data_compressed":81378,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383742,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":882140,"data_compressed":27836,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645380,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1171456,"data_stored":1285292,"data_compressed":52420,"data_compressed_allocated":69632,"data_compressed_original":745472,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808905,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":882376,"data_compressed":27836,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070544,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480124,"data_compressed":5787,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332181,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":1082324,"data_compressed":44440,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495707,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":488549,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":560315,"data_compressed":10514,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":489820,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":562056,"data_compressed":10514,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182508,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1013676,"data_compressed":33992,"data_compressed_allocated":45056,"data_compressed_original":540672,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":490300,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757345,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":1211903,"data_compressed":50757,"data_compressed_allocated":65536,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":71973,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":73714,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":206465,"data_compressed":7980,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T11:30:30.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:30:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:30:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:30:30.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:30:30.481 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 93 KiB/s wr, 38 op/s 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:30.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:30 smithi033 ceph-mon[27845]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:31.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 93 KiB/s wr, 38 op/s 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:31.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:31.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:31.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:31.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:30 smithi194 ceph-mon[33017]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 93 KiB/s wr, 38 op/s 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi096:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi194:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:31.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:30 smithi096 ceph-mon[32753]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:30:32.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 -- 172.21.15.33:0/1425628934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 msgr2=0x7f1830113dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 --2- 172.21.15.33:0/1425628934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f1830113dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f181c009990 tx=0x7f181c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:30:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 -- 172.21.15.33:0/1425628934 shutdown_connections 2024-04-26T11:30:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 --2- 172.21.15.33:0/1425628934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1830114300 0x7f18301166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 --2- 172.21.15.33:0/1425628934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1830077950 0x7f1830077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 --2- 172.21.15.33:0/1425628934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f1830113dc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:32.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 -- 172.21.15.33:0/1425628934 >> 172.21.15.33:0/1425628934 conn(0x7f1830101150 msgr2=0x7f1830103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:32.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 -- 172.21.15.33:0/1425628934 shutdown_connections 2024-04-26T11:30:32.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.812+0000 7f1835600700 1 -- 172.21.15.33:0/1425628934 wait complete. 2024-04-26T11:30:32.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.813+0000 7f1835600700 1 Processor -- start 2024-04-26T11:30:32.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.814+0000 7f1835600700 1 -- start start 2024-04-26T11:30:32.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.814+0000 7f1835600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1830077950 0x7f18301a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:32.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.814+0000 7f1835600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f18301a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.814+0000 7f1835600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1830114300 0x7f18301a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.814+0000 7f1835600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1830119cd0 con 0x7f1830077950 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.814+0000 7f1835600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1830119b50 con 0x7f1830114300 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.814+0000 7f1835600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1830119e50 con 0x7f1830078270 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1830077950 0x7f18301a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182e000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f18301a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1830077950 0x7f18301a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56720/0 (socket says 172.21.15.33:56720) 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182e000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f18301a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:37456/0 (socket says 172.21.15.33:37456) 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182ea00700 1 -- 172.21.15.33:0/2083505267 learned_addr learned my addr 172.21.15.33:0/2083505267 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182f400700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1830114300 0x7f18301a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182e000700 1 -- 172.21.15.33:0/2083505267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1830114300 msgr2=0x7f18301a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182e000700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1830114300 0x7f18301a4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182e000700 1 -- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1830077950 msgr2=0x7f18301a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:32.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182e000700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1830077950 0x7f18301a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:32.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.815+0000 7f182e000700 1 -- 172.21.15.33:0/2083505267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18301a5230 con 0x7f1830078270 2024-04-26T11:30:32.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.816+0000 7f182e000700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f18301a0760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f181c0098e0 tx=0x7f181c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:32.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.816+0000 7f1827400700 1 -- 172.21.15.33:0/2083505267 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c027070 con 0x7f1830078270 2024-04-26T11:30:32.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.816+0000 7f1827400700 1 -- 172.21.15.33:0/2083505267 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181c022470 con 0x7f1830078270 2024-04-26T11:30:32.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.816+0000 7f1827400700 1 -- 172.21.15.33:0/2083505267 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c017650 con 0x7f1830078270 2024-04-26T11:30:32.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.816+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18301a54c0 con 0x7f1830078270 2024-04-26T11:30:32.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.816+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18301acda0 con 0x7f1830078270 2024-04-26T11:30:32.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.817+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f183006a110 con 0x7f1830078270 2024-04-26T11:30:32.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.819+0000 7f1827400700 1 -- 172.21.15.33:0/2083505267 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f181c02a070 con 0x7f1830078270 2024-04-26T11:30:32.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.820+0000 7f1827400700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f18140a3e40 0x7f18140a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:32.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.821+0000 7f1827400700 1 -- 172.21.15.33:0/2083505267 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f181c0d1700 con 0x7f1830078270 2024-04-26T11:30:32.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.821+0000 7f182ea00700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f18140a3e40 0x7f18140a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:32.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.821+0000 7f182ea00700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f18140a3e40 0x7f18140a6300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1820009d20 tx=0x7f1820009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:32.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:32.824+0000 7f1827400700 1 -- 172.21.15.33:0/2083505267 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f181c05eff0 con 0x7f1830078270 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:30:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: Updating smithi033:/var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:30:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:30:33.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.051+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 --> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f18301a1580 con 0x7f18140a3e40 2024-04-26T11:30:33.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.075+0000 7f1827400700 1 -- 172.21.15.33:0/2083505267 <== mgr.14931 v2:172.21.15.33:6800/516906866 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258064 (secure 0 0 0) 0x7f18301a1580 con 0x7f18140a3e40 2024-04-26T11:30:33.080 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:33.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.083+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f18140a3e40 msgr2=0x7f18140a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:33.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.083+0000 7f1835600700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f18140a3e40 0x7f18140a6300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1820009d20 tx=0x7f1820009260 comp rx=0 tx=0).stop 2024-04-26T11:30:33.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 msgr2=0x7f18301a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:33.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f18301a0760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f181c0098e0 tx=0x7f181c0057c0 comp rx=0 tx=0).stop 2024-04-26T11:30:33.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 shutdown_connections 2024-04-26T11:30:33.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f18140a3e40 0x7f18140a6300 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:33.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1830114300 0x7f18301a4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:33.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1830077950 0x7f18301a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:33.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 --2- 172.21.15.33:0/2083505267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1830078270 0x7f18301a0760 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:33.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 >> 172.21.15.33:0/2083505267 conn(0x7f1830101150 msgr2=0x7f1830103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:33.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 shutdown_connections 2024-04-26T11:30:33.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:33.084+0000 7f1835600700 1 -- 172.21.15.33:0/2083505267 wait complete. 2024-04-26T11:30:33.088 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-26T11:30:33.886 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-04-26T11:30:31.190801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389338,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1912,"num_read_kb":1385,"num_write":628,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":962,"ondisk_log_size":962,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357408,"kb_used_data":12372,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584416,"statfs":{"total":1151940427776,"available":1151574441984,"internally_reserved":0,"allocated":12668928,"data_stored":10028118,"data_compressed":320475,"data_compressed_allocated":430080,"data_compressed_original":4190208,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":386,"num_read_kb":463,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.989409"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308540+0000","last_change":"2024-04-26T11:26:29.018818+0000","last_active":"2024-04-26T11:30:23.308540+0000","last_peered":"2024-04-26T11:30:23.308540+0000","last_clean":"2024-04-26T11:30:23.308540+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:30:23.308540+0000","last_undegraded":"2024-04-26T11:30:23.308540+0000","last_fullsized":"2024-04-26T11:30:23.308540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:05.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518402+0000","last_change":"2024-04-26T11:26:29.021338+0000","last_active":"2024-04-26T11:30:23.518402+0000","last_peered":"2024-04-26T11:30:23.518402+0000","last_clean":"2024-04-26T11:30:23.518402+0000","last_became_active":"2024-04-26T11:26:29.021019+0000","last_became_peered":"2024-04-26T11:26:29.021019+0000","last_unstale":"2024-04-26T11:30:23.518402+0000","last_undegraded":"2024-04-26T11:30:23.518402+0000","last_fullsized":"2024-04-26T11:30:23.518402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:52:44.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518696+0000","last_change":"2024-04-26T11:26:29.014166+0000","last_active":"2024-04-26T11:30:23.518696+0000","last_peered":"2024-04-26T11:30:23.518696+0000","last_clean":"2024-04-26T11:30:23.518696+0000","last_became_active":"2024-04-26T11:26:29.013812+0000","last_became_peered":"2024-04-26T11:26:29.013812+0000","last_unstale":"2024-04-26T11:30:23.518696+0000","last_undegraded":"2024-04-26T11:30:23.518696+0000","last_fullsized":"2024-04-26T11:30:23.518696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:08.683633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513524+0000","last_change":"2024-04-26T11:26:28.999249+0000","last_active":"2024-04-26T11:30:23.513524+0000","last_peered":"2024-04-26T11:30:23.513524+0000","last_clean":"2024-04-26T11:30:23.513524+0000","last_became_active":"2024-04-26T11:26:28.998927+0000","last_became_peered":"2024-04-26T11:26:28.998927+0000","last_unstale":"2024-04-26T11:30:23.513524+0000","last_undegraded":"2024-04-26T11:30:23.513524+0000","last_fullsized":"2024-04-26T11:30:23.513524+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:56.674537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698533+0000","last_change":"2024-04-26T11:26:29.017202+0000","last_active":"2024-04-26T11:30:23.698533+0000","last_peered":"2024-04-26T11:30:23.698533+0000","last_clean":"2024-04-26T11:30:23.698533+0000","last_became_active":"2024-04-26T11:26:29.016849+0000","last_became_peered":"2024-04-26T11:26:29.016849+0000","last_unstale":"2024-04-26T11:30:23.698533+0000","last_undegraded":"2024-04-26T11:30:23.698533+0000","last_fullsized":"2024-04-26T11:30:23.698533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:58:22.164118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165326+0000","last_change":"2024-04-26T11:27:19.278294+0000","last_active":"2024-04-26T11:30:23.165326+0000","last_peered":"2024-04-26T11:30:23.165326+0000","last_clean":"2024-04-26T11:30:23.165326+0000","last_became_active":"2024-04-26T11:27:19.277853+0000","last_became_peered":"2024-04-26T11:27:19.277853+0000","last_unstale":"2024-04-26T11:30:23.165326+0000","last_undegraded":"2024-04-26T11:30:23.165326+0000","last_fullsized":"2024-04-26T11:30:23.165326+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:12:18.733166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515577+0000","last_change":"2024-04-26T11:26:28.991406+0000","last_active":"2024-04-26T11:30:23.515577+0000","last_peered":"2024-04-26T11:30:23.515577+0000","last_clean":"2024-04-26T11:30:23.515577+0000","last_became_active":"2024-04-26T11:26:28.989303+0000","last_became_peered":"2024-04-26T11:26:28.989303+0000","last_unstale":"2024-04-26T11:30:23.515577+0000","last_undegraded":"2024-04-26T11:30:23.515577+0000","last_fullsized":"2024-04-26T11:30:23.515577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:42:59.449078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163073+0000","last_change":"2024-04-26T11:26:29.020922+0000","last_active":"2024-04-26T11:30:23.163073+0000","last_peered":"2024-04-26T11:30:23.163073+0000","last_clean":"2024-04-26T11:30:23.163073+0000","last_became_active":"2024-04-26T11:26:29.020576+0000","last_became_peered":"2024-04-26T11:26:29.020576+0000","last_unstale":"2024-04-26T11:30:23.163073+0000","last_undegraded":"2024-04-26T11:30:23.163073+0000","last_fullsized":"2024-04-26T11:30:23.163073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:48.770319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520362+0000","last_change":"2024-04-26T11:26:29.004171+0000","last_active":"2024-04-26T11:30:23.520362+0000","last_peered":"2024-04-26T11:30:23.520362+0000","last_clean":"2024-04-26T11:30:23.520362+0000","last_became_active":"2024-04-26T11:26:29.003816+0000","last_became_peered":"2024-04-26T11:26:29.003816+0000","last_unstale":"2024-04-26T11:30:23.520362+0000","last_undegraded":"2024-04-26T11:30:23.520362+0000","last_fullsized":"2024-04-26T11:30:23.520362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:45:14.807634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165819+0000","last_change":"2024-04-26T11:26:28.999453+0000","last_active":"2024-04-26T11:30:23.165819+0000","last_peered":"2024-04-26T11:30:23.165819+0000","last_clean":"2024-04-26T11:30:23.165819+0000","last_became_active":"2024-04-26T11:26:28.999104+0000","last_became_peered":"2024-04-26T11:26:28.999104+0000","last_unstale":"2024-04-26T11:30:23.165819+0000","last_undegraded":"2024-04-26T11:30:23.165819+0000","last_fullsized":"2024-04-26T11:30:23.165819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:55.408949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515355+0000","last_change":"2024-04-26T11:26:28.999218+0000","last_active":"2024-04-26T11:30:23.515355+0000","last_peered":"2024-04-26T11:30:23.515355+0000","last_clean":"2024-04-26T11:30:23.515355+0000","last_became_active":"2024-04-26T11:26:28.998584+0000","last_became_peered":"2024-04-26T11:26:28.998584+0000","last_unstale":"2024-04-26T11:30:23.515355+0000","last_undegraded":"2024-04-26T11:30:23.515355+0000","last_fullsized":"2024-04-26T11:30:23.515355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:18.040520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513411+0000","last_change":"2024-04-26T11:26:28.997121+0000","last_active":"2024-04-26T11:30:23.513411+0000","last_peered":"2024-04-26T11:30:23.513411+0000","last_clean":"2024-04-26T11:30:23.513411+0000","last_became_active":"2024-04-26T11:26:28.996556+0000","last_became_peered":"2024-04-26T11:26:28.996556+0000","last_unstale":"2024-04-26T11:30:23.513411+0000","last_undegraded":"2024-04-26T11:30:23.513411+0000","last_fullsized":"2024-04-26T11:30:23.513411+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:18:55.251371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514338+0000","last_change":"2024-04-26T11:26:28.999525+0000","last_active":"2024-04-26T11:30:23.514338+0000","last_peered":"2024-04-26T11:30:23.514338+0000","last_clean":"2024-04-26T11:30:23.514338+0000","last_became_active":"2024-04-26T11:26:28.998578+0000","last_became_peered":"2024-04-26T11:26:28.998578+0000","last_unstale":"2024-04-26T11:30:23.514338+0000","last_undegraded":"2024-04-26T11:30:23.514338+0000","last_fullsized":"2024-04-26T11:30:23.514338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:32:35.523804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518043+0000","last_change":"2024-04-26T11:26:29.008427+0000","last_active":"2024-04-26T11:30:23.518043+0000","last_peered":"2024-04-26T11:30:23.518043+0000","last_clean":"2024-04-26T11:30:23.518043+0000","last_became_active":"2024-04-26T11:26:29.008073+0000","last_became_peered":"2024-04-26T11:26:29.008073+0000","last_unstale":"2024-04-26T11:30:23.518043+0000","last_undegraded":"2024-04-26T11:30:23.518043+0000","last_fullsized":"2024-04-26T11:30:23.518043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:03.185941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518029+0000","last_change":"2024-04-26T11:26:29.015136+0000","last_active":"2024-04-26T11:30:23.518029+0000","last_peered":"2024-04-26T11:30:23.518029+0000","last_clean":"2024-04-26T11:30:23.518029+0000","last_became_active":"2024-04-26T11:26:29.014800+0000","last_became_peered":"2024-04-26T11:26:29.014800+0000","last_unstale":"2024-04-26T11:30:23.518029+0000","last_undegraded":"2024-04-26T11:30:23.518029+0000","last_fullsized":"2024-04-26T11:30:23.518029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:28:35.871046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519847+0000","last_change":"2024-04-26T11:26:29.024795+0000","last_active":"2024-04-26T11:30:23.519847+0000","last_peered":"2024-04-26T11:30:23.519847+0000","last_clean":"2024-04-26T11:30:23.519847+0000","last_became_active":"2024-04-26T11:26:29.024430+0000","last_became_peered":"2024-04-26T11:26:29.024430+0000","last_unstale":"2024-04-26T11:30:23.519847+0000","last_undegraded":"2024-04-26T11:30:23.519847+0000","last_fullsized":"2024-04-26T11:30:23.519847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:05:20.172869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308685+0000","last_change":"2024-04-26T11:26:29.018822+0000","last_active":"2024-04-26T11:30:23.308685+0000","last_peered":"2024-04-26T11:30:23.308685+0000","last_clean":"2024-04-26T11:30:23.308685+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:30:23.308685+0000","last_undegraded":"2024-04-26T11:30:23.308685+0000","last_fullsized":"2024-04-26T11:30:23.308685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:04.475400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306306+0000","last_change":"2024-04-26T11:26:29.001984+0000","last_active":"2024-04-26T11:30:23.306306+0000","last_peered":"2024-04-26T11:30:23.306306+0000","last_clean":"2024-04-26T11:30:23.306306+0000","last_became_active":"2024-04-26T11:26:28.999492+0000","last_became_peered":"2024-04-26T11:26:28.999492+0000","last_unstale":"2024-04-26T11:30:23.306306+0000","last_undegraded":"2024-04-26T11:30:23.306306+0000","last_fullsized":"2024-04-26T11:30:23.306306+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:21.398823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165815+0000","last_change":"2024-04-26T11:26:29.019677+0000","last_active":"2024-04-26T11:30:23.165815+0000","last_peered":"2024-04-26T11:30:23.165815+0000","last_clean":"2024-04-26T11:30:23.165815+0000","last_became_active":"2024-04-26T11:26:29.019315+0000","last_became_peered":"2024-04-26T11:26:29.019315+0000","last_unstale":"2024-04-26T11:30:23.165815+0000","last_undegraded":"2024-04-26T11:30:23.165815+0000","last_fullsized":"2024-04-26T11:30:23.165815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:20.239140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306242+0000","last_change":"2024-04-26T11:26:24.964887+0000","last_active":"2024-04-26T11:30:23.306242+0000","last_peered":"2024-04-26T11:30:23.306242+0000","last_clean":"2024-04-26T11:30:23.306242+0000","last_became_active":"2024-04-26T11:26:24.964380+0000","last_became_peered":"2024-04-26T11:26:24.964380+0000","last_unstale":"2024-04-26T11:30:23.306242+0000","last_undegraded":"2024-04-26T11:30:23.306242+0000","last_fullsized":"2024-04-26T11:30:23.306242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:59:41.347542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306031+0000","last_change":"2024-04-26T11:27:19.244889+0000","last_active":"2024-04-26T11:30:23.306031+0000","last_peered":"2024-04-26T11:30:23.306031+0000","last_clean":"2024-04-26T11:30:23.306031+0000","last_became_active":"2024-04-26T11:27:19.244428+0000","last_became_peered":"2024-04-26T11:27:19.244428+0000","last_unstale":"2024-04-26T11:30:23.306031+0000","last_undegraded":"2024-04-26T11:30:23.306031+0000","last_fullsized":"2024-04-26T11:30:23.306031+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:09:39.366242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166356+0000","last_change":"2024-04-26T11:26:25.082741+0000","last_active":"2024-04-26T11:30:23.166356+0000","last_peered":"2024-04-26T11:30:23.166356+0000","last_clean":"2024-04-26T11:30:23.166356+0000","last_became_active":"2024-04-26T11:26:25.082361+0000","last_became_peered":"2024-04-26T11:26:25.082361+0000","last_unstale":"2024-04-26T11:30:23.166356+0000","last_undegraded":"2024-04-26T11:30:23.166356+0000","last_fullsized":"2024-04-26T11:30:23.166356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:23:17.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519051+0000","last_change":"2024-04-26T11:26:29.014817+0000","last_active":"2024-04-26T11:30:23.519051+0000","last_peered":"2024-04-26T11:30:23.519051+0000","last_clean":"2024-04-26T11:30:23.519051+0000","last_became_active":"2024-04-26T11:26:29.014419+0000","last_became_peered":"2024-04-26T11:26:29.014419+0000","last_unstale":"2024-04-26T11:30:23.519051+0000","last_undegraded":"2024-04-26T11:30:23.519051+0000","last_fullsized":"2024-04-26T11:30:23.519051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:33:28.355075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698995+0000","last_change":"2024-04-26T11:26:24.960224+0000","last_active":"2024-04-26T11:30:23.698995+0000","last_peered":"2024-04-26T11:30:23.698995+0000","last_clean":"2024-04-26T11:30:23.698995+0000","last_became_active":"2024-04-26T11:26:24.959848+0000","last_became_peered":"2024-04-26T11:26:24.959848+0000","last_unstale":"2024-04-26T11:30:23.698995+0000","last_undegraded":"2024-04-26T11:30:23.698995+0000","last_fullsized":"2024-04-26T11:30:23.698995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:49:10.932187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520146+0000","last_change":"2024-04-26T11:26:29.001946+0000","last_active":"2024-04-26T11:30:23.520146+0000","last_peered":"2024-04-26T11:30:23.520146+0000","last_clean":"2024-04-26T11:30:23.520146+0000","last_became_active":"2024-04-26T11:26:29.001579+0000","last_became_peered":"2024-04-26T11:26:29.001579+0000","last_unstale":"2024-04-26T11:30:23.520146+0000","last_undegraded":"2024-04-26T11:30:23.520146+0000","last_fullsized":"2024-04-26T11:30:23.520146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:14:43.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698818+0000","last_change":"2024-04-26T11:26:24.956148+0000","last_active":"2024-04-26T11:30:23.698818+0000","last_peered":"2024-04-26T11:30:23.698818+0000","last_clean":"2024-04-26T11:30:23.698818+0000","last_became_active":"2024-04-26T11:26:24.955736+0000","last_became_peered":"2024-04-26T11:26:24.955736+0000","last_unstale":"2024-04-26T11:30:23.698818+0000","last_undegraded":"2024-04-26T11:30:23.698818+0000","last_fullsized":"2024-04-26T11:30:23.698818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:43.705470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317648+0000","last_change":"2024-04-26T11:27:19.415134+0000","last_active":"2024-04-26T11:30:23.317648+0000","last_peered":"2024-04-26T11:30:23.317648+0000","last_clean":"2024-04-26T11:30:23.317648+0000","last_became_active":"2024-04-26T11:27:19.414703+0000","last_became_peered":"2024-04-26T11:27:19.414703+0000","last_unstale":"2024-04-26T11:30:23.317648+0000","last_undegraded":"2024-04-26T11:30:23.317648+0000","last_fullsized":"2024-04-26T11:30:23.317648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:41:11.268296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":254,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:27.445778+0000","last_change":"2024-04-26T11:26:24.958350+0000","last_active":"2024-04-26T11:30:27.445778+0000","last_peered":"2024-04-26T11:30:27.445778+0000","last_clean":"2024-04-26T11:30:27.445778+0000","last_became_active":"2024-04-26T11:26:24.957382+0000","last_became_peered":"2024-04-26T11:26:24.957382+0000","last_unstale":"2024-04-26T11:30:27.445778+0000","last_undegraded":"2024-04-26T11:30:27.445778+0000","last_fullsized":"2024-04-26T11:30:27.445778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:03.854227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698584+0000","last_change":"2024-04-26T11:26:28.997568+0000","last_active":"2024-04-26T11:30:23.698584+0000","last_peered":"2024-04-26T11:30:23.698584+0000","last_clean":"2024-04-26T11:30:23.698584+0000","last_became_active":"2024-04-26T11:26:28.997161+0000","last_became_peered":"2024-04-26T11:26:28.997161+0000","last_unstale":"2024-04-26T11:30:23.698584+0000","last_undegraded":"2024-04-26T11:30:23.698584+0000","last_fullsized":"2024-04-26T11:30:23.698584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:04.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165161+0000","last_change":"2024-04-26T11:26:24.958093+0000","last_active":"2024-04-26T11:30:23.165161+0000","last_peered":"2024-04-26T11:30:23.165161+0000","last_clean":"2024-04-26T11:30:23.165161+0000","last_became_active":"2024-04-26T11:26:24.957730+0000","last_became_peered":"2024-04-26T11:26:24.957730+0000","last_unstale":"2024-04-26T11:30:23.165161+0000","last_undegraded":"2024-04-26T11:30:23.165161+0000","last_fullsized":"2024-04-26T11:30:23.165161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:09:29.827063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306012+0000","last_change":"2024-04-26T11:27:19.248406+0000","last_active":"2024-04-26T11:30:23.306012+0000","last_peered":"2024-04-26T11:30:23.306012+0000","last_clean":"2024-04-26T11:30:23.306012+0000","last_became_active":"2024-04-26T11:27:19.247980+0000","last_became_peered":"2024-04-26T11:27:19.247980+0000","last_unstale":"2024-04-26T11:30:23.306012+0000","last_undegraded":"2024-04-26T11:30:23.306012+0000","last_fullsized":"2024-04-26T11:30:23.306012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:51:34.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514048+0000","last_change":"2024-04-26T11:26:24.963463+0000","last_active":"2024-04-26T11:30:23.514048+0000","last_peered":"2024-04-26T11:30:23.514048+0000","last_clean":"2024-04-26T11:30:23.514048+0000","last_became_active":"2024-04-26T11:26:24.963030+0000","last_became_peered":"2024-04-26T11:26:24.963030+0000","last_unstale":"2024-04-26T11:30:23.514048+0000","last_undegraded":"2024-04-26T11:30:23.514048+0000","last_fullsized":"2024-04-26T11:30:23.514048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:04:17.844500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163342+0000","last_change":"2024-04-26T11:26:29.013899+0000","last_active":"2024-04-26T11:30:23.163342+0000","last_peered":"2024-04-26T11:30:23.163342+0000","last_clean":"2024-04-26T11:30:23.163342+0000","last_became_active":"2024-04-26T11:26:29.012674+0000","last_became_peered":"2024-04-26T11:26:29.012674+0000","last_unstale":"2024-04-26T11:30:23.163342+0000","last_undegraded":"2024-04-26T11:30:23.163342+0000","last_fullsized":"2024-04-26T11:30:23.163342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:33:46.259841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317199+0000","last_change":"2024-04-26T11:26:24.957803+0000","last_active":"2024-04-26T11:30:23.317199+0000","last_peered":"2024-04-26T11:30:23.317199+0000","last_clean":"2024-04-26T11:30:23.317199+0000","last_became_active":"2024-04-26T11:26:24.957496+0000","last_became_peered":"2024-04-26T11:26:24.957496+0000","last_unstale":"2024-04-26T11:30:23.317199+0000","last_undegraded":"2024-04-26T11:30:23.317199+0000","last_fullsized":"2024-04-26T11:30:23.317199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:15.497106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306343+0000","last_change":"2024-04-26T11:26:29.021648+0000","last_active":"2024-04-26T11:30:23.306343+0000","last_peered":"2024-04-26T11:30:23.306343+0000","last_clean":"2024-04-26T11:30:23.306343+0000","last_became_active":"2024-04-26T11:26:29.020689+0000","last_became_peered":"2024-04-26T11:26:29.020689+0000","last_unstale":"2024-04-26T11:30:23.306343+0000","last_undegraded":"2024-04-26T11:30:23.306343+0000","last_fullsized":"2024-04-26T11:30:23.306343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:15:42.321851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":249,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:27.445215+0000","last_change":"2024-04-26T11:26:24.966184+0000","last_active":"2024-04-26T11:30:27.445215+0000","last_peered":"2024-04-26T11:30:27.445215+0000","last_clean":"2024-04-26T11:30:27.445215+0000","last_became_active":"2024-04-26T11:26:24.965850+0000","last_became_peered":"2024-04-26T11:26:24.965850+0000","last_unstale":"2024-04-26T11:30:27.445215+0000","last_undegraded":"2024-04-26T11:30:27.445215+0000","last_fullsized":"2024-04-26T11:30:27.445215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:49.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520125+0000","last_change":"2024-04-26T11:26:29.003570+0000","last_active":"2024-04-26T11:30:23.520125+0000","last_peered":"2024-04-26T11:30:23.520125+0000","last_clean":"2024-04-26T11:30:23.520125+0000","last_became_active":"2024-04-26T11:26:29.003039+0000","last_became_peered":"2024-04-26T11:26:29.003039+0000","last_unstale":"2024-04-26T11:30:23.520125+0000","last_undegraded":"2024-04-26T11:30:23.520125+0000","last_fullsized":"2024-04-26T11:30:23.520125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:29:16.166432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515513+0000","last_change":"2024-04-26T11:26:24.954968+0000","last_active":"2024-04-26T11:30:23.515513+0000","last_peered":"2024-04-26T11:30:23.515513+0000","last_clean":"2024-04-26T11:30:23.515513+0000","last_became_active":"2024-04-26T11:26:24.954595+0000","last_became_peered":"2024-04-26T11:26:24.954595+0000","last_unstale":"2024-04-26T11:30:23.515513+0000","last_undegraded":"2024-04-26T11:30:23.515513+0000","last_fullsized":"2024-04-26T11:30:23.515513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:52:21.988942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518513+0000","last_change":"2024-04-26T11:26:29.015601+0000","last_active":"2024-04-26T11:30:23.518513+0000","last_peered":"2024-04-26T11:30:23.518513+0000","last_clean":"2024-04-26T11:30:23.518513+0000","last_became_active":"2024-04-26T11:26:29.015199+0000","last_became_peered":"2024-04-26T11:26:29.015199+0000","last_unstale":"2024-04-26T11:30:23.518513+0000","last_undegraded":"2024-04-26T11:30:23.518513+0000","last_fullsized":"2024-04-26T11:30:23.518513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:15:04.573214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317777+0000","last_change":"2024-04-26T11:26:25.085542+0000","last_active":"2024-04-26T11:30:23.317777+0000","last_peered":"2024-04-26T11:30:23.317777+0000","last_clean":"2024-04-26T11:30:23.317777+0000","last_became_active":"2024-04-26T11:26:25.085230+0000","last_became_peered":"2024-04-26T11:26:25.085230+0000","last_unstale":"2024-04-26T11:30:23.317777+0000","last_undegraded":"2024-04-26T11:30:23.317777+0000","last_fullsized":"2024-04-26T11:30:23.317777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:51.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520229+0000","last_change":"2024-04-26T11:26:29.003954+0000","last_active":"2024-04-26T11:30:23.520229+0000","last_peered":"2024-04-26T11:30:23.520229+0000","last_clean":"2024-04-26T11:30:23.520229+0000","last_became_active":"2024-04-26T11:26:29.003576+0000","last_became_peered":"2024-04-26T11:26:29.003576+0000","last_unstale":"2024-04-26T11:30:23.520229+0000","last_undegraded":"2024-04-26T11:30:23.520229+0000","last_fullsized":"2024-04-26T11:30:23.520229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:24.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":511,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:28.446722+0000","last_change":"2024-04-26T11:26:24.957823+0000","last_active":"2024-04-26T11:30:28.446722+0000","last_peered":"2024-04-26T11:30:28.446722+0000","last_clean":"2024-04-26T11:30:28.446722+0000","last_became_active":"2024-04-26T11:26:24.957434+0000","last_became_peered":"2024-04-26T11:26:24.957434+0000","last_unstale":"2024-04-26T11:30:28.446722+0000","last_undegraded":"2024-04-26T11:30:28.446722+0000","last_fullsized":"2024-04-26T11:30:28.446722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:38:43.247282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71973,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":444,"num_read_kb":292,"num_write":39,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698802+0000","last_change":"2024-04-26T11:26:28.994764+0000","last_active":"2024-04-26T11:30:23.698802+0000","last_peered":"2024-04-26T11:30:23.698802+0000","last_clean":"2024-04-26T11:30:23.698802+0000","last_became_active":"2024-04-26T11:26:28.994421+0000","last_became_peered":"2024-04-26T11:26:28.994421+0000","last_unstale":"2024-04-26T11:30:23.698802+0000","last_undegraded":"2024-04-26T11:30:23.698802+0000","last_fullsized":"2024-04-26T11:30:23.698802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:06:06.499434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165530+0000","last_change":"2024-04-26T11:26:24.968036+0000","last_active":"2024-04-26T11:30:23.165530+0000","last_peered":"2024-04-26T11:30:23.165530+0000","last_clean":"2024-04-26T11:30:23.165530+0000","last_became_active":"2024-04-26T11:26:24.967684+0000","last_became_peered":"2024-04-26T11:26:24.967684+0000","last_unstale":"2024-04-26T11:30:23.165530+0000","last_undegraded":"2024-04-26T11:30:23.165530+0000","last_fullsized":"2024-04-26T11:30:23.165530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:00:27.467411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514528+0000","last_change":"2024-04-26T11:26:29.020642+0000","last_active":"2024-04-26T11:30:23.514528+0000","last_peered":"2024-04-26T11:30:23.514528+0000","last_clean":"2024-04-26T11:30:23.514528+0000","last_became_active":"2024-04-26T11:26:29.020278+0000","last_became_peered":"2024-04-26T11:26:29.020278+0000","last_unstale":"2024-04-26T11:30:23.514528+0000","last_undegraded":"2024-04-26T11:30:23.514528+0000","last_fullsized":"2024-04-26T11:30:23.514528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:48.932064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317403+0000","last_change":"2024-04-26T11:26:24.970111+0000","last_active":"2024-04-26T11:30:23.317403+0000","last_peered":"2024-04-26T11:30:23.317403+0000","last_clean":"2024-04-26T11:30:23.317403+0000","last_became_active":"2024-04-26T11:26:24.969629+0000","last_became_peered":"2024-04-26T11:26:24.969629+0000","last_unstale":"2024-04-26T11:30:23.317403+0000","last_undegraded":"2024-04-26T11:30:23.317403+0000","last_fullsized":"2024-04-26T11:30:23.317403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:52:07.859419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515448+0000","last_change":"2024-04-26T11:26:29.002986+0000","last_active":"2024-04-26T11:30:23.515448+0000","last_peered":"2024-04-26T11:30:23.515448+0000","last_clean":"2024-04-26T11:30:23.515448+0000","last_became_active":"2024-04-26T11:26:29.002610+0000","last_became_peered":"2024-04-26T11:26:29.002610+0000","last_unstale":"2024-04-26T11:30:23.515448+0000","last_undegraded":"2024-04-26T11:30:23.515448+0000","last_fullsized":"2024-04-26T11:30:23.515448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:04.877356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":257,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:27.619602+0000","last_change":"2024-04-26T11:26:24.957825+0000","last_active":"2024-04-26T11:30:27.619602+0000","last_peered":"2024-04-26T11:30:27.619602+0000","last_clean":"2024-04-26T11:30:27.619602+0000","last_became_active":"2024-04-26T11:26:24.957464+0000","last_became_peered":"2024-04-26T11:26:24.957464+0000","last_unstale":"2024-04-26T11:30:27.619602+0000","last_undegraded":"2024-04-26T11:30:27.619602+0000","last_fullsized":"2024-04-26T11:30:27.619602+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:06:49.350268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":222,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515219+0000","last_change":"2024-04-26T11:26:28.994223+0000","last_active":"2024-04-26T11:30:23.515219+0000","last_peered":"2024-04-26T11:30:23.515219+0000","last_clean":"2024-04-26T11:30:23.515219+0000","last_became_active":"2024-04-26T11:26:28.993388+0000","last_became_peered":"2024-04-26T11:26:28.993388+0000","last_unstale":"2024-04-26T11:30:23.515219+0000","last_undegraded":"2024-04-26T11:30:23.515219+0000","last_fullsized":"2024-04-26T11:30:23.515219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:07:24.208028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698494+0000","last_change":"2024-04-26T11:26:25.078473+0000","last_active":"2024-04-26T11:30:23.698494+0000","last_peered":"2024-04-26T11:30:23.698494+0000","last_clean":"2024-04-26T11:30:23.698494+0000","last_became_active":"2024-04-26T11:26:25.078138+0000","last_became_peered":"2024-04-26T11:26:25.078138+0000","last_unstale":"2024-04-26T11:30:23.698494+0000","last_undegraded":"2024-04-26T11:30:23.698494+0000","last_fullsized":"2024-04-26T11:30:23.698494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:19.692345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317226+0000","last_change":"2024-04-26T11:26:28.998687+0000","last_active":"2024-04-26T11:30:23.317226+0000","last_peered":"2024-04-26T11:30:23.317226+0000","last_clean":"2024-04-26T11:30:23.317226+0000","last_became_active":"2024-04-26T11:26:28.998358+0000","last_became_peered":"2024-04-26T11:26:28.998358+0000","last_unstale":"2024-04-26T11:30:23.317226+0000","last_undegraded":"2024-04-26T11:30:23.317226+0000","last_fullsized":"2024-04-26T11:30:23.317226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:40:51.358188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698939+0000","last_change":"2024-04-26T11:26:24.956213+0000","last_active":"2024-04-26T11:30:23.698939+0000","last_peered":"2024-04-26T11:30:23.698939+0000","last_clean":"2024-04-26T11:30:23.698939+0000","last_became_active":"2024-04-26T11:26:24.955773+0000","last_became_peered":"2024-04-26T11:26:24.955773+0000","last_unstale":"2024-04-26T11:30:23.698939+0000","last_undegraded":"2024-04-26T11:30:23.698939+0000","last_fullsized":"2024-04-26T11:30:23.698939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:38.253367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308369+0000","last_change":"2024-04-26T11:26:29.001389+0000","last_active":"2024-04-26T11:30:23.308369+0000","last_peered":"2024-04-26T11:30:23.308369+0000","last_clean":"2024-04-26T11:30:23.308369+0000","last_became_active":"2024-04-26T11:26:29.001023+0000","last_became_peered":"2024-04-26T11:26:29.001023+0000","last_unstale":"2024-04-26T11:30:23.308369+0000","last_undegraded":"2024-04-26T11:30:23.308369+0000","last_fullsized":"2024-04-26T11:30:23.308369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:57:23.606518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306250+0000","last_change":"2024-04-26T11:27:19.893959+0000","last_active":"2024-04-26T11:30:23.306250+0000","last_peered":"2024-04-26T11:30:23.306250+0000","last_clean":"2024-04-26T11:30:23.306250+0000","last_became_active":"2024-04-26T11:27:19.892819+0000","last_became_peered":"2024-04-26T11:27:19.892819+0000","last_unstale":"2024-04-26T11:30:23.306250+0000","last_undegraded":"2024-04-26T11:30:23.306250+0000","last_fullsized":"2024-04-26T11:30:23.306250+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:53:39.829381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165605+0000","last_change":"2024-04-26T11:26:29.019906+0000","last_active":"2024-04-26T11:30:23.165605+0000","last_peered":"2024-04-26T11:30:23.165605+0000","last_clean":"2024-04-26T11:30:23.165605+0000","last_became_active":"2024-04-26T11:26:29.019545+0000","last_became_peered":"2024-04-26T11:26:29.019545+0000","last_unstale":"2024-04-26T11:30:23.165605+0000","last_undegraded":"2024-04-26T11:30:23.165605+0000","last_fullsized":"2024-04-26T11:30:23.165605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:48.617269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515436+0000","last_change":"2024-04-26T11:26:24.956400+0000","last_active":"2024-04-26T11:30:23.515436+0000","last_peered":"2024-04-26T11:30:23.515436+0000","last_clean":"2024-04-26T11:30:23.515436+0000","last_became_active":"2024-04-26T11:26:24.956030+0000","last_became_peered":"2024-04-26T11:26:24.956030+0000","last_unstale":"2024-04-26T11:30:23.515436+0000","last_undegraded":"2024-04-26T11:30:23.515436+0000","last_fullsized":"2024-04-26T11:30:23.515436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:47.841006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165227+0000","last_change":"2024-04-26T11:26:29.013087+0000","last_active":"2024-04-26T11:30:23.165227+0000","last_peered":"2024-04-26T11:30:23.165227+0000","last_clean":"2024-04-26T11:30:23.165227+0000","last_became_active":"2024-04-26T11:26:29.012330+0000","last_became_peered":"2024-04-26T11:26:29.012330+0000","last_unstale":"2024-04-26T11:30:23.165227+0000","last_undegraded":"2024-04-26T11:30:23.165227+0000","last_fullsized":"2024-04-26T11:30:23.165227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:12:50.752329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306001+0000","last_change":"2024-04-26T11:27:19.245410+0000","last_active":"2024-04-26T11:30:23.306001+0000","last_peered":"2024-04-26T11:30:23.306001+0000","last_clean":"2024-04-26T11:30:23.306001+0000","last_became_active":"2024-04-26T11:27:19.244578+0000","last_became_peered":"2024-04-26T11:27:19.244578+0000","last_unstale":"2024-04-26T11:30:23.306001+0000","last_undegraded":"2024-04-26T11:30:23.306001+0000","last_fullsized":"2024-04-26T11:30:23.306001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:07:50.576496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":251,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:27.618587+0000","last_change":"2024-04-26T11:26:25.082517+0000","last_active":"2024-04-26T11:30:27.618587+0000","last_peered":"2024-04-26T11:30:27.618587+0000","last_clean":"2024-04-26T11:30:27.618587+0000","last_became_active":"2024-04-26T11:26:25.081724+0000","last_became_peered":"2024-04-26T11:26:25.081724+0000","last_unstale":"2024-04-26T11:30:27.618587+0000","last_undegraded":"2024-04-26T11:30:27.618587+0000","last_fullsized":"2024-04-26T11:30:27.618587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:59:40.006615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.164078+0000","last_change":"2024-04-26T11:26:29.021425+0000","last_active":"2024-04-26T11:30:23.164078+0000","last_peered":"2024-04-26T11:30:23.164078+0000","last_clean":"2024-04-26T11:30:23.164078+0000","last_became_active":"2024-04-26T11:26:29.020881+0000","last_became_peered":"2024-04-26T11:26:29.020881+0000","last_unstale":"2024-04-26T11:30:23.164078+0000","last_undegraded":"2024-04-26T11:30:23.164078+0000","last_fullsized":"2024-04-26T11:30:23.164078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:18.350323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513430+0000","last_change":"2024-04-26T11:26:24.963614+0000","last_active":"2024-04-26T11:30:23.513430+0000","last_peered":"2024-04-26T11:30:23.513430+0000","last_clean":"2024-04-26T11:30:23.513430+0000","last_became_active":"2024-04-26T11:26:24.963312+0000","last_became_peered":"2024-04-26T11:26:24.963312+0000","last_unstale":"2024-04-26T11:30:23.513430+0000","last_undegraded":"2024-04-26T11:30:23.513430+0000","last_fullsized":"2024-04-26T11:30:23.513430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:55.886637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'148","reported_seq":407,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:25.328828+0000","last_change":"2024-04-26T11:24:54.292436+0000","last_active":"2024-04-26T11:30:25.328828+0000","last_peered":"2024-04-26T11:30:25.328828+0000","last_clean":"2024-04-26T11:30:25.328828+0000","last_became_active":"2024-04-26T11:24:54.291921+0000","last_became_peered":"2024-04-26T11:24:54.291921+0000","last_unstale":"2024-04-26T11:30:25.328828+0000","last_undegraded":"2024-04-26T11:30:25.328828+0000","last_fullsized":"2024-04-26T11:30:25.328828+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_clean_scrub_stamp":"2024-04-26T11:21:00.617277+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:11:01.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513834+0000","last_change":"2024-04-26T11:26:29.000905+0000","last_active":"2024-04-26T11:30:23.513834+0000","last_peered":"2024-04-26T11:30:23.513834+0000","last_clean":"2024-04-26T11:30:23.513834+0000","last_became_active":"2024-04-26T11:26:29.000517+0000","last_became_peered":"2024-04-26T11:26:29.000517+0000","last_unstale":"2024-04-26T11:30:23.513834+0000","last_undegraded":"2024-04-26T11:30:23.513834+0000","last_fullsized":"2024-04-26T11:30:23.513834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:38:55.394316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515508+0000","last_change":"2024-04-26T11:26:24.957489+0000","last_active":"2024-04-26T11:30:23.515508+0000","last_peered":"2024-04-26T11:30:23.515508+0000","last_clean":"2024-04-26T11:30:23.515508+0000","last_became_active":"2024-04-26T11:26:24.957151+0000","last_became_peered":"2024-04-26T11:26:24.957151+0000","last_unstale":"2024-04-26T11:30:23.515508+0000","last_undegraded":"2024-04-26T11:30:23.515508+0000","last_fullsized":"2024-04-26T11:30:23.515508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:42:25.343105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698886+0000","last_change":"2024-04-26T11:26:25.086767+0000","last_active":"2024-04-26T11:30:23.698886+0000","last_peered":"2024-04-26T11:30:23.698886+0000","last_clean":"2024-04-26T11:30:23.698886+0000","last_became_active":"2024-04-26T11:26:25.086440+0000","last_became_peered":"2024-04-26T11:26:25.086440+0000","last_unstale":"2024-04-26T11:30:23.698886+0000","last_undegraded":"2024-04-26T11:30:23.698886+0000","last_fullsized":"2024-04-26T11:30:23.698886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:02.294644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514542+0000","last_change":"2024-04-26T11:26:29.003683+0000","last_active":"2024-04-26T11:30:23.514542+0000","last_peered":"2024-04-26T11:30:23.514542+0000","last_clean":"2024-04-26T11:30:23.514542+0000","last_became_active":"2024-04-26T11:26:29.003307+0000","last_became_peered":"2024-04-26T11:26:29.003307+0000","last_unstale":"2024-04-26T11:30:23.514542+0000","last_undegraded":"2024-04-26T11:30:23.514542+0000","last_fullsized":"2024-04-26T11:30:23.514542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:41:00.124540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'102","reported_seq":235,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:25.800292+0000","last_change":"2024-04-26T11:27:19.849965+0000","last_active":"2024-04-26T11:30:25.800292+0000","last_peered":"2024-04-26T11:30:25.800292+0000","last_clean":"2024-04-26T11:30:25.800292+0000","last_became_active":"2024-04-26T11:27:19.840545+0000","last_became_peered":"2024-04-26T11:27:19.840545+0000","last_unstale":"2024-04-26T11:30:25.800292+0000","last_undegraded":"2024-04-26T11:30:25.800292+0000","last_fullsized":"2024-04-26T11:30:25.800292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:08.003704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166953+0000","last_change":"2024-04-26T11:26:29.016783+0000","last_active":"2024-04-26T11:30:23.166953+0000","last_peered":"2024-04-26T11:30:23.166953+0000","last_clean":"2024-04-26T11:30:23.166953+0000","last_became_active":"2024-04-26T11:26:29.013998+0000","last_became_peered":"2024-04-26T11:26:29.013998+0000","last_unstale":"2024-04-26T11:30:23.166953+0000","last_undegraded":"2024-04-26T11:30:23.166953+0000","last_fullsized":"2024-04-26T11:30:23.166953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:24:44.453094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513649+0000","last_change":"2024-04-26T11:26:25.902330+0000","last_active":"2024-04-26T11:30:23.513649+0000","last_peered":"2024-04-26T11:30:23.513649+0000","last_clean":"2024-04-26T11:30:23.513649+0000","last_became_active":"2024-04-26T11:26:25.901948+0000","last_became_peered":"2024-04-26T11:26:25.901948+0000","last_unstale":"2024-04-26T11:30:23.513649+0000","last_undegraded":"2024-04-26T11:30:23.513649+0000","last_fullsized":"2024-04-26T11:30:23.513649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:00:23.541393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518542+0000","last_change":"2024-04-26T11:26:29.025692+0000","last_active":"2024-04-26T11:30:23.518542+0000","last_peered":"2024-04-26T11:30:23.518542+0000","last_clean":"2024-04-26T11:30:23.518542+0000","last_became_active":"2024-04-26T11:26:29.025112+0000","last_became_peered":"2024-04-26T11:26:29.025112+0000","last_unstale":"2024-04-26T11:30:23.518542+0000","last_undegraded":"2024-04-26T11:30:23.518542+0000","last_fullsized":"2024-04-26T11:30:23.518542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:39:22.637865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519849+0000","last_change":"2024-04-26T11:26:25.080270+0000","last_active":"2024-04-26T11:30:23.519849+0000","last_peered":"2024-04-26T11:30:23.519849+0000","last_clean":"2024-04-26T11:30:23.519849+0000","last_became_active":"2024-04-26T11:26:25.079895+0000","last_became_peered":"2024-04-26T11:26:25.079895+0000","last_unstale":"2024-04-26T11:30:23.519849+0000","last_undegraded":"2024-04-26T11:30:23.519849+0000","last_fullsized":"2024-04-26T11:30:23.519849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:47:08.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698571+0000","last_change":"2024-04-26T11:26:28.996540+0000","last_active":"2024-04-26T11:30:23.698571+0000","last_peered":"2024-04-26T11:30:23.698571+0000","last_clean":"2024-04-26T11:30:23.698571+0000","last_became_active":"2024-04-26T11:26:28.994724+0000","last_became_peered":"2024-04-26T11:26:28.994724+0000","last_unstale":"2024-04-26T11:30:23.698571+0000","last_undegraded":"2024-04-26T11:30:23.698571+0000","last_fullsized":"2024-04-26T11:30:23.698571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:13.382332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163378+0000","last_change":"2024-04-26T11:26:24.967935+0000","last_active":"2024-04-26T11:30:23.163378+0000","last_peered":"2024-04-26T11:30:23.163378+0000","last_clean":"2024-04-26T11:30:23.163378+0000","last_became_active":"2024-04-26T11:26:24.966465+0000","last_became_peered":"2024-04-26T11:26:24.966465+0000","last_unstale":"2024-04-26T11:30:23.163378+0000","last_undegraded":"2024-04-26T11:30:23.163378+0000","last_fullsized":"2024-04-26T11:30:23.163378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:27:38.817316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308221+0000","last_change":"2024-04-26T11:26:29.024243+0000","last_active":"2024-04-26T11:30:23.308221+0000","last_peered":"2024-04-26T11:30:23.308221+0000","last_clean":"2024-04-26T11:30:23.308221+0000","last_became_active":"2024-04-26T11:26:29.023900+0000","last_became_peered":"2024-04-26T11:26:29.023900+0000","last_unstale":"2024-04-26T11:30:23.308221+0000","last_undegraded":"2024-04-26T11:30:23.308221+0000","last_fullsized":"2024-04-26T11:30:23.308221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:28:23.446568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165372+0000","last_change":"2024-04-26T11:26:24.969953+0000","last_active":"2024-04-26T11:30:23.165372+0000","last_peered":"2024-04-26T11:30:23.165372+0000","last_clean":"2024-04-26T11:30:23.165372+0000","last_became_active":"2024-04-26T11:26:24.969371+0000","last_became_peered":"2024-04-26T11:26:24.969371+0000","last_unstale":"2024-04-26T11:30:23.165372+0000","last_undegraded":"2024-04-26T11:30:23.165372+0000","last_fullsized":"2024-04-26T11:30:23.165372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:07:24.323833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163379+0000","last_change":"2024-04-26T11:26:29.029771+0000","last_active":"2024-04-26T11:30:23.163379+0000","last_peered":"2024-04-26T11:30:23.163379+0000","last_clean":"2024-04-26T11:30:23.163379+0000","last_became_active":"2024-04-26T11:26:29.029387+0000","last_became_peered":"2024-04-26T11:26:29.029387+0000","last_unstale":"2024-04-26T11:30:23.163379+0000","last_undegraded":"2024-04-26T11:30:23.163379+0000","last_fullsized":"2024-04-26T11:30:23.163379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:40.856935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698580+0000","last_change":"2024-04-26T11:26:24.958608+0000","last_active":"2024-04-26T11:30:23.698580+0000","last_peered":"2024-04-26T11:30:23.698580+0000","last_clean":"2024-04-26T11:30:23.698580+0000","last_became_active":"2024-04-26T11:26:24.958188+0000","last_became_peered":"2024-04-26T11:26:24.958188+0000","last_unstale":"2024-04-26T11:30:23.698580+0000","last_undegraded":"2024-04-26T11:30:23.698580+0000","last_fullsized":"2024-04-26T11:30:23.698580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:31:41.442551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515634+0000","last_change":"2024-04-26T11:26:29.000207+0000","last_active":"2024-04-26T11:30:23.515634+0000","last_peered":"2024-04-26T11:30:23.515634+0000","last_clean":"2024-04-26T11:30:23.515634+0000","last_became_active":"2024-04-26T11:26:28.999894+0000","last_became_peered":"2024-04-26T11:26:28.999894+0000","last_unstale":"2024-04-26T11:30:23.515634+0000","last_undegraded":"2024-04-26T11:30:23.515634+0000","last_fullsized":"2024-04-26T11:30:23.515634+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:52:10.349511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165748+0000","last_change":"2024-04-26T11:26:25.902036+0000","last_active":"2024-04-26T11:30:23.165748+0000","last_peered":"2024-04-26T11:30:23.165748+0000","last_clean":"2024-04-26T11:30:23.165748+0000","last_became_active":"2024-04-26T11:26:25.901567+0000","last_became_peered":"2024-04-26T11:26:25.901567+0000","last_unstale":"2024-04-26T11:30:23.165748+0000","last_undegraded":"2024-04-26T11:30:23.165748+0000","last_fullsized":"2024-04-26T11:30:23.165748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:58.757780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515510+0000","last_change":"2024-04-26T11:26:28.998915+0000","last_active":"2024-04-26T11:30:23.515510+0000","last_peered":"2024-04-26T11:30:23.515510+0000","last_clean":"2024-04-26T11:30:23.515510+0000","last_became_active":"2024-04-26T11:26:28.998180+0000","last_became_peered":"2024-04-26T11:26:28.998180+0000","last_unstale":"2024-04-26T11:30:23.515510+0000","last_undegraded":"2024-04-26T11:30:23.515510+0000","last_fullsized":"2024-04-26T11:30:23.515510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:19.972697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'65","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:25.292250+0000","last_change":"2024-04-26T11:26:24.966029+0000","last_active":"2024-04-26T11:30:25.292250+0000","last_peered":"2024-04-26T11:30:25.292250+0000","last_clean":"2024-04-26T11:30:25.292250+0000","last_became_active":"2024-04-26T11:26:24.965676+0000","last_became_peered":"2024-04-26T11:26:24.965676+0000","last_unstale":"2024-04-26T11:30:25.292250+0000","last_undegraded":"2024-04-26T11:30:25.292250+0000","last_fullsized":"2024-04-26T11:30:25.292250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:18:06.904563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515320+0000","last_change":"2024-04-26T11:26:29.020093+0000","last_active":"2024-04-26T11:30:23.515320+0000","last_peered":"2024-04-26T11:30:23.515320+0000","last_clean":"2024-04-26T11:30:23.515320+0000","last_became_active":"2024-04-26T11:26:29.019690+0000","last_became_peered":"2024-04-26T11:26:29.019690+0000","last_unstale":"2024-04-26T11:30:23.515320+0000","last_undegraded":"2024-04-26T11:30:23.515320+0000","last_fullsized":"2024-04-26T11:30:23.515320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:35:44.501217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515925+0000","last_change":"2024-04-26T11:26:24.966757+0000","last_active":"2024-04-26T11:30:23.515925+0000","last_peered":"2024-04-26T11:30:23.515925+0000","last_clean":"2024-04-26T11:30:23.515925+0000","last_became_active":"2024-04-26T11:26:24.966298+0000","last_became_peered":"2024-04-26T11:26:24.966298+0000","last_unstale":"2024-04-26T11:30:23.515925+0000","last_undegraded":"2024-04-26T11:30:23.515925+0000","last_fullsized":"2024-04-26T11:30:23.515925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:17:40.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515641+0000","last_change":"2024-04-26T11:26:28.997774+0000","last_active":"2024-04-26T11:30:23.515641+0000","last_peered":"2024-04-26T11:30:23.515641+0000","last_clean":"2024-04-26T11:30:23.515641+0000","last_became_active":"2024-04-26T11:26:28.997455+0000","last_became_peered":"2024-04-26T11:26:28.997455+0000","last_unstale":"2024-04-26T11:30:23.515641+0000","last_undegraded":"2024-04-26T11:30:23.515641+0000","last_fullsized":"2024-04-26T11:30:23.515641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:56.097022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308289+0000","last_change":"2024-04-26T11:26:28.999714+0000","last_active":"2024-04-26T11:30:23.308289+0000","last_peered":"2024-04-26T11:30:23.308289+0000","last_clean":"2024-04-26T11:30:23.308289+0000","last_became_active":"2024-04-26T11:26:28.999377+0000","last_became_peered":"2024-04-26T11:26:28.999377+0000","last_unstale":"2024-04-26T11:30:23.308289+0000","last_undegraded":"2024-04-26T11:30:23.308289+0000","last_fullsized":"2024-04-26T11:30:23.308289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:39:59.141556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165347+0000","last_change":"2024-04-26T11:27:19.247154+0000","last_active":"2024-04-26T11:30:23.165347+0000","last_peered":"2024-04-26T11:30:23.165347+0000","last_clean":"2024-04-26T11:30:23.165347+0000","last_became_active":"2024-04-26T11:27:19.246590+0000","last_became_peered":"2024-04-26T11:27:19.246590+0000","last_unstale":"2024-04-26T11:30:23.165347+0000","last_undegraded":"2024-04-26T11:30:23.165347+0000","last_fullsized":"2024-04-26T11:30:23.165347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:12:58.401699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165282+0000","last_change":"2024-04-26T11:26:29.019983+0000","last_active":"2024-04-26T11:30:23.165282+0000","last_peered":"2024-04-26T11:30:23.165282+0000","last_clean":"2024-04-26T11:30:23.165282+0000","last_became_active":"2024-04-26T11:26:29.019536+0000","last_became_peered":"2024-04-26T11:26:29.019536+0000","last_unstale":"2024-04-26T11:30:23.165282+0000","last_undegraded":"2024-04-26T11:30:23.165282+0000","last_fullsized":"2024-04-26T11:30:23.165282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:15:06.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518424+0000","last_change":"2024-04-26T11:26:29.027089+0000","last_active":"2024-04-26T11:30:23.518424+0000","last_peered":"2024-04-26T11:30:23.518424+0000","last_clean":"2024-04-26T11:30:23.518424+0000","last_became_active":"2024-04-26T11:26:29.026030+0000","last_became_peered":"2024-04-26T11:26:29.026030+0000","last_unstale":"2024-04-26T11:30:23.518424+0000","last_undegraded":"2024-04-26T11:30:23.518424+0000","last_fullsized":"2024-04-26T11:30:23.518424+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:24:06.105600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514398+0000","last_change":"2024-04-26T11:26:29.018472+0000","last_active":"2024-04-26T11:30:23.514398+0000","last_peered":"2024-04-26T11:30:23.514398+0000","last_clean":"2024-04-26T11:30:23.514398+0000","last_became_active":"2024-04-26T11:26:29.015947+0000","last_became_peered":"2024-04-26T11:26:29.015947+0000","last_unstale":"2024-04-26T11:30:23.514398+0000","last_undegraded":"2024-04-26T11:30:23.514398+0000","last_fullsized":"2024-04-26T11:30:23.514398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:35:48.082709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308522+0000","last_change":"2024-04-26T11:26:28.997582+0000","last_active":"2024-04-26T11:30:23.308522+0000","last_peered":"2024-04-26T11:30:23.308522+0000","last_clean":"2024-04-26T11:30:23.308522+0000","last_became_active":"2024-04-26T11:26:28.997202+0000","last_became_peered":"2024-04-26T11:26:28.997202+0000","last_unstale":"2024-04-26T11:30:23.308522+0000","last_undegraded":"2024-04-26T11:30:23.308522+0000","last_fullsized":"2024-04-26T11:30:23.308522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:59.776455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165909+0000","last_change":"2024-04-26T11:26:28.999428+0000","last_active":"2024-04-26T11:30:23.165909+0000","last_peered":"2024-04-26T11:30:23.165909+0000","last_clean":"2024-04-26T11:30:23.165909+0000","last_became_active":"2024-04-26T11:26:28.999093+0000","last_became_peered":"2024-04-26T11:26:28.999093+0000","last_unstale":"2024-04-26T11:30:23.165909+0000","last_undegraded":"2024-04-26T11:30:23.165909+0000","last_fullsized":"2024-04-26T11:30:23.165909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:07.222231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166613+0000","last_change":"2024-04-26T11:26:29.019673+0000","last_active":"2024-04-26T11:30:23.166613+0000","last_peered":"2024-04-26T11:30:23.166613+0000","last_clean":"2024-04-26T11:30:23.166613+0000","last_became_active":"2024-04-26T11:26:29.017289+0000","last_became_peered":"2024-04-26T11:26:29.017289+0000","last_unstale":"2024-04-26T11:30:23.166613+0000","last_undegraded":"2024-04-26T11:30:23.166613+0000","last_fullsized":"2024-04-26T11:30:23.166613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:35.848004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317338+0000","last_change":"2024-04-26T11:26:29.017234+0000","last_active":"2024-04-26T11:30:23.317338+0000","last_peered":"2024-04-26T11:30:23.317338+0000","last_clean":"2024-04-26T11:30:23.317338+0000","last_became_active":"2024-04-26T11:26:29.016863+0000","last_became_peered":"2024-04-26T11:26:29.016863+0000","last_unstale":"2024-04-26T11:30:23.317338+0000","last_undegraded":"2024-04-26T11:30:23.317338+0000","last_fullsized":"2024-04-26T11:30:23.317338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:14:31.136286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163192+0000","last_change":"2024-04-26T11:26:29.018122+0000","last_active":"2024-04-26T11:30:23.163192+0000","last_peered":"2024-04-26T11:30:23.163192+0000","last_clean":"2024-04-26T11:30:23.163192+0000","last_became_active":"2024-04-26T11:26:29.016269+0000","last_became_peered":"2024-04-26T11:26:29.016269+0000","last_unstale":"2024-04-26T11:30:23.163192+0000","last_undegraded":"2024-04-26T11:30:23.163192+0000","last_fullsized":"2024-04-26T11:30:23.163192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:22.135086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520050+0000","last_change":"2024-04-26T11:27:19.278596+0000","last_active":"2024-04-26T11:30:23.520050+0000","last_peered":"2024-04-26T11:30:23.520050+0000","last_clean":"2024-04-26T11:30:23.520050+0000","last_became_active":"2024-04-26T11:27:19.278138+0000","last_became_peered":"2024-04-26T11:27:19.278138+0000","last_unstale":"2024-04-26T11:30:23.520050+0000","last_undegraded":"2024-04-26T11:30:23.520050+0000","last_fullsized":"2024-04-26T11:30:23.520050+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:17.038527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514500+0000","last_change":"2024-04-26T11:26:29.026905+0000","last_active":"2024-04-26T11:30:23.514500+0000","last_peered":"2024-04-26T11:30:23.514500+0000","last_clean":"2024-04-26T11:30:23.514500+0000","last_became_active":"2024-04-26T11:26:29.026539+0000","last_became_peered":"2024-04-26T11:26:29.026539+0000","last_unstale":"2024-04-26T11:30:23.514500+0000","last_undegraded":"2024-04-26T11:30:23.514500+0000","last_fullsized":"2024-04-26T11:30:23.514500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:41:12.169466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308385+0000","last_change":"2024-04-26T11:26:28.999953+0000","last_active":"2024-04-26T11:30:23.308385+0000","last_peered":"2024-04-26T11:30:23.308385+0000","last_clean":"2024-04-26T11:30:23.308385+0000","last_became_active":"2024-04-26T11:26:28.998344+0000","last_became_peered":"2024-04-26T11:26:28.998344+0000","last_unstale":"2024-04-26T11:30:23.308385+0000","last_undegraded":"2024-04-26T11:30:23.308385+0000","last_fullsized":"2024-04-26T11:30:23.308385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:30:53.269171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107250,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":798736,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1675,"num_read_kb":1055,"num_write":425,"num_write_kb":2359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2396208,"data_compressed":108945,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383743,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":882140,"data_compressed":27979,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645381,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32344,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712808,"statfs":{"total":95995035648,"available":95961915392,"internally_reserved":0,"allocated":1196032,"data_stored":1481924,"data_compressed":61466,"data_compressed_allocated":81920,"data_compressed_original":929792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808906,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":882376,"data_compressed":27979,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070545,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":489080,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332182,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1082324,"data_compressed":44440,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495708,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":488549,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":560315,"data_compressed":10514,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":489820,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":562056,"data_compressed":10514,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182509,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":1210308,"data_compressed":43181,"data_compressed_allocated":57344,"data_compressed_original":724992,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":490300,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757346,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1114112,"data_stored":1408926,"data_compressed":59642,"data_compressed_allocated":77824,"data_compressed_original":921600,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":71973,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":73714,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330099,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":403097,"data_compressed":17026,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T11:30:33.890 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T11:30:33.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T11:30:34.425 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:34.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:34 smithi033 ceph-mon[27845]: from='client.25129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:34.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:34 smithi033 ceph-mon[27845]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T11:30:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:34 smithi194 ceph-mon[33017]: from='client.25129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:34 smithi194 ceph-mon[33017]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T11:30:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:34 smithi096 ceph-mon[32753]: from='client.25129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:34 smithi096 ceph-mon[32753]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T11:30:36.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 -- 172.21.15.33:0/1035723004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394105640 msgr2=0x7f7394105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:36.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 --2- 172.21.15.33:0/1035723004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394105640 0x7f7394105ac0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7380009990 tx=0x7f7380009ca0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 -- 172.21.15.33:0/1035723004 shutdown_connections 2024-04-26T11:30:36.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 --2- 172.21.15.33:0/1035723004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394106000 0x7f7394112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 --2- 172.21.15.33:0/1035723004 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f739410b640 0x7f739410ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 --2- 172.21.15.33:0/1035723004 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394105640 0x7f7394105ac0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 -- 172.21.15.33:0/1035723004 >> 172.21.15.33:0/1035723004 conn(0x7f7394101130 msgr2=0x7f7394103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:36.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 -- 172.21.15.33:0/1035723004 shutdown_connections 2024-04-26T11:30:36.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.707+0000 7f739c600700 1 -- 172.21.15.33:0/1035723004 wait complete. 2024-04-26T11:30:36.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.708+0000 7f739c600700 1 Processor -- start 2024-04-26T11:30:36.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.708+0000 7f739c600700 1 -- start start 2024-04-26T11:30:36.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f739c600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394105640 0x7f739419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:36.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f739c600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394106000 0x7f739419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:36.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f739c600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f739410b640 0x7f73941a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:36.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f739c600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7394115890 con 0x7f7394106000 2024-04-26T11:30:36.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f739c600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7394115710 con 0x7f739410b640 2024-04-26T11:30:36.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f739c600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7394115a10 con 0x7f7394105640 2024-04-26T11:30:36.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f7399200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394106000 0x7f739419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:36.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f7399c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394105640 0x7f739419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:36.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f7399c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394105640 0x7f739419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:37476/0 (socket says 172.21.15.33:37476) 2024-04-26T11:30:36.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.709+0000 7f7399c00700 1 -- 172.21.15.33:0/1258800090 learned_addr learned my addr 172.21.15.33:0/1258800090 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:36.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.710+0000 7f739a600700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f739410b640 0x7f73941a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:36.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.710+0000 7f7399c00700 1 -- 172.21.15.33:0/1258800090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f739410b640 msgr2=0x7f73941a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:36.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.710+0000 7f7399c00700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f739410b640 0x7f73941a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.710+0000 7f7399c00700 1 -- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394106000 msgr2=0x7f739419c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:36.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.710+0000 7f7399c00700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394106000 0x7f739419c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.710+0000 7f7399c00700 1 -- 172.21.15.33:0/1258800090 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73941a0d20 con 0x7f7394105640 2024-04-26T11:30:36.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.710+0000 7f7399c00700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394105640 0x7f739419bd60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f738800e9a0 tx=0x7f738800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:36.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.711+0000 7f7392000700 1 -- 172.21.15.33:0/1258800090 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738800ca10 con 0x7f7394105640 2024-04-26T11:30:36.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.711+0000 7f7392000700 1 -- 172.21.15.33:0/1258800090 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f738800cbb0 con 0x7f7394105640 2024-04-26T11:30:36.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.711+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73941a1010 con 0x7f7394105640 2024-04-26T11:30:36.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.711+0000 7f7392000700 1 -- 172.21.15.33:0/1258800090 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388018680 con 0x7f7394105640 2024-04-26T11:30:36.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.711+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73941a8890 con 0x7f7394105640 2024-04-26T11:30:36.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.712+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739406a110 con 0x7f7394105640 2024-04-26T11:30:36.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.714+0000 7f7392000700 1 -- 172.21.15.33:0/1258800090 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f7388010430 con 0x7f7394105640 2024-04-26T11:30:36.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:36 smithi033 ceph-mon[27845]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T11:30:36.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.716+0000 7f7392000700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f737c0a3ec0 0x7f737c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:36.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.716+0000 7f7392000700 1 -- 172.21.15.33:0/1258800090 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f73880d0a90 con 0x7f7394105640 2024-04-26T11:30:36.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.716+0000 7f7399200700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f737c0a3ec0 0x7f737c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:36.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.717+0000 7f7399200700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f737c0a3ec0 0x7f737c0a6380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7394103700 tx=0x7f7380005fd0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:36.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.719+0000 7f7392000700 1 -- 172.21.15.33:0/1258800090 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f738809cf70 con 0x7f7394105640 2024-04-26T11:30:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:36 smithi194 ceph-mon[33017]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T11:30:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:36 smithi096 ceph-mon[32753]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T11:30:36.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.945+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 --> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7394063860 con 0x7f737c0a3ec0 2024-04-26T11:30:36.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.954+0000 7f7392000700 1 -- 172.21.15.33:0/1258800090 <== mgr.14931 v2:172.21.15.33:6800/516906866 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258065 (secure 0 0 0) 0x7f7380067030 con 0x7f737c0a3ec0 2024-04-26T11:30:36.959 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f737c0a3ec0 msgr2=0x7f737c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f737c0a3ec0 0x7f737c0a6380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7394103700 tx=0x7f7380005fd0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394105640 msgr2=0x7f739419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394105640 0x7f739419bd60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f738800e9a0 tx=0x7f738800ecb0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 shutdown_connections 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f737c0a3ec0 0x7f737c0a6380 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f739410b640 0x7f73941a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7394105640 0x7f739419bd60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 --2- 172.21.15.33:0/1258800090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394106000 0x7f739419c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:36.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 >> 172.21.15.33:0/1258800090 conn(0x7f7394101130 msgr2=0x7f7394101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:36.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 shutdown_connections 2024-04-26T11:30:36.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:36.960+0000 7f739c600700 1 -- 172.21.15.33:0/1258800090 wait complete. 2024-04-26T11:30:36.964 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-26T11:30:37.876 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-26T11:30:35.192110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389338,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1942,"num_read_kb":1400,"num_write":628,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":962,"ondisk_log_size":962,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357408,"kb_used_data":12372,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584416,"statfs":{"total":1151940427776,"available":1151574441984,"internally_reserved":0,"allocated":12668928,"data_stored":10028118,"data_compressed":320475,"data_compressed_allocated":430080,"data_compressed_original":4190208,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":416,"num_read_kb":478,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.990718"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308540+0000","last_change":"2024-04-26T11:26:29.018818+0000","last_active":"2024-04-26T11:30:23.308540+0000","last_peered":"2024-04-26T11:30:23.308540+0000","last_clean":"2024-04-26T11:30:23.308540+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:30:23.308540+0000","last_undegraded":"2024-04-26T11:30:23.308540+0000","last_fullsized":"2024-04-26T11:30:23.308540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:05.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518402+0000","last_change":"2024-04-26T11:26:29.021338+0000","last_active":"2024-04-26T11:30:23.518402+0000","last_peered":"2024-04-26T11:30:23.518402+0000","last_clean":"2024-04-26T11:30:23.518402+0000","last_became_active":"2024-04-26T11:26:29.021019+0000","last_became_peered":"2024-04-26T11:26:29.021019+0000","last_unstale":"2024-04-26T11:30:23.518402+0000","last_undegraded":"2024-04-26T11:30:23.518402+0000","last_fullsized":"2024-04-26T11:30:23.518402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:52:44.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518696+0000","last_change":"2024-04-26T11:26:29.014166+0000","last_active":"2024-04-26T11:30:23.518696+0000","last_peered":"2024-04-26T11:30:23.518696+0000","last_clean":"2024-04-26T11:30:23.518696+0000","last_became_active":"2024-04-26T11:26:29.013812+0000","last_became_peered":"2024-04-26T11:26:29.013812+0000","last_unstale":"2024-04-26T11:30:23.518696+0000","last_undegraded":"2024-04-26T11:30:23.518696+0000","last_fullsized":"2024-04-26T11:30:23.518696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:08.683633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513524+0000","last_change":"2024-04-26T11:26:28.999249+0000","last_active":"2024-04-26T11:30:23.513524+0000","last_peered":"2024-04-26T11:30:23.513524+0000","last_clean":"2024-04-26T11:30:23.513524+0000","last_became_active":"2024-04-26T11:26:28.998927+0000","last_became_peered":"2024-04-26T11:26:28.998927+0000","last_unstale":"2024-04-26T11:30:23.513524+0000","last_undegraded":"2024-04-26T11:30:23.513524+0000","last_fullsized":"2024-04-26T11:30:23.513524+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:56.674537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698533+0000","last_change":"2024-04-26T11:26:29.017202+0000","last_active":"2024-04-26T11:30:23.698533+0000","last_peered":"2024-04-26T11:30:23.698533+0000","last_clean":"2024-04-26T11:30:23.698533+0000","last_became_active":"2024-04-26T11:26:29.016849+0000","last_became_peered":"2024-04-26T11:26:29.016849+0000","last_unstale":"2024-04-26T11:30:23.698533+0000","last_undegraded":"2024-04-26T11:30:23.698533+0000","last_fullsized":"2024-04-26T11:30:23.698533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:58:22.164118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165326+0000","last_change":"2024-04-26T11:27:19.278294+0000","last_active":"2024-04-26T11:30:23.165326+0000","last_peered":"2024-04-26T11:30:23.165326+0000","last_clean":"2024-04-26T11:30:23.165326+0000","last_became_active":"2024-04-26T11:27:19.277853+0000","last_became_peered":"2024-04-26T11:27:19.277853+0000","last_unstale":"2024-04-26T11:30:23.165326+0000","last_undegraded":"2024-04-26T11:30:23.165326+0000","last_fullsized":"2024-04-26T11:30:23.165326+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:12:18.733166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515577+0000","last_change":"2024-04-26T11:26:28.991406+0000","last_active":"2024-04-26T11:30:23.515577+0000","last_peered":"2024-04-26T11:30:23.515577+0000","last_clean":"2024-04-26T11:30:23.515577+0000","last_became_active":"2024-04-26T11:26:28.989303+0000","last_became_peered":"2024-04-26T11:26:28.989303+0000","last_unstale":"2024-04-26T11:30:23.515577+0000","last_undegraded":"2024-04-26T11:30:23.515577+0000","last_fullsized":"2024-04-26T11:30:23.515577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:42:59.449078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163073+0000","last_change":"2024-04-26T11:26:29.020922+0000","last_active":"2024-04-26T11:30:23.163073+0000","last_peered":"2024-04-26T11:30:23.163073+0000","last_clean":"2024-04-26T11:30:23.163073+0000","last_became_active":"2024-04-26T11:26:29.020576+0000","last_became_peered":"2024-04-26T11:26:29.020576+0000","last_unstale":"2024-04-26T11:30:23.163073+0000","last_undegraded":"2024-04-26T11:30:23.163073+0000","last_fullsized":"2024-04-26T11:30:23.163073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:48.770319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520362+0000","last_change":"2024-04-26T11:26:29.004171+0000","last_active":"2024-04-26T11:30:23.520362+0000","last_peered":"2024-04-26T11:30:23.520362+0000","last_clean":"2024-04-26T11:30:23.520362+0000","last_became_active":"2024-04-26T11:26:29.003816+0000","last_became_peered":"2024-04-26T11:26:29.003816+0000","last_unstale":"2024-04-26T11:30:23.520362+0000","last_undegraded":"2024-04-26T11:30:23.520362+0000","last_fullsized":"2024-04-26T11:30:23.520362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:45:14.807634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165819+0000","last_change":"2024-04-26T11:26:28.999453+0000","last_active":"2024-04-26T11:30:23.165819+0000","last_peered":"2024-04-26T11:30:23.165819+0000","last_clean":"2024-04-26T11:30:23.165819+0000","last_became_active":"2024-04-26T11:26:28.999104+0000","last_became_peered":"2024-04-26T11:26:28.999104+0000","last_unstale":"2024-04-26T11:30:23.165819+0000","last_undegraded":"2024-04-26T11:30:23.165819+0000","last_fullsized":"2024-04-26T11:30:23.165819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:55.408949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515355+0000","last_change":"2024-04-26T11:26:28.999218+0000","last_active":"2024-04-26T11:30:23.515355+0000","last_peered":"2024-04-26T11:30:23.515355+0000","last_clean":"2024-04-26T11:30:23.515355+0000","last_became_active":"2024-04-26T11:26:28.998584+0000","last_became_peered":"2024-04-26T11:26:28.998584+0000","last_unstale":"2024-04-26T11:30:23.515355+0000","last_undegraded":"2024-04-26T11:30:23.515355+0000","last_fullsized":"2024-04-26T11:30:23.515355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:18.040520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513411+0000","last_change":"2024-04-26T11:26:28.997121+0000","last_active":"2024-04-26T11:30:23.513411+0000","last_peered":"2024-04-26T11:30:23.513411+0000","last_clean":"2024-04-26T11:30:23.513411+0000","last_became_active":"2024-04-26T11:26:28.996556+0000","last_became_peered":"2024-04-26T11:26:28.996556+0000","last_unstale":"2024-04-26T11:30:23.513411+0000","last_undegraded":"2024-04-26T11:30:23.513411+0000","last_fullsized":"2024-04-26T11:30:23.513411+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:18:55.251371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514338+0000","last_change":"2024-04-26T11:26:28.999525+0000","last_active":"2024-04-26T11:30:23.514338+0000","last_peered":"2024-04-26T11:30:23.514338+0000","last_clean":"2024-04-26T11:30:23.514338+0000","last_became_active":"2024-04-26T11:26:28.998578+0000","last_became_peered":"2024-04-26T11:26:28.998578+0000","last_unstale":"2024-04-26T11:30:23.514338+0000","last_undegraded":"2024-04-26T11:30:23.514338+0000","last_fullsized":"2024-04-26T11:30:23.514338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:32:35.523804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518043+0000","last_change":"2024-04-26T11:26:29.008427+0000","last_active":"2024-04-26T11:30:23.518043+0000","last_peered":"2024-04-26T11:30:23.518043+0000","last_clean":"2024-04-26T11:30:23.518043+0000","last_became_active":"2024-04-26T11:26:29.008073+0000","last_became_peered":"2024-04-26T11:26:29.008073+0000","last_unstale":"2024-04-26T11:30:23.518043+0000","last_undegraded":"2024-04-26T11:30:23.518043+0000","last_fullsized":"2024-04-26T11:30:23.518043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:03.185941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518029+0000","last_change":"2024-04-26T11:26:29.015136+0000","last_active":"2024-04-26T11:30:23.518029+0000","last_peered":"2024-04-26T11:30:23.518029+0000","last_clean":"2024-04-26T11:30:23.518029+0000","last_became_active":"2024-04-26T11:26:29.014800+0000","last_became_peered":"2024-04-26T11:26:29.014800+0000","last_unstale":"2024-04-26T11:30:23.518029+0000","last_undegraded":"2024-04-26T11:30:23.518029+0000","last_fullsized":"2024-04-26T11:30:23.518029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:28:35.871046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519847+0000","last_change":"2024-04-26T11:26:29.024795+0000","last_active":"2024-04-26T11:30:23.519847+0000","last_peered":"2024-04-26T11:30:23.519847+0000","last_clean":"2024-04-26T11:30:23.519847+0000","last_became_active":"2024-04-26T11:26:29.024430+0000","last_became_peered":"2024-04-26T11:26:29.024430+0000","last_unstale":"2024-04-26T11:30:23.519847+0000","last_undegraded":"2024-04-26T11:30:23.519847+0000","last_fullsized":"2024-04-26T11:30:23.519847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:05:20.172869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308685+0000","last_change":"2024-04-26T11:26:29.018822+0000","last_active":"2024-04-26T11:30:23.308685+0000","last_peered":"2024-04-26T11:30:23.308685+0000","last_clean":"2024-04-26T11:30:23.308685+0000","last_became_active":"2024-04-26T11:26:29.018453+0000","last_became_peered":"2024-04-26T11:26:29.018453+0000","last_unstale":"2024-04-26T11:30:23.308685+0000","last_undegraded":"2024-04-26T11:30:23.308685+0000","last_fullsized":"2024-04-26T11:30:23.308685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:04.475400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306306+0000","last_change":"2024-04-26T11:26:29.001984+0000","last_active":"2024-04-26T11:30:23.306306+0000","last_peered":"2024-04-26T11:30:23.306306+0000","last_clean":"2024-04-26T11:30:23.306306+0000","last_became_active":"2024-04-26T11:26:28.999492+0000","last_became_peered":"2024-04-26T11:26:28.999492+0000","last_unstale":"2024-04-26T11:30:23.306306+0000","last_undegraded":"2024-04-26T11:30:23.306306+0000","last_fullsized":"2024-04-26T11:30:23.306306+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:21.398823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165815+0000","last_change":"2024-04-26T11:26:29.019677+0000","last_active":"2024-04-26T11:30:23.165815+0000","last_peered":"2024-04-26T11:30:23.165815+0000","last_clean":"2024-04-26T11:30:23.165815+0000","last_became_active":"2024-04-26T11:26:29.019315+0000","last_became_peered":"2024-04-26T11:26:29.019315+0000","last_unstale":"2024-04-26T11:30:23.165815+0000","last_undegraded":"2024-04-26T11:30:23.165815+0000","last_fullsized":"2024-04-26T11:30:23.165815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:20.239140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306303+0000","last_change":"2024-04-26T11:26:24.964887+0000","last_active":"2024-04-26T11:30:23.306303+0000","last_peered":"2024-04-26T11:30:23.306303+0000","last_clean":"2024-04-26T11:30:23.306303+0000","last_became_active":"2024-04-26T11:26:24.964380+0000","last_became_peered":"2024-04-26T11:26:24.964380+0000","last_unstale":"2024-04-26T11:30:23.306303+0000","last_undegraded":"2024-04-26T11:30:23.306303+0000","last_fullsized":"2024-04-26T11:30:23.306303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:59:41.347542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306031+0000","last_change":"2024-04-26T11:27:19.244889+0000","last_active":"2024-04-26T11:30:23.306031+0000","last_peered":"2024-04-26T11:30:23.306031+0000","last_clean":"2024-04-26T11:30:23.306031+0000","last_became_active":"2024-04-26T11:27:19.244428+0000","last_became_peered":"2024-04-26T11:27:19.244428+0000","last_unstale":"2024-04-26T11:30:23.306031+0000","last_undegraded":"2024-04-26T11:30:23.306031+0000","last_fullsized":"2024-04-26T11:30:23.306031+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:09:39.366242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166356+0000","last_change":"2024-04-26T11:26:25.082741+0000","last_active":"2024-04-26T11:30:23.166356+0000","last_peered":"2024-04-26T11:30:23.166356+0000","last_clean":"2024-04-26T11:30:23.166356+0000","last_became_active":"2024-04-26T11:26:25.082361+0000","last_became_peered":"2024-04-26T11:26:25.082361+0000","last_unstale":"2024-04-26T11:30:23.166356+0000","last_undegraded":"2024-04-26T11:30:23.166356+0000","last_fullsized":"2024-04-26T11:30:23.166356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:23:17.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519051+0000","last_change":"2024-04-26T11:26:29.014817+0000","last_active":"2024-04-26T11:30:23.519051+0000","last_peered":"2024-04-26T11:30:23.519051+0000","last_clean":"2024-04-26T11:30:23.519051+0000","last_became_active":"2024-04-26T11:26:29.014419+0000","last_became_peered":"2024-04-26T11:26:29.014419+0000","last_unstale":"2024-04-26T11:30:23.519051+0000","last_undegraded":"2024-04-26T11:30:23.519051+0000","last_fullsized":"2024-04-26T11:30:23.519051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:33:28.355075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698995+0000","last_change":"2024-04-26T11:26:24.960224+0000","last_active":"2024-04-26T11:30:23.698995+0000","last_peered":"2024-04-26T11:30:23.698995+0000","last_clean":"2024-04-26T11:30:23.698995+0000","last_became_active":"2024-04-26T11:26:24.959848+0000","last_became_peered":"2024-04-26T11:26:24.959848+0000","last_unstale":"2024-04-26T11:30:23.698995+0000","last_undegraded":"2024-04-26T11:30:23.698995+0000","last_fullsized":"2024-04-26T11:30:23.698995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:49:10.932187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520146+0000","last_change":"2024-04-26T11:26:29.001946+0000","last_active":"2024-04-26T11:30:23.520146+0000","last_peered":"2024-04-26T11:30:23.520146+0000","last_clean":"2024-04-26T11:30:23.520146+0000","last_became_active":"2024-04-26T11:26:29.001579+0000","last_became_peered":"2024-04-26T11:26:29.001579+0000","last_unstale":"2024-04-26T11:30:23.520146+0000","last_undegraded":"2024-04-26T11:30:23.520146+0000","last_fullsized":"2024-04-26T11:30:23.520146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:14:43.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698818+0000","last_change":"2024-04-26T11:26:24.956148+0000","last_active":"2024-04-26T11:30:23.698818+0000","last_peered":"2024-04-26T11:30:23.698818+0000","last_clean":"2024-04-26T11:30:23.698818+0000","last_became_active":"2024-04-26T11:26:24.955736+0000","last_became_peered":"2024-04-26T11:26:24.955736+0000","last_unstale":"2024-04-26T11:30:23.698818+0000","last_undegraded":"2024-04-26T11:30:23.698818+0000","last_fullsized":"2024-04-26T11:30:23.698818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:43.705470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317648+0000","last_change":"2024-04-26T11:27:19.415134+0000","last_active":"2024-04-26T11:30:23.317648+0000","last_peered":"2024-04-26T11:30:23.317648+0000","last_clean":"2024-04-26T11:30:23.317648+0000","last_became_active":"2024-04-26T11:27:19.414703+0000","last_became_peered":"2024-04-26T11:27:19.414703+0000","last_unstale":"2024-04-26T11:30:23.317648+0000","last_undegraded":"2024-04-26T11:30:23.317648+0000","last_fullsized":"2024-04-26T11:30:23.317648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:41:11.268296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":259,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:32.452437+0000","last_change":"2024-04-26T11:26:24.958350+0000","last_active":"2024-04-26T11:30:32.452437+0000","last_peered":"2024-04-26T11:30:32.452437+0000","last_clean":"2024-04-26T11:30:32.452437+0000","last_became_active":"2024-04-26T11:26:24.957382+0000","last_became_peered":"2024-04-26T11:26:24.957382+0000","last_unstale":"2024-04-26T11:30:32.452437+0000","last_undegraded":"2024-04-26T11:30:32.452437+0000","last_fullsized":"2024-04-26T11:30:32.452437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:03.854227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698584+0000","last_change":"2024-04-26T11:26:28.997568+0000","last_active":"2024-04-26T11:30:23.698584+0000","last_peered":"2024-04-26T11:30:23.698584+0000","last_clean":"2024-04-26T11:30:23.698584+0000","last_became_active":"2024-04-26T11:26:28.997161+0000","last_became_peered":"2024-04-26T11:26:28.997161+0000","last_unstale":"2024-04-26T11:30:23.698584+0000","last_undegraded":"2024-04-26T11:30:23.698584+0000","last_fullsized":"2024-04-26T11:30:23.698584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:04.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165161+0000","last_change":"2024-04-26T11:26:24.958093+0000","last_active":"2024-04-26T11:30:23.165161+0000","last_peered":"2024-04-26T11:30:23.165161+0000","last_clean":"2024-04-26T11:30:23.165161+0000","last_became_active":"2024-04-26T11:26:24.957730+0000","last_became_peered":"2024-04-26T11:26:24.957730+0000","last_unstale":"2024-04-26T11:30:23.165161+0000","last_undegraded":"2024-04-26T11:30:23.165161+0000","last_fullsized":"2024-04-26T11:30:23.165161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:09:29.827063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306012+0000","last_change":"2024-04-26T11:27:19.248406+0000","last_active":"2024-04-26T11:30:23.306012+0000","last_peered":"2024-04-26T11:30:23.306012+0000","last_clean":"2024-04-26T11:30:23.306012+0000","last_became_active":"2024-04-26T11:27:19.247980+0000","last_became_peered":"2024-04-26T11:27:19.247980+0000","last_unstale":"2024-04-26T11:30:23.306012+0000","last_undegraded":"2024-04-26T11:30:23.306012+0000","last_fullsized":"2024-04-26T11:30:23.306012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:51:34.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514048+0000","last_change":"2024-04-26T11:26:24.963463+0000","last_active":"2024-04-26T11:30:23.514048+0000","last_peered":"2024-04-26T11:30:23.514048+0000","last_clean":"2024-04-26T11:30:23.514048+0000","last_became_active":"2024-04-26T11:26:24.963030+0000","last_became_peered":"2024-04-26T11:26:24.963030+0000","last_unstale":"2024-04-26T11:30:23.514048+0000","last_undegraded":"2024-04-26T11:30:23.514048+0000","last_fullsized":"2024-04-26T11:30:23.514048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:04:17.844500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163342+0000","last_change":"2024-04-26T11:26:29.013899+0000","last_active":"2024-04-26T11:30:23.163342+0000","last_peered":"2024-04-26T11:30:23.163342+0000","last_clean":"2024-04-26T11:30:23.163342+0000","last_became_active":"2024-04-26T11:26:29.012674+0000","last_became_peered":"2024-04-26T11:26:29.012674+0000","last_unstale":"2024-04-26T11:30:23.163342+0000","last_undegraded":"2024-04-26T11:30:23.163342+0000","last_fullsized":"2024-04-26T11:30:23.163342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:33:46.259841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317199+0000","last_change":"2024-04-26T11:26:24.957803+0000","last_active":"2024-04-26T11:30:23.317199+0000","last_peered":"2024-04-26T11:30:23.317199+0000","last_clean":"2024-04-26T11:30:23.317199+0000","last_became_active":"2024-04-26T11:26:24.957496+0000","last_became_peered":"2024-04-26T11:26:24.957496+0000","last_unstale":"2024-04-26T11:30:23.317199+0000","last_undegraded":"2024-04-26T11:30:23.317199+0000","last_fullsized":"2024-04-26T11:30:23.317199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:15.497106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306343+0000","last_change":"2024-04-26T11:26:29.021648+0000","last_active":"2024-04-26T11:30:23.306343+0000","last_peered":"2024-04-26T11:30:23.306343+0000","last_clean":"2024-04-26T11:30:23.306343+0000","last_became_active":"2024-04-26T11:26:29.020689+0000","last_became_peered":"2024-04-26T11:26:29.020689+0000","last_unstale":"2024-04-26T11:30:23.306343+0000","last_undegraded":"2024-04-26T11:30:23.306343+0000","last_fullsized":"2024-04-26T11:30:23.306343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:15:42.321851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":254,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:32.451958+0000","last_change":"2024-04-26T11:26:24.966184+0000","last_active":"2024-04-26T11:30:32.451958+0000","last_peered":"2024-04-26T11:30:32.451958+0000","last_clean":"2024-04-26T11:30:32.451958+0000","last_became_active":"2024-04-26T11:26:24.965850+0000","last_became_peered":"2024-04-26T11:26:24.965850+0000","last_unstale":"2024-04-26T11:30:32.451958+0000","last_undegraded":"2024-04-26T11:30:32.451958+0000","last_fullsized":"2024-04-26T11:30:32.451958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:49.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":221,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520125+0000","last_change":"2024-04-26T11:26:29.003570+0000","last_active":"2024-04-26T11:30:23.520125+0000","last_peered":"2024-04-26T11:30:23.520125+0000","last_clean":"2024-04-26T11:30:23.520125+0000","last_became_active":"2024-04-26T11:26:29.003039+0000","last_became_peered":"2024-04-26T11:26:29.003039+0000","last_unstale":"2024-04-26T11:30:23.520125+0000","last_undegraded":"2024-04-26T11:30:23.520125+0000","last_fullsized":"2024-04-26T11:30:23.520125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:29:16.166432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515513+0000","last_change":"2024-04-26T11:26:24.954968+0000","last_active":"2024-04-26T11:30:23.515513+0000","last_peered":"2024-04-26T11:30:23.515513+0000","last_clean":"2024-04-26T11:30:23.515513+0000","last_became_active":"2024-04-26T11:26:24.954595+0000","last_became_peered":"2024-04-26T11:26:24.954595+0000","last_unstale":"2024-04-26T11:30:23.515513+0000","last_undegraded":"2024-04-26T11:30:23.515513+0000","last_fullsized":"2024-04-26T11:30:23.515513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:52:21.988942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518513+0000","last_change":"2024-04-26T11:26:29.015601+0000","last_active":"2024-04-26T11:30:23.518513+0000","last_peered":"2024-04-26T11:30:23.518513+0000","last_clean":"2024-04-26T11:30:23.518513+0000","last_became_active":"2024-04-26T11:26:29.015199+0000","last_became_peered":"2024-04-26T11:26:29.015199+0000","last_unstale":"2024-04-26T11:30:23.518513+0000","last_undegraded":"2024-04-26T11:30:23.518513+0000","last_fullsized":"2024-04-26T11:30:23.518513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:15:04.573214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317777+0000","last_change":"2024-04-26T11:26:25.085542+0000","last_active":"2024-04-26T11:30:23.317777+0000","last_peered":"2024-04-26T11:30:23.317777+0000","last_clean":"2024-04-26T11:30:23.317777+0000","last_became_active":"2024-04-26T11:26:25.085230+0000","last_became_peered":"2024-04-26T11:26:25.085230+0000","last_unstale":"2024-04-26T11:30:23.317777+0000","last_undegraded":"2024-04-26T11:30:23.317777+0000","last_fullsized":"2024-04-26T11:30:23.317777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:51.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520229+0000","last_change":"2024-04-26T11:26:29.003954+0000","last_active":"2024-04-26T11:30:23.520229+0000","last_peered":"2024-04-26T11:30:23.520229+0000","last_clean":"2024-04-26T11:30:23.520229+0000","last_became_active":"2024-04-26T11:26:29.003576+0000","last_became_peered":"2024-04-26T11:26:29.003576+0000","last_unstale":"2024-04-26T11:30:23.520229+0000","last_undegraded":"2024-04-26T11:30:23.520229+0000","last_fullsized":"2024-04-26T11:30:23.520229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:49:24.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":521,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:33.453100+0000","last_change":"2024-04-26T11:26:24.957823+0000","last_active":"2024-04-26T11:30:33.453100+0000","last_peered":"2024-04-26T11:30:33.453100+0000","last_clean":"2024-04-26T11:30:33.453100+0000","last_became_active":"2024-04-26T11:26:24.957434+0000","last_became_peered":"2024-04-26T11:26:24.957434+0000","last_unstale":"2024-04-26T11:30:33.453100+0000","last_undegraded":"2024-04-26T11:30:33.453100+0000","last_fullsized":"2024-04-26T11:30:33.453100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:38:43.247282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71973,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":454,"num_read_kb":297,"num_write":39,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698802+0000","last_change":"2024-04-26T11:26:28.994764+0000","last_active":"2024-04-26T11:30:23.698802+0000","last_peered":"2024-04-26T11:30:23.698802+0000","last_clean":"2024-04-26T11:30:23.698802+0000","last_became_active":"2024-04-26T11:26:28.994421+0000","last_became_peered":"2024-04-26T11:26:28.994421+0000","last_unstale":"2024-04-26T11:30:23.698802+0000","last_undegraded":"2024-04-26T11:30:23.698802+0000","last_fullsized":"2024-04-26T11:30:23.698802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:06:06.499434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165530+0000","last_change":"2024-04-26T11:26:24.968036+0000","last_active":"2024-04-26T11:30:23.165530+0000","last_peered":"2024-04-26T11:30:23.165530+0000","last_clean":"2024-04-26T11:30:23.165530+0000","last_became_active":"2024-04-26T11:26:24.967684+0000","last_became_peered":"2024-04-26T11:26:24.967684+0000","last_unstale":"2024-04-26T11:30:23.165530+0000","last_undegraded":"2024-04-26T11:30:23.165530+0000","last_fullsized":"2024-04-26T11:30:23.165530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:00:27.467411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514528+0000","last_change":"2024-04-26T11:26:29.020642+0000","last_active":"2024-04-26T11:30:23.514528+0000","last_peered":"2024-04-26T11:30:23.514528+0000","last_clean":"2024-04-26T11:30:23.514528+0000","last_became_active":"2024-04-26T11:26:29.020278+0000","last_became_peered":"2024-04-26T11:26:29.020278+0000","last_unstale":"2024-04-26T11:30:23.514528+0000","last_undegraded":"2024-04-26T11:30:23.514528+0000","last_fullsized":"2024-04-26T11:30:23.514528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:48.932064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317403+0000","last_change":"2024-04-26T11:26:24.970111+0000","last_active":"2024-04-26T11:30:23.317403+0000","last_peered":"2024-04-26T11:30:23.317403+0000","last_clean":"2024-04-26T11:30:23.317403+0000","last_became_active":"2024-04-26T11:26:24.969629+0000","last_became_peered":"2024-04-26T11:26:24.969629+0000","last_unstale":"2024-04-26T11:30:23.317403+0000","last_undegraded":"2024-04-26T11:30:23.317403+0000","last_fullsized":"2024-04-26T11:30:23.317403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:52:07.859419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515448+0000","last_change":"2024-04-26T11:26:29.002986+0000","last_active":"2024-04-26T11:30:23.515448+0000","last_peered":"2024-04-26T11:30:23.515448+0000","last_clean":"2024-04-26T11:30:23.515448+0000","last_became_active":"2024-04-26T11:26:29.002610+0000","last_became_peered":"2024-04-26T11:26:29.002610+0000","last_unstale":"2024-04-26T11:30:23.515448+0000","last_undegraded":"2024-04-26T11:30:23.515448+0000","last_fullsized":"2024-04-26T11:30:23.515448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:04.877356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":262,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:32.629086+0000","last_change":"2024-04-26T11:26:24.957825+0000","last_active":"2024-04-26T11:30:32.629086+0000","last_peered":"2024-04-26T11:30:32.629086+0000","last_clean":"2024-04-26T11:30:32.629086+0000","last_became_active":"2024-04-26T11:26:24.957464+0000","last_became_peered":"2024-04-26T11:26:24.957464+0000","last_unstale":"2024-04-26T11:30:32.629086+0000","last_undegraded":"2024-04-26T11:30:32.629086+0000","last_fullsized":"2024-04-26T11:30:32.629086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:06:49.350268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":227,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515219+0000","last_change":"2024-04-26T11:26:28.994223+0000","last_active":"2024-04-26T11:30:23.515219+0000","last_peered":"2024-04-26T11:30:23.515219+0000","last_clean":"2024-04-26T11:30:23.515219+0000","last_became_active":"2024-04-26T11:26:28.993388+0000","last_became_peered":"2024-04-26T11:26:28.993388+0000","last_unstale":"2024-04-26T11:30:23.515219+0000","last_undegraded":"2024-04-26T11:30:23.515219+0000","last_fullsized":"2024-04-26T11:30:23.515219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:07:24.208028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698494+0000","last_change":"2024-04-26T11:26:25.078473+0000","last_active":"2024-04-26T11:30:23.698494+0000","last_peered":"2024-04-26T11:30:23.698494+0000","last_clean":"2024-04-26T11:30:23.698494+0000","last_became_active":"2024-04-26T11:26:25.078138+0000","last_became_peered":"2024-04-26T11:26:25.078138+0000","last_unstale":"2024-04-26T11:30:23.698494+0000","last_undegraded":"2024-04-26T11:30:23.698494+0000","last_fullsized":"2024-04-26T11:30:23.698494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:19.692345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317226+0000","last_change":"2024-04-26T11:26:28.998687+0000","last_active":"2024-04-26T11:30:23.317226+0000","last_peered":"2024-04-26T11:30:23.317226+0000","last_clean":"2024-04-26T11:30:23.317226+0000","last_became_active":"2024-04-26T11:26:28.998358+0000","last_became_peered":"2024-04-26T11:26:28.998358+0000","last_unstale":"2024-04-26T11:30:23.317226+0000","last_undegraded":"2024-04-26T11:30:23.317226+0000","last_fullsized":"2024-04-26T11:30:23.317226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:40:51.358188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698939+0000","last_change":"2024-04-26T11:26:24.956213+0000","last_active":"2024-04-26T11:30:23.698939+0000","last_peered":"2024-04-26T11:30:23.698939+0000","last_clean":"2024-04-26T11:30:23.698939+0000","last_became_active":"2024-04-26T11:26:24.955773+0000","last_became_peered":"2024-04-26T11:26:24.955773+0000","last_unstale":"2024-04-26T11:30:23.698939+0000","last_undegraded":"2024-04-26T11:30:23.698939+0000","last_fullsized":"2024-04-26T11:30:23.698939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:38.253367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308369+0000","last_change":"2024-04-26T11:26:29.001389+0000","last_active":"2024-04-26T11:30:23.308369+0000","last_peered":"2024-04-26T11:30:23.308369+0000","last_clean":"2024-04-26T11:30:23.308369+0000","last_became_active":"2024-04-26T11:26:29.001023+0000","last_became_peered":"2024-04-26T11:26:29.001023+0000","last_unstale":"2024-04-26T11:30:23.308369+0000","last_undegraded":"2024-04-26T11:30:23.308369+0000","last_fullsized":"2024-04-26T11:30:23.308369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:57:23.606518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306289+0000","last_change":"2024-04-26T11:27:19.893959+0000","last_active":"2024-04-26T11:30:23.306289+0000","last_peered":"2024-04-26T11:30:23.306289+0000","last_clean":"2024-04-26T11:30:23.306289+0000","last_became_active":"2024-04-26T11:27:19.892819+0000","last_became_peered":"2024-04-26T11:27:19.892819+0000","last_unstale":"2024-04-26T11:30:23.306289+0000","last_undegraded":"2024-04-26T11:30:23.306289+0000","last_fullsized":"2024-04-26T11:30:23.306289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:53:39.829381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165605+0000","last_change":"2024-04-26T11:26:29.019906+0000","last_active":"2024-04-26T11:30:23.165605+0000","last_peered":"2024-04-26T11:30:23.165605+0000","last_clean":"2024-04-26T11:30:23.165605+0000","last_became_active":"2024-04-26T11:26:29.019545+0000","last_became_peered":"2024-04-26T11:26:29.019545+0000","last_unstale":"2024-04-26T11:30:23.165605+0000","last_undegraded":"2024-04-26T11:30:23.165605+0000","last_fullsized":"2024-04-26T11:30:23.165605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:48.617269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515436+0000","last_change":"2024-04-26T11:26:24.956400+0000","last_active":"2024-04-26T11:30:23.515436+0000","last_peered":"2024-04-26T11:30:23.515436+0000","last_clean":"2024-04-26T11:30:23.515436+0000","last_became_active":"2024-04-26T11:26:24.956030+0000","last_became_peered":"2024-04-26T11:26:24.956030+0000","last_unstale":"2024-04-26T11:30:23.515436+0000","last_undegraded":"2024-04-26T11:30:23.515436+0000","last_fullsized":"2024-04-26T11:30:23.515436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:57:47.841006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165227+0000","last_change":"2024-04-26T11:26:29.013087+0000","last_active":"2024-04-26T11:30:23.165227+0000","last_peered":"2024-04-26T11:30:23.165227+0000","last_clean":"2024-04-26T11:30:23.165227+0000","last_became_active":"2024-04-26T11:26:29.012330+0000","last_became_peered":"2024-04-26T11:26:29.012330+0000","last_unstale":"2024-04-26T11:30:23.165227+0000","last_undegraded":"2024-04-26T11:30:23.165227+0000","last_fullsized":"2024-04-26T11:30:23.165227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:12:50.752329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.306001+0000","last_change":"2024-04-26T11:27:19.245410+0000","last_active":"2024-04-26T11:30:23.306001+0000","last_peered":"2024-04-26T11:30:23.306001+0000","last_clean":"2024-04-26T11:30:23.306001+0000","last_became_active":"2024-04-26T11:27:19.244578+0000","last_became_peered":"2024-04-26T11:27:19.244578+0000","last_unstale":"2024-04-26T11:30:23.306001+0000","last_undegraded":"2024-04-26T11:30:23.306001+0000","last_fullsized":"2024-04-26T11:30:23.306001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:07:50.576496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":257,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:33.630848+0000","last_change":"2024-04-26T11:26:25.082517+0000","last_active":"2024-04-26T11:30:33.630848+0000","last_peered":"2024-04-26T11:30:33.630848+0000","last_clean":"2024-04-26T11:30:33.630848+0000","last_became_active":"2024-04-26T11:26:25.081724+0000","last_became_peered":"2024-04-26T11:26:25.081724+0000","last_unstale":"2024-04-26T11:30:33.630848+0000","last_undegraded":"2024-04-26T11:30:33.630848+0000","last_fullsized":"2024-04-26T11:30:33.630848+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:59:40.006615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":221,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.164078+0000","last_change":"2024-04-26T11:26:29.021425+0000","last_active":"2024-04-26T11:30:23.164078+0000","last_peered":"2024-04-26T11:30:23.164078+0000","last_clean":"2024-04-26T11:30:23.164078+0000","last_became_active":"2024-04-26T11:26:29.020881+0000","last_became_peered":"2024-04-26T11:26:29.020881+0000","last_unstale":"2024-04-26T11:30:23.164078+0000","last_undegraded":"2024-04-26T11:30:23.164078+0000","last_fullsized":"2024-04-26T11:30:23.164078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:18.350323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513430+0000","last_change":"2024-04-26T11:26:24.963614+0000","last_active":"2024-04-26T11:30:23.513430+0000","last_peered":"2024-04-26T11:30:23.513430+0000","last_clean":"2024-04-26T11:30:23.513430+0000","last_became_active":"2024-04-26T11:26:24.963312+0000","last_became_peered":"2024-04-26T11:26:24.963312+0000","last_unstale":"2024-04-26T11:30:23.513430+0000","last_undegraded":"2024-04-26T11:30:23.513430+0000","last_fullsized":"2024-04-26T11:30:23.513430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:32:55.886637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'148","reported_seq":407,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:25.328828+0000","last_change":"2024-04-26T11:24:54.292436+0000","last_active":"2024-04-26T11:30:25.328828+0000","last_peered":"2024-04-26T11:30:25.328828+0000","last_clean":"2024-04-26T11:30:25.328828+0000","last_became_active":"2024-04-26T11:24:54.291921+0000","last_became_peered":"2024-04-26T11:24:54.291921+0000","last_unstale":"2024-04-26T11:30:25.328828+0000","last_undegraded":"2024-04-26T11:30:25.328828+0000","last_fullsized":"2024-04-26T11:30:25.328828+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:21:00.617277+0000","last_clean_scrub_stamp":"2024-04-26T11:21:00.617277+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:11:01.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513834+0000","last_change":"2024-04-26T11:26:29.000905+0000","last_active":"2024-04-26T11:30:23.513834+0000","last_peered":"2024-04-26T11:30:23.513834+0000","last_clean":"2024-04-26T11:30:23.513834+0000","last_became_active":"2024-04-26T11:26:29.000517+0000","last_became_peered":"2024-04-26T11:26:29.000517+0000","last_unstale":"2024-04-26T11:30:23.513834+0000","last_undegraded":"2024-04-26T11:30:23.513834+0000","last_fullsized":"2024-04-26T11:30:23.513834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:38:55.394316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515508+0000","last_change":"2024-04-26T11:26:24.957489+0000","last_active":"2024-04-26T11:30:23.515508+0000","last_peered":"2024-04-26T11:30:23.515508+0000","last_clean":"2024-04-26T11:30:23.515508+0000","last_became_active":"2024-04-26T11:26:24.957151+0000","last_became_peered":"2024-04-26T11:26:24.957151+0000","last_unstale":"2024-04-26T11:30:23.515508+0000","last_undegraded":"2024-04-26T11:30:23.515508+0000","last_fullsized":"2024-04-26T11:30:23.515508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:42:25.343105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698886+0000","last_change":"2024-04-26T11:26:25.086767+0000","last_active":"2024-04-26T11:30:23.698886+0000","last_peered":"2024-04-26T11:30:23.698886+0000","last_clean":"2024-04-26T11:30:23.698886+0000","last_became_active":"2024-04-26T11:26:25.086440+0000","last_became_peered":"2024-04-26T11:26:25.086440+0000","last_unstale":"2024-04-26T11:30:23.698886+0000","last_undegraded":"2024-04-26T11:30:23.698886+0000","last_fullsized":"2024-04-26T11:30:23.698886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:02.294644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514542+0000","last_change":"2024-04-26T11:26:29.003683+0000","last_active":"2024-04-26T11:30:23.514542+0000","last_peered":"2024-04-26T11:30:23.514542+0000","last_clean":"2024-04-26T11:30:23.514542+0000","last_became_active":"2024-04-26T11:26:29.003307+0000","last_became_peered":"2024-04-26T11:26:29.003307+0000","last_unstale":"2024-04-26T11:30:23.514542+0000","last_undegraded":"2024-04-26T11:30:23.514542+0000","last_fullsized":"2024-04-26T11:30:23.514542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:41:00.124540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'102","reported_seq":235,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:25.800292+0000","last_change":"2024-04-26T11:27:19.849965+0000","last_active":"2024-04-26T11:30:25.800292+0000","last_peered":"2024-04-26T11:30:25.800292+0000","last_clean":"2024-04-26T11:30:25.800292+0000","last_became_active":"2024-04-26T11:27:19.840545+0000","last_became_peered":"2024-04-26T11:27:19.840545+0000","last_unstale":"2024-04-26T11:30:25.800292+0000","last_undegraded":"2024-04-26T11:30:25.800292+0000","last_fullsized":"2024-04-26T11:30:25.800292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:08.003704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166953+0000","last_change":"2024-04-26T11:26:29.016783+0000","last_active":"2024-04-26T11:30:23.166953+0000","last_peered":"2024-04-26T11:30:23.166953+0000","last_clean":"2024-04-26T11:30:23.166953+0000","last_became_active":"2024-04-26T11:26:29.013998+0000","last_became_peered":"2024-04-26T11:26:29.013998+0000","last_unstale":"2024-04-26T11:30:23.166953+0000","last_undegraded":"2024-04-26T11:30:23.166953+0000","last_fullsized":"2024-04-26T11:30:23.166953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:24:44.453094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.513649+0000","last_change":"2024-04-26T11:26:25.902330+0000","last_active":"2024-04-26T11:30:23.513649+0000","last_peered":"2024-04-26T11:30:23.513649+0000","last_clean":"2024-04-26T11:30:23.513649+0000","last_became_active":"2024-04-26T11:26:25.901948+0000","last_became_peered":"2024-04-26T11:26:25.901948+0000","last_unstale":"2024-04-26T11:30:23.513649+0000","last_undegraded":"2024-04-26T11:30:23.513649+0000","last_fullsized":"2024-04-26T11:30:23.513649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:00:23.541393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518542+0000","last_change":"2024-04-26T11:26:29.025692+0000","last_active":"2024-04-26T11:30:23.518542+0000","last_peered":"2024-04-26T11:30:23.518542+0000","last_clean":"2024-04-26T11:30:23.518542+0000","last_became_active":"2024-04-26T11:26:29.025112+0000","last_became_peered":"2024-04-26T11:26:29.025112+0000","last_unstale":"2024-04-26T11:30:23.518542+0000","last_undegraded":"2024-04-26T11:30:23.518542+0000","last_fullsized":"2024-04-26T11:30:23.518542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:39:22.637865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.519849+0000","last_change":"2024-04-26T11:26:25.080270+0000","last_active":"2024-04-26T11:30:23.519849+0000","last_peered":"2024-04-26T11:30:23.519849+0000","last_clean":"2024-04-26T11:30:23.519849+0000","last_became_active":"2024-04-26T11:26:25.079895+0000","last_became_peered":"2024-04-26T11:26:25.079895+0000","last_unstale":"2024-04-26T11:30:23.519849+0000","last_undegraded":"2024-04-26T11:30:23.519849+0000","last_fullsized":"2024-04-26T11:30:23.519849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:47:08.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698571+0000","last_change":"2024-04-26T11:26:28.996540+0000","last_active":"2024-04-26T11:30:23.698571+0000","last_peered":"2024-04-26T11:30:23.698571+0000","last_clean":"2024-04-26T11:30:23.698571+0000","last_became_active":"2024-04-26T11:26:28.994724+0000","last_became_peered":"2024-04-26T11:26:28.994724+0000","last_unstale":"2024-04-26T11:30:23.698571+0000","last_undegraded":"2024-04-26T11:30:23.698571+0000","last_fullsized":"2024-04-26T11:30:23.698571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:19:13.382332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163378+0000","last_change":"2024-04-26T11:26:24.967935+0000","last_active":"2024-04-26T11:30:23.163378+0000","last_peered":"2024-04-26T11:30:23.163378+0000","last_clean":"2024-04-26T11:30:23.163378+0000","last_became_active":"2024-04-26T11:26:24.966465+0000","last_became_peered":"2024-04-26T11:26:24.966465+0000","last_unstale":"2024-04-26T11:30:23.163378+0000","last_undegraded":"2024-04-26T11:30:23.163378+0000","last_fullsized":"2024-04-26T11:30:23.163378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:27:38.817316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308221+0000","last_change":"2024-04-26T11:26:29.024243+0000","last_active":"2024-04-26T11:30:23.308221+0000","last_peered":"2024-04-26T11:30:23.308221+0000","last_clean":"2024-04-26T11:30:23.308221+0000","last_became_active":"2024-04-26T11:26:29.023900+0000","last_became_peered":"2024-04-26T11:26:29.023900+0000","last_unstale":"2024-04-26T11:30:23.308221+0000","last_undegraded":"2024-04-26T11:30:23.308221+0000","last_fullsized":"2024-04-26T11:30:23.308221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:28:23.446568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165372+0000","last_change":"2024-04-26T11:26:24.969953+0000","last_active":"2024-04-26T11:30:23.165372+0000","last_peered":"2024-04-26T11:30:23.165372+0000","last_clean":"2024-04-26T11:30:23.165372+0000","last_became_active":"2024-04-26T11:26:24.969371+0000","last_became_peered":"2024-04-26T11:26:24.969371+0000","last_unstale":"2024-04-26T11:30:23.165372+0000","last_undegraded":"2024-04-26T11:30:23.165372+0000","last_fullsized":"2024-04-26T11:30:23.165372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:07:24.323833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163379+0000","last_change":"2024-04-26T11:26:29.029771+0000","last_active":"2024-04-26T11:30:23.163379+0000","last_peered":"2024-04-26T11:30:23.163379+0000","last_clean":"2024-04-26T11:30:23.163379+0000","last_became_active":"2024-04-26T11:26:29.029387+0000","last_became_peered":"2024-04-26T11:26:29.029387+0000","last_unstale":"2024-04-26T11:30:23.163379+0000","last_undegraded":"2024-04-26T11:30:23.163379+0000","last_fullsized":"2024-04-26T11:30:23.163379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:40.856935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.698580+0000","last_change":"2024-04-26T11:26:24.958608+0000","last_active":"2024-04-26T11:30:23.698580+0000","last_peered":"2024-04-26T11:30:23.698580+0000","last_clean":"2024-04-26T11:30:23.698580+0000","last_became_active":"2024-04-26T11:26:24.958188+0000","last_became_peered":"2024-04-26T11:26:24.958188+0000","last_unstale":"2024-04-26T11:30:23.698580+0000","last_undegraded":"2024-04-26T11:30:23.698580+0000","last_fullsized":"2024-04-26T11:30:23.698580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:31:41.442551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515634+0000","last_change":"2024-04-26T11:26:29.000207+0000","last_active":"2024-04-26T11:30:23.515634+0000","last_peered":"2024-04-26T11:30:23.515634+0000","last_clean":"2024-04-26T11:30:23.515634+0000","last_became_active":"2024-04-26T11:26:28.999894+0000","last_became_peered":"2024-04-26T11:26:28.999894+0000","last_unstale":"2024-04-26T11:30:23.515634+0000","last_undegraded":"2024-04-26T11:30:23.515634+0000","last_fullsized":"2024-04-26T11:30:23.515634+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:52:10.349511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165748+0000","last_change":"2024-04-26T11:26:25.902036+0000","last_active":"2024-04-26T11:30:23.165748+0000","last_peered":"2024-04-26T11:30:23.165748+0000","last_clean":"2024-04-26T11:30:23.165748+0000","last_became_active":"2024-04-26T11:26:25.901567+0000","last_became_peered":"2024-04-26T11:26:25.901567+0000","last_unstale":"2024-04-26T11:30:23.165748+0000","last_undegraded":"2024-04-26T11:30:23.165748+0000","last_fullsized":"2024-04-26T11:30:23.165748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:58.757780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515510+0000","last_change":"2024-04-26T11:26:28.998915+0000","last_active":"2024-04-26T11:30:23.515510+0000","last_peered":"2024-04-26T11:30:23.515510+0000","last_clean":"2024-04-26T11:30:23.515510+0000","last_became_active":"2024-04-26T11:26:28.998180+0000","last_became_peered":"2024-04-26T11:26:28.998180+0000","last_unstale":"2024-04-26T11:30:23.515510+0000","last_undegraded":"2024-04-26T11:30:23.515510+0000","last_fullsized":"2024-04-26T11:30:23.515510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:12:19.972697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'65","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:25.292250+0000","last_change":"2024-04-26T11:26:24.966029+0000","last_active":"2024-04-26T11:30:25.292250+0000","last_peered":"2024-04-26T11:30:25.292250+0000","last_clean":"2024-04-26T11:30:25.292250+0000","last_became_active":"2024-04-26T11:26:24.965676+0000","last_became_peered":"2024-04-26T11:26:24.965676+0000","last_unstale":"2024-04-26T11:30:25.292250+0000","last_undegraded":"2024-04-26T11:30:25.292250+0000","last_fullsized":"2024-04-26T11:30:25.292250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T17:18:06.904563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515320+0000","last_change":"2024-04-26T11:26:29.020093+0000","last_active":"2024-04-26T11:30:23.515320+0000","last_peered":"2024-04-26T11:30:23.515320+0000","last_clean":"2024-04-26T11:30:23.515320+0000","last_became_active":"2024-04-26T11:26:29.019690+0000","last_became_peered":"2024-04-26T11:26:29.019690+0000","last_unstale":"2024-04-26T11:30:23.515320+0000","last_undegraded":"2024-04-26T11:30:23.515320+0000","last_fullsized":"2024-04-26T11:30:23.515320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:35:44.501217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515925+0000","last_change":"2024-04-26T11:26:24.966757+0000","last_active":"2024-04-26T11:30:23.515925+0000","last_peered":"2024-04-26T11:30:23.515925+0000","last_clean":"2024-04-26T11:30:23.515925+0000","last_became_active":"2024-04-26T11:26:24.966298+0000","last_became_peered":"2024-04-26T11:26:24.966298+0000","last_unstale":"2024-04-26T11:30:23.515925+0000","last_undegraded":"2024-04-26T11:30:23.515925+0000","last_fullsized":"2024-04-26T11:30:23.515925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:23.925253+0000","last_clean_scrub_stamp":"2024-04-26T11:26:23.925253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:17:40.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.515641+0000","last_change":"2024-04-26T11:26:28.997774+0000","last_active":"2024-04-26T11:30:23.515641+0000","last_peered":"2024-04-26T11:30:23.515641+0000","last_clean":"2024-04-26T11:30:23.515641+0000","last_became_active":"2024-04-26T11:26:28.997455+0000","last_became_peered":"2024-04-26T11:26:28.997455+0000","last_unstale":"2024-04-26T11:30:23.515641+0000","last_undegraded":"2024-04-26T11:30:23.515641+0000","last_fullsized":"2024-04-26T11:30:23.515641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:09:56.097022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308289+0000","last_change":"2024-04-26T11:26:28.999714+0000","last_active":"2024-04-26T11:30:23.308289+0000","last_peered":"2024-04-26T11:30:23.308289+0000","last_clean":"2024-04-26T11:30:23.308289+0000","last_became_active":"2024-04-26T11:26:28.999377+0000","last_became_peered":"2024-04-26T11:26:28.999377+0000","last_unstale":"2024-04-26T11:30:23.308289+0000","last_undegraded":"2024-04-26T11:30:23.308289+0000","last_fullsized":"2024-04-26T11:30:23.308289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:39:59.141556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165347+0000","last_change":"2024-04-26T11:27:19.247154+0000","last_active":"2024-04-26T11:30:23.165347+0000","last_peered":"2024-04-26T11:30:23.165347+0000","last_clean":"2024-04-26T11:30:23.165347+0000","last_became_active":"2024-04-26T11:27:19.246590+0000","last_became_peered":"2024-04-26T11:27:19.246590+0000","last_unstale":"2024-04-26T11:30:23.165347+0000","last_undegraded":"2024-04-26T11:30:23.165347+0000","last_fullsized":"2024-04-26T11:30:23.165347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:12:58.401699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165282+0000","last_change":"2024-04-26T11:26:29.019983+0000","last_active":"2024-04-26T11:30:23.165282+0000","last_peered":"2024-04-26T11:30:23.165282+0000","last_clean":"2024-04-26T11:30:23.165282+0000","last_became_active":"2024-04-26T11:26:29.019536+0000","last_became_peered":"2024-04-26T11:26:29.019536+0000","last_unstale":"2024-04-26T11:30:23.165282+0000","last_undegraded":"2024-04-26T11:30:23.165282+0000","last_fullsized":"2024-04-26T11:30:23.165282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:15:06.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.518424+0000","last_change":"2024-04-26T11:26:29.027089+0000","last_active":"2024-04-26T11:30:23.518424+0000","last_peered":"2024-04-26T11:30:23.518424+0000","last_clean":"2024-04-26T11:30:23.518424+0000","last_became_active":"2024-04-26T11:26:29.026030+0000","last_became_peered":"2024-04-26T11:26:29.026030+0000","last_unstale":"2024-04-26T11:30:23.518424+0000","last_undegraded":"2024-04-26T11:30:23.518424+0000","last_fullsized":"2024-04-26T11:30:23.518424+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T15:24:06.105600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514398+0000","last_change":"2024-04-26T11:26:29.018472+0000","last_active":"2024-04-26T11:30:23.514398+0000","last_peered":"2024-04-26T11:30:23.514398+0000","last_clean":"2024-04-26T11:30:23.514398+0000","last_became_active":"2024-04-26T11:26:29.015947+0000","last_became_peered":"2024-04-26T11:26:29.015947+0000","last_unstale":"2024-04-26T11:30:23.514398+0000","last_undegraded":"2024-04-26T11:30:23.514398+0000","last_fullsized":"2024-04-26T11:30:23.514398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:35:48.082709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308522+0000","last_change":"2024-04-26T11:26:28.997582+0000","last_active":"2024-04-26T11:30:23.308522+0000","last_peered":"2024-04-26T11:30:23.308522+0000","last_clean":"2024-04-26T11:30:23.308522+0000","last_became_active":"2024-04-26T11:26:28.997202+0000","last_became_peered":"2024-04-26T11:26:28.997202+0000","last_unstale":"2024-04-26T11:30:23.308522+0000","last_undegraded":"2024-04-26T11:30:23.308522+0000","last_fullsized":"2024-04-26T11:30:23.308522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:59.776455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.165909+0000","last_change":"2024-04-26T11:26:28.999428+0000","last_active":"2024-04-26T11:30:23.165909+0000","last_peered":"2024-04-26T11:30:23.165909+0000","last_clean":"2024-04-26T11:30:23.165909+0000","last_became_active":"2024-04-26T11:26:28.999093+0000","last_became_peered":"2024-04-26T11:26:28.999093+0000","last_unstale":"2024-04-26T11:30:23.165909+0000","last_undegraded":"2024-04-26T11:30:23.165909+0000","last_fullsized":"2024-04-26T11:30:23.165909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:07.222231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.166613+0000","last_change":"2024-04-26T11:26:29.019673+0000","last_active":"2024-04-26T11:30:23.166613+0000","last_peered":"2024-04-26T11:30:23.166613+0000","last_clean":"2024-04-26T11:30:23.166613+0000","last_became_active":"2024-04-26T11:26:29.017289+0000","last_became_peered":"2024-04-26T11:26:29.017289+0000","last_unstale":"2024-04-26T11:30:23.166613+0000","last_undegraded":"2024-04-26T11:30:23.166613+0000","last_fullsized":"2024-04-26T11:30:23.166613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:35.848004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.317338+0000","last_change":"2024-04-26T11:26:29.017234+0000","last_active":"2024-04-26T11:30:23.317338+0000","last_peered":"2024-04-26T11:30:23.317338+0000","last_clean":"2024-04-26T11:30:23.317338+0000","last_became_active":"2024-04-26T11:26:29.016863+0000","last_became_peered":"2024-04-26T11:26:29.016863+0000","last_unstale":"2024-04-26T11:30:23.317338+0000","last_undegraded":"2024-04-26T11:30:23.317338+0000","last_fullsized":"2024-04-26T11:30:23.317338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:14:31.136286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.163192+0000","last_change":"2024-04-26T11:26:29.018122+0000","last_active":"2024-04-26T11:30:23.163192+0000","last_peered":"2024-04-26T11:30:23.163192+0000","last_clean":"2024-04-26T11:30:23.163192+0000","last_became_active":"2024-04-26T11:26:29.016269+0000","last_became_peered":"2024-04-26T11:26:29.016269+0000","last_unstale":"2024-04-26T11:30:23.163192+0000","last_undegraded":"2024-04-26T11:30:23.163192+0000","last_fullsized":"2024-04-26T11:30:23.163192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:33:22.135086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.520050+0000","last_change":"2024-04-26T11:27:19.278596+0000","last_active":"2024-04-26T11:30:23.520050+0000","last_peered":"2024-04-26T11:30:23.520050+0000","last_clean":"2024-04-26T11:30:23.520050+0000","last_became_active":"2024-04-26T11:27:19.278138+0000","last_became_peered":"2024-04-26T11:27:19.278138+0000","last_unstale":"2024-04-26T11:30:23.520050+0000","last_undegraded":"2024-04-26T11:30:23.520050+0000","last_fullsized":"2024-04-26T11:30:23.520050+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:17.038527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.514500+0000","last_change":"2024-04-26T11:26:29.026905+0000","last_active":"2024-04-26T11:30:23.514500+0000","last_peered":"2024-04-26T11:30:23.514500+0000","last_clean":"2024-04-26T11:30:23.514500+0000","last_became_active":"2024-04-26T11:26:29.026539+0000","last_became_peered":"2024-04-26T11:26:29.026539+0000","last_unstale":"2024-04-26T11:30:23.514500+0000","last_undegraded":"2024-04-26T11:30:23.514500+0000","last_fullsized":"2024-04-26T11:30:23.514500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T16:41:12.169466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T11:30:23.308385+0000","last_change":"2024-04-26T11:26:28.999953+0000","last_active":"2024-04-26T11:30:23.308385+0000","last_peered":"2024-04-26T11:30:23.308385+0000","last_clean":"2024-04-26T11:30:23.308385+0000","last_became_active":"2024-04-26T11:26:28.998344+0000","last_became_peered":"2024-04-26T11:26:28.998344+0000","last_unstale":"2024-04-26T11:30:23.308385+0000","last_undegraded":"2024-04-26T11:30:23.308385+0000","last_fullsized":"2024-04-26T11:30:23.308385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T11:26:27.962210+0000","last_clean_scrub_stamp":"2024-04-26T11:26:27.962210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:30:53.269171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107250,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":798736,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1705,"num_read_kb":1070,"num_write":425,"num_write_kb":2359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2396208,"data_compressed":108945,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383744,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":882140,"data_compressed":27979,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645382,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32344,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712808,"statfs":{"total":95995035648,"available":95961915392,"internally_reserved":0,"allocated":1196032,"data_stored":1481924,"data_compressed":61466,"data_compressed_allocated":81920,"data_compressed_original":929792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":882376,"data_compressed":27979,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070546,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":489080,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332183,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1082324,"data_compressed":44440,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495709,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":488549,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":560315,"data_compressed":10514,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":489820,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":562056,"data_compressed":10514,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182510,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":1210308,"data_compressed":43181,"data_compressed_allocated":57344,"data_compressed_original":724992,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920872,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":490300,"data_compressed":8690,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757347,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1114112,"data_stored":1408926,"data_compressed":59642,"data_compressed_allocated":77824,"data_compressed_original":921600,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":71973,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":73714,"data_compressed":1824,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330099,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":403097,"data_compressed":17026,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T11:30:37.880 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:30:38.396 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:38.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:38 smithi033 ceph-mon[27845]: from='client.25135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:38.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:38 smithi033 ceph-mon[27845]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T11:30:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:38 smithi194 ceph-mon[33017]: from='client.25135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:38 smithi194 ceph-mon[33017]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T11:30:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:38 smithi096 ceph-mon[32753]: from='client.25135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T11:30:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:38 smithi096 ceph-mon[32753]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T11:30:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 -- 172.21.15.33:0/1765977253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 msgr2=0x7f8cf8103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 --2- 172.21.15.33:0/1765977253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 0x7f8cf8103cf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009860 tx=0x7f8ce802a3d0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 -- 172.21.15.33:0/1765977253 shutdown_connections 2024-04-26T11:30:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 --2- 172.21.15.33:0/1765977253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cf810cd30 0x7f8cf810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 --2- 172.21.15.33:0/1765977253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf8104230 0x7f8cf810c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 --2- 172.21.15.33:0/1765977253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 0x7f8cf8103cf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 -- 172.21.15.33:0/1765977253 >> 172.21.15.33:0/1765977253 conn(0x7f8cf80fd150 msgr2=0x7f8cf80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 -- 172.21.15.33:0/1765977253 shutdown_connections 2024-04-26T11:30:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.209+0000 7f8cfca00640 1 -- 172.21.15.33:0/1765977253 wait complete. 2024-04-26T11:30:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 Processor -- start 2024-04-26T11:30:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 -- start start 2024-04-26T11:30:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 0x7f8cf819b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cf8104230 0x7f8cf819b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf810cd30 0x7f8cf819f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf8112a30 con 0x7f8cf8103910 2024-04-26T11:30:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81128b0 con 0x7f8cf810cd30 2024-04-26T11:30:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.211+0000 7f8cfca00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf8112bb0 con 0x7f8cf8104230 2024-04-26T11:30:39.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 0x7f8cf819b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 0x7f8cf819b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56746/0 (socket says 172.21.15.33:56746) 2024-04-26T11:30:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cefe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cf8104230 0x7f8cf819b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cf7400640 1 -- 172.21.15.33:0/2046396137 learned_addr learned my addr 172.21.15.33:0/2046396137 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf810cd30 0x7f8cf819f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cefe00640 1 -- 172.21.15.33:0/2046396137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf810cd30 msgr2=0x7f8cf819f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cefe00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf810cd30 0x7f8cf819f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cefe00640 1 -- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 msgr2=0x7f8cf819b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cefe00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 0x7f8cf819b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.212+0000 7f8cefe00640 1 -- 172.21.15.33:0/2046396137 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf81a0120 con 0x7f8cf8104230 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.213+0000 7f8cefe00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cf8104230 0x7f8cf819b670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9c0 tx=0x7f8ce000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.213+0000 7f8cf4c00640 1 -- 172.21.15.33:0/2046396137 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000ccc0 con 0x7f8cf8104230 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.213+0000 7f8cf4c00640 1 -- 172.21.15.33:0/2046396137 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ce00035f0 con 0x7f8cf8104230 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.213+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf81a0410 con 0x7f8cf8104230 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.213+0000 7f8cf4c00640 1 -- 172.21.15.33:0/2046396137 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000ce60 con 0x7f8cf8104230 2024-04-26T11:30:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.213+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf81a7ca0 con 0x7f8cf8104230 2024-04-26T11:30:39.218 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.217+0000 7f8cf4c00640 1 -- 172.21.15.33:0/2046396137 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f8ce0004000 con 0x7f8cf8104230 2024-04-26T11:30:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.217+0000 7f8d00dc4740 1 Processor -- start 2024-04-26T11:30:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8d00dc4740 1 -- start start 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea95243c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea95315b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bea9489380 con 0x55bea9523fe0 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bea9425c30 con 0x55bea9524900 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bea9485340 con 0x55bea952f1c0 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea95243c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8cefe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea95315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea95243c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56758/0 (socket says 172.21.15.33:56758) 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8cf7e00640 1 -- 172.21.15.33:0/3441509851 learned_addr learned my addr 172.21.15.33:0/3441509851 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.218+0000 7f8cf7400640 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf4c00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8cdc0a3e20 0x7f8cdc0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf7400640 1 -- 172.21.15.33:0/3441509851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 msgr2=0x55bea95315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf7400640 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea95315b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf7400640 1 -- 172.21.15.33:0/3441509851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 msgr2=0x55bea95243c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf7400640 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea95243c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf7400640 1 -- 172.21.15.33:0/3441509851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bea9447940 con 0x55bea9524900 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf4c00640 1 -- 172.21.15.33:0/2046396137 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8ce00d0a70 con 0x7f8cf8104230 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8cdc0a3e20 0x7f8cdc0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.219+0000 7f8cf7400640 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952ec80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009830 tx=0x7f8ce8032f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd3f7d7cd194a028 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.220+0000 7f8ce7e00640 1 -- 172.21.15.33:0/3441509851 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8004050 con 0x55bea9524900 2024-04-26T11:30:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.220+0000 7f8ce7e00640 1 -- 172.21.15.33:0/3441509851 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ce8030e30 con 0x55bea9524900 2024-04-26T11:30:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.220+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8cdc0a3e20 0x7f8cdc0a62e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0011a00 tx=0x7f8cf0004ab0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.220+0000 7f8ce7e00640 1 -- 172.21.15.33:0/3441509851 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80055b0 con 0x55bea9524900 2024-04-26T11:30:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3441509851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 msgr2=0x55bea952ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952ec80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009830 tx=0x7f8ce8032f90 comp rx=0 tx=0).stop 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3441509851 shutdown_connections 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea95315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952ec80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3441509851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea95243c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3441509851 >> 172.21.15.33:0/3441509851 conn(0x55bea95204f0 msgr2=0x55bea95208d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3441509851 shutdown_connections 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3441509851 wait complete. 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.221+0000 7f8d00dc4740 1 -- start start 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 1 Processor -- start 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea952c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea952f1c0 0x55bea952cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bea9531d90 con 0x55bea952f1c0 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bea9410c60 con 0x55bea9524900 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bea9531f10 con 0x55bea9523fe0 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.222+0000 7f8d00dc4740 10 client.0 _ll_register_callbacks cb 0x55bea943ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea952f1c0 0x55bea952cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea952c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea952c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:37526/0 (socket says 172.21.15.33:37526) 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 -- 172.21.15.33:0/3978272649 learned_addr learned my addr 172.21.15.33:0/3978272649 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cefe00640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 -- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 msgr2=0x55bea952c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 -- 172.21.15.33:0/3978272649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea952f1c0 msgr2=0x55bea952cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea952f1c0 0x55bea952cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 -- 172.21.15.33:0/3978272649 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bea941f510 con 0x55bea9523fe0 2024-04-26T11:30:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea952f1c0 0x55bea952cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cefe00640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8cf7400640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea952c050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8031000 tx=0x7f8ce8004280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4077a2cd5ae53efb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.223+0000 7f8ce4c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.224+0000 7f8ce4c00640 1 -- 172.21.15.33:0/3978272649 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8055060 con 0x55bea9523fe0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.224+0000 7f8ce4c00640 1 -- 172.21.15.33:0/3978272649 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8ce8030b90 con 0x55bea9523fe0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.224+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce8030e30 con 0x55bea9523fe0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.224+0000 7f8ce4c00640 1 -- 172.21.15.33:0/3978272649 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8ce80044d0 con 0x55bea9523fe0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.224+0000 7f8d00dc4740 10 client.25147 fetch_fsmap learned FSMap version 26 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.224+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bea919b210 con 0x55bea9523fe0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.224+0000 7f8ce4c00640 1 -- 172.21.15.33:0/3978272649 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ce8030e30 con 0x55bea9523fe0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8ce4c00640 1 -- 172.21.15.33:0/3978272649 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8ce80035f0 con 0x55bea9523fe0 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8d00dc4740 10 client.25147 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8d00dc4740 10 client.25147 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8d00dc4740 20 client.25147 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea953ec40 0x55bea9541030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8d00dc4740 4 client.25147 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bea9496f90 con 0x55bea953ec40 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.225+0000 7f8cefe00640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea953ec40 0x55bea9541030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.226+0000 7f8cefe00640 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea953ec40 0x55bea9541030 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98dfafdea7aa5bd1 server_cookie=5c789010ec93bd5d in_seq=0 out_seq=0 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.226+0000 7f8ce4c00640 10 client.25147 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8ce4c00640 1 -- 172.21.15.33:0/3978272649 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bea9496f90 con 0x55bea953ec40 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8ce4c00640 10 client.25147 handle_command_reply: tid=0 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 client.25147 shutdown 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 msgr2=0x55bea952c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea952c050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8031000 tx=0x7f8ce8004280 comp rx=0 tx=0).stop 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 shutdown_connections 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea953ec40 0x55bea9541030 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea952f1c0 0x55bea952cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea952c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/3978272649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea952c050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8ce4c00640 0 client.25147 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 >> 172.21.15.33:0/3978272649 conn(0x55bea95204f0 msgr2=0x55bea9526270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 shutdown_connections 2024-04-26T11:30:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.227+0000 7f8d00dc4740 1 -- 172.21.15.33:0/3978272649 wait complete. 2024-04-26T11:30:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.228+0000 7f8d00dc4740 20 client.25147 trim_cache size 0 max 16384 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 Processor -- start 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 -- start start 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea953ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea95269a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea952b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81b2e70 con 0x55bea9523fe0 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81b0740 con 0x55bea9524900 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.244+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81ae590 con 0x55bea952f1c0 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea953ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea953ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56762/0 (socket says 172.21.15.33:56762) 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 -- 172.21.15.33:0/892654822 learned_addr learned my addr 172.21.15.33:0/892654822 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea952b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cefe00640 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea95269a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 -- 172.21.15.33:0/892654822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 msgr2=0x55bea952b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea952b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 -- 172.21.15.33:0/892654822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 msgr2=0x55bea95269a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea95269a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 -- 172.21.15.33:0/892654822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bea9267620 con 0x55bea9523fe0 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.245+0000 7f8cf7400640 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea953ac50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8003a10 tx=0x7f8ce802aad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1131721145ad54b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8ce7400640 1 -- 172.21.15.33:0/892654822 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80042f0 con 0x55bea9523fe0 2024-04-26T11:30:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8ce7400640 1 -- 172.21.15.33:0/892654822 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ce8004490 con 0x55bea9523fe0 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8ce7400640 1 -- 172.21.15.33:0/892654822 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8030df0 con 0x55bea9523fe0 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 -- 172.21.15.33:0/892654822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 msgr2=0x55bea953ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea953ac50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8003a10 tx=0x7f8ce802aad0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 -- 172.21.15.33:0/892654822 shutdown_connections 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea952f1c0 0x55bea952b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea9524900 0x55bea95269a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/892654822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9523fe0 0x55bea953ac50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 -- 172.21.15.33:0/892654822 >> 172.21.15.33:0/892654822 conn(0x55bea95204f0 msgr2=0x55bea952e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.246+0000 7f8d00dc4740 1 -- 172.21.15.33:0/892654822 shutdown_connections 2024-04-26T11:30:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.247+0000 7f8d00dc4740 1 -- 172.21.15.33:0/892654822 wait complete. 2024-04-26T11:30:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.247+0000 7f8d00dc4740 1 -- start start 2024-04-26T11:30:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.247+0000 7f8d00dc4740 1 Processor -- start 2024-04-26T11:30:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.247+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea9532120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.247+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9524900 0x55bea9532660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.247+0000 7f8d00dc4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea952f1c0 0x55bea9532ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.247+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81aec00 con 0x55bea9524900 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81b0070 con 0x55bea952f1c0 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8d00dc4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf81af2d0 con 0x55bea9523fe0 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8d00dc4740 10 client.0 _ll_register_callbacks cb 0x7f8cf0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea952f1c0 0x55bea9532ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cefe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9524900 0x55bea9532660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea952f1c0 0x55bea9532ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36222/0 (socket says 172.21.15.33:36222) 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 -- 172.21.15.33:0/1734706217 learned_addr learned my addr 172.21.15.33:0/1734706217 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7400640 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea9532120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 -- 172.21.15.33:0/1734706217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 msgr2=0x55bea9532120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea9532120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 -- 172.21.15.33:0/1734706217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9524900 msgr2=0x55bea9532660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9524900 0x55bea9532660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cf7e00640 1 -- 172.21.15.33:0/1734706217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bea9443cb0 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.248+0000 7f8cefe00640 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9524900 0x55bea9532660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8cf7e00640 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea952f1c0 0x55bea9532ba0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000fd90 tx=0x7f8cf0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf0d68ba367ad8a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8ce4c00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8ce4c00640 1 -- 172.21.15.33:0/1734706217 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf001a040 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce8004490 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8ce4c00640 1 -- 172.21.15.33:0/1734706217 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8cf0004b20 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8ce4c00640 1 -- 172.21.15.33:0/1734706217 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8cf0004d20 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8d00dc4740 10 client.25169 fetch_fsmap learned FSMap version 26 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bea95a42c0 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.249+0000 7f8ce4c00640 1 -- 172.21.15.33:0/1734706217 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ce8004490 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8ce4c00640 1 -- 172.21.15.33:0/1734706217 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8cf002a070 con 0x55bea952f1c0 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8d00dc4740 10 client.25169 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8d00dc4740 10 client.25169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:30:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8d00dc4740 20 client.25169 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea95a4ac0 0x55bea95a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8d00dc4740 4 client.25169 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:30:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bea9490e00 con 0x55bea95a4ac0 2024-04-26T11:30:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.250+0000 7f8cf7400640 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea95a4ac0 0x55bea95a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.251+0000 7f8cf7400640 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea95a4ac0 0x55bea95a6ee0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da1d4f640c1c0be server_cookie=673016636f5e1c0 in_seq=0 out_seq=0 2024-04-26T11:30:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.251+0000 7f8ce4c00640 10 client.25169 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8ce4c00640 1 -- 172.21.15.33:0/1734706217 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bea9490e00 con 0x55bea95a4ac0 2024-04-26T11:30:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8ce4c00640 10 client.25169 handle_command_reply: tid=0 2024-04-26T11:30:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 client.25169 shutdown 2024-04-26T11:30:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea952f1c0 msgr2=0x55bea9532ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea952f1c0 0x55bea9532ba0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000fd90 tx=0x7f8cf0004a40 comp rx=0 tx=0).stop 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 shutdown_connections 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55bea95a4ac0 0x55bea95a6ee0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bea952f1c0 0x55bea9532ba0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bea9524900 0x55bea9532660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 --2- 172.21.15.33:0/1734706217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bea9523fe0 0x55bea9532120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 >> 172.21.15.33:0/1734706217 conn(0x55bea95204f0 msgr2=0x55bea952d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8ce4c00640 0 client.25169 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 shutdown_connections 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 1 -- 172.21.15.33:0/1734706217 wait complete. 2024-04-26T11:30:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.263+0000 7f8d00dc4740 20 client.25169 trim_cache size 0 max 16384 2024-04-26T11:30:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:30:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:30:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:30:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:30:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8cdc0a3e20 msgr2=0x7f8cdc0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8cdc0a3e20 0x7f8cdc0a62e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0011a00 tx=0x7f8cf0004ab0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cf8104230 msgr2=0x7f8cf819b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cf8104230 0x7f8cf819b670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9c0 tx=0x7f8ce000ee80 comp rx=0 tx=0).stop 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 shutdown_connections 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8cdc0a3e20 0x7f8cdc0a62e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cf8104230 0x7f8cf819b670 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf810cd30 0x7f8cf819f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 --2- 172.21.15.33:0/2046396137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8cf8103910 0x7f8cf819b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.264+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 >> 172.21.15.33:0/2046396137 conn(0x7f8cf80fd150 msgr2=0x7f8cf8101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.265+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 shutdown_connections 2024-04-26T11:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:39.265+0000 7f8cfca00640 1 -- 172.21.15.33:0/2046396137 wait complete. 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.513+0000 7f2f64a00700 1 -- 172.21.15.33:0/941480882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006b9c0 msgr2=0x7f2f6006be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 --2- 172.21.15.33:0/941480882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006b9c0 0x7f2f6006be20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009990 tx=0x7f2f4c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 -- 172.21.15.33:0/941480882 shutdown_connections 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 --2- 172.21.15.33:0/941480882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f6006c360 0x7f2f601102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 --2- 172.21.15.33:0/941480882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f60109490 0x7f2f60109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 --2- 172.21.15.33:0/941480882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006b9c0 0x7f2f6006be20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 -- 172.21.15.33:0/941480882 >> 172.21.15.33:0/941480882 conn(0x7f2f60079290 msgr2=0x7f2f600796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 -- 172.21.15.33:0/941480882 shutdown_connections 2024-04-26T11:30:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.514+0000 7f2f64a00700 1 -- 172.21.15.33:0/941480882 wait complete. 2024-04-26T11:30:40.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 Processor -- start 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 -- start start 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f6006b9c0 0x7f2f601023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006c360 0x7f2f60102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f60109490 0x7f2f60102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2f60113610 con 0x7f2f6006c360 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2f60113490 con 0x7f2f60109490 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.515+0000 7f2f64a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2f60113790 con 0x7f2f6006b9c0 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006c360 0x7f2f60102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006c360 0x7f2f60102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52776/0 (socket says 172.21.15.33:52776) 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 -- 172.21.15.33:0/1825348703 learned_addr learned my addr 172.21.15.33:0/1825348703 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5de00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f6006b9c0 0x7f2f601023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:40.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5e800700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f60109490 0x7f2f60102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:40.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 -- 172.21.15.33:0/1825348703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f6006b9c0 msgr2=0x7f2f601023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:40.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f6006b9c0 0x7f2f601023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 -- 172.21.15.33:0/1825348703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f60109490 msgr2=0x7f2f60102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f60109490 0x7f2f60102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5d400700 1 -- 172.21.15.33:0/1825348703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f601a8040 con 0x7f2f6006c360 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.516+0000 7f2f5e800700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f60109490 0x7f2f60102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.517+0000 7f2f5d400700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006c360 0x7f2f60102920 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c005910 tx=0x7f2f4c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.517+0000 7f2f56a00700 1 -- 172.21.15.33:0/1825348703 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c027070 con 0x7f2f6006c360 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.517+0000 7f2f56a00700 1 -- 172.21.15.33:0/1825348703 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f4c022470 con 0x7f2f6006c360 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.517+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f601a82d0 con 0x7f2f6006c360 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.517+0000 7f2f56a00700 1 -- 172.21.15.33:0/1825348703 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c017650 con 0x7f2f6006c360 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.517+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f601a8720 con 0x7f2f6006c360 2024-04-26T11:30:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.519+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f6006a110 con 0x7f2f6006c360 2024-04-26T11:30:40.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.521+0000 7f2f56a00700 1 -- 172.21.15.33:0/1825348703 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f2f4c02a070 con 0x7f2f6006c360 2024-04-26T11:30:40.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.523+0000 7f2f56a00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f2f440a3e40 0x7f2f440a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:40.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.523+0000 7f2f56a00700 1 -- 172.21.15.33:0/1825348703 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2f4c0d16b0 con 0x7f2f6006c360 2024-04-26T11:30:40.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.523+0000 7f2f5de00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f2f440a3e40 0x7f2f440a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:40.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.524+0000 7f2f5de00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f2f440a3e40 0x7f2f440a6300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009be0 tx=0x7f2f50009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:40.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.529+0000 7f2f56a00700 1 -- 172.21.15.33:0/1825348703 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f4c013070 con 0x7f2f6006c360 2024-04-26T11:30:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:40 smithi033 ceph-mon[27845]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-26T11:30:40.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.815+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f6002e490 con 0x7f2f6006c360 2024-04-26T11:30:40.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.817+0000 7f2f56a00700 1 -- 172.21.15.33:0/1825348703 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f2f4c099070 con 0x7f2f6006c360 2024-04-26T11:30:40.819 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:40.820 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:30:23.037562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24488,24527]},"id":1}]} 2024-04-26T11:30:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f2f440a3e40 msgr2=0x7f2f440a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:40.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f2f440a3e40 0x7f2f440a6300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009be0 tx=0x7f2f50009260 comp rx=0 tx=0).stop 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006c360 msgr2=0x7f2f60102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006c360 0x7f2f60102920 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c005910 tx=0x7f2f4c00b670 comp rx=0 tx=0).stop 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 shutdown_connections 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f2f440a3e40 0x7f2f440a6300 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f60109490 0x7f2f60102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f6006b9c0 0x7f2f601023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 --2- 172.21.15.33:0/1825348703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2f6006c360 0x7f2f60102920 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.820+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 >> 172.21.15.33:0/1825348703 conn(0x7f2f60079290 msgr2=0x7f2f60100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.821+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 shutdown_connections 2024-04-26T11:30:40.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:40.821+0000 7f2f64a00700 1 -- 172.21.15.33:0/1825348703 wait complete. 2024-04-26T11:30:40.824 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-04-26T11:30:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:40 smithi194 ceph-mon[33017]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-26T11:30:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:40 smithi096 ceph-mon[32753]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-26T11:30:41.740 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:41 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1825348703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:41.745 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:30:41.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:41 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1825348703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:41 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1825348703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:42.216 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:42 smithi033 ceph-mon[27845]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 37 KiB/s wr, 33 op/s 2024-04-26T11:30:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:42 smithi194 ceph-mon[33017]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 37 KiB/s wr, 33 op/s 2024-04-26T11:30:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:42 smithi096 ceph-mon[32753]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 37 KiB/s wr, 33 op/s 2024-04-26T11:30:44.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 -- 172.21.15.33:0/2821928151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c10b630 msgr2=0x7f248c10ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 --2- 172.21.15.33:0/2821928151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c10b630 0x7f248c10ba10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f247c009990 tx=0x7f247c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 -- 172.21.15.33:0/2821928151 shutdown_connections 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 --2- 172.21.15.33:0/2821928151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c105ff0 0x7f248c112500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 --2- 172.21.15.33:0/2821928151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c10b630 0x7f248c10ba10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 --2- 172.21.15.33:0/2821928151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 0x7f248c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 -- 172.21.15.33:0/2821928151 >> 172.21.15.33:0/2821928151 conn(0x7f248c101160 msgr2=0x7f248c103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 -- 172.21.15.33:0/2821928151 shutdown_connections 2024-04-26T11:30:44.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.417+0000 7f2491400700 1 -- 172.21.15.33:0/2821928151 wait complete. 2024-04-26T11:30:44.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.419+0000 7f2491400700 1 Processor -- start 2024-04-26T11:30:44.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.419+0000 7f2491400700 1 -- start start 2024-04-26T11:30:44.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f2491400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 0x7f248c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:44.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f2491400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c105ff0 0x7f248c19c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:44.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f2491400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c10b630 0x7f248c1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f2491400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f248c115860 con 0x7f248c10b630 2024-04-26T11:30:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f2491400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1156e0 con 0x7f248c105630 2024-04-26T11:30:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f2491400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1159e0 con 0x7f248c105ff0 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f248a800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 0x7f248c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f248b200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c10b630 0x7f248c1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f248b200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c10b630 0x7f248c1a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52788/0 (socket says 172.21.15.33:52788) 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f248b200700 1 -- 172.21.15.33:0/3035488632 learned_addr learned my addr 172.21.15.33:0/3035488632 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f2489e00700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c105ff0 0x7f248c19c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.420+0000 7f248a800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 0x7f248c19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45344/0 (socket says 172.21.15.33:45344) 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2489e00700 1 -- 172.21.15.33:0/3035488632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 msgr2=0x7f248c19bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2489e00700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 0x7f248c19bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2489e00700 1 -- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c10b630 msgr2=0x7f248c1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2489e00700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c10b630 0x7f248c1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2489e00700 1 -- 172.21.15.33:0/3035488632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248c1a0d40 con 0x7f248c105ff0 2024-04-26T11:30:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f248a800700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 0x7f248c19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f248b200700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c10b630 0x7f248c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2489e00700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c105ff0 0x7f248c19c270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f247800e9f0 tx=0x7f247800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2483400700 1 -- 172.21.15.33:0/3035488632 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247800c9c0 con 0x7f248c105ff0 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2483400700 1 -- 172.21.15.33:0/3035488632 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f247800cb60 con 0x7f248c105ff0 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248c1a1030 con 0x7f248c105ff0 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.421+0000 7f2483400700 1 -- 172.21.15.33:0/3035488632 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478018680 con 0x7f248c105ff0 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.422+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248c1a88b0 con 0x7f248c105ff0 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.423+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f248c10dbc0 con 0x7f248c105ff0 2024-04-26T11:30:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.426+0000 7f2483400700 1 -- 172.21.15.33:0/3035488632 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f2478010430 con 0x7f248c105ff0 2024-04-26T11:30:44.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.427+0000 7f2483400700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f24700a3e40 0x7f24700a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:44.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.427+0000 7f2483400700 1 -- 172.21.15.33:0/3035488632 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f24780d0e70 con 0x7f248c105ff0 2024-04-26T11:30:44.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.427+0000 7f248a800700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f24700a3e40 0x7f24700a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:44.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.428+0000 7f248a800700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f24700a3e40 0x7f24700a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f247c006010 tx=0x7f247c00b260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:44.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.429+0000 7f2483400700 1 -- 172.21.15.33:0/3035488632 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f247809d350 con 0x7f248c105ff0 2024-04-26T11:30:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:44 smithi033 ceph-mon[27845]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:30:44.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.713+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f248c06a110 con 0x7f248c105ff0 2024-04-26T11:30:44.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.715+0000 7f2483400700 1 -- 172.21.15.33:0/3035488632 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f247809cc50 con 0x7f248c105ff0 2024-04-26T11:30:44.717 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:44.717 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:30:23.037562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24488,24527]},"id":1}]} 2024-04-26T11:30:44.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f24700a3e40 msgr2=0x7f24700a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:44.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f24700a3e40 0x7f24700a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f247c006010 tx=0x7f247c00b260 comp rx=0 tx=0).stop 2024-04-26T11:30:44.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c105ff0 msgr2=0x7f248c19c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:44.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c105ff0 0x7f248c19c270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f247800e9f0 tx=0x7f247800ed00 comp rx=0 tx=0).stop 2024-04-26T11:30:44.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 shutdown_connections 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f24700a3e40 0x7f24700a6300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f248c10b630 0x7f248c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248c105630 0x7f248c19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.718+0000 7f2491400700 1 --2- 172.21.15.33:0/3035488632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f248c105ff0 0x7f248c19c270 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.719+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 >> 172.21.15.33:0/3035488632 conn(0x7f248c101160 msgr2=0x7f248c101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.719+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 shutdown_connections 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:44.719+0000 7f2491400700 1 -- 172.21.15.33:0/3035488632 wait complete. 2024-04-26T11:30:44.721 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-04-26T11:30:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:44 smithi194 ceph-mon[33017]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:30:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:44 smithi096 ceph-mon[32753]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:30:45.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T11:30:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:45 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3035488632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:45.975 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:45 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3035488632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:45.975 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:45 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3035488632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:46.053 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:46 smithi194 ceph-mon[33017]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-26T11:30:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:46 smithi033 ceph-mon[27845]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-26T11:30:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:46 smithi096 ceph-mon[32753]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 -- 172.21.15.33:0/3056539192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 msgr2=0x7fc794113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 --2- 172.21.15.33:0/3056539192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 0x7fc794113d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc780009990 tx=0x7fc780009ca0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 -- 172.21.15.33:0/3056539192 shutdown_connections 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 --2- 172.21.15.33:0/3056539192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 0x7fc7941166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 --2- 172.21.15.33:0/3056539192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc794077950 0x7fc794077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 --2- 172.21.15.33:0/3056539192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 0x7fc794113d90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 -- 172.21.15.33:0/3056539192 >> 172.21.15.33:0/3056539192 conn(0x7fc794101120 msgr2=0x7fc794103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 -- 172.21.15.33:0/3056539192 shutdown_connections 2024-04-26T11:30:48.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.334+0000 7fc79a000700 1 -- 172.21.15.33:0/3056539192 wait complete. 2024-04-26T11:30:48.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.335+0000 7fc79a000700 1 Processor -- start 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.335+0000 7fc79a000700 1 -- start start 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc79a000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc794077950 0x7fc794113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc79a000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 0x7fc794113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc79a000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 0x7fc79410cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc79a000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc794119cb0 con 0x7fc794078270 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc79a000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc794119b30 con 0x7fc794077950 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc79a000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc794119e30 con 0x7fc7941142d0 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc793e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 0x7fc79410cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc793e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 0x7fc79410cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:60086/0 (socket says 172.21.15.33:60086) 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc793e00700 1 -- 172.21.15.33:0/4092336453 learned_addr learned my addr 172.21.15.33:0/4092336453 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.336+0000 7fc793400700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc794077950 0x7fc794113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc792a00700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 0x7fc794113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:48.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc793e00700 1 -- 172.21.15.33:0/4092336453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc794077950 msgr2=0x7fc794113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:48.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc793e00700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc794077950 0x7fc794113850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc793e00700 1 -- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 msgr2=0x7fc794113d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:48.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc793e00700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 0x7fc794113d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc793e00700 1 -- 172.21.15.33:0/4092336453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79410d430 con 0x7fc7941142d0 2024-04-26T11:30:48.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc792a00700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 0x7fc794113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:30:48.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.337+0000 7fc793e00700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 0x7fc79410cb30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009fd0 tx=0x7fc78c00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:48.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.338+0000 7fc78be00700 1 -- 172.21.15.33:0/4092336453 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0092e0 con 0x7fc7941142d0 2024-04-26T11:30:48.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.338+0000 7fc78be00700 1 -- 172.21.15.33:0/4092336453 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc78c004bb0 con 0x7fc7941142d0 2024-04-26T11:30:48.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.338+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79410d6c0 con 0x7fc7941142d0 2024-04-26T11:30:48.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.338+0000 7fc78be00700 1 -- 172.21.15.33:0/4092336453 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c004280 con 0x7fc7941142d0 2024-04-26T11:30:48.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.338+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc794111660 con 0x7fc7941142d0 2024-04-26T11:30:48.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.339+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc794079090 con 0x7fc7941142d0 2024-04-26T11:30:48.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.341+0000 7fc78be00700 1 -- 172.21.15.33:0/4092336453 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fc78c01b030 con 0x7fc7941142d0 2024-04-26T11:30:48.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.343+0000 7fc78be00700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7780a3ec0 0x7fc7780a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:48.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.343+0000 7fc78be00700 1 -- 172.21.15.33:0/4092336453 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc78c0d0450 con 0x7fc7941142d0 2024-04-26T11:30:48.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.343+0000 7fc793400700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7780a3ec0 0x7fc7780a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:48.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.344+0000 7fc793400700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7780a3ec0 0x7fc7780a6380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc784009680 tx=0x7fc784009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:48.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.347+0000 7fc78be00700 1 -- 172.21.15.33:0/4092336453 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc78c022020 con 0x7fc7941142d0 2024-04-26T11:30:48.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.582+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc794112130 con 0x7fc7941142d0 2024-04-26T11:30:48.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.584+0000 7fc78be00700 1 -- 172.21.15.33:0/4092336453 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22968 (secure 0 0 0) 0x7fc78c010040 con 0x7fc7941142d0 2024-04-26T11:30:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:48.587 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":91,"fsid":"5cc7f146-03be-11ef-bc93-c7b262605968","created":"2024-04-26T11:17:08.866250+0000","modified":"2024-04-26T11:30:23.152277+0000","last_up_change":"2024-04-26T11:25:15.355440+0000","last_in_change":"2024-04-26T11:24:56.007797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T11:20:59.833195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T11:26:23.006236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T11:26:27.948404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1df23-f689-46b9-a67c-a8846bd7c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6803","nonce":2032124728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6805","nonce":2032124728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6809","nonce":2032124728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2032124728},{"type":"v1","addr":"172.21.15.33:6807","nonce":2032124728}]},"public_addr":"172.21.15.33:6803/2032124728","cluster_addr":"172.21.15.33:6805/2032124728","heartbeat_back_addr":"172.21.15.33:6809/2032124728","heartbeat_front_addr":"172.21.15.33:6807/2032124728","state":["exists","up"]},{"osd":1,"uuid":"8d3789d6-b7e0-41b5-b81f-5f2f2492d663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6801","nonce":2891673076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6803","nonce":2891673076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6807","nonce":2891673076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2891673076},{"type":"v1","addr":"172.21.15.96:6805","nonce":2891673076}]},"public_addr":"172.21.15.96:6801/2891673076","cluster_addr":"172.21.15.96:6803/2891673076","heartbeat_back_addr":"172.21.15.96:6807/2891673076","heartbeat_front_addr":"172.21.15.96:6805/2891673076","state":["exists","up"]},{"osd":2,"uuid":"05e220c5-ccfa-45ed-97a6-d2281a055fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6801","nonce":3454447938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6803","nonce":3454447938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6807","nonce":3454447938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3454447938},{"type":"v1","addr":"172.21.15.194:6805","nonce":3454447938}]},"public_addr":"172.21.15.194:6801/3454447938","cluster_addr":"172.21.15.194:6803/3454447938","heartbeat_back_addr":"172.21.15.194:6807/3454447938","heartbeat_front_addr":"172.21.15.194:6805/3454447938","state":["exists","up"]},{"osd":3,"uuid":"f814ef05-a751-446c-b5c9-319aa93738c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6811","nonce":2572626709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6813","nonce":2572626709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6817","nonce":2572626709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":2572626709},{"type":"v1","addr":"172.21.15.33:6815","nonce":2572626709}]},"public_addr":"172.21.15.33:6811/2572626709","cluster_addr":"172.21.15.33:6813/2572626709","heartbeat_back_addr":"172.21.15.33:6817/2572626709","heartbeat_front_addr":"172.21.15.33:6815/2572626709","state":["exists","up"]},{"osd":4,"uuid":"f5f7754c-be64-4277-8ce2-6d3117800d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6809","nonce":237079136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6811","nonce":237079136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6815","nonce":237079136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":237079136},{"type":"v1","addr":"172.21.15.96:6813","nonce":237079136}]},"public_addr":"172.21.15.96:6809/237079136","cluster_addr":"172.21.15.96:6811/237079136","heartbeat_back_addr":"172.21.15.96:6815/237079136","heartbeat_front_addr":"172.21.15.96:6813/237079136","state":["exists","up"]},{"osd":5,"uuid":"a138a4e3-39a6-4646-9eb8-27184d6dda61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6809","nonce":769884619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6811","nonce":769884619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6815","nonce":769884619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":769884619},{"type":"v1","addr":"172.21.15.194:6813","nonce":769884619}]},"public_addr":"172.21.15.194:6809/769884619","cluster_addr":"172.21.15.194:6811/769884619","heartbeat_back_addr":"172.21.15.194:6815/769884619","heartbeat_front_addr":"172.21.15.194:6813/769884619","state":["exists","up"]},{"osd":6,"uuid":"79871b86-d637-4952-b923-615080fad108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6819","nonce":3152193593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6821","nonce":3152193593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6825","nonce":3152193593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3152193593},{"type":"v1","addr":"172.21.15.33:6823","nonce":3152193593}]},"public_addr":"172.21.15.33:6819/3152193593","cluster_addr":"172.21.15.33:6821/3152193593","heartbeat_back_addr":"172.21.15.33:6825/3152193593","heartbeat_front_addr":"172.21.15.33:6823/3152193593","state":["exists","up"]},{"osd":7,"uuid":"514a0fd5-1191-4c6c-8724-d13762ffb9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6817","nonce":1631656381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6819","nonce":1631656381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6823","nonce":1631656381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1631656381},{"type":"v1","addr":"172.21.15.96:6821","nonce":1631656381}]},"public_addr":"172.21.15.96:6817/1631656381","cluster_addr":"172.21.15.96:6819/1631656381","heartbeat_back_addr":"172.21.15.96:6823/1631656381","heartbeat_front_addr":"172.21.15.96:6821/1631656381","state":["exists","up"]},{"osd":8,"uuid":"6b44bf32-22e5-4c02-8be5-79db01d9f8a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6817","nonce":529571686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6819","nonce":529571686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6823","nonce":529571686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":529571686},{"type":"v1","addr":"172.21.15.194:6821","nonce":529571686}]},"public_addr":"172.21.15.194:6817/529571686","cluster_addr":"172.21.15.194:6819/529571686","heartbeat_back_addr":"172.21.15.194:6823/529571686","heartbeat_front_addr":"172.21.15.194:6821/529571686","state":["exists","up"]},{"osd":9,"uuid":"9a0a3c0f-0405-46c7-bff5-886d3232f60c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6827","nonce":1639452107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6829","nonce":1639452107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6833","nonce":1639452107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1639452107},{"type":"v1","addr":"172.21.15.33:6831","nonce":1639452107}]},"public_addr":"172.21.15.33:6827/1639452107","cluster_addr":"172.21.15.33:6829/1639452107","heartbeat_back_addr":"172.21.15.33:6833/1639452107","heartbeat_front_addr":"172.21.15.33:6831/1639452107","state":["exists","up"]},{"osd":10,"uuid":"9a7b1104-145f-4393-bf38-a75dcaf06b8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6825","nonce":2356422092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6827","nonce":2356422092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6831","nonce":2356422092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2356422092},{"type":"v1","addr":"172.21.15.96:6829","nonce":2356422092}]},"public_addr":"172.21.15.96:6825/2356422092","cluster_addr":"172.21.15.96:6827/2356422092","heartbeat_back_addr":"172.21.15.96:6831/2356422092","heartbeat_front_addr":"172.21.15.96:6829/2356422092","state":["exists","up"]},{"osd":11,"uuid":"58375031-d243-4316-9c22-1edf99890668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6825","nonce":2370303582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6827","nonce":2370303582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6831","nonce":2370303582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2370303582},{"type":"v1","addr":"172.21.15.194:6829","nonce":2370303582}]},"public_addr":"172.21.15.194:6825/2370303582","cluster_addr":"172.21.15.194:6827/2370303582","heartbeat_back_addr":"172.21.15.194:6831/2370303582","heartbeat_front_addr":"172.21.15.194:6829/2370303582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:08.907094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:33.729282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:20:56.816176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:29.820861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:21:55.944929+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:21.033650+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:22:55.571901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:21.464933+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:23:47.109865+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:21.120797+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:24:48.260455+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T11:25:13.846036+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:0/1763622610":"2024-04-27T11:30:23.152235+0000","172.21.15.33:0/3707727076":"2024-04-27T11:30:23.152235+0000","172.21.15.33:0/1149083719":"2024-04-27T11:30:23.152235+0000","172.21.15.33:6801/657098496":"2024-04-27T11:30:23.152235+0000","172.21.15.33:0/2640535669":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/580146805":"2024-04-27T11:18:17.660858+0000","172.21.15.33:6801/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/1152006795":"2024-04-27T11:30:23.152235+0000","172.21.15.33:6800/657098496":"2024-04-27T11:30:23.152235+0000","172.21.15.33:6800/334919779":"2024-04-27T11:18:17.660858+0000","172.21.15.33:0/2936895268":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/2839525448":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1725605750":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6801/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:6800/927259861":"2024-04-27T11:17:42.466956+0000","172.21.15.33:0/1539238418":"2024-04-27T11:18:17.660858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T11:30:48.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:48 smithi033 ceph-mon[27845]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7780a3ec0 msgr2=0x7fc7780a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7780a3ec0 0x7fc7780a6380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc784009680 tx=0x7fc784009260 comp rx=0 tx=0).stop 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 msgr2=0x7fc79410cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 0x7fc79410cb30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009fd0 tx=0x7fc78c00ef90 comp rx=0 tx=0).stop 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 shutdown_connections 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7780a3ec0 0x7fc7780a6380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7941142d0 0x7fc79410cb30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc794077950 0x7fc794113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 --2- 172.21.15.33:0/4092336453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc794078270 0x7fc794113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 >> 172.21.15.33:0/4092336453 conn(0x7fc794101120 msgr2=0x7fc794103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:48.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 shutdown_connections 2024-04-26T11:30:48.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:48.589+0000 7fc79a000700 1 -- 172.21.15.33:0/4092336453 wait complete. 2024-04-26T11:30:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:48 smithi194 ceph-mon[33017]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:48 smithi096 ceph-mon[32753]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:49.507 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-04-26T11:30:49.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:49 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/4092336453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:30:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:49 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/4092336453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:30:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:49 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/4092336453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T11:30:49.949 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:50 smithi194 ceph-mon[33017]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-26T11:30:50.891 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:50 smithi033 ceph-mon[27845]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-26T11:30:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:50 smithi096 ceph-mon[32753]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 -- 172.21.15.33:0/868852377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85541016e0 msgr2=0x7f8554101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 --2- 172.21.15.33:0/868852377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85541016e0 0x7f8554101ac0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8544009900 tx=0x7f854402a470 comp rx=0 tx=0).stop 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 -- 172.21.15.33:0/868852377 shutdown_connections 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 --2- 172.21.15.33:0/868852377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8554106ae0 0x7f8554106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 --2- 172.21.15.33:0/868852377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 0x7f85541064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 --2- 172.21.15.33:0/868852377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85541016e0 0x7f8554101ac0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 -- 172.21.15.33:0/868852377 >> 172.21.15.33:0/868852377 conn(0x7f85540fd150 msgr2=0x7f85540ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 -- 172.21.15.33:0/868852377 shutdown_connections 2024-04-26T11:30:52.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.417+0000 7f8553200640 1 -- 172.21.15.33:0/868852377 wait complete. 2024-04-26T11:30:52.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.418+0000 7f8553200640 1 Processor -- start 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.418+0000 7f8553200640 1 -- start start 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8553200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85541016e0 0x7f855419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8553200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 0x7f855419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8553200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8554106ae0 0x7f85541a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8553200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8554118fe0 con 0x7f8554102000 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8553200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8554118e60 con 0x7f85541016e0 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8553200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8554119160 con 0x7f8554106ae0 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8549400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 0x7f855419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8549400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 0x7f855419fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44060/0 (socket says 172.21.15.33:44060) 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8551e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85541016e0 0x7f855419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8549400640 1 -- 172.21.15.33:0/3761950763 learned_addr learned my addr 172.21.15.33:0/3761950763 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:52.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8552800640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8554106ae0 0x7f85541a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8549400640 1 -- 172.21.15.33:0/3761950763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8554106ae0 msgr2=0x7f85541a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.419+0000 7f8549400640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8554106ae0 0x7f85541a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f8549400640 1 -- 172.21.15.33:0/3761950763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85541016e0 msgr2=0x7f855419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f8549400640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85541016e0 0x7f855419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f8549400640 1 -- 172.21.15.33:0/3761950763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85541a44f0 con 0x7f8554102000 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f8551e00640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85541016e0 0x7f855419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f8549400640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 0x7f855419fa40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c970 tx=0x7f853c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f854b400640 1 -- 172.21.15.33:0/3761950763 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c007bb0 con 0x7f8554102000 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f854b400640 1 -- 172.21.15.33:0/3761950763 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853c007d50 con 0x7f8554102000 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85541a47e0 con 0x7f8554102000 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f854b400640 1 -- 172.21.15.33:0/3761950763 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c018660 con 0x7f8554102000 2024-04-26T11:30:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.420+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85541ac070 con 0x7f8554102000 2024-04-26T11:30:52.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 Processor -- start 2024-04-26T11:30:52.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f854b400640 1 -- 172.21.15.33:0/3761950763 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f853c0188c0 con 0x7f8554102000 2024-04-26T11:30:52.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 -- start start 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f05fe0 0x5634e6f063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f10c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f111c0 0x5634e6f135b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5634e6e6b380 con 0x5634e6f06900 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5634e6e07c30 con 0x5634e6f111c0 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f855b9f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5634e6e67340 con 0x5634e6f05fe0 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.424+0000 7f8551e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f10c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8551e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f10c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44070/0 (socket says 172.21.15.33:44070) 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8551e00640 1 -- 172.21.15.33:0/1863307128 learned_addr learned my addr 172.21.15.33:0/1863307128 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8552800640 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f05fe0 0x5634e6f063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8549400640 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f111c0 0x5634e6f135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8551e00640 1 -- 172.21.15.33:0/1863307128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f05fe0 msgr2=0x5634e6f063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8551e00640 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f05fe0 0x5634e6f063c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8551e00640 1 -- 172.21.15.33:0/1863307128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f111c0 msgr2=0x5634e6f135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8551e00640 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f111c0 0x5634e6f135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.425+0000 7f8551e00640 1 -- 172.21.15.33:0/1863307128 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634e6e29940 con 0x5634e6f06900 2024-04-26T11:30:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.426+0000 7f854b400640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f85380a3da0 0x7f85380a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.426+0000 7f8551e00640 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f10c80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8544009230 tx=0x7f85440045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a28035b6952526 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.426+0000 7f854b400640 1 -- 172.21.15.33:0/3761950763 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f853c0d09d0 con 0x7f8554102000 2024-04-26T11:30:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.426+0000 7f8552800640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f85380a3da0 0x7f85380a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.426+0000 7f8543000640 1 -- 172.21.15.33:0/1863307128 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544035040 con 0x5634e6f06900 2024-04-26T11:30:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.426+0000 7f8543000640 1 -- 172.21.15.33:0/1863307128 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8544005480 con 0x5634e6f06900 2024-04-26T11:30:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.426+0000 7f8543000640 1 -- 172.21.15.33:0/1863307128 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544003b20 con 0x5634e6f06900 2024-04-26T11:30:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.427+0000 7f855b9f8740 1 -- 172.21.15.33:0/1863307128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 msgr2=0x5634e6f10c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.427+0000 7f8552800640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f85380a3da0 0x7f85380a6260 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f854c005fd0 tx=0x7f854c009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.427+0000 7f855b9f8740 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f10c80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8544009230 tx=0x7f85440045b0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.427+0000 7f855b9f8740 1 -- 172.21.15.33:0/1863307128 shutdown_connections 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f111c0 0x5634e6f135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f10c80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 --2- 172.21.15.33:0/1863307128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f05fe0 0x5634e6f063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 -- 172.21.15.33:0/1863307128 >> 172.21.15.33:0/1863307128 conn(0x5634e6f024f0 msgr2=0x5634e6f028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 -- 172.21.15.33:0/1863307128 shutdown_connections 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 -- 172.21.15.33:0/1863307128 wait complete. 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 -- start start 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.428+0000 7f855b9f8740 1 Processor -- start 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f0e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f855b9f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5634e6f13d90 con 0x5634e6f06900 2024-04-26T11:30:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f855b9f8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5634e6df2c60 con 0x5634e6f05fe0 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f855b9f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5634e6f13f10 con 0x5634e6f111c0 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f855b9f8740 10 client.0 _ll_register_callbacks cb 0x5634e6e1ee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f8551e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f8549400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f0e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.429+0000 7f8551e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:47946/0 (socket says 172.21.15.33:47946) 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8551e00640 1 -- 172.21.15.33:0/2721098193 learned_addr learned my addr 172.21.15.33:0/2721098193 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8552800640 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8549400640 1 -- 172.21.15.33:0/2721098193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 msgr2=0x5634e6f0ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8549400640 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8549400640 1 -- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 msgr2=0x5634e6f0e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8549400640 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8549400640 1 -- 172.21.15.33:0/2721098193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634e6e01510 con 0x5634e6f06900 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8551e00640 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8549400640 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f0e680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f853c0ce010 tx=0x7f853c098a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc21c86deb57ac16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.430+0000 7f8537400640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:30:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f8537400640 1 -- 172.21.15.33:0/2721098193 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c09c110 con 0x5634e6f06900 2024-04-26T11:30:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f8537400640 1 -- 172.21.15.33:0/2721098193 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f853c0a5190 con 0x5634e6f06900 2024-04-26T11:30:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8544005480 con 0x5634e6f06900 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f8537400640 1 -- 172.21.15.33:0/2721098193 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f853c09ceb0 con 0x5634e6f06900 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f855b9f8740 10 client.15078 fetch_fsmap learned FSMap version 26 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5634e6b7d210 con 0x5634e6f06900 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f8537400640 1 -- 172.21.15.33:0/2721098193 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8544005480 con 0x5634e6f06900 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f8537400640 1 -- 172.21.15.33:0/2721098193 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f853c018bb0 con 0x5634e6f06900 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f855b9f8740 10 client.15078 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f855b9f8740 10 client.15078 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.431+0000 7f855b9f8740 20 client.15078 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.432+0000 7f855b9f8740 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6f20ef0 0x5634e6f23380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.432+0000 7f855b9f8740 4 client.15078 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.432+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634e6e78f90 con 0x5634e6f20ef0 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.432+0000 7f8551e00640 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6f20ef0 0x5634e6f23380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.432+0000 7f8551e00640 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6f20ef0 0x5634e6f23380 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcaa5687b3876a07 server_cookie=fe75158977e16d64 in_seq=0 out_seq=0 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.433+0000 7f8537400640 10 client.15078 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.433+0000 7f8537400640 1 -- 172.21.15.33:0/2721098193 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634e6e78f90 con 0x5634e6f20ef0 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.433+0000 7f8537400640 10 client.15078 handle_command_reply: tid=0 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.433+0000 7f855b9f8740 1 client.15078 shutdown 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 msgr2=0x5634e6f0e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f0e680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f853c0ce010 tx=0x7f853c098a50 comp rx=0 tx=0).stop 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 shutdown_connections 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6f20ef0 0x5634e6f23380 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f0e680 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 --2- 172.21.15.33:0/2721098193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f8537400640 0 client.15078 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 >> 172.21.15.33:0/2721098193 conn(0x5634e6f024f0 msgr2=0x5634e6f08270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 shutdown_connections 2024-04-26T11:30:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 1 -- 172.21.15.33:0/2721098193 wait complete. 2024-04-26T11:30:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.434+0000 7f855b9f8740 20 client.15078 trim_cache size 0 max 16384 2024-04-26T11:30:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.450+0000 7f855b9f8740 1 Processor -- start 2024-04-26T11:30:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f855b9f8740 1 -- start start 2024-04-26T11:30:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f1cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f24b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f855b9f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f85541aea20 con 0x5634e6f06900 2024-04-26T11:30:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f855b9f8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f85541ac340 con 0x5634e6f05fe0 2024-04-26T11:30:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f855b9f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f85541a0850 con 0x5634e6f111c0 2024-04-26T11:30:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f8549400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f24b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f8549400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f24b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44082/0 (socket says 172.21.15.33:44082) 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f8549400640 1 -- 172.21.15.33:0/670993673 learned_addr learned my addr 172.21.15.33:0/670993673 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f8551e00640 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f1cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.451+0000 7f8552800640 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8549400640 1 -- 172.21.15.33:0/670993673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 msgr2=0x5634e6f0ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8549400640 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8549400640 1 -- 172.21.15.33:0/670993673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 msgr2=0x5634e6f1cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8549400640 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f1cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8549400640 1 -- 172.21.15.33:0/670993673 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634e6c49620 con 0x5634e6f06900 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8551e00640 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f1cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8549400640 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f24b40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f853c099130 tx=0x7f853c098b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fbb584e8a748b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8542600640 1 -- 172.21.15.33:0/670993673 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c09c110 con 0x5634e6f06900 2024-04-26T11:30:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8542600640 1 -- 172.21.15.33:0/670993673 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853c0b9020 con 0x5634e6f06900 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.452+0000 7f8542600640 1 -- 172.21.15.33:0/670993673 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c0ce2c0 con 0x5634e6f06900 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 -- 172.21.15.33:0/670993673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 msgr2=0x5634e6f24b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f24b40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f853c099130 tx=0x7f853c098b80 comp rx=0 tx=0).stop 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 -- 172.21.15.33:0/670993673 shutdown_connections 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f0ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f24b40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 --2- 172.21.15.33:0/670993673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f1cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 -- 172.21.15.33:0/670993673 >> 172.21.15.33:0/670993673 conn(0x5634e6f024f0 msgr2=0x5634e6f0f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 -- 172.21.15.33:0/670993673 shutdown_connections 2024-04-26T11:30:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 -- 172.21.15.33:0/670993673 wait complete. 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.453+0000 7f855b9f8740 1 -- start start 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 1 Processor -- start 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f19160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f4ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f855410f9d0 con 0x5634e6f06900 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8554110e00 con 0x5634e6f05fe0 2024-04-26T11:30:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8554110060 con 0x5634e6f111c0 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.454+0000 7f855b9f8740 10 client.0 _ll_register_callbacks cb 0x7f853c00af00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8549400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f19160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:47960/0 (socket says 172.21.15.33:47960) 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8549400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f19160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44086/0 (socket says 172.21.15.33:44086) 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 -- 172.21.15.33:0/4175485255 learned_addr learned my addr 172.21.15.33:0/4175485255 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8552800640 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f4ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 -- 172.21.15.33:0/4175485255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 msgr2=0x5634e6f4ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f4ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 -- 172.21.15.33:0/4175485255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 msgr2=0x5634e6f19160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f19160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 -- 172.21.15.33:0/4175485255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634e6e25cb0 con 0x5634e6f05fe0 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8552800640 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f4ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:30:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8551e00640 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0be30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85440036d0 tx=0x7f8544009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ace9066e39a8e459 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8537400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.455+0000 7f8537400640 1 -- 172.21.15.33:0/4175485255 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544035040 con 0x5634e6f05fe0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f8537400640 1 -- 172.21.15.33:0/4175485255 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8544003a30 con 0x5634e6f05fe0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f853c0b9020 con 0x5634e6f05fe0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f8537400640 1 -- 172.21.15.33:0/4175485255 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8544005550 con 0x5634e6f05fe0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f855b9f8740 10 client.25205 fetch_fsmap learned FSMap version 26 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5634e6fa8f30 con 0x5634e6f05fe0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f8537400640 1 -- 172.21.15.33:0/4175485255 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853c0b9020 con 0x5634e6f05fe0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f8537400640 1 -- 172.21.15.33:0/4175485255 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f85440046e0 con 0x5634e6f05fe0 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f855b9f8740 10 client.25205 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f855b9f8740 10 client.25205 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f855b9f8740 20 client.25205 populate_metadata read hostname 'smithi033' 2024-04-26T11:30:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.456+0000 7f855b9f8740 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6fa9270 0x5634e6fab660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.457+0000 7f855b9f8740 4 client.25205 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:30:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.457+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5634e6e72e00 con 0x5634e6fa9270 2024-04-26T11:30:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.457+0000 7f8549400640 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6fa9270 0x5634e6fab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.457+0000 7f8549400640 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6fa9270 0x5634e6fab660 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723d960409932695 server_cookie=e5394d5e152b346f in_seq=0 out_seq=0 2024-04-26T11:30:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.457+0000 7f8537400640 10 client.25205 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f8537400640 1 -- 172.21.15.33:0/4175485255 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5634e6e72e00 con 0x5634e6fa9270 2024-04-26T11:30:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f8537400640 10 client.25205 handle_command_reply: tid=0 2024-04-26T11:30:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 client.25205 shutdown 2024-04-26T11:30:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 msgr2=0x5634e6f0be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0be30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85440036d0 tx=0x7f8544009260 comp rx=0 tx=0).stop 2024-04-26T11:30:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 shutdown_connections 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5634e6f111c0 0x5634e6f4ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5634e6fa9270 0x5634e6fab660 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5634e6f06900 0x5634e6f19160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 --2- 172.21.15.33:0/4175485255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5634e6f05fe0 0x5634e6f0be30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 >> 172.21.15.33:0/4175485255 conn(0x5634e6f024f0 msgr2=0x5634e6f00dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f8537400640 0 client.25205 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 shutdown_connections 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.466+0000 7f855b9f8740 1 -- 172.21.15.33:0/4175485255 wait complete. 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f855b9f8740 20 client.25205 trim_cache size 0 max 16384 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:30:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:30:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:30:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:30:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f85380a3da0 msgr2=0x7f85380a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f85380a3da0 0x7f85380a6260 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f854c005fd0 tx=0x7f854c009500 comp rx=0 tx=0).stop 2024-04-26T11:30:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 msgr2=0x7f855419fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 0x7f855419fa40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c970 tx=0x7f853c00cd70 comp rx=0 tx=0).stop 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 shutdown_connections 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f85380a3da0 0x7f85380a6260 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8554106ae0 0x7f85541a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8554102000 0x7f855419fa40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 --2- 172.21.15.33:0/3761950763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85541016e0 0x7f855419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.467+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 >> 172.21.15.33:0/3761950763 conn(0x7f85540fd150 msgr2=0x7f85540fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.468+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 shutdown_connections 2024-04-26T11:30:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:30:52.468+0000 7f8553200640 1 -- 172.21.15.33:0/3761950763 wait complete. 2024-04-26T11:30:52.763 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:52 smithi033 ceph-mon[27845]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:52.833 INFO:teuthology.orchestra.run.smithi033.stdout:{"run_dir":"/var/run/ceph"} 2024-04-26T11:30:52.833 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-04-26T11:30:52.834 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 4375 2024-04-26T11:30:52.834 DEBUG:tasks.quiescer:Yielding 2024-04-26T11:30:52.834 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T11:30:52.844 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T11:30:52.844 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi033.front.sepia.ceph.com 2024-04-26T11:30:52.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-26T11:30:52.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:30:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:52 smithi194 ceph-mon[33017]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:52 smithi096 ceph-mon[32753]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:52.954 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:30:52.999 INFO:teuthology.orchestra.run.smithi033.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-26T11:30:52.999 INFO:teuthology.orchestra.run.smithi033.stdout:0@5cc7f146-03be-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/2b1eefe1-b994-4781-af0e-f418abad6198 340G 0 340G 0% /tmp/cephfs 2024-04-26T11:30:53.008 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-26T11:30:53.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-26T11:30:53.254 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-26T11:30:53.309 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:54 smithi033 ceph-mon[27845]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:54 smithi194 ceph-mon[33017]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:54 smithi096 ceph-mon[32753]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T11:30:55.723 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-26T11:30:55.824 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-26T11:30:55.831 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-26T11:30:55.957 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/: 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 . 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 .. 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 data 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data: 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:30:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:30:56.063 INFO:teuthology.orchestra.run.smithi033.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 .. 2024-04-26T11:30:56.070 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-26T11:30:56.201 INFO:teuthology.orchestra.run.smithi033.stdout:[Service] 2024-04-26T11:30:56.202 INFO:teuthology.orchestra.run.smithi033.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-26T11:30:56.202 INFO:teuthology.orchestra.run.smithi033.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-26T11:30:56.208 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-26T11:30:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 -- 172.21.15.33:0/4033302758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c10b660 msgr2=0x7fab2c10ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 --2- 172.21.15.33:0/4033302758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c10b660 0x7fab2c10ba40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0098f0 tx=0x7fab1c009c00 comp rx=0 tx=0).stop 2024-04-26T11:30:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 -- 172.21.15.33:0/4033302758 shutdown_connections 2024-04-26T11:30:56.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 --2- 172.21.15.33:0/4033302758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab2c106020 0x7fab2c112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 --2- 172.21.15.33:0/4033302758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c10b660 0x7fab2c10ba40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 --2- 172.21.15.33:0/4033302758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c105660 0x7fab2c105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 -- 172.21.15.33:0/4033302758 >> 172.21.15.33:0/4033302758 conn(0x7fab2c101150 msgr2=0x7fab2c103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 -- 172.21.15.33:0/4033302758 shutdown_connections 2024-04-26T11:30:56.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.219+0000 7fab30a00700 1 -- 172.21.15.33:0/4033302758 wait complete. 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.220+0000 7fab30a00700 1 Processor -- start 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab30a00700 1 -- start start 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab30a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c105660 0x7fab2c19bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab30a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c106020 0x7fab2c19c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab30a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab2c10b660 0x7fab2c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab30a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1158b0 con 0x7fab2c10b660 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab30a00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c115730 con 0x7fab2c105660 2024-04-26T11:30:56.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab30a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c115a30 con 0x7fab2c106020 2024-04-26T11:30:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab29400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c106020 0x7fab2c19c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab29e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c105660 0x7fab2c19bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab29400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c106020 0x7fab2c19c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:58650/0 (socket says 172.21.15.33:58650) 2024-04-26T11:30:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab29400700 1 -- 172.21.15.33:0/1887683170 learned_addr learned my addr 172.21.15.33:0/1887683170 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:30:56.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.221+0000 7fab2a800700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab2c10b660 0x7fab2c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.222+0000 7fab29400700 1 -- 172.21.15.33:0/1887683170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c105660 msgr2=0x7fab2c19bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.222+0000 7fab29400700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c105660 0x7fab2c19bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.222+0000 7fab29400700 1 -- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab2c10b660 msgr2=0x7fab2c1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.222+0000 7fab29400700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab2c10b660 0x7fab2c1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.222+0000 7fab29400700 1 -- 172.21.15.33:0/1887683170 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c1a0dd0 con 0x7fab2c106020 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.222+0000 7fab29e00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c105660 0x7fab2c19bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.222+0000 7fab29400700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c106020 0x7fab2c19c300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9c0 tx=0x7fab1800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.223+0000 7fab22a00700 1 -- 172.21.15.33:0/1887683170 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1800cbd0 con 0x7fab2c106020 2024-04-26T11:30:56.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.223+0000 7fab22a00700 1 -- 172.21.15.33:0/1887683170 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab1800ee60 con 0x7fab2c106020 2024-04-26T11:30:56.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.223+0000 7fab22a00700 1 -- 172.21.15.33:0/1887683170 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab180055c0 con 0x7fab2c106020 2024-04-26T11:30:56.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.223+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c1a10c0 con 0x7fab2c106020 2024-04-26T11:30:56.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.223+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c1a8940 con 0x7fab2c106020 2024-04-26T11:30:56.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.224+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2c06a110 con 0x7fab2c106020 2024-04-26T11:30:56.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.227+0000 7fab22a00700 1 -- 172.21.15.33:0/1887683170 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fab18004750 con 0x7fab2c106020 2024-04-26T11:30:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.229+0000 7fab22a00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fab100a3ec0 0x7fab100a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:30:56.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.229+0000 7fab22a00700 1 -- 172.21.15.33:0/1887683170 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fab180d0e40 con 0x7fab2c106020 2024-04-26T11:30:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.229+0000 7fab29e00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fab100a3ec0 0x7fab100a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:30:56.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.230+0000 7fab29e00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fab100a3ec0 0x7fab100a6380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0098c0 tx=0x7fab1c019040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:30:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.230+0000 7fab22a00700 1 -- 172.21.15.33:0/1887683170 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab18053ff0 con 0x7fab2c106020 2024-04-26T11:30:56.418 INFO:teuthology.orchestra.run.smithi033.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-26T11:30:56.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.528+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab2c02e490 con 0x7fab2c106020 2024-04-26T11:30:56.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.529+0000 7fab22a00700 1 -- 172.21.15.33:0/1887683170 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7fab1809d320 con 0x7fab2c106020 2024-04-26T11:30:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:30:56.531 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:30:23.037562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24488,24527]},"id":1}]} 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fab100a3ec0 msgr2=0x7fab100a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fab100a3ec0 0x7fab100a6380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0098c0 tx=0x7fab1c019040 comp rx=0 tx=0).stop 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c106020 msgr2=0x7fab2c19c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c106020 0x7fab2c19c300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9c0 tx=0x7fab1800ecd0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 shutdown_connections 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fab100a3ec0 0x7fab100a6380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab2c10b660 0x7fab2c1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab2c105660 0x7fab2c19bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 --2- 172.21.15.33:0/1887683170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab2c106020 0x7fab2c19c300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:30:56.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 >> 172.21.15.33:0/1887683170 conn(0x7fab2c101150 msgr2=0x7fab2c101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:30:56.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.533+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 shutdown_connections 2024-04-26T11:30:56.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:30:56.534+0000 7fab30a00700 1 -- 172.21.15.33:0/1887683170 wait complete. 2024-04-26T11:30:56.536 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 26 2024-04-26T11:30:56.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:56 smithi033 ceph-mon[27845]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 13 op/s 2024-04-26T11:30:56.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:56 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1887683170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:56 smithi194 ceph-mon[33017]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 13 op/s 2024-04-26T11:30:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:56 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1887683170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:56 smithi096 ceph-mon[32753]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 13 op/s 2024-04-26T11:30:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:56 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1887683170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:30:57.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T11:26:32.073203+0000', 'modified': '2024-04-26T11:30:23.037562+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24527, 'mds_1': 24488, 'mds_2': 24484}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6835/1403509498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 1403509498}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 1403509498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6835/1002586260', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6834', 'nonce': 1002586260}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1002586260}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6837/1265312544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 1265312544}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 1265312544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.194:6839/517247009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 517247009}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 517247009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.96:6837/823522377', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 823522377}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 823522377}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.96:6839/1966519759', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 1966519759}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 1966519759}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24484, 24488, 24527]} 2024-04-26T11:30:57.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T11:30:57.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T11:30:57.942 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:30:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:30:58 smithi194 ceph-mon[33017]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-26T11:30:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:30:58 smithi033 ceph-mon[27845]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-26T11:30:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:30:58 smithi096 ceph-mon[32753]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-26T11:31:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 -- 172.21.15.33:0/3194511664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa34810fe20 msgr2=0x7fa348112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 --2- 172.21.15.33:0/3194511664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa34810fe20 0x7fa348112210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa34000b0a0 tx=0x7fa34000b3b0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 -- 172.21.15.33:0/3194511664 shutdown_connections 2024-04-26T11:31:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 --2- 172.21.15.33:0/3194511664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa34810fe20 0x7fa348112210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 --2- 172.21.15.33:0/3194511664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075690 0x7fa348075a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 --2- 172.21.15.33:0/3194511664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa348075fb0 0x7fa34810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 -- 172.21.15.33:0/3194511664 >> 172.21.15.33:0/3194511664 conn(0x7fa34807a610 msgr2=0x7fa34807aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 -- 172.21.15.33:0/3194511664 shutdown_connections 2024-04-26T11:31:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.324+0000 7fa34d800700 1 -- 172.21.15.33:0/3194511664 wait complete. 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.325+0000 7fa34d800700 1 Processor -- start 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.325+0000 7fa34d800700 1 -- start start 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa34d800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348075690 0x7fa34819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa34d800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075fb0 0x7fa34819c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa34d800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa34810fe20 0x7fa3481a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa34d800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa348115860 con 0x7fa348075690 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa34d800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa3481156e0 con 0x7fa348075fb0 2024-04-26T11:31:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa34d800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa3481159e0 con 0x7fa34810fe20 2024-04-26T11:31:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa346a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348075690 0x7fa34819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa346000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075fb0 0x7fa34819c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa347400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa34810fe20 0x7fa3481a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa346000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075fb0 0x7fa34819c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45822/0 (socket says 172.21.15.33:45822) 2024-04-26T11:31:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa346000700 1 -- 172.21.15.33:0/188268075 learned_addr learned my addr 172.21.15.33:0/188268075 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.326+0000 7fa346a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348075690 0x7fa34819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58666/0 (socket says 172.21.15.33:58666) 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa346a00700 1 -- 172.21.15.33:0/188268075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa34810fe20 msgr2=0x7fa3481a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa346a00700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa34810fe20 0x7fa3481a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa346a00700 1 -- 172.21.15.33:0/188268075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075fb0 msgr2=0x7fa34819c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa346a00700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075fb0 0x7fa34819c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa346a00700 1 -- 172.21.15.33:0/188268075 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3481a0d40 con 0x7fa348075690 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa347400700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa34810fe20 0x7fa3481a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa346000700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075fb0 0x7fa34819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa346a00700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348075690 0x7fa34819bd30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e9f0 tx=0x7fa33800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa33f400700 1 -- 172.21.15.33:0/188268075 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33800c9c0 con 0x7fa348075690 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3481a1030 con 0x7fa348075690 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa33f400700 1 -- 172.21.15.33:0/188268075 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa33800cb60 con 0x7fa348075690 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.327+0000 7fa33f400700 1 -- 172.21.15.33:0/188268075 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa338018680 con 0x7fa348075690 2024-04-26T11:31:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.328+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3481a88b0 con 0x7fa348075690 2024-04-26T11:31:00.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.331+0000 7fa33f400700 1 -- 172.21.15.33:0/188268075 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fa338010430 con 0x7fa348075690 2024-04-26T11:31:00.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa33f400700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa32c0a3e40 0x7fa32c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa358276b80 1 Processor -- start 2024-04-26T11:31:00.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa33f400700 1 -- 172.21.15.33:0/188268075 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa3380d0b50 con 0x7fa348075690 2024-04-26T11:31:00.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa346000700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa32c0a3e40 0x7fa32c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa358276b80 1 -- start start 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa358276b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a87c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa358276b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a876640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa358276b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a876b80 0x558c6a882cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.333+0000 7fa358276b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c6a7c9990 con 0x558c6a87bf10 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa358276b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558c6a7681f0 con 0x558c6a876b80 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa358276b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558c6a7c5950 con 0x558c6a8761c0 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a87c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a87c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58676/0 (socket says 172.21.15.33:58676) 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 -- 172.21.15.33:0/1212550996 learned_addr learned my addr 172.21.15.33:0/1212550996 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:00.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa347400700 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a876b80 0x558c6a882cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346000700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa32c0a3e40 0x7fa32c0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa348111b40 tx=0x7fa334009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346000700 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a876640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 -- 172.21.15.33:0/1212550996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 msgr2=0x558c6a876640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a876640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 -- 172.21.15.33:0/1212550996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a876b80 msgr2=0x558c6a882cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a876b80 0x558c6a882cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346a00700 1 -- 172.21.15.33:0/1212550996 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6a770710 con 0x558c6a87bf10 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.334+0000 7fa346000700 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a876640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.335+0000 7fa346a00700 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a87c2f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa33809c8c0 tx=0x7fa33809c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3a55abf8b79b8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.335+0000 7fa333e00700 1 -- 172.21.15.33:0/1212550996 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33809c930 con 0x558c6a87bf10 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.335+0000 7fa333e00700 1 -- 172.21.15.33:0/1212550996 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3380a0ce0 con 0x558c6a87bf10 2024-04-26T11:31:00.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.335+0000 7fa333e00700 1 -- 172.21.15.33:0/1212550996 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33809cd90 con 0x558c6a87bf10 2024-04-26T11:31:00.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 -- 172.21.15.33:0/1212550996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 msgr2=0x558c6a87c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a87c2f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa33809c8c0 tx=0x7fa33809c380 comp rx=0 tx=0).stop 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 -- 172.21.15.33:0/1212550996 shutdown_connections 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a876b80 0x558c6a882cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a87c2f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 --2- 172.21.15.33:0/1212550996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a876640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 -- 172.21.15.33:0/1212550996 >> 172.21.15.33:0/1212550996 conn(0x558c6a7a43f0 msgr2=0x558c6a870ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 -- 172.21.15.33:0/1212550996 shutdown_connections 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.336+0000 7fa358276b80 1 -- 172.21.15.33:0/1212550996 wait complete. 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.337+0000 7fa358276b80 1 -- start start 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.337+0000 7fa358276b80 1 Processor -- start 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa358276b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a8828e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa358276b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a876b80 0x558c6a87fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa358276b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a880120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa358276b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c6a883210 con 0x558c6a876b80 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa358276b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558c6a764440 con 0x558c6a87bf10 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa358276b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558c6a883390 con 0x558c6a8761c0 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa358276b80 10 client.0 _ll_register_callbacks cb 0x558c6a75d330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a876b80 0x558c6a87fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa347400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a880120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a876b80 0x558c6a87fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58688/0 (socket says 172.21.15.33:58688) 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 -- 172.21.15.33:0/3744508139 learned_addr learned my addr 172.21.15.33:0/3744508139 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346a00700 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a8828e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 -- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 msgr2=0x558c6a8828e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a8828e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 -- 172.21.15.33:0/3744508139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 msgr2=0x558c6a880120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a880120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa346000700 1 -- 172.21.15.33:0/3744508139 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6a716880 con 0x558c6a876b80 2024-04-26T11:31:00.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.338+0000 7fa347400700 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a880120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:00.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.339+0000 7fa346000700 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a876b80 0x558c6a87fbe0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa33401bce0 tx=0x7fa3340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3944138ce3af8bcc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:00.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.339+0000 7fa330c00700 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.339+0000 7fa330c00700 1 -- 172.21.15.33:0/3744508139 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33401dcb0 con 0x558c6a876b80 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.339+0000 7fa330c00700 1 -- 172.21.15.33:0/3744508139 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa3340273f0 con 0x558c6a876b80 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.339+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3380a0ce0 con 0x558c6a876b80 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa330c00700 1 -- 172.21.15.33:0/3744508139 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa33401de50 con 0x558c6a876b80 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 10 client.15102 fetch_fsmap learned FSMap version 26 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558c6a88d8d0 con 0x558c6a876b80 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa330c00700 1 -- 172.21.15.33:0/3744508139 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3380a0ce0 con 0x558c6a876b80 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa330c00700 1 -- 172.21.15.33:0/3744508139 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa334030890 con 0x558c6a876b80 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 10 client.15102 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 20 client.15102 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:00.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a88ff70 0x558c6a8913e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 4 client.15102 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.340+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 --> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c6a842b20 con 0x558c6a88ff70 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.341+0000 7fa346a00700 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a88ff70 0x558c6a8913e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.341+0000 7fa346a00700 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a88ff70 0x558c6a8913e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95b7508a95368ca6 server_cookie=e713d0156f38e5e1 in_seq=0 out_seq=0 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.341+0000 7fa330c00700 10 client.15102 ms_handle_connect on v2:172.21.15.194:6838/517247009 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.342+0000 7fa330c00700 1 -- 172.21.15.33:0/3744508139 <== mds.2 v2:172.21.15.194:6838/517247009 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c6a842b20 con 0x558c6a88ff70 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.342+0000 7fa330c00700 10 client.15102 handle_command_reply: tid=0 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.342+0000 7fa358276b80 1 client.15102 shutdown 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a876b80 msgr2=0x558c6a87fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a876b80 0x558c6a87fbe0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa33401bce0 tx=0x7fa3340047c0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 shutdown_connections 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a88ff70 0x558c6a8913e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a880120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a8761c0 0x558c6a8828e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 --2- 172.21.15.33:0/3744508139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a876b80 0x558c6a87fbe0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa330c00700 0 client.15102 ms_handle_reset on v2:172.21.15.194:6838/517247009 2024-04-26T11:31:00.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 >> 172.21.15.33:0/3744508139 conn(0x558c6a7a43f0 msgr2=0x558c6a86b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:00.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 shutdown_connections 2024-04-26T11:31:00.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 1 -- 172.21.15.33:0/3744508139 wait complete. 2024-04-26T11:31:00.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.343+0000 7fa358276b80 20 client.15102 trim_cache size 0 max 16384 2024-04-26T11:31:00.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.366+0000 7fa358276b80 1 Processor -- start 2024-04-26T11:31:00.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.366+0000 7fa358276b80 1 -- start start 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa358276b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a8761c0 0x558c6a88c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa358276b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a87ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa358276b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a87b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa358276b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c6a88f720 con 0x558c6a8761c0 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa358276b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558c6a88fdf0 con 0x558c6a87bf10 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa358276b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558c6a8904c0 con 0x558c6a876b80 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa346a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a8761c0 0x558c6a88c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa346a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a8761c0 0x558c6a88c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58694/0 (socket says 172.21.15.33:58694) 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa346a00700 1 -- 172.21.15.33:0/3883934584 learned_addr learned my addr 172.21.15.33:0/3883934584 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:00.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa347400700 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a87b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.367+0000 7fa346000700 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a87ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa346a00700 1 -- 172.21.15.33:0/3883934584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 msgr2=0x558c6a87ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa346a00700 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a87ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa346a00700 1 -- 172.21.15.33:0/3883934584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 msgr2=0x558c6a87b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa346a00700 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a87b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa346a00700 1 -- 172.21.15.33:0/3883934584 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6a86d730 con 0x558c6a8761c0 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa347400700 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a87b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa346000700 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a87ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa346a00700 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a8761c0 0x558c6a88c130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa3380a5070 tx=0x7fa338009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1debf90c66a7c375 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:00.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa333400700 1 -- 172.21.15.33:0/3883934584 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3380a0ce0 con 0x558c6a8761c0 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa333400700 1 -- 172.21.15.33:0/3883934584 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3380870b0 con 0x558c6a8761c0 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.368+0000 7fa333400700 1 -- 172.21.15.33:0/3883934584 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3380a5370 con 0x558c6a8761c0 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 -- 172.21.15.33:0/3883934584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a8761c0 msgr2=0x558c6a88c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a8761c0 0x558c6a88c130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa3380a5070 tx=0x7fa338009790 comp rx=0 tx=0).stop 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 -- 172.21.15.33:0/3883934584 shutdown_connections 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a87bf10 0x558c6a87b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a8761c0 0x558c6a88c130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 --2- 172.21.15.33:0/3883934584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a87ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 -- 172.21.15.33:0/3883934584 >> 172.21.15.33:0/3883934584 conn(0x558c6a7a43f0 msgr2=0x558c6a87e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 -- 172.21.15.33:0/3883934584 shutdown_connections 2024-04-26T11:31:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.369+0000 7fa358276b80 1 -- 172.21.15.33:0/3883934584 wait complete. 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.370+0000 7fa358276b80 1 -- start start 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.370+0000 7fa358276b80 1 Processor -- start 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.370+0000 7fa358276b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a8761c0 0x558c6a89a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa358276b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a89ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa358276b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a878dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa358276b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558c6a891260 con 0x558c6a87bf10 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa358276b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558c6a890b90 con 0x558c6a8761c0 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa358276b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558c6a891930 con 0x558c6a876b80 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa358276b80 10 client.0 _ll_register_callbacks cb 0x558c6a75d330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a8761c0 0x558c6a89a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa347400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a878dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a8761c0 0x558c6a89a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45862/0 (socket says 172.21.15.33:45862) 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 -- 172.21.15.33:0/3089187640 learned_addr learned my addr 172.21.15.33:0/3089187640 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346000700 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a89ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 -- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 msgr2=0x558c6a89ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a89ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 -- 172.21.15.33:0/3089187640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 msgr2=0x558c6a878dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a878dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.371+0000 7fa346a00700 1 -- 172.21.15.33:0/3089187640 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6a877840 con 0x558c6a8761c0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa346a00700 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a8761c0 0x558c6a89a850 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa33800c5b0 tx=0x7fa33809d0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b4adcd432a5f72b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa330c00700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa330c00700 1 -- 172.21.15.33:0/3089187640 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3380a0ce0 con 0x558c6a8761c0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa330c00700 1 -- 172.21.15.33:0/3089187640 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa33809ac30 con 0x558c6a8761c0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3380870b0 con 0x558c6a8761c0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa330c00700 1 -- 172.21.15.33:0/3089187640 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa3380672b0 con 0x558c6a8761c0 2024-04-26T11:31:00.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa358276b80 10 client.25229 fetch_fsmap learned FSMap version 26 2024-04-26T11:31:00.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558c6a8f71d0 con 0x558c6a8761c0 2024-04-26T11:31:00.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.372+0000 7fa330c00700 1 -- 172.21.15.33:0/3089187640 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3380870b0 con 0x558c6a8761c0 2024-04-26T11:31:00.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa330c00700 1 -- 172.21.15.33:0/3089187640 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa338010720 con 0x558c6a8761c0 2024-04-26T11:31:00.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa358276b80 10 client.25229 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa358276b80 10 client.25229 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa358276b80 20 client.25229 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa358276b80 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a8fb650 0x558c6a8fa360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa358276b80 4 client.25229 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 --> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] -- command(tid 0: {"prefix": "status"}) -- 0x558c6a7d7370 con 0x558c6a8fb650 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.373+0000 7fa346000700 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a8fb650 0x558c6a8fa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa346000700 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a8fb650 0x558c6a8fa360 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6718b9ac742aa35c server_cookie=184fd55fe148012 in_seq=0 out_seq=0 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa330c00700 10 client.25229 ms_handle_connect on v2:172.21.15.194:6838/517247009 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa330c00700 1 -- 172.21.15.33:0/3089187640 <== mds.2 v2:172.21.15.194:6838/517247009 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558c6a7d7370 con 0x558c6a8fb650 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa330c00700 10 client.25229 handle_command_reply: tid=0 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 client.25229 shutdown 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a8761c0 msgr2=0x558c6a89a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a8761c0 0x558c6a89a850 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa33800c5b0 tx=0x7fa33809d0b0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 shutdown_connections 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] conn(0x558c6a8fb650 0x558c6a8fa360 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558c6a87bf10 0x558c6a878dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c6a8761c0 0x558c6a89a850 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 --2- 172.21.15.33:0/3089187640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c6a876b80 0x558c6a89ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa330c00700 0 client.25229 ms_handle_reset on v2:172.21.15.194:6838/517247009 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.374+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 >> 172.21.15.33:0/3089187640 conn(0x558c6a7a43f0 msgr2=0x558c6a86ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.375+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 shutdown_connections 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.375+0000 7fa358276b80 1 -- 172.21.15.33:0/3089187640 wait complete. 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.375+0000 7fa358276b80 20 client.25229 trim_cache size 0 max 16384 2024-04-26T11:31:00.378 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24484, 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 250.26587331600001, 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 26, 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 91, 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 302.455925869 2024-04-26T11:31:00.379 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:31:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.375+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa32c0a3e40 msgr2=0x7fa32c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.375+0000 7fa34d800700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa32c0a3e40 0x7fa32c0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa348111b40 tx=0x7fa334009260 comp rx=0 tx=0).stop 2024-04-26T11:31:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348075690 msgr2=0x7fa34819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348075690 0x7fa34819bd30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e9f0 tx=0x7fa33800ed00 comp rx=0 tx=0).stop 2024-04-26T11:31:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 shutdown_connections 2024-04-26T11:31:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa32c0a3e40 0x7fa32c0a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa34810fe20 0x7fa3481a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa348075690 0x7fa34819bd30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 --2- 172.21.15.33:0/188268075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa348075fb0 0x7fa34819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:00.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 >> 172.21.15.33:0/188268075 conn(0x7fa34807a610 msgr2=0x7fa348100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:00.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 shutdown_connections 2024-04-26T11:31:00.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:00.376+0000 7fa34d800700 1 -- 172.21.15.33:0/188268075 wait complete. 2024-04-26T11:31:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:00 smithi033 ceph-mon[27845]: pgmap v21: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-04-26T11:31:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:00 smithi194 ceph-mon[33017]: pgmap v21: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-04-26T11:31:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:00 smithi096 ceph-mon[32753]: pgmap v21: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-04-26T11:31:01.177 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:31:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:31:00.852+0000 7fb09ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:31:01.177 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:31:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:31:00.852+0000 7fb09ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:31:01.177 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:31:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:31:00.852+0000 7fb09ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:31:01.177 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:31:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:31:00.852+0000 7fb09ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:31:01.177 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 26 11:31:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mgr-x[28082]: 2024-04-26T11:31:00.852+0000 7fb09ac00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T11:31:01.298 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-26T11:31:01.782 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:31:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:02 smithi194 ceph-mon[33017]: pgmap v22: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-26T11:31:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:02 smithi194 ceph-mon[33017]: mgrmap e26: x(active, since 38s), standbys: z, y 2024-04-26T11:31:02.892 INFO:teuthology.orchestra.run.smithi033.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-26T11:31:02.894 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:02 smithi033 ceph-mon[27845]: pgmap v22: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-26T11:31:02.894 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:02 smithi033 ceph-mon[27845]: mgrmap e26: x(active, since 38s), standbys: z, y 2024-04-26T11:31:02.902 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T11:31:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:02 smithi096 ceph-mon[32753]: pgmap v22: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-26T11:31:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:02 smithi096 ceph-mon[32753]: mgrmap e26: x(active, since 38s), standbys: z, y 2024-04-26T11:31:03.037 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/: 2024-04-26T11:31:03.037 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.037 INFO:teuthology.orchestra.run.smithi033.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 . 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 .. 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 data 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data: 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:total 35 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 . 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 .. 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 base 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 26 11:30 global 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 log 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_commit_ts 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_dynshmem 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 26 11:30 pg_hba.conf 2024-04-26T11:31:03.038 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 26 11:30 pg_ident.conf 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 pg_logical 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 11:30 pg_multixact 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_notify 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_replslot 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_serial 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_snapshots 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_stat 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_stat_tmp 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 pg_subtrans 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_tblspc 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 pg_twophase 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:30 PG_VERSION 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 11:30 pg_wal 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 pg_xact 2024-04-26T11:31:03.039 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 26 11:30 postgresql.auto.conf 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 26 11:30 postgresql.conf 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/base: 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 . 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 11:30 1 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 11:31 13493 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 11:31 13494 2024-04-26T11:31:03.040 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:total 7777 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 11:30 . 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 .. 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 112 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 113 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 11:30 1247 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1247_fsm 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1247_vm 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 11:30 1249 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1249_fsm 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1249_vm 2024-04-26T11:31:03.041 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 11:30 1255 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1255_fsm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1255_vm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 11:31 1259 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1259_fsm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1259_vm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 11:30 13344 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 13344_fsm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13344_vm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 13346 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13348 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13349 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 13349_fsm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13349_vm 2024-04-26T11:31:03.042 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 13351 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13353 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13354 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 13354_fsm 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13354_vm 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 13356 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13358 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13359 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 13359_fsm 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13359_vm 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 13361 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13363 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 1417 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 1418 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 174 2024-04-26T11:31:03.043 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 175 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2187 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2224 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2328 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2336 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2337 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2579 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2600 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2600_fsm 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2600_vm 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2601 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2601_fsm 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2601_vm 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:30 2602 2024-04-26T11:31:03.044 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2602_fsm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2602_vm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2603 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2603_fsm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2603_vm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2604 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2605 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2605_fsm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2605_vm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2606 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2606_fsm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2606_vm 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2607 2024-04-26T11:31:03.045 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2607_fsm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2607_vm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 11:30 2608 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2608_fsm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2608_vm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 11:30 2609 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2609_fsm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2609_vm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2610 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2610_fsm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2610_vm 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2611 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2612 2024-04-26T11:31:03.046 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2612_fsm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2612_vm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2613 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2615 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2615_fsm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2615_vm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2616 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2616_fsm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2616_vm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 11:30 2617 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2617_fsm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2617_vm 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 11:30 2618 2024-04-26T11:31:03.047 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2618_fsm 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2618_vm 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 11:30 2619 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2619_fsm 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2619_vm 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2620 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2650 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2651 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2652 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:30 2653 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:30 2654 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2655 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2656 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2657 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 11:30 2658 2024-04-26T11:31:03.048 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 11:30 2659 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2660 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2661 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2662 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:30 2663 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2664 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2665 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2666 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2667 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2668 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2669 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2670 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 11:30 2673 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 11:30 2674 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 11:30 2675 2024-04-26T11:31:03.049 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2678 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2679 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2680 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2681 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2682 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2683 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2684 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2685 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2686 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2687 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:30 2688 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:30 2689 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 11:30 2690 2024-04-26T11:31:03.050 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 11:30 2691 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2692 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2693 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2696 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2699 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2701 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2702 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2703 2024-04-26T11:31:03.051 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:30 2704 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2753 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2753_fsm 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2753_vm 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2754 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2755 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2756 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2757 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2830 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2831 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2832 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2833 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2834 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2835 2024-04-26T11:31:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2836 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2837 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 11:30 2838 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2838_fsm 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2838_vm 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2839 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2840 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2840_fsm 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2840_vm 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2841 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2995 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2996 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3079 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3079_fsm 2024-04-26T11:31:03.053 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3079_vm 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3080 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3081 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:30 3085 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3118 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3119 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 11:30 3164 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3256 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3257 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3258 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3350 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3351 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3379 2024-04-26T11:31:03.054 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3380 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3381 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3394 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3394_fsm 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3394_vm 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3395 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3429 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3430 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3431 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3433 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3439 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3440 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:30 3455 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 11:30 3456 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3456_fsm 2024-04-26T11:31:03.055 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3456_vm 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3466 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3467 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3468 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3501 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3502 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3503 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3534 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3541 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3541_fsm 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3541_vm 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3542 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3574 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3575 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3576 2024-04-26T11:31:03.056 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3596 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3597 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3598 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3599 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3600 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3600_fsm 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3600_vm 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3601 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3601_fsm 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3601_vm 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3602 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3602_fsm 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3602_vm 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3603 2024-04-26T11:31:03.057 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3603_fsm 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3603_vm 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3604 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3605 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3606 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3607 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3608 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 3609 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3712 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3764 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3764_fsm 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3764_vm 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3766 2024-04-26T11:31:03.058 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3767 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3997 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4143 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4144 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4145 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4146 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4147 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4148 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4149 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4150 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4151 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4152 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4153 2024-04-26T11:31:03.059 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4154 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4155 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4156 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4157 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4158 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4159 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4160 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4161 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4162 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4163 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4164 2024-04-26T11:31:03.060 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4165 2024-04-26T11:31:03.061 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4166 2024-04-26T11:31:03.061 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4167 2024-04-26T11:31:03.061 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4168 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4169 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4170 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4171 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4172 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4173 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4174 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 5002 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 548 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 549 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 6102 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 6104 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 6106 2024-04-26T11:31:03.062 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6110 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6111 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6112 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6113 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6117 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 826 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 827 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 828 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 11:30 pg_filenode.map 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:30 PG_VERSION 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:total 7777 2024-04-26T11:31:03.063 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 11:31 . 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 .. 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 112 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 113 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 11:31 1247 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1247_fsm 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1247_vm 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 11:31 1249 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1249_fsm 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1249_vm 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 11:30 1255 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 1255_fsm 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1255_vm 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 11:31 1259 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 1259_fsm 2024-04-26T11:31:03.064 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1259_vm 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 11:30 13344 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 13344_fsm 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13344_vm 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 13346 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13348 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13349 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 13349_fsm 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13349_vm 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 13351 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13353 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13354 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 13354_fsm 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13354_vm 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 13356 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13358 2024-04-26T11:31:03.065 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13359 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 13359_fsm 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13359_vm 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 13361 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 13363 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 1417 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 1418 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 174 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 175 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2187 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2224 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2328 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2336 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2337 2024-04-26T11:31:03.066 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2579 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2600 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2600_fsm 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2600_vm 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2601 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2601_fsm 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2601_vm 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:30 2602 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2602_fsm 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2602_vm 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2603 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2603_fsm 2024-04-26T11:31:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2603_vm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2604 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2605 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2605_fsm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2605_vm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2606 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2606_fsm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2606_vm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2607 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2607_fsm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2607_vm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 11:31 2608 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2608_fsm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2608_vm 2024-04-26T11:31:03.068 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 11:30 2609 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2609_fsm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2609_vm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2610 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2610_fsm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2610_vm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2611 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2612 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2612_fsm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2612_vm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2613 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2615 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2615_fsm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2615_vm 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2616 2024-04-26T11:31:03.069 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2616_fsm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2616_vm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 11:30 2617 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2617_fsm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2617_vm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 11:30 2618 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2618_fsm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2618_vm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 11:30 2619 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2619_fsm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2619_vm 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2620 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2650 2024-04-26T11:31:03.070 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2651 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2652 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 2653 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 2654 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2655 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2656 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2657 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 11:30 2658 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 11:31 2659 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2660 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2661 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2662 2024-04-26T11:31:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:31 2663 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2664 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2665 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2666 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2667 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2668 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2669 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2670 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 11:31 2673 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 11:31 2674 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 11:30 2675 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2678 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2679 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2680 2024-04-26T11:31:03.072 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2681 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2682 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2683 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2684 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2685 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2686 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2687 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:30 2688 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:30 2689 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 11:31 2690 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 11:31 2691 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2692 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2693 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2696 2024-04-26T11:31:03.073 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2699 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2701 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2702 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2703 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 2704 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2753 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2753_fsm 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2753_vm 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2754 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2755 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2756 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 2757 2024-04-26T11:31:03.074 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2830 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2831 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2832 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2833 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2834 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2835 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2836 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2837 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 11:31 2838 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2838_fsm 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2838_vm 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2839 2024-04-26T11:31:03.075 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2840 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2840_fsm 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2840_vm 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2841 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2995 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2996 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3079 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3079_fsm 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3079_vm 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3080 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3081 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 3085 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3118 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3119 2024-04-26T11:31:03.076 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 11:31 3164 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3256 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3257 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3258 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3350 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3351 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3379 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3380 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3381 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3394 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3394_fsm 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3394_vm 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3395 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3429 2024-04-26T11:31:03.077 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3430 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3431 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3433 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3439 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3440 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 3455 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 11:30 3456 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3456_fsm 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3456_vm 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3466 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3467 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3468 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3501 2024-04-26T11:31:03.078 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3502 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3503 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3534 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3541 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3541_fsm 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3541_vm 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3542 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3574 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3575 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3576 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3596 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3597 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3598 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3599 2024-04-26T11:31:03.079 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3600 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3600_fsm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3600_vm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3601 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3601_fsm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3601_vm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3602 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3602_fsm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3602_vm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3603 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3603_fsm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3603_vm 2024-04-26T11:31:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3604 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3605 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3606 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3607 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3608 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:30 3609 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3712 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3764 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 3764_fsm 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3764_vm 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3766 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 3767 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3997 2024-04-26T11:31:03.081 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4143 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4144 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4145 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4146 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4147 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4148 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4149 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4150 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4151 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4152 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4153 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4154 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4155 2024-04-26T11:31:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4156 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4157 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4158 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4159 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4160 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4161 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4162 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4163 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4164 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4165 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4166 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4167 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4168 2024-04-26T11:31:03.083 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4169 2024-04-26T11:31:03.084 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4170 2024-04-26T11:31:03.084 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4171 2024-04-26T11:31:03.084 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4172 2024-04-26T11:31:03.084 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4173 2024-04-26T11:31:03.084 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4174 2024-04-26T11:31:03.084 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 5002 2024-04-26T11:31:03.084 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 548 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 549 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 6102 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 6104 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 6106 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6110 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6111 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6112 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6113 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6117 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 826 2024-04-26T11:31:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 827 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 828 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 11:31 pg_filenode.map 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 PG_VERSION 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:total 7777 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 11:31 . 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 .. 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 112 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 113 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 11:31 1247 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 1247_fsm 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 1247_vm 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 11:31 1249 2024-04-26T11:31:03.087 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 1249_fsm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 1249_vm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 11:31 1255 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 1255_fsm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 1255_vm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 11:31 1259 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 1259_fsm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 1259_vm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 11:31 13344 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 13344_fsm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13344_vm 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 13346 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13348 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13349 2024-04-26T11:31:03.088 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 13349_fsm 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13349_vm 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 13351 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13353 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13354 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 13354_fsm 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13354_vm 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 13356 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13358 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13359 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 13359_fsm 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13359_vm 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 13361 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 13363 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 1417 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 1418 2024-04-26T11:31:03.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 174 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 175 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2187 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2224 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2328 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2336 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2337 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2579 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2600 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2600_fsm 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2600_vm 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2601 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2601_fsm 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2601_vm 2024-04-26T11:31:03.090 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:31 2602 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2602_fsm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2602_vm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2603 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2603_fsm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2603_vm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2604 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2605 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2605_fsm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2605_vm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2606 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2606_fsm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2606_vm 2024-04-26T11:31:03.091 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2607 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2607_fsm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2607_vm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 11:31 2608 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2608_fsm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2608_vm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 11:31 2609 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2609_fsm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2609_vm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2610 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2610_fsm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2610_vm 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2611 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2612 2024-04-26T11:31:03.092 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2612_fsm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2612_vm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2613 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2615 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2615_fsm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2615_vm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2616 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2616_fsm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2616_vm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 11:31 2617 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2617_fsm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2617_vm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 11:31 2618 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2618_fsm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2618_vm 2024-04-26T11:31:03.093 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 11:31 2619 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2619_fsm 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2619_vm 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2620 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2650 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2651 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2652 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 2653 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 2654 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2655 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2656 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2657 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 11:31 2658 2024-04-26T11:31:03.094 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 11:31 2659 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2660 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2661 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2662 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:31 2663 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2664 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2665 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2666 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2667 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2668 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2669 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2670 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 11:31 2673 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 11:31 2674 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 11:31 2675 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2678 2024-04-26T11:31:03.095 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2679 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2680 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2681 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2682 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2683 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2684 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2685 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2686 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2687 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 2688 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 11:31 2689 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 11:31 2690 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 11:31 2691 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2692 2024-04-26T11:31:03.096 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2693 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2696 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2699 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2701 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2702 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2703 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 2704 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2753 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2753_fsm 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2753_vm 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2754 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2755 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2756 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 2757 2024-04-26T11:31:03.097 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2830 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2831 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2832 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2833 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2834 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2835 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2836 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2837 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 11:31 2838 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2838_fsm 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2838_vm 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2839 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2840 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 2840_fsm 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2840_vm 2024-04-26T11:31:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2841 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 2995 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2996 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3079 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3079_fsm 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3079_vm 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3080 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3081 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 3085 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3118 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3119 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 11:31 3164 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3256 2024-04-26T11:31:03.099 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3257 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3258 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3350 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3351 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3379 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3380 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3381 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3394 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3394_fsm 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3394_vm 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3395 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3429 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3430 2024-04-26T11:31:03.100 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3431 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3433 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3439 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3440 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 11:31 3455 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 11:31 3456 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3456_fsm 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3456_vm 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3466 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3467 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3468 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3501 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3502 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3503 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3534 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3541 2024-04-26T11:31:03.101 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3541_fsm 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3541_vm 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3542 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3574 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3575 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3576 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3596 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3597 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 3598 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3599 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3600 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3600_fsm 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3600_vm 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3601 2024-04-26T11:31:03.102 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3601_fsm 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3601_vm 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3602 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3602_fsm 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3602_vm 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3603 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3603_fsm 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3603_vm 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3604 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3605 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3606 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3607 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3608 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 11:31 3609 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3712 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3764 2024-04-26T11:31:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 3764_fsm 2024-04-26T11:31:03.104 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3764_vm 2024-04-26T11:31:03.104 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3766 2024-04-26T11:31:03.104 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 3767 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 3997 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4143 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4144 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4145 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4146 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4147 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4148 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4149 2024-04-26T11:31:03.106 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4150 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4151 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4152 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4153 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4154 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4155 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4156 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4157 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4158 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4159 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4160 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4161 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4162 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4163 2024-04-26T11:31:03.107 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4164 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4165 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4166 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4167 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4168 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4169 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4170 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4171 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4172 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 4173 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 4174 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 5002 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 548 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 549 2024-04-26T11:31:03.108 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 6102 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 6104 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 6106 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6110 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6111 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6112 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6113 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 6117 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 826 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 827 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 828 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 11:31 pg_filenode.map 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 PG_VERSION 2024-04-26T11:31:03.109 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/global: 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:total 537 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 26 11:30 . 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1213 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1213_fsm 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1213_vm 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1214 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1214_fsm 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1214_vm 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 1232 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 1233 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1260 2024-04-26T11:31:03.110 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1260_fsm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1260_vm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1261 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 1261_fsm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 1261_vm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 1262 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:31 1262_fsm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 1262_vm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2396 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 11:30 2396_fsm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 2396_vm 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2397 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2671 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:31 2672 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2676 2024-04-26T11:31:03.111 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2677 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2694 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2695 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2697 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 11:30 2698 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2846 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2847 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2964 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2965 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 2966 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 2967 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 3592 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 3593 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4060 2024-04-26T11:31:03.112 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4061 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4175 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4176 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4177 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4178 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4181 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4182 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4183 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4184 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 4185 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 4186 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 6000 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6001 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6002 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 6100 2024-04-26T11:31:03.113 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6114 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 6115 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 pg_control 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 11:30 pg_filenode.map 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/log: 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:31 . 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.114 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:total 1 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 . 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 mappings 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 26 11:31 replorigin_checkpoint 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 snapshots 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.115 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 .. 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 11:31 .. 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 11:30 . 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.116 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 members 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 offsets 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:total 8 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 . 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 11:30 .. 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 0000 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:total 8 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 . 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 11:30 .. 2024-04-26T11:31:03.117 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 0000 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-26T11:31:03.118 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.119 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:total 8 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 . 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:30 0000 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.120 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:total 16384 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 11:30 . 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 11:31 000000010000000000000001 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 archive_status 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:total 0 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 11:30 . 2024-04-26T11:31:03.121 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 11:30 .. 2024-04-26T11:31:03.122 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:03.122 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-26T11:31:03.122 INFO:teuthology.orchestra.run.smithi033.stdout:total 8 2024-04-26T11:31:03.122 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 11:30 . 2024-04-26T11:31:03.122 INFO:teuthology.orchestra.run.smithi033.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 11:31 .. 2024-04-26T11:31:03.122 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 11:31 0000 2024-04-26T11:31:03.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-26T11:31:03.399 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-26T11:31:03.503 INFO:teuthology.orchestra.run.smithi033.stderr:dropping old tables... 2024-04-26T11:31:03.504 INFO:teuthology.orchestra.run.smithi033.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-26T11:31:03.504 INFO:teuthology.orchestra.run.smithi033.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-26T11:31:03.504 INFO:teuthology.orchestra.run.smithi033.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-26T11:31:03.504 INFO:teuthology.orchestra.run.smithi033.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-26T11:31:03.504 INFO:teuthology.orchestra.run.smithi033.stderr:creating tables... 2024-04-26T11:31:03.526 INFO:teuthology.orchestra.run.smithi033.stderr:generating data (client-side)... 2024-04-26T11:31:03.690 INFO:teuthology.orchestra.run.smithi033.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.11 s, remaining 3.50 s) 2024-04-26T11:31:04.088 INFO:teuthology.orchestra.run.smithi033.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.51 s, remaining 7.66 s) 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.132+0000 7fbc61c00700 1 -- 172.21.15.33:0/510351443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 msgr2=0x7fbc5c116920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.132+0000 7fbc61c00700 1 --2- 172.21.15.33:0/510351443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c116920 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009f90 tx=0x7fbc5400b330 comp rx=0 tx=0).stop 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.133+0000 7fbc61c00700 1 -- 172.21.15.33:0/510351443 shutdown_connections 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.133+0000 7fbc61c00700 1 --2- 172.21.15.33:0/510351443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c116920 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.133+0000 7fbc61c00700 1 --2- 172.21.15.33:0/510351443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c10fa50 0x7fbc5c10fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.133+0000 7fbc61c00700 1 --2- 172.21.15.33:0/510351443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc5c109a50 0x7fbc5c109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.133+0000 7fbc61c00700 1 -- 172.21.15.33:0/510351443 >> 172.21.15.33:0/510351443 conn(0x7fbc5c076bf0 msgr2=0x7fbc5c108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.133+0000 7fbc61c00700 1 -- 172.21.15.33:0/510351443 shutdown_connections 2024-04-26T11:31:04.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.133+0000 7fbc61c00700 1 -- 172.21.15.33:0/510351443 wait complete. 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.134+0000 7fbc61c00700 1 Processor -- start 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.134+0000 7fbc61c00700 1 -- start start 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.134+0000 7fbc61c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc5c109a50 0x7fbc5c1a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.134+0000 7fbc61c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc61c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c10fa50 0x7fbc5c1a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc61c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c119ca0 con 0x7fbc5c10fa50 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc61c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c119b20 con 0x7fbc5c10a410 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc61c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c119e20 con 0x7fbc5c109a50 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c1a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45896/0 (socket says 172.21.15.33:45896) 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 -- 172.21.15.33:0/2023446339 learned_addr learned my addr 172.21.15.33:0/2023446339 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc5c109a50 0x7fbc5c1a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc5b400700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c10fa50 0x7fbc5c1a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 -- 172.21.15.33:0/2023446339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc5c109a50 msgr2=0x7fbc5c1a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc5c109a50 0x7fbc5c1a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 -- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c10fa50 msgr2=0x7fbc5c1a49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c10fa50 0x7fbc5c1a49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.135+0000 7fbc53e00700 1 -- 172.21.15.33:0/2023446339 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c1a50f0 con 0x7fbc5c10a410 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.136+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c1a0670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400e9c0 tx=0x7fbc4400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.136+0000 7fbc53400700 1 -- 172.21.15.33:0/2023446339 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4400cbd0 con 0x7fbc5c10a410 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.136+0000 7fbc53400700 1 -- 172.21.15.33:0/2023446339 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc4400cd70 con 0x7fbc5c10a410 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.136+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c1a53e0 con 0x7fbc5c10a410 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.136+0000 7fbc53400700 1 -- 172.21.15.33:0/2023446339 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44005610 con 0x7fbc5c10a410 2024-04-26T11:31:04.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.136+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c1acc60 con 0x7fbc5c10a410 2024-04-26T11:31:04.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.140+0000 7fbc53400700 1 -- 172.21.15.33:0/2023446339 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7fbc44004750 con 0x7fbc5c10a410 2024-04-26T11:31:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.141+0000 7fbc53400700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbc400a3ed0 0x7fbc400a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.141+0000 7fbc6c68bb80 1 Processor -- start 2024-04-26T11:31:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.141+0000 7fbc53400700 1 -- 172.21.15.33:0/2023446339 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbc440d0e40 con 0x7fbc5c10a410 2024-04-26T11:31:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.141+0000 7fbc6c68bb80 1 -- start start 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbc400a3ed0 0x7fbc400a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f655790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f650b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f6514c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af6f5a3840 con 0x55af6f651080 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af6f542160 con 0x55af6f655cd0 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55af6f59f800 con 0x55af6f6553b0 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f650b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f650b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45900/0 (socket says 172.21.15.33:45900) 2024-04-26T11:31:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.142+0000 7fbc53e00700 1 -- 172.21.15.33:0/2073057302 learned_addr learned my addr 172.21.15.33:0/2073057302 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbc400a3ed0 0x7fbc400a6390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009750 tx=0x7fbc4c009310 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:04.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc53e00700 1 -- 172.21.15.33:0/2073057302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 msgr2=0x55af6f655790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2073057302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f655790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc53e00700 1 -- 172.21.15.33:0/2073057302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 msgr2=0x55af6f6514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2073057302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f6514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc53e00700 1 -- 172.21.15.33:0/2073057302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af6f54a5e0 con 0x55af6f655cd0 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2073057302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f650b40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc4409c840 tx=0x7fbc4409c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8da9d78bdc674f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.143+0000 7fbc4be00700 1 -- 172.21.15.33:0/2073057302 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc440b6070 con 0x55af6f655cd0 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.144+0000 7fbc4be00700 1 -- 172.21.15.33:0/2073057302 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc4409cb40 con 0x55af6f655cd0 2024-04-26T11:31:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.144+0000 7fbc4be00700 1 -- 172.21.15.33:0/2073057302 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4409cfc0 con 0x55af6f655cd0 2024-04-26T11:31:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2073057302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 msgr2=0x55af6f650b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2073057302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f650b40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc4409c840 tx=0x7fbc4409c750 comp rx=0 tx=0).stop 2024-04-26T11:31:04.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2073057302 shutdown_connections 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2073057302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f6514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2073057302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f655790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2073057302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f650b40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2073057302 >> 172.21.15.33:0/2073057302 conn(0x55af6f57e2a0 msgr2=0x55af6f64b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2073057302 shutdown_connections 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.145+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2073057302 wait complete. 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 -- start start 2024-04-26T11:31:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 Processor -- start 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f659e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f65a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af6f65d810 con 0x55af6f655cd0 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af6f53e3b0 con 0x55af6f6553b0 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.146+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55af6f65d990 con 0x55af6f651080 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc6c68bb80 10 client.0 _ll_register_callbacks cb 0x55af6f537330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f659e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc5b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f65a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f659e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45908/0 (socket says 172.21.15.33:45908) 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc5aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 -- 172.21.15.33:0/2255766392 learned_addr learned my addr 172.21.15.33:0/2255766392 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 -- 172.21.15.33:0/2255766392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 msgr2=0x55af6f65cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 -- 172.21.15.33:0/2255766392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 msgr2=0x55af6f65a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f65a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.147+0000 7fbc53e00700 1 -- 172.21.15.33:0/2255766392 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af6f4f0880 con 0x55af6f6553b0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc53e00700 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f659e50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400c5b0 tx=0x7fbc4400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ca2585afa24cc63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc48c00700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc48c00700 1 -- 172.21.15.33:0/2255766392 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc440b6040 con 0x55af6f6553b0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc48c00700 1 -- 172.21.15.33:0/2255766392 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbc440a5320 con 0x55af6f6553b0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc4409cb40 con 0x55af6f6553b0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc48c00700 1 -- 172.21.15.33:0/2255766392 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc440a57b0 con 0x55af6f6553b0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc6c68bb80 10 client.25247 fetch_fsmap learned FSMap version 26 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55af6f668110 con 0x55af6f6553b0 2024-04-26T11:31:04.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc48c00700 1 -- 172.21.15.33:0/2255766392 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc4409cb40 con 0x55af6f6553b0 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.148+0000 7fbc48c00700 1 -- 172.21.15.33:0/2255766392 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbc44098890 con 0x55af6f6553b0 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.149+0000 7fbc6c68bb80 10 client.25247 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.149+0000 7fbc6c68bb80 10 client.25247 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.149+0000 7fbc6c68bb80 20 client.25247 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.149+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f669460 0x55af6f66b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.149+0000 7fbc6c68bb80 4 client.25247 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.149+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 --> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af6f5b1220 con 0x55af6f669460 2024-04-26T11:31:04.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.149+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f669460 0x55af6f66b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.150+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f669460 0x55af6f66b5d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=341ce4bad96687cf server_cookie=873b98d0166ef262 in_seq=0 out_seq=0 2024-04-26T11:31:04.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.150+0000 7fbc48c00700 10 client.25247 ms_handle_connect on v2:172.21.15.96:6836/823522377 2024-04-26T11:31:04.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.150+0000 7fbc48c00700 1 -- 172.21.15.33:0/2255766392 <== mds.1 v2:172.21.15.96:6836/823522377 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af6f5b1220 con 0x55af6f669460 2024-04-26T11:31:04.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.150+0000 7fbc48c00700 10 client.25247 handle_command_reply: tid=0 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.150+0000 7fbc6c68bb80 1 client.25247 shutdown 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 msgr2=0x55af6f659e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f659e50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400c5b0 tx=0x7fbc4400ef10 comp rx=0 tx=0).stop 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 shutdown_connections 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f65a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f669460 0x55af6f66b5d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/2255766392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f659e50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc48c00700 0 client.25247 ms_handle_reset on v2:172.21.15.96:6836/823522377 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 >> 172.21.15.33:0/2255766392 conn(0x55af6f57e2a0 msgr2=0x55af6f645e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 shutdown_connections 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/2255766392 wait complete. 2024-04-26T11:31:04.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.151+0000 7fbc6c68bb80 20 client.25247 trim_cache size 0 max 16384 2024-04-26T11:31:04.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.190+0000 7fbc6c68bb80 1 Processor -- start 2024-04-26T11:31:04.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.190+0000 7fbc6c68bb80 1 -- start start 2024-04-26T11:31:04.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.191+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.191+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f657180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.191+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f6576c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.191+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af6f669910 con 0x55af6f655cd0 2024-04-26T11:31:04.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.191+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af6f669fe0 con 0x55af6f6553b0 2024-04-26T11:31:04.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.191+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55af6f66a6b0 con 0x55af6f651080 2024-04-26T11:31:04.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.192+0000 7fbc5aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.192+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f657180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.192+0000 7fbc5aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:59664/0 (socket says 172.21.15.33:59664) 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.192+0000 7fbc5aa00700 1 -- 172.21.15.33:0/1863660200 learned_addr learned my addr 172.21.15.33:0/1863660200 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.192+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f657180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45910/0 (socket says 172.21.15.33:45910) 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.192+0000 7fbc5b400700 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f6576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.193+0000 7fbc5aa00700 1 -- 172.21.15.33:0/1863660200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 msgr2=0x55af6f657180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.193+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f657180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.193+0000 7fbc5aa00700 1 -- 172.21.15.33:0/1863660200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 msgr2=0x55af6f6576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.193+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f6576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.193+0000 7fbc5aa00700 1 -- 172.21.15.33:0/1863660200 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af6f66c530 con 0x55af6f651080 2024-04-26T11:31:04.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.193+0000 7fbc53e00700 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f657180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:04.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.194+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65c690 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c005fd0 tx=0x7fbc4c013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c92be7501e122bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:04.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.194+0000 7fbc4b400700 1 -- 172.21.15.33:0/1863660200 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c029250 con 0x55af6f651080 2024-04-26T11:31:04.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.194+0000 7fbc4b400700 1 -- 172.21.15.33:0/1863660200 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc4c0298a0 con 0x55af6f651080 2024-04-26T11:31:04.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.194+0000 7fbc4b400700 1 -- 172.21.15.33:0/1863660200 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c004010 con 0x55af6f651080 2024-04-26T11:31:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.195+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/1863660200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 msgr2=0x55af6f65c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.195+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65c690 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c005fd0 tx=0x7fbc4c013040 comp rx=0 tx=0).stop 2024-04-26T11:31:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.195+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/1863660200 shutdown_connections 2024-04-26T11:31:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.195+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f655cd0 0x55af6f6576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.195+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f651080 0x55af6f65c690 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.195+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/1863660200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f6553b0 0x55af6f657180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.195+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/1863660200 >> 172.21.15.33:0/1863660200 conn(0x55af6f57e2a0 msgr2=0x55af6f64b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.196+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/1863660200 shutdown_connections 2024-04-26T11:31:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.196+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/1863660200 wait complete. 2024-04-26T11:31:04.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.197+0000 7fbc6c68bb80 1 -- start start 2024-04-26T11:31:04.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.198+0000 7fbc6c68bb80 1 Processor -- start 2024-04-26T11:31:04.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.199+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f674a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.199+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f674f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.199+0000 7fbc6c68bb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f66d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.199+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af6f66b450 con 0x55af6f651080 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af6f66ad80 con 0x55af6f655cd0 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc6c68bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55af6f66bb20 con 0x55af6f6553b0 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f674a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc6c68bb80 10 client.0 _ll_register_callbacks cb 0x55af6f537330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f674a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58788/0 (socket says 172.21.15.33:58788) 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 -- 172.21.15.33:0/3326409730 learned_addr learned my addr 172.21.15.33:0/3326409730 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc53e00700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f674f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5b400700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f66d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 -- 172.21.15.33:0/3326409730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 msgr2=0x55af6f674f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f674f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 -- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 msgr2=0x55af6f66d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f66d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc5aa00700 1 -- 172.21.15.33:0/3326409730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af6f66e230 con 0x55af6f651080 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.200+0000 7fbc53e00700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f674f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.201+0000 7fbc5aa00700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f674a50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0095c0 tx=0x7fbc4c0037e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71557ce781a24dea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.201+0000 7fbc48c00700 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.201+0000 7fbc48c00700 1 -- 172.21.15.33:0/3326409730 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c03f060 con 0x55af6f651080 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.201+0000 7fbc48c00700 1 -- 172.21.15.33:0/3326409730 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbc4c003eb0 con 0x55af6f651080 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.202+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc4c0298a0 con 0x55af6f651080 2024-04-26T11:31:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.202+0000 7fbc48c00700 1 -- 172.21.15.33:0/3326409730 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc4c005040 con 0x55af6f651080 2024-04-26T11:31:04.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.202+0000 7fbc6c68bb80 10 client.15129 fetch_fsmap learned FSMap version 26 2024-04-26T11:31:04.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.202+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55af6f6d1cd0 con 0x55af6f651080 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.202+0000 7fbc48c00700 1 -- 172.21.15.33:0/3326409730 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc4c0298a0 con 0x55af6f651080 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.203+0000 7fbc48c00700 1 -- 172.21.15.33:0/3326409730 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbc4c029550 con 0x55af6f651080 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.203+0000 7fbc6c68bb80 10 client.15129 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.203+0000 7fbc6c68bb80 10 client.15129 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.203+0000 7fbc6c68bb80 20 client.15129 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.203+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f6d64b0 0x55af6f6d51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.203+0000 7fbc6c68bb80 4 client.15129 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.203+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 --> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] -- command(tid 0: {"prefix": "status"}) -- 0x55af6f5ab2c0 con 0x55af6f6d64b0 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.204+0000 7fbc53e00700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f6d64b0 0x55af6f6d51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.204+0000 7fbc53e00700 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f6d64b0 0x55af6f6d51c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=625be41df44cb629 server_cookie=a191e5b3dcb9f34e in_seq=0 out_seq=0 2024-04-26T11:31:04.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.204+0000 7fbc48c00700 10 client.15129 ms_handle_connect on v2:172.21.15.96:6836/823522377 2024-04-26T11:31:04.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc48c00700 1 -- 172.21.15.33:0/3326409730 <== mds.1 v2:172.21.15.96:6836/823522377 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55af6f5ab2c0 con 0x55af6f6d64b0 2024-04-26T11:31:04.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc48c00700 10 client.15129 handle_command_reply: tid=0 2024-04-26T11:31:04.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 client.15129 shutdown 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 msgr2=0x55af6f674a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f674a50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0095c0 tx=0x7fbc4c0037e0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 shutdown_connections 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:6836/823522377,v1:172.21.15.96:6837/823522377] conn(0x55af6f6d64b0 0x55af6f6d51c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc48c00700 0 client.15129 ms_handle_reset on v2:172.21.15.96:6836/823522377 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af6f655cd0 0x55af6f66d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af6f651080 0x55af6f674a50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 --2- 172.21.15.33:0/3326409730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af6f6553b0 0x55af6f674f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 >> 172.21.15.33:0/3326409730 conn(0x55af6f57e2a0 msgr2=0x55af6f653f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.205+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 shutdown_connections 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.206+0000 7fbc6c68bb80 1 -- 172.21.15.33:0/3326409730 wait complete. 2024-04-26T11:31:04.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.206+0000 7fbc6c68bb80 20 client.15129 trim_cache size 0 max 16384 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24488, 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 255.119220587, 2024-04-26T11:31:04.208 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 26, 2024-04-26T11:31:04.209 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 91, 2024-04-26T11:31:04.209 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T11:31:04.209 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 309.15185279299999 2024-04-26T11:31:04.209 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbc400a3ed0 msgr2=0x7fbc400a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbc400a3ed0 0x7fbc400a6390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009750 tx=0x7fbc4c009310 comp rx=0 tx=0).stop 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 msgr2=0x7fbc5c1a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c1a0670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400e9c0 tx=0x7fbc4400ecd0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 shutdown_connections 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbc400a3ed0 0x7fbc400a6390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c10fa50 0x7fbc5c1a49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc5c109a50 0x7fbc5c1a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 --2- 172.21.15.33:0/2023446339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c10a410 0x7fbc5c1a0670 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 >> 172.21.15.33:0/2023446339 conn(0x7fbc5c076bf0 msgr2=0x7fbc5c0771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:04.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.207+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 shutdown_connections 2024-04-26T11:31:04.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:04.208+0000 7fbc61c00700 1 -- 172.21.15.33:0/2023446339 wait complete. 2024-04-26T11:31:04.271 INFO:teuthology.orchestra.run.smithi033.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.69 s, remaining 6.71 s) 2024-04-26T11:31:04.559 INFO:teuthology.orchestra.run.smithi033.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.98 s, remaining 6.87 s) 2024-04-26T11:31:04.846 INFO:teuthology.orchestra.run.smithi033.stderr:500000 of 3200000 tuples (15%) done (elapsed 1.27 s, remaining 6.86 s) 2024-04-26T11:31:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:04 smithi194 ceph-mon[33017]: pgmap v23: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-26T11:31:04.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:04 smithi033 ceph-mon[27845]: pgmap v23: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-26T11:31:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:04 smithi096 ceph-mon[32753]: pgmap v23: 97 pgs: 97 active+clean; 22 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-26T11:31:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:600000 of 3200000 tuples (18%) done (elapsed 1.43 s, remaining 6.21 s) 2024-04-26T11:31:05.390 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T11:31:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:700000 of 3200000 tuples (21%) done (elapsed 1.81 s, remaining 6.47 s) 2024-04-26T11:31:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.652+0000 7f440ae00640 1 -- 172.21.15.33:0/4021192518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404101eb0 msgr2=0x7f440410eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.652+0000 7f440ae00640 1 --2- 172.21.15.33:0/4021192518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404101eb0 0x7f440410eae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009900 tx=0x7f43f002a410 comp rx=0 tx=0).stop 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.653+0000 7f440ae00640 1 -- 172.21.15.33:0/4021192518 shutdown_connections 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.653+0000 7f440ae00640 1 --2- 172.21.15.33:0/4021192518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f440410f0b0 0x7f44041114a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.653+0000 7f440ae00640 1 --2- 172.21.15.33:0/4021192518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4404101eb0 0x7f440410eae0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.653+0000 7f440ae00640 1 --2- 172.21.15.33:0/4021192518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4404101500 0x7f44041018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.653+0000 7f440ae00640 1 -- 172.21.15.33:0/4021192518 >> 172.21.15.33:0/4021192518 conn(0x7f44040fd150 msgr2=0x7f44040ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.653+0000 7f440ae00640 1 -- 172.21.15.33:0/4021192518 shutdown_connections 2024-04-26T11:31:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.653+0000 7f440ae00640 1 -- 172.21.15.33:0/4021192518 wait complete. 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 Processor -- start 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 -- start start 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4404101500 0x7f440419b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4404101eb0 0x7f440419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440410f0b0 0x7f440419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4404114c40 con 0x7f4404101500 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4404114ac0 con 0x7f440410f0b0 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.654+0000 7f440ae00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4404114dc0 con 0x7f4404101eb0 2024-04-26T11:31:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440410f0b0 0x7f440419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f4409000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4404101eb0 0x7f440419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440410f0b0 0x7f440419fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45916/0 (socket says 172.21.15.33:45916) 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 -- 172.21.15.33:0/685139295 learned_addr learned my addr 172.21.15.33:0/685139295 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f4409000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4404101eb0 0x7f440419b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:59692/0 (socket says 172.21.15.33:59692) 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f4409a00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4404101500 0x7f440419b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 -- 172.21.15.33:0/685139295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4404101eb0 msgr2=0x7f440419b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4404101eb0 0x7f440419b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 -- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4404101500 msgr2=0x7f440419b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4404101500 0x7f440419b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f440a400640 1 -- 172.21.15.33:0/685139295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44041a0210 con 0x7f440410f0b0 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.655+0000 7f4409000640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4404101eb0 0x7f440419b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.656+0000 7f440a400640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440410f0b0 0x7f440419fae0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00eb10 tx=0x7f43fc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.656+0000 7f4402000640 1 -- 172.21.15.33:0/685139295 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc004b20 con 0x7f440410f0b0 2024-04-26T11:31:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.656+0000 7f4402000640 1 -- 172.21.15.33:0/685139295 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43fc004cc0 con 0x7f440410f0b0 2024-04-26T11:31:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.656+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44041a0500 con 0x7f440410f0b0 2024-04-26T11:31:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.656+0000 7f4402000640 1 -- 172.21.15.33:0/685139295 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc0107b0 con 0x7f440410f0b0 2024-04-26T11:31:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.656+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44041a7d90 con 0x7f440410f0b0 2024-04-26T11:31:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.659+0000 7f4402000640 1 -- 172.21.15.33:0/685139295 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f43fc005020 con 0x7f440410f0b0 2024-04-26T11:31:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.661+0000 7f440f27d740 1 Processor -- start 2024-04-26T11:31:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f440f27d740 1 -- start start 2024-04-26T11:31:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f440f27d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d78fe0 0x561969d793c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f4402000640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f43ec0a3eb0 0x7f43ec0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f440f27d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d83c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f440f27d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d841c0 0x561969d865b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f440f27d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561969cde380 con 0x561969d79900 2024-04-26T11:31:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f440f27d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561969c7ac30 con 0x561969d841c0 2024-04-26T11:31:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f440f27d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561969cda340 con 0x561969d78fe0 2024-04-26T11:31:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.662+0000 7f4402000640 1 -- 172.21.15.33:0/685139295 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f43fc0d10c0 con 0x7f440410f0b0 2024-04-26T11:31:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d78fe0 0x561969d793c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f440a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d841c0 0x561969d865b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d78fe0 0x561969d793c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:59694/0 (socket says 172.21.15.33:59694) 2024-04-26T11:31:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 -- 172.21.15.33:0/1046783813 learned_addr learned my addr 172.21.15.33:0/1046783813 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f440a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d841c0 0x561969d865b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45926/0 (socket says 172.21.15.33:45926) 2024-04-26T11:31:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d83c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409a00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f43ec0a3eb0 0x7f43ec0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 -- 172.21.15.33:0/1046783813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d841c0 msgr2=0x561969d865b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d841c0 0x561969d865b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 -- 172.21.15.33:0/1046783813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 msgr2=0x561969d83c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d83c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.663+0000 7f4409000640 1 -- 172.21.15.33:0/1046783813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561969c9c940 con 0x561969d78fe0 2024-04-26T11:31:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.664+0000 7f4409000640 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d78fe0 0x561969d793c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43f0005e00 tx=0x7f43f0005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a43f792b755cfebe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.664+0000 7f4409a00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f43ec0a3eb0 0x7f43ec0a6370 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009bd0 tx=0x7f43f8005f90 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.664+0000 7f43f6a00640 1 -- 172.21.15.33:0/1046783813 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0035030 con 0x561969d78fe0 2024-04-26T11:31:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.664+0000 7f43f6a00640 1 -- 172.21.15.33:0/1046783813 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43f00054d0 con 0x561969d78fe0 2024-04-26T11:31:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.664+0000 7f43f6a00640 1 -- 172.21.15.33:0/1046783813 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0004a60 con 0x561969d78fe0 2024-04-26T11:31:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.665+0000 7f440f27d740 1 -- 172.21.15.33:0/1046783813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d78fe0 msgr2=0x561969d793c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.665+0000 7f440f27d740 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d78fe0 0x561969d793c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43f0005e00 tx=0x7f43f0005d90 comp rx=0 tx=0).stop 2024-04-26T11:31:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.665+0000 7f440f27d740 1 -- 172.21.15.33:0/1046783813 shutdown_connections 2024-04-26T11:31:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.665+0000 7f440f27d740 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d841c0 0x561969d865b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.665+0000 7f440f27d740 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d83c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.666+0000 7f440f27d740 1 --2- 172.21.15.33:0/1046783813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d78fe0 0x561969d793c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.666+0000 7f440f27d740 1 -- 172.21.15.33:0/1046783813 >> 172.21.15.33:0/1046783813 conn(0x561969d754f0 msgr2=0x561969d758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.666+0000 7f440f27d740 1 -- 172.21.15.33:0/1046783813 shutdown_connections 2024-04-26T11:31:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.666+0000 7f440f27d740 1 -- 172.21.15.33:0/1046783813 wait complete. 2024-04-26T11:31:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.666+0000 7f440f27d740 1 -- start start 2024-04-26T11:31:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.667+0000 7f440f27d740 1 Processor -- start 2024-04-26T11:31:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.667+0000 7f440f27d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d81140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f440f27d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d81680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f440f27d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d81bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f440f27d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561969d86d90 con 0x561969d841c0 2024-04-26T11:31:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f440f27d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561969c65c60 con 0x561969d78fe0 2024-04-26T11:31:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f440f27d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561969d86f10 con 0x561969d79900 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f440f27d740 10 client.0 _ll_register_callbacks cb 0x561969c91e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d81140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d81140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45928/0 (socket says 172.21.15.33:45928) 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 -- 172.21.15.33:0/2198152338 learned_addr learned my addr 172.21.15.33:0/2198152338 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f440a400640 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d81bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409000640 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d81680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 -- 172.21.15.33:0/2198152338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 msgr2=0x561969d81680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d81680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 -- 172.21.15.33:0/2198152338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 msgr2=0x561969d81bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d81bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.668+0000 7f4409a00640 1 -- 172.21.15.33:0/2198152338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561969c74510 con 0x561969d78fe0 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f4409a00640 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d81140 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f43f800ecd0 tx=0x7f43f8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1967e85b8419607 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f43eb400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f43eb400640 1 -- 172.21.15.33:0/2198152338 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8004220 con 0x561969d78fe0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f43eb400640 1 -- 172.21.15.33:0/2198152338 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f43f80043c0 con 0x561969d78fe0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43f00054d0 con 0x561969d78fe0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f43eb400640 1 -- 172.21.15.33:0/2198152338 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f43f801c8a0 con 0x561969d78fe0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f440f27d740 10 client.25265 fetch_fsmap learned FSMap version 26 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.669+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5619699f0210 con 0x561969d78fe0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f43eb400640 1 -- 172.21.15.33:0/2198152338 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43f00054d0 con 0x561969d78fe0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f43eb400640 1 -- 172.21.15.33:0/2198152338 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f43f8027430 con 0x561969d78fe0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f440f27d740 10 client.25265 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f440f27d740 10 client.25265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f440f27d740 20 client.25265 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f440f27d740 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969d93ef0 0x561969d962e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f440f27d740 4 client.25265 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.670+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561969cebf90 con 0x561969d93ef0 2024-04-26T11:31:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.671+0000 7f4409000640 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969d93ef0 0x561969d962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.671+0000 7f4409000640 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969d93ef0 0x561969d962e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbdcce4e6b692fcb server_cookie=c87c92c4b3205870 in_seq=0 out_seq=0 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.671+0000 7f43eb400640 10 client.25265 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f43eb400640 1 -- 172.21.15.33:0/2198152338 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561969cebf90 con 0x561969d93ef0 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f43eb400640 10 client.25265 handle_command_reply: tid=0 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 client.25265 shutdown 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 msgr2=0x561969d81140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d81140 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f43f800ecd0 tx=0x7f43f8004730 comp rx=0 tx=0).stop 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 shutdown_connections 2024-04-26T11:31:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d81680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969d93ef0 0x561969d962e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d81bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 --2- 172.21.15.33:0/2198152338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d81140 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f43eb400640 0 client.25265 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 >> 172.21.15.33:0/2198152338 conn(0x561969d754f0 msgr2=0x561969d7b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.672+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 shutdown_connections 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.673+0000 7f440f27d740 1 -- 172.21.15.33:0/2198152338 wait complete. 2024-04-26T11:31:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.673+0000 7f440f27d740 20 client.25265 trim_cache size 0 max 16384 2024-04-26T11:31:05.699 INFO:teuthology.orchestra.run.smithi033.stderr:800000 of 3200000 tuples (25%) done (elapsed 2.12 s, remaining 6.37 s) 2024-04-26T11:31:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.707+0000 7f440f27d740 1 Processor -- start 2024-04-26T11:31:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.708+0000 7f440f27d740 1 -- start start 2024-04-26T11:31:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.708+0000 7f440f27d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d8fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.708+0000 7f440f27d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d7b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.708+0000 7f440f27d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d841c0 0x561969d9bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.708+0000 7f440f27d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44041b2ec0 con 0x561969d79900 2024-04-26T11:31:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.708+0000 7f440f27d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44041b0790 con 0x561969d78fe0 2024-04-26T11:31:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.708+0000 7f440f27d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44041ae5e0 con 0x561969d841c0 2024-04-26T11:31:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d8fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d7b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d8fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45936/0 (socket says 172.21.15.33:45936) 2024-04-26T11:31:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 -- 172.21.15.33:0/3861154516 learned_addr learned my addr 172.21.15.33:0/3861154516 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409000640 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d7b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58816/0 (socket says 172.21.15.33:58816) 2024-04-26T11:31:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f440a400640 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d841c0 0x561969d9bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 -- 172.21.15.33:0/3861154516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d841c0 msgr2=0x561969d9bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d841c0 0x561969d9bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 -- 172.21.15.33:0/3861154516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 msgr2=0x561969d7b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d7b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f4409a00640 1 -- 172.21.15.33:0/3861154516 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561969abc620 con 0x561969d78fe0 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.709+0000 7f440a400640 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d841c0 0x561969d9bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.710+0000 7f4409a00640 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d8fc50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f43f8016040 tx=0x7f43f800e8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d7622cc97f5668b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.710+0000 7f43f6000640 1 -- 172.21.15.33:0/3861154516 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f803d070 con 0x561969d78fe0 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.710+0000 7f43f6000640 1 -- 172.21.15.33:0/3861154516 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43f8027ac0 con 0x561969d78fe0 2024-04-26T11:31:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.710+0000 7f43f6000640 1 -- 172.21.15.33:0/3861154516 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8034070 con 0x561969d78fe0 2024-04-26T11:31:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 -- 172.21.15.33:0/3861154516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 msgr2=0x561969d8fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d8fc50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f43f8016040 tx=0x7f43f800e8b0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 -- 172.21.15.33:0/3861154516 shutdown_connections 2024-04-26T11:31:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d841c0 0x561969d9bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d79900 0x561969d7b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 --2- 172.21.15.33:0/3861154516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d8fc50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 -- 172.21.15.33:0/3861154516 >> 172.21.15.33:0/3861154516 conn(0x561969d754f0 msgr2=0x561969d830f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 -- 172.21.15.33:0/3861154516 shutdown_connections 2024-04-26T11:31:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.711+0000 7f440f27d740 1 -- 172.21.15.33:0/3861154516 wait complete. 2024-04-26T11:31:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.712+0000 7f440f27d740 1 -- start start 2024-04-26T11:31:05.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.712+0000 7f440f27d740 1 Processor -- start 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f440f27d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d7df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f440f27d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d7e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f440f27d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d7e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f440f27d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44041aec50 con 0x561969d841c0 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f440f27d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44041b00c0 con 0x561969d78fe0 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f440f27d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44041af320 con 0x561969d79900 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f440f27d740 10 client.0 _ll_register_callbacks cb 0x7f43f8009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f4409a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d7df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f4409a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d7df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45942/0 (socket says 172.21.15.33:45942) 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f4409a00640 1 -- 172.21.15.33:0/2107692237 learned_addr learned my addr 172.21.15.33:0/2107692237 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.713+0000 7f4409000640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d7e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f440a400640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d7e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f4409a00640 1 -- 172.21.15.33:0/2107692237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 msgr2=0x561969d7e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f4409a00640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d7e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f4409a00640 1 -- 172.21.15.33:0/2107692237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 msgr2=0x561969d7e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f4409a00640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d7e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f4409a00640 1 -- 172.21.15.33:0/2107692237 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561969c98cb0 con 0x561969d78fe0 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f440a400640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d7e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f4409a00640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d7df30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f43f800a010 tx=0x7f43f8014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b376617d76e12d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f43eb400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f43eb400640 1 -- 172.21.15.33:0/2107692237 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f803d030 con 0x561969d78fe0 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f43eb400640 1 -- 172.21.15.33:0/2107692237 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f43f801cdd0 con 0x561969d78fe0 2024-04-26T11:31:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.714+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43f8027ac0 con 0x561969d78fe0 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f43eb400640 1 -- 172.21.15.33:0/2107692237 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f43f8034040 con 0x561969d78fe0 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 10 client.25271 fetch_fsmap learned FSMap version 26 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561969dfa650 con 0x561969d78fe0 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f43eb400640 1 -- 172.21.15.33:0/2107692237 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43f8027ac0 con 0x561969d78fe0 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f43eb400640 1 -- 172.21.15.33:0/2107692237 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f43f80444c0 con 0x561969d78fe0 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 10 client.25271 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 10 client.25271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 20 client.25271 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969dfac60 0x561969dfd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 4 client.25271 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:31:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561969ce5e00 con 0x561969dfac60 2024-04-26T11:31:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.715+0000 7f4409000640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969dfac60 0x561969dfd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.716+0000 7f4409000640 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969dfac60 0x561969dfd080 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=412ddeb05037b854 server_cookie=b9bf6db5267c62b0 in_seq=0 out_seq=0 2024-04-26T11:31:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.716+0000 7f43eb400640 10 client.25271 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.975+0000 7f43eb400640 1 -- 172.21.15.33:0/2107692237 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561969ce5e00 con 0x561969dfac60 2024-04-26T11:31:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.975+0000 7f43eb400640 10 client.25271 handle_command_reply: tid=0 2024-04-26T11:31:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.975+0000 7f440f27d740 1 client.25271 shutdown 2024-04-26T11:31:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 msgr2=0x561969d7df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d7df30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f43f800a010 tx=0x7f43f8014000 comp rx=0 tx=0).stop 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 shutdown_connections 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561969d79900 0x561969d7e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561969dfac60 0x561969dfd080 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561969d841c0 0x561969d7e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 --2- 172.21.15.33:0/2107692237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561969d78fe0 0x561969d7df30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f43eb400640 0 client.25271 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 >> 172.21.15.33:0/2107692237 conn(0x561969d754f0 msgr2=0x561969d73dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 shutdown_connections 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 1 -- 172.21.15.33:0/2107692237 wait complete. 2024-04-26T11:31:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.976+0000 7f440f27d740 20 client.25271 trim_cache size 0 max 16384 2024-04-26T11:31:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:31:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:31:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:31:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f43ec0a3eb0 msgr2=0x7f43ec0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f43ec0a3eb0 0x7f43ec0a6370 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009bd0 tx=0x7f43f8005f90 comp rx=0 tx=0).stop 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440410f0b0 msgr2=0x7f440419fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440410f0b0 0x7f440419fae0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00eb10 tx=0x7f43fc00c2d0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 shutdown_connections 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f43ec0a3eb0 0x7f43ec0a6370 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4404101eb0 0x7f440419b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440410f0b0 0x7f440419fae0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 --2- 172.21.15.33:0/685139295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4404101500 0x7f440419b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 >> 172.21.15.33:0/685139295 conn(0x7f44040fd150 msgr2=0x7f44040ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 shutdown_connections 2024-04-26T11:31:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:05.977+0000 7f440ae00640 1 -- 172.21.15.33:0/685139295 wait complete. 2024-04-26T11:31:06.010 INFO:teuthology.orchestra.run.smithi033.stderr:900000 of 3200000 tuples (28%) done (elapsed 2.43 s, remaining 6.22 s) 2024-04-26T11:31:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:31:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:1000000 of 3200000 tuples (31%) done (elapsed 2.77 s, remaining 6.10 s) 2024-04-26T11:31:06.774 INFO:teuthology.orchestra.run.smithi033.stderr:1100000 of 3200000 tuples (34%) done (elapsed 3.20 s, remaining 6.10 s) 2024-04-26T11:31:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:06 smithi194 ceph-mon[33017]: pgmap v24: 97 pgs: 97 active+clean; 72 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 6.8 MiB/s wr, 171 op/s 2024-04-26T11:31:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:06 smithi033 ceph-mon[27845]: pgmap v24: 97 pgs: 97 active+clean; 72 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 6.8 MiB/s wr, 171 op/s 2024-04-26T11:31:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:06 smithi096 ceph-mon[32753]: pgmap v24: 97 pgs: 97 active+clean; 72 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 6.8 MiB/s wr, 171 op/s 2024-04-26T11:31:06.958 INFO:teuthology.orchestra.run.smithi033.stderr:1200000 of 3200000 tuples (37%) done (elapsed 3.38 s, remaining 5.64 s) 2024-04-26T11:31:07.378 INFO:teuthology.orchestra.run.smithi033.stderr:1300000 of 3200000 tuples (40%) done (elapsed 3.80 s, remaining 5.56 s) 2024-04-26T11:31:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:07 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:31:07.699 INFO:teuthology.orchestra.run.smithi033.stderr:1400000 of 3200000 tuples (43%) done (elapsed 4.12 s, remaining 5.30 s) 2024-04-26T11:31:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:07 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:31:07.882 INFO:teuthology.orchestra.run.smithi033.stderr:1500000 of 3200000 tuples (46%) done (elapsed 4.31 s, remaining 4.88 s) 2024-04-26T11:31:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:07 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.258+0000 7faa42a00700 1 -- 172.21.15.33:0/3882525776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c1094a0 msgr2=0x7faa3c109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.258+0000 7faa42a00700 1 --2- 172.21.15.33:0/3882525776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c1094a0 0x7faa3c109880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0098f0 tx=0x7faa2c009c00 comp rx=0 tx=0).stop 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.258+0000 7faa42a00700 1 -- 172.21.15.33:0/3882525776 shutdown_connections 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.258+0000 7faa42a00700 1 --2- 172.21.15.33:0/3882525776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa3c06c4f0 0x7faa3c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.258+0000 7faa42a00700 1 --2- 172.21.15.33:0/3882525776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c1094a0 0x7faa3c109880 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.258+0000 7faa42a00700 1 --2- 172.21.15.33:0/3882525776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3c06bb50 0x7faa3c06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.258+0000 7faa42a00700 1 -- 172.21.15.33:0/3882525776 >> 172.21.15.33:0/3882525776 conn(0x7faa3c0792d0 msgr2=0x7faa3c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.259+0000 7faa42a00700 1 -- 172.21.15.33:0/3882525776 shutdown_connections 2024-04-26T11:31:08.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.259+0000 7faa42a00700 1 -- 172.21.15.33:0/3882525776 wait complete. 2024-04-26T11:31:08.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.260+0000 7faa42a00700 1 Processor -- start 2024-04-26T11:31:08.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.260+0000 7faa42a00700 1 -- start start 2024-04-26T11:31:08.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa42a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c06bb50 0x7faa3c102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa42a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3c06c4f0 0x7faa3c102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa42a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa3c1094a0 0x7faa3c102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa42a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c113600 con 0x7faa3c06bb50 2024-04-26T11:31:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa42a00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c113480 con 0x7faa3c06c4f0 2024-04-26T11:31:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa3be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c06bb50 0x7faa3c102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa3be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c06bb50 0x7faa3c102420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58840/0 (socket says 172.21.15.33:58840) 2024-04-26T11:31:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa3be00700 1 -- 172.21.15.33:0/1370492793 learned_addr learned my addr 172.21.15.33:0/1370492793 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c113780 con 0x7faa3c1094a0 2024-04-26T11:31:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa40a00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa3c1094a0 0x7faa3c102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.261+0000 7faa3b400700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3c06c4f0 0x7faa3c102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.262+0000 7faa3be00700 1 -- 172.21.15.33:0/1370492793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa3c1094a0 msgr2=0x7faa3c102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.262+0000 7faa3be00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa3c1094a0 0x7faa3c102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.262+0000 7faa3be00700 1 -- 172.21.15.33:0/1370492793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3c06c4f0 msgr2=0x7faa3c102960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.262+0000 7faa3be00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3c06c4f0 0x7faa3c102960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.262+0000 7faa3be00700 1 -- 172.21.15.33:0/1370492793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c1a80e0 con 0x7faa3c06bb50 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.262+0000 7faa3b400700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3c06c4f0 0x7faa3c102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.263+0000 7faa3be00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c06bb50 0x7faa3c102420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009230 tx=0x7faa2c0094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.263+0000 7faa38c00700 1 -- 172.21.15.33:0/1370492793 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c005140 con 0x7faa3c06bb50 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.263+0000 7faa38c00700 1 -- 172.21.15.33:0/1370492793 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa2c00b560 con 0x7faa3c06bb50 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.263+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3c1a8370 con 0x7faa3c06bb50 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.263+0000 7faa38c00700 1 -- 172.21.15.33:0/1370492793 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c003a80 con 0x7faa3c06bb50 2024-04-26T11:31:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.263+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa3c1a87c0 con 0x7faa3c06bb50 2024-04-26T11:31:08.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.266+0000 7faa38c00700 1 -- 172.21.15.33:0/1370492793 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7faa2c003c40 con 0x7faa3c06bb50 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.269+0000 7faa38c00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faa200a3ed0 0x7faa200a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.269+0000 7faa4d37eb80 1 Processor -- start 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.269+0000 7faa38c00700 1 -- 172.21.15.33:0/1370492793 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faa2c0d1990 con 0x7faa3c06bb50 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.269+0000 7faa3b400700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faa200a3ed0 0x7faa200a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.269+0000 7faa4d37eb80 1 -- start start 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.270+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86c2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.270+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.270+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86be080 0x55b2f86be4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.270+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b2f8610840 con 0x55b2f86c23b0 2024-04-26T11:31:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.270+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b2f85af160 con 0x55b2f86c2cd0 2024-04-26T11:31:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.270+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b2f860c800 con 0x55b2f86be080 2024-04-26T11:31:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.270+0000 7faa3b400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45978/0 (socket says 172.21.15.33:45978) 2024-04-26T11:31:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 -- 172.21.15.33:0/3866175469 learned_addr learned my addr 172.21.15.33:0/3866175469 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faa200a3ed0 0x7faa200a6390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa28009750 tx=0x7faa28009310 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3be00700 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86c2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 -- 172.21.15.33:0/3866175469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86be080 msgr2=0x55b2f86be4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86be080 0x55b2f86be4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 -- 172.21.15.33:0/3866175469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 msgr2=0x55b2f86c2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86c2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 -- 172.21.15.33:0/3866175469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2f85b75e0 con 0x55b2f86c2cd0 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.271+0000 7faa3b400700 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bdb40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faa2800f600 tx=0x7faa28014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89ef2cf34325b045 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.272+0000 7faa31600700 1 -- 172.21.15.33:0/3866175469 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28004230 con 0x55b2f86c2cd0 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.272+0000 7faa31600700 1 -- 172.21.15.33:0/3866175469 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa280043d0 con 0x55b2f86c2cd0 2024-04-26T11:31:08.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.272+0000 7faa31600700 1 -- 172.21.15.33:0/3866175469 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2801c8d0 con 0x55b2f86c2cd0 2024-04-26T11:31:08.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3866175469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 msgr2=0x55b2f86bdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bdb40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faa2800f600 tx=0x7faa28014000 comp rx=0 tx=0).stop 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3866175469 shutdown_connections 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86be080 0x55b2f86be4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86c2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3866175469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bdb40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3866175469 >> 172.21.15.33:0/3866175469 conn(0x55b2f85eb2a0 msgr2=0x55b2f86b8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3866175469 shutdown_connections 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.273+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3866175469 wait complete. 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 -- start start 2024-04-26T11:31:08.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 Processor -- start 2024-04-26T11:31:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c2cd0 0x55b2f86c7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86ca810 con 0x55b2f86c2cd0 2024-04-26T11:31:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b2f85ab3b0 con 0x55b2f86be080 2024-04-26T11:31:08.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.274+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86ca990 con 0x55b2f86c23b0 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa4d37eb80 10 client.0 _ll_register_callbacks cb 0x55b2f85a4330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45994/0 (socket says 172.21.15.33:45994) 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 -- 172.21.15.33:0/1346561255 learned_addr learned my addr 172.21.15.33:0/1346561255 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa40a00700 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c2cd0 0x55b2f86c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 -- 172.21.15.33:0/1346561255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 msgr2=0x55b2f86c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 -- 172.21.15.33:0/1346561255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c2cd0 msgr2=0x55b2f86c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c2cd0 0x55b2f86c7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 -- 172.21.15.33:0/1346561255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2f855d880 con 0x55b2f86be080 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa40a00700 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c2cd0 0x55b2f86c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:08.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa3be00700 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c99d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faa2c09d6b0 tx=0x7faa2c09d100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31302139d8eafe85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.275+0000 7faa26000700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa26000700 1 -- 172.21.15.33:0/1346561255 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c09db80 con 0x55b2f86be080 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa26000700 1 -- 172.21.15.33:0/1346561255 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faa2c0a5e70 con 0x55b2f86be080 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa280043d0 con 0x55b2f86be080 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa26000700 1 -- 172.21.15.33:0/1346561255 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faa2c09d720 con 0x55b2f86be080 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa4d37eb80 10 client.25283 fetch_fsmap learned FSMap version 27 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b2f86d5020 con 0x55b2f86be080 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa26000700 1 -- 172.21.15.33:0/1346561255 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa280043d0 con 0x55b2f86be080 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa26000700 1 -- 172.21.15.33:0/1346561255 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7faa2c0cf1f0 con 0x55b2f86be080 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.276+0000 7faa4d37eb80 10 client.25283 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.277+0000 7faa4d37eb80 10 client.25283 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.277+0000 7faa4d37eb80 20 client.25283 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.277+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f86d7770 0x55b2f86d8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.277+0000 7faa4d37eb80 4 client.25283 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:08.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.277+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2f861e220 con 0x55b2f86d7770 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.277+0000 7faa3b400700 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f86d7770 0x55b2f86d8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.278+0000 7faa3b400700 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f86d7770 0x55b2f86d8be0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=911fbe5eb199c338 server_cookie=9cf14107d20650d7 in_seq=0 out_seq=0 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.278+0000 7faa26000700 10 client.25283 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.278+0000 7faa26000700 1 -- 172.21.15.33:0/1346561255 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2f861e220 con 0x55b2f86d7770 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.278+0000 7faa26000700 10 client.25283 handle_command_reply: tid=0 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 client.25283 shutdown 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 msgr2=0x55b2f86c99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c99d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faa2c09d6b0 tx=0x7faa2c09d100 comp rx=0 tx=0).stop 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 shutdown_connections 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f86d7770 0x55b2f86d8be0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c2cd0 0x55b2f86c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c99d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/1346561255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 >> 172.21.15.33:0/1346561255 conn(0x55b2f85eb2a0 msgr2=0x55b2f86b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa26000700 0 client.25283 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 shutdown_connections 2024-04-26T11:31:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 1 -- 172.21.15.33:0/1346561255 wait complete. 2024-04-26T11:31:08.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.279+0000 7faa4d37eb80 20 client.25283 trim_cache size 0 max 16384 2024-04-26T11:31:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 Processor -- start 2024-04-26T11:31:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 -- start start 2024-04-26T11:31:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86be080 0x55b2f86b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86d6f20 con 0x55b2f86be080 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86d75f0 con 0x55b2f86c2cd0 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.302+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86d7cc0 con 0x55b2f86c23b0 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86be080 0x55b2f86b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa40a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:59740/0 (socket says 172.21.15.33:59740) 2024-04-26T11:31:08.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 -- 172.21.15.33:0/775561122 learned_addr learned my addr 172.21.15.33:0/775561122 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 -- 172.21.15.33:0/775561122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 msgr2=0x55b2f86bbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 --2- 172.21.15.33:0/775561122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 -- 172.21.15.33:0/775561122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86be080 msgr2=0x55b2f86b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 --2- 172.21.15.33:0/775561122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86be080 0x55b2f86b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.303+0000 7faa3b400700 1 -- 172.21.15.33:0/775561122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2f86be8d0 con 0x55b2f86c23b0 2024-04-26T11:31:08.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.304+0000 7faa3b400700 1 --2- 172.21.15.33:0/775561122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c9510 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faa2801c480 tx=0x7faa28005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6fee7b7d645f2ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.304+0000 7faa30c00700 1 -- 172.21.15.33:0/775561122 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2803a070 con 0x55b2f86c23b0 2024-04-26T11:31:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.304+0000 7faa30c00700 1 -- 172.21.15.33:0/775561122 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa280347b0 con 0x55b2f86c23b0 2024-04-26T11:31:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.304+0000 7faa30c00700 1 -- 172.21.15.33:0/775561122 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2801cbb0 con 0x55b2f86c23b0 2024-04-26T11:31:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 -- 172.21.15.33:0/775561122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 msgr2=0x55b2f86c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/775561122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c9510 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faa2801c480 tx=0x7faa28005a90 comp rx=0 tx=0).stop 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 -- 172.21.15.33:0/775561122 shutdown_connections 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/775561122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86c2cd0 0x55b2f86bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/775561122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86be080 0x55b2f86b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/775561122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c23b0 0x55b2f86c9510 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 -- 172.21.15.33:0/775561122 >> 172.21.15.33:0/775561122 conn(0x55b2f85eb2a0 msgr2=0x55b2f86b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 -- 172.21.15.33:0/775561122 shutdown_connections 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 -- 172.21.15.33:0/775561122 wait complete. 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.305+0000 7faa4d37eb80 1 -- start start 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 1 Processor -- start 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86e2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c2cd0 0x55b2f86e25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86d8a60 con 0x55b2f86c23b0 2024-04-26T11:31:08.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86d8390 con 0x55b2f86be080 2024-04-26T11:31:08.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b2f86d9130 con 0x55b2f86c2cd0 2024-04-26T11:31:08.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa4d37eb80 10 client.0 _ll_register_callbacks cb 0x55b2f85a4330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.306+0000 7faa3b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86e2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86e2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58890/0 (socket says 172.21.15.33:58890) 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 -- 172.21.15.33:0/3586949623 learned_addr learned my addr 172.21.15.33:0/3586949623 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3be00700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa40a00700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c2cd0 0x55b2f86e25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 -- 172.21.15.33:0/3586949623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c2cd0 msgr2=0x55b2f86e25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c2cd0 0x55b2f86e25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 -- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 msgr2=0x55b2f86c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 -- 172.21.15.33:0/3586949623 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2f86bb6e0 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa40a00700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c2cd0 0x55b2f86e25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa3b400700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86e2080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7faa2800f5d0 tx=0x7faa2800ff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6440528964d5a5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.307+0000 7faa26000700 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.308+0000 7faa26000700 1 -- 172.21.15.33:0/3586949623 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2803a070 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.308+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa280347b0 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.308+0000 7faa26000700 1 -- 172.21.15.33:0/3586949623 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faa280343c0 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.308+0000 7faa26000700 1 -- 172.21.15.33:0/3586949623 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faa280345c0 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.308+0000 7faa4d37eb80 10 client.15159 fetch_fsmap learned FSMap version 27 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.308+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b2f873ee60 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa26000700 1 -- 172.21.15.33:0/3586949623 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa280347b0 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa26000700 1 -- 172.21.15.33:0/3586949623 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7faa28025720 con 0x55b2f86c23b0 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa4d37eb80 10 client.15159 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa4d37eb80 10 client.15159 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T11:31:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa4d37eb80 20 client.15159 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f8743360 0x55b2f8742040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa4d37eb80 4 client.15159 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T11:31:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "status"}) -- 0x55b2f86182c0 con 0x55b2f8743360 2024-04-26T11:31:08.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.309+0000 7faa3be00700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f8743360 0x55b2f8742040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa3be00700 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f8743360 0x55b2f8742040 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5a4a63902aba9c server_cookie=6602825f4571fd57 in_seq=0 out_seq=0 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa26000700 10 client.15159 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa26000700 1 -- 172.21.15.33:0/3586949623 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b2f86182c0 con 0x55b2f8743360 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa26000700 10 client.15159 handle_command_reply: tid=0 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 client.15159 shutdown 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 msgr2=0x55b2f86e2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86e2080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7faa2800f5d0 tx=0x7faa2800ff10 comp rx=0 tx=0).stop 2024-04-26T11:31:08.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 shutdown_connections 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55b2f8743360 0x55b2f8742040 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b2f86c2cd0 0x55b2f86e25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b2f86be080 0x55b2f86c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa26000700 0 client.15159 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 --2- 172.21.15.33:0/3586949623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b2f86c23b0 0x55b2f86e2080 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.310+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 >> 172.21.15.33:0/3586949623 conn(0x55b2f85eb2a0 msgr2=0x55b2f86b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.311+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 shutdown_connections 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.311+0000 7faa4d37eb80 1 -- 172.21.15.33:0/3586949623 wait complete. 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.311+0000 7faa4d37eb80 20 client.15159 trim_cache size 0 max 16384 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:31:08.314 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "5cc7f146-03be-11ef-bc93-c7b262605968", 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24527, 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 275.28289221799997, 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 27, 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 91, 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 296.89489821500001 2024-04-26T11:31:08.315 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.311+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faa200a3ed0 msgr2=0x7faa200a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.311+0000 7faa42a00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faa200a3ed0 0x7faa200a6390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa28009750 tx=0x7faa28009310 comp rx=0 tx=0).stop 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.311+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c06bb50 msgr2=0x7faa3c102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.311+0000 7faa42a00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c06bb50 0x7faa3c102420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009230 tx=0x7faa2c0094a0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 shutdown_connections 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faa200a3ed0 0x7faa200a6390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa3c1094a0 0x7faa3c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faa3c06bb50 0x7faa3c102420 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 --2- 172.21.15.33:0/1370492793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3c06c4f0 0x7faa3c102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:08.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 >> 172.21.15.33:0/1370492793 conn(0x7faa3c0792d0 msgr2=0x7faa3c100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:08.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 shutdown_connections 2024-04-26T11:31:08.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:08.312+0000 7faa42a00700 1 -- 172.21.15.33:0/1370492793 wait complete. 2024-04-26T11:31:08.317 INFO:teuthology.orchestra.run.smithi033.stderr:1600000 of 3200000 tuples (50%) done (elapsed 4.74 s, remaining 4.74 s) 2024-04-26T11:31:08.557 INFO:teuthology.orchestra.run.smithi033.stderr:1700000 of 3200000 tuples (53%) done (elapsed 4.98 s, remaining 4.39 s) 2024-04-26T11:31:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:08 smithi194 ceph-mon[33017]: pgmap v25: 97 pgs: 97 active+clean; 72 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 6.8 MiB/s wr, 167 op/s 2024-04-26T11:31:08.883 INFO:teuthology.orchestra.run.smithi033.stderr:1800000 of 3200000 tuples (56%) done (elapsed 5.31 s, remaining 4.13 s) 2024-04-26T11:31:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:08 smithi033 ceph-mon[27845]: pgmap v25: 97 pgs: 97 active+clean; 72 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 6.8 MiB/s wr, 167 op/s 2024-04-26T11:31:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:08 smithi096 ceph-mon[32753]: pgmap v25: 97 pgs: 97 active+clean; 72 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 6.8 MiB/s wr, 167 op/s 2024-04-26T11:31:09.267 INFO:teuthology.orchestra.run.smithi033.stderr:1900000 of 3200000 tuples (59%) done (elapsed 5.69 s, remaining 3.89 s) 2024-04-26T11:31:09.286 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-26T11:31:09.463 INFO:teuthology.orchestra.run.smithi033.stderr:2000000 of 3200000 tuples (62%) done (elapsed 5.89 s, remaining 3.53 s) 2024-04-26T11:31:09.869 INFO:teuthology.orchestra.run.smithi033.stderr:2100000 of 3200000 tuples (65%) done (elapsed 6.29 s, remaining 3.30 s) 2024-04-26T11:31:10.273 INFO:teuthology.orchestra.run.smithi033.stderr:2200000 of 3200000 tuples (68%) done (elapsed 6.70 s, remaining 3.04 s) 2024-04-26T11:31:10.455 INFO:teuthology.orchestra.run.smithi033.stderr:2300000 of 3200000 tuples (71%) done (elapsed 6.88 s, remaining 2.69 s) 2024-04-26T11:31:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:10 smithi194 ceph-mon[33017]: pgmap v26: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 36 MiB/s wr, 367 op/s 2024-04-26T11:31:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:10 smithi033 ceph-mon[27845]: pgmap v26: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 36 MiB/s wr, 367 op/s 2024-04-26T11:31:10.928 INFO:teuthology.orchestra.run.smithi033.stderr:2400000 of 3200000 tuples (75%) done (elapsed 7.35 s, remaining 2.45 s) 2024-04-26T11:31:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:10 smithi096 ceph-mon[32753]: pgmap v26: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 36 MiB/s wr, 367 op/s 2024-04-26T11:31:11.257 INFO:teuthology.orchestra.run.smithi033.stderr:2500000 of 3200000 tuples (78%) done (elapsed 7.68 s, remaining 2.15 s) 2024-04-26T11:31:11.442 INFO:teuthology.orchestra.run.smithi033.stderr:2600000 of 3200000 tuples (81%) done (elapsed 7.87 s, remaining 1.82 s) 2024-04-26T11:31:11.757 INFO:teuthology.orchestra.run.smithi033.stderr:2700000 of 3200000 tuples (84%) done (elapsed 8.18 s, remaining 1.51 s) 2024-04-26T11:31:12.112 INFO:teuthology.orchestra.run.smithi033.stderr:2800000 of 3200000 tuples (87%) done (elapsed 8.54 s, remaining 1.22 s) 2024-04-26T11:31:12.206 INFO:teuthology.orchestra.run.smithi033.stderr:2900000 of 3200000 tuples (90%) done (elapsed 8.63 s, remaining 0.89 s) 2024-04-26T11:31:12.711 INFO:teuthology.orchestra.run.smithi033.stderr:3000000 of 3200000 tuples (93%) done (elapsed 9.13 s, remaining 0.61 s) 2024-04-26T11:31:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:12 smithi033 ceph-mon[27845]: pgmap v27: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 34 MiB/s wr, 350 op/s 2024-04-26T11:31:12.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:12 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:31:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:12 smithi096 ceph-mon[32753]: pgmap v27: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 34 MiB/s wr, 350 op/s 2024-04-26T11:31:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:12 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:31:13.065 INFO:teuthology.orchestra.run.smithi033.stderr:3100000 of 3200000 tuples (96%) done (elapsed 9.49 s, remaining 0.31 s) 2024-04-26T11:31:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:12 smithi194 ceph-mon[33017]: pgmap v27: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 34 MiB/s wr, 350 op/s 2024-04-26T11:31:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:12 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:31:13.170 INFO:teuthology.orchestra.run.smithi033.stderr:3200000 of 3200000 tuples (100%) done (elapsed 9.59 s, remaining 0.00 s) 2024-04-26T11:31:13.263 INFO:teuthology.orchestra.run.smithi033.stderr:vacuuming... 2024-04-26T11:31:14.297 INFO:teuthology.orchestra.run.smithi033.stderr:creating primary keys... 2024-04-26T11:31:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:14 smithi033 ceph-mon[27845]: pgmap v28: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 34 MiB/s wr, 350 op/s 2024-04-26T11:31:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:14 smithi096 ceph-mon[32753]: pgmap v28: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 34 MiB/s wr, 350 op/s 2024-04-26T11:31:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:14 smithi194 ceph-mon[33017]: pgmap v28: 97 pgs: 97 active+clean; 245 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 34 MiB/s wr, 350 op/s 2024-04-26T11:31:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:16 smithi096 ceph-mon[32753]: pgmap v29: 97 pgs: 97 active+clean; 382 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 58 MiB/s wr, 369 op/s 2024-04-26T11:31:16.935 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:16 smithi033 ceph-mon[27845]: pgmap v29: 97 pgs: 97 active+clean; 382 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 58 MiB/s wr, 369 op/s 2024-04-26T11:31:16.962 INFO:teuthology.orchestra.run.smithi033.stderr:done in 13.46 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 9.74 s, vacuum 1.03 s, primary keys 2.66 s). 2024-04-26T11:31:16.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-26T11:31:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:16 smithi194 ceph-mon[33017]: pgmap v29: 97 pgs: 97 active+clean; 382 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 58 MiB/s wr, 369 op/s 2024-04-26T11:31:17.183 INFO:teuthology.orchestra.run.smithi033.stderr:starting vacuum...end. 2024-04-26T11:31:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:18 smithi033 ceph-mon[27845]: pgmap v30: 97 pgs: 97 active+clean; 382 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 52 MiB/s wr, 217 op/s 2024-04-26T11:31:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:18 smithi096 ceph-mon[32753]: pgmap v30: 97 pgs: 97 active+clean; 382 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 52 MiB/s wr, 217 op/s 2024-04-26T11:31:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:18 smithi194 ceph-mon[33017]: pgmap v30: 97 pgs: 97 active+clean; 382 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 52 MiB/s wr, 217 op/s 2024-04-26T11:31:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.160+0000 7fbce8600640 1 -- 172.21.15.33:0/3909640308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 msgr2=0x7fbce0075010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.160+0000 7fbce8600640 1 --2- 172.21.15.33:0/3909640308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce0075010 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009900 tx=0x7fbccc02a490 comp rx=0 tx=0).stop 2024-04-26T11:31:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.160+0000 7fbce8600640 1 -- 172.21.15.33:0/3909640308 shutdown_connections 2024-04-26T11:31:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.160+0000 7fbce8600640 1 --2- 172.21.15.33:0/3909640308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce0075550 0x7fbce010ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.160+0000 7fbce8600640 1 --2- 172.21.15.33:0/3909640308 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbce010f220 0x7fbce0111610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.160+0000 7fbce8600640 1 --2- 172.21.15.33:0/3909640308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce0075010 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.160+0000 7fbce8600640 1 -- 172.21.15.33:0/3909640308 >> 172.21.15.33:0/3909640308 conn(0x7fbce0079dc0 msgr2=0x7fbce007a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.161+0000 7fbce8600640 1 -- 172.21.15.33:0/3909640308 shutdown_connections 2024-04-26T11:31:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.161+0000 7fbce8600640 1 -- 172.21.15.33:0/3909640308 wait complete. 2024-04-26T11:31:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.162+0000 7fbce8600640 1 Processor -- start 2024-04-26T11:31:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.162+0000 7fbce8600640 1 -- start start 2024-04-26T11:31:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce8600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce010eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce8600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbce0075550 0x7fbce010bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce8600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce010f220 0x7fbce010c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce8600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0114e10 con 0x7fbce0075550 2024-04-26T11:31:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce8600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0114c90 con 0x7fbce0074c30 2024-04-26T11:31:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce8600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0114f90 con 0x7fbce010f220 2024-04-26T11:31:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce010eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce010eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49478/0 (socket says 172.21.15.33:49478) 2024-04-26T11:31:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce5c00640 1 -- 172.21.15.33:0/2848020688 learned_addr learned my addr 172.21.15.33:0/2848020688 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce5200640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbce0075550 0x7fbce010bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.163+0000 7fbce6600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce010f220 0x7fbce010c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.164+0000 7fbce5c00640 1 -- 172.21.15.33:0/2848020688 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce010f220 msgr2=0x7fbce010c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.164+0000 7fbce5c00640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce010f220 0x7fbce010c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.164+0000 7fbce5c00640 1 -- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbce0075550 msgr2=0x7fbce010bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.164+0000 7fbce5c00640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbce0075550 0x7fbce010bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.164+0000 7fbce5c00640 1 -- 172.21.15.33:0/2848020688 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce010c970 con 0x7fbce0074c30 2024-04-26T11:31:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.164+0000 7fbce5200640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbce0075550 0x7fbce010bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:31:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.165+0000 7fbce5c00640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce010eab0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbccc005ab0 tx=0x7fbccc003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.165+0000 7fbcde000640 1 -- 172.21.15.33:0/2848020688 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc03f070 con 0x7fbce0074c30 2024-04-26T11:31:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.165+0000 7fbcde000640 1 -- 172.21.15.33:0/2848020688 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbccc031d60 con 0x7fbce0074c30 2024-04-26T11:31:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.165+0000 7fbcde000640 1 -- 172.21.15.33:0/2848020688 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc035030 con 0x7fbce0074c30 2024-04-26T11:31:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.165+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce01abb90 con 0x7fbce0074c30 2024-04-26T11:31:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.165+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce01ac090 con 0x7fbce0074c30 2024-04-26T11:31:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.169+0000 7fbcde000640 1 -- 172.21.15.33:0/2848020688 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7fbccc042070 con 0x7fbce0074c30 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 Processor -- start 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 -- start start 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c8e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6c98a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 0x5581c6c9b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5581c6bf3380 con 0x5581c6c8ddd0 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5581c6b8fc30 con 0x5581c6c8e6f0 2024-04-26T11:31:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.170+0000 7fbceb57f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5581c6bef340 con 0x5581c6c98fb0 2024-04-26T11:31:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6c98a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6c98a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49482/0 (socket says 172.21.15.33:49482) 2024-04-26T11:31:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 -- 172.21.15.33:0/2476413628 learned_addr learned my addr 172.21.15.33:0/2476413628 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5200640 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 0x5581c6c9b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce6600640 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c8e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 -- 172.21.15.33:0/2476413628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 msgr2=0x5581c6c9b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 0x5581c6c9b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 -- 172.21.15.33:0/2476413628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 msgr2=0x5581c6c8e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c8e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.171+0000 7fbce5c00640 1 -- 172.21.15.33:0/2476413628 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c6bb1940 con 0x5581c6c8e6f0 2024-04-26T11:31:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.172+0000 7fbce5c00640 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6c98a70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0ee7a0 tx=0x7fbccc0f9540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5004c25290d1bc06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.172+0000 7fbcde000640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcc80a3f60 0x7fbcc80a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.172+0000 7fbcd2a00640 1 -- 172.21.15.33:0/2476413628 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0b92f0 con 0x5581c6c8e6f0 2024-04-26T11:31:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.172+0000 7fbcd2a00640 1 -- 172.21.15.33:0/2476413628 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbccc0fe030 con 0x5581c6c8e6f0 2024-04-26T11:31:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.172+0000 7fbce6600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcc80a3f60 0x7fbcc80a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.172+0000 7fbcd2a00640 1 -- 172.21.15.33:0/2476413628 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0bc4a0 con 0x5581c6c8e6f0 2024-04-26T11:31:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.175+0000 7fbceb57f740 1 -- 172.21.15.33:0/2476413628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 msgr2=0x5581c6c98a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.175+0000 7fbceb57f740 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6c98a70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0ee7a0 tx=0x7fbccc0f9540 comp rx=0 tx=0).stop 2024-04-26T11:31:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.177+0000 7fbcde000640 1 -- 172.21.15.33:0/2848020688 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbccc0f1220 con 0x7fbce0074c30 2024-04-26T11:31:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.177+0000 7fbce6600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcc80a3f60 0x7fbcc80a6420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbce01006c0 tx=0x7fbcd80095f0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.177+0000 7fbceb57f740 1 -- 172.21.15.33:0/2476413628 shutdown_connections 2024-04-26T11:31:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.177+0000 7fbceb57f740 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 0x5581c6c9b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.177+0000 7fbceb57f740 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6c98a70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.177+0000 7fbceb57f740 1 --2- 172.21.15.33:0/2476413628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c8e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.177+0000 7fbceb57f740 1 -- 172.21.15.33:0/2476413628 >> 172.21.15.33:0/2476413628 conn(0x5581c6c8a2e0 msgr2=0x5581c6c8a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.178+0000 7fbceb57f740 1 -- 172.21.15.33:0/2476413628 shutdown_connections 2024-04-26T11:31:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.178+0000 7fbceb57f740 1 -- 172.21.15.33:0/2476413628 wait complete. 2024-04-26T11:31:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.178+0000 7fbceb57f740 1 -- start start 2024-04-26T11:31:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.178+0000 7fbceb57f740 1 Processor -- start 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c95ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c96410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c98fb0 0x5581c6c96950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbceb57f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5581c6c9b9a0 con 0x5581c6c8ddd0 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbceb57f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5581c6b7ac60 con 0x5581c6c98fb0 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbceb57f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5581c6c9bb20 con 0x5581c6c8e6f0 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbceb57f740 10 client.0 _ll_register_callbacks cb 0x5581c6ba6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.179+0000 7fbce5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c96410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.180+0000 7fbce5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c96410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52414/0 (socket says 172.21.15.33:52414) 2024-04-26T11:31:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.180+0000 7fbce5200640 1 -- 172.21.15.33:0/3783652544 learned_addr learned my addr 172.21.15.33:0/3783652544 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.180+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c95ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.180+0000 7fbce6600640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c98fb0 0x5581c6c96950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.180+0000 7fbce5200640 1 -- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c98fb0 msgr2=0x5581c6c96950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.181+0000 7fbce5200640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c98fb0 0x5581c6c96950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.181+0000 7fbce5200640 1 -- 172.21.15.33:0/3783652544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 msgr2=0x5581c6c95ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.181+0000 7fbce5200640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c95ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.181+0000 7fbce5200640 1 -- 172.21.15.33:0/3783652544 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c6b89510 con 0x5581c6c8e6f0 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.181+0000 7fbce6600640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c98fb0 0x5581c6c96950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.181+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c95ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.182+0000 7fbce5200640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c96410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4011730 tx=0x7fbcd4011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caafc75a89d6500e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.182+0000 7fbcc7400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.182+0000 7fbcc7400640 1 -- 172.21.15.33:0/3783652544 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400fe70 con 0x5581c6c8e6f0 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.182+0000 7fbcc7400640 1 -- 172.21.15.33:0/3783652544 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbcd4004020 con 0x5581c6c8e6f0 2024-04-26T11:31:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.182+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbccc0fe030 con 0x5581c6c8e6f0 2024-04-26T11:31:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.184+0000 7fbcc7400640 1 -- 172.21.15.33:0/3783652544 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbcd4004220 con 0x5581c6c8e6f0 2024-04-26T11:31:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.184+0000 7fbceb57f740 10 client.25273 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.184+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581c6905210 con 0x5581c6c8e6f0 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbcc7400640 1 -- 172.21.15.33:0/3783652544 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbccc0fe030 con 0x5581c6c8e6f0 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbcc7400640 1 -- 172.21.15.33:0/3783652544 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbcd4018030 con 0x5581c6c8e6f0 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbceb57f740 10 client.25273 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbceb57f740 10 client.25273 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbceb57f740 20 client.25273 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6ca8a80 0x5581c6caaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbceb57f740 4 client.25273 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.185+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581c6c00f90 con 0x5581c6ca8a80 2024-04-26T11:31:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.186+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6ca8a80 0x5581c6caaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.188+0000 7fbcc7400640 10 client.25273 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.189+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6ca8a80 0x5581c6caaef0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb41a81a9a1a7bb server_cookie=697c5c7921465a6c in_seq=0 out_seq=0 2024-04-26T11:31:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.189+0000 7fbcc7400640 1 -- 172.21.15.33:0/3783652544 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581c6c00f90 con 0x5581c6ca8a80 2024-04-26T11:31:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.189+0000 7fbcc7400640 10 client.25273 handle_command_reply: tid=0 2024-04-26T11:31:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.190+0000 7fbceb57f740 1 client.25273 shutdown 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.190+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 msgr2=0x5581c6c96410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.190+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c96410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4011730 tx=0x7fbcd4011bf0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 shutdown_connections 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c96410 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6ca8a80 0x5581c6caaef0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c98fb0 0x5581c6c96950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3783652544 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6c95ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 >> 172.21.15.33:0/3783652544 conn(0x5581c6c8a2e0 msgr2=0x5581c6c90060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbcc7400640 0 client.25273 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 shutdown_connections 2024-04-26T11:31:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 1 -- 172.21.15.33:0/3783652544 wait complete. 2024-04-26T11:31:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.191+0000 7fbceb57f740 20 client.25273 trim_cache size 0 max 16384 2024-04-26T11:31:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.225+0000 7fbceb57f740 1 Processor -- start 2024-04-26T11:31:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbceb57f740 1 -- start start 2024-04-26T11:31:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c90790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c98fb0 0x5581c6c94eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbceb57f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01b7480 con 0x5581c6c98fb0 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbceb57f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01b4d50 con 0x5581c6c8ddd0 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbceb57f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01b2ba0 con 0x5581c6c8e6f0 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.226+0000 7fbce5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49506/0 (socket says 172.21.15.33:49506) 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5c00640 1 -- 172.21.15.33:0/3351697666 learned_addr learned my addr 172.21.15.33:0/3351697666 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5200640 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c90790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce6600640 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c98fb0 0x5581c6c94eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5200640 1 -- 172.21.15.33:0/3351697666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8ddd0 msgr2=0x5581c6ca47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5200640 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5200640 1 -- 172.21.15.33:0/3351697666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c98fb0 msgr2=0x5581c6c94eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5200640 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c98fb0 0x5581c6c94eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.227+0000 7fbce5200640 1 -- 172.21.15.33:0/3351697666 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c69d1620 con 0x5581c6c8e6f0 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.228+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.228+0000 7fbce5200640 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c90790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4004c70 tx=0x7fbcd400c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13b963e611b5e85e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.228+0000 7fbcd2000640 1 -- 172.21.15.33:0/3351697666 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4028070 con 0x5581c6c8e6f0 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.228+0000 7fbcd2000640 1 -- 172.21.15.33:0/3351697666 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd400f790 con 0x5581c6c8e6f0 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.228+0000 7fbcd2000640 1 -- 172.21.15.33:0/3351697666 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4011290 con 0x5581c6c8e6f0 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.229+0000 7fbceb57f740 1 -- 172.21.15.33:0/3351697666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 msgr2=0x5581c6c90790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.229+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c90790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4004c70 tx=0x7fbcd400c560 comp rx=0 tx=0).stop 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.230+0000 7fbceb57f740 1 -- 172.21.15.33:0/3351697666 shutdown_connections 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.230+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c8e6f0 0x5581c6c90790 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.230+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c98fb0 0x5581c6c94eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.230+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3351697666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.230+0000 7fbceb57f740 1 -- 172.21.15.33:0/3351697666 >> 172.21.15.33:0/3351697666 conn(0x5581c6c8a2e0 msgr2=0x5581c6c947a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.230+0000 7fbceb57f740 1 -- 172.21.15.33:0/3351697666 shutdown_connections 2024-04-26T11:31:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.230+0000 7fbceb57f740 1 -- 172.21.15.33:0/3351697666 wait complete. 2024-04-26T11:31:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.231+0000 7fbceb57f740 1 -- start start 2024-04-26T11:31:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.231+0000 7fbceb57f740 1 Processor -- start 2024-04-26T11:31:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6ca1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbceb57f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 0x5581c6c9c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbceb57f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01b3210 con 0x5581c6c8ddd0 2024-04-26T11:31:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbceb57f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01b4680 con 0x5581c6c8e6f0 2024-04-26T11:31:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbceb57f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01b38e0 con 0x5581c6c98fb0 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbceb57f740 10 client.0 _ll_register_callbacks cb 0x7fbcd400c750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbce5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6ca1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbce5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6ca1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49522/0 (socket says 172.21.15.33:49522) 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.232+0000 7fbce5200640 1 -- 172.21.15.33:0/3235425323 learned_addr learned my addr 172.21.15.33:0/3235425323 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5200640 1 -- 172.21.15.33:0/3235425323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 msgr2=0x5581c6c9c460 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5200640 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 0x5581c6c9c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5200640 1 -- 172.21.15.33:0/3235425323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 msgr2=0x5581c6ca0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5200640 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5200640 1 -- 172.21.15.33:0/3235425323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c6badcb0 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbcc7400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.233+0000 7fbce5200640 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6ca1240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4004d10 tx=0x7fbcd40045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7008938545f9b041 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.234+0000 7fbcc7400640 1 -- 172.21.15.33:0/3235425323 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4028070 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.234+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcd400f790 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.234+0000 7fbcc7400640 1 -- 172.21.15.33:0/3235425323 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbcd4031070 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.234+0000 7fbcc7400640 1 -- 172.21.15.33:0/3235425323 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbcd4028070 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.234+0000 7fbceb57f740 10 client.25307 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.234+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581c6d0eb90 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.235+0000 7fbcc7400640 1 -- 172.21.15.33:0/3235425323 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd400f790 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.236+0000 7fbcc7400640 1 -- 172.21.15.33:0/3235425323 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbcd4018070 con 0x5581c6c8e6f0 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.236+0000 7fbceb57f740 10 client.25307 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.236+0000 7fbceb57f740 10 client.25307 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.236+0000 7fbceb57f740 20 client.25307 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.236+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6d0f1c0 0x5581c6d115e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.236+0000 7fbceb57f740 4 client.25307 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:31:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.236+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5581c6bfae00 con 0x5581c6d0f1c0 2024-04-26T11:31:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.237+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6d0f1c0 0x5581c6d115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.237+0000 7fbcc7400640 10 client.25307 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.237+0000 7fbce5c00640 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6d0f1c0 0x5581c6d115e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c9f54d2ce77e399 server_cookie=748a824790b5fef4 in_seq=0 out_seq=0 2024-04-26T11:31:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.262+0000 7fbcc7400640 1 -- 172.21.15.33:0/3235425323 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5581c6bfae00 con 0x5581c6d0f1c0 2024-04-26T11:31:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.263+0000 7fbcc7400640 10 client.25307 handle_command_reply: tid=0 2024-04-26T11:31:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.263+0000 7fbceb57f740 1 client.25307 shutdown 2024-04-26T11:31:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.263+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 msgr2=0x5581c6ca1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.263+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6ca1240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4004d10 tx=0x7fbcd40045d0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.264+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 shutdown_connections 2024-04-26T11:31:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.264+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581c6c98fb0 0x5581c6c9c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.264+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581c6d0f1c0 0x5581c6d115e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.264+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581c6c8e6f0 0x5581c6ca1240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.264+0000 7fbceb57f740 1 --2- 172.21.15.33:0/3235425323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581c6c8ddd0 0x5581c6ca0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.264+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 >> 172.21.15.33:0/3235425323 conn(0x5581c6c8a2e0 msgr2=0x5581c6c95e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.264+0000 7fbcc7400640 0 client.25307 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.265+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 shutdown_connections 2024-04-26T11:31:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.265+0000 7fbceb57f740 1 -- 172.21.15.33:0/3235425323 wait complete. 2024-04-26T11:31:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.265+0000 7fbceb57f740 20 client.25307 trim_cache size 0 max 16384 2024-04-26T11:31:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:31:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:31:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:31:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:31:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcc80a3f60 msgr2=0x7fbcc80a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcc80a3f60 0x7fbcc80a6420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbce01006c0 tx=0x7fbcd80095f0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 msgr2=0x7fbce010eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce010eab0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbccc005ab0 tx=0x7fbccc003cd0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 shutdown_connections 2024-04-26T11:31:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcc80a3f60 0x7fbcc80a6420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce010f220 0x7fbce010c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbce0075550 0x7fbce010bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 --2- 172.21.15.33:0/2848020688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbce0074c30 0x7fbce010eab0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.266+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 >> 172.21.15.33:0/2848020688 conn(0x7fbce0079dc0 msgr2=0x7fbce00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.267+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 shutdown_connections 2024-04-26T11:31:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:19.267+0000 7fbce8600640 1 -- 172.21.15.33:0/2848020688 wait complete. 2024-04-26T11:31:20.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:20 smithi033 ceph-mon[27845]: pgmap v31: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 70 MiB/s wr, 269 op/s 2024-04-26T11:31:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:20 smithi096 ceph-mon[32753]: pgmap v31: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 70 MiB/s wr, 269 op/s 2024-04-26T11:31:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:20 smithi194 ceph-mon[33017]: pgmap v31: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 70 MiB/s wr, 269 op/s 2024-04-26T11:31:22.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 5.0 s, 2787.0 tps, lat 34.423 ms stddev 36.815 2024-04-26T11:31:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:22 smithi033 ceph-mon[27845]: pgmap v32: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 41 MiB/s wr, 70 op/s 2024-04-26T11:31:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:22 smithi096 ceph-mon[32753]: pgmap v32: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 41 MiB/s wr, 70 op/s 2024-04-26T11:31:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:22 smithi194 ceph-mon[33017]: pgmap v32: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 41 MiB/s wr, 70 op/s 2024-04-26T11:31:24.286 INFO:tasks.quiescer.fs.[cephfs]:Total duration (36.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T11:31:24.286 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.0 2024-04-26T11:31:24.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:31:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:31:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:24 smithi033 ceph-mon[27845]: pgmap v33: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 41 MiB/s wr, 70 op/s 2024-04-26T11:31:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:24 smithi096 ceph-mon[32753]: pgmap v33: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 41 MiB/s wr, 70 op/s 2024-04-26T11:31:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:24 smithi194 ceph-mon[33017]: pgmap v33: 97 pgs: 97 active+clean; 523 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 41 MiB/s wr, 70 op/s 2024-04-26T11:31:26.885 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:26 smithi033 ceph-mon[27845]: pgmap v34: 97 pgs: 97 active+clean; 898 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 43 MiB/s wr, 312 op/s 2024-04-26T11:31:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:26 smithi096 ceph-mon[32753]: pgmap v34: 97 pgs: 97 active+clean; 898 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 43 MiB/s wr, 312 op/s 2024-04-26T11:31:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:26 smithi194 ceph-mon[33017]: pgmap v34: 97 pgs: 97 active+clean; 898 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 43 MiB/s wr, 312 op/s 2024-04-26T11:31:27.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 10.0 s, 2957.7 tps, lat 33.806 ms stddev 38.715 2024-04-26T11:31:27.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.459+0000 7f7192c00700 1 -- 172.21.15.33:0/4096818459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c108350 msgr2=0x7f718c06b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:27.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.459+0000 7f7192c00700 1 --2- 172.21.15.33:0/4096818459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c108350 0x7f718c06b130 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f717c009990 tx=0x7f717c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.459+0000 7f7192c00700 1 -- 172.21.15.33:0/4096818459 shutdown_connections 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.459+0000 7f7192c00700 1 --2- 172.21.15.33:0/4096818459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f718c105e60 0x7f718c1062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.459+0000 7f7192c00700 1 --2- 172.21.15.33:0/4096818459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c108350 0x7f718c06b130 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.459+0000 7f7192c00700 1 --2- 172.21.15.33:0/4096818459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f718c06b670 0x7f718c06bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.459+0000 7f7192c00700 1 -- 172.21.15.33:0/4096818459 >> 172.21.15.33:0/4096818459 conn(0x7f718c0792d0 msgr2=0x7f718c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.460+0000 7f7192c00700 1 -- 172.21.15.33:0/4096818459 shutdown_connections 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.460+0000 7f7192c00700 1 -- 172.21.15.33:0/4096818459 wait complete. 2024-04-26T11:31:27.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.461+0000 7f7192c00700 1 Processor -- start 2024-04-26T11:31:27.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.461+0000 7f7192c00700 1 -- start start 2024-04-26T11:31:27.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.461+0000 7f7192c00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c06b670 0x7f718c074c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:27.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.461+0000 7f7192c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f718c105e60 0x7f718c06dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:27.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f7192c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f718c108350 0x7f718c06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:27.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f7192c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f718c119be0 con 0x7f718c108350 2024-04-26T11:31:27.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f7192c00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f718c119a60 con 0x7f718c06b670 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f7192c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f718c119d60 con 0x7f718c105e60 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f7190c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f718c108350 0x7f718c06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f7190c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f718c108350 0x7f718c06e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47972/0 (socket says 172.21.15.33:47972) 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f718be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c06b670 0x7f718c074c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f7190c00700 1 -- 172.21.15.33:0/1828023114 learned_addr learned my addr 172.21.15.33:0/1828023114 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f718b400700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f718c105e60 0x7f718c06dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f718be00700 1 -- 172.21.15.33:0/1828023114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f718c105e60 msgr2=0x7f718c06dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f718be00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f718c105e60 0x7f718c06dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f718be00700 1 -- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f718c108350 msgr2=0x7f718c06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f718be00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f718c108350 0x7f718c06e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.462+0000 7f718be00700 1 -- 172.21.15.33:0/1828023114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718c06e9e0 con 0x7f718c06b670 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.463+0000 7f718be00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c06b670 0x7f718c074c70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f717c000c00 tx=0x7f717c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.463+0000 7f7188c00700 1 -- 172.21.15.33:0/1828023114 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c026070 con 0x7f718c06b670 2024-04-26T11:31:27.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.463+0000 7f7188c00700 1 -- 172.21.15.33:0/1828023114 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f717c021410 con 0x7f718c06b670 2024-04-26T11:31:27.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.463+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718c072810 con 0x7f718c06b670 2024-04-26T11:31:27.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.463+0000 7f7188c00700 1 -- 172.21.15.33:0/1828023114 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c017650 con 0x7f718c06b670 2024-04-26T11:31:27.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.463+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f718c072d40 con 0x7f718c06b670 2024-04-26T11:31:27.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.466+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f718c102940 con 0x7f718c06b670 2024-04-26T11:31:27.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.469+0000 7f7188c00700 1 -- 172.21.15.33:0/1828023114 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f717c029070 con 0x7f718c06b670 2024-04-26T11:31:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.472+0000 7f7188c00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f71700a3f50 0x7f71700a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.472+0000 7f7188c00700 1 -- 172.21.15.33:0/1828023114 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f717c0d16c0 con 0x7f718c06b670 2024-04-26T11:31:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.472+0000 7f718b400700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f71700a3f50 0x7f71700a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:27.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.475+0000 7f718b400700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f71700a3f50 0x7f71700a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f718c06b3d0 tx=0x7f717800a380 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:27.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.476+0000 7f7188c00700 1 -- 172.21.15.33:0/1828023114 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f717c05dff0 con 0x7f718c06b670 2024-04-26T11:31:27.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.913+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f718c06f2a0 con 0x7f718c06b670 2024-04-26T11:31:27.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.914+0000 7f7188c00700 1 -- 172.21.15.33:0/1828023114 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11868 (secure 0 0 0) 0x7f717c013070 con 0x7f718c06b670 2024-04-26T11:31:27.918 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:27.919 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:31:11.426188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:31:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.919+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f71700a3f50 msgr2=0x7f71700a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.919+0000 7f7192c00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f71700a3f50 0x7f71700a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f718c06b3d0 tx=0x7f717800a380 comp rx=0 tx=0).stop 2024-04-26T11:31:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.919+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c06b670 msgr2=0x7f718c074c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.919+0000 7f7192c00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c06b670 0x7f718c074c70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f717c000c00 tx=0x7f717c0047b0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.920+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 shutdown_connections 2024-04-26T11:31:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.920+0000 7f7192c00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f71700a3f50 0x7f71700a6410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.920+0000 7f7192c00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f718c108350 0x7f718c06e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.920+0000 7f7192c00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f718c06b670 0x7f718c074c70 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f717c000c00 tx=0x7f717c0047b0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.920+0000 7f7192c00700 1 --2- 172.21.15.33:0/1828023114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f718c105e60 0x7f718c06dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.920+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 >> 172.21.15.33:0/1828023114 conn(0x7f718c0792d0 msgr2=0x7f718c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.921+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 shutdown_connections 2024-04-26T11:31:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:27.921+0000 7f7192c00700 1 -- 172.21.15.33:0/1828023114 wait complete. 2024-04-26T11:31:27.927 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-04-26T11:31:28.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:28 smithi033 ceph-mon[27845]: pgmap v35: 97 pgs: 97 active+clean; 898 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 20 MiB/s wr, 293 op/s 2024-04-26T11:31:28.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:28 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/1828023114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:31:28.798 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24527 quiesce db / --timeout 60 --expiration 78.0 --await-for 42.0' 2024-04-26T11:31:28.799 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24527 quiesce db / --timeout 60 --expiration 78.0 --await-for 42.0 2024-04-26T11:31:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:28 smithi096 ceph-mon[32753]: pgmap v35: 97 pgs: 97 active+clean; 898 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 20 MiB/s wr, 293 op/s 2024-04-26T11:31:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:28 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/1828023114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:31:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:28 smithi194 ceph-mon[33017]: pgmap v35: 97 pgs: 97 active+clean; 898 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 20 MiB/s wr, 293 op/s 2024-04-26T11:31:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:28 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/1828023114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:31:29.468 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:31:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:30 smithi033 ceph-mon[27845]: pgmap v36: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 25 MiB/s wr, 602 op/s 2024-04-26T11:31:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:30 smithi096 ceph-mon[32753]: pgmap v36: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 25 MiB/s wr, 602 op/s 2024-04-26T11:31:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:30 smithi194 ceph-mon[33017]: pgmap v36: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 25 MiB/s wr, 602 op/s 2024-04-26T11:31:31.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.996+0000 7f1b22800700 1 -- 172.21.15.33:0/3433805392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c10a960 msgr2=0x7f1b1c10ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:31.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.996+0000 7f1b22800700 1 --2- 172.21.15.33:0/3433805392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c10a960 0x7f1b1c10ade0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1b040099c0 tx=0x7f1b04009cd0 comp rx=0 tx=0).stop 2024-04-26T11:31:31.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.996+0000 7f1b22800700 1 -- 172.21.15.33:0/3433805392 shutdown_connections 2024-04-26T11:31:31.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.996+0000 7f1b22800700 1 --2- 172.21.15.33:0/3433805392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b1c11cc00 0x7f1b1c11eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:31.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.996+0000 7f1b22800700 1 --2- 172.21.15.33:0/3433805392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b1c111c70 0x7f1b1c10a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.996+0000 7f1b22800700 1 --2- 172.21.15.33:0/3433805392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c10a960 0x7f1b1c10ade0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1b040099c0 tx=0x7f1b04009cd0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.996+0000 7f1b22800700 1 -- 172.21.15.33:0/3433805392 >> 172.21.15.33:0/3433805392 conn(0x7f1b1c06e590 msgr2=0x7f1b1c06e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.997+0000 7f1b22800700 1 -- 172.21.15.33:0/3433805392 shutdown_connections 2024-04-26T11:31:32.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.997+0000 7f1b22800700 1 -- 172.21.15.33:0/3433805392 wait complete. 2024-04-26T11:31:32.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:31.999+0000 7f1b22800700 1 Processor -- start 2024-04-26T11:31:32.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.000+0000 7f1b22800700 1 -- start start 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.000+0000 7f1b22800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b1c111c70 0x7f1b1c1ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.000+0000 7f1b22800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b1c11cc00 0x7f1b1c1aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.000+0000 7f1b22800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c1a88e0 0x7f1b1c1a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.000+0000 7f1b22800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c122480 con 0x7f1b1c111c70 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.000+0000 7f1b22800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c122300 con 0x7f1b1c11cc00 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.000+0000 7f1b22800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c122600 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.001+0000 7f1b1be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c1a88e0 0x7f1b1c1a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.001+0000 7f1b1be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c1a88e0 0x7f1b1c1a8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52604/0 (socket says 172.21.15.33:52604) 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.001+0000 7f1b1be00700 1 -- 172.21.15.33:0/570251029 learned_addr learned my addr 172.21.15.33:0/570251029 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.001+0000 7f1b12a00700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b1c11cc00 0x7f1b1c1aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.001+0000 7f1b1be00700 1 -- 172.21.15.33:0/570251029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b1c11cc00 msgr2=0x7f1b1c1aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.001+0000 7f1b1be00700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b1c11cc00 0x7f1b1c1aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.001+0000 7f1b1be00700 1 -- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b1c111c70 msgr2=0x7f1b1c1ae720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T11:31:32.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.002+0000 7f1b1b400700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b1c111c70 0x7f1b1c1ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.002+0000 7f1b1be00700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b1c111c70 0x7f1b1c1ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.002+0000 7f1b1be00700 1 -- 172.21.15.33:0/570251029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1c1a9460 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.003+0000 7f1b1b400700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b1c111c70 0x7f1b1c1ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.003+0000 7f1b1be00700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c1a88e0 0x7f1b1c1a8d60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400d980 tx=0x7f1b1400dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.003+0000 7f1b18c00700 1 -- 172.21.15.33:0/570251029 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b14017070 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.003+0000 7f1b18c00700 1 -- 172.21.15.33:0/570251029 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b14009df0 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.003+0000 7f1b18c00700 1 -- 172.21.15.33:0/570251029 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b140053a0 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.003+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b1c1b55f0 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.004+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b1c1b5990 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.010+0000 7f1b18c00700 1 -- 172.21.15.33:0/570251029 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f1b14005540 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.013+0000 7f1b18c00700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f1b000a40c0 0x7f1b000a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.013+0000 7f1b18c00700 1 -- 172.21.15.33:0/570251029 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1b140d09d0 con 0x7f1b1c1a88e0 2024-04-26T11:31:32.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.014+0000 7f1b1b400700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f1b000a40c0 0x7f1b000a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.017+0000 7f1b1b400700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f1b000a40c0 0x7f1b000a6580 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c1a9a20 tx=0x7f1b0c00b040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.019+0000 7f1b2d1a0b80 1 Processor -- start 2024-04-26T11:31:32.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.019+0000 7f1b2d1a0b80 1 -- start start 2024-04-26T11:31:32.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb765f0 0x56217cb769d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb80780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cb830b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56217cac9b70 con 0x56217cb765f0 2024-04-26T11:31:32.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56217ca683d0 con 0x56217cb80cc0 2024-04-26T11:31:32.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56217cac5b30 con 0x56217cb76f10 2024-04-26T11:31:32.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b1be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cb830b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b1be00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cb830b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42786/0 (socket says 172.21.15.33:42786) 2024-04-26T11:31:32.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.020+0000 7f1b1be00700 1 -- 172.21.15.33:0/157132862 learned_addr learned my addr 172.21.15.33:0/157132862 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b1be00700 1 -- 172.21.15.33:0/157132862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 msgr2=0x56217cb80780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b1be00700 1 --2- 172.21.15.33:0/157132862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb80780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b1be00700 1 -- 172.21.15.33:0/157132862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb765f0 msgr2=0x56217cb769d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b1be00700 1 --2- 172.21.15.33:0/157132862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb765f0 0x56217cb769d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b1be00700 1 -- 172.21.15.33:0/157132862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56217ca708f0 con 0x56217cb80cc0 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b1be00700 1 --2- 172.21.15.33:0/157132862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cb830b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b1409c3d0 tx=0x7f1b1409ced0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475ae28e223576d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b10c00700 1 -- 172.21.15.33:0/157132862 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b140b2070 con 0x56217cb80cc0 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.021+0000 7f1b10c00700 1 -- 172.21.15.33:0/157132862 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b1409a770 con 0x56217cb80cc0 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/157132862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 msgr2=0x56217cb830b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/157132862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cb830b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b1409c3d0 tx=0x7f1b1409ced0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/157132862 shutdown_connections 2024-04-26T11:31:32.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/157132862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cb830b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/157132862 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb765f0 0x56217cb769d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/157132862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb80780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/157132862 >> 172.21.15.33:0/157132862 conn(0x56217caa45d0 msgr2=0x56217cb71930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.022+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/157132862 shutdown_connections 2024-04-26T11:31:32.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.023+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/157132862 wait complete. 2024-04-26T11:31:32.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.023+0000 7f1b2d1a0b80 1 -- start start 2024-04-26T11:31:32.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.023+0000 7f1b2d1a0b80 1 Processor -- start 2024-04-26T11:31:32.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.024+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb80400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.024+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb76f10 0x56217cb7d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.024+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb7dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.024+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56217cb838d0 con 0x56217cb80cc0 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.024+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56217ca64620 con 0x56217cb76f10 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.024+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56217cb83a50 con 0x56217cb765f0 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.024+0000 7f1b2d1a0b80 10 client.0 _ll_register_callbacks cb 0x56217ca5d510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb76f10 0x56217cb7d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb76f10 0x56217cb7d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42802/0 (socket says 172.21.15.33:42802) 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 -- 172.21.15.33:0/720837903 learned_addr learned my addr 172.21.15.33:0/720837903 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b1b400700 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb80400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b1be00700 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb7dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 -- 172.21.15.33:0/720837903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 msgr2=0x56217cb80400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb80400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 -- 172.21.15.33:0/720837903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 msgr2=0x56217cb7dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb7dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.025+0000 7f1b12a00700 1 -- 172.21.15.33:0/720837903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56217ca16a60 con 0x56217cb76f10 2024-04-26T11:31:32.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.026+0000 7f1b12a00700 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb76f10 0x56217cb7d700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1b04000c00 tx=0x7f1b04005810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fe7d63b5ba6cf44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.026+0000 7f1b09600700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:32.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.027+0000 7f1b09600700 1 -- 172.21.15.33:0/720837903 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04027060 con 0x56217cb76f10 2024-04-26T11:31:32.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.027+0000 7f1b09600700 1 -- 172.21.15.33:0/720837903 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1b040039c0 con 0x56217cb76f10 2024-04-26T11:31:32.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.027+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b1409a770 con 0x56217cb76f10 2024-04-26T11:31:32.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.029+0000 7f1b09600700 1 -- 172.21.15.33:0/720837903 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1b04003bc0 con 0x56217cb76f10 2024-04-26T11:31:32.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.029+0000 7f1b2d1a0b80 10 client.25325 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:32.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.029+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56217cb8e190 con 0x56217cb76f10 2024-04-26T11:31:32.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.029+0000 7f1b09600700 1 -- 172.21.15.33:0/720837903 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b1409a770 con 0x56217cb76f10 2024-04-26T11:31:32.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.030+0000 7f1b09600700 1 -- 172.21.15.33:0/720837903 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1b04021030 con 0x56217cb76f10 2024-04-26T11:31:32.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.030+0000 7f1b2d1a0b80 10 client.25325 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:32.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.030+0000 7f1b2d1a0b80 10 client.25325 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:31:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.030+0000 7f1b2d1a0b80 20 client.25325 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.031+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cb90720 0x56217cb92b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.031+0000 7f1b2d1a0b80 4 client.25325 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.031+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56217cb42d00 con 0x56217cb90720 2024-04-26T11:31:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.031+0000 7f1b1b400700 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cb90720 0x56217cb92b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.032+0000 7f1b1b400700 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cb90720 0x56217cb92b10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9277ef15b63a522 server_cookie=d3e745d19997cce8 in_seq=0 out_seq=0 2024-04-26T11:31:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.032+0000 7f1b09600700 10 client.25325 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.032+0000 7f1b09600700 1 -- 172.21.15.33:0/720837903 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56217cb42d00 con 0x56217cb90720 2024-04-26T11:31:32.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.032+0000 7f1b09600700 10 client.25325 handle_command_reply: tid=0 2024-04-26T11:31:32.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 client.25325 shutdown 2024-04-26T11:31:32.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb76f10 msgr2=0x56217cb7d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb76f10 0x56217cb7d700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1b04000c00 tx=0x7f1b04005810 comp rx=0 tx=0).stop 2024-04-26T11:31:32.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 shutdown_connections 2024-04-26T11:31:32.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cb90720 0x56217cb92b10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb7dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb80400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/720837903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb76f10 0x56217cb7d700 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 >> 172.21.15.33:0/720837903 conn(0x56217caa45d0 msgr2=0x56217cb735d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b09600700 0 client.25325 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 shutdown_connections 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.033+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/720837903 wait complete. 2024-04-26T11:31:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.034+0000 7f1b2d1a0b80 20 client.25325 trim_cache size 0 max 16384 2024-04-26T11:31:32.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.061+0000 7f1b2d1a0b80 1 Processor -- start 2024-04-26T11:31:32.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.061+0000 7f1b2d1a0b80 1 -- start start 2024-04-26T11:31:32.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb765f0 0x56217cb7c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb78cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb983b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c1bf8c0 con 0x56217cb80cc0 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c1bd150 con 0x56217cb765f0 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b2d1a0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c1bafa0 con 0x56217cb76f10 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b12a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb78cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b12a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb78cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52634/0 (socket says 172.21.15.33:52634) 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b12a00700 1 -- 172.21.15.33:0/790773484 learned_addr learned my addr 172.21.15.33:0/790773484 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b1be00700 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.062+0000 7f1b1b400700 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb765f0 0x56217cb7c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.063+0000 7f1b12a00700 1 -- 172.21.15.33:0/790773484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb765f0 msgr2=0x56217cb7c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.063+0000 7f1b12a00700 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb765f0 0x56217cb7c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.063+0000 7f1b12a00700 1 -- 172.21.15.33:0/790773484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 msgr2=0x56217cb983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.063+0000 7f1b12a00700 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.063+0000 7f1b12a00700 1 -- 172.21.15.33:0/790773484 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56217cb711e0 con 0x56217cb76f10 2024-04-26T11:31:32.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.063+0000 7f1b1b400700 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb765f0 0x56217cb7c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:32.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.063+0000 7f1b12a00700 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb78cd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b04003e90 tx=0x7f1b0400bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=260e4e43d22fd458 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.064+0000 7f1b0be00700 1 -- 172.21.15.33:0/790773484 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04027040 con 0x56217cb76f10 2024-04-26T11:31:32.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.064+0000 7f1b0be00700 1 -- 172.21.15.33:0/790773484 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b040039c0 con 0x56217cb76f10 2024-04-26T11:31:32.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.064+0000 7f1b0be00700 1 -- 172.21.15.33:0/790773484 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04031080 con 0x56217cb76f10 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/790773484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 msgr2=0x56217cb78cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb78cd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b04003e90 tx=0x7f1b0400bf20 comp rx=0 tx=0).stop 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/790773484 shutdown_connections 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb80cc0 0x56217cb983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb765f0 0x56217cb7c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/790773484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb76f10 0x56217cb78cd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/790773484 >> 172.21.15.33:0/790773484 conn(0x56217caa45d0 msgr2=0x56217cb785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/790773484 shutdown_connections 2024-04-26T11:31:32.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.065+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/790773484 wait complete. 2024-04-26T11:31:32.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.066+0000 7f1b2d1a0b80 1 -- start start 2024-04-26T11:31:32.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.066+0000 7f1b2d1a0b80 1 Processor -- start 2024-04-26T11:31:32.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.067+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb85ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.067+0000 7f1b2d1a0b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb76f10 0x56217cb86420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.067+0000 7f1b1b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb85ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.067+0000 7f1b1b400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb85ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52650/0 (socket says 172.21.15.33:52650) 2024-04-26T11:31:32.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.068+0000 7f1b1b400700 1 -- 172.21.15.33:0/3543129640 learned_addr learned my addr 172.21.15.33:0/3543129640 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.068+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cbf6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.068+0000 7f1b12a00700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb76f10 0x56217cb86420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.068+0000 7f1b1be00700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cbf6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.068+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c1bb610 con 0x56217cb76f10 2024-04-26T11:31:32.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.068+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c1bca80 con 0x56217cb80cc0 2024-04-26T11:31:32.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.068+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c1bbce0 con 0x56217cb765f0 2024-04-26T11:31:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b2d1a0b80 10 client.0 _ll_register_callbacks cb 0x56217ca5d510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b1be00700 1 -- 172.21.15.33:0/3543129640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 msgr2=0x56217cb85ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b1be00700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb85ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b1be00700 1 -- 172.21.15.33:0/3543129640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb76f10 msgr2=0x56217cb86420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b1be00700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb76f10 0x56217cb86420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b1be00700 1 -- 172.21.15.33:0/3543129640 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56217cbf71a0 con 0x56217cb80cc0 2024-04-26T11:31:32.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b1b400700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb85ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:32.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b1be00700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cbf6c30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x56217cb774a0 tx=0x7f1b14087200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133d2cbe5d94bff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.069+0000 7f1b09600700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:32.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.070+0000 7f1b09600700 1 -- 172.21.15.33:0/3543129640 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b14080060 con 0x56217cb80cc0 2024-04-26T11:31:32.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.070+0000 7f1b09600700 1 -- 172.21.15.33:0/3543129640 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1b140a4eb0 con 0x56217cb80cc0 2024-04-26T11:31:32.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.070+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b040039c0 con 0x56217cb80cc0 2024-04-26T11:31:32.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.070+0000 7f1b09600700 1 -- 172.21.15.33:0/3543129640 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1b140bb070 con 0x56217cb80cc0 2024-04-26T11:31:32.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.070+0000 7f1b2d1a0b80 10 client.25331 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:32.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.070+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56217cbf8440 con 0x56217cb80cc0 2024-04-26T11:31:32.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b09600700 1 -- 172.21.15.33:0/3543129640 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b040039c0 con 0x56217cb80cc0 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b09600700 1 -- 172.21.15.33:0/3543129640 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1b1409bb20 con 0x56217cb80cc0 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b2d1a0b80 10 client.25331 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b2d1a0b80 10 client.25331 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b2d1a0b80 20 client.25331 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cbf8790 0x56217cbfabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b2d1a0b80 4 client.25331 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.0, "await_for": 42.0}] 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.0, "await_for": 42.0}) -- 0x56217cad7550 con 0x56217cbf8790 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.071+0000 7f1b1b400700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cbf8790 0x56217cbfabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.072+0000 7f1b09600700 10 client.25331 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.072+0000 7f1b1b400700 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cbf8790 0x56217cbfabb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a95f8de2db928f server_cookie=3e6957fef27bb007 in_seq=0 out_seq=0 2024-04-26T11:31:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 15.0 s, 2895.0 tps, lat 33.659 ms stddev 36.692 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 -- 172.21.15.33:0/2182662645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51601038f0 msgr2=0x7f5160103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 --2- 172.21.15.33:0/2182662645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51601038f0 0x7f5160103cd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5150009860 tx=0x7f515002a3d0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 -- 172.21.15.33:0/2182662645 shutdown_connections 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 --2- 172.21.15.33:0/2182662645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5160104210 0x7f516010c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 --2- 172.21.15.33:0/2182662645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f516010cd10 0x7f516010f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 --2- 172.21.15.33:0/2182662645 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f51601038f0 0x7f5160103cd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 -- 172.21.15.33:0/2182662645 >> 172.21.15.33:0/2182662645 conn(0x7f51600fd150 msgr2=0x7f51600ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 -- 172.21.15.33:0/2182662645 shutdown_connections 2024-04-26T11:31:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.411+0000 7f515fe00640 1 -- 172.21.15.33:0/2182662645 wait complete. 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 Processor -- start 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 -- start start 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51601038f0 0x7f5160101540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5160104210 0x7f5160101a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f516010cd10 0x7f5160101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112a10 con 0x7f516010cd10 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112890 con 0x7f5160104210 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.412+0000 7f515fe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112b90 con 0x7f51601038f0 2024-04-26T11:31:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f516010cd10 0x7f5160101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51601038f0 0x7f5160101540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5160104210 0x7f5160101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51601038f0 0x7f5160101540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52672/0 (socket says 172.21.15.33:52672) 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5160104210 0x7f5160101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42834/0 (socket says 172.21.15.33:42834) 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 -- 172.21.15.33:0/1255540276 learned_addr learned my addr 172.21.15.33:0/1255540276 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 -- 172.21.15.33:0/1255540276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5160104210 msgr2=0x7f5160101a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5160104210 0x7f5160101a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 -- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f516010cd10 msgr2=0x7f5160101fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f516010cd10 0x7f5160101fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515ea00640 1 -- 172.21.15.33:0/1255540276 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51601a74b0 con 0x7f51601038f0 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515e000640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5160104210 0x7f5160101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.413+0000 7f515f400640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f516010cd10 0x7f5160101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.414+0000 7f515ea00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51601038f0 0x7f5160101540 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5150009980 tx=0x7f5150030fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.414+0000 7f5157400640 1 -- 172.21.15.33:0/1255540276 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5150040070 con 0x7f51601038f0 2024-04-26T11:31:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.414+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51601a7740 con 0x7f51601038f0 2024-04-26T11:31:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.414+0000 7f5157400640 1 -- 172.21.15.33:0/1255540276 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5150005570 con 0x7f51601038f0 2024-04-26T11:31:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.414+0000 7f5157400640 1 -- 172.21.15.33:0/1255540276 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51500046c0 con 0x7f51601038f0 2024-04-26T11:31:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.414+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51601a7c30 con 0x7f51601038f0 2024-04-26T11:31:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.417+0000 7f5168a3e740 1 Processor -- start 2024-04-26T11:31:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.417+0000 7f5168a3e740 1 -- start start 2024-04-26T11:31:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da8ff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8ff900 0x5581da909c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da90c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f5168a3e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5581da864380 con 0x5581da8ff900 2024-04-26T11:31:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f5168a3e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5581da800c30 con 0x5581da90a1c0 2024-04-26T11:31:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f5168a3e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5581da860340 con 0x5581da8fefe0 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da8ff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f5157400640 1 -- 172.21.15.33:0/1255540276 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f5150043070 con 0x7f51601038f0 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da90c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da8ff3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52688/0 (socket says 172.21.15.33:52688) 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515f400640 1 -- 172.21.15.33:0/720969768 learned_addr learned my addr 172.21.15.33:0/720969768 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515ea00640 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8ff900 0x5581da909c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515e000640 1 -- 172.21.15.33:0/720969768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 msgr2=0x5581da8ff3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515e000640 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da8ff3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515e000640 1 -- 172.21.15.33:0/720969768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8ff900 msgr2=0x5581da909c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515e000640 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8ff900 0x5581da909c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515e000640 1 -- 172.21.15.33:0/720969768 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581da822940 con 0x5581da90a1c0 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.418+0000 7f515ea00640 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8ff900 0x5581da909c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.419+0000 7f515e000640 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da90c5b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f514c009f90 tx=0x7f514c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e1e02200fc39f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.419+0000 7f5157400640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f51440a3e30 0x7f51440a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f514be00640 1 -- 172.21.15.33:0/720969768 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c004030 con 0x5581da90a1c0 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f514be00640 1 -- 172.21.15.33:0/720969768 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f514c0054d0 con 0x5581da90a1c0 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f514be00640 1 -- 172.21.15.33:0/720969768 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c004ca0 con 0x5581da90a1c0 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f5157400640 1 -- 172.21.15.33:0/1255540276 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f51500f14a0 con 0x7f51601038f0 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f515f400640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f51440a3e30 0x7f51440a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f5168a3e740 1 -- 172.21.15.33:0/720969768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 msgr2=0x5581da90c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f5168a3e740 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da90c5b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f514c009f90 tx=0x7f514c0118d0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.420+0000 7f515f400640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f51440a3e30 0x7f51440a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5158005c40 tx=0x7f51580095c0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 -- 172.21.15.33:0/720969768 shutdown_connections 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da90c5b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8ff900 0x5581da909c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 --2- 172.21.15.33:0/720969768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da8ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 -- 172.21.15.33:0/720969768 >> 172.21.15.33:0/720969768 conn(0x5581da8fb4f0 msgr2=0x5581da8fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 -- 172.21.15.33:0/720969768 shutdown_connections 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 -- 172.21.15.33:0/720969768 wait complete. 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 -- start start 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.421+0000 7f5168a3e740 1 Processor -- start 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da907600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da907b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f5168a3e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5581da90cdf0 con 0x5581da90a1c0 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f5168a3e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5581da7ebc60 con 0x5581da8ff900 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f5168a3e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5581da90cf70 con 0x5581da8fefe0 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f5168a3e740 10 client.0 _ll_register_callbacks cb 0x5581da817e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da907600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da907600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42860/0 (socket says 172.21.15.33:42860) 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515e000640 1 -- 172.21.15.33:0/3639153610 learned_addr learned my addr 172.21.15.33:0/3639153610 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515ea00640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515f400640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da907b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515e000640 1 -- 172.21.15.33:0/3639153610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 msgr2=0x5581da9070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515e000640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515e000640 1 -- 172.21.15.33:0/3639153610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 msgr2=0x5581da907b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.422+0000 7f515e000640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da907b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.423+0000 7f515e000640 1 -- 172.21.15.33:0/3639153610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581da7fa510 con 0x5581da8ff900 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.423+0000 7f515f400640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da907b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.423+0000 7f515e000640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da907600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f514c009350 tx=0x7f514c00fcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d05331a0ccb182 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.424+0000 7f5148c00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.424+0000 7f5148c00640 1 -- 172.21.15.33:0/3639153610 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c029020 con 0x5581da8ff900 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.424+0000 7f5148c00640 1 -- 172.21.15.33:0/3639153610 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f514c004bf0 con 0x5581da8ff900 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.424+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f514c0054d0 con 0x5581da8ff900 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.424+0000 7f5148c00640 1 -- 172.21.15.33:0/3639153610 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f514c034050 con 0x5581da8ff900 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.424+0000 7f5168a3e740 10 client.25343 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.424+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581da576210 con 0x5581da8ff900 2024-04-26T11:31:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5148c00640 1 -- 172.21.15.33:0/3639153610 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f514c0054d0 con 0x5581da8ff900 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5148c00640 1 -- 172.21.15.33:0/3639153610 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f514c025470 con 0x5581da8ff900 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5168a3e740 10 client.25343 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5168a3e740 10 client.25343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5168a3e740 20 client.25343 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da919d90 0x5581da91c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5168a3e740 4 client.25343 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.425+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581da871f90 con 0x5581da919d90 2024-04-26T11:31:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.428+0000 7f515ea00640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da919d90 0x5581da91c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.428+0000 7f515ea00640 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da919d90 0x5581da91c180 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d44d0181c60b5fd6 server_cookie=7e4816b86478c49c in_seq=0 out_seq=0 2024-04-26T11:31:32.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.428+0000 7f5148c00640 10 client.25343 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.439+0000 7f5148c00640 1 -- 172.21.15.33:0/3639153610 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581da871f90 con 0x5581da919d90 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.439+0000 7f5148c00640 10 client.25343 handle_command_reply: tid=0 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.439+0000 7f5168a3e740 1 client.25343 shutdown 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 msgr2=0x5581da907600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da907600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f514c009350 tx=0x7f514c00fcb0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 shutdown_connections 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da919d90 0x5581da91c180 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da907b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da907600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3639153610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 >> 172.21.15.33:0/3639153610 conn(0x5581da8fb4f0 msgr2=0x5581da901270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5148c00640 0 client.25343 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 shutdown_connections 2024-04-26T11:31:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 1 -- 172.21.15.33:0/3639153610 wait complete. 2024-04-26T11:31:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.440+0000 7f5168a3e740 20 client.25343 trim_cache size 0 max 16384 2024-04-26T11:31:32.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 Processor -- start 2024-04-26T11:31:32.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 -- start start 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8fefe0 0x5581da915cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8ff900 0x5581da9019a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da921a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b2eb0 con 0x5581da8fefe0 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b0780 con 0x5581da90a1c0 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.457+0000 7f5168a3e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51601ae5d0 con 0x5581da8ff900 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8ff900 0x5581da9019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da921a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da921a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42872/0 (socket says 172.21.15.33:42872) 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515f400640 1 -- 172.21.15.33:0/2053908586 learned_addr learned my addr 172.21.15.33:0/2053908586 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8ff900 0x5581da9019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52704/0 (socket says 172.21.15.33:52704) 2024-04-26T11:31:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515ea00640 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8fefe0 0x5581da915cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515e000640 1 -- 172.21.15.33:0/2053908586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 msgr2=0x5581da921a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515e000640 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da921a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515e000640 1 -- 172.21.15.33:0/2053908586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8fefe0 msgr2=0x5581da915cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515e000640 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8fefe0 0x5581da915cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.458+0000 7f515e000640 1 -- 172.21.15.33:0/2053908586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581da642620 con 0x5581da8ff900 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.459+0000 7f515e000640 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8ff900 0x5581da9019a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f514c0257e0 tx=0x7f514c009d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad46a93c3f095e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.459+0000 7f514b400640 1 -- 172.21.15.33:0/2053908586 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c029040 con 0x5581da8ff900 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.459+0000 7f514b400640 1 -- 172.21.15.33:0/2053908586 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f514c0044e0 con 0x5581da8ff900 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.459+0000 7f514b400640 1 -- 172.21.15.33:0/2053908586 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c000e60 con 0x5581da8ff900 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 -- 172.21.15.33:0/2053908586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8ff900 msgr2=0x5581da9019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8ff900 0x5581da9019a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f514c0257e0 tx=0x7f514c009d10 comp rx=0 tx=0).stop 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 -- 172.21.15.33:0/2053908586 shutdown_connections 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8ff900 0x5581da9019a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da90a1c0 0x5581da921a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 --2- 172.21.15.33:0/2053908586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da8fefe0 0x5581da915cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 -- 172.21.15.33:0/2053908586 >> 172.21.15.33:0/2053908586 conn(0x5581da8fb4f0 msgr2=0x5581da909070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 -- 172.21.15.33:0/2053908586 shutdown_connections 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 -- 172.21.15.33:0/2053908586 wait complete. 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 -- start start 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.460+0000 7f5168a3e740 1 Processor -- start 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da903bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f5168a3e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da904310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f5168a3e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f51601aec40 con 0x5581da90a1c0 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f5168a3e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b00b0 con 0x5581da8ff900 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f5168a3e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51601af310 con 0x5581da8fefe0 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f5168a3e740 10 client.0 _ll_register_callbacks cb 0x7f5158009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f515ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f515e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da903bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f515f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da904310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f515ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9126d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52716/0 (socket says 172.21.15.33:52716) 2024-04-26T11:31:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.461+0000 7f515ea00640 1 -- 172.21.15.33:0/3472178577 learned_addr learned my addr 172.21.15.33:0/3472178577 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f515f400640 1 -- 172.21.15.33:0/3472178577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 msgr2=0x5581da9126d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f515f400640 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9126d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f515f400640 1 -- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 msgr2=0x5581da903bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f515f400640 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da903bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f515f400640 1 -- 172.21.15.33:0/3472178577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581da81ecb0 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f515e000640 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da903bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f515f400640 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da904310 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5158009a60 tx=0x7f5158004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50cb554f67a442f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f5148c00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f5148c00640 1 -- 172.21.15.33:0/3472178577 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515802c070 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f5148c00640 1 -- 172.21.15.33:0/3472178577 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5158028840 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.462+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f514c0044e0 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5148c00640 1 -- 172.21.15.33:0/3472178577 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5158038050 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 10 client.15201 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581da9801f0 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5148c00640 1 -- 172.21.15.33:0/3472178577 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f514c0044e0 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5148c00640 1 -- 172.21.15.33:0/3472178577 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f515800b040 con 0x5581da90a1c0 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 10 client.15201 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 10 client.15201 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 20 client.15201 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da980a00 0x5581da982e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 4 client.15201 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:31:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.463+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5581da86be00 con 0x5581da980a00 2024-04-26T11:31:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.464+0000 7f515ea00640 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da980a00 0x5581da982e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.465+0000 7f515ea00640 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da980a00 0x5581da982e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90cb44ec5f0951ee server_cookie=31ed90ac7fb4cde0 in_seq=0 out_seq=0 2024-04-26T11:31:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.465+0000 7f5148c00640 10 client.15201 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.762+0000 7f5148c00640 1 -- 172.21.15.33:0/3472178577 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5581da86be00 con 0x5581da980a00 2024-04-26T11:31:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.762+0000 7f5148c00640 10 client.15201 handle_command_reply: tid=0 2024-04-26T11:31:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.762+0000 7f5168a3e740 1 client.15201 shutdown 2024-04-26T11:31:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 msgr2=0x5581da904310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da904310 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5158009a60 tx=0x7f5158004a40 comp rx=0 tx=0).stop 2024-04-26T11:31:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 shutdown_connections 2024-04-26T11:31:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5581da980a00 0x5581da982e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5581da90a1c0 0x5581da904310 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5581da8ff900 0x5581da903bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 --2- 172.21.15.33:0/3472178577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581da8fefe0 0x5581da9126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 >> 172.21.15.33:0/3472178577 conn(0x5581da8fb4f0 msgr2=0x5581da907060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5148c00640 0 client.15201 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 shutdown_connections 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.763+0000 7f5168a3e740 1 -- 172.21.15.33:0/3472178577 wait complete. 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.764+0000 7f5168a3e740 20 client.15201 trim_cache size 0 max 16384 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:31:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:31:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:31:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f51440a3e30 msgr2=0x7f51440a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f51440a3e30 0x7f51440a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5158005c40 tx=0x7f51580095c0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51601038f0 msgr2=0x7f5160101540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51601038f0 0x7f5160101540 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5150009980 tx=0x7f5150030fa0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 shutdown_connections 2024-04-26T11:31:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f51440a3e30 0x7f51440a62f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f516010cd10 0x7f5160101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5160104210 0x7f5160101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 --2- 172.21.15.33:0/1255540276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51601038f0 0x7f5160101540 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 >> 172.21.15.33:0/1255540276 conn(0x7f51600fd150 msgr2=0x7f51600febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.765+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 shutdown_connections 2024-04-26T11:31:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:32.766+0000 7f515fe00640 1 -- 172.21.15.33:0/1255540276 wait complete. 2024-04-26T11:31:32.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.810+0000 7f1b09600700 1 -- 172.21.15.33:0/3543129640 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x56217cad7550 con 0x56217cbf8790 2024-04-26T11:31:32.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b09600700 10 client.25331 handle_command_reply: tid=0 2024-04-26T11:31:32.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 client.25331 shutdown 2024-04-26T11:31:32.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 msgr2=0x56217cbf6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cbf6c30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x56217cb774a0 tx=0x7f1b14087200 comp rx=0 tx=0).stop 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 shutdown_connections 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56217cbf8790 0x56217cbfabb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56217cb80cc0 0x56217cbf6c30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b09600700 0 client.25331 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56217cb765f0 0x56217cb85ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 --2- 172.21.15.33:0/3543129640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56217cb76f10 0x56217cb86420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.811+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 >> 172.21.15.33:0/3543129640 conn(0x56217caa45d0 msgr2=0x56217cb75e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.812+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 shutdown_connections 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.812+0000 7f1b2d1a0b80 1 -- 172.21.15.33:0/3543129640 wait complete. 2024-04-26T11:31:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.812+0000 7f1b2d1a0b80 20 client.25331 trim_cache size 0 max 16384 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24527, 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 2, 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "5514a944": { 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 2, 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:31:32.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 60.0, 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 78.0, 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:32.816 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.813+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f1b000a40c0 msgr2=0x7f1b000a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.813+0000 7f1b22800700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f1b000a40c0 0x7f1b000a6580 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c1a9a20 tx=0x7f1b0c00b040 comp rx=0 tx=0).stop 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.813+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c1a88e0 msgr2=0x7f1b1c1a8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.813+0000 7f1b22800700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c1a88e0 0x7f1b1c1a8d60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400d980 tx=0x7f1b1400dd40 comp rx=0 tx=0).stop 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 shutdown_connections 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f1b000a40c0 0x7f1b000a6580 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b1c1a88e0 0x7f1b1c1a8d60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b1c111c70 0x7f1b1c1ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 --2- 172.21.15.33:0/570251029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b1c11cc00 0x7f1b1c1aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:32.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 >> 172.21.15.33:0/570251029 conn(0x7f1b1c06e590 msgr2=0x7f1b1c11abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:32.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 shutdown_connections 2024-04-26T11:31:32.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:32.814+0000 7f1b22800700 1 -- 172.21.15.33:0/570251029 wait complete. 2024-04-26T11:31:32.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:32 smithi194 ceph-mon[33017]: pgmap v37: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 MiB/s wr, 551 op/s 2024-04-26T11:31:32.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:31:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:32 smithi033 ceph-mon[27845]: pgmap v37: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 MiB/s wr, 551 op/s 2024-04-26T11:31:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:31:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:32 smithi096 ceph-mon[32753]: pgmap v37: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 MiB/s wr, 551 op/s 2024-04-26T11:31:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:31:33.691 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5514a944', quiesce took 9.4 seconds. Will release after: 18.0 2024-04-26T11:31:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:31:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:31:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:31:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:31:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:31:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:31:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:31:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:31:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:31:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:34 smithi033 ceph-mon[27845]: pgmap v38: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 MiB/s wr, 551 op/s 2024-04-26T11:31:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:34 smithi096 ceph-mon[32753]: pgmap v38: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 MiB/s wr, 551 op/s 2024-04-26T11:31:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:34 smithi194 ceph-mon[33017]: pgmap v38: 97 pgs: 97 active+clean; 938 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 MiB/s wr, 551 op/s 2024-04-26T11:31:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:36 smithi033 ceph-mon[27845]: pgmap v39: 97 pgs: 97 active+clean; 944 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 9.0 MiB/s wr, 814 op/s 2024-04-26T11:31:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:36 smithi096 ceph-mon[32753]: pgmap v39: 97 pgs: 97 active+clean; 944 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 9.0 MiB/s wr, 814 op/s 2024-04-26T11:31:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:36 smithi194 ceph-mon[33017]: pgmap v39: 97 pgs: 97 active+clean; 944 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 9.0 MiB/s wr, 814 op/s 2024-04-26T11:31:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 20.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:31:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:38 smithi096 ceph-mon[32753]: pgmap v40: 97 pgs: 97 active+clean; 944 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 6.3 MiB/s wr, 571 op/s 2024-04-26T11:31:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:38 smithi194 ceph-mon[33017]: pgmap v40: 97 pgs: 97 active+clean; 944 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 6.3 MiB/s wr, 571 op/s 2024-04-26T11:31:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:38 smithi033 ceph-mon[27845]: pgmap v40: 97 pgs: 97 active+clean; 944 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 6.3 MiB/s wr, 571 op/s 2024-04-26T11:31:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:40 smithi096 ceph-mon[32753]: pgmap v41: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 7.2 MiB/s wr, 584 op/s 2024-04-26T11:31:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:40 smithi194 ceph-mon[33017]: pgmap v41: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 7.2 MiB/s wr, 584 op/s 2024-04-26T11:31:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:40 smithi033 ceph-mon[27845]: pgmap v41: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 7.2 MiB/s wr, 584 op/s 2024-04-26T11:31:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 25.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:31:43.010 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:42 smithi096 ceph-mon[32753]: pgmap v42: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-26T11:31:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:42 smithi194 ceph-mon[33017]: pgmap v42: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-26T11:31:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:42 smithi033 ceph-mon[27845]: pgmap v42: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-26T11:31:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:44 smithi194 ceph-mon[33017]: pgmap v43: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-26T11:31:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:44 smithi033 ceph-mon[27845]: pgmap v43: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-26T11:31:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:44 smithi096 ceph-mon[32753]: pgmap v43: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-26T11:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 -- 172.21.15.33:0/2654877067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 msgr2=0x7f31a4109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2654877067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a4109570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3194009900 tx=0x7f319402a410 comp rx=0 tx=0).stop 2024-04-26T11:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 -- 172.21.15.33:0/2654877067 shutdown_connections 2024-04-26T11:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2654877067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a410e260 0x7f31a4115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2654877067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a4109b40 0x7f31a410db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2654877067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a4109570 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 -- 172.21.15.33:0/2654877067 >> 172.21.15.33:0/2654877067 conn(0x7f31a4076ec0 msgr2=0x7f31a40792e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 -- 172.21.15.33:0/2654877067 shutdown_connections 2024-04-26T11:31:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.915+0000 7f31a9e00640 1 -- 172.21.15.33:0/2654877067 wait complete. 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.916+0000 7f31a9e00640 1 Processor -- start 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.916+0000 7f31a9e00640 1 -- start start 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a410e260 0x7f31a419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a9e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a41a3f80 0x7f31a41a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a9e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4119220 con 0x7f31a41a3f80 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a9e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41190a0 con 0x7f31a410e260 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a9e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41193a0 con 0x7f31a4109190 2024-04-26T11:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a3400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a41a3f80 0x7f31a41a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:46738/0 (socket says 172.21.15.33:46738) 2024-04-26T11:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f31a2a00640 1 -- 172.21.15.33:0/2239152421 learned_addr learned my addr 172.21.15.33:0/2239152421 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.917+0000 7f319be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a410e260 0x7f31a419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a2a00640 1 -- 172.21.15.33:0/2239152421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a410e260 msgr2=0x7f31a419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a2a00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a410e260 0x7f31a419fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a2a00640 1 -- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a41a3f80 msgr2=0x7f31a41a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a2a00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a41a3f80 0x7f31a41a4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a2a00640 1 -- 172.21.15.33:0/2239152421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a41a4a70 con 0x7f31a4109190 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a2a00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a419f6c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3194005ab0 tx=0x7f3194003c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a8a00640 1 -- 172.21.15.33:0/2239152421 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319403f070 con 0x7f31a4109190 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a8a00640 1 -- 172.21.15.33:0/2239152421 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3194031d60 con 0x7f31a4109190 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.918+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a41a4d00 con 0x7f31a4109190 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.919+0000 7f31a8a00640 1 -- 172.21.15.33:0/2239152421 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3194035030 con 0x7f31a4109190 2024-04-26T11:31:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.919+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a41ac520 con 0x7f31a4109190 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 Processor -- start 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31a8a00640 1 -- 172.21.15.33:0/2239152421 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f3194042070 con 0x7f31a4109190 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 -- start start 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf3031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf30da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559dcf268380 con 0x559dcf3036f0 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559dcf204c30 con 0x559dcf30dfb0 2024-04-26T11:31:45.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.922+0000 7f31acc38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559dcf264340 con 0x559dcf302dd0 2024-04-26T11:31:45.925 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf30da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.925 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf30da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39992/0 (socket says 172.21.15.33:39992) 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf3031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 -- 172.21.15.33:0/1777969781 learned_addr learned my addr 172.21.15.33:0/1777969781 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f319be00640 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 -- 172.21.15.33:0/1777969781 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 msgr2=0x559dcf3031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf3031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 -- 172.21.15.33:0/1777969781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 msgr2=0x559dcf3103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 -- 172.21.15.33:0/1777969781 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dcf226940 con 0x559dcf3036f0 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.923+0000 7f31a2a00640 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf30da70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f31940ee620 tx=0x7f31940f93b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298283a549182a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.924+0000 7f31a8a00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f31880a3f60 0x7f31880a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.924+0000 7f31a8a00640 1 -- 172.21.15.33:0/2239152421 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f31940f1100 con 0x7f31a4109190 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.924+0000 7f3193e00640 1 -- 172.21.15.33:0/1777969781 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31940b9220 con 0x559dcf3036f0 2024-04-26T11:31:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.924+0000 7f31a3400640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f31880a3f60 0x7f31880a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.924+0000 7f3193e00640 1 -- 172.21.15.33:0/1777969781 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31940fe030 con 0x559dcf3036f0 2024-04-26T11:31:45.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.924+0000 7f3193e00640 1 -- 172.21.15.33:0/1777969781 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31940bc8e0 con 0x559dcf3036f0 2024-04-26T11:31:45.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 -- 172.21.15.33:0/1777969781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 msgr2=0x559dcf30da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf30da70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f31940ee620 tx=0x7f31940f93b0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31a3400640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f31880a3f60 0x7f31880a6420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f319c005fd0 tx=0x7f319c009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:45.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 -- 172.21.15.33:0/1777969781 shutdown_connections 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf30da70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 --2- 172.21.15.33:0/1777969781 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf3031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 -- 172.21.15.33:0/1777969781 >> 172.21.15.33:0/1777969781 conn(0x559dcf2ff2e0 msgr2=0x559dcf2ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 -- 172.21.15.33:0/1777969781 shutdown_connections 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 -- 172.21.15.33:0/1777969781 wait complete. 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.925+0000 7f31acc38740 1 -- start start 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 1 Processor -- start 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf30aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf3036f0 0x559dcf30b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf30dfb0 0x559dcf30b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559dcf3109a0 con 0x559dcf30dfb0 2024-04-26T11:31:45.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559dcf1efc60 con 0x559dcf3036f0 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559dcf310b20 con 0x559dcf302dd0 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31acc38740 10 client.0 _ll_register_callbacks cb 0x559dcf21be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.926+0000 7f31a3400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf30dfb0 0x559dcf30b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f319be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf3036f0 0x559dcf30b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f31a3400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf30dfb0 0x559dcf30b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40006/0 (socket says 172.21.15.33:40006) 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f31a3400640 1 -- 172.21.15.33:0/1962540481 learned_addr learned my addr 172.21.15.33:0/1962540481 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f31a2a00640 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf30aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f319be00640 1 -- 172.21.15.33:0/1962540481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 msgr2=0x559dcf30aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f319be00640 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf30aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f319be00640 1 -- 172.21.15.33:0/1962540481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf30dfb0 msgr2=0x559dcf30b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f319be00640 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf30dfb0 0x559dcf30b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f319be00640 1 -- 172.21.15.33:0/1962540481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dcf1fe510 con 0x559dcf3036f0 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f31a3400640 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf30dfb0 0x559dcf30b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:45.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f319be00640 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf3036f0 0x559dcf30b410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x559dcf300470 tx=0x7f318c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=123ddd25ee0cb359 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:45.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.927+0000 7f3190c00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:45.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.928+0000 7f3190c00640 1 -- 172.21.15.33:0/1962540481 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c004020 con 0x559dcf3036f0 2024-04-26T11:31:45.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.928+0000 7f3190c00640 1 -- 172.21.15.33:0/1962540481 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f318c0041c0 con 0x559dcf3036f0 2024-04-26T11:31:45.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.928+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31940fe030 con 0x559dcf3036f0 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.928+0000 7f3190c00640 1 -- 172.21.15.33:0/1962540481 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f318c0043c0 con 0x559dcf3036f0 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.928+0000 7f31acc38740 10 client.25361 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.928+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559dcef7a210 con 0x559dcf3036f0 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f3190c00640 1 -- 172.21.15.33:0/1962540481 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31940fe030 con 0x559dcf3036f0 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f3190c00640 1 -- 172.21.15.33:0/1962540481 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f318c004ca0 con 0x559dcf3036f0 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f31acc38740 10 client.25361 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f31acc38740 10 client.25361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f31acc38740 20 client.25361 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:45.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f31acc38740 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf31daa0 0x559dcf31feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f31acc38740 4 client.25361 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.929+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dcf275f90 con 0x559dcf31daa0 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.930+0000 7f31a2a00640 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf31daa0 0x559dcf31feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.930+0000 7f31a2a00640 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf31daa0 0x559dcf31feb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=190cfb8ce27a391b server_cookie=38a6e48c4732b386 in_seq=0 out_seq=0 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.930+0000 7f3190c00640 10 client.25361 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f3190c00640 1 -- 172.21.15.33:0/1962540481 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559dcf275f90 con 0x559dcf31daa0 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f3190c00640 10 client.25361 handle_command_reply: tid=0 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f31acc38740 1 client.25361 shutdown 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf3036f0 msgr2=0x559dcf30b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f31acc38740 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf3036f0 0x559dcf30b410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x559dcf300470 tx=0x7f318c0118d0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 shutdown_connections 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f31acc38740 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf31daa0 0x559dcf31feb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.931+0000 7f31acc38740 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf30dfb0 0x559dcf30b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.932+0000 7f3190c00640 0 client.25361 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.932+0000 7f31acc38740 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf3036f0 0x559dcf30b410 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.932+0000 7f31acc38740 1 --2- 172.21.15.33:0/1962540481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf30aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.932+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 >> 172.21.15.33:0/1962540481 conn(0x559dcf2ff2e0 msgr2=0x559dcf305010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:45.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.932+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 shutdown_connections 2024-04-26T11:31:45.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.932+0000 7f31acc38740 1 -- 172.21.15.33:0/1962540481 wait complete. 2024-04-26T11:31:45.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.932+0000 7f31acc38740 20 client.25361 trim_cache size 0 max 16384 2024-04-26T11:31:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.948+0000 7f31acc38740 1 Processor -- start 2024-04-26T11:31:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31acc38740 1 -- start start 2024-04-26T11:31:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31acc38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf319740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31acc38740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf305790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31acc38740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3257c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31acc38740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41b73f0 con 0x559dcf3036f0 2024-04-26T11:31:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31acc38740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41b4cc0 con 0x559dcf30dfb0 2024-04-26T11:31:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31acc38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41b2b10 con 0x559dcf302dd0 2024-04-26T11:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf319740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f319be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf305790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf319740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:46766/0 (socket says 172.21.15.33:46766) 2024-04-26T11:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f319be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf305790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40014/0 (socket says 172.21.15.33:40014) 2024-04-26T11:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31a2a00640 1 -- 172.21.15.33:0/465031621 learned_addr learned my addr 172.21.15.33:0/465031621 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.949+0000 7f31a3400640 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3257c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f31a2a00640 1 -- 172.21.15.33:0/465031621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 msgr2=0x559dcf3257c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f31a2a00640 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3257c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f31a2a00640 1 -- 172.21.15.33:0/465031621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 msgr2=0x559dcf305790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f31a2a00640 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf305790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f31a2a00640 1 -- 172.21.15.33:0/465031621 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dcf046620 con 0x559dcf302dd0 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f319be00640 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf305790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f31a2a00640 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf319740 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31940c5610 tx=0x7f319402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52833ff903b0d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f3193400640 1 -- 172.21.15.33:0/465031621 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3194049040 con 0x559dcf302dd0 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f3193400640 1 -- 172.21.15.33:0/465031621 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31940fe070 con 0x559dcf302dd0 2024-04-26T11:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.950+0000 7f3193400640 1 -- 172.21.15.33:0/465031621 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31940bcac0 con 0x559dcf302dd0 2024-04-26T11:31:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 -- 172.21.15.33:0/465031621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 msgr2=0x559dcf319740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf319740 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31940c5610 tx=0x7f319402a410 comp rx=0 tx=0).stop 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 -- 172.21.15.33:0/465031621 shutdown_connections 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf3257c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf3036f0 0x559dcf305790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 --2- 172.21.15.33:0/465031621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf302dd0 0x559dcf319740 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 -- 172.21.15.33:0/465031621 >> 172.21.15.33:0/465031621 conn(0x559dcf2ff2e0 msgr2=0x559dcf2ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 -- 172.21.15.33:0/465031621 shutdown_connections 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.951+0000 7f31acc38740 1 -- 172.21.15.33:0/465031621 wait complete. 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 -- start start 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 Processor -- start 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf302dd0 0x559dcf3160e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf3036f0 0x559dcf307980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf307f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41b3180 con 0x559dcf302dd0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41b45f0 con 0x559dcf30dfb0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41b3850 con 0x559dcf3036f0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.952+0000 7f31acc38740 10 client.0 _ll_register_callbacks cb 0x7f318c00bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf302dd0 0x559dcf3160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf302dd0 0x559dcf3160e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40028/0 (socket says 172.21.15.33:40028) 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f31a2a00640 1 -- 172.21.15.33:0/2803267038 learned_addr learned my addr 172.21.15.33:0/2803267038 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f319be00640 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf3036f0 0x559dcf307980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f31a3400640 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf307f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f319be00640 1 -- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 msgr2=0x559dcf307f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f319be00640 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf307f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f319be00640 1 -- 172.21.15.33:0/2803267038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf302dd0 msgr2=0x559dcf3160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f319be00640 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf302dd0 0x559dcf3160e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.953+0000 7f319be00640 1 -- 172.21.15.33:0/2803267038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dcf222cb0 con 0x559dcf3036f0 2024-04-26T11:31:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f319be00640 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf3036f0 0x559dcf307980 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f318c001eb0 tx=0x7f318c0052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76d811e26b9dd51f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f3190c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f3190c00640 1 -- 172.21.15.33:0/2803267038 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c000e60 con 0x559dcf3036f0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f3190c00640 1 -- 172.21.15.33:0/2803267038 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f318c017070 con 0x559dcf3036f0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31940fe070 con 0x559dcf3036f0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f3190c00640 1 -- 172.21.15.33:0/2803267038 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f318c026030 con 0x559dcf3036f0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f31acc38740 10 client.25330 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.954+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559dcf383f10 con 0x559dcf3036f0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f3190c00640 1 -- 172.21.15.33:0/2803267038 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31940fe070 con 0x559dcf3036f0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f3190c00640 1 -- 172.21.15.33:0/2803267038 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f318c004b30 con 0x559dcf3036f0 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f31acc38740 10 client.25330 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f31acc38740 10 client.25330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f31acc38740 20 client.25330 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f31acc38740 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf384740 0x559dcf386b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f31acc38740 4 client.25330 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:31:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559dcf26fe00 con 0x559dcf384740 2024-04-26T11:31:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.955+0000 7f31a2a00640 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf384740 0x559dcf386b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.956+0000 7f31a2a00640 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf384740 0x559dcf386b60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd1b20f62bd2f42 server_cookie=42fbc474b24fb27b in_seq=0 out_seq=0 2024-04-26T11:31:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.956+0000 7f3190c00640 10 client.25330 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f3190c00640 1 -- 172.21.15.33:0/2803267038 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559dcf26fe00 con 0x559dcf384740 2024-04-26T11:31:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f3190c00640 10 client.25330 handle_command_reply: tid=0 2024-04-26T11:31:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 client.25330 shutdown 2024-04-26T11:31:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf3036f0 msgr2=0x559dcf307980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf3036f0 0x559dcf307980 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f318c001eb0 tx=0x7f318c0052f0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 shutdown_connections 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559dcf3036f0 0x559dcf307980 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559dcf384740 0x559dcf386b60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559dcf30dfb0 0x559dcf307f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 --2- 172.21.15.33:0/2803267038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559dcf302dd0 0x559dcf3160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f3190c00640 0 client.25330 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.983+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 >> 172.21.15.33:0/2803267038 conn(0x559dcf2ff2e0 msgr2=0x559dcf30ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.984+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 shutdown_connections 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.984+0000 7f31acc38740 1 -- 172.21.15.33:0/2803267038 wait complete. 2024-04-26T11:31:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.984+0000 7f31acc38740 20 client.25330 trim_cache size 0 max 16384 2024-04-26T11:31:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:31:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:31:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:31:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f31880a3f60 msgr2=0x7f31880a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f31880a3f60 0x7f31880a6420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f319c005fd0 tx=0x7f319c009500 comp rx=0 tx=0).stop 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 msgr2=0x7f31a419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a419f6c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3194005ab0 tx=0x7f3194003c80 comp rx=0 tx=0).stop 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 shutdown_connections 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f31880a3f60 0x7f31880a6420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f31a41a3f80 0x7f31a41a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a410e260 0x7f31a419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 --2- 172.21.15.33:0/2239152421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a4109190 0x7f31a419f6c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 >> 172.21.15.33:0/2239152421 conn(0x7f31a4076ec0 msgr2=0x7f31a4077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 shutdown_connections 2024-04-26T11:31:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:45.985+0000 7f31a9e00640 1 -- 172.21.15.33:0/2239152421 wait complete. 2024-04-26T11:31:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:46 smithi194 ceph-mon[33017]: pgmap v44: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 2.7 MiB/s wr, 279 op/s 2024-04-26T11:31:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:46 smithi033 ceph-mon[27845]: pgmap v44: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 2.7 MiB/s wr, 279 op/s 2024-04-26T11:31:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:46 smithi096 ceph-mon[32753]: pgmap v44: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 2.7 MiB/s wr, 279 op/s 2024-04-26T11:31:47.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 30.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:31:48.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:48 smithi033 ceph-mon[27845]: pgmap v45: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 837 KiB/s wr, 16 op/s 2024-04-26T11:31:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:48 smithi194 ceph-mon[33017]: pgmap v45: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 837 KiB/s wr, 16 op/s 2024-04-26T11:31:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:48 smithi096 ceph-mon[32753]: pgmap v45: 97 pgs: 97 active+clean; 945 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 837 KiB/s wr, 16 op/s 2024-04-26T11:31:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:50 smithi194 ceph-mon[33017]: pgmap v46: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.9 MiB/s wr, 32 op/s 2024-04-26T11:31:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:50 smithi033 ceph-mon[27845]: pgmap v46: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.9 MiB/s wr, 32 op/s 2024-04-26T11:31:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:50 smithi096 ceph-mon[32753]: pgmap v46: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.9 MiB/s wr, 32 op/s 2024-04-26T11:31:51.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:31:52.145 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:31:52.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 35.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:31:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:52 smithi033 ceph-mon[27845]: pgmap v47: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-04-26T11:31:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:52 smithi194 ceph-mon[33017]: pgmap v47: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-04-26T11:31:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:52 smithi096 ceph-mon[32753]: pgmap v47: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-04-26T11:31:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:53 smithi033 ceph-mon[27845]: pgmap v48: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-04-26T11:31:54.002 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:53 smithi096 ceph-mon[32753]: pgmap v48: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-04-26T11:31:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:53 smithi194 ceph-mon[33017]: pgmap v48: 97 pgs: 97 active+clean; 945 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-04-26T11:31:54.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 -- 172.21.15.33:0/19927394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd8102e60 msgr2=0x7fbcd8106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:54.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/19927394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd8102e60 0x7fbcd8106eb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbcc40099c0 tx=0x7fbcc4009cd0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 -- 172.21.15.33:0/19927394 shutdown_connections 2024-04-26T11:31:54.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/19927394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcd8107590 0x7fbcd810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/19927394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd81024b0 0x7fbcd8102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/19927394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd8102e60 0x7fbcd8106eb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 -- 172.21.15.33:0/19927394 >> 172.21.15.33:0/19927394 conn(0x7fbcd80fdd40 msgr2=0x7fbcd8100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 -- 172.21.15.33:0/19927394 shutdown_connections 2024-04-26T11:31:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.498+0000 7fbcd7e00700 1 -- 172.21.15.33:0/19927394 wait complete. 2024-04-26T11:31:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.499+0000 7fbcd7e00700 1 Processor -- start 2024-04-26T11:31:54.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.500+0000 7fbcd7e00700 1 -- start start 2024-04-26T11:31:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.500+0000 7fbcd7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd81024b0 0x7fbcd8103f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.500+0000 7fbcd7e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd8102e60 0x7fbcd8104490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.500+0000 7fbcd7e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcd8107590 0x7fbcd81049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.500+0000 7fbcd7e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd8112400 con 0x7fbcd81024b0 2024-04-26T11:31:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.500+0000 7fbcd7e00700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd8112280 con 0x7fbcd8107590 2024-04-26T11:31:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.500+0000 7fbcd7e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd8112580 con 0x7fbcd8102e60 2024-04-26T11:31:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd81024b0 0x7fbcd8103f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd81024b0 0x7fbcd8103f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42664/0 (socket says 172.21.15.33:42664) 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd7400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcd8107590 0x7fbcd81049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd6a00700 1 -- 172.21.15.33:0/2975129992 learned_addr learned my addr 172.21.15.33:0/2975129992 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd6000700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd8102e60 0x7fbcd8104490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd7400700 1 -- 172.21.15.33:0/2975129992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd8102e60 msgr2=0x7fbcd8104490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd7400700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd8102e60 0x7fbcd8104490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd7400700 1 -- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd81024b0 msgr2=0x7fbcd8103f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd7400700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd81024b0 0x7fbcd8103f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.501+0000 7fbcd7400700 1 -- 172.21.15.33:0/2975129992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd81a7f60 con 0x7fbcd8107590 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.502+0000 7fbcd7400700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcd8107590 0x7fbcd81049d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000efb0 tx=0x7fbcd000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:54.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.502+0000 7fbccf400700 1 -- 172.21.15.33:0/2975129992 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0003a00 con 0x7fbcd8107590 2024-04-26T11:31:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.502+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd81a8250 con 0x7fbcd8107590 2024-04-26T11:31:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.502+0000 7fbccf400700 1 -- 172.21.15.33:0/2975129992 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd000ce50 con 0x7fbcd8107590 2024-04-26T11:31:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.502+0000 7fbccf400700 1 -- 172.21.15.33:0/2975129992 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00047c0 con 0x7fbcd8107590 2024-04-26T11:31:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.502+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd81a8710 con 0x7fbcd8107590 2024-04-26T11:31:54.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.503+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd806a110 con 0x7fbcd8107590 2024-04-26T11:31:54.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.505+0000 7fbccf400700 1 -- 172.21.15.33:0/2975129992 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7fbcd0012050 con 0x7fbcd8107590 2024-04-26T11:31:54.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.507+0000 7fbccf400700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcbc0a3ed0 0x7fbcbc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:54.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.507+0000 7fbccf400700 1 -- 172.21.15.33:0/2975129992 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbcd00d0320 con 0x7fbcd8107590 2024-04-26T11:31:54.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.507+0000 7fbcd6a00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcbc0a3ed0 0x7fbcbc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:54.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.508+0000 7fbcd6a00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcbc0a3ed0 0x7fbcbc0a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009d20 tx=0x7fbcc8009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:54.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.511+0000 7fbccf400700 1 -- 172.21.15.33:0/2975129992 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbcd0022060 con 0x7fbcd8107590 2024-04-26T11:31:54.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.794+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcd802e490 con 0x7fbcd8107590 2024-04-26T11:31:54.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.796+0000 7fbccf400700 1 -- 172.21.15.33:0/2975129992 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11868 (secure 0 0 0) 0x7fbcd0010040 con 0x7fbcd8107590 2024-04-26T11:31:54.797 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:31:54.797 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:31:11.426188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:31:54.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcbc0a3ed0 msgr2=0x7fbcbc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:54.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcbc0a3ed0 0x7fbcbc0a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009d20 tx=0x7fbcc8009260 comp rx=0 tx=0).stop 2024-04-26T11:31:54.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcd8107590 msgr2=0x7fbcd81049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:54.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcd8107590 0x7fbcd81049d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000efb0 tx=0x7fbcd000c3a0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 shutdown_connections 2024-04-26T11:31:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fbcbc0a3ed0 0x7fbcbc0a6390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbcd8107590 0x7fbcd81049d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbcd81024b0 0x7fbcd8103f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 --2- 172.21.15.33:0/2975129992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd8102e60 0x7fbcd8104490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 >> 172.21.15.33:0/2975129992 conn(0x7fbcd80fdd40 msgr2=0x7fbcd806b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 shutdown_connections 2024-04-26T11:31:54.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:54.799+0000 7fbcd7e00700 1 -- 172.21.15.33:0/2975129992 wait complete. 2024-04-26T11:31:54.801 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-04-26T11:31:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:54 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2975129992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:31:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:54 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2975129992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:31:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:54 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2975129992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:31:55.704 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24527 quiesce db --set-id 5514a944 --release --await' 2024-04-26T11:31:55.705 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24527 quiesce db --set-id 5514a944 --release --await 2024-04-26T11:31:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:55 smithi033 ceph-mon[27845]: pgmap v49: 97 pgs: 97 active+clean; 949 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 MiB/s wr, 25 op/s 2024-04-26T11:31:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:55 smithi096 ceph-mon[32753]: pgmap v49: 97 pgs: 97 active+clean; 949 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 MiB/s wr, 25 op/s 2024-04-26T11:31:56.184 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:31:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:55 smithi194 ceph-mon[33017]: pgmap v49: 97 pgs: 97 active+clean; 949 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 MiB/s wr, 25 op/s 2024-04-26T11:31:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:56 smithi033 ceph-mon[27845]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-04-26T11:31:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:56 smithi096 ceph-mon[32753]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-04-26T11:31:57.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 40.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:31:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:56 smithi194 ceph-mon[33017]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-04-26T11:31:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:31:57 smithi033 ceph-mon[27845]: pgmap v50: 97 pgs: 97 active+clean; 949 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 MiB/s wr, 21 op/s 2024-04-26T11:31:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:31:57 smithi096 ceph-mon[32753]: pgmap v50: 97 pgs: 97 active+clean; 949 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 MiB/s wr, 21 op/s 2024-04-26T11:31:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:31:57 smithi194 ceph-mon[33017]: pgmap v50: 97 pgs: 97 active+clean; 949 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 MiB/s wr, 21 op/s 2024-04-26T11:31:58.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.690+0000 7f3d6b200700 1 -- 172.21.15.33:0/1011377663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 msgr2=0x7f3d6406b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.690+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1011377663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6406b590 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009990 tx=0x7f3d4c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.691+0000 7f3d6b200700 1 -- 172.21.15.33:0/1011377663 shutdown_connections 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.691+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1011377663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d6406bad0 0x7f3d641102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.691+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1011377663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d641094a0 0x7f3d64109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.691+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1011377663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6406b590 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.691+0000 7f3d6b200700 1 -- 172.21.15.33:0/1011377663 >> 172.21.15.33:0/1011377663 conn(0x7f3d640792d0 msgr2=0x7f3d640796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.691+0000 7f3d6b200700 1 -- 172.21.15.33:0/1011377663 shutdown_connections 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.691+0000 7f3d6b200700 1 -- 172.21.15.33:0/1011377663 wait complete. 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.692+0000 7f3d6b200700 1 Processor -- start 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.692+0000 7f3d6b200700 1 -- start start 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d6b200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d6b200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d6406bad0 0x7f3d6419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d6b200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d641094a0 0x7f3d641a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d6b200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64113600 con 0x7f3d6406b130 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d6b200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64113480 con 0x7f3d6406bad0 2024-04-26T11:31:58.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d6b200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64113780 con 0x7f3d641094a0 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6419bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42684/0 (socket says 172.21.15.33:42684) 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d63e00700 1 -- 172.21.15.33:0/1289169211 learned_addr learned my addr 172.21.15.33:0/1289169211 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d5b400700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d6406bad0 0x7f3d6419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.693+0000 7f3d69200700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d641094a0 0x7f3d641a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d63e00700 1 -- 172.21.15.33:0/1289169211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d641094a0 msgr2=0x7f3d641a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d63e00700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d641094a0 0x7f3d641a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d63e00700 1 -- 172.21.15.33:0/1289169211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d6406bad0 msgr2=0x7f3d6419c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d63e00700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d6406bad0 0x7f3d6419c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d63e00700 1 -- 172.21.15.33:0/1289169211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d641a0dc0 con 0x7f3d6406b130 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d63e00700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6419bdb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400eb30 tx=0x7f3d5400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d61600700 1 -- 172.21.15.33:0/1289169211 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5400cc90 con 0x7f3d6406b130 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d61600700 1 -- 172.21.15.33:0/1289169211 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d540044d0 con 0x7f3d6406b130 2024-04-26T11:31:58.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d61600700 1 -- 172.21.15.33:0/1289169211 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54010430 con 0x7f3d6406b130 2024-04-26T11:31:58.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.694+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d641a10b0 con 0x7f3d6406b130 2024-04-26T11:31:58.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.695+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d641a8930 con 0x7f3d6406b130 2024-04-26T11:31:58.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.698+0000 7f3d61600700 1 -- 172.21.15.33:0/1289169211 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3d54003ea0 con 0x7f3d6406b130 2024-04-26T11:31:58.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.699+0000 7f3d61600700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f3d440a3e30 0x7f3d440a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.699+0000 7f3d75ae4b80 1 Processor -- start 2024-04-26T11:31:58.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d61600700 1 -- 172.21.15.33:0/1289169211 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3d540d09c0 con 0x7f3d6406b130 2024-04-26T11:31:58.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d5b400700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f3d440a3e30 0x7f3d440a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d75ae4b80 1 -- start start 2024-04-26T11:31:58.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f2df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f27ee0 0x559105f28340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f28880 0x559105f34900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559105e7a990 con 0x559105f2dbb0 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559105e192b0 con 0x559105f28880 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.700+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559105e76950 con 0x559105f27ee0 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f2df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f2df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42686/0 (socket says 172.21.15.33:42686) 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d5b400700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f3d440a3e30 0x7f3d440a62f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d64079980 tx=0x7f3d4c00b260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d69200700 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f28880 0x559105f34900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d63e00700 1 -- 172.21.15.33:0/4203701157 learned_addr learned my addr 172.21.15.33:0/4203701157 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:58.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d5b400700 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f27ee0 0x559105f28340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d69200700 1 -- 172.21.15.33:0/4203701157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f27ee0 msgr2=0x559105f28340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d69200700 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f27ee0 0x559105f28340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d69200700 1 -- 172.21.15.33:0/4203701157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 msgr2=0x559105f2df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d69200700 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f2df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.701+0000 7f3d69200700 1 -- 172.21.15.33:0/4203701157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559105e21730 con 0x559105f28880 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.702+0000 7f3d69200700 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f28880 0x559105f34900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009310 tx=0x7f3d5c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f464f9a050f116b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.702+0000 7f3d59600700 1 -- 172.21.15.33:0/4203701157 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c019020 con 0x559105f28880 2024-04-26T11:31:58.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.702+0000 7f3d59600700 1 -- 172.21.15.33:0/4203701157 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d5c015af0 con 0x559105f28880 2024-04-26T11:31:58.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.702+0000 7f3d59600700 1 -- 172.21.15.33:0/4203701157 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c014980 con 0x559105f28880 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/4203701157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f28880 msgr2=0x559105f34900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f28880 0x559105f34900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009310 tx=0x7f3d5c009920 comp rx=0 tx=0).stop 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/4203701157 shutdown_connections 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f28880 0x559105f34900 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f2df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/4203701157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f27ee0 0x559105f28340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/4203701157 >> 172.21.15.33:0/4203701157 conn(0x559105e553f0 msgr2=0x559105f22a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:58.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/4203701157 shutdown_connections 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.703+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/4203701157 wait complete. 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.704+0000 7f3d75ae4b80 1 -- start start 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.704+0000 7f3d75ae4b80 1 Processor -- start 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.704+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f34470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.704+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f31770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.704+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f31cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559105f34e40 con 0x559105f28880 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559105e15500 con 0x559105f27ee0 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559105f34fc0 con 0x559105f2dbb0 2024-04-26T11:31:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d75ae4b80 10 client.0 _ll_register_callbacks cb 0x559105e0e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:58.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d5b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f31770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d5b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f31770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42690/0 (socket says 172.21.15.33:42690) 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d5b400700 1 -- 172.21.15.33:0/1889362540 learned_addr learned my addr 172.21.15.33:0/1889362540 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d63e00700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f34470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d69200700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f31cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d63e00700 1 -- 172.21.15.33:0/1889362540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 msgr2=0x559105f31cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d63e00700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f31cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d63e00700 1 -- 172.21.15.33:0/1889362540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 msgr2=0x559105f31770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d63e00700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f31770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d63e00700 1 -- 172.21.15.33:0/1889362540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559105dc79d0 con 0x559105f27ee0 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d69200700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f31cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.705+0000 7f3d5b400700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f31770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d63e00700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f34470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3d540a5050 tx=0x7f3d5409c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6188b4db2674c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d52000700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d52000700 1 -- 172.21.15.33:0/1889362540 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5409d010 con 0x559105f27ee0 2024-04-26T11:31:58.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d52000700 1 -- 172.21.15.33:0/1889362540 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3d54004190 con 0x559105f27ee0 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d5c015af0 con 0x559105f27ee0 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d52000700 1 -- 172.21.15.33:0/1889362540 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3d5409c910 con 0x559105f27ee0 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d75ae4b80 10 client.25385 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.706+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559105f3f500 con 0x559105f27ee0 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d52000700 1 -- 172.21.15.33:0/1889362540 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d5c015af0 con 0x559105f27ee0 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d52000700 1 -- 172.21.15.33:0/1889362540 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f3d5409cb70 con 0x559105f27ee0 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d75ae4b80 10 client.25385 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d75ae4b80 10 client.25385 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d75ae4b80 20 client.25385 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105f41ae0 0x559105f43f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d75ae4b80 4 client.25385 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559105e88370 con 0x559105f41ae0 2024-04-26T11:31:58.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.707+0000 7f3d5b400700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105f41ae0 0x559105f43f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.708+0000 7f3d5b400700 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105f41ae0 0x559105f43f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f17aea89b3a5ffe server_cookie=45310023133d40b0 in_seq=0 out_seq=0 2024-04-26T11:31:58.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.708+0000 7f3d52000700 10 client.25385 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d52000700 1 -- 172.21.15.33:0/1889362540 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559105e88370 con 0x559105f41ae0 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d52000700 10 client.25385 handle_command_reply: tid=0 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 client.25385 shutdown 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 msgr2=0x559105f34470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f34470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3d540a5050 tx=0x7f3d5409c4d0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 shutdown_connections 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105f41ae0 0x559105f43f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f31cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f34470 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/1889362540 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f31770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d52000700 0 client.25385 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 >> 172.21.15.33:0/1889362540 conn(0x559105e553f0 msgr2=0x559105f2cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.709+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 shutdown_connections 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.710+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/1889362540 wait complete. 2024-04-26T11:31:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.710+0000 7f3d75ae4b80 20 client.25385 trim_cache size 0 max 16384 2024-04-26T11:31:58.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.731+0000 7f3d75ae4b80 1 Processor -- start 2024-04-26T11:31:58.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.731+0000 7f3d75ae4b80 1 -- start start 2024-04-26T11:31:58.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.731+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f1d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.731+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f3dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.731+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f308f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.731+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3d641b28a0 con 0x559105f28880 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d641b0130 con 0x559105f27ee0 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d641adf80 con 0x559105f2dbb0 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d5b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f3dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f1d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d5b400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f3dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42702/0 (socket says 172.21.15.33:42702) 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d5b400700 1 -- 172.21.15.33:0/2676396901 learned_addr learned my addr 172.21.15.33:0/2676396901 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:58.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d69200700 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d63e00700 1 -- 172.21.15.33:0/2676396901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 msgr2=0x559105f308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d63e00700 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f308f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d63e00700 1 -- 172.21.15.33:0/2676396901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 msgr2=0x559105f3dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d63e00700 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f3dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.732+0000 7f3d63e00700 1 -- 172.21.15.33:0/2676396901 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559105f1f3a0 con 0x559105f27ee0 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.733+0000 7f3d63e00700 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f1d080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400eb00 tx=0x7f3d54017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e60931f733468e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.733+0000 7f3d58c00700 1 -- 172.21.15.33:0/2676396901 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5409c270 con 0x559105f27ee0 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.733+0000 7f3d58c00700 1 -- 172.21.15.33:0/2676396901 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d54085e50 con 0x559105f27ee0 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.733+0000 7f3d58c00700 1 -- 172.21.15.33:0/2676396901 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d540870b0 con 0x559105f27ee0 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/2676396901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 msgr2=0x559105f1d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f1d080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400eb00 tx=0x7f3d54017000 comp rx=0 tx=0).stop 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/2676396901 shutdown_connections 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f2dbb0 0x559105f308f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f1d080 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/2676396901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f28880 0x559105f3dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/2676396901 >> 172.21.15.33:0/2676396901 conn(0x559105e553f0 msgr2=0x559105f26ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/2676396901 shutdown_connections 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/2676396901 wait complete. 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.734+0000 7f3d75ae4b80 1 -- start start 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 1 Processor -- start 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f3a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f28880 0x559105f3a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f491c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3d641ae5f0 con 0x559105f2dbb0 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d641afa60 con 0x559105f27ee0 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d641aecc0 con 0x559105f28880 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d75ae4b80 10 client.0 _ll_register_callbacks cb 0x559105e0e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.735+0000 7f3d69200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f3a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f3a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55552/0 (socket says 172.21.15.33:55552) 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 -- 172.21.15.33:0/265040157 learned_addr learned my addr 172.21.15.33:0/265040157 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d69200700 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42708/0 (socket says 172.21.15.33:42708) 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d5b400700 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f28880 0x559105f3a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 -- 172.21.15.33:0/265040157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f28880 msgr2=0x559105f3a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f28880 0x559105f3a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 -- 172.21.15.33:0/265040157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 msgr2=0x559105f491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f491c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 -- 172.21.15.33:0/265040157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559105f1d8b0 con 0x559105f27ee0 2024-04-26T11:31:58.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d63e00700 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f3a210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3d5409c610 tx=0x7f3d5409c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=872815b394b37691 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:58.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d52000700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:31:58.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.736+0000 7f3d52000700 1 -- 172.21.15.33:0/265040157 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d540a5580 con 0x559105f27ee0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d52000700 1 -- 172.21.15.33:0/265040157 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3d540870d0 con 0x559105f27ee0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d54085e50 con 0x559105f27ee0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d52000700 1 -- 172.21.15.33:0/265040157 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3d5409d010 con 0x559105f27ee0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 10 client.25391 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559105f1d8b0 con 0x559105f27ee0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d52000700 1 -- 172.21.15.33:0/265040157 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d54085e50 con 0x559105f27ee0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d52000700 1 -- 172.21.15.33:0/265040157 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f3d540bbb60 con 0x559105f27ee0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 10 client.25391 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 10 client.25391 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 20 client.25391 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105fa8a40 0x559105faae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 4 client.25391 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5514a944", "release": true, "await": true}] 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.737+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5514a944", "release": true, "await": true}) -- 0x559105e82410 con 0x559105fa8a40 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.738+0000 7f3d5b400700 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105fa8a40 0x559105faae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:58.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.738+0000 7f3d5b400700 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105fa8a40 0x559105faae60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb2daf28fbd96a0a server_cookie=29ee6dd1211e5fb9 in_seq=0 out_seq=0 2024-04-26T11:31:58.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.739+0000 7f3d52000700 10 client.25391 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:58.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.768+0000 7f3d52000700 1 -- 172.21.15.33:0/265040157 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x559105e82410 con 0x559105fa8a40 2024-04-26T11:31:58.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.768+0000 7f3d52000700 10 client.25391 handle_command_reply: tid=0 2024-04-26T11:31:58.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.768+0000 7f3d75ae4b80 1 client.25391 shutdown 2024-04-26T11:31:58.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.769+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 msgr2=0x559105f3a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.769+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f3a210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3d5409c610 tx=0x7f3d5409c4d0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 shutdown_connections 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559105fa8a40 0x559105faae60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559105f2dbb0 0x559105f491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559105f27ee0 0x559105f3a210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d75ae4b80 1 --2- 172.21.15.33:0/265040157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559105f28880 0x559105f3a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d52000700 0 client.25391 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 >> 172.21.15.33:0/265040157 conn(0x559105e553f0 msgr2=0x559105f318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.771+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 shutdown_connections 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.772+0000 7f3d75ae4b80 1 -- 172.21.15.33:0/265040157 wait complete. 2024-04-26T11:31:58.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.772+0000 7f3d75ae4b80 20 client.25391 trim_cache size 0 max 16384 2024-04-26T11:31:58.774 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:31:58.774 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-04-26T11:31:58.774 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24527, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 4, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "5514a944": { 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 4, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 60.0, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 78.0, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-04-26T11:31:58.775 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.775+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f3d440a3e30 msgr2=0x7f3d440a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.775+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f3d440a3e30 0x7f3d440a62f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d64079980 tx=0x7f3d4c00b260 comp rx=0 tx=0).stop 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.775+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 msgr2=0x7f3d6419bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.775+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6419bdb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400eb30 tx=0x7f3d5400ee40 comp rx=0 tx=0).stop 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 shutdown_connections 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f3d440a3e30 0x7f3d440a62f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d641094a0 0x7f3d641a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d6406b130 0x7f3d6419bdb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 --2- 172.21.15.33:0/1289169211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d6406bad0 0x7f3d6419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 >> 172.21.15.33:0/1289169211 conn(0x7f3d640792d0 msgr2=0x7f3d6410e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:58.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 shutdown_connections 2024-04-26T11:31:58.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:31:58.776+0000 7f3d6b200700 1 -- 172.21.15.33:0/1289169211 wait complete. 2024-04-26T11:31:59.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.167+0000 7f3566200640 1 -- 172.21.15.33:0/4069940937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 msgr2=0x7f356010aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.167+0000 7f3566200640 1 --2- 172.21.15.33:0/4069940937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 0x7f356010aff0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3550009930 tx=0x7f355002a420 comp rx=0 tx=0).stop 2024-04-26T11:31:59.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.168+0000 7f3566200640 1 -- 172.21.15.33:0/4069940937 shutdown_connections 2024-04-26T11:31:59.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.168+0000 7f3566200640 1 --2- 172.21.15.33:0/4069940937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560105510 0x7f35601119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.168+0000 7f3566200640 1 --2- 172.21.15.33:0/4069940937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560104b50 0x7f3560104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.168+0000 7f3566200640 1 --2- 172.21.15.33:0/4069940937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 0x7f356010aff0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.168+0000 7f3566200640 1 -- 172.21.15.33:0/4069940937 >> 172.21.15.33:0/4069940937 conn(0x7f3560100600 msgr2=0x7f3560102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.168+0000 7f3566200640 1 -- 172.21.15.33:0/4069940937 shutdown_connections 2024-04-26T11:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.168+0000 7f3566200640 1 -- 172.21.15.33:0/4069940937 wait complete. 2024-04-26T11:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.171+0000 7f3566200640 1 Processor -- start 2024-04-26T11:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.171+0000 7f3566200640 1 -- start start 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f3566200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560104b50 0x7f356019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f3566200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560105510 0x7f356019b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f3566200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 0x7f356019fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f3566200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3560114e10 con 0x7f3560105510 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f3566200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3560114c90 con 0x7f3560104b50 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f3566200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3560114f90 con 0x7f356010ac10 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560104b50 0x7f356019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560104b50 0x7f356019b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55570/0 (socket says 172.21.15.33:55570) 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 -- 172.21.15.33:0/1610235787 learned_addr learned my addr 172.21.15.33:0/1610235787 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355fe00640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 0x7f356019fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 -- 172.21.15.33:0/1610235787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 msgr2=0x7f356019fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f3557e00640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560105510 0x7f356019b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 0x7f356019fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 -- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560105510 msgr2=0x7f356019b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560105510 0x7f356019b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355f400640 1 -- 172.21.15.33:0/1610235787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35601a0150 con 0x7f3560104b50 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.172+0000 7f355fe00640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 0x7f356019fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.173+0000 7f3557e00640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560105510 0x7f356019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.173+0000 7f355f400640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560104b50 0x7f356019b1b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3550031440 tx=0x7f3550003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.174+0000 7f355cc00640 1 -- 172.21.15.33:0/1610235787 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355003f070 con 0x7f3560104b50 2024-04-26T11:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.174+0000 7f355cc00640 1 -- 172.21.15.33:0/1610235787 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3550031d60 con 0x7f3560104b50 2024-04-26T11:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.174+0000 7f355cc00640 1 -- 172.21.15.33:0/1610235787 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550035030 con 0x7f3560104b50 2024-04-26T11:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.174+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35601a03e0 con 0x7f3560104b50 2024-04-26T11:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.175+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35601a7cd0 con 0x7f3560104b50 2024-04-26T11:31:59.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.179+0000 7f355cc00640 1 -- 172.21.15.33:0/1610235787 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3550042070 con 0x7f3560104b50 2024-04-26T11:31:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.180+0000 7f356902a740 1 Processor -- start 2024-04-26T11:31:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f356902a740 1 -- start start 2024-04-26T11:31:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f356902a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f356902a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550bda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f356902a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550c03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f356902a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cc55018380 con 0x55cc550b2dd0 2024-04-26T11:31:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f356902a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cc54fb4c30 con 0x55cc550bdfb0 2024-04-26T11:31:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f356902a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc55014340 con 0x55cc550b36f0 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.181+0000 7f355f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550bda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550bda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50840/0 (socket says 172.21.15.33:50840) 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 -- 172.21.15.33:0/561843984 learned_addr learned my addr 172.21.15.33:0/561843984 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355fe00640 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f3557e00640 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550c03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355cc00640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f35440a3f20 0x7f35440a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 -- 172.21.15.33:0/561843984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 msgr2=0x55cc550c03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550c03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 -- 172.21.15.33:0/561843984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 msgr2=0x55cc550b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 -- 172.21.15.33:0/561843984 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc54fd6940 con 0x55cc550b36f0 2024-04-26T11:31:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f3557e00640 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550c03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:31:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355fe00640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f35440a3f20 0x7f35440a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.182+0000 7f355f400640 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550bda70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35500eebc0 tx=0x7f35500b9ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7fd9c53f651fccb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.183+0000 7f354fe00640 1 -- 172.21.15.33:0/561843984 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500bd540 con 0x55cc550b36f0 2024-04-26T11:31:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.183+0000 7f354fe00640 1 -- 172.21.15.33:0/561843984 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35500c4150 con 0x55cc550b36f0 2024-04-26T11:31:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.183+0000 7f354fe00640 1 -- 172.21.15.33:0/561843984 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500bc470 con 0x55cc550b36f0 2024-04-26T11:31:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.183+0000 7f355cc00640 1 -- 172.21.15.33:0/1610235787 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f35500f1580 con 0x7f3560104b50 2024-04-26T11:31:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.185+0000 7f356902a740 1 -- 172.21.15.33:0/561843984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 msgr2=0x55cc550bda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.185+0000 7f356902a740 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550bda70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35500eebc0 tx=0x7f35500b9ad0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.185+0000 7f355fe00640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f35440a3f20 0x7f35440a63e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3558005fd0 tx=0x7f3558004a40 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.186+0000 7f356902a740 1 -- 172.21.15.33:0/561843984 shutdown_connections 2024-04-26T11:31:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.186+0000 7f356902a740 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550bda70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.186+0000 7f356902a740 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550c03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.186+0000 7f356902a740 1 --2- 172.21.15.33:0/561843984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.186+0000 7f356902a740 1 -- 172.21.15.33:0/561843984 >> 172.21.15.33:0/561843984 conn(0x55cc550af2e0 msgr2=0x55cc550af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.186+0000 7f356902a740 1 -- 172.21.15.33:0/561843984 shutdown_connections 2024-04-26T11:31:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.186+0000 7f356902a740 1 -- 172.21.15.33:0/561843984 wait complete. 2024-04-26T11:31:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.187+0000 7f356902a740 1 -- start start 2024-04-26T11:31:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.187+0000 7f356902a740 1 Processor -- start 2024-04-26T11:31:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f356902a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550baf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f356902a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550bb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f356902a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550bb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f356902a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cc550c09a0 con 0x55cc550b36f0 2024-04-26T11:31:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f356902a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cc54f9fc60 con 0x55cc550bdfb0 2024-04-26T11:31:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f356902a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc550c0b20 con 0x55cc550b2dd0 2024-04-26T11:31:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f356902a740 10 client.0 _ll_register_callbacks cb 0x55cc54fcbe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f355f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550baf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f355f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550baf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50842/0 (socket says 172.21.15.33:50842) 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f355f400640 1 -- 172.21.15.33:0/2190495943 learned_addr learned my addr 172.21.15.33:0/2190495943 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.188+0000 7f3557e00640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550bb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355fe00640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550bb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355f400640 1 -- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 msgr2=0x55cc550bb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355f400640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550bb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355f400640 1 -- 172.21.15.33:0/2190495943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 msgr2=0x55cc550bb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355f400640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550bb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355f400640 1 -- 172.21.15.33:0/2190495943 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc54fae510 con 0x55cc550b2dd0 2024-04-26T11:31:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355fe00640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550bb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f3557e00640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550bb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:31:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f355f400640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550baf60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35500c3c50 tx=0x7f3550031bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfac1edbbb8ef584 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.189+0000 7f354cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:31:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.190+0000 7f354cc00640 1 -- 172.21.15.33:0/2190495943 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550049070 con 0x55cc550b2dd0 2024-04-26T11:31:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.190+0000 7f354cc00640 1 -- 172.21.15.33:0/2190495943 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f35500bba70 con 0x55cc550b2dd0 2024-04-26T11:31:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.190+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cc55025f90 con 0x55cc550b2dd0 2024-04-26T11:31:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.191+0000 7f354cc00640 1 -- 172.21.15.33:0/2190495943 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35500bd0d0 con 0x55cc550b2dd0 2024-04-26T11:31:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.191+0000 7f356902a740 10 client.25363 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.191+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc54d2a210 con 0x55cc550b2dd0 2024-04-26T11:31:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.191+0000 7f354cc00640 1 -- 172.21.15.33:0/2190495943 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cc55025f90 con 0x55cc550b2dd0 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.191+0000 7f354cc00640 1 -- 172.21.15.33:0/2190495943 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f35500b9de0 con 0x55cc550b2dd0 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.192+0000 7f356902a740 10 client.25363 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.192+0000 7f356902a740 10 client.25363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.192+0000 7f356902a740 20 client.25363 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.192+0000 7f356902a740 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc550cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.192+0000 7f356902a740 4 client.25363 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.192+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc5501fe00 con 0x55cc550cd920 2024-04-26T11:31:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.193+0000 7f3557e00640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc550cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.193+0000 7f3557e00640 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc550cfd10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93e6d3e05f2783a6 server_cookie=ddb12cd93f2db743 in_seq=0 out_seq=0 2024-04-26T11:31:59.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.193+0000 7f354cc00640 10 client.25363 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:59.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.195+0000 7f354cc00640 1 -- 172.21.15.33:0/2190495943 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc5501fe00 con 0x55cc550cd920 2024-04-26T11:31:59.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.195+0000 7f354cc00640 10 client.25363 handle_command_reply: tid=0 2024-04-26T11:31:59.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.195+0000 7f356902a740 1 client.25363 shutdown 2024-04-26T11:31:59.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.196+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 msgr2=0x55cc550baf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.196+0000 7f356902a740 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550baf60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35500c3c50 tx=0x7f3550031bd0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.198+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 shutdown_connections 2024-04-26T11:31:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.198+0000 7f356902a740 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc550cfd10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.198+0000 7f354cc00640 0 client.25363 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.199+0000 7f356902a740 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550bb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.199+0000 7f356902a740 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550bb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.199+0000 7f356902a740 1 --2- 172.21.15.33:0/2190495943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550baf60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.199+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 >> 172.21.15.33:0/2190495943 conn(0x55cc550af2e0 msgr2=0x55cc550b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.200+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 shutdown_connections 2024-04-26T11:31:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.200+0000 7f356902a740 1 -- 172.21.15.33:0/2190495943 wait complete. 2024-04-26T11:31:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.205+0000 7f356902a740 20 client.25363 trim_cache size 0 max 16384 2024-04-26T11:31:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 Processor -- start 2024-04-26T11:31:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 -- start start 2024-04-26T11:31:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550d5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550d5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b2e60 con 0x55cc550b2dd0 2024-04-26T11:31:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b0730 con 0x55cc550bdfb0 2024-04-26T11:31:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.240+0000 7f356902a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f35601ae540 con 0x55cc550b36f0 2024-04-26T11:31:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f3557e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550d5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f3557e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550d5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50848/0 (socket says 172.21.15.33:50848) 2024-04-26T11:31:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f3557e00640 1 -- 172.21.15.33:0/2719181375 learned_addr learned my addr 172.21.15.33:0/2719181375 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f355f400640 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f3557e00640 1 -- 172.21.15.33:0/2719181375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 msgr2=0x55cc550d5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f355fe00640 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550d5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f3557e00640 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550d5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.241+0000 7f3557e00640 1 -- 172.21.15.33:0/2719181375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 msgr2=0x55cc550c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f3557e00640 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f3557e00640 1 -- 172.21.15.33:0/2719181375 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc550b3520 con 0x55cc550b36f0 2024-04-26T11:31:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f355fe00640 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550d5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f3557e00640 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550d5580 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x55cc550b0700 tx=0x7f354800e780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63d4fd15d914d75c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f355f400640 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f354f400640 1 -- 172.21.15.33:0/2719181375 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3548004c60 con 0x55cc550b36f0 2024-04-26T11:31:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f354f400640 1 -- 172.21.15.33:0/2719181375 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3548004e00 con 0x55cc550b36f0 2024-04-26T11:31:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.244+0000 7f354f400640 1 -- 172.21.15.33:0/2719181375 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3548011a70 con 0x55cc550b36f0 2024-04-26T11:31:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.245+0000 7f356902a740 1 -- 172.21.15.33:0/2719181375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 msgr2=0x55cc550d5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.245+0000 7f356902a740 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550d5580 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x55cc550b0700 tx=0x7f354800e780 comp rx=0 tx=0).stop 2024-04-26T11:31:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 -- 172.21.15.33:0/2719181375 shutdown_connections 2024-04-26T11:31:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b36f0 0x55cc550d5580 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550d5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 --2- 172.21.15.33:0/2719181375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b2dd0 0x55cc550c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 -- 172.21.15.33:0/2719181375 >> 172.21.15.33:0/2719181375 conn(0x55cc550af2e0 msgr2=0x55cc550b6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 -- 172.21.15.33:0/2719181375 shutdown_connections 2024-04-26T11:31:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 -- 172.21.15.33:0/2719181375 wait complete. 2024-04-26T11:31:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.246+0000 7f356902a740 1 -- start start 2024-04-26T11:31:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.247+0000 7f356902a740 1 Processor -- start 2024-04-26T11:31:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.247+0000 7f356902a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f356902a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f356902a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550b7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f356902a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35601aebf0 con 0x55cc550b36f0 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f356902a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b0060 con 0x55cc550bdfb0 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f356902a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f35601af2c0 con 0x55cc550b2dd0 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f356902a740 10 client.0 _ll_register_callbacks cb 0x7f35500f9300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f355f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f355fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550b7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f3557e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f355f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550c5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50864/0 (socket says 172.21.15.33:50864) 2024-04-26T11:31:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.248+0000 7f355f400640 1 -- 172.21.15.33:0/1924885383 learned_addr learned my addr 172.21.15.33:0/1924885383 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:31:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f355f400640 1 -- 172.21.15.33:0/1924885383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 msgr2=0x55cc550b7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f355f400640 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550b7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f355f400640 1 -- 172.21.15.33:0/1924885383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 msgr2=0x55cc550c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:31:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f355f400640 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f355f400640 1 -- 172.21.15.33:0/1924885383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc54df6620 con 0x55cc550b2dd0 2024-04-26T11:31:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f3557e00640 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:31:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f355f400640 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550c5ce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f355002a9e0 tx=0x7f3550005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17035ae8de7ddb9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:31:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.249+0000 7f354cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:31:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.250+0000 7f354cc00640 1 -- 172.21.15.33:0/1924885383 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500bd0d0 con 0x55cc550b2dd0 2024-04-26T11:31:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.252+0000 7f354cc00640 1 -- 172.21.15.33:0/1924885383 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f35500b9de0 con 0x55cc550b2dd0 2024-04-26T11:31:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.252+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3548004e00 con 0x55cc550b2dd0 2024-04-26T11:31:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.252+0000 7f354cc00640 1 -- 172.21.15.33:0/1924885383 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35500bd2b0 con 0x55cc550b2dd0 2024-04-26T11:31:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.253+0000 7f356902a740 10 client.25369 fetch_fsmap learned FSMap version 28 2024-04-26T11:31:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.253+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc54fd2cb0 con 0x55cc550b2dd0 2024-04-26T11:31:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.253+0000 7f354cc00640 1 -- 172.21.15.33:0/1924885383 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3548004e00 con 0x55cc550b2dd0 2024-04-26T11:31:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.253+0000 7f354cc00640 1 -- 172.21.15.33:0/1924885383 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f355003b610 con 0x55cc550b2dd0 2024-04-26T11:31:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.253+0000 7f356902a740 10 client.25369 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:31:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.254+0000 7f356902a740 10 client.25369 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:31:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.254+0000 7f356902a740 20 client.25369 populate_metadata read hostname 'smithi033' 2024-04-26T11:31:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.254+0000 7f356902a740 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc5513e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:31:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.254+0000 7f356902a740 4 client.25369 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:31:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.254+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc54f9b4b0 con 0x55cc550cd920 2024-04-26T11:31:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.254+0000 7f3557e00640 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc5513e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:31:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.255+0000 7f3557e00640 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc5513e010 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b12da876946a18af server_cookie=9fe506e9eac02d32 in_seq=0 out_seq=0 2024-04-26T11:31:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.255+0000 7f354cc00640 10 client.25369 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.268+0000 7f354cc00640 1 -- 172.21.15.33:0/1924885383 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc54f9b4b0 con 0x55cc550cd920 2024-04-26T11:31:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.268+0000 7f354cc00640 10 client.25369 handle_command_reply: tid=0 2024-04-26T11:31:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.268+0000 7f356902a740 1 client.25369 shutdown 2024-04-26T11:31:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 msgr2=0x55cc550c5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550c5ce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f355002a9e0 tx=0x7f3550005b70 comp rx=0 tx=0).stop 2024-04-26T11:31:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 shutdown_connections 2024-04-26T11:31:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55cc550cd920 0x55cc5513e010 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cc550bdfb0 0x55cc550b7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cc550b36f0 0x55cc550c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 --2- 172.21.15.33:0/1924885383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc550b2dd0 0x55cc550c5ce0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f354cc00640 0 client.25369 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 >> 172.21.15.33:0/1924885383 conn(0x55cc550af2e0 msgr2=0x55cc550bc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.269+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 shutdown_connections 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.270+0000 7f356902a740 1 -- 172.21.15.33:0/1924885383 wait complete. 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.270+0000 7f356902a740 20 client.25369 trim_cache size 0 max 16384 2024-04-26T11:31:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:31:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:31:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:31:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.271+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f35440a3f20 msgr2=0x7f35440a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.271+0000 7f3566200640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f35440a3f20 0x7f35440a63e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3558005fd0 tx=0x7f3558004a40 comp rx=0 tx=0).stop 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.271+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560104b50 msgr2=0x7f356019b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.271+0000 7f3566200640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560104b50 0x7f356019b1b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3550031440 tx=0x7f3550003d50 comp rx=0 tx=0).stop 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 shutdown_connections 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f35440a3f20 0x7f35440a63e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356010ac10 0x7f356019fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3560105510 0x7f356019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 --2- 172.21.15.33:0/1610235787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3560104b50 0x7f356019b1b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 >> 172.21.15.33:0/1610235787 conn(0x7f3560100600 msgr2=0x7f3560101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:31:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 shutdown_connections 2024-04-26T11:31:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:31:59.272+0000 7f3566200640 1 -- 172.21.15.33:0/1610235787 wait complete. 2024-04-26T11:31:59.736 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5514a944', seconds elapsed: 35.4 2024-04-26T11:31:59.737 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.0 2024-04-26T11:31:59.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:32:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:32:00.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:00 smithi033 ceph-mon[27845]: pgmap v51: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.4 MiB/s wr, 92 op/s 2024-04-26T11:32:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:00 smithi194 ceph-mon[33017]: pgmap v51: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.4 MiB/s wr, 92 op/s 2024-04-26T11:32:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:00 smithi096 ceph-mon[32753]: pgmap v51: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.4 MiB/s wr, 92 op/s 2024-04-26T11:32:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 45.0 s, 809.8 tps, lat 740.778 ms stddev 4149.726 2024-04-26T11:32:02.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:02 smithi033 ceph-mon[27845]: pgmap v52: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-04-26T11:32:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:02 smithi194 ceph-mon[33017]: pgmap v52: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-04-26T11:32:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:02 smithi096 ceph-mon[32753]: pgmap v52: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-04-26T11:32:02.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.942+0000 7f4e7b000700 1 -- 172.21.15.33:0/3580085195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406b130 msgr2=0x7f4e7406b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:02.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.942+0000 7f4e7b000700 1 --2- 172.21.15.33:0/3580085195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406b130 0x7f4e7406b510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009990 tx=0x7f4e64009ca0 comp rx=0 tx=0).stop 2024-04-26T11:32:02.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.942+0000 7f4e7b000700 1 -- 172.21.15.33:0/3580085195 shutdown_connections 2024-04-26T11:32:02.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.943+0000 7f4e7b000700 1 --2- 172.21.15.33:0/3580085195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 0x7f4e741122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:02.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.943+0000 7f4e7b000700 1 --2- 172.21.15.33:0/3580085195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406b130 0x7f4e7406b510 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:02.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.943+0000 7f4e7b000700 1 --2- 172.21.15.33:0/3580085195 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7406ba50 0x7f4e7410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:02.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.943+0000 7f4e7b000700 1 -- 172.21.15.33:0/3580085195 >> 172.21.15.33:0/3580085195 conn(0x7f4e740792d0 msgr2=0x7f4e740796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:02.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.943+0000 7f4e7b000700 1 -- 172.21.15.33:0/3580085195 shutdown_connections 2024-04-26T11:32:02.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.944+0000 7f4e7b000700 1 -- 172.21.15.33:0/3580085195 wait complete. 2024-04-26T11:32:02.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.945+0000 7f4e7b000700 1 Processor -- start 2024-04-26T11:32:02.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.945+0000 7f4e7b000700 1 -- start start 2024-04-26T11:32:02.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.946+0000 7f4e7b000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7406b130 0x7f4e7419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:02.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.946+0000 7f4e7b000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406ba50 0x7f4e7419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:02.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.946+0000 7f4e7b000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 0x7f4e741a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:02.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.946+0000 7f4e7b000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e74115830 con 0x7f4e7406b130 2024-04-26T11:32:02.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.946+0000 7f4e7b000700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741156b0 con 0x7f4e7410fed0 2024-04-26T11:32:02.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.946+0000 7f4e7b000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741159b0 con 0x7f4e7406ba50 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.946+0000 7f4e79000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 0x7f4e741a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e79000700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 0x7f4e741a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58474/0 (socket says 172.21.15.33:58474) 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e79000700 1 -- 172.21.15.33:0/949306814 learned_addr learned my addr 172.21.15.33:0/949306814 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e73e00700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7406b130 0x7f4e7419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e73400700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406ba50 0x7f4e7419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e79000700 1 -- 172.21.15.33:0/949306814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406ba50 msgr2=0x7f4e7419c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e79000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406ba50 0x7f4e7419c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e79000700 1 -- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7406b130 msgr2=0x7f4e7419bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e79000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7406b130 0x7f4e7419bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e79000700 1 -- 172.21.15.33:0/949306814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e741a0d50 con 0x7f4e7410fed0 2024-04-26T11:32:02.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.947+0000 7f4e73400700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406ba50 0x7f4e7419c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:02.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.948+0000 7f4e79000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 0x7f4e741a0610 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c00c110 tx=0x7f4e6c00c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:02.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.948+0000 7f4e70c00700 1 -- 172.21.15.33:0/949306814 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0092e0 con 0x7f4e7410fed0 2024-04-26T11:32:02.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.948+0000 7f4e70c00700 1 -- 172.21.15.33:0/949306814 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e6c003a00 con 0x7f4e7410fed0 2024-04-26T11:32:02.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.948+0000 7f4e70c00700 1 -- 172.21.15.33:0/949306814 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c004650 con 0x7f4e7410fed0 2024-04-26T11:32:02.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.948+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e741a0fe0 con 0x7f4e7410fed0 2024-04-26T11:32:02.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.949+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e741a88c0 con 0x7f4e7410fed0 2024-04-26T11:32:02.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.950+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7406a110 con 0x7f4e7410fed0 2024-04-26T11:32:02.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.953+0000 7f4e70c00700 1 -- 172.21.15.33:0/949306814 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4e6c01a020 con 0x7f4e7410fed0 2024-04-26T11:32:02.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.954+0000 7f4e70c00700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4e580a3f00 0x7f4e580a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:02.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.954+0000 7f4e70c00700 1 -- 172.21.15.33:0/949306814 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4e6c0cfac0 con 0x7f4e7410fed0 2024-04-26T11:32:02.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.955+0000 7f4e73e00700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4e580a3f00 0x7f4e580a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:02.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.956+0000 7f4e73e00700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4e580a3f00 0x7f4e580a63c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4e64006010 tx=0x7f4e6400b260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:02.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:02.959+0000 7f4e70c00700 1 -- 172.21.15.33:0/949306814 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e6c010040 con 0x7f4e7410fed0 2024-04-26T11:32:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.326+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e7402e490 con 0x7f4e7410fed0 2024-04-26T11:32:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.327+0000 7f4e70c00700 1 -- 172.21.15.33:0/949306814 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11868 (secure 0 0 0) 0x7f4e6c003eb0 con 0x7f4e7410fed0 2024-04-26T11:32:03.328 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:32:03.330 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:31:11.426188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:32:03.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.332+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4e580a3f00 msgr2=0x7f4e580a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.332+0000 7f4e7b000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4e580a3f00 0x7f4e580a63c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4e64006010 tx=0x7f4e6400b260 comp rx=0 tx=0).stop 2024-04-26T11:32:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.332+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 msgr2=0x7f4e741a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.332+0000 7f4e7b000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 0x7f4e741a0610 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c00c110 tx=0x7f4e6c00c420 comp rx=0 tx=0).stop 2024-04-26T11:32:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.333+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 shutdown_connections 2024-04-26T11:32:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.333+0000 7f4e7b000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4e580a3f00 0x7f4e580a63c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.333+0000 7f4e7b000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e7410fed0 0x7f4e741a0610 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.333+0000 7f4e7b000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e7406b130 0x7f4e7419bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.333+0000 7f4e7b000700 1 --2- 172.21.15.33:0/949306814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e7406ba50 0x7f4e7419c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.333+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 >> 172.21.15.33:0/949306814 conn(0x7f4e740792d0 msgr2=0x7f4e74111380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.333+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 shutdown_connections 2024-04-26T11:32:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:03.334+0000 7f4e7b000700 1 -- 172.21.15.33:0/949306814 wait complete. 2024-04-26T11:32:03.336 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-04-26T11:32:04.230 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24527 quiesce db / --timeout 60 --expiration 78.0 --await-for 42.0' 2024-04-26T11:32:04.230 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24527 quiesce db / --timeout 60 --expiration 78.0 --await-for 42.0 2024-04-26T11:32:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:04 smithi033 ceph-mon[27845]: pgmap v53: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-04-26T11:32:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:04 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/949306814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:32:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:04 smithi194 ceph-mon[33017]: pgmap v53: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-04-26T11:32:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:04 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/949306814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:32:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:04 smithi096 ceph-mon[32753]: pgmap v53: 97 pgs: 97 active+clean; 950 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-04-26T11:32:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:04 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/949306814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:32:04.756 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:32:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:06 smithi194 ceph-mon[33017]: pgmap v54: 97 pgs: 97 active+clean; 954 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 364 op/s 2024-04-26T11:32:06.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:06 smithi033 ceph-mon[27845]: pgmap v54: 97 pgs: 97 active+clean; 954 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 364 op/s 2024-04-26T11:32:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:06 smithi096 ceph-mon[32753]: pgmap v54: 97 pgs: 97 active+clean; 954 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 364 op/s 2024-04-26T11:32:07.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 50.0 s, 262.7 tps, lat 316.079 ms stddev 204.077 2024-04-26T11:32:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.628+0000 7f5fad400700 1 -- 172.21.15.33:0/3769577519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa810d890 msgr2=0x7f5fa810fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.628+0000 7f5fad400700 1 --2- 172.21.15.33:0/3769577519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa810d890 0x7f5fa810fd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009f90 tx=0x7f5fa000b330 comp rx=0 tx=0).stop 2024-04-26T11:32:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.628+0000 7f5fad400700 1 -- 172.21.15.33:0/3769577519 shutdown_connections 2024-04-26T11:32:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.628+0000 7f5fad400700 1 --2- 172.21.15.33:0/3769577519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa810d890 0x7f5fa810fd10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.628+0000 7f5fad400700 1 --2- 172.21.15.33:0/3769577519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 0x7f5fa8104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.628+0000 7f5fad400700 1 --2- 172.21.15.33:0/3769577519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa8104dd0 0x7f5fa810d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.628+0000 7f5fad400700 1 -- 172.21.15.33:0/3769577519 >> 172.21.15.33:0/3769577519 conn(0x7f5fa80fdc60 msgr2=0x7f5fa8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.632+0000 7f5fad400700 1 -- 172.21.15.33:0/3769577519 shutdown_connections 2024-04-26T11:32:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.634+0000 7f5fad400700 1 -- 172.21.15.33:0/3769577519 wait complete. 2024-04-26T11:32:07.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.636+0000 7f5fad400700 1 Processor -- start 2024-04-26T11:32:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.638+0000 7f5fad400700 1 -- start start 2024-04-26T11:32:07.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.639+0000 7f5fad400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 0x7f5fa8074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fad400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa8104dd0 0x7f5fa8075040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fa7e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 0x7f5fa8074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fa7e00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 0x7f5fa8074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58490/0 (socket says 172.21.15.33:58490) 2024-04-26T11:32:07.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fad400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa810d890 0x7f5fa806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fad400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81134b0 con 0x7f5fa810d890 2024-04-26T11:32:07.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fad400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8113330 con 0x7f5fa81044b0 2024-04-26T11:32:07.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fad400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8113630 con 0x7f5fa8104dd0 2024-04-26T11:32:07.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.640+0000 7f5fa7e00700 1 -- 172.21.15.33:0/3256300375 learned_addr learned my addr 172.21.15.33:0/3256300375 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:07.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.641+0000 7f5fa7400700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa8104dd0 0x7f5fa8075040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.641+0000 7f5faca00700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa810d890 0x7f5fa806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.642+0000 7f5fa7e00700 1 -- 172.21.15.33:0/3256300375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa8104dd0 msgr2=0x7f5fa8075040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.642+0000 7f5fa7e00700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa8104dd0 0x7f5fa8075040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.642+0000 7f5fa7e00700 1 -- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa810d890 msgr2=0x7f5fa806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.642+0000 7f5fa7e00700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa810d890 0x7f5fa806dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.642+0000 7f5fa7e00700 1 -- 172.21.15.33:0/3256300375 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa806e5d0 con 0x7f5fa81044b0 2024-04-26T11:32:07.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.643+0000 7f5fa7e00700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 0x7f5fa8074b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800e9f0 tx=0x7f5f9800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.644+0000 7f5fa4c00700 1 -- 172.21.15.33:0/3256300375 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98004a30 con 0x7f5fa81044b0 2024-04-26T11:32:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.644+0000 7f5fa4c00700 1 -- 172.21.15.33:0/3256300375 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f9800cc70 con 0x7f5fa81044b0 2024-04-26T11:32:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.644+0000 7f5fa4c00700 1 -- 172.21.15.33:0/3256300375 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98005600 con 0x7f5fa81044b0 2024-04-26T11:32:07.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.645+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa806e8c0 con 0x7f5fa81044b0 2024-04-26T11:32:07.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.645+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa80728b0 con 0x7f5fa81044b0 2024-04-26T11:32:07.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.654+0000 7f5fb93f5b80 1 Processor -- start 2024-04-26T11:32:07.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.655+0000 7f5fb93f5b80 1 -- start start 2024-04-26T11:32:07.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.656+0000 7f5fa4c00700 1 -- 172.21.15.33:0/3256300375 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5f98004c60 con 0x7f5fa81044b0 2024-04-26T11:32:07.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.656+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b9d70 0x564def6b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.656+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.656+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6c8d30 0x564def6cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.656+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564def608a20 con 0x564def6b6610 2024-04-26T11:32:07.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.656+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564def5a7340 con 0x564def6c8d30 2024-04-26T11:32:07.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.656+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564def6049e0 con 0x564def6b9d70 2024-04-26T11:32:07.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.657+0000 7f5faca00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b9d70 0x564def6b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.657+0000 7f5faca00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b9d70 0x564def6b60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:49090/0 (socket says 172.21.15.33:49090) 2024-04-26T11:32:07.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.657+0000 7f5faca00700 1 -- 172.21.15.33:0/1268067790 learned_addr learned my addr 172.21.15.33:0/1268067790 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:07.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.657+0000 7f5fa7e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.657+0000 7f5fa7400700 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6c8d30 0x564def6cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.658+0000 7f5faca00700 1 -- 172.21.15.33:0/1268067790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6c8d30 msgr2=0x564def6cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:07.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.658+0000 7f5faca00700 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6c8d30 0x564def6cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.658+0000 7f5faca00700 1 -- 172.21.15.33:0/1268067790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 msgr2=0x564def6b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:07.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.658+0000 7f5faca00700 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.658+0000 7f5faca00700 1 -- 172.21.15.33:0/1268067790 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564def5af7c0 con 0x564def6b9d70 2024-04-26T11:32:07.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.659+0000 7f5faca00700 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b9d70 0x564def6b60d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00093c0 tx=0x7f5fa00093f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44fd7371a357c0ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:07.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.659+0000 7f5f9d600700 1 -- 172.21.15.33:0/1268067790 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0020070 con 0x564def6b9d70 2024-04-26T11:32:07.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.659+0000 7f5f9d600700 1 -- 172.21.15.33:0/1268067790 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa0003dc0 con 0x564def6b9d70 2024-04-26T11:32:07.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.659+0000 7f5f9d600700 1 -- 172.21.15.33:0/1268067790 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0007c60 con 0x564def6b9d70 2024-04-26T11:32:07.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.659+0000 7f5fa4c00700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5f8c0a3fb0 0x7f5f8c0a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.660+0000 7f5fa4c00700 1 -- 172.21.15.33:0/3256300375 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5f980d0920 con 0x7f5fa81044b0 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.660+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1268067790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b9d70 msgr2=0x564def6b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.660+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b9d70 0x564def6b60d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00093c0 tx=0x7f5fa00093f0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.660+0000 7f5fa7e00700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5f8c0a3fb0 0x7f5f8c0a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.661+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1268067790 shutdown_connections 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.661+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6c8d30 0x564def6cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.661+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b9d70 0x564def6b60d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.661+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1268067790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.661+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1268067790 >> 172.21.15.33:0/1268067790 conn(0x564def5e3480 msgr2=0x564def6b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.661+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1268067790 shutdown_connections 2024-04-26T11:32:07.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.661+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1268067790 wait complete. 2024-04-26T11:32:07.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.662+0000 7f5fb93f5b80 1 -- start start 2024-04-26T11:32:07.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.662+0000 7f5fb93f5b80 1 Processor -- start 2024-04-26T11:32:07.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fa7e00700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5f8c0a3fb0 0x7f5f8c0a6470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f980ce130 tx=0x7f5f980d4680 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:07.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6c8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 0x564def6c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564def6cb7b0 con 0x564def6b6610 2024-04-26T11:32:07.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564def5a3590 con 0x564def6b9d70 2024-04-26T11:32:07.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564def6cb930 con 0x564def6c8d30 2024-04-26T11:32:07.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fb93f5b80 10 client.0 _ll_register_callbacks cb 0x564def59c510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:07.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5faca00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.663+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58500/0 (socket says 172.21.15.33:58500) 2024-04-26T11:32:07.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5fa7400700 1 -- 172.21.15.33:0/1013699280 learned_addr learned my addr 172.21.15.33:0/1013699280 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:07.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5faca00700 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 0x564def6c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5fa7400700 1 -- 172.21.15.33:0/1013699280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 msgr2=0x564def6c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5fa7400700 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 0x564def6c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5fa7400700 1 -- 172.21.15.33:0/1013699280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 msgr2=0x564def6c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5fa7400700 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5fa7400700 1 -- 172.21.15.33:0/1013699280 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564def555a60 con 0x564def6b9d70 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5faca00700 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5fa7400700 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6c5810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400df20 tx=0x7f5f94012730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aafd7836a1cb5fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.664+0000 7f5f92000700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.665+0000 7f5f92000700 1 -- 172.21.15.33:0/1013699280 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94012e60 con 0x564def6b9d70 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.665+0000 7f5f92000700 1 -- 172.21.15.33:0/1013699280 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5f94003680 con 0x564def6b9d70 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.665+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa0003dc0 con 0x564def6b9d70 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.665+0000 7f5f92000700 1 -- 172.21.15.33:0/1013699280 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f94004500 con 0x564def6b9d70 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.665+0000 7f5fb93f5b80 10 client.25418 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:07.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.665+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564def6d6130 con 0x564def6b9d70 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.665+0000 7f5f92000700 1 -- 172.21.15.33:0/1013699280 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa0003dc0 con 0x564def6b9d70 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5f92000700 1 -- 172.21.15.33:0/1013699280 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f5f94019020 con 0x564def6b9d70 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5fb93f5b80 10 client.25418 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5fb93f5b80 10 client.25418 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5fb93f5b80 20 client.25418 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def6d86c0 0x564def6daab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5fb93f5b80 4 client.25418 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564def616400 con 0x564def6d86c0 2024-04-26T11:32:07.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.666+0000 7f5faca00700 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def6d86c0 0x564def6daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.667+0000 7f5faca00700 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def6d86c0 0x564def6daab0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=901a0cfe7c305c6c server_cookie=a66227da52c6cadf in_seq=0 out_seq=0 2024-04-26T11:32:07.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.667+0000 7f5f92000700 10 client.25418 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:07.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.667+0000 7f5f92000700 1 -- 172.21.15.33:0/1013699280 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564def616400 con 0x564def6d86c0 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.667+0000 7f5f92000700 10 client.25418 handle_command_reply: tid=0 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.667+0000 7f5fb93f5b80 1 client.25418 shutdown 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 msgr2=0x564def6c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6c5810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400df20 tx=0x7f5f94012730 comp rx=0 tx=0).stop 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 shutdown_connections 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def6d86c0 0x564def6daab0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 0x564def6c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/1013699280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6c5810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 >> 172.21.15.33:0/1013699280 conn(0x564def5e3480 msgr2=0x564def6ab500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5f92000700 0 client.25418 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 shutdown_connections 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.668+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/1013699280 wait complete. 2024-04-26T11:32:07.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.669+0000 7f5fb93f5b80 20 client.25418 trim_cache size 0 max 16384 2024-04-26T11:32:07.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.717+0000 7f5fb93f5b80 1 Processor -- start 2024-04-26T11:32:07.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.717+0000 7f5fb93f5b80 1 -- start start 2024-04-26T11:32:07.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.717+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b6610 0x564def6c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.717+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6c8d30 0x564def6e0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81b6850 con 0x564def6c8d30 2024-04-26T11:32:07.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81b40e0 con 0x564def6b9d70 2024-04-26T11:32:07.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81b1f50 con 0x564def6b6610 2024-04-26T11:32:07.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58508/0 (socket says 172.21.15.33:58508) 2024-04-26T11:32:07.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 -- 172.21.15.33:0/2829670226 learned_addr learned my addr 172.21.15.33:0/2829670226 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5faca00700 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b6610 0x564def6c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5faca00700 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6c8d30 0x564def6e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 -- 172.21.15.33:0/2829670226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b6610 msgr2=0x564def6c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b6610 0x564def6c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 -- 172.21.15.33:0/2829670226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6c8d30 msgr2=0x564def6e0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6c8d30 0x564def6e0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.718+0000 7f5fa7400700 1 -- 172.21.15.33:0/2829670226 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564def6c55e0 con 0x564def6b9d70 2024-04-26T11:32:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.719+0000 7f5fa7400700 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d4670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5f940046c0 tx=0x7f5f9400f3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb37591f109e00e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:07.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.720+0000 7f5f9cc00700 1 -- 172.21.15.33:0/2829670226 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94012e60 con 0x564def6b9d70 2024-04-26T11:32:07.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.720+0000 7f5f9cc00700 1 -- 172.21.15.33:0/2829670226 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f94005030 con 0x564def6b9d70 2024-04-26T11:32:07.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.720+0000 7f5f9cc00700 1 -- 172.21.15.33:0/2829670226 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94004970 con 0x564def6b9d70 2024-04-26T11:32:07.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.721+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/2829670226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 msgr2=0x564def6d4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.721+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d4670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5f940046c0 tx=0x7f5f9400f3f0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.721+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/2829670226 shutdown_connections 2024-04-26T11:32:07.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.722+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6c8d30 0x564def6e0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.722+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6b6610 0x564def6c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.722+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/2829670226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d4670 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.722+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/2829670226 >> 172.21.15.33:0/2829670226 conn(0x564def5e3480 msgr2=0x564def6c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.724+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/2829670226 shutdown_connections 2024-04-26T11:32:07.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.725+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/2829670226 wait complete. 2024-04-26T11:32:07.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.725+0000 7f5fb93f5b80 1 -- start start 2024-04-26T11:32:07.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.726+0000 7f5fb93f5b80 1 Processor -- start 2024-04-26T11:32:07.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.727+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.727+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.728+0000 7f5fb93f5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 0x564def6cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.728+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81b25a0 con 0x564def6b6610 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.728+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81b3a10 con 0x564def6b9d70 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.728+0000 7f5fb93f5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81b2c70 con 0x564def6c8d30 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.728+0000 7f5fb93f5b80 10 client.0 _ll_register_callbacks cb 0x564def59c510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.728+0000 7f5faca00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.728+0000 7f5faca00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6d0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60706/0 (socket says 172.21.15.33:60706) 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.729+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.729+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58512/0 (socket says 172.21.15.33:58512) 2024-04-26T11:32:07.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.729+0000 7f5fa7400700 1 -- 172.21.15.33:0/668322350 learned_addr learned my addr 172.21.15.33:0/668322350 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:07.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.729+0000 7f5fa7400700 1 -- 172.21.15.33:0/668322350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 msgr2=0x564def6cdb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T11:32:07.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.729+0000 7f5fa7400700 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 0x564def6cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.729+0000 7f5fa7400700 1 -- 172.21.15.33:0/668322350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 msgr2=0x564def6d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:07.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.730+0000 7f5fa7400700 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.730+0000 7f5fa7400700 1 -- 172.21.15.33:0/668322350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564def6ce370 con 0x564def6b9d70 2024-04-26T11:32:07.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.731+0000 7f5fa7400700 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d10c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f9400d630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=520c3a707971d0c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:07.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.731+0000 7f5f92000700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:32:07.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.732+0000 7f5f92000700 1 -- 172.21.15.33:0/668322350 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f940051e0 con 0x564def6b9d70 2024-04-26T11:32:07.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.734+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f94005030 con 0x564def6b9d70 2024-04-26T11:32:07.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.734+0000 7f5f92000700 1 -- 172.21.15.33:0/668322350 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5f94005380 con 0x564def6b9d70 2024-04-26T11:32:07.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.734+0000 7f5f92000700 1 -- 172.21.15.33:0/668322350 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f94005580 con 0x564def6b9d70 2024-04-26T11:32:07.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.734+0000 7f5fb93f5b80 10 client.25424 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:07.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.734+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564def73f670 con 0x564def6b9d70 2024-04-26T11:32:07.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.735+0000 7f5f92000700 1 -- 172.21.15.33:0/668322350 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f94005030 con 0x564def6b9d70 2024-04-26T11:32:07.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.735+0000 7f5f92000700 1 -- 172.21.15.33:0/668322350 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f5f94019070 con 0x564def6b9d70 2024-04-26T11:32:07.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.736+0000 7f5fb93f5b80 10 client.25424 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:07.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.736+0000 7f5fb93f5b80 10 client.25424 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:32:07.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.736+0000 7f5fb93f5b80 20 client.25424 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:07.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.736+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def73fd20 0x564def742140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:07.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.736+0000 7f5fb93f5b80 4 client.25424 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.0, "await_for": 42.0}] 2024-04-26T11:32:07.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.736+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.0, "await_for": 42.0}) -- 0x564def6104a0 con 0x564def73fd20 2024-04-26T11:32:07.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.736+0000 7f5faca00700 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def73fd20 0x564def742140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:07.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.737+0000 7f5faca00700 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def73fd20 0x564def742140 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65fe0cea7c0d6c26 server_cookie=f9675210f6e03de0 in_seq=0 out_seq=0 2024-04-26T11:32:07.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.737+0000 7f5f92000700 10 client.25424 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.947+0000 7f5f92000700 1 -- 172.21.15.33:0/668322350 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x564def6104a0 con 0x564def73fd20 2024-04-26T11:32:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.947+0000 7f5f92000700 10 client.25424 handle_command_reply: tid=0 2024-04-26T11:32:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.947+0000 7f5fb93f5b80 1 client.25424 shutdown 2024-04-26T11:32:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 msgr2=0x564def6d10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d10c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f9400d630 comp rx=0 tx=0).stop 2024-04-26T11:32:07.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 shutdown_connections 2024-04-26T11:32:07.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564def73fd20 0x564def742140 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564def6c8d30 0x564def6cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564def6b6610 0x564def6d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 --2- 172.21.15.33:0/668322350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564def6b9d70 0x564def6d10c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5f92000700 0 client.25424 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.948+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 >> 172.21.15.33:0/668322350 conn(0x564def5e3480 msgr2=0x564def6aac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.950+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 shutdown_connections 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.950+0000 7f5fb93f5b80 1 -- 172.21.15.33:0/668322350 wait complete. 2024-04-26T11:32:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.950+0000 7f5fb93f5b80 20 client.25424 trim_cache size 0 max 16384 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24527, 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 6, 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "ab712af7": { 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 6, 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-26T11:32:07.953 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 60.0, 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 78.0, 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "QUIESCED", 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:07.954 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:32:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.957+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5f8c0a3fb0 msgr2=0x7f5f8c0a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.957+0000 7f5fad400700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5f8c0a3fb0 0x7f5f8c0a6470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f980ce130 tx=0x7f5f980d4680 comp rx=0 tx=0).stop 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.957+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 msgr2=0x7f5fa8074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.957+0000 7f5fad400700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 0x7f5fa8074b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800e9f0 tx=0x7f5f9800ed00 comp rx=0 tx=0).stop 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.958+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 shutdown_connections 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.958+0000 7f5fad400700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f5f8c0a3fb0 0x7f5f8c0a6470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.958+0000 7f5fad400700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5fa810d890 0x7f5fa806dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.958+0000 7f5fad400700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa81044b0 0x7f5fa8074b00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.958+0000 7f5fad400700 1 --2- 172.21.15.33:0/3256300375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5fa8104dd0 0x7f5fa8075040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:07.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.958+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 >> 172.21.15.33:0/3256300375 conn(0x7f5fa80fdc60 msgr2=0x7f5fa80fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.958+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 shutdown_connections 2024-04-26T11:32:07.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:07.959+0000 7f5fad400700 1 -- 172.21.15.33:0/3256300375 wait complete. 2024-04-26T11:32:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:08 smithi194 ceph-mon[33017]: pgmap v55: 97 pgs: 97 active+clean; 954 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 358 op/s 2024-04-26T11:32:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:08 smithi033 ceph-mon[27845]: pgmap v55: 97 pgs: 97 active+clean; 954 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 358 op/s 2024-04-26T11:32:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:08 smithi096 ceph-mon[32753]: pgmap v55: 97 pgs: 97 active+clean; 954 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 358 op/s 2024-04-26T11:32:08.857 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ab712af7', quiesce took 9.1 seconds. Will release after: 18.0 2024-04-26T11:32:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:10 smithi194 ceph-mon[33017]: pgmap v56: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 35 MiB/s wr, 1.07k op/s 2024-04-26T11:32:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:10 smithi033 ceph-mon[27845]: pgmap v56: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 35 MiB/s wr, 1.07k op/s 2024-04-26T11:32:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:10 smithi096 ceph-mon[32753]: pgmap v56: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 35 MiB/s wr, 1.07k op/s 2024-04-26T11:32:12.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 55.0 s, 16.2 tps, lat 418.910 ms stddev 285.418 2024-04-26T11:32:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.428+0000 7f2844a00640 1 -- 172.21.15.33:0/3387767474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840077780 msgr2=0x7f28401130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.428+0000 7f2844a00640 1 --2- 172.21.15.33:0/3387767474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840077780 0x7f28401130c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f282c009930 tx=0x7f282c02a440 comp rx=0 tx=0).stop 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.428+0000 7f2844a00640 1 -- 172.21.15.33:0/3387767474 shutdown_connections 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.428+0000 7f2844a00640 1 --2- 172.21.15.33:0/3387767474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2840113600 0x7f28401159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.429+0000 7f2844a00640 1 --2- 172.21.15.33:0/3387767474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840077780 0x7f28401130c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.429+0000 7f2844a00640 1 --2- 172.21.15.33:0/3387767474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840076e60 0x7f2840077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.429+0000 7f2844a00640 1 -- 172.21.15.33:0/3387767474 >> 172.21.15.33:0/3387767474 conn(0x7f2840100410 msgr2=0x7f2840102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.429+0000 7f2844a00640 1 -- 172.21.15.33:0/3387767474 shutdown_connections 2024-04-26T11:32:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.429+0000 7f2844a00640 1 -- 172.21.15.33:0/3387767474 wait complete. 2024-04-26T11:32:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.430+0000 7f2844a00640 1 Processor -- start 2024-04-26T11:32:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.430+0000 7f2844a00640 1 -- start start 2024-04-26T11:32:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f2844a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2840076e60 0x7f284019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f2844a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840077780 0x7f284019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f2844a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840113600 0x7f28401a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f2844a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2840119010 con 0x7f2840077780 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f2844a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2840118e90 con 0x7f2840113600 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f2844a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2840119190 con 0x7f2840076e60 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f283ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840077780 0x7f284019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f283fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840113600 0x7f28401a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f283ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840077780 0x7f284019fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43236/0 (socket says 172.21.15.33:43236) 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f283ea00640 1 -- 172.21.15.33:0/3583680499 learned_addr learned my addr 172.21.15.33:0/3583680499 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f283f400640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2840076e60 0x7f284019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f283fe00640 1 -- 172.21.15.33:0/3583680499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2840076e60 msgr2=0x7f284019f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.431+0000 7f283fe00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2840076e60 0x7f284019f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f283fe00640 1 -- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840077780 msgr2=0x7f284019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f283fe00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840077780 0x7f284019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f283fe00640 1 -- 172.21.15.33:0/3583680499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28401a4610 con 0x7f2840113600 2024-04-26T11:32:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f283ea00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840077780 0x7f284019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f283fe00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840113600 0x7f28401a3ee0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2838005fd0 tx=0x7f283800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f2837e00640 1 -- 172.21.15.33:0/3583680499 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283800cdf0 con 0x7f2840113600 2024-04-26T11:32:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f2837e00640 1 -- 172.21.15.33:0/3583680499 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2838005020 con 0x7f2840113600 2024-04-26T11:32:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28401a4900 con 0x7f2840113600 2024-04-26T11:32:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f2837e00640 1 -- 172.21.15.33:0/3583680499 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2838010690 con 0x7f2840113600 2024-04-26T11:32:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.432+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28401ac190 con 0x7f2840113600 2024-04-26T11:32:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2837e00640 1 -- 172.21.15.33:0/3583680499 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2838005460 con 0x7f2840113600 2024-04-26T11:32:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 Processor -- start 2024-04-26T11:32:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 -- start start 2024-04-26T11:32:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903de3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a0903e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a0903eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a090343380 con 0x55a0903e91c0 2024-04-26T11:32:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a0902dfc30 con 0x55a0903de900 2024-04-26T11:32:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.436+0000 7f2848e53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a09033f340 con 0x55a0903ddfe0 2024-04-26T11:32:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903de3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a0903e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a0903eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a0903e8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58434/0 (socket says 172.21.15.33:58434) 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 -- 172.21.15.33:0/1799855890 learned_addr learned my addr 172.21.15.33:0/1799855890 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 -- 172.21.15.33:0/1799855890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 msgr2=0x55a0903de3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 --2- 172.21.15.33:0/1799855890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903de3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 -- 172.21.15.33:0/1799855890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 msgr2=0x55a0903eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 --2- 172.21.15.33:0/1799855890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a0903eb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.437+0000 7f283f400640 1 -- 172.21.15.33:0/1799855890 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a090301940 con 0x55a0903de900 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.438+0000 7f2837e00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f28240a3d10 0x7f28240a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.438+0000 7f283f400640 1 --2- 172.21.15.33:0/1799855890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a0903e8c80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2830009cf0 tx=0x7f2830011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ee68733c5d40890 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.438+0000 7f2837e00640 1 -- 172.21.15.33:0/3583680499 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f28380d0e00 con 0x7f2840113600 2024-04-26T11:32:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.438+0000 7f283ea00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f28240a3d10 0x7f28240a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.438+0000 7f282be00640 1 -- 172.21.15.33:0/1799855890 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283001e070 con 0x55a0903de900 2024-04-26T11:32:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.438+0000 7f282be00640 1 -- 172.21.15.33:0/1799855890 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2830009de0 con 0x55a0903de900 2024-04-26T11:32:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.438+0000 7f282be00640 1 -- 172.21.15.33:0/1799855890 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830019450 con 0x55a0903de900 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 -- 172.21.15.33:0/1799855890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 msgr2=0x55a0903e8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f283ea00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f28240a3d10 0x7f28240a61d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f282c000c00 tx=0x7f282c0058d0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 --2- 172.21.15.33:0/1799855890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a0903e8c80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2830009cf0 tx=0x7f2830011810 comp rx=0 tx=0).stop 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 -- 172.21.15.33:0/1799855890 shutdown_connections 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 --2- 172.21.15.33:0/1799855890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a0903eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 --2- 172.21.15.33:0/1799855890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a0903e8c80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 --2- 172.21.15.33:0/1799855890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903de3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 -- 172.21.15.33:0/1799855890 >> 172.21.15.33:0/1799855890 conn(0x55a0903da4f0 msgr2=0x55a0903da8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 -- 172.21.15.33:0/1799855890 shutdown_connections 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 -- 172.21.15.33:0/1799855890 wait complete. 2024-04-26T11:32:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.439+0000 7f2848e53740 1 -- start start 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 1 Processor -- start 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903e6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a0903ebdf0 con 0x55a0903de900 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a0902cac60 con 0x55a0903e91c0 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a0903ebf70 con 0x55a0903ddfe0 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.440+0000 7f2848e53740 10 client.0 _ll_register_callbacks cb 0x55a0902f6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43244/0 (socket says 172.21.15.33:43244) 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283ea00640 1 -- 172.21.15.33:0/861704276 learned_addr learned my addr 172.21.15.33:0/861704276 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283f400640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283fe00640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283f400640 1 -- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 msgr2=0x55a0903e6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283f400640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283f400640 1 -- 172.21.15.33:0/861704276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 msgr2=0x55a0903e6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283f400640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283f400640 1 -- 172.21.15.33:0/861704276 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0902d9510 con 0x55a0903ddfe0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283ea00640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f283f400640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903e6130 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2830009cf0 tx=0x7f2830005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0591efe14a1b180 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.441+0000 7f2828c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.442+0000 7f2828c00640 1 -- 172.21.15.33:0/861704276 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830027060 con 0x55a0903ddfe0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.442+0000 7f2828c00640 1 -- 172.21.15.33:0/861704276 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2830005460 con 0x55a0903ddfe0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.442+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2830009de0 con 0x55a0903ddfe0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.442+0000 7f2828c00640 1 -- 172.21.15.33:0/861704276 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f283001e040 con 0x55a0903ddfe0 2024-04-26T11:32:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.442+0000 7f2848e53740 10 client.25402 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.442+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a090055210 con 0x55a0903ddfe0 2024-04-26T11:32:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2828c00640 1 -- 172.21.15.33:0/861704276 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2830009de0 con 0x55a0903ddfe0 2024-04-26T11:32:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2828c00640 1 -- 172.21.15.33:0/861704276 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f2830021070 con 0x55a0903ddfe0 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2848e53740 10 client.25402 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2848e53740 10 client.25402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2848e53740 20 client.25402 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2848e53740 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a0903f8e20 0x55a0903fb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2848e53740 4 client.25402 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.443+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a090350f90 con 0x55a0903f8e20 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.444+0000 7f283ea00640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a0903f8e20 0x55a0903fb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.445+0000 7f283ea00640 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a0903f8e20 0x55a0903fb210 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c0c4fbf6894448 server_cookie=ae449320771ee8d7 in_seq=0 out_seq=0 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.445+0000 7f2828c00640 10 client.25402 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.445+0000 7f2828c00640 1 -- 172.21.15.33:0/861704276 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a090350f90 con 0x55a0903f8e20 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.445+0000 7f2828c00640 10 client.25402 handle_command_reply: tid=0 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.445+0000 7f2848e53740 1 client.25402 shutdown 2024-04-26T11:32:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 msgr2=0x55a0903e6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903e6130 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2830009cf0 tx=0x7f2830005270 comp rx=0 tx=0).stop 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 shutdown_connections 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a0903f8e20 0x55a0903fb210 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 --2- 172.21.15.33:0/861704276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903e6130 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 >> 172.21.15.33:0/861704276 conn(0x55a0903da4f0 msgr2=0x55a0903e0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2828c00640 0 client.25402 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 shutdown_connections 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 1 -- 172.21.15.33:0/861704276 wait complete. 2024-04-26T11:32:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.446+0000 7f2848e53740 20 client.25402 trim_cache size 0 max 16384 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.462+0000 7f2848e53740 1 Processor -- start 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.462+0000 7f2848e53740 1 -- start start 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f2848e53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f2848e53740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a090400a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f2848e53740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a090400fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f2848e53740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28401b73b0 con 0x55a0903e91c0 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f2848e53740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f28401b4c80 con 0x55a0903de900 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f2848e53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f28401b2ad0 con 0x55a0903ddfe0 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f283fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a090400fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f283fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a090400fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43252/0 (socket says 172.21.15.33:43252) 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f283fe00640 1 -- 172.21.15.33:0/1128166209 learned_addr learned my addr 172.21.15.33:0/1128166209 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f283ea00640 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a090400a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f283f400640 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.463+0000 7f283fe00640 1 -- 172.21.15.33:0/1128166209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 msgr2=0x55a0903f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f283fe00640 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f283fe00640 1 -- 172.21.15.33:0/1128166209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 msgr2=0x55a090400a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f283fe00640 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a090400a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f283fe00640 1 -- 172.21.15.33:0/1128166209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0903de730 con 0x55a0903e91c0 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f283ea00640 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a090400a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f283fe00640 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a090400fa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f283809ca50 tx=0x7f2838005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cc9c81ff3d7752 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f282b400640 1 -- 172.21.15.33:0/1128166209 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28380b5070 con 0x55a0903e91c0 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f282b400640 1 -- 172.21.15.33:0/1128166209 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28380a59c0 con 0x55a0903e91c0 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.464+0000 7f282b400640 1 -- 172.21.15.33:0/1128166209 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28380a5530 con 0x55a0903e91c0 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 -- 172.21.15.33:0/1128166209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 msgr2=0x55a090400fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a090400fa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f283809ca50 tx=0x7f2838005770 comp rx=0 tx=0).stop 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 -- 172.21.15.33:0/1128166209 shutdown_connections 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903e91c0 0x55a090400fa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903de900 0x55a090400a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 --2- 172.21.15.33:0/1128166209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 -- 172.21.15.33:0/1128166209 >> 172.21.15.33:0/1128166209 conn(0x55a0903da4f0 msgr2=0x55a0903da8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 -- 172.21.15.33:0/1128166209 shutdown_connections 2024-04-26T11:32:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 -- 172.21.15.33:0/1128166209 wait complete. 2024-04-26T11:32:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.465+0000 7f2848e53740 1 -- start start 2024-04-26T11:32:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 1 Processor -- start 2024-04-26T11:32:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28401b3140 con 0x55a0903de900 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f28401b45b0 con 0x55a0903e91c0 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f28401b3810 con 0x55a0903ddfe0 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f2848e53740 10 client.0 _ll_register_callbacks cb 0x7f283001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.466+0000 7f283ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43260/0 (socket says 172.21.15.33:43260) 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 -- 172.21.15.33:0/3966088194 learned_addr learned my addr 172.21.15.33:0/3966088194 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283f400640 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283fe00640 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 -- 172.21.15.33:0/3966088194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 msgr2=0x55a0903f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 -- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 msgr2=0x55a0903e3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:12.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 -- 172.21.15.33:0/3966088194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a090121620 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f283ea00640 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e2bb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f282c009230 tx=0x7f282c02a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe2f9a3d55dd2e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.467+0000 7f2828c00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2828c00640 1 -- 172.21.15.33:0/3966088194 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c035040 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2828c00640 1 -- 172.21.15.33:0/3966088194 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f282c054740 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28380a59c0 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2828c00640 1 -- 172.21.15.33:0/3966088194 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f282c063050 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a0902fdcb0 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2828c00640 1 -- 172.21.15.33:0/3966088194 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28380a59c0 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2828c00640 1 -- 172.21.15.33:0/3966088194 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f282c049040 con 0x55a0903de900 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 10 client.15291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 20 client.15291 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:12.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a09045f790 0x55a090461bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:12.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 4 client.15291 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:32:12.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.468+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a09034ae00 con 0x55a09045f790 2024-04-26T11:32:12.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.469+0000 7f283f400640 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a09045f790 0x55a090461bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:12.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.469+0000 7f283f400640 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a09045f790 0x55a090461bb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af20d8966e565fd5 server_cookie=fcd66e44d06d046e in_seq=0 out_seq=0 2024-04-26T11:32:12.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.469+0000 7f2828c00640 10 client.15291 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2828c00640 1 -- 172.21.15.33:0/3966088194 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a09034ae00 con 0x55a09045f790 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2828c00640 10 client.15291 handle_command_reply: tid=0 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 client.15291 shutdown 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 msgr2=0x55a0903e2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e2bb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f282c009230 tx=0x7f282c02a440 comp rx=0 tx=0).stop 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 shutdown_connections 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a09045f790 0x55a090461bb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0903e91c0 0x55a0903e3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a0903de900 0x55a0903e2bb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 --2- 172.21.15.33:0/3966088194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a0903ddfe0 0x55a0903f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 >> 172.21.15.33:0/3966088194 conn(0x55a0903da4f0 msgr2=0x55a0903df990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2828c00640 0 client.15291 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 shutdown_connections 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.488+0000 7f2848e53740 1 -- 172.21.15.33:0/3966088194 wait complete. 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.489+0000 7f2848e53740 20 client.15291 trim_cache size 0 max 16384 2024-04-26T11:32:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:32:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:32:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:32:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:32:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.489+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f28240a3d10 msgr2=0x7f28240a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.489+0000 7f2844a00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f28240a3d10 0x7f28240a61d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f282c000c00 tx=0x7f282c0058d0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.489+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840113600 msgr2=0x7f28401a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.489+0000 7f2844a00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840113600 0x7f28401a3ee0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2838005fd0 tx=0x7f283800ef10 comp rx=0 tx=0).stop 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 shutdown_connections 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f28240a3d10 0x7f28240a61d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840113600 0x7f28401a3ee0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2840077780 0x7f284019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 --2- 172.21.15.33:0/3583680499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2840076e60 0x7f284019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 >> 172.21.15.33:0/3583680499 conn(0x7f2840100410 msgr2=0x7f2840102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 shutdown_connections 2024-04-26T11:32:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:12.490+0000 7f2844a00640 1 -- 172.21.15.33:0/3583680499 wait complete. 2024-04-26T11:32:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:12 smithi194 ceph-mon[33017]: pgmap v57: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 35 MiB/s wr, 997 op/s 2024-04-26T11:32:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:12 smithi033 ceph-mon[27845]: pgmap v57: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 35 MiB/s wr, 997 op/s 2024-04-26T11:32:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:12 smithi096 ceph-mon[32753]: pgmap v57: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 35 MiB/s wr, 997 op/s 2024-04-26T11:32:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:14 smithi194 ceph-mon[33017]: pgmap v58: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 35 MiB/s wr, 997 op/s 2024-04-26T11:32:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:14 smithi033 ceph-mon[27845]: pgmap v58: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 35 MiB/s wr, 997 op/s 2024-04-26T11:32:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:14 smithi096 ceph-mon[32753]: pgmap v58: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 35 MiB/s wr, 997 op/s 2024-04-26T11:32:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:16 smithi194 ceph-mon[33017]: pgmap v59: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 35 MiB/s wr, 1.01k op/s 2024-04-26T11:32:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:16 smithi033 ceph-mon[27845]: pgmap v59: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 35 MiB/s wr, 1.01k op/s 2024-04-26T11:32:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:16 smithi096 ceph-mon[32753]: pgmap v59: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 35 MiB/s wr, 1.01k op/s 2024-04-26T11:32:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 60.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:32:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:18 smithi194 ceph-mon[33017]: pgmap v60: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 33 MiB/s wr, 719 op/s 2024-04-26T11:32:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:18 smithi033 ceph-mon[27845]: pgmap v60: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 33 MiB/s wr, 719 op/s 2024-04-26T11:32:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:18 smithi096 ceph-mon[32753]: pgmap v60: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 33 MiB/s wr, 719 op/s 2024-04-26T11:32:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:20 smithi194 ceph-mon[33017]: pgmap v61: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 33 MiB/s wr, 725 op/s 2024-04-26T11:32:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:20 smithi033 ceph-mon[27845]: pgmap v61: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 33 MiB/s wr, 725 op/s 2024-04-26T11:32:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:20 smithi096 ceph-mon[32753]: pgmap v61: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 33 MiB/s wr, 725 op/s 2024-04-26T11:32:22.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 65.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:32:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:22 smithi194 ceph-mon[33017]: pgmap v62: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-26T11:32:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:22 smithi033 ceph-mon[27845]: pgmap v62: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-26T11:32:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:22 smithi096 ceph-mon[32753]: pgmap v62: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-26T11:32:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:24 smithi194 ceph-mon[33017]: pgmap v63: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-26T11:32:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:24 smithi033 ceph-mon[27845]: pgmap v63: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-26T11:32:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:24 smithi096 ceph-mon[32753]: pgmap v63: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 -- 172.21.15.33:0/782260849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 msgr2=0x7f6ea4104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/782260849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea4104df0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009900 tx=0x7f6e9002a400 comp rx=0 tx=0).stop 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 -- 172.21.15.33:0/782260849 shutdown_connections 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/782260849 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea4105330 0x7f6ea4111840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/782260849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea4104df0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/782260849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea410a980 0x7f6ea410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 -- 172.21.15.33:0/782260849 >> 172.21.15.33:0/782260849 conn(0x7f6ea4100440 msgr2=0x7f6ea4102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 -- 172.21.15.33:0/782260849 shutdown_connections 2024-04-26T11:32:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.639+0000 7f6eaaa00640 1 -- 172.21.15.33:0/782260849 wait complete. 2024-04-26T11:32:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.640+0000 7f6eaaa00640 1 Processor -- start 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.640+0000 7f6eaaa00640 1 -- start start 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6eaaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6eaaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea4105330 0x7f6ea419b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6eaaa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea410a980 0x7f6ea419fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6eaaa00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4114c10 con 0x7f6ea410a980 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6eaaa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4114a90 con 0x7f6ea4104970 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6eaaa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4114d90 con 0x7f6ea4105330 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6ea9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6ea9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea419b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49374/0 (socket says 172.21.15.33:49374) 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6ea9600640 1 -- 172.21.15.33:0/629539166 learned_addr learned my addr 172.21.15.33:0/629539166 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6eaa000640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea410a980 0x7f6ea419fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.641+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea4105330 0x7f6ea419b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea9600640 1 -- 172.21.15.33:0/629539166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea4105330 msgr2=0x7f6ea419b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea9600640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea4105330 0x7f6ea419b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea9600640 1 -- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea410a980 msgr2=0x7f6ea419fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea9600640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea410a980 0x7f6ea419fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea9600640 1 -- 172.21.15.33:0/629539166 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea41a0180 con 0x7f6ea4104970 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea9600640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea419b190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800ea60 tx=0x7f6e9800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea2000640 1 -- 172.21.15.33:0/629539166 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800cde0 con 0x7f6ea4104970 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea2000640 1 -- 172.21.15.33:0/629539166 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e980035f0 con 0x7f6ea4104970 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea41a0470 con 0x7f6ea4104970 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6ea2000640 1 -- 172.21.15.33:0/629539166 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98010690 con 0x7f6ea4104970 2024-04-26T11:32:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.642+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea41a7d00 con 0x7f6ea4104970 2024-04-26T11:32:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 Processor -- start 2024-04-26T11:32:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 -- start start 2024-04-26T11:32:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6ea2000640 1 -- 172.21.15.33:0/629539166 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6e980037a0 con 0x7f6ea4104970 2024-04-26T11:32:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af5392f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af53939c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af53894380 con 0x55af5392f900 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af53830c30 con 0x55af5393a1c0 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.646+0000 7f6eaee32740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55af53890340 con 0x55af5392efe0 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6ea9600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af53939c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6ea9600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af53939c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41646/0 (socket says 172.21.15.33:41646) 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6ea9600640 1 -- 172.21.15.33:0/3074516338 learned_addr learned my addr 172.21.15.33:0/3074516338 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af5392f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6eaa000640 1 -- 172.21.15.33:0/3074516338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 msgr2=0x55af5393c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6eaa000640 1 -- 172.21.15.33:0/3074516338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 msgr2=0x55af53939c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af53939c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6eaa000640 1 -- 172.21.15.33:0/3074516338 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af53852940 con 0x55af5392efe0 2024-04-26T11:32:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.647+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af5392f3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009f50 tx=0x7f6e9c003e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d90c5fae12b7b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6e96600640 1 -- 172.21.15.33:0/3074516338 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c011440 con 0x55af5392efe0 2024-04-26T11:32:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6e96600640 1 -- 172.21.15.33:0/3074516338 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e9c015030 con 0x55af5392efe0 2024-04-26T11:32:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6e96600640 1 -- 172.21.15.33:0/3074516338 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c010920 con 0x55af5392efe0 2024-04-26T11:32:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6ea2000640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f6e8c0a3d90 0x7f6e8c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6ea2000640 1 -- 172.21.15.33:0/629539166 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6e980d09e0 con 0x7f6ea4104970 2024-04-26T11:32:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6ea9600640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f6e8c0a3d90 0x7f6e8c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6eaee32740 1 -- 172.21.15.33:0/3074516338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 msgr2=0x55af5392f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.648+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af5392f3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009f50 tx=0x7f6e9c003e00 comp rx=0 tx=0).stop 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 -- 172.21.15.33:0/3074516338 shutdown_connections 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af53939c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3074516338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af5392f3c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 -- 172.21.15.33:0/3074516338 >> 172.21.15.33:0/3074516338 conn(0x55af5392b4f0 msgr2=0x55af5392b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 -- 172.21.15.33:0/3074516338 shutdown_connections 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 -- 172.21.15.33:0/3074516338 wait complete. 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6ea9600640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f6e8c0a3d90 0x7f6e8c0a6250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e98098d90 tx=0x7f6e980d6530 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 -- start start 2024-04-26T11:32:25.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.649+0000 7f6eaee32740 1 Processor -- start 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af539370a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af539375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af53937b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaee32740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55af5393cdf0 con 0x55af5392f900 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaee32740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af5381bc60 con 0x55af5393a1c0 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaee32740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55af5393cf70 con 0x55af5392efe0 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaee32740 10 client.0 _ll_register_callbacks cb 0x55af53847e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6ea8c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af539375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaa000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af539370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6ea8c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af539375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41656/0 (socket says 172.21.15.33:41656) 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6ea8c00640 1 -- 172.21.15.33:0/3558751997 learned_addr learned my addr 172.21.15.33:0/3558751997 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.650+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af53937b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6ea8c00640 1 -- 172.21.15.33:0/3558751997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 msgr2=0x55af539370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af539370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6ea8c00640 1 -- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 msgr2=0x55af53937b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af53937b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6ea8c00640 1 -- 172.21.15.33:0/3558751997 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af5382a510 con 0x55af5392f900 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af53937b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af539375e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6e9002a920 tx=0x7f6e90031e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70f04f559e47c8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6e8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3558751997 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90035040 con 0x55af5392f900 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3558751997 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6e900316f0 con 0x55af5392f900 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.651+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9c015030 con 0x55af5392f900 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3558751997 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e90004040 con 0x55af5392f900 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 10 client.15303 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55af535a6210 con 0x55af5392f900 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3558751997 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e9c015030 con 0x55af5392f900 2024-04-26T11:32:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3558751997 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f6e900047a0 con 0x55af5392f900 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 10 client.15303 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 10 client.15303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 20 client.15303 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af53949eb0 0x55af5394c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 4 client.15303 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.652+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af538a1f90 con 0x55af53949eb0 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.653+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af53949eb0 0x55af5394c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.653+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af53949eb0 0x55af5394c2a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87465e291c888927 server_cookie=6a2aaddb074f0773 in_seq=0 out_seq=0 2024-04-26T11:32:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.653+0000 7f6e8aa00640 10 client.15303 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.654+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3558751997 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af538a1f90 con 0x55af53949eb0 2024-04-26T11:32:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.654+0000 7f6e8aa00640 10 client.15303 handle_command_reply: tid=0 2024-04-26T11:32:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.654+0000 7f6eaee32740 1 client.15303 shutdown 2024-04-26T11:32:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.654+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 msgr2=0x55af539375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.654+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af539375e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6e9002a920 tx=0x7f6e90031e10 comp rx=0 tx=0).stop 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 shutdown_connections 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af53949eb0 0x55af5394c2a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af53937b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392f900 0x55af539375e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6e8aa00640 0 client.15303 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3558751997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392efe0 0x55af539370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 >> 172.21.15.33:0/3558751997 conn(0x55af5392b4f0 msgr2=0x55af53931270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 shutdown_connections 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 1 -- 172.21.15.33:0/3558751997 wait complete. 2024-04-26T11:32:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.655+0000 7f6eaee32740 20 client.15303 trim_cache size 0 max 16384 2024-04-26T11:32:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 Processor -- start 2024-04-26T11:32:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 -- start start 2024-04-26T11:32:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5392efe0 0x55af53945cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af53936200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5393a1c0 0x55af53937940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b2f30 con 0x55af5393a1c0 2024-04-26T11:32:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b0800 con 0x55af5392efe0 2024-04-26T11:32:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.672+0000 7f6eaee32740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41ae650 con 0x55af5392f900 2024-04-26T11:32:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af53936200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6eaa000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5393a1c0 0x55af53937940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af53936200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:53436/0 (socket says 172.21.15.33:53436) 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 -- 172.21.15.33:0/2755182085 learned_addr learned my addr 172.21.15.33:0/2755182085 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6eaa000640 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5392efe0 0x55af53945cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 -- 172.21.15.33:0/2755182085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5392efe0 msgr2=0x55af53945cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5392efe0 0x55af53945cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 -- 172.21.15.33:0/2755182085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5393a1c0 msgr2=0x55af53937940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5393a1c0 0x55af53937940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6ea8c00640 1 -- 172.21.15.33:0/2755182085 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af5392f730 con 0x55af5392f900 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.673+0000 7f6eaa000640 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5392efe0 0x55af53945cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:32:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.674+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af53936200 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6e90004d60 tx=0x7f6e900095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ef021093275a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.674+0000 7f6e95c00640 1 -- 172.21.15.33:0/2755182085 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90035070 con 0x55af5392f900 2024-04-26T11:32:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.674+0000 7f6e95c00640 1 -- 172.21.15.33:0/2755182085 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e900319a0 con 0x55af5392f900 2024-04-26T11:32:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.674+0000 7f6e95c00640 1 -- 172.21.15.33:0/2755182085 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90053040 con 0x55af5392f900 2024-04-26T11:32:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 -- 172.21.15.33:0/2755182085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 msgr2=0x55af53936200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af53936200 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6e90004d60 tx=0x7f6e900095f0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 -- 172.21.15.33:0/2755182085 shutdown_connections 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af53936200 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5393a1c0 0x55af53937940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 --2- 172.21.15.33:0/2755182085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5392efe0 0x55af53945cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 -- 172.21.15.33:0/2755182085 >> 172.21.15.33:0/2755182085 conn(0x55af5392b4f0 msgr2=0x55af539387c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 -- 172.21.15.33:0/2755182085 shutdown_connections 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 -- 172.21.15.33:0/2755182085 wait complete. 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 -- start start 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.675+0000 7f6eaee32740 1 Processor -- start 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392efe0 0x55af53942410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af5393d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaee32740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaee32740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41aecc0 con 0x55af5392efe0 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaee32740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b0130 con 0x55af5393a1c0 2024-04-26T11:32:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaee32740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41af390 con 0x55af5392f900 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaee32740 10 client.0 _ll_register_callbacks cb 0x7f6e90005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6eaa000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392efe0 0x55af53942410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6ea8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af5393d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6ea8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af5393d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:53440/0 (socket says 172.21.15.33:53440) 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.676+0000 7f6ea8c00640 1 -- 172.21.15.33:0/3716757071 learned_addr learned my addr 172.21.15.33:0/3716757071 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaa000640 1 -- 172.21.15.33:0/3716757071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 msgr2=0x55af5393d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af5393d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaa000640 1 -- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 msgr2=0x55af5393dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaa000640 1 -- 172.21.15.33:0/3716757071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af53672620 con 0x55af5392efe0 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af5393d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaa000640 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392efe0 0x55af53942410 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0105c0 tx=0x7f6e9c003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e627d462e9ef34 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6e8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:32:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3716757071 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c024040 con 0x55af5392efe0 2024-04-26T11:32:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3716757071 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6e9c02c5e0 con 0x55af5392efe0 2024-04-26T11:32:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e900319a0 con 0x55af5392efe0 2024-04-26T11:32:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.677+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3716757071 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e9c02c800 con 0x55af5392efe0 2024-04-26T11:32:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 10 client.15309 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55af5384ecb0 con 0x55af5392efe0 2024-04-26T11:32:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3716757071 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e900319a0 con 0x55af5392efe0 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3716757071 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f6e9c010680 con 0x55af5392efe0 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 10 client.15309 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 10 client.15309 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 20 client.15309 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af539b0370 0x55af539b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 4 client.15309 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af5389be00 con 0x55af539b0370 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.678+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af539b0370 0x55af539b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.679+0000 7f6ea8c00640 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af539b0370 0x55af539b2790 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b31d7a65dc3edc5 server_cookie=948d0d859eb39d02 in_seq=0 out_seq=0 2024-04-26T11:32:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.679+0000 7f6e8aa00640 10 client.15309 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6e8aa00640 1 -- 172.21.15.33:0/3716757071 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af5389be00 con 0x55af539b0370 2024-04-26T11:32:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6e8aa00640 10 client.15309 handle_command_reply: tid=0 2024-04-26T11:32:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 client.15309 shutdown 2024-04-26T11:32:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392efe0 msgr2=0x55af53942410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392efe0 0x55af53942410 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0105c0 tx=0x7f6e9c003e00 comp rx=0 tx=0).stop 2024-04-26T11:32:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 shutdown_connections 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55af5392f900 0x55af5393d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55af539b0370 0x55af539b2790 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af5393a1c0 0x55af5393dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 --2- 172.21.15.33:0/3716757071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55af5392efe0 0x55af53942410 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6e8aa00640 0 client.15309 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 >> 172.21.15.33:0/3716757071 conn(0x55af5392b4f0 msgr2=0x55af53931220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 shutdown_connections 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 1 -- 172.21.15.33:0/3716757071 wait complete. 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.697+0000 7f6eaee32740 20 client.15309 trim_cache size 0 max 16384 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:32:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:32:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:32:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:32:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f6e8c0a3d90 msgr2=0x7f6e8c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f6e8c0a3d90 0x7f6e8c0a6250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e98098d90 tx=0x7f6e980d6530 comp rx=0 tx=0).stop 2024-04-26T11:32:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 msgr2=0x7f6ea419b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea419b190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800ea60 tx=0x7f6e9800ef20 comp rx=0 tx=0).stop 2024-04-26T11:32:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 shutdown_connections 2024-04-26T11:32:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f6e8c0a3d90 0x7f6e8c0a6250 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea4105330 0x7f6ea419b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ea410a980 0x7f6ea419fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 --2- 172.21.15.33:0/629539166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ea4104970 0x7f6ea419b190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.698+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 >> 172.21.15.33:0/629539166 conn(0x7f6ea4100440 msgr2=0x7f6ea4100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.699+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 shutdown_connections 2024-04-26T11:32:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:25.699+0000 7f6eaaa00640 1 -- 172.21.15.33:0/629539166 wait complete. 2024-04-26T11:32:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:26 smithi194 ceph-mon[33017]: pgmap v64: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 65 KiB/s wr, 18 op/s 2024-04-26T11:32:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:26 smithi033 ceph-mon[27845]: pgmap v64: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 65 KiB/s wr, 18 op/s 2024-04-26T11:32:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:26 smithi096 ceph-mon[32753]: pgmap v64: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 65 KiB/s wr, 18 op/s 2024-04-26T11:32:26.859 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:32:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 70.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:32:27.328 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:32:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:28 smithi194 ceph-mon[33017]: pgmap v65: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 52 KiB/s wr, 9 op/s 2024-04-26T11:32:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:28 smithi033 ceph-mon[27845]: pgmap v65: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 52 KiB/s wr, 9 op/s 2024-04-26T11:32:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:28 smithi096 ceph-mon[32753]: pgmap v65: 97 pgs: 97 active+clean; 962 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 52 KiB/s wr, 9 op/s 2024-04-26T11:32:29.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.664+0000 7f0c79600700 1 -- 172.21.15.33:0/25584331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74078270 msgr2=0x7f0c74113dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.664+0000 7f0c79600700 1 --2- 172.21.15.33:0/25584331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74078270 0x7f0c74113dc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009990 tx=0x7f0c5c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.664+0000 7f0c79600700 1 -- 172.21.15.33:0/25584331 shutdown_connections 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.664+0000 7f0c79600700 1 --2- 172.21.15.33:0/25584331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c74114300 0x7f0c741166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.664+0000 7f0c79600700 1 --2- 172.21.15.33:0/25584331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c74077950 0x7f0c74077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.664+0000 7f0c79600700 1 --2- 172.21.15.33:0/25584331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74078270 0x7f0c74113dc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.664+0000 7f0c79600700 1 -- 172.21.15.33:0/25584331 >> 172.21.15.33:0/25584331 conn(0x7f0c74101150 msgr2=0x7f0c74103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.665+0000 7f0c79600700 1 -- 172.21.15.33:0/25584331 shutdown_connections 2024-04-26T11:32:29.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.665+0000 7f0c79600700 1 -- 172.21.15.33:0/25584331 wait complete. 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 Processor -- start 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 -- start start 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74077950 0x7f0c741a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c74078270 0x7f0c741a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c74114300 0x7f0c741a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74119cd0 con 0x7f0c74077950 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74119b50 con 0x7f0c74114300 2024-04-26T11:32:29.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.666+0000 7f0c79600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74119e50 con 0x7f0c74078270 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c72a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74077950 0x7f0c741a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c6a000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c74078270 0x7f0c741a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c6a000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c74078270 0x7f0c741a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:53444/0 (socket says 172.21.15.33:53444) 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c6a000700 1 -- 172.21.15.33:0/2340595832 learned_addr learned my addr 172.21.15.33:0/2340595832 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c73400700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c74114300 0x7f0c741a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c73400700 1 -- 172.21.15.33:0/2340595832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c74078270 msgr2=0x7f0c741a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c73400700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c74078270 0x7f0c741a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c73400700 1 -- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74077950 msgr2=0x7f0c741a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c73400700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74077950 0x7f0c741a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.667+0000 7f0c73400700 1 -- 172.21.15.33:0/2340595832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c741a5210 con 0x7f0c74114300 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.668+0000 7f0c73400700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c74114300 0x7f0c741a4ad0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0c7407bc40 tx=0x7f0c6c00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.668+0000 7f0c6be00700 1 -- 172.21.15.33:0/2340595832 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0092e0 con 0x7f0c74114300 2024-04-26T11:32:29.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.668+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c741a54a0 con 0x7f0c74114300 2024-04-26T11:32:29.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.668+0000 7f0c6be00700 1 -- 172.21.15.33:0/2340595832 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c6c004bb0 con 0x7f0c74114300 2024-04-26T11:32:29.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.668+0000 7f0c6be00700 1 -- 172.21.15.33:0/2340595832 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c004020 con 0x7f0c74114300 2024-04-26T11:32:29.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.668+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c741acd80 con 0x7f0c74114300 2024-04-26T11:32:29.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.670+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c7406a110 con 0x7f0c74114300 2024-04-26T11:32:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.671+0000 7f0c6be00700 1 -- 172.21.15.33:0/2340595832 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0c6c01b030 con 0x7f0c74114300 2024-04-26T11:32:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.673+0000 7f0c6be00700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f0c580a3db0 0x7f0c580a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.673+0000 7f0c6be00700 1 -- 172.21.15.33:0/2340595832 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0c6c0d0260 con 0x7f0c74114300 2024-04-26T11:32:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.673+0000 7f0c72a00700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f0c580a3db0 0x7f0c580a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.674+0000 7f0c72a00700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f0c580a3db0 0x7f0c580a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009680 tx=0x7f0c64009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.677+0000 7f0c6be00700 1 -- 172.21.15.33:0/2340595832 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c6c022020 con 0x7f0c74114300 2024-04-26T11:32:29.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.960+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c7402e490 con 0x7f0c74114300 2024-04-26T11:32:29.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.961+0000 7f0c6be00700 1 -- 172.21.15.33:0/2340595832 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11868 (secure 0 0 0) 0x7f0c6c010040 con 0x7f0c74114300 2024-04-26T11:32:29.962 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:32:29.963 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:31:11.426188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.96:6837/823522377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":823522377},{"type":"v1","addr":"172.21.15.96:6837","nonce":823522377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24527,24484]},"id":1}]} 2024-04-26T11:32:29.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f0c580a3db0 msgr2=0x7f0c580a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:29.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f0c580a3db0 0x7f0c580a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009680 tx=0x7f0c64009260 comp rx=0 tx=0).stop 2024-04-26T11:32:29.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c74114300 msgr2=0x7f0c741a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:29.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c74114300 0x7f0c741a4ad0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0c7407bc40 tx=0x7f0c6c00ef90 comp rx=0 tx=0).stop 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 shutdown_connections 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f0c580a3db0 0x7f0c580a6270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c74114300 0x7f0c741a4ad0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c74077950 0x7f0c741a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.964+0000 7f0c79600700 1 --2- 172.21.15.33:0/2340595832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c74078270 0x7f0c741a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.965+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 >> 172.21.15.33:0/2340595832 conn(0x7f0c74101150 msgr2=0x7f0c74103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.965+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 shutdown_connections 2024-04-26T11:32:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:29.965+0000 7f0c79600700 1 -- 172.21.15.33:0/2340595832 wait complete. 2024-04-26T11:32:29.967 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 28 2024-04-26T11:32:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:30 smithi033 ceph-mon[27845]: pgmap v66: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 345 KiB/s wr, 51 op/s 2024-04-26T11:32:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:30 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/2340595832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:32:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:30 smithi096 ceph-mon[32753]: pgmap v66: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 345 KiB/s wr, 51 op/s 2024-04-26T11:32:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:30 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/2340595832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:32:30.790 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24527 quiesce db --set-id ab712af7 --release --await' 2024-04-26T11:32:30.790 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24527 quiesce db --set-id ab712af7 --release --await 2024-04-26T11:32:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:30 smithi194 ceph-mon[33017]: pgmap v66: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 345 KiB/s wr, 51 op/s 2024-04-26T11:32:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:30 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/2340595832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:32:31.259 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:32:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 75.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:32:32.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:32 smithi194 ceph-mon[33017]: pgmap v67: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 297 KiB/s wr, 45 op/s 2024-04-26T11:32:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:32 smithi033 ceph-mon[27845]: pgmap v67: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 297 KiB/s wr, 45 op/s 2024-04-26T11:32:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:32 smithi096 ceph-mon[32753]: pgmap v67: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 297 KiB/s wr, 45 op/s 2024-04-26T11:32:33.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 -- 172.21.15.33:0/571056639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 msgr2=0x7f9fd0105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 --2- 172.21.15.33:0/571056639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 0x7f9fd0105ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009990 tx=0x7f9fbc009ca0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 -- 172.21.15.33:0/571056639 shutdown_connections 2024-04-26T11:32:33.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 --2- 172.21.15.33:0/571056639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 0x7f9fd0112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 --2- 172.21.15.33:0/571056639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd010b640 0x7f9fd010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 --2- 172.21.15.33:0/571056639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 0x7f9fd0105ac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 -- 172.21.15.33:0/571056639 >> 172.21.15.33:0/571056639 conn(0x7f9fd0101130 msgr2=0x7f9fd0103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:33.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 -- 172.21.15.33:0/571056639 shutdown_connections 2024-04-26T11:32:33.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.545+0000 7f9fd6200700 1 -- 172.21.15.33:0/571056639 wait complete. 2024-04-26T11:32:33.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.546+0000 7f9fd6200700 1 Processor -- start 2024-04-26T11:32:33.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.547+0000 7f9fd6200700 1 -- start start 2024-04-26T11:32:33.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.547+0000 7f9fd6200700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 0x7f9fd01a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.547+0000 7f9fd6200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 0x7f9fd01a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.547+0000 7f9fd6200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd010b640 0x7f9fd019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.547+0000 7f9fd6200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01158d0 con 0x7f9fd0106000 2024-04-26T11:32:33.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.547+0000 7f9fd6200700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0115750 con 0x7f9fd0105640 2024-04-26T11:32:33.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.547+0000 7f9fd6200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0115a50 con 0x7f9fd010b640 2024-04-26T11:32:33.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 0x7f9fd01a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 0x7f9fd01a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45308/0 (socket says 172.21.15.33:45308) 2024-04-26T11:32:33.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 -- 172.21.15.33:0/2034960790 learned_addr learned my addr 172.21.15.33:0/2034960790 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:33.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd010b640 0x7f9fd019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcf400700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 0x7f9fd01a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 -- 172.21.15.33:0/2034960790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd010b640 msgr2=0x7f9fd019bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd010b640 0x7f9fd019bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 -- 172.21.15.33:0/2034960790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 msgr2=0x7f9fd01a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 0x7f9fd01a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.548+0000 7f9fcea00700 1 -- 172.21.15.33:0/2034960790 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd019c450 con 0x7f9fd0106000 2024-04-26T11:32:33.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.549+0000 7f9fcea00700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 0x7f9fd01a1f90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc000c00 tx=0x7f9fbc0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:33.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.549+0000 7f9fc7e00700 1 -- 172.21.15.33:0/2034960790 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc027070 con 0x7f9fd0106000 2024-04-26T11:32:33.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.549+0000 7f9fc7e00700 1 -- 172.21.15.33:0/2034960790 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fbc022470 con 0x7f9fd0106000 2024-04-26T11:32:33.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.549+0000 7f9fc7e00700 1 -- 172.21.15.33:0/2034960790 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc017650 con 0x7f9fd0106000 2024-04-26T11:32:33.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.549+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd019c6e0 con 0x7f9fd0106000 2024-04-26T11:32:33.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.549+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd01a8800 con 0x7f9fd0106000 2024-04-26T11:32:33.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.553+0000 7f9fc7e00700 1 -- 172.21.15.33:0/2034960790 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9fbc02a070 con 0x7f9fd0106000 2024-04-26T11:32:33.556 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.554+0000 7f9fc7e00700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9fb40a3e30 0x7f9fb40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.555+0000 7f9fc7e00700 1 -- 172.21.15.33:0/2034960790 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9fbc0d1510 con 0x7f9fd0106000 2024-04-26T11:32:33.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.555+0000 7f9fe0b6db80 1 Processor -- start 2024-04-26T11:32:33.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.555+0000 7f9fcf400700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9fb40a3e30 0x7f9fb40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.555+0000 7f9fe0b6db80 1 -- start start 2024-04-26T11:32:33.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 0x5618be0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0cb880 0x5618be0d7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618be01d990 con 0x5618be0d0bb0 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5618bdfbc2b0 con 0x5618be0caee0 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5618be019950 con 0x5618be0cb880 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fcea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fcfe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0cb880 0x5618be0d7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fcea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57150/0 (socket says 172.21.15.33:57150) 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.556+0000 7f9fcea00700 1 -- 172.21.15.33:0/2641300156 learned_addr learned my addr 172.21.15.33:0/2641300156 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fcfe00700 1 -- 172.21.15.33:0/2641300156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 msgr2=0x5618be0cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fcfe00700 1 -- 172.21.15.33:0/2641300156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 msgr2=0x5618be0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 0x5618be0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fcfe00700 1 -- 172.21.15.33:0/2641300156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618bdfc4730 con 0x5618be0cb880 2024-04-26T11:32:33.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fcea00700 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:33.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0cb880 0x5618be0d7900 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8011580 tx=0x7f9fc8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d31d57e70a9deda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:33.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fc4c00700 1 -- 172.21.15.33:0/2641300156 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8011e80 con 0x5618be0cb880 2024-04-26T11:32:33.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fc4c00700 1 -- 172.21.15.33:0/2641300156 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fc8004bb0 con 0x5618be0cb880 2024-04-26T11:32:33.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.557+0000 7f9fc4c00700 1 -- 172.21.15.33:0/2641300156 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8004760 con 0x5618be0cb880 2024-04-26T11:32:33.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.558+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2641300156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0cb880 msgr2=0x5618be0d7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.558+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0cb880 0x5618be0d7900 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8011580 tx=0x7f9fc8009920 comp rx=0 tx=0).stop 2024-04-26T11:32:33.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.558+0000 7f9fcf400700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9fb40a3e30 0x7f9fb40a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0009b10 tx=0x7f9fc0009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:33.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2641300156 shutdown_connections 2024-04-26T11:32:33.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0cb880 0x5618be0d7900 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 0x5618be0d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2641300156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2641300156 >> 172.21.15.33:0/2641300156 conn(0x5618bdff83f0 msgr2=0x5618be0c5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:33.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2641300156 shutdown_connections 2024-04-26T11:32:33.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2641300156 wait complete. 2024-04-26T11:32:33.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.559+0000 7f9fe0b6db80 1 -- start start 2024-04-26T11:32:33.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 1 Processor -- start 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0caee0 0x5618be0d7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0cb880 0x5618be0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 0x5618be0d4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5618be0d7e40 con 0x5618be0d0bb0 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5618bdfb8500 con 0x5618be0cb880 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5618be0d7fc0 con 0x5618be0caee0 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.560+0000 7f9fe0b6db80 10 client.0 _ll_register_callbacks cb 0x5618bdfb1480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:33.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0cb880 0x5618be0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcfe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 0x5618be0d4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0cb880 0x5618be0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57164/0 (socket says 172.21.15.33:57164) 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 -- 172.21.15.33:0/919148749 learned_addr learned my addr 172.21.15.33:0/919148749 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcf400700 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0caee0 0x5618be0d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 -- 172.21.15.33:0/919148749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0caee0 msgr2=0x5618be0d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0caee0 0x5618be0d7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 -- 172.21.15.33:0/919148749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 msgr2=0x5618be0d4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 0x5618be0d4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 -- 172.21.15.33:0/919148749 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618bdf6a9d0 con 0x5618be0cb880 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.561+0000 7f9fcea00700 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0cb880 0x5618be0d4800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc005f50 tx=0x7f9fbc09d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abe7ecbddf326672 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:33.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.562+0000 7f9fb9600700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.562+0000 7f9fb9600700 1 -- 172.21.15.33:0/919148749 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc09d950 con 0x5618be0cb880 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.562+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fc8004bb0 con 0x5618be0cb880 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.562+0000 7f9fb9600700 1 -- 172.21.15.33:0/919148749 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9fbc0a5c40 con 0x5618be0cb880 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.562+0000 7f9fb9600700 1 -- 172.21.15.33:0/919148749 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9fbc09dbf0 con 0x5618be0cb880 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.562+0000 7f9fe0b6db80 10 client.25475 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.562+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5618be0e2630 con 0x5618be0cb880 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fb9600700 1 -- 172.21.15.33:0/919148749 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fc8004bb0 con 0x5618be0cb880 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fb9600700 1 -- 172.21.15.33:0/919148749 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f9fbc0bc170 con 0x5618be0cb880 2024-04-26T11:32:33.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fe0b6db80 10 client.25475 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fe0b6db80 10 client.25475 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fe0b6db80 20 client.25475 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be0e4c80 0x5618be0e70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fe0b6db80 4 client.25475 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618be02b370 con 0x5618be0e4c80 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.563+0000 7f9fcf400700 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be0e4c80 0x5618be0e70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.564+0000 7f9fb9600700 10 client.25475 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.564+0000 7f9fcf400700 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be0e4c80 0x5618be0e70a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2e33f07def4c988 server_cookie=2dd6798a00317235 in_seq=0 out_seq=0 2024-04-26T11:32:33.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fb9600700 1 -- 172.21.15.33:0/919148749 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618be02b370 con 0x5618be0e4c80 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fb9600700 10 client.25475 handle_command_reply: tid=0 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 client.25475 shutdown 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0cb880 msgr2=0x5618be0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0cb880 0x5618be0d4800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc005f50 tx=0x7f9fbc09d3b0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 shutdown_connections 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be0e4c80 0x5618be0e70a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0d0bb0 0x5618be0d4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0caee0 0x5618be0d7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/919148749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0cb880 0x5618be0d4800 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fb9600700 0 client.25475 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:33.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.565+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 >> 172.21.15.33:0/919148749 conn(0x5618bdff83f0 msgr2=0x5618be0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:33.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.566+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 shutdown_connections 2024-04-26T11:32:33.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.566+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/919148749 wait complete. 2024-04-26T11:32:33.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.566+0000 7f9fe0b6db80 20 client.25475 trim_cache size 0 max 16384 2024-04-26T11:32:33.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 Processor -- start 2024-04-26T11:32:33.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 -- start start 2024-04-26T11:32:33.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0e0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0ec970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01b2690 con 0x5618be0cb880 2024-04-26T11:32:33.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01aff20 con 0x5618be0caee0 2024-04-26T11:32:33.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.592+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01add70 con 0x5618be0d0bb0 2024-04-26T11:32:33.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcf400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0e0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcfe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcf400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0e0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57170/0 (socket says 172.21.15.33:57170) 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcf400700 1 -- 172.21.15.33:0/1243345600 learned_addr learned my addr 172.21.15.33:0/1243345600 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcea00700 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0ec970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcfe00700 1 -- 172.21.15.33:0/1243345600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 msgr2=0x5618be0e0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0e0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcfe00700 1 -- 172.21.15.33:0/1243345600 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 msgr2=0x5618be0ec970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0ec970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.593+0000 7f9fcfe00700 1 -- 172.21.15.33:0/1243345600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618be0cb680 con 0x5618be0d0bb0 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.594+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0eceb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8004260 tx=0x7f9fc8009390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63d0c8dfb1261dbc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.594+0000 7f9fbbe00700 1 -- 172.21.15.33:0/1243345600 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc80055b0 con 0x5618be0d0bb0 2024-04-26T11:32:33.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.594+0000 7f9fbbe00700 1 -- 172.21.15.33:0/1243345600 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fc8003a00 con 0x5618be0d0bb0 2024-04-26T11:32:33.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.594+0000 7f9fbbe00700 1 -- 172.21.15.33:0/1243345600 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8007880 con 0x5618be0d0bb0 2024-04-26T11:32:33.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.595+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/1243345600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 msgr2=0x5618be0eceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.595+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0eceb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8004260 tx=0x7f9fc8009390 comp rx=0 tx=0).stop 2024-04-26T11:32:33.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.595+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/1243345600 shutdown_connections 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.595+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0eceb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.595+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0e0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.595+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/1243345600 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0ec970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.595+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/1243345600 >> 172.21.15.33:0/1243345600 conn(0x5618bdff83f0 msgr2=0x5618be0ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.596+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/1243345600 shutdown_connections 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.596+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/1243345600 wait complete. 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.596+0000 7f9fe0b6db80 1 -- start start 2024-04-26T11:32:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.597+0000 7f9fe0b6db80 1 Processor -- start 2024-04-26T11:32:33.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0dd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0dd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fe0b6db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01ae3e0 con 0x5618be0cb880 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01af850 con 0x5618be0caee0 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fe0b6db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01aeab0 con 0x5618be0d0bb0 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fe0b6db80 10 client.0 _ll_register_callbacks cb 0x5618bdfb1480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fcf400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0dd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fcea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0dd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fcf400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0dd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57176/0 (socket says 172.21.15.33:57176) 2024-04-26T11:32:33.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fcf400700 1 -- 172.21.15.33:0/2526656810 learned_addr learned my addr 172.21.15.33:0/2526656810 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:33.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.598+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fcf400700 1 -- 172.21.15.33:0/2526656810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 msgr2=0x5618be0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fcf400700 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fcf400700 1 -- 172.21.15.33:0/2526656810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 msgr2=0x5618be0dd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:33.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fcf400700 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0dd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fcf400700 1 -- 172.21.15.33:0/2526656810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618be0ccc10 con 0x5618be0caee0 2024-04-26T11:32:33.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fcfe00700 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:32:33.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fcf400700 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0dd210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000fa80 tx=0x7f9fc0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=637e3362d6c7f045 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fb9600700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fb9600700 1 -- 172.21.15.33:0/2526656810 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0011da0 con 0x5618be0caee0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fb9600700 1 -- 172.21.15.33:0/2526656810 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9fc0028300 con 0x5618be0caee0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.599+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fc8003a00 con 0x5618be0caee0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fb9600700 1 -- 172.21.15.33:0/2526656810 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9fc0028500 con 0x5618be0caee0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 10 client.25481 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5618be14ccb0 con 0x5618be0caee0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fb9600700 1 -- 172.21.15.33:0/2526656810 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fc8003a00 con 0x5618be0caee0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fb9600700 1 -- 172.21.15.33:0/2526656810 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f9fc0037050 con 0x5618be0caee0 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 10 client.25481 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 10 client.25481 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 20 client.25481 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be14d320 0x5618be14f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:33.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 4 client.25481 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ab712af7", "release": true, "await": true}] 2024-04-26T11:32:33.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.600+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ab712af7", "release": true, "await": true}) -- 0x5618be025410 con 0x5618be14d320 2024-04-26T11:32:33.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.601+0000 7f9fcea00700 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be14d320 0x5618be14f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:33.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.601+0000 7f9fcea00700 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be14d320 0x5618be14f740 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af170198627a405f server_cookie=59760c804486c102 in_seq=0 out_seq=0 2024-04-26T11:32:33.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.601+0000 7f9fb9600700 10 client.25481 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.629+0000 7f9fb9600700 1 -- 172.21.15.33:0/2526656810 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5618be025410 con 0x5618be14d320 2024-04-26T11:32:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.629+0000 7f9fb9600700 10 client.25481 handle_command_reply: tid=0 2024-04-26T11:32:33.634 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:32:33.634 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24527, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 8, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "ab712af7": { 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 8, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 60.0, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 78.0, 2024-04-26T11:32:33.635 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RELEASED", 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:32:33.636 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.629+0000 7f9fe0b6db80 1 client.25481 shutdown 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.629+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 msgr2=0x5618be0dd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.629+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0dd210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000fa80 tx=0x7f9fc0004750 comp rx=0 tx=0).stop 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.630+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 shutdown_connections 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.630+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5618be14d320 0x5618be14f740 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.630+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5618be0d0bb0 0x5618be0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.630+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5618be0caee0 0x5618be0dd210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.630+0000 7f9fe0b6db80 1 --2- 172.21.15.33:0/2526656810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5618be0cb880 0x5618be0dd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.630+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 >> 172.21.15.33:0/2526656810 conn(0x5618bdff83f0 msgr2=0x5618be0d6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:33.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.630+0000 7f9fb9600700 0 client.25481 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.631+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 shutdown_connections 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.631+0000 7f9fe0b6db80 1 -- 172.21.15.33:0/2526656810 wait complete. 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.631+0000 7f9fe0b6db80 20 client.25481 trim_cache size 0 max 16384 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9fb40a3e30 msgr2=0x7f9fb40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9fb40a3e30 0x7f9fb40a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0009b10 tx=0x7f9fc0009260 comp rx=0 tx=0).stop 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 msgr2=0x7f9fd01a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 0x7f9fd01a1f90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc000c00 tx=0x7f9fbc0057c0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 shutdown_connections 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9fb40a3e30 0x7f9fb40a62f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd010b640 0x7f9fd019bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fd0105640 0x7f9fd01a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 --2- 172.21.15.33:0/2034960790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9fd0106000 0x7f9fd01a1f90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.633+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 >> 172.21.15.33:0/2034960790 conn(0x7f9fd0101130 msgr2=0x7f9fd01018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:33.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.634+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 shutdown_connections 2024-04-26T11:32:33.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:32:33.634+0000 7f9fd6200700 1 -- 172.21.15.33:0/2034960790 wait complete. 2024-04-26T11:32:34.377 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:34 smithi033 ceph-mon[27845]: pgmap v68: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 297 KiB/s wr, 45 op/s 2024-04-26T11:32:34.378 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:32:34.621 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ab712af7', seconds elapsed: 34.9 2024-04-26T11:32:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:34 smithi096 ceph-mon[32753]: pgmap v68: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 297 KiB/s wr, 45 op/s 2024-04-26T11:32:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:32:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:34 smithi194 ceph-mon[33017]: pgmap v68: 97 pgs: 97 active+clean; 961 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 297 KiB/s wr, 45 op/s 2024-04-26T11:32:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:32:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:32:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:32:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:32:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:32:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:32:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:32:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:32:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:32:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:32:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:36 smithi033 ceph-mon[27845]: pgmap v69: 97 pgs: 97 active+clean; 964 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 581 KiB/s wr, 81 op/s 2024-04-26T11:32:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:36 smithi096 ceph-mon[32753]: pgmap v69: 97 pgs: 97 active+clean; 964 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 581 KiB/s wr, 81 op/s 2024-04-26T11:32:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:36 smithi194 ceph-mon[33017]: pgmap v69: 97 pgs: 97 active+clean; 964 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 581 KiB/s wr, 81 op/s 2024-04-26T11:32:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 80.0 s, 1650.2 tps, lat 371.909 ms stddev 2976.964 2024-04-26T11:32:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:38 smithi033 ceph-mon[27845]: pgmap v70: 97 pgs: 97 active+clean; 964 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 577 KiB/s wr, 77 op/s 2024-04-26T11:32:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:38 smithi096 ceph-mon[32753]: pgmap v70: 97 pgs: 97 active+clean; 964 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 577 KiB/s wr, 77 op/s 2024-04-26T11:32:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:38 smithi194 ceph-mon[33017]: pgmap v70: 97 pgs: 97 active+clean; 964 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 577 KiB/s wr, 77 op/s 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 -- 172.21.15.33:0/4037273786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe810a980 msgr2=0x7fcbe810ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 --2- 172.21.15.33:0/4037273786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe810a980 0x7fcbe810ad60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009890 tx=0x7fcbdc02a380 comp rx=0 tx=0).stop 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 -- 172.21.15.33:0/4037273786 shutdown_connections 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 --2- 172.21.15.33:0/4037273786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8105330 0x7fcbe8111840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 --2- 172.21.15.33:0/4037273786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbe8104970 0x7fcbe8104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 --2- 172.21.15.33:0/4037273786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe810a980 0x7fcbe810ad60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 -- 172.21.15.33:0/4037273786 >> 172.21.15.33:0/4037273786 conn(0x7fcbe8100440 msgr2=0x7fcbe8102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 -- 172.21.15.33:0/4037273786 shutdown_connections 2024-04-26T11:32:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.890+0000 7fcbee400640 1 -- 172.21.15.33:0/4037273786 wait complete. 2024-04-26T11:32:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.891+0000 7fcbee400640 1 Processor -- start 2024-04-26T11:32:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.891+0000 7fcbee400640 1 -- start start 2024-04-26T11:32:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbee400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8104970 0x7fcbe819b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbee400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe8105330 0x7fcbe819b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbee400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbe810a980 0x7fcbe819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbee400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe8114c60 con 0x7fcbe8104970 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbee400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe8114ae0 con 0x7fcbe810a980 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbee400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe8114de0 con 0x7fcbe8105330 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbed000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8104970 0x7fcbe819b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbed000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8104970 0x7fcbe819b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45374/0 (socket says 172.21.15.33:45374) 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbed000640 1 -- 172.21.15.33:0/1177751088 learned_addr learned my addr 172.21.15.33:0/1177751088 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbeda00640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbe810a980 0x7fcbe819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.892+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe8105330 0x7fcbe819b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.893+0000 7fcbed000640 1 -- 172.21.15.33:0/1177751088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe8105330 msgr2=0x7fcbe819b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.893+0000 7fcbed000640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe8105330 0x7fcbe819b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.893+0000 7fcbed000640 1 -- 172.21.15.33:0/1177751088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbe810a980 msgr2=0x7fcbe819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.893+0000 7fcbed000640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbe810a980 0x7fcbe819fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.893+0000 7fcbed000640 1 -- 172.21.15.33:0/1177751088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe81a01e0 con 0x7fcbe8104970 2024-04-26T11:32:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.893+0000 7fcbed000640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8104970 0x7fcbe819b1f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc02a8f0 tx=0x7fcbdc032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.894+0000 7fcbe5600640 1 -- 172.21.15.33:0/1177751088 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc042070 con 0x7fcbe8104970 2024-04-26T11:32:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.894+0000 7fcbe5600640 1 -- 172.21.15.33:0/1177751088 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbdc030e10 con 0x7fcbe8104970 2024-04-26T11:32:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.894+0000 7fcbe5600640 1 -- 172.21.15.33:0/1177751088 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc037040 con 0x7fcbe8104970 2024-04-26T11:32:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.894+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe81a0470 con 0x7fcbe8104970 2024-04-26T11:32:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.895+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe81a7d60 con 0x7fcbe8104970 2024-04-26T11:32:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 Processor -- start 2024-04-26T11:32:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 -- start start 2024-04-26T11:32:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e02fe0 0x561367e033c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e03900 0x561367e0dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e105b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561367d68380 con 0x561367e03900 2024-04-26T11:32:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561367d04c30 con 0x561367e0e1c0 2024-04-26T11:32:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.901+0000 7fcbf2831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561367d64340 con 0x561367e02fe0 2024-04-26T11:32:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e105b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e105b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57190/0 (socket says 172.21.15.33:57190) 2024-04-26T11:32:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 -- 172.21.15.33:0/1286548003 learned_addr learned my addr 172.21.15.33:0/1286548003 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 -- 172.21.15.33:0/1286548003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e02fe0 msgr2=0x561367e033c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T11:32:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/1286548003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e02fe0 0x561367e033c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 -- 172.21.15.33:0/1286548003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e03900 msgr2=0x561367e0dc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T11:32:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/1286548003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e03900 0x561367e0dc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 -- 172.21.15.33:0/1286548003 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561367d26940 con 0x561367e0e1c0 2024-04-26T11:32:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/1286548003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e105b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009c90 tx=0x7fcbd4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d062f440dfa16ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.902+0000 7fcbda000640 1 -- 172.21.15.33:0/1286548003 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd401aa50 con 0x561367e0e1c0 2024-04-26T11:32:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.903+0000 7fcbda000640 1 -- 172.21.15.33:0/1286548003 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd4009e40 con 0x561367e0e1c0 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.903+0000 7fcbda000640 1 -- 172.21.15.33:0/1286548003 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4004b20 con 0x561367e0e1c0 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.903+0000 7fcbf2831740 1 -- 172.21.15.33:0/1286548003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 msgr2=0x561367e105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.903+0000 7fcbf2831740 1 --2- 172.21.15.33:0/1286548003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e105b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009c90 tx=0x7fcbd4011810 comp rx=0 tx=0).stop 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.904+0000 7fcbf2831740 1 -- 172.21.15.33:0/1286548003 shutdown_connections 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.904+0000 7fcbf2831740 1 --2- 172.21.15.33:0/1286548003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e105b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.904+0000 7fcbf2831740 1 --2- 172.21.15.33:0/1286548003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e03900 0x561367e0dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.904+0000 7fcbf2831740 1 --2- 172.21.15.33:0/1286548003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e02fe0 0x561367e033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.904+0000 7fcbf2831740 1 -- 172.21.15.33:0/1286548003 >> 172.21.15.33:0/1286548003 conn(0x561367dff4f0 msgr2=0x561367dff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:38.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.905+0000 7fcbe5600640 1 -- 172.21.15.33:0/1177751088 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fcbdc0053e0 con 0x7fcbe8104970 2024-04-26T11:32:38.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.905+0000 7fcbf2831740 1 -- 172.21.15.33:0/1286548003 shutdown_connections 2024-04-26T11:32:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.907+0000 7fcbe5600640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fcbd00a3c70 0x7fcbd00a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.907+0000 7fcbe5600640 1 -- 172.21.15.33:0/1177751088 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcbdc0efe60 con 0x7fcbe8104970 2024-04-26T11:32:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.908+0000 7fcbf2831740 1 -- 172.21.15.33:0/1286548003 wait complete. 2024-04-26T11:32:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.908+0000 7fcbf2831740 1 -- start start 2024-04-26T11:32:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.908+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fcbd00a3c70 0x7fcbd00a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbf2831740 1 Processor -- start 2024-04-26T11:32:38.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fcbd00a3c70 0x7fcbd00a6130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4024040 tx=0x7fcbd401af10 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:38.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e02fe0 0x561367e0b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e0b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e0bad0 0x561367e1a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbf2831740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561367e10e40 con 0x561367e0bad0 2024-04-26T11:32:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbf2831740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561367cefc60 con 0x561367e02fe0 2024-04-26T11:32:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.909+0000 7fcbf2831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561367e10fc0 con 0x561367e0e1c0 2024-04-26T11:32:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbf2831740 10 client.0 _ll_register_callbacks cb 0x561367d1be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbed000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e02fe0 0x561367e0b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbed000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e02fe0 0x561367e0b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57206/0 (socket says 172.21.15.33:57206) 2024-04-26T11:32:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbed000640 1 -- 172.21.15.33:0/3753193827 learned_addr learned my addr 172.21.15.33:0/3753193827 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e0b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbeda00640 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e0bad0 0x561367e1a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbe7e00640 1 -- 172.21.15.33:0/3753193827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e02fe0 msgr2=0x561367e0b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e02fe0 0x561367e0b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbe7e00640 1 -- 172.21.15.33:0/3753193827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e0bad0 msgr2=0x561367e1a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.910+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e0bad0 0x561367e1a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.911+0000 7fcbe7e00640 1 -- 172.21.15.33:0/3753193827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561367cfe510 con 0x561367e0e1c0 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.911+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e0b590 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcbd401afd0 tx=0x7fcbd401ae70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f0e982272130993 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.911+0000 7fcbcea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:32:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.912+0000 7fcbcea00640 1 -- 172.21.15.33:0/3753193827 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd403d060 con 0x561367e0e1c0 2024-04-26T11:32:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.912+0000 7fcbcea00640 1 -- 172.21.15.33:0/3753193827 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcbd402e470 con 0x561367e0e1c0 2024-04-26T11:32:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.912+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561367d75f90 con 0x561367e0e1c0 2024-04-26T11:32:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.912+0000 7fcbcea00640 1 -- 172.21.15.33:0/3753193827 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbd402e670 con 0x561367e0e1c0 2024-04-26T11:32:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.912+0000 7fcbf2831740 10 client.25459 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.912+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561367a7a210 con 0x561367e0e1c0 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.913+0000 7fcbcea00640 1 -- 172.21.15.33:0/3753193827 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561367d75f90 con 0x561367e0e1c0 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbcea00640 1 -- 172.21.15.33:0/3753193827 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fcbd4022820 con 0x561367e0e1c0 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbf2831740 10 client.25459 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbf2831740 10 client.25459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbf2831740 20 client.25459 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e1e0a0 0x561367e20490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbf2831740 4 client.25459 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561367d6fe00 con 0x561367e1e0a0 2024-04-26T11:32:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.914+0000 7fcbed000640 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e1e0a0 0x561367e20490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.915+0000 7fcbed000640 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e1e0a0 0x561367e20490 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fc2ced53e292793 server_cookie=8a5ab19067169ebb in_seq=0 out_seq=0 2024-04-26T11:32:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.915+0000 7fcbcea00640 10 client.25459 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.916+0000 7fcbcea00640 1 -- 172.21.15.33:0/3753193827 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561367d6fe00 con 0x561367e1e0a0 2024-04-26T11:32:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.916+0000 7fcbcea00640 10 client.25459 handle_command_reply: tid=0 2024-04-26T11:32:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.916+0000 7fcbf2831740 1 client.25459 shutdown 2024-04-26T11:32:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 msgr2=0x561367e0b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e0b590 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcbd401afd0 tx=0x7fcbd401ae70 comp rx=0 tx=0).stop 2024-04-26T11:32:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 shutdown_connections 2024-04-26T11:32:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e0b590 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e1e0a0 0x561367e20490 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e0bad0 0x561367e1a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3753193827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e02fe0 0x561367e0b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 >> 172.21.15.33:0/3753193827 conn(0x561367dff4f0 msgr2=0x561367e05220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.917+0000 7fcbcea00640 0 client.25459 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.918+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 shutdown_connections 2024-04-26T11:32:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.918+0000 7fcbf2831740 1 -- 172.21.15.33:0/3753193827 wait complete. 2024-04-26T11:32:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.918+0000 7fcbf2831740 20 client.25459 trim_cache size 0 max 16384 2024-04-26T11:32:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 Processor -- start 2024-04-26T11:32:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 -- start start 2024-04-26T11:32:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e0cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0bad0 0x561367e25b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e26060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe81aeb10 con 0x561367e02fe0 2024-04-26T11:32:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe81ac3e0 con 0x561367e0bad0 2024-04-26T11:32:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.944+0000 7fcbf2831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe81aa230 con 0x561367e0e1c0 2024-04-26T11:32:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.945+0000 7fcbe7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0bad0 0x561367e25b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.945+0000 7fcbe7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0bad0 0x561367e25b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57222/0 (socket says 172.21.15.33:57222) 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.945+0000 7fcbe7e00640 1 -- 172.21.15.33:0/3958845167 learned_addr learned my addr 172.21.15.33:0/3958845167 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.945+0000 7fcbeda00640 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e26060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.946+0000 7fcbe7e00640 1 -- 172.21.15.33:0/3958845167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 msgr2=0x561367e26060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.946+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e26060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.946+0000 7fcbe7e00640 1 -- 172.21.15.33:0/3958845167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 msgr2=0x561367e0cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.946+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e0cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.946+0000 7fcbe7e00640 1 -- 172.21.15.33:0/3958845167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561367b46620 con 0x561367e0bad0 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.946+0000 7fcbeda00640 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e26060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:32:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.946+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0bad0 0x561367e25b20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4011850 tx=0x7fcbd40099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b002a272b9e64e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.947+0000 7fcbd9600640 1 -- 172.21.15.33:0/3958845167 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd403d070 con 0x561367e0bad0 2024-04-26T11:32:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.947+0000 7fcbd9600640 1 -- 172.21.15.33:0/3958845167 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd4022e10 con 0x561367e0bad0 2024-04-26T11:32:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.947+0000 7fcbd9600640 1 -- 172.21.15.33:0/3958845167 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4047080 con 0x561367e0bad0 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 -- 172.21.15.33:0/3958845167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0bad0 msgr2=0x561367e25b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0bad0 0x561367e25b20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4011850 tx=0x7fcbd40099b0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 -- 172.21.15.33:0/3958845167 shutdown_connections 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0e1c0 0x561367e26060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0bad0 0x561367e25b20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 --2- 172.21.15.33:0/3958845167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e0cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 -- 172.21.15.33:0/3958845167 >> 172.21.15.33:0/3958845167 conn(0x561367dff4f0 msgr2=0x561367e0c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 -- 172.21.15.33:0/3958845167 shutdown_connections 2024-04-26T11:32:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.948+0000 7fcbf2831740 1 -- 172.21.15.33:0/3958845167 wait complete. 2024-04-26T11:32:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.949+0000 7fcbf2831740 1 -- start start 2024-04-26T11:32:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.949+0000 7fcbf2831740 1 Processor -- start 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e166f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0bad0 0x561367e11a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbf2831740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e11fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbf2831740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe81aa8a0 con 0x561367e02fe0 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbf2831740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe81abd10 con 0x561367e0e1c0 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbf2831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe81aaf70 con 0x561367e0bad0 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbf2831740 10 client.0 _ll_register_callbacks cb 0x7fcbd4009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbe7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0bad0 0x561367e11a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbeda00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e11fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbe7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0bad0 0x561367e11a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52268/0 (socket says 172.21.15.33:52268) 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbe7e00640 1 -- 172.21.15.33:0/4002955174 learned_addr learned my addr 172.21.15.33:0/4002955174 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbeda00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e11fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57236/0 (socket says 172.21.15.33:57236) 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.950+0000 7fcbed000640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e166f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.951+0000 7fcbe7e00640 1 -- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 msgr2=0x561367e11fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.951+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e11fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.951+0000 7fcbe7e00640 1 -- 172.21.15.33:0/4002955174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 msgr2=0x561367e166f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.952+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e166f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.952+0000 7fcbe7e00640 1 -- 172.21.15.33:0/4002955174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561367d22cb0 con 0x561367e0bad0 2024-04-26T11:32:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.952+0000 7fcbeda00640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e11fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.952+0000 7fcbed000640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.954+0000 7fcbe7e00640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0bad0 0x561367e11a60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006010 tx=0x7fcbd40229a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=942b3f9cf60fdc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.954+0000 7fcbcea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:32:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.954+0000 7fcbcea00640 1 -- 172.21.15.33:0/4002955174 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd403d070 con 0x561367e0bad0 2024-04-26T11:32:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.955+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbd4022e10 con 0x561367e0bad0 2024-04-26T11:32:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.956+0000 7fcbcea00640 1 -- 172.21.15.33:0/4002955174 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcbd4035040 con 0x561367e0bad0 2024-04-26T11:32:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.956+0000 7fcbcea00640 1 -- 172.21.15.33:0/4002955174 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbd403d070 con 0x561367e0bad0 2024-04-26T11:32:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.957+0000 7fcbf2831740 10 client.25465 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.957+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561367e95260 con 0x561367e0bad0 2024-04-26T11:32:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.958+0000 7fcbcea00640 1 -- 172.21.15.33:0/4002955174 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd4022e10 con 0x561367e0bad0 2024-04-26T11:32:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.958+0000 7fcbcea00640 1 -- 172.21.15.33:0/4002955174 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fcbd401a420 con 0x561367e0bad0 2024-04-26T11:32:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.958+0000 7fcbf2831740 10 client.25465 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.959+0000 7fcbf2831740 10 client.25465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.959+0000 7fcbf2831740 20 client.25465 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.959+0000 7fcbf2831740 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e958e0 0x561367e97d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.959+0000 7fcbf2831740 4 client.25465 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:32:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.959+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561367ceb4b0 con 0x561367e958e0 2024-04-26T11:32:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.960+0000 7fcbed000640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e958e0 0x561367e97d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.962+0000 7fcbed000640 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e958e0 0x561367e97d00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7564ebd713dcf1cf server_cookie=4f86841f527fa7c7 in_seq=0 out_seq=0 2024-04-26T11:32:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.962+0000 7fcbcea00640 10 client.25465 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.975+0000 7fcbcea00640 1 -- 172.21.15.33:0/4002955174 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561367ceb4b0 con 0x561367e958e0 2024-04-26T11:32:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.975+0000 7fcbcea00640 10 client.25465 handle_command_reply: tid=0 2024-04-26T11:32:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.975+0000 7fcbf2831740 1 client.25465 shutdown 2024-04-26T11:32:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.975+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0bad0 msgr2=0x561367e11a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.975+0000 7fcbf2831740 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0bad0 0x561367e11a60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006010 tx=0x7fcbd40229a0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 shutdown_connections 2024-04-26T11:32:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561367e0bad0 0x561367e11a60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561367e958e0 0x561367e97d00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561367e0e1c0 0x561367e11fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 --2- 172.21.15.33:0/4002955174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561367e02fe0 0x561367e166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbcea00640 0 client.25465 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 >> 172.21.15.33:0/4002955174 conn(0x561367dff4f0 msgr2=0x561367e0d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 shutdown_connections 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 1 -- 172.21.15.33:0/4002955174 wait complete. 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.976+0000 7fcbf2831740 20 client.25465 trim_cache size 0 max 16384 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:32:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fcbd00a3c70 msgr2=0x7fcbd00a6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fcbd00a3c70 0x7fcbd00a6130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4024040 tx=0x7fcbd401af10 comp rx=0 tx=0).stop 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8104970 msgr2=0x7fcbe819b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8104970 0x7fcbe819b1f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc02a8f0 tx=0x7fcbdc032d40 comp rx=0 tx=0).stop 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 shutdown_connections 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fcbd00a3c70 0x7fcbd00a6130 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbe8105330 0x7fcbe819b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbe810a980 0x7fcbe819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 --2- 172.21.15.33:0/1177751088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbe8104970 0x7fcbe819b1f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 >> 172.21.15.33:0/1177751088 conn(0x7fcbe8100440 msgr2=0x7fcbe8100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 shutdown_connections 2024-04-26T11:32:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:38.977+0000 7fcbee400640 1 -- 172.21.15.33:0/1177751088 wait complete. 2024-04-26T11:32:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:40 smithi033 ceph-mon[27845]: pgmap v71: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 6.2 MiB/s wr, 546 op/s 2024-04-26T11:32:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:40 smithi096 ceph-mon[32753]: pgmap v71: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 6.2 MiB/s wr, 546 op/s 2024-04-26T11:32:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:40 smithi194 ceph-mon[33017]: pgmap v71: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 6.2 MiB/s wr, 546 op/s 2024-04-26T11:32:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 85.0 s, 2492.4 tps, lat 40.150 ms stddev 42.644 2024-04-26T11:32:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:42 smithi033 ceph-mon[27845]: pgmap v72: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 5.9 MiB/s wr, 505 op/s 2024-04-26T11:32:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:42 smithi096 ceph-mon[32753]: pgmap v72: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 5.9 MiB/s wr, 505 op/s 2024-04-26T11:32:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:42 smithi194 ceph-mon[33017]: pgmap v72: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 5.9 MiB/s wr, 505 op/s 2024-04-26T11:32:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:44 smithi033 ceph-mon[27845]: pgmap v73: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 5.9 MiB/s wr, 505 op/s 2024-04-26T11:32:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:44 smithi096 ceph-mon[32753]: pgmap v73: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 5.9 MiB/s wr, 505 op/s 2024-04-26T11:32:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:44 smithi194 ceph-mon[33017]: pgmap v73: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 5.9 MiB/s wr, 505 op/s 2024-04-26T11:32:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:46 smithi033 ceph-mon[27845]: pgmap v74: 97 pgs: 97 active+clean; 1.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 8.7 MiB/s wr, 791 op/s 2024-04-26T11:32:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:46 smithi096 ceph-mon[32753]: pgmap v74: 97 pgs: 97 active+clean; 1.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 8.7 MiB/s wr, 791 op/s 2024-04-26T11:32:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:46 smithi194 ceph-mon[33017]: pgmap v74: 97 pgs: 97 active+clean; 1.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 8.7 MiB/s wr, 791 op/s 2024-04-26T11:32:47.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 90.0 s, 2863.0 tps, lat 34.801 ms stddev 35.866 2024-04-26T11:32:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:48 smithi033 ceph-mon[27845]: pgmap v75: 97 pgs: 97 active+clean; 1.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.4 MiB/s wr, 755 op/s 2024-04-26T11:32:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:48 smithi096 ceph-mon[32753]: pgmap v75: 97 pgs: 97 active+clean; 1.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.4 MiB/s wr, 755 op/s 2024-04-26T11:32:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:48 smithi194 ceph-mon[33017]: pgmap v75: 97 pgs: 97 active+clean; 1.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.4 MiB/s wr, 755 op/s 2024-04-26T11:32:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:50 smithi033 ceph-mon[27845]: pgmap v76: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-26T11:32:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:50 smithi096 ceph-mon[32753]: pgmap v76: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-26T11:32:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:50 smithi194 ceph-mon[33017]: pgmap v76: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-26T11:32:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 -- 172.21.15.33:0/2952474999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca14105510 msgr2=0x7fca141119d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 --2- 172.21.15.33:0/2952474999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca14105510 0x7fca141119d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009cd0 tx=0x7fca0c02a8f0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 -- 172.21.15.33:0/2952474999 shutdown_connections 2024-04-26T11:32:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 --2- 172.21.15.33:0/2952474999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca14105510 0x7fca141119d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 --2- 172.21.15.33:0/2952474999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca14104b50 0x7fca14104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 --2- 172.21.15.33:0/2952474999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca1410ac10 0x7fca1410aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 -- 172.21.15.33:0/2952474999 >> 172.21.15.33:0/2952474999 conn(0x7fca14100600 msgr2=0x7fca14102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 -- 172.21.15.33:0/2952474999 shutdown_connections 2024-04-26T11:32:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.153+0000 7fca18a00640 1 -- 172.21.15.33:0/2952474999 wait complete. 2024-04-26T11:32:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.155+0000 7fca18a00640 1 Processor -- start 2024-04-26T11:32:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.155+0000 7fca18a00640 1 -- start start 2024-04-26T11:32:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.156+0000 7fca18a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca14104b50 0x7fca14074050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.156+0000 7fca18a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca14105510 0x7fca14074590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.156+0000 7fca18a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca1410ac10 0x7fca1406e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.156+0000 7fca18a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fca1406cc10 con 0x7fca14104b50 2024-04-26T11:32:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.156+0000 7fca18a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca1406ca90 con 0x7fca14105510 2024-04-26T11:32:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.156+0000 7fca18a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca1406cd90 con 0x7fca1410ac10 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca14104b50 0x7fca14074050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca14105510 0x7fca14074590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca14105510 0x7fca14074590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:39570/0 (socket says 172.21.15.33:39570) 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 -- 172.21.15.33:0/1246650476 learned_addr learned my addr 172.21.15.33:0/1246650476 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca12800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca1410ac10 0x7fca1406e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 -- 172.21.15.33:0/1246650476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca1410ac10 msgr2=0x7fca1406e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca1410ac10 0x7fca1406e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 -- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca14104b50 msgr2=0x7fca14074050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca14104b50 0x7fca14074050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca11400640 1 -- 172.21.15.33:0/1246650476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1406ebc0 con 0x7fca14105510 2024-04-26T11:32:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.157+0000 7fca12800640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca1410ac10 0x7fca1406e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:32:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.158+0000 7fca11400640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca14105510 0x7fca14074590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fca0000ea60 tx=0x7fca0000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.158+0000 7fca0aa00640 1 -- 172.21.15.33:0/1246650476 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0000cde0 con 0x7fca14105510 2024-04-26T11:32:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.158+0000 7fca0aa00640 1 -- 172.21.15.33:0/1246650476 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca000035f0 con 0x7fca14105510 2024-04-26T11:32:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.158+0000 7fca0aa00640 1 -- 172.21.15.33:0/1246650476 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00010690 con 0x7fca14105510 2024-04-26T11:32:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.158+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1406ee50 con 0x7fca14105510 2024-04-26T11:32:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.158+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca14076f50 con 0x7fca14105510 2024-04-26T11:32:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.162+0000 7fca1b879740 1 Processor -- start 2024-04-26T11:32:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca0aa00640 1 -- 172.21.15.33:0/1246650476 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fca00004020 con 0x7fca14105510 2024-04-26T11:32:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca1b879740 1 -- start start 2024-04-26T11:32:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca1b879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aa91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca1b879740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca1b879740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541ab3fb0 0x565541ab63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca1b879740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565541a0e380 con 0x565541aa96f0 2024-04-26T11:32:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca1b879740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5655419aac30 con 0x565541ab3fb0 2024-04-26T11:32:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.163+0000 7fca1b879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x565541a0a340 con 0x565541aa8dd0 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541ab3fb0 0x565541ab63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541ab3fb0 0x565541ab63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:39574/0 (socket says 172.21.15.33:39574) 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 -- 172.21.15.33:0/1755060896 learned_addr learned my addr 172.21.15.33:0/1755060896 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11e00640 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca12800640 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aa91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 -- 172.21.15.33:0/1755060896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 msgr2=0x565541aa91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aa91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 -- 172.21.15.33:0/1755060896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 msgr2=0x565541ab3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.164+0000 7fca11400640 1 -- 172.21.15.33:0/1755060896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655419cc940 con 0x565541ab3fb0 2024-04-26T11:32:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.165+0000 7fca0aa00640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc9f80a3dc0 0x7fc9f80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.165+0000 7fca11400640 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541ab3fb0 0x565541ab63a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fca000ce220 tx=0x7fca000d96a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=549dedb9c64e4329 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.165+0000 7fca0aa00640 1 -- 172.21.15.33:0/1246650476 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fca000d0be0 con 0x7fca14105510 2024-04-26T11:32:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.166+0000 7fca12800640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc9f80a3dc0 0x7fc9f80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.166+0000 7fc9ff000640 1 -- 172.21.15.33:0/1755060896 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000e0020 con 0x565541ab3fb0 2024-04-26T11:32:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.166+0000 7fc9ff000640 1 -- 172.21.15.33:0/1755060896 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca00004310 con 0x565541ab3fb0 2024-04-26T11:32:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.167+0000 7fc9ff000640 1 -- 172.21.15.33:0/1755060896 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0009c900 con 0x565541ab3fb0 2024-04-26T11:32:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.168+0000 7fca1b879740 1 -- 172.21.15.33:0/1755060896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541ab3fb0 msgr2=0x565541ab63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.168+0000 7fca1b879740 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541ab3fb0 0x565541ab63a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fca000ce220 tx=0x7fca000d96a0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca12800640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc9f80a3dc0 0x7fc9f80a6280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fca0c004af0 tx=0x7fca0c004a40 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca1b879740 1 -- 172.21.15.33:0/1755060896 shutdown_connections 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca1b879740 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541ab3fb0 0x565541ab63a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca1b879740 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca1b879740 1 --2- 172.21.15.33:0/1755060896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aa91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca1b879740 1 -- 172.21.15.33:0/1755060896 >> 172.21.15.33:0/1755060896 conn(0x565541aa52e0 msgr2=0x565541aa56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca1b879740 1 -- 172.21.15.33:0/1755060896 shutdown_connections 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.169+0000 7fca1b879740 1 -- 172.21.15.33:0/1755060896 wait complete. 2024-04-26T11:32:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 -- start start 2024-04-26T11:32:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 Processor -- start 2024-04-26T11:32:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa8dd0 0x565541ab0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541ab19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x565541ab6be0 con 0x565541aa96f0 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x565541995c60 con 0x565541aa8dd0 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x565541ab6d60 con 0x565541ab3fb0 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.170+0000 7fca1b879740 10 client.0 _ll_register_callbacks cb 0x5655419c1e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541ab19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541ab19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:48828/0 (socket says 172.21.15.33:48828) 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca11e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa8dd0 0x565541ab0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 -- 172.21.15.33:0/2610286123 learned_addr learned my addr 172.21.15.33:0/2610286123 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca11400640 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 -- 172.21.15.33:0/2610286123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa8dd0 msgr2=0x565541ab0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa8dd0 0x565541ab0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 -- 172.21.15.33:0/2610286123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 msgr2=0x565541ab14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.171+0000 7fca12800640 1 -- 172.21.15.33:0/2610286123 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5655419a4510 con 0x565541ab3fb0 2024-04-26T11:32:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.172+0000 7fca12800640 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541ab19e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009cd0 tx=0x7fca0c02a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4602bf7a24c815ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.172+0000 7fc9f3400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.172+0000 7fc9f3400640 1 -- 172.21.15.33:0/2610286123 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c003970 con 0x565541ab3fb0 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.172+0000 7fc9f3400640 1 -- 172.21.15.33:0/2610286123 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fca0c003b10 con 0x565541ab3fb0 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.173+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565541a1bf90 con 0x565541ab3fb0 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.173+0000 7fc9f3400640 1 -- 172.21.15.33:0/2610286123 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fca0c003d10 con 0x565541ab3fb0 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.173+0000 7fca1b879740 10 client.25477 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.173+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565541720210 con 0x565541ab3fb0 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fc9f3400640 1 -- 172.21.15.33:0/2610286123 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x565541a1bf90 con 0x565541ab3fb0 2024-04-26T11:32:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fc9f3400640 1 -- 172.21.15.33:0/2610286123 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fca0c053940 con 0x565541ab3fb0 2024-04-26T11:32:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fca1b879740 10 client.25477 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fca1b879740 10 client.25477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fca1b879740 20 client.25477 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fca1b879740 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541ac3d60 0x565541ac6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fca1b879740 4 client.25477 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:32:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.174+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565541a15e00 con 0x565541ac3d60 2024-04-26T11:32:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.175+0000 7fca11e00640 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541ac3d60 0x565541ac6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.177+0000 7fca11e00640 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541ac3d60 0x565541ac6150 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3884f5ca6036694f server_cookie=c9d6032cdb805188 in_seq=0 out_seq=0 2024-04-26T11:32:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.177+0000 7fc9f3400640 10 client.25477 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.177+0000 7fc9f3400640 1 -- 172.21.15.33:0/2610286123 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565541a15e00 con 0x565541ac3d60 2024-04-26T11:32:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.177+0000 7fc9f3400640 10 client.25477 handle_command_reply: tid=0 2024-04-26T11:32:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.177+0000 7fca1b879740 1 client.25477 shutdown 2024-04-26T11:32:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 msgr2=0x565541ab19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541ab19e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009cd0 tx=0x7fca0c02a8f0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 shutdown_connections 2024-04-26T11:32:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541ab19e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541ac3d60 0x565541ac6150 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa96f0 0x565541ab14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 --2- 172.21.15.33:0/2610286123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa8dd0 0x565541ab0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 >> 172.21.15.33:0/2610286123 conn(0x565541aa52e0 msgr2=0x565541aab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fc9f3400640 0 client.25477 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 shutdown_connections 2024-04-26T11:32:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.178+0000 7fca1b879740 1 -- 172.21.15.33:0/2610286123 wait complete. 2024-04-26T11:32:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.179+0000 7fca1b879740 20 client.25477 trim_cache size 0 max 16384 2024-04-26T11:32:52.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 95.0 s, 3051.4 tps, lat 32.856 ms stddev 37.151 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 Processor -- start 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 -- start start 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa8dd0 0x565541abfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aa56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541acba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fca141b7540 con 0x565541aa8dd0 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca141b4e10 con 0x565541aa96f0 2024-04-26T11:32:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.218+0000 7fca1b879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca141b2cc0 con 0x565541ab3fb0 2024-04-26T11:32:52.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aa56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aa56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:39606/0 (socket says 172.21.15.33:39606) 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca12800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541acba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 -- 172.21.15.33:0/1458099713 learned_addr learned my addr 172.21.15.33:0/1458099713 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11e00640 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa8dd0 0x565541abfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 -- 172.21.15.33:0/1458099713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 msgr2=0x565541acba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541acba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 -- 172.21.15.33:0/1458099713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa8dd0 msgr2=0x565541abfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa8dd0 0x565541abfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca11400640 1 -- 172.21.15.33:0/1458099713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565541aaff70 con 0x565541aa96f0 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.219+0000 7fca12800640 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541acba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:32:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.220+0000 7fca11e00640 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa8dd0 0x565541abfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.220+0000 7fca11400640 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aa56c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca000d9ba0 tx=0x7fca00003e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd882ca098366b35 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.220+0000 7fc9fe600640 1 -- 172.21.15.33:0/1458099713 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0009de60 con 0x565541aa96f0 2024-04-26T11:32:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.220+0000 7fc9fe600640 1 -- 172.21.15.33:0/1458099713 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca000d44e0 con 0x565541aa96f0 2024-04-26T11:32:52.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.220+0000 7fc9fe600640 1 -- 172.21.15.33:0/1458099713 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000e0040 con 0x565541aa96f0 2024-04-26T11:32:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.224+0000 7fca1b879740 1 -- 172.21.15.33:0/1458099713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 msgr2=0x565541aa56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.224+0000 7fca1b879740 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aa56c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca000d9ba0 tx=0x7fca00003e90 comp rx=0 tx=0).stop 2024-04-26T11:32:52.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.225+0000 7fca1b879740 1 -- 172.21.15.33:0/1458099713 shutdown_connections 2024-04-26T11:32:52.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.225+0000 7fca1b879740 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541ab3fb0 0x565541acba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.225+0000 7fca1b879740 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aa56c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.225+0000 7fca1b879740 1 --2- 172.21.15.33:0/1458099713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541aa8dd0 0x565541abfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.225+0000 7fca1b879740 1 -- 172.21.15.33:0/1458099713 >> 172.21.15.33:0/1458099713 conn(0x565541aa52e0 msgr2=0x565541aa4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.225+0000 7fca1b879740 1 -- 172.21.15.33:0/1458099713 shutdown_connections 2024-04-26T11:32:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.225+0000 7fca1b879740 1 -- 172.21.15.33:0/1458099713 wait complete. 2024-04-26T11:32:52.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.226+0000 7fca1b879740 1 -- start start 2024-04-26T11:32:52.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.227+0000 7fca1b879740 1 Processor -- start 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca1b879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aadf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca1b879740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aae470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca1b879740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541ab3fb0 0x565541aae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca1b879740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fca141b3310 con 0x565541ab3fb0 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca1b879740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca141b4740 con 0x565541aa96f0 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca1b879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca141b39a0 con 0x565541aa8dd0 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca1b879740 10 client.0 _ll_register_callbacks cb 0x7fca04005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca11e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aadf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca11400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca11e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aadf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:48842/0 (socket says 172.21.15.33:48842) 2024-04-26T11:32:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca11e00640 1 -- 172.21.15.33:0/790671467 learned_addr learned my addr 172.21.15.33:0/790671467 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:32:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.228+0000 7fca12800640 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541ab3fb0 0x565541aae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.229+0000 7fca11400640 1 -- 172.21.15.33:0/790671467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 msgr2=0x565541aadf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.229+0000 7fca11400640 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aadf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.229+0000 7fca11400640 1 -- 172.21.15.33:0/790671467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541ab3fb0 msgr2=0x565541aae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:32:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.229+0000 7fca11400640 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541ab3fb0 0x565541aae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.229+0000 7fca11400640 1 -- 172.21.15.33:0/790671467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565541aabd80 con 0x565541aa96f0 2024-04-26T11:32:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.229+0000 7fca11e00640 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aadf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:32:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.230+0000 7fca11400640 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aae470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c760 tx=0x7fca00098e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b499bbb0dfa1fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:32:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.230+0000 7fc9f3400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:32:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.230+0000 7fc9f3400640 1 -- 172.21.15.33:0/790671467 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000e0070 con 0x565541aa96f0 2024-04-26T11:32:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.230+0000 7fc9f3400640 1 -- 172.21.15.33:0/790671467 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fca000bd0b0 con 0x565541aa96f0 2024-04-26T11:32:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.230+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca000d44e0 con 0x565541aa96f0 2024-04-26T11:32:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.231+0000 7fc9f3400640 1 -- 172.21.15.33:0/790671467 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fca0009de60 con 0x565541aa96f0 2024-04-26T11:32:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.232+0000 7fca1b879740 10 client.25517 fetch_fsmap learned FSMap version 28 2024-04-26T11:32:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.232+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5655417ec620 con 0x565541aa96f0 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fc9f3400640 1 -- 172.21.15.33:0/790671467 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca000d44e0 con 0x565541aa96f0 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fc9f3400640 1 -- 172.21.15.33:0/790671467 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fca000ce560 con 0x565541aa96f0 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fca1b879740 10 client.25517 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fca1b879740 10 client.25517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fca1b879740 20 client.25517 populate_metadata read hostname 'smithi033' 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fca1b879740 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541b2a760 0x565541b2cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fca1b879740 4 client.25517 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.233+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5655419914b0 con 0x565541b2a760 2024-04-26T11:32:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.234+0000 7fca11e00640 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541b2a760 0x565541b2cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:32:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.234+0000 7fc9f3400640 10 client.25517 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.235+0000 7fca11e00640 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541b2a760 0x565541b2cb50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698ca523fb2adceb server_cookie=81ce704aa924e5ba in_seq=0 out_seq=0 2024-04-26T11:32:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.249+0000 7fc9f3400640 1 -- 172.21.15.33:0/790671467 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5655419914b0 con 0x565541b2a760 2024-04-26T11:32:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.249+0000 7fc9f3400640 10 client.25517 handle_command_reply: tid=0 2024-04-26T11:32:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.250+0000 7fca1b879740 1 client.25517 shutdown 2024-04-26T11:32:52.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.250+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 msgr2=0x565541aae470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.250+0000 7fca1b879740 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aae470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c760 tx=0x7fca00098e70 comp rx=0 tx=0).stop 2024-04-26T11:32:52.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.251+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 shutdown_connections 2024-04-26T11:32:52.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.251+0000 7fca1b879740 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x565541b2a760 0x565541b2cb50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.251+0000 7fca1b879740 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x565541ab3fb0 0x565541aae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.251+0000 7fca1b879740 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x565541aa96f0 0x565541aae470 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.251+0000 7fca1b879740 1 --2- 172.21.15.33:0/790671467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565541aa8dd0 0x565541aadf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.251+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 >> 172.21.15.33:0/790671467 conn(0x565541aa52e0 msgr2=0x565541aa4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.251+0000 7fc9f3400640 0 client.25517 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:32:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.252+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 shutdown_connections 2024-04-26T11:32:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.252+0000 7fca1b879740 1 -- 172.21.15.33:0/790671467 wait complete. 2024-04-26T11:32:52.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.252+0000 7fca1b879740 20 client.25517 trim_cache size 0 max 16384 2024-04-26T11:32:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:32:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:32:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:32:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.253+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc9f80a3dc0 msgr2=0x7fc9f80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.253+0000 7fca18a00640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc9f80a3dc0 0x7fc9f80a6280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fca0c004af0 tx=0x7fca0c004a40 comp rx=0 tx=0).stop 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.253+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca14105510 msgr2=0x7fca14074590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.253+0000 7fca18a00640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca14105510 0x7fca14074590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fca0000ea60 tx=0x7fca0000ef20 comp rx=0 tx=0).stop 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 shutdown_connections 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc9f80a3dc0 0x7fc9f80a6280 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca1410ac10 0x7fca1406e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca14105510 0x7fca14074590 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 --2- 172.21.15.33:0/1246650476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca14104b50 0x7fca14074050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 >> 172.21.15.33:0/1246650476 conn(0x7fca14100600 msgr2=0x7fca14101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 shutdown_connections 2024-04-26T11:32:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:32:52.254+0000 7fca18a00640 1 -- 172.21.15.33:0/1246650476 wait complete. 2024-04-26T11:32:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:52 smithi096 ceph-mon[32753]: pgmap v77: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-26T11:32:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:52 smithi194 ceph-mon[33017]: pgmap v77: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-26T11:32:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:52 smithi033 ceph-mon[27845]: pgmap v77: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-26T11:32:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:54 smithi096 ceph-mon[32753]: pgmap v78: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-26T11:32:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:54 smithi194 ceph-mon[33017]: pgmap v78: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-26T11:32:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:54 smithi033 ceph-mon[27845]: pgmap v78: 97 pgs: 97 active+clean; 1.0 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-26T11:32:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:56 smithi194 ceph-mon[33017]: pgmap v79: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 10 MiB/s wr, 902 op/s 2024-04-26T11:32:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:56 smithi033 ceph-mon[27845]: pgmap v79: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 10 MiB/s wr, 902 op/s 2024-04-26T11:32:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:56 smithi096 ceph-mon[32753]: pgmap v79: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 10 MiB/s wr, 902 op/s 2024-04-26T11:32:57.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 100.0 s, 2943.5 tps, lat 33.937 ms stddev 36.323 2024-04-26T11:32:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:32:58 smithi194 ceph-mon[33017]: pgmap v80: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.2 MiB/s wr, 616 op/s 2024-04-26T11:32:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:32:58 smithi033 ceph-mon[27845]: pgmap v80: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.2 MiB/s wr, 616 op/s 2024-04-26T11:32:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:32:58 smithi096 ceph-mon[32753]: pgmap v80: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.2 MiB/s wr, 616 op/s 2024-04-26T11:33:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:00 smithi194 ceph-mon[33017]: pgmap v81: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 10 MiB/s wr, 904 op/s 2024-04-26T11:33:00.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:00 smithi033 ceph-mon[27845]: pgmap v81: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 10 MiB/s wr, 904 op/s 2024-04-26T11:33:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:00 smithi096 ceph-mon[32753]: pgmap v81: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 10 MiB/s wr, 904 op/s 2024-04-26T11:33:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 105.0 s, 1784.5 tps, lat 51.363 ms stddev 68.785 2024-04-26T11:33:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:02 smithi194 ceph-mon[33017]: pgmap v82: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 604 op/s 2024-04-26T11:33:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:02 smithi033 ceph-mon[27845]: pgmap v82: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 604 op/s 2024-04-26T11:33:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:02 smithi096 ceph-mon[32753]: pgmap v82: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 604 op/s 2024-04-26T11:33:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:04 smithi194 ceph-mon[33017]: pgmap v83: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 604 op/s 2024-04-26T11:33:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:04 smithi033 ceph-mon[27845]: pgmap v83: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 604 op/s 2024-04-26T11:33:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:04 smithi096 ceph-mon[32753]: pgmap v83: 97 pgs: 97 active+clean; 1.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 604 op/s 2024-04-26T11:33:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.445+0000 7f8e6c000640 1 -- 172.21.15.33:0/3064164716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e641137b0 msgr2=0x7f8e64115ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.445+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3064164716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e641137b0 0x7f8e64115ba0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009ca0 tx=0x7f8e5c02a8e0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.446+0000 7f8e6c000640 1 -- 172.21.15.33:0/3064164716 shutdown_connections 2024-04-26T11:33:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.446+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3064164716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e641137b0 0x7f8e64115ba0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.446+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3064164716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64077780 0x7f8e64113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.446+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3064164716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e64076e60 0x7f8e64077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.446+0000 7f8e6c000640 1 -- 172.21.15.33:0/3064164716 >> 172.21.15.33:0/3064164716 conn(0x7f8e64100620 msgr2=0x7f8e64102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.447+0000 7f8e6c000640 1 -- 172.21.15.33:0/3064164716 shutdown_connections 2024-04-26T11:33:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.447+0000 7f8e6c000640 1 -- 172.21.15.33:0/3064164716 wait complete. 2024-04-26T11:33:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.448+0000 7f8e6c000640 1 Processor -- start 2024-04-26T11:33:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.448+0000 7f8e6c000640 1 -- start start 2024-04-26T11:33:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e6c000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64076e60 0x7f8e6419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e6c000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e64077780 0x7f8e6419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e6c000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e641137b0 0x7f8e641a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e69600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64076e60 0x7f8e6419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e69600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64076e60 0x7f8e6419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:46504/0 (socket says 172.21.15.33:46504) 2024-04-26T11:33:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e6c000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e64119220 con 0x7f8e64077780 2024-04-26T11:33:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e6c000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8e641190a0 con 0x7f8e64076e60 2024-04-26T11:33:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.449+0000 7f8e6c000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8e641193a0 con 0x7f8e641137b0 2024-04-26T11:33:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e641137b0 0x7f8e641a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e69600640 1 -- 172.21.15.33:0/3710111971 learned_addr learned my addr 172.21.15.33:0/3710111971 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e68c00640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e64077780 0x7f8e6419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e69600640 1 -- 172.21.15.33:0/3710111971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e641137b0 msgr2=0x7f8e641a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e69600640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e641137b0 0x7f8e641a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e69600640 1 -- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e64077780 msgr2=0x7f8e6419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e69600640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e64077780 0x7f8e6419fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.450+0000 7f8e69600640 1 -- 172.21.15.33:0/3710111971 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e641a46b0 con 0x7f8e64076e60 2024-04-26T11:33:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.451+0000 7f8e68c00640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e64077780 0x7f8e6419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:33:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.451+0000 7f8e69600640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64076e60 0x7f8e6419f6c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800d950 tx=0x7f8e5800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.451+0000 7f8e62000640 1 -- 172.21.15.33:0/3710111971 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e58017070 con 0x7f8e64076e60 2024-04-26T11:33:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.451+0000 7f8e62000640 1 -- 172.21.15.33:0/3710111971 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e58009c30 con 0x7f8e64076e60 2024-04-26T11:33:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.451+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e641a49a0 con 0x7f8e64076e60 2024-04-26T11:33:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.452+0000 7f8e62000640 1 -- 172.21.15.33:0/3710111971 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5800f620 con 0x7f8e64076e60 2024-04-26T11:33:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.452+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e641ac230 con 0x7f8e64076e60 2024-04-26T11:33:05.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 Processor -- start 2024-04-26T11:33:05.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 -- start start 2024-04-26T11:33:05.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bcf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bdc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3c8b34380 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d3c8ad0c30 con 0x55d3c8bcedd0 2024-04-26T11:33:05.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.471+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d3c8b30340 con 0x55d3c8bcf6f0 2024-04-26T11:33:05.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.472+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bcf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.472+0000 7f8e62000640 1 -- 172.21.15.33:0/3710111971 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8e58010460 con 0x7f8e64076e60 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.472+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bcf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:46518/0 (socket says 172.21.15.33:46518) 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.472+0000 7f8e6a000640 1 -- 172.21.15.33:0/4072411809 learned_addr learned my addr 172.21.15.33:0/4072411809 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.472+0000 7f8e69600640 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.473+0000 7f8e6a000640 1 -- 172.21.15.33:0/4072411809 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 msgr2=0x55d3c8bd9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.473+0000 7f8e6a000640 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.473+0000 7f8e6a000640 1 -- 172.21.15.33:0/4072411809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bd9fb0 msgr2=0x55d3c8bdc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.473+0000 7f8e6a000640 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bdc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.473+0000 7f8e6a000640 1 -- 172.21.15.33:0/4072411809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c8af2940 con 0x55d3c8bcedd0 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.473+0000 7f8e6a000640 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bcf1b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009ca0 tx=0x7f8e5c0315b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ad071b94a73ddd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.474+0000 7f8e56600640 1 -- 172.21.15.33:0/4072411809 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c003b70 con 0x55d3c8bcedd0 2024-04-26T11:33:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.474+0000 7f8e56600640 1 -- 172.21.15.33:0/4072411809 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e5c036040 con 0x55d3c8bcedd0 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.474+0000 7f8e56600640 1 -- 172.21.15.33:0/4072411809 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c004580 con 0x55d3c8bcedd0 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.475+0000 7f8e6edbf740 1 -- 172.21.15.33:0/4072411809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcedd0 msgr2=0x55d3c8bcf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.475+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bcf1b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009ca0 tx=0x7f8e5c0315b0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.475+0000 7f8e6edbf740 1 -- 172.21.15.33:0/4072411809 shutdown_connections 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.475+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.475+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bdc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.475+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/4072411809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bcf1b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.475+0000 7f8e6edbf740 1 -- 172.21.15.33:0/4072411809 >> 172.21.15.33:0/4072411809 conn(0x55d3c8bcb2e0 msgr2=0x55d3c8bcb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.477+0000 7f8e6edbf740 1 -- 172.21.15.33:0/4072411809 shutdown_connections 2024-04-26T11:33:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.477+0000 7f8e6edbf740 1 -- 172.21.15.33:0/4072411809 wait complete. 2024-04-26T11:33:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.477+0000 7f8e62000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8e4c0a3da0 0x7f8e4c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.477+0000 7f8e6edbf740 1 -- start start 2024-04-26T11:33:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.478+0000 7f8e62000640 1 -- 172.21.15.33:0/3710111971 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8e580d0310 con 0x7f8e64076e60 2024-04-26T11:33:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.478+0000 7f8e6edbf740 1 Processor -- start 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.478+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bd6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.478+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e69600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bd6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:46530/0 (socket says 172.21.15.33:46530) 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e69600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bd6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57682/0 (socket says 172.21.15.33:57682) 2024-04-26T11:33:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e6a000640 1 -- 172.21.15.33:0/11726429 learned_addr learned my addr 172.21.15.33:0/11726429 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3c8bdcb80 con 0x55d3c8bcf6f0 2024-04-26T11:33:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d3c8abbc60 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.479+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d3c8bdcd00 con 0x55d3c8bcedd0 2024-04-26T11:33:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.480+0000 7f8e6edbf740 10 client.0 _ll_register_callbacks cb 0x55d3c8ae7e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.480+0000 7f8e6a000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8e4c0a3da0 0x7f8e4c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.480+0000 7f8e68c00640 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e69600640 1 -- 172.21.15.33:0/11726429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 msgr2=0x55d3c8bd7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e69600640 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e69600640 1 -- 172.21.15.33:0/11726429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcf6f0 msgr2=0x55d3c8bd73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e69600640 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e69600640 1 -- 172.21.15.33:0/11726429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c8aca510 con 0x55d3c8bcedd0 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e69600640 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bd6e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e580cda70 tx=0x7f8e5809bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114a0845e1a34ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:33:05.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e68c00640 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:33:05.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.481+0000 7f8e4aa00640 1 -- 172.21.15.33:0/11726429 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e58020070 con 0x55d3c8bcedd0 2024-04-26T11:33:05.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.482+0000 7f8e4aa00640 1 -- 172.21.15.33:0/11726429 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8e580d3b70 con 0x55d3c8bcedd0 2024-04-26T11:33:05.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.482+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e5c036040 con 0x55d3c8bcedd0 2024-04-26T11:33:05.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.482+0000 7f8e4aa00640 1 -- 172.21.15.33:0/11726429 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e5809c2c0 con 0x55d3c8bcedd0 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.482+0000 7f8e6edbf740 10 client.25495 fetch_fsmap learned FSMap version 28 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.482+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d3c8846210 con 0x55d3c8bcedd0 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.482+0000 7f8e4aa00640 1 -- 172.21.15.33:0/11726429 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e5c036040 con 0x55d3c8bcedd0 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.482+0000 7f8e4aa00640 1 -- 172.21.15.33:0/11726429 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8e5809b710 con 0x55d3c8bcedd0 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.483+0000 7f8e6edbf740 10 client.25495 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.483+0000 7f8e6edbf740 10 client.25495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.483+0000 7f8e6edbf740 20 client.25495 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.483+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8be9ce0 0x55d3c8bec140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.483+0000 7f8e6edbf740 4 client.25495 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:33:05.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.483+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3c8b41f90 con 0x55d3c8be9ce0 2024-04-26T11:33:05.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.483+0000 7f8e68c00640 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8be9ce0 0x55d3c8bec140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.484+0000 7f8e6a000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8e4c0a3da0 0x7f8e4c0a6260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c004160 tx=0x7f8e5c0041c0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:05.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.484+0000 7f8e68c00640 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8be9ce0 0x55d3c8bec140 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de97865ce191e2b2 server_cookie=2548b2c5fa7fa8c7 in_seq=0 out_seq=0 2024-04-26T11:33:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.484+0000 7f8e4aa00640 10 client.25495 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e4aa00640 1 -- 172.21.15.33:0/11726429 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3c8b41f90 con 0x55d3c8be9ce0 2024-04-26T11:33:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e4aa00640 10 client.25495 handle_command_reply: tid=0 2024-04-26T11:33:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 client.25495 shutdown 2024-04-26T11:33:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcedd0 msgr2=0x55d3c8bd6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bd6e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e580cda70 tx=0x7f8e5809bec0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 shutdown_connections 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8be9ce0 0x55d3c8bec140 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/11726429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcedd0 0x55d3c8bd6e90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.485+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 >> 172.21.15.33:0/11726429 conn(0x55d3c8bcb2e0 msgr2=0x55d3c8bd1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.486+0000 7f8e4aa00640 0 client.25495 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.486+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 shutdown_connections 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.486+0000 7f8e6edbf740 1 -- 172.21.15.33:0/11726429 wait complete. 2024-04-26T11:33:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.486+0000 7f8e6edbf740 20 client.25495 trim_cache size 0 max 16384 2024-04-26T11:33:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.512+0000 7f8e6edbf740 1 Processor -- start 2024-04-26T11:33:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.512+0000 7f8e6edbf740 1 -- start start 2024-04-26T11:33:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e44010fc0 con 0x55d3c8bcedd0 2024-04-26T11:33:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4400e890 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4400c6e0 con 0x55d3c8bcf6f0 2024-04-26T11:33:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e68c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:46536/0 (socket says 172.21.15.33:46536) 2024-04-26T11:33:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e6a000640 1 -- 172.21.15.33:0/2877355355 learned_addr learned my addr 172.21.15.33:0/2877355355 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.513+0000 7f8e69600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.515+0000 7f8e6a000640 1 -- 172.21.15.33:0/2877355355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 msgr2=0x55d3c8bd1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.515+0000 7f8e6a000640 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.515+0000 7f8e6a000640 1 -- 172.21.15.33:0/2877355355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 msgr2=0x55d3c8be5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.515+0000 7f8e6a000640 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.515+0000 7f8e6a000640 1 -- 172.21.15.33:0/2877355355 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c8912620 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.516+0000 7f8e68c00640 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.516+0000 7f8e69600640 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:33:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.516+0000 7f8e6a000640 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c004190 tx=0x7f8e5c02ade0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5132c88df54ffcef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e55c00640 1 -- 172.21.15.33:0/2877355355 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c004250 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e55c00640 1 -- 172.21.15.33:0/2877355355 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e5c036040 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e55c00640 1 -- 172.21.15.33:0/2877355355 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c044430 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e6edbf740 1 -- 172.21.15.33:0/2877355355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 msgr2=0x55d3c8bd7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c004190 tx=0x7f8e5c02ade0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e6edbf740 1 -- 172.21.15.33:0/2877355355 shutdown_connections 2024-04-26T11:33:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8bd1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd7730 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/2877355355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.517+0000 7f8e6edbf740 1 -- 172.21.15.33:0/2877355355 >> 172.21.15.33:0/2877355355 conn(0x55d3c8bcb2e0 msgr2=0x55d3c8bca440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.518+0000 7f8e6edbf740 1 -- 172.21.15.33:0/2877355355 shutdown_connections 2024-04-26T11:33:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.518+0000 7f8e6edbf740 1 -- 172.21.15.33:0/2877355355 wait complete. 2024-04-26T11:33:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.518+0000 7f8e6edbf740 1 -- start start 2024-04-26T11:33:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.518+0000 7f8e6edbf740 1 Processor -- start 2024-04-26T11:33:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8be24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6edbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4400cd50 con 0x55d3c8bcedd0 2024-04-26T11:33:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4400e1c0 con 0x55d3c8bcf6f0 2024-04-26T11:33:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6edbf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4400d420 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6edbf740 10 client.0 _ll_register_callbacks cb 0x7f8e58010cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e68c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8be24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.519+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e69600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57694/0 (socket says 172.21.15.33:57694) 2024-04-26T11:33:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e6a000640 1 -- 172.21.15.33:0/3233905725 learned_addr learned my addr 172.21.15.33:0/3233905725 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e6a000640 1 -- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcf6f0 msgr2=0x55d3c8be24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e6a000640 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8be24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e6a000640 1 -- 172.21.15.33:0/3233905725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 msgr2=0x55d3c8be1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e6a000640 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.520+0000 7f8e6a000640 1 -- 172.21.15.33:0/3233905725 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c8aeecb0 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.521+0000 7f8e68c00640 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8be24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.521+0000 7f8e6a000640 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd36c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c031840 tx=0x7f8e5c032f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=739ec5872afaecdd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.521+0000 7f8e4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:33:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.521+0000 7f8e4aa00640 1 -- 172.21.15.33:0/3233905725 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c06a060 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.522+0000 7f8e4aa00640 1 -- 172.21.15.33:0/3233905725 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8e5c0307e0 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.522+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e5c036040 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.522+0000 7f8e4aa00640 1 -- 172.21.15.33:0/3233905725 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e5c030ed0 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.522+0000 7f8e6edbf740 10 client.25501 fetch_fsmap learned FSMap version 28 2024-04-26T11:33:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.522+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d3c8c4fd40 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.522+0000 7f8e4aa00640 1 -- 172.21.15.33:0/3233905725 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e5c036040 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e4aa00640 1 -- 172.21.15.33:0/3233905725 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8e5c0050d0 con 0x55d3c8bd9fb0 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e6edbf740 10 client.25501 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e6edbf740 10 client.25501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e6edbf740 20 client.25501 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8c50010 0x55d3c8c52430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e6edbf740 4 client.25501 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:33:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d3c8b3be00 con 0x55d3c8c50010 2024-04-26T11:33:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.523+0000 7f8e69600640 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8c50010 0x55d3c8c52430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.524+0000 7f8e69600640 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8c50010 0x55d3c8c52430 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5dd9b9f78b568f0 server_cookie=4eab244d510d7e32 in_seq=0 out_seq=0 2024-04-26T11:33:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.524+0000 7f8e4aa00640 10 client.25501 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.538+0000 7f8e4aa00640 1 -- 172.21.15.33:0/3233905725 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d3c8b3be00 con 0x55d3c8c50010 2024-04-26T11:33:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.538+0000 7f8e4aa00640 10 client.25501 handle_command_reply: tid=0 2024-04-26T11:33:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.538+0000 7f8e6edbf740 1 client.25501 shutdown 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.538+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bd9fb0 msgr2=0x55d3c8bd36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.538+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd36c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c031840 tx=0x7f8e5c032f10 comp rx=0 tx=0).stop 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 shutdown_connections 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3c8bd9fb0 0x55d3c8bd36c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55d3c8c50010 0x55d3c8c52430 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e4aa00640 0 client.25501 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d3c8bcf6f0 0x55d3c8be24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e6edbf740 1 --2- 172.21.15.33:0/3233905725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d3c8bcedd0 0x55d3c8be1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 >> 172.21.15.33:0/3233905725 conn(0x55d3c8bcb2e0 msgr2=0x55d3c8bd8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.539+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 shutdown_connections 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.540+0000 7f8e6edbf740 1 -- 172.21.15.33:0/3233905725 wait complete. 2024-04-26T11:33:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.540+0000 7f8e6edbf740 20 client.25501 trim_cache size 0 max 16384 2024-04-26T11:33:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:33:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:33:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:33:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8e4c0a3da0 msgr2=0x7f8e4c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8e4c0a3da0 0x7f8e4c0a6260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c004160 tx=0x7f8e5c0041c0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64076e60 msgr2=0x7f8e6419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64076e60 0x7f8e6419f6c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800d950 tx=0x7f8e5800de10 comp rx=0 tx=0).stop 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 shutdown_connections 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f8e4c0a3da0 0x7f8e4c0a6260 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e641137b0 0x7f8e641a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e64077780 0x7f8e6419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 --2- 172.21.15.33:0/3710111971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e64076e60 0x7f8e6419f6c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.541+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 >> 172.21.15.33:0/3710111971 conn(0x7f8e64100620 msgr2=0x7f8e64102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.542+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 shutdown_connections 2024-04-26T11:33:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:05.542+0000 7f8e6c000640 1 -- 172.21.15.33:0/3710111971 wait complete. 2024-04-26T11:33:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:06 smithi194 ceph-mon[33017]: pgmap v84: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-26T11:33:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:06 smithi033 ceph-mon[27845]: pgmap v84: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-26T11:33:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:06 smithi096 ceph-mon[32753]: pgmap v84: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-26T11:33:07.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 110.0 s, 2103.8 tps, lat 51.048 ms stddev 130.810 2024-04-26T11:33:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:08 smithi194 ceph-mon[33017]: pgmap v85: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 30 MiB/s wr, 1.74k op/s 2024-04-26T11:33:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:08 smithi033 ceph-mon[27845]: pgmap v85: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 30 MiB/s wr, 1.74k op/s 2024-04-26T11:33:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:08 smithi096 ceph-mon[32753]: pgmap v85: 97 pgs: 97 active+clean; 1.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 30 MiB/s wr, 1.74k op/s 2024-04-26T11:33:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:10 smithi194 ceph-mon[33017]: pgmap v86: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 37 MiB/s wr, 2.28k op/s 2024-04-26T11:33:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:10 smithi033 ceph-mon[27845]: pgmap v86: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 37 MiB/s wr, 2.28k op/s 2024-04-26T11:33:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:10 smithi096 ceph-mon[32753]: pgmap v86: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 37 MiB/s wr, 2.28k op/s 2024-04-26T11:33:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:11 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:11 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:11 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:12.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 115.0 s, 2747.4 tps, lat 36.740 ms stddev 53.127 2024-04-26T11:33:12.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:12 smithi096 ceph-mon[32753]: pgmap v87: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 34 MiB/s wr, 1.99k op/s 2024-04-26T11:33:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:12 smithi194 ceph-mon[33017]: pgmap v87: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 34 MiB/s wr, 1.99k op/s 2024-04-26T11:33:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:12 smithi033 ceph-mon[27845]: pgmap v87: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 34 MiB/s wr, 1.99k op/s 2024-04-26T11:33:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:14 smithi194 ceph-mon[33017]: pgmap v88: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 34 MiB/s wr, 1.99k op/s 2024-04-26T11:33:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:14 smithi033 ceph-mon[27845]: pgmap v88: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 34 MiB/s wr, 1.99k op/s 2024-04-26T11:33:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:14 smithi096 ceph-mon[32753]: pgmap v88: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 34 MiB/s wr, 1.99k op/s 2024-04-26T11:33:15.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ec725a700 time 2024-04-26T11:33:15.067281+0000 2024-04-26T11:33:15.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0ed18833dd] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.434 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.435 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.435 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.435 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: clone() 2024-04-26T11:33:15.435 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.068+0000 7f0ec725a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ec725a700 time 2024-04-26T11:33:15.067281+0000 2024-04-26T11:33:15.435 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:15.435 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.435 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0ed18833dd] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.436 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: clone() 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: *** Caught signal (Aborted) ** 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: in thread 7f0ec725a700 thread_name:ms_dispatch 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ed060bd20] 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: gsignal() 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: abort() 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0ed1883437] 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.437 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 16: clone() 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.069+0000 7f0ec725a700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: in thread 7f0ec725a700 thread_name:ms_dispatch 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.438 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ed060bd20] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: gsignal() 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: abort() 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0ed1883437] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.439 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.440 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 16: clone() 2024-04-26T11:33:15.440 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:33:15.440 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: -1> 2024-04-26T11:33:15.068+0000 7f0ec725a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ec725a700 time 2024-04-26T11:33:15.067281+0000 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0ed18833dd] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.441 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: clone() 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 0> 2024-04-26T11:33:15.069+0000 7f0ec725a700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: in thread 7f0ec725a700 thread_name:ms_dispatch 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ed060bd20] 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: gsignal() 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: abort() 2024-04-26T11:33:15.442 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0ed1883437] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 16: clone() 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.443 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: -1> 2024-04-26T11:33:15.068+0000 7f0ec725a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ec725a700 time 2024-04-26T11:33:15.067281+0000 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0ed18833dd] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.444 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: clone() 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 0> 2024-04-26T11:33:15.069+0000 7f0ec725a700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: in thread 7f0ec725a700 thread_name:ms_dispatch 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ed060bd20] 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2: gsignal() 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 3: abort() 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0ed1883437] 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0ed18835a3] 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e89b129822] 2024-04-26T11:33:15.445 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e89b099598] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e89b09bc17] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e89af94d0b] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89af952ac] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e89af7dfad] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0ed1b17e20] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 13: (DispatchQueue::entry()+0x51f) [0x7f0ed1b14fdf] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0ed1be7fc1] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0ed06011ca] 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 16: clone() 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[62080]: 2024-04-26T11:33:15.446 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 podman[66367]: 2024-04-26 11:33:15.131212797 +0000 UTC m=+0.021234447 container died 292a5264335ecf7fd36f0a7d2e9795878cafb321dd9dfa06cfca425d9f97a973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:33:16.223 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 podman[66367]: 2024-04-26 11:33:15.959348744 +0000 UTC m=+0.849370390 container remove 292a5264335ecf7fd36f0a7d2e9795878cafb321dd9dfa06cfca425d9f97a973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T11:33:16.224 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:15 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:33:16.493 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:16 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-26T11:33:16.493 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:16 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 2.744s CPU time. 2024-04-26T11:33:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:16 smithi194 ceph-mon[33017]: pgmap v89: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 37 MiB/s wr, 2.29k op/s 2024-04-26T11:33:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:16 smithi033 ceph-mon[27845]: pgmap v89: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 37 MiB/s wr, 2.29k op/s 2024-04-26T11:33:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:16 smithi096 ceph-mon[32753]: pgmap v89: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 37 MiB/s wr, 2.29k op/s 2024-04-26T11:33:17.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 120.0 s, 3011.6 tps, lat 33.188 ms stddev 37.386 2024-04-26T11:33:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.693+0000 7f574a400640 1 -- 172.21.15.33:0/298786423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 msgr2=0x7f574410c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.693+0000 7f574a400640 1 --2- 172.21.15.33:0/298786423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 0x7f574410c9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f572c009900 tx=0x7f572c02a490 comp rx=0 tx=0).stop 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.694+0000 7f574a400640 1 -- 172.21.15.33:0/298786423 shutdown_connections 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.694+0000 7f574a400640 1 --2- 172.21.15.33:0/298786423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f574410cf90 0x7f574410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.694+0000 7f574a400640 1 --2- 172.21.15.33:0/298786423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 0x7f574410c9c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.694+0000 7f574a400640 1 --2- 172.21.15.33:0/298786423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5744103bc0 0x7f5744103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.694+0000 7f574a400640 1 -- 172.21.15.33:0/298786423 >> 172.21.15.33:0/298786423 conn(0x7f5744078940 msgr2=0x7f5744078d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.694+0000 7f574a400640 1 -- 172.21.15.33:0/298786423 shutdown_connections 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.694+0000 7f574a400640 1 -- 172.21.15.33:0/298786423 wait complete. 2024-04-26T11:33:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.695+0000 7f574a400640 1 Processor -- start 2024-04-26T11:33:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.695+0000 7f574a400640 1 -- start start 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.695+0000 7f574a400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5744103bc0 0x7f574419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.695+0000 7f574a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 0x7f574419b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f574a400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f574410cf90 0x7f574419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f574a400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5744112bd0 con 0x7f574410cf90 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f574a400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5744112a50 con 0x7f57441044e0 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f574a400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5744112d50 con 0x7f5744103bc0 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f574410cf90 0x7f574419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f574410cf90 0x7f574419fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57168/0 (socket says 172.21.15.33:57168) 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 -- 172.21.15.33:0/1603156670 learned_addr learned my addr 172.21.15.33:0/1603156670 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f573aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 0x7f574419b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743400640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5744103bc0 0x7f574419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 -- 172.21.15.33:0/1603156670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5744103bc0 msgr2=0x7f574419b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5744103bc0 0x7f574419b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 -- 172.21.15.33:0/1603156670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 msgr2=0x7f574419b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 0x7f574419b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f5743e00640 1 -- 172.21.15.33:0/1603156670 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57441a0340 con 0x7f574410cf90 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.696+0000 7f573aa00640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 0x7f574419b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.697+0000 7f5743e00640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f574410cf90 0x7f574419fc10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e970 tx=0x7f573c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.697+0000 7f5740c00640 1 -- 172.21.15.33:0/1603156670 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c00ccc0 con 0x7f574410cf90 2024-04-26T11:33:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.697+0000 7f5740c00640 1 -- 172.21.15.33:0/1603156670 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f573c0054d0 con 0x7f574410cf90 2024-04-26T11:33:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.697+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57441a0630 con 0x7f574410cf90 2024-04-26T11:33:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.697+0000 7f5740c00640 1 -- 172.21.15.33:0/1603156670 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c010430 con 0x7f574410cf90 2024-04-26T11:33:18.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.697+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57441a7ec0 con 0x7f574410cf90 2024-04-26T11:33:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.700+0000 7f574d393740 1 Processor -- start 2024-04-26T11:33:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.700+0000 7f5740c00640 1 -- 172.21.15.33:0/1603156670 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f573c005020 con 0x7f574410cf90 2024-04-26T11:33:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.700+0000 7f574d393740 1 -- start start 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f574d393740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f574d393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8d16f0 0x55e83c8dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f574d393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8dbfb0 0x55e83c8de3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f574d393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e83c836380 con 0x55e83c8d16f0 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f574d393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e83c7d2c30 con 0x55e83c8dbfb0 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f574d393740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e83c832340 con 0x55e83c8d0dd0 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8d16f0 0x55e83c8dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f573aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8dbfb0 0x55e83c8de3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8d16f0 0x55e83c8dba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57180/0 (socket says 172.21.15.33:57180) 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 -- 172.21.15.33:0/2310680358 learned_addr learned my addr 172.21.15.33:0/2310680358 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f573aa00640 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 -- 172.21.15.33:0/2310680358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 msgr2=0x55e83c8d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 -- 172.21.15.33:0/2310680358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8dbfb0 msgr2=0x55e83c8de3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8dbfb0 0x55e83c8de3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f5743400640 1 -- 172.21.15.33:0/2310680358 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e83c7f4940 con 0x55e83c8d16f0 2024-04-26T11:33:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.701+0000 7f573aa00640 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8dbfb0 0x55e83c8de3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.702+0000 7f5743400640 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8d16f0 0x55e83c8dba70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5734010400 tx=0x7f57340108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cb54b1451fb2f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.702+0000 7f5738c00640 1 -- 172.21.15.33:0/2310680358 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734003b70 con 0x55e83c8d16f0 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.702+0000 7f5738c00640 1 -- 172.21.15.33:0/2310680358 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5734003d10 con 0x55e83c8d16f0 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.702+0000 7f5738c00640 1 -- 172.21.15.33:0/2310680358 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734011a70 con 0x55e83c8d16f0 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.702+0000 7f5740c00640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57280a3e40 0x7f57280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.702+0000 7f5740c00640 1 -- 172.21.15.33:0/1603156670 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f573c0d0b10 con 0x7f574410cf90 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f573aa00640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57280a3e40 0x7f57280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 -- 172.21.15.33:0/2310680358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8d16f0 msgr2=0x55e83c8dba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8d16f0 0x55e83c8dba70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5734010400 tx=0x7f57340108c0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 -- 172.21.15.33:0/2310680358 shutdown_connections 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8dbfb0 0x55e83c8de3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8d16f0 0x55e83c8dba70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 --2- 172.21.15.33:0/2310680358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 -- 172.21.15.33:0/2310680358 >> 172.21.15.33:0/2310680358 conn(0x55e83c8cd2e0 msgr2=0x55e83c8cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 -- 172.21.15.33:0/2310680358 shutdown_connections 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f574d393740 1 -- 172.21.15.33:0/2310680358 wait complete. 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.703+0000 7f573aa00640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57280a3e40 0x7f57280a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f572c031440 tx=0x7f572c005780 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 -- start start 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 Processor -- start 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e83c8debe0 con 0x55e83c8dbfb0 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e83c7bdc60 con 0x55e83c8d16f0 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.704+0000 7f574d393740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e83c8ded60 con 0x55e83c8d0dd0 2024-04-26T11:33:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f574d393740 10 client.0 _ll_register_callbacks cb 0x55e83c7e9e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:47642/0 (socket says 172.21.15.33:47642) 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 -- 172.21.15.33:0/2377452236 learned_addr learned my addr 172.21.15.33:0/2377452236 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f573aa00640 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 -- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 msgr2=0x55e83c8d93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 -- 172.21.15.33:0/2377452236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 msgr2=0x55e83c8d98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 -- 172.21.15.33:0/2377452236 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e83c7cc510 con 0x55e83c8d0dd0 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f573aa00640 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5743400640 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d8e70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5734004c70 tx=0x7f57340115d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=596b414bbb6d2cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.705+0000 7f5731600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:33:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.706+0000 7f5731600640 1 -- 172.21.15.33:0/2377452236 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734004e50 con 0x55e83c8d0dd0 2024-04-26T11:33:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.706+0000 7f5731600640 1 -- 172.21.15.33:0/2377452236 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5734011980 con 0x55e83c8d0dd0 2024-04-26T11:33:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.706+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5734003d10 con 0x55e83c8d0dd0 2024-04-26T11:33:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.706+0000 7f5731600640 1 -- 172.21.15.33:0/2377452236 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5734011b80 con 0x55e83c8d0dd0 2024-04-26T11:33:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.706+0000 7f574d393740 10 client.25513 fetch_fsmap learned FSMap version 29 2024-04-26T11:33:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.706+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e83c548210 con 0x55e83c8d0dd0 2024-04-26T11:33:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.706+0000 7f5731600640 1 -- 172.21.15.33:0/2377452236 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5734003d10 con 0x55e83c8d0dd0 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f5731600640 1 -- 172.21.15.33:0/2377452236 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f5734005040 con 0x55e83c8d0dd0 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f574d393740 10 client.25513 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f574d393740 10 client.25513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f574d393740 20 client.25513 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f574d393740 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c8ebb70 0x55e83c8edf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f574d393740 4 client.25513 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e83c843f90 con 0x55e83c8ebb70 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.707+0000 7f573aa00640 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c8ebb70 0x55e83c8edf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.708+0000 7f573aa00640 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c8ebb70 0x55e83c8edf60 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de2f6e297f1888e4 server_cookie=ebbcef81533043e1 in_seq=0 out_seq=0 2024-04-26T11:33:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.708+0000 7f5731600640 10 client.25513 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.708+0000 7f5731600640 1 -- 172.21.15.33:0/2377452236 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e83c843f90 con 0x55e83c8ebb70 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.708+0000 7f5731600640 10 client.25513 handle_command_reply: tid=0 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.708+0000 7f574d393740 1 client.25513 shutdown 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 msgr2=0x55e83c8d8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d8e70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5734004c70 tx=0x7f57340115d0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 shutdown_connections 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c8ebb70 0x55e83c8edf60 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f5731600640 0 client.25513 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 --2- 172.21.15.33:0/2377452236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d8e70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 >> 172.21.15.33:0/2377452236 conn(0x55e83c8cd2e0 msgr2=0x55e83c8d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 shutdown_connections 2024-04-26T11:33:18.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 1 -- 172.21.15.33:0/2377452236 wait complete. 2024-04-26T11:33:18.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.709+0000 7f574d393740 20 client.25513 trim_cache size 0 max 16384 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 Processor -- start 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 -- start start 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d0dd0 0x55e83c8e7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d16f0 0x55e83c8f3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f57441b3100 con 0x55e83c8dbfb0 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f57441b09a0 con 0x55e83c8d0dd0 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f574d393740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f57441ae7f0 con 0x55e83c8d16f0 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f5743e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f5743e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57192/0 (socket says 172.21.15.33:57192) 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f5743e00640 1 -- 172.21.15.33:0/3262599346 learned_addr learned my addr 172.21.15.33:0/3262599346 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f5743400640 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d0dd0 0x55e83c8e7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.725+0000 7f573aa00640 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d16f0 0x55e83c8f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5743e00640 1 -- 172.21.15.33:0/3262599346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d16f0 msgr2=0x55e83c8f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5743e00640 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d16f0 0x55e83c8f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5743e00640 1 -- 172.21.15.33:0/3262599346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d0dd0 msgr2=0x55e83c8e7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5743e00640 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d0dd0 0x55e83c8e7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5743e00640 1 -- 172.21.15.33:0/3262599346 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e83c614620 con 0x55e83c8dbfb0 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5743400640 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d0dd0 0x55e83c8e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5743e00640 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8f3ad0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e940 tx=0x7f573c09c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c0e800787cbc83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5733e00640 1 -- 172.21.15.33:0/3262599346 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c09d160 con 0x55e83c8dbfb0 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5733e00640 1 -- 172.21.15.33:0/3262599346 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f573c09ca60 con 0x55e83c8dbfb0 2024-04-26T11:33:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.726+0000 7f5733e00640 1 -- 172.21.15.33:0/3262599346 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c0990d0 con 0x55e83c8dbfb0 2024-04-26T11:33:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 -- 172.21.15.33:0/3262599346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 msgr2=0x55e83c8f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8f3ad0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e940 tx=0x7f573c09c4d0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 -- 172.21.15.33:0/3262599346 shutdown_connections 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d16f0 0x55e83c8f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8f3ad0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 --2- 172.21.15.33:0/3262599346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d0dd0 0x55e83c8e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 -- 172.21.15.33:0/3262599346 >> 172.21.15.33:0/3262599346 conn(0x55e83c8cd2e0 msgr2=0x55e83c8d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 -- 172.21.15.33:0/3262599346 shutdown_connections 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 -- 172.21.15.33:0/3262599346 wait complete. 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.727+0000 7f574d393740 1 -- start start 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 1 Processor -- start 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f57441aee60 con 0x55e83c8dbfb0 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f57441b02d0 con 0x55e83c8d16f0 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f57441af530 con 0x55e83c8d0dd0 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f574d393740 10 client.0 _ll_register_callbacks cb 0x7f573c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f5743e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.728+0000 7f5743400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:47670/0 (socket says 172.21.15.33:47670) 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 -- 172.21.15.33:0/1012713503 learned_addr learned my addr 172.21.15.33:0/1012713503 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f573aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 -- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 msgr2=0x55e83c8d6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 -- 172.21.15.33:0/1012713503 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 msgr2=0x55e83c8d6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 -- 172.21.15.33:0/1012713503 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e83c7f0cb0 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f573aa00640 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5743400640 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d5bd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5734005fd0 tx=0x7f57340116e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=233bdba650b4176c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5731600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5731600640 1 -- 172.21.15.33:0/1012713503 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573403a060 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f5731600640 1 -- 172.21.15.33:0/1012713503 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5734011980 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.729+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f573c09ca60 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f5731600640 1 -- 172.21.15.33:0/1012713503 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5734004ca0 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 10 client.25519 fetch_fsmap learned FSMap version 29 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e83c951fa0 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f5731600640 1 -- 172.21.15.33:0/1012713503 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f573c09ca60 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f5731600640 1 -- 172.21.15.33:0/1012713503 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f5734035c10 con 0x55e83c8d0dd0 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 10 client.25519 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 10 client.25519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:33:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 20 client.25519 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c9526c0 0x55e83c954ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 4 client.25519 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:33:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e83c83de00 con 0x55e83c9526c0 2024-04-26T11:33:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.730+0000 7f573aa00640 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c9526c0 0x55e83c954ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.731+0000 7f573aa00640 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c9526c0 0x55e83c954ae0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7b5ddc38b0fde3b server_cookie=fb6b7890abc7a5f in_seq=0 out_seq=0 2024-04-26T11:33:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:18.731+0000 7f5731600640 10 client.25519 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:18.833 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:18 smithi033 ceph-mon[27845]: pgmap v90: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 836 op/s 2024-04-26T11:33:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:18 smithi194 ceph-mon[33017]: pgmap v90: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 836 op/s 2024-04-26T11:33:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:18 smithi096 ceph-mon[32753]: pgmap v90: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 836 op/s 2024-04-26T11:33:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:20 smithi194 ceph-mon[33017]: pgmap v91: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-26T11:33:20.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:20 smithi033 ceph-mon[27845]: pgmap v91: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-26T11:33:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:20 smithi096 ceph-mon[32753]: pgmap v91: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-26T11:33:22.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 125.0 s, 488.0 tps, lat 35.092 ms stddev 35.192 2024-04-26T11:33:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:22 smithi194 ceph-mon[33017]: pgmap v92: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T11:33:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:22 smithi033 ceph-mon[27845]: pgmap v92: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T11:33:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:22 smithi096 ceph-mon[32753]: pgmap v92: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T11:33:24.850 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:24 smithi033 ceph-mon[27845]: pgmap v93: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T11:33:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:24 smithi194 ceph-mon[33017]: pgmap v93: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T11:33:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:24 smithi096 ceph-mon[32753]: pgmap v93: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T11:33:26.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:26 smithi096 ceph-mon[32753]: pgmap v94: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.8 MiB/s wr, 636 op/s 2024-04-26T11:33:26.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:33:26.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 systemd[1]: Stopped Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:33:26.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 2.744s CPU time. 2024-04-26T11:33:26.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 systemd[1]: Starting Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:33:26.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 podman[66462]: 2024-04-26 11:33:26.580810547 +0000 UTC m=+0.018250933 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:33:26.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 podman[66462]: 2024-04-26 11:33:26.685493175 +0000 UTC m=+0.122933564 container create b306ec7e8a256eac47223d3cd06811e15f8693c72ee63581545d5c4c2beaa0f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True) 2024-04-26T11:33:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:26 smithi194 ceph-mon[33017]: pgmap v94: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.8 MiB/s wr, 636 op/s 2024-04-26T11:33:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:26 smithi033 ceph-mon[27845]: pgmap v94: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.8 MiB/s wr, 636 op/s 2024-04-26T11:33:26.997 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 podman[66462]: 2024-04-26 11:33:26.896713212 +0000 UTC m=+0.334153598 container init b306ec7e8a256eac47223d3cd06811e15f8693c72ee63581545d5c4c2beaa0f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:33:26.997 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 podman[66462]: 2024-04-26 11:33:26.899477822 +0000 UTC m=+0.336918210 container start b306ec7e8a256eac47223d3cd06811e15f8693c72ee63581545d5c4c2beaa0f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T11:33:26.997 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 ceph-mds[66482]: -- 172.21.15.96:0/885781317 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5572ad02fa40 con 0x5572ad002800 2024-04-26T11:33:26.997 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: starting mds.h at 2024-04-26T11:33:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 130.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:33:27.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 bash[66462]: b306ec7e8a256eac47223d3cd06811e15f8693c72ee63581545d5c4c2beaa0f8 2024-04-26T11:33:27.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:33:26 smithi096 systemd[1]: Started Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: daemon mds.h restarted 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:boot 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:33:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:27 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: daemon mds.h restarted 2024-04-26T11:33:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:33:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:33:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T11:33:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:boot 2024-04-26T11:33:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:33:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:33:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:27 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: daemon mds.h restarted 2024-04-26T11:33:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:33:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:33:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T11:33:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:boot 2024-04-26T11:33:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:33:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:33:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:27 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:28 smithi194 ceph-mon[33017]: pgmap v96: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-04-26T11:33:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:28 smithi033 ceph-mon[27845]: pgmap v96: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-04-26T11:33:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:28 smithi096 ceph-mon[32753]: pgmap v96: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-04-26T11:33:29.806 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:29 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:resolve 2024-04-26T11:33:29.807 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:29 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:29 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:resolve 2024-04-26T11:33:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:29 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:29 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:resolve 2024-04-26T11:33:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:29 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.581+0000 7f5731600640 1 -- 172.21.15.33:0/1012713503 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e83c83de00 con 0x55e83c9526c0 2024-04-26T11:33:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.581+0000 7f5731600640 10 client.25519 handle_command_reply: tid=0 2024-04-26T11:33:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 client.25519 shutdown 2024-04-26T11:33:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 msgr2=0x55e83c8d5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d5bd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5734005fd0 tx=0x7f57340116e0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 shutdown_connections 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55e83c9526c0 0x55e83c954ae0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e83c8dbfb0 0x55e83c8d6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e83c8d16f0 0x55e83c8d6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 --2- 172.21.15.33:0/1012713503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e83c8d0dd0 0x55e83c8d5bd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 >> 172.21.15.33:0/1012713503 conn(0x55e83c8cd2e0 msgr2=0x55e83c8d2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.582+0000 7f5731600640 0 client.25519 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.583+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 shutdown_connections 2024-04-26T11:33:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.583+0000 7f574d393740 1 -- 172.21.15.33:0/1012713503 wait complete. 2024-04-26T11:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.583+0000 7f574d393740 20 client.25519 trim_cache size 0 max 16384 2024-04-26T11:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.584+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57280a3e40 msgr2=0x7f57280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.584+0000 7f574a400640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57280a3e40 0x7f57280a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f572c031440 tx=0x7f572c005780 comp rx=0 tx=0).stop 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.584+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f574410cf90 msgr2=0x7f574419fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.584+0000 7f574a400640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f574410cf90 0x7f574419fc10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e970 tx=0x7f573c00ee30 comp rx=0 tx=0).stop 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 shutdown_connections 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f57280a3e40 0x7f57280a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f574410cf90 0x7f574419fc10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57441044e0 0x7f574419b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 --2- 172.21.15.33:0/1603156670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5744103bc0 0x7f574419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 >> 172.21.15.33:0/1603156670 conn(0x7f5744078940 msgr2=0x7f5744108dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 shutdown_connections 2024-04-26T11:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:30.585+0000 7f574a400640 1 -- 172.21.15.33:0/1603156670 wait complete. 2024-04-26T11:33:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:30 smithi194 ceph-mon[33017]: pgmap v97: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:30 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:reconnect 2024-04-26T11:33:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:30 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:30 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:33:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:30 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0149997 2024-04-26T11:33:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:30 smithi033 ceph-mon[27845]: pgmap v97: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:30 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:reconnect 2024-04-26T11:33:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:30 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:30 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:33:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:30 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0149997 2024-04-26T11:33:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:30 smithi096 ceph-mon[32753]: pgmap v97: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:30 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:reconnect 2024-04-26T11:33:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:30 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:30 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:33:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:30 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0149997 2024-04-26T11:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:rejoin 2024-04-26T11:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:31 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:31 smithi194 ceph-mon[33017]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:31 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:31 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:active 2024-04-26T11:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:31 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:rejoin 2024-04-26T11:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:31 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:31 smithi033 ceph-mon[27845]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:31 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:31 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:active 2024-04-26T11:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:31 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:rejoin 2024-04-26T11:33:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:31 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:31 smithi096 ceph-mon[32753]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:33:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:31 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:33:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:31 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:33:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1002586260,v1:172.21.15.96:6835/1002586260] up:active 2024-04-26T11:33:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:31 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 135.0 s, 576.8 tps, lat 490.648 ms stddev 2452.005 2024-04-26T11:33:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:32 smithi194 ceph-mon[33017]: pgmap v98: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:32 smithi033 ceph-mon[27845]: pgmap v98: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:32 smithi096 ceph-mon[32753]: pgmap v98: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:34 smithi194 ceph-mon[33017]: pgmap v99: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:34 smithi033 ceph-mon[27845]: pgmap v99: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:34 smithi096 ceph-mon[32753]: pgmap v99: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 998 KiB/s wr, 131 op/s 2024-04-26T11:33:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:33:35.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:33:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:33:36.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:36 smithi033 ceph-mon[27845]: pgmap v100: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-04-26T11:33:36.591 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcbb57e5700 time 2024-04-26T11:33:36.221811+0000 2024-04-26T11:33:36.591 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:36.591 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcbbfe0e3dd] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.592 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: clone() 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.222+0000 7fcbb57e5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcbb57e5700 time 2024-04-26T11:33:36.221811+0000 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcbbfe0e3dd] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.593 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: clone() 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: *** Caught signal (Aborted) ** 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: in thread 7fcbb57e5700 thread_name:ms_dispatch 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcbbeb96d20] 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: gsignal() 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: abort() 2024-04-26T11:33:36.594 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcbbfe0e437] 2024-04-26T11:33:36.595 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.595 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.595 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.595 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.595 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 16: clone() 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.223+0000 7fcbb57e5700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: in thread 7fcbb57e5700 thread_name:ms_dispatch 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.596 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcbbeb96d20] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: gsignal() 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: abort() 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcbbfe0e437] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.597 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 16: clone() 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: -1> 2024-04-26T11:33:36.222+0000 7fcbb57e5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcbb57e5700 time 2024-04-26T11:33:36.221811+0000 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcbbfe0e3dd] 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.598 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: clone() 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 0> 2024-04-26T11:33:36.223+0000 7fcbb57e5700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: in thread 7fcbb57e5700 thread_name:ms_dispatch 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.599 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcbbeb96d20] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: gsignal() 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: abort() 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcbbfe0e437] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.600 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 16: clone() 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: -1> 2024-04-26T11:33:36.222+0000 7fcbb57e5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcbb57e5700 time 2024-04-26T11:33:36.221811+0000 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcbbfe0e3dd] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.601 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: clone() 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 0> 2024-04-26T11:33:36.223+0000 7fcbb57e5700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: in thread 7fcbb57e5700 thread_name:ms_dispatch 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcbbeb96d20] 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2: gsignal() 2024-04-26T11:33:36.602 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 3: abort() 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcbbfe0e437] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcbbfe0e5a3] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56368b593822] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56368b503598] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56368b505c17] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56368b3fed0b] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56368b3ff2ac] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56368b3e7fad] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcbc00a2e20] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 13: (DispatchQueue::entry()+0x51f) [0x7fcbc009ffdf] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcbc0172fc1] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcbbeb8c1ca] 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 16: clone() 2024-04-26T11:33:36.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:33:36.604 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[61776]: 2024-04-26T11:33:36.604 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:36 smithi096 podman[66614]: 2024-04-26 11:33:36.279764913 +0000 UTC m=+0.018629601 container died c88e058eb2a1057dfffb0124e6bdbbb208a41a2c57fc37e99447983e205eb666 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T11:33:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:36 smithi194 ceph-mon[33017]: pgmap v100: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-04-26T11:33:36.910 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:36 smithi096 ceph-mon[32753]: pgmap v100: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-04-26T11:33:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 140.0 s, 3262.2 tps, lat 30.618 ms stddev 32.274 2024-04-26T11:33:37.462 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:37 smithi096 podman[66614]: 2024-04-26 11:33:37.20991396 +0000 UTC m=+0.948778648 container remove c88e058eb2a1057dfffb0124e6bdbbb208a41a2c57fc37e99447983e205eb666 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T11:33:37.462 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:37 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:33:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:33:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:33:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:33:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:33:37.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:33:37.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:33:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:33:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:33:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:33:37.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:37 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-26T11:33:37.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:37 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.417s CPU time. 2024-04-26T11:33:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:38 smithi033 ceph-mon[27845]: pgmap v101: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-04-26T11:33:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:38 smithi096 ceph-mon[32753]: pgmap v101: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-04-26T11:33:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:38 smithi194 ceph-mon[33017]: pgmap v101: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-04-26T11:33:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:40 smithi033 ceph-mon[27845]: pgmap v102: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.4 MiB/s wr, 607 op/s 2024-04-26T11:33:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:40 smithi096 ceph-mon[32753]: pgmap v102: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.4 MiB/s wr, 607 op/s 2024-04-26T11:33:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:40 smithi194 ceph-mon[33017]: pgmap v102: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.4 MiB/s wr, 607 op/s 2024-04-26T11:33:41.479 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 15.6 2024-04-26T11:33:41.480 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T11:33:42.139 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:33:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 145.0 s, 2499.4 tps, lat 39.798 ms stddev 44.170 2024-04-26T11:33:42.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:42 smithi033 ceph-mon[27845]: pgmap v103: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.1 MiB/s wr, 556 op/s 2024-04-26T11:33:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:42 smithi096 ceph-mon[32753]: pgmap v103: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.1 MiB/s wr, 556 op/s 2024-04-26T11:33:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:42 smithi194 ceph-mon[33017]: pgmap v103: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.1 MiB/s wr, 556 op/s 2024-04-26T11:33:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.767+0000 7f4179800640 1 -- 172.21.15.33:0/1180175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 msgr2=0x7f41741119b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.767+0000 7f4179800640 1 --2- 172.21.15.33:0/1180175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f41741119b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f416c009930 tx=0x7f416c02a440 comp rx=0 tx=0).stop 2024-04-26T11:33:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.768+0000 7f4179800640 1 -- 172.21.15.33:0/1180175033 shutdown_connections 2024-04-26T11:33:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.768+0000 7f4179800640 1 --2- 172.21.15.33:0/1180175033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4174104b30 0x7f4174104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.768+0000 7f4179800640 1 --2- 172.21.15.33:0/1180175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f41741119b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.768+0000 7f4179800640 1 --2- 172.21.15.33:0/1180175033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f417410abf0 0x7f417410afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.768+0000 7f4179800640 1 -- 172.21.15.33:0/1180175033 >> 172.21.15.33:0/1180175033 conn(0x7f4174100620 msgr2=0x7f4174102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.768+0000 7f4179800640 1 -- 172.21.15.33:0/1180175033 shutdown_connections 2024-04-26T11:33:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.769+0000 7f4179800640 1 -- 172.21.15.33:0/1180175033 wait complete. 2024-04-26T11:33:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.770+0000 7f4179800640 1 Processor -- start 2024-04-26T11:33:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.770+0000 7f4179800640 1 -- start start 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4179800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4174104b30 0x7f417419b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4179800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f417419b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4179800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f417410abf0 0x7f417419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4179800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4174114e30 con 0x7f417410abf0 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4179800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4174114cb0 con 0x7f41741054f0 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4172000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f417419b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4172000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f417419b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33558/0 (socket says 172.21.15.33:33558) 2024-04-26T11:33:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4172000640 1 -- 172.21.15.33:0/1122848420 learned_addr learned my addr 172.21.15.33:0/1122848420 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4172a00640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4174104b30 0x7f417419b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4174114fb0 con 0x7f4174104b30 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.771+0000 7f4173400640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f417410abf0 0x7f417419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f4172000640 1 -- 172.21.15.33:0/1122848420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4174104b30 msgr2=0x7f417419b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f4172000640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4174104b30 0x7f417419b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f4172000640 1 -- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f417410abf0 msgr2=0x7f417419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f4172000640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f417410abf0 0x7f417419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f4172000640 1 -- 172.21.15.33:0/1122848420 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41741a0190 con 0x7f41741054f0 2024-04-26T11:33:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f4172000640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f417419b730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4164009d50 tx=0x7f416400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f416b400640 1 -- 172.21.15.33:0/1122848420 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164003b70 con 0x7f41741054f0 2024-04-26T11:33:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f416b400640 1 -- 172.21.15.33:0/1122848420 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4164003d10 con 0x7f41741054f0 2024-04-26T11:33:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f416b400640 1 -- 172.21.15.33:0/1122848420 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41640108b0 con 0x7f41741054f0 2024-04-26T11:33:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.772+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41741a0480 con 0x7f41741054f0 2024-04-26T11:33:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.773+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41741a7d10 con 0x7f41741054f0 2024-04-26T11:33:43.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.778+0000 7f417c619740 1 Processor -- start 2024-04-26T11:33:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.779+0000 7f416b400640 1 -- 172.21.15.33:0/1122848420 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4164004000 con 0x7f41741054f0 2024-04-26T11:33:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.779+0000 7f417c619740 1 -- start start 2024-04-26T11:33:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.780+0000 7f417c619740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.780+0000 7f417c619740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.781+0000 7f4173400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.781+0000 7f4173400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93bf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33572/0 (socket says 172.21.15.33:33572) 2024-04-26T11:33:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f4172a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f4172a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57458/0 (socket says 172.21.15.33:57458) 2024-04-26T11:33:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f417c619740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f417c619740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5628b9324380 con 0x5628b93bf6f0 2024-04-26T11:33:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f417c619740 1 -- 172.21.15.33:0/1685584757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5628b92c0c30 con 0x5628b93bedd0 2024-04-26T11:33:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f4173400640 1 -- 172.21.15.33:0/1685584757 learned_addr learned my addr 172.21.15.33:0/1685584757 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f417c619740 1 -- 172.21.15.33:0/1685584757 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5628b9320340 con 0x5628b93c9fb0 2024-04-26T11:33:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f416b400640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f41580a3e40 0x7f41580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.782+0000 7f4172000640 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.783+0000 7f416b400640 1 -- 172.21.15.33:0/1122848420 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8584+0+0 (secure 0 0 0) 0x7f41640d0ee0 con 0x7f41741054f0 2024-04-26T11:33:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.783+0000 7f4173400640 1 -- 172.21.15.33:0/1685584757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 msgr2=0x5628b93cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.783+0000 7f4173400640 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.783+0000 7f4173400640 1 -- 172.21.15.33:0/1685584757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 msgr2=0x5628b93c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.784+0000 7f4173400640 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.784+0000 7f4173400640 1 -- 172.21.15.33:0/1685584757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b92e2940 con 0x5628b93bedd0 2024-04-26T11:33:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.784+0000 7f4172000640 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.784+0000 7f4172a00640 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.784+0000 7f4173400640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f41580a3e40 0x7f41580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.786+0000 7f4173400640 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93bf1b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f416c009930 tx=0x7f416c0041c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e4c9936748d9b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.787+0000 7f4163e00640 1 -- 172.21.15.33:0/1685584757 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c04d070 con 0x5628b93bedd0 2024-04-26T11:33:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.787+0000 7f4163e00640 1 -- 172.21.15.33:0/1685584757 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f416c004d70 con 0x5628b93bedd0 2024-04-26T11:33:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.787+0000 7f4163e00640 1 -- 172.21.15.33:0/1685584757 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c035040 con 0x5628b93bedd0 2024-04-26T11:33:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.788+0000 7f417c619740 1 -- 172.21.15.33:0/1685584757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 msgr2=0x5628b93bf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.788+0000 7f417c619740 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93bf1b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f416c009930 tx=0x7f416c0041c0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.788+0000 7f4173400640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f41580a3e40 0x7f41580a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f416c04a000 tx=0x7f416c04b000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.789+0000 7f417c619740 1 -- 172.21.15.33:0/1685584757 shutdown_connections 2024-04-26T11:33:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.789+0000 7f417c619740 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.789+0000 7f417c619740 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.789+0000 7f417c619740 1 --2- 172.21.15.33:0/1685584757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93bf1b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.789+0000 7f417c619740 1 -- 172.21.15.33:0/1685584757 >> 172.21.15.33:0/1685584757 conn(0x5628b93bb2e0 msgr2=0x5628b93bb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.790+0000 7f417c619740 1 -- 172.21.15.33:0/1685584757 shutdown_connections 2024-04-26T11:33:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.790+0000 7f417c619740 1 -- 172.21.15.33:0/1685584757 wait complete. 2024-04-26T11:33:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.790+0000 7f417c619740 1 -- start start 2024-04-26T11:33:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.791+0000 7f417c619740 1 Processor -- start 2024-04-26T11:33:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.791+0000 7f417c619740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bedd0 0x5628b93c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.791+0000 7f417c619740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.791+0000 7f417c619740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93c9fb0 0x5628b93c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.791+0000 7f417c619740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5628b93ccbe0 con 0x5628b93bf6f0 2024-04-26T11:33:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.791+0000 7f417c619740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5628b92abc60 con 0x5628b93c9fb0 2024-04-26T11:33:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.791+0000 7f417c619740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5628b93ccd60 con 0x5628b93bedd0 2024-04-26T11:33:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f417c619740 10 client.0 _ll_register_callbacks cb 0x5628b92d7e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bedd0 0x5628b93c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bedd0 0x5628b93c6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:42016/0 (socket says 172.21.15.33:42016) 2024-04-26T11:33:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 -- 172.21.15.33:0/286057331 learned_addr learned my addr 172.21.15.33:0/286057331 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4173400640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93c9fb0 0x5628b93c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 -- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93c9fb0 msgr2=0x5628b93c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93c9fb0 0x5628b93c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 -- 172.21.15.33:0/286057331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 msgr2=0x5628b93c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172000640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.792+0000 7f4172a00640 1 -- 172.21.15.33:0/286057331 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b92ba510 con 0x5628b93bedd0 2024-04-26T11:33:43.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.793+0000 7f4172000640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:33:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.793+0000 7f4172a00640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bedd0 0x5628b93c6f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f415c011470 tx=0x7f415c011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e65617e1a31f0a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.793+0000 7f4160c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:33:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.793+0000 7f4160c00640 1 -- 172.21.15.33:0/286057331 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c004020 con 0x5628b93bedd0 2024-04-26T11:33:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.793+0000 7f4160c00640 1 -- 172.21.15.33:0/286057331 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8584+0+0 (secure 0 0 0) 0x7f415c0041c0 con 0x5628b93bedd0 2024-04-26T11:33:43.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.794+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f416c004d70 con 0x5628b93bedd0 2024-04-26T11:33:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.794+0000 7f4160c00640 1 -- 172.21.15.33:0/286057331 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f415c0043c0 con 0x5628b93bedd0 2024-04-26T11:33:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.794+0000 7f417c619740 10 client.25531 fetch_fsmap learned FSMap version 35 2024-04-26T11:33:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.794+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628b9036210 con 0x5628b93bedd0 2024-04-26T11:33:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.795+0000 7f4160c00640 1 -- 172.21.15.33:0/286057331 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f416c004d70 con 0x5628b93bedd0 2024-04-26T11:33:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.795+0000 7f4160c00640 1 -- 172.21.15.33:0/286057331 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f415c003af0 con 0x5628b93bedd0 2024-04-26T11:33:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.795+0000 7f417c619740 10 client.25531 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T11:33:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.795+0000 7f417c619740 10 client.25531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.795+0000 7f417c619740 20 client.25531 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.795+0000 7f417c619740 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b93d9df0 0x5628b93dc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.796+0000 7f417c619740 4 client.25531 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.796+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628b9331f90 con 0x5628b93d9df0 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.796+0000 7f4172000640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b93d9df0 0x5628b93dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.797+0000 7f4172000640 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b93d9df0 0x5628b93dc260 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caeb7e6c29c72fda server_cookie=77cc6aa7bbac8fb1 in_seq=0 out_seq=0 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.797+0000 7f4160c00640 10 client.25531 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.797+0000 7f4160c00640 1 -- 172.21.15.33:0/286057331 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628b9331f90 con 0x5628b93d9df0 2024-04-26T11:33:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.797+0000 7f4160c00640 10 client.25531 handle_command_reply: tid=0 2024-04-26T11:33:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.798+0000 7f417c619740 1 client.25531 shutdown 2024-04-26T11:33:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.798+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bedd0 msgr2=0x5628b93c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.798+0000 7f417c619740 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bedd0 0x5628b93c6f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f415c011470 tx=0x7f415c011930 comp rx=0 tx=0).stop 2024-04-26T11:33:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 shutdown_connections 2024-04-26T11:33:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f417c619740 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b93d9df0 0x5628b93dc260 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f417c619740 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93c9fb0 0x5628b93c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f417c619740 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f417c619740 1 --2- 172.21.15.33:0/286057331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bedd0 0x5628b93c6f90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f4160c00640 0 client.25531 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 >> 172.21.15.33:0/286057331 conn(0x5628b93bb2e0 msgr2=0x5628b93c1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.800+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 shutdown_connections 2024-04-26T11:33:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.801+0000 7f417c619740 1 -- 172.21.15.33:0/286057331 wait complete. 2024-04-26T11:33:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.801+0000 7f417c619740 20 client.25531 trim_cache size 0 max 16384 2024-04-26T11:33:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 Processor -- start 2024-04-26T11:33:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 -- start start 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bf6f0 0x5628b93babc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93c9fb0 0x5628b93c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41741b3080 con 0x5628b93c9fb0 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41741b08f0 con 0x5628b93bedd0 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.829+0000 7f417c619740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f41741ae740 con 0x5628b93bf6f0 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.830+0000 7f4172000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bf6f0 0x5628b93babc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.830+0000 7f4172a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.830+0000 7f4172a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33586/0 (socket says 172.21.15.33:33586) 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.830+0000 7f4172a00640 1 -- 172.21.15.33:0/2626102647 learned_addr learned my addr 172.21.15.33:0/2626102647 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.830+0000 7f4172000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bf6f0 0x5628b93babc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:42028/0 (socket says 172.21.15.33:42028) 2024-04-26T11:33:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.830+0000 7f4173400640 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93c9fb0 0x5628b93c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.831+0000 7f4172a00640 1 -- 172.21.15.33:0/2626102647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bf6f0 msgr2=0x5628b93babc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.831+0000 7f4172a00640 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bf6f0 0x5628b93babc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.831+0000 7f4172a00640 1 -- 172.21.15.33:0/2626102647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93c9fb0 msgr2=0x5628b93c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.831+0000 7f4172a00640 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93c9fb0 0x5628b93c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.831+0000 7f4172a00640 1 -- 172.21.15.33:0/2626102647 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b9102620 con 0x5628b93bedd0 2024-04-26T11:33:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.832+0000 7f4172a00640 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d5aa0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f415c006010 tx=0x7f415c01a720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9768d67e6a47734 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.833+0000 7f4163400640 1 -- 172.21.15.33:0/2626102647 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c004000 con 0x5628b93bedd0 2024-04-26T11:33:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.833+0000 7f4163400640 1 -- 172.21.15.33:0/2626102647 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415c0041a0 con 0x5628b93bedd0 2024-04-26T11:33:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.833+0000 7f4163400640 1 -- 172.21.15.33:0/2626102647 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c0051d0 con 0x5628b93bedd0 2024-04-26T11:33:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.834+0000 7f417c619740 1 -- 172.21.15.33:0/2626102647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 msgr2=0x5628b93d5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.834+0000 7f417c619740 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d5aa0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f415c006010 tx=0x7f415c01a720 comp rx=0 tx=0).stop 2024-04-26T11:33:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 -- 172.21.15.33:0/2626102647 shutdown_connections 2024-04-26T11:33:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93bf6f0 0x5628b93babc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93c9fb0 0x5628b93c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 --2- 172.21.15.33:0/2626102647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d5aa0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 -- 172.21.15.33:0/2626102647 >> 172.21.15.33:0/2626102647 conn(0x5628b93bb2e0 msgr2=0x5628b93c86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 -- 172.21.15.33:0/2626102647 shutdown_connections 2024-04-26T11:33:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 -- 172.21.15.33:0/2626102647 wait complete. 2024-04-26T11:33:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.835+0000 7f417c619740 1 -- start start 2024-04-26T11:33:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.836+0000 7f417c619740 1 Processor -- start 2024-04-26T11:33:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.836+0000 7f417c619740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.836+0000 7f417c619740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.836+0000 7f417c619740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f417c619740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41741aedb0 con 0x5628b93bf6f0 2024-04-26T11:33:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f417c619740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41741b0220 con 0x5628b93bedd0 2024-04-26T11:33:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f417c619740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f41741af480 con 0x5628b93c9fb0 2024-04-26T11:33:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f417c619740 10 client.0 _ll_register_callbacks cb 0x7f415c00bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f4172a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f4173400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f4173400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:42030/0 (socket says 172.21.15.33:42030) 2024-04-26T11:33:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f4173400640 1 -- 172.21.15.33:0/2416327345 learned_addr learned my addr 172.21.15.33:0/2416327345 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.837+0000 7f4172a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33588/0 (socket says 172.21.15.33:33588) 2024-04-26T11:33:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.838+0000 7f4172000640 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.838+0000 7f4172a00640 1 -- 172.21.15.33:0/2416327345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 msgr2=0x5628b93cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.838+0000 7f4172a00640 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.838+0000 7f4172a00640 1 -- 172.21.15.33:0/2416327345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 msgr2=0x5628b93d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.838+0000 7f4172a00640 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.838+0000 7f4172a00640 1 -- 172.21.15.33:0/2416327345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b92decb0 con 0x5628b93bedd0 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.838+0000 7f4172000640 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.839+0000 7f4172a00640 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d1fb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f415c005d00 tx=0x7f415c011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77de4bfbe725ae0b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.839+0000 7f4160c00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.839+0000 7f4160c00640 1 -- 172.21.15.33:0/2416327345 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c03a060 con 0x5628b93bedd0 2024-04-26T11:33:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.840+0000 7f4160c00640 1 -- 172.21.15.33:0/2416327345 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8584+0+0 (secure 0 0 0) 0x7f415c029040 con 0x5628b93bedd0 2024-04-26T11:33:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.840+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f415c0041a0 con 0x5628b93bedd0 2024-04-26T11:33:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.840+0000 7f4160c00640 1 -- 172.21.15.33:0/2416327345 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f415c004000 con 0x5628b93bedd0 2024-04-26T11:33:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.840+0000 7f417c619740 10 client.25574 fetch_fsmap learned FSMap version 35 2024-04-26T11:33:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.840+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628b9449310 con 0x5628b93bedd0 2024-04-26T11:33:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.840+0000 7f4160c00640 1 -- 172.21.15.33:0/2416327345 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415c0041a0 con 0x5628b93bedd0 2024-04-26T11:33:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.841+0000 7f4160c00640 1 -- 172.21.15.33:0/2416327345 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f415c010820 con 0x5628b93bedd0 2024-04-26T11:33:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.841+0000 7f417c619740 10 client.25574 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T11:33:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.841+0000 7f417c619740 10 client.25574 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:33:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.841+0000 7f417c619740 20 client.25574 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.841+0000 7f417c619740 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b94499d0 0x5628b944bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.841+0000 7f417c619740 4 client.25574 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:33:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.841+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628b932be00 con 0x5628b94499d0 2024-04-26T11:33:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.842+0000 7f4172000640 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b94499d0 0x5628b944bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.842+0000 7f4160c00640 10 client.25574 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:43.843+0000 7f4172000640 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b94499d0 0x5628b944bdf0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbddc01147654774 server_cookie=d19bdb825864a2b8 in_seq=0 out_seq=0 2024-04-26T11:33:44.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 -- 172.21.15.33:0/1172935791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075650 msgr2=0x7fa684075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:44.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 --2- 172.21.15.33:0/1172935791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075650 0x7fa684075a30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa6740098f0 tx=0x7fa674009c00 comp rx=0 tx=0).stop 2024-04-26T11:33:44.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 -- 172.21.15.33:0/1172935791 shutdown_connections 2024-04-26T11:33:44.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 --2- 172.21.15.33:0/1172935791 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa68410fe20 0x7fa684112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:44.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 --2- 172.21.15.33:0/1172935791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075650 0x7fa684075a30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:44.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 --2- 172.21.15.33:0/1172935791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa684075f70 0x7fa68410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:44.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 -- 172.21.15.33:0/1172935791 >> 172.21.15.33:0/1172935791 conn(0x7fa68407a580 msgr2=0x7fa68407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:44.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 -- 172.21.15.33:0/1172935791 shutdown_connections 2024-04-26T11:33:44.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.824+0000 7fa68a400700 1 -- 172.21.15.33:0/1172935791 wait complete. 2024-04-26T11:33:44.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.825+0000 7fa68a400700 1 Processor -- start 2024-04-26T11:33:44.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.825+0000 7fa68a400700 1 -- start start 2024-04-26T11:33:44.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa68a400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa684075650 0x7fa68419bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:44.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa68a400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075f70 0x7fa68419c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:44.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa68a400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa68410fe20 0x7fa6841a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:44.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa68a400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa684115800 con 0x7fa684075650 2024-04-26T11:33:44.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa68a400700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa684115680 con 0x7fa684075f70 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa68a400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa684115980 con 0x7fa68410fe20 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa683400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa684075650 0x7fa68419bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa683e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa68410fe20 0x7fa6841a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa683400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa684075650 0x7fa68419bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57516/0 (socket says 172.21.15.33:57516) 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa683400700 1 -- 172.21.15.33:0/3257809648 learned_addr learned my addr 172.21.15.33:0/3257809648 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.826+0000 7fa682a00700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075f70 0x7fa68419c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa682a00700 1 -- 172.21.15.33:0/3257809648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa68410fe20 msgr2=0x7fa6841a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa682a00700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa68410fe20 0x7fa6841a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa682a00700 1 -- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa684075650 msgr2=0x7fa68419bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa682a00700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa684075650 0x7fa68419bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:44.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa682a00700 1 -- 172.21.15.33:0/3257809648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6841a0d20 con 0x7fa684075f70 2024-04-26T11:33:44.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa682a00700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075f70 0x7fa68419c250 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa67000e9c0 tx=0x7fa67000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:44.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa67be00700 1 -- 172.21.15.33:0/3257809648 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67000c9c0 con 0x7fa684075f70 2024-04-26T11:33:44.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6841a1010 con 0x7fa684075f70 2024-04-26T11:33:44.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa67be00700 1 -- 172.21.15.33:0/3257809648 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa67000cb60 con 0x7fa684075f70 2024-04-26T11:33:44.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa67be00700 1 -- 172.21.15.33:0/3257809648 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670018420 con 0x7fa684075f70 2024-04-26T11:33:44.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.827+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6841a8890 con 0x7fa684075f70 2024-04-26T11:33:44.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.828+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa68406a110 con 0x7fa684075f70 2024-04-26T11:33:44.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.831+0000 7fa67be00700 1 -- 172.21.15.33:0/3257809648 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa670010430 con 0x7fa684075f70 2024-04-26T11:33:44.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.832+0000 7fa67be00700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa6680a3e30 0x7fa6680a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:44.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.833+0000 7fa67be00700 1 -- 172.21.15.33:0/3257809648 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8584+0+0 (secure 0 0 0) 0x7fa6700d0db0 con 0x7fa684075f70 2024-04-26T11:33:44.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.833+0000 7fa683400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa6680a3e30 0x7fa6680a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:44.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.833+0000 7fa683400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa6680a3e30 0x7fa6680a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa674009680 tx=0x7fa674019040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:44.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:44.834+0000 7fa67be00700 1 -- 172.21.15.33:0/3257809648 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa67009d3b0 con 0x7fa684075f70 2024-04-26T11:33:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:44 smithi033 ceph-mon[27845]: pgmap v104: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.1 MiB/s wr, 556 op/s 2024-04-26T11:33:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:44 smithi096 ceph-mon[32753]: pgmap v104: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.1 MiB/s wr, 556 op/s 2024-04-26T11:33:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:44 smithi194 ceph-mon[33017]: pgmap v104: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 7.1 MiB/s wr, 556 op/s 2024-04-26T11:33:45.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.129+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa68419d070 con 0x7fa684075f70 2024-04-26T11:33:45.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.130+0000 7fa67be00700 1 -- 172.21.15.33:0/3257809648 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11873 (secure 0 0 0) 0x7fa67009ccb0 con 0x7fa684075f70 2024-04-26T11:33:45.132 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-26T11:33:45.132 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4017784507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4017784507},{"type":"v1","addr":"172.21.15.33:6835","nonce":4017784507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14394,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/500691090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":500691090},{"type":"v1","addr":"172.21.15.33:6839","nonce":500691090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14400,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/3408204812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":3408204812},{"type":"v1","addr":"172.21.15.33:6841","nonce":3408204812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3159968353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3159968353},{"type":"v1","addr":"172.21.15.194:6833","nonce":3159968353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3830261408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3830261408},{"type":"v1","addr":"172.21.15.96:6833","nonce":3830261408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3073208972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3073208972},{"type":"v1","addr":"172.21.15.33:6837","nonce":3073208972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T11:26:32.073203+0000","modified":"2024-04-26T11:33:31.164054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24470,"mds_2":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/1403509498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":1403509498},{"type":"v1","addr":"172.21.15.194:6835","nonce":1403509498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"e","rank":1,"incarnation":31,"state":"up:active","state_seq":120,"addr":"172.21.15.96:6835/1002586260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1002586260},{"type":"v1","addr":"172.21.15.96:6835","nonce":1002586260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/1265312544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1265312544},{"type":"v1","addr":"172.21.15.194:6837","nonce":1265312544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6839/517247009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":517247009},{"type":"v1","addr":"172.21.15.194:6839","nonce":517247009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.96:6839/1966519759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1966519759},{"type":"v1","addr":"172.21.15.96:6839","nonce":1966519759}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25556":{"gid":25556,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/1439931910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":1439931910},{"type":"v1","addr":"172.21.15.96:6837","nonce":1439931910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24484,24470]},"id":1}]} 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa6680a3e30 msgr2=0x7fa6680a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa6680a3e30 0x7fa6680a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa674009680 tx=0x7fa674019040 comp rx=0 tx=0).stop 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075f70 msgr2=0x7fa68419c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075f70 0x7fa68419c250 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa67000e9c0 tx=0x7fa67000ecd0 comp rx=0 tx=0).stop 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 shutdown_connections 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa6680a3e30 0x7fa6680a62f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa68410fe20 0x7fa6841a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa684075650 0x7fa68419bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 --2- 172.21.15.33:0/3257809648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa684075f70 0x7fa68419c250 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 >> 172.21.15.33:0/3257809648 conn(0x7fa68407a580 msgr2=0x7fa684100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 shutdown_connections 2024-04-26T11:33:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:45.133+0000 7fa68a400700 1 -- 172.21.15.33:0/3257809648 wait complete. 2024-04-26T11:33:45.136 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 35 2024-04-26T11:33:45.844 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:45 smithi033 ceph-mon[27845]: from='client.? 172.21.15.33:0/3257809648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:33:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:45 smithi096 ceph-mon[32753]: from='client.? 172.21.15.33:0/3257809648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:33:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:45 smithi194 ceph-mon[33017]: from='client.? 172.21.15.33:0/3257809648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T11:33:46.180 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24527 quiesce db / --timeout 60 --expiration 75.6 --await-for 42.0' 2024-04-26T11:33:46.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 5cc7f146-03be-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24527 quiesce db / --timeout 60 --expiration 75.6 --await-for 42.0 2024-04-26T11:33:46.636 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/5cc7f146-03be-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T11:33:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:46 smithi033 ceph-mon[27845]: pgmap v105: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 9.7 MiB/s wr, 839 op/s 2024-04-26T11:33:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:46 smithi096 ceph-mon[32753]: pgmap v105: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 9.7 MiB/s wr, 839 op/s 2024-04-26T11:33:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:46 smithi194 ceph-mon[33017]: pgmap v105: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 9.7 MiB/s wr, 839 op/s 2024-04-26T11:33:47.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 150.0 s, 773.6 tps, lat 47.822 ms stddev 64.512 2024-04-26T11:33:47.952 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:47 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:33:47.952 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:47 smithi096 systemd[1]: Stopped Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:33:47.952 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:47 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.417s CPU time. 2024-04-26T11:33:47.952 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:47 smithi096 systemd[1]: Starting Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:33:47.953 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:47 smithi096 podman[66753]: 2024-04-26 11:33:47.832949688 +0000 UTC m=+0.018146521 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:33:47.953 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:47 smithi096 podman[66753]: 2024-04-26 11:33:47.952267411 +0000 UTC m=+0.137464219 container create 92a09f568a26e590de3b2c387b2d9ceceb24ae51618e047af2c03bc8d92d0294 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-26T11:33:48.222 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:48 smithi096 podman[66753]: 2024-04-26 11:33:48.197161569 +0000 UTC m=+0.382358377 container init 92a09f568a26e590de3b2c387b2d9ceceb24ae51618e047af2c03bc8d92d0294 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-26T11:33:48.223 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:48 smithi096 podman[66753]: 2024-04-26 11:33:48.19991234 +0000 UTC m=+0.385109154 container start 92a09f568a26e590de3b2c387b2d9ceceb24ae51618e047af2c03bc8d92d0294 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:33:48.633 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mds[66772]: -- 172.21.15.96:0/3396292247 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573aae51a40 con 0x5573aae24800 2024-04-26T11:33:48.634 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: starting mds.e at 2024-04-26T11:33:48.634 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:48 smithi096 bash[66753]: 92a09f568a26e590de3b2c387b2d9ceceb24ae51618e047af2c03bc8d92d0294 2024-04-26T11:33:48.634 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:33:48 smithi096 systemd[1]: Started Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:33:48.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: pgmap v106: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 634 op/s 2024-04-26T11:33:48.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: daemon mds.e restarted 2024-04-26T11:33:48.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:33:48.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:33:48.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T11:33:48.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:boot 2024-04-26T11:33:48.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:33:48.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:33:48.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:48 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: pgmap v106: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 634 op/s 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: daemon mds.e restarted 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:boot 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:33:48.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:48 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:48.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 -- 172.21.15.33:0/348781681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03a810b640 msgr2=0x7f03a810ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:48.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 --2- 172.21.15.33:0/348781681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03a810b640 0x7f03a810ba20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f039c009990 tx=0x7f039c009ca0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 -- 172.21.15.33:0/348781681 shutdown_connections 2024-04-26T11:33:48.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 --2- 172.21.15.33:0/348781681 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a8106000 0x7f03a8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 --2- 172.21.15.33:0/348781681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03a810b640 0x7f03a810ba20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 --2- 172.21.15.33:0/348781681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03a8105640 0x7f03a8105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 -- 172.21.15.33:0/348781681 >> 172.21.15.33:0/348781681 conn(0x7f03a8101130 msgr2=0x7f03a8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:48.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 -- 172.21.15.33:0/348781681 shutdown_connections 2024-04-26T11:33:48.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.952+0000 7f03b0800700 1 -- 172.21.15.33:0/348781681 wait complete. 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.953+0000 7f03b0800700 1 Processor -- start 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.953+0000 7f03b0800700 1 -- start start 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03b0800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03a8105640 0x7f03a81a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03b0800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03a8106000 0x7f03a81a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03b0800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a810b640 0x7f03a819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03b0800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03a8115870 con 0x7f03a8105640 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03b0800700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81156f0 con 0x7f03a8106000 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03b0800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81159f0 con 0x7f03a810b640 2024-04-26T11:33:48.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03ade00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03a8105640 0x7f03a81a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03ae800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a810b640 0x7f03a819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03ad400700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03a8106000 0x7f03a81a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03ade00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03a8105640 0x7f03a81a1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57528/0 (socket says 172.21.15.33:57528) 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.954+0000 7f03ade00700 1 -- 172.21.15.33:0/4014196248 learned_addr learned my addr 172.21.15.33:0/4014196248 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03ad400700 1 -- 172.21.15.33:0/4014196248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a810b640 msgr2=0x7f03a819ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03ad400700 1 --2- 172.21.15.33:0/4014196248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a810b640 0x7f03a819ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03ad400700 1 -- 172.21.15.33:0/4014196248 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03a8105640 msgr2=0x7f03a81a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03ad400700 1 --2- 172.21.15.33:0/4014196248 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03a8105640 0x7f03a81a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03ad400700 1 -- 172.21.15.33:0/4014196248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a819c1b0 con 0x7f03a8106000 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03ad400700 1 --2- 172.21.15.33:0/4014196248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03a8106000 0x7f03a81a1dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f039800e9f0 tx=0x7f039800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03a2a00700 1 -- 172.21.15.33:0/4014196248 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039800c9c0 con 0x7f03a8106000 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03a2a00700 1 -- 172.21.15.33:0/4014196248 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f039800cb60 con 0x7f03a8106000 2024-04-26T11:33:48.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03a2a00700 1 -- 172.21.15.33:0/4014196248 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398018680 con 0x7f03a8106000 2024-04-26T11:33:48.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03b0800700 1 -- 172.21.15.33:0/4014196248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a819c4a0 con 0x7f03a8106000 2024-04-26T11:33:48.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.955+0000 7f03b0800700 1 -- 172.21.15.33:0/4014196248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a81a8760 con 0x7f03a8106000 2024-04-26T11:33:48.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.959+0000 7f03a2a00700 1 -- 172.21.15.33:0/4014196248 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0398010430 con 0x7f03a8106000 2024-04-26T11:33:48.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.960+0000 7f03bb264b80 1 Processor -- start 2024-04-26T11:33:48.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.960+0000 7f03bb264b80 1 -- start start 2024-04-26T11:33:48.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.960+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a4095429d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.960+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a40954c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a40954ccc0 0x55a40954f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03bb264b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a409495b70 con 0x55a409542f10 2024-04-26T11:33:48.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03bb264b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a4094343d0 con 0x55a4095425f0 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03bb264b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a409491b30 con 0x55a40954ccc0 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03a2a00700 1 --2- 172.21.15.33:0/4014196248 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f03900a3ed0 0x7f03900a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03ae800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a4095429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03ae800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a4095429d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33630/0 (socket says 172.21.15.33:33630) 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03ae800700 1 -- 172.21.15.33:0/1691443940 learned_addr learned my addr 172.21.15.33:0/1691443940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03ad400700 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a40954ccc0 0x55a40954f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03ade00700 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a40954c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.961+0000 7f03ae800700 1 --2- 172.21.15.33:0/4014196248 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f03900a3ed0 0x7f03900a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03a2a00700 1 -- 172.21.15.33:0/4014196248 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f03980d09e0 con 0x7f03a8106000 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ade00700 1 -- 172.21.15.33:0/1691443940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a40954ccc0 msgr2=0x55a40954f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ade00700 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a40954ccc0 0x55a40954f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ade00700 1 -- 172.21.15.33:0/1691443940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 msgr2=0x55a4095429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:48.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ade00700 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a4095429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ade00700 1 -- 172.21.15.33:0/1691443940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a40943c8f0 con 0x55a409542f10 2024-04-26T11:33:48.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ae800700 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a4095429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ade00700 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a40954c780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f03a819cb90 tx=0x7f039c005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd8ae7d57ee60ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f03ae800700 1 --2- 172.21.15.33:0/4014196248 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f03900a3ed0 0x7f03900a6390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f03a4003fd0 tx=0x7f03a4004b40 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.962+0000 7f0397400700 1 -- 172.21.15.33:0/1691443940 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c027070 con 0x55a409542f10 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f0397400700 1 -- 172.21.15.33:0/1691443940 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f039c0056a0 con 0x55a409542f10 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f0397400700 1 -- 172.21.15.33:0/1691443940 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c003bb0 con 0x55a409542f10 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f03bb264b80 1 -- 172.21.15.33:0/1691443940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 msgr2=0x55a40954c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f03bb264b80 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a40954c780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f03a819cb90 tx=0x7f039c005bc0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f03bb264b80 1 -- 172.21.15.33:0/1691443940 shutdown_connections 2024-04-26T11:33:48.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f03bb264b80 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a40954ccc0 0x55a40954f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f03bb264b80 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a4095429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f03bb264b80 1 --2- 172.21.15.33:0/1691443940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a40954c780 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.963+0000 7f03bb264b80 1 -- 172.21.15.33:0/1691443940 >> 172.21.15.33:0/1691443940 conn(0x55a4094705d0 msgr2=0x55a40953d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:48.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.964+0000 7f03bb264b80 1 -- 172.21.15.33:0/1691443940 shutdown_connections 2024-04-26T11:33:48.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.964+0000 7f03bb264b80 1 -- 172.21.15.33:0/1691443940 wait complete. 2024-04-26T11:33:48.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.964+0000 7f03bb264b80 1 -- start start 2024-04-26T11:33:48.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.964+0000 7f03bb264b80 1 Processor -- start 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4095425f0 0x55a40954c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a409549760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a409549ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03bb264b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a40954f7d0 con 0x55a4095425f0 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03bb264b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a409430620 con 0x55a40954ccc0 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03bb264b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a40954f950 con 0x55a409542f10 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03bb264b80 10 client.0 _ll_register_callbacks cb 0x55a409429510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03ade00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4095425f0 0x55a40954c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03ad400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a409549760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03ae800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a409549ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03ade00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4095425f0 0x55a40954c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57558/0 (socket says 172.21.15.33:57558) 2024-04-26T11:33:48.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03ade00700 1 -- 172.21.15.33:0/935588333 learned_addr learned my addr 172.21.15.33:0/935588333 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.965+0000 7f03ae800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a409549ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33642/0 (socket says 172.21.15.33:33642) 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f03ae800700 1 -- 172.21.15.33:0/935588333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 msgr2=0x55a409549760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f03ae800700 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a409549760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f03ae800700 1 -- 172.21.15.33:0/935588333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4095425f0 msgr2=0x55a40954c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f03ae800700 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4095425f0 0x55a40954c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f03ae800700 1 -- 172.21.15.33:0/935588333 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4093e2a60 con 0x55a40954ccc0 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f03ae800700 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a409549ca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f03a4004840 tx=0x7f03a4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc84689a3fd12151 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f038be00700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f038be00700 1 -- 172.21.15.33:0/935588333 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a402a060 con 0x55a40954ccc0 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f038be00700 1 -- 172.21.15.33:0/935588333 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f03a4007d70 con 0x55a40954ccc0 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.966+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f039c0056a0 con 0x55a40954ccc0 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f038be00700 1 -- 172.21.15.33:0/935588333 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f03a4031050 con 0x55a40954ccc0 2024-04-26T11:33:48.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 10 client.25598 fetch_fsmap learned FSMap version 37 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a40955a100 con 0x55a40954ccc0 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f038be00700 1 -- 172.21.15.33:0/935588333 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f039c0056a0 con 0x55a40954ccc0 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f038be00700 1 -- 172.21.15.33:0/935588333 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f03a4018040 con 0x55a40954ccc0 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 10 client.25598 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 10 client.25598 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 20 client.25598 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a40955c740 0x55a40955eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 4 client.25598 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:33:48.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.967+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a40950ed00 con 0x55a40955c740 2024-04-26T11:33:48.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.968+0000 7f03ade00700 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a40955c740 0x55a40955eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:48.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.968+0000 7f03ade00700 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a40955c740 0x55a40955eb70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfa367fd2375eb9b server_cookie=47ad2a4b54b3eacc in_seq=0 out_seq=0 2024-04-26T11:33:48.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:48.968+0000 7f038be00700 10 client.25598 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: pgmap v106: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 634 op/s 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: daemon mds.e restarted 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:boot 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:48 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:50.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:50 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:resolve 2024-04-26T11:33:50.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:50 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:50.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:50 smithi033 ceph-mon[27845]: pgmap v108: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:50 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:resolve 2024-04-26T11:33:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:50 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:50 smithi194 ceph-mon[33017]: pgmap v108: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:50 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:resolve 2024-04-26T11:33:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:50 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:50 smithi096 ceph-mon[32753]: pgmap v108: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.275+0000 7f4160c00640 1 -- 172.21.15.33:0/2416327345 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628b932be00 con 0x5628b94499d0 2024-04-26T11:33:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.275+0000 7f4160c00640 10 client.25574 handle_command_reply: tid=0 2024-04-26T11:33:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.275+0000 7f417c619740 1 client.25574 shutdown 2024-04-26T11:33:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 msgr2=0x5628b93d1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d1fb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f415c005d00 tx=0x7f415c011930 comp rx=0 tx=0).stop 2024-04-26T11:33:51.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.275+0000 7f038be00700 1 -- 172.21.15.33:0/935588333 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a40950ed00 con 0x55a40955c740 2024-04-26T11:33:51.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f038be00700 10 client.25598 handle_command_reply: tid=0 2024-04-26T11:33:51.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f03bb264b80 1 client.25598 shutdown 2024-04-26T11:33:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 shutdown_connections 2024-04-26T11:33:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628b93c9fb0 0x5628b93cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5628b94499d0 0x5628b944bdf0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5628b93bf6f0 0x5628b93d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 --2- 172.21.15.33:0/2416327345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628b93bedd0 0x5628b93d1fb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f4160c00640 0 client.25574 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.276+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 >> 172.21.15.33:0/2416327345 conn(0x5628b93bb2e0 msgr2=0x5628b93c1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 shutdown_connections 2024-04-26T11:33:51.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 msgr2=0x55a409549ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:51.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a409549ca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f03a4004840 tx=0x7f03a4004ad0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f417c619740 1 -- 172.21.15.33:0/2416327345 wait complete. 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 shutdown_connections 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a40955c740 0x55a40955eb70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f038be00700 0 client.25598 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a409549ca0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4095425f0 0x55a40954c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 --2- 172.21.15.33:0/935588333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a409549760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.277+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 >> 172.21.15.33:0/935588333 conn(0x55a4094705d0 msgr2=0x55a40953f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:51.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.278+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 shutdown_connections 2024-04-26T11:33:51.283 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.278+0000 7f417c619740 20 client.25574 trim_cache size 0 max 16384 2024-04-26T11:33:51.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.278+0000 7f03bb264b80 1 -- 172.21.15.33:0/935588333 wait complete. 2024-04-26T11:33:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:33:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:33:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:33:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:33:51.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.278+0000 7f03bb264b80 20 client.25598 trim_cache size 0 max 16384 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f41580a3e40 msgr2=0x7f41580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f41580a3e40 0x7f41580a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f416c04a000 tx=0x7f416c04b000 comp rx=0 tx=0).stop 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 msgr2=0x7f417419b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f417419b730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4164009d50 tx=0x7f416400c6a0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 shutdown_connections 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f41580a3e40 0x7f41580a6300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f417410abf0 0x7f417419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41741054f0 0x7f417419b730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 --2- 172.21.15.33:0/1122848420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4174104b30 0x7f417419b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 >> 172.21.15.33:0/1122848420 conn(0x7f4174100620 msgr2=0x7f4174101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.279+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 shutdown_connections 2024-04-26T11:33:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:33:51.280+0000 7f4179800640 1 -- 172.21.15.33:0/1122848420 wait complete. 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.305+0000 7f03bb264b80 1 Processor -- start 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.305+0000 7f03bb264b80 1 -- start start 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a409549d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a40954bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a40954ccc0 0x55a4095644d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03bb264b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81b0840 con 0x55a40954ccc0 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03bb264b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81adfc0 con 0x55a4095425f0 2024-04-26T11:33:51.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03bb264b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81b0170 con 0x55a409542f10 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03ade00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a409549d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03ade00700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a409549d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45460/0 (socket says 172.21.15.33:45460) 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.306+0000 7f03ade00700 1 -- 172.21.15.33:0/2094891451 learned_addr learned my addr 172.21.15.33:0/2094891451 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ae800700 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a40954ccc0 0x55a4095644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ad400700 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a40954bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ade00700 1 -- 172.21.15.33:0/2094891451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 msgr2=0x55a40954bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ade00700 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a40954bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ade00700 1 -- 172.21.15.33:0/2094891451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a40954ccc0 msgr2=0x55a4095644d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:51.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ade00700 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a40954ccc0 0x55a4095644d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ade00700 1 -- 172.21.15.33:0/2094891451 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a40955f1f0 con 0x55a4095425f0 2024-04-26T11:33:51.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.307+0000 7f03ade00700 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a409549d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b610 tx=0x7f039c005530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3bdd2db17c7c9dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:51.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.308+0000 7f0396a00700 1 -- 172.21.15.33:0/2094891451 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c032070 con 0x55a4095425f0 2024-04-26T11:33:51.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.308+0000 7f0396a00700 1 -- 172.21.15.33:0/2094891451 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f039c028040 con 0x55a4095425f0 2024-04-26T11:33:51.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.308+0000 7f0396a00700 1 -- 172.21.15.33:0/2094891451 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c032e60 con 0x55a4095425f0 2024-04-26T11:33:51.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 -- 172.21.15.33:0/2094891451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 msgr2=0x55a409549d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:51.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a409549d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b610 tx=0x7f039c005530 comp rx=0 tx=0).stop 2024-04-26T11:33:51.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 -- 172.21.15.33:0/2094891451 shutdown_connections 2024-04-26T11:33:51.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a40954ccc0 0x55a4095644d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4095425f0 0x55a409549d90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2094891451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a409542f10 0x55a40954bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 -- 172.21.15.33:0/2094891451 >> 172.21.15.33:0/2094891451 conn(0x55a4094705d0 msgr2=0x55a409544ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 -- 172.21.15.33:0/2094891451 shutdown_connections 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.309+0000 7f03bb264b80 1 -- 172.21.15.33:0/2094891451 wait complete. 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.310+0000 7f03bb264b80 1 -- start start 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.310+0000 7f03bb264b80 1 Processor -- start 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a4095425f0 0x55a409554c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a409552190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03bb264b80 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a4095526d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03bb264b80 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81afaa0 con 0x55a409542f10 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03bb264b80 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81ae630 con 0x55a40954ccc0 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03bb264b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81aed00 con 0x55a4095425f0 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03bb264b80 10 client.0 _ll_register_callbacks cb 0x55a409429510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03ad400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a409552190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03ae800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a4095526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03ae800700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a4095526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45462/0 (socket says 172.21.15.33:45462) 2024-04-26T11:33:51.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03ad400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a409552190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36072/0 (socket says 172.21.15.33:36072) 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03ae800700 1 -- 172.21.15.33:0/2760085619 learned_addr learned my addr 172.21.15.33:0/2760085619 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.311+0000 7f03ade00700 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a4095425f0 0x55a409554c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f03ae800700 1 -- 172.21.15.33:0/2760085619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a4095425f0 msgr2=0x55a409554c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f03ae800700 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a4095425f0 0x55a409554c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f03ae800700 1 -- 172.21.15.33:0/2760085619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 msgr2=0x55a409552190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f03ae800700 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a409552190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f03ae800700 1 -- 172.21.15.33:0/2760085619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a409552c10 con 0x55a40954ccc0 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f03ade00700 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a4095425f0 0x55a409554c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:33:51.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f03ae800700 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a4095526d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f03a40195e0 tx=0x7f03a4009170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b5462d674f9a95f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:33:51.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.312+0000 7f038be00700 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:33:51.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.313+0000 7f038be00700 1 -- 172.21.15.33:0/2760085619 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4005c60 con 0x55a40954ccc0 2024-04-26T11:33:51.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.313+0000 7f038be00700 1 -- 172.21.15.33:0/2760085619 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f03a4005e00 con 0x55a40954ccc0 2024-04-26T11:33:51.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.313+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f039c028040 con 0x55a40954ccc0 2024-04-26T11:33:51.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.313+0000 7f038be00700 1 -- 172.21.15.33:0/2760085619 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f03a4025400 con 0x55a40954ccc0 2024-04-26T11:33:51.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.313+0000 7f03bb264b80 10 client.25604 fetch_fsmap learned FSMap version 40 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.313+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a4095c4a90 con 0x55a40954ccc0 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f038be00700 1 -- 172.21.15.33:0/2760085619 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f039c028040 con 0x55a40954ccc0 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f038be00700 1 -- 172.21.15.33:0/2760085619 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 40) ==== 8350+0+0 (secure 0 0 0) 0x7f03a4035480 con 0x55a40954ccc0 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f03bb264b80 10 client.25604 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f03bb264b80 10 client.25604 resolve_mds: validated gid 24527 aka daemon mds.b 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f03bb264b80 20 client.25604 populate_metadata read hostname 'smithi033' 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a4095c52f0 0x55a4095c7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f03bb264b80 4 client.25604 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 75.6, "await_for": 42.0}] 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 75.6, "await_for": 42.0}) -- 0x55a4094a3550 con 0x55a4095c52f0 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.314+0000 7f03ade00700 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a4095c52f0 0x55a4095c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.315+0000 7f03ade00700 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a4095c52f0 0x55a4095c7710 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce73ca706e137a47 server_cookie=ed68515481c98595 in_seq=0 out_seq=0 2024-04-26T11:33:51.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:51.315+0000 7f038be00700 10 client.25604 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:51 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:reconnect 2024-04-26T11:33:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:51 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:51 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:33:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:51 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:33:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:51 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:reconnect 2024-04-26T11:33:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:51 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:51 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:33:51.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:51 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:33:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:51 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:reconnect 2024-04-26T11:33:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:51 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:51 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:33:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:51 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:33:52.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 155.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:33:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.237+0000 7f038be00700 1 -- 172.21.15.33:0/2760085619 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: -9 ) ==== 8+0+586 (crc 0 0 0) 0x55a4094a3550 con 0x55a4095c52f0 2024-04-26T11:33:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.237+0000 7f038be00700 10 client.25604 handle_command_reply: tid=0 2024-04-26T11:33:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.237+0000 7f03bb264b80 1 client.25604 shutdown 2024-04-26T11:33:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.238+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 msgr2=0x55a4095526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:33:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.238+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a4095526d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f03a40195e0 tx=0x7f03a4009170 comp rx=0 tx=0).stop 2024-04-26T11:33:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.238+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 shutdown_connections 2024-04-26T11:33:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.238+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a4095c52f0 0x55a4095c7710 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:52.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.239+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a40954ccc0 0x55a4095526d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:52.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.239+0000 7f038be00700 0 client.25604 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:33:52.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.239+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a4095425f0 0x55a409554c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:52.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.239+0000 7f03bb264b80 1 --2- 172.21.15.33:0/2760085619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a409542f10 0x55a409552190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:33:52.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.239+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 >> 172.21.15.33:0/2760085619 conn(0x55a4094705d0 msgr2=0x55a4095415e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:33:52.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.239+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 shutdown_connections 2024-04-26T11:33:52.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.239+0000 7f03bb264b80 1 -- 172.21.15.33:0/2760085619 wait complete. 2024-04-26T11:33:52.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-26T11:33:52.240+0000 7f03bb264b80 20 client.25604 trim_cache size 0 max 16384 2024-04-26T11:33:52.242 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-26T11:33:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 40, 2024-04-26T11:33:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: "leader": 24527, 2024-04-26T11:33:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: "set_version": 10, 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "sets": { 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "519a5833": { 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "version": 10, 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "age_ref": 0.0, 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FAILED", 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "timeout": 60.0, 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "expiration": 75.6, 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "members": { 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "file:/": { 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "excluded": false, 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "state": { 2024-04-26T11:33:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FAILED", 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-26T11:33:52.244 INFO:teuthology.orchestra.run.smithi033.stderr:Error EBADF: 2024-04-26T11:33:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:52 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:rejoin 2024-04-26T11:33:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:52 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:52 smithi194 ceph-mon[33017]: pgmap v109: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:52 smithi194 ceph-mon[33017]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:33:52.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:52 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:rejoin 2024-04-26T11:33:52.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:52 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:52.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:52 smithi033 ceph-mon[27845]: pgmap v109: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:52.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:52 smithi033 ceph-mon[27845]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:33:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:52 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:rejoin 2024-04-26T11:33:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:52 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:52 smithi096 ceph-mon[32753]: pgmap v109: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:52 smithi096 ceph-mon[32753]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:33:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 9 2024-04-26T11:33:53.251 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 9 (EBADF), stdout: { "epoch": 40, "leader": 24527, "set_version": 10, "sets": { "519a5833": { "version": 10, "age_ref": 0.0, "state": { "name": "FAILED", "age": 0.0 }, "timeout": 60.0, "expiration": 75.6, "members": { "file:/": { "excluded": false, "state": { "name": "FAILED", "age": 0.0 } } } } } } 2024-04-26T11:33:53.251 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks/quiescer.py", line 354, in _run self.do_quiesce(d) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks/quiescer.py", line 310, in do_quiesce raise RuntimeError(f"Error quiescing set '{set_id}': {rcinfo(rc)}") RuntimeError: Error quiescing set '519a5833': 9 (EBADF) 2024-04-26T11:33:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:53 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:33:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:53 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:33:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:53 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:active 2024-04-26T11:33:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:53 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:53 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:53 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:33:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:53 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:33:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:53 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:active 2024-04-26T11:33:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:53 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:53.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:53 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:53 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:33:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:53 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:33:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:53 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1439931910,v1:172.21.15.96:6837/1439931910] up:active 2024-04-26T11:33:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:53 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:33:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:53 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:33:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:54 smithi096 ceph-mon[32753]: pgmap v110: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:54 smithi194 ceph-mon[33017]: pgmap v110: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:54 smithi033 ceph-mon[27845]: pgmap v110: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-04-26T11:33:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:56 smithi194 ceph-mon[33017]: pgmap v111: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-04-26T11:33:56.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:56 smithi033 ceph-mon[27845]: pgmap v111: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-04-26T11:33:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:56 smithi096 ceph-mon[32753]: pgmap v111: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-04-26T11:33:57.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 160.0 s, 2910.2 tps, lat 90.573 ms stddev 700.312 2024-04-26T11:33:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:33:58 smithi194 ceph-mon[33017]: pgmap v112: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-04-26T11:33:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:33:58 smithi033 ceph-mon[27845]: pgmap v112: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-04-26T11:33:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:33:58 smithi096 ceph-mon[32753]: pgmap v112: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-04-26T11:34:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:00 smithi194 ceph-mon[33017]: pgmap v113: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.6 MiB/s wr, 574 op/s 2024-04-26T11:34:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:00 smithi033 ceph-mon[27845]: pgmap v113: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.6 MiB/s wr, 574 op/s 2024-04-26T11:34:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:00 smithi096 ceph-mon[32753]: pgmap v113: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.6 MiB/s wr, 574 op/s 2024-04-26T11:34:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 165.0 s, 1824.6 tps, lat 52.358 ms stddev 89.987 2024-04-26T11:34:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:02 smithi194 ceph-mon[33017]: pgmap v114: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.4 MiB/s wr, 445 op/s 2024-04-26T11:34:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:02 smithi033 ceph-mon[27845]: pgmap v114: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.4 MiB/s wr, 445 op/s 2024-04-26T11:34:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:02 smithi096 ceph-mon[32753]: pgmap v114: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.4 MiB/s wr, 445 op/s 2024-04-26T11:34:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:03 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:03 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:03 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.451+0000 7f117aa00640 1 -- 172.21.15.33:0/3211028467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1174075550 msgr2=0x7f117410ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.451+0000 7f117aa00640 1 --2- 172.21.15.33:0/3211028467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1174075550 0x7f117410ece0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1160009900 tx=0x7f116002a480 comp rx=0 tx=0).stop 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.452+0000 7f117aa00640 1 -- 172.21.15.33:0/3211028467 shutdown_connections 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.452+0000 7f117aa00640 1 --2- 172.21.15.33:0/3211028467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1174075550 0x7f117410ece0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.452+0000 7f117aa00640 1 --2- 172.21.15.33:0/3211028467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f117410f220 0x7f1174111610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.452+0000 7f117aa00640 1 --2- 172.21.15.33:0/3211028467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1174074c30 0x7f1174075010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.452+0000 7f117aa00640 1 -- 172.21.15.33:0/3211028467 >> 172.21.15.33:0/3211028467 conn(0x7f1174079dc0 msgr2=0x7f117407a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.452+0000 7f117aa00640 1 -- 172.21.15.33:0/3211028467 shutdown_connections 2024-04-26T11:34:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.452+0000 7f117aa00640 1 -- 172.21.15.33:0/3211028467 wait complete. 2024-04-26T11:34:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.453+0000 7f117aa00640 1 Processor -- start 2024-04-26T11:34:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.453+0000 7f117aa00640 1 -- start start 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.453+0000 7f117aa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1174074c30 0x7f117419b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f117aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174075550 0x7f117419b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f117aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f117410f220 0x7f117419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f117aa00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1174114e50 con 0x7f1174074c30 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f117aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1174114cd0 con 0x7f1174075550 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f117aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1174114fd0 con 0x7f117410f220 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174075550 0x7f117419b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1174074c30 0x7f117419b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1178a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f117410f220 0x7f117419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174075550 0x7f117419b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34602/0 (socket says 172.21.15.33:34602) 2024-04-26T11:34:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173400640 1 -- 172.21.15.33:0/1876359500 learned_addr learned my addr 172.21.15.33:0/1876359500 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173400640 1 -- 172.21.15.33:0/1876359500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f117410f220 msgr2=0x7f117419fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173400640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f117410f220 0x7f117419fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173400640 1 -- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1174074c30 msgr2=0x7f117419b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.454+0000 7f1173400640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1174074c30 0x7f117419b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.455+0000 7f1173400640 1 -- 172.21.15.33:0/1876359500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11741a0200 con 0x7f1174075550 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.455+0000 7f1178a00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f117410f220 0x7f117419fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.455+0000 7f1173400640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174075550 0x7f117419b750 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1160005ab0 tx=0x7f1160003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.455+0000 7f1170c00640 1 -- 172.21.15.33:0/1876359500 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116003f070 con 0x7f1174075550 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.455+0000 7f1170c00640 1 -- 172.21.15.33:0/1876359500 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1160031b60 con 0x7f1174075550 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.455+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11741a0490 con 0x7f1174075550 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.455+0000 7f1170c00640 1 -- 172.21.15.33:0/1876359500 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160035030 con 0x7f1174075550 2024-04-26T11:34:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.456+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11741a7d30 con 0x7f1174075550 2024-04-26T11:34:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f1170c00640 1 -- 172.21.15.33:0/1876359500 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1160042070 con 0x7f1174075550 2024-04-26T11:34:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 Processor -- start 2024-04-26T11:34:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 -- start start 2024-04-26T11:34:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b151b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b1fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b223a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563656a7a380 con 0x563656b1ffb0 2024-04-26T11:34:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563656a16c30 con 0x563656b156f0 2024-04-26T11:34:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.460+0000 7f117d876740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563656a76340 con 0x563656b14dd0 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1173400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b223a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1173e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b1fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1178a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1173400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b223a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59492/0 (socket says 172.21.15.33:59492) 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1173400640 1 -- 172.21.15.33:0/1491751264 learned_addr learned my addr 172.21.15.33:0/1491751264 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1178a00640 1 -- 172.21.15.33:0/1491751264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 msgr2=0x563656b1fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1178a00640 1 --2- 172.21.15.33:0/1491751264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b1fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1178a00640 1 -- 172.21.15.33:0/1491751264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 msgr2=0x563656b223a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1178a00640 1 --2- 172.21.15.33:0/1491751264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b223a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.461+0000 7f1178a00640 1 -- 172.21.15.33:0/1491751264 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563656a38940 con 0x563656b14dd0 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.462+0000 7f1170c00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f11580a3e40 0x7f11580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.462+0000 7f1178a00640 1 --2- 172.21.15.33:0/1491751264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b151b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f116c011760 tx=0x7f116c011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf6e59c94ddae24c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.462+0000 7f1170c00640 1 -- 172.21.15.33:0/1876359500 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f11600f0e40 con 0x7f1174075550 2024-04-26T11:34:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.462+0000 7f1173e00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f11580a3e40 0x7f11580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.462+0000 7f1169200640 1 -- 172.21.15.33:0/1491751264 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c004d20 con 0x563656b14dd0 2024-04-26T11:34:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.462+0000 7f1169200640 1 -- 172.21.15.33:0/1491751264 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116c00f9a0 con 0x563656b14dd0 2024-04-26T11:34:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.462+0000 7f1169200640 1 -- 172.21.15.33:0/1491751264 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c0043d0 con 0x563656b14dd0 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 -- 172.21.15.33:0/1491751264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 msgr2=0x563656b151b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 --2- 172.21.15.33:0/1491751264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b151b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f116c011760 tx=0x7f116c011c20 comp rx=0 tx=0).stop 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f1173e00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f11580a3e40 0x7f11580a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x563656b16370 tx=0x7f1164005f90 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 -- 172.21.15.33:0/1491751264 shutdown_connections 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 --2- 172.21.15.33:0/1491751264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b223a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 --2- 172.21.15.33:0/1491751264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b1fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 --2- 172.21.15.33:0/1491751264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b151b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 -- 172.21.15.33:0/1491751264 >> 172.21.15.33:0/1491751264 conn(0x563656b112e0 msgr2=0x563656b116c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 -- 172.21.15.33:0/1491751264 shutdown_connections 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 -- 172.21.15.33:0/1491751264 wait complete. 2024-04-26T11:34:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.463+0000 7f117d876740 1 -- start start 2024-04-26T11:34:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.464+0000 7f117d876740 1 Processor -- start 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f117d876740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b1cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f117d876740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b1d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f117d876740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b1d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f117d876740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563656b22be0 con 0x563656b14dd0 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f117d876740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563656a01c60 con 0x563656b1ffb0 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f117d876740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563656b22d60 con 0x563656b156f0 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f117d876740 10 client.0 _ll_register_callbacks cb 0x563656a2de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f1173e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b1cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f1178a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b1d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f1173e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b1cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59506/0 (socket says 172.21.15.33:59506) 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.465+0000 7f1173e00640 1 -- 172.21.15.33:0/1913608851 learned_addr learned my addr 172.21.15.33:0/1913608851 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f1178a00640 1 -- 172.21.15.33:0/1913608851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 msgr2=0x563656b1d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f1178a00640 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b1d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f1178a00640 1 -- 172.21.15.33:0/1913608851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 msgr2=0x563656b1cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f1178a00640 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b1cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f1178a00640 1 -- 172.21.15.33:0/1913608851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563656a10510 con 0x563656b1ffb0 2024-04-26T11:34:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f1178a00640 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b1d9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f116c011760 tx=0x7f116c004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eac1c1ad0244eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f115d600640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:34:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.466+0000 7f115d600640 1 -- 172.21.15.33:0/1913608851 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c003a30 con 0x563656b1ffb0 2024-04-26T11:34:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.467+0000 7f115d600640 1 -- 172.21.15.33:0/1913608851 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f116c003bd0 con 0x563656b1ffb0 2024-04-26T11:34:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.467+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f116c00f9a0 con 0x563656b1ffb0 2024-04-26T11:34:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.467+0000 7f115d600640 1 -- 172.21.15.33:0/1913608851 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f116c032050 con 0x563656b1ffb0 2024-04-26T11:34:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.467+0000 7f117d876740 10 client.25616 fetch_fsmap learned FSMap version 43 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.467+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56365678c210 con 0x563656b1ffb0 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.467+0000 7f115d600640 1 -- 172.21.15.33:0/1913608851 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116c00f9a0 con 0x563656b1ffb0 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.467+0000 7f115d600640 1 -- 172.21.15.33:0/1913608851 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 43) ==== 8342+0+0 (secure 0 0 0) 0x7f116c018070 con 0x563656b1ffb0 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f117d876740 10 client.25616 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f117d876740 10 client.25616 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f117d876740 20 client.25616 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f117d876740 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b2fc60 0x563656b32050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f117d876740 4 client.25616 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563656a87f90 con 0x563656b2fc60 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f1173e00640 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b2fc60 0x563656b32050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f115d600640 10 client.25616 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.468+0000 7f1173e00640 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b2fc60 0x563656b32050 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=901ba6d54e5fd994 server_cookie=aece56049dabeb4c in_seq=0 out_seq=0 2024-04-26T11:34:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.469+0000 7f115d600640 1 -- 172.21.15.33:0/1913608851 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563656a87f90 con 0x563656b2fc60 2024-04-26T11:34:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.469+0000 7f115d600640 10 client.25616 handle_command_reply: tid=0 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.469+0000 7f117d876740 1 client.25616 shutdown 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 msgr2=0x563656b1d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b1d9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f116c011760 tx=0x7f116c004cf0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 shutdown_connections 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b1d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b2fc60 0x563656b32050 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b1d9e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 --2- 172.21.15.33:0/1913608851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b1cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f115d600640 0 client.25616 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 >> 172.21.15.33:0/1913608851 conn(0x563656b112e0 msgr2=0x563656b17010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 shutdown_connections 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 1 -- 172.21.15.33:0/1913608851 wait complete. 2024-04-26T11:34:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.470+0000 7f117d876740 20 client.25616 trim_cache size 0 max 16384 2024-04-26T11:34:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.492+0000 7f117d876740 1 Processor -- start 2024-04-26T11:34:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.492+0000 7f117d876740 1 -- start start 2024-04-26T11:34:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f117d876740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b1f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f117d876740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b2baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f117d876740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b1d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f117d876740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b30d0 con 0x563656b1ffb0 2024-04-26T11:34:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f117d876740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b0970 con 0x563656b156f0 2024-04-26T11:34:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f117d876740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11741ae7c0 con 0x563656b14dd0 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b2baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b1f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b2baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34624/0 (socket says 172.21.15.33:34624) 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b1f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:44298/0 (socket says 172.21.15.33:44298) 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173400640 1 -- 172.21.15.33:0/1941135582 learned_addr learned my addr 172.21.15.33:0/1941135582 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1178a00640 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b1d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173e00640 1 -- 172.21.15.33:0/1941135582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 msgr2=0x563656b2baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173e00640 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b2baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173e00640 1 -- 172.21.15.33:0/1941135582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 msgr2=0x563656b1d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.493+0000 7f1173e00640 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b1d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.494+0000 7f1173e00640 1 -- 172.21.15.33:0/1941135582 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563656858620 con 0x563656b14dd0 2024-04-26T11:34:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.494+0000 7f1173400640 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b2baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.494+0000 7f1173e00640 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b1f690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x563656b22e80 tx=0x7f116401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6652e7a0807fc719 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.494+0000 7f115fe00640 1 -- 172.21.15.33:0/1941135582 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164003a40 con 0x563656b14dd0 2024-04-26T11:34:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.494+0000 7f115fe00640 1 -- 172.21.15.33:0/1941135582 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116402b8d0 con 0x563656b14dd0 2024-04-26T11:34:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.494+0000 7f115fe00640 1 -- 172.21.15.33:0/1941135582 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11640047a0 con 0x563656b14dd0 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 -- 172.21.15.33:0/1941135582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 msgr2=0x563656b1f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b1f690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x563656b22e80 tx=0x7f116401b000 comp rx=0 tx=0).stop 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 -- 172.21.15.33:0/1941135582 shutdown_connections 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b1ffb0 0x563656b1d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b156f0 0x563656b2baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 --2- 172.21.15.33:0/1941135582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b14dd0 0x563656b1f690 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 -- 172.21.15.33:0/1941135582 >> 172.21.15.33:0/1941135582 conn(0x563656b112e0 msgr2=0x563656b1cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.495+0000 7f117d876740 1 -- 172.21.15.33:0/1941135582 shutdown_connections 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.496+0000 7f117d876740 1 -- 172.21.15.33:0/1941135582 wait complete. 2024-04-26T11:34:04.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.496+0000 7f117d876740 1 -- start start 2024-04-26T11:34:04.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.496+0000 7f117d876740 1 Processor -- start 2024-04-26T11:34:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f117d876740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b28400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f117d876740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b199c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f117d876740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b19f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f117d876740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f11741aee30 con 0x563656b14dd0 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f117d876740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b02a0 con 0x563656b1ffb0 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f117d876740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11741af500 con 0x563656b156f0 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f117d876740 10 client.0 _ll_register_callbacks cb 0x7f11600f4860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f1173400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b199c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f1173400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b199c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:44306/0 (socket says 172.21.15.33:44306) 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f1173400640 1 -- 172.21.15.33:0/3279794895 learned_addr learned my addr 172.21.15.33:0/3279794895 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.497+0000 7f1178a00640 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b19f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f1178a00640 1 -- 172.21.15.33:0/3279794895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 msgr2=0x563656b199c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f1178a00640 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b199c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f1178a00640 1 -- 172.21.15.33:0/3279794895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 msgr2=0x563656b28400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f1178a00640 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b28400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f1178a00640 1 -- 172.21.15.33:0/3279794895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563656a34cb0 con 0x563656b1ffb0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f1173400640 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b199c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f1178a00640 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b19f20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f116c005210 tx=0x7f116c011490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=811327c79ce6e61c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f115d600640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.498+0000 7f115d600640 1 -- 172.21.15.33:0/3279794895 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c03b060 con 0x563656b1ffb0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.499+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f116402b8d0 con 0x563656b1ffb0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.499+0000 7f115d600640 1 -- 172.21.15.33:0/3279794895 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f116c005430 con 0x563656b1ffb0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.499+0000 7f115d600640 1 -- 172.21.15.33:0/3279794895 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f116c042050 con 0x563656b1ffb0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.499+0000 7f117d876740 10 client.25622 fetch_fsmap learned FSMap version 43 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.499+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563656b9e790 con 0x563656b1ffb0 2024-04-26T11:34:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.499+0000 7f115d600640 1 -- 172.21.15.33:0/3279794895 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116402b8d0 con 0x563656b1ffb0 2024-04-26T11:34:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f115d600640 1 -- 172.21.15.33:0/3279794895 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 43) ==== 8342+0+0 (secure 0 0 0) 0x7f116c018040 con 0x563656b1ffb0 2024-04-26T11:34:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f117d876740 10 client.25622 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T11:34:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f117d876740 10 client.25622 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f117d876740 20 client.25622 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f117d876740 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b9ee80 0x563656ba12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f117d876740 4 client.25622 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:34:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563656a81e00 con 0x563656b9ee80 2024-04-26T11:34:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.500+0000 7f1173e00640 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b9ee80 0x563656ba12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.501+0000 7f1173e00640 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b9ee80 0x563656ba12a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5265ef417ae7088d server_cookie=82ddbac44aac9a6f in_seq=0 out_seq=0 2024-04-26T11:34:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.501+0000 7f115d600640 10 client.25622 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.514+0000 7f115d600640 1 -- 172.21.15.33:0/3279794895 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563656a81e00 con 0x563656b9ee80 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.514+0000 7f115d600640 10 client.25622 handle_command_reply: tid=0 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.514+0000 7f117d876740 1 client.25622 shutdown 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 msgr2=0x563656b19f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b19f20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f116c005210 tx=0x7f116c011490 comp rx=0 tx=0).stop 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 shutdown_connections 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563656b156f0 0x563656b199c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x563656b9ee80 0x563656ba12a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563656b1ffb0 0x563656b19f20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f115d600640 0 client.25622 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 --2- 172.21.15.33:0/3279794895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563656b14dd0 0x563656b28400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 >> 172.21.15.33:0/3279794895 conn(0x563656b112e0 msgr2=0x563656b17010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 shutdown_connections 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 1 -- 172.21.15.33:0/3279794895 wait complete. 2024-04-26T11:34:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.515+0000 7f117d876740 20 client.25622 trim_cache size 0 max 16384 2024-04-26T11:34:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:34:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:34:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:34:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f11580a3e40 msgr2=0x7f11580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f11580a3e40 0x7f11580a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x563656b16370 tx=0x7f1164005f90 comp rx=0 tx=0).stop 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174075550 msgr2=0x7f117419b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174075550 0x7f117419b750 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1160005ab0 tx=0x7f1160003c10 comp rx=0 tx=0).stop 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 shutdown_connections 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f11580a3e40 0x7f11580a6300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f117410f220 0x7f117419fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1174075550 0x7f117419b750 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 --2- 172.21.15.33:0/1876359500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1174074c30 0x7f117419b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 >> 172.21.15.33:0/1876359500 conn(0x7f1174079dc0 msgr2=0x7f11740ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 shutdown_connections 2024-04-26T11:34:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:04.516+0000 7f117aa00640 1 -- 172.21.15.33:0/1876359500 wait complete. 2024-04-26T11:34:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:04 smithi194 ceph-mon[33017]: pgmap v115: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.4 MiB/s wr, 445 op/s 2024-04-26T11:34:04.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:04 smithi033 ceph-mon[27845]: pgmap v115: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.4 MiB/s wr, 445 op/s 2024-04-26T11:34:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:04 smithi096 ceph-mon[32753]: pgmap v115: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.4 MiB/s wr, 445 op/s 2024-04-26T11:34:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:06 smithi033 ceph-mon[27845]: pgmap v116: 97 pgs: 97 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 6.8 MiB/s wr, 659 op/s 2024-04-26T11:34:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:06 smithi096 ceph-mon[32753]: pgmap v116: 97 pgs: 97 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 6.8 MiB/s wr, 659 op/s 2024-04-26T11:34:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:06 smithi194 ceph-mon[33017]: pgmap v116: 97 pgs: 97 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 6.8 MiB/s wr, 659 op/s 2024-04-26T11:34:07.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 170.0 s, 587.8 tps, lat 177.671 ms stddev 411.895 2024-04-26T11:34:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:08 smithi033 ceph-mon[27845]: pgmap v117: 97 pgs: 97 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.1 MiB/s wr, 504 op/s 2024-04-26T11:34:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:08 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-mon[32753]: pgmap v117: 97 pgs: 97 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.1 MiB/s wr, 504 op/s 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7faf382ba700 time 2024-04-26T11:34:08.545087+0000 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf428e33dd] 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: clone() 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: *** Caught signal (Aborted) ** 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: in thread 7faf382ba700 thread_name:ms_dispatch 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf4166bd20] 2024-04-26T11:34:08.685 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: gsignal() 2024-04-26T11:34:08.685 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: abort() 2024-04-26T11:34:08.685 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf428e3437] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.686 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 16: clone() 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.545+0000 7faf382ba700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7faf382ba700 time 2024-04-26T11:34:08.545087+0000 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf428e33dd] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.687 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: clone() 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.548+0000 7faf382ba700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: in thread 7faf382ba700 thread_name:ms_dispatch 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf4166bd20] 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: gsignal() 2024-04-26T11:34:08.688 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: abort() 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf428e3437] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 16: clone() 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: -1> 2024-04-26T11:34:08.545+0000 7faf382ba700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7faf382ba700 time 2024-04-26T11:34:08.545087+0000 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf428e33dd] 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: clone() 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 0> 2024-04-26T11:34:08.548+0000 7faf382ba700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: in thread 7faf382ba700 thread_name:ms_dispatch 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf4166bd20] 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: gsignal() 2024-04-26T11:34:08.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: abort() 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf428e3437] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 16: clone() 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:34:08.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: -1> 2024-04-26T11:34:08.545+0000 7faf382ba700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7faf382ba700 time 2024-04-26T11:34:08.545087+0000 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf428e33dd] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: clone() 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 0> 2024-04-26T11:34:08.548+0000 7faf382ba700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: in thread 7faf382ba700 thread_name:ms_dispatch 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf4166bd20] 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2: gsignal() 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 3: abort() 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf428e3437] 2024-04-26T11:34:08.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf428e35a3] 2024-04-26T11:34:08.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5572aaac4822] 2024-04-26T11:34:08.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5572aaa34598] 2024-04-26T11:34:08.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5572aaa36c17] 2024-04-26T11:34:08.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5572aa92fd0b] 2024-04-26T11:34:08.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5572aa9302ac] 2024-04-26T11:34:08.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5572aa918fad] 2024-04-26T11:34:08.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7faf42b77e20] 2024-04-26T11:34:08.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 13: (DispatchQueue::entry()+0x51f) [0x7faf42b74fdf] 2024-04-26T11:34:08.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf42c47fc1] 2024-04-26T11:34:08.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faf416611ca] 2024-04-26T11:34:08.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 16: clone() 2024-04-26T11:34:08.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:34:08.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66478]: 2024-04-26T11:34:08.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:08 smithi096 podman[66812]: 2024-04-26 11:34:08.616387201 +0000 UTC m=+0.020199203 container died b306ec7e8a256eac47223d3cd06811e15f8693c72ee63581545d5c4c2beaa0f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0) 2024-04-26T11:34:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:08 smithi194 ceph-mon[33017]: pgmap v117: 97 pgs: 97 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.1 MiB/s wr, 504 op/s 2024-04-26T11:34:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:08 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:09.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:09 smithi096 podman[66812]: 2024-04-26 11:34:09.310788042 +0000 UTC m=+0.714600034 container remove b306ec7e8a256eac47223d3cd06811e15f8693c72ee63581545d5c4c2beaa0f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T11:34:09.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:09 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:34:09.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:09 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-26T11:34:10.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:10 smithi033 ceph-mon[27845]: pgmap v118: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 32 MiB/s wr, 2.02k op/s 2024-04-26T11:34:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:10 smithi096 ceph-mon[32753]: pgmap v118: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 32 MiB/s wr, 2.02k op/s 2024-04-26T11:34:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:10 smithi194 ceph-mon[33017]: pgmap v118: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 32 MiB/s wr, 2.02k op/s 2024-04-26T11:34:12.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 175.0 s, 1840.0 tps, lat 29.321 ms stddev 32.041 2024-04-26T11:34:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:12 smithi033 ceph-mon[27845]: pgmap v119: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 MiB/s wr, 1.73k op/s 2024-04-26T11:34:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:12 smithi096 ceph-mon[32753]: pgmap v119: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 MiB/s wr, 1.73k op/s 2024-04-26T11:34:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:12 smithi194 ceph-mon[33017]: pgmap v119: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 MiB/s wr, 1.73k op/s 2024-04-26T11:34:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:14 smithi033 ceph-mon[27845]: pgmap v120: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 MiB/s wr, 1.73k op/s 2024-04-26T11:34:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:14 smithi096 ceph-mon[32753]: pgmap v120: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 MiB/s wr, 1.73k op/s 2024-04-26T11:34:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:14 smithi194 ceph-mon[33017]: pgmap v120: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 MiB/s wr, 1.73k op/s 2024-04-26T11:34:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:16 smithi033 ceph-mon[27845]: pgmap v121: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 31 MiB/s wr, 1.89k op/s 2024-04-26T11:34:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:16 smithi096 ceph-mon[32753]: pgmap v121: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 31 MiB/s wr, 1.89k op/s 2024-04-26T11:34:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:16 smithi194 ceph-mon[33017]: pgmap v121: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 31 MiB/s wr, 1.89k op/s 2024-04-26T11:34:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 180.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:34:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 -- 172.21.15.33:0/2857278258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0654076e60 msgr2=0x7f0654077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 --2- 172.21.15.33:0/2857278258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0654076e60 0x7f0654077240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0644009930 tx=0x7f064402a420 comp rx=0 tx=0).stop 2024-04-26T11:34:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 -- 172.21.15.33:0/2857278258 shutdown_connections 2024-04-26T11:34:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 --2- 172.21.15.33:0/2857278258 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06541137b0 0x7f0654115ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 --2- 172.21.15.33:0/2857278258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654077780 0x7f0654113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 --2- 172.21.15.33:0/2857278258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0654076e60 0x7f0654077240 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 -- 172.21.15.33:0/2857278258 >> 172.21.15.33:0/2857278258 conn(0x7f0654100620 msgr2=0x7f0654102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 -- 172.21.15.33:0/2857278258 shutdown_connections 2024-04-26T11:34:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.671+0000 7f065b000640 1 -- 172.21.15.33:0/2857278258 wait complete. 2024-04-26T11:34:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.672+0000 7f065b000640 1 Processor -- start 2024-04-26T11:34:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.672+0000 7f065b000640 1 -- start start 2024-04-26T11:34:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.672+0000 7f065b000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0654076e60 0x7f065419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.672+0000 7f065b000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654077780 0x7f065419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.672+0000 7f065b000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06541137b0 0x7f06541a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.672+0000 7f065b000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f06541191d0 con 0x7f0654076e60 2024-04-26T11:34:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.672+0000 7f065b000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0654119050 con 0x7f0654077780 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f065b000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0654119350 con 0x7f06541137b0 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0654076e60 0x7f065419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0654076e60 0x7f065419f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60694/0 (socket says 172.21.15.33:60694) 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 -- 172.21.15.33:0/484778618 learned_addr learned my addr 172.21.15.33:0/484778618 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f064be00640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654077780 0x7f065419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0659000640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06541137b0 0x7f06541a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 -- 172.21.15.33:0/484778618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06541137b0 msgr2=0x7f06541a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06541137b0 0x7f06541a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 -- 172.21.15.33:0/484778618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654077780 msgr2=0x7f065419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654077780 0x7f065419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.673+0000 7f0653e00640 1 -- 172.21.15.33:0/484778618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06541a4720 con 0x7f0654076e60 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.674+0000 7f0653e00640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0654076e60 0x7f065419f730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0644005940 tx=0x7f06440040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.674+0000 7f0651600640 1 -- 172.21.15.33:0/484778618 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064403f030 con 0x7f0654076e60 2024-04-26T11:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.674+0000 7f0651600640 1 -- 172.21.15.33:0/484778618 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06440044c0 con 0x7f0654076e60 2024-04-26T11:34:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.674+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06541a49b0 con 0x7f0654076e60 2024-04-26T11:34:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.674+0000 7f0651600640 1 -- 172.21.15.33:0/484778618 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644035070 con 0x7f0654076e60 2024-04-26T11:34:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.674+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06541ac2a0 con 0x7f0654076e60 2024-04-26T11:34:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.677+0000 7f065de6e740 1 Processor -- start 2024-04-26T11:34:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.677+0000 7f065de6e740 1 -- start start 2024-04-26T11:34:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.677+0000 7f0651600640 1 -- 172.21.15.33:0/484778618 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0644042030 con 0x7f0654076e60 2024-04-26T11:34:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f065de6e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40f9c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f065de6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fa6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f065de6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f065de6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557a40f01380 con 0x557a40fa6fb0 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f065de6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557a40e9dc30 con 0x557a40f9c6f0 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f065de6e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a40efd340 con 0x557a40f9bdd0 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f0653e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fa6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f0659000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40f9c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f0653e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fa6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51428/0 (socket says 172.21.15.33:51428) 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f0653e00640 1 -- 172.21.15.33:0/795298075 learned_addr learned my addr 172.21.15.33:0/795298075 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f0659000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40f9c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:35574/0 (socket says 172.21.15.33:35574) 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.678+0000 7f064be00640 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0659000640 1 -- 172.21.15.33:0/795298075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 msgr2=0x557a40fa6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0659000640 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fa6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0659000640 1 -- 172.21.15.33:0/795298075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 msgr2=0x557a40fa93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0659000640 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0659000640 1 -- 172.21.15.33:0/795298075 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a40ebf940 con 0x557a40f9bdd0 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0659000640 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40f9c1b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f064c005fd0 tx=0x7f064c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a0078a601e1975f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0651600640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f06380a3dc0 0x7f06380a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.679+0000 7f0649600640 1 -- 172.21.15.33:0/795298075 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c004d20 con 0x557a40f9bdd0 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f0649600640 1 -- 172.21.15.33:0/795298075 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064c004050 con 0x557a40f9bdd0 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f0651600640 1 -- 172.21.15.33:0/484778618 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f06440f0c20 con 0x7f0654076e60 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f0649600640 1 -- 172.21.15.33:0/795298075 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c005280 con 0x557a40f9bdd0 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f0653e00640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f06380a3dc0 0x7f06380a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 -- 172.21.15.33:0/795298075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 msgr2=0x557a40f9c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40f9c1b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f064c005fd0 tx=0x7f064c011810 comp rx=0 tx=0).stop 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f0653e00640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f06380a3dc0 0x7f06380a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f06440bcb10 tx=0x7f06440f46d0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 -- 172.21.15.33:0/795298075 shutdown_connections 2024-04-26T11:34:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fa6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 --2- 172.21.15.33:0/795298075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40f9c1b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 -- 172.21.15.33:0/795298075 >> 172.21.15.33:0/795298075 conn(0x557a40f982e0 msgr2=0x557a40f986c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 -- 172.21.15.33:0/795298075 shutdown_connections 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.680+0000 7f065de6e740 1 -- 172.21.15.33:0/795298075 wait complete. 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.681+0000 7f065de6e740 1 -- start start 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.681+0000 7f065de6e740 1 Processor -- start 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f065de6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9bdd0 0x557a40fa3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f065de6e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f065de6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f065de6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557a40fa9be0 con 0x557a40fa6fb0 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f065de6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557a40e88c60 con 0x557a40f9bdd0 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f065de6e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a40fa9d60 con 0x557a40f9c6f0 2024-04-26T11:34:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f065de6e740 10 client.0 _ll_register_callbacks cb 0x557a40eb4e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f0659000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9bdd0 0x557a40fa3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f0659000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:35580/0 (socket says 172.21.15.33:35580) 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 -- 172.21.15.33:0/1386010598 learned_addr learned my addr 172.21.15.33:0/1386010598 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 -- 172.21.15.33:0/1386010598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9bdd0 msgr2=0x557a40fa3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9bdd0 0x557a40fa3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 -- 172.21.15.33:0/1386010598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 msgr2=0x557a40fa4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.682+0000 7f064be00640 1 -- 172.21.15.33:0/1386010598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a40e97510 con 0x557a40f9c6f0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f064be00640 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa43f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f063c012700 tx=0x7f063c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e5d4b47848257b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f0642000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f0642000640 1 -- 172.21.15.33:0/1386010598 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c004020 con 0x557a40f9c6f0 2024-04-26T11:34:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f064c004050 con 0x557a40f9c6f0 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f0642000640 1 -- 172.21.15.33:0/1386010598 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f063c0041c0 con 0x557a40f9c6f0 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f0642000640 1 -- 172.21.15.33:0/1386010598 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f063c0043c0 con 0x557a40f9c6f0 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f065de6e740 10 client.25591 fetch_fsmap learned FSMap version 44 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x557a40c13210 con 0x557a40f9c6f0 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.683+0000 7f0642000640 1 -- 172.21.15.33:0/1386010598 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064c004050 con 0x557a40f9c6f0 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f0642000640 1 -- 172.21.15.33:0/1386010598 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f063c003ab0 con 0x557a40f9c6f0 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f065de6e740 10 client.25591 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f065de6e740 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f065de6e740 20 client.25591 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f065de6e740 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a40fb6d90 0x557a40fb9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f065de6e740 4 client.25591 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a40f0ef90 con 0x557a40fb6d90 2024-04-26T11:34:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.684+0000 7f0659000640 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a40fb6d90 0x557a40fb9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.685+0000 7f0659000640 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a40fb6d90 0x557a40fb9180 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93c763913f989fb0 server_cookie=9eeff2e5baaa570f in_seq=0 out_seq=0 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.685+0000 7f0642000640 10 client.25591 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f0642000640 1 -- 172.21.15.33:0/1386010598 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a40f0ef90 con 0x557a40fb6d90 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f0642000640 10 client.25591 handle_command_reply: tid=0 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 client.25591 shutdown 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 msgr2=0x557a40fa43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa43f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f063c012700 tx=0x7f063c012bc0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 shutdown_connections 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa43f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a40fb6d90 0x557a40fb9180 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fa4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 --2- 172.21.15.33:0/1386010598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9bdd0 0x557a40fa3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f0642000640 0 client.25591 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 >> 172.21.15.33:0/1386010598 conn(0x557a40f982e0 msgr2=0x557a40f9e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 shutdown_connections 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 1 -- 172.21.15.33:0/1386010598 wait complete. 2024-04-26T11:34:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.686+0000 7f065de6e740 20 client.25591 trim_cache size 0 max 16384 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 Processor -- start 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 -- start start 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40fb2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fbea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fbefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f06541b7520 con 0x557a40fa6fb0 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06541b4dc0 con 0x557a40f9c6f0 2024-04-26T11:34:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.703+0000 7f065de6e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f06541b2c10 con 0x557a40f9bdd0 2024-04-26T11:34:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fbea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f0659000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40fb2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fbea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51458/0 (socket says 172.21.15.33:51458) 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 -- 172.21.15.33:0/431010095 learned_addr learned my addr 172.21.15.33:0/431010095 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f0659000640 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fbefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 -- 172.21.15.33:0/431010095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 msgr2=0x557a40fb2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40fb2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 -- 172.21.15.33:0/431010095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 msgr2=0x557a40fbefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fbefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.704+0000 7f064be00640 1 -- 172.21.15.33:0/431010095 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a40f9c520 con 0x557a40f9c6f0 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f064be00640 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fbea90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f063c003f20 tx=0x7f063c009830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd191d9b036baef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f0648c00640 1 -- 172.21.15.33:0/431010095 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c01c670 con 0x557a40f9c6f0 2024-04-26T11:34:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f0648c00640 1 -- 172.21.15.33:0/431010095 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063c01ce40 con 0x557a40f9c6f0 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f0648c00640 1 -- 172.21.15.33:0/431010095 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c00fdc0 con 0x557a40f9c6f0 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f065de6e740 1 -- 172.21.15.33:0/431010095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 msgr2=0x557a40fbea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f065de6e740 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fbea90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f063c003f20 tx=0x7f063c009830 comp rx=0 tx=0).stop 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f065de6e740 1 -- 172.21.15.33:0/431010095 shutdown_connections 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f065de6e740 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40fa6fb0 0x557a40fbefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.705+0000 7f065de6e740 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40f9c6f0 0x557a40fbea90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.706+0000 7f065de6e740 1 --2- 172.21.15.33:0/431010095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9bdd0 0x557a40fb2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.706+0000 7f065de6e740 1 -- 172.21.15.33:0/431010095 >> 172.21.15.33:0/431010095 conn(0x557a40f982e0 msgr2=0x557a40fa27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.706+0000 7f065de6e740 1 -- 172.21.15.33:0/431010095 shutdown_connections 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.706+0000 7f065de6e740 1 -- 172.21.15.33:0/431010095 wait complete. 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.706+0000 7f065de6e740 1 -- start start 2024-04-26T11:34:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.706+0000 7f065de6e740 1 Processor -- start 2024-04-26T11:34:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f065de6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40f9bdd0 0x557a40faf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f065de6e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f065de6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40fa6fb0 0x557a40fa0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f065de6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f06541b3280 con 0x557a40f9bdd0 2024-04-26T11:34:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f065de6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06541b46f0 con 0x557a40fa6fb0 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f065de6e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f06541b3950 con 0x557a40f9c6f0 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f065de6e740 10 client.0 _ll_register_callbacks cb 0x7f063c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f0659000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40f9bdd0 0x557a40faf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f0659000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40f9bdd0 0x557a40faf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:60714/0 (socket says 172.21.15.33:60714) 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f0659000640 1 -- 172.21.15.33:0/324624982 learned_addr learned my addr 172.21.15.33:0/324624982 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f064be00640 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f0659000640 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40fa6fb0 0x557a40fa0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.707+0000 7f0659000640 1 -- 172.21.15.33:0/324624982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 msgr2=0x557a40fa0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0659000640 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0659000640 1 -- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40fa6fb0 msgr2=0x557a40fa0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0659000640 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40fa6fb0 0x557a40fa0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0659000640 1 -- 172.21.15.33:0/324624982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a40cdf620 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0659000640 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40f9bdd0 0x557a40faf240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f064c005fd0 tx=0x7f064c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff793a9ae20c19fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0642000640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0642000640 1 -- 172.21.15.33:0/324624982 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c003e30 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0642000640 1 -- 172.21.15.33:0/324624982 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8670+0+0 (secure 0 0 0) 0x7f064c004260 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557a40f08e00 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f0642000640 1 -- 172.21.15.33:0/324624982 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f064c041050 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f065de6e740 10 client.15456 fetch_fsmap learned FSMap version 44 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.708+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x557a40ebbcb0 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f0642000640 1 -- 172.21.15.33:0/324624982 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557a40f08e00 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f0642000640 1 -- 172.21.15.33:0/324624982 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f064c025040 con 0x557a40f9bdd0 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f065de6e740 10 client.15456 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f065de6e740 10 client.15456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f065de6e740 20 client.15456 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f065de6e740 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a4101d360 0x557a4101f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f065de6e740 4 client.15456 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:34:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a40e844b0 con 0x557a4101d360 2024-04-26T11:34:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.709+0000 7f064be00640 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a4101d360 0x557a4101f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.710+0000 7f064be00640 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a4101d360 0x557a4101f780 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dfa2915ad5682c3 server_cookie=15dcf9d665b95395 in_seq=0 out_seq=0 2024-04-26T11:34:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:17.710+0000 7f0642000640 10 client.15456 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:18.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:18 smithi033 ceph-mon[27845]: pgmap v122: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.68k op/s 2024-04-26T11:34:18.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:18 smithi096 ceph-mon[32753]: pgmap v122: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.68k op/s 2024-04-26T11:34:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:18 smithi194 ceph-mon[33017]: pgmap v122: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.68k op/s 2024-04-26T11:34:20.020 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:19 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:34:20.020 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:19 smithi096 systemd[1]: Stopped Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:34:20.020 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:19 smithi096 systemd[1]: Starting Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:34:20.020 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:19 smithi096 podman[66910]: 2024-04-26 11:34:19.821871157 +0000 UTC m=+0.018410926 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:34:20.280 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:20 smithi096 podman[66910]: 2024-04-26 11:34:20.019724136 +0000 UTC m=+0.216263907 container create 8cfcdf5c67cb20fdea885a98a9111937dc5f90afdd989f46c002e6835355829e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:34:20.280 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:20 smithi096 podman[66910]: 2024-04-26 11:34:20.25637996 +0000 UTC m=+0.452919737 container init 8cfcdf5c67cb20fdea885a98a9111937dc5f90afdd989f46c002e6835355829e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T11:34:20.280 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:20 smithi096 podman[66910]: 2024-04-26 11:34:20.259042631 +0000 UTC m=+0.455582399 container start 8cfcdf5c67cb20fdea885a98a9111937dc5f90afdd989f46c002e6835355829e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T11:34:20.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: pgmap v123: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 1.73k op/s 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: daemon mds.h restarted 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:boot 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:34:20.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:20 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: pgmap v123: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 1.73k op/s 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: daemon mds.h restarted 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:boot 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-mds[66928]: -- 172.21.15.96:0/1030446324 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ebd0339a40 con 0x55ebd030d000 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:20 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: starting mds.h at 2024-04-26T11:34:20.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:20 smithi096 bash[66910]: 8cfcdf5c67cb20fdea885a98a9111937dc5f90afdd989f46c002e6835355829e 2024-04-26T11:34:20.684 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:34:20 smithi096 systemd[1]: Started Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: pgmap v123: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 1.73k op/s 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: daemon mds.h restarted 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:boot 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:34:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:20 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:22.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 185.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:34:22.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:22 smithi033 ceph-mon[27845]: pgmap v125: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-04-26T11:34:22.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:22 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:resolve 2024-04-26T11:34:22.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:22 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:22 smithi096 ceph-mon[32753]: pgmap v125: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-04-26T11:34:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:22 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:resolve 2024-04-26T11:34:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:22 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:22 smithi194 ceph-mon[33017]: pgmap v125: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-04-26T11:34:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:22 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:resolve 2024-04-26T11:34:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:22 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.457+0000 7f0642000640 1 -- 172.21.15.33:0/324624982 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a40e844b0 con 0x557a4101d360 2024-04-26T11:34:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.457+0000 7f0642000640 10 client.15456 handle_command_reply: tid=0 2024-04-26T11:34:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.457+0000 7f065de6e740 1 client.15456 shutdown 2024-04-26T11:34:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40f9bdd0 msgr2=0x557a40faf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40f9bdd0 0x557a40faf240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f064c005fd0 tx=0x7f064c000f40 comp rx=0 tx=0).stop 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 shutdown_connections 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a40f9c6f0 0x557a40fa0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557a4101d360 0x557a4101f780 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557a40fa6fb0 0x557a40fa0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 --2- 172.21.15.33:0/324624982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557a40f9bdd0 0x557a40faf240 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 >> 172.21.15.33:0/324624982 conn(0x557a40f982e0 msgr2=0x557a40fa55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.458+0000 7f0642000640 0 client.15456 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.459+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 shutdown_connections 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.459+0000 7f065de6e740 1 -- 172.21.15.33:0/324624982 wait complete. 2024-04-26T11:34:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.459+0000 7f065de6e740 20 client.15456 trim_cache size 0 max 16384 2024-04-26T11:34:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:34:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:34:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:34:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:34:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.461+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f06380a3dc0 msgr2=0x7f06380a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.461+0000 7f065b000640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f06380a3dc0 0x7f06380a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f06440bcb10 tx=0x7f06440f46d0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.461+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0654076e60 msgr2=0x7f065419f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.461+0000 7f065b000640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0654076e60 0x7f065419f730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0644005940 tx=0x7f06440040c0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.461+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 shutdown_connections 2024-04-26T11:34:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.461+0000 7f065b000640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f06380a3dc0 0x7f06380a6280 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.462+0000 7f065b000640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06541137b0 0x7f06541a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.462+0000 7f065b000640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654077780 0x7f065419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.462+0000 7f065b000640 1 --2- 172.21.15.33:0/484778618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0654076e60 0x7f065419f730 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.462+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 >> 172.21.15.33:0/484778618 conn(0x7f0654100620 msgr2=0x7f0654102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.462+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 shutdown_connections 2024-04-26T11:34:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:23.462+0000 7f065b000640 1 -- 172.21.15.33:0/484778618 wait complete. 2024-04-26T11:34:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:23 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:reconnect 2024-04-26T11:34:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:23 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:23 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00999979 2024-04-26T11:34:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:23 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:34:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:23 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:reconnect 2024-04-26T11:34:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:23 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:23 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00999979 2024-04-26T11:34:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:23 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:34:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:23 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:reconnect 2024-04-26T11:34:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:23 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:23 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00999979 2024-04-26T11:34:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:23 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:24 smithi194 ceph-mon[33017]: pgmap v126: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-04-26T11:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:rejoin 2024-04-26T11:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:24 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:24 smithi194 ceph-mon[33017]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:34:24.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:24 smithi033 ceph-mon[27845]: pgmap v126: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-04-26T11:34:24.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:rejoin 2024-04-26T11:34:24.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:24 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:24.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:24 smithi033 ceph-mon[27845]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:34:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:24 smithi096 ceph-mon[32753]: pgmap v126: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-04-26T11:34:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:rejoin 2024-04-26T11:34:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:24 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:24 smithi096 ceph-mon[32753]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:34:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:25 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:34:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:25 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:34:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:active 2024-04-26T11:34:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:25 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:25 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:25 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:34:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:25 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:34:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:active 2024-04-26T11:34:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:25 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:25 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:25 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:34:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:25 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:34:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/2114098577,v1:172.21.15.96:6835/2114098577] up:active 2024-04-26T11:34:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:25 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:34:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:25 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:26 smithi194 ceph-mon[33017]: pgmap v127: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 834 KiB/s wr, 114 op/s 2024-04-26T11:34:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:26 smithi033 ceph-mon[27845]: pgmap v127: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 834 KiB/s wr, 114 op/s 2024-04-26T11:34:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:26 smithi096 ceph-mon[32753]: pgmap v127: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 834 KiB/s wr, 114 op/s 2024-04-26T11:34:27.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 190.0 s, 1362.3 tps, lat 254.019 ms stddev 1782.707 2024-04-26T11:34:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:28 smithi194 ceph-mon[33017]: pgmap v128: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 834 KiB/s wr, 114 op/s 2024-04-26T11:34:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:28 smithi033 ceph-mon[27845]: pgmap v128: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 834 KiB/s wr, 114 op/s 2024-04-26T11:34:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:28 smithi096 ceph-mon[32753]: pgmap v128: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 834 KiB/s wr, 114 op/s 2024-04-26T11:34:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:30 smithi194 ceph-mon[33017]: pgmap v129: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-04-26T11:34:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:30 smithi033 ceph-mon[27845]: pgmap v129: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-04-26T11:34:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:30 smithi096 ceph-mon[32753]: pgmap v129: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-04-26T11:34:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 195.0 s, 2534.7 tps, lat 39.279 ms stddev 43.729 2024-04-26T11:34:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:32 smithi194 ceph-mon[33017]: pgmap v130: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 4.5 MiB/s wr, 334 op/s 2024-04-26T11:34:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:32 smithi033 ceph-mon[27845]: pgmap v130: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 4.5 MiB/s wr, 334 op/s 2024-04-26T11:34:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:32 smithi096 ceph-mon[32753]: pgmap v130: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 4.5 MiB/s wr, 334 op/s 2024-04-26T11:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:34 smithi194 ceph-mon[33017]: pgmap v131: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-04-26T11:34:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:34 smithi033 ceph-mon[27845]: pgmap v131: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-04-26T11:34:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:34 smithi096 ceph-mon[32753]: pgmap v131: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 -- 172.21.15.33:0/549669894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 msgr2=0x7f00d4104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 --2- 172.21.15.33:0/549669894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 0x7f00d4104fb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009930 tx=0x7f00c002a440 comp rx=0 tx=0).stop 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 -- 172.21.15.33:0/549669894 shutdown_connections 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 --2- 172.21.15.33:0/549669894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 0x7f00d4104fb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 --2- 172.21.15.33:0/549669894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d41054f0 0x7f00d41119b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 --2- 172.21.15.33:0/549669894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d410abf0 0x7f00d410afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 -- 172.21.15.33:0/549669894 >> 172.21.15.33:0/549669894 conn(0x7f00d4100620 msgr2=0x7f00d4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 -- 172.21.15.33:0/549669894 shutdown_connections 2024-04-26T11:34:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.654+0000 7f00db400640 1 -- 172.21.15.33:0/549669894 wait complete. 2024-04-26T11:34:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.657+0000 7f00db400640 1 Processor -- start 2024-04-26T11:34:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.657+0000 7f00db400640 1 -- start start 2024-04-26T11:34:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.659+0000 7f00db400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 0x7f00d4111720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.659+0000 7f00db400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d41054f0 0x7f00d410e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.659+0000 7f00db400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d410abf0 0x7f00d410ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.659+0000 7f00db400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4114df0 con 0x7f00d410abf0 2024-04-26T11:34:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.659+0000 7f00db400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4114c70 con 0x7f00d41054f0 2024-04-26T11:34:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.659+0000 7f00db400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4114f70 con 0x7f00d4104b30 2024-04-26T11:34:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.659+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d41054f0 0x7f00d410e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d41054f0 0x7f00d410e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:44534/0 (socket says 172.21.15.33:44534) 2024-04-26T11:34:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d3e00640 1 -- 172.21.15.33:0/3971739695 learned_addr learned my addr 172.21.15.33:0/3971739695 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d8a00640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 0x7f00d4111720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d9400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d410abf0 0x7f00d410ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d3e00640 1 -- 172.21.15.33:0/3971739695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 msgr2=0x7f00d4111720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d3e00640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 0x7f00d4111720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d3e00640 1 -- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d410abf0 msgr2=0x7f00d410ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d3e00640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d410abf0 0x7f00d410ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d3e00640 1 -- 172.21.15.33:0/3971739695 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d410f610 con 0x7f00d41054f0 2024-04-26T11:34:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.660+0000 7f00d8a00640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 0x7f00d4111720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:34:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.661+0000 7f00d3e00640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d41054f0 0x7f00d410e810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f00c0031440 tx=0x7f00c0003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.661+0000 7f00d1600640 1 -- 172.21.15.33:0/3971739695 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c003f070 con 0x7f00d41054f0 2024-04-26T11:34:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.661+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d41abaf0 con 0x7f00d41054f0 2024-04-26T11:34:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.661+0000 7f00d1600640 1 -- 172.21.15.33:0/3971739695 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00c0031d60 con 0x7f00d41054f0 2024-04-26T11:34:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.661+0000 7f00d1600640 1 -- 172.21.15.33:0/3971739695 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0035030 con 0x7f00d41054f0 2024-04-26T11:34:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.661+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d41ac000 con 0x7f00d41054f0 2024-04-26T11:34:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.667+0000 7f00de1d1740 1 Processor -- start 2024-04-26T11:34:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.668+0000 7f00de1d1740 1 -- start start 2024-04-26T11:34:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.668+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95a3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.668+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95a35d0 0x5576d95bea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.668+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95befc0 0x5576d95c13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.668+0000 7f00de1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5576d9508380 con 0x5576d95a2cb0 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.668+0000 7f00de1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5576d94a4c30 con 0x5576d95a35d0 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.668+0000 7f00de1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5576d9504340 con 0x5576d95befc0 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.669+0000 7f00d8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95a35d0 0x5576d95bea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.669+0000 7f00d9400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.669+0000 7f00d8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95a35d0 0x5576d95bea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:44538/0 (socket says 172.21.15.33:44538) 2024-04-26T11:34:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.669+0000 7f00d8a00640 1 -- 172.21.15.33:0/2284844546 learned_addr learned my addr 172.21.15.33:0/2284844546 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.669+0000 7f00d8a00640 1 -- 172.21.15.33:0/2284844546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95befc0 msgr2=0x5576d95c13b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T11:34:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.670+0000 7f00d8a00640 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95befc0 0x5576d95c13b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.670+0000 7f00d8a00640 1 -- 172.21.15.33:0/2284844546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 msgr2=0x5576d95a3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.670+0000 7f00d8a00640 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95a3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.670+0000 7f00d8a00640 1 -- 172.21.15.33:0/2284844546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576d94c6940 con 0x5576d95a35d0 2024-04-26T11:34:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.670+0000 7f00d1600640 1 -- 172.21.15.33:0/3971739695 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f00c0042070 con 0x7f00d41054f0 2024-04-26T11:34:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.670+0000 7f00d9400640 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95a3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.671+0000 7f00d8a00640 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95a35d0 0x5576d95bea80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f00c800bc50 tx=0x7f00c800d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8569c94db2992d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.672+0000 7f00d1600640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f00bc0a3e40 0x7f00bc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.672+0000 7f00d1600640 1 -- 172.21.15.33:0/3971739695 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f00c00f11e0 con 0x7f00d41054f0 2024-04-26T11:34:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.672+0000 7f00d9400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f00bc0a3e40 0x7f00bc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.673+0000 7f00c5c00640 1 -- 172.21.15.33:0/2284844546 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c8017620 con 0x5576d95a35d0 2024-04-26T11:34:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.673+0000 7f00c5c00640 1 -- 172.21.15.33:0/2284844546 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00c8017c70 con 0x5576d95a35d0 2024-04-26T11:34:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.673+0000 7f00c5c00640 1 -- 172.21.15.33:0/2284844546 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c800fc00 con 0x5576d95a35d0 2024-04-26T11:34:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00de1d1740 1 -- 172.21.15.33:0/2284844546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95a35d0 msgr2=0x5576d95bea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95a35d0 0x5576d95bea80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f00c800bc50 tx=0x7f00c800d710 comp rx=0 tx=0).stop 2024-04-26T11:34:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00d9400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f00bc0a3e40 0x7f00bc0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f00cc011550 tx=0x7f00cc007680 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00de1d1740 1 -- 172.21.15.33:0/2284844546 shutdown_connections 2024-04-26T11:34:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95befc0 0x5576d95c13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95a35d0 0x5576d95bea80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2284844546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95a3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.674+0000 7f00de1d1740 1 -- 172.21.15.33:0/2284844546 >> 172.21.15.33:0/2284844546 conn(0x5576d959f2e0 msgr2=0x5576d959f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.675+0000 7f00de1d1740 1 -- 172.21.15.33:0/2284844546 shutdown_connections 2024-04-26T11:34:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.677+0000 7f00de1d1740 1 -- 172.21.15.33:0/2284844546 wait complete. 2024-04-26T11:34:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.678+0000 7f00de1d1740 1 -- start start 2024-04-26T11:34:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.678+0000 7f00de1d1740 1 Processor -- start 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95bc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95bc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95bcb50 0x5576d95cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00de1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5576d95c1bf0 con 0x5576d95a2cb0 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00de1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5576d948fc60 con 0x5576d95befc0 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00de1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5576d95c1d70 con 0x5576d95bcb50 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00de1d1740 10 client.0 _ll_register_callbacks cb 0x5576d94bbe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95bc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d8a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95bc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95bc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:44552/0 (socket says 172.21.15.33:44552) 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 -- 172.21.15.33:0/169819682 learned_addr learned my addr 172.21.15.33:0/169819682 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d9400640 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95bcb50 0x5576d95cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 -- 172.21.15.33:0/169819682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95bcb50 msgr2=0x5576d95cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95bcb50 0x5576d95cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 -- 172.21.15.33:0/169819682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 msgr2=0x5576d95bc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95bc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.679+0000 7f00d3e00640 1 -- 172.21.15.33:0/169819682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576d949e510 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00d3e00640 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95bc610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00c0000c00 tx=0x7f00c00bcbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df9a74602c367b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00ba000640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00ba000640 1 -- 172.21.15.33:0/169819682 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0049070 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00ba000640 1 -- 172.21.15.33:0/169819682 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f00c00c1080 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00c8017c70 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00ba000640 1 -- 172.21.15.33:0/169819682 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f00c00bd090 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00de1d1740 10 client.25655 fetch_fsmap learned FSMap version 51 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.680+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5576d921a210 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00ba000640 1 -- 172.21.15.33:0/169819682 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00c8017c70 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00ba000640 1 -- 172.21.15.33:0/169819682 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 51) ==== 8338+0+0 (secure 0 0 0) 0x7f00c00c5870 con 0x5576d95befc0 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00de1d1740 10 client.25655 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T11:34:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00de1d1740 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00de1d1740 20 client.25655 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00de1d1740 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d95cf2e0 0x5576d95d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00de1d1740 4 client.25655 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:34:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576d9515f90 con 0x5576d95cf2e0 2024-04-26T11:34:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.681+0000 7f00d8a00640 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d95cf2e0 0x5576d95d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.682+0000 7f00ba000640 10 client.25655 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.682+0000 7f00d8a00640 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d95cf2e0 0x5576d95d16f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd1064480a10a3ef server_cookie=de88a2903eb26406 in_seq=0 out_seq=0 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.683+0000 7f00ba000640 1 -- 172.21.15.33:0/169819682 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576d9515f90 con 0x5576d95cf2e0 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.683+0000 7f00ba000640 10 client.25655 handle_command_reply: tid=0 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.683+0000 7f00de1d1740 1 client.25655 shutdown 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 msgr2=0x5576d95bc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95bc610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00c0000c00 tx=0x7f00c00bcbd0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 shutdown_connections 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95bcb50 0x5576d95cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d95cf2e0 0x5576d95d16f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95bc610 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 --2- 172.21.15.33:0/169819682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95a2cb0 0x5576d95bc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 >> 172.21.15.33:0/169819682 conn(0x5576d959f2e0 msgr2=0x5576d95a50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00ba000640 0 client.25655 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.684+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 shutdown_connections 2024-04-26T11:34:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.685+0000 7f00de1d1740 1 -- 172.21.15.33:0/169819682 wait complete. 2024-04-26T11:34:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.685+0000 7f00de1d1740 20 client.25655 trim_cache size 0 max 16384 2024-04-26T11:34:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.715+0000 7f00de1d1740 1 Processor -- start 2024-04-26T11:34:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.715+0000 7f00de1d1740 1 -- start start 2024-04-26T11:34:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.716+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95d7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.716+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95bcb50 0x5576d95d7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.716+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95cab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.716+0000 7f00de1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00d41aed40 con 0x5576d95bcb50 2024-04-26T11:34:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.716+0000 7f00de1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00d41ac610 con 0x5576d95befc0 2024-04-26T11:34:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.716+0000 7f00de1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4050700 con 0x5576d95a2cb0 2024-04-26T11:34:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:44568/0 (socket says 172.21.15.33:44568) 2024-04-26T11:34:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 -- 172.21.15.33:0/2193563156 learned_addr learned my addr 172.21.15.33:0/2193563156 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d3e00640 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95bcb50 0x5576d95d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d8a00640 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 -- 172.21.15.33:0/2193563156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 msgr2=0x5576d95d7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95d7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 -- 172.21.15.33:0/2193563156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95bcb50 msgr2=0x5576d95d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95bcb50 0x5576d95d7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d9400640 1 -- 172.21.15.33:0/2193563156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576d92e6620 con 0x5576d95befc0 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.717+0000 7f00d8a00640 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95d7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.718+0000 7f00d3e00640 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95bcb50 0x5576d95d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.718+0000 7f00d9400640 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95cab20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00f690 tx=0x7f00cc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b41ae31651fcedf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.718+0000 7f00c5200640 1 -- 172.21.15.33:0/2193563156 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc02b070 con 0x5576d95befc0 2024-04-26T11:34:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.718+0000 7f00c5200640 1 -- 172.21.15.33:0/2193563156 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00cc0268c0 con 0x5576d95befc0 2024-04-26T11:34:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.718+0000 7f00c5200640 1 -- 172.21.15.33:0/2193563156 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc01d910 con 0x5576d95befc0 2024-04-26T11:34:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.719+0000 7f00de1d1740 1 -- 172.21.15.33:0/2193563156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 msgr2=0x5576d95cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.719+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95cab20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00f690 tx=0x7f00cc014000 comp rx=0 tx=0).stop 2024-04-26T11:34:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.720+0000 7f00de1d1740 1 -- 172.21.15.33:0/2193563156 shutdown_connections 2024-04-26T11:34:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.720+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95befc0 0x5576d95cab20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.720+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95bcb50 0x5576d95d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.720+0000 7f00de1d1740 1 --2- 172.21.15.33:0/2193563156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95d7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.720+0000 7f00de1d1740 1 -- 172.21.15.33:0/2193563156 >> 172.21.15.33:0/2193563156 conn(0x5576d959f2e0 msgr2=0x5576d95ba9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.721+0000 7f00de1d1740 1 -- 172.21.15.33:0/2193563156 shutdown_connections 2024-04-26T11:34:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.721+0000 7f00de1d1740 1 -- 172.21.15.33:0/2193563156 wait complete. 2024-04-26T11:34:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.722+0000 7f00de1d1740 1 -- start start 2024-04-26T11:34:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.722+0000 7f00de1d1740 1 Processor -- start 2024-04-26T11:34:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.723+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95c7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.723+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95bcb50 0x5576d95b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.723+0000 7f00de1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95befc0 0x5576d95b92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.723+0000 7f00d8a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d8a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95c7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:41276/0 (socket says 172.21.15.33:41276) 2024-04-26T11:34:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00de1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00d40733f0 con 0x5576d95befc0 2024-04-26T11:34:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00de1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4074860 con 0x5576d95bcb50 2024-04-26T11:34:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00de1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4073ac0 con 0x5576d95a2cb0 2024-04-26T11:34:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00de1d1740 10 client.0 _ll_register_callbacks cb 0x7f00c0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95bcb50 0x5576d95b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95bcb50 0x5576d95b8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:44574/0 (socket says 172.21.15.33:44574) 2024-04-26T11:34:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 -- 172.21.15.33:0/4198967653 learned_addr learned my addr 172.21.15.33:0/4198967653 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d9400640 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95befc0 0x5576d95b92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 -- 172.21.15.33:0/4198967653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 msgr2=0x5576d95c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 -- 172.21.15.33:0/4198967653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95befc0 msgr2=0x5576d95b92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95befc0 0x5576d95b92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 -- 172.21.15.33:0/4198967653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576d94c2cb0 con 0x5576d95bcb50 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00d3e00640 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95bcb50 0x5576d95b8ba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f00c00b9a60 tx=0x7f00c00bcd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e08310c3f78a81b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.724+0000 7f00ba000640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:34:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.725+0000 7f00ba000640 1 -- 172.21.15.33:0/4198967653 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0049040 con 0x5576d95bcb50 2024-04-26T11:34:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.725+0000 7f00ba000640 1 -- 172.21.15.33:0/4198967653 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f00c0031ab0 con 0x5576d95bcb50 2024-04-26T11:34:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.726+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00cc0268c0 con 0x5576d95bcb50 2024-04-26T11:34:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.727+0000 7f00ba000640 1 -- 172.21.15.33:0/4198967653 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f00c00c1440 con 0x5576d95bcb50 2024-04-26T11:34:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.727+0000 7f00de1d1740 10 client.25661 fetch_fsmap learned FSMap version 51 2024-04-26T11:34:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.727+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5576d94c2cb0 con 0x5576d95bcb50 2024-04-26T11:34:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.727+0000 7f00ba000640 1 -- 172.21.15.33:0/4198967653 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00cc0268c0 con 0x5576d95bcb50 2024-04-26T11:34:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.727+0000 7f00ba000640 1 -- 172.21.15.33:0/4198967653 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 51) ==== 8338+0+0 (secure 0 0 0) 0x7f00c00b8fe0 con 0x5576d95bcb50 2024-04-26T11:34:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.728+0000 7f00de1d1740 10 client.25661 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T11:34:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.728+0000 7f00de1d1740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.728+0000 7f00de1d1740 20 client.25661 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.728+0000 7f00de1d1740 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d9635c00 0x5576d9638020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.728+0000 7f00de1d1740 4 client.25661 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:34:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.728+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576d950fe00 con 0x5576d9635c00 2024-04-26T11:34:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.728+0000 7f00d8a00640 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d9635c00 0x5576d9638020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.729+0000 7f00d8a00640 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d9635c00 0x5576d9638020 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d64c93b521c5ff9 server_cookie=6012908145517314 in_seq=0 out_seq=0 2024-04-26T11:34:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.729+0000 7f00ba000640 10 client.25661 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.770+0000 7f00ba000640 1 -- 172.21.15.33:0/4198967653 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5576d950fe00 con 0x5576d9635c00 2024-04-26T11:34:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.770+0000 7f00ba000640 10 client.25661 handle_command_reply: tid=0 2024-04-26T11:34:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.770+0000 7f00de1d1740 1 client.25661 shutdown 2024-04-26T11:34:36.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.771+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95bcb50 msgr2=0x5576d95b8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.771+0000 7f00de1d1740 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95bcb50 0x5576d95b8ba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f00c00b9a60 tx=0x7f00c00bcd00 comp rx=0 tx=0).stop 2024-04-26T11:34:36.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 shutdown_connections 2024-04-26T11:34:36.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5576d9635c00 0x5576d9638020 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5576d95befc0 0x5576d95b92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576d95bcb50 0x5576d95b8ba0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 --2- 172.21.15.33:0/4198967653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d95a2cb0 0x5576d95c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 >> 172.21.15.33:0/4198967653 conn(0x5576d959f2e0 msgr2=0x5576d959dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00ba000640 0 client.25661 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 shutdown_connections 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.772+0000 7f00de1d1740 1 -- 172.21.15.33:0/4198967653 wait complete. 2024-04-26T11:34:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.773+0000 7f00de1d1740 20 client.25661 trim_cache size 0 max 16384 2024-04-26T11:34:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:34:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:34:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:34:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:34:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f00bc0a3e40 msgr2=0x7f00bc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f00bc0a3e40 0x7f00bc0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f00cc011550 tx=0x7f00cc007680 comp rx=0 tx=0).stop 2024-04-26T11:34:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d41054f0 msgr2=0x7f00d410e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d41054f0 0x7f00d410e810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f00c0031440 tx=0x7f00c0003d50 comp rx=0 tx=0).stop 2024-04-26T11:34:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 shutdown_connections 2024-04-26T11:34:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f00bc0a3e40 0x7f00bc0a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00d410abf0 0x7f00d410ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00d41054f0 0x7f00d410e810 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 --2- 172.21.15.33:0/3971739695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00d4104b30 0x7f00d4111720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.775+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 >> 172.21.15.33:0/3971739695 conn(0x7f00d4100620 msgr2=0x7f00d4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:36.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.776+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 shutdown_connections 2024-04-26T11:34:36.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:36.776+0000 7f00db400640 1 -- 172.21.15.33:0/3971739695 wait complete. 2024-04-26T11:34:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:36 smithi194 ceph-mon[33017]: pgmap v132: 97 pgs: 97 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 6.8 MiB/s wr, 596 op/s 2024-04-26T11:34:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:36 smithi033 ceph-mon[27845]: pgmap v132: 97 pgs: 97 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 6.8 MiB/s wr, 596 op/s 2024-04-26T11:34:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:36 smithi096 ceph-mon[32753]: pgmap v132: 97 pgs: 97 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 6.8 MiB/s wr, 596 op/s 2024-04-26T11:34:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 200.0 s, 2488.6 tps, lat 40.309 ms stddev 48.204 2024-04-26T11:34:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:38 smithi194 ceph-mon[33017]: pgmap v133: 97 pgs: 97 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 6.4 MiB/s wr, 549 op/s 2024-04-26T11:34:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:34:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:38 smithi033 ceph-mon[27845]: pgmap v133: 97 pgs: 97 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 6.4 MiB/s wr, 549 op/s 2024-04-26T11:34:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:34:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:38 smithi096 ceph-mon[32753]: pgmap v133: 97 pgs: 97 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 6.4 MiB/s wr, 549 op/s 2024-04-26T11:34:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:34:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:34:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:34:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:34:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:40 smithi194 ceph-mon[33017]: pgmap v134: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 9.1 MiB/s wr, 736 op/s 2024-04-26T11:34:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:40 smithi033 ceph-mon[27845]: pgmap v134: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 9.1 MiB/s wr, 736 op/s 2024-04-26T11:34:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:40 smithi096 ceph-mon[32753]: pgmap v134: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 9.1 MiB/s wr, 736 op/s 2024-04-26T11:34:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 205.0 s, 2328.6 tps, lat 42.988 ms stddev 48.461 2024-04-26T11:34:42.749 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:42 smithi096 ceph-mon[32753]: pgmap v135: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.7 MiB/s wr, 666 op/s 2024-04-26T11:34:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:42 smithi194 ceph-mon[33017]: pgmap v135: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.7 MiB/s wr, 666 op/s 2024-04-26T11:34:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:42 smithi033 ceph-mon[27845]: pgmap v135: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.7 MiB/s wr, 666 op/s 2024-04-26T11:34:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:44 smithi194 ceph-mon[33017]: pgmap v136: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 622 op/s 2024-04-26T11:34:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:44 smithi033 ceph-mon[27845]: pgmap v136: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 622 op/s 2024-04-26T11:34:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:44 smithi096 ceph-mon[32753]: pgmap v136: 97 pgs: 97 active+clean; 1.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 622 op/s 2024-04-26T11:34:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:46 smithi194 ceph-mon[33017]: pgmap v137: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 921 op/s 2024-04-26T11:34:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:46 smithi033 ceph-mon[27845]: pgmap v137: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 921 op/s 2024-04-26T11:34:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:46 smithi096 ceph-mon[32753]: pgmap v137: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 921 op/s 2024-04-26T11:34:47.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 210.0 s, 2893.2 tps, lat 34.680 ms stddev 37.493 2024-04-26T11:34:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:47 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:47 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:47 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:48.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:48 smithi033 ceph-mon[27845]: pgmap v138: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 632 op/s 2024-04-26T11:34:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:48 smithi194 ceph-mon[33017]: pgmap v138: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 632 op/s 2024-04-26T11:34:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:48 smithi096 ceph-mon[32753]: pgmap v138: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 632 op/s 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 -- 172.21.15.33:0/127511788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 msgr2=0x7fc7e4115ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 --2- 172.21.15.33:0/127511788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 0x7fc7e4115ba0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009cd0 tx=0x7fc7dc02a8f0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 -- 172.21.15.33:0/127511788 shutdown_connections 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 --2- 172.21.15.33:0/127511788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 0x7fc7e4115ba0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 --2- 172.21.15.33:0/127511788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7e4077780 0x7fc7e4113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 --2- 172.21.15.33:0/127511788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7e4076e60 0x7fc7e4077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 -- 172.21.15.33:0/127511788 >> 172.21.15.33:0/127511788 conn(0x7fc7e4100620 msgr2=0x7fc7e4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 -- 172.21.15.33:0/127511788 shutdown_connections 2024-04-26T11:34:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.959+0000 7fc7eba00640 1 -- 172.21.15.33:0/127511788 wait complete. 2024-04-26T11:34:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 Processor -- start 2024-04-26T11:34:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 -- start start 2024-04-26T11:34:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7e4076e60 0x7fc7e419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7e4077780 0x7fc7e419fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 0x7fc7e41a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e4119180 con 0x7fc7e4076e60 2024-04-26T11:34:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e4119000 con 0x7fc7e4077780 2024-04-26T11:34:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.961+0000 7fc7eba00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e4119300 con 0x7fc7e41137b0 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 0x7fc7e41a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 0x7fc7e41a3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:55398/0 (socket says 172.21.15.33:55398) 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e9a00640 1 -- 172.21.15.33:0/165206492 learned_addr learned my addr 172.21.15.33:0/165206492 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7e4077780 0x7fc7e419fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e9000640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7e4076e60 0x7fc7e419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e3e00640 1 -- 172.21.15.33:0/165206492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 msgr2=0x7fc7e41a3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 0x7fc7e41a3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e3e00640 1 -- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7e4076e60 msgr2=0x7fc7e419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:49.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.962+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7e4076e60 0x7fc7e419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.963+0000 7fc7e3e00640 1 -- 172.21.15.33:0/165206492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e41a4670 con 0x7fc7e4077780 2024-04-26T11:34:49.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.963+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7e4077780 0x7fc7e419fbc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000e9f0 tx=0x7fc7d000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:49.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.963+0000 7fc7e1600640 1 -- 172.21.15.33:0/165206492 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d000ccc0 con 0x7fc7e4077780 2024-04-26T11:34:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.963+0000 7fc7e1600640 1 -- 172.21.15.33:0/165206492 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d00108e0 con 0x7fc7e4077780 2024-04-26T11:34:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.963+0000 7fc7e1600640 1 -- 172.21.15.33:0/165206492 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0017820 con 0x7fc7e4077780 2024-04-26T11:34:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.964+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e41a4960 con 0x7fc7e4077780 2024-04-26T11:34:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.964+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e41ac1f0 con 0x7fc7e4077780 2024-04-26T11:34:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.970+0000 7fc7e1600640 1 -- 172.21.15.33:0/165206492 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc7d0010a90 con 0x7fc7e4077780 2024-04-26T11:34:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.971+0000 7fc7ee80c740 1 Processor -- start 2024-04-26T11:34:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.971+0000 7fc7ee80c740 1 -- start start 2024-04-26T11:34:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.971+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad62d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.971+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad637a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.971+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad63a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5632ad592380 con 0x5632ad62d6f0 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5632ad52ec30 con 0x5632ad637fb0 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5632ad58e340 con 0x5632ad62cdd0 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad62d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad62d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:55410/0 (socket says 172.21.15.33:55410) 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e9a00640 1 -- 172.21.15.33:0/4164035140 learned_addr learned my addr 172.21.15.33:0/4164035140 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e1600640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7cc0a3e90 0x7fc7cc0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e9000640 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad637a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad63a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7cc0a3e90 0x7fc7cc0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.972+0000 7fc7e1600640 1 -- 172.21.15.33:0/165206492 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7fc7d00d0e80 con 0x7fc7e4077780 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.973+0000 7fc7e9a00640 1 -- 172.21.15.33:0/4164035140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 msgr2=0x5632ad63a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.973+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad63a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.973+0000 7fc7e9a00640 1 -- 172.21.15.33:0/4164035140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 msgr2=0x5632ad637a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.973+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad637a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.973+0000 7fc7e9a00640 1 -- 172.21.15.33:0/4164035140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ad550940 con 0x5632ad62cdd0 2024-04-26T11:34:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.973+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad63a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.973+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad62d1b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc030770 tx=0x7fc7dc030800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972b25f6c6e8c501 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.974+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7cc0a3e90 0x7fc7cc0a6350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc048000 tx=0x7fc7dc04a000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.975+0000 7fc7d6000640 1 -- 172.21.15.33:0/4164035140 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc003b70 con 0x5632ad62cdd0 2024-04-26T11:34:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.975+0000 7fc7d6000640 1 -- 172.21.15.33:0/4164035140 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7dc036070 con 0x5632ad62cdd0 2024-04-26T11:34:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.975+0000 7fc7d6000640 1 -- 172.21.15.33:0/4164035140 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc04a6d0 con 0x5632ad62cdd0 2024-04-26T11:34:49.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.976+0000 7fc7ee80c740 1 -- 172.21.15.33:0/4164035140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 msgr2=0x5632ad62d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:49.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.976+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad62d1b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc030770 tx=0x7fc7dc030800 comp rx=0 tx=0).stop 2024-04-26T11:34:49.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.977+0000 7fc7ee80c740 1 -- 172.21.15.33:0/4164035140 shutdown_connections 2024-04-26T11:34:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.977+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad63a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.977+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad637a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.977+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/4164035140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad62d1b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.977+0000 7fc7ee80c740 1 -- 172.21.15.33:0/4164035140 >> 172.21.15.33:0/4164035140 conn(0x5632ad6292e0 msgr2=0x5632ad6296c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.977+0000 7fc7ee80c740 1 -- 172.21.15.33:0/4164035140 shutdown_connections 2024-04-26T11:34:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.977+0000 7fc7ee80c740 1 -- 172.21.15.33:0/4164035140 wait complete. 2024-04-26T11:34:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 -- start start 2024-04-26T11:34:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 Processor -- start 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad634f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad635490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad6359d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5632ad63ab20 con 0x5632ad62d6f0 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5632ad519c60 con 0x5632ad637fb0 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5632ad63aca0 con 0x5632ad62cdd0 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.978+0000 7fc7ee80c740 10 client.0 _ll_register_callbacks cb 0x5632ad545e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad634f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad634f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:55422/0 (socket says 172.21.15.33:55422) 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 -- 172.21.15.33:0/3272499714 learned_addr learned my addr 172.21.15.33:0/3272499714 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad6359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad635490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 -- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 msgr2=0x5632ad6359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad6359d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 -- 172.21.15.33:0/3272499714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 msgr2=0x5632ad635490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad635490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 -- 172.21.15.33:0/3272499714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ad528510 con 0x5632ad62cdd0 2024-04-26T11:34:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad635490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7e9000640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad634f50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8012740 tx=0x7fc7d8012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=225dd56ef357e3fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.979+0000 7fc7caa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.980+0000 7fc7caa00640 1 -- 172.21.15.33:0/3272499714 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d800f920 con 0x5632ad62cdd0 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.980+0000 7fc7caa00640 1 -- 172.21.15.33:0/3272499714 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7fc7d800fac0 con 0x5632ad62cdd0 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.981+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7dc036070 con 0x5632ad62cdd0 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.981+0000 7fc7caa00640 1 -- 172.21.15.33:0/3272499714 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fc7d800fcc0 con 0x5632ad62cdd0 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.982+0000 7fc7ee80c740 10 client.25621 fetch_fsmap learned FSMap version 52 2024-04-26T11:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.982+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5632ad2a4210 con 0x5632ad62cdd0 2024-04-26T11:34:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.982+0000 7fc7caa00640 1 -- 172.21.15.33:0/3272499714 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7dc036070 con 0x5632ad62cdd0 2024-04-26T11:34:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.982+0000 7fc7caa00640 1 -- 172.21.15.33:0/3272499714 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7fc7d8010430 con 0x5632ad62cdd0 2024-04-26T11:34:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.983+0000 7fc7ee80c740 10 client.25621 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T11:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.983+0000 7fc7ee80c740 10 client.25621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.983+0000 7fc7ee80c740 20 client.25621 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.983+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad647ce0 0x5632ad64a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.983+0000 7fc7ee80c740 4 client.25621 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.983+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632ad59ff90 con 0x5632ad647ce0 2024-04-26T11:34:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.984+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad647ce0 0x5632ad64a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.986+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad647ce0 0x5632ad64a110 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=796d479b053cc89a server_cookie=191610e40b05e94a in_seq=0 out_seq=0 2024-04-26T11:34:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.986+0000 7fc7caa00640 10 client.25621 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.986+0000 7fc7caa00640 1 -- 172.21.15.33:0/3272499714 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632ad59ff90 con 0x5632ad647ce0 2024-04-26T11:34:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.986+0000 7fc7caa00640 10 client.25621 handle_command_reply: tid=0 2024-04-26T11:34:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 client.25621 shutdown 2024-04-26T11:34:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 msgr2=0x5632ad634f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad634f50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8012740 tx=0x7fc7d8012c00 comp rx=0 tx=0).stop 2024-04-26T11:34:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 shutdown_connections 2024-04-26T11:34:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad647ce0 0x5632ad64a110 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad6359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad635490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/3272499714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad634f50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 >> 172.21.15.33:0/3272499714 conn(0x5632ad6292e0 msgr2=0x5632ad62f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.987+0000 7fc7caa00640 0 client.25621 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.988+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 shutdown_connections 2024-04-26T11:34:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.988+0000 7fc7ee80c740 1 -- 172.21.15.33:0/3272499714 wait complete. 2024-04-26T11:34:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:49.988+0000 7fc7ee80c740 20 client.25621 trim_cache size 0 max 16384 2024-04-26T11:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.017+0000 7fc7ee80c740 1 Processor -- start 2024-04-26T11:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.017+0000 7fc7ee80c740 1 -- start start 2024-04-26T11:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad6439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad64fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad64ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e41b76b0 con 0x5632ad62d6f0 2024-04-26T11:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e41b4f50 con 0x5632ad637fb0 2024-04-26T11:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e41b2da0 con 0x5632ad62cdd0 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7e3e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad64fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7e3e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad64fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:43234/0 (socket says 172.21.15.33:43234) 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7e3e00640 1 -- 172.21.15.33:0/428324770 learned_addr learned my addr 172.21.15.33:0/428324770 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.018+0000 7fc7e9a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad64ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.019+0000 7fc7e9000640 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad6439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.019+0000 7fc7e9a00640 1 -- 172.21.15.33:0/428324770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 msgr2=0x5632ad6439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.019+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad6439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.019+0000 7fc7e9a00640 1 -- 172.21.15.33:0/428324770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 msgr2=0x5632ad64fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.019+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad64fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.019+0000 7fc7e9a00640 1 -- 172.21.15.33:0/428324770 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ad370620 con 0x5632ad637fb0 2024-04-26T11:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.019+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad64fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.020+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad64ff60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0307a0 tx=0x7fc7dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76e288afc96f5511 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.021+0000 7fc7d5600640 1 -- 172.21.15.33:0/428324770 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc030c90 con 0x5632ad637fb0 2024-04-26T11:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.021+0000 7fc7d5600640 1 -- 172.21.15.33:0/428324770 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7dc036040 con 0x5632ad637fb0 2024-04-26T11:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.021+0000 7fc7d5600640 1 -- 172.21.15.33:0/428324770 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc04a6d0 con 0x5632ad637fb0 2024-04-26T11:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.022+0000 7fc7ee80c740 1 -- 172.21.15.33:0/428324770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 msgr2=0x5632ad64ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.022+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad64ff60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0307a0 tx=0x7fc7dc0046c0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.024+0000 7fc7ee80c740 1 -- 172.21.15.33:0/428324770 shutdown_connections 2024-04-26T11:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.024+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad637fb0 0x5632ad64ff60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.024+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad62d6f0 0x5632ad64fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.024+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/428324770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62cdd0 0x5632ad6439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.024+0000 7fc7ee80c740 1 -- 172.21.15.33:0/428324770 >> 172.21.15.33:0/428324770 conn(0x5632ad6292e0 msgr2=0x5632ad630290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.024+0000 7fc7ee80c740 1 -- 172.21.15.33:0/428324770 shutdown_connections 2024-04-26T11:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.024+0000 7fc7ee80c740 1 -- 172.21.15.33:0/428324770 wait complete. 2024-04-26T11:34:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 -- start start 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 Processor -- start 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad62cdd0 0x5632ad63bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62d6f0 0x5632ad63c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad637fb0 0x5632ad63c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e41b3410 con 0x5632ad637fb0 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e41b4880 con 0x5632ad62cdd0 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e41b3ae0 con 0x5632ad62d6f0 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.025+0000 7fc7ee80c740 10 client.0 _ll_register_callbacks cb 0x7fc7d8009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62d6f0 0x5632ad63c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62d6f0 0x5632ad63c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:55440/0 (socket says 172.21.15.33:55440) 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 -- 172.21.15.33:0/705991391 learned_addr learned my addr 172.21.15.33:0/705991391 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e9000640 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad62cdd0 0x5632ad63bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e9a00640 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad637fb0 0x5632ad63c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 -- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad62cdd0 msgr2=0x5632ad63bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad62cdd0 0x5632ad63bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 -- 172.21.15.33:0/705991391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad637fb0 msgr2=0x5632ad63c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:34:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad637fb0 0x5632ad63c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.026+0000 7fc7e3e00640 1 -- 172.21.15.33:0/705991391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ad54ccb0 con 0x5632ad62d6f0 2024-04-26T11:34:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.027+0000 7fc7e3e00640 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62d6f0 0x5632ad63c130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc7d009ca30 tx=0x7fc7d0009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dbc471dd0d3459a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:34:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.027+0000 7fc7caa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:34:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.028+0000 7fc7caa00640 1 -- 172.21.15.33:0/705991391 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00b5070 con 0x5632ad62d6f0 2024-04-26T11:34:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.028+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7dc036040 con 0x5632ad62d6f0 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.028+0000 7fc7caa00640 1 -- 172.21.15.33:0/705991391 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7fc7d009b030 con 0x5632ad62d6f0 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.029+0000 7fc7caa00640 1 -- 172.21.15.33:0/705991391 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fc7d00b5070 con 0x5632ad62d6f0 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.029+0000 7fc7ee80c740 10 client.25627 fetch_fsmap learned FSMap version 52 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.029+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5632ad6ae710 con 0x5632ad62d6f0 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.029+0000 7fc7caa00640 1 -- 172.21.15.33:0/705991391 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7dc036040 con 0x5632ad62d6f0 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.029+0000 7fc7caa00640 1 -- 172.21.15.33:0/705991391 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7fc7d00993a0 con 0x5632ad62d6f0 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7ee80c740 10 client.25627 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7ee80c740 10 client.25627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7ee80c740 20 client.25627 populate_metadata read hostname 'smithi033' 2024-04-26T11:34:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad6aeea0 0x5632ad6b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:34:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7ee80c740 4 client.25627 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:34:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632ad599e00 con 0x5632ad6aeea0 2024-04-26T11:34:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7e9000640 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad6aeea0 0x5632ad6b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:34:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.030+0000 7fc7e9000640 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad6aeea0 0x5632ad6b12c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b330fec3dea667b server_cookie=2d59e3c71b8b5e57 in_seq=0 out_seq=0 2024-04-26T11:34:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.031+0000 7fc7caa00640 10 client.25627 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.044+0000 7fc7caa00640 1 -- 172.21.15.33:0/705991391 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632ad599e00 con 0x5632ad6aeea0 2024-04-26T11:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.044+0000 7fc7caa00640 10 client.25627 handle_command_reply: tid=0 2024-04-26T11:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.044+0000 7fc7ee80c740 1 client.25627 shutdown 2024-04-26T11:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62d6f0 msgr2=0x5632ad63c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62d6f0 0x5632ad63c130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc7d009ca30 tx=0x7fc7d0009700 comp rx=0 tx=0).stop 2024-04-26T11:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 shutdown_connections 2024-04-26T11:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ad62d6f0 0x5632ad63c130 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5632ad6aeea0 0x5632ad6b12c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5632ad637fb0 0x5632ad63c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 --2- 172.21.15.33:0/705991391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5632ad62cdd0 0x5632ad63bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7caa00640 0 client.25627 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 >> 172.21.15.33:0/705991391 conn(0x5632ad6292e0 msgr2=0x5632ad62e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 shutdown_connections 2024-04-26T11:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 1 -- 172.21.15.33:0/705991391 wait complete. 2024-04-26T11:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.045+0000 7fc7ee80c740 20 client.25627 trim_cache size 0 max 16384 2024-04-26T11:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.046+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7cc0a3e90 msgr2=0x7fc7cc0a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.046+0000 7fc7eba00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7cc0a3e90 0x7fc7cc0a6350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc048000 tx=0x7fc7dc04a000 comp rx=0 tx=0).stop 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.046+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7e4077780 msgr2=0x7fc7e419fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.046+0000 7fc7eba00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7e4077780 0x7fc7e419fbc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000e9f0 tx=0x7fc7d000eeb0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.047+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 shutdown_connections 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.047+0000 7fc7eba00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fc7cc0a3e90 0x7fc7cc0a6350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.047+0000 7fc7eba00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7e41137b0 0x7fc7e41a3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.047+0000 7fc7eba00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7e4077780 0x7fc7e419fbc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.047+0000 7fc7eba00640 1 --2- 172.21.15.33:0/165206492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7e4076e60 0x7fc7e419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.047+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 >> 172.21.15.33:0/165206492 conn(0x7fc7e4100620 msgr2=0x7fc7e4102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.047+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 shutdown_connections 2024-04-26T11:34:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:34:50.048+0000 7fc7eba00640 1 -- 172.21.15.33:0/165206492 wait complete. 2024-04-26T11:34:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:50 smithi194 ceph-mon[33017]: pgmap v139: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 876 op/s 2024-04-26T11:34:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:50 smithi033 ceph-mon[27845]: pgmap v139: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 876 op/s 2024-04-26T11:34:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:50 smithi096 ceph-mon[32753]: pgmap v139: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 876 op/s 2024-04-26T11:34:52.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 215.0 s, 2615.8 tps, lat 38.044 ms stddev 43.669 2024-04-26T11:34:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:52 smithi194 ceph-mon[33017]: pgmap v140: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 8.2 MiB/s wr, 748 op/s 2024-04-26T11:34:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:52 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:52 smithi033 ceph-mon[27845]: pgmap v140: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 8.2 MiB/s wr, 748 op/s 2024-04-26T11:34:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:52 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:52 smithi096 ceph-mon[32753]: pgmap v140: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 8.2 MiB/s wr, 748 op/s 2024-04-26T11:34:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:52 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:34:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:54 smithi194 ceph-mon[33017]: pgmap v141: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 5.8 MiB/s wr, 603 op/s 2024-04-26T11:34:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:54 smithi033 ceph-mon[27845]: pgmap v141: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 5.8 MiB/s wr, 603 op/s 2024-04-26T11:34:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:54 smithi096 ceph-mon[32753]: pgmap v141: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 5.8 MiB/s wr, 603 op/s 2024-04-26T11:34:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:56 smithi194 ceph-mon[33017]: pgmap v142: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.8 MiB/s wr, 890 op/s 2024-04-26T11:34:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:56 smithi033 ceph-mon[27845]: pgmap v142: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.8 MiB/s wr, 890 op/s 2024-04-26T11:34:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:56 smithi096 ceph-mon[32753]: pgmap v142: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.8 MiB/s wr, 890 op/s 2024-04-26T11:34:57.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 220.0 s, 2569.6 tps, lat 38.889 ms stddev 43.839 2024-04-26T11:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:34:58 smithi194 ceph-mon[33017]: pgmap v143: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.9 MiB/s wr, 591 op/s 2024-04-26T11:34:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:34:58 smithi033 ceph-mon[27845]: pgmap v143: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.9 MiB/s wr, 591 op/s 2024-04-26T11:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:34:58 smithi096 ceph-mon[32753]: pgmap v143: 97 pgs: 97 active+clean; 2.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.9 MiB/s wr, 591 op/s 2024-04-26T11:35:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:00 smithi194 ceph-mon[33017]: pgmap v144: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 9.2 MiB/s wr, 828 op/s 2024-04-26T11:35:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:00 smithi033 ceph-mon[27845]: pgmap v144: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 9.2 MiB/s wr, 828 op/s 2024-04-26T11:35:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:00 smithi096 ceph-mon[32753]: pgmap v144: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 9.2 MiB/s wr, 828 op/s 2024-04-26T11:35:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 225.0 s, 1985.3 tps, lat 47.248 ms stddev 83.212 2024-04-26T11:35:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:02 smithi194 ceph-mon[33017]: pgmap v145: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 637 op/s 2024-04-26T11:35:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:02 smithi033 ceph-mon[27845]: pgmap v145: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 637 op/s 2024-04-26T11:35:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:02 smithi096 ceph-mon[32753]: pgmap v145: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 637 op/s 2024-04-26T11:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.204+0000 7f4f7d200640 1 -- 172.21.15.33:0/4209031343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 msgr2=0x7f4f78104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.204+0000 7f4f7d200640 1 --2- 172.21.15.33:0/4209031343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 0x7f4f78104fc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009900 tx=0x7f4f6002a400 comp rx=0 tx=0).stop 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.205+0000 7f4f7d200640 1 -- 172.21.15.33:0/4209031343 shutdown_connections 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.205+0000 7f4f7d200640 1 --2- 172.21.15.33:0/4209031343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f78105500 0x7f4f781119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.205+0000 7f4f7d200640 1 --2- 172.21.15.33:0/4209031343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 0x7f4f78104fc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.205+0000 7f4f7d200640 1 --2- 172.21.15.33:0/4209031343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 0x7f4f7810afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.205+0000 7f4f7d200640 1 -- 172.21.15.33:0/4209031343 >> 172.21.15.33:0/4209031343 conn(0x7f4f78100630 msgr2=0x7f4f78102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.205+0000 7f4f7d200640 1 -- 172.21.15.33:0/4209031343 shutdown_connections 2024-04-26T11:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.205+0000 7f4f7d200640 1 -- 172.21.15.33:0/4209031343 wait complete. 2024-04-26T11:35:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 Processor -- start 2024-04-26T11:35:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 -- start start 2024-04-26T11:35:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 0x7f4f7819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f78105500 0x7f4f7819b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 0x7f4f7819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f78114df0 con 0x7f4f78104b40 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f78114c70 con 0x7f4f7810ac00 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f7d200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4f78114f70 con 0x7f4f78105500 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f76800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 0x7f4f7819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f75e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 0x7f4f7819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f76800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 0x7f4f7819fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:43060/0 (socket says 172.21.15.33:43060) 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f75e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 0x7f4f7819b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34144/0 (socket says 172.21.15.33:34144) 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.206+0000 7f4f76800640 1 -- 172.21.15.33:0/2172535846 learned_addr learned my addr 172.21.15.33:0/2172535846 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f75400640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f78105500 0x7f4f7819b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f76800640 1 -- 172.21.15.33:0/2172535846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f78105500 msgr2=0x7f4f7819b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f76800640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f78105500 0x7f4f7819b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f76800640 1 -- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 msgr2=0x7f4f7819b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f76800640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 0x7f4f7819b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f76800640 1 -- 172.21.15.33:0/2172535846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f781a0170 con 0x7f4f7810ac00 2024-04-26T11:35:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f76800640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 0x7f4f7819fa40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800c970 tx=0x7f4f6800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.207+0000 7f4f6ea00640 1 -- 172.21.15.33:0/2172535846 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f68007bb0 con 0x7f4f7810ac00 2024-04-26T11:35:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.208+0000 7f4f6ea00640 1 -- 172.21.15.33:0/2172535846 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f68007d50 con 0x7f4f7810ac00 2024-04-26T11:35:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.208+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f781a0460 con 0x7f4f7810ac00 2024-04-26T11:35:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.208+0000 7f4f6ea00640 1 -- 172.21.15.33:0/2172535846 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f68018770 con 0x7f4f7810ac00 2024-04-26T11:35:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.208+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f781a7cf0 con 0x7f4f7810ac00 2024-04-26T11:35:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.211+0000 7f4f6ea00640 1 -- 172.21.15.33:0/2172535846 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4f6800f990 con 0x7f4f7810ac00 2024-04-26T11:35:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.211+0000 7f4f7ff6f740 1 Processor -- start 2024-04-26T11:35:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.211+0000 7f4f7ff6f740 1 -- start start 2024-04-26T11:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.211+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.211+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d85a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.211+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d883a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fdd9ce0380 con 0x55fdd9d7add0 2024-04-26T11:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fdd9c7cc30 con 0x55fdd9d85fb0 2024-04-26T11:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fdd9cdc340 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d85a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34152/0 (socket says 172.21.15.33:34152) 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75400640 1 -- 172.21.15.33:0/2025948354 learned_addr learned my addr 172.21.15.33:0/2025948354 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75400640 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d883a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75e00640 1 -- 172.21.15.33:0/2025948354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 msgr2=0x55fdd9d883a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75e00640 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d883a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75e00640 1 -- 172.21.15.33:0/2025948354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 msgr2=0x55fdd9d7b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75e00640 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.212+0000 7f4f75e00640 1 -- 172.21.15.33:0/2025948354 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdd9c9e940 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.213+0000 7f4f6ea00640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4f5c0a3e40 0x7f4f5c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.213+0000 7f4f75e00640 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d85a70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009fa0 tx=0x7f4f700118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78452dc6aa6dd423 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.213+0000 7f4f6ea00640 1 -- 172.21.15.33:0/2172535846 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f4f680d0940 con 0x7f4f7810ac00 2024-04-26T11:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.213+0000 7f4f75400640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4f5c0a3e40 0x7f4f5c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.213+0000 7f4f67400640 1 -- 172.21.15.33:0/2025948354 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70003c30 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.213+0000 7f4f67400640 1 -- 172.21.15.33:0/2025948354 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f70003dd0 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.213+0000 7f4f67400640 1 -- 172.21.15.33:0/2025948354 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000f9c0 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2025948354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7b6f0 msgr2=0x55fdd9d85a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d85a70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009fa0 tx=0x7f4f700118d0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f75400640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4f5c0a3e40 0x7f4f5c0a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55fdd9d78700 tx=0x7f4f60000bc0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2025948354 shutdown_connections 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d85a70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d883a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2025948354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2025948354 >> 172.21.15.33:0/2025948354 conn(0x55fdd9d772e0 msgr2=0x55fdd9d776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2025948354 shutdown_connections 2024-04-26T11:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2025948354 wait complete. 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.214+0000 7f4f7ff6f740 1 -- start start 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 1 Processor -- start 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d82f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d83480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d839c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fdd9d88be0 con 0x55fdd9d7add0 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fdd9c67c60 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fdd9d88d60 con 0x55fdd9d85fb0 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f7ff6f740 10 client.0 _ll_register_callbacks cb 0x55fdd9c93e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.215+0000 7f4f75400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d83480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d83480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:43090/0 (socket says 172.21.15.33:43090) 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 -- 172.21.15.33:0/3141787239 learned_addr learned my addr 172.21.15.33:0/3141787239 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75e00640 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d82f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f76800640 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d839c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 -- 172.21.15.33:0/3141787239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 msgr2=0x55fdd9d839c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d839c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 -- 172.21.15.33:0/3141787239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 msgr2=0x55fdd9d82f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d82f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 -- 172.21.15.33:0/3141787239 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdd9c76510 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f75400640 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d83480 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4f60033000 tx=0x7f4f6002a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54e4827178d175e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.216+0000 7f4f5be00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f5be00640 1 -- 172.21.15.33:0/3141787239 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6004f030 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f70003dd0 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f5be00640 1 -- 172.21.15.33:0/3141787239 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f4f60004960 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f5be00640 1 -- 172.21.15.33:0/3141787239 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4f60046070 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f7ff6f740 10 client.25691 fetch_fsmap learned FSMap version 53 2024-04-26T11:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55fdd99f2210 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f5be00640 1 -- 172.21.15.33:0/3141787239 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f70003dd0 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f5be00640 1 -- 172.21.15.33:0/3141787239 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f4f6003dba0 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f7ff6f740 10 client.25691 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T11:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f7ff6f740 10 client.25691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.217+0000 7f4f7ff6f740 20 client.25691 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.218+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9d95e00 0x55fdd9d98250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.218+0000 7f4f7ff6f740 4 client.25691 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.218+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdd9cedf90 con 0x55fdd9d95e00 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.218+0000 7f4f75e00640 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9d95e00 0x55fdd9d98250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.219+0000 7f4f75e00640 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9d95e00 0x55fdd9d98250 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5617ef3da258ff2f server_cookie=5aeb01fd89f4afa2 in_seq=0 out_seq=0 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.219+0000 7f4f5be00640 10 client.25691 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.219+0000 7f4f5be00640 1 -- 172.21.15.33:0/3141787239 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdd9cedf90 con 0x55fdd9d95e00 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.219+0000 7f4f5be00640 10 client.25691 handle_command_reply: tid=0 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.219+0000 7f4f7ff6f740 1 client.25691 shutdown 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7b6f0 msgr2=0x55fdd9d83480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d83480 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4f60033000 tx=0x7f4f6002a400 comp rx=0 tx=0).stop 2024-04-26T11:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 shutdown_connections 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d839c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9d95e00 0x55fdd9d98250 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d83480 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3141787239 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7add0 0x55fdd9d82f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f5be00640 0 client.25691 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 >> 172.21.15.33:0/3141787239 conn(0x55fdd9d772e0 msgr2=0x55fdd9d7d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 shutdown_connections 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3141787239 wait complete. 2024-04-26T11:35:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.220+0000 7f4f7ff6f740 20 client.25691 trim_cache size 0 max 16384 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 Processor -- start 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 -- start start 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7add0 0x55fdd9d85670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d91aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d7cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781b3070 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781b08e0 con 0x55fdd9d85fb0 2024-04-26T11:35:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.238+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781ae730 con 0x55fdd9d7add0 2024-04-26T11:35:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d91aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d91aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34156/0 (socket says 172.21.15.33:34156) 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 -- 172.21.15.33:0/2706978153 learned_addr learned my addr 172.21.15.33:0/2706978153 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f76800640 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d7cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75e00640 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7add0 0x55fdd9d85670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 -- 172.21.15.33:0/2706978153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7add0 msgr2=0x55fdd9d85670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7add0 0x55fdd9d85670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 -- 172.21.15.33:0/2706978153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 msgr2=0x55fdd9d7cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d7cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.239+0000 7f4f75400640 1 -- 172.21.15.33:0/2706978153 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdd9abe620 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.240+0000 7f4f75400640 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d91aa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4f6002a920 tx=0x7f4f60031640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6fe56f1ac0cff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.240+0000 7f4f66a00640 1 -- 172.21.15.33:0/2706978153 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6004f070 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.240+0000 7f4f66a00640 1 -- 172.21.15.33:0/2706978153 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f60004e10 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.240+0000 7f4f66a00640 1 -- 172.21.15.33:0/2706978153 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60035040 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2706978153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 msgr2=0x55fdd9d91aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d91aa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4f6002a920 tx=0x7f4f60031640 comp rx=0 tx=0).stop 2024-04-26T11:35:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2706978153 shutdown_connections 2024-04-26T11:35:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d7cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d91aa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/2706978153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d7add0 0x55fdd9d85670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2706978153 >> 172.21.15.33:0/2706978153 conn(0x55fdd9d772e0 msgr2=0x55fdd9d7c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2706978153 shutdown_connections 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/2706978153 wait complete. 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 -- start start 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.241+0000 7f4f7ff6f740 1 Processor -- start 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d80290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f7ff6f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781aeda0 con 0x55fdd9d7b6f0 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781b0210 con 0x55fdd9d7add0 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f7ff6f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781af470 con 0x55fdd9d85fb0 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f7ff6f740 10 client.0 _ll_register_callbacks cb 0x7f4f60032e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f75e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f75e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:43096/0 (socket says 172.21.15.33:43096) 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f75e00640 1 -- 172.21.15.33:0/3386065469 learned_addr learned my addr 172.21.15.33:0/3386065469 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f76800640 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.242+0000 7f4f75400640 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d80290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f75e00640 1 -- 172.21.15.33:0/3386065469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 msgr2=0x55fdd9d807d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f75e00640 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d807d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f75e00640 1 -- 172.21.15.33:0/3386065469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 msgr2=0x55fdd9d80290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f75e00640 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d80290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f75e00640 1 -- 172.21.15.33:0/3386065469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdd9c9acb0 con 0x55fdd9d7add0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f75e00640 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7fd50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009fd0 tx=0x7f4f70009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3d1e051009fa4f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f5be00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f5be00640 1 -- 172.21.15.33:0/3386065469 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7002bb50 con 0x55fdd9d7add0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f60004e10 con 0x55fdd9d7add0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f5be00640 1 -- 172.21.15.33:0/3386065469 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f4f7002bcf0 con 0x55fdd9d7add0 2024-04-26T11:35:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.243+0000 7f4f5be00640 1 -- 172.21.15.33:0/3386065469 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4f70039050 con 0x55fdd9d7add0 2024-04-26T11:35:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 10 client.25697 fetch_fsmap learned FSMap version 53 2024-04-26T11:35:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55fdd9dfc620 con 0x55fdd9d7add0 2024-04-26T11:35:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f5be00640 1 -- 172.21.15.33:0/3386065469 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f60004e10 con 0x55fdd9d7add0 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f5be00640 1 -- 172.21.15.33:0/3386065469 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f4f7000f8e0 con 0x55fdd9d7add0 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 10 client.25697 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 20 client.25697 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9dfcc40 0x55fdd9dff060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 4 client.25697 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdd9ce7e00 con 0x55fdd9dfcc40 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.244+0000 7f4f75400640 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9dfcc40 0x55fdd9dff060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.245+0000 7f4f75400640 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9dfcc40 0x55fdd9dff060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da46bfe74cb3598 server_cookie=38e393d4e66f7839 in_seq=0 out_seq=0 2024-04-26T11:35:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.245+0000 7f4f5be00640 10 client.25697 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f5be00640 1 -- 172.21.15.33:0/3386065469 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdd9ce7e00 con 0x55fdd9dfcc40 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f5be00640 10 client.25697 handle_command_reply: tid=0 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 client.25697 shutdown 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7add0 msgr2=0x55fdd9d7fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7fd50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009fd0 tx=0x7f4f70009820 comp rx=0 tx=0).stop 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 shutdown_connections 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdd9d85fb0 0x55fdd9d807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55fdd9dfcc40 0x55fdd9dff060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fdd9d7b6f0 0x55fdd9d80290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 --2- 172.21.15.33:0/3386065469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fdd9d7add0 0x55fdd9d7fd50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f5be00640 0 client.25697 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.258+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 >> 172.21.15.33:0/3386065469 conn(0x55fdd9d772e0 msgr2=0x55fdd9d76440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:03.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.259+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 shutdown_connections 2024-04-26T11:35:03.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.259+0000 7f4f7ff6f740 1 -- 172.21.15.33:0/3386065469 wait complete. 2024-04-26T11:35:03.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.259+0000 7f4f7ff6f740 20 client.25697 trim_cache size 0 max 16384 2024-04-26T11:35:03.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:35:03.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:35:03.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:35:03.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:35:03.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.259+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4f5c0a3e40 msgr2=0x7f4f5c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.259+0000 7f4f7d200640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4f5c0a3e40 0x7f4f5c0a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55fdd9d78700 tx=0x7f4f60000bc0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 msgr2=0x7f4f7819fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 0x7f4f7819fa40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800c970 tx=0x7f4f6800ce30 comp rx=0 tx=0).stop 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 shutdown_connections 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4f5c0a3e40 0x7f4f5c0a6300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f78105500 0x7f4f7819b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f7810ac00 0x7f4f7819fa40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 --2- 172.21.15.33:0/2172535846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f78104b40 0x7f4f7819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 >> 172.21.15.33:0/2172535846 conn(0x7f4f78100630 msgr2=0x7f4f78101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 shutdown_connections 2024-04-26T11:35:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:03.260+0000 7f4f7d200640 1 -- 172.21.15.33:0/2172535846 wait complete. 2024-04-26T11:35:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:04 smithi194 ceph-mon[33017]: pgmap v146: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 578 op/s 2024-04-26T11:35:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:04 smithi033 ceph-mon[27845]: pgmap v146: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 578 op/s 2024-04-26T11:35:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:04 smithi096 ceph-mon[32753]: pgmap v146: 97 pgs: 97 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 578 op/s 2024-04-26T11:35:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:06 smithi194 ceph-mon[33017]: pgmap v147: 97 pgs: 97 active+clean; 2.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.0 MiB/s wr, 783 op/s 2024-04-26T11:35:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:06 smithi033 ceph-mon[27845]: pgmap v147: 97 pgs: 97 active+clean; 2.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.0 MiB/s wr, 783 op/s 2024-04-26T11:35:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:06 smithi096 ceph-mon[32753]: pgmap v147: 97 pgs: 97 active+clean; 2.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.0 MiB/s wr, 783 op/s 2024-04-26T11:35:07.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 230.0 s, 216.0 tps, lat 457.461 ms stddev 281.208 2024-04-26T11:35:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:08 smithi194 ceph-mon[33017]: pgmap v148: 97 pgs: 97 active+clean; 2.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 495 op/s 2024-04-26T11:35:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:08 smithi033 ceph-mon[27845]: pgmap v148: 97 pgs: 97 active+clean; 2.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 495 op/s 2024-04-26T11:35:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:08 smithi096 ceph-mon[32753]: pgmap v148: 97 pgs: 97 active+clean; 2.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 495 op/s 2024-04-26T11:35:09.667 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fce805e4700 time 2024-04-26T11:35:09.191273+0000 2024-04-26T11:35:09.667 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:35:09.667 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.667 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8ac0d3dd] 2024-04-26T11:35:09.667 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: clone() 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.192+0000 7fce805e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fce805e4700 time 2024-04-26T11:35:09.191273+0000 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:35:09.668 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8ac0d3dd] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.669 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: clone() 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: *** Caught signal (Aborted) ** 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: in thread 7fce805e4700 thread_name:ms_dispatch 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fce89995d20] 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: gsignal() 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: abort() 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fce8ac0d437] 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.670 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 16: clone() 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.193+0000 7fce805e4700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: in thread 7fce805e4700 thread_name:ms_dispatch 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fce89995d20] 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: gsignal() 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: abort() 2024-04-26T11:35:09.671 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fce8ac0d437] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 16: clone() 2024-04-26T11:35:09.672 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: -1> 2024-04-26T11:35:09.192+0000 7fce805e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fce805e4700 time 2024-04-26T11:35:09.191273+0000 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8ac0d3dd] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.673 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: clone() 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 0> 2024-04-26T11:35:09.193+0000 7fce805e4700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: in thread 7fce805e4700 thread_name:ms_dispatch 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fce89995d20] 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: gsignal() 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: abort() 2024-04-26T11:35:09.674 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fce8ac0d437] 2024-04-26T11:35:09.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 16: clone() 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: -1> 2024-04-26T11:35:09.192+0000 7fce805e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fce805e4700 time 2024-04-26T11:35:09.191273+0000 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8ac0d3dd] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: clone() 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 0> 2024-04-26T11:35:09.193+0000 7fce805e4700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: in thread 7fce805e4700 thread_name:ms_dispatch 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fce89995d20] 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2: gsignal() 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 3: abort() 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fce8ac0d437] 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fce8ac0d5a3] 2024-04-26T11:35:09.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5573a8cd5822] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5573a8c45598] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5573a8c47c17] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5573a8b40d0b] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5573a8b412ac] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5573a8b29fad] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fce8aea1e20] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 13: (DispatchQueue::entry()+0x51f) [0x7fce8ae9efdf] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fce8af71fc1] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fce8998b1ca] 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 16: clone() 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[66768]: 2024-04-26T11:35:09.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:09 smithi096 podman[67117]: 2024-04-26 11:35:09.288447992 +0000 UTC m=+0.044698870 container died 92a09f568a26e590de3b2c387b2d9ceceb24ae51618e047af2c03bc8d92d0294 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T11:35:10.294 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:10 smithi096 podman[67117]: 2024-04-26 11:35:10.020135205 +0000 UTC m=+0.776386070 container remove 92a09f568a26e590de3b2c387b2d9ceceb24ae51618e047af2c03bc8d92d0294 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T11:35:10.294 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:10 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:35:10.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:10 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-26T11:35:10.603 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:10 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.227s CPU time. 2024-04-26T11:35:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:10 smithi194 ceph-mon[33017]: pgmap v149: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 MiB/s wr, 1.33k op/s 2024-04-26T11:35:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:10 smithi033 ceph-mon[27845]: pgmap v149: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 MiB/s wr, 1.33k op/s 2024-04-26T11:35:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:10 smithi096 ceph-mon[32753]: pgmap v149: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 MiB/s wr, 1.33k op/s 2024-04-26T11:35:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 235.0 s, 1689.3 tps, lat 63.433 ms stddev 185.333 2024-04-26T11:35:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:12 smithi033 ceph-mon[27845]: pgmap v150: 97 pgs: 97 active+clean; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 27 MiB/s wr, 1.51k op/s 2024-04-26T11:35:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:12 smithi096 ceph-mon[32753]: pgmap v150: 97 pgs: 97 active+clean; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 27 MiB/s wr, 1.51k op/s 2024-04-26T11:35:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:12 smithi194 ceph-mon[33017]: pgmap v150: 97 pgs: 97 active+clean; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 27 MiB/s wr, 1.51k op/s 2024-04-26T11:35:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:14 smithi033 ceph-mon[27845]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.46k op/s 2024-04-26T11:35:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:14 smithi096 ceph-mon[32753]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.46k op/s 2024-04-26T11:35:15.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:14 smithi194 ceph-mon[33017]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.46k op/s 2024-04-26T11:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.412+0000 7f4dffe00640 1 -- 172.21.15.33:0/1066417734 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00103100 msgr2=0x7f4e0010f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.412+0000 7f4dffe00640 1 --2- 172.21.15.33:0/1066417734 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00103100 0x7f4e0010f610 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009cd0 tx=0x7f4df802a8e0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.413+0000 7f4dffe00640 1 -- 172.21.15.33:0/1066417734 shutdown_connections 2024-04-26T11:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.413+0000 7f4dffe00640 1 --2- 172.21.15.33:0/1066417734 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00103100 0x7f4e0010f610 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.413+0000 7f4dffe00640 1 --2- 172.21.15.33:0/1066417734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 0x7f4e00102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.413+0000 7f4dffe00640 1 --2- 172.21.15.33:0/1066417734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e00108750 0x7f4e00108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.413+0000 7f4dffe00640 1 -- 172.21.15.33:0/1066417734 >> 172.21.15.33:0/1066417734 conn(0x7f4e000fe270 msgr2=0x7f4e00100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.413+0000 7f4dffe00640 1 -- 172.21.15.33:0/1066417734 shutdown_connections 2024-04-26T11:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.413+0000 7f4dffe00640 1 -- 172.21.15.33:0/1066417734 wait complete. 2024-04-26T11:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 Processor -- start 2024-04-26T11:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 -- start start 2024-04-26T11:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 0x7f4e00198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e00103100 0x7f4e001994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00108750 0x7f4e0019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e00112a60 con 0x7f4e00103100 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001128e0 con 0x7f4e00102740 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.414+0000 7f4dffe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e00112be0 con 0x7f4e00108750 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 0x7f4e00198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfe000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e00103100 0x7f4e001994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 0x7f4e00198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33160/0 (socket says 172.21.15.33:33160) 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 -- 172.21.15.33:0/2457905511 learned_addr learned my addr 172.21.15.33:0/2457905511 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dff400640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00108750 0x7f4e0019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 -- 172.21.15.33:0/2457905511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00108750 msgr2=0x7f4e0019d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00108750 0x7f4e0019d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 -- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e00103100 msgr2=0x7f4e001994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e00103100 0x7f4e001994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 -- 172.21.15.33:0/2457905511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0019df40 con 0x7f4e00102740 2024-04-26T11:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfe000640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e00103100 0x7f4e001994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.415+0000 7f4dfea00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 0x7f4e00198fa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4df000e9f0 tx=0x7f4df000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.416+0000 7f4df7400640 1 -- 172.21.15.33:0/2457905511 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df000cd80 con 0x7f4e00102740 2024-04-26T11:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.416+0000 7f4df7400640 1 -- 172.21.15.33:0/2457905511 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4df00108e0 con 0x7f4e00102740 2024-04-26T11:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.416+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0019e230 con 0x7f4e00102740 2024-04-26T11:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.416+0000 7f4df7400640 1 -- 172.21.15.33:0/2457905511 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df0018980 con 0x7f4e00102740 2024-04-26T11:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.416+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e001a5ac0 con 0x7f4e00102740 2024-04-26T11:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.419+0000 7f4e08baf740 1 Processor -- start 2024-04-26T11:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.419+0000 7f4df7400640 1 -- 172.21.15.33:0/2457905511 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4df0010430 con 0x7f4e00102740 2024-04-26T11:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.419+0000 7f4e08baf740 1 -- start start 2024-04-26T11:35:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cdb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4e08baf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557511c2f380 con 0x557511cc9ec0 2024-04-26T11:35:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4e08baf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557511bcbc30 con 0x557511cd95d0 2024-04-26T11:35:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4e08baf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557511c2b340 con 0x557511cca7e0 2024-04-26T11:35:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4dff400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4dff400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39412/0 (socket says 172.21.15.33:39412) 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4dff400640 1 -- 172.21.15.33:0/1963475335 learned_addr learned my addr 172.21.15.33:0/1963475335 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4dfea00640 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.420+0000 7f4dfe000640 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cdb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4dff400640 1 -- 172.21.15.33:0/1963475335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 msgr2=0x557511cd9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4dff400640 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4dff400640 1 -- 172.21.15.33:0/1963475335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 msgr2=0x557511cdb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4dff400640 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cdb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4dff400640 1 -- 172.21.15.33:0/1963475335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557511bed940 con 0x557511cc9ec0 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4dfe000640 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cdb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4df7400640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4de40a3d90 0x7f4de40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4dff400640 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cca2a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4df8035000 tx=0x7f4df8005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd63184d741884f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.421+0000 7f4df7400640 1 -- 172.21.15.33:0/2457905511 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f4df00d0b20 con 0x7f4e00102740 2024-04-26T11:35:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.422+0000 7f4debe00640 1 -- 172.21.15.33:0/1963475335 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8031850 con 0x557511cc9ec0 2024-04-26T11:35:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.422+0000 7f4dfea00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4de40a3d90 0x7f4de40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.422+0000 7f4debe00640 1 -- 172.21.15.33:0/1963475335 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4df8036070 con 0x557511cc9ec0 2024-04-26T11:35:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.422+0000 7f4debe00640 1 -- 172.21.15.33:0/1963475335 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80044e0 con 0x557511cc9ec0 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.422+0000 7f4e08baf740 1 -- 172.21.15.33:0/1963475335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 msgr2=0x557511cca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.422+0000 7f4dfea00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4de40a3d90 0x7f4de40a6250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4df0098e80 tx=0x7f4df00d45d0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.422+0000 7f4e08baf740 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cca2a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4df8035000 tx=0x7f4df8005f80 comp rx=0 tx=0).stop 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 -- 172.21.15.33:0/1963475335 shutdown_connections 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cdb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 --2- 172.21.15.33:0/1963475335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cca2a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 -- 172.21.15.33:0/1963475335 >> 172.21.15.33:0/1963475335 conn(0x557511cc64f0 msgr2=0x557511cc68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 -- 172.21.15.33:0/1963475335 shutdown_connections 2024-04-26T11:35:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 -- 172.21.15.33:0/1963475335 wait complete. 2024-04-26T11:35:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 -- start start 2024-04-26T11:35:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.423+0000 7f4e08baf740 1 Processor -- start 2024-04-26T11:35:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cd65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511cd6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cd95d0 0x557511cd7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4e08baf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557511cdc1a0 con 0x557511cc9ec0 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4e08baf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557511bb6c60 con 0x557511cca7e0 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4e08baf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557511cdc320 con 0x557511cd95d0 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4e08baf740 10 client.0 _ll_register_callbacks cb 0x557511be2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4dff400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cd65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4dff400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cd65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:39414/0 (socket says 172.21.15.33:39414) 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4dfe000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511cd6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4dff400640 1 -- 172.21.15.33:0/2913237193 learned_addr learned my addr 172.21.15.33:0/2913237193 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.424+0000 7f4dff400640 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cd95d0 0x557511cd7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4dfe000640 1 -- 172.21.15.33:0/2913237193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cd95d0 msgr2=0x557511cd7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4dfe000640 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cd95d0 0x557511cd7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4dfe000640 1 -- 172.21.15.33:0/2913237193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 msgr2=0x557511cd65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4dfe000640 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cd65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4dfe000640 1 -- 172.21.15.33:0/2913237193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557511bc5510 con 0x557511cca7e0 2024-04-26T11:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4dff400640 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cd65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4dfe000640 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511cd6b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4e00100900 tx=0x7f4dec012ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68c8e20695bc5d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4de8c00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4de8c00640 1 -- 172.21.15.33:0/2913237193 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec004030 con 0x557511cca7e0 2024-04-26T11:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4de8c00640 1 -- 172.21.15.33:0/2913237193 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f4dec0041d0 con 0x557511cca7e0 2024-04-26T11:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4df8036070 con 0x557511cca7e0 2024-04-26T11:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4de8c00640 1 -- 172.21.15.33:0/2913237193 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4dec0043d0 con 0x557511cca7e0 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4e08baf740 10 client.25709 fetch_fsmap learned FSMap version 53 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.425+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x557511941210 con 0x557511cca7e0 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4de8c00640 1 -- 172.21.15.33:0/2913237193 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4df8036070 con 0x557511cca7e0 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4de8c00640 1 -- 172.21.15.33:0/2913237193 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f4dec003ae0 con 0x557511cca7e0 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4e08baf740 10 client.25709 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4e08baf740 10 client.25709 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4e08baf740 20 client.25709 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511ce94c0 0x557511ceb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4e08baf740 4 client.25709 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.426+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557511c3cf90 con 0x557511ce94c0 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.427+0000 7f4dff400640 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511ce94c0 0x557511ceb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.427+0000 7f4dff400640 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511ce94c0 0x557511ceb8d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da326d09e0631332 server_cookie=b9326970d48a18d2 in_seq=0 out_seq=0 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.427+0000 7f4de8c00640 10 client.25709 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.428+0000 7f4de8c00640 1 -- 172.21.15.33:0/2913237193 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557511c3cf90 con 0x557511ce94c0 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.428+0000 7f4de8c00640 10 client.25709 handle_command_reply: tid=0 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.428+0000 7f4e08baf740 1 client.25709 shutdown 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 msgr2=0x557511cd6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511cd6b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4e00100900 tx=0x7f4dec012ca0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 shutdown_connections 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cd95d0 0x557511cd7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511ce94c0 0x557511ceb8d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511cd6b00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2913237193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511cd65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4de8c00640 0 client.25709 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 >> 172.21.15.33:0/2913237193 conn(0x557511cc64f0 msgr2=0x557511ccc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 shutdown_connections 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 1 -- 172.21.15.33:0/2913237193 wait complete. 2024-04-26T11:35:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.429+0000 7f4e08baf740 20 client.25709 trim_cache size 0 max 16384 2024-04-26T11:35:16.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.446+0000 7f4e08baf740 1 Processor -- start 2024-04-26T11:35:16.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4e08baf740 1 -- start start 2024-04-26T11:35:16.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511ce50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cd83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4e08baf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001aec90 con 0x557511cc9ec0 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4e08baf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001ac500 con 0x557511cd95d0 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4e08baf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001aa350 con 0x557511cca7e0 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4dfe000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.447+0000 7f4dff400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511ce50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:38550/0 (socket says 172.21.15.33:38550) 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 -- 172.21.15.33:0/2869271274 learned_addr learned my addr 172.21.15.33:0/2869271274 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dff400640 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cd83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 -- 172.21.15.33:0/2869271274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 msgr2=0x557511cd83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cd83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 -- 172.21.15.33:0/2869271274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 msgr2=0x557511ce50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511ce50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 -- 172.21.15.33:0/2869271274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557511a0d620 con 0x557511cca7e0 2024-04-26T11:35:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dff400640 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511ce50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:35:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.448+0000 7f4dfe000640 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd5720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0058c0 tx=0x7f4dec005600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a927ba7c3d47ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4deb400640 1 -- 172.21.15.33:0/2869271274 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec004000 con 0x557511cca7e0 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4deb400640 1 -- 172.21.15.33:0/2869271274 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dec0041a0 con 0x557511cca7e0 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4deb400640 1 -- 172.21.15.33:0/2869271274 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec01c600 con 0x557511cca7e0 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4e08baf740 1 -- 172.21.15.33:0/2869271274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 msgr2=0x557511cd5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd5720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0058c0 tx=0x7f4dec005600 comp rx=0 tx=0).stop 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4e08baf740 1 -- 172.21.15.33:0/2869271274 shutdown_connections 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cca7e0 0x557511cd5720 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cd95d0 0x557511cd83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4e08baf740 1 --2- 172.21.15.33:0/2869271274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cc9ec0 0x557511ce50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.449+0000 7f4e08baf740 1 -- 172.21.15.33:0/2869271274 >> 172.21.15.33:0/2869271274 conn(0x557511cc64f0 msgr2=0x557511cd7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.450+0000 7f4e08baf740 1 -- 172.21.15.33:0/2869271274 shutdown_connections 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.450+0000 7f4e08baf740 1 -- 172.21.15.33:0/2869271274 wait complete. 2024-04-26T11:35:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.450+0000 7f4e08baf740 1 -- start start 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.450+0000 7f4e08baf740 1 Processor -- start 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cc9ec0 0x557511ce15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511ce1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4e08baf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cd95d0 0x557511cdcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4e08baf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001aa9c0 con 0x557511cd95d0 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4e08baf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001abe30 con 0x557511cca7e0 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4e08baf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e001ab090 con 0x557511cc9ec0 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4e08baf740 10 client.0 _ll_register_callbacks cb 0x7f4dec005a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4dff400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cc9ec0 0x557511ce15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4dff400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cd95d0 0x557511cdcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4dff400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cc9ec0 0x557511ce15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:38552/0 (socket says 172.21.15.33:38552) 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4dfe000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511ce1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.451+0000 7f4dff400640 1 -- 172.21.15.33:0/4127660139 learned_addr learned my addr 172.21.15.33:0/4127660139 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4dff400640 1 -- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 msgr2=0x557511ce1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4dff400640 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511ce1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4dff400640 1 -- 172.21.15.33:0/4127660139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cd95d0 msgr2=0x557511cdcb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4dff400640 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cd95d0 0x557511cdcb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4dff400640 1 -- 172.21.15.33:0/4127660139 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557511be9cb0 con 0x557511cc9ec0 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4dfe000640 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511ce1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4dff400640 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cc9ec0 0x557511ce15a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4df8033040 tx=0x7f4df8057040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d5f5e0daab88f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4de8c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4de8c00640 1 -- 172.21.15.33:0/4127660139 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8049040 con 0x557511cc9ec0 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dec0041a0 con 0x557511cc9ec0 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4de8c00640 1 -- 172.21.15.33:0/4127660139 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8756+0+0 (secure 0 0 0) 0x7f4df80043f0 con 0x557511cc9ec0 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4de8c00640 1 -- 172.21.15.33:0/4127660139 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4df80045f0 con 0x557511cc9ec0 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.452+0000 7f4e08baf740 10 client.25660 fetch_fsmap learned FSMap version 53 2024-04-26T11:35:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x557511d4f4c0 con 0x557511cc9ec0 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4de8c00640 1 -- 172.21.15.33:0/4127660139 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dec0041a0 con 0x557511cc9ec0 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4de8c00640 1 -- 172.21.15.33:0/4127660139 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f4df804f040 con 0x557511cc9ec0 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4e08baf740 10 client.25660 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4e08baf740 10 client.25660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4e08baf740 20 client.25660 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4e08baf740 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511d4f7b0 0x557511d51ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4e08baf740 4 client.25660 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.453+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557511c36e00 con 0x557511d4f7b0 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.454+0000 7f4dfe000640 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511d4f7b0 0x557511d51ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.454+0000 7f4dfe000640 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511d4f7b0 0x557511d51ba0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbfcc4c44fc48e62 server_cookie=69288f6b271eea0a in_seq=0 out_seq=0 2024-04-26T11:35:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:16.454+0000 7f4de8c00640 10 client.25660 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:16 smithi033 ceph-mon[27845]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 31 MiB/s wr, 1.86k op/s 2024-04-26T11:35:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:16 smithi096 ceph-mon[32753]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 31 MiB/s wr, 1.86k op/s 2024-04-26T11:35:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:16 smithi194 ceph-mon[33017]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 31 MiB/s wr, 1.86k op/s 2024-04-26T11:35:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 240.0 s, 710.4 tps, lat 39.245 ms stddev 46.049 2024-04-26T11:35:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:18 smithi033 ceph-mon[27845]: pgmap v153: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 MiB/s wr, 1.65k op/s 2024-04-26T11:35:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:18 smithi096 ceph-mon[32753]: pgmap v153: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 MiB/s wr, 1.65k op/s 2024-04-26T11:35:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:18 smithi194 ceph-mon[33017]: pgmap v153: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 MiB/s wr, 1.65k op/s 2024-04-26T11:35:20.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:20 smithi033 ceph-mon[27845]: pgmap v154: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 30 MiB/s wr, 1.70k op/s 2024-04-26T11:35:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:20 smithi096 ceph-mon[32753]: pgmap v154: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 30 MiB/s wr, 1.70k op/s 2024-04-26T11:35:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:20 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:35:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:20 smithi096 systemd[1]: Stopped Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:35:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:20 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.227s CPU time. 2024-04-26T11:35:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:20 smithi096 systemd[1]: Starting Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:35:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:20 smithi096 podman[67214]: 2024-04-26 11:35:20.566367992 +0000 UTC m=+0.018393766 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:35:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:20 smithi194 ceph-mon[33017]: pgmap v154: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 30 MiB/s wr, 1.70k op/s 2024-04-26T11:35:21.265 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:21 smithi096 podman[67214]: 2024-04-26 11:35:21.012313185 +0000 UTC m=+0.464338951 container create f18e2e65f92f9aa4f6152881a9132cc87ad0239532f0f3ada0468dfec3733969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T11:35:21.265 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:21 smithi096 podman[67214]: 2024-04-26 11:35:21.264951217 +0000 UTC m=+0.716976989 container init f18e2e65f92f9aa4f6152881a9132cc87ad0239532f0f3ada0468dfec3733969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T11:35:21.630 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:21 smithi096 podman[67214]: 2024-04-26 11:35:21.268083452 +0000 UTC m=+0.720109226 container start f18e2e65f92f9aa4f6152881a9132cc87ad0239532f0f3ada0468dfec3733969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T11:35:21.630 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mds[67232]: -- 172.21.15.96:0/886275710 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d00f11ba40 con 0x55d00f0ef000 2024-04-26T11:35:21.631 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: starting mds.e at 2024-04-26T11:35:21.631 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:21 smithi096 bash[67214]: f18e2e65f92f9aa4f6152881a9132cc87ad0239532f0f3ada0468dfec3733969 2024-04-26T11:35:21.631 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:35:21 smithi096 systemd[1]: Started Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:35:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: daemon mds.e restarted 2024-04-26T11:35:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:35:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:35:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T11:35:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:boot 2024-04-26T11:35:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:35:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:35:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:21 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: daemon mds.e restarted 2024-04-26T11:35:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:35:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:35:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T11:35:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:boot 2024-04-26T11:35:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:35:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:35:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:21 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: daemon mds.e restarted 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:boot 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:35:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:21 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:22.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 245.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:35:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:22 smithi033 ceph-mon[27845]: pgmap v156: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 569 op/s 2024-04-26T11:35:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:22 smithi096 ceph-mon[32753]: pgmap v156: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 569 op/s 2024-04-26T11:35:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:22 smithi194 ceph-mon[33017]: pgmap v156: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 569 op/s 2024-04-26T11:35:23.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:23 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:resolve 2024-04-26T11:35:23.910 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:23 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:23 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:resolve 2024-04-26T11:35:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:23 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:23 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:resolve 2024-04-26T11:35:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:23 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.682+0000 7f4de8c00640 1 -- 172.21.15.33:0/4127660139 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557511c36e00 con 0x557511d4f7b0 2024-04-26T11:35:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.682+0000 7f4de8c00640 10 client.25660 handle_command_reply: tid=0 2024-04-26T11:35:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.682+0000 7f4e08baf740 1 client.25660 shutdown 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.682+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cc9ec0 msgr2=0x557511ce15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.682+0000 7f4e08baf740 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cc9ec0 0x557511ce15a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4df8033040 tx=0x7f4df8057040 comp rx=0 tx=0).stop 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 shutdown_connections 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557511d4f7b0 0x557511d51ba0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557511cd95d0 0x557511cdcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557511cca7e0 0x557511ce1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 --2- 172.21.15.33:0/4127660139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557511cc9ec0 0x557511ce15a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4de8c00640 0 client.25660 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 >> 172.21.15.33:0/4127660139 conn(0x557511cc64f0 msgr2=0x557511cd6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 shutdown_connections 2024-04-26T11:35:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.683+0000 7f4e08baf740 1 -- 172.21.15.33:0/4127660139 wait complete. 2024-04-26T11:35:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.684+0000 7f4e08baf740 20 client.25660 trim_cache size 0 max 16384 2024-04-26T11:35:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:35:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:35:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:35:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4de40a3d90 msgr2=0x7f4de40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4de40a3d90 0x7f4de40a6250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4df0098e80 tx=0x7f4df00d45d0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 msgr2=0x7f4e00198fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 0x7f4e00198fa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4df000e9f0 tx=0x7f4df000eeb0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 shutdown_connections 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f4de40a3d90 0x7f4de40a6250 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e00108750 0x7f4e0019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4e00103100 0x7f4e001994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 --2- 172.21.15.33:0/2457905511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e00102740 0x7f4e00198fa0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 >> 172.21.15.33:0/2457905511 conn(0x7f4e000fe270 msgr2=0x7f4e000fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 shutdown_connections 2024-04-26T11:35:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:24.685+0000 7f4dffe00640 1 -- 172.21.15.33:0/2457905511 wait complete. 2024-04-26T11:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:24 smithi033 ceph-mon[27845]: pgmap v157: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 569 op/s 2024-04-26T11:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:24 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:reconnect 2024-04-26T11:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:24 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:24 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999979 2024-04-26T11:35:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:24 smithi096 ceph-mon[32753]: pgmap v157: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 569 op/s 2024-04-26T11:35:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:24 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:35:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:reconnect 2024-04-26T11:35:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:24 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:24 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999979 2024-04-26T11:35:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:24 smithi194 ceph-mon[33017]: pgmap v157: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 569 op/s 2024-04-26T11:35:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:24 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:35:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:reconnect 2024-04-26T11:35:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:24 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:24 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999979 2024-04-26T11:35:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:rejoin 2024-04-26T11:35:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:25 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:25 smithi096 ceph-mon[32753]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:35:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:rejoin 2024-04-26T11:35:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:25 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:25 smithi194 ceph-mon[33017]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:35:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:rejoin 2024-04-26T11:35:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:25 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:25 smithi033 ceph-mon[27845]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:35:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:26 smithi096 ceph-mon[32753]: pgmap v158: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T11:35:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:26 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:35:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:26 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:35:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:26 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:active 2024-04-26T11:35:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:26 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:26.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:26 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:35:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:26 smithi194 ceph-mon[33017]: pgmap v158: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T11:35:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:26 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:35:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:26 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:35:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:26 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:active 2024-04-26T11:35:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:26 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:26 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:35:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:26 smithi033 ceph-mon[27845]: pgmap v158: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T11:35:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:26 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:35:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:26 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:35:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:26 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1777769457,v1:172.21.15.96:6837/1777769457] up:active 2024-04-26T11:35:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:26 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:35:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:26 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:35:27.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 250.0 s, 766.6 tps, lat 355.684 ms stddev 1967.385 2024-04-26T11:35:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:28 smithi194 ceph-mon[33017]: pgmap v159: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T11:35:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:28 smithi033 ceph-mon[27845]: pgmap v159: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T11:35:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:28 smithi096 ceph-mon[32753]: pgmap v159: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T11:35:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:30 smithi194 ceph-mon[33017]: pgmap v160: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 4.0 MiB/s wr, 274 op/s 2024-04-26T11:35:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:30 smithi033 ceph-mon[27845]: pgmap v160: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 4.0 MiB/s wr, 274 op/s 2024-04-26T11:35:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:30 smithi096 ceph-mon[32753]: pgmap v160: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 4.0 MiB/s wr, 274 op/s 2024-04-26T11:35:32.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:31 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:35:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:31 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:35:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:31 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:35:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 255.0 s, 2443.6 tps, lat 40.788 ms stddev 50.416 2024-04-26T11:35:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:32 smithi194 ceph-mon[33017]: pgmap v161: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 4.2 MiB/s wr, 301 op/s 2024-04-26T11:35:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:32 smithi033 ceph-mon[27845]: pgmap v161: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 4.2 MiB/s wr, 301 op/s 2024-04-26T11:35:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:32 smithi096 ceph-mon[32753]: pgmap v161: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 4.2 MiB/s wr, 301 op/s 2024-04-26T11:35:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:34 smithi194 ceph-mon[33017]: pgmap v162: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-04-26T11:35:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:34 smithi033 ceph-mon[27845]: pgmap v162: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-04-26T11:35:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:34 smithi096 ceph-mon[32753]: pgmap v162: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-04-26T11:35:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:36 smithi194 ceph-mon[33017]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 6.4 MiB/s wr, 550 op/s 2024-04-26T11:35:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:36 smithi033 ceph-mon[27845]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 6.4 MiB/s wr, 550 op/s 2024-04-26T11:35:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:36 smithi096 ceph-mon[32753]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 6.4 MiB/s wr, 550 op/s 2024-04-26T11:35:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 260.0 s, 2974.2 tps, lat 33.700 ms stddev 37.859 2024-04-26T11:35:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.875+0000 7f196fc00640 1 -- 172.21.15.33:0/3949977688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1968101b10 msgr2=0x7f1968101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.875+0000 7f196fc00640 1 --2- 172.21.15.33:0/3949977688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1968101b10 0x7f1968101f70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1954009890 tx=0x7f195402a370 comp rx=0 tx=0).stop 2024-04-26T11:35:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.876+0000 7f196fc00640 1 -- 172.21.15.33:0/3949977688 shutdown_connections 2024-04-26T11:35:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.876+0000 7f196fc00640 1 --2- 172.21.15.33:0/3949977688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f19681024b0 0x7f196810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.876+0000 7f196fc00640 1 --2- 172.21.15.33:0/3949977688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1968101b10 0x7f1968101f70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.876+0000 7f196fc00640 1 --2- 172.21.15.33:0/3949977688 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1968107970 0x7f1968107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.876+0000 7f196fc00640 1 -- 172.21.15.33:0/3949977688 >> 172.21.15.33:0/3949977688 conn(0x7f1968078810 msgr2=0x7f1968078c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.876+0000 7f196fc00640 1 -- 172.21.15.33:0/3949977688 shutdown_connections 2024-04-26T11:35:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.876+0000 7f196fc00640 1 -- 172.21.15.33:0/3949977688 wait complete. 2024-04-26T11:35:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.878+0000 7f196fc00640 1 Processor -- start 2024-04-26T11:35:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.878+0000 7f196fc00640 1 -- start start 2024-04-26T11:35:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.878+0000 7f196fc00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1968101b10 0x7f196819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.878+0000 7f196fc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19681024b0 0x7f196819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f196fc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1968107970 0x7f196819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f196fc00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1968112c20 con 0x7f1968101b10 2024-04-26T11:35:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f196fc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1968112aa0 con 0x7f19681024b0 2024-04-26T11:35:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f196fc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1968112da0 con 0x7f1968107970 2024-04-26T11:35:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f1967e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19681024b0 0x7f196819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f1967e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19681024b0 0x7f196819b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:59796/0 (socket says 172.21.15.33:59796) 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f1967e00640 1 -- 172.21.15.33:0/2576045701 learned_addr learned my addr 172.21.15.33:0/2576045701 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.879+0000 7f196d200640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1968101b10 0x7f196819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.880+0000 7f196dc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1968107970 0x7f196819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.880+0000 7f1967e00640 1 -- 172.21.15.33:0/2576045701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1968107970 msgr2=0x7f196819fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.880+0000 7f1967e00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1968107970 0x7f196819fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.880+0000 7f1967e00640 1 -- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1968101b10 msgr2=0x7f196819b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.880+0000 7f1967e00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1968101b10 0x7f196819b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.880+0000 7f1967e00640 1 -- 172.21.15.33:0/2576045701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19681a0290 con 0x7f19681024b0 2024-04-26T11:35:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.880+0000 7f196d200640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1968101b10 0x7f196819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:35:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.881+0000 7f1967e00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19681024b0 0x7f196819b7e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1954030bb0 tx=0x7f1954030be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.881+0000 7f1965600640 1 -- 172.21.15.33:0/2576045701 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954040020 con 0x7f19681024b0 2024-04-26T11:35:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.881+0000 7f1965600640 1 -- 172.21.15.33:0/2576045701 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1954038040 con 0x7f19681024b0 2024-04-26T11:35:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.881+0000 7f1965600640 1 -- 172.21.15.33:0/2576045701 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954037070 con 0x7f19681024b0 2024-04-26T11:35:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.882+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19681a0520 con 0x7f19681024b0 2024-04-26T11:35:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.882+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19681a7e10 con 0x7f19681024b0 2024-04-26T11:35:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.887+0000 7f1965600640 1 -- 172.21.15.33:0/2576045701 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1954043020 con 0x7f19681024b0 2024-04-26T11:35:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.888+0000 7f1972bd6740 1 Processor -- start 2024-04-26T11:35:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.889+0000 7f1972bd6740 1 -- start start 2024-04-26T11:35:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f3cdd0 0x564296f3d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f47a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f47fb0 0x564296f4a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1972bd6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564296ea2380 con 0x564296f47fb0 2024-04-26T11:35:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1972bd6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564296e3ec30 con 0x564296f3d6f0 2024-04-26T11:35:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1972bd6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564296e9e340 con 0x564296f3cdd0 2024-04-26T11:35:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1967e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f47fb0 0x564296f4a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f196d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f47a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1967e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f47fb0 0x564296f4a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50896/0 (socket says 172.21.15.33:50896) 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.890+0000 7f1967e00640 1 -- 172.21.15.33:0/4271275352 learned_addr learned my addr 172.21.15.33:0/4271275352 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f196dc00640 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f3cdd0 0x564296f3d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f1967e00640 1 -- 172.21.15.33:0/4271275352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f3cdd0 msgr2=0x564296f3d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f1967e00640 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f3cdd0 0x564296f3d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f1967e00640 1 -- 172.21.15.33:0/4271275352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 msgr2=0x564296f47a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f1967e00640 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f47a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f1967e00640 1 -- 172.21.15.33:0/4271275352 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564296e60940 con 0x564296f47fb0 2024-04-26T11:35:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f1965600640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f19500a3e40 0x7f19500a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.891+0000 7f1967e00640 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f47fb0 0x564296f4a3a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f19540ed900 tx=0x7f19540f83d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9333069b29239c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.892+0000 7f195a000640 1 -- 172.21.15.33:0/4271275352 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19540b8480 con 0x564296f47fb0 2024-04-26T11:35:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.892+0000 7f195a000640 1 -- 172.21.15.33:0/4271275352 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19540bc8e0 con 0x564296f47fb0 2024-04-26T11:35:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.892+0000 7f195a000640 1 -- 172.21.15.33:0/4271275352 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19540d0a20 con 0x564296f47fb0 2024-04-26T11:35:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.892+0000 7f1965600640 1 -- 172.21.15.33:0/2576045701 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7f19540f02d0 con 0x7f19681024b0 2024-04-26T11:35:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.892+0000 7f196dc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f19500a3e40 0x7f19500a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f196dc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f19500a3e40 0x7f19500a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1960005fd0 tx=0x7f1960009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f1972bd6740 1 -- 172.21.15.33:0/4271275352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f47fb0 msgr2=0x564296f4a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f1972bd6740 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f47fb0 0x564296f4a3a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f19540ed900 tx=0x7f19540f83d0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f1972bd6740 1 -- 172.21.15.33:0/4271275352 shutdown_connections 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f1972bd6740 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f47fb0 0x564296f4a3a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f1972bd6740 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f47a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f1972bd6740 1 --2- 172.21.15.33:0/4271275352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f3cdd0 0x564296f3d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.895+0000 7f1972bd6740 1 -- 172.21.15.33:0/4271275352 >> 172.21.15.33:0/4271275352 conn(0x564296f392e0 msgr2=0x564296f396c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.896+0000 7f1972bd6740 1 -- 172.21.15.33:0/4271275352 shutdown_connections 2024-04-26T11:35:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.896+0000 7f1972bd6740 1 -- 172.21.15.33:0/4271275352 wait complete. 2024-04-26T11:35:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.897+0000 7f1972bd6740 1 -- start start 2024-04-26T11:35:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.897+0000 7f1972bd6740 1 Processor -- start 2024-04-26T11:35:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3cdd0 0x564296f44e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3d6f0 0x564296f453d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f1972bd6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564296ea2380 con 0x564296f3d6f0 2024-04-26T11:35:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f1972bd6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564296e29c60 con 0x564296f3cdd0 2024-04-26T11:35:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f1972bd6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564296f531b0 con 0x564296f47fb0 2024-04-26T11:35:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f196d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3cdd0 0x564296f44e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f1967e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3d6f0 0x564296f453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f196d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3cdd0 0x564296f44e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:59826/0 (socket says 172.21.15.33:59826) 2024-04-26T11:35:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.899+0000 7f196d200640 1 -- 172.21.15.33:0/997291618 learned_addr learned my addr 172.21.15.33:0/997291618 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f1972bd6740 10 client.0 _ll_register_callbacks cb 0x564296e55e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f196dc00640 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f196d200640 1 -- 172.21.15.33:0/997291618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 msgr2=0x564296f45910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f196d200640 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f196d200640 1 -- 172.21.15.33:0/997291618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3d6f0 msgr2=0x564296f453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f196d200640 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3d6f0 0x564296f453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f196d200640 1 -- 172.21.15.33:0/997291618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564296e38510 con 0x564296f3cdd0 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f1967e00640 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3d6f0 0x564296f453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.900+0000 7f196d200640 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3cdd0 0x564296f44e90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f195c011900 tx=0x7f195c011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca93141ed1d2cc98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.901+0000 7f194ea00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.901+0000 7f194ea00640 1 -- 172.21.15.33:0/997291618 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c00fbf0 con 0x564296f3cdd0 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.901+0000 7f194ea00640 1 -- 172.21.15.33:0/997291618 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7f195c00fd90 con 0x564296f3cdd0 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.901+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19540bc8e0 con 0x564296f3cdd0 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.901+0000 7f194ea00640 1 -- 172.21.15.33:0/997291618 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f195c0233f0 con 0x564296f3cdd0 2024-04-26T11:35:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.901+0000 7f1972bd6740 10 client.25733 fetch_fsmap learned FSMap version 61 2024-04-26T11:35:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.901+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x564296bb4210 con 0x564296f3cdd0 2024-04-26T11:35:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.902+0000 7f194ea00640 1 -- 172.21.15.33:0/997291618 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19540bc8e0 con 0x564296f3cdd0 2024-04-26T11:35:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.902+0000 7f194ea00640 1 -- 172.21.15.33:0/997291618 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7f195c022030 con 0x564296f3cdd0 2024-04-26T11:35:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.903+0000 7f1972bd6740 10 client.25733 fetch_fsmap finished waiting for FSMap version 61 2024-04-26T11:35:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.903+0000 7f1972bd6740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.903+0000 7f1972bd6740 20 client.25733 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.903+0000 7f1972bd6740 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296f60120 0x564296f62550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.903+0000 7f1972bd6740 4 client.25733 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:35:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.903+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564296eaff90 con 0x564296f60120 2024-04-26T11:35:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.905+0000 7f1967e00640 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296f60120 0x564296f62550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.907+0000 7f1967e00640 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296f60120 0x564296f62550 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d172303d79b4d19b server_cookie=671c0094226598ca in_seq=0 out_seq=0 2024-04-26T11:35:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.907+0000 7f194ea00640 10 client.25733 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.910+0000 7f194ea00640 1 -- 172.21.15.33:0/997291618 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564296eaff90 con 0x564296f60120 2024-04-26T11:35:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.910+0000 7f194ea00640 10 client.25733 handle_command_reply: tid=0 2024-04-26T11:35:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.910+0000 7f1972bd6740 1 client.25733 shutdown 2024-04-26T11:35:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3cdd0 msgr2=0x564296f44e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3cdd0 0x564296f44e90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f195c011900 tx=0x7f195c011dc0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 shutdown_connections 2024-04-26T11:35:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296f60120 0x564296f62550 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3d6f0 0x564296f453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 --2- 172.21.15.33:0/997291618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3cdd0 0x564296f44e90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f194ea00640 0 client.25733 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.911+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 >> 172.21.15.33:0/997291618 conn(0x564296f392e0 msgr2=0x564296f3f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.912+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 shutdown_connections 2024-04-26T11:35:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.912+0000 7f1972bd6740 1 -- 172.21.15.33:0/997291618 wait complete. 2024-04-26T11:35:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.913+0000 7f1972bd6740 20 client.25733 trim_cache size 0 max 16384 2024-04-26T11:35:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 Processor -- start 2024-04-26T11:35:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 -- start start 2024-04-26T11:35:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f5be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f43e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f19681af010 con 0x564296f3cdd0 2024-04-26T11:35:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f19681ac820 con 0x564296f3d6f0 2024-04-26T11:35:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.944+0000 7f1972bd6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f19681aa690 con 0x564296f47fb0 2024-04-26T11:35:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f43e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f43e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:59838/0 (socket says 172.21.15.33:59838) 2024-04-26T11:35:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 -- 172.21.15.33:0/2211129606 learned_addr learned my addr 172.21.15.33:0/2211129606 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f196dc00640 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f196d200640 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f5be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 -- 172.21.15.33:0/2211129606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 msgr2=0x564296f45730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 -- 172.21.15.33:0/2211129606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 msgr2=0x564296f5be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f5be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.945+0000 7f1967e00640 1 -- 172.21.15.33:0/2211129606 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564296c80620 con 0x564296f3d6f0 2024-04-26T11:35:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.947+0000 7f1967e00640 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f43e70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1954000c00 tx=0x7f1954003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b986d763a2ffbe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.947+0000 7f1959600640 1 -- 172.21.15.33:0/2211129606 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19540d0c80 con 0x564296f3d6f0 2024-04-26T11:35:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.947+0000 7f1959600640 1 -- 172.21.15.33:0/2211129606 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564296eaff90 con 0x564296f3d6f0 2024-04-26T11:35:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.947+0000 7f1959600640 1 -- 172.21.15.33:0/2211129606 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19540d0e50 con 0x564296f3d6f0 2024-04-26T11:35:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.951+0000 7f1972bd6740 1 -- 172.21.15.33:0/2211129606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 msgr2=0x564296f43e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.951+0000 7f1972bd6740 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f43e70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1954000c00 tx=0x7f1954003680 comp rx=0 tx=0).stop 2024-04-26T11:35:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.951+0000 7f1972bd6740 1 -- 172.21.15.33:0/2211129606 shutdown_connections 2024-04-26T11:35:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.951+0000 7f1972bd6740 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f45730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.951+0000 7f1972bd6740 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f43e70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.951+0000 7f1972bd6740 1 --2- 172.21.15.33:0/2211129606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f5be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.952+0000 7f1972bd6740 1 -- 172.21.15.33:0/2211129606 >> 172.21.15.33:0/2211129606 conn(0x564296f392e0 msgr2=0x564296f43760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.953+0000 7f1972bd6740 1 -- 172.21.15.33:0/2211129606 shutdown_connections 2024-04-26T11:35:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.954+0000 7f1972bd6740 1 -- 172.21.15.33:0/2211129606 wait complete. 2024-04-26T11:35:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.954+0000 7f1972bd6740 1 -- start start 2024-04-26T11:35:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.954+0000 7f1972bd6740 1 Processor -- start 2024-04-26T11:35:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.956+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f58290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.956+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f587d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.957+0000 7f1972bd6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f53780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.957+0000 7f196dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f53780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.958+0000 7f196dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f53780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:47162/0 (socket says 172.21.15.33:47162) 2024-04-26T11:35:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.958+0000 7f196dc00640 1 -- 172.21.15.33:0/3866659536 learned_addr learned my addr 172.21.15.33:0/3866659536 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.958+0000 7f1967e00640 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.958+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f19681aace0 con 0x564296f3cdd0 2024-04-26T11:35:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.958+0000 7f196d200640 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f58290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.958+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f19681ac150 con 0x564296f3d6f0 2024-04-26T11:35:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.958+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f19681ab3b0 con 0x564296f47fb0 2024-04-26T11:35:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.959+0000 7f1972bd6740 10 client.0 _ll_register_callbacks cb 0x7f195c011560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.959+0000 7f196dc00640 1 -- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 msgr2=0x564296f587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.959+0000 7f196dc00640 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.959+0000 7f196dc00640 1 -- 172.21.15.33:0/3866659536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 msgr2=0x564296f58290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.959+0000 7f196dc00640 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f58290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.959+0000 7f196dc00640 1 -- 172.21.15.33:0/3866659536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564296e5ccb0 con 0x564296f47fb0 2024-04-26T11:35:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.961+0000 7f196dc00640 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f53780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f19600127d0 tx=0x7f1960004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94ce492713127e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.961+0000 7f194ea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:35:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.961+0000 7f194ea00640 1 -- 172.21.15.33:0/3866659536 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960011d70 con 0x564296f47fb0 2024-04-26T11:35:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.961+0000 7f194ea00640 1 -- 172.21.15.33:0/3866659536 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7f196002a470 con 0x564296f47fb0 2024-04-26T11:35:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.961+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564296eaff90 con 0x564296f47fb0 2024-04-26T11:35:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.962+0000 7f194ea00640 1 -- 172.21.15.33:0/3866659536 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f1960012e10 con 0x564296f47fb0 2024-04-26T11:35:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.962+0000 7f1972bd6740 10 client.25678 fetch_fsmap learned FSMap version 61 2024-04-26T11:35:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.962+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x564296fc6060 con 0x564296f47fb0 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.962+0000 7f194ea00640 1 -- 172.21.15.33:0/3866659536 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564296eaff90 con 0x564296f47fb0 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.963+0000 7f194ea00640 1 -- 172.21.15.33:0/3866659536 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7f196000f040 con 0x564296f47fb0 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.963+0000 7f1972bd6740 10 client.25678 fetch_fsmap finished waiting for FSMap version 61 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.963+0000 7f1972bd6740 10 client.25678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.963+0000 7f1972bd6740 20 client.25678 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.963+0000 7f1972bd6740 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296fc6820 0x564296fc8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.963+0000 7f1972bd6740 4 client.25678 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.963+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564296ea9e00 con 0x564296fc6820 2024-04-26T11:35:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.964+0000 7f196d200640 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296fc6820 0x564296fc8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.965+0000 7f194ea00640 10 client.25678 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.966+0000 7f196d200640 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296fc6820 0x564296fc8c40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a56b0031959c4f server_cookie=a0a0134458240f9b in_seq=0 out_seq=0 2024-04-26T11:35:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.978+0000 7f194ea00640 1 -- 172.21.15.33:0/3866659536 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564296ea9e00 con 0x564296fc6820 2024-04-26T11:35:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.978+0000 7f194ea00640 10 client.25678 handle_command_reply: tid=0 2024-04-26T11:35:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.978+0000 7f1972bd6740 1 client.25678 shutdown 2024-04-26T11:35:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.979+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 msgr2=0x564296f53780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.979+0000 7f1972bd6740 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f53780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f19600127d0 tx=0x7f1960004ab0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 shutdown_connections 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564296f47fb0 0x564296f53780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x564296fc6820 0x564296fc8c40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564296f3d6f0 0x564296f587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 --2- 172.21.15.33:0/3866659536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564296f3cdd0 0x564296f58290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 >> 172.21.15.33:0/3866659536 conn(0x564296f392e0 msgr2=0x564296f3f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f194ea00640 0 client.25678 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 shutdown_connections 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 1 -- 172.21.15.33:0/3866659536 wait complete. 2024-04-26T11:35:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.980+0000 7f1972bd6740 20 client.25678 trim_cache size 0 max 16384 2024-04-26T11:35:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:35:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:35:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:35:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:35:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f19500a3e40 msgr2=0x7f19500a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f19500a3e40 0x7f19500a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1960005fd0 tx=0x7f1960009500 comp rx=0 tx=0).stop 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19681024b0 msgr2=0x7f196819b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19681024b0 0x7f196819b7e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1954030bb0 tx=0x7f1954030be0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 shutdown_connections 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f19500a3e40 0x7f19500a6300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1968107970 0x7f196819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19681024b0 0x7f196819b7e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 --2- 172.21.15.33:0/2576045701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1968101b10 0x7f196819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 >> 172.21.15.33:0/2576045701 conn(0x7f1968078810 msgr2=0x7f1968100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 shutdown_connections 2024-04-26T11:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:37.981+0000 7f196fc00640 1 -- 172.21.15.33:0/2576045701 wait complete. 2024-04-26T11:35:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:38 smithi194 ceph-mon[33017]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 6.1 MiB/s wr, 508 op/s 2024-04-26T11:35:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:38 smithi033 ceph-mon[27845]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 6.1 MiB/s wr, 508 op/s 2024-04-26T11:35:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:38 smithi096 ceph-mon[32753]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 6.1 MiB/s wr, 508 op/s 2024-04-26T11:35:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:35:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:35:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:35:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:40 smithi194 ceph-mon[33017]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 9.3 MiB/s wr, 733 op/s 2024-04-26T11:35:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:40 smithi033 ceph-mon[27845]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 9.3 MiB/s wr, 733 op/s 2024-04-26T11:35:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:40 smithi096 ceph-mon[32753]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 9.3 MiB/s wr, 733 op/s 2024-04-26T11:35:42.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:35:42.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:35:42.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:35:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:35:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:35:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:35:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:35:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:35:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:35:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 265.0 s, 2118.0 tps, lat 47.223 ms stddev 57.632 2024-04-26T11:35:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:42 smithi194 ceph-mon[33017]: pgmap v166: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 641 op/s 2024-04-26T11:35:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:42 smithi033 ceph-mon[27845]: pgmap v166: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 641 op/s 2024-04-26T11:35:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:42 smithi096 ceph-mon[32753]: pgmap v166: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 641 op/s 2024-04-26T11:35:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:44 smithi194 ceph-mon[33017]: pgmap v167: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.0 MiB/s wr, 594 op/s 2024-04-26T11:35:45.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:44 smithi033 ceph-mon[27845]: pgmap v167: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.0 MiB/s wr, 594 op/s 2024-04-26T11:35:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:44 smithi096 ceph-mon[32753]: pgmap v167: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.0 MiB/s wr, 594 op/s 2024-04-26T11:35:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:46 smithi194 ceph-mon[33017]: pgmap v168: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 10 MiB/s wr, 923 op/s 2024-04-26T11:35:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:46 smithi033 ceph-mon[27845]: pgmap v168: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 10 MiB/s wr, 923 op/s 2024-04-26T11:35:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:46 smithi096 ceph-mon[32753]: pgmap v168: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 10 MiB/s wr, 923 op/s 2024-04-26T11:35:47.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 270.0 s, 2981.8 tps, lat 33.568 ms stddev 38.283 2024-04-26T11:35:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:48 smithi194 ceph-mon[33017]: pgmap v169: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 624 op/s 2024-04-26T11:35:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:48 smithi033 ceph-mon[27845]: pgmap v169: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 624 op/s 2024-04-26T11:35:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:48 smithi096 ceph-mon[32753]: pgmap v169: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 624 op/s 2024-04-26T11:35:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:50 smithi194 ceph-mon[33017]: pgmap v170: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 891 op/s 2024-04-26T11:35:51.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.170+0000 7ff10ac00640 1 -- 172.21.15.33:0/1933798180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 msgr2=0x7ff1041116e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.170+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1933798180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 0x7ff1041116e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009ca0 tx=0x7ff0fc02a8b0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.171+0000 7ff10ac00640 1 -- 172.21.15.33:0/1933798180 shutdown_connections 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.171+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1933798180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104102060 0x7ff10410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.171+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1933798180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 0x7ff1041116e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.171+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1933798180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff104101740 0x7ff104101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.171+0000 7ff10ac00640 1 -- 172.21.15.33:0/1933798180 >> 172.21.15.33:0/1933798180 conn(0x7ff1040789b0 msgr2=0x7ff104078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.171+0000 7ff10ac00640 1 -- 172.21.15.33:0/1933798180 shutdown_connections 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.171+0000 7ff10ac00640 1 -- 172.21.15.33:0/1933798180 wait complete. 2024-04-26T11:35:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.172+0000 7ff10ac00640 1 Processor -- start 2024-04-26T11:35:51.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.172+0000 7ff10ac00640 1 -- start start 2024-04-26T11:35:51.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.173+0000 7ff10ac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104101740 0x7ff104103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.173+0000 7ff10ac00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff104102060 0x7ff104103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.173+0000 7ff10ac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 0x7ff1041044a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.173+0000 7ff10ac00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff104114dd0 con 0x7ff104102060 2024-04-26T11:35:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.173+0000 7ff10ac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff104114c50 con 0x7ff10410f2f0 2024-04-26T11:35:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.173+0000 7ff10ac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff104114f50 con 0x7ff104101740 2024-04-26T11:35:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.174+0000 7ff103e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104101740 0x7ff104103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.174+0000 7ff103e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104101740 0x7ff104103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57064/0 (socket says 172.21.15.33:57064) 2024-04-26T11:35:51.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.174+0000 7ff103e00640 1 -- 172.21.15.33:0/1175554712 learned_addr learned my addr 172.21.15.33:0/1175554712 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:51.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.174+0000 7ff103400640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff104102060 0x7ff104103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:50 smithi033 ceph-mon[27845]: pgmap v170: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 891 op/s 2024-04-26T11:35:51.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.177+0000 7ff108c00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 0x7ff1041044a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.177+0000 7ff103e00640 1 -- 172.21.15.33:0/1175554712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 msgr2=0x7ff1041044a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.177+0000 7ff103e00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 0x7ff1041044a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.177+0000 7ff103e00640 1 -- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff104102060 msgr2=0x7ff104103f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.177+0000 7ff103e00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff104102060 0x7ff104103f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.177+0000 7ff103e00640 1 -- 172.21.15.33:0/1175554712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1041ab8f0 con 0x7ff104101740 2024-04-26T11:35:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.180+0000 7ff103e00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104101740 0x7ff104103a20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400d980 tx=0x7ff0f400de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:50 smithi096 ceph-mon[32753]: pgmap v170: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 891 op/s 2024-04-26T11:35:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.182+0000 7ff100c00640 1 -- 172.21.15.33:0/1175554712 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4017070 con 0x7ff104101740 2024-04-26T11:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.182+0000 7ff100c00640 1 -- 172.21.15.33:0/1175554712 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f4009c30 con 0x7ff104101740 2024-04-26T11:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.182+0000 7ff100c00640 1 -- 172.21.15.33:0/1175554712 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4017070 con 0x7ff104101740 2024-04-26T11:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.182+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1041abbe0 con 0x7ff104101740 2024-04-26T11:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.182+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1041ac020 con 0x7ff104101740 2024-04-26T11:35:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.190+0000 7ff10da4f740 1 Processor -- start 2024-04-26T11:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.193+0000 7ff100c00640 1 -- 172.21.15.33:0/1175554712 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff0f4010460 con 0x7ff104101740 2024-04-26T11:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.193+0000 7ff10da4f740 1 -- start start 2024-04-26T11:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.194+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b11dd0 0x560fa5b121b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.195+0000 7ff108c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b11dd0 0x560fa5b121b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.195+0000 7ff108c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b11dd0 0x560fa5b121b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49034/0 (socket says 172.21.15.33:49034) 2024-04-26T11:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.195+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b126f0 0x560fa5b1ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.196+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.196+0000 7ff10da4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fa5a77380 con 0x560fa5b11dd0 2024-04-26T11:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.196+0000 7ff10da4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560fa5a13c30 con 0x560fa5b1cfb0 2024-04-26T11:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.196+0000 7ff10da4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560fa5a73340 con 0x560fa5b126f0 2024-04-26T11:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.196+0000 7ff108c00640 1 -- 172.21.15.33:0/2508992957 learned_addr learned my addr 172.21.15.33:0/2508992957 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.196+0000 7ff100c00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff0e80a4090 0x7ff0e80a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.196+0000 7ff100c00640 1 -- 172.21.15.33:0/1175554712 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7ff0f40d01d0 con 0x7ff104101740 2024-04-26T11:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.197+0000 7ff103400640 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.198+0000 7ff103e00640 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b126f0 0x560fa5b1ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.198+0000 7ff103400640 1 -- 172.21.15.33:0/2508992957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b126f0 msgr2=0x560fa5b1ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.198+0000 7ff103400640 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b126f0 0x560fa5b1ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.198+0000 7ff103400640 1 -- 172.21.15.33:0/2508992957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b11dd0 msgr2=0x560fa5b121b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.198+0000 7ff103400640 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b11dd0 0x560fa5b121b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.198+0000 7ff103400640 1 -- 172.21.15.33:0/2508992957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa5a35940 con 0x560fa5b1cfb0 2024-04-26T11:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.199+0000 7ff108c00640 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b11dd0 0x560fa5b121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.199+0000 7ff103400640 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1f3a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff104105220 tx=0x7ff0f000f360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=267c2fbcac55351 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.201+0000 7ff0f9200640 1 -- 172.21.15.33:0/2508992957 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0015070 con 0x560fa5b1cfb0 2024-04-26T11:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.201+0000 7ff0f9200640 1 -- 172.21.15.33:0/2508992957 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f000fd90 con 0x560fa5b1cfb0 2024-04-26T11:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.201+0000 7ff0f9200640 1 -- 172.21.15.33:0/2508992957 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0011440 con 0x560fa5b1cfb0 2024-04-26T11:35:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.202+0000 7ff10da4f740 1 -- 172.21.15.33:0/2508992957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b1cfb0 msgr2=0x560fa5b1f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.202+0000 7ff10da4f740 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1f3a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff104105220 tx=0x7ff0f000f360 comp rx=0 tx=0).stop 2024-04-26T11:35:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.202+0000 7ff10da4f740 1 -- 172.21.15.33:0/2508992957 shutdown_connections 2024-04-26T11:35:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.202+0000 7ff10da4f740 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b126f0 0x560fa5b1ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.202+0000 7ff10da4f740 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1f3a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.202+0000 7ff10da4f740 1 --2- 172.21.15.33:0/2508992957 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b11dd0 0x560fa5b121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.202+0000 7ff10da4f740 1 -- 172.21.15.33:0/2508992957 >> 172.21.15.33:0/2508992957 conn(0x560fa5b0e2e0 msgr2=0x560fa5b0e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.203+0000 7ff10da4f740 1 -- 172.21.15.33:0/2508992957 shutdown_connections 2024-04-26T11:35:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.203+0000 7ff10da4f740 1 -- 172.21.15.33:0/2508992957 wait complete. 2024-04-26T11:35:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.204+0000 7ff108c00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff0e80a4090 0x7ff0e80a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.204+0000 7ff10da4f740 1 -- start start 2024-04-26T11:35:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.204+0000 7ff10da4f740 1 Processor -- start 2024-04-26T11:35:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff108c00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff0e80a4090 0x7ff0e80a6550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc031040 tx=0x7ff0fc0095c0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b11dd0 0x560fa5b19fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b126f0 0x560fa5b1a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff10da4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fa5b1fbe0 con 0x560fa5b126f0 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff10da4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560fa59fec60 con 0x560fa5b11dd0 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff10da4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560fa5b1fd60 con 0x560fa5b1cfb0 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.206+0000 7ff10da4f740 10 client.0 _ll_register_callbacks cb 0x560fa5a2ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.207+0000 7ff108c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.207+0000 7ff108c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57074/0 (socket says 172.21.15.33:57074) 2024-04-26T11:35:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.207+0000 7ff108c00640 1 -- 172.21.15.33:0/1994450019 learned_addr learned my addr 172.21.15.33:0/1994450019 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.208+0000 7ff103e00640 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b11dd0 0x560fa5b19fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.208+0000 7ff103400640 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b126f0 0x560fa5b1a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.208+0000 7ff108c00640 1 -- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b11dd0 msgr2=0x560fa5b19fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.208+0000 7ff108c00640 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b11dd0 0x560fa5b19fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.208+0000 7ff108c00640 1 -- 172.21.15.33:0/1994450019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b126f0 msgr2=0x560fa5b1a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.208+0000 7ff108c00640 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b126f0 0x560fa5b1a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.208+0000 7ff108c00640 1 -- 172.21.15.33:0/1994450019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa5a0d510 con 0x560fa5b1cfb0 2024-04-26T11:35:51.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.209+0000 7ff108c00640 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1aa40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc035000 tx=0x7ff0fc02adb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d56868921dd5b1f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:51.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.209+0000 7ff0ed600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:35:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.210+0000 7ff0ed600640 1 -- 172.21.15.33:0/1994450019 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc003e40 con 0x560fa5b1cfb0 2024-04-26T11:35:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.210+0000 7ff0ed600640 1 -- 172.21.15.33:0/1994450019 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7ff0fc031680 con 0x560fa5b1cfb0 2024-04-26T11:35:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.210+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0f000fd90 con 0x560fa5b1cfb0 2024-04-26T11:35:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.210+0000 7ff0ed600640 1 -- 172.21.15.33:0/1994450019 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7ff0fc031880 con 0x560fa5b1cfb0 2024-04-26T11:35:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.211+0000 7ff10da4f740 10 client.25690 fetch_fsmap learned FSMap version 61 2024-04-26T11:35:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.211+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x560fa5789210 con 0x560fa5b1cfb0 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.211+0000 7ff0ed600640 1 -- 172.21.15.33:0/1994450019 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f000fd90 con 0x560fa5b1cfb0 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.213+0000 7ff0ed600640 1 -- 172.21.15.33:0/1994450019 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7ff0fc045440 con 0x560fa5b1cfb0 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.213+0000 7ff10da4f740 10 client.25690 fetch_fsmap finished waiting for FSMap version 61 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.213+0000 7ff10da4f740 10 client.25690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.213+0000 7ff10da4f740 20 client.25690 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.213+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b2ce20 0x560fa5b2f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.213+0000 7ff10da4f740 4 client.25690 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.213+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fa5a84f90 con 0x560fa5b2ce20 2024-04-26T11:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.214+0000 7ff103e00640 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b2ce20 0x560fa5b2f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.218+0000 7ff103e00640 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b2ce20 0x560fa5b2f250 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df859eec7cf66ba1 server_cookie=f67b443a538f7760 in_seq=0 out_seq=0 2024-04-26T11:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.218+0000 7ff0ed600640 10 client.25690 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.219+0000 7ff0ed600640 1 -- 172.21.15.33:0/1994450019 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff0f407ae50 con 0x560fa5b2ce20 2024-04-26T11:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.219+0000 7ff0ed600640 10 client.25690 handle_command_reply: tid=0 2024-04-26T11:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.219+0000 7ff10da4f740 1 client.25690 shutdown 2024-04-26T11:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b1cfb0 msgr2=0x560fa5b1aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1aa40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc035000 tx=0x7ff0fc02adb0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 shutdown_connections 2024-04-26T11:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1aa40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b2ce20 0x560fa5b2f250 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b126f0 0x560fa5b1a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1994450019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b11dd0 0x560fa5b19fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 >> 172.21.15.33:0/1994450019 conn(0x560fa5b0e2e0 msgr2=0x560fa5b14060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.220+0000 7ff0ed600640 0 client.25690 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.221+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 shutdown_connections 2024-04-26T11:35:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.221+0000 7ff10da4f740 1 -- 172.21.15.33:0/1994450019 wait complete. 2024-04-26T11:35:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.222+0000 7ff10da4f740 20 client.25690 trim_cache size 0 max 16384 2024-04-26T11:35:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.258+0000 7ff10da4f740 1 Processor -- start 2024-04-26T11:35:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.259+0000 7ff10da4f740 1 -- start start 2024-04-26T11:35:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.259+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b28aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.259+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.259+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.259+0000 7ff10da4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1041b7670 con 0x560fa5b1cfb0 2024-04-26T11:35:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.259+0000 7ff10da4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff1041b4e80 con 0x560fa5b126f0 2024-04-26T11:35:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.259+0000 7ff10da4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff1041b2d80 con 0x560fa5b11dd0 2024-04-26T11:35:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.260+0000 7ff103e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b28aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.260+0000 7ff103e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b28aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57078/0 (socket says 172.21.15.33:57078) 2024-04-26T11:35:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.260+0000 7ff103e00640 1 -- 172.21.15.33:0/1691722884 learned_addr learned my addr 172.21.15.33:0/1691722884 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.260+0000 7ff103400640 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.261+0000 7ff103400640 1 -- 172.21.15.33:0/1691722884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 msgr2=0x560fa5b28aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.261+0000 7ff103400640 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b28aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.261+0000 7ff103400640 1 -- 172.21.15.33:0/1691722884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 msgr2=0x560fa5b1a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.261+0000 7ff103400640 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.261+0000 7ff103400640 1 -- 172.21.15.33:0/1691722884 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa5b1f710 con 0x560fa5b126f0 2024-04-26T11:35:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.262+0000 7ff103400640 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b147e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x560fa5b0f700 tx=0x7ff0f000ba80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df89c0799b53870d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.263+0000 7ff0efe00640 1 -- 172.21.15.33:0/1691722884 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0015070 con 0x560fa5b126f0 2024-04-26T11:35:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.263+0000 7ff0efe00640 1 -- 172.21.15.33:0/1691722884 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f0005030 con 0x560fa5b126f0 2024-04-26T11:35:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.263+0000 7ff0efe00640 1 -- 172.21.15.33:0/1691722884 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0004a40 con 0x560fa5b126f0 2024-04-26T11:35:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.264+0000 7ff10da4f740 1 -- 172.21.15.33:0/1691722884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 msgr2=0x560fa5b147e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.264+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b147e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x560fa5b0f700 tx=0x7ff0f000ba80 comp rx=0 tx=0).stop 2024-04-26T11:35:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.264+0000 7ff10da4f740 1 -- 172.21.15.33:0/1691722884 shutdown_connections 2024-04-26T11:35:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.264+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b1a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.264+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b147e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.264+0000 7ff10da4f740 1 --2- 172.21.15.33:0/1691722884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b28aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.264+0000 7ff10da4f740 1 -- 172.21.15.33:0/1691722884 >> 172.21.15.33:0/1691722884 conn(0x560fa5b0e2e0 msgr2=0x560fa5b1bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.265+0000 7ff10da4f740 1 -- 172.21.15.33:0/1691722884 shutdown_connections 2024-04-26T11:35:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.265+0000 7ff10da4f740 1 -- 172.21.15.33:0/1691722884 wait complete. 2024-04-26T11:35:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.266+0000 7ff10da4f740 1 -- start start 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.266+0000 7ff10da4f740 1 Processor -- start 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.268+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b25280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.268+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b16890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.268+0000 7ff10da4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b16dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.268+0000 7ff10da4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff1041b3340 con 0x560fa5b1cfb0 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.268+0000 7ff10da4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff1041b47b0 con 0x560fa5b126f0 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.268+0000 7ff10da4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff1041b3a10 con 0x560fa5b11dd0 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.269+0000 7ff103e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b25280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.269+0000 7ff10da4f740 10 client.0 _ll_register_callbacks cb 0x7ff0fc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:35:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.269+0000 7ff103e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b25280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57080/0 (socket says 172.21.15.33:57080) 2024-04-26T11:35:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.269+0000 7ff103e00640 1 -- 172.21.15.33:0/587288687 learned_addr learned my addr 172.21.15.33:0/587288687 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:35:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.269+0000 7ff108c00640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b16dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.269+0000 7ff103400640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b16890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.270+0000 7ff108c00640 1 -- 172.21.15.33:0/587288687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 msgr2=0x560fa5b25280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.270+0000 7ff108c00640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b25280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.270+0000 7ff108c00640 1 -- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 msgr2=0x560fa5b16890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.270+0000 7ff108c00640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b16890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.270+0000 7ff108c00640 1 -- 172.21.15.33:0/587288687 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa5855620 con 0x560fa5b1cfb0 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.270+0000 7ff103e00640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b25280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:35:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.271+0000 7ff108c00640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b16dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc004af0 tx=0x7ff0fc030420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a5e186045d175e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:35:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.271+0000 7ff0ed600640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:35:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.271+0000 7ff0ed600640 1 -- 172.21.15.33:0/587288687 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc045c90 con 0x560fa5b1cfb0 2024-04-26T11:35:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.271+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0f0005030 con 0x560fa5b1cfb0 2024-04-26T11:35:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.271+0000 7ff0ed600640 1 -- 172.21.15.33:0/587288687 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7ff0fc045e30 con 0x560fa5b1cfb0 2024-04-26T11:35:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.272+0000 7ff0ed600640 1 -- 172.21.15.33:0/587288687 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7ff0fc067050 con 0x560fa5b1cfb0 2024-04-26T11:35:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.272+0000 7ff10da4f740 10 client.15555 fetch_fsmap learned FSMap version 61 2024-04-26T11:35:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.272+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x560fa5a31cb0 con 0x560fa5b1cfb0 2024-04-26T11:35:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.272+0000 7ff0ed600640 1 -- 172.21.15.33:0/587288687 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0fc04c030 con 0x560fa5b1cfb0 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.273+0000 7ff0ed600640 1 -- 172.21.15.33:0/587288687 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7ff0fc05e070 con 0x560fa5b1cfb0 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.273+0000 7ff10da4f740 10 client.15555 fetch_fsmap finished waiting for FSMap version 61 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.273+0000 7ff10da4f740 10 client.15555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.273+0000 7ff10da4f740 20 client.15555 populate_metadata read hostname 'smithi033' 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.273+0000 7ff10da4f740 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b936f0 0x560fa5b95b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.273+0000 7ff10da4f740 4 client.15555 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.273+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff0f0005030 con 0x560fa5b936f0 2024-04-26T11:35:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.274+0000 7ff103e00640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b936f0 0x560fa5b95b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:35:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.274+0000 7ff103e00640 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b936f0 0x560fa5b95b10 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e5806bd90e6f6c4 server_cookie=8455fe8042196449 in_seq=0 out_seq=0 2024-04-26T11:35:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.274+0000 7ff0ed600640 10 client.15555 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.287+0000 7ff0ed600640 1 -- 172.21.15.33:0/587288687 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff0f0005030 con 0x560fa5b936f0 2024-04-26T11:35:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.287+0000 7ff0ed600640 10 client.15555 handle_command_reply: tid=0 2024-04-26T11:35:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.287+0000 7ff10da4f740 1 client.15555 shutdown 2024-04-26T11:35:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 msgr2=0x560fa5b16dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b16dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc004af0 tx=0x7ff0fc030420 comp rx=0 tx=0).stop 2024-04-26T11:35:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 shutdown_connections 2024-04-26T11:35:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x560fa5b936f0 0x560fa5b95b10 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fa5b1cfb0 0x560fa5b16dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fa5b126f0 0x560fa5b16890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 --2- 172.21.15.33:0/587288687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560fa5b11dd0 0x560fa5b25280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff0ed600640 0 client.15555 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 >> 172.21.15.33:0/587288687 conn(0x560fa5b0e2e0 msgr2=0x560fa5b1beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.288+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 shutdown_connections 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.289+0000 7ff10da4f740 1 -- 172.21.15.33:0/587288687 wait complete. 2024-04-26T11:35:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.289+0000 7ff10da4f740 20 client.15555 trim_cache size 0 max 16384 2024-04-26T11:35:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:35:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:35:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:35:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:35:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.290+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff0e80a4090 msgr2=0x7ff0e80a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.290+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff0e80a4090 0x7ff0e80a6550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc031040 tx=0x7ff0fc0095c0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.290+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104101740 msgr2=0x7ff104103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.290+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104101740 0x7ff104103a20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400d980 tx=0x7ff0f400de40 comp rx=0 tx=0).stop 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 shutdown_connections 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff0e80a4090 0x7ff0e80a6550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff10410f2f0 0x7ff1041044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff104102060 0x7ff104103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 --2- 172.21.15.33:0/1175554712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff104101740 0x7ff104103a20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 >> 172.21.15.33:0/1175554712 conn(0x7ff1040789b0 msgr2=0x7ff104110780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 shutdown_connections 2024-04-26T11:35:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:35:51.291+0000 7ff10ac00640 1 -- 172.21.15.33:0/1175554712 wait complete. 2024-04-26T11:35:52.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 275.0 s, 3281.3 tps, lat 30.479 ms stddev 33.349 2024-04-26T11:35:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:52 smithi194 ceph-mon[33017]: pgmap v171: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.9 MiB/s wr, 716 op/s 2024-04-26T11:35:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:52 smithi033 ceph-mon[27845]: pgmap v171: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.9 MiB/s wr, 716 op/s 2024-04-26T11:35:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:52 smithi096 ceph-mon[32753]: pgmap v171: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.9 MiB/s wr, 716 op/s 2024-04-26T11:35:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:53 smithi194 ceph-mon[33017]: pgmap v172: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 644 op/s 2024-04-26T11:35:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:53 smithi033 ceph-mon[27845]: pgmap v172: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 644 op/s 2024-04-26T11:35:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:53 smithi096 ceph-mon[32753]: pgmap v172: 97 pgs: 97 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 644 op/s 2024-04-26T11:35:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:56 smithi033 ceph-mon[27845]: pgmap v173: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 971 op/s 2024-04-26T11:35:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:56 smithi096 ceph-mon[32753]: pgmap v173: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 971 op/s 2024-04-26T11:35:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:56 smithi194 ceph-mon[33017]: pgmap v173: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 971 op/s 2024-04-26T11:35:57.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 280.0 s, 3181.6 tps, lat 31.189 ms stddev 33.523 2024-04-26T11:35:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:35:58 smithi033 ceph-mon[27845]: pgmap v174: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 642 op/s 2024-04-26T11:35:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:35:58 smithi096 ceph-mon[32753]: pgmap v174: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 642 op/s 2024-04-26T11:35:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:35:58 smithi194 ceph-mon[33017]: pgmap v174: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 642 op/s 2024-04-26T11:36:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:00 smithi033 ceph-mon[27845]: pgmap v175: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-26T11:36:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-mon[32753]: pgmap v175: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f61868f9700 time 2024-04-26T11:36:00.297856+0000 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6190f223dd] 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.689 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: clone() 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.298+0000 7f61868f9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f61868f9700 time 2024-04-26T11:36:00.297856+0000 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:00.690 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6190f223dd] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.691 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: clone() 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: *** Caught signal (Aborted) ** 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: in thread 7f61868f9700 thread_name:ms_dispatch 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f618fcaad20] 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: gsignal() 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: abort() 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6190f22437] 2024-04-26T11:36:00.692 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.693 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 16: clone() 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.300+0000 7f61868f9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: in thread 7f61868f9700 thread_name:ms_dispatch 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f618fcaad20] 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: gsignal() 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: abort() 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6190f22437] 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.694 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 16: clone() 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.695 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: -1> 2024-04-26T11:36:00.298+0000 7f61868f9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f61868f9700 time 2024-04-26T11:36:00.297856+0000 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6190f223dd] 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.696 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: clone() 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 0> 2024-04-26T11:36:00.300+0000 7f61868f9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: in thread 7f61868f9700 thread_name:ms_dispatch 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.697 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f618fcaad20] 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: gsignal() 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: abort() 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6190f22437] 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.698 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 16: clone() 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: -5> 2024-04-26T11:36:00.298+0000 7f61868f9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f61868f9700 time 2024-04-26T11:36:00.297856+0000 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:00.699 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6190f223dd] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.700 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: clone() 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: -4> 2024-04-26T11:36:00.300+0000 7f61868f9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: in thread 7f61868f9700 thread_name:ms_dispatch 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f618fcaad20] 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2: gsignal() 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 3: abort() 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6190f22437] 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6190f225a3] 2024-04-26T11:36:00.701 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ebce321822] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ebce291598] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ebce293c17] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ebce18cd0b] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebce18d2ac] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ebce175fad] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f61911b6e20] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 13: (DispatchQueue::entry()+0x51f) [0x7f61911b3fdf] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6191286fc1] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f618fca01ca] 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 16: clone() 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:00.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[66924]: 2024-04-26T11:36:00.703 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:00 smithi096 podman[67406]: 2024-04-26 11:36:00.358238476 +0000 UTC m=+0.019062221 container died 8cfcdf5c67cb20fdea885a98a9111937dc5f90afdd989f46c002e6835355829e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:36:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:00 smithi194 ceph-mon[33017]: pgmap v175: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-26T11:36:01.304 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:01 smithi096 podman[67406]: 2024-04-26 11:36:01.025860709 +0000 UTC m=+0.686684526 container remove 8cfcdf5c67cb20fdea885a98a9111937dc5f90afdd989f46c002e6835355829e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T11:36:01.304 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:01 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:36:01.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:01 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-26T11:36:01.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:01 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.147s CPU time. 2024-04-26T11:36:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 285.0 s, 2441.9 tps, lat 41.119 ms stddev 47.290 2024-04-26T11:36:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:02 smithi033 ceph-mon[27845]: pgmap v176: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.7 MiB/s wr, 678 op/s 2024-04-26T11:36:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:02 smithi096 ceph-mon[32753]: pgmap v176: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.7 MiB/s wr, 678 op/s 2024-04-26T11:36:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:02 smithi194 ceph-mon[33017]: pgmap v176: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.7 MiB/s wr, 678 op/s 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.453+0000 7f933a400640 1 -- 172.21.15.33:0/2432369537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334108960 msgr2=0x7f9334108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.453+0000 7f933a400640 1 --2- 172.21.15.33:0/2432369537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334108960 0x7f9334108d40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f931c009930 tx=0x7f931c02a420 comp rx=0 tx=0).stop 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.454+0000 7f933a400640 1 -- 172.21.15.33:0/2432369537 shutdown_connections 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.454+0000 7f933a400640 1 --2- 172.21.15.33:0/2432369537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9334103310 0x7f933410f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.454+0000 7f933a400640 1 --2- 172.21.15.33:0/2432369537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9334102950 0x7f9334102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.454+0000 7f933a400640 1 --2- 172.21.15.33:0/2432369537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334108960 0x7f9334108d40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.454+0000 7f933a400640 1 -- 172.21.15.33:0/2432369537 >> 172.21.15.33:0/2432369537 conn(0x7f93340fe3e0 msgr2=0x7f9334100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.454+0000 7f933a400640 1 -- 172.21.15.33:0/2432369537 shutdown_connections 2024-04-26T11:36:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.455+0000 7f933a400640 1 -- 172.21.15.33:0/2432369537 wait complete. 2024-04-26T11:36:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.456+0000 7f933a400640 1 Processor -- start 2024-04-26T11:36:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.457+0000 7f933a400640 1 -- start start 2024-04-26T11:36:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.459+0000 7f933a400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334102950 0x7f93340740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.459+0000 7f9333400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334102950 0x7f93340740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.459+0000 7f9333400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334102950 0x7f93340740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:39860/0 (socket says 172.21.15.33:39860) 2024-04-26T11:36:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.459+0000 7f933a400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9334103310 0x7f9334074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.459+0000 7f933a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9334108960 0x7f933406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f933a400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9334112c40 con 0x7f9334103310 2024-04-26T11:36:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f933a400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9334112ac0 con 0x7f9334108960 2024-04-26T11:36:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f933a400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9334112dc0 con 0x7f9334102950 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f9333400640 1 -- 172.21.15.33:0/265846213 learned_addr learned my addr 172.21.15.33:0/265846213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f9333e00640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9334108960 0x7f933406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f9333400640 1 -- 172.21.15.33:0/265846213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9334108960 msgr2=0x7f933406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f9333400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9334108960 0x7f933406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f9333400640 1 -- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9334103310 msgr2=0x7f9334074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f9333400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9334103310 0x7f9334074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.460+0000 7f9333400640 1 -- 172.21.15.33:0/265846213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933406dca0 con 0x7f9334102950 2024-04-26T11:36:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.461+0000 7f9333400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334102950 0x7f93340740e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f931c031440 tx=0x7f931c003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.461+0000 7f9330c00640 1 -- 172.21.15.33:0/265846213 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c035030 con 0x7f9334102950 2024-04-26T11:36:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.462+0000 7f9330c00640 1 -- 172.21.15.33:0/265846213 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c04a070 con 0x7f9334102950 2024-04-26T11:36:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.462+0000 7f9330c00640 1 -- 172.21.15.33:0/265846213 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c03b460 con 0x7f9334102950 2024-04-26T11:36:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.463+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f933406df30 con 0x7f9334102950 2024-04-26T11:36:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.464+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9334071de0 con 0x7f9334102950 2024-04-26T11:36:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.469+0000 7f9330c00640 1 -- 172.21.15.33:0/265846213 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f931c0314a0 con 0x7f9334102950 2024-04-26T11:36:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.470+0000 7f933d377740 1 Processor -- start 2024-04-26T11:36:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.470+0000 7f933d377740 1 -- start start 2024-04-26T11:36:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.471+0000 7f933d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x5573079141b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.471+0000 7f933d377740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.471+0000 7f933d377740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55730791efb0 0x5573079213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.471+0000 7f933d377740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557307879380 con 0x55730791efb0 2024-04-26T11:36:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.471+0000 7f933d377740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557307815c30 con 0x5573079146f0 2024-04-26T11:36:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.471+0000 7f933d377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557307875340 con 0x557307913dd0 2024-04-26T11:36:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55410/0 (socket says 172.21.15.33:55410) 2024-04-26T11:36:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 -- 172.21.15.33:0/2401087968 learned_addr learned my addr 172.21.15.33:0/2401087968 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333e00640 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x5573079141b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f932be00640 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55730791efb0 0x5573079213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 -- 172.21.15.33:0/2401087968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 msgr2=0x5573079141b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x5573079141b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 -- 172.21.15.33:0/2401087968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55730791efb0 msgr2=0x5573079213a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55730791efb0 0x5573079213a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 -- 172.21.15.33:0/2401087968 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557307837940 con 0x5573079146f0 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f932be00640 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55730791efb0 0x5573079213a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:36:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.472+0000 7f9333400640 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791ea70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f931c0f62d0 tx=0x7f931c0fcf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48eff0c95d940829 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.474+0000 7f9330c00640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f93180a3f10 0x7f93180a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.474+0000 7f9328c00640 1 -- 172.21.15.33:0/2401087968 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c108070 con 0x5573079146f0 2024-04-26T11:36:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.474+0000 7f9328c00640 1 -- 172.21.15.33:0/2401087968 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c101e20 con 0x5573079146f0 2024-04-26T11:36:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.474+0000 7f9328c00640 1 -- 172.21.15.33:0/2401087968 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0c09c0 con 0x5573079146f0 2024-04-26T11:36:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.474+0000 7f9330c00640 1 -- 172.21.15.33:0/265846213 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7f931c0f8de0 con 0x7f9334102950 2024-04-26T11:36:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.475+0000 7f9333e00640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f93180a3f10 0x7f93180a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f933d377740 1 -- 172.21.15.33:0/2401087968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 msgr2=0x55730791ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:04.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f9333e00640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f93180a3f10 0x7f93180a63d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f932c005fd0 tx=0x7f932c009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:04.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f933d377740 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791ea70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f931c0f62d0 tx=0x7f931c0fcf90 comp rx=0 tx=0).stop 2024-04-26T11:36:04.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f933d377740 1 -- 172.21.15.33:0/2401087968 shutdown_connections 2024-04-26T11:36:04.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f933d377740 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55730791efb0 0x5573079213a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f933d377740 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791ea70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f933d377740 1 --2- 172.21.15.33:0/2401087968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x5573079141b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.476+0000 7f933d377740 1 -- 172.21.15.33:0/2401087968 >> 172.21.15.33:0/2401087968 conn(0x5573079102e0 msgr2=0x5573079106c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:04.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.477+0000 7f933d377740 1 -- 172.21.15.33:0/2401087968 shutdown_connections 2024-04-26T11:36:04.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.477+0000 7f933d377740 1 -- 172.21.15.33:0/2401087968 wait complete. 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.477+0000 7f933d377740 1 -- start start 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.477+0000 7f933d377740 1 Processor -- start 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f933d377740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557307913dd0 0x55730791bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f933d377740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x55730791c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f933d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x55730791c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f933d377740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5573079219a0 con 0x5573079146f0 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f933d377740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557307800c60 con 0x557307913dd0 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f933d377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557307921b20 con 0x55730791efb0 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f933d377740 10 client.0 _ll_register_callbacks cb 0x55730782ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557307913dd0 0x55730791bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557307913dd0 0x55730791bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55424/0 (socket says 172.21.15.33:55424) 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 -- 172.21.15.33:0/2936896687 learned_addr learned my addr 172.21.15.33:0/2936896687 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f932be00640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x55730791c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333e00640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x55730791c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 -- 172.21.15.33:0/2936896687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 msgr2=0x55730791c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x55730791c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 -- 172.21.15.33:0/2936896687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 msgr2=0x55730791c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x55730791c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333400640 1 -- 172.21.15.33:0/2936896687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55730780f510 con 0x557307913dd0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.478+0000 7f9333e00640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x55730791c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f9333400640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557307913dd0 0x55730791bf00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f931c1018e0 tx=0x7f931c02a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=853ece8e27dcb344 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f9321600640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f9321600640 1 -- 172.21.15.33:0/2936896687 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0c3970 con 0x557307913dd0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f9321600640 1 -- 172.21.15.33:0/2936896687 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7f931c03d310 con 0x557307913dd0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f931c101e20 con 0x557307913dd0 2024-04-26T11:36:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f9321600640 1 -- 172.21.15.33:0/2936896687 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f931c0c40c0 con 0x557307913dd0 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f933d377740 10 client.25766 fetch_fsmap learned FSMap version 61 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x55730758b210 con 0x557307913dd0 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f9321600640 1 -- 172.21.15.33:0/2936896687 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c101e20 con 0x557307913dd0 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f9321600640 1 -- 172.21.15.33:0/2936896687 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7f931c0c12e0 con 0x557307913dd0 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f933d377740 10 client.25766 fetch_fsmap finished waiting for FSMap version 61 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.479+0000 7f933d377740 10 client.25766 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.480+0000 7f933d377740 20 client.25766 populate_metadata read hostname 'smithi033' 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.480+0000 7f933d377740 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55730792e830 0x557307930c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.480+0000 7f933d377740 4 client.25766 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.480+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557307886f90 con 0x55730792e830 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.480+0000 7f932be00640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55730792e830 0x557307930c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.480+0000 7f932be00640 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55730792e830 0x557307930c20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=564996f5198660ed server_cookie=97e3fbfc11db3230 in_seq=0 out_seq=0 2024-04-26T11:36:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.480+0000 7f9321600640 10 client.25766 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:04.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f9321600640 1 -- 172.21.15.33:0/2936896687 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557307886f90 con 0x55730792e830 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f9321600640 10 client.25766 handle_command_reply: tid=0 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 client.25766 shutdown 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557307913dd0 msgr2=0x55730791bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557307913dd0 0x55730791bf00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f931c1018e0 tx=0x7f931c02a420 comp rx=0 tx=0).stop 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 shutdown_connections 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x55730791c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55730792e830 0x557307930c20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x55730791c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 --2- 172.21.15.33:0/2936896687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557307913dd0 0x55730791bf00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f9321600640 0 client.25766 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 >> 172.21.15.33:0/2936896687 conn(0x5573079102e0 msgr2=0x557307916010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.481+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 shutdown_connections 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.482+0000 7f933d377740 1 -- 172.21.15.33:0/2936896687 wait complete. 2024-04-26T11:36:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.482+0000 7f933d377740 20 client.25766 trim_cache size 0 max 16384 2024-04-26T11:36:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.497+0000 7f933d377740 1 Processor -- start 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.497+0000 7f933d377740 1 -- start start 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.497+0000 7f933d377740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557307913dd0 0x55730792a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.497+0000 7f933d377740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f933d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x557307936420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f933d377740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f93341b7820 con 0x557307913dd0 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f933d377740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93341b5030 con 0x5573079146f0 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f933d377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93341b2e80 con 0x55730791efb0 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557307913dd0 0x55730792a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557307913dd0 0x55730792a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49470/0 (socket says 172.21.15.33:49470) 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 -- 172.21.15.33:0/787904859 learned_addr learned my addr 172.21.15.33:0/787904859 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f932be00640 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333e00640 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x557307936420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 -- 172.21.15.33:0/787904859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 msgr2=0x557307936420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x557307936420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 -- 172.21.15.33:0/787904859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 msgr2=0x55730791aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 -- 172.21.15.33:0/787904859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557307914520 con 0x557307913dd0 2024-04-26T11:36:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.498+0000 7f9333400640 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557307913dd0 0x55730792a760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f931c006010 tx=0x7f931c02a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338f288ba921d302 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f9323e00640 1 -- 172.21.15.33:0/787904859 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0dd070 con 0x557307913dd0 2024-04-26T11:36:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f9323e00640 1 -- 172.21.15.33:0/787904859 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c0c4c70 con 0x557307913dd0 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f9323e00640 1 -- 172.21.15.33:0/787904859 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c108070 con 0x557307913dd0 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 -- 172.21.15.33:0/787904859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557307913dd0 msgr2=0x55730792a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557307913dd0 0x55730792a760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f931c006010 tx=0x7f931c02a920 comp rx=0 tx=0).stop 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 -- 172.21.15.33:0/787904859 shutdown_connections 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55730791efb0 0x557307936420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573079146f0 0x55730791aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 --2- 172.21.15.33:0/787904859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557307913dd0 0x55730792a760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 -- 172.21.15.33:0/787904859 >> 172.21.15.33:0/787904859 conn(0x5573079102e0 msgr2=0x55730791a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.499+0000 7f933d377740 1 -- 172.21.15.33:0/787904859 shutdown_connections 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 -- 172.21.15.33:0/787904859 wait complete. 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 -- start start 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 Processor -- start 2024-04-26T11:36:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x557307926c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x5573079271b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55730791efb0 0x557307918730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f93341b34f0 con 0x5573079146f0 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93341b4960 con 0x55730791efb0 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.500+0000 7f933d377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93341b3bc0 con 0x557307913dd0 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f933d377740 10 client.0 _ll_register_callbacks cb 0x7f931c0fcd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x557307926c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x557307926c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:39908/0 (socket says 172.21.15.33:39908) 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 -- 172.21.15.33:0/617775224 learned_addr learned my addr 172.21.15.33:0/617775224 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333e00640 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55730791efb0 0x557307918730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f932be00640 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x5573079271b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 -- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55730791efb0 msgr2=0x557307918730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55730791efb0 0x557307918730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 -- 172.21.15.33:0/617775224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 msgr2=0x5573079271b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x5573079271b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 -- 172.21.15.33:0/617775224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557307657620 con 0x557307913dd0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9333400640 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x557307926c70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f931c0c4970 tx=0x7f931c02a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c261d4a0401a381 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.501+0000 7f9321600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f9321600640 1 -- 172.21.15.33:0/617775224 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0dd070 con 0x557307913dd0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f9321600640 1 -- 172.21.15.33:0/617775224 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8842+0+0 (secure 0 0 0) 0x7f931c0fc5e0 con 0x557307913dd0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f931c0c4c70 con 0x557307913dd0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f9321600640 1 -- 172.21.15.33:0/617775224 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f931c108050 con 0x557307913dd0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 10 client.25714 fetch_fsmap learned FSMap version 61 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x557307833cb0 con 0x557307913dd0 2024-04-26T11:36:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f9321600640 1 -- 172.21.15.33:0/617775224 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c0c4c70 con 0x557307913dd0 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f9321600640 1 -- 172.21.15.33:0/617775224 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7f931c0f6610 con 0x557307913dd0 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 10 client.25714 fetch_fsmap finished waiting for FSMap version 61 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 10 client.25714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 20 client.25714 populate_metadata read hostname 'smithi033' 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557307994850 0x557307996c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 4 client.25714 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.502+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557307880e00 con 0x557307994850 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.503+0000 7f932be00640 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557307994850 0x557307996c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.503+0000 7f932be00640 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557307994850 0x557307996c70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4bafee7ee812da3 server_cookie=53f9b79f8f270f57 in_seq=0 out_seq=0 2024-04-26T11:36:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:04.503+0000 7f9321600640 10 client.25714 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:04 smithi096 ceph-mon[32753]: pgmap v177: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.3 MiB/s wr, 629 op/s 2024-04-26T11:36:04.853 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:04 smithi033 ceph-mon[27845]: pgmap v177: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.3 MiB/s wr, 629 op/s 2024-04-26T11:36:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:04 smithi194 ceph-mon[33017]: pgmap v177: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.3 MiB/s wr, 629 op/s 2024-04-26T11:36:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:06 smithi194 ceph-mon[33017]: pgmap v178: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 10 MiB/s wr, 911 op/s 2024-04-26T11:36:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:06 smithi033 ceph-mon[27845]: pgmap v178: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 10 MiB/s wr, 911 op/s 2024-04-26T11:36:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:06 smithi096 ceph-mon[32753]: pgmap v178: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 10 MiB/s wr, 911 op/s 2024-04-26T11:36:07.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 290.0 s, 936.4 tps, lat 36.068 ms stddev 41.273 2024-04-26T11:36:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:08 smithi194 ceph-mon[33017]: pgmap v179: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 584 op/s 2024-04-26T11:36:08.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:08 smithi033 ceph-mon[27845]: pgmap v179: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 584 op/s 2024-04-26T11:36:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:08 smithi096 ceph-mon[32753]: pgmap v179: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 584 op/s 2024-04-26T11:36:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:10 smithi194 ceph-mon[33017]: pgmap v180: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.0 MiB/s wr, 638 op/s 2024-04-26T11:36:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:10 smithi033 ceph-mon[27845]: pgmap v180: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.0 MiB/s wr, 638 op/s 2024-04-26T11:36:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:10 smithi096 ceph-mon[32753]: pgmap v180: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.0 MiB/s wr, 638 op/s 2024-04-26T11:36:11.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:11 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:36:11.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:11 smithi096 systemd[1]: Stopped Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:36:11.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:11 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.147s CPU time. 2024-04-26T11:36:11.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:11 smithi096 systemd[1]: Starting Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:36:11.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:11 smithi096 podman[67505]: 2024-04-26 11:36:11.569613218 +0000 UTC m=+0.018214082 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:36:12.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 295.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:36:12.339 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:12 smithi096 podman[67505]: 2024-04-26 11:36:12.075824123 +0000 UTC m=+0.524424976 container create 27c4a322db345f5521478a2e012ba82cfb169d602852c6702fc1ddcc66ba30c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:36:12.339 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:12 smithi096 podman[67505]: 2024-04-26 11:36:12.311749688 +0000 UTC m=+0.760350548 container init 27c4a322db345f5521478a2e012ba82cfb169d602852c6702fc1ddcc66ba30c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T11:36:12.339 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:12 smithi096 podman[67505]: 2024-04-26 11:36:12.314402841 +0000 UTC m=+0.763003695 container start 27c4a322db345f5521478a2e012ba82cfb169d602852c6702fc1ddcc66ba30c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424) 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: pgmap v181: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 MiB/s wr, 411 op/s 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: daemon mds.h restarted 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:boot 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:12.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:12.632 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-mds[67525]: -- 172.21.15.96:0/1023230520 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5616f8c8da40 con 0x5616f8c61000 2024-04-26T11:36:12.632 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:12 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: starting mds.h at 2024-04-26T11:36:12.632 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:12 smithi096 bash[67505]: 27c4a322db345f5521478a2e012ba82cfb169d602852c6702fc1ddcc66ba30c7 2024-04-26T11:36:12.632 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:12 smithi096 systemd[1]: Started Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:36:12.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: pgmap v181: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 MiB/s wr, 411 op/s 2024-04-26T11:36:12.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: daemon mds.h restarted 2024-04-26T11:36:12.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:12.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:36:12.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T11:36:12.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:boot 2024-04-26T11:36:12.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:36:12.858 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:12.858 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:12 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: pgmap v181: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 MiB/s wr, 411 op/s 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: daemon mds.h restarted 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:boot 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:12 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:14.721 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:14 smithi033 ceph-mon[27845]: pgmap v183: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-04-26T11:36:14.721 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:14 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:resolve 2024-04-26T11:36:14.721 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:14 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:14 smithi194 ceph-mon[33017]: pgmap v183: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-04-26T11:36:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:14 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:resolve 2024-04-26T11:36:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:14 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:14 smithi096 ceph-mon[32753]: pgmap v183: 97 pgs: 97 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-04-26T11:36:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:14 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:resolve 2024-04-26T11:36:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:14 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.487+0000 7f9321600640 1 -- 172.21.15.33:0/617775224 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557307880e00 con 0x557307994850 2024-04-26T11:36:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.487+0000 7f9321600640 10 client.25714 handle_command_reply: tid=0 2024-04-26T11:36:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 client.25714 shutdown 2024-04-26T11:36:15.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 msgr2=0x557307926c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:15.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x557307926c70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f931c0c4970 tx=0x7f931c02a420 comp rx=0 tx=0).stop 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 shutdown_connections 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x557307994850 0x557307996c70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55730791efb0 0x557307918730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573079146f0 0x5573079271b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 --2- 172.21.15.33:0/617775224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557307913dd0 0x557307926c70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 >> 172.21.15.33:0/617775224 conn(0x5573079102e0 msgr2=0x55730790ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.488+0000 7f9321600640 0 client.25714 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.489+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 shutdown_connections 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.489+0000 7f933d377740 1 -- 172.21.15.33:0/617775224 wait complete. 2024-04-26T11:36:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.489+0000 7f933d377740 20 client.25714 trim_cache size 0 max 16384 2024-04-26T11:36:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:36:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:36:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:36:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.490+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f93180a3f10 msgr2=0x7f93180a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.490+0000 7f933a400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f93180a3f10 0x7f93180a63d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f932c005fd0 tx=0x7f932c009500 comp rx=0 tx=0).stop 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.490+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334102950 msgr2=0x7f93340740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.490+0000 7f933a400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334102950 0x7f93340740e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f931c031440 tx=0x7f931c003c00 comp rx=0 tx=0).stop 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 shutdown_connections 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f93180a3f10 0x7f93180a63d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9334108960 0x7f933406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9334103310 0x7f9334074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 --2- 172.21.15.33:0/265846213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9334102950 0x7f93340740e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 >> 172.21.15.33:0/265846213 conn(0x7f93340fe3e0 msgr2=0x7f93340feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 shutdown_connections 2024-04-26T11:36:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:15.491+0000 7f933a400640 1 -- 172.21.15.33:0/265846213 wait complete. 2024-04-26T11:36:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:15 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:reconnect 2024-04-26T11:36:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:15 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:15 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:36:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:15 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:36:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:15 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:reconnect 2024-04-26T11:36:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:15 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:15 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:36:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:15 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:36:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:15 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:reconnect 2024-04-26T11:36:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:15 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:15 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:36:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:15 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:36:16.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: pgmap v184: 97 pgs: 97 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T11:36:16.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:rejoin 2024-04-26T11:36:16.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:16.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:36:16.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:16.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:36:16.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:active 2024-04-26T11:36:16.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:16 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: pgmap v184: 97 pgs: 97 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T11:36:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:rejoin 2024-04-26T11:36:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:36:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:36:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:active 2024-04-26T11:36:16.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:16 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: pgmap v184: 97 pgs: 97 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:rejoin 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3977415905,v1:172.21.15.96:6835/3977415905] up:active 2024-04-26T11:36:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:16 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:17.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 300.0 s, 91.1 tps, lat 2699.532 ms stddev 5084.367 2024-04-26T11:36:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:18 smithi194 ceph-mon[33017]: pgmap v185: 97 pgs: 97 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T11:36:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:18 smithi033 ceph-mon[27845]: pgmap v185: 97 pgs: 97 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T11:36:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:18 smithi096 ceph-mon[32753]: pgmap v185: 97 pgs: 97 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T11:36:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:20 smithi194 ceph-mon[33017]: pgmap v186: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 28 MiB/s wr, 1.06k op/s 2024-04-26T11:36:20.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:20 smithi033 ceph-mon[27845]: pgmap v186: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 28 MiB/s wr, 1.06k op/s 2024-04-26T11:36:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:20 smithi096 ceph-mon[32753]: pgmap v186: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 28 MiB/s wr, 1.06k op/s 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c1785b700 time 2024-04-26T11:36:21.294782+0000 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c21e843dd] 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.675 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: clone() 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.295+0000 7f2c1785b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c1785b700 time 2024-04-26T11:36:21.294782+0000 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:21.676 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.677 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.677 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c21e843dd] 2024-04-26T11:36:21.677 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.677 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.677 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: clone() 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: *** Caught signal (Aborted) ** 2024-04-26T11:36:21.678 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: in thread 7f2c1785b700 thread_name:ms_dispatch 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c20c0cd20] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: gsignal() 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: abort() 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c21e84437] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.679 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 16: clone() 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.296+0000 7f2c1785b700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: in thread 7f2c1785b700 thread_name:ms_dispatch 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c20c0cd20] 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: gsignal() 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: abort() 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c21e84437] 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.680 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 16: clone() 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: -1> 2024-04-26T11:36:21.295+0000 7f2c1785b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c1785b700 time 2024-04-26T11:36:21.294782+0000 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:21.681 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c21e843dd] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.682 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: clone() 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 0> 2024-04-26T11:36:21.296+0000 7f2c1785b700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: in thread 7f2c1785b700 thread_name:ms_dispatch 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c20c0cd20] 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: gsignal() 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: abort() 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c21e84437] 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 16: clone() 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: -4> 2024-04-26T11:36:21.295+0000 7f2c1785b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c1785b700 time 2024-04-26T11:36:21.294782+0000 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.685 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c21e843dd] 2024-04-26T11:36:21.685 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.685 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.685 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.685 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.685 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: clone() 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: -3> 2024-04-26T11:36:21.296+0000 7f2c1785b700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: in thread 7f2c1785b700 thread_name:ms_dispatch 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c20c0cd20] 2024-04-26T11:36:21.686 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2: gsignal() 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 3: abort() 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c21e84437] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c21e845a3] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d00d6c2822] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d00d632598] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d00d634c17] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d00d52dd0b] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d00d52e2ac] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d00d516fad] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c22118e20] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c22115fdf] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c221e8fc1] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c20c021ca] 2024-04-26T11:36:21.687 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 16: clone() 2024-04-26T11:36:21.688 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:21.688 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67228]: 2024-04-26T11:36:21.688 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 podman[67575]: 2024-04-26 11:36:21.377943275 +0000 UTC m=+0.022425122 container died f18e2e65f92f9aa4f6152881a9132cc87ad0239532f0f3ada0468dfec3733969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T11:36:22.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 305.0 s, 1535.5 tps, lat 69.172 ms stddev 384.357 2024-04-26T11:36:22.253 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 podman[67575]: 2024-04-26 11:36:21.992641776 +0000 UTC m=+0.637123605 container remove f18e2e65f92f9aa4f6152881a9132cc87ad0239532f0f3ada0468dfec3733969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T11:36:22.253 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:21 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:36:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:22 smithi033 ceph-mon[27845]: pgmap v187: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 41 MiB/s wr, 1.49k op/s 2024-04-26T11:36:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:22 smithi096 ceph-mon[32753]: pgmap v187: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 41 MiB/s wr, 1.49k op/s 2024-04-26T11:36:22.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:22 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-26T11:36:22.683 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:22 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.201s CPU time. 2024-04-26T11:36:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:22 smithi194 ceph-mon[33017]: pgmap v187: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 41 MiB/s wr, 1.49k op/s 2024-04-26T11:36:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:24 smithi194 ceph-mon[33017]: pgmap v188: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 38 MiB/s wr, 1.36k op/s 2024-04-26T11:36:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:24 smithi033 ceph-mon[27845]: pgmap v188: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 38 MiB/s wr, 1.36k op/s 2024-04-26T11:36:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:24 smithi096 ceph-mon[32753]: pgmap v188: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 38 MiB/s wr, 1.36k op/s 2024-04-26T11:36:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:26 smithi194 ceph-mon[33017]: pgmap v189: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 42 MiB/s wr, 1.45k op/s 2024-04-26T11:36:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:26 smithi033 ceph-mon[27845]: pgmap v189: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 42 MiB/s wr, 1.45k op/s 2024-04-26T11:36:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:26 smithi096 ceph-mon[32753]: pgmap v189: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 42 MiB/s wr, 1.45k op/s 2024-04-26T11:36:27.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 310.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:36:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.645+0000 7faf8fe00640 1 -- 172.21.15.33:0/1625812490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 msgr2=0x7faf9010c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.645+0000 7faf8fe00640 1 --2- 172.21.15.33:0/1625812490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9010c7b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faf78009be0 tx=0x7faf7802a750 comp rx=0 tx=0).stop 2024-04-26T11:36:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.645+0000 7faf8fe00640 1 -- 172.21.15.33:0/1625812490 shutdown_connections 2024-04-26T11:36:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.645+0000 7faf8fe00640 1 --2- 172.21.15.33:0/1625812490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 0x7faf9010f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.646+0000 7faf8fe00640 1 --2- 172.21.15.33:0/1625812490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9010c7b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.646+0000 7faf8fe00640 1 --2- 172.21.15.33:0/1625812490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf901039b0 0x7faf90103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.646+0000 7faf8fe00640 1 -- 172.21.15.33:0/1625812490 >> 172.21.15.33:0/1625812490 conn(0x7faf900fd150 msgr2=0x7faf900ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.646+0000 7faf8fe00640 1 -- 172.21.15.33:0/1625812490 shutdown_connections 2024-04-26T11:36:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.646+0000 7faf8fe00640 1 -- 172.21.15.33:0/1625812490 wait complete. 2024-04-26T11:36:28.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.647+0000 7faf8fe00640 1 Processor -- start 2024-04-26T11:36:28.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.647+0000 7faf8fe00640 1 -- start start 2024-04-26T11:36:28.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf901039b0 0x7faf9019b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9019b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 0x7faf9019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8fe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf901129c0 con 0x7faf9010cd80 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf90112840 con 0x7faf901042d0 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8fe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf90112b40 con 0x7faf901039b0 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9019b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 0x7faf9019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf901039b0 0x7faf9019b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9019b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:43672/0 (socket says 172.21.15.33:43672) 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8e000640 1 -- 172.21.15.33:0/3916899779 learned_addr learned my addr 172.21.15.33:0/3916899779 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 0x7faf9019fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58292/0 (socket says 172.21.15.33:58292) 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8e000640 1 -- 172.21.15.33:0/3916899779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf901039b0 msgr2=0x7faf9019b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8e000640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf901039b0 0x7faf9019b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8e000640 1 -- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 msgr2=0x7faf9019fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.648+0000 7faf8e000640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 0x7faf9019fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf8e000640 1 -- 172.21.15.33:0/3916899779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf901a01b0 con 0x7faf901042d0 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf8f400640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 0x7faf9019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf8e000640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9019b700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faf7802ad40 tx=0x7faf78003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf87400640 1 -- 172.21.15.33:0/3916899779 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78041020 con 0x7faf901042d0 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf87400640 1 -- 172.21.15.33:0/3916899779 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf78003e10 con 0x7faf901042d0 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf901a0440 con 0x7faf901042d0 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf87400640 1 -- 172.21.15.33:0/3916899779 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78036040 con 0x7faf901042d0 2024-04-26T11:36:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.649+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf901a7d30 con 0x7faf901042d0 2024-04-26T11:36:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 Processor -- start 2024-04-26T11:36:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 -- start start 2024-04-26T11:36:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf87400640 1 -- 172.21.15.33:0/3916899779 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7faf78044020 con 0x7faf901042d0 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22c0fe0 0x555bb22c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22cc1c0 0x555bb22ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555bb2226380 con 0x555bb22c0fe0 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555bb21c2c30 con 0x555bb22c1900 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.654+0000 7faf983cd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555bb2222340 con 0x555bb22cc1c0 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22c0fe0 0x555bb22c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22c0fe0 0x555bb22c13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58304/0 (socket says 172.21.15.33:58304) 2024-04-26T11:36:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 -- 172.21.15.33:0/3289744617 learned_addr learned my addr 172.21.15.33:0/3289744617 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8e000640 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22cc1c0 0x555bb22ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8ea00640 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 -- 172.21.15.33:0/3289744617 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22cc1c0 msgr2=0x555bb22ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22cc1c0 0x555bb22ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 -- 172.21.15.33:0/3289744617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 msgr2=0x555bb22cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.655+0000 7faf8f400640 1 -- 172.21.15.33:0/3289744617 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb21e4940 con 0x555bb22c0fe0 2024-04-26T11:36:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.656+0000 7faf8f400640 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22c0fe0 0x555bb22c13c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf88009f90 tx=0x7faf880118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1686b57a71ce7dab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.656+0000 7faf7f400640 1 -- 172.21.15.33:0/3289744617 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8801e070 con 0x555bb22c0fe0 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.656+0000 7faf7f400640 1 -- 172.21.15.33:0/3289744617 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf88004310 con 0x555bb22c0fe0 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.656+0000 7faf7f400640 1 -- 172.21.15.33:0/3289744617 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8800f840 con 0x555bb22c0fe0 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.656+0000 7faf87400640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faf700a3d10 0x7faf700a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.656+0000 7faf87400640 1 -- 172.21.15.33:0/3916899779 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8928+0+0 (secure 0 0 0) 0x7faf780f1430 con 0x7faf901042d0 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.656+0000 7faf8ea00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faf700a3d10 0x7faf700a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 -- 172.21.15.33:0/3289744617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22c0fe0 msgr2=0x555bb22c13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22c0fe0 0x555bb22c13c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf88009f90 tx=0x7faf880118d0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 -- 172.21.15.33:0/3289744617 shutdown_connections 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22cc1c0 0x555bb22ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 --2- 172.21.15.33:0/3289744617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22c0fe0 0x555bb22c13c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 -- 172.21.15.33:0/3289744617 >> 172.21.15.33:0/3289744617 conn(0x555bb22bd4f0 msgr2=0x555bb22bd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 -- 172.21.15.33:0/3289744617 shutdown_connections 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf983cd740 1 -- 172.21.15.33:0/3289744617 wait complete. 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.657+0000 7faf8ea00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faf700a3d10 0x7faf700a61d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x555bb22c2580 tx=0x7faf80009260 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 -- start start 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 Processor -- start 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555bb22cedf0 con 0x555bb22cc1c0 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555bb21adc60 con 0x555bb22c1900 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555bb22cef70 con 0x555bb22c0fe0 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.658+0000 7faf983cd740 10 client.0 _ll_register_callbacks cb 0x555bb21d9e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:36:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58314/0 (socket says 172.21.15.33:58314) 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 -- 172.21.15.33:0/4248633047 learned_addr learned my addr 172.21.15.33:0/4248633047 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8e000640 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8ea00640 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 -- 172.21.15.33:0/4248633047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 msgr2=0x555bb22c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 -- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 msgr2=0x555bb22c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 -- 172.21.15.33:0/4248633047 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb21bc510 con 0x555bb22cc1c0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.659+0000 7faf8f400640 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c9ba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faf880056e0 tx=0x7faf88005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9993295ab2e7efe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.660+0000 7faf77e00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.660+0000 7faf77e00640 1 -- 172.21.15.33:0/4248633047 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88027020 con 0x555bb22cc1c0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.660+0000 7faf77e00640 1 -- 172.21.15.33:0/4248633047 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8928+0+0 (secure 0 0 0) 0x7faf88003800 con 0x555bb22cc1c0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.660+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf88004310 con 0x555bb22cc1c0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.660+0000 7faf77e00640 1 -- 172.21.15.33:0/4248633047 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7faf88032050 con 0x555bb22cc1c0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.660+0000 7faf983cd740 10 client.15579 fetch_fsmap learned FSMap version 67 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.660+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x555bb1f38210 con 0x555bb22cc1c0 2024-04-26T11:36:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf77e00640 1 -- 172.21.15.33:0/4248633047 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf88004310 con 0x555bb22cc1c0 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf77e00640 1 -- 172.21.15.33:0/4248633047 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 67) ==== 8346+0+0 (secure 0 0 0) 0x7faf8800f5c0 con 0x555bb22cc1c0 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf983cd740 10 client.15579 fetch_fsmap finished waiting for FSMap version 67 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf983cd740 10 client.15579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf983cd740 20 client.15579 populate_metadata read hostname 'smithi033' 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf983cd740 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb22dbe20 0x555bb22de210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf983cd740 4 client.15579 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bb2233f90 con 0x555bb22dbe20 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.661+0000 7faf8ea00640 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb22dbe20 0x555bb22de210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.662+0000 7faf8ea00640 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb22dbe20 0x555bb22de210 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80f950c35c677134 server_cookie=9f37408f4c07176 in_seq=0 out_seq=0 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.662+0000 7faf77e00640 10 client.15579 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf77e00640 1 -- 172.21.15.33:0/4248633047 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555bb2233f90 con 0x555bb22dbe20 2024-04-26T11:36:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf77e00640 10 client.15579 handle_command_reply: tid=0 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 client.15579 shutdown 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 msgr2=0x555bb22c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c9ba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faf880056e0 tx=0x7faf88005710 comp rx=0 tx=0).stop 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 shutdown_connections 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb22dbe20 0x555bb22de210 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c9ba0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 --2- 172.21.15.33:0/4248633047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf77e00640 0 client.15579 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 >> 172.21.15.33:0/4248633047 conn(0x555bb22bd4f0 msgr2=0x555bb22c3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 shutdown_connections 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.663+0000 7faf983cd740 1 -- 172.21.15.33:0/4248633047 wait complete. 2024-04-26T11:36:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.664+0000 7faf983cd740 20 client.15579 trim_cache size 0 max 16384 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 Processor -- start 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 -- start start 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c0fe0 0x555bb22cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c1900 0x555bb22d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf901b32a0 con 0x555bb22cc1c0 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf901b0b10 con 0x555bb22c0fe0 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.680+0000 7faf983cd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf901ae940 con 0x555bb22c1900 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22e39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58324/0 (socket says 172.21.15.33:58324) 2024-04-26T11:36:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 -- 172.21.15.33:0/70482392 learned_addr learned my addr 172.21.15.33:0/70482392 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8ea00640 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c0fe0 0x555bb22cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8e000640 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c1900 0x555bb22d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 -- 172.21.15.33:0/70482392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c1900 msgr2=0x555bb22d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c1900 0x555bb22d7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 -- 172.21.15.33:0/70482392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c0fe0 msgr2=0x555bb22cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c0fe0 0x555bb22cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.681+0000 7faf8f400640 1 -- 172.21.15.33:0/70482392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb2004620 con 0x555bb22cc1c0 2024-04-26T11:36:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.682+0000 7faf8f400640 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22e39a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf88004020 tx=0x7faf88001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368a688717800744 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.682+0000 7faf7ea00640 1 -- 172.21.15.33:0/70482392 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88027030 con 0x555bb22cc1c0 2024-04-26T11:36:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.682+0000 7faf7ea00640 1 -- 172.21.15.33:0/70482392 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf8801f040 con 0x555bb22cc1c0 2024-04-26T11:36:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.682+0000 7faf7ea00640 1 -- 172.21.15.33:0/70482392 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8801e070 con 0x555bb22cc1c0 2024-04-26T11:36:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 -- 172.21.15.33:0/70482392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 msgr2=0x555bb22e39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22e39a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf88004020 tx=0x7faf88001e50 comp rx=0 tx=0).stop 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 -- 172.21.15.33:0/70482392 shutdown_connections 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c1900 0x555bb22d7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22e39a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 --2- 172.21.15.33:0/70482392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c0fe0 0x555bb22cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 -- 172.21.15.33:0/70482392 >> 172.21.15.33:0/70482392 conn(0x555bb22bd4f0 msgr2=0x555bb22bc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 -- 172.21.15.33:0/70482392 shutdown_connections 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 -- 172.21.15.33:0/70482392 wait complete. 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 -- start start 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.683+0000 7faf983cd740 1 Processor -- start 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf983cd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22d41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf983cd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22d4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf983cd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf983cd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faf901aefd0 con 0x555bb22cc1c0 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf983cd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf901b0440 con 0x555bb22c1900 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf983cd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf901af6a0 con 0x555bb22c0fe0 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf983cd740 10 client.0 _ll_register_callbacks cb 0x7faf8801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf8f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58338/0 (socket says 172.21.15.33:58338) 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf8f400640 1 -- 172.21.15.33:0/1488995254 learned_addr learned my addr 172.21.15.33:0/1488995254 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf8ea00640 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22d41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.684+0000 7faf8e000640 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22d4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf8f400640 1 -- 172.21.15.33:0/1488995254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 msgr2=0x555bb22d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf8f400640 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22d41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf8f400640 1 -- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 msgr2=0x555bb22d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf8f400640 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22d4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf8f400640 1 -- 172.21.15.33:0/1488995254 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb21e0cb0 con 0x555bb22cc1c0 2024-04-26T11:36:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf8f400640 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c5950 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faf88011910 tx=0x7faf88005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9846acceaede4894 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf77e00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:36:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.685+0000 7faf77e00640 1 -- 172.21.15.33:0/1488995254 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88027070 con 0x555bb22cc1c0 2024-04-26T11:36:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf8801f040 con 0x555bb22cc1c0 2024-04-26T11:36:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf77e00640 1 -- 172.21.15.33:0/1488995254 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8928+0+0 (secure 0 0 0) 0x7faf88003800 con 0x555bb22cc1c0 2024-04-26T11:36:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf77e00640 1 -- 172.21.15.33:0/1488995254 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7faf8801e040 con 0x555bb22cc1c0 2024-04-26T11:36:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 10 client.15585 fetch_fsmap learned FSMap version 67 2024-04-26T11:36:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x555bb2341c40 con 0x555bb22cc1c0 2024-04-26T11:36:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf77e00640 1 -- 172.21.15.33:0/1488995254 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf8801f040 con 0x555bb22cc1c0 2024-04-26T11:36:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf77e00640 1 -- 172.21.15.33:0/1488995254 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 67) ==== 8346+0+0 (secure 0 0 0) 0x7faf88037940 con 0x555bb22cc1c0 2024-04-26T11:36:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 10 client.15585 fetch_fsmap finished waiting for FSMap version 67 2024-04-26T11:36:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 10 client.15585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:36:28.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 20 client.15585 populate_metadata read hostname 'smithi033' 2024-04-26T11:36:28.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb23420d0 0x555bb23444f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:28.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 4 client.15585 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:36:28.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.686+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555bb222de00 con 0x555bb23420d0 2024-04-26T11:36:28.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.687+0000 7faf8ea00640 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb23420d0 0x555bb23444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:28.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.687+0000 7faf8ea00640 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb23420d0 0x555bb23444f0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a1ca0a6727c6289 server_cookie=8ba31c8fd4c64c67 in_seq=0 out_seq=0 2024-04-26T11:36:28.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:28.687+0000 7faf77e00640 10 client.15585 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:28 smithi194 ceph-mon[33017]: pgmap v190: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 41 MiB/s wr, 1.36k op/s 2024-04-26T11:36:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:28 smithi033 ceph-mon[27845]: pgmap v190: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 41 MiB/s wr, 1.36k op/s 2024-04-26T11:36:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:28 smithi096 ceph-mon[32753]: pgmap v190: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 41 MiB/s wr, 1.36k op/s 2024-04-26T11:36:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:30 smithi194 ceph-mon[33017]: pgmap v191: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 41 MiB/s wr, 1.39k op/s 2024-04-26T11:36:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:30 smithi033 ceph-mon[27845]: pgmap v191: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 41 MiB/s wr, 1.39k op/s 2024-04-26T11:36:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:30 smithi096 ceph-mon[32753]: pgmap v191: 97 pgs: 97 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 41 MiB/s wr, 1.39k op/s 2024-04-26T11:36:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 315.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:36:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:32 smithi096 ceph-mon[32753]: pgmap v192: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 19 MiB/s wr, 634 op/s 2024-04-26T11:36:32.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:36:32.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 systemd[1]: Stopped Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:36:32.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.201s CPU time. 2024-04-26T11:36:32.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 systemd[1]: Starting Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:36:32.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 podman[67675]: 2024-04-26 11:36:32.572727012 +0000 UTC m=+0.018149106 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:36:32.684 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 podman[67675]: 2024-04-26 11:36:32.684129509 +0000 UTC m=+0.129551597 container create 03bbbac3c9b9fcedd975f96845920dcc94903d4640c9676ac6fbaff19150cc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T11:36:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:32 smithi194 ceph-mon[33017]: pgmap v192: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 19 MiB/s wr, 634 op/s 2024-04-26T11:36:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:32 smithi033 ceph-mon[27845]: pgmap v192: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 19 MiB/s wr, 634 op/s 2024-04-26T11:36:32.953 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 podman[67675]: 2024-04-26 11:36:32.92919066 +0000 UTC m=+0.374612746 container init 03bbbac3c9b9fcedd975f96845920dcc94903d4640c9676ac6fbaff19150cc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T11:36:32.954 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 podman[67675]: 2024-04-26 11:36:32.931825304 +0000 UTC m=+0.377247391 container start 03bbbac3c9b9fcedd975f96845920dcc94903d4640c9676ac6fbaff19150cc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T11:36:33.432 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 ceph-mds[67694]: -- 172.21.15.96:0/1387095784 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556d9ef25a40 con 0x556d9eef9000 2024-04-26T11:36:33.433 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:32 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: starting mds.e at 2024-04-26T11:36:33.433 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:33 smithi096 bash[67675]: 03bbbac3c9b9fcedd975f96845920dcc94903d4640c9676ac6fbaff19150cc71 2024-04-26T11:36:33.433 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:36:33 smithi096 systemd[1]: Started Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: daemon mds.e restarted 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:boot 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:33 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: daemon mds.e restarted 2024-04-26T11:36:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:36:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T11:36:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:boot 2024-04-26T11:36:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:36:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:33 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: daemon mds.e restarted 2024-04-26T11:36:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:36:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T11:36:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:boot 2024-04-26T11:36:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:36:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:33 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:34 smithi194 ceph-mon[33017]: pgmap v194: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 299 op/s 2024-04-26T11:36:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:34 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:resolve 2024-04-26T11:36:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:34 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:34 smithi033 ceph-mon[27845]: pgmap v194: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 299 op/s 2024-04-26T11:36:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:34 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:resolve 2024-04-26T11:36:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:34 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:34 smithi096 ceph-mon[32753]: pgmap v194: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 299 op/s 2024-04-26T11:36:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:34 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:resolve 2024-04-26T11:36:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:34 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.008+0000 7faf77e00640 1 -- 172.21.15.33:0/1488995254 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555bb222de00 con 0x555bb23420d0 2024-04-26T11:36:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.008+0000 7faf77e00640 10 client.15585 handle_command_reply: tid=0 2024-04-26T11:36:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.008+0000 7faf983cd740 1 client.15585 shutdown 2024-04-26T11:36:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 msgr2=0x555bb22c5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c5950 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faf88011910 tx=0x7faf88005750 comp rx=0 tx=0).stop 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 shutdown_connections 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555bb23420d0 0x555bb23444f0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555bb22cc1c0 0x555bb22c5950 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555bb22c1900 0x555bb22d4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 --2- 172.21.15.33:0/1488995254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bb22c0fe0 0x555bb22d41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 >> 172.21.15.33:0/1488995254 conn(0x555bb22bd4f0 msgr2=0x555bb22c2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.009+0000 7faf77e00640 0 client.15585 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.010+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 shutdown_connections 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.010+0000 7faf983cd740 1 -- 172.21.15.33:0/1488995254 wait complete. 2024-04-26T11:36:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.010+0000 7faf983cd740 20 client.15585 trim_cache size 0 max 16384 2024-04-26T11:36:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:36:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:36:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:36:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faf700a3d10 msgr2=0x7faf700a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faf700a3d10 0x7faf700a61d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x555bb22c2580 tx=0x7faf80009260 comp rx=0 tx=0).stop 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 msgr2=0x7faf9019b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9019b700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faf7802ad40 tx=0x7faf78003c20 comp rx=0 tx=0).stop 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 shutdown_connections 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7faf700a3d10 0x7faf700a61d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faf9010cd80 0x7faf9019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf901042d0 0x7faf9019b700 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 --2- 172.21.15.33:0/3916899779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf901039b0 0x7faf9019b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 >> 172.21.15.33:0/3916899779 conn(0x7faf900fd150 msgr2=0x7faf90102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 shutdown_connections 2024-04-26T11:36:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:36.012+0000 7faf8fe00640 1 -- 172.21.15.33:0/3916899779 wait complete. 2024-04-26T11:36:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:35 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:36:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:35 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:reconnect 2024-04-26T11:36:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:35 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:35 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119997 2024-04-26T11:36:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:35 smithi194 ceph-mon[33017]: pgmap v195: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 759 KiB/s wr, 103 op/s 2024-04-26T11:36:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:35 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:36:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:35 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:reconnect 2024-04-26T11:36:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:35 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:35 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119997 2024-04-26T11:36:36.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:35 smithi033 ceph-mon[27845]: pgmap v195: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 759 KiB/s wr, 103 op/s 2024-04-26T11:36:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:35 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:36:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:35 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:reconnect 2024-04-26T11:36:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:35 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:35 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119997 2024-04-26T11:36:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:35 smithi096 ceph-mon[32753]: pgmap v195: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 759 KiB/s wr, 103 op/s 2024-04-26T11:36:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 320.0 s, 367.0 tps, lat 856.703 ms stddev 3417.122 2024-04-26T11:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:36 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:rejoin 2024-04-26T11:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:36 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:37 smithi194 ceph-mon[33017]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:37 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:37 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:37 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:active 2024-04-26T11:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:37 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:36 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:rejoin 2024-04-26T11:36:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:36 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:36 smithi033 ceph-mon[27845]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:36:37.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:36 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:37.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:36 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:36:37.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:37 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:active 2024-04-26T11:36:37.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:37 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:36 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:rejoin 2024-04-26T11:36:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:36 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:37 smithi096 ceph-mon[32753]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:36:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:37 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:37 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:36:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:37 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/404453146,v1:172.21.15.96:6837/404453146] up:active 2024-04-26T11:36:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:37 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:38 smithi194 ceph-mon[33017]: pgmap v196: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 759 KiB/s wr, 103 op/s 2024-04-26T11:36:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:38 smithi033 ceph-mon[27845]: pgmap v196: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 759 KiB/s wr, 103 op/s 2024-04-26T11:36:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:38 smithi096 ceph-mon[32753]: pgmap v196: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 759 KiB/s wr, 103 op/s 2024-04-26T11:36:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:40 smithi033 ceph-mon[27845]: pgmap v197: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 6.3 MiB/s wr, 209 op/s 2024-04-26T11:36:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:40 smithi096 ceph-mon[32753]: pgmap v197: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 6.3 MiB/s wr, 209 op/s 2024-04-26T11:36:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:40 smithi194 ceph-mon[33017]: pgmap v197: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 6.3 MiB/s wr, 209 op/s 2024-04-26T11:36:41.667 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7df52a0700 time 2024-04-26T11:36:41.289795+0000 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7dff8c93dd] 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.668 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: clone() 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.290+0000 7f7df52a0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7df52a0700 time 2024-04-26T11:36:41.289795+0000 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.669 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7dff8c93dd] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.670 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: clone() 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: *** Caught signal (Aborted) ** 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: in thread 7f7df52a0700 thread_name:ms_dispatch 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7dfe651d20] 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: gsignal() 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: abort() 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7dff8c9437] 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.671 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.672 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.672 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.672 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 16: clone() 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.291+0000 7f7df52a0700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: in thread 7f7df52a0700 thread_name:ms_dispatch 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7dfe651d20] 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: gsignal() 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: abort() 2024-04-26T11:36:41.673 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7dff8c9437] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 16: clone() 2024-04-26T11:36:41.674 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: -1> 2024-04-26T11:36:41.290+0000 7f7df52a0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7df52a0700 time 2024-04-26T11:36:41.289795+0000 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7dff8c93dd] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.675 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: clone() 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 0> 2024-04-26T11:36:41.291+0000 7f7df52a0700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: in thread 7f7df52a0700 thread_name:ms_dispatch 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7dfe651d20] 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: gsignal() 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: abort() 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7dff8c9437] 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.676 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.677 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.677 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.677 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.677 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 16: clone() 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: -1> 2024-04-26T11:36:41.290+0000 7f7df52a0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7df52a0700 time 2024-04-26T11:36:41.289795+0000 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.678 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7dff8c93dd] 2024-04-26T11:36:41.679 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.679 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:36:41.680 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.680 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.680 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.680 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.680 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.680 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: clone() 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 0> 2024-04-26T11:36:41.291+0000 7f7df52a0700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: in thread 7f7df52a0700 thread_name:ms_dispatch 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7dfe651d20] 2024-04-26T11:36:41.681 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2: gsignal() 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 3: abort() 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7dff8c9437] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7dff8c95a3] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5616f62a2822] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5616f6212598] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5616f6214c17] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5616f610dd0b] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5616f610e2ac] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5616f60f6fad] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7dffb5de20] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 13: (DispatchQueue::entry()+0x51f) [0x7f7dffb5afdf] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7dffc2dfc1] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7dfe6471ca] 2024-04-26T11:36:41.682 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 16: clone() 2024-04-26T11:36:41.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:36:41.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67521]: 2024-04-26T11:36:41.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 podman[67732]: 2024-04-26 11:36:41.348738569 +0000 UTC m=+0.019461520 container died 27c4a322db345f5521478a2e012ba82cfb169d602852c6702fc1ddcc66ba30c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T11:36:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:36:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 325.0 s, 2109.7 tps, lat 43.900 ms stddev 58.127 2024-04-26T11:36:42.283 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 podman[67732]: 2024-04-26 11:36:41.959468561 +0000 UTC m=+0.630191512 container remove 27c4a322db345f5521478a2e012ba82cfb169d602852c6702fc1ddcc66ba30c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T11:36:42.283 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:41 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:36:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:42 smithi096 ceph-mon[32753]: pgmap v198: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 11 MiB/s wr, 277 op/s 2024-04-26T11:36:42.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:42 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-26T11:36:42.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:42 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.077s CPU time. 2024-04-26T11:36:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:42 smithi194 ceph-mon[33017]: pgmap v198: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 11 MiB/s wr, 277 op/s 2024-04-26T11:36:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:42 smithi033 ceph-mon[27845]: pgmap v198: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 11 MiB/s wr, 277 op/s 2024-04-26T11:36:43.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:36:43.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:36:43.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:36:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:36:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:36:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:36:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:36:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:36:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:36:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:44 smithi194 ceph-mon[33017]: pgmap v199: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 11 MiB/s wr, 266 op/s 2024-04-26T11:36:45.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:44 smithi096 ceph-mon[32753]: pgmap v199: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 11 MiB/s wr, 266 op/s 2024-04-26T11:36:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:44 smithi033 ceph-mon[27845]: pgmap v199: 97 pgs: 97 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 11 MiB/s wr, 266 op/s 2024-04-26T11:36:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:46 smithi194 ceph-mon[33017]: pgmap v200: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 18 MiB/s wr, 569 op/s 2024-04-26T11:36:47.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:46 smithi033 ceph-mon[27845]: pgmap v200: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 18 MiB/s wr, 569 op/s 2024-04-26T11:36:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:46 smithi096 ceph-mon[32753]: pgmap v200: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 18 MiB/s wr, 569 op/s 2024-04-26T11:36:47.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 330.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:36:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:48 smithi194 ceph-mon[33017]: pgmap v201: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 17 MiB/s wr, 524 op/s 2024-04-26T11:36:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:48 smithi033 ceph-mon[27845]: pgmap v201: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 17 MiB/s wr, 524 op/s 2024-04-26T11:36:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:48 smithi096 ceph-mon[32753]: pgmap v201: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 17 MiB/s wr, 524 op/s 2024-04-26T11:36:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 -- 172.21.15.33:0/2981778309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06b2a0 msgr2=0x7fa93c06b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 --2- 172.21.15.33:0/2981778309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06b2a0 0x7fa93c06b720 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa924009930 tx=0x7fa92402a440 comp rx=0 tx=0).stop 2024-04-26T11:36:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 -- 172.21.15.33:0/2981778309 shutdown_connections 2024-04-26T11:36:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 --2- 172.21.15.33:0/2981778309 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa93c06bc60 0x7fa93c10f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 --2- 172.21.15.33:0/2981778309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06b2a0 0x7fa93c06b720 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 --2- 172.21.15.33:0/2981778309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa93c1089e0 0x7fa93c108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 -- 172.21.15.33:0/2981778309 >> 172.21.15.33:0/2981778309 conn(0x7fa93c077b70 msgr2=0x7fa93c077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 -- 172.21.15.33:0/2981778309 shutdown_connections 2024-04-26T11:36:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.196+0000 7fa941200640 1 -- 172.21.15.33:0/2981778309 wait complete. 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.197+0000 7fa941200640 1 Processor -- start 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.197+0000 7fa941200640 1 -- start start 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa941200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa93c06b2a0 0x7fa93c199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa941200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06bc60 0x7fa93c199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa941200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa93c1089e0 0x7fa93c19d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa941200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c112be0 con 0x7fa93c06bc60 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa941200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c112a60 con 0x7fa93c06b2a0 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa941200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c112d60 con 0x7fa93c1089e0 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa933e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06bc60 0x7fa93c199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa933e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06bc60 0x7fa93c199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44910/0 (socket says 172.21.15.33:44910) 2024-04-26T11:36:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa933e00640 1 -- 172.21.15.33:0/1185467919 learned_addr learned my addr 172.21.15.33:0/1185467919 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa93a800640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa93c1089e0 0x7fa93c19d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.198+0000 7fa939e00640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa93c06b2a0 0x7fa93c199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.199+0000 7fa933e00640 1 -- 172.21.15.33:0/1185467919 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa93c1089e0 msgr2=0x7fa93c19d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.199+0000 7fa933e00640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa93c1089e0 0x7fa93c19d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.199+0000 7fa933e00640 1 -- 172.21.15.33:0/1185467919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa93c06b2a0 msgr2=0x7fa93c199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.199+0000 7fa933e00640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa93c06b2a0 0x7fa93c199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.199+0000 7fa933e00640 1 -- 172.21.15.33:0/1185467919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93c19e010 con 0x7fa93c06bc60 2024-04-26T11:36:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.199+0000 7fa933e00640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06bc60 0x7fa93c199560 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa924009230 tx=0x7fa924004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.199+0000 7fa932a00640 1 -- 172.21.15.33:0/1185467919 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92403f030 con 0x7fa93c06bc60 2024-04-26T11:36:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.200+0000 7fa932a00640 1 -- 172.21.15.33:0/1185467919 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa924004440 con 0x7fa93c06bc60 2024-04-26T11:36:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.200+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93c19e2a0 con 0x7fa93c06bc60 2024-04-26T11:36:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.200+0000 7fa932a00640 1 -- 172.21.15.33:0/1185467919 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924035070 con 0x7fa93c06bc60 2024-04-26T11:36:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.200+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa93c1a5b90 con 0x7fa93c06bc60 2024-04-26T11:36:49.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.203+0000 7fa9441cc740 1 Processor -- start 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa9441cc740 1 -- start start 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed53135c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa932a00640 1 -- 172.21.15.33:0/1185467919 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa924042030 con 0x7fa93c06bc60 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed5313b00 0x561ed531de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed53207b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa9441cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ed5274380 con 0x561ed531e3c0 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa9441cc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ed5210c30 con 0x561ed53131e0 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa9441cc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561ed5270340 con 0x561ed5313b00 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa939e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed5313b00 0x561ed531de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa939e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed5313b00 0x561ed531de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:34056/0 (socket says 172.21.15.33:34056) 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa939e00640 1 -- 172.21.15.33:0/164313143 learned_addr learned my addr 172.21.15.33:0/164313143 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa933e00640 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed53207b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.204+0000 7fa93a800640 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed53135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa939e00640 1 -- 172.21.15.33:0/164313143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 msgr2=0x561ed53135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa939e00640 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed53135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa939e00640 1 -- 172.21.15.33:0/164313143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 msgr2=0x561ed53207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa939e00640 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed53207b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa939e00640 1 -- 172.21.15.33:0/164313143 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed5232940 con 0x561ed5313b00 2024-04-26T11:36:49.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa93a800640 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed53135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:36:49.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa939e00640 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed5313b00 0x561ed531de80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009cf0 tx=0x7fa92c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba6452c2375c401b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:49.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa92b400640 1 -- 172.21.15.33:0/164313143 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c004230 con 0x561ed5313b00 2024-04-26T11:36:49.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa92b400640 1 -- 172.21.15.33:0/164313143 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa92c0043d0 con 0x561ed5313b00 2024-04-26T11:36:49.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.205+0000 7fa92b400640 1 -- 172.21.15.33:0/164313143 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c019880 con 0x561ed5313b00 2024-04-26T11:36:49.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa932a00640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa9200a3e40 0x7fa9200a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa932a00640 1 -- 172.21.15.33:0/1185467919 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9014+0+0 (secure 0 0 0) 0x7fa9240f0cb0 con 0x7fa93c06bc60 2024-04-26T11:36:49.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa93a800640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa9200a3e40 0x7fa9200a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa9441cc740 1 -- 172.21.15.33:0/164313143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed5313b00 msgr2=0x561ed531de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa9441cc740 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed5313b00 0x561ed531de80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009cf0 tx=0x7fa92c011810 comp rx=0 tx=0).stop 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa9441cc740 1 -- 172.21.15.33:0/164313143 shutdown_connections 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa9441cc740 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed5313b00 0x561ed531de80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa9441cc740 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed53207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa9441cc740 1 --2- 172.21.15.33:0/164313143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed53135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.206+0000 7fa9441cc740 1 -- 172.21.15.33:0/164313143 >> 172.21.15.33:0/164313143 conn(0x561ed530f6f0 msgr2=0x561ed530fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.207+0000 7fa9441cc740 1 -- 172.21.15.33:0/164313143 shutdown_connections 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.207+0000 7fa9441cc740 1 -- 172.21.15.33:0/164313143 wait complete. 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.207+0000 7fa93a800640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa9200a3e40 0x7fa9200a6300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa934009fd0 tx=0x7fa934009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.207+0000 7fa9441cc740 1 -- start start 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.207+0000 7fa9441cc740 1 Processor -- start 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed531b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed531b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed531e3c0 0x561ed531bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa9441cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561ed5320ff0 con 0x561ed5313b00 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa9441cc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ed51fbc60 con 0x561ed53131e0 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa9441cc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561ed5321170 con 0x561ed531e3c0 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa9441cc740 10 client.0 _ll_register_callbacks cb 0x561ed5227e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa93a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed531e3c0 0x561ed531bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa933e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed531b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa939e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed531b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa933e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed531b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44920/0 (socket says 172.21.15.33:44920) 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa933e00640 1 -- 172.21.15.33:0/1575938349 learned_addr learned my addr 172.21.15.33:0/1575938349 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa939e00640 1 -- 172.21.15.33:0/1575938349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed531e3c0 msgr2=0x561ed531bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa939e00640 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed531e3c0 0x561ed531bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa939e00640 1 -- 172.21.15.33:0/1575938349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 msgr2=0x561ed531b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa939e00640 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed531b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.208+0000 7fa939e00640 1 -- 172.21.15.33:0/1575938349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed520a510 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.209+0000 7fa939e00640 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed531b340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa92c003f50 tx=0x7fa92c003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c54a176ff17f6b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.209+0000 7fa91fe00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.209+0000 7fa91fe00640 1 -- 172.21.15.33:0/1575938349 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c027070 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.209+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa92c0043d0 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.209+0000 7fa91fe00640 1 -- 172.21.15.33:0/1575938349 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9014+0+0 (secure 0 0 0) 0x7fa92c005240 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa91fe00640 1 -- 172.21.15.33:0/1575938349 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa92c033050 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 10 client.25814 fetch_fsmap learned FSMap version 73 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x561ed4f86210 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa91fe00640 1 -- 172.21.15.33:0/1575938349 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa92c0043d0 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa91fe00640 1 -- 172.21.15.33:0/1575938349 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 73) ==== 8346+0+0 (secure 0 0 0) 0x7fa92c004b70 con 0x561ed53131e0 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 10 client.25814 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T11:36:49.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 10 client.25814 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 20 client.25814 populate_metadata read hostname 'smithi033' 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed532e0a0 0x561ed5330490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 4 client.25814 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.210+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ed5281f90 con 0x561ed532e0a0 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.211+0000 7fa933e00640 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed532e0a0 0x561ed5330490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.211+0000 7fa933e00640 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed532e0a0 0x561ed5330490 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c380053705c89e3 server_cookie=5b698836ed36e943 in_seq=0 out_seq=0 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.211+0000 7fa91fe00640 10 client.25814 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa91fe00640 1 -- 172.21.15.33:0/1575938349 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ed5281f90 con 0x561ed532e0a0 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa91fe00640 10 client.25814 handle_command_reply: tid=0 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 client.25814 shutdown 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 msgr2=0x561ed531b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed531b340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa92c003f50 tx=0x7fa92c003820 comp rx=0 tx=0).stop 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 shutdown_connections 2024-04-26T11:36:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed531e3c0 0x561ed531bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed532e0a0 0x561ed5330490 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed531b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1575938349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed53131e0 0x561ed531b340 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa91fe00640 0 client.25814 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 >> 172.21.15.33:0/1575938349 conn(0x561ed530f6f0 msgr2=0x561ed530a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 shutdown_connections 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 1 -- 172.21.15.33:0/1575938349 wait complete. 2024-04-26T11:36:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.212+0000 7fa9441cc740 20 client.25814 trim_cache size 0 max 16384 2024-04-26T11:36:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 Processor -- start 2024-04-26T11:36:49.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 -- start start 2024-04-26T11:36:49.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed531da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed5313b00 0x561ed5329eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed530a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c1aefc0 con 0x561ed531e3c0 2024-04-26T11:36:49.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c1ac830 con 0x561ed5313b00 2024-04-26T11:36:49.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.229+0000 7fa9441cc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c1aa680 con 0x561ed53131e0 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa93a800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed530a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed5313b00 0x561ed5329eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed5313b00 0x561ed5329eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:43374/0 (socket says 172.21.15.33:43374) 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa93a800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed530a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44926/0 (socket says 172.21.15.33:44926) 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 -- 172.21.15.33:0/1334819763 learned_addr learned my addr 172.21.15.33:0/1334819763 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa939e00640 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed531da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 -- 172.21.15.33:0/1334819763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 msgr2=0x561ed531da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed531da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 -- 172.21.15.33:0/1334819763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 msgr2=0x561ed530a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed530a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 -- 172.21.15.33:0/1334819763 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed5313930 con 0x561ed5313b00 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.230+0000 7fa933e00640 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed5313b00 0x561ed5329eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa9240bca40 tx=0x7fa9240f4760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdbf8ad5dd6c73af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa92aa00640 1 -- 172.21.15.33:0/1334819763 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924049040 con 0x561ed5313b00 2024-04-26T11:36:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa92aa00640 1 -- 172.21.15.33:0/1334819763 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9240bcab0 con 0x561ed5313b00 2024-04-26T11:36:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa92aa00640 1 -- 172.21.15.33:0/1334819763 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9240b9120 con 0x561ed5313b00 2024-04-26T11:36:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa9441cc740 1 -- 172.21.15.33:0/1334819763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed5313b00 msgr2=0x561ed5329eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed5313b00 0x561ed5329eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa9240bca40 tx=0x7fa9240f4760 comp rx=0 tx=0).stop 2024-04-26T11:36:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa9441cc740 1 -- 172.21.15.33:0/1334819763 shutdown_connections 2024-04-26T11:36:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed531e3c0 0x561ed530a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed5313b00 0x561ed5329eb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa9441cc740 1 --2- 172.21.15.33:0/1334819763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed531da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.231+0000 7fa9441cc740 1 -- 172.21.15.33:0/1334819763 >> 172.21.15.33:0/1334819763 conn(0x561ed530f6f0 msgr2=0x561ed5309bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.232+0000 7fa9441cc740 1 -- 172.21.15.33:0/1334819763 shutdown_connections 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.232+0000 7fa9441cc740 1 -- 172.21.15.33:0/1334819763 wait complete. 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.232+0000 7fa9441cc740 1 -- start start 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.232+0000 7fa9441cc740 1 Processor -- start 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed5318730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed5318c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa9441cc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed531e3c0 0x561ed5393520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa9441cc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c1aacf0 con 0x561ed5313b00 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa9441cc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c1ac160 con 0x561ed531e3c0 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa9441cc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c1ab3c0 con 0x561ed53131e0 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa9441cc740 10 client.0 _ll_register_callbacks cb 0x7fa9240f4630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:36:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed531e3c0 0x561ed5393520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed531e3c0 0x561ed5393520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:43388/0 (socket says 172.21.15.33:43388) 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 -- 172.21.15.33:0/2825844281 learned_addr learned my addr 172.21.15.33:0/2825844281 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa933e00640 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed5318c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa939e00640 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed5318730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 -- 172.21.15.33:0/2825844281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 msgr2=0x561ed5318730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed5318730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 -- 172.21.15.33:0/2825844281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 msgr2=0x561ed5318c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed5318c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.233+0000 7fa93a800640 1 -- 172.21.15.33:0/2825844281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed5052620 con 0x561ed531e3c0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa93a800640 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed531e3c0 0x561ed5393520 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa934009d20 tx=0x7fa934004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=878abc56ac7d0b14 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa91fe00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa91fe00640 1 -- 172.21.15.33:0/2825844281 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93402e060 con 0x561ed531e3c0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9240bcab0 con 0x561ed531e3c0 2024-04-26T11:36:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa91fe00640 1 -- 172.21.15.33:0/2825844281 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9014+0+0 (secure 0 0 0) 0x7fa934003b80 con 0x561ed531e3c0 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa91fe00640 1 -- 172.21.15.33:0/2825844281 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fa934035050 con 0x561ed531e3c0 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa9441cc740 10 client.25820 fetch_fsmap learned FSMap version 73 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x561ed522ecb0 con 0x561ed531e3c0 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.234+0000 7fa91fe00640 1 -- 172.21.15.33:0/2825844281 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9240bcab0 con 0x561ed531e3c0 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa91fe00640 1 -- 172.21.15.33:0/2825844281 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 73) ==== 8346+0+0 (secure 0 0 0) 0x7fa93401e560 con 0x561ed531e3c0 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa9441cc740 10 client.25820 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa9441cc740 10 client.25820 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa9441cc740 20 client.25820 populate_metadata read hostname 'smithi033' 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa9441cc740 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed53951e0 0x561ed5397600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa9441cc740 4 client.25820 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:36:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561ed527be00 con 0x561ed53951e0 2024-04-26T11:36:49.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.235+0000 7fa939e00640 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed53951e0 0x561ed5397600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:36:49.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.236+0000 7fa939e00640 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed53951e0 0x561ed5397600 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de5ec0e3c41be27e server_cookie=e9cc16c684a9ead5 in_seq=0 out_seq=0 2024-04-26T11:36:49.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:49.236+0000 7fa91fe00640 10 client.25820 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:50 smithi194 ceph-mon[33017]: pgmap v202: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 18 MiB/s wr, 559 op/s 2024-04-26T11:36:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:50 smithi033 ceph-mon[27845]: pgmap v202: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 18 MiB/s wr, 559 op/s 2024-04-26T11:36:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:50 smithi096 ceph-mon[32753]: pgmap v202: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 18 MiB/s wr, 559 op/s 2024-04-26T11:36:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 335.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:36:52.702 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:36:52.703 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 systemd[1]: Stopped Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:36:52.703 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.077s CPU time. 2024-04-26T11:36:52.703 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 systemd[1]: Starting Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:36:52.703 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 podman[67971]: 2024-04-26 11:36:52.574087104 +0000 UTC m=+0.018408083 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:36:52.703 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 podman[67971]: 2024-04-26 11:36:52.701334784 +0000 UTC m=+0.145655743 container create 8f194aa04823abfb651a560a65f19d6a2f981f29a35b20cfd0e66c6ddcad11ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T11:36:52.954 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:52 smithi096 ceph-mon[32753]: pgmap v203: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 459 op/s 2024-04-26T11:36:52.955 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 podman[67971]: 2024-04-26 11:36:52.929167737 +0000 UTC m=+0.373488721 container init 8f194aa04823abfb651a560a65f19d6a2f981f29a35b20cfd0e66c6ddcad11ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True) 2024-04-26T11:36:52.955 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 podman[67971]: 2024-04-26 11:36:52.931942861 +0000 UTC m=+0.376263813 container start 8f194aa04823abfb651a560a65f19d6a2f981f29a35b20cfd0e66c6ddcad11ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:36:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:52 smithi194 ceph-mon[33017]: pgmap v203: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 459 op/s 2024-04-26T11:36:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:52 smithi033 ceph-mon[27845]: pgmap v203: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 459 op/s 2024-04-26T11:36:53.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 ceph-mds[67989]: -- 172.21.15.96:0/1533557237 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fafae1a40 con 0x564fafab4c00 2024-04-26T11:36:53.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:52 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: starting mds.h at 2024-04-26T11:36:53.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:53 smithi096 bash[67971]: 8f194aa04823abfb651a560a65f19d6a2f981f29a35b20cfd0e66c6ddcad11ab 2024-04-26T11:36:53.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:36:53 smithi096 systemd[1]: Started Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: daemon mds.h restarted 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:boot 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:53 smithi194 ceph-mon[33017]: pgmap v205: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 468 op/s 2024-04-26T11:36:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: daemon mds.h restarted 2024-04-26T11:36:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:36:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T11:36:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:boot 2024-04-26T11:36:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:36:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:53 smithi033 ceph-mon[27845]: pgmap v205: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 468 op/s 2024-04-26T11:36:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: daemon mds.h restarted 2024-04-26T11:36:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:36:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:36:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T11:36:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:boot 2024-04-26T11:36:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:36:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:36:54.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:54.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:53 smithi096 ceph-mon[32753]: pgmap v205: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 468 op/s 2024-04-26T11:36:55.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:54 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:resolve 2024-04-26T11:36:55.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:54 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:54 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:resolve 2024-04-26T11:36:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:54 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:54 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:resolve 2024-04-26T11:36:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:54 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:56.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.018+0000 7fa91fe00640 1 -- 172.21.15.33:0/2825844281 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561ed527be00 con 0x561ed53951e0 2024-04-26T11:36:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.018+0000 7fa91fe00640 10 client.25820 handle_command_reply: tid=0 2024-04-26T11:36:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.018+0000 7fa9441cc740 1 client.25820 shutdown 2024-04-26T11:36:56.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.018+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed531e3c0 msgr2=0x561ed5393520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.018+0000 7fa9441cc740 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed531e3c0 0x561ed5393520 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa934009d20 tx=0x7fa934004a40 comp rx=0 tx=0).stop 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 shutdown_connections 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa9441cc740 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x561ed53951e0 0x561ed5397600 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa9441cc740 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ed531e3c0 0x561ed5393520 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa9441cc740 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561ed5313b00 0x561ed5318c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa9441cc740 1 --2- 172.21.15.33:0/2825844281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ed53131e0 0x561ed5318730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa91fe00640 0 client.25820 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:36:56.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 >> 172.21.15.33:0/2825844281 conn(0x561ed530f6f0 msgr2=0x561ed530a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.019+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 shutdown_connections 2024-04-26T11:36:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.020+0000 7fa9441cc740 1 -- 172.21.15.33:0/2825844281 wait complete. 2024-04-26T11:36:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.020+0000 7fa9441cc740 20 client.25820 trim_cache size 0 max 16384 2024-04-26T11:36:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:36:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:36:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:36:56.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:36:56.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa9200a3e40 msgr2=0x7fa9200a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:56.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa9200a3e40 0x7fa9200a6300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa934009fd0 tx=0x7fa934009500 comp rx=0 tx=0).stop 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06bc60 msgr2=0x7fa93c199560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06bc60 0x7fa93c199560 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa924009230 tx=0x7fa924004040 comp rx=0 tx=0).stop 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 shutdown_connections 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fa9200a3e40 0x7fa9200a6300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa93c1089e0 0x7fa93c19d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa93c06bc60 0x7fa93c199560 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 --2- 172.21.15.33:0/1185467919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa93c06b2a0 0x7fa93c199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.021+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 >> 172.21.15.33:0/1185467919 conn(0x7fa93c077b70 msgr2=0x7fa93c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.022+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 shutdown_connections 2024-04-26T11:36:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:36:56.022+0000 7fa941200640 1 -- 172.21.15.33:0/1185467919 wait complete. 2024-04-26T11:36:56.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:56 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399992 2024-04-26T11:36:56.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:56 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:reconnect 2024-04-26T11:36:56.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:56 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:56.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:56 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:36:56.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:56 smithi033 ceph-mon[27845]: pgmap v206: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-26T11:36:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:56 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399992 2024-04-26T11:36:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:56 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:reconnect 2024-04-26T11:36:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:56 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:56 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:36:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:56 smithi194 ceph-mon[33017]: pgmap v206: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-26T11:36:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:56 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399992 2024-04-26T11:36:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:56 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:reconnect 2024-04-26T11:36:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:56 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:56 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:36:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:56 smithi096 ceph-mon[32753]: pgmap v206: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-26T11:36:57.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 340.0 s, 411.9 tps, lat 746.360 ms stddev 3160.644 2024-04-26T11:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:57 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:rejoin 2024-04-26T11:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:57 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:57 smithi194 ceph-mon[33017]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:57 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:57 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:57 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:active 2024-04-26T11:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:57 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:57 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:rejoin 2024-04-26T11:36:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:57 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:57 smithi033 ceph-mon[27845]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:36:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:57 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:57 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:36:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:57 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:active 2024-04-26T11:36:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:57 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:57 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:rejoin 2024-04-26T11:36:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:57 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:36:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:57 smithi096 ceph-mon[32753]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:36:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:57 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:36:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:57 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:36:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:57 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/3788090565,v1:172.21.15.96:6835/3788090565] up:active 2024-04-26T11:36:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:57 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:36:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:36:58 smithi194 ceph-mon[33017]: pgmap v207: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-26T11:36:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:36:58 smithi033 ceph-mon[27845]: pgmap v207: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-26T11:36:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:36:58 smithi096 ceph-mon[32753]: pgmap v207: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-26T11:37:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:00 smithi033 ceph-mon[27845]: pgmap v208: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 6.7 MiB/s wr, 221 op/s 2024-04-26T11:37:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:00 smithi096 ceph-mon[32753]: pgmap v208: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 6.7 MiB/s wr, 221 op/s 2024-04-26T11:37:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:00 smithi194 ceph-mon[33017]: pgmap v208: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 6.7 MiB/s wr, 221 op/s 2024-04-26T11:37:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 345.0 s, 1850.4 tps, lat 54.024 ms stddev 75.200 2024-04-26T11:37:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:02 smithi096 ceph-mon[32753]: pgmap v209: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 11 MiB/s wr, 281 op/s 2024-04-26T11:37:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:02 smithi194 ceph-mon[33017]: pgmap v209: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 11 MiB/s wr, 281 op/s 2024-04-26T11:37:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:02 smithi033 ceph-mon[27845]: pgmap v209: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 11 MiB/s wr, 281 op/s 2024-04-26T11:37:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:04 smithi194 ceph-mon[33017]: pgmap v210: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 10 MiB/s wr, 270 op/s 2024-04-26T11:37:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:04 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:04 smithi033 ceph-mon[27845]: pgmap v210: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 10 MiB/s wr, 270 op/s 2024-04-26T11:37:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:04 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:04 smithi096 ceph-mon[32753]: pgmap v210: 97 pgs: 97 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 10 MiB/s wr, 270 op/s 2024-04-26T11:37:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:04 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:06 smithi194 ceph-mon[33017]: pgmap v211: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 25 MiB/s wr, 1.19k op/s 2024-04-26T11:37:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:06 smithi033 ceph-mon[27845]: pgmap v211: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 25 MiB/s wr, 1.19k op/s 2024-04-26T11:37:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:06 smithi096 ceph-mon[32753]: pgmap v211: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 25 MiB/s wr, 1.19k op/s 2024-04-26T11:37:07.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 350.0 s, 2413.8 tps, lat 41.451 ms stddev 49.477 2024-04-26T11:37:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:07 smithi194 ceph-mon[33017]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T11:37:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:07 smithi033 ceph-mon[27845]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T11:37:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:07 smithi096 ceph-mon[32753]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T11:37:08.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-mon[32753]: pgmap v213: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 30 MiB/s wr, 1.37k op/s 2024-04-26T11:37:08.759 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:08 smithi033 ceph-mon[27845]: pgmap v213: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 30 MiB/s wr, 1.37k op/s 2024-04-26T11:37:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:08 smithi194 ceph-mon[33017]: pgmap v213: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 30 MiB/s wr, 1.37k op/s 2024-04-26T11:37:09.149 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efe3a84a700 time 2024-04-26T11:37:08.717230+0000 2024-04-26T11:37:09.149 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe44e733dd] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.150 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.151 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.151 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: clone() 2024-04-26T11:37:09.151 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:08.717+0000 7efe3a84a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efe3a84a700 time 2024-04-26T11:37:08.717230+0000 2024-04-26T11:37:09.151 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:09.151 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.152 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.152 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe44e733dd] 2024-04-26T11:37:09.152 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.152 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.152 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.152 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: clone() 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: *** Caught signal (Aborted) ** 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: in thread 7efe3a84a700 thread_name:ms_dispatch 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.153 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe43bfbd20] 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: gsignal() 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: abort() 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe44e73437] 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.155 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 16: clone() 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:08.719+0000 7efe3a84a700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: in thread 7efe3a84a700 thread_name:ms_dispatch 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe43bfbd20] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: gsignal() 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: abort() 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe44e73437] 2024-04-26T11:37:09.156 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 16: clone() 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.157 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: -1> 2024-04-26T11:37:08.717+0000 7efe3a84a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efe3a84a700 time 2024-04-26T11:37:08.717230+0000 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe44e733dd] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.158 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: clone() 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 0> 2024-04-26T11:37:08.719+0000 7efe3a84a700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: in thread 7efe3a84a700 thread_name:ms_dispatch 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe43bfbd20] 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: gsignal() 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: abort() 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe44e73437] 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.159 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 16: clone() 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: -1> 2024-04-26T11:37:08.717+0000 7efe3a84a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efe3a84a700 time 2024-04-26T11:37:08.717230+0000 2024-04-26T11:37:09.160 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efe44e733dd] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.161 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: clone() 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 0> 2024-04-26T11:37:08.719+0000 7efe3a84a700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: in thread 7efe3a84a700 thread_name:ms_dispatch 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efe43bfbd20] 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2: gsignal() 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 3: abort() 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efe44e73437] 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7efe44e735a3] 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556d9c41b822] 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556d9c38b598] 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d9c38dc17] 2024-04-26T11:37:09.162 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d9c286d0b] 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d9c2872ac] 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556d9c26ffad] 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7efe45107e20] 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 13: (DispatchQueue::entry()+0x51f) [0x7efe45104fdf] 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efe451d7fc1] 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efe43bf11ca] 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 16: clone() 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[67690]: 2024-04-26T11:37:09.163 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:08 smithi096 podman[68035]: 2024-04-26 11:37:08.780140869 +0000 UTC m=+0.023252845 container died 03bbbac3c9b9fcedd975f96845920dcc94903d4640c9676ac6fbaff19150cc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T11:37:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 -- 172.21.15.33:0/4181411378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1060109190 msgr2=0x7f1060109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 --2- 172.21.15.33:0/4181411378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1060109190 0x7f1060109570 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1050009ca0 tx=0x7f105002a8c0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 -- 172.21.15.33:0/4181411378 shutdown_connections 2024-04-26T11:37:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 --2- 172.21.15.33:0/4181411378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1060109b40 0x7f106010db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 --2- 172.21.15.33:0/4181411378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f106010e260 0x7f1060115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 --2- 172.21.15.33:0/4181411378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1060109190 0x7f1060109570 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 -- 172.21.15.33:0/4181411378 >> 172.21.15.33:0/4181411378 conn(0x7f1060076ec0 msgr2=0x7f10600792e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 -- 172.21.15.33:0/4181411378 shutdown_connections 2024-04-26T11:37:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.165+0000 7f1065600640 1 -- 172.21.15.33:0/4181411378 wait complete. 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.166+0000 7f1065600640 1 Processor -- start 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.166+0000 7f1065600640 1 -- start start 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1065600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1060109190 0x7f106019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1065600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106010e260 0x7f106019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1065600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10601a3f50 0x7f10601a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1065600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10601191f0 con 0x7f1060109190 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1065600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1060119070 con 0x7f106010e260 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1065600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1060119370 con 0x7f10601a3f50 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f105ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1060109190 0x7f106019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f105ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1060109190 0x7f106019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:32970/0 (socket says 172.21.15.33:32970) 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f105ea00640 1 -- 172.21.15.33:0/460486250 learned_addr learned my addr 172.21.15.33:0/460486250 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1056000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106010e260 0x7f106019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f105f400640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10601a3f50 0x7f10601a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1056000640 1 -- 172.21.15.33:0/460486250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10601a3f50 msgr2=0x7f10601a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1056000640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10601a3f50 0x7f10601a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1056000640 1 -- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1060109190 msgr2=0x7f106019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1056000640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1060109190 0x7f106019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f1056000640 1 -- 172.21.15.33:0/460486250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10601a4a40 con 0x7f106010e260 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.167+0000 7f105ea00640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1060109190 0x7f106019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.168+0000 7f1056000640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106010e260 0x7f106019fbd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1048005fd0 tx=0x7f104800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.168+0000 7f1057e00640 1 -- 172.21.15.33:0/460486250 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048009c00 con 0x7f106010e260 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.168+0000 7f1057e00640 1 -- 172.21.15.33:0/460486250 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1048009da0 con 0x7f106010e260 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.168+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10601a4d30 con 0x7f106010e260 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.168+0000 7f1057e00640 1 -- 172.21.15.33:0/460486250 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048003e40 con 0x7f106010e260 2024-04-26T11:37:09.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.168+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10601ac570 con 0x7f106010e260 2024-04-26T11:37:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.172+0000 7f1068478740 1 Processor -- start 2024-04-26T11:37:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1068478740 1 -- start start 2024-04-26T11:37:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1057e00640 1 -- 172.21.15.33:0/460486250 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1048004000 con 0x7f106010e260 2024-04-26T11:37:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1068478740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e0fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1068478740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e106a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1068478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e1093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1068478740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555d0e061380 con 0x555d0e0fc6f0 2024-04-26T11:37:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1068478740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555d0dffdc30 con 0x555d0e0fbdd0 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1068478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555d0e05d340 con 0x555d0e106fb0 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f105f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e0fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f105ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f105f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e0fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33004/0 (socket says 172.21.15.33:33004) 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f105f400640 1 -- 172.21.15.33:0/668320343 learned_addr learned my addr 172.21.15.33:0/668320343 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.173+0000 7f1056000640 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e1093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.174+0000 7f105f400640 1 -- 172.21.15.33:0/668320343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 msgr2=0x555d0e1093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.174+0000 7f105f400640 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e1093a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.174+0000 7f105f400640 1 -- 172.21.15.33:0/668320343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 msgr2=0x555d0e106a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.174+0000 7f105f400640 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e106a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.174+0000 7f105f400640 1 -- 172.21.15.33:0/668320343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d0e01f940 con 0x555d0e0fbdd0 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.174+0000 7f105f400640 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e0fc1b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1058009c20 tx=0x7f1058011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42ab15f7d1b34b06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f104f400640 1 -- 172.21.15.33:0/668320343 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105801e070 con 0x555d0e0fbdd0 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f104f400640 1 -- 172.21.15.33:0/668320343 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1058019ae0 con 0x555d0e0fbdd0 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f104f400640 1 -- 172.21.15.33:0/668320343 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105800e900 con 0x555d0e0fbdd0 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f1057e00640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f10440a3e40 0x7f10440a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f1057e00640 1 -- 172.21.15.33:0/460486250 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9188+0+0 (secure 0 0 0) 0x7f10480d0550 con 0x7f106010e260 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f105ea00640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f10440a3e40 0x7f10440a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f1068478740 1 -- 172.21.15.33:0/668320343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 msgr2=0x555d0e0fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.175+0000 7f1068478740 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e0fc1b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1058009c20 tx=0x7f1058011810 comp rx=0 tx=0).stop 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 -- 172.21.15.33:0/668320343 shutdown_connections 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e1093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 --2- 172.21.15.33:0/668320343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e0fc1b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 -- 172.21.15.33:0/668320343 >> 172.21.15.33:0/668320343 conn(0x555d0e0f82e0 msgr2=0x555d0e0f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 -- 172.21.15.33:0/668320343 shutdown_connections 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 -- 172.21.15.33:0/668320343 wait complete. 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f105ea00640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f10440a3e40 0x7f10440a6300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x555d0e0fd370 tx=0x7f1050005d80 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.176+0000 7f1068478740 1 -- start start 2024-04-26T11:37:09.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 1 Processor -- start 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e103f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e104450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e104990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555d0e109be0 con 0x555d0e0fc6f0 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555d0dfe8c60 con 0x555d0e0fbdd0 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555d0e109d60 con 0x555d0e106fb0 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1068478740 10 client.0 _ll_register_callbacks cb 0x555d0e014e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f105ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e103f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f105ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e103f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33008/0 (socket says 172.21.15.33:33008) 2024-04-26T11:37:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f105ea00640 1 -- 172.21.15.33:0/647599940 learned_addr learned my addr 172.21.15.33:0/647599940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f105f400640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e104990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.177+0000 7f1056000640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e104450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f105f400640 1 -- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 msgr2=0x555d0e103f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f105f400640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e103f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f105f400640 1 -- 172.21.15.33:0/647599940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 msgr2=0x555d0e104450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f105f400640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e104450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f105f400640 1 -- 172.21.15.33:0/647599940 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d0dff7510 con 0x555d0e106fb0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f105ea00640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f105f400640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e104990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1058011d10 tx=0x7f1058009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22d1a0b255d43077 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f1043e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f1043e00640 1 -- 172.21.15.33:0/647599940 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105801e050 con 0x555d0e106fb0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.178+0000 7f1043e00640 1 -- 172.21.15.33:0/647599940 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9188+0+0 (secure 0 0 0) 0x7f1058019740 con 0x555d0e106fb0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1058019ae0 con 0x555d0e106fb0 2024-04-26T11:37:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1043e00640 1 -- 172.21.15.33:0/647599940 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f1058032050 con 0x555d0e106fb0 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 10 client.25777 fetch_fsmap learned FSMap version 80 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x555d0dd73210 con 0x555d0e106fb0 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1043e00640 1 -- 172.21.15.33:0/647599940 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1058019ae0 con 0x555d0e106fb0 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1043e00640 1 -- 172.21.15.33:0/647599940 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 80) ==== 8350+0+0 (secure 0 0 0) 0x7f1058004b50 con 0x555d0e106fb0 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 10 client.25777 fetch_fsmap finished waiting for FSMap version 80 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 10 client.25777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 20 client.25777 populate_metadata read hostname 'smithi033' 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e116c00 0x555d0e118ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 4 client.25777 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.179+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d0e06ef90 con 0x555d0e116c00 2024-04-26T11:37:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.180+0000 7f105ea00640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e116c00 0x555d0e118ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.183+0000 7f105ea00640 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e116c00 0x555d0e118ff0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b8ee68b37497bc0 server_cookie=9ccf18534d52b26 in_seq=0 out_seq=0 2024-04-26T11:37:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.186+0000 7f1043e00640 10 client.25777 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.187+0000 7f1043e00640 1 -- 172.21.15.33:0/647599940 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d0e06ef90 con 0x555d0e116c00 2024-04-26T11:37:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.187+0000 7f1043e00640 10 client.25777 handle_command_reply: tid=0 2024-04-26T11:37:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.187+0000 7f1068478740 1 client.25777 shutdown 2024-04-26T11:37:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.190+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 msgr2=0x555d0e104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.190+0000 7f1068478740 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e104990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1058011d10 tx=0x7f1058009f00 comp rx=0 tx=0).stop 2024-04-26T11:37:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.191+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 shutdown_connections 2024-04-26T11:37:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.191+0000 7f1068478740 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e104990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.192+0000 7f1068478740 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e116c00 0x555d0e118ff0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.192+0000 7f1068478740 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e104450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.192+0000 7f1068478740 1 --2- 172.21.15.33:0/647599940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fbdd0 0x555d0e103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.192+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 >> 172.21.15.33:0/647599940 conn(0x555d0e0f82e0 msgr2=0x555d0e0fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.192+0000 7f1043e00640 0 client.25777 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.193+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 shutdown_connections 2024-04-26T11:37:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.194+0000 7f1068478740 1 -- 172.21.15.33:0/647599940 wait complete. 2024-04-26T11:37:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.195+0000 7f1068478740 20 client.25777 trim_cache size 0 max 16384 2024-04-26T11:37:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 Processor -- start 2024-04-26T11:37:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 -- start start 2024-04-26T11:37:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fbdd0 0x555d0e112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fc6f0 0x555d0e0f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e11e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10601b7b70 con 0x555d0e0fbdd0 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10601b53b0 con 0x555d0e0fc6f0 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.220+0000 7f1068478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f10601b3220 con 0x555d0e106fb0 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f1056000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fc6f0 0x555d0e0f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f1056000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fc6f0 0x555d0e0f72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33018/0 (socket says 172.21.15.33:33018) 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f1056000640 1 -- 172.21.15.33:0/3305315738 learned_addr learned my addr 172.21.15.33:0/3305315738 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105f400640 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e11e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105ea00640 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fbdd0 0x555d0e112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105f400640 1 -- 172.21.15.33:0/3305315738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fc6f0 msgr2=0x555d0e0f72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105f400640 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fc6f0 0x555d0e0f72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105f400640 1 -- 172.21.15.33:0/3305315738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fbdd0 msgr2=0x555d0e112aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105f400640 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fbdd0 0x555d0e112aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105f400640 1 -- 172.21.15.33:0/3305315738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d0e0fc520 con 0x555d0e106fb0 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f1056000640 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fc6f0 0x555d0e0f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.221+0000 7f105f400640 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e11e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f105800e7b0 tx=0x7f1058005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d66fb24fa48ba5ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f104ea00640 1 -- 172.21.15.33:0/3305315738 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058015070 con 0x555d0e106fb0 2024-04-26T11:37:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f104ea00640 1 -- 172.21.15.33:0/3305315738 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f105801f040 con 0x555d0e106fb0 2024-04-26T11:37:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f104ea00640 1 -- 172.21.15.33:0/3305315738 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105801e070 con 0x555d0e106fb0 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 -- 172.21.15.33:0/3305315738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 msgr2=0x555d0e11e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e11e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f105800e7b0 tx=0x7f1058005970 comp rx=0 tx=0).stop 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 -- 172.21.15.33:0/3305315738 shutdown_connections 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e106fb0 0x555d0e11e760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e0fc6f0 0x555d0e0f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 --2- 172.21.15.33:0/3305315738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fbdd0 0x555d0e112aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 -- 172.21.15.33:0/3305315738 >> 172.21.15.33:0/3305315738 conn(0x555d0e0f82e0 msgr2=0x555d0e0f6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.222+0000 7f1068478740 1 -- 172.21.15.33:0/3305315738 shutdown_connections 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.223+0000 7f1068478740 1 -- 172.21.15.33:0/3305315738 wait complete. 2024-04-26T11:37:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.223+0000 7f1068478740 1 -- start start 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.223+0000 7f1068478740 1 Processor -- start 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.223+0000 7f1068478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e0fbdd0 0x555d0e10f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1068478740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e100860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1068478740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e106fb0 0x555d0e100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1068478740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10601b3870 con 0x555d0e0fc6f0 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1068478740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10601b4ce0 con 0x555d0e106fb0 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1068478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f10601b3f40 con 0x555d0e0fbdd0 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1068478740 10 client.0 _ll_register_callbacks cb 0x7f105801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e0fbdd0 0x555d0e10f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e0fbdd0 0x555d0e10f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:49326/0 (socket says 172.21.15.33:49326) 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 -- 172.21.15.33:0/1111788270 learned_addr learned my addr 172.21.15.33:0/1111788270 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1056000640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105f400640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e106fb0 0x555d0e100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 -- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e106fb0 msgr2=0x555d0e100da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e106fb0 0x555d0e100da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 -- 172.21.15.33:0/1111788270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 msgr2=0x555d0e100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e100860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 -- 172.21.15.33:0/1111788270 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d0de3f620 con 0x555d0e0fbdd0 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f1056000640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.224+0000 7f105ea00640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e0fbdd0 0x555d0e10f280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x555d0e119700 tx=0x7f1050042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=502414b6a108b90e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1043e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1043e00640 1 -- 172.21.15.33:0/1111788270 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050003dd0 con 0x555d0e0fbdd0 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f105801f040 con 0x555d0e0fbdd0 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1043e00640 1 -- 172.21.15.33:0/1111788270 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9188+0+0 (secure 0 0 0) 0x7f1050056410 con 0x555d0e0fbdd0 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1043e00640 1 -- 172.21.15.33:0/1111788270 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f1050065050 con 0x555d0e0fbdd0 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1068478740 10 client.25783 fetch_fsmap learned FSMap version 80 2024-04-26T11:37:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x555d0e01bcb0 con 0x555d0e0fbdd0 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.225+0000 7f1043e00640 1 -- 172.21.15.33:0/1111788270 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f105801f040 con 0x555d0e0fbdd0 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1043e00640 1 -- 172.21.15.33:0/1111788270 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 80) ==== 8350+0+0 (secure 0 0 0) 0x7f10500046b0 con 0x555d0e0fbdd0 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1068478740 10 client.25783 fetch_fsmap finished waiting for FSMap version 80 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1068478740 10 client.25783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1068478740 20 client.25783 populate_metadata read hostname 'smithi033' 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1068478740 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e17d2e0 0x555d0e17f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1068478740 4 client.25783 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d0e068e00 con 0x555d0e17d2e0 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1056000640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e17d2e0 0x555d0e17f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1056000640 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e17d2e0 0x555d0e17f700 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a930fe50fa4fb127 server_cookie=fed4b1838055720a in_seq=0 out_seq=0 2024-04-26T11:37:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:09.226+0000 7f1043e00640 10 client.25783 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:09.734 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:09 smithi096 podman[68035]: 2024-04-26 11:37:09.476103855 +0000 UTC m=+0.719215814 container remove 03bbbac3c9b9fcedd975f96845920dcc94903d4640c9676ac6fbaff19150cc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-26T11:37:09.734 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:09 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:37:10.182 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:09 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-26T11:37:10.183 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:09 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.209s CPU time. 2024-04-26T11:37:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:10 smithi194 ceph-mon[33017]: pgmap v214: 97 pgs: 97 active+clean; 3.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 29 MiB/s wr, 1.42k op/s 2024-04-26T11:37:10.898 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:10 smithi033 ceph-mon[27845]: pgmap v214: 97 pgs: 97 active+clean; 3.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 29 MiB/s wr, 1.42k op/s 2024-04-26T11:37:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:10 smithi096 ceph-mon[32753]: pgmap v214: 97 pgs: 97 active+clean; 3.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 29 MiB/s wr, 1.42k op/s 2024-04-26T11:37:12.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 355.0 s, 861.4 tps, lat 41.389 ms stddev 45.902 2024-04-26T11:37:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:12 smithi194 ceph-mon[33017]: pgmap v215: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 30 MiB/s wr, 1.48k op/s 2024-04-26T11:37:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:12 smithi033 ceph-mon[27845]: pgmap v215: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 30 MiB/s wr, 1.48k op/s 2024-04-26T11:37:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:12 smithi096 ceph-mon[32753]: pgmap v215: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 30 MiB/s wr, 1.48k op/s 2024-04-26T11:37:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:14 smithi033 ceph-mon[27845]: pgmap v216: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 30 MiB/s wr, 1.48k op/s 2024-04-26T11:37:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:14 smithi096 ceph-mon[32753]: pgmap v216: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 30 MiB/s wr, 1.48k op/s 2024-04-26T11:37:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:14 smithi194 ceph-mon[33017]: pgmap v216: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 30 MiB/s wr, 1.48k op/s 2024-04-26T11:37:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:16 smithi096 ceph-mon[32753]: pgmap v217: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-04-26T11:37:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:16 smithi096 ceph-mon[32753]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T11:37:16.850 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:16 smithi033 ceph-mon[27845]: pgmap v217: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-04-26T11:37:16.850 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:16 smithi033 ceph-mon[27845]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T11:37:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:16 smithi194 ceph-mon[33017]: pgmap v217: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-04-26T11:37:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:16 smithi194 ceph-mon[33017]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T11:37:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 360.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:37:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:18 smithi194 ceph-mon[33017]: pgmap v219: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-04-26T11:37:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:18 smithi033 ceph-mon[27845]: pgmap v219: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-04-26T11:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:18 smithi096 ceph-mon[32753]: pgmap v219: 97 pgs: 97 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-04-26T11:37:20.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:19 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:37:20.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:19 smithi096 systemd[1]: Stopped Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:37:20.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:19 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.209s CPU time. 2024-04-26T11:37:20.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:19 smithi096 systemd[1]: Starting Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:37:20.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 podman[68130]: 2024-04-26 11:37:20.069803982 +0000 UTC m=+0.018315522 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:37:20.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 podman[68130]: 2024-04-26 11:37:20.192959978 +0000 UTC m=+0.141471503 container create 3872f6b4111db0308f246660d37e6c6d07001ce80d4013f0af816c2109829558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2) 2024-04-26T11:37:20.445 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:20 smithi096 ceph-mon[32753]: pgmap v220: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 425 op/s 2024-04-26T11:37:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:20 smithi194 ceph-mon[33017]: pgmap v220: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 425 op/s 2024-04-26T11:37:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:20 smithi033 ceph-mon[27845]: pgmap v220: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 425 op/s 2024-04-26T11:37:20.932 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 podman[68130]: 2024-04-26 11:37:20.446003038 +0000 UTC m=+0.394514563 container init 3872f6b4111db0308f246660d37e6c6d07001ce80d4013f0af816c2109829558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:37:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 podman[68130]: 2024-04-26 11:37:20.449359293 +0000 UTC m=+0.397870820 container start 3872f6b4111db0308f246660d37e6c6d07001ce80d4013f0af816c2109829558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:37:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 ceph-mds[68149]: -- 172.21.15.96:0/159652327 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd9c4b3a40 con 0x55bd9c486c00 2024-04-26T11:37:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: starting mds.e at 2024-04-26T11:37:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 bash[68130]: 3872f6b4111db0308f246660d37e6c6d07001ce80d4013f0af816c2109829558 2024-04-26T11:37:20.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:20 smithi096 systemd[1]: Started Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: daemon mds.e restarted 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:boot 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:37:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:21 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: daemon mds.e restarted 2024-04-26T11:37:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:37:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:37:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T11:37:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:boot 2024-04-26T11:37:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:37:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:37:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:21 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: daemon mds.e restarted 2024-04-26T11:37:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:37:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:37:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T11:37:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:boot 2024-04-26T11:37:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:37:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:37:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:21 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:22.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 365.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:37:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:22 smithi194 ceph-mon[33017]: pgmap v222: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.3 MiB/s wr, 382 op/s 2024-04-26T11:37:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:22 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:resolve 2024-04-26T11:37:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:22 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:22 smithi033 ceph-mon[27845]: pgmap v222: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.3 MiB/s wr, 382 op/s 2024-04-26T11:37:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:22 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:resolve 2024-04-26T11:37:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:22 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:22 smithi096 ceph-mon[32753]: pgmap v222: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.3 MiB/s wr, 382 op/s 2024-04-26T11:37:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:22 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:resolve 2024-04-26T11:37:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:22 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.469+0000 7f1043e00640 1 -- 172.21.15.33:0/1111788270 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d0e068e00 con 0x555d0e17d2e0 2024-04-26T11:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.469+0000 7f1043e00640 10 client.25783 handle_command_reply: tid=0 2024-04-26T11:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.469+0000 7f1068478740 1 client.25783 shutdown 2024-04-26T11:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e0fbdd0 msgr2=0x555d0e10f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e0fbdd0 0x555d0e10f280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x555d0e119700 tx=0x7f1050042000 comp rx=0 tx=0).stop 2024-04-26T11:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 shutdown_connections 2024-04-26T11:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x555d0e17d2e0 0x555d0e17f700 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555d0e106fb0 0x555d0e100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555d0e0fc6f0 0x555d0e100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1043e00640 0 client.25783 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 --2- 172.21.15.33:0/1111788270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d0e0fbdd0 0x555d0e10f280 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 >> 172.21.15.33:0/1111788270 conn(0x555d0e0f82e0 msgr2=0x555d0e0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 shutdown_connections 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 1 -- 172.21.15.33:0/1111788270 wait complete. 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.470+0000 7f1068478740 20 client.25783 trim_cache size 0 max 16384 2024-04-26T11:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.471+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f10440a3e40 msgr2=0x7f10440a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.471+0000 7f1065600640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f10440a3e40 0x7f10440a6300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x555d0e0fd370 tx=0x7f1050005d80 comp rx=0 tx=0).stop 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.471+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106010e260 msgr2=0x7f106019fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.471+0000 7f1065600640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106010e260 0x7f106019fbd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1048005fd0 tx=0x7f104800dde0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 shutdown_connections 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f10440a3e40 0x7f10440a6300 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10601a3f50 0x7f10601a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106010e260 0x7f106019fbd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 --2- 172.21.15.33:0/460486250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1060109190 0x7f106019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 >> 172.21.15.33:0/460486250 conn(0x7f1060076ec0 msgr2=0x7f1060079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 shutdown_connections 2024-04-26T11:37:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:24.472+0000 7f1065600640 1 -- 172.21.15.33:0/460486250 wait complete. 2024-04-26T11:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:24 smithi194 ceph-mon[33017]: pgmap v223: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 506 KiB/s wr, 66 op/s 2024-04-26T11:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:reconnect 2024-04-26T11:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:24 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:24 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00599987 2024-04-26T11:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:24 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:37:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:24 smithi033 ceph-mon[27845]: pgmap v223: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 506 KiB/s wr, 66 op/s 2024-04-26T11:37:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:reconnect 2024-04-26T11:37:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:24 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:24 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00599987 2024-04-26T11:37:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:24 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:37:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:24 smithi096 ceph-mon[32753]: pgmap v223: 97 pgs: 97 active+clean; 3.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 506 KiB/s wr, 66 op/s 2024-04-26T11:37:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:reconnect 2024-04-26T11:37:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:24 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:24 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00599987 2024-04-26T11:37:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:24 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:37:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:rejoin 2024-04-26T11:37:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:25 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:25 smithi194 ceph-mon[33017]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:37:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:rejoin 2024-04-26T11:37:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:25 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:25 smithi033 ceph-mon[27845]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:37:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:rejoin 2024-04-26T11:37:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:25 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:25 smithi096 ceph-mon[32753]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:37:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:26 smithi194 ceph-mon[33017]: pgmap v224: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 10 MiB/s wr, 997 op/s 2024-04-26T11:37:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:26 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:37:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:26 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:37:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:26 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:active 2024-04-26T11:37:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:26 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:26 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:26 smithi033 ceph-mon[27845]: pgmap v224: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 10 MiB/s wr, 997 op/s 2024-04-26T11:37:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:26 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:37:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:26 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:37:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:26 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:active 2024-04-26T11:37:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:26 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:26 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:26 smithi096 ceph-mon[32753]: pgmap v224: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 10 MiB/s wr, 997 op/s 2024-04-26T11:37:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:26 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:37:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:26 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:37:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:26 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/1265802935,v1:172.21.15.96:6837/1265802935] up:active 2024-04-26T11:37:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:26 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:26 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:27.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 370.0 s, 887.5 tps, lat 410.785 ms stddev 2506.614 2024-04-26T11:37:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:28 smithi194 ceph-mon[33017]: pgmap v225: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.1 MiB/s wr, 907 op/s 2024-04-26T11:37:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:28 smithi033 ceph-mon[27845]: pgmap v225: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.1 MiB/s wr, 907 op/s 2024-04-26T11:37:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:28 smithi096 ceph-mon[32753]: pgmap v225: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.1 MiB/s wr, 907 op/s 2024-04-26T11:37:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:29 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:29 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:29 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:30 smithi194 ceph-mon[33017]: pgmap v226: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-04-26T11:37:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:30 smithi033 ceph-mon[27845]: pgmap v226: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-04-26T11:37:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-mon[32753]: pgmap v226: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8c5dd0f700 time 2024-04-26T11:37:30.583992+0000 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8c683383dd] 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: clone() 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.584+0000 7f8c5dd0f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8c5dd0f700 time 2024-04-26T11:37:30.583992+0000 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:30.934 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8c683383dd] 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: clone() 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: *** Caught signal (Aborted) ** 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: in thread 7f8c5dd0f700 thread_name:ms_dispatch 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c670c0d20] 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: gsignal() 2024-04-26T11:37:30.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: abort() 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8c68338437] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 16: clone() 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.586+0000 7f8c5dd0f700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:30.937 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: in thread 7f8c5dd0f700 thread_name:ms_dispatch 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c670c0d20] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: gsignal() 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: abort() 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8c68338437] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.938 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 16: clone() 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: -1> 2024-04-26T11:37:30.584+0000 7f8c5dd0f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8c5dd0f700 time 2024-04-26T11:37:30.583992+0000 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8c683383dd] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.939 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: clone() 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 0> 2024-04-26T11:37:30.586+0000 7f8c5dd0f700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: in thread 7f8c5dd0f700 thread_name:ms_dispatch 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c670c0d20] 2024-04-26T11:37:30.940 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: gsignal() 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: abort() 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8c68338437] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 16: clone() 2024-04-26T11:37:30.941 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: -1> 2024-04-26T11:37:30.584+0000 7f8c5dd0f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8c5dd0f700 time 2024-04-26T11:37:30.583992+0000 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8c683383dd] 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.942 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: clone() 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 0> 2024-04-26T11:37:30.586+0000 7f8c5dd0f700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: in thread 7f8c5dd0f700 thread_name:ms_dispatch 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.943 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c670c0d20] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2: gsignal() 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 3: abort() 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8c68338437] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8c683385a3] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564fae654822] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564fae5c4598] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564fae5c6c17] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564fae4bfd0b] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564fae4c02ac] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564fae4a8fad] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f8c685cce20] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 13: (DispatchQueue::entry()+0x51f) [0x7f8c685c9fdf] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c6869cfc1] 2024-04-26T11:37:30.944 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8c670b61ca] 2024-04-26T11:37:30.945 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 16: clone() 2024-04-26T11:37:30.945 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:30.945 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[67985]: 2024-04-26T11:37:30.945 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:30 smithi096 podman[68187]: 2024-04-26 11:37:30.64583589 +0000 UTC m=+0.021143428 container died 8f194aa04823abfb651a560a65f19d6a2f981f29a35b20cfd0e66c6ddcad11ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:37:31.582 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:31 smithi096 podman[68187]: 2024-04-26 11:37:31.32598038 +0000 UTC m=+0.701287898 container remove 8f194aa04823abfb651a560a65f19d6a2f981f29a35b20cfd0e66c6ddcad11ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T11:37:31.582 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:31 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:37:31.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:31 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-26T11:37:31.933 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:31 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.011s CPU time. 2024-04-26T11:37:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 375.0 s, 2070.5 tps, lat 37.248 ms stddev 48.323 2024-04-26T11:37:32.850 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:32 smithi033 ceph-mon[27845]: pgmap v227: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 18 MiB/s wr, 1.03k op/s 2024-04-26T11:37:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:32 smithi194 ceph-mon[33017]: pgmap v227: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 18 MiB/s wr, 1.03k op/s 2024-04-26T11:37:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:32 smithi096 ceph-mon[32753]: pgmap v227: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 18 MiB/s wr, 1.03k op/s 2024-04-26T11:37:34.834 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:34 smithi033 ceph-mon[27845]: pgmap v228: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 16 MiB/s wr, 937 op/s 2024-04-26T11:37:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:34 smithi194 ceph-mon[33017]: pgmap v228: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 16 MiB/s wr, 937 op/s 2024-04-26T11:37:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:34 smithi096 ceph-mon[32753]: pgmap v228: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 16 MiB/s wr, 937 op/s 2024-04-26T11:37:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:36 smithi194 ceph-mon[33017]: pgmap v229: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-04-26T11:37:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:36 smithi033 ceph-mon[27845]: pgmap v229: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-04-26T11:37:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:36 smithi096 ceph-mon[32753]: pgmap v229: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-04-26T11:37:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 380.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:37:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 -- 172.21.15.33:0/2823370490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 msgr2=0x7f3564104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 --2- 172.21.15.33:0/2823370490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 0x7f3564104fc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3550009860 tx=0x7f355002a340 comp rx=0 tx=0).stop 2024-04-26T11:37:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 -- 172.21.15.33:0/2823370490 shutdown_connections 2024-04-26T11:37:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 --2- 172.21.15.33:0/2823370490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3564105500 0x7f35641119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 --2- 172.21.15.33:0/2823370490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 0x7f3564104fc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 --2- 172.21.15.33:0/2823370490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f356410ac00 0x7f356410afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 -- 172.21.15.33:0/2823370490 >> 172.21.15.33:0/2823370490 conn(0x7f3564100630 msgr2=0x7f3564102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 -- 172.21.15.33:0/2823370490 shutdown_connections 2024-04-26T11:37:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.640+0000 7f356b800640 1 -- 172.21.15.33:0/2823370490 wait complete. 2024-04-26T11:37:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.641+0000 7f356b800640 1 Processor -- start 2024-04-26T11:37:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.641+0000 7f356b800640 1 -- start start 2024-04-26T11:37:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f356b800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 0x7f356419b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f356b800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3564105500 0x7f356419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f356b800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f356410ac00 0x7f356419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f356b800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3564114df0 con 0x7f3564104b40 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f356b800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3564114c70 con 0x7f356410ac00 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f356b800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3564114f70 con 0x7f3564105500 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f3568e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 0x7f356419b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f3563e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3564105500 0x7f356419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f3563e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3564105500 0x7f356419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50560/0 (socket says 172.21.15.33:50560) 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f3563e00640 1 -- 172.21.15.33:0/3088766554 learned_addr learned my addr 172.21.15.33:0/3088766554 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f3569800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f356410ac00 0x7f356419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.642+0000 7f3563e00640 1 -- 172.21.15.33:0/3088766554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f356410ac00 msgr2=0x7f356419fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3563e00640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f356410ac00 0x7f356419fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3563e00640 1 -- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 msgr2=0x7f356419b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3563e00640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 0x7f356419b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3563e00640 1 -- 172.21.15.33:0/3088766554 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35641a0210 con 0x7f3564105500 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3568e00640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 0x7f356419b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3569800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f356410ac00 0x7f356419fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3563e00640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3564105500 0x7f356419b7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3550032d50 tx=0x7f3550032d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3561600640 1 -- 172.21.15.33:0/3088766554 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550037040 con 0x7f3564105500 2024-04-26T11:37:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35641a04a0 con 0x7f3564105500 2024-04-26T11:37:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3561600640 1 -- 172.21.15.33:0/3088766554 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3550041070 con 0x7f3564105500 2024-04-26T11:37:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f3561600640 1 -- 172.21.15.33:0/3088766554 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550003850 con 0x7f3564105500 2024-04-26T11:37:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.643+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35641a7d90 con 0x7f3564105500 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.646+0000 7f356e5bb740 1 Processor -- start 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f356e5bb740 1 -- start start 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f3561600640 1 -- 172.21.15.33:0/3088766554 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3550030480 con 0x7f3564105500 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c19341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c19346f0 0x55a5c193ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c193efb0 0x55a5c19413a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f356e5bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a5c1899380 con 0x55a5c193efb0 2024-04-26T11:37:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f356e5bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a5c1835c30 con 0x55a5c19346f0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f356e5bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a5c1895340 con 0x55a5c1933dd0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f3568e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c19346f0 0x55a5c193ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f3569800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c19341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f3569800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c19341b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:50564/0 (socket says 172.21.15.33:50564) 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f3569800640 1 -- 172.21.15.33:0/1610936488 learned_addr learned my addr 172.21.15.33:0/1610936488 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.647+0000 7f3563e00640 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c193efb0 0x55a5c19413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.648+0000 7f3568e00640 1 -- 172.21.15.33:0/1610936488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 msgr2=0x55a5c19341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.648+0000 7f3568e00640 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c19341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.648+0000 7f3568e00640 1 -- 172.21.15.33:0/1610936488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c193efb0 msgr2=0x55a5c19413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.648+0000 7f3568e00640 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c193efb0 0x55a5c19413a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.648+0000 7f3568e00640 1 -- 172.21.15.33:0/1610936488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c1857940 con 0x55a5c19346f0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.648+0000 7f3568e00640 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c19346f0 0x55a5c193ea70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3558011440 tx=0x7f3558011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1a958730365cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.648+0000 7f3556000640 1 -- 172.21.15.33:0/1610936488 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558004310 con 0x55a5c19346f0 2024-04-26T11:37:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f3556000640 1 -- 172.21.15.33:0/1610936488 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35580044b0 con 0x55a5c19346f0 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f3556000640 1 -- 172.21.15.33:0/1610936488 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355800fa70 con 0x55a5c19346f0 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f3561600640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f354c0a3e40 0x7f354c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f3561600640 1 -- 172.21.15.33:0/3088766554 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9186+0+0 (secure 0 0 0) 0x7f35500f0520 con 0x7f3564105500 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f3569800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f354c0a3e40 0x7f354c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f356e5bb740 1 -- 172.21.15.33:0/1610936488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c19346f0 msgr2=0x55a5c193ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c19346f0 0x55a5c193ea70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3558011440 tx=0x7f3558011900 comp rx=0 tx=0).stop 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f356e5bb740 1 -- 172.21.15.33:0/1610936488 shutdown_connections 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c193efb0 0x55a5c19413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c19346f0 0x55a5c193ea70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.649+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1610936488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c19341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.650+0000 7f356e5bb740 1 -- 172.21.15.33:0/1610936488 >> 172.21.15.33:0/1610936488 conn(0x55a5c19302e0 msgr2=0x55a5c19306c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.650+0000 7f356e5bb740 1 -- 172.21.15.33:0/1610936488 shutdown_connections 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.650+0000 7f356e5bb740 1 -- 172.21.15.33:0/1610936488 wait complete. 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.650+0000 7f3569800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f354c0a3e40 0x7f354c0a6300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f355c009fd0 tx=0x7f355c009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.650+0000 7f356e5bb740 1 -- start start 2024-04-26T11:37:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.650+0000 7f356e5bb740 1 Processor -- start 2024-04-26T11:37:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c1933dd0 0x55a5c193bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c19346f0 0x55a5c193c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c193c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f356e5bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a5c1941be0 con 0x55a5c1933dd0 2024-04-26T11:37:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f356e5bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a5c1820c60 con 0x55a5c193efb0 2024-04-26T11:37:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f356e5bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a5c1941d60 con 0x55a5c19346f0 2024-04-26T11:37:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f356e5bb740 10 client.0 _ll_register_callbacks cb 0x55a5c184ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f3568e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c1933dd0 0x55a5c193bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f3568e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c1933dd0 0x55a5c193bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53822/0 (socket says 172.21.15.33:53822) 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f3568e00640 1 -- 172.21.15.33:0/3162171329 learned_addr learned my addr 172.21.15.33:0/3162171329 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f3563e00640 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c19346f0 0x55a5c193c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.651+0000 7f3569800640 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c193c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f3568e00640 1 -- 172.21.15.33:0/3162171329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c19346f0 msgr2=0x55a5c193c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f3568e00640 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c19346f0 0x55a5c193c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f3568e00640 1 -- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 msgr2=0x55a5c193c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f3568e00640 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c193c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f3568e00640 1 -- 172.21.15.33:0/3162171329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c182f510 con 0x55a5c1933dd0 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f3568e00640 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c1933dd0 0x55a5c193bf30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f355801b040 tx=0x7f3558005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7570708afc1969e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f354aa00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:37:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f354aa00640 1 -- 172.21.15.33:0/3162171329 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558029020 con 0x55a5c1933dd0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.652+0000 7f354aa00640 1 -- 172.21.15.33:0/3162171329 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9186+0+0 (secure 0 0 0) 0x7f355800f9f0 con 0x55a5c1933dd0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35580044b0 con 0x55a5c1933dd0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f354aa00640 1 -- 172.21.15.33:0/3162171329 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f3558034050 con 0x55a5c1933dd0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f356e5bb740 10 client.15633 fetch_fsmap learned FSMap version 88 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x55a5c15ab210 con 0x55a5c1933dd0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f354aa00640 1 -- 172.21.15.33:0/3162171329 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35580044b0 con 0x55a5c1933dd0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f354aa00640 1 -- 172.21.15.33:0/3162171329 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 88) ==== 8350+0+0 (secure 0 0 0) 0x7f3558004c90 con 0x55a5c1933dd0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f356e5bb740 10 client.15633 fetch_fsmap finished waiting for FSMap version 88 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.653+0000 7f356e5bb740 10 client.15633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.654+0000 7f356e5bb740 20 client.15633 populate_metadata read hostname 'smithi033' 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.654+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c194ece0 0x55a5c19510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.654+0000 7f356e5bb740 4 client.15633 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.654+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5c18a6f90 con 0x55a5c194ece0 2024-04-26T11:37:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.654+0000 7f3563e00640 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c194ece0 0x55a5c19510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.654+0000 7f3563e00640 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c194ece0 0x55a5c19510d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abafb38b09acc47a server_cookie=f30eba754bafd1eb in_seq=0 out_seq=0 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.654+0000 7f354aa00640 10 client.15633 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.655+0000 7f354aa00640 1 -- 172.21.15.33:0/3162171329 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5c18a6f90 con 0x55a5c194ece0 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.655+0000 7f354aa00640 10 client.15633 handle_command_reply: tid=0 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.655+0000 7f356e5bb740 1 client.15633 shutdown 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.655+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c1933dd0 msgr2=0x55a5c193bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.655+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c1933dd0 0x55a5c193bf30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f355801b040 tx=0x7f3558005850 comp rx=0 tx=0).stop 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 shutdown_connections 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c19346f0 0x55a5c193c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c194ece0 0x55a5c19510d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c193c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3162171329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c1933dd0 0x55a5c193bf30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f354aa00640 0 client.15633 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 >> 172.21.15.33:0/3162171329 conn(0x55a5c19302e0 msgr2=0x55a5c1936010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 shutdown_connections 2024-04-26T11:37:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 1 -- 172.21.15.33:0/3162171329 wait complete. 2024-04-26T11:37:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.656+0000 7f356e5bb740 20 client.15633 trim_cache size 0 max 16384 2024-04-26T11:37:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.672+0000 7f356e5bb740 1 Processor -- start 2024-04-26T11:37:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.672+0000 7f356e5bb740 1 -- start start 2024-04-26T11:37:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c1933dd0 0x55a5c193e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c194aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c193efb0 0x55a5c19306c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f356e5bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35641b35b0 con 0x55a5c19346f0 2024-04-26T11:37:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f356e5bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35641b0df0 con 0x55a5c1933dd0 2024-04-26T11:37:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f356e5bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f35641aec40 con 0x55a5c193efb0 2024-04-26T11:37:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3563e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c194aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3563e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c194aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53834/0 (socket says 172.21.15.33:53834) 2024-04-26T11:37:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3563e00640 1 -- 172.21.15.33:0/1231115596 learned_addr learned my addr 172.21.15.33:0/1231115596 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3569800640 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c193efb0 0x55a5c19306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3568e00640 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c1933dd0 0x55a5c193e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3563e00640 1 -- 172.21.15.33:0/1231115596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c193efb0 msgr2=0x55a5c19306c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3563e00640 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c193efb0 0x55a5c19306c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3563e00640 1 -- 172.21.15.33:0/1231115596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c1933dd0 msgr2=0x55a5c193e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.673+0000 7f3563e00640 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c1933dd0 0x55a5c193e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.674+0000 7f3563e00640 1 -- 172.21.15.33:0/1231115596 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c193aee0 con 0x55a5c19346f0 2024-04-26T11:37:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.674+0000 7f3568e00640 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c1933dd0 0x55a5c193e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.674+0000 7f3563e00640 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c194aaa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35500bc330 tx=0x7f35500f29b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bc95d1952fd5bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.674+0000 7f3555600640 1 -- 172.21.15.33:0/1231115596 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500b8640 con 0x55a5c19346f0 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.674+0000 7f3555600640 1 -- 172.21.15.33:0/1231115596 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35500bcaa0 con 0x55a5c19346f0 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.674+0000 7f3555600640 1 -- 172.21.15.33:0/1231115596 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500bbd00 con 0x55a5c19346f0 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 -- 172.21.15.33:0/1231115596 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 msgr2=0x55a5c194aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c194aaa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35500bc330 tx=0x7f35500f29b0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 -- 172.21.15.33:0/1231115596 shutdown_connections 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c193efb0 0x55a5c19306c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c194aaa0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 --2- 172.21.15.33:0/1231115596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c1933dd0 0x55a5c193e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 -- 172.21.15.33:0/1231115596 >> 172.21.15.33:0/1231115596 conn(0x55a5c19302e0 msgr2=0x55a5c1936010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 -- 172.21.15.33:0/1231115596 shutdown_connections 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 -- 172.21.15.33:0/1231115596 wait complete. 2024-04-26T11:37:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 -- start start 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.675+0000 7f356e5bb740 1 Processor -- start 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c1938e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c19393a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f356e5bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c19398e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f356e5bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35641af2b0 con 0x55a5c19346f0 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f356e5bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35641b0720 con 0x55a5c193efb0 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f356e5bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f35641af980 con 0x55a5c1933dd0 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f356e5bb740 10 client.0 _ll_register_callbacks cb 0x7f355c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f3563e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c19393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f3563e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c19393a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53844/0 (socket says 172.21.15.33:53844) 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f3563e00640 1 -- 172.21.15.33:0/3185124067 learned_addr learned my addr 172.21.15.33:0/3185124067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f3568e00640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c1938e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.676+0000 7f3569800640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c19398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f3563e00640 1 -- 172.21.15.33:0/3185124067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 msgr2=0x55a5c1938e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f3563e00640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c1938e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f3563e00640 1 -- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 msgr2=0x55a5c19398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f3563e00640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c19398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f3563e00640 1 -- 172.21.15.33:0/3185124067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c1677620 con 0x55a5c19346f0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f3569800640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c19398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f3563e00640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c19393a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35500b8800 tx=0x7f35500bbd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3254c5c2edb4aa4b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f354aa00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f354aa00640 1 -- 172.21.15.33:0/3185124067 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500bbee0 con 0x55a5c19346f0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f354aa00640 1 -- 172.21.15.33:0/3185124067 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9186+0+0 (secure 0 0 0) 0x7f35500f0200 con 0x55a5c19346f0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35500bcaa0 con 0x55a5c19346f0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f354aa00640 1 -- 172.21.15.33:0/3185124067 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f35500bc170 con 0x55a5c19346f0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f356e5bb740 10 client.15639 fetch_fsmap learned FSMap version 88 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.677+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=4+}) -- 0x55a5c1853cb0 con 0x55a5c19346f0 2024-04-26T11:37:37.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f354aa00640 1 -- 172.21.15.33:0/3185124067 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35500bcaa0 con 0x55a5c19346f0 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f354aa00640 1 -- 172.21.15.33:0/3185124067 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 88) ==== 8350+0+0 (secure 0 0 0) 0x7f35500b8a10 con 0x55a5c19346f0 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f356e5bb740 10 client.15639 fetch_fsmap finished waiting for FSMap version 88 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f356e5bb740 10 client.15639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f356e5bb740 20 client.15639 populate_metadata read hostname 'smithi033' 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c19b57a0 0x55a5c19b7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f356e5bb740 4 client.15639 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a5c18a0e00 con 0x55a5c19b57a0 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.678+0000 7f3568e00640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c19b57a0 0x55a5c19b7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.679+0000 7f3568e00640 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c19b57a0 0x55a5c19b7bc0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=102e99fa0643c284 server_cookie=925b8962a5c57fa4 in_seq=0 out_seq=0 2024-04-26T11:37:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:37.679+0000 7f354aa00640 10 client.15639 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:38 smithi194 ceph-mon[33017]: pgmap v230: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 11 MiB/s wr, 408 op/s 2024-04-26T11:37:38.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:38 smithi033 ceph-mon[27845]: pgmap v230: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 11 MiB/s wr, 408 op/s 2024-04-26T11:37:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:38 smithi096 ceph-mon[32753]: pgmap v230: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 11 MiB/s wr, 408 op/s 2024-04-26T11:37:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:40 smithi194 ceph-mon[33017]: pgmap v231: 97 pgs: 97 active+clean; 3.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 11 MiB/s wr, 443 op/s 2024-04-26T11:37:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:40 smithi033 ceph-mon[27845]: pgmap v231: 97 pgs: 97 active+clean; 3.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 11 MiB/s wr, 443 op/s 2024-04-26T11:37:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:40 smithi096 ceph-mon[32753]: pgmap v231: 97 pgs: 97 active+clean; 3.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 11 MiB/s wr, 443 op/s 2024-04-26T11:37:42.032 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:41 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:37:42.032 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:41 smithi096 systemd[1]: Stopped Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:37:42.032 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:41 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.011s CPU time. 2024-04-26T11:37:42.032 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:41 smithi096 systemd[1]: Starting Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:37:42.032 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:41 smithi096 podman[68282]: 2024-04-26 11:37:41.808435418 +0000 UTC m=+0.018102013 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:37:42.032 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:41 smithi096 podman[68282]: 2024-04-26 11:37:41.917607698 +0000 UTC m=+0.127274293 container create b96121d0903d7389d17f5dc2570fad2d11809a3c543e92666f5e92d332b65a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-04-26T11:37:42.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 385.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:37:42.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:42 smithi096 podman[68282]: 2024-04-26 11:37:42.161978551 +0000 UTC m=+0.371645146 container init b96121d0903d7389d17f5dc2570fad2d11809a3c543e92666f5e92d332b65a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-26T11:37:42.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:42 smithi096 podman[68282]: 2024-04-26 11:37:42.165034145 +0000 UTC m=+0.374700750 container start b96121d0903d7389d17f5dc2570fad2d11809a3c543e92666f5e92d332b65a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux ) 2024-04-26T11:37:42.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mds[68301]: -- 172.21.15.96:0/3618395594 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d5b5075a40 con 0x55d5b5048c00 2024-04-26T11:37:42.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: starting mds.h at 2024-04-26T11:37:42.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:42 smithi096 bash[68282]: b96121d0903d7389d17f5dc2570fad2d11809a3c543e92666f5e92d332b65a39 2024-04-26T11:37:42.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:37:42 smithi096 systemd[1]: Started Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: pgmap v232: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.4 MiB/s wr, 304 op/s 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: daemon mds.h restarted 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:boot 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:42 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: pgmap v232: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.4 MiB/s wr, 304 op/s 2024-04-26T11:37:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: daemon mds.h restarted 2024-04-26T11:37:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:37:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:37:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T11:37:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:boot 2024-04-26T11:37:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:37:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:37:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:42 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: pgmap v232: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.4 MiB/s wr, 304 op/s 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: daemon mds.h restarted 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:boot 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:42 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:37:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:37:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:37:44.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:44 smithi033 ceph-mon[27845]: pgmap v234: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.5 MiB/s wr, 277 op/s 2024-04-26T11:37:44.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:44 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:resolve 2024-04-26T11:37:44.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:44 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:44.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:37:44.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:37:44.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:44 smithi194 ceph-mon[33017]: pgmap v234: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.5 MiB/s wr, 277 op/s 2024-04-26T11:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:44 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:resolve 2024-04-26T11:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:44 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:44 smithi096 ceph-mon[32753]: pgmap v234: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.5 MiB/s wr, 277 op/s 2024-04-26T11:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:44 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:resolve 2024-04-26T11:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:44 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:37:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.594+0000 7f354aa00640 1 -- 172.21.15.33:0/3185124067 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a5c18a0e00 con 0x55a5c19b57a0 2024-04-26T11:37:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.594+0000 7f354aa00640 10 client.15639 handle_command_reply: tid=0 2024-04-26T11:37:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.594+0000 7f356e5bb740 1 client.15639 shutdown 2024-04-26T11:37:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 msgr2=0x55a5c19393a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c19393a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35500b8800 tx=0x7f35500bbd00 comp rx=0 tx=0).stop 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 shutdown_connections 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55a5c19b57a0 0x55a5c19b7bc0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a5c193efb0 0x55a5c19398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a5c19346f0 0x55a5c19393a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 --2- 172.21.15.33:0/3185124067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5c1933dd0 0x55a5c1938e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 >> 172.21.15.33:0/3185124067 conn(0x55a5c19302e0 msgr2=0x55a5c1936010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f354aa00640 0 client.15639 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 shutdown_connections 2024-04-26T11:37:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.595+0000 7f356e5bb740 1 -- 172.21.15.33:0/3185124067 wait complete. 2024-04-26T11:37:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.596+0000 7f356e5bb740 20 client.15639 trim_cache size 0 max 16384 2024-04-26T11:37:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:37:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:37:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:37:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:37:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f354c0a3e40 msgr2=0x7f354c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f354c0a3e40 0x7f354c0a6300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f355c009fd0 tx=0x7f355c009500 comp rx=0 tx=0).stop 2024-04-26T11:37:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3564105500 msgr2=0x7f356419b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3564105500 0x7f356419b7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3550032d50 tx=0x7f3550032d80 comp rx=0 tx=0).stop 2024-04-26T11:37:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 shutdown_connections 2024-04-26T11:37:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f354c0a3e40 0x7f354c0a6300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3564105500 0x7f356419b7b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f356410ac00 0x7f356419fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 --2- 172.21.15.33:0/3088766554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3564104b40 0x7f356419b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.597+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 >> 172.21.15.33:0/3088766554 conn(0x7f3564100630 msgr2=0x7f3564101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.598+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 shutdown_connections 2024-04-26T11:37:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:45.598+0000 7f356b800640 1 -- 172.21.15.33:0/3088766554 wait complete. 2024-04-26T11:37:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:45 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:reconnect 2024-04-26T11:37:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:45 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:45 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:37:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:45 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:37:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:45 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:reconnect 2024-04-26T11:37:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:45 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:45 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:37:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:45 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:37:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:45 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:reconnect 2024-04-26T11:37:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:45 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:45 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899981 2024-04-26T11:37:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:45 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: pgmap v235: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:rejoin 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:active 2024-04-26T11:37:46.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:46 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: pgmap v235: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:rejoin 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:active 2024-04-26T11:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:46 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: pgmap v235: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:rejoin 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/486225699,v1:172.21.15.96:6835/486225699] up:active 2024-04-26T11:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:46 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:37:47.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 390.0 s, 369.1 tps, lat 875.602 ms stddev 3512.188 2024-04-26T11:37:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:48 smithi194 ceph-mon[33017]: pgmap v236: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:37:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:48 smithi033 ceph-mon[27845]: pgmap v236: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:37:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:48 smithi096 ceph-mon[32753]: pgmap v236: 97 pgs: 97 active+clean; 3.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:37:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:50 smithi194 ceph-mon[33017]: pgmap v237: 97 pgs: 97 active+clean; 3.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-04-26T11:37:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:50 smithi033 ceph-mon[27845]: pgmap v237: 97 pgs: 97 active+clean; 3.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-04-26T11:37:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:50 smithi096 ceph-mon[32753]: pgmap v237: 97 pgs: 97 active+clean; 3.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-04-26T11:37:52.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 395.0 s, 2383.2 tps, lat 41.876 ms stddev 68.268 2024-04-26T11:37:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:52 smithi194 ceph-mon[33017]: pgmap v238: 97 pgs: 97 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 13 MiB/s wr, 971 op/s 2024-04-26T11:37:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:52 smithi033 ceph-mon[27845]: pgmap v238: 97 pgs: 97 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 13 MiB/s wr, 971 op/s 2024-04-26T11:37:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:52 smithi096 ceph-mon[32753]: pgmap v238: 97 pgs: 97 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 13 MiB/s wr, 971 op/s 2024-04-26T11:37:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:54 smithi194 ceph-mon[33017]: pgmap v239: 97 pgs: 97 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T11:37:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:54 smithi033 ceph-mon[27845]: pgmap v239: 97 pgs: 97 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T11:37:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-mon[32753]: pgmap v239: 97 pgs: 97 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f93a3335700 time 2024-04-26T11:37:54.642331+0000 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93ad95e3dd] 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: clone() 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.643+0000 7f93a3335700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f93a3335700 time 2024-04-26T11:37:54.642331+0000 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:54.934 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93ad95e3dd] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.936 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: clone() 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: *** Caught signal (Aborted) ** 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: in thread 7f93a3335700 thread_name:ms_dispatch 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93ac6e6d20] 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: gsignal() 2024-04-26T11:37:54.937 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: abort() 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93ad95e437] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.938 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 16: clone() 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.644+0000 7f93a3335700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: in thread 7f93a3335700 thread_name:ms_dispatch 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93ac6e6d20] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: gsignal() 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: abort() 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93ad95e437] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.939 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 16: clone() 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: -1> 2024-04-26T11:37:54.643+0000 7f93a3335700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f93a3335700 time 2024-04-26T11:37:54.642331+0000 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.940 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93ad95e3dd] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.941 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.942 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.942 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: clone() 2024-04-26T11:37:54.942 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.944 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 0> 2024-04-26T11:37:54.644+0000 7f93a3335700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:54.944 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: in thread 7f93a3335700 thread_name:ms_dispatch 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93ac6e6d20] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: gsignal() 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: abort() 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93ad95e437] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.945 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 16: clone() 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: -1> 2024-04-26T11:37:54.643+0000 7f93a3335700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f93a3335700 time 2024-04-26T11:37:54.642331+0000 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93ad95e3dd] 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.946 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: clone() 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 0> 2024-04-26T11:37:54.644+0000 7f93a3335700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: in thread 7f93a3335700 thread_name:ms_dispatch 2024-04-26T11:37:54.947 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93ac6e6d20] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2: gsignal() 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 3: abort() 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93ad95e437] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93ad95e5a3] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55bd9b3fd822] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55bd9b36d598] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bd9b36fc17] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bd9b268d0b] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bd9b2692ac] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55bd9b251fad] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f93adbf2e20] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 13: (DispatchQueue::entry()+0x51f) [0x7f93adbeffdf] 2024-04-26T11:37:54.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f93adcc2fc1] 2024-04-26T11:37:54.949 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93ac6dc1ca] 2024-04-26T11:37:54.949 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 16: clone() 2024-04-26T11:37:54.949 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:37:54.949 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68145]: 2024-04-26T11:37:54.949 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:54 smithi096 podman[68469]: 2024-04-26 11:37:54.705515515 +0000 UTC m=+0.020516272 container died 3872f6b4111db0308f246660d37e6c6d07001ce80d4013f0af816c2109829558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T11:37:55.626 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:55 smithi096 podman[68469]: 2024-04-26 11:37:55.333501077 +0000 UTC m=+0.648501831 container remove 3872f6b4111db0308f246660d37e6c6d07001ce80d4013f0af816c2109829558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T11:37:55.626 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:55 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:37:55.932 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:55 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-26T11:37:55.933 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:37:55 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.670s CPU time. 2024-04-26T11:37:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:56 smithi033 ceph-mon[27845]: pgmap v240: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 17 MiB/s wr, 1.21k op/s 2024-04-26T11:37:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:56 smithi096 ceph-mon[32753]: pgmap v240: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 17 MiB/s wr, 1.21k op/s 2024-04-26T11:37:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:56 smithi194 ceph-mon[33017]: pgmap v240: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 17 MiB/s wr, 1.21k op/s 2024-04-26T11:37:57.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 400.0 s, 2578.4 tps, lat 35.783 ms stddev 43.802 2024-04-26T11:37:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 -- 172.21.15.33:0/1820533599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020103100 msgr2=0x7f702010f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 --2- 172.21.15.33:0/1820533599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020103100 0x7f702010f610 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7018009930 tx=0x7f701802a420 comp rx=0 tx=0).stop 2024-04-26T11:37:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 -- 172.21.15.33:0/1820533599 shutdown_connections 2024-04-26T11:37:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 --2- 172.21.15.33:0/1820533599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020103100 0x7f702010f610 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 --2- 172.21.15.33:0/1820533599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7020102740 0x7f7020102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 --2- 172.21.15.33:0/1820533599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7020108750 0x7f7020108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 -- 172.21.15.33:0/1820533599 >> 172.21.15.33:0/1820533599 conn(0x7f70200fe270 msgr2=0x7f7020100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:58.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 -- 172.21.15.33:0/1820533599 shutdown_connections 2024-04-26T11:37:58.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.743+0000 7f7024a00640 1 -- 172.21.15.33:0/1820533599 wait complete. 2024-04-26T11:37:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.744+0000 7f7024a00640 1 Processor -- start 2024-04-26T11:37:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.744+0000 7f7024a00640 1 -- start start 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7024a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020102740 0x7f7020074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7024a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7020103100 0x7f7020075170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7024a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7020108750 0x7f70200756b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7024a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7020112a20 con 0x7f7020103100 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7024a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f70201128a0 con 0x7f7020108750 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7024a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7020112ba0 con 0x7f7020102740 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020102740 0x7f7020074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7016a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7020103100 0x7f7020075170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020102740 0x7f7020074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57736/0 (socket says 172.21.15.33:57736) 2024-04-26T11:37:58.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f7016a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7020103100 0x7f7020075170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47794/0 (socket says 172.21.15.33:47794) 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 -- 172.21.15.33:0/914598725 learned_addr learned my addr 172.21.15.33:0/914598725 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701fe00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7020108750 0x7f70200756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 -- 172.21.15.33:0/914598725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7020108750 msgr2=0x7f70200756b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7020108750 0x7f70200756b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 -- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7020103100 msgr2=0x7f7020075170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7020103100 0x7f7020075170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.745+0000 7f701f400640 1 -- 172.21.15.33:0/914598725 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70201a7460 con 0x7f7020102740 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.746+0000 7f701f400640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020102740 0x7f7020074c30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f701000db80 tx=0x7f701000b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:58.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.746+0000 7f701cc00640 1 -- 172.21.15.33:0/914598725 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010017070 con 0x7f7020102740 2024-04-26T11:37:58.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.746+0000 7f701cc00640 1 -- 172.21.15.33:0/914598725 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7010003b70 con 0x7f7020102740 2024-04-26T11:37:58.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.746+0000 7f701cc00640 1 -- 172.21.15.33:0/914598725 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70100106c0 con 0x7f7020102740 2024-04-26T11:37:58.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.746+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70201a7750 con 0x7f7020102740 2024-04-26T11:37:58.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.746+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70201a7cf0 con 0x7f7020102740 2024-04-26T11:37:58.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.750+0000 7f7028def740 1 Processor -- start 2024-04-26T11:37:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7028def740 1 -- start start 2024-04-26T11:37:58.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f701cc00640 1 -- 172.21.15.33:0/914598725 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7010004020 con 0x7f7020102740 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7028def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da20053c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7028def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7028def740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da20125b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7028def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556da1f6a380 con 0x556da2004fe0 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7028def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556da1f06c30 con 0x556da2005900 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7028def740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556da1f66340 con 0x556da20101c0 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f701fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da20053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f701f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f701fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da20053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47804/0 (socket says 172.21.15.33:47804) 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f701fe00640 1 -- 172.21.15.33:0/2260300255 learned_addr learned my addr 172.21.15.33:0/2260300255 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.751+0000 7f7016a00640 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da20125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701fe00640 1 -- 172.21.15.33:0/2260300255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 msgr2=0x556da20125b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701fe00640 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da20125b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701fe00640 1 -- 172.21.15.33:0/2260300255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 msgr2=0x556da200fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701fe00640 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701fe00640 1 -- 172.21.15.33:0/2260300255 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556da1f28940 con 0x556da2004fe0 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701f400640 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f7016a00640 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da20125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701fe00640 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da20053c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7018009900 tx=0x7f7018004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690311a86cee082b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f701cc00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f70040a3d90 0x7f70040a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.752+0000 7f700fe00640 1 -- 172.21.15.33:0/2260300255 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018040070 con 0x556da2004fe0 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f700fe00640 1 -- 172.21.15.33:0/2260300255 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7018003aa0 con 0x556da2004fe0 2024-04-26T11:37:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f700fe00640 1 -- 172.21.15.33:0/2260300255 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018035040 con 0x556da2004fe0 2024-04-26T11:37:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f701cc00640 1 -- 172.21.15.33:0/914598725 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9272+0+0 (secure 0 0 0) 0x7f70100d0980 con 0x7f7020102740 2024-04-26T11:37:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f701f400640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f70040a3d90 0x7f70040a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 -- 172.21.15.33:0/2260300255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 msgr2=0x556da20053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da20053c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7018009900 tx=0x7f7018004130 comp rx=0 tx=0).stop 2024-04-26T11:37:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 -- 172.21.15.33:0/2260300255 shutdown_connections 2024-04-26T11:37:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da20125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 --2- 172.21.15.33:0/2260300255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da20053c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 -- 172.21.15.33:0/2260300255 >> 172.21.15.33:0/2260300255 conn(0x556da20014f0 msgr2=0x556da20018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f701f400640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f70040a3d90 0x7f70040a6250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70100cde90 tx=0x7f70100d6530 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.753+0000 7f7028def740 1 -- 172.21.15.33:0/2260300255 shutdown_connections 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.754+0000 7f7028def740 1 -- 172.21.15.33:0/2260300255 wait complete. 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.754+0000 7f7028def740 1 -- start start 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.754+0000 7f7028def740 1 Processor -- start 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7028def740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2004fe0 0x556da200d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7028def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2005900 0x556da200d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7028def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da200dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7028def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556da2012d90 con 0x556da2005900 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7028def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556da1ef1c60 con 0x556da20101c0 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7028def740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556da2012f10 con 0x556da2004fe0 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7028def740 10 client.0 _ll_register_callbacks cb 0x556da1f1de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2005900 0x556da200d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2005900 0x556da200d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47808/0 (socket says 172.21.15.33:47808) 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 -- 172.21.15.33:0/365018106 learned_addr learned my addr 172.21.15.33:0/365018106 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f701fe00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2004fe0 0x556da200d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f701fe00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da200dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 -- 172.21.15.33:0/365018106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2004fe0 msgr2=0x556da200d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2004fe0 0x556da200d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 -- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 msgr2=0x556da200dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da200dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f7016a00640 1 -- 172.21.15.33:0/365018106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556da1f00510 con 0x556da2005900 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.755+0000 7f701fe00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2004fe0 0x556da200d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f7016a00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2005900 0x556da200d680 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7008012700 tx=0x7f7008012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c232d0af237d3746 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f700cc00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f700cc00640 1 -- 172.21.15.33:0/365018106 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008004050 con 0x556da2005900 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f700cc00640 1 -- 172.21.15.33:0/365018106 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9272+0+0 (secure 0 0 0) 0x7f70080041f0 con 0x556da2005900 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7018003aa0 con 0x556da2005900 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f700cc00640 1 -- 172.21.15.33:0/365018106 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f7008024050 con 0x556da2005900 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f7028def740 10 client.15651 fetch_fsmap learned FSMap version 95 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.756+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x556da1c7c210 con 0x556da2005900 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f700cc00640 1 -- 172.21.15.33:0/365018106 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7018003aa0 con 0x556da2005900 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f700cc00640 1 -- 172.21.15.33:0/365018106 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 95) ==== 8342+0+0 (secure 0 0 0) 0x7f7008003990 con 0x556da2005900 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f7028def740 10 client.15651 fetch_fsmap finished waiting for FSMap version 95 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f7028def740 10 client.15651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f7028def740 20 client.15651 populate_metadata read hostname 'smithi033' 2024-04-26T11:37:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f7028def740 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da201ffb0 0x556da20223e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f7028def740 4 client.15651 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556da1f77f90 con 0x556da201ffb0 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.757+0000 7f701fe00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da201ffb0 0x556da20223e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.758+0000 7f701fe00640 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da201ffb0 0x556da20223e0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab297377b026fc3 server_cookie=7668ab0149c4f4a3 in_seq=0 out_seq=0 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.758+0000 7f700cc00640 10 client.15651 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.758+0000 7f700cc00640 1 -- 172.21.15.33:0/365018106 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556da1f77f90 con 0x556da201ffb0 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.758+0000 7f700cc00640 10 client.15651 handle_command_reply: tid=0 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 client.15651 shutdown 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2005900 msgr2=0x556da200d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2005900 0x556da200d680 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7008012700 tx=0x7f7008012bc0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 shutdown_connections 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da201ffb0 0x556da20223e0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da200dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2005900 0x556da200d680 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f700cc00640 0 client.15651 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 --2- 172.21.15.33:0/365018106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2004fe0 0x556da200d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 >> 172.21.15.33:0/365018106 conn(0x556da20014f0 msgr2=0x556da2007270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 shutdown_connections 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 1 -- 172.21.15.33:0/365018106 wait complete. 2024-04-26T11:37:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.759+0000 7f7028def740 20 client.15651 trim_cache size 0 max 16384 2024-04-26T11:37:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.775+0000 7f7028def740 1 Processor -- start 2024-04-26T11:37:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.775+0000 7f7028def740 1 -- start start 2024-04-26T11:37:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7028def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da201bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7028def740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2005900 0x556da20079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7028def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da2027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7028def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b3690 con 0x556da2004fe0 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7028def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b0f30 con 0x556da20101c0 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7028def740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f70201aed80 con 0x556da2005900 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da201bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da201bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47810/0 (socket says 172.21.15.33:47810) 2024-04-26T11:37:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 -- 172.21.15.33:0/75371457 learned_addr learned my addr 172.21.15.33:0/75371457 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7016a00640 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2005900 0x556da20079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da2027cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 -- 172.21.15.33:0/75371457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2005900 msgr2=0x556da20079a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2005900 0x556da20079a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 -- 172.21.15.33:0/75371457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 msgr2=0x556da2027cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da2027cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f701fe00640 1 -- 172.21.15.33:0/75371457 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556da1d48620 con 0x556da2004fe0 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.776+0000 7f7016a00640 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2005900 0x556da20079a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.777+0000 7f701fe00640 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da201bc50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7018009930 tx=0x7f7018004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea83778110637a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.777+0000 7f700f400640 1 -- 172.21.15.33:0/75371457 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018035040 con 0x556da2004fe0 2024-04-26T11:37:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.777+0000 7f700f400640 1 -- 172.21.15.33:0/75371457 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70180054d0 con 0x556da2004fe0 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.777+0000 7f700f400640 1 -- 172.21.15.33:0/75371457 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018040040 con 0x556da2004fe0 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 -- 172.21.15.33:0/75371457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 msgr2=0x556da201bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da201bc50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7018009930 tx=0x7f7018004130 comp rx=0 tx=0).stop 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 -- 172.21.15.33:0/75371457 shutdown_connections 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da2005900 0x556da20079a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da20101c0 0x556da2027cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 --2- 172.21.15.33:0/75371457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da201bc50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 -- 172.21.15.33:0/75371457 >> 172.21.15.33:0/75371457 conn(0x556da20014f0 msgr2=0x556da200f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 -- 172.21.15.33:0/75371457 shutdown_connections 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 -- 172.21.15.33:0/75371457 wait complete. 2024-04-26T11:37:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 -- start start 2024-04-26T11:37:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.778+0000 7f7028def740 1 Processor -- start 2024-04-26T11:37:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7028def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da200a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7028def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7028def740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da2085560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7028def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f70201af3f0 con 0x556da2004fe0 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7028def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b0860 con 0x556da2005900 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7028def740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f70201afac0 con 0x556da20101c0 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7028def740 10 client.0 _ll_register_callbacks cb 0x7f70080097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f701fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da200a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:52896/0 (socket says 172.21.15.33:52896) 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 -- 172.21.15.33:0/2781396711 learned_addr learned my addr 172.21.15.33:0/2781396711 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f701fe00640 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da2085560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 -- 172.21.15.33:0/2781396711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 msgr2=0x556da2085560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da2085560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 -- 172.21.15.33:0/2781396711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 msgr2=0x556da200a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da200a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f7016a00640 1 -- 172.21.15.33:0/2781396711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556da1f24cb0 con 0x556da2005900 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.779+0000 7f701fe00640 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da2085560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f7016a00640 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200a930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x556da200e710 tx=0x7f700800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be64c37c992c499f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f700cc00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f700cc00640 1 -- 172.21.15.33:0/2781396711 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700800fbb0 con 0x556da2005900 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f700cc00640 1 -- 172.21.15.33:0/2781396711 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9272+0+0 (secure 0 0 0) 0x7f700800fd50 con 0x556da2005900 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70180054d0 con 0x556da2005900 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f700cc00640 1 -- 172.21.15.33:0/2781396711 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f7008039050 con 0x556da2005900 2024-04-26T11:37:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f7028def740 10 client.25892 fetch_fsmap learned FSMap version 95 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.780+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x556da2086a80 con 0x556da2005900 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f700cc00640 1 -- 172.21.15.33:0/2781396711 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70180054d0 con 0x556da2005900 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f700cc00640 1 -- 172.21.15.33:0/2781396711 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 95) ==== 8342+0+0 (secure 0 0 0) 0x7f7008005430 con 0x556da2005900 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f7028def740 10 client.25892 fetch_fsmap finished waiting for FSMap version 95 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f7028def740 10 client.25892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f7028def740 20 client.25892 populate_metadata read hostname 'smithi033' 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f7028def740 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da2087230 0x556da2089650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f7028def740 4 client.25892 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556da1f71e00 con 0x556da2087230 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.781+0000 7f701fe00640 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da2087230 0x556da2089650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.782+0000 7f700cc00640 10 client.25892 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:37:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:37:58.782+0000 7f701fe00640 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da2087230 0x556da2089650 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=addeccf8c54d28de server_cookie=be404cb065d74cdc in_seq=0 out_seq=0 2024-04-26T11:37:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:37:58 smithi033 ceph-mon[27845]: pgmap v241: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 17 MiB/s wr, 1.12k op/s 2024-04-26T11:37:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:37:58 smithi096 ceph-mon[32753]: pgmap v241: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 17 MiB/s wr, 1.12k op/s 2024-04-26T11:37:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:37:58 smithi194 ceph-mon[33017]: pgmap v241: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 17 MiB/s wr, 1.12k op/s 2024-04-26T11:38:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:00 smithi033 ceph-mon[27845]: pgmap v242: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 19 MiB/s wr, 1.27k op/s 2024-04-26T11:38:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:00 smithi096 ceph-mon[32753]: pgmap v242: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 19 MiB/s wr, 1.27k op/s 2024-04-26T11:38:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:00 smithi194 ceph-mon[33017]: pgmap v242: 97 pgs: 97 active+clean; 3.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 19 MiB/s wr, 1.27k op/s 2024-04-26T11:38:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 405.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:38:02.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:02 smithi194 ceph-mon[33017]: pgmap v243: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 15 MiB/s wr, 916 op/s 2024-04-26T11:38:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:02 smithi033 ceph-mon[27845]: pgmap v243: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 15 MiB/s wr, 916 op/s 2024-04-26T11:38:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:02 smithi096 ceph-mon[32753]: pgmap v243: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 15 MiB/s wr, 916 op/s 2024-04-26T11:38:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:04 smithi033 ceph-mon[27845]: pgmap v244: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 9.4 MiB/s wr, 623 op/s 2024-04-26T11:38:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:04 smithi096 ceph-mon[32753]: pgmap v244: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 9.4 MiB/s wr, 623 op/s 2024-04-26T11:38:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:04 smithi194 ceph-mon[33017]: pgmap v244: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 9.4 MiB/s wr, 623 op/s 2024-04-26T11:38:06.183 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:05 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:38:06.183 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:05 smithi096 systemd[1]: Stopped Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:38:06.183 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:05 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.670s CPU time. 2024-04-26T11:38:06.183 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:05 smithi096 systemd[1]: Starting Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:38:06.183 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:05 smithi096 podman[68567]: 2024-04-26 11:38:05.819398628 +0000 UTC m=+0.018281036 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:38:06.528 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:06 smithi096 podman[68567]: 2024-04-26 11:38:06.225808981 +0000 UTC m=+0.424691378 container create 328cb3e3d82528d5262fe8b7a93fc39132e628bba22be2860d097496bc5d4cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T11:38:06.529 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:06 smithi096 podman[68567]: 2024-04-26 11:38:06.436459327 +0000 UTC m=+0.635341722 container init 328cb3e3d82528d5262fe8b7a93fc39132e628bba22be2860d097496bc5d4cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424) 2024-04-26T11:38:06.529 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:06 smithi096 podman[68567]: 2024-04-26 11:38:06.439611849 +0000 UTC m=+0.638494250 container start 328cb3e3d82528d5262fe8b7a93fc39132e628bba22be2860d097496bc5d4cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T11:38:06.529 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mds[68585]: -- 172.21.15.96:0/1120134450 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ff53dba40 con 0x560ff53ae800 2024-04-26T11:38:06.529 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: starting mds.e at 2024-04-26T11:38:06.529 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:06 smithi096 bash[68567]: 328cb3e3d82528d5262fe8b7a93fc39132e628bba22be2860d097496bc5d4cc2 2024-04-26T11:38:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: pgmap v245: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 9.7 MiB/s wr, 669 op/s 2024-04-26T11:38:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: daemon mds.e restarted 2024-04-26T11:38:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:38:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T11:38:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:boot 2024-04-26T11:38:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:38:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:38:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:06 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:06.932 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:06 smithi096 systemd[1]: Started Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: pgmap v245: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 9.7 MiB/s wr, 669 op/s 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: daemon mds.e restarted 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:boot 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:06 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: pgmap v245: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 9.7 MiB/s wr, 669 op/s 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: daemon mds.e restarted 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:boot 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T11:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:06 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:07.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 410.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:38:08.913 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:08 smithi033 ceph-mon[27845]: pgmap v247: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 325 op/s 2024-04-26T11:38:08.913 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:08 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:resolve 2024-04-26T11:38:08.913 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:08 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:08 smithi096 ceph-mon[32753]: pgmap v247: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 325 op/s 2024-04-26T11:38:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:08 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:resolve 2024-04-26T11:38:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:08 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:08 smithi194 ceph-mon[33017]: pgmap v247: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 325 op/s 2024-04-26T11:38:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:08 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:resolve 2024-04-26T11:38:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:08 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.683+0000 7f700cc00640 1 -- 172.21.15.33:0/2781396711 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556da1f71e00 con 0x556da2087230 2024-04-26T11:38:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.683+0000 7f700cc00640 10 client.25892 handle_command_reply: tid=0 2024-04-26T11:38:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.683+0000 7f7028def740 1 client.25892 shutdown 2024-04-26T11:38:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 msgr2=0x556da200a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200a930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x556da200e710 tx=0x7f700800f790 comp rx=0 tx=0).stop 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 shutdown_connections 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556da20101c0 0x556da2085560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x556da2087230 0x556da2089650 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556da2005900 0x556da200a930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 --2- 172.21.15.33:0/2781396711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556da2004fe0 0x556da200a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f700cc00640 0 client.25892 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.684+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 >> 172.21.15.33:0/2781396711 conn(0x556da20014f0 msgr2=0x556da2000650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.685+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 shutdown_connections 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.685+0000 7f7028def740 1 -- 172.21.15.33:0/2781396711 wait complete. 2024-04-26T11:38:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.685+0000 7f7028def740 20 client.25892 trim_cache size 0 max 16384 2024-04-26T11:38:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:38:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:38:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:38:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f70040a3d90 msgr2=0x7f70040a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f70040a3d90 0x7f70040a6250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70100cde90 tx=0x7f70100d6530 comp rx=0 tx=0).stop 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020102740 msgr2=0x7f7020074c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020102740 0x7f7020074c30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f701000db80 tx=0x7f701000b630 comp rx=0 tx=0).stop 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 shutdown_connections 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f70040a3d90 0x7f70040a6250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7020108750 0x7f70200756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7020103100 0x7f7020075170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 --2- 172.21.15.33:0/914598725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7020102740 0x7f7020074c30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 >> 172.21.15.33:0/914598725 conn(0x7f70200fe270 msgr2=0x7f70200fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 shutdown_connections 2024-04-26T11:38:09.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:09.687+0000 7f7024a00640 1 -- 172.21.15.33:0/914598725 wait complete. 2024-04-26T11:38:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:09 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:38:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:09 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:reconnect 2024-04-26T11:38:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:09 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:09 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119997 2024-04-26T11:38:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:09 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:38:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:09 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:reconnect 2024-04-26T11:38:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:09 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:09 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119997 2024-04-26T11:38:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:09 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:38:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:09 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:reconnect 2024-04-26T11:38:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:09 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:09 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119997 2024-04-26T11:38:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:10 smithi096 ceph-mon[32753]: pgmap v248: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-04-26T11:38:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:10 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:rejoin 2024-04-26T11:38:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:10 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:10.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:10 smithi096 ceph-mon[32753]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:38:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:10 smithi194 ceph-mon[33017]: pgmap v248: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-04-26T11:38:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:10 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:rejoin 2024-04-26T11:38:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:10 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:10 smithi194 ceph-mon[33017]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:38:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:10 smithi033 ceph-mon[27845]: pgmap v248: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-04-26T11:38:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:10 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:rejoin 2024-04-26T11:38:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:10 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:10 smithi033 ceph-mon[27845]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-26T11:38:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:11 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:38:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:11 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:38:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:11 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:active 2024-04-26T11:38:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:11 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:11 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:38:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:11 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:38:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:11 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:38:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:11 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:active 2024-04-26T11:38:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:11 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:11 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:38:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:11 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:38:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:11 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:38:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:11 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6836/166226875,v1:172.21.15.96:6837/166226875] up:active 2024-04-26T11:38:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:11 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T11:38:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:11 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T11:38:12.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 415.0 s, 279.5 tps, lat 1081.711 ms stddev 3587.824 2024-04-26T11:38:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:12 smithi194 ceph-mon[33017]: pgmap v249: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 424 KiB/s wr, 58 op/s 2024-04-26T11:38:13.138 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:12 smithi096 ceph-mon[32753]: pgmap v249: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 424 KiB/s wr, 58 op/s 2024-04-26T11:38:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:12 smithi033 ceph-mon[27845]: pgmap v249: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 424 KiB/s wr, 58 op/s 2024-04-26T11:38:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:14 smithi194 ceph-mon[33017]: pgmap v250: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 424 KiB/s wr, 58 op/s 2024-04-26T11:38:15.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:14 smithi033 ceph-mon[27845]: pgmap v250: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 424 KiB/s wr, 58 op/s 2024-04-26T11:38:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:14 smithi096 ceph-mon[32753]: pgmap v250: 97 pgs: 97 active+clean; 3.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 424 KiB/s wr, 58 op/s 2024-04-26T11:38:15.740 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e4145c700 time 2024-04-26T11:38:15.703054+0000 2024-04-26T11:38:15.740 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:15.740 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.740 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e4ba853dd] 2024-04-26T11:38:15.740 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.740 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.740 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: clone() 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.703+0000 7f6e4145c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e4145c700 time 2024-04-26T11:38:15.703054+0000 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:15.741 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.742 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e4ba853dd] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.743 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: clone() 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: *** Caught signal (Aborted) ** 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: in thread 7f6e4145c700 thread_name:ms_dispatch 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e4a80dd20] 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: gsignal() 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: abort() 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e4ba85437] 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.744 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 16: clone() 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.705+0000 7f6e4145c700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: in thread 7f6e4145c700 thread_name:ms_dispatch 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.745 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e4a80dd20] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: gsignal() 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: abort() 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e4ba85437] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.746 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 16: clone() 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: -1> 2024-04-26T11:38:15.703+0000 7f6e4145c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e4145c700 time 2024-04-26T11:38:15.703054+0000 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e4ba853dd] 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.747 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: clone() 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 0> 2024-04-26T11:38:15.705+0000 7f6e4145c700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: in thread 7f6e4145c700 thread_name:ms_dispatch 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.748 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e4a80dd20] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: gsignal() 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: abort() 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e4ba85437] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.749 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 16: clone() 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: -1> 2024-04-26T11:38:15.703+0000 7f6e4145c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e4145c700 time 2024-04-26T11:38:15.703054+0000 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e4ba853dd] 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.750 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: clone() 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 0> 2024-04-26T11:38:15.705+0000 7f6e4145c700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:38:15.751 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: in thread 7f6e4145c700 thread_name:ms_dispatch 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e4a80dd20] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2: gsignal() 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 3: abort() 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e4ba85437] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e4ba855a3] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d5b3f06822] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d5b3e76598] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d5b3e78c17] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d5b3d71d0b] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d5b3d722ac] 2024-04-26T11:38:15.752 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d5b3d5afad] 2024-04-26T11:38:15.753 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e4bd19e20] 2024-04-26T11:38:15.753 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e4bd16fdf] 2024-04-26T11:38:15.753 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e4bde9fc1] 2024-04-26T11:38:15.753 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e4a8031ca] 2024-04-26T11:38:15.753 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 16: clone() 2024-04-26T11:38:15.753 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:38:15.753 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h[68297]: 2024-04-26T11:38:16.065 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:15 smithi096 podman[68633]: 2024-04-26 11:38:15.770874453 +0000 UTC m=+0.021595382 container died b96121d0903d7389d17f5dc2570fad2d11809a3c543e92666f5e92d332b65a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:38:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:16 smithi096 ceph-mon[32753]: pgmap v251: 97 pgs: 97 active+clean; 3.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 14 MiB/s wr, 1.34k op/s 2024-04-26T11:38:17.078 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:16 smithi096 podman[68633]: 2024-04-26 11:38:16.708587692 +0000 UTC m=+0.959308612 container remove b96121d0903d7389d17f5dc2570fad2d11809a3c543e92666f5e92d332b65a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T11:38:17.078 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:16 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:38:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:16 smithi194 ceph-mon[33017]: pgmap v251: 97 pgs: 97 active+clean; 3.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 14 MiB/s wr, 1.34k op/s 2024-04-26T11:38:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:16 smithi033 ceph-mon[27845]: pgmap v251: 97 pgs: 97 active+clean; 3.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 14 MiB/s wr, 1.34k op/s 2024-04-26T11:38:17.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 420.0 s, 2425.8 tps, lat 43.501 ms stddev 157.283 2024-04-26T11:38:17.432 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:17 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-26T11:38:17.433 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:17 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.175s CPU time. 2024-04-26T11:38:18.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:17 smithi033 ceph-mon[27845]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T11:38:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:17 smithi194 ceph-mon[33017]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T11:38:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:17 smithi096 ceph-mon[32753]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T11:38:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:18 smithi194 ceph-mon[33017]: pgmap v253: 97 pgs: 97 active+clean; 3.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 14 MiB/s wr, 1.34k op/s 2024-04-26T11:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:18 smithi033 ceph-mon[27845]: pgmap v253: 97 pgs: 97 active+clean; 3.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 14 MiB/s wr, 1.34k op/s 2024-04-26T11:38:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:18 smithi096 ceph-mon[32753]: pgmap v253: 97 pgs: 97 active+clean; 3.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 14 MiB/s wr, 1.34k op/s 2024-04-26T11:38:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:20 smithi194 ceph-mon[33017]: pgmap v254: 97 pgs: 97 active+clean; 3.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 17 MiB/s wr, 1.58k op/s 2024-04-26T11:38:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:20 smithi033 ceph-mon[27845]: pgmap v254: 97 pgs: 97 active+clean; 3.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 17 MiB/s wr, 1.58k op/s 2024-04-26T11:38:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:20 smithi096 ceph-mon[32753]: pgmap v254: 97 pgs: 97 active+clean; 3.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 17 MiB/s wr, 1.58k op/s 2024-04-26T11:38:22.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 425.0 s, 830.6 tps, lat 34.397 ms stddev 40.476 2024-04-26T11:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 -- 172.21.15.33:0/2314401388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc1014f0 msgr2=0x7f44bc1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/2314401388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc1014f0 0x7f44bc1018d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009900 tx=0x7f44b002a410 comp rx=0 tx=0).stop 2024-04-26T11:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 -- 172.21.15.33:0/2314401388 shutdown_connections 2024-04-26T11:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/2314401388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc101ea0 0x7f44bc10eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/2314401388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44bc10f0a0 0x7f44bc111490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/2314401388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc1014f0 0x7f44bc1018d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 -- 172.21.15.33:0/2314401388 >> 172.21.15.33:0/2314401388 conn(0x7f44bc0fd170 msgr2=0x7f44bc0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.842+0000 7f44c2200640 1 -- 172.21.15.33:0/2314401388 shutdown_connections 2024-04-26T11:38:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.843+0000 7f44c2200640 1 -- 172.21.15.33:0/2314401388 wait complete. 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 Processor -- start 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 -- start start 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44bc1014f0 0x7f44bc19b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc101ea0 0x7f44bc19b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc10f0a0 0x7f44bc19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc114c50 con 0x7f44bc101ea0 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc114ad0 con 0x7f44bc1014f0 2024-04-26T11:38:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.844+0000 7f44c2200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc114dd0 con 0x7f44bc10f0a0 2024-04-26T11:38:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc101ea0 0x7f44bc19b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc101ea0 0x7f44bc19b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50748/0 (socket says 172.21.15.33:50748) 2024-04-26T11:38:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44c0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44bc1014f0 0x7f44bc19b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44bbe00640 1 -- 172.21.15.33:0/432838530 learned_addr learned my addr 172.21.15.33:0/432838530 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44c1800640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc10f0a0 0x7f44bc19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44c0e00640 1 -- 172.21.15.33:0/432838530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc10f0a0 msgr2=0x7f44bc19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44c0e00640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc10f0a0 0x7f44bc19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44c0e00640 1 -- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc101ea0 msgr2=0x7f44bc19b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44c0e00640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc101ea0 0x7f44bc19b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44c0e00640 1 -- 172.21.15.33:0/432838530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44bc1a0200 con 0x7f44bc1014f0 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.845+0000 7f44bbe00640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc101ea0 0x7f44bc19b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.846+0000 7f44c0e00640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44bc1014f0 0x7f44bc19b210 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f44b002aa00 tx=0x7f44b0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.846+0000 7f44b9600640 1 -- 172.21.15.33:0/432838530 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b003f070 con 0x7f44bc1014f0 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.846+0000 7f44b9600640 1 -- 172.21.15.33:0/432838530 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b0031b50 con 0x7f44bc1014f0 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.846+0000 7f44b9600640 1 -- 172.21.15.33:0/432838530 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b0035030 con 0x7f44bc1014f0 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.846+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44bc1a0490 con 0x7f44bc1014f0 2024-04-26T11:38:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.846+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44bc1a0900 con 0x7f44bc1014f0 2024-04-26T11:38:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.849+0000 7f44c657f740 1 Processor -- start 2024-04-26T11:38:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.849+0000 7f44b9600640 1 -- 172.21.15.33:0/432838530 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f44b0042070 con 0x7f44bc1014f0 2024-04-26T11:38:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.849+0000 7f44c657f740 1 -- start start 2024-04-26T11:38:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c657f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c657f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c657f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c657f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562996157380 con 0x5629961fd1c0 2024-04-26T11:38:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c657f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5629960f3c30 con 0x5629961f1fe0 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c657f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562996153340 con 0x5629961f2900 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58018/0 (socket says 172.21.15.33:58018) 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c1800640 1 -- 172.21.15.33:0/3374744254 learned_addr learned my addr 172.21.15.33:0/3374744254 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44bbe00640 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.850+0000 7f44c0e00640 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44c1800640 1 -- 172.21.15.33:0/3374744254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 msgr2=0x5629961fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44c1800640 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44c1800640 1 -- 172.21.15.33:0/3374744254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 msgr2=0x5629961ff5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44c1800640 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961ff5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44c1800640 1 -- 172.21.15.33:0/3374744254 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562996115940 con 0x5629961f1fe0 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44bbe00640 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44c1800640 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961f23c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009f30 tx=0x7f44b40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18582cfc2882911d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44b9600640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f44a40a3d10 0x7f44a40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44ae000640 1 -- 172.21.15.33:0/3374744254 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b4004d20 con 0x5629961f1fe0 2024-04-26T11:38:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44b9600640 1 -- 172.21.15.33:0/432838530 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9446+0+0 (secure 0 0 0) 0x7f44b00f10e0 con 0x7f44bc1014f0 2024-04-26T11:38:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44ae000640 1 -- 172.21.15.33:0/3374744254 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b4004050 con 0x5629961f1fe0 2024-04-26T11:38:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.851+0000 7f44ae000640 1 -- 172.21.15.33:0/3374744254 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b4005260 con 0x5629961f1fe0 2024-04-26T11:38:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c0e00640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f44a40a3d10 0x7f44a40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 -- 172.21.15.33:0/3374744254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 msgr2=0x5629961f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961f23c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009f30 tx=0x7f44b40118d0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 -- 172.21.15.33:0/3374744254 shutdown_connections 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 --2- 172.21.15.33:0/3374744254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961f23c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 -- 172.21.15.33:0/3374744254 >> 172.21.15.33:0/3374744254 conn(0x5629961ee4f0 msgr2=0x5629961ee8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 -- 172.21.15.33:0/3374744254 shutdown_connections 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c0e00640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f44a40a3d10 0x7f44a40a61d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x5629961f3580 tx=0x7f44b0031400 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.852+0000 7f44c657f740 1 -- 172.21.15.33:0/3374744254 wait complete. 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 -- start start 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 Processor -- start 2024-04-26T11:38:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5629961ffdf0 con 0x5629961fd1c0 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5629960dec60 con 0x5629961f1fe0 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.853+0000 7f44c657f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5629961fff70 con 0x5629961f2900 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44c657f740 10 client.0 _ll_register_callbacks cb 0x56299610ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44c1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44c1800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44c1800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50780/0 (socket says 172.21.15.33:50780) 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44c1800640 1 -- 172.21.15.33:0/124228724 learned_addr learned my addr 172.21.15.33:0/124228724 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44bbe00640 1 -- 172.21.15.33:0/124228724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 msgr2=0x5629961fa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44bbe00640 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961fa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44bbe00640 1 -- 172.21.15.33:0/124228724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 msgr2=0x5629961fab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44bbe00640 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.854+0000 7f44bbe00640 1 -- 172.21.15.33:0/124228724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629960ed510 con 0x5629961f2900 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44bbe00640 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fa600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x5629961ef680 tx=0x7f44a8012d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9afceddf3ad88ba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44a2a00640 1 -- 172.21.15.33:0/124228724 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8004020 con 0x5629961f2900 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44b4004050 con 0x5629961f2900 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44a2a00640 1 -- 172.21.15.33:0/124228724 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9446+0+0 (secure 0 0 0) 0x7f44a80041c0 con 0x5629961f2900 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44a2a00640 1 -- 172.21.15.33:0/124228724 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f44a8024050 con 0x5629961f2900 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44c657f740 10 client.25834 fetch_fsmap learned FSMap version 102 2024-04-26T11:38:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.855+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x562995e69210 con 0x5629961f2900 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44a2a00640 1 -- 172.21.15.33:0/124228724 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b4004050 con 0x5629961f2900 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44a2a00640 1 -- 172.21.15.33:0/124228724 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 102) ==== 8338+0+0 (secure 0 0 0) 0x7f44a8003950 con 0x5629961f2900 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44c657f740 10 client.25834 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44c657f740 10 client.25834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44c657f740 20 client.25834 populate_metadata read hostname 'smithi033' 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44c657f740 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299620cfa0 0x56299620f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44c657f740 4 client.25834 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562996164f90 con 0x56299620cfa0 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.856+0000 7f44c1800640 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299620cfa0 0x56299620f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.857+0000 7f44c1800640 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299620cfa0 0x56299620f3b0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5206b7d126027026 server_cookie=ccd3055930bf7950 in_seq=0 out_seq=0 2024-04-26T11:38:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.857+0000 7f44a2a00640 10 client.25834 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44a2a00640 1 -- 172.21.15.33:0/124228724 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562996164f90 con 0x56299620cfa0 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44a2a00640 10 client.25834 handle_command_reply: tid=0 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 client.25834 shutdown 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 msgr2=0x5629961fa600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fa600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x5629961ef680 tx=0x7f44a8012d90 comp rx=0 tx=0).stop 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 shutdown_connections 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961fa600 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299620cfa0 0x56299620f3b0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 --2- 172.21.15.33:0/124228724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x5629961fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 >> 172.21.15.33:0/124228724 conn(0x5629961ee4f0 msgr2=0x5629961f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44a2a00640 0 client.25834 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 shutdown_connections 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.858+0000 7f44c657f740 1 -- 172.21.15.33:0/124228724 wait complete. 2024-04-26T11:38:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.859+0000 7f44c657f740 20 client.25834 trim_cache size 0 max 16384 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 Processor -- start 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 -- start start 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x562996208ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc1b37d0 con 0x5629961fd1c0 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc1b10a0 con 0x5629961f1fe0 2024-04-26T11:38:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.875+0000 7f44c657f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc1aef10 con 0x5629961f2900 2024-04-26T11:38:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961f9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:34266/0 (socket says 172.21.15.33:34266) 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 -- 172.21.15.33:0/3248471055 learned_addr learned my addr 172.21.15.33:0/3248471055 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44c1800640 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x562996208ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44c1800640 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 -- 172.21.15.33:0/3248471055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 msgr2=0x562996208ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x562996208ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 -- 172.21.15.33:0/3248471055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 msgr2=0x5629961fbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44bbe00640 1 -- 172.21.15.33:0/3248471055 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562995f35620 con 0x5629961f2900 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.876+0000 7f44c1800640 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.877+0000 7f44bbe00640 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961f9220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f44a8003dc0 tx=0x7f44a801c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c16ec81f9a714e82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.877+0000 7f44ad600640 1 -- 172.21.15.33:0/3248471055 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a800f970 con 0x5629961f2900 2024-04-26T11:38:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.877+0000 7f44ad600640 1 -- 172.21.15.33:0/3248471055 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44a8021030 con 0x5629961f2900 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.877+0000 7f44ad600640 1 -- 172.21.15.33:0/3248471055 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a80052d0 con 0x5629961f2900 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.877+0000 7f44c657f740 1 -- 172.21.15.33:0/3248471055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 msgr2=0x5629961f9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.877+0000 7f44c657f740 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961f9220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f44a8003dc0 tx=0x7f44a801c6a0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 -- 172.21.15.33:0/3248471055 shutdown_connections 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f2900 0x5629961f9220 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961fd1c0 0x5629961fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 --2- 172.21.15.33:0/3248471055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961f1fe0 0x562996208ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 -- 172.21.15.33:0/3248471055 >> 172.21.15.33:0/3248471055 conn(0x5629961ee4f0 msgr2=0x5629961fb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 -- 172.21.15.33:0/3248471055 shutdown_connections 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 -- 172.21.15.33:0/3248471055 wait complete. 2024-04-26T11:38:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 -- start start 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.878+0000 7f44c657f740 1 Processor -- start 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c657f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f1fe0 0x5629962051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c657f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961f2900 0x562996205730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c657f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961fd1c0 0x5629961f6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c657f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc1af560 con 0x5629961f2900 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c657f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc1b09d0 con 0x5629961fd1c0 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c657f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc1afc30 con 0x5629961f1fe0 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c657f740 10 client.0 _ll_register_callbacks cb 0x7f44a8009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:38:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961f2900 0x562996205730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961f2900 0x562996205730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50792/0 (socket says 172.21.15.33:50792) 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44bbe00640 1 -- 172.21.15.33:0/4053801741 learned_addr learned my addr 172.21.15.33:0/4053801741 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c1800640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f1fe0 0x5629962051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.879+0000 7f44c1800640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961fd1c0 0x5629961f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44bbe00640 1 -- 172.21.15.33:0/4053801741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f1fe0 msgr2=0x5629962051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44bbe00640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f1fe0 0x5629962051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44bbe00640 1 -- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961fd1c0 msgr2=0x5629961f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44bbe00640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961fd1c0 0x5629961f6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44bbe00640 1 -- 172.21.15.33:0/4053801741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562996111cb0 con 0x5629961f2900 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44c1800640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961fd1c0 0x5629961f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44bbe00640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961f2900 0x562996205730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44a8003ac0 tx=0x7f44a8005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e31091a31d64651 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44a2a00640 1 -- 172.21.15.33:0/4053801741 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a801c470 con 0x5629961f2900 2024-04-26T11:38:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44a2a00640 1 -- 172.21.15.33:0/4053801741 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9446+0+0 (secure 0 0 0) 0x7f44a8018040 con 0x5629961f2900 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.880+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44a8021030 con 0x5629961f2900 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44a2a00640 1 -- 172.21.15.33:0/4053801741 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f44a800fb80 con 0x5629961f2900 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 10 client.15675 fetch_fsmap learned FSMap version 102 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x56299627b730 con 0x5629961f2900 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44a2a00640 1 -- 172.21.15.33:0/4053801741 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44a8021030 con 0x5629961f2900 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44a2a00640 1 -- 172.21.15.33:0/4053801741 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 102) ==== 8338+0+0 (secure 0 0 0) 0x7f44a801c8a0 con 0x5629961f2900 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 10 client.15675 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 10 client.15675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 20 client.15675 populate_metadata read hostname 'smithi033' 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299627be10 0x56299627e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 4 client.15675 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.881+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56299615ee00 con 0x56299627be10 2024-04-26T11:38:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.882+0000 7f44c1800640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299627be10 0x56299627e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.882+0000 7f44c1800640 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299627be10 0x56299627e230 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca8333bf2d6b126d server_cookie=5770e2fb8ac9af7d in_seq=0 out_seq=0 2024-04-26T11:38:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:22.882+0000 7f44a2a00640 10 client.15675 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:22 smithi194 ceph-mon[33017]: pgmap v255: 97 pgs: 97 active+clean; 3.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-26T11:38:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:22 smithi033 ceph-mon[27845]: pgmap v255: 97 pgs: 97 active+clean; 3.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-26T11:38:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:22 smithi096 ceph-mon[32753]: pgmap v255: 97 pgs: 97 active+clean; 3.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-26T11:38:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:24 smithi194 ceph-mon[33017]: pgmap v256: 97 pgs: 97 active+clean; 3.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-26T11:38:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:24 smithi033 ceph-mon[27845]: pgmap v256: 97 pgs: 97 active+clean; 3.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-26T11:38:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:24 smithi096 ceph-mon[32753]: pgmap v256: 97 pgs: 97 active+clean; 3.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-26T11:38:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:26 smithi194 ceph-mon[33017]: pgmap v257: 97 pgs: 97 active+clean; 3.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 471 op/s 2024-04-26T11:38:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:26 smithi194 ceph-mon[33017]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T11:38:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:26 smithi033 ceph-mon[27845]: pgmap v257: 97 pgs: 97 active+clean; 3.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 471 op/s 2024-04-26T11:38:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:26 smithi033 ceph-mon[27845]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T11:38:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:26 smithi096 ceph-mon[32753]: pgmap v257: 97 pgs: 97 active+clean; 3.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 471 op/s 2024-04-26T11:38:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:26 smithi096 ceph-mon[32753]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T11:38:27.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 430.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:38:27.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:27 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 6. 2024-04-26T11:38:27.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:27 smithi096 systemd[1]: Stopped Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:38:27.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:27 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.175s CPU time. 2024-04-26T11:38:27.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:27 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Start request repeated too quickly. 2024-04-26T11:38:27.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:27 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-26T11:38:27.683 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 26 11:38:27 smithi096 systemd[1]: Failed to start Ceph mds.h for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:38:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:28 smithi194 ceph-mon[33017]: pgmap v259: 97 pgs: 97 active+clean; 3.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 471 op/s 2024-04-26T11:38:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:28 smithi033 ceph-mon[27845]: pgmap v259: 97 pgs: 97 active+clean; 3.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 471 op/s 2024-04-26T11:38:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:28 smithi096 ceph-mon[32753]: pgmap v259: 97 pgs: 97 active+clean; 3.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 471 op/s 2024-04-26T11:38:31.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:30 smithi194 ceph-mon[33017]: pgmap v260: 97 pgs: 97 active+clean; 3.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.8 MiB/s wr, 715 op/s 2024-04-26T11:38:31.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:30 smithi033 ceph-mon[27845]: pgmap v260: 97 pgs: 97 active+clean; 3.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.8 MiB/s wr, 715 op/s 2024-04-26T11:38:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:30 smithi096 ceph-mon[32753]: pgmap v260: 97 pgs: 97 active+clean; 3.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.8 MiB/s wr, 715 op/s 2024-04-26T11:38:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:31 smithi194 ceph-mon[33017]: Replacing daemon mds.h as rank 1 with standby daemon mds.e 2024-04-26T11:38:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:31 smithi194 ceph-mon[33017]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-26T11:38:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:31 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:31 smithi194 ceph-mon[33017]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T11:38:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:31 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:31 smithi033 ceph-mon[27845]: Replacing daemon mds.h as rank 1 with standby daemon mds.e 2024-04-26T11:38:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:31 smithi033 ceph-mon[27845]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-26T11:38:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:31 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:31 smithi033 ceph-mon[27845]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T11:38:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:31 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:31 smithi096 ceph-mon[32753]: Replacing daemon mds.h as rank 1 with standby daemon mds.e 2024-04-26T11:38:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:31 smithi096 ceph-mon[32753]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-26T11:38:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:31 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:31 smithi096 ceph-mon[32753]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T11:38:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:31 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 435.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:38:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:32 smithi194 ceph-mon[33017]: pgmap v262: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-04-26T11:38:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:32 smithi033 ceph-mon[27845]: pgmap v262: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-04-26T11:38:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:32 smithi096 ceph-mon[32753]: pgmap v262: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-04-26T11:38:34.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:33 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:resolve 2024-04-26T11:38:34.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:33 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:33 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:resolve 2024-04-26T11:38:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:33 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:34.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:33 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:resolve 2024-04-26T11:38:34.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:33 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.840+0000 7f44a2a00640 1 -- 172.21.15.33:0/4053801741 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56299615ee00 con 0x56299627be10 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.840+0000 7f44a2a00640 10 client.15675 handle_command_reply: tid=0 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.840+0000 7f44c657f740 1 client.15675 shutdown 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961f2900 msgr2=0x562996205730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961f2900 0x562996205730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44a8003ac0 tx=0x7f44a8005f10 comp rx=0 tx=0).stop 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 shutdown_connections 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56299627be10 0x56299627e230 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5629961fd1c0 0x5629961f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5629961f2900 0x562996205730 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 --2- 172.21.15.33:0/4053801741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629961f1fe0 0x5629962051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44a2a00640 0 client.15675 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 >> 172.21.15.33:0/4053801741 conn(0x5629961ee4f0 msgr2=0x5629961f3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 shutdown_connections 2024-04-26T11:38:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 1 -- 172.21.15.33:0/4053801741 wait complete. 2024-04-26T11:38:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.841+0000 7f44c657f740 20 client.15675 trim_cache size 0 max 16384 2024-04-26T11:38:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:38:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:38:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:38:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:38:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f44a40a3d10 msgr2=0x7f44a40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f44a40a3d10 0x7f44a40a61d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x5629961f3580 tx=0x7f44b0031400 comp rx=0 tx=0).stop 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44bc1014f0 msgr2=0x7f44bc19b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44bc1014f0 0x7f44bc19b210 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f44b002aa00 tx=0x7f44b0003b90 comp rx=0 tx=0).stop 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 shutdown_connections 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f44a40a3d10 0x7f44a40a61d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc10f0a0 0x7f44bc19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44bc101ea0 0x7f44bc19b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 --2- 172.21.15.33:0/432838530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44bc1014f0 0x7f44bc19b210 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.842+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 >> 172.21.15.33:0/432838530 conn(0x7f44bc0fd170 msgr2=0x7f44bc0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.843+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 shutdown_connections 2024-04-26T11:38:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:34.843+0000 7f44c2200640 1 -- 172.21.15.33:0/432838530 wait complete. 2024-04-26T11:38:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:34 smithi194 ceph-mon[33017]: pgmap v263: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9.5 MiB/s wr, 1.01k op/s 2024-04-26T11:38:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:34 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:38:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:34 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:reconnect 2024-04-26T11:38:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:34 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:34 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:38:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:34 smithi033 ceph-mon[27845]: pgmap v263: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9.5 MiB/s wr, 1.01k op/s 2024-04-26T11:38:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:34 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:38:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:34 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:reconnect 2024-04-26T11:38:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:34 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:34 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:38:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:34 smithi096 ceph-mon[32753]: pgmap v263: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9.5 MiB/s wr, 1.01k op/s 2024-04-26T11:38:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:34 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:38:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:34 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:reconnect 2024-04-26T11:38:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:34 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:34 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:38:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:35 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:rejoin 2024-04-26T11:38:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:35 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:35 smithi194 ceph-mon[33017]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:38:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:35 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:rejoin 2024-04-26T11:38:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:35 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:35 smithi033 ceph-mon[27845]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:38:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:35 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:rejoin 2024-04-26T11:38:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:35 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:35 smithi096 ceph-mon[32753]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-26T11:38:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:36 smithi194 ceph-mon[33017]: pgmap v264: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8.8 MiB/s wr, 948 op/s 2024-04-26T11:38:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:36 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:38:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:36 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:38:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:36 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:active 2024-04-26T11:38:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:36 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:36 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-04-26T11:38:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:36 smithi033 ceph-mon[27845]: pgmap v264: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8.8 MiB/s wr, 948 op/s 2024-04-26T11:38:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:36 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:38:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:36 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:38:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:36 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:active 2024-04-26T11:38:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:36 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:36 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-04-26T11:38:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:36 smithi096 ceph-mon[32753]: pgmap v264: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8.8 MiB/s wr, 948 op/s 2024-04-26T11:38:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:36 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:38:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:36 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:38:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:36 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/1417181376,v1:172.21.15.96:6835/1417181376] up:active 2024-04-26T11:38:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:36 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 2 up:standby-replay 6 up:standby 2024-04-26T11:38:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:36 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-04-26T11:38:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 440.0 s, 760.7 tps, lat 487.849 ms stddev 2771.380 2024-04-26T11:38:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:37 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-04-26T11:38:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:37 smithi033 ceph-mon[27845]: pgmap v265: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.0 MiB/s wr, 863 op/s 2024-04-26T11:38:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:37 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-04-26T11:38:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:37 smithi096 ceph-mon[32753]: pgmap v265: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.0 MiB/s wr, 863 op/s 2024-04-26T11:38:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:37 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2024-04-26T11:38:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:37 smithi194 ceph-mon[33017]: pgmap v265: 97 pgs: 97 active+clean; 3.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.0 MiB/s wr, 863 op/s 2024-04-26T11:38:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:40 smithi194 ceph-mon[33017]: pgmap v266: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 5.8 MiB/s wr, 550 op/s 2024-04-26T11:38:40.861 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-mon[32753]: pgmap v266: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 5.8 MiB/s wr, 550 op/s 2024-04-26T11:38:40.904 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:40 smithi033 ceph-mon[27845]: pgmap v266: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 5.8 MiB/s wr, 550 op/s 2024-04-26T11:38:41.184 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f37545ee700 time 2024-04-26T11:38:40.861747+0000 2024-04-26T11:38:41.184 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:41.184 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.184 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f375ec173dd] 2024-04-26T11:38:41.185 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.185 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.185 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.185 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.185 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.186 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.186 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.186 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.186 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.186 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.186 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.187 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: clone() 2024-04-26T11:38:41.187 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:40.862+0000 7f37545ee700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f37545ee700 time 2024-04-26T11:38:40.861747+0000 2024-04-26T11:38:41.187 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:41.187 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.189 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.190 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f375ec173dd] 2024-04-26T11:38:41.190 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.190 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.190 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.190 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.191 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.191 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.191 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.191 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.191 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.191 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.192 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.192 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: clone() 2024-04-26T11:38:41.192 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.192 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: *** Caught signal (Aborted) ** 2024-04-26T11:38:41.192 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: in thread 7f37545ee700 thread_name:ms_dispatch 2024-04-26T11:38:41.192 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f375d99fd20] 2024-04-26T11:38:41.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: gsignal() 2024-04-26T11:38:41.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: abort() 2024-04-26T11:38:41.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f375ec17437] 2024-04-26T11:38:41.193 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.194 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.194 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.194 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.194 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.194 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.194 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.195 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.195 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.195 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.195 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.195 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 16: clone() 2024-04-26T11:38:41.195 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:40.863+0000 7f37545ee700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:38:41.196 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: in thread 7f37545ee700 thread_name:ms_dispatch 2024-04-26T11:38:41.196 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.196 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.196 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f375d99fd20] 2024-04-26T11:38:41.196 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: gsignal() 2024-04-26T11:38:41.197 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: abort() 2024-04-26T11:38:41.197 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f375ec17437] 2024-04-26T11:38:41.197 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.197 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.197 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.197 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.197 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.198 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.198 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.198 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.198 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.198 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.199 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.199 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 16: clone() 2024-04-26T11:38:41.199 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:38:41.199 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.199 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: -1> 2024-04-26T11:38:40.862+0000 7f37545ee700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f37545ee700 time 2024-04-26T11:38:40.861747+0000 2024-04-26T11:38:41.199 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:41.200 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.200 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.200 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f375ec173dd] 2024-04-26T11:38:41.200 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.200 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.200 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.201 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.201 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.201 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.201 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.201 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.201 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.202 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.202 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.202 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: clone() 2024-04-26T11:38:41.202 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.202 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 0> 2024-04-26T11:38:40.863+0000 7f37545ee700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:38:41.202 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: in thread 7f37545ee700 thread_name:ms_dispatch 2024-04-26T11:38:41.203 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.203 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.203 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f375d99fd20] 2024-04-26T11:38:41.203 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: gsignal() 2024-04-26T11:38:41.203 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: abort() 2024-04-26T11:38:41.203 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f375ec17437] 2024-04-26T11:38:41.204 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.204 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.204 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.204 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.204 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.204 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.205 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 16: clone() 2024-04-26T11:38:41.206 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:38:41.206 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.206 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: -1> 2024-04-26T11:38:40.862+0000 7f37545ee700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f37545ee700 time 2024-04-26T11:38:40.861747+0000 2024-04-26T11:38:41.206 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:38:41.206 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.206 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.207 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f375ec173dd] 2024-04-26T11:38:41.207 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.207 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.207 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.207 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.207 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.208 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.208 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.208 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.208 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.208 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.208 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.209 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: clone() 2024-04-26T11:38:41.209 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.209 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 0> 2024-04-26T11:38:40.863+0000 7f37545ee700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:38:41.209 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: in thread 7f37545ee700 thread_name:ms_dispatch 2024-04-26T11:38:41.209 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.209 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:38:41.210 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f375d99fd20] 2024-04-26T11:38:41.210 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2: gsignal() 2024-04-26T11:38:41.210 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 3: abort() 2024-04-26T11:38:41.210 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f375ec17437] 2024-04-26T11:38:41.210 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f375ec175a3] 2024-04-26T11:38:41.211 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560ff3258822] 2024-04-26T11:38:41.211 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560ff31c8598] 2024-04-26T11:38:41.211 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ff31cac17] 2024-04-26T11:38:41.211 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ff30c3d0b] 2024-04-26T11:38:41.211 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ff30c42ac] 2024-04-26T11:38:41.212 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560ff30acfad] 2024-04-26T11:38:41.212 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f375eeabe20] 2024-04-26T11:38:41.212 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 13: (DispatchQueue::entry()+0x51f) [0x7f375eea8fdf] 2024-04-26T11:38:41.212 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f375ef7bfc1] 2024-04-26T11:38:41.212 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f375d9951ca] 2024-04-26T11:38:41.212 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 16: clone() 2024-04-26T11:38:41.213 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:38:41.213 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e[68581]: 2024-04-26T11:38:41.213 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:40 smithi096 podman[68678]: 2024-04-26 11:38:40.922011269 +0000 UTC m=+0.020161722 container died 328cb3e3d82528d5262fe8b7a93fc39132e628bba22be2860d097496bc5d4cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T11:38:41.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:41 smithi096 podman[68678]: 2024-04-26 11:38:41.689918375 +0000 UTC m=+0.788068824 container remove 328cb3e3d82528d5262fe8b7a93fc39132e628bba22be2860d097496bc5d4cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:38:41.948 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:41 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:38:42.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 445.0 s, 2761.4 tps, lat 36.518 ms stddev 48.792 2024-04-26T11:38:42.432 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:41 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-26T11:38:42.433 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:41 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.124s CPU time. 2024-04-26T11:38:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:42 smithi194 ceph-mon[33017]: pgmap v267: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.7 MiB/s wr, 311 op/s 2024-04-26T11:38:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:42 smithi033 ceph-mon[27845]: pgmap v267: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.7 MiB/s wr, 311 op/s 2024-04-26T11:38:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:42 smithi096 ceph-mon[32753]: pgmap v267: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.7 MiB/s wr, 311 op/s 2024-04-26T11:38:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:44 smithi194 ceph-mon[33017]: pgmap v268: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 3.1 MiB/s wr, 261 op/s 2024-04-26T11:38:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:44 smithi033 ceph-mon[27845]: pgmap v268: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 3.1 MiB/s wr, 261 op/s 2024-04-26T11:38:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:44 smithi096 ceph-mon[32753]: pgmap v268: 97 pgs: 97 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 3.1 MiB/s wr, 261 op/s 2024-04-26T11:38:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:38:45.877 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:38:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:38:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:46 smithi033 ceph-mon[27845]: pgmap v269: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 6.9 MiB/s wr, 558 op/s 2024-04-26T11:38:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:46 smithi096 ceph-mon[32753]: pgmap v269: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 6.9 MiB/s wr, 558 op/s 2024-04-26T11:38:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:46 smithi194 ceph-mon[33017]: pgmap v269: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 6.9 MiB/s wr, 558 op/s 2024-04-26T11:38:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:47.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 450.0 s, 1070.2 tps, lat 36.483 ms stddev 57.795 2024-04-26T11:38:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:38:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:38:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:38:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:38:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 -- 172.21.15.33:0/3475033680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0010ac00 msgr2=0x7f7d0010afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 --2- 172.21.15.33:0/3475033680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0010ac00 0x7f7d0010afe0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009ca0 tx=0x7f7cf002a8c0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 -- 172.21.15.33:0/3475033680 shutdown_connections 2024-04-26T11:38:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 --2- 172.21.15.33:0/3475033680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d00105500 0x7f7d001119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 --2- 172.21.15.33:0/3475033680 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 0x7f7d00104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 --2- 172.21.15.33:0/3475033680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d0010ac00 0x7f7d0010afe0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 -- 172.21.15.33:0/3475033680 >> 172.21.15.33:0/3475033680 conn(0x7f7d00100630 msgr2=0x7f7d00102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 -- 172.21.15.33:0/3475033680 shutdown_connections 2024-04-26T11:38:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.005+0000 7f7d06a00640 1 -- 172.21.15.33:0/3475033680 wait complete. 2024-04-26T11:38:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.006+0000 7f7d06a00640 1 Processor -- start 2024-04-26T11:38:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.006+0000 7f7d06a00640 1 -- start start 2024-04-26T11:38:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7d06a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 0x7f7d0019b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7d06a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d00105500 0x7f7d0019b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7d06a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d0010ac00 0x7f7d0019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7d06a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7d00114df0 con 0x7f7d00105500 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7d06a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d00114c70 con 0x7f7d00104b40 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7d06a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d00114f70 con 0x7f7d0010ac00 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 0x7f7d0019b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 0x7f7d0019b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34478/0 (socket says 172.21.15.33:34478) 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7cffe00640 1 -- 172.21.15.33:0/1940855669 learned_addr learned my addr 172.21.15.33:0/1940855669 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7cff400640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d00105500 0x7f7d0019b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.007+0000 7f7d04a00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d0010ac00 0x7f7d0019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cffe00640 1 -- 172.21.15.33:0/1940855669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d0010ac00 msgr2=0x7f7d0019fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cffe00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d0010ac00 0x7f7d0019fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cffe00640 1 -- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d00105500 msgr2=0x7f7d0019b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cffe00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d00105500 0x7f7d0019b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cffe00640 1 -- 172.21.15.33:0/1940855669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d001a01b0 con 0x7f7d00104b40 2024-04-26T11:38:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cffe00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 0x7f7d0019b210 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0006010 tx=0x7f7cf00316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cfcc00640 1 -- 172.21.15.33:0/1940855669 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0003b70 con 0x7f7d00104b40 2024-04-26T11:38:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cfcc00640 1 -- 172.21.15.33:0/1940855669 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf0003d10 con 0x7f7d00104b40 2024-04-26T11:38:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d001a0440 con 0x7f7d00104b40 2024-04-26T11:38:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.008+0000 7f7cfcc00640 1 -- 172.21.15.33:0/1940855669 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf002fa00 con 0x7f7d00104b40 2024-04-26T11:38:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.009+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d001a7d30 con 0x7f7d00104b40 2024-04-26T11:38:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7cfcc00640 1 -- 172.21.15.33:0/1940855669 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7cf002fba0 con 0x7f7d00104b40 2024-04-26T11:38:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 Processor -- start 2024-04-26T11:38:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 -- start start 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c28dd0 0x559da9c291b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c33a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c33fb0 0x559da9c363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559da9b8e380 con 0x559da9c296f0 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559da9b2ac30 con 0x559da9c33fb0 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.012+0000 7f7d09874740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559da9b8a340 con 0x559da9c28dd0 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c33a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c28dd0 0x559da9c291b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c28dd0 0x559da9c291b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:40272/0 (socket says 172.21.15.33:40272) 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 -- 172.21.15.33:0/928751027 learned_addr learned my addr 172.21.15.33:0/928751027 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7cff400640 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c33fb0 0x559da9c363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 -- 172.21.15.33:0/928751027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c33fb0 msgr2=0x559da9c363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c33fb0 0x559da9c363a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 -- 172.21.15.33:0/928751027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 msgr2=0x559da9c33a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c33a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 -- 172.21.15.33:0/928751027 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da9b4c940 con 0x559da9c28dd0 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7d04a00640 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c28dd0 0x559da9c291b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009ca0 tx=0x7f7cf8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd17c128d0b8a001 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.013+0000 7f7cfcc00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7ce40a3dc0 0x7f7ce40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.014+0000 7f7cfcc00640 1 -- 172.21.15.33:0/1940855669 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9444+0+0 (secure 0 0 0) 0x7f7cf00f20e0 con 0x7f7d00104b40 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.014+0000 7f7cf5200640 1 -- 172.21.15.33:0/928751027 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf801e070 con 0x559da9c28dd0 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.014+0000 7f7cffe00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7ce40a3dc0 0x7f7ce40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.014+0000 7f7cf5200640 1 -- 172.21.15.33:0/928751027 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf8009e50 con 0x559da9c28dd0 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.014+0000 7f7cf5200640 1 -- 172.21.15.33:0/928751027 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800ea10 con 0x559da9c28dd0 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 -- 172.21.15.33:0/928751027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c28dd0 msgr2=0x559da9c291b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7cffe00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7ce40a3dc0 0x7f7ce40a6280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x559da9c2a370 tx=0x7f7cf00f5b90 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c28dd0 0x559da9c291b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009ca0 tx=0x7f7cf8011810 comp rx=0 tx=0).stop 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 -- 172.21.15.33:0/928751027 shutdown_connections 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c33fb0 0x559da9c363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c33a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 --2- 172.21.15.33:0/928751027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c28dd0 0x559da9c291b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 -- 172.21.15.33:0/928751027 >> 172.21.15.33:0/928751027 conn(0x559da9c252e0 msgr2=0x559da9c256c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 -- 172.21.15.33:0/928751027 shutdown_connections 2024-04-26T11:38:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 -- 172.21.15.33:0/928751027 wait complete. 2024-04-26T11:38:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.015+0000 7f7d09874740 1 -- start start 2024-04-26T11:38:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 1 Processor -- start 2024-04-26T11:38:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c30f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c296f0 0x559da9c31470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c33fb0 0x559da9c319b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559da9c36be0 con 0x559da9c33fb0 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559da9b15c60 con 0x559da9c28dd0 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559da9c36d60 con 0x559da9c296f0 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d09874740 10 client.0 _ll_register_callbacks cb 0x559da9b41e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7cff400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c296f0 0x559da9c31470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7cff400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c296f0 0x559da9c31470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:40288/0 (socket says 172.21.15.33:40288) 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7cff400640 1 -- 172.21.15.33:0/466597512 learned_addr learned my addr 172.21.15.33:0/466597512 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d04a00640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c30f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.016+0000 7f7d04a00640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c33fb0 0x559da9c319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7cff400640 1 -- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 msgr2=0x559da9c30f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7cff400640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c30f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7cff400640 1 -- 172.21.15.33:0/466597512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c33fb0 msgr2=0x559da9c319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7cff400640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c33fb0 0x559da9c319b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7cff400640 1 -- 172.21.15.33:0/466597512 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da9b24510 con 0x559da9c296f0 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7d04a00640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c30f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7cff400640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c296f0 0x559da9c31470 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8012700 tx=0x7f7ce8012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=230443054d8795a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7ced600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7ced600640 1 -- 172.21.15.33:0/466597512 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800f920 con 0x559da9c296f0 2024-04-26T11:38:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cf8009e50 con 0x559da9c296f0 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.017+0000 7f7ced600640 1 -- 172.21.15.33:0/466597512 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9444+0+0 (secure 0 0 0) 0x7f7ce800fac0 con 0x559da9c296f0 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7ced600640 1 -- 172.21.15.33:0/466597512 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f7ce800fcc0 con 0x559da9c296f0 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 10 client.25852 fetch_fsmap learned FSMap version 109 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x559da98a0210 con 0x559da9c296f0 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7ced600640 1 -- 172.21.15.33:0/466597512 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf8009e50 con 0x559da9c296f0 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7ced600640 1 -- 172.21.15.33:0/466597512 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 109) ==== 7737+0+0 (secure 0 0 0) 0x7f7ce801c420 con 0x559da9c296f0 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 10 client.25852 fetch_fsmap finished waiting for FSMap version 109 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 10 client.25852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 20 client.25852 populate_metadata read hostname 'smithi033' 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9c43e20 0x559da9c46210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 4 client.25852 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.018+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559da9b9bf90 con 0x559da9c43e20 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.019+0000 7f7d04a00640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9c43e20 0x559da9c46210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.019+0000 7f7d04a00640 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9c43e20 0x559da9c46210 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f92355ed197f9d server_cookie=c7b5e983d116e64 in_seq=0 out_seq=0 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.019+0000 7f7ced600640 10 client.25852 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7ced600640 1 -- 172.21.15.33:0/466597512 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559da9b9bf90 con 0x559da9c43e20 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7ced600640 10 client.25852 handle_command_reply: tid=0 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 client.25852 shutdown 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c296f0 msgr2=0x559da9c31470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c296f0 0x559da9c31470 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8012700 tx=0x7f7ce8012bc0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 shutdown_connections 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c296f0 0x559da9c31470 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9c43e20 0x559da9c46210 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c33fb0 0x559da9c319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 --2- 172.21.15.33:0/466597512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c30f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7ced600640 0 client.25852 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.020+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 >> 172.21.15.33:0/466597512 conn(0x559da9c252e0 msgr2=0x559da9c2b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.021+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 shutdown_connections 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.021+0000 7f7d09874740 1 -- 172.21.15.33:0/466597512 wait complete. 2024-04-26T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.021+0000 7f7d09874740 20 client.25852 trim_cache size 0 max 16384 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 Processor -- start 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 -- start start 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c3faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c32d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c4bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001b2960 con 0x559da9c296f0 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001b3030 con 0x559da9c28dd0 2024-04-26T11:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.037+0000 7f7d09874740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001b0940 con 0x559da9c33fb0 2024-04-26T11:38:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7cff400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c32d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c3faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c3faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34504/0 (socket says 172.21.15.33:34504) 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 -- 172.21.15.33:0/168582490 learned_addr learned my addr 172.21.15.33:0/168582490 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c4bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 -- 172.21.15.33:0/168582490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 msgr2=0x559da9c4bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c4bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 -- 172.21.15.33:0/168582490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 msgr2=0x559da9c32d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c32d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 -- 172.21.15.33:0/168582490 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da9c29520 con 0x559da9c28dd0 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.038+0000 7f7d04a00640 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c3faa0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800e6c0 tx=0x7f7cf8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a82d77ab616413c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7cefe00640 1 -- 172.21.15.33:0/168582490 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf801e040 con 0x559da9c28dd0 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7cefe00640 1 -- 172.21.15.33:0/168582490 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf80054f0 con 0x559da9c28dd0 2024-04-26T11:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7cefe00640 1 -- 172.21.15.33:0/168582490 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8003b20 con 0x559da9c28dd0 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 -- 172.21.15.33:0/168582490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 msgr2=0x559da9c3faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c3faa0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800e6c0 tx=0x7f7cf8005f90 comp rx=0 tx=0).stop 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 -- 172.21.15.33:0/168582490 shutdown_connections 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c4bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c296f0 0x559da9c32d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 --2- 172.21.15.33:0/168582490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c28dd0 0x559da9c3faa0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 -- 172.21.15.33:0/168582490 >> 172.21.15.33:0/168582490 conn(0x559da9c252e0 msgr2=0x559da9c32650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.039+0000 7f7d09874740 1 -- 172.21.15.33:0/168582490 shutdown_connections 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.040+0000 7f7d09874740 1 -- 172.21.15.33:0/168582490 wait complete. 2024-04-26T11:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.040+0000 7f7d09874740 1 -- start start 2024-04-26T11:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.040+0000 7f7d09874740 1 Processor -- start 2024-04-26T11:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.040+0000 7f7d09874740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c28dd0 0x559da9c2da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d09874740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c296f0 0x559da9c2dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d09874740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c2e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d09874740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001b0270 con 0x559da9c28dd0 2024-04-26T11:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d09874740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001aee60 con 0x559da9c296f0 2024-04-26T11:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d09874740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001af4d0 con 0x559da9c33fb0 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d09874740 10 client.0 _ll_register_callbacks cb 0x7f7ce8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d04a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c28dd0 0x559da9c2da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c296f0 0x559da9c2dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c296f0 0x559da9c2dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34520/0 (socket says 172.21.15.33:34520) 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 -- 172.21.15.33:0/875808620 learned_addr learned my addr 172.21.15.33:0/875808620 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 -- 172.21.15.33:0/875808620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 msgr2=0x559da9c2e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d04a00640 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c2e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c2e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 -- 172.21.15.33:0/875808620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c28dd0 msgr2=0x559da9c2da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c28dd0 0x559da9c2da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7cff400640 1 -- 172.21.15.33:0/875808620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da996c620 con 0x559da9c296f0 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.041+0000 7f7d04a00640 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c2e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7cff400640 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c296f0 0x559da9c2dfd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x559da9c30790 tx=0x7f7ce800ffc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f36513a7944a30b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7ced600640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7ced600640 1 -- 172.21.15.33:0/875808620 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce802f060 con 0x559da9c296f0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7ced600640 1 -- 172.21.15.33:0/875808620 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9444+0+0 (secure 0 0 0) 0x7f7ce801ca80 con 0x559da9c296f0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cf80054f0 con 0x559da9c296f0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7ced600640 1 -- 172.21.15.33:0/875808620 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f7ce801cc80 con 0x559da9c296f0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7d09874740 10 client.25934 fetch_fsmap learned FSMap version 109 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x559da9b48cb0 con 0x559da9c296f0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.042+0000 7f7ced600640 1 -- 172.21.15.33:0/875808620 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf80054f0 con 0x559da9c296f0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7ced600640 1 -- 172.21.15.33:0/875808620 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 109) ==== 7737+0+0 (secure 0 0 0) 0x7f7ce8003710 con 0x559da9c296f0 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7d09874740 10 client.25934 fetch_fsmap finished waiting for FSMap version 109 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7d09874740 10 client.25934 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7d09874740 20 client.25934 populate_metadata read hostname 'smithi033' 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7d09874740 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9caa930 0x559da9cacd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7d09874740 4 client.25934 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559da9b95e00 con 0x559da9caa930 2024-04-26T11:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.043+0000 7f7d04a00640 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9caa930 0x559da9cacd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.044+0000 7f7d04a00640 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9caa930 0x559da9cacd50 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad7ae67f897c9e7e server_cookie=5093957050231d45 in_seq=0 out_seq=0 2024-04-26T11:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:48.044+0000 7f7ced600640 10 client.25934 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:38:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:38:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:38:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:48 smithi033 ceph-mon[27845]: pgmap v270: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 6.6 MiB/s wr, 512 op/s 2024-04-26T11:38:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:48 smithi096 ceph-mon[32753]: pgmap v270: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 6.6 MiB/s wr, 512 op/s 2024-04-26T11:38:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:48 smithi194 ceph-mon[33017]: pgmap v270: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 6.6 MiB/s wr, 512 op/s 2024-04-26T11:38:50.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:50 smithi033 ceph-mon[27845]: pgmap v271: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 8.1 MiB/s wr, 620 op/s 2024-04-26T11:38:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:50 smithi096 ceph-mon[32753]: pgmap v271: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 8.1 MiB/s wr, 620 op/s 2024-04-26T11:38:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:50 smithi194 ceph-mon[33017]: pgmap v271: 97 pgs: 97 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 8.1 MiB/s wr, 620 op/s 2024-04-26T11:38:52.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 455.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:38:52.640 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:52 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-04-26T11:38:52.640 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:52 smithi096 systemd[1]: Stopped Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:38:52.641 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:52 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.124s CPU time. 2024-04-26T11:38:52.641 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:52 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Start request repeated too quickly. 2024-04-26T11:38:52.641 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:52 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-26T11:38:52.641 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 26 11:38:52 smithi096 systemd[1]: Failed to start Ceph mds.e for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:38:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:52 smithi033 ceph-mon[27845]: pgmap v272: 97 pgs: 97 active+clean; 3.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.6 MiB/s wr, 529 op/s 2024-04-26T11:38:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:52 smithi096 ceph-mon[32753]: pgmap v272: 97 pgs: 97 active+clean; 3.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.6 MiB/s wr, 529 op/s 2024-04-26T11:38:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:52 smithi194 ceph-mon[33017]: pgmap v272: 97 pgs: 97 active+clean; 3.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.6 MiB/s wr, 529 op/s 2024-04-26T11:38:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:54 smithi033 ceph-mon[27845]: pgmap v273: 97 pgs: 97 active+clean; 3.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 455 op/s 2024-04-26T11:38:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:54 smithi096 ceph-mon[32753]: pgmap v273: 97 pgs: 97 active+clean; 3.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 455 op/s 2024-04-26T11:38:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:54 smithi194 ceph-mon[33017]: pgmap v273: 97 pgs: 97 active+clean; 3.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 455 op/s 2024-04-26T11:38:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:56 smithi033 ceph-mon[27845]: pgmap v274: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 541 op/s 2024-04-26T11:38:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:56 smithi033 ceph-mon[27845]: Replacing daemon mds.e as rank 1 with standby daemon mds.d 2024-04-26T11:38:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:56 smithi033 ceph-mon[27845]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-26T11:38:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:56 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:56 smithi033 ceph-mon[27845]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T11:38:56.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:56 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:38:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:56 smithi096 ceph-mon[32753]: pgmap v274: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 541 op/s 2024-04-26T11:38:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:56 smithi096 ceph-mon[32753]: Replacing daemon mds.e as rank 1 with standby daemon mds.d 2024-04-26T11:38:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:56 smithi096 ceph-mon[32753]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-26T11:38:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:56 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:56 smithi096 ceph-mon[32753]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T11:38:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:56 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:38:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:56 smithi194 ceph-mon[33017]: pgmap v274: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 541 op/s 2024-04-26T11:38:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:56 smithi194 ceph-mon[33017]: Replacing daemon mds.e as rank 1 with standby daemon mds.d 2024-04-26T11:38:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:56 smithi194 ceph-mon[33017]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-26T11:38:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:56 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:38:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:56 smithi194 ceph-mon[33017]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T11:38:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:56 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:38:57.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 460.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:38:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:58 smithi033 ceph-mon[27845]: pgmap v276: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-04-26T11:38:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:resolve 2024-04-26T11:38:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:58 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:38:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:58 smithi096 ceph-mon[32753]: pgmap v276: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-04-26T11:38:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:resolve 2024-04-26T11:38:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:58 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:38:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:58 smithi194 ceph-mon[33017]: pgmap v276: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-04-26T11:38:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:resolve 2024-04-26T11:38:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:58 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:38:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.689+0000 7f7ced600640 1 -- 172.21.15.33:0/875808620 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559da9b95e00 con 0x559da9caa930 2024-04-26T11:38:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.689+0000 7f7ced600640 10 client.25934 handle_command_reply: tid=0 2024-04-26T11:38:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.689+0000 7f7d09874740 1 client.25934 shutdown 2024-04-26T11:38:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c296f0 msgr2=0x559da9c2dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c296f0 0x559da9c2dfd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x559da9c30790 tx=0x7f7ce800ffc0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 shutdown_connections 2024-04-26T11:38:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559da9c33fb0 0x559da9c2e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x559da9caa930 0x559da9cacd50 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559da9c296f0 0x559da9c2dfd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 --2- 172.21.15.33:0/875808620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559da9c28dd0 0x559da9c2da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 >> 172.21.15.33:0/875808620 conn(0x559da9c252e0 msgr2=0x559da9c31780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7ced600640 0 client.25934 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 shutdown_connections 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.690+0000 7f7d09874740 1 -- 172.21.15.33:0/875808620 wait complete. 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.691+0000 7f7d09874740 20 client.25934 trim_cache size 0 max 16384 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:38:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:38:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:38:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7ce40a3dc0 msgr2=0x7f7ce40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7ce40a3dc0 0x7f7ce40a6280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x559da9c2a370 tx=0x7f7cf00f5b90 comp rx=0 tx=0).stop 2024-04-26T11:38:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 msgr2=0x7f7d0019b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:38:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 0x7f7d0019b210 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0006010 tx=0x7f7cf00316f0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 shutdown_connections 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7ce40a3dc0 0x7f7ce40a6280 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d0010ac00 0x7f7d0019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d00105500 0x7f7d0019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 --2- 172.21.15.33:0/1940855669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d00104b40 0x7f7d0019b210 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 >> 172.21.15.33:0/1940855669 conn(0x7f7d00100630 msgr2=0x7f7d00101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 shutdown_connections 2024-04-26T11:38:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:38:59.692+0000 7f7d06a00640 1 -- 172.21.15.33:0/1940855669 wait complete. 2024-04-26T11:38:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:59 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:38:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:reconnect 2024-04-26T11:38:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:59 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:38:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:38:59 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119998 2024-04-26T11:39:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:59 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:39:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:reconnect 2024-04-26T11:39:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:59 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:38:59 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119998 2024-04-26T11:39:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:59 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:39:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:reconnect 2024-04-26T11:39:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:59 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:38:59 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119998 2024-04-26T11:39:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:00 smithi194 ceph-mon[33017]: pgmap v277: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.6 MiB/s wr, 199 op/s 2024-04-26T11:39:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:00 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:rejoin 2024-04-26T11:39:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:00 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:00 smithi194 ceph-mon[33017]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:39:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:00 smithi033 ceph-mon[27845]: pgmap v277: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.6 MiB/s wr, 199 op/s 2024-04-26T11:39:01.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:00 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:rejoin 2024-04-26T11:39:01.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:00 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:01.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:00 smithi033 ceph-mon[27845]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:39:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:00 smithi096 ceph-mon[32753]: pgmap v277: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.6 MiB/s wr, 199 op/s 2024-04-26T11:39:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:00 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:rejoin 2024-04-26T11:39:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:00 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:00 smithi096 ceph-mon[32753]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:39:02.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:01 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:02.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:01 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:39:02.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:01 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:active 2024-04-26T11:39:02.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:01 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:02.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:01 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:01 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:01 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:39:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:01 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:active 2024-04-26T11:39:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:01 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:01 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:02.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 465.0 s, 553.4 tps, lat 652.654 ms stddev 3106.479 2024-04-26T11:39:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:01 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:01 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:39:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:01 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3073208972,v1:172.21.15.33:6837/3073208972] up:active 2024-04-26T11:39:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:01 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:01 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:02 smithi194 ceph-mon[33017]: pgmap v278: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-04-26T11:39:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:02 smithi033 ceph-mon[27845]: pgmap v278: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-04-26T11:39:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:02 smithi096 ceph-mon[32753]: pgmap v278: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-04-26T11:39:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:04 smithi194 ceph-mon[33017]: pgmap v279: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-04-26T11:39:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:04 smithi033 ceph-mon[27845]: pgmap v279: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-04-26T11:39:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:04 smithi096 ceph-mon[32753]: pgmap v279: 97 pgs: 97 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-04-26T11:39:06.170 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1734200700 time 2024-04-26T11:39:05.733386+0000 2024-04-26T11:39:06.170 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:06.170 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.170 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173fb1c3dd] 2024-04-26T11:39:06.170 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.171 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: clone() 2024-04-26T11:39:06.172 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:05.733+0000 7f1734200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1734200700 time 2024-04-26T11:39:05.733386+0000 2024-04-26T11:39:06.172 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:06.172 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173fb1c3dd] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.173 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: clone() 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: *** Caught signal (Aborted) ** 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: in thread 7f1734200700 thread_name:ms_dispatch 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f173e8a4d20] 2024-04-26T11:39:06.174 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: gsignal() 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: abort() 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173fb1c437] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.175 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 16: clone() 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:05.736+0000 7f1734200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: in thread 7f1734200700 thread_name:ms_dispatch 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f173e8a4d20] 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: gsignal() 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: abort() 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173fb1c437] 2024-04-26T11:39:06.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 16: clone() 2024-04-26T11:39:06.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: -1> 2024-04-26T11:39:05.733+0000 7f1734200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1734200700 time 2024-04-26T11:39:05.733386+0000 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173fb1c3dd] 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: clone() 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 0> 2024-04-26T11:39:05.736+0000 7f1734200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:06.179 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: in thread 7f1734200700 thread_name:ms_dispatch 2024-04-26T11:39:06.180 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.180 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.180 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f173e8a4d20] 2024-04-26T11:39:06.180 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: gsignal() 2024-04-26T11:39:06.180 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: abort() 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173fb1c437] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.181 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 16: clone() 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: -1> 2024-04-26T11:39:05.733+0000 7f1734200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1734200700 time 2024-04-26T11:39:05.733386+0000 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173fb1c3dd] 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.182 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.183 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: clone() 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 0> 2024-04-26T11:39:05.736+0000 7f1734200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: in thread 7f1734200700 thread_name:ms_dispatch 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f173e8a4d20] 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2: gsignal() 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 3: abort() 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173fb1c437] 2024-04-26T11:39:06.184 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f173fb1c5a3] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558145e30822] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558145da0598] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558145da2c17] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558145c9bd0b] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558145c9c2ac] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558145c84fad] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f173fdb0e20] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 13: (DispatchQueue::entry()+0x51f) [0x7f173fdadfdf] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173fe80fc1] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f173e89a1ca] 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 16: clone() 2024-04-26T11:39:06.185 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:06.186 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[67822]: 2024-04-26T11:39:06.186 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:05 smithi033 podman[91446]: 2024-04-26 11:39:05.92143786 +0000 UTC m=+0.053679755 container died c177a1841c35c10c29b13a9ed17d5941c255b19be4b08f394e9c25be7ad75bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T11:39:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:06 smithi194 ceph-mon[33017]: pgmap v280: 97 pgs: 97 active+clean; 3.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-04-26T11:39:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:06 smithi033 ceph-mon[27845]: pgmap v280: 97 pgs: 97 active+clean; 3.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-04-26T11:39:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:06 smithi096 ceph-mon[32753]: pgmap v280: 97 pgs: 97 active+clean; 3.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-04-26T11:39:07.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 470.0 s, 1768.0 tps, lat 55.675 ms stddev 69.416 2024-04-26T11:39:07.676 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:07 smithi033 podman[91446]: 2024-04-26 11:39:07.405456445 +0000 UTC m=+1.537698317 container remove c177a1841c35c10c29b13a9ed17d5941c255b19be4b08f394e9c25be7ad75bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:39:07.676 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:07 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:39:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:07 smithi194 ceph-mon[33017]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T11:39:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:07 smithi033 ceph-mon[27845]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T11:39:08.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:07 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-26T11:39:08.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:07 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.608s CPU time. 2024-04-26T11:39:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:07 smithi096 ceph-mon[32753]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T11:39:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:08 smithi194 ceph-mon[33017]: pgmap v282: 97 pgs: 97 active+clean; 3.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-04-26T11:39:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:08 smithi033 ceph-mon[27845]: pgmap v282: 97 pgs: 97 active+clean; 3.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-04-26T11:39:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:08 smithi096 ceph-mon[32753]: pgmap v282: 97 pgs: 97 active+clean; 3.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-04-26T11:39:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:10 smithi194 ceph-mon[33017]: pgmap v283: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 13 MiB/s wr, 1.28k op/s 2024-04-26T11:39:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:10 smithi033 ceph-mon[27845]: pgmap v283: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 13 MiB/s wr, 1.28k op/s 2024-04-26T11:39:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:10 smithi096 ceph-mon[32753]: pgmap v283: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 13 MiB/s wr, 1.28k op/s 2024-04-26T11:39:12.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 475.0 s, 564.2 tps, lat 51.993 ms stddev 73.383 2024-04-26T11:39:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:12 smithi194 ceph-mon[33017]: pgmap v284: 97 pgs: 97 active+clean; 3.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 14 MiB/s wr, 1.41k op/s 2024-04-26T11:39:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:12 smithi033 ceph-mon[27845]: pgmap v284: 97 pgs: 97 active+clean; 3.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 14 MiB/s wr, 1.41k op/s 2024-04-26T11:39:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:12 smithi096 ceph-mon[32753]: pgmap v284: 97 pgs: 97 active+clean; 3.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 14 MiB/s wr, 1.41k op/s 2024-04-26T11:39:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 -- 172.21.15.33:0/995606683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6174077780 msgr2=0x7f6174113280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 --2- 172.21.15.33:0/995606683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6174077780 0x7f6174113280 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6160009890 tx=0x7f616002a360 comp rx=0 tx=0).stop 2024-04-26T11:39:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 -- 172.21.15.33:0/995606683 shutdown_connections 2024-04-26T11:39:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 --2- 172.21.15.33:0/995606683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61741137c0 0x7f6174115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 --2- 172.21.15.33:0/995606683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6174077780 0x7f6174113280 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 --2- 172.21.15.33:0/995606683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6174076e60 0x7f6174077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 -- 172.21.15.33:0/995606683 >> 172.21.15.33:0/995606683 conn(0x7f6174100630 msgr2=0x7f6174102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 -- 172.21.15.33:0/995606683 shutdown_connections 2024-04-26T11:39:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.855+0000 7f617a200640 1 -- 172.21.15.33:0/995606683 wait complete. 2024-04-26T11:39:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.856+0000 7f617a200640 1 Processor -- start 2024-04-26T11:39:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.856+0000 7f617a200640 1 -- start start 2024-04-26T11:39:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.856+0000 7f617a200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6174076e60 0x7f617419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f617a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6174077780 0x7f617419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f617a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61741137c0 0x7f61741a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f617a200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61741191c0 con 0x7f6174076e60 2024-04-26T11:39:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f617a200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6174119040 con 0x7f61741137c0 2024-04-26T11:39:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f617a200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6174119340 con 0x7f6174077780 2024-04-26T11:39:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6174076e60 0x7f617419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6174076e60 0x7f617419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33618/0 (socket says 172.21.15.33:33618) 2024-04-26T11:39:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 -- 172.21.15.33:0/4148272880 learned_addr learned my addr 172.21.15.33:0/4148272880 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6172a00640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6174077780 0x7f617419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173e00640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61741137c0 0x7f61741a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 -- 172.21.15.33:0/4148272880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6174077780 msgr2=0x7f617419fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6174077780 0x7f617419fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 -- 172.21.15.33:0/4148272880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61741137c0 msgr2=0x7f61741a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61741137c0 0x7f61741a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.857+0000 7f6173400640 1 -- 172.21.15.33:0/4148272880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61741a46e0 con 0x7f6174076e60 2024-04-26T11:39:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.858+0000 7f6173400640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6174076e60 0x7f617419f740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f616400e9c0 tx=0x7f616400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.858+0000 7f616be00640 1 -- 172.21.15.33:0/4148272880 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616400cde0 con 0x7f6174076e60 2024-04-26T11:39:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.858+0000 7f616be00640 1 -- 172.21.15.33:0/4148272880 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61640035f0 con 0x7f6174076e60 2024-04-26T11:39:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.858+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61741a49d0 con 0x7f6174076e60 2024-04-26T11:39:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.858+0000 7f616be00640 1 -- 172.21.15.33:0/4148272880 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6164010690 con 0x7f6174076e60 2024-04-26T11:39:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.858+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61741ac260 con 0x7f6174076e60 2024-04-26T11:39:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.862+0000 7f617d1d1740 1 Processor -- start 2024-04-26T11:39:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f617d1d1740 1 -- start start 2024-04-26T11:39:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f616be00640 1 -- 172.21.15.33:0/4148272880 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6164004000 con 0x7f6174076e60 2024-04-26T11:39:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc396f0 0x5565abc43a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f617d1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565abb9e380 con 0x5565abc396f0 2024-04-26T11:39:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f617d1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5565abb3ac30 con 0x5565abc43fb0 2024-04-26T11:39:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.863+0000 7f617d1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5565abb9a340 con 0x5565abc38dd0 2024-04-26T11:39:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc396f0 0x5565abc43a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57938/0 (socket says 172.21.15.33:57938) 2024-04-26T11:39:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 -- 172.21.15.33:0/1665355677 learned_addr learned my addr 172.21.15.33:0/1665355677 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6172a00640 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 -- 172.21.15.33:0/1665355677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 msgr2=0x5565abc463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 -- 172.21.15.33:0/1665355677 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc396f0 msgr2=0x5565abc43a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc396f0 0x5565abc43a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173e00640 1 -- 172.21.15.33:0/1665355677 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565abb5c940 con 0x5565abc38dd0 2024-04-26T11:39:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.864+0000 7f6173400640 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc396f0 0x5565abc43a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.865+0000 7f6173e00640 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc391b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f616c009f30 tx=0x7f616c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1a8283da739540 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.865+0000 7f615fe00640 1 -- 172.21.15.33:0/1665355677 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c004d20 con 0x5565abc38dd0 2024-04-26T11:39:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.865+0000 7f615fe00640 1 -- 172.21.15.33:0/1665355677 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f616c004050 con 0x5565abc38dd0 2024-04-26T11:39:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.865+0000 7f615fe00640 1 -- 172.21.15.33:0/1665355677 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c005220 con 0x5565abc38dd0 2024-04-26T11:39:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.865+0000 7f616be00640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f61580a3dc0 0x7f61580a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.865+0000 7f616be00640 1 -- 172.21.15.33:0/4148272880 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9618+0+0 (secure 0 0 0) 0x7f61640d0fb0 con 0x7f6174076e60 2024-04-26T11:39:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.865+0000 7f6173400640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f61580a3dc0 0x7f61580a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 -- 172.21.15.33:0/1665355677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 msgr2=0x5565abc391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc391b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f616c009f30 tx=0x7f616c0118d0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 -- 172.21.15.33:0/1665355677 shutdown_connections 2024-04-26T11:39:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc396f0 0x5565abc43a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1665355677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc391b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 -- 172.21.15.33:0/1665355677 >> 172.21.15.33:0/1665355677 conn(0x5565abc352e0 msgr2=0x5565abc356c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 -- 172.21.15.33:0/1665355677 shutdown_connections 2024-04-26T11:39:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 -- 172.21.15.33:0/1665355677 wait complete. 2024-04-26T11:39:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f6173400640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f61580a3dc0 0x7f61580a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5565abc3a370 tx=0x7f61640d4a60 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.866+0000 7f617d1d1740 1 -- start start 2024-04-26T11:39:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 1 Processor -- start 2024-04-26T11:39:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc38dd0 0x5565abc40ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc41430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc41970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565abc46be0 con 0x5565abc43fb0 2024-04-26T11:39:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5565abb25c60 con 0x5565abc38dd0 2024-04-26T11:39:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5565abc46d60 con 0x5565abc396f0 2024-04-26T11:39:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f617d1d1740 10 client.0 _ll_register_callbacks cb 0x5565abb51e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:39:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f6173e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc41970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.867+0000 7f6172a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc41430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc41430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:57940/0 (socket says 172.21.15.33:57940) 2024-04-26T11:39:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 -- 172.21.15.33:0/1900998656 learned_addr learned my addr 172.21.15.33:0/1900998656 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6173e00640 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc38dd0 0x5565abc40ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 -- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc38dd0 msgr2=0x5565abc40ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc38dd0 0x5565abc40ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 -- 172.21.15.33:0/1900998656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 msgr2=0x5565abc41970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc41970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 -- 172.21.15.33:0/1900998656 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565abb34510 con 0x5565abc396f0 2024-04-26T11:39:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6173e00640 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc41970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f6172a00640 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc41430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6160005e00 tx=0x7f616002ae90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c600e8fdb3a5192 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.868+0000 7f615cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:39:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f615cc00640 1 -- 172.21.15.33:0/1900998656 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160042070 con 0x5565abc396f0 2024-04-26T11:39:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f615cc00640 1 -- 172.21.15.33:0/1900998656 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9618+0+0 (secure 0 0 0) 0x7f6160004040 con 0x5565abc396f0 2024-04-26T11:39:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f616c004050 con 0x5565abc396f0 2024-04-26T11:39:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f615cc00640 1 -- 172.21.15.33:0/1900998656 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f616004e050 con 0x5565abc396f0 2024-04-26T11:39:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f617d1d1740 10 client.25867 fetch_fsmap learned FSMap version 115 2024-04-26T11:39:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x5565ab8b0210 con 0x5565abc396f0 2024-04-26T11:39:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f615cc00640 1 -- 172.21.15.33:0/1900998656 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f616c004050 con 0x5565abc396f0 2024-04-26T11:39:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f615cc00640 1 -- 172.21.15.33:0/1900998656 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 115) ==== 7128+0+0 (secure 0 0 0) 0x7f6160003680 con 0x5565abc396f0 2024-04-26T11:39:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f617d1d1740 10 client.25867 fetch_fsmap finished waiting for FSMap version 115 2024-04-26T11:39:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f617d1d1740 10 client.25867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:39:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.869+0000 7f617d1d1740 20 client.25867 populate_metadata read hostname 'smithi033' 2024-04-26T11:39:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.870+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abc53cb0 0x5565abc560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.870+0000 7f617d1d1740 4 client.25867 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:39:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.870+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565abbabf90 con 0x5565abc53cb0 2024-04-26T11:39:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.870+0000 7f6173e00640 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abc53cb0 0x5565abc560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.870+0000 7f6173e00640 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abc53cb0 0x5565abc560c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9699d30c3f78cd23 server_cookie=f73e6bebe948c5f7 in_seq=0 out_seq=0 2024-04-26T11:39:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.870+0000 7f615cc00640 10 client.25867 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f615cc00640 1 -- 172.21.15.33:0/1900998656 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565abbabf90 con 0x5565abc53cb0 2024-04-26T11:39:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f615cc00640 10 client.25867 handle_command_reply: tid=0 2024-04-26T11:39:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 client.25867 shutdown 2024-04-26T11:39:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 msgr2=0x5565abc41430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc41430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6160005e00 tx=0x7f616002ae90 comp rx=0 tx=0).stop 2024-04-26T11:39:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 shutdown_connections 2024-04-26T11:39:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc41430 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abc53cb0 0x5565abc560c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc41970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1900998656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc38dd0 0x5565abc40ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f615cc00640 0 client.25867 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.871+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 >> 172.21.15.33:0/1900998656 conn(0x5565abc352e0 msgr2=0x5565abc3b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.872+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 shutdown_connections 2024-04-26T11:39:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.872+0000 7f617d1d1740 1 -- 172.21.15.33:0/1900998656 wait complete. 2024-04-26T11:39:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.872+0000 7f617d1d1740 20 client.25867 trim_cache size 0 max 16384 2024-04-26T11:39:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.888+0000 7f617d1d1740 1 Processor -- start 2024-04-26T11:39:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.888+0000 7f617d1d1740 1 -- start start 2024-04-26T11:39:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.888+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc38dd0 0x5565abc4faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.888+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc40050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.888+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc5b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.888+0000 7f617d1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b61f0 con 0x5565abc38dd0 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.888+0000 7f617d1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b68c0 con 0x5565abc43fb0 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f617d1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b6f90 con 0x5565abc396f0 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6173e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc38dd0 0x5565abc4faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6172a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc40050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6173e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc38dd0 0x5565abc4faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33636/0 (socket says 172.21.15.33:33636) 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6173e00640 1 -- 172.21.15.33:0/1390595720 learned_addr learned my addr 172.21.15.33:0/1390595720 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6173e00640 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc5b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6172a00640 1 -- 172.21.15.33:0/1390595720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 msgr2=0x5565abc5b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6172a00640 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc5b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6172a00640 1 -- 172.21.15.33:0/1390595720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc38dd0 msgr2=0x5565abc4faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6172a00640 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc38dd0 0x5565abc4faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6172a00640 1 -- 172.21.15.33:0/1390595720 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565abc39520 con 0x5565abc396f0 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6173e00640 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc5b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.889+0000 7f6172a00640 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc40050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f61600095f0 tx=0x7f6160005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82465e3b2a91bf3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f615f400640 1 -- 172.21.15.33:0/1390595720 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160057070 con 0x5565abc396f0 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f615f400640 1 -- 172.21.15.33:0/1390595720 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61600307f0 con 0x5565abc396f0 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f615f400640 1 -- 172.21.15.33:0/1390595720 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160042040 con 0x5565abc396f0 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 -- 172.21.15.33:0/1390595720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 msgr2=0x5565abc40050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc40050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f61600095f0 tx=0x7f6160005c90 comp rx=0 tx=0).stop 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 -- 172.21.15.33:0/1390595720 shutdown_connections 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc396f0 0x5565abc40050 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc43fb0 0x5565abc5b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 --2- 172.21.15.33:0/1390595720 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc38dd0 0x5565abc4faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 -- 172.21.15.33:0/1390595720 >> 172.21.15.33:0/1390595720 conn(0x5565abc352e0 msgr2=0x5565abc356c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.890+0000 7f617d1d1740 1 -- 172.21.15.33:0/1390595720 shutdown_connections 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.891+0000 7f617d1d1740 1 -- 172.21.15.33:0/1390595720 wait complete. 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.891+0000 7f617d1d1740 1 -- start start 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.891+0000 7f617d1d1740 1 Processor -- start 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.891+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.891+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc396f0 0x5565abc47f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.891+0000 7f617d1d1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc48470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f617d1d1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b34f0 con 0x5565abc43fb0 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f617d1d1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b48e0 con 0x5565abc396f0 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f617d1d1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b3b40 con 0x5565abc38dd0 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f617d1d1740 10 client.0 _ll_register_callbacks cb 0x7f6164005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:39:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc396f0 0x5565abc47f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6173e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc48470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc396f0 0x5565abc47f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:43154/0 (socket says 172.21.15.33:43154) 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 -- 172.21.15.33:0/3264180046 learned_addr learned my addr 172.21.15.33:0/3264180046 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6173e00640 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 -- 172.21.15.33:0/3264180046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 msgr2=0x5565abc479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 -- 172.21.15.33:0/3264180046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 msgr2=0x5565abc48470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc48470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 -- 172.21.15.33:0/3264180046 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ab97c620 con 0x5565abc396f0 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6173e00640 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f6172a00640 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc396f0 0x5565abc47f30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f616002a880 tx=0x7f6160000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b383864a31e66f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.892+0000 7f615cc00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:39:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f615cc00640 1 -- 172.21.15.33:0/3264180046 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160057030 con 0x5565abc396f0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61600307f0 con 0x5565abc396f0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f615cc00640 1 -- 172.21.15.33:0/3264180046 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9618+0+0 (secure 0 0 0) 0x7f6160030ce0 con 0x5565abc396f0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f615cc00640 1 -- 172.21.15.33:0/3264180046 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f6160064050 con 0x5565abc396f0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 10 client.25949 fetch_fsmap learned FSMap version 115 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x5565abb58cb0 con 0x5565abc396f0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f615cc00640 1 -- 172.21.15.33:0/3264180046 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61600307f0 con 0x5565abc396f0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f615cc00640 1 -- 172.21.15.33:0/3264180046 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 115) ==== 7128+0+0 (secure 0 0 0) 0x7f6160060cc0 con 0x5565abc396f0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 10 client.25949 fetch_fsmap finished waiting for FSMap version 115 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 10 client.25949 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 20 client.25949 populate_metadata read hostname 'smithi033' 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abcba750 0x5565abcbcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 4 client.25949 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.893+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565abba5e00 con 0x5565abcba750 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.894+0000 7f6173e00640 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abcba750 0x5565abcbcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.894+0000 7f6173e00640 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abcba750 0x5565abcbcb70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c67fa69b332ad5e7 server_cookie=b8e3dbc40b559d7f in_seq=0 out_seq=0 2024-04-26T11:39:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:12.894+0000 7f615cc00640 10 client.25949 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:14 smithi194 ceph-mon[33017]: pgmap v285: 97 pgs: 97 active+clean; 3.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 14 MiB/s wr, 1.41k op/s 2024-04-26T11:39:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:14 smithi033 ceph-mon[27845]: pgmap v285: 97 pgs: 97 active+clean; 3.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 14 MiB/s wr, 1.41k op/s 2024-04-26T11:39:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:14 smithi096 ceph-mon[32753]: pgmap v285: 97 pgs: 97 active+clean; 3.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 14 MiB/s wr, 1.41k op/s 2024-04-26T11:39:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:16 smithi194 ceph-mon[33017]: pgmap v286: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-04-26T11:39:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:16 smithi194 ceph-mon[33017]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T11:39:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:16 smithi033 ceph-mon[27845]: pgmap v286: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-04-26T11:39:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:16 smithi033 ceph-mon[27845]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T11:39:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:16 smithi096 ceph-mon[32753]: pgmap v286: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-04-26T11:39:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:16 smithi096 ceph-mon[32753]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T11:39:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 480.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:39:18.489 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:39:18.489 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 systemd[1]: Stopped Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:39:18.490 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.608s CPU time. 2024-04-26T11:39:18.490 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 systemd[1]: Starting Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:39:18.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 podman[91581]: 2024-04-26 11:39:18.482855879 +0000 UTC m=+0.036551676 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:39:18.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 podman[91581]: 2024-04-26 11:39:18.606376893 +0000 UTC m=+0.160072715 container create 8675d4027dfb7e3ad92d3ee9e7fb61ae3a31daf565b4ca7889af886fca4a1c45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T11:39:18.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:18 smithi033 ceph-mon[27845]: pgmap v288: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-04-26T11:39:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:18 smithi194 ceph-mon[33017]: pgmap v288: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-04-26T11:39:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:18 smithi096 ceph-mon[32753]: pgmap v288: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-04-26T11:39:19.112 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 podman[91581]: 2024-04-26 11:39:18.860696563 +0000 UTC m=+0.414392391 container init 8675d4027dfb7e3ad92d3ee9e7fb61ae3a31daf565b4ca7889af886fca4a1c45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T11:39:19.112 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 podman[91581]: 2024-04-26 11:39:18.873403133 +0000 UTC m=+0.427098925 container start 8675d4027dfb7e3ad92d3ee9e7fb61ae3a31daf565b4ca7889af886fca4a1c45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T11:39:19.113 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 ceph-mds[91601]: -- 172.21.15.33:0/889182391 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a0f4e37a40 con 0x55a0f4e0ac00 2024-04-26T11:39:19.113 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: starting mds.d at 2024-04-26T11:39:19.113 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 bash[91581]: 8675d4027dfb7e3ad92d3ee9e7fb61ae3a31daf565b4ca7889af886fca4a1c45 2024-04-26T11:39:19.113 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:18 smithi033 systemd[1]: Started Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: daemon mds.d restarted 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:boot 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:19 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: daemon mds.d restarted 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:boot 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:19 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: daemon mds.d restarted 2024-04-26T11:39:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:39:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T11:39:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:boot 2024-04-26T11:39:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:39:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:39:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:39:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:19 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:20 smithi033 ceph-mon[27845]: pgmap v290: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-04-26T11:39:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:20 smithi194 ceph-mon[33017]: pgmap v290: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-04-26T11:39:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:20 smithi096 ceph-mon[32753]: pgmap v290: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-04-26T11:39:21.758 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:21 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:resolve 2024-04-26T11:39:21.758 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:21 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:21 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:resolve 2024-04-26T11:39:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:21 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:21 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:resolve 2024-04-26T11:39:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:21 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:22.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 485.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:39:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.532+0000 7f615cc00640 1 -- 172.21.15.33:0/3264180046 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565abba5e00 con 0x5565abcba750 2024-04-26T11:39:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.532+0000 7f615cc00640 10 client.25949 handle_command_reply: tid=0 2024-04-26T11:39:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.532+0000 7f617d1d1740 1 client.25949 shutdown 2024-04-26T11:39:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc396f0 msgr2=0x5565abc47f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc396f0 0x5565abc47f30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f616002a880 tx=0x7f6160000bc0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 shutdown_connections 2024-04-26T11:39:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5565abcba750 0x5565abcbcb70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565abc43fb0 0x5565abc48470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565abc396f0 0x5565abc47f30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 --2- 172.21.15.33:0/3264180046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5565abc38dd0 0x5565abc479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 >> 172.21.15.33:0/3264180046 conn(0x5565abc352e0 msgr2=0x5565abc40eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f615cc00640 0 client.25949 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 shutdown_connections 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.533+0000 7f617d1d1740 1 -- 172.21.15.33:0/3264180046 wait complete. 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.534+0000 7f617d1d1740 20 client.25949 trim_cache size 0 max 16384 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:39:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:39:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.535+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f61580a3dc0 msgr2=0x7f61580a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.535+0000 7f617a200640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f61580a3dc0 0x7f61580a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5565abc3a370 tx=0x7f61640d4a60 comp rx=0 tx=0).stop 2024-04-26T11:39:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.535+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6174076e60 msgr2=0x7f617419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.535+0000 7f617a200640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6174076e60 0x7f617419f740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f616400e9c0 tx=0x7f616400ee80 comp rx=0 tx=0).stop 2024-04-26T11:39:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.535+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 shutdown_connections 2024-04-26T11:39:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.536+0000 7f617a200640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f61580a3dc0 0x7f61580a6280 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.536+0000 7f617a200640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6174077780 0x7f617419fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.536+0000 7f617a200640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61741137c0 0x7f61741a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.536+0000 7f617a200640 1 --2- 172.21.15.33:0/4148272880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6174076e60 0x7f617419f740 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.536+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 >> 172.21.15.33:0/4148272880 conn(0x7f6174100630 msgr2=0x7f6174114f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.536+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 shutdown_connections 2024-04-26T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:22.536+0000 7f617a200640 1 -- 172.21.15.33:0/4148272880 wait complete. 2024-04-26T11:39:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:22 smithi194 ceph-mon[33017]: pgmap v291: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-04-26T11:39:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:22 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:39:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:22 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:reconnect 2024-04-26T11:39:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:22 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:22 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999979 2024-04-26T11:39:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:22 smithi033 ceph-mon[27845]: pgmap v291: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-04-26T11:39:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:22 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:39:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:22 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:reconnect 2024-04-26T11:39:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:22 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:22 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999979 2024-04-26T11:39:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:22 smithi096 ceph-mon[32753]: pgmap v291: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-04-26T11:39:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:22 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:39:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:22 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:reconnect 2024-04-26T11:39:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:22 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:22 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999979 2024-04-26T11:39:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:23 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:rejoin 2024-04-26T11:39:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:23 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:23 smithi194 ceph-mon[33017]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:39:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:23 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:rejoin 2024-04-26T11:39:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:23 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:23 smithi033 ceph-mon[27845]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:39:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:23 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:rejoin 2024-04-26T11:39:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:23 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:23 smithi096 ceph-mon[32753]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:24 smithi194 ceph-mon[33017]: pgmap v292: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 967 KiB/s wr, 113 op/s 2024-04-26T11:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:24 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:24 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:active 2024-04-26T11:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:24 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:24 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:24 smithi033 ceph-mon[27845]: pgmap v292: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 967 KiB/s wr, 113 op/s 2024-04-26T11:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:24 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:24 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:active 2024-04-26T11:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:24 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:24 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:24 smithi096 ceph-mon[32753]: pgmap v292: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 967 KiB/s wr, 113 op/s 2024-04-26T11:39:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:24 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:24 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:39:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/3830261408,v1:172.21.15.96:6833/3830261408] up:active 2024-04-26T11:39:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:24 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:24 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:26 smithi194 ceph-mon[33017]: pgmap v293: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.2 MiB/s wr, 244 op/s 2024-04-26T11:39:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:26 smithi033 ceph-mon[27845]: pgmap v293: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.2 MiB/s wr, 244 op/s 2024-04-26T11:39:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:26 smithi096 ceph-mon[32753]: pgmap v293: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.2 MiB/s wr, 244 op/s 2024-04-26T11:39:27.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 490.0 s, 2362.2 tps, lat 157.639 ms stddev 1378.959 2024-04-26T11:39:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:28 smithi194 ceph-mon[33017]: pgmap v294: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 2.0 MiB/s wr, 222 op/s 2024-04-26T11:39:28.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:28 smithi033 ceph-mon[27845]: pgmap v294: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 2.0 MiB/s wr, 222 op/s 2024-04-26T11:39:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-mon[32753]: pgmap v294: 97 pgs: 97 active+clean; 3.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 2.0 MiB/s wr, 222 op/s 2024-04-26T11:39:28.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8972688700 time 2024-04-26T11:39:28.556148+0000 2024-04-26T11:39:28.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:28.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f897ccb13dd] 2024-04-26T11:39:28.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: clone() 2024-04-26T11:39:28.934 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.556+0000 7f8972688700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8972688700 time 2024-04-26T11:39:28.556148+0000 2024-04-26T11:39:28.935 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:28.935 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f897ccb13dd] 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.939 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: clone() 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.940 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: *** Caught signal (Aborted) ** 2024-04-26T11:39:28.941 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: in thread 7f8972688700 thread_name:ms_dispatch 2024-04-26T11:39:28.941 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.941 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f897ba39d20] 2024-04-26T11:39:28.941 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: gsignal() 2024-04-26T11:39:28.941 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: abort() 2024-04-26T11:39:28.941 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f897ccb1437] 2024-04-26T11:39:28.941 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 16: clone() 2024-04-26T11:39:28.942 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.558+0000 7f8972688700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: in thread 7f8972688700 thread_name:ms_dispatch 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f897ba39d20] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: gsignal() 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: abort() 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f897ccb1437] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.943 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 16: clone() 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: -1> 2024-04-26T11:39:28.556+0000 7f8972688700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8972688700 time 2024-04-26T11:39:28.556148+0000 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.944 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f897ccb13dd] 2024-04-26T11:39:28.945 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.945 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.945 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.945 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.945 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.946 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.946 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.946 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.946 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.947 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.947 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.947 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: clone() 2024-04-26T11:39:28.947 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.948 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 0> 2024-04-26T11:39:28.558+0000 7f8972688700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:28.948 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: in thread 7f8972688700 thread_name:ms_dispatch 2024-04-26T11:39:28.948 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.948 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.948 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f897ba39d20] 2024-04-26T11:39:28.949 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: gsignal() 2024-04-26T11:39:28.949 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: abort() 2024-04-26T11:39:28.949 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f897ccb1437] 2024-04-26T11:39:28.949 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.949 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.949 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.950 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.950 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.950 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.950 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.950 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.951 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.951 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.951 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.951 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 16: clone() 2024-04-26T11:39:28.951 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:28.951 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.952 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: -1> 2024-04-26T11:39:28.556+0000 7f8972688700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8972688700 time 2024-04-26T11:39:28.556148+0000 2024-04-26T11:39:28.952 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:28.952 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.952 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.952 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f897ccb13dd] 2024-04-26T11:39:28.952 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.953 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.953 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.953 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.953 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.953 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.954 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.954 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.954 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.954 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.954 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.955 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: clone() 2024-04-26T11:39:28.955 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.955 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 0> 2024-04-26T11:39:28.558+0000 7f8972688700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:28.955 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: in thread 7f8972688700 thread_name:ms_dispatch 2024-04-26T11:39:28.955 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.955 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:28.956 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f897ba39d20] 2024-04-26T11:39:28.956 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2: gsignal() 2024-04-26T11:39:28.956 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 3: abort() 2024-04-26T11:39:28.956 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f897ccb1437] 2024-04-26T11:39:28.956 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f897ccb15a3] 2024-04-26T11:39:28.956 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563d6c514822] 2024-04-26T11:39:28.957 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563d6c484598] 2024-04-26T11:39:28.957 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d6c486c17] 2024-04-26T11:39:28.957 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563d6c37fd0b] 2024-04-26T11:39:28.957 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d6c3802ac] 2024-04-26T11:39:28.957 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563d6c368fad] 2024-04-26T11:39:28.958 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f897cf45e20] 2024-04-26T11:39:28.958 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 13: (DispatchQueue::entry()+0x51f) [0x7f897cf42fdf] 2024-04-26T11:39:28.958 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f897d015fc1] 2024-04-26T11:39:28.958 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f897ba2f1ca] 2024-04-26T11:39:28.958 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 16: clone() 2024-04-26T11:39:28.958 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:28.959 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[61451]: 2024-04-26T11:39:28.959 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:28 smithi096 podman[68866]: 2024-04-26 11:39:28.623424112 +0000 UTC m=+0.020925706 container died e770095a5864e754c4ae734bb51cfe9dac95d6d0220c100a5964aa403b152613 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-26T11:39:29.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:29 smithi096 podman[68866]: 2024-04-26 11:39:29.426170201 +0000 UTC m=+0.823671795 container remove e770095a5864e754c4ae734bb51cfe9dac95d6d0220c100a5964aa403b152613 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T11:39:29.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:29 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:39:30.182 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:29 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-26T11:39:30.183 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:29 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.256s CPU time. 2024-04-26T11:39:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:30 smithi033 ceph-mon[27845]: pgmap v295: 97 pgs: 97 active+clean; 3.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-26T11:39:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:30 smithi194 ceph-mon[33017]: pgmap v295: 97 pgs: 97 active+clean; 3.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-26T11:39:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:30 smithi096 ceph-mon[32753]: pgmap v295: 97 pgs: 97 active+clean; 3.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-26T11:39:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 495.0 s, 1036.2 tps, lat 45.634 ms stddev 97.930 2024-04-26T11:39:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:32 smithi194 ceph-mon[33017]: pgmap v296: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-26T11:39:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:32 smithi033 ceph-mon[27845]: pgmap v296: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-26T11:39:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:32 smithi096 ceph-mon[32753]: pgmap v296: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-26T11:39:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:34 smithi194 ceph-mon[33017]: pgmap v297: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 13 MiB/s wr, 1.23k op/s 2024-04-26T11:39:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:34 smithi033 ceph-mon[27845]: pgmap v297: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 13 MiB/s wr, 1.23k op/s 2024-04-26T11:39:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:34 smithi096 ceph-mon[32753]: pgmap v297: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 13 MiB/s wr, 1.23k op/s 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.685+0000 7f7f75c00640 1 -- 172.21.15.33:0/1408201499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f70108e70 msgr2=0x7f7f701092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.685+0000 7f7f75c00640 1 --2- 172.21.15.33:0/1408201499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f70108e70 0x7f7f701092b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009890 tx=0x7f7f5802a360 comp rx=0 tx=0).stop 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.686+0000 7f7f75c00640 1 -- 172.21.15.33:0/1408201499 shutdown_connections 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.686+0000 7f7f75c00640 1 --2- 172.21.15.33:0/1408201499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f701097f0 0x7f7f70115c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.686+0000 7f7f75c00640 1 --2- 172.21.15.33:0/1408201499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f70108e70 0x7f7f701092b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.686+0000 7f7f75c00640 1 --2- 172.21.15.33:0/1408201499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f7010ee00 0x7f7f7010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.686+0000 7f7f75c00640 1 -- 172.21.15.33:0/1408201499 >> 172.21.15.33:0/1408201499 conn(0x7f7f70078710 msgr2=0x7f7f70107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.686+0000 7f7f75c00640 1 -- 172.21.15.33:0/1408201499 shutdown_connections 2024-04-26T11:39:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.686+0000 7f7f75c00640 1 -- 172.21.15.33:0/1408201499 wait complete. 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 Processor -- start 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 -- start start 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f70108e70 0x7f7f7019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f701097f0 0x7f7f7019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f7010ee00 0x7f7f701a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f70118fd0 con 0x7f7f7010ee00 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f70118e50 con 0x7f7f701097f0 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.687+0000 7f7f75c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f70119150 con 0x7f7f70108e70 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f7010ee00 0x7f7f701a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f7010ee00 0x7f7f701a3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49096/0 (socket says 172.21.15.33:49096) 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 -- 172.21.15.33:0/312033999 learned_addr learned my addr 172.21.15.33:0/312033999 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f67e00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f701097f0 0x7f7f7019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f70108e70 0x7f7f7019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 -- 172.21.15.33:0/312033999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f70108e70 msgr2=0x7f7f7019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f70108e70 0x7f7f7019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 -- 172.21.15.33:0/312033999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f701097f0 msgr2=0x7f7f7019fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f701097f0 0x7f7f7019fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 -- 172.21.15.33:0/312033999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f701a4520 con 0x7f7f7010ee00 2024-04-26T11:39:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.688+0000 7f7f75200640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f7010ee00 0x7f7f701a3df0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800eb10 tx=0x7f7f6800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.689+0000 7f7f6d600640 1 -- 172.21.15.33:0/312033999 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6800ce20 con 0x7f7f7010ee00 2024-04-26T11:39:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.689+0000 7f7f6d600640 1 -- 172.21.15.33:0/312033999 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f68005020 con 0x7f7f7010ee00 2024-04-26T11:39:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.689+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f701a4810 con 0x7f7f7010ee00 2024-04-26T11:39:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.689+0000 7f7f6d600640 1 -- 172.21.15.33:0/312033999 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68010690 con 0x7f7f7010ee00 2024-04-26T11:39:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.689+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f701ac0a0 con 0x7f7f7010ee00 2024-04-26T11:39:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.692+0000 7f7f7a02e740 1 Processor -- start 2024-04-26T11:39:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.692+0000 7f7f6d600640 1 -- 172.21.15.33:0/312033999 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7f68005510 con 0x7f7f7010ee00 2024-04-26T11:39:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.693+0000 7f7f7a02e740 1 -- start start 2024-04-26T11:39:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.693+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x55825475e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.693+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254768c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.693+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x55825476b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.693+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5582546c3380 con 0x5582547691c0 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.693+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55825465fc30 con 0x55825475e900 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.693+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5582546bf340 con 0x55825475dfe0 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254768c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254768c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45860/0 (socket says 172.21.15.33:45860) 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 -- 172.21.15.33:0/2452096064 learned_addr learned my addr 172.21.15.33:0/2452096064 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f67e00640 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x55825476b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f67e00640 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x55825475e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 -- 172.21.15.33:0/2452096064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 msgr2=0x55825475e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x55825475e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 -- 172.21.15.33:0/2452096064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 msgr2=0x55825476b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x55825476b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 -- 172.21.15.33:0/2452096064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558254681940 con 0x55825475e900 2024-04-26T11:39:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6d600640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7f540a3d90 0x7f7f540a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254768c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009c50 tx=0x7f7f60011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb17c2973d7d09d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.694+0000 7f7f6d600640 1 -- 172.21.15.33:0/312033999 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9616+0+0 (secure 0 0 0) 0x7f7f680d12e0 con 0x7f7f7010ee00 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.695+0000 7f7f65600640 1 -- 172.21.15.33:0/2452096064 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6001e070 con 0x55825475e900 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.695+0000 7f7f67e00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7f540a3d90 0x7f7f540a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.695+0000 7f7f65600640 1 -- 172.21.15.33:0/2452096064 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f60019ae0 con 0x55825475e900 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.695+0000 7f7f65600640 1 -- 172.21.15.33:0/2452096064 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6000e980 con 0x55825475e900 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2452096064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 msgr2=0x558254768c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254768c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009c50 tx=0x7f7f60011810 comp rx=0 tx=0).stop 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f67e00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7f540a3d90 0x7f7f540a6250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f58035040 tx=0x7f7f58000bc0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2452096064 shutdown_connections 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x55825476b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254768c80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2452096064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x55825475e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2452096064 >> 172.21.15.33:0/2452096064 conn(0x55825475a4f0 msgr2=0x55825475a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2452096064 shutdown_connections 2024-04-26T11:39:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2452096064 wait complete. 2024-04-26T11:39:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.696+0000 7f7f7a02e740 1 -- start start 2024-04-26T11:39:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 1 Processor -- start 2024-04-26T11:39:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x5582547660a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55825475e900 0x5582547665e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5582547691c0 0x558254766b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55825476bdf0 con 0x55825475e900 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55825464ac60 con 0x5582547691c0 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55825476bf70 con 0x55825475dfe0 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f7a02e740 10 client.0 _ll_register_callbacks cb 0x558254676e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f67e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55825475e900 0x5582547665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f6fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x5582547660a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f67e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55825475e900 0x5582547665e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49118/0 (socket says 172.21.15.33:49118) 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f67e00640 1 -- 172.21.15.33:0/483348869 learned_addr learned my addr 172.21.15.33:0/483348869 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.697+0000 7f7f75200640 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5582547691c0 0x558254766b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f6fe00640 1 -- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5582547691c0 msgr2=0x558254766b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5582547691c0 0x558254766b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f6fe00640 1 -- 172.21.15.33:0/483348869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55825475e900 msgr2=0x5582547665e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55825475e900 0x5582547665e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f6fe00640 1 -- 172.21.15.33:0/483348869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558254659510 con 0x55825475dfe0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x5582547660a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f60006010 tx=0x7f7f60005e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fe401166477365e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f5e000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f5e000640 1 -- 172.21.15.33:0/483348869 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60028030 con 0x55825475dfe0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.698+0000 7f7f5e000640 1 -- 172.21.15.33:0/483348869 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9616+0+0 (secure 0 0 0) 0x7f7f60004020 con 0x55825475dfe0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f60019ae0 con 0x55825475dfe0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f5e000640 1 -- 172.21.15.33:0/483348869 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f7f60032050 con 0x55825475dfe0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f7a02e740 10 client.25891 fetch_fsmap learned FSMap version 122 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=4+}) -- 0x5582543d5210 con 0x55825475dfe0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f5e000640 1 -- 172.21.15.33:0/483348869 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f60019ae0 con 0x55825475dfe0 2024-04-26T11:39:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f5e000640 1 -- 172.21.15.33:0/483348869 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 122) ==== 7128+0+0 (secure 0 0 0) 0x7f7f60021030 con 0x55825475dfe0 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f7a02e740 10 client.25891 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f7a02e740 10 client.25891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.699+0000 7f7f7a02e740 20 client.25891 populate_metadata read hostname 'smithi033' 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.700+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x558254778da0 0x55825477b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.700+0000 7f7f7a02e740 4 client.25891 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.700+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582546d0f90 con 0x558254778da0 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.700+0000 7f7f67e00640 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x558254778da0 0x55825477b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.701+0000 7f7f67e00640 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x558254778da0 0x55825477b1d0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce6c2bb885a7710f server_cookie=599a709bee45c39a in_seq=0 out_seq=0 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.701+0000 7f7f5e000640 10 client.25891 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.701+0000 7f7f5e000640 1 -- 172.21.15.33:0/483348869 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582546d0f90 con 0x558254778da0 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.701+0000 7f7f5e000640 10 client.25891 handle_command_reply: tid=0 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.701+0000 7f7f7a02e740 1 client.25891 shutdown 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 msgr2=0x5582547660a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x5582547660a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f60006010 tx=0x7f7f60005e10 comp rx=0 tx=0).stop 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 shutdown_connections 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x558254778da0 0x55825477b1d0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5582547691c0 0x558254766b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55825475e900 0x5582547665e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f5e000640 0 client.25891 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/483348869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x5582547660a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 >> 172.21.15.33:0/483348869 conn(0x55825475a4f0 msgr2=0x558254760220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 shutdown_connections 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 1 -- 172.21.15.33:0/483348869 wait complete. 2024-04-26T11:39:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.702+0000 7f7f7a02e740 20 client.25891 trim_cache size 0 max 16384 2024-04-26T11:39:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.718+0000 7f7f7a02e740 1 Processor -- start 2024-04-26T11:39:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f7a02e740 1 -- start start 2024-04-26T11:39:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x558254774cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254765200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x5582547600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701b6110 con 0x5582547691c0 2024-04-26T11:39:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701b67e0 con 0x55825475e900 2024-04-26T11:39:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701b6eb0 con 0x55825475dfe0 2024-04-26T11:39:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f75200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x5582547600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.719+0000 7f7f67e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254765200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x5582547600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49124/0 (socket says 172.21.15.33:49124) 2024-04-26T11:39:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f67e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254765200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45878/0 (socket says 172.21.15.33:45878) 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 -- 172.21.15.33:0/676189775 learned_addr learned my addr 172.21.15.33:0/676189775 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x558254774cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 -- 172.21.15.33:0/676189775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 msgr2=0x558254774cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x558254774cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 -- 172.21.15.33:0/676189775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 msgr2=0x558254765200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254765200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 -- 172.21.15.33:0/676189775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825475e730 con 0x5582547691c0 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x558254774cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f67e00640 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254765200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.720+0000 7f7f75200640 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x5582547600a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7f68000f80 tx=0x7f7f68099120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad37ce20a09ef5c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f64c00640 1 -- 172.21.15.33:0/676189775 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f680a5a70 con 0x5582547691c0 2024-04-26T11:39:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f64c00640 1 -- 172.21.15.33:0/676189775 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f6809b100 con 0x5582547691c0 2024-04-26T11:39:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f64c00640 1 -- 172.21.15.33:0/676189775 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68098b50 con 0x5582547691c0 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 -- 172.21.15.33:0/676189775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 msgr2=0x5582547600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x5582547600a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7f68000f80 tx=0x7f7f68099120 comp rx=0 tx=0).stop 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 -- 172.21.15.33:0/676189775 shutdown_connections 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x5582547600a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475e900 0x558254765200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/676189775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475dfe0 0x558254774cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 -- 172.21.15.33:0/676189775 >> 172.21.15.33:0/676189775 conn(0x55825475a4f0 msgr2=0x55825475f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 -- 172.21.15.33:0/676189775 shutdown_connections 2024-04-26T11:39:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.721+0000 7f7f7a02e740 1 -- 172.21.15.33:0/676189775 wait complete. 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.722+0000 7f7f7a02e740 1 -- start start 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.722+0000 7f7f7a02e740 1 Processor -- start 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.722+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475dfe0 0x558254762d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.722+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475e900 0x5582547632c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.722+0000 7f7f7a02e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x558254763800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.722+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701b33d0 con 0x5582547691c0 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701b4800 con 0x55825475dfe0 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f7a02e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701b3a60 con 0x55825475e900 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f7a02e740 10 client.0 _ll_register_callbacks cb 0x7f7f6001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f75200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x558254763800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f6fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475dfe0 0x558254762d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f75200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x558254763800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49126/0 (socket says 172.21.15.33:49126) 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f6fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475dfe0 0x558254762d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45886/0 (socket says 172.21.15.33:45886) 2024-04-26T11:39:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f75200640 1 -- 172.21.15.33:0/2154815425 learned_addr learned my addr 172.21.15.33:0/2154815425 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f67e00640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475e900 0x5582547632c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f67e00640 1 -- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475dfe0 msgr2=0x558254762d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f67e00640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475dfe0 0x558254762d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f67e00640 1 -- 172.21.15.33:0/2154815425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 msgr2=0x558254763800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f67e00640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x558254763800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f67e00640 1 -- 172.21.15.33:0/2154815425 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582544a1620 con 0x55825475e900 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f75200640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x558254763800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.723+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475dfe0 0x558254762d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f67e00640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475e900 0x5582547632c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55825477ba50 tx=0x7f7f580056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc709808d65c0035 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f5e000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f5e000640 1 -- 172.21.15.33:0/2154815425 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f58044070 con 0x55825475e900 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f6809b100 con 0x55825475e900 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f5e000640 1 -- 172.21.15.33:0/2154815425 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9616+0+0 (secure 0 0 0) 0x7f7f58052d00 con 0x55825475e900 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f5e000640 1 -- 172.21.15.33:0/2154815425 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f7f58062050 con 0x55825475e900 2024-04-26T11:39:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f7a02e740 10 client.25897 fetch_fsmap learned FSMap version 122 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=4+}) -- 0x55825467dcb0 con 0x55825475e900 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.724+0000 7f7f5e000640 1 -- 172.21.15.33:0/2154815425 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f6809b100 con 0x55825475e900 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f5e000640 1 -- 172.21.15.33:0/2154815425 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 122) ==== 7128+0+0 (secure 0 0 0) 0x7f7f5803d040 con 0x55825475e900 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f7a02e740 10 client.25897 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f7a02e740 10 client.25897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f7a02e740 20 client.25897 populate_metadata read hostname 'smithi033' 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5582547df970 0x5582547e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f7a02e740 4 client.25897 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582546cae00 con 0x5582547df970 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5582547df970 0x5582547e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f6fe00640 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5582547df970 0x5582547e1d90 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd5e495a9ccde6ce server_cookie=fc3a062999f11b00 in_seq=0 out_seq=0 2024-04-26T11:39:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:35.725+0000 7f7f5e000640 10 client.25897 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:36 smithi194 ceph-mon[33017]: pgmap v298: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:39:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:36 smithi033 ceph-mon[27845]: pgmap v298: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:39:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:36 smithi096 ceph-mon[32753]: pgmap v298: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:39:37.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 500.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:39:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:38 smithi194 ceph-mon[33017]: pgmap v299: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-26T11:39:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:38 smithi033 ceph-mon[27845]: pgmap v299: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-26T11:39:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:38 smithi096 ceph-mon[32753]: pgmap v299: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-26T11:39:40.291 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:39 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:39:40.291 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:39 smithi096 systemd[1]: Stopped Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:39:40.291 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:39 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.256s CPU time. 2024-04-26T11:39:40.291 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:39 smithi096 systemd[1]: Starting Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:39:40.291 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 podman[68959]: 2024-04-26 11:39:40.065541349 +0000 UTC m=+0.017930898 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:39:40.291 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 podman[68959]: 2024-04-26 11:39:40.176937684 +0000 UTC m=+0.129327223 container create f0b582d27651368009c69b4d9f2310db93372adcfe570590b30f262d1ef75ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T11:39:40.573 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 podman[68959]: 2024-04-26 11:39:40.413388328 +0000 UTC m=+0.365777870 container init f0b582d27651368009c69b4d9f2310db93372adcfe570590b30f262d1ef75ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T11:39:40.573 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 podman[68959]: 2024-04-26 11:39:40.416238302 +0000 UTC m=+0.368627848 container start f0b582d27651368009c69b4d9f2310db93372adcfe570590b30f262d1ef75ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T11:39:40.573 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mds[68979]: -- 172.21.15.96:0/795132067 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55833b87ba40 con 0x55833b84ec00 2024-04-26T11:39:40.573 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: starting mds.k at 2024-04-26T11:39:40.573 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 bash[68959]: f0b582d27651368009c69b4d9f2310db93372adcfe570590b30f262d1ef75ed9 2024-04-26T11:39:40.573 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:39:40 smithi096 systemd[1]: Started Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: pgmap v300: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 1.29k op/s 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: daemon mds.k restarted 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:boot 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:39:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:40 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: pgmap v300: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 1.29k op/s 2024-04-26T11:39:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: daemon mds.k restarted 2024-04-26T11:39:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:39:40.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:39:40.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T11:39:40.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:boot 2024-04-26T11:39:40.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:39:40.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:39:40.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:40 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: pgmap v300: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 1.29k op/s 2024-04-26T11:39:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: daemon mds.k restarted 2024-04-26T11:39:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:39:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:39:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T11:39:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:boot 2024-04-26T11:39:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:39:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:39:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:40 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 505.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:39:42.857 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:42 smithi033 ceph-mon[27845]: pgmap v302: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T11:39:42.858 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:42 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:resolve 2024-04-26T11:39:42.858 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:42 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:42 smithi096 ceph-mon[32753]: pgmap v302: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T11:39:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:42 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:resolve 2024-04-26T11:39:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:42 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:42 smithi194 ceph-mon[33017]: pgmap v302: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T11:39:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:42 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:resolve 2024-04-26T11:39:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:42 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.627+0000 7f7f5e000640 1 -- 172.21.15.33:0/2154815425 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582546cae00 con 0x5582547df970 2024-04-26T11:39:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.627+0000 7f7f5e000640 10 client.25897 handle_command_reply: tid=0 2024-04-26T11:39:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.627+0000 7f7f7a02e740 1 client.25897 shutdown 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475e900 msgr2=0x5582547632c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475e900 0x5582547632c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55825477ba50 tx=0x7f7f580056d0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 shutdown_connections 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55825475e900 0x5582547632c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x5582547df970 0x5582547e1d90 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5582547691c0 0x558254763800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 --2- 172.21.15.33:0/2154815425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55825475dfe0 0x558254762d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f5e000640 0 client.25897 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.628+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 >> 172.21.15.33:0/2154815425 conn(0x55825475a4f0 msgr2=0x558254760220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:43.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.629+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 shutdown_connections 2024-04-26T11:39:43.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.629+0000 7f7f7a02e740 1 -- 172.21.15.33:0/2154815425 wait complete. 2024-04-26T11:39:43.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.629+0000 7f7f7a02e740 20 client.25897 trim_cache size 0 max 16384 2024-04-26T11:39:43.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:39:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:39:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:39:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:39:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7f540a3d90 msgr2=0x7f7f540a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7f540a3d90 0x7f7f540a6250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f58035040 tx=0x7f7f58000bc0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f7010ee00 msgr2=0x7f7f701a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f7010ee00 0x7f7f701a3df0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800eb10 tx=0x7f7f6800ef10 comp rx=0 tx=0).stop 2024-04-26T11:39:43.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 shutdown_connections 2024-04-26T11:39:43.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f7f540a3d90 0x7f7f540a6250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f7010ee00 0x7f7f701a3df0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f701097f0 0x7f7f7019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 --2- 172.21.15.33:0/312033999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f70108e70 0x7f7f7019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:43.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 >> 172.21.15.33:0/312033999 conn(0x7f7f70078710 msgr2=0x7f7f70079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:43.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 shutdown_connections 2024-04-26T11:39:43.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:43.631+0000 7f7f75c00640 1 -- 172.21.15.33:0/312033999 wait complete. 2024-04-26T11:39:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:43 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:39:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:43 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:reconnect 2024-04-26T11:39:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:43 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:43 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:39:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:43 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:39:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:43 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:reconnect 2024-04-26T11:39:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:43 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:43 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:39:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:43 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:39:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:43 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:reconnect 2024-04-26T11:39:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:43 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:43 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:44 smithi033 ceph-mon[27845]: pgmap v303: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T11:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:44 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:rejoin 2024-04-26T11:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:44 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:44 smithi033 ceph-mon[27845]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:44 smithi096 ceph-mon[32753]: pgmap v303: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T11:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:44 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:rejoin 2024-04-26T11:39:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:44 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:44 smithi096 ceph-mon[32753]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:39:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:44 smithi194 ceph-mon[33017]: pgmap v303: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T11:39:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:44 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:rejoin 2024-04-26T11:39:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:44 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:44 smithi194 ceph-mon[33017]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:39:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:45 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:45 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:39:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:45 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:active 2024-04-26T11:39:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:45 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:45 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:45 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:45 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:39:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:45 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:active 2024-04-26T11:39:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:45 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:45 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:45 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:39:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:45 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:39:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:45 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1139227988,v1:172.21.15.33:6837/1139227988] up:active 2024-04-26T11:39:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:45 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:39:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:45 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:46 smithi033 ceph-mon[27845]: pgmap v304: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-26T11:39:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:46 smithi096 ceph-mon[32753]: pgmap v304: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-26T11:39:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:46 smithi194 ceph-mon[33017]: pgmap v304: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-26T11:39:47.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 510.0 s, 1145.0 tps, lat 308.173 ms stddev 1993.887 2024-04-26T11:39:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:39:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:39:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:39:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:48 smithi194 ceph-mon[33017]: pgmap v305: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-26T11:39:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:48 smithi033 ceph-mon[27845]: pgmap v305: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-26T11:39:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:48 smithi096 ceph-mon[32753]: pgmap v305: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-26T11:39:49.936 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc707600700 time 2024-04-26T11:39:49.674092+0000 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc712e723dd] 2024-04-26T11:39:49.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: clone() 2024-04-26T11:39:49.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.675+0000 7fc707600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc707600700 time 2024-04-26T11:39:49.674092+0000 2024-04-26T11:39:49.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:49.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc712e723dd] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: clone() 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: *** Caught signal (Aborted) ** 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: in thread 7fc707600700 thread_name:ms_dispatch 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc711bfad20] 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: gsignal() 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: abort() 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc712e72437] 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 16: clone() 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.677+0000 7fc707600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: in thread 7fc707600700 thread_name:ms_dispatch 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.946 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc711bfad20] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: gsignal() 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: abort() 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc712e72437] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.947 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.948 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 16: clone() 2024-04-26T11:39:49.948 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:49.948 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.948 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: -1> 2024-04-26T11:39:49.675+0000 7fc707600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc707600700 time 2024-04-26T11:39:49.674092+0000 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc712e723dd] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.949 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: clone() 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 0> 2024-04-26T11:39:49.677+0000 7fc707600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: in thread 7fc707600700 thread_name:ms_dispatch 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc711bfad20] 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: gsignal() 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: abort() 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc712e72437] 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.950 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 16: clone() 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: -1> 2024-04-26T11:39:49.675+0000 7fc707600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc707600700 time 2024-04-26T11:39:49.674092+0000 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.951 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc712e723dd] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: clone() 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.952 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 0> 2024-04-26T11:39:49.677+0000 7fc707600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: in thread 7fc707600700 thread_name:ms_dispatch 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc711bfad20] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2: gsignal() 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 3: abort() 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc712e72437] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc712e725a3] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a0bda1b822] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a0bd98b598] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a0bd98dc17] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a0bd886d0b] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a0bd8872ac] 2024-04-26T11:39:49.953 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a0bd86ffad] 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc713106e20] 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 13: (DispatchQueue::entry()+0x51f) [0x7fc713103fdf] 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc7131d6fc1] 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc711bf01ca] 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 16: clone() 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91596]: 2024-04-26T11:39:49.954 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:49 smithi033 podman[91825]: 2024-04-26 11:39:49.849234476 +0000 UTC m=+0.045247072 container died 8675d4027dfb7e3ad92d3ee9e7fb61ae3a31daf565b4ca7889af886fca4a1c45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T11:39:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:49 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:39:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:39:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:39:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:49 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:39:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:39:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:39:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:39:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:50 smithi194 ceph-mon[33017]: pgmap v306: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 3.3 MiB/s wr, 324 op/s 2024-04-26T11:39:51.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:50 smithi033 podman[91825]: 2024-04-26 11:39:50.697848415 +0000 UTC m=+0.893861001 container remove 8675d4027dfb7e3ad92d3ee9e7fb61ae3a31daf565b4ca7889af886fca4a1c45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T11:39:51.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:50 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:39:51.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:50 smithi033 ceph-mon[27845]: pgmap v306: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 3.3 MiB/s wr, 324 op/s 2024-04-26T11:39:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:50 smithi096 ceph-mon[32753]: pgmap v306: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 3.3 MiB/s wr, 324 op/s 2024-04-26T11:39:51.675 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:51 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-26T11:39:51.676 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:39:51 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.045s CPU time. 2024-04-26T11:39:52.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 515.0 s, 1973.6 tps, lat 50.603 ms stddev 57.038 2024-04-26T11:39:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:52 smithi194 ceph-mon[33017]: pgmap v307: 97 pgs: 97 active+clean; 3.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.7 MiB/s wr, 374 op/s 2024-04-26T11:39:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:52 smithi033 ceph-mon[27845]: pgmap v307: 97 pgs: 97 active+clean; 3.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.7 MiB/s wr, 374 op/s 2024-04-26T11:39:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:52 smithi096 ceph-mon[32753]: pgmap v307: 97 pgs: 97 active+clean; 3.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.7 MiB/s wr, 374 op/s 2024-04-26T11:39:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:54 smithi194 ceph-mon[33017]: pgmap v308: 97 pgs: 97 active+clean; 3.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 3.4 MiB/s wr, 342 op/s 2024-04-26T11:39:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:54 smithi033 ceph-mon[27845]: pgmap v308: 97 pgs: 97 active+clean; 3.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 3.4 MiB/s wr, 342 op/s 2024-04-26T11:39:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:54 smithi096 ceph-mon[32753]: pgmap v308: 97 pgs: 97 active+clean; 3.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 3.4 MiB/s wr, 342 op/s 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 -- 172.21.15.33:0/3832506335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4076e60 msgr2=0x7f9aa4077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 --2- 172.21.15.33:0/3832506335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4076e60 0x7f9aa4077240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009890 tx=0x7f9a9802a360 comp rx=0 tx=0).stop 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 -- 172.21.15.33:0/3832506335 shutdown_connections 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 --2- 172.21.15.33:0/3832506335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa4077780 0x7f9aa4113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 --2- 172.21.15.33:0/3832506335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 0x7f9aa4115ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 --2- 172.21.15.33:0/3832506335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4076e60 0x7f9aa4077240 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 -- 172.21.15.33:0/3832506335 >> 172.21.15.33:0/3832506335 conn(0x7f9aa4100620 msgr2=0x7f9aa4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 -- 172.21.15.33:0/3832506335 shutdown_connections 2024-04-26T11:39:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.790+0000 7f9aab800640 1 -- 172.21.15.33:0/3832506335 wait complete. 2024-04-26T11:39:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.791+0000 7f9aab800640 1 Processor -- start 2024-04-26T11:39:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aab800640 1 -- start start 2024-04-26T11:39:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aab800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa4076e60 0x7f9aa419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aab800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4077780 0x7f9aa419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aab800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 0x7f9aa41a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aab800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41191b0 con 0x7f9aa41137b0 2024-04-26T11:39:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aab800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4119030 con 0x7f9aa4077780 2024-04-26T11:39:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aab800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4119330 con 0x7f9aa4076e60 2024-04-26T11:39:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.792+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 0x7f9aa41a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa8e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa4076e60 0x7f9aa419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4077780 0x7f9aa419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 0x7f9aa41a3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52344/0 (socket says 172.21.15.33:52344) 2024-04-26T11:39:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4077780 0x7f9aa419fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55368/0 (socket says 172.21.15.33:55368) 2024-04-26T11:39:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa9800640 1 -- 172.21.15.33:0/3352158209 learned_addr learned my addr 172.21.15.33:0/3352158209 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa8e00640 1 -- 172.21.15.33:0/3352158209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4077780 msgr2=0x7f9aa419fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4077780 0x7f9aa419fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa8e00640 1 -- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 msgr2=0x7f9aa41a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 0x7f9aa41a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa8e00640 1 -- 172.21.15.33:0/3352158209 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa41a4660 con 0x7f9aa4076e60 2024-04-26T11:39:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa3e00640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4077780 0x7f9aa419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.793+0000 7f9aa9800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 0x7f9aa41a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.794+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa4076e60 0x7f9aa419f670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009780 tx=0x7f9a98032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.794+0000 7f9aa1600640 1 -- 172.21.15.33:0/3352158209 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98037040 con 0x7f9aa4076e60 2024-04-26T11:39:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.794+0000 7f9aa1600640 1 -- 172.21.15.33:0/3352158209 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a98004050 con 0x7f9aa4076e60 2024-04-26T11:39:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.794+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa41a48f0 con 0x7f9aa4076e60 2024-04-26T11:39:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.794+0000 7f9aa1600640 1 -- 172.21.15.33:0/3352158209 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980035f0 con 0x7f9aa4076e60 2024-04-26T11:39:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.794+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa41ac1e0 con 0x7f9aa4076e60 2024-04-26T11:39:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.798+0000 7f9aae77e740 1 Processor -- start 2024-04-26T11:39:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.799+0000 7f9aae77e740 1 -- start start 2024-04-26T11:39:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.799+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cf981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.799+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfa2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.799+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.799+0000 7f9aae77e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55922cefd380 con 0x55922cfa2fb0 2024-04-26T11:39:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.799+0000 7f9aae77e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55922ce99c30 con 0x55922cf986f0 2024-04-26T11:39:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.799+0000 7f9aae77e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55922cef9340 con 0x55922cf97dd0 2024-04-26T11:39:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.800+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cf981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.800+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cf981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:43130/0 (socket says 172.21.15.33:43130) 2024-04-26T11:39:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.800+0000 7f9aa9800640 1 -- 172.21.15.33:0/4065303097 learned_addr learned my addr 172.21.15.33:0/4065303097 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.800+0000 7f9aa1600640 1 -- 172.21.15.33:0/3352158209 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9a98043020 con 0x7f9aa4076e60 2024-04-26T11:39:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.801+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfa2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.801+0000 7f9aa8e00640 1 -- 172.21.15.33:0/4065303097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 msgr2=0x55922cf981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.801+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cf981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.801+0000 7f9aa8e00640 1 -- 172.21.15.33:0/4065303097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 msgr2=0x55922cfa53a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.802+0000 7f9aa3e00640 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.803+0000 7f9aa1600640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9a8c0a3e40 0x7f9a8c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.803+0000 7f9aa1600640 1 -- 172.21.15.33:0/3352158209 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9702+0+0 (secure 0 0 0) 0x7f9a980efd60 con 0x7f9aa4076e60 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.804+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.804+0000 7f9aa8e00640 1 -- 172.21.15.33:0/4065303097 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55922cebb940 con 0x55922cf986f0 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.804+0000 7f9aa3e00640 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T11:39:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.805+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfa2a70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9a980ed0c0 tx=0x7f9a98005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25f8b79fbfdc5cda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.806+0000 7f9a96000640 1 -- 172.21.15.33:0/4065303097 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980b7bb0 con 0x55922cf986f0 2024-04-26T11:39:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.806+0000 7f9a96000640 1 -- 172.21.15.33:0/4065303097 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a980a4e50 con 0x55922cf986f0 2024-04-26T11:39:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.806+0000 7f9a96000640 1 -- 172.21.15.33:0/4065303097 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980bb270 con 0x55922cf986f0 2024-04-26T11:39:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.807+0000 7f9aa9800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9a8c0a3e40 0x7f9a8c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.807+0000 7f9aae77e740 1 -- 172.21.15.33:0/4065303097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 msgr2=0x55922cfa2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.807+0000 7f9aae77e740 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfa2a70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9a980ed0c0 tx=0x7f9a98005f90 comp rx=0 tx=0).stop 2024-04-26T11:39:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.808+0000 7f9aae77e740 1 -- 172.21.15.33:0/4065303097 shutdown_connections 2024-04-26T11:39:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.808+0000 7f9aae77e740 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.808+0000 7f9aae77e740 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfa2a70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.808+0000 7f9aae77e740 1 --2- 172.21.15.33:0/4065303097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cf981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.808+0000 7f9aae77e740 1 -- 172.21.15.33:0/4065303097 >> 172.21.15.33:0/4065303097 conn(0x55922cf942e0 msgr2=0x55922cf946c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.809+0000 7f9aa9800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9a8c0a3e40 0x7f9a8c0a6300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0094d0 tx=0x7f9a9c009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.810+0000 7f9aae77e740 1 -- 172.21.15.33:0/4065303097 shutdown_connections 2024-04-26T11:39:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.811+0000 7f9aae77e740 1 -- 172.21.15.33:0/4065303097 wait complete. 2024-04-26T11:39:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.814+0000 7f9aae77e740 1 -- start start 2024-04-26T11:39:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.814+0000 7f9aae77e740 1 Processor -- start 2024-04-26T11:39:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.816+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cf9ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.816+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf986f0 0x55922cfa0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.816+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.816+0000 7f9aae77e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55922ce99c30 con 0x55922cfa2fb0 2024-04-26T11:39:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.816+0000 7f9aae77e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55922ce84c60 con 0x55922cf97dd0 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.816+0000 7f9aae77e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55922cfa59a0 con 0x55922cf986f0 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.816+0000 7f9aae77e740 10 client.0 _ll_register_callbacks cb 0x55922ceb0e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.817+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.817+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52354/0 (socket says 172.21.15.33:52354) 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.817+0000 7f9aa9800640 1 -- 172.21.15.33:0/2351648324 learned_addr learned my addr 172.21.15.33:0/2351648324 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.817+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cf9ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.818+0000 7f9aa9800640 1 -- 172.21.15.33:0/2351648324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf986f0 msgr2=0x55922cfa0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.818+0000 7f9aa9800640 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf986f0 0x55922cfa0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.818+0000 7f9aa9800640 1 -- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 msgr2=0x55922cf9ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.818+0000 7f9aa9800640 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cf9ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.818+0000 7f9aa9800640 1 -- 172.21.15.33:0/2351648324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55922ce93510 con 0x55922cfa2fb0 2024-04-26T11:39:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.819+0000 7f9aa9800640 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa0980 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c010b30 tx=0x7f9a9c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32669c5793aaaeca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.819+0000 7f9a8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:39:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.820+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2351648324 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c003970 con 0x55922cfa2fb0 2024-04-26T11:39:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.820+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a980a4e50 con 0x55922cfa2fb0 2024-04-26T11:39:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.820+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2351648324 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9702+0+0 (secure 0 0 0) 0x7f9a9c003b10 con 0x55922cfa2fb0 2024-04-26T11:39:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.820+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2351648324 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (secure 0 0 0) 0x7f9a9c055050 con 0x55922cfa2fb0 2024-04-26T11:39:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.821+0000 7f9aae77e740 10 client.15744 fetch_fsmap learned FSMap version 130 2024-04-26T11:39:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.821+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=4+}) -- 0x55922cc0f210 con 0x55922cfa2fb0 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2351648324 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a980a4e50 con 0x55922cfa2fb0 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2351648324 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 130) ==== 7132+0+0 (secure 0 0 0) 0x7f9a9c004210 con 0x55922cfa2fb0 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9aae77e740 10 client.15744 fetch_fsmap finished waiting for FSMap version 130 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9aae77e740 10 client.15744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9aae77e740 20 client.15744 populate_metadata read hostname 'smithi033' 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922cfb00f0 0x55922cfb2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9aae77e740 4 client.15744 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:39:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.822+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55922cf0af90 con 0x55922cfb00f0 2024-04-26T11:39:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.826+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922cfb00f0 0x55922cfb2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.827+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922cfb00f0 0x55922cfb2570 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ba0039d5e1d0f34 server_cookie=83e11c72d0a3236a in_seq=0 out_seq=0 2024-04-26T11:39:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.827+0000 7f9a8aa00640 10 client.15744 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.828+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2351648324 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55922cf0af90 con 0x55922cfb00f0 2024-04-26T11:39:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.828+0000 7f9a8aa00640 10 client.15744 handle_command_reply: tid=0 2024-04-26T11:39:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.828+0000 7f9aae77e740 1 client.15744 shutdown 2024-04-26T11:39:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.830+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 msgr2=0x55922cfa0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.830+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa0980 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c010b30 tx=0x7f9a9c004ab0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 shutdown_connections 2024-04-26T11:39:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf986f0 0x55922cfa0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922cfb00f0 0x55922cfb2570 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfa0980 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2351648324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cf9ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 >> 172.21.15.33:0/2351648324 conn(0x55922cf942e0 msgr2=0x55922cf9a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9a8aa00640 0 client.15744 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 shutdown_connections 2024-04-26T11:39:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.831+0000 7f9aae77e740 1 -- 172.21.15.33:0/2351648324 wait complete. 2024-04-26T11:39:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.832+0000 7f9aae77e740 20 client.15744 trim_cache size 0 max 16384 2024-04-26T11:39:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 Processor -- start 2024-04-26T11:39:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 -- start start 2024-04-26T11:39:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cfae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cf986f0 0x55922cfbf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cfa2fb0 0x55922cfb7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41ada20 con 0x55922cf986f0 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41ae0f0 con 0x55922cf97dd0 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.852+0000 7f9aae77e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41ae7c0 con 0x55922cfa2fb0 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cf986f0 0x55922cfbf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cfa2fb0 0x55922cfb7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cfae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cfa2fb0 0x55922cfb7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:43144/0 (socket says 172.21.15.33:43144) 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cf986f0 0x55922cfbf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52358/0 (socket says 172.21.15.33:52358) 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa9800640 1 -- 172.21.15.33:0/2172819568 learned_addr learned my addr 172.21.15.33:0/2172819568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cfae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55392/0 (socket says 172.21.15.33:55392) 2024-04-26T11:39:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 -- 172.21.15.33:0/2172819568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cfa2fb0 msgr2=0x55922cfb7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cfa2fb0 0x55922cfb7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 -- 172.21.15.33:0/2172819568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cf986f0 msgr2=0x55922cfbf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cf986f0 0x55922cfbf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 -- 172.21.15.33:0/2172819568 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55922ccdb620 con 0x55922cf97dd0 2024-04-26T11:39:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa9800640 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cfa2fb0 0x55922cfb7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.853+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cfae710 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9a980b7d70 tx=0x7f9a980a6040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebf3460c312e27a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.854+0000 7f9a95600640 1 -- 172.21.15.33:0/2172819568 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980bb450 con 0x55922cf97dd0 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.854+0000 7f9a95600640 1 -- 172.21.15.33:0/2172819568 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a980328f0 con 0x55922cf97dd0 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.854+0000 7f9a95600640 1 -- 172.21.15.33:0/2172819568 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980bb6e0 con 0x55922cf97dd0 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.854+0000 7f9aae77e740 1 -- 172.21.15.33:0/2172819568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 msgr2=0x55922cfae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.854+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cfae710 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9a980b7d70 tx=0x7f9a980a6040 comp rx=0 tx=0).stop 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 -- 172.21.15.33:0/2172819568 shutdown_connections 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cfa2fb0 0x55922cfb7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cf986f0 0x55922cfbf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2172819568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf97dd0 0x55922cfae710 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 -- 172.21.15.33:0/2172819568 >> 172.21.15.33:0/2172819568 conn(0x55922cf942e0 msgr2=0x55922cf9b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:39:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 -- 172.21.15.33:0/2172819568 shutdown_connections 2024-04-26T11:39:56.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 -- 172.21.15.33:0/2172819568 wait complete. 2024-04-26T11:39:56.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 -- start start 2024-04-26T11:39:56.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.855+0000 7f9aae77e740 1 Processor -- start 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cfaac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aae77e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfb7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aae77e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa40734a0 con 0x55922cfa2fb0 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aae77e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa40748b0 con 0x55922cf986f0 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aae77e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4073b10 con 0x55922cf97dd0 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aae77e740 10 client.0 _ll_register_callbacks cb 0x7f9a9c0112e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55404/0 (socket says 172.21.15.33:55404) 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aa3e00640 1 -- 172.21.15.33:0/2660951852 learned_addr learned my addr 172.21.15.33:0/2660951852 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aa9800640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfb7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.856+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cfaac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aa3e00640 1 -- 172.21.15.33:0/2660951852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 msgr2=0x55922cfaac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aa3e00640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cfaac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aa3e00640 1 -- 172.21.15.33:0/2660951852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 msgr2=0x55922cfb7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aa3e00640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfb7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aa3e00640 1 -- 172.21.15.33:0/2660951852 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55922ceb7cb0 con 0x55922cf986f0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cfaac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aa3e00640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfab160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9aa40799f0 tx=0x7f9a90004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b635d95c39cdf255 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9a8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2660951852 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90020060 con 0x55922cf986f0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a980328f0 con 0x55922cf986f0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.857+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2660951852 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9702+0+0 (secure 0 0 0) 0x7f9a90012a90 con 0x55922cf986f0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2660951852 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (secure 0 0 0) 0x7f9a90027050 con 0x55922cf986f0 2024-04-26T11:39:56.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 10 client.25997 fetch_fsmap learned FSMap version 130 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=4+}) -- 0x55922d018f00 con 0x55922cf986f0 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2660951852 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a980328f0 con 0x55922cf986f0 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2660951852 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 130) ==== 7132+0+0 (secure 0 0 0) 0x7f9a9000f3c0 con 0x55922cf986f0 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 10 client.25997 fetch_fsmap finished waiting for FSMap version 130 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 10 client.25997 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 20 client.25997 populate_metadata read hostname 'smithi033' 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922d0196b0 0x55922d01bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 4 client.25997 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55922cf04e00 con 0x55922d0196b0 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.858+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922d0196b0 0x55922d01bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.859+0000 7f9aa8e00640 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922d0196b0 0x55922d01bad0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cc7eb229f1f8f3 server_cookie=84f07b477c9b8d13 in_seq=0 out_seq=0 2024-04-26T11:39:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:39:56.859+0000 7f9a8aa00640 10 client.25997 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:39:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:56 smithi194 ceph-mon[33017]: pgmap v309: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 6.1 MiB/s wr, 619 op/s 2024-04-26T11:39:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:56 smithi033 ceph-mon[27845]: pgmap v309: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 6.1 MiB/s wr, 619 op/s 2024-04-26T11:39:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:56 smithi096 ceph-mon[32753]: pgmap v309: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 6.1 MiB/s wr, 619 op/s 2024-04-26T11:39:57.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 520.0 s, 636.7 tps, lat 46.978 ms stddev 50.074 2024-04-26T11:39:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:39:58 smithi033 ceph-mon[27845]: pgmap v310: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 5.8 MiB/s wr, 571 op/s 2024-04-26T11:39:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:39:58 smithi096 ceph-mon[32753]: pgmap v310: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 5.8 MiB/s wr, 571 op/s 2024-04-26T11:39:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:39:58 smithi194 ceph-mon[33017]: pgmap v310: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 5.8 MiB/s wr, 571 op/s 2024-04-26T11:40:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:00 smithi033 ceph-mon[27845]: pgmap v311: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 6.1 MiB/s wr, 617 op/s 2024-04-26T11:40:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:00 smithi033 ceph-mon[27845]: overall HEALTH_OK 2024-04-26T11:40:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:00 smithi096 ceph-mon[32753]: pgmap v311: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 6.1 MiB/s wr, 617 op/s 2024-04-26T11:40:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:00 smithi096 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T11:40:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:00 smithi194 ceph-mon[33017]: pgmap v311: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 6.1 MiB/s wr, 617 op/s 2024-04-26T11:40:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:00 smithi194 ceph-mon[33017]: overall HEALTH_OK 2024-04-26T11:40:01.854 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:01 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:40:01.854 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:01 smithi033 systemd[1]: Stopped Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:40:01.854 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:01 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.045s CPU time. 2024-04-26T11:40:01.854 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:01 smithi033 systemd[1]: Starting Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:40:02.121 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:01 smithi033 ceph-mon[27845]: pgmap v312: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.0 MiB/s wr, 431 op/s 2024-04-26T11:40:02.122 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:01 smithi033 podman[91962]: 2024-04-26 11:40:01.754186934 +0000 UTC m=+0.034607564 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:40:02.122 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:01 smithi033 podman[91962]: 2024-04-26 11:40:01.875560705 +0000 UTC m=+0.155981390 container create 8d808b21a8e0c75436839671474b91b05c9d8c8c76e0415a5821d6476ce897cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:40:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:01 smithi096 ceph-mon[32753]: pgmap v312: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.0 MiB/s wr, 431 op/s 2024-04-26T11:40:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 525.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:40:02.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:01 smithi194 ceph-mon[33017]: pgmap v312: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.0 MiB/s wr, 431 op/s 2024-04-26T11:40:02.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:02 smithi033 podman[91962]: 2024-04-26 11:40:02.119948831 +0000 UTC m=+0.400369505 container init 8d808b21a8e0c75436839671474b91b05c9d8c8c76e0415a5821d6476ce897cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T11:40:02.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:02 smithi033 podman[91962]: 2024-04-26 11:40:02.132106976 +0000 UTC m=+0.412527650 container start 8d808b21a8e0c75436839671474b91b05c9d8c8c76e0415a5821d6476ce897cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:40:02.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mds[91980]: -- 172.21.15.33:0/2464339118 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0537efa40 con 0x55d0537c2c00 2024-04-26T11:40:02.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: starting mds.d at 2024-04-26T11:40:02.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:02 smithi033 bash[91962]: 8d808b21a8e0c75436839671474b91b05c9d8c8c76e0415a5821d6476ce897cb 2024-04-26T11:40:02.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:40:02 smithi033 systemd[1]: Started Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:40:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: daemon mds.d restarted 2024-04-26T11:40:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:40:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T11:40:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:boot 2024-04-26T11:40:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:40:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:02 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: daemon mds.d restarted 2024-04-26T11:40:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:40:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T11:40:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:boot 2024-04-26T11:40:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:40:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:02 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: daemon mds.d restarted 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:boot 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:02 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:04.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:04 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:resolve 2024-04-26T11:40:04.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:04 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:04.469 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:04 smithi033 ceph-mon[27845]: pgmap v314: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.8 MiB/s wr, 410 op/s 2024-04-26T11:40:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:04 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:resolve 2024-04-26T11:40:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:04 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:04 smithi194 ceph-mon[33017]: pgmap v314: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.8 MiB/s wr, 410 op/s 2024-04-26T11:40:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:04 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:resolve 2024-04-26T11:40:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:04 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:04 smithi096 ceph-mon[32753]: pgmap v314: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.8 MiB/s wr, 410 op/s 2024-04-26T11:40:05.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.236+0000 7f9a8aa00640 1 -- 172.21.15.33:0/2660951852 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55922cf04e00 con 0x55922d0196b0 2024-04-26T11:40:05.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.236+0000 7f9a8aa00640 10 client.25997 handle_command_reply: tid=0 2024-04-26T11:40:05.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.236+0000 7f9aae77e740 1 client.25997 shutdown 2024-04-26T11:40:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 msgr2=0x55922cfab160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfab160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9aa40799f0 tx=0x7f9a90004e40 comp rx=0 tx=0).stop 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 shutdown_connections 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55922d0196b0 0x55922d01bad0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55922cfa2fb0 0x55922cfb7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55922cf986f0 0x55922cfab160 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 --2- 172.21.15.33:0/2660951852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55922cf97dd0 0x55922cfaac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9a8aa00640 0 client.25997 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.237+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 >> 172.21.15.33:0/2660951852 conn(0x55922cf942e0 msgr2=0x55922cf99780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.238+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 shutdown_connections 2024-04-26T11:40:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.238+0000 7f9aae77e740 1 -- 172.21.15.33:0/2660951852 wait complete. 2024-04-26T11:40:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.238+0000 7f9aae77e740 20 client.25997 trim_cache size 0 max 16384 2024-04-26T11:40:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:40:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:40:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:40:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:40:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9a8c0a3e40 msgr2=0x7f9a8c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9a8c0a3e40 0x7f9a8c0a6300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0094d0 tx=0x7f9a9c009500 comp rx=0 tx=0).stop 2024-04-26T11:40:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa4076e60 msgr2=0x7f9aa419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa4076e60 0x7f9aa419f670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009780 tx=0x7f9a98032d50 comp rx=0 tx=0).stop 2024-04-26T11:40:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 shutdown_connections 2024-04-26T11:40:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f9a8c0a3e40 0x7f9a8c0a6300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa41137b0 0x7f9aa41a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9aa4077780 0x7f9aa419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 --2- 172.21.15.33:0/3352158209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa4076e60 0x7f9aa419f670 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 >> 172.21.15.33:0/3352158209 conn(0x7f9aa4100620 msgr2=0x7f9aa4102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 shutdown_connections 2024-04-26T11:40:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:05.240+0000 7f9aab800640 1 -- 172.21.15.33:0/3352158209 wait complete. 2024-04-26T11:40:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:05 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:40:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:05 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:reconnect 2024-04-26T11:40:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:05 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:05 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:40:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:05 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:40:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:05 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:reconnect 2024-04-26T11:40:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:05 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:05 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:40:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:05 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:40:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:05 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:reconnect 2024-04-26T11:40:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:05 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:05 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:40:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:06 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:rejoin 2024-04-26T11:40:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:06 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:06 smithi194 ceph-mon[33017]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:40:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:06 smithi194 ceph-mon[33017]: pgmap v315: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 982 KiB/s wr, 132 op/s 2024-04-26T11:40:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:06 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:rejoin 2024-04-26T11:40:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:06 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:06 smithi033 ceph-mon[27845]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:40:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:06 smithi033 ceph-mon[27845]: pgmap v315: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 982 KiB/s wr, 132 op/s 2024-04-26T11:40:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:06 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:rejoin 2024-04-26T11:40:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:06 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:06 smithi096 ceph-mon[32753]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:40:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:06 smithi096 ceph-mon[32753]: pgmap v315: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 982 KiB/s wr, 132 op/s 2024-04-26T11:40:07.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 530.0 s, 162.4 tps, lat 1666.311 ms stddev 4218.896 2024-04-26T11:40:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:07 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:07 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:40:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:07 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:active 2024-04-26T11:40:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:07 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:07 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:07 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:07 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:40:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:07 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:active 2024-04-26T11:40:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:07 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:07 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:07 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:07 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:40:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:07 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1109850524,v1:172.21.15.96:6833/1109850524] up:active 2024-04-26T11:40:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:07 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:07 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:08 smithi194 ceph-mon[33017]: pgmap v316: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 982 KiB/s wr, 132 op/s 2024-04-26T11:40:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:08 smithi033 ceph-mon[27845]: pgmap v316: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 982 KiB/s wr, 132 op/s 2024-04-26T11:40:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:08 smithi096 ceph-mon[32753]: pgmap v316: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 982 KiB/s wr, 132 op/s 2024-04-26T11:40:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:10 smithi194 ceph-mon[33017]: pgmap v317: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 6.8 MiB/s wr, 738 op/s 2024-04-26T11:40:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:10 smithi033 ceph-mon[27845]: pgmap v317: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 6.8 MiB/s wr, 738 op/s 2024-04-26T11:40:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:10 smithi096 ceph-mon[32753]: pgmap v317: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 6.8 MiB/s wr, 738 op/s 2024-04-26T11:40:11.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5e094e700 time 2024-04-26T11:40:11.264138+0000 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5eaf773dd] 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: clone() 2024-04-26T11:40:11.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.263+0000 7fa5e094e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5e094e700 time 2024-04-26T11:40:11.264138+0000 2024-04-26T11:40:11.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:11.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5eaf773dd] 2024-04-26T11:40:11.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: clone() 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: *** Caught signal (Aborted) ** 2024-04-26T11:40:11.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: in thread 7fa5e094e700 thread_name:ms_dispatch 2024-04-26T11:40:11.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5e9cffd20] 2024-04-26T11:40:11.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: gsignal() 2024-04-26T11:40:11.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: abort() 2024-04-26T11:40:11.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5eaf77437] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 16: clone() 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.265+0000 7fa5e094e700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: in thread 7fa5e094e700 thread_name:ms_dispatch 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5e9cffd20] 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: gsignal() 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: abort() 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5eaf77437] 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 16: clone() 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: -1> 2024-04-26T11:40:11.263+0000 7fa5e094e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5e094e700 time 2024-04-26T11:40:11.264138+0000 2024-04-26T11:40:11.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5eaf773dd] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: clone() 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 0> 2024-04-26T11:40:11.265+0000 7fa5e094e700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: in thread 7fa5e094e700 thread_name:ms_dispatch 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5e9cffd20] 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: gsignal() 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: abort() 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5eaf77437] 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 16: clone() 2024-04-26T11:40:11.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: -1> 2024-04-26T11:40:11.263+0000 7fa5e094e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5e094e700 time 2024-04-26T11:40:11.264138+0000 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5eaf773dd] 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.697 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.697 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.697 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.697 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.697 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.697 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.697 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.698 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.698 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.698 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: clone() 2024-04-26T11:40:11.698 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.698 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 0> 2024-04-26T11:40:11.265+0000 7fa5e094e700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:11.698 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: in thread 7fa5e094e700 thread_name:ms_dispatch 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5e9cffd20] 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2: gsignal() 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 3: abort() 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5eaf77437] 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5eaf775a3] 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558339858822] 2024-04-26T11:40:11.699 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5583397c8598] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5583397cac17] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5583396c3d0b] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5583396c42ac] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5583396acfad] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5eb20be20] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5eb208fdf] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5eb2dbfc1] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5e9cf51ca] 2024-04-26T11:40:11.700 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 16: clone() 2024-04-26T11:40:11.701 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:11.701 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[68974]: 2024-04-26T11:40:11.701 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:11 smithi096 podman[69157]: 2024-04-26 11:40:11.352536213 +0000 UTC m=+0.047176384 container died f0b582d27651368009c69b4d9f2310db93372adcfe570590b30f262d1ef75ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T11:40:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 535.0 s, 2026.3 tps, lat 49.137 ms stddev 67.116 2024-04-26T11:40:12.514 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:12 smithi096 podman[69157]: 2024-04-26 11:40:12.277268715 +0000 UTC m=+0.971908875 container remove f0b582d27651368009c69b4d9f2310db93372adcfe570590b30f262d1ef75ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T11:40:12.514 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:12 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:40:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:12 smithi194 ceph-mon[33017]: pgmap v318: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-26T11:40:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:12 smithi194 ceph-mon[33017]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T11:40:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:12 smithi033 ceph-mon[27845]: pgmap v318: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-26T11:40:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:12 smithi033 ceph-mon[27845]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T11:40:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:12 smithi096 ceph-mon[32753]: pgmap v318: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-26T11:40:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:12 smithi096 ceph-mon[32753]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T11:40:12.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:12 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-26T11:40:12.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:12 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.061s CPU time. 2024-04-26T11:40:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:14 smithi194 ceph-mon[33017]: pgmap v320: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-26T11:40:14.905 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:14 smithi033 ceph-mon[27845]: pgmap v320: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-26T11:40:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:14 smithi096 ceph-mon[32753]: pgmap v320: 97 pgs: 97 active+clean; 3.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-26T11:40:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:16 smithi194 ceph-mon[33017]: pgmap v321: 97 pgs: 97 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 13 MiB/s wr, 1.31k op/s 2024-04-26T11:40:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:16 smithi194 ceph-mon[33017]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T11:40:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:16 smithi033 ceph-mon[27845]: pgmap v321: 97 pgs: 97 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 13 MiB/s wr, 1.31k op/s 2024-04-26T11:40:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:16 smithi033 ceph-mon[27845]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T11:40:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:16 smithi096 ceph-mon[32753]: pgmap v321: 97 pgs: 97 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 13 MiB/s wr, 1.31k op/s 2024-04-26T11:40:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:16 smithi096 ceph-mon[32753]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T11:40:17.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 540.0 s, 2.8 tps, lat 76.878 ms stddev 22.577 2024-04-26T11:40:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 -- 172.21.15.33:0/1639425343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff290077780 msgr2=0x7ff290113280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 --2- 172.21.15.33:0/1639425343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff290077780 0x7ff290113280 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff278009900 tx=0x7ff27802a3f0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 -- 172.21.15.33:0/1639425343 shutdown_connections 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 --2- 172.21.15.33:0/1639425343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2901137c0 0x7ff290115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 --2- 172.21.15.33:0/1639425343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff290077780 0x7ff290113280 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 --2- 172.21.15.33:0/1639425343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290076e60 0x7ff290077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 -- 172.21.15.33:0/1639425343 >> 172.21.15.33:0/1639425343 conn(0x7ff290100630 msgr2=0x7ff290102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 -- 172.21.15.33:0/1639425343 shutdown_connections 2024-04-26T11:40:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.416+0000 7ff297000640 1 -- 172.21.15.33:0/1639425343 wait complete. 2024-04-26T11:40:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.417+0000 7ff297000640 1 Processor -- start 2024-04-26T11:40:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff297000640 1 -- start start 2024-04-26T11:40:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff297000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff290076e60 0x7ff29019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff297000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290077780 0x7ff29019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff297000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2901137c0 0x7ff2901a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff297000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff290119210 con 0x7ff2901137c0 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff297000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff290119090 con 0x7ff290076e60 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff297000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff290119390 con 0x7ff290077780 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff287400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290077780 0x7ff29019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff28fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff290076e60 0x7ff29019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff287400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290077780 0x7ff29019fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52380/0 (socket says 172.21.15.33:52380) 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff287400640 1 -- 172.21.15.33:0/2005198587 learned_addr learned my addr 172.21.15.33:0/2005198587 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.418+0000 7ff295000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2901137c0 0x7ff2901a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff287400640 1 -- 172.21.15.33:0/2005198587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff290076e60 msgr2=0x7ff29019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff287400640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff290076e60 0x7ff29019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff287400640 1 -- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2901137c0 msgr2=0x7ff2901a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff287400640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2901137c0 0x7ff2901a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff287400640 1 -- 172.21.15.33:0/2005198587 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2901a46a0 con 0x7ff290077780 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff28fe00640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff290076e60 0x7ff29019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff287400640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290077780 0x7ff29019fbf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff27802a9e0 tx=0x7ff278003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff28d600640 1 -- 172.21.15.33:0/2005198587 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27803f070 con 0x7ff290077780 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2901a4930 con 0x7ff290077780 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff28d600640 1 -- 172.21.15.33:0/2005198587 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff27803bb20 con 0x7ff290077780 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.419+0000 7ff28d600640 1 -- 172.21.15.33:0/2005198587 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278035030 con 0x7ff290077780 2024-04-26T11:40:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.420+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2901ac220 con 0x7ff290077780 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 Processor -- start 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 -- start start 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff28d600640 1 -- 172.21.15.33:0/2005198587 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff278042070 con 0x7ff290077780 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4af6f0 0x55623b4b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4bc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55623b414380 con 0x55623b4b9fb0 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55623b3b0c30 con 0x55623b4af6f0 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff299e4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55623b410340 con 0x55623b4aedd0 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff287400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4bc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.423+0000 7ff28fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4af6f0 0x55623b4b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4af6f0 0x55623b4b9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58730/0 (socket says 172.21.15.33:58730) 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 -- 172.21.15.33:0/2793865776 learned_addr learned my addr 172.21.15.33:0/2793865776 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff295000640 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 -- 172.21.15.33:0/2793865776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 msgr2=0x55623b4af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 -- 172.21.15.33:0/2793865776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 msgr2=0x55623b4bc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4bc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 -- 172.21.15.33:0/2793865776 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55623b3d2940 con 0x55623b4af6f0 2024-04-26T11:40:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff28fe00640 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4af6f0 0x55623b4b9a70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff280011540 tx=0x7ff280011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b75bb61ceebbe8f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.424+0000 7ff285600640 1 -- 172.21.15.33:0/2793865776 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28000f640 con 0x55623b4af6f0 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff285600640 1 -- 172.21.15.33:0/2793865776 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff28000fc90 con 0x55623b4af6f0 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff285600640 1 -- 172.21.15.33:0/2793865776 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280019c80 con 0x55623b4af6f0 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff28d600640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff2740a3dc0 0x7ff2740a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff28d600640 1 -- 172.21.15.33:0/2005198587 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9832+0+0 (secure 0 0 0) 0x7ff2780f13d0 con 0x7ff290077780 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff295000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff2740a3dc0 0x7ff2740a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff299e4f740 1 -- 172.21.15.33:0/2793865776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4af6f0 msgr2=0x55623b4b9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4af6f0 0x55623b4b9a70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff280011540 tx=0x7ff280011a00 comp rx=0 tx=0).stop 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff299e4f740 1 -- 172.21.15.33:0/2793865776 shutdown_connections 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4bc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4af6f0 0x55623b4b9a70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2793865776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.425+0000 7ff299e4f740 1 -- 172.21.15.33:0/2793865776 >> 172.21.15.33:0/2793865776 conn(0x55623b4ab2e0 msgr2=0x55623b4ab6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.426+0000 7ff299e4f740 1 -- 172.21.15.33:0/2793865776 shutdown_connections 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.426+0000 7ff299e4f740 1 -- 172.21.15.33:0/2793865776 wait complete. 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.426+0000 7ff295000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff2740a3dc0 0x7ff2740a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff29006be30 tx=0x7ff288009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.426+0000 7ff299e4f740 1 -- start start 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.426+0000 7ff299e4f740 1 Processor -- start 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4af6f0 0x55623b4b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff299e4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55623b4bcbe0 con 0x55623b4af6f0 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff299e4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55623b39bc60 con 0x55623b4b9fb0 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff299e4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55623b4bcd60 con 0x55623b4aedd0 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff299e4f740 10 client.0 _ll_register_callbacks cb 0x55623b3c7e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff287400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4af6f0 0x55623b4b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff28fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff28fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4b6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:52402/0 (socket says 172.21.15.33:52402) 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff28fe00640 1 -- 172.21.15.33:0/618697151 learned_addr learned my addr 172.21.15.33:0/618697151 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff287400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4af6f0 0x55623b4b7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45614/0 (socket says 172.21.15.33:45614) 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff295000640 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff28fe00640 1 -- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 msgr2=0x55623b4b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff28fe00640 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff28fe00640 1 -- 172.21.15.33:0/618697151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4af6f0 msgr2=0x55623b4b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.427+0000 7ff28fe00640 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4af6f0 0x55623b4b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.428+0000 7ff28fe00640 1 -- 172.21.15.33:0/618697151 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55623b3aa510 con 0x55623b4aedd0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.428+0000 7ff287400640 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4af6f0 0x55623b4b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.428+0000 7ff28fe00640 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4b6f30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff280000c00 tx=0x7ff28000f870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a95f5ab9c0ffe10b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.428+0000 7ff27e000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.428+0000 7ff27e000640 1 -- 172.21.15.33:0/618697151 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280026020 con 0x55623b4aedd0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.428+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff28000fc90 con 0x55623b4aedd0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.428+0000 7ff27e000640 1 -- 172.21.15.33:0/618697151 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9832+0+0 (secure 0 0 0) 0x7ff280019980 con 0x55623b4aedd0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff27e000640 1 -- 172.21.15.33:0/618697151 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7ff280031050 con 0x55623b4aedd0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 10 client.25930 fetch_fsmap learned FSMap version 137 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x55623b126210 con 0x55623b4aedd0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff27e000640 1 -- 172.21.15.33:0/618697151 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff28000fc90 con 0x55623b4aedd0 2024-04-26T11:40:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff27e000640 1 -- 172.21.15.33:0/618697151 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 137) ==== 7132+0+0 (secure 0 0 0) 0x7ff2800046c0 con 0x55623b4aedd0 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 10 client.25930 fetch_fsmap finished waiting for FSMap version 137 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 10 client.25930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 20 client.25930 populate_metadata read hostname 'smithi033' 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b4c9c80 0x55623b4cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 4 client.25930 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.429+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55623b421f90 con 0x55623b4c9c80 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.430+0000 7ff287400640 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b4c9c80 0x55623b4cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.430+0000 7ff287400640 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b4c9c80 0x55623b4cc0b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c32e67a9aa790c31 server_cookie=8003accb627e2288 in_seq=0 out_seq=0 2024-04-26T11:40:18.436 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.430+0000 7ff27e000640 10 client.25930 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.431+0000 7ff27e000640 1 -- 172.21.15.33:0/618697151 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55623b421f90 con 0x55623b4c9c80 2024-04-26T11:40:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.431+0000 7ff27e000640 10 client.25930 handle_command_reply: tid=0 2024-04-26T11:40:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.431+0000 7ff299e4f740 1 client.25930 shutdown 2024-04-26T11:40:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.431+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 msgr2=0x55623b4b6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.431+0000 7ff299e4f740 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4b6f30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff280000c00 tx=0x7ff28000f870 comp rx=0 tx=0).stop 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 shutdown_connections 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b4c9c80 0x55623b4cc0b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4af6f0 0x55623b4b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 --2- 172.21.15.33:0/618697151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4aedd0 0x55623b4b6f30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff27e000640 0 client.25930 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 >> 172.21.15.33:0/618697151 conn(0x55623b4ab2e0 msgr2=0x55623b4b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 shutdown_connections 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 1 -- 172.21.15.33:0/618697151 wait complete. 2024-04-26T11:40:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.432+0000 7ff299e4f740 20 client.25930 trim_cache size 0 max 16384 2024-04-26T11:40:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.448+0000 7ff299e4f740 1 Processor -- start 2024-04-26T11:40:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.448+0000 7ff299e4f740 1 -- start start 2024-04-26T11:40:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4aedd0 0x55623b4b9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff299e4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901b6320 con 0x55623b4aedd0 2024-04-26T11:40:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff299e4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901b6a20 con 0x55623b4b9fb0 2024-04-26T11:40:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff299e4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901b70f0 con 0x55623b4af6f0 2024-04-26T11:40:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff295000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff28fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4aedd0 0x55623b4b9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff287400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff295000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58758/0 (socket says 172.21.15.33:58758) 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.449+0000 7ff295000640 1 -- 172.21.15.33:0/3936910633 learned_addr learned my addr 172.21.15.33:0/3936910633 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff295000640 1 -- 172.21.15.33:0/3936910633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 msgr2=0x55623b4c5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff295000640 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4c5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff295000640 1 -- 172.21.15.33:0/3936910633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4aedd0 msgr2=0x55623b4b9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff295000640 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4aedd0 0x55623b4b9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff295000640 1 -- 172.21.15.33:0/3936910633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55623b1f2620 con 0x55623b4b9fb0 2024-04-26T11:40:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff28fe00640 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4aedd0 0x55623b4b9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff295000640 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b0e90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff288003ea0 tx=0x7ff28801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afdf0ccf7a0aa412 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff284c00640 1 -- 172.21.15.33:0/3936910633 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28800f540 con 0x55623b4b9fb0 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff284c00640 1 -- 172.21.15.33:0/3936910633 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff28800fb90 con 0x55623b4b9fb0 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.450+0000 7ff284c00640 1 -- 172.21.15.33:0/3936910633 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28801ec50 con 0x55623b4b9fb0 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 -- 172.21.15.33:0/3936910633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 msgr2=0x55623b4b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b0e90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff288003ea0 tx=0x7ff28801a000 comp rx=0 tx=0).stop 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 -- 172.21.15.33:0/3936910633 shutdown_connections 2024-04-26T11:40:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4c5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4b9fb0 0x55623b4b0e90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 --2- 172.21.15.33:0/3936910633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4aedd0 0x55623b4b9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 -- 172.21.15.33:0/3936910633 >> 172.21.15.33:0/3936910633 conn(0x55623b4ab2e0 msgr2=0x55623b4b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 -- 172.21.15.33:0/3936910633 shutdown_connections 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.451+0000 7ff299e4f740 1 -- 172.21.15.33:0/3936910633 wait complete. 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 -- start start 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 Processor -- start 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4aedd0 0x55623b4c2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901b3620 con 0x55623b4b9fb0 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901b4a10 con 0x55623b4aedd0 2024-04-26T11:40:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901b3c70 con 0x55623b4af6f0 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.452+0000 7ff299e4f740 10 client.0 _ll_register_callbacks cb 0x7ff2780f4d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff295000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff28fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4aedd0 0x55623b4c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff295000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45638/0 (socket says 172.21.15.33:45638) 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff287400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff295000640 1 -- 172.21.15.33:0/2649465136 learned_addr learned my addr 172.21.15.33:0/2649465136 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff28fe00640 1 -- 172.21.15.33:0/2649465136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 msgr2=0x55623b4b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff28fe00640 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff28fe00640 1 -- 172.21.15.33:0/2649465136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 msgr2=0x55623b4b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff28fe00640 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff28fe00640 1 -- 172.21.15.33:0/2649465136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55623b3cecb0 con 0x55623b4aedd0 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff295000640 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:40:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff28fe00640 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4aedd0 0x55623b4c2380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff280004a90 tx=0x7ff280005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13bcb8ef473059b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.453+0000 7ff27e000640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff27e000640 1 -- 172.21.15.33:0/2649465136 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280026070 con 0x55623b4aedd0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff27e000640 1 -- 172.21.15.33:0/2649465136 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9832+0+0 (secure 0 0 0) 0x7ff280019b10 con 0x55623b4aedd0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff28800fb90 con 0x55623b4aedd0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff27e000640 1 -- 172.21.15.33:0/2649465136 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7ff28000fe80 con 0x55623b4aedd0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff299e4f740 10 client.26021 fetch_fsmap learned FSMap version 137 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x55623b530020 con 0x55623b4aedd0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff27e000640 1 -- 172.21.15.33:0/2649465136 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff28800fb90 con 0x55623b4aedd0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff27e000640 1 -- 172.21.15.33:0/2649465136 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 137) ==== 7132+0+0 (secure 0 0 0) 0x7ff2800046c0 con 0x55623b4aedd0 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff299e4f740 10 client.26021 fetch_fsmap finished waiting for FSMap version 137 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff299e4f740 10 client.26021 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff299e4f740 20 client.26021 populate_metadata read hostname 'smithi033' 2024-04-26T11:40:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.454+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b5305d0 0x55623b5329c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.455+0000 7ff299e4f740 4 client.26021 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:40:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.455+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55623b41be00 con 0x55623b5305d0 2024-04-26T11:40:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.455+0000 7ff287400640 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b5305d0 0x55623b5329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.455+0000 7ff287400640 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b5305d0 0x55623b5329c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e867abf8bafb9fdb server_cookie=13902cc46e95d405 in_seq=0 out_seq=0 2024-04-26T11:40:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:18.455+0000 7ff27e000640 10 client.26021 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:18 smithi194 ceph-mon[33017]: pgmap v323: 97 pgs: 97 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-04-26T11:40:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:18 smithi033 ceph-mon[27845]: pgmap v323: 97 pgs: 97 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-04-26T11:40:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:18 smithi096 ceph-mon[32753]: pgmap v323: 97 pgs: 97 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-04-26T11:40:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:20 smithi194 ceph-mon[33017]: pgmap v324: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.8 MiB/s wr, 409 op/s 2024-04-26T11:40:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:20 smithi033 ceph-mon[27845]: pgmap v324: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.8 MiB/s wr, 409 op/s 2024-04-26T11:40:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:20 smithi096 ceph-mon[32753]: pgmap v324: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.8 MiB/s wr, 409 op/s 2024-04-26T11:40:22.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 545.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:40:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:22 smithi194 ceph-mon[33017]: pgmap v325: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.4 MiB/s wr, 365 op/s 2024-04-26T11:40:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:22 smithi033 ceph-mon[27845]: pgmap v325: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.4 MiB/s wr, 365 op/s 2024-04-26T11:40:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:22 smithi096 ceph-mon[32753]: pgmap v325: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.4 MiB/s wr, 365 op/s 2024-04-26T11:40:22.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:22 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:40:22.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:22 smithi096 systemd[1]: Stopped Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:40:22.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:22 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.061s CPU time. 2024-04-26T11:40:22.933 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:22 smithi096 systemd[1]: Starting Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:40:23.306 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:22 smithi096 podman[69249]: 2024-04-26 11:40:22.840313545 +0000 UTC m=+0.018197378 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:40:23.306 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:23 smithi096 podman[69249]: 2024-04-26 11:40:23.035729593 +0000 UTC m=+0.213613417 container create 57a57fe22001788e6a8908507078fa762e06f8d9e538cd45c6685f332f336b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T11:40:23.306 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:23 smithi096 podman[69249]: 2024-04-26 11:40:23.30554805 +0000 UTC m=+0.483431876 container init 57a57fe22001788e6a8908507078fa762e06f8d9e538cd45c6685f332f336b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: daemon mds.k restarted 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:boot 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:23.670 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:23 smithi096 podman[69249]: 2024-04-26 11:40:23.308241389 +0000 UTC m=+0.486125213 container start 57a57fe22001788e6a8908507078fa762e06f8d9e538cd45c6685f332f336b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-26T11:40:23.671 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-mds[69268]: -- 172.21.15.96:0/207518127 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b6b5cda40 con 0x564b6b5a0800 2024-04-26T11:40:23.671 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:23 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: starting mds.k at 2024-04-26T11:40:23.671 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:23 smithi096 bash[69249]: 57a57fe22001788e6a8908507078fa762e06f8d9e538cd45c6685f332f336b56 2024-04-26T11:40:23.671 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:23 smithi096 systemd[1]: Started Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: daemon mds.k restarted 2024-04-26T11:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T11:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:boot 2024-04-26T11:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:40:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:23 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: daemon mds.k restarted 2024-04-26T11:40:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:40:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T11:40:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:boot 2024-04-26T11:40:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:40:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:23 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:24 smithi033 ceph-mon[27845]: pgmap v327: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T11:40:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:24 smithi194 ceph-mon[33017]: pgmap v327: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T11:40:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:24 smithi096 ceph-mon[32753]: pgmap v327: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T11:40:25.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:resolve 2024-04-26T11:40:25.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:25 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:resolve 2024-04-26T11:40:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:25 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:resolve 2024-04-26T11:40:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:25 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.587+0000 7ff27e000640 1 -- 172.21.15.33:0/2649465136 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55623b41be00 con 0x55623b5305d0 2024-04-26T11:40:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.587+0000 7ff27e000640 10 client.26021 handle_command_reply: tid=0 2024-04-26T11:40:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.588+0000 7ff299e4f740 1 client.26021 shutdown 2024-04-26T11:40:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.588+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4aedd0 msgr2=0x55623b4c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.588+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4aedd0 0x55623b4c2380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff280004a90 tx=0x7ff280005f10 comp rx=0 tx=0).stop 2024-04-26T11:40:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 shutdown_connections 2024-04-26T11:40:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55623b4af6f0 0x55623b4b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55623b5305d0 0x55623b5329c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55623b4b9fb0 0x55623b4b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 --2- 172.21.15.33:0/2649465136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55623b4aedd0 0x55623b4c2380 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff27e000640 0 client.26021 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 >> 172.21.15.33:0/2649465136 conn(0x55623b4ab2e0 msgr2=0x55623b4b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 shutdown_connections 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 1 -- 172.21.15.33:0/2649465136 wait complete. 2024-04-26T11:40:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.589+0000 7ff299e4f740 20 client.26021 trim_cache size 0 max 16384 2024-04-26T11:40:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:40:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:40:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:40:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:40:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.590+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff2740a3dc0 msgr2=0x7ff2740a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.590+0000 7ff297000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff2740a3dc0 0x7ff2740a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff29006be30 tx=0x7ff288009500 comp rx=0 tx=0).stop 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.590+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290077780 msgr2=0x7ff29019fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.590+0000 7ff297000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290077780 0x7ff29019fbf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff27802a9e0 tx=0x7ff278003c10 comp rx=0 tx=0).stop 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 shutdown_connections 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7ff2740a3dc0 0x7ff2740a6280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff290077780 0x7ff29019fbf0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff2901137c0 0x7ff2901a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 --2- 172.21.15.33:0/2005198587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff290076e60 0x7ff29019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 >> 172.21.15.33:0/2005198587 conn(0x7ff290100630 msgr2=0x7ff290114da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 shutdown_connections 2024-04-26T11:40:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:26.591+0000 7ff297000640 1 -- 172.21.15.33:0/2005198587 wait complete. 2024-04-26T11:40:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:26 smithi194 ceph-mon[33017]: pgmap v328: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-26T11:40:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:26 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:40:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:26 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:reconnect 2024-04-26T11:40:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:26 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:26 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:40:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:26 smithi033 ceph-mon[27845]: pgmap v328: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-26T11:40:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:26 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:40:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:26 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:reconnect 2024-04-26T11:40:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:26 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:26 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:40:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:26 smithi096 ceph-mon[32753]: pgmap v328: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-26T11:40:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:26 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:40:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:26 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:reconnect 2024-04-26T11:40:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:26 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:26 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:40:27.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 550.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:40:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:27 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:rejoin 2024-04-26T11:40:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:27 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:27 smithi194 ceph-mon[33017]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:40:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:27 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:rejoin 2024-04-26T11:40:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:27 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:27 smithi033 ceph-mon[27845]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:40:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:27 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:rejoin 2024-04-26T11:40:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:27 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:27 smithi096 ceph-mon[32753]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:40:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:28 smithi194 ceph-mon[33017]: pgmap v329: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-04-26T11:40:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:28 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:28 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:40:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:28 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:active 2024-04-26T11:40:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:28 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:28 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:28 smithi033 ceph-mon[27845]: pgmap v329: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-04-26T11:40:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:28 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:28 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:40:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:28 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:active 2024-04-26T11:40:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:28 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:28 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:28 smithi096 ceph-mon[32753]: pgmap v329: 97 pgs: 97 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-04-26T11:40:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:28 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:28 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:40:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:28 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3159968353,v1:172.21.15.194:6833/3159968353] up:active 2024-04-26T11:40:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:28 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:28 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:30 smithi194 ceph-mon[33017]: pgmap v330: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-04-26T11:40:30.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:30 smithi033 ceph-mon[27845]: pgmap v330: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-04-26T11:40:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:30 smithi096 ceph-mon[32753]: pgmap v330: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-04-26T11:40:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 555.0 s, 1763.5 tps, lat 226.633 ms stddev 1641.162 2024-04-26T11:40:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-mon[33017]: pgmap v331: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-04-26T11:40:32.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa0ad482700 time 2024-04-26T11:40:32.601253+0000 2024-04-26T11:40:32.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:32.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0b7aab3dd] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.855 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: clone() 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.602+0000 7fa0ad482700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa0ad482700 time 2024-04-26T11:40:32.601253+0000 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0b7aab3dd] 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.856 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: clone() 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: *** Caught signal (Aborted) ** 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: in thread 7fa0ad482700 thread_name:ms_dispatch 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0b6833d20] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: gsignal() 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: abort() 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0b7aab437] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.857 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.858 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.858 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.858 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 16: clone() 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.603+0000 7fa0ad482700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: in thread 7fa0ad482700 thread_name:ms_dispatch 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0b6833d20] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: gsignal() 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: abort() 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0b7aab437] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 16: clone() 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: -1> 2024-04-26T11:40:32.602+0000 7fa0ad482700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa0ad482700 time 2024-04-26T11:40:32.601253+0000 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0b7aab3dd] 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.861 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: clone() 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 0> 2024-04-26T11:40:32.603+0000 7fa0ad482700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: in thread 7fa0ad482700 thread_name:ms_dispatch 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.862 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0b6833d20] 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: gsignal() 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: abort() 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0b7aab437] 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 16: clone() 2024-04-26T11:40:32.864 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: -4> 2024-04-26T11:40:32.602+0000 7fa0ad482700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa0ad482700 time 2024-04-26T11:40:32.601253+0000 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0b7aab3dd] 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.865 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.866 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.867 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.867 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: clone() 2024-04-26T11:40:32.867 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.867 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: -3> 2024-04-26T11:40:32.603+0000 7fa0ad482700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:32.867 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: in thread 7fa0ad482700 thread_name:ms_dispatch 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0b6833d20] 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2: gsignal() 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 3: abort() 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0b7aab437] 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa0b7aab5a3] 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5560f751d822] 2024-04-26T11:40:32.868 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5560f748d598] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5560f748fc17] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5560f7388d0b] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5560f73892ac] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5560f7371fad] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa0b7d3fe20] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 13: (DispatchQueue::entry()+0x51f) [0x7fa0b7d3cfdf] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0b7e0ffc1] 2024-04-26T11:40:32.869 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0b68291ca] 2024-04-26T11:40:32.870 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 16: clone() 2024-04-26T11:40:32.870 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:32.870 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[63510]: 2024-04-26T11:40:32.870 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:32 smithi194 podman[69643]: 2024-04-26 11:40:32.664469462 +0000 UTC m=+0.021133302 container died e1dea43708931cbcdb48f9e9a27e5c523b175729ee602345e9995f87e84497b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T11:40:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:32 smithi033 ceph-mon[27845]: pgmap v331: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-04-26T11:40:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:32 smithi096 ceph-mon[32753]: pgmap v331: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-04-26T11:40:33.669 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:33 smithi194 podman[69643]: 2024-04-26 11:40:33.408544265 +0000 UTC m=+0.765208098 container remove e1dea43708931cbcdb48f9e9a27e5c523b175729ee602345e9995f87e84497b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-26T11:40:33.669 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:33 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:40:34.104 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:33 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-26T11:40:34.104 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:33 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.341s CPU time. 2024-04-26T11:40:34.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:34 smithi033 ceph-mon[27845]: pgmap v332: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-04-26T11:40:34.928 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:34 smithi194 ceph-mon[33017]: pgmap v332: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-04-26T11:40:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:34 smithi096 ceph-mon[32753]: pgmap v332: 97 pgs: 97 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-04-26T11:40:36.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:36 smithi033 ceph-mon[27845]: pgmap v333: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 9.0 MiB/s wr, 973 op/s 2024-04-26T11:40:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:36 smithi096 ceph-mon[32753]: pgmap v333: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 9.0 MiB/s wr, 973 op/s 2024-04-26T11:40:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:36 smithi194 ceph-mon[33017]: pgmap v333: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 9.0 MiB/s wr, 973 op/s 2024-04-26T11:40:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 560.0 s, 391.1 tps, lat 49.876 ms stddev 60.780 2024-04-26T11:40:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:38 smithi033 ceph-mon[27845]: pgmap v334: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 8.5 MiB/s wr, 912 op/s 2024-04-26T11:40:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:38 smithi096 ceph-mon[32753]: pgmap v334: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 8.5 MiB/s wr, 912 op/s 2024-04-26T11:40:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:38 smithi194 ceph-mon[33017]: pgmap v334: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 8.5 MiB/s wr, 912 op/s 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 -- 172.21.15.33:0/4041090676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 msgr2=0x7fafd410c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4041090676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 0x7fafd410c6f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009900 tx=0x7fafbc02a480 comp rx=0 tx=0).stop 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 -- 172.21.15.33:0/4041090676 shutdown_connections 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4041090676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 0x7fafd410c6f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4041090676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafd410cd10 0x7fafd410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4041090676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 0x7fafd4103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 -- 172.21.15.33:0/4041090676 >> 172.21.15.33:0/4041090676 conn(0x7fafd40fd150 msgr2=0x7fafd40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 -- 172.21.15.33:0/4041090676 shutdown_connections 2024-04-26T11:40:39.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.744+0000 7fafd3e00640 1 -- 172.21.15.33:0/4041090676 wait complete. 2024-04-26T11:40:39.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.745+0000 7fafd3e00640 1 Processor -- start 2024-04-26T11:40:39.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.745+0000 7fafd3e00640 1 -- start start 2024-04-26T11:40:39.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 0x7fafd4198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 0x7fafd41994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafd410cd10 0x7fafd419d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd3e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41129c0 con 0x7fafd41038f0 2024-04-26T11:40:39.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd3e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fafd4112840 con 0x7fafd410cd10 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fafd4112b40 con 0x7fafd4104210 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 0x7fafd4198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 0x7fafd4198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53280/0 (socket says 172.21.15.33:53280) 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafd2a00640 1 -- 172.21.15.33:0/4228582948 learned_addr learned my addr 172.21.15.33:0/4228582948 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.746+0000 7fafcbe00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 0x7fafd41994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd3400640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafd410cd10 0x7fafd419d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd2a00640 1 -- 172.21.15.33:0/4228582948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 msgr2=0x7fafd41994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd2a00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 0x7fafd41994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd2a00640 1 -- 172.21.15.33:0/4228582948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafd410cd10 msgr2=0x7fafd419d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd2a00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafd410cd10 0x7fafd419d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd2a00640 1 -- 172.21.15.33:0/4228582948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd419df90 con 0x7fafd41038f0 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd2a00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 0x7fafd4198fa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9f0 tx=0x7fafc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:39.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafcb400640 1 -- 172.21.15.33:0/4228582948 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc400cd80 con 0x7fafd41038f0 2024-04-26T11:40:39.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd419e280 con 0x7fafd41038f0 2024-04-26T11:40:39.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafcb400640 1 -- 172.21.15.33:0/4228582948 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc40108e0 con 0x7fafd41038f0 2024-04-26T11:40:39.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafcb400640 1 -- 172.21.15.33:0/4228582948 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4018980 con 0x7fafd41038f0 2024-04-26T11:40:39.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.747+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd41a5b10 con 0x7fafd41038f0 2024-04-26T11:40:39.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 Processor -- start 2024-04-26T11:40:39.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafcb400640 1 -- 172.21.15.33:0/4228582948 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fafc4010430 con 0x7fafd41038f0 2024-04-26T11:40:39.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 -- start start 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55857f457380 con 0x55857f4f1fe0 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55857f3f3c30 con 0x55857f4fd1c0 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.751+0000 7fafdc670740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55857f453340 con 0x55857f4f2900 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafd3400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4ff5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36886/0 (socket says 172.21.15.33:36886) 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 -- 172.21.15.33:0/645277858 learned_addr learned my addr 172.21.15.33:0/645277858 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafd2a00640 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 -- 172.21.15.33:0/645277858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 msgr2=0x55857f4fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 -- 172.21.15.33:0/645277858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 msgr2=0x55857f4f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafcbe00640 1 -- 172.21.15.33:0/645277858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55857f415940 con 0x55857f4fd1c0 2024-04-26T11:40:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.752+0000 7fafd3400640 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.753+0000 7fafcbe00640 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4ff5b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009900 tx=0x7fafbc032b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff4a5699d5d1ae84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.753+0000 7fafcb400640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fafb80a3d10 0x7fafb80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.753+0000 7fafcb400640 1 -- 172.21.15.33:0/4228582948 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9874+0+0 (secure 0 0 0) 0x7fafc40d11f0 con 0x7fafd41038f0 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.753+0000 7fafc3e00640 1 -- 172.21.15.33:0/645277858 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc03f070 con 0x55857f4fd1c0 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.753+0000 7fafc3e00640 1 -- 172.21.15.33:0/645277858 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafbc032e40 con 0x55857f4fd1c0 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.753+0000 7fafc3e00640 1 -- 172.21.15.33:0/645277858 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc035070 con 0x55857f4fd1c0 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.753+0000 7fafd3400640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fafb80a3d10 0x7fafb80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 -- 172.21.15.33:0/645277858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 msgr2=0x55857f4ff5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafd3400640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fafb80a3d10 0x7fafb80a61d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fafcc005fd0 tx=0x7fafcc009500 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4ff5b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009900 tx=0x7fafbc032b90 comp rx=0 tx=0).stop 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 -- 172.21.15.33:0/645277858 shutdown_connections 2024-04-26T11:40:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4ff5b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 --2- 172.21.15.33:0/645277858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 -- 172.21.15.33:0/645277858 >> 172.21.15.33:0/645277858 conn(0x55857f4ee4f0 msgr2=0x55857f4ee8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 -- 172.21.15.33:0/645277858 shutdown_connections 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.754+0000 7fafdc670740 1 -- 172.21.15.33:0/645277858 wait complete. 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 -- start start 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 Processor -- start 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4fab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55857f4ffd90 con 0x55857f4f1fe0 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55857f3dec60 con 0x55857f4fd1c0 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55857f4fff10 con 0x55857f4f2900 2024-04-26T11:40:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.755+0000 7fafdc670740 10 client.0 _ll_register_callbacks cb 0x55857f40ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fa080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53310/0 (socket says 172.21.15.33:53310) 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 -- 172.21.15.33:0/618905343 learned_addr learned my addr 172.21.15.33:0/618905343 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafcbe00640 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd3400640 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4fab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 -- 172.21.15.33:0/618905343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 msgr2=0x55857f4fa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 -- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 msgr2=0x55857f4fab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4fab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 -- 172.21.15.33:0/618905343 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55857f3ed510 con 0x55857f4f1fe0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafd2a00640 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fa080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000c00 tx=0x7fafc400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b519c19f87895d8f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.756+0000 7fafc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafc0c00640 1 -- 172.21.15.33:0/618905343 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4010720 con 0x55857f4f1fe0 2024-04-26T11:40:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafc0c00640 1 -- 172.21.15.33:0/618905343 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9874+0+0 (secure 0 0 0) 0x7fafc40ce610 con 0x55857f4f1fe0 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55857f464f90 con 0x55857f4f1fe0 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafc0c00640 1 -- 172.21.15.33:0/618905343 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7fafc40ce810 con 0x55857f4f1fe0 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafdc670740 10 client.15786 fetch_fsmap learned FSMap version 144 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55857f169210 con 0x55857f4f1fe0 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafc0c00640 1 -- 172.21.15.33:0/618905343 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55857f464f90 con 0x55857f4f1fe0 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.757+0000 7fafc0c00640 1 -- 172.21.15.33:0/618905343 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 144) ==== 7132+0+0 (secure 0 0 0) 0x7fafc40d7cc0 con 0x55857f4f1fe0 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.758+0000 7fafdc670740 10 client.15786 fetch_fsmap finished waiting for FSMap version 144 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.758+0000 7fafdc670740 10 client.15786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.758+0000 7fafdc670740 20 client.15786 populate_metadata read hostname 'smithi033' 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.758+0000 7fafdc670740 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f5112e0 0x55857f5136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.758+0000 7fafdc670740 4 client.15786 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.758+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55857f45ee00 con 0x55857f5112e0 2024-04-26T11:40:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.758+0000 7fafcbe00640 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f5112e0 0x55857f5136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.759+0000 7fafcbe00640 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f5112e0 0x55857f5136d0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=317a6e6373e18950 server_cookie=62e58318886f1c45 in_seq=0 out_seq=0 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.759+0000 7fafc0c00640 10 client.15786 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafc0c00640 1 -- 172.21.15.33:0/618905343 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55857f45ee00 con 0x55857f5112e0 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafc0c00640 10 client.15786 handle_command_reply: tid=0 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 client.15786 shutdown 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 msgr2=0x55857f4fa080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fa080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000c00 tx=0x7fafc400cf20 comp rx=0 tx=0).stop 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 shutdown_connections 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4f2900 0x55857f4fa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f5112e0 0x55857f5136d0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4fd1c0 0x55857f4fab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 --2- 172.21.15.33:0/618905343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fa080 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 >> 172.21.15.33:0/618905343 conn(0x55857f4ee4f0 msgr2=0x55857f4f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafc0c00640 0 client.15786 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 shutdown_connections 2024-04-26T11:40:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 1 -- 172.21.15.33:0/618905343 wait complete. 2024-04-26T11:40:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.760+0000 7fafdc670740 20 client.15786 trim_cache size 0 max 16384 2024-04-26T11:40:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 Processor -- start 2024-04-26T11:40:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 -- start start 2024-04-26T11:40:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f2900 0x55857f4f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f518ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41adac0 con 0x55857f4f1fe0 2024-04-26T11:40:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41ae1c0 con 0x55857f4f2900 2024-04-26T11:40:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.777+0000 7fafdc670740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41ae890 con 0x55857f4fd1c0 2024-04-26T11:40:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53314/0 (socket says 172.21.15.33:53314) 2024-04-26T11:40:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 -- 172.21.15.33:0/2803098932 learned_addr learned my addr 172.21.15.33:0/2803098932 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd3400640 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f518ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafcbe00640 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f2900 0x55857f4f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 -- 172.21.15.33:0/2803098932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 msgr2=0x55857f518ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f518ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 -- 172.21.15.33:0/2803098932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f2900 msgr2=0x55857f4f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f2900 0x55857f4f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 -- 172.21.15.33:0/2803098932 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55857f235620 con 0x55857f4f1fe0 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafcbe00640 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f2900 0x55857f4f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.778+0000 7fafd2a00640 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fc7b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fafc40995b0 tx=0x7fafc4010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5649cd47b5628dec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafc3400640 1 -- 172.21.15.33:0/2803098932 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc409d2e0 con 0x55857f4f1fe0 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafc3400640 1 -- 172.21.15.33:0/2803098932 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc409cbe0 con 0x55857f4f1fe0 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafc3400640 1 -- 172.21.15.33:0/2803098932 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc40ced40 con 0x55857f4f1fe0 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 -- 172.21.15.33:0/2803098932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 msgr2=0x55857f4fc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fc7b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fafc40995b0 tx=0x7fafc4010ca0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 -- 172.21.15.33:0/2803098932 shutdown_connections 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f518ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f2900 0x55857f4f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 --2- 172.21.15.33:0/2803098932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f1fe0 0x55857f4fc7b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 -- 172.21.15.33:0/2803098932 >> 172.21.15.33:0/2803098932 conn(0x55857f4ee4f0 msgr2=0x55857f4f8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 -- 172.21.15.33:0/2803098932 shutdown_connections 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.779+0000 7fafdc670740 1 -- 172.21.15.33:0/2803098932 wait complete. 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.780+0000 7fafdc670740 1 -- start start 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.780+0000 7fafdc670740 1 Processor -- start 2024-04-26T11:40:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.780+0000 7fafdc670740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f1fe0 0x55857f5098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.780+0000 7fafdc670740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f2900 0x55857f4f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.780+0000 7fafdc670740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f4f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafdc670740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41aada0 con 0x55857f4f2900 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafdc670740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41ac1b0 con 0x55857f4f1fe0 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafdc670740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41ab410 con 0x55857f4fd1c0 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafdc670740 10 client.0 _ll_register_callbacks cb 0x7fafc40d4990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f2900 0x55857f4f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f2900 0x55857f4f69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53324/0 (socket says 172.21.15.33:53324) 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 -- 172.21.15.33:0/717694147 learned_addr learned my addr 172.21.15.33:0/717694147 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafd3400640 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f4f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafd2a00640 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f1fe0 0x55857f5098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 -- 172.21.15.33:0/717694147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 msgr2=0x55857f4f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f4f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 -- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f1fe0 msgr2=0x55857f5098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:40:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f1fe0 0x55857f5098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 -- 172.21.15.33:0/717694147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55857f411cb0 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafcbe00640 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f2900 0x55857f4f69b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafbc033000 tx=0x7fafbc02ab00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b511aaae617941a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.781+0000 7fafc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.782+0000 7fafc0c00640 1 -- 172.21.15.33:0/717694147 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc053020 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.782+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc409cbe0 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.782+0000 7fafc0c00640 1 -- 172.21.15.33:0/717694147 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9874+0+0 (secure 0 0 0) 0x7fafbc0318b0 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.782+0000 7fafc0c00640 1 -- 172.21.15.33:0/717694147 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7fafbc05e050 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.782+0000 7fafdc670740 10 client.15792 fetch_fsmap learned FSMap version 144 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.782+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55857f5774b0 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.782+0000 7fafc0c00640 1 -- 172.21.15.33:0/717694147 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc409cbe0 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafc0c00640 1 -- 172.21.15.33:0/717694147 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 144) ==== 7132+0+0 (secure 0 0 0) 0x7fafbc003b10 con 0x55857f4f2900 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafdc670740 10 client.15792 fetch_fsmap finished waiting for FSMap version 144 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafdc670740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafdc670740 20 client.15792 populate_metadata read hostname 'smithi033' 2024-04-26T11:40:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafdc670740 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f577a00 0x55857f579e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:40:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafdc670740 4 client.15792 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:40:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55857f3da4b0 con 0x55857f577a00 2024-04-26T11:40:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.783+0000 7fafd2a00640 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f577a00 0x55857f579e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:40:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.784+0000 7fafd2a00640 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f577a00 0x55857f579e20 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f37461e6edc940c server_cookie=eba951de43da0809 in_seq=0 out_seq=0 2024-04-26T11:40:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:39.784+0000 7fafc0c00640 10 client.15792 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:40.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:40 smithi033 ceph-mon[27845]: pgmap v335: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 8.8 MiB/s wr, 948 op/s 2024-04-26T11:40:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:40 smithi096 ceph-mon[32753]: pgmap v335: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 8.8 MiB/s wr, 948 op/s 2024-04-26T11:40:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:40 smithi194 ceph-mon[33017]: pgmap v335: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 8.8 MiB/s wr, 948 op/s 2024-04-26T11:40:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 565.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:40:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:42 smithi033 ceph-mon[27845]: pgmap v336: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.0 MiB/s wr, 878 op/s 2024-04-26T11:40:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:42 smithi096 ceph-mon[32753]: pgmap v336: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.0 MiB/s wr, 878 op/s 2024-04-26T11:40:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:42 smithi194 ceph-mon[33017]: pgmap v336: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.0 MiB/s wr, 878 op/s 2024-04-26T11:40:44.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:43 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:40:44.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:43 smithi194 systemd[1]: Stopped Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:40:44.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:43 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.341s CPU time. 2024-04-26T11:40:44.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:43 smithi194 systemd[1]: Starting Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:40:44.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 podman[69760]: 2024-04-26 11:40:43.999630573 +0000 UTC m=+0.018621046 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:40:44.381 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 podman[69760]: 2024-04-26 11:40:44.110794387 +0000 UTC m=+0.129784877 container create f4589d30da905ff9a562bd9a9b39c6db369d526fd85e42f747f00f3d9f1be02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD) 2024-04-26T11:40:44.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 podman[69760]: 2024-04-26 11:40:44.381627848 +0000 UTC m=+0.400618314 container init f4589d30da905ff9a562bd9a9b39c6db369d526fd85e42f747f00f3d9f1be02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:40:44.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 podman[69760]: 2024-04-26 11:40:44.384464258 +0000 UTC m=+0.403454714 container start f4589d30da905ff9a562bd9a9b39c6db369d526fd85e42f747f00f3d9f1be02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T11:40:44.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mds[69779]: -- 172.21.15.194:0/1462711001 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b8cded1a40 con 0x55b8cdea5000 2024-04-26T11:40:44.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: starting mds.l at 2024-04-26T11:40:44.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 bash[69760]: f4589d30da905ff9a562bd9a9b39c6db369d526fd85e42f747f00f3d9f1be02f 2024-04-26T11:40:44.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:40:44 smithi194 systemd[1]: Started Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:40:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: pgmap v337: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 845 op/s 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: daemon mds.l restarted 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:boot 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:44 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: pgmap v337: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 845 op/s 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: daemon mds.l restarted 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:boot 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:44 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: pgmap v337: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 845 op/s 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: daemon mds.l restarted 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:boot 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:44 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:46.900 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:46 smithi033 ceph-mon[27845]: pgmap v339: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 933 KiB/s wr, 125 op/s 2024-04-26T11:40:46.901 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:46 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:resolve 2024-04-26T11:40:46.901 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:46 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:46 smithi096 ceph-mon[32753]: pgmap v339: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 933 KiB/s wr, 125 op/s 2024-04-26T11:40:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:46 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:resolve 2024-04-26T11:40:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:46 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:46 smithi194 ceph-mon[33017]: pgmap v339: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 933 KiB/s wr, 125 op/s 2024-04-26T11:40:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:46 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:resolve 2024-04-26T11:40:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:46 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 570.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.666+0000 7fafc0c00640 1 -- 172.21.15.33:0/717694147 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55857f3da4b0 con 0x55857f577a00 2024-04-26T11:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.666+0000 7fafc0c00640 10 client.15792 handle_command_reply: tid=0 2024-04-26T11:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.666+0000 7fafdc670740 1 client.15792 shutdown 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.666+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f2900 msgr2=0x55857f4f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.666+0000 7fafdc670740 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f2900 0x55857f4f69b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafbc033000 tx=0x7fafbc02ab00 comp rx=0 tx=0).stop 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 shutdown_connections 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55857f4fd1c0 0x55857f4f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55857f577a00 0x55857f579e20 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55857f4f2900 0x55857f4f69b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 --2- 172.21.15.33:0/717694147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55857f4f1fe0 0x55857f5098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafc0c00640 0 client.15792 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 >> 172.21.15.33:0/717694147 conn(0x55857f4ee4f0 msgr2=0x55857f4fa040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 shutdown_connections 2024-04-26T11:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.667+0000 7fafdc670740 1 -- 172.21.15.33:0/717694147 wait complete. 2024-04-26T11:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.668+0000 7fafdc670740 20 client.15792 trim_cache size 0 max 16384 2024-04-26T11:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:40:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.669+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fafb80a3d10 msgr2=0x7fafb80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.669+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fafb80a3d10 0x7fafb80a61d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fafcc005fd0 tx=0x7fafcc009500 comp rx=0 tx=0).stop 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.669+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 msgr2=0x7fafd4198fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.669+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 0x7fafd4198fa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9f0 tx=0x7fafc400eeb0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 shutdown_connections 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7fafb80a3d10 0x7fafb80a61d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafd4104210 0x7fafd41994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafd410cd10 0x7fafd419d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 --2- 172.21.15.33:0/4228582948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafd41038f0 0x7fafd4198fa0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 >> 172.21.15.33:0/4228582948 conn(0x7fafd40fd150 msgr2=0x7fafd40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 shutdown_connections 2024-04-26T11:40:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:40:47.670+0000 7fafd3e00640 1 -- 172.21.15.33:0/4228582948 wait complete. 2024-04-26T11:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:47 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999979 2024-04-26T11:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:47 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:reconnect 2024-04-26T11:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:47 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:47 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:40:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:47 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999979 2024-04-26T11:40:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:47 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:reconnect 2024-04-26T11:40:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:47 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:47 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:40:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:47 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999979 2024-04-26T11:40:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:47 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:reconnect 2024-04-26T11:40:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:47 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:47 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00899981 2024-04-26T11:40:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:48 smithi033 ceph-mon[27845]: pgmap v340: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 933 KiB/s wr, 125 op/s 2024-04-26T11:40:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:48 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:rejoin 2024-04-26T11:40:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:48 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:48 smithi033 ceph-mon[27845]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:40:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:48 smithi096 ceph-mon[32753]: pgmap v340: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 933 KiB/s wr, 125 op/s 2024-04-26T11:40:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:48 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:rejoin 2024-04-26T11:40:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:48 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:48 smithi096 ceph-mon[32753]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:40:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:48 smithi194 ceph-mon[33017]: pgmap v340: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 933 KiB/s wr, 125 op/s 2024-04-26T11:40:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:48 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:rejoin 2024-04-26T11:40:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:48 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:48 smithi194 ceph-mon[33017]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:40:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:49 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:49 smithi033 ceph-mon[27845]: Cluster is now healthy 2024-04-26T11:40:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:49 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:active 2024-04-26T11:40:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:49 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:49 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:49 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:49 smithi096 ceph-mon[32753]: Cluster is now healthy 2024-04-26T11:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:49 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:active 2024-04-26T11:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:49 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:49 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:40:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:49 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:40:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:49 smithi194 ceph-mon[33017]: Cluster is now healthy 2024-04-26T11:40:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:49 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/1362944124,v1:172.21.15.96:6833/1362944124] up:active 2024-04-26T11:40:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:49 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:40:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:49 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:40:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:40:51.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:50 smithi033 ceph-mon[27845]: pgmap v341: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-04-26T11:40:51.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:50 smithi194 ceph-mon[33017]: pgmap v341: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-04-26T11:40:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:50 smithi096 ceph-mon[32753]: pgmap v341: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-04-26T11:40:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:51.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:52.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 575.0 s, 1280.4 tps, lat 297.409 ms stddev 1939.276 2024-04-26T11:40:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: pgmap v342: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:40:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: pgmap v342: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:40:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: pgmap v342: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:40:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:53.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:53.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:54.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.068 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f91900cc700 time 2024-04-26T11:40:53.722182+0000 2024-04-26T11:40:54.068 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f919a6f53dd] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.069 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: clone() 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:53.722+0000 7f91900cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f91900cc700 time 2024-04-26T11:40:53.722182+0000 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f919a6f53dd] 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.070 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: clone() 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: *** Caught signal (Aborted) ** 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: in thread 7f91900cc700 thread_name:ms_dispatch 2024-04-26T11:40:54.071 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f919947dd20] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: gsignal() 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: abort() 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f919a6f5437] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.072 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 16: clone() 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:53.723+0000 7f91900cc700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: in thread 7f91900cc700 thread_name:ms_dispatch 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f919947dd20] 2024-04-26T11:40:54.073 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: gsignal() 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: abort() 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f919a6f5437] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.074 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 16: clone() 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: -1> 2024-04-26T11:40:53.722+0000 7f91900cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f91900cc700 time 2024-04-26T11:40:53.722182+0000 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f919a6f53dd] 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.075 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: clone() 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 0> 2024-04-26T11:40:53.723+0000 7f91900cc700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: in thread 7f91900cc700 thread_name:ms_dispatch 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f919947dd20] 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: gsignal() 2024-04-26T11:40:54.076 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: abort() 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f919a6f5437] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.077 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 16: clone() 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: -1> 2024-04-26T11:40:53.722+0000 7f91900cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f91900cc700 time 2024-04-26T11:40:53.722182+0000 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f919a6f53dd] 2024-04-26T11:40:54.078 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.079 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.079 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.079 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.079 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.079 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.079 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: clone() 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 0> 2024-04-26T11:40:53.723+0000 7f91900cc700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: in thread 7f91900cc700 thread_name:ms_dispatch 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f919947dd20] 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2: gsignal() 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 3: abort() 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f919a6f5437] 2024-04-26T11:40:54.080 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f919a6f55a3] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564b6a777822] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564b6a6e7598] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b6a6e9c17] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564b6a5e2d0b] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b6a5e32ac] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564b6a5cbfad] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f919a989e20] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 13: (DispatchQueue::entry()+0x51f) [0x7f919a986fdf] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f919aa59fc1] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91994731ca] 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 16: clone() 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[69264]: 2024-04-26T11:40:54.081 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:53 smithi096 podman[69861]: 2024-04-26 11:40:53.786131333 +0000 UTC m=+0.021769889 container died 57a57fe22001788e6a8908507078fa762e06f8d9e538cd45c6685f332f336b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:40:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:40:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:40:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:54.743 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:54 smithi096 podman[69861]: 2024-04-26 11:40:54.433575321 +0000 UTC m=+0.669213866 container remove 57a57fe22001788e6a8908507078fa762e06f8d9e538cd45c6685f332f336b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:40:54.743 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:54 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:40:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: pgmap v343: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:40:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:40:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:55.182 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:54 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-26T11:40:55.182 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:40:54 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.047s CPU time. 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: pgmap v343: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:40:55.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:40:55.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:40:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:40:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: pgmap v343: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:40:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:40:55.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:40:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:55 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:40:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:55 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:40:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:55 smithi033 ceph-mon[27845]: pgmap v344: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.0 MiB/s wr, 340 op/s 2024-04-26T11:40:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:55 smithi033 ceph-mon[27845]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-26T11:40:56.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:55 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:40:56.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:55 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:40:56.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:55 smithi096 ceph-mon[32753]: pgmap v344: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.0 MiB/s wr, 340 op/s 2024-04-26T11:40:56.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:55 smithi096 ceph-mon[32753]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-26T11:40:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:55 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:40:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:55 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:40:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:55 smithi194 ceph-mon[33017]: pgmap v344: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.0 MiB/s wr, 340 op/s 2024-04-26T11:40:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:55 smithi194 ceph-mon[33017]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-26T11:40:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:56 smithi033 ceph-mon[27845]: pgmap v345: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-04-26T11:40:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:56 smithi096 ceph-mon[32753]: pgmap v345: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-04-26T11:40:57.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 580.0 s, 2415.9 tps, lat 41.525 ms stddev 47.799 2024-04-26T11:40:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:56 smithi194 ceph-mon[33017]: pgmap v345: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-04-26T11:40:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:40:59 smithi194 ceph-mon[33017]: pgmap v346: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-04-26T11:40:59.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:40:59 smithi033 ceph-mon[27845]: pgmap v346: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-04-26T11:40:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:40:59 smithi096 ceph-mon[32753]: pgmap v346: 97 pgs: 97 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-04-26T11:41:00.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.819+0000 7f38ee600640 1 -- 172.21.15.33:0/832559610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 msgr2=0x7f38e81018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.819+0000 7f38ee600640 1 --2- 172.21.15.33:0/832559610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 0x7f38e81018b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009860 tx=0x7f38dc02a340 comp rx=0 tx=0).stop 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.820+0000 7f38ee600640 1 -- 172.21.15.33:0/832559610 shutdown_connections 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.820+0000 7f38ee600640 1 --2- 172.21.15.33:0/832559610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e810f080 0x7f38e8111470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.820+0000 7f38ee600640 1 --2- 172.21.15.33:0/832559610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38e8101e80 0x7f38e810eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.820+0000 7f38ee600640 1 --2- 172.21.15.33:0/832559610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 0x7f38e81018b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.820+0000 7f38ee600640 1 -- 172.21.15.33:0/832559610 >> 172.21.15.33:0/832559610 conn(0x7f38e80fd150 msgr2=0x7f38e80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.820+0000 7f38ee600640 1 -- 172.21.15.33:0/832559610 shutdown_connections 2024-04-26T11:41:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.820+0000 7f38ee600640 1 -- 172.21.15.33:0/832559610 wait complete. 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 Processor -- start 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 -- start start 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 0x7f38e819b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e8101e80 0x7f38e819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38e810f080 0x7f38e819fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38e8114be0 con 0x7f38e8101e80 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38e8114a60 con 0x7f38e810f080 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ee600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f38e8114d60 con 0x7f38e81014d0 2024-04-26T11:41:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e8101e80 0x7f38e819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e8101e80 0x7f38e819b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54822/0 (socket says 172.21.15.33:54822) 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38e7e00640 1 -- 172.21.15.33:0/2252630173 learned_addr learned my addr 172.21.15.33:0/2252630173 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38ed200640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 0x7f38e819b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.821+0000 7f38edc00640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38e810f080 0x7f38e819fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e7e00640 1 -- 172.21.15.33:0/2252630173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 msgr2=0x7f38e819b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e7e00640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 0x7f38e819b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e7e00640 1 -- 172.21.15.33:0/2252630173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38e810f080 msgr2=0x7f38e819fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e7e00640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38e810f080 0x7f38e819fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e7e00640 1 -- 172.21.15.33:0/2252630173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e81a0200 con 0x7f38e8101e80 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e7e00640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e8101e80 0x7f38e819b7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f38d400e9f0 tx=0x7f38d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e5600640 1 -- 172.21.15.33:0/2252630173 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d400cd80 con 0x7f38e8101e80 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e5600640 1 -- 172.21.15.33:0/2252630173 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d40108e0 con 0x7f38e8101e80 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e81a04f0 con 0x7f38e8101e80 2024-04-26T11:41:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.822+0000 7f38e5600640 1 -- 172.21.15.33:0/2252630173 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4018980 con 0x7f38e8101e80 2024-04-26T11:41:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.823+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e81a7d80 con 0x7f38e8101e80 2024-04-26T11:41:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.826+0000 7f38f2980740 1 Processor -- start 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.826+0000 7f38e5600640 1 -- 172.21.15.33:0/2252630173 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f38d4010430 con 0x7f38e8101e80 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.826+0000 7f38f2980740 1 -- start start 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.826+0000 7f38f2980740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c481fe0 0x55713c4823c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.826+0000 7f38f2980740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c482900 0x55713c48cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38f2980740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c48d1c0 0x55713c48f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38f2980740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55713c3e7380 con 0x55713c481fe0 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38f2980740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55713c383c30 con 0x55713c48d1c0 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38f2980740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55713c3e3340 con 0x55713c482900 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38edc00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c481fe0 0x55713c4823c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c48d1c0 0x55713c48f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38edc00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c481fe0 0x55713c4823c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54836/0 (socket says 172.21.15.33:54836) 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38edc00640 1 -- 172.21.15.33:0/3912440830 learned_addr learned my addr 172.21.15.33:0/3912440830 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38ed200640 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c482900 0x55713c48cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38e7e00640 1 -- 172.21.15.33:0/3912440830 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c482900 msgr2=0x55713c48cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38e7e00640 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c482900 0x55713c48cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38e7e00640 1 -- 172.21.15.33:0/3912440830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c481fe0 msgr2=0x55713c4823c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38e7e00640 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c481fe0 0x55713c4823c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38e7e00640 1 -- 172.21.15.33:0/3912440830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55713c3a5940 con 0x55713c48d1c0 2024-04-26T11:41:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.827+0000 7f38e7e00640 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c48d1c0 0x55713c48f5b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ce230 tx=0x7f38d40d9730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e0d1ed6e6955cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.828+0000 7f38da000640 1 -- 172.21.15.33:0/3912440830 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40e4070 con 0x55713c48d1c0 2024-04-26T11:41:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.828+0000 7f38e5600640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f38d00a3d10 0x7f38d00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.828+0000 7f38da000640 1 -- 172.21.15.33:0/3912440830 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d40d1980 con 0x55713c48d1c0 2024-04-26T11:41:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.828+0000 7f38da000640 1 -- 172.21.15.33:0/3912440830 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d409c670 con 0x55713c48d1c0 2024-04-26T11:41:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.828+0000 7f38e5600640 1 -- 172.21.15.33:0/2252630173 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9960+0+0 (secure 0 0 0) 0x7f38d40d10e0 con 0x7f38e8101e80 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.828+0000 7f38edc00640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f38d00a3d10 0x7f38d00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 -- 172.21.15.33:0/3912440830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c48d1c0 msgr2=0x55713c48f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c48d1c0 0x55713c48f5b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ce230 tx=0x7f38d40d9730 comp rx=0 tx=0).stop 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 -- 172.21.15.33:0/3912440830 shutdown_connections 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c482900 0x55713c48cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c48d1c0 0x55713c48f5b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 --2- 172.21.15.33:0/3912440830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c481fe0 0x55713c4823c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 -- 172.21.15.33:0/3912440830 >> 172.21.15.33:0/3912440830 conn(0x55713c47e4f0 msgr2=0x55713c47e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 -- 172.21.15.33:0/3912440830 shutdown_connections 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38edc00640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f38d00a3d10 0x7f38d00a61d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f38e000e5a0 tx=0x7f38e0010040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 -- 172.21.15.33:0/3912440830 wait complete. 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 -- start start 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.829+0000 7f38f2980740 1 Processor -- start 2024-04-26T11:41:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38f2980740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c481fe0 0x55713c48a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38f2980740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c482900 0x55713c48a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38f2980740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c48d1c0 0x55713c48ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38f2980740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55713c48fdf0 con 0x55713c48d1c0 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38f2980740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55713c36ec60 con 0x55713c482900 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38f2980740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55713c48ff70 con 0x55713c481fe0 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38f2980740 10 client.0 _ll_register_callbacks cb 0x55713c39ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38edc00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c48d1c0 0x55713c48ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38ed200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c481fe0 0x55713c48a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c482900 0x55713c48a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38ed200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c481fe0 0x55713c48a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:42690/0 (socket says 172.21.15.33:42690) 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38ed200640 1 -- 172.21.15.33:0/3552662213 learned_addr learned my addr 172.21.15.33:0/3552662213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38e7e00640 1 -- 172.21.15.33:0/3552662213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c481fe0 msgr2=0x55713c48a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38e7e00640 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c481fe0 0x55713c48a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38e7e00640 1 -- 172.21.15.33:0/3552662213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c48d1c0 msgr2=0x55713c48ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.830+0000 7f38e7e00640 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c48d1c0 0x55713c48ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38e7e00640 1 -- 172.21.15.33:0/3552662213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55713c37d510 con 0x55713c482900 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38ed200640 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c481fe0 0x55713c48a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38edc00640 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c48d1c0 0x55713c48ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:41:00.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38e7e00640 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c482900 0x55713c48a5f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f38d409abd0 tx=0x7f38d400cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e840812fa4c46699 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38cea00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38cea00640 1 -- 172.21.15.33:0/3552662213 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d409b0f0 con 0x55713c482900 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d40d1980 con 0x55713c482900 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38cea00640 1 -- 172.21.15.33:0/3552662213 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9960+0+0 (secure 0 0 0) 0x7f38d40d4c80 con 0x55713c482900 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38cea00640 1 -- 172.21.15.33:0/3552662213 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) ==== 24+0+0 (secure 0 0 0) 0x7f38d409b2e0 con 0x55713c482900 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38f2980740 10 client.26060 fetch_fsmap learned FSMap version 151 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.831+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=4+}) -- 0x55713c0f9210 con 0x55713c482900 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38cea00640 1 -- 172.21.15.33:0/3552662213 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d40d1980 con 0x55713c482900 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38cea00640 1 -- 172.21.15.33:0/3552662213 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 151) ==== 7132+0+0 (secure 0 0 0) 0x7f38d40991a0 con 0x55713c482900 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38f2980740 10 client.26060 fetch_fsmap finished waiting for FSMap version 151 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38f2980740 10 client.26060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38f2980740 20 client.26060 populate_metadata read hostname 'smithi033' 2024-04-26T11:41:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38f2980740 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c49ccf0 0x55713c49f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38f2980740 4 client.26060 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:41:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55713c3f4f90 con 0x55713c49ccf0 2024-04-26T11:41:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.832+0000 7f38ed200640 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c49ccf0 0x55713c49f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.833+0000 7f38ed200640 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c49ccf0 0x55713c49f0e0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23713d91b13de836 server_cookie=113ceb9128279e3f in_seq=0 out_seq=0 2024-04-26T11:41:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.833+0000 7f38cea00640 10 client.26060 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.833+0000 7f38cea00640 1 -- 172.21.15.33:0/3552662213 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55713c3f4f90 con 0x55713c49ccf0 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.833+0000 7f38cea00640 10 client.26060 handle_command_reply: tid=0 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.833+0000 7f38f2980740 1 client.26060 shutdown 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.833+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c482900 msgr2=0x55713c48a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.833+0000 7f38f2980740 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c482900 0x55713c48a5f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f38d409abd0 tx=0x7f38d400cf20 comp rx=0 tx=0).stop 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 shutdown_connections 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c49ccf0 0x55713c49f0e0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c48d1c0 0x55713c48ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c482900 0x55713c48a5f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38cea00640 0 client.26060 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 --2- 172.21.15.33:0/3552662213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c481fe0 0x55713c48a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 >> 172.21.15.33:0/3552662213 conn(0x55713c47e4f0 msgr2=0x55713c484270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 shutdown_connections 2024-04-26T11:41:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 1 -- 172.21.15.33:0/3552662213 wait complete. 2024-04-26T11:41:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.834+0000 7f38f2980740 20 client.26060 trim_cache size 0 max 16384 2024-04-26T11:41:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.849+0000 7f38f2980740 1 Processor -- start 2024-04-26T11:41:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38f2980740 1 -- start start 2024-04-26T11:41:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38f2980740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c498cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38f2980740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4849a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38f2980740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4a4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38f2980740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38e81b2020 con 0x55713c482900 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38f2980740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38e81b26f0 con 0x55713c481fe0 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38f2980740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f38e81b2dc0 con 0x55713c48d1c0 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4849a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38edc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38ed200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c498cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38edc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4a4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:42706/0 (socket says 172.21.15.33:42706) 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38edc00640 1 -- 172.21.15.33:0/78857535 learned_addr learned my addr 172.21.15.33:0/78857535 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.850+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4849a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54852/0 (socket says 172.21.15.33:54852) 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38edc00640 1 -- 172.21.15.33:0/78857535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 msgr2=0x55713c498cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38edc00640 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c498cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38edc00640 1 -- 172.21.15.33:0/78857535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 msgr2=0x55713c4849a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38edc00640 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4849a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38edc00640 1 -- 172.21.15.33:0/78857535 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55713c1c5620 con 0x55713c48d1c0 2024-04-26T11:41:00.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38e7e00640 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4849a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:41:00.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38edc00640 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4a4790 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f38e0004040 tx=0x7f38e0016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e47f935f571f63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38d9600640 1 -- 172.21.15.33:0/78857535 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e000d8b0 con 0x55713c48d1c0 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38d9600640 1 -- 172.21.15.33:0/78857535 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e002b070 con 0x55713c48d1c0 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.851+0000 7f38d9600640 1 -- 172.21.15.33:0/78857535 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0026420 con 0x55713c48d1c0 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 -- 172.21.15.33:0/78857535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 msgr2=0x55713c4a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4a4790 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f38e0004040 tx=0x7f38e0016000 comp rx=0 tx=0).stop 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 -- 172.21.15.33:0/78857535 shutdown_connections 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4a4790 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4849a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 --2- 172.21.15.33:0/78857535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c498cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 -- 172.21.15.33:0/78857535 >> 172.21.15.33:0/78857535 conn(0x55713c47e4f0 msgr2=0x55713c4992c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 -- 172.21.15.33:0/78857535 shutdown_connections 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 -- 172.21.15.33:0/78857535 wait complete. 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 -- start start 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.852+0000 7f38f2980740 1 Processor -- start 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38f2980740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c495180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38f2980740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38f2980740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4907b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38f2980740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38e81af300 con 0x55713c482900 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38f2980740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38e81b0710 con 0x55713c481fe0 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38f2980740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f38e81af970 con 0x55713c48d1c0 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38f2980740 10 client.0 _ll_register_callbacks cb 0x7f38d40e2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38ed200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c495180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38edc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4907b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:54860/0 (socket says 172.21.15.33:54860) 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.853+0000 7f38e7e00640 1 -- 172.21.15.33:0/3928869799 learned_addr learned my addr 172.21.15.33:0/3928869799 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38edc00640 1 -- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 msgr2=0x55713c495180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38edc00640 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c495180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38edc00640 1 -- 172.21.15.33:0/3928869799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 msgr2=0x55713c4956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38edc00640 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38edc00640 1 -- 172.21.15.33:0/3928869799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55713c3a1cb0 con 0x55713c48d1c0 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38e7e00640 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38ed200640 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c495180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38edc00640 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4907b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f38e0004810 tx=0x7f38e0009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1272b4c907ab71ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38cea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38cea00640 1 -- 172.21.15.33:0/3928869799 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0003d20 con 0x55713c48d1c0 2024-04-26T11:41:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38e002b070 con 0x55713c48d1c0 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.854+0000 7f38cea00640 1 -- 172.21.15.33:0/3928869799 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9960+0+0 (secure 0 0 0) 0x7f38e001a450 con 0x55713c48d1c0 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38cea00640 1 -- 172.21.15.33:0/3928869799 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) ==== 24+0+0 (secure 0 0 0) 0x7f38e0042050 con 0x55713c48d1c0 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 10 client.25978 fetch_fsmap learned FSMap version 151 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=4+}) -- 0x55713c502a90 con 0x55713c48d1c0 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38cea00640 1 -- 172.21.15.33:0/3928869799 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e002b070 con 0x55713c48d1c0 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38cea00640 1 -- 172.21.15.33:0/3928869799 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 151) ==== 7132+0+0 (secure 0 0 0) 0x7f38e000b040 con 0x55713c48d1c0 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 10 client.25978 fetch_fsmap finished waiting for FSMap version 151 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 10 client.25978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 20 client.25978 populate_metadata read hostname 'smithi033' 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c503140 0x55713c505560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 4 client.25978 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55713c3eee00 con 0x55713c503140 2024-04-26T11:41:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.855+0000 7f38ed200640 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c503140 0x55713c505560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:00.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.856+0000 7f38ed200640 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c503140 0x55713c505560 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa7a264048e48a8e server_cookie=4cf25d6043d87346 in_seq=0 out_seq=0 2024-04-26T11:41:00.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:00.856+0000 7f38cea00640 10 client.25978 ms_handle_connect on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:01 smithi194 ceph-mon[33017]: pgmap v347: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 674 op/s 2024-04-26T11:41:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:01 smithi033 ceph-mon[27845]: pgmap v347: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 674 op/s 2024-04-26T11:41:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:01 smithi096 ceph-mon[32753]: pgmap v347: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 674 op/s 2024-04-26T11:41:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 585.0 s, 1133.5 tps, lat 40.210 ms stddev 45.280 2024-04-26T11:41:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:03 smithi194 ceph-mon[33017]: pgmap v348: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-04-26T11:41:03.858 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:03 smithi033 ceph-mon[27845]: pgmap v348: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-04-26T11:41:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:03 smithi096 ceph-mon[32753]: pgmap v348: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-04-26T11:41:05.308 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:04 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:41:05.308 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:04 smithi096 systemd[1]: Stopped Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:05.308 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:04 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.047s CPU time. 2024-04-26T11:41:05.308 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:04 smithi096 systemd[1]: Starting Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:41:05.308 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 podman[69990]: 2024-04-26 11:41:05.069866183 +0000 UTC m=+0.017914887 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:41:05.308 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 podman[69990]: 2024-04-26 11:41:05.167531876 +0000 UTC m=+0.115580549 container create 98aadb1d70854a2155b4ea61a2e0a9a1687a5e75c30ab6e793c976e29ddabb34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:05 smithi096 ceph-mon[32753]: pgmap v349: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.2 MiB/s wr, 688 op/s 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:05 smithi096 ceph-mon[32753]: daemon mds.k restarted 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:05 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 podman[69990]: 2024-04-26 11:41:05.428989826 +0000 UTC m=+0.377038514 container init 98aadb1d70854a2155b4ea61a2e0a9a1687a5e75c30ab6e793c976e29ddabb34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 podman[69990]: 2024-04-26 11:41:05.431943425 +0000 UTC m=+0.379992105 container start 98aadb1d70854a2155b4ea61a2e0a9a1687a5e75c30ab6e793c976e29ddabb34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 ceph-mds[70010]: -- 172.21.15.96:0/246459517 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557b04c47a40 con 0x557b04c1a800 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: starting mds.k at 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 bash[69990]: 98aadb1d70854a2155b4ea61a2e0a9a1687a5e75c30ab6e793c976e29ddabb34 2024-04-26T11:41:05.613 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:05 smithi096 systemd[1]: Started Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:05 smithi194 ceph-mon[33017]: pgmap v349: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.2 MiB/s wr, 688 op/s 2024-04-26T11:41:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:05 smithi194 ceph-mon[33017]: daemon mds.k restarted 2024-04-26T11:41:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:05 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:05 smithi033 ceph-mon[27845]: pgmap v349: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.2 MiB/s wr, 688 op/s 2024-04-26T11:41:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:05 smithi033 ceph-mon[27845]: daemon mds.k restarted 2024-04-26T11:41:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:05 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:41:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:06 smithi194 ceph-mon[33017]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T11:41:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:06 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:boot 2024-04-26T11:41:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:06 smithi194 ceph-mon[33017]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:41:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:06 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:06 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:41:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:06 smithi033 ceph-mon[27845]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T11:41:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:06 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:boot 2024-04-26T11:41:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:06 smithi033 ceph-mon[27845]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:41:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:06 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:06 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:41:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:06 smithi096 ceph-mon[32753]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T11:41:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:06 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:boot 2024-04-26T11:41:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:06 smithi096 ceph-mon[32753]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:41:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:06 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:06 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:07.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 590.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:41:07.758 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:07 smithi033 ceph-mon[27845]: pgmap v351: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-04-26T11:41:07.759 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:07 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:resolve 2024-04-26T11:41:07.759 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:07 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:07 smithi194 ceph-mon[33017]: pgmap v351: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-04-26T11:41:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:07 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:resolve 2024-04-26T11:41:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:07 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:07 smithi096 ceph-mon[32753]: pgmap v351: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-04-26T11:41:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:07 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:resolve 2024-04-26T11:41:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:07 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.529+0000 7f38cea00640 1 -- 172.21.15.33:0/3928869799 <== mds.0 v2:172.21.15.96:6838/1966519759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55713c3eee00 con 0x55713c503140 2024-04-26T11:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.529+0000 7f38cea00640 10 client.25978 handle_command_reply: tid=0 2024-04-26T11:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.530+0000 7f38f2980740 1 client.25978 shutdown 2024-04-26T11:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.530+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 msgr2=0x55713c4907b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.530+0000 7f38f2980740 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4907b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f38e0004810 tx=0x7f38e0009fd0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 shutdown_connections 2024-04-26T11:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55713c48d1c0 0x55713c4907b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x55713c503140 0x55713c505560 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55713c482900 0x55713c4956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 --2- 172.21.15.33:0/3928869799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55713c481fe0 0x55713c495180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38cea00640 0 client.25978 ms_handle_reset on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 >> 172.21.15.33:0/3928869799 conn(0x55713c47e4f0 msgr2=0x55713c48b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 shutdown_connections 2024-04-26T11:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.531+0000 7f38f2980740 1 -- 172.21.15.33:0/3928869799 wait complete. 2024-04-26T11:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.532+0000 7f38f2980740 20 client.25978 trim_cache size 0 max 16384 2024-04-26T11:41:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:{ 2024-04-26T11:41:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "message": "", 2024-04-26T11:41:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout: "return_code": 0 2024-04-26T11:41:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stdout:} 2024-04-26T11:41:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f38d00a3d10 msgr2=0x7f38d00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f38d00a3d10 0x7f38d00a61d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f38e000e5a0 tx=0x7f38e0010040 comp rx=0 tx=0).stop 2024-04-26T11:41:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e8101e80 msgr2=0x7f38e819b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e8101e80 0x7f38e819b7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f38d400e9f0 tx=0x7f38d400eeb0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 shutdown_connections 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7f38d00a3d10 0x7f38d00a61d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38e810f080 0x7f38e819fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38e8101e80 0x7f38e819b7a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 --2- 172.21.15.33:0/2252630173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f38e81014d0 0x7f38e819b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 >> 172.21.15.33:0/2252630173 conn(0x7f38e80fd150 msgr2=0x7f38e80ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 shutdown_connections 2024-04-26T11:41:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:08.533+0000 7f38ee600640 1 -- 172.21.15.33:0/2252630173 wait complete. 2024-04-26T11:41:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:08 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00099998 2024-04-26T11:41:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:08 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:reconnect 2024-04-26T11:41:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:08 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:08 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00799984 2024-04-26T11:41:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:08 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00099998 2024-04-26T11:41:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:08 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:reconnect 2024-04-26T11:41:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:08 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:08 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00799984 2024-04-26T11:41:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:08 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00099998 2024-04-26T11:41:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:08 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:reconnect 2024-04-26T11:41:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:08 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:08 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00799984 2024-04-26T11:41:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:09 smithi194 ceph-mon[33017]: pgmap v352: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-04-26T11:41:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:09 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:rejoin 2024-04-26T11:41:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:09 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:09 smithi194 ceph-mon[33017]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:41:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:09 smithi033 ceph-mon[27845]: pgmap v352: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-04-26T11:41:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:09 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:rejoin 2024-04-26T11:41:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:09 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:09 smithi033 ceph-mon[27845]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:41:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:09 smithi096 ceph-mon[32753]: pgmap v352: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-04-26T11:41:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:09 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:rejoin 2024-04-26T11:41:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:09 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:09 smithi096 ceph-mon[32753]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:41:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:10 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:41:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:10 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:active 2024-04-26T11:41:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:10 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:10 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:10 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:41:10.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:10 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:active 2024-04-26T11:41:10.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:10 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:10.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:10 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:10 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:41:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:10 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/3458200426,v1:172.21.15.194:6833/3458200426] up:active 2024-04-26T11:41:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:10 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:10 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:11 smithi194 ceph-mon[33017]: pgmap v353: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.9 MiB/s wr, 321 op/s 2024-04-26T11:41:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:11 smithi033 ceph-mon[27845]: pgmap v353: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.9 MiB/s wr, 321 op/s 2024-04-26T11:41:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:11 smithi096 ceph-mon[32753]: pgmap v353: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.9 MiB/s wr, 321 op/s 2024-04-26T11:41:12.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 595.0 s, 617.6 tps, lat 412.131 ms stddev 1820.264 2024-04-26T11:41:13.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:13 smithi033 ceph-mon[27845]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T11:41:13.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:13 smithi033 ceph-mon[27845]: pgmap v355: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-04-26T11:41:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:13 smithi096 ceph-mon[32753]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T11:41:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:13 smithi096 ceph-mon[32753]: pgmap v355: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-04-26T11:41:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:13 smithi194 ceph-mon[33017]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T11:41:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:13 smithi194 ceph-mon[33017]: pgmap v355: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-04-26T11:41:14.624 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4ca42a5700 time 2024-04-26T11:41:14.557822+0000 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4cae8ce3dd] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.625 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.626 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.626 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.626 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: clone() 2024-04-26T11:41:14.626 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.558+0000 7f4ca42a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4ca42a5700 time 2024-04-26T11:41:14.557822+0000 2024-04-26T11:41:14.626 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:14.626 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.627 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.627 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4cae8ce3dd] 2024-04-26T11:41:14.627 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.627 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.627 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: clone() 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: *** Caught signal (Aborted) ** 2024-04-26T11:41:14.628 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: in thread 7f4ca42a5700 thread_name:ms_dispatch 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4cad656d20] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: gsignal() 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: abort() 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4cae8ce437] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.629 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 16: clone() 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.560+0000 7f4ca42a5700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: in thread 7f4ca42a5700 thread_name:ms_dispatch 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4cad656d20] 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: gsignal() 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: abort() 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4cae8ce437] 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.630 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 16: clone() 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: -1> 2024-04-26T11:41:14.558+0000 7f4ca42a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4ca42a5700 time 2024-04-26T11:41:14.557822+0000 2024-04-26T11:41:14.631 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4cae8ce3dd] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.632 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: clone() 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 0> 2024-04-26T11:41:14.560+0000 7f4ca42a5700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: in thread 7f4ca42a5700 thread_name:ms_dispatch 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4cad656d20] 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: gsignal() 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: abort() 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4cae8ce437] 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.633 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 16: clone() 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: -1> 2024-04-26T11:41:14.558+0000 7f4ca42a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4ca42a5700 time 2024-04-26T11:41:14.557822+0000 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:14.634 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4cae8ce3dd] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.635 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: clone() 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 0> 2024-04-26T11:41:14.560+0000 7f4ca42a5700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: in thread 7f4ca42a5700 thread_name:ms_dispatch 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4cad656d20] 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2: gsignal() 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 3: abort() 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4cae8ce437] 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f4cae8ce5a3] 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55b8cc7ff822] 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55b8cc76f598] 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b8cc771c17] 2024-04-26T11:41:14.636 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b8cc66ad0b] 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b8cc66b2ac] 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55b8cc653fad] 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f4caeb62e20] 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 13: (DispatchQueue::entry()+0x51f) [0x7f4caeb5ffdf] 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4caec32fc1] 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4cad64c1ca] 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 16: clone() 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[69775]: 2024-04-26T11:41:14.637 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:14 smithi194 podman[70390]: 2024-04-26 11:41:14.623959979 +0000 UTC m=+0.020789425 container died f4589d30da905ff9a562bd9a9b39c6db369d526fd85e42f747f00f3d9f1be02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:41:15.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:15 smithi033 ceph-mon[27845]: pgmap v356: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T11:41:15.835 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:15 smithi194 ceph-mon[33017]: pgmap v356: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T11:41:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-mon[32753]: pgmap v356: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T11:41:15.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f31c6e02700 time 2024-04-26T11:41:15.534670+0000 2024-04-26T11:41:15.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:15.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f31d142b3dd] 2024-04-26T11:41:15.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: clone() 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: *** Caught signal (Aborted) ** 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: in thread 7f31c6e02700 thread_name:ms_dispatch 2024-04-26T11:41:15.934 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.535+0000 7f31c6e02700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f31c6e02700 time 2024-04-26T11:41:15.534670+0000 2024-04-26T11:41:15.935 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:15.935 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.935 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.935 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f31d142b3dd] 2024-04-26T11:41:15.935 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.935 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.935 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: clone() 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f31d01b3d20] 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: gsignal() 2024-04-26T11:41:15.936 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: abort() 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f31d142b437] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 16: clone() 2024-04-26T11:41:15.937 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.537+0000 7f31c6e02700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: in thread 7f31c6e02700 thread_name:ms_dispatch 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f31d01b3d20] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: gsignal() 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: abort() 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f31d142b437] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.938 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.939 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.939 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.939 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.939 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.939 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 16: clone() 2024-04-26T11:41:15.939 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:15.939 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: -1> 2024-04-26T11:41:15.535+0000 7f31c6e02700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f31c6e02700 time 2024-04-26T11:41:15.534670+0000 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f31d142b3dd] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.940 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: clone() 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 0> 2024-04-26T11:41:15.537+0000 7f31c6e02700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: in thread 7f31c6e02700 thread_name:ms_dispatch 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.941 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f31d01b3d20] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: gsignal() 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: abort() 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f31d142b437] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.942 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 16: clone() 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: -4> 2024-04-26T11:41:15.535+0000 7f31c6e02700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f31c6e02700 time 2024-04-26T11:41:15.534670+0000 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f31d142b3dd] 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.943 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: clone() 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: -3> 2024-04-26T11:41:15.537+0000 7f31c6e02700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: in thread 7f31c6e02700 thread_name:ms_dispatch 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f31d01b3d20] 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2: gsignal() 2024-04-26T11:41:15.944 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 3: abort() 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f31d142b437] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f31d142b5a3] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564998d51822] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564998cc1598] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564998cc3c17] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564998bbcd0b] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564998bbd2ac] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564998ba5fad] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f31d16bfe20] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 13: (DispatchQueue::entry()+0x51f) [0x7f31d16bcfdf] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f31d178ffc1] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f31d01a91ca] 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 16: clone() 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:15.945 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[62393]: 2024-04-26T11:41:15.946 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:15 smithi096 podman[70056]: 2024-04-26 11:41:15.604100211 +0000 UTC m=+0.021110141 container died 3d77cf91fa21b8169e895ebe16d137ebfa380aeec6b4e62cf396833ad3facc29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T11:41:16.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:16 smithi194 podman[70390]: 2024-04-26 11:41:16.103168863 +0000 UTC m=+1.499998344 container remove f4589d30da905ff9a562bd9a9b39c6db369d526fd85e42f747f00f3d9f1be02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T11:41:16.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:16 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:41:16.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:16 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-26T11:41:16.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:16 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.127s CPU time. 2024-04-26T11:41:16.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:16 smithi096 podman[70056]: 2024-04-26 11:41:16.660950094 +0000 UTC m=+1.077959987 container remove 3d77cf91fa21b8169e895ebe16d137ebfa380aeec6b4e62cf396833ad3facc29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T11:41:16.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:16 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:41:16.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:16 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-26T11:41:16.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:16 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.b.service: Consumed 1min 7.881s CPU time. 2024-04-26T11:41:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 600.0 s, 963.7 tps, lat 71.179 ms stddev 91.397 2024-04-26T11:41:17.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:17 smithi033 ceph-mon[27845]: pgmap v357: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-26T11:41:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:17 smithi194 ceph-mon[33017]: pgmap v357: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-26T11:41:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:17 smithi096 ceph-mon[32753]: pgmap v357: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-26T11:41:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:19 smithi194 ceph-mon[33017]: pgmap v358: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-26T11:41:19.859 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:19 smithi033 ceph-mon[27845]: pgmap v358: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-26T11:41:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:19 smithi096 ceph-mon[32753]: pgmap v358: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 19 MiB/s wr, 1.49k op/s 2024-04-26T11:41:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 -- 172.21.15.33:0/2116262285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef00077780 msgr2=0x7fef001130a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 --2- 172.21.15.33:0/2116262285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef00077780 0x7fef001130a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feee8009930 tx=0x7feee802a420 comp rx=0 tx=0).stop 2024-04-26T11:41:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 -- 172.21.15.33:0/2116262285 shutdown_connections 2024-04-26T11:41:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 --2- 172.21.15.33:0/2116262285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef00077780 0x7fef001130a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 --2- 172.21.15.33:0/2116262285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef001135e0 0x7fef001159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 --2- 172.21.15.33:0/2116262285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef00076e60 0x7fef00077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 -- 172.21.15.33:0/2116262285 >> 172.21.15.33:0/2116262285 conn(0x7fef00100450 msgr2=0x7fef00102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 -- 172.21.15.33:0/2116262285 shutdown_connections 2024-04-26T11:41:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.689+0000 7fef06600640 1 -- 172.21.15.33:0/2116262285 wait complete. 2024-04-26T11:41:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.691+0000 7fef06600640 1 Processor -- start 2024-04-26T11:41:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.691+0000 7fef06600640 1 -- start start 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.691+0000 7fef06600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00076e60 0x7fef0019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef06600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef00077780 0x7fef0019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef06600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef001135e0 0x7fef001a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef06600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fef00119030 con 0x7fef00076e60 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef06600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef00118eb0 con 0x7fef001135e0 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef06600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fef001191b0 con 0x7fef00077780 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef05200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00076e60 0x7fef0019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef05c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef001135e0 0x7fef001a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7feef7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef00077780 0x7fef0019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef05200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00076e60 0x7fef0019f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:32836/0 (socket says 172.21.15.33:32836) 2024-04-26T11:41:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 learned_addr learned my addr 172.21.15.33:0/848178078 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7feef7e00640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef001135e0 msgr2=0x7fef001a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7feef7e00640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef001135e0 0x7fef001a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7feef7e00640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00076e60 msgr2=0x7fef0019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7feef7e00640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00076e60 0x7fef0019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7feef7e00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef001a4570 con 0x7fef00077780 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.692+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fef00076e60 0x7fef0019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.693+0000 7feef7e00640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef00077780 0x7fef0019fb10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feee802a940 tx=0x7feee8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.693+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee803f070 con 0x7fef00077780 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.693+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feee8031b60 con 0x7fef00077780 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.693+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8035030 con 0x7fef00077780 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.693+0000 7fef06600640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef001a4800 con 0x7fef00077780 2024-04-26T11:41:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.693+0000 7fef06600640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef001a4c70 con 0x7fef00077780 2024-04-26T11:41:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 Processor -- start 2024-04-26T11:41:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7feee8042070 con 0x7fef00077780 2024-04-26T11:41:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 -- start start 2024-04-26T11:41:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564385820fe0 0x5643858213c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385821900 0x56438582bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56438582c1c0 0x56438582e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564385786380 con 0x564385821900 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564385722c30 con 0x56438582c1c0 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.697+0000 7fef0a9fa740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564385782340 con 0x564385820fe0 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7feef7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56438582c1c0 0x56438582e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564385820fe0 0x5643858213c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385821900 0x56438582bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7feef7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56438582c1c0 0x56438582e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45538/0 (socket says 172.21.15.33:45538) 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564385820fe0 0x5643858213c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.33:35576/0 (socket says 172.21.15.33:35576) 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7feef7e00640 1 -- 172.21.15.33:0/455470189 learned_addr learned my addr 172.21.15.33:0/455470189 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 -- 172.21.15.33:0/455470189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56438582c1c0 msgr2=0x56438582e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56438582c1c0 0x56438582e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 -- 172.21.15.33:0/455470189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385821900 msgr2=0x56438582bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385821900 0x56438582bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 -- 172.21.15.33:0/455470189 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564385744940 con 0x564385820fe0 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05200640 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385821900 0x56438582bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.698+0000 7fef05c00640 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564385820fe0 0x5643858213c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feef8005fd0 tx=0x7feef8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58092ae9a39d868b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.699+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.699+0000 7feef6000640 1 -- 172.21.15.33:0/455470189 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8004d20 con 0x564385820fe0 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.699+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 10090+0+0 (secure 0 0 0) 0x7feee80f1360 con 0x7fef00077780 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.699+0000 7feef6000640 1 -- 172.21.15.33:0/455470189 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef8004050 con 0x564385820fe0 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.699+0000 7feef6000640 1 -- 172.21.15.33:0/455470189 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8005280 con 0x564385820fe0 2024-04-26T11:41:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.699+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 -- 172.21.15.33:0/455470189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564385820fe0 msgr2=0x5643858213c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564385820fe0 0x5643858213c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feef8005fd0 tx=0x7feef8011810 comp rx=0 tx=0).stop 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40a61d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feef0009620 tx=0x7feef0005f90 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 -- 172.21.15.33:0/455470189 shutdown_connections 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56438582c1c0 0x56438582e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385821900 0x56438582bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 --2- 172.21.15.33:0/455470189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564385820fe0 0x5643858213c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 -- 172.21.15.33:0/455470189 >> 172.21.15.33:0/455470189 conn(0x56438581d4f0 msgr2=0x56438581d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 -- 172.21.15.33:0/455470189 shutdown_connections 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.700+0000 7fef0a9fa740 1 -- 172.21.15.33:0/455470189 wait complete. 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 -- start start 2024-04-26T11:41:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 Processor -- start 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385820fe0 0x5643858290e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564385821900 0x564385829620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56438582c1c0 0x564385829b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56438582edf0 con 0x564385820fe0 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56438570dc60 con 0x564385821900 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56438582ef70 con 0x56438582c1c0 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.701+0000 7fef0a9fa740 10 client.0 _ll_register_callbacks cb 0x564385739e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7fef05200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385820fe0 0x5643858290e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feef7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564385821900 0x564385829620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7fef05200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385820fe0 0x5643858290e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:32840/0 (socket says 172.21.15.33:32840) 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7fef05c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56438582c1c0 0x564385829b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 learned_addr learned my addr 172.21.15.33:0/3958398144 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feef7e00640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56438582c1c0 msgr2=0x564385829b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feef7e00640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56438582c1c0 0x564385829b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feef7e00640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385820fe0 msgr2=0x5643858290e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feef7e00640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564385820fe0 0x5643858290e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feef7e00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56438571c510 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feef7e00640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564385821900 0x564385829620 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7feee80bca30 tx=0x7feee803b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3016434a4adf47a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feeeea00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8049070 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 10090+0+0 (secure 0 0 0) 0x7feee80b93a0 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.702+0000 7fef0a9fa740 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef8004050 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=158) ==== 24+0+0 (secure 0 0 0) 0x7feee80bcb00 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 10 client.26084 fetch_fsmap learned FSMap version 158 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=158,monmap=4+}) -- 0x564385498210 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef8004050 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 158) ==== 7132+0+0 (secure 0 0 0) 0x7feee80ee6b0 con 0x564385821900 2024-04-26T11:41:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 10 client.26084 fetch_fsmap finished waiting for FSMap version 158 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 10 client.26084 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 20 client.26084 populate_metadata read hostname 'smithi033' 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 4 client.26084 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef0a9fa740 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564385793f90 con 0x56438583bed0 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.703+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T11:41:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:21 smithi194 ceph-mon[33017]: pgmap v359: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 20 MiB/s wr, 1.48k op/s 2024-04-26T11:41:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:21 smithi194 ceph-mon[33017]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T11:41:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.905+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.905+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T11:41:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:21.905+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:21 smithi033 ceph-mon[27845]: pgmap v359: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 20 MiB/s wr, 1.48k op/s 2024-04-26T11:41:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:21 smithi033 ceph-mon[27845]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T11:41:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:21 smithi096 ceph-mon[32753]: pgmap v359: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 20 MiB/s wr, 1.48k op/s 2024-04-26T11:41:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:21 smithi096 ceph-mon[32753]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T11:41:22.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 605.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:41:22.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:22.306+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:22.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:22.306+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:22.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:22.306+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T11:41:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:23.107+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:23.107+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:23.107+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T11:41:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:23 smithi194 ceph-mon[33017]: pgmap v361: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 20 MiB/s wr, 1.48k op/s 2024-04-26T11:41:23.898 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:23 smithi033 ceph-mon[27845]: pgmap v361: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 20 MiB/s wr, 1.48k op/s 2024-04-26T11:41:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:23 smithi096 ceph-mon[32753]: pgmap v361: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 20 MiB/s wr, 1.48k op/s 2024-04-26T11:41:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:24.709+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:24.709+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:24.709+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T11:41:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:25 smithi194 ceph-mon[33017]: pgmap v362: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 681 op/s 2024-04-26T11:41:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:25 smithi033 ceph-mon[27845]: pgmap v362: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 681 op/s 2024-04-26T11:41:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:25 smithi096 ceph-mon[32753]: pgmap v362: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 681 op/s 2024-04-26T11:41:26.870 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:26 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:41:26.871 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:26 smithi194 systemd[1]: Stopped Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:26.871 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:26 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.127s CPU time. 2024-04-26T11:41:26.871 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:26 smithi194 systemd[1]: Starting Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:41:26.871 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:26 smithi194 podman[70486]: 2024-04-26 11:41:26.73969217 +0000 UTC m=+0.018515321 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:41:27.141 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:26 smithi194 podman[70486]: 2024-04-26 11:41:26.870578933 +0000 UTC m=+0.149402046 container create b67d49f3bfa69c05a8f094d14034a43d137c8e1c3a4de8a3bde7fff8d00b0be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5) 2024-04-26T11:41:27.141 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:27 smithi194 podman[70486]: 2024-04-26 11:41:27.115130066 +0000 UTC m=+0.393953190 container init b67d49f3bfa69c05a8f094d14034a43d137c8e1c3a4de8a3bde7fff8d00b0be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T11:41:27.141 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:27 smithi194 podman[70486]: 2024-04-26 11:41:27.117588259 +0000 UTC m=+0.396411381 container start b67d49f3bfa69c05a8f094d14034a43d137c8e1c3a4de8a3bde7fff8d00b0be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T11:41:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 610.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:41:27.275 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:26 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:41:27.276 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:26 smithi096 systemd[1]: Stopped Ceph mds.b for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:27.276 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:26 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.b.service: Consumed 1min 7.881s CPU time. 2024-04-26T11:41:27.276 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:26 smithi096 systemd[1]: Starting Ceph mds.b for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:41:27.276 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 podman[70151]: 2024-04-26 11:41:27.06289991 +0000 UTC m=+0.017737045 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:41:27.276 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 podman[70151]: 2024-04-26 11:41:27.161483903 +0000 UTC m=+0.116321030 container create c465ddc9a31af43aa5a731b086da0faa3300905394b1efff5831f7567bb55914 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T11:41:27.403 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mds[70505]: -- 172.21.15.194:0/384894223 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55773507da40 con 0x557735050800 2024-04-26T11:41:27.403 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: starting mds.l at 2024-04-26T11:41:27.403 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:27 smithi194 bash[70486]: b67d49f3bfa69c05a8f094d14034a43d137c8e1c3a4de8a3bde7fff8d00b0be3 2024-04-26T11:41:27.403 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:41:27 smithi194 systemd[1]: Started Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:27.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 podman[70151]: 2024-04-26 11:41:27.397597384 +0000 UTC m=+0.352434511 container init c465ddc9a31af43aa5a731b086da0faa3300905394b1efff5831f7567bb55914 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T11:41:27.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 podman[70151]: 2024-04-26 11:41:27.400127165 +0000 UTC m=+0.354964295 container start c465ddc9a31af43aa5a731b086da0faa3300905394b1efff5831f7567bb55914 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=) 2024-04-26T11:41:27.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mds[70170]: -- 172.21.15.96:0/3658237376 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5613f9d3fa40 con 0x5613f9d12c00 2024-04-26T11:41:27.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: starting mds.b at 2024-04-26T11:41:27.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 bash[70151]: c465ddc9a31af43aa5a731b086da0faa3300905394b1efff5831f7567bb55914 2024-04-26T11:41:27.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:41:27 smithi096 systemd[1]: Started Ceph mds.b for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: pgmap v363: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.6 MiB/s wr, 248 op/s 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: daemon mds.l restarted 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:boot 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: daemon mds.b restarted 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:boot 2024-04-26T11:41:27.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-26T11:41:27.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:27 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {1=k=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: pgmap v363: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.6 MiB/s wr, 248 op/s 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: daemon mds.l restarted 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:boot 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: daemon mds.b restarted 2024-04-26T11:41:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:41:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T11:41:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:boot 2024-04-26T11:41:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-26T11:41:27.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:27 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {1=k=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-26T11:41:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:27.912+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:27.912+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:27.912+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T11:41:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: pgmap v363: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.6 MiB/s wr, 248 op/s 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: daemon mds.l restarted 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:boot 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: daemon mds.b restarted 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:boot 2024-04-26T11:41:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-26T11:41:27.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:27 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {1=k=up:replay,2=i=up:active} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-26T11:41:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:28 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=c=up:replay,1=k=up:replay,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:28 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=c=up:replay,1=k=up:replay,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:28 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=c=up:replay,1=k=up:replay,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:29.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:29 smithi033 ceph-mon[27845]: pgmap v366: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 579 KiB/s wr, 75 op/s 2024-04-26T11:41:29.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:29 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:resolve 2024-04-26T11:41:29.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:29 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:resolve 2024-04-26T11:41:29.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:29 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=c=up:resolve,1=k=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:29 smithi194 ceph-mon[33017]: pgmap v366: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 579 KiB/s wr, 75 op/s 2024-04-26T11:41:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:29 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:resolve 2024-04-26T11:41:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:29 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:resolve 2024-04-26T11:41:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:29 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=c=up:resolve,1=k=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:29 smithi096 ceph-mon[32753]: pgmap v366: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 579 KiB/s wr, 75 op/s 2024-04-26T11:41:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:29 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:resolve 2024-04-26T11:41:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:29 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:resolve 2024-04-26T11:41:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:29 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=c=up:resolve,1=k=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:30 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:41:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:30 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:41:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:30 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:reconnect 2024-04-26T11:41:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:30 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:reconnect 2024-04-26T11:41:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:30 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=c=up:reconnect,1=k=up:reconnect,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:30 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0189996 2024-04-26T11:41:30.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:30 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.357992 2024-04-26T11:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:30 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:30 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:30 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:reconnect 2024-04-26T11:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:30 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:reconnect 2024-04-26T11:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:30 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=c=up:reconnect,1=k=up:reconnect,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:30 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0189996 2024-04-26T11:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:30 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.357992 2024-04-26T11:41:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:30 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:41:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:30 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:41:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:30 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:reconnect 2024-04-26T11:41:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:30 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:reconnect 2024-04-26T11:41:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:30 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=c=up:reconnect,1=k=up:reconnect,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:30 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0189996 2024-04-26T11:41:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:30 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.357992 2024-04-26T11:41:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: pgmap v367: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 967 KiB/s wr, 127 op/s 2024-04-26T11:41:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:rejoin 2024-04-26T11:41:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:rejoin 2024-04-26T11:41:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=c=up:rejoin,1=k=up:rejoin,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-26T11:41:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:41:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:41:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:active 2024-04-26T11:41:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:active 2024-04-26T11:41:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:31 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=c=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: pgmap v367: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 967 KiB/s wr, 127 op/s 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:rejoin 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:rejoin 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=c=up:rejoin,1=k=up:rejoin,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:active 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:active 2024-04-26T11:41:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:31 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=c=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: pgmap v367: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 967 KiB/s wr, 127 op/s 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:rejoin 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:rejoin 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=c=up:rejoin,1=k=up:rejoin,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1265312544,v1:172.21.15.194:6837/1265312544] up:active 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/466401967,v1:172.21.15.96:6833/466401967] up:active 2024-04-26T11:41:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:31 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=c=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 615.0 s, 376.4 tps, lat 880.947 ms stddev 3556.541 2024-04-26T11:41:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:33 smithi194 ceph-mon[33017]: pgmap v368: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 651 KiB/s wr, 86 op/s 2024-04-26T11:41:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:33 smithi033 ceph-mon[27845]: pgmap v368: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 651 KiB/s wr, 86 op/s 2024-04-26T11:41:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:33 smithi096 ceph-mon[32753]: pgmap v368: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 651 KiB/s wr, 86 op/s 2024-04-26T11:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:34.319+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:34.320 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:34.319+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:34.320 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:34.319+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-26T11:41:34.604 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:34 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T11:41:34.604 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:34 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:41:34.604 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:34 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T11:41:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:35 smithi194 ceph-mon[33017]: pgmap v369: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 6.7 MiB/s wr, 256 op/s 2024-04-26T11:41:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:35 smithi033 ceph-mon[27845]: pgmap v369: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 6.7 MiB/s wr, 256 op/s 2024-04-26T11:41:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:35 smithi096 ceph-mon[32753]: pgmap v369: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 6.7 MiB/s wr, 256 op/s 2024-04-26T11:41:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=c=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fead3de9700 time 2024-04-26T11:41:36.396284+0000 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feade4123dd] 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.683 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: clone() 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: *** Caught signal (Aborted) ** 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: in thread 7fead3de9700 thread_name:ms_dispatch 2024-04-26T11:41:36.684 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.396+0000 7fead3de9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fead3de9700 time 2024-04-26T11:41:36.396284+0000 2024-04-26T11:41:36.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:36.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feade4123dd] 2024-04-26T11:41:36.685 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: clone() 2024-04-26T11:41:36.686 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feadd19ad20] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: gsignal() 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: abort() 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feade412437] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.687 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 16: clone() 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.398+0000 7fead3de9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: in thread 7fead3de9700 thread_name:ms_dispatch 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feadd19ad20] 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: gsignal() 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: abort() 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feade412437] 2024-04-26T11:41:36.688 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 16: clone() 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: -1> 2024-04-26T11:41:36.396+0000 7fead3de9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fead3de9700 time 2024-04-26T11:41:36.396284+0000 2024-04-26T11:41:36.689 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feade4123dd] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.690 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: clone() 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 0> 2024-04-26T11:41:36.398+0000 7fead3de9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: in thread 7fead3de9700 thread_name:ms_dispatch 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feadd19ad20] 2024-04-26T11:41:36.691 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: gsignal() 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: abort() 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feade412437] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.692 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 16: clone() 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: -1> 2024-04-26T11:41:36.396+0000 7fead3de9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fead3de9700 time 2024-04-26T11:41:36.396284+0000 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feade4123dd] 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.693 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: clone() 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 0> 2024-04-26T11:41:36.398+0000 7fead3de9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: in thread 7fead3de9700 thread_name:ms_dispatch 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feadd19ad20] 2024-04-26T11:41:36.694 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2: gsignal() 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 3: abort() 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feade412437] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feade4125a3] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557b02fb1822] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557b02f21598] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557b02f23c17] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557b02e1cd0b] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557b02e1d2ac] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557b02e05fad] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7feade6a6e20] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 13: (DispatchQueue::entry()+0x51f) [0x7feade6a3fdf] 2024-04-26T11:41:36.695 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7feade776fc1] 2024-04-26T11:41:36.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feadd1901ca] 2024-04-26T11:41:36.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 16: clone() 2024-04-26T11:41:36.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:36.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70005]: 2024-04-26T11:41:36.696 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:36 smithi096 podman[70207]: 2024-04-26 11:41:36.496074689 +0000 UTC m=+0.030445683 container died 98aadb1d70854a2155b4ea61a2e0a9a1687a5e75c30ab6e793c976e29ddabb34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5) 2024-04-26T11:41:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:36 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=c=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:36 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=c=up:active,1=k=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:41:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 620.0 s, 2085.1 tps, lat 46.356 ms stddev 54.259 2024-04-26T11:41:37.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:37 smithi096 ceph-mon[32753]: pgmap v370: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 7.5 MiB/s wr, 314 op/s 2024-04-26T11:41:37.583 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:37 smithi096 podman[70207]: 2024-04-26 11:41:37.304227697 +0000 UTC m=+0.838598680 container remove 98aadb1d70854a2155b4ea61a2e0a9a1687a5e75c30ab6e793c976e29ddabb34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:41:37.583 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:37 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:41:37.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:37 smithi194 ceph-mon[33017]: pgmap v370: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 7.5 MiB/s wr, 314 op/s 2024-04-26T11:41:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:37 smithi033 ceph-mon[27845]: pgmap v370: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 7.5 MiB/s wr, 314 op/s 2024-04-26T11:41:37.932 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:37 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-26T11:41:37.932 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:37 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.200s CPU time. 2024-04-26T11:41:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:39 smithi194 ceph-mon[33017]: pgmap v371: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 6.4 MiB/s wr, 265 op/s 2024-04-26T11:41:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:39 smithi033 ceph-mon[27845]: pgmap v371: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 6.4 MiB/s wr, 265 op/s 2024-04-26T11:41:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:39 smithi096 ceph-mon[32753]: pgmap v371: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 6.4 MiB/s wr, 265 op/s 2024-04-26T11:41:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:41 smithi194 ceph-mon[33017]: pgmap v372: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 10 MiB/s wr, 444 op/s 2024-04-26T11:41:41.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:41 smithi033 ceph-mon[27845]: pgmap v372: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 10 MiB/s wr, 444 op/s 2024-04-26T11:41:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:41 smithi096 ceph-mon[32753]: pgmap v372: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 10 MiB/s wr, 444 op/s 2024-04-26T11:41:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 625.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:41:43.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3bea7fb700 time 2024-04-26T11:41:42.981408+0000 2024-04-26T11:41:43.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:43.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf4e243dd] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: clone() 2024-04-26T11:41:43.356 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: *** Caught signal (Aborted) ** 2024-04-26T11:41:43.356 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: in thread 7f3bea7fb700 thread_name:ms_dispatch 2024-04-26T11:41:43.356 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:42.982+0000 7f3bea7fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3bea7fb700 time 2024-04-26T11:41:42.981408+0000 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf4e243dd] 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.358 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: clone() 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3bacd20] 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: gsignal() 2024-04-26T11:41:43.359 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: abort() 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf4e24437] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.360 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 16: clone() 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:42.984+0000 7f3bea7fb700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: in thread 7f3bea7fb700 thread_name:ms_dispatch 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3bacd20] 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: gsignal() 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: abort() 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf4e24437] 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.361 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 16: clone() 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1734> 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1732> 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1730> 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1> 2024-04-26T11:41:42.982+0000 7f3bea7fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3bea7fb700 time 2024-04-26T11:41:42.981408+0000 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:43.362 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf4e243dd] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.363 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: clone() 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 0> 2024-04-26T11:41:42.984+0000 7f3bea7fb700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: in thread 7f3bea7fb700 thread_name:ms_dispatch 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3bacd20] 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: gsignal() 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: abort() 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf4e24437] 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.364 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 16: clone() 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1734> 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1732> 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1730> 2024-04-26T11:41:34.227+0000 7f3be7ff6700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T11:41:43.365 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: -1> 2024-04-26T11:41:42.982+0000 7f3bea7fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3bea7fb700 time 2024-04-26T11:41:42.981408+0000 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf4e243dd] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.366 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: clone() 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 0> 2024-04-26T11:41:42.984+0000 7f3bea7fb700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: in thread 7f3bea7fb700 thread_name:ms_dispatch 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3bacd20] 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2: gsignal() 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 3: abort() 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf4e24437] 2024-04-26T11:41:43.367 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3bf4e245a3] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55719ad0a822] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55719ac7a598] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55719ac7cc17] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55719ab75d0b] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55719ab762ac] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55719ab5efad] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f3bf50b8e20] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf50b5fdf] 2024-04-26T11:41:43.368 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf5188fc1] 2024-04-26T11:41:43.369 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf3ba21ca] 2024-04-26T11:41:43.369 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 16: clone() 2024-04-26T11:41:43.369 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:41:43.369 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[64127]: 2024-04-26T11:41:43.369 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 podman[70565]: 2024-04-26 11:41:43.09142096 +0000 UTC m=+0.042002121 container died 6d29997d79c4238c9474081edc60fe900af09f1f442008b8ead04052a404bc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:41:43.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:43 smithi033 ceph-mon[27845]: pgmap v373: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 9.8 MiB/s wr, 404 op/s 2024-04-26T11:41:43.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:43 smithi194 ceph-mon[33017]: pgmap v373: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 9.8 MiB/s wr, 404 op/s 2024-04-26T11:41:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:43 smithi096 ceph-mon[32753]: pgmap v373: 97 pgs: 97 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 9.8 MiB/s wr, 404 op/s 2024-04-26T11:41:44.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 podman[70565]: 2024-04-26 11:41:43.713773255 +0000 UTC m=+0.664354404 container remove 6d29997d79c4238c9474081edc60fe900af09f1f442008b8ead04052a404bc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-26T11:41:44.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:41:44.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T11:41:44.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:43 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 2.976s CPU time. 2024-04-26T11:41:45.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:45 smithi033 ceph-mon[27845]: pgmap v374: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 21 MiB/s wr, 1.07k op/s 2024-04-26T11:41:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:45 smithi194 ceph-mon[33017]: pgmap v374: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 21 MiB/s wr, 1.07k op/s 2024-04-26T11:41:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:45 smithi096 ceph-mon[32753]: pgmap v374: 97 pgs: 97 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 21 MiB/s wr, 1.07k op/s 2024-04-26T11:41:47.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:47.128+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:47.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:47.128+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:41:47.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:41:47.128+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:41:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 630.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:41:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:47 smithi194 ceph-mon[33017]: pgmap v375: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 25 MiB/s wr, 1.39k op/s 2024-04-26T11:41:47.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:47 smithi033 ceph-mon[27845]: pgmap v375: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 25 MiB/s wr, 1.39k op/s 2024-04-26T11:41:47.923 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:47 smithi096 ceph-mon[32753]: pgmap v375: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 25 MiB/s wr, 1.39k op/s 2024-04-26T11:41:47.923 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:47 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:41:47.923 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:47 smithi096 systemd[1]: Stopped Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:47.923 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:47 smithi096 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.200s CPU time. 2024-04-26T11:41:47.923 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:47 smithi096 systemd[1]: Starting Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:41:48.183 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:47 smithi096 podman[70320]: 2024-04-26 11:41:47.826215649 +0000 UTC m=+0.018279573 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:41:48.183 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:47 smithi096 podman[70320]: 2024-04-26 11:41:47.938306374 +0000 UTC m=+0.130370280 container create 4829d4d49a43e81a3edf9c0202f9010776adebf00e115fd438abd85d2acdce20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-26T11:41:48.183 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:48 smithi096 podman[70320]: 2024-04-26 11:41:48.166412923 +0000 UTC m=+0.358476829 container init 4829d4d49a43e81a3edf9c0202f9010776adebf00e115fd438abd85d2acdce20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T11:41:48.183 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:48 smithi096 podman[70320]: 2024-04-26 11:41:48.168951601 +0000 UTC m=+0.361015521 container start 4829d4d49a43e81a3edf9c0202f9010776adebf00e115fd438abd85d2acdce20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:41:48.583 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mds[70339]: -- 172.21.15.96:0/794099449 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8f863fa40 con 0x55a8f8612c00 2024-04-26T11:41:48.584 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-k[70335]: starting mds.k at 2024-04-26T11:41:48.584 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:48 smithi096 bash[70320]: 4829d4d49a43e81a3edf9c0202f9010776adebf00e115fd438abd85d2acdce20 2024-04-26T11:41:48.584 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 26 11:41:48 smithi096 systemd[1]: Started Ceph mds.k for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: daemon mds.k restarted 2024-04-26T11:41:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:41:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T11:41:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6832/3136378694,v1:172.21.15.96:6833/3136378694] up:boot 2024-04-26T11:41:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:41:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=c=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:48 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=c=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: daemon mds.k restarted 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6832/3136378694,v1:172.21.15.96:6833/3136378694] up:boot 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=c=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:48 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=c=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: daemon mds.k restarted 2024-04-26T11:41:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:41:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T11:41:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T11:41:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6832/3136378694,v1:172.21.15.96:6833/3136378694] up:boot 2024-04-26T11:41:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:41:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=c=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:41:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:48 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=c=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:49 smithi194 ceph-mon[33017]: pgmap v377: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 1.58k op/s 2024-04-26T11:41:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:49 smithi033 ceph-mon[27845]: pgmap v377: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 1.58k op/s 2024-04-26T11:41:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:49 smithi096 ceph-mon[32753]: pgmap v377: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 1.58k op/s 2024-04-26T11:41:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:50 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:resolve 2024-04-26T11:41:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:50 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=c=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:50 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:resolve 2024-04-26T11:41:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:50 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=c=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:50 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:resolve 2024-04-26T11:41:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:50 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=c=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:41:51.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:51 smithi033 ceph-mon[27845]: pgmap v378: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T11:41:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:51 smithi096 ceph-mon[32753]: pgmap v378: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T11:41:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:51 smithi194 ceph-mon[33017]: pgmap v378: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T11:41:52.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 635.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:41:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:53 smithi033 ceph-mon[27845]: pgmap v379: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T11:41:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:53 smithi096 ceph-mon[32753]: pgmap v379: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T11:41:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:53 smithi194 ceph-mon[33017]: pgmap v379: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T11:41:54.373 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:41:54.373 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 systemd[1]: Stopped Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:54.373 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 2.976s CPU time. 2024-04-26T11:41:54.374 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 systemd[1]: Starting Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:41:54.374 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 podman[70661]: 2024-04-26 11:41:54.244047899 +0000 UTC m=+0.019052699 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:41:54.374 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 podman[70661]: 2024-04-26 11:41:54.373236446 +0000 UTC m=+0.148241220 container create 00de932448a3db59ea9f658f4af88c1c476325b79672c1f1272557335b034ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-26T11:41:54.635 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 podman[70661]: 2024-04-26 11:41:54.609778789 +0000 UTC m=+0.384783574 container init 00de932448a3db59ea9f658f4af88c1c476325b79672c1f1272557335b034ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T11:41:54.635 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 podman[70661]: 2024-04-26 11:41:54.612327009 +0000 UTC m=+0.387331785 container start 00de932448a3db59ea9f658f4af88c1c476325b79672c1f1272557335b034ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:41:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 ceph-mds[70680]: -- 172.21.15.194:0/1051856468 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562dab659a40 con 0x562dab62d000 2024-04-26T11:41:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: starting mds.c at 2024-04-26T11:41:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 bash[70661]: 00de932448a3db59ea9f658f4af88c1c476325b79672c1f1272557335b034ec6 2024-04-26T11:41:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:41:54 smithi194 systemd[1]: Started Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:41:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: pgmap v380: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 538 op/s 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: daemon mds.c restarted 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:boot 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {1=l=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:replay,1=l=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:55.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:41:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: pgmap v380: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 538 op/s 2024-04-26T11:41:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: daemon mds.c restarted 2024-04-26T11:41:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:41:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T11:41:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:boot 2024-04-26T11:41:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T11:41:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {1=l=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-26T11:41:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:replay,1=l=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: pgmap v380: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 538 op/s 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: daemon mds.c restarted 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:boot 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {1=l=up:resolve,2=i=up:active} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-26T11:41:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:replay,1=l=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:56.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:41:56.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:56 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:resolve 2024-04-26T11:41:56.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:56 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:resolve,1=l=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:56.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:41:56.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:41:56.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:41:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:56 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:resolve 2024-04-26T11:41:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:56 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:resolve,1=l=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:41:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:41:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:41:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:56 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:resolve 2024-04-26T11:41:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:56 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:resolve,1=l=up:resolve,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:41:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:41:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:41:57.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 640.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:41:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: pgmap v382: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-26T11:41:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: pgmap v383: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 45 KiB/s wr, 11 op/s 2024-04-26T11:41:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999979 2024-04-26T11:41:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:reconnect 2024-04-26T11:41:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:reconnect 2024-04-26T11:41:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:reconnect,1=l=up:reconnect,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.0169997 2024-04-26T11:41:58.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0189996 2024-04-26T11:41:58.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:57 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0199996 2024-04-26T11:41:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: pgmap v382: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-26T11:41:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: pgmap v383: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 45 KiB/s wr, 11 op/s 2024-04-26T11:41:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999979 2024-04-26T11:41:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:reconnect 2024-04-26T11:41:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:reconnect 2024-04-26T11:41:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:reconnect,1=l=up:reconnect,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.0169997 2024-04-26T11:41:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0189996 2024-04-26T11:41:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:57 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0199996 2024-04-26T11:41:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: pgmap v382: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: pgmap v383: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 45 KiB/s wr, 11 op/s 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999979 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:reconnect 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:reconnect 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:reconnect,1=l=up:reconnect,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.0169997 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0189996 2024-04-26T11:41:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:57 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0199996 2024-04-26T11:41:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:rejoin 2024-04-26T11:41:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:rejoin 2024-04-26T11:41:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:58 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:rejoin,1=l=up:rejoin,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:58 smithi194 ceph-mon[33017]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T11:41:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:58 smithi194 ceph-mon[33017]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:41:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:rejoin 2024-04-26T11:41:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:rejoin 2024-04-26T11:41:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:58 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:rejoin,1=l=up:rejoin,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:58 smithi033 ceph-mon[27845]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T11:41:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:58 smithi033 ceph-mon[27845]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:rejoin 2024-04-26T11:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:rejoin 2024-04-26T11:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:58 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:rejoin,1=l=up:rejoin,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:41:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:58 smithi096 ceph-mon[32753]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T11:41:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:58 smithi096 ceph-mon[32753]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:59 smithi194 ceph-mon[33017]: pgmap v384: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-26T11:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:59 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:active 2024-04-26T11:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:active 2024-04-26T11:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:59 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:41:59 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:59 smithi033 ceph-mon[27845]: pgmap v384: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-26T11:42:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:59 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:active 2024-04-26T11:42:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:active 2024-04-26T11:42:00.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:59 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:42:00.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:41:59 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:59 smithi096 ceph-mon[32753]: pgmap v384: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-26T11:42:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:59 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:active 2024-04-26T11:42:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/1771392146,v1:172.21.15.194:6833/1771392146] up:active 2024-04-26T11:42:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:59 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 1 up:standby-replay 6 up:standby 2024-04-26T11:42:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:41:59 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:01 smithi194 ceph-mon[33017]: pgmap v385: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 692 KiB/s wr, 50 op/s 2024-04-26T11:42:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:02.143+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:02.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:02.143+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:02.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:02.143+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:42:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:01 smithi033 ceph-mon[27845]: pgmap v385: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 692 KiB/s wr, 50 op/s 2024-04-26T11:42:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:01 smithi096 ceph-mon[32753]: pgmap v385: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 692 KiB/s wr, 50 op/s 2024-04-26T11:42:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 645.0 s, 1259.1 tps, lat 399.827 ms stddev 2729.990 2024-04-26T11:42:02.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:42:02 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:42:02.465+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T11:42:02.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:42:02 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:42:02.465+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:42:02.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:42:02 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:42:02.465+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T11:42:04.074 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:03 smithi033 ceph-mon[27845]: pgmap v386: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-04-26T11:42:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-mon[33017]: pgmap v386: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-04-26T11:42:04.105 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0035c97700 time 2024-04-26T11:42:03.720115+0000 2024-04-26T11:42:04.105 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:04.105 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.105 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00402c03dd] 2024-04-26T11:42:04.106 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.106 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.106 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.106 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.106 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.106 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.107 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.107 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.107 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.107 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.107 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.107 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: clone() 2024-04-26T11:42:04.108 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:03.720+0000 7f0035c97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0035c97700 time 2024-04-26T11:42:03.720115+0000 2024-04-26T11:42:04.108 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:04.108 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.108 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.108 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00402c03dd] 2024-04-26T11:42:04.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.110 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.110 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.110 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.110 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.110 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: clone() 2024-04-26T11:42:04.110 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: *** Caught signal (Aborted) ** 2024-04-26T11:42:04.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: in thread 7f0035c97700 thread_name:ms_dispatch 2024-04-26T11:42:04.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f003f048d20] 2024-04-26T11:42:04.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: gsignal() 2024-04-26T11:42:04.111 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: abort() 2024-04-26T11:42:04.112 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00402c0437] 2024-04-26T11:42:04.112 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.112 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.112 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.112 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.112 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.114 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.114 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.114 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.115 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.115 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.115 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.115 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 16: clone() 2024-04-26T11:42:04.115 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:03.721+0000 7f0035c97700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:04.115 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: in thread 7f0035c97700 thread_name:ms_dispatch 2024-04-26T11:42:04.116 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.116 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.116 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f003f048d20] 2024-04-26T11:42:04.116 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: gsignal() 2024-04-26T11:42:04.116 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: abort() 2024-04-26T11:42:04.116 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00402c0437] 2024-04-26T11:42:04.117 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.117 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.117 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.117 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.117 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.117 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.117 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.118 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.118 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.118 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.118 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.118 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 16: clone() 2024-04-26T11:42:04.119 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:04.119 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.119 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: -1> 2024-04-26T11:42:03.720+0000 7f0035c97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0035c97700 time 2024-04-26T11:42:03.720115+0000 2024-04-26T11:42:04.119 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:04.119 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.119 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.119 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00402c03dd] 2024-04-26T11:42:04.120 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.120 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.120 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.120 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.120 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.121 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.121 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.121 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.121 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.121 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.121 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.121 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: clone() 2024-04-26T11:42:04.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 0> 2024-04-26T11:42:03.721+0000 7f0035c97700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:04.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: in thread 7f0035c97700 thread_name:ms_dispatch 2024-04-26T11:42:04.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f003f048d20] 2024-04-26T11:42:04.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: gsignal() 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: abort() 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00402c0437] 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 16: clone() 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.124 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: -1> 2024-04-26T11:42:03.720+0000 7f0035c97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0035c97700 time 2024-04-26T11:42:03.720115+0000 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f00402c03dd] 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.125 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.126 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.126 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.126 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.126 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.126 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.126 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: clone() 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 0> 2024-04-26T11:42:03.721+0000 7f0035c97700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: in thread 7f0035c97700 thread_name:ms_dispatch 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f003f048d20] 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2: gsignal() 2024-04-26T11:42:04.127 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 3: abort() 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f00402c0437] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f00402c05a3] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557732995822] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557732905598] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557732907c17] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557732800d0b] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5577328012ac] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5577327e9fad] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0040554e20] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 13: (DispatchQueue::entry()+0x51f) [0x7f0040551fdf] 2024-04-26T11:42:04.128 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0040624fc1] 2024-04-26T11:42:04.129 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f003f03e1ca] 2024-04-26T11:42:04.129 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 16: clone() 2024-04-26T11:42:04.129 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:04.129 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70501]: 2024-04-26T11:42:04.129 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:03 smithi194 podman[70852]: 2024-04-26 11:42:03.815662456 +0000 UTC m=+0.051063812 container died b67d49f3bfa69c05a8f094d14034a43d137c8e1c3a4de8a3bde7fff8d00b0be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T11:42:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:03 smithi096 ceph-mon[32753]: pgmap v386: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-04-26T11:42:05.033 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:04 smithi194 podman[70852]: 2024-04-26 11:42:04.757855612 +0000 UTC m=+0.993256944 container remove b67d49f3bfa69c05a8f094d14034a43d137c8e1c3a4de8a3bde7fff8d00b0be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T11:42:05.033 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:04 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:42:05.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:05 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-26T11:42:05.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:05 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.176s CPU time. 2024-04-26T11:42:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:05 smithi194 ceph-mon[33017]: pgmap v387: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 1.2 MiB/s wr, 64 op/s 2024-04-26T11:42:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:05 smithi033 ceph-mon[27845]: pgmap v387: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 1.2 MiB/s wr, 64 op/s 2024-04-26T11:42:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:05 smithi096 ceph-mon[32753]: pgmap v387: 97 pgs: 97 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 1.2 MiB/s wr, 64 op/s 2024-04-26T11:42:07.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 650.0 s, 1159.6 tps, lat 44.951 ms stddev 52.835 2024-04-26T11:42:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:07 smithi194 ceph-mon[33017]: pgmap v388: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-26T11:42:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:07 smithi033 ceph-mon[27845]: pgmap v388: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-26T11:42:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:07 smithi096 ceph-mon[32753]: pgmap v388: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-26T11:42:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:08 smithi194 ceph-mon[33017]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T11:42:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:08 smithi033 ceph-mon[27845]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T11:42:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:08 smithi096 ceph-mon[32753]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T11:42:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:09 smithi194 ceph-mon[33017]: pgmap v390: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-26T11:42:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:09 smithi033 ceph-mon[27845]: pgmap v390: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-26T11:42:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:09 smithi096 ceph-mon[32753]: pgmap v390: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-26T11:42:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:11 smithi194 ceph-mon[33017]: pgmap v391: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 612 op/s 2024-04-26T11:42:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:11 smithi033 ceph-mon[27845]: pgmap v391: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 612 op/s 2024-04-26T11:42:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:11 smithi096 ceph-mon[32753]: pgmap v391: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 612 op/s 2024-04-26T11:42:12.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 655.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:42:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:13 smithi194 ceph-mon[33017]: pgmap v392: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 610 op/s 2024-04-26T11:42:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:13 smithi033 ceph-mon[27845]: pgmap v392: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 610 op/s 2024-04-26T11:42:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:13 smithi096 ceph-mon[32753]: pgmap v392: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 610 op/s 2024-04-26T11:42:15.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:42:15.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 systemd[1]: Stopped Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:42:15.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.176s CPU time. 2024-04-26T11:42:15.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 systemd[1]: Starting Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:42:15.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 podman[70952]: 2024-04-26 11:42:15.240438027 +0000 UTC m=+0.018439441 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:42:15.622 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 podman[70952]: 2024-04-26 11:42:15.358963355 +0000 UTC m=+0.136964756 container create a0b3516fcba7a6ab30d586f7288ed493257d9f5ff5f1f05fa345a69df52b346f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T11:42:15.623 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 podman[70952]: 2024-04-26 11:42:15.595519181 +0000 UTC m=+0.373520589 container init a0b3516fcba7a6ab30d586f7288ed493257d9f5ff5f1f05fa345a69df52b346f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T11:42:15.623 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 podman[70952]: 2024-04-26 11:42:15.599546115 +0000 UTC m=+0.377547525 container start a0b3516fcba7a6ab30d586f7288ed493257d9f5ff5f1f05fa345a69df52b346f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:42:16.104 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mds[70971]: -- 172.21.15.194:0/1241707167 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564596577a40 con 0x56459654b000 2024-04-26T11:42:16.104 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: starting mds.l at 2024-04-26T11:42:16.104 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 bash[70952]: a0b3516fcba7a6ab30d586f7288ed493257d9f5ff5f1f05fa345a69df52b346f 2024-04-26T11:42:16.104 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:15 smithi194 systemd[1]: Started Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:42:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: pgmap v393: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 610 op/s 2024-04-26T11:42:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: daemon mds.l restarted 2024-04-26T11:42:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:16.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T11:42:16.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:boot 2024-04-26T11:42:16.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:42:16.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:16.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:42:16.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:15 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: pgmap v393: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 610 op/s 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: daemon mds.l restarted 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:boot 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:42:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:15 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: pgmap v393: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 610 op/s 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: daemon mds.l restarted 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:boot 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:42:16.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:15 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:17.146+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:17.146+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:17.146+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:42:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 660.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:42:17.404 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:17 smithi194 ceph-mon[33017]: pgmap v395: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 223 op/s 2024-04-26T11:42:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:17 smithi033 ceph-mon[27845]: pgmap v395: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 223 op/s 2024-04-26T11:42:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:17 smithi033 ceph-mon[27845]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T11:42:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:17 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:resolve 2024-04-26T11:42:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:17 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:17 smithi096 ceph-mon[32753]: pgmap v395: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 223 op/s 2024-04-26T11:42:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:17 smithi096 ceph-mon[32753]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T11:42:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:17 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:resolve 2024-04-26T11:42:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:17 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:17 smithi194 ceph-mon[33017]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T11:42:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:17 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:resolve 2024-04-26T11:42:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:17 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:18 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:42:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:18 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:reconnect 2024-04-26T11:42:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:18 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:18 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:42:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:18 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:42:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:18 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:reconnect 2024-04-26T11:42:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:18 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:19.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:18 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:42:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:18 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:42:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:18 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:reconnect 2024-04-26T11:42:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:18 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:18 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:42:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:19 smithi194 ceph-mon[33017]: pgmap v397: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 758 KiB/s wr, 99 op/s 2024-04-26T11:42:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:19 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:rejoin 2024-04-26T11:42:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:19 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:19 smithi194 ceph-mon[33017]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:42:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:19 smithi033 ceph-mon[27845]: pgmap v397: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 758 KiB/s wr, 99 op/s 2024-04-26T11:42:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:19 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:rejoin 2024-04-26T11:42:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:19 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:19 smithi033 ceph-mon[27845]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:42:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:19 smithi096 ceph-mon[32753]: pgmap v397: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 758 KiB/s wr, 99 op/s 2024-04-26T11:42:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:19 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:rejoin 2024-04-26T11:42:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:19 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:19 smithi096 ceph-mon[32753]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:42:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:20 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:20 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:active 2024-04-26T11:42:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:20 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:20 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:20 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:20 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:active 2024-04-26T11:42:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:20 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:20 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:20 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:20 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/3924730707,v1:172.21.15.194:6837/3924730707] up:active 2024-04-26T11:42:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:20 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:20 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:21 smithi194 ceph-mon[33017]: pgmap v398: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 921 KiB/s wr, 122 op/s 2024-04-26T11:42:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:21 smithi033 ceph-mon[27845]: pgmap v398: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 921 KiB/s wr, 122 op/s 2024-04-26T11:42:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:21 smithi096 ceph-mon[32753]: pgmap v398: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 921 KiB/s wr, 122 op/s 2024-04-26T11:42:22.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 665.0 s, 1037.1 tps, lat 336.375 ms stddev 2090.765 2024-04-26T11:42:22.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:42:22 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:42:22.466+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:42:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:23 smithi033 ceph-mon[27845]: pgmap v399: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.0 MiB/s wr, 142 op/s 2024-04-26T11:42:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:23 smithi096 ceph-mon[32753]: pgmap v399: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.0 MiB/s wr, 142 op/s 2024-04-26T11:42:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:23 smithi194 ceph-mon[33017]: pgmap v399: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.0 MiB/s wr, 142 op/s 2024-04-26T11:42:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:25.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0996566700 time 2024-04-26T11:42:24.838752+0000 2024-04-26T11:42:25.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a0b8f3dd] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: clone() 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:24.838+0000 7f0996566700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0996566700 time 2024-04-26T11:42:24.838752+0000 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a0b8f3dd] 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: clone() 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: *** Caught signal (Aborted) ** 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: in thread 7f0996566700 thread_name:ms_dispatch 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099f917d20] 2024-04-26T11:42:25.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: gsignal() 2024-04-26T11:42:25.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: abort() 2024-04-26T11:42:25.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0b8f437] 2024-04-26T11:42:25.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 16: clone() 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:24.839+0000 7f0996566700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: in thread 7f0996566700 thread_name:ms_dispatch 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099f917d20] 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: gsignal() 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: abort() 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0b8f437] 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 16: clone() 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: -1> 2024-04-26T11:42:24.838+0000 7f0996566700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0996566700 time 2024-04-26T11:42:24.838752+0000 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a0b8f3dd] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: clone() 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 0> 2024-04-26T11:42:24.839+0000 7f0996566700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:25.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: in thread 7f0996566700 thread_name:ms_dispatch 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099f917d20] 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: gsignal() 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: abort() 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0b8f437] 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 16: clone() 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: -1> 2024-04-26T11:42:24.838+0000 7f0996566700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0996566700 time 2024-04-26T11:42:24.838752+0000 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a0b8f3dd] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: clone() 2024-04-26T11:42:25.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 0> 2024-04-26T11:42:24.839+0000 7f0996566700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: in thread 7f0996566700 thread_name:ms_dispatch 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099f917d20] 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2: gsignal() 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 3: abort() 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0b8f437] 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a0b8f5a3] 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562daa711822] 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562daa681598] 2024-04-26T11:42:25.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562daa683c17] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562daa57cd0b] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562daa57d2ac] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562daa565fad] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f09a0e23e20] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 13: (DispatchQueue::entry()+0x51f) [0x7f09a0e20fdf] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09a0ef3fc1] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f099f90d1ca] 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 16: clone() 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[70676]: 2024-04-26T11:42:25.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:24 smithi194 podman[71010]: 2024-04-26 11:42:24.900961044 +0000 UTC m=+0.019137474 container died 00de932448a3db59ea9f658f4af88c1c476325b79672c1f1272557335b034ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T11:42:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:24 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:24 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:25.853 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:25 smithi194 podman[71010]: 2024-04-26 11:42:25.559128287 +0000 UTC m=+0.677304704 container remove 00de932448a3db59ea9f658f4af88c1c476325b79672c1f1272557335b034ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T11:42:25.853 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:25 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:25 smithi033 ceph-mon[27845]: pgmap v400: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 490 KiB/s wr, 66 op/s 2024-04-26T11:42:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:25 smithi096 ceph-mon[32753]: pgmap v400: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 490 KiB/s wr, 66 op/s 2024-04-26T11:42:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:25 smithi194 ceph-mon[33017]: pgmap v400: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 490 KiB/s wr, 66 op/s 2024-04-26T11:42:26.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:25 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T11:42:26.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:25 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.228s CPU time. 2024-04-26T11:42:27.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 670.0 s, 2018.6 tps, lat 43.307 ms stddev 49.276 2024-04-26T11:42:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:27 smithi033 ceph-mon[27845]: pgmap v401: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 7.3 MiB/s wr, 372 op/s 2024-04-26T11:42:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:27 smithi096 ceph-mon[32753]: pgmap v401: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 7.3 MiB/s wr, 372 op/s 2024-04-26T11:42:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:27 smithi194 ceph-mon[33017]: pgmap v401: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 7.3 MiB/s wr, 372 op/s 2024-04-26T11:42:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:29 smithi033 ceph-mon[27845]: pgmap v402: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 6.3 MiB/s wr, 320 op/s 2024-04-26T11:42:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:29 smithi096 ceph-mon[32753]: pgmap v402: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 6.3 MiB/s wr, 320 op/s 2024-04-26T11:42:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:29 smithi194 ceph-mon[33017]: pgmap v402: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 6.3 MiB/s wr, 320 op/s 2024-04-26T11:42:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:32.162+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:32.162+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:42:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:32.162+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:31 smithi033 ceph-mon[27845]: pgmap v403: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 8.8 MiB/s wr, 461 op/s 2024-04-26T11:42:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:31 smithi096 ceph-mon[32753]: pgmap v403: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 8.8 MiB/s wr, 461 op/s 2024-04-26T11:42:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 675.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:42:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:31 smithi194 ceph-mon[33017]: pgmap v403: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 8.8 MiB/s wr, 461 op/s 2024-04-26T11:42:34.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:33 smithi194 ceph-mon[33017]: pgmap v404: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 9.0 MiB/s wr, 478 op/s 2024-04-26T11:42:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:33 smithi033 ceph-mon[27845]: pgmap v404: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 9.0 MiB/s wr, 478 op/s 2024-04-26T11:42:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:33 smithi096 ceph-mon[32753]: pgmap v404: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 9.0 MiB/s wr, 478 op/s 2024-04-26T11:42:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:35 smithi033 ceph-mon[27845]: pgmap v405: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 8.9 MiB/s wr, 465 op/s 2024-04-26T11:42:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:35 smithi096 ceph-mon[32753]: pgmap v405: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 8.9 MiB/s wr, 465 op/s 2024-04-26T11:42:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:35 smithi194 ceph-mon[33017]: pgmap v405: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 8.9 MiB/s wr, 465 op/s 2024-04-26T11:42:36.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:42:36.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 systemd[1]: Stopped Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:42:36.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.228s CPU time. 2024-04-26T11:42:36.355 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 systemd[1]: Starting Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:42:36.671 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 podman[71115]: 2024-04-26 11:42:36.258350339 +0000 UTC m=+0.019953788 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:42:36.671 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 podman[71115]: 2024-04-26 11:42:36.376727299 +0000 UTC m=+0.138330748 container create 6f176fdbf760fe41393fa82f668b50dd8e96ec3254759ddb07cc76500e3a50de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:42:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: daemon mds.c restarted 2024-04-26T11:42:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:37.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:42:37.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T11:42:37.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:boot 2024-04-26T11:42:37.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:42:37.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:37.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:37.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 podman[71115]: 2024-04-26 11:42:36.671132338 +0000 UTC m=+0.432735783 container init 6f176fdbf760fe41393fa82f668b50dd8e96ec3254759ddb07cc76500e3a50de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T11:42:37.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 podman[71115]: 2024-04-26 11:42:36.673878133 +0000 UTC m=+0.435481580 container start 6f176fdbf760fe41393fa82f668b50dd8e96ec3254759ddb07cc76500e3a50de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T11:42:37.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-mds[71135]: -- 172.21.15.194:0/2329102688 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e4e5063a40 con 0x55e4e5036c00 2024-04-26T11:42:37.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: starting mds.c at 2024-04-26T11:42:37.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 bash[71115]: 6f176fdbf760fe41393fa82f668b50dd8e96ec3254759ddb07cc76500e3a50de 2024-04-26T11:42:37.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:42:36 smithi194 systemd[1]: Started Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:42:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: daemon mds.c restarted 2024-04-26T11:42:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:42:37.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T11:42:37.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:boot 2024-04-26T11:42:37.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:42:37.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:37.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:36 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: daemon mds.c restarted 2024-04-26T11:42:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:42:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T11:42:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:boot 2024-04-26T11:42:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:42:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:36 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 680.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:42:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:37 smithi033 ceph-mon[27845]: pgmap v406: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 10 MiB/s wr, 597 op/s 2024-04-26T11:42:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:37 smithi096 ceph-mon[32753]: pgmap v406: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 10 MiB/s wr, 597 op/s 2024-04-26T11:42:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:37 smithi194 ceph-mon[33017]: pgmap v406: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 10 MiB/s wr, 597 op/s 2024-04-26T11:42:39.201 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:38 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:resolve 2024-04-26T11:42:39.201 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:38 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:39.201 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:38 smithi033 ceph-mon[27845]: pgmap v408: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 400 op/s 2024-04-26T11:42:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:38 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:resolve 2024-04-26T11:42:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:38 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:38 smithi194 ceph-mon[33017]: pgmap v408: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 400 op/s 2024-04-26T11:42:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:38 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:resolve 2024-04-26T11:42:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:38 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:38 smithi096 ceph-mon[32753]: pgmap v408: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 400 op/s 2024-04-26T11:42:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:39 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:42:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:39 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:reconnect 2024-04-26T11:42:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:39 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:39 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119998 2024-04-26T11:42:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:39 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:42:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:39 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:reconnect 2024-04-26T11:42:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:39 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:39 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119998 2024-04-26T11:42:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:39 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:42:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:39 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:reconnect 2024-04-26T11:42:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:39 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:39 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0119998 2024-04-26T11:42:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:40 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:rejoin 2024-04-26T11:42:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:40 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:40 smithi194 ceph-mon[33017]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:42:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:40 smithi194 ceph-mon[33017]: pgmap v409: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 MiB/s wr, 265 op/s 2024-04-26T11:42:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:40 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:rejoin 2024-04-26T11:42:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:40 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:40 smithi033 ceph-mon[27845]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:42:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:40 smithi033 ceph-mon[27845]: pgmap v409: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 MiB/s wr, 265 op/s 2024-04-26T11:42:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:40 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:rejoin 2024-04-26T11:42:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:40 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:40 smithi096 ceph-mon[32753]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:42:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:40 smithi096 ceph-mon[32753]: pgmap v409: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 MiB/s wr, 265 op/s 2024-04-26T11:42:42.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 685.0 s, 582.6 tps, lat 536.390 ms stddev 2638.572 2024-04-26T11:42:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:41 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:41 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:active 2024-04-26T11:42:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:41 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:41 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:41 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:41 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:active 2024-04-26T11:42:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:41 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:41 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:41 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:42:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:41 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/986307035,v1:172.21.15.194:6833/986307035] up:active 2024-04-26T11:42:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:41 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:41 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:42.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:42:42 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:42:42.467+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:42:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:43 smithi194 ceph-mon[33017]: pgmap v410: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T11:42:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:43 smithi033 ceph-mon[27845]: pgmap v410: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T11:42:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:43 smithi096 ceph-mon[32753]: pgmap v410: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T11:42:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:45 smithi194 ceph-mon[33017]: pgmap v411: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T11:42:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:45 smithi033 ceph-mon[27845]: pgmap v411: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T11:42:45.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:45 smithi096 ceph-mon[32753]: pgmap v411: 97 pgs: 97 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T11:42:46.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f251c963700 time 2024-04-26T11:42:46.011561+0000 2024-04-26T11:42:46.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:46.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2526f8c3dd] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: clone() 2024-04-26T11:42:46.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.011+0000 7f251c963700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f251c963700 time 2024-04-26T11:42:46.011561+0000 2024-04-26T11:42:46.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:46.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2526f8c3dd] 2024-04-26T11:42:46.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: clone() 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: *** Caught signal (Aborted) ** 2024-04-26T11:42:46.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: in thread 7f251c963700 thread_name:ms_dispatch 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2525d14d20] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: gsignal() 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: abort() 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2526f8c437] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 16: clone() 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.013+0000 7f251c963700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: in thread 7f251c963700 thread_name:ms_dispatch 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2525d14d20] 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: gsignal() 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: abort() 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2526f8c437] 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 16: clone() 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: -1> 2024-04-26T11:42:46.011+0000 7f251c963700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f251c963700 time 2024-04-26T11:42:46.011561+0000 2024-04-26T11:42:46.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2526f8c3dd] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: clone() 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 0> 2024-04-26T11:42:46.013+0000 7f251c963700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: in thread 7f251c963700 thread_name:ms_dispatch 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2525d14d20] 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: gsignal() 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: abort() 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2526f8c437] 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 16: clone() 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: -1> 2024-04-26T11:42:46.011+0000 7f251c963700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f251c963700 time 2024-04-26T11:42:46.011561+0000 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2526f8c3dd] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: clone() 2024-04-26T11:42:46.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 0> 2024-04-26T11:42:46.013+0000 7f251c963700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: in thread 7f251c963700 thread_name:ms_dispatch 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2525d14d20] 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2: gsignal() 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 3: abort() 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2526f8c437] 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2526f8c5a3] 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564594d83822] 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564594cf3598] 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564594cf5c17] 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564594beed0b] 2024-04-26T11:42:46.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564594bef2ac] 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564594bd7fad] 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2527220e20] 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 13: (DispatchQueue::entry()+0x51f) [0x7f252721dfdf] 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f25272f0fc1] 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2525d0a1ca] 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 16: clone() 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[70967]: 2024-04-26T11:42:46.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 podman[71174]: 2024-04-26 11:42:46.069019767 +0000 UTC m=+0.019280777 container died a0b3516fcba7a6ab30d586f7288ed493257d9f5ff5f1f05fa345a69df52b346f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=) 2024-04-26T11:42:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:47.177+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:47.177+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:42:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:42:47.177+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:42:47.182 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 690.0 s, 2001.4 tps, lat 49.623 ms stddev 78.441 2024-04-26T11:42:47.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:47 smithi194 ceph-mon[33017]: pgmap v412: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 13 MiB/s wr, 1.16k op/s 2024-04-26T11:42:47.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:47 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:47.191 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 podman[71174]: 2024-04-26 11:42:46.902785498 +0000 UTC m=+0.853046498 container remove a0b3516fcba7a6ab30d586f7288ed493257d9f5ff5f1f05fa345a69df52b346f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:42:47.191 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:46 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:42:47.357 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:47 smithi033 ceph-mon[27845]: pgmap v412: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 13 MiB/s wr, 1.16k op/s 2024-04-26T11:42:47.357 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:47 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:47 smithi096 ceph-mon[32753]: pgmap v412: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 13 MiB/s wr, 1.16k op/s 2024-04-26T11:42:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:47 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:42:47.604 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:47 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-26T11:42:47.604 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:47 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.144s CPU time. 2024-04-26T11:42:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:49 smithi033 ceph-mon[27845]: pgmap v413: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T11:42:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:49 smithi096 ceph-mon[32753]: pgmap v413: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T11:42:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:49 smithi194 ceph-mon[33017]: pgmap v413: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T11:42:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:51 smithi033 ceph-mon[27845]: pgmap v414: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T11:42:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:51 smithi096 ceph-mon[32753]: pgmap v414: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T11:42:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:51 smithi194 ceph-mon[33017]: pgmap v414: 97 pgs: 97 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T11:42:52.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 695.0 s, 369.7 tps, lat 56.128 ms stddev 56.509 2024-04-26T11:42:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:53 smithi033 ceph-mon[27845]: pgmap v415: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-04-26T11:42:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:53 smithi096 ceph-mon[32753]: pgmap v415: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-04-26T11:42:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:53 smithi194 ceph-mon[33017]: pgmap v415: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-04-26T11:42:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:55 smithi033 ceph-mon[27845]: pgmap v416: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 15 MiB/s wr, 1.17k op/s 2024-04-26T11:42:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:55 smithi096 ceph-mon[32753]: pgmap v416: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 15 MiB/s wr, 1.17k op/s 2024-04-26T11:42:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:55 smithi194 ceph-mon[33017]: pgmap v416: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 15 MiB/s wr, 1.17k op/s 2024-04-26T11:42:57.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 700.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:42:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:57 smithi033 ceph-mon[27845]: pgmap v417: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 16 MiB/s wr, 1.22k op/s 2024-04-26T11:42:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:42:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:57 smithi096 ceph-mon[32753]: pgmap v417: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 16 MiB/s wr, 1.22k op/s 2024-04-26T11:42:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:42:57.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:57 smithi194 ceph-mon[33017]: pgmap v417: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 16 MiB/s wr, 1.22k op/s 2024-04-26T11:42:57.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:42:57.580 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:42:57.580 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 systemd[1]: Stopped Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:42:57.580 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.144s CPU time. 2024-04-26T11:42:57.581 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 systemd[1]: Starting Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:42:57.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 podman[71372]: 2024-04-26 11:42:57.482434162 +0000 UTC m=+0.017884121 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:42:57.854 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 podman[71372]: 2024-04-26 11:42:57.653868189 +0000 UTC m=+0.189318177 container create f7363689181993734dc6587b1a410da1e3370678ea9c1e0bbda14d5fac37c4f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T11:42:58.122 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 podman[71372]: 2024-04-26 11:42:57.906756066 +0000 UTC m=+0.442206011 container init f7363689181993734dc6587b1a410da1e3370678ea9c1e0bbda14d5fac37c4f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T11:42:58.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 podman[71372]: 2024-04-26 11:42:57.909369241 +0000 UTC m=+0.444819189 container start f7363689181993734dc6587b1a410da1e3370678ea9c1e0bbda14d5fac37c4f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True) 2024-04-26T11:42:58.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 ceph-mds[71392]: -- 172.21.15.194:0/3085174192 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56477ff2ba40 con 0x56477fefe800 2024-04-26T11:42:58.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:57 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: starting mds.l at 2024-04-26T11:42:58.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:58 smithi194 bash[71372]: f7363689181993734dc6587b1a410da1e3370678ea9c1e0bbda14d5fac37c4f2 2024-04-26T11:42:58.123 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:42:58 smithi194 systemd[1]: Started Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: pgmap v418: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 315 op/s 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: pgmap v419: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.5 MiB/s wr, 381 op/s 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: daemon mds.l restarted 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:42:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T11:42:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:boot 2024-04-26T11:42:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:42:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:42:58 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: pgmap v418: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 315 op/s 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: pgmap v419: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.5 MiB/s wr, 381 op/s 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: daemon mds.l restarted 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:boot 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:42:58 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:42:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:42:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:42:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: pgmap v418: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 315 op/s 2024-04-26T11:42:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: pgmap v419: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.5 MiB/s wr, 381 op/s 2024-04-26T11:42:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:42:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: daemon mds.l restarted 2024-04-26T11:42:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:42:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:42:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T11:42:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:boot 2024-04-26T11:42:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:42:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:42:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:42:58 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:00.450 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:00 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:resolve 2024-04-26T11:43:00.450 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:00 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:00.451 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:00 smithi033 ceph-mon[27845]: pgmap v421: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 137 op/s 2024-04-26T11:43:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:00 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:resolve 2024-04-26T11:43:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:00 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:00 smithi194 ceph-mon[33017]: pgmap v421: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 137 op/s 2024-04-26T11:43:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:00 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:resolve 2024-04-26T11:43:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:00 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:00 smithi096 ceph-mon[32753]: pgmap v421: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 137 op/s 2024-04-26T11:43:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:01 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:reconnect 2024-04-26T11:43:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:01 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:01 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0149997 2024-04-26T11:43:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:01 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.106998 2024-04-26T11:43:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:01 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:reconnect 2024-04-26T11:43:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:01 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:01 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0149997 2024-04-26T11:43:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:01 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.106998 2024-04-26T11:43:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:01 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:reconnect 2024-04-26T11:43:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:01 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:01 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0149997 2024-04-26T11:43:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:01 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.106998 2024-04-26T11:43:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 705.0 s, 304.5 tps, lat 912.435 ms stddev 3299.241 2024-04-26T11:43:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:02.192+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:02.192+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:43:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:02.192+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:02 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:rejoin 2024-04-26T11:43:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:02 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:02 smithi194 ceph-mon[33017]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:43:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:02 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:02 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:active 2024-04-26T11:43:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:02 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:02 smithi194 ceph-mon[33017]: pgmap v422: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-26T11:43:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:02 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:rejoin 2024-04-26T11:43:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:02 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:02 smithi033 ceph-mon[27845]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:43:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:02 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:02 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:active 2024-04-26T11:43:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:02 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:02 smithi033 ceph-mon[27845]: pgmap v422: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:43:02.469+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:rejoin 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-mon[32753]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1031537986,v1:172.21.15.194:6837/1031537986] up:active 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:02 smithi096 ceph-mon[32753]: pgmap v422: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-26T11:43:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:04 smithi194 ceph-mon[33017]: pgmap v423: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 556 KiB/s wr, 73 op/s 2024-04-26T11:43:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:04 smithi033 ceph-mon[27845]: pgmap v423: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 556 KiB/s wr, 73 op/s 2024-04-26T11:43:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:04 smithi096 ceph-mon[32753]: pgmap v423: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 556 KiB/s wr, 73 op/s 2024-04-26T11:43:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:06 smithi194 ceph-mon[33017]: pgmap v424: 97 pgs: 97 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-04-26T11:43:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:06 smithi033 ceph-mon[27845]: pgmap v424: 97 pgs: 97 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-04-26T11:43:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:06 smithi096 ceph-mon[32753]: pgmap v424: 97 pgs: 97 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-04-26T11:43:07.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 710.0 s, 1614.6 tps, lat 63.118 ms stddev 83.811 2024-04-26T11:43:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-mon[33017]: pgmap v425: 97 pgs: 97 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 9.0 MiB/s wr, 928 op/s 2024-04-26T11:43:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-mon[33017]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T11:43:09.075 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:08 smithi033 ceph-mon[27845]: pgmap v425: 97 pgs: 97 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 9.0 MiB/s wr, 928 op/s 2024-04-26T11:43:09.075 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:08 smithi033 ceph-mon[27845]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T11:43:09.160 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6dd0e75700 time 2024-04-26T11:43:08.875537+0000 2024-04-26T11:43:09.160 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ddb49e3dd] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.161 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: clone() 2024-04-26T11:43:09.162 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:08.875+0000 7f6dd0e75700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6dd0e75700 time 2024-04-26T11:43:08.875537+0000 2024-04-26T11:43:09.162 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:09.162 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.162 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.162 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ddb49e3dd] 2024-04-26T11:43:09.162 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.162 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.163 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: clone() 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: *** Caught signal (Aborted) ** 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: in thread 7f6dd0e75700 thread_name:ms_dispatch 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6dda226d20] 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: gsignal() 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: abort() 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ddb49e437] 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.164 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 16: clone() 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:08.877+0000 7f6dd0e75700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: in thread 7f6dd0e75700 thread_name:ms_dispatch 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6dda226d20] 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: gsignal() 2024-04-26T11:43:09.165 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: abort() 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ddb49e437] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 16: clone() 2024-04-26T11:43:09.166 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: -1> 2024-04-26T11:43:08.875+0000 7f6dd0e75700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6dd0e75700 time 2024-04-26T11:43:08.875537+0000 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ddb49e3dd] 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.168 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: clone() 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 0> 2024-04-26T11:43:08.877+0000 7f6dd0e75700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: in thread 7f6dd0e75700 thread_name:ms_dispatch 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6dda226d20] 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: gsignal() 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: abort() 2024-04-26T11:43:09.169 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ddb49e437] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 16: clone() 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:09.170 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: -1> 2024-04-26T11:43:08.875+0000 7f6dd0e75700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6dd0e75700 time 2024-04-26T11:43:08.875537+0000 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6ddb49e3dd] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.171 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: clone() 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 0> 2024-04-26T11:43:08.877+0000 7f6dd0e75700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: in thread 7f6dd0e75700 thread_name:ms_dispatch 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6dda226d20] 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2: gsignal() 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 3: abort() 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6ddb49e437] 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6ddb49e5a3] 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55e4e267c822] 2024-04-26T11:43:09.172 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55e4e25ec598] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e4e25eec17] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e4e24e7d0b] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e4e24e82ac] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55e4e24d0fad] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6ddb732e20] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 13: (DispatchQueue::entry()+0x51f) [0x7f6ddb72ffdf] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6ddb802fc1] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6dda21c1ca] 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 16: clone() 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71131]: 2024-04-26T11:43:09.173 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:08 smithi194 podman[71481]: 2024-04-26 11:43:08.956338374 +0000 UTC m=+0.038702105 container died 6f176fdbf760fe41393fa82f668b50dd8e96ec3254759ddb07cc76500e3a50de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:43:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:08 smithi096 ceph-mon[32753]: pgmap v425: 97 pgs: 97 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 9.0 MiB/s wr, 928 op/s 2024-04-26T11:43:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:08 smithi096 ceph-mon[32753]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T11:43:09.875 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:09 smithi194 podman[71481]: 2024-04-26 11:43:09.617715829 +0000 UTC m=+0.700079547 container remove 6f176fdbf760fe41393fa82f668b50dd8e96ec3254759ddb07cc76500e3a50de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T11:43:09.875 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:09 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:43:10.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:09 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T11:43:10.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:09 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.735s CPU time. 2024-04-26T11:43:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:10 smithi194 ceph-mon[33017]: pgmap v427: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 13 MiB/s wr, 1.29k op/s 2024-04-26T11:43:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:10 smithi033 ceph-mon[27845]: pgmap v427: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 13 MiB/s wr, 1.29k op/s 2024-04-26T11:43:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:10 smithi096 ceph-mon[32753]: pgmap v427: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 13 MiB/s wr, 1.29k op/s 2024-04-26T11:43:12.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 715.0 s, 505.3 tps, lat 74.739 ms stddev 104.536 2024-04-26T11:43:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:12 smithi194 ceph-mon[33017]: pgmap v428: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:43:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:12 smithi033 ceph-mon[27845]: pgmap v428: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:43:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:12 smithi096 ceph-mon[32753]: pgmap v428: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:43:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:14 smithi194 ceph-mon[33017]: pgmap v429: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:43:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:14 smithi033 ceph-mon[27845]: pgmap v429: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:43:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:14 smithi096 ceph-mon[32753]: pgmap v429: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-04-26T11:43:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:16 smithi194 ceph-mon[33017]: pgmap v430: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.5 MiB/s wr, 587 op/s 2024-04-26T11:43:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:16 smithi194 ceph-mon[33017]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T11:43:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:16 smithi033 ceph-mon[27845]: pgmap v430: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.5 MiB/s wr, 587 op/s 2024-04-26T11:43:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:16 smithi033 ceph-mon[27845]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T11:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:16 smithi096 ceph-mon[32753]: pgmap v430: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.5 MiB/s wr, 587 op/s 2024-04-26T11:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:16 smithi096 ceph-mon[32753]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T11:43:17.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 720.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:43:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:17.207+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:17.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:17.207+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:17.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:17.207+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:43:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:18 smithi194 ceph-mon[33017]: pgmap v432: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 6.6 MiB/s wr, 592 op/s 2024-04-26T11:43:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:18 smithi033 ceph-mon[27845]: pgmap v432: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 6.6 MiB/s wr, 592 op/s 2024-04-26T11:43:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:18 smithi096 ceph-mon[32753]: pgmap v432: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 6.6 MiB/s wr, 592 op/s 2024-04-26T11:43:20.357 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:43:20.357 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 systemd[1]: Stopped Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:43:20.357 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.735s CPU time. 2024-04-26T11:43:20.357 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 systemd[1]: Starting Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:43:20.628 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 podman[71573]: 2024-04-26 11:43:20.259058071 +0000 UTC m=+0.018251672 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:43:20.628 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 podman[71573]: 2024-04-26 11:43:20.381171343 +0000 UTC m=+0.140364913 container create 721361dc291170d3b036ea9530d14602c091b5211295e636c87035e316b2ea00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:43:20.628 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 podman[71573]: 2024-04-26 11:43:20.628110339 +0000 UTC m=+0.387303909 container init 721361dc291170d3b036ea9530d14602c091b5211295e636c87035e316b2ea00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T11:43:21.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 podman[71573]: 2024-04-26 11:43:20.630992455 +0000 UTC m=+0.390186039 container start 721361dc291170d3b036ea9530d14602c091b5211295e636c87035e316b2ea00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:43:21.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mds[71592]: -- 172.21.15.194:0/2417814257 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564619abfa40 con 0x564619a93000 2024-04-26T11:43:21.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: starting mds.c at 2024-04-26T11:43:21.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 bash[71573]: 721361dc291170d3b036ea9530d14602c091b5211295e636c87035e316b2ea00 2024-04-26T11:43:21.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:20 smithi194 systemd[1]: Started Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:43:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: pgmap v433: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.1 MiB/s wr, 456 op/s 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: daemon mds.c restarted 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:boot 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:20 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: pgmap v433: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.1 MiB/s wr, 456 op/s 2024-04-26T11:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: daemon mds.c restarted 2024-04-26T11:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T11:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:boot 2024-04-26T11:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:20 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: pgmap v433: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.1 MiB/s wr, 456 op/s 2024-04-26T11:43:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: daemon mds.c restarted 2024-04-26T11:43:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:43:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T11:43:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:boot 2024-04-26T11:43:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:43:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:20 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:22.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 725.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:43:22.946 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:22 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:resolve 2024-04-26T11:43:22.946 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:22 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:22.946 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:22 smithi033 ceph-mon[27845]: pgmap v435: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.5 MiB/s wr, 558 op/s 2024-04-26T11:43:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:22 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:resolve 2024-04-26T11:43:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:22 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:22 smithi194 ceph-mon[33017]: pgmap v435: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.5 MiB/s wr, 558 op/s 2024-04-26T11:43:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:22 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:resolve 2024-04-26T11:43:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:22 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:22 smithi096 ceph-mon[32753]: pgmap v435: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.5 MiB/s wr, 558 op/s 2024-04-26T11:43:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:23 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00099998 2024-04-26T11:43:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:23 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:reconnect 2024-04-26T11:43:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:23 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:23 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:23 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00099998 2024-04-26T11:43:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:23 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:reconnect 2024-04-26T11:43:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:23 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:23 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:23 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00099998 2024-04-26T11:43:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:23 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:reconnect 2024-04-26T11:43:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:23 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:23 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:rejoin 2024-04-26T11:43:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:24 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:24 smithi194 ceph-mon[33017]: pgmap v436: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-04-26T11:43:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:24 smithi194 ceph-mon[33017]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:43:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:rejoin 2024-04-26T11:43:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:24 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:24 smithi033 ceph-mon[27845]: pgmap v436: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-04-26T11:43:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:24 smithi033 ceph-mon[27845]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:43:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:rejoin 2024-04-26T11:43:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:24 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:24 smithi096 ceph-mon[32753]: pgmap v436: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-04-26T11:43:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:24 smithi096 ceph-mon[32753]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:43:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:25 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:active 2024-04-26T11:43:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:25 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:25 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:25 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:active 2024-04-26T11:43:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:25 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:25 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:25 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/610478393,v1:172.21.15.194:6833/610478393] up:active 2024-04-26T11:43:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:25 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:25 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:26 smithi194 ceph-mon[33017]: pgmap v437: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.6 MiB/s wr, 449 op/s 2024-04-26T11:43:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:26 smithi033 ceph-mon[27845]: pgmap v437: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.6 MiB/s wr, 449 op/s 2024-04-26T11:43:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:26 smithi096 ceph-mon[32753]: pgmap v437: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.6 MiB/s wr, 449 op/s 2024-04-26T11:43:27.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 730.0 s, 3.8 tps, lat 15708.588 ms stddev 28.336 2024-04-26T11:43:27.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:43:27 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:43:27.470+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:43:28.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f73cb6b0700 time 2024-04-26T11:43:27.950291+0000 2024-04-26T11:43:28.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:28.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73d5cd93dd] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: clone() 2024-04-26T11:43:28.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:27.950+0000 7f73cb6b0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f73cb6b0700 time 2024-04-26T11:43:27.950291+0000 2024-04-26T11:43:28.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:28.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73d5cd93dd] 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: clone() 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: *** Caught signal (Aborted) ** 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: in thread 7f73cb6b0700 thread_name:ms_dispatch 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73d4a61d20] 2024-04-26T11:43:28.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: gsignal() 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: abort() 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f73d5cd9437] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 16: clone() 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:27.952+0000 7f73cb6b0700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: in thread 7f73cb6b0700 thread_name:ms_dispatch 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73d4a61d20] 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: gsignal() 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: abort() 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f73d5cd9437] 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 16: clone() 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: -3> 2024-04-26T11:43:27.950+0000 7f73cb6b0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f73cb6b0700 time 2024-04-26T11:43:27.950291+0000 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73d5cd93dd] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: clone() 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 0> 2024-04-26T11:43:27.952+0000 7f73cb6b0700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: in thread 7f73cb6b0700 thread_name:ms_dispatch 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73d4a61d20] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: gsignal() 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: abort() 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f73d5cd9437] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 16: clone() 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: -4> 2024-04-26T11:43:27.950+0000 7f73cb6b0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f73cb6b0700 time 2024-04-26T11:43:27.950291+0000 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73d5cd93dd] 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: clone() 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: -1> 2024-04-26T11:43:27.952+0000 7f73cb6b0700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: in thread 7f73cb6b0700 thread_name:ms_dispatch 2024-04-26T11:43:28.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f73d4a61d20] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2: gsignal() 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 3: abort() 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f73d5cd9437] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73d5cd95a3] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56477e999822] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x56477e909598] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56477e90bc17] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56477e804d0b] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56477e8052ac] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56477e7edfad] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f73d5f6de20] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 13: (DispatchQueue::entry()+0x51f) [0x7f73d5f6afdf] 2024-04-26T11:43:28.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f73d603dfc1] 2024-04-26T11:43:28.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f73d4a571ca] 2024-04-26T11:43:28.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 16: clone() 2024-04-26T11:43:28.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:28.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:27 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71387]: 2024-04-26T11:43:28.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:28 smithi194 podman[71629]: 2024-04-26 11:43:28.016420572 +0000 UTC m=+0.020836807 container died f7363689181993734dc6587b1a410da1e3370678ea9c1e0bbda14d5fac37c4f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T11:43:28.920 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:28 smithi194 ceph-mon[33017]: pgmap v438: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-04-26T11:43:28.920 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:28 smithi194 podman[71629]: 2024-04-26 11:43:28.6586308 +0000 UTC m=+0.663047024 container remove f7363689181993734dc6587b1a410da1e3370678ea9c1e0bbda14d5fac37c4f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T11:43:28.920 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:28 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:43:29.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:28 smithi033 ceph-mon[27845]: pgmap v438: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-04-26T11:43:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:28 smithi096 ceph-mon[32753]: pgmap v438: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-04-26T11:43:29.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:28 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-26T11:43:29.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:28 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.158s CPU time. 2024-04-26T11:43:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:30 smithi194 ceph-mon[33017]: pgmap v439: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.6 MiB/s wr, 217 op/s 2024-04-26T11:43:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:30 smithi033 ceph-mon[27845]: pgmap v439: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.6 MiB/s wr, 217 op/s 2024-04-26T11:43:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:30 smithi096 ceph-mon[32753]: pgmap v439: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.6 MiB/s wr, 217 op/s 2024-04-26T11:43:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 735.0 s, 395.9 tps, lat 847.878 ms stddev 3761.110 2024-04-26T11:43:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:32.209+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:32.209+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:43:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:32.209+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:32 smithi194 ceph-mon[33017]: pgmap v440: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-26T11:43:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:32 smithi033 ceph-mon[27845]: pgmap v440: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-26T11:43:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:32 smithi096 ceph-mon[32753]: pgmap v440: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-26T11:43:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:34 smithi194 ceph-mon[33017]: pgmap v441: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-26T11:43:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:34 smithi033 ceph-mon[27845]: pgmap v441: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-26T11:43:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:34 smithi096 ceph-mon[32753]: pgmap v441: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-26T11:43:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:36 smithi194 ceph-mon[33017]: pgmap v442: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-04-26T11:43:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:36 smithi033 ceph-mon[27845]: pgmap v442: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-04-26T11:43:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:36 smithi096 ceph-mon[32753]: pgmap v442: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-04-26T11:43:37.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 740.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:43:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:38 smithi194 ceph-mon[33017]: pgmap v443: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-04-26T11:43:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:38 smithi033 ceph-mon[27845]: pgmap v443: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-04-26T11:43:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:38 smithi096 ceph-mon[32753]: pgmap v443: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-04-26T11:43:39.376 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:43:39.377 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 systemd[1]: Stopped Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:43:39.377 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.158s CPU time. 2024-04-26T11:43:39.377 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 systemd[1]: Starting Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:43:39.377 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 podman[71729]: 2024-04-26 11:43:39.246491002 +0000 UTC m=+0.018874432 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:43:39.377 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 podman[71729]: 2024-04-26 11:43:39.376052223 +0000 UTC m=+0.148435685 container create 288f543a76c602838df0560ee823ac151efbf704c87183d5b3f50b0e4ee60b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T11:43:40.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: daemon mds.l restarted 2024-04-26T11:43:40.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:40.108 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:43:40.108 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T11:43:40.108 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:boot 2024-04-26T11:43:40.108 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:43:40.108 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:40.108 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:40.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 podman[71729]: 2024-04-26 11:43:39.636870567 +0000 UTC m=+0.409253995 container init 288f543a76c602838df0560ee823ac151efbf704c87183d5b3f50b0e4ee60b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T11:43:40.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 podman[71729]: 2024-04-26 11:43:39.639402675 +0000 UTC m=+0.411786108 container start 288f543a76c602838df0560ee823ac151efbf704c87183d5b3f50b0e4ee60b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T11:43:40.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-mds[71748]: -- 172.21.15.194:0/4028253228 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560069d7ba40 con 0x560069d4ec00 2024-04-26T11:43:40.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: starting mds.l at 2024-04-26T11:43:40.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 bash[71729]: 288f543a76c602838df0560ee823ac151efbf704c87183d5b3f50b0e4ee60b2f 2024-04-26T11:43:40.109 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:43:39 smithi194 systemd[1]: Started Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:43:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: daemon mds.l restarted 2024-04-26T11:43:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:43:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T11:43:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:boot 2024-04-26T11:43:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:43:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:39 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: daemon mds.l restarted 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:boot 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:39 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:40 smithi194 ceph-mon[33017]: pgmap v445: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 796 KiB/s wr, 42 op/s 2024-04-26T11:43:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:40 smithi033 ceph-mon[27845]: pgmap v445: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 796 KiB/s wr, 42 op/s 2024-04-26T11:43:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:40 smithi096 ceph-mon[32753]: pgmap v445: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 796 KiB/s wr, 42 op/s 2024-04-26T11:43:42.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:41 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:resolve 2024-04-26T11:43:42.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:41 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:41 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:resolve 2024-04-26T11:43:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:41 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:41 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:resolve 2024-04-26T11:43:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:41 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:42.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 745.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:43:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:42 smithi194 ceph-mon[33017]: pgmap v446: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T11:43:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:42 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:43:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:42 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:reconnect 2024-04-26T11:43:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:42 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:42 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:42 smithi033 ceph-mon[27845]: pgmap v446: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T11:43:43.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:42 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:43:43.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:42 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:reconnect 2024-04-26T11:43:43.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:42 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:43.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:42 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:42 smithi096 ceph-mon[32753]: pgmap v446: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T11:43:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:42 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299994 2024-04-26T11:43:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:42 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:reconnect 2024-04-26T11:43:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:42 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:42 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:44.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:43 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:rejoin 2024-04-26T11:43:44.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:43 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:44.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:43 smithi096 ceph-mon[32753]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:43:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:rejoin 2024-04-26T11:43:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-mon[33017]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:43:44.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcee7291700 time 2024-04-26T11:43:43.846279+0000 2024-04-26T11:43:44.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef18ba3dd] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: clone() 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:43.846+0000 7fcee7291700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcee7291700 time 2024-04-26T11:43:43.846279+0000 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef18ba3dd] 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: clone() 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: *** Caught signal (Aborted) ** 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: in thread 7fcee7291700 thread_name:ms_dispatch 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef0642d20] 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: gsignal() 2024-04-26T11:43:44.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: abort() 2024-04-26T11:43:44.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef18ba437] 2024-04-26T11:43:44.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 16: clone() 2024-04-26T11:43:44.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:43.848+0000 7fcee7291700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: in thread 7fcee7291700 thread_name:ms_dispatch 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef0642d20] 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: gsignal() 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: abort() 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef18ba437] 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 16: clone() 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: -1> 2024-04-26T11:43:43.846+0000 7fcee7291700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcee7291700 time 2024-04-26T11:43:43.846279+0000 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef18ba3dd] 2024-04-26T11:43:44.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: clone() 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 0> 2024-04-26T11:43:43.848+0000 7fcee7291700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:44.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: in thread 7fcee7291700 thread_name:ms_dispatch 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef0642d20] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: gsignal() 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: abort() 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef18ba437] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 16: clone() 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: -1> 2024-04-26T11:43:43.846+0000 7fcee7291700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fcee7291700 time 2024-04-26T11:43:43.846279+0000 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef18ba3dd] 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: clone() 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 0> 2024-04-26T11:43:43.848+0000 7fcee7291700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: in thread 7fcee7291700 thread_name:ms_dispatch 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef0642d20] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2: gsignal() 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 3: abort() 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef18ba437] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fcef18ba5a3] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564617537822] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5646174a7598] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5646174a9c17] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5646173a2d0b] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5646173a32ac] 2024-04-26T11:43:44.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56461738bfad] 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fcef1b4ee20] 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef1b4bfdf] 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef1c1efc1] 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef06381ca] 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 16: clone() 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71588]: 2024-04-26T11:43:44.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:43 smithi194 podman[71783]: 2024-04-26 11:43:43.909230247 +0000 UTC m=+0.021395852 container died 721361dc291170d3b036ea9530d14602c091b5211295e636c87035e316b2ea00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5) 2024-04-26T11:43:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:43 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:rejoin 2024-04-26T11:43:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:43 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:43 smithi033 ceph-mon[27845]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:43:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:44 smithi194 ceph-mon[33017]: pgmap v447: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T11:43:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:44 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:44 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:active 2024-04-26T11:43:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:44 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:44 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:44.854 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:44 smithi194 podman[71783]: 2024-04-26 11:43:44.594827627 +0000 UTC m=+0.706993276 container remove 721361dc291170d3b036ea9530d14602c091b5211295e636c87035e316b2ea00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:43:44.854 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:44 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:43:44.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:44 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T11:43:44.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:44 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.063s CPU time. 2024-04-26T11:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:44 smithi033 ceph-mon[27845]: pgmap v447: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T11:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:44 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:44 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:active 2024-04-26T11:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:44 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:44 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:44 smithi096 ceph-mon[32753]: pgmap v447: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T11:43:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:44 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:43:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:44 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/2388381177,v1:172.21.15.194:6837/2388381177] up:active 2024-04-26T11:43:45.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:44 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:45.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:44 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:43:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:46 smithi194 ceph-mon[33017]: pgmap v448: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 87 KiB/s wr, 22 op/s 2024-04-26T11:43:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:46 smithi033 ceph-mon[27845]: pgmap v448: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 87 KiB/s wr, 22 op/s 2024-04-26T11:43:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:46 smithi096 ceph-mon[32753]: pgmap v448: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 87 KiB/s wr, 22 op/s 2024-04-26T11:43:47.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 750.0 s, 5.4 tps, lat 15098.133 ms stddev 7.316 2024-04-26T11:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:47.224+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:47.224+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:43:47.224+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:43:47.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:43:47 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:43:47.471+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:43:49.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:48 smithi194 ceph-mon[33017]: pgmap v449: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 87 KiB/s wr, 22 op/s 2024-04-26T11:43:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:48 smithi033 ceph-mon[27845]: pgmap v449: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 87 KiB/s wr, 22 op/s 2024-04-26T11:43:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:48 smithi096 ceph-mon[32753]: pgmap v449: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 87 KiB/s wr, 22 op/s 2024-04-26T11:43:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:50 smithi194 ceph-mon[33017]: pgmap v450: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 413 KiB/s wr, 62 op/s 2024-04-26T11:43:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:50 smithi033 ceph-mon[27845]: pgmap v450: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 413 KiB/s wr, 62 op/s 2024-04-26T11:43:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:50 smithi096 ceph-mon[32753]: pgmap v450: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 413 KiB/s wr, 62 op/s 2024-04-26T11:43:52.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 755.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:43:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:52 smithi194 ceph-mon[33017]: pgmap v451: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 426 KiB/s wr, 62 op/s 2024-04-26T11:43:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:52 smithi033 ceph-mon[27845]: pgmap v451: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 426 KiB/s wr, 62 op/s 2024-04-26T11:43:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:52 smithi096 ceph-mon[32753]: pgmap v451: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 426 KiB/s wr, 62 op/s 2024-04-26T11:43:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:54 smithi194 ceph-mon[33017]: pgmap v452: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 422 KiB/s wr, 61 op/s 2024-04-26T11:43:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:54 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:43:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:54 smithi194 systemd[1]: Stopped Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:43:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:54 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.063s CPU time. 2024-04-26T11:43:55.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:54 smithi194 systemd[1]: Starting Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:43:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:54 smithi033 ceph-mon[27845]: pgmap v452: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 422 KiB/s wr, 61 op/s 2024-04-26T11:43:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:54 smithi096 ceph-mon[32753]: pgmap v452: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 422 KiB/s wr, 61 op/s 2024-04-26T11:43:55.399 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 podman[71876]: 2024-04-26 11:43:55.013942856 +0000 UTC m=+0.019494865 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:43:55.400 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 podman[71876]: 2024-04-26 11:43:55.129529498 +0000 UTC m=+0.135081491 container create 689e1d4604643e4894404d82b8f760a1830c6529ee8e505c476518031c6eaa89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T11:43:55.400 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 podman[71876]: 2024-04-26 11:43:55.399601847 +0000 UTC m=+0.405153841 container init 689e1d4604643e4894404d82b8f760a1830c6529ee8e505c476518031c6eaa89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T11:43:55.830 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 podman[71876]: 2024-04-26 11:43:55.402333173 +0000 UTC m=+0.407885175 container start 689e1d4604643e4894404d82b8f760a1830c6529ee8e505c476518031c6eaa89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T11:43:55.830 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mds[71895]: -- 172.21.15.194:0/1384979762 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cfe731a40 con 0x564cfe704c00 2024-04-26T11:43:55.830 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: starting mds.c at 2024-04-26T11:43:55.831 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 bash[71876]: 689e1d4604643e4894404d82b8f760a1830c6529ee8e505c476518031c6eaa89 2024-04-26T11:43:55.831 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:43:55 smithi194 systemd[1]: Started Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: daemon mds.c restarted 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:boot 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:55 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: daemon mds.c restarted 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:boot 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:56.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:55 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: daemon mds.c restarted 2024-04-26T11:43:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:43:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:43:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T11:43:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:boot 2024-04-26T11:43:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-04-26T11:43:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-26T11:43:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:55 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:56 smithi194 ceph-mon[33017]: pgmap v454: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-04-26T11:43:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:56 smithi033 ceph-mon[27845]: pgmap v454: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-04-26T11:43:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:56 smithi096 ceph-mon[32753]: pgmap v454: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-04-26T11:43:57.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 760.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:43:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:57 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:resolve 2024-04-26T11:43:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:57 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:58.105 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:57 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:resolve 2024-04-26T11:43:58.105 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:57 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:57 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:resolve 2024-04-26T11:43:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:57 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:58 smithi194 ceph-mon[33017]: pgmap v455: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-04-26T11:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:58 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:reconnect 2024-04-26T11:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:58 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:58 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:43:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:58 smithi033 ceph-mon[27845]: pgmap v455: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-04-26T11:43:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:58 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:43:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:reconnect 2024-04-26T11:43:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:58 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:58 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:43:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:58 smithi096 ceph-mon[32753]: pgmap v455: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-04-26T11:43:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:58 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199996 2024-04-26T11:43:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:reconnect 2024-04-26T11:43:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:58 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:43:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:58 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:43:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:44:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:rejoin 2024-04-26T11:44:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:59 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:44:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:44:00.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:43:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:44:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:rejoin 2024-04-26T11:44:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:59 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:44:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:44:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:43:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:44:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:rejoin 2024-04-26T11:44:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:59 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:44:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:44:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:43:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:44:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:00 smithi033 ceph-mon[27845]: pgmap v456: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 526 KiB/s wr, 69 op/s 2024-04-26T11:44:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:00 smithi033 ceph-mon[27845]: pgmap v457: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 538 KiB/s wr, 71 op/s 2024-04-26T11:44:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:00 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:clientreplay 2024-04-26T11:44:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:00 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:clientreplay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:00 smithi033 ceph-mon[27845]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:44:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:00 smithi096 ceph-mon[32753]: pgmap v456: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 526 KiB/s wr, 69 op/s 2024-04-26T11:44:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:00 smithi096 ceph-mon[32753]: pgmap v457: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 538 KiB/s wr, 71 op/s 2024-04-26T11:44:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:00 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:clientreplay 2024-04-26T11:44:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:00 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:clientreplay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:00 smithi096 ceph-mon[32753]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:44:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:00 smithi194 ceph-mon[33017]: pgmap v456: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 526 KiB/s wr, 69 op/s 2024-04-26T11:44:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:00 smithi194 ceph-mon[33017]: pgmap v457: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 538 KiB/s wr, 71 op/s 2024-04-26T11:44:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:00 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:clientreplay 2024-04-26T11:44:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:00 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:clientreplay,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:00 smithi194 ceph-mon[33017]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-04-26T11:44:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:01 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:01 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:active 2024-04-26T11:44:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:01 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:01 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:44:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:01 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:01 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:active 2024-04-26T11:44:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:01 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:01 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:44:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 765.0 s, 602.0 tps, lat 971.060 ms stddev 5172.856 2024-04-26T11:44:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:02.239+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:02.239+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:44:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:02.239+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:01 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:01 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2584928981,v1:172.21.15.194:6833/2584928981] up:active 2024-04-26T11:44:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:01 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 2 up:standby-replay 5 up:standby 2024-04-26T11:44:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:01 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby 2024-04-26T11:44:02.909 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:44:02 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:44:02.471+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:44:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:02 smithi033 ceph-mon[27845]: pgmap v458: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.1 MiB/s wr, 148 op/s 2024-04-26T11:44:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:02 smithi096 ceph-mon[32753]: pgmap v458: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.1 MiB/s wr, 148 op/s 2024-04-26T11:44:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:02 smithi194 ceph-mon[33017]: pgmap v458: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.1 MiB/s wr, 148 op/s 2024-04-26T11:44:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:04 smithi033 ceph-mon[27845]: pgmap v459: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 615 KiB/s wr, 82 op/s 2024-04-26T11:44:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:04 smithi096 ceph-mon[32753]: pgmap v459: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 615 KiB/s wr, 82 op/s 2024-04-26T11:44:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:04 smithi194 ceph-mon[33017]: pgmap v459: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 615 KiB/s wr, 82 op/s 2024-04-26T11:44:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:06 smithi096 ceph-mon[32753]: pgmap v460: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-04-26T11:44:07.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 770.0 s, 316.3 tps, lat 60.341 ms stddev 85.298 2024-04-26T11:44:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:06 smithi194 ceph-mon[33017]: pgmap v460: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-04-26T11:44:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:06 smithi033 ceph-mon[27845]: pgmap v460: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.1 MiB/s wr, 236 op/s 2024-04-26T11:44:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:08 smithi033 ceph-mon[27845]: pgmap v461: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-04-26T11:44:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:08 smithi033 ceph-mon[27845]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T11:44:09.349 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-mon[33017]: pgmap v461: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-04-26T11:44:09.350 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-mon[33017]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T11:44:09.352 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c67f06700 time 2024-04-26T11:44:08.872039+0000 2024-04-26T11:44:09.352 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:09.352 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.352 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c7252f3dd] 2024-04-26T11:44:09.352 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.352 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.353 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: clone() 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: *** Caught signal (Aborted) ** 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: in thread 7f2c67f06700 thread_name:ms_dispatch 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:08.872+0000 7f2c67f06700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c67f06700 time 2024-04-26T11:44:08.872039+0000 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c7252f3dd] 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.355 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: clone() 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c712b7d20] 2024-04-26T11:44:09.356 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: gsignal() 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: abort() 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c7252f437] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.357 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 16: clone() 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:08.874+0000 7f2c67f06700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: in thread 7f2c67f06700 thread_name:ms_dispatch 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c712b7d20] 2024-04-26T11:44:09.358 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: gsignal() 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: abort() 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c7252f437] 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.359 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 16: clone() 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: -1> 2024-04-26T11:44:08.872+0000 7f2c67f06700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c67f06700 time 2024-04-26T11:44:08.872039+0000 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:09.360 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c7252f3dd] 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.361 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: clone() 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 0> 2024-04-26T11:44:08.874+0000 7f2c67f06700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: in thread 7f2c67f06700 thread_name:ms_dispatch 2024-04-26T11:44:09.362 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c712b7d20] 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: gsignal() 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: abort() 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c7252f437] 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.363 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 16: clone() 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:09.364 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: -1> 2024-04-26T11:44:08.872+0000 7f2c67f06700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f2c67f06700 time 2024-04-26T11:44:08.872039+0000 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2c7252f3dd] 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.365 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: clone() 2024-04-26T11:44:09.366 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 0> 2024-04-26T11:44:08.874+0000 7f2c67f06700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: in thread 7f2c67f06700 thread_name:ms_dispatch 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2c712b7d20] 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2: gsignal() 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 3: abort() 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2c7252f437] 2024-04-26T11:44:09.367 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2c7252f5a3] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x56006854a822] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5600684ba598] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5600684bcc17] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5600683b5d0b] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5600683b62ac] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x56006839efad] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f2c727c3e20] 2024-04-26T11:44:09.368 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 13: (DispatchQueue::entry()+0x51f) [0x7f2c727c0fdf] 2024-04-26T11:44:09.369 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2c72893fc1] 2024-04-26T11:44:09.369 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2c712ad1ca] 2024-04-26T11:44:09.369 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 16: clone() 2024-04-26T11:44:09.369 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:09.369 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l[71744]: 2024-04-26T11:44:09.369 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:08 smithi194 podman[72069]: 2024-04-26 11:44:08.932204416 +0000 UTC m=+0.018933754 container died 288f543a76c602838df0560ee823ac151efbf704c87183d5b3f50b0e4ee60b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:44:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:08 smithi096 ceph-mon[32753]: pgmap v461: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 3.1 MiB/s wr, 281 op/s 2024-04-26T11:44:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:08 smithi096 ceph-mon[32753]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T11:44:09.911 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:09 smithi194 podman[72069]: 2024-04-26 11:44:09.655575608 +0000 UTC m=+0.742304929 container remove 288f543a76c602838df0560ee823ac151efbf704c87183d5b3f50b0e4ee60b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T11:44:09.911 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:09 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:44:10.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:09 smithi033 ceph-mon[27845]: pgmap v463: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 3.0 MiB/s wr, 269 op/s 2024-04-26T11:44:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:09 smithi194 ceph-mon[33017]: pgmap v463: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 3.0 MiB/s wr, 269 op/s 2024-04-26T11:44:10.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:09 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-26T11:44:10.354 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:09 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.272s CPU time. 2024-04-26T11:44:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:09 smithi096 ceph-mon[32753]: pgmap v463: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 3.0 MiB/s wr, 269 op/s 2024-04-26T11:44:12.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 775.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:44:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:12 smithi194 ceph-mon[33017]: pgmap v464: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 3.1 MiB/s wr, 277 op/s 2024-04-26T11:44:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:12 smithi194 ceph-mon[33017]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T11:44:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:12 smithi033 ceph-mon[27845]: pgmap v464: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 3.1 MiB/s wr, 277 op/s 2024-04-26T11:44:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:12 smithi033 ceph-mon[27845]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T11:44:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:12 smithi096 ceph-mon[32753]: pgmap v464: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 3.1 MiB/s wr, 277 op/s 2024-04-26T11:44:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:12 smithi096 ceph-mon[32753]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T11:44:14.735 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:14 smithi096 ceph-mon[32753]: pgmap v466: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-04-26T11:44:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:14 smithi194 ceph-mon[33017]: pgmap v466: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-04-26T11:44:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:14 smithi033 ceph-mon[27845]: pgmap v466: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-04-26T11:44:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:16 smithi194 ceph-mon[33017]: pgmap v467: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 997 KiB/s wr, 132 op/s 2024-04-26T11:44:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:16 smithi033 ceph-mon[27845]: pgmap v467: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 997 KiB/s wr, 132 op/s 2024-04-26T11:44:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:16 smithi096 ceph-mon[32753]: pgmap v467: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 997 KiB/s wr, 132 op/s 2024-04-26T11:44:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 780.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:44:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:17.242+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:17.242+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:17.242+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:44:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:18 smithi194 ceph-mon[33017]: pgmap v468: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 944 KiB/s wr, 125 op/s 2024-04-26T11:44:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:18 smithi033 ceph-mon[27845]: pgmap v468: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 944 KiB/s wr, 125 op/s 2024-04-26T11:44:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:18 smithi096 ceph-mon[32753]: pgmap v468: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 944 KiB/s wr, 125 op/s 2024-04-26T11:44:20.488 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:20 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-04-26T11:44:20.488 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:20 smithi194 systemd[1]: Stopped Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:44:20.488 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:20 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.272s CPU time. 2024-04-26T11:44:20.488 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:20 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-04-26T11:44:20.488 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:20 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-26T11:44:20.489 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 26 11:44:20 smithi194 systemd[1]: Failed to start Ceph mds.l for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:44:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:20 smithi194 ceph-mon[33017]: pgmap v469: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 883 KiB/s wr, 116 op/s 2024-04-26T11:44:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:20 smithi033 ceph-mon[27845]: pgmap v469: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 883 KiB/s wr, 116 op/s 2024-04-26T11:44:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:20 smithi096 ceph-mon[32753]: pgmap v469: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 883 KiB/s wr, 116 op/s 2024-04-26T11:44:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:21.723+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00034c0 con 0x7fef00077780 2024-04-26T11:44:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:21.723+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00034c0 con 0x7fef00077780 2024-04-26T11:44:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:21.734+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0034c0 con 0x564385821900 2024-04-26T11:44:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:21.735+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0034c0 con 0x564385821900 2024-04-26T11:44:22.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 785.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:44:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:22 smithi194 ceph-mon[33017]: pgmap v470: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:44:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:22 smithi033 ceph-mon[27845]: pgmap v470: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:44:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:22 smithi096 ceph-mon[32753]: pgmap v470: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-26T11:44:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:24 smithi194 ceph-mon[33017]: pgmap v471: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-04-26T11:44:24.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:24 smithi033 ceph-mon[27845]: pgmap v471: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-04-26T11:44:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:24 smithi096 ceph-mon[32753]: pgmap v471: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-04-26T11:44:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:26 smithi194 ceph-mon[33017]: pgmap v472: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-04-26T11:44:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:26 smithi194 ceph-mon[33017]: Replacing daemon mds.l as rank 1 with standby daemon mds.c 2024-04-26T11:44:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:26 smithi194 ceph-mon[33017]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-26T11:44:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:26 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:26 smithi194 ceph-mon[33017]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T11:44:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:26 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:26 smithi033 ceph-mon[27845]: pgmap v472: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-04-26T11:44:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:26 smithi033 ceph-mon[27845]: Replacing daemon mds.l as rank 1 with standby daemon mds.c 2024-04-26T11:44:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:26 smithi033 ceph-mon[27845]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-26T11:44:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:26 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:26 smithi033 ceph-mon[27845]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T11:44:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:26 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:26 smithi096 ceph-mon[32753]: pgmap v472: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-04-26T11:44:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:26 smithi096 ceph-mon[32753]: Replacing daemon mds.l as rank 1 with standby daemon mds.c 2024-04-26T11:44:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:26 smithi096 ceph-mon[32753]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-26T11:44:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:26 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:26 smithi096 ceph-mon[32753]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T11:44:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:26 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 790.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:44:28.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:28 smithi033 ceph-mon[27845]: pgmap v474: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-04-26T11:44:28.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:28 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:resolve 2024-04-26T11:44:28.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:28 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:28 smithi194 ceph-mon[33017]: pgmap v474: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-04-26T11:44:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:28 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:resolve 2024-04-26T11:44:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:28 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:28 smithi096 ceph-mon[32753]: pgmap v474: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-04-26T11:44:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:28 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:resolve 2024-04-26T11:44:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:28 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:29 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:reconnect 2024-04-26T11:44:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:29 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:29 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00799984 2024-04-26T11:44:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:29 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:44:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:29 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:reconnect 2024-04-26T11:44:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:29 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:29 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00799984 2024-04-26T11:44:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:29 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:29 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:reconnect 2024-04-26T11:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:29 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:29 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00799984 2024-04-26T11:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:29 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:44:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-mon[33017]: pgmap v475: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-04-26T11:44:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:rejoin 2024-04-26T11:44:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-mon[33017]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:44:30.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9796e83700 time 2024-04-26T11:44:30.593692+0000 2024-04-26T11:44:30.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:30.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f97a14ac3dd] 2024-04-26T11:44:30.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.855 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: clone() 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.595+0000 7f9796e83700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9796e83700 time 2024-04-26T11:44:30.593692+0000 2024-04-26T11:44:30.856 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f97a14ac3dd] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.857 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: clone() 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: *** Caught signal (Aborted) ** 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: in thread 7f9796e83700 thread_name:ms_dispatch 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f97a0234d20] 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: gsignal() 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: abort() 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f97a14ac437] 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.858 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.859 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.859 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 16: clone() 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.596+0000 7f9796e83700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: in thread 7f9796e83700 thread_name:ms_dispatch 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f97a0234d20] 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: gsignal() 2024-04-26T11:44:30.860 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: abort() 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f97a14ac437] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 16: clone() 2024-04-26T11:44:30.861 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: -1> 2024-04-26T11:44:30.595+0000 7f9796e83700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9796e83700 time 2024-04-26T11:44:30.593692+0000 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f97a14ac3dd] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.862 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: clone() 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 0> 2024-04-26T11:44:30.596+0000 7f9796e83700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: in thread 7f9796e83700 thread_name:ms_dispatch 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f97a0234d20] 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: gsignal() 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: abort() 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f97a14ac437] 2024-04-26T11:44:30.863 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 16: clone() 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: -3> 2024-04-26T11:44:30.595+0000 7f9796e83700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9796e83700 time 2024-04-26T11:44:30.593692+0000 2024-04-26T11:44:30.864 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f97a14ac3dd] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.865 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: clone() 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: -2> 2024-04-26T11:44:30.596+0000 7f9796e83700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: in thread 7f9796e83700 thread_name:ms_dispatch 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f97a0234d20] 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2: gsignal() 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 3: abort() 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f97a14ac437] 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f97a14ac5a3] 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x564cfc177822] 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x564cfc0e7598] 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564cfc0e9c17] 2024-04-26T11:44:30.866 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564cfbfe2d0b] 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564cfbfe32ac] 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x564cfbfcbfad] 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f97a1740e20] 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 13: (DispatchQueue::entry()+0x51f) [0x7f97a173dfdf] 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f97a1810fc1] 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f97a022a1ca] 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 16: clone() 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[71891]: 2024-04-26T11:44:30.867 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:30 smithi194 podman[72119]: 2024-04-26 11:44:30.656619247 +0000 UTC m=+0.019477970 container died 689e1d4604643e4894404d82b8f760a1830c6529ee8e505c476518031c6eaa89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T11:44:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:30 smithi033 ceph-mon[27845]: pgmap v475: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-04-26T11:44:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:30 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:rejoin 2024-04-26T11:44:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:30 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:30 smithi033 ceph-mon[27845]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:44:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:30 smithi096 ceph-mon[32753]: pgmap v475: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-04-26T11:44:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:30 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:rejoin 2024-04-26T11:44:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:30 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:30 smithi096 ceph-mon[32753]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:44:31.576 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:31 smithi194 podman[72119]: 2024-04-26 11:44:31.318292099 +0000 UTC m=+0.681150857 container remove 689e1d4604643e4894404d82b8f760a1830c6529ee8e505c476518031c6eaa89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T11:44:31.576 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:31 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:44:31.576 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:31 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T11:44:31.576 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:31 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.206s CPU time. 2024-04-26T11:44:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:31 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:active 2024-04-26T11:44:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:31 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:31 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:44:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:31 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:active 2024-04-26T11:44:31.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:31 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:31.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:31 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:44:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:31 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6836/1168242101,v1:172.21.15.194:6837/1168242101] up:active 2024-04-26T11:44:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:31 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:31 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:44:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 795.0 s, 4.4 tps, lat 27564.462 ms stddev 13.925 2024-04-26T11:44:32.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:32.244+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:32.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:32.244+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:44:32.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:32.244+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:32.833 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:32 smithi033 ceph-mon[27845]: pgmap v476: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 594 KiB/s wr, 80 op/s 2024-04-26T11:44:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:32 smithi194 ceph-mon[33017]: pgmap v476: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 594 KiB/s wr, 80 op/s 2024-04-26T11:44:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:32 smithi096 ceph-mon[32753]: pgmap v476: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 594 KiB/s wr, 80 op/s 2024-04-26T11:44:32.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:44:32 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:44:32.477+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:44:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:34 smithi194 ceph-mon[33017]: pgmap v477: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 594 KiB/s wr, 80 op/s 2024-04-26T11:44:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:34 smithi033 ceph-mon[27845]: pgmap v477: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 594 KiB/s wr, 80 op/s 2024-04-26T11:44:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:34 smithi096 ceph-mon[32753]: pgmap v477: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 594 KiB/s wr, 80 op/s 2024-04-26T11:44:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:36 smithi194 ceph-mon[33017]: pgmap v478: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 695 KiB/s wr, 92 op/s 2024-04-26T11:44:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:36 smithi033 ceph-mon[27845]: pgmap v478: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 695 KiB/s wr, 92 op/s 2024-04-26T11:44:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:36 smithi096 ceph-mon[32753]: pgmap v478: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 695 KiB/s wr, 92 op/s 2024-04-26T11:44:37.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 800.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:44:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:38 smithi194 ceph-mon[33017]: pgmap v479: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 642 KiB/s wr, 85 op/s 2024-04-26T11:44:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:38 smithi033 ceph-mon[27845]: pgmap v479: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 642 KiB/s wr, 85 op/s 2024-04-26T11:44:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:38 smithi096 ceph-mon[32753]: pgmap v479: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 642 KiB/s wr, 85 op/s 2024-04-26T11:44:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:40 smithi194 ceph-mon[33017]: pgmap v480: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 579 KiB/s wr, 76 op/s 2024-04-26T11:44:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:40 smithi033 ceph-mon[27845]: pgmap v480: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 579 KiB/s wr, 76 op/s 2024-04-26T11:44:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:40 smithi096 ceph-mon[32753]: pgmap v480: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 579 KiB/s wr, 76 op/s 2024-04-26T11:44:41.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:41 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:44:41.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:41 smithi194 systemd[1]: Stopped Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:44:41.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:41 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.206s CPU time. 2024-04-26T11:44:41.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:41 smithi194 systemd[1]: Starting Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:44:41.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:41 smithi194 podman[72228]: 2024-04-26 11:44:41.736647198 +0000 UTC m=+0.018325609 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:44:41.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:41 smithi194 podman[72228]: 2024-04-26 11:44:41.835457896 +0000 UTC m=+0.117136295 container create dbdb3091957a25cf1421f8fa703235a2c04cfcc55e84d077e675c03392df124b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:44:42.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 805.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:44:42.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:42 smithi194 podman[72228]: 2024-04-26 11:44:42.096669957 +0000 UTC m=+0.378348353 container init dbdb3091957a25cf1421f8fa703235a2c04cfcc55e84d077e675c03392df124b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T11:44:42.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:42 smithi194 podman[72228]: 2024-04-26 11:44:42.099652908 +0000 UTC m=+0.381331305 container start dbdb3091957a25cf1421f8fa703235a2c04cfcc55e84d077e675c03392df124b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:44:42.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mds[72247]: -- 172.21.15.194:0/3564062318 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557686189a40 con 0x55768615cc00 2024-04-26T11:44:42.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: starting mds.c at 2024-04-26T11:44:42.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:42 smithi194 bash[72228]: dbdb3091957a25cf1421f8fa703235a2c04cfcc55e84d077e675c03392df124b 2024-04-26T11:44:42.602 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:44:42 smithi194 systemd[1]: Started Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:44:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: pgmap v481: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 583 KiB/s wr, 80 op/s 2024-04-26T11:44:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: daemon mds.c restarted 2024-04-26T11:44:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:44:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T11:44:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:boot 2024-04-26T11:44:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:44:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-26T11:44:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:42 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: pgmap v481: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 583 KiB/s wr, 80 op/s 2024-04-26T11:44:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: daemon mds.c restarted 2024-04-26T11:44:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:44:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T11:44:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:boot 2024-04-26T11:44:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:44:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-26T11:44:42.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:42 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: pgmap v481: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 583 KiB/s wr, 80 op/s 2024-04-26T11:44:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: daemon mds.c restarted 2024-04-26T11:44:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T11:44:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T11:44:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:boot 2024-04-26T11:44:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:44:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-26T11:44:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:42 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:44.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:44 smithi033 ceph-mon[27845]: pgmap v483: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 546 KiB/s wr, 73 op/s 2024-04-26T11:44:44.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:44 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:resolve 2024-04-26T11:44:44.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:44 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:44 smithi096 ceph-mon[32753]: pgmap v483: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 546 KiB/s wr, 73 op/s 2024-04-26T11:44:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:44 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:resolve 2024-04-26T11:44:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:44 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:44 smithi194 ceph-mon[33017]: pgmap v483: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 546 KiB/s wr, 73 op/s 2024-04-26T11:44:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:44 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:resolve 2024-04-26T11:44:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:44 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:45 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:44:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:45 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:reconnect 2024-04-26T11:44:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:45 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:45.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:45 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:44:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:45 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:44:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:45 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:reconnect 2024-04-26T11:44:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:45 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:45 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:44:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:45 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:44:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:45 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:reconnect 2024-04-26T11:44:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:45 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:45 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:44:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-mon[27845]: pgmap v484: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 64 KiB/s wr, 14 op/s 2024-04-26T11:44:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:rejoin 2024-04-26T11:44:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-mon[27845]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:44:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:active 2024-04-26T11:44:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe8d3a00700 time 2024-04-26T11:44:46.529060+0000 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8df3b33dd] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: clone() 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.530+0000 7fe8d3a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe8d3a00700 time 2024-04-26T11:44:46.529060+0000 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8df3b33dd] 2024-04-26T11:44:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: clone() 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: *** Caught signal (Aborted) ** 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: in thread 7fe8d3a00700 thread_name:ms_dispatch 2024-04-26T11:44:46.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8de13bd20] 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: gsignal() 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: abort() 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8df3b3437] 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:46 smithi096 ceph-mon[32753]: pgmap v484: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 64 KiB/s wr, 14 op/s 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:46 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:rejoin 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:46 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:46 smithi096 ceph-mon[32753]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:46 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:46 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:active 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:46 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 16: clone() 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.534+0000 7fe8d3a00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: in thread 7fe8d3a00700 thread_name:ms_dispatch 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8de13bd20] 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: gsignal() 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: abort() 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8df3b3437] 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 16: clone() 2024-04-26T11:44:46.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: -1> 2024-04-26T11:44:46.530+0000 7fe8d3a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe8d3a00700 time 2024-04-26T11:44:46.529060+0000 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8df3b33dd] 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: clone() 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 0> 2024-04-26T11:44:46.534+0000 7fe8d3a00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: in thread 7fe8d3a00700 thread_name:ms_dispatch 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8de13bd20] 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: gsignal() 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: abort() 2024-04-26T11:44:46.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8df3b3437] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 16: clone() 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:46.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: -1> 2024-04-26T11:44:46.530+0000 7fe8d3a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe8d3a00700 time 2024-04-26T11:44:46.529060+0000 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8df3b33dd] 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: clone() 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 0> 2024-04-26T11:44:46.534+0000 7fe8d3a00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: in thread 7fe8d3a00700 thread_name:ms_dispatch 2024-04-26T11:44:46.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8de13bd20] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2: gsignal() 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 3: abort() 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8df3b3437] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe8df3b35a3] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55d033e63822] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55d033dd3598] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d033dd5c17] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d033cced0b] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d033ccf2ac] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55d033cb7fad] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe8df647e20] 2024-04-26T11:44:46.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8df644fdf] 2024-04-26T11:44:46.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8df717fc1] 2024-04-26T11:44:46.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8de1311ca] 2024-04-26T11:44:46.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 16: clone() 2024-04-26T11:44:46.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:44:46.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[91976]: 2024-04-26T11:44:46.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:46 smithi033 podman[93063]: 2024-04-26 11:44:46.694894047 +0000 UTC m=+0.048609324 container died 8d808b21a8e0c75436839671474b91b05c9d8c8c76e0415a5821d6476ce897cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T11:44:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:46 smithi194 ceph-mon[33017]: pgmap v484: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 64 KiB/s wr, 14 op/s 2024-04-26T11:44:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:46 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:rejoin 2024-04-26T11:44:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:46 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:46 smithi194 ceph-mon[33017]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:44:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:46 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:44:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:46 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2024122458,v1:172.21.15.33:6837/2024122458] up:active 2024-04-26T11:44:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:46 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:44:47.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 810.0 s, 247.9 tps, lat 3067.328 ms stddev 10702.797 2024-04-26T11:44:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:47.259+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:47.259+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:44:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:44:47.259+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:44:47.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:44:47 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:44:47.478+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:44:48.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:47 smithi033 podman[93063]: 2024-04-26 11:44:47.758344215 +0000 UTC m=+1.112059515 container remove 8d808b21a8e0c75436839671474b91b05c9d8c8c76e0415a5821d6476ce897cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:44:48.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:47 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:44:48.646 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:48 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-26T11:44:48.646 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:48 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.155s CPU time. 2024-04-26T11:44:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:48 smithi033 ceph-mon[27845]: pgmap v485: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-26T11:44:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:48 smithi096 ceph-mon[32753]: pgmap v485: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-26T11:44:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:48 smithi194 ceph-mon[33017]: pgmap v485: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-26T11:44:50.917 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:50 smithi194 ceph-mon[33017]: pgmap v486: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-26T11:44:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:50 smithi033 ceph-mon[27845]: pgmap v486: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-26T11:44:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:50 smithi096 ceph-mon[32753]: pgmap v486: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-26T11:44:52.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 815.0 s, 44.0 tps, lat 33.738 ms stddev 34.566 2024-04-26T11:44:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:52 smithi033 ceph-mon[27845]: pgmap v487: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 706 KiB/s wr, 53 op/s 2024-04-26T11:44:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:52 smithi096 ceph-mon[32753]: pgmap v487: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 706 KiB/s wr, 53 op/s 2024-04-26T11:44:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:52 smithi194 ceph-mon[33017]: pgmap v487: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 706 KiB/s wr, 53 op/s 2024-04-26T11:44:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:54 smithi033 ceph-mon[27845]: pgmap v488: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 633 KiB/s wr, 47 op/s 2024-04-26T11:44:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:54 smithi096 ceph-mon[32753]: pgmap v488: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 633 KiB/s wr, 47 op/s 2024-04-26T11:44:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:54 smithi194 ceph-mon[33017]: pgmap v488: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 633 KiB/s wr, 47 op/s 2024-04-26T11:44:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:56 smithi033 ceph-mon[27845]: pgmap v489: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 870 KiB/s wr, 81 op/s 2024-04-26T11:44:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:56 smithi096 ceph-mon[32753]: pgmap v489: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 870 KiB/s wr, 81 op/s 2024-04-26T11:44:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:56 smithi194 ceph-mon[33017]: pgmap v489: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 870 KiB/s wr, 81 op/s 2024-04-26T11:44:57.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 820.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:44:58.797 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:58 smithi033 ceph-mon[27845]: pgmap v490: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 919 KiB/s wr, 85 op/s 2024-04-26T11:44:58.798 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:58 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:44:58.798 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:58 smithi033 systemd[1]: Stopped Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:44:58.798 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:58 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.155s CPU time. 2024-04-26T11:44:58.798 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:58 smithi033 systemd[1]: Starting Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:44:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:58 smithi096 ceph-mon[32753]: pgmap v490: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 919 KiB/s wr, 85 op/s 2024-04-26T11:44:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:58 smithi194 ceph-mon[33017]: pgmap v490: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 919 KiB/s wr, 85 op/s 2024-04-26T11:44:59.130 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:58 smithi033 podman[93155]: 2024-04-26 11:44:58.703153991 +0000 UTC m=+0.034679818 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:44:59.130 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:58 smithi033 podman[93155]: 2024-04-26 11:44:58.817519891 +0000 UTC m=+0.149045780 container create e1459739bc7bb2c8c76b6958ee3ada2c0fbf714709f120bbd1790b07174979c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-26T11:44:59.426 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:59 smithi033 podman[93155]: 2024-04-26 11:44:59.12927706 +0000 UTC m=+0.460802887 container init e1459739bc7bb2c8c76b6958ee3ada2c0fbf714709f120bbd1790b07174979c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:44:59.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:59 smithi033 podman[93155]: 2024-04-26 11:44:59.139468785 +0000 UTC m=+0.470994609 container start e1459739bc7bb2c8c76b6958ee3ada2c0fbf714709f120bbd1790b07174979c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-26T11:44:59.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mds[93173]: -- 172.21.15.33:0/3856039846 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f18de5a40 con 0x563f18db9000 2024-04-26T11:44:59.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: starting mds.d at 2024-04-26T11:44:59.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:59 smithi033 bash[93155]: e1459739bc7bb2c8c76b6958ee3ada2c0fbf714709f120bbd1790b07174979c1 2024-04-26T11:44:59.427 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:44:59 smithi033 systemd[1]: Started Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: daemon mds.d restarted 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: osdmap e138: 12 total, 12 up, 12 in 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:boot 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:44:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:59.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:44:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: daemon mds.d restarted 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: osdmap e138: 12 total, 12 up, 12 in 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:boot 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: daemon mds.d restarted 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: osdmap e138: 12 total, 12 up, 12 in 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:boot 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:replay,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:45:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:00 smithi096 ceph-mon[32753]: pgmap v492: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-04-26T11:45:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:00 smithi194 ceph-mon[33017]: pgmap v492: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-04-26T11:45:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:00 smithi033 ceph-mon[27845]: pgmap v492: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-04-26T11:45:01.944 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:01 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:resolve 2024-04-26T11:45:01.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:01 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:01.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:45:01.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:45:01.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:01 smithi033 ceph-mon[27845]: pgmap v493: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 812 KiB/s wr, 108 op/s 2024-04-26T11:45:01.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:45:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:01 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:resolve 2024-04-26T11:45:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:01 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:45:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:45:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:01 smithi194 ceph-mon[33017]: pgmap v493: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 812 KiB/s wr, 108 op/s 2024-04-26T11:45:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:45:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:01 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:resolve 2024-04-26T11:45:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:01 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:resolve,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:45:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:45:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:01 smithi096 ceph-mon[32753]: pgmap v493: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 812 KiB/s wr, 108 op/s 2024-04-26T11:45:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:45:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 825.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:02.274+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:02.274+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:45:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:02.274+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:02 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:reconnect 2024-04-26T11:45:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:02 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:02 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899982 2024-04-26T11:45:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:02 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:45:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:02 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:reconnect 2024-04-26T11:45:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:02 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:02 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899982 2024-04-26T11:45:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:02 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:45:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:02 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:reconnect 2024-04-26T11:45:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:02 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:reconnect,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:02 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00899982 2024-04-26T11:45:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:02 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0099998 2024-04-26T11:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:rejoin 2024-04-26T11:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-mon[33017]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-mon[33017]: pgmap v494: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 812 KiB/s wr, 108 op/s 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6271bd9700 time 2024-04-26T11:45:03.771331+0000 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f627c2023dd] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.105 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: clone() 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:03.771+0000 7f6271bd9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6271bd9700 time 2024-04-26T11:45:03.771331+0000 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f627c2023dd] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.106 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: clone() 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: *** Caught signal (Aborted) ** 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: in thread 7f6271bd9700 thread_name:ms_dispatch 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f627af8ad20] 2024-04-26T11:45:04.107 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: gsignal() 2024-04-26T11:45:04.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: abort() 2024-04-26T11:45:04.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f627c202437] 2024-04-26T11:45:04.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.108 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 16: clone() 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:03.773+0000 7f6271bd9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:04.109 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: in thread 7f6271bd9700 thread_name:ms_dispatch 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f627af8ad20] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: gsignal() 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: abort() 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f627c202437] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.110 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 16: clone() 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: -1> 2024-04-26T11:45:03.771+0000 7f6271bd9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6271bd9700 time 2024-04-26T11:45:03.771331+0000 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f627c2023dd] 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.111 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: clone() 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 0> 2024-04-26T11:45:03.773+0000 7f6271bd9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: in thread 7f6271bd9700 thread_name:ms_dispatch 2024-04-26T11:45:04.112 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f627af8ad20] 2024-04-26T11:45:04.113 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: gsignal() 2024-04-26T11:45:04.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: abort() 2024-04-26T11:45:04.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f627c202437] 2024-04-26T11:45:04.114 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 16: clone() 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:04.115 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: -1> 2024-04-26T11:45:03.771+0000 7f6271bd9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6271bd9700 time 2024-04-26T11:45:03.771331+0000 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f627c2023dd] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.116 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: clone() 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 0> 2024-04-26T11:45:03.773+0000 7f6271bd9700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: in thread 7f6271bd9700 thread_name:ms_dispatch 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f627af8ad20] 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2: gsignal() 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 3: abort() 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f627c202437] 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f627c2025a3] 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557684c89822] 2024-04-26T11:45:04.117 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557684bf9598] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557684bfbc17] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557684af4d0b] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557684af52ac] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557684addfad] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f627c496e20] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 13: (DispatchQueue::entry()+0x51f) [0x7f627c493fdf] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f627c566fc1] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f627af801ca] 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 16: clone() 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c[72243]: 2024-04-26T11:45:04.118 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:03 smithi194 podman[72415]: 2024-04-26 11:45:03.836614259 +0000 UTC m=+0.021328466 container died dbdb3091957a25cf1421f8fa703235a2c04cfcc55e84d077e675c03392df124b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T11:45:04.170 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:03 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:rejoin 2024-04-26T11:45:04.171 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:03 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:04.171 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:03 smithi033 ceph-mon[27845]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:45:04.171 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:03 smithi033 ceph-mon[27845]: pgmap v494: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 812 KiB/s wr, 108 op/s 2024-04-26T11:45:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:03 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:rejoin 2024-04-26T11:45:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:03 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=c=up:rejoin,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:03 smithi096 ceph-mon[32753]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-26T11:45:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:03 smithi096 ceph-mon[32753]: pgmap v494: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 812 KiB/s wr, 108 op/s 2024-04-26T11:45:04.721 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:04 smithi194 podman[72415]: 2024-04-26 11:45:04.46865956 +0000 UTC m=+0.653373749 container remove dbdb3091957a25cf1421f8fa703235a2c04cfcc55e84d077e675c03392df124b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:45:04.721 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:04 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:45:05.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:04 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T11:45:05.104 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:04 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.255s CPU time. 2024-04-26T11:45:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:04 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:04 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:active 2024-04-26T11:45:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:04 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:04 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:45:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:04 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:04 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:active 2024-04-26T11:45:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:04 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:04 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:45:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:04 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:04 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6832/2530658283,v1:172.21.15.194:6833/2530658283] up:active 2024-04-26T11:45:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:04 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 2 up:standby-replay 4 up:standby 2024-04-26T11:45:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:04 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=c=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby 2024-04-26T11:45:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:05 smithi194 ceph-mon[33017]: pgmap v495: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-26T11:45:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:05 smithi033 ceph-mon[27845]: pgmap v495: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-26T11:45:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:05 smithi096 ceph-mon[32753]: pgmap v495: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-26T11:45:07.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 830.0 s, 1097.5 tps, lat 351.053 ms stddev 2228.378 2024-04-26T11:45:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:07 smithi194 ceph-mon[33017]: pgmap v496: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.6 MiB/s wr, 130 op/s 2024-04-26T11:45:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:07 smithi033 ceph-mon[27845]: pgmap v496: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.6 MiB/s wr, 130 op/s 2024-04-26T11:45:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:07 smithi096 ceph-mon[32753]: pgmap v496: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.6 MiB/s wr, 130 op/s 2024-04-26T11:45:07.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:45:07 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:45:07.479+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:45:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:09 smithi194 ceph-mon[33017]: pgmap v497: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 129 op/s 2024-04-26T11:45:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:09 smithi033 ceph-mon[27845]: pgmap v497: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 129 op/s 2024-04-26T11:45:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:09 smithi096 ceph-mon[32753]: pgmap v497: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 129 op/s 2024-04-26T11:45:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:11 smithi194 ceph-mon[33017]: pgmap v498: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 3.1 MiB/s wr, 280 op/s 2024-04-26T11:45:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:11 smithi033 ceph-mon[27845]: pgmap v498: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 3.1 MiB/s wr, 280 op/s 2024-04-26T11:45:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:11 smithi096 ceph-mon[32753]: pgmap v498: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 3.1 MiB/s wr, 280 op/s 2024-04-26T11:45:12.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 835.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:13 smithi194 ceph-mon[33017]: pgmap v499: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.7 MiB/s wr, 227 op/s 2024-04-26T11:45:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:13 smithi033 ceph-mon[27845]: pgmap v499: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.7 MiB/s wr, 227 op/s 2024-04-26T11:45:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:13 smithi096 ceph-mon[32753]: pgmap v499: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.7 MiB/s wr, 227 op/s 2024-04-26T11:45:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:15 smithi194 ceph-mon[33017]: pgmap v500: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-26T11:45:15.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:15 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 6. 2024-04-26T11:45:15.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:15 smithi194 systemd[1]: Stopped Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:45:15.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:15 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.255s CPU time. 2024-04-26T11:45:15.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:15 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Start request repeated too quickly. 2024-04-26T11:45:15.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:15 smithi194 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T11:45:15.354 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 26 11:45:15 smithi194 systemd[1]: Failed to start Ceph mds.c for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:45:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:15 smithi033 ceph-mon[27845]: pgmap v500: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-26T11:45:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:15 smithi096 ceph-mon[32753]: pgmap v500: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-26T11:45:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 840.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:17.289+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:17.289+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:45:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:17.289+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:17 smithi194 ceph-mon[33017]: pgmap v501: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-26T11:45:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:17 smithi033 ceph-mon[27845]: pgmap v501: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-26T11:45:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:17 smithi096 ceph-mon[32753]: pgmap v501: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-26T11:45:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:19 smithi194 ceph-mon[33017]: pgmap v502: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-04-26T11:45:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:19 smithi033 ceph-mon[27845]: pgmap v502: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-04-26T11:45:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:19 smithi096 ceph-mon[32753]: pgmap v502: 97 pgs: 97 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-04-26T11:45:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:21 smithi194 ceph-mon[33017]: pgmap v503: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-04-26T11:45:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:21 smithi033 ceph-mon[27845]: pgmap v503: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-04-26T11:45:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:21 smithi096 ceph-mon[32753]: pgmap v503: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-04-26T11:45:22.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 845.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:22 smithi194 ceph-mon[33017]: Replacing daemon mds.c as rank 1 with standby daemon mds.d 2024-04-26T11:45:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:22 smithi194 ceph-mon[33017]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-26T11:45:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:22 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:22 smithi194 ceph-mon[33017]: osdmap e139: 12 total, 12 up, 12 in 2024-04-26T11:45:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:22 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:22 smithi033 ceph-mon[27845]: Replacing daemon mds.c as rank 1 with standby daemon mds.d 2024-04-26T11:45:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:22 smithi033 ceph-mon[27845]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-26T11:45:22.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:22 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:22.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:22 smithi033 ceph-mon[27845]: osdmap e139: 12 total, 12 up, 12 in 2024-04-26T11:45:22.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:22 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:22 smithi096 ceph-mon[32753]: Replacing daemon mds.c as rank 1 with standby daemon mds.d 2024-04-26T11:45:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:22 smithi096 ceph-mon[32753]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-26T11:45:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:22 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:22 smithi096 ceph-mon[32753]: osdmap e139: 12 total, 12 up, 12 in 2024-04-26T11:45:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:22 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:23 smithi194 ceph-mon[33017]: pgmap v505: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 MiB/s wr, 195 op/s 2024-04-26T11:45:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:23 smithi033 ceph-mon[27845]: pgmap v505: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 MiB/s wr, 195 op/s 2024-04-26T11:45:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:23 smithi096 ceph-mon[32753]: pgmap v505: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 MiB/s wr, 195 op/s 2024-04-26T11:45:24.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:resolve 2024-04-26T11:45:24.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:24 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:resolve 2024-04-26T11:45:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:24 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:resolve 2024-04-26T11:45:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:24 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:25 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:45:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:reconnect 2024-04-26T11:45:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:25 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:25 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0199997 2024-04-26T11:45:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:25 smithi194 ceph-mon[33017]: pgmap v506: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-26T11:45:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:25 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:45:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:reconnect 2024-04-26T11:45:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:25 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:25 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0199997 2024-04-26T11:45:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:25 smithi033 ceph-mon[27845]: pgmap v506: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-26T11:45:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:25 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:45:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:reconnect 2024-04-26T11:45:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:25 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:25 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0199997 2024-04-26T11:45:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:25 smithi096 ceph-mon[32753]: pgmap v506: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-26T11:45:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:26 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:rejoin 2024-04-26T11:45:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:26 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:26.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:rejoin 2024-04-26T11:45:26.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:26 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:rejoin 2024-04-26T11:45:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:26 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:26.747 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5cf4200700 time 2024-04-26T11:45:26.544149+0000 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5cffa493dd] 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.748 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: clone() 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.545+0000 7f5cf4200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5cf4200700 time 2024-04-26T11:45:26.544149+0000 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5cffa493dd] 2024-04-26T11:45:26.749 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: clone() 2024-04-26T11:45:26.750 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: *** Caught signal (Aborted) ** 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: in thread 7f5cf4200700 thread_name:ms_dispatch 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cfe7d1d20] 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: gsignal() 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: abort() 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5cffa49437] 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.751 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 16: clone() 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.548+0000 7f5cf4200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: in thread 7f5cf4200700 thread_name:ms_dispatch 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cfe7d1d20] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: gsignal() 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: abort() 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5cffa49437] 2024-04-26T11:45:26.752 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.753 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.754 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 16: clone() 2024-04-26T11:45:26.754 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: -1> 2024-04-26T11:45:26.545+0000 7f5cf4200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5cf4200700 time 2024-04-26T11:45:26.544149+0000 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5cffa493dd] 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.755 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: clone() 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 0> 2024-04-26T11:45:26.548+0000 7f5cf4200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: in thread 7f5cf4200700 thread_name:ms_dispatch 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cfe7d1d20] 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: gsignal() 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: abort() 2024-04-26T11:45:26.756 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5cffa49437] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 16: clone() 2024-04-26T11:45:26.757 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: -1> 2024-04-26T11:45:26.545+0000 7f5cf4200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5cf4200700 time 2024-04-26T11:45:26.544149+0000 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5cffa493dd] 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.758 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.759 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: clone() 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 0> 2024-04-26T11:45:26.548+0000 7f5cf4200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: in thread 7f5cf4200700 thread_name:ms_dispatch 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cfe7d1d20] 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2: gsignal() 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 3: abort() 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5cffa49437] 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5cffa495a3] 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563f0f0ae822] 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563f0f01e598] 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563f0f020c17] 2024-04-26T11:45:26.760 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563f0ef19d0b] 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563f0ef1a2ac] 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563f0ef02fad] 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5cffcdde20] 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 13: (DispatchQueue::entry()+0x51f) [0x7f5cffcdafdf] 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cffdadfc1] 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5cfe7c71ca] 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 16: clone() 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93169]: 2024-04-26T11:45:26.761 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:26 smithi033 podman[93362]: 2024-04-26 11:45:26.743645052 +0000 UTC m=+0.056459905 container died e1459739bc7bb2c8c76b6958ee3ada2c0fbf714709f120bbd1790b07174979c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:45:27.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 850.0 s, 157.7 tps, lat 2365.780 ms stddev 6434.517 2024-04-26T11:45:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:27 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:clientreplay 2024-04-26T11:45:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:27 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:27 smithi194 ceph-mon[33017]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:45:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:27 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:27 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:active 2024-04-26T11:45:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:27 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:45:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:27 smithi194 ceph-mon[33017]: pgmap v507: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:45:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:27 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:clientreplay 2024-04-26T11:45:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:27 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:27 smithi033 ceph-mon[27845]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:45:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:27 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:27 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:active 2024-04-26T11:45:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:27 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:45:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:27 smithi033 ceph-mon[27845]: pgmap v507: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:45:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:clientreplay 2024-04-26T11:45:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-mon[32753]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:45:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1954472819,v1:172.21.15.33:6837/1954472819] up:active 2024-04-26T11:45:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:45:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-mon[32753]: pgmap v507: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:45:27.926 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:27 smithi033 podman[93362]: 2024-04-26 11:45:27.526263604 +0000 UTC m=+0.839078475 container remove e1459739bc7bb2c8c76b6958ee3ada2c0fbf714709f120bbd1790b07174979c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:45:27.926 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:45:27.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:45:27 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:45:27.480+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:45:28.426 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-26T11:45:28.426 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.330s CPU time. 2024-04-26T11:45:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:29 smithi033 ceph-mon[27845]: pgmap v508: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:45:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:29 smithi194 ceph-mon[33017]: pgmap v508: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:45:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:29 smithi096 ceph-mon[32753]: pgmap v508: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 MiB/s wr, 190 op/s 2024-04-26T11:45:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:31 smithi033 ceph-mon[27845]: pgmap v509: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-26T11:45:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:31 smithi194 ceph-mon[33017]: pgmap v509: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-26T11:45:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:31 smithi096 ceph-mon[32753]: pgmap v509: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-26T11:45:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 855.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:32.305+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:32.305+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:32.305+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:45:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:33 smithi194 ceph-mon[33017]: osdmap e140: 12 total, 12 up, 12 in 2024-04-26T11:45:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:33 smithi194 ceph-mon[33017]: pgmap v511: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-26T11:45:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:33 smithi033 ceph-mon[27845]: osdmap e140: 12 total, 12 up, 12 in 2024-04-26T11:45:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:33 smithi033 ceph-mon[27845]: pgmap v511: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-26T11:45:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:33 smithi096 ceph-mon[32753]: osdmap e140: 12 total, 12 up, 12 in 2024-04-26T11:45:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:33 smithi096 ceph-mon[32753]: pgmap v511: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-26T11:45:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:35 smithi033 ceph-mon[27845]: pgmap v512: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:35 smithi194 ceph-mon[33017]: pgmap v512: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:35 smithi096 ceph-mon[32753]: pgmap v512: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:37.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 860.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:37 smithi194 ceph-mon[33017]: pgmap v513: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:37 smithi033 ceph-mon[27845]: pgmap v513: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:37 smithi096 ceph-mon[32753]: pgmap v513: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:38.583 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:45:38.584 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 systemd[1]: Stopped Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:45:38.584 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.330s CPU time. 2024-04-26T11:45:38.584 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 systemd[1]: Starting Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:45:38.915 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 podman[93454]: 2024-04-26 11:45:38.487991701 +0000 UTC m=+0.034634304 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:45:38.915 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 podman[93454]: 2024-04-26 11:45:38.60222816 +0000 UTC m=+0.148870790 container create 620d0bb41eeab96461157d7a0ce8052f95005da258dee69d6864d7a07f5c15b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:45:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: pgmap v514: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: daemon mds.d restarted 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: osdmap e141: 12 total, 12 up, 12 in 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:boot 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:39 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 podman[93454]: 2024-04-26 11:45:38.914281694 +0000 UTC m=+0.460924367 container init 620d0bb41eeab96461157d7a0ce8052f95005da258dee69d6864d7a07f5c15b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 podman[93454]: 2024-04-26 11:45:38.926262753 +0000 UTC m=+0.472905354 container start 620d0bb41eeab96461157d7a0ce8052f95005da258dee69d6864d7a07f5c15b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:45:39.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 ceph-mds[93473]: -- 172.21.15.33:0/4133720620 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584b6baba40 con 0x5584b6b7e800 2024-04-26T11:45:39.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:38 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: starting mds.d at 2024-04-26T11:45:39.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:39 smithi033 bash[93454]: 620d0bb41eeab96461157d7a0ce8052f95005da258dee69d6864d7a07f5c15b8 2024-04-26T11:45:39.178 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:45:39 smithi033 systemd[1]: Started Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: pgmap v514: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: daemon mds.d restarted 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: osdmap e141: 12 total, 12 up, 12 in 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:boot 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:45:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:39 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: pgmap v514: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: daemon mds.d restarted 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: osdmap e141: 12 total, 12 up, 12 in 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:boot 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:45:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:39 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:41.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:41 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:resolve 2024-04-26T11:45:41.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:41 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:41.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:41 smithi033 ceph-mon[27845]: pgmap v516: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 643 KiB/s wr, 87 op/s 2024-04-26T11:45:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:41 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:resolve 2024-04-26T11:45:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:41 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:41 smithi194 ceph-mon[33017]: pgmap v516: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 643 KiB/s wr, 87 op/s 2024-04-26T11:45:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:41 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:resolve 2024-04-26T11:45:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:41 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:41 smithi096 ceph-mon[32753]: pgmap v516: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 643 KiB/s wr, 87 op/s 2024-04-26T11:45:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 865.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:42 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:45:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:42 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:reconnect 2024-04-26T11:45:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:42 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:42 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:45:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:42 smithi194 ceph-mon[33017]: osdmap e142: 12 total, 12 up, 12 in 2024-04-26T11:45:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:42 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:45:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:42 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:reconnect 2024-04-26T11:45:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:42 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:42 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:45:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:42 smithi033 ceph-mon[27845]: osdmap e142: 12 total, 12 up, 12 in 2024-04-26T11:45:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:42 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:45:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:42 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:reconnect 2024-04-26T11:45:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:42 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:42 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:45:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:42 smithi096 ceph-mon[32753]: osdmap e142: 12 total, 12 up, 12 in 2024-04-26T11:45:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:43 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:rejoin 2024-04-26T11:45:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:43 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:43 smithi194 ceph-mon[33017]: pgmap v518: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 288 KiB/s wr, 40 op/s 2024-04-26T11:45:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:43 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:rejoin 2024-04-26T11:45:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:43 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:43 smithi033 ceph-mon[27845]: pgmap v518: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 288 KiB/s wr, 40 op/s 2024-04-26T11:45:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:43 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:rejoin 2024-04-26T11:45:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:43 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:43 smithi096 ceph-mon[32753]: pgmap v518: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 288 KiB/s wr, 40 op/s 2024-04-26T11:45:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:44 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:clientreplay 2024-04-26T11:45:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:44 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:44 smithi194 ceph-mon[33017]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:45:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:clientreplay 2024-04-26T11:45:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-mon[27845]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:45:44.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1a11e00700 time 2024-04-26T11:45:44.092839+0000 2024-04-26T11:45:44.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:44.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1a1d61f3dd] 2024-04-26T11:45:44.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: clone() 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: *** Caught signal (Aborted) ** 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: in thread 7f1a11e00700 thread_name:ms_dispatch 2024-04-26T11:45:44.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.093+0000 7f1a11e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1a11e00700 time 2024-04-26T11:45:44.092839+0000 2024-04-26T11:45:44.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:44.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1a1d61f3dd] 2024-04-26T11:45:44.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: clone() 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1a1c3a7d20] 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: gsignal() 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: abort() 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1a1d61f437] 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 16: clone() 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.095+0000 7f1a11e00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: in thread 7f1a11e00700 thread_name:ms_dispatch 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1a1c3a7d20] 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: gsignal() 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: abort() 2024-04-26T11:45:44.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1a1d61f437] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 16: clone() 2024-04-26T11:45:44.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:44.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:44 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:clientreplay 2024-04-26T11:45:44.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:44 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:44.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:44 smithi096 ceph-mon[32753]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:45:44.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: -1> 2024-04-26T11:45:44.093+0000 7f1a11e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1a11e00700 time 2024-04-26T11:45:44.092839+0000 2024-04-26T11:45:44.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:44.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1a1d61f3dd] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: clone() 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 0> 2024-04-26T11:45:44.095+0000 7f1a11e00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: in thread 7f1a11e00700 thread_name:ms_dispatch 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1a1c3a7d20] 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: gsignal() 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: abort() 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1a1d61f437] 2024-04-26T11:45:44.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 16: clone() 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:44.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: -1> 2024-04-26T11:45:44.093+0000 7f1a11e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1a11e00700 time 2024-04-26T11:45:44.092839+0000 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1a1d61f3dd] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: clone() 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 0> 2024-04-26T11:45:44.095+0000 7f1a11e00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: in thread 7f1a11e00700 thread_name:ms_dispatch 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1a1c3a7d20] 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2: gsignal() 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 3: abort() 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1a1d61f437] 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1a1d61f5a3] 2024-04-26T11:45:44.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563a67582822] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563a674f2598] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563a674f4c17] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563a673edd0b] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563a673ee2ac] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563a673d6fad] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1a1d8b3e20] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 13: (DispatchQueue::entry()+0x51f) [0x7f1a1d8b0fdf] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1a1d983fc1] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1a1c39d1ca] 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 16: clone() 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[69063]: 2024-04-26T11:45:44.442 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 podman[93507]: 2024-04-26 11:45:44.289794131 +0000 UTC m=+0.067015293 container died 72590b12b61c3be371a043d086898a0d923cdbb6d1823f6c75260a431ec6ee0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T11:45:45.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:45 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:45.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:45 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:active 2024-04-26T11:45:45.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:45 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:45.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:45 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:45:45.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:45 smithi096 ceph-mon[32753]: pgmap v519: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 419 KiB/s wr, 47 op/s 2024-04-26T11:45:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:45 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:45 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:active 2024-04-26T11:45:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:45 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:45 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:45:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:45 smithi194 ceph-mon[33017]: pgmap v519: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 419 KiB/s wr, 47 op/s 2024-04-26T11:45:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:45 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:45:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:45 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/3408204812,v1:172.21.15.33:6841/3408204812] up:active 2024-04-26T11:45:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:45 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:45 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:45:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:45 smithi033 ceph-mon[27845]: pgmap v519: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 419 KiB/s wr, 47 op/s 2024-04-26T11:45:45.426 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 podman[93507]: 2024-04-26 11:45:44.970231397 +0000 UTC m=+0.747452547 container remove 72590b12b61c3be371a043d086898a0d923cdbb6d1823f6c75260a431ec6ee0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T11:45:45.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:44 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:45:45.925 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:45 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-26T11:45:45.926 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:45 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.871s CPU time. 2024-04-26T11:45:47.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:47 smithi033 ceph-mon[27845]: pgmap v520: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 516 KiB/s wr, 60 op/s 2024-04-26T11:45:47.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 870.0 s, 1531.6 tps, lat 287.523 ms stddev 2260.202 2024-04-26T11:45:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:47.307+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:47.307+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:45:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:45:47.307+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:45:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:47 smithi194 ceph-mon[33017]: pgmap v520: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 516 KiB/s wr, 60 op/s 2024-04-26T11:45:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:47 smithi096 ceph-mon[32753]: pgmap v520: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 516 KiB/s wr, 60 op/s 2024-04-26T11:45:47.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:45:47 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:45:47.482+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:45:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:49 smithi033 ceph-mon[27845]: pgmap v521: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 414 KiB/s wr, 48 op/s 2024-04-26T11:45:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:49 smithi194 ceph-mon[33017]: pgmap v521: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 414 KiB/s wr, 48 op/s 2024-04-26T11:45:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:49 smithi096 ceph-mon[32753]: pgmap v521: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 414 KiB/s wr, 48 op/s 2024-04-26T11:45:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:51 smithi194 ceph-mon[33017]: pgmap v522: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 3.7 MiB/s wr, 297 op/s 2024-04-26T11:45:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:51 smithi033 ceph-mon[27845]: pgmap v522: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 3.7 MiB/s wr, 297 op/s 2024-04-26T11:45:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:51 smithi096 ceph-mon[32753]: pgmap v522: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 3.7 MiB/s wr, 297 op/s 2024-04-26T11:45:52.186 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 875.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:53 smithi033 ceph-mon[27845]: pgmap v523: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 3.2 MiB/s wr, 259 op/s 2024-04-26T11:45:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:53 smithi194 ceph-mon[33017]: pgmap v523: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 3.2 MiB/s wr, 259 op/s 2024-04-26T11:45:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:53 smithi096 ceph-mon[32753]: pgmap v523: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 3.2 MiB/s wr, 259 op/s 2024-04-26T11:45:56.016 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:55 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:45:56.016 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:55 smithi033 systemd[1]: Stopped Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:45:56.016 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:55 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.871s CPU time. 2024-04-26T11:45:56.016 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:55 smithi033 systemd[1]: Starting Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:45:56.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:56 smithi033 ceph-mon[27845]: pgmap v524: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 3.6 MiB/s wr, 319 op/s 2024-04-26T11:45:56.268 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 podman[93599]: 2024-04-26 11:45:55.983988025 +0000 UTC m=+0.036199307 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:45:56.268 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 podman[93599]: 2024-04-26 11:45:56.104572615 +0000 UTC m=+0.156783934 container create aa7c1c86438d58ff70f15d8a40ad0b850c556feca504fbd6cbf35872b133bab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T11:45:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:56 smithi194 ceph-mon[33017]: pgmap v524: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 3.6 MiB/s wr, 319 op/s 2024-04-26T11:45:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:56 smithi096 ceph-mon[32753]: pgmap v524: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 3.6 MiB/s wr, 319 op/s 2024-04-26T11:45:56.676 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 podman[93599]: 2024-04-26 11:45:56.367693179 +0000 UTC m=+0.419904504 container init aa7c1c86438d58ff70f15d8a40ad0b850c556feca504fbd6cbf35872b133bab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T11:45:56.677 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 podman[93599]: 2024-04-26 11:45:56.379066684 +0000 UTC m=+0.431278036 container start aa7c1c86438d58ff70f15d8a40ad0b850c556feca504fbd6cbf35872b133bab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T11:45:56.677 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 ceph-mds[93618]: -- 172.21.15.33:0/4027021389 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a9bef13a40 con 0x55a9beee7000 2024-04-26T11:45:56.677 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: starting mds.a at 2024-04-26T11:45:56.677 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 bash[93599]: aa7c1c86438d58ff70f15d8a40ad0b850c556feca504fbd6cbf35872b133bab8 2024-04-26T11:45:56.677 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:45:56 smithi033 systemd[1]: Started Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:45:57.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 880.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: daemon mds.a restarted 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: osdmap e143: 12 total, 12 up, 12 in 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:boot 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:45:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:57 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: daemon mds.a restarted 2024-04-26T11:45:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: osdmap e143: 12 total, 12 up, 12 in 2024-04-26T11:45:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:boot 2024-04-26T11:45:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:45:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:45:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:45:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:57 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: daemon mds.a restarted 2024-04-26T11:45:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:45:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: osdmap e143: 12 total, 12 up, 12 in 2024-04-26T11:45:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:boot 2024-04-26T11:45:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:45:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:45:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:45:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:57 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:58.726 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:58 smithi033 ceph-mon[27845]: pgmap v526: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 366 op/s 2024-04-26T11:45:58.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:resolve 2024-04-26T11:45:58.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:58 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:58 smithi194 ceph-mon[33017]: pgmap v526: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 366 op/s 2024-04-26T11:45:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:resolve 2024-04-26T11:45:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:58 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:58 smithi096 ceph-mon[32753]: pgmap v526: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 366 op/s 2024-04-26T11:45:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:resolve 2024-04-26T11:45:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:58 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:59 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:45:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:reconnect 2024-04-26T11:45:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:59 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:45:59 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:45:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:59 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:45:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:reconnect 2024-04-26T11:45:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:59 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:45:59 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:45:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:59 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:45:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:reconnect 2024-04-26T11:45:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:59 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:45:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:45:59 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:46:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:00 smithi194 ceph-mon[33017]: pgmap v527: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 366 op/s 2024-04-26T11:46:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:00 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:rejoin 2024-04-26T11:46:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:00 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:00 smithi194 ceph-mon[33017]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:46:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-mon[27845]: pgmap v527: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 366 op/s 2024-04-26T11:46:00.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:rejoin 2024-04-26T11:46:00.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:00.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-mon[27845]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:46:00.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4312600700 time 2024-04-26T11:46:00.535264+0000 2024-04-26T11:46:00.929 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f431dfb83dd] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: clone() 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.536+0000 7f4312600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4312600700 time 2024-04-26T11:46:00.535264+0000 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f431dfb83dd] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.931 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: clone() 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: *** Caught signal (Aborted) ** 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: in thread 7f4312600700 thread_name:ms_dispatch 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f431cd40d20] 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: gsignal() 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: abort() 2024-04-26T11:46:00.932 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f431dfb8437] 2024-04-26T11:46:00.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.933 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:00 smithi096 ceph-mon[32753]: pgmap v527: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 366 op/s 2024-04-26T11:46:00.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:00 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:rejoin 2024-04-26T11:46:00.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:00 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:00.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:00 smithi096 ceph-mon[32753]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:46:00.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.934 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 16: clone() 2024-04-26T11:46:00.935 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.539+0000 7f4312600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:00.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: in thread 7f4312600700 thread_name:ms_dispatch 2024-04-26T11:46:00.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f431cd40d20] 2024-04-26T11:46:00.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: gsignal() 2024-04-26T11:46:00.936 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: abort() 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f431dfb8437] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 16: clone() 2024-04-26T11:46:00.937 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: -20> 2024-04-26T11:46:00.536+0000 7f4312600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4312600700 time 2024-04-26T11:46:00.535264+0000 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f431dfb83dd] 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.938 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.939 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: clone() 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 0> 2024-04-26T11:46:00.539+0000 7f4312600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: in thread 7f4312600700 thread_name:ms_dispatch 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f431cd40d20] 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: gsignal() 2024-04-26T11:46:00.940 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: abort() 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f431dfb8437] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 16: clone() 2024-04-26T11:46:00.941 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: -21> 2024-04-26T11:46:00.536+0000 7f4312600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4312600700 time 2024-04-26T11:46:00.535264+0000 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f431dfb83dd] 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.942 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: clone() 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: -1> 2024-04-26T11:46:00.539+0000 7f4312600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: in thread 7f4312600700 thread_name:ms_dispatch 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f431cd40d20] 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2: gsignal() 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 3: abort() 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f431dfb8437] 2024-04-26T11:46:00.943 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f431dfb85a3] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55849a188822] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55849a0f8598] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55849a0fac17] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558499ff3d0b] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558499ff42ac] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558499fdcfad] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f431e24ce20] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 13: (DispatchQueue::entry()+0x51f) [0x7f431e249fdf] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f431e31cfc1] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f431cd361ca] 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 16: clone() 2024-04-26T11:46:00.944 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:00.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93469]: 2024-04-26T11:46:00.945 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:00 smithi033 podman[93653]: 2024-04-26 11:46:00.735968139 +0000 UTC m=+0.076148953 container died 620d0bb41eeab96461157d7a0ce8052f95005da258dee69d6864d7a07f5c15b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T11:46:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:01 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:01 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:active 2024-04-26T11:46:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:01 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:01 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:46:01.928 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:01 smithi033 podman[93653]: 2024-04-26 11:46:01.589137208 +0000 UTC m=+0.929318006 container remove 620d0bb41eeab96461157d7a0ce8052f95005da258dee69d6864d7a07f5c15b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T11:46:01.962 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:01 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:46:01.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:01 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:01.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:01 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:active 2024-04-26T11:46:01.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:01 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:01.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:01 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:01.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:46:01.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:01 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:01.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:01 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/757670101,v1:172.21.15.33:6837/757670101] up:active 2024-04-26T11:46:01.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:01 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:01.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:01 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:01.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:46:02.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 885.0 s, 296.0 tps, lat 935.338 ms stddev 3385.526 2024-04-26T11:46:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:02.322+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:02.322+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:02.322+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:46:02.425 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:02 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-26T11:46:02.426 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:02 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.220s CPU time. 2024-04-26T11:46:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:02 smithi194 ceph-mon[33017]: pgmap v528: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-26T11:46:02.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:02 smithi033 ceph-mon[27845]: pgmap v528: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-26T11:46:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:02 smithi096 ceph-mon[32753]: pgmap v528: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-26T11:46:02.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:46:02 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:46:02.482+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:46:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:46:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:46:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:03 smithi194 ceph-mon[33017]: pgmap v529: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T11:46:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:03 smithi194 ceph-mon[33017]: pgmap v530: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 672 KiB/s wr, 89 op/s 2024-04-26T11:46:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:46:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:46:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:46:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:03 smithi033 ceph-mon[27845]: pgmap v529: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T11:46:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:03 smithi033 ceph-mon[27845]: pgmap v530: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 672 KiB/s wr, 89 op/s 2024-04-26T11:46:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:46:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:46:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:46:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:03 smithi096 ceph-mon[32753]: pgmap v529: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T11:46:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:03 smithi096 ceph-mon[32753]: pgmap v530: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 672 KiB/s wr, 89 op/s 2024-04-26T11:46:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:46:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:05 smithi033 ceph-mon[27845]: pgmap v531: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-26T11:46:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:05 smithi096 ceph-mon[32753]: pgmap v531: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-26T11:46:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:05 smithi194 ceph-mon[33017]: pgmap v531: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-26T11:46:07.184 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 890.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:46:07.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:07 smithi033 ceph-mon[27845]: pgmap v532: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T11:46:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:07 smithi096 ceph-mon[32753]: pgmap v532: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T11:46:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:07 smithi194 ceph-mon[33017]: pgmap v532: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T11:46:09.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:09 smithi194 ceph-mon[33017]: pgmap v533: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T11:46:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:09 smithi033 ceph-mon[27845]: pgmap v533: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T11:46:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:09 smithi096 ceph-mon[32753]: pgmap v533: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T11:46:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:11 smithi033 ceph-mon[27845]: pgmap v534: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-04-26T11:46:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:11 smithi096 ceph-mon[32753]: pgmap v534: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-04-26T11:46:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:11 smithi194 ceph-mon[33017]: pgmap v534: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-04-26T11:46:12.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 895.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:46:12.192 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:46:12.588 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 systemd[1]: Stopped Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:46:12.588 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.220s CPU time. 2024-04-26T11:46:12.588 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 systemd[1]: Starting Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:46:12.889 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 podman[93884]: 2024-04-26 11:46:12.49482675 +0000 UTC m=+0.037865189 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:46:12.889 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 podman[93884]: 2024-04-26 11:46:12.60699313 +0000 UTC m=+0.150031596 container create 976d5eaf459242638af8a0939bc423aa969cf38e5f9b8756840dfe02d56b8a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T11:46:12.889 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 podman[93884]: 2024-04-26 11:46:12.876989923 +0000 UTC m=+0.420028340 container init 976d5eaf459242638af8a0939bc423aa969cf38e5f9b8756840dfe02d56b8a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T11:46:13.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 podman[93884]: 2024-04-26 11:46:12.912137478 +0000 UTC m=+0.455175886 container start 976d5eaf459242638af8a0939bc423aa969cf38e5f9b8756840dfe02d56b8a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T11:46:13.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 ceph-mds[93902]: -- 172.21.15.33:0/510851757 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e258a7a40 con 0x563e2587ac00 2024-04-26T11:46:13.176 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:12 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: starting mds.d at 2024-04-26T11:46:13.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:13 smithi033 bash[93884]: 976d5eaf459242638af8a0939bc423aa969cf38e5f9b8756840dfe02d56b8a9d 2024-04-26T11:46:13.177 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:13 smithi033 systemd[1]: Started Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:46:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: pgmap v535: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: daemon mds.d restarted 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: osdmap e144: 12 total, 12 up, 12 in 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:boot 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:46:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:13 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: pgmap v535: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: daemon mds.d restarted 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: osdmap e144: 12 total, 12 up, 12 in 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:boot 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:13 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: pgmap v535: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T11:46:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: daemon mds.d restarted 2024-04-26T11:46:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T11:46:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: osdmap e144: 12 total, 12 up, 12 in 2024-04-26T11:46:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:boot 2024-04-26T11:46:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:46:14.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:46:14.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:13 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:15.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:14 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:resolve 2024-04-26T11:46:15.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:14 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:15.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:14 smithi033 ceph-mon[27845]: pgmap v537: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 914 KiB/s wr, 120 op/s 2024-04-26T11:46:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:14 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:resolve 2024-04-26T11:46:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:14 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:14 smithi194 ceph-mon[33017]: pgmap v537: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 914 KiB/s wr, 120 op/s 2024-04-26T11:46:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:14 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:resolve 2024-04-26T11:46:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:14 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:14 smithi096 ceph-mon[32753]: pgmap v537: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 914 KiB/s wr, 120 op/s 2024-04-26T11:46:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:16 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:46:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:16 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:reconnect 2024-04-26T11:46:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:16 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:16 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:46:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:46:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:reconnect 2024-04-26T11:46:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:46:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:16 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:46:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:16 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:reconnect 2024-04-26T11:46:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:16 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:16 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:46:16.830 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f948f400700 time 2024-04-26T11:46:16.611792+0000 2024-04-26T11:46:16.830 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:16.830 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f949acbb3dd] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.831 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: clone() 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: *** Caught signal (Aborted) ** 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: in thread 7f948f400700 thread_name:ms_dispatch 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.613+0000 7f948f400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f948f400700 time 2024-04-26T11:46:16.611792+0000 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f949acbb3dd] 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.832 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: clone() 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9499a43d20] 2024-04-26T11:46:16.833 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: gsignal() 2024-04-26T11:46:16.834 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: abort() 2024-04-26T11:46:16.834 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f949acbb437] 2024-04-26T11:46:16.834 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.834 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.834 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.834 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.834 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.835 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.835 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.835 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.835 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.835 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.835 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 16: clone() 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.617+0000 7f948f400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: in thread 7f948f400700 thread_name:ms_dispatch 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9499a43d20] 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: gsignal() 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: abort() 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f949acbb437] 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.836 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 16: clone() 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: -1> 2024-04-26T11:46:16.613+0000 7f948f400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f948f400700 time 2024-04-26T11:46:16.611792+0000 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:16.837 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f949acbb3dd] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.838 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: clone() 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 0> 2024-04-26T11:46:16.617+0000 7f948f400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: in thread 7f948f400700 thread_name:ms_dispatch 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9499a43d20] 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: gsignal() 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: abort() 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f949acbb437] 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.839 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 16: clone() 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.840 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: -1> 2024-04-26T11:46:16.613+0000 7f948f400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f948f400700 time 2024-04-26T11:46:16.611792+0000 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f949acbb3dd] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.841 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: clone() 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 0> 2024-04-26T11:46:16.617+0000 7f948f400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: in thread 7f948f400700 thread_name:ms_dispatch 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9499a43d20] 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2: gsignal() 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 3: abort() 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f949acbb437] 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f949acbb5a3] 2024-04-26T11:46:16.842 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a98115f822] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a9810cf598] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a9810d1c17] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a980fcad0b] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a980fcb2ac] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a980fb3fad] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f949af4fe20] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 13: (DispatchQueue::entry()+0x51f) [0x7f949af4cfdf] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f949b01ffc1] 2024-04-26T11:46:16.843 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9499a391ca] 2024-04-26T11:46:16.844 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 16: clone() 2024-04-26T11:46:16.844 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:16.844 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[93614]: 2024-04-26T11:46:16.844 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:16 smithi033 podman[93938]: 2024-04-26 11:46:16.828470276 +0000 UTC m=+0.063878484 container died aa7c1c86438d58ff70f15d8a40ad0b850c556feca504fbd6cbf35872b133bab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:46:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:17 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:rejoin 2024-04-26T11:46:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:17 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:17 smithi033 ceph-mon[27845]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:46:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:17 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:17 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:active 2024-04-26T11:46:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:17 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:17 smithi033 ceph-mon[27845]: pgmap v538: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 845 KiB/s wr, 110 op/s 2024-04-26T11:46:17.183 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 900.0 s, 204.8 tps, lat 1572.865 ms stddev 4696.824 2024-04-26T11:46:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:17.337+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:17.337+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:17.337+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:46:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:17 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:rejoin 2024-04-26T11:46:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:17 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:17 smithi194 ceph-mon[33017]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:46:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:17 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:17 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:active 2024-04-26T11:46:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:17 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:17 smithi194 ceph-mon[33017]: pgmap v538: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 845 KiB/s wr, 110 op/s 2024-04-26T11:46:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:rejoin 2024-04-26T11:46:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-mon[32753]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:46:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/116761636,v1:172.21.15.33:6841/116761636] up:active 2024-04-26T11:46:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-mon[32753]: pgmap v538: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 845 KiB/s wr, 110 op/s 2024-04-26T11:46:17.926 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:17 smithi033 podman[93938]: 2024-04-26 11:46:17.632914851 +0000 UTC m=+0.868323074 container remove aa7c1c86438d58ff70f15d8a40ad0b850c556feca504fbd6cbf35872b133bab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True) 2024-04-26T11:46:17.926 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:17 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:46:17.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:46:17 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:46:17.483+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:46:18.426 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-26T11:46:18.426 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.168s CPU time. 2024-04-26T11:46:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:19 smithi033 ceph-mon[27845]: pgmap v539: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 845 KiB/s wr, 110 op/s 2024-04-26T11:46:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:19 smithi096 ceph-mon[32753]: pgmap v539: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 845 KiB/s wr, 110 op/s 2024-04-26T11:46:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:19 smithi194 ceph-mon[33017]: pgmap v539: 97 pgs: 97 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 845 KiB/s wr, 110 op/s 2024-04-26T11:46:21.880 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:21 smithi033 ceph-mon[27845]: pgmap v540: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-04-26T11:46:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:21 smithi096 ceph-mon[32753]: pgmap v540: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-04-26T11:46:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:21 smithi194 ceph-mon[33017]: pgmap v540: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-04-26T11:46:22.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 905.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:46:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:23 smithi033 ceph-mon[27845]: pgmap v541: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 2.9 MiB/s wr, 360 op/s 2024-04-26T11:46:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:23 smithi096 ceph-mon[32753]: pgmap v541: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 2.9 MiB/s wr, 360 op/s 2024-04-26T11:46:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:23 smithi194 ceph-mon[33017]: pgmap v541: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 2.9 MiB/s wr, 360 op/s 2024-04-26T11:46:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:25 smithi033 ceph-mon[27845]: pgmap v542: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.6 MiB/s wr, 317 op/s 2024-04-26T11:46:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:25 smithi096 ceph-mon[32753]: pgmap v542: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.6 MiB/s wr, 317 op/s 2024-04-26T11:46:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:25 smithi194 ceph-mon[33017]: pgmap v542: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.6 MiB/s wr, 317 op/s 2024-04-26T11:46:27.188 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 910.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:46:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:27 smithi033 ceph-mon[27845]: pgmap v543: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.3 MiB/s wr, 285 op/s 2024-04-26T11:46:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:27 smithi096 ceph-mon[32753]: pgmap v543: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.3 MiB/s wr, 285 op/s 2024-04-26T11:46:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:27 smithi194 ceph-mon[33017]: pgmap v543: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.3 MiB/s wr, 285 op/s 2024-04-26T11:46:28.597 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:46:28.598 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 systemd[1]: Stopped Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:46:28.598 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.168s CPU time. 2024-04-26T11:46:28.598 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 systemd[1]: Starting Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:46:28.914 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 podman[94027]: 2024-04-26 11:46:28.505612394 +0000 UTC m=+0.037992299 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:46:28.914 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 podman[94027]: 2024-04-26 11:46:28.62595948 +0000 UTC m=+0.158339452 container create 6145e2f11822d717bc6334b7dccd54c66afec6d51cd30c29b4d593afe3af1644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T11:46:29.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 podman[94027]: 2024-04-26 11:46:28.912871304 +0000 UTC m=+0.445251237 container init 6145e2f11822d717bc6334b7dccd54c66afec6d51cd30c29b4d593afe3af1644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T11:46:29.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 podman[94027]: 2024-04-26 11:46:28.924241218 +0000 UTC m=+0.456621123 container start 6145e2f11822d717bc6334b7dccd54c66afec6d51cd30c29b4d593afe3af1644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T11:46:29.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 ceph-mds[94045]: -- 172.21.15.33:0/2316753215 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f05e5a7a40 con 0x55f05e57ac00 2024-04-26T11:46:29.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:28 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: starting mds.a at 2024-04-26T11:46:29.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:29 smithi033 bash[94027]: 6145e2f11822d717bc6334b7dccd54c66afec6d51cd30c29b4d593afe3af1644 2024-04-26T11:46:29.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:29 smithi033 systemd[1]: Started Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: pgmap v544: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: daemon mds.a restarted 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: osdmap e145: 12 total, 12 up, 12 in 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:boot 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:46:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:29 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: pgmap v544: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: daemon mds.a restarted 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: osdmap e145: 12 total, 12 up, 12 in 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:boot 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:29 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: pgmap v544: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-04-26T11:46:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: daemon mds.a restarted 2024-04-26T11:46:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:46:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: osdmap e145: 12 total, 12 up, 12 in 2024-04-26T11:46:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:boot 2024-04-26T11:46:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-04-26T11:46:30.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-26T11:46:30.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:29 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:31.919 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:31 smithi033 ceph-mon[27845]: pgmap v546: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 353 KiB/s wr, 46 op/s 2024-04-26T11:46:31.919 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:resolve 2024-04-26T11:46:31.919 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:31 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:31 smithi096 ceph-mon[32753]: pgmap v546: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 353 KiB/s wr, 46 op/s 2024-04-26T11:46:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:resolve 2024-04-26T11:46:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:31 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:31 smithi194 ceph-mon[33017]: pgmap v546: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 353 KiB/s wr, 46 op/s 2024-04-26T11:46:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:resolve 2024-04-26T11:46:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:31 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:progress: 915.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T11:46:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:32.341+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:32.341+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:32.341+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:transaction type: 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:scaling factor: 32 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:query mode: simple 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:number of clients: 100 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:number of threads: 4 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:duration: 900 s 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:number of transactions actually processed: 773705 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:latency average = 118.297 ms 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:latency stddev = 1201.981 ms 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:tps = 844.963239 (including connections establishing) 2024-04-26T11:46:32.850 INFO:teuthology.orchestra.run.smithi033.stdout:tps = 844.969283 (excluding connections establishing) 2024-04-26T11:46:32.857 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-26T11:46:32.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:32 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:46:32.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:32 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:reconnect 2024-04-26T11:46:32.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:32 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:32.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:32 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:46:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:32 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:46:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:32 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:reconnect 2024-04-26T11:46:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:32 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:32 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:46:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:32 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:46:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:32 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:reconnect 2024-04-26T11:46:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:32 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:32 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:46:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:33 smithi194 ceph-mon[33017]: pgmap v547: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 63 KiB/s wr, 13 op/s 2024-04-26T11:46:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:33 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:rejoin 2024-04-26T11:46:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:33 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:33 smithi033 ceph-mon[27845]: pgmap v547: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 63 KiB/s wr, 13 op/s 2024-04-26T11:46:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:33 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:rejoin 2024-04-26T11:46:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:33 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:33 smithi096 ceph-mon[32753]: pgmap v547: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 63 KiB/s wr, 13 op/s 2024-04-26T11:46:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:33 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:rejoin 2024-04-26T11:46:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:33 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:clientreplay 2024-04-26T11:46:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:35.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-mon[27845]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:46:35.080 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4399400700 time 2024-04-26T11:46:34.730896+0000 2024-04-26T11:46:35.080 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:35.080 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.080 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a4c4a3dd] 2024-04-26T11:46:35.080 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: clone() 2024-04-26T11:46:35.081 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: *** Caught signal (Aborted) ** 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: in thread 7f4399400700 thread_name:ms_dispatch 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:34.730+0000 7f4399400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4399400700 time 2024-04-26T11:46:34.730896+0000 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a4c4a3dd] 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.082 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: clone() 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a39d2d20] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: gsignal() 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: abort() 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a4c4a437] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.083 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.084 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.084 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.084 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.084 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.084 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 16: clone() 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:34.733+0000 7f4399400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: in thread 7f4399400700 thread_name:ms_dispatch 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a39d2d20] 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: gsignal() 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: abort() 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a4c4a437] 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.085 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 16: clone() 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: -1> 2024-04-26T11:46:34.730+0000 7f4399400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4399400700 time 2024-04-26T11:46:34.730896+0000 2024-04-26T11:46:35.086 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a4c4a3dd] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.087 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: clone() 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 0> 2024-04-26T11:46:34.733+0000 7f4399400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: in thread 7f4399400700 thread_name:ms_dispatch 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a39d2d20] 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: gsignal() 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: abort() 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a4c4a437] 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.088 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 16: clone() 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: -1> 2024-04-26T11:46:34.730+0000 7f4399400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4399400700 time 2024-04-26T11:46:34.730896+0000 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.089 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.090 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a4c4a3dd] 2024-04-26T11:46:35.090 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.090 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.090 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.090 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: clone() 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 0> 2024-04-26T11:46:34.733+0000 7f4399400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: in thread 7f4399400700 thread_name:ms_dispatch 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.091 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a39d2d20] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2: gsignal() 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 3: abort() 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a4c4a437] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f43a4c4a5a3] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563e1021a822] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563e1018a598] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563e1018cc17] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563e10085d0b] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563e100862ac] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563e1006efad] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f43a4edee20] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a4edbfdf] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a4faefc1] 2024-04-26T11:46:35.092 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a39c81ca] 2024-04-26T11:46:35.093 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 16: clone() 2024-04-26T11:46:35.093 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:35.093 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d[93898]: 2024-04-26T11:46:35.093 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:34 smithi033 podman[94100]: 2024-04-26 11:46:34.853419802 +0000 UTC m=+0.034200548 container died 976d5eaf459242638af8a0939bc423aa969cf38e5f9b8756840dfe02d56b8a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-04-26T11:46:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:34 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:clientreplay 2024-04-26T11:46:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:34 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:34 smithi194 ceph-mon[33017]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:46:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:34 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:clientreplay 2024-04-26T11:46:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:34 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:clientreplay,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:34 smithi096 ceph-mon[32753]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-04-26T11:46:35.926 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:35 smithi033 podman[94100]: 2024-04-26 11:46:35.567924017 +0000 UTC m=+0.748704807 container remove 976d5eaf459242638af8a0939bc423aa969cf38e5f9b8756840dfe02d56b8a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T11:46:35.926 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:35 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:46:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:35 smithi033 ceph-mon[27845]: pgmap v548: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-04-26T11:46:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:35 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:35 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:active 2024-04-26T11:46:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:35 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:35 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:35 smithi194 ceph-mon[33017]: pgmap v548: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-04-26T11:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:35 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:35 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:active 2024-04-26T11:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:35 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:35 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:35 smithi096 ceph-mon[32753]: pgmap v548: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-04-26T11:46:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:35 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:35 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4001641037,v1:172.21.15.33:6837/4001641037] up:active 2024-04-26T11:46:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:35 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 2 up:standby-replay 3 up:standby 2024-04-26T11:46:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:35 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby 2024-04-26T11:46:36.676 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:36 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-26T11:46:36.676 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:36 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.471s CPU time. 2024-04-26T11:46:37.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:37 smithi033 ceph-mon[27845]: pgmap v549: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-04-26T11:46:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:37 smithi096 ceph-mon[32753]: pgmap v549: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-04-26T11:46:37.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:46:37 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:46:37.484+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:46:38.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:37 smithi194 ceph-mon[33017]: pgmap v549: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-04-26T11:46:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:39 smithi194 ceph-mon[33017]: pgmap v550: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-04-26T11:46:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:39 smithi033 ceph-mon[27845]: pgmap v550: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-04-26T11:46:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:39 smithi096 ceph-mon[32753]: pgmap v550: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 16 op/s 2024-04-26T11:46:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:41 smithi194 ceph-mon[33017]: pgmap v551: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 48 KiB/s wr, 17 op/s 2024-04-26T11:46:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:41 smithi033 ceph-mon[27845]: pgmap v551: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 48 KiB/s wr, 17 op/s 2024-04-26T11:46:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:41 smithi096 ceph-mon[32753]: pgmap v551: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 48 KiB/s wr, 17 op/s 2024-04-26T11:46:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:43 smithi194 ceph-mon[33017]: pgmap v552: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 53 KiB/s wr, 15 op/s 2024-04-26T11:46:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:43 smithi033 ceph-mon[27845]: pgmap v552: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 53 KiB/s wr, 15 op/s 2024-04-26T11:46:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:43 smithi096 ceph-mon[32753]: pgmap v552: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 53 KiB/s wr, 15 op/s 2024-04-26T11:46:45.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:45 smithi096 ceph-mon[32753]: pgmap v553: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 64 KiB/s wr, 18 op/s 2024-04-26T11:46:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:45 smithi194 ceph-mon[33017]: pgmap v553: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 64 KiB/s wr, 18 op/s 2024-04-26T11:46:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:45 smithi033 ceph-mon[27845]: pgmap v553: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 64 KiB/s wr, 18 op/s 2024-04-26T11:46:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:46 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-04-26T11:46:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:46 smithi033 systemd[1]: Stopped Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:46:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:46 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.471s CPU time. 2024-04-26T11:46:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:46 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-04-26T11:46:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:46 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-26T11:46:46.930 INFO:journalctl@ceph.mds.d.smithi033.stdout:Apr 26 11:46:46 smithi033 systemd[1]: Failed to start Ceph mds.d for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:46:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:47.352+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:47.352+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:46:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:46:47.352+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:46:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:47 smithi194 ceph-mon[33017]: pgmap v554: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 65 KiB/s wr, 13 op/s 2024-04-26T11:46:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:47 smithi033 ceph-mon[27845]: pgmap v554: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 65 KiB/s wr, 13 op/s 2024-04-26T11:46:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:47 smithi096 ceph-mon[32753]: pgmap v554: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 65 KiB/s wr, 13 op/s 2024-04-26T11:46:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:49 smithi194 ceph-mon[33017]: pgmap v555: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 37 KiB/s wr, 9 op/s 2024-04-26T11:46:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:49 smithi033 ceph-mon[27845]: pgmap v555: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 37 KiB/s wr, 9 op/s 2024-04-26T11:46:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:49 smithi096 ceph-mon[32753]: pgmap v555: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 37 KiB/s wr, 9 op/s 2024-04-26T11:46:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:51 smithi194 ceph-mon[33017]: pgmap v556: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 37 KiB/s wr, 11 op/s 2024-04-26T11:46:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:51 smithi194 ceph-mon[33017]: Replacing daemon mds.d as rank 1 with standby daemon mds.a 2024-04-26T11:46:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:51 smithi194 ceph-mon[33017]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-26T11:46:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:51 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:51 smithi194 ceph-mon[33017]: osdmap e146: 12 total, 12 up, 12 in 2024-04-26T11:46:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:51 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:51 smithi033 ceph-mon[27845]: pgmap v556: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 37 KiB/s wr, 11 op/s 2024-04-26T11:46:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:51 smithi033 ceph-mon[27845]: Replacing daemon mds.d as rank 1 with standby daemon mds.a 2024-04-26T11:46:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:51 smithi033 ceph-mon[27845]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-26T11:46:52.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:51 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:52.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:51 smithi033 ceph-mon[27845]: osdmap e146: 12 total, 12 up, 12 in 2024-04-26T11:46:52.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:51 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:51 smithi096 ceph-mon[32753]: pgmap v556: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 37 KiB/s wr, 11 op/s 2024-04-26T11:46:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:51 smithi096 ceph-mon[32753]: Replacing daemon mds.d as rank 1 with standby daemon mds.a 2024-04-26T11:46:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:51 smithi096 ceph-mon[32753]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-26T11:46:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:51 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:46:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:51 smithi096 ceph-mon[32753]: osdmap e146: 12 total, 12 up, 12 in 2024-04-26T11:46:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:51 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:54.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:53 smithi033 ceph-mon[27845]: pgmap v558: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-26T11:46:54.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:53 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:resolve 2024-04-26T11:46:54.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:53 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:53 smithi194 ceph-mon[33017]: pgmap v558: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-26T11:46:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:53 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:resolve 2024-04-26T11:46:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:53 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:53 smithi096 ceph-mon[32753]: pgmap v558: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-26T11:46:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:53 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:resolve 2024-04-26T11:46:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:53 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:55.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:54 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:reconnect 2024-04-26T11:46:55.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:54 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:55.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:54 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999984 2024-04-26T11:46:55.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:54 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0139998 2024-04-26T11:46:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:54 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:reconnect 2024-04-26T11:46:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:54 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:54 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999984 2024-04-26T11:46:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:54 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0139998 2024-04-26T11:46:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:54 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:reconnect 2024-04-26T11:46:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:54 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:54 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.000999984 2024-04-26T11:46:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:54 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0139998 2024-04-26T11:46:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:55 smithi194 ceph-mon[33017]: pgmap v559: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2024-04-26T11:46:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:55 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:rejoin 2024-04-26T11:46:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:55 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:55 smithi033 ceph-mon[27845]: pgmap v559: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2024-04-26T11:46:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:55 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:rejoin 2024-04-26T11:46:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:55 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:55 smithi096 ceph-mon[32753]: pgmap v559: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2024-04-26T11:46:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:55 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:rejoin 2024-04-26T11:46:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:55 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:56 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:clientreplay 2024-04-26T11:46:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:56 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:clientreplay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:56 smithi194 ceph-mon[33017]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:46:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:56 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:56 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:active 2024-04-26T11:46:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:56 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:46:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:56 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:clientreplay 2024-04-26T11:46:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:56 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:clientreplay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:56 smithi033 ceph-mon[27845]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:46:57.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:56 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:57.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:56 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:active 2024-04-26T11:46:57.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:56 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:46:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:56 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:clientreplay 2024-04-26T11:46:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:56 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:clientreplay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:46:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:56 smithi096 ceph-mon[32753]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:46:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:56 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:46:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:56 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6840/332420901,v1:172.21.15.33:6841/332420901] up:active 2024-04-26T11:46:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:56 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:46:57.791 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:46:57 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:46:57.485+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:46:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:57 smithi194 ceph-mon[33017]: pgmap v560: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-26T11:46:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:57 smithi033 ceph-mon[27845]: pgmap v560: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-26T11:46:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:57 smithi096 ceph-mon[32753]: pgmap v560: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-26T11:46:59.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd43c000700 time 2024-04-26T11:46:58.997494+0000 2024-04-26T11:46:59.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:59.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.428 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd4479be3dd] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: clone() 2024-04-26T11:46:59.429 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.001+0000 7fd43c000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd43c000700 time 2024-04-26T11:46:58.997494+0000 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd4479be3dd] 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.430 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: clone() 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: *** Caught signal (Aborted) ** 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: in thread 7fd43c000700 thread_name:ms_dispatch 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd446746d20] 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: gsignal() 2024-04-26T11:46:59.431 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: abort() 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd4479be437] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.432 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 16: clone() 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.006+0000 7fd43c000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: in thread 7fd43c000700 thread_name:ms_dispatch 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd446746d20] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: gsignal() 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: abort() 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd4479be437] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.433 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 16: clone() 2024-04-26T11:46:59.434 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: -1> 2024-04-26T11:46:59.001+0000 7fd43c000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd43c000700 time 2024-04-26T11:46:58.997494+0000 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd4479be3dd] 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.435 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: clone() 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 0> 2024-04-26T11:46:59.006+0000 7fd43c000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: in thread 7fd43c000700 thread_name:ms_dispatch 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.436 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd446746d20] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: gsignal() 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: abort() 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd4479be437] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.437 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 16: clone() 2024-04-26T11:46:59.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:59.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: -1> 2024-04-26T11:46:59.001+0000 7fd43c000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd43c000700 time 2024-04-26T11:46:58.997494+0000 2024-04-26T11:46:59.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:46:59.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.438 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd4479be3dd] 2024-04-26T11:46:59.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.439 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: clone() 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 0> 2024-04-26T11:46:59.006+0000 7fd43c000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: in thread 7fd43c000700 thread_name:ms_dispatch 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd446746d20] 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2: gsignal() 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 3: abort() 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd4479be437] 2024-04-26T11:46:59.440 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fd4479be5a3] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55f0280ba822] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55f02802a598] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f02802cc17] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f027f25d0b] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f027f262ac] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55f027f0efad] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fd447c52e20] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 13: (DispatchQueue::entry()+0x51f) [0x7fd447c4ffdf] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd447d22fc1] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd44673c1ca] 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 16: clone() 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94041]: 2024-04-26T11:46:59.441 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 podman[94171]: 2024-04-26 11:46:59.189332131 +0000 UTC m=+0.042634050 container died 6145e2f11822d717bc6334b7dccd54c66afec6d51cd30c29b4d593afe3af1644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True) 2024-04-26T11:47:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:46:59 smithi194 ceph-mon[33017]: pgmap v561: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-26T11:47:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:46:59 smithi033 ceph-mon[27845]: pgmap v561: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-26T11:47:00.176 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 podman[94171]: 2024-04-26 11:46:59.830795262 +0000 UTC m=+0.684097218 container remove 6145e2f11822d717bc6334b7dccd54c66afec6d51cd30c29b4d593afe3af1644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:47:00.176 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:46:59 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:47:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:46:59 smithi096 ceph-mon[32753]: pgmap v561: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-26T11:47:00.676 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:00 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-26T11:47:00.676 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:00 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.461s CPU time. 2024-04-26T11:47:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:01 smithi194 ceph-mon[33017]: pgmap v562: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 80 KiB/s wr, 20 op/s 2024-04-26T11:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:01 smithi033 ceph-mon[27845]: pgmap v562: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 80 KiB/s wr, 20 op/s 2024-04-26T11:47:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:01 smithi096 ceph-mon[32753]: pgmap v562: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 80 KiB/s wr, 20 op/s 2024-04-26T11:47:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:02.368+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:02.368+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:02.368+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:47:04.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:03 smithi194 ceph-mon[33017]: pgmap v563: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 72 KiB/s wr, 18 op/s 2024-04-26T11:47:04.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:47:04.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:03 smithi194 ceph-mon[33017]: osdmap e147: 12 total, 12 up, 12 in 2024-04-26T11:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:03 smithi033 ceph-mon[27845]: pgmap v563: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 72 KiB/s wr, 18 op/s 2024-04-26T11:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:03 smithi033 ceph-mon[27845]: osdmap e147: 12 total, 12 up, 12 in 2024-04-26T11:47:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:03 smithi096 ceph-mon[32753]: pgmap v563: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 72 KiB/s wr, 18 op/s 2024-04-26T11:47:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:47:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:03 smithi096 ceph-mon[32753]: osdmap e147: 12 total, 12 up, 12 in 2024-04-26T11:47:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:47:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:47:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:47:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:47:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:47:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:47:05.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:47:05.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:47:05.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:47:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:05 smithi194 ceph-mon[33017]: pgmap v565: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 82 KiB/s wr, 18 op/s 2024-04-26T11:47:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:05 smithi194 ceph-mon[33017]: pgmap v566: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 100 KiB/s wr, 22 op/s 2024-04-26T11:47:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:05 smithi033 ceph-mon[27845]: pgmap v565: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 82 KiB/s wr, 18 op/s 2024-04-26T11:47:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:05 smithi033 ceph-mon[27845]: pgmap v566: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 100 KiB/s wr, 22 op/s 2024-04-26T11:47:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:05 smithi096 ceph-mon[32753]: pgmap v565: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 82 KiB/s wr, 18 op/s 2024-04-26T11:47:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:05 smithi096 ceph-mon[32753]: pgmap v566: 97 pgs: 97 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 100 KiB/s wr, 22 op/s 2024-04-26T11:47:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:07 smithi194 ceph-mon[33017]: pgmap v567: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 101 KiB/s wr, 28 op/s 2024-04-26T11:47:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:07 smithi033 ceph-mon[27845]: pgmap v567: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 101 KiB/s wr, 28 op/s 2024-04-26T11:47:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:07 smithi096 ceph-mon[32753]: pgmap v567: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 101 KiB/s wr, 28 op/s 2024-04-26T11:47:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:09 smithi194 ceph-mon[33017]: pgmap v568: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T11:47:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:09 smithi033 ceph-mon[27845]: pgmap v568: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T11:47:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:09 smithi096 ceph-mon[32753]: pgmap v568: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T11:47:10.849 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:47:10.849 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:10 smithi033 systemd[1]: Stopped Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:47:10.850 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.461s CPU time. 2024-04-26T11:47:10.850 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:10 smithi033 systemd[1]: Starting Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:47:11.176 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:10 smithi033 podman[94393]: 2024-04-26 11:47:10.754321517 +0000 UTC m=+0.037829306 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:47:11.176 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:10 smithi033 podman[94393]: 2024-04-26 11:47:10.881957952 +0000 UTC m=+0.165465781 container create ec0acf6c5f9f56065247ca28f7bb64d84b5922ae2353696e13e5ecf950df9d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:47:11.678 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:11 smithi033 podman[94393]: 2024-04-26 11:47:11.186687908 +0000 UTC m=+0.470195760 container init ec0acf6c5f9f56065247ca28f7bb64d84b5922ae2353696e13e5ecf950df9d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T11:47:11.678 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:11 smithi033 podman[94393]: 2024-04-26 11:47:11.199073837 +0000 UTC m=+0.482581658 container start ec0acf6c5f9f56065247ca28f7bb64d84b5922ae2353696e13e5ecf950df9d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:47:11.678 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mds[94411]: -- 172.21.15.33:0/2257204395 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ac4b31a40 con 0x562ac4b04800 2024-04-26T11:47:11.678 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: starting mds.a at 2024-04-26T11:47:11.678 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:11 smithi033 bash[94393]: ec0acf6c5f9f56065247ca28f7bb64d84b5922ae2353696e13e5ecf950df9d19 2024-04-26T11:47:11.678 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:11 smithi033 systemd[1]: Started Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: pgmap v569: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: daemon mds.a restarted 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: osdmap e148: 12 total, 12 up, 12 in 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:boot 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:12.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:11 smithi194 ceph-mon[33017]: osdmap e149: 12 total, 12 up, 12 in 2024-04-26T11:47:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: pgmap v569: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-26T11:47:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: daemon mds.a restarted 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: osdmap e148: 12 total, 12 up, 12 in 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:boot 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:11 smithi033 ceph-mon[27845]: osdmap e149: 12 total, 12 up, 12 in 2024-04-26T11:47:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: pgmap v569: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: daemon mds.a restarted 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: osdmap e148: 12 total, 12 up, 12 in 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:boot 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:11 smithi096 ceph-mon[32753]: osdmap e149: 12 total, 12 up, 12 in 2024-04-26T11:47:13.796 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:13 smithi033 ceph-mon[27845]: pgmap v572: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T11:47:13.796 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:13 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:resolve 2024-04-26T11:47:13.797 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:13 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:13 smithi194 ceph-mon[33017]: pgmap v572: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T11:47:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:13 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:resolve 2024-04-26T11:47:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:13 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:13 smithi096 ceph-mon[32753]: pgmap v572: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T11:47:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:13 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:resolve 2024-04-26T11:47:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:13 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:14 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:47:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:14 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:reconnect 2024-04-26T11:47:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:14 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:14 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:47:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:14 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:47:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:14 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:reconnect 2024-04-26T11:47:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:14 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:14 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:47:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:14 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:47:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:14 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:reconnect 2024-04-26T11:47:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:14 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:14 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:47:15.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:15 smithi096 ceph-mon[32753]: pgmap v573: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:47:15.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:15 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:rejoin 2024-04-26T11:47:15.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:15 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:15 smithi194 ceph-mon[33017]: pgmap v573: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:47:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:15 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:rejoin 2024-04-26T11:47:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:15 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:15 smithi033 ceph-mon[27845]: pgmap v573: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:47:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:15 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:rejoin 2024-04-26T11:47:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:15 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:16 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:clientreplay 2024-04-26T11:47:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:16 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:clientreplay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:16 smithi194 ceph-mon[33017]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:47:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:16 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:16 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:active 2024-04-26T11:47:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:16 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:16.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:clientreplay 2024-04-26T11:47:16.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:clientreplay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:16.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-mon[27845]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:47:16.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:16.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:active 2024-04-26T11:47:16.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:16.915 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5c2e600700 time 2024-04-26T11:47:16.571759+0000 2024-04-26T11:47:16.915 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:16.915 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c39f1a3dd] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: clone() 2024-04-26T11:47:16.916 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: *** Caught signal (Aborted) ** 2024-04-26T11:47:16.917 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: in thread 7f5c2e600700 thread_name:ms_dispatch 2024-04-26T11:47:16.917 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.574+0000 7f5c2e600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5c2e600700 time 2024-04-26T11:47:16.571759+0000 2024-04-26T11:47:16.917 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:16.917 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.917 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.917 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c39f1a3dd] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: clone() 2024-04-26T11:47:16.918 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c38ca2d20] 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: gsignal() 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: abort() 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c39f1a437] 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.919 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 16: clone() 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.576+0000 7f5c2e600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: in thread 7f5c2e600700 thread_name:ms_dispatch 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c38ca2d20] 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: gsignal() 2024-04-26T11:47:16.920 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: abort() 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c39f1a437] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 16: clone() 2024-04-26T11:47:16.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:16.923 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.923 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: -9> 2024-04-26T11:47:16.574+0000 7f5c2e600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5c2e600700 time 2024-04-26T11:47:16.571759+0000 2024-04-26T11:47:16.923 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:16.923 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.923 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.923 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c39f1a3dd] 2024-04-26T11:47:16.923 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: clone() 2024-04-26T11:47:16.924 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.925 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 0> 2024-04-26T11:47:16.576+0000 7f5c2e600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:16.925 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: in thread 7f5c2e600700 thread_name:ms_dispatch 2024-04-26T11:47:16.925 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.925 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.925 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c38ca2d20] 2024-04-26T11:47:16.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: gsignal() 2024-04-26T11:47:16.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: abort() 2024-04-26T11:47:16.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c39f1a437] 2024-04-26T11:47:16.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 16: clone() 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: -27> 2024-04-26T11:47:16.574+0000 7f5c2e600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5c2e600700 time 2024-04-26T11:47:16.571759+0000 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c39f1a3dd] 2024-04-26T11:47:16.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: clone() 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: -18> 2024-04-26T11:47:16.576+0000 7f5c2e600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: in thread 7f5c2e600700 thread_name:ms_dispatch 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c38ca2d20] 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2: gsignal() 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 3: abort() 2024-04-26T11:47:16.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c39f1a437] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c39f1a5a3] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x555e54e8e822] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x555e54dfe598] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555e54e00c17] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555e54cf9d0b] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555e54cfa2ac] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x555e54ce2fad] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f5c3a1aee20] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 13: (DispatchQueue::entry()+0x51f) [0x7f5c3a1abfdf] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5c3a27efc1] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5c38c981ca] 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 16: clone() 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:16.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[68650]: 2024-04-26T11:47:16.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:16 smithi033 podman[94446]: 2024-04-26 11:47:16.708348641 +0000 UTC m=+0.034585257 container died cb67d2abbe02999b4decde9512b00dacf0f0812f0c8ea0fc15e562aacf930589 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:47:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:16 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:clientreplay 2024-04-26T11:47:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:16 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:clientreplay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:16 smithi096 ceph-mon[32753]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:47:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:16 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:16 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/500691090,v1:172.21.15.33:6839/500691090] up:active 2024-04-26T11:47:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:16 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:17.383+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:17.383+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:17.385 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:17.383+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:47:17.588 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:17 smithi033 podman[94446]: 2024-04-26 11:47:17.398834707 +0000 UTC m=+0.725071365 container remove cb67d2abbe02999b4decde9512b00dacf0f0812f0c8ea0fc15e562aacf930589 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T11:47:17.588 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:17 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:47:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:17 smithi194 ceph-mon[33017]: pgmap v574: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6.7 KiB/s wr, 8 op/s 2024-04-26T11:47:17.877 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:17 smithi033 ceph-mon[27845]: pgmap v574: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6.7 KiB/s wr, 8 op/s 2024-04-26T11:47:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:17 smithi096 ceph-mon[32753]: pgmap v574: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6.7 KiB/s wr, 8 op/s 2024-04-26T11:47:17.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:47:17 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:47:17.487+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:47:18.176 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:17 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-26T11:47:18.176 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:17 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.773s CPU time. 2024-04-26T11:47:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:19 smithi194 ceph-mon[33017]: pgmap v575: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6.6 KiB/s wr, 4 op/s 2024-04-26T11:47:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:19 smithi033 ceph-mon[27845]: pgmap v575: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6.6 KiB/s wr, 4 op/s 2024-04-26T11:47:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:19 smithi096 ceph-mon[32753]: pgmap v575: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6.6 KiB/s wr, 4 op/s 2024-04-26T11:47:21.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:21.754+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0005850 con 0x7fef00077780 2024-04-26T11:47:21.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:21.755+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0005850 con 0x7fef00077780 2024-04-26T11:47:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:21.763+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc005820 con 0x564385821900 2024-04-26T11:47:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:21.764+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc005820 con 0x564385821900 2024-04-26T11:47:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:21 smithi194 ceph-mon[33017]: pgmap v576: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 66 KiB/s wr, 35 op/s 2024-04-26T11:47:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:21 smithi033 ceph-mon[27845]: pgmap v576: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 66 KiB/s wr, 35 op/s 2024-04-26T11:47:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:21 smithi096 ceph-mon[32753]: pgmap v576: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 66 KiB/s wr, 35 op/s 2024-04-26T11:47:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:23 smithi194 ceph-mon[33017]: pgmap v577: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 55 KiB/s wr, 29 op/s 2024-04-26T11:47:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:23 smithi033 ceph-mon[27845]: pgmap v577: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 55 KiB/s wr, 29 op/s 2024-04-26T11:47:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:23 smithi096 ceph-mon[32753]: pgmap v577: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 55 KiB/s wr, 29 op/s 2024-04-26T11:47:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:25 smithi033 ceph-mon[27845]: pgmap v578: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 50 KiB/s wr, 27 op/s 2024-04-26T11:47:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:25 smithi096 ceph-mon[32753]: pgmap v578: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 50 KiB/s wr, 27 op/s 2024-04-26T11:47:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:25 smithi194 ceph-mon[33017]: pgmap v578: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 50 KiB/s wr, 27 op/s 2024-04-26T11:47:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:27 smithi033 ceph-mon[27845]: pgmap v579: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 50 KiB/s wr, 28 op/s 2024-04-26T11:47:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:27 smithi096 ceph-mon[32753]: pgmap v579: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 50 KiB/s wr, 28 op/s 2024-04-26T11:47:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:27 smithi194 ceph-mon[33017]: pgmap v579: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 50 KiB/s wr, 28 op/s 2024-04-26T11:47:28.340 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:47:28.341 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:27 smithi033 systemd[1]: Stopped Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:47:28.341 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.773s CPU time. 2024-04-26T11:47:28.341 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:27 smithi033 systemd[1]: Starting Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:47:28.655 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 podman[94553]: 2024-04-26 11:47:28.247456602 +0000 UTC m=+0.042502837 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:47:28.655 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 podman[94553]: 2024-04-26 11:47:28.375038541 +0000 UTC m=+0.170084759 container create 45bc70d112acbfb28945fb252694b70dad0d3df7e413a2f5f27479f1fde6966d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:47:28.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 podman[94553]: 2024-04-26 11:47:28.654164146 +0000 UTC m=+0.449210423 container init 45bc70d112acbfb28945fb252694b70dad0d3df7e413a2f5f27479f1fde6966d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:47:28.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 podman[94553]: 2024-04-26 11:47:28.664275224 +0000 UTC m=+0.459321428 container start 45bc70d112acbfb28945fb252694b70dad0d3df7e413a2f5f27479f1fde6966d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T11:47:28.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 ceph-mds[94572]: -- 172.21.15.33:0/3552723098 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559fae5e1a40 con 0x559fae5b5000 2024-04-26T11:47:28.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: starting mds.g at 2024-04-26T11:47:28.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 bash[94553]: 45bc70d112acbfb28945fb252694b70dad0d3df7e413a2f5f27479f1fde6966d 2024-04-26T11:47:28.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:28 smithi033 systemd[1]: Started Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:47:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: pgmap v580: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-04-26T11:47:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: daemon mds.g restarted 2024-04-26T11:47:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:47:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: osdmap e150: 12 total, 12 up, 12 in 2024-04-26T11:47:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:boot 2024-04-26T11:47:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:47:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:29 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: pgmap v580: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-04-26T11:47:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: daemon mds.g restarted 2024-04-26T11:47:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:47:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: osdmap e150: 12 total, 12 up, 12 in 2024-04-26T11:47:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:boot 2024-04-26T11:47:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:47:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:29 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: pgmap v580: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: daemon mds.g restarted 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: osdmap e150: 12 total, 12 up, 12 in 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:boot 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:29 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:31.000 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:30 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:resolve 2024-04-26T11:47:31.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:30 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:30 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:resolve 2024-04-26T11:47:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:30 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:30 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:resolve 2024-04-26T11:47:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:30 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:31 smithi194 ceph-mon[33017]: pgmap v582: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:47:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:31 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:47:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:reconnect 2024-04-26T11:47:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:31 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:31 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:47:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:31 smithi033 ceph-mon[27845]: pgmap v582: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:47:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:31 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:47:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:reconnect 2024-04-26T11:47:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:31 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:31 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:47:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:31 smithi096 ceph-mon[32753]: pgmap v582: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:47:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:31 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:47:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:reconnect 2024-04-26T11:47:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:31 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:31 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:47:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:32.387+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:32.387+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:32.387+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:47:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:32 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:rejoin 2024-04-26T11:47:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:32 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:32 smithi194 ceph-mon[33017]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:47:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:rejoin 2024-04-26T11:47:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-mon[27845]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:47:33.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe72a400700 time 2024-04-26T11:47:32.821249+0000 2024-04-26T11:47:33.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:33.177 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe735da43dd] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.178 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: clone() 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:32.822+0000 7fe72a400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe72a400700 time 2024-04-26T11:47:32.821249+0000 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe735da43dd] 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.179 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: clone() 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: *** Caught signal (Aborted) ** 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: in thread 7fe72a400700 thread_name:ms_dispatch 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe734b2cd20] 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: gsignal() 2024-04-26T11:47:33.180 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: abort() 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe735da4437] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.181 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 16: clone() 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:32.824+0000 7fe72a400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: in thread 7fe72a400700 thread_name:ms_dispatch 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe734b2cd20] 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: gsignal() 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: abort() 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe735da4437] 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.182 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 16: clone() 2024-04-26T11:47:33.183 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:32 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:rejoin 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:32 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:32 smithi096 ceph-mon[32753]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: -1> 2024-04-26T11:47:32.822+0000 7fe72a400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe72a400700 time 2024-04-26T11:47:32.821249+0000 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.184 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe735da43dd] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: clone() 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.185 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 0> 2024-04-26T11:47:32.824+0000 7fe72a400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: in thread 7fe72a400700 thread_name:ms_dispatch 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe734b2cd20] 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: gsignal() 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: abort() 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe735da4437] 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.186 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 16: clone() 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: -1> 2024-04-26T11:47:32.822+0000 7fe72a400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe72a400700 time 2024-04-26T11:47:32.821249+0000 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:33.187 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.188 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.188 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe735da43dd] 2024-04-26T11:47:33.188 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.188 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.188 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.188 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.188 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: clone() 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 0> 2024-04-26T11:47:32.824+0000 7fe72a400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: in thread 7fe72a400700 thread_name:ms_dispatch 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe734b2cd20] 2024-04-26T11:47:33.189 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2: gsignal() 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 3: abort() 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe735da4437] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe735da45a3] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562ab7cf0822] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562ab7c60598] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562ab7c62c17] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562ab7b5bd0b] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562ab7b5c2ac] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562ab7b44fad] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fe736038e20] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 13: (DispatchQueue::entry()+0x51f) [0x7fe736035fdf] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe736108fc1] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe734b221ca] 2024-04-26T11:47:33.190 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 16: clone() 2024-04-26T11:47:33.191 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:33.191 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94407]: 2024-04-26T11:47:33.191 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:32 smithi033 podman[94606]: 2024-04-26 11:47:32.968231427 +0000 UTC m=+0.054610566 container died ec0acf6c5f9f56065247ca28f7bb64d84b5922ae2353696e13e5ecf950df9d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:47:33.926 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:33 smithi033 podman[94606]: 2024-04-26 11:47:33.725561974 +0000 UTC m=+0.811941140 container remove ec0acf6c5f9f56065247ca28f7bb64d84b5922ae2353696e13e5ecf950df9d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:47:33.926 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:33 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:47:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:33 smithi033 ceph-mon[27845]: pgmap v583: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:47:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:33 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:33 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:active 2024-04-26T11:47:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:33 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:33 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:33 smithi194 ceph-mon[33017]: pgmap v583: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:47:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:33 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:33 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:active 2024-04-26T11:47:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:33 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:33 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:33 smithi096 ceph-mon[32753]: pgmap v583: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:47:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:33 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:33 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/3458282153,v1:172.21.15.33:6837/3458282153] up:active 2024-04-26T11:47:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:33 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:33 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:34.426 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:34 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-26T11:47:34.667 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:34 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.190s CPU time. 2024-04-26T11:47:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:35 smithi194 ceph-mon[33017]: pgmap v584: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 307 B/s wr, 7 op/s 2024-04-26T11:47:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:35 smithi033 ceph-mon[27845]: pgmap v584: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 307 B/s wr, 7 op/s 2024-04-26T11:47:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:35 smithi096 ceph-mon[32753]: pgmap v584: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 307 B/s wr, 7 op/s 2024-04-26T11:47:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:37 smithi194 ceph-mon[33017]: pgmap v585: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-26T11:47:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:37 smithi033 ceph-mon[27845]: pgmap v585: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-26T11:47:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:37 smithi096 ceph-mon[32753]: pgmap v585: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-26T11:47:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:39 smithi194 ceph-mon[33017]: pgmap v586: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-26T11:47:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:39 smithi033 ceph-mon[27845]: pgmap v586: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-26T11:47:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:39 smithi096 ceph-mon[32753]: pgmap v586: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-26T11:47:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:41 smithi194 ceph-mon[33017]: pgmap v587: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-26T11:47:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:41 smithi033 ceph-mon[27845]: pgmap v587: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-26T11:47:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:41 smithi096 ceph-mon[32753]: pgmap v587: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-26T11:47:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:43 smithi194 ceph-mon[33017]: pgmap v588: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-26T11:47:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:43 smithi194 ceph-mon[33017]: osdmap e151: 12 total, 12 up, 12 in 2024-04-26T11:47:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:43 smithi033 ceph-mon[27845]: pgmap v588: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-26T11:47:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:43 smithi033 ceph-mon[27845]: osdmap e151: 12 total, 12 up, 12 in 2024-04-26T11:47:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:43 smithi096 ceph-mon[32753]: pgmap v588: 97 pgs: 97 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-26T11:47:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:43 smithi096 ceph-mon[32753]: osdmap e151: 12 total, 12 up, 12 in 2024-04-26T11:47:44.830 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:44 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:47:44.830 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:44 smithi033 systemd[1]: Stopped Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:47:44.830 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:44 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.190s CPU time. 2024-04-26T11:47:44.830 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:44 smithi033 systemd[1]: Starting Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:47:45.106 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:44 smithi033 podman[94694]: 2024-04-26 11:47:44.735044456 +0000 UTC m=+0.034802194 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:47:45.106 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:44 smithi033 podman[94694]: 2024-04-26 11:47:44.851280639 +0000 UTC m=+0.151038412 container create 7588f7e49d5d19546bd79eecc673a54932f1e2f709e27dc791df4336db6019ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:47:45.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:45 smithi033 podman[94694]: 2024-04-26 11:47:45.105371812 +0000 UTC m=+0.405129592 container init 7588f7e49d5d19546bd79eecc673a54932f1e2f709e27dc791df4336db6019ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:47:45.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:45 smithi033 podman[94694]: 2024-04-26 11:47:45.116742869 +0000 UTC m=+0.416500652 container start 7588f7e49d5d19546bd79eecc673a54932f1e2f709e27dc791df4336db6019ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-04-26T11:47:45.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mds[94712]: -- 172.21.15.33:0/3026549181 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557250011a40 con 0x55724ffe5000 2024-04-26T11:47:45.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: starting mds.a at 2024-04-26T11:47:45.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:45 smithi033 bash[94694]: 7588f7e49d5d19546bd79eecc673a54932f1e2f709e27dc791df4336db6019ad 2024-04-26T11:47:45.427 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:47:45 smithi033 systemd[1]: Started Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:47:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: pgmap v590: 97 pgs: 97 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 51 KiB/s wr, 16 op/s 2024-04-26T11:47:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: daemon mds.a restarted 2024-04-26T11:47:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:46.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: osdmap e152: 12 total, 12 up, 12 in 2024-04-26T11:47:46.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:boot 2024-04-26T11:47:46.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:47:46.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:46.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:47:46.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:45 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: pgmap v590: 97 pgs: 97 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 51 KiB/s wr, 16 op/s 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: daemon mds.a restarted 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: osdmap e152: 12 total, 12 up, 12 in 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:boot 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:47:46.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:45 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: pgmap v590: 97 pgs: 97 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 51 KiB/s wr, 16 op/s 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: daemon mds.a restarted 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: osdmap e152: 12 total, 12 up, 12 in 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:boot 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T11:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:45 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:47.395+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:47.395+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:47:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:47:47.395+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:47:48.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:47 smithi033 ceph-mon[27845]: pgmap v592: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-26T11:47:48.074 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:47 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:resolve 2024-04-26T11:47:48.074 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:47 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:47 smithi194 ceph-mon[33017]: pgmap v592: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-26T11:47:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:47 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:resolve 2024-04-26T11:47:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:47 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:47 smithi096 ceph-mon[32753]: pgmap v592: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-26T11:47:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:47 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:resolve 2024-04-26T11:47:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:47 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:48 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:47:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:48 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:reconnect 2024-04-26T11:47:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:48 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:48 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:47:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:48 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:47:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:48 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:reconnect 2024-04-26T11:47:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:48 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:48 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:47:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:48 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:47:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:48 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:reconnect 2024-04-26T11:47:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:48 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:48 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:47:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-mon[27845]: pgmap v593: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-04-26T11:47:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:rejoin 2024-04-26T11:47:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-mon[27845]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9843200700 time 2024-04-26T11:47:49.886807+0000 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f984eafc3dd] 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: clone() 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:49.888+0000 7f9843200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9843200700 time 2024-04-26T11:47:49.886807+0000 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:50.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f984eafc3dd] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: clone() 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: *** Caught signal (Aborted) ** 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: in thread 7f9843200700 thread_name:ms_dispatch 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f984d884d20] 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: gsignal() 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: abort() 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f984eafc437] 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:49 smithi096 ceph-mon[32753]: pgmap v593: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-04-26T11:47:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:49 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:rejoin 2024-04-26T11:47:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:49 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:49 smithi096 ceph-mon[32753]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 16: clone() 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:49.891+0000 7f9843200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: in thread 7f9843200700 thread_name:ms_dispatch 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f984d884d20] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: gsignal() 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: abort() 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f984eafc437] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 16: clone() 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: -1> 2024-04-26T11:47:49.888+0000 7f9843200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9843200700 time 2024-04-26T11:47:49.886807+0000 2024-04-26T11:47:50.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f984eafc3dd] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: clone() 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 0> 2024-04-26T11:47:49.891+0000 7f9843200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: in thread 7f9843200700 thread_name:ms_dispatch 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f984d884d20] 2024-04-26T11:47:50.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: gsignal() 2024-04-26T11:47:50.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: abort() 2024-04-26T11:47:50.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f984eafc437] 2024-04-26T11:47:50.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 16: clone() 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: -1> 2024-04-26T11:47:49.888+0000 7f9843200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9843200700 time 2024-04-26T11:47:49.886807+0000 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f984eafc3dd] 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: clone() 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 0> 2024-04-26T11:47:49.891+0000 7f9843200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: in thread 7f9843200700 thread_name:ms_dispatch 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f984d884d20] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2: gsignal() 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 3: abort() 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f984eafc437] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f984eafc5a3] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559f8528b822] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559f851fb598] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f851fdc17] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f850f6d0b] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f850f72ac] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559f850dffad] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f984ed90e20] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 13: (DispatchQueue::entry()+0x51f) [0x7f984ed8dfdf] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f984ee60fc1] 2024-04-26T11:47:50.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f984d87a1ca] 2024-04-26T11:47:50.194 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 16: clone() 2024-04-26T11:47:50.194 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:47:50.194 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:49 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94568]: 2024-04-26T11:47:50.194 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:50 smithi033 podman[94746]: 2024-04-26 11:47:50.017893957 +0000 UTC m=+0.042966715 container died 45bc70d112acbfb28945fb252694b70dad0d3df7e413a2f5f27479f1fde6966d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS) 2024-04-26T11:47:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:49 smithi194 ceph-mon[33017]: pgmap v593: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-04-26T11:47:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:49 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:rejoin 2024-04-26T11:47:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:49 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:49 smithi194 ceph-mon[33017]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:47:51.176 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:51 smithi033 podman[94746]: 2024-04-26 11:47:51.027008687 +0000 UTC m=+1.052081470 container remove 45bc70d112acbfb28945fb252694b70dad0d3df7e413a2f5f27479f1fde6966d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:47:51.176 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:51 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:47:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:50 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:51.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:50 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:active 2024-04-26T11:47:51.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:50 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:51.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:50 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:50 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:50 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:active 2024-04-26T11:47:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:50 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:50 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:50 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:47:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:50 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/95222549,v1:172.21.15.33:6839/95222549] up:active 2024-04-26T11:47:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:50 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:47:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:50 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:47:51.871 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:51 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-26T11:47:51.871 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:47:51 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.153s CPU time. 2024-04-26T11:47:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:51 smithi033 ceph-mon[27845]: pgmap v594: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-04-26T11:47:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:51 smithi033 ceph-mon[27845]: osdmap e153: 12 total, 12 up, 12 in 2024-04-26T11:47:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:51 smithi096 ceph-mon[32753]: pgmap v594: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-04-26T11:47:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:51 smithi096 ceph-mon[32753]: osdmap e153: 12 total, 12 up, 12 in 2024-04-26T11:47:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:51 smithi194 ceph-mon[33017]: pgmap v594: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-04-26T11:47:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:51 smithi194 ceph-mon[33017]: osdmap e153: 12 total, 12 up, 12 in 2024-04-26T11:47:52.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:47:52 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:47:52.489+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:47:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:53 smithi033 ceph-mon[27845]: pgmap v596: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-04-26T11:47:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:53 smithi096 ceph-mon[32753]: pgmap v596: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-04-26T11:47:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:53 smithi194 ceph-mon[33017]: pgmap v596: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-04-26T11:47:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:55 smithi033 ceph-mon[27845]: pgmap v597: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-04-26T11:47:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:55 smithi096 ceph-mon[32753]: pgmap v597: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-04-26T11:47:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:55 smithi194 ceph-mon[33017]: pgmap v597: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-04-26T11:47:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:57 smithi033 ceph-mon[27845]: pgmap v598: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-04-26T11:47:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:57 smithi096 ceph-mon[32753]: pgmap v598: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-04-26T11:47:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:57 smithi194 ceph-mon[33017]: pgmap v598: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-04-26T11:48:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:47:59 smithi033 ceph-mon[27845]: pgmap v599: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-04-26T11:48:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:47:59 smithi096 ceph-mon[32753]: pgmap v599: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-04-26T11:48:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:47:59 smithi194 ceph-mon[33017]: pgmap v599: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-04-26T11:48:02.090 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:01 smithi033 ceph-mon[27845]: pgmap v600: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-26T11:48:02.090 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:01 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:48:02.090 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:01 smithi033 systemd[1]: Stopped Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:02.090 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:01 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.153s CPU time. 2024-04-26T11:48:02.090 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:01 smithi033 systemd[1]: Starting Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:48:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:01 smithi096 ceph-mon[32753]: pgmap v600: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-26T11:48:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:01 smithi194 ceph-mon[33017]: pgmap v600: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-26T11:48:02.408 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 podman[94836]: 2024-04-26 11:48:01.994127227 +0000 UTC m=+0.034091408 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:48:02.408 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 podman[94836]: 2024-04-26 11:48:02.11117696 +0000 UTC m=+0.151141166 container create 106b80c56393e29ab9c3e2044ac0386ea5e1407d4f5701f7277dae3f3c413b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T11:48:02.408 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 podman[94836]: 2024-04-26 11:48:02.406856585 +0000 UTC m=+0.446820798 container init 106b80c56393e29ab9c3e2044ac0386ea5e1407d4f5701f7277dae3f3c413b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T11:48:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:02.410+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:02.410+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:02.411+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:48:02.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 podman[94836]: 2024-04-26 11:48:02.41791449 +0000 UTC m=+0.457878681 container start 106b80c56393e29ab9c3e2044ac0386ea5e1407d4f5701f7277dae3f3c413b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:48:02.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mds[94855]: -- 172.21.15.33:0/2752804131 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56001f933a40 con 0x56001f906800 2024-04-26T11:48:02.677 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: starting mds.g at 2024-04-26T11:48:02.677 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 bash[94836]: 106b80c56393e29ab9c3e2044ac0386ea5e1407d4f5701f7277dae3f3c413b36 2024-04-26T11:48:02.677 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:02 smithi033 systemd[1]: Started Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: daemon mds.g restarted 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: osdmap e154: 12 total, 12 up, 12 in 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:boot 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:48:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:02 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: daemon mds.g restarted 2024-04-26T11:48:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: osdmap e154: 12 total, 12 up, 12 in 2024-04-26T11:48:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:boot 2024-04-26T11:48:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:48:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:48:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:48:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:02 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: daemon mds.g restarted 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: osdmap e154: 12 total, 12 up, 12 in 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:boot 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:02 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:03 smithi033 ceph-mon[27845]: pgmap v601: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-26T11:48:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:03 smithi096 ceph-mon[32753]: pgmap v601: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-26T11:48:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:03 smithi194 ceph-mon[33017]: pgmap v601: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-26T11:48:05.185 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:04 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:resolve 2024-04-26T11:48:05.185 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:04 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:05.185 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:48:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:04 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:resolve 2024-04-26T11:48:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:04 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:48:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:04 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:resolve 2024-04-26T11:48:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:04 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:48:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:05 smithi194 ceph-mon[33017]: pgmap v603: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-26T11:48:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:05 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:48:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:05 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:reconnect 2024-04-26T11:48:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:05 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:05 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:48:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:05 smithi033 ceph-mon[27845]: pgmap v603: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-26T11:48:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:05 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:48:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:05 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:reconnect 2024-04-26T11:48:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:05 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:05 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:48:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:05 smithi096 ceph-mon[32753]: pgmap v603: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-26T11:48:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:05 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:48:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:05 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:reconnect 2024-04-26T11:48:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:05 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:05 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:48:06.928 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1f9ec00700 time 2024-04-26T11:48:06.605231+0000 2024-04-26T11:48:06.928 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:06.928 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.928 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1faa4f43dd] 2024-04-26T11:48:06.928 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.928 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.929 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: clone() 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: *** Caught signal (Aborted) ** 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: in thread 7f1f9ec00700 thread_name:ms_dispatch 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.606+0000 7f1f9ec00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1f9ec00700 time 2024-04-26T11:48:06.605231+0000 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1faa4f43dd] 2024-04-26T11:48:06.930 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: clone() 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.931 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1fa927cd20] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: gsignal() 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: abort() 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1faa4f4437] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.932 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 16: clone() 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.609+0000 7f1f9ec00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: in thread 7f1f9ec00700 thread_name:ms_dispatch 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1fa927cd20] 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: gsignal() 2024-04-26T11:48:06.933 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: abort() 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1faa4f4437] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.934 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 16: clone() 2024-04-26T11:48:06.935 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:06.935 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.935 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: -1> 2024-04-26T11:48:06.606+0000 7f1f9ec00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1f9ec00700 time 2024-04-26T11:48:06.605231+0000 2024-04-26T11:48:06.935 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:06.935 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.935 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1faa4f43dd] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.936 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: clone() 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 0> 2024-04-26T11:48:06.609+0000 7f1f9ec00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: in thread 7f1f9ec00700 thread_name:ms_dispatch 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1fa927cd20] 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: gsignal() 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: abort() 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1faa4f4437] 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.937 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 16: clone() 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: -1> 2024-04-26T11:48:06.606+0000 7f1f9ec00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1f9ec00700 time 2024-04-26T11:48:06.605231+0000 2024-04-26T11:48:06.938 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:06.939 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.939 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.939 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1faa4f43dd] 2024-04-26T11:48:06.939 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.939 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.939 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: clone() 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 0> 2024-04-26T11:48:06.609+0000 7f1f9ec00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:06.940 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: in thread 7f1f9ec00700 thread_name:ms_dispatch 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1fa927cd20] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2: gsignal() 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 3: abort() 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1faa4f4437] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1faa4f45a3] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55722d5a4822] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55722d514598] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55722d516c17] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55722d40fd0b] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55722d4102ac] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55722d3f8fad] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f1faa788e20] 2024-04-26T11:48:06.941 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 13: (DispatchQueue::entry()+0x51f) [0x7f1faa785fdf] 2024-04-26T11:48:06.942 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1faa858fc1] 2024-04-26T11:48:06.942 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1fa92721ca] 2024-04-26T11:48:06.942 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 16: clone() 2024-04-26T11:48:06.942 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:06.942 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a[94708]: 2024-04-26T11:48:06.942 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:06 smithi033 podman[95018]: 2024-04-26 11:48:06.744593523 +0000 UTC m=+0.036282538 container died 7588f7e49d5d19546bd79eecc673a54932f1e2f709e27dc791df4336db6019ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:48:07.266 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:rejoin 2024-04-26T11:48:07.266 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:07.266 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:48:07.266 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:48:07.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:48:07.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:48:07.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:07.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:active 2024-04-26T11:48:07.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:06 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:rejoin 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:active 2024-04-26T11:48:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:06 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:48:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:rejoin 2024-04-26T11:48:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=i=up:active} 2 up:standby-replay 2 up:standby 2024-04-26T11:48:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T11:48:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:48:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:48:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:48:07.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:07.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/4268980260,v1:172.21.15.33:6837/4268980260] up:active 2024-04-26T11:48:07.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:06 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby 2024-04-26T11:48:07.926 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:07 smithi033 podman[95018]: 2024-04-26 11:48:07.648406458 +0000 UTC m=+0.940095518 container remove 7588f7e49d5d19546bd79eecc673a54932f1e2f709e27dc791df4336db6019ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T11:48:07.926 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:07 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:48:07.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:48:07 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:48:07.490+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:48:08.226 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:07 smithi033 ceph-mon[27845]: pgmap v604: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 419 B/s wr, 8 op/s 2024-04-26T11:48:08.226 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:07 smithi033 ceph-mon[27845]: pgmap v605: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 527 B/s wr, 10 op/s 2024-04-26T11:48:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:07 smithi194 ceph-mon[33017]: pgmap v604: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 419 B/s wr, 8 op/s 2024-04-26T11:48:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:07 smithi194 ceph-mon[33017]: pgmap v605: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 527 B/s wr, 10 op/s 2024-04-26T11:48:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:07 smithi096 ceph-mon[32753]: pgmap v604: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 419 B/s wr, 8 op/s 2024-04-26T11:48:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:07 smithi096 ceph-mon[32753]: pgmap v605: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 527 B/s wr, 10 op/s 2024-04-26T11:48:08.676 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:08 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-26T11:48:08.676 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:08 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.204s CPU time. 2024-04-26T11:48:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:09 smithi194 ceph-mon[33017]: pgmap v606: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-26T11:48:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:09 smithi033 ceph-mon[27845]: pgmap v606: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-26T11:48:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:09 smithi096 ceph-mon[32753]: pgmap v606: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-26T11:48:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:11 smithi194 ceph-mon[33017]: pgmap v607: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 65 KiB/s wr, 40 op/s 2024-04-26T11:48:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:11 smithi033 ceph-mon[27845]: pgmap v607: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 65 KiB/s wr, 40 op/s 2024-04-26T11:48:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:11 smithi096 ceph-mon[32753]: pgmap v607: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 65 KiB/s wr, 40 op/s 2024-04-26T11:48:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:13 smithi194 ceph-mon[33017]: pgmap v608: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 53 KiB/s wr, 32 op/s 2024-04-26T11:48:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:13 smithi033 ceph-mon[27845]: pgmap v608: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 53 KiB/s wr, 32 op/s 2024-04-26T11:48:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:13 smithi096 ceph-mon[32753]: pgmap v608: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 53 KiB/s wr, 32 op/s 2024-04-26T11:48:16.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:15 smithi096 ceph-mon[32753]: pgmap v609: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 52 KiB/s wr, 31 op/s 2024-04-26T11:48:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:15 smithi194 ceph-mon[33017]: pgmap v609: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 52 KiB/s wr, 31 op/s 2024-04-26T11:48:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:15 smithi033 ceph-mon[27845]: pgmap v609: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 52 KiB/s wr, 31 op/s 2024-04-26T11:48:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:17.426+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:17.426+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:48:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:17.426+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:17 smithi194 ceph-mon[33017]: pgmap v610: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 51 KiB/s wr, 29 op/s 2024-04-26T11:48:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:17 smithi033 ceph-mon[27845]: pgmap v610: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 51 KiB/s wr, 29 op/s 2024-04-26T11:48:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:17 smithi096 ceph-mon[32753]: pgmap v610: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 51 KiB/s wr, 29 op/s 2024-04-26T11:48:18.864 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:48:18.864 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:18 smithi033 systemd[1]: Stopped Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:18.864 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.204s CPU time. 2024-04-26T11:48:18.864 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-04-26T11:48:18.864 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-26T11:48:18.865 INFO:journalctl@ceph.mds.a.smithi033.stdout:Apr 26 11:48:18 smithi033 systemd[1]: Failed to start Ceph mds.a for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:19 smithi033 ceph-mon[27845]: pgmap v611: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-04-26T11:48:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:19 smithi194 ceph-mon[33017]: pgmap v611: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-04-26T11:48:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:19 smithi096 ceph-mon[32753]: pgmap v611: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-04-26T11:48:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:21 smithi033 ceph-mon[27845]: pgmap v612: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 42 KiB/s wr, 27 op/s 2024-04-26T11:48:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:21 smithi194 ceph-mon[33017]: pgmap v612: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 42 KiB/s wr, 27 op/s 2024-04-26T11:48:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:21 smithi096 ceph-mon[32753]: pgmap v612: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 42 KiB/s wr, 27 op/s 2024-04-26T11:48:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:22 smithi194 ceph-mon[33017]: Replacing daemon mds.a as rank 1 with standby daemon mds.g 2024-04-26T11:48:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:22 smithi194 ceph-mon[33017]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-26T11:48:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:22 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:22 smithi194 ceph-mon[33017]: osdmap e155: 12 total, 12 up, 12 in 2024-04-26T11:48:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:22 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:22 smithi033 ceph-mon[27845]: Replacing daemon mds.a as rank 1 with standby daemon mds.g 2024-04-26T11:48:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:22 smithi033 ceph-mon[27845]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-26T11:48:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:22 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:22 smithi033 ceph-mon[27845]: osdmap e155: 12 total, 12 up, 12 in 2024-04-26T11:48:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:22 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:22 smithi096 ceph-mon[32753]: Replacing daemon mds.a as rank 1 with standby daemon mds.g 2024-04-26T11:48:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:22 smithi096 ceph-mon[32753]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-26T11:48:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:22 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:22 smithi096 ceph-mon[32753]: osdmap e155: 12 total, 12 up, 12 in 2024-04-26T11:48:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:22 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:23.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:23 smithi033 ceph-mon[27845]: pgmap v614: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T11:48:23.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:23 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:resolve 2024-04-26T11:48:23.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:23 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:23 smithi194 ceph-mon[33017]: pgmap v614: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T11:48:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:23 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:resolve 2024-04-26T11:48:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:23 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:23 smithi096 ceph-mon[32753]: pgmap v614: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T11:48:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:23 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:resolve 2024-04-26T11:48:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:23 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:24 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:48:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:reconnect 2024-04-26T11:48:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:24 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:24 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:48:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:24 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:48:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:reconnect 2024-04-26T11:48:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:24 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:24 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:48:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:24 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:48:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:reconnect 2024-04-26T11:48:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:24 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:24 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:48:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-mon[27845]: pgmap v615: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T11:48:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:rejoin 2024-04-26T11:48:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-mon[27845]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:48:25.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0d54200700 time 2024-04-26T11:48:25.651374+0000 2024-04-26T11:48:25.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:25.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0d5faf33dd] 2024-04-26T11:48:25.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: clone() 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.652+0000 7f0d54200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0d54200700 time 2024-04-26T11:48:25.651374+0000 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:25.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0d5faf33dd] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: clone() 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: *** Caught signal (Aborted) ** 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: in thread 7f0d54200700 thread_name:ms_dispatch 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d5e87bd20] 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: gsignal() 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: abort() 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0d5faf3437] 2024-04-26T11:48:25.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 16: clone() 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.654+0000 7f0d54200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: in thread 7f0d54200700 thread_name:ms_dispatch 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d5e87bd20] 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: gsignal() 2024-04-26T11:48:25.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: abort() 2024-04-26T11:48:25.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0d5faf3437] 2024-04-26T11:48:25.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 16: clone() 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: -1> 2024-04-26T11:48:25.652+0000 7f0d54200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0d54200700 time 2024-04-26T11:48:25.651374+0000 2024-04-26T11:48:25.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:25.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0d5faf33dd] 2024-04-26T11:48:25.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: clone() 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 0> 2024-04-26T11:48:25.654+0000 7f0d54200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: in thread 7f0d54200700 thread_name:ms_dispatch 2024-04-26T11:48:25.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d5e87bd20] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: gsignal() 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: abort() 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0d5faf3437] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 16: clone() 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: -1> 2024-04-26T11:48:25.652+0000 7f0d54200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0d54200700 time 2024-04-26T11:48:25.651374+0000 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0d5faf33dd] 2024-04-26T11:48:25.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.939 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.939 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:25 smithi096 ceph-mon[32753]: pgmap v615: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T11:48:25.939 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:rejoin 2024-04-26T11:48:25.939 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:25 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:25.939 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:25 smithi096 ceph-mon[32753]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: clone() 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 0> 2024-04-26T11:48:25.654+0000 7f0d54200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: in thread 7f0d54200700 thread_name:ms_dispatch 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0d5e87bd20] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2: gsignal() 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 3: abort() 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0d5faf3437] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f0d5faf35a3] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55ffe905c822] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55ffe8fcc598] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffe8fcec17] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffe8ec7d0b] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffe8ec82ac] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55ffe8eb0fad] 2024-04-26T11:48:25.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f0d5fd87e20] 2024-04-26T11:48:25.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 13: (DispatchQueue::entry()+0x51f) [0x7f0d5fd84fdf] 2024-04-26T11:48:25.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0d5fe57fc1] 2024-04-26T11:48:25.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0d5e8711ca] 2024-04-26T11:48:25.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 16: clone() 2024-04-26T11:48:25.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:25.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[94851]: 2024-04-26T11:48:25.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:25 smithi033 podman[95086]: 2024-04-26 11:48:25.787562992 +0000 UTC m=+0.041384117 container died 106b80c56393e29ab9c3e2044ac0386ea5e1407d4f5701f7277dae3f3c413b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424) 2024-04-26T11:48:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:25 smithi194 ceph-mon[33017]: pgmap v615: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T11:48:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:rejoin 2024-04-26T11:48:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:25 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:25 smithi194 ceph-mon[33017]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:48:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:26 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:26 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:active 2024-04-26T11:48:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:26 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:26.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:26 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:48:26.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:26 smithi033 podman[95086]: 2024-04-26 11:48:26.508830413 +0000 UTC m=+0.762651581 container remove 106b80c56393e29ab9c3e2044ac0386ea5e1407d4f5701f7277dae3f3c413b36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:48:26.927 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:26 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:48:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:26 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:26 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:active 2024-04-26T11:48:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:26 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:26 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:48:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:26 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:26 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6838/165909200,v1:172.21.15.33:6839/165909200] up:active 2024-04-26T11:48:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:26 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:26 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:48:27.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-26T11:48:27.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.263s CPU time. 2024-04-26T11:48:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:27 smithi033 ceph-mon[27845]: pgmap v616: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:48:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:27 smithi096 ceph-mon[32753]: pgmap v616: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:48:27.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:48:27 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:48:27.491+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:48:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:27 smithi194 ceph-mon[33017]: pgmap v616: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:48:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:29 smithi033 ceph-mon[27845]: pgmap v617: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:48:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:29 smithi096 ceph-mon[32753]: pgmap v617: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:48:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:29 smithi194 ceph-mon[33017]: pgmap v617: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:48:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:31 smithi033 ceph-mon[27845]: pgmap v618: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 53 KiB/s wr, 17 op/s 2024-04-26T11:48:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:31 smithi096 ceph-mon[32753]: pgmap v618: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 53 KiB/s wr, 17 op/s 2024-04-26T11:48:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:31 smithi194 ceph-mon[33017]: pgmap v618: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 53 KiB/s wr, 17 op/s 2024-04-26T11:48:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:32.441+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:32.441+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:32.441+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:48:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:33 smithi033 ceph-mon[27845]: pgmap v619: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 50 KiB/s wr, 16 op/s 2024-04-26T11:48:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:33 smithi033 ceph-mon[27845]: osdmap e156: 12 total, 12 up, 12 in 2024-04-26T11:48:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:33 smithi096 ceph-mon[32753]: pgmap v619: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 50 KiB/s wr, 16 op/s 2024-04-26T11:48:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:33 smithi096 ceph-mon[32753]: osdmap e156: 12 total, 12 up, 12 in 2024-04-26T11:48:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:33 smithi194 ceph-mon[33017]: pgmap v619: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 50 KiB/s wr, 16 op/s 2024-04-26T11:48:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:33 smithi194 ceph-mon[33017]: osdmap e156: 12 total, 12 up, 12 in 2024-04-26T11:48:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:35 smithi033 ceph-mon[27845]: pgmap v621: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 53 KiB/s wr, 17 op/s 2024-04-26T11:48:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:35 smithi096 ceph-mon[32753]: pgmap v621: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 53 KiB/s wr, 17 op/s 2024-04-26T11:48:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:35 smithi194 ceph-mon[33017]: pgmap v621: 97 pgs: 97 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 53 KiB/s wr, 17 op/s 2024-04-26T11:48:37.580 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:48:37.580 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 systemd[1]: Stopped Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:37.580 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.263s CPU time. 2024-04-26T11:48:37.581 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 systemd[1]: Starting Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:48:37.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:37 smithi033 ceph-mon[27845]: pgmap v622: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 53 KiB/s wr, 16 op/s 2024-04-26T11:48:37.857 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 podman[95177]: 2024-04-26 11:48:37.48618612 +0000 UTC m=+0.036947508 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:48:37.857 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 podman[95177]: 2024-04-26 11:48:37.592716778 +0000 UTC m=+0.143478177 container create 35a5e91ebbf5ac78b790e66391e03169d6a643d364461e2b548bbb8f46d74134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T11:48:37.857 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 podman[95177]: 2024-04-26 11:48:37.855433657 +0000 UTC m=+0.406195097 container init 35a5e91ebbf5ac78b790e66391e03169d6a643d364461e2b548bbb8f46d74134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T11:48:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:37 smithi096 ceph-mon[32753]: pgmap v622: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 53 KiB/s wr, 16 op/s 2024-04-26T11:48:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:37 smithi194 ceph-mon[33017]: pgmap v622: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 53 KiB/s wr, 16 op/s 2024-04-26T11:48:38.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 podman[95177]: 2024-04-26 11:48:37.867259821 +0000 UTC m=+0.418021191 container start 35a5e91ebbf5ac78b790e66391e03169d6a643d364461e2b548bbb8f46d74134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:48:38.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 ceph-mds[95196]: -- 172.21.15.33:0/304527856 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56037e901a40 con 0x56037e8d4800 2024-04-26T11:48:38.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: starting mds.g at 2024-04-26T11:48:38.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 bash[95177]: 35a5e91ebbf5ac78b790e66391e03169d6a643d364461e2b548bbb8f46d74134 2024-04-26T11:48:38.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:37 smithi033 systemd[1]: Started Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: daemon mds.g restarted 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: osdmap e157: 12 total, 12 up, 12 in 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:boot 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:48:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:38 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: daemon mds.g restarted 2024-04-26T11:48:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: osdmap e157: 12 total, 12 up, 12 in 2024-04-26T11:48:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:boot 2024-04-26T11:48:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:48:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:48:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:48:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:38 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: daemon mds.g restarted 2024-04-26T11:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: osdmap e157: 12 total, 12 up, 12 in 2024-04-26T11:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:boot 2024-04-26T11:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:48:39.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:38 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:39 smithi194 ceph-mon[33017]: pgmap v624: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T11:48:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:39 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:resolve 2024-04-26T11:48:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:39 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:39 smithi033 ceph-mon[27845]: pgmap v624: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T11:48:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:39 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:resolve 2024-04-26T11:48:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:39 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:39 smithi096 ceph-mon[32753]: pgmap v624: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T11:48:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:39 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:resolve 2024-04-26T11:48:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:39 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:41 smithi194 ceph-mon[33017]: pgmap v625: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-26T11:48:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:41 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:48:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:41 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:reconnect 2024-04-26T11:48:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:41 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:41 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:48:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:41 smithi194 ceph-mon[33017]: osdmap e158: 12 total, 12 up, 12 in 2024-04-26T11:48:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:41 smithi033 ceph-mon[27845]: pgmap v625: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-26T11:48:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:41 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:48:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:41 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:reconnect 2024-04-26T11:48:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:41 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:42.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:41 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:48:42.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:41 smithi033 ceph-mon[27845]: osdmap e158: 12 total, 12 up, 12 in 2024-04-26T11:48:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:41 smithi096 ceph-mon[32753]: pgmap v625: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-26T11:48:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:41 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:48:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:41 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:reconnect 2024-04-26T11:48:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:41 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:41 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:48:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:41 smithi096 ceph-mon[32753]: osdmap e158: 12 total, 12 up, 12 in 2024-04-26T11:48:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:42 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:rejoin 2024-04-26T11:48:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:42 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:42 smithi194 ceph-mon[33017]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:48:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:rejoin 2024-04-26T11:48:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-mon[27845]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:48:43.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc828200700 time 2024-04-26T11:48:42.734684+0000 2024-04-26T11:48:43.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc833a333dd] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: clone() 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:42.734+0000 7fc828200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc828200700 time 2024-04-26T11:48:42.734684+0000 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc833a333dd] 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: clone() 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: *** Caught signal (Aborted) ** 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: in thread 7fc828200700 thread_name:ms_dispatch 2024-04-26T11:48:43.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc8327bbd20] 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: gsignal() 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: abort() 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc833a33437] 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 16: clone() 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:42.737+0000 7fc828200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: in thread 7fc828200700 thread_name:ms_dispatch 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc8327bbd20] 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: gsignal() 2024-04-26T11:48:43.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: abort() 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc833a33437] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 16: clone() 2024-04-26T11:48:43.184 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:43.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:42 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:rejoin 2024-04-26T11:48:43.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:42 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:43.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:42 smithi096 ceph-mon[32753]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: -1> 2024-04-26T11:48:42.734+0000 7fc828200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc828200700 time 2024-04-26T11:48:42.734684+0000 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc833a333dd] 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: clone() 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 0> 2024-04-26T11:48:42.737+0000 7fc828200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: in thread 7fc828200700 thread_name:ms_dispatch 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc8327bbd20] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: gsignal() 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: abort() 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc833a33437] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 16: clone() 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: -1> 2024-04-26T11:48:42.734+0000 7fc828200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc828200700 time 2024-04-26T11:48:42.734684+0000 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc833a333dd] 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: clone() 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 0> 2024-04-26T11:48:42.737+0000 7fc828200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: in thread 7fc828200700 thread_name:ms_dispatch 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc8327bbd20] 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2: gsignal() 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 3: abort() 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc833a33437] 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fc833a335a3] 2024-04-26T11:48:43.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x563b6d030822] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x563b6cfa0598] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563b6cfa2c17] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563b6ce9bd0b] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563b6ce9c2ac] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x563b6ce84fad] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fc833cc7e20] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 13: (DispatchQueue::entry()+0x51f) [0x7fc833cc4fdf] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc833d97fc1] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc8327b11ca] 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 16: clone() 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[67407]: 2024-04-26T11:48:43.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:42 smithi033 podman[95230]: 2024-04-26 11:48:42.863188096 +0000 UTC m=+0.036036263 container died 09e1ce08f377bb1f5748908391a920a0fed9730559d97ad8f60dfa01c74be379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T11:48:43.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:43 smithi033 podman[95230]: 2024-04-26 11:48:43.550478384 +0000 UTC m=+0.723326593 container remove 09e1ce08f377bb1f5748908391a920a0fed9730559d97ad8f60dfa01c74be379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T11:48:43.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:43 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:48:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:43 smithi033 ceph-mon[27845]: pgmap v627: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-26T11:48:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:43 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:43 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:active 2024-04-26T11:48:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:43 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:43 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:48:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:43 smithi194 ceph-mon[33017]: pgmap v627: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-26T11:48:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:43 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:43 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:active 2024-04-26T11:48:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:43 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:43 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:48:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:43 smithi096 ceph-mon[32753]: pgmap v627: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-26T11:48:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:43 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:48:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:43 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4017784507,v1:172.21.15.33:6835/4017784507] up:active 2024-04-26T11:48:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:43 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:43 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:48:44.426 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:44 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-26T11:48:44.426 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:44 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.740s CPU time. 2024-04-26T11:48:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:45 smithi194 ceph-mon[33017]: pgmap v628: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:48:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:45 smithi033 ceph-mon[27845]: pgmap v628: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:48:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:45 smithi096 ceph-mon[32753]: pgmap v628: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:48:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:47.457+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:47.457+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:48:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:48:47.457+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:48:47.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:47 smithi096 ceph-mon[32753]: pgmap v629: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 17 KiB/s wr, 16 op/s 2024-04-26T11:48:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:47 smithi194 ceph-mon[33017]: pgmap v629: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 17 KiB/s wr, 16 op/s 2024-04-26T11:48:48.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:47 smithi033 ceph-mon[27845]: pgmap v629: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 17 KiB/s wr, 16 op/s 2024-04-26T11:48:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:49 smithi194 ceph-mon[33017]: pgmap v630: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-26T11:48:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:49 smithi033 ceph-mon[27845]: pgmap v630: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-26T11:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:49 smithi096 ceph-mon[32753]: pgmap v630: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-26T11:48:51.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:51 smithi033 ceph-mon[27845]: pgmap v631: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-26T11:48:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:51 smithi194 ceph-mon[33017]: pgmap v631: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-26T11:48:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:51 smithi096 ceph-mon[32753]: pgmap v631: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-26T11:48:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:53 smithi194 ceph-mon[33017]: pgmap v632: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-26T11:48:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:53 smithi033 ceph-mon[27845]: pgmap v632: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-26T11:48:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:53 smithi096 ceph-mon[32753]: pgmap v632: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-26T11:48:54.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-04-26T11:48:54.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 systemd[1]: Stopped Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:54.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.740s CPU time. 2024-04-26T11:48:54.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 systemd[1]: Starting Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:48:54.900 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 podman[95324]: 2024-04-26 11:48:54.502443332 +0000 UTC m=+0.038272437 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:48:54.900 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 podman[95324]: 2024-04-26 11:48:54.609495686 +0000 UTC m=+0.145324818 container create e3cef9a640100ed08096254e149bab588d597a675997636d226e52744907ebf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T11:48:54.900 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 podman[95324]: 2024-04-26 11:48:54.898288088 +0000 UTC m=+0.434117274 container init e3cef9a640100ed08096254e149bab588d597a675997636d226e52744907ebf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T11:48:55.176 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 podman[95324]: 2024-04-26 11:48:54.910378359 +0000 UTC m=+0.446207461 container start e3cef9a640100ed08096254e149bab588d597a675997636d226e52744907ebf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T11:48:55.176 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 ceph-mds[95343]: -- 172.21.15.33:0/1567131801 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6d7455a40 con 0x55c6d7428c00 2024-04-26T11:48:55.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: starting mds.j at 2024-04-26T11:48:55.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:54 smithi033 bash[95324]: e3cef9a640100ed08096254e149bab588d597a675997636d226e52744907ebf0 2024-04-26T11:48:55.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:48:55 smithi033 systemd[1]: Started Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: pgmap v633: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: daemon mds.j restarted 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: osdmap e159: 12 total, 12 up, 12 in 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:boot 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:55 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: pgmap v633: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-26T11:48:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: daemon mds.j restarted 2024-04-26T11:48:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:48:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: osdmap e159: 12 total, 12 up, 12 in 2024-04-26T11:48:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:boot 2024-04-26T11:48:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:48:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:48:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:55 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: pgmap v633: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-26T11:48:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: daemon mds.j restarted 2024-04-26T11:48:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:48:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:48:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: osdmap e159: 12 total, 12 up, 12 in 2024-04-26T11:48:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:boot 2024-04-26T11:48:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:48:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:48:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:55 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:57.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:56 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:resolve 2024-04-26T11:48:57.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:56 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:56 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:resolve 2024-04-26T11:48:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:56 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:56 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:resolve 2024-04-26T11:48:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:56 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:58 smithi194 ceph-mon[33017]: pgmap v635: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T11:48:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:58 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:48:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:reconnect 2024-04-26T11:48:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:58 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:58 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:48:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:58 smithi033 ceph-mon[27845]: pgmap v635: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T11:48:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:58 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:48:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:reconnect 2024-04-26T11:48:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:58 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:58 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:48:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:58 smithi096 ceph-mon[32753]: pgmap v635: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T11:48:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:58 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00399994 2024-04-26T11:48:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:reconnect 2024-04-26T11:48:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:58 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:58 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:48:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:rejoin 2024-04-26T11:48:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:59 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:59 smithi194 ceph-mon[33017]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:48:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:48:59 smithi194 ceph-mon[33017]: pgmap v636: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T11:48:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:rejoin 2024-04-26T11:48:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-mon[27845]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:48:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-mon[27845]: pgmap v636: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f86a1600700 time 2024-04-26T11:48:59.070231+0000 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86acff03dd] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.428 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.429 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.429 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.429 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.429 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: clone() 2024-04-26T11:48:59.429 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.072+0000 7f86a1600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f86a1600700 time 2024-04-26T11:48:59.070231+0000 2024-04-26T11:48:59.429 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:59.429 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.430 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.430 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86acff03dd] 2024-04-26T11:48:59.430 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.430 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.430 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: clone() 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: *** Caught signal (Aborted) ** 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: in thread 7f86a1600700 thread_name:ms_dispatch 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.431 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86abd78d20] 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: gsignal() 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: abort() 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86acff0437] 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.432 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:rejoin 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:59 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:59 smithi096 ceph-mon[32753]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:48:59 smithi096 ceph-mon[32753]: pgmap v636: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.433 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 16: clone() 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.076+0000 7f86a1600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: in thread 7f86a1600700 thread_name:ms_dispatch 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86abd78d20] 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: gsignal() 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: abort() 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86acff0437] 2024-04-26T11:48:59.434 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 16: clone() 2024-04-26T11:48:59.439 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:59.440 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.440 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: -1> 2024-04-26T11:48:59.072+0000 7f86a1600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f86a1600700 time 2024-04-26T11:48:59.070231+0000 2024-04-26T11:48:59.440 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:59.440 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86acff03dd] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.441 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: clone() 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 0> 2024-04-26T11:48:59.076+0000 7f86a1600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: in thread 7f86a1600700 thread_name:ms_dispatch 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86abd78d20] 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: gsignal() 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: abort() 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86acff0437] 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.442 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 16: clone() 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: -1> 2024-04-26T11:48:59.072+0000 7f86a1600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f86a1600700 time 2024-04-26T11:48:59.070231+0000 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:48:59.443 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86acff03dd] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.444 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: clone() 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 0> 2024-04-26T11:48:59.076+0000 7f86a1600700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: in thread 7f86a1600700 thread_name:ms_dispatch 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86abd78d20] 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2: gsignal() 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 3: abort() 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86acff0437] 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f86acff05a3] 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560379c07822] 2024-04-26T11:48:59.445 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560379b77598] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560379b79c17] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560379a72d0b] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560379a732ac] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560379a5bfad] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f86ad284e20] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 13: (DispatchQueue::entry()+0x51f) [0x7f86ad281fdf] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f86ad354fc1] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86abd6e1ca] 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 16: clone() 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95191]: 2024-04-26T11:48:59.446 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 podman[95377]: 2024-04-26 11:48:59.230655899 +0000 UTC m=+0.057265670 container died 35a5e91ebbf5ac78b790e66391e03169d6a643d364461e2b548bbb8f46d74134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:49:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:00 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:00 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:active 2024-04-26T11:49:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:00 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:00 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:00 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:00 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:active 2024-04-26T11:49:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:00 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:00 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:00.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 podman[95377]: 2024-04-26 11:48:59.968330474 +0000 UTC m=+0.794940267 container remove 35a5e91ebbf5ac78b790e66391e03169d6a643d364461e2b548bbb8f46d74134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T11:49:00.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:48:59 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:49:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:00 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:00 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2612651701,v1:172.21.15.33:6837/2612651701] up:active 2024-04-26T11:49:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:00 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:00 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:00.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:00 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-26T11:49:00.926 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:00 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.193s CPU time. 2024-04-26T11:49:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:01 smithi194 ceph-mon[33017]: pgmap v637: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:01 smithi033 ceph-mon[27845]: pgmap v637: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:01 smithi096 ceph-mon[32753]: pgmap v637: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:02.468+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:02.468+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:49:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:02.468+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:02.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:49:02 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:49:02.493+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:49:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:03 smithi194 ceph-mon[33017]: pgmap v638: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:03 smithi033 ceph-mon[27845]: pgmap v638: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:03 smithi096 ceph-mon[32753]: pgmap v638: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:05 smithi194 ceph-mon[33017]: pgmap v639: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:05 smithi033 ceph-mon[27845]: pgmap v639: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:05 smithi096 ceph-mon[32753]: pgmap v639: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-04-26T11:49:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:07 smithi194 ceph-mon[33017]: pgmap v640: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-04-26T11:49:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:49:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:07 smithi033 ceph-mon[27845]: pgmap v640: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-04-26T11:49:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:49:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:07 smithi096 ceph-mon[32753]: pgmap v640: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-04-26T11:49:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:49:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:09 smithi194 ceph-mon[33017]: pgmap v641: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-04-26T11:49:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:49:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:49:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:09 smithi194 ceph-mon[33017]: pgmap v642: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-04-26T11:49:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:09 smithi194 ceph-mon[33017]: pgmap v643: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 239 B/s wr, 6 op/s 2024-04-26T11:49:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:09 smithi033 ceph-mon[27845]: pgmap v641: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-04-26T11:49:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:49:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:49:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:09 smithi033 ceph-mon[27845]: pgmap v642: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-04-26T11:49:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:09 smithi033 ceph-mon[27845]: pgmap v643: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 239 B/s wr, 6 op/s 2024-04-26T11:49:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:09 smithi096 ceph-mon[32753]: pgmap v641: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-04-26T11:49:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:49:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:49:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:09 smithi096 ceph-mon[32753]: pgmap v642: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-04-26T11:49:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:09 smithi096 ceph-mon[32753]: pgmap v643: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 239 B/s wr, 6 op/s 2024-04-26T11:49:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:49:10.871 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:49:10.871 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:10 smithi033 systemd[1]: Stopped Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:49:10.871 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.193s CPU time. 2024-04-26T11:49:10.871 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:10 smithi033 systemd[1]: Starting Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:49:11.173 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:10 smithi033 podman[95690]: 2024-04-26 11:49:10.859512622 +0000 UTC m=+0.033182654 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:49:11.174 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:10 smithi033 podman[95690]: 2024-04-26 11:49:10.977852586 +0000 UTC m=+0.151522679 container create 021a59fcd17d23a6032bab9861d9ab452a65764a3bce7978d281f787008927d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T11:49:11.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:11 smithi033 podman[95690]: 2024-04-26 11:49:11.264981789 +0000 UTC m=+0.438651865 container init 021a59fcd17d23a6032bab9861d9ab452a65764a3bce7978d281f787008927d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:49:11.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:11 smithi033 podman[95690]: 2024-04-26 11:49:11.275677372 +0000 UTC m=+0.449347422 container start 021a59fcd17d23a6032bab9861d9ab452a65764a3bce7978d281f787008927d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:49:11.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mds[95717]: -- 172.21.15.33:0/1134385595 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562e60337a40 con 0x562e6030b000 2024-04-26T11:49:11.677 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: starting mds.g at 2024-04-26T11:49:11.677 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:11 smithi033 bash[95690]: 021a59fcd17d23a6032bab9861d9ab452a65764a3bce7978d281f787008927d4 2024-04-26T11:49:11.677 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:11 smithi033 systemd[1]: Started Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: pgmap v644: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 B/s wr, 9 op/s 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: daemon mds.g restarted 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: osdmap e160: 12 total, 12 up, 12 in 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:boot 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:11 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: pgmap v644: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 B/s wr, 9 op/s 2024-04-26T11:49:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: daemon mds.g restarted 2024-04-26T11:49:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: osdmap e160: 12 total, 12 up, 12 in 2024-04-26T11:49:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:boot 2024-04-26T11:49:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:49:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:49:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:11 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: pgmap v644: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 B/s wr, 9 op/s 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: daemon mds.g restarted 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: osdmap e160: 12 total, 12 up, 12 in 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:boot 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:49:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:11 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:14.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:13 smithi033 ceph-mon[27845]: pgmap v646: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:49:14.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:13 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:resolve 2024-04-26T11:49:14.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:13 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:13 smithi194 ceph-mon[33017]: pgmap v646: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:49:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:13 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:resolve 2024-04-26T11:49:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:13 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:13 smithi096 ceph-mon[32753]: pgmap v646: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:49:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:13 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:resolve 2024-04-26T11:49:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:13 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:14 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:49:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:14 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:reconnect 2024-04-26T11:49:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:14 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:14 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:49:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:14 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:49:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:14 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:reconnect 2024-04-26T11:49:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:14 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:14 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:49:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:14 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:49:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:14 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:reconnect 2024-04-26T11:49:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:14 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:14 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:49:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:15 smithi194 ceph-mon[33017]: pgmap v647: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-26T11:49:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:15 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:rejoin 2024-04-26T11:49:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:15 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:15 smithi194 ceph-mon[33017]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:49:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-mon[27845]: pgmap v647: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-26T11:49:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:rejoin 2024-04-26T11:49:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-mon[27845]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:49:16.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f01d1200700 time 2024-04-26T11:49:15.827984+0000 2024-04-26T11:49:16.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01dcb013dd] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: clone() 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:15.829+0000 7f01d1200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f01d1200700 time 2024-04-26T11:49:15.827984+0000 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01dcb013dd] 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.179 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: clone() 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: *** Caught signal (Aborted) ** 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: in thread 7f01d1200700 thread_name:ms_dispatch 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.180 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01db889d20] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: gsignal() 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: abort() 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01dcb01437] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.181 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 16: clone() 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:15.832+0000 7f01d1200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: in thread 7f01d1200700 thread_name:ms_dispatch 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01db889d20] 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: gsignal() 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: abort() 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01dcb01437] 2024-04-26T11:49:16.182 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 16: clone() 2024-04-26T11:49:16.183 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:15 smithi096 ceph-mon[32753]: pgmap v647: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:15 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:rejoin 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:15 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:15 smithi096 ceph-mon[32753]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: -1> 2024-04-26T11:49:15.829+0000 7f01d1200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f01d1200700 time 2024-04-26T11:49:15.827984+0000 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.184 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01dcb013dd] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: clone() 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.185 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 0> 2024-04-26T11:49:15.832+0000 7f01d1200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: in thread 7f01d1200700 thread_name:ms_dispatch 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01db889d20] 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: gsignal() 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: abort() 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01dcb01437] 2024-04-26T11:49:16.186 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 16: clone() 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: -1> 2024-04-26T11:49:15.829+0000 7f01d1200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f01d1200700 time 2024-04-26T11:49:15.827984+0000 2024-04-26T11:49:16.187 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:16.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01dcb013dd] 2024-04-26T11:49:16.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.188 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: clone() 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 0> 2024-04-26T11:49:15.832+0000 7f01d1200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:16.189 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: in thread 7f01d1200700 thread_name:ms_dispatch 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01db889d20] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2: gsignal() 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 3: abort() 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01dcb01437] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01dcb015a3] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55c69d822822] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55c69d792598] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c69d794c17] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c69d68dd0b] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c69d68e2ac] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55c69d676fad] 2024-04-26T11:49:16.190 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f01dcd95e20] 2024-04-26T11:49:16.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 13: (DispatchQueue::entry()+0x51f) [0x7f01dcd92fdf] 2024-04-26T11:49:16.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f01dce65fc1] 2024-04-26T11:49:16.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f01db87f1ca] 2024-04-26T11:49:16.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 16: clone() 2024-04-26T11:49:16.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:16.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95338]: 2024-04-26T11:49:16.191 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:15 smithi033 podman[95753]: 2024-04-26 11:49:15.981774655 +0000 UTC m=+0.035180043 container died e3cef9a640100ed08096254e149bab588d597a675997636d226e52744907ebf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T11:49:17.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:16 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:17.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:16 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:active 2024-04-26T11:49:17.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:16 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:17.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:16 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:16 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:16 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:active 2024-04-26T11:49:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:16 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:16 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:16 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:16 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/273697747,v1:172.21.15.33:6835/273697747] up:active 2024-04-26T11:49:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:16 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:16 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:17.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:16 smithi033 podman[95753]: 2024-04-26 11:49:16.744520828 +0000 UTC m=+0.797926257 container remove e3cef9a640100ed08096254e149bab588d597a675997636d226e52744907ebf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T11:49:17.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:16 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:49:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:17.484+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:17.484+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:17.484+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:49:17.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:17 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-26T11:49:17.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:17 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.185s CPU time. 2024-04-26T11:49:17.806 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:49:17 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:49:17.494+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:49:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:17 smithi194 ceph-mon[33017]: pgmap v648: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-26T11:49:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:17 smithi033 ceph-mon[27845]: pgmap v648: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-26T11:49:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:17 smithi096 ceph-mon[32753]: pgmap v648: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-26T11:49:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:19 smithi194 ceph-mon[33017]: pgmap v649: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:49:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:19 smithi033 ceph-mon[27845]: pgmap v649: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:49:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:19 smithi096 ceph-mon[32753]: pgmap v649: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T11:49:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:21 smithi194 ceph-mon[33017]: pgmap v650: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 46 KiB/s wr, 29 op/s 2024-04-26T11:49:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:21 smithi033 ceph-mon[27845]: pgmap v650: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 46 KiB/s wr, 29 op/s 2024-04-26T11:49:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:21 smithi096 ceph-mon[32753]: pgmap v650: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 46 KiB/s wr, 29 op/s 2024-04-26T11:49:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:23 smithi194 ceph-mon[33017]: pgmap v651: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 40 KiB/s wr, 26 op/s 2024-04-26T11:49:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:23 smithi033 ceph-mon[27845]: pgmap v651: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 40 KiB/s wr, 26 op/s 2024-04-26T11:49:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:23 smithi096 ceph-mon[32753]: pgmap v651: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 40 KiB/s wr, 26 op/s 2024-04-26T11:49:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:25 smithi194 ceph-mon[33017]: pgmap v652: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 38 KiB/s wr, 27 op/s 2024-04-26T11:49:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:25 smithi033 ceph-mon[27845]: pgmap v652: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 38 KiB/s wr, 27 op/s 2024-04-26T11:49:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:25 smithi096 ceph-mon[32753]: pgmap v652: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 38 KiB/s wr, 27 op/s 2024-04-26T11:49:27.834 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-04-26T11:49:27.834 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:27 smithi033 systemd[1]: Stopped Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:49:27.834 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.185s CPU time. 2024-04-26T11:49:27.834 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:27 smithi033 systemd[1]: Starting Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:49:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:27 smithi194 ceph-mon[33017]: pgmap v653: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 38 KiB/s wr, 24 op/s 2024-04-26T11:49:28.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:27 smithi033 ceph-mon[27845]: pgmap v653: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 38 KiB/s wr, 24 op/s 2024-04-26T11:49:28.151 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:27 smithi033 podman[95841]: 2024-04-26 11:49:27.741732182 +0000 UTC m=+0.035825489 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:49:28.151 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:27 smithi033 podman[95841]: 2024-04-26 11:49:27.870408692 +0000 UTC m=+0.164502032 container create 64cd5176b9af9dcc9148822aea8af061c08ea6d126d77aa21e036430a2e4a54c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:49:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:27 smithi096 ceph-mon[32753]: pgmap v653: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 38 KiB/s wr, 24 op/s 2024-04-26T11:49:28.426 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:28 smithi033 podman[95841]: 2024-04-26 11:49:28.149724741 +0000 UTC m=+0.443818087 container init 64cd5176b9af9dcc9148822aea8af061c08ea6d126d77aa21e036430a2e4a54c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T11:49:28.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:28 smithi033 podman[95841]: 2024-04-26 11:49:28.160156402 +0000 UTC m=+0.454249715 container start 64cd5176b9af9dcc9148822aea8af061c08ea6d126d77aa21e036430a2e4a54c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T11:49:28.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mds[95859]: -- 172.21.15.33:0/3807267778 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5652e3d51a40 con 0x5652e3d24800 2024-04-26T11:49:28.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: starting mds.j at 2024-04-26T11:49:28.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:28 smithi033 bash[95841]: 64cd5176b9af9dcc9148822aea8af061c08ea6d126d77aa21e036430a2e4a54c 2024-04-26T11:49:28.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:28 smithi033 systemd[1]: Started Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: daemon mds.j restarted 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: osdmap e161: 12 total, 12 up, 12 in 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:boot 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:28 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: daemon mds.j restarted 2024-04-26T11:49:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: osdmap e161: 12 total, 12 up, 12 in 2024-04-26T11:49:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:boot 2024-04-26T11:49:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:49:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:49:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:28 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: daemon mds.j restarted 2024-04-26T11:49:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: osdmap e161: 12 total, 12 up, 12 in 2024-04-26T11:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:boot 2024-04-26T11:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:28 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:29 smithi194 ceph-mon[33017]: pgmap v655: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 46 KiB/s wr, 29 op/s 2024-04-26T11:49:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:29 smithi033 ceph-mon[27845]: pgmap v655: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 46 KiB/s wr, 29 op/s 2024-04-26T11:49:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:29 smithi096 ceph-mon[32753]: pgmap v655: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 46 KiB/s wr, 29 op/s 2024-04-26T11:49:31.114 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:30 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:resolve 2024-04-26T11:49:31.114 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:30 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:30 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:resolve 2024-04-26T11:49:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:30 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:30 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:resolve 2024-04-26T11:49:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:30 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:31 smithi033 ceph-mon[27845]: pgmap v656: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-26T11:49:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:31 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:49:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:31 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:reconnect 2024-04-26T11:49:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:31 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:31 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:49:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:31 smithi096 ceph-mon[32753]: pgmap v656: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-26T11:49:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:31 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:49:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:31 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:reconnect 2024-04-26T11:49:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:31 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:31 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:49:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:31 smithi194 ceph-mon[33017]: pgmap v656: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-26T11:49:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:31 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:49:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:31 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:reconnect 2024-04-26T11:49:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:31 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:31 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:49:32.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:32.498+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:32.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:32.498+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:32.498+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:49:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:rejoin 2024-04-26T11:49:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-mon[27845]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:49:33.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cb7000700 time 2024-04-26T11:49:32.928702+0000 2024-04-26T11:49:33.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:33.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7cc28833dd] 2024-04-26T11:49:33.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.177 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: clone() 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:32.928+0000 7f7cb7000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cb7000700 time 2024-04-26T11:49:32.928702+0000 2024-04-26T11:49:33.178 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:33.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.179 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7cc28833dd] 2024-04-26T11:49:33.180 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.180 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.180 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.180 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.180 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.180 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.181 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: clone() 2024-04-26T11:49:33.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: *** Caught signal (Aborted) ** 2024-04-26T11:49:33.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: in thread 7f7cb7000700 thread_name:ms_dispatch 2024-04-26T11:49:33.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cc160bd20] 2024-04-26T11:49:33.182 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: gsignal() 2024-04-26T11:49:33.183 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: abort() 2024-04-26T11:49:33.183 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7cc2883437] 2024-04-26T11:49:33.183 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.183 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.183 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.183 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.183 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:32 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:rejoin 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:32 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:32 smithi096 ceph-mon[32753]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 16: clone() 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:32.931+0000 7f7cb7000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: in thread 7f7cb7000700 thread_name:ms_dispatch 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cc160bd20] 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: gsignal() 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: abort() 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7cc2883437] 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.185 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 16: clone() 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: -1> 2024-04-26T11:49:32.928+0000 7f7cb7000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cb7000700 time 2024-04-26T11:49:32.928702+0000 2024-04-26T11:49:33.186 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7cc28833dd] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.187 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: clone() 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 0> 2024-04-26T11:49:32.931+0000 7f7cb7000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: in thread 7f7cb7000700 thread_name:ms_dispatch 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cc160bd20] 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: gsignal() 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: abort() 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7cc2883437] 2024-04-26T11:49:33.188 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 16: clone() 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:33.189 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: -1> 2024-04-26T11:49:32.928+0000 7f7cb7000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cb7000700 time 2024-04-26T11:49:32.928702+0000 2024-04-26T11:49:33.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:33.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7cc28833dd] 2024-04-26T11:49:33.190 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.191 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: clone() 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 0> 2024-04-26T11:49:32.931+0000 7f7cb7000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: in thread 7f7cb7000700 thread_name:ms_dispatch 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cc160bd20] 2024-04-26T11:49:33.192 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2: gsignal() 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 3: abort() 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7cc2883437] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7cc28835a3] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x562e3e43a822] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x562e3e3aa598] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562e3e3acc17] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562e3e2a5d0b] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562e3e2a62ac] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x562e3e28efad] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f7cc2b17e20] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 13: (DispatchQueue::entry()+0x51f) [0x7f7cc2b14fdf] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7cc2be7fc1] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cc16011ca] 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 16: clone() 2024-04-26T11:49:33.193 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:33.194 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:32 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95711]: 2024-04-26T11:49:33.194 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:33 smithi033 podman[95893]: 2024-04-26 11:49:33.06646756 +0000 UTC m=+0.037946049 container died 021a59fcd17d23a6032bab9861d9ab452a65764a3bce7978d281f787008927d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:49:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:32 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:rejoin 2024-04-26T11:49:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:32 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:32 smithi194 ceph-mon[33017]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:49:34.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:33 smithi033 ceph-mon[27845]: pgmap v657: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:49:34.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:33 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:34.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:33 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:active 2024-04-26T11:49:34.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:33 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:34.129 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:33 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:34.129 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:33 smithi033 podman[95893]: 2024-04-26 11:49:33.720159521 +0000 UTC m=+0.691638053 container remove 021a59fcd17d23a6032bab9861d9ab452a65764a3bce7978d281f787008927d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0) 2024-04-26T11:49:34.129 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:33 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:49:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:33 smithi096 ceph-mon[32753]: pgmap v657: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:49:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:33 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:33 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:active 2024-04-26T11:49:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:33 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:33 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:33 smithi194 ceph-mon[33017]: pgmap v657: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:49:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:33 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:33 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/1581758099,v1:172.21.15.33:6837/1581758099] up:active 2024-04-26T11:49:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:33 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:33 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:34.392 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:34 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-26T11:49:34.392 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:34 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.154s CPU time. 2024-04-26T11:49:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:35 smithi033 ceph-mon[27845]: pgmap v658: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-26T11:49:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:35 smithi096 ceph-mon[32753]: pgmap v658: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-26T11:49:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:35 smithi194 ceph-mon[33017]: pgmap v658: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-26T11:49:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:36.785+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0005b50 con 0x7fef00077780 2024-04-26T11:49:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:36.787+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0005b50 con 0x7fef00077780 2024-04-26T11:49:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:36.792+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc005ae0 con 0x564385821900 2024-04-26T11:49:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:36.792+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc005ae0 con 0x564385821900 2024-04-26T11:49:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:37 smithi033 ceph-mon[27845]: pgmap v659: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 38 KiB/s wr, 15 op/s 2024-04-26T11:49:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:37 smithi096 ceph-mon[32753]: pgmap v659: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 38 KiB/s wr, 15 op/s 2024-04-26T11:49:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:37 smithi194 ceph-mon[33017]: pgmap v659: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 38 KiB/s wr, 15 op/s 2024-04-26T11:49:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:39 smithi033 ceph-mon[27845]: pgmap v660: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 36 KiB/s wr, 14 op/s 2024-04-26T11:49:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:39 smithi096 ceph-mon[32753]: pgmap v660: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 36 KiB/s wr, 14 op/s 2024-04-26T11:49:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:39 smithi194 ceph-mon[33017]: pgmap v660: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 36 KiB/s wr, 14 op/s 2024-04-26T11:49:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:41 smithi033 ceph-mon[27845]: pgmap v661: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-04-26T11:49:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:41 smithi096 ceph-mon[32753]: pgmap v661: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-04-26T11:49:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:41 smithi194 ceph-mon[33017]: pgmap v661: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 32 KiB/s wr, 15 op/s 2024-04-26T11:49:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:43 smithi096 ceph-mon[32753]: pgmap v662: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-26T11:49:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:43 smithi096 ceph-mon[32753]: osdmap e162: 12 total, 12 up, 12 in 2024-04-26T11:49:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:43 smithi194 ceph-mon[33017]: pgmap v662: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-26T11:49:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:43 smithi194 ceph-mon[33017]: osdmap e162: 12 total, 12 up, 12 in 2024-04-26T11:49:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:43 smithi033 ceph-mon[27845]: pgmap v662: 97 pgs: 97 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-26T11:49:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:43 smithi033 ceph-mon[27845]: osdmap e162: 12 total, 12 up, 12 in 2024-04-26T11:49:44.844 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:44 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:49:44.844 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:44 smithi033 systemd[1]: Stopped Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:49:44.844 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:44 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.154s CPU time. 2024-04-26T11:49:44.845 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:44 smithi033 systemd[1]: Starting Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:49:45.168 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:44 smithi033 podman[95983]: 2024-04-26 11:49:44.748147504 +0000 UTC m=+0.032929520 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:49:45.168 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:44 smithi033 podman[95983]: 2024-04-26 11:49:44.887772336 +0000 UTC m=+0.172554419 container create ea1da4ad8a0cc2c58000c716590ecc3722fbcadda7274be7edba919d729d7505 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T11:49:45.168 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:45 smithi033 podman[95983]: 2024-04-26 11:49:45.166970639 +0000 UTC m=+0.451752699 container init ea1da4ad8a0cc2c58000c716590ecc3722fbcadda7274be7edba919d729d7505 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-26T11:49:45.426 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:45 smithi033 podman[95983]: 2024-04-26 11:49:45.177920492 +0000 UTC m=+0.462702519 container start ea1da4ad8a0cc2c58000c716590ecc3722fbcadda7274be7edba919d729d7505 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T11:49:45.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mds[96001]: -- 172.21.15.33:0/4176859029 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560dc6be9a40 con 0x560dc6bbc800 2024-04-26T11:49:45.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: starting mds.g at 2024-04-26T11:49:45.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:45 smithi033 bash[95983]: ea1da4ad8a0cc2c58000c716590ecc3722fbcadda7274be7edba919d729d7505 2024-04-26T11:49:45.427 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:49:45 smithi033 systemd[1]: Started Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:49:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: pgmap v664: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:49:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: daemon mds.g restarted 2024-04-26T11:49:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: osdmap e163: 12 total, 12 up, 12 in 2024-04-26T11:49:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:boot 2024-04-26T11:49:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:49:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:46.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:49:46.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:45 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: pgmap v664: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:49:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: daemon mds.g restarted 2024-04-26T11:49:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: osdmap e163: 12 total, 12 up, 12 in 2024-04-26T11:49:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:boot 2024-04-26T11:49:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:49:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:49:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:45 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: pgmap v664: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: daemon mds.g restarted 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: osdmap e163: 12 total, 12 up, 12 in 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:boot 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T11:49:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:45 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:47.514+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:47.514+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:49:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:49:47.514+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:49:48.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:47 smithi033 ceph-mon[27845]: pgmap v666: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:49:48.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:47 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:resolve 2024-04-26T11:49:48.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:47 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:48.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:47 smithi096 ceph-mon[32753]: pgmap v666: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:49:48.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:47 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:resolve 2024-04-26T11:49:48.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:47 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:47 smithi194 ceph-mon[33017]: pgmap v666: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:49:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:47 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:resolve 2024-04-26T11:49:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:47 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:48 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:49:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:48 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:reconnect 2024-04-26T11:49:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:48 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:48 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:48 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:48 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:reconnect 2024-04-26T11:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:48 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:48 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:48 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:48 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:reconnect 2024-04-26T11:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:48 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:48 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:49:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:49 smithi194 ceph-mon[33017]: pgmap v667: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T11:49:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:49 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:rejoin 2024-04-26T11:49:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:49 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:49 smithi194 ceph-mon[33017]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:49:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:49 smithi033 ceph-mon[27845]: pgmap v667: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T11:49:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:49 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:rejoin 2024-04-26T11:49:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:49 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:49 smithi033 ceph-mon[27845]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:49:50.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5c3000700 time 2024-04-26T11:49:50.023324+0000 2024-04-26T11:49:50.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:50.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5ce9e83dd] 2024-04-26T11:49:50.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: clone() 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.023+0000 7fa5c3000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5c3000700 time 2024-04-26T11:49:50.023324+0000 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:50.428 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5ce9e83dd] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.429 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: clone() 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: *** Caught signal (Aborted) ** 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: in thread 7fa5c3000700 thread_name:ms_dispatch 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5cd770d20] 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: gsignal() 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: abort() 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5ce9e8437] 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.431 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 16: clone() 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.026+0000 7fa5c3000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: in thread 7fa5c3000700 thread_name:ms_dispatch 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5cd770d20] 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: gsignal() 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: abort() 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5ce9e8437] 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.432 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.433 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.433 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.433 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.433 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.433 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.433 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 16: clone() 2024-04-26T11:49:50.433 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:49 smithi096 ceph-mon[32753]: pgmap v667: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:49 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:rejoin 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:49 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:49 smithi096 ceph-mon[32753]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: -1> 2024-04-26T11:49:50.023+0000 7fa5c3000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5c3000700 time 2024-04-26T11:49:50.023324+0000 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.434 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5ce9e83dd] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.435 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: clone() 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 0> 2024-04-26T11:49:50.026+0000 7fa5c3000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: in thread 7fa5c3000700 thread_name:ms_dispatch 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5cd770d20] 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: gsignal() 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: abort() 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5ce9e8437] 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.436 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 16: clone() 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: -1> 2024-04-26T11:49:50.023+0000 7fa5c3000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa5c3000700 time 2024-04-26T11:49:50.023324+0000 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.437 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa5ce9e83dd] 2024-04-26T11:49:50.438 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.438 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.438 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.438 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.438 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.438 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.438 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: clone() 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 0> 2024-04-26T11:49:50.026+0000 7fa5c3000700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: in thread 7fa5c3000700 thread_name:ms_dispatch 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa5cd770d20] 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2: gsignal() 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 3: abort() 2024-04-26T11:49:50.439 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa5ce9e8437] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa5ce9e85a3] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x5652c0a62822] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x5652c09d2598] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5652c09d4c17] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5652c08cdd0b] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5652c08ce2ac] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x5652c08b6fad] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fa5cec7ce20] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 13: (DispatchQueue::entry()+0x51f) [0x7fa5cec79fdf] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa5ced4cfc1] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa5cd7661ca] 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 16: clone() 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:49:50.440 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[95855]: 2024-04-26T11:49:50.441 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:50 smithi033 podman[96035]: 2024-04-26 11:49:50.143239185 +0000 UTC m=+0.033875598 container died 64cd5176b9af9dcc9148822aea8af061c08ea6d126d77aa21e036430a2e4a54c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T11:49:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:active 2024-04-26T11:49:51.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:51.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:50 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:50 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:50 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:active 2024-04-26T11:49:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:50 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:50 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:50 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:49:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:50 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/4104464194,v1:172.21.15.33:6835/4104464194] up:active 2024-04-26T11:49:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:50 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:49:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:50 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:49:51.800 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:51 smithi033 podman[96035]: 2024-04-26 11:49:51.464544383 +0000 UTC m=+1.355180843 container remove 64cd5176b9af9dcc9148822aea8af061c08ea6d126d77aa21e036430a2e4a54c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T11:49:51.800 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:51 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:49:52.176 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:52 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-26T11:49:52.176 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:49:52 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.154s CPU time. 2024-04-26T11:49:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:51 smithi033 ceph-mon[27845]: pgmap v668: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:49:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:51 smithi033 ceph-mon[27845]: osdmap e164: 12 total, 12 up, 12 in 2024-04-26T11:49:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:51 smithi194 ceph-mon[33017]: pgmap v668: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:49:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:51 smithi194 ceph-mon[33017]: osdmap e164: 12 total, 12 up, 12 in 2024-04-26T11:49:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:51 smithi096 ceph-mon[32753]: pgmap v668: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:49:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:51 smithi096 ceph-mon[32753]: osdmap e164: 12 total, 12 up, 12 in 2024-04-26T11:49:52.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:49:52 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:49:52.496+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:49:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:54 smithi194 ceph-mon[33017]: pgmap v670: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-26T11:49:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:54 smithi033 ceph-mon[27845]: pgmap v670: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-26T11:49:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:54 smithi096 ceph-mon[32753]: pgmap v670: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-26T11:49:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:56 smithi194 ceph-mon[33017]: pgmap v671: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-26T11:49:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:56 smithi033 ceph-mon[27845]: pgmap v671: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-26T11:49:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:56 smithi096 ceph-mon[32753]: pgmap v671: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-26T11:49:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:57 smithi033 ceph-mon[27845]: pgmap v672: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-26T11:49:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:57 smithi096 ceph-mon[32753]: pgmap v672: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-26T11:49:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:57 smithi194 ceph-mon[33017]: pgmap v672: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-26T11:50:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:49:59 smithi194 ceph-mon[33017]: pgmap v673: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-26T11:50:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:49:59 smithi033 ceph-mon[27845]: pgmap v673: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-26T11:50:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:49:59 smithi096 ceph-mon[32753]: pgmap v673: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-26T11:50:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:00 smithi194 ceph-mon[33017]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-26T11:50:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:00 smithi194 ceph-mon[33017]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-26T11:50:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:00 smithi194 ceph-mon[33017]: daemon mds.e on smithi096 is in error state 2024-04-26T11:50:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:00 smithi194 ceph-mon[33017]: daemon mds.h on smithi096 is in error state 2024-04-26T11:50:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:00 smithi033 ceph-mon[27845]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-26T11:50:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:00 smithi033 ceph-mon[27845]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-26T11:50:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:00 smithi033 ceph-mon[27845]: daemon mds.e on smithi096 is in error state 2024-04-26T11:50:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:00 smithi033 ceph-mon[27845]: daemon mds.h on smithi096 is in error state 2024-04-26T11:50:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:00 smithi096 ceph-mon[32753]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-26T11:50:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:00 smithi096 ceph-mon[32753]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-26T11:50:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:00 smithi096 ceph-mon[32753]: daemon mds.e on smithi096 is in error state 2024-04-26T11:50:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:00 smithi096 ceph-mon[32753]: daemon mds.h on smithi096 is in error state 2024-04-26T11:50:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:01 smithi194 ceph-mon[33017]: pgmap v674: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-04-26T11:50:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:01 smithi033 ceph-mon[27845]: pgmap v674: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-04-26T11:50:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:01 smithi096 ceph-mon[32753]: pgmap v674: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-04-26T11:50:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:02.531+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:02.531+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:02.531+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:50:02.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-04-26T11:50:02.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 systemd[1]: Stopped Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:02.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.154s CPU time. 2024-04-26T11:50:02.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 systemd[1]: Starting Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:50:02.895 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 podman[96126]: 2024-04-26 11:50:02.50267952 +0000 UTC m=+0.034619621 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:50:02.896 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 podman[96126]: 2024-04-26 11:50:02.615690418 +0000 UTC m=+0.147630609 container create e74cb2fedb1c8160bcd01587e5fc69fddf6572d42069c2057599983f2c7b52f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:50:03.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 podman[96126]: 2024-04-26 11:50:02.894941115 +0000 UTC m=+0.426881260 container init e74cb2fedb1c8160bcd01587e5fc69fddf6572d42069c2057599983f2c7b52f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T11:50:03.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 podman[96126]: 2024-04-26 11:50:02.905536933 +0000 UTC m=+0.437477098 container start e74cb2fedb1c8160bcd01587e5fc69fddf6572d42069c2057599983f2c7b52f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T11:50:03.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 ceph-mds[96145]: -- 172.21.15.33:0/571784990 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a722ae5a40 con 0x55a722ab9000 2024-04-26T11:50:03.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:02 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: starting mds.j at 2024-04-26T11:50:03.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:03 smithi033 bash[96126]: e74cb2fedb1c8160bcd01587e5fc69fddf6572d42069c2057599983f2c7b52f9 2024-04-26T11:50:03.178 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:03 smithi033 systemd[1]: Started Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: pgmap v675: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-26T11:50:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: daemon mds.j restarted 2024-04-26T11:50:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: osdmap e165: 12 total, 12 up, 12 in 2024-04-26T11:50:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:boot 2024-04-26T11:50:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:50:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:04.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:04.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:03 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: pgmap v675: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-26T11:50:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: daemon mds.j restarted 2024-04-26T11:50:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: osdmap e165: 12 total, 12 up, 12 in 2024-04-26T11:50:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:boot 2024-04-26T11:50:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:50:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:03 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: pgmap v675: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-26T11:50:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: daemon mds.j restarted 2024-04-26T11:50:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: osdmap e165: 12 total, 12 up, 12 in 2024-04-26T11:50:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:boot 2024-04-26T11:50:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-04-26T11:50:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:03 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:05.241 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:04 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:resolve 2024-04-26T11:50:05.241 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:04 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:04 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:resolve 2024-04-26T11:50:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:04 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:04 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:resolve 2024-04-26T11:50:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:04 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:06 smithi194 ceph-mon[33017]: pgmap v677: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:06 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:50:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:06 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:reconnect 2024-04-26T11:50:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:06 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:06 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:50:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-mon[27845]: pgmap v677: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:50:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:reconnect 2024-04-26T11:50:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:06.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:50:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:06 smithi096 ceph-mon[32753]: pgmap v677: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:06 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00199997 2024-04-26T11:50:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:06 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:reconnect 2024-04-26T11:50:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:06 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:06 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:50:06.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e3e200700 time 2024-04-26T11:50:06.667697+0000 2024-04-26T11:50:06.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:06.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e49b243dd] 2024-04-26T11:50:06.928 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.929 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.930 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: clone() 2024-04-26T11:50:06.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.668+0000 7f6e3e200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e3e200700 time 2024-04-26T11:50:06.667697+0000 2024-04-26T11:50:06.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:06.931 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e49b243dd] 2024-04-26T11:50:06.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.932 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.933 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.934 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: clone() 2024-04-26T11:50:06.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: *** Caught signal (Aborted) ** 2024-04-26T11:50:06.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: in thread 7f6e3e200700 thread_name:ms_dispatch 2024-04-26T11:50:06.935 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e488acd20] 2024-04-26T11:50:06.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: gsignal() 2024-04-26T11:50:06.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: abort() 2024-04-26T11:50:06.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e49b24437] 2024-04-26T11:50:06.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.936 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.937 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.938 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 16: clone() 2024-04-26T11:50:06.939 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.671+0000 7f6e3e200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:06.939 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: in thread 7f6e3e200700 thread_name:ms_dispatch 2024-04-26T11:50:06.939 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.939 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.939 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e488acd20] 2024-04-26T11:50:06.939 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: gsignal() 2024-04-26T11:50:06.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: abort() 2024-04-26T11:50:06.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e49b24437] 2024-04-26T11:50:06.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.940 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.941 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 16: clone() 2024-04-26T11:50:06.942 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:06.944 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.944 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: -1> 2024-04-26T11:50:06.668+0000 7f6e3e200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e3e200700 time 2024-04-26T11:50:06.667697+0000 2024-04-26T11:50:06.944 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:06.944 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.945 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.945 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e49b243dd] 2024-04-26T11:50:06.945 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.945 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.945 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.946 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.946 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.946 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.946 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.946 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.947 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.947 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.947 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.947 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: clone() 2024-04-26T11:50:06.947 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.947 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 0> 2024-04-26T11:50:06.671+0000 7f6e3e200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:06.948 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: in thread 7f6e3e200700 thread_name:ms_dispatch 2024-04-26T11:50:06.948 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.948 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.948 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e488acd20] 2024-04-26T11:50:06.948 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: gsignal() 2024-04-26T11:50:06.949 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: abort() 2024-04-26T11:50:06.949 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e49b24437] 2024-04-26T11:50:06.949 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.949 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.949 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.949 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.950 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.950 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.950 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.950 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.950 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.950 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.951 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.951 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 16: clone() 2024-04-26T11:50:06.951 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:06.951 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.951 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: -1> 2024-04-26T11:50:06.668+0000 7f6e3e200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6e3e200700 time 2024-04-26T11:50:06.667697+0000 2024-04-26T11:50:06.952 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:06.952 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.952 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.952 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e49b243dd] 2024-04-26T11:50:06.952 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.952 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.953 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.953 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.953 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.953 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.953 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.953 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.954 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.954 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.954 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.954 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: clone() 2024-04-26T11:50:06.957 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.958 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 0> 2024-04-26T11:50:06.671+0000 7f6e3e200700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:06.958 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: in thread 7f6e3e200700 thread_name:ms_dispatch 2024-04-26T11:50:06.958 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.958 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:06.958 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e488acd20] 2024-04-26T11:50:06.959 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2: gsignal() 2024-04-26T11:50:06.959 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 3: abort() 2024-04-26T11:50:06.959 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e49b24437] 2024-04-26T11:50:06.959 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e49b245a3] 2024-04-26T11:50:06.959 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x560da3eba822] 2024-04-26T11:50:06.959 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x560da3e2a598] 2024-04-26T11:50:06.960 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560da3e2cc17] 2024-04-26T11:50:06.960 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560da3d25d0b] 2024-04-26T11:50:06.960 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560da3d262ac] 2024-04-26T11:50:06.960 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x560da3d0efad] 2024-04-26T11:50:06.960 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f6e49db8e20] 2024-04-26T11:50:06.960 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 13: (DispatchQueue::entry()+0x51f) [0x7f6e49db5fdf] 2024-04-26T11:50:06.961 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6e49e88fc1] 2024-04-26T11:50:06.961 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6e488a21ca] 2024-04-26T11:50:06.961 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 16: clone() 2024-04-26T11:50:06.961 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:06.961 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g[95997]: 2024-04-26T11:50:06.961 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:06 smithi033 podman[96180]: 2024-04-26 11:50:06.828691967 +0000 UTC m=+0.044635423 container died ea1da4ad8a0cc2c58000c716590ecc3722fbcadda7274be7edba919d729d7505 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T11:50:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:07 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:rejoin 2024-04-26T11:50:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:07 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:07 smithi194 ceph-mon[33017]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:50:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:07 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:07 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:active 2024-04-26T11:50:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:07 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:07 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:rejoin 2024-04-26T11:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:07 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:07 smithi033 ceph-mon[27845]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:07 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:07 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:active 2024-04-26T11:50:07.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:07 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:07 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:rejoin 2024-04-26T11:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:07 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=i=up:active} 2 up:standby-replay 1 up:standby 2024-04-26T11:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:07 smithi096 ceph-mon[32753]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T11:50:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:07 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:07 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6836/2847426000,v1:172.21.15.33:6837/2847426000] up:active 2024-04-26T11:50:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:07 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 2024-04-26T11:50:07.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:07 smithi033 podman[96180]: 2024-04-26 11:50:07.532360556 +0000 UTC m=+0.748304056 container remove ea1da4ad8a0cc2c58000c716590ecc3722fbcadda7274be7edba919d729d7505 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-26T11:50:07.921 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:07 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:50:07.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:50:07 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:50:07.497+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:50:08.184 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:08 smithi033 ceph-mon[27845]: pgmap v678: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:08.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:08 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-26T11:50:08.184 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:08 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.285s CPU time. 2024-04-26T11:50:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:08 smithi194 ceph-mon[33017]: pgmap v678: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:08 smithi096 ceph-mon[32753]: pgmap v678: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:50:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:50:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:50:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:10 smithi033 ceph-mon[27845]: pgmap v679: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:10 smithi194 ceph-mon[33017]: pgmap v679: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:10 smithi096 ceph-mon[32753]: pgmap v679: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T11:50:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:50:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:50:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:50:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:50:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:50:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:50:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:50:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:50:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:50:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:12 smithi033 ceph-mon[27845]: pgmap v680: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 47 KiB/s wr, 31 op/s 2024-04-26T11:50:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:12 smithi033 ceph-mon[27845]: pgmap v681: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 59 KiB/s wr, 39 op/s 2024-04-26T11:50:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:12 smithi194 ceph-mon[33017]: pgmap v680: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 47 KiB/s wr, 31 op/s 2024-04-26T11:50:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:12 smithi194 ceph-mon[33017]: pgmap v681: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 59 KiB/s wr, 39 op/s 2024-04-26T11:50:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:12 smithi096 ceph-mon[32753]: pgmap v680: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 47 KiB/s wr, 31 op/s 2024-04-26T11:50:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:12 smithi096 ceph-mon[32753]: pgmap v681: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 59 KiB/s wr, 39 op/s 2024-04-26T11:50:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:13 smithi033 ceph-mon[27845]: pgmap v682: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 48 KiB/s wr, 33 op/s 2024-04-26T11:50:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:13 smithi194 ceph-mon[33017]: pgmap v682: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 48 KiB/s wr, 33 op/s 2024-04-26T11:50:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:13 smithi096 ceph-mon[32753]: pgmap v682: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 48 KiB/s wr, 33 op/s 2024-04-26T11:50:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:15 smithi033 ceph-mon[27845]: pgmap v683: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 47 KiB/s wr, 27 op/s 2024-04-26T11:50:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:15 smithi096 ceph-mon[32753]: pgmap v683: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 47 KiB/s wr, 27 op/s 2024-04-26T11:50:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:15 smithi194 ceph-mon[33017]: pgmap v683: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 47 KiB/s wr, 27 op/s 2024-04-26T11:50:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:17.546+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:17.546+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:17.546+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:50:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:17 smithi033 ceph-mon[27845]: pgmap v684: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-04-26T11:50:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:17 smithi096 ceph-mon[32753]: pgmap v684: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-04-26T11:50:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:17 smithi194 ceph-mon[33017]: pgmap v684: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-04-26T11:50:18.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-04-26T11:50:18.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:18 smithi033 systemd[1]: Stopped Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:18.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Consumed 2.285s CPU time. 2024-04-26T11:50:18.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-04-26T11:50:18.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:18 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-26T11:50:18.676 INFO:journalctl@ceph.mds.g.smithi033.stdout:Apr 26 11:50:18 smithi033 systemd[1]: Failed to start Ceph mds.g for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:19 smithi033 ceph-mon[27845]: pgmap v685: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-04-26T11:50:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:19 smithi096 ceph-mon[32753]: pgmap v685: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-04-26T11:50:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:19 smithi194 ceph-mon[33017]: pgmap v685: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 47 KiB/s wr, 30 op/s 2024-04-26T11:50:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:21 smithi033 ceph-mon[27845]: pgmap v686: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:21 smithi096 ceph-mon[32753]: pgmap v686: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:21.794+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0005ed0 con 0x7fef00077780 2024-04-26T11:50:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:21.795+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0005ed0 con 0x7fef00077780 2024-04-26T11:50:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:21.799+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc005de0 con 0x564385821900 2024-04-26T11:50:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:21.800+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc005de0 con 0x564385821900 2024-04-26T11:50:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:21 smithi194 ceph-mon[33017]: pgmap v686: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:22 smithi033 ceph-mon[27845]: Replacing daemon mds.g as rank 1 with standby daemon mds.j 2024-04-26T11:50:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:22 smithi033 ceph-mon[27845]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-26T11:50:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:22 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:22 smithi033 ceph-mon[27845]: osdmap e166: 12 total, 12 up, 12 in 2024-04-26T11:50:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:22 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:22 smithi096 ceph-mon[32753]: Replacing daemon mds.g as rank 1 with standby daemon mds.j 2024-04-26T11:50:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:22 smithi096 ceph-mon[32753]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-26T11:50:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:22 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:22 smithi096 ceph-mon[32753]: osdmap e166: 12 total, 12 up, 12 in 2024-04-26T11:50:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:22 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:22 smithi194 ceph-mon[33017]: Replacing daemon mds.g as rank 1 with standby daemon mds.j 2024-04-26T11:50:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:22 smithi194 ceph-mon[33017]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-26T11:50:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:22 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:22 smithi194 ceph-mon[33017]: osdmap e166: 12 total, 12 up, 12 in 2024-04-26T11:50:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:22 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:23 smithi033 ceph-mon[27845]: pgmap v688: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:23 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:resolve 2024-04-26T11:50:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:23 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:23 smithi096 ceph-mon[32753]: pgmap v688: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:23 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:resolve 2024-04-26T11:50:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:23 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:23 smithi194 ceph-mon[33017]: pgmap v688: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:23 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:resolve 2024-04-26T11:50:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:23 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:24 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:24 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:reconnect 2024-04-26T11:50:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:24 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:24 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:50:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:24 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:24 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:reconnect 2024-04-26T11:50:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:24 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:24 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:50:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:24 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:24 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:reconnect 2024-04-26T11:50:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:24 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:24 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:50:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-mon[27845]: pgmap v689: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:rejoin 2024-04-26T11:50:25.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:25.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-mon[27845]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:25.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7382400700 time 2024-04-26T11:50:25.704890+0000 2024-04-26T11:50:25.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:25.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f738dcc73dd] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: clone() 2024-04-26T11:50:25.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.706+0000 7f7382400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7382400700 time 2024-04-26T11:50:25.704890+0000 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f738dcc73dd] 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: clone() 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: *** Caught signal (Aborted) ** 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: in thread 7f7382400700 thread_name:ms_dispatch 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f738ca4fd20] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: gsignal() 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: abort() 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f738dcc7437] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 16: clone() 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.708+0000 7f7382400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: in thread 7f7382400700 thread_name:ms_dispatch 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f738ca4fd20] 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: gsignal() 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: abort() 2024-04-26T11:50:25.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f738dcc7437] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 16: clone() 2024-04-26T11:50:25.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:25.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:25 smithi096 ceph-mon[32753]: pgmap v689: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:25.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:25 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:rejoin 2024-04-26T11:50:25.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:25 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:25.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:25 smithi096 ceph-mon[32753]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: -1> 2024-04-26T11:50:25.706+0000 7f7382400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7382400700 time 2024-04-26T11:50:25.704890+0000 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f738dcc73dd] 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: clone() 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 0> 2024-04-26T11:50:25.708+0000 7f7382400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: in thread 7f7382400700 thread_name:ms_dispatch 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f738ca4fd20] 2024-04-26T11:50:25.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: gsignal() 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: abort() 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f738dcc7437] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 16: clone() 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: -1> 2024-04-26T11:50:25.706+0000 7f7382400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7382400700 time 2024-04-26T11:50:25.704890+0000 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f738dcc73dd] 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: clone() 2024-04-26T11:50:25.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 0> 2024-04-26T11:50:25.708+0000 7f7382400700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: in thread 7f7382400700 thread_name:ms_dispatch 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f738ca4fd20] 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2: gsignal() 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 3: abort() 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f738dcc7437] 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f738dcc75a3] 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x55a6ef8a8822] 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x55a6ef818598] 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a6ef81ac17] 2024-04-26T11:50:25.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a6ef713d0b] 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a6ef7142ac] 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x55a6ef6fcfad] 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f738df5be20] 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 13: (DispatchQueue::entry()+0x51f) [0x7f738df58fdf] 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f738e02bfc1] 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f738ca451ca] 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 16: clone() 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96141]: 2024-04-26T11:50:25.941 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:25 smithi033 podman[96368]: 2024-04-26 11:50:25.850029382 +0000 UTC m=+0.042279869 container died e74cb2fedb1c8160bcd01587e5fc69fddf6572d42069c2057599983f2c7b52f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:50:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:25 smithi194 ceph-mon[33017]: pgmap v689: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-26T11:50:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:25 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:rejoin 2024-04-26T11:50:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:25 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:25 smithi194 ceph-mon[33017]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:26 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:26 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:active 2024-04-26T11:50:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:26 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:26.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:26 smithi033 podman[96368]: 2024-04-26 11:50:26.629136266 +0000 UTC m=+0.821386820 container remove e74cb2fedb1c8160bcd01587e5fc69fddf6572d42069c2057599983f2c7b52f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:50:26.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:26 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:50:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:26 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:26 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:active 2024-04-26T11:50:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:26 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:26 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:26 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/2270541947,v1:172.21.15.33:6835/2270541947] up:active 2024-04-26T11:50:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:26 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:27.426 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-26T11:50:27.426 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:27 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.234s CPU time. 2024-04-26T11:50:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:27 smithi096 ceph-mon[32753]: pgmap v690: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T11:50:27.933 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:50:27 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:50:27.498+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:50:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:27 smithi194 ceph-mon[33017]: pgmap v690: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T11:50:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:27 smithi033 ceph-mon[27845]: pgmap v690: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T11:50:29.944 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:29 smithi033 ceph-mon[27845]: pgmap v691: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T11:50:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:29 smithi194 ceph-mon[33017]: pgmap v691: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T11:50:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:29 smithi096 ceph-mon[32753]: pgmap v691: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T11:50:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:31 smithi194 ceph-mon[33017]: pgmap v692: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 14 op/s 2024-04-26T11:50:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:31 smithi033 ceph-mon[27845]: pgmap v692: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 14 op/s 2024-04-26T11:50:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:31 smithi096 ceph-mon[32753]: pgmap v692: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 14 op/s 2024-04-26T11:50:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:32.561+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:32.561+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:50:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:32.561+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:33 smithi194 ceph-mon[33017]: pgmap v693: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 42 KiB/s wr, 13 op/s 2024-04-26T11:50:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:33 smithi194 ceph-mon[33017]: osdmap e167: 12 total, 12 up, 12 in 2024-04-26T11:50:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:33 smithi033 ceph-mon[27845]: pgmap v693: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 42 KiB/s wr, 13 op/s 2024-04-26T11:50:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:33 smithi033 ceph-mon[27845]: osdmap e167: 12 total, 12 up, 12 in 2024-04-26T11:50:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:33 smithi096 ceph-mon[32753]: pgmap v693: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 42 KiB/s wr, 13 op/s 2024-04-26T11:50:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:33 smithi096 ceph-mon[32753]: osdmap e167: 12 total, 12 up, 12 in 2024-04-26T11:50:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:35 smithi194 ceph-mon[33017]: pgmap v695: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 14 op/s 2024-04-26T11:50:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:35 smithi033 ceph-mon[27845]: pgmap v695: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 14 op/s 2024-04-26T11:50:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:35 smithi096 ceph-mon[32753]: pgmap v695: 97 pgs: 97 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 14 op/s 2024-04-26T11:50:37.597 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-04-26T11:50:37.598 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 systemd[1]: Stopped Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:37.598 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.234s CPU time. 2024-04-26T11:50:37.598 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 systemd[1]: Starting Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:50:37.874 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:37 smithi033 ceph-mon[27845]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 12 op/s 2024-04-26T11:50:37.874 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 podman[96461]: 2024-04-26 11:50:37.502642763 +0000 UTC m=+0.040186321 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:50:37.874 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 podman[96461]: 2024-04-26 11:50:37.612728271 +0000 UTC m=+0.150271872 container create 9b029e846692b7b95044c2dfb6934c26695c0ae0e22435b816d4c7193356d445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:50:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:37 smithi194 ceph-mon[33017]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 12 op/s 2024-04-26T11:50:38.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 podman[96461]: 2024-04-26 11:50:37.88408167 +0000 UTC m=+0.421625258 container init 9b029e846692b7b95044c2dfb6934c26695c0ae0e22435b816d4c7193356d445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T11:50:38.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 podman[96461]: 2024-04-26 11:50:37.89553426 +0000 UTC m=+0.433077809 container start 9b029e846692b7b95044c2dfb6934c26695c0ae0e22435b816d4c7193356d445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-26T11:50:38.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 ceph-mds[96479]: -- 172.21.15.33:0/3732578275 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556bd7659a40 con 0x556bd762d000 2024-04-26T11:50:38.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: starting mds.j at 2024-04-26T11:50:38.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 bash[96461]: 9b029e846692b7b95044c2dfb6934c26695c0ae0e22435b816d4c7193356d445 2024-04-26T11:50:38.177 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:37 smithi033 systemd[1]: Started Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:37 smithi096 ceph-mon[32753]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 12 op/s 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: daemon mds.j restarted 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: osdmap e168: 12 total, 12 up, 12 in 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:boot 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:38 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: daemon mds.j restarted 2024-04-26T11:50:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: osdmap e168: 12 total, 12 up, 12 in 2024-04-26T11:50:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:boot 2024-04-26T11:50:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:50:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:38 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: daemon mds.j restarted 2024-04-26T11:50:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: osdmap e168: 12 total, 12 up, 12 in 2024-04-26T11:50:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:boot 2024-04-26T11:50:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:50:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:38 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:40.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:39 smithi033 ceph-mon[27845]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-26T11:50:40.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:39 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:resolve 2024-04-26T11:50:40.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:39 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:39 smithi194 ceph-mon[33017]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-26T11:50:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:39 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:resolve 2024-04-26T11:50:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:39 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:39 smithi096 ceph-mon[32753]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-26T11:50:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:39 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:resolve 2024-04-26T11:50:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:39 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:40 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:40 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:reconnect 2024-04-26T11:50:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:40 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:40 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:50:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:40 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:40 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:reconnect 2024-04-26T11:50:41.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:40 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:41.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:40 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:50:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:40 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:40 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:reconnect 2024-04-26T11:50:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:40 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:40 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.0109998 2024-04-26T11:50:42.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:42 smithi194 ceph-mon[33017]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:50:42.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:42 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:rejoin 2024-04-26T11:50:42.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:42 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:42.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:42 smithi194 ceph-mon[33017]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:42.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:42 smithi194 ceph-mon[33017]: osdmap e169: 12 total, 12 up, 12 in 2024-04-26T11:50:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-mon[27845]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:50:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:rejoin 2024-04-26T11:50:42.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:42.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-mon[27845]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:42.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-mon[27845]: osdmap e169: 12 total, 12 up, 12 in 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff84fc00700 time 2024-04-26T11:50:42.050307+0000 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff85b48d3dd] 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.332 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: clone() 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.051+0000 7ff84fc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff84fc00700 time 2024-04-26T11:50:42.050307+0000 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:42.333 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff85b48d3dd] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.334 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: clone() 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: *** Caught signal (Aborted) ** 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: in thread 7ff84fc00700 thread_name:ms_dispatch 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff85a215d20] 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: gsignal() 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: abort() 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff85b48d437] 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.335 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.337 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.337 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.337 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.337 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.337 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.337 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 16: clone() 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.055+0000 7ff84fc00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: in thread 7ff84fc00700 thread_name:ms_dispatch 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff85a215d20] 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: gsignal() 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: abort() 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff85b48d437] 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.338 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 16: clone() 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: -1> 2024-04-26T11:50:42.051+0000 7ff84fc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff84fc00700 time 2024-04-26T11:50:42.050307+0000 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:42.339 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff85b48d3dd] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.340 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: clone() 2024-04-26T11:50:42.341 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.341 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 0> 2024-04-26T11:50:42.055+0000 7ff84fc00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:42.341 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: in thread 7ff84fc00700 thread_name:ms_dispatch 2024-04-26T11:50:42.341 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.341 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.341 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff85a215d20] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: gsignal() 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: abort() 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff85b48d437] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.342 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 16: clone() 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: -1> 2024-04-26T11:50:42.051+0000 7ff84fc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff84fc00700 time 2024-04-26T11:50:42.050307+0000 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff85b48d3dd] 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.343 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: clone() 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 0> 2024-04-26T11:50:42.055+0000 7ff84fc00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: in thread 7ff84fc00700 thread_name:ms_dispatch 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff85a215d20] 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2: gsignal() 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 3: abort() 2024-04-26T11:50:42.344 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff85b48d437] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff85b48d5a3] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x556bd1494822] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x556bd1404598] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556bd1406c17] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556bd12ffd0b] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556bd13002ac] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x556bd12e8fad] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7ff85b721e20] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 13: (DispatchQueue::entry()+0x51f) [0x7ff85b71efdf] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff85b7f1fc1] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff85a20b1ca] 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 16: clone() 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:42.345 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96475]: 2024-04-26T11:50:42.346 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:42 smithi033 podman[96513]: 2024-04-26 11:50:42.210659567 +0000 UTC m=+0.035930158 container died 9b029e846692b7b95044c2dfb6934c26695c0ae0e22435b816d4c7193356d445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T11:50:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:42 smithi096 ceph-mon[32753]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:50:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:42 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:rejoin 2024-04-26T11:50:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:42 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:42 smithi096 ceph-mon[32753]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:42 smithi096 ceph-mon[32753]: osdmap e169: 12 total, 12 up, 12 in 2024-04-26T11:50:42.932 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 26 11:50:42 smithi096 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-b[70166]: 2024-04-26T11:50:42.499+0000 7f040d55a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T11:50:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:43 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:43 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:active 2024-04-26T11:50:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:43 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:43 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:43 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:active 2024-04-26T11:50:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:43 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:43.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:43 smithi033 podman[96513]: 2024-04-26 11:50:43.046522959 +0000 UTC m=+0.871793591 container remove 9b029e846692b7b95044c2dfb6934c26695c0ae0e22435b816d4c7193356d445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T11:50:43.427 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:43 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:50:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:43 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:43 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/3935779982,v1:172.21.15.33:6835/3935779982] up:active 2024-04-26T11:50:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:43 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:43.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:43 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-26T11:50:43.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:43 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.131s CPU time. 2024-04-26T11:50:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:44 smithi194 ceph-mon[33017]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:50:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:44 smithi033 ceph-mon[27845]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:50:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:44 smithi096 ceph-mon[32753]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T11:50:46.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:46 smithi033 ceph-mon[27845]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:50:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:46 smithi194 ceph-mon[33017]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:46 smithi096 ceph-mon[32753]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 127 B/s wr, 4 op/s 2024-04-26T11:50:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:47.577+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:47.577+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:50:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:50:47.577+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:50:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:48 smithi194 ceph-mon[33017]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-26T11:50:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:48 smithi033 ceph-mon[27845]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-26T11:50:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:48 smithi096 ceph-mon[32753]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-26T11:50:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:49 smithi033 ceph-mon[27845]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:50:49.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:49 smithi096 ceph-mon[32753]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:50:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:49 smithi194 ceph-mon[33017]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:50:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:51 smithi033 ceph-mon[27845]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-26T11:50:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:51 smithi096 ceph-mon[32753]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-26T11:50:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:51 smithi194 ceph-mon[33017]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-26T11:50:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:53 smithi033 ceph-mon[27845]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-04-26T11:50:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:53 smithi096 ceph-mon[32753]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-04-26T11:50:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:53 smithi194 ceph-mon[33017]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-04-26T11:50:54.110 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:53 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-04-26T11:50:54.110 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:53 smithi033 systemd[1]: Stopped Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:54.110 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:53 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.131s CPU time. 2024-04-26T11:50:54.110 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:53 smithi033 systemd[1]: Starting Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968... 2024-04-26T11:50:54.413 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 podman[96604]: 2024-04-26 11:50:54.016838613 +0000 UTC m=+0.043045106 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T11:50:54.413 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 podman[96604]: 2024-04-26 11:50:54.140964544 +0000 UTC m=+0.167171061 container create c3ead0c1f116a053d49dd966d3965a57801175c78b6b77cfcac055d78f4970a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T11:50:54.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 podman[96604]: 2024-04-26 11:50:54.411960152 +0000 UTC m=+0.438166676 container init c3ead0c1f116a053d49dd966d3965a57801175c78b6b77cfcac055d78f4970a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T11:50:54.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 podman[96604]: 2024-04-26 11:50:54.423117441 +0000 UTC m=+0.449323978 container start c3ead0c1f116a053d49dd966d3965a57801175c78b6b77cfcac055d78f4970a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T11:50:54.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 ceph-mds[96623]: -- 172.21.15.33:0/3266303540 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559032161a40 con 0x559032135000 2024-04-26T11:50:54.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: starting mds.j at 2024-04-26T11:50:54.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 bash[96604]: c3ead0c1f116a053d49dd966d3965a57801175c78b6b77cfcac055d78f4970a1 2024-04-26T11:50:54.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:54 smithi033 systemd[1]: Started Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:50:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: daemon mds.j restarted 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: osdmap e170: 12 total, 12 up, 12 in 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:boot 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:55 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: daemon mds.j restarted 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: osdmap e170: 12 total, 12 up, 12 in 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:boot 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:55 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-04-26T11:50:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: daemon mds.j restarted 2024-04-26T11:50:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T11:50:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: osdmap e170: 12 total, 12 up, 12 in 2024-04-26T11:50:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:boot 2024-04-26T11:50:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-04-26T11:50:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-26T11:50:55.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T11:50:55.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:55 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:56.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:56 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:resolve 2024-04-26T11:50:56.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:56 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:56 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:resolve 2024-04-26T11:50:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:56 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:56 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:resolve 2024-04-26T11:50:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:56 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:57 smithi194 ceph-mon[33017]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:50:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:57 smithi194 ceph-mon[33017]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:57 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:reconnect 2024-04-26T11:50:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:57 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:57 smithi194 ceph-mon[33017]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:50:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:57 smithi033 ceph-mon[27845]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:50:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:57 smithi033 ceph-mon[27845]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:57.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:57 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:reconnect 2024-04-26T11:50:57.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:57 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:57.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:57 smithi033 ceph-mon[27845]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:50:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:57 smithi096 ceph-mon[32753]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:50:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:57 smithi096 ceph-mon[32753]: reconnect by client.25216 172.21.15.33:0/4125416142 after 0.00299995 2024-04-26T11:50:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:57 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:reconnect 2024-04-26T11:50:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:57 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:57 smithi096 ceph-mon[32753]: reconnect by client.14952 192.168.0.1:0/3404657790 after 0.00999984 2024-04-26T11:50:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:58 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:rejoin 2024-04-26T11:50:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:58 smithi194 ceph-mon[33017]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:58 smithi194 ceph-mon[33017]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:rejoin 2024-04-26T11:50:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-mon[27845]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-mon[27845]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:58.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5585e00700 time 2024-04-26T11:50:58.569525+0000 2024-04-26T11:50:58.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:58.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f559165d3dd] 2024-04-26T11:50:58.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.927 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: clone() 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.571+0000 7f5585e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5585e00700 time 2024-04-26T11:50:58.569525+0000 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:58.928 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f559165d3dd] 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.929 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: clone() 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: *** Caught signal (Aborted) ** 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: in thread 7f5585e00700 thread_name:ms_dispatch 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f55903e5d20] 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: gsignal() 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: abort() 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f559165d437] 2024-04-26T11:50:58.930 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.931 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 16: clone() 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.574+0000 7f5585e00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: in thread 7f5585e00700 thread_name:ms_dispatch 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f55903e5d20] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: gsignal() 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: abort() 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f559165d437] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.932 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 16: clone() 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:58 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:rejoin 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:58 smithi096 ceph-mon[32753]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:58 smithi096 ceph-mon[32753]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: -1> 2024-04-26T11:50:58.571+0000 7f5585e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5585e00700 time 2024-04-26T11:50:58.569525+0000 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f559165d3dd] 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.934 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: clone() 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 0> 2024-04-26T11:50:58.574+0000 7f5585e00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: in thread 7f5585e00700 thread_name:ms_dispatch 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.935 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f55903e5d20] 2024-04-26T11:50:58.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: gsignal() 2024-04-26T11:50:58.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: abort() 2024-04-26T11:50:58.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f559165d437] 2024-04-26T11:50:58.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.936 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 16: clone() 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: -1> 2024-04-26T11:50:58.571+0000 7f5585e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5585e00700 time 2024-04-26T11:50:58.569525+0000 2024-04-26T11:50:58.937 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f559165d3dd] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.938 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: clone() 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 0> 2024-04-26T11:50:58.574+0000 7f5585e00700 -1 *** Caught signal (Aborted) ** 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: in thread 7f5585e00700 thread_name:ms_dispatch 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f55903e5d20] 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2: gsignal() 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 3: abort() 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f559165d437] 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f559165d5a3] 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x559011502822] 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x559011472598] 2024-04-26T11:50:58.939 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559011474c17] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55901136dd0b] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55901136e2ac] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x559011356fad] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f55918f1e20] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 13: (DispatchQueue::entry()+0x51f) [0x7f55918eefdf] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55919c1fc1] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f55903db1ca] 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 16: clone() 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j[96619]: 2024-04-26T11:50:58.940 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:58 smithi033 podman[96657]: 2024-04-26 11:50:58.733089716 +0000 UTC m=+0.040937653 container died c3ead0c1f116a053d49dd966d3965a57801175c78b6b77cfcac055d78f4970a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T11:50:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:59 smithi194 ceph-mon[33017]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:50:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:59 smithi194 ceph-mon[33017]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:59 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:active 2024-04-26T11:50:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:50:59 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:59.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:59 smithi033 podman[96657]: 2024-04-26 11:50:59.542442398 +0000 UTC m=+0.850290368 container remove c3ead0c1f116a053d49dd966d3965a57801175c78b6b77cfcac055d78f4970a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-5cc7f146-03be-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T11:50:59.926 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:50:59 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-26T11:50:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:59 smithi033 ceph-mon[27845]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:50:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:59 smithi033 ceph-mon[27845]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:59 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:active 2024-04-26T11:50:59.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:50:59 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:50:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:59 smithi096 ceph-mon[32753]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:50:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:59 smithi096 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T11:50:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:59 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.33:6834/991410812,v1:172.21.15.33:6835/991410812] up:active 2024-04-26T11:50:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:50:59 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=i=up:active} 2 up:standby-replay 2024-04-26T11:51:00.426 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:00 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-26T11:51:00.426 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:00 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.069s CPU time. 2024-04-26T11:51:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:01 smithi194 ceph-mon[33017]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-26T11:51:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:01 smithi033 ceph-mon[27845]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-26T11:51:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:01 smithi096 ceph-mon[32753]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-26T11:51:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:02.593+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:02.594+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:02.594+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:51:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:03 smithi194 ceph-mon[33017]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:51:03.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:03 smithi033 ceph-mon[27845]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:51:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:03 smithi096 ceph-mon[32753]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:51:05.840 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:05 smithi033 ceph-mon[27845]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:51:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:05 smithi194 ceph-mon[33017]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:51:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:05 smithi096 ceph-mon[32753]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-26T11:51:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:07 smithi194 ceph-mon[33017]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 9.6 KiB/s wr, 11 op/s 2024-04-26T11:51:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:07 smithi033 ceph-mon[27845]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 9.6 KiB/s wr, 11 op/s 2024-04-26T11:51:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:07 smithi096 ceph-mon[32753]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 9.6 KiB/s wr, 11 op/s 2024-04-26T11:51:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:09 smithi194 ceph-mon[33017]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-04-26T11:51:09.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:09 smithi033 ceph-mon[27845]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-04-26T11:51:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:09 smithi096 ceph-mon[32753]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-04-26T11:51:10.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-04-26T11:51:10.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:10 smithi033 systemd[1]: Stopped Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:51:10.676 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Consumed 2.069s CPU time. 2024-04-26T11:51:10.677 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-04-26T11:51:10.677 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:10 smithi033 systemd[1]: ceph-5cc7f146-03be-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-26T11:51:10.677 INFO:journalctl@ceph.mds.j.smithi033.stdout:Apr 26 11:51:10 smithi033 systemd[1]: Failed to start Ceph mds.j for 5cc7f146-03be-11ef-bc93-c7b262605968. 2024-04-26T11:51:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:11 smithi194 ceph-mon[33017]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 9.5 KiB/s wr, 12 op/s 2024-04-26T11:51:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:51:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:11 smithi033 ceph-mon[27845]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 9.5 KiB/s wr, 12 op/s 2024-04-26T11:51:11.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:51:11.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:11 smithi096 ceph-mon[32753]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 9.5 KiB/s wr, 12 op/s 2024-04-26T11:51:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:51:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:13 smithi194 ceph-mon[33017]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T11:51:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:13 smithi033 ceph-mon[27845]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T11:51:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:13 smithi096 ceph-mon[32753]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T11:51:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:14.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:51:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:51:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:51:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:51:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:15 smithi194 ceph-mon[33017]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T11:51:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:51:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:51:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:51:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:51:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:15 smithi033 ceph-mon[27845]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T11:51:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T11:51:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T11:51:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T11:51:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T11:51:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:15 smithi096 ceph-mon[32753]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T11:51:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:17.609+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:17.609+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:17.609+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:51:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:51:17.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:51:17.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:51:17.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:51:17.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:17.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:51:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:17.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-26T11:51:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:51:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:18.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:51:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:18 smithi033 ceph-mon[27845]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-26T11:51:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:18 smithi096 ceph-mon[32753]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-26T11:51:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:18 smithi194 ceph-mon[33017]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-26T11:51:19.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:19 smithi033 ceph-mon[27845]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T11:51:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:19 smithi096 ceph-mon[32753]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T11:51:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:19 smithi194 ceph-mon[33017]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T11:51:22.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:21 smithi096 ceph-mon[32753]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:21 smithi194 ceph-mon[33017]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:21 smithi033 ceph-mon[27845]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:23 smithi194 ceph-mon[33017]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:23 smithi033 ceph-mon[27845]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:23 smithi096 ceph-mon[32753]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:25 smithi194 ceph-mon[33017]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:25 smithi033 ceph-mon[27845]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:25 smithi096 ceph-mon[32753]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:51:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:27 smithi194 ceph-mon[33017]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:51:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:27 smithi033 ceph-mon[27845]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:51:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:27 smithi096 ceph-mon[32753]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:51:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:29 smithi194 ceph-mon[33017]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:29 smithi033 ceph-mon[27845]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:29 smithi096 ceph-mon[32753]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:31 smithi194 ceph-mon[33017]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:32.115 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:31 smithi096 ceph-mon[32753]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:31 smithi033 ceph-mon[27845]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:32.625+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:32.625+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:51:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:32.625+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:33 smithi194 ceph-mon[33017]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:33 smithi033 ceph-mon[27845]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:33 smithi096 ceph-mon[32753]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:35 smithi194 ceph-mon[33017]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:35 smithi033 ceph-mon[27845]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:35 smithi096 ceph-mon[32753]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:37 smithi194 ceph-mon[33017]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:37 smithi033 ceph-mon[27845]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:37 smithi096 ceph-mon[32753]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:39 smithi194 ceph-mon[33017]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:39 smithi033 ceph-mon[27845]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:39 smithi096 ceph-mon[32753]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:41 smithi194 ceph-mon[33017]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:41 smithi033 ceph-mon[27845]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:41 smithi096 ceph-mon[32753]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:43 smithi194 ceph-mon[33017]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:43 smithi033 ceph-mon[27845]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:43 smithi096 ceph-mon[32753]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:45 smithi194 ceph-mon[33017]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:45 smithi033 ceph-mon[27845]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:45 smithi096 ceph-mon[32753]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:47.637+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:47.637+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:51:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:51:47.637+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:51:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:47 smithi194 ceph-mon[33017]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:47 smithi033 ceph-mon[27845]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:47 smithi096 ceph-mon[32753]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:49 smithi194 ceph-mon[33017]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:49 smithi033 ceph-mon[27845]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:49 smithi096 ceph-mon[32753]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-26T11:51:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:51 smithi033 ceph-mon[27845]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:51 smithi096 ceph-mon[32753]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:51 smithi194 ceph-mon[33017]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T11:51:54.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:53 smithi033 ceph-mon[27845]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:53 smithi096 ceph-mon[32753]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:53 smithi194 ceph-mon[33017]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:55 smithi033 ceph-mon[27845]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:55 smithi096 ceph-mon[32753]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:55 smithi194 ceph-mon[33017]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:51:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:57 smithi033 ceph-mon[27845]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:57 smithi096 ceph-mon[32753]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:51:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:57 smithi194 ceph-mon[33017]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:51:59 smithi033 ceph-mon[27845]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:51:59 smithi096 ceph-mon[32753]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:51:59 smithi194 ceph-mon[33017]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:01 smithi033 ceph-mon[27845]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:01 smithi096 ceph-mon[32753]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:01 smithi194 ceph-mon[33017]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:02.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:02.653+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:02.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:02.653+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:02.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:02.653+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:52:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:03 smithi033 ceph-mon[27845]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:03 smithi096 ceph-mon[32753]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:03 smithi194 ceph-mon[33017]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:05 smithi033 ceph-mon[27845]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:05 smithi096 ceph-mon[32753]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:05 smithi194 ceph-mon[33017]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:07 smithi033 ceph-mon[27845]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:07 smithi096 ceph-mon[32753]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:07 smithi194 ceph-mon[33017]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:09 smithi033 ceph-mon[27845]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:09 smithi096 ceph-mon[32753]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:09 smithi194 ceph-mon[33017]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:11 smithi033 ceph-mon[27845]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:11 smithi096 ceph-mon[32753]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:11 smithi194 ceph-mon[33017]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:13 smithi033 ceph-mon[27845]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:13 smithi096 ceph-mon[32753]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:13 smithi194 ceph-mon[33017]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:15 smithi033 ceph-mon[27845]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:15 smithi096 ceph-mon[32753]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:15 smithi194 ceph-mon[33017]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:17.668+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:17.668+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:17.668+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:52:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:17 smithi194 ceph-mon[33017]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:52:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:17 smithi033 ceph-mon[27845]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:52:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:17 smithi096 ceph-mon[32753]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:52:19.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:52:19.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:52:19.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:52:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:52:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:52:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:52:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:52:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:52:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:52:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:19 smithi194 ceph-mon[33017]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:19 smithi194 ceph-mon[33017]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:19 smithi033 ceph-mon[27845]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:19 smithi033 ceph-mon[27845]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:19 smithi096 ceph-mon[32753]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:19 smithi096 ceph-mon[32753]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:21 smithi194 ceph-mon[33017]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:22.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:21 smithi033 ceph-mon[27845]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:21 smithi096 ceph-mon[32753]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:23 smithi194 ceph-mon[33017]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:23 smithi033 ceph-mon[27845]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:23 smithi096 ceph-mon[32753]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:25 smithi194 ceph-mon[33017]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T11:52:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:25 smithi033 ceph-mon[27845]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T11:52:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:25 smithi096 ceph-mon[32753]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T11:52:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:27 smithi194 ceph-mon[33017]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T11:52:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:27 smithi033 ceph-mon[27845]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T11:52:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:27 smithi096 ceph-mon[32753]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T11:52:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:29 smithi194 ceph-mon[33017]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:52:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:29 smithi033 ceph-mon[27845]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:52:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:29 smithi096 ceph-mon[32753]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:52:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:31 smithi194 ceph-mon[33017]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:52:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:31 smithi033 ceph-mon[27845]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:52:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:31 smithi096 ceph-mon[32753]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:52:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:32.684+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:32.684+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:52:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:32.684+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:33 smithi194 ceph-mon[33017]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:52:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:33 smithi033 ceph-mon[27845]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:52:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:33 smithi096 ceph-mon[32753]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:52:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:35 smithi194 ceph-mon[33017]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:52:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:35 smithi033 ceph-mon[27845]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:52:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:35 smithi096 ceph-mon[32753]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:52:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:37 smithi194 ceph-mon[33017]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:37 smithi033 ceph-mon[27845]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:37 smithi096 ceph-mon[32753]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:39 smithi194 ceph-mon[33017]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:39 smithi033 ceph-mon[27845]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:39 smithi096 ceph-mon[32753]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:41 smithi194 ceph-mon[33017]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:41 smithi033 ceph-mon[27845]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:41 smithi096 ceph-mon[32753]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:43 smithi194 ceph-mon[33017]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:43 smithi033 ceph-mon[27845]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:43 smithi096 ceph-mon[32753]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:45 smithi194 ceph-mon[33017]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:52:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:45 smithi033 ceph-mon[27845]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:52:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:45 smithi096 ceph-mon[32753]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:52:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:47.692+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:47.692+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:52:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:52:47.692+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:52:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:47 smithi194 ceph-mon[33017]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:47 smithi033 ceph-mon[27845]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:47 smithi096 ceph-mon[32753]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:49 smithi194 ceph-mon[33017]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:49 smithi033 ceph-mon[27845]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:49 smithi096 ceph-mon[32753]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:51 smithi194 ceph-mon[33017]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:51 smithi033 ceph-mon[27845]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:51 smithi096 ceph-mon[32753]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:52:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:53 smithi194 ceph-mon[33017]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:53 smithi033 ceph-mon[27845]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:53 smithi096 ceph-mon[32753]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:52:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:55 smithi194 ceph-mon[33017]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:52:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:55 smithi033 ceph-mon[27845]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:52:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:55 smithi096 ceph-mon[32753]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:52:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:57 smithi194 ceph-mon[33017]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:57 smithi033 ceph-mon[27845]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:52:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:57 smithi096 ceph-mon[32753]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:52:59 smithi194 ceph-mon[33017]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:52:59 smithi033 ceph-mon[27845]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:52:59 smithi096 ceph-mon[32753]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:01 smithi194 ceph-mon[33017]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:01 smithi033 ceph-mon[27845]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:01 smithi096 ceph-mon[32753]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:02.700+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:02.700+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:53:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:02.700+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:03 smithi194 ceph-mon[33017]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:03 smithi033 ceph-mon[27845]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:03 smithi096 ceph-mon[32753]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:05 smithi194 ceph-mon[33017]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:53:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:05 smithi033 ceph-mon[27845]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:53:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:05 smithi096 ceph-mon[32753]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:53:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:07 smithi194 ceph-mon[33017]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:07 smithi033 ceph-mon[27845]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:07 smithi096 ceph-mon[32753]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:09 smithi194 ceph-mon[33017]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:09 smithi033 ceph-mon[27845]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:09 smithi096 ceph-mon[32753]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:11 smithi194 ceph-mon[33017]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:11 smithi033 ceph-mon[27845]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:11 smithi096 ceph-mon[32753]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:13 smithi194 ceph-mon[33017]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:13 smithi033 ceph-mon[27845]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:13 smithi096 ceph-mon[32753]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:15 smithi194 ceph-mon[33017]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:53:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:15 smithi033 ceph-mon[27845]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:15 smithi096 ceph-mon[32753]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:53:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:17.710+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:17.710+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:17.710+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:53:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:17 smithi194 ceph-mon[33017]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:17 smithi033 ceph-mon[27845]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:17 smithi096 ceph-mon[32753]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:53:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:53:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:53:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:19 smithi033 ceph-mon[27845]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:19 smithi096 ceph-mon[32753]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:19 smithi194 ceph-mon[33017]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:53:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:53:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:53:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:53:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:53:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:53:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:53:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:53:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:53:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:21.822+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00060f0 con 0x7fef00077780 2024-04-26T11:53:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:21.823+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00060f0 con 0x7fef00077780 2024-04-26T11:53:21.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:21.826+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc005fb0 con 0x564385821900 2024-04-26T11:53:21.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:21.827+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc005fb0 con 0x564385821900 2024-04-26T11:53:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:21 smithi033 ceph-mon[27845]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T11:53:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:21 smithi033 ceph-mon[27845]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:21 smithi096 ceph-mon[32753]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T11:53:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:21 smithi096 ceph-mon[32753]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:21 smithi194 ceph-mon[33017]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T11:53:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:21 smithi194 ceph-mon[33017]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:23 smithi033 ceph-mon[27845]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:23 smithi096 ceph-mon[32753]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:23 smithi194 ceph-mon[33017]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:25 smithi033 ceph-mon[27845]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T11:53:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:25 smithi096 ceph-mon[32753]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T11:53:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:25 smithi194 ceph-mon[33017]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T11:53:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:27 smithi033 ceph-mon[27845]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:27 smithi096 ceph-mon[32753]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:27 smithi194 ceph-mon[33017]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:29 smithi033 ceph-mon[27845]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:29 smithi096 ceph-mon[32753]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:29 smithi194 ceph-mon[33017]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:53:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:31 smithi033 ceph-mon[27845]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:53:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:31 smithi096 ceph-mon[32753]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:53:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:31 smithi194 ceph-mon[33017]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:53:32.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:32.717+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:32.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:32.717+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:53:32.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:32.717+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:33 smithi033 ceph-mon[27845]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:33 smithi096 ceph-mon[32753]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:33 smithi194 ceph-mon[33017]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:35 smithi033 ceph-mon[27845]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:35 smithi096 ceph-mon[32753]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:35 smithi194 ceph-mon[33017]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:37 smithi033 ceph-mon[27845]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:37 smithi096 ceph-mon[32753]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:37 smithi194 ceph-mon[33017]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:39 smithi194 ceph-mon[33017]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:39 smithi033 ceph-mon[27845]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:39 smithi096 ceph-mon[32753]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:41 smithi194 ceph-mon[33017]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:41 smithi033 ceph-mon[27845]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:41 smithi096 ceph-mon[32753]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:43 smithi194 ceph-mon[33017]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:43 smithi033 ceph-mon[27845]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:43 smithi096 ceph-mon[32753]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:45 smithi194 ceph-mon[33017]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:45 smithi033 ceph-mon[27845]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:45 smithi096 ceph-mon[32753]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:47.731+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:47.731+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:53:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:53:47.731+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:53:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:47 smithi194 ceph-mon[33017]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:47 smithi033 ceph-mon[27845]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:47 smithi096 ceph-mon[32753]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:50.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:49 smithi096 ceph-mon[32753]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:49 smithi194 ceph-mon[33017]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:49 smithi033 ceph-mon[27845]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:52.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:51 smithi033 ceph-mon[27845]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:51 smithi194 ceph-mon[33017]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:51 smithi096 ceph-mon[32753]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:53 smithi194 ceph-mon[33017]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:53 smithi033 ceph-mon[27845]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:53 smithi096 ceph-mon[32753]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:55 smithi194 ceph-mon[33017]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:55 smithi033 ceph-mon[27845]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:55 smithi096 ceph-mon[32753]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:53:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:57 smithi194 ceph-mon[33017]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:57 smithi033 ceph-mon[27845]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:53:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:57 smithi096 ceph-mon[32753]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:53:59 smithi194 ceph-mon[33017]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:53:59 smithi033 ceph-mon[27845]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:53:59 smithi096 ceph-mon[32753]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:01 smithi194 ceph-mon[33017]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:01 smithi033 ceph-mon[27845]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:01 smithi096 ceph-mon[32753]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:02.747+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:02.747+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:02.747+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:54:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 181.682063 secs 2024-04-26T11:54:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:03 smithi194 ceph-mon[33017]: slow request 181.682062 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T11:54:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 181.682063 secs 2024-04-26T11:54:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:03 smithi033 ceph-mon[27845]: slow request 181.682062 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T11:54:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 181.682063 secs 2024-04-26T11:54:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:03 smithi096 ceph-mon[32753]: slow request 181.682062 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T11:54:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:04 smithi194 ceph-mon[33017]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:04 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 184.389837 secs 2024-04-26T11:54:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:04 smithi194 ceph-mon[33017]: slow request 184.389836 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T11:54:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:04 smithi033 ceph-mon[27845]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:04 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 184.389837 secs 2024-04-26T11:54:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:04 smithi033 ceph-mon[27845]: slow request 184.389836 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T11:54:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:04 smithi096 ceph-mon[32753]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:04 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 184.389837 secs 2024-04-26T11:54:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:04 smithi096 ceph-mon[32753]: slow request 184.389836 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T11:54:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:06 smithi194 ceph-mon[33017]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:06 smithi033 ceph-mon[27845]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:06 smithi096 ceph-mon[32753]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:07 smithi194 ceph-mon[33017]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-26T11:54:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:07 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:active 2024-04-26T11:54:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:07 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:54:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:07 smithi033 ceph-mon[27845]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-26T11:54:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:07 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:active 2024-04-26T11:54:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:07 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:54:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:07 smithi096 ceph-mon[32753]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-26T11:54:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:07 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.96:6834/4062130572,v1:172.21.15.96:6835/4062130572] up:active 2024-04-26T11:54:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:07 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:54:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:08 smithi194 ceph-mon[33017]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:08 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:54:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:08 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:54:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 186.682122 secs 2024-04-26T11:54:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:08 smithi033 ceph-mon[27845]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:08 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:54:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:08 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:54:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 186.682122 secs 2024-04-26T11:54:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:08 smithi096 ceph-mon[32753]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:08 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:54:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:08 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:54:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 186.682122 secs 2024-04-26T11:54:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 189.389952 secs 2024-04-26T11:54:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 189.389952 secs 2024-04-26T11:54:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 189.389952 secs 2024-04-26T11:54:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:10 smithi194 ceph-mon[33017]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:10 smithi033 ceph-mon[27845]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:10 smithi096 ceph-mon[32753]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:12 smithi194 ceph-mon[33017]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:12 smithi194 ceph-mon[33017]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-26T11:54:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:12 smithi033 ceph-mon[27845]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:12 smithi033 ceph-mon[27845]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-26T11:54:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:12 smithi096 ceph-mon[32753]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:12 smithi096 ceph-mon[32753]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-26T11:54:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 191.682184 secs 2024-04-26T11:54:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 191.682184 secs 2024-04-26T11:54:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 191.682184 secs 2024-04-26T11:54:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:14 smithi194 ceph-mon[33017]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 194.390098 secs 2024-04-26T11:54:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:14 smithi033 ceph-mon[27845]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 194.390098 secs 2024-04-26T11:54:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:14 smithi096 ceph-mon[32753]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 194.390098 secs 2024-04-26T11:54:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:16 smithi194 ceph-mon[33017]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:16 smithi033 ceph-mon[27845]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:16 smithi096 ceph-mon[32753]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:17 smithi194 ceph-mon[33017]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:17 smithi033 ceph-mon[27845]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:17 smithi096 ceph-mon[32753]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:17.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:17.753+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:17.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:17.753+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:17.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:17.753+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:54:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 196.682243 secs 2024-04-26T11:54:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 196.682243 secs 2024-04-26T11:54:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 196.682243 secs 2024-04-26T11:54:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 199.390238 secs 2024-04-26T11:54:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:19 smithi194 ceph-mon[33017]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 199.390238 secs 2024-04-26T11:54:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:19 smithi033 ceph-mon[27845]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 199.390238 secs 2024-04-26T11:54:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:19 smithi096 ceph-mon[32753]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:21 smithi194 ceph-mon[33017]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:54:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:21 smithi033 ceph-mon[27845]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:54:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:21 smithi096 ceph-mon[32753]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:54:22.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:54:22.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:54:22.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:22 smithi033 ceph-mon[27845]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:22.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:22 smithi033 ceph-mon[27845]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:22.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:54:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:54:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:54:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:22 smithi194 ceph-mon[33017]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:22 smithi194 ceph-mon[33017]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:54:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:54:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:54:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:22 smithi096 ceph-mon[32753]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:22 smithi096 ceph-mon[32753]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:54:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 201.682326 secs 2024-04-26T11:54:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 204.390381 secs 2024-04-26T11:54:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 201.682326 secs 2024-04-26T11:54:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 204.390381 secs 2024-04-26T11:54:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 201.682326 secs 2024-04-26T11:54:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 204.390381 secs 2024-04-26T11:54:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:24 smithi194 ceph-mon[33017]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:24 smithi033 ceph-mon[27845]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:24 smithi096 ceph-mon[32753]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:26 smithi033 ceph-mon[27845]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:26 smithi096 ceph-mon[32753]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:26 smithi194 ceph-mon[33017]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 206.682386 secs 2024-04-26T11:54:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 206.682386 secs 2024-04-26T11:54:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 206.682386 secs 2024-04-26T11:54:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:28 smithi033 ceph-mon[27845]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 209.390500 secs 2024-04-26T11:54:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:28 smithi096 ceph-mon[32753]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 209.390500 secs 2024-04-26T11:54:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:28 smithi194 ceph-mon[33017]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 209.390500 secs 2024-04-26T11:54:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:30 smithi033 ceph-mon[27845]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:30 smithi096 ceph-mon[32753]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:30 smithi194 ceph-mon[33017]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:54:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:32.754+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:32.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:32.755+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:32.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:32.755+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:54:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:32 smithi033 ceph-mon[27845]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:54:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 211.682469 secs 2024-04-26T11:54:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:32 smithi096 ceph-mon[32753]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:54:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 211.682469 secs 2024-04-26T11:54:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:32 smithi194 ceph-mon[33017]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:54:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 211.682469 secs 2024-04-26T11:54:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 214.390625 secs 2024-04-26T11:54:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 214.390625 secs 2024-04-26T11:54:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 214.390625 secs 2024-04-26T11:54:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:34 smithi096 ceph-mon[32753]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:34 smithi194 ceph-mon[33017]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:34 smithi033 ceph-mon[27845]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:36 smithi194 ceph-mon[33017]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:36 smithi033 ceph-mon[27845]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:36 smithi096 ceph-mon[32753]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 216.682550 secs 2024-04-26T11:54:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 216.682550 secs 2024-04-26T11:54:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 216.682550 secs 2024-04-26T11:54:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:38 smithi194 ceph-mon[33017]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 219.390707 secs 2024-04-26T11:54:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:38 smithi033 ceph-mon[27845]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 219.390707 secs 2024-04-26T11:54:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:38 smithi096 ceph-mon[32753]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 219.390707 secs 2024-04-26T11:54:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:40 smithi194 ceph-mon[33017]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:40 smithi033 ceph-mon[27845]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:40 smithi096 ceph-mon[32753]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:42 smithi194 ceph-mon[33017]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 221.682636 secs 2024-04-26T11:54:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:42 smithi033 ceph-mon[27845]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 221.682636 secs 2024-04-26T11:54:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:42 smithi096 ceph-mon[32753]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 221.682636 secs 2024-04-26T11:54:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 224.390773 secs 2024-04-26T11:54:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 224.390773 secs 2024-04-26T11:54:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 224.390773 secs 2024-04-26T11:54:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:44 smithi194 ceph-mon[33017]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:44 smithi033 ceph-mon[27845]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:44 smithi096 ceph-mon[32753]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:46 smithi194 ceph-mon[33017]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:46 smithi033 ceph-mon[27845]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:46 smithi096 ceph-mon[32753]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:47.768+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:47.768+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:54:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:54:47.768+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:54:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 226.682718 secs 2024-04-26T11:54:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 226.682718 secs 2024-04-26T11:54:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 226.682718 secs 2024-04-26T11:54:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:48 smithi194 ceph-mon[33017]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 229.390907 secs 2024-04-26T11:54:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:48 smithi033 ceph-mon[27845]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 229.390907 secs 2024-04-26T11:54:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:48 smithi096 ceph-mon[32753]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 229.390907 secs 2024-04-26T11:54:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:50 smithi194 ceph-mon[33017]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:50 smithi033 ceph-mon[27845]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:50 smithi096 ceph-mon[32753]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:53 smithi194 ceph-mon[33017]: pgmap v832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 231.682803 secs 2024-04-26T11:54:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:53 smithi033 ceph-mon[27845]: pgmap v832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 231.682803 secs 2024-04-26T11:54:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:53 smithi096 ceph-mon[32753]: pgmap v832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 231.682803 secs 2024-04-26T11:54:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 234.391042 secs 2024-04-26T11:54:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 234.391042 secs 2024-04-26T11:54:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 234.391042 secs 2024-04-26T11:54:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:55 smithi194 ceph-mon[33017]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:55 smithi033 ceph-mon[27845]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:55 smithi096 ceph-mon[32753]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:57 smithi194 ceph-mon[33017]: pgmap v834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:57 smithi033 ceph-mon[27845]: pgmap v834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:57 smithi096 ceph-mon[32753]: pgmap v834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:54:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 236.682913 secs 2024-04-26T11:54:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 236.682913 secs 2024-04-26T11:54:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 236.682913 secs 2024-04-26T11:54:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:59 smithi194 ceph-mon[33017]: pgmap v835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 239.391184 secs 2024-04-26T11:54:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:54:59 smithi194 ceph-mon[33017]: slow request 181.118973 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T11:54:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:59 smithi033 ceph-mon[27845]: pgmap v835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 239.391184 secs 2024-04-26T11:54:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:54:59 smithi033 ceph-mon[27845]: slow request 181.118973 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T11:54:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:59 smithi096 ceph-mon[32753]: pgmap v835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:54:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 239.391184 secs 2024-04-26T11:54:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:54:59 smithi096 ceph-mon[32753]: slow request 181.118973 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T11:55:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:00 smithi194 ceph-mon[33017]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:55:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:00 smithi194 ceph-mon[33017]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:55:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:00 smithi033 ceph-mon[27845]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:55:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:00 smithi033 ceph-mon[27845]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:55:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:00 smithi096 ceph-mon[32753]: mds.? [v2:172.21.15.194:6838/517247009,v1:172.21.15.194:6839/517247009] up:active 2024-04-26T11:55:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:00 smithi096 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=i=up:active} 2 up:standby-replay 2024-04-26T11:55:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:01 smithi194 ceph-mon[33017]: pgmap v836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:01 smithi033 ceph-mon[27845]: pgmap v836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:01 smithi096 ceph-mon[32753]: pgmap v836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:02.783+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:02.783+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:55:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:02.783+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:03 smithi194 ceph-mon[33017]: pgmap v837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:03.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 241.682999 secs 2024-04-26T11:55:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:03 smithi033 ceph-mon[27845]: pgmap v837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 241.682999 secs 2024-04-26T11:55:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:03 smithi096 ceph-mon[32753]: pgmap v837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 241.682999 secs 2024-04-26T11:55:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 244.391332 secs 2024-04-26T11:55:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 244.391332 secs 2024-04-26T11:55:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 244.391332 secs 2024-04-26T11:55:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:05 smithi194 ceph-mon[33017]: pgmap v838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:05 smithi033 ceph-mon[27845]: pgmap v838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:05 smithi096 ceph-mon[32753]: pgmap v838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:07 smithi194 ceph-mon[33017]: pgmap v839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:07 smithi033 ceph-mon[27845]: pgmap v839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:07 smithi096 ceph-mon[32753]: pgmap v839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 246.683057 secs 2024-04-26T11:55:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 246.683057 secs 2024-04-26T11:55:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 246.683057 secs 2024-04-26T11:55:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:09 smithi194 ceph-mon[33017]: pgmap v840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 249.391452 secs 2024-04-26T11:55:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:09 smithi033 ceph-mon[27845]: pgmap v840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 249.391452 secs 2024-04-26T11:55:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:09 smithi096 ceph-mon[32753]: pgmap v840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 249.391452 secs 2024-04-26T11:55:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:10 smithi033 ceph-mon[27845]: pgmap v841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:10 smithi096 ceph-mon[32753]: pgmap v841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:10 smithi194 ceph-mon[33017]: pgmap v841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:12 smithi033 ceph-mon[27845]: pgmap v842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 251.683119 secs 2024-04-26T11:55:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:12 smithi096 ceph-mon[32753]: pgmap v842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 251.683119 secs 2024-04-26T11:55:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:12 smithi194 ceph-mon[33017]: pgmap v842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 251.683119 secs 2024-04-26T11:55:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 254.391535 secs 2024-04-26T11:55:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 254.391535 secs 2024-04-26T11:55:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 254.391535 secs 2024-04-26T11:55:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:14 smithi096 ceph-mon[32753]: pgmap v843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:14 smithi194 ceph-mon[33017]: pgmap v843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:14 smithi033 ceph-mon[27845]: pgmap v843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:16 smithi194 ceph-mon[33017]: pgmap v844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:16 smithi033 ceph-mon[27845]: pgmap v844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:16 smithi096 ceph-mon[32753]: pgmap v844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:17.796+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:17.796+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:17.796+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:55:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 256.683178 secs 2024-04-26T11:55:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 256.683178 secs 2024-04-26T11:55:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 256.683178 secs 2024-04-26T11:55:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:18 smithi194 ceph-mon[33017]: pgmap v845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 259.391596 secs 2024-04-26T11:55:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:18 smithi033 ceph-mon[27845]: pgmap v845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 259.391596 secs 2024-04-26T11:55:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:18 smithi096 ceph-mon[32753]: pgmap v845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 259.391596 secs 2024-04-26T11:55:21.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:20 smithi096 ceph-mon[32753]: pgmap v846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:20 smithi194 ceph-mon[33017]: pgmap v846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:20 smithi033 ceph-mon[27845]: pgmap v846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:22 smithi033 ceph-mon[27845]: pgmap v847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:55:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 261.683242 secs 2024-04-26T11:55:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:22 smithi194 ceph-mon[33017]: pgmap v847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:55:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 261.683242 secs 2024-04-26T11:55:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:22 smithi096 ceph-mon[32753]: pgmap v847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:55:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 261.683242 secs 2024-04-26T11:55:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 264.391650 secs 2024-04-26T11:55:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 264.391650 secs 2024-04-26T11:55:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 264.391650 secs 2024-04-26T11:55:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:24 smithi194 ceph-mon[33017]: pgmap v848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:24 smithi033 ceph-mon[27845]: pgmap v848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:24 smithi096 ceph-mon[32753]: pgmap v848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T11:55:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:26 smithi033 ceph-mon[27845]: pgmap v849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:26 smithi096 ceph-mon[32753]: pgmap v849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:26 smithi194 ceph-mon[33017]: pgmap v849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T11:55:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:28.306 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 266.683298 secs 2024-04-26T11:55:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 266.683298 secs 2024-04-26T11:55:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 266.683298 secs 2024-04-26T11:55:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:28 smithi033 ceph-mon[27845]: pgmap v850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 269.391787 secs 2024-04-26T11:55:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:28 smithi194 ceph-mon[33017]: pgmap v850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 269.391787 secs 2024-04-26T11:55:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:28 smithi096 ceph-mon[32753]: pgmap v850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 269.391787 secs 2024-04-26T11:55:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:55:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:55:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:30 smithi194 ceph-mon[33017]: pgmap v851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T11:55:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:30 smithi194 ceph-mon[33017]: pgmap v852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T11:55:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:55:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:55:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:30 smithi033 ceph-mon[27845]: pgmap v851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T11:55:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:30 smithi033 ceph-mon[27845]: pgmap v852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T11:55:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:55:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:55:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:30 smithi096 ceph-mon[32753]: pgmap v851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T11:55:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:30 smithi096 ceph-mon[32753]: pgmap v852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T11:55:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:55:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:32 smithi194 ceph-mon[33017]: pgmap v853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:32 smithi033 ceph-mon[27845]: pgmap v853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:32 smithi096 ceph-mon[32753]: pgmap v853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:32.812 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:32.811+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:32.812 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:32.811+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:55:32.812 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:32.811+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 271.683385 secs 2024-04-26T11:55:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 274.391923 secs 2024-04-26T11:55:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 271.683385 secs 2024-04-26T11:55:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 274.391923 secs 2024-04-26T11:55:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 271.683385 secs 2024-04-26T11:55:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 274.391923 secs 2024-04-26T11:55:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:34 smithi194 ceph-mon[33017]: pgmap v854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:34 smithi033 ceph-mon[27845]: pgmap v854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:34 smithi096 ceph-mon[32753]: pgmap v854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:36 smithi194 ceph-mon[33017]: pgmap v855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:36 smithi033 ceph-mon[27845]: pgmap v855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:36 smithi096 ceph-mon[32753]: pgmap v855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:38 smithi194 ceph-mon[33017]: pgmap v856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 276.683444 secs 2024-04-26T11:55:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 279.392060 secs 2024-04-26T11:55:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:38 smithi033 ceph-mon[27845]: pgmap v856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 276.683444 secs 2024-04-26T11:55:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 279.392060 secs 2024-04-26T11:55:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:38 smithi096 ceph-mon[32753]: pgmap v856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:55:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 276.683444 secs 2024-04-26T11:55:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 279.392060 secs 2024-04-26T11:55:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:40 smithi194 ceph-mon[33017]: pgmap v857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:55:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:40 smithi033 ceph-mon[27845]: pgmap v857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:55:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:40 smithi096 ceph-mon[32753]: pgmap v857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:55:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:42 smithi194 ceph-mon[33017]: pgmap v858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:42 smithi033 ceph-mon[27845]: pgmap v858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:42 smithi096 ceph-mon[32753]: pgmap v858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 281.683529 secs 2024-04-26T11:55:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 284.392212 secs 2024-04-26T11:55:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 281.683529 secs 2024-04-26T11:55:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 284.392212 secs 2024-04-26T11:55:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 281.683529 secs 2024-04-26T11:55:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 284.392212 secs 2024-04-26T11:55:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:44 smithi194 ceph-mon[33017]: pgmap v859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:44 smithi033 ceph-mon[27845]: pgmap v859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:44 smithi096 ceph-mon[32753]: pgmap v859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:46 smithi194 ceph-mon[33017]: pgmap v860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:46 smithi033 ceph-mon[27845]: pgmap v860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:46 smithi096 ceph-mon[32753]: pgmap v860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:47.823+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:47.823+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:55:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:55:47.823+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:55:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:48 smithi194 ceph-mon[33017]: pgmap v861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 286.683610 secs 2024-04-26T11:55:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 289.392278 secs 2024-04-26T11:55:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:48 smithi033 ceph-mon[27845]: pgmap v861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 286.683610 secs 2024-04-26T11:55:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 289.392278 secs 2024-04-26T11:55:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:48 smithi096 ceph-mon[32753]: pgmap v861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 286.683610 secs 2024-04-26T11:55:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 289.392278 secs 2024-04-26T11:55:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:50 smithi194 ceph-mon[33017]: pgmap v862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:50 smithi033 ceph-mon[27845]: pgmap v862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:50 smithi096 ceph-mon[32753]: pgmap v862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:52 smithi194 ceph-mon[33017]: pgmap v863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:52 smithi033 ceph-mon[27845]: pgmap v863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:52 smithi096 ceph-mon[32753]: pgmap v863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 291.683695 secs 2024-04-26T11:55:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 294.392421 secs 2024-04-26T11:55:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 291.683695 secs 2024-04-26T11:55:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 294.392421 secs 2024-04-26T11:55:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 291.683695 secs 2024-04-26T11:55:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 294.392421 secs 2024-04-26T11:55:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:54 smithi033 ceph-mon[27845]: pgmap v864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:54 smithi096 ceph-mon[32753]: pgmap v864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:54 smithi194 ceph-mon[33017]: pgmap v864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:56 smithi033 ceph-mon[27845]: pgmap v865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:56 smithi096 ceph-mon[32753]: pgmap v865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:56 smithi194 ceph-mon[33017]: pgmap v865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:55:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:58 smithi033 ceph-mon[27845]: pgmap v866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 296.683768 secs 2024-04-26T11:55:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:55:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 299.392500 secs 2024-04-26T11:55:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:58 smithi096 ceph-mon[32753]: pgmap v866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 296.683768 secs 2024-04-26T11:55:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:55:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 299.392500 secs 2024-04-26T11:55:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:58 smithi194 ceph-mon[33017]: pgmap v866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:55:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 296.683768 secs 2024-04-26T11:55:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:55:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 299.392500 secs 2024-04-26T11:56:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:00 smithi033 ceph-mon[27845]: pgmap v867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:00 smithi096 ceph-mon[32753]: pgmap v867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:00 smithi194 ceph-mon[33017]: pgmap v867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:02 smithi033 ceph-mon[27845]: pgmap v868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:02 smithi096 ceph-mon[32753]: pgmap v868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:02.831+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:02.831+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:02.831+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:56:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:02 smithi194 ceph-mon[33017]: pgmap v868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 301.683812 secs 2024-04-26T11:56:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 304.392565 secs 2024-04-26T11:56:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 301.683812 secs 2024-04-26T11:56:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 304.392565 secs 2024-04-26T11:56:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 301.683812 secs 2024-04-26T11:56:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 304.392565 secs 2024-04-26T11:56:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:04 smithi033 ceph-mon[27845]: pgmap v869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:04 smithi096 ceph-mon[32753]: pgmap v869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:04 smithi194 ceph-mon[33017]: pgmap v869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:06 smithi033 ceph-mon[27845]: pgmap v870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:06 smithi096 ceph-mon[32753]: pgmap v870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:06 smithi194 ceph-mon[33017]: pgmap v870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:08 smithi033 ceph-mon[27845]: pgmap v871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 306.683911 secs 2024-04-26T11:56:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 309.392639 secs 2024-04-26T11:56:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:08 smithi096 ceph-mon[32753]: pgmap v871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 306.683911 secs 2024-04-26T11:56:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 309.392639 secs 2024-04-26T11:56:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:08 smithi194 ceph-mon[33017]: pgmap v871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 306.683911 secs 2024-04-26T11:56:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 309.392639 secs 2024-04-26T11:56:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:10 smithi033 ceph-mon[27845]: pgmap v872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:10 smithi096 ceph-mon[32753]: pgmap v872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:10 smithi194 ceph-mon[33017]: pgmap v872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:12 smithi033 ceph-mon[27845]: pgmap v873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:12 smithi096 ceph-mon[32753]: pgmap v873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:12 smithi194 ceph-mon[33017]: pgmap v873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 311.684015 secs 2024-04-26T11:56:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 314.392757 secs 2024-04-26T11:56:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 311.684015 secs 2024-04-26T11:56:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 314.392757 secs 2024-04-26T11:56:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 311.684015 secs 2024-04-26T11:56:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 314.392757 secs 2024-04-26T11:56:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:14 smithi194 ceph-mon[33017]: pgmap v874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:14 smithi033 ceph-mon[27845]: pgmap v874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:14 smithi096 ceph-mon[32753]: pgmap v874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:16 smithi194 ceph-mon[33017]: pgmap v875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:16 smithi033 ceph-mon[27845]: pgmap v875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:16 smithi096 ceph-mon[32753]: pgmap v875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:17.840+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:17.840+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:17.840+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:56:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:18 smithi194 ceph-mon[33017]: pgmap v876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 316.684077 secs 2024-04-26T11:56:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 319.392914 secs 2024-04-26T11:56:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:18 smithi033 ceph-mon[27845]: pgmap v876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 316.684077 secs 2024-04-26T11:56:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 319.392914 secs 2024-04-26T11:56:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:18 smithi096 ceph-mon[32753]: pgmap v876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 316.684077 secs 2024-04-26T11:56:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 319.392914 secs 2024-04-26T11:56:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:20 smithi194 ceph-mon[33017]: pgmap v877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:20 smithi033 ceph-mon[27845]: pgmap v877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:20 smithi096 ceph-mon[32753]: pgmap v877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T11:56:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T11:56:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40a61d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feef0009620 tx=0x7feef0005f90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T11:56:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40a61d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feef0009620 tx=0x7feef0005f90 comp rx=0 tx=0).stop 2024-04-26T11:56:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40a61d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T11:56:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40a61d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T11:56:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40a61d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T11:56:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.701+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 0x7feee40b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T11:56:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.702+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 0x7feee40b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T11:56:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.703+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 0x7feee40b0f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0019000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T11:56:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.852+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0006310 con 0x7fef00077780 2024-04-26T11:56:21.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.853+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0006310 con 0x7fef00077780 2024-04-26T11:56:21.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.854+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0061d0 con 0x564385821900 2024-04-26T11:56:21.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:21.855+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0061d0 con 0x564385821900 2024-04-26T11:56:22.735 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:22 smithi096 ceph-mon[32753]: pgmap v878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:22 smithi194 ceph-mon[33017]: pgmap v878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:22 smithi033 ceph-mon[27845]: pgmap v878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:23.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 321.684164 secs 2024-04-26T11:56:23.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 324.393051 secs 2024-04-26T11:56:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 321.684164 secs 2024-04-26T11:56:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 324.393051 secs 2024-04-26T11:56:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 321.684164 secs 2024-04-26T11:56:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 324.393051 secs 2024-04-26T11:56:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:24 smithi194 ceph-mon[33017]: pgmap v879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:24 smithi033 ceph-mon[27845]: pgmap v879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:24 smithi096 ceph-mon[32753]: pgmap v879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:26 smithi194 ceph-mon[33017]: pgmap v880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:26 smithi033 ceph-mon[27845]: pgmap v880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:26 smithi096 ceph-mon[32753]: pgmap v880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:28.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:28 smithi033 ceph-mon[27845]: pgmap v881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:28.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 326.684244 secs 2024-04-26T11:56:28.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 329.393158 secs 2024-04-26T11:56:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:28 smithi194 ceph-mon[33017]: pgmap v881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 326.684244 secs 2024-04-26T11:56:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 329.393158 secs 2024-04-26T11:56:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:28 smithi096 ceph-mon[32753]: pgmap v881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:29.160 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 326.684244 secs 2024-04-26T11:56:29.160 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 329.393158 secs 2024-04-26T11:56:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:30 smithi194 ceph-mon[33017]: pgmap v882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:56:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:30 smithi033 ceph-mon[27845]: pgmap v882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:56:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:30 smithi096 ceph-mon[32753]: pgmap v882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:56:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:56:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:56:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:31 smithi194 ceph-mon[33017]: pgmap v883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T11:56:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:31 smithi194 ceph-mon[33017]: pgmap v884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:56:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:56:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:56:31.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:31 smithi033 ceph-mon[27845]: pgmap v883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T11:56:31.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:31 smithi033 ceph-mon[27845]: pgmap v884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:31.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:56:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:56:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:56:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:31 smithi096 ceph-mon[32753]: pgmap v883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T11:56:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:31 smithi096 ceph-mon[32753]: pgmap v884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:56:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:32.852+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:32.852+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:56:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:32.852+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 331.684331 secs 2024-04-26T11:56:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 331.684331 secs 2024-04-26T11:56:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 331.684331 secs 2024-04-26T11:56:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:33 smithi194 ceph-mon[33017]: pgmap v885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 334.393204 secs 2024-04-26T11:56:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:33 smithi033 ceph-mon[27845]: pgmap v885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 334.393204 secs 2024-04-26T11:56:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:33 smithi096 ceph-mon[32753]: pgmap v885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 334.393204 secs 2024-04-26T11:56:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:35 smithi033 ceph-mon[27845]: pgmap v886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:35 smithi096 ceph-mon[32753]: pgmap v886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:35 smithi194 ceph-mon[33017]: pgmap v886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:37 smithi033 ceph-mon[27845]: pgmap v887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 336.684414 secs 2024-04-26T11:56:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:37 smithi096 ceph-mon[32753]: pgmap v887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 336.684414 secs 2024-04-26T11:56:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:37 smithi194 ceph-mon[33017]: pgmap v887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 336.684414 secs 2024-04-26T11:56:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 339.393346 secs 2024-04-26T11:56:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 339.393346 secs 2024-04-26T11:56:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 339.393346 secs 2024-04-26T11:56:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:39 smithi033 ceph-mon[27845]: pgmap v888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:39 smithi096 ceph-mon[32753]: pgmap v888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:39 smithi194 ceph-mon[33017]: pgmap v888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:56:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:41 smithi033 ceph-mon[27845]: pgmap v889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:56:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:41 smithi096 ceph-mon[32753]: pgmap v889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:56:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:41 smithi194 ceph-mon[33017]: pgmap v889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:56:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 341.684498 secs 2024-04-26T11:56:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 341.684498 secs 2024-04-26T11:56:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 341.684498 secs 2024-04-26T11:56:44.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:43 smithi033 ceph-mon[27845]: pgmap v890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:44.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 344.393417 secs 2024-04-26T11:56:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:43 smithi194 ceph-mon[33017]: pgmap v890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 344.393417 secs 2024-04-26T11:56:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:43 smithi096 ceph-mon[32753]: pgmap v890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 344.393417 secs 2024-04-26T11:56:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:45 smithi194 ceph-mon[33017]: pgmap v891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:45 smithi033 ceph-mon[27845]: pgmap v891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:45 smithi096 ceph-mon[32753]: pgmap v891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:47.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:47.859+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:47.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:47.859+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:56:47.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:56:47.859+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:56:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:47 smithi194 ceph-mon[33017]: pgmap v892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 346.684555 secs 2024-04-26T11:56:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:47 smithi033 ceph-mon[27845]: pgmap v892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 346.684555 secs 2024-04-26T11:56:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:47 smithi096 ceph-mon[32753]: pgmap v892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 346.684555 secs 2024-04-26T11:56:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 349.393491 secs 2024-04-26T11:56:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 349.393491 secs 2024-04-26T11:56:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 349.393491 secs 2024-04-26T11:56:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:49 smithi194 ceph-mon[33017]: pgmap v893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:49 smithi033 ceph-mon[27845]: pgmap v893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:49 smithi096 ceph-mon[32753]: pgmap v893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:51 smithi033 ceph-mon[27845]: pgmap v894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:51 smithi096 ceph-mon[32753]: pgmap v894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:51 smithi194 ceph-mon[33017]: pgmap v894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 351.684639 secs 2024-04-26T11:56:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 351.684639 secs 2024-04-26T11:56:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 351.684639 secs 2024-04-26T11:56:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:53 smithi033 ceph-mon[27845]: pgmap v895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 354.393627 secs 2024-04-26T11:56:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:53 smithi096 ceph-mon[32753]: pgmap v895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 354.393627 secs 2024-04-26T11:56:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:53 smithi194 ceph-mon[33017]: pgmap v895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 354.393627 secs 2024-04-26T11:56:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:55 smithi096 ceph-mon[32753]: pgmap v896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:55 smithi194 ceph-mon[33017]: pgmap v896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:55 smithi033 ceph-mon[27845]: pgmap v896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:56:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:57 smithi096 ceph-mon[32753]: pgmap v897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 356.684722 secs 2024-04-26T11:56:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:57 smithi194 ceph-mon[33017]: pgmap v897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 356.684722 secs 2024-04-26T11:56:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:57 smithi033 ceph-mon[27845]: pgmap v897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:56:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 356.684722 secs 2024-04-26T11:56:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 359.393707 secs 2024-04-26T11:56:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 359.393707 secs 2024-04-26T11:56:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 359.393707 secs 2024-04-26T11:57:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:56:59 smithi194 ceph-mon[33017]: pgmap v898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:56:59 smithi033 ceph-mon[27845]: pgmap v898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:56:59 smithi096 ceph-mon[32753]: pgmap v898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:01 smithi194 ceph-mon[33017]: pgmap v899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:01 smithi033 ceph-mon[27845]: pgmap v899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:01 smithi096 ceph-mon[32753]: pgmap v899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:02.873+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:02.873+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:02.873+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:57:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:02 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 361.684807 secs 2024-04-26T11:57:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:02 smithi194 ceph-mon[33017]: slow request 361.684807 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T11:57:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:02 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 361.684807 secs 2024-04-26T11:57:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:02 smithi033 ceph-mon[27845]: slow request 361.684807 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T11:57:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:02 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 361.684807 secs 2024-04-26T11:57:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:02 smithi096 ceph-mon[32753]: slow request 361.684807 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T11:57:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:03 smithi194 ceph-mon[33017]: pgmap v900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:03 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 364.393769 secs 2024-04-26T11:57:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:03 smithi194 ceph-mon[33017]: slow request 364.393769 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T11:57:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:03 smithi033 ceph-mon[27845]: pgmap v900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:03 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 364.393769 secs 2024-04-26T11:57:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:03 smithi033 ceph-mon[27845]: slow request 364.393769 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T11:57:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:03 smithi096 ceph-mon[32753]: pgmap v900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:03 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 364.393769 secs 2024-04-26T11:57:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:03 smithi096 ceph-mon[32753]: slow request 364.393769 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T11:57:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:05 smithi194 ceph-mon[33017]: pgmap v901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:05 smithi033 ceph-mon[27845]: pgmap v901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:05 smithi096 ceph-mon[32753]: pgmap v901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:07 smithi194 ceph-mon[33017]: pgmap v902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 366.684890 secs 2024-04-26T11:57:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:07 smithi033 ceph-mon[27845]: pgmap v902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 366.684890 secs 2024-04-26T11:57:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:07 smithi096 ceph-mon[32753]: pgmap v902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 366.684890 secs 2024-04-26T11:57:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 369.393896 secs 2024-04-26T11:57:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 369.393896 secs 2024-04-26T11:57:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 369.393896 secs 2024-04-26T11:57:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:09 smithi194 ceph-mon[33017]: pgmap v903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:09 smithi033 ceph-mon[27845]: pgmap v903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:09 smithi096 ceph-mon[32753]: pgmap v903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:11 smithi194 ceph-mon[33017]: pgmap v904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:11 smithi033 ceph-mon[27845]: pgmap v904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:11 smithi096 ceph-mon[32753]: pgmap v904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 371.684992 secs 2024-04-26T11:57:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 371.684992 secs 2024-04-26T11:57:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 371.684992 secs 2024-04-26T11:57:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:14 smithi194 ceph-mon[33017]: pgmap v905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 374.394019 secs 2024-04-26T11:57:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:14 smithi033 ceph-mon[27845]: pgmap v905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 374.394019 secs 2024-04-26T11:57:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:14 smithi096 ceph-mon[32753]: pgmap v905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 374.394019 secs 2024-04-26T11:57:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:16 smithi194 ceph-mon[33017]: pgmap v906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:16 smithi033 ceph-mon[27845]: pgmap v906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:16 smithi096 ceph-mon[32753]: pgmap v906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:17.889+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:17.889+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:17.889+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:57:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:18 smithi194 ceph-mon[33017]: pgmap v907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 376.685086 secs 2024-04-26T11:57:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:18 smithi033 ceph-mon[27845]: pgmap v907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 376.685086 secs 2024-04-26T11:57:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:18 smithi096 ceph-mon[32753]: pgmap v907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 376.685086 secs 2024-04-26T11:57:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 379.394163 secs 2024-04-26T11:57:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 379.394163 secs 2024-04-26T11:57:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 379.394163 secs 2024-04-26T11:57:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:20 smithi194 ceph-mon[33017]: pgmap v908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:20 smithi033 ceph-mon[27845]: pgmap v908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:20 smithi096 ceph-mon[32753]: pgmap v908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:22 smithi194 ceph-mon[33017]: pgmap v909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:22 smithi033 ceph-mon[27845]: pgmap v909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:22 smithi096 ceph-mon[32753]: pgmap v909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:23.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 381.685147 secs 2024-04-26T11:57:23.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 381.685147 secs 2024-04-26T11:57:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 381.685147 secs 2024-04-26T11:57:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:24 smithi194 ceph-mon[33017]: pgmap v910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 384.394260 secs 2024-04-26T11:57:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:24 smithi033 ceph-mon[27845]: pgmap v910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 384.394260 secs 2024-04-26T11:57:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:24 smithi096 ceph-mon[32753]: pgmap v910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 384.394260 secs 2024-04-26T11:57:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:26 smithi194 ceph-mon[33017]: pgmap v911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:26 smithi033 ceph-mon[27845]: pgmap v911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:26 smithi096 ceph-mon[32753]: pgmap v911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:28 smithi194 ceph-mon[33017]: pgmap v912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 386.685207 secs 2024-04-26T11:57:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:28 smithi033 ceph-mon[27845]: pgmap v912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 386.685207 secs 2024-04-26T11:57:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:28 smithi096 ceph-mon[32753]: pgmap v912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 386.685207 secs 2024-04-26T11:57:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 389.394323 secs 2024-04-26T11:57:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 389.394323 secs 2024-04-26T11:57:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 389.394323 secs 2024-04-26T11:57:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:30 smithi194 ceph-mon[33017]: pgmap v913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:30 smithi033 ceph-mon[27845]: pgmap v913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:30 smithi096 ceph-mon[32753]: pgmap v913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:32 smithi194 ceph-mon[33017]: pgmap v914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:57:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:32 smithi033 ceph-mon[27845]: pgmap v914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:57:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:32 smithi096 ceph-mon[32753]: pgmap v914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:57:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:32.904+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:32.904+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:57:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:32.904+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 391.685283 secs 2024-04-26T11:57:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:57:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:57:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:57:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 391.685283 secs 2024-04-26T11:57:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:57:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:57:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:57:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 391.685283 secs 2024-04-26T11:57:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:57:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:57:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:57:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:34 smithi194 ceph-mon[33017]: pgmap v915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:34 smithi194 ceph-mon[33017]: pgmap v916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 394.394404 secs 2024-04-26T11:57:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:34 smithi033 ceph-mon[27845]: pgmap v915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:34 smithi033 ceph-mon[27845]: pgmap v916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 394.394404 secs 2024-04-26T11:57:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:34 smithi096 ceph-mon[32753]: pgmap v915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:34 smithi096 ceph-mon[32753]: pgmap v916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 394.394404 secs 2024-04-26T11:57:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:36 smithi194 ceph-mon[33017]: pgmap v917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:57:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:36 smithi033 ceph-mon[27845]: pgmap v917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:57:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:36 smithi096 ceph-mon[32753]: pgmap v917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T11:57:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:38 smithi194 ceph-mon[33017]: pgmap v918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 396.685341 secs 2024-04-26T11:57:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:38 smithi033 ceph-mon[27845]: pgmap v918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:38.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 396.685341 secs 2024-04-26T11:57:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:38 smithi096 ceph-mon[32753]: pgmap v918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 396.685341 secs 2024-04-26T11:57:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 399.394482 secs 2024-04-26T11:57:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 399.394482 secs 2024-04-26T11:57:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 399.394482 secs 2024-04-26T11:57:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:40 smithi033 ceph-mon[27845]: pgmap v919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:40 smithi096 ceph-mon[32753]: pgmap v919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:40 smithi194 ceph-mon[33017]: pgmap v919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:42 smithi033 ceph-mon[27845]: pgmap v920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:42 smithi096 ceph-mon[32753]: pgmap v920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:42 smithi194 ceph-mon[33017]: pgmap v920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:57:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 401.685415 secs 2024-04-26T11:57:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 401.685415 secs 2024-04-26T11:57:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 401.685415 secs 2024-04-26T11:57:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:44 smithi033 ceph-mon[27845]: pgmap v921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:57:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 404.394568 secs 2024-04-26T11:57:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:44 smithi096 ceph-mon[32753]: pgmap v921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:57:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 404.394568 secs 2024-04-26T11:57:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:44 smithi194 ceph-mon[33017]: pgmap v921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:57:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 404.394568 secs 2024-04-26T11:57:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:45 smithi033 ceph-mon[27845]: pgmap v922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:45 smithi096 ceph-mon[32753]: pgmap v922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:45 smithi194 ceph-mon[33017]: pgmap v922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:47.916+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:47.916+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:57:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:47.916+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:57:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:47 smithi033 ceph-mon[27845]: pgmap v923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:57:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 406.685495 secs 2024-04-26T11:57:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:47 smithi096 ceph-mon[32753]: pgmap v923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:57:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 406.685495 secs 2024-04-26T11:57:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:47 smithi194 ceph-mon[33017]: pgmap v923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:57:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 406.685495 secs 2024-04-26T11:57:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 409.394647 secs 2024-04-26T11:57:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 409.394647 secs 2024-04-26T11:57:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 409.394647 secs 2024-04-26T11:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:49 smithi033 ceph-mon[27845]: pgmap v924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:49 smithi096 ceph-mon[32753]: pgmap v924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:50.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:49 smithi194 ceph-mon[33017]: pgmap v924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:51.867+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T11:57:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:51.867+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T11:57:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:51.868+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T11:57:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:57:51.868+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T11:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:51 smithi033 ceph-mon[27845]: pgmap v925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:51 smithi096 ceph-mon[32753]: pgmap v925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:51 smithi194 ceph-mon[33017]: pgmap v925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 411.685579 secs 2024-04-26T11:57:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 411.685579 secs 2024-04-26T11:57:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 411.685579 secs 2024-04-26T11:57:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:53 smithi033 ceph-mon[27845]: pgmap v926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 414.394732 secs 2024-04-26T11:57:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:53 smithi096 ceph-mon[32753]: pgmap v926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 414.394732 secs 2024-04-26T11:57:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:53 smithi194 ceph-mon[33017]: pgmap v926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:57:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 414.394732 secs 2024-04-26T11:57:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:55 smithi033 ceph-mon[27845]: pgmap v927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:55 smithi096 ceph-mon[32753]: pgmap v927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:55 smithi194 ceph-mon[33017]: pgmap v927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:57:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:57 smithi033 ceph-mon[27845]: pgmap v928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:57:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 416.685657 secs 2024-04-26T11:57:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:57 smithi096 ceph-mon[32753]: pgmap v928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:57:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 416.685657 secs 2024-04-26T11:57:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:57 smithi194 ceph-mon[33017]: pgmap v928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:57:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 416.685657 secs 2024-04-26T11:57:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:58 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 419.394795 secs 2024-04-26T11:57:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:58 smithi096 ceph-mon[32753]: slow request 361.122584 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T11:57:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:58 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 419.394795 secs 2024-04-26T11:57:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:58 smithi194 ceph-mon[33017]: slow request 361.122584 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T11:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:58 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 419.394795 secs 2024-04-26T11:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:58 smithi033 ceph-mon[27845]: slow request 361.122584 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T11:57:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:57:59 smithi096 ceph-mon[32753]: pgmap v929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:57:59 smithi194 ceph-mon[33017]: pgmap v929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:57:59 smithi033 ceph-mon[27845]: pgmap v929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:01 smithi194 ceph-mon[33017]: pgmap v930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:01 smithi033 ceph-mon[27845]: pgmap v930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:01 smithi096 ceph-mon[32753]: pgmap v930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:02.924+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:02.924+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:02.924+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:58:03.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 421.685743 secs 2024-04-26T11:58:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 421.685743 secs 2024-04-26T11:58:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 421.685743 secs 2024-04-26T11:58:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:03 smithi194 ceph-mon[33017]: pgmap v931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 424.394882 secs 2024-04-26T11:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:03 smithi033 ceph-mon[27845]: pgmap v931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 424.394882 secs 2024-04-26T11:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:03 smithi096 ceph-mon[32753]: pgmap v931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 424.394882 secs 2024-04-26T11:58:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:05 smithi194 ceph-mon[33017]: pgmap v932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:05 smithi033 ceph-mon[27845]: pgmap v932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:05 smithi096 ceph-mon[32753]: pgmap v932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:07 smithi194 ceph-mon[33017]: pgmap v933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:08.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 426.685824 secs 2024-04-26T11:58:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:07 smithi033 ceph-mon[27845]: pgmap v933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 426.685824 secs 2024-04-26T11:58:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:07 smithi096 ceph-mon[32753]: pgmap v933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 426.685824 secs 2024-04-26T11:58:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 429.395034 secs 2024-04-26T11:58:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 429.395034 secs 2024-04-26T11:58:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 429.395034 secs 2024-04-26T11:58:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:09 smithi194 ceph-mon[33017]: pgmap v934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:09 smithi033 ceph-mon[27845]: pgmap v934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:09 smithi096 ceph-mon[32753]: pgmap v934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:11 smithi194 ceph-mon[33017]: pgmap v935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:11 smithi033 ceph-mon[27845]: pgmap v935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:11 smithi096 ceph-mon[32753]: pgmap v935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 431.685908 secs 2024-04-26T11:58:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 431.685908 secs 2024-04-26T11:58:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 431.685908 secs 2024-04-26T11:58:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:13 smithi194 ceph-mon[33017]: pgmap v936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 434.395133 secs 2024-04-26T11:58:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:13 smithi033 ceph-mon[27845]: pgmap v936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 434.395133 secs 2024-04-26T11:58:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:13 smithi096 ceph-mon[32753]: pgmap v936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 434.395133 secs 2024-04-26T11:58:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:15 smithi194 ceph-mon[33017]: pgmap v937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:15 smithi033 ceph-mon[27845]: pgmap v937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:15 smithi096 ceph-mon[32753]: pgmap v937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:17.927+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:17.927+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:17.927+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:58:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:17 smithi194 ceph-mon[33017]: pgmap v938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 436.686004 secs 2024-04-26T11:58:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:17 smithi033 ceph-mon[27845]: pgmap v938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 436.686004 secs 2024-04-26T11:58:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:17 smithi096 ceph-mon[32753]: pgmap v938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 436.686004 secs 2024-04-26T11:58:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 439.395212 secs 2024-04-26T11:58:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 439.395212 secs 2024-04-26T11:58:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 439.395212 secs 2024-04-26T11:58:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:19 smithi194 ceph-mon[33017]: pgmap v939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:19 smithi033 ceph-mon[27845]: pgmap v939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:19 smithi096 ceph-mon[32753]: pgmap v939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:21 smithi194 ceph-mon[33017]: pgmap v940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:21 smithi033 ceph-mon[27845]: pgmap v940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:21 smithi096 ceph-mon[32753]: pgmap v940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:23.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 441.686083 secs 2024-04-26T11:58:23.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 441.686083 secs 2024-04-26T11:58:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 441.686083 secs 2024-04-26T11:58:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:23 smithi194 ceph-mon[33017]: pgmap v941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 444.395298 secs 2024-04-26T11:58:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:23 smithi033 ceph-mon[27845]: pgmap v941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 444.395298 secs 2024-04-26T11:58:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:23 smithi096 ceph-mon[32753]: pgmap v941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 444.395298 secs 2024-04-26T11:58:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:25 smithi194 ceph-mon[33017]: pgmap v942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:25 smithi033 ceph-mon[27845]: pgmap v942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:25 smithi096 ceph-mon[32753]: pgmap v942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:27 smithi194 ceph-mon[33017]: pgmap v943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 446.686167 secs 2024-04-26T11:58:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:27 smithi033 ceph-mon[27845]: pgmap v943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 446.686167 secs 2024-04-26T11:58:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:27 smithi096 ceph-mon[32753]: pgmap v943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T11:58:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 446.686167 secs 2024-04-26T11:58:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 449.395365 secs 2024-04-26T11:58:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 449.395365 secs 2024-04-26T11:58:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 449.395365 secs 2024-04-26T11:58:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:29 smithi194 ceph-mon[33017]: pgmap v944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:30.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:29 smithi033 ceph-mon[27845]: pgmap v944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:29 smithi096 ceph-mon[32753]: pgmap v944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:31 smithi194 ceph-mon[33017]: pgmap v945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:31 smithi033 ceph-mon[27845]: pgmap v945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:31 smithi096 ceph-mon[32753]: pgmap v945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:32.942+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:32.942+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:32.942+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:58:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 451.686271 secs 2024-04-26T11:58:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 451.686271 secs 2024-04-26T11:58:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 451.686271 secs 2024-04-26T11:58:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:33 smithi194 ceph-mon[33017]: pgmap v946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:58:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 454.395464 secs 2024-04-26T11:58:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:33 smithi033 ceph-mon[27845]: pgmap v946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:58:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 454.395464 secs 2024-04-26T11:58:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:33 smithi096 ceph-mon[32753]: pgmap v946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:58:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 454.395464 secs 2024-04-26T11:58:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:58:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:58:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:58:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:58:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:58:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:58:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:58:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:58:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:58:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:35 smithi194 ceph-mon[33017]: pgmap v947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:35 smithi194 ceph-mon[33017]: pgmap v948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-04-26T11:58:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:35 smithi033 ceph-mon[27845]: pgmap v947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:35 smithi033 ceph-mon[27845]: pgmap v948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-04-26T11:58:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:35 smithi096 ceph-mon[32753]: pgmap v947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:35 smithi096 ceph-mon[32753]: pgmap v948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-04-26T11:58:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:37 smithi194 ceph-mon[33017]: pgmap v949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 456.686352 secs 2024-04-26T11:58:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:37 smithi033 ceph-mon[27845]: pgmap v949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 456.686352 secs 2024-04-26T11:58:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:37 smithi096 ceph-mon[32753]: pgmap v949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 456.686352 secs 2024-04-26T11:58:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 459.395622 secs 2024-04-26T11:58:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 459.395622 secs 2024-04-26T11:58:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 459.395622 secs 2024-04-26T11:58:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:39 smithi194 ceph-mon[33017]: pgmap v950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:39 smithi033 ceph-mon[27845]: pgmap v950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:39 smithi096 ceph-mon[32753]: pgmap v950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:41 smithi033 ceph-mon[27845]: pgmap v951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:41 smithi096 ceph-mon[32753]: pgmap v951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:41 smithi194 ceph-mon[33017]: pgmap v951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 461.686439 secs 2024-04-26T11:58:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 461.686439 secs 2024-04-26T11:58:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 461.686439 secs 2024-04-26T11:58:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:43 smithi033 ceph-mon[27845]: pgmap v952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 464.395766 secs 2024-04-26T11:58:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:43 smithi096 ceph-mon[32753]: pgmap v952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 464.395766 secs 2024-04-26T11:58:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:43 smithi194 ceph-mon[33017]: pgmap v952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T11:58:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 464.395766 secs 2024-04-26T11:58:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:45 smithi033 ceph-mon[27845]: pgmap v953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:58:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:45 smithi096 ceph-mon[32753]: pgmap v953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:58:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:45 smithi194 ceph-mon[33017]: pgmap v953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:58:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:47.957+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:47.957+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:58:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:58:47.957+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:58:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:47 smithi033 ceph-mon[27845]: pgmap v954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 466.686521 secs 2024-04-26T11:58:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:47 smithi096 ceph-mon[32753]: pgmap v954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 466.686521 secs 2024-04-26T11:58:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:47 smithi194 ceph-mon[33017]: pgmap v954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 466.686521 secs 2024-04-26T11:58:49.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 469.395896 secs 2024-04-26T11:58:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 469.395896 secs 2024-04-26T11:58:49.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 469.395896 secs 2024-04-26T11:58:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:49 smithi033 ceph-mon[27845]: pgmap v955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:49 smithi096 ceph-mon[32753]: pgmap v955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:49 smithi194 ceph-mon[33017]: pgmap v955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:51 smithi033 ceph-mon[27845]: pgmap v956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:51 smithi096 ceph-mon[32753]: pgmap v956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:51 smithi194 ceph-mon[33017]: pgmap v956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 471.686605 secs 2024-04-26T11:58:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 471.686605 secs 2024-04-26T11:58:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 471.686605 secs 2024-04-26T11:58:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:53 smithi033 ceph-mon[27845]: pgmap v957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 474.396032 secs 2024-04-26T11:58:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:53 smithi096 ceph-mon[32753]: pgmap v957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 474.396032 secs 2024-04-26T11:58:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:53 smithi194 ceph-mon[33017]: pgmap v957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 474.396032 secs 2024-04-26T11:58:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:55 smithi033 ceph-mon[27845]: pgmap v958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:55 smithi096 ceph-mon[32753]: pgmap v958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:55 smithi194 ceph-mon[33017]: pgmap v958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:58:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:57 smithi033 ceph-mon[27845]: pgmap v959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 476.686687 secs 2024-04-26T11:58:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:57 smithi096 ceph-mon[32753]: pgmap v959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 476.686687 secs 2024-04-26T11:58:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:57 smithi194 ceph-mon[33017]: pgmap v959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:58:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 476.686687 secs 2024-04-26T11:58:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 479.396160 secs 2024-04-26T11:58:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 479.396160 secs 2024-04-26T11:58:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 479.396160 secs 2024-04-26T11:59:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:58:59 smithi194 ceph-mon[33017]: pgmap v960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:58:59 smithi033 ceph-mon[27845]: pgmap v960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:58:59 smithi096 ceph-mon[32753]: pgmap v960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:01 smithi194 ceph-mon[33017]: pgmap v961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:01 smithi033 ceph-mon[27845]: pgmap v961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:01 smithi096 ceph-mon[32753]: pgmap v961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:02.965+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:02.965+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:02.965+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:59:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 481.686772 secs 2024-04-26T11:59:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 481.686772 secs 2024-04-26T11:59:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 481.686772 secs 2024-04-26T11:59:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:03 smithi194 ceph-mon[33017]: pgmap v962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 484.396302 secs 2024-04-26T11:59:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:03 smithi033 ceph-mon[27845]: pgmap v962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 484.396302 secs 2024-04-26T11:59:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:03 smithi096 ceph-mon[32753]: pgmap v962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 484.396302 secs 2024-04-26T11:59:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:05 smithi194 ceph-mon[33017]: pgmap v963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:05 smithi033 ceph-mon[27845]: pgmap v963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:05 smithi096 ceph-mon[32753]: pgmap v963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:07 smithi194 ceph-mon[33017]: pgmap v964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 486.686855 secs 2024-04-26T11:59:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:07 smithi033 ceph-mon[27845]: pgmap v964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 486.686855 secs 2024-04-26T11:59:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:07 smithi096 ceph-mon[32753]: pgmap v964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 486.686855 secs 2024-04-26T11:59:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 489.396395 secs 2024-04-26T11:59:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 489.396395 secs 2024-04-26T11:59:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 489.396395 secs 2024-04-26T11:59:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:09 smithi194 ceph-mon[33017]: pgmap v965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:09 smithi033 ceph-mon[27845]: pgmap v965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:09 smithi096 ceph-mon[32753]: pgmap v965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:11 smithi194 ceph-mon[33017]: pgmap v966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:11 smithi033 ceph-mon[27845]: pgmap v966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:11 smithi096 ceph-mon[32753]: pgmap v966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 491.686940 secs 2024-04-26T11:59:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 491.686940 secs 2024-04-26T11:59:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 491.686940 secs 2024-04-26T11:59:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:13 smithi194 ceph-mon[33017]: pgmap v967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 494.396515 secs 2024-04-26T11:59:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:13 smithi033 ceph-mon[27845]: pgmap v967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 494.396515 secs 2024-04-26T11:59:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:13 smithi096 ceph-mon[32753]: pgmap v967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 494.396515 secs 2024-04-26T11:59:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:16 smithi194 ceph-mon[33017]: pgmap v968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:16 smithi033 ceph-mon[27845]: pgmap v968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:16 smithi096 ceph-mon[32753]: pgmap v968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:17.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:17.976+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:17.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:17.976+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:17.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:17.976+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:59:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:18 smithi194 ceph-mon[33017]: pgmap v969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 496.686997 secs 2024-04-26T11:59:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:18 smithi033 ceph-mon[27845]: pgmap v969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 496.686997 secs 2024-04-26T11:59:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:18 smithi096 ceph-mon[32753]: pgmap v969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 496.686997 secs 2024-04-26T11:59:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 499.396577 secs 2024-04-26T11:59:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 499.396577 secs 2024-04-26T11:59:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 499.396577 secs 2024-04-26T11:59:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:20 smithi033 ceph-mon[27845]: pgmap v970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:20 smithi194 ceph-mon[33017]: pgmap v970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:20 smithi096 ceph-mon[32753]: pgmap v970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:21.883+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T11:59:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:21.883+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T11:59:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:21.884+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T11:59:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:21.884+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T11:59:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:22 smithi194 ceph-mon[33017]: pgmap v971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:22 smithi033 ceph-mon[27845]: pgmap v971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:22 smithi096 ceph-mon[32753]: pgmap v971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:23.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 501.687058 secs 2024-04-26T11:59:23.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 501.687058 secs 2024-04-26T11:59:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 501.687058 secs 2024-04-26T11:59:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:24 smithi194 ceph-mon[33017]: pgmap v972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 504.396710 secs 2024-04-26T11:59:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:24 smithi033 ceph-mon[27845]: pgmap v972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 504.396710 secs 2024-04-26T11:59:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:24 smithi096 ceph-mon[32753]: pgmap v972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 504.396710 secs 2024-04-26T11:59:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:26 smithi194 ceph-mon[33017]: pgmap v973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:26 smithi033 ceph-mon[27845]: pgmap v973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:26 smithi096 ceph-mon[32753]: pgmap v973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:28 smithi194 ceph-mon[33017]: pgmap v974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 506.687117 secs 2024-04-26T11:59:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:28 smithi033 ceph-mon[27845]: pgmap v974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 506.687117 secs 2024-04-26T11:59:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:28 smithi096 ceph-mon[32753]: pgmap v974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 506.687117 secs 2024-04-26T11:59:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 509.396833 secs 2024-04-26T11:59:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 509.396833 secs 2024-04-26T11:59:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 509.396833 secs 2024-04-26T11:59:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:30 smithi194 ceph-mon[33017]: pgmap v975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:30 smithi033 ceph-mon[27845]: pgmap v975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:30 smithi096 ceph-mon[32753]: pgmap v975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:32 smithi194 ceph-mon[33017]: pgmap v976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:32 smithi033 ceph-mon[27845]: pgmap v976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:32 smithi096 ceph-mon[32753]: pgmap v976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:32.992+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:32.992+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:32.992+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:59:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 511.687181 secs 2024-04-26T11:59:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 511.687181 secs 2024-04-26T11:59:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 511.687181 secs 2024-04-26T11:59:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:34 smithi194 ceph-mon[33017]: pgmap v977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 514.396955 secs 2024-04-26T11:59:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:34 smithi033 ceph-mon[27845]: pgmap v977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 514.396955 secs 2024-04-26T11:59:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:34 smithi096 ceph-mon[32753]: pgmap v977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 514.396955 secs 2024-04-26T11:59:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:59:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:59:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T11:59:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:36 smithi194 ceph-mon[33017]: pgmap v978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:36 smithi033 ceph-mon[27845]: pgmap v978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:36 smithi096 ceph-mon[32753]: pgmap v978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:59:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:59:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:59:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:59:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T11:59:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T11:59:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T11:59:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:38 smithi033 ceph-mon[27845]: pgmap v979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:38 smithi033 ceph-mon[27845]: pgmap v980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:59:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:38 smithi033 ceph-mon[27845]: pgmap v981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 516.687240 secs 2024-04-26T11:59:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:38 smithi096 ceph-mon[32753]: pgmap v979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:38 smithi096 ceph-mon[32753]: pgmap v980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:59:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:38 smithi096 ceph-mon[32753]: pgmap v981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 516.687240 secs 2024-04-26T11:59:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:38 smithi194 ceph-mon[33017]: pgmap v979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:38 smithi194 ceph-mon[33017]: pgmap v980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T11:59:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:38 smithi194 ceph-mon[33017]: pgmap v981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 516.687240 secs 2024-04-26T11:59:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 519.397128 secs 2024-04-26T11:59:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 519.397128 secs 2024-04-26T11:59:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 519.397128 secs 2024-04-26T11:59:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:40 smithi033 ceph-mon[27845]: pgmap v982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T11:59:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:40 smithi096 ceph-mon[32753]: pgmap v982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T11:59:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:40 smithi194 ceph-mon[33017]: pgmap v982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T11:59:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:42 smithi033 ceph-mon[27845]: pgmap v983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:42 smithi096 ceph-mon[32753]: pgmap v983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:42 smithi194 ceph-mon[33017]: pgmap v983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 521.687302 secs 2024-04-26T11:59:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 521.687302 secs 2024-04-26T11:59:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 521.687302 secs 2024-04-26T11:59:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:44 smithi033 ceph-mon[27845]: pgmap v984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 524.397188 secs 2024-04-26T11:59:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:44 smithi096 ceph-mon[32753]: pgmap v984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 524.397188 secs 2024-04-26T11:59:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:44 smithi194 ceph-mon[33017]: pgmap v984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 524.397188 secs 2024-04-26T11:59:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:46 smithi033 ceph-mon[27845]: pgmap v985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:46 smithi096 ceph-mon[32753]: pgmap v985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:46 smithi194 ceph-mon[33017]: pgmap v985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T11:59:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:47 smithi194 ceph-mon[33017]: pgmap v986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:59:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:47 smithi033 ceph-mon[27845]: pgmap v986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:59:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:47 smithi096 ceph-mon[32753]: pgmap v986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T11:59:47.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:47.994+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:47.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:47.994+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T11:59:47.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T11:59:47.994+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T11:59:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 526.687382 secs 2024-04-26T11:59:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 526.687382 secs 2024-04-26T11:59:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 526.687382 secs 2024-04-26T11:59:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 529.397330 secs 2024-04-26T11:59:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:49 smithi194 ceph-mon[33017]: pgmap v987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 529.397330 secs 2024-04-26T11:59:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:49 smithi033 ceph-mon[27845]: pgmap v987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 529.397330 secs 2024-04-26T11:59:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:49 smithi096 ceph-mon[32753]: pgmap v987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:51 smithi194 ceph-mon[33017]: pgmap v988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:51 smithi033 ceph-mon[27845]: pgmap v988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:51 smithi096 ceph-mon[32753]: pgmap v988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:53.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 531.687442 secs 2024-04-26T11:59:53.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 531.687442 secs 2024-04-26T11:59:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 531.687442 secs 2024-04-26T11:59:54.352 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:54 smithi033 ceph-mon[27845]: pgmap v989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:54.352 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 534.397399 secs 2024-04-26T11:59:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:54 smithi194 ceph-mon[33017]: pgmap v989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 534.397399 secs 2024-04-26T11:59:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:54 smithi096 ceph-mon[32753]: pgmap v989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 534.397399 secs 2024-04-26T11:59:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:56 smithi194 ceph-mon[33017]: pgmap v990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:56 smithi033 ceph-mon[27845]: pgmap v990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:56 smithi096 ceph-mon[32753]: pgmap v990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T11:59:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:58 smithi194 ceph-mon[33017]: pgmap v991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 536.687522 secs 2024-04-26T11:59:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:58 smithi033 ceph-mon[27845]: pgmap v991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 536.687522 secs 2024-04-26T11:59:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:58 smithi096 ceph-mon[32753]: pgmap v991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T11:59:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 536.687522 secs 2024-04-26T11:59:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 11:59:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 539.397505 secs 2024-04-26T11:59:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 11:59:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 539.397505 secs 2024-04-26T11:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 11:59:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 539.397505 secs 2024-04-26T12:00:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: pgmap v992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:00:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-04-26T12:00:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.j on smithi033 is in error state 2024-04-26T12:00:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.d on smithi033 is in error state 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.g on smithi033 is in error state 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.a on smithi033 is in error state 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.e on smithi096 is in error state 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.h on smithi096 is in error state 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.l on smithi194 is in error state 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: daemon mds.c on smithi194 is in error state 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2024-04-26T12:00:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:00 smithi194 ceph-mon[33017]: mds.b(mds.0): 1 slow requests are blocked > 180 secs 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: pgmap v992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.j on smithi033 is in error state 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.d on smithi033 is in error state 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.g on smithi033 is in error state 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.a on smithi033 is in error state 2024-04-26T12:00:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.e on smithi096 is in error state 2024-04-26T12:00:00.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.h on smithi096 is in error state 2024-04-26T12:00:00.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.l on smithi194 is in error state 2024-04-26T12:00:00.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: daemon mds.c on smithi194 is in error state 2024-04-26T12:00:00.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-04-26T12:00:00.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2024-04-26T12:00:00.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:00 smithi033 ceph-mon[27845]: mds.b(mds.0): 1 slow requests are blocked > 180 secs 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: pgmap v992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.j on smithi033 is in error state 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.d on smithi033 is in error state 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.g on smithi033 is in error state 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.a on smithi033 is in error state 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.e on smithi096 is in error state 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.h on smithi096 is in error state 2024-04-26T12:00:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.l on smithi194 is in error state 2024-04-26T12:00:00.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: daemon mds.c on smithi194 is in error state 2024-04-26T12:00:00.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-04-26T12:00:00.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2024-04-26T12:00:00.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:00 smithi096 ceph-mon[32753]: mds.b(mds.0): 1 slow requests are blocked > 180 secs 2024-04-26T12:00:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:02 smithi194 ceph-mon[33017]: pgmap v993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:02 smithi033 ceph-mon[27845]: pgmap v993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:02 smithi096 ceph-mon[32753]: pgmap v993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:02.997+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:02.997+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:00:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:02.997+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 541.687607 secs 2024-04-26T12:00:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 541.687607 secs 2024-04-26T12:00:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 541.687607 secs 2024-04-26T12:00:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:04 smithi194 ceph-mon[33017]: pgmap v994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 544.397630 secs 2024-04-26T12:00:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:04 smithi033 ceph-mon[27845]: pgmap v994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 544.397630 secs 2024-04-26T12:00:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:04 smithi096 ceph-mon[32753]: pgmap v994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 544.397630 secs 2024-04-26T12:00:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:06 smithi194 ceph-mon[33017]: pgmap v995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:06 smithi033 ceph-mon[27845]: pgmap v995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:06 smithi096 ceph-mon[32753]: pgmap v995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:08 smithi194 ceph-mon[33017]: pgmap v996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 546.687686 secs 2024-04-26T12:00:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:08 smithi033 ceph-mon[27845]: pgmap v996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 546.687686 secs 2024-04-26T12:00:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:08 smithi096 ceph-mon[32753]: pgmap v996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 546.687686 secs 2024-04-26T12:00:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 549.397785 secs 2024-04-26T12:00:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 549.397785 secs 2024-04-26T12:00:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 549.397785 secs 2024-04-26T12:00:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:10 smithi194 ceph-mon[33017]: pgmap v997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:10 smithi033 ceph-mon[27845]: pgmap v997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:10 smithi096 ceph-mon[32753]: pgmap v997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:12 smithi194 ceph-mon[33017]: pgmap v998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:12 smithi033 ceph-mon[27845]: pgmap v998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:12 smithi096 ceph-mon[32753]: pgmap v998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 551.687770 secs 2024-04-26T12:00:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 551.687770 secs 2024-04-26T12:00:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 551.687770 secs 2024-04-26T12:00:14.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:14 smithi194 ceph-mon[33017]: pgmap v999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:14.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 554.397910 secs 2024-04-26T12:00:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:14 smithi033 ceph-mon[27845]: pgmap v999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 554.397910 secs 2024-04-26T12:00:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:14 smithi096 ceph-mon[32753]: pgmap v999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 554.397910 secs 2024-04-26T12:00:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:16 smithi194 ceph-mon[33017]: pgmap v1000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:16 smithi033 ceph-mon[27845]: pgmap v1000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:16 smithi096 ceph-mon[32753]: pgmap v1000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:18.013+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:18.014+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:18.014+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:00:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:18 smithi033 ceph-mon[27845]: pgmap v1001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 556.687853 secs 2024-04-26T12:00:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:18 smithi096 ceph-mon[32753]: pgmap v1001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 556.687853 secs 2024-04-26T12:00:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:18 smithi194 ceph-mon[33017]: pgmap v1001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 556.687853 secs 2024-04-26T12:00:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 559.398051 secs 2024-04-26T12:00:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 559.398051 secs 2024-04-26T12:00:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 559.398051 secs 2024-04-26T12:00:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:20 smithi033 ceph-mon[27845]: pgmap v1002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:20 smithi096 ceph-mon[32753]: pgmap v1002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:20 smithi194 ceph-mon[33017]: pgmap v1002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:22 smithi033 ceph-mon[27845]: pgmap v1003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:22 smithi096 ceph-mon[32753]: pgmap v1003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:22 smithi194 ceph-mon[33017]: pgmap v1003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 561.687937 secs 2024-04-26T12:00:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 561.687937 secs 2024-04-26T12:00:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 561.687937 secs 2024-04-26T12:00:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:24 smithi033 ceph-mon[27845]: pgmap v1004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 564.398176 secs 2024-04-26T12:00:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:24 smithi096 ceph-mon[32753]: pgmap v1004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 564.398176 secs 2024-04-26T12:00:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:24 smithi194 ceph-mon[33017]: pgmap v1004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 564.398176 secs 2024-04-26T12:00:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:26 smithi033 ceph-mon[27845]: pgmap v1005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:26 smithi096 ceph-mon[32753]: pgmap v1005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:26 smithi194 ceph-mon[33017]: pgmap v1005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:28 smithi033 ceph-mon[27845]: pgmap v1006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 566.688022 secs 2024-04-26T12:00:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:28 smithi096 ceph-mon[32753]: pgmap v1006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 566.688022 secs 2024-04-26T12:00:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:28 smithi194 ceph-mon[33017]: pgmap v1006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 566.688022 secs 2024-04-26T12:00:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 569.398299 secs 2024-04-26T12:00:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 569.398299 secs 2024-04-26T12:00:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 569.398299 secs 2024-04-26T12:00:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:30 smithi033 ceph-mon[27845]: pgmap v1007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:30 smithi096 ceph-mon[32753]: pgmap v1007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:30 smithi194 ceph-mon[33017]: pgmap v1007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:31 smithi096 ceph-mon[32753]: pgmap v1008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:31 smithi194 ceph-mon[33017]: pgmap v1008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:31 smithi033 ceph-mon[27845]: pgmap v1008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 571.688103 secs 2024-04-26T12:00:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 571.688103 secs 2024-04-26T12:00:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:33.021+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:33.021+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:33.021+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:00:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 571.688103 secs 2024-04-26T12:00:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:33 smithi033 ceph-mon[27845]: pgmap v1009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 574.398424 secs 2024-04-26T12:00:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:33 smithi096 ceph-mon[32753]: pgmap v1009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 574.398424 secs 2024-04-26T12:00:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:33 smithi194 ceph-mon[33017]: pgmap v1009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 574.398424 secs 2024-04-26T12:00:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:36 smithi194 ceph-mon[33017]: pgmap v1010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:36 smithi033 ceph-mon[27845]: pgmap v1010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:36 smithi096 ceph-mon[32753]: pgmap v1010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:37 smithi033 ceph-mon[27845]: pgmap v1011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:00:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 576.688185 secs 2024-04-26T12:00:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:37 smithi096 ceph-mon[32753]: pgmap v1011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:00:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 576.688185 secs 2024-04-26T12:00:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:37 smithi194 ceph-mon[33017]: pgmap v1011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:00:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 576.688185 secs 2024-04-26T12:00:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 579.398502 secs 2024-04-26T12:00:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 579.398502 secs 2024-04-26T12:00:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 579.398502 secs 2024-04-26T12:00:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:39 smithi033 ceph-mon[27845]: pgmap v1012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:00:40.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:00:40.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:39 smithi096 ceph-mon[32753]: pgmap v1012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:00:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:00:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:39 smithi194 ceph-mon[33017]: pgmap v1012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:00:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:00:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:00:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:40 smithi033 ceph-mon[27845]: pgmap v1013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T12:00:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:40 smithi033 ceph-mon[27845]: pgmap v1014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:41.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:40 smithi096 ceph-mon[32753]: pgmap v1013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T12:00:41.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:40 smithi096 ceph-mon[32753]: pgmap v1014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:40 smithi194 ceph-mon[33017]: pgmap v1013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T12:00:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:40 smithi194 ceph-mon[33017]: pgmap v1014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:42 smithi033 ceph-mon[27845]: pgmap v1015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 581.688244 secs 2024-04-26T12:00:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:42 smithi096 ceph-mon[32753]: pgmap v1015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 581.688244 secs 2024-04-26T12:00:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:42 smithi194 ceph-mon[33017]: pgmap v1015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 581.688244 secs 2024-04-26T12:00:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 584.398606 secs 2024-04-26T12:00:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 584.398606 secs 2024-04-26T12:00:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 584.398606 secs 2024-04-26T12:00:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:44 smithi033 ceph-mon[27845]: pgmap v1016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:00:45.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:44 smithi096 ceph-mon[32753]: pgmap v1016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:00:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:44 smithi194 ceph-mon[33017]: pgmap v1016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:00:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:46 smithi033 ceph-mon[27845]: pgmap v1017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:47.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:46 smithi096 ceph-mon[32753]: pgmap v1017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:46 smithi194 ceph-mon[33017]: pgmap v1017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:48.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:48.031+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:00:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:00:48.031+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:00:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 586.688325 secs 2024-04-26T12:00:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 586.688325 secs 2024-04-26T12:00:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 586.688325 secs 2024-04-26T12:00:49.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:48 smithi194 ceph-mon[33017]: pgmap v1018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:49.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 589.398733 secs 2024-04-26T12:00:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:48 smithi033 ceph-mon[27845]: pgmap v1018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 589.398733 secs 2024-04-26T12:00:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:48 smithi096 ceph-mon[32753]: pgmap v1018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:00:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 589.398733 secs 2024-04-26T12:00:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:50 smithi194 ceph-mon[33017]: pgmap v1019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:00:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:50 smithi033 ceph-mon[27845]: pgmap v1019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:00:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:50 smithi096 ceph-mon[32753]: pgmap v1019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:00:53.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:52 smithi033 ceph-mon[27845]: pgmap v1020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:53.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 591.688410 secs 2024-04-26T12:00:53.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:52 smithi096 ceph-mon[32753]: pgmap v1020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:53.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 591.688410 secs 2024-04-26T12:00:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:52 smithi194 ceph-mon[33017]: pgmap v1020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 591.688410 secs 2024-04-26T12:00:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 594.398860 secs 2024-04-26T12:00:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 594.398860 secs 2024-04-26T12:00:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 594.398860 secs 2024-04-26T12:00:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:54 smithi194 ceph-mon[33017]: pgmap v1021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:54 smithi033 ceph-mon[27845]: pgmap v1021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:54 smithi096 ceph-mon[32753]: pgmap v1021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:56 smithi194 ceph-mon[33017]: pgmap v1022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:56 smithi033 ceph-mon[27845]: pgmap v1022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:56 smithi096 ceph-mon[32753]: pgmap v1022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:00:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 596.688491 secs 2024-04-26T12:00:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 596.688491 secs 2024-04-26T12:00:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 596.688491 secs 2024-04-26T12:00:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:58 smithi194 ceph-mon[33017]: pgmap v1023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:00:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 599.398941 secs 2024-04-26T12:00:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:58 smithi033 ceph-mon[27845]: pgmap v1023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:00:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 599.398941 secs 2024-04-26T12:00:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:58 smithi096 ceph-mon[32753]: pgmap v1023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:00:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:00:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 599.398941 secs 2024-04-26T12:01:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:00 smithi194 ceph-mon[33017]: pgmap v1024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:00 smithi033 ceph-mon[27845]: pgmap v1024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:00 smithi096 ceph-mon[32753]: pgmap v1024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:03.040+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:03.040+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:03.040+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:01:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:02 smithi194 ceph-mon[33017]: pgmap v1025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 601.688574 secs 2024-04-26T12:01:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:02 smithi033 ceph-mon[27845]: pgmap v1025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 601.688574 secs 2024-04-26T12:01:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:02 smithi096 ceph-mon[32753]: pgmap v1025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 601.688574 secs 2024-04-26T12:01:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 604.399058 secs 2024-04-26T12:01:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 604.399058 secs 2024-04-26T12:01:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 604.399058 secs 2024-04-26T12:01:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:04 smithi194 ceph-mon[33017]: pgmap v1026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:04 smithi033 ceph-mon[27845]: pgmap v1026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:04 smithi096 ceph-mon[32753]: pgmap v1026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:07 smithi194 ceph-mon[33017]: pgmap v1027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:07 smithi033 ceph-mon[27845]: pgmap v1027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:07 smithi096 ceph-mon[32753]: pgmap v1027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 606.688629 secs 2024-04-26T12:01:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 606.688629 secs 2024-04-26T12:01:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 606.688629 secs 2024-04-26T12:01:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:09 smithi033 ceph-mon[27845]: pgmap v1028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 609.399179 secs 2024-04-26T12:01:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:09 smithi194 ceph-mon[33017]: pgmap v1028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 609.399179 secs 2024-04-26T12:01:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:09 smithi096 ceph-mon[32753]: pgmap v1028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 609.399179 secs 2024-04-26T12:01:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:11 smithi194 ceph-mon[33017]: pgmap v1029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:11 smithi033 ceph-mon[27845]: pgmap v1029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:11 smithi096 ceph-mon[32753]: pgmap v1029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:13 smithi194 ceph-mon[33017]: pgmap v1030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 611.688712 secs 2024-04-26T12:01:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:13 smithi033 ceph-mon[27845]: pgmap v1030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 611.688712 secs 2024-04-26T12:01:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:13 smithi096 ceph-mon[32753]: pgmap v1030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 611.688712 secs 2024-04-26T12:01:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 614.399299 secs 2024-04-26T12:01:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 614.399299 secs 2024-04-26T12:01:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 614.399299 secs 2024-04-26T12:01:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:15 smithi194 ceph-mon[33017]: pgmap v1031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:15 smithi033 ceph-mon[27845]: pgmap v1031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:15 smithi096 ceph-mon[32753]: pgmap v1031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:17 smithi194 ceph-mon[33017]: pgmap v1032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:17 smithi033 ceph-mon[27845]: pgmap v1032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:17 smithi096 ceph-mon[32753]: pgmap v1032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:18.057 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:18.056+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:18.057 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:18.056+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:18.057 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:18.056+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:01:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 616.688795 secs 2024-04-26T12:01:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 616.688795 secs 2024-04-26T12:01:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 616.688795 secs 2024-04-26T12:01:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:19 smithi194 ceph-mon[33017]: pgmap v1033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 619.399409 secs 2024-04-26T12:01:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:19 smithi033 ceph-mon[27845]: pgmap v1033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 619.399409 secs 2024-04-26T12:01:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:19 smithi096 ceph-mon[32753]: pgmap v1033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 619.399409 secs 2024-04-26T12:01:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:21 smithi194 ceph-mon[33017]: pgmap v1034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:21 smithi033 ceph-mon[27845]: pgmap v1034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:21 smithi096 ceph-mon[32753]: pgmap v1034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:23 smithi194 ceph-mon[33017]: pgmap v1035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 621.688878 secs 2024-04-26T12:01:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:23 smithi033 ceph-mon[27845]: pgmap v1035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 621.688878 secs 2024-04-26T12:01:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:23 smithi096 ceph-mon[32753]: pgmap v1035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 621.688878 secs 2024-04-26T12:01:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 624.399526 secs 2024-04-26T12:01:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 624.399526 secs 2024-04-26T12:01:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 624.399526 secs 2024-04-26T12:01:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:25 smithi194 ceph-mon[33017]: pgmap v1036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:25 smithi033 ceph-mon[27845]: pgmap v1036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:25 smithi096 ceph-mon[32753]: pgmap v1036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:27 smithi194 ceph-mon[33017]: pgmap v1037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:27 smithi033 ceph-mon[27845]: pgmap v1037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:27 smithi096 ceph-mon[32753]: pgmap v1037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 626.688958 secs 2024-04-26T12:01:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 626.688958 secs 2024-04-26T12:01:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 626.688958 secs 2024-04-26T12:01:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:29 smithi194 ceph-mon[33017]: pgmap v1038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 629.399645 secs 2024-04-26T12:01:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:29 smithi033 ceph-mon[27845]: pgmap v1038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 629.399645 secs 2024-04-26T12:01:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:29 smithi096 ceph-mon[32753]: pgmap v1038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 629.399645 secs 2024-04-26T12:01:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:31 smithi033 ceph-mon[27845]: pgmap v1039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:31 smithi096 ceph-mon[32753]: pgmap v1039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:31 smithi194 ceph-mon[33017]: pgmap v1039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:01:33.072 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:33.072+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:33.072+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:33.072+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:01:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:33 smithi033 ceph-mon[27845]: pgmap v1040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 631.689049 secs 2024-04-26T12:01:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:33 smithi096 ceph-mon[32753]: pgmap v1040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 631.689049 secs 2024-04-26T12:01:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:33 smithi194 ceph-mon[33017]: pgmap v1040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 631.689049 secs 2024-04-26T12:01:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 634.399766 secs 2024-04-26T12:01:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 634.399766 secs 2024-04-26T12:01:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 634.399766 secs 2024-04-26T12:01:35.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:35 smithi033 ceph-mon[27845]: pgmap v1041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:35 smithi096 ceph-mon[32753]: pgmap v1041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:35 smithi194 ceph-mon[33017]: pgmap v1041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:37 smithi033 ceph-mon[27845]: pgmap v1042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:37 smithi096 ceph-mon[32753]: pgmap v1042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:37 smithi194 ceph-mon[33017]: pgmap v1042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 636.689108 secs 2024-04-26T12:01:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 636.689108 secs 2024-04-26T12:01:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 636.689108 secs 2024-04-26T12:01:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:39 smithi033 ceph-mon[27845]: pgmap v1043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 639.399910 secs 2024-04-26T12:01:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:39 smithi096 ceph-mon[32753]: pgmap v1043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 639.399910 secs 2024-04-26T12:01:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:39 smithi194 ceph-mon[33017]: pgmap v1043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 639.399910 secs 2024-04-26T12:01:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:01:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:01:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:01:41.424 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:41 smithi194 ceph-mon[33017]: pgmap v1044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:41.424 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.424 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.424 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.424 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:41 smithi033 ceph-mon[27845]: pgmap v1044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:41 smithi096 ceph-mon[32753]: pgmap v1044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:43 smithi033 ceph-mon[27845]: pgmap v1045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 641.689175 secs 2024-04-26T12:01:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:43 smithi096 ceph-mon[32753]: pgmap v1045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 641.689175 secs 2024-04-26T12:01:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:43 smithi194 ceph-mon[33017]: pgmap v1045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 641.689175 secs 2024-04-26T12:01:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 644.399997 secs 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 644.399997 secs 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 644.399997 secs 2024-04-26T12:01:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:44.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:45 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:01:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:45 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:01:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:45 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:01:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:45 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:01:45.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:45 smithi033 ceph-mon[27845]: pgmap v1046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:45.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:45.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:45 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:01:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:45 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:01:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:45 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:01:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:45 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:01:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:45 smithi096 ceph-mon[32753]: pgmap v1046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:45 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:01:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:45 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:01:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:45 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:01:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:45 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:01:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:45 smithi194 ceph-mon[33017]: pgmap v1046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:01:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: pgmap v1047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: pgmap v1048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: pgmap v1049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: pgmap v1047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:01:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:01:46.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:01:46.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: pgmap v1048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:01:46.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: pgmap v1049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:46.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: pgmap v1047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:01:46.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:01:46.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:01:46.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:01:46.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:01:46.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: pgmap v1048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:01:46.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: pgmap v1049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:46.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:01:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 646.689235 secs 2024-04-26T12:01:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 646.689235 secs 2024-04-26T12:01:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:48.083+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:48.085 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:48.083+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:01:48.085 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:01:48.083+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:01:48.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 646.689235 secs 2024-04-26T12:01:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 649.400142 secs 2024-04-26T12:01:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:48 smithi033 ceph-mon[27845]: pgmap v1050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:01:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 649.400142 secs 2024-04-26T12:01:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:48 smithi096 ceph-mon[32753]: pgmap v1050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:01:49.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 649.400142 secs 2024-04-26T12:01:49.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:48 smithi194 ceph-mon[33017]: pgmap v1050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:01:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:51 smithi033 ceph-mon[27845]: pgmap v1051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:51 smithi096 ceph-mon[32753]: pgmap v1051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:51 smithi194 ceph-mon[33017]: pgmap v1051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:53 smithi033 ceph-mon[27845]: pgmap v1052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 651.689316 secs 2024-04-26T12:01:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:53 smithi096 ceph-mon[32753]: pgmap v1052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 651.689316 secs 2024-04-26T12:01:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:53 smithi194 ceph-mon[33017]: pgmap v1052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:01:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 651.689316 secs 2024-04-26T12:01:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 654.400233 secs 2024-04-26T12:01:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 654.400233 secs 2024-04-26T12:01:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 654.400233 secs 2024-04-26T12:01:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:55 smithi033 ceph-mon[27845]: pgmap v1053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:01:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:55 smithi096 ceph-mon[32753]: pgmap v1053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:01:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:55 smithi194 ceph-mon[33017]: pgmap v1053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:01:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:57 smithi033 ceph-mon[27845]: pgmap v1054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:01:57.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:57 smithi096 ceph-mon[32753]: pgmap v1054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:01:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:57 smithi194 ceph-mon[33017]: pgmap v1054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:01:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 656.689398 secs 2024-04-26T12:01:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 656.689398 secs 2024-04-26T12:01:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 656.689398 secs 2024-04-26T12:01:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 659.400350 secs 2024-04-26T12:01:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:01:59 smithi033 ceph-mon[27845]: pgmap v1055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 659.400350 secs 2024-04-26T12:01:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:01:59 smithi096 ceph-mon[32753]: pgmap v1055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:01:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 659.400350 secs 2024-04-26T12:01:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:01:59 smithi194 ceph-mon[33017]: pgmap v1055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:01 smithi033 ceph-mon[27845]: pgmap v1056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:01 smithi096 ceph-mon[32753]: pgmap v1056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:01 smithi194 ceph-mon[33017]: pgmap v1056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:03.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:03.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:03.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:02:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:03 smithi033 ceph-mon[27845]: pgmap v1057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 661.689480 secs 2024-04-26T12:02:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:03 smithi096 ceph-mon[32753]: pgmap v1057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 661.689480 secs 2024-04-26T12:02:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:03 smithi194 ceph-mon[33017]: pgmap v1057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 661.689480 secs 2024-04-26T12:02:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 664.400462 secs 2024-04-26T12:02:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 664.400462 secs 2024-04-26T12:02:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 664.400462 secs 2024-04-26T12:02:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:05 smithi033 ceph-mon[27845]: pgmap v1058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:05 smithi096 ceph-mon[32753]: pgmap v1058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:05 smithi194 ceph-mon[33017]: pgmap v1058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:07 smithi033 ceph-mon[27845]: pgmap v1059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:07 smithi096 ceph-mon[32753]: pgmap v1059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:07 smithi194 ceph-mon[33017]: pgmap v1059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 666.689563 secs 2024-04-26T12:02:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 666.689563 secs 2024-04-26T12:02:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 666.689563 secs 2024-04-26T12:02:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 669.400587 secs 2024-04-26T12:02:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:09 smithi194 ceph-mon[33017]: pgmap v1060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 669.400587 secs 2024-04-26T12:02:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:09 smithi033 ceph-mon[27845]: pgmap v1060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 669.400587 secs 2024-04-26T12:02:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:09 smithi096 ceph-mon[32753]: pgmap v1060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:10 smithi194 ceph-mon[33017]: pgmap v1061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:10 smithi033 ceph-mon[27845]: pgmap v1061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:10 smithi096 ceph-mon[32753]: pgmap v1061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:13.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:13 smithi033 ceph-mon[27845]: pgmap v1062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:13.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 671.689648 secs 2024-04-26T12:02:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:13 smithi096 ceph-mon[32753]: pgmap v1062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 671.689648 secs 2024-04-26T12:02:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:13 smithi194 ceph-mon[33017]: pgmap v1062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 671.689648 secs 2024-04-26T12:02:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 674.400713 secs 2024-04-26T12:02:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 674.400713 secs 2024-04-26T12:02:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 674.400713 secs 2024-04-26T12:02:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:15 smithi033 ceph-mon[27845]: pgmap v1063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:15 smithi096 ceph-mon[32753]: pgmap v1063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:15 smithi194 ceph-mon[33017]: pgmap v1063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:17 smithi033 ceph-mon[27845]: pgmap v1064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:17 smithi096 ceph-mon[32753]: pgmap v1064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:17 smithi194 ceph-mon[33017]: pgmap v1064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:18.110+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:18.110+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:02:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:18.110+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 676.689731 secs 2024-04-26T12:02:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 676.689731 secs 2024-04-26T12:02:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 676.689731 secs 2024-04-26T12:02:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 679.400857 secs 2024-04-26T12:02:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:19 smithi033 ceph-mon[27845]: pgmap v1065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 679.400857 secs 2024-04-26T12:02:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:19 smithi096 ceph-mon[32753]: pgmap v1065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 679.400857 secs 2024-04-26T12:02:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:19 smithi194 ceph-mon[33017]: pgmap v1065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:21 smithi033 ceph-mon[27845]: pgmap v1066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:21 smithi096 ceph-mon[32753]: pgmap v1066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:21 smithi194 ceph-mon[33017]: pgmap v1066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:21.912+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:02:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:21.912+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:02:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:21.913+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:02:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:21.913+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:02:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:23 smithi033 ceph-mon[27845]: pgmap v1067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 681.689817 secs 2024-04-26T12:02:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:23 smithi096 ceph-mon[32753]: pgmap v1067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 681.689817 secs 2024-04-26T12:02:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:23 smithi194 ceph-mon[33017]: pgmap v1067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 681.689817 secs 2024-04-26T12:02:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 684.400983 secs 2024-04-26T12:02:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 684.400983 secs 2024-04-26T12:02:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 684.400983 secs 2024-04-26T12:02:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:25 smithi033 ceph-mon[27845]: pgmap v1068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:25 smithi096 ceph-mon[32753]: pgmap v1068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:25 smithi194 ceph-mon[33017]: pgmap v1068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:27 smithi096 ceph-mon[32753]: pgmap v1069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:27 smithi194 ceph-mon[33017]: pgmap v1069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:27 smithi033 ceph-mon[27845]: pgmap v1069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 686.689905 secs 2024-04-26T12:02:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 686.689905 secs 2024-04-26T12:02:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 686.689905 secs 2024-04-26T12:02:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 689.401080 secs 2024-04-26T12:02:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:29 smithi194 ceph-mon[33017]: pgmap v1070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 689.401080 secs 2024-04-26T12:02:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:29 smithi033 ceph-mon[27845]: pgmap v1070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 689.401080 secs 2024-04-26T12:02:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:29 smithi096 ceph-mon[32753]: pgmap v1070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:30 smithi194 ceph-mon[33017]: pgmap v1071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:30 smithi033 ceph-mon[27845]: pgmap v1071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:30 smithi096 ceph-mon[32753]: pgmap v1071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:33.112+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:33.112+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:33.112+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:02:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:33 smithi033 ceph-mon[27845]: pgmap v1072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 691.690006 secs 2024-04-26T12:02:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:33 smithi096 ceph-mon[32753]: pgmap v1072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 691.690006 secs 2024-04-26T12:02:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:33 smithi194 ceph-mon[33017]: pgmap v1072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 691.690006 secs 2024-04-26T12:02:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 694.401198 secs 2024-04-26T12:02:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 694.401198 secs 2024-04-26T12:02:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 694.401198 secs 2024-04-26T12:02:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:35 smithi033 ceph-mon[27845]: pgmap v1073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:35 smithi096 ceph-mon[32753]: pgmap v1073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:35 smithi194 ceph-mon[33017]: pgmap v1073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:37 smithi033 ceph-mon[27845]: pgmap v1074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:37 smithi096 ceph-mon[32753]: pgmap v1074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:37 smithi194 ceph-mon[33017]: pgmap v1074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 696.690094 secs 2024-04-26T12:02:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 696.690094 secs 2024-04-26T12:02:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 696.690094 secs 2024-04-26T12:02:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 699.401264 secs 2024-04-26T12:02:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:39 smithi033 ceph-mon[27845]: pgmap v1075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 699.401264 secs 2024-04-26T12:02:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:39 smithi096 ceph-mon[32753]: pgmap v1075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 699.401264 secs 2024-04-26T12:02:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:39 smithi194 ceph-mon[33017]: pgmap v1075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:41 smithi033 ceph-mon[27845]: pgmap v1076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:41 smithi096 ceph-mon[32753]: pgmap v1076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:41 smithi194 ceph-mon[33017]: pgmap v1076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:42 smithi194 ceph-mon[33017]: pgmap v1077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 701.690181 secs 2024-04-26T12:02:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:42 smithi033 ceph-mon[27845]: pgmap v1077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 701.690181 secs 2024-04-26T12:02:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:42 smithi096 ceph-mon[32753]: pgmap v1077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 701.690181 secs 2024-04-26T12:02:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 704.401389 secs 2024-04-26T12:02:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 704.401389 secs 2024-04-26T12:02:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 704.401389 secs 2024-04-26T12:02:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:44 smithi194 ceph-mon[33017]: pgmap v1078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:44 smithi033 ceph-mon[27845]: pgmap v1078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:44 smithi096 ceph-mon[32753]: pgmap v1078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:02:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:47 smithi033 ceph-mon[27845]: pgmap v1079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:02:47.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:47 smithi096 ceph-mon[32753]: pgmap v1079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:47.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:02:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:47 smithi194 ceph-mon[33017]: pgmap v1079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:02:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:02:48.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:48.117+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:48.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:48.117+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:02:48.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:02:48.118+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:02:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 706.690240 secs 2024-04-26T12:02:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:02:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:02:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:02:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 706.690240 secs 2024-04-26T12:02:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:02:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:02:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:02:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 706.690240 secs 2024-04-26T12:02:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:02:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:02:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:02:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:49 smithi033 ceph-mon[27845]: pgmap v1080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:49 smithi033 ceph-mon[27845]: pgmap v1081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:49.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 709.401538 secs 2024-04-26T12:02:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:49 smithi096 ceph-mon[32753]: pgmap v1080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:49 smithi096 ceph-mon[32753]: pgmap v1081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 709.401538 secs 2024-04-26T12:02:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:49 smithi194 ceph-mon[33017]: pgmap v1080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:02:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:49 smithi194 ceph-mon[33017]: pgmap v1081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:49.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 709.401538 secs 2024-04-26T12:02:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:51 smithi033 ceph-mon[27845]: pgmap v1082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:51 smithi096 ceph-mon[32753]: pgmap v1082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:51 smithi194 ceph-mon[33017]: pgmap v1082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:53 smithi033 ceph-mon[27845]: pgmap v1083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 711.690325 secs 2024-04-26T12:02:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:53 smithi096 ceph-mon[32753]: pgmap v1083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 711.690325 secs 2024-04-26T12:02:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:53 smithi194 ceph-mon[33017]: pgmap v1083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 711.690325 secs 2024-04-26T12:02:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 714.401610 secs 2024-04-26T12:02:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 714.401610 secs 2024-04-26T12:02:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 714.401610 secs 2024-04-26T12:02:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:55 smithi033 ceph-mon[27845]: pgmap v1084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:55 smithi096 ceph-mon[32753]: pgmap v1084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:55 smithi194 ceph-mon[33017]: pgmap v1084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:57 smithi096 ceph-mon[32753]: pgmap v1085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:57 smithi194 ceph-mon[33017]: pgmap v1085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:57 smithi033 ceph-mon[27845]: pgmap v1085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:02:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 716.690407 secs 2024-04-26T12:02:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 716.690407 secs 2024-04-26T12:02:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 716.690407 secs 2024-04-26T12:02:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:59 smithi194 ceph-mon[33017]: pgmap v1086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:02:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 719.401749 secs 2024-04-26T12:02:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:59 smithi033 ceph-mon[27845]: pgmap v1086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:02:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 719.401749 secs 2024-04-26T12:02:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:59 smithi096 ceph-mon[32753]: pgmap v1086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:02:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 719.401749 secs 2024-04-26T12:03:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:00 smithi194 ceph-mon[33017]: pgmap v1087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:00 smithi033 ceph-mon[27845]: pgmap v1087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:00 smithi096 ceph-mon[32753]: pgmap v1087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:02 smithi194 ceph-mon[33017]: pgmap v1088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:02 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 721.690496 secs 2024-04-26T12:03:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:02 smithi194 ceph-mon[33017]: slow request 721.690495 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:03:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:03.125+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:03.125+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:03.125+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:03:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:02 smithi033 ceph-mon[27845]: pgmap v1088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:02 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 721.690496 secs 2024-04-26T12:03:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:02 smithi033 ceph-mon[27845]: slow request 721.690495 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:03:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:02 smithi096 ceph-mon[32753]: pgmap v1088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:02 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 721.690496 secs 2024-04-26T12:03:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:02 smithi096 ceph-mon[32753]: slow request 721.690495 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:03:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:03 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 724.401871 secs 2024-04-26T12:03:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:03 smithi194 ceph-mon[33017]: slow request 724.401870 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:03:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:03 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 724.401871 secs 2024-04-26T12:03:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:03 smithi033 ceph-mon[27845]: slow request 724.401870 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:03:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:03 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 724.401871 secs 2024-04-26T12:03:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:03 smithi096 ceph-mon[32753]: slow request 724.401870 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:03:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:04 smithi194 ceph-mon[33017]: pgmap v1089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:04 smithi033 ceph-mon[27845]: pgmap v1089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:04 smithi096 ceph-mon[32753]: pgmap v1089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:06 smithi033 ceph-mon[27845]: pgmap v1090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:07.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:06 smithi096 ceph-mon[32753]: pgmap v1090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:06 smithi194 ceph-mon[33017]: pgmap v1090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 726.690578 secs 2024-04-26T12:03:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 726.690578 secs 2024-04-26T12:03:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 726.690578 secs 2024-04-26T12:03:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:08 smithi033 ceph-mon[27845]: pgmap v1091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 729.401969 secs 2024-04-26T12:03:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:08 smithi096 ceph-mon[32753]: pgmap v1091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 729.401969 secs 2024-04-26T12:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:08 smithi194 ceph-mon[33017]: pgmap v1091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 729.401969 secs 2024-04-26T12:03:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:10 smithi033 ceph-mon[27845]: pgmap v1092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:11.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:10 smithi096 ceph-mon[32753]: pgmap v1092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:10 smithi194 ceph-mon[33017]: pgmap v1092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:12 smithi033 ceph-mon[27845]: pgmap v1093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 731.690663 secs 2024-04-26T12:03:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:12 smithi096 ceph-mon[32753]: pgmap v1093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 731.690663 secs 2024-04-26T12:03:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:12 smithi194 ceph-mon[33017]: pgmap v1093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 731.690663 secs 2024-04-26T12:03:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 734.402092 secs 2024-04-26T12:03:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 734.402092 secs 2024-04-26T12:03:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 734.402092 secs 2024-04-26T12:03:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:14 smithi033 ceph-mon[27845]: pgmap v1094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:14 smithi096 ceph-mon[32753]: pgmap v1094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:14 smithi194 ceph-mon[33017]: pgmap v1094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:16 smithi033 ceph-mon[27845]: pgmap v1095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:17.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:16 smithi096 ceph-mon[32753]: pgmap v1095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:16 smithi194 ceph-mon[33017]: pgmap v1095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:18.136+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:18.136+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:18.136+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:03:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 736.690745 secs 2024-04-26T12:03:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 736.690745 secs 2024-04-26T12:03:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 736.690745 secs 2024-04-26T12:03:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:18 smithi033 ceph-mon[27845]: pgmap v1096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 739.402214 secs 2024-04-26T12:03:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:18 smithi096 ceph-mon[32753]: pgmap v1096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 739.402214 secs 2024-04-26T12:03:19.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:18 smithi194 ceph-mon[33017]: pgmap v1096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:19.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 739.402214 secs 2024-04-26T12:03:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:20 smithi096 ceph-mon[32753]: pgmap v1097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:20 smithi194 ceph-mon[33017]: pgmap v1097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:20 smithi033 ceph-mon[27845]: pgmap v1097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:22 smithi096 ceph-mon[32753]: pgmap v1098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 741.690827 secs 2024-04-26T12:03:23.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:22 smithi033 ceph-mon[27845]: pgmap v1098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:23.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 741.690827 secs 2024-04-26T12:03:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:22 smithi194 ceph-mon[33017]: pgmap v1098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 741.690827 secs 2024-04-26T12:03:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 744.402337 secs 2024-04-26T12:03:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 744.402337 secs 2024-04-26T12:03:24.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 744.402337 secs 2024-04-26T12:03:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:24 smithi194 ceph-mon[33017]: pgmap v1099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:24 smithi033 ceph-mon[27845]: pgmap v1099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:24 smithi096 ceph-mon[32753]: pgmap v1099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:26 smithi194 ceph-mon[33017]: pgmap v1100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:26 smithi033 ceph-mon[27845]: pgmap v1100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:26 smithi096 ceph-mon[32753]: pgmap v1100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 746.690909 secs 2024-04-26T12:03:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 746.690909 secs 2024-04-26T12:03:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 746.690909 secs 2024-04-26T12:03:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:28 smithi194 ceph-mon[33017]: pgmap v1101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 749.402417 secs 2024-04-26T12:03:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:28 smithi033 ceph-mon[27845]: pgmap v1101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 749.402417 secs 2024-04-26T12:03:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:28 smithi096 ceph-mon[32753]: pgmap v1101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 749.402417 secs 2024-04-26T12:03:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:30 smithi194 ceph-mon[33017]: pgmap v1102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:30 smithi033 ceph-mon[27845]: pgmap v1102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:30 smithi096 ceph-mon[32753]: pgmap v1102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:33.145+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:33.145+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:33.145+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:03:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:32 smithi194 ceph-mon[33017]: pgmap v1103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 751.691009 secs 2024-04-26T12:03:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:32 smithi033 ceph-mon[27845]: pgmap v1103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 751.691009 secs 2024-04-26T12:03:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:32 smithi096 ceph-mon[32753]: pgmap v1103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 751.691009 secs 2024-04-26T12:03:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 754.402477 secs 2024-04-26T12:03:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 754.402477 secs 2024-04-26T12:03:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 754.402477 secs 2024-04-26T12:03:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:34 smithi194 ceph-mon[33017]: pgmap v1104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:34 smithi033 ceph-mon[27845]: pgmap v1104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:34 smithi096 ceph-mon[32753]: pgmap v1104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:36 smithi194 ceph-mon[33017]: pgmap v1105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:36 smithi033 ceph-mon[27845]: pgmap v1105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:36 smithi096 ceph-mon[32753]: pgmap v1105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 756.691090 secs 2024-04-26T12:03:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 756.691090 secs 2024-04-26T12:03:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 756.691090 secs 2024-04-26T12:03:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:39 smithi194 ceph-mon[33017]: pgmap v1106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 759.402561 secs 2024-04-26T12:03:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:39 smithi033 ceph-mon[27845]: pgmap v1106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 759.402561 secs 2024-04-26T12:03:39.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:39 smithi096 ceph-mon[32753]: pgmap v1106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:39.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 759.402561 secs 2024-04-26T12:03:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:41 smithi194 ceph-mon[33017]: pgmap v1107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:41 smithi033 ceph-mon[27845]: pgmap v1107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:41 smithi096 ceph-mon[32753]: pgmap v1107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:43 smithi194 ceph-mon[33017]: pgmap v1108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 761.691152 secs 2024-04-26T12:03:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:43 smithi033 ceph-mon[27845]: pgmap v1108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 761.691152 secs 2024-04-26T12:03:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:43 smithi096 ceph-mon[32753]: pgmap v1108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 761.691152 secs 2024-04-26T12:03:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 764.402683 secs 2024-04-26T12:03:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 764.402683 secs 2024-04-26T12:03:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 764.402683 secs 2024-04-26T12:03:45.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:45 smithi033 ceph-mon[27845]: pgmap v1109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:45 smithi194 ceph-mon[33017]: pgmap v1109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:45 smithi096 ceph-mon[32753]: pgmap v1109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:47 smithi194 ceph-mon[33017]: pgmap v1110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:47 smithi033 ceph-mon[27845]: pgmap v1110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:47 smithi096 ceph-mon[32753]: pgmap v1110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:03:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:48.160+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:48.160+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:03:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:03:48.160+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:03:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 766.691237 secs 2024-04-26T12:03:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 766.691237 secs 2024-04-26T12:03:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 766.691237 secs 2024-04-26T12:03:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:49 smithi194 ceph-mon[33017]: pgmap v1111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 769.402816 secs 2024-04-26T12:03:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:03:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:49 smithi033 ceph-mon[27845]: pgmap v1111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 769.402816 secs 2024-04-26T12:03:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:03:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:49 smithi096 ceph-mon[32753]: pgmap v1111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:03:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 769.402816 secs 2024-04-26T12:03:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:03:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:03:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:03:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:03:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:03:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:03:50.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:03:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:03:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:03:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:03:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:51 smithi194 ceph-mon[33017]: pgmap v1112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:03:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:51 smithi194 ceph-mon[33017]: pgmap v1113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:51 smithi033 ceph-mon[27845]: pgmap v1112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:03:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:51 smithi033 ceph-mon[27845]: pgmap v1113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:51 smithi096 ceph-mon[32753]: pgmap v1112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:03:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:51 smithi096 ceph-mon[32753]: pgmap v1113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:53 smithi194 ceph-mon[33017]: pgmap v1114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 771.691320 secs 2024-04-26T12:03:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:53 smithi033 ceph-mon[27845]: pgmap v1114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 771.691320 secs 2024-04-26T12:03:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:53 smithi096 ceph-mon[32753]: pgmap v1114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 771.691320 secs 2024-04-26T12:03:54.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 774.402947 secs 2024-04-26T12:03:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 774.402947 secs 2024-04-26T12:03:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 774.402947 secs 2024-04-26T12:03:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:55 smithi194 ceph-mon[33017]: pgmap v1115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:55 smithi033 ceph-mon[27845]: pgmap v1115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:55 smithi096 ceph-mon[32753]: pgmap v1115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:57 smithi194 ceph-mon[33017]: pgmap v1116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:57 smithi033 ceph-mon[27845]: pgmap v1116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:57 smithi096 ceph-mon[32753]: pgmap v1116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 776.691404 secs 2024-04-26T12:03:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 776.691404 secs 2024-04-26T12:03:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 776.691404 secs 2024-04-26T12:03:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:59 smithi194 ceph-mon[33017]: pgmap v1117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 779.403081 secs 2024-04-26T12:03:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:03:59 smithi194 ceph-mon[33017]: slow request 721.130870 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:03:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:59 smithi033 ceph-mon[27845]: pgmap v1117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 779.403081 secs 2024-04-26T12:03:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:03:59 smithi033 ceph-mon[27845]: slow request 721.130870 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:03:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:59 smithi096 ceph-mon[32753]: pgmap v1117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:03:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 779.403081 secs 2024-04-26T12:03:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:03:59 smithi096 ceph-mon[32753]: slow request 721.130870 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:04:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:01 smithi194 ceph-mon[33017]: pgmap v1118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:04:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:01 smithi033 ceph-mon[27845]: pgmap v1118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:04:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:01 smithi096 ceph-mon[32753]: pgmap v1118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:04:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:03.172+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:03.172+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:03.172+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:04:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:03 smithi194 ceph-mon[33017]: pgmap v1119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 781.691490 secs 2024-04-26T12:04:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:03 smithi033 ceph-mon[27845]: pgmap v1119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 781.691490 secs 2024-04-26T12:04:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:03 smithi096 ceph-mon[32753]: pgmap v1119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 781.691490 secs 2024-04-26T12:04:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 784.403204 secs 2024-04-26T12:04:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 784.403204 secs 2024-04-26T12:04:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 784.403204 secs 2024-04-26T12:04:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:05 smithi033 ceph-mon[27845]: pgmap v1120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:05 smithi096 ceph-mon[32753]: pgmap v1120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:05 smithi194 ceph-mon[33017]: pgmap v1120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:07 smithi033 ceph-mon[27845]: pgmap v1121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:07 smithi096 ceph-mon[32753]: pgmap v1121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:07 smithi194 ceph-mon[33017]: pgmap v1121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 786.691571 secs 2024-04-26T12:04:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 786.691571 secs 2024-04-26T12:04:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 786.691571 secs 2024-04-26T12:04:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:09 smithi033 ceph-mon[27845]: pgmap v1122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 789.403304 secs 2024-04-26T12:04:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:09 smithi096 ceph-mon[32753]: pgmap v1122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 789.403304 secs 2024-04-26T12:04:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:09 smithi194 ceph-mon[33017]: pgmap v1122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 789.403304 secs 2024-04-26T12:04:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:11 smithi033 ceph-mon[27845]: pgmap v1123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:11 smithi096 ceph-mon[32753]: pgmap v1123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:11 smithi194 ceph-mon[33017]: pgmap v1123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:13 smithi033 ceph-mon[27845]: pgmap v1124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 791.691652 secs 2024-04-26T12:04:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:13 smithi096 ceph-mon[32753]: pgmap v1124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 791.691652 secs 2024-04-26T12:04:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:13 smithi194 ceph-mon[33017]: pgmap v1124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 791.691652 secs 2024-04-26T12:04:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 794.403369 secs 2024-04-26T12:04:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 794.403369 secs 2024-04-26T12:04:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 794.403369 secs 2024-04-26T12:04:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:15 smithi033 ceph-mon[27845]: pgmap v1125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:15 smithi096 ceph-mon[32753]: pgmap v1125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:15 smithi194 ceph-mon[33017]: pgmap v1125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:17 smithi033 ceph-mon[27845]: pgmap v1126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:17 smithi096 ceph-mon[32753]: pgmap v1126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:17 smithi194 ceph-mon[33017]: pgmap v1126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:18.175+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:18.175+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:18.175+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:04:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 796.691733 secs 2024-04-26T12:04:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 796.691733 secs 2024-04-26T12:04:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 796.691733 secs 2024-04-26T12:04:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:19 smithi033 ceph-mon[27845]: pgmap v1127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 799.403505 secs 2024-04-26T12:04:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:19 smithi096 ceph-mon[32753]: pgmap v1127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 799.403505 secs 2024-04-26T12:04:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:19 smithi194 ceph-mon[33017]: pgmap v1127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 799.403505 secs 2024-04-26T12:04:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:21 smithi096 ceph-mon[32753]: pgmap v1128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:21 smithi194 ceph-mon[33017]: pgmap v1128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:21 smithi033 ceph-mon[27845]: pgmap v1128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:23 smithi194 ceph-mon[33017]: pgmap v1129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 801.691813 secs 2024-04-26T12:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:23 smithi096 ceph-mon[32753]: pgmap v1129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 801.691813 secs 2024-04-26T12:04:23.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:23 smithi033 ceph-mon[27845]: pgmap v1129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:23.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 801.691813 secs 2024-04-26T12:04:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 804.403565 secs 2024-04-26T12:04:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 804.403565 secs 2024-04-26T12:04:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 804.403565 secs 2024-04-26T12:04:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:25 smithi194 ceph-mon[33017]: pgmap v1130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:25 smithi033 ceph-mon[27845]: pgmap v1130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:25 smithi096 ceph-mon[32753]: pgmap v1130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:27 smithi194 ceph-mon[33017]: pgmap v1131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:27 smithi033 ceph-mon[27845]: pgmap v1131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:27 smithi096 ceph-mon[32753]: pgmap v1131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 806.691895 secs 2024-04-26T12:04:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 806.691895 secs 2024-04-26T12:04:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 806.691895 secs 2024-04-26T12:04:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:29 smithi194 ceph-mon[33017]: pgmap v1132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 809.403691 secs 2024-04-26T12:04:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:29 smithi033 ceph-mon[27845]: pgmap v1132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 809.403691 secs 2024-04-26T12:04:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:29 smithi096 ceph-mon[32753]: pgmap v1132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 809.403691 secs 2024-04-26T12:04:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:30 smithi194 ceph-mon[33017]: pgmap v1133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:30 smithi033 ceph-mon[27845]: pgmap v1133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:30 smithi096 ceph-mon[32753]: pgmap v1133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:32 smithi194 ceph-mon[33017]: pgmap v1134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:32 smithi033 ceph-mon[27845]: pgmap v1134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:32 smithi096 ceph-mon[32753]: pgmap v1134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:33.186+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:33.186+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:33.186+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:04:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 811.691941 secs 2024-04-26T12:04:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 814.403797 secs 2024-04-26T12:04:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 811.691941 secs 2024-04-26T12:04:33.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 814.403797 secs 2024-04-26T12:04:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 811.691941 secs 2024-04-26T12:04:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 814.403797 secs 2024-04-26T12:04:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:34 smithi194 ceph-mon[33017]: pgmap v1135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:34 smithi033 ceph-mon[27845]: pgmap v1135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:34 smithi096 ceph-mon[32753]: pgmap v1135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:36 smithi194 ceph-mon[33017]: pgmap v1136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:36 smithi033 ceph-mon[27845]: pgmap v1136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:36 smithi096 ceph-mon[32753]: pgmap v1136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 816.691989 secs 2024-04-26T12:04:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 816.691989 secs 2024-04-26T12:04:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 816.691989 secs 2024-04-26T12:04:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:38 smithi033 ceph-mon[27845]: pgmap v1137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 819.403923 secs 2024-04-26T12:04:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:38 smithi096 ceph-mon[32753]: pgmap v1137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 819.403923 secs 2024-04-26T12:04:39.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:38 smithi194 ceph-mon[33017]: pgmap v1137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:39.108 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 819.403923 secs 2024-04-26T12:04:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:40 smithi033 ceph-mon[27845]: pgmap v1138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:40 smithi096 ceph-mon[32753]: pgmap v1138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:40 smithi194 ceph-mon[33017]: pgmap v1138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:42 smithi033 ceph-mon[27845]: pgmap v1139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 821.692072 secs 2024-04-26T12:04:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:42 smithi096 ceph-mon[32753]: pgmap v1139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 821.692072 secs 2024-04-26T12:04:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:42 smithi194 ceph-mon[33017]: pgmap v1139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 821.692072 secs 2024-04-26T12:04:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 824.404048 secs 2024-04-26T12:04:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 824.404048 secs 2024-04-26T12:04:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 824.404048 secs 2024-04-26T12:04:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:44 smithi033 ceph-mon[27845]: pgmap v1140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:44 smithi096 ceph-mon[32753]: pgmap v1140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:44 smithi194 ceph-mon[33017]: pgmap v1140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:46 smithi033 ceph-mon[27845]: pgmap v1141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:46 smithi096 ceph-mon[32753]: pgmap v1141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:46 smithi194 ceph-mon[33017]: pgmap v1141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 826.692127 secs 2024-04-26T12:04:48.049 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 826.692127 secs 2024-04-26T12:04:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 826.692127 secs 2024-04-26T12:04:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:48.201+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:48.201+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:04:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:04:48.201+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:04:49.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:48 smithi194 ceph-mon[33017]: pgmap v1142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:49.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 829.404173 secs 2024-04-26T12:04:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:48 smithi033 ceph-mon[27845]: pgmap v1142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 829.404173 secs 2024-04-26T12:04:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:48 smithi096 ceph-mon[32753]: pgmap v1142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 829.404173 secs 2024-04-26T12:04:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:50 smithi194 ceph-mon[33017]: pgmap v1143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:04:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:50 smithi033 ceph-mon[27845]: pgmap v1143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:04:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:50 smithi096 ceph-mon[32753]: pgmap v1143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:04:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:04:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:04:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:04:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:04:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:04:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:04:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:04:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:04:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:04:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:04:53.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:52 smithi033 ceph-mon[27845]: pgmap v1144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:53.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:52 smithi033 ceph-mon[27845]: pgmap v1145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:53.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 831.692215 secs 2024-04-26T12:04:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:52 smithi194 ceph-mon[33017]: pgmap v1144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:52 smithi194 ceph-mon[33017]: pgmap v1145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 831.692215 secs 2024-04-26T12:04:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:52 smithi096 ceph-mon[32753]: pgmap v1144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:04:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:52 smithi096 ceph-mon[32753]: pgmap v1145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 831.692215 secs 2024-04-26T12:04:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 834.404296 secs 2024-04-26T12:04:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 834.404296 secs 2024-04-26T12:04:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 834.404296 secs 2024-04-26T12:04:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:54 smithi194 ceph-mon[33017]: pgmap v1146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:54 smithi033 ceph-mon[27845]: pgmap v1146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:54 smithi096 ceph-mon[32753]: pgmap v1146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:56 smithi194 ceph-mon[33017]: pgmap v1147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:56 smithi033 ceph-mon[27845]: pgmap v1147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:56 smithi096 ceph-mon[32753]: pgmap v1147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 836.692274 secs 2024-04-26T12:04:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 836.692274 secs 2024-04-26T12:04:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 836.692274 secs 2024-04-26T12:04:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:58 smithi194 ceph-mon[33017]: pgmap v1148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:59.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:04:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 839.404418 secs 2024-04-26T12:04:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:58 smithi033 ceph-mon[27845]: pgmap v1148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:04:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 839.404418 secs 2024-04-26T12:04:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:58 smithi096 ceph-mon[32753]: pgmap v1148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:04:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:04:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 839.404418 secs 2024-04-26T12:05:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:00 smithi194 ceph-mon[33017]: pgmap v1149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:05:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:00 smithi033 ceph-mon[27845]: pgmap v1149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:05:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:00 smithi096 ceph-mon[32753]: pgmap v1149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:05:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:02 smithi194 ceph-mon[33017]: pgmap v1150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:05:03.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 841.692333 secs 2024-04-26T12:05:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:02 smithi033 ceph-mon[27845]: pgmap v1150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:05:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 841.692333 secs 2024-04-26T12:05:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:02 smithi096 ceph-mon[32753]: pgmap v1150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:05:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 841.692333 secs 2024-04-26T12:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:03.215+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:03.215+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:03.215+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:05:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 844.404541 secs 2024-04-26T12:05:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 844.404541 secs 2024-04-26T12:05:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 844.404541 secs 2024-04-26T12:05:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:04 smithi194 ceph-mon[33017]: pgmap v1151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:04 smithi033 ceph-mon[27845]: pgmap v1151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:05.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:04 smithi096 ceph-mon[32753]: pgmap v1151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:06 smithi194 ceph-mon[33017]: pgmap v1152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:06 smithi033 ceph-mon[27845]: pgmap v1152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:07.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:06 smithi096 ceph-mon[32753]: pgmap v1152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 846.692387 secs 2024-04-26T12:05:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 846.692387 secs 2024-04-26T12:05:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 846.692387 secs 2024-04-26T12:05:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:08 smithi194 ceph-mon[33017]: pgmap v1153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 849.404689 secs 2024-04-26T12:05:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:08 smithi033 ceph-mon[27845]: pgmap v1153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 849.404689 secs 2024-04-26T12:05:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:08 smithi096 ceph-mon[32753]: pgmap v1153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 849.404689 secs 2024-04-26T12:05:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:10 smithi194 ceph-mon[33017]: pgmap v1154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:05:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:10 smithi033 ceph-mon[27845]: pgmap v1154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:05:11.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:10 smithi096 ceph-mon[32753]: pgmap v1154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:05:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:12 smithi194 ceph-mon[33017]: pgmap v1155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 851.692446 secs 2024-04-26T12:05:13.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:12 smithi033 ceph-mon[27845]: pgmap v1155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:13.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 851.692446 secs 2024-04-26T12:05:13.626 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:12 smithi096 ceph-mon[32753]: pgmap v1155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:13.626 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 851.692446 secs 2024-04-26T12:05:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 854.404815 secs 2024-04-26T12:05:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 854.404815 secs 2024-04-26T12:05:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 854.404815 secs 2024-04-26T12:05:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:14 smithi194 ceph-mon[33017]: pgmap v1156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:14 smithi033 ceph-mon[27845]: pgmap v1156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:14 smithi096 ceph-mon[32753]: pgmap v1156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:16 smithi194 ceph-mon[33017]: pgmap v1157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:16 smithi033 ceph-mon[27845]: pgmap v1157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:16 smithi096 ceph-mon[32753]: pgmap v1157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:18.230+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:18.230+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:05:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:18.230+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:18.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 856.692501 secs 2024-04-26T12:05:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 856.692501 secs 2024-04-26T12:05:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 856.692501 secs 2024-04-26T12:05:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:19 smithi194 ceph-mon[33017]: pgmap v1158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 859.404944 secs 2024-04-26T12:05:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:19 smithi033 ceph-mon[27845]: pgmap v1158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 859.404944 secs 2024-04-26T12:05:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:19 smithi096 ceph-mon[32753]: pgmap v1158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 859.404944 secs 2024-04-26T12:05:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:21 smithi194 ceph-mon[33017]: pgmap v1159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:21 smithi033 ceph-mon[27845]: pgmap v1159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:21 smithi096 ceph-mon[32753]: pgmap v1159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:21.941+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:05:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:21.942+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:05:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:21.942+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:05:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:21.942+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:05:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:23 smithi194 ceph-mon[33017]: pgmap v1160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 861.692584 secs 2024-04-26T12:05:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:23 smithi033 ceph-mon[27845]: pgmap v1160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 861.692584 secs 2024-04-26T12:05:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:23 smithi096 ceph-mon[32753]: pgmap v1160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 861.692584 secs 2024-04-26T12:05:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 864.405007 secs 2024-04-26T12:05:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 864.405007 secs 2024-04-26T12:05:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 864.405007 secs 2024-04-26T12:05:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:25 smithi194 ceph-mon[33017]: pgmap v1161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:25 smithi033 ceph-mon[27845]: pgmap v1161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:25 smithi096 ceph-mon[32753]: pgmap v1161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:27 smithi194 ceph-mon[33017]: pgmap v1162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:27 smithi033 ceph-mon[27845]: pgmap v1162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:27 smithi096 ceph-mon[32753]: pgmap v1162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 866.692668 secs 2024-04-26T12:05:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 866.692668 secs 2024-04-26T12:05:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 866.692668 secs 2024-04-26T12:05:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:29 smithi194 ceph-mon[33017]: pgmap v1163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 869.405153 secs 2024-04-26T12:05:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:29 smithi033 ceph-mon[27845]: pgmap v1163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 869.405153 secs 2024-04-26T12:05:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:29 smithi096 ceph-mon[32753]: pgmap v1163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 869.405153 secs 2024-04-26T12:05:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:31 smithi194 ceph-mon[33017]: pgmap v1164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:31 smithi033 ceph-mon[27845]: pgmap v1164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:31 smithi096 ceph-mon[32753]: pgmap v1164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:33.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:33.242+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:33.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:33.242+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:33.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:33.242+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:05:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:33 smithi194 ceph-mon[33017]: pgmap v1165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 871.692752 secs 2024-04-26T12:05:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:33 smithi033 ceph-mon[27845]: pgmap v1165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 871.692752 secs 2024-04-26T12:05:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:33 smithi096 ceph-mon[32753]: pgmap v1165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 871.692752 secs 2024-04-26T12:05:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 874.405266 secs 2024-04-26T12:05:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 874.405266 secs 2024-04-26T12:05:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 874.405266 secs 2024-04-26T12:05:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:35 smithi194 ceph-mon[33017]: pgmap v1166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:35 smithi033 ceph-mon[27845]: pgmap v1166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:35 smithi096 ceph-mon[32753]: pgmap v1166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:37 smithi194 ceph-mon[33017]: pgmap v1167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:37 smithi033 ceph-mon[27845]: pgmap v1167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:37 smithi096 ceph-mon[32753]: pgmap v1167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 876.692836 secs 2024-04-26T12:05:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 876.692836 secs 2024-04-26T12:05:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 876.692836 secs 2024-04-26T12:05:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:39 smithi194 ceph-mon[33017]: pgmap v1168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 879.405363 secs 2024-04-26T12:05:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:39 smithi033 ceph-mon[27845]: pgmap v1168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 879.405363 secs 2024-04-26T12:05:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:39 smithi096 ceph-mon[32753]: pgmap v1168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 879.405363 secs 2024-04-26T12:05:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:41 smithi194 ceph-mon[33017]: pgmap v1169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:41 smithi033 ceph-mon[27845]: pgmap v1169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:41 smithi096 ceph-mon[32753]: pgmap v1169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:43 smithi033 ceph-mon[27845]: pgmap v1170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 881.692919 secs 2024-04-26T12:05:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:43 smithi096 ceph-mon[32753]: pgmap v1170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 881.692919 secs 2024-04-26T12:05:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:43 smithi194 ceph-mon[33017]: pgmap v1170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 881.692919 secs 2024-04-26T12:05:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 884.405461 secs 2024-04-26T12:05:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 884.405461 secs 2024-04-26T12:05:44.609 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 884.405461 secs 2024-04-26T12:05:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:45 smithi033 ceph-mon[27845]: pgmap v1171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:45 smithi096 ceph-mon[32753]: pgmap v1171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:45 smithi194 ceph-mon[33017]: pgmap v1171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:47 smithi033 ceph-mon[27845]: pgmap v1172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:47 smithi096 ceph-mon[32753]: pgmap v1172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:47 smithi194 ceph-mon[33017]: pgmap v1172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:05:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:48.257+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:48.258+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:05:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:05:48.258+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:05:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 886.693026 secs 2024-04-26T12:05:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 886.693026 secs 2024-04-26T12:05:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 886.693026 secs 2024-04-26T12:05:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:49 smithi033 ceph-mon[27845]: pgmap v1173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 889.405553 secs 2024-04-26T12:05:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:49 smithi096 ceph-mon[32753]: pgmap v1173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 889.405553 secs 2024-04-26T12:05:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:49 smithi194 ceph-mon[33017]: pgmap v1173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 889.405553 secs 2024-04-26T12:05:51.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:51 smithi033 ceph-mon[27845]: pgmap v1174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:51 smithi096 ceph-mon[32753]: pgmap v1174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:51 smithi194 ceph-mon[33017]: pgmap v1174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:05:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:05:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:05:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:53 smithi033 ceph-mon[27845]: pgmap v1175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 891.693088 secs 2024-04-26T12:05:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:05:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:05:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:05:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:53 smithi096 ceph-mon[32753]: pgmap v1175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 891.693088 secs 2024-04-26T12:05:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:05:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:05:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:05:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:53 smithi194 ceph-mon[33017]: pgmap v1175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 891.693088 secs 2024-04-26T12:05:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:05:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:05:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:05:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:54 smithi033 ceph-mon[27845]: pgmap v1176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:54 smithi033 ceph-mon[27845]: pgmap v1177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 894.405679 secs 2024-04-26T12:05:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:54 smithi096 ceph-mon[32753]: pgmap v1176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:54 smithi096 ceph-mon[32753]: pgmap v1177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 894.405679 secs 2024-04-26T12:05:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:54 smithi194 ceph-mon[33017]: pgmap v1176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:05:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:54 smithi194 ceph-mon[33017]: pgmap v1177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 894.405679 secs 2024-04-26T12:05:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:56 smithi096 ceph-mon[32753]: pgmap v1178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:56 smithi194 ceph-mon[33017]: pgmap v1178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:56 smithi033 ceph-mon[27845]: pgmap v1178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:58 smithi096 ceph-mon[32753]: pgmap v1179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 896.693172 secs 2024-04-26T12:05:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:58 smithi194 ceph-mon[33017]: pgmap v1179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 896.693172 secs 2024-04-26T12:05:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:58 smithi033 ceph-mon[27845]: pgmap v1179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:05:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 896.693172 secs 2024-04-26T12:05:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 899.405814 secs 2024-04-26T12:05:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 899.405814 secs 2024-04-26T12:05:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 899.405814 secs 2024-04-26T12:06:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:00 smithi194 ceph-mon[33017]: pgmap v1180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:06:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:00 smithi033 ceph-mon[27845]: pgmap v1180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:06:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:00 smithi096 ceph-mon[32753]: pgmap v1180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:06:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:02 smithi194 ceph-mon[33017]: pgmap v1181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:02.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:02 smithi033 ceph-mon[27845]: pgmap v1181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:02 smithi096 ceph-mon[32753]: pgmap v1181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:03.272+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:03.272+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:03.272+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:06:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 901.693277 secs 2024-04-26T12:06:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 901.693277 secs 2024-04-26T12:06:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 901.693277 secs 2024-04-26T12:06:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:04 smithi194 ceph-mon[33017]: pgmap v1182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:06:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 904.405932 secs 2024-04-26T12:06:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:04 smithi033 ceph-mon[27845]: pgmap v1182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:06:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 904.405932 secs 2024-04-26T12:06:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:04 smithi096 ceph-mon[32753]: pgmap v1182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:06:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 904.405932 secs 2024-04-26T12:06:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:06 smithi194 ceph-mon[33017]: pgmap v1183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:06 smithi033 ceph-mon[27845]: pgmap v1183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:06 smithi096 ceph-mon[32753]: pgmap v1183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:06.949+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:06:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:06.949+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:06:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:06.950+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:06:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:06.950+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:06:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:08 smithi194 ceph-mon[33017]: pgmap v1184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 906.693338 secs 2024-04-26T12:06:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:08 smithi033 ceph-mon[27845]: pgmap v1184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 906.693338 secs 2024-04-26T12:06:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:08 smithi096 ceph-mon[32753]: pgmap v1184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 906.693338 secs 2024-04-26T12:06:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 909.406067 secs 2024-04-26T12:06:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 909.406067 secs 2024-04-26T12:06:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 909.406067 secs 2024-04-26T12:06:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:10 smithi194 ceph-mon[33017]: pgmap v1185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:10 smithi033 ceph-mon[27845]: pgmap v1185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:10 smithi096 ceph-mon[32753]: pgmap v1185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:11 smithi194 ceph-mon[33017]: pgmap v1186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:11 smithi033 ceph-mon[27845]: pgmap v1186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:11 smithi096 ceph-mon[32753]: pgmap v1186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 911.693395 secs 2024-04-26T12:06:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 911.693395 secs 2024-04-26T12:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 911.693395 secs 2024-04-26T12:06:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:13 smithi033 ceph-mon[27845]: pgmap v1187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 914.406194 secs 2024-04-26T12:06:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:13 smithi096 ceph-mon[32753]: pgmap v1187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 914.406194 secs 2024-04-26T12:06:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:13 smithi194 ceph-mon[33017]: pgmap v1187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 914.406194 secs 2024-04-26T12:06:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:16 smithi194 ceph-mon[33017]: pgmap v1188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:16 smithi033 ceph-mon[27845]: pgmap v1188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:16 smithi096 ceph-mon[32753]: pgmap v1188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:18.287+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:18.287+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:18.287+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:06:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:18 smithi194 ceph-mon[33017]: pgmap v1189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 916.693453 secs 2024-04-26T12:06:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:18 smithi033 ceph-mon[27845]: pgmap v1189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 916.693453 secs 2024-04-26T12:06:18.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:18 smithi096 ceph-mon[32753]: pgmap v1189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:18.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 916.693453 secs 2024-04-26T12:06:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 919.406266 secs 2024-04-26T12:06:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 919.406266 secs 2024-04-26T12:06:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 919.406266 secs 2024-04-26T12:06:20.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:20 smithi033 ceph-mon[27845]: pgmap v1190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:20 smithi194 ceph-mon[33017]: pgmap v1190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:20 smithi096 ceph-mon[32753]: pgmap v1190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:22 smithi194 ceph-mon[33017]: pgmap v1191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:22 smithi033 ceph-mon[27845]: pgmap v1191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:22 smithi096 ceph-mon[32753]: pgmap v1191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:23.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 921.693510 secs 2024-04-26T12:06:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 921.693510 secs 2024-04-26T12:06:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 921.693510 secs 2024-04-26T12:06:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:24 smithi096 ceph-mon[32753]: pgmap v1192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 924.406383 secs 2024-04-26T12:06:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:24 smithi194 ceph-mon[33017]: pgmap v1192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 924.406383 secs 2024-04-26T12:06:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:24 smithi033 ceph-mon[27845]: pgmap v1192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 924.406383 secs 2024-04-26T12:06:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:26 smithi194 ceph-mon[33017]: pgmap v1193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:26 smithi033 ceph-mon[27845]: pgmap v1193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:26 smithi096 ceph-mon[32753]: pgmap v1193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:28 smithi194 ceph-mon[33017]: pgmap v1194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:28.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 926.693568 secs 2024-04-26T12:06:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:28 smithi033 ceph-mon[27845]: pgmap v1194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 926.693568 secs 2024-04-26T12:06:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:28 smithi096 ceph-mon[32753]: pgmap v1194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 926.693568 secs 2024-04-26T12:06:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 929.406480 secs 2024-04-26T12:06:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 929.406480 secs 2024-04-26T12:06:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 929.406480 secs 2024-04-26T12:06:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:30 smithi194 ceph-mon[33017]: pgmap v1195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:30 smithi033 ceph-mon[27845]: pgmap v1195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:30 smithi096 ceph-mon[32753]: pgmap v1195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:32 smithi194 ceph-mon[33017]: pgmap v1196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:32 smithi033 ceph-mon[27845]: pgmap v1196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:32 smithi096 ceph-mon[32753]: pgmap v1196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:33.303+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:33.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:33.303+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:33.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:33.303+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:06:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 931.693652 secs 2024-04-26T12:06:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 931.693652 secs 2024-04-26T12:06:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 931.693652 secs 2024-04-26T12:06:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:34 smithi194 ceph-mon[33017]: pgmap v1197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 934.406543 secs 2024-04-26T12:06:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:34 smithi033 ceph-mon[27845]: pgmap v1197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 934.406543 secs 2024-04-26T12:06:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:34 smithi096 ceph-mon[32753]: pgmap v1197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 934.406543 secs 2024-04-26T12:06:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:36 smithi033 ceph-mon[27845]: pgmap v1198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:36 smithi096 ceph-mon[32753]: pgmap v1198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:36 smithi194 ceph-mon[33017]: pgmap v1198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:38 smithi033 ceph-mon[27845]: pgmap v1199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 936.693735 secs 2024-04-26T12:06:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:38 smithi096 ceph-mon[32753]: pgmap v1199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 936.693735 secs 2024-04-26T12:06:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:38 smithi194 ceph-mon[33017]: pgmap v1199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 936.693735 secs 2024-04-26T12:06:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 939.406663 secs 2024-04-26T12:06:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 939.406663 secs 2024-04-26T12:06:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 939.406663 secs 2024-04-26T12:06:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:40 smithi033 ceph-mon[27845]: pgmap v1200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:40 smithi096 ceph-mon[32753]: pgmap v1200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:40 smithi194 ceph-mon[33017]: pgmap v1200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:42 smithi033 ceph-mon[27845]: pgmap v1201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:42 smithi096 ceph-mon[32753]: pgmap v1201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:42 smithi194 ceph-mon[33017]: pgmap v1201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 941.693821 secs 2024-04-26T12:06:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 941.693821 secs 2024-04-26T12:06:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 941.693821 secs 2024-04-26T12:06:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:44 smithi033 ceph-mon[27845]: pgmap v1202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 944.406798 secs 2024-04-26T12:06:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:44 smithi096 ceph-mon[32753]: pgmap v1202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 944.406798 secs 2024-04-26T12:06:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:44 smithi194 ceph-mon[33017]: pgmap v1202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 944.406798 secs 2024-04-26T12:06:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:46 smithi033 ceph-mon[27845]: pgmap v1203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:47.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:46 smithi194 ceph-mon[33017]: pgmap v1203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:47.167 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:46 smithi096 ceph-mon[32753]: pgmap v1203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:48.309+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:48.309+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:06:48.309+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:06:48.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:48 smithi033 ceph-mon[27845]: pgmap v1204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:48.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 946.693880 secs 2024-04-26T12:06:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:48 smithi096 ceph-mon[32753]: pgmap v1204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 946.693880 secs 2024-04-26T12:06:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:48 smithi194 ceph-mon[33017]: pgmap v1204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 946.693880 secs 2024-04-26T12:06:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 949.406938 secs 2024-04-26T12:06:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 949.406938 secs 2024-04-26T12:06:49.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 949.406938 secs 2024-04-26T12:06:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:50 smithi033 ceph-mon[27845]: pgmap v1205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:50 smithi096 ceph-mon[32753]: pgmap v1205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:50 smithi194 ceph-mon[33017]: pgmap v1205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:52 smithi033 ceph-mon[27845]: pgmap v1206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:52 smithi096 ceph-mon[32753]: pgmap v1206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:52 smithi194 ceph-mon[33017]: pgmap v1206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:06:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 951.693988 secs 2024-04-26T12:06:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 951.693988 secs 2024-04-26T12:06:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 951.693988 secs 2024-04-26T12:06:54.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:54 smithi096 ceph-mon[32753]: pgmap v1207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:54.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 954.407072 secs 2024-04-26T12:06:54.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:06:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:54 smithi194 ceph-mon[33017]: pgmap v1207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 954.407072 secs 2024-04-26T12:06:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:06:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:54 smithi033 ceph-mon[27845]: pgmap v1207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:06:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 954.407072 secs 2024-04-26T12:06:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:06:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:06:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:06:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:06:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:06:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:06:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:06:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:06:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:06:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:06:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:56 smithi194 ceph-mon[33017]: pgmap v1208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:06:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:56 smithi194 ceph-mon[33017]: pgmap v1209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:56 smithi033 ceph-mon[27845]: pgmap v1208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:06:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:56 smithi033 ceph-mon[27845]: pgmap v1209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:56 smithi096 ceph-mon[32753]: pgmap v1208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:06:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:56 smithi096 ceph-mon[32753]: pgmap v1209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:58 smithi194 ceph-mon[33017]: pgmap v1210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 956.694029 secs 2024-04-26T12:06:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:58 smithi033 ceph-mon[27845]: pgmap v1210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 956.694029 secs 2024-04-26T12:06:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:58 smithi096 ceph-mon[32753]: pgmap v1210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:06:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 956.694029 secs 2024-04-26T12:06:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 959.407193 secs 2024-04-26T12:06:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 959.407193 secs 2024-04-26T12:06:59.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 959.407193 secs 2024-04-26T12:07:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:00 smithi194 ceph-mon[33017]: pgmap v1211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:00 smithi033 ceph-mon[27845]: pgmap v1211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:00 smithi096 ceph-mon[32753]: pgmap v1211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:02 smithi194 ceph-mon[33017]: pgmap v1212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:02 smithi033 ceph-mon[27845]: pgmap v1212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:02 smithi096 ceph-mon[32753]: pgmap v1212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:03.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:03.320+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:03.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:03.320+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:03.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:03.320+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:07:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 961.694091 secs 2024-04-26T12:07:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 961.694091 secs 2024-04-26T12:07:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 961.694091 secs 2024-04-26T12:07:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:04 smithi194 ceph-mon[33017]: pgmap v1213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 964.407304 secs 2024-04-26T12:07:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:04 smithi033 ceph-mon[27845]: pgmap v1213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 964.407304 secs 2024-04-26T12:07:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:04 smithi096 ceph-mon[32753]: pgmap v1213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 964.407304 secs 2024-04-26T12:07:05.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:05 smithi194 ceph-mon[33017]: pgmap v1214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:07:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:05 smithi033 ceph-mon[27845]: pgmap v1214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:07:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:05 smithi096 ceph-mon[32753]: pgmap v1214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:07:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:07 smithi194 ceph-mon[33017]: pgmap v1215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 966.694175 secs 2024-04-26T12:07:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:07 smithi033 ceph-mon[27845]: pgmap v1215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 966.694175 secs 2024-04-26T12:07:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:07 smithi096 ceph-mon[32753]: pgmap v1215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 966.694175 secs 2024-04-26T12:07:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 969.407383 secs 2024-04-26T12:07:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 969.407383 secs 2024-04-26T12:07:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 969.407383 secs 2024-04-26T12:07:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:09 smithi194 ceph-mon[33017]: pgmap v1216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:09 smithi033 ceph-mon[27845]: pgmap v1216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:10.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:09 smithi096 ceph-mon[32753]: pgmap v1216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:11 smithi194 ceph-mon[33017]: pgmap v1217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:11 smithi033 ceph-mon[27845]: pgmap v1217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:12.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:11 smithi096 ceph-mon[32753]: pgmap v1217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 971.694235 secs 2024-04-26T12:07:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 971.694235 secs 2024-04-26T12:07:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 971.694235 secs 2024-04-26T12:07:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:13 smithi194 ceph-mon[33017]: pgmap v1218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 974.407505 secs 2024-04-26T12:07:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:13 smithi033 ceph-mon[27845]: pgmap v1218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 974.407505 secs 2024-04-26T12:07:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:13 smithi096 ceph-mon[32753]: pgmap v1218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 974.407505 secs 2024-04-26T12:07:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:15 smithi194 ceph-mon[33017]: pgmap v1219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:15 smithi033 ceph-mon[27845]: pgmap v1219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:15 smithi096 ceph-mon[32753]: pgmap v1219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:17 smithi194 ceph-mon[33017]: pgmap v1220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 976.694292 secs 2024-04-26T12:07:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:17 smithi033 ceph-mon[27845]: pgmap v1220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 976.694292 secs 2024-04-26T12:07:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:17 smithi096 ceph-mon[32753]: pgmap v1220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 976.694292 secs 2024-04-26T12:07:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:18.320+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:18.320+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:18.320+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:07:19.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 979.407615 secs 2024-04-26T12:07:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 979.407615 secs 2024-04-26T12:07:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 979.407615 secs 2024-04-26T12:07:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:19 smithi194 ceph-mon[33017]: pgmap v1221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:19 smithi033 ceph-mon[27845]: pgmap v1221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:19 smithi096 ceph-mon[32753]: pgmap v1221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:21 smithi194 ceph-mon[33017]: pgmap v1222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:22.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:21 smithi033 ceph-mon[27845]: pgmap v1222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:21 smithi096 ceph-mon[32753]: pgmap v1222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 981.694350 secs 2024-04-26T12:07:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 981.694350 secs 2024-04-26T12:07:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 981.694350 secs 2024-04-26T12:07:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:23 smithi096 ceph-mon[32753]: pgmap v1223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 984.407733 secs 2024-04-26T12:07:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:23 smithi194 ceph-mon[33017]: pgmap v1223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 984.407733 secs 2024-04-26T12:07:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:23 smithi033 ceph-mon[27845]: pgmap v1223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 984.407733 secs 2024-04-26T12:07:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:25 smithi194 ceph-mon[33017]: pgmap v1224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:25 smithi033 ceph-mon[27845]: pgmap v1224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:25 smithi096 ceph-mon[32753]: pgmap v1224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:27 smithi194 ceph-mon[33017]: pgmap v1225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 986.694408 secs 2024-04-26T12:07:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:27 smithi033 ceph-mon[27845]: pgmap v1225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 986.694408 secs 2024-04-26T12:07:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:27 smithi096 ceph-mon[32753]: pgmap v1225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 986.694408 secs 2024-04-26T12:07:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 989.407866 secs 2024-04-26T12:07:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 989.407866 secs 2024-04-26T12:07:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 989.407866 secs 2024-04-26T12:07:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:29 smithi194 ceph-mon[33017]: pgmap v1226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:30.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:29 smithi033 ceph-mon[27845]: pgmap v1226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:29 smithi096 ceph-mon[32753]: pgmap v1226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:31 smithi194 ceph-mon[33017]: pgmap v1227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:31 smithi033 ceph-mon[27845]: pgmap v1227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:31 smithi096 ceph-mon[32753]: pgmap v1227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 991.694467 secs 2024-04-26T12:07:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 991.694467 secs 2024-04-26T12:07:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 991.694467 secs 2024-04-26T12:07:33.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:33.336+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:33.336+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:33.336+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:07:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:33 smithi194 ceph-mon[33017]: pgmap v1228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 994.407985 secs 2024-04-26T12:07:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:33 smithi033 ceph-mon[27845]: pgmap v1228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 994.407985 secs 2024-04-26T12:07:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:33 smithi096 ceph-mon[32753]: pgmap v1228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 994.407985 secs 2024-04-26T12:07:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:35 smithi194 ceph-mon[33017]: pgmap v1229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:35 smithi033 ceph-mon[27845]: pgmap v1229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:35 smithi096 ceph-mon[32753]: pgmap v1229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:37 smithi194 ceph-mon[33017]: pgmap v1230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 996.694542 secs 2024-04-26T12:07:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:37 smithi033 ceph-mon[27845]: pgmap v1230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 996.694542 secs 2024-04-26T12:07:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:37 smithi096 ceph-mon[32753]: pgmap v1230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 996.694542 secs 2024-04-26T12:07:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 999.408137 secs 2024-04-26T12:07:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 999.408137 secs 2024-04-26T12:07:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 999.408137 secs 2024-04-26T12:07:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:39 smithi194 ceph-mon[33017]: pgmap v1231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:39 smithi033 ceph-mon[27845]: pgmap v1231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:39 smithi096 ceph-mon[32753]: pgmap v1231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:41 smithi194 ceph-mon[33017]: pgmap v1232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:41 smithi033 ceph-mon[27845]: pgmap v1232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:41 smithi096 ceph-mon[32753]: pgmap v1232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1001.694600 secs 2024-04-26T12:07:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1001.694600 secs 2024-04-26T12:07:43.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1001.694600 secs 2024-04-26T12:07:44.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:43 smithi033 ceph-mon[27845]: pgmap v1233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:44.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1004.408203 secs 2024-04-26T12:07:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:43 smithi096 ceph-mon[32753]: pgmap v1233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1004.408203 secs 2024-04-26T12:07:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:43 smithi194 ceph-mon[33017]: pgmap v1233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1004.408203 secs 2024-04-26T12:07:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:45 smithi033 ceph-mon[27845]: pgmap v1234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:45 smithi096 ceph-mon[32753]: pgmap v1234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:45 smithi194 ceph-mon[33017]: pgmap v1234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:47 smithi033 ceph-mon[27845]: pgmap v1235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1006.694657 secs 2024-04-26T12:07:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:47 smithi096 ceph-mon[32753]: pgmap v1235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1006.694657 secs 2024-04-26T12:07:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:48.352+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:48.352+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:07:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:07:48.352+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:07:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:47 smithi194 ceph-mon[33017]: pgmap v1235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1006.694657 secs 2024-04-26T12:07:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1009.408287 secs 2024-04-26T12:07:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1009.408287 secs 2024-04-26T12:07:49.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1009.408287 secs 2024-04-26T12:07:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:49 smithi033 ceph-mon[27845]: pgmap v1236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:49 smithi096 ceph-mon[32753]: pgmap v1236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:49 smithi194 ceph-mon[33017]: pgmap v1236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:51 smithi033 ceph-mon[27845]: pgmap v1237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:51 smithi096 ceph-mon[32753]: pgmap v1237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:51 smithi194 ceph-mon[33017]: pgmap v1237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1011.694716 secs 2024-04-26T12:07:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1011.694716 secs 2024-04-26T12:07:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1011.694716 secs 2024-04-26T12:07:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:53 smithi033 ceph-mon[27845]: pgmap v1238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1014.408367 secs 2024-04-26T12:07:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:53 smithi096 ceph-mon[32753]: pgmap v1238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1014.408367 secs 2024-04-26T12:07:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:53 smithi194 ceph-mon[33017]: pgmap v1238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1014.408367 secs 2024-04-26T12:07:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:55 smithi033 ceph-mon[27845]: pgmap v1239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:07:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:55 smithi096 ceph-mon[32753]: pgmap v1239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:07:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:55 smithi194 ceph-mon[33017]: pgmap v1239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:07:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:07:57.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:07:57.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:07:57.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:07:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:07:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:07:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:07:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:07:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:07:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:07:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:57 smithi033 ceph-mon[27845]: pgmap v1240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:57 smithi033 ceph-mon[27845]: pgmap v1241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1016.694791 secs 2024-04-26T12:07:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:57 smithi096 ceph-mon[32753]: pgmap v1240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:57 smithi096 ceph-mon[32753]: pgmap v1241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1016.694791 secs 2024-04-26T12:07:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:57 smithi194 ceph-mon[33017]: pgmap v1240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:07:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:57 smithi194 ceph-mon[33017]: pgmap v1241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:07:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1016.694791 secs 2024-04-26T12:07:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1019.408493 secs 2024-04-26T12:07:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1019.408493 secs 2024-04-26T12:07:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1019.408493 secs 2024-04-26T12:08:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:07:59 smithi096 ceph-mon[32753]: pgmap v1242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:07:59 smithi194 ceph-mon[33017]: pgmap v1242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:07:59 smithi033 ceph-mon[27845]: pgmap v1242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:01 smithi194 ceph-mon[33017]: pgmap v1243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:01 smithi033 ceph-mon[27845]: pgmap v1243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:01 smithi096 ceph-mon[32753]: pgmap v1243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1021.694849 secs 2024-04-26T12:08:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:03.368+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:03.368+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:03.368+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:08:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1021.694849 secs 2024-04-26T12:08:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1021.694849 secs 2024-04-26T12:08:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:03 smithi194 ceph-mon[33017]: pgmap v1244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1024.408614 secs 2024-04-26T12:08:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:03 smithi033 ceph-mon[27845]: pgmap v1244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1024.408614 secs 2024-04-26T12:08:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:03 smithi096 ceph-mon[32753]: pgmap v1244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1024.408614 secs 2024-04-26T12:08:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:05 smithi194 ceph-mon[33017]: pgmap v1245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:08:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:05 smithi033 ceph-mon[27845]: pgmap v1245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:08:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:05 smithi096 ceph-mon[32753]: pgmap v1245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:08:08.240 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:07 smithi033 ceph-mon[27845]: pgmap v1246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:08:08.241 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1026.694907 secs 2024-04-26T12:08:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:07 smithi194 ceph-mon[33017]: pgmap v1246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:08:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1026.694907 secs 2024-04-26T12:08:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:07 smithi096 ceph-mon[32753]: pgmap v1246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:08:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1026.694907 secs 2024-04-26T12:08:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1029.408750 secs 2024-04-26T12:08:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1029.408750 secs 2024-04-26T12:08:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1029.408750 secs 2024-04-26T12:08:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:09 smithi194 ceph-mon[33017]: pgmap v1247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:09 smithi033 ceph-mon[27845]: pgmap v1247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:09 smithi096 ceph-mon[32753]: pgmap v1247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:11 smithi194 ceph-mon[33017]: pgmap v1248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:11 smithi033 ceph-mon[27845]: pgmap v1248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:11 smithi096 ceph-mon[32753]: pgmap v1248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1031.694972 secs 2024-04-26T12:08:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1031.694972 secs 2024-04-26T12:08:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1031.694972 secs 2024-04-26T12:08:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:13 smithi194 ceph-mon[33017]: pgmap v1249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1034.408883 secs 2024-04-26T12:08:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:13 smithi033 ceph-mon[27845]: pgmap v1249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1034.408883 secs 2024-04-26T12:08:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:13 smithi096 ceph-mon[32753]: pgmap v1249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1034.408883 secs 2024-04-26T12:08:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:15 smithi194 ceph-mon[33017]: pgmap v1250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:15 smithi033 ceph-mon[27845]: pgmap v1250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:15 smithi096 ceph-mon[32753]: pgmap v1250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:17 smithi194 ceph-mon[33017]: pgmap v1251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1036.695056 secs 2024-04-26T12:08:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:18.370+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:18.370+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:18.370+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:08:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:17 smithi033 ceph-mon[27845]: pgmap v1251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1036.695056 secs 2024-04-26T12:08:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:17 smithi096 ceph-mon[32753]: pgmap v1251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1036.695056 secs 2024-04-26T12:08:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1039.409019 secs 2024-04-26T12:08:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1039.409019 secs 2024-04-26T12:08:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1039.409019 secs 2024-04-26T12:08:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:20 smithi194 ceph-mon[33017]: pgmap v1252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:20 smithi033 ceph-mon[27845]: pgmap v1252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:20 smithi096 ceph-mon[32753]: pgmap v1252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:21.974+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:08:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:21.974+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:08:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:21.974+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:08:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:21.974+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:08:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:22 smithi194 ceph-mon[33017]: pgmap v1253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:22 smithi033 ceph-mon[27845]: pgmap v1253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:22 smithi096 ceph-mon[32753]: pgmap v1253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:23.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1041.695145 secs 2024-04-26T12:08:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1041.695145 secs 2024-04-26T12:08:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1041.695145 secs 2024-04-26T12:08:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:24 smithi096 ceph-mon[32753]: pgmap v1254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1044.409155 secs 2024-04-26T12:08:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:24 smithi194 ceph-mon[33017]: pgmap v1254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1044.409155 secs 2024-04-26T12:08:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:24 smithi033 ceph-mon[27845]: pgmap v1254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1044.409155 secs 2024-04-26T12:08:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:26 smithi194 ceph-mon[33017]: pgmap v1255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:26 smithi033 ceph-mon[27845]: pgmap v1255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:26 smithi096 ceph-mon[32753]: pgmap v1255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:28 smithi194 ceph-mon[33017]: pgmap v1256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1046.695232 secs 2024-04-26T12:08:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:28 smithi033 ceph-mon[27845]: pgmap v1256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1046.695232 secs 2024-04-26T12:08:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:28 smithi096 ceph-mon[32753]: pgmap v1256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1046.695232 secs 2024-04-26T12:08:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1049.409215 secs 2024-04-26T12:08:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1049.409215 secs 2024-04-26T12:08:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1049.409215 secs 2024-04-26T12:08:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:30 smithi194 ceph-mon[33017]: pgmap v1257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:30 smithi033 ceph-mon[27845]: pgmap v1257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:30 smithi096 ceph-mon[32753]: pgmap v1257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:32 smithi194 ceph-mon[33017]: pgmap v1258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:32 smithi033 ceph-mon[27845]: pgmap v1258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:32 smithi096 ceph-mon[32753]: pgmap v1258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1051.695293 secs 2024-04-26T12:08:33.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:33.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:33.371+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:08:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:33.371+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1051.695293 secs 2024-04-26T12:08:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1051.695293 secs 2024-04-26T12:08:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:34 smithi194 ceph-mon[33017]: pgmap v1259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1054.409295 secs 2024-04-26T12:08:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:34 smithi033 ceph-mon[27845]: pgmap v1259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1054.409295 secs 2024-04-26T12:08:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:34 smithi096 ceph-mon[32753]: pgmap v1259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1054.409295 secs 2024-04-26T12:08:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:36 smithi194 ceph-mon[33017]: pgmap v1260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:36 smithi033 ceph-mon[27845]: pgmap v1260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:36 smithi096 ceph-mon[32753]: pgmap v1260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:38 smithi194 ceph-mon[33017]: pgmap v1261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1056.695384 secs 2024-04-26T12:08:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:38 smithi033 ceph-mon[27845]: pgmap v1261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1056.695384 secs 2024-04-26T12:08:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:38 smithi096 ceph-mon[32753]: pgmap v1261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1056.695384 secs 2024-04-26T12:08:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1059.409417 secs 2024-04-26T12:08:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1059.409417 secs 2024-04-26T12:08:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1059.409417 secs 2024-04-26T12:08:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:40 smithi194 ceph-mon[33017]: pgmap v1262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:40 smithi033 ceph-mon[27845]: pgmap v1262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:40 smithi096 ceph-mon[32753]: pgmap v1262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:42 smithi194 ceph-mon[33017]: pgmap v1263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:42 smithi033 ceph-mon[27845]: pgmap v1263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:42 smithi096 ceph-mon[32753]: pgmap v1263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1061.695472 secs 2024-04-26T12:08:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1061.695472 secs 2024-04-26T12:08:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1061.695472 secs 2024-04-26T12:08:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:44 smithi033 ceph-mon[27845]: pgmap v1264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1064.409535 secs 2024-04-26T12:08:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:44 smithi096 ceph-mon[32753]: pgmap v1264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1064.409535 secs 2024-04-26T12:08:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:44 smithi194 ceph-mon[33017]: pgmap v1264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1064.409535 secs 2024-04-26T12:08:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:46 smithi033 ceph-mon[27845]: pgmap v1265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:46 smithi096 ceph-mon[32753]: pgmap v1265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:46 smithi194 ceph-mon[33017]: pgmap v1265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:48.385+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:48.385+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:08:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:08:48.385+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:08:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:48 smithi033 ceph-mon[27845]: pgmap v1266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1066.695560 secs 2024-04-26T12:08:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:48 smithi096 ceph-mon[32753]: pgmap v1266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1066.695560 secs 2024-04-26T12:08:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:48 smithi194 ceph-mon[33017]: pgmap v1266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1066.695560 secs 2024-04-26T12:08:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1069.409682 secs 2024-04-26T12:08:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1069.409682 secs 2024-04-26T12:08:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1069.409682 secs 2024-04-26T12:08:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:50 smithi033 ceph-mon[27845]: pgmap v1267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:50 smithi096 ceph-mon[32753]: pgmap v1267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:50 smithi194 ceph-mon[33017]: pgmap v1267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:52 smithi033 ceph-mon[27845]: pgmap v1268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:52 smithi096 ceph-mon[32753]: pgmap v1268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:52 smithi194 ceph-mon[33017]: pgmap v1268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:08:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1071.695648 secs 2024-04-26T12:08:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1071.695648 secs 2024-04-26T12:08:53.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1071.695648 secs 2024-04-26T12:08:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:54 smithi033 ceph-mon[27845]: pgmap v1269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1074.409797 secs 2024-04-26T12:08:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:54 smithi096 ceph-mon[32753]: pgmap v1269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1074.409797 secs 2024-04-26T12:08:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:54 smithi194 ceph-mon[33017]: pgmap v1269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1074.409797 secs 2024-04-26T12:08:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:56 smithi033 ceph-mon[27845]: pgmap v1270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:56 smithi096 ceph-mon[32753]: pgmap v1270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:56 smithi194 ceph-mon[33017]: pgmap v1270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:08:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:08:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:08:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:58 smithi096 ceph-mon[32753]: pgmap v1271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1076.695737 secs 2024-04-26T12:08:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:08:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:08:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:08:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:58 smithi194 ceph-mon[33017]: pgmap v1271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1076.695737 secs 2024-04-26T12:08:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:08:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:08:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:08:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:58 smithi033 ceph-mon[27845]: pgmap v1271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:08:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1076.695737 secs 2024-04-26T12:08:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:08:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:08:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:08:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1079.409909 secs 2024-04-26T12:08:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:59 smithi194 ceph-mon[33017]: pgmap v1272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:08:59 smithi194 ceph-mon[33017]: pgmap v1273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1079.409909 secs 2024-04-26T12:08:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:59 smithi033 ceph-mon[27845]: pgmap v1272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:08:59 smithi033 ceph-mon[27845]: pgmap v1273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:08:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1079.409909 secs 2024-04-26T12:08:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:59 smithi096 ceph-mon[32753]: pgmap v1272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:08:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:08:59 smithi096 ceph-mon[32753]: pgmap v1273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:01 smithi194 ceph-mon[33017]: pgmap v1274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:01 smithi033 ceph-mon[27845]: pgmap v1274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:01 smithi096 ceph-mon[32753]: pgmap v1274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:03.400+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:03.401+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:03.401+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:09:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:03 smithi194 ceph-mon[33017]: pgmap v1275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1081.695824 secs 2024-04-26T12:09:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:03 smithi033 ceph-mon[27845]: pgmap v1275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1081.695824 secs 2024-04-26T12:09:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:03 smithi096 ceph-mon[32753]: pgmap v1275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1081.695824 secs 2024-04-26T12:09:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1084.409981 secs 2024-04-26T12:09:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1084.409981 secs 2024-04-26T12:09:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1084.409981 secs 2024-04-26T12:09:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:05 smithi194 ceph-mon[33017]: pgmap v1276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:09:05.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:05 smithi033 ceph-mon[27845]: pgmap v1276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:09:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:05 smithi096 ceph-mon[32753]: pgmap v1276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:09:07.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:07 smithi194 ceph-mon[33017]: pgmap v1277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:07 smithi033 ceph-mon[27845]: pgmap v1277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:07 smithi096 ceph-mon[32753]: pgmap v1277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:09:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1086.695913 secs 2024-04-26T12:09:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1086.695913 secs 2024-04-26T12:09:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1086.695913 secs 2024-04-26T12:09:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1089.410127 secs 2024-04-26T12:09:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:09 smithi194 ceph-mon[33017]: pgmap v1278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:09:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1089.410127 secs 2024-04-26T12:09:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:09 smithi033 ceph-mon[27845]: pgmap v1278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:09:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1089.410127 secs 2024-04-26T12:09:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:09 smithi096 ceph-mon[32753]: pgmap v1278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:09:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:11 smithi194 ceph-mon[33017]: pgmap v1279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:11 smithi033 ceph-mon[27845]: pgmap v1279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:11 smithi096 ceph-mon[32753]: pgmap v1279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:13 smithi194 ceph-mon[33017]: pgmap v1280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1091.695979 secs 2024-04-26T12:09:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:13 smithi033 ceph-mon[27845]: pgmap v1280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1091.695979 secs 2024-04-26T12:09:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:13 smithi096 ceph-mon[32753]: pgmap v1280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1091.695979 secs 2024-04-26T12:09:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1094.410262 secs 2024-04-26T12:09:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1094.410262 secs 2024-04-26T12:09:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1094.410262 secs 2024-04-26T12:09:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:15 smithi194 ceph-mon[33017]: pgmap v1281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:15 smithi033 ceph-mon[27845]: pgmap v1281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:15 smithi096 ceph-mon[32753]: pgmap v1281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:17 smithi194 ceph-mon[33017]: pgmap v1282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:17 smithi033 ceph-mon[27845]: pgmap v1282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:17 smithi096 ceph-mon[32753]: pgmap v1282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:18.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:18.410+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:18.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:18.410+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:18.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:18.410+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:09:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1096.696067 secs 2024-04-26T12:09:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1096.696067 secs 2024-04-26T12:09:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1096.696067 secs 2024-04-26T12:09:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1099.410387 secs 2024-04-26T12:09:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:19 smithi194 ceph-mon[33017]: pgmap v1283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1099.410387 secs 2024-04-26T12:09:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:19 smithi033 ceph-mon[27845]: pgmap v1283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1099.410387 secs 2024-04-26T12:09:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:19 smithi096 ceph-mon[32753]: pgmap v1283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:20 smithi194 ceph-mon[33017]: pgmap v1284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:20 smithi033 ceph-mon[27845]: pgmap v1284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:20 smithi096 ceph-mon[32753]: pgmap v1284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:23.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:23 smithi033 ceph-mon[27845]: pgmap v1285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:23.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1101.696127 secs 2024-04-26T12:09:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:23 smithi096 ceph-mon[32753]: pgmap v1285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1101.696127 secs 2024-04-26T12:09:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:23 smithi194 ceph-mon[33017]: pgmap v1285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1101.696127 secs 2024-04-26T12:09:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1104.410524 secs 2024-04-26T12:09:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1104.410524 secs 2024-04-26T12:09:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1104.410524 secs 2024-04-26T12:09:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:25 smithi033 ceph-mon[27845]: pgmap v1286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:25 smithi096 ceph-mon[32753]: pgmap v1286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:25 smithi194 ceph-mon[33017]: pgmap v1286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:27 smithi033 ceph-mon[27845]: pgmap v1287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:27 smithi096 ceph-mon[32753]: pgmap v1287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:27 smithi194 ceph-mon[33017]: pgmap v1287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1106.696215 secs 2024-04-26T12:09:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1106.696215 secs 2024-04-26T12:09:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1106.696215 secs 2024-04-26T12:09:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1109.410643 secs 2024-04-26T12:09:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:29 smithi033 ceph-mon[27845]: pgmap v1288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1109.410643 secs 2024-04-26T12:09:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:29 smithi096 ceph-mon[32753]: pgmap v1288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1109.410643 secs 2024-04-26T12:09:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:29 smithi194 ceph-mon[33017]: pgmap v1288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:31 smithi033 ceph-mon[27845]: pgmap v1289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:31 smithi096 ceph-mon[32753]: pgmap v1289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:31 smithi194 ceph-mon[33017]: pgmap v1289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:33.417+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:33.417+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:33.417+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:09:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:33 smithi033 ceph-mon[27845]: pgmap v1290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1111.696305 secs 2024-04-26T12:09:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:33 smithi096 ceph-mon[32753]: pgmap v1290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1111.696305 secs 2024-04-26T12:09:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:33 smithi194 ceph-mon[33017]: pgmap v1290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1111.696305 secs 2024-04-26T12:09:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1114.410798 secs 2024-04-26T12:09:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1114.410798 secs 2024-04-26T12:09:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1114.410798 secs 2024-04-26T12:09:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:35 smithi033 ceph-mon[27845]: pgmap v1291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:35 smithi096 ceph-mon[32753]: pgmap v1291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:35 smithi194 ceph-mon[33017]: pgmap v1291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:37 smithi033 ceph-mon[27845]: pgmap v1292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:37 smithi096 ceph-mon[32753]: pgmap v1292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:37 smithi194 ceph-mon[33017]: pgmap v1292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1116.696367 secs 2024-04-26T12:09:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1116.696367 secs 2024-04-26T12:09:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1116.696367 secs 2024-04-26T12:09:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1119.410929 secs 2024-04-26T12:09:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:39 smithi096 ceph-mon[32753]: pgmap v1293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1119.410929 secs 2024-04-26T12:09:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:39 smithi194 ceph-mon[33017]: pgmap v1293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1119.410929 secs 2024-04-26T12:09:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:39 smithi033 ceph-mon[27845]: pgmap v1293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:41.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:41 smithi096 ceph-mon[32753]: pgmap v1294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:41 smithi194 ceph-mon[33017]: pgmap v1294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:41 smithi033 ceph-mon[27845]: pgmap v1294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:43 smithi194 ceph-mon[33017]: pgmap v1295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1121.696456 secs 2024-04-26T12:09:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:43 smithi033 ceph-mon[27845]: pgmap v1295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1121.696456 secs 2024-04-26T12:09:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:43 smithi096 ceph-mon[32753]: pgmap v1295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1121.696456 secs 2024-04-26T12:09:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1124.411068 secs 2024-04-26T12:09:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1124.411068 secs 2024-04-26T12:09:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1124.411068 secs 2024-04-26T12:09:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:45 smithi194 ceph-mon[33017]: pgmap v1296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:45 smithi033 ceph-mon[27845]: pgmap v1296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:45 smithi096 ceph-mon[32753]: pgmap v1296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:47 smithi194 ceph-mon[33017]: pgmap v1297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:47 smithi033 ceph-mon[27845]: pgmap v1297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:47 smithi096 ceph-mon[32753]: pgmap v1297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:48.432+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:48.432+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:09:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:09:48.432+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:09:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1126.696544 secs 2024-04-26T12:09:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1126.696544 secs 2024-04-26T12:09:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1126.696544 secs 2024-04-26T12:09:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1129.411193 secs 2024-04-26T12:09:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:49 smithi194 ceph-mon[33017]: pgmap v1298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1129.411193 secs 2024-04-26T12:09:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:49 smithi033 ceph-mon[27845]: pgmap v1298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1129.411193 secs 2024-04-26T12:09:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:49 smithi096 ceph-mon[32753]: pgmap v1298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:51 smithi194 ceph-mon[33017]: pgmap v1299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:51 smithi033 ceph-mon[27845]: pgmap v1299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:51 smithi096 ceph-mon[32753]: pgmap v1299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:53.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:53 smithi033 ceph-mon[27845]: pgmap v1300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:53.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1131.696629 secs 2024-04-26T12:09:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:53 smithi194 ceph-mon[33017]: pgmap v1300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1131.696629 secs 2024-04-26T12:09:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:53 smithi096 ceph-mon[32753]: pgmap v1300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1131.696629 secs 2024-04-26T12:09:54.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1134.411287 secs 2024-04-26T12:09:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1134.411287 secs 2024-04-26T12:09:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1134.411287 secs 2024-04-26T12:09:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:55 smithi194 ceph-mon[33017]: pgmap v1301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:55 smithi033 ceph-mon[27845]: pgmap v1301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:55 smithi096 ceph-mon[32753]: pgmap v1301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:57.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:57 smithi194 ceph-mon[33017]: pgmap v1302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:57 smithi033 ceph-mon[27845]: pgmap v1302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:57 smithi096 ceph-mon[32753]: pgmap v1302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:09:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1136.696718 secs 2024-04-26T12:09:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1136.696718 secs 2024-04-26T12:09:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1136.696718 secs 2024-04-26T12:09:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1139.411414 secs 2024-04-26T12:09:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:59 smithi194 ceph-mon[33017]: pgmap v1303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:59.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:09:59.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:09:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:09:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1139.411414 secs 2024-04-26T12:09:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:59 smithi033 ceph-mon[27845]: pgmap v1303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:09:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:09:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:09:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1139.411414 secs 2024-04-26T12:09:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:59 smithi096 ceph-mon[32753]: pgmap v1303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:09:59.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:09:59.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:09:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:10:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:10:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:10:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:00 smithi194 ceph-mon[33017]: pgmap v1304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:10:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:10:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:00 smithi033 ceph-mon[27845]: pgmap v1304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:00.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:10:00.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:10:00.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:00 smithi096 ceph-mon[32753]: pgmap v1304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:10:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:10:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:01 smithi194 ceph-mon[33017]: pgmap v1305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:10:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:01 smithi194 ceph-mon[33017]: pgmap v1306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:10:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:10:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:10:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:01 smithi033 ceph-mon[27845]: pgmap v1305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:10:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:01 smithi033 ceph-mon[27845]: pgmap v1306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:10:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:10:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:10:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:01 smithi096 ceph-mon[32753]: pgmap v1305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:10:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:01 smithi096 ceph-mon[32753]: pgmap v1306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:10:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1141.696806 secs 2024-04-26T12:10:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1141.696806 secs 2024-04-26T12:10:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1141.696806 secs 2024-04-26T12:10:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:03.443+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:03.443+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:03.443+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:10:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:03 smithi194 ceph-mon[33017]: pgmap v1307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1144.411476 secs 2024-04-26T12:10:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:03 smithi033 ceph-mon[27845]: pgmap v1307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1144.411476 secs 2024-04-26T12:10:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:03 smithi096 ceph-mon[32753]: pgmap v1307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1144.411476 secs 2024-04-26T12:10:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:05 smithi194 ceph-mon[33017]: pgmap v1308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:05 smithi033 ceph-mon[27845]: pgmap v1308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:05 smithi096 ceph-mon[32753]: pgmap v1308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:07 smithi194 ceph-mon[33017]: pgmap v1309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1146.696894 secs 2024-04-26T12:10:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:07 smithi033 ceph-mon[27845]: pgmap v1309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1146.696894 secs 2024-04-26T12:10:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:07 smithi096 ceph-mon[32753]: pgmap v1309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:10:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1146.696894 secs 2024-04-26T12:10:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1149.411606 secs 2024-04-26T12:10:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1149.411606 secs 2024-04-26T12:10:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1149.411606 secs 2024-04-26T12:10:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:09 smithi194 ceph-mon[33017]: pgmap v1310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:10:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:09 smithi033 ceph-mon[27845]: pgmap v1310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:10:10.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:09 smithi096 ceph-mon[32753]: pgmap v1310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:10:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:11 smithi194 ceph-mon[33017]: pgmap v1311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:10:12.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:11 smithi033 ceph-mon[27845]: pgmap v1311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:10:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:11 smithi096 ceph-mon[32753]: pgmap v1311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:10:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1151.696984 secs 2024-04-26T12:10:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1151.696984 secs 2024-04-26T12:10:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1151.696984 secs 2024-04-26T12:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:13 smithi194 ceph-mon[33017]: pgmap v1312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1154.411731 secs 2024-04-26T12:10:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:13 smithi033 ceph-mon[27845]: pgmap v1312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1154.411731 secs 2024-04-26T12:10:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:13 smithi096 ceph-mon[32753]: pgmap v1312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1154.411731 secs 2024-04-26T12:10:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:15 smithi194 ceph-mon[33017]: pgmap v1313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:15 smithi033 ceph-mon[27845]: pgmap v1313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:15 smithi096 ceph-mon[32753]: pgmap v1313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:17 smithi194 ceph-mon[33017]: pgmap v1314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1156.697071 secs 2024-04-26T12:10:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:17 smithi033 ceph-mon[27845]: pgmap v1314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1156.697071 secs 2024-04-26T12:10:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:17 smithi096 ceph-mon[32753]: pgmap v1314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1156.697071 secs 2024-04-26T12:10:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:18.457+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:18.457+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:10:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:18.457+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:19.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1159.411853 secs 2024-04-26T12:10:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1159.411853 secs 2024-04-26T12:10:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1159.411853 secs 2024-04-26T12:10:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:19 smithi194 ceph-mon[33017]: pgmap v1315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:19 smithi033 ceph-mon[27845]: pgmap v1315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:19 smithi096 ceph-mon[32753]: pgmap v1315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:21 smithi194 ceph-mon[33017]: pgmap v1316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:21 smithi033 ceph-mon[27845]: pgmap v1316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:21 smithi096 ceph-mon[32753]: pgmap v1316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:23.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1161.697161 secs 2024-04-26T12:10:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1161.697161 secs 2024-04-26T12:10:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1161.697161 secs 2024-04-26T12:10:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:23 smithi096 ceph-mon[32753]: pgmap v1317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1164.411979 secs 2024-04-26T12:10:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:23 smithi194 ceph-mon[33017]: pgmap v1317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1164.411979 secs 2024-04-26T12:10:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:23 smithi033 ceph-mon[27845]: pgmap v1317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1164.411979 secs 2024-04-26T12:10:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:25 smithi194 ceph-mon[33017]: pgmap v1318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:25 smithi033 ceph-mon[27845]: pgmap v1318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:25 smithi096 ceph-mon[32753]: pgmap v1318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:27 smithi194 ceph-mon[33017]: pgmap v1319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1166.697247 secs 2024-04-26T12:10:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:27 smithi033 ceph-mon[27845]: pgmap v1319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:28.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1166.697247 secs 2024-04-26T12:10:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:27 smithi096 ceph-mon[32753]: pgmap v1319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1166.697247 secs 2024-04-26T12:10:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1169.412136 secs 2024-04-26T12:10:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1169.412136 secs 2024-04-26T12:10:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1169.412136 secs 2024-04-26T12:10:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:29 smithi194 ceph-mon[33017]: pgmap v1320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:29 smithi033 ceph-mon[27845]: pgmap v1320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:29 smithi096 ceph-mon[32753]: pgmap v1320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:31 smithi194 ceph-mon[33017]: pgmap v1321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:31 smithi033 ceph-mon[27845]: pgmap v1321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:32.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:31 smithi096 ceph-mon[32753]: pgmap v1321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1171.697335 secs 2024-04-26T12:10:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1171.697335 secs 2024-04-26T12:10:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1171.697335 secs 2024-04-26T12:10:33.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:33.474+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:33.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:33.474+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:33.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:33.474+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:10:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:33 smithi194 ceph-mon[33017]: pgmap v1322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1174.412262 secs 2024-04-26T12:10:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:33 smithi033 ceph-mon[27845]: pgmap v1322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1174.412262 secs 2024-04-26T12:10:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:33 smithi096 ceph-mon[32753]: pgmap v1322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1174.412262 secs 2024-04-26T12:10:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:35 smithi194 ceph-mon[33017]: pgmap v1323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:35 smithi033 ceph-mon[27845]: pgmap v1323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:35 smithi096 ceph-mon[32753]: pgmap v1323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:37 smithi194 ceph-mon[33017]: pgmap v1324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1176.697422 secs 2024-04-26T12:10:38.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:37 smithi033 ceph-mon[27845]: pgmap v1324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:38.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1176.697422 secs 2024-04-26T12:10:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:37 smithi096 ceph-mon[32753]: pgmap v1324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1176.697422 secs 2024-04-26T12:10:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1179.412343 secs 2024-04-26T12:10:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1179.412343 secs 2024-04-26T12:10:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1179.412343 secs 2024-04-26T12:10:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:39 smithi194 ceph-mon[33017]: pgmap v1325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:39 smithi033 ceph-mon[27845]: pgmap v1325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:39 smithi096 ceph-mon[32753]: pgmap v1325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:41 smithi194 ceph-mon[33017]: pgmap v1326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:41 smithi033 ceph-mon[27845]: pgmap v1326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:41 smithi096 ceph-mon[32753]: pgmap v1326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1181.697510 secs 2024-04-26T12:10:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1181.697510 secs 2024-04-26T12:10:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1181.697510 secs 2024-04-26T12:10:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:43 smithi194 ceph-mon[33017]: pgmap v1327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1184.412488 secs 2024-04-26T12:10:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:43 smithi033 ceph-mon[27845]: pgmap v1327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1184.412488 secs 2024-04-26T12:10:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:43 smithi096 ceph-mon[32753]: pgmap v1327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:10:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1184.412488 secs 2024-04-26T12:10:46.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:45 smithi033 ceph-mon[27845]: pgmap v1328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:45 smithi096 ceph-mon[32753]: pgmap v1328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:45 smithi194 ceph-mon[33017]: pgmap v1328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:10:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:47 smithi033 ceph-mon[27845]: pgmap v1329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1186.697598 secs 2024-04-26T12:10:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:47 smithi096 ceph-mon[32753]: pgmap v1329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1186.697598 secs 2024-04-26T12:10:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:47 smithi194 ceph-mon[33017]: pgmap v1329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1186.697598 secs 2024-04-26T12:10:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:48.489+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:48.489+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:10:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:10:48.489+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:10:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1189.412610 secs 2024-04-26T12:10:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1189.412610 secs 2024-04-26T12:10:49.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1189.412610 secs 2024-04-26T12:10:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:49 smithi033 ceph-mon[27845]: pgmap v1330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:49 smithi096 ceph-mon[32753]: pgmap v1330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:49 smithi194 ceph-mon[33017]: pgmap v1330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:51 smithi033 ceph-mon[27845]: pgmap v1331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:51 smithi096 ceph-mon[32753]: pgmap v1331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:51 smithi194 ceph-mon[33017]: pgmap v1331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1191.697657 secs 2024-04-26T12:10:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1191.697657 secs 2024-04-26T12:10:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1191.697657 secs 2024-04-26T12:10:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:53 smithi033 ceph-mon[27845]: pgmap v1332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1194.412708 secs 2024-04-26T12:10:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:53 smithi096 ceph-mon[32753]: pgmap v1332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1194.412708 secs 2024-04-26T12:10:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:53 smithi194 ceph-mon[33017]: pgmap v1332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1194.412708 secs 2024-04-26T12:10:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:55 smithi033 ceph-mon[27845]: pgmap v1333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:55 smithi096 ceph-mon[32753]: pgmap v1333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:55 smithi194 ceph-mon[33017]: pgmap v1333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:10:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:57 smithi033 ceph-mon[27845]: pgmap v1334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1196.697714 secs 2024-04-26T12:10:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:57 smithi096 ceph-mon[32753]: pgmap v1334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1196.697714 secs 2024-04-26T12:10:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:57 smithi194 ceph-mon[33017]: pgmap v1334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:10:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1196.697714 secs 2024-04-26T12:10:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1199.412828 secs 2024-04-26T12:10:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1199.412828 secs 2024-04-26T12:10:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1199.412828 secs 2024-04-26T12:11:00.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:10:59 smithi033 ceph-mon[27845]: pgmap v1335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:10:59 smithi096 ceph-mon[32753]: pgmap v1335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:10:59 smithi194 ceph-mon[33017]: pgmap v1335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:01 smithi033 ceph-mon[27845]: pgmap v1336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:11:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:01 smithi096 ceph-mon[32753]: pgmap v1336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:11:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:01 smithi194 ceph-mon[33017]: pgmap v1336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:11:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:11:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:11:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:11:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1201.697774 secs 2024-04-26T12:11:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:11:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:11:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:11:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1201.697774 secs 2024-04-26T12:11:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:11:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:11:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:11:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1201.697774 secs 2024-04-26T12:11:03.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:03.501+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:03.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:03.501+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:11:03.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:03.501+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:03 smithi194 ceph-mon[33017]: pgmap v1337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:03 smithi194 ceph-mon[33017]: pgmap v1338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1204.412978 secs 2024-04-26T12:11:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:03 smithi033 ceph-mon[27845]: pgmap v1337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:03 smithi033 ceph-mon[27845]: pgmap v1338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:04.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1204.412978 secs 2024-04-26T12:11:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:03 smithi096 ceph-mon[32753]: pgmap v1337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:03 smithi096 ceph-mon[32753]: pgmap v1338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1204.412978 secs 2024-04-26T12:11:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:05 smithi194 ceph-mon[33017]: pgmap v1339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:11:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:05 smithi033 ceph-mon[27845]: pgmap v1339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:11:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:05 smithi096 ceph-mon[32753]: pgmap v1339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:11:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:07 smithi194 ceph-mon[33017]: pgmap v1340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:08.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1206.697832 secs 2024-04-26T12:11:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:07 smithi033 ceph-mon[27845]: pgmap v1340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1206.697832 secs 2024-04-26T12:11:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:07 smithi096 ceph-mon[32753]: pgmap v1340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1206.697832 secs 2024-04-26T12:11:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1209.413135 secs 2024-04-26T12:11:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1209.413135 secs 2024-04-26T12:11:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1209.413135 secs 2024-04-26T12:11:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:09 smithi194 ceph-mon[33017]: pgmap v1341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:09 smithi033 ceph-mon[27845]: pgmap v1341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:09 smithi096 ceph-mon[32753]: pgmap v1341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:12.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:11 smithi194 ceph-mon[33017]: pgmap v1342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:11 smithi033 ceph-mon[27845]: pgmap v1342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:11 smithi096 ceph-mon[32753]: pgmap v1342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1211.697874 secs 2024-04-26T12:11:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1211.697874 secs 2024-04-26T12:11:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1211.697874 secs 2024-04-26T12:11:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:13 smithi194 ceph-mon[33017]: pgmap v1343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:11:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1214.413204 secs 2024-04-26T12:11:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:13 smithi033 ceph-mon[27845]: pgmap v1343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:11:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1214.413204 secs 2024-04-26T12:11:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:13 smithi096 ceph-mon[32753]: pgmap v1343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:11:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1214.413204 secs 2024-04-26T12:11:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:15 smithi194 ceph-mon[33017]: pgmap v1344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:15 smithi033 ceph-mon[27845]: pgmap v1344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:15 smithi096 ceph-mon[32753]: pgmap v1344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:18.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:17 smithi194 ceph-mon[33017]: pgmap v1345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:18.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1216.697979 secs 2024-04-26T12:11:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:17 smithi033 ceph-mon[27845]: pgmap v1345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1216.697979 secs 2024-04-26T12:11:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:17 smithi096 ceph-mon[32753]: pgmap v1345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1216.697979 secs 2024-04-26T12:11:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:18.507+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:18.507+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:11:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:18.507+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:19.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1219.413293 secs 2024-04-26T12:11:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1219.413293 secs 2024-04-26T12:11:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1219.413293 secs 2024-04-26T12:11:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:20 smithi194 ceph-mon[33017]: pgmap v1346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:20 smithi033 ceph-mon[27845]: pgmap v1346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:20 smithi096 ceph-mon[32753]: pgmap v1346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 msgr2=0x7feee40b0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T12:11:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 msgr2=0x7feee40b0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T12:11:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 0x7feee40b0f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0019000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T12:11:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 0x7feee40b0f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0019000 comp rx=0 tx=0).stop 2024-04-26T12:11:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 msgr2=0x7feee40b0f20 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T12:11:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 msgr2=0x7feee40b0f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T12:11:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40aeb30 0x7feee40b0f20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T12:11:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.703+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T12:11:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.704+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T12:11:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:21.706+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40b1460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7feef0024000 tx=0x7feef0012040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T12:11:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:22.004+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:11:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:22.004+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:11:22.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:22.012+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:11:22.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:22.012+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:11:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:22 smithi194 ceph-mon[33017]: pgmap v1347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:22 smithi033 ceph-mon[27845]: pgmap v1347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:22 smithi096 ceph-mon[32753]: pgmap v1347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:23.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1221.698068 secs 2024-04-26T12:11:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1221.698068 secs 2024-04-26T12:11:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1221.698068 secs 2024-04-26T12:11:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:24 smithi096 ceph-mon[32753]: pgmap v1348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1224.413379 secs 2024-04-26T12:11:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:24 smithi194 ceph-mon[33017]: pgmap v1348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1224.413379 secs 2024-04-26T12:11:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:24 smithi033 ceph-mon[27845]: pgmap v1348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1224.413379 secs 2024-04-26T12:11:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:26 smithi194 ceph-mon[33017]: pgmap v1349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:26 smithi033 ceph-mon[27845]: pgmap v1349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:26 smithi096 ceph-mon[32753]: pgmap v1349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:28 smithi194 ceph-mon[33017]: pgmap v1350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1226.698167 secs 2024-04-26T12:11:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:28 smithi033 ceph-mon[27845]: pgmap v1350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1226.698167 secs 2024-04-26T12:11:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:28 smithi096 ceph-mon[32753]: pgmap v1350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1226.698167 secs 2024-04-26T12:11:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1229.413540 secs 2024-04-26T12:11:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1229.413540 secs 2024-04-26T12:11:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1229.413540 secs 2024-04-26T12:11:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:30 smithi033 ceph-mon[27845]: pgmap v1351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:30 smithi194 ceph-mon[33017]: pgmap v1351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:30 smithi096 ceph-mon[32753]: pgmap v1351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:32 smithi194 ceph-mon[33017]: pgmap v1352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:32 smithi033 ceph-mon[27845]: pgmap v1352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:32 smithi096 ceph-mon[32753]: pgmap v1352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1231.698257 secs 2024-04-26T12:11:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1231.698257 secs 2024-04-26T12:11:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1231.698257 secs 2024-04-26T12:11:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:33.520+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:33.520+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:11:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:33.520+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:34 smithi194 ceph-mon[33017]: pgmap v1353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1234.413616 secs 2024-04-26T12:11:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:34 smithi033 ceph-mon[27845]: pgmap v1353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1234.413616 secs 2024-04-26T12:11:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:34 smithi096 ceph-mon[32753]: pgmap v1353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1234.413616 secs 2024-04-26T12:11:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:36 smithi194 ceph-mon[33017]: pgmap v1354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:36 smithi033 ceph-mon[27845]: pgmap v1354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:36 smithi096 ceph-mon[32753]: pgmap v1354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:38 smithi194 ceph-mon[33017]: pgmap v1355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1236.698340 secs 2024-04-26T12:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:38 smithi033 ceph-mon[27845]: pgmap v1355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1236.698340 secs 2024-04-26T12:11:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:38 smithi096 ceph-mon[32753]: pgmap v1355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1236.698340 secs 2024-04-26T12:11:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1239.413680 secs 2024-04-26T12:11:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1239.413680 secs 2024-04-26T12:11:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1239.413680 secs 2024-04-26T12:11:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:40 smithi194 ceph-mon[33017]: pgmap v1356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:40 smithi033 ceph-mon[27845]: pgmap v1356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:40 smithi096 ceph-mon[32753]: pgmap v1356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:42 smithi194 ceph-mon[33017]: pgmap v1357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:42 smithi033 ceph-mon[27845]: pgmap v1357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:42 smithi096 ceph-mon[32753]: pgmap v1357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1241.698443 secs 2024-04-26T12:11:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1241.698443 secs 2024-04-26T12:11:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1241.698443 secs 2024-04-26T12:11:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:44 smithi033 ceph-mon[27845]: pgmap v1358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1244.413804 secs 2024-04-26T12:11:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:44 smithi096 ceph-mon[32753]: pgmap v1358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1244.413804 secs 2024-04-26T12:11:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:44 smithi194 ceph-mon[33017]: pgmap v1358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1244.413804 secs 2024-04-26T12:11:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:46 smithi033 ceph-mon[27845]: pgmap v1359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:46 smithi096 ceph-mon[32753]: pgmap v1359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:46 smithi194 ceph-mon[33017]: pgmap v1359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:48 smithi033 ceph-mon[27845]: pgmap v1360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1246.698503 secs 2024-04-26T12:11:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:48 smithi096 ceph-mon[32753]: pgmap v1360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1246.698503 secs 2024-04-26T12:11:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:48.522+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:48.523+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:11:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:11:48.523+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:11:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:48 smithi194 ceph-mon[33017]: pgmap v1360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1246.698503 secs 2024-04-26T12:11:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1249.413925 secs 2024-04-26T12:11:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1249.413925 secs 2024-04-26T12:11:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1249.413925 secs 2024-04-26T12:11:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:50 smithi033 ceph-mon[27845]: pgmap v1361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:50 smithi096 ceph-mon[32753]: pgmap v1361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:50 smithi194 ceph-mon[33017]: pgmap v1361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:52.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:52 smithi033 ceph-mon[27845]: pgmap v1362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:52 smithi096 ceph-mon[32753]: pgmap v1362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:52 smithi194 ceph-mon[33017]: pgmap v1362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:11:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1251.698590 secs 2024-04-26T12:11:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1251.698590 secs 2024-04-26T12:11:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1251.698590 secs 2024-04-26T12:11:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:54 smithi033 ceph-mon[27845]: pgmap v1363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1254.414061 secs 2024-04-26T12:11:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:54 smithi096 ceph-mon[32753]: pgmap v1363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1254.414061 secs 2024-04-26T12:11:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:54 smithi194 ceph-mon[33017]: pgmap v1363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:11:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1254.414061 secs 2024-04-26T12:11:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:56 smithi033 ceph-mon[27845]: pgmap v1364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:56 smithi096 ceph-mon[32753]: pgmap v1364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:56 smithi194 ceph-mon[33017]: pgmap v1364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:11:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:58 smithi033 ceph-mon[27845]: pgmap v1365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1256.698671 secs 2024-04-26T12:11:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:58 smithi096 ceph-mon[32753]: pgmap v1365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1256.698671 secs 2024-04-26T12:11:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:58 smithi194 ceph-mon[33017]: pgmap v1365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:11:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1256.698671 secs 2024-04-26T12:11:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1259.414181 secs 2024-04-26T12:11:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1259.414181 secs 2024-04-26T12:11:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1259.414181 secs 2024-04-26T12:12:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:00 smithi033 ceph-mon[27845]: pgmap v1366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:00 smithi096 ceph-mon[32753]: pgmap v1366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:00 smithi194 ceph-mon[33017]: pgmap v1366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:02 smithi096 ceph-mon[32753]: pgmap v1367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:02 smithi194 ceph-mon[33017]: pgmap v1367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:02 smithi033 ceph-mon[27845]: pgmap v1367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:03.538+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:03.538+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:03.538+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:12:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1261.698756 secs 2024-04-26T12:12:03.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:12:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1261.698756 secs 2024-04-26T12:12:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:12:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1261.698756 secs 2024-04-26T12:12:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:12:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:04 smithi194 ceph-mon[33017]: pgmap v1368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1264.414247 secs 2024-04-26T12:12:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:04 smithi096 ceph-mon[32753]: pgmap v1368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1264.414247 secs 2024-04-26T12:12:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:04 smithi033 ceph-mon[27845]: pgmap v1368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1264.414247 secs 2024-04-26T12:12:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: pgmap v1369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:12:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: pgmap v1369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: pgmap v1369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:12:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:06.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:06.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:07 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:12:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:07 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:12:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:07 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:12:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:07 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:12:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:07 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:12:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:07 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:12:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:07 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:12:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:07 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:12:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:07 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:12:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:07 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:12:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:07 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:12:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:07 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:12:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:08 smithi194 ceph-mon[33017]: pgmap v1370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:12:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1266.698838 secs 2024-04-26T12:12:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:08 smithi033 ceph-mon[27845]: pgmap v1370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:12:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1266.698838 secs 2024-04-26T12:12:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:08 smithi096 ceph-mon[32753]: pgmap v1370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:12:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1266.698838 secs 2024-04-26T12:12:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1269.414367 secs 2024-04-26T12:12:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:12:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:12:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1269.414367 secs 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:12:09.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:12:09.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1269.414367 secs 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:12:09.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:12:09.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:12:09.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:12:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:10 smithi194 ceph-mon[33017]: pgmap v1371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:10 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:12:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:10 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:12:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:10 smithi194 ceph-mon[33017]: pgmap v1372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:12:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:10 smithi194 ceph-mon[33017]: pgmap v1373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:12:10.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:10 smithi033 ceph-mon[27845]: pgmap v1371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:10.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:10 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:12:10.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:10 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:12:10.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:10 smithi033 ceph-mon[27845]: pgmap v1372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:12:10.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:10 smithi033 ceph-mon[27845]: pgmap v1373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:12:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:10 smithi096 ceph-mon[32753]: pgmap v1371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:10 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:12:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:10 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:12:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:10 smithi096 ceph-mon[32753]: pgmap v1372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:12:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:10 smithi096 ceph-mon[32753]: pgmap v1373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:12:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:11 smithi194 ceph-mon[33017]: pgmap v1374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:11 smithi033 ceph-mon[27845]: pgmap v1374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:11 smithi096 ceph-mon[32753]: pgmap v1374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1271.698913 secs 2024-04-26T12:12:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1271.698913 secs 2024-04-26T12:12:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1271.698913 secs 2024-04-26T12:12:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:14 smithi033 ceph-mon[27845]: pgmap v1375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T12:12:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1274.414482 secs 2024-04-26T12:12:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:14 smithi096 ceph-mon[32753]: pgmap v1375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T12:12:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1274.414482 secs 2024-04-26T12:12:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:14 smithi194 ceph-mon[33017]: pgmap v1375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T12:12:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1274.414482 secs 2024-04-26T12:12:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:16 smithi033 ceph-mon[27845]: pgmap v1376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:16 smithi096 ceph-mon[32753]: pgmap v1376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:16 smithi194 ceph-mon[33017]: pgmap v1376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:18 smithi033 ceph-mon[27845]: pgmap v1377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1276.699000 secs 2024-04-26T12:12:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:18 smithi096 ceph-mon[32753]: pgmap v1377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1276.699000 secs 2024-04-26T12:12:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:18.546+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:18.546+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:18.546+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:12:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:18 smithi194 ceph-mon[33017]: pgmap v1377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:12:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1276.699000 secs 2024-04-26T12:12:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1279.414618 secs 2024-04-26T12:12:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1279.414618 secs 2024-04-26T12:12:19.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1279.414618 secs 2024-04-26T12:12:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:20 smithi096 ceph-mon[32753]: pgmap v1378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:12:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:20 smithi194 ceph-mon[33017]: pgmap v1378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:12:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:20 smithi033 ceph-mon[27845]: pgmap v1378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:12:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:22 smithi096 ceph-mon[32753]: pgmap v1379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:22 smithi194 ceph-mon[33017]: pgmap v1379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:22 smithi033 ceph-mon[27845]: pgmap v1379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:23.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1281.699084 secs 2024-04-26T12:12:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1281.699084 secs 2024-04-26T12:12:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1281.699084 secs 2024-04-26T12:12:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:24 smithi096 ceph-mon[32753]: pgmap v1380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1284.414744 secs 2024-04-26T12:12:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:24 smithi194 ceph-mon[33017]: pgmap v1380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1284.414744 secs 2024-04-26T12:12:24.612 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:24 smithi033 ceph-mon[27845]: pgmap v1380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:24.612 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1284.414744 secs 2024-04-26T12:12:26.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:26 smithi194 ceph-mon[33017]: pgmap v1381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:26.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:26 smithi033 ceph-mon[27845]: pgmap v1381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:26 smithi096 ceph-mon[32753]: pgmap v1381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:28 smithi194 ceph-mon[33017]: pgmap v1382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1286.699171 secs 2024-04-26T12:12:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:28 smithi033 ceph-mon[27845]: pgmap v1382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1286.699171 secs 2024-04-26T12:12:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:28 smithi096 ceph-mon[32753]: pgmap v1382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1286.699171 secs 2024-04-26T12:12:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1289.414878 secs 2024-04-26T12:12:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1289.414878 secs 2024-04-26T12:12:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1289.414878 secs 2024-04-26T12:12:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:30 smithi194 ceph-mon[33017]: pgmap v1383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:30 smithi033 ceph-mon[27845]: pgmap v1383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:30 smithi096 ceph-mon[32753]: pgmap v1383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:32 smithi194 ceph-mon[33017]: pgmap v1384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:32 smithi033 ceph-mon[27845]: pgmap v1384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:32 smithi096 ceph-mon[32753]: pgmap v1384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:33.553+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:33.553+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:33.553+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:12:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1291.699275 secs 2024-04-26T12:12:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1291.699275 secs 2024-04-26T12:12:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1291.699275 secs 2024-04-26T12:12:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:34 smithi194 ceph-mon[33017]: pgmap v1385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1294.414995 secs 2024-04-26T12:12:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:34 smithi033 ceph-mon[27845]: pgmap v1385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1294.414995 secs 2024-04-26T12:12:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:34 smithi096 ceph-mon[32753]: pgmap v1385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1294.414995 secs 2024-04-26T12:12:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:36 smithi194 ceph-mon[33017]: pgmap v1386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:36 smithi033 ceph-mon[27845]: pgmap v1386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:36 smithi096 ceph-mon[32753]: pgmap v1386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:38 smithi194 ceph-mon[33017]: pgmap v1387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1296.699363 secs 2024-04-26T12:12:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:38 smithi033 ceph-mon[27845]: pgmap v1387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1296.699363 secs 2024-04-26T12:12:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:38 smithi096 ceph-mon[32753]: pgmap v1387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1296.699363 secs 2024-04-26T12:12:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1299.415118 secs 2024-04-26T12:12:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1299.415118 secs 2024-04-26T12:12:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1299.415118 secs 2024-04-26T12:12:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:40 smithi194 ceph-mon[33017]: pgmap v1388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:40 smithi033 ceph-mon[27845]: pgmap v1388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:40 smithi096 ceph-mon[32753]: pgmap v1388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:42 smithi194 ceph-mon[33017]: pgmap v1389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:42 smithi033 ceph-mon[27845]: pgmap v1389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:42 smithi096 ceph-mon[32753]: pgmap v1389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1301.699446 secs 2024-04-26T12:12:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1301.699446 secs 2024-04-26T12:12:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1301.699446 secs 2024-04-26T12:12:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:44 smithi194 ceph-mon[33017]: pgmap v1390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1304.415240 secs 2024-04-26T12:12:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:44 smithi033 ceph-mon[27845]: pgmap v1390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1304.415240 secs 2024-04-26T12:12:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:44 smithi096 ceph-mon[32753]: pgmap v1390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1304.415240 secs 2024-04-26T12:12:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:46 smithi194 ceph-mon[33017]: pgmap v1391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:46 smithi033 ceph-mon[27845]: pgmap v1391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:46 smithi096 ceph-mon[32753]: pgmap v1391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:48.558+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:48.558+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:12:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:12:48.558+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:12:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:48 smithi194 ceph-mon[33017]: pgmap v1392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1306.699528 secs 2024-04-26T12:12:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:48 smithi033 ceph-mon[27845]: pgmap v1392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1306.699528 secs 2024-04-26T12:12:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:48 smithi096 ceph-mon[32753]: pgmap v1392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1306.699528 secs 2024-04-26T12:12:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1309.415323 secs 2024-04-26T12:12:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1309.415323 secs 2024-04-26T12:12:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1309.415323 secs 2024-04-26T12:12:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:50 smithi194 ceph-mon[33017]: pgmap v1393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:50 smithi033 ceph-mon[27845]: pgmap v1393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:50 smithi096 ceph-mon[32753]: pgmap v1393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:52 smithi194 ceph-mon[33017]: pgmap v1394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:52 smithi033 ceph-mon[27845]: pgmap v1394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:52 smithi096 ceph-mon[32753]: pgmap v1394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:53.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1311.699612 secs 2024-04-26T12:12:53.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1314.415441 secs 2024-04-26T12:12:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1311.699612 secs 2024-04-26T12:12:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1314.415441 secs 2024-04-26T12:12:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1311.699612 secs 2024-04-26T12:12:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1314.415441 secs 2024-04-26T12:12:54.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:54 smithi096 ceph-mon[32753]: pgmap v1395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:54 smithi194 ceph-mon[33017]: pgmap v1395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:54 smithi033 ceph-mon[27845]: pgmap v1395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:55 smithi194 ceph-mon[33017]: pgmap v1396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:55 smithi033 ceph-mon[27845]: pgmap v1396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:55 smithi096 ceph-mon[32753]: pgmap v1396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:12:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:57 smithi033 ceph-mon[27845]: pgmap v1397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:58.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1316.699695 secs 2024-04-26T12:12:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:57 smithi096 ceph-mon[32753]: pgmap v1397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1316.699695 secs 2024-04-26T12:12:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:57 smithi194 ceph-mon[33017]: pgmap v1397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:12:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1316.699695 secs 2024-04-26T12:12:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1319.415575 secs 2024-04-26T12:12:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1319.415575 secs 2024-04-26T12:12:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1319.415575 secs 2024-04-26T12:13:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:12:59 smithi096 ceph-mon[32753]: pgmap v1398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:12:59 smithi194 ceph-mon[33017]: pgmap v1398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:12:59 smithi033 ceph-mon[27845]: pgmap v1398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:01 smithi194 ceph-mon[33017]: pgmap v1399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:01 smithi033 ceph-mon[27845]: pgmap v1399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:01 smithi096 ceph-mon[32753]: pgmap v1399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:03.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1321.699777 secs 2024-04-26T12:13:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1321.699777 secs 2024-04-26T12:13:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1321.699777 secs 2024-04-26T12:13:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:03.566+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:03.566+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:13:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:03.566+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:03 smithi194 ceph-mon[33017]: pgmap v1400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1324.415703 secs 2024-04-26T12:13:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:03 smithi033 ceph-mon[27845]: pgmap v1400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1324.415703 secs 2024-04-26T12:13:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:03 smithi096 ceph-mon[32753]: pgmap v1400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1324.415703 secs 2024-04-26T12:13:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:05 smithi194 ceph-mon[33017]: pgmap v1401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:13:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:05 smithi033 ceph-mon[27845]: pgmap v1401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:13:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:05 smithi096 ceph-mon[32753]: pgmap v1401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:13:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:07 smithi194 ceph-mon[33017]: pgmap v1402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1326.699836 secs 2024-04-26T12:13:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:07 smithi033 ceph-mon[27845]: pgmap v1402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1326.699836 secs 2024-04-26T12:13:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:07 smithi096 ceph-mon[32753]: pgmap v1402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1326.699836 secs 2024-04-26T12:13:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1329.415845 secs 2024-04-26T12:13:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1329.415845 secs 2024-04-26T12:13:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1329.415845 secs 2024-04-26T12:13:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:09 smithi194 ceph-mon[33017]: pgmap v1403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:13:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:09 smithi033 ceph-mon[27845]: pgmap v1403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:13:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:09 smithi096 ceph-mon[32753]: pgmap v1403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:13:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:13:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:13:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:13:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:13:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:13:11.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:13:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:13:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:13:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:13:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:11 smithi194 ceph-mon[33017]: pgmap v1404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T12:13:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:11 smithi194 ceph-mon[33017]: pgmap v1405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:13:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:11 smithi033 ceph-mon[27845]: pgmap v1404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T12:13:12.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:11 smithi033 ceph-mon[27845]: pgmap v1405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:13:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:11 smithi096 ceph-mon[32753]: pgmap v1404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T12:13:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:11 smithi096 ceph-mon[32753]: pgmap v1405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:13:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1331.699921 secs 2024-04-26T12:13:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1331.699921 secs 2024-04-26T12:13:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1331.699921 secs 2024-04-26T12:13:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:14 smithi096 ceph-mon[32753]: pgmap v1406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:13:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1334.415990 secs 2024-04-26T12:13:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:14 smithi194 ceph-mon[33017]: pgmap v1406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:13:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1334.415990 secs 2024-04-26T12:13:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:14 smithi033 ceph-mon[27845]: pgmap v1406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:13:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1334.415990 secs 2024-04-26T12:13:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:16 smithi096 ceph-mon[32753]: pgmap v1407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:16 smithi194 ceph-mon[33017]: pgmap v1407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:16 smithi033 ceph-mon[27845]: pgmap v1407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:13:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:18.566+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:18.566+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:18.566+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:13:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:18 smithi194 ceph-mon[33017]: pgmap v1408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1336.700001 secs 2024-04-26T12:13:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:18 smithi033 ceph-mon[27845]: pgmap v1408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1336.700001 secs 2024-04-26T12:13:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:18 smithi096 ceph-mon[32753]: pgmap v1408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1336.700001 secs 2024-04-26T12:13:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1339.416141 secs 2024-04-26T12:13:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1339.416141 secs 2024-04-26T12:13:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1339.416141 secs 2024-04-26T12:13:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:20 smithi194 ceph-mon[33017]: pgmap v1409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:20 smithi033 ceph-mon[27845]: pgmap v1409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:20 smithi096 ceph-mon[32753]: pgmap v1409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:22 smithi194 ceph-mon[33017]: pgmap v1410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:13:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:22 smithi033 ceph-mon[27845]: pgmap v1410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:13:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:22 smithi096 ceph-mon[32753]: pgmap v1410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:13:23.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1341.700087 secs 2024-04-26T12:13:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1341.700087 secs 2024-04-26T12:13:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1341.700087 secs 2024-04-26T12:13:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:24 smithi096 ceph-mon[32753]: pgmap v1411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1344.416260 secs 2024-04-26T12:13:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:24 smithi194 ceph-mon[33017]: pgmap v1411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1344.416260 secs 2024-04-26T12:13:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:24 smithi033 ceph-mon[27845]: pgmap v1411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1344.416260 secs 2024-04-26T12:13:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:26 smithi194 ceph-mon[33017]: pgmap v1412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:26 smithi033 ceph-mon[27845]: pgmap v1412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:26 smithi096 ceph-mon[32753]: pgmap v1412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:28 smithi194 ceph-mon[33017]: pgmap v1413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1346.700170 secs 2024-04-26T12:13:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:28 smithi033 ceph-mon[27845]: pgmap v1413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1346.700170 secs 2024-04-26T12:13:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:28 smithi096 ceph-mon[32753]: pgmap v1413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1346.700170 secs 2024-04-26T12:13:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1349.416384 secs 2024-04-26T12:13:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1349.416384 secs 2024-04-26T12:13:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1349.416384 secs 2024-04-26T12:13:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:30 smithi194 ceph-mon[33017]: pgmap v1414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:30 smithi033 ceph-mon[27845]: pgmap v1414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:30 smithi096 ceph-mon[32753]: pgmap v1414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:32 smithi194 ceph-mon[33017]: pgmap v1415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:32 smithi033 ceph-mon[27845]: pgmap v1415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:32 smithi096 ceph-mon[32753]: pgmap v1415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:33.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:33.582+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:33.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:33.582+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:33.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:33.582+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:13:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1351.700226 secs 2024-04-26T12:13:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1351.700226 secs 2024-04-26T12:13:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1351.700226 secs 2024-04-26T12:13:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:34 smithi194 ceph-mon[33017]: pgmap v1416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1354.416445 secs 2024-04-26T12:13:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:34 smithi033 ceph-mon[27845]: pgmap v1416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1354.416445 secs 2024-04-26T12:13:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:34 smithi096 ceph-mon[32753]: pgmap v1416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1354.416445 secs 2024-04-26T12:13:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:36 smithi194 ceph-mon[33017]: pgmap v1417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:36 smithi033 ceph-mon[27845]: pgmap v1417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:36 smithi096 ceph-mon[32753]: pgmap v1417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:38 smithi194 ceph-mon[33017]: pgmap v1418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1356.700285 secs 2024-04-26T12:13:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:38 smithi033 ceph-mon[27845]: pgmap v1418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1356.700285 secs 2024-04-26T12:13:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:38 smithi096 ceph-mon[32753]: pgmap v1418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1356.700285 secs 2024-04-26T12:13:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1359.416573 secs 2024-04-26T12:13:39.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1359.416573 secs 2024-04-26T12:13:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1359.416573 secs 2024-04-26T12:13:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:40 smithi194 ceph-mon[33017]: pgmap v1419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:40 smithi033 ceph-mon[27845]: pgmap v1419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:40 smithi096 ceph-mon[32753]: pgmap v1419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:42 smithi194 ceph-mon[33017]: pgmap v1420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:42 smithi033 ceph-mon[27845]: pgmap v1420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:42 smithi096 ceph-mon[32753]: pgmap v1420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1361.700370 secs 2024-04-26T12:13:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1361.700370 secs 2024-04-26T12:13:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1361.700370 secs 2024-04-26T12:13:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:44 smithi194 ceph-mon[33017]: pgmap v1421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1364.416701 secs 2024-04-26T12:13:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:44 smithi033 ceph-mon[27845]: pgmap v1421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1364.416701 secs 2024-04-26T12:13:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:44 smithi096 ceph-mon[32753]: pgmap v1421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1364.416701 secs 2024-04-26T12:13:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:46 smithi194 ceph-mon[33017]: pgmap v1422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:46 smithi033 ceph-mon[27845]: pgmap v1422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:46 smithi096 ceph-mon[32753]: pgmap v1422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:48.593+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:48.594+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:13:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:13:48.594+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:13:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:48 smithi194 ceph-mon[33017]: pgmap v1423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1366.700453 secs 2024-04-26T12:13:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:48 smithi033 ceph-mon[27845]: pgmap v1423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1366.700453 secs 2024-04-26T12:13:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:48 smithi096 ceph-mon[32753]: pgmap v1423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1366.700453 secs 2024-04-26T12:13:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1369.416844 secs 2024-04-26T12:13:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1369.416844 secs 2024-04-26T12:13:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1369.416844 secs 2024-04-26T12:13:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:50 smithi194 ceph-mon[33017]: pgmap v1424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:50 smithi033 ceph-mon[27845]: pgmap v1424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:50 smithi096 ceph-mon[32753]: pgmap v1424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:51 smithi033 ceph-mon[27845]: pgmap v1425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:51 smithi096 ceph-mon[32753]: pgmap v1425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:51 smithi194 ceph-mon[33017]: pgmap v1425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:13:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1371.700533 secs 2024-04-26T12:13:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:53 smithi194 ceph-mon[33017]: pgmap v1426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1374.416958 secs 2024-04-26T12:13:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1371.700533 secs 2024-04-26T12:13:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:53 smithi033 ceph-mon[27845]: pgmap v1426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1374.416958 secs 2024-04-26T12:13:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1371.700533 secs 2024-04-26T12:13:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:53 smithi096 ceph-mon[32753]: pgmap v1426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1374.416958 secs 2024-04-26T12:13:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:55 smithi033 ceph-mon[27845]: pgmap v1427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:55 smithi096 ceph-mon[32753]: pgmap v1427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:55 smithi194 ceph-mon[33017]: pgmap v1427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:13:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:57 smithi033 ceph-mon[27845]: pgmap v1428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:57 smithi096 ceph-mon[32753]: pgmap v1428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:57 smithi194 ceph-mon[33017]: pgmap v1428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:13:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1376.700619 secs 2024-04-26T12:13:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1379.417082 secs 2024-04-26T12:13:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1376.700619 secs 2024-04-26T12:13:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1379.417082 secs 2024-04-26T12:13:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1376.700619 secs 2024-04-26T12:13:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1379.417082 secs 2024-04-26T12:13:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:13:59 smithi033 ceph-mon[27845]: pgmap v1429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:13:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:13:59 smithi096 ceph-mon[32753]: pgmap v1429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:13:59 smithi194 ceph-mon[33017]: pgmap v1429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:01 smithi033 ceph-mon[27845]: pgmap v1430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:01 smithi096 ceph-mon[32753]: pgmap v1430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:01 smithi194 ceph-mon[33017]: pgmap v1430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:03.598+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:03.598+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:03.598+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:14:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1381.700697 secs 2024-04-26T12:14:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:03 smithi033 ceph-mon[27845]: pgmap v1431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1384.417211 secs 2024-04-26T12:14:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1381.700697 secs 2024-04-26T12:14:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:03 smithi096 ceph-mon[32753]: pgmap v1431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1384.417211 secs 2024-04-26T12:14:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1381.700697 secs 2024-04-26T12:14:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:03 smithi194 ceph-mon[33017]: pgmap v1431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1384.417211 secs 2024-04-26T12:14:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:05 smithi033 ceph-mon[27845]: pgmap v1432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:05 smithi096 ceph-mon[32753]: pgmap v1432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:05 smithi194 ceph-mon[33017]: pgmap v1432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:07 smithi033 ceph-mon[27845]: pgmap v1433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:14:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:07 smithi096 ceph-mon[32753]: pgmap v1433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:14:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:07 smithi194 ceph-mon[33017]: pgmap v1433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:14:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1386.700782 secs 2024-04-26T12:14:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1389.417273 secs 2024-04-26T12:14:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1386.700782 secs 2024-04-26T12:14:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1389.417273 secs 2024-04-26T12:14:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1386.700782 secs 2024-04-26T12:14:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1389.417273 secs 2024-04-26T12:14:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:09 smithi033 ceph-mon[27845]: pgmap v1434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:09 smithi096 ceph-mon[32753]: pgmap v1434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:09 smithi194 ceph-mon[33017]: pgmap v1434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:11 smithi033 ceph-mon[27845]: pgmap v1435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:14:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:11 smithi096 ceph-mon[32753]: pgmap v1435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:14:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:11 smithi194 ceph-mon[33017]: pgmap v1435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:14:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:14:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:14:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:12 smithi096 ceph-mon[32753]: pgmap v1436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:12 smithi096 ceph-mon[32753]: pgmap v1437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:14:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1391.700868 secs 2024-04-26T12:14:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:14:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:14:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:12 smithi194 ceph-mon[33017]: pgmap v1436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:12 smithi194 ceph-mon[33017]: pgmap v1437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:14:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1391.700868 secs 2024-04-26T12:14:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:14:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:14:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:12 smithi033 ceph-mon[27845]: pgmap v1436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:12 smithi033 ceph-mon[27845]: pgmap v1437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:14:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1391.700868 secs 2024-04-26T12:14:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1394.417404 secs 2024-04-26T12:14:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1394.417404 secs 2024-04-26T12:14:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1394.417404 secs 2024-04-26T12:14:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:14 smithi194 ceph-mon[33017]: pgmap v1438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:14:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:14 smithi033 ceph-mon[27845]: pgmap v1438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:14:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:14 smithi096 ceph-mon[32753]: pgmap v1438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:14:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:17 smithi194 ceph-mon[33017]: pgmap v1439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:17 smithi033 ceph-mon[27845]: pgmap v1439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:17 smithi096 ceph-mon[32753]: pgmap v1439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1396.700951 secs 2024-04-26T12:14:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:18.610+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:18.610+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:14:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:18.610+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1396.700951 secs 2024-04-26T12:14:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1396.700951 secs 2024-04-26T12:14:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1399.417519 secs 2024-04-26T12:14:19.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:19 smithi194 ceph-mon[33017]: pgmap v1440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1399.417519 secs 2024-04-26T12:14:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:19 smithi033 ceph-mon[27845]: pgmap v1440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1399.417519 secs 2024-04-26T12:14:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:19 smithi096 ceph-mon[32753]: pgmap v1440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:21 smithi194 ceph-mon[33017]: pgmap v1441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:21 smithi033 ceph-mon[27845]: pgmap v1441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:21 smithi096 ceph-mon[32753]: pgmap v1441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:14:22.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.034+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:14:22.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.034+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:14:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.035+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:14:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.035+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:14:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.035+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:14:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.036+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0070 con 0x564385821900 2024-04-26T12:14:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.036+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T12:14:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:22.036+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:14:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:23 smithi194 ceph-mon[33017]: pgmap v1442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:14:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1401.701015 secs 2024-04-26T12:14:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:23 smithi033 ceph-mon[27845]: pgmap v1442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:14:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1401.701015 secs 2024-04-26T12:14:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:23 smithi096 ceph-mon[32753]: pgmap v1442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:14:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1401.701015 secs 2024-04-26T12:14:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1404.417642 secs 2024-04-26T12:14:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1404.417642 secs 2024-04-26T12:14:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1404.417642 secs 2024-04-26T12:14:25.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:25 smithi096 ceph-mon[32753]: pgmap v1443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:25 smithi194 ceph-mon[33017]: pgmap v1443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:25 smithi033 ceph-mon[27845]: pgmap v1443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:27 smithi194 ceph-mon[33017]: pgmap v1444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:27 smithi033 ceph-mon[27845]: pgmap v1444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:27 smithi096 ceph-mon[32753]: pgmap v1444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1406.701076 secs 2024-04-26T12:14:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1406.701076 secs 2024-04-26T12:14:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1406.701076 secs 2024-04-26T12:14:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1409.417723 secs 2024-04-26T12:14:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:29 smithi194 ceph-mon[33017]: pgmap v1445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1409.417723 secs 2024-04-26T12:14:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:29 smithi033 ceph-mon[27845]: pgmap v1445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1409.417723 secs 2024-04-26T12:14:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:29 smithi096 ceph-mon[32753]: pgmap v1445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:31 smithi194 ceph-mon[33017]: pgmap v1446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:31 smithi033 ceph-mon[27845]: pgmap v1446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:31.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:31 smithi096 ceph-mon[32753]: pgmap v1446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:33 smithi194 ceph-mon[33017]: pgmap v1447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1411.701161 secs 2024-04-26T12:14:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:33.626+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:33.626+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:14:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:33.626+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:33 smithi033 ceph-mon[27845]: pgmap v1447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1411.701161 secs 2024-04-26T12:14:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:33 smithi096 ceph-mon[32753]: pgmap v1447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1411.701161 secs 2024-04-26T12:14:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1414.417850 secs 2024-04-26T12:14:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1414.417850 secs 2024-04-26T12:14:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1414.417850 secs 2024-04-26T12:14:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:35 smithi194 ceph-mon[33017]: pgmap v1448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:35 smithi033 ceph-mon[27845]: pgmap v1448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:35 smithi096 ceph-mon[32753]: pgmap v1448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:37 smithi194 ceph-mon[33017]: pgmap v1449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:37 smithi033 ceph-mon[27845]: pgmap v1449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:37 smithi096 ceph-mon[32753]: pgmap v1449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1416.701222 secs 2024-04-26T12:14:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1416.701222 secs 2024-04-26T12:14:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1416.701222 secs 2024-04-26T12:14:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1419.417968 secs 2024-04-26T12:14:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:39 smithi194 ceph-mon[33017]: pgmap v1450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1419.417968 secs 2024-04-26T12:14:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:39 smithi033 ceph-mon[27845]: pgmap v1450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1419.417968 secs 2024-04-26T12:14:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:39 smithi096 ceph-mon[32753]: pgmap v1450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:41 smithi194 ceph-mon[33017]: pgmap v1451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:41 smithi033 ceph-mon[27845]: pgmap v1451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:41.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:41 smithi096 ceph-mon[32753]: pgmap v1451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:43 smithi194 ceph-mon[33017]: pgmap v1452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1421.701308 secs 2024-04-26T12:14:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:43 smithi033 ceph-mon[27845]: pgmap v1452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1421.701308 secs 2024-04-26T12:14:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:43 smithi096 ceph-mon[32753]: pgmap v1452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1421.701308 secs 2024-04-26T12:14:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1424.418131 secs 2024-04-26T12:14:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1424.418131 secs 2024-04-26T12:14:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1424.418131 secs 2024-04-26T12:14:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:45 smithi194 ceph-mon[33017]: pgmap v1453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:45 smithi033 ceph-mon[27845]: pgmap v1453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:45 smithi096 ceph-mon[32753]: pgmap v1453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:47 smithi194 ceph-mon[33017]: pgmap v1454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:47 smithi033 ceph-mon[27845]: pgmap v1454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:47 smithi096 ceph-mon[32753]: pgmap v1454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:48.641+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:48.641+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:14:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:14:48.641+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:14:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1426.701394 secs 2024-04-26T12:14:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1429.418255 secs 2024-04-26T12:14:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:48 smithi033 ceph-mon[27845]: pgmap v1455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1426.701394 secs 2024-04-26T12:14:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1429.418255 secs 2024-04-26T12:14:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:48 smithi096 ceph-mon[32753]: pgmap v1455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1426.701394 secs 2024-04-26T12:14:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1429.418255 secs 2024-04-26T12:14:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:48 smithi194 ceph-mon[33017]: pgmap v1455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:51.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:51 smithi194 ceph-mon[33017]: pgmap v1456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:51 smithi033 ceph-mon[27845]: pgmap v1456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:51 smithi096 ceph-mon[32753]: pgmap v1456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:53.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:53 smithi033 ceph-mon[27845]: pgmap v1457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:53.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1431.701455 secs 2024-04-26T12:14:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:53 smithi194 ceph-mon[33017]: pgmap v1457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1431.701455 secs 2024-04-26T12:14:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:53 smithi096 ceph-mon[32753]: pgmap v1457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1431.701455 secs 2024-04-26T12:14:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1434.418377 secs 2024-04-26T12:14:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1434.418377 secs 2024-04-26T12:14:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1434.418377 secs 2024-04-26T12:14:55.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:55 smithi096 ceph-mon[32753]: pgmap v1458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:55 smithi194 ceph-mon[33017]: pgmap v1458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:55 smithi033 ceph-mon[27845]: pgmap v1458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:57 smithi194 ceph-mon[33017]: pgmap v1459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:57 smithi033 ceph-mon[27845]: pgmap v1459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:57 smithi096 ceph-mon[32753]: pgmap v1459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:14:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1436.701539 secs 2024-04-26T12:14:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1436.701539 secs 2024-04-26T12:14:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1436.701539 secs 2024-04-26T12:14:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1439.418509 secs 2024-04-26T12:14:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:14:59 smithi194 ceph-mon[33017]: pgmap v1460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1439.418509 secs 2024-04-26T12:14:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:14:59 smithi033 ceph-mon[27845]: pgmap v1460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:14:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1439.418509 secs 2024-04-26T12:14:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:14:59 smithi096 ceph-mon[32753]: pgmap v1460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:01 smithi194 ceph-mon[33017]: pgmap v1461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:01 smithi033 ceph-mon[27845]: pgmap v1461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:01 smithi096 ceph-mon[32753]: pgmap v1461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:03 smithi194 ceph-mon[33017]: pgmap v1462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 1441.701623 secs 2024-04-26T12:15:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:03 smithi194 ceph-mon[33017]: slow request 1441.701623 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:15:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:03.644+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:03.644+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:15:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:03.644+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:03 smithi033 ceph-mon[27845]: pgmap v1462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 1441.701623 secs 2024-04-26T12:15:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:03 smithi033 ceph-mon[27845]: slow request 1441.701623 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:15:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:03 smithi096 ceph-mon[32753]: pgmap v1462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 1441.701623 secs 2024-04-26T12:15:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:03 smithi096 ceph-mon[32753]: slow request 1441.701623 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:15:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 1444.418592 secs 2024-04-26T12:15:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:04 smithi194 ceph-mon[33017]: slow request 1444.418591 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:15:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 1444.418592 secs 2024-04-26T12:15:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:04 smithi033 ceph-mon[27845]: slow request 1444.418591 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:15:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 1444.418592 secs 2024-04-26T12:15:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:04 smithi096 ceph-mon[32753]: slow request 1444.418591 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:15:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:05 smithi194 ceph-mon[33017]: pgmap v1463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:05 smithi033 ceph-mon[27845]: pgmap v1463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:05 smithi096 ceph-mon[32753]: pgmap v1463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:07 smithi194 ceph-mon[33017]: pgmap v1464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:07 smithi033 ceph-mon[27845]: pgmap v1464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:07 smithi096 ceph-mon[32753]: pgmap v1464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1446.701706 secs 2024-04-26T12:15:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1446.701706 secs 2024-04-26T12:15:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1446.701706 secs 2024-04-26T12:15:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1449.418677 secs 2024-04-26T12:15:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:09 smithi194 ceph-mon[33017]: pgmap v1465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1449.418677 secs 2024-04-26T12:15:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:09 smithi033 ceph-mon[27845]: pgmap v1465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1449.418677 secs 2024-04-26T12:15:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:09 smithi096 ceph-mon[32753]: pgmap v1465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:11 smithi194 ceph-mon[33017]: pgmap v1466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:11 smithi033 ceph-mon[27845]: pgmap v1466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:11 smithi096 ceph-mon[32753]: pgmap v1466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:13 smithi194 ceph-mon[33017]: pgmap v1467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1451.701792 secs 2024-04-26T12:15:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:15:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:13 smithi033 ceph-mon[27845]: pgmap v1467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1451.701792 secs 2024-04-26T12:15:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:15:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:13 smithi096 ceph-mon[32753]: pgmap v1467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1451.701792 secs 2024-04-26T12:15:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:15:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1454.418774 secs 2024-04-26T12:15:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:15:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:15:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:15:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1454.418774 secs 2024-04-26T12:15:14.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:15:14.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:15:14.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:15:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1454.418774 secs 2024-04-26T12:15:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:15:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:15:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:15:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:15 smithi194 ceph-mon[33017]: pgmap v1468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:15 smithi194 ceph-mon[33017]: pgmap v1469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:15 smithi033 ceph-mon[27845]: pgmap v1468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:15 smithi033 ceph-mon[27845]: pgmap v1469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:15 smithi096 ceph-mon[32753]: pgmap v1468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:15 smithi096 ceph-mon[32753]: pgmap v1469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:17 smithi194 ceph-mon[33017]: pgmap v1470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:17 smithi033 ceph-mon[27845]: pgmap v1470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:17 smithi096 ceph-mon[32753]: pgmap v1470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1456.701876 secs 2024-04-26T12:15:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1459.418898 secs 2024-04-26T12:15:18.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:18.650+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:18.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:18.650+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:18.650+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:15:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1456.701876 secs 2024-04-26T12:15:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1459.418898 secs 2024-04-26T12:15:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1456.701876 secs 2024-04-26T12:15:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1459.418898 secs 2024-04-26T12:15:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:19 smithi194 ceph-mon[33017]: pgmap v1471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:19 smithi033 ceph-mon[27845]: pgmap v1471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:19 smithi096 ceph-mon[32753]: pgmap v1471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:20 smithi033 ceph-mon[27845]: pgmap v1472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:20 smithi096 ceph-mon[32753]: pgmap v1472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:20 smithi194 ceph-mon[33017]: pgmap v1472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:22 smithi033 ceph-mon[27845]: pgmap v1473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:23.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1461.701957 secs 2024-04-26T12:15:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:22 smithi096 ceph-mon[32753]: pgmap v1473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:23.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1461.701957 secs 2024-04-26T12:15:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:22 smithi194 ceph-mon[33017]: pgmap v1473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:15:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1461.701957 secs 2024-04-26T12:15:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1464.419017 secs 2024-04-26T12:15:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1464.419017 secs 2024-04-26T12:15:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1464.419017 secs 2024-04-26T12:15:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:24 smithi033 ceph-mon[27845]: pgmap v1474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:15:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:24 smithi096 ceph-mon[32753]: pgmap v1474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:15:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:24 smithi194 ceph-mon[33017]: pgmap v1474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:15:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:26 smithi033 ceph-mon[27845]: pgmap v1475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:26 smithi096 ceph-mon[32753]: pgmap v1475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:26 smithi194 ceph-mon[33017]: pgmap v1475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1466.702020 secs 2024-04-26T12:15:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1466.702020 secs 2024-04-26T12:15:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1466.702020 secs 2024-04-26T12:15:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:28 smithi033 ceph-mon[27845]: pgmap v1476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1469.419169 secs 2024-04-26T12:15:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:28 smithi096 ceph-mon[32753]: pgmap v1476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1469.419169 secs 2024-04-26T12:15:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:28 smithi194 ceph-mon[33017]: pgmap v1476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1469.419169 secs 2024-04-26T12:15:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:30 smithi033 ceph-mon[27845]: pgmap v1477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:30 smithi096 ceph-mon[32753]: pgmap v1477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:30 smithi194 ceph-mon[33017]: pgmap v1477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:32 smithi096 ceph-mon[32753]: pgmap v1478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1471.702081 secs 2024-04-26T12:15:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:32 smithi194 ceph-mon[33017]: pgmap v1478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1471.702081 secs 2024-04-26T12:15:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:32 smithi033 ceph-mon[27845]: pgmap v1478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1471.702081 secs 2024-04-26T12:15:33.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:33.655+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:33.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:33.655+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:33.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:33.655+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:15:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1474.419294 secs 2024-04-26T12:15:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1474.419294 secs 2024-04-26T12:15:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1474.419294 secs 2024-04-26T12:15:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:34 smithi194 ceph-mon[33017]: pgmap v1479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:34 smithi033 ceph-mon[27845]: pgmap v1479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:34 smithi096 ceph-mon[32753]: pgmap v1479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:36 smithi194 ceph-mon[33017]: pgmap v1480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:36 smithi033 ceph-mon[27845]: pgmap v1480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:36 smithi096 ceph-mon[32753]: pgmap v1480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1476.702144 secs 2024-04-26T12:15:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1476.702144 secs 2024-04-26T12:15:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1476.702144 secs 2024-04-26T12:15:39.314 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:38 smithi033 ceph-mon[27845]: pgmap v1481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:39.314 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1479.419422 secs 2024-04-26T12:15:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:38 smithi194 ceph-mon[33017]: pgmap v1481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1479.419422 secs 2024-04-26T12:15:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:38 smithi096 ceph-mon[32753]: pgmap v1481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1479.419422 secs 2024-04-26T12:15:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:40 smithi194 ceph-mon[33017]: pgmap v1482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:40 smithi033 ceph-mon[27845]: pgmap v1482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:40 smithi096 ceph-mon[32753]: pgmap v1482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:42 smithi194 ceph-mon[33017]: pgmap v1483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1481.702203 secs 2024-04-26T12:15:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:42 smithi033 ceph-mon[27845]: pgmap v1483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1481.702203 secs 2024-04-26T12:15:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:42 smithi096 ceph-mon[32753]: pgmap v1483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1481.702203 secs 2024-04-26T12:15:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1484.419503 secs 2024-04-26T12:15:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1484.419503 secs 2024-04-26T12:15:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1484.419503 secs 2024-04-26T12:15:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:44 smithi194 ceph-mon[33017]: pgmap v1484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:44 smithi033 ceph-mon[27845]: pgmap v1484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:44 smithi096 ceph-mon[32753]: pgmap v1484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:46 smithi194 ceph-mon[33017]: pgmap v1485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:46 smithi033 ceph-mon[27845]: pgmap v1485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:46 smithi096 ceph-mon[32753]: pgmap v1485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1486.702265 secs 2024-04-26T12:15:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1486.702265 secs 2024-04-26T12:15:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1486.702265 secs 2024-04-26T12:15:48.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:48.672+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:48.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:48.672+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:15:48.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:15:48.672+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:15:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:48 smithi194 ceph-mon[33017]: pgmap v1486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1489.419588 secs 2024-04-26T12:15:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:48 smithi033 ceph-mon[27845]: pgmap v1486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1489.419588 secs 2024-04-26T12:15:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:48 smithi096 ceph-mon[32753]: pgmap v1486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1489.419588 secs 2024-04-26T12:15:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:51 smithi194 ceph-mon[33017]: pgmap v1487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:51 smithi033 ceph-mon[27845]: pgmap v1487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:51 smithi096 ceph-mon[32753]: pgmap v1487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:53 smithi194 ceph-mon[33017]: pgmap v1488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1491.702347 secs 2024-04-26T12:15:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:53 smithi033 ceph-mon[27845]: pgmap v1488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1491.702347 secs 2024-04-26T12:15:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:53 smithi096 ceph-mon[32753]: pgmap v1488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1491.702347 secs 2024-04-26T12:15:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1494.419733 secs 2024-04-26T12:15:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1494.419733 secs 2024-04-26T12:15:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1494.419733 secs 2024-04-26T12:15:55.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:55 smithi096 ceph-mon[32753]: pgmap v1489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:55 smithi194 ceph-mon[33017]: pgmap v1489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:55 smithi033 ceph-mon[27845]: pgmap v1489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:57 smithi194 ceph-mon[33017]: pgmap v1490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:57 smithi033 ceph-mon[27845]: pgmap v1490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:57 smithi096 ceph-mon[32753]: pgmap v1490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:15:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1496.702430 secs 2024-04-26T12:15:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1496.702430 secs 2024-04-26T12:15:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1496.702430 secs 2024-04-26T12:15:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:59 smithi194 ceph-mon[33017]: pgmap v1491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:59 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 1499.419858 secs 2024-04-26T12:15:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:15:59 smithi194 ceph-mon[33017]: slow request 1441.147646 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:15:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:59 smithi033 ceph-mon[27845]: pgmap v1491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:59 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 1499.419858 secs 2024-04-26T12:15:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:15:59 smithi033 ceph-mon[27845]: slow request 1441.147646 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:15:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:59 smithi096 ceph-mon[32753]: pgmap v1491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:15:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:59 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 1499.419858 secs 2024-04-26T12:15:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:15:59 smithi096 ceph-mon[32753]: slow request 1441.147646 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:16:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:01 smithi194 ceph-mon[33017]: pgmap v1492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:01 smithi033 ceph-mon[27845]: pgmap v1492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:01 smithi096 ceph-mon[32753]: pgmap v1492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:03 smithi194 ceph-mon[33017]: pgmap v1493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1501.702513 secs 2024-04-26T12:16:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:03 smithi033 ceph-mon[27845]: pgmap v1493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1501.702513 secs 2024-04-26T12:16:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:03 smithi096 ceph-mon[32753]: pgmap v1493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1501.702513 secs 2024-04-26T12:16:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:03.680+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:03.680+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:16:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:03.680+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1504.419998 secs 2024-04-26T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1504.419998 secs 2024-04-26T12:16:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1504.419998 secs 2024-04-26T12:16:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:05 smithi194 ceph-mon[33017]: pgmap v1494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:05 smithi033 ceph-mon[27845]: pgmap v1494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:05 smithi096 ceph-mon[32753]: pgmap v1494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:07 smithi194 ceph-mon[33017]: pgmap v1495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:16:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:07 smithi033 ceph-mon[27845]: pgmap v1495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:16:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:07 smithi096 ceph-mon[32753]: pgmap v1495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:16:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1506.702599 secs 2024-04-26T12:16:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1506.702599 secs 2024-04-26T12:16:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1506.702599 secs 2024-04-26T12:16:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:09 smithi194 ceph-mon[33017]: pgmap v1496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1509.420149 secs 2024-04-26T12:16:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:09 smithi033 ceph-mon[27845]: pgmap v1496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1509.420149 secs 2024-04-26T12:16:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:09 smithi096 ceph-mon[32753]: pgmap v1496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1509.420149 secs 2024-04-26T12:16:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:11 smithi194 ceph-mon[33017]: pgmap v1497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:16:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:11 smithi033 ceph-mon[27845]: pgmap v1497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:16:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:11 smithi096 ceph-mon[32753]: pgmap v1497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:13 smithi194 ceph-mon[33017]: pgmap v1498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1511.702678 secs 2024-04-26T12:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:13 smithi033 ceph-mon[27845]: pgmap v1498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1511.702678 secs 2024-04-26T12:16:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:13 smithi096 ceph-mon[32753]: pgmap v1498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1511.702678 secs 2024-04-26T12:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1514.420281 secs 2024-04-26T12:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1514.420281 secs 2024-04-26T12:16:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1514.420281 secs 2024-04-26T12:16:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:15 smithi194 ceph-mon[33017]: pgmap v1499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:16:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:15 smithi033 ceph-mon[27845]: pgmap v1499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:16:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:15 smithi096 ceph-mon[32753]: pgmap v1499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:16:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:16:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:16:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:16:16.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:16:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:16:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:16:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:16:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:16:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:16:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:16:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:17 smithi033 ceph-mon[27845]: pgmap v1500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T12:16:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:17 smithi033 ceph-mon[27845]: pgmap v1501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:17 smithi096 ceph-mon[32753]: pgmap v1500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T12:16:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:17 smithi096 ceph-mon[32753]: pgmap v1501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:17 smithi194 ceph-mon[33017]: pgmap v1500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T12:16:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:17 smithi194 ceph-mon[33017]: pgmap v1501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1516.702761 secs 2024-04-26T12:16:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1516.702761 secs 2024-04-26T12:16:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1516.702761 secs 2024-04-26T12:16:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:18.693+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:18.693+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:18.693+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:16:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:19 smithi033 ceph-mon[27845]: pgmap v1502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1519.420403 secs 2024-04-26T12:16:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:19 smithi096 ceph-mon[32753]: pgmap v1502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1519.420403 secs 2024-04-26T12:16:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:19 smithi194 ceph-mon[33017]: pgmap v1502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1519.420403 secs 2024-04-26T12:16:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:21 smithi033 ceph-mon[27845]: pgmap v1503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:21 smithi096 ceph-mon[32753]: pgmap v1503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:21 smithi194 ceph-mon[33017]: pgmap v1503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:23.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:23 smithi194 ceph-mon[33017]: pgmap v1504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:23.407 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1521.702846 secs 2024-04-26T12:16:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:23 smithi033 ceph-mon[27845]: pgmap v1504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1521.702846 secs 2024-04-26T12:16:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:23 smithi096 ceph-mon[32753]: pgmap v1504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1521.702846 secs 2024-04-26T12:16:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1524.420524 secs 2024-04-26T12:16:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1524.420524 secs 2024-04-26T12:16:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1524.420524 secs 2024-04-26T12:16:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:25 smithi033 ceph-mon[27845]: pgmap v1505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:25 smithi096 ceph-mon[32753]: pgmap v1505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:25 smithi194 ceph-mon[33017]: pgmap v1505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:16:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:27 smithi033 ceph-mon[27845]: pgmap v1506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:16:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:27 smithi096 ceph-mon[32753]: pgmap v1506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:16:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:27 smithi194 ceph-mon[33017]: pgmap v1506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:16:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1526.702928 secs 2024-04-26T12:16:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1526.702928 secs 2024-04-26T12:16:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1526.702928 secs 2024-04-26T12:16:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:29 smithi033 ceph-mon[27845]: pgmap v1507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1529.420653 secs 2024-04-26T12:16:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:29 smithi096 ceph-mon[32753]: pgmap v1507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1529.420653 secs 2024-04-26T12:16:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:29 smithi194 ceph-mon[33017]: pgmap v1507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1529.420653 secs 2024-04-26T12:16:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:31 smithi033 ceph-mon[27845]: pgmap v1508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:31 smithi096 ceph-mon[32753]: pgmap v1508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:31 smithi194 ceph-mon[33017]: pgmap v1508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:33 smithi096 ceph-mon[32753]: pgmap v1509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1531.703002 secs 2024-04-26T12:16:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:33 smithi194 ceph-mon[33017]: pgmap v1509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1531.703002 secs 2024-04-26T12:16:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:33 smithi033 ceph-mon[27845]: pgmap v1509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1531.703002 secs 2024-04-26T12:16:33.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:33.697+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:33.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:33.697+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:16:33.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:33.697+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1534.420797 secs 2024-04-26T12:16:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1534.420797 secs 2024-04-26T12:16:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1534.420797 secs 2024-04-26T12:16:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:35 smithi194 ceph-mon[33017]: pgmap v1510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:35 smithi033 ceph-mon[27845]: pgmap v1510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:35 smithi096 ceph-mon[32753]: pgmap v1510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:37 smithi194 ceph-mon[33017]: pgmap v1511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:37 smithi033 ceph-mon[27845]: pgmap v1511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:37 smithi096 ceph-mon[32753]: pgmap v1511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1536.703061 secs 2024-04-26T12:16:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1536.703061 secs 2024-04-26T12:16:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1536.703061 secs 2024-04-26T12:16:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:39 smithi194 ceph-mon[33017]: pgmap v1512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1539.420939 secs 2024-04-26T12:16:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:39 smithi033 ceph-mon[27845]: pgmap v1512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1539.420939 secs 2024-04-26T12:16:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:39 smithi096 ceph-mon[32753]: pgmap v1512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1539.420939 secs 2024-04-26T12:16:41.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:41 smithi033 ceph-mon[27845]: pgmap v1513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:41 smithi194 ceph-mon[33017]: pgmap v1513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:41.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:41 smithi096 ceph-mon[32753]: pgmap v1513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:43 smithi194 ceph-mon[33017]: pgmap v1514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1541.703120 secs 2024-04-26T12:16:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:43 smithi033 ceph-mon[27845]: pgmap v1514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1541.703120 secs 2024-04-26T12:16:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:43 smithi096 ceph-mon[32753]: pgmap v1514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1541.703120 secs 2024-04-26T12:16:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1544.421085 secs 2024-04-26T12:16:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1544.421085 secs 2024-04-26T12:16:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1544.421085 secs 2024-04-26T12:16:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:45 smithi194 ceph-mon[33017]: pgmap v1515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:45 smithi033 ceph-mon[27845]: pgmap v1515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:45 smithi096 ceph-mon[32753]: pgmap v1515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:47 smithi194 ceph-mon[33017]: pgmap v1516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:47 smithi033 ceph-mon[27845]: pgmap v1516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:47 smithi096 ceph-mon[32753]: pgmap v1516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1546.703211 secs 2024-04-26T12:16:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1546.703211 secs 2024-04-26T12:16:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1546.703211 secs 2024-04-26T12:16:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:48.704+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:48.704+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:16:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:16:48.704+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:16:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:49 smithi194 ceph-mon[33017]: pgmap v1517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1549.421186 secs 2024-04-26T12:16:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:49 smithi033 ceph-mon[27845]: pgmap v1517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1549.421186 secs 2024-04-26T12:16:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:49 smithi096 ceph-mon[32753]: pgmap v1517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1549.421186 secs 2024-04-26T12:16:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:51 smithi194 ceph-mon[33017]: pgmap v1518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:51 smithi033 ceph-mon[27845]: pgmap v1518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:51 smithi096 ceph-mon[32753]: pgmap v1518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:53 smithi194 ceph-mon[33017]: pgmap v1519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1551.703273 secs 2024-04-26T12:16:53.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:53 smithi033 ceph-mon[27845]: pgmap v1519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:53.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1551.703273 secs 2024-04-26T12:16:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:53 smithi096 ceph-mon[32753]: pgmap v1519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1551.703273 secs 2024-04-26T12:16:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1554.421250 secs 2024-04-26T12:16:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1554.421250 secs 2024-04-26T12:16:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1554.421250 secs 2024-04-26T12:16:55.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:55 smithi096 ceph-mon[32753]: pgmap v1520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:55 smithi194 ceph-mon[33017]: pgmap v1520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:55 smithi033 ceph-mon[27845]: pgmap v1520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:57 smithi033 ceph-mon[27845]: pgmap v1521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:57 smithi194 ceph-mon[33017]: pgmap v1521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:57 smithi096 ceph-mon[32753]: pgmap v1521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:16:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1556.703357 secs 2024-04-26T12:16:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1556.703357 secs 2024-04-26T12:16:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1556.703357 secs 2024-04-26T12:16:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:59 smithi194 ceph-mon[33017]: pgmap v1522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1559.421395 secs 2024-04-26T12:16:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:59 smithi033 ceph-mon[27845]: pgmap v1522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1559.421395 secs 2024-04-26T12:16:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:59 smithi096 ceph-mon[32753]: pgmap v1522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:16:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1559.421395 secs 2024-04-26T12:17:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:01 smithi194 ceph-mon[33017]: pgmap v1523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:01 smithi033 ceph-mon[27845]: pgmap v1523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:01 smithi096 ceph-mon[32753]: pgmap v1523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:03 smithi194 ceph-mon[33017]: pgmap v1524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1561.703440 secs 2024-04-26T12:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:03 smithi033 ceph-mon[27845]: pgmap v1524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1561.703440 secs 2024-04-26T12:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:03 smithi096 ceph-mon[32753]: pgmap v1524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1561.703440 secs 2024-04-26T12:17:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:03.714+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:03.715+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:03.715+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:17:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1564.421522 secs 2024-04-26T12:17:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1564.421522 secs 2024-04-26T12:17:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1564.421522 secs 2024-04-26T12:17:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:05 smithi194 ceph-mon[33017]: pgmap v1525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:05 smithi033 ceph-mon[27845]: pgmap v1525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:05 smithi096 ceph-mon[32753]: pgmap v1525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:07 smithi194 ceph-mon[33017]: pgmap v1526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:07 smithi033 ceph-mon[27845]: pgmap v1526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:07 smithi096 ceph-mon[32753]: pgmap v1526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1566.703521 secs 2024-04-26T12:17:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1566.703521 secs 2024-04-26T12:17:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1566.703521 secs 2024-04-26T12:17:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:09 smithi194 ceph-mon[33017]: pgmap v1527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1569.421647 secs 2024-04-26T12:17:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:09 smithi033 ceph-mon[27845]: pgmap v1527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1569.421647 secs 2024-04-26T12:17:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:09 smithi096 ceph-mon[32753]: pgmap v1527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1569.421647 secs 2024-04-26T12:17:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:11 smithi194 ceph-mon[33017]: pgmap v1528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:11 smithi033 ceph-mon[27845]: pgmap v1528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:11 smithi096 ceph-mon[32753]: pgmap v1528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:13 smithi033 ceph-mon[27845]: pgmap v1529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1571.703606 secs 2024-04-26T12:17:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1574.421716 secs 2024-04-26T12:17:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:13 smithi096 ceph-mon[32753]: pgmap v1529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1571.703606 secs 2024-04-26T12:17:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1574.421716 secs 2024-04-26T12:17:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:13 smithi194 ceph-mon[33017]: pgmap v1529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1571.703606 secs 2024-04-26T12:17:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1574.421716 secs 2024-04-26T12:17:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:15 smithi033 ceph-mon[27845]: pgmap v1530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:15 smithi096 ceph-mon[32753]: pgmap v1530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:15 smithi194 ceph-mon[33017]: pgmap v1530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:17:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:17:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:17:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:17 smithi033 ceph-mon[27845]: pgmap v1531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:17:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:17:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:17:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:17 smithi096 ceph-mon[32753]: pgmap v1531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:17:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:17:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:17:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:17 smithi194 ceph-mon[33017]: pgmap v1531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:17:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:17:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:17:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:18 smithi033 ceph-mon[27845]: pgmap v1532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:18 smithi033 ceph-mon[27845]: pgmap v1533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1576.703694 secs 2024-04-26T12:17:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1579.421839 secs 2024-04-26T12:17:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:18 smithi096 ceph-mon[32753]: pgmap v1532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:18 smithi096 ceph-mon[32753]: pgmap v1533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1576.703694 secs 2024-04-26T12:17:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1579.421839 secs 2024-04-26T12:17:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:18.728+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:18.728+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:17:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:18.728+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:18 smithi194 ceph-mon[33017]: pgmap v1532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:18 smithi194 ceph-mon[33017]: pgmap v1533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:18.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1576.703694 secs 2024-04-26T12:17:18.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1579.421839 secs 2024-04-26T12:17:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:19 smithi033 ceph-mon[27845]: pgmap v1534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:17:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:19 smithi096 ceph-mon[32753]: pgmap v1534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:17:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:19 smithi194 ceph-mon[33017]: pgmap v1534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:17:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:22.065+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:17:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:22.065+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:17:22.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:22.066+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:17:22.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:22.066+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:17:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:22 smithi194 ceph-mon[33017]: pgmap v1535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:22 smithi033 ceph-mon[27845]: pgmap v1535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:22 smithi096 ceph-mon[32753]: pgmap v1535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:23.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1581.703777 secs 2024-04-26T12:17:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1581.703777 secs 2024-04-26T12:17:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1581.703777 secs 2024-04-26T12:17:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1584.421973 secs 2024-04-26T12:17:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:24 smithi194 ceph-mon[33017]: pgmap v1536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1584.421973 secs 2024-04-26T12:17:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:24 smithi033 ceph-mon[27845]: pgmap v1536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1584.421973 secs 2024-04-26T12:17:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:24 smithi096 ceph-mon[32753]: pgmap v1536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:26 smithi194 ceph-mon[33017]: pgmap v1537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:26.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:26 smithi033 ceph-mon[27845]: pgmap v1537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:26 smithi096 ceph-mon[32753]: pgmap v1537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:28 smithi194 ceph-mon[33017]: pgmap v1538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:17:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1586.703851 secs 2024-04-26T12:17:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:28 smithi033 ceph-mon[27845]: pgmap v1538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:17:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1586.703851 secs 2024-04-26T12:17:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:28 smithi096 ceph-mon[32753]: pgmap v1538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:17:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1586.703851 secs 2024-04-26T12:17:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1589.422076 secs 2024-04-26T12:17:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1589.422076 secs 2024-04-26T12:17:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1589.422076 secs 2024-04-26T12:17:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:30 smithi194 ceph-mon[33017]: pgmap v1539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:30 smithi033 ceph-mon[27845]: pgmap v1539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:30 smithi096 ceph-mon[32753]: pgmap v1539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:32 smithi194 ceph-mon[33017]: pgmap v1540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:32 smithi033 ceph-mon[27845]: pgmap v1540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:32 smithi096 ceph-mon[32753]: pgmap v1540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1591.703936 secs 2024-04-26T12:17:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1591.703936 secs 2024-04-26T12:17:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1591.703936 secs 2024-04-26T12:17:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:33.729+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:33.729+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:17:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:33.729+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1594.422196 secs 2024-04-26T12:17:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:34 smithi194 ceph-mon[33017]: pgmap v1541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1594.422196 secs 2024-04-26T12:17:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:34 smithi033 ceph-mon[27845]: pgmap v1541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1594.422196 secs 2024-04-26T12:17:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:34 smithi096 ceph-mon[32753]: pgmap v1541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:36 smithi194 ceph-mon[33017]: pgmap v1542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:36.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:36 smithi033 ceph-mon[27845]: pgmap v1542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:36 smithi096 ceph-mon[32753]: pgmap v1542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:38 smithi194 ceph-mon[33017]: pgmap v1543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1596.704023 secs 2024-04-26T12:17:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:38 smithi033 ceph-mon[27845]: pgmap v1543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1596.704023 secs 2024-04-26T12:17:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:38 smithi096 ceph-mon[32753]: pgmap v1543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1596.704023 secs 2024-04-26T12:17:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1599.422260 secs 2024-04-26T12:17:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1599.422260 secs 2024-04-26T12:17:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1599.422260 secs 2024-04-26T12:17:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:40 smithi033 ceph-mon[27845]: pgmap v1544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:40 smithi096 ceph-mon[32753]: pgmap v1544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:40 smithi194 ceph-mon[33017]: pgmap v1544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:42 smithi033 ceph-mon[27845]: pgmap v1545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:42 smithi096 ceph-mon[32753]: pgmap v1545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:42 smithi194 ceph-mon[33017]: pgmap v1545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:43.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1601.704087 secs 2024-04-26T12:17:43.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1604.422340 secs 2024-04-26T12:17:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1601.704087 secs 2024-04-26T12:17:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1604.422340 secs 2024-04-26T12:17:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1601.704087 secs 2024-04-26T12:17:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1604.422340 secs 2024-04-26T12:17:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:44 smithi033 ceph-mon[27845]: pgmap v1546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:44 smithi096 ceph-mon[32753]: pgmap v1546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:44 smithi194 ceph-mon[33017]: pgmap v1546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:45 smithi033 ceph-mon[27845]: pgmap v1547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:45 smithi096 ceph-mon[32753]: pgmap v1547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:45 smithi194 ceph-mon[33017]: pgmap v1547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:48 smithi194 ceph-mon[33017]: pgmap v1548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1606.704174 secs 2024-04-26T12:17:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:48 smithi033 ceph-mon[27845]: pgmap v1548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1606.704174 secs 2024-04-26T12:17:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:48 smithi096 ceph-mon[32753]: pgmap v1548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1606.704174 secs 2024-04-26T12:17:48.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:48.733+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:48.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:48.733+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:17:48.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:17:48.733+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:17:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1609.422458 secs 2024-04-26T12:17:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1609.422458 secs 2024-04-26T12:17:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1609.422458 secs 2024-04-26T12:17:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:50 smithi194 ceph-mon[33017]: pgmap v1549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:50 smithi033 ceph-mon[27845]: pgmap v1549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:50 smithi096 ceph-mon[32753]: pgmap v1549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:17:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:52 smithi194 ceph-mon[33017]: pgmap v1550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:52 smithi033 ceph-mon[27845]: pgmap v1550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:52 smithi096 ceph-mon[32753]: pgmap v1550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:17:53.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1611.704261 secs 2024-04-26T12:17:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1611.704261 secs 2024-04-26T12:17:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1611.704261 secs 2024-04-26T12:17:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1614.422581 secs 2024-04-26T12:17:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:54 smithi194 ceph-mon[33017]: pgmap v1551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1614.422581 secs 2024-04-26T12:17:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:54 smithi033 ceph-mon[27845]: pgmap v1551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1614.422581 secs 2024-04-26T12:17:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:54 smithi096 ceph-mon[32753]: pgmap v1551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:56 smithi194 ceph-mon[33017]: pgmap v1552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:56 smithi033 ceph-mon[27845]: pgmap v1552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:56 smithi096 ceph-mon[32753]: pgmap v1552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:17:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:58 smithi194 ceph-mon[33017]: pgmap v1553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1616.704322 secs 2024-04-26T12:17:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:58 smithi033 ceph-mon[27845]: pgmap v1553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1616.704322 secs 2024-04-26T12:17:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:58 smithi096 ceph-mon[32753]: pgmap v1553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:17:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1616.704322 secs 2024-04-26T12:17:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1619.422720 secs 2024-04-26T12:17:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1619.422720 secs 2024-04-26T12:17:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1619.422720 secs 2024-04-26T12:18:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:00 smithi194 ceph-mon[33017]: pgmap v1554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:18:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:00 smithi033 ceph-mon[27845]: pgmap v1554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:18:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:00 smithi096 ceph-mon[32753]: pgmap v1554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:18:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:02 smithi033 ceph-mon[27845]: pgmap v1555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:02 smithi096 ceph-mon[32753]: pgmap v1555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:02 smithi194 ceph-mon[33017]: pgmap v1555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1621.704387 secs 2024-04-26T12:18:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1624.422848 secs 2024-04-26T12:18:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1621.704387 secs 2024-04-26T12:18:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1624.422848 secs 2024-04-26T12:18:03.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:03.746+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:03.746+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:03.746+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:18:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1621.704387 secs 2024-04-26T12:18:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1624.422848 secs 2024-04-26T12:18:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:04 smithi033 ceph-mon[27845]: pgmap v1556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:04 smithi096 ceph-mon[32753]: pgmap v1556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:04 smithi194 ceph-mon[33017]: pgmap v1556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:06 smithi033 ceph-mon[27845]: pgmap v1557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:06 smithi096 ceph-mon[32753]: pgmap v1557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:06 smithi194 ceph-mon[33017]: pgmap v1557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:08 smithi033 ceph-mon[27845]: pgmap v1558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:08.686 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1626.704469 secs 2024-04-26T12:18:08.686 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1629.423004 secs 2024-04-26T12:18:08.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:08 smithi096 ceph-mon[32753]: pgmap v1558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:08.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1626.704469 secs 2024-04-26T12:18:08.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1629.423004 secs 2024-04-26T12:18:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:08 smithi194 ceph-mon[33017]: pgmap v1558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1626.704469 secs 2024-04-26T12:18:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1629.423004 secs 2024-04-26T12:18:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:10 smithi033 ceph-mon[27845]: pgmap v1559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:18:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:10 smithi096 ceph-mon[32753]: pgmap v1559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:18:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:10 smithi194 ceph-mon[33017]: pgmap v1559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:18:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:11 smithi194 ceph-mon[33017]: pgmap v1560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:11 smithi033 ceph-mon[27845]: pgmap v1560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:11 smithi096 ceph-mon[32753]: pgmap v1560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1631.704556 secs 2024-04-26T12:18:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1631.704556 secs 2024-04-26T12:18:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1631.704556 secs 2024-04-26T12:18:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1634.423117 secs 2024-04-26T12:18:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:13 smithi194 ceph-mon[33017]: pgmap v1561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1634.423117 secs 2024-04-26T12:18:14.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:13 smithi033 ceph-mon[27845]: pgmap v1561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1634.423117 secs 2024-04-26T12:18:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:13 smithi096 ceph-mon[32753]: pgmap v1561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:16 smithi194 ceph-mon[33017]: pgmap v1562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:16 smithi033 ceph-mon[27845]: pgmap v1562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:16 smithi096 ceph-mon[32753]: pgmap v1562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:18 smithi194 ceph-mon[33017]: pgmap v1563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1636.704616 secs 2024-04-26T12:18:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:18:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:18 smithi033 ceph-mon[27845]: pgmap v1563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1636.704616 secs 2024-04-26T12:18:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:18:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:18 smithi096 ceph-mon[32753]: pgmap v1563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1636.704616 secs 2024-04-26T12:18:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:18:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:18.761+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:18.761+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:18.761+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:18:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1639.423204 secs 2024-04-26T12:18:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:18:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:18:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:18:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1639.423204 secs 2024-04-26T12:18:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:18:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:18:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:18:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1639.423204 secs 2024-04-26T12:18:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:18:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:18:19.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:18:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:20 smithi194 ceph-mon[33017]: pgmap v1564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:20 smithi194 ceph-mon[33017]: pgmap v1565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:18:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:20 smithi033 ceph-mon[27845]: pgmap v1564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:20 smithi033 ceph-mon[27845]: pgmap v1565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:18:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:20 smithi096 ceph-mon[32753]: pgmap v1564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:20 smithi096 ceph-mon[32753]: pgmap v1565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:18:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:22 smithi194 ceph-mon[33017]: pgmap v1566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:22 smithi033 ceph-mon[27845]: pgmap v1566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:22 smithi096 ceph-mon[32753]: pgmap v1566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1641.704703 secs 2024-04-26T12:18:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1641.704703 secs 2024-04-26T12:18:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1641.704703 secs 2024-04-26T12:18:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:24 smithi194 ceph-mon[33017]: pgmap v1567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1644.423348 secs 2024-04-26T12:18:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:24 smithi033 ceph-mon[27845]: pgmap v1567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1644.423348 secs 2024-04-26T12:18:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:24 smithi096 ceph-mon[32753]: pgmap v1567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1644.423348 secs 2024-04-26T12:18:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:26 smithi194 ceph-mon[33017]: pgmap v1568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:26.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:26 smithi033 ceph-mon[27845]: pgmap v1568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:26 smithi096 ceph-mon[32753]: pgmap v1568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:28 smithi033 ceph-mon[27845]: pgmap v1569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1646.704786 secs 2024-04-26T12:18:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:28 smithi096 ceph-mon[32753]: pgmap v1569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1646.704786 secs 2024-04-26T12:18:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:28 smithi194 ceph-mon[33017]: pgmap v1569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:18:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1646.704786 secs 2024-04-26T12:18:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1649.423495 secs 2024-04-26T12:18:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1649.423495 secs 2024-04-26T12:18:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1649.423495 secs 2024-04-26T12:18:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:30 smithi033 ceph-mon[27845]: pgmap v1570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:18:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:30 smithi096 ceph-mon[32753]: pgmap v1570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:18:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:30 smithi194 ceph-mon[33017]: pgmap v1570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:18:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:32 smithi033 ceph-mon[27845]: pgmap v1571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:32 smithi096 ceph-mon[32753]: pgmap v1571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:32 smithi194 ceph-mon[33017]: pgmap v1571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1651.704876 secs 2024-04-26T12:18:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1654.423577 secs 2024-04-26T12:18:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1651.704876 secs 2024-04-26T12:18:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1654.423577 secs 2024-04-26T12:18:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:33.771+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:33.771+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:33.771+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:18:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1651.704876 secs 2024-04-26T12:18:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1654.423577 secs 2024-04-26T12:18:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:34 smithi033 ceph-mon[27845]: pgmap v1572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:34 smithi096 ceph-mon[32753]: pgmap v1572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:34 smithi194 ceph-mon[33017]: pgmap v1572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:36 smithi033 ceph-mon[27845]: pgmap v1573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:36 smithi096 ceph-mon[32753]: pgmap v1573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:36 smithi194 ceph-mon[33017]: pgmap v1573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:37 smithi194 ceph-mon[33017]: pgmap v1574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:37.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:37 smithi033 ceph-mon[27845]: pgmap v1574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:37 smithi096 ceph-mon[32753]: pgmap v1574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1656.704972 secs 2024-04-26T12:18:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1659.423701 secs 2024-04-26T12:18:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1656.704972 secs 2024-04-26T12:18:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1659.423701 secs 2024-04-26T12:18:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1656.704972 secs 2024-04-26T12:18:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1659.423701 secs 2024-04-26T12:18:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:39 smithi194 ceph-mon[33017]: pgmap v1575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:39 smithi033 ceph-mon[27845]: pgmap v1575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:39 smithi096 ceph-mon[32753]: pgmap v1575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:41 smithi194 ceph-mon[33017]: pgmap v1576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:41 smithi033 ceph-mon[27845]: pgmap v1576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:41 smithi096 ceph-mon[32753]: pgmap v1576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1661.705078 secs 2024-04-26T12:18:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1661.705078 secs 2024-04-26T12:18:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1661.705078 secs 2024-04-26T12:18:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:43 smithi194 ceph-mon[33017]: pgmap v1577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1664.423809 secs 2024-04-26T12:18:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:43 smithi033 ceph-mon[27845]: pgmap v1577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1664.423809 secs 2024-04-26T12:18:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:43 smithi096 ceph-mon[32753]: pgmap v1577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1664.423809 secs 2024-04-26T12:18:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:45 smithi194 ceph-mon[33017]: pgmap v1578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:45 smithi033 ceph-mon[27845]: pgmap v1578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:45 smithi096 ceph-mon[32753]: pgmap v1578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:47 smithi194 ceph-mon[33017]: pgmap v1579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1666.705142 secs 2024-04-26T12:18:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:47 smithi033 ceph-mon[27845]: pgmap v1579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1666.705142 secs 2024-04-26T12:18:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:47 smithi096 ceph-mon[32753]: pgmap v1579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1666.705142 secs 2024-04-26T12:18:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:48.787+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:48.787+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:18:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:18:48.787+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:18:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1669.423906 secs 2024-04-26T12:18:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1669.423906 secs 2024-04-26T12:18:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1669.423906 secs 2024-04-26T12:18:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:50 smithi194 ceph-mon[33017]: pgmap v1580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:50 smithi033 ceph-mon[27845]: pgmap v1580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:50 smithi096 ceph-mon[32753]: pgmap v1580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:52 smithi194 ceph-mon[33017]: pgmap v1581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:52 smithi033 ceph-mon[27845]: pgmap v1581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:52 smithi096 ceph-mon[32753]: pgmap v1581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:53.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1671.705229 secs 2024-04-26T12:18:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1671.705229 secs 2024-04-26T12:18:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1671.705229 secs 2024-04-26T12:18:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:54 smithi194 ceph-mon[33017]: pgmap v1582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1674.423999 secs 2024-04-26T12:18:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:54 smithi033 ceph-mon[27845]: pgmap v1582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1674.423999 secs 2024-04-26T12:18:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:54 smithi096 ceph-mon[32753]: pgmap v1582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1674.423999 secs 2024-04-26T12:18:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:56 smithi194 ceph-mon[33017]: pgmap v1583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:56 smithi033 ceph-mon[27845]: pgmap v1583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:56.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:56 smithi096 ceph-mon[32753]: pgmap v1583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:18:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:58 smithi194 ceph-mon[33017]: pgmap v1584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1676.705310 secs 2024-04-26T12:18:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:58 smithi033 ceph-mon[27845]: pgmap v1584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1676.705310 secs 2024-04-26T12:18:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:58 smithi096 ceph-mon[32753]: pgmap v1584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:18:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1676.705310 secs 2024-04-26T12:18:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:18:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1679.424135 secs 2024-04-26T12:18:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:18:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1679.424135 secs 2024-04-26T12:18:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:18:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1679.424135 secs 2024-04-26T12:19:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:00 smithi194 ceph-mon[33017]: pgmap v1585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:00 smithi033 ceph-mon[27845]: pgmap v1585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:00 smithi096 ceph-mon[32753]: pgmap v1585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:02 smithi194 ceph-mon[33017]: pgmap v1586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:02 smithi033 ceph-mon[27845]: pgmap v1586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:02 smithi096 ceph-mon[32753]: pgmap v1586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1681.705397 secs 2024-04-26T12:19:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1681.705397 secs 2024-04-26T12:19:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1681.705397 secs 2024-04-26T12:19:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:03.800+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:03.800+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:03.800+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:19:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:04 smithi194 ceph-mon[33017]: pgmap v1587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1684.424178 secs 2024-04-26T12:19:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:04 smithi033 ceph-mon[27845]: pgmap v1587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:04.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1684.424178 secs 2024-04-26T12:19:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:04 smithi096 ceph-mon[32753]: pgmap v1587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1684.424178 secs 2024-04-26T12:19:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:06 smithi194 ceph-mon[33017]: pgmap v1588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:06 smithi033 ceph-mon[27845]: pgmap v1588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:06 smithi096 ceph-mon[32753]: pgmap v1588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:08 smithi194 ceph-mon[33017]: pgmap v1589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:08.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1686.705479 secs 2024-04-26T12:19:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:08 smithi033 ceph-mon[27845]: pgmap v1589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1686.705479 secs 2024-04-26T12:19:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:08 smithi096 ceph-mon[32753]: pgmap v1589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1686.705479 secs 2024-04-26T12:19:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1689.424296 secs 2024-04-26T12:19:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1689.424296 secs 2024-04-26T12:19:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1689.424296 secs 2024-04-26T12:19:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:10 smithi194 ceph-mon[33017]: pgmap v1590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:10 smithi033 ceph-mon[27845]: pgmap v1590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:10 smithi096 ceph-mon[32753]: pgmap v1590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:12 smithi194 ceph-mon[33017]: pgmap v1591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:12 smithi033 ceph-mon[27845]: pgmap v1591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:12 smithi096 ceph-mon[32753]: pgmap v1591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1691.705562 secs 2024-04-26T12:19:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1691.705562 secs 2024-04-26T12:19:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1691.705562 secs 2024-04-26T12:19:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:14 smithi194 ceph-mon[33017]: pgmap v1592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1694.424415 secs 2024-04-26T12:19:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:14 smithi033 ceph-mon[27845]: pgmap v1592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1694.424415 secs 2024-04-26T12:19:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:14 smithi096 ceph-mon[32753]: pgmap v1592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1694.424415 secs 2024-04-26T12:19:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:16 smithi033 ceph-mon[27845]: pgmap v1593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:16 smithi096 ceph-mon[32753]: pgmap v1593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:16 smithi194 ceph-mon[33017]: pgmap v1593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:18 smithi033 ceph-mon[27845]: pgmap v1594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1696.705649 secs 2024-04-26T12:19:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:18 smithi096 ceph-mon[32753]: pgmap v1594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1696.705649 secs 2024-04-26T12:19:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:18 smithi194 ceph-mon[33017]: pgmap v1594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1696.705649 secs 2024-04-26T12:19:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:18.802+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:18.803+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:18.803+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:19:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1699.424533 secs 2024-04-26T12:19:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1699.424533 secs 2024-04-26T12:19:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1699.424533 secs 2024-04-26T12:19:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:20 smithi033 ceph-mon[27845]: pgmap v1595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:19:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:20 smithi096 ceph-mon[32753]: pgmap v1595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:19:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:20 smithi194 ceph-mon[33017]: pgmap v1595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:19:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:19:21.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:19:21.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:19:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:19:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:19:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:19:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:19:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:19:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:19:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:22 smithi033 ceph-mon[27845]: pgmap v1596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:19:22.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:22 smithi033 ceph-mon[27845]: pgmap v1597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:22 smithi096 ceph-mon[32753]: pgmap v1596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:19:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:22 smithi096 ceph-mon[32753]: pgmap v1597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:22 smithi194 ceph-mon[33017]: pgmap v1596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:19:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:22 smithi194 ceph-mon[33017]: pgmap v1597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1701.705733 secs 2024-04-26T12:19:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1701.705733 secs 2024-04-26T12:19:23.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1701.705733 secs 2024-04-26T12:19:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:24 smithi033 ceph-mon[27845]: pgmap v1598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1704.424649 secs 2024-04-26T12:19:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:24 smithi096 ceph-mon[32753]: pgmap v1598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1704.424649 secs 2024-04-26T12:19:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:24 smithi194 ceph-mon[33017]: pgmap v1598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1704.424649 secs 2024-04-26T12:19:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:26 smithi033 ceph-mon[27845]: pgmap v1599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:26 smithi096 ceph-mon[32753]: pgmap v1599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:26 smithi194 ceph-mon[33017]: pgmap v1599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:28 smithi033 ceph-mon[27845]: pgmap v1600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:28.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1706.705816 secs 2024-04-26T12:19:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:28 smithi096 ceph-mon[32753]: pgmap v1600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1706.705816 secs 2024-04-26T12:19:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:28 smithi194 ceph-mon[33017]: pgmap v1600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1706.705816 secs 2024-04-26T12:19:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1709.424766 secs 2024-04-26T12:19:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1709.424766 secs 2024-04-26T12:19:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1709.424766 secs 2024-04-26T12:19:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:30 smithi096 ceph-mon[32753]: pgmap v1601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:30 smithi194 ceph-mon[33017]: pgmap v1601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:30 smithi033 ceph-mon[27845]: pgmap v1601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:19:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:32 smithi194 ceph-mon[33017]: pgmap v1602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:19:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:32 smithi033 ceph-mon[27845]: pgmap v1602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:19:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:32 smithi096 ceph-mon[32753]: pgmap v1602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:19:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1711.705900 secs 2024-04-26T12:19:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1711.705900 secs 2024-04-26T12:19:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1711.705900 secs 2024-04-26T12:19:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:33.817+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:33.817+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:19:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:33.817+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:34 smithi194 ceph-mon[33017]: pgmap v1603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1714.424838 secs 2024-04-26T12:19:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:34 smithi033 ceph-mon[27845]: pgmap v1603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1714.424838 secs 2024-04-26T12:19:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:34 smithi096 ceph-mon[32753]: pgmap v1603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1714.424838 secs 2024-04-26T12:19:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:36 smithi194 ceph-mon[33017]: pgmap v1604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:36 smithi033 ceph-mon[27845]: pgmap v1604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:36 smithi096 ceph-mon[32753]: pgmap v1604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:38 smithi194 ceph-mon[33017]: pgmap v1605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1716.706001 secs 2024-04-26T12:19:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:38 smithi033 ceph-mon[27845]: pgmap v1605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1716.706001 secs 2024-04-26T12:19:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:38 smithi096 ceph-mon[32753]: pgmap v1605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1716.706001 secs 2024-04-26T12:19:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1719.424907 secs 2024-04-26T12:19:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1719.424907 secs 2024-04-26T12:19:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1719.424907 secs 2024-04-26T12:19:40.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:40 smithi194 ceph-mon[33017]: pgmap v1606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:40 smithi033 ceph-mon[27845]: pgmap v1606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:40 smithi096 ceph-mon[32753]: pgmap v1606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:42 smithi194 ceph-mon[33017]: pgmap v1607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:42 smithi033 ceph-mon[27845]: pgmap v1607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:42 smithi096 ceph-mon[32753]: pgmap v1607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1721.706092 secs 2024-04-26T12:19:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1721.706092 secs 2024-04-26T12:19:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1721.706092 secs 2024-04-26T12:19:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:44 smithi194 ceph-mon[33017]: pgmap v1608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1724.425021 secs 2024-04-26T12:19:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:44 smithi033 ceph-mon[27845]: pgmap v1608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1724.425021 secs 2024-04-26T12:19:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:44 smithi096 ceph-mon[32753]: pgmap v1608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1724.425021 secs 2024-04-26T12:19:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:46 smithi194 ceph-mon[33017]: pgmap v1609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:46 smithi033 ceph-mon[27845]: pgmap v1609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:46 smithi096 ceph-mon[32753]: pgmap v1609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:48 smithi033 ceph-mon[27845]: pgmap v1610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1726.706184 secs 2024-04-26T12:19:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:48 smithi194 ceph-mon[33017]: pgmap v1610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1726.706184 secs 2024-04-26T12:19:48.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:48 smithi096 ceph-mon[32753]: pgmap v1610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:48.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1726.706184 secs 2024-04-26T12:19:48.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:48.831+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:48.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:48.831+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:19:48.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:19:48.831+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:19:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1729.425119 secs 2024-04-26T12:19:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1729.425119 secs 2024-04-26T12:19:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1729.425119 secs 2024-04-26T12:19:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:50 smithi194 ceph-mon[33017]: pgmap v1611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:50 smithi033 ceph-mon[27845]: pgmap v1611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:50 smithi096 ceph-mon[32753]: pgmap v1611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:52 smithi194 ceph-mon[33017]: pgmap v1612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:52 smithi033 ceph-mon[27845]: pgmap v1612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:52 smithi096 ceph-mon[32753]: pgmap v1612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:53.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1731.706286 secs 2024-04-26T12:19:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1731.706286 secs 2024-04-26T12:19:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1731.706286 secs 2024-04-26T12:19:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:54 smithi194 ceph-mon[33017]: pgmap v1613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1734.425231 secs 2024-04-26T12:19:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:54 smithi033 ceph-mon[27845]: pgmap v1613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1734.425231 secs 2024-04-26T12:19:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:54 smithi096 ceph-mon[32753]: pgmap v1613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1734.425231 secs 2024-04-26T12:19:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:56 smithi194 ceph-mon[33017]: pgmap v1614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:56 smithi033 ceph-mon[27845]: pgmap v1614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:56 smithi096 ceph-mon[32753]: pgmap v1614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:19:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:58 smithi194 ceph-mon[33017]: pgmap v1615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1736.706390 secs 2024-04-26T12:19:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:58 smithi033 ceph-mon[27845]: pgmap v1615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1736.706390 secs 2024-04-26T12:19:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:58 smithi096 ceph-mon[32753]: pgmap v1615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:19:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1736.706390 secs 2024-04-26T12:19:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:19:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1739.425348 secs 2024-04-26T12:19:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:19:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1739.425348 secs 2024-04-26T12:19:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:19:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1739.425348 secs 2024-04-26T12:20:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:00 smithi194 ceph-mon[33017]: pgmap v1616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:20:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:00 smithi033 ceph-mon[27845]: pgmap v1616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:20:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:00 smithi096 ceph-mon[32753]: pgmap v1616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:20:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:02 smithi194 ceph-mon[33017]: pgmap v1617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:02 smithi033 ceph-mon[27845]: pgmap v1617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:02 smithi096 ceph-mon[32753]: pgmap v1617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1741.706492 secs 2024-04-26T12:20:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1741.706492 secs 2024-04-26T12:20:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1741.706492 secs 2024-04-26T12:20:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:03.846+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:03.846+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:03.846+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:20:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:04 smithi194 ceph-mon[33017]: pgmap v1618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:04.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1744.425464 secs 2024-04-26T12:20:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:04 smithi033 ceph-mon[27845]: pgmap v1618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1744.425464 secs 2024-04-26T12:20:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:04 smithi096 ceph-mon[32753]: pgmap v1618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1744.425464 secs 2024-04-26T12:20:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:06 smithi194 ceph-mon[33017]: pgmap v1619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:06 smithi033 ceph-mon[27845]: pgmap v1619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:06 smithi096 ceph-mon[32753]: pgmap v1619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:08 smithi194 ceph-mon[33017]: pgmap v1620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1746.706617 secs 2024-04-26T12:20:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:08 smithi033 ceph-mon[27845]: pgmap v1620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1746.706617 secs 2024-04-26T12:20:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:08 smithi096 ceph-mon[32753]: pgmap v1620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1746.706617 secs 2024-04-26T12:20:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1749.425581 secs 2024-04-26T12:20:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1749.425581 secs 2024-04-26T12:20:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1749.425581 secs 2024-04-26T12:20:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:10 smithi033 ceph-mon[27845]: pgmap v1621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:10 smithi096 ceph-mon[32753]: pgmap v1621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:10 smithi194 ceph-mon[33017]: pgmap v1621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:12 smithi033 ceph-mon[27845]: pgmap v1622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:12 smithi096 ceph-mon[32753]: pgmap v1622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:12 smithi194 ceph-mon[33017]: pgmap v1622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1751.706719 secs 2024-04-26T12:20:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1751.706719 secs 2024-04-26T12:20:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1751.706719 secs 2024-04-26T12:20:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:14 smithi033 ceph-mon[27845]: pgmap v1623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1754.425665 secs 2024-04-26T12:20:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:14 smithi096 ceph-mon[32753]: pgmap v1623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1754.425665 secs 2024-04-26T12:20:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:14 smithi194 ceph-mon[33017]: pgmap v1623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1754.425665 secs 2024-04-26T12:20:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:16 smithi033 ceph-mon[27845]: pgmap v1624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:16 smithi096 ceph-mon[32753]: pgmap v1624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:16 smithi194 ceph-mon[33017]: pgmap v1624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:18 smithi033 ceph-mon[27845]: pgmap v1625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1756.706820 secs 2024-04-26T12:20:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1759.425787 secs 2024-04-26T12:20:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:18 smithi096 ceph-mon[32753]: pgmap v1625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1756.706820 secs 2024-04-26T12:20:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1759.425787 secs 2024-04-26T12:20:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:18 smithi194 ceph-mon[33017]: pgmap v1625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1756.706820 secs 2024-04-26T12:20:18.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1759.425787 secs 2024-04-26T12:20:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:18.862+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:18.862+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:18.863+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:20:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:20 smithi033 ceph-mon[27845]: pgmap v1626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:20 smithi096 ceph-mon[32753]: pgmap v1626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:20 smithi194 ceph-mon[33017]: pgmap v1626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:21.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:20:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:20:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:20:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:22.095+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:20:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:22.095+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:20:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:22.096+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T12:20:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:22.096+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:20:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:22 smithi033 ceph-mon[27845]: pgmap v1627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:22 smithi096 ceph-mon[32753]: pgmap v1627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:22 smithi194 ceph-mon[33017]: pgmap v1627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:22.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1761.706923 secs 2024-04-26T12:20:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1764.425851 secs 2024-04-26T12:20:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:20:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:20:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1761.706923 secs 2024-04-26T12:20:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1764.425851 secs 2024-04-26T12:20:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:20:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:20:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1761.706923 secs 2024-04-26T12:20:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1764.425851 secs 2024-04-26T12:20:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:20:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:20:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:20:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:24 smithi033 ceph-mon[27845]: pgmap v1628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:24 smithi033 ceph-mon[27845]: pgmap v1629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:20:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:24 smithi033 ceph-mon[27845]: pgmap v1630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:20:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:24 smithi096 ceph-mon[32753]: pgmap v1628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:24 smithi096 ceph-mon[32753]: pgmap v1629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:20:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:24 smithi096 ceph-mon[32753]: pgmap v1630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:20:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:24 smithi194 ceph-mon[33017]: pgmap v1628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:24 smithi194 ceph-mon[33017]: pgmap v1629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:20:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:24 smithi194 ceph-mon[33017]: pgmap v1630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:20:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:25 smithi096 ceph-mon[32753]: pgmap v1631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:25 smithi194 ceph-mon[33017]: pgmap v1631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:25 smithi033 ceph-mon[27845]: pgmap v1631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:28 smithi194 ceph-mon[33017]: pgmap v1632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1766.707007 secs 2024-04-26T12:20:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:28 smithi033 ceph-mon[27845]: pgmap v1632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1766.707007 secs 2024-04-26T12:20:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:28 smithi096 ceph-mon[32753]: pgmap v1632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1766.707007 secs 2024-04-26T12:20:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1769.425985 secs 2024-04-26T12:20:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1769.425985 secs 2024-04-26T12:20:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1769.425985 secs 2024-04-26T12:20:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:30 smithi194 ceph-mon[33017]: pgmap v1633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:20:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:30 smithi033 ceph-mon[27845]: pgmap v1633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:20:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:30 smithi096 ceph-mon[32753]: pgmap v1633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:20:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:32 smithi194 ceph-mon[33017]: pgmap v1634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:32 smithi033 ceph-mon[27845]: pgmap v1634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:32 smithi096 ceph-mon[32753]: pgmap v1634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:20:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1771.707099 secs 2024-04-26T12:20:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1771.707099 secs 2024-04-26T12:20:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1771.707099 secs 2024-04-26T12:20:33.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:33.871+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:33.871+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:33.871+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:20:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1774.426047 secs 2024-04-26T12:20:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:34 smithi194 ceph-mon[33017]: pgmap v1635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:20:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1774.426047 secs 2024-04-26T12:20:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:34 smithi033 ceph-mon[27845]: pgmap v1635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:20:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1774.426047 secs 2024-04-26T12:20:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:34 smithi096 ceph-mon[32753]: pgmap v1635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:20:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:36 smithi033 ceph-mon[27845]: pgmap v1636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:36 smithi096 ceph-mon[32753]: pgmap v1636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:36 smithi194 ceph-mon[33017]: pgmap v1636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:38 smithi033 ceph-mon[27845]: pgmap v1637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1776.707161 secs 2024-04-26T12:20:38.688 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:38 smithi096 ceph-mon[32753]: pgmap v1637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:38.688 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1776.707161 secs 2024-04-26T12:20:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:38 smithi194 ceph-mon[33017]: pgmap v1637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1776.707161 secs 2024-04-26T12:20:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1779.426163 secs 2024-04-26T12:20:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1779.426163 secs 2024-04-26T12:20:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1779.426163 secs 2024-04-26T12:20:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:40 smithi033 ceph-mon[27845]: pgmap v1638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:40 smithi096 ceph-mon[32753]: pgmap v1638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:40 smithi194 ceph-mon[33017]: pgmap v1638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:42 smithi033 ceph-mon[27845]: pgmap v1639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:42 smithi096 ceph-mon[32753]: pgmap v1639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:42 smithi194 ceph-mon[33017]: pgmap v1639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1781.707222 secs 2024-04-26T12:20:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1781.707222 secs 2024-04-26T12:20:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1781.707222 secs 2024-04-26T12:20:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1784.426280 secs 2024-04-26T12:20:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:44 smithi033 ceph-mon[27845]: pgmap v1640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1784.426280 secs 2024-04-26T12:20:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:44 smithi096 ceph-mon[32753]: pgmap v1640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1784.426280 secs 2024-04-26T12:20:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:44 smithi194 ceph-mon[33017]: pgmap v1640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:46 smithi033 ceph-mon[27845]: pgmap v1641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:46 smithi096 ceph-mon[32753]: pgmap v1641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:46 smithi194 ceph-mon[33017]: pgmap v1641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:48 smithi033 ceph-mon[27845]: pgmap v1642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1786.707284 secs 2024-04-26T12:20:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1789.426389 secs 2024-04-26T12:20:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:48 smithi096 ceph-mon[32753]: pgmap v1642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1786.707284 secs 2024-04-26T12:20:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1789.426389 secs 2024-04-26T12:20:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:48 smithi194 ceph-mon[33017]: pgmap v1642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1786.707284 secs 2024-04-26T12:20:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1789.426389 secs 2024-04-26T12:20:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:48.888+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:48.888+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:20:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:20:48.888+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:20:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:50 smithi033 ceph-mon[27845]: pgmap v1643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:50 smithi096 ceph-mon[32753]: pgmap v1643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:50 smithi194 ceph-mon[33017]: pgmap v1643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:52 smithi033 ceph-mon[27845]: pgmap v1644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:52 smithi096 ceph-mon[32753]: pgmap v1644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:52 smithi194 ceph-mon[33017]: pgmap v1644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1791.707385 secs 2024-04-26T12:20:53.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1794.426505 secs 2024-04-26T12:20:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1791.707385 secs 2024-04-26T12:20:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1794.426505 secs 2024-04-26T12:20:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1791.707385 secs 2024-04-26T12:20:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1794.426505 secs 2024-04-26T12:20:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:54 smithi033 ceph-mon[27845]: pgmap v1645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:54 smithi096 ceph-mon[32753]: pgmap v1645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:54 smithi194 ceph-mon[33017]: pgmap v1645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:55 smithi194 ceph-mon[33017]: pgmap v1646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:55 smithi033 ceph-mon[27845]: pgmap v1646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:55 smithi096 ceph-mon[32753]: pgmap v1646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:20:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:58 smithi194 ceph-mon[33017]: pgmap v1647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1796.707485 secs 2024-04-26T12:20:58.636 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:58 smithi096 ceph-mon[32753]: pgmap v1647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:58.636 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1796.707485 secs 2024-04-26T12:20:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:58 smithi033 ceph-mon[27845]: pgmap v1647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:20:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1796.707485 secs 2024-04-26T12:20:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:20:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1799.426570 secs 2024-04-26T12:20:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:20:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1799.426570 secs 2024-04-26T12:20:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:20:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1799.426570 secs 2024-04-26T12:21:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:00 smithi194 ceph-mon[33017]: pgmap v1648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:00 smithi033 ceph-mon[27845]: pgmap v1648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:00 smithi096 ceph-mon[32753]: pgmap v1648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:02 smithi194 ceph-mon[33017]: pgmap v1649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:02.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:02 smithi033 ceph-mon[27845]: pgmap v1649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:02 smithi096 ceph-mon[32753]: pgmap v1649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1801.707585 secs 2024-04-26T12:21:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1801.707585 secs 2024-04-26T12:21:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1801.707585 secs 2024-04-26T12:21:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:03.892+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:03.892+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:21:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:03.892+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1804.426686 secs 2024-04-26T12:21:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:04 smithi194 ceph-mon[33017]: pgmap v1650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1804.426686 secs 2024-04-26T12:21:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:04 smithi033 ceph-mon[27845]: pgmap v1650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1804.426686 secs 2024-04-26T12:21:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:04 smithi096 ceph-mon[32753]: pgmap v1650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:06 smithi194 ceph-mon[33017]: pgmap v1651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:06 smithi033 ceph-mon[27845]: pgmap v1651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:06 smithi096 ceph-mon[32753]: pgmap v1651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:08 smithi194 ceph-mon[33017]: pgmap v1652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1806.707684 secs 2024-04-26T12:21:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:08 smithi033 ceph-mon[27845]: pgmap v1652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1806.707684 secs 2024-04-26T12:21:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:08 smithi096 ceph-mon[32753]: pgmap v1652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1806.707684 secs 2024-04-26T12:21:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1809.426756 secs 2024-04-26T12:21:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1809.426756 secs 2024-04-26T12:21:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1809.426756 secs 2024-04-26T12:21:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:10 smithi194 ceph-mon[33017]: pgmap v1653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:10 smithi033 ceph-mon[27845]: pgmap v1653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:10 smithi096 ceph-mon[32753]: pgmap v1653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:12 smithi194 ceph-mon[33017]: pgmap v1654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:12 smithi033 ceph-mon[27845]: pgmap v1654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:12 smithi096 ceph-mon[32753]: pgmap v1654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1811.707786 secs 2024-04-26T12:21:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1811.707786 secs 2024-04-26T12:21:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1811.707786 secs 2024-04-26T12:21:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1814.426853 secs 2024-04-26T12:21:14.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:14 smithi033 ceph-mon[27845]: pgmap v1655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1814.426853 secs 2024-04-26T12:21:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:14 smithi096 ceph-mon[32753]: pgmap v1655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1814.426853 secs 2024-04-26T12:21:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:14 smithi194 ceph-mon[33017]: pgmap v1655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:16 smithi033 ceph-mon[27845]: pgmap v1656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:16 smithi096 ceph-mon[32753]: pgmap v1656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:16 smithi194 ceph-mon[33017]: pgmap v1656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:18 smithi033 ceph-mon[27845]: pgmap v1657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1816.707886 secs 2024-04-26T12:21:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:18 smithi096 ceph-mon[32753]: pgmap v1657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1816.707886 secs 2024-04-26T12:21:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:18 smithi194 ceph-mon[33017]: pgmap v1657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1816.707886 secs 2024-04-26T12:21:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:18.907+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:18.908+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:18.908+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:21:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1819.426977 secs 2024-04-26T12:21:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1819.426977 secs 2024-04-26T12:21:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1819.426977 secs 2024-04-26T12:21:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:20 smithi033 ceph-mon[27845]: pgmap v1658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:20 smithi096 ceph-mon[32753]: pgmap v1658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:20 smithi194 ceph-mon[33017]: pgmap v1658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:22 smithi033 ceph-mon[27845]: pgmap v1659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:22 smithi096 ceph-mon[32753]: pgmap v1659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:22 smithi194 ceph-mon[33017]: pgmap v1659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1821.707994 secs 2024-04-26T12:21:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1824.427095 secs 2024-04-26T12:21:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1821.707994 secs 2024-04-26T12:21:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1824.427095 secs 2024-04-26T12:21:23.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1821.707994 secs 2024-04-26T12:21:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1824.427095 secs 2024-04-26T12:21:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:24 smithi033 ceph-mon[27845]: pgmap v1660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:21:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:24 smithi096 ceph-mon[32753]: pgmap v1660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:21:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:24 smithi194 ceph-mon[33017]: pgmap v1660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:21:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:21:25.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:21:25.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:21:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:21:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:21:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:21:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:21:25.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:21:25.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:21:26.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:26 smithi033 ceph-mon[27845]: pgmap v1661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:21:26.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:26 smithi033 ceph-mon[27845]: pgmap v1662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:26 smithi096 ceph-mon[32753]: pgmap v1661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:21:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:26 smithi096 ceph-mon[32753]: pgmap v1662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:26 smithi194 ceph-mon[33017]: pgmap v1661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:21:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:26 smithi194 ceph-mon[33017]: pgmap v1662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:28 smithi033 ceph-mon[27845]: pgmap v1663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1826.708057 secs 2024-04-26T12:21:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1829.427188 secs 2024-04-26T12:21:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:28 smithi096 ceph-mon[32753]: pgmap v1663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1826.708057 secs 2024-04-26T12:21:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1829.427188 secs 2024-04-26T12:21:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:28 smithi194 ceph-mon[33017]: pgmap v1663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1826.708057 secs 2024-04-26T12:21:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1829.427188 secs 2024-04-26T12:21:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:30 smithi096 ceph-mon[32753]: pgmap v1664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:30 smithi194 ceph-mon[33017]: pgmap v1664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:30 smithi033 ceph-mon[27845]: pgmap v1664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:31 smithi194 ceph-mon[33017]: pgmap v1665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:31 smithi033 ceph-mon[27845]: pgmap v1665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:31 smithi096 ceph-mon[32753]: pgmap v1665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1831.708118 secs 2024-04-26T12:21:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1831.708118 secs 2024-04-26T12:21:33.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1831.708118 secs 2024-04-26T12:21:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:33.919+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:33.919+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:33.919+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:21:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:33 smithi194 ceph-mon[33017]: pgmap v1666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1834.427305 secs 2024-04-26T12:21:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:33 smithi033 ceph-mon[27845]: pgmap v1666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:34.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1834.427305 secs 2024-04-26T12:21:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:33 smithi096 ceph-mon[32753]: pgmap v1666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:21:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1834.427305 secs 2024-04-26T12:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:35 smithi194 ceph-mon[33017]: pgmap v1667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:21:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:35 smithi033 ceph-mon[27845]: pgmap v1667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:21:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:35 smithi096 ceph-mon[32753]: pgmap v1667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:21:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:38 smithi194 ceph-mon[33017]: pgmap v1668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1836.708205 secs 2024-04-26T12:21:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:38 smithi033 ceph-mon[27845]: pgmap v1668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1836.708205 secs 2024-04-26T12:21:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:38 smithi096 ceph-mon[32753]: pgmap v1668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1836.708205 secs 2024-04-26T12:21:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1839.427424 secs 2024-04-26T12:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1839.427424 secs 2024-04-26T12:21:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1839.427424 secs 2024-04-26T12:21:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:40 smithi194 ceph-mon[33017]: pgmap v1669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:40 smithi033 ceph-mon[27845]: pgmap v1669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:40 smithi096 ceph-mon[32753]: pgmap v1669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:42 smithi194 ceph-mon[33017]: pgmap v1670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:42 smithi033 ceph-mon[27845]: pgmap v1670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:42 smithi096 ceph-mon[32753]: pgmap v1670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1841.708295 secs 2024-04-26T12:21:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1841.708295 secs 2024-04-26T12:21:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1841.708295 secs 2024-04-26T12:21:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:44 smithi194 ceph-mon[33017]: pgmap v1671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:44.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1844.427564 secs 2024-04-26T12:21:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:44 smithi033 ceph-mon[27845]: pgmap v1671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1844.427564 secs 2024-04-26T12:21:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:44 smithi096 ceph-mon[32753]: pgmap v1671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1844.427564 secs 2024-04-26T12:21:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:46 smithi194 ceph-mon[33017]: pgmap v1672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:46 smithi033 ceph-mon[27845]: pgmap v1672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:46 smithi096 ceph-mon[32753]: pgmap v1672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:48 smithi194 ceph-mon[33017]: pgmap v1673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1846.708354 secs 2024-04-26T12:21:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:48 smithi033 ceph-mon[27845]: pgmap v1673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:48.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1846.708354 secs 2024-04-26T12:21:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:48 smithi096 ceph-mon[32753]: pgmap v1673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1846.708354 secs 2024-04-26T12:21:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:48.934+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:48.934+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:21:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:21:48.934+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:21:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1849.427706 secs 2024-04-26T12:21:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1849.427706 secs 2024-04-26T12:21:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1849.427706 secs 2024-04-26T12:21:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:50 smithi194 ceph-mon[33017]: pgmap v1674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:50 smithi033 ceph-mon[27845]: pgmap v1674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:50 smithi096 ceph-mon[32753]: pgmap v1674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:52 smithi194 ceph-mon[33017]: pgmap v1675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:52 smithi033 ceph-mon[27845]: pgmap v1675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:52 smithi096 ceph-mon[32753]: pgmap v1675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:53.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1851.708414 secs 2024-04-26T12:21:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1851.708414 secs 2024-04-26T12:21:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1851.708414 secs 2024-04-26T12:21:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:54 smithi194 ceph-mon[33017]: pgmap v1676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1854.427824 secs 2024-04-26T12:21:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:54 smithi033 ceph-mon[27845]: pgmap v1676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1854.427824 secs 2024-04-26T12:21:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:54 smithi096 ceph-mon[32753]: pgmap v1676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1854.427824 secs 2024-04-26T12:21:56.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:56 smithi194 ceph-mon[33017]: pgmap v1677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:56 smithi033 ceph-mon[27845]: pgmap v1677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:56 smithi096 ceph-mon[32753]: pgmap v1677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:21:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:58 smithi194 ceph-mon[33017]: pgmap v1678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1856.708498 secs 2024-04-26T12:21:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:58 smithi033 ceph-mon[27845]: pgmap v1678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1856.708498 secs 2024-04-26T12:21:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:58 smithi096 ceph-mon[32753]: pgmap v1678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:21:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1856.708498 secs 2024-04-26T12:21:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:21:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1859.427932 secs 2024-04-26T12:21:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:21:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1859.427932 secs 2024-04-26T12:21:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:21:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1859.427932 secs 2024-04-26T12:22:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:00 smithi194 ceph-mon[33017]: pgmap v1679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:00 smithi033 ceph-mon[27845]: pgmap v1679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:00 smithi096 ceph-mon[32753]: pgmap v1679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:02 smithi033 ceph-mon[27845]: pgmap v1680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:02 smithi096 ceph-mon[32753]: pgmap v1680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:02 smithi194 ceph-mon[33017]: pgmap v1680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1861.708583 secs 2024-04-26T12:22:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1861.708583 secs 2024-04-26T12:22:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1861.708583 secs 2024-04-26T12:22:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:03.936+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:03.936+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:03.936+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:22:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:04 smithi033 ceph-mon[27845]: pgmap v1681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:04.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1864.428041 secs 2024-04-26T12:22:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:04 smithi096 ceph-mon[32753]: pgmap v1681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1864.428041 secs 2024-04-26T12:22:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:04 smithi194 ceph-mon[33017]: pgmap v1681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1864.428041 secs 2024-04-26T12:22:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:06 smithi033 ceph-mon[27845]: pgmap v1682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:06 smithi096 ceph-mon[32753]: pgmap v1682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:06 smithi194 ceph-mon[33017]: pgmap v1682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:08.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:08 smithi033 ceph-mon[27845]: pgmap v1683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:08.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1866.708666 secs 2024-04-26T12:22:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:08 smithi096 ceph-mon[32753]: pgmap v1683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1866.708666 secs 2024-04-26T12:22:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:08 smithi194 ceph-mon[33017]: pgmap v1683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1866.708666 secs 2024-04-26T12:22:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1869.428166 secs 2024-04-26T12:22:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1869.428166 secs 2024-04-26T12:22:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1869.428166 secs 2024-04-26T12:22:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:10 smithi033 ceph-mon[27845]: pgmap v1684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:10 smithi096 ceph-mon[32753]: pgmap v1684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:10 smithi194 ceph-mon[33017]: pgmap v1684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:12 smithi033 ceph-mon[27845]: pgmap v1685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:12 smithi096 ceph-mon[32753]: pgmap v1685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:12 smithi194 ceph-mon[33017]: pgmap v1685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1871.708751 secs 2024-04-26T12:22:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1871.708751 secs 2024-04-26T12:22:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1871.708751 secs 2024-04-26T12:22:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:14 smithi033 ceph-mon[27845]: pgmap v1686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1874.428291 secs 2024-04-26T12:22:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:14 smithi096 ceph-mon[32753]: pgmap v1686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1874.428291 secs 2024-04-26T12:22:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:14 smithi194 ceph-mon[33017]: pgmap v1686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1874.428291 secs 2024-04-26T12:22:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:16 smithi033 ceph-mon[27845]: pgmap v1687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:16 smithi096 ceph-mon[32753]: pgmap v1687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:16 smithi194 ceph-mon[33017]: pgmap v1687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:18 smithi033 ceph-mon[27845]: pgmap v1688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1876.708839 secs 2024-04-26T12:22:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:18 smithi096 ceph-mon[32753]: pgmap v1688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1876.708839 secs 2024-04-26T12:22:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:18 smithi194 ceph-mon[33017]: pgmap v1688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1876.708839 secs 2024-04-26T12:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:18.949+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:18.949+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:18.949+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:22:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1879.428408 secs 2024-04-26T12:22:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1879.428408 secs 2024-04-26T12:22:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1879.428408 secs 2024-04-26T12:22:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:20 smithi096 ceph-mon[32753]: pgmap v1689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:20 smithi194 ceph-mon[33017]: pgmap v1689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:20 smithi033 ceph-mon[27845]: pgmap v1689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:22 smithi194 ceph-mon[33017]: pgmap v1690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:22 smithi033 ceph-mon[27845]: pgmap v1690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:22 smithi096 ceph-mon[32753]: pgmap v1690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:23.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1881.708922 secs 2024-04-26T12:22:23.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1881.708922 secs 2024-04-26T12:22:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1881.708922 secs 2024-04-26T12:22:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:24 smithi194 ceph-mon[33017]: pgmap v1691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:24.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1884.428563 secs 2024-04-26T12:22:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:24 smithi033 ceph-mon[27845]: pgmap v1691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1884.428563 secs 2024-04-26T12:22:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:24 smithi096 ceph-mon[32753]: pgmap v1691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1884.428563 secs 2024-04-26T12:22:26.602 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:26 smithi033 ceph-mon[27845]: pgmap v1692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:26.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:22:26.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:26 smithi194 ceph-mon[33017]: pgmap v1692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:22:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:26 smithi096 ceph-mon[32753]: pgmap v1692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:22:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:26.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:28 smithi194 ceph-mon[33017]: pgmap v1693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1886.709029 secs 2024-04-26T12:22:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:28 smithi033 ceph-mon[27845]: pgmap v1693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1886.709029 secs 2024-04-26T12:22:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:28 smithi096 ceph-mon[32753]: pgmap v1693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1886.709029 secs 2024-04-26T12:22:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1889.428631 secs 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1889.428631 secs 2024-04-26T12:22:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1889.428631 secs 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:29.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:30 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:22:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:30 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:22:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:30 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:22:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:30 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:22:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:30 smithi194 ceph-mon[33017]: pgmap v1694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:30 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:22:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:30 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:22:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:30 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:22:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:30 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:22:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:30 smithi033 ceph-mon[27845]: pgmap v1694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:30 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:22:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:30 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:22:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:30 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:22:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:30 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:22:30.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:30 smithi096 ceph-mon[32753]: pgmap v1694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:30.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:30.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: pgmap v1695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:22:32.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:22:32.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:22:32.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: pgmap v1696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:22:32.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: pgmap v1697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:32.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:32.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: pgmap v1695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:22:32.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:22:32.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: pgmap v1696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:22:32.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: pgmap v1697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:32.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: pgmap v1695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: pgmap v1696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: pgmap v1697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:32.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:22:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1891.709089 secs 2024-04-26T12:22:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1891.709089 secs 2024-04-26T12:22:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1891.709089 secs 2024-04-26T12:22:33.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:33.964+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:33.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:33.964+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:33.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:33.964+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:22:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1894.428703 secs 2024-04-26T12:22:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:34 smithi194 ceph-mon[33017]: pgmap v1698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:22:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1894.428703 secs 2024-04-26T12:22:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:34 smithi033 ceph-mon[27845]: pgmap v1698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:22:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1894.428703 secs 2024-04-26T12:22:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:34 smithi096 ceph-mon[32753]: pgmap v1698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:22:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:36 smithi194 ceph-mon[33017]: pgmap v1699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:36 smithi033 ceph-mon[27845]: pgmap v1699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:36 smithi096 ceph-mon[32753]: pgmap v1699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:37.117+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:22:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:37.117+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:22:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:37.118+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:22:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:37.118+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T12:22:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:38 smithi194 ceph-mon[33017]: pgmap v1700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1896.709175 secs 2024-04-26T12:22:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:38 smithi033 ceph-mon[27845]: pgmap v1700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1896.709175 secs 2024-04-26T12:22:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:38 smithi096 ceph-mon[32753]: pgmap v1700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:22:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1896.709175 secs 2024-04-26T12:22:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1899.428839 secs 2024-04-26T12:22:39.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1899.428839 secs 2024-04-26T12:22:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1899.428839 secs 2024-04-26T12:22:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:40 smithi194 ceph-mon[33017]: pgmap v1701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:22:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:40 smithi033 ceph-mon[27845]: pgmap v1701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:22:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:40 smithi096 ceph-mon[32753]: pgmap v1701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:22:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:42 smithi194 ceph-mon[33017]: pgmap v1702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:22:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:42 smithi033 ceph-mon[27845]: pgmap v1702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:22:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:42 smithi096 ceph-mon[32753]: pgmap v1702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:22:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1901.709261 secs 2024-04-26T12:22:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1901.709261 secs 2024-04-26T12:22:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1901.709261 secs 2024-04-26T12:22:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1904.428958 secs 2024-04-26T12:22:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:44 smithi194 ceph-mon[33017]: pgmap v1703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1904.428958 secs 2024-04-26T12:22:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:44 smithi033 ceph-mon[27845]: pgmap v1703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1904.428958 secs 2024-04-26T12:22:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:44 smithi096 ceph-mon[32753]: pgmap v1703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:46 smithi033 ceph-mon[27845]: pgmap v1704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:46 smithi096 ceph-mon[32753]: pgmap v1704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:46 smithi194 ceph-mon[33017]: pgmap v1704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:48 smithi033 ceph-mon[27845]: pgmap v1705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1906.709324 secs 2024-04-26T12:22:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:48 smithi096 ceph-mon[32753]: pgmap v1705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1906.709324 secs 2024-04-26T12:22:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:48 smithi194 ceph-mon[33017]: pgmap v1705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1906.709324 secs 2024-04-26T12:22:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:48.980+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:48.980+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:22:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:22:48.980+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:22:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1909.429079 secs 2024-04-26T12:22:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1909.429079 secs 2024-04-26T12:22:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1909.429079 secs 2024-04-26T12:22:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:50 smithi033 ceph-mon[27845]: pgmap v1706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:50.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:50 smithi096 ceph-mon[32753]: pgmap v1706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:50 smithi194 ceph-mon[33017]: pgmap v1706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:52 smithi033 ceph-mon[27845]: pgmap v1707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:52 smithi096 ceph-mon[32753]: pgmap v1707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:52 smithi194 ceph-mon[33017]: pgmap v1707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:53.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1911.709410 secs 2024-04-26T12:22:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1911.709410 secs 2024-04-26T12:22:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1911.709410 secs 2024-04-26T12:22:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1914.429197 secs 2024-04-26T12:22:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:54 smithi033 ceph-mon[27845]: pgmap v1708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1914.429197 secs 2024-04-26T12:22:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:54 smithi096 ceph-mon[32753]: pgmap v1708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1914.429197 secs 2024-04-26T12:22:54.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:54 smithi194 ceph-mon[33017]: pgmap v1708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:56 smithi033 ceph-mon[27845]: pgmap v1709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:56 smithi096 ceph-mon[32753]: pgmap v1709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:56 smithi194 ceph-mon[33017]: pgmap v1709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:22:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:58 smithi033 ceph-mon[27845]: pgmap v1710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1916.709492 secs 2024-04-26T12:22:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:22:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1919.429335 secs 2024-04-26T12:22:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:58 smithi096 ceph-mon[32753]: pgmap v1710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1916.709492 secs 2024-04-26T12:22:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:22:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1919.429335 secs 2024-04-26T12:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:58 smithi194 ceph-mon[33017]: pgmap v1710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1916.709492 secs 2024-04-26T12:22:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:22:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1919.429335 secs 2024-04-26T12:23:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:00 smithi033 ceph-mon[27845]: pgmap v1711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:00 smithi096 ceph-mon[32753]: pgmap v1711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:00 smithi194 ceph-mon[33017]: pgmap v1711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:02.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:02 smithi033 ceph-mon[27845]: pgmap v1712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:02 smithi096 ceph-mon[32753]: pgmap v1712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:02 smithi194 ceph-mon[33017]: pgmap v1712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1921.709564 secs 2024-04-26T12:23:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1924.429446 secs 2024-04-26T12:23:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1921.709564 secs 2024-04-26T12:23:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1924.429446 secs 2024-04-26T12:23:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1921.709564 secs 2024-04-26T12:23:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1924.429446 secs 2024-04-26T12:23:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:03.994+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:03.994+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:23:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:03.994+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:04.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:04 smithi033 ceph-mon[27845]: pgmap v1713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:04 smithi096 ceph-mon[32753]: pgmap v1713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:04 smithi194 ceph-mon[33017]: pgmap v1713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:06 smithi194 ceph-mon[33017]: pgmap v1714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:06 smithi033 ceph-mon[27845]: pgmap v1714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:06 smithi096 ceph-mon[32753]: pgmap v1714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:08 smithi194 ceph-mon[33017]: pgmap v1715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:08.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1926.709625 secs 2024-04-26T12:23:08.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1929.429512 secs 2024-04-26T12:23:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:08 smithi033 ceph-mon[27845]: pgmap v1715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1926.709625 secs 2024-04-26T12:23:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1929.429512 secs 2024-04-26T12:23:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:08 smithi096 ceph-mon[32753]: pgmap v1715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1926.709625 secs 2024-04-26T12:23:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1929.429512 secs 2024-04-26T12:23:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:10 smithi194 ceph-mon[33017]: pgmap v1716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:10 smithi033 ceph-mon[27845]: pgmap v1716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:10 smithi096 ceph-mon[32753]: pgmap v1716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:12 smithi194 ceph-mon[33017]: pgmap v1717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:12 smithi033 ceph-mon[27845]: pgmap v1717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:12 smithi096 ceph-mon[32753]: pgmap v1717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1931.709719 secs 2024-04-26T12:23:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1934.429632 secs 2024-04-26T12:23:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:13 smithi194 ceph-mon[33017]: pgmap v1718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1931.709719 secs 2024-04-26T12:23:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1934.429632 secs 2024-04-26T12:23:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:13 smithi033 ceph-mon[27845]: pgmap v1718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1931.709719 secs 2024-04-26T12:23:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1934.429632 secs 2024-04-26T12:23:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:13 smithi096 ceph-mon[32753]: pgmap v1718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:16 smithi194 ceph-mon[33017]: pgmap v1719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:16 smithi033 ceph-mon[27845]: pgmap v1719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:16 smithi096 ceph-mon[32753]: pgmap v1719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:18 smithi194 ceph-mon[33017]: pgmap v1720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1936.709799 secs 2024-04-26T12:23:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:18 smithi033 ceph-mon[27845]: pgmap v1720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1936.709799 secs 2024-04-26T12:23:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:18 smithi096 ceph-mon[32753]: pgmap v1720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1936.709799 secs 2024-04-26T12:23:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:19.006+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:19.006+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:19.006+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:23:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1939.429752 secs 2024-04-26T12:23:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1939.429752 secs 2024-04-26T12:23:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1939.429752 secs 2024-04-26T12:23:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:20 smithi033 ceph-mon[27845]: pgmap v1721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:20 smithi096 ceph-mon[32753]: pgmap v1721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:20 smithi194 ceph-mon[33017]: pgmap v1721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:22.126+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:23:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:22.126+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:23:22.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:22.126+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:23:22.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:22.126+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:23:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:22 smithi033 ceph-mon[27845]: pgmap v1722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:22 smithi096 ceph-mon[32753]: pgmap v1722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:22 smithi194 ceph-mon[33017]: pgmap v1722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:23.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1941.709883 secs 2024-04-26T12:23:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1941.709883 secs 2024-04-26T12:23:23.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1941.709883 secs 2024-04-26T12:23:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1944.429884 secs 2024-04-26T12:23:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:24 smithi033 ceph-mon[27845]: pgmap v1723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1944.429884 secs 2024-04-26T12:23:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:24 smithi096 ceph-mon[32753]: pgmap v1723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1944.429884 secs 2024-04-26T12:23:24.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:24 smithi194 ceph-mon[33017]: pgmap v1723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:26 smithi033 ceph-mon[27845]: pgmap v1724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:26 smithi096 ceph-mon[32753]: pgmap v1724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:26 smithi194 ceph-mon[33017]: pgmap v1724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:28 smithi033 ceph-mon[27845]: pgmap v1725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1946.709945 secs 2024-04-26T12:23:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:28 smithi096 ceph-mon[32753]: pgmap v1725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1946.709945 secs 2024-04-26T12:23:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:28 smithi194 ceph-mon[33017]: pgmap v1725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1946.709945 secs 2024-04-26T12:23:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1949.430002 secs 2024-04-26T12:23:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1949.430002 secs 2024-04-26T12:23:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1949.430002 secs 2024-04-26T12:23:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:30 smithi033 ceph-mon[27845]: pgmap v1726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:23:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:30 smithi096 ceph-mon[32753]: pgmap v1726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:23:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:30 smithi194 ceph-mon[33017]: pgmap v1726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:23:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:32 smithi033 ceph-mon[27845]: pgmap v1727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:23:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:32 smithi096 ceph-mon[32753]: pgmap v1727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:23:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:32 smithi194 ceph-mon[33017]: pgmap v1727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:23:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1951.710023 secs 2024-04-26T12:23:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:23:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:23:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:23:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1954.430136 secs 2024-04-26T12:23:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1951.710023 secs 2024-04-26T12:23:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:23:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:23:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:23:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1954.430136 secs 2024-04-26T12:23:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1951.710023 secs 2024-04-26T12:23:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:23:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:23:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:23:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1954.430136 secs 2024-04-26T12:23:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:34.019+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:34.019+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:23:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:34.019+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:34 smithi033 ceph-mon[27845]: pgmap v1728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:34 smithi033 ceph-mon[27845]: pgmap v1729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:34 smithi096 ceph-mon[32753]: pgmap v1728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:34 smithi096 ceph-mon[32753]: pgmap v1729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:34 smithi194 ceph-mon[33017]: pgmap v1728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:34 smithi194 ceph-mon[33017]: pgmap v1729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:36 smithi033 ceph-mon[27845]: pgmap v1730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:36 smithi096 ceph-mon[32753]: pgmap v1730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:36 smithi194 ceph-mon[33017]: pgmap v1730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:38 smithi096 ceph-mon[32753]: pgmap v1731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1956.710106 secs 2024-04-26T12:23:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1959.430260 secs 2024-04-26T12:23:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:38 smithi194 ceph-mon[33017]: pgmap v1731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:38.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1956.710106 secs 2024-04-26T12:23:38.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1959.430260 secs 2024-04-26T12:23:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:38 smithi033 ceph-mon[27845]: pgmap v1731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1956.710106 secs 2024-04-26T12:23:38.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1959.430260 secs 2024-04-26T12:23:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:40 smithi096 ceph-mon[32753]: pgmap v1732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:23:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:40 smithi194 ceph-mon[33017]: pgmap v1732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:23:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:40 smithi033 ceph-mon[27845]: pgmap v1732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:23:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:42 smithi194 ceph-mon[33017]: pgmap v1733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:42.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:42 smithi033 ceph-mon[27845]: pgmap v1733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:42 smithi096 ceph-mon[32753]: pgmap v1733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:23:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1961.710189 secs 2024-04-26T12:23:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1964.430322 secs 2024-04-26T12:23:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1961.710189 secs 2024-04-26T12:23:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1964.430322 secs 2024-04-26T12:23:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1961.710189 secs 2024-04-26T12:23:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1964.430322 secs 2024-04-26T12:23:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:44 smithi194 ceph-mon[33017]: pgmap v1734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:23:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:44 smithi033 ceph-mon[27845]: pgmap v1734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:23:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:44 smithi096 ceph-mon[32753]: pgmap v1734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:23:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:46 smithi194 ceph-mon[33017]: pgmap v1735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:46 smithi033 ceph-mon[27845]: pgmap v1735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:46 smithi096 ceph-mon[32753]: pgmap v1735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:23:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:48 smithi194 ceph-mon[33017]: pgmap v1736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1966.710280 secs 2024-04-26T12:23:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1969.430447 secs 2024-04-26T12:23:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:48 smithi033 ceph-mon[27845]: pgmap v1736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1966.710280 secs 2024-04-26T12:23:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1969.430447 secs 2024-04-26T12:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:48 smithi096 ceph-mon[32753]: pgmap v1736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1966.710280 secs 2024-04-26T12:23:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1969.430447 secs 2024-04-26T12:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:49.035+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:49.035+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:23:49.035+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:23:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:49 smithi194 ceph-mon[33017]: pgmap v1737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:49 smithi033 ceph-mon[27845]: pgmap v1737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:49 smithi096 ceph-mon[32753]: pgmap v1737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:23:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:52 smithi194 ceph-mon[33017]: pgmap v1738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:23:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:52 smithi033 ceph-mon[27845]: pgmap v1738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:23:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:52 smithi096 ceph-mon[32753]: pgmap v1738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:23:53.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1971.710371 secs 2024-04-26T12:23:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1971.710371 secs 2024-04-26T12:23:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1971.710371 secs 2024-04-26T12:23:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:54 smithi194 ceph-mon[33017]: pgmap v1739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:23:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1974.430520 secs 2024-04-26T12:23:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:54 smithi033 ceph-mon[27845]: pgmap v1739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:23:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1974.430520 secs 2024-04-26T12:23:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:54 smithi096 ceph-mon[32753]: pgmap v1739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:23:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1974.430520 secs 2024-04-26T12:23:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:56 smithi194 ceph-mon[33017]: pgmap v1740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:23:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:56 smithi033 ceph-mon[27845]: pgmap v1740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:23:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:56 smithi096 ceph-mon[32753]: pgmap v1740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:23:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:58 smithi194 ceph-mon[33017]: pgmap v1741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:23:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1976.710454 secs 2024-04-26T12:23:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:58 smithi033 ceph-mon[27845]: pgmap v1741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:23:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1976.710454 secs 2024-04-26T12:23:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:58 smithi096 ceph-mon[32753]: pgmap v1741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:23:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1976.710454 secs 2024-04-26T12:23:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:23:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1979.430632 secs 2024-04-26T12:23:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:23:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1979.430632 secs 2024-04-26T12:23:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:23:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1979.430632 secs 2024-04-26T12:24:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:00 smithi194 ceph-mon[33017]: pgmap v1742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:24:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:00 smithi033 ceph-mon[27845]: pgmap v1742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:24:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:00 smithi096 ceph-mon[32753]: pgmap v1742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:24:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:02 smithi194 ceph-mon[33017]: pgmap v1743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:24:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:02 smithi033 ceph-mon[27845]: pgmap v1743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:24:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:02 smithi096 ceph-mon[32753]: pgmap v1743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:24:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1981.710536 secs 2024-04-26T12:24:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1981.710536 secs 2024-04-26T12:24:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1981.710536 secs 2024-04-26T12:24:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:04.049+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:04.049+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:04.049+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:24:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1984.430749 secs 2024-04-26T12:24:04.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:04 smithi033 ceph-mon[27845]: pgmap v1744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1984.430749 secs 2024-04-26T12:24:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:04 smithi096 ceph-mon[32753]: pgmap v1744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1984.430749 secs 2024-04-26T12:24:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:04 smithi194 ceph-mon[33017]: pgmap v1744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:06.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:06 smithi033 ceph-mon[27845]: pgmap v1745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:06 smithi096 ceph-mon[32753]: pgmap v1745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:06 smithi194 ceph-mon[33017]: pgmap v1745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:08 smithi033 ceph-mon[27845]: pgmap v1746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1986.710622 secs 2024-04-26T12:24:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:08 smithi096 ceph-mon[32753]: pgmap v1746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1986.710622 secs 2024-04-26T12:24:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:08 smithi194 ceph-mon[33017]: pgmap v1746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1986.710622 secs 2024-04-26T12:24:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1989.430866 secs 2024-04-26T12:24:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1989.430866 secs 2024-04-26T12:24:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1989.430866 secs 2024-04-26T12:24:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:10 smithi033 ceph-mon[27845]: pgmap v1747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:10 smithi096 ceph-mon[32753]: pgmap v1747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:10 smithi194 ceph-mon[33017]: pgmap v1747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:12 smithi033 ceph-mon[27845]: pgmap v1748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:12 smithi096 ceph-mon[32753]: pgmap v1748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:12 smithi194 ceph-mon[33017]: pgmap v1748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:13.431 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1991.710707 secs 2024-04-26T12:24:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1991.710707 secs 2024-04-26T12:24:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1991.710707 secs 2024-04-26T12:24:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1994.431002 secs 2024-04-26T12:24:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:14 smithi033 ceph-mon[27845]: pgmap v1749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1994.431002 secs 2024-04-26T12:24:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:14 smithi096 ceph-mon[32753]: pgmap v1749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1994.431002 secs 2024-04-26T12:24:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:14 smithi194 ceph-mon[33017]: pgmap v1749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:16 smithi033 ceph-mon[27845]: pgmap v1750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:16 smithi096 ceph-mon[32753]: pgmap v1750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:16 smithi194 ceph-mon[33017]: pgmap v1750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:18 smithi033 ceph-mon[27845]: pgmap v1751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 1996.710782 secs 2024-04-26T12:24:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:18 smithi096 ceph-mon[32753]: pgmap v1751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 1996.710782 secs 2024-04-26T12:24:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:18 smithi194 ceph-mon[33017]: pgmap v1751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 1996.710782 secs 2024-04-26T12:24:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:19.057+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:19.057+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:24:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:19.057+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 1999.431121 secs 2024-04-26T12:24:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 1999.431121 secs 2024-04-26T12:24:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 1999.431121 secs 2024-04-26T12:24:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:20 smithi033 ceph-mon[27845]: pgmap v1752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:20 smithi096 ceph-mon[32753]: pgmap v1752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:20 smithi194 ceph-mon[33017]: pgmap v1752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:22 smithi096 ceph-mon[32753]: pgmap v1753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:22 smithi194 ceph-mon[33017]: pgmap v1753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:22 smithi033 ceph-mon[27845]: pgmap v1753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:23.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2001.710866 secs 2024-04-26T12:24:23.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2001.710866 secs 2024-04-26T12:24:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2001.710866 secs 2024-04-26T12:24:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2004.431158 secs 2024-04-26T12:24:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:24 smithi194 ceph-mon[33017]: pgmap v1754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2004.431158 secs 2024-04-26T12:24:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:24 smithi033 ceph-mon[27845]: pgmap v1754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2004.431158 secs 2024-04-26T12:24:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:24 smithi096 ceph-mon[32753]: pgmap v1754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:26 smithi194 ceph-mon[33017]: pgmap v1755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:26 smithi033 ceph-mon[27845]: pgmap v1755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:26 smithi096 ceph-mon[32753]: pgmap v1755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:28.485 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:28 smithi096 ceph-mon[32753]: pgmap v1756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:28.485 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2006.710951 secs 2024-04-26T12:24:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:28 smithi194 ceph-mon[33017]: pgmap v1756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2006.710951 secs 2024-04-26T12:24:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:28 smithi033 ceph-mon[27845]: pgmap v1756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2006.710951 secs 2024-04-26T12:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2009.431232 secs 2024-04-26T12:24:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2009.431232 secs 2024-04-26T12:24:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2009.431232 secs 2024-04-26T12:24:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:30 smithi194 ceph-mon[33017]: pgmap v1757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:30 smithi033 ceph-mon[27845]: pgmap v1757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:30 smithi096 ceph-mon[32753]: pgmap v1757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:32 smithi194 ceph-mon[33017]: pgmap v1758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:32.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:32 smithi033 ceph-mon[27845]: pgmap v1758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:32 smithi096 ceph-mon[32753]: pgmap v1758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2011.711036 secs 2024-04-26T12:24:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2011.711036 secs 2024-04-26T12:24:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2011.711036 secs 2024-04-26T12:24:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:34.071+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:34.071+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:24:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:34.071+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2014.431343 secs 2024-04-26T12:24:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:34 smithi194 ceph-mon[33017]: pgmap v1759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:24:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2014.431343 secs 2024-04-26T12:24:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:34 smithi033 ceph-mon[27845]: pgmap v1759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:24:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2014.431343 secs 2024-04-26T12:24:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:34 smithi096 ceph-mon[32753]: pgmap v1759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:24:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:24:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:24:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:24:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:24:35.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:24:35.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:24:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:24:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:24:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:24:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:36 smithi194 ceph-mon[33017]: pgmap v1760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:24:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:36 smithi194 ceph-mon[33017]: pgmap v1761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:36.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:36 smithi033 ceph-mon[27845]: pgmap v1760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:24:36.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:36 smithi033 ceph-mon[27845]: pgmap v1761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:36 smithi096 ceph-mon[32753]: pgmap v1760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:24:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:36 smithi096 ceph-mon[32753]: pgmap v1761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:38 smithi194 ceph-mon[33017]: pgmap v1762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:38.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2016.711121 secs 2024-04-26T12:24:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:38 smithi033 ceph-mon[27845]: pgmap v1762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2016.711121 secs 2024-04-26T12:24:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:38 smithi096 ceph-mon[32753]: pgmap v1762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2016.711121 secs 2024-04-26T12:24:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2019.431455 secs 2024-04-26T12:24:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2019.431455 secs 2024-04-26T12:24:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2019.431455 secs 2024-04-26T12:24:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:40 smithi194 ceph-mon[33017]: pgmap v1763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:40.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:40 smithi033 ceph-mon[27845]: pgmap v1763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:40 smithi096 ceph-mon[32753]: pgmap v1763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:42 smithi194 ceph-mon[33017]: pgmap v1764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:42 smithi033 ceph-mon[27845]: pgmap v1764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:42 smithi096 ceph-mon[32753]: pgmap v1764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2021.711206 secs 2024-04-26T12:24:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2021.711206 secs 2024-04-26T12:24:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2021.711206 secs 2024-04-26T12:24:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:44 smithi194 ceph-mon[33017]: pgmap v1765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2024.431545 secs 2024-04-26T12:24:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:44 smithi033 ceph-mon[27845]: pgmap v1765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2024.431545 secs 2024-04-26T12:24:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:44 smithi096 ceph-mon[32753]: pgmap v1765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:24:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2024.431545 secs 2024-04-26T12:24:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:46 smithi194 ceph-mon[33017]: pgmap v1766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:24:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:46 smithi033 ceph-mon[27845]: pgmap v1766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:24:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:46 smithi096 ceph-mon[32753]: pgmap v1766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:24:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:48 smithi194 ceph-mon[33017]: pgmap v1767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:48.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2026.711290 secs 2024-04-26T12:24:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:48 smithi033 ceph-mon[27845]: pgmap v1767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2026.711290 secs 2024-04-26T12:24:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:48 smithi096 ceph-mon[32753]: pgmap v1767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2026.711290 secs 2024-04-26T12:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:49.086+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:49.086+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:24:49.086+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:24:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2029.431663 secs 2024-04-26T12:24:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2029.431663 secs 2024-04-26T12:24:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2029.431663 secs 2024-04-26T12:24:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:50 smithi194 ceph-mon[33017]: pgmap v1768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:50 smithi033 ceph-mon[27845]: pgmap v1768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:50 smithi096 ceph-mon[32753]: pgmap v1768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:52 smithi194 ceph-mon[33017]: pgmap v1769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:52.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:52 smithi033 ceph-mon[27845]: pgmap v1769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:52 smithi096 ceph-mon[32753]: pgmap v1769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:24:53.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2031.711352 secs 2024-04-26T12:24:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2031.711352 secs 2024-04-26T12:24:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2031.711352 secs 2024-04-26T12:24:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:54 smithi194 ceph-mon[33017]: pgmap v1770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2034.431799 secs 2024-04-26T12:24:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:54 smithi033 ceph-mon[27845]: pgmap v1770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2034.431799 secs 2024-04-26T12:24:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:54 smithi096 ceph-mon[32753]: pgmap v1770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:24:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2034.431799 secs 2024-04-26T12:24:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:56 smithi194 ceph-mon[33017]: pgmap v1771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:24:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:56 smithi033 ceph-mon[27845]: pgmap v1771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:24:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:56 smithi096 ceph-mon[32753]: pgmap v1771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:24:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:58 smithi194 ceph-mon[33017]: pgmap v1772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:24:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2036.711437 secs 2024-04-26T12:24:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:58 smithi033 ceph-mon[27845]: pgmap v1772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:24:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2036.711437 secs 2024-04-26T12:24:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:58 smithi096 ceph-mon[32753]: pgmap v1772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:24:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2036.711437 secs 2024-04-26T12:24:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2039.431935 secs 2024-04-26T12:24:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2039.431935 secs 2024-04-26T12:24:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2039.431935 secs 2024-04-26T12:25:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:00 smithi194 ceph-mon[33017]: pgmap v1773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:25:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:00 smithi033 ceph-mon[27845]: pgmap v1773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:25:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:00 smithi096 ceph-mon[32753]: pgmap v1773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:25:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:02 smithi194 ceph-mon[33017]: pgmap v1774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:25:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:02 smithi033 ceph-mon[27845]: pgmap v1774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:25:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:02 smithi096 ceph-mon[32753]: pgmap v1774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:25:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2041.711528 secs 2024-04-26T12:25:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2041.711528 secs 2024-04-26T12:25:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2041.711528 secs 2024-04-26T12:25:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:04.101+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:04.101+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:25:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:04.101+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:04 smithi033 ceph-mon[27845]: pgmap v1775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:25:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2044.432032 secs 2024-04-26T12:25:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:04 smithi096 ceph-mon[32753]: pgmap v1775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:25:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2044.432032 secs 2024-04-26T12:25:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:04 smithi194 ceph-mon[33017]: pgmap v1775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:25:04.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2044.432032 secs 2024-04-26T12:25:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:06 smithi033 ceph-mon[27845]: pgmap v1776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:25:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:06 smithi096 ceph-mon[32753]: pgmap v1776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:25:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:06 smithi194 ceph-mon[33017]: pgmap v1776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:25:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:08 smithi033 ceph-mon[27845]: pgmap v1777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:08.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2046.711611 secs 2024-04-26T12:25:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:08 smithi096 ceph-mon[32753]: pgmap v1777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2046.711611 secs 2024-04-26T12:25:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:08 smithi194 ceph-mon[33017]: pgmap v1777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:08.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2046.711611 secs 2024-04-26T12:25:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2049.432164 secs 2024-04-26T12:25:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2049.432164 secs 2024-04-26T12:25:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2049.432164 secs 2024-04-26T12:25:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:10 smithi033 ceph-mon[27845]: pgmap v1778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:10 smithi096 ceph-mon[32753]: pgmap v1778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:10 smithi194 ceph-mon[33017]: pgmap v1778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:12 smithi033 ceph-mon[27845]: pgmap v1779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:12 smithi096 ceph-mon[32753]: pgmap v1779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:12 smithi194 ceph-mon[33017]: pgmap v1779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2051.711694 secs 2024-04-26T12:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2054.432257 secs 2024-04-26T12:25:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2051.711694 secs 2024-04-26T12:25:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2054.432257 secs 2024-04-26T12:25:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2051.711694 secs 2024-04-26T12:25:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2054.432257 secs 2024-04-26T12:25:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:14 smithi194 ceph-mon[33017]: pgmap v1780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:14 smithi033 ceph-mon[27845]: pgmap v1780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:14 smithi096 ceph-mon[32753]: pgmap v1780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:16 smithi194 ceph-mon[33017]: pgmap v1781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:16 smithi033 ceph-mon[27845]: pgmap v1781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:16 smithi096 ceph-mon[32753]: pgmap v1781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:18 smithi194 ceph-mon[33017]: pgmap v1782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2056.711777 secs 2024-04-26T12:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2059.432351 secs 2024-04-26T12:25:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:18 smithi033 ceph-mon[27845]: pgmap v1782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2056.711777 secs 2024-04-26T12:25:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2059.432351 secs 2024-04-26T12:25:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:18 smithi096 ceph-mon[32753]: pgmap v1782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2056.711777 secs 2024-04-26T12:25:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2059.432351 secs 2024-04-26T12:25:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:19.116+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:19.117+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:19.117+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:25:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:20 smithi194 ceph-mon[33017]: pgmap v1783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:20 smithi033 ceph-mon[27845]: pgmap v1783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:20 smithi096 ceph-mon[32753]: pgmap v1783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:22 smithi194 ceph-mon[33017]: pgmap v1784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:22 smithi033 ceph-mon[27845]: pgmap v1784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:22 smithi096 ceph-mon[32753]: pgmap v1784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2061.711867 secs 2024-04-26T12:25:23.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2064.432457 secs 2024-04-26T12:25:23.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2061.711867 secs 2024-04-26T12:25:23.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2064.432457 secs 2024-04-26T12:25:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2061.711867 secs 2024-04-26T12:25:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2064.432457 secs 2024-04-26T12:25:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:24 smithi194 ceph-mon[33017]: pgmap v1785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:24 smithi033 ceph-mon[27845]: pgmap v1785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:24 smithi096 ceph-mon[32753]: pgmap v1785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:25 smithi194 ceph-mon[33017]: pgmap v1786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:25 smithi033 ceph-mon[27845]: pgmap v1786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:25 smithi096 ceph-mon[32753]: pgmap v1786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:27 smithi194 ceph-mon[33017]: pgmap v1787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2066.711954 secs 2024-04-26T12:25:28.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:27 smithi033 ceph-mon[27845]: pgmap v1787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:28.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2066.711954 secs 2024-04-26T12:25:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:27 smithi096 ceph-mon[32753]: pgmap v1787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2066.711954 secs 2024-04-26T12:25:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2069.432528 secs 2024-04-26T12:25:29.137 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2069.432528 secs 2024-04-26T12:25:29.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2069.432528 secs 2024-04-26T12:25:30.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:29 smithi194 ceph-mon[33017]: pgmap v1788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:29 smithi033 ceph-mon[27845]: pgmap v1788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:29 smithi096 ceph-mon[32753]: pgmap v1788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:31 smithi194 ceph-mon[33017]: pgmap v1789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:31 smithi033 ceph-mon[27845]: pgmap v1789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:31 smithi096 ceph-mon[32753]: pgmap v1789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2071.712042 secs 2024-04-26T12:25:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2071.712042 secs 2024-04-26T12:25:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2071.712042 secs 2024-04-26T12:25:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:33 smithi194 ceph-mon[33017]: pgmap v1790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2074.432647 secs 2024-04-26T12:25:34.133 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:34.131+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:34.131+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:25:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:34.131+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:33 smithi033 ceph-mon[27845]: pgmap v1790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:34.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2074.432647 secs 2024-04-26T12:25:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:33 smithi096 ceph-mon[32753]: pgmap v1790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2074.432647 secs 2024-04-26T12:25:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:35 smithi194 ceph-mon[33017]: pgmap v1791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:25:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:35 smithi033 ceph-mon[27845]: pgmap v1791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:25:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:35 smithi096 ceph-mon[32753]: pgmap v1791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:25:38.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:38 smithi033 ceph-mon[27845]: pgmap v1792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:38.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2076.712111 secs 2024-04-26T12:25:38.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2079.432738 secs 2024-04-26T12:25:38.837 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:38 smithi096 ceph-mon[32753]: pgmap v1792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:38.838 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2076.712111 secs 2024-04-26T12:25:38.838 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2079.432738 secs 2024-04-26T12:25:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:38 smithi194 ceph-mon[33017]: pgmap v1792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:38.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2076.712111 secs 2024-04-26T12:25:38.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2079.432738 secs 2024-04-26T12:25:40.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:40 smithi033 ceph-mon[27845]: pgmap v1793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:40.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:40 smithi096 ceph-mon[32753]: pgmap v1793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:40.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:40 smithi194 ceph-mon[33017]: pgmap v1793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:40.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:40.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:42 smithi033 ceph-mon[27845]: pgmap v1794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:25:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:25:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:42 smithi096 ceph-mon[32753]: pgmap v1794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:25:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:25:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:42 smithi194 ceph-mon[33017]: pgmap v1794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:25:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:25:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:25:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:43 smithi033 ceph-mon[27845]: pgmap v1795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:43 smithi033 ceph-mon[27845]: pgmap v1796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2081.712174 secs 2024-04-26T12:25:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:43 smithi096 ceph-mon[32753]: pgmap v1795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:43 smithi096 ceph-mon[32753]: pgmap v1796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2081.712174 secs 2024-04-26T12:25:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:43 smithi194 ceph-mon[33017]: pgmap v1795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:43 smithi194 ceph-mon[33017]: pgmap v1796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2081.712174 secs 2024-04-26T12:25:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2084.432856 secs 2024-04-26T12:25:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2084.432856 secs 2024-04-26T12:25:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2084.432856 secs 2024-04-26T12:25:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:45 smithi033 ceph-mon[27845]: pgmap v1797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T12:25:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:45 smithi096 ceph-mon[32753]: pgmap v1797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T12:25:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:45 smithi194 ceph-mon[33017]: pgmap v1797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T12:25:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:47 smithi033 ceph-mon[27845]: pgmap v1798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:47 smithi096 ceph-mon[32753]: pgmap v1798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:47 smithi194 ceph-mon[33017]: pgmap v1798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2086.712273 secs 2024-04-26T12:25:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2086.712273 secs 2024-04-26T12:25:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2086.712273 secs 2024-04-26T12:25:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:49.147+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:49.147+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:25:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:25:49.147+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:25:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2089.432991 secs 2024-04-26T12:25:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:49 smithi194 ceph-mon[33017]: pgmap v1799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2089.432991 secs 2024-04-26T12:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:49 smithi033 ceph-mon[27845]: pgmap v1799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2089.432991 secs 2024-04-26T12:25:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:49 smithi096 ceph-mon[32753]: pgmap v1799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:51 smithi033 ceph-mon[27845]: pgmap v1800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:51 smithi096 ceph-mon[32753]: pgmap v1800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:51 smithi194 ceph-mon[33017]: pgmap v1800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:25:53.672 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:53 smithi194 ceph-mon[33017]: pgmap v1801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:25:53.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2091.712373 secs 2024-04-26T12:25:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:53 smithi033 ceph-mon[27845]: pgmap v1801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:25:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2091.712373 secs 2024-04-26T12:25:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:53 smithi096 ceph-mon[32753]: pgmap v1801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:25:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2091.712373 secs 2024-04-26T12:25:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2094.433124 secs 2024-04-26T12:25:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2094.433124 secs 2024-04-26T12:25:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2094.433124 secs 2024-04-26T12:25:55.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:55 smithi096 ceph-mon[32753]: pgmap v1802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:55 smithi194 ceph-mon[33017]: pgmap v1802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:55 smithi033 ceph-mon[27845]: pgmap v1802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:57 smithi194 ceph-mon[33017]: pgmap v1803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:57 smithi033 ceph-mon[27845]: pgmap v1803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:57.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:57 smithi096 ceph-mon[32753]: pgmap v1803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:25:58.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2096.712463 secs 2024-04-26T12:25:58.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2099.433246 secs 2024-04-26T12:25:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2096.712463 secs 2024-04-26T12:25:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2099.433246 secs 2024-04-26T12:25:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2096.712463 secs 2024-04-26T12:25:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2099.433246 secs 2024-04-26T12:25:59.847 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:25:59 smithi096 ceph-mon[32753]: pgmap v1804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:25:59 smithi194 ceph-mon[33017]: pgmap v1804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:25:59.857 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:25:59 smithi033 ceph-mon[27845]: pgmap v1804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:01 smithi194 ceph-mon[33017]: pgmap v1805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:01 smithi033 ceph-mon[27845]: pgmap v1805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:01 smithi096 ceph-mon[32753]: pgmap v1805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:03 smithi194 ceph-mon[33017]: pgmap v1806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2101.712564 secs 2024-04-26T12:26:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2104.433363 secs 2024-04-26T12:26:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:03 smithi033 ceph-mon[27845]: pgmap v1806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2101.712564 secs 2024-04-26T12:26:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2104.433363 secs 2024-04-26T12:26:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:03 smithi096 ceph-mon[32753]: pgmap v1806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2101.712564 secs 2024-04-26T12:26:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2104.433363 secs 2024-04-26T12:26:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:04.164+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:04.164+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:04.164+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:26:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:05 smithi194 ceph-mon[33017]: pgmap v1807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:05 smithi033 ceph-mon[27845]: pgmap v1807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:05 smithi096 ceph-mon[32753]: pgmap v1807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:07 smithi194 ceph-mon[33017]: pgmap v1808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:07 smithi033 ceph-mon[27845]: pgmap v1808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:07 smithi096 ceph-mon[32753]: pgmap v1808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2106.712624 secs 2024-04-26T12:26:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2109.433487 secs 2024-04-26T12:26:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2106.712624 secs 2024-04-26T12:26:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2109.433487 secs 2024-04-26T12:26:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2106.712624 secs 2024-04-26T12:26:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2109.433487 secs 2024-04-26T12:26:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:09 smithi194 ceph-mon[33017]: pgmap v1809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:09 smithi033 ceph-mon[27845]: pgmap v1809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:09 smithi096 ceph-mon[32753]: pgmap v1809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:11 smithi194 ceph-mon[33017]: pgmap v1810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:11 smithi033 ceph-mon[27845]: pgmap v1810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:11 smithi096 ceph-mon[32753]: pgmap v1810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:13 smithi194 ceph-mon[33017]: pgmap v1811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2111.712710 secs 2024-04-26T12:26:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2114.433622 secs 2024-04-26T12:26:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:13 smithi033 ceph-mon[27845]: pgmap v1811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2111.712710 secs 2024-04-26T12:26:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2114.433622 secs 2024-04-26T12:26:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:13 smithi096 ceph-mon[32753]: pgmap v1811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2111.712710 secs 2024-04-26T12:26:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2114.433622 secs 2024-04-26T12:26:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:14 smithi194 ceph-mon[33017]: pgmap v1812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:14 smithi033 ceph-mon[27845]: pgmap v1812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:14 smithi096 ceph-mon[32753]: pgmap v1812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:17 smithi194 ceph-mon[33017]: pgmap v1813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:17 smithi033 ceph-mon[27845]: pgmap v1813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:17 smithi096 ceph-mon[32753]: pgmap v1813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2116.712770 secs 2024-04-26T12:26:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2116.712770 secs 2024-04-26T12:26:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2116.712770 secs 2024-04-26T12:26:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:19.179+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:19.179+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:19.179+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:26:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2119.433743 secs 2024-04-26T12:26:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:19 smithi194 ceph-mon[33017]: pgmap v1814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2119.433743 secs 2024-04-26T12:26:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:19 smithi033 ceph-mon[27845]: pgmap v1814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2119.433743 secs 2024-04-26T12:26:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:19 smithi096 ceph-mon[32753]: pgmap v1814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:21 smithi194 ceph-mon[33017]: pgmap v1815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:21.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:21 smithi033 ceph-mon[27845]: pgmap v1815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:21 smithi096 ceph-mon[32753]: pgmap v1815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40b1460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7feef0024000 tx=0x7feef0012040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40b1460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7feef0024000 tx=0x7feef0012040 comp rx=0 tx=0).stop 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40b1460 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40b1460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40b1460 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.707+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 0x7feee40af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T12:26:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.708+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 0x7feee40af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:21.710+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 0x7feee40af500 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feee80038f0 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T12:26:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:22.156+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:26:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:22.156+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:26:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:22.156+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T12:26:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:22.156+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:26:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:23 smithi194 ceph-mon[33017]: pgmap v1816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2121.712856 secs 2024-04-26T12:26:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:23 smithi033 ceph-mon[27845]: pgmap v1816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2121.712856 secs 2024-04-26T12:26:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:23 smithi096 ceph-mon[32753]: pgmap v1816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2121.712856 secs 2024-04-26T12:26:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2124.433866 secs 2024-04-26T12:26:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2124.433866 secs 2024-04-26T12:26:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2124.433866 secs 2024-04-26T12:26:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:25 smithi194 ceph-mon[33017]: pgmap v1817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:25 smithi033 ceph-mon[27845]: pgmap v1817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:25 smithi096 ceph-mon[32753]: pgmap v1817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:27 smithi194 ceph-mon[33017]: pgmap v1818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:27 smithi033 ceph-mon[27845]: pgmap v1818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:27 smithi096 ceph-mon[32753]: pgmap v1818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:28.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2126.712941 secs 2024-04-26T12:26:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2126.712941 secs 2024-04-26T12:26:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2126.712941 secs 2024-04-26T12:26:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2129.433984 secs 2024-04-26T12:26:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:29 smithi194 ceph-mon[33017]: pgmap v1819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2129.433984 secs 2024-04-26T12:26:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:29 smithi033 ceph-mon[27845]: pgmap v1819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2129.433984 secs 2024-04-26T12:26:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:29 smithi096 ceph-mon[32753]: pgmap v1819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:31 smithi033 ceph-mon[27845]: pgmap v1820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:31 smithi194 ceph-mon[33017]: pgmap v1820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:31.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:31 smithi096 ceph-mon[32753]: pgmap v1820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:33 smithi194 ceph-mon[33017]: pgmap v1821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2131.713020 secs 2024-04-26T12:26:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:33 smithi033 ceph-mon[27845]: pgmap v1821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2131.713020 secs 2024-04-26T12:26:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:33 smithi096 ceph-mon[32753]: pgmap v1821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2131.713020 secs 2024-04-26T12:26:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:34.191+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:34.191+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:26:34.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:34.191+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2134.434132 secs 2024-04-26T12:26:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2134.434132 secs 2024-04-26T12:26:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2134.434132 secs 2024-04-26T12:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:35 smithi194 ceph-mon[33017]: pgmap v1822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:35.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:35 smithi033 ceph-mon[27845]: pgmap v1822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:35 smithi096 ceph-mon[32753]: pgmap v1822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:37 smithi194 ceph-mon[33017]: pgmap v1823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:37 smithi033 ceph-mon[27845]: pgmap v1823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:37 smithi096 ceph-mon[32753]: pgmap v1823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2136.713083 secs 2024-04-26T12:26:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2136.713083 secs 2024-04-26T12:26:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2136.713083 secs 2024-04-26T12:26:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2139.434249 secs 2024-04-26T12:26:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:39 smithi194 ceph-mon[33017]: pgmap v1824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2139.434249 secs 2024-04-26T12:26:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:39 smithi033 ceph-mon[27845]: pgmap v1824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2139.434249 secs 2024-04-26T12:26:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:39 smithi096 ceph-mon[32753]: pgmap v1824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:41 smithi194 ceph-mon[33017]: pgmap v1825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:41 smithi033 ceph-mon[27845]: pgmap v1825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:41.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:41 smithi096 ceph-mon[32753]: pgmap v1825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:43 smithi194 ceph-mon[33017]: pgmap v1826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2141.713186 secs 2024-04-26T12:26:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:26:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2144.434314 secs 2024-04-26T12:26:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:43 smithi033 ceph-mon[27845]: pgmap v1826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2141.713186 secs 2024-04-26T12:26:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:26:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2144.434314 secs 2024-04-26T12:26:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:43 smithi096 ceph-mon[32753]: pgmap v1826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2141.713186 secs 2024-04-26T12:26:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:26:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2144.434314 secs 2024-04-26T12:26:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:26:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:26:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:26:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:26:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:26:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:26:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:26:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:45 smithi194 ceph-mon[33017]: pgmap v1827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:45 smithi194 ceph-mon[33017]: pgmap v1828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:45.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:45 smithi033 ceph-mon[27845]: pgmap v1827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:45.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:45 smithi033 ceph-mon[27845]: pgmap v1828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:45 smithi096 ceph-mon[32753]: pgmap v1827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:45 smithi096 ceph-mon[32753]: pgmap v1828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:46 smithi194 ceph-mon[33017]: pgmap v1829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:46 smithi033 ceph-mon[27845]: pgmap v1829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:46 smithi096 ceph-mon[32753]: pgmap v1829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2146.713248 secs 2024-04-26T12:26:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2146.713248 secs 2024-04-26T12:26:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2146.713248 secs 2024-04-26T12:26:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:48 smithi194 ceph-mon[33017]: pgmap v1830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2149.434434 secs 2024-04-26T12:26:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:48 smithi033 ceph-mon[27845]: pgmap v1830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2149.434434 secs 2024-04-26T12:26:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:48 smithi096 ceph-mon[32753]: pgmap v1830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2149.434434 secs 2024-04-26T12:26:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:49.206+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:49.206+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:26:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:26:49.206+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:26:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:50 smithi033 ceph-mon[27845]: pgmap v1831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:50 smithi096 ceph-mon[32753]: pgmap v1831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:50 smithi194 ceph-mon[33017]: pgmap v1831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:53.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:52 smithi194 ceph-mon[33017]: pgmap v1832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:53.157 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2151.713333 secs 2024-04-26T12:26:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:52 smithi033 ceph-mon[27845]: pgmap v1832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:53.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2151.713333 secs 2024-04-26T12:26:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:52 smithi096 ceph-mon[32753]: pgmap v1832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:26:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2151.713333 secs 2024-04-26T12:26:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2154.434554 secs 2024-04-26T12:26:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2154.434554 secs 2024-04-26T12:26:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2154.434554 secs 2024-04-26T12:26:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:54 smithi033 ceph-mon[27845]: pgmap v1833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:26:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:54 smithi096 ceph-mon[32753]: pgmap v1833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:26:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:54 smithi194 ceph-mon[33017]: pgmap v1833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:26:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:56 smithi033 ceph-mon[27845]: pgmap v1834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:56 smithi096 ceph-mon[32753]: pgmap v1834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:56 smithi194 ceph-mon[33017]: pgmap v1834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:26:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2156.713415 secs 2024-04-26T12:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2156.713415 secs 2024-04-26T12:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2156.713415 secs 2024-04-26T12:26:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:58 smithi194 ceph-mon[33017]: pgmap v1835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:26:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2159.434623 secs 2024-04-26T12:26:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:58 smithi033 ceph-mon[27845]: pgmap v1835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:26:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2159.434623 secs 2024-04-26T12:26:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:58 smithi096 ceph-mon[32753]: pgmap v1835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:26:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:26:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2159.434623 secs 2024-04-26T12:27:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:00 smithi194 ceph-mon[33017]: pgmap v1836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:00 smithi033 ceph-mon[27845]: pgmap v1836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:00 smithi096 ceph-mon[32753]: pgmap v1836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:02 smithi194 ceph-mon[33017]: pgmap v1837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2161.713497 secs 2024-04-26T12:27:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:02 smithi033 ceph-mon[27845]: pgmap v1837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2161.713497 secs 2024-04-26T12:27:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:02 smithi096 ceph-mon[32753]: pgmap v1837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2161.713497 secs 2024-04-26T12:27:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:04.209+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:04.210+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:04.210+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:27:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2164.434743 secs 2024-04-26T12:27:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2164.434743 secs 2024-04-26T12:27:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2164.434743 secs 2024-04-26T12:27:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:04 smithi194 ceph-mon[33017]: pgmap v1838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:04 smithi033 ceph-mon[27845]: pgmap v1838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:05.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:04 smithi096 ceph-mon[32753]: pgmap v1838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:06 smithi194 ceph-mon[33017]: pgmap v1839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:06 smithi033 ceph-mon[27845]: pgmap v1839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:06 smithi096 ceph-mon[32753]: pgmap v1839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2166.713586 secs 2024-04-26T12:27:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2166.713586 secs 2024-04-26T12:27:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2166.713586 secs 2024-04-26T12:27:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:08 smithi194 ceph-mon[33017]: pgmap v1840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2169.434858 secs 2024-04-26T12:27:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:08 smithi033 ceph-mon[27845]: pgmap v1840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2169.434858 secs 2024-04-26T12:27:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:08 smithi096 ceph-mon[32753]: pgmap v1840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2169.434858 secs 2024-04-26T12:27:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:10 smithi194 ceph-mon[33017]: pgmap v1841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:10 smithi033 ceph-mon[27845]: pgmap v1841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:10 smithi096 ceph-mon[32753]: pgmap v1841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:12 smithi194 ceph-mon[33017]: pgmap v1842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2171.713670 secs 2024-04-26T12:27:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:12 smithi033 ceph-mon[27845]: pgmap v1842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2171.713670 secs 2024-04-26T12:27:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:12 smithi096 ceph-mon[32753]: pgmap v1842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2171.713670 secs 2024-04-26T12:27:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2174.434991 secs 2024-04-26T12:27:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2174.434991 secs 2024-04-26T12:27:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2174.434991 secs 2024-04-26T12:27:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:14 smithi194 ceph-mon[33017]: pgmap v1843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:14 smithi033 ceph-mon[27845]: pgmap v1843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:14 smithi096 ceph-mon[32753]: pgmap v1843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:17 smithi194 ceph-mon[33017]: pgmap v1844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:17 smithi033 ceph-mon[27845]: pgmap v1844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:17 smithi096 ceph-mon[32753]: pgmap v1844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2176.713759 secs 2024-04-26T12:27:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2176.713759 secs 2024-04-26T12:27:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2176.713759 secs 2024-04-26T12:27:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:19.225+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:19.225+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:19.225+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:27:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:19 smithi194 ceph-mon[33017]: pgmap v1845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2179.435129 secs 2024-04-26T12:27:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:19 smithi033 ceph-mon[27845]: pgmap v1845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2179.435129 secs 2024-04-26T12:27:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:19 smithi096 ceph-mon[32753]: pgmap v1845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2179.435129 secs 2024-04-26T12:27:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:21 smithi194 ceph-mon[33017]: pgmap v1846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:21 smithi033 ceph-mon[27845]: pgmap v1846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:21 smithi096 ceph-mon[32753]: pgmap v1846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:23 smithi194 ceph-mon[33017]: pgmap v1847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2181.713852 secs 2024-04-26T12:27:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:23 smithi033 ceph-mon[27845]: pgmap v1847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2181.713852 secs 2024-04-26T12:27:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:23 smithi096 ceph-mon[32753]: pgmap v1847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2181.713852 secs 2024-04-26T12:27:24.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2184.435246 secs 2024-04-26T12:27:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2184.435246 secs 2024-04-26T12:27:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2184.435246 secs 2024-04-26T12:27:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:25 smithi194 ceph-mon[33017]: pgmap v1848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:25 smithi033 ceph-mon[27845]: pgmap v1848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:25 smithi096 ceph-mon[32753]: pgmap v1848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:27 smithi194 ceph-mon[33017]: pgmap v1849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:27 smithi033 ceph-mon[27845]: pgmap v1849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:27 smithi096 ceph-mon[32753]: pgmap v1849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:28.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2186.713940 secs 2024-04-26T12:27:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2186.713940 secs 2024-04-26T12:27:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2186.713940 secs 2024-04-26T12:27:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:29 smithi194 ceph-mon[33017]: pgmap v1850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2189.435310 secs 2024-04-26T12:27:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:29 smithi033 ceph-mon[27845]: pgmap v1850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2189.435310 secs 2024-04-26T12:27:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:29 smithi096 ceph-mon[32753]: pgmap v1850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2189.435310 secs 2024-04-26T12:27:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:31 smithi194 ceph-mon[33017]: pgmap v1851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:31 smithi033 ceph-mon[27845]: pgmap v1851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:31 smithi096 ceph-mon[32753]: pgmap v1851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:33 smithi194 ceph-mon[33017]: pgmap v1852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2191.714033 secs 2024-04-26T12:27:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:33 smithi033 ceph-mon[27845]: pgmap v1852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2191.714033 secs 2024-04-26T12:27:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:33 smithi096 ceph-mon[32753]: pgmap v1852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2191.714033 secs 2024-04-26T12:27:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:34.228+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:34.228+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:34.228+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:27:34.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2194.435434 secs 2024-04-26T12:27:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2194.435434 secs 2024-04-26T12:27:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2194.435434 secs 2024-04-26T12:27:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:35 smithi194 ceph-mon[33017]: pgmap v1853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:35 smithi033 ceph-mon[27845]: pgmap v1853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:35 smithi096 ceph-mon[32753]: pgmap v1853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:37 smithi194 ceph-mon[33017]: pgmap v1854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:37 smithi033 ceph-mon[27845]: pgmap v1854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:37 smithi096 ceph-mon[32753]: pgmap v1854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2196.714122 secs 2024-04-26T12:27:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2196.714122 secs 2024-04-26T12:27:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2196.714122 secs 2024-04-26T12:27:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:39 smithi033 ceph-mon[27845]: pgmap v1855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:39.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2199.435550 secs 2024-04-26T12:27:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:39 smithi096 ceph-mon[32753]: pgmap v1855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2199.435550 secs 2024-04-26T12:27:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:39 smithi194 ceph-mon[33017]: pgmap v1855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2199.435550 secs 2024-04-26T12:27:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:41 smithi033 ceph-mon[27845]: pgmap v1856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:41 smithi096 ceph-mon[32753]: pgmap v1856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:41 smithi194 ceph-mon[33017]: pgmap v1856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:27:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:43 smithi033 ceph-mon[27845]: pgmap v1857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2201.714212 secs 2024-04-26T12:27:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:43 smithi096 ceph-mon[32753]: pgmap v1857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2201.714212 secs 2024-04-26T12:27:43.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:43 smithi194 ceph-mon[33017]: pgmap v1857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:43.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2201.714212 secs 2024-04-26T12:27:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2204.435683 secs 2024-04-26T12:27:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2204.435683 secs 2024-04-26T12:27:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2204.435683 secs 2024-04-26T12:27:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:45 smithi033 ceph-mon[27845]: pgmap v1858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:27:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:45 smithi096 ceph-mon[32753]: pgmap v1858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:27:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:45 smithi194 ceph-mon[33017]: pgmap v1858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:27:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:27:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:27:46.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:27:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:27:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:27:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:27:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:27:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:27:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:27:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:47 smithi033 ceph-mon[27845]: pgmap v1859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:27:47.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:47 smithi033 ceph-mon[27845]: pgmap v1860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:47 smithi096 ceph-mon[32753]: pgmap v1859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:27:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:47 smithi096 ceph-mon[32753]: pgmap v1860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:47 smithi194 ceph-mon[33017]: pgmap v1859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:27:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:47 smithi194 ceph-mon[33017]: pgmap v1860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2206.714296 secs 2024-04-26T12:27:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2206.714296 secs 2024-04-26T12:27:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2206.714296 secs 2024-04-26T12:27:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:49 smithi033 ceph-mon[27845]: pgmap v1861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2209.435800 secs 2024-04-26T12:27:49.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:49.240+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:49.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:49.240+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:27:49.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:27:49.240+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:27:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:49 smithi096 ceph-mon[32753]: pgmap v1861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2209.435800 secs 2024-04-26T12:27:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:49 smithi194 ceph-mon[33017]: pgmap v1861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2209.435800 secs 2024-04-26T12:27:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:51 smithi033 ceph-mon[27845]: pgmap v1862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:51 smithi096 ceph-mon[32753]: pgmap v1862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:51 smithi194 ceph-mon[33017]: pgmap v1862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:53 smithi033 ceph-mon[27845]: pgmap v1863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2211.714379 secs 2024-04-26T12:27:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:53 smithi096 ceph-mon[32753]: pgmap v1863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2211.714379 secs 2024-04-26T12:27:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:53 smithi194 ceph-mon[33017]: pgmap v1863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2211.714379 secs 2024-04-26T12:27:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2214.435921 secs 2024-04-26T12:27:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2214.435921 secs 2024-04-26T12:27:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2214.435921 secs 2024-04-26T12:27:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:55 smithi096 ceph-mon[32753]: pgmap v1864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:55 smithi194 ceph-mon[33017]: pgmap v1864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:55 smithi033 ceph-mon[27845]: pgmap v1864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:27:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:57 smithi194 ceph-mon[33017]: pgmap v1865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:27:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:57 smithi033 ceph-mon[27845]: pgmap v1865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:27:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:57 smithi096 ceph-mon[32753]: pgmap v1865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:27:58.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2216.714464 secs 2024-04-26T12:27:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2216.714464 secs 2024-04-26T12:27:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2216.714464 secs 2024-04-26T12:27:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:59 smithi194 ceph-mon[33017]: pgmap v1866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2219.436063 secs 2024-04-26T12:27:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:59 smithi033 ceph-mon[27845]: pgmap v1866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2219.436063 secs 2024-04-26T12:27:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:59 smithi096 ceph-mon[32753]: pgmap v1866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:27:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2219.436063 secs 2024-04-26T12:28:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:01 smithi194 ceph-mon[33017]: pgmap v1867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:01 smithi033 ceph-mon[27845]: pgmap v1867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:01 smithi096 ceph-mon[32753]: pgmap v1867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:03 smithi194 ceph-mon[33017]: pgmap v1868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2221.714552 secs 2024-04-26T12:28:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:03 smithi033 ceph-mon[27845]: pgmap v1868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2221.714552 secs 2024-04-26T12:28:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:03 smithi096 ceph-mon[32753]: pgmap v1868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2221.714552 secs 2024-04-26T12:28:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:04.254+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:04.254+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:04.254+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:28:04.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2224.436179 secs 2024-04-26T12:28:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2224.436179 secs 2024-04-26T12:28:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2224.436179 secs 2024-04-26T12:28:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:05 smithi194 ceph-mon[33017]: pgmap v1869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:05 smithi033 ceph-mon[27845]: pgmap v1869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:05 smithi096 ceph-mon[32753]: pgmap v1869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:07 smithi194 ceph-mon[33017]: pgmap v1870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:07 smithi033 ceph-mon[27845]: pgmap v1870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:07 smithi096 ceph-mon[32753]: pgmap v1870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2226.714636 secs 2024-04-26T12:28:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2226.714636 secs 2024-04-26T12:28:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2226.714636 secs 2024-04-26T12:28:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:09 smithi194 ceph-mon[33017]: pgmap v1871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2229.436297 secs 2024-04-26T12:28:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:09 smithi033 ceph-mon[27845]: pgmap v1871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2229.436297 secs 2024-04-26T12:28:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:09 smithi096 ceph-mon[32753]: pgmap v1871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2229.436297 secs 2024-04-26T12:28:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:11 smithi194 ceph-mon[33017]: pgmap v1872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:11 smithi033 ceph-mon[27845]: pgmap v1872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:11 smithi096 ceph-mon[32753]: pgmap v1872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:13.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:13 smithi194 ceph-mon[33017]: pgmap v1873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:13.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2231.714742 secs 2024-04-26T12:28:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:13 smithi033 ceph-mon[27845]: pgmap v1873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:13.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2231.714742 secs 2024-04-26T12:28:13.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:13 smithi096 ceph-mon[32753]: pgmap v1873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:13.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2231.714742 secs 2024-04-26T12:28:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2234.436414 secs 2024-04-26T12:28:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2234.436414 secs 2024-04-26T12:28:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2234.436414 secs 2024-04-26T12:28:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:15 smithi194 ceph-mon[33017]: pgmap v1874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:15 smithi033 ceph-mon[27845]: pgmap v1874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:15 smithi096 ceph-mon[32753]: pgmap v1874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:17 smithi194 ceph-mon[33017]: pgmap v1875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:17 smithi033 ceph-mon[27845]: pgmap v1875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:17 smithi096 ceph-mon[32753]: pgmap v1875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2236.714829 secs 2024-04-26T12:28:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2236.714829 secs 2024-04-26T12:28:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2236.714829 secs 2024-04-26T12:28:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:19.269+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:19.269+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:19.269+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:28:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:19 smithi194 ceph-mon[33017]: pgmap v1876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2239.436477 secs 2024-04-26T12:28:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:19 smithi033 ceph-mon[27845]: pgmap v1876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2239.436477 secs 2024-04-26T12:28:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:19 smithi096 ceph-mon[32753]: pgmap v1876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2239.436477 secs 2024-04-26T12:28:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:21 smithi194 ceph-mon[33017]: pgmap v1877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:21 smithi033 ceph-mon[27845]: pgmap v1877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:21 smithi096 ceph-mon[32753]: pgmap v1877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:23 smithi194 ceph-mon[33017]: pgmap v1878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2241.714916 secs 2024-04-26T12:28:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:23 smithi033 ceph-mon[27845]: pgmap v1878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2241.714916 secs 2024-04-26T12:28:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:23 smithi096 ceph-mon[32753]: pgmap v1878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2241.714916 secs 2024-04-26T12:28:24.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2244.436541 secs 2024-04-26T12:28:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2244.436541 secs 2024-04-26T12:28:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2244.436541 secs 2024-04-26T12:28:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:25 smithi194 ceph-mon[33017]: pgmap v1879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:25 smithi033 ceph-mon[27845]: pgmap v1879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:25 smithi096 ceph-mon[32753]: pgmap v1879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:27 smithi194 ceph-mon[33017]: pgmap v1880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:27 smithi033 ceph-mon[27845]: pgmap v1880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:27 smithi096 ceph-mon[32753]: pgmap v1880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:28.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2246.715006 secs 2024-04-26T12:28:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2246.715006 secs 2024-04-26T12:28:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2246.715006 secs 2024-04-26T12:28:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:29 smithi194 ceph-mon[33017]: pgmap v1881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2249.436671 secs 2024-04-26T12:28:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:29 smithi033 ceph-mon[27845]: pgmap v1881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2249.436671 secs 2024-04-26T12:28:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:29 smithi096 ceph-mon[32753]: pgmap v1881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2249.436671 secs 2024-04-26T12:28:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:31 smithi194 ceph-mon[33017]: pgmap v1882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:31 smithi033 ceph-mon[27845]: pgmap v1882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:31.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:31 smithi096 ceph-mon[32753]: pgmap v1882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:33 smithi194 ceph-mon[33017]: pgmap v1883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:33.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2251.715097 secs 2024-04-26T12:28:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:33 smithi033 ceph-mon[27845]: pgmap v1883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2251.715097 secs 2024-04-26T12:28:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:33 smithi096 ceph-mon[32753]: pgmap v1883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2251.715097 secs 2024-04-26T12:28:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:34.270+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:34.270+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:34.270+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:28:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2254.436789 secs 2024-04-26T12:28:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2254.436789 secs 2024-04-26T12:28:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2254.436789 secs 2024-04-26T12:28:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:35 smithi194 ceph-mon[33017]: pgmap v1884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:35 smithi033 ceph-mon[27845]: pgmap v1884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:35 smithi096 ceph-mon[32753]: pgmap v1884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:37 smithi194 ceph-mon[33017]: pgmap v1885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:37 smithi033 ceph-mon[27845]: pgmap v1885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:37 smithi096 ceph-mon[32753]: pgmap v1885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2256.715161 secs 2024-04-26T12:28:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2256.715161 secs 2024-04-26T12:28:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2256.715161 secs 2024-04-26T12:28:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:39 smithi033 ceph-mon[27845]: pgmap v1886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2259.436870 secs 2024-04-26T12:28:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:39 smithi096 ceph-mon[32753]: pgmap v1886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2259.436870 secs 2024-04-26T12:28:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:39 smithi194 ceph-mon[33017]: pgmap v1886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:39.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2259.436870 secs 2024-04-26T12:28:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:41 smithi033 ceph-mon[27845]: pgmap v1887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:41.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:41 smithi096 ceph-mon[32753]: pgmap v1887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:41 smithi194 ceph-mon[33017]: pgmap v1887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:43 smithi033 ceph-mon[27845]: pgmap v1888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2261.715269 secs 2024-04-26T12:28:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:43 smithi096 ceph-mon[32753]: pgmap v1888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2261.715269 secs 2024-04-26T12:28:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:43 smithi194 ceph-mon[33017]: pgmap v1888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2261.715269 secs 2024-04-26T12:28:44.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2264.437005 secs 2024-04-26T12:28:44.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2264.437005 secs 2024-04-26T12:28:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2264.437005 secs 2024-04-26T12:28:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:45 smithi033 ceph-mon[27845]: pgmap v1889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:45 smithi096 ceph-mon[32753]: pgmap v1889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:45 smithi194 ceph-mon[33017]: pgmap v1889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:28:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:28:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:28:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:47 smithi033 ceph-mon[27845]: pgmap v1890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:28:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:28:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:28:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:47 smithi096 ceph-mon[32753]: pgmap v1890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:28:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:28:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:28:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:47 smithi194 ceph-mon[33017]: pgmap v1890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:28:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:28:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:28:47.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:28:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:48 smithi033 ceph-mon[27845]: pgmap v1891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:48 smithi033 ceph-mon[27845]: pgmap v1892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2266.715332 secs 2024-04-26T12:28:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:48 smithi096 ceph-mon[32753]: pgmap v1891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:48 smithi096 ceph-mon[32753]: pgmap v1892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2266.715332 secs 2024-04-26T12:28:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:48 smithi194 ceph-mon[33017]: pgmap v1891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:28:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:48 smithi194 ceph-mon[33017]: pgmap v1892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2266.715332 secs 2024-04-26T12:28:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:49.283+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:49.283+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:28:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:28:49.283+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:28:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2269.437175 secs 2024-04-26T12:28:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2269.437175 secs 2024-04-26T12:28:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2269.437175 secs 2024-04-26T12:28:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:50 smithi033 ceph-mon[27845]: pgmap v1893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:28:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:50 smithi096 ceph-mon[32753]: pgmap v1893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:28:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:50 smithi194 ceph-mon[33017]: pgmap v1893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:28:52.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:52 smithi033 ceph-mon[27845]: pgmap v1894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:52 smithi096 ceph-mon[32753]: pgmap v1894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:52 smithi194 ceph-mon[33017]: pgmap v1894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:53.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2271.715435 secs 2024-04-26T12:28:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2271.715435 secs 2024-04-26T12:28:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2271.715435 secs 2024-04-26T12:28:54.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2274.437298 secs 2024-04-26T12:28:54.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:54 smithi033 ceph-mon[27845]: pgmap v1895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2274.437298 secs 2024-04-26T12:28:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:54 smithi194 ceph-mon[33017]: pgmap v1895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2274.437298 secs 2024-04-26T12:28:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:54 smithi096 ceph-mon[32753]: pgmap v1895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:56 smithi194 ceph-mon[33017]: pgmap v1896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:56 smithi033 ceph-mon[27845]: pgmap v1896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:56 smithi096 ceph-mon[32753]: pgmap v1896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:28:58.735 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:58 smithi096 ceph-mon[32753]: pgmap v1897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:28:58.735 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2276.715536 secs 2024-04-26T12:28:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:58 smithi194 ceph-mon[33017]: pgmap v1897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:28:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2276.715536 secs 2024-04-26T12:28:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:58 smithi033 ceph-mon[27845]: pgmap v1897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:28:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2276.715536 secs 2024-04-26T12:28:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:28:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2279.437375 secs 2024-04-26T12:28:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:28:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2279.437375 secs 2024-04-26T12:28:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:28:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2279.437375 secs 2024-04-26T12:29:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:00 smithi194 ceph-mon[33017]: pgmap v1898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:00 smithi033 ceph-mon[27845]: pgmap v1898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:00 smithi096 ceph-mon[32753]: pgmap v1898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:02 smithi194 ceph-mon[33017]: pgmap v1899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:02 smithi033 ceph-mon[27845]: pgmap v1899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:02 smithi096 ceph-mon[32753]: pgmap v1899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2281.715637 secs 2024-04-26T12:29:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2284.437502 secs 2024-04-26T12:29:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2281.715637 secs 2024-04-26T12:29:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2284.437502 secs 2024-04-26T12:29:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2281.715637 secs 2024-04-26T12:29:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2284.437502 secs 2024-04-26T12:29:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:04.286+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:04.286+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:04.286+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:29:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:04 smithi194 ceph-mon[33017]: pgmap v1900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:04 smithi033 ceph-mon[27845]: pgmap v1900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:04 smithi096 ceph-mon[32753]: pgmap v1900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:06 smithi194 ceph-mon[33017]: pgmap v1901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:06 smithi033 ceph-mon[27845]: pgmap v1901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:06 smithi096 ceph-mon[32753]: pgmap v1901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:08 smithi194 ceph-mon[33017]: pgmap v1902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2286.715733 secs 2024-04-26T12:29:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2289.437638 secs 2024-04-26T12:29:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:08 smithi033 ceph-mon[27845]: pgmap v1902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2286.715733 secs 2024-04-26T12:29:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2289.437638 secs 2024-04-26T12:29:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:08 smithi096 ceph-mon[32753]: pgmap v1902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2286.715733 secs 2024-04-26T12:29:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2289.437638 secs 2024-04-26T12:29:10.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:10 smithi194 ceph-mon[33017]: pgmap v1903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:10.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:10 smithi033 ceph-mon[27845]: pgmap v1903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:10 smithi096 ceph-mon[32753]: pgmap v1903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:12 smithi194 ceph-mon[33017]: pgmap v1904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:12 smithi033 ceph-mon[27845]: pgmap v1904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:12 smithi096 ceph-mon[32753]: pgmap v1904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2291.715835 secs 2024-04-26T12:29:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2294.437762 secs 2024-04-26T12:29:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:13 smithi194 ceph-mon[33017]: pgmap v1905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2291.715835 secs 2024-04-26T12:29:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2294.437762 secs 2024-04-26T12:29:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:13 smithi033 ceph-mon[27845]: pgmap v1905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2291.715835 secs 2024-04-26T12:29:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2294.437762 secs 2024-04-26T12:29:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:13 smithi096 ceph-mon[32753]: pgmap v1905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:16 smithi194 ceph-mon[33017]: pgmap v1906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:16 smithi033 ceph-mon[27845]: pgmap v1906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:16 smithi096 ceph-mon[32753]: pgmap v1906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:18 smithi194 ceph-mon[33017]: pgmap v1907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2296.715924 secs 2024-04-26T12:29:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:18 smithi033 ceph-mon[27845]: pgmap v1907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2296.715924 secs 2024-04-26T12:29:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:18 smithi096 ceph-mon[32753]: pgmap v1907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2296.715924 secs 2024-04-26T12:29:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:19.297+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:19.297+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:19.297+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:29:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2299.437902 secs 2024-04-26T12:29:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2299.437902 secs 2024-04-26T12:29:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2299.437902 secs 2024-04-26T12:29:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:20 smithi194 ceph-mon[33017]: pgmap v1908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:20 smithi033 ceph-mon[27845]: pgmap v1908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:20 smithi096 ceph-mon[32753]: pgmap v1908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:22.186+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:29:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:22.186+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:29:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:22.186+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:29:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:22.187+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:29:22.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:22 smithi033 ceph-mon[27845]: pgmap v1909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:22 smithi194 ceph-mon[33017]: pgmap v1909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:22 smithi096 ceph-mon[32753]: pgmap v1909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2301.716017 secs 2024-04-26T12:29:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2301.716017 secs 2024-04-26T12:29:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2301.716017 secs 2024-04-26T12:29:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2304.438022 secs 2024-04-26T12:29:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:24 smithi194 ceph-mon[33017]: pgmap v1910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2304.438022 secs 2024-04-26T12:29:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:24 smithi033 ceph-mon[27845]: pgmap v1910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2304.438022 secs 2024-04-26T12:29:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:24 smithi096 ceph-mon[32753]: pgmap v1910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:26 smithi033 ceph-mon[27845]: pgmap v1911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:26 smithi096 ceph-mon[32753]: pgmap v1911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:26 smithi194 ceph-mon[33017]: pgmap v1911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:28 smithi033 ceph-mon[27845]: pgmap v1912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2306.716120 secs 2024-04-26T12:29:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:28 smithi096 ceph-mon[32753]: pgmap v1912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2306.716120 secs 2024-04-26T12:29:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:28 smithi194 ceph-mon[33017]: pgmap v1912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2306.716120 secs 2024-04-26T12:29:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2309.438164 secs 2024-04-26T12:29:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2309.438164 secs 2024-04-26T12:29:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2309.438164 secs 2024-04-26T12:29:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:30 smithi033 ceph-mon[27845]: pgmap v1913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:30 smithi096 ceph-mon[32753]: pgmap v1913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:30 smithi194 ceph-mon[33017]: pgmap v1913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:32 smithi033 ceph-mon[27845]: pgmap v1914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:32.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:32 smithi096 ceph-mon[32753]: pgmap v1914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:32 smithi194 ceph-mon[33017]: pgmap v1914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2311.716179 secs 2024-04-26T12:29:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2311.716179 secs 2024-04-26T12:29:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2311.716179 secs 2024-04-26T12:29:34.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:34.313+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:34.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:34.313+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:34.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:34.313+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:29:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2314.438297 secs 2024-04-26T12:29:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:34 smithi033 ceph-mon[27845]: pgmap v1915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2314.438297 secs 2024-04-26T12:29:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:34 smithi096 ceph-mon[32753]: pgmap v1915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2314.438297 secs 2024-04-26T12:29:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:34 smithi194 ceph-mon[33017]: pgmap v1915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:36 smithi033 ceph-mon[27845]: pgmap v1916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:36.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:36 smithi096 ceph-mon[32753]: pgmap v1916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:36 smithi194 ceph-mon[33017]: pgmap v1916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:38 smithi033 ceph-mon[27845]: pgmap v1917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2316.716242 secs 2024-04-26T12:29:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:38 smithi096 ceph-mon[32753]: pgmap v1917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2316.716242 secs 2024-04-26T12:29:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:38 smithi194 ceph-mon[33017]: pgmap v1917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2316.716242 secs 2024-04-26T12:29:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2319.438440 secs 2024-04-26T12:29:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2319.438440 secs 2024-04-26T12:29:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2319.438440 secs 2024-04-26T12:29:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:40 smithi033 ceph-mon[27845]: pgmap v1918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:40 smithi096 ceph-mon[32753]: pgmap v1918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:40 smithi194 ceph-mon[33017]: pgmap v1918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:42 smithi033 ceph-mon[27845]: pgmap v1919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:42 smithi096 ceph-mon[32753]: pgmap v1919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:42 smithi194 ceph-mon[33017]: pgmap v1919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2321.716329 secs 2024-04-26T12:29:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2321.716329 secs 2024-04-26T12:29:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2321.716329 secs 2024-04-26T12:29:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2324.438566 secs 2024-04-26T12:29:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:44 smithi194 ceph-mon[33017]: pgmap v1920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2324.438566 secs 2024-04-26T12:29:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:44 smithi033 ceph-mon[27845]: pgmap v1920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2324.438566 secs 2024-04-26T12:29:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:44 smithi096 ceph-mon[32753]: pgmap v1920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:46 smithi194 ceph-mon[33017]: pgmap v1921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:46 smithi033 ceph-mon[27845]: pgmap v1921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:46 smithi096 ceph-mon[32753]: pgmap v1921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:29:48.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:48 smithi194 ceph-mon[33017]: pgmap v1922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:48.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2326.716414 secs 2024-04-26T12:29:48.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:29:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:48 smithi033 ceph-mon[27845]: pgmap v1922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2326.716414 secs 2024-04-26T12:29:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:29:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:48 smithi096 ceph-mon[32753]: pgmap v1922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2326.716414 secs 2024-04-26T12:29:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:29:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:49.323+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:49.323+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:29:49.326 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:29:49.323+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2329.438671 secs 2024-04-26T12:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:29:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2329.438671 secs 2024-04-26T12:29:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:29:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:29:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:29:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2329.438671 secs 2024-04-26T12:29:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:29:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:29:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:29:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:50 smithi194 ceph-mon[33017]: pgmap v1923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:50 smithi194 ceph-mon[33017]: pgmap v1924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:50 smithi033 ceph-mon[27845]: pgmap v1923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:50 smithi033 ceph-mon[27845]: pgmap v1924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:50 smithi096 ceph-mon[32753]: pgmap v1923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:29:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:50 smithi096 ceph-mon[32753]: pgmap v1924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:52.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:52 smithi194 ceph-mon[33017]: pgmap v1925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:52 smithi033 ceph-mon[27845]: pgmap v1925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:52 smithi096 ceph-mon[32753]: pgmap v1925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2331.716503 secs 2024-04-26T12:29:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2334.438795 secs 2024-04-26T12:29:53.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2331.716503 secs 2024-04-26T12:29:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2334.438795 secs 2024-04-26T12:29:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2331.716503 secs 2024-04-26T12:29:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2334.438795 secs 2024-04-26T12:29:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:54 smithi194 ceph-mon[33017]: pgmap v1926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:54 smithi033 ceph-mon[27845]: pgmap v1926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:54 smithi096 ceph-mon[32753]: pgmap v1926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:56 smithi194 ceph-mon[33017]: pgmap v1927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:56 smithi033 ceph-mon[27845]: pgmap v1927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:56 smithi096 ceph-mon[32753]: pgmap v1927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:58 smithi194 ceph-mon[33017]: pgmap v1928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2336.716585 secs 2024-04-26T12:29:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:29:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2339.438918 secs 2024-04-26T12:29:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:58 smithi033 ceph-mon[27845]: pgmap v1928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2336.716585 secs 2024-04-26T12:29:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:29:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2339.438918 secs 2024-04-26T12:29:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:58 smithi096 ceph-mon[32753]: pgmap v1928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:29:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2336.716585 secs 2024-04-26T12:29:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:29:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2339.438918 secs 2024-04-26T12:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:00 smithi194 ceph-mon[33017]: pgmap v1929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:00 smithi033 ceph-mon[27845]: pgmap v1929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:30:00.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:30:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:00 smithi096 ceph-mon[32753]: pgmap v1929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:30:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:30:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:02 smithi194 ceph-mon[33017]: pgmap v1930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:02 smithi033 ceph-mon[27845]: pgmap v1930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:02 smithi096 ceph-mon[32753]: pgmap v1930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2341.716672 secs 2024-04-26T12:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2344.439016 secs 2024-04-26T12:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2341.716672 secs 2024-04-26T12:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2344.439016 secs 2024-04-26T12:30:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2341.716672 secs 2024-04-26T12:30:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2344.439016 secs 2024-04-26T12:30:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:04.340+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:04.340+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:04.340+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:30:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:04 smithi194 ceph-mon[33017]: pgmap v1931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:04 smithi033 ceph-mon[27845]: pgmap v1931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:04 smithi096 ceph-mon[32753]: pgmap v1931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:06 smithi194 ceph-mon[33017]: pgmap v1932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:06 smithi033 ceph-mon[27845]: pgmap v1932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:06 smithi096 ceph-mon[32753]: pgmap v1932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:08 smithi194 ceph-mon[33017]: pgmap v1933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2346.716755 secs 2024-04-26T12:30:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2349.439148 secs 2024-04-26T12:30:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:08 smithi033 ceph-mon[27845]: pgmap v1933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2346.716755 secs 2024-04-26T12:30:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2349.439148 secs 2024-04-26T12:30:08.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:08 smithi096 ceph-mon[32753]: pgmap v1933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:08.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2346.716755 secs 2024-04-26T12:30:08.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2349.439148 secs 2024-04-26T12:30:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:10 smithi194 ceph-mon[33017]: pgmap v1934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:10 smithi033 ceph-mon[27845]: pgmap v1934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:10 smithi096 ceph-mon[32753]: pgmap v1934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:11 smithi194 ceph-mon[33017]: pgmap v1935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:11 smithi033 ceph-mon[27845]: pgmap v1935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:11 smithi096 ceph-mon[32753]: pgmap v1935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2351.716840 secs 2024-04-26T12:30:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2351.716840 secs 2024-04-26T12:30:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2351.716840 secs 2024-04-26T12:30:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:13 smithi194 ceph-mon[33017]: pgmap v1936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2354.439189 secs 2024-04-26T12:30:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:13 smithi033 ceph-mon[27845]: pgmap v1936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2354.439189 secs 2024-04-26T12:30:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:13 smithi096 ceph-mon[32753]: pgmap v1936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2354.439189 secs 2024-04-26T12:30:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:15 smithi194 ceph-mon[33017]: pgmap v1937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:15 smithi033 ceph-mon[27845]: pgmap v1937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:15 smithi096 ceph-mon[32753]: pgmap v1937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:17 smithi194 ceph-mon[33017]: pgmap v1938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2356.716922 secs 2024-04-26T12:30:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:17 smithi033 ceph-mon[27845]: pgmap v1938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2356.716922 secs 2024-04-26T12:30:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:17 smithi096 ceph-mon[32753]: pgmap v1938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2356.716922 secs 2024-04-26T12:30:19.347 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:19.346+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:19.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:19.346+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:19.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:19.346+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:30:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2359.439270 secs 2024-04-26T12:30:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2359.439270 secs 2024-04-26T12:30:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2359.439270 secs 2024-04-26T12:30:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:20 smithi194 ceph-mon[33017]: pgmap v1939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:20 smithi033 ceph-mon[27845]: pgmap v1939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:20 smithi096 ceph-mon[32753]: pgmap v1939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:22 smithi194 ceph-mon[33017]: pgmap v1940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:22 smithi033 ceph-mon[27845]: pgmap v1940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:22 smithi096 ceph-mon[32753]: pgmap v1940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2361.717010 secs 2024-04-26T12:30:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2361.717010 secs 2024-04-26T12:30:23.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2361.717010 secs 2024-04-26T12:30:24.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:24 smithi033 ceph-mon[27845]: pgmap v1941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:24.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2364.439390 secs 2024-04-26T12:30:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:24 smithi194 ceph-mon[33017]: pgmap v1941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2364.439390 secs 2024-04-26T12:30:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:24 smithi096 ceph-mon[32753]: pgmap v1941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2364.439390 secs 2024-04-26T12:30:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:26 smithi194 ceph-mon[33017]: pgmap v1942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:26 smithi033 ceph-mon[27845]: pgmap v1942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:26 smithi096 ceph-mon[32753]: pgmap v1942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:28 smithi194 ceph-mon[33017]: pgmap v1943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2366.717099 secs 2024-04-26T12:30:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:28 smithi033 ceph-mon[27845]: pgmap v1943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2366.717099 secs 2024-04-26T12:30:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:28 smithi096 ceph-mon[32753]: pgmap v1943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2366.717099 secs 2024-04-26T12:30:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2369.439458 secs 2024-04-26T12:30:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2369.439458 secs 2024-04-26T12:30:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2369.439458 secs 2024-04-26T12:30:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:30 smithi194 ceph-mon[33017]: pgmap v1944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:30 smithi033 ceph-mon[27845]: pgmap v1944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:30 smithi096 ceph-mon[32753]: pgmap v1944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:32 smithi194 ceph-mon[33017]: pgmap v1945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:32 smithi033 ceph-mon[27845]: pgmap v1945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:32 smithi096 ceph-mon[32753]: pgmap v1945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2371.717187 secs 2024-04-26T12:30:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2371.717187 secs 2024-04-26T12:30:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2371.717187 secs 2024-04-26T12:30:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:34 smithi194 ceph-mon[33017]: pgmap v1946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2374.439601 secs 2024-04-26T12:30:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:34.359+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:34.359+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:30:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:34.359+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:34 smithi033 ceph-mon[27845]: pgmap v1946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2374.439601 secs 2024-04-26T12:30:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:34 smithi096 ceph-mon[32753]: pgmap v1946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2374.439601 secs 2024-04-26T12:30:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:36 smithi194 ceph-mon[33017]: pgmap v1947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:36 smithi033 ceph-mon[27845]: pgmap v1947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:36 smithi096 ceph-mon[32753]: pgmap v1947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:38 smithi194 ceph-mon[33017]: pgmap v1948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2376.717275 secs 2024-04-26T12:30:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:38 smithi033 ceph-mon[27845]: pgmap v1948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2376.717275 secs 2024-04-26T12:30:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:38 smithi096 ceph-mon[32753]: pgmap v1948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2376.717275 secs 2024-04-26T12:30:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2379.439717 secs 2024-04-26T12:30:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2379.439717 secs 2024-04-26T12:30:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2379.439717 secs 2024-04-26T12:30:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:40 smithi033 ceph-mon[27845]: pgmap v1949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:40 smithi194 ceph-mon[33017]: pgmap v1949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:40 smithi096 ceph-mon[32753]: pgmap v1949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:42 smithi194 ceph-mon[33017]: pgmap v1950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:42 smithi033 ceph-mon[27845]: pgmap v1950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:42 smithi096 ceph-mon[32753]: pgmap v1950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2381.717337 secs 2024-04-26T12:30:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2381.717337 secs 2024-04-26T12:30:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2381.717337 secs 2024-04-26T12:30:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:44 smithi033 ceph-mon[27845]: pgmap v1951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2384.439832 secs 2024-04-26T12:30:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:44 smithi096 ceph-mon[32753]: pgmap v1951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2384.439832 secs 2024-04-26T12:30:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:44 smithi194 ceph-mon[33017]: pgmap v1951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2384.439832 secs 2024-04-26T12:30:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:46 smithi033 ceph-mon[27845]: pgmap v1952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:46 smithi096 ceph-mon[32753]: pgmap v1952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:46 smithi194 ceph-mon[33017]: pgmap v1952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:48 smithi033 ceph-mon[27845]: pgmap v1953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2386.717400 secs 2024-04-26T12:30:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:48 smithi096 ceph-mon[32753]: pgmap v1953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2386.717400 secs 2024-04-26T12:30:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:48 smithi194 ceph-mon[33017]: pgmap v1953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2386.717400 secs 2024-04-26T12:30:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:49.373+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:49.373+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:30:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:49.373+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:30:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2389.439966 secs 2024-04-26T12:30:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2389.439966 secs 2024-04-26T12:30:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2389.439966 secs 2024-04-26T12:30:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:50 smithi033 ceph-mon[27845]: pgmap v1954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:50.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:30:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:50 smithi096 ceph-mon[32753]: pgmap v1954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:30:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:50 smithi194 ceph-mon[33017]: pgmap v1954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:30:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:51.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:52.199+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:30:52.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:52.199+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:30:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:52.200+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:30:52.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:30:52.200+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:30:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:52 smithi194 ceph-mon[33017]: pgmap v1955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:30:52.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:30:52.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:52 smithi033 ceph-mon[27845]: pgmap v1955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:30:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:30:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:52 smithi096 ceph-mon[32753]: pgmap v1955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:30:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:30:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:30:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:30:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:53 smithi194 ceph-mon[33017]: pgmap v1956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:53 smithi194 ceph-mon[33017]: pgmap v1957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2391.717461 secs 2024-04-26T12:30:53.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2394.440101 secs 2024-04-26T12:30:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:53 smithi033 ceph-mon[27845]: pgmap v1956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:53 smithi033 ceph-mon[27845]: pgmap v1957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2391.717461 secs 2024-04-26T12:30:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2394.440101 secs 2024-04-26T12:30:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:53 smithi096 ceph-mon[32753]: pgmap v1956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:30:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:53 smithi096 ceph-mon[32753]: pgmap v1957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2391.717461 secs 2024-04-26T12:30:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2394.440101 secs 2024-04-26T12:30:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:54 smithi194 ceph-mon[33017]: pgmap v1958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:54 smithi033 ceph-mon[27845]: pgmap v1958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:54 smithi096 ceph-mon[32753]: pgmap v1958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:57 smithi194 ceph-mon[33017]: pgmap v1959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:57 smithi033 ceph-mon[27845]: pgmap v1959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:57 smithi096 ceph-mon[32753]: pgmap v1959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:58.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2396.717543 secs 2024-04-26T12:30:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2396.717543 secs 2024-04-26T12:30:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2396.717543 secs 2024-04-26T12:30:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2399.440216 secs 2024-04-26T12:30:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:30:59 smithi194 ceph-mon[33017]: pgmap v1960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2399.440216 secs 2024-04-26T12:30:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:30:59 smithi033 ceph-mon[27845]: pgmap v1960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:30:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2399.440216 secs 2024-04-26T12:30:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:30:59 smithi096 ceph-mon[32753]: pgmap v1960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:01 smithi194 ceph-mon[33017]: pgmap v1961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:01.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:01 smithi033 ceph-mon[27845]: pgmap v1961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:01 smithi096 ceph-mon[32753]: pgmap v1961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:03 smithi194 ceph-mon[33017]: pgmap v1962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:31:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2401.717633 secs 2024-04-26T12:31:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2404.440305 secs 2024-04-26T12:31:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:03 smithi033 ceph-mon[27845]: pgmap v1962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:31:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2401.717633 secs 2024-04-26T12:31:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2404.440305 secs 2024-04-26T12:31:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:03 smithi096 ceph-mon[32753]: pgmap v1962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:31:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2401.717633 secs 2024-04-26T12:31:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2404.440305 secs 2024-04-26T12:31:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:04.389+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:04.389+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:04.389+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:31:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:05 smithi194 ceph-mon[33017]: pgmap v1963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:05 smithi033 ceph-mon[27845]: pgmap v1963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:05 smithi096 ceph-mon[32753]: pgmap v1963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:07 smithi194 ceph-mon[33017]: pgmap v1964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:07 smithi033 ceph-mon[27845]: pgmap v1964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:07 smithi096 ceph-mon[32753]: pgmap v1964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2406.717694 secs 2024-04-26T12:31:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2409.440450 secs 2024-04-26T12:31:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2406.717694 secs 2024-04-26T12:31:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2409.440450 secs 2024-04-26T12:31:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2406.717694 secs 2024-04-26T12:31:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2409.440450 secs 2024-04-26T12:31:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:09 smithi194 ceph-mon[33017]: pgmap v1965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:09 smithi033 ceph-mon[27845]: pgmap v1965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:09 smithi096 ceph-mon[32753]: pgmap v1965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:11 smithi194 ceph-mon[33017]: pgmap v1966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:11 smithi033 ceph-mon[27845]: pgmap v1966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:11 smithi096 ceph-mon[32753]: pgmap v1966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:13 smithi194 ceph-mon[33017]: pgmap v1967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2411.717778 secs 2024-04-26T12:31:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2414.440588 secs 2024-04-26T12:31:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:13 smithi033 ceph-mon[27845]: pgmap v1967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2411.717778 secs 2024-04-26T12:31:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2414.440588 secs 2024-04-26T12:31:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:13 smithi096 ceph-mon[32753]: pgmap v1967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2411.717778 secs 2024-04-26T12:31:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2414.440588 secs 2024-04-26T12:31:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:15 smithi194 ceph-mon[33017]: pgmap v1968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:15 smithi033 ceph-mon[27845]: pgmap v1968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:15 smithi096 ceph-mon[32753]: pgmap v1968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:17 smithi194 ceph-mon[33017]: pgmap v1969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:17 smithi033 ceph-mon[27845]: pgmap v1969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:17 smithi096 ceph-mon[32753]: pgmap v1969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2416.717861 secs 2024-04-26T12:31:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2419.440721 secs 2024-04-26T12:31:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2416.717861 secs 2024-04-26T12:31:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2419.440721 secs 2024-04-26T12:31:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2416.717861 secs 2024-04-26T12:31:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2419.440721 secs 2024-04-26T12:31:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:19.405+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:19.405+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:19.405+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:31:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:19 smithi194 ceph-mon[33017]: pgmap v1970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:19 smithi033 ceph-mon[27845]: pgmap v1970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:19 smithi096 ceph-mon[32753]: pgmap v1970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:21 smithi194 ceph-mon[33017]: pgmap v1971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:21 smithi033 ceph-mon[27845]: pgmap v1971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:21 smithi096 ceph-mon[32753]: pgmap v1971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:23 smithi194 ceph-mon[33017]: pgmap v1972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2421.717948 secs 2024-04-26T12:31:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2424.440858 secs 2024-04-26T12:31:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:23 smithi033 ceph-mon[27845]: pgmap v1972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2421.717948 secs 2024-04-26T12:31:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2424.440858 secs 2024-04-26T12:31:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:23 smithi096 ceph-mon[32753]: pgmap v1972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2421.717948 secs 2024-04-26T12:31:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2424.440858 secs 2024-04-26T12:31:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:25 smithi194 ceph-mon[33017]: pgmap v1973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:25 smithi033 ceph-mon[27845]: pgmap v1973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:25 smithi096 ceph-mon[32753]: pgmap v1973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:27 smithi194 ceph-mon[33017]: pgmap v1974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:27 smithi033 ceph-mon[27845]: pgmap v1974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:27 smithi096 ceph-mon[32753]: pgmap v1974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:28.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2426.718037 secs 2024-04-26T12:31:28.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2429.441009 secs 2024-04-26T12:31:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2426.718037 secs 2024-04-26T12:31:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2429.441009 secs 2024-04-26T12:31:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2426.718037 secs 2024-04-26T12:31:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2429.441009 secs 2024-04-26T12:31:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:29 smithi194 ceph-mon[33017]: pgmap v1975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:29 smithi033 ceph-mon[27845]: pgmap v1975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:29 smithi096 ceph-mon[32753]: pgmap v1975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:30 smithi194 ceph-mon[33017]: pgmap v1976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:30 smithi033 ceph-mon[27845]: pgmap v1976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:30 smithi096 ceph-mon[32753]: pgmap v1976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:33 smithi194 ceph-mon[33017]: pgmap v1977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2431.718096 secs 2024-04-26T12:31:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:33 smithi033 ceph-mon[27845]: pgmap v1977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2431.718096 secs 2024-04-26T12:31:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:33 smithi096 ceph-mon[32753]: pgmap v1977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2431.718096 secs 2024-04-26T12:31:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:34.416+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:34.416+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:34.416+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:31:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2434.441141 secs 2024-04-26T12:31:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2434.441141 secs 2024-04-26T12:31:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2434.441141 secs 2024-04-26T12:31:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:35 smithi194 ceph-mon[33017]: pgmap v1978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:35 smithi033 ceph-mon[27845]: pgmap v1978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:35 smithi096 ceph-mon[32753]: pgmap v1978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:37 smithi194 ceph-mon[33017]: pgmap v1979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:37 smithi033 ceph-mon[27845]: pgmap v1979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:37 smithi096 ceph-mon[32753]: pgmap v1979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2436.718181 secs 2024-04-26T12:31:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2439.441259 secs 2024-04-26T12:31:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2436.718181 secs 2024-04-26T12:31:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2439.441259 secs 2024-04-26T12:31:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2436.718181 secs 2024-04-26T12:31:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2439.441259 secs 2024-04-26T12:31:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:39 smithi194 ceph-mon[33017]: pgmap v1980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:39 smithi033 ceph-mon[27845]: pgmap v1980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:39 smithi096 ceph-mon[32753]: pgmap v1980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:41 smithi194 ceph-mon[33017]: pgmap v1981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:41.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:41 smithi033 ceph-mon[27845]: pgmap v1981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:41.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:41 smithi096 ceph-mon[32753]: pgmap v1981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:43 smithi194 ceph-mon[33017]: pgmap v1982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2441.718264 secs 2024-04-26T12:31:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2444.441381 secs 2024-04-26T12:31:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:43 smithi033 ceph-mon[27845]: pgmap v1982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2441.718264 secs 2024-04-26T12:31:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2444.441381 secs 2024-04-26T12:31:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:43 smithi096 ceph-mon[32753]: pgmap v1982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2441.718264 secs 2024-04-26T12:31:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2444.441381 secs 2024-04-26T12:31:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:45 smithi194 ceph-mon[33017]: pgmap v1983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:45 smithi033 ceph-mon[27845]: pgmap v1983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:45 smithi096 ceph-mon[32753]: pgmap v1983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:47 smithi194 ceph-mon[33017]: pgmap v1984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:47 smithi033 ceph-mon[27845]: pgmap v1984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:47 smithi096 ceph-mon[32753]: pgmap v1984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2446.718325 secs 2024-04-26T12:31:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2449.441501 secs 2024-04-26T12:31:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2446.718325 secs 2024-04-26T12:31:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2449.441501 secs 2024-04-26T12:31:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2446.718325 secs 2024-04-26T12:31:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2449.441501 secs 2024-04-26T12:31:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:49.431+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:49.431+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:31:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:31:49.431+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:31:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:49 smithi194 ceph-mon[33017]: pgmap v1985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:49 smithi033 ceph-mon[27845]: pgmap v1985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:49 smithi096 ceph-mon[32753]: pgmap v1985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:51 smithi194 ceph-mon[33017]: pgmap v1986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:51 smithi033 ceph-mon[27845]: pgmap v1986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:51 smithi096 ceph-mon[32753]: pgmap v1986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:31:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:53 smithi194 ceph-mon[33017]: pgmap v1987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:53.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2451.718366 secs 2024-04-26T12:31:53.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:31:53.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2454.441591 secs 2024-04-26T12:31:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:53 smithi033 ceph-mon[27845]: pgmap v1987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2451.718366 secs 2024-04-26T12:31:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:31:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2454.441591 secs 2024-04-26T12:31:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:53 smithi096 ceph-mon[32753]: pgmap v1987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2451.718366 secs 2024-04-26T12:31:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:31:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2454.441591 secs 2024-04-26T12:31:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:31:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:31:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:31:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:31:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:31:54.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:31:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:31:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:31:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:31:55.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:55 smithi033 ceph-mon[27845]: pgmap v1988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:55.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:55 smithi033 ceph-mon[27845]: pgmap v1989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:31:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:55 smithi194 ceph-mon[33017]: pgmap v1988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:55 smithi194 ceph-mon[33017]: pgmap v1989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:31:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:55 smithi096 ceph-mon[32753]: pgmap v1988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:31:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:55 smithi096 ceph-mon[32753]: pgmap v1989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:31:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:57 smithi194 ceph-mon[33017]: pgmap v1990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:57 smithi033 ceph-mon[27845]: pgmap v1990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:57 smithi096 ceph-mon[32753]: pgmap v1990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:58.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2456.718451 secs 2024-04-26T12:31:58.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2459.441711 secs 2024-04-26T12:31:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2456.718451 secs 2024-04-26T12:31:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2459.441711 secs 2024-04-26T12:31:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2456.718451 secs 2024-04-26T12:31:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2459.441711 secs 2024-04-26T12:31:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:31:59 smithi194 ceph-mon[33017]: pgmap v1991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:31:59 smithi033 ceph-mon[27845]: pgmap v1991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:31:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:31:59 smithi096 ceph-mon[32753]: pgmap v1991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:32:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:00 smithi194 ceph-mon[33017]: pgmap v1992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:32:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:00 smithi033 ceph-mon[27845]: pgmap v1992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:32:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:00 smithi096 ceph-mon[32753]: pgmap v1992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:32:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:03 smithi033 ceph-mon[27845]: pgmap v1993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:32:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2461.718532 secs 2024-04-26T12:32:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:03 smithi096 ceph-mon[32753]: pgmap v1993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:32:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2461.718532 secs 2024-04-26T12:32:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:03 smithi194 ceph-mon[33017]: pgmap v1993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:32:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2461.718532 secs 2024-04-26T12:32:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2464.441828 secs 2024-04-26T12:32:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:04.440+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:04.440+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:04.440+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:32:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2464.441828 secs 2024-04-26T12:32:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2464.441828 secs 2024-04-26T12:32:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:05 smithi194 ceph-mon[33017]: pgmap v1994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:32:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:05 smithi033 ceph-mon[27845]: pgmap v1994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:32:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:05 smithi096 ceph-mon[32753]: pgmap v1994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:32:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:07 smithi194 ceph-mon[33017]: pgmap v1995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:07 smithi033 ceph-mon[27845]: pgmap v1995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:07 smithi096 ceph-mon[32753]: pgmap v1995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2466.718612 secs 2024-04-26T12:32:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2466.718612 secs 2024-04-26T12:32:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2466.718612 secs 2024-04-26T12:32:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2469.441939 secs 2024-04-26T12:32:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:09 smithi194 ceph-mon[33017]: pgmap v1996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2469.441939 secs 2024-04-26T12:32:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:09 smithi033 ceph-mon[27845]: pgmap v1996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2469.441939 secs 2024-04-26T12:32:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:09 smithi096 ceph-mon[32753]: pgmap v1996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:11 smithi194 ceph-mon[33017]: pgmap v1997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:11 smithi033 ceph-mon[27845]: pgmap v1997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:11 smithi096 ceph-mon[32753]: pgmap v1997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:13 smithi033 ceph-mon[27845]: pgmap v1998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2471.718698 secs 2024-04-26T12:32:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:13 smithi194 ceph-mon[33017]: pgmap v1998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2471.718698 secs 2024-04-26T12:32:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:13 smithi096 ceph-mon[32753]: pgmap v1998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2471.718698 secs 2024-04-26T12:32:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2474.442075 secs 2024-04-26T12:32:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2474.442075 secs 2024-04-26T12:32:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2474.442075 secs 2024-04-26T12:32:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:15 smithi194 ceph-mon[33017]: pgmap v1999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:15 smithi033 ceph-mon[27845]: pgmap v1999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:15 smithi096 ceph-mon[32753]: pgmap v1999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:17 smithi194 ceph-mon[33017]: pgmap v2000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:17 smithi033 ceph-mon[27845]: pgmap v2000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:17 smithi096 ceph-mon[32753]: pgmap v2000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2476.718783 secs 2024-04-26T12:32:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2476.718783 secs 2024-04-26T12:32:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2476.718783 secs 2024-04-26T12:32:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:19.451+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:19.451+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:19.451+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:32:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2479.442192 secs 2024-04-26T12:32:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:19 smithi194 ceph-mon[33017]: pgmap v2001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2479.442192 secs 2024-04-26T12:32:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:19 smithi033 ceph-mon[27845]: pgmap v2001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2479.442192 secs 2024-04-26T12:32:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:19 smithi096 ceph-mon[32753]: pgmap v2001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:21 smithi194 ceph-mon[33017]: pgmap v2002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:21 smithi033 ceph-mon[27845]: pgmap v2002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:21 smithi096 ceph-mon[32753]: pgmap v2002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:22.216+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:32:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:22.216+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:32:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:22.216+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:32:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:22.216+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0020 con 0x7fef00077780 2024-04-26T12:32:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:23 smithi194 ceph-mon[33017]: pgmap v2003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2481.718861 secs 2024-04-26T12:32:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:23 smithi033 ceph-mon[27845]: pgmap v2003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2481.718861 secs 2024-04-26T12:32:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:23 smithi096 ceph-mon[32753]: pgmap v2003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2481.718861 secs 2024-04-26T12:32:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2484.442308 secs 2024-04-26T12:32:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2484.442308 secs 2024-04-26T12:32:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2484.442308 secs 2024-04-26T12:32:25.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:25 smithi033 ceph-mon[27845]: pgmap v2004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:25 smithi194 ceph-mon[33017]: pgmap v2004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:25 smithi096 ceph-mon[32753]: pgmap v2004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:27 smithi194 ceph-mon[33017]: pgmap v2005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:27 smithi033 ceph-mon[27845]: pgmap v2005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:27 smithi096 ceph-mon[32753]: pgmap v2005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:28.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2486.718944 secs 2024-04-26T12:32:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2486.718944 secs 2024-04-26T12:32:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2486.718944 secs 2024-04-26T12:32:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2489.442428 secs 2024-04-26T12:32:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:29 smithi194 ceph-mon[33017]: pgmap v2006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2489.442428 secs 2024-04-26T12:32:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:29 smithi033 ceph-mon[27845]: pgmap v2006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2489.442428 secs 2024-04-26T12:32:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:29 smithi096 ceph-mon[32753]: pgmap v2006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:31 smithi194 ceph-mon[33017]: pgmap v2007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:31 smithi033 ceph-mon[27845]: pgmap v2007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:31.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:31 smithi096 ceph-mon[32753]: pgmap v2007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:33 smithi194 ceph-mon[33017]: pgmap v2008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:33.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2491.719030 secs 2024-04-26T12:32:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:33 smithi033 ceph-mon[27845]: pgmap v2008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2491.719030 secs 2024-04-26T12:32:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:33 smithi096 ceph-mon[32753]: pgmap v2008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2491.719030 secs 2024-04-26T12:32:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:34.464+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:34.464+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:34.464+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:32:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2494.442521 secs 2024-04-26T12:32:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2494.442521 secs 2024-04-26T12:32:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2494.442521 secs 2024-04-26T12:32:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:35 smithi194 ceph-mon[33017]: pgmap v2009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:35 smithi033 ceph-mon[27845]: pgmap v2009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:35 smithi096 ceph-mon[32753]: pgmap v2009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:37 smithi194 ceph-mon[33017]: pgmap v2010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:37 smithi033 ceph-mon[27845]: pgmap v2010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:37 smithi096 ceph-mon[32753]: pgmap v2010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2496.719119 secs 2024-04-26T12:32:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2496.719119 secs 2024-04-26T12:32:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2496.719119 secs 2024-04-26T12:32:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2499.442646 secs 2024-04-26T12:32:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:39 smithi194 ceph-mon[33017]: pgmap v2011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2499.442646 secs 2024-04-26T12:32:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:39 smithi033 ceph-mon[27845]: pgmap v2011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2499.442646 secs 2024-04-26T12:32:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:39 smithi096 ceph-mon[32753]: pgmap v2011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:41 smithi194 ceph-mon[33017]: pgmap v2012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:41 smithi033 ceph-mon[27845]: pgmap v2012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:41.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:41 smithi096 ceph-mon[32753]: pgmap v2012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:43 smithi194 ceph-mon[33017]: pgmap v2013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2501.719178 secs 2024-04-26T12:32:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:43 smithi033 ceph-mon[27845]: pgmap v2013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2501.719178 secs 2024-04-26T12:32:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:43 smithi096 ceph-mon[32753]: pgmap v2013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2501.719178 secs 2024-04-26T12:32:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2504.442767 secs 2024-04-26T12:32:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2504.442767 secs 2024-04-26T12:32:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2504.442767 secs 2024-04-26T12:32:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:45 smithi033 ceph-mon[27845]: pgmap v2014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:45 smithi096 ceph-mon[32753]: pgmap v2014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:45 smithi194 ceph-mon[33017]: pgmap v2014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:47 smithi033 ceph-mon[27845]: pgmap v2015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:47 smithi096 ceph-mon[32753]: pgmap v2015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:47 smithi194 ceph-mon[33017]: pgmap v2015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:48.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2506.719278 secs 2024-04-26T12:32:48.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2509.442898 secs 2024-04-26T12:32:48.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:48 smithi033 ceph-mon[27845]: pgmap v2016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2506.719278 secs 2024-04-26T12:32:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2509.442898 secs 2024-04-26T12:32:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:48 smithi096 ceph-mon[32753]: pgmap v2016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2506.719278 secs 2024-04-26T12:32:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2509.442898 secs 2024-04-26T12:32:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:48 smithi194 ceph-mon[33017]: pgmap v2016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:49.478+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:49.478+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:32:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:32:49.478+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:32:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:51 smithi194 ceph-mon[33017]: pgmap v2017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:51 smithi033 ceph-mon[27845]: pgmap v2017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:51 smithi096 ceph-mon[32753]: pgmap v2017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:53 smithi194 ceph-mon[33017]: pgmap v2018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2511.719363 secs 2024-04-26T12:32:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:53 smithi033 ceph-mon[27845]: pgmap v2018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2511.719363 secs 2024-04-26T12:32:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:53 smithi096 ceph-mon[32753]: pgmap v2018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2511.719363 secs 2024-04-26T12:32:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2514.443021 secs 2024-04-26T12:32:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2514.443021 secs 2024-04-26T12:32:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2514.443021 secs 2024-04-26T12:32:55.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:55 smithi033 ceph-mon[27845]: pgmap v2019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:55.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:32:55.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:55 smithi194 ceph-mon[33017]: pgmap v2019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:32:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:55 smithi096 ceph-mon[32753]: pgmap v2019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:32:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:57 smithi194 ceph-mon[33017]: pgmap v2020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:57.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:57 smithi033 ceph-mon[27845]: pgmap v2020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:57.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:57.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:57 smithi096 ceph-mon[32753]: pgmap v2020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:32:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2516.719433 secs 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2516.719433 secs 2024-04-26T12:32:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2516.719433 secs 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2519.443141 secs 2024-04-26T12:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: pgmap v2021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:59.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:59.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:32:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:32:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:32:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:32:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:32:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2519.443141 secs 2024-04-26T12:32:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: pgmap v2021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:59.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:32:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:32:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:32:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:32:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:32:59.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2519.443141 secs 2024-04-26T12:32:59.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: pgmap v2021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:32:59.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:32:59.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:32:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:33:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: pgmap v2022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:33:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: pgmap v2023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: pgmap v2024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: pgmap v2022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:33:01.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:33:01.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:33:01.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:33:01.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: pgmap v2023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T12:33:01.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: pgmap v2024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:01.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: pgmap v2022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:33:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:33:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:33:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:33:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:33:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: pgmap v2023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T12:33:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: pgmap v2024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:33:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:02 smithi033 ceph-mon[27845]: pgmap v2025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:02 smithi096 ceph-mon[32753]: pgmap v2025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:02 smithi194 ceph-mon[33017]: pgmap v2025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2521.719514 secs 2024-04-26T12:33:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2524.443251 secs 2024-04-26T12:33:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2521.719514 secs 2024-04-26T12:33:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2524.443251 secs 2024-04-26T12:33:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2521.719514 secs 2024-04-26T12:33:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2524.443251 secs 2024-04-26T12:33:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:04.486+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:04.486+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:33:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:04.486+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:04 smithi033 ceph-mon[27845]: pgmap v2026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:04 smithi096 ceph-mon[32753]: pgmap v2026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:04 smithi194 ceph-mon[33017]: pgmap v2026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:06 smithi033 ceph-mon[27845]: pgmap v2027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:06 smithi096 ceph-mon[32753]: pgmap v2027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:06 smithi194 ceph-mon[33017]: pgmap v2027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:33:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2526.719591 secs 2024-04-26T12:33:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2526.719591 secs 2024-04-26T12:33:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2526.719591 secs 2024-04-26T12:33:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2529.443366 secs 2024-04-26T12:33:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:08 smithi194 ceph-mon[33017]: pgmap v2028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:33:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2529.443366 secs 2024-04-26T12:33:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:08 smithi033 ceph-mon[27845]: pgmap v2028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:33:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2529.443366 secs 2024-04-26T12:33:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:08 smithi096 ceph-mon[32753]: pgmap v2028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:33:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:10 smithi033 ceph-mon[27845]: pgmap v2029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:33:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:10 smithi096 ceph-mon[32753]: pgmap v2029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:33:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:10 smithi194 ceph-mon[33017]: pgmap v2029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:33:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:12 smithi033 ceph-mon[27845]: pgmap v2030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:12 smithi096 ceph-mon[32753]: pgmap v2030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:13.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:12 smithi194 ceph-mon[33017]: pgmap v2030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:13.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2531.719676 secs 2024-04-26T12:33:13.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2534.443493 secs 2024-04-26T12:33:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2531.719676 secs 2024-04-26T12:33:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2534.443493 secs 2024-04-26T12:33:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2531.719676 secs 2024-04-26T12:33:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2534.443493 secs 2024-04-26T12:33:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:14 smithi033 ceph-mon[27845]: pgmap v2031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:14 smithi096 ceph-mon[32753]: pgmap v2031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:14 smithi194 ceph-mon[33017]: pgmap v2031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:16 smithi033 ceph-mon[27845]: pgmap v2032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:16 smithi096 ceph-mon[32753]: pgmap v2032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:16 smithi194 ceph-mon[33017]: pgmap v2032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2536.719735 secs 2024-04-26T12:33:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2536.719735 secs 2024-04-26T12:33:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2536.719735 secs 2024-04-26T12:33:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2539.443558 secs 2024-04-26T12:33:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:18 smithi194 ceph-mon[33017]: pgmap v2033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2539.443558 secs 2024-04-26T12:33:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:18 smithi033 ceph-mon[27845]: pgmap v2033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2539.443558 secs 2024-04-26T12:33:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:18 smithi096 ceph-mon[32753]: pgmap v2033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:19.503+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:19.503+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:19.503+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:33:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:20 smithi033 ceph-mon[27845]: pgmap v2034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:20 smithi096 ceph-mon[32753]: pgmap v2034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:20 smithi194 ceph-mon[33017]: pgmap v2034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:22.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:22 smithi033 ceph-mon[27845]: pgmap v2035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:22 smithi096 ceph-mon[32753]: pgmap v2035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:22 smithi194 ceph-mon[33017]: pgmap v2035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2541.719817 secs 2024-04-26T12:33:23.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2544.443670 secs 2024-04-26T12:33:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2541.719817 secs 2024-04-26T12:33:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2544.443670 secs 2024-04-26T12:33:24.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2541.719817 secs 2024-04-26T12:33:24.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2544.443670 secs 2024-04-26T12:33:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:24 smithi033 ceph-mon[27845]: pgmap v2036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:24 smithi096 ceph-mon[32753]: pgmap v2036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:25.018 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:24 smithi194 ceph-mon[33017]: pgmap v2036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:26 smithi033 ceph-mon[27845]: pgmap v2037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:26 smithi096 ceph-mon[32753]: pgmap v2037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:26 smithi194 ceph-mon[33017]: pgmap v2037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2546.719904 secs 2024-04-26T12:33:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2546.719904 secs 2024-04-26T12:33:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2546.719904 secs 2024-04-26T12:33:29.085 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2549.443802 secs 2024-04-26T12:33:29.085 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:28 smithi096 ceph-mon[32753]: pgmap v2038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2549.443802 secs 2024-04-26T12:33:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:28 smithi194 ceph-mon[33017]: pgmap v2038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2549.443802 secs 2024-04-26T12:33:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:28 smithi033 ceph-mon[27845]: pgmap v2038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:30 smithi033 ceph-mon[27845]: pgmap v2039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:30 smithi096 ceph-mon[32753]: pgmap v2039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:30 smithi194 ceph-mon[33017]: pgmap v2039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:32 smithi033 ceph-mon[27845]: pgmap v2040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:32 smithi096 ceph-mon[32753]: pgmap v2040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:32 smithi194 ceph-mon[33017]: pgmap v2040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2551.719986 secs 2024-04-26T12:33:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2554.443899 secs 2024-04-26T12:33:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2551.719986 secs 2024-04-26T12:33:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2554.443899 secs 2024-04-26T12:33:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2551.719986 secs 2024-04-26T12:33:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2554.443899 secs 2024-04-26T12:33:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:34.517+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:34.517+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:33:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:34.517+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:34 smithi033 ceph-mon[27845]: pgmap v2041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:34 smithi096 ceph-mon[32753]: pgmap v2041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:34 smithi194 ceph-mon[33017]: pgmap v2041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:36 smithi033 ceph-mon[27845]: pgmap v2042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:36 smithi096 ceph-mon[32753]: pgmap v2042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:36 smithi194 ceph-mon[33017]: pgmap v2042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2556.720046 secs 2024-04-26T12:33:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2556.720046 secs 2024-04-26T12:33:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2556.720046 secs 2024-04-26T12:33:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2559.443992 secs 2024-04-26T12:33:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:38 smithi194 ceph-mon[33017]: pgmap v2043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2559.443992 secs 2024-04-26T12:33:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:38 smithi033 ceph-mon[27845]: pgmap v2043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2559.443992 secs 2024-04-26T12:33:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:38 smithi096 ceph-mon[32753]: pgmap v2043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:40 smithi033 ceph-mon[27845]: pgmap v2044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:40 smithi096 ceph-mon[32753]: pgmap v2044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:40 smithi194 ceph-mon[33017]: pgmap v2044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2561.720107 secs 2024-04-26T12:33:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:43 smithi033 ceph-mon[27845]: pgmap v2045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2564.444089 secs 2024-04-26T12:33:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2561.720107 secs 2024-04-26T12:33:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:43 smithi096 ceph-mon[32753]: pgmap v2045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2564.444089 secs 2024-04-26T12:33:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2561.720107 secs 2024-04-26T12:33:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:43 smithi194 ceph-mon[33017]: pgmap v2045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2564.444089 secs 2024-04-26T12:33:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:45 smithi033 ceph-mon[27845]: pgmap v2046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:45 smithi096 ceph-mon[32753]: pgmap v2046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:46.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:45 smithi194 ceph-mon[33017]: pgmap v2046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:47 smithi033 ceph-mon[27845]: pgmap v2047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:47 smithi096 ceph-mon[32753]: pgmap v2047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:47 smithi194 ceph-mon[33017]: pgmap v2047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2566.720180 secs 2024-04-26T12:33:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2569.444151 secs 2024-04-26T12:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2566.720180 secs 2024-04-26T12:33:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2569.444151 secs 2024-04-26T12:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2566.720180 secs 2024-04-26T12:33:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2569.444151 secs 2024-04-26T12:33:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:49.519+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:49.519+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:33:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:33:49.519+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:33:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:49 smithi033 ceph-mon[27845]: pgmap v2048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:49 smithi096 ceph-mon[32753]: pgmap v2048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:49 smithi194 ceph-mon[33017]: pgmap v2048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:51 smithi033 ceph-mon[27845]: pgmap v2049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:51 smithi096 ceph-mon[32753]: pgmap v2049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:51 smithi194 ceph-mon[33017]: pgmap v2049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:33:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2571.720244 secs 2024-04-26T12:33:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:53 smithi096 ceph-mon[32753]: pgmap v2050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2574.444266 secs 2024-04-26T12:33:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2571.720244 secs 2024-04-26T12:33:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:53 smithi194 ceph-mon[33017]: pgmap v2050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2574.444266 secs 2024-04-26T12:33:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2571.720244 secs 2024-04-26T12:33:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:53 smithi033 ceph-mon[27845]: pgmap v2050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:33:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2574.444266 secs 2024-04-26T12:33:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:55 smithi194 ceph-mon[33017]: pgmap v2051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:33:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:55 smithi033 ceph-mon[27845]: pgmap v2051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:33:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:55 smithi096 ceph-mon[32753]: pgmap v2051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:33:57.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:57 smithi033 ceph-mon[27845]: pgmap v2052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:33:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:57 smithi194 ceph-mon[33017]: pgmap v2052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:33:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:57 smithi096 ceph-mon[32753]: pgmap v2052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:33:59.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2576.720326 secs 2024-04-26T12:33:59.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2579.444382 secs 2024-04-26T12:33:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2576.720326 secs 2024-04-26T12:33:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2579.444382 secs 2024-04-26T12:33:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2576.720326 secs 2024-04-26T12:33:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2579.444382 secs 2024-04-26T12:34:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:33:59 smithi194 ceph-mon[33017]: pgmap v2053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:33:59 smithi033 ceph-mon[27845]: pgmap v2053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:33:59 smithi096 ceph-mon[32753]: pgmap v2053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:01 smithi194 ceph-mon[33017]: pgmap v2054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:34:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:34:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:01 smithi033 ceph-mon[27845]: pgmap v2054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:34:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:34:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:01 smithi096 ceph-mon[32753]: pgmap v2054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:34:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:34:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:34:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:34:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:02 smithi194 ceph-mon[33017]: pgmap v2055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:02 smithi194 ceph-mon[33017]: pgmap v2056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:34:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:34:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:34:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:34:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:02 smithi033 ceph-mon[27845]: pgmap v2055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:02 smithi033 ceph-mon[27845]: pgmap v2056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:34:03.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:34:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:34:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:34:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:02 smithi096 ceph-mon[32753]: pgmap v2055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:02 smithi096 ceph-mon[32753]: pgmap v2056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T12:34:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:34:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2581.720390 secs 2024-04-26T12:34:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2584.444495 secs 2024-04-26T12:34:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2581.720390 secs 2024-04-26T12:34:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2584.444495 secs 2024-04-26T12:34:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2581.720390 secs 2024-04-26T12:34:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2584.444495 secs 2024-04-26T12:34:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:04.535+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:04.535+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:04.535+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:34:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:04 smithi194 ceph-mon[33017]: pgmap v2057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:04 smithi033 ceph-mon[27845]: pgmap v2057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:04 smithi096 ceph-mon[32753]: pgmap v2057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:34:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:07 smithi194 ceph-mon[33017]: pgmap v2058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:07 smithi033 ceph-mon[27845]: pgmap v2058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:07 smithi096 ceph-mon[32753]: pgmap v2058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2586.720449 secs 2024-04-26T12:34:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2586.720449 secs 2024-04-26T12:34:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2586.720449 secs 2024-04-26T12:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2589.444574 secs 2024-04-26T12:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:09 smithi194 ceph-mon[33017]: pgmap v2059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2589.444574 secs 2024-04-26T12:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:09 smithi033 ceph-mon[27845]: pgmap v2059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:09.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2589.444574 secs 2024-04-26T12:34:09.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:09 smithi096 ceph-mon[32753]: pgmap v2059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:11 smithi194 ceph-mon[33017]: pgmap v2060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:11 smithi033 ceph-mon[27845]: pgmap v2060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:11 smithi096 ceph-mon[32753]: pgmap v2060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:13 smithi194 ceph-mon[33017]: pgmap v2061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:34:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2591.720511 secs 2024-04-26T12:34:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:13 smithi033 ceph-mon[27845]: pgmap v2061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:34:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2591.720511 secs 2024-04-26T12:34:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:13 smithi096 ceph-mon[32753]: pgmap v2061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:34:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2591.720511 secs 2024-04-26T12:34:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2594.444691 secs 2024-04-26T12:34:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2594.444691 secs 2024-04-26T12:34:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2594.444691 secs 2024-04-26T12:34:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:15 smithi194 ceph-mon[33017]: pgmap v2062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:15 smithi033 ceph-mon[27845]: pgmap v2062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:15 smithi096 ceph-mon[32753]: pgmap v2062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:17 smithi194 ceph-mon[33017]: pgmap v2063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:17 smithi033 ceph-mon[27845]: pgmap v2063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:17 smithi096 ceph-mon[32753]: pgmap v2063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2596.720590 secs 2024-04-26T12:34:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2596.720590 secs 2024-04-26T12:34:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2596.720590 secs 2024-04-26T12:34:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:19.551+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:19.551+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:19.551+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:34:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2599.444829 secs 2024-04-26T12:34:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:19 smithi194 ceph-mon[33017]: pgmap v2064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2599.444829 secs 2024-04-26T12:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:19 smithi033 ceph-mon[27845]: pgmap v2064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2599.444829 secs 2024-04-26T12:34:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:19 smithi096 ceph-mon[32753]: pgmap v2064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:21 smithi033 ceph-mon[27845]: pgmap v2065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:21 smithi096 ceph-mon[32753]: pgmap v2065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:21 smithi194 ceph-mon[33017]: pgmap v2065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:23 smithi033 ceph-mon[27845]: pgmap v2066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2601.720650 secs 2024-04-26T12:34:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:23 smithi096 ceph-mon[32753]: pgmap v2066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2601.720650 secs 2024-04-26T12:34:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:23 smithi194 ceph-mon[33017]: pgmap v2066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2601.720650 secs 2024-04-26T12:34:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2604.444965 secs 2024-04-26T12:34:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2604.444965 secs 2024-04-26T12:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2604.444965 secs 2024-04-26T12:34:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:25 smithi033 ceph-mon[27845]: pgmap v2067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:25 smithi096 ceph-mon[32753]: pgmap v2067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:25.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:25 smithi194 ceph-mon[33017]: pgmap v2067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:27 smithi033 ceph-mon[27845]: pgmap v2068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:27 smithi096 ceph-mon[32753]: pgmap v2068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:27 smithi194 ceph-mon[33017]: pgmap v2068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2606.720722 secs 2024-04-26T12:34:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2606.720722 secs 2024-04-26T12:34:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2606.720722 secs 2024-04-26T12:34:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2609.445080 secs 2024-04-26T12:34:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:29 smithi033 ceph-mon[27845]: pgmap v2069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2609.445080 secs 2024-04-26T12:34:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:29 smithi096 ceph-mon[32753]: pgmap v2069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2609.445080 secs 2024-04-26T12:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:29 smithi194 ceph-mon[33017]: pgmap v2069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:31 smithi033 ceph-mon[27845]: pgmap v2070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:31.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:31 smithi096 ceph-mon[32753]: pgmap v2070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:31 smithi194 ceph-mon[33017]: pgmap v2070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:33 smithi033 ceph-mon[27845]: pgmap v2071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2611.720805 secs 2024-04-26T12:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:33 smithi096 ceph-mon[32753]: pgmap v2071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2611.720805 secs 2024-04-26T12:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:33 smithi194 ceph-mon[33017]: pgmap v2071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2611.720805 secs 2024-04-26T12:34:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:34.560+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:34.560+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:34:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:34.560+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:34.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2614.445195 secs 2024-04-26T12:34:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2614.445195 secs 2024-04-26T12:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2614.445195 secs 2024-04-26T12:34:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:35 smithi033 ceph-mon[27845]: pgmap v2072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:35 smithi096 ceph-mon[32753]: pgmap v2072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:35 smithi194 ceph-mon[33017]: pgmap v2072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:37 smithi033 ceph-mon[27845]: pgmap v2073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:37 smithi096 ceph-mon[32753]: pgmap v2073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:37 smithi194 ceph-mon[33017]: pgmap v2073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2616.720889 secs 2024-04-26T12:34:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2616.720889 secs 2024-04-26T12:34:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2616.720889 secs 2024-04-26T12:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2619.445300 secs 2024-04-26T12:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:39 smithi194 ceph-mon[33017]: pgmap v2074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2619.445300 secs 2024-04-26T12:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:39 smithi033 ceph-mon[27845]: pgmap v2074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2619.445300 secs 2024-04-26T12:34:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:39 smithi096 ceph-mon[32753]: pgmap v2074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:41 smithi194 ceph-mon[33017]: pgmap v2075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:41 smithi033 ceph-mon[27845]: pgmap v2075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:41.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:41 smithi096 ceph-mon[32753]: pgmap v2075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:43 smithi194 ceph-mon[33017]: pgmap v2076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2621.720993 secs 2024-04-26T12:34:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:43 smithi033 ceph-mon[27845]: pgmap v2076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2621.720993 secs 2024-04-26T12:34:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:43 smithi096 ceph-mon[32753]: pgmap v2076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2621.720993 secs 2024-04-26T12:34:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2624.445436 secs 2024-04-26T12:34:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2624.445436 secs 2024-04-26T12:34:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2624.445436 secs 2024-04-26T12:34:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:45 smithi194 ceph-mon[33017]: pgmap v2077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:45 smithi033 ceph-mon[27845]: pgmap v2077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:45 smithi096 ceph-mon[32753]: pgmap v2077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:47 smithi194 ceph-mon[33017]: pgmap v2078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:47 smithi033 ceph-mon[27845]: pgmap v2078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:47 smithi096 ceph-mon[32753]: pgmap v2078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2626.721080 secs 2024-04-26T12:34:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2626.721080 secs 2024-04-26T12:34:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2626.721080 secs 2024-04-26T12:34:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:49.562+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:49.562+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:34:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:34:49.562+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:34:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2629.445553 secs 2024-04-26T12:34:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:49 smithi033 ceph-mon[27845]: pgmap v2079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2629.445553 secs 2024-04-26T12:34:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:49 smithi194 ceph-mon[33017]: pgmap v2079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2629.445553 secs 2024-04-26T12:34:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:49 smithi096 ceph-mon[32753]: pgmap v2079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:51 smithi194 ceph-mon[33017]: pgmap v2080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:51 smithi033 ceph-mon[27845]: pgmap v2080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:51 smithi096 ceph-mon[32753]: pgmap v2080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:34:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:53 smithi194 ceph-mon[33017]: pgmap v2081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2631.721161 secs 2024-04-26T12:34:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:53 smithi033 ceph-mon[27845]: pgmap v2081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2631.721161 secs 2024-04-26T12:34:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:53 smithi096 ceph-mon[32753]: pgmap v2081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2631.721161 secs 2024-04-26T12:34:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2634.445616 secs 2024-04-26T12:34:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2634.445616 secs 2024-04-26T12:34:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2634.445616 secs 2024-04-26T12:34:55.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:55 smithi194 ceph-mon[33017]: pgmap v2082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:55 smithi033 ceph-mon[27845]: pgmap v2082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:55 smithi096 ceph-mon[32753]: pgmap v2082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:34:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:57 smithi194 ceph-mon[33017]: pgmap v2083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:57 smithi033 ceph-mon[27845]: pgmap v2083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:57 smithi096 ceph-mon[32753]: pgmap v2083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2636.721223 secs 2024-04-26T12:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2639.445735 secs 2024-04-26T12:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2636.721223 secs 2024-04-26T12:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2639.445735 secs 2024-04-26T12:34:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2636.721223 secs 2024-04-26T12:34:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2639.445735 secs 2024-04-26T12:34:59.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:34:59 smithi096 ceph-mon[32753]: pgmap v2084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:34:59 smithi194 ceph-mon[33017]: pgmap v2084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:34:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:34:59 smithi033 ceph-mon[27845]: pgmap v2084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:01 smithi194 ceph-mon[33017]: pgmap v2085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:01 smithi033 ceph-mon[27845]: pgmap v2085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:01 smithi096 ceph-mon[32753]: pgmap v2085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:03 smithi194 ceph-mon[33017]: pgmap v2086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2641.721283 secs 2024-04-26T12:35:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:35:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2644.445808 secs 2024-04-26T12:35:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:03 smithi033 ceph-mon[27845]: pgmap v2086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2641.721283 secs 2024-04-26T12:35:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:35:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2644.445808 secs 2024-04-26T12:35:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:03 smithi096 ceph-mon[32753]: pgmap v2086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2641.721283 secs 2024-04-26T12:35:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:35:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2644.445808 secs 2024-04-26T12:35:04.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:04.569+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:04.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:04.569+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:04.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:04.569+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:35:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:35:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:35:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:35:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:35:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:35:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:35:04.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:35:04.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:35:04.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:35:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:05 smithi194 ceph-mon[33017]: pgmap v2087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:05 smithi194 ceph-mon[33017]: pgmap v2088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:35:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:05 smithi033 ceph-mon[27845]: pgmap v2087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:05 smithi033 ceph-mon[27845]: pgmap v2088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:35:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:05 smithi096 ceph-mon[32753]: pgmap v2087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:05 smithi096 ceph-mon[32753]: pgmap v2088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:35:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:07 smithi194 ceph-mon[33017]: pgmap v2089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:07 smithi033 ceph-mon[27845]: pgmap v2089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:07 smithi096 ceph-mon[32753]: pgmap v2089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2646.721371 secs 2024-04-26T12:35:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2649.445899 secs 2024-04-26T12:35:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2646.721371 secs 2024-04-26T12:35:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2649.445899 secs 2024-04-26T12:35:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2646.721371 secs 2024-04-26T12:35:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2649.445899 secs 2024-04-26T12:35:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:09 smithi194 ceph-mon[33017]: pgmap v2090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:09 smithi033 ceph-mon[27845]: pgmap v2090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:09 smithi096 ceph-mon[32753]: pgmap v2090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:11 smithi194 ceph-mon[33017]: pgmap v2091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:11 smithi033 ceph-mon[27845]: pgmap v2091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:11 smithi096 ceph-mon[32753]: pgmap v2091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:13 smithi194 ceph-mon[33017]: pgmap v2092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2651.721459 secs 2024-04-26T12:35:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2654.446022 secs 2024-04-26T12:35:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:13 smithi033 ceph-mon[27845]: pgmap v2092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2651.721459 secs 2024-04-26T12:35:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2654.446022 secs 2024-04-26T12:35:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:13 smithi096 ceph-mon[32753]: pgmap v2092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:35:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2651.721459 secs 2024-04-26T12:35:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2654.446022 secs 2024-04-26T12:35:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:15 smithi194 ceph-mon[33017]: pgmap v2093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:35:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:15 smithi033 ceph-mon[27845]: pgmap v2093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:35:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:15 smithi096 ceph-mon[32753]: pgmap v2093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:35:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:17 smithi194 ceph-mon[33017]: pgmap v2094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:17 smithi033 ceph-mon[27845]: pgmap v2094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:17 smithi096 ceph-mon[32753]: pgmap v2094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2656.721559 secs 2024-04-26T12:35:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2659.446145 secs 2024-04-26T12:35:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2656.721559 secs 2024-04-26T12:35:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2659.446145 secs 2024-04-26T12:35:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2656.721559 secs 2024-04-26T12:35:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2659.446145 secs 2024-04-26T12:35:19.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:19.583+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:19.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:19.583+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:19.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:19.583+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:35:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:19 smithi194 ceph-mon[33017]: pgmap v2095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:19 smithi033 ceph-mon[27845]: pgmap v2095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:19 smithi096 ceph-mon[32753]: pgmap v2095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:21 smithi194 ceph-mon[33017]: pgmap v2096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:21 smithi033 ceph-mon[27845]: pgmap v2096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:21 smithi096 ceph-mon[32753]: pgmap v2096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:22.246+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:35:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:22.246+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:35:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:22.247+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0100 con 0x564385821900 2024-04-26T12:35:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:22.247+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T12:35:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:23 smithi194 ceph-mon[33017]: pgmap v2097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2661.721841 secs 2024-04-26T12:35:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2664.446185 secs 2024-04-26T12:35:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:23 smithi033 ceph-mon[27845]: pgmap v2097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2661.721841 secs 2024-04-26T12:35:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2664.446185 secs 2024-04-26T12:35:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:23 smithi096 ceph-mon[32753]: pgmap v2097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2661.721841 secs 2024-04-26T12:35:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2664.446185 secs 2024-04-26T12:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:24 smithi033 ceph-mon[27845]: pgmap v2098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:24 smithi096 ceph-mon[32753]: pgmap v2098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:24 smithi194 ceph-mon[33017]: pgmap v2098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:27 smithi194 ceph-mon[33017]: pgmap v2099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:27 smithi033 ceph-mon[27845]: pgmap v2099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:27 smithi096 ceph-mon[32753]: pgmap v2099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2666.721900 secs 2024-04-26T12:35:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2666.721900 secs 2024-04-26T12:35:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2666.721900 secs 2024-04-26T12:35:29.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:29 smithi096 ceph-mon[32753]: pgmap v2100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:29.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2669.446259 secs 2024-04-26T12:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:29 smithi194 ceph-mon[33017]: pgmap v2100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2669.446259 secs 2024-04-26T12:35:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:29 smithi033 ceph-mon[27845]: pgmap v2100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2669.446259 secs 2024-04-26T12:35:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:31 smithi194 ceph-mon[33017]: pgmap v2101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:31 smithi033 ceph-mon[27845]: pgmap v2101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:31.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:31 smithi096 ceph-mon[32753]: pgmap v2101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:33 smithi194 ceph-mon[33017]: pgmap v2102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2671.721958 secs 2024-04-26T12:35:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:33 smithi033 ceph-mon[27845]: pgmap v2102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2671.721958 secs 2024-04-26T12:35:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:33 smithi096 ceph-mon[32753]: pgmap v2102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2671.721958 secs 2024-04-26T12:35:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2674.446380 secs 2024-04-26T12:35:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2674.446380 secs 2024-04-26T12:35:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2674.446380 secs 2024-04-26T12:35:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:34.599+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:34.599+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:34.599+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:35:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:35 smithi194 ceph-mon[33017]: pgmap v2103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:35 smithi033 ceph-mon[27845]: pgmap v2103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:35 smithi096 ceph-mon[32753]: pgmap v2103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:37 smithi194 ceph-mon[33017]: pgmap v2104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:37 smithi033 ceph-mon[27845]: pgmap v2104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:37 smithi096 ceph-mon[32753]: pgmap v2104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2676.721992 secs 2024-04-26T12:35:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2676.721992 secs 2024-04-26T12:35:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2676.721992 secs 2024-04-26T12:35:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:39 smithi194 ceph-mon[33017]: pgmap v2105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2679.446506 secs 2024-04-26T12:35:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:39 smithi033 ceph-mon[27845]: pgmap v2105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2679.446506 secs 2024-04-26T12:35:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:39 smithi096 ceph-mon[32753]: pgmap v2105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2679.446506 secs 2024-04-26T12:35:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:41 smithi194 ceph-mon[33017]: pgmap v2106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:41 smithi033 ceph-mon[27845]: pgmap v2106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:41 smithi096 ceph-mon[32753]: pgmap v2106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:43 smithi194 ceph-mon[33017]: pgmap v2107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2681.722035 secs 2024-04-26T12:35:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:43 smithi033 ceph-mon[27845]: pgmap v2107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2681.722035 secs 2024-04-26T12:35:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:43 smithi096 ceph-mon[32753]: pgmap v2107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2681.722035 secs 2024-04-26T12:35:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2684.446582 secs 2024-04-26T12:35:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2684.446582 secs 2024-04-26T12:35:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2684.446582 secs 2024-04-26T12:35:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:45 smithi033 ceph-mon[27845]: pgmap v2108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:45.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:45 smithi096 ceph-mon[32753]: pgmap v2108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:45 smithi194 ceph-mon[33017]: pgmap v2108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:47 smithi033 ceph-mon[27845]: pgmap v2109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:47 smithi096 ceph-mon[32753]: pgmap v2109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:47 smithi194 ceph-mon[33017]: pgmap v2109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2686.722101 secs 2024-04-26T12:35:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2686.722101 secs 2024-04-26T12:35:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2686.722101 secs 2024-04-26T12:35:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:49 smithi033 ceph-mon[27845]: pgmap v2110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2689.446700 secs 2024-04-26T12:35:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:49 smithi096 ceph-mon[32753]: pgmap v2110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2689.446700 secs 2024-04-26T12:35:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:49 smithi194 ceph-mon[33017]: pgmap v2110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2689.446700 secs 2024-04-26T12:35:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:49.610+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:49.610+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:35:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:35:49.610+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:35:51.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:51 smithi033 ceph-mon[27845]: pgmap v2111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:51.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:51 smithi096 ceph-mon[32753]: pgmap v2111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:51 smithi194 ceph-mon[33017]: pgmap v2111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:53 smithi033 ceph-mon[27845]: pgmap v2112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2691.722162 secs 2024-04-26T12:35:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:53 smithi096 ceph-mon[32753]: pgmap v2112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2691.722162 secs 2024-04-26T12:35:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:53 smithi194 ceph-mon[33017]: pgmap v2112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2691.722162 secs 2024-04-26T12:35:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2694.446828 secs 2024-04-26T12:35:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2694.446828 secs 2024-04-26T12:35:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2694.446828 secs 2024-04-26T12:35:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:55 smithi033 ceph-mon[27845]: pgmap v2113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:55 smithi096 ceph-mon[32753]: pgmap v2113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:55 smithi194 ceph-mon[33017]: pgmap v2113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:57.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:57 smithi194 ceph-mon[33017]: pgmap v2114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:57 smithi033 ceph-mon[27845]: pgmap v2114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:57 smithi096 ceph-mon[32753]: pgmap v2114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:35:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2696.722251 secs 2024-04-26T12:35:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2696.722251 secs 2024-04-26T12:35:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2696.722251 secs 2024-04-26T12:35:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:59 smithi033 ceph-mon[27845]: pgmap v2115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2699.446948 secs 2024-04-26T12:35:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:59 smithi096 ceph-mon[32753]: pgmap v2115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2699.446948 secs 2024-04-26T12:35:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:59 smithi194 ceph-mon[33017]: pgmap v2115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:35:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2699.446948 secs 2024-04-26T12:36:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:01 smithi033 ceph-mon[27845]: pgmap v2116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:01 smithi096 ceph-mon[32753]: pgmap v2116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:01 smithi194 ceph-mon[33017]: pgmap v2116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:03 smithi096 ceph-mon[32753]: pgmap v2117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2701.722351 secs 2024-04-26T12:36:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:03 smithi194 ceph-mon[33017]: pgmap v2117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2701.722351 secs 2024-04-26T12:36:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:03 smithi033 ceph-mon[27845]: pgmap v2117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2701.722351 secs 2024-04-26T12:36:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2704.447063 secs 2024-04-26T12:36:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:04.617+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:04.617+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:04.617+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:36:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2704.447063 secs 2024-04-26T12:36:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2704.447063 secs 2024-04-26T12:36:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:05 smithi194 ceph-mon[33017]: pgmap v2118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:36:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:05 smithi033 ceph-mon[27845]: pgmap v2118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:36:05.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:05 smithi096 ceph-mon[32753]: pgmap v2118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:05.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:36:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:36:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:36:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:36:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:36:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:36:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:36:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:36:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:36:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:36:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:07 smithi033 ceph-mon[27845]: pgmap v2119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:36:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:07 smithi033 ceph-mon[27845]: pgmap v2120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:07 smithi194 ceph-mon[33017]: pgmap v2119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:36:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:07 smithi194 ceph-mon[33017]: pgmap v2120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:07 smithi096 ceph-mon[32753]: pgmap v2119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:36:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:07 smithi096 ceph-mon[32753]: pgmap v2120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2706.722453 secs 2024-04-26T12:36:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2706.722453 secs 2024-04-26T12:36:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2706.722453 secs 2024-04-26T12:36:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:09 smithi194 ceph-mon[33017]: pgmap v2121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2709.447196 secs 2024-04-26T12:36:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:09 smithi033 ceph-mon[27845]: pgmap v2121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2709.447196 secs 2024-04-26T12:36:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:09 smithi096 ceph-mon[32753]: pgmap v2121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2709.447196 secs 2024-04-26T12:36:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:11 smithi194 ceph-mon[33017]: pgmap v2122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:11 smithi033 ceph-mon[27845]: pgmap v2122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:11 smithi096 ceph-mon[32753]: pgmap v2122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:13 smithi194 ceph-mon[33017]: pgmap v2123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2711.722557 secs 2024-04-26T12:36:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:13 smithi033 ceph-mon[27845]: pgmap v2123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2711.722557 secs 2024-04-26T12:36:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:13 smithi096 ceph-mon[32753]: pgmap v2123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2711.722557 secs 2024-04-26T12:36:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2714.447275 secs 2024-04-26T12:36:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2714.447275 secs 2024-04-26T12:36:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2714.447275 secs 2024-04-26T12:36:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:15 smithi194 ceph-mon[33017]: pgmap v2124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:15 smithi033 ceph-mon[27845]: pgmap v2124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:15 smithi096 ceph-mon[32753]: pgmap v2124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:36:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:17 smithi194 ceph-mon[33017]: pgmap v2125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:36:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:17 smithi033 ceph-mon[27845]: pgmap v2125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:36:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:17 smithi096 ceph-mon[32753]: pgmap v2125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:36:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2716.722658 secs 2024-04-26T12:36:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2716.722658 secs 2024-04-26T12:36:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2716.722658 secs 2024-04-26T12:36:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:19 smithi194 ceph-mon[33017]: pgmap v2126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2719.447388 secs 2024-04-26T12:36:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:19.631+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:19.631+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:19.631+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:36:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:19 smithi033 ceph-mon[27845]: pgmap v2126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2719.447388 secs 2024-04-26T12:36:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:19 smithi096 ceph-mon[32753]: pgmap v2126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2719.447388 secs 2024-04-26T12:36:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:21 smithi194 ceph-mon[33017]: pgmap v2127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:21 smithi033 ceph-mon[27845]: pgmap v2127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:21 smithi096 ceph-mon[32753]: pgmap v2127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:23 smithi194 ceph-mon[33017]: pgmap v2128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2721.722759 secs 2024-04-26T12:36:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:23 smithi033 ceph-mon[27845]: pgmap v2128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2721.722759 secs 2024-04-26T12:36:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:23 smithi096 ceph-mon[32753]: pgmap v2128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2721.722759 secs 2024-04-26T12:36:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2724.447510 secs 2024-04-26T12:36:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2724.447510 secs 2024-04-26T12:36:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2724.447510 secs 2024-04-26T12:36:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:25 smithi194 ceph-mon[33017]: pgmap v2129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:25 smithi033 ceph-mon[27845]: pgmap v2129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:25 smithi096 ceph-mon[32753]: pgmap v2129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:27.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:27 smithi194 ceph-mon[33017]: pgmap v2130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:27 smithi033 ceph-mon[27845]: pgmap v2130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:27 smithi096 ceph-mon[32753]: pgmap v2130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2726.722849 secs 2024-04-26T12:36:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2726.722849 secs 2024-04-26T12:36:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2726.722849 secs 2024-04-26T12:36:29.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:29 smithi096 ceph-mon[32753]: pgmap v2131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:29.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2729.447590 secs 2024-04-26T12:36:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:29 smithi194 ceph-mon[33017]: pgmap v2131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2729.447590 secs 2024-04-26T12:36:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:29 smithi033 ceph-mon[27845]: pgmap v2131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2729.447590 secs 2024-04-26T12:36:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:31 smithi194 ceph-mon[33017]: pgmap v2132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:31 smithi033 ceph-mon[27845]: pgmap v2132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:31.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:31 smithi096 ceph-mon[32753]: pgmap v2132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:33 smithi194 ceph-mon[33017]: pgmap v2133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2731.722947 secs 2024-04-26T12:36:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:33 smithi033 ceph-mon[27845]: pgmap v2133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2731.722947 secs 2024-04-26T12:36:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:33 smithi096 ceph-mon[32753]: pgmap v2133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2731.722947 secs 2024-04-26T12:36:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2734.447705 secs 2024-04-26T12:36:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:34.646+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:34.646+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:36:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:34.646+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2734.447705 secs 2024-04-26T12:36:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2734.447705 secs 2024-04-26T12:36:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:35 smithi194 ceph-mon[33017]: pgmap v2134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:35 smithi033 ceph-mon[27845]: pgmap v2134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:35 smithi096 ceph-mon[32753]: pgmap v2134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:37 smithi194 ceph-mon[33017]: pgmap v2135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:37 smithi033 ceph-mon[27845]: pgmap v2135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:37 smithi096 ceph-mon[32753]: pgmap v2135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2736.723034 secs 2024-04-26T12:36:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2736.723034 secs 2024-04-26T12:36:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2736.723034 secs 2024-04-26T12:36:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:39 smithi033 ceph-mon[27845]: pgmap v2136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2739.447838 secs 2024-04-26T12:36:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:39 smithi096 ceph-mon[32753]: pgmap v2136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2739.447838 secs 2024-04-26T12:36:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:39 smithi194 ceph-mon[33017]: pgmap v2136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2739.447838 secs 2024-04-26T12:36:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:41 smithi033 ceph-mon[27845]: pgmap v2137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:41 smithi096 ceph-mon[32753]: pgmap v2137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:41 smithi194 ceph-mon[33017]: pgmap v2137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:43 smithi033 ceph-mon[27845]: pgmap v2138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2741.723123 secs 2024-04-26T12:36:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:43 smithi096 ceph-mon[32753]: pgmap v2138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2741.723123 secs 2024-04-26T12:36:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:43 smithi194 ceph-mon[33017]: pgmap v2138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2741.723123 secs 2024-04-26T12:36:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2744.447953 secs 2024-04-26T12:36:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2744.447953 secs 2024-04-26T12:36:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2744.447953 secs 2024-04-26T12:36:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:45 smithi033 ceph-mon[27845]: pgmap v2139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:45 smithi096 ceph-mon[32753]: pgmap v2139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:45 smithi194 ceph-mon[33017]: pgmap v2139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:47 smithi033 ceph-mon[27845]: pgmap v2140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:47 smithi096 ceph-mon[32753]: pgmap v2140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:47 smithi194 ceph-mon[33017]: pgmap v2140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2746.723216 secs 2024-04-26T12:36:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2746.723216 secs 2024-04-26T12:36:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2746.723216 secs 2024-04-26T12:36:49.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:49.662+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:49.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:49.662+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:36:49.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:36:49.662+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:36:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:49 smithi033 ceph-mon[27845]: pgmap v2141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2749.448053 secs 2024-04-26T12:36:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:49 smithi096 ceph-mon[32753]: pgmap v2141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2749.448053 secs 2024-04-26T12:36:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:49 smithi194 ceph-mon[33017]: pgmap v2141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2749.448053 secs 2024-04-26T12:36:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:51 smithi033 ceph-mon[27845]: pgmap v2142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:51 smithi096 ceph-mon[32753]: pgmap v2142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:51 smithi194 ceph-mon[33017]: pgmap v2142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:53 smithi033 ceph-mon[27845]: pgmap v2143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2751.723305 secs 2024-04-26T12:36:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:53 smithi096 ceph-mon[32753]: pgmap v2143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:53.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2751.723305 secs 2024-04-26T12:36:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:53 smithi194 ceph-mon[33017]: pgmap v2143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2751.723305 secs 2024-04-26T12:36:54.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2754.448180 secs 2024-04-26T12:36:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2754.448180 secs 2024-04-26T12:36:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2754.448180 secs 2024-04-26T12:36:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:55 smithi033 ceph-mon[27845]: pgmap v2144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:55 smithi096 ceph-mon[32753]: pgmap v2144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:55 smithi194 ceph-mon[33017]: pgmap v2144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:57.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:57 smithi096 ceph-mon[32753]: pgmap v2145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:57.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:57 smithi194 ceph-mon[33017]: pgmap v2145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:57.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:57 smithi033 ceph-mon[27845]: pgmap v2145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:36:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2756.723368 secs 2024-04-26T12:36:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2756.723368 secs 2024-04-26T12:36:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2756.723368 secs 2024-04-26T12:36:59.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:59 smithi096 ceph-mon[32753]: pgmap v2146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:59.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2759.448297 secs 2024-04-26T12:36:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:59 smithi194 ceph-mon[33017]: pgmap v2146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2759.448297 secs 2024-04-26T12:36:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:59 smithi033 ceph-mon[27845]: pgmap v2146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:36:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2759.448297 secs 2024-04-26T12:37:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:01 smithi194 ceph-mon[33017]: pgmap v2147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:01 smithi033 ceph-mon[27845]: pgmap v2147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:01 smithi096 ceph-mon[32753]: pgmap v2147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:03 smithi194 ceph-mon[33017]: pgmap v2148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2761.723472 secs 2024-04-26T12:37:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:03 smithi033 ceph-mon[27845]: pgmap v2148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2761.723472 secs 2024-04-26T12:37:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:03 smithi096 ceph-mon[32753]: pgmap v2148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2761.723472 secs 2024-04-26T12:37:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:04.676+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:04.676+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:04.676+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:37:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2764.448375 secs 2024-04-26T12:37:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2764.448375 secs 2024-04-26T12:37:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2764.448375 secs 2024-04-26T12:37:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:05 smithi194 ceph-mon[33017]: pgmap v2149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:05 smithi033 ceph-mon[27845]: pgmap v2149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:05 smithi096 ceph-mon[32753]: pgmap v2149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:37:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:37:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:37:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:07 smithi194 ceph-mon[33017]: pgmap v2150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:37:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:37:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:37:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:07 smithi033 ceph-mon[27845]: pgmap v2150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:07.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:37:07.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:37:07.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:37:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:07 smithi096 ceph-mon[32753]: pgmap v2150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:37:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:37:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:37:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:08 smithi194 ceph-mon[33017]: pgmap v2151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:08 smithi194 ceph-mon[33017]: pgmap v2152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:08.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2766.723551 secs 2024-04-26T12:37:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:08 smithi033 ceph-mon[27845]: pgmap v2151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:08 smithi033 ceph-mon[27845]: pgmap v2152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2766.723551 secs 2024-04-26T12:37:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:08 smithi096 ceph-mon[32753]: pgmap v2151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:08 smithi096 ceph-mon[32753]: pgmap v2152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2766.723551 secs 2024-04-26T12:37:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2769.448494 secs 2024-04-26T12:37:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2769.448494 secs 2024-04-26T12:37:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2769.448494 secs 2024-04-26T12:37:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:10 smithi194 ceph-mon[33017]: pgmap v2153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:37:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:10 smithi033 ceph-mon[27845]: pgmap v2153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:37:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:10 smithi096 ceph-mon[32753]: pgmap v2153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:37:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:12 smithi194 ceph-mon[33017]: pgmap v2154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:12 smithi033 ceph-mon[27845]: pgmap v2154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:12 smithi096 ceph-mon[32753]: pgmap v2154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2771.723652 secs 2024-04-26T12:37:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2774.448571 secs 2024-04-26T12:37:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:13 smithi194 ceph-mon[33017]: pgmap v2155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2771.723652 secs 2024-04-26T12:37:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2774.448571 secs 2024-04-26T12:37:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:13 smithi033 ceph-mon[27845]: pgmap v2155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2771.723652 secs 2024-04-26T12:37:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2774.448571 secs 2024-04-26T12:37:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:13 smithi096 ceph-mon[32753]: pgmap v2155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:16 smithi033 ceph-mon[27845]: pgmap v2156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:16 smithi096 ceph-mon[32753]: pgmap v2156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:16 smithi194 ceph-mon[33017]: pgmap v2156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:18 smithi033 ceph-mon[27845]: pgmap v2157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:37:18.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2776.723760 secs 2024-04-26T12:37:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:18 smithi096 ceph-mon[32753]: pgmap v2157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:37:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2776.723760 secs 2024-04-26T12:37:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:18 smithi194 ceph-mon[33017]: pgmap v2157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:37:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2776.723760 secs 2024-04-26T12:37:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2779.448670 secs 2024-04-26T12:37:19.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2779.448670 secs 2024-04-26T12:37:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:19.692+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:19.692+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:37:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:19.692+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2779.448670 secs 2024-04-26T12:37:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:20 smithi033 ceph-mon[27845]: pgmap v2158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:20 smithi096 ceph-mon[32753]: pgmap v2158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:20 smithi194 ceph-mon[33017]: pgmap v2158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:22 smithi033 ceph-mon[27845]: pgmap v2159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:22 smithi096 ceph-mon[32753]: pgmap v2159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:22 smithi194 ceph-mon[33017]: pgmap v2159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2781.723858 secs 2024-04-26T12:37:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2781.723858 secs 2024-04-26T12:37:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2781.723858 secs 2024-04-26T12:37:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2784.448785 secs 2024-04-26T12:37:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:24 smithi033 ceph-mon[27845]: pgmap v2160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2784.448785 secs 2024-04-26T12:37:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:24 smithi096 ceph-mon[32753]: pgmap v2160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2784.448785 secs 2024-04-26T12:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:24 smithi194 ceph-mon[33017]: pgmap v2160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:26 smithi033 ceph-mon[27845]: pgmap v2161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:26 smithi096 ceph-mon[32753]: pgmap v2161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:26 smithi194 ceph-mon[33017]: pgmap v2161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:28 smithi033 ceph-mon[27845]: pgmap v2162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2786.723947 secs 2024-04-26T12:37:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:28 smithi096 ceph-mon[32753]: pgmap v2162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2786.723947 secs 2024-04-26T12:37:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:28 smithi194 ceph-mon[33017]: pgmap v2162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2786.723947 secs 2024-04-26T12:37:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2789.448886 secs 2024-04-26T12:37:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2789.448886 secs 2024-04-26T12:37:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2789.448886 secs 2024-04-26T12:37:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:30 smithi096 ceph-mon[32753]: pgmap v2163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:30 smithi194 ceph-mon[33017]: pgmap v2163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:30 smithi033 ceph-mon[27845]: pgmap v2163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:32 smithi194 ceph-mon[33017]: pgmap v2164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:32 smithi033 ceph-mon[27845]: pgmap v2164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:32 smithi096 ceph-mon[32753]: pgmap v2164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2791.724027 secs 2024-04-26T12:37:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2791.724027 secs 2024-04-26T12:37:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2791.724027 secs 2024-04-26T12:37:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:34.696+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:34.696+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:34.696+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:37:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2794.449022 secs 2024-04-26T12:37:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:34 smithi194 ceph-mon[33017]: pgmap v2165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2794.449022 secs 2024-04-26T12:37:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:34 smithi033 ceph-mon[27845]: pgmap v2165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2794.449022 secs 2024-04-26T12:37:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:34 smithi096 ceph-mon[32753]: pgmap v2165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:36 smithi194 ceph-mon[33017]: pgmap v2166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:36 smithi033 ceph-mon[27845]: pgmap v2166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:36 smithi096 ceph-mon[32753]: pgmap v2166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:38 smithi194 ceph-mon[33017]: pgmap v2167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2796.724113 secs 2024-04-26T12:37:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:38 smithi033 ceph-mon[27845]: pgmap v2167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2796.724113 secs 2024-04-26T12:37:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:38 smithi096 ceph-mon[32753]: pgmap v2167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2796.724113 secs 2024-04-26T12:37:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2799.449160 secs 2024-04-26T12:37:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2799.449160 secs 2024-04-26T12:37:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2799.449160 secs 2024-04-26T12:37:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:40 smithi033 ceph-mon[27845]: pgmap v2168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:40 smithi194 ceph-mon[33017]: pgmap v2168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:40.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:40 smithi096 ceph-mon[32753]: pgmap v2168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:42 smithi194 ceph-mon[33017]: pgmap v2169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:42 smithi033 ceph-mon[27845]: pgmap v2169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:42 smithi096 ceph-mon[32753]: pgmap v2169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2801.724175 secs 2024-04-26T12:37:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2801.724175 secs 2024-04-26T12:37:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2801.724175 secs 2024-04-26T12:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2804.449312 secs 2024-04-26T12:37:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:44 smithi194 ceph-mon[33017]: pgmap v2170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2804.449312 secs 2024-04-26T12:37:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:44 smithi033 ceph-mon[27845]: pgmap v2170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2804.449312 secs 2024-04-26T12:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:44 smithi096 ceph-mon[32753]: pgmap v2170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:46 smithi194 ceph-mon[33017]: pgmap v2171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:46 smithi033 ceph-mon[27845]: pgmap v2171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:46 smithi096 ceph-mon[32753]: pgmap v2171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:48 smithi194 ceph-mon[33017]: pgmap v2172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2806.724271 secs 2024-04-26T12:37:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:48 smithi033 ceph-mon[27845]: pgmap v2172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2806.724271 secs 2024-04-26T12:37:48.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:48 smithi096 ceph-mon[32753]: pgmap v2172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:48.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2806.724271 secs 2024-04-26T12:37:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:49.698+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:49.698+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:37:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:37:49.698+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:37:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2809.449452 secs 2024-04-26T12:37:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2809.449452 secs 2024-04-26T12:37:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2809.449452 secs 2024-04-26T12:37:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:50 smithi194 ceph-mon[33017]: pgmap v2173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:50 smithi033 ceph-mon[27845]: pgmap v2173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:50 smithi096 ceph-mon[32753]: pgmap v2173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:37:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:52 smithi194 ceph-mon[33017]: pgmap v2174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:52 smithi033 ceph-mon[27845]: pgmap v2174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:52 smithi096 ceph-mon[32753]: pgmap v2174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:37:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2811.724346 secs 2024-04-26T12:37:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2811.724346 secs 2024-04-26T12:37:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2811.724346 secs 2024-04-26T12:37:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2814.449570 secs 2024-04-26T12:37:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:54 smithi194 ceph-mon[33017]: pgmap v2175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2814.449570 secs 2024-04-26T12:37:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:54 smithi033 ceph-mon[27845]: pgmap v2175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2814.449570 secs 2024-04-26T12:37:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:54 smithi096 ceph-mon[32753]: pgmap v2175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:56 smithi194 ceph-mon[33017]: pgmap v2176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:56 smithi033 ceph-mon[27845]: pgmap v2176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:56 smithi096 ceph-mon[32753]: pgmap v2176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:37:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:58 smithi194 ceph-mon[33017]: pgmap v2177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2816.724448 secs 2024-04-26T12:37:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:37:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2819.449640 secs 2024-04-26T12:37:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:58 smithi033 ceph-mon[27845]: pgmap v2177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2816.724448 secs 2024-04-26T12:37:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:37:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2819.449640 secs 2024-04-26T12:37:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:58 smithi096 ceph-mon[32753]: pgmap v2177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:37:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2816.724448 secs 2024-04-26T12:37:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:37:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2819.449640 secs 2024-04-26T12:38:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:00 smithi194 ceph-mon[33017]: pgmap v2178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:00 smithi033 ceph-mon[27845]: pgmap v2178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:00 smithi096 ceph-mon[32753]: pgmap v2178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:02 smithi194 ceph-mon[33017]: pgmap v2179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:38:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:02 smithi033 ceph-mon[27845]: pgmap v2179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:38:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:02 smithi096 ceph-mon[32753]: pgmap v2179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:38:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2821.724540 secs 2024-04-26T12:38:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2824.449735 secs 2024-04-26T12:38:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2821.724540 secs 2024-04-26T12:38:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2824.449735 secs 2024-04-26T12:38:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2821.724540 secs 2024-04-26T12:38:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2824.449735 secs 2024-04-26T12:38:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:04.713+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:04.713+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:04.713+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:38:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:04 smithi194 ceph-mon[33017]: pgmap v2180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:04 smithi033 ceph-mon[27845]: pgmap v2180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:04 smithi096 ceph-mon[32753]: pgmap v2180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:06 smithi194 ceph-mon[33017]: pgmap v2181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:06 smithi033 ceph-mon[27845]: pgmap v2181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:06 smithi096 ceph-mon[32753]: pgmap v2181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:08 smithi194 ceph-mon[33017]: pgmap v2182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2826.724642 secs 2024-04-26T12:38:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:38:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2829.449827 secs 2024-04-26T12:38:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:08 smithi033 ceph-mon[27845]: pgmap v2182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2826.724642 secs 2024-04-26T12:38:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:38:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2829.449827 secs 2024-04-26T12:38:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:08 smithi096 ceph-mon[32753]: pgmap v2182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2826.724642 secs 2024-04-26T12:38:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:38:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2829.449827 secs 2024-04-26T12:38:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:38:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:38:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:38:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:38:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:38:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:38:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:38:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:38:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:38:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:10 smithi194 ceph-mon[33017]: pgmap v2183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:10 smithi194 ceph-mon[33017]: pgmap v2184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:38:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:10 smithi033 ceph-mon[27845]: pgmap v2183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:10 smithi033 ceph-mon[27845]: pgmap v2184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:38:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:10 smithi096 ceph-mon[32753]: pgmap v2183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:10 smithi096 ceph-mon[32753]: pgmap v2184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T12:38:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:12 smithi194 ceph-mon[33017]: pgmap v2185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:12 smithi033 ceph-mon[27845]: pgmap v2185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:12 smithi096 ceph-mon[32753]: pgmap v2185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2831.724741 secs 2024-04-26T12:38:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2834.449962 secs 2024-04-26T12:38:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2831.724741 secs 2024-04-26T12:38:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2834.449962 secs 2024-04-26T12:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2831.724741 secs 2024-04-26T12:38:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2834.449962 secs 2024-04-26T12:38:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:14 smithi033 ceph-mon[27845]: pgmap v2186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:14 smithi096 ceph-mon[32753]: pgmap v2186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:14 smithi194 ceph-mon[33017]: pgmap v2186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:16 smithi033 ceph-mon[27845]: pgmap v2187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:16 smithi096 ceph-mon[32753]: pgmap v2187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:16 smithi194 ceph-mon[33017]: pgmap v2187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:18 smithi033 ceph-mon[27845]: pgmap v2188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2836.724841 secs 2024-04-26T12:38:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2839.450025 secs 2024-04-26T12:38:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:18 smithi096 ceph-mon[32753]: pgmap v2188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2836.724841 secs 2024-04-26T12:38:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2839.450025 secs 2024-04-26T12:38:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:18 smithi194 ceph-mon[33017]: pgmap v2188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:38:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2836.724841 secs 2024-04-26T12:38:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2839.450025 secs 2024-04-26T12:38:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:19.730+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:19.730+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:19.730+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:38:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:20 smithi033 ceph-mon[27845]: pgmap v2189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:38:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:20 smithi096 ceph-mon[32753]: pgmap v2189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:38:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:20 smithi194 ceph-mon[33017]: pgmap v2189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:38:22.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:22.278+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:38:22.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:22.278+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:38:22.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:22.278+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:38:22.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:22.278+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:38:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:22 smithi033 ceph-mon[27845]: pgmap v2190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:22 smithi096 ceph-mon[32753]: pgmap v2190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:22 smithi194 ceph-mon[33017]: pgmap v2190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2841.724903 secs 2024-04-26T12:38:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2844.450147 secs 2024-04-26T12:38:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2841.724903 secs 2024-04-26T12:38:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2844.450147 secs 2024-04-26T12:38:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2841.724903 secs 2024-04-26T12:38:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2844.450147 secs 2024-04-26T12:38:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:24 smithi033 ceph-mon[27845]: pgmap v2191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:24 smithi096 ceph-mon[32753]: pgmap v2191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:24 smithi194 ceph-mon[33017]: pgmap v2191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:26 smithi033 ceph-mon[27845]: pgmap v2192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:26 smithi096 ceph-mon[32753]: pgmap v2192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:26 smithi194 ceph-mon[33017]: pgmap v2192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:27 smithi033 ceph-mon[27845]: pgmap v2193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:27 smithi096 ceph-mon[32753]: pgmap v2193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:28.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:27 smithi194 ceph-mon[33017]: pgmap v2193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2846.724990 secs 2024-04-26T12:38:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2849.450218 secs 2024-04-26T12:38:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2846.724990 secs 2024-04-26T12:38:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2849.450218 secs 2024-04-26T12:38:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2846.724990 secs 2024-04-26T12:38:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2849.450218 secs 2024-04-26T12:38:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:29 smithi033 ceph-mon[27845]: pgmap v2194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:29 smithi096 ceph-mon[32753]: pgmap v2194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:29 smithi194 ceph-mon[33017]: pgmap v2194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:32 smithi194 ceph-mon[33017]: pgmap v2195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:32 smithi033 ceph-mon[27845]: pgmap v2195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:32 smithi096 ceph-mon[32753]: pgmap v2195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2851.725084 secs 2024-04-26T12:38:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2851.725084 secs 2024-04-26T12:38:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2851.725084 secs 2024-04-26T12:38:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:34 smithi194 ceph-mon[33017]: pgmap v2196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2854.450334 secs 2024-04-26T12:38:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:34 smithi033 ceph-mon[27845]: pgmap v2196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2854.450334 secs 2024-04-26T12:38:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:34 smithi096 ceph-mon[32753]: pgmap v2196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2854.450334 secs 2024-04-26T12:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:34.742+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:34.742+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:34.742+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:38:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:36 smithi194 ceph-mon[33017]: pgmap v2197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:36 smithi033 ceph-mon[27845]: pgmap v2197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:36 smithi096 ceph-mon[32753]: pgmap v2197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:38 smithi194 ceph-mon[33017]: pgmap v2198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2856.725147 secs 2024-04-26T12:38:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:38 smithi033 ceph-mon[27845]: pgmap v2198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2856.725147 secs 2024-04-26T12:38:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:38 smithi096 ceph-mon[32753]: pgmap v2198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2856.725147 secs 2024-04-26T12:38:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2859.450451 secs 2024-04-26T12:38:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2859.450451 secs 2024-04-26T12:38:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2859.450451 secs 2024-04-26T12:38:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:40 smithi194 ceph-mon[33017]: pgmap v2199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:40 smithi033 ceph-mon[27845]: pgmap v2199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:40 smithi096 ceph-mon[32753]: pgmap v2199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:42 smithi194 ceph-mon[33017]: pgmap v2200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:42 smithi033 ceph-mon[27845]: pgmap v2200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:42 smithi096 ceph-mon[32753]: pgmap v2200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2861.725229 secs 2024-04-26T12:38:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2861.725229 secs 2024-04-26T12:38:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2861.725229 secs 2024-04-26T12:38:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:44 smithi033 ceph-mon[27845]: pgmap v2201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2864.450551 secs 2024-04-26T12:38:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:44 smithi096 ceph-mon[32753]: pgmap v2201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2864.450551 secs 2024-04-26T12:38:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:44 smithi194 ceph-mon[33017]: pgmap v2201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2864.450551 secs 2024-04-26T12:38:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:46 smithi033 ceph-mon[27845]: pgmap v2202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:46 smithi096 ceph-mon[32753]: pgmap v2202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:46 smithi194 ceph-mon[33017]: pgmap v2202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:48 smithi033 ceph-mon[27845]: pgmap v2203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2866.725313 secs 2024-04-26T12:38:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:48 smithi096 ceph-mon[32753]: pgmap v2203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2866.725313 secs 2024-04-26T12:38:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:48 smithi194 ceph-mon[33017]: pgmap v2203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2866.725313 secs 2024-04-26T12:38:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2869.450669 secs 2024-04-26T12:38:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2869.450669 secs 2024-04-26T12:38:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2869.450669 secs 2024-04-26T12:38:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:49.745+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:49.745+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:38:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:38:49.745+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:38:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:50 smithi033 ceph-mon[27845]: pgmap v2204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:50.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:50 smithi096 ceph-mon[32753]: pgmap v2204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:50 smithi194 ceph-mon[33017]: pgmap v2204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:52 smithi033 ceph-mon[27845]: pgmap v2205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:52 smithi096 ceph-mon[32753]: pgmap v2205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:52 smithi194 ceph-mon[33017]: pgmap v2205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2871.725401 secs 2024-04-26T12:38:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:53 smithi194 ceph-mon[33017]: pgmap v2206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:54.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2874.450789 secs 2024-04-26T12:38:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2871.725401 secs 2024-04-26T12:38:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:53 smithi033 ceph-mon[27845]: pgmap v2206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2874.450789 secs 2024-04-26T12:38:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2871.725401 secs 2024-04-26T12:38:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:53 smithi096 ceph-mon[32753]: pgmap v2206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2874.450789 secs 2024-04-26T12:38:56.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:56 smithi194 ceph-mon[33017]: pgmap v2207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:56 smithi033 ceph-mon[27845]: pgmap v2207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:56 smithi096 ceph-mon[32753]: pgmap v2207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:38:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:58 smithi194 ceph-mon[33017]: pgmap v2208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2876.725462 secs 2024-04-26T12:38:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:58 smithi033 ceph-mon[27845]: pgmap v2208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2876.725462 secs 2024-04-26T12:38:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:58 smithi096 ceph-mon[32753]: pgmap v2208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:38:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2876.725462 secs 2024-04-26T12:38:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2879.450914 secs 2024-04-26T12:38:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2879.450914 secs 2024-04-26T12:38:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2879.450914 secs 2024-04-26T12:39:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:00 smithi194 ceph-mon[33017]: pgmap v2209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:00 smithi033 ceph-mon[27845]: pgmap v2209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:00 smithi096 ceph-mon[32753]: pgmap v2209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:02 smithi194 ceph-mon[33017]: pgmap v2210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:02 smithi033 ceph-mon[27845]: pgmap v2210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:02 smithi096 ceph-mon[32753]: pgmap v2210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:03 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 2881.725544 secs 2024-04-26T12:39:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:03 smithi194 ceph-mon[33017]: slow request 2881.725544 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:39:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:03 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 2881.725544 secs 2024-04-26T12:39:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:03 smithi033 ceph-mon[27845]: slow request 2881.725544 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:39:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:03 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 2881.725544 secs 2024-04-26T12:39:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:03 smithi096 ceph-mon[32753]: slow request 2881.725544 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T12:39:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:04 smithi194 ceph-mon[33017]: pgmap v2211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 2884.451045 secs 2024-04-26T12:39:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:04 smithi194 ceph-mon[33017]: slow request 2884.451045 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:39:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:04 smithi033 ceph-mon[27845]: pgmap v2211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 2884.451045 secs 2024-04-26T12:39:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:04 smithi033 ceph-mon[27845]: slow request 2884.451045 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:39:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:04 smithi096 ceph-mon[32753]: pgmap v2211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 2884.451045 secs 2024-04-26T12:39:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:04 smithi096 ceph-mon[32753]: slow request 2884.451045 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T12:39:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:04.757+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:04.757+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:39:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:04.757+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:06 smithi033 ceph-mon[27845]: pgmap v2212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:06 smithi096 ceph-mon[32753]: pgmap v2212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:06 smithi194 ceph-mon[33017]: pgmap v2212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:07.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:07.285+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:39:07.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:07.285+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:39:07.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:07.285+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:39:07.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:07.285+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:39:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:08 smithi033 ceph-mon[27845]: pgmap v2213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2886.725647 secs 2024-04-26T12:39:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:08 smithi096 ceph-mon[32753]: pgmap v2213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2886.725647 secs 2024-04-26T12:39:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:08 smithi194 ceph-mon[33017]: pgmap v2213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2886.725647 secs 2024-04-26T12:39:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2889.451156 secs 2024-04-26T12:39:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2889.451156 secs 2024-04-26T12:39:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2889.451156 secs 2024-04-26T12:39:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:10 smithi033 ceph-mon[27845]: pgmap v2214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:39:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:10 smithi096 ceph-mon[32753]: pgmap v2214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:39:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:10 smithi194 ceph-mon[33017]: pgmap v2214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:39:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:39:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:39:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:39:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:39:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:39:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:39:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:39:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:39:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:39:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:12 smithi033 ceph-mon[27845]: pgmap v2215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:39:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:12 smithi033 ceph-mon[27845]: pgmap v2216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:12 smithi096 ceph-mon[32753]: pgmap v2215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:39:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:12 smithi096 ceph-mon[32753]: pgmap v2216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:12 smithi194 ceph-mon[33017]: pgmap v2215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:39:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:12 smithi194 ceph-mon[33017]: pgmap v2216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2891.725731 secs 2024-04-26T12:39:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2891.725731 secs 2024-04-26T12:39:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2891.725731 secs 2024-04-26T12:39:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:14 smithi033 ceph-mon[27845]: pgmap v2217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2894.451263 secs 2024-04-26T12:39:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:14 smithi096 ceph-mon[32753]: pgmap v2217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2894.451263 secs 2024-04-26T12:39:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:14 smithi194 ceph-mon[33017]: pgmap v2217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2894.451263 secs 2024-04-26T12:39:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:16 smithi033 ceph-mon[27845]: pgmap v2218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:16 smithi096 ceph-mon[32753]: pgmap v2218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:16 smithi194 ceph-mon[33017]: pgmap v2218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:18 smithi033 ceph-mon[27845]: pgmap v2219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2896.725817 secs 2024-04-26T12:39:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:18 smithi096 ceph-mon[32753]: pgmap v2219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2896.725817 secs 2024-04-26T12:39:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:18 smithi194 ceph-mon[33017]: pgmap v2219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2896.725817 secs 2024-04-26T12:39:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2899.451362 secs 2024-04-26T12:39:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2899.451362 secs 2024-04-26T12:39:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2899.451362 secs 2024-04-26T12:39:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:19.768+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:19.768+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:19.768+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:39:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:20 smithi033 ceph-mon[27845]: pgmap v2220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:20 smithi096 ceph-mon[32753]: pgmap v2220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:20 smithi194 ceph-mon[33017]: pgmap v2220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:39:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:22 smithi096 ceph-mon[32753]: pgmap v2221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:39:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:22 smithi194 ceph-mon[33017]: pgmap v2221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:39:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:22 smithi033 ceph-mon[27845]: pgmap v2221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:39:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2901.725906 secs 2024-04-26T12:39:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2901.725906 secs 2024-04-26T12:39:23.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2901.725906 secs 2024-04-26T12:39:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:24 smithi194 ceph-mon[33017]: pgmap v2222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2904.451471 secs 2024-04-26T12:39:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:24 smithi033 ceph-mon[27845]: pgmap v2222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2904.451471 secs 2024-04-26T12:39:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:24 smithi096 ceph-mon[32753]: pgmap v2222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2904.451471 secs 2024-04-26T12:39:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:26 smithi194 ceph-mon[33017]: pgmap v2223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:26.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:26 smithi033 ceph-mon[27845]: pgmap v2223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:26 smithi096 ceph-mon[32753]: pgmap v2223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:28.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:28 smithi033 ceph-mon[27845]: pgmap v2224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:28.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2906.726006 secs 2024-04-26T12:39:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:28 smithi194 ceph-mon[33017]: pgmap v2224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2906.726006 secs 2024-04-26T12:39:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:28 smithi096 ceph-mon[32753]: pgmap v2224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2906.726006 secs 2024-04-26T12:39:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2909.451587 secs 2024-04-26T12:39:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2909.451587 secs 2024-04-26T12:39:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2909.451587 secs 2024-04-26T12:39:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:30 smithi194 ceph-mon[33017]: pgmap v2225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:30 smithi033 ceph-mon[27845]: pgmap v2225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:30 smithi096 ceph-mon[32753]: pgmap v2225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:32.485 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:32 smithi096 ceph-mon[32753]: pgmap v2226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:32 smithi194 ceph-mon[33017]: pgmap v2226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:32 smithi033 ceph-mon[27845]: pgmap v2226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2911.726096 secs 2024-04-26T12:39:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2911.726096 secs 2024-04-26T12:39:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2911.726096 secs 2024-04-26T12:39:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:34 smithi194 ceph-mon[33017]: pgmap v2227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2914.451702 secs 2024-04-26T12:39:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:34 smithi033 ceph-mon[27845]: pgmap v2227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:34.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2914.451702 secs 2024-04-26T12:39:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:34 smithi096 ceph-mon[32753]: pgmap v2227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2914.451702 secs 2024-04-26T12:39:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:34.783+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:34.783+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:39:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:34.783+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:36 smithi194 ceph-mon[33017]: pgmap v2228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:36 smithi033 ceph-mon[27845]: pgmap v2228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:36 smithi096 ceph-mon[32753]: pgmap v2228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:38 smithi194 ceph-mon[33017]: pgmap v2229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2916.726182 secs 2024-04-26T12:39:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:38 smithi033 ceph-mon[27845]: pgmap v2229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2916.726182 secs 2024-04-26T12:39:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:38 smithi096 ceph-mon[32753]: pgmap v2229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2916.726182 secs 2024-04-26T12:39:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2919.451800 secs 2024-04-26T12:39:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2919.451800 secs 2024-04-26T12:39:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2919.451800 secs 2024-04-26T12:39:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:40 smithi194 ceph-mon[33017]: pgmap v2230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:40 smithi033 ceph-mon[27845]: pgmap v2230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:40 smithi096 ceph-mon[32753]: pgmap v2230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:42 smithi194 ceph-mon[33017]: pgmap v2231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:42 smithi033 ceph-mon[27845]: pgmap v2231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:42 smithi096 ceph-mon[32753]: pgmap v2231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2921.726267 secs 2024-04-26T12:39:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2921.726267 secs 2024-04-26T12:39:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2921.726267 secs 2024-04-26T12:39:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:44 smithi194 ceph-mon[33017]: pgmap v2232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2924.451915 secs 2024-04-26T12:39:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:44 smithi033 ceph-mon[27845]: pgmap v2232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2924.451915 secs 2024-04-26T12:39:44.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:44 smithi096 ceph-mon[32753]: pgmap v2232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:44.687 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2924.451915 secs 2024-04-26T12:39:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:46 smithi194 ceph-mon[33017]: pgmap v2233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:46 smithi033 ceph-mon[27845]: pgmap v2233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:46 smithi096 ceph-mon[32753]: pgmap v2233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:48 smithi194 ceph-mon[33017]: pgmap v2234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2926.726351 secs 2024-04-26T12:39:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:48 smithi033 ceph-mon[27845]: pgmap v2234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2926.726351 secs 2024-04-26T12:39:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:48 smithi096 ceph-mon[32753]: pgmap v2234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2926.726351 secs 2024-04-26T12:39:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2929.452031 secs 2024-04-26T12:39:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2929.452031 secs 2024-04-26T12:39:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2929.452031 secs 2024-04-26T12:39:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:49.799+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:49.799+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:39:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:39:49.799+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:39:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:50 smithi194 ceph-mon[33017]: pgmap v2235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:50 smithi033 ceph-mon[27845]: pgmap v2235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:50 smithi096 ceph-mon[32753]: pgmap v2235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:52 smithi194 ceph-mon[33017]: pgmap v2236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:52 smithi033 ceph-mon[27845]: pgmap v2236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:52 smithi096 ceph-mon[32753]: pgmap v2236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2931.726438 secs 2024-04-26T12:39:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2931.726438 secs 2024-04-26T12:39:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2931.726438 secs 2024-04-26T12:39:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:54 smithi194 ceph-mon[33017]: pgmap v2237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2934.452143 secs 2024-04-26T12:39:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:54 smithi033 ceph-mon[27845]: pgmap v2237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2934.452143 secs 2024-04-26T12:39:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:54 smithi096 ceph-mon[32753]: pgmap v2237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2934.452143 secs 2024-04-26T12:39:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:56 smithi194 ceph-mon[33017]: pgmap v2238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:56 smithi033 ceph-mon[27845]: pgmap v2238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:56 smithi096 ceph-mon[32753]: pgmap v2238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:39:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:58 smithi194 ceph-mon[33017]: pgmap v2239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2936.726495 secs 2024-04-26T12:39:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:58 smithi033 ceph-mon[27845]: pgmap v2239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2936.726495 secs 2024-04-26T12:39:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:58 smithi096 ceph-mon[32753]: pgmap v2239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:39:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2936.726495 secs 2024-04-26T12:39:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 2939.452243 secs 2024-04-26T12:39:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:39:59 smithi194 ceph-mon[33017]: slow request 2881.180032 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:39:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 2939.452243 secs 2024-04-26T12:39:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:39:59 smithi033 ceph-mon[27845]: slow request 2881.180032 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:39:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 2939.452243 secs 2024-04-26T12:39:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:39:59 smithi096 ceph-mon[32753]: slow request 2881.180032 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T12:40:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:00 smithi194 ceph-mon[33017]: pgmap v2240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:40:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:00 smithi033 ceph-mon[27845]: pgmap v2240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:40:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:00 smithi096 ceph-mon[32753]: pgmap v2240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:40:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:02 smithi033 ceph-mon[27845]: pgmap v2241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:02 smithi194 ceph-mon[33017]: pgmap v2241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:02 smithi096 ceph-mon[32753]: pgmap v2241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2941.726578 secs 2024-04-26T12:40:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2941.726578 secs 2024-04-26T12:40:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2941.726578 secs 2024-04-26T12:40:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:04 smithi033 ceph-mon[27845]: pgmap v2242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2944.452360 secs 2024-04-26T12:40:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:04 smithi096 ceph-mon[32753]: pgmap v2242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2944.452360 secs 2024-04-26T12:40:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:04.804+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:04.804+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:04.804+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:40:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:04 smithi194 ceph-mon[33017]: pgmap v2242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2944.452360 secs 2024-04-26T12:40:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:06 smithi033 ceph-mon[27845]: pgmap v2243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:06 smithi096 ceph-mon[32753]: pgmap v2243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:06 smithi194 ceph-mon[33017]: pgmap v2243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:08 smithi033 ceph-mon[27845]: pgmap v2244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2946.726672 secs 2024-04-26T12:40:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:08 smithi096 ceph-mon[32753]: pgmap v2244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2946.726672 secs 2024-04-26T12:40:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:08 smithi194 ceph-mon[33017]: pgmap v2244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2946.726672 secs 2024-04-26T12:40:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2949.452476 secs 2024-04-26T12:40:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2949.452476 secs 2024-04-26T12:40:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2949.452476 secs 2024-04-26T12:40:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:10 smithi033 ceph-mon[27845]: pgmap v2245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:10 smithi096 ceph-mon[32753]: pgmap v2245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:10 smithi194 ceph-mon[33017]: pgmap v2245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:40:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:40:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:40:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:12 smithi033 ceph-mon[27845]: pgmap v2246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:40:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:40:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:12 smithi096 ceph-mon[32753]: pgmap v2246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:40:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:40:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:12 smithi194 ceph-mon[33017]: pgmap v2246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:40:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:40:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:13 smithi033 ceph-mon[27845]: pgmap v2247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:13 smithi033 ceph-mon[27845]: pgmap v2248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:40:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2951.726763 secs 2024-04-26T12:40:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:13 smithi096 ceph-mon[32753]: pgmap v2247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:13 smithi096 ceph-mon[32753]: pgmap v2248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:40:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2951.726763 secs 2024-04-26T12:40:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:13 smithi194 ceph-mon[33017]: pgmap v2247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:13 smithi194 ceph-mon[33017]: pgmap v2248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:40:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2951.726763 secs 2024-04-26T12:40:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2954.452550 secs 2024-04-26T12:40:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2954.452550 secs 2024-04-26T12:40:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2954.452550 secs 2024-04-26T12:40:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:15 smithi194 ceph-mon[33017]: pgmap v2249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:40:15.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:15 smithi033 ceph-mon[27845]: pgmap v2249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:40:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:15 smithi096 ceph-mon[32753]: pgmap v2249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:40:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:17 smithi194 ceph-mon[33017]: pgmap v2250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:17 smithi033 ceph-mon[27845]: pgmap v2250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:17 smithi096 ceph-mon[32753]: pgmap v2250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2956.726868 secs 2024-04-26T12:40:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2956.726868 secs 2024-04-26T12:40:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2956.726868 secs 2024-04-26T12:40:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:19.820+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:19.820+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:19.820+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:40:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2959.452623 secs 2024-04-26T12:40:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:19 smithi194 ceph-mon[33017]: pgmap v2251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2959.452623 secs 2024-04-26T12:40:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:19 smithi033 ceph-mon[27845]: pgmap v2251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2959.452623 secs 2024-04-26T12:40:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:19 smithi096 ceph-mon[32753]: pgmap v2251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:21 smithi194 ceph-mon[33017]: pgmap v2252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:21 smithi033 ceph-mon[27845]: pgmap v2252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:21 smithi096 ceph-mon[32753]: pgmap v2252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:23 smithi194 ceph-mon[33017]: pgmap v2253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:40:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2961.726976 secs 2024-04-26T12:40:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:23 smithi033 ceph-mon[27845]: pgmap v2253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:40:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2961.726976 secs 2024-04-26T12:40:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:23 smithi096 ceph-mon[32753]: pgmap v2253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:40:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2961.726976 secs 2024-04-26T12:40:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2964.452742 secs 2024-04-26T12:40:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2964.452742 secs 2024-04-26T12:40:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2964.452742 secs 2024-04-26T12:40:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:25 smithi194 ceph-mon[33017]: pgmap v2254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:25 smithi033 ceph-mon[27845]: pgmap v2254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:25 smithi096 ceph-mon[32753]: pgmap v2254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:27 smithi194 ceph-mon[33017]: pgmap v2255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:27 smithi033 ceph-mon[27845]: pgmap v2255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:27 smithi096 ceph-mon[32753]: pgmap v2255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:28.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2966.727039 secs 2024-04-26T12:40:28.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2966.727039 secs 2024-04-26T12:40:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2966.727039 secs 2024-04-26T12:40:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2969.452857 secs 2024-04-26T12:40:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:29 smithi194 ceph-mon[33017]: pgmap v2256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2969.452857 secs 2024-04-26T12:40:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:29 smithi033 ceph-mon[27845]: pgmap v2256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2969.452857 secs 2024-04-26T12:40:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:29 smithi096 ceph-mon[32753]: pgmap v2256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:31 smithi033 ceph-mon[27845]: pgmap v2257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:31 smithi194 ceph-mon[33017]: pgmap v2257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:31 smithi096 ceph-mon[32753]: pgmap v2257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:33 smithi194 ceph-mon[33017]: pgmap v2258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2971.727125 secs 2024-04-26T12:40:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:33 smithi033 ceph-mon[27845]: pgmap v2258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2971.727125 secs 2024-04-26T12:40:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:33 smithi096 ceph-mon[32753]: pgmap v2258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2971.727125 secs 2024-04-26T12:40:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:34.835+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:34.835+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:40:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:34.835+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2974.452982 secs 2024-04-26T12:40:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2974.452982 secs 2024-04-26T12:40:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2974.452982 secs 2024-04-26T12:40:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:35 smithi194 ceph-mon[33017]: pgmap v2259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:35 smithi033 ceph-mon[27845]: pgmap v2259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:35 smithi096 ceph-mon[32753]: pgmap v2259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:37 smithi194 ceph-mon[33017]: pgmap v2260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:37 smithi033 ceph-mon[27845]: pgmap v2260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:37 smithi096 ceph-mon[32753]: pgmap v2260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2976.727209 secs 2024-04-26T12:40:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2976.727209 secs 2024-04-26T12:40:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2976.727209 secs 2024-04-26T12:40:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2979.453101 secs 2024-04-26T12:40:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:39 smithi194 ceph-mon[33017]: pgmap v2261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2979.453101 secs 2024-04-26T12:40:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:39 smithi033 ceph-mon[27845]: pgmap v2261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2979.453101 secs 2024-04-26T12:40:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:39 smithi096 ceph-mon[32753]: pgmap v2261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:41 smithi194 ceph-mon[33017]: pgmap v2262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:41 smithi033 ceph-mon[27845]: pgmap v2262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:41.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:41 smithi096 ceph-mon[32753]: pgmap v2262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:43 smithi194 ceph-mon[33017]: pgmap v2263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2981.727310 secs 2024-04-26T12:40:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:43 smithi033 ceph-mon[27845]: pgmap v2263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2981.727310 secs 2024-04-26T12:40:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:43 smithi096 ceph-mon[32753]: pgmap v2263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2981.727310 secs 2024-04-26T12:40:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2984.453221 secs 2024-04-26T12:40:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2984.453221 secs 2024-04-26T12:40:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2984.453221 secs 2024-04-26T12:40:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:45 smithi194 ceph-mon[33017]: pgmap v2264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:45 smithi033 ceph-mon[27845]: pgmap v2264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:45 smithi096 ceph-mon[32753]: pgmap v2264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:47 smithi194 ceph-mon[33017]: pgmap v2265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:47 smithi033 ceph-mon[27845]: pgmap v2265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:47 smithi096 ceph-mon[32753]: pgmap v2265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2986.727410 secs 2024-04-26T12:40:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2989.453375 secs 2024-04-26T12:40:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2986.727410 secs 2024-04-26T12:40:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2989.453375 secs 2024-04-26T12:40:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2986.727410 secs 2024-04-26T12:40:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2989.453375 secs 2024-04-26T12:40:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:49.841+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:49.842+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:40:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:40:49.842+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:40:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:49 smithi194 ceph-mon[33017]: pgmap v2266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:49 smithi033 ceph-mon[27845]: pgmap v2266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:49 smithi096 ceph-mon[32753]: pgmap v2266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:51 smithi194 ceph-mon[33017]: pgmap v2267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:51 smithi033 ceph-mon[27845]: pgmap v2267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:51 smithi096 ceph-mon[32753]: pgmap v2267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:40:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:53 smithi194 ceph-mon[33017]: pgmap v2268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2991.727495 secs 2024-04-26T12:40:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2994.453485 secs 2024-04-26T12:40:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:53 smithi033 ceph-mon[27845]: pgmap v2268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2991.727495 secs 2024-04-26T12:40:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2994.453485 secs 2024-04-26T12:40:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:53 smithi096 ceph-mon[32753]: pgmap v2268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2991.727495 secs 2024-04-26T12:40:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2994.453485 secs 2024-04-26T12:40:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:55 smithi194 ceph-mon[33017]: pgmap v2269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:55 smithi033 ceph-mon[27845]: pgmap v2269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:55 smithi096 ceph-mon[32753]: pgmap v2269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:40:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:57 smithi194 ceph-mon[33017]: pgmap v2270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:57 smithi033 ceph-mon[27845]: pgmap v2270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:57 smithi096 ceph-mon[32753]: pgmap v2270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:40:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 2996.727600 secs 2024-04-26T12:40:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 2999.453623 secs 2024-04-26T12:40:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 2996.727600 secs 2024-04-26T12:40:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 2999.453623 secs 2024-04-26T12:40:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 2996.727600 secs 2024-04-26T12:40:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 2999.453623 secs 2024-04-26T12:40:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:40:59 smithi033 ceph-mon[27845]: pgmap v2271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:40:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:40:59 smithi096 ceph-mon[32753]: pgmap v2271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:40:59 smithi194 ceph-mon[33017]: pgmap v2271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:01 smithi033 ceph-mon[27845]: pgmap v2272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:01 smithi096 ceph-mon[32753]: pgmap v2272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:01 smithi194 ceph-mon[33017]: pgmap v2272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:03 smithi033 ceph-mon[27845]: pgmap v2273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3001.727704 secs 2024-04-26T12:41:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3004.453737 secs 2024-04-26T12:41:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:03 smithi096 ceph-mon[32753]: pgmap v2273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3001.727704 secs 2024-04-26T12:41:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3004.453737 secs 2024-04-26T12:41:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:03 smithi194 ceph-mon[33017]: pgmap v2273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3001.727704 secs 2024-04-26T12:41:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3004.453737 secs 2024-04-26T12:41:04.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:04.856+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:04.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:04.856+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:04.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:04.856+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:41:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:05 smithi033 ceph-mon[27845]: pgmap v2274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:41:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:05 smithi096 ceph-mon[32753]: pgmap v2274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:41:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:05 smithi194 ceph-mon[33017]: pgmap v2274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:41:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:07 smithi033 ceph-mon[27845]: pgmap v2275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:41:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:07 smithi096 ceph-mon[32753]: pgmap v2275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:41:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:07 smithi194 ceph-mon[33017]: pgmap v2275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:41:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3006.727807 secs 2024-04-26T12:41:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3009.453821 secs 2024-04-26T12:41:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3006.727807 secs 2024-04-26T12:41:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3009.453821 secs 2024-04-26T12:41:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3006.727807 secs 2024-04-26T12:41:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3009.453821 secs 2024-04-26T12:41:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:09 smithi033 ceph-mon[27845]: pgmap v2276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:09 smithi096 ceph-mon[32753]: pgmap v2276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:09 smithi194 ceph-mon[33017]: pgmap v2276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:11 smithi033 ceph-mon[27845]: pgmap v2277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:11 smithi096 ceph-mon[32753]: pgmap v2277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:11 smithi194 ceph-mon[33017]: pgmap v2277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:13 smithi033 ceph-mon[27845]: pgmap v2278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3011.727908 secs 2024-04-26T12:41:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:41:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3014.453909 secs 2024-04-26T12:41:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:13.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:13 smithi096 ceph-mon[32753]: pgmap v2278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3011.727908 secs 2024-04-26T12:41:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:41:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3014.453909 secs 2024-04-26T12:41:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:13 smithi194 ceph-mon[33017]: pgmap v2278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3011.727908 secs 2024-04-26T12:41:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:41:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3014.453909 secs 2024-04-26T12:41:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:15 smithi033 ceph-mon[27845]: pgmap v2279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:41:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:41:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:41:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:15 smithi096 ceph-mon[32753]: pgmap v2279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:41:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:41:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:41:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:15 smithi194 ceph-mon[33017]: pgmap v2279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:41:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:41:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:41:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:41:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:16 smithi033 ceph-mon[27845]: pgmap v2280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:41:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:16 smithi033 ceph-mon[27845]: pgmap v2281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:16 smithi096 ceph-mon[32753]: pgmap v2280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:41:16.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:16 smithi096 ceph-mon[32753]: pgmap v2281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:16 smithi194 ceph-mon[33017]: pgmap v2280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:41:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:16 smithi194 ceph-mon[33017]: pgmap v2281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:18 smithi096 ceph-mon[32753]: pgmap v2282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3016.728013 secs 2024-04-26T12:41:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3019.454045 secs 2024-04-26T12:41:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:18 smithi194 ceph-mon[33017]: pgmap v2282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3016.728013 secs 2024-04-26T12:41:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3019.454045 secs 2024-04-26T12:41:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:18 smithi033 ceph-mon[27845]: pgmap v2282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3016.728013 secs 2024-04-26T12:41:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3019.454045 secs 2024-04-26T12:41:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:19.872+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:19.872+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:19.872+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:41:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:20 smithi194 ceph-mon[33017]: pgmap v2283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:41:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:20 smithi033 ceph-mon[27845]: pgmap v2283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:41:21.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:20 smithi096 ceph-mon[32753]: pgmap v2283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:41:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 msgr2=0x7feee40af500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T12:41:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 msgr2=0x7feee40af500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T12:41:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 0x7feee40af500 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feee80038f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T12:41:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 0x7feee40af500 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feee80038f0 comp rx=0 tx=0).stop 2024-04-26T12:41:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 msgr2=0x7feee40af500 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T12:41:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 msgr2=0x7feee40af500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T12:41:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af120 0x7feee40af500 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T12:41:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.710+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T12:41:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.711+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T12:41:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:21.713+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40afc50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feef0024000 tx=0x7feef0038040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T12:41:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:21 smithi194 ceph-mon[33017]: pgmap v2284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:21 smithi033 ceph-mon[27845]: pgmap v2284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:21 smithi096 ceph-mon[32753]: pgmap v2284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:41:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:22.308+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:41:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:22.308+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:41:22.310 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:22.309+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:41:22.310 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:22.309+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:41:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3021.728115 secs 2024-04-26T12:41:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3021.728115 secs 2024-04-26T12:41:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3021.728115 secs 2024-04-26T12:41:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:23 smithi033 ceph-mon[27845]: pgmap v2285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T12:41:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3024.454164 secs 2024-04-26T12:41:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:23 smithi096 ceph-mon[32753]: pgmap v2285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T12:41:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3024.454164 secs 2024-04-26T12:41:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:23 smithi194 ceph-mon[33017]: pgmap v2285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T12:41:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3024.454164 secs 2024-04-26T12:41:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:26 smithi194 ceph-mon[33017]: pgmap v2286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:41:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:26 smithi033 ceph-mon[27845]: pgmap v2286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:41:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:26 smithi096 ceph-mon[32753]: pgmap v2286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:41:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:28 smithi194 ceph-mon[33017]: pgmap v2287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3026.728176 secs 2024-04-26T12:41:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:28 smithi033 ceph-mon[27845]: pgmap v2287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3026.728176 secs 2024-04-26T12:41:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:28 smithi096 ceph-mon[32753]: pgmap v2287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:28.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3026.728176 secs 2024-04-26T12:41:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3029.454283 secs 2024-04-26T12:41:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3029.454283 secs 2024-04-26T12:41:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3029.454283 secs 2024-04-26T12:41:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:30 smithi194 ceph-mon[33017]: pgmap v2288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:30 smithi033 ceph-mon[27845]: pgmap v2288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:30 smithi096 ceph-mon[32753]: pgmap v2288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:32 smithi194 ceph-mon[33017]: pgmap v2289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:32 smithi033 ceph-mon[27845]: pgmap v2289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:32 smithi096 ceph-mon[32753]: pgmap v2289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:33.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3031.728238 secs 2024-04-26T12:41:33.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3031.728238 secs 2024-04-26T12:41:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3031.728238 secs 2024-04-26T12:41:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:34 smithi194 ceph-mon[33017]: pgmap v2290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3034.454405 secs 2024-04-26T12:41:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:34 smithi033 ceph-mon[27845]: pgmap v2290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3034.454405 secs 2024-04-26T12:41:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:34 smithi096 ceph-mon[32753]: pgmap v2290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3034.454405 secs 2024-04-26T12:41:34.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:34.877+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:34.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:34.877+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:34.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:34.877+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:41:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:36 smithi194 ceph-mon[33017]: pgmap v2291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:36 smithi033 ceph-mon[27845]: pgmap v2291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:36 smithi096 ceph-mon[32753]: pgmap v2291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:38 smithi194 ceph-mon[33017]: pgmap v2292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3036.728299 secs 2024-04-26T12:41:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:38 smithi033 ceph-mon[27845]: pgmap v2292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3036.728299 secs 2024-04-26T12:41:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:38 smithi096 ceph-mon[32753]: pgmap v2292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3036.728299 secs 2024-04-26T12:41:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3039.454501 secs 2024-04-26T12:41:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3039.454501 secs 2024-04-26T12:41:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3039.454501 secs 2024-04-26T12:41:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:40 smithi194 ceph-mon[33017]: pgmap v2293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:40 smithi033 ceph-mon[27845]: pgmap v2293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:40 smithi096 ceph-mon[32753]: pgmap v2293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:42 smithi194 ceph-mon[33017]: pgmap v2294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:42 smithi033 ceph-mon[27845]: pgmap v2294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:42 smithi096 ceph-mon[32753]: pgmap v2294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3041.728386 secs 2024-04-26T12:41:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3041.728386 secs 2024-04-26T12:41:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3041.728386 secs 2024-04-26T12:41:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:44 smithi194 ceph-mon[33017]: pgmap v2295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3044.454636 secs 2024-04-26T12:41:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:44 smithi033 ceph-mon[27845]: pgmap v2295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3044.454636 secs 2024-04-26T12:41:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:44 smithi096 ceph-mon[32753]: pgmap v2295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3044.454636 secs 2024-04-26T12:41:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:46 smithi033 ceph-mon[27845]: pgmap v2296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:46 smithi096 ceph-mon[32753]: pgmap v2296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:46 smithi194 ceph-mon[33017]: pgmap v2296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:48 smithi033 ceph-mon[27845]: pgmap v2297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3046.728448 secs 2024-04-26T12:41:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:48 smithi096 ceph-mon[32753]: pgmap v2297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3046.728448 secs 2024-04-26T12:41:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:48 smithi194 ceph-mon[33017]: pgmap v2297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3046.728448 secs 2024-04-26T12:41:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3049.454758 secs 2024-04-26T12:41:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3049.454758 secs 2024-04-26T12:41:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3049.454758 secs 2024-04-26T12:41:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:49.878+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:49.878+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:41:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:41:49.878+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:41:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:50 smithi033 ceph-mon[27845]: pgmap v2298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:50 smithi096 ceph-mon[32753]: pgmap v2298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:50 smithi194 ceph-mon[33017]: pgmap v2298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:52 smithi033 ceph-mon[27845]: pgmap v2299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:52 smithi096 ceph-mon[32753]: pgmap v2299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:52 smithi194 ceph-mon[33017]: pgmap v2299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3051.728533 secs 2024-04-26T12:41:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3051.728533 secs 2024-04-26T12:41:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3051.728533 secs 2024-04-26T12:41:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:54 smithi033 ceph-mon[27845]: pgmap v2300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3054.454873 secs 2024-04-26T12:41:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:54 smithi096 ceph-mon[32753]: pgmap v2300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3054.454873 secs 2024-04-26T12:41:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:54 smithi194 ceph-mon[33017]: pgmap v2300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3054.454873 secs 2024-04-26T12:41:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:56 smithi033 ceph-mon[27845]: pgmap v2301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:56 smithi096 ceph-mon[32753]: pgmap v2301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:56 smithi194 ceph-mon[33017]: pgmap v2301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:41:58.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:58 smithi194 ceph-mon[33017]: pgmap v2302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:58.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3056.728616 secs 2024-04-26T12:41:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:58 smithi033 ceph-mon[27845]: pgmap v2302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3056.728616 secs 2024-04-26T12:41:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:58 smithi096 ceph-mon[32753]: pgmap v2302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:41:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3056.728616 secs 2024-04-26T12:41:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:41:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3059.455014 secs 2024-04-26T12:41:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:41:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3059.455014 secs 2024-04-26T12:41:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:41:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3059.455014 secs 2024-04-26T12:42:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:00 smithi033 ceph-mon[27845]: pgmap v2303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:00 smithi096 ceph-mon[32753]: pgmap v2303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:00 smithi194 ceph-mon[33017]: pgmap v2303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:02 smithi033 ceph-mon[27845]: pgmap v2304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:02 smithi096 ceph-mon[32753]: pgmap v2304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:02 smithi194 ceph-mon[33017]: pgmap v2304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3061.728700 secs 2024-04-26T12:42:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3061.728700 secs 2024-04-26T12:42:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3061.728700 secs 2024-04-26T12:42:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:04 smithi194 ceph-mon[33017]: pgmap v2305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3064.455129 secs 2024-04-26T12:42:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:04 smithi033 ceph-mon[27845]: pgmap v2305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3064.455129 secs 2024-04-26T12:42:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:04 smithi096 ceph-mon[32753]: pgmap v2305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3064.455129 secs 2024-04-26T12:42:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:04.879+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:04.879+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:04.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:04.879+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:42:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:06 smithi194 ceph-mon[33017]: pgmap v2306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:06 smithi033 ceph-mon[27845]: pgmap v2306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:06 smithi096 ceph-mon[32753]: pgmap v2306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:08 smithi194 ceph-mon[33017]: pgmap v2307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3066.728785 secs 2024-04-26T12:42:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:08 smithi033 ceph-mon[27845]: pgmap v2307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3066.728785 secs 2024-04-26T12:42:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:08 smithi096 ceph-mon[32753]: pgmap v2307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3066.728785 secs 2024-04-26T12:42:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3069.455247 secs 2024-04-26T12:42:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3069.455247 secs 2024-04-26T12:42:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3069.455247 secs 2024-04-26T12:42:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:10 smithi194 ceph-mon[33017]: pgmap v2308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:10 smithi033 ceph-mon[27845]: pgmap v2308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:10 smithi096 ceph-mon[32753]: pgmap v2308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:12 smithi194 ceph-mon[33017]: pgmap v2309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:12 smithi033 ceph-mon[27845]: pgmap v2309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:12 smithi096 ceph-mon[32753]: pgmap v2309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3071.728869 secs 2024-04-26T12:42:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3071.728869 secs 2024-04-26T12:42:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3071.728869 secs 2024-04-26T12:42:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:14 smithi194 ceph-mon[33017]: pgmap v2310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3074.455368 secs 2024-04-26T12:42:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:14 smithi033 ceph-mon[27845]: pgmap v2310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3074.455368 secs 2024-04-26T12:42:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:14 smithi096 ceph-mon[32753]: pgmap v2310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3074.455368 secs 2024-04-26T12:42:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:16 smithi194 ceph-mon[33017]: pgmap v2311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:42:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:16 smithi033 ceph-mon[27845]: pgmap v2311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:42:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:16 smithi096 ceph-mon[32753]: pgmap v2311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:42:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:42:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:42:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:17 smithi194 ceph-mon[33017]: pgmap v2312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:17 smithi194 ceph-mon[33017]: pgmap v2313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:42:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:42:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:42:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:17 smithi033 ceph-mon[27845]: pgmap v2312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:17 smithi033 ceph-mon[27845]: pgmap v2313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:42:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:42:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:42:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:17 smithi096 ceph-mon[32753]: pgmap v2312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:17 smithi096 ceph-mon[32753]: pgmap v2313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:42:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3076.728954 secs 2024-04-26T12:42:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3076.728954 secs 2024-04-26T12:42:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3076.728954 secs 2024-04-26T12:42:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3079.455461 secs 2024-04-26T12:42:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:19 smithi194 ceph-mon[33017]: pgmap v2314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3079.455461 secs 2024-04-26T12:42:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:19 smithi033 ceph-mon[27845]: pgmap v2314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3079.455461 secs 2024-04-26T12:42:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:19 smithi096 ceph-mon[32753]: pgmap v2314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:19.881+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:19.881+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:19.881+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:42:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:21 smithi033 ceph-mon[27845]: pgmap v2315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:21 smithi096 ceph-mon[32753]: pgmap v2315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:21 smithi194 ceph-mon[33017]: pgmap v2315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3081.729000 secs 2024-04-26T12:42:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:23 smithi033 ceph-mon[27845]: pgmap v2316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3084.455598 secs 2024-04-26T12:42:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3081.729000 secs 2024-04-26T12:42:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:23 smithi096 ceph-mon[32753]: pgmap v2316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3084.455598 secs 2024-04-26T12:42:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3081.729000 secs 2024-04-26T12:42:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:23 smithi194 ceph-mon[33017]: pgmap v2316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3084.455598 secs 2024-04-26T12:42:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:25 smithi033 ceph-mon[27845]: pgmap v2317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:25 smithi096 ceph-mon[32753]: pgmap v2317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:25 smithi194 ceph-mon[33017]: pgmap v2317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:42:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:27 smithi033 ceph-mon[27845]: pgmap v2318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:42:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:27 smithi096 ceph-mon[32753]: pgmap v2318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:42:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:27 smithi194 ceph-mon[33017]: pgmap v2318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:42:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3086.729063 secs 2024-04-26T12:42:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3089.455733 secs 2024-04-26T12:42:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3086.729063 secs 2024-04-26T12:42:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3089.455733 secs 2024-04-26T12:42:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3086.729063 secs 2024-04-26T12:42:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3089.455733 secs 2024-04-26T12:42:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:29 smithi096 ceph-mon[32753]: pgmap v2319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:29 smithi194 ceph-mon[33017]: pgmap v2319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:30.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:29 smithi033 ceph-mon[27845]: pgmap v2319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:31 smithi194 ceph-mon[33017]: pgmap v2320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:31 smithi033 ceph-mon[27845]: pgmap v2320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:31 smithi096 ceph-mon[32753]: pgmap v2320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:33.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3091.729149 secs 2024-04-26T12:42:33.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:33 smithi096 ceph-mon[32753]: pgmap v2321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:33.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3094.455842 secs 2024-04-26T12:42:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3091.729149 secs 2024-04-26T12:42:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:33 smithi194 ceph-mon[33017]: pgmap v2321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3094.455842 secs 2024-04-26T12:42:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3091.729149 secs 2024-04-26T12:42:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:33 smithi033 ceph-mon[27845]: pgmap v2321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3094.455842 secs 2024-04-26T12:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:34.887+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:34.887+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:34.887+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:42:36.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:35 smithi194 ceph-mon[33017]: pgmap v2322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:35 smithi033 ceph-mon[27845]: pgmap v2322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:35 smithi096 ceph-mon[32753]: pgmap v2322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:37 smithi194 ceph-mon[33017]: pgmap v2323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:37 smithi033 ceph-mon[27845]: pgmap v2323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:37 smithi096 ceph-mon[32753]: pgmap v2323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3096.729232 secs 2024-04-26T12:42:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3099.455954 secs 2024-04-26T12:42:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:38 smithi194 ceph-mon[33017]: pgmap v2324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3096.729232 secs 2024-04-26T12:42:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3099.455954 secs 2024-04-26T12:42:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:38 smithi033 ceph-mon[27845]: pgmap v2324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3096.729232 secs 2024-04-26T12:42:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3099.455954 secs 2024-04-26T12:42:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:38 smithi096 ceph-mon[32753]: pgmap v2324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:41 smithi033 ceph-mon[27845]: pgmap v2325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:41 smithi096 ceph-mon[32753]: pgmap v2325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:41 smithi194 ceph-mon[33017]: pgmap v2325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3101.729292 secs 2024-04-26T12:42:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:43 smithi033 ceph-mon[27845]: pgmap v2326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3104.456088 secs 2024-04-26T12:42:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3101.729292 secs 2024-04-26T12:42:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:43 smithi096 ceph-mon[32753]: pgmap v2326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3104.456088 secs 2024-04-26T12:42:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3101.729292 secs 2024-04-26T12:42:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:43 smithi194 ceph-mon[33017]: pgmap v2326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3104.456088 secs 2024-04-26T12:42:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:45 smithi033 ceph-mon[27845]: pgmap v2327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:45 smithi096 ceph-mon[32753]: pgmap v2327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:45 smithi194 ceph-mon[33017]: pgmap v2327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:47 smithi033 ceph-mon[27845]: pgmap v2328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:47 smithi096 ceph-mon[32753]: pgmap v2328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:47 smithi194 ceph-mon[33017]: pgmap v2328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3106.729366 secs 2024-04-26T12:42:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3109.456211 secs 2024-04-26T12:42:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3106.729366 secs 2024-04-26T12:42:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3109.456211 secs 2024-04-26T12:42:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3106.729366 secs 2024-04-26T12:42:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3109.456211 secs 2024-04-26T12:42:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:49.902+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:49.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:49.902+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:42:49.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:42:49.902+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:42:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:49 smithi194 ceph-mon[33017]: pgmap v2329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:49 smithi033 ceph-mon[27845]: pgmap v2329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:49 smithi096 ceph-mon[32753]: pgmap v2329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:52.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:51 smithi033 ceph-mon[27845]: pgmap v2330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:51 smithi194 ceph-mon[33017]: pgmap v2330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:51 smithi096 ceph-mon[32753]: pgmap v2330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3111.729428 secs 2024-04-26T12:42:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:53 smithi194 ceph-mon[33017]: pgmap v2331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3114.456333 secs 2024-04-26T12:42:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3111.729428 secs 2024-04-26T12:42:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:53 smithi033 ceph-mon[27845]: pgmap v2331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3114.456333 secs 2024-04-26T12:42:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3111.729428 secs 2024-04-26T12:42:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:53 smithi096 ceph-mon[32753]: pgmap v2331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3114.456333 secs 2024-04-26T12:42:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:55 smithi194 ceph-mon[33017]: pgmap v2332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:55 smithi033 ceph-mon[27845]: pgmap v2332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:55 smithi096 ceph-mon[32753]: pgmap v2332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:42:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:56 smithi194 ceph-mon[33017]: pgmap v2333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:57.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:56 smithi033 ceph-mon[27845]: pgmap v2333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:57.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:56 smithi096 ceph-mon[32753]: pgmap v2333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3116.729515 secs 2024-04-26T12:42:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3116.729515 secs 2024-04-26T12:42:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3116.729515 secs 2024-04-26T12:42:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3119.456464 secs 2024-04-26T12:42:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:42:58 smithi033 ceph-mon[27845]: pgmap v2334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3119.456464 secs 2024-04-26T12:42:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:42:58 smithi096 ceph-mon[32753]: pgmap v2334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:42:59.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3119.456464 secs 2024-04-26T12:42:59.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:42:58 smithi194 ceph-mon[33017]: pgmap v2334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:01 smithi033 ceph-mon[27845]: pgmap v2335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:01 smithi096 ceph-mon[32753]: pgmap v2335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:01 smithi194 ceph-mon[33017]: pgmap v2335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3121.729599 secs 2024-04-26T12:43:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:03 smithi033 ceph-mon[27845]: pgmap v2336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3124.456583 secs 2024-04-26T12:43:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3121.729599 secs 2024-04-26T12:43:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:03 smithi096 ceph-mon[32753]: pgmap v2336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3124.456583 secs 2024-04-26T12:43:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3121.729599 secs 2024-04-26T12:43:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:03 smithi194 ceph-mon[33017]: pgmap v2336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3124.456583 secs 2024-04-26T12:43:04.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:04.916+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:04.916+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:04.916+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:43:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:05 smithi096 ceph-mon[32753]: pgmap v2337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:05 smithi194 ceph-mon[33017]: pgmap v2337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:05 smithi033 ceph-mon[27845]: pgmap v2337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:07.952 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:07 smithi033 ceph-mon[27845]: pgmap v2338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:07 smithi194 ceph-mon[33017]: pgmap v2338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:07 smithi096 ceph-mon[32753]: pgmap v2338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3126.729684 secs 2024-04-26T12:43:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3129.456701 secs 2024-04-26T12:43:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3126.729684 secs 2024-04-26T12:43:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3129.456701 secs 2024-04-26T12:43:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3126.729684 secs 2024-04-26T12:43:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3129.456701 secs 2024-04-26T12:43:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:09 smithi194 ceph-mon[33017]: pgmap v2339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:09 smithi033 ceph-mon[27845]: pgmap v2339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:09 smithi096 ceph-mon[32753]: pgmap v2339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:11 smithi194 ceph-mon[33017]: pgmap v2340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:11 smithi033 ceph-mon[27845]: pgmap v2340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:11 smithi096 ceph-mon[32753]: pgmap v2340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3131.729765 secs 2024-04-26T12:43:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:13 smithi194 ceph-mon[33017]: pgmap v2341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3134.456799 secs 2024-04-26T12:43:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3131.729765 secs 2024-04-26T12:43:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:13 smithi033 ceph-mon[27845]: pgmap v2341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3134.456799 secs 2024-04-26T12:43:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3131.729765 secs 2024-04-26T12:43:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:13 smithi096 ceph-mon[32753]: pgmap v2341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3134.456799 secs 2024-04-26T12:43:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:14 smithi194 ceph-mon[33017]: pgmap v2342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:14 smithi033 ceph-mon[27845]: pgmap v2342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:14 smithi096 ceph-mon[32753]: pgmap v2342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:17 smithi033 ceph-mon[27845]: pgmap v2343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:43:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:17 smithi096 ceph-mon[32753]: pgmap v2343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:43:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:17 smithi194 ceph-mon[33017]: pgmap v2343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:43:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3136.729851 secs 2024-04-26T12:43:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3139.456868 secs 2024-04-26T12:43:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3136.729851 secs 2024-04-26T12:43:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3139.456868 secs 2024-04-26T12:43:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3136.729851 secs 2024-04-26T12:43:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3139.456868 secs 2024-04-26T12:43:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:19.926+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:19.927+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:19.927+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:43:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:19 smithi194 ceph-mon[33017]: pgmap v2344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:19 smithi033 ceph-mon[27845]: pgmap v2344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:19 smithi096 ceph-mon[32753]: pgmap v2344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:21.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:21 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:43:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:21 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:43:22.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:21 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:43:22.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:21 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:43:22.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:21 smithi194 ceph-mon[33017]: pgmap v2345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:22.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:22.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:21 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:43:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:21 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:43:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:21 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:43:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:21 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:43:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:21 smithi033 ceph-mon[27845]: pgmap v2345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:22.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:21 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:43:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:21 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:43:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:21 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:43:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:21 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:43:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:21 smithi096 ceph-mon[32753]: pgmap v2345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:22.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3141.729933 secs 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: pgmap v2346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:43:24.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:43:24.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:43:24.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:43:24.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: pgmap v2347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:43:24.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: pgmap v2348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:43:24.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:24.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3144.457008 secs 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3141.729933 secs 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: pgmap v2346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: pgmap v2347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: pgmap v2348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:24.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3144.457008 secs 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3141.729933 secs 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: pgmap v2346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:43:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:43:24.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:43:24.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:43:24.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:43:24.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: pgmap v2347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:43:24.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: pgmap v2348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:43:24.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:43:24.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3144.457008 secs 2024-04-26T12:43:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:25 smithi033 ceph-mon[27845]: pgmap v2349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:25 smithi096 ceph-mon[32753]: pgmap v2349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:25 smithi194 ceph-mon[33017]: pgmap v2349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:27 smithi033 ceph-mon[27845]: pgmap v2350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:28.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3146.730026 secs 2024-04-26T12:43:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:27 smithi096 ceph-mon[32753]: pgmap v2350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3146.730026 secs 2024-04-26T12:43:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:27 smithi194 ceph-mon[33017]: pgmap v2350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3146.730026 secs 2024-04-26T12:43:29.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3149.457144 secs 2024-04-26T12:43:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3149.457144 secs 2024-04-26T12:43:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3149.457144 secs 2024-04-26T12:43:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:29 smithi194 ceph-mon[33017]: pgmap v2351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:43:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:29 smithi033 ceph-mon[27845]: pgmap v2351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:43:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:29 smithi096 ceph-mon[32753]: pgmap v2351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:43:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:31 smithi194 ceph-mon[33017]: pgmap v2352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:31 smithi033 ceph-mon[27845]: pgmap v2352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:31 smithi096 ceph-mon[32753]: pgmap v2352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:33.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3151.730115 secs 2024-04-26T12:43:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3151.730115 secs 2024-04-26T12:43:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3151.730115 secs 2024-04-26T12:43:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:33 smithi194 ceph-mon[33017]: pgmap v2353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:43:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3154.457276 secs 2024-04-26T12:43:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:33 smithi033 ceph-mon[27845]: pgmap v2353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:43:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3154.457276 secs 2024-04-26T12:43:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:33 smithi096 ceph-mon[32753]: pgmap v2353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:43:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3154.457276 secs 2024-04-26T12:43:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:34.935+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:34.935+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:34.935+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:43:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:35 smithi194 ceph-mon[33017]: pgmap v2354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:35 smithi033 ceph-mon[27845]: pgmap v2354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:35 smithi096 ceph-mon[32753]: pgmap v2354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:37 smithi194 ceph-mon[33017]: pgmap v2355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3156.730176 secs 2024-04-26T12:43:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:37 smithi033 ceph-mon[27845]: pgmap v2355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3156.730176 secs 2024-04-26T12:43:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:37 smithi096 ceph-mon[32753]: pgmap v2355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3156.730176 secs 2024-04-26T12:43:39.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3159.457394 secs 2024-04-26T12:43:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3159.457394 secs 2024-04-26T12:43:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3159.457394 secs 2024-04-26T12:43:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:39 smithi194 ceph-mon[33017]: pgmap v2356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:39 smithi033 ceph-mon[27845]: pgmap v2356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:39 smithi096 ceph-mon[32753]: pgmap v2356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:41 smithi194 ceph-mon[33017]: pgmap v2357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:41 smithi033 ceph-mon[27845]: pgmap v2357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:41 smithi096 ceph-mon[32753]: pgmap v2357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3161.730260 secs 2024-04-26T12:43:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3161.730260 secs 2024-04-26T12:43:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3161.730260 secs 2024-04-26T12:43:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:43 smithi194 ceph-mon[33017]: pgmap v2358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3164.457517 secs 2024-04-26T12:43:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:43 smithi033 ceph-mon[27845]: pgmap v2358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3164.457517 secs 2024-04-26T12:43:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:43 smithi096 ceph-mon[32753]: pgmap v2358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:43:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3164.457517 secs 2024-04-26T12:43:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:46 smithi194 ceph-mon[33017]: pgmap v2359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:46 smithi033 ceph-mon[27845]: pgmap v2359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:46 smithi096 ceph-mon[32753]: pgmap v2359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:43:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:48 smithi194 ceph-mon[33017]: pgmap v2360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3166.730342 secs 2024-04-26T12:43:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:48 smithi033 ceph-mon[27845]: pgmap v2360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3166.730342 secs 2024-04-26T12:43:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:48 smithi096 ceph-mon[32753]: pgmap v2360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3166.730342 secs 2024-04-26T12:43:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3169.457610 secs 2024-04-26T12:43:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3169.457610 secs 2024-04-26T12:43:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3169.457610 secs 2024-04-26T12:43:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:49.939+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:49.939+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:43:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:43:49.939+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:43:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:50 smithi194 ceph-mon[33017]: pgmap v2361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:50 smithi033 ceph-mon[27845]: pgmap v2361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:50 smithi096 ceph-mon[32753]: pgmap v2361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:52 smithi194 ceph-mon[33017]: pgmap v2362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:52 smithi033 ceph-mon[27845]: pgmap v2362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:52 smithi096 ceph-mon[32753]: pgmap v2362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3171.730401 secs 2024-04-26T12:43:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3171.730401 secs 2024-04-26T12:43:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3171.730401 secs 2024-04-26T12:43:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:54 smithi194 ceph-mon[33017]: pgmap v2363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3174.457747 secs 2024-04-26T12:43:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:54 smithi033 ceph-mon[27845]: pgmap v2363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3174.457747 secs 2024-04-26T12:43:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:54 smithi096 ceph-mon[32753]: pgmap v2363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3174.457747 secs 2024-04-26T12:43:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:56 smithi194 ceph-mon[33017]: pgmap v2364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:56 smithi033 ceph-mon[27845]: pgmap v2364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:56 smithi096 ceph-mon[32753]: pgmap v2364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:43:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:58 smithi194 ceph-mon[33017]: pgmap v2365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3176.730482 secs 2024-04-26T12:43:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:58 smithi033 ceph-mon[27845]: pgmap v2365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3176.730482 secs 2024-04-26T12:43:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:58 smithi096 ceph-mon[32753]: pgmap v2365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:43:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3176.730482 secs 2024-04-26T12:43:59.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3179.457918 secs 2024-04-26T12:43:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3179.457918 secs 2024-04-26T12:43:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3179.457918 secs 2024-04-26T12:44:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:00 smithi194 ceph-mon[33017]: pgmap v2366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:00 smithi033 ceph-mon[27845]: pgmap v2366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:00 smithi096 ceph-mon[32753]: pgmap v2366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:02 smithi194 ceph-mon[33017]: pgmap v2367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:02 smithi033 ceph-mon[27845]: pgmap v2367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:02 smithi096 ceph-mon[32753]: pgmap v2367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:03.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3181.730567 secs 2024-04-26T12:44:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3181.730567 secs 2024-04-26T12:44:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3181.730567 secs 2024-04-26T12:44:04.346 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:04 smithi096 ceph-mon[32753]: pgmap v2368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:04.346 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3184.458058 secs 2024-04-26T12:44:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:04 smithi194 ceph-mon[33017]: pgmap v2368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3184.458058 secs 2024-04-26T12:44:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:04 smithi033 ceph-mon[27845]: pgmap v2368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3184.458058 secs 2024-04-26T12:44:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:04.940+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:04.940+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:04.940+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:44:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:06 smithi194 ceph-mon[33017]: pgmap v2369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:06 smithi033 ceph-mon[27845]: pgmap v2369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:06 smithi096 ceph-mon[32753]: pgmap v2369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:08 smithi194 ceph-mon[33017]: pgmap v2370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3186.730650 secs 2024-04-26T12:44:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:08 smithi033 ceph-mon[27845]: pgmap v2370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3186.730650 secs 2024-04-26T12:44:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:08 smithi096 ceph-mon[32753]: pgmap v2370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3186.730650 secs 2024-04-26T12:44:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3189.458173 secs 2024-04-26T12:44:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3189.458173 secs 2024-04-26T12:44:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3189.458173 secs 2024-04-26T12:44:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:10 smithi033 ceph-mon[27845]: pgmap v2371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:10 smithi194 ceph-mon[33017]: pgmap v2371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:10 smithi096 ceph-mon[32753]: pgmap v2371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:12 smithi033 ceph-mon[27845]: pgmap v2372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:12 smithi096 ceph-mon[32753]: pgmap v2372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:12.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:12 smithi194 ceph-mon[33017]: pgmap v2372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3191.730739 secs 2024-04-26T12:44:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3191.730739 secs 2024-04-26T12:44:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3191.730739 secs 2024-04-26T12:44:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:14 smithi033 ceph-mon[27845]: pgmap v2373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3194.458290 secs 2024-04-26T12:44:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:14 smithi096 ceph-mon[32753]: pgmap v2373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3194.458290 secs 2024-04-26T12:44:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:14 smithi194 ceph-mon[33017]: pgmap v2373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3194.458290 secs 2024-04-26T12:44:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:16 smithi033 ceph-mon[27845]: pgmap v2374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:16.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:16 smithi096 ceph-mon[32753]: pgmap v2374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:16 smithi194 ceph-mon[33017]: pgmap v2374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:18 smithi033 ceph-mon[27845]: pgmap v2375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3196.730822 secs 2024-04-26T12:44:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:18 smithi096 ceph-mon[32753]: pgmap v2375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3196.730822 secs 2024-04-26T12:44:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:18 smithi194 ceph-mon[33017]: pgmap v2375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3196.730822 secs 2024-04-26T12:44:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3199.458444 secs 2024-04-26T12:44:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3199.458444 secs 2024-04-26T12:44:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3199.458444 secs 2024-04-26T12:44:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:19.953+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:19.953+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:19.953+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:44:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:20 smithi033 ceph-mon[27845]: pgmap v2376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:20 smithi096 ceph-mon[32753]: pgmap v2376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:20.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:20 smithi194 ceph-mon[33017]: pgmap v2376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:22.337+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:44:22.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:22.337+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:44:22.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:22.338+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:44:22.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:22.338+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T12:44:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:22 smithi033 ceph-mon[27845]: pgmap v2377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:22 smithi096 ceph-mon[32753]: pgmap v2377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:22 smithi194 ceph-mon[33017]: pgmap v2377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3201.730904 secs 2024-04-26T12:44:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3201.730904 secs 2024-04-26T12:44:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3201.730904 secs 2024-04-26T12:44:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:24 smithi033 ceph-mon[27845]: pgmap v2378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3204.458571 secs 2024-04-26T12:44:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:44:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:24 smithi096 ceph-mon[32753]: pgmap v2378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3204.458571 secs 2024-04-26T12:44:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:44:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:24 smithi194 ceph-mon[33017]: pgmap v2378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3204.458571 secs 2024-04-26T12:44:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:44:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:44:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:44:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:25 smithi033 ceph-mon[27845]: pgmap v2379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:44:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:25 smithi033 ceph-mon[27845]: pgmap v2380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:25.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:44:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:44:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:44:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:25 smithi096 ceph-mon[32753]: pgmap v2379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:44:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:25 smithi096 ceph-mon[32753]: pgmap v2380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:44:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:44:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:44:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:25 smithi194 ceph-mon[33017]: pgmap v2379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:44:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:25 smithi194 ceph-mon[33017]: pgmap v2380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:44:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:27 smithi033 ceph-mon[27845]: pgmap v2381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:27 smithi096 ceph-mon[32753]: pgmap v2381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:27 smithi194 ceph-mon[33017]: pgmap v2381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3206.730987 secs 2024-04-26T12:44:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3209.458698 secs 2024-04-26T12:44:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3206.730987 secs 2024-04-26T12:44:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3209.458698 secs 2024-04-26T12:44:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3206.730987 secs 2024-04-26T12:44:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3209.458698 secs 2024-04-26T12:44:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:29 smithi033 ceph-mon[27845]: pgmap v2382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:29 smithi096 ceph-mon[32753]: pgmap v2382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:29 smithi194 ceph-mon[33017]: pgmap v2382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:31 smithi033 ceph-mon[27845]: pgmap v2383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:31 smithi096 ceph-mon[32753]: pgmap v2383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:31 smithi194 ceph-mon[33017]: pgmap v2383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3211.731047 secs 2024-04-26T12:44:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:33 smithi033 ceph-mon[27845]: pgmap v2384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3214.458842 secs 2024-04-26T12:44:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3211.731047 secs 2024-04-26T12:44:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:33 smithi096 ceph-mon[32753]: pgmap v2384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3214.458842 secs 2024-04-26T12:44:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3211.731047 secs 2024-04-26T12:44:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:33 smithi194 ceph-mon[33017]: pgmap v2384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:44:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3214.458842 secs 2024-04-26T12:44:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:34.963+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:34.963+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:34.963+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:44:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:35 smithi033 ceph-mon[27845]: pgmap v2385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:44:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:35 smithi096 ceph-mon[32753]: pgmap v2385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:44:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:35 smithi194 ceph-mon[33017]: pgmap v2385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:44:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:37 smithi033 ceph-mon[27845]: pgmap v2386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:37 smithi096 ceph-mon[32753]: pgmap v2386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:37 smithi194 ceph-mon[33017]: pgmap v2386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3216.731134 secs 2024-04-26T12:44:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3219.458922 secs 2024-04-26T12:44:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3216.731134 secs 2024-04-26T12:44:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3219.458922 secs 2024-04-26T12:44:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3216.731134 secs 2024-04-26T12:44:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3219.458922 secs 2024-04-26T12:44:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:39 smithi194 ceph-mon[33017]: pgmap v2387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:39 smithi033 ceph-mon[27845]: pgmap v2387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:39 smithi096 ceph-mon[32753]: pgmap v2387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:41 smithi194 ceph-mon[33017]: pgmap v2388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:41 smithi033 ceph-mon[27845]: pgmap v2388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:41 smithi096 ceph-mon[32753]: pgmap v2388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3221.731219 secs 2024-04-26T12:44:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:43 smithi194 ceph-mon[33017]: pgmap v2389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3224.459059 secs 2024-04-26T12:44:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3221.731219 secs 2024-04-26T12:44:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:43 smithi033 ceph-mon[27845]: pgmap v2389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3224.459059 secs 2024-04-26T12:44:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3221.731219 secs 2024-04-26T12:44:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:43 smithi096 ceph-mon[32753]: pgmap v2389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3224.459059 secs 2024-04-26T12:44:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:45 smithi194 ceph-mon[33017]: pgmap v2390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:45 smithi033 ceph-mon[27845]: pgmap v2390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:45 smithi096 ceph-mon[32753]: pgmap v2390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:47 smithi194 ceph-mon[33017]: pgmap v2391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:47 smithi033 ceph-mon[27845]: pgmap v2391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:47 smithi096 ceph-mon[32753]: pgmap v2391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3226.731302 secs 2024-04-26T12:44:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3229.459184 secs 2024-04-26T12:44:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3226.731302 secs 2024-04-26T12:44:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3229.459184 secs 2024-04-26T12:44:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3226.731302 secs 2024-04-26T12:44:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3229.459184 secs 2024-04-26T12:44:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:49.979+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:49.979+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:44:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:44:49.979+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:44:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:49 smithi194 ceph-mon[33017]: pgmap v2392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:49 smithi033 ceph-mon[27845]: pgmap v2392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:49 smithi096 ceph-mon[32753]: pgmap v2392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:51 smithi194 ceph-mon[33017]: pgmap v2393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:51 smithi033 ceph-mon[27845]: pgmap v2393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:51 smithi096 ceph-mon[32753]: pgmap v2393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3231.731364 secs 2024-04-26T12:44:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:53 smithi194 ceph-mon[33017]: pgmap v2394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3234.459313 secs 2024-04-26T12:44:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3231.731364 secs 2024-04-26T12:44:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:53 smithi033 ceph-mon[27845]: pgmap v2394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3234.459313 secs 2024-04-26T12:44:54.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3231.731364 secs 2024-04-26T12:44:54.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:53 smithi096 ceph-mon[32753]: pgmap v2394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:54.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3234.459313 secs 2024-04-26T12:44:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:54 smithi194 ceph-mon[33017]: pgmap v2395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:54 smithi033 ceph-mon[27845]: pgmap v2395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:54 smithi096 ceph-mon[32753]: pgmap v2395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:44:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:57 smithi033 ceph-mon[27845]: pgmap v2396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:57.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:57 smithi096 ceph-mon[32753]: pgmap v2396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:57 smithi194 ceph-mon[33017]: pgmap v2396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3236.731425 secs 2024-04-26T12:44:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3239.459425 secs 2024-04-26T12:44:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3236.731425 secs 2024-04-26T12:44:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3239.459425 secs 2024-04-26T12:44:59.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3236.731425 secs 2024-04-26T12:44:59.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3239.459425 secs 2024-04-26T12:44:59.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:44:59 smithi033 ceph-mon[27845]: pgmap v2397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:44:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:44:59 smithi096 ceph-mon[32753]: pgmap v2397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:44:59 smithi194 ceph-mon[33017]: pgmap v2397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:01 smithi033 ceph-mon[27845]: pgmap v2398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:01 smithi096 ceph-mon[32753]: pgmap v2398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:01 smithi194 ceph-mon[33017]: pgmap v2398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3241.731485 secs 2024-04-26T12:45:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:03 smithi096 ceph-mon[32753]: pgmap v2399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3244.459542 secs 2024-04-26T12:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3241.731485 secs 2024-04-26T12:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:03 smithi194 ceph-mon[33017]: pgmap v2399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3244.459542 secs 2024-04-26T12:45:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3241.731485 secs 2024-04-26T12:45:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:03 smithi033 ceph-mon[27845]: pgmap v2399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3244.459542 secs 2024-04-26T12:45:04.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:04.989+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:04.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:04.989+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:45:04.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:04.989+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:05 smithi096 ceph-mon[32753]: pgmap v2400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:05 smithi194 ceph-mon[33017]: pgmap v2400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:05 smithi033 ceph-mon[27845]: pgmap v2400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:07 smithi194 ceph-mon[33017]: pgmap v2401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:07 smithi033 ceph-mon[27845]: pgmap v2401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:07 smithi096 ceph-mon[32753]: pgmap v2401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3246.731568 secs 2024-04-26T12:45:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3249.459679 secs 2024-04-26T12:45:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3246.731568 secs 2024-04-26T12:45:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3249.459679 secs 2024-04-26T12:45:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3246.731568 secs 2024-04-26T12:45:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3249.459679 secs 2024-04-26T12:45:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:09 smithi194 ceph-mon[33017]: pgmap v2402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:09 smithi033 ceph-mon[27845]: pgmap v2402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:09 smithi096 ceph-mon[32753]: pgmap v2402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:11.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:11 smithi033 ceph-mon[27845]: pgmap v2403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:11 smithi194 ceph-mon[33017]: pgmap v2403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:12.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:11 smithi096 ceph-mon[32753]: pgmap v2403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3251.731628 secs 2024-04-26T12:45:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:13 smithi194 ceph-mon[33017]: pgmap v2404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3254.459810 secs 2024-04-26T12:45:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3251.731628 secs 2024-04-26T12:45:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:13 smithi033 ceph-mon[27845]: pgmap v2404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3254.459810 secs 2024-04-26T12:45:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3251.731628 secs 2024-04-26T12:45:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:13 smithi096 ceph-mon[32753]: pgmap v2404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3254.459810 secs 2024-04-26T12:45:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:15 smithi194 ceph-mon[33017]: pgmap v2405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:15 smithi033 ceph-mon[27845]: pgmap v2405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:15 smithi096 ceph-mon[32753]: pgmap v2405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:17 smithi194 ceph-mon[33017]: pgmap v2406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:17 smithi033 ceph-mon[27845]: pgmap v2406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:17 smithi096 ceph-mon[32753]: pgmap v2406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3256.731714 secs 2024-04-26T12:45:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3259.459909 secs 2024-04-26T12:45:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3256.731714 secs 2024-04-26T12:45:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3259.459909 secs 2024-04-26T12:45:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3256.731714 secs 2024-04-26T12:45:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3259.459909 secs 2024-04-26T12:45:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:19.996+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:19.996+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:19.996+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:45:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:19 smithi194 ceph-mon[33017]: pgmap v2407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:19 smithi033 ceph-mon[27845]: pgmap v2407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:19 smithi096 ceph-mon[32753]: pgmap v2407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:21 smithi194 ceph-mon[33017]: pgmap v2408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:21 smithi033 ceph-mon[27845]: pgmap v2408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:21 smithi096 ceph-mon[32753]: pgmap v2408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3261.731826 secs 2024-04-26T12:45:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:23 smithi194 ceph-mon[33017]: pgmap v2409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3264.459984 secs 2024-04-26T12:45:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3261.731826 secs 2024-04-26T12:45:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:23 smithi033 ceph-mon[27845]: pgmap v2409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3264.459984 secs 2024-04-26T12:45:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3261.731826 secs 2024-04-26T12:45:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:23 smithi096 ceph-mon[32753]: pgmap v2409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3264.459984 secs 2024-04-26T12:45:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:24 smithi194 ceph-mon[33017]: pgmap v2410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:24 smithi033 ceph-mon[27845]: pgmap v2410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:24 smithi096 ceph-mon[32753]: pgmap v2410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:45:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:45:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:45:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:45:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:45:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:26 smithi194 ceph-mon[33017]: pgmap v2411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:26 smithi194 ceph-mon[33017]: pgmap v2412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:45:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:45:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:45:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:26 smithi033 ceph-mon[27845]: pgmap v2411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:26 smithi033 ceph-mon[27845]: pgmap v2412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:45:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:45:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:45:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:26 smithi096 ceph-mon[32753]: pgmap v2411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:26 smithi096 ceph-mon[32753]: pgmap v2412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:45:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3266.731909 secs 2024-04-26T12:45:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3266.731909 secs 2024-04-26T12:45:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3266.731909 secs 2024-04-26T12:45:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3269.460127 secs 2024-04-26T12:45:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:28 smithi033 ceph-mon[27845]: pgmap v2413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3269.460127 secs 2024-04-26T12:45:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:28 smithi096 ceph-mon[32753]: pgmap v2413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:29.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3269.460127 secs 2024-04-26T12:45:29.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:28 smithi194 ceph-mon[33017]: pgmap v2413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:31 smithi033 ceph-mon[27845]: pgmap v2414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:31.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:31 smithi096 ceph-mon[32753]: pgmap v2414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:31.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:31 smithi194 ceph-mon[33017]: pgmap v2414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:33 smithi033 ceph-mon[27845]: pgmap v2415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3271.732012 secs 2024-04-26T12:45:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:33 smithi096 ceph-mon[32753]: pgmap v2415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3271.732012 secs 2024-04-26T12:45:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:33 smithi194 ceph-mon[33017]: pgmap v2415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:45:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3271.732012 secs 2024-04-26T12:45:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3274.460287 secs 2024-04-26T12:45:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3274.460287 secs 2024-04-26T12:45:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3274.460287 secs 2024-04-26T12:45:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:35.004+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:35.004+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:35.004+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:45:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:35 smithi033 ceph-mon[27845]: pgmap v2416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:45:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:35 smithi096 ceph-mon[32753]: pgmap v2416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:45:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:35 smithi194 ceph-mon[33017]: pgmap v2416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:45:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:37 smithi033 ceph-mon[27845]: pgmap v2417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:45:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:37 smithi096 ceph-mon[32753]: pgmap v2417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:45:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:37 smithi194 ceph-mon[33017]: pgmap v2417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:45:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3276.732113 secs 2024-04-26T12:45:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3276.732113 secs 2024-04-26T12:45:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3276.732113 secs 2024-04-26T12:45:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3279.460392 secs 2024-04-26T12:45:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:39 smithi033 ceph-mon[27845]: pgmap v2418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3279.460392 secs 2024-04-26T12:45:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:39 smithi096 ceph-mon[32753]: pgmap v2418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3279.460392 secs 2024-04-26T12:45:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:39 smithi194 ceph-mon[33017]: pgmap v2418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:41 smithi033 ceph-mon[27845]: pgmap v2419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:41.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:41 smithi096 ceph-mon[32753]: pgmap v2419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:41 smithi194 ceph-mon[33017]: pgmap v2419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:43 smithi033 ceph-mon[27845]: pgmap v2420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3281.732216 secs 2024-04-26T12:45:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:43 smithi096 ceph-mon[32753]: pgmap v2420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3281.732216 secs 2024-04-26T12:45:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:43 smithi194 ceph-mon[33017]: pgmap v2420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3281.732216 secs 2024-04-26T12:45:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3284.460511 secs 2024-04-26T12:45:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3284.460511 secs 2024-04-26T12:45:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3284.460511 secs 2024-04-26T12:45:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:45 smithi194 ceph-mon[33017]: pgmap v2421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:45 smithi033 ceph-mon[27845]: pgmap v2421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:45 smithi096 ceph-mon[32753]: pgmap v2421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:47 smithi194 ceph-mon[33017]: pgmap v2422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:47 smithi033 ceph-mon[27845]: pgmap v2422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:47 smithi096 ceph-mon[32753]: pgmap v2422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3286.732279 secs 2024-04-26T12:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3286.732279 secs 2024-04-26T12:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3286.732279 secs 2024-04-26T12:45:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3289.460648 secs 2024-04-26T12:45:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:49 smithi194 ceph-mon[33017]: pgmap v2423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3289.460648 secs 2024-04-26T12:45:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:49 smithi033 ceph-mon[27845]: pgmap v2423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3289.460648 secs 2024-04-26T12:45:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:49 smithi096 ceph-mon[32753]: pgmap v2423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:50.013+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:50.013+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:45:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:45:50.013+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:51 smithi194 ceph-mon[33017]: pgmap v2424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:51 smithi033 ceph-mon[27845]: pgmap v2424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:51 smithi096 ceph-mon[32753]: pgmap v2424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:53 smithi194 ceph-mon[33017]: pgmap v2425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3291.732340 secs 2024-04-26T12:45:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:53 smithi033 ceph-mon[27845]: pgmap v2425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3291.732340 secs 2024-04-26T12:45:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:53 smithi096 ceph-mon[32753]: pgmap v2425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3291.732340 secs 2024-04-26T12:45:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3294.460795 secs 2024-04-26T12:45:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3294.460795 secs 2024-04-26T12:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3294.460795 secs 2024-04-26T12:45:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:55 smithi194 ceph-mon[33017]: pgmap v2426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:55 smithi033 ceph-mon[27845]: pgmap v2426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:55 smithi096 ceph-mon[32753]: pgmap v2426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:45:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:57 smithi194 ceph-mon[33017]: pgmap v2427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:57 smithi033 ceph-mon[27845]: pgmap v2427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:57 smithi096 ceph-mon[32753]: pgmap v2427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3296.732400 secs 2024-04-26T12:45:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3296.732400 secs 2024-04-26T12:45:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3296.732400 secs 2024-04-26T12:45:59.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3299.460880 secs 2024-04-26T12:45:59.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:45:59 smithi194 ceph-mon[33017]: pgmap v2428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:59.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3299.460880 secs 2024-04-26T12:45:59.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:45:59 smithi033 ceph-mon[27845]: pgmap v2428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:45:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3299.460880 secs 2024-04-26T12:45:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:45:59 smithi096 ceph-mon[32753]: pgmap v2428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:01.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:01 smithi194 ceph-mon[33017]: pgmap v2429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:01 smithi033 ceph-mon[27845]: pgmap v2429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:01 smithi096 ceph-mon[32753]: pgmap v2429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:03 smithi194 ceph-mon[33017]: pgmap v2430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3301.732488 secs 2024-04-26T12:46:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:03 smithi033 ceph-mon[27845]: pgmap v2430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3301.732488 secs 2024-04-26T12:46:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:03 smithi096 ceph-mon[32753]: pgmap v2430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3301.732488 secs 2024-04-26T12:46:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3304.460958 secs 2024-04-26T12:46:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3304.460958 secs 2024-04-26T12:46:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3304.460958 secs 2024-04-26T12:46:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:05.023+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:05.023+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:05.023+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:46:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:05 smithi194 ceph-mon[33017]: pgmap v2431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:05 smithi033 ceph-mon[27845]: pgmap v2431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:05 smithi096 ceph-mon[32753]: pgmap v2431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:07 smithi194 ceph-mon[33017]: pgmap v2432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:07 smithi033 ceph-mon[27845]: pgmap v2432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:07 smithi096 ceph-mon[32753]: pgmap v2432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3306.732574 secs 2024-04-26T12:46:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3306.732574 secs 2024-04-26T12:46:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3306.732574 secs 2024-04-26T12:46:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3309.461078 secs 2024-04-26T12:46:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:09 smithi194 ceph-mon[33017]: pgmap v2433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3309.461078 secs 2024-04-26T12:46:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:09 smithi033 ceph-mon[27845]: pgmap v2433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3309.461078 secs 2024-04-26T12:46:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:09 smithi096 ceph-mon[32753]: pgmap v2433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:11 smithi194 ceph-mon[33017]: pgmap v2434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:11 smithi033 ceph-mon[27845]: pgmap v2434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:11 smithi096 ceph-mon[32753]: pgmap v2434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:13 smithi194 ceph-mon[33017]: pgmap v2435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3311.732658 secs 2024-04-26T12:46:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:13 smithi033 ceph-mon[27845]: pgmap v2435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3311.732658 secs 2024-04-26T12:46:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:13 smithi096 ceph-mon[32753]: pgmap v2435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3311.732658 secs 2024-04-26T12:46:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3314.461177 secs 2024-04-26T12:46:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3314.461177 secs 2024-04-26T12:46:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3314.461177 secs 2024-04-26T12:46:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:15 smithi194 ceph-mon[33017]: pgmap v2436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:15 smithi033 ceph-mon[27845]: pgmap v2436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:15 smithi096 ceph-mon[32753]: pgmap v2436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:17 smithi194 ceph-mon[33017]: pgmap v2437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:17 smithi033 ceph-mon[27845]: pgmap v2437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:17 smithi096 ceph-mon[32753]: pgmap v2437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3316.732744 secs 2024-04-26T12:46:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3316.732744 secs 2024-04-26T12:46:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3316.732744 secs 2024-04-26T12:46:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3319.461273 secs 2024-04-26T12:46:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:19 smithi194 ceph-mon[33017]: pgmap v2438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3319.461273 secs 2024-04-26T12:46:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:19 smithi033 ceph-mon[27845]: pgmap v2438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3319.461273 secs 2024-04-26T12:46:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:19 smithi096 ceph-mon[32753]: pgmap v2438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:20.035+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:20.035+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:46:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:20.035+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:21 smithi194 ceph-mon[33017]: pgmap v2439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:21 smithi033 ceph-mon[27845]: pgmap v2439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:21 smithi096 ceph-mon[32753]: pgmap v2439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:23 smithi194 ceph-mon[33017]: pgmap v2440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3321.732828 secs 2024-04-26T12:46:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:23 smithi033 ceph-mon[27845]: pgmap v2440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3321.732828 secs 2024-04-26T12:46:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:23 smithi096 ceph-mon[32753]: pgmap v2440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3321.732828 secs 2024-04-26T12:46:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3324.461393 secs 2024-04-26T12:46:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3324.461393 secs 2024-04-26T12:46:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3324.461393 secs 2024-04-26T12:46:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:25 smithi033 ceph-mon[27845]: pgmap v2441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:25.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:25 smithi096 ceph-mon[32753]: pgmap v2441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:25 smithi194 ceph-mon[33017]: pgmap v2441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:27 smithi033 ceph-mon[27845]: pgmap v2442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:46:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:27 smithi096 ceph-mon[32753]: pgmap v2442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:46:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:27 smithi194 ceph-mon[33017]: pgmap v2442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:46:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3326.732888 secs 2024-04-26T12:46:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3329.461510 secs 2024-04-26T12:46:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:46:28.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:46:28.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:46:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3326.732888 secs 2024-04-26T12:46:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3329.461510 secs 2024-04-26T12:46:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:46:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:46:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:46:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3326.732888 secs 2024-04-26T12:46:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3329.461510 secs 2024-04-26T12:46:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:46:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:46:29.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:46:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:29 smithi033 ceph-mon[27845]: pgmap v2443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:29 smithi033 ceph-mon[27845]: pgmap v2444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:29 smithi096 ceph-mon[32753]: pgmap v2443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:29 smithi096 ceph-mon[32753]: pgmap v2444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:29 smithi194 ceph-mon[33017]: pgmap v2443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:29 smithi194 ceph-mon[33017]: pgmap v2444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:31 smithi033 ceph-mon[27845]: pgmap v2445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:31 smithi096 ceph-mon[32753]: pgmap v2445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:31 smithi194 ceph-mon[33017]: pgmap v2445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:33 smithi033 ceph-mon[27845]: pgmap v2446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3331.732993 secs 2024-04-26T12:46:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3334.461629 secs 2024-04-26T12:46:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:33 smithi096 ceph-mon[32753]: pgmap v2446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3331.732993 secs 2024-04-26T12:46:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3334.461629 secs 2024-04-26T12:46:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:33 smithi194 ceph-mon[33017]: pgmap v2446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3331.732993 secs 2024-04-26T12:46:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3334.461629 secs 2024-04-26T12:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:35.042+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:35.042+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:35.042+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:46:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:35 smithi033 ceph-mon[27845]: pgmap v2447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:46:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:35 smithi096 ceph-mon[32753]: pgmap v2447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:35 smithi194 ceph-mon[33017]: pgmap v2447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:46:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:37 smithi033 ceph-mon[27845]: pgmap v2448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:37.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:37 smithi096 ceph-mon[32753]: pgmap v2448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:37 smithi194 ceph-mon[33017]: pgmap v2448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:46:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3336.733082 secs 2024-04-26T12:46:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3339.461748 secs 2024-04-26T12:46:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3336.733082 secs 2024-04-26T12:46:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3339.461748 secs 2024-04-26T12:46:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3336.733082 secs 2024-04-26T12:46:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3339.461748 secs 2024-04-26T12:46:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:39 smithi033 ceph-mon[27845]: pgmap v2449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:46:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:39 smithi096 ceph-mon[32753]: pgmap v2449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:46:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:39 smithi194 ceph-mon[33017]: pgmap v2449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:46:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:41 smithi033 ceph-mon[27845]: pgmap v2450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:41 smithi096 ceph-mon[32753]: pgmap v2450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:41 smithi194 ceph-mon[33017]: pgmap v2450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:43 smithi033 ceph-mon[27845]: pgmap v2451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3341.733169 secs 2024-04-26T12:46:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3344.461866 secs 2024-04-26T12:46:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:43 smithi096 ceph-mon[32753]: pgmap v2451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3341.733169 secs 2024-04-26T12:46:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3344.461866 secs 2024-04-26T12:46:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:43 smithi194 ceph-mon[33017]: pgmap v2451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3341.733169 secs 2024-04-26T12:46:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3344.461866 secs 2024-04-26T12:46:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:45 smithi033 ceph-mon[27845]: pgmap v2452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:45.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:45 smithi096 ceph-mon[32753]: pgmap v2452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:45 smithi194 ceph-mon[33017]: pgmap v2452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:47 smithi096 ceph-mon[32753]: pgmap v2453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:47 smithi194 ceph-mon[33017]: pgmap v2453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:47 smithi033 ceph-mon[27845]: pgmap v2453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3346.733228 secs 2024-04-26T12:46:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3349.461990 secs 2024-04-26T12:46:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3346.733228 secs 2024-04-26T12:46:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3349.461990 secs 2024-04-26T12:46:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3346.733228 secs 2024-04-26T12:46:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3349.461990 secs 2024-04-26T12:46:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:50.052+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:50.052+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:46:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:46:50.052+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:46:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:49 smithi194 ceph-mon[33017]: pgmap v2454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:49 smithi033 ceph-mon[27845]: pgmap v2454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:49 smithi096 ceph-mon[32753]: pgmap v2454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:51 smithi194 ceph-mon[33017]: pgmap v2455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:51 smithi033 ceph-mon[27845]: pgmap v2455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:51 smithi096 ceph-mon[32753]: pgmap v2455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:53 smithi194 ceph-mon[33017]: pgmap v2456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3351.733316 secs 2024-04-26T12:46:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3354.462070 secs 2024-04-26T12:46:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:53 smithi033 ceph-mon[27845]: pgmap v2456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3351.733316 secs 2024-04-26T12:46:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3354.462070 secs 2024-04-26T12:46:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:53 smithi096 ceph-mon[32753]: pgmap v2456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3351.733316 secs 2024-04-26T12:46:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3354.462070 secs 2024-04-26T12:46:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:55 smithi194 ceph-mon[33017]: pgmap v2457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:55 smithi033 ceph-mon[27845]: pgmap v2457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:55 smithi096 ceph-mon[32753]: pgmap v2457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:46:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:57 smithi194 ceph-mon[33017]: pgmap v2458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:57 smithi033 ceph-mon[27845]: pgmap v2458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:57 smithi096 ceph-mon[32753]: pgmap v2458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:46:59.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3356.733394 secs 2024-04-26T12:46:59.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3359.462209 secs 2024-04-26T12:46:59.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3356.733394 secs 2024-04-26T12:46:59.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3359.462209 secs 2024-04-26T12:46:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3356.733394 secs 2024-04-26T12:46:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3359.462209 secs 2024-04-26T12:47:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:46:59 smithi194 ceph-mon[33017]: pgmap v2459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:46:59 smithi033 ceph-mon[27845]: pgmap v2459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:46:59 smithi096 ceph-mon[32753]: pgmap v2459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:01 smithi194 ceph-mon[33017]: pgmap v2460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:01 smithi033 ceph-mon[27845]: pgmap v2460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:01 smithi096 ceph-mon[32753]: pgmap v2460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:03 smithi194 ceph-mon[33017]: pgmap v2461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3361.733480 secs 2024-04-26T12:47:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3364.462344 secs 2024-04-26T12:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:03 smithi033 ceph-mon[27845]: pgmap v2461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3361.733480 secs 2024-04-26T12:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3364.462344 secs 2024-04-26T12:47:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:03 smithi096 ceph-mon[32753]: pgmap v2461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3361.733480 secs 2024-04-26T12:47:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3364.462344 secs 2024-04-26T12:47:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:05.068+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:05.068+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:05.068+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:47:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:05 smithi194 ceph-mon[33017]: pgmap v2462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:05 smithi033 ceph-mon[27845]: pgmap v2462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:05 smithi096 ceph-mon[32753]: pgmap v2462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:07 smithi194 ceph-mon[33017]: pgmap v2463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:07 smithi033 ceph-mon[27845]: pgmap v2463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:08.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:07 smithi096 ceph-mon[32753]: pgmap v2463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3366.733563 secs 2024-04-26T12:47:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3369.462465 secs 2024-04-26T12:47:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:08 smithi194 ceph-mon[33017]: pgmap v2464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3366.733563 secs 2024-04-26T12:47:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3369.462465 secs 2024-04-26T12:47:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:08 smithi033 ceph-mon[27845]: pgmap v2464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3366.733563 secs 2024-04-26T12:47:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3369.462465 secs 2024-04-26T12:47:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:08 smithi096 ceph-mon[32753]: pgmap v2464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:11.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:11 smithi033 ceph-mon[27845]: pgmap v2465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:11 smithi096 ceph-mon[32753]: pgmap v2465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:11 smithi194 ceph-mon[33017]: pgmap v2465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:13 smithi033 ceph-mon[27845]: pgmap v2466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3371.733645 secs 2024-04-26T12:47:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:13 smithi096 ceph-mon[32753]: pgmap v2466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:13.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3371.733645 secs 2024-04-26T12:47:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:13 smithi194 ceph-mon[33017]: pgmap v2466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3371.733645 secs 2024-04-26T12:47:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3374.462578 secs 2024-04-26T12:47:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3374.462578 secs 2024-04-26T12:47:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3374.462578 secs 2024-04-26T12:47:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:15 smithi033 ceph-mon[27845]: pgmap v2467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:15 smithi096 ceph-mon[32753]: pgmap v2467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:15 smithi194 ceph-mon[33017]: pgmap v2467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:17 smithi033 ceph-mon[27845]: pgmap v2468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:17 smithi096 ceph-mon[32753]: pgmap v2468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:17 smithi194 ceph-mon[33017]: pgmap v2468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3376.733729 secs 2024-04-26T12:47:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3376.733729 secs 2024-04-26T12:47:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3376.733729 secs 2024-04-26T12:47:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3379.462713 secs 2024-04-26T12:47:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:19 smithi033 ceph-mon[27845]: pgmap v2469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3379.462713 secs 2024-04-26T12:47:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:19 smithi096 ceph-mon[32753]: pgmap v2469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3379.462713 secs 2024-04-26T12:47:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:19 smithi194 ceph-mon[33017]: pgmap v2469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:20.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:20.078+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:20.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:20.078+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:20.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:20.078+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:47:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:21 smithi033 ceph-mon[27845]: pgmap v2470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:21 smithi096 ceph-mon[32753]: pgmap v2470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:21 smithi194 ceph-mon[33017]: pgmap v2470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.366+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T12:47:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.366+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T12:47:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.367+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T12:47:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.367+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feee80d0020 con 0x7fef00077780 2024-04-26T12:47:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.367+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:47:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.367+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:47:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.368+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:47:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:22.368+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:47:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:23 smithi033 ceph-mon[27845]: pgmap v2471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3381.733815 secs 2024-04-26T12:47:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:23 smithi096 ceph-mon[32753]: pgmap v2471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3381.733815 secs 2024-04-26T12:47:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:23 smithi194 ceph-mon[33017]: pgmap v2471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3381.733815 secs 2024-04-26T12:47:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3384.462864 secs 2024-04-26T12:47:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3384.462864 secs 2024-04-26T12:47:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3384.462864 secs 2024-04-26T12:47:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:25 smithi194 ceph-mon[33017]: pgmap v2472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:25 smithi033 ceph-mon[27845]: pgmap v2472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:25 smithi096 ceph-mon[32753]: pgmap v2472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:27 smithi194 ceph-mon[33017]: pgmap v2473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:27 smithi033 ceph-mon[27845]: pgmap v2473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:27 smithi096 ceph-mon[32753]: pgmap v2473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3386.733901 secs 2024-04-26T12:47:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3386.733901 secs 2024-04-26T12:47:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3386.733901 secs 2024-04-26T12:47:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3389.463011 secs 2024-04-26T12:47:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:29 smithi194 ceph-mon[33017]: pgmap v2474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:47:29.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3389.463011 secs 2024-04-26T12:47:29.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:29 smithi033 ceph-mon[27845]: pgmap v2474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:29.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:47:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3389.463011 secs 2024-04-26T12:47:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:29 smithi096 ceph-mon[32753]: pgmap v2474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:47:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:47:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:47:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:30 smithi194 ceph-mon[33017]: pgmap v2475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:47:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:30 smithi194 ceph-mon[33017]: pgmap v2476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:30.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:47:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:47:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:47:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:30 smithi033 ceph-mon[27845]: pgmap v2475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:47:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:30 smithi033 ceph-mon[27845]: pgmap v2476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:30.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:47:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:47:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:47:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:30 smithi096 ceph-mon[32753]: pgmap v2475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:47:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:30 smithi096 ceph-mon[32753]: pgmap v2476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:47:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:31 smithi194 ceph-mon[33017]: pgmap v2477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:32.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:31 smithi033 ceph-mon[27845]: pgmap v2477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:31 smithi096 ceph-mon[32753]: pgmap v2477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3391.733986 secs 2024-04-26T12:47:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3391.733986 secs 2024-04-26T12:47:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3391.733986 secs 2024-04-26T12:47:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3394.463075 secs 2024-04-26T12:47:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:33 smithi096 ceph-mon[32753]: pgmap v2478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3394.463075 secs 2024-04-26T12:47:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:33 smithi194 ceph-mon[33017]: pgmap v2478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3394.463075 secs 2024-04-26T12:47:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:33 smithi033 ceph-mon[27845]: pgmap v2478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:35.091+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:35.091+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:35.091+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:47:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:35 smithi194 ceph-mon[33017]: pgmap v2479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:35 smithi033 ceph-mon[27845]: pgmap v2479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:35 smithi096 ceph-mon[32753]: pgmap v2479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:37 smithi194 ceph-mon[33017]: pgmap v2480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:37 smithi033 ceph-mon[27845]: pgmap v2480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:37 smithi096 ceph-mon[32753]: pgmap v2480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:47:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3396.734076 secs 2024-04-26T12:47:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3399.463204 secs 2024-04-26T12:47:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3396.734076 secs 2024-04-26T12:47:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3399.463204 secs 2024-04-26T12:47:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3396.734076 secs 2024-04-26T12:47:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3399.463204 secs 2024-04-26T12:47:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:39 smithi194 ceph-mon[33017]: pgmap v2481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:47:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:39 smithi033 ceph-mon[27845]: pgmap v2481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:47:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:39 smithi096 ceph-mon[32753]: pgmap v2481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:47:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:41 smithi194 ceph-mon[33017]: pgmap v2482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:41 smithi033 ceph-mon[27845]: pgmap v2482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:42.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:41 smithi096 ceph-mon[32753]: pgmap v2482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3401.734160 secs 2024-04-26T12:47:43.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3401.734160 secs 2024-04-26T12:47:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3401.734160 secs 2024-04-26T12:47:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3404.463319 secs 2024-04-26T12:47:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:43 smithi096 ceph-mon[32753]: pgmap v2483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3404.463319 secs 2024-04-26T12:47:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:43 smithi194 ceph-mon[33017]: pgmap v2483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3404.463319 secs 2024-04-26T12:47:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:43 smithi033 ceph-mon[27845]: pgmap v2483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:45 smithi194 ceph-mon[33017]: pgmap v2484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:45 smithi033 ceph-mon[27845]: pgmap v2484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:46.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:45 smithi096 ceph-mon[32753]: pgmap v2484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:47 smithi194 ceph-mon[33017]: pgmap v2485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:47 smithi033 ceph-mon[27845]: pgmap v2485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:47 smithi096 ceph-mon[32753]: pgmap v2485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3406.734222 secs 2024-04-26T12:47:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3409.463431 secs 2024-04-26T12:47:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3406.734222 secs 2024-04-26T12:47:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3409.463431 secs 2024-04-26T12:47:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3406.734222 secs 2024-04-26T12:47:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3409.463431 secs 2024-04-26T12:47:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:50.098+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:50.098+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:47:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:47:50.098+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:47:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:49 smithi194 ceph-mon[33017]: pgmap v2486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:49 smithi033 ceph-mon[27845]: pgmap v2486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:49 smithi096 ceph-mon[32753]: pgmap v2486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:51 smithi194 ceph-mon[33017]: pgmap v2487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:51 smithi033 ceph-mon[27845]: pgmap v2487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:51 smithi096 ceph-mon[32753]: pgmap v2487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3411.734311 secs 2024-04-26T12:47:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3411.734311 secs 2024-04-26T12:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3411.734311 secs 2024-04-26T12:47:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3414.463554 secs 2024-04-26T12:47:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:53 smithi096 ceph-mon[32753]: pgmap v2488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3414.463554 secs 2024-04-26T12:47:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:53 smithi194 ceph-mon[33017]: pgmap v2488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3414.463554 secs 2024-04-26T12:47:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:53 smithi033 ceph-mon[27845]: pgmap v2488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:55 smithi194 ceph-mon[33017]: pgmap v2489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:55 smithi033 ceph-mon[27845]: pgmap v2489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:55 smithi096 ceph-mon[32753]: pgmap v2489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:47:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:57 smithi194 ceph-mon[33017]: pgmap v2490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:57 smithi033 ceph-mon[27845]: pgmap v2490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:57 smithi096 ceph-mon[32753]: pgmap v2490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:47:59.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3416.734372 secs 2024-04-26T12:47:59.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3419.463640 secs 2024-04-26T12:47:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3416.734372 secs 2024-04-26T12:47:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3419.463640 secs 2024-04-26T12:47:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3416.734372 secs 2024-04-26T12:47:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3419.463640 secs 2024-04-26T12:48:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:47:59 smithi194 ceph-mon[33017]: pgmap v2491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:47:59 smithi033 ceph-mon[27845]: pgmap v2491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:47:59 smithi096 ceph-mon[32753]: pgmap v2491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:01 smithi194 ceph-mon[33017]: pgmap v2492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:01 smithi033 ceph-mon[27845]: pgmap v2492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:01 smithi096 ceph-mon[32753]: pgmap v2492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3421.734431 secs 2024-04-26T12:48:03.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3421.734431 secs 2024-04-26T12:48:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3421.734431 secs 2024-04-26T12:48:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3424.463702 secs 2024-04-26T12:48:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:03 smithi096 ceph-mon[32753]: pgmap v2493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3424.463702 secs 2024-04-26T12:48:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:03 smithi194 ceph-mon[33017]: pgmap v2493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3424.463702 secs 2024-04-26T12:48:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:03 smithi033 ceph-mon[27845]: pgmap v2493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:05.113+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:05.113+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:05.113+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:48:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:05 smithi194 ceph-mon[33017]: pgmap v2494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:05 smithi033 ceph-mon[27845]: pgmap v2494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:05 smithi096 ceph-mon[32753]: pgmap v2494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:07 smithi194 ceph-mon[33017]: pgmap v2495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:07 smithi033 ceph-mon[27845]: pgmap v2495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:07 smithi096 ceph-mon[32753]: pgmap v2495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3426.734515 secs 2024-04-26T12:48:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3429.463824 secs 2024-04-26T12:48:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3426.734515 secs 2024-04-26T12:48:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3429.463824 secs 2024-04-26T12:48:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3426.734515 secs 2024-04-26T12:48:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3429.463824 secs 2024-04-26T12:48:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:09 smithi194 ceph-mon[33017]: pgmap v2496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:09 smithi033 ceph-mon[27845]: pgmap v2496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:09 smithi096 ceph-mon[32753]: pgmap v2496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:11 smithi194 ceph-mon[33017]: pgmap v2497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:11 smithi033 ceph-mon[27845]: pgmap v2497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:11 smithi096 ceph-mon[32753]: pgmap v2497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3431.734597 secs 2024-04-26T12:48:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3431.734597 secs 2024-04-26T12:48:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3431.734597 secs 2024-04-26T12:48:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3434.463945 secs 2024-04-26T12:48:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:13 smithi096 ceph-mon[32753]: pgmap v2498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3434.463945 secs 2024-04-26T12:48:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:13 smithi194 ceph-mon[33017]: pgmap v2498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3434.463945 secs 2024-04-26T12:48:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:13 smithi033 ceph-mon[27845]: pgmap v2498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:15 smithi194 ceph-mon[33017]: pgmap v2499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:15 smithi033 ceph-mon[27845]: pgmap v2499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:15 smithi096 ceph-mon[32753]: pgmap v2499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:17 smithi194 ceph-mon[33017]: pgmap v2500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:17 smithi033 ceph-mon[27845]: pgmap v2500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:17 smithi096 ceph-mon[32753]: pgmap v2500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3436.734683 secs 2024-04-26T12:48:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3439.464089 secs 2024-04-26T12:48:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3436.734683 secs 2024-04-26T12:48:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3439.464089 secs 2024-04-26T12:48:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3436.734683 secs 2024-04-26T12:48:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3439.464089 secs 2024-04-26T12:48:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:19 smithi194 ceph-mon[33017]: pgmap v2501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:20.125+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:20.125+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:20.125+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:48:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:19 smithi033 ceph-mon[27845]: pgmap v2501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:19 smithi096 ceph-mon[32753]: pgmap v2501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:21 smithi194 ceph-mon[33017]: pgmap v2502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:21 smithi033 ceph-mon[27845]: pgmap v2502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:21 smithi096 ceph-mon[32753]: pgmap v2502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3441.734767 secs 2024-04-26T12:48:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3441.734767 secs 2024-04-26T12:48:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3441.734767 secs 2024-04-26T12:48:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3444.464210 secs 2024-04-26T12:48:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:23 smithi096 ceph-mon[32753]: pgmap v2503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3444.464210 secs 2024-04-26T12:48:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:23 smithi194 ceph-mon[33017]: pgmap v2503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3444.464210 secs 2024-04-26T12:48:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:23 smithi033 ceph-mon[27845]: pgmap v2503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:25 smithi194 ceph-mon[33017]: pgmap v2504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:25 smithi033 ceph-mon[27845]: pgmap v2504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:25 smithi096 ceph-mon[32753]: pgmap v2504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:27 smithi194 ceph-mon[33017]: pgmap v2505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:27 smithi033 ceph-mon[27845]: pgmap v2505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:27 smithi096 ceph-mon[32753]: pgmap v2505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:29.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3446.734850 secs 2024-04-26T12:48:29.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3449.464338 secs 2024-04-26T12:48:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3446.734850 secs 2024-04-26T12:48:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3449.464338 secs 2024-04-26T12:48:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3446.734850 secs 2024-04-26T12:48:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3449.464338 secs 2024-04-26T12:48:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:29 smithi194 ceph-mon[33017]: pgmap v2506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:29 smithi033 ceph-mon[27845]: pgmap v2506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:29 smithi096 ceph-mon[32753]: pgmap v2506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:48:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:48:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:48:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:48:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:48:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:31 smithi194 ceph-mon[33017]: pgmap v2507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:31 smithi194 ceph-mon[33017]: pgmap v2508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:48:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:48:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:48:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:31 smithi033 ceph-mon[27845]: pgmap v2507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:31 smithi033 ceph-mon[27845]: pgmap v2508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:48:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:48:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:48:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:31 smithi096 ceph-mon[32753]: pgmap v2507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:31 smithi096 ceph-mon[32753]: pgmap v2508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:48:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3451.734934 secs 2024-04-26T12:48:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3451.734934 secs 2024-04-26T12:48:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3451.734934 secs 2024-04-26T12:48:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3454.464467 secs 2024-04-26T12:48:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:33 smithi096 ceph-mon[32753]: pgmap v2509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3454.464467 secs 2024-04-26T12:48:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:33 smithi194 ceph-mon[33017]: pgmap v2509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3454.464467 secs 2024-04-26T12:48:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:33 smithi033 ceph-mon[27845]: pgmap v2509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:35.142+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:35.142+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:35.142+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:48:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:36 smithi194 ceph-mon[33017]: pgmap v2510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:36 smithi033 ceph-mon[27845]: pgmap v2510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:36 smithi096 ceph-mon[32753]: pgmap v2510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:38 smithi194 ceph-mon[33017]: pgmap v2511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3456.735004 secs 2024-04-26T12:48:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:38 smithi033 ceph-mon[27845]: pgmap v2511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3456.735004 secs 2024-04-26T12:48:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:38 smithi096 ceph-mon[32753]: pgmap v2511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3456.735004 secs 2024-04-26T12:48:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3459.464613 secs 2024-04-26T12:48:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3459.464613 secs 2024-04-26T12:48:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3459.464613 secs 2024-04-26T12:48:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:40 smithi194 ceph-mon[33017]: pgmap v2512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:40 smithi033 ceph-mon[27845]: pgmap v2512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:40 smithi096 ceph-mon[32753]: pgmap v2512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:48:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:42 smithi194 ceph-mon[33017]: pgmap v2513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:48:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:42 smithi033 ceph-mon[27845]: pgmap v2513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:48:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:42 smithi096 ceph-mon[32753]: pgmap v2513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:48:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3461.735066 secs 2024-04-26T12:48:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3461.735066 secs 2024-04-26T12:48:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3461.735066 secs 2024-04-26T12:48:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3464.464694 secs 2024-04-26T12:48:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:44 smithi194 ceph-mon[33017]: pgmap v2514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3464.464694 secs 2024-04-26T12:48:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:44 smithi033 ceph-mon[27845]: pgmap v2514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3464.464694 secs 2024-04-26T12:48:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:44 smithi096 ceph-mon[32753]: pgmap v2514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:46 smithi194 ceph-mon[33017]: pgmap v2515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:46 smithi033 ceph-mon[27845]: pgmap v2515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:46 smithi096 ceph-mon[32753]: pgmap v2515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:48 smithi194 ceph-mon[33017]: pgmap v2516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3466.735126 secs 2024-04-26T12:48:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:48 smithi033 ceph-mon[27845]: pgmap v2516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3466.735126 secs 2024-04-26T12:48:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:48 smithi096 ceph-mon[32753]: pgmap v2516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3466.735126 secs 2024-04-26T12:48:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3469.464795 secs 2024-04-26T12:48:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3469.464795 secs 2024-04-26T12:48:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3469.464795 secs 2024-04-26T12:48:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:50.151+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:50.151+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:48:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:48:50.151+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:48:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:50 smithi194 ceph-mon[33017]: pgmap v2517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:50 smithi033 ceph-mon[27845]: pgmap v2517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:50 smithi096 ceph-mon[32753]: pgmap v2517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:52 smithi194 ceph-mon[33017]: pgmap v2518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:52 smithi033 ceph-mon[27845]: pgmap v2518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:52.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:52 smithi096 ceph-mon[32753]: pgmap v2518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3471.735188 secs 2024-04-26T12:48:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3471.735188 secs 2024-04-26T12:48:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3471.735188 secs 2024-04-26T12:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3474.464921 secs 2024-04-26T12:48:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:54 smithi194 ceph-mon[33017]: pgmap v2519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3474.464921 secs 2024-04-26T12:48:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:54 smithi033 ceph-mon[27845]: pgmap v2519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3474.464921 secs 2024-04-26T12:48:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:54 smithi096 ceph-mon[32753]: pgmap v2519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:56 smithi194 ceph-mon[33017]: pgmap v2520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:56 smithi033 ceph-mon[27845]: pgmap v2520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:56 smithi096 ceph-mon[32753]: pgmap v2520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:48:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:58 smithi033 ceph-mon[27845]: pgmap v2521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3476.735250 secs 2024-04-26T12:48:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:58 smithi096 ceph-mon[32753]: pgmap v2521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3476.735250 secs 2024-04-26T12:48:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:58 smithi194 ceph-mon[33017]: pgmap v2521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:48:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3476.735250 secs 2024-04-26T12:48:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3479.465050 secs 2024-04-26T12:48:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3479.465050 secs 2024-04-26T12:49:00.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3479.465050 secs 2024-04-26T12:49:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:00 smithi033 ceph-mon[27845]: pgmap v2522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:00 smithi096 ceph-mon[32753]: pgmap v2522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:00 smithi194 ceph-mon[33017]: pgmap v2522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:02 smithi033 ceph-mon[27845]: pgmap v2523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:02 smithi096 ceph-mon[32753]: pgmap v2523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:02 smithi194 ceph-mon[33017]: pgmap v2523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3481.735334 secs 2024-04-26T12:49:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3481.735334 secs 2024-04-26T12:49:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3481.735334 secs 2024-04-26T12:49:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3484.465186 secs 2024-04-26T12:49:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:04 smithi033 ceph-mon[27845]: pgmap v2524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3484.465186 secs 2024-04-26T12:49:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:04 smithi096 ceph-mon[32753]: pgmap v2524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3484.465186 secs 2024-04-26T12:49:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:04 smithi194 ceph-mon[33017]: pgmap v2524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:05.165+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:05.165+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:05.165+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:49:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:06 smithi033 ceph-mon[27845]: pgmap v2525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:06 smithi096 ceph-mon[32753]: pgmap v2525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:06 smithi194 ceph-mon[33017]: pgmap v2525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:08 smithi033 ceph-mon[27845]: pgmap v2526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3486.735394 secs 2024-04-26T12:49:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3489.465269 secs 2024-04-26T12:49:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:08 smithi096 ceph-mon[32753]: pgmap v2526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3486.735394 secs 2024-04-26T12:49:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3489.465269 secs 2024-04-26T12:49:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:08 smithi194 ceph-mon[33017]: pgmap v2526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3486.735394 secs 2024-04-26T12:49:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3489.465269 secs 2024-04-26T12:49:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:10 smithi033 ceph-mon[27845]: pgmap v2527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:10 smithi096 ceph-mon[32753]: pgmap v2527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:10 smithi194 ceph-mon[33017]: pgmap v2527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:12 smithi033 ceph-mon[27845]: pgmap v2528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:12 smithi096 ceph-mon[32753]: pgmap v2528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:12 smithi194 ceph-mon[33017]: pgmap v2528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3491.735455 secs 2024-04-26T12:49:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3494.465398 secs 2024-04-26T12:49:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3491.735455 secs 2024-04-26T12:49:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3494.465398 secs 2024-04-26T12:49:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3491.735455 secs 2024-04-26T12:49:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3494.465398 secs 2024-04-26T12:49:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:14 smithi033 ceph-mon[27845]: pgmap v2529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:14 smithi096 ceph-mon[32753]: pgmap v2529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:14 smithi194 ceph-mon[33017]: pgmap v2529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:16 smithi033 ceph-mon[27845]: pgmap v2530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:16 smithi096 ceph-mon[32753]: pgmap v2530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:16 smithi194 ceph-mon[33017]: pgmap v2530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:18 smithi033 ceph-mon[27845]: pgmap v2531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3496.735531 secs 2024-04-26T12:49:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3499.465501 secs 2024-04-26T12:49:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:18 smithi096 ceph-mon[32753]: pgmap v2531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3496.735531 secs 2024-04-26T12:49:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3499.465501 secs 2024-04-26T12:49:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:18 smithi194 ceph-mon[33017]: pgmap v2531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3496.735531 secs 2024-04-26T12:49:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3499.465501 secs 2024-04-26T12:49:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:20.180+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:20.180+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:20.180+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:49:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:20 smithi096 ceph-mon[32753]: pgmap v2532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:20 smithi194 ceph-mon[33017]: pgmap v2532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:20 smithi033 ceph-mon[27845]: pgmap v2532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:22 smithi096 ceph-mon[32753]: pgmap v2533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:22 smithi194 ceph-mon[33017]: pgmap v2533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:22 smithi033 ceph-mon[27845]: pgmap v2533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3501.735615 secs 2024-04-26T12:49:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3504.465588 secs 2024-04-26T12:49:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3501.735615 secs 2024-04-26T12:49:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3504.465588 secs 2024-04-26T12:49:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3501.735615 secs 2024-04-26T12:49:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3504.465588 secs 2024-04-26T12:49:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:24 smithi194 ceph-mon[33017]: pgmap v2534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:24 smithi033 ceph-mon[27845]: pgmap v2534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:24 smithi096 ceph-mon[32753]: pgmap v2534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:49:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:26 smithi194 ceph-mon[33017]: pgmap v2535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:26 smithi033 ceph-mon[27845]: pgmap v2535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:26 smithi096 ceph-mon[32753]: pgmap v2535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:28 smithi194 ceph-mon[33017]: pgmap v2536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3506.735699 secs 2024-04-26T12:49:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3509.465728 secs 2024-04-26T12:49:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:28 smithi033 ceph-mon[27845]: pgmap v2536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3506.735699 secs 2024-04-26T12:49:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3509.465728 secs 2024-04-26T12:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:28 smithi096 ceph-mon[32753]: pgmap v2536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3506.735699 secs 2024-04-26T12:49:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3509.465728 secs 2024-04-26T12:49:30.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:30 smithi033 ceph-mon[27845]: pgmap v2537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:30 smithi194 ceph-mon[33017]: pgmap v2537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:30 smithi096 ceph-mon[32753]: pgmap v2537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:32 smithi194 ceph-mon[33017]: pgmap v2538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:49:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:32 smithi033 ceph-mon[27845]: pgmap v2538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:49:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:32 smithi096 ceph-mon[32753]: pgmap v2538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:49:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3511.735746 secs 2024-04-26T12:49:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3514.465875 secs 2024-04-26T12:49:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:49:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:49:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:49:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3511.735746 secs 2024-04-26T12:49:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3514.465875 secs 2024-04-26T12:49:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:49:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:49:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:49:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3511.735746 secs 2024-04-26T12:49:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3514.465875 secs 2024-04-26T12:49:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:49:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:49:34.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:49:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:34 smithi194 ceph-mon[33017]: pgmap v2539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:34 smithi194 ceph-mon[33017]: pgmap v2540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:34 smithi033 ceph-mon[27845]: pgmap v2539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:34 smithi033 ceph-mon[27845]: pgmap v2540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:34 smithi096 ceph-mon[32753]: pgmap v2539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:34 smithi096 ceph-mon[32753]: pgmap v2540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:35.187+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:35.187+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:49:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:35.187+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:37.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:36 smithi033 ceph-mon[27845]: pgmap v2541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:36 smithi194 ceph-mon[33017]: pgmap v2541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:36 smithi096 ceph-mon[32753]: pgmap v2541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:39.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:38 smithi194 ceph-mon[33017]: pgmap v2542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:39.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3516.735829 secs 2024-04-26T12:49:39.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3519.465941 secs 2024-04-26T12:49:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:38 smithi033 ceph-mon[27845]: pgmap v2542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3516.735829 secs 2024-04-26T12:49:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3519.465941 secs 2024-04-26T12:49:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:38 smithi096 ceph-mon[32753]: pgmap v2542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3516.735829 secs 2024-04-26T12:49:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3519.465941 secs 2024-04-26T12:49:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:40 smithi194 ceph-mon[33017]: pgmap v2543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:49:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:40 smithi033 ceph-mon[27845]: pgmap v2543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:49:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:40 smithi096 ceph-mon[32753]: pgmap v2543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:49:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:42 smithi194 ceph-mon[33017]: pgmap v2544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:42 smithi033 ceph-mon[27845]: pgmap v2544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:42 smithi096 ceph-mon[32753]: pgmap v2544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3521.735914 secs 2024-04-26T12:49:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3524.466034 secs 2024-04-26T12:49:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3521.735914 secs 2024-04-26T12:49:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3524.466034 secs 2024-04-26T12:49:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3521.735914 secs 2024-04-26T12:49:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3524.466034 secs 2024-04-26T12:49:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:44 smithi194 ceph-mon[33017]: pgmap v2545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:49:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:44 smithi033 ceph-mon[27845]: pgmap v2545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:49:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:44 smithi096 ceph-mon[32753]: pgmap v2545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:49:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:46 smithi194 ceph-mon[33017]: pgmap v2546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:46 smithi033 ceph-mon[27845]: pgmap v2546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:46 smithi096 ceph-mon[32753]: pgmap v2546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:48 smithi194 ceph-mon[33017]: pgmap v2547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3526.735977 secs 2024-04-26T12:49:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3529.466153 secs 2024-04-26T12:49:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:48 smithi033 ceph-mon[27845]: pgmap v2547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3526.735977 secs 2024-04-26T12:49:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3529.466153 secs 2024-04-26T12:49:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:48 smithi096 ceph-mon[32753]: pgmap v2547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3526.735977 secs 2024-04-26T12:49:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3529.466153 secs 2024-04-26T12:49:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:50.200+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:50.200+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:49:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:49:50.200+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:49:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:50 smithi194 ceph-mon[33017]: pgmap v2548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:49:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:50 smithi033 ceph-mon[27845]: pgmap v2548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:49:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:50 smithi096 ceph-mon[32753]: pgmap v2548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:49:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:52 smithi194 ceph-mon[33017]: pgmap v2549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:52 smithi033 ceph-mon[27845]: pgmap v2549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:52 smithi096 ceph-mon[32753]: pgmap v2549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:49:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3531.736002 secs 2024-04-26T12:49:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3534.466272 secs 2024-04-26T12:49:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3531.736002 secs 2024-04-26T12:49:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3534.466272 secs 2024-04-26T12:49:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3531.736002 secs 2024-04-26T12:49:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3534.466272 secs 2024-04-26T12:49:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:54 smithi194 ceph-mon[33017]: pgmap v2550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:54 smithi033 ceph-mon[27845]: pgmap v2550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:54 smithi096 ceph-mon[32753]: pgmap v2550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:55 smithi194 ceph-mon[33017]: pgmap v2551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:55 smithi033 ceph-mon[27845]: pgmap v2551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:55 smithi096 ceph-mon[32753]: pgmap v2551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:49:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:58 smithi194 ceph-mon[33017]: pgmap v2552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3536.736064 secs 2024-04-26T12:49:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:58 smithi033 ceph-mon[27845]: pgmap v2552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3536.736064 secs 2024-04-26T12:49:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:58 smithi096 ceph-mon[32753]: pgmap v2552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:49:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3536.736064 secs 2024-04-26T12:49:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3539.466405 secs 2024-04-26T12:49:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3539.466405 secs 2024-04-26T12:49:59.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3539.466405 secs 2024-04-26T12:50:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:00 smithi033 ceph-mon[27845]: pgmap v2553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:50:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:00 smithi194 ceph-mon[33017]: pgmap v2553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:50:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:00 smithi096 ceph-mon[32753]: pgmap v2553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T12:50:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:02 smithi194 ceph-mon[33017]: pgmap v2554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:02.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:02 smithi033 ceph-mon[27845]: pgmap v2554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:02.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:02 smithi096 ceph-mon[32753]: pgmap v2554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3541.736128 secs 2024-04-26T12:50:03.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3541.736128 secs 2024-04-26T12:50:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3541.736128 secs 2024-04-26T12:50:04.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3544.466527 secs 2024-04-26T12:50:04.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:04 smithi096 ceph-mon[32753]: pgmap v2555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3544.466527 secs 2024-04-26T12:50:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:04 smithi194 ceph-mon[33017]: pgmap v2555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3544.466527 secs 2024-04-26T12:50:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:04 smithi033 ceph-mon[27845]: pgmap v2555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:05.201+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:05.201+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:50:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:05.201+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:06 smithi194 ceph-mon[33017]: pgmap v2556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:06 smithi033 ceph-mon[27845]: pgmap v2556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:06 smithi096 ceph-mon[32753]: pgmap v2556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:08 smithi194 ceph-mon[33017]: pgmap v2557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3546.736192 secs 2024-04-26T12:50:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:08 smithi033 ceph-mon[27845]: pgmap v2557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3546.736192 secs 2024-04-26T12:50:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:08 smithi096 ceph-mon[32753]: pgmap v2557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3546.736192 secs 2024-04-26T12:50:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3549.466665 secs 2024-04-26T12:50:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3549.466665 secs 2024-04-26T12:50:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3549.466665 secs 2024-04-26T12:50:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:10 smithi194 ceph-mon[33017]: pgmap v2558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:10 smithi033 ceph-mon[27845]: pgmap v2558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:10 smithi096 ceph-mon[32753]: pgmap v2558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:12 smithi033 ceph-mon[27845]: pgmap v2559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:12.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:12 smithi096 ceph-mon[32753]: pgmap v2559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:12 smithi194 ceph-mon[33017]: pgmap v2559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3551.736254 secs 2024-04-26T12:50:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3551.736254 secs 2024-04-26T12:50:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3551.736254 secs 2024-04-26T12:50:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3554.466802 secs 2024-04-26T12:50:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:14 smithi033 ceph-mon[27845]: pgmap v2560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3554.466802 secs 2024-04-26T12:50:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:14 smithi096 ceph-mon[32753]: pgmap v2560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3554.466802 secs 2024-04-26T12:50:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:14 smithi194 ceph-mon[33017]: pgmap v2560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:16 smithi033 ceph-mon[27845]: pgmap v2561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:16 smithi096 ceph-mon[32753]: pgmap v2561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:16 smithi194 ceph-mon[33017]: pgmap v2561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:18 smithi033 ceph-mon[27845]: pgmap v2562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3556.736335 secs 2024-04-26T12:50:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:18 smithi096 ceph-mon[32753]: pgmap v2562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3556.736335 secs 2024-04-26T12:50:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:18 smithi194 ceph-mon[33017]: pgmap v2562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3556.736335 secs 2024-04-26T12:50:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3559.466920 secs 2024-04-26T12:50:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3559.466920 secs 2024-04-26T12:50:19.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3559.466920 secs 2024-04-26T12:50:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:20.204+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:20.204+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:20.204+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:50:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:20 smithi033 ceph-mon[27845]: pgmap v2563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:20 smithi096 ceph-mon[32753]: pgmap v2563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:20 smithi194 ceph-mon[33017]: pgmap v2563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:22.396+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:50:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:22.396+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:50:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:22.401+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:50:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:22.401+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:50:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:22 smithi033 ceph-mon[27845]: pgmap v2564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:22 smithi096 ceph-mon[32753]: pgmap v2564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:22 smithi194 ceph-mon[33017]: pgmap v2564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3561.736398 secs 2024-04-26T12:50:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3561.736398 secs 2024-04-26T12:50:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3561.736398 secs 2024-04-26T12:50:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3564.467036 secs 2024-04-26T12:50:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:24 smithi033 ceph-mon[27845]: pgmap v2565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3564.467036 secs 2024-04-26T12:50:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:24 smithi096 ceph-mon[32753]: pgmap v2565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3564.467036 secs 2024-04-26T12:50:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:24 smithi194 ceph-mon[33017]: pgmap v2565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:26 smithi033 ceph-mon[27845]: pgmap v2566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:26 smithi096 ceph-mon[32753]: pgmap v2566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:26 smithi194 ceph-mon[33017]: pgmap v2566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:28 smithi096 ceph-mon[32753]: pgmap v2567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3566.736456 secs 2024-04-26T12:50:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:28 smithi194 ceph-mon[33017]: pgmap v2567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3566.736456 secs 2024-04-26T12:50:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:28 smithi033 ceph-mon[27845]: pgmap v2567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3566.736456 secs 2024-04-26T12:50:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3569.467153 secs 2024-04-26T12:50:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3569.467153 secs 2024-04-26T12:50:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3569.467153 secs 2024-04-26T12:50:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:30 smithi194 ceph-mon[33017]: pgmap v2568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:30 smithi033 ceph-mon[27845]: pgmap v2568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:30 smithi096 ceph-mon[32753]: pgmap v2568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:50:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:32 smithi194 ceph-mon[33017]: pgmap v2569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:32 smithi033 ceph-mon[27845]: pgmap v2569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:32 smithi096 ceph-mon[32753]: pgmap v2569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3571.736518 secs 2024-04-26T12:50:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3571.736518 secs 2024-04-26T12:50:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3571.736518 secs 2024-04-26T12:50:34.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3574.467262 secs 2024-04-26T12:50:34.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:34 smithi096 ceph-mon[32753]: pgmap v2570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:34.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:50:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3574.467262 secs 2024-04-26T12:50:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:34 smithi194 ceph-mon[33017]: pgmap v2570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:50:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3574.467262 secs 2024-04-26T12:50:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:34 smithi033 ceph-mon[27845]: pgmap v2570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:50:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:35.207+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:35.208+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:35.208+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:50:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:50:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:50:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:50:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:50:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:50:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:50:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:50:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:50:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:50:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:36 smithi194 ceph-mon[33017]: pgmap v2571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:50:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:36 smithi194 ceph-mon[33017]: pgmap v2572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:36 smithi033 ceph-mon[27845]: pgmap v2571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:50:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:36 smithi033 ceph-mon[27845]: pgmap v2572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:36 smithi096 ceph-mon[32753]: pgmap v2571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:50:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:36 smithi096 ceph-mon[32753]: pgmap v2572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:38 smithi194 ceph-mon[33017]: pgmap v2573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3576.736605 secs 2024-04-26T12:50:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:38 smithi033 ceph-mon[27845]: pgmap v2573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3576.736605 secs 2024-04-26T12:50:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:38 smithi096 ceph-mon[32753]: pgmap v2573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3576.736605 secs 2024-04-26T12:50:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3579.467334 secs 2024-04-26T12:50:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3579.467334 secs 2024-04-26T12:50:39.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3579.467334 secs 2024-04-26T12:50:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:40 smithi194 ceph-mon[33017]: pgmap v2574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:50:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:40 smithi033 ceph-mon[27845]: pgmap v2574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:50:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:40 smithi096 ceph-mon[32753]: pgmap v2574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T12:50:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:42 smithi194 ceph-mon[33017]: pgmap v2575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:42 smithi033 ceph-mon[27845]: pgmap v2575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:42 smithi096 ceph-mon[32753]: pgmap v2575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3581.736675 secs 2024-04-26T12:50:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3581.736675 secs 2024-04-26T12:50:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3581.736675 secs 2024-04-26T12:50:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:44 smithi194 ceph-mon[33017]: pgmap v2576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3584.467452 secs 2024-04-26T12:50:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:44 smithi033 ceph-mon[27845]: pgmap v2576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:44.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3584.467452 secs 2024-04-26T12:50:44.942 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:44 smithi096 ceph-mon[32753]: pgmap v2576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:50:44.942 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3584.467452 secs 2024-04-26T12:50:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:46 smithi194 ceph-mon[33017]: pgmap v2577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:50:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:46 smithi033 ceph-mon[27845]: pgmap v2577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:50:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:46 smithi096 ceph-mon[32753]: pgmap v2577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:50:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:48 smithi194 ceph-mon[33017]: pgmap v2578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3586.736761 secs 2024-04-26T12:50:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:48 smithi033 ceph-mon[27845]: pgmap v2578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3586.736761 secs 2024-04-26T12:50:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:48 smithi096 ceph-mon[32753]: pgmap v2578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3586.736761 secs 2024-04-26T12:50:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3589.467569 secs 2024-04-26T12:50:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3589.467569 secs 2024-04-26T12:50:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3589.467569 secs 2024-04-26T12:50:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:50.220+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:50.220+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:50:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:50:50.220+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:50:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:50 smithi194 ceph-mon[33017]: pgmap v2579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:50 smithi033 ceph-mon[27845]: pgmap v2579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:50 smithi096 ceph-mon[32753]: pgmap v2579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:52 smithi194 ceph-mon[33017]: pgmap v2580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:52 smithi033 ceph-mon[27845]: pgmap v2580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:52 smithi096 ceph-mon[32753]: pgmap v2580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3591.736846 secs 2024-04-26T12:50:53.872 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3591.736846 secs 2024-04-26T12:50:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3591.736846 secs 2024-04-26T12:50:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:54 smithi194 ceph-mon[33017]: pgmap v2581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3594.467686 secs 2024-04-26T12:50:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:54 smithi033 ceph-mon[27845]: pgmap v2581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3594.467686 secs 2024-04-26T12:50:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:54 smithi096 ceph-mon[32753]: pgmap v2581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3594.467686 secs 2024-04-26T12:50:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:56 smithi194 ceph-mon[33017]: pgmap v2582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:56 smithi033 ceph-mon[27845]: pgmap v2582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:56 smithi096 ceph-mon[32753]: pgmap v2582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:50:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:58 smithi194 ceph-mon[33017]: pgmap v2583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3596.736929 secs 2024-04-26T12:50:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:58 smithi033 ceph-mon[27845]: pgmap v2583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3596.736929 secs 2024-04-26T12:50:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:58 smithi096 ceph-mon[32753]: pgmap v2583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:50:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3596.736929 secs 2024-04-26T12:50:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3599.467802 secs 2024-04-26T12:50:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3599.467802 secs 2024-04-26T12:50:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3599.467802 secs 2024-04-26T12:51:00.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:00 smithi033 ceph-mon[27845]: pgmap v2584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:00 smithi194 ceph-mon[33017]: pgmap v2584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:00 smithi096 ceph-mon[32753]: pgmap v2584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:02 smithi194 ceph-mon[33017]: pgmap v2585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:02 smithi033 ceph-mon[27845]: pgmap v2585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:02 smithi096 ceph-mon[32753]: pgmap v2585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3601.737005 secs 2024-04-26T12:51:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3601.737005 secs 2024-04-26T12:51:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3601.737005 secs 2024-04-26T12:51:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:04 smithi194 ceph-mon[33017]: pgmap v2586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3604.467916 secs 2024-04-26T12:51:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:04 smithi033 ceph-mon[27845]: pgmap v2586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3604.467916 secs 2024-04-26T12:51:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:04 smithi096 ceph-mon[32753]: pgmap v2586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3604.467916 secs 2024-04-26T12:51:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:05.232+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:05.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:05.232+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:05.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:05.232+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:51:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:06 smithi194 ceph-mon[33017]: pgmap v2587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:06 smithi033 ceph-mon[27845]: pgmap v2587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:06 smithi096 ceph-mon[32753]: pgmap v2587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:08 smithi033 ceph-mon[27845]: pgmap v2588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3606.737090 secs 2024-04-26T12:51:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:08 smithi096 ceph-mon[32753]: pgmap v2588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3606.737090 secs 2024-04-26T12:51:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:08 smithi194 ceph-mon[33017]: pgmap v2588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3606.737090 secs 2024-04-26T12:51:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3609.468033 secs 2024-04-26T12:51:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3609.468033 secs 2024-04-26T12:51:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3609.468033 secs 2024-04-26T12:51:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:10 smithi033 ceph-mon[27845]: pgmap v2589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:10 smithi096 ceph-mon[32753]: pgmap v2589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:10 smithi194 ceph-mon[33017]: pgmap v2589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:12 smithi033 ceph-mon[27845]: pgmap v2590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:12 smithi096 ceph-mon[32753]: pgmap v2590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:12 smithi194 ceph-mon[33017]: pgmap v2590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3611.737152 secs 2024-04-26T12:51:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3611.737152 secs 2024-04-26T12:51:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3611.737152 secs 2024-04-26T12:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:14 smithi033 ceph-mon[27845]: pgmap v2591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3614.468157 secs 2024-04-26T12:51:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:14 smithi096 ceph-mon[32753]: pgmap v2591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3614.468157 secs 2024-04-26T12:51:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:14 smithi194 ceph-mon[33017]: pgmap v2591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3614.468157 secs 2024-04-26T12:51:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:16 smithi033 ceph-mon[27845]: pgmap v2592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:16 smithi096 ceph-mon[32753]: pgmap v2592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:16 smithi194 ceph-mon[33017]: pgmap v2592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:18 smithi033 ceph-mon[27845]: pgmap v2593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3616.737217 secs 2024-04-26T12:51:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3619.468255 secs 2024-04-26T12:51:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:18 smithi096 ceph-mon[32753]: pgmap v2593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3616.737217 secs 2024-04-26T12:51:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3619.468255 secs 2024-04-26T12:51:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:18 smithi194 ceph-mon[33017]: pgmap v2593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3616.737217 secs 2024-04-26T12:51:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3619.468255 secs 2024-04-26T12:51:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:20.245+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:20.246+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:20.246+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:51:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:20 smithi033 ceph-mon[27845]: pgmap v2594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:20 smithi096 ceph-mon[32753]: pgmap v2594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:20 smithi194 ceph-mon[33017]: pgmap v2594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:22 smithi033 ceph-mon[27845]: pgmap v2595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:22 smithi096 ceph-mon[32753]: pgmap v2595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:23.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:22 smithi194 ceph-mon[33017]: pgmap v2595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3621.737271 secs 2024-04-26T12:51:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3624.468341 secs 2024-04-26T12:51:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3621.737271 secs 2024-04-26T12:51:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3624.468341 secs 2024-04-26T12:51:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3621.737271 secs 2024-04-26T12:51:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3624.468341 secs 2024-04-26T12:51:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:24 smithi033 ceph-mon[27845]: pgmap v2596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:24 smithi096 ceph-mon[32753]: pgmap v2596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:24 smithi194 ceph-mon[33017]: pgmap v2596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:26 smithi096 ceph-mon[32753]: pgmap v2597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:26 smithi194 ceph-mon[33017]: pgmap v2597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:26 smithi033 ceph-mon[27845]: pgmap v2597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:28 smithi096 ceph-mon[32753]: pgmap v2598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3626.737357 secs 2024-04-26T12:51:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3629.468480 secs 2024-04-26T12:51:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:28 smithi194 ceph-mon[33017]: pgmap v2598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3626.737357 secs 2024-04-26T12:51:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3629.468480 secs 2024-04-26T12:51:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:28 smithi033 ceph-mon[27845]: pgmap v2598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3626.737357 secs 2024-04-26T12:51:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3629.468480 secs 2024-04-26T12:51:30.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:30 smithi033 ceph-mon[27845]: pgmap v2599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:30 smithi194 ceph-mon[33017]: pgmap v2599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:30 smithi096 ceph-mon[32753]: pgmap v2599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:32 smithi194 ceph-mon[33017]: pgmap v2600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:32 smithi033 ceph-mon[27845]: pgmap v2600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:32 smithi096 ceph-mon[32753]: pgmap v2600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3631.737438 secs 2024-04-26T12:51:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3634.468604 secs 2024-04-26T12:51:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3631.737438 secs 2024-04-26T12:51:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3634.468604 secs 2024-04-26T12:51:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3631.737438 secs 2024-04-26T12:51:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3634.468604 secs 2024-04-26T12:51:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:34 smithi194 ceph-mon[33017]: pgmap v2601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:34 smithi033 ceph-mon[27845]: pgmap v2601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:34 smithi096 ceph-mon[32753]: pgmap v2601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:35.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:35.262+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:35.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:35.262+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:35.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:35.262+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:51:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:51:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:51:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:51:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:36 smithi194 ceph-mon[33017]: pgmap v2602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:36 smithi033 ceph-mon[27845]: pgmap v2602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:36 smithi096 ceph-mon[32753]: pgmap v2602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:37.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:51:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:51:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:51:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:51:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:51:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:51:38.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:51:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:38 smithi194 ceph-mon[33017]: pgmap v2603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3636.737521 secs 2024-04-26T12:51:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:38 smithi194 ceph-mon[33017]: pgmap v2604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:51:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:38 smithi194 ceph-mon[33017]: pgmap v2605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3639.468731 secs 2024-04-26T12:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:38 smithi033 ceph-mon[27845]: pgmap v2603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3636.737521 secs 2024-04-26T12:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:38 smithi033 ceph-mon[27845]: pgmap v2604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:38 smithi033 ceph-mon[27845]: pgmap v2605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3639.468731 secs 2024-04-26T12:51:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:38 smithi096 ceph-mon[32753]: pgmap v2603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3636.737521 secs 2024-04-26T12:51:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:38 smithi096 ceph-mon[32753]: pgmap v2604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:51:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:38 smithi096 ceph-mon[32753]: pgmap v2605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3639.468731 secs 2024-04-26T12:51:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:40 smithi194 ceph-mon[33017]: pgmap v2606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:40 smithi033 ceph-mon[27845]: pgmap v2606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:40 smithi096 ceph-mon[32753]: pgmap v2606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:42 smithi194 ceph-mon[33017]: pgmap v2607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:42 smithi033 ceph-mon[27845]: pgmap v2607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:42 smithi096 ceph-mon[32753]: pgmap v2607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3641.737610 secs 2024-04-26T12:51:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3644.468849 secs 2024-04-26T12:51:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3641.737610 secs 2024-04-26T12:51:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3644.468849 secs 2024-04-26T12:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3641.737610 secs 2024-04-26T12:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3644.468849 secs 2024-04-26T12:51:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:44 smithi194 ceph-mon[33017]: pgmap v2608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:51:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:44 smithi033 ceph-mon[27845]: pgmap v2608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:51:45.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:44 smithi096 ceph-mon[32753]: pgmap v2608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:51:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:46 smithi194 ceph-mon[33017]: pgmap v2609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:46 smithi033 ceph-mon[27845]: pgmap v2609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:47.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:46 smithi096 ceph-mon[32753]: pgmap v2609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T12:51:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3646.737696 secs 2024-04-26T12:51:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:48 smithi194 ceph-mon[33017]: pgmap v2610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:51:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3649.468924 secs 2024-04-26T12:51:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3646.737696 secs 2024-04-26T12:51:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:48 smithi033 ceph-mon[27845]: pgmap v2610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:51:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3649.468924 secs 2024-04-26T12:51:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3646.737696 secs 2024-04-26T12:51:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:48 smithi096 ceph-mon[32753]: pgmap v2610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:51:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3649.468924 secs 2024-04-26T12:51:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:50.276+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:50.276+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:51:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:51:50.276+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:51:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:50 smithi194 ceph-mon[33017]: pgmap v2611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:50 smithi033 ceph-mon[27845]: pgmap v2611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:50 smithi096 ceph-mon[32753]: pgmap v2611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:52 smithi194 ceph-mon[33017]: pgmap v2612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:52 smithi033 ceph-mon[27845]: pgmap v2612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:52 smithi096 ceph-mon[32753]: pgmap v2612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3651.737775 secs 2024-04-26T12:51:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3654.469059 secs 2024-04-26T12:51:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3651.737775 secs 2024-04-26T12:51:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3654.469059 secs 2024-04-26T12:51:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3651.737775 secs 2024-04-26T12:51:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3654.469059 secs 2024-04-26T12:51:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:54 smithi194 ceph-mon[33017]: pgmap v2613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:54 smithi033 ceph-mon[27845]: pgmap v2613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:54 smithi096 ceph-mon[32753]: pgmap v2613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:56 smithi194 ceph-mon[33017]: pgmap v2614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:56 smithi033 ceph-mon[27845]: pgmap v2614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:56 smithi096 ceph-mon[32753]: pgmap v2614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:51:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3656.737858 secs 2024-04-26T12:51:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:58 smithi194 ceph-mon[33017]: pgmap v2615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:51:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3659.469158 secs 2024-04-26T12:51:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3656.737858 secs 2024-04-26T12:51:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:58 smithi033 ceph-mon[27845]: pgmap v2615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:51:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3659.469158 secs 2024-04-26T12:51:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3656.737858 secs 2024-04-26T12:51:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:58 smithi096 ceph-mon[32753]: pgmap v2615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:51:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:51:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3659.469158 secs 2024-04-26T12:52:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:00 smithi194 ceph-mon[33017]: pgmap v2616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:00 smithi033 ceph-mon[27845]: pgmap v2616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:00 smithi096 ceph-mon[32753]: pgmap v2616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:02 smithi194 ceph-mon[33017]: pgmap v2617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:02 smithi033 ceph-mon[27845]: pgmap v2617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:02 smithi096 ceph-mon[32753]: pgmap v2617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:04.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3661.737938 secs 2024-04-26T12:52:04.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3664.469277 secs 2024-04-26T12:52:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3661.737938 secs 2024-04-26T12:52:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3664.469277 secs 2024-04-26T12:52:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3661.737938 secs 2024-04-26T12:52:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3664.469277 secs 2024-04-26T12:52:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:04 smithi194 ceph-mon[33017]: pgmap v2618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:04 smithi033 ceph-mon[27845]: pgmap v2618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:04 smithi096 ceph-mon[32753]: pgmap v2618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:05.291+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:05.291+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:52:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:05.291+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:05 smithi194 ceph-mon[33017]: pgmap v2619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:05 smithi033 ceph-mon[27845]: pgmap v2619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:05 smithi096 ceph-mon[32753]: pgmap v2619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3666.738023 secs 2024-04-26T12:52:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:08 smithi033 ceph-mon[27845]: pgmap v2620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3666.738023 secs 2024-04-26T12:52:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:08 smithi096 ceph-mon[32753]: pgmap v2620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3666.738023 secs 2024-04-26T12:52:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:08 smithi194 ceph-mon[33017]: pgmap v2620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3669.469401 secs 2024-04-26T12:52:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3669.469401 secs 2024-04-26T12:52:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3669.469401 secs 2024-04-26T12:52:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:10 smithi033 ceph-mon[27845]: pgmap v2621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:10.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:10 smithi096 ceph-mon[32753]: pgmap v2621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:10 smithi194 ceph-mon[33017]: pgmap v2621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:12 smithi033 ceph-mon[27845]: pgmap v2622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:12 smithi096 ceph-mon[32753]: pgmap v2622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:12 smithi194 ceph-mon[33017]: pgmap v2622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3671.738084 secs 2024-04-26T12:52:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3671.738084 secs 2024-04-26T12:52:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3671.738084 secs 2024-04-26T12:52:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3674.469463 secs 2024-04-26T12:52:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:14 smithi033 ceph-mon[27845]: pgmap v2623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3674.469463 secs 2024-04-26T12:52:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:14 smithi096 ceph-mon[32753]: pgmap v2623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3674.469463 secs 2024-04-26T12:52:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:14 smithi194 ceph-mon[33017]: pgmap v2623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:16 smithi033 ceph-mon[27845]: pgmap v2624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:16 smithi096 ceph-mon[32753]: pgmap v2624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:16 smithi194 ceph-mon[33017]: pgmap v2624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3676.738146 secs 2024-04-26T12:52:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:18 smithi033 ceph-mon[27845]: pgmap v2625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3676.738146 secs 2024-04-26T12:52:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:18 smithi096 ceph-mon[32753]: pgmap v2625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3676.738146 secs 2024-04-26T12:52:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:18 smithi194 ceph-mon[33017]: pgmap v2625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3679.469577 secs 2024-04-26T12:52:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3679.469577 secs 2024-04-26T12:52:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3679.469577 secs 2024-04-26T12:52:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:20.307+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:20.307+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:20.307+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:52:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:20 smithi033 ceph-mon[27845]: pgmap v2626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:20 smithi096 ceph-mon[32753]: pgmap v2626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:20 smithi194 ceph-mon[33017]: pgmap v2626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:22 smithi033 ceph-mon[27845]: pgmap v2627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:22 smithi096 ceph-mon[32753]: pgmap v2627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:22 smithi194 ceph-mon[33017]: pgmap v2627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3681.738232 secs 2024-04-26T12:52:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3684.469731 secs 2024-04-26T12:52:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3681.738232 secs 2024-04-26T12:52:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3684.469731 secs 2024-04-26T12:52:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3681.738232 secs 2024-04-26T12:52:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3684.469731 secs 2024-04-26T12:52:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:24 smithi096 ceph-mon[32753]: pgmap v2628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:24 smithi194 ceph-mon[33017]: pgmap v2628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:24 smithi033 ceph-mon[27845]: pgmap v2628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:26 smithi096 ceph-mon[32753]: pgmap v2629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:27.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:26 smithi033 ceph-mon[27845]: pgmap v2629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:26 smithi194 ceph-mon[33017]: pgmap v2629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3686.738290 secs 2024-04-26T12:52:29.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:28 smithi194 ceph-mon[33017]: pgmap v2630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:29.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3689.469854 secs 2024-04-26T12:52:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3686.738290 secs 2024-04-26T12:52:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:28 smithi033 ceph-mon[27845]: pgmap v2630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3689.469854 secs 2024-04-26T12:52:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3686.738290 secs 2024-04-26T12:52:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:28 smithi096 ceph-mon[32753]: pgmap v2630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3689.469854 secs 2024-04-26T12:52:30.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:30 smithi033 ceph-mon[27845]: pgmap v2631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:30 smithi194 ceph-mon[33017]: pgmap v2631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:30 smithi096 ceph-mon[32753]: pgmap v2631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:32 smithi194 ceph-mon[33017]: pgmap v2632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:32 smithi033 ceph-mon[27845]: pgmap v2632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:32 smithi096 ceph-mon[32753]: pgmap v2632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3691.738375 secs 2024-04-26T12:52:34.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3694.469918 secs 2024-04-26T12:52:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3691.738375 secs 2024-04-26T12:52:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3694.469918 secs 2024-04-26T12:52:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3691.738375 secs 2024-04-26T12:52:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3694.469918 secs 2024-04-26T12:52:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:34 smithi194 ceph-mon[33017]: pgmap v2633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:34 smithi033 ceph-mon[27845]: pgmap v2633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:34 smithi096 ceph-mon[32753]: pgmap v2633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:35.322+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:35.322+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:52:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:35.322+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:36 smithi194 ceph-mon[33017]: pgmap v2634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:36 smithi033 ceph-mon[27845]: pgmap v2634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:36 smithi096 ceph-mon[32753]: pgmap v2634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3696.738462 secs 2024-04-26T12:52:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:38 smithi194 ceph-mon[33017]: pgmap v2635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:52:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3699.470015 secs 2024-04-26T12:52:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3696.738462 secs 2024-04-26T12:52:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:38 smithi033 ceph-mon[27845]: pgmap v2635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:52:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3699.470015 secs 2024-04-26T12:52:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3696.738462 secs 2024-04-26T12:52:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:38 smithi096 ceph-mon[32753]: pgmap v2635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:52:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3699.470015 secs 2024-04-26T12:52:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:52:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:52:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:52:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:52:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:52:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:52:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:52:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:52:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:52:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:40 smithi194 ceph-mon[33017]: pgmap v2636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:40 smithi194 ceph-mon[33017]: pgmap v2637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:52:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:40 smithi033 ceph-mon[27845]: pgmap v2636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:40 smithi033 ceph-mon[27845]: pgmap v2637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:52:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:40 smithi096 ceph-mon[32753]: pgmap v2636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:41.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:40 smithi096 ceph-mon[32753]: pgmap v2637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:52:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:42 smithi194 ceph-mon[33017]: pgmap v2638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:42 smithi033 ceph-mon[27845]: pgmap v2638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:42 smithi096 ceph-mon[32753]: pgmap v2638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3701.738525 secs 2024-04-26T12:52:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3704.470167 secs 2024-04-26T12:52:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3701.738525 secs 2024-04-26T12:52:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3704.470167 secs 2024-04-26T12:52:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3701.738525 secs 2024-04-26T12:52:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3704.470167 secs 2024-04-26T12:52:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:44 smithi194 ceph-mon[33017]: pgmap v2639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:44 smithi033 ceph-mon[27845]: pgmap v2639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:44 smithi096 ceph-mon[32753]: pgmap v2639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:46 smithi194 ceph-mon[33017]: pgmap v2640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:46 smithi033 ceph-mon[27845]: pgmap v2640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:46 smithi096 ceph-mon[32753]: pgmap v2640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:48 smithi194 ceph-mon[33017]: pgmap v2641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3706.738610 secs 2024-04-26T12:52:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3709.470281 secs 2024-04-26T12:52:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:48 smithi033 ceph-mon[27845]: pgmap v2641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3706.738610 secs 2024-04-26T12:52:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3709.470281 secs 2024-04-26T12:52:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:48 smithi096 ceph-mon[32753]: pgmap v2641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:52:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3706.738610 secs 2024-04-26T12:52:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3709.470281 secs 2024-04-26T12:52:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:50.338+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:50.338+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:52:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:52:50.338+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:52:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:50 smithi194 ceph-mon[33017]: pgmap v2642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:52:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:50 smithi033 ceph-mon[27845]: pgmap v2642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:52:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:50 smithi096 ceph-mon[32753]: pgmap v2642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:52:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:52 smithi194 ceph-mon[33017]: pgmap v2643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:52 smithi033 ceph-mon[27845]: pgmap v2643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:52 smithi096 ceph-mon[32753]: pgmap v2643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3711.738694 secs 2024-04-26T12:52:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3714.470346 secs 2024-04-26T12:52:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3711.738694 secs 2024-04-26T12:52:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3714.470346 secs 2024-04-26T12:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3711.738694 secs 2024-04-26T12:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3714.470346 secs 2024-04-26T12:52:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:54 smithi194 ceph-mon[33017]: pgmap v2644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:54 smithi033 ceph-mon[27845]: pgmap v2644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:54 smithi096 ceph-mon[32753]: pgmap v2644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:56 smithi194 ceph-mon[33017]: pgmap v2645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:56 smithi033 ceph-mon[27845]: pgmap v2645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:56 smithi096 ceph-mon[32753]: pgmap v2645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:52:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:58 smithi194 ceph-mon[33017]: pgmap v2646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3716.738772 secs 2024-04-26T12:52:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:52:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3719.470468 secs 2024-04-26T12:52:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:58 smithi033 ceph-mon[27845]: pgmap v2646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3716.738772 secs 2024-04-26T12:52:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:52:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3719.470468 secs 2024-04-26T12:52:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:58 smithi096 ceph-mon[32753]: pgmap v2646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:52:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3716.738772 secs 2024-04-26T12:52:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:52:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3719.470468 secs 2024-04-26T12:53:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:00 smithi194 ceph-mon[33017]: pgmap v2647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:00 smithi033 ceph-mon[27845]: pgmap v2647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:00 smithi096 ceph-mon[32753]: pgmap v2647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:02 smithi194 ceph-mon[33017]: pgmap v2648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:02 smithi033 ceph-mon[27845]: pgmap v2648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:02 smithi096 ceph-mon[32753]: pgmap v2648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:04.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3721.738858 secs 2024-04-26T12:53:04.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3724.470585 secs 2024-04-26T12:53:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3721.738858 secs 2024-04-26T12:53:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3724.470585 secs 2024-04-26T12:53:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3721.738858 secs 2024-04-26T12:53:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3724.470585 secs 2024-04-26T12:53:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:04 smithi194 ceph-mon[33017]: pgmap v2649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:04 smithi033 ceph-mon[27845]: pgmap v2649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:04 smithi096 ceph-mon[32753]: pgmap v2649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:05.350+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:05.350+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:53:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:05.350+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:06 smithi194 ceph-mon[33017]: pgmap v2650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:06 smithi033 ceph-mon[27845]: pgmap v2650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:06 smithi096 ceph-mon[32753]: pgmap v2650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:08 smithi194 ceph-mon[33017]: pgmap v2651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3726.738941 secs 2024-04-26T12:53:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3729.470676 secs 2024-04-26T12:53:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:08 smithi033 ceph-mon[27845]: pgmap v2651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3726.738941 secs 2024-04-26T12:53:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3729.470676 secs 2024-04-26T12:53:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:08 smithi096 ceph-mon[32753]: pgmap v2651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3726.738941 secs 2024-04-26T12:53:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3729.470676 secs 2024-04-26T12:53:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:10 smithi194 ceph-mon[33017]: pgmap v2652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:10 smithi033 ceph-mon[27845]: pgmap v2652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:10 smithi096 ceph-mon[32753]: pgmap v2652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:12 smithi194 ceph-mon[33017]: pgmap v2653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:13.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:12 smithi033 ceph-mon[27845]: pgmap v2653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:12 smithi096 ceph-mon[32753]: pgmap v2653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3731.739002 secs 2024-04-26T12:53:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3734.470791 secs 2024-04-26T12:53:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:13 smithi194 ceph-mon[33017]: pgmap v2654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3731.739002 secs 2024-04-26T12:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3734.470791 secs 2024-04-26T12:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:13 smithi033 ceph-mon[27845]: pgmap v2654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3731.739002 secs 2024-04-26T12:53:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3734.470791 secs 2024-04-26T12:53:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:13 smithi096 ceph-mon[32753]: pgmap v2654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:16 smithi194 ceph-mon[33017]: pgmap v2655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:16 smithi033 ceph-mon[27845]: pgmap v2655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:16 smithi096 ceph-mon[32753]: pgmap v2655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:18 smithi194 ceph-mon[33017]: pgmap v2656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3736.739092 secs 2024-04-26T12:53:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:18 smithi033 ceph-mon[27845]: pgmap v2656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3736.739092 secs 2024-04-26T12:53:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:18 smithi096 ceph-mon[32753]: pgmap v2656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3736.739092 secs 2024-04-26T12:53:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3739.470909 secs 2024-04-26T12:53:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3739.470909 secs 2024-04-26T12:53:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3739.470909 secs 2024-04-26T12:53:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:20.358+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:20.358+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:20.358+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:53:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:20 smithi194 ceph-mon[33017]: pgmap v2657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:20 smithi033 ceph-mon[27845]: pgmap v2657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:20 smithi096 ceph-mon[32753]: pgmap v2657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:22.427+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:53:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:22.427+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:53:22.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:22.427+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:53:22.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:22.427+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:53:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:22 smithi194 ceph-mon[33017]: pgmap v2658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:22 smithi033 ceph-mon[27845]: pgmap v2658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:22 smithi096 ceph-mon[32753]: pgmap v2658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3741.739154 secs 2024-04-26T12:53:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3741.739154 secs 2024-04-26T12:53:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3741.739154 secs 2024-04-26T12:53:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3744.471044 secs 2024-04-26T12:53:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:24 smithi194 ceph-mon[33017]: pgmap v2659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3744.471044 secs 2024-04-26T12:53:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:24 smithi033 ceph-mon[27845]: pgmap v2659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3744.471044 secs 2024-04-26T12:53:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:24 smithi096 ceph-mon[32753]: pgmap v2659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:26 smithi194 ceph-mon[33017]: pgmap v2660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:26 smithi033 ceph-mon[27845]: pgmap v2660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:26 smithi096 ceph-mon[32753]: pgmap v2660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:28 smithi194 ceph-mon[33017]: pgmap v2661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3746.739240 secs 2024-04-26T12:53:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:28 smithi033 ceph-mon[27845]: pgmap v2661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3746.739240 secs 2024-04-26T12:53:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:28 smithi096 ceph-mon[32753]: pgmap v2661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3746.739240 secs 2024-04-26T12:53:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3749.471203 secs 2024-04-26T12:53:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3749.471203 secs 2024-04-26T12:53:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3749.471203 secs 2024-04-26T12:53:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:30 smithi194 ceph-mon[33017]: pgmap v2662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:30 smithi033 ceph-mon[27845]: pgmap v2662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:30 smithi096 ceph-mon[32753]: pgmap v2662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:32 smithi194 ceph-mon[33017]: pgmap v2663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:32 smithi033 ceph-mon[27845]: pgmap v2663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:32 smithi096 ceph-mon[32753]: pgmap v2663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3751.739302 secs 2024-04-26T12:53:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3751.739302 secs 2024-04-26T12:53:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3751.739302 secs 2024-04-26T12:53:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3754.471315 secs 2024-04-26T12:53:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:34 smithi033 ceph-mon[27845]: pgmap v2664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3754.471315 secs 2024-04-26T12:53:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:34 smithi096 ceph-mon[32753]: pgmap v2664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3754.471315 secs 2024-04-26T12:53:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:34 smithi194 ceph-mon[33017]: pgmap v2664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:53:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:35.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:35.371+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:35.371+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:53:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:36 smithi033 ceph-mon[27845]: pgmap v2665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:36 smithi096 ceph-mon[32753]: pgmap v2665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:36 smithi194 ceph-mon[33017]: pgmap v2665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:38 smithi033 ceph-mon[27845]: pgmap v2666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3756.739365 secs 2024-04-26T12:53:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:38 smithi096 ceph-mon[32753]: pgmap v2666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:38.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3756.739365 secs 2024-04-26T12:53:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:38 smithi194 ceph-mon[33017]: pgmap v2666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3756.739365 secs 2024-04-26T12:53:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3759.471383 secs 2024-04-26T12:53:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3759.471383 secs 2024-04-26T12:53:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3759.471383 secs 2024-04-26T12:53:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:40 smithi033 ceph-mon[27845]: pgmap v2667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:53:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:53:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:40 smithi096 ceph-mon[32753]: pgmap v2667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:53:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:53:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:40 smithi194 ceph-mon[33017]: pgmap v2667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:53:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:53:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:42 smithi033 ceph-mon[27845]: pgmap v2668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:42.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:42 smithi096 ceph-mon[32753]: pgmap v2668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:42.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:42 smithi194 ceph-mon[33017]: pgmap v2668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3761.739433 secs 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3761.739433 secs 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3761.739433 secs 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:43.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:53:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:53:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3764.471476 secs 2024-04-26T12:53:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:53:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:53:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: pgmap v2669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:53:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:53:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3764.471476 secs 2024-04-26T12:53:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:53:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:53:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: pgmap v2669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3764.471476 secs 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: pgmap v2669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: pgmap v2670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: pgmap v2671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: pgmap v2672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: pgmap v2670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: pgmap v2671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: pgmap v2672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: pgmap v2670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:53:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: pgmap v2671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: pgmap v2672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:46.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:53:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3766.739510 secs 2024-04-26T12:53:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:48 smithi033 ceph-mon[27845]: pgmap v2673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T12:53:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3766.739510 secs 2024-04-26T12:53:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:48 smithi096 ceph-mon[32753]: pgmap v2673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T12:53:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3766.739510 secs 2024-04-26T12:53:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:48 smithi194 ceph-mon[33017]: pgmap v2673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T12:53:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3769.471555 secs 2024-04-26T12:53:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3769.471555 secs 2024-04-26T12:53:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3769.471555 secs 2024-04-26T12:53:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:50.376+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:50.376+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:53:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:53:50.376+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:53:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:50 smithi033 ceph-mon[27845]: pgmap v2674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:50 smithi096 ceph-mon[32753]: pgmap v2674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:50 smithi194 ceph-mon[33017]: pgmap v2674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:52 smithi033 ceph-mon[27845]: pgmap v2675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:52 smithi096 ceph-mon[32753]: pgmap v2675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:52 smithi194 ceph-mon[33017]: pgmap v2675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:53:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3771.739593 secs 2024-04-26T12:53:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3771.739593 secs 2024-04-26T12:53:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3771.739593 secs 2024-04-26T12:53:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3774.471671 secs 2024-04-26T12:53:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:54 smithi033 ceph-mon[27845]: pgmap v2676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:53:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3774.471671 secs 2024-04-26T12:53:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:54 smithi096 ceph-mon[32753]: pgmap v2676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:53:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3774.471671 secs 2024-04-26T12:53:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:54 smithi194 ceph-mon[33017]: pgmap v2676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T12:53:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:56 smithi033 ceph-mon[27845]: pgmap v2677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:53:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:56 smithi096 ceph-mon[32753]: pgmap v2677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:53:57.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:56 smithi194 ceph-mon[33017]: pgmap v2677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:53:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3776.739677 secs 2024-04-26T12:53:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:58 smithi033 ceph-mon[27845]: pgmap v2678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3776.739677 secs 2024-04-26T12:53:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:58 smithi096 ceph-mon[32753]: pgmap v2678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3776.739677 secs 2024-04-26T12:53:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:58 smithi194 ceph-mon[33017]: pgmap v2678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:53:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3779.471783 secs 2024-04-26T12:53:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3779.471783 secs 2024-04-26T12:54:00.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3779.471783 secs 2024-04-26T12:54:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:00 smithi033 ceph-mon[27845]: pgmap v2679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:00 smithi096 ceph-mon[32753]: pgmap v2679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:00 smithi194 ceph-mon[33017]: pgmap v2679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:02 smithi096 ceph-mon[32753]: pgmap v2680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:02 smithi194 ceph-mon[33017]: pgmap v2680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:02 smithi033 ceph-mon[27845]: pgmap v2680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3781.739762 secs 2024-04-26T12:54:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3784.471923 secs 2024-04-26T12:54:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3781.739762 secs 2024-04-26T12:54:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3784.471923 secs 2024-04-26T12:54:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3781.739762 secs 2024-04-26T12:54:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3784.471923 secs 2024-04-26T12:54:05.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:04 smithi096 ceph-mon[32753]: pgmap v2681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:04 smithi194 ceph-mon[33017]: pgmap v2681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:04 smithi033 ceph-mon[27845]: pgmap v2681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:05.384+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:05.385 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:05.384+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:05.385 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:05.384+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:54:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:06 smithi096 ceph-mon[32753]: pgmap v2682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:06 smithi194 ceph-mon[33017]: pgmap v2682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:06 smithi033 ceph-mon[27845]: pgmap v2682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3786.739850 secs 2024-04-26T12:54:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:08 smithi194 ceph-mon[33017]: pgmap v2683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3789.472057 secs 2024-04-26T12:54:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3786.739850 secs 2024-04-26T12:54:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:08 smithi033 ceph-mon[27845]: pgmap v2683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3789.472057 secs 2024-04-26T12:54:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3786.739850 secs 2024-04-26T12:54:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:08 smithi096 ceph-mon[32753]: pgmap v2683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3789.472057 secs 2024-04-26T12:54:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:10 smithi194 ceph-mon[33017]: pgmap v2684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:10 smithi033 ceph-mon[27845]: pgmap v2684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:10 smithi096 ceph-mon[32753]: pgmap v2684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:12 smithi194 ceph-mon[33017]: pgmap v2685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:12 smithi033 ceph-mon[27845]: pgmap v2685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:12 smithi096 ceph-mon[32753]: pgmap v2685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3791.739933 secs 2024-04-26T12:54:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3794.472174 secs 2024-04-26T12:54:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3791.739933 secs 2024-04-26T12:54:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3794.472174 secs 2024-04-26T12:54:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3791.739933 secs 2024-04-26T12:54:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3794.472174 secs 2024-04-26T12:54:15.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:14 smithi033 ceph-mon[27845]: pgmap v2686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:14 smithi194 ceph-mon[33017]: pgmap v2686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:14 smithi096 ceph-mon[32753]: pgmap v2686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:16 smithi194 ceph-mon[33017]: pgmap v2687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:16 smithi033 ceph-mon[27845]: pgmap v2687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:16 smithi096 ceph-mon[32753]: pgmap v2687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3796.740018 secs 2024-04-26T12:54:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:18 smithi194 ceph-mon[33017]: pgmap v2688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3799.472271 secs 2024-04-26T12:54:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3796.740018 secs 2024-04-26T12:54:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:18 smithi033 ceph-mon[27845]: pgmap v2688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3799.472271 secs 2024-04-26T12:54:19.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3796.740018 secs 2024-04-26T12:54:19.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:18 smithi096 ceph-mon[32753]: pgmap v2688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:19.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3799.472271 secs 2024-04-26T12:54:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:20.396+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:20.396+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:20.396+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:54:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:20 smithi194 ceph-mon[33017]: pgmap v2689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:20 smithi033 ceph-mon[27845]: pgmap v2689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:20 smithi096 ceph-mon[32753]: pgmap v2689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:22 smithi194 ceph-mon[33017]: pgmap v2690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:22 smithi033 ceph-mon[27845]: pgmap v2690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:22 smithi096 ceph-mon[32753]: pgmap v2690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3801.740080 secs 2024-04-26T12:54:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3804.472391 secs 2024-04-26T12:54:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3801.740080 secs 2024-04-26T12:54:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3804.472391 secs 2024-04-26T12:54:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3801.740080 secs 2024-04-26T12:54:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3804.472391 secs 2024-04-26T12:54:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:24 smithi194 ceph-mon[33017]: pgmap v2691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:24 smithi033 ceph-mon[27845]: pgmap v2691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:24 smithi096 ceph-mon[32753]: pgmap v2691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:26 smithi194 ceph-mon[33017]: pgmap v2692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:26 smithi033 ceph-mon[27845]: pgmap v2692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:26 smithi096 ceph-mon[32753]: pgmap v2692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3806.740143 secs 2024-04-26T12:54:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:28 smithi194 ceph-mon[33017]: pgmap v2693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3809.472511 secs 2024-04-26T12:54:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3806.740143 secs 2024-04-26T12:54:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:28 smithi033 ceph-mon[27845]: pgmap v2693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3809.472511 secs 2024-04-26T12:54:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3806.740143 secs 2024-04-26T12:54:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:28 smithi096 ceph-mon[32753]: pgmap v2693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3809.472511 secs 2024-04-26T12:54:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:30 smithi194 ceph-mon[33017]: pgmap v2694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:30 smithi033 ceph-mon[27845]: pgmap v2694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:30 smithi096 ceph-mon[32753]: pgmap v2694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:32 smithi194 ceph-mon[33017]: pgmap v2695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:32 smithi033 ceph-mon[27845]: pgmap v2695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:32 smithi096 ceph-mon[32753]: pgmap v2695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3811.740201 secs 2024-04-26T12:54:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3814.472592 secs 2024-04-26T12:54:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3811.740201 secs 2024-04-26T12:54:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3814.472592 secs 2024-04-26T12:54:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3811.740201 secs 2024-04-26T12:54:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3814.472592 secs 2024-04-26T12:54:35.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:34 smithi096 ceph-mon[32753]: pgmap v2696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:34 smithi194 ceph-mon[33017]: pgmap v2696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:34 smithi033 ceph-mon[27845]: pgmap v2696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:35.404+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:35.404+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:35.404+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:54:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:36 smithi194 ceph-mon[33017]: pgmap v2697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:36 smithi033 ceph-mon[27845]: pgmap v2697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:36 smithi096 ceph-mon[32753]: pgmap v2697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3816.740262 secs 2024-04-26T12:54:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:38 smithi194 ceph-mon[33017]: pgmap v2698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3819.472691 secs 2024-04-26T12:54:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3816.740262 secs 2024-04-26T12:54:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:38 smithi033 ceph-mon[27845]: pgmap v2698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3819.472691 secs 2024-04-26T12:54:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3816.740262 secs 2024-04-26T12:54:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:38 smithi096 ceph-mon[32753]: pgmap v2698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3819.472691 secs 2024-04-26T12:54:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:40 smithi194 ceph-mon[33017]: pgmap v2699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:40 smithi033 ceph-mon[27845]: pgmap v2699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:40 smithi096 ceph-mon[32753]: pgmap v2699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:43.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:42 smithi194 ceph-mon[33017]: pgmap v2700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:42 smithi033 ceph-mon[27845]: pgmap v2700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:42 smithi096 ceph-mon[32753]: pgmap v2700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3821.740323 secs 2024-04-26T12:54:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3824.472811 secs 2024-04-26T12:54:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3821.740323 secs 2024-04-26T12:54:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3824.472811 secs 2024-04-26T12:54:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3821.740323 secs 2024-04-26T12:54:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3824.472811 secs 2024-04-26T12:54:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:44 smithi194 ceph-mon[33017]: pgmap v2701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:44 smithi033 ceph-mon[27845]: pgmap v2701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:44 smithi096 ceph-mon[32753]: pgmap v2701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:46 smithi194 ceph-mon[33017]: pgmap v2702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:54:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:46 smithi033 ceph-mon[27845]: pgmap v2702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:54:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:46 smithi096 ceph-mon[32753]: pgmap v2702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:54:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:54:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:54:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:54:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:54:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:54:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:54:48.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:54:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:54:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:54:48.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:54:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:48 smithi194 ceph-mon[33017]: pgmap v2703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:48 smithi194 ceph-mon[33017]: pgmap v2704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3826.740388 secs 2024-04-26T12:54:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3829.472910 secs 2024-04-26T12:54:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:48 smithi033 ceph-mon[27845]: pgmap v2703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:48 smithi033 ceph-mon[27845]: pgmap v2704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3826.740388 secs 2024-04-26T12:54:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3829.472910 secs 2024-04-26T12:54:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:48 smithi096 ceph-mon[32753]: pgmap v2703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:54:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:48 smithi096 ceph-mon[32753]: pgmap v2704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3826.740388 secs 2024-04-26T12:54:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3829.472910 secs 2024-04-26T12:54:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:50.420+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:50.420+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:54:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:54:50.420+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:54:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:50 smithi194 ceph-mon[33017]: pgmap v2705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:54:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:50 smithi033 ceph-mon[27845]: pgmap v2705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:54:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:50 smithi096 ceph-mon[32753]: pgmap v2705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T12:54:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:52 smithi033 ceph-mon[27845]: pgmap v2706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:52 smithi096 ceph-mon[32753]: pgmap v2706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:52 smithi194 ceph-mon[33017]: pgmap v2706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3831.740449 secs 2024-04-26T12:54:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3834.473031 secs 2024-04-26T12:54:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:53 smithi033 ceph-mon[27845]: pgmap v2707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3831.740449 secs 2024-04-26T12:54:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3834.473031 secs 2024-04-26T12:54:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:53 smithi096 ceph-mon[32753]: pgmap v2707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3831.740449 secs 2024-04-26T12:54:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3834.473031 secs 2024-04-26T12:54:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:53 smithi194 ceph-mon[33017]: pgmap v2707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:56 smithi194 ceph-mon[33017]: pgmap v2708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:56 smithi033 ceph-mon[27845]: pgmap v2708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:56 smithi096 ceph-mon[32753]: pgmap v2708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:54:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:58 smithi194 ceph-mon[33017]: pgmap v2709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:54:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3836.740532 secs 2024-04-26T12:54:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:58 smithi033 ceph-mon[27845]: pgmap v2709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:54:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3836.740532 secs 2024-04-26T12:54:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:58 smithi096 ceph-mon[32753]: pgmap v2709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:54:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3836.740532 secs 2024-04-26T12:54:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3839.473149 secs 2024-04-26T12:54:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3839.473149 secs 2024-04-26T12:54:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3839.473149 secs 2024-04-26T12:55:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:00 smithi033 ceph-mon[27845]: pgmap v2710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:00 smithi096 ceph-mon[32753]: pgmap v2710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:00 smithi194 ceph-mon[33017]: pgmap v2710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:02.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:02 smithi033 ceph-mon[27845]: pgmap v2711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:02 smithi096 ceph-mon[32753]: pgmap v2711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:02 smithi194 ceph-mon[33017]: pgmap v2711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3841.740620 secs 2024-04-26T12:55:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3841.740620 secs 2024-04-26T12:55:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3841.740620 secs 2024-04-26T12:55:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3844.473274 secs 2024-04-26T12:55:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:04 smithi033 ceph-mon[27845]: pgmap v2712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3844.473274 secs 2024-04-26T12:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:04 smithi096 ceph-mon[32753]: pgmap v2712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3844.473274 secs 2024-04-26T12:55:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:04 smithi194 ceph-mon[33017]: pgmap v2712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:05.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:05.433+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:05.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:05.433+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:05.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:05.433+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:55:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:06 smithi033 ceph-mon[27845]: pgmap v2713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:06 smithi096 ceph-mon[32753]: pgmap v2713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:06 smithi194 ceph-mon[33017]: pgmap v2713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:08 smithi033 ceph-mon[27845]: pgmap v2714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3846.740704 secs 2024-04-26T12:55:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:08 smithi096 ceph-mon[32753]: pgmap v2714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3846.740704 secs 2024-04-26T12:55:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:08 smithi194 ceph-mon[33017]: pgmap v2714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3846.740704 secs 2024-04-26T12:55:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3849.473392 secs 2024-04-26T12:55:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3849.473392 secs 2024-04-26T12:55:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3849.473392 secs 2024-04-26T12:55:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:10 smithi033 ceph-mon[27845]: pgmap v2715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:10 smithi096 ceph-mon[32753]: pgmap v2715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:10 smithi194 ceph-mon[33017]: pgmap v2715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:12 smithi033 ceph-mon[27845]: pgmap v2716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:12 smithi096 ceph-mon[32753]: pgmap v2716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:12 smithi194 ceph-mon[33017]: pgmap v2716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3851.740791 secs 2024-04-26T12:55:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3851.740791 secs 2024-04-26T12:55:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3851.740791 secs 2024-04-26T12:55:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3854.473511 secs 2024-04-26T12:55:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:14 smithi096 ceph-mon[32753]: pgmap v2717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3854.473511 secs 2024-04-26T12:55:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:14 smithi194 ceph-mon[33017]: pgmap v2717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3854.473511 secs 2024-04-26T12:55:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:14 smithi033 ceph-mon[27845]: pgmap v2717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:16 smithi096 ceph-mon[32753]: pgmap v2718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:16 smithi194 ceph-mon[33017]: pgmap v2718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:16 smithi033 ceph-mon[27845]: pgmap v2718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:18 smithi194 ceph-mon[33017]: pgmap v2719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3856.740878 secs 2024-04-26T12:55:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:18 smithi033 ceph-mon[27845]: pgmap v2719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3856.740878 secs 2024-04-26T12:55:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:18 smithi096 ceph-mon[32753]: pgmap v2719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3856.740878 secs 2024-04-26T12:55:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3859.473655 secs 2024-04-26T12:55:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3859.473655 secs 2024-04-26T12:55:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3859.473655 secs 2024-04-26T12:55:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:20.449+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:20.449+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:20.449+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:55:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:20 smithi194 ceph-mon[33017]: pgmap v2720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:20 smithi033 ceph-mon[27845]: pgmap v2720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:20 smithi096 ceph-mon[32753]: pgmap v2720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:22 smithi194 ceph-mon[33017]: pgmap v2721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:22 smithi033 ceph-mon[27845]: pgmap v2721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:22 smithi096 ceph-mon[32753]: pgmap v2721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3861.740967 secs 2024-04-26T12:55:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3861.740967 secs 2024-04-26T12:55:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3861.740967 secs 2024-04-26T12:55:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3864.473769 secs 2024-04-26T12:55:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:24 smithi194 ceph-mon[33017]: pgmap v2722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3864.473769 secs 2024-04-26T12:55:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:24 smithi033 ceph-mon[27845]: pgmap v2722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3864.473769 secs 2024-04-26T12:55:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:24 smithi096 ceph-mon[32753]: pgmap v2722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:26 smithi194 ceph-mon[33017]: pgmap v2723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:26 smithi033 ceph-mon[27845]: pgmap v2723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:26 smithi096 ceph-mon[32753]: pgmap v2723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:28 smithi194 ceph-mon[33017]: pgmap v2724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3866.741048 secs 2024-04-26T12:55:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:28 smithi033 ceph-mon[27845]: pgmap v2724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3866.741048 secs 2024-04-26T12:55:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:28 smithi096 ceph-mon[32753]: pgmap v2724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3866.741048 secs 2024-04-26T12:55:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3869.473868 secs 2024-04-26T12:55:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3869.473868 secs 2024-04-26T12:55:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3869.473868 secs 2024-04-26T12:55:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:30 smithi194 ceph-mon[33017]: pgmap v2725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:30 smithi033 ceph-mon[27845]: pgmap v2725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:30 smithi096 ceph-mon[32753]: pgmap v2725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:55:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:32 smithi194 ceph-mon[33017]: pgmap v2726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:32.913 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:32 smithi033 ceph-mon[27845]: pgmap v2726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:32 smithi096 ceph-mon[32753]: pgmap v2726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3871.741109 secs 2024-04-26T12:55:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3871.741109 secs 2024-04-26T12:55:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3871.741109 secs 2024-04-26T12:55:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3874.473994 secs 2024-04-26T12:55:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:34 smithi194 ceph-mon[33017]: pgmap v2727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3874.473994 secs 2024-04-26T12:55:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:34 smithi033 ceph-mon[27845]: pgmap v2727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3874.473994 secs 2024-04-26T12:55:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:34 smithi096 ceph-mon[32753]: pgmap v2727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:35.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:35.464+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:35.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:35.464+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:35.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:35.464+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:55:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:36 smithi194 ceph-mon[33017]: pgmap v2728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:36 smithi033 ceph-mon[27845]: pgmap v2728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:36 smithi096 ceph-mon[32753]: pgmap v2728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:37.447+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:55:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:37.447+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:55:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:37.448+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:55:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:37.448+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:55:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:38 smithi194 ceph-mon[33017]: pgmap v2729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3876.741170 secs 2024-04-26T12:55:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:38 smithi033 ceph-mon[27845]: pgmap v2729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3876.741170 secs 2024-04-26T12:55:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:38 smithi096 ceph-mon[32753]: pgmap v2729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3876.741170 secs 2024-04-26T12:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3879.474097 secs 2024-04-26T12:55:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3879.474097 secs 2024-04-26T12:55:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3879.474097 secs 2024-04-26T12:55:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:40 smithi194 ceph-mon[33017]: pgmap v2730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:40 smithi033 ceph-mon[27845]: pgmap v2730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:40 smithi096 ceph-mon[32753]: pgmap v2730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:42 smithi194 ceph-mon[33017]: pgmap v2731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:42 smithi033 ceph-mon[27845]: pgmap v2731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:42 smithi096 ceph-mon[32753]: pgmap v2731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3881.741231 secs 2024-04-26T12:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3881.741231 secs 2024-04-26T12:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3881.741231 secs 2024-04-26T12:55:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3884.474203 secs 2024-04-26T12:55:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:44 smithi194 ceph-mon[33017]: pgmap v2732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3884.474203 secs 2024-04-26T12:55:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:44 smithi033 ceph-mon[27845]: pgmap v2732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3884.474203 secs 2024-04-26T12:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:44 smithi096 ceph-mon[32753]: pgmap v2732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:46 smithi194 ceph-mon[33017]: pgmap v2733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:46 smithi033 ceph-mon[27845]: pgmap v2733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:46 smithi096 ceph-mon[32753]: pgmap v2733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:48 smithi194 ceph-mon[33017]: pgmap v2734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3886.741319 secs 2024-04-26T12:55:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:55:48.915 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:48 smithi096 ceph-mon[32753]: pgmap v2734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:48.916 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3886.741319 secs 2024-04-26T12:55:48.916 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:55:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:48 smithi033 ceph-mon[27845]: pgmap v2734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3886.741319 secs 2024-04-26T12:55:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:55:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3889.474296 secs 2024-04-26T12:55:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3889.474296 secs 2024-04-26T12:55:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3889.474296 secs 2024-04-26T12:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:50.478+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:50.478+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:55:50.478+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:55:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:50 smithi194 ceph-mon[33017]: pgmap v2735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:50 smithi033 ceph-mon[27845]: pgmap v2735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:50 smithi096 ceph-mon[32753]: pgmap v2735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:55:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:52 smithi096 ceph-mon[32753]: pgmap v2736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T12:55:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:52 smithi194 ceph-mon[33017]: pgmap v2736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T12:55:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:52 smithi033 ceph-mon[27845]: pgmap v2736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T12:55:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:53.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3891.741377 secs 2024-04-26T12:55:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3891.741377 secs 2024-04-26T12:55:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3891.741377 secs 2024-04-26T12:55:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3894.474447 secs 2024-04-26T12:55:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:54 smithi194 ceph-mon[33017]: pgmap v2737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3894.474447 secs 2024-04-26T12:55:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:54 smithi033 ceph-mon[27845]: pgmap v2737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3894.474447 secs 2024-04-26T12:55:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:54 smithi096 ceph-mon[32753]: pgmap v2737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:55:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:55:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:55:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:55 smithi194 ceph-mon[33017]: pgmap v2738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:55:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:55 smithi194 ceph-mon[33017]: pgmap v2739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:55:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:55:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:55 smithi033 ceph-mon[27845]: pgmap v2738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:55:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:55 smithi033 ceph-mon[27845]: pgmap v2739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:55:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:55:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:55 smithi096 ceph-mon[32753]: pgmap v2738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:55:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:55 smithi096 ceph-mon[32753]: pgmap v2739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:56.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:55:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:57 smithi194 ceph-mon[33017]: pgmap v2740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:57 smithi033 ceph-mon[27845]: pgmap v2740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:57 smithi096 ceph-mon[32753]: pgmap v2740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:55:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3896.741436 secs 2024-04-26T12:55:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3899.474564 secs 2024-04-26T12:55:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3896.741436 secs 2024-04-26T12:55:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3899.474564 secs 2024-04-26T12:55:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3896.741436 secs 2024-04-26T12:55:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3899.474564 secs 2024-04-26T12:56:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:55:59 smithi033 ceph-mon[27845]: pgmap v2741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T12:56:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:55:59 smithi194 ceph-mon[33017]: pgmap v2741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T12:56:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:55:59 smithi096 ceph-mon[32753]: pgmap v2741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T12:56:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:01 smithi194 ceph-mon[33017]: pgmap v2742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:01 smithi033 ceph-mon[27845]: pgmap v2742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:01 smithi096 ceph-mon[32753]: pgmap v2742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3901.741498 secs 2024-04-26T12:56:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:03 smithi194 ceph-mon[33017]: pgmap v2743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3904.474681 secs 2024-04-26T12:56:04.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3901.741498 secs 2024-04-26T12:56:04.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:03 smithi033 ceph-mon[27845]: pgmap v2743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:04.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3904.474681 secs 2024-04-26T12:56:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3901.741498 secs 2024-04-26T12:56:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:03 smithi096 ceph-mon[32753]: pgmap v2743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3904.474681 secs 2024-04-26T12:56:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:05.489+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:05.489+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:05.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:05.489+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:56:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:05 smithi194 ceph-mon[33017]: pgmap v2744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:56:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:05 smithi033 ceph-mon[27845]: pgmap v2744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:56:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:05 smithi096 ceph-mon[32753]: pgmap v2744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:56:08.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:07 smithi194 ceph-mon[33017]: pgmap v2745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:07 smithi033 ceph-mon[27845]: pgmap v2745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:07 smithi096 ceph-mon[32753]: pgmap v2745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3906.741558 secs 2024-04-26T12:56:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3909.474816 secs 2024-04-26T12:56:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3906.741558 secs 2024-04-26T12:56:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3909.474816 secs 2024-04-26T12:56:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3906.741558 secs 2024-04-26T12:56:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3909.474816 secs 2024-04-26T12:56:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:09 smithi194 ceph-mon[33017]: pgmap v2746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:09 smithi033 ceph-mon[27845]: pgmap v2746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:09 smithi096 ceph-mon[32753]: pgmap v2746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:11 smithi194 ceph-mon[33017]: pgmap v2747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:11 smithi033 ceph-mon[27845]: pgmap v2747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:11 smithi096 ceph-mon[32753]: pgmap v2747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3911.741643 secs 2024-04-26T12:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:13 smithi194 ceph-mon[33017]: pgmap v2748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3914.474935 secs 2024-04-26T12:56:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3911.741643 secs 2024-04-26T12:56:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:13 smithi033 ceph-mon[27845]: pgmap v2748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3914.474935 secs 2024-04-26T12:56:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3911.741643 secs 2024-04-26T12:56:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:13 smithi096 ceph-mon[32753]: pgmap v2748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3914.474935 secs 2024-04-26T12:56:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:15 smithi194 ceph-mon[33017]: pgmap v2749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:15 smithi033 ceph-mon[27845]: pgmap v2749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:15 smithi096 ceph-mon[32753]: pgmap v2749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:17 smithi194 ceph-mon[33017]: pgmap v2750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:17 smithi033 ceph-mon[27845]: pgmap v2750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:17 smithi096 ceph-mon[32753]: pgmap v2750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3916.741725 secs 2024-04-26T12:56:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3919.475051 secs 2024-04-26T12:56:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3916.741725 secs 2024-04-26T12:56:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3919.475051 secs 2024-04-26T12:56:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3916.741725 secs 2024-04-26T12:56:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3919.475051 secs 2024-04-26T12:56:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:19 smithi194 ceph-mon[33017]: pgmap v2751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:19 smithi033 ceph-mon[27845]: pgmap v2751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:19 smithi096 ceph-mon[32753]: pgmap v2751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:20.493+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:20.493+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:56:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:20.493+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:21.714+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000704) for more than 900000000 us, fault. 2024-04-26T12:56:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:21.714+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40afc50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feef0024000 tx=0x7feef0038040 comp rx=0 tx=0).stop 2024-04-26T12:56:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:21.714+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 msgr2=0x7feee40afc50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T12:56:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:21.714+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40a3d10 0x7feee40afc50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T12:56:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:21.714+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 0x7feee40b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T12:56:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:21.715+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 0x7feee40b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T12:56:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:21.717+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 0x7feee40b2580 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0038040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T12:56:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:21 smithi194 ceph-mon[33017]: pgmap v2752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:21 smithi033 ceph-mon[27845]: pgmap v2752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:21 smithi096 ceph-mon[32753]: pgmap v2752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:22.456+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:56:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:22.456+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:56:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:22.456+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:56:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:22.456+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:56:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3921.741785 secs 2024-04-26T12:56:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:23 smithi033 ceph-mon[27845]: pgmap v2753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3924.475172 secs 2024-04-26T12:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3921.741785 secs 2024-04-26T12:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:23 smithi096 ceph-mon[32753]: pgmap v2753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3924.475172 secs 2024-04-26T12:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3921.741785 secs 2024-04-26T12:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:23 smithi194 ceph-mon[33017]: pgmap v2753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3924.475172 secs 2024-04-26T12:56:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:25 smithi033 ceph-mon[27845]: pgmap v2754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:25 smithi096 ceph-mon[32753]: pgmap v2754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:25 smithi194 ceph-mon[33017]: pgmap v2754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:27 smithi033 ceph-mon[27845]: pgmap v2755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:27 smithi096 ceph-mon[32753]: pgmap v2755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:27 smithi194 ceph-mon[33017]: pgmap v2755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3926.741845 secs 2024-04-26T12:56:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3929.475295 secs 2024-04-26T12:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3926.741845 secs 2024-04-26T12:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3929.475295 secs 2024-04-26T12:56:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3926.741845 secs 2024-04-26T12:56:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3929.475295 secs 2024-04-26T12:56:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:29 smithi033 ceph-mon[27845]: pgmap v2756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:29 smithi096 ceph-mon[32753]: pgmap v2756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:30.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:29 smithi194 ceph-mon[33017]: pgmap v2756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:30 smithi033 ceph-mon[27845]: pgmap v2757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:30 smithi096 ceph-mon[32753]: pgmap v2757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:30 smithi194 ceph-mon[33017]: pgmap v2757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3931.741929 secs 2024-04-26T12:56:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:33 smithi194 ceph-mon[33017]: pgmap v2758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3934.475373 secs 2024-04-26T12:56:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3931.741929 secs 2024-04-26T12:56:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:33 smithi033 ceph-mon[27845]: pgmap v2758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3934.475373 secs 2024-04-26T12:56:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3931.741929 secs 2024-04-26T12:56:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:33 smithi096 ceph-mon[32753]: pgmap v2758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3934.475373 secs 2024-04-26T12:56:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:35.502+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:35.502+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:56:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:35.502+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:35 smithi194 ceph-mon[33017]: pgmap v2759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:35 smithi033 ceph-mon[27845]: pgmap v2759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:35 smithi096 ceph-mon[32753]: pgmap v2759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:37 smithi194 ceph-mon[33017]: pgmap v2760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:37 smithi033 ceph-mon[27845]: pgmap v2760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:37 smithi096 ceph-mon[32753]: pgmap v2760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3936.742025 secs 2024-04-26T12:56:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3939.475502 secs 2024-04-26T12:56:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3936.742025 secs 2024-04-26T12:56:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3939.475502 secs 2024-04-26T12:56:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3936.742025 secs 2024-04-26T12:56:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3939.475502 secs 2024-04-26T12:56:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:39 smithi194 ceph-mon[33017]: pgmap v2761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:39 smithi033 ceph-mon[27845]: pgmap v2761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:39 smithi096 ceph-mon[32753]: pgmap v2761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:41 smithi194 ceph-mon[33017]: pgmap v2762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:41 smithi033 ceph-mon[27845]: pgmap v2762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:41 smithi096 ceph-mon[32753]: pgmap v2762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3941.742088 secs 2024-04-26T12:56:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:43 smithi194 ceph-mon[33017]: pgmap v2763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3944.475650 secs 2024-04-26T12:56:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3941.742088 secs 2024-04-26T12:56:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:43 smithi033 ceph-mon[27845]: pgmap v2763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3944.475650 secs 2024-04-26T12:56:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3941.742088 secs 2024-04-26T12:56:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:43 smithi096 ceph-mon[32753]: pgmap v2763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3944.475650 secs 2024-04-26T12:56:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:45 smithi194 ceph-mon[33017]: pgmap v2764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:45 smithi033 ceph-mon[27845]: pgmap v2764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:45 smithi096 ceph-mon[32753]: pgmap v2764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:47 smithi194 ceph-mon[33017]: pgmap v2765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:47 smithi033 ceph-mon[27845]: pgmap v2765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:47 smithi096 ceph-mon[32753]: pgmap v2765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3946.742150 secs 2024-04-26T12:56:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3949.475772 secs 2024-04-26T12:56:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3946.742150 secs 2024-04-26T12:56:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3949.475772 secs 2024-04-26T12:56:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3946.742150 secs 2024-04-26T12:56:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3949.475772 secs 2024-04-26T12:56:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:49 smithi194 ceph-mon[33017]: pgmap v2766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:49 smithi033 ceph-mon[27845]: pgmap v2766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:49 smithi096 ceph-mon[32753]: pgmap v2766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:50.509+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:50.509+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:56:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:56:50.509+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:56:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:51 smithi194 ceph-mon[33017]: pgmap v2767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:51 smithi033 ceph-mon[27845]: pgmap v2767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:51 smithi096 ceph-mon[32753]: pgmap v2767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3951.742240 secs 2024-04-26T12:56:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:53 smithi033 ceph-mon[27845]: pgmap v2768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3954.475899 secs 2024-04-26T12:56:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3951.742240 secs 2024-04-26T12:56:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:53 smithi096 ceph-mon[32753]: pgmap v2768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3954.475899 secs 2024-04-26T12:56:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3951.742240 secs 2024-04-26T12:56:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:53 smithi194 ceph-mon[33017]: pgmap v2768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3954.475899 secs 2024-04-26T12:56:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:55 smithi033 ceph-mon[27845]: pgmap v2769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:56:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:55 smithi096 ceph-mon[32753]: pgmap v2769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:56:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:55 smithi194 ceph-mon[33017]: pgmap v2769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:56:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:56:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:56:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:56:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:56:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:56:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:56:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:56:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:56:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:56:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:56:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:57 smithi033 ceph-mon[27845]: pgmap v2770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:57 smithi033 ceph-mon[27845]: pgmap v2771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:57 smithi096 ceph-mon[32753]: pgmap v2770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:57 smithi096 ceph-mon[32753]: pgmap v2771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:57 smithi194 ceph-mon[33017]: pgmap v2770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:56:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:57 smithi194 ceph-mon[33017]: pgmap v2771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:56:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3956.742320 secs 2024-04-26T12:56:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3959.476046 secs 2024-04-26T12:56:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3956.742320 secs 2024-04-26T12:56:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3959.476046 secs 2024-04-26T12:56:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3956.742320 secs 2024-04-26T12:56:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3959.476046 secs 2024-04-26T12:57:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:56:59 smithi033 ceph-mon[27845]: pgmap v2772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:56:59 smithi096 ceph-mon[32753]: pgmap v2772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:00.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:56:59 smithi194 ceph-mon[33017]: pgmap v2772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:00 smithi033 ceph-mon[27845]: pgmap v2773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:00 smithi096 ceph-mon[32753]: pgmap v2773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:00 smithi194 ceph-mon[33017]: pgmap v2773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:03 smithi194 ceph-mon[33017]: pgmap v2774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3961.742381 secs 2024-04-26T12:57:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:03 smithi033 ceph-mon[27845]: pgmap v2774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3961.742381 secs 2024-04-26T12:57:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:03 smithi096 ceph-mon[32753]: pgmap v2774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:57:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3961.742381 secs 2024-04-26T12:57:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3964.476172 secs 2024-04-26T12:57:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3964.476172 secs 2024-04-26T12:57:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3964.476172 secs 2024-04-26T12:57:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:05.524+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:05.524+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:57:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:05.524+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:05 smithi194 ceph-mon[33017]: pgmap v2775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:57:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:05 smithi033 ceph-mon[27845]: pgmap v2775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:57:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:05 smithi096 ceph-mon[32753]: pgmap v2775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T12:57:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:07 smithi194 ceph-mon[33017]: pgmap v2776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:57:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:07 smithi033 ceph-mon[27845]: pgmap v2776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:57:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:07 smithi096 ceph-mon[32753]: pgmap v2776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:57:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3966.742463 secs 2024-04-26T12:57:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3966.742463 secs 2024-04-26T12:57:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3966.742463 secs 2024-04-26T12:57:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3969.476293 secs 2024-04-26T12:57:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:09 smithi194 ceph-mon[33017]: pgmap v2777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3969.476293 secs 2024-04-26T12:57:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:09 smithi033 ceph-mon[27845]: pgmap v2777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3969.476293 secs 2024-04-26T12:57:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:09 smithi096 ceph-mon[32753]: pgmap v2777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:11 smithi194 ceph-mon[33017]: pgmap v2778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:11.902 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:11 smithi033 ceph-mon[27845]: pgmap v2778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:11 smithi096 ceph-mon[32753]: pgmap v2778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:13 smithi194 ceph-mon[33017]: pgmap v2779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3971.742546 secs 2024-04-26T12:57:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:13 smithi033 ceph-mon[27845]: pgmap v2779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3971.742546 secs 2024-04-26T12:57:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:13 smithi096 ceph-mon[32753]: pgmap v2779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3971.742546 secs 2024-04-26T12:57:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3974.476393 secs 2024-04-26T12:57:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3974.476393 secs 2024-04-26T12:57:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3974.476393 secs 2024-04-26T12:57:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:15 smithi194 ceph-mon[33017]: pgmap v2780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:15 smithi033 ceph-mon[27845]: pgmap v2780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:15 smithi096 ceph-mon[32753]: pgmap v2780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:17 smithi194 ceph-mon[33017]: pgmap v2781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:17 smithi033 ceph-mon[27845]: pgmap v2781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:17 smithi096 ceph-mon[32753]: pgmap v2781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3976.742629 secs 2024-04-26T12:57:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3976.742629 secs 2024-04-26T12:57:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3976.742629 secs 2024-04-26T12:57:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3979.476519 secs 2024-04-26T12:57:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:19 smithi194 ceph-mon[33017]: pgmap v2782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3979.476519 secs 2024-04-26T12:57:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:19 smithi033 ceph-mon[27845]: pgmap v2782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3979.476519 secs 2024-04-26T12:57:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:19 smithi096 ceph-mon[32753]: pgmap v2782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:20.540+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:20.540+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:57:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:20.540+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:21 smithi194 ceph-mon[33017]: pgmap v2783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:21 smithi033 ceph-mon[27845]: pgmap v2783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:21 smithi096 ceph-mon[32753]: pgmap v2783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:23 smithi194 ceph-mon[33017]: pgmap v2784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3981.742712 secs 2024-04-26T12:57:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:23 smithi033 ceph-mon[27845]: pgmap v2784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3981.742712 secs 2024-04-26T12:57:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:23 smithi096 ceph-mon[32753]: pgmap v2784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3981.742712 secs 2024-04-26T12:57:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3984.476624 secs 2024-04-26T12:57:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3984.476624 secs 2024-04-26T12:57:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3984.476624 secs 2024-04-26T12:57:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:25 smithi033 ceph-mon[27845]: pgmap v2785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:25 smithi096 ceph-mon[32753]: pgmap v2785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:25 smithi194 ceph-mon[33017]: pgmap v2785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:27 smithi033 ceph-mon[27845]: pgmap v2786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:27 smithi096 ceph-mon[32753]: pgmap v2786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:27 smithi194 ceph-mon[33017]: pgmap v2786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3986.742796 secs 2024-04-26T12:57:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3986.742796 secs 2024-04-26T12:57:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3986.742796 secs 2024-04-26T12:57:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3989.476769 secs 2024-04-26T12:57:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:29 smithi033 ceph-mon[27845]: pgmap v2787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3989.476769 secs 2024-04-26T12:57:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:29 smithi096 ceph-mon[32753]: pgmap v2787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3989.476769 secs 2024-04-26T12:57:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:29 smithi194 ceph-mon[33017]: pgmap v2787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:31 smithi033 ceph-mon[27845]: pgmap v2788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:31 smithi096 ceph-mon[32753]: pgmap v2788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:31 smithi194 ceph-mon[33017]: pgmap v2788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:33 smithi033 ceph-mon[27845]: pgmap v2789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3991.742855 secs 2024-04-26T12:57:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:33 smithi096 ceph-mon[32753]: pgmap v2789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:33.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3991.742855 secs 2024-04-26T12:57:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:33 smithi194 ceph-mon[33017]: pgmap v2789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3991.742855 secs 2024-04-26T12:57:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3994.476894 secs 2024-04-26T12:57:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3994.476894 secs 2024-04-26T12:57:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3994.476894 secs 2024-04-26T12:57:35.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:35.550+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:35.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:35.550+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:35.551 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:35.550+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:57:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:35 smithi033 ceph-mon[27845]: pgmap v2790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:35 smithi096 ceph-mon[32753]: pgmap v2790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:35 smithi194 ceph-mon[33017]: pgmap v2790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:37 smithi033 ceph-mon[27845]: pgmap v2791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:37 smithi096 ceph-mon[32753]: pgmap v2791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:37 smithi194 ceph-mon[33017]: pgmap v2791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 3996.742937 secs 2024-04-26T12:57:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 3996.742937 secs 2024-04-26T12:57:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 3996.742937 secs 2024-04-26T12:57:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 3999.477048 secs 2024-04-26T12:57:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:39 smithi033 ceph-mon[27845]: pgmap v2792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 3999.477048 secs 2024-04-26T12:57:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:39 smithi096 ceph-mon[32753]: pgmap v2792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 3999.477048 secs 2024-04-26T12:57:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:39 smithi194 ceph-mon[33017]: pgmap v2792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:41 smithi096 ceph-mon[32753]: pgmap v2793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:41 smithi194 ceph-mon[33017]: pgmap v2793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:41 smithi033 ceph-mon[27845]: pgmap v2793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:43 smithi096 ceph-mon[32753]: pgmap v2794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4001.742993 secs 2024-04-26T12:57:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:43 smithi194 ceph-mon[33017]: pgmap v2794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4001.742993 secs 2024-04-26T12:57:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:43 smithi033 ceph-mon[27845]: pgmap v2794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4001.742993 secs 2024-04-26T12:57:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4004.477166 secs 2024-04-26T12:57:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4004.477166 secs 2024-04-26T12:57:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4004.477166 secs 2024-04-26T12:57:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:45 smithi194 ceph-mon[33017]: pgmap v2795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:45 smithi033 ceph-mon[27845]: pgmap v2795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:45 smithi096 ceph-mon[32753]: pgmap v2795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:47 smithi194 ceph-mon[33017]: pgmap v2796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:47 smithi033 ceph-mon[27845]: pgmap v2796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:47 smithi096 ceph-mon[32753]: pgmap v2796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4006.743076 secs 2024-04-26T12:57:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4009.477285 secs 2024-04-26T12:57:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4006.743076 secs 2024-04-26T12:57:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4009.477285 secs 2024-04-26T12:57:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4006.743076 secs 2024-04-26T12:57:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4009.477285 secs 2024-04-26T12:57:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:49 smithi194 ceph-mon[33017]: pgmap v2797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:49 smithi033 ceph-mon[27845]: pgmap v2797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:49 smithi096 ceph-mon[32753]: pgmap v2797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:50.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:50.552+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:50.554 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:50.552+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:57:50.554 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:57:50.552+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:57:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:51 smithi194 ceph-mon[33017]: pgmap v2798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:51 smithi033 ceph-mon[27845]: pgmap v2798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:51 smithi096 ceph-mon[32753]: pgmap v2798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:53 smithi194 ceph-mon[33017]: pgmap v2799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4011.743163 secs 2024-04-26T12:57:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4014.477415 secs 2024-04-26T12:57:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:53 smithi033 ceph-mon[27845]: pgmap v2799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4011.743163 secs 2024-04-26T12:57:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4014.477415 secs 2024-04-26T12:57:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:53 smithi096 ceph-mon[32753]: pgmap v2799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4011.743163 secs 2024-04-26T12:57:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4014.477415 secs 2024-04-26T12:57:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:55 smithi194 ceph-mon[33017]: pgmap v2800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:55 smithi033 ceph-mon[27845]: pgmap v2800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:55 smithi096 ceph-mon[32753]: pgmap v2800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:57:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:57 smithi194 ceph-mon[33017]: pgmap v2801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:57:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:57 smithi033 ceph-mon[27845]: pgmap v2801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:57:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:57 smithi096 ceph-mon[32753]: pgmap v2801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:57:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:57:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4016.743229 secs 2024-04-26T12:57:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4019.477539 secs 2024-04-26T12:57:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:57:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:57:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4016.743229 secs 2024-04-26T12:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4019.477539 secs 2024-04-26T12:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:57:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4016.743229 secs 2024-04-26T12:57:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4019.477539 secs 2024-04-26T12:57:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:57:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:57:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:58:00.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:59 smithi194 ceph-mon[33017]: pgmap v2802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:00.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:57:59 smithi194 ceph-mon[33017]: pgmap v2803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:59 smithi033 ceph-mon[27845]: pgmap v2802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:57:59 smithi033 ceph-mon[27845]: pgmap v2803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:59 smithi096 ceph-mon[32753]: pgmap v2802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:57:59 smithi096 ceph-mon[32753]: pgmap v2803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:01 smithi194 ceph-mon[33017]: pgmap v2804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:01 smithi033 ceph-mon[27845]: pgmap v2804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:01 smithi096 ceph-mon[32753]: pgmap v2804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:03 smithi194 ceph-mon[33017]: pgmap v2805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4021.743313 secs 2024-04-26T12:58:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4024.477645 secs 2024-04-26T12:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:03 smithi033 ceph-mon[27845]: pgmap v2805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4021.743313 secs 2024-04-26T12:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4024.477645 secs 2024-04-26T12:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:03 smithi096 ceph-mon[32753]: pgmap v2805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4021.743313 secs 2024-04-26T12:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4024.477645 secs 2024-04-26T12:58:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:05.567+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:05.567+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:05.567+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:58:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:05 smithi194 ceph-mon[33017]: pgmap v2806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:58:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:05 smithi033 ceph-mon[27845]: pgmap v2806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:58:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:05 smithi096 ceph-mon[32753]: pgmap v2806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T12:58:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:07 smithi194 ceph-mon[33017]: pgmap v2807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:07 smithi033 ceph-mon[27845]: pgmap v2807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:07 smithi096 ceph-mon[32753]: pgmap v2807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:58:09.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4026.743374 secs 2024-04-26T12:58:09.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4029.477770 secs 2024-04-26T12:58:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4026.743374 secs 2024-04-26T12:58:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4029.477770 secs 2024-04-26T12:58:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4026.743374 secs 2024-04-26T12:58:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4029.477770 secs 2024-04-26T12:58:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:09 smithi194 ceph-mon[33017]: pgmap v2808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:58:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:09 smithi033 ceph-mon[27845]: pgmap v2808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:58:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:09 smithi096 ceph-mon[32753]: pgmap v2808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:58:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:11 smithi194 ceph-mon[33017]: pgmap v2809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:11 smithi033 ceph-mon[27845]: pgmap v2809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:11 smithi096 ceph-mon[32753]: pgmap v2809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:13 smithi194 ceph-mon[33017]: pgmap v2810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4031.743460 secs 2024-04-26T12:58:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4034.477888 secs 2024-04-26T12:58:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:13 smithi033 ceph-mon[27845]: pgmap v2810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4031.743460 secs 2024-04-26T12:58:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4034.477888 secs 2024-04-26T12:58:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:13 smithi096 ceph-mon[32753]: pgmap v2810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4031.743460 secs 2024-04-26T12:58:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4034.477888 secs 2024-04-26T12:58:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:15 smithi194 ceph-mon[33017]: pgmap v2811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:15 smithi033 ceph-mon[27845]: pgmap v2811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:15 smithi096 ceph-mon[32753]: pgmap v2811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:17 smithi194 ceph-mon[33017]: pgmap v2812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:17 smithi033 ceph-mon[27845]: pgmap v2812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:17 smithi096 ceph-mon[32753]: pgmap v2812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4036.743540 secs 2024-04-26T12:58:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4039.478014 secs 2024-04-26T12:58:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4036.743540 secs 2024-04-26T12:58:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4039.478014 secs 2024-04-26T12:58:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4036.743540 secs 2024-04-26T12:58:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4039.478014 secs 2024-04-26T12:58:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:19 smithi194 ceph-mon[33017]: pgmap v2813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:19 smithi033 ceph-mon[27845]: pgmap v2813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:19 smithi096 ceph-mon[32753]: pgmap v2813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T12:58:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:20.583+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:20.583+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:20.583+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:58:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:21 smithi194 ceph-mon[33017]: pgmap v2814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:21 smithi033 ceph-mon[27845]: pgmap v2814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:21 smithi096 ceph-mon[32753]: pgmap v2814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T12:58:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:23 smithi194 ceph-mon[33017]: pgmap v2815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4041.743600 secs 2024-04-26T12:58:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4044.478163 secs 2024-04-26T12:58:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:23 smithi033 ceph-mon[27845]: pgmap v2815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4041.743600 secs 2024-04-26T12:58:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4044.478163 secs 2024-04-26T12:58:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:23 smithi096 ceph-mon[32753]: pgmap v2815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4041.743600 secs 2024-04-26T12:58:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4044.478163 secs 2024-04-26T12:58:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:25 smithi194 ceph-mon[33017]: pgmap v2816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:25 smithi033 ceph-mon[27845]: pgmap v2816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:25 smithi096 ceph-mon[32753]: pgmap v2816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:27 smithi194 ceph-mon[33017]: pgmap v2817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:27 smithi033 ceph-mon[27845]: pgmap v2817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:27 smithi096 ceph-mon[32753]: pgmap v2817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4046.743706 secs 2024-04-26T12:58:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4049.478246 secs 2024-04-26T12:58:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4046.743706 secs 2024-04-26T12:58:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4049.478246 secs 2024-04-26T12:58:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4046.743706 secs 2024-04-26T12:58:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4049.478246 secs 2024-04-26T12:58:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:29 smithi194 ceph-mon[33017]: pgmap v2818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:29 smithi033 ceph-mon[27845]: pgmap v2818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:29 smithi096 ceph-mon[32753]: pgmap v2818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:31 smithi033 ceph-mon[27845]: pgmap v2819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:31 smithi096 ceph-mon[32753]: pgmap v2819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:31 smithi194 ceph-mon[33017]: pgmap v2819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:33 smithi033 ceph-mon[27845]: pgmap v2820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4051.743767 secs 2024-04-26T12:58:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4054.478366 secs 2024-04-26T12:58:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:33 smithi096 ceph-mon[32753]: pgmap v2820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4051.743767 secs 2024-04-26T12:58:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4054.478366 secs 2024-04-26T12:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:33 smithi194 ceph-mon[33017]: pgmap v2820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4051.743767 secs 2024-04-26T12:58:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4054.478366 secs 2024-04-26T12:58:35.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:35.598+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:35.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:35.598+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:35.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:35.598+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:58:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:35 smithi033 ceph-mon[27845]: pgmap v2821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:35 smithi096 ceph-mon[32753]: pgmap v2821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:35 smithi194 ceph-mon[33017]: pgmap v2821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:37 smithi033 ceph-mon[27845]: pgmap v2822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:37 smithi096 ceph-mon[32753]: pgmap v2822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:37 smithi194 ceph-mon[33017]: pgmap v2822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4056.743827 secs 2024-04-26T12:58:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4059.478529 secs 2024-04-26T12:58:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4056.743827 secs 2024-04-26T12:58:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4059.478529 secs 2024-04-26T12:58:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4056.743827 secs 2024-04-26T12:58:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4059.478529 secs 2024-04-26T12:58:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:39 smithi033 ceph-mon[27845]: pgmap v2823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:39 smithi096 ceph-mon[32753]: pgmap v2823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:39 smithi194 ceph-mon[33017]: pgmap v2823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:41 smithi033 ceph-mon[27845]: pgmap v2824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:41 smithi096 ceph-mon[32753]: pgmap v2824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:41 smithi194 ceph-mon[33017]: pgmap v2824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:43 smithi033 ceph-mon[27845]: pgmap v2825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4061.743886 secs 2024-04-26T12:58:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4064.478655 secs 2024-04-26T12:58:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:43 smithi096 ceph-mon[32753]: pgmap v2825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4061.743886 secs 2024-04-26T12:58:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4064.478655 secs 2024-04-26T12:58:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:43 smithi194 ceph-mon[33017]: pgmap v2825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4061.743886 secs 2024-04-26T12:58:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4064.478655 secs 2024-04-26T12:58:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:45 smithi033 ceph-mon[27845]: pgmap v2826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:45 smithi096 ceph-mon[32753]: pgmap v2826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:45 smithi194 ceph-mon[33017]: pgmap v2826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:47 smithi033 ceph-mon[27845]: pgmap v2827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:47 smithi096 ceph-mon[32753]: pgmap v2827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:47 smithi194 ceph-mon[33017]: pgmap v2827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4066.743947 secs 2024-04-26T12:58:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4069.478796 secs 2024-04-26T12:58:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4066.743947 secs 2024-04-26T12:58:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4069.478796 secs 2024-04-26T12:58:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4066.743947 secs 2024-04-26T12:58:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4069.478796 secs 2024-04-26T12:58:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:49 smithi033 ceph-mon[27845]: pgmap v2828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:50.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:49 smithi096 ceph-mon[32753]: pgmap v2828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:49 smithi194 ceph-mon[33017]: pgmap v2828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:50.600+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:50.600+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:58:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:58:50.600+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:58:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:50 smithi096 ceph-mon[32753]: pgmap v2829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:50 smithi194 ceph-mon[33017]: pgmap v2829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:50 smithi033 ceph-mon[27845]: pgmap v2829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:53 smithi194 ceph-mon[33017]: pgmap v2830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4071.744005 secs 2024-04-26T12:58:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:53 smithi033 ceph-mon[27845]: pgmap v2830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4071.744005 secs 2024-04-26T12:58:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:53 smithi096 ceph-mon[32753]: pgmap v2830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4071.744005 secs 2024-04-26T12:58:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4074.478934 secs 2024-04-26T12:58:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4074.478934 secs 2024-04-26T12:58:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4074.478934 secs 2024-04-26T12:58:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:55 smithi033 ceph-mon[27845]: pgmap v2831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:55 smithi194 ceph-mon[33017]: pgmap v2831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:55 smithi096 ceph-mon[32753]: pgmap v2831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:57 smithi194 ceph-mon[33017]: pgmap v2832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:57 smithi033 ceph-mon[27845]: pgmap v2832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:57.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:57 smithi096 ceph-mon[32753]: pgmap v2832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:58:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4076.744029 secs 2024-04-26T12:58:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4076.744029 secs 2024-04-26T12:58:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4076.744029 secs 2024-04-26T12:58:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4079.479062 secs 2024-04-26T12:58:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:59 smithi194 ceph-mon[33017]: pgmap v2833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:58:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:58:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4079.479062 secs 2024-04-26T12:58:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:59 smithi033 ceph-mon[27845]: pgmap v2833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:58:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:58:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4079.479062 secs 2024-04-26T12:58:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:59 smithi096 ceph-mon[32753]: pgmap v2833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:58:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:58:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T12:59:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:59:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:59:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:00 smithi194 ceph-mon[33017]: pgmap v2834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:59:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:00 smithi194 ceph-mon[33017]: pgmap v2835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:59:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:59:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:59:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:00 smithi033 ceph-mon[27845]: pgmap v2834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:59:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:00 smithi033 ceph-mon[27845]: pgmap v2835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:59:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T12:59:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T12:59:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:00 smithi096 ceph-mon[32753]: pgmap v2834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T12:59:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:00 smithi096 ceph-mon[32753]: pgmap v2835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T12:59:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:02 smithi194 ceph-mon[33017]: pgmap v2836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:02 smithi033 ceph-mon[27845]: pgmap v2836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:02 smithi096 ceph-mon[32753]: pgmap v2836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4081.744090 secs 2024-04-26T12:59:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4084.479177 secs 2024-04-26T12:59:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4081.744090 secs 2024-04-26T12:59:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4084.479177 secs 2024-04-26T12:59:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4081.744090 secs 2024-04-26T12:59:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4084.479177 secs 2024-04-26T12:59:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:04 smithi033 ceph-mon[27845]: pgmap v2837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:04 smithi096 ceph-mon[32753]: pgmap v2837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:04 smithi194 ceph-mon[33017]: pgmap v2837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:05.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:05.615+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:05.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:05.615+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:05.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:05.615+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:59:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:06 smithi033 ceph-mon[27845]: pgmap v2838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:06 smithi096 ceph-mon[32753]: pgmap v2838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:06 smithi194 ceph-mon[33017]: pgmap v2838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4086.744173 secs 2024-04-26T12:59:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:08 smithi033 ceph-mon[27845]: pgmap v2839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4089.479297 secs 2024-04-26T12:59:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4086.744173 secs 2024-04-26T12:59:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:08 smithi096 ceph-mon[32753]: pgmap v2839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4089.479297 secs 2024-04-26T12:59:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4086.744173 secs 2024-04-26T12:59:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:08 smithi194 ceph-mon[33017]: pgmap v2839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T12:59:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4089.479297 secs 2024-04-26T12:59:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:10 smithi033 ceph-mon[27845]: pgmap v2840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:59:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:10 smithi096 ceph-mon[32753]: pgmap v2840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:59:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:10 smithi194 ceph-mon[33017]: pgmap v2840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T12:59:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:12 smithi033 ceph-mon[27845]: pgmap v2841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:12 smithi096 ceph-mon[32753]: pgmap v2841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:12 smithi194 ceph-mon[33017]: pgmap v2841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4091.744235 secs 2024-04-26T12:59:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4094.479388 secs 2024-04-26T12:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4091.744235 secs 2024-04-26T12:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4094.479388 secs 2024-04-26T12:59:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4091.744235 secs 2024-04-26T12:59:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4094.479388 secs 2024-04-26T12:59:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:14 smithi033 ceph-mon[27845]: pgmap v2842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:14 smithi096 ceph-mon[32753]: pgmap v2842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:14 smithi194 ceph-mon[33017]: pgmap v2842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:16 smithi033 ceph-mon[27845]: pgmap v2843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:16 smithi096 ceph-mon[32753]: pgmap v2843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:16 smithi194 ceph-mon[33017]: pgmap v2843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4096.744319 secs 2024-04-26T12:59:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:18 smithi033 ceph-mon[27845]: pgmap v2844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4099.479524 secs 2024-04-26T12:59:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4096.744319 secs 2024-04-26T12:59:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:18 smithi096 ceph-mon[32753]: pgmap v2844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4099.479524 secs 2024-04-26T12:59:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4096.744319 secs 2024-04-26T12:59:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:18 smithi194 ceph-mon[33017]: pgmap v2844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4099.479524 secs 2024-04-26T12:59:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:20.619+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:20.619+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:20.619+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:59:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:20 smithi033 ceph-mon[27845]: pgmap v2845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:20 smithi096 ceph-mon[32753]: pgmap v2845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:20 smithi194 ceph-mon[33017]: pgmap v2845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:22.485+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T12:59:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:22.485+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T12:59:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:22.485+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T12:59:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:22.486+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T12:59:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:22 smithi033 ceph-mon[27845]: pgmap v2846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:22 smithi096 ceph-mon[32753]: pgmap v2846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:22 smithi194 ceph-mon[33017]: pgmap v2846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4101.744405 secs 2024-04-26T12:59:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4104.479661 secs 2024-04-26T12:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4101.744405 secs 2024-04-26T12:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4104.479661 secs 2024-04-26T12:59:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4101.744405 secs 2024-04-26T12:59:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4104.479661 secs 2024-04-26T12:59:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:24 smithi033 ceph-mon[27845]: pgmap v2847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:24 smithi096 ceph-mon[32753]: pgmap v2847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:24 smithi194 ceph-mon[33017]: pgmap v2847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:25 smithi033 ceph-mon[27845]: pgmap v2848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:25 smithi194 ceph-mon[33017]: pgmap v2848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:25 smithi096 ceph-mon[32753]: pgmap v2848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4106.744464 secs 2024-04-26T12:59:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:28 smithi033 ceph-mon[27845]: pgmap v2849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4109.479787 secs 2024-04-26T12:59:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4106.744464 secs 2024-04-26T12:59:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:28 smithi096 ceph-mon[32753]: pgmap v2849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4109.479787 secs 2024-04-26T12:59:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4106.744464 secs 2024-04-26T12:59:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:28 smithi194 ceph-mon[33017]: pgmap v2849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4109.479787 secs 2024-04-26T12:59:31.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:30 smithi194 ceph-mon[33017]: pgmap v2850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:30 smithi033 ceph-mon[27845]: pgmap v2850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:30 smithi096 ceph-mon[32753]: pgmap v2850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:32 smithi033 ceph-mon[27845]: pgmap v2851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:32 smithi096 ceph-mon[32753]: pgmap v2851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:32 smithi194 ceph-mon[33017]: pgmap v2851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4111.744550 secs 2024-04-26T12:59:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4114.479913 secs 2024-04-26T12:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4111.744550 secs 2024-04-26T12:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4114.479913 secs 2024-04-26T12:59:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4111.744550 secs 2024-04-26T12:59:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4114.479913 secs 2024-04-26T12:59:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:34 smithi033 ceph-mon[27845]: pgmap v2852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:34 smithi096 ceph-mon[32753]: pgmap v2852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:34 smithi194 ceph-mon[33017]: pgmap v2852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:35.628+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:35.628+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:35.628+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:59:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:36 smithi033 ceph-mon[27845]: pgmap v2853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:36 smithi096 ceph-mon[32753]: pgmap v2853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:36 smithi194 ceph-mon[33017]: pgmap v2853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4116.744638 secs 2024-04-26T12:59:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:38 smithi033 ceph-mon[27845]: pgmap v2854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4119.479975 secs 2024-04-26T12:59:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4116.744638 secs 2024-04-26T12:59:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:38 smithi096 ceph-mon[32753]: pgmap v2854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4119.479975 secs 2024-04-26T12:59:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4116.744638 secs 2024-04-26T12:59:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:38 smithi194 ceph-mon[33017]: pgmap v2854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4119.479975 secs 2024-04-26T12:59:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:40 smithi033 ceph-mon[27845]: pgmap v2855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:40 smithi096 ceph-mon[32753]: pgmap v2855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:40 smithi194 ceph-mon[33017]: pgmap v2855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:42 smithi033 ceph-mon[27845]: pgmap v2856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:42 smithi096 ceph-mon[32753]: pgmap v2856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:42 smithi194 ceph-mon[33017]: pgmap v2856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4121.744697 secs 2024-04-26T12:59:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4124.480133 secs 2024-04-26T12:59:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4121.744697 secs 2024-04-26T12:59:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4124.480133 secs 2024-04-26T12:59:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4121.744697 secs 2024-04-26T12:59:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4124.480133 secs 2024-04-26T12:59:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:44 smithi033 ceph-mon[27845]: pgmap v2857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:44 smithi096 ceph-mon[32753]: pgmap v2857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:44 smithi194 ceph-mon[33017]: pgmap v2857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:46 smithi033 ceph-mon[27845]: pgmap v2858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:46 smithi096 ceph-mon[32753]: pgmap v2858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:46 smithi194 ceph-mon[33017]: pgmap v2858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:47 smithi194 ceph-mon[33017]: pgmap v2859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:47 smithi033 ceph-mon[27845]: pgmap v2859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:47 smithi096 ceph-mon[32753]: pgmap v2859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4126.744771 secs 2024-04-26T12:59:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4129.480226 secs 2024-04-26T12:59:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4126.744771 secs 2024-04-26T12:59:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4129.480226 secs 2024-04-26T12:59:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4126.744771 secs 2024-04-26T12:59:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4129.480226 secs 2024-04-26T12:59:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:49 smithi194 ceph-mon[33017]: pgmap v2860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:49 smithi033 ceph-mon[27845]: pgmap v2860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:49 smithi096 ceph-mon[32753]: pgmap v2860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:50.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:50.643+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:50.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:50.643+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T12:59:50.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T12:59:50.643+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T12:59:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:52 smithi033 ceph-mon[27845]: pgmap v2861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:52 smithi096 ceph-mon[32753]: pgmap v2861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:52 smithi194 ceph-mon[33017]: pgmap v2861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4131.744858 secs 2024-04-26T12:59:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4134.480361 secs 2024-04-26T12:59:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4131.744858 secs 2024-04-26T12:59:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4134.480361 secs 2024-04-26T12:59:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4131.744858 secs 2024-04-26T12:59:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4134.480361 secs 2024-04-26T12:59:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:54 smithi033 ceph-mon[27845]: pgmap v2862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:54 smithi096 ceph-mon[32753]: pgmap v2862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:54 smithi194 ceph-mon[33017]: pgmap v2862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:56 smithi033 ceph-mon[27845]: pgmap v2863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:56 smithi096 ceph-mon[32753]: pgmap v2863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:56 smithi194 ceph-mon[33017]: pgmap v2863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T12:59:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4136.744941 secs 2024-04-26T12:59:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:58 smithi033 ceph-mon[27845]: pgmap v2864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 12:59:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4139.480490 secs 2024-04-26T12:59:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4136.744941 secs 2024-04-26T12:59:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:58 smithi096 ceph-mon[32753]: pgmap v2864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 12:59:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4139.480490 secs 2024-04-26T12:59:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4136.744941 secs 2024-04-26T12:59:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:58 smithi194 ceph-mon[33017]: pgmap v2864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T12:59:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 12:59:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4139.480490 secs 2024-04-26T13:00:01.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:00 smithi194 ceph-mon[33017]: pgmap v2865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:01.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:00:01.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:00:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:00 smithi033 ceph-mon[27845]: pgmap v2865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:00:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:00:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:00 smithi096 ceph-mon[32753]: pgmap v2865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:00:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:00:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:00:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:00:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:00:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:00:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:00:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:00:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:00:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:00:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:00:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:02 smithi033 ceph-mon[27845]: pgmap v2866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:02 smithi033 ceph-mon[27845]: pgmap v2867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:02 smithi096 ceph-mon[32753]: pgmap v2866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:02 smithi096 ceph-mon[32753]: pgmap v2867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:02 smithi194 ceph-mon[33017]: pgmap v2866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:02 smithi194 ceph-mon[33017]: pgmap v2867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4141.745026 secs 2024-04-26T13:00:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4144.480602 secs 2024-04-26T13:00:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4141.745026 secs 2024-04-26T13:00:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4144.480602 secs 2024-04-26T13:00:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4141.745026 secs 2024-04-26T13:00:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4144.480602 secs 2024-04-26T13:00:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:04 smithi096 ceph-mon[32753]: pgmap v2868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:04 smithi194 ceph-mon[33017]: pgmap v2868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:04 smithi033 ceph-mon[27845]: pgmap v2868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:05.658+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:05.658+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:05.658+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:00:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:05 smithi194 ceph-mon[33017]: pgmap v2869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:05 smithi033 ceph-mon[27845]: pgmap v2869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:05 smithi096 ceph-mon[32753]: pgmap v2869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:08 smithi194 ceph-mon[33017]: pgmap v2870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4146.745086 secs 2024-04-26T13:00:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:08 smithi033 ceph-mon[27845]: pgmap v2870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4146.745086 secs 2024-04-26T13:00:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:08 smithi096 ceph-mon[32753]: pgmap v2870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:00:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4146.745086 secs 2024-04-26T13:00:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4149.480740 secs 2024-04-26T13:00:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4149.480740 secs 2024-04-26T13:00:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4149.480740 secs 2024-04-26T13:00:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:10 smithi194 ceph-mon[33017]: pgmap v2871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:00:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:10 smithi033 ceph-mon[27845]: pgmap v2871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:00:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:10 smithi096 ceph-mon[32753]: pgmap v2871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:00:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:12 smithi194 ceph-mon[33017]: pgmap v2872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:00:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:12 smithi033 ceph-mon[27845]: pgmap v2872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:00:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:12 smithi096 ceph-mon[32753]: pgmap v2872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:00:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4151.745145 secs 2024-04-26T13:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4151.745145 secs 2024-04-26T13:00:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4151.745145 secs 2024-04-26T13:00:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4154.480865 secs 2024-04-26T13:00:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:14 smithi194 ceph-mon[33017]: pgmap v2873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4154.480865 secs 2024-04-26T13:00:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:14 smithi033 ceph-mon[27845]: pgmap v2873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4154.480865 secs 2024-04-26T13:00:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:14 smithi096 ceph-mon[32753]: pgmap v2873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:16 smithi194 ceph-mon[33017]: pgmap v2874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:16 smithi033 ceph-mon[27845]: pgmap v2874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:16 smithi096 ceph-mon[32753]: pgmap v2874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:18 smithi194 ceph-mon[33017]: pgmap v2875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4156.745231 secs 2024-04-26T13:00:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:18 smithi033 ceph-mon[27845]: pgmap v2875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4156.745231 secs 2024-04-26T13:00:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:18 smithi096 ceph-mon[32753]: pgmap v2875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4156.745231 secs 2024-04-26T13:00:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4159.481004 secs 2024-04-26T13:00:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4159.481004 secs 2024-04-26T13:00:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4159.481004 secs 2024-04-26T13:00:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:20.667+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:20.667+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:20.667+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:00:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:20 smithi194 ceph-mon[33017]: pgmap v2876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:20 smithi033 ceph-mon[27845]: pgmap v2876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:20 smithi096 ceph-mon[32753]: pgmap v2876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:22 smithi194 ceph-mon[33017]: pgmap v2877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:22 smithi033 ceph-mon[27845]: pgmap v2877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:22 smithi096 ceph-mon[32753]: pgmap v2877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4161.745309 secs 2024-04-26T13:00:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4161.745309 secs 2024-04-26T13:00:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4161.745309 secs 2024-04-26T13:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4164.481131 secs 2024-04-26T13:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:24 smithi194 ceph-mon[33017]: pgmap v2878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4164.481131 secs 2024-04-26T13:00:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:24 smithi033 ceph-mon[27845]: pgmap v2878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4164.481131 secs 2024-04-26T13:00:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:24 smithi096 ceph-mon[32753]: pgmap v2878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:26 smithi194 ceph-mon[33017]: pgmap v2879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:26 smithi033 ceph-mon[27845]: pgmap v2879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:26 smithi096 ceph-mon[32753]: pgmap v2879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:28 smithi033 ceph-mon[27845]: pgmap v2880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4166.745394 secs 2024-04-26T13:00:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:28 smithi096 ceph-mon[32753]: pgmap v2880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4166.745394 secs 2024-04-26T13:00:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:28 smithi194 ceph-mon[33017]: pgmap v2880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4166.745394 secs 2024-04-26T13:00:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4169.481269 secs 2024-04-26T13:00:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4169.481269 secs 2024-04-26T13:00:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4169.481269 secs 2024-04-26T13:00:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:30 smithi033 ceph-mon[27845]: pgmap v2881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:30 smithi096 ceph-mon[32753]: pgmap v2881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:31.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:30 smithi194 ceph-mon[33017]: pgmap v2881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:32 smithi033 ceph-mon[27845]: pgmap v2882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:32 smithi096 ceph-mon[32753]: pgmap v2882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:32 smithi194 ceph-mon[33017]: pgmap v2882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4171.745485 secs 2024-04-26T13:00:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4171.745485 secs 2024-04-26T13:00:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4171.745485 secs 2024-04-26T13:00:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4174.481409 secs 2024-04-26T13:00:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:34 smithi033 ceph-mon[27845]: pgmap v2883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4174.481409 secs 2024-04-26T13:00:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:34 smithi096 ceph-mon[32753]: pgmap v2883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4174.481409 secs 2024-04-26T13:00:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:34 smithi194 ceph-mon[33017]: pgmap v2883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:35.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:35.681+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:35.681+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:35.681+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:00:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:36 smithi033 ceph-mon[27845]: pgmap v2884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:36 smithi096 ceph-mon[32753]: pgmap v2884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:36 smithi194 ceph-mon[33017]: pgmap v2884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:38 smithi033 ceph-mon[27845]: pgmap v2885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4176.745571 secs 2024-04-26T13:00:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:38 smithi096 ceph-mon[32753]: pgmap v2885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4176.745571 secs 2024-04-26T13:00:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:38 smithi194 ceph-mon[33017]: pgmap v2885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4176.745571 secs 2024-04-26T13:00:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4179.481566 secs 2024-04-26T13:00:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4179.481566 secs 2024-04-26T13:00:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4179.481566 secs 2024-04-26T13:00:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:40 smithi033 ceph-mon[27845]: pgmap v2886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:40 smithi096 ceph-mon[32753]: pgmap v2886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:40 smithi194 ceph-mon[33017]: pgmap v2886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:42 smithi096 ceph-mon[32753]: pgmap v2887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:42 smithi194 ceph-mon[33017]: pgmap v2887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:42 smithi033 ceph-mon[27845]: pgmap v2887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4181.745650 secs 2024-04-26T13:00:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4181.745650 secs 2024-04-26T13:00:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4181.745650 secs 2024-04-26T13:00:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4184.481693 secs 2024-04-26T13:00:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:44 smithi194 ceph-mon[33017]: pgmap v2888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4184.481693 secs 2024-04-26T13:00:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:44 smithi033 ceph-mon[27845]: pgmap v2888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4184.481693 secs 2024-04-26T13:00:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:44 smithi096 ceph-mon[32753]: pgmap v2888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:46 smithi194 ceph-mon[33017]: pgmap v2889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:46 smithi033 ceph-mon[27845]: pgmap v2889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:46 smithi096 ceph-mon[32753]: pgmap v2889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:48 smithi194 ceph-mon[33017]: pgmap v2890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4186.745708 secs 2024-04-26T13:00:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:48 smithi033 ceph-mon[27845]: pgmap v2890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4186.745708 secs 2024-04-26T13:00:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:48 smithi096 ceph-mon[32753]: pgmap v2890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4186.745708 secs 2024-04-26T13:00:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4189.481810 secs 2024-04-26T13:00:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4189.481810 secs 2024-04-26T13:00:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4189.481810 secs 2024-04-26T13:00:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:50.696+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:50.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:50.696+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:00:50.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:00:50.696+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:00:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:50 smithi194 ceph-mon[33017]: pgmap v2891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:50 smithi033 ceph-mon[27845]: pgmap v2891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:50 smithi096 ceph-mon[32753]: pgmap v2891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:52 smithi194 ceph-mon[33017]: pgmap v2892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:52 smithi033 ceph-mon[27845]: pgmap v2892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:52 smithi096 ceph-mon[32753]: pgmap v2892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4191.745792 secs 2024-04-26T13:00:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4191.745792 secs 2024-04-26T13:00:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4191.745792 secs 2024-04-26T13:00:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4194.481929 secs 2024-04-26T13:00:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:54 smithi194 ceph-mon[33017]: pgmap v2893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4194.481929 secs 2024-04-26T13:00:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:54 smithi033 ceph-mon[27845]: pgmap v2893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4194.481929 secs 2024-04-26T13:00:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:54 smithi096 ceph-mon[32753]: pgmap v2893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:56 smithi194 ceph-mon[33017]: pgmap v2894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:56 smithi033 ceph-mon[27845]: pgmap v2894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:56 smithi096 ceph-mon[32753]: pgmap v2894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:00:59.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:58 smithi033 ceph-mon[27845]: pgmap v2895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:59.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4196.745878 secs 2024-04-26T13:00:59.056 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:00:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4199.482064 secs 2024-04-26T13:00:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:58 smithi194 ceph-mon[33017]: pgmap v2895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4196.745878 secs 2024-04-26T13:00:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:00:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4199.482064 secs 2024-04-26T13:00:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:58 smithi096 ceph-mon[32753]: pgmap v2895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:00:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4196.745878 secs 2024-04-26T13:00:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:00:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4199.482064 secs 2024-04-26T13:01:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:00 smithi194 ceph-mon[33017]: pgmap v2896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:00 smithi033 ceph-mon[27845]: pgmap v2896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:00 smithi096 ceph-mon[32753]: pgmap v2896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:02 smithi194 ceph-mon[33017]: pgmap v2897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:01:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:02 smithi033 ceph-mon[27845]: pgmap v2897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:01:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:02 smithi096 ceph-mon[32753]: pgmap v2897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:01:04.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4201.745989 secs 2024-04-26T13:01:04.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4204.482188 secs 2024-04-26T13:01:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4201.745989 secs 2024-04-26T13:01:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4204.482188 secs 2024-04-26T13:01:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4201.745989 secs 2024-04-26T13:01:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4204.482188 secs 2024-04-26T13:01:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:04 smithi033 ceph-mon[27845]: pgmap v2898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:01:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:01:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:04 smithi096 ceph-mon[32753]: pgmap v2898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:01:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:01:05.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:04 smithi194 ceph-mon[33017]: pgmap v2898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:01:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:01:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:01:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:05.701+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:05.701+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:05.701+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:01:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:05 smithi033 ceph-mon[27845]: pgmap v2899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:01:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:05 smithi033 ceph-mon[27845]: pgmap v2900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:01:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:05 smithi096 ceph-mon[32753]: pgmap v2899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:01:06.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:05 smithi096 ceph-mon[32753]: pgmap v2900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:01:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:05 smithi194 ceph-mon[33017]: pgmap v2899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:01:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:05 smithi194 ceph-mon[33017]: pgmap v2900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:01:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:07 smithi096 ceph-mon[32753]: pgmap v2901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:07 smithi194 ceph-mon[33017]: pgmap v2901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:07 smithi033 ceph-mon[27845]: pgmap v2901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4206.746058 secs 2024-04-26T13:01:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4209.482305 secs 2024-04-26T13:01:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4206.746058 secs 2024-04-26T13:01:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4209.482305 secs 2024-04-26T13:01:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4206.746058 secs 2024-04-26T13:01:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4209.482305 secs 2024-04-26T13:01:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:09 smithi194 ceph-mon[33017]: pgmap v2902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:01:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:09 smithi033 ceph-mon[27845]: pgmap v2902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:01:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:09 smithi096 ceph-mon[32753]: pgmap v2902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:01:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:10 smithi194 ceph-mon[33017]: pgmap v2903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:10 smithi033 ceph-mon[27845]: pgmap v2903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:10 smithi096 ceph-mon[32753]: pgmap v2903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:12 smithi194 ceph-mon[33017]: pgmap v2904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4211.746138 secs 2024-04-26T13:01:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:12 smithi033 ceph-mon[27845]: pgmap v2904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4211.746138 secs 2024-04-26T13:01:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:12 smithi096 ceph-mon[32753]: pgmap v2904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:01:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4211.746138 secs 2024-04-26T13:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4214.482424 secs 2024-04-26T13:01:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4214.482424 secs 2024-04-26T13:01:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4214.482424 secs 2024-04-26T13:01:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:15 smithi194 ceph-mon[33017]: pgmap v2905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:01:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:15 smithi033 ceph-mon[27845]: pgmap v2905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:01:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:15 smithi096 ceph-mon[32753]: pgmap v2905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:01:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:17 smithi194 ceph-mon[33017]: pgmap v2906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:17 smithi033 ceph-mon[27845]: pgmap v2906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:17 smithi096 ceph-mon[32753]: pgmap v2906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4216.746228 secs 2024-04-26T13:01:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4216.746228 secs 2024-04-26T13:01:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4216.746228 secs 2024-04-26T13:01:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:19 smithi194 ceph-mon[33017]: pgmap v2907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4219.482539 secs 2024-04-26T13:01:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:19 smithi033 ceph-mon[27845]: pgmap v2907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4219.482539 secs 2024-04-26T13:01:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:19 smithi096 ceph-mon[32753]: pgmap v2907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4219.482539 secs 2024-04-26T13:01:20.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:20.716+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:20.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:20.716+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:20.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:20.716+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:01:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:21 smithi194 ceph-mon[33017]: pgmap v2908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:21 smithi033 ceph-mon[27845]: pgmap v2908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:21 smithi096 ceph-mon[32753]: pgmap v2908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:23 smithi194 ceph-mon[33017]: pgmap v2909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4221.746315 secs 2024-04-26T13:01:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:23 smithi033 ceph-mon[27845]: pgmap v2909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4221.746315 secs 2024-04-26T13:01:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:23 smithi096 ceph-mon[32753]: pgmap v2909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4221.746315 secs 2024-04-26T13:01:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4224.482633 secs 2024-04-26T13:01:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4224.482633 secs 2024-04-26T13:01:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4224.482633 secs 2024-04-26T13:01:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:25 smithi194 ceph-mon[33017]: pgmap v2910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:25 smithi033 ceph-mon[27845]: pgmap v2910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:25 smithi096 ceph-mon[32753]: pgmap v2910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:27 smithi194 ceph-mon[33017]: pgmap v2911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:27 smithi033 ceph-mon[27845]: pgmap v2911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:27 smithi096 ceph-mon[32753]: pgmap v2911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4226.746378 secs 2024-04-26T13:01:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4226.746378 secs 2024-04-26T13:01:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4226.746378 secs 2024-04-26T13:01:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:29 smithi033 ceph-mon[27845]: pgmap v2912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4229.482758 secs 2024-04-26T13:01:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:29 smithi096 ceph-mon[32753]: pgmap v2912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4229.482758 secs 2024-04-26T13:01:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:29 smithi194 ceph-mon[33017]: pgmap v2912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4229.482758 secs 2024-04-26T13:01:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:31 smithi033 ceph-mon[27845]: pgmap v2913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:31 smithi096 ceph-mon[32753]: pgmap v2913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:31 smithi194 ceph-mon[33017]: pgmap v2913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:33.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:33 smithi194 ceph-mon[33017]: pgmap v2914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:33.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4231.746438 secs 2024-04-26T13:01:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:33 smithi033 ceph-mon[27845]: pgmap v2914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4231.746438 secs 2024-04-26T13:01:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:33 smithi096 ceph-mon[32753]: pgmap v2914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4231.746438 secs 2024-04-26T13:01:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4234.482878 secs 2024-04-26T13:01:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4234.482878 secs 2024-04-26T13:01:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4234.482878 secs 2024-04-26T13:01:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:35 smithi033 ceph-mon[27845]: pgmap v2915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:35 smithi096 ceph-mon[32753]: pgmap v2915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:35.733+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:35.733+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:35.733+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:01:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:35 smithi194 ceph-mon[33017]: pgmap v2915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:37 smithi033 ceph-mon[27845]: pgmap v2916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:37 smithi096 ceph-mon[32753]: pgmap v2916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:37 smithi194 ceph-mon[33017]: pgmap v2916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4236.746520 secs 2024-04-26T13:01:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4236.746520 secs 2024-04-26T13:01:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4236.746520 secs 2024-04-26T13:01:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:39 smithi033 ceph-mon[27845]: pgmap v2917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4239.483002 secs 2024-04-26T13:01:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:39 smithi096 ceph-mon[32753]: pgmap v2917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4239.483002 secs 2024-04-26T13:01:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:39 smithi194 ceph-mon[33017]: pgmap v2917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4239.483002 secs 2024-04-26T13:01:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:41 smithi033 ceph-mon[27845]: pgmap v2918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:41 smithi096 ceph-mon[32753]: pgmap v2918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:41 smithi194 ceph-mon[33017]: pgmap v2918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:43 smithi033 ceph-mon[27845]: pgmap v2919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4241.746593 secs 2024-04-26T13:01:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:43 smithi096 ceph-mon[32753]: pgmap v2919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4241.746593 secs 2024-04-26T13:01:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:43 smithi194 ceph-mon[33017]: pgmap v2919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4241.746593 secs 2024-04-26T13:01:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4244.483136 secs 2024-04-26T13:01:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4244.483136 secs 2024-04-26T13:01:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4244.483136 secs 2024-04-26T13:01:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:45 smithi033 ceph-mon[27845]: pgmap v2920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:45 smithi096 ceph-mon[32753]: pgmap v2920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:45 smithi194 ceph-mon[33017]: pgmap v2920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:47 smithi033 ceph-mon[27845]: pgmap v2921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:47 smithi096 ceph-mon[32753]: pgmap v2921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:47 smithi194 ceph-mon[33017]: pgmap v2921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4246.746707 secs 2024-04-26T13:01:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4246.746707 secs 2024-04-26T13:01:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4246.746707 secs 2024-04-26T13:01:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:49 smithi194 ceph-mon[33017]: pgmap v2922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4249.483247 secs 2024-04-26T13:01:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:49 smithi033 ceph-mon[27845]: pgmap v2922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4249.483247 secs 2024-04-26T13:01:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:49 smithi096 ceph-mon[32753]: pgmap v2922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4249.483247 secs 2024-04-26T13:01:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:50.748+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:50.748+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:01:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:01:50.748+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:01:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:51 smithi194 ceph-mon[33017]: pgmap v2923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:51 smithi033 ceph-mon[27845]: pgmap v2923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:51 smithi096 ceph-mon[32753]: pgmap v2923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:53 smithi194 ceph-mon[33017]: pgmap v2924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4251.746791 secs 2024-04-26T13:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:53 smithi033 ceph-mon[27845]: pgmap v2924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4251.746791 secs 2024-04-26T13:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:53 smithi096 ceph-mon[32753]: pgmap v2924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4251.746791 secs 2024-04-26T13:01:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4254.483331 secs 2024-04-26T13:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4254.483331 secs 2024-04-26T13:01:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4254.483331 secs 2024-04-26T13:01:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:55 smithi194 ceph-mon[33017]: pgmap v2925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:55 smithi033 ceph-mon[27845]: pgmap v2925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:55.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:55 smithi096 ceph-mon[32753]: pgmap v2925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:57 smithi194 ceph-mon[33017]: pgmap v2926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:57 smithi033 ceph-mon[27845]: pgmap v2926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:57 smithi096 ceph-mon[32753]: pgmap v2926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:01:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4256.746897 secs 2024-04-26T13:01:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4256.746897 secs 2024-04-26T13:01:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4256.746897 secs 2024-04-26T13:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:59 smithi194 ceph-mon[33017]: pgmap v2927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4259.483441 secs 2024-04-26T13:01:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:59 smithi033 ceph-mon[27845]: pgmap v2927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4259.483441 secs 2024-04-26T13:01:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:59 smithi096 ceph-mon[32753]: pgmap v2927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:01:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4259.483441 secs 2024-04-26T13:02:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:01 smithi194 ceph-mon[33017]: pgmap v2928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:01 smithi033 ceph-mon[27845]: pgmap v2928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:01 smithi096 ceph-mon[32753]: pgmap v2928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:03 smithi194 ceph-mon[33017]: pgmap v2929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4261.746985 secs 2024-04-26T13:02:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:03 smithi033 ceph-mon[27845]: pgmap v2929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4261.746985 secs 2024-04-26T13:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:03 smithi096 ceph-mon[32753]: pgmap v2929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4261.746985 secs 2024-04-26T13:02:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4264.483564 secs 2024-04-26T13:02:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:02:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4264.483564 secs 2024-04-26T13:02:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:02:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4264.483564 secs 2024-04-26T13:02:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:02:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:05.762+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:05.762+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:05.763+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:02:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:05 smithi194 ceph-mon[33017]: pgmap v2930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:05 smithi033 ceph-mon[27845]: pgmap v2930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:05 smithi096 ceph-mon[32753]: pgmap v2930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:07 smithi194 ceph-mon[33017]: pgmap v2931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:02:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:02:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:07 smithi194 ceph-mon[33017]: pgmap v2932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:02:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:07 smithi194 ceph-mon[33017]: pgmap v2933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:07 smithi033 ceph-mon[27845]: pgmap v2931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:02:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:02:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:07 smithi033 ceph-mon[27845]: pgmap v2932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:02:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:07 smithi033 ceph-mon[27845]: pgmap v2933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:07 smithi096 ceph-mon[32753]: pgmap v2931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:02:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:02:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:07 smithi096 ceph-mon[32753]: pgmap v2932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:02:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:07 smithi096 ceph-mon[32753]: pgmap v2933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:07.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:02:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4266.747047 secs 2024-04-26T13:02:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4266.747047 secs 2024-04-26T13:02:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4266.747047 secs 2024-04-26T13:02:09.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4269.483694 secs 2024-04-26T13:02:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:09 smithi096 ceph-mon[32753]: pgmap v2934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:02:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4269.483694 secs 2024-04-26T13:02:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:09 smithi194 ceph-mon[33017]: pgmap v2934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:02:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4269.483694 secs 2024-04-26T13:02:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:09 smithi033 ceph-mon[27845]: pgmap v2934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:02:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:11 smithi194 ceph-mon[33017]: pgmap v2935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:11 smithi033 ceph-mon[27845]: pgmap v2935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:11 smithi096 ceph-mon[32753]: pgmap v2935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4271.747129 secs 2024-04-26T13:02:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:13 smithi194 ceph-mon[33017]: pgmap v2936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4271.747129 secs 2024-04-26T13:02:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:13 smithi033 ceph-mon[27845]: pgmap v2936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4271.747129 secs 2024-04-26T13:02:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:13 smithi096 ceph-mon[32753]: pgmap v2936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4274.483815 secs 2024-04-26T13:02:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4274.483815 secs 2024-04-26T13:02:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4274.483815 secs 2024-04-26T13:02:16.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:15 smithi033 ceph-mon[27845]: pgmap v2937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:15 smithi194 ceph-mon[33017]: pgmap v2937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:15 smithi096 ceph-mon[32753]: pgmap v2937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:02:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:17 smithi194 ceph-mon[33017]: pgmap v2938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:02:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:17 smithi033 ceph-mon[27845]: pgmap v2938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:02:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:17 smithi096 ceph-mon[32753]: pgmap v2938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:02:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4276.747213 secs 2024-04-26T13:02:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4279.483885 secs 2024-04-26T13:02:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4276.747213 secs 2024-04-26T13:02:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4279.483885 secs 2024-04-26T13:02:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4276.747213 secs 2024-04-26T13:02:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4279.483885 secs 2024-04-26T13:02:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:19 smithi194 ceph-mon[33017]: pgmap v2939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:19 smithi033 ceph-mon[27845]: pgmap v2939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:19 smithi096 ceph-mon[32753]: pgmap v2939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:20.777+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:20.777+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:20.777+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:02:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:21 smithi194 ceph-mon[33017]: pgmap v2940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:21 smithi033 ceph-mon[27845]: pgmap v2940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:21 smithi096 ceph-mon[32753]: pgmap v2940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:22.516+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:02:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:22.516+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:02:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:22.516+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T13:02:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:22.516+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T13:02:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4281.747299 secs 2024-04-26T13:02:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:23 smithi194 ceph-mon[33017]: pgmap v2941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4284.484001 secs 2024-04-26T13:02:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4281.747299 secs 2024-04-26T13:02:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:23 smithi033 ceph-mon[27845]: pgmap v2941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4284.484001 secs 2024-04-26T13:02:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4281.747299 secs 2024-04-26T13:02:24.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:23 smithi096 ceph-mon[32753]: pgmap v2941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:24.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4284.484001 secs 2024-04-26T13:02:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:25 smithi194 ceph-mon[33017]: pgmap v2942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:25 smithi033 ceph-mon[27845]: pgmap v2942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:25 smithi096 ceph-mon[32753]: pgmap v2942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:27 smithi194 ceph-mon[33017]: pgmap v2943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:27 smithi033 ceph-mon[27845]: pgmap v2943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:27 smithi096 ceph-mon[32753]: pgmap v2943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4286.747373 secs 2024-04-26T13:02:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4289.484154 secs 2024-04-26T13:02:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4286.747373 secs 2024-04-26T13:02:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4289.484154 secs 2024-04-26T13:02:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4286.747373 secs 2024-04-26T13:02:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4289.484154 secs 2024-04-26T13:02:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:29 smithi194 ceph-mon[33017]: pgmap v2944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:29 smithi033 ceph-mon[27845]: pgmap v2944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:29 smithi096 ceph-mon[32753]: pgmap v2944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:31 smithi194 ceph-mon[33017]: pgmap v2945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:31 smithi033 ceph-mon[27845]: pgmap v2945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:31 smithi096 ceph-mon[32753]: pgmap v2945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4291.747459 secs 2024-04-26T13:02:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:33 smithi194 ceph-mon[33017]: pgmap v2946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4294.484237 secs 2024-04-26T13:02:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4291.747459 secs 2024-04-26T13:02:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:33 smithi033 ceph-mon[27845]: pgmap v2946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4294.484237 secs 2024-04-26T13:02:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4291.747459 secs 2024-04-26T13:02:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:33 smithi096 ceph-mon[32753]: pgmap v2946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4294.484237 secs 2024-04-26T13:02:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:35.790+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:35.790+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:35.790+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:02:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:35 smithi194 ceph-mon[33017]: pgmap v2947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:35 smithi033 ceph-mon[27845]: pgmap v2947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:35 smithi096 ceph-mon[32753]: pgmap v2947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:37 smithi194 ceph-mon[33017]: pgmap v2948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:37 smithi033 ceph-mon[27845]: pgmap v2948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:37 smithi096 ceph-mon[32753]: pgmap v2948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4296.747549 secs 2024-04-26T13:02:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4299.484355 secs 2024-04-26T13:02:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4296.747549 secs 2024-04-26T13:02:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4299.484355 secs 2024-04-26T13:02:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4296.747549 secs 2024-04-26T13:02:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4299.484355 secs 2024-04-26T13:02:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:39 smithi194 ceph-mon[33017]: pgmap v2949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:39 smithi033 ceph-mon[27845]: pgmap v2949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:39 smithi096 ceph-mon[32753]: pgmap v2949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:41 smithi194 ceph-mon[33017]: pgmap v2950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:41 smithi033 ceph-mon[27845]: pgmap v2950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:41 smithi096 ceph-mon[32753]: pgmap v2950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4301.747634 secs 2024-04-26T13:02:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:43 smithi194 ceph-mon[33017]: pgmap v2951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4304.484476 secs 2024-04-26T13:02:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4301.747634 secs 2024-04-26T13:02:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:43 smithi033 ceph-mon[27845]: pgmap v2951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4304.484476 secs 2024-04-26T13:02:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4301.747634 secs 2024-04-26T13:02:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:43 smithi096 ceph-mon[32753]: pgmap v2951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4304.484476 secs 2024-04-26T13:02:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:45 smithi033 ceph-mon[27845]: pgmap v2952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:45 smithi096 ceph-mon[32753]: pgmap v2952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:45 smithi194 ceph-mon[33017]: pgmap v2952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:47 smithi033 ceph-mon[27845]: pgmap v2953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:47 smithi096 ceph-mon[32753]: pgmap v2953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:47 smithi194 ceph-mon[33017]: pgmap v2953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4306.747721 secs 2024-04-26T13:02:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4309.484613 secs 2024-04-26T13:02:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4306.747721 secs 2024-04-26T13:02:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4309.484613 secs 2024-04-26T13:02:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4306.747721 secs 2024-04-26T13:02:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4309.484613 secs 2024-04-26T13:02:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:49 smithi033 ceph-mon[27845]: pgmap v2954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:49 smithi096 ceph-mon[32753]: pgmap v2954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:49 smithi194 ceph-mon[33017]: pgmap v2954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:50.793+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:50.793+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:02:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:02:50.793+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:02:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:51 smithi033 ceph-mon[27845]: pgmap v2955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:51 smithi096 ceph-mon[32753]: pgmap v2955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:51 smithi194 ceph-mon[33017]: pgmap v2955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4311.747806 secs 2024-04-26T13:02:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:53 smithi033 ceph-mon[27845]: pgmap v2956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4314.484726 secs 2024-04-26T13:02:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4311.747806 secs 2024-04-26T13:02:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:53 smithi096 ceph-mon[32753]: pgmap v2956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4314.484726 secs 2024-04-26T13:02:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4311.747806 secs 2024-04-26T13:02:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:53 smithi194 ceph-mon[33017]: pgmap v2956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4314.484726 secs 2024-04-26T13:02:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:55 smithi033 ceph-mon[27845]: pgmap v2957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:55 smithi096 ceph-mon[32753]: pgmap v2957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:55 smithi194 ceph-mon[33017]: pgmap v2957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:02:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:57 smithi033 ceph-mon[27845]: pgmap v2958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:57 smithi096 ceph-mon[32753]: pgmap v2958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:57 smithi194 ceph-mon[33017]: pgmap v2958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:02:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4316.747889 secs 2024-04-26T13:02:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4319.484852 secs 2024-04-26T13:02:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4316.747889 secs 2024-04-26T13:02:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4319.484852 secs 2024-04-26T13:02:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4316.747889 secs 2024-04-26T13:02:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4319.484852 secs 2024-04-26T13:03:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:02:59 smithi033 ceph-mon[27845]: pgmap v2959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:02:59 smithi096 ceph-mon[32753]: pgmap v2959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:02:59 smithi194 ceph-mon[33017]: pgmap v2959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:01 smithi033 ceph-mon[27845]: pgmap v2960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:01 smithi096 ceph-mon[32753]: pgmap v2960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:01 smithi194 ceph-mon[33017]: pgmap v2960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4321.747977 secs 2024-04-26T13:03:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:03 smithi033 ceph-mon[27845]: pgmap v2961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4324.484952 secs 2024-04-26T13:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4321.747977 secs 2024-04-26T13:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:03 smithi096 ceph-mon[32753]: pgmap v2961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4324.484952 secs 2024-04-26T13:03:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4321.747977 secs 2024-04-26T13:03:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:03 smithi194 ceph-mon[33017]: pgmap v2961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4324.484952 secs 2024-04-26T13:03:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:05.801+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:05.801+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:05.801+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:03:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:05 smithi033 ceph-mon[27845]: pgmap v2962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:05 smithi096 ceph-mon[32753]: pgmap v2962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:05 smithi194 ceph-mon[33017]: pgmap v2962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:07 smithi096 ceph-mon[32753]: pgmap v2963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:03:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:07 smithi194 ceph-mon[33017]: pgmap v2963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:03:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:07 smithi033 ceph-mon[27845]: pgmap v2963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:03:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4326.748047 secs 2024-04-26T13:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4329.485068 secs 2024-04-26T13:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:03:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4326.748047 secs 2024-04-26T13:03:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4329.485068 secs 2024-04-26T13:03:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:03:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:03:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:03:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4326.748047 secs 2024-04-26T13:03:09.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4329.485068 secs 2024-04-26T13:03:09.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:03:09.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:03:09.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:03:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:09 smithi194 ceph-mon[33017]: pgmap v2964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:09 smithi194 ceph-mon[33017]: pgmap v2965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:09 smithi033 ceph-mon[27845]: pgmap v2964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:09 smithi033 ceph-mon[27845]: pgmap v2965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:09 smithi096 ceph-mon[32753]: pgmap v2964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:09 smithi096 ceph-mon[32753]: pgmap v2965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:11 smithi194 ceph-mon[33017]: pgmap v2966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:11 smithi033 ceph-mon[27845]: pgmap v2966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:11 smithi096 ceph-mon[32753]: pgmap v2966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:13 smithi194 ceph-mon[33017]: pgmap v2967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4331.748131 secs 2024-04-26T13:03:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4334.485186 secs 2024-04-26T13:03:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:13 smithi033 ceph-mon[27845]: pgmap v2967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4331.748131 secs 2024-04-26T13:03:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4334.485186 secs 2024-04-26T13:03:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:13 smithi096 ceph-mon[32753]: pgmap v2967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4331.748131 secs 2024-04-26T13:03:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4334.485186 secs 2024-04-26T13:03:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:15 smithi194 ceph-mon[33017]: pgmap v2968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:15 smithi033 ceph-mon[27845]: pgmap v2968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:15 smithi096 ceph-mon[32753]: pgmap v2968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:18.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:17 smithi033 ceph-mon[27845]: pgmap v2969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:17 smithi194 ceph-mon[33017]: pgmap v2969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:17 smithi096 ceph-mon[32753]: pgmap v2969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:03:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4336.748194 secs 2024-04-26T13:03:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4339.485305 secs 2024-04-26T13:03:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4336.748194 secs 2024-04-26T13:03:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4339.485305 secs 2024-04-26T13:03:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4336.748194 secs 2024-04-26T13:03:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4339.485305 secs 2024-04-26T13:03:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:19 smithi194 ceph-mon[33017]: pgmap v2970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:03:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:19 smithi033 ceph-mon[27845]: pgmap v2970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:03:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:19 smithi096 ceph-mon[32753]: pgmap v2970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:03:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:20.815+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:20.815+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:20.815+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:03:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:20 smithi194 ceph-mon[33017]: pgmap v2971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:20 smithi033 ceph-mon[27845]: pgmap v2971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:20 smithi096 ceph-mon[32753]: pgmap v2971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:23 smithi033 ceph-mon[27845]: pgmap v2972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4341.748288 secs 2024-04-26T13:03:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:23 smithi096 ceph-mon[32753]: pgmap v2972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4341.748288 secs 2024-04-26T13:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:23 smithi194 ceph-mon[33017]: pgmap v2972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4341.748288 secs 2024-04-26T13:03:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4344.485423 secs 2024-04-26T13:03:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4344.485423 secs 2024-04-26T13:03:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4344.485423 secs 2024-04-26T13:03:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:25 smithi033 ceph-mon[27845]: pgmap v2973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:25 smithi096 ceph-mon[32753]: pgmap v2973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:25 smithi194 ceph-mon[33017]: pgmap v2973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:27 smithi033 ceph-mon[27845]: pgmap v2974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:27 smithi096 ceph-mon[32753]: pgmap v2974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:27 smithi194 ceph-mon[33017]: pgmap v2974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4346.748389 secs 2024-04-26T13:03:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4346.748389 secs 2024-04-26T13:03:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4346.748389 secs 2024-04-26T13:03:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4349.485539 secs 2024-04-26T13:03:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:29 smithi033 ceph-mon[27845]: pgmap v2975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4349.485539 secs 2024-04-26T13:03:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:29 smithi096 ceph-mon[32753]: pgmap v2975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4349.485539 secs 2024-04-26T13:03:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:29 smithi194 ceph-mon[33017]: pgmap v2975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:31 smithi033 ceph-mon[27845]: pgmap v2976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:31 smithi096 ceph-mon[32753]: pgmap v2976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:31 smithi194 ceph-mon[33017]: pgmap v2976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:33 smithi033 ceph-mon[27845]: pgmap v2977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4351.748494 secs 2024-04-26T13:03:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:33 smithi096 ceph-mon[32753]: pgmap v2977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4351.748494 secs 2024-04-26T13:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:33 smithi194 ceph-mon[33017]: pgmap v2977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4351.748494 secs 2024-04-26T13:03:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4354.485641 secs 2024-04-26T13:03:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4354.485641 secs 2024-04-26T13:03:34.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4354.485641 secs 2024-04-26T13:03:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:35 smithi033 ceph-mon[27845]: pgmap v2978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:35 smithi096 ceph-mon[32753]: pgmap v2978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:35.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:35.818+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:35.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:35.818+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:35.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:35.819+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:03:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:35 smithi194 ceph-mon[33017]: pgmap v2978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:37 smithi033 ceph-mon[27845]: pgmap v2979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:37 smithi096 ceph-mon[32753]: pgmap v2979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:37 smithi194 ceph-mon[33017]: pgmap v2979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4356.748596 secs 2024-04-26T13:03:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4356.748596 secs 2024-04-26T13:03:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4356.748596 secs 2024-04-26T13:03:39.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4359.485741 secs 2024-04-26T13:03:39.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:39 smithi096 ceph-mon[32753]: pgmap v2980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4359.485741 secs 2024-04-26T13:03:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:39 smithi194 ceph-mon[33017]: pgmap v2980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4359.485741 secs 2024-04-26T13:03:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:39 smithi033 ceph-mon[27845]: pgmap v2980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:41 smithi194 ceph-mon[33017]: pgmap v2981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:41 smithi033 ceph-mon[27845]: pgmap v2981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:41 smithi096 ceph-mon[32753]: pgmap v2981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:43 smithi194 ceph-mon[33017]: pgmap v2982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4361.748696 secs 2024-04-26T13:03:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:43 smithi033 ceph-mon[27845]: pgmap v2982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4361.748696 secs 2024-04-26T13:03:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:43 smithi096 ceph-mon[32753]: pgmap v2982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4361.748696 secs 2024-04-26T13:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4364.485857 secs 2024-04-26T13:03:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4364.485857 secs 2024-04-26T13:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4364.485857 secs 2024-04-26T13:03:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:45 smithi194 ceph-mon[33017]: pgmap v2983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:45 smithi033 ceph-mon[27845]: pgmap v2983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:45 smithi096 ceph-mon[32753]: pgmap v2983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:47 smithi194 ceph-mon[33017]: pgmap v2984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:47 smithi033 ceph-mon[27845]: pgmap v2984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:47 smithi096 ceph-mon[32753]: pgmap v2984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4366.748794 secs 2024-04-26T13:03:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4366.748794 secs 2024-04-26T13:03:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4366.748794 secs 2024-04-26T13:03:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4369.485973 secs 2024-04-26T13:03:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:49 smithi194 ceph-mon[33017]: pgmap v2985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4369.485973 secs 2024-04-26T13:03:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:49 smithi033 ceph-mon[27845]: pgmap v2985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4369.485973 secs 2024-04-26T13:03:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:49 smithi096 ceph-mon[32753]: pgmap v2985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:50.834+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:50.834+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:03:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:50.834+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:03:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:51 smithi194 ceph-mon[33017]: pgmap v2986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:51 smithi033 ceph-mon[27845]: pgmap v2986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:51 smithi096 ceph-mon[32753]: pgmap v2986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:52.533+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:03:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:52.533+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:03:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:52.534+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:03:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:03:52.534+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:03:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:53 smithi194 ceph-mon[33017]: pgmap v2987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4371.748857 secs 2024-04-26T13:03:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:53 smithi033 ceph-mon[27845]: pgmap v2987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4371.748857 secs 2024-04-26T13:03:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:53 smithi096 ceph-mon[32753]: pgmap v2987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4371.748857 secs 2024-04-26T13:03:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4374.486122 secs 2024-04-26T13:03:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4374.486122 secs 2024-04-26T13:03:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4374.486122 secs 2024-04-26T13:03:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:55 smithi194 ceph-mon[33017]: pgmap v2988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:55 smithi033 ceph-mon[27845]: pgmap v2988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:55 smithi096 ceph-mon[32753]: pgmap v2988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:03:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:57 smithi194 ceph-mon[33017]: pgmap v2989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:57 smithi033 ceph-mon[27845]: pgmap v2989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:57 smithi096 ceph-mon[32753]: pgmap v2989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4376.748942 secs 2024-04-26T13:03:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4376.748942 secs 2024-04-26T13:03:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4376.748942 secs 2024-04-26T13:03:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4379.486212 secs 2024-04-26T13:03:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:03:59 smithi194 ceph-mon[33017]: pgmap v2990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4379.486212 secs 2024-04-26T13:03:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:03:59 smithi033 ceph-mon[27845]: pgmap v2990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:03:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4379.486212 secs 2024-04-26T13:03:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:03:59 smithi096 ceph-mon[32753]: pgmap v2990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:01 smithi194 ceph-mon[33017]: pgmap v2991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:01 smithi033 ceph-mon[27845]: pgmap v2991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:01.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:01 smithi096 ceph-mon[32753]: pgmap v2991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:03 smithi194 ceph-mon[33017]: pgmap v2992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4381.749029 secs 2024-04-26T13:04:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:03 smithi033 ceph-mon[27845]: pgmap v2992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4381.749029 secs 2024-04-26T13:04:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:03 smithi096 ceph-mon[32753]: pgmap v2992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4381.749029 secs 2024-04-26T13:04:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4384.486338 secs 2024-04-26T13:04:04.912 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4384.486338 secs 2024-04-26T13:04:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4384.486338 secs 2024-04-26T13:04:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:05.850+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:05.850+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:05.850+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:04:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:05 smithi194 ceph-mon[33017]: pgmap v2993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:05 smithi033 ceph-mon[27845]: pgmap v2993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:05 smithi096 ceph-mon[32753]: pgmap v2993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:07 smithi194 ceph-mon[33017]: pgmap v2994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:07 smithi033 ceph-mon[27845]: pgmap v2994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:07 smithi096 ceph-mon[32753]: pgmap v2994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4386.749093 secs 2024-04-26T13:04:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4386.749093 secs 2024-04-26T13:04:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4386.749093 secs 2024-04-26T13:04:09.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4389.486456 secs 2024-04-26T13:04:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:09 smithi096 ceph-mon[32753]: pgmap v2995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:04:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4389.486456 secs 2024-04-26T13:04:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:09 smithi194 ceph-mon[33017]: pgmap v2995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:04:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.901 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4389.486456 secs 2024-04-26T13:04:09.901 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:09 smithi033 ceph-mon[27845]: pgmap v2995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:09.901 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:04:09.902 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.902 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.902 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:09.902 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:11 smithi194 ceph-mon[33017]: pgmap v2996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:11 smithi033 ceph-mon[27845]: pgmap v2996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:11 smithi096 ceph-mon[32753]: pgmap v2996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:12.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:04:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:04:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:04:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:04:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: pgmap v2997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4391.749176 secs 2024-04-26T13:04:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:13.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: pgmap v2997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4391.749176 secs 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:04:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:04:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:04:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:04:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: pgmap v2997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:04:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4391.749176 secs 2024-04-26T13:04:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4394.486574 secs 2024-04-26T13:04:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4394.486574 secs 2024-04-26T13:04:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4394.486574 secs 2024-04-26T13:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: pgmap v2998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:04:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:04:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:04:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:04:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: pgmap v2999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:04:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: pgmap v3000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:15.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: pgmap v2998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: pgmap v2999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: pgmap v3000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: pgmap v2998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: pgmap v2999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: pgmap v3000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:04:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:17 smithi194 ceph-mon[33017]: pgmap v3001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:17 smithi033 ceph-mon[27845]: pgmap v3001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:17 smithi096 ceph-mon[32753]: pgmap v3001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4396.749239 secs 2024-04-26T13:04:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4396.749239 secs 2024-04-26T13:04:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4396.749239 secs 2024-04-26T13:04:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4399.486690 secs 2024-04-26T13:04:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:19 smithi194 ceph-mon[33017]: pgmap v3002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:04:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4399.486690 secs 2024-04-26T13:04:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:19 smithi033 ceph-mon[27845]: pgmap v3002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:04:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4399.486690 secs 2024-04-26T13:04:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:19 smithi096 ceph-mon[32753]: pgmap v3002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:04:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:20.860+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:20.860+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:20.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:20.860+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:04:22.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:21 smithi194 ceph-mon[33017]: pgmap v3003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:21 smithi033 ceph-mon[27845]: pgmap v3003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:21 smithi096 ceph-mon[32753]: pgmap v3003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:04:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4401.749327 secs 2024-04-26T13:04:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:23 smithi194 ceph-mon[33017]: pgmap v3004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:04:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4401.749327 secs 2024-04-26T13:04:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:23 smithi033 ceph-mon[27845]: pgmap v3004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:04:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4401.749327 secs 2024-04-26T13:04:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:23 smithi096 ceph-mon[32753]: pgmap v3004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:04:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4404.486806 secs 2024-04-26T13:04:25.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4404.486806 secs 2024-04-26T13:04:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4404.486806 secs 2024-04-26T13:04:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:25 smithi194 ceph-mon[33017]: pgmap v3005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:04:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:25 smithi033 ceph-mon[27845]: pgmap v3005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:04:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:25 smithi096 ceph-mon[32753]: pgmap v3005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:04:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:27 smithi194 ceph-mon[33017]: pgmap v3006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:27 smithi033 ceph-mon[27845]: pgmap v3006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:27 smithi096 ceph-mon[32753]: pgmap v3006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4406.749388 secs 2024-04-26T13:04:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4406.749388 secs 2024-04-26T13:04:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4406.749388 secs 2024-04-26T13:04:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4409.486927 secs 2024-04-26T13:04:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:29 smithi194 ceph-mon[33017]: pgmap v3007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4409.486927 secs 2024-04-26T13:04:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:29 smithi033 ceph-mon[27845]: pgmap v3007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4409.486927 secs 2024-04-26T13:04:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:29 smithi096 ceph-mon[32753]: pgmap v3007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:31 smithi194 ceph-mon[33017]: pgmap v3008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:31 smithi033 ceph-mon[27845]: pgmap v3008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:31 smithi096 ceph-mon[32753]: pgmap v3008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4411.749475 secs 2024-04-26T13:04:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:33 smithi194 ceph-mon[33017]: pgmap v3009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4411.749475 secs 2024-04-26T13:04:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:33 smithi033 ceph-mon[27845]: pgmap v3009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4411.749475 secs 2024-04-26T13:04:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:33 smithi096 ceph-mon[32753]: pgmap v3009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:35.020 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4414.487050 secs 2024-04-26T13:04:35.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4414.487050 secs 2024-04-26T13:04:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4414.487050 secs 2024-04-26T13:04:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:35.876+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:35.876+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:35.876+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:04:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:35 smithi194 ceph-mon[33017]: pgmap v3010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:36.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:35 smithi033 ceph-mon[27845]: pgmap v3010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:35 smithi096 ceph-mon[32753]: pgmap v3010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:37 smithi194 ceph-mon[33017]: pgmap v3011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:37 smithi033 ceph-mon[27845]: pgmap v3011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:37 smithi096 ceph-mon[32753]: pgmap v3011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:39.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4416.749539 secs 2024-04-26T13:04:39.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4419.487193 secs 2024-04-26T13:04:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4416.749539 secs 2024-04-26T13:04:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4419.487193 secs 2024-04-26T13:04:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4416.749539 secs 2024-04-26T13:04:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4419.487193 secs 2024-04-26T13:04:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:39 smithi194 ceph-mon[33017]: pgmap v3012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:39 smithi033 ceph-mon[27845]: pgmap v3012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:39 smithi096 ceph-mon[32753]: pgmap v3012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:41 smithi194 ceph-mon[33017]: pgmap v3013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:41 smithi033 ceph-mon[27845]: pgmap v3013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:41 smithi096 ceph-mon[32753]: pgmap v3013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4421.749621 secs 2024-04-26T13:04:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:43 smithi194 ceph-mon[33017]: pgmap v3014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4424.487259 secs 2024-04-26T13:04:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4421.749621 secs 2024-04-26T13:04:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:43 smithi033 ceph-mon[27845]: pgmap v3014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4424.487259 secs 2024-04-26T13:04:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4421.749621 secs 2024-04-26T13:04:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:43 smithi096 ceph-mon[32753]: pgmap v3014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4424.487259 secs 2024-04-26T13:04:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:45 smithi194 ceph-mon[33017]: pgmap v3015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:45 smithi033 ceph-mon[27845]: pgmap v3015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:45 smithi096 ceph-mon[32753]: pgmap v3015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:47 smithi194 ceph-mon[33017]: pgmap v3016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:47 smithi033 ceph-mon[27845]: pgmap v3016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:47 smithi096 ceph-mon[32753]: pgmap v3016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4426.749707 secs 2024-04-26T13:04:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4429.487371 secs 2024-04-26T13:04:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4426.749707 secs 2024-04-26T13:04:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4429.487371 secs 2024-04-26T13:04:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4426.749707 secs 2024-04-26T13:04:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4429.487371 secs 2024-04-26T13:04:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:49 smithi194 ceph-mon[33017]: pgmap v3017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:49 smithi033 ceph-mon[27845]: pgmap v3017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:49 smithi096 ceph-mon[32753]: pgmap v3017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:50.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:50.881+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:50.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:50.881+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:04:50.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:04:50.881+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:04:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:51 smithi194 ceph-mon[33017]: pgmap v3018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:51 smithi033 ceph-mon[27845]: pgmap v3018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:51 smithi096 ceph-mon[32753]: pgmap v3018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4431.749792 secs 2024-04-26T13:04:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:53 smithi194 ceph-mon[33017]: pgmap v3019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4434.487450 secs 2024-04-26T13:04:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4431.749792 secs 2024-04-26T13:04:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:53 smithi033 ceph-mon[27845]: pgmap v3019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4434.487450 secs 2024-04-26T13:04:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4431.749792 secs 2024-04-26T13:04:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:53 smithi096 ceph-mon[32753]: pgmap v3019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4434.487450 secs 2024-04-26T13:04:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:55 smithi194 ceph-mon[33017]: pgmap v3020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:55 smithi033 ceph-mon[27845]: pgmap v3020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:55 smithi096 ceph-mon[32753]: pgmap v3020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:04:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:57 smithi194 ceph-mon[33017]: pgmap v3021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:57 smithi033 ceph-mon[27845]: pgmap v3021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:57 smithi096 ceph-mon[32753]: pgmap v3021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:04:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4436.749879 secs 2024-04-26T13:04:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4439.487570 secs 2024-04-26T13:04:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4436.749879 secs 2024-04-26T13:04:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4439.487570 secs 2024-04-26T13:04:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4436.749879 secs 2024-04-26T13:04:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4439.487570 secs 2024-04-26T13:05:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:04:59 smithi194 ceph-mon[33017]: pgmap v3022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:04:59 smithi033 ceph-mon[27845]: pgmap v3022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:04:59 smithi096 ceph-mon[32753]: pgmap v3022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:01 smithi194 ceph-mon[33017]: pgmap v3023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:01 smithi033 ceph-mon[27845]: pgmap v3023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:01 smithi096 ceph-mon[32753]: pgmap v3023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4441.749991 secs 2024-04-26T13:05:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:03 smithi033 ceph-mon[27845]: pgmap v3024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4444.487684 secs 2024-04-26T13:05:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4441.749991 secs 2024-04-26T13:05:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:03 smithi096 ceph-mon[32753]: pgmap v3024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4444.487684 secs 2024-04-26T13:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4441.749991 secs 2024-04-26T13:05:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:03 smithi194 ceph-mon[33017]: pgmap v3024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4444.487684 secs 2024-04-26T13:05:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:05.896+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:05.896+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:05.896+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:05:06.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:05 smithi194 ceph-mon[33017]: pgmap v3025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:05 smithi033 ceph-mon[27845]: pgmap v3025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:05 smithi096 ceph-mon[32753]: pgmap v3025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:07 smithi033 ceph-mon[27845]: pgmap v3026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:07 smithi096 ceph-mon[32753]: pgmap v3026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:07 smithi194 ceph-mon[33017]: pgmap v3026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4446.750076 secs 2024-04-26T13:05:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4449.487763 secs 2024-04-26T13:05:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4446.750076 secs 2024-04-26T13:05:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4449.487763 secs 2024-04-26T13:05:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4446.750076 secs 2024-04-26T13:05:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4449.487763 secs 2024-04-26T13:05:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:09 smithi033 ceph-mon[27845]: pgmap v3027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:09 smithi096 ceph-mon[32753]: pgmap v3027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:09 smithi194 ceph-mon[33017]: pgmap v3027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:11 smithi033 ceph-mon[27845]: pgmap v3028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:11 smithi096 ceph-mon[32753]: pgmap v3028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:11 smithi194 ceph-mon[33017]: pgmap v3028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4451.750164 secs 2024-04-26T13:05:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:13 smithi033 ceph-mon[27845]: pgmap v3029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4454.487912 secs 2024-04-26T13:05:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4451.750164 secs 2024-04-26T13:05:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:13 smithi096 ceph-mon[32753]: pgmap v3029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4454.487912 secs 2024-04-26T13:05:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4451.750164 secs 2024-04-26T13:05:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:13 smithi194 ceph-mon[33017]: pgmap v3029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4454.487912 secs 2024-04-26T13:05:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:15 smithi033 ceph-mon[27845]: pgmap v3030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:05:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:15 smithi096 ceph-mon[32753]: pgmap v3030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:05:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:15 smithi194 ceph-mon[33017]: pgmap v3030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:05:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:05:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:05:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:05:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:05:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:05:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:05:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:05:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:05:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:05:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:17 smithi033 ceph-mon[27845]: pgmap v3031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:17 smithi033 ceph-mon[27845]: pgmap v3032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:17 smithi096 ceph-mon[32753]: pgmap v3031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:17 smithi096 ceph-mon[32753]: pgmap v3032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:17 smithi194 ceph-mon[33017]: pgmap v3031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:17 smithi194 ceph-mon[33017]: pgmap v3032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4456.750224 secs 2024-04-26T13:05:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4459.488047 secs 2024-04-26T13:05:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4456.750224 secs 2024-04-26T13:05:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4459.488047 secs 2024-04-26T13:05:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4456.750224 secs 2024-04-26T13:05:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4459.488047 secs 2024-04-26T13:05:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:19 smithi033 ceph-mon[27845]: pgmap v3033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:19 smithi096 ceph-mon[32753]: pgmap v3033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:19 smithi194 ceph-mon[33017]: pgmap v3033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:20.908+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:20.908+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:20.908+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:05:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:21 smithi096 ceph-mon[32753]: pgmap v3034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:22.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:21 smithi033 ceph-mon[27845]: pgmap v3034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:21 smithi194 ceph-mon[33017]: pgmap v3034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:22.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:22.549+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:05:22.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:22.549+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:05:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:22.549+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:05:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:22.549+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:05:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:23 smithi096 ceph-mon[32753]: pgmap v3035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4461.750309 secs 2024-04-26T13:05:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4464.488160 secs 2024-04-26T13:05:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:23 smithi194 ceph-mon[33017]: pgmap v3035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4461.750309 secs 2024-04-26T13:05:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4464.488160 secs 2024-04-26T13:05:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:23 smithi033 ceph-mon[27845]: pgmap v3035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4461.750309 secs 2024-04-26T13:05:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4464.488160 secs 2024-04-26T13:05:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:25 smithi194 ceph-mon[33017]: pgmap v3036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:25 smithi033 ceph-mon[27845]: pgmap v3036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:25 smithi096 ceph-mon[32753]: pgmap v3036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:05:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:27 smithi194 ceph-mon[33017]: pgmap v3037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:05:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:27 smithi033 ceph-mon[27845]: pgmap v3037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:05:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:27 smithi096 ceph-mon[32753]: pgmap v3037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:05:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4466.750400 secs 2024-04-26T13:05:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4469.488283 secs 2024-04-26T13:05:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4466.750400 secs 2024-04-26T13:05:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4469.488283 secs 2024-04-26T13:05:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4466.750400 secs 2024-04-26T13:05:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4469.488283 secs 2024-04-26T13:05:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:29 smithi194 ceph-mon[33017]: pgmap v3038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:29 smithi033 ceph-mon[27845]: pgmap v3038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:29 smithi096 ceph-mon[32753]: pgmap v3038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:31 smithi194 ceph-mon[33017]: pgmap v3039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:31 smithi033 ceph-mon[27845]: pgmap v3039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:31 smithi096 ceph-mon[32753]: pgmap v3039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:33 smithi194 ceph-mon[33017]: pgmap v3040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4471.750481 secs 2024-04-26T13:05:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4474.488400 secs 2024-04-26T13:05:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:33 smithi033 ceph-mon[27845]: pgmap v3040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4471.750481 secs 2024-04-26T13:05:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4474.488400 secs 2024-04-26T13:05:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:33 smithi096 ceph-mon[32753]: pgmap v3040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4471.750481 secs 2024-04-26T13:05:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4474.488400 secs 2024-04-26T13:05:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:35.921+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:35.921+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:35.921+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:05:36.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:35 smithi033 ceph-mon[27845]: pgmap v3041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:35 smithi194 ceph-mon[33017]: pgmap v3041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:35 smithi096 ceph-mon[32753]: pgmap v3041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:37 smithi194 ceph-mon[33017]: pgmap v3042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:37 smithi033 ceph-mon[27845]: pgmap v3042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:37 smithi096 ceph-mon[32753]: pgmap v3042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:39.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4476.750545 secs 2024-04-26T13:05:39.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4479.488519 secs 2024-04-26T13:05:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4476.750545 secs 2024-04-26T13:05:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4479.488519 secs 2024-04-26T13:05:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4476.750545 secs 2024-04-26T13:05:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4479.488519 secs 2024-04-26T13:05:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:39 smithi194 ceph-mon[33017]: pgmap v3043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:39 smithi033 ceph-mon[27845]: pgmap v3043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:39 smithi096 ceph-mon[32753]: pgmap v3043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:41 smithi194 ceph-mon[33017]: pgmap v3044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:41 smithi033 ceph-mon[27845]: pgmap v3044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:41 smithi096 ceph-mon[32753]: pgmap v3044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:05:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:43 smithi194 ceph-mon[33017]: pgmap v3045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4481.750634 secs 2024-04-26T13:05:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:43 smithi033 ceph-mon[27845]: pgmap v3045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4481.750634 secs 2024-04-26T13:05:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:43 smithi096 ceph-mon[32753]: pgmap v3045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4481.750634 secs 2024-04-26T13:05:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4484.488639 secs 2024-04-26T13:05:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4484.488639 secs 2024-04-26T13:05:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4484.488639 secs 2024-04-26T13:05:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:45 smithi194 ceph-mon[33017]: pgmap v3046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:45 smithi033 ceph-mon[27845]: pgmap v3046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:45 smithi096 ceph-mon[32753]: pgmap v3046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:47 smithi096 ceph-mon[32753]: pgmap v3047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:47 smithi194 ceph-mon[33017]: pgmap v3047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:47 smithi033 ceph-mon[27845]: pgmap v3047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4486.750720 secs 2024-04-26T13:05:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4486.750720 secs 2024-04-26T13:05:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4486.750720 secs 2024-04-26T13:05:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4489.488752 secs 2024-04-26T13:05:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:49 smithi194 ceph-mon[33017]: pgmap v3048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4489.488752 secs 2024-04-26T13:05:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:49 smithi033 ceph-mon[27845]: pgmap v3048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4489.488752 secs 2024-04-26T13:05:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:49 smithi096 ceph-mon[32753]: pgmap v3048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:50.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:50.936+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:50.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:50.936+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:05:50.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:05:50.936+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:05:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:51 smithi194 ceph-mon[33017]: pgmap v3049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:51 smithi033 ceph-mon[27845]: pgmap v3049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:51 smithi096 ceph-mon[32753]: pgmap v3049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:53 smithi194 ceph-mon[33017]: pgmap v3050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4491.750805 secs 2024-04-26T13:05:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:53 smithi033 ceph-mon[27845]: pgmap v3050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4491.750805 secs 2024-04-26T13:05:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:53 smithi096 ceph-mon[32753]: pgmap v3050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:05:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4491.750805 secs 2024-04-26T13:05:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4494.488868 secs 2024-04-26T13:05:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4494.488868 secs 2024-04-26T13:05:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4494.488868 secs 2024-04-26T13:05:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:55 smithi194 ceph-mon[33017]: pgmap v3051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:55 smithi033 ceph-mon[27845]: pgmap v3051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:55 smithi096 ceph-mon[32753]: pgmap v3051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:05:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:57 smithi194 ceph-mon[33017]: pgmap v3052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:57 smithi033 ceph-mon[27845]: pgmap v3052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:57 smithi096 ceph-mon[32753]: pgmap v3052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4496.750890 secs 2024-04-26T13:05:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4496.750890 secs 2024-04-26T13:05:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4496.750890 secs 2024-04-26T13:05:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4499.489019 secs 2024-04-26T13:05:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:05:59 smithi194 ceph-mon[33017]: pgmap v3053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4499.489019 secs 2024-04-26T13:05:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:05:59 smithi033 ceph-mon[27845]: pgmap v3053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:05:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4499.489019 secs 2024-04-26T13:05:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:05:59 smithi096 ceph-mon[32753]: pgmap v3053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:01 smithi194 ceph-mon[33017]: pgmap v3054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:01 smithi033 ceph-mon[27845]: pgmap v3054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:01 smithi096 ceph-mon[32753]: pgmap v3054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:03 smithi194 ceph-mon[33017]: pgmap v3055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4501.750994 secs 2024-04-26T13:06:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:03 smithi033 ceph-mon[27845]: pgmap v3055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4501.750994 secs 2024-04-26T13:06:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:03 smithi096 ceph-mon[32753]: pgmap v3055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4501.750994 secs 2024-04-26T13:06:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4504.489152 secs 2024-04-26T13:06:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4504.489152 secs 2024-04-26T13:06:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4504.489152 secs 2024-04-26T13:06:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:05 smithi194 ceph-mon[33017]: pgmap v3056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:05 smithi033 ceph-mon[27845]: pgmap v3056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:05 smithi096 ceph-mon[32753]: pgmap v3056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:05.951+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:05.952+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:05.952+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:06:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:07 smithi194 ceph-mon[33017]: pgmap v3057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:07 smithi033 ceph-mon[27845]: pgmap v3057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:07 smithi096 ceph-mon[32753]: pgmap v3057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4506.751056 secs 2024-04-26T13:06:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4506.751056 secs 2024-04-26T13:06:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4506.751056 secs 2024-04-26T13:06:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4509.489269 secs 2024-04-26T13:06:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:09 smithi033 ceph-mon[27845]: pgmap v3058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4509.489269 secs 2024-04-26T13:06:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:09 smithi096 ceph-mon[32753]: pgmap v3058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4509.489269 secs 2024-04-26T13:06:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:09 smithi194 ceph-mon[33017]: pgmap v3058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:11 smithi194 ceph-mon[33017]: pgmap v3059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:11 smithi033 ceph-mon[27845]: pgmap v3059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:11 smithi096 ceph-mon[32753]: pgmap v3059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:13 smithi194 ceph-mon[33017]: pgmap v3060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4511.751145 secs 2024-04-26T13:06:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:13 smithi033 ceph-mon[27845]: pgmap v3060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4511.751145 secs 2024-04-26T13:06:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:13 smithi096 ceph-mon[32753]: pgmap v3060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4511.751145 secs 2024-04-26T13:06:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4514.489369 secs 2024-04-26T13:06:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4514.489369 secs 2024-04-26T13:06:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4514.489369 secs 2024-04-26T13:06:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:15 smithi194 ceph-mon[33017]: pgmap v3061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:15 smithi033 ceph-mon[27845]: pgmap v3061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:15 smithi096 ceph-mon[32753]: pgmap v3061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:17 smithi194 ceph-mon[33017]: pgmap v3062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:06:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:17 smithi033 ceph-mon[27845]: pgmap v3062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:06:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:17 smithi096 ceph-mon[32753]: pgmap v3062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4516.751228 secs 2024-04-26T13:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:06:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:06:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:06:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4516.751228 secs 2024-04-26T13:06:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:06:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:06:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:06:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4516.751228 secs 2024-04-26T13:06:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:06:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:06:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:06:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:19 smithi194 ceph-mon[33017]: pgmap v3063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:19 smithi194 ceph-mon[33017]: pgmap v3064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4519.489468 secs 2024-04-26T13:06:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:19 smithi033 ceph-mon[27845]: pgmap v3063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:19 smithi033 ceph-mon[27845]: pgmap v3064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4519.489468 secs 2024-04-26T13:06:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:19 smithi096 ceph-mon[32753]: pgmap v3063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:19 smithi096 ceph-mon[32753]: pgmap v3064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4519.489468 secs 2024-04-26T13:06:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:20.962+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:20.962+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:20.963+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:06:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:21 smithi194 ceph-mon[33017]: pgmap v3065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:21 smithi033 ceph-mon[27845]: pgmap v3065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:21 smithi096 ceph-mon[32753]: pgmap v3065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:23 smithi194 ceph-mon[33017]: pgmap v3066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4521.751288 secs 2024-04-26T13:06:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:23 smithi033 ceph-mon[27845]: pgmap v3066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4521.751288 secs 2024-04-26T13:06:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:23 smithi096 ceph-mon[32753]: pgmap v3066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4521.751288 secs 2024-04-26T13:06:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4524.489584 secs 2024-04-26T13:06:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4524.489584 secs 2024-04-26T13:06:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4524.489584 secs 2024-04-26T13:06:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:25 smithi194 ceph-mon[33017]: pgmap v3067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:06:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:25 smithi033 ceph-mon[27845]: pgmap v3067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:06:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:25 smithi096 ceph-mon[32753]: pgmap v3067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:06:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:27 smithi194 ceph-mon[33017]: pgmap v3068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:27 smithi033 ceph-mon[27845]: pgmap v3068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:27 smithi096 ceph-mon[32753]: pgmap v3068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:06:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4526.751346 secs 2024-04-26T13:06:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4526.751346 secs 2024-04-26T13:06:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4526.751346 secs 2024-04-26T13:06:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:29 smithi033 ceph-mon[27845]: pgmap v3069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:06:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4529.489717 secs 2024-04-26T13:06:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:29 smithi096 ceph-mon[32753]: pgmap v3069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:06:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4529.489717 secs 2024-04-26T13:06:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:29 smithi194 ceph-mon[33017]: pgmap v3069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:06:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4529.489717 secs 2024-04-26T13:06:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:31 smithi033 ceph-mon[27845]: pgmap v3070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:31 smithi096 ceph-mon[32753]: pgmap v3070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:31 smithi194 ceph-mon[33017]: pgmap v3070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:33 smithi033 ceph-mon[27845]: pgmap v3071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4531.751407 secs 2024-04-26T13:06:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:33 smithi096 ceph-mon[32753]: pgmap v3071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4531.751407 secs 2024-04-26T13:06:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:33 smithi194 ceph-mon[33017]: pgmap v3071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4531.751407 secs 2024-04-26T13:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4534.489838 secs 2024-04-26T13:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4534.489838 secs 2024-04-26T13:06:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4534.489838 secs 2024-04-26T13:06:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:35 smithi033 ceph-mon[27845]: pgmap v3072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:35 smithi096 ceph-mon[32753]: pgmap v3072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:35.968+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:35.968+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:35.968+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:06:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:35 smithi194 ceph-mon[33017]: pgmap v3072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:37.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:37 smithi194 ceph-mon[33017]: pgmap v3073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:37 smithi033 ceph-mon[27845]: pgmap v3073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:37 smithi096 ceph-mon[32753]: pgmap v3073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4536.751492 secs 2024-04-26T13:06:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4536.751492 secs 2024-04-26T13:06:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4536.751492 secs 2024-04-26T13:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:39 smithi033 ceph-mon[27845]: pgmap v3074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4539.489972 secs 2024-04-26T13:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:39 smithi096 ceph-mon[32753]: pgmap v3074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4539.489972 secs 2024-04-26T13:06:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:39 smithi194 ceph-mon[33017]: pgmap v3074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4539.489972 secs 2024-04-26T13:06:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:41 smithi033 ceph-mon[27845]: pgmap v3075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:41 smithi096 ceph-mon[32753]: pgmap v3075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:41 smithi194 ceph-mon[33017]: pgmap v3075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:43 smithi033 ceph-mon[27845]: pgmap v3076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4541.751577 secs 2024-04-26T13:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:43 smithi096 ceph-mon[32753]: pgmap v3076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4541.751577 secs 2024-04-26T13:06:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:43 smithi194 ceph-mon[33017]: pgmap v3076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4541.751577 secs 2024-04-26T13:06:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4544.490091 secs 2024-04-26T13:06:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4544.490091 secs 2024-04-26T13:06:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4544.490091 secs 2024-04-26T13:06:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:45 smithi096 ceph-mon[32753]: pgmap v3077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:45 smithi194 ceph-mon[33017]: pgmap v3077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:45 smithi033 ceph-mon[27845]: pgmap v3077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:47 smithi194 ceph-mon[33017]: pgmap v3078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:47 smithi033 ceph-mon[27845]: pgmap v3078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:47 smithi096 ceph-mon[32753]: pgmap v3078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4546.751638 secs 2024-04-26T13:06:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4546.751638 secs 2024-04-26T13:06:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4546.751638 secs 2024-04-26T13:06:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:49 smithi194 ceph-mon[33017]: pgmap v3079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4549.490208 secs 2024-04-26T13:06:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:49 smithi033 ceph-mon[27845]: pgmap v3079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4549.490208 secs 2024-04-26T13:06:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:49 smithi096 ceph-mon[32753]: pgmap v3079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4549.490208 secs 2024-04-26T13:06:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:50.978+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:50.978+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:06:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:06:50.978+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:06:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:51 smithi194 ceph-mon[33017]: pgmap v3080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:51 smithi033 ceph-mon[27845]: pgmap v3080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:51 smithi096 ceph-mon[32753]: pgmap v3080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:53 smithi194 ceph-mon[33017]: pgmap v3081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4551.751724 secs 2024-04-26T13:06:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:53 smithi033 ceph-mon[27845]: pgmap v3081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4551.751724 secs 2024-04-26T13:06:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:53 smithi096 ceph-mon[32753]: pgmap v3081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4551.751724 secs 2024-04-26T13:06:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4554.490326 secs 2024-04-26T13:06:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4554.490326 secs 2024-04-26T13:06:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4554.490326 secs 2024-04-26T13:06:56.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:55 smithi033 ceph-mon[27845]: pgmap v3082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:55 smithi194 ceph-mon[33017]: pgmap v3082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:55 smithi096 ceph-mon[32753]: pgmap v3082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:06:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:57 smithi194 ceph-mon[33017]: pgmap v3083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:57 smithi033 ceph-mon[27845]: pgmap v3083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:57 smithi096 ceph-mon[32753]: pgmap v3083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:06:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4556.751814 secs 2024-04-26T13:06:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4556.751814 secs 2024-04-26T13:06:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4556.751814 secs 2024-04-26T13:07:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:59 smithi194 ceph-mon[33017]: pgmap v3084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4559.490444 secs 2024-04-26T13:07:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:59 smithi033 ceph-mon[27845]: pgmap v3084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4559.490444 secs 2024-04-26T13:07:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:59 smithi096 ceph-mon[32753]: pgmap v3084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4559.490444 secs 2024-04-26T13:07:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:01 smithi194 ceph-mon[33017]: pgmap v3085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:01 smithi033 ceph-mon[27845]: pgmap v3085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:01 smithi096 ceph-mon[32753]: pgmap v3085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:03 smithi194 ceph-mon[33017]: pgmap v3086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4561.751898 secs 2024-04-26T13:07:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:03 smithi033 ceph-mon[27845]: pgmap v3086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4561.751898 secs 2024-04-26T13:07:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:03 smithi096 ceph-mon[32753]: pgmap v3086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4561.751898 secs 2024-04-26T13:07:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4564.490566 secs 2024-04-26T13:07:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4564.490566 secs 2024-04-26T13:07:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4564.490566 secs 2024-04-26T13:07:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:05.979+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:05.979+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:05.979+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:07:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:05 smithi194 ceph-mon[33017]: pgmap v3087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:05 smithi033 ceph-mon[27845]: pgmap v3087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:05 smithi096 ceph-mon[32753]: pgmap v3087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:07 smithi194 ceph-mon[33017]: pgmap v3088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:07 smithi033 ceph-mon[27845]: pgmap v3088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:07 smithi096 ceph-mon[32753]: pgmap v3088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:09.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4566.751996 secs 2024-04-26T13:07:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4566.751996 secs 2024-04-26T13:07:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4566.751996 secs 2024-04-26T13:07:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:09 smithi194 ceph-mon[33017]: pgmap v3089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4569.490682 secs 2024-04-26T13:07:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:09 smithi033 ceph-mon[27845]: pgmap v3089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4569.490682 secs 2024-04-26T13:07:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:09 smithi096 ceph-mon[32753]: pgmap v3089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4569.490682 secs 2024-04-26T13:07:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:11 smithi194 ceph-mon[33017]: pgmap v3090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:11 smithi033 ceph-mon[27845]: pgmap v3090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:11 smithi096 ceph-mon[32753]: pgmap v3090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:13 smithi194 ceph-mon[33017]: pgmap v3091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4571.752081 secs 2024-04-26T13:07:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4574.490800 secs 2024-04-26T13:07:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:13 smithi033 ceph-mon[27845]: pgmap v3091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4571.752081 secs 2024-04-26T13:07:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4574.490800 secs 2024-04-26T13:07:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:13 smithi096 ceph-mon[32753]: pgmap v3091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4571.752081 secs 2024-04-26T13:07:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4574.490800 secs 2024-04-26T13:07:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:15 smithi194 ceph-mon[33017]: pgmap v3092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:15 smithi033 ceph-mon[27845]: pgmap v3092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:15 smithi096 ceph-mon[32753]: pgmap v3092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:17 smithi194 ceph-mon[33017]: pgmap v3093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:17 smithi033 ceph-mon[27845]: pgmap v3093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:17 smithi096 ceph-mon[32753]: pgmap v3093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4576.752165 secs 2024-04-26T13:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4579.490923 secs 2024-04-26T13:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4576.752165 secs 2024-04-26T13:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4579.490923 secs 2024-04-26T13:07:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:07:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4576.752165 secs 2024-04-26T13:07:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4579.490923 secs 2024-04-26T13:07:19.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:07:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:19 smithi194 ceph-mon[33017]: pgmap v3094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:07:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:07:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:07:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:19 smithi033 ceph-mon[27845]: pgmap v3094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:07:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:07:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:07:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:19 smithi096 ceph-mon[32753]: pgmap v3094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:07:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:07:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:07:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:20.993+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:20.993+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:20.993+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:07:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:20 smithi194 ceph-mon[33017]: pgmap v3095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:07:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:20 smithi194 ceph-mon[33017]: pgmap v3096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:20 smithi033 ceph-mon[27845]: pgmap v3095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:07:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:20 smithi033 ceph-mon[27845]: pgmap v3096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:20 smithi096 ceph-mon[32753]: pgmap v3095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:07:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:20 smithi096 ceph-mon[32753]: pgmap v3096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:22 smithi194 ceph-mon[33017]: pgmap v3097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:22 smithi033 ceph-mon[27845]: pgmap v3097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:22 smithi096 ceph-mon[32753]: pgmap v3097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4581.752253 secs 2024-04-26T13:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4584.491038 secs 2024-04-26T13:07:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4581.752253 secs 2024-04-26T13:07:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4584.491038 secs 2024-04-26T13:07:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4581.752253 secs 2024-04-26T13:07:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4584.491038 secs 2024-04-26T13:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:24 smithi194 ceph-mon[33017]: pgmap v3098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:24 smithi033 ceph-mon[27845]: pgmap v3098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:24 smithi096 ceph-mon[32753]: pgmap v3098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:27.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:26 smithi033 ceph-mon[27845]: pgmap v3099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:26 smithi096 ceph-mon[32753]: pgmap v3099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:26 smithi194 ceph-mon[33017]: pgmap v3099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4586.752339 secs 2024-04-26T13:07:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:28 smithi033 ceph-mon[27845]: pgmap v3100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4589.491157 secs 2024-04-26T13:07:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4586.752339 secs 2024-04-26T13:07:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:28 smithi096 ceph-mon[32753]: pgmap v3100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4589.491157 secs 2024-04-26T13:07:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4586.752339 secs 2024-04-26T13:07:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:28 smithi194 ceph-mon[33017]: pgmap v3100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:07:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4589.491157 secs 2024-04-26T13:07:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:30 smithi033 ceph-mon[27845]: pgmap v3101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:07:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:30 smithi096 ceph-mon[32753]: pgmap v3101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:07:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:30 smithi194 ceph-mon[33017]: pgmap v3101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:07:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:32 smithi033 ceph-mon[27845]: pgmap v3102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:32 smithi096 ceph-mon[32753]: pgmap v3102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:32 smithi194 ceph-mon[33017]: pgmap v3102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4591.752422 secs 2024-04-26T13:07:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4594.491235 secs 2024-04-26T13:07:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4591.752422 secs 2024-04-26T13:07:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4594.491235 secs 2024-04-26T13:07:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4591.752422 secs 2024-04-26T13:07:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4594.491235 secs 2024-04-26T13:07:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:34 smithi033 ceph-mon[27845]: pgmap v3103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:34 smithi096 ceph-mon[32753]: pgmap v3103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:34 smithi194 ceph-mon[33017]: pgmap v3103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:36.009+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:36.009+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:36.009+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:07:37.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:36 smithi194 ceph-mon[33017]: pgmap v3104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:36 smithi033 ceph-mon[27845]: pgmap v3104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:36 smithi096 ceph-mon[32753]: pgmap v3104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4596.752505 secs 2024-04-26T13:07:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:38 smithi033 ceph-mon[27845]: pgmap v3105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4599.491333 secs 2024-04-26T13:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4596.752505 secs 2024-04-26T13:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:38 smithi096 ceph-mon[32753]: pgmap v3105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4599.491333 secs 2024-04-26T13:07:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4596.752505 secs 2024-04-26T13:07:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:38 smithi194 ceph-mon[33017]: pgmap v3105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4599.491333 secs 2024-04-26T13:07:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:40 smithi033 ceph-mon[27845]: pgmap v3106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:40 smithi096 ceph-mon[32753]: pgmap v3106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:40 smithi194 ceph-mon[33017]: pgmap v3106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:42 smithi033 ceph-mon[27845]: pgmap v3107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:42 smithi096 ceph-mon[32753]: pgmap v3107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:42 smithi194 ceph-mon[33017]: pgmap v3107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4601.752588 secs 2024-04-26T13:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4604.491453 secs 2024-04-26T13:07:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4601.752588 secs 2024-04-26T13:07:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4604.491453 secs 2024-04-26T13:07:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4601.752588 secs 2024-04-26T13:07:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4604.491453 secs 2024-04-26T13:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:44 smithi033 ceph-mon[27845]: pgmap v3108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:44 smithi096 ceph-mon[32753]: pgmap v3108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:44 smithi194 ceph-mon[33017]: pgmap v3108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:46 smithi033 ceph-mon[27845]: pgmap v3109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:46 smithi096 ceph-mon[32753]: pgmap v3109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:46 smithi194 ceph-mon[33017]: pgmap v3109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4606.752670 secs 2024-04-26T13:07:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:48 smithi096 ceph-mon[32753]: pgmap v3110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4609.491531 secs 2024-04-26T13:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4606.752670 secs 2024-04-26T13:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:48 smithi194 ceph-mon[33017]: pgmap v3110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4609.491531 secs 2024-04-26T13:07:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4606.752670 secs 2024-04-26T13:07:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:48 smithi033 ceph-mon[27845]: pgmap v3110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4609.491531 secs 2024-04-26T13:07:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:51.024+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:51.024+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:07:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:07:51.024+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:07:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:50 smithi194 ceph-mon[33017]: pgmap v3111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:50 smithi033 ceph-mon[27845]: pgmap v3111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:50 smithi096 ceph-mon[32753]: pgmap v3111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:52 smithi194 ceph-mon[33017]: pgmap v3112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:52 smithi033 ceph-mon[27845]: pgmap v3112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:52 smithi096 ceph-mon[32753]: pgmap v3112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4611.752754 secs 2024-04-26T13:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4614.491665 secs 2024-04-26T13:07:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4611.752754 secs 2024-04-26T13:07:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4614.491665 secs 2024-04-26T13:07:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4611.752754 secs 2024-04-26T13:07:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4614.491665 secs 2024-04-26T13:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:54 smithi194 ceph-mon[33017]: pgmap v3113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:54 smithi033 ceph-mon[27845]: pgmap v3113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:54 smithi096 ceph-mon[32753]: pgmap v3113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:56 smithi194 ceph-mon[33017]: pgmap v3114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:56 smithi033 ceph-mon[27845]: pgmap v3114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:56 smithi096 ceph-mon[32753]: pgmap v3114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4616.752813 secs 2024-04-26T13:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:58 smithi194 ceph-mon[33017]: pgmap v3115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:07:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4619.491785 secs 2024-04-26T13:07:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4616.752813 secs 2024-04-26T13:07:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:58 smithi033 ceph-mon[27845]: pgmap v3115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:07:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4619.491785 secs 2024-04-26T13:07:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4616.752813 secs 2024-04-26T13:07:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:58 smithi096 ceph-mon[32753]: pgmap v3115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:07:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:07:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4619.491785 secs 2024-04-26T13:08:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:00 smithi194 ceph-mon[33017]: pgmap v3116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:00 smithi033 ceph-mon[27845]: pgmap v3116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:00 smithi096 ceph-mon[32753]: pgmap v3116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:02 smithi194 ceph-mon[33017]: pgmap v3117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:02 smithi033 ceph-mon[27845]: pgmap v3117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:03.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:02 smithi096 ceph-mon[32753]: pgmap v3117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4621.752898 secs 2024-04-26T13:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4624.491902 secs 2024-04-26T13:08:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4621.752898 secs 2024-04-26T13:08:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4624.491902 secs 2024-04-26T13:08:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4621.752898 secs 2024-04-26T13:08:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4624.491902 secs 2024-04-26T13:08:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:04 smithi194 ceph-mon[33017]: pgmap v3118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:04 smithi033 ceph-mon[27845]: pgmap v3118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:04 smithi096 ceph-mon[32753]: pgmap v3118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:06.040+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:06.040+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:06.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:06.040+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:08:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:06 smithi194 ceph-mon[33017]: pgmap v3119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:06 smithi033 ceph-mon[27845]: pgmap v3119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:06 smithi096 ceph-mon[32753]: pgmap v3119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4626.752981 secs 2024-04-26T13:08:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:08 smithi194 ceph-mon[33017]: pgmap v3120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4629.492027 secs 2024-04-26T13:08:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4626.752981 secs 2024-04-26T13:08:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:08 smithi033 ceph-mon[27845]: pgmap v3120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4629.492027 secs 2024-04-26T13:08:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4626.752981 secs 2024-04-26T13:08:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:08 smithi096 ceph-mon[32753]: pgmap v3120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4629.492027 secs 2024-04-26T13:08:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:11 smithi194 ceph-mon[33017]: pgmap v3121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:11 smithi033 ceph-mon[27845]: pgmap v3121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:11 smithi096 ceph-mon[32753]: pgmap v3121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:13 smithi194 ceph-mon[33017]: pgmap v3122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4631.753067 secs 2024-04-26T13:08:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:13 smithi033 ceph-mon[27845]: pgmap v3122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4631.753067 secs 2024-04-26T13:08:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:13 smithi096 ceph-mon[32753]: pgmap v3122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4631.753067 secs 2024-04-26T13:08:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4634.492146 secs 2024-04-26T13:08:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4634.492146 secs 2024-04-26T13:08:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4634.492146 secs 2024-04-26T13:08:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:15 smithi194 ceph-mon[33017]: pgmap v3123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:15 smithi033 ceph-mon[27845]: pgmap v3123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:15 smithi096 ceph-mon[32753]: pgmap v3123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:17 smithi194 ceph-mon[33017]: pgmap v3124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:17 smithi033 ceph-mon[27845]: pgmap v3124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:17 smithi096 ceph-mon[32753]: pgmap v3124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4636.753129 secs 2024-04-26T13:08:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4636.753129 secs 2024-04-26T13:08:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4636.753129 secs 2024-04-26T13:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:19 smithi194 ceph-mon[33017]: pgmap v3125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4639.492265 secs 2024-04-26T13:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:19 smithi033 ceph-mon[27845]: pgmap v3125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4639.492265 secs 2024-04-26T13:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:19 smithi096 ceph-mon[32753]: pgmap v3125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4639.492265 secs 2024-04-26T13:08:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:20 smithi194 ceph-mon[33017]: pgmap v3126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:08:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:20 smithi033 ceph-mon[27845]: pgmap v3126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:08:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:20 smithi096 ceph-mon[32753]: pgmap v3126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:08:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:21.056+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:21.056+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:21.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:21.056+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:08:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:08:21.738 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:08:21.739 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:08:21.739 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:08:21.739 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:08:21.739 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:08:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:08:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:08:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:08:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:22.579+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:08:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:22.579+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:08:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:22.580+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T13:08:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:22.580+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T13:08:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:22 smithi033 ceph-mon[27845]: pgmap v3127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:22 smithi033 ceph-mon[27845]: pgmap v3128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:22 smithi096 ceph-mon[32753]: pgmap v3127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:22 smithi096 ceph-mon[32753]: pgmap v3128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:22 smithi194 ceph-mon[33017]: pgmap v3127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:22 smithi194 ceph-mon[33017]: pgmap v3128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4641.753192 secs 2024-04-26T13:08:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4641.753192 secs 2024-04-26T13:08:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4641.753192 secs 2024-04-26T13:08:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4644.492340 secs 2024-04-26T13:08:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:24 smithi033 ceph-mon[27845]: pgmap v3129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4644.492340 secs 2024-04-26T13:08:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:24 smithi096 ceph-mon[32753]: pgmap v3129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4644.492340 secs 2024-04-26T13:08:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:24 smithi194 ceph-mon[33017]: pgmap v3129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:26 smithi033 ceph-mon[27845]: pgmap v3130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:26 smithi096 ceph-mon[32753]: pgmap v3130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:26 smithi194 ceph-mon[33017]: pgmap v3130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:28 smithi033 ceph-mon[27845]: pgmap v3131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4646.753278 secs 2024-04-26T13:08:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:28 smithi096 ceph-mon[32753]: pgmap v3131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4646.753278 secs 2024-04-26T13:08:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:28 smithi194 ceph-mon[33017]: pgmap v3131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4646.753278 secs 2024-04-26T13:08:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4649.492456 secs 2024-04-26T13:08:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4649.492456 secs 2024-04-26T13:08:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4649.492456 secs 2024-04-26T13:08:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:30 smithi033 ceph-mon[27845]: pgmap v3132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:30 smithi096 ceph-mon[32753]: pgmap v3132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:30 smithi194 ceph-mon[33017]: pgmap v3132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:08:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:32 smithi033 ceph-mon[27845]: pgmap v3133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:08:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:32 smithi096 ceph-mon[32753]: pgmap v3133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:08:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:32 smithi194 ceph-mon[33017]: pgmap v3133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:08:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4651.753339 secs 2024-04-26T13:08:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4651.753339 secs 2024-04-26T13:08:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4651.753339 secs 2024-04-26T13:08:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4654.492572 secs 2024-04-26T13:08:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:34 smithi096 ceph-mon[32753]: pgmap v3134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4654.492572 secs 2024-04-26T13:08:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:34 smithi194 ceph-mon[33017]: pgmap v3134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4654.492572 secs 2024-04-26T13:08:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:34 smithi033 ceph-mon[27845]: pgmap v3134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:36.071+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:36.071+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:36.071+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:08:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:36 smithi194 ceph-mon[33017]: pgmap v3135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:36 smithi033 ceph-mon[27845]: pgmap v3135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:36 smithi096 ceph-mon[32753]: pgmap v3135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:38 smithi194 ceph-mon[33017]: pgmap v3136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4656.753404 secs 2024-04-26T13:08:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:38 smithi033 ceph-mon[27845]: pgmap v3136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4656.753404 secs 2024-04-26T13:08:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:38 smithi096 ceph-mon[32753]: pgmap v3136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4656.753404 secs 2024-04-26T13:08:39.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4659.492635 secs 2024-04-26T13:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4659.492635 secs 2024-04-26T13:08:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4659.492635 secs 2024-04-26T13:08:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:40 smithi194 ceph-mon[33017]: pgmap v3137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:40 smithi033 ceph-mon[27845]: pgmap v3137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:40 smithi096 ceph-mon[32753]: pgmap v3137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:42.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:42 smithi096 ceph-mon[32753]: pgmap v3138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:42 smithi194 ceph-mon[33017]: pgmap v3138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:42 smithi033 ceph-mon[27845]: pgmap v3138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4661.753464 secs 2024-04-26T13:08:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4661.753464 secs 2024-04-26T13:08:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4661.753464 secs 2024-04-26T13:08:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4664.492757 secs 2024-04-26T13:08:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:44 smithi194 ceph-mon[33017]: pgmap v3139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4664.492757 secs 2024-04-26T13:08:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:44 smithi033 ceph-mon[27845]: pgmap v3139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4664.492757 secs 2024-04-26T13:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:44 smithi096 ceph-mon[32753]: pgmap v3139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:46 smithi194 ceph-mon[33017]: pgmap v3140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:46 smithi033 ceph-mon[27845]: pgmap v3140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:46 smithi096 ceph-mon[32753]: pgmap v3140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:48 smithi194 ceph-mon[33017]: pgmap v3141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4666.753549 secs 2024-04-26T13:08:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:48 smithi033 ceph-mon[27845]: pgmap v3141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4666.753549 secs 2024-04-26T13:08:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:48 smithi096 ceph-mon[32753]: pgmap v3141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4666.753549 secs 2024-04-26T13:08:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4669.492888 secs 2024-04-26T13:08:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4669.492888 secs 2024-04-26T13:08:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4669.492888 secs 2024-04-26T13:08:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:50 smithi194 ceph-mon[33017]: pgmap v3142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:50 smithi033 ceph-mon[27845]: pgmap v3142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:50 smithi096 ceph-mon[32753]: pgmap v3142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:51.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:51.081+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:51.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:51.081+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:08:51.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:08:51.081+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:08:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:52 smithi194 ceph-mon[33017]: pgmap v3143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:52 smithi033 ceph-mon[27845]: pgmap v3143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:52 smithi096 ceph-mon[32753]: pgmap v3143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4671.753611 secs 2024-04-26T13:08:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4671.753611 secs 2024-04-26T13:08:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4671.753611 secs 2024-04-26T13:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4674.493030 secs 2024-04-26T13:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:54 smithi194 ceph-mon[33017]: pgmap v3144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4674.493030 secs 2024-04-26T13:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:54 smithi033 ceph-mon[27845]: pgmap v3144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4674.493030 secs 2024-04-26T13:08:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:54 smithi096 ceph-mon[32753]: pgmap v3144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:56 smithi194 ceph-mon[33017]: pgmap v3145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:56 smithi033 ceph-mon[27845]: pgmap v3145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:56 smithi096 ceph-mon[32753]: pgmap v3145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:08:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:58 smithi194 ceph-mon[33017]: pgmap v3146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4676.753695 secs 2024-04-26T13:08:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:58 smithi033 ceph-mon[27845]: pgmap v3146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4676.753695 secs 2024-04-26T13:08:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:58 smithi096 ceph-mon[32753]: pgmap v3146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:08:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4676.753695 secs 2024-04-26T13:08:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4679.493131 secs 2024-04-26T13:08:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4679.493131 secs 2024-04-26T13:08:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4679.493131 secs 2024-04-26T13:09:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:00 smithi033 ceph-mon[27845]: pgmap v3147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:00 smithi194 ceph-mon[33017]: pgmap v3147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:00 smithi096 ceph-mon[32753]: pgmap v3147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:02 smithi194 ceph-mon[33017]: pgmap v3148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:02 smithi033 ceph-mon[27845]: pgmap v3148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:02 smithi096 ceph-mon[32753]: pgmap v3148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4681.753778 secs 2024-04-26T13:09:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4681.753778 secs 2024-04-26T13:09:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4681.753778 secs 2024-04-26T13:09:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4684.493243 secs 2024-04-26T13:09:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:04 smithi194 ceph-mon[33017]: pgmap v3149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4684.493243 secs 2024-04-26T13:09:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:04 smithi033 ceph-mon[27845]: pgmap v3149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4684.493243 secs 2024-04-26T13:09:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:04 smithi096 ceph-mon[32753]: pgmap v3149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:06.089+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:06.089+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:06.089+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:09:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:06 smithi194 ceph-mon[33017]: pgmap v3150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:06 smithi033 ceph-mon[27845]: pgmap v3150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:06 smithi096 ceph-mon[32753]: pgmap v3150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:08 smithi194 ceph-mon[33017]: pgmap v3151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4686.753865 secs 2024-04-26T13:09:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:08 smithi033 ceph-mon[27845]: pgmap v3151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4686.753865 secs 2024-04-26T13:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:08 smithi096 ceph-mon[32753]: pgmap v3151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4686.753865 secs 2024-04-26T13:09:09.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4689.493377 secs 2024-04-26T13:09:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4689.493377 secs 2024-04-26T13:09:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4689.493377 secs 2024-04-26T13:09:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:10 smithi194 ceph-mon[33017]: pgmap v3152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:10 smithi033 ceph-mon[27845]: pgmap v3152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:10 smithi096 ceph-mon[32753]: pgmap v3152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:12 smithi194 ceph-mon[33017]: pgmap v3153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:12 smithi033 ceph-mon[27845]: pgmap v3153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:12 smithi096 ceph-mon[32753]: pgmap v3153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4691.753949 secs 2024-04-26T13:09:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4691.753949 secs 2024-04-26T13:09:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4691.753949 secs 2024-04-26T13:09:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4694.493500 secs 2024-04-26T13:09:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:14 smithi194 ceph-mon[33017]: pgmap v3154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4694.493500 secs 2024-04-26T13:09:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:14 smithi033 ceph-mon[27845]: pgmap v3154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4694.493500 secs 2024-04-26T13:09:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:14 smithi096 ceph-mon[32753]: pgmap v3154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:16 smithi033 ceph-mon[27845]: pgmap v3155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:16 smithi096 ceph-mon[32753]: pgmap v3155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:16 smithi194 ceph-mon[33017]: pgmap v3155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:18 smithi033 ceph-mon[27845]: pgmap v3156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4696.754012 secs 2024-04-26T13:09:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:18 smithi096 ceph-mon[32753]: pgmap v3156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4696.754012 secs 2024-04-26T13:09:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:18 smithi194 ceph-mon[33017]: pgmap v3156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4696.754012 secs 2024-04-26T13:09:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4699.493638 secs 2024-04-26T13:09:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4699.493638 secs 2024-04-26T13:09:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4699.493638 secs 2024-04-26T13:09:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:20 smithi033 ceph-mon[27845]: pgmap v3157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:20 smithi096 ceph-mon[32753]: pgmap v3157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:21.098+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:21.098+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:09:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:21.098+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:20 smithi194 ceph-mon[33017]: pgmap v3157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:22 smithi033 ceph-mon[27845]: pgmap v3158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:09:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:22 smithi096 ceph-mon[32753]: pgmap v3158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:09:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:22 smithi194 ceph-mon[33017]: pgmap v3158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:23.174 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:09:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4701.754097 secs 2024-04-26T13:09:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:09:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:09:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:23 smithi033 ceph-mon[27845]: pgmap v3159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:23 smithi033 ceph-mon[27845]: pgmap v3160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:09:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4701.754097 secs 2024-04-26T13:09:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:09:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:09:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:23 smithi096 ceph-mon[32753]: pgmap v3159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:23 smithi096 ceph-mon[32753]: pgmap v3160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:09:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4701.754097 secs 2024-04-26T13:09:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:09:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:09:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:23 smithi194 ceph-mon[33017]: pgmap v3159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:23 smithi194 ceph-mon[33017]: pgmap v3160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:09:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4704.493750 secs 2024-04-26T13:09:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4704.493750 secs 2024-04-26T13:09:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4704.493750 secs 2024-04-26T13:09:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:25 smithi033 ceph-mon[27845]: pgmap v3161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:25 smithi096 ceph-mon[32753]: pgmap v3161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:25 smithi194 ceph-mon[33017]: pgmap v3161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:27 smithi194 ceph-mon[33017]: pgmap v3162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:27 smithi033 ceph-mon[27845]: pgmap v3162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:27 smithi096 ceph-mon[32753]: pgmap v3162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4706.754188 secs 2024-04-26T13:09:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4706.754188 secs 2024-04-26T13:09:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4706.754188 secs 2024-04-26T13:09:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4709.493866 secs 2024-04-26T13:09:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:29 smithi033 ceph-mon[27845]: pgmap v3163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:09:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4709.493866 secs 2024-04-26T13:09:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:29 smithi096 ceph-mon[32753]: pgmap v3163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:09:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4709.493866 secs 2024-04-26T13:09:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:29 smithi194 ceph-mon[33017]: pgmap v3163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:09:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:31 smithi194 ceph-mon[33017]: pgmap v3164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:31 smithi033 ceph-mon[27845]: pgmap v3164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:31 smithi096 ceph-mon[32753]: pgmap v3164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:09:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4711.754279 secs 2024-04-26T13:09:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:33 smithi194 ceph-mon[33017]: pgmap v3165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:09:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4711.754279 secs 2024-04-26T13:09:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:33 smithi033 ceph-mon[27845]: pgmap v3165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:09:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4711.754279 secs 2024-04-26T13:09:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:33 smithi096 ceph-mon[32753]: pgmap v3165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:09:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4714.493961 secs 2024-04-26T13:09:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4714.493961 secs 2024-04-26T13:09:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4714.493961 secs 2024-04-26T13:09:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:35 smithi194 ceph-mon[33017]: pgmap v3166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:35 smithi033 ceph-mon[27845]: pgmap v3166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:35 smithi096 ceph-mon[32753]: pgmap v3166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:36.113+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:36.113+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:36.113+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:09:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:37 smithi194 ceph-mon[33017]: pgmap v3167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:37 smithi033 ceph-mon[27845]: pgmap v3167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:37 smithi096 ceph-mon[32753]: pgmap v3167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4716.754369 secs 2024-04-26T13:09:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4716.754369 secs 2024-04-26T13:09:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4716.754369 secs 2024-04-26T13:09:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4719.494052 secs 2024-04-26T13:09:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:39 smithi033 ceph-mon[27845]: pgmap v3168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4719.494052 secs 2024-04-26T13:09:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:39 smithi096 ceph-mon[32753]: pgmap v3168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4719.494052 secs 2024-04-26T13:09:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:39 smithi194 ceph-mon[33017]: pgmap v3168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:41 smithi194 ceph-mon[33017]: pgmap v3169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:41 smithi033 ceph-mon[27845]: pgmap v3169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:41 smithi096 ceph-mon[32753]: pgmap v3169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4721.754467 secs 2024-04-26T13:09:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:43 smithi194 ceph-mon[33017]: pgmap v3170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4721.754467 secs 2024-04-26T13:09:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:43 smithi033 ceph-mon[27845]: pgmap v3170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4721.754467 secs 2024-04-26T13:09:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:43 smithi096 ceph-mon[32753]: pgmap v3170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4724.494154 secs 2024-04-26T13:09:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4724.494154 secs 2024-04-26T13:09:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4724.494154 secs 2024-04-26T13:09:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:45 smithi033 ceph-mon[27845]: pgmap v3171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:45 smithi096 ceph-mon[32753]: pgmap v3171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:45 smithi194 ceph-mon[33017]: pgmap v3171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:47 smithi194 ceph-mon[33017]: pgmap v3172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:47 smithi033 ceph-mon[27845]: pgmap v3172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:47 smithi096 ceph-mon[32753]: pgmap v3172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4726.754551 secs 2024-04-26T13:09:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4726.754551 secs 2024-04-26T13:09:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4726.754551 secs 2024-04-26T13:09:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4729.494198 secs 2024-04-26T13:09:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:49 smithi033 ceph-mon[27845]: pgmap v3173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4729.494198 secs 2024-04-26T13:09:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:49 smithi096 ceph-mon[32753]: pgmap v3173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4729.494198 secs 2024-04-26T13:09:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:49 smithi194 ceph-mon[33017]: pgmap v3173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:51.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:51.119+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:51.119+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:09:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:09:51.119+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:09:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:51 smithi033 ceph-mon[27845]: pgmap v3174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:51 smithi096 ceph-mon[32753]: pgmap v3174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:51 smithi194 ceph-mon[33017]: pgmap v3174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4731.754636 secs 2024-04-26T13:09:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:53 smithi194 ceph-mon[33017]: pgmap v3175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4731.754636 secs 2024-04-26T13:09:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:53 smithi033 ceph-mon[27845]: pgmap v3175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4731.754636 secs 2024-04-26T13:09:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:53 smithi096 ceph-mon[32753]: pgmap v3175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4734.494262 secs 2024-04-26T13:09:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4734.494262 secs 2024-04-26T13:09:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4734.494262 secs 2024-04-26T13:09:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:55 smithi033 ceph-mon[27845]: pgmap v3176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:55 smithi096 ceph-mon[32753]: pgmap v3176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:55 smithi194 ceph-mon[33017]: pgmap v3176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:09:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:57 smithi033 ceph-mon[27845]: pgmap v3177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:57 smithi096 ceph-mon[32753]: pgmap v3177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:57 smithi194 ceph-mon[33017]: pgmap v3177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4736.754739 secs 2024-04-26T13:09:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4736.754739 secs 2024-04-26T13:09:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4736.754739 secs 2024-04-26T13:09:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4739.494380 secs 2024-04-26T13:09:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:09:59 smithi033 ceph-mon[27845]: pgmap v3178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4739.494380 secs 2024-04-26T13:09:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:09:59 smithi096 ceph-mon[32753]: pgmap v3178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:09:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4739.494380 secs 2024-04-26T13:09:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:09:59 smithi194 ceph-mon[33017]: pgmap v3178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:10:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:10:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:10:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:01 smithi033 ceph-mon[27845]: pgmap v3179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:01 smithi096 ceph-mon[32753]: pgmap v3179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:01 smithi194 ceph-mon[33017]: pgmap v3179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:03.318 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4741.754841 secs 2024-04-26T13:10:03.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:03 smithi033 ceph-mon[27845]: pgmap v3180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4741.754841 secs 2024-04-26T13:10:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:03 smithi194 ceph-mon[33017]: pgmap v3180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4741.754841 secs 2024-04-26T13:10:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:03 smithi096 ceph-mon[32753]: pgmap v3180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4744.494446 secs 2024-04-26T13:10:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4744.494446 secs 2024-04-26T13:10:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4744.494446 secs 2024-04-26T13:10:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:05 smithi033 ceph-mon[27845]: pgmap v3181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:05 smithi096 ceph-mon[32753]: pgmap v3181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:05 smithi194 ceph-mon[33017]: pgmap v3181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:06.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:06.135+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:06.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:06.135+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:10:06.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:06.135+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:07 smithi033 ceph-mon[27845]: pgmap v3182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:07 smithi096 ceph-mon[32753]: pgmap v3182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:07.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:07 smithi194 ceph-mon[33017]: pgmap v3182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4746.754943 secs 2024-04-26T13:10:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4746.754943 secs 2024-04-26T13:10:08.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4746.754943 secs 2024-04-26T13:10:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4749.494543 secs 2024-04-26T13:10:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:09 smithi194 ceph-mon[33017]: pgmap v3183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4749.494543 secs 2024-04-26T13:10:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:09 smithi033 ceph-mon[27845]: pgmap v3183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4749.494543 secs 2024-04-26T13:10:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:09 smithi096 ceph-mon[32753]: pgmap v3183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:12 smithi033 ceph-mon[27845]: pgmap v3184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:12 smithi096 ceph-mon[32753]: pgmap v3184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:12 smithi194 ceph-mon[33017]: pgmap v3184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4751.755030 secs 2024-04-26T13:10:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4751.755030 secs 2024-04-26T13:10:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4751.755030 secs 2024-04-26T13:10:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:14 smithi033 ceph-mon[27845]: pgmap v3185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4754.494660 secs 2024-04-26T13:10:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:14 smithi096 ceph-mon[32753]: pgmap v3185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4754.494660 secs 2024-04-26T13:10:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:14 smithi194 ceph-mon[33017]: pgmap v3185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4754.494660 secs 2024-04-26T13:10:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:16 smithi033 ceph-mon[27845]: pgmap v3186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:16 smithi096 ceph-mon[32753]: pgmap v3186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:16 smithi194 ceph-mon[33017]: pgmap v3186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:18 smithi033 ceph-mon[27845]: pgmap v3187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4756.755117 secs 2024-04-26T13:10:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:18 smithi096 ceph-mon[32753]: pgmap v3187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4756.755117 secs 2024-04-26T13:10:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:18 smithi194 ceph-mon[33017]: pgmap v3187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4756.755117 secs 2024-04-26T13:10:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4759.494776 secs 2024-04-26T13:10:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4759.494776 secs 2024-04-26T13:10:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4759.494776 secs 2024-04-26T13:10:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:20 smithi033 ceph-mon[27845]: pgmap v3188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:20 smithi096 ceph-mon[32753]: pgmap v3188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:20 smithi194 ceph-mon[33017]: pgmap v3188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:21.150+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:21.151+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:21.151+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:10:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:22 smithi033 ceph-mon[27845]: pgmap v3189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:22 smithi096 ceph-mon[32753]: pgmap v3189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:22 smithi194 ceph-mon[33017]: pgmap v3189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4761.755177 secs 2024-04-26T13:10:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4761.755177 secs 2024-04-26T13:10:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4761.755177 secs 2024-04-26T13:10:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:24 smithi033 ceph-mon[27845]: pgmap v3190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4764.494913 secs 2024-04-26T13:10:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:10:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:24 smithi096 ceph-mon[32753]: pgmap v3190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4764.494913 secs 2024-04-26T13:10:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:10:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:24 smithi194 ceph-mon[33017]: pgmap v3190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4764.494913 secs 2024-04-26T13:10:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:10:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:10:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:10:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:25 smithi033 ceph-mon[27845]: pgmap v3191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:10:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:25 smithi033 ceph-mon[27845]: pgmap v3192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:10:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:10:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:10:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:25 smithi096 ceph-mon[32753]: pgmap v3191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:10:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:25 smithi096 ceph-mon[32753]: pgmap v3192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:10:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:10:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:10:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:25 smithi194 ceph-mon[33017]: pgmap v3191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:10:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:25 smithi194 ceph-mon[33017]: pgmap v3192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:10:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:27 smithi194 ceph-mon[33017]: pgmap v3193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:27 smithi033 ceph-mon[27845]: pgmap v3193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:27 smithi096 ceph-mon[32753]: pgmap v3193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4766.755238 secs 2024-04-26T13:10:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4766.755238 secs 2024-04-26T13:10:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4766.755238 secs 2024-04-26T13:10:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4769.495049 secs 2024-04-26T13:10:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:29 smithi194 ceph-mon[33017]: pgmap v3194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4769.495049 secs 2024-04-26T13:10:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:29 smithi033 ceph-mon[27845]: pgmap v3194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4769.495049 secs 2024-04-26T13:10:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:29 smithi096 ceph-mon[32753]: pgmap v3194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:31 smithi194 ceph-mon[33017]: pgmap v3195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:31 smithi033 ceph-mon[27845]: pgmap v3195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:31 smithi096 ceph-mon[32753]: pgmap v3195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4771.755323 secs 2024-04-26T13:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:33 smithi194 ceph-mon[33017]: pgmap v3196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4771.755323 secs 2024-04-26T13:10:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:33 smithi033 ceph-mon[27845]: pgmap v3196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4771.755323 secs 2024-04-26T13:10:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:33 smithi096 ceph-mon[32753]: pgmap v3196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:10:35.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4774.495166 secs 2024-04-26T13:10:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4774.495166 secs 2024-04-26T13:10:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4774.495166 secs 2024-04-26T13:10:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:35 smithi194 ceph-mon[33017]: pgmap v3197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:10:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:36.164+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:36.164+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:10:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:36.164+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:35 smithi033 ceph-mon[27845]: pgmap v3197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:10:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:35 smithi096 ceph-mon[32753]: pgmap v3197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:10:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:37 smithi194 ceph-mon[33017]: pgmap v3198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:37 smithi033 ceph-mon[27845]: pgmap v3198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:37 smithi096 ceph-mon[32753]: pgmap v3198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:39.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4776.755407 secs 2024-04-26T13:10:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4776.755407 secs 2024-04-26T13:10:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4776.755407 secs 2024-04-26T13:10:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4779.495289 secs 2024-04-26T13:10:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:39 smithi194 ceph-mon[33017]: pgmap v3199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4779.495289 secs 2024-04-26T13:10:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:39 smithi033 ceph-mon[27845]: pgmap v3199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4779.495289 secs 2024-04-26T13:10:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:39 smithi096 ceph-mon[32753]: pgmap v3199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:41 smithi194 ceph-mon[33017]: pgmap v3200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:41 smithi033 ceph-mon[27845]: pgmap v3200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:41 smithi096 ceph-mon[32753]: pgmap v3200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4781.755490 secs 2024-04-26T13:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:43 smithi194 ceph-mon[33017]: pgmap v3201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4781.755490 secs 2024-04-26T13:10:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:43 smithi033 ceph-mon[27845]: pgmap v3201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4781.755490 secs 2024-04-26T13:10:44.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:43 smithi096 ceph-mon[32753]: pgmap v3201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4784.495430 secs 2024-04-26T13:10:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4784.495430 secs 2024-04-26T13:10:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4784.495430 secs 2024-04-26T13:10:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:45 smithi194 ceph-mon[33017]: pgmap v3202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:45 smithi033 ceph-mon[27845]: pgmap v3202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:45 smithi096 ceph-mon[32753]: pgmap v3202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:47 smithi194 ceph-mon[33017]: pgmap v3203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:47 smithi033 ceph-mon[27845]: pgmap v3203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:47 smithi096 ceph-mon[32753]: pgmap v3203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4786.755575 secs 2024-04-26T13:10:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4786.755575 secs 2024-04-26T13:10:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4786.755575 secs 2024-04-26T13:10:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4789.495557 secs 2024-04-26T13:10:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:49 smithi194 ceph-mon[33017]: pgmap v3204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4789.495557 secs 2024-04-26T13:10:50.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:49 smithi033 ceph-mon[27845]: pgmap v3204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:50.247 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4789.495557 secs 2024-04-26T13:10:50.248 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:49 smithi096 ceph-mon[32753]: pgmap v3204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:51.179+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:51.179+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:10:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:10:51.179+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:10:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:51 smithi194 ceph-mon[33017]: pgmap v3205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:51 smithi033 ceph-mon[27845]: pgmap v3205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:51 smithi096 ceph-mon[32753]: pgmap v3205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4791.755659 secs 2024-04-26T13:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:53 smithi194 ceph-mon[33017]: pgmap v3206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4794.495678 secs 2024-04-26T13:10:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4791.755659 secs 2024-04-26T13:10:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:53 smithi033 ceph-mon[27845]: pgmap v3206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4794.495678 secs 2024-04-26T13:10:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4791.755659 secs 2024-04-26T13:10:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:53 smithi096 ceph-mon[32753]: pgmap v3206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4794.495678 secs 2024-04-26T13:10:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:55 smithi194 ceph-mon[33017]: pgmap v3207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:55 smithi033 ceph-mon[27845]: pgmap v3207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:55 smithi096 ceph-mon[32753]: pgmap v3207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:10:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:57 smithi194 ceph-mon[33017]: pgmap v3208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:57 smithi033 ceph-mon[27845]: pgmap v3208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:57 smithi096 ceph-mon[32753]: pgmap v3208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:10:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4796.755745 secs 2024-04-26T13:10:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4799.495792 secs 2024-04-26T13:10:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4796.755745 secs 2024-04-26T13:10:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4799.495792 secs 2024-04-26T13:10:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4796.755745 secs 2024-04-26T13:10:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4799.495792 secs 2024-04-26T13:11:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:10:59 smithi033 ceph-mon[27845]: pgmap v3209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:10:59 smithi096 ceph-mon[32753]: pgmap v3209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:10:59 smithi194 ceph-mon[33017]: pgmap v3209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:01 smithi033 ceph-mon[27845]: pgmap v3210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:01 smithi096 ceph-mon[32753]: pgmap v3210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:01 smithi194 ceph-mon[33017]: pgmap v3210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4801.755830 secs 2024-04-26T13:11:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:03 smithi033 ceph-mon[27845]: pgmap v3211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4804.495916 secs 2024-04-26T13:11:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4801.755830 secs 2024-04-26T13:11:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:03 smithi096 ceph-mon[32753]: pgmap v3211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4804.495916 secs 2024-04-26T13:11:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4801.755830 secs 2024-04-26T13:11:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:03 smithi194 ceph-mon[33017]: pgmap v3211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4804.495916 secs 2024-04-26T13:11:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:05 smithi033 ceph-mon[27845]: pgmap v3212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:05 smithi096 ceph-mon[32753]: pgmap v3212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:06.183+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:06.183+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:06.183+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:11:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:05 smithi194 ceph-mon[33017]: pgmap v3212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:07 smithi033 ceph-mon[27845]: pgmap v3213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:07 smithi096 ceph-mon[32753]: pgmap v3213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:07 smithi194 ceph-mon[33017]: pgmap v3213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4806.755933 secs 2024-04-26T13:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4809.496039 secs 2024-04-26T13:11:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4806.755933 secs 2024-04-26T13:11:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4809.496039 secs 2024-04-26T13:11:09.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4806.755933 secs 2024-04-26T13:11:09.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4809.496039 secs 2024-04-26T13:11:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:09 smithi033 ceph-mon[27845]: pgmap v3214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:09 smithi096 ceph-mon[32753]: pgmap v3214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:09 smithi194 ceph-mon[33017]: pgmap v3214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:11 smithi033 ceph-mon[27845]: pgmap v3215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:11 smithi096 ceph-mon[32753]: pgmap v3215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:11 smithi194 ceph-mon[33017]: pgmap v3215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4811.756003 secs 2024-04-26T13:11:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:13 smithi033 ceph-mon[27845]: pgmap v3216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4814.496156 secs 2024-04-26T13:11:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4811.756003 secs 2024-04-26T13:11:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:13 smithi096 ceph-mon[32753]: pgmap v3216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4814.496156 secs 2024-04-26T13:11:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4811.756003 secs 2024-04-26T13:11:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:13 smithi194 ceph-mon[33017]: pgmap v3216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4814.496156 secs 2024-04-26T13:11:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:15 smithi033 ceph-mon[27845]: pgmap v3217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:15 smithi096 ceph-mon[32753]: pgmap v3217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:15 smithi194 ceph-mon[33017]: pgmap v3217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:17 smithi033 ceph-mon[27845]: pgmap v3218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:17 smithi096 ceph-mon[32753]: pgmap v3218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:17 smithi194 ceph-mon[33017]: pgmap v3218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4816.756066 secs 2024-04-26T13:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4819.496277 secs 2024-04-26T13:11:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4816.756066 secs 2024-04-26T13:11:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4819.496277 secs 2024-04-26T13:11:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4816.756066 secs 2024-04-26T13:11:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4819.496277 secs 2024-04-26T13:11:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:19 smithi096 ceph-mon[32753]: pgmap v3219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:19 smithi194 ceph-mon[33017]: pgmap v3219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:19 smithi033 ceph-mon[27845]: pgmap v3219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.194+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.194+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.194+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 msgr2=0x7feee40b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 msgr2=0x7feee40b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 0x7feee40b2580 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 0x7feee40b2580 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0038040 comp rx=0 tx=0).stop 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 msgr2=0x7feee40b2580 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 msgr2=0x7feee40b2580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0190 0x7feee40b2580 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T13:11:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.717+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 0x7feee40b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T13:11:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.718+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 0x7feee40b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T13:11:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:21.720+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 0x7feee40b2ac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0018000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T13:11:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:21 smithi194 ceph-mon[33017]: pgmap v3220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:21 smithi033 ceph-mon[27845]: pgmap v3220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:21 smithi096 ceph-mon[32753]: pgmap v3220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:22.610+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:11:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:22.610+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:11:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:22.612+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:11:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:22.612+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:11:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4821.756126 secs 2024-04-26T13:11:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:23 smithi194 ceph-mon[33017]: pgmap v3221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4824.496357 secs 2024-04-26T13:11:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4821.756126 secs 2024-04-26T13:11:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:23 smithi033 ceph-mon[27845]: pgmap v3221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4824.496357 secs 2024-04-26T13:11:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4821.756126 secs 2024-04-26T13:11:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:23 smithi096 ceph-mon[32753]: pgmap v3221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4824.496357 secs 2024-04-26T13:11:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:25 smithi194 ceph-mon[33017]: pgmap v3222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:11:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:25 smithi033 ceph-mon[27845]: pgmap v3222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:11:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:25 smithi096 ceph-mon[32753]: pgmap v3222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:11:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:11:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:11:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:11:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:11:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:11:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:11:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:11:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:11:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:11:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:27 smithi194 ceph-mon[33017]: pgmap v3223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:27 smithi194 ceph-mon[33017]: pgmap v3224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:27 smithi033 ceph-mon[27845]: pgmap v3223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:27 smithi033 ceph-mon[27845]: pgmap v3224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:27 smithi096 ceph-mon[32753]: pgmap v3223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:27 smithi096 ceph-mon[32753]: pgmap v3224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4826.756185 secs 2024-04-26T13:11:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4829.496494 secs 2024-04-26T13:11:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4826.756185 secs 2024-04-26T13:11:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4829.496494 secs 2024-04-26T13:11:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4826.756185 secs 2024-04-26T13:11:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4829.496494 secs 2024-04-26T13:11:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:29 smithi194 ceph-mon[33017]: pgmap v3225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:29 smithi033 ceph-mon[27845]: pgmap v3225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:29 smithi096 ceph-mon[32753]: pgmap v3225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:31 smithi194 ceph-mon[33017]: pgmap v3226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:31 smithi033 ceph-mon[27845]: pgmap v3226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:31 smithi096 ceph-mon[32753]: pgmap v3226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:33 smithi194 ceph-mon[33017]: pgmap v3227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4831.756267 secs 2024-04-26T13:11:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4834.496609 secs 2024-04-26T13:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:33 smithi033 ceph-mon[27845]: pgmap v3227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4831.756267 secs 2024-04-26T13:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4834.496609 secs 2024-04-26T13:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:33 smithi096 ceph-mon[32753]: pgmap v3227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4831.756267 secs 2024-04-26T13:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4834.496609 secs 2024-04-26T13:11:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:36.210+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:36.210+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:36.210+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:11:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:35 smithi194 ceph-mon[33017]: pgmap v3228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:35 smithi033 ceph-mon[27845]: pgmap v3228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:35 smithi096 ceph-mon[32753]: pgmap v3228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:11:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:37 smithi194 ceph-mon[33017]: pgmap v3229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:11:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:37 smithi033 ceph-mon[27845]: pgmap v3229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:37 smithi096 ceph-mon[32753]: pgmap v3229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:11:39.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4836.756355 secs 2024-04-26T13:11:39.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4839.496732 secs 2024-04-26T13:11:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4836.756355 secs 2024-04-26T13:11:39.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4839.496732 secs 2024-04-26T13:11:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4836.756355 secs 2024-04-26T13:11:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4839.496732 secs 2024-04-26T13:11:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:40 smithi194 ceph-mon[33017]: pgmap v3230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:40 smithi033 ceph-mon[27845]: pgmap v3230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:40 smithi096 ceph-mon[32753]: pgmap v3230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:42 smithi194 ceph-mon[33017]: pgmap v3231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:42 smithi033 ceph-mon[27845]: pgmap v3231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:42 smithi096 ceph-mon[32753]: pgmap v3231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:44 smithi194 ceph-mon[33017]: pgmap v3232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4841.756439 secs 2024-04-26T13:11:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4844.496871 secs 2024-04-26T13:11:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:44 smithi033 ceph-mon[27845]: pgmap v3232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4841.756439 secs 2024-04-26T13:11:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4844.496871 secs 2024-04-26T13:11:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:44 smithi096 ceph-mon[32753]: pgmap v3232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4841.756439 secs 2024-04-26T13:11:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4844.496871 secs 2024-04-26T13:11:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:46 smithi194 ceph-mon[33017]: pgmap v3233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:46 smithi033 ceph-mon[27845]: pgmap v3233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:46 smithi096 ceph-mon[32753]: pgmap v3233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:48 smithi194 ceph-mon[33017]: pgmap v3234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4846.756525 secs 2024-04-26T13:11:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:48 smithi033 ceph-mon[27845]: pgmap v3234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4846.756525 secs 2024-04-26T13:11:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:48 smithi096 ceph-mon[32753]: pgmap v3234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4846.756525 secs 2024-04-26T13:11:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4849.496999 secs 2024-04-26T13:11:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4849.496999 secs 2024-04-26T13:11:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4849.496999 secs 2024-04-26T13:11:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:50 smithi194 ceph-mon[33017]: pgmap v3235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:50 smithi033 ceph-mon[27845]: pgmap v3235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:50 smithi096 ceph-mon[32753]: pgmap v3235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:51.225+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:51.226+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:11:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:11:51.226+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:11:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:52 smithi033 ceph-mon[27845]: pgmap v3236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:52 smithi194 ceph-mon[33017]: pgmap v3236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:52 smithi096 ceph-mon[32753]: pgmap v3236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4851.756610 secs 2024-04-26T13:11:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4851.756610 secs 2024-04-26T13:11:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4851.756610 secs 2024-04-26T13:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:54 smithi194 ceph-mon[33017]: pgmap v3237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4854.497155 secs 2024-04-26T13:11:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:54 smithi033 ceph-mon[27845]: pgmap v3237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4854.497155 secs 2024-04-26T13:11:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:54 smithi096 ceph-mon[32753]: pgmap v3237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4854.497155 secs 2024-04-26T13:11:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:56 smithi194 ceph-mon[33017]: pgmap v3238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:56 smithi033 ceph-mon[27845]: pgmap v3238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:56 smithi096 ceph-mon[32753]: pgmap v3238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:11:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:58 smithi194 ceph-mon[33017]: pgmap v3239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4856.756672 secs 2024-04-26T13:11:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:58 smithi033 ceph-mon[27845]: pgmap v3239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4856.756672 secs 2024-04-26T13:11:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:58 smithi096 ceph-mon[32753]: pgmap v3239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:11:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4856.756672 secs 2024-04-26T13:11:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4859.497289 secs 2024-04-26T13:11:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4859.497289 secs 2024-04-26T13:11:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4859.497289 secs 2024-04-26T13:12:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:00 smithi194 ceph-mon[33017]: pgmap v3240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:00 smithi033 ceph-mon[27845]: pgmap v3240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:00 smithi096 ceph-mon[32753]: pgmap v3240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:01 smithi194 ceph-mon[33017]: pgmap v3241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:01 smithi033 ceph-mon[27845]: pgmap v3241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:01 smithi096 ceph-mon[32753]: pgmap v3241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:03 smithi194 ceph-mon[33017]: pgmap v3242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4861.756758 secs 2024-04-26T13:12:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:03 smithi033 ceph-mon[27845]: pgmap v3242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4861.756758 secs 2024-04-26T13:12:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:03 smithi096 ceph-mon[32753]: pgmap v3242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4861.756758 secs 2024-04-26T13:12:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4864.497420 secs 2024-04-26T13:12:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4864.497420 secs 2024-04-26T13:12:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4864.497420 secs 2024-04-26T13:12:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:05 smithi194 ceph-mon[33017]: pgmap v3243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:05 smithi033 ceph-mon[27845]: pgmap v3243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:05 smithi096 ceph-mon[32753]: pgmap v3243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:06.226+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:06.226+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:06.226+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:12:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:07.618+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:12:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:07.618+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:12:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:07.618+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:12:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:07.618+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:12:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:07 smithi194 ceph-mon[33017]: pgmap v3244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:07 smithi033 ceph-mon[27845]: pgmap v3244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:07 smithi096 ceph-mon[32753]: pgmap v3244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4866.756844 secs 2024-04-26T13:12:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4866.756844 secs 2024-04-26T13:12:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4866.756844 secs 2024-04-26T13:12:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4869.497510 secs 2024-04-26T13:12:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:09 smithi194 ceph-mon[33017]: pgmap v3245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:09.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4869.497510 secs 2024-04-26T13:12:09.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:09 smithi033 ceph-mon[27845]: pgmap v3245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4869.497510 secs 2024-04-26T13:12:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:09 smithi096 ceph-mon[32753]: pgmap v3245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:11 smithi194 ceph-mon[33017]: pgmap v3246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:11 smithi033 ceph-mon[27845]: pgmap v3246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:11 smithi096 ceph-mon[32753]: pgmap v3246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:13 smithi194 ceph-mon[33017]: pgmap v3247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4871.756930 secs 2024-04-26T13:12:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:13 smithi033 ceph-mon[27845]: pgmap v3247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4871.756930 secs 2024-04-26T13:12:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:13 smithi096 ceph-mon[32753]: pgmap v3247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4871.756930 secs 2024-04-26T13:12:14.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4874.497606 secs 2024-04-26T13:12:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4874.497606 secs 2024-04-26T13:12:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4874.497606 secs 2024-04-26T13:12:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:15 smithi194 ceph-mon[33017]: pgmap v3248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:15 smithi033 ceph-mon[27845]: pgmap v3248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:15 smithi096 ceph-mon[32753]: pgmap v3248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:17 smithi194 ceph-mon[33017]: pgmap v3249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:17 smithi033 ceph-mon[27845]: pgmap v3249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:17 smithi096 ceph-mon[32753]: pgmap v3249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4876.757007 secs 2024-04-26T13:12:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4876.757007 secs 2024-04-26T13:12:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4876.757007 secs 2024-04-26T13:12:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4879.497724 secs 2024-04-26T13:12:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:19 smithi194 ceph-mon[33017]: pgmap v3250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4879.497724 secs 2024-04-26T13:12:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:19 smithi033 ceph-mon[27845]: pgmap v3250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4879.497724 secs 2024-04-26T13:12:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:19 smithi096 ceph-mon[32753]: pgmap v3250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:21.237+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:21.237+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:21.237+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:12:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:21 smithi194 ceph-mon[33017]: pgmap v3251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:21 smithi033 ceph-mon[27845]: pgmap v3251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:21 smithi096 ceph-mon[32753]: pgmap v3251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:23 smithi194 ceph-mon[33017]: pgmap v3252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4881.757069 secs 2024-04-26T13:12:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:23 smithi033 ceph-mon[27845]: pgmap v3252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4881.757069 secs 2024-04-26T13:12:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:23 smithi096 ceph-mon[32753]: pgmap v3252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4881.757069 secs 2024-04-26T13:12:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4884.497840 secs 2024-04-26T13:12:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4884.497840 secs 2024-04-26T13:12:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4884.497840 secs 2024-04-26T13:12:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:25 smithi194 ceph-mon[33017]: pgmap v3253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:25 smithi033 ceph-mon[27845]: pgmap v3253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:25 smithi096 ceph-mon[32753]: pgmap v3253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:27 smithi194 ceph-mon[33017]: pgmap v3254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:12:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:27 smithi033 ceph-mon[27845]: pgmap v3254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:12:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:27 smithi096 ceph-mon[32753]: pgmap v3254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4886.757159 secs 2024-04-26T13:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4886.757159 secs 2024-04-26T13:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4886.757159 secs 2024-04-26T13:12:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4889.497978 secs 2024-04-26T13:12:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:29 smithi033 ceph-mon[27845]: pgmap v3255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:12:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:12:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:29 smithi033 ceph-mon[27845]: pgmap v3256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:12:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:29 smithi033 ceph-mon[27845]: pgmap v3257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:12:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4889.497978 secs 2024-04-26T13:12:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:29 smithi096 ceph-mon[32753]: pgmap v3255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:12:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:12:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:29 smithi096 ceph-mon[32753]: pgmap v3256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:12:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:29 smithi096 ceph-mon[32753]: pgmap v3257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:12:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4889.497978 secs 2024-04-26T13:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:29 smithi194 ceph-mon[33017]: pgmap v3255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:29 smithi194 ceph-mon[33017]: pgmap v3256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:29 smithi194 ceph-mon[33017]: pgmap v3257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:12:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:32 smithi194 ceph-mon[33017]: pgmap v3258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:32 smithi033 ceph-mon[27845]: pgmap v3258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:32 smithi096 ceph-mon[32753]: pgmap v3258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4891.757266 secs 2024-04-26T13:12:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4891.757266 secs 2024-04-26T13:12:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4891.757266 secs 2024-04-26T13:12:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:34 smithi194 ceph-mon[33017]: pgmap v3259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:12:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4894.498102 secs 2024-04-26T13:12:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:34 smithi033 ceph-mon[27845]: pgmap v3259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:12:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4894.498102 secs 2024-04-26T13:12:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:34 smithi096 ceph-mon[32753]: pgmap v3259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:12:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4894.498102 secs 2024-04-26T13:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:36.254+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:36.254+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:12:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:36.254+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:36 smithi194 ceph-mon[33017]: pgmap v3260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:36 smithi033 ceph-mon[27845]: pgmap v3260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:36 smithi096 ceph-mon[32753]: pgmap v3260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:38 smithi194 ceph-mon[33017]: pgmap v3261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4896.757348 secs 2024-04-26T13:12:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:38 smithi033 ceph-mon[27845]: pgmap v3261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4896.757348 secs 2024-04-26T13:12:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:38 smithi096 ceph-mon[32753]: pgmap v3261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:12:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4896.757348 secs 2024-04-26T13:12:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4899.498224 secs 2024-04-26T13:12:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4899.498224 secs 2024-04-26T13:12:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4899.498224 secs 2024-04-26T13:12:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:40 smithi194 ceph-mon[33017]: pgmap v3262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:12:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:40 smithi033 ceph-mon[27845]: pgmap v3262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:12:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:40 smithi096 ceph-mon[32753]: pgmap v3262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:12:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:41 smithi194 ceph-mon[33017]: pgmap v3263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:41 smithi033 ceph-mon[27845]: pgmap v3263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:41 smithi096 ceph-mon[32753]: pgmap v3263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4901.757436 secs 2024-04-26T13:12:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4901.757436 secs 2024-04-26T13:12:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4901.757436 secs 2024-04-26T13:12:44.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:44 smithi096 ceph-mon[32753]: pgmap v3264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:44.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4904.498303 secs 2024-04-26T13:12:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:44 smithi194 ceph-mon[33017]: pgmap v3264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4904.498303 secs 2024-04-26T13:12:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:44 smithi033 ceph-mon[27845]: pgmap v3264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4904.498303 secs 2024-04-26T13:12:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:46 smithi194 ceph-mon[33017]: pgmap v3265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:46 smithi033 ceph-mon[27845]: pgmap v3265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:46 smithi096 ceph-mon[32753]: pgmap v3265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:47 smithi194 ceph-mon[33017]: pgmap v3266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:47 smithi033 ceph-mon[27845]: pgmap v3266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:47 smithi096 ceph-mon[32753]: pgmap v3266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4906.757521 secs 2024-04-26T13:12:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4906.757521 secs 2024-04-26T13:12:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4906.757521 secs 2024-04-26T13:12:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4909.498419 secs 2024-04-26T13:12:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:49 smithi033 ceph-mon[27845]: pgmap v3267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4909.498419 secs 2024-04-26T13:12:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:49 smithi096 ceph-mon[32753]: pgmap v3267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:49.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4909.498419 secs 2024-04-26T13:12:49.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:49 smithi194 ceph-mon[33017]: pgmap v3267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:51.269+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:51.269+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:12:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:12:51.269+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:12:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:52 smithi194 ceph-mon[33017]: pgmap v3268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:52 smithi033 ceph-mon[27845]: pgmap v3268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:52 smithi096 ceph-mon[32753]: pgmap v3268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4911.757609 secs 2024-04-26T13:12:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4911.757609 secs 2024-04-26T13:12:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4911.757609 secs 2024-04-26T13:12:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:54 smithi194 ceph-mon[33017]: pgmap v3269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4914.498521 secs 2024-04-26T13:12:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:54 smithi033 ceph-mon[27845]: pgmap v3269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4914.498521 secs 2024-04-26T13:12:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:54 smithi096 ceph-mon[32753]: pgmap v3269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4914.498521 secs 2024-04-26T13:12:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:56 smithi194 ceph-mon[33017]: pgmap v3270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:56 smithi033 ceph-mon[27845]: pgmap v3270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:56 smithi096 ceph-mon[32753]: pgmap v3270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:12:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:58 smithi194 ceph-mon[33017]: pgmap v3271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4916.757696 secs 2024-04-26T13:12:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:58 smithi033 ceph-mon[27845]: pgmap v3271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4916.757696 secs 2024-04-26T13:12:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:58 smithi096 ceph-mon[32753]: pgmap v3271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4916.757696 secs 2024-04-26T13:12:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4919.498599 secs 2024-04-26T13:12:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:12:59 smithi194 ceph-mon[33017]: pgmap v3272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4919.498599 secs 2024-04-26T13:12:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:12:59 smithi033 ceph-mon[27845]: pgmap v3272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:12:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4919.498599 secs 2024-04-26T13:12:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:12:59 smithi096 ceph-mon[32753]: pgmap v3272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:02 smithi194 ceph-mon[33017]: pgmap v3273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:02 smithi033 ceph-mon[27845]: pgmap v3273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:02 smithi096 ceph-mon[32753]: pgmap v3273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4921.757783 secs 2024-04-26T13:13:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4921.757783 secs 2024-04-26T13:13:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4921.757783 secs 2024-04-26T13:13:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:04 smithi194 ceph-mon[33017]: pgmap v3274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4924.498734 secs 2024-04-26T13:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:04 smithi033 ceph-mon[27845]: pgmap v3274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4924.498734 secs 2024-04-26T13:13:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:04 smithi096 ceph-mon[32753]: pgmap v3274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4924.498734 secs 2024-04-26T13:13:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:06.281+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:06.281+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:13:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:06.281+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:06 smithi194 ceph-mon[33017]: pgmap v3275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:06 smithi033 ceph-mon[27845]: pgmap v3275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:06 smithi096 ceph-mon[32753]: pgmap v3275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:07 smithi194 ceph-mon[33017]: pgmap v3276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:07 smithi033 ceph-mon[27845]: pgmap v3276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:07 smithi096 ceph-mon[32753]: pgmap v3276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4926.757869 secs 2024-04-26T13:13:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4926.757869 secs 2024-04-26T13:13:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4926.757869 secs 2024-04-26T13:13:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4929.498851 secs 2024-04-26T13:13:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:09 smithi033 ceph-mon[27845]: pgmap v3277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4929.498851 secs 2024-04-26T13:13:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:09 smithi096 ceph-mon[32753]: pgmap v3277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4929.498851 secs 2024-04-26T13:13:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:09 smithi194 ceph-mon[33017]: pgmap v3277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:12 smithi194 ceph-mon[33017]: pgmap v3278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:12 smithi033 ceph-mon[27845]: pgmap v3278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:12 smithi096 ceph-mon[32753]: pgmap v3278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4931.757955 secs 2024-04-26T13:13:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4931.757955 secs 2024-04-26T13:13:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4931.757955 secs 2024-04-26T13:13:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:14 smithi194 ceph-mon[33017]: pgmap v3279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4934.498987 secs 2024-04-26T13:13:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:14 smithi033 ceph-mon[27845]: pgmap v3279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4934.498987 secs 2024-04-26T13:13:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:14 smithi096 ceph-mon[32753]: pgmap v3279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4934.498987 secs 2024-04-26T13:13:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:15 smithi033 ceph-mon[27845]: pgmap v3280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:15 smithi096 ceph-mon[32753]: pgmap v3280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:15 smithi194 ceph-mon[33017]: pgmap v3280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:18 smithi194 ceph-mon[33017]: pgmap v3281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4936.758041 secs 2024-04-26T13:13:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:18 smithi033 ceph-mon[27845]: pgmap v3281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4936.758041 secs 2024-04-26T13:13:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:18 smithi096 ceph-mon[32753]: pgmap v3281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4936.758041 secs 2024-04-26T13:13:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4939.499063 secs 2024-04-26T13:13:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4939.499063 secs 2024-04-26T13:13:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4939.499063 secs 2024-04-26T13:13:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:20 smithi194 ceph-mon[33017]: pgmap v3282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:20 smithi033 ceph-mon[27845]: pgmap v3282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:20 smithi096 ceph-mon[32753]: pgmap v3282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:21.287+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:21.287+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:21.287+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:13:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:21 smithi033 ceph-mon[27845]: pgmap v3283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:21 smithi096 ceph-mon[32753]: pgmap v3283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:21 smithi194 ceph-mon[33017]: pgmap v3283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4941.758131 secs 2024-04-26T13:13:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4941.758131 secs 2024-04-26T13:13:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4941.758131 secs 2024-04-26T13:13:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:24 smithi194 ceph-mon[33017]: pgmap v3284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4944.499156 secs 2024-04-26T13:13:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:24 smithi033 ceph-mon[27845]: pgmap v3284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4944.499156 secs 2024-04-26T13:13:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:24 smithi096 ceph-mon[32753]: pgmap v3284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4944.499156 secs 2024-04-26T13:13:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:26 smithi194 ceph-mon[33017]: pgmap v3285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:26 smithi033 ceph-mon[27845]: pgmap v3285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:26 smithi096 ceph-mon[32753]: pgmap v3285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:27 smithi033 ceph-mon[27845]: pgmap v3286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:27 smithi096 ceph-mon[32753]: pgmap v3286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:27 smithi194 ceph-mon[33017]: pgmap v3286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4946.758218 secs 2024-04-26T13:13:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4946.758218 secs 2024-04-26T13:13:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4946.758218 secs 2024-04-26T13:13:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4949.499265 secs 2024-04-26T13:13:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:29 smithi033 ceph-mon[27845]: pgmap v3287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4949.499265 secs 2024-04-26T13:13:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:29 smithi096 ceph-mon[32753]: pgmap v3287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4949.499265 secs 2024-04-26T13:13:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:29 smithi194 ceph-mon[33017]: pgmap v3287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:13:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:13:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:13:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:13:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:13:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:31 smithi033 ceph-mon[27845]: pgmap v3288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:13:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:31 smithi033 ceph-mon[27845]: pgmap v3289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:13:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:13:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:13:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:31 smithi096 ceph-mon[32753]: pgmap v3288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:13:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:31 smithi096 ceph-mon[32753]: pgmap v3289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:13:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:13:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:13:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:31 smithi194 ceph-mon[33017]: pgmap v3288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:13:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:31 smithi194 ceph-mon[33017]: pgmap v3289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:13:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4951.758295 secs 2024-04-26T13:13:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:33 smithi194 ceph-mon[33017]: pgmap v3290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4951.758295 secs 2024-04-26T13:13:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:33 smithi033 ceph-mon[27845]: pgmap v3290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4951.758295 secs 2024-04-26T13:13:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:33 smithi096 ceph-mon[32753]: pgmap v3290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4954.499388 secs 2024-04-26T13:13:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4954.499388 secs 2024-04-26T13:13:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4954.499388 secs 2024-04-26T13:13:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:35 smithi194 ceph-mon[33017]: pgmap v3291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:35 smithi033 ceph-mon[27845]: pgmap v3291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:35 smithi096 ceph-mon[32753]: pgmap v3291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:36.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:36.300+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:36.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:36.300+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:13:36.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:36.300+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:37 smithi194 ceph-mon[33017]: pgmap v3292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:37 smithi033 ceph-mon[27845]: pgmap v3292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:37 smithi096 ceph-mon[32753]: pgmap v3292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:39.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4956.758354 secs 2024-04-26T13:13:39.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4956.758354 secs 2024-04-26T13:13:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4956.758354 secs 2024-04-26T13:13:40.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4959.499505 secs 2024-04-26T13:13:40.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:39 smithi194 ceph-mon[33017]: pgmap v3293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:40.614 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4959.499505 secs 2024-04-26T13:13:40.615 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:39 smithi096 ceph-mon[32753]: pgmap v3293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:40.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4959.499505 secs 2024-04-26T13:13:40.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:39 smithi033 ceph-mon[27845]: pgmap v3293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:13:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:41 smithi194 ceph-mon[33017]: pgmap v3294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:13:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:41 smithi033 ceph-mon[27845]: pgmap v3294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:13:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:41 smithi096 ceph-mon[32753]: pgmap v3294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:13:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4961.758456 secs 2024-04-26T13:13:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:43 smithi194 ceph-mon[33017]: pgmap v3295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4961.758456 secs 2024-04-26T13:13:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:43 smithi033 ceph-mon[27845]: pgmap v3295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4961.758456 secs 2024-04-26T13:13:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:43 smithi096 ceph-mon[32753]: pgmap v3295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:45.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4964.499642 secs 2024-04-26T13:13:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4964.499642 secs 2024-04-26T13:13:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4964.499642 secs 2024-04-26T13:13:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:45 smithi194 ceph-mon[33017]: pgmap v3296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:45 smithi033 ceph-mon[27845]: pgmap v3296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:45 smithi096 ceph-mon[32753]: pgmap v3296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:47 smithi194 ceph-mon[33017]: pgmap v3297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:47 smithi033 ceph-mon[27845]: pgmap v3297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:47 smithi096 ceph-mon[32753]: pgmap v3297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4966.758543 secs 2024-04-26T13:13:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4966.758543 secs 2024-04-26T13:13:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4966.758543 secs 2024-04-26T13:13:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4969.499758 secs 2024-04-26T13:13:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:49 smithi194 ceph-mon[33017]: pgmap v3298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4969.499758 secs 2024-04-26T13:13:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:49 smithi033 ceph-mon[27845]: pgmap v3298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4969.499758 secs 2024-04-26T13:13:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:49 smithi096 ceph-mon[32753]: pgmap v3298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:51.308+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:51.308+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:13:51.309 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:13:51.308+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:13:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:51 smithi194 ceph-mon[33017]: pgmap v3299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:51 smithi033 ceph-mon[27845]: pgmap v3299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:51 smithi096 ceph-mon[32753]: pgmap v3299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4971.758629 secs 2024-04-26T13:13:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:53 smithi194 ceph-mon[33017]: pgmap v3300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4971.758629 secs 2024-04-26T13:13:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:53 smithi033 ceph-mon[27845]: pgmap v3300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4971.758629 secs 2024-04-26T13:13:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:53 smithi096 ceph-mon[32753]: pgmap v3300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4974.499876 secs 2024-04-26T13:13:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4974.499876 secs 2024-04-26T13:13:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4974.499876 secs 2024-04-26T13:13:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:55 smithi194 ceph-mon[33017]: pgmap v3301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:55 smithi033 ceph-mon[27845]: pgmap v3301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:55 smithi096 ceph-mon[32753]: pgmap v3301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:13:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:57 smithi194 ceph-mon[33017]: pgmap v3302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:57 smithi033 ceph-mon[27845]: pgmap v3302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:57 smithi096 ceph-mon[32753]: pgmap v3302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:13:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4976.758716 secs 2024-04-26T13:13:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4976.758716 secs 2024-04-26T13:13:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4976.758716 secs 2024-04-26T13:14:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4979.500016 secs 2024-04-26T13:14:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:13:59 smithi194 ceph-mon[33017]: pgmap v3303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4979.500016 secs 2024-04-26T13:14:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:13:59 smithi033 ceph-mon[27845]: pgmap v3303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4979.500016 secs 2024-04-26T13:14:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:13:59 smithi096 ceph-mon[32753]: pgmap v3303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:01 smithi194 ceph-mon[33017]: pgmap v3304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:01 smithi033 ceph-mon[27845]: pgmap v3304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:01 smithi096 ceph-mon[32753]: pgmap v3304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4981.758778 secs 2024-04-26T13:14:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:03 smithi194 ceph-mon[33017]: pgmap v3305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4984.500134 secs 2024-04-26T13:14:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4981.758778 secs 2024-04-26T13:14:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:03 smithi033 ceph-mon[27845]: pgmap v3305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:04.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4984.500134 secs 2024-04-26T13:14:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4981.758778 secs 2024-04-26T13:14:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:03 smithi096 ceph-mon[32753]: pgmap v3305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4984.500134 secs 2024-04-26T13:14:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:05 smithi033 ceph-mon[27845]: pgmap v3306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:05 smithi096 ceph-mon[32753]: pgmap v3306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:06.323+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:06.323+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:06.323+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:14:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:05 smithi194 ceph-mon[33017]: pgmap v3306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:07 smithi033 ceph-mon[27845]: pgmap v3307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:07 smithi096 ceph-mon[32753]: pgmap v3307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:07 smithi194 ceph-mon[33017]: pgmap v3307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4986.758865 secs 2024-04-26T13:14:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4989.500285 secs 2024-04-26T13:14:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4986.758865 secs 2024-04-26T13:14:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4989.500285 secs 2024-04-26T13:14:09.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4986.758865 secs 2024-04-26T13:14:09.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4989.500285 secs 2024-04-26T13:14:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:09 smithi033 ceph-mon[27845]: pgmap v3308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:09 smithi096 ceph-mon[32753]: pgmap v3308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:09 smithi194 ceph-mon[33017]: pgmap v3308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:12.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:11 smithi033 ceph-mon[27845]: pgmap v3309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:11 smithi096 ceph-mon[32753]: pgmap v3309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:11 smithi194 ceph-mon[33017]: pgmap v3309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4991.758951 secs 2024-04-26T13:14:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:13 smithi033 ceph-mon[27845]: pgmap v3310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4994.500404 secs 2024-04-26T13:14:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4991.758951 secs 2024-04-26T13:14:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:13 smithi096 ceph-mon[32753]: pgmap v3310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4994.500404 secs 2024-04-26T13:14:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4991.758951 secs 2024-04-26T13:14:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:13 smithi194 ceph-mon[33017]: pgmap v3310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4994.500404 secs 2024-04-26T13:14:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:15 smithi033 ceph-mon[27845]: pgmap v3311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:15 smithi096 ceph-mon[32753]: pgmap v3311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:15 smithi194 ceph-mon[33017]: pgmap v3311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:17 smithi033 ceph-mon[27845]: pgmap v3312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:17 smithi096 ceph-mon[32753]: pgmap v3312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:17 smithi194 ceph-mon[33017]: pgmap v3312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 4996.759007 secs 2024-04-26T13:14:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 4999.500470 secs 2024-04-26T13:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 4996.759007 secs 2024-04-26T13:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 4999.500470 secs 2024-04-26T13:14:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 4996.759007 secs 2024-04-26T13:14:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 4999.500470 secs 2024-04-26T13:14:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:19 smithi033 ceph-mon[27845]: pgmap v3313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:19 smithi096 ceph-mon[32753]: pgmap v3313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:19 smithi194 ceph-mon[33017]: pgmap v3313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:21.338+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:21.338+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:21.338+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:14:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:21 smithi033 ceph-mon[27845]: pgmap v3314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:21 smithi096 ceph-mon[32753]: pgmap v3314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:21 smithi194 ceph-mon[33017]: pgmap v3314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:22.641+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:14:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:22.641+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:14:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:22.642+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:14:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:22.642+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:14:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5001.759097 secs 2024-04-26T13:14:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:23 smithi096 ceph-mon[32753]: pgmap v3315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5004.500546 secs 2024-04-26T13:14:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5001.759097 secs 2024-04-26T13:14:24.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:23 smithi194 ceph-mon[33017]: pgmap v3315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:24.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5004.500546 secs 2024-04-26T13:14:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5001.759097 secs 2024-04-26T13:14:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:23 smithi033 ceph-mon[27845]: pgmap v3315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5004.500546 secs 2024-04-26T13:14:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:25 smithi194 ceph-mon[33017]: pgmap v3316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:25 smithi033 ceph-mon[27845]: pgmap v3316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:25 smithi096 ceph-mon[32753]: pgmap v3316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:27 smithi194 ceph-mon[33017]: pgmap v3317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:27 smithi033 ceph-mon[27845]: pgmap v3317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:27 smithi096 ceph-mon[32753]: pgmap v3317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5006.759175 secs 2024-04-26T13:14:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5009.500664 secs 2024-04-26T13:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5006.759175 secs 2024-04-26T13:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5009.500664 secs 2024-04-26T13:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5006.759175 secs 2024-04-26T13:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5009.500664 secs 2024-04-26T13:14:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:29 smithi194 ceph-mon[33017]: pgmap v3318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:29 smithi033 ceph-mon[27845]: pgmap v3318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:29 smithi096 ceph-mon[32753]: pgmap v3318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:31 smithi194 ceph-mon[33017]: pgmap v3319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:14:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:31 smithi033 ceph-mon[27845]: pgmap v3319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:14:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:31 smithi096 ceph-mon[32753]: pgmap v3319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:14:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5011.759236 secs 2024-04-26T13:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:33 smithi194 ceph-mon[33017]: pgmap v3320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5014.500731 secs 2024-04-26T13:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5011.759236 secs 2024-04-26T13:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:33 smithi033 ceph-mon[27845]: pgmap v3320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5014.500731 secs 2024-04-26T13:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5011.759236 secs 2024-04-26T13:14:34.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:33 smithi096 ceph-mon[32753]: pgmap v3320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:34.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5014.500731 secs 2024-04-26T13:14:34.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:34.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:36.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:36.352+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:36.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:36.352+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:36.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:36.352+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:14:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:35 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:14:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:35 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:14:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:35 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:14:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:35 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:14:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:35 smithi194 ceph-mon[33017]: pgmap v3321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:35 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:35 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:35 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:35 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:35 smithi033 ceph-mon[27845]: pgmap v3321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:35 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:14:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:35 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:14:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:35 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:14:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:35 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:14:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:35 smithi096 ceph-mon[32753]: pgmap v3321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:14:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: pgmap v3322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:14:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: pgmap v3322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:14:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:14:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: pgmap v3322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:14:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:14:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:14:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:14:39.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:38 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:14:39.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:38 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:14:39.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:38 smithi194 ceph-mon[33017]: pgmap v3323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:14:39.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:38 smithi194 ceph-mon[33017]: pgmap v3324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:14:39.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5016.759296 secs 2024-04-26T13:14:39.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5019.500850 secs 2024-04-26T13:14:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:38 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:14:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:38 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:14:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:38 smithi033 ceph-mon[27845]: pgmap v3323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:14:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:38 smithi033 ceph-mon[27845]: pgmap v3324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:14:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5016.759296 secs 2024-04-26T13:14:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5019.500850 secs 2024-04-26T13:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:38 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:38 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:38 smithi096 ceph-mon[32753]: pgmap v3323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:38 smithi096 ceph-mon[32753]: pgmap v3324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5016.759296 secs 2024-04-26T13:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5019.500850 secs 2024-04-26T13:14:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:41 smithi194 ceph-mon[33017]: pgmap v3325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:14:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:41 smithi033 ceph-mon[27845]: pgmap v3325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:14:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:41 smithi096 ceph-mon[32753]: pgmap v3325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:14:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:43 smithi033 ceph-mon[27845]: pgmap v3326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:43 smithi194 ceph-mon[33017]: pgmap v3326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:43 smithi096 ceph-mon[32753]: pgmap v3326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5021.759379 secs 2024-04-26T13:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5024.500971 secs 2024-04-26T13:14:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5021.759379 secs 2024-04-26T13:14:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5024.500971 secs 2024-04-26T13:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5021.759379 secs 2024-04-26T13:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5024.500971 secs 2024-04-26T13:14:45.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:45 smithi096 ceph-mon[32753]: pgmap v3327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:14:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:45 smithi194 ceph-mon[33017]: pgmap v3327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:14:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:45 smithi033 ceph-mon[27845]: pgmap v3327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:14:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:47 smithi194 ceph-mon[33017]: pgmap v3328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:47 smithi033 ceph-mon[27845]: pgmap v3328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:47 smithi096 ceph-mon[32753]: pgmap v3328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:49 smithi194 ceph-mon[33017]: pgmap v3329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5026.759441 secs 2024-04-26T13:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5029.501130 secs 2024-04-26T13:14:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:49 smithi033 ceph-mon[27845]: pgmap v3329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:14:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5026.759441 secs 2024-04-26T13:14:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5029.501130 secs 2024-04-26T13:14:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:49 smithi096 ceph-mon[32753]: pgmap v3329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:14:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5026.759441 secs 2024-04-26T13:14:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5029.501130 secs 2024-04-26T13:14:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:51 smithi194 ceph-mon[33017]: pgmap v3330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:51.366+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:51.366+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:14:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:14:51.366+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:14:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:51 smithi033 ceph-mon[27845]: pgmap v3330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:51 smithi096 ceph-mon[32753]: pgmap v3330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:53 smithi194 ceph-mon[33017]: pgmap v3331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:53 smithi033 ceph-mon[27845]: pgmap v3331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:53 smithi096 ceph-mon[32753]: pgmap v3331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5031.759499 secs 2024-04-26T13:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5034.501247 secs 2024-04-26T13:14:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5031.759499 secs 2024-04-26T13:14:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5034.501247 secs 2024-04-26T13:14:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5031.759499 secs 2024-04-26T13:14:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5034.501247 secs 2024-04-26T13:14:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:55 smithi194 ceph-mon[33017]: pgmap v3332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:55 smithi033 ceph-mon[27845]: pgmap v3332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:55 smithi096 ceph-mon[32753]: pgmap v3332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:57 smithi194 ceph-mon[33017]: pgmap v3333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:14:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:57 smithi033 ceph-mon[27845]: pgmap v3333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:14:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:57 smithi096 ceph-mon[32753]: pgmap v3333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:14:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5036.759557 secs 2024-04-26T13:14:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5036.759557 secs 2024-04-26T13:14:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5036.759557 secs 2024-04-26T13:14:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:59 smithi194 ceph-mon[33017]: pgmap v3334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5039.501335 secs 2024-04-26T13:14:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:59 smithi033 ceph-mon[27845]: pgmap v3334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5039.501335 secs 2024-04-26T13:14:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:59 smithi096 ceph-mon[32753]: pgmap v3334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:14:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5039.501335 secs 2024-04-26T13:15:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:01 smithi194 ceph-mon[33017]: pgmap v3335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:15:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:01 smithi033 ceph-mon[27845]: pgmap v3335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:15:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:01 smithi096 ceph-mon[32753]: pgmap v3335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:15:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:03 smithi194 ceph-mon[33017]: pgmap v3336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:15:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5041.759640 secs 2024-04-26T13:15:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:03 smithi033 ceph-mon[27845]: pgmap v3336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:15:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5041.759640 secs 2024-04-26T13:15:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:03 smithi096 ceph-mon[32753]: pgmap v3336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:15:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5041.759640 secs 2024-04-26T13:15:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5044.501416 secs 2024-04-26T13:15:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5044.501416 secs 2024-04-26T13:15:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5044.501416 secs 2024-04-26T13:15:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:05 smithi194 ceph-mon[33017]: pgmap v3337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:05 smithi033 ceph-mon[27845]: pgmap v3337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:05 smithi096 ceph-mon[32753]: pgmap v3337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:06.378+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:06.379+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:06.379+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:15:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:07 smithi194 ceph-mon[33017]: pgmap v3338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:07 smithi033 ceph-mon[27845]: pgmap v3338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:07 smithi096 ceph-mon[32753]: pgmap v3338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5046.759696 secs 2024-04-26T13:15:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5046.759696 secs 2024-04-26T13:15:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5046.759696 secs 2024-04-26T13:15:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:09 smithi033 ceph-mon[27845]: pgmap v3339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5049.501542 secs 2024-04-26T13:15:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:09 smithi096 ceph-mon[32753]: pgmap v3339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5049.501542 secs 2024-04-26T13:15:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:09 smithi194 ceph-mon[33017]: pgmap v3339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5049.501542 secs 2024-04-26T13:15:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:10 smithi033 ceph-mon[27845]: pgmap v3340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:10 smithi096 ceph-mon[32753]: pgmap v3340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:10 smithi194 ceph-mon[33017]: pgmap v3340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:12 smithi033 ceph-mon[27845]: pgmap v3341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:12 smithi096 ceph-mon[32753]: pgmap v3341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:12 smithi194 ceph-mon[33017]: pgmap v3341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5051.759778 secs 2024-04-26T13:15:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5051.759778 secs 2024-04-26T13:15:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5051.759778 secs 2024-04-26T13:15:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5054.501617 secs 2024-04-26T13:15:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:14 smithi033 ceph-mon[27845]: pgmap v3342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5054.501617 secs 2024-04-26T13:15:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:14 smithi194 ceph-mon[33017]: pgmap v3342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5054.501617 secs 2024-04-26T13:15:14.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:14 smithi096 ceph-mon[32753]: pgmap v3342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:16.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:16 smithi096 ceph-mon[32753]: pgmap v3343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:16 smithi194 ceph-mon[33017]: pgmap v3343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:16 smithi033 ceph-mon[27845]: pgmap v3343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:18 smithi194 ceph-mon[33017]: pgmap v3344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5056.759858 secs 2024-04-26T13:15:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:18 smithi033 ceph-mon[27845]: pgmap v3344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5056.759858 secs 2024-04-26T13:15:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:18 smithi096 ceph-mon[32753]: pgmap v3344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5056.759858 secs 2024-04-26T13:15:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5059.501733 secs 2024-04-26T13:15:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5059.501733 secs 2024-04-26T13:15:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5059.501733 secs 2024-04-26T13:15:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:20 smithi194 ceph-mon[33017]: pgmap v3345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:20 smithi033 ceph-mon[27845]: pgmap v3345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:20 smithi096 ceph-mon[32753]: pgmap v3345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:21.393+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:21.393+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:21.393+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:15:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:22 smithi194 ceph-mon[33017]: pgmap v3346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:22 smithi033 ceph-mon[27845]: pgmap v3346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:22 smithi096 ceph-mon[32753]: pgmap v3346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5061.759944 secs 2024-04-26T13:15:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5061.759944 secs 2024-04-26T13:15:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5061.759944 secs 2024-04-26T13:15:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5064.501810 secs 2024-04-26T13:15:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:24 smithi194 ceph-mon[33017]: pgmap v3347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5064.501810 secs 2024-04-26T13:15:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:24 smithi033 ceph-mon[27845]: pgmap v3347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5064.501810 secs 2024-04-26T13:15:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:24 smithi096 ceph-mon[32753]: pgmap v3347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:26 smithi194 ceph-mon[33017]: pgmap v3348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:26 smithi033 ceph-mon[27845]: pgmap v3348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:26 smithi096 ceph-mon[32753]: pgmap v3348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:28 smithi194 ceph-mon[33017]: pgmap v3349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5066.760031 secs 2024-04-26T13:15:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:28 smithi033 ceph-mon[27845]: pgmap v3349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5066.760031 secs 2024-04-26T13:15:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:28 smithi096 ceph-mon[32753]: pgmap v3349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5066.760031 secs 2024-04-26T13:15:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5069.501926 secs 2024-04-26T13:15:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5069.501926 secs 2024-04-26T13:15:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5069.501926 secs 2024-04-26T13:15:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:30 smithi194 ceph-mon[33017]: pgmap v3350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:30 smithi033 ceph-mon[27845]: pgmap v3350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:30 smithi096 ceph-mon[32753]: pgmap v3350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:32 smithi194 ceph-mon[33017]: pgmap v3351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:32 smithi033 ceph-mon[27845]: pgmap v3351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:32 smithi096 ceph-mon[32753]: pgmap v3351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5071.760146 secs 2024-04-26T13:15:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5071.760146 secs 2024-04-26T13:15:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5071.760146 secs 2024-04-26T13:15:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5074.502041 secs 2024-04-26T13:15:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:34 smithi033 ceph-mon[27845]: pgmap v3352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5074.502041 secs 2024-04-26T13:15:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:34 smithi096 ceph-mon[32753]: pgmap v3352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5074.502041 secs 2024-04-26T13:15:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:34 smithi194 ceph-mon[33017]: pgmap v3352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:36.397+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:36.397+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:36.397+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:15:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:36 smithi033 ceph-mon[27845]: pgmap v3353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:36 smithi096 ceph-mon[32753]: pgmap v3353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:36 smithi194 ceph-mon[33017]: pgmap v3353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:15:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:15:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:15:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:38 smithi033 ceph-mon[27845]: pgmap v3354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5076.760211 secs 2024-04-26T13:15:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:38 smithi096 ceph-mon[32753]: pgmap v3354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5076.760211 secs 2024-04-26T13:15:39.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:38 smithi194 ceph-mon[33017]: pgmap v3354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:39.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5076.760211 secs 2024-04-26T13:15:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5079.502162 secs 2024-04-26T13:15:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:15:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:15:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:39 smithi033 ceph-mon[27845]: pgmap v3355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:39 smithi033 ceph-mon[27845]: pgmap v3356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:15:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5079.502162 secs 2024-04-26T13:15:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:15:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:15:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:39 smithi096 ceph-mon[32753]: pgmap v3355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:39 smithi096 ceph-mon[32753]: pgmap v3356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5079.502162 secs 2024-04-26T13:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:39 smithi194 ceph-mon[33017]: pgmap v3355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:39 smithi194 ceph-mon[33017]: pgmap v3356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:15:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:41 smithi033 ceph-mon[27845]: pgmap v3357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:41 smithi096 ceph-mon[32753]: pgmap v3357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:41 smithi194 ceph-mon[33017]: pgmap v3357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5081.760297 secs 2024-04-26T13:15:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:43 smithi033 ceph-mon[27845]: pgmap v3358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5084.502297 secs 2024-04-26T13:15:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5081.760297 secs 2024-04-26T13:15:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:43 smithi096 ceph-mon[32753]: pgmap v3358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5084.502297 secs 2024-04-26T13:15:44.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5081.760297 secs 2024-04-26T13:15:44.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:43 smithi194 ceph-mon[33017]: pgmap v3358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:44.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5084.502297 secs 2024-04-26T13:15:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:45 smithi033 ceph-mon[27845]: pgmap v3359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:45 smithi096 ceph-mon[32753]: pgmap v3359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:45 smithi194 ceph-mon[33017]: pgmap v3359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:47 smithi033 ceph-mon[27845]: pgmap v3360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:47 smithi096 ceph-mon[32753]: pgmap v3360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:47 smithi194 ceph-mon[33017]: pgmap v3360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:15:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5086.760395 secs 2024-04-26T13:15:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5089.502407 secs 2024-04-26T13:15:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5086.760395 secs 2024-04-26T13:15:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5089.502407 secs 2024-04-26T13:15:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5086.760395 secs 2024-04-26T13:15:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5089.502407 secs 2024-04-26T13:15:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:49 smithi033 ceph-mon[27845]: pgmap v3361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:15:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:49 smithi096 ceph-mon[32753]: pgmap v3361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:15:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:49 smithi194 ceph-mon[33017]: pgmap v3361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:15:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:51.413+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:51.413+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:15:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:15:51.413+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:15:52.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:51 smithi033 ceph-mon[27845]: pgmap v3362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:51 smithi096 ceph-mon[32753]: pgmap v3362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:51 smithi194 ceph-mon[33017]: pgmap v3362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5091.760497 secs 2024-04-26T13:15:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:53 smithi033 ceph-mon[27845]: pgmap v3363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5094.502522 secs 2024-04-26T13:15:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5091.760497 secs 2024-04-26T13:15:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:53 smithi096 ceph-mon[32753]: pgmap v3363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5094.502522 secs 2024-04-26T13:15:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5091.760497 secs 2024-04-26T13:15:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:53 smithi194 ceph-mon[33017]: pgmap v3363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5094.502522 secs 2024-04-26T13:15:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:55 smithi033 ceph-mon[27845]: pgmap v3364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:55 smithi096 ceph-mon[32753]: pgmap v3364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:55 smithi194 ceph-mon[33017]: pgmap v3364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:15:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:57 smithi033 ceph-mon[27845]: pgmap v3365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:57 smithi096 ceph-mon[32753]: pgmap v3365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:57 smithi194 ceph-mon[33017]: pgmap v3365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:15:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5096.760600 secs 2024-04-26T13:15:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5099.502638 secs 2024-04-26T13:15:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5096.760600 secs 2024-04-26T13:15:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5099.502638 secs 2024-04-26T13:15:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5096.760600 secs 2024-04-26T13:15:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5099.502638 secs 2024-04-26T13:16:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:15:59 smithi096 ceph-mon[32753]: pgmap v3366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:15:59 smithi194 ceph-mon[33017]: pgmap v3366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:15:59 smithi033 ceph-mon[27845]: pgmap v3366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:01 smithi096 ceph-mon[32753]: pgmap v3367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:01 smithi194 ceph-mon[33017]: pgmap v3367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:01 smithi033 ceph-mon[27845]: pgmap v3367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5101.760658 secs 2024-04-26T13:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:03 smithi194 ceph-mon[33017]: pgmap v3368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5104.502787 secs 2024-04-26T13:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5101.760658 secs 2024-04-26T13:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:03 smithi033 ceph-mon[27845]: pgmap v3368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5104.502787 secs 2024-04-26T13:16:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5101.760658 secs 2024-04-26T13:16:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:03 smithi096 ceph-mon[32753]: pgmap v3368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5104.502787 secs 2024-04-26T13:16:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:05 smithi194 ceph-mon[33017]: pgmap v3369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:06.421+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:06.421+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:06.421+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:16:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:05 smithi033 ceph-mon[27845]: pgmap v3369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:05 smithi096 ceph-mon[32753]: pgmap v3369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:07 smithi194 ceph-mon[33017]: pgmap v3370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:07 smithi033 ceph-mon[27845]: pgmap v3370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:07 smithi096 ceph-mon[32753]: pgmap v3370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:09.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5106.760741 secs 2024-04-26T13:16:09.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5109.502904 secs 2024-04-26T13:16:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5106.760741 secs 2024-04-26T13:16:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5109.502904 secs 2024-04-26T13:16:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5106.760741 secs 2024-04-26T13:16:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5109.502904 secs 2024-04-26T13:16:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:09 smithi194 ceph-mon[33017]: pgmap v3371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:09 smithi033 ceph-mon[27845]: pgmap v3371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:09 smithi096 ceph-mon[32753]: pgmap v3371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:11 smithi194 ceph-mon[33017]: pgmap v3372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:11 smithi033 ceph-mon[27845]: pgmap v3372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:11 smithi096 ceph-mon[32753]: pgmap v3372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5111.760824 secs 2024-04-26T13:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:13 smithi194 ceph-mon[33017]: pgmap v3373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5114.503026 secs 2024-04-26T13:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5111.760824 secs 2024-04-26T13:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:13 smithi033 ceph-mon[27845]: pgmap v3373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5114.503026 secs 2024-04-26T13:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5111.760824 secs 2024-04-26T13:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:13 smithi096 ceph-mon[32753]: pgmap v3373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5114.503026 secs 2024-04-26T13:16:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:15 smithi194 ceph-mon[33017]: pgmap v3374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:15 smithi033 ceph-mon[27845]: pgmap v3374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:15 smithi096 ceph-mon[32753]: pgmap v3374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:18.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:17 smithi194 ceph-mon[33017]: pgmap v3375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:17 smithi033 ceph-mon[27845]: pgmap v3375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:17 smithi096 ceph-mon[32753]: pgmap v3375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5116.760908 secs 2024-04-26T13:16:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5119.503161 secs 2024-04-26T13:16:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5116.760908 secs 2024-04-26T13:16:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5119.503161 secs 2024-04-26T13:16:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5116.760908 secs 2024-04-26T13:16:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5119.503161 secs 2024-04-26T13:16:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:20 smithi194 ceph-mon[33017]: pgmap v3376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:20 smithi033 ceph-mon[27845]: pgmap v3376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:20 smithi096 ceph-mon[32753]: pgmap v3376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:21.432+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:21.432+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:21.432+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:16:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:22 smithi194 ceph-mon[33017]: pgmap v3377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:22 smithi033 ceph-mon[27845]: pgmap v3377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:22 smithi096 ceph-mon[32753]: pgmap v3377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5121.761010 secs 2024-04-26T13:16:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:24 smithi194 ceph-mon[33017]: pgmap v3378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5124.503289 secs 2024-04-26T13:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5121.761010 secs 2024-04-26T13:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:24 smithi033 ceph-mon[27845]: pgmap v3378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5124.503289 secs 2024-04-26T13:16:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5121.761010 secs 2024-04-26T13:16:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:24 smithi096 ceph-mon[32753]: pgmap v3378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5124.503289 secs 2024-04-26T13:16:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:26 smithi194 ceph-mon[33017]: pgmap v3379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:26 smithi033 ceph-mon[27845]: pgmap v3379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:26 smithi096 ceph-mon[32753]: pgmap v3379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:28 smithi194 ceph-mon[33017]: pgmap v3380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:28 smithi033 ceph-mon[27845]: pgmap v3380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:28 smithi096 ceph-mon[32753]: pgmap v3380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5126.761072 secs 2024-04-26T13:16:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5129.503396 secs 2024-04-26T13:16:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5126.761072 secs 2024-04-26T13:16:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5129.503396 secs 2024-04-26T13:16:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5126.761072 secs 2024-04-26T13:16:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5129.503396 secs 2024-04-26T13:16:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:30 smithi194 ceph-mon[33017]: pgmap v3381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:30 smithi033 ceph-mon[27845]: pgmap v3381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:30 smithi096 ceph-mon[32753]: pgmap v3381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:32 smithi033 ceph-mon[27845]: pgmap v3382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:32 smithi194 ceph-mon[33017]: pgmap v3382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:32 smithi096 ceph-mon[32753]: pgmap v3382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5131.761173 secs 2024-04-26T13:16:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:34 smithi194 ceph-mon[33017]: pgmap v3383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5134.503505 secs 2024-04-26T13:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5131.761173 secs 2024-04-26T13:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:34 smithi033 ceph-mon[27845]: pgmap v3383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5134.503505 secs 2024-04-26T13:16:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5131.761173 secs 2024-04-26T13:16:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:34 smithi096 ceph-mon[32753]: pgmap v3383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5134.503505 secs 2024-04-26T13:16:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:36 smithi194 ceph-mon[33017]: pgmap v3384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:36 smithi033 ceph-mon[27845]: pgmap v3384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:36 smithi096 ceph-mon[32753]: pgmap v3384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:36.448+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:36.448+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:36.448+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:16:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:38 smithi194 ceph-mon[33017]: pgmap v3385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:38 smithi033 ceph-mon[27845]: pgmap v3385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:38 smithi096 ceph-mon[32753]: pgmap v3385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:39.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5136.761234 secs 2024-04-26T13:16:39.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5139.503644 secs 2024-04-26T13:16:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5136.761234 secs 2024-04-26T13:16:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5139.503644 secs 2024-04-26T13:16:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5136.761234 secs 2024-04-26T13:16:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5139.503644 secs 2024-04-26T13:16:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:40 smithi194 ceph-mon[33017]: pgmap v3386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:16:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:40 smithi033 ceph-mon[27845]: pgmap v3386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:16:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:40 smithi096 ceph-mon[32753]: pgmap v3386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:16:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:16:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:16:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:16:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:16:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:16:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:16:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:16:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:16:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:16:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:42 smithi194 ceph-mon[33017]: pgmap v3387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:16:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:42 smithi194 ceph-mon[33017]: pgmap v3388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:42 smithi033 ceph-mon[27845]: pgmap v3387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:16:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:42 smithi033 ceph-mon[27845]: pgmap v3388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:42 smithi096 ceph-mon[32753]: pgmap v3387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:16:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:42 smithi096 ceph-mon[32753]: pgmap v3388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5141.761295 secs 2024-04-26T13:16:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5141.761295 secs 2024-04-26T13:16:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5141.761295 secs 2024-04-26T13:16:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:44 smithi194 ceph-mon[33017]: pgmap v3389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5144.503778 secs 2024-04-26T13:16:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:44 smithi033 ceph-mon[27845]: pgmap v3389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5144.503778 secs 2024-04-26T13:16:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:44 smithi096 ceph-mon[32753]: pgmap v3389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5144.503778 secs 2024-04-26T13:16:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:46 smithi194 ceph-mon[33017]: pgmap v3390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:46 smithi033 ceph-mon[27845]: pgmap v3390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:46 smithi096 ceph-mon[32753]: pgmap v3390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:48 smithi194 ceph-mon[33017]: pgmap v3391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5146.761382 secs 2024-04-26T13:16:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:48 smithi033 ceph-mon[27845]: pgmap v3391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5146.761382 secs 2024-04-26T13:16:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:48 smithi096 ceph-mon[32753]: pgmap v3391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5146.761382 secs 2024-04-26T13:16:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5149.503911 secs 2024-04-26T13:16:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5149.503911 secs 2024-04-26T13:16:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5149.503911 secs 2024-04-26T13:16:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:50 smithi033 ceph-mon[27845]: pgmap v3392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:50 smithi096 ceph-mon[32753]: pgmap v3392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:50 smithi194 ceph-mon[33017]: pgmap v3392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:16:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:51.464+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:51.464+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:16:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:16:51.464+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:16:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:52 smithi033 ceph-mon[27845]: pgmap v3393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:16:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:52 smithi096 ceph-mon[32753]: pgmap v3393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:16:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:52 smithi194 ceph-mon[33017]: pgmap v3393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:16:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5151.761469 secs 2024-04-26T13:16:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5151.761469 secs 2024-04-26T13:16:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5151.761469 secs 2024-04-26T13:16:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:54 smithi033 ceph-mon[27845]: pgmap v3394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5154.504001 secs 2024-04-26T13:16:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:54 smithi096 ceph-mon[32753]: pgmap v3394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5154.504001 secs 2024-04-26T13:16:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:54 smithi194 ceph-mon[33017]: pgmap v3394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5154.504001 secs 2024-04-26T13:16:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:56 smithi033 ceph-mon[27845]: pgmap v3395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:56 smithi096 ceph-mon[32753]: pgmap v3395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:56 smithi194 ceph-mon[33017]: pgmap v3395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:16:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:58 smithi033 ceph-mon[27845]: pgmap v3396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5156.761551 secs 2024-04-26T13:16:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:58 smithi096 ceph-mon[32753]: pgmap v3396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5156.761551 secs 2024-04-26T13:16:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:58 smithi194 ceph-mon[33017]: pgmap v3396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:16:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5156.761551 secs 2024-04-26T13:16:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5159.504152 secs 2024-04-26T13:16:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5159.504152 secs 2024-04-26T13:16:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5159.504152 secs 2024-04-26T13:17:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:00 smithi033 ceph-mon[27845]: pgmap v3397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:00 smithi096 ceph-mon[32753]: pgmap v3397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:00 smithi194 ceph-mon[33017]: pgmap v3397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:01 smithi033 ceph-mon[27845]: pgmap v3398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:01 smithi096 ceph-mon[32753]: pgmap v3398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:01 smithi194 ceph-mon[33017]: pgmap v3398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:03 smithi194 ceph-mon[33017]: pgmap v3399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5161.761636 secs 2024-04-26T13:17:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:03 smithi033 ceph-mon[27845]: pgmap v3399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5161.761636 secs 2024-04-26T13:17:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:03 smithi096 ceph-mon[32753]: pgmap v3399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5161.761636 secs 2024-04-26T13:17:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5164.504212 secs 2024-04-26T13:17:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5164.504212 secs 2024-04-26T13:17:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5164.504212 secs 2024-04-26T13:17:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:05 smithi194 ceph-mon[33017]: pgmap v3400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:05 smithi033 ceph-mon[27845]: pgmap v3400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:05 smithi096 ceph-mon[32753]: pgmap v3400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:06.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:06.477+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:06.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:06.477+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:17:06.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:06.477+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:07 smithi194 ceph-mon[33017]: pgmap v3401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:07 smithi033 ceph-mon[27845]: pgmap v3401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:07 smithi096 ceph-mon[32753]: pgmap v3401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5166.761738 secs 2024-04-26T13:17:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5166.761738 secs 2024-04-26T13:17:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5166.761738 secs 2024-04-26T13:17:09.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5169.504331 secs 2024-04-26T13:17:09.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:09 smithi033 ceph-mon[27845]: pgmap v3402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5169.504331 secs 2024-04-26T13:17:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:09 smithi194 ceph-mon[33017]: pgmap v3402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5169.504331 secs 2024-04-26T13:17:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:09 smithi096 ceph-mon[32753]: pgmap v3402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:11 smithi194 ceph-mon[33017]: pgmap v3403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:11 smithi033 ceph-mon[27845]: pgmap v3403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:11 smithi096 ceph-mon[32753]: pgmap v3403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:13 smithi194 ceph-mon[33017]: pgmap v3404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5171.761826 secs 2024-04-26T13:17:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:13 smithi033 ceph-mon[27845]: pgmap v3404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5171.761826 secs 2024-04-26T13:17:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:13 smithi096 ceph-mon[32753]: pgmap v3404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5171.761826 secs 2024-04-26T13:17:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5174.504441 secs 2024-04-26T13:17:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5174.504441 secs 2024-04-26T13:17:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5174.504441 secs 2024-04-26T13:17:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:15 smithi033 ceph-mon[27845]: pgmap v3405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:15 smithi096 ceph-mon[32753]: pgmap v3405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:15 smithi194 ceph-mon[33017]: pgmap v3405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:17 smithi096 ceph-mon[32753]: pgmap v3406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:17 smithi194 ceph-mon[33017]: pgmap v3406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:17 smithi033 ceph-mon[27845]: pgmap v3406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:19.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5176.761914 secs 2024-04-26T13:17:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5176.761914 secs 2024-04-26T13:17:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5176.761914 secs 2024-04-26T13:17:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5179.504571 secs 2024-04-26T13:17:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:19 smithi194 ceph-mon[33017]: pgmap v3407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5179.504571 secs 2024-04-26T13:17:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:19 smithi033 ceph-mon[27845]: pgmap v3407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5179.504571 secs 2024-04-26T13:17:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:19 smithi096 ceph-mon[32753]: pgmap v3407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:21.481+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:21.481+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:21.481+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:17:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:21 smithi194 ceph-mon[33017]: pgmap v3408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:21 smithi033 ceph-mon[27845]: pgmap v3408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:21 smithi096 ceph-mon[32753]: pgmap v3408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:22.671+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:17:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:22.671+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:17:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:22.671+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T13:17:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:22.672+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T13:17:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:23 smithi194 ceph-mon[33017]: pgmap v3409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5181.762017 secs 2024-04-26T13:17:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:23 smithi033 ceph-mon[27845]: pgmap v3409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5181.762017 secs 2024-04-26T13:17:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:23 smithi096 ceph-mon[32753]: pgmap v3409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5181.762017 secs 2024-04-26T13:17:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5184.504694 secs 2024-04-26T13:17:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5184.504694 secs 2024-04-26T13:17:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5184.504694 secs 2024-04-26T13:17:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:25 smithi194 ceph-mon[33017]: pgmap v3410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:25 smithi033 ceph-mon[27845]: pgmap v3410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:25 smithi096 ceph-mon[32753]: pgmap v3410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:27 smithi194 ceph-mon[33017]: pgmap v3411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:27 smithi033 ceph-mon[27845]: pgmap v3411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:27 smithi096 ceph-mon[32753]: pgmap v3411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5186.762103 secs 2024-04-26T13:17:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5186.762103 secs 2024-04-26T13:17:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5186.762103 secs 2024-04-26T13:17:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5189.504834 secs 2024-04-26T13:17:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:29 smithi194 ceph-mon[33017]: pgmap v3412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5189.504834 secs 2024-04-26T13:17:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:29 smithi033 ceph-mon[27845]: pgmap v3412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5189.504834 secs 2024-04-26T13:17:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:29 smithi096 ceph-mon[32753]: pgmap v3412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:31 smithi194 ceph-mon[33017]: pgmap v3413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:31 smithi033 ceph-mon[27845]: pgmap v3413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:31 smithi096 ceph-mon[32753]: pgmap v3413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:33 smithi194 ceph-mon[33017]: pgmap v3414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5191.762191 secs 2024-04-26T13:17:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:33 smithi033 ceph-mon[27845]: pgmap v3414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5191.762191 secs 2024-04-26T13:17:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:33 smithi096 ceph-mon[32753]: pgmap v3414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5191.762191 secs 2024-04-26T13:17:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5194.504953 secs 2024-04-26T13:17:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5194.504953 secs 2024-04-26T13:17:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5194.504953 secs 2024-04-26T13:17:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:35 smithi194 ceph-mon[33017]: pgmap v3415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:35 smithi033 ceph-mon[27845]: pgmap v3415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:35 smithi096 ceph-mon[32753]: pgmap v3415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:36.483+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:36.483+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:36.483+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:17:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:37 smithi194 ceph-mon[33017]: pgmap v3416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:37 smithi033 ceph-mon[27845]: pgmap v3416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:37 smithi096 ceph-mon[32753]: pgmap v3416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:39.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5196.762296 secs 2024-04-26T13:17:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5196.762296 secs 2024-04-26T13:17:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5196.762296 secs 2024-04-26T13:17:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5199.505096 secs 2024-04-26T13:17:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:39 smithi194 ceph-mon[33017]: pgmap v3417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5199.505096 secs 2024-04-26T13:17:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:39 smithi033 ceph-mon[27845]: pgmap v3417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5199.505096 secs 2024-04-26T13:17:40.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:39 smithi096 ceph-mon[32753]: pgmap v3417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:17:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:17:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:17:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:41 smithi194 ceph-mon[33017]: pgmap v3418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:41 smithi033 ceph-mon[27845]: pgmap v3418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:41 smithi096 ceph-mon[32753]: pgmap v3418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:17:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:17:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:42 smithi194 ceph-mon[33017]: pgmap v3419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:42 smithi194 ceph-mon[33017]: pgmap v3420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:17:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:17:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:17:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:42 smithi033 ceph-mon[27845]: pgmap v3419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:42 smithi033 ceph-mon[27845]: pgmap v3420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:17:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:17:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:17:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:42 smithi096 ceph-mon[32753]: pgmap v3419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:42 smithi096 ceph-mon[32753]: pgmap v3420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:17:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5201.762387 secs 2024-04-26T13:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5201.762387 secs 2024-04-26T13:17:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5201.762387 secs 2024-04-26T13:17:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5204.505220 secs 2024-04-26T13:17:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:44 smithi194 ceph-mon[33017]: pgmap v3421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:17:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5204.505220 secs 2024-04-26T13:17:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:44 smithi033 ceph-mon[27845]: pgmap v3421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:17:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5204.505220 secs 2024-04-26T13:17:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:44 smithi096 ceph-mon[32753]: pgmap v3421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:17:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:47 smithi033 ceph-mon[27845]: pgmap v3422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:47 smithi096 ceph-mon[32753]: pgmap v3422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:47 smithi194 ceph-mon[33017]: pgmap v3422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5206.762445 secs 2024-04-26T13:17:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5206.762445 secs 2024-04-26T13:17:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5206.762445 secs 2024-04-26T13:17:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5209.505305 secs 2024-04-26T13:17:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:49 smithi033 ceph-mon[27845]: pgmap v3423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5209.505305 secs 2024-04-26T13:17:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:49 smithi096 ceph-mon[32753]: pgmap v3423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5209.505305 secs 2024-04-26T13:17:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:49 smithi194 ceph-mon[33017]: pgmap v3423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:51 smithi033 ceph-mon[27845]: pgmap v3424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:51 smithi096 ceph-mon[32753]: pgmap v3424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:51.500+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:51.500+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:17:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:17:51.500+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:17:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:51 smithi194 ceph-mon[33017]: pgmap v3424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:17:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:52 smithi033 ceph-mon[27845]: pgmap v3425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:17:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:52 smithi096 ceph-mon[32753]: pgmap v3425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:17:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:52 smithi194 ceph-mon[33017]: pgmap v3425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:17:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5211.762531 secs 2024-04-26T13:17:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5211.762531 secs 2024-04-26T13:17:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5211.762531 secs 2024-04-26T13:17:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5214.505379 secs 2024-04-26T13:17:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:54 smithi096 ceph-mon[32753]: pgmap v3426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5214.505379 secs 2024-04-26T13:17:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:54 smithi194 ceph-mon[33017]: pgmap v3426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5214.505379 secs 2024-04-26T13:17:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:54 smithi033 ceph-mon[27845]: pgmap v3426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:57 smithi033 ceph-mon[27845]: pgmap v3427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:57 smithi096 ceph-mon[32753]: pgmap v3427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:57 smithi194 ceph-mon[33017]: pgmap v3427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:17:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5216.762616 secs 2024-04-26T13:17:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5216.762616 secs 2024-04-26T13:17:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5216.762616 secs 2024-04-26T13:17:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5219.505497 secs 2024-04-26T13:17:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:17:59 smithi033 ceph-mon[27845]: pgmap v3428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5219.505497 secs 2024-04-26T13:17:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:17:59 smithi096 ceph-mon[32753]: pgmap v3428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:17:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5219.505497 secs 2024-04-26T13:17:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:17:59 smithi194 ceph-mon[33017]: pgmap v3428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:00 smithi033 ceph-mon[27845]: pgmap v3429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:00 smithi096 ceph-mon[32753]: pgmap v3429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:00 smithi194 ceph-mon[33017]: pgmap v3429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:03 smithi033 ceph-mon[27845]: pgmap v3430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5221.762701 secs 2024-04-26T13:18:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:03 smithi096 ceph-mon[32753]: pgmap v3430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5221.762701 secs 2024-04-26T13:18:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:03 smithi194 ceph-mon[33017]: pgmap v3430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5221.762701 secs 2024-04-26T13:18:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5224.505620 secs 2024-04-26T13:18:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5224.505620 secs 2024-04-26T13:18:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5224.505620 secs 2024-04-26T13:18:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:05 smithi033 ceph-mon[27845]: pgmap v3431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:05 smithi096 ceph-mon[32753]: pgmap v3431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:05 smithi194 ceph-mon[33017]: pgmap v3431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:06 smithi096 ceph-mon[32753]: pgmap v3432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:06.515+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:06.515+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:06.515+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:18:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:06 smithi194 ceph-mon[33017]: pgmap v3432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:06 smithi033 ceph-mon[27845]: pgmap v3432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5226.762785 secs 2024-04-26T13:18:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5226.762785 secs 2024-04-26T13:18:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5226.762785 secs 2024-04-26T13:18:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5229.505743 secs 2024-04-26T13:18:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:09 smithi033 ceph-mon[27845]: pgmap v3433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5229.505743 secs 2024-04-26T13:18:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:09 smithi096 ceph-mon[32753]: pgmap v3433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5229.505743 secs 2024-04-26T13:18:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:09 smithi194 ceph-mon[33017]: pgmap v3433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:10 smithi194 ceph-mon[33017]: pgmap v3434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:10 smithi033 ceph-mon[27845]: pgmap v3434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:10 smithi096 ceph-mon[32753]: pgmap v3434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:13 smithi033 ceph-mon[27845]: pgmap v3435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5231.762845 secs 2024-04-26T13:18:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:13 smithi096 ceph-mon[32753]: pgmap v3435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5231.762845 secs 2024-04-26T13:18:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:13 smithi194 ceph-mon[33017]: pgmap v3435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5231.762845 secs 2024-04-26T13:18:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5234.505864 secs 2024-04-26T13:18:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:14 smithi194 ceph-mon[33017]: pgmap v3436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5234.505864 secs 2024-04-26T13:18:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:14 smithi033 ceph-mon[27845]: pgmap v3436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5234.505864 secs 2024-04-26T13:18:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:14 smithi096 ceph-mon[32753]: pgmap v3436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:17 smithi033 ceph-mon[27845]: pgmap v3437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:17 smithi096 ceph-mon[32753]: pgmap v3437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:17 smithi194 ceph-mon[33017]: pgmap v3437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5236.762901 secs 2024-04-26T13:18:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:18 smithi033 ceph-mon[27845]: pgmap v3438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5236.762901 secs 2024-04-26T13:18:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:18 smithi096 ceph-mon[32753]: pgmap v3438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5236.762901 secs 2024-04-26T13:18:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:18 smithi194 ceph-mon[33017]: pgmap v3438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5239.505990 secs 2024-04-26T13:18:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5239.505990 secs 2024-04-26T13:18:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5239.505990 secs 2024-04-26T13:18:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:20 smithi194 ceph-mon[33017]: pgmap v3439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:20 smithi033 ceph-mon[27845]: pgmap v3439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:20 smithi096 ceph-mon[32753]: pgmap v3439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:21.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:21.521+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:21.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:21.521+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:21.521+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:18:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:22 smithi194 ceph-mon[33017]: pgmap v3440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:22 smithi033 ceph-mon[27845]: pgmap v3440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:22 smithi096 ceph-mon[32753]: pgmap v3440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5241.762982 secs 2024-04-26T13:18:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5241.762982 secs 2024-04-26T13:18:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5241.762982 secs 2024-04-26T13:18:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5244.506156 secs 2024-04-26T13:18:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:24 smithi194 ceph-mon[33017]: pgmap v3441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5244.506156 secs 2024-04-26T13:18:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:24 smithi033 ceph-mon[27845]: pgmap v3441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5244.506156 secs 2024-04-26T13:18:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:24 smithi096 ceph-mon[32753]: pgmap v3441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:26 smithi194 ceph-mon[33017]: pgmap v3442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:26 smithi033 ceph-mon[27845]: pgmap v3442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:26 smithi096 ceph-mon[32753]: pgmap v3442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5246.763040 secs 2024-04-26T13:18:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5246.763040 secs 2024-04-26T13:18:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5246.763040 secs 2024-04-26T13:18:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5249.506291 secs 2024-04-26T13:18:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:29 smithi033 ceph-mon[27845]: pgmap v3443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5249.506291 secs 2024-04-26T13:18:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:29 smithi096 ceph-mon[32753]: pgmap v3443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5249.506291 secs 2024-04-26T13:18:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:29 smithi194 ceph-mon[33017]: pgmap v3443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:31 smithi033 ceph-mon[27845]: pgmap v3444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:31 smithi096 ceph-mon[32753]: pgmap v3444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:31 smithi194 ceph-mon[33017]: pgmap v3444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:33 smithi033 ceph-mon[27845]: pgmap v3445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5251.763127 secs 2024-04-26T13:18:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:33 smithi096 ceph-mon[32753]: pgmap v3445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5251.763127 secs 2024-04-26T13:18:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:33 smithi194 ceph-mon[33017]: pgmap v3445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5251.763127 secs 2024-04-26T13:18:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5254.506414 secs 2024-04-26T13:18:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5254.506414 secs 2024-04-26T13:18:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5254.506414 secs 2024-04-26T13:18:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:35 smithi033 ceph-mon[27845]: pgmap v3446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:35 smithi096 ceph-mon[32753]: pgmap v3446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:35 smithi194 ceph-mon[33017]: pgmap v3446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:36.535+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:36.535+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:36.535+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:18:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:37 smithi033 ceph-mon[27845]: pgmap v3447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:37 smithi096 ceph-mon[32753]: pgmap v3447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:37 smithi194 ceph-mon[33017]: pgmap v3447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5256.763190 secs 2024-04-26T13:18:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5256.763190 secs 2024-04-26T13:18:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5256.763190 secs 2024-04-26T13:18:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5259.506538 secs 2024-04-26T13:18:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:39 smithi033 ceph-mon[27845]: pgmap v3448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5259.506538 secs 2024-04-26T13:18:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:39 smithi096 ceph-mon[32753]: pgmap v3448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:39.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5259.506538 secs 2024-04-26T13:18:39.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:39 smithi194 ceph-mon[33017]: pgmap v3448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:40 smithi194 ceph-mon[33017]: pgmap v3449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:40 smithi033 ceph-mon[27845]: pgmap v3449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:40 smithi096 ceph-mon[32753]: pgmap v3449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:42 smithi194 ceph-mon[33017]: pgmap v3450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:42 smithi033 ceph-mon[27845]: pgmap v3450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:42 smithi096 ceph-mon[32753]: pgmap v3450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:18:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5261.763256 secs 2024-04-26T13:18:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:18:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5261.763256 secs 2024-04-26T13:18:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:18:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5261.763256 secs 2024-04-26T13:18:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5264.506627 secs 2024-04-26T13:18:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:18:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:18:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:44 smithi194 ceph-mon[33017]: pgmap v3451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:44 smithi194 ceph-mon[33017]: pgmap v3452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5264.506627 secs 2024-04-26T13:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:44 smithi033 ceph-mon[27845]: pgmap v3451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:44 smithi033 ceph-mon[27845]: pgmap v3452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5264.506627 secs 2024-04-26T13:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:44 smithi096 ceph-mon[32753]: pgmap v3451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:44 smithi096 ceph-mon[32753]: pgmap v3452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:18:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:46 smithi033 ceph-mon[27845]: pgmap v3453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:46 smithi096 ceph-mon[32753]: pgmap v3453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:46 smithi194 ceph-mon[33017]: pgmap v3453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5266.763344 secs 2024-04-26T13:18:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:48 smithi033 ceph-mon[27845]: pgmap v3454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5266.763344 secs 2024-04-26T13:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:48 smithi096 ceph-mon[32753]: pgmap v3454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5266.763344 secs 2024-04-26T13:18:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:48 smithi194 ceph-mon[33017]: pgmap v3454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5269.506750 secs 2024-04-26T13:18:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5269.506750 secs 2024-04-26T13:18:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5269.506750 secs 2024-04-26T13:18:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:50 smithi033 ceph-mon[27845]: pgmap v3455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:50 smithi096 ceph-mon[32753]: pgmap v3455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:50 smithi194 ceph-mon[33017]: pgmap v3455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:51.541+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:51.541+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:18:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:18:51.541+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:18:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:52 smithi033 ceph-mon[27845]: pgmap v3456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:52 smithi096 ceph-mon[32753]: pgmap v3456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:52 smithi194 ceph-mon[33017]: pgmap v3456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:18:54.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5271.763441 secs 2024-04-26T13:18:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5271.763441 secs 2024-04-26T13:18:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5271.763441 secs 2024-04-26T13:18:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5274.506871 secs 2024-04-26T13:18:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:54 smithi033 ceph-mon[27845]: pgmap v3457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:18:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5274.506871 secs 2024-04-26T13:18:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:54 smithi096 ceph-mon[32753]: pgmap v3457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:18:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5274.506871 secs 2024-04-26T13:18:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:54 smithi194 ceph-mon[33017]: pgmap v3457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:18:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:56 smithi033 ceph-mon[27845]: pgmap v3458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:56 smithi096 ceph-mon[32753]: pgmap v3458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:56 smithi194 ceph-mon[33017]: pgmap v3458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:18:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5276.763502 secs 2024-04-26T13:18:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:58 smithi033 ceph-mon[27845]: pgmap v3459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:18:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5279.507031 secs 2024-04-26T13:18:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5276.763502 secs 2024-04-26T13:18:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:58 smithi096 ceph-mon[32753]: pgmap v3459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:18:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5279.507031 secs 2024-04-26T13:18:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5276.763502 secs 2024-04-26T13:18:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:58 smithi194 ceph-mon[33017]: pgmap v3459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:18:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:18:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5279.507031 secs 2024-04-26T13:19:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:00 smithi033 ceph-mon[27845]: pgmap v3460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:00 smithi096 ceph-mon[32753]: pgmap v3460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:00 smithi194 ceph-mon[33017]: pgmap v3460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:02 smithi033 ceph-mon[27845]: pgmap v3461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:02 smithi096 ceph-mon[32753]: pgmap v3461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:02 smithi194 ceph-mon[33017]: pgmap v3461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5281.763603 secs 2024-04-26T13:19:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5284.507129 secs 2024-04-26T13:19:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5281.763603 secs 2024-04-26T13:19:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5284.507129 secs 2024-04-26T13:19:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5281.763603 secs 2024-04-26T13:19:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5284.507129 secs 2024-04-26T13:19:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:04 smithi096 ceph-mon[32753]: pgmap v3462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:04 smithi194 ceph-mon[33017]: pgmap v3462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:04 smithi033 ceph-mon[27845]: pgmap v3462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:06.542+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:06.542+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:06.542+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:19:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:06 smithi194 ceph-mon[33017]: pgmap v3463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:06 smithi033 ceph-mon[27845]: pgmap v3463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:06 smithi096 ceph-mon[32753]: pgmap v3463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5286.763705 secs 2024-04-26T13:19:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:08 smithi033 ceph-mon[27845]: pgmap v3464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5289.507229 secs 2024-04-26T13:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5286.763705 secs 2024-04-26T13:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:08 smithi194 ceph-mon[33017]: pgmap v3464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5289.507229 secs 2024-04-26T13:19:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5286.763705 secs 2024-04-26T13:19:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:08 smithi096 ceph-mon[32753]: pgmap v3464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5289.507229 secs 2024-04-26T13:19:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:10 smithi194 ceph-mon[33017]: pgmap v3465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:10 smithi033 ceph-mon[27845]: pgmap v3465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:10 smithi096 ceph-mon[32753]: pgmap v3465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:12 smithi194 ceph-mon[33017]: pgmap v3466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:12 smithi033 ceph-mon[27845]: pgmap v3466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:12 smithi096 ceph-mon[32753]: pgmap v3466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5291.763803 secs 2024-04-26T13:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5294.507343 secs 2024-04-26T13:19:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5291.763803 secs 2024-04-26T13:19:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5294.507343 secs 2024-04-26T13:19:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5291.763803 secs 2024-04-26T13:19:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5294.507343 secs 2024-04-26T13:19:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:14 smithi194 ceph-mon[33017]: pgmap v3467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:14 smithi033 ceph-mon[27845]: pgmap v3467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:14 smithi096 ceph-mon[32753]: pgmap v3467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:16 smithi194 ceph-mon[33017]: pgmap v3468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:16 smithi033 ceph-mon[27845]: pgmap v3468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:16 smithi096 ceph-mon[32753]: pgmap v3468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:19.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5296.763903 secs 2024-04-26T13:19:19.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:18 smithi096 ceph-mon[32753]: pgmap v3469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:19.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5299.507417 secs 2024-04-26T13:19:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5296.763903 secs 2024-04-26T13:19:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:18 smithi194 ceph-mon[33017]: pgmap v3469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5299.507417 secs 2024-04-26T13:19:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5296.763903 secs 2024-04-26T13:19:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:18 smithi033 ceph-mon[27845]: pgmap v3469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5299.507417 secs 2024-04-26T13:19:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:20 smithi194 ceph-mon[33017]: pgmap v3470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:20 smithi033 ceph-mon[27845]: pgmap v3470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:20 smithi096 ceph-mon[32753]: pgmap v3470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:21.545+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:21.545+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:21.545+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:19:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:22 smithi194 ceph-mon[33017]: pgmap v3471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:22 smithi033 ceph-mon[27845]: pgmap v3471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:22 smithi096 ceph-mon[32753]: pgmap v3471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:24.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5301.763998 secs 2024-04-26T13:19:24.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5304.507528 secs 2024-04-26T13:19:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5301.763998 secs 2024-04-26T13:19:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5304.507528 secs 2024-04-26T13:19:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5301.763998 secs 2024-04-26T13:19:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5304.507528 secs 2024-04-26T13:19:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:24 smithi194 ceph-mon[33017]: pgmap v3472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:24 smithi033 ceph-mon[27845]: pgmap v3472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:24 smithi096 ceph-mon[32753]: pgmap v3472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:27 smithi194 ceph-mon[33017]: pgmap v3473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:27 smithi033 ceph-mon[27845]: pgmap v3473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:27 smithi096 ceph-mon[32753]: pgmap v3473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5306.764086 secs 2024-04-26T13:19:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:29 smithi194 ceph-mon[33017]: pgmap v3474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5309.507645 secs 2024-04-26T13:19:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5306.764086 secs 2024-04-26T13:19:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:29 smithi033 ceph-mon[27845]: pgmap v3474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5309.507645 secs 2024-04-26T13:19:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5306.764086 secs 2024-04-26T13:19:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:29 smithi096 ceph-mon[32753]: pgmap v3474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5309.507645 secs 2024-04-26T13:19:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:31 smithi194 ceph-mon[33017]: pgmap v3475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:31 smithi033 ceph-mon[27845]: pgmap v3475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:31 smithi096 ceph-mon[32753]: pgmap v3475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:33 smithi194 ceph-mon[33017]: pgmap v3476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:33 smithi033 ceph-mon[27845]: pgmap v3476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:33 smithi096 ceph-mon[32753]: pgmap v3476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5311.764173 secs 2024-04-26T13:19:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5314.507768 secs 2024-04-26T13:19:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5311.764173 secs 2024-04-26T13:19:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5314.507768 secs 2024-04-26T13:19:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5311.764173 secs 2024-04-26T13:19:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5314.507768 secs 2024-04-26T13:19:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:35 smithi194 ceph-mon[33017]: pgmap v3477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:35 smithi033 ceph-mon[27845]: pgmap v3477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:35 smithi096 ceph-mon[32753]: pgmap v3477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:36.554+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:36.554+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:36.554+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:19:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:37 smithi194 ceph-mon[33017]: pgmap v3478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:37 smithi033 ceph-mon[27845]: pgmap v3478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:37 smithi096 ceph-mon[32753]: pgmap v3478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:39.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5316.764262 secs 2024-04-26T13:19:39.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:39 smithi033 ceph-mon[27845]: pgmap v3479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:39.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5319.507919 secs 2024-04-26T13:19:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5316.764262 secs 2024-04-26T13:19:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:39 smithi194 ceph-mon[33017]: pgmap v3479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5319.507919 secs 2024-04-26T13:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5316.764262 secs 2024-04-26T13:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:39 smithi096 ceph-mon[32753]: pgmap v3479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5319.507919 secs 2024-04-26T13:19:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:41 smithi194 ceph-mon[33017]: pgmap v3480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:41 smithi033 ceph-mon[27845]: pgmap v3480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:41 smithi096 ceph-mon[32753]: pgmap v3480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:19:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:43 smithi194 ceph-mon[33017]: pgmap v3481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:43 smithi033 ceph-mon[27845]: pgmap v3481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:43 smithi096 ceph-mon[32753]: pgmap v3481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5321.764350 secs 2024-04-26T13:19:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5324.508055 secs 2024-04-26T13:19:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5321.764350 secs 2024-04-26T13:19:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5324.508055 secs 2024-04-26T13:19:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5321.764350 secs 2024-04-26T13:19:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5324.508055 secs 2024-04-26T13:19:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:45 smithi194 ceph-mon[33017]: pgmap v3482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:19:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:45 smithi033 ceph-mon[27845]: pgmap v3482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:19:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:45 smithi096 ceph-mon[32753]: pgmap v3482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:19:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:19:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:19:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:19:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:19:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:19:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:19:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:19:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:19:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:19:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:47 smithi194 ceph-mon[33017]: pgmap v3483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:19:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:47 smithi194 ceph-mon[33017]: pgmap v3484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:47 smithi033 ceph-mon[27845]: pgmap v3483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:19:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:47 smithi033 ceph-mon[27845]: pgmap v3484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:47 smithi096 ceph-mon[32753]: pgmap v3483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:19:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:47 smithi096 ceph-mon[32753]: pgmap v3484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5326.764459 secs 2024-04-26T13:19:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5326.764459 secs 2024-04-26T13:19:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5326.764459 secs 2024-04-26T13:19:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:49 smithi194 ceph-mon[33017]: pgmap v3485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5329.508141 secs 2024-04-26T13:19:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:49 smithi033 ceph-mon[27845]: pgmap v3485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5329.508141 secs 2024-04-26T13:19:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:49 smithi096 ceph-mon[32753]: pgmap v3485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5329.508141 secs 2024-04-26T13:19:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:51 smithi194 ceph-mon[33017]: pgmap v3486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:51 smithi033 ceph-mon[27845]: pgmap v3486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:51 smithi096 ceph-mon[32753]: pgmap v3486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:51.571+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:51.571+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:19:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:19:51.571+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:19:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:53 smithi033 ceph-mon[27845]: pgmap v3487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5331.764548 secs 2024-04-26T13:19:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:53 smithi096 ceph-mon[32753]: pgmap v3487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5331.764548 secs 2024-04-26T13:19:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:53 smithi194 ceph-mon[33017]: pgmap v3487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5331.764548 secs 2024-04-26T13:19:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5334.508253 secs 2024-04-26T13:19:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5334.508253 secs 2024-04-26T13:19:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5334.508253 secs 2024-04-26T13:19:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:55 smithi033 ceph-mon[27845]: pgmap v3488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:55 smithi096 ceph-mon[32753]: pgmap v3488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:55 smithi194 ceph-mon[33017]: pgmap v3488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:19:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:57 smithi033 ceph-mon[27845]: pgmap v3489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:19:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:57 smithi096 ceph-mon[32753]: pgmap v3489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:19:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:57 smithi194 ceph-mon[33017]: pgmap v3489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:19:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5336.764650 secs 2024-04-26T13:19:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5336.764650 secs 2024-04-26T13:19:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5336.764650 secs 2024-04-26T13:19:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:59 smithi033 ceph-mon[27845]: pgmap v3490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:19:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5339.508388 secs 2024-04-26T13:19:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:59 smithi096 ceph-mon[32753]: pgmap v3490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:19:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5339.508388 secs 2024-04-26T13:19:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:59 smithi194 ceph-mon[33017]: pgmap v3490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:19:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:19:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5339.508388 secs 2024-04-26T13:20:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:20:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:20:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:20:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:01 smithi033 ceph-mon[27845]: pgmap v3491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:01 smithi096 ceph-mon[32753]: pgmap v3491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:01 smithi194 ceph-mon[33017]: pgmap v3491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:03 smithi033 ceph-mon[27845]: pgmap v3492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5341.764759 secs 2024-04-26T13:20:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:03 smithi096 ceph-mon[32753]: pgmap v3492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5341.764759 secs 2024-04-26T13:20:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:03 smithi194 ceph-mon[33017]: pgmap v3492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5341.764759 secs 2024-04-26T13:20:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5344.508505 secs 2024-04-26T13:20:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5344.508505 secs 2024-04-26T13:20:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5344.508505 secs 2024-04-26T13:20:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:05 smithi033 ceph-mon[27845]: pgmap v3493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:05 smithi096 ceph-mon[32753]: pgmap v3493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:05 smithi194 ceph-mon[33017]: pgmap v3493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:06.583+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:06.583+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:06.583+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:20:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:07 smithi033 ceph-mon[27845]: pgmap v3494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:07 smithi096 ceph-mon[32753]: pgmap v3494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:07 smithi194 ceph-mon[33017]: pgmap v3494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5346.764857 secs 2024-04-26T13:20:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5346.764857 secs 2024-04-26T13:20:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5346.764857 secs 2024-04-26T13:20:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:09 smithi194 ceph-mon[33017]: pgmap v3495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5349.508630 secs 2024-04-26T13:20:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:09 smithi033 ceph-mon[27845]: pgmap v3495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5349.508630 secs 2024-04-26T13:20:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:09 smithi096 ceph-mon[32753]: pgmap v3495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5349.508630 secs 2024-04-26T13:20:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:10 smithi194 ceph-mon[33017]: pgmap v3496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:10 smithi033 ceph-mon[27845]: pgmap v3496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:10 smithi096 ceph-mon[32753]: pgmap v3496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:12 smithi194 ceph-mon[33017]: pgmap v3497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:12 smithi033 ceph-mon[27845]: pgmap v3497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:12 smithi096 ceph-mon[32753]: pgmap v3497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5351.764955 secs 2024-04-26T13:20:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5351.764955 secs 2024-04-26T13:20:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5351.764955 secs 2024-04-26T13:20:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5354.508740 secs 2024-04-26T13:20:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:14 smithi194 ceph-mon[33017]: pgmap v3498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5354.508740 secs 2024-04-26T13:20:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:14 smithi033 ceph-mon[27845]: pgmap v3498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5354.508740 secs 2024-04-26T13:20:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:14 smithi096 ceph-mon[32753]: pgmap v3498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:16 smithi194 ceph-mon[33017]: pgmap v3499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:16 smithi033 ceph-mon[27845]: pgmap v3499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:16 smithi096 ceph-mon[32753]: pgmap v3499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:18 smithi194 ceph-mon[33017]: pgmap v3500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5356.765005 secs 2024-04-26T13:20:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:18 smithi033 ceph-mon[27845]: pgmap v3500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5356.765005 secs 2024-04-26T13:20:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:18 smithi096 ceph-mon[32753]: pgmap v3500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5356.765005 secs 2024-04-26T13:20:19.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5359.508860 secs 2024-04-26T13:20:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5359.508860 secs 2024-04-26T13:20:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5359.508860 secs 2024-04-26T13:20:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:20 smithi194 ceph-mon[33017]: pgmap v3501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:20 smithi033 ceph-mon[27845]: pgmap v3501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:20 smithi096 ceph-mon[32753]: pgmap v3501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:21.596+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:21.596+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:21.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:21.596+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.700+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.700+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.701+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.701+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.701+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x564385821900 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.701+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0020 con 0x7fef00077780 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.701+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0020 con 0x564385821900 2024-04-26T13:20:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:22.702+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:20:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:22 smithi194 ceph-mon[33017]: pgmap v3502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:22 smithi033 ceph-mon[27845]: pgmap v3502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:22 smithi096 ceph-mon[32753]: pgmap v3502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5361.765105 secs 2024-04-26T13:20:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5361.765105 secs 2024-04-26T13:20:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5361.765105 secs 2024-04-26T13:20:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5364.508995 secs 2024-04-26T13:20:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:24 smithi194 ceph-mon[33017]: pgmap v3503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5364.508995 secs 2024-04-26T13:20:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:24 smithi033 ceph-mon[27845]: pgmap v3503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5364.508995 secs 2024-04-26T13:20:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:24 smithi096 ceph-mon[32753]: pgmap v3503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:26 smithi194 ceph-mon[33017]: pgmap v3504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:26 smithi033 ceph-mon[27845]: pgmap v3504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:26 smithi096 ceph-mon[32753]: pgmap v3504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:28 smithi033 ceph-mon[27845]: pgmap v3505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5366.765165 secs 2024-04-26T13:20:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:28 smithi096 ceph-mon[32753]: pgmap v3505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5366.765165 secs 2024-04-26T13:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:28 smithi194 ceph-mon[33017]: pgmap v3505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5366.765165 secs 2024-04-26T13:20:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5369.509149 secs 2024-04-26T13:20:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5369.509149 secs 2024-04-26T13:20:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5369.509149 secs 2024-04-26T13:20:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:30 smithi033 ceph-mon[27845]: pgmap v3506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:30 smithi096 ceph-mon[32753]: pgmap v3506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:30 smithi194 ceph-mon[33017]: pgmap v3506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:32 smithi033 ceph-mon[27845]: pgmap v3507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:32 smithi096 ceph-mon[32753]: pgmap v3507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:32 smithi194 ceph-mon[33017]: pgmap v3507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:33.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5371.765223 secs 2024-04-26T13:20:33.639 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5371.765223 secs 2024-04-26T13:20:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5371.765223 secs 2024-04-26T13:20:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5374.509269 secs 2024-04-26T13:20:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:34 smithi033 ceph-mon[27845]: pgmap v3508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5374.509269 secs 2024-04-26T13:20:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:34 smithi096 ceph-mon[32753]: pgmap v3508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5374.509269 secs 2024-04-26T13:20:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:34 smithi194 ceph-mon[33017]: pgmap v3508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:36.596+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:36.596+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:36.596+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:20:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:36 smithi033 ceph-mon[27845]: pgmap v3509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:36 smithi096 ceph-mon[32753]: pgmap v3509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:36 smithi194 ceph-mon[33017]: pgmap v3509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:38 smithi033 ceph-mon[27845]: pgmap v3510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5376.765309 secs 2024-04-26T13:20:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:38 smithi096 ceph-mon[32753]: pgmap v3510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5376.765309 secs 2024-04-26T13:20:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:38 smithi194 ceph-mon[33017]: pgmap v3510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5376.765309 secs 2024-04-26T13:20:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5379.509410 secs 2024-04-26T13:20:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5379.509410 secs 2024-04-26T13:20:40.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5379.509410 secs 2024-04-26T13:20:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:40 smithi033 ceph-mon[27845]: pgmap v3511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:40 smithi096 ceph-mon[32753]: pgmap v3511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:40 smithi194 ceph-mon[33017]: pgmap v3511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:42 smithi033 ceph-mon[27845]: pgmap v3512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:42 smithi096 ceph-mon[32753]: pgmap v3512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:42 smithi194 ceph-mon[33017]: pgmap v3512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5381.765399 secs 2024-04-26T13:20:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5381.765399 secs 2024-04-26T13:20:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5381.765399 secs 2024-04-26T13:20:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5384.509501 secs 2024-04-26T13:20:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:44 smithi096 ceph-mon[32753]: pgmap v3513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5384.509501 secs 2024-04-26T13:20:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:44 smithi194 ceph-mon[33017]: pgmap v3513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5384.509501 secs 2024-04-26T13:20:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:44 smithi033 ceph-mon[27845]: pgmap v3513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:46 smithi194 ceph-mon[33017]: pgmap v3514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:20:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:46 smithi033 ceph-mon[27845]: pgmap v3514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:20:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:46 smithi096 ceph-mon[32753]: pgmap v3514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:20:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:47 smithi194 ceph-mon[33017]: pgmap v3515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:47 smithi194 ceph-mon[33017]: pgmap v3516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:20:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:20:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:20:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:47 smithi033 ceph-mon[27845]: pgmap v3515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:47 smithi033 ceph-mon[27845]: pgmap v3516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:20:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:20:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:20:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:47 smithi096 ceph-mon[32753]: pgmap v3515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:47 smithi096 ceph-mon[32753]: pgmap v3516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:20:49.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5386.765456 secs 2024-04-26T13:20:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5386.765456 secs 2024-04-26T13:20:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5386.765456 secs 2024-04-26T13:20:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5389.509629 secs 2024-04-26T13:20:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:49 smithi194 ceph-mon[33017]: pgmap v3517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:20:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5389.509629 secs 2024-04-26T13:20:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:49 smithi033 ceph-mon[27845]: pgmap v3517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:20:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5389.509629 secs 2024-04-26T13:20:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:49 smithi096 ceph-mon[32753]: pgmap v3517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:20:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:51 smithi194 ceph-mon[33017]: pgmap v3518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:51.606+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:51.607+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:20:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:20:51.607+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:20:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:51 smithi033 ceph-mon[27845]: pgmap v3518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:51 smithi096 ceph-mon[32753]: pgmap v3518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5391.765557 secs 2024-04-26T13:20:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5391.765557 secs 2024-04-26T13:20:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5391.765557 secs 2024-04-26T13:20:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5394.509702 secs 2024-04-26T13:20:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:54 smithi033 ceph-mon[27845]: pgmap v3519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5394.509702 secs 2024-04-26T13:20:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:54 smithi096 ceph-mon[32753]: pgmap v3519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5394.509702 secs 2024-04-26T13:20:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:54 smithi194 ceph-mon[33017]: pgmap v3519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:55 smithi194 ceph-mon[33017]: pgmap v3520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:55 smithi033 ceph-mon[27845]: pgmap v3520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:55 smithi096 ceph-mon[32753]: pgmap v3520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:20:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:57 smithi033 ceph-mon[27845]: pgmap v3521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:20:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:57 smithi194 ceph-mon[33017]: pgmap v3521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:20:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:57 smithi096 ceph-mon[32753]: pgmap v3521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:20:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5396.765655 secs 2024-04-26T13:20:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5396.765655 secs 2024-04-26T13:20:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5396.765655 secs 2024-04-26T13:20:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5399.509821 secs 2024-04-26T13:20:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:20:59 smithi194 ceph-mon[33017]: pgmap v3522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5399.509821 secs 2024-04-26T13:20:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:20:59 smithi033 ceph-mon[27845]: pgmap v3522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:20:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5399.509821 secs 2024-04-26T13:20:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:20:59 smithi096 ceph-mon[32753]: pgmap v3522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:01 smithi194 ceph-mon[33017]: pgmap v3523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:01 smithi033 ceph-mon[27845]: pgmap v3523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:01 smithi096 ceph-mon[32753]: pgmap v3523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5401.765753 secs 2024-04-26T13:21:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5401.765753 secs 2024-04-26T13:21:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5401.765753 secs 2024-04-26T13:21:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5404.509938 secs 2024-04-26T13:21:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:04 smithi033 ceph-mon[27845]: pgmap v3524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5404.509938 secs 2024-04-26T13:21:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:04 smithi096 ceph-mon[32753]: pgmap v3524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5404.509938 secs 2024-04-26T13:21:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:04 smithi194 ceph-mon[33017]: pgmap v3524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:05 smithi194 ceph-mon[33017]: pgmap v3525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:05 smithi033 ceph-mon[27845]: pgmap v3525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:05 smithi096 ceph-mon[32753]: pgmap v3525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:06.619+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:06.619+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:06.619+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:21:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:07 smithi194 ceph-mon[33017]: pgmap v3526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:07 smithi033 ceph-mon[27845]: pgmap v3526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:07 smithi096 ceph-mon[32753]: pgmap v3526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5406.765818 secs 2024-04-26T13:21:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5406.765818 secs 2024-04-26T13:21:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5406.765818 secs 2024-04-26T13:21:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5409.510060 secs 2024-04-26T13:21:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:09 smithi033 ceph-mon[27845]: pgmap v3527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5409.510060 secs 2024-04-26T13:21:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:09 smithi194 ceph-mon[33017]: pgmap v3527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5409.510060 secs 2024-04-26T13:21:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:09 smithi096 ceph-mon[32753]: pgmap v3527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:11 smithi194 ceph-mon[33017]: pgmap v3528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:11 smithi033 ceph-mon[27845]: pgmap v3528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:11 smithi096 ceph-mon[32753]: pgmap v3528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5411.765902 secs 2024-04-26T13:21:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5411.765902 secs 2024-04-26T13:21:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5411.765902 secs 2024-04-26T13:21:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5414.510173 secs 2024-04-26T13:21:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:14 smithi033 ceph-mon[27845]: pgmap v3529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5414.510173 secs 2024-04-26T13:21:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:14 smithi096 ceph-mon[32753]: pgmap v3529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5414.510173 secs 2024-04-26T13:21:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:14 smithi194 ceph-mon[33017]: pgmap v3529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:15 smithi194 ceph-mon[33017]: pgmap v3530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:15 smithi033 ceph-mon[27845]: pgmap v3530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:15 smithi096 ceph-mon[32753]: pgmap v3530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:17 smithi194 ceph-mon[33017]: pgmap v3531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:17 smithi033 ceph-mon[27845]: pgmap v3531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:17 smithi096 ceph-mon[32753]: pgmap v3531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5416.765985 secs 2024-04-26T13:21:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5416.765985 secs 2024-04-26T13:21:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5416.765985 secs 2024-04-26T13:21:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5419.510297 secs 2024-04-26T13:21:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:19 smithi096 ceph-mon[32753]: pgmap v3532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5419.510297 secs 2024-04-26T13:21:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:19 smithi194 ceph-mon[33017]: pgmap v3532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5419.510297 secs 2024-04-26T13:21:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:19 smithi033 ceph-mon[27845]: pgmap v3532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:21 smithi194 ceph-mon[33017]: pgmap v3533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:21.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:21.635+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:21.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:21.635+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:21.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:21.635+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:21:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:21 smithi033 ceph-mon[27845]: pgmap v3533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:21 smithi096 ceph-mon[32753]: pgmap v3533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5421.766071 secs 2024-04-26T13:21:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5421.766071 secs 2024-04-26T13:21:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5421.766071 secs 2024-04-26T13:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5424.510413 secs 2024-04-26T13:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:24 smithi033 ceph-mon[27845]: pgmap v3534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5424.510413 secs 2024-04-26T13:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:24 smithi096 ceph-mon[32753]: pgmap v3534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5424.510413 secs 2024-04-26T13:21:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:24 smithi194 ceph-mon[33017]: pgmap v3534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:25 smithi194 ceph-mon[33017]: pgmap v3535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:25 smithi033 ceph-mon[27845]: pgmap v3535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:25 smithi096 ceph-mon[32753]: pgmap v3535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:27 smithi194 ceph-mon[33017]: pgmap v3536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:27 smithi033 ceph-mon[27845]: pgmap v3536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:27 smithi096 ceph-mon[32753]: pgmap v3536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5426.766154 secs 2024-04-26T13:21:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5426.766154 secs 2024-04-26T13:21:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5426.766154 secs 2024-04-26T13:21:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5429.510529 secs 2024-04-26T13:21:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:29 smithi194 ceph-mon[33017]: pgmap v3537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5429.510529 secs 2024-04-26T13:21:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:29 smithi033 ceph-mon[27845]: pgmap v3537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5429.510529 secs 2024-04-26T13:21:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:29 smithi096 ceph-mon[32753]: pgmap v3537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:31 smithi194 ceph-mon[33017]: pgmap v3538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:31 smithi033 ceph-mon[27845]: pgmap v3538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:31 smithi096 ceph-mon[32753]: pgmap v3538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5431.766243 secs 2024-04-26T13:21:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5431.766243 secs 2024-04-26T13:21:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5431.766243 secs 2024-04-26T13:21:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5434.510644 secs 2024-04-26T13:21:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:34 smithi033 ceph-mon[27845]: pgmap v3539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5434.510644 secs 2024-04-26T13:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:34 smithi096 ceph-mon[32753]: pgmap v3539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5434.510644 secs 2024-04-26T13:21:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:34 smithi194 ceph-mon[33017]: pgmap v3539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:35 smithi194 ceph-mon[33017]: pgmap v3540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:35 smithi033 ceph-mon[27845]: pgmap v3540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:35 smithi096 ceph-mon[32753]: pgmap v3540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:36.649+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:36.649+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:36.649+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:21:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:37 smithi194 ceph-mon[33017]: pgmap v3541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:37 smithi033 ceph-mon[27845]: pgmap v3541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:37 smithi096 ceph-mon[32753]: pgmap v3541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5436.766303 secs 2024-04-26T13:21:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5436.766303 secs 2024-04-26T13:21:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5436.766303 secs 2024-04-26T13:21:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5439.510777 secs 2024-04-26T13:21:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:39 smithi033 ceph-mon[27845]: pgmap v3542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5439.510777 secs 2024-04-26T13:21:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:39 smithi194 ceph-mon[33017]: pgmap v3542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5439.510777 secs 2024-04-26T13:21:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:39 smithi096 ceph-mon[32753]: pgmap v3542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:41 smithi194 ceph-mon[33017]: pgmap v3543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:41 smithi033 ceph-mon[27845]: pgmap v3543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:41 smithi096 ceph-mon[32753]: pgmap v3543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5441.766385 secs 2024-04-26T13:21:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5441.766385 secs 2024-04-26T13:21:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5441.766385 secs 2024-04-26T13:21:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5444.510919 secs 2024-04-26T13:21:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:44 smithi033 ceph-mon[27845]: pgmap v3544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5444.510919 secs 2024-04-26T13:21:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:44 smithi096 ceph-mon[32753]: pgmap v3544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5444.510919 secs 2024-04-26T13:21:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:44 smithi194 ceph-mon[33017]: pgmap v3544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:45 smithi194 ceph-mon[33017]: pgmap v3545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:45 smithi033 ceph-mon[27845]: pgmap v3545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:45 smithi096 ceph-mon[32753]: pgmap v3545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:21:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:47 smithi194 ceph-mon[33017]: pgmap v3546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:47 smithi033 ceph-mon[27845]: pgmap v3546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:47 smithi096 ceph-mon[32753]: pgmap v3546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:21:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5446.766470 secs 2024-04-26T13:21:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:21:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5446.766470 secs 2024-04-26T13:21:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:21:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5446.766470 secs 2024-04-26T13:21:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5449.511035 secs 2024-04-26T13:21:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:21:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:21:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:49 smithi096 ceph-mon[32753]: pgmap v3547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:49 smithi096 ceph-mon[32753]: pgmap v3548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:49.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:21:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5449.511035 secs 2024-04-26T13:21:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:21:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:21:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:49 smithi194 ceph-mon[33017]: pgmap v3547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:49 smithi194 ceph-mon[33017]: pgmap v3548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:21:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5449.511035 secs 2024-04-26T13:21:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:21:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:21:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:49 smithi033 ceph-mon[27845]: pgmap v3547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:21:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:49 smithi033 ceph-mon[27845]: pgmap v3548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:21:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:51.660+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:51.660+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:21:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:21:51.660+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:21:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:51 smithi033 ceph-mon[27845]: pgmap v3549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:51 smithi096 ceph-mon[32753]: pgmap v3549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:51 smithi194 ceph-mon[33017]: pgmap v3549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5451.766553 secs 2024-04-26T13:21:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:53 smithi033 ceph-mon[27845]: pgmap v3550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5451.766553 secs 2024-04-26T13:21:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:53 smithi096 ceph-mon[32753]: pgmap v3550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5451.766553 secs 2024-04-26T13:21:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:53 smithi194 ceph-mon[33017]: pgmap v3550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5454.511138 secs 2024-04-26T13:21:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5454.511138 secs 2024-04-26T13:21:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5454.511138 secs 2024-04-26T13:21:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:55 smithi033 ceph-mon[27845]: pgmap v3551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:55 smithi096 ceph-mon[32753]: pgmap v3551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:55 smithi194 ceph-mon[33017]: pgmap v3551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:57 smithi033 ceph-mon[27845]: pgmap v3552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:57 smithi096 ceph-mon[32753]: pgmap v3552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:57 smithi194 ceph-mon[33017]: pgmap v3552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:21:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5456.766639 secs 2024-04-26T13:21:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5456.766639 secs 2024-04-26T13:21:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5456.766639 secs 2024-04-26T13:22:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5459.511266 secs 2024-04-26T13:22:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:21:59 smithi033 ceph-mon[27845]: pgmap v3553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:22:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5459.511266 secs 2024-04-26T13:22:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:21:59 smithi096 ceph-mon[32753]: pgmap v3553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:22:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5459.511266 secs 2024-04-26T13:22:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:21:59 smithi194 ceph-mon[33017]: pgmap v3553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:22:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:01 smithi033 ceph-mon[27845]: pgmap v3554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:01 smithi096 ceph-mon[32753]: pgmap v3554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:01 smithi194 ceph-mon[33017]: pgmap v3554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5461.766728 secs 2024-04-26T13:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:03 smithi096 ceph-mon[32753]: pgmap v3555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5464.511359 secs 2024-04-26T13:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5461.766728 secs 2024-04-26T13:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:03 smithi194 ceph-mon[33017]: pgmap v3555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5464.511359 secs 2024-04-26T13:22:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5461.766728 secs 2024-04-26T13:22:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:03 smithi033 ceph-mon[27845]: pgmap v3555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5464.511359 secs 2024-04-26T13:22:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:05 smithi096 ceph-mon[32753]: pgmap v3556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:05 smithi194 ceph-mon[33017]: pgmap v3556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:05 smithi033 ceph-mon[27845]: pgmap v3556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:06.676+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:06.676+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:06.676+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:22:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:07 smithi194 ceph-mon[33017]: pgmap v3557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:07 smithi033 ceph-mon[27845]: pgmap v3557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:07 smithi096 ceph-mon[32753]: pgmap v3557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5466.766814 secs 2024-04-26T13:22:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5469.511472 secs 2024-04-26T13:22:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5466.766814 secs 2024-04-26T13:22:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5469.511472 secs 2024-04-26T13:22:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5466.766814 secs 2024-04-26T13:22:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5469.511472 secs 2024-04-26T13:22:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:09 smithi194 ceph-mon[33017]: pgmap v3558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:09 smithi033 ceph-mon[27845]: pgmap v3558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:09 smithi096 ceph-mon[32753]: pgmap v3558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:11 smithi194 ceph-mon[33017]: pgmap v3559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:11 smithi033 ceph-mon[27845]: pgmap v3559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:11 smithi096 ceph-mon[32753]: pgmap v3559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5471.766896 secs 2024-04-26T13:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:13 smithi194 ceph-mon[33017]: pgmap v3560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5474.511540 secs 2024-04-26T13:22:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5471.766896 secs 2024-04-26T13:22:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:13 smithi033 ceph-mon[27845]: pgmap v3560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5474.511540 secs 2024-04-26T13:22:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5471.766896 secs 2024-04-26T13:22:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:13 smithi096 ceph-mon[32753]: pgmap v3560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5474.511540 secs 2024-04-26T13:22:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:15 smithi194 ceph-mon[33017]: pgmap v3561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:15 smithi033 ceph-mon[27845]: pgmap v3561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:15 smithi096 ceph-mon[32753]: pgmap v3561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:17 smithi194 ceph-mon[33017]: pgmap v3562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:17 smithi033 ceph-mon[27845]: pgmap v3562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:17 smithi096 ceph-mon[32753]: pgmap v3562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:19.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5476.767000 secs 2024-04-26T13:22:19.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5479.511637 secs 2024-04-26T13:22:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5476.767000 secs 2024-04-26T13:22:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5479.511637 secs 2024-04-26T13:22:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5476.767000 secs 2024-04-26T13:22:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5479.511637 secs 2024-04-26T13:22:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:19 smithi194 ceph-mon[33017]: pgmap v3563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:19 smithi033 ceph-mon[27845]: pgmap v3563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:19 smithi096 ceph-mon[32753]: pgmap v3563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:21.691+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:21.691+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:21.691+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:22:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:21 smithi194 ceph-mon[33017]: pgmap v3564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:21 smithi033 ceph-mon[27845]: pgmap v3564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:21 smithi096 ceph-mon[32753]: pgmap v3564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5481.767060 secs 2024-04-26T13:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:23 smithi194 ceph-mon[33017]: pgmap v3565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5484.511758 secs 2024-04-26T13:22:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5481.767060 secs 2024-04-26T13:22:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:23 smithi033 ceph-mon[27845]: pgmap v3565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5484.511758 secs 2024-04-26T13:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5481.767060 secs 2024-04-26T13:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:23 smithi096 ceph-mon[32753]: pgmap v3565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5484.511758 secs 2024-04-26T13:22:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:25 smithi194 ceph-mon[33017]: pgmap v3566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:25 smithi033 ceph-mon[27845]: pgmap v3566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:25 smithi096 ceph-mon[32753]: pgmap v3566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:28 smithi194 ceph-mon[33017]: pgmap v3567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:28 smithi033 ceph-mon[27845]: pgmap v3567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:28 smithi096 ceph-mon[32753]: pgmap v3567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5486.767146 secs 2024-04-26T13:22:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5489.511877 secs 2024-04-26T13:22:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5486.767146 secs 2024-04-26T13:22:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5489.511877 secs 2024-04-26T13:22:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5486.767146 secs 2024-04-26T13:22:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5489.511877 secs 2024-04-26T13:22:30.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:30 smithi033 ceph-mon[27845]: pgmap v3568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:30 smithi194 ceph-mon[33017]: pgmap v3568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:30 smithi096 ceph-mon[32753]: pgmap v3568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:32 smithi194 ceph-mon[33017]: pgmap v3569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:32 smithi033 ceph-mon[27845]: pgmap v3569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:32 smithi096 ceph-mon[32753]: pgmap v3569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5491.767234 secs 2024-04-26T13:22:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:34 smithi194 ceph-mon[33017]: pgmap v3570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5494.511988 secs 2024-04-26T13:22:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5491.767234 secs 2024-04-26T13:22:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:34 smithi033 ceph-mon[27845]: pgmap v3570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5494.511988 secs 2024-04-26T13:22:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5491.767234 secs 2024-04-26T13:22:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:34 smithi096 ceph-mon[32753]: pgmap v3570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5494.511988 secs 2024-04-26T13:22:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:36 smithi194 ceph-mon[33017]: pgmap v3571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:36 smithi033 ceph-mon[27845]: pgmap v3571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:36 smithi096 ceph-mon[32753]: pgmap v3571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:36.706+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:36.706+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:36.706+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:22:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:38 smithi194 ceph-mon[33017]: pgmap v3572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:38 smithi033 ceph-mon[27845]: pgmap v3572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:38 smithi096 ceph-mon[32753]: pgmap v3572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:39.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5496.767295 secs 2024-04-26T13:22:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5499.512148 secs 2024-04-26T13:22:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5496.767295 secs 2024-04-26T13:22:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5499.512148 secs 2024-04-26T13:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5496.767295 secs 2024-04-26T13:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5499.512148 secs 2024-04-26T13:22:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:40 smithi194 ceph-mon[33017]: pgmap v3573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:40 smithi033 ceph-mon[27845]: pgmap v3573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:40 smithi096 ceph-mon[32753]: pgmap v3573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:42 smithi194 ceph-mon[33017]: pgmap v3574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:42 smithi033 ceph-mon[27845]: pgmap v3574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:42 smithi096 ceph-mon[32753]: pgmap v3574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5501.767382 secs 2024-04-26T13:22:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:44 smithi194 ceph-mon[33017]: pgmap v3575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5504.512311 secs 2024-04-26T13:22:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5501.767382 secs 2024-04-26T13:22:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:44 smithi033 ceph-mon[27845]: pgmap v3575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5504.512311 secs 2024-04-26T13:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5501.767382 secs 2024-04-26T13:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:44 smithi096 ceph-mon[32753]: pgmap v3575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5504.512311 secs 2024-04-26T13:22:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:46 smithi194 ceph-mon[33017]: pgmap v3576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:46 smithi033 ceph-mon[27845]: pgmap v3576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:46 smithi096 ceph-mon[32753]: pgmap v3576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:48 smithi194 ceph-mon[33017]: pgmap v3577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:48 smithi033 ceph-mon[27845]: pgmap v3577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:48 smithi096 ceph-mon[32753]: pgmap v3577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:49.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5506.767482 secs 2024-04-26T13:22:49.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5509.512387 secs 2024-04-26T13:22:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5506.767482 secs 2024-04-26T13:22:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5509.512387 secs 2024-04-26T13:22:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5506.767482 secs 2024-04-26T13:22:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5509.512387 secs 2024-04-26T13:22:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:50 smithi194 ceph-mon[33017]: pgmap v3578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:22:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:50 smithi033 ceph-mon[27845]: pgmap v3578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:22:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:50 smithi096 ceph-mon[32753]: pgmap v3578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:22:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:22:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:51.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:51.721+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:51.722+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:22:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:22:51.722+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:22:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:52 smithi033 ceph-mon[27845]: pgmap v3579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:22:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:22:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:52 smithi096 ceph-mon[32753]: pgmap v3579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:22:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:22:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:52 smithi194 ceph-mon[33017]: pgmap v3579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:22:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:22:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:22:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:22:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:53 smithi033 ceph-mon[27845]: pgmap v3580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:22:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:53 smithi033 ceph-mon[27845]: pgmap v3581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5511.767541 secs 2024-04-26T13:22:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:53 smithi096 ceph-mon[32753]: pgmap v3580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:22:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:53 smithi096 ceph-mon[32753]: pgmap v3581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5511.767541 secs 2024-04-26T13:22:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:53 smithi194 ceph-mon[33017]: pgmap v3580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:22:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:53 smithi194 ceph-mon[33017]: pgmap v3581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5511.767541 secs 2024-04-26T13:22:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5514.512517 secs 2024-04-26T13:22:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5514.512517 secs 2024-04-26T13:22:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5514.512517 secs 2024-04-26T13:22:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:55 smithi033 ceph-mon[27845]: pgmap v3582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:22:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:55 smithi096 ceph-mon[32753]: pgmap v3582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:22:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:55 smithi194 ceph-mon[33017]: pgmap v3582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:22:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:57 smithi033 ceph-mon[27845]: pgmap v3583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:57 smithi096 ceph-mon[32753]: pgmap v3583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:57 smithi194 ceph-mon[33017]: pgmap v3583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5516.767623 secs 2024-04-26T13:22:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5516.767623 secs 2024-04-26T13:22:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5516.767623 secs 2024-04-26T13:22:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:59 smithi033 ceph-mon[27845]: pgmap v3584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5519.512644 secs 2024-04-26T13:22:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:59 smithi096 ceph-mon[32753]: pgmap v3584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5519.512644 secs 2024-04-26T13:22:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:59 smithi194 ceph-mon[33017]: pgmap v3584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:22:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5519.512644 secs 2024-04-26T13:23:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:01 smithi033 ceph-mon[27845]: pgmap v3585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:23:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:01 smithi096 ceph-mon[32753]: pgmap v3585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:23:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:01 smithi194 ceph-mon[33017]: pgmap v3585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:23:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:03 smithi033 ceph-mon[27845]: pgmap v3586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:23:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5521.767706 secs 2024-04-26T13:23:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:03 smithi096 ceph-mon[32753]: pgmap v3586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:23:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5521.767706 secs 2024-04-26T13:23:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:03 smithi194 ceph-mon[33017]: pgmap v3586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:23:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5521.767706 secs 2024-04-26T13:23:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5524.512766 secs 2024-04-26T13:23:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5524.512766 secs 2024-04-26T13:23:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5524.512766 secs 2024-04-26T13:23:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:05 smithi194 ceph-mon[33017]: pgmap v3587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:05 smithi033 ceph-mon[27845]: pgmap v3587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:05 smithi096 ceph-mon[32753]: pgmap v3587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:06.734+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:06.734+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:06.734+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:23:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:07 smithi194 ceph-mon[33017]: pgmap v3588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:07 smithi033 ceph-mon[27845]: pgmap v3588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:07 smithi096 ceph-mon[32753]: pgmap v3588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5526.767766 secs 2024-04-26T13:23:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5526.767766 secs 2024-04-26T13:23:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5526.767766 secs 2024-04-26T13:23:09.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:09 smithi033 ceph-mon[27845]: pgmap v3589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5529.512916 secs 2024-04-26T13:23:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:09 smithi194 ceph-mon[33017]: pgmap v3589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5529.512916 secs 2024-04-26T13:23:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:09 smithi096 ceph-mon[32753]: pgmap v3589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5529.512916 secs 2024-04-26T13:23:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:11 smithi194 ceph-mon[33017]: pgmap v3590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:11 smithi033 ceph-mon[27845]: pgmap v3590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:11 smithi096 ceph-mon[32753]: pgmap v3590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:13 smithi194 ceph-mon[33017]: pgmap v3591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5531.767849 secs 2024-04-26T13:23:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:13 smithi033 ceph-mon[27845]: pgmap v3591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5531.767849 secs 2024-04-26T13:23:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:13 smithi096 ceph-mon[32753]: pgmap v3591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5531.767849 secs 2024-04-26T13:23:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5534.513059 secs 2024-04-26T13:23:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5534.513059 secs 2024-04-26T13:23:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5534.513059 secs 2024-04-26T13:23:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:15 smithi194 ceph-mon[33017]: pgmap v3592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:15 smithi033 ceph-mon[27845]: pgmap v3592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:15 smithi096 ceph-mon[32753]: pgmap v3592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:17 smithi194 ceph-mon[33017]: pgmap v3593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:17 smithi033 ceph-mon[27845]: pgmap v3593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:17 smithi096 ceph-mon[32753]: pgmap v3593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:18 smithi194 ceph-mon[33017]: pgmap v3594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5536.767935 secs 2024-04-26T13:23:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:18 smithi033 ceph-mon[27845]: pgmap v3594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5536.767935 secs 2024-04-26T13:23:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:18 smithi096 ceph-mon[32753]: pgmap v3594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5536.767935 secs 2024-04-26T13:23:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5539.513214 secs 2024-04-26T13:23:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5539.513214 secs 2024-04-26T13:23:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5539.513214 secs 2024-04-26T13:23:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:20 smithi194 ceph-mon[33017]: pgmap v3595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:20 smithi033 ceph-mon[27845]: pgmap v3595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:20 smithi096 ceph-mon[32753]: pgmap v3595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:21.736+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:21.736+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:21.736+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:23:22.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:22.730+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:23:22.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:22.730+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T13:23:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:22.731+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:23:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:22.731+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:23:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:22 smithi194 ceph-mon[33017]: pgmap v3596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:22 smithi033 ceph-mon[27845]: pgmap v3596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:22 smithi096 ceph-mon[32753]: pgmap v3596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5541.768019 secs 2024-04-26T13:23:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5541.768019 secs 2024-04-26T13:23:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5541.768019 secs 2024-04-26T13:23:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5544.513298 secs 2024-04-26T13:23:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:24 smithi194 ceph-mon[33017]: pgmap v3597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5544.513298 secs 2024-04-26T13:23:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:24 smithi033 ceph-mon[27845]: pgmap v3597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5544.513298 secs 2024-04-26T13:23:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:24 smithi096 ceph-mon[32753]: pgmap v3597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:26 smithi194 ceph-mon[33017]: pgmap v3598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:26 smithi033 ceph-mon[27845]: pgmap v3598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:26 smithi096 ceph-mon[32753]: pgmap v3598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:28 smithi194 ceph-mon[33017]: pgmap v3599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5546.768104 secs 2024-04-26T13:23:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:28 smithi033 ceph-mon[27845]: pgmap v3599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5546.768104 secs 2024-04-26T13:23:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:28 smithi096 ceph-mon[32753]: pgmap v3599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5546.768104 secs 2024-04-26T13:23:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5549.513429 secs 2024-04-26T13:23:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5549.513429 secs 2024-04-26T13:23:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5549.513429 secs 2024-04-26T13:23:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:30 smithi033 ceph-mon[27845]: pgmap v3600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:30 smithi096 ceph-mon[32753]: pgmap v3600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:30 smithi194 ceph-mon[33017]: pgmap v3600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:32 smithi033 ceph-mon[27845]: pgmap v3601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:32 smithi096 ceph-mon[32753]: pgmap v3601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:32 smithi194 ceph-mon[33017]: pgmap v3601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5551.768188 secs 2024-04-26T13:23:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5551.768188 secs 2024-04-26T13:23:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5551.768188 secs 2024-04-26T13:23:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5554.513561 secs 2024-04-26T13:23:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:34 smithi033 ceph-mon[27845]: pgmap v3602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5554.513561 secs 2024-04-26T13:23:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:34 smithi096 ceph-mon[32753]: pgmap v3602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5554.513561 secs 2024-04-26T13:23:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:34 smithi194 ceph-mon[33017]: pgmap v3602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:36.737+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:36.737+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:36.737+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:23:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:36 smithi033 ceph-mon[27845]: pgmap v3603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:36 smithi096 ceph-mon[32753]: pgmap v3603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:36 smithi194 ceph-mon[33017]: pgmap v3603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:23:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:38 smithi033 ceph-mon[27845]: pgmap v3604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5556.768278 secs 2024-04-26T13:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:38 smithi096 ceph-mon[32753]: pgmap v3604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5556.768278 secs 2024-04-26T13:23:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:38 smithi194 ceph-mon[33017]: pgmap v3604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5556.768278 secs 2024-04-26T13:23:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5559.513688 secs 2024-04-26T13:23:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5559.513688 secs 2024-04-26T13:23:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5559.513688 secs 2024-04-26T13:23:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:40 smithi033 ceph-mon[27845]: pgmap v3605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:40 smithi096 ceph-mon[32753]: pgmap v3605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:40 smithi194 ceph-mon[33017]: pgmap v3605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:42 smithi033 ceph-mon[27845]: pgmap v3606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:42 smithi096 ceph-mon[32753]: pgmap v3606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:42 smithi194 ceph-mon[33017]: pgmap v3606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5561.768339 secs 2024-04-26T13:23:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5561.768339 secs 2024-04-26T13:23:44.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5561.768339 secs 2024-04-26T13:23:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5564.513812 secs 2024-04-26T13:23:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:44 smithi033 ceph-mon[27845]: pgmap v3607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5564.513812 secs 2024-04-26T13:23:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:44 smithi096 ceph-mon[32753]: pgmap v3607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5564.513812 secs 2024-04-26T13:23:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:44 smithi194 ceph-mon[33017]: pgmap v3607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:46 smithi033 ceph-mon[27845]: pgmap v3608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:46 smithi096 ceph-mon[32753]: pgmap v3608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:46 smithi194 ceph-mon[33017]: pgmap v3608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5566.768398 secs 2024-04-26T13:23:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:48 smithi096 ceph-mon[32753]: pgmap v3609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5566.768398 secs 2024-04-26T13:23:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:48 smithi194 ceph-mon[33017]: pgmap v3609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5566.768398 secs 2024-04-26T13:23:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:48 smithi033 ceph-mon[27845]: pgmap v3609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:23:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5569.513873 secs 2024-04-26T13:23:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5569.513873 secs 2024-04-26T13:23:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5569.513873 secs 2024-04-26T13:23:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:50 smithi194 ceph-mon[33017]: pgmap v3610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:50 smithi033 ceph-mon[27845]: pgmap v3610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:50 smithi096 ceph-mon[32753]: pgmap v3610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:23:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:51.750+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:51.750+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:23:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:23:51.750+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:23:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:52 smithi194 ceph-mon[33017]: pgmap v3611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:23:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:52 smithi033 ceph-mon[27845]: pgmap v3611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:23:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:52 smithi096 ceph-mon[32753]: pgmap v3611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5571.768483 secs 2024-04-26T13:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:53 smithi194 ceph-mon[33017]: pgmap v3612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:53 smithi194 ceph-mon[33017]: pgmap v3613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5571.768483 secs 2024-04-26T13:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:53 smithi033 ceph-mon[27845]: pgmap v3612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:53 smithi033 ceph-mon[27845]: pgmap v3613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:23:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5571.768483 secs 2024-04-26T13:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:53 smithi096 ceph-mon[32753]: pgmap v3612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:53 smithi096 ceph-mon[32753]: pgmap v3613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:23:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5574.513993 secs 2024-04-26T13:23:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5574.513993 secs 2024-04-26T13:23:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5574.513993 secs 2024-04-26T13:23:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:55 smithi194 ceph-mon[33017]: pgmap v3614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:23:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:55 smithi033 ceph-mon[27845]: pgmap v3614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:23:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:55 smithi096 ceph-mon[32753]: pgmap v3614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:23:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:58 smithi194 ceph-mon[33017]: pgmap v3615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:23:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5576.768568 secs 2024-04-26T13:23:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:58 smithi033 ceph-mon[27845]: pgmap v3615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:23:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5576.768568 secs 2024-04-26T13:23:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:58 smithi096 ceph-mon[32753]: pgmap v3615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:23:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5576.768568 secs 2024-04-26T13:23:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5579.514125 secs 2024-04-26T13:23:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:23:59 smithi194 ceph-mon[33017]: pgmap v3616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 955 B/s rd, 1 op/s 2024-04-26T13:23:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5579.514125 secs 2024-04-26T13:23:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:23:59 smithi033 ceph-mon[27845]: pgmap v3616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 955 B/s rd, 1 op/s 2024-04-26T13:23:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5579.514125 secs 2024-04-26T13:23:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:23:59 smithi096 ceph-mon[32753]: pgmap v3616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 955 B/s rd, 1 op/s 2024-04-26T13:24:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:02 smithi194 ceph-mon[33017]: pgmap v3617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:24:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:02 smithi033 ceph-mon[27845]: pgmap v3617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:24:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:02 smithi096 ceph-mon[32753]: pgmap v3617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:24:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5581.768651 secs 2024-04-26T13:24:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:03 smithi194 ceph-mon[33017]: pgmap v3618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:24:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5581.768651 secs 2024-04-26T13:24:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:03 smithi033 ceph-mon[27845]: pgmap v3618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:24:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5581.768651 secs 2024-04-26T13:24:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:03 smithi096 ceph-mon[32753]: pgmap v3618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:24:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5584.514249 secs 2024-04-26T13:24:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5584.514249 secs 2024-04-26T13:24:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5584.514249 secs 2024-04-26T13:24:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:05 smithi194 ceph-mon[33017]: pgmap v3619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:24:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:05 smithi033 ceph-mon[27845]: pgmap v3619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:24:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:05 smithi096 ceph-mon[32753]: pgmap v3619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:24:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:06.766+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:06.766+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:06.766+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:24:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:08 smithi194 ceph-mon[33017]: pgmap v3620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5586.768735 secs 2024-04-26T13:24:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:08 smithi033 ceph-mon[27845]: pgmap v3620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5586.768735 secs 2024-04-26T13:24:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:08 smithi096 ceph-mon[32753]: pgmap v3620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5586.768735 secs 2024-04-26T13:24:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5589.514373 secs 2024-04-26T13:24:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:09 smithi033 ceph-mon[27845]: pgmap v3621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5589.514373 secs 2024-04-26T13:24:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:09 smithi194 ceph-mon[33017]: pgmap v3621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5589.514373 secs 2024-04-26T13:24:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:09 smithi096 ceph-mon[32753]: pgmap v3621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:11 smithi194 ceph-mon[33017]: pgmap v3622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:11 smithi033 ceph-mon[27845]: pgmap v3622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:11 smithi096 ceph-mon[32753]: pgmap v3622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5591.768817 secs 2024-04-26T13:24:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5591.768817 secs 2024-04-26T13:24:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5591.768817 secs 2024-04-26T13:24:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5594.514500 secs 2024-04-26T13:24:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:14 smithi033 ceph-mon[27845]: pgmap v3623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5594.514500 secs 2024-04-26T13:24:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:14 smithi096 ceph-mon[32753]: pgmap v3623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:14.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5594.514500 secs 2024-04-26T13:24:14.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:14 smithi194 ceph-mon[33017]: pgmap v3623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:15 smithi194 ceph-mon[33017]: pgmap v3624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:15 smithi033 ceph-mon[27845]: pgmap v3624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:15 smithi096 ceph-mon[32753]: pgmap v3624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:17 smithi194 ceph-mon[33017]: pgmap v3625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:17 smithi033 ceph-mon[27845]: pgmap v3625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:17 smithi096 ceph-mon[32753]: pgmap v3625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5596.768903 secs 2024-04-26T13:24:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5596.768903 secs 2024-04-26T13:24:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5596.768903 secs 2024-04-26T13:24:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5599.514564 secs 2024-04-26T13:24:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:19 smithi096 ceph-mon[32753]: pgmap v3626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5599.514564 secs 2024-04-26T13:24:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:19 smithi194 ceph-mon[33017]: pgmap v3626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5599.514564 secs 2024-04-26T13:24:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:19 smithi033 ceph-mon[27845]: pgmap v3626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:21 smithi194 ceph-mon[33017]: pgmap v3627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:21 smithi033 ceph-mon[27845]: pgmap v3627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:21 smithi096 ceph-mon[32753]: pgmap v3627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:21.779+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:21.779+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:21.779+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:24:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5601.768988 secs 2024-04-26T13:24:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5601.768988 secs 2024-04-26T13:24:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5601.768988 secs 2024-04-26T13:24:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5604.514644 secs 2024-04-26T13:24:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:24 smithi033 ceph-mon[27845]: pgmap v3628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5604.514644 secs 2024-04-26T13:24:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:24 smithi096 ceph-mon[32753]: pgmap v3628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5604.514644 secs 2024-04-26T13:24:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:24 smithi194 ceph-mon[33017]: pgmap v3628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:25 smithi194 ceph-mon[33017]: pgmap v3629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:25 smithi033 ceph-mon[27845]: pgmap v3629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:25 smithi096 ceph-mon[32753]: pgmap v3629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:27 smithi194 ceph-mon[33017]: pgmap v3630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:27 smithi033 ceph-mon[27845]: pgmap v3630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:27 smithi096 ceph-mon[32753]: pgmap v3630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5606.769047 secs 2024-04-26T13:24:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5606.769047 secs 2024-04-26T13:24:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5606.769047 secs 2024-04-26T13:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5609.514787 secs 2024-04-26T13:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:29 smithi194 ceph-mon[33017]: pgmap v3631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5609.514787 secs 2024-04-26T13:24:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:29 smithi033 ceph-mon[27845]: pgmap v3631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5609.514787 secs 2024-04-26T13:24:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:29 smithi096 ceph-mon[32753]: pgmap v3631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:31.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:31 smithi194 ceph-mon[33017]: pgmap v3632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:31 smithi033 ceph-mon[27845]: pgmap v3632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:31 smithi096 ceph-mon[32753]: pgmap v3632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5611.769107 secs 2024-04-26T13:24:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5611.769107 secs 2024-04-26T13:24:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5611.769107 secs 2024-04-26T13:24:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5614.514909 secs 2024-04-26T13:24:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:34 smithi033 ceph-mon[27845]: pgmap v3633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5614.514909 secs 2024-04-26T13:24:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:34 smithi096 ceph-mon[32753]: pgmap v3633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5614.514909 secs 2024-04-26T13:24:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:34 smithi194 ceph-mon[33017]: pgmap v3633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:35 smithi194 ceph-mon[33017]: pgmap v3634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:35 smithi033 ceph-mon[27845]: pgmap v3634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:35 smithi096 ceph-mon[32753]: pgmap v3634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:36.785+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:36.785+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:36.785+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:24:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:37 smithi194 ceph-mon[33017]: pgmap v3635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:37 smithi033 ceph-mon[27845]: pgmap v3635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:37 smithi096 ceph-mon[32753]: pgmap v3635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5616.769197 secs 2024-04-26T13:24:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5616.769197 secs 2024-04-26T13:24:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5616.769197 secs 2024-04-26T13:24:39.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5619.515038 secs 2024-04-26T13:24:39.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:39 smithi033 ceph-mon[27845]: pgmap v3636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5619.515038 secs 2024-04-26T13:24:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:39 smithi194 ceph-mon[33017]: pgmap v3636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5619.515038 secs 2024-04-26T13:24:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:39 smithi096 ceph-mon[32753]: pgmap v3636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:41 smithi194 ceph-mon[33017]: pgmap v3637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:41 smithi033 ceph-mon[27845]: pgmap v3637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:41 smithi096 ceph-mon[32753]: pgmap v3637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5621.769278 secs 2024-04-26T13:24:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5621.769278 secs 2024-04-26T13:24:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5621.769278 secs 2024-04-26T13:24:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5624.515159 secs 2024-04-26T13:24:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:44 smithi033 ceph-mon[27845]: pgmap v3638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5624.515159 secs 2024-04-26T13:24:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:44 smithi096 ceph-mon[32753]: pgmap v3638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5624.515159 secs 2024-04-26T13:24:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:44 smithi194 ceph-mon[33017]: pgmap v3638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:45 smithi194 ceph-mon[33017]: pgmap v3639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:45 smithi033 ceph-mon[27845]: pgmap v3639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:45 smithi096 ceph-mon[32753]: pgmap v3639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:47 smithi194 ceph-mon[33017]: pgmap v3640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:47 smithi033 ceph-mon[27845]: pgmap v3640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:47 smithi096 ceph-mon[32753]: pgmap v3640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5626.769338 secs 2024-04-26T13:24:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5626.769338 secs 2024-04-26T13:24:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5626.769338 secs 2024-04-26T13:24:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5629.515280 secs 2024-04-26T13:24:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:49 smithi096 ceph-mon[32753]: pgmap v3641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5629.515280 secs 2024-04-26T13:24:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:49 smithi194 ceph-mon[33017]: pgmap v3641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5629.515280 secs 2024-04-26T13:24:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:49 smithi033 ceph-mon[27845]: pgmap v3641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:51 smithi194 ceph-mon[33017]: pgmap v3642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:51 smithi033 ceph-mon[27845]: pgmap v3642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:51 smithi096 ceph-mon[32753]: pgmap v3642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:51.799+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:51.799+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:24:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:24:51.799+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:24:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5631.769419 secs 2024-04-26T13:24:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5631.769419 secs 2024-04-26T13:24:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5631.769419 secs 2024-04-26T13:24:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5634.515363 secs 2024-04-26T13:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:54 smithi033 ceph-mon[27845]: pgmap v3643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5634.515363 secs 2024-04-26T13:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:54 smithi096 ceph-mon[32753]: pgmap v3643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5634.515363 secs 2024-04-26T13:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:54 smithi194 ceph-mon[33017]: pgmap v3643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:55 smithi194 ceph-mon[33017]: pgmap v3644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:55 smithi033 ceph-mon[27845]: pgmap v3644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:55 smithi096 ceph-mon[32753]: pgmap v3644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:57.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:24:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:58 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:24:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:58 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:24:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:58 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:24:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:58 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:24:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:58 smithi194 ceph-mon[33017]: pgmap v3645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:58 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:24:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:58 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:24:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:58 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:24:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:58 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:24:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:58 smithi033 ceph-mon[27845]: pgmap v3645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:58 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:24:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:58 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:24:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:58 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:24:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:58 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:24:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:58 smithi096 ceph-mon[32753]: pgmap v3645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:24:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5636.769481 secs 2024-04-26T13:24:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5639.515443 secs 2024-04-26T13:24:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:24:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5636.769481 secs 2024-04-26T13:24:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5639.515443 secs 2024-04-26T13:24:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:24:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5636.769481 secs 2024-04-26T13:24:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5639.515443 secs 2024-04-26T13:24:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:24:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:24:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: pgmap v3646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:25:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:25:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:25:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: pgmap v3646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: pgmap v3646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:25:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:25:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:25:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:01 smithi194 ceph-mon[33017]: pgmap v3647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T13:25:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:01 smithi194 ceph-mon[33017]: pgmap v3648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:25:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:25:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:01 smithi033 ceph-mon[27845]: pgmap v3647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T13:25:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:01 smithi033 ceph-mon[27845]: pgmap v3648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:25:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:25:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:01 smithi096 ceph-mon[32753]: pgmap v3647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T13:25:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:01 smithi096 ceph-mon[32753]: pgmap v3648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:03 smithi033 ceph-mon[27845]: pgmap v3649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:03 smithi096 ceph-mon[32753]: pgmap v3649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:03 smithi194 ceph-mon[33017]: pgmap v3649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5641.769567 secs 2024-04-26T13:25:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5644.515551 secs 2024-04-26T13:25:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5641.769567 secs 2024-04-26T13:25:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5644.515551 secs 2024-04-26T13:25:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5641.769567 secs 2024-04-26T13:25:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5644.515551 secs 2024-04-26T13:25:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:05 smithi033 ceph-mon[27845]: pgmap v3650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:25:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:05 smithi096 ceph-mon[32753]: pgmap v3650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:25:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:05 smithi194 ceph-mon[33017]: pgmap v3650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:25:06.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:06.815+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:06.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:06.816+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:06.816+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:25:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:07 smithi033 ceph-mon[27845]: pgmap v3651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:07 smithi096 ceph-mon[32753]: pgmap v3651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:07 smithi194 ceph-mon[33017]: pgmap v3651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5646.769650 secs 2024-04-26T13:25:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5646.769650 secs 2024-04-26T13:25:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5646.769650 secs 2024-04-26T13:25:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:09 smithi033 ceph-mon[27845]: pgmap v3652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5649.515695 secs 2024-04-26T13:25:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:09 smithi096 ceph-mon[32753]: pgmap v3652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5649.515695 secs 2024-04-26T13:25:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:09 smithi194 ceph-mon[33017]: pgmap v3652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:25:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5649.515695 secs 2024-04-26T13:25:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:11 smithi033 ceph-mon[27845]: pgmap v3653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:25:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:11 smithi096 ceph-mon[32753]: pgmap v3653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:25:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:11 smithi194 ceph-mon[33017]: pgmap v3653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:25:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:13 smithi033 ceph-mon[27845]: pgmap v3654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5651.769734 secs 2024-04-26T13:25:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:13 smithi096 ceph-mon[32753]: pgmap v3654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5651.769734 secs 2024-04-26T13:25:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:13 smithi194 ceph-mon[33017]: pgmap v3654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5651.769734 secs 2024-04-26T13:25:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5654.515835 secs 2024-04-26T13:25:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:14 smithi194 ceph-mon[33017]: pgmap v3655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5654.515835 secs 2024-04-26T13:25:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:14 smithi033 ceph-mon[27845]: pgmap v3655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5654.515835 secs 2024-04-26T13:25:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:14 smithi096 ceph-mon[32753]: pgmap v3655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:16 smithi194 ceph-mon[33017]: pgmap v3656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:16 smithi033 ceph-mon[27845]: pgmap v3656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:16 smithi096 ceph-mon[32753]: pgmap v3656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5656.769819 secs 2024-04-26T13:25:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:18 smithi033 ceph-mon[27845]: pgmap v3657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5656.769819 secs 2024-04-26T13:25:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:18 smithi096 ceph-mon[32753]: pgmap v3657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5656.769819 secs 2024-04-26T13:25:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:18 smithi194 ceph-mon[33017]: pgmap v3657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5659.515993 secs 2024-04-26T13:25:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5659.515993 secs 2024-04-26T13:25:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5659.515993 secs 2024-04-26T13:25:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:20 smithi033 ceph-mon[27845]: pgmap v3658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:20 smithi096 ceph-mon[32753]: pgmap v3658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:20 smithi194 ceph-mon[33017]: pgmap v3658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:21.829+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:21.829+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:21.829+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:25:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:22 smithi033 ceph-mon[27845]: pgmap v3659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:22 smithi096 ceph-mon[32753]: pgmap v3659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:22 smithi194 ceph-mon[33017]: pgmap v3659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5661.769904 secs 2024-04-26T13:25:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5661.769904 secs 2024-04-26T13:25:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5661.769904 secs 2024-04-26T13:25:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5664.516139 secs 2024-04-26T13:25:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:24 smithi033 ceph-mon[27845]: pgmap v3660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5664.516139 secs 2024-04-26T13:25:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:24 smithi096 ceph-mon[32753]: pgmap v3660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5664.516139 secs 2024-04-26T13:25:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:24 smithi194 ceph-mon[33017]: pgmap v3660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:26 smithi033 ceph-mon[27845]: pgmap v3661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:26 smithi096 ceph-mon[32753]: pgmap v3661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:26 smithi194 ceph-mon[33017]: pgmap v3661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5666.769989 secs 2024-04-26T13:25:29.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:28 smithi033 ceph-mon[27845]: pgmap v3662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5666.769989 secs 2024-04-26T13:25:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:28 smithi096 ceph-mon[32753]: pgmap v3662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5666.769989 secs 2024-04-26T13:25:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:28 smithi194 ceph-mon[33017]: pgmap v3662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5669.516235 secs 2024-04-26T13:25:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5669.516235 secs 2024-04-26T13:25:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5669.516235 secs 2024-04-26T13:25:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:30 smithi033 ceph-mon[27845]: pgmap v3663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:30 smithi096 ceph-mon[32753]: pgmap v3663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:30 smithi194 ceph-mon[33017]: pgmap v3663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:32 smithi033 ceph-mon[27845]: pgmap v3664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:32 smithi096 ceph-mon[32753]: pgmap v3664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:32 smithi194 ceph-mon[33017]: pgmap v3664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5671.770047 secs 2024-04-26T13:25:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5671.770047 secs 2024-04-26T13:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5671.770047 secs 2024-04-26T13:25:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5674.516307 secs 2024-04-26T13:25:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:34 smithi096 ceph-mon[32753]: pgmap v3665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5674.516307 secs 2024-04-26T13:25:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:34 smithi194 ceph-mon[33017]: pgmap v3665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5674.516307 secs 2024-04-26T13:25:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:34 smithi033 ceph-mon[27845]: pgmap v3665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:36.830+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:36.830+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:25:36.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:36.830+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:37.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:36 smithi033 ceph-mon[27845]: pgmap v3666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:36 smithi194 ceph-mon[33017]: pgmap v3666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:36 smithi096 ceph-mon[32753]: pgmap v3666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5676.770105 secs 2024-04-26T13:25:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:38 smithi194 ceph-mon[33017]: pgmap v3667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5676.770105 secs 2024-04-26T13:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:38 smithi033 ceph-mon[27845]: pgmap v3667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5676.770105 secs 2024-04-26T13:25:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:38 smithi096 ceph-mon[32753]: pgmap v3667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5679.516422 secs 2024-04-26T13:25:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5679.516422 secs 2024-04-26T13:25:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5679.516422 secs 2024-04-26T13:25:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:40 smithi194 ceph-mon[33017]: pgmap v3668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:40 smithi033 ceph-mon[27845]: pgmap v3668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:40 smithi096 ceph-mon[32753]: pgmap v3668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:42 smithi194 ceph-mon[33017]: pgmap v3669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:42 smithi033 ceph-mon[27845]: pgmap v3669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:42 smithi096 ceph-mon[32753]: pgmap v3669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5681.770195 secs 2024-04-26T13:25:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5684.516544 secs 2024-04-26T13:25:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5681.770195 secs 2024-04-26T13:25:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5684.516544 secs 2024-04-26T13:25:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5681.770195 secs 2024-04-26T13:25:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5684.516544 secs 2024-04-26T13:25:45.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:44 smithi194 ceph-mon[33017]: pgmap v3670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:44 smithi033 ceph-mon[27845]: pgmap v3670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:44 smithi096 ceph-mon[32753]: pgmap v3670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:46 smithi194 ceph-mon[33017]: pgmap v3671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:46 smithi033 ceph-mon[27845]: pgmap v3671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:46 smithi096 ceph-mon[32753]: pgmap v3671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5686.770283 secs 2024-04-26T13:25:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:48 smithi194 ceph-mon[33017]: pgmap v3672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5689.516659 secs 2024-04-26T13:25:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5686.770283 secs 2024-04-26T13:25:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:48 smithi033 ceph-mon[27845]: pgmap v3672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5689.516659 secs 2024-04-26T13:25:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5686.770283 secs 2024-04-26T13:25:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:48 smithi096 ceph-mon[32753]: pgmap v3672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5689.516659 secs 2024-04-26T13:25:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:50 smithi194 ceph-mon[33017]: pgmap v3673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:50 smithi033 ceph-mon[27845]: pgmap v3673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:50 smithi096 ceph-mon[32753]: pgmap v3673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:51.844+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:51.845+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:25:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:25:51.845+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:25:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:52 smithi194 ceph-mon[33017]: pgmap v3674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:52 smithi033 ceph-mon[27845]: pgmap v3674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:52 smithi096 ceph-mon[32753]: pgmap v3674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5691.770371 secs 2024-04-26T13:25:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5694.516783 secs 2024-04-26T13:25:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5691.770371 secs 2024-04-26T13:25:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5694.516783 secs 2024-04-26T13:25:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5691.770371 secs 2024-04-26T13:25:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5694.516783 secs 2024-04-26T13:25:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:55 smithi194 ceph-mon[33017]: pgmap v3675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:55 smithi033 ceph-mon[27845]: pgmap v3675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:55 smithi096 ceph-mon[32753]: pgmap v3675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:57 smithi194 ceph-mon[33017]: pgmap v3676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:57 smithi033 ceph-mon[27845]: pgmap v3676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:57 smithi096 ceph-mon[32753]: pgmap v3676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:25:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5696.770433 secs 2024-04-26T13:25:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:59 smithi194 ceph-mon[33017]: pgmap v3677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:25:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5699.516928 secs 2024-04-26T13:25:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5696.770433 secs 2024-04-26T13:25:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:59 smithi033 ceph-mon[27845]: pgmap v3677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:25:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5699.516928 secs 2024-04-26T13:25:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5696.770433 secs 2024-04-26T13:25:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:59 smithi096 ceph-mon[32753]: pgmap v3677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:25:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:25:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5699.516928 secs 2024-04-26T13:26:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:26:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:26:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:26:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:01 smithi194 ceph-mon[33017]: pgmap v3678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:01.368 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:01 smithi096 ceph-mon[32753]: pgmap v3678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:01 smithi033 ceph-mon[27845]: pgmap v3678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:03 smithi194 ceph-mon[33017]: pgmap v3679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:03.323 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:03 smithi033 ceph-mon[27845]: pgmap v3679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:03 smithi096 ceph-mon[32753]: pgmap v3679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5701.770518 secs 2024-04-26T13:26:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5704.517021 secs 2024-04-26T13:26:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5701.770518 secs 2024-04-26T13:26:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5704.517021 secs 2024-04-26T13:26:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5701.770518 secs 2024-04-26T13:26:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5704.517021 secs 2024-04-26T13:26:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:05 smithi194 ceph-mon[33017]: pgmap v3680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:05 smithi033 ceph-mon[27845]: pgmap v3680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:05 smithi096 ceph-mon[32753]: pgmap v3680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:06.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:06.860+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:06.860+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:06.860+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:07 smithi194 ceph-mon[33017]: pgmap v3681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:07 smithi033 ceph-mon[27845]: pgmap v3681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:07 smithi096 ceph-mon[32753]: pgmap v3681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:26:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:08 smithi194 ceph-mon[33017]: pgmap v3682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T13:26:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:08 smithi194 ceph-mon[33017]: pgmap v3683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5706.770602 secs 2024-04-26T13:26:08.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:08 smithi033 ceph-mon[27845]: pgmap v3682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T13:26:08.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:08 smithi033 ceph-mon[27845]: pgmap v3683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:08.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5706.770602 secs 2024-04-26T13:26:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:08 smithi096 ceph-mon[32753]: pgmap v3682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T13:26:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:08 smithi096 ceph-mon[32753]: pgmap v3683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5706.770602 secs 2024-04-26T13:26:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5709.517103 secs 2024-04-26T13:26:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5709.517103 secs 2024-04-26T13:26:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5709.517103 secs 2024-04-26T13:26:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:10 smithi194 ceph-mon[33017]: pgmap v3684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:26:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:10 smithi033 ceph-mon[27845]: pgmap v3684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:26:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:10 smithi096 ceph-mon[32753]: pgmap v3684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T13:26:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:11 smithi194 ceph-mon[33017]: pgmap v3685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:11 smithi033 ceph-mon[27845]: pgmap v3685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:11 smithi096 ceph-mon[32753]: pgmap v3685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5711.770661 secs 2024-04-26T13:26:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:13 smithi194 ceph-mon[33017]: pgmap v3686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5711.770661 secs 2024-04-26T13:26:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:13 smithi033 ceph-mon[27845]: pgmap v3686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:14.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5711.770661 secs 2024-04-26T13:26:14.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:13 smithi096 ceph-mon[32753]: pgmap v3686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5714.517253 secs 2024-04-26T13:26:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5714.517253 secs 2024-04-26T13:26:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5714.517253 secs 2024-04-26T13:26:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:15 smithi194 ceph-mon[33017]: pgmap v3687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:15 smithi033 ceph-mon[27845]: pgmap v3687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:15 smithi096 ceph-mon[32753]: pgmap v3687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T13:26:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:17 smithi194 ceph-mon[33017]: pgmap v3688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:26:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:17 smithi033 ceph-mon[27845]: pgmap v3688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:26:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:17 smithi096 ceph-mon[32753]: pgmap v3688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:26:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5716.770746 secs 2024-04-26T13:26:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5716.770746 secs 2024-04-26T13:26:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5716.770746 secs 2024-04-26T13:26:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5719.517365 secs 2024-04-26T13:26:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:19 smithi033 ceph-mon[27845]: pgmap v3689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5719.517365 secs 2024-04-26T13:26:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:19 smithi096 ceph-mon[32753]: pgmap v3689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5719.517365 secs 2024-04-26T13:26:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:19 smithi194 ceph-mon[33017]: pgmap v3689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 msgr2=0x7feee40b2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T13:26:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 msgr2=0x7feee40b2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T13:26:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 0x7feee40b2ac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0018000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T13:26:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 0x7feee40b2ac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0018000 comp rx=0 tx=0).stop 2024-04-26T13:26:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 msgr2=0x7feee40b2ac0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T13:26:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 msgr2=0x7feee40b2ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T13:26:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af920 0x7feee40b2ac0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T13:26:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.721+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 0x7feee40b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T13:26:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.722+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 0x7feee40b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T13:26:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.724+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 0x7feee40b3000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0032000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T13:26:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.861+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.861+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:21.861+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:26:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:21 smithi033 ceph-mon[27845]: pgmap v3690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:21 smithi096 ceph-mon[32753]: pgmap v3690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:21 smithi194 ceph-mon[33017]: pgmap v3690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:22.761+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:26:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:22.761+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T13:26:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:22.762+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:26:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:22.762+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:26:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5721.770833 secs 2024-04-26T13:26:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:23 smithi033 ceph-mon[27845]: pgmap v3691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5721.770833 secs 2024-04-26T13:26:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:23 smithi096 ceph-mon[32753]: pgmap v3691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5721.770833 secs 2024-04-26T13:26:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:23 smithi194 ceph-mon[33017]: pgmap v3691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5724.517483 secs 2024-04-26T13:26:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5724.517483 secs 2024-04-26T13:26:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5724.517483 secs 2024-04-26T13:26:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:25 smithi033 ceph-mon[27845]: pgmap v3692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:25 smithi096 ceph-mon[32753]: pgmap v3692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:25 smithi194 ceph-mon[33017]: pgmap v3692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:27 smithi033 ceph-mon[27845]: pgmap v3693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:27 smithi096 ceph-mon[32753]: pgmap v3693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:27 smithi194 ceph-mon[33017]: pgmap v3693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5726.770915 secs 2024-04-26T13:26:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5726.770915 secs 2024-04-26T13:26:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5726.770915 secs 2024-04-26T13:26:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5729.517601 secs 2024-04-26T13:26:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:29 smithi033 ceph-mon[27845]: pgmap v3694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5729.517601 secs 2024-04-26T13:26:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:29 smithi096 ceph-mon[32753]: pgmap v3694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5729.517601 secs 2024-04-26T13:26:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:29 smithi194 ceph-mon[33017]: pgmap v3694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:31 smithi033 ceph-mon[27845]: pgmap v3695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:31 smithi096 ceph-mon[32753]: pgmap v3695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:31 smithi194 ceph-mon[33017]: pgmap v3695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5731.770999 secs 2024-04-26T13:26:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:33 smithi033 ceph-mon[27845]: pgmap v3696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5731.770999 secs 2024-04-26T13:26:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:33 smithi096 ceph-mon[32753]: pgmap v3696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5731.770999 secs 2024-04-26T13:26:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:33 smithi194 ceph-mon[33017]: pgmap v3696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5734.517721 secs 2024-04-26T13:26:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5734.517721 secs 2024-04-26T13:26:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5734.517721 secs 2024-04-26T13:26:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:35 smithi033 ceph-mon[27845]: pgmap v3697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:35 smithi096 ceph-mon[32753]: pgmap v3697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:35 smithi194 ceph-mon[33017]: pgmap v3697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:36.872+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:36.872+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:26:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:36.872+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:37 smithi096 ceph-mon[32753]: pgmap v3698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:37 smithi194 ceph-mon[33017]: pgmap v3698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:37 smithi033 ceph-mon[27845]: pgmap v3698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5736.771083 secs 2024-04-26T13:26:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5736.771083 secs 2024-04-26T13:26:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5736.771083 secs 2024-04-26T13:26:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5739.517837 secs 2024-04-26T13:26:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:39 smithi194 ceph-mon[33017]: pgmap v3699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5739.517837 secs 2024-04-26T13:26:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:39 smithi033 ceph-mon[27845]: pgmap v3699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5739.517837 secs 2024-04-26T13:26:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:39 smithi096 ceph-mon[32753]: pgmap v3699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:41 smithi194 ceph-mon[33017]: pgmap v3700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:41 smithi033 ceph-mon[27845]: pgmap v3700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:41 smithi096 ceph-mon[32753]: pgmap v3700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5741.771173 secs 2024-04-26T13:26:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:43 smithi194 ceph-mon[33017]: pgmap v3701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5741.771173 secs 2024-04-26T13:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:43 smithi033 ceph-mon[27845]: pgmap v3701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5741.771173 secs 2024-04-26T13:26:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:43 smithi096 ceph-mon[32753]: pgmap v3701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:45.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5744.517955 secs 2024-04-26T13:26:45.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5744.517955 secs 2024-04-26T13:26:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5744.517955 secs 2024-04-26T13:26:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:45 smithi194 ceph-mon[33017]: pgmap v3702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:45 smithi033 ceph-mon[27845]: pgmap v3702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:45 smithi096 ceph-mon[32753]: pgmap v3702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:47 smithi194 ceph-mon[33017]: pgmap v3703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:47 smithi033 ceph-mon[27845]: pgmap v3703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:47 smithi096 ceph-mon[32753]: pgmap v3703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5746.771235 secs 2024-04-26T13:26:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5749.518072 secs 2024-04-26T13:26:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5746.771235 secs 2024-04-26T13:26:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5749.518072 secs 2024-04-26T13:26:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5746.771235 secs 2024-04-26T13:26:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5749.518072 secs 2024-04-26T13:26:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:49 smithi194 ceph-mon[33017]: pgmap v3704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:49 smithi033 ceph-mon[27845]: pgmap v3704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:49 smithi096 ceph-mon[32753]: pgmap v3704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:51.876+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:51.876+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:26:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:26:51.876+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:26:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:51 smithi194 ceph-mon[33017]: pgmap v3705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:51 smithi033 ceph-mon[27845]: pgmap v3705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:51 smithi096 ceph-mon[32753]: pgmap v3705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5751.771296 secs 2024-04-26T13:26:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:53 smithi194 ceph-mon[33017]: pgmap v3706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5754.518207 secs 2024-04-26T13:26:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5751.771296 secs 2024-04-26T13:26:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:53 smithi033 ceph-mon[27845]: pgmap v3706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5754.518207 secs 2024-04-26T13:26:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5751.771296 secs 2024-04-26T13:26:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:53 smithi096 ceph-mon[32753]: pgmap v3706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5754.518207 secs 2024-04-26T13:26:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:55 smithi194 ceph-mon[33017]: pgmap v3707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:55 smithi033 ceph-mon[27845]: pgmap v3707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:55 smithi096 ceph-mon[32753]: pgmap v3707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:26:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:58 smithi194 ceph-mon[33017]: pgmap v3708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:58 smithi033 ceph-mon[27845]: pgmap v3708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:58 smithi096 ceph-mon[32753]: pgmap v3708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:26:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5756.771379 secs 2024-04-26T13:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:26:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5759.518321 secs 2024-04-26T13:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5756.771379 secs 2024-04-26T13:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:26:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5759.518321 secs 2024-04-26T13:26:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5756.771379 secs 2024-04-26T13:26:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:26:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5759.518321 secs 2024-04-26T13:27:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:00 smithi194 ceph-mon[33017]: pgmap v3709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:00 smithi033 ceph-mon[27845]: pgmap v3709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:00 smithi096 ceph-mon[32753]: pgmap v3709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:02 smithi194 ceph-mon[33017]: pgmap v3710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:02 smithi033 ceph-mon[27845]: pgmap v3710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:02 smithi096 ceph-mon[32753]: pgmap v3710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:04 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 5761.771465 secs 2024-04-26T13:27:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:04 smithi194 ceph-mon[33017]: slow request 5761.771465 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T13:27:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:04 smithi194 ceph-mon[33017]: pgmap v3711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:04 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 5764.518446 secs 2024-04-26T13:27:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:04 smithi194 ceph-mon[33017]: slow request 5764.518445 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T13:27:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:04 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 5761.771465 secs 2024-04-26T13:27:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:04 smithi033 ceph-mon[27845]: slow request 5761.771465 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T13:27:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:04 smithi033 ceph-mon[27845]: pgmap v3711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:04 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 5764.518446 secs 2024-04-26T13:27:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:04 smithi033 ceph-mon[27845]: slow request 5764.518445 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T13:27:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:04 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 5761.771465 secs 2024-04-26T13:27:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:04 smithi096 ceph-mon[32753]: slow request 5761.771465 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T13:27:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:04 smithi096 ceph-mon[32753]: pgmap v3711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:04 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 5764.518446 secs 2024-04-26T13:27:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:04 smithi096 ceph-mon[32753]: slow request 5764.518445 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T13:27:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:06 smithi194 ceph-mon[33017]: pgmap v3712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:06 smithi033 ceph-mon[27845]: pgmap v3712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:06 smithi096 ceph-mon[32753]: pgmap v3712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:06.882+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:06.882+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:06.882+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:27:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:08 smithi194 ceph-mon[33017]: pgmap v3713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:27:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:08 smithi033 ceph-mon[27845]: pgmap v3713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:27:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:08 smithi096 ceph-mon[32753]: pgmap v3713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:27:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5766.771552 secs 2024-04-26T13:27:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5769.518522 secs 2024-04-26T13:27:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:27:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:27:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:27:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5766.771552 secs 2024-04-26T13:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5769.518522 secs 2024-04-26T13:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5766.771552 secs 2024-04-26T13:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5769.518522 secs 2024-04-26T13:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:27:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:10 smithi194 ceph-mon[33017]: pgmap v3714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:10 smithi194 ceph-mon[33017]: pgmap v3715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:10 smithi033 ceph-mon[27845]: pgmap v3714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:10 smithi033 ceph-mon[27845]: pgmap v3715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:10 smithi096 ceph-mon[32753]: pgmap v3714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:10 smithi096 ceph-mon[32753]: pgmap v3715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:12 smithi194 ceph-mon[33017]: pgmap v3716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:12 smithi033 ceph-mon[27845]: pgmap v3716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:12 smithi096 ceph-mon[32753]: pgmap v3716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:14 smithi194 ceph-mon[33017]: pgmap v3717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5771.771634 secs 2024-04-26T13:27:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5774.518645 secs 2024-04-26T13:27:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:14 smithi033 ceph-mon[27845]: pgmap v3717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5771.771634 secs 2024-04-26T13:27:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5774.518645 secs 2024-04-26T13:27:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:14 smithi096 ceph-mon[32753]: pgmap v3717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5771.771634 secs 2024-04-26T13:27:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5774.518645 secs 2024-04-26T13:27:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:16 smithi194 ceph-mon[33017]: pgmap v3718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:27:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:16 smithi033 ceph-mon[27845]: pgmap v3718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:27:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:16 smithi096 ceph-mon[32753]: pgmap v3718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:27:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:18 smithi194 ceph-mon[33017]: pgmap v3719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:18 smithi033 ceph-mon[27845]: pgmap v3719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:18 smithi096 ceph-mon[32753]: pgmap v3719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5776.771719 secs 2024-04-26T13:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5779.518736 secs 2024-04-26T13:27:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5776.771719 secs 2024-04-26T13:27:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5779.518736 secs 2024-04-26T13:27:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5776.771719 secs 2024-04-26T13:27:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5779.518736 secs 2024-04-26T13:27:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:20 smithi194 ceph-mon[33017]: pgmap v3720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:27:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:20 smithi033 ceph-mon[27845]: pgmap v3720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:27:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:20 smithi096 ceph-mon[32753]: pgmap v3720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:27:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:21.898+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:21.898+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:21.898+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:27:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:22 smithi194 ceph-mon[33017]: pgmap v3721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:22 smithi033 ceph-mon[27845]: pgmap v3721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:22 smithi096 ceph-mon[32753]: pgmap v3721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:24 smithi194 ceph-mon[33017]: pgmap v3722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5781.771777 secs 2024-04-26T13:27:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5784.518852 secs 2024-04-26T13:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:24 smithi033 ceph-mon[27845]: pgmap v3722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5781.771777 secs 2024-04-26T13:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5784.518852 secs 2024-04-26T13:27:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:24 smithi096 ceph-mon[32753]: pgmap v3722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5781.771777 secs 2024-04-26T13:27:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5784.518852 secs 2024-04-26T13:27:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:26 smithi194 ceph-mon[33017]: pgmap v3723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:26 smithi033 ceph-mon[27845]: pgmap v3723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:26 smithi096 ceph-mon[32753]: pgmap v3723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:28 smithi033 ceph-mon[27845]: pgmap v3724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:28 smithi096 ceph-mon[32753]: pgmap v3724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:28 smithi194 ceph-mon[33017]: pgmap v3724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5786.771862 secs 2024-04-26T13:27:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5789.518970 secs 2024-04-26T13:27:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5786.771862 secs 2024-04-26T13:27:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5789.518970 secs 2024-04-26T13:27:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5786.771862 secs 2024-04-26T13:27:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5789.518970 secs 2024-04-26T13:27:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:30 smithi033 ceph-mon[27845]: pgmap v3725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:30 smithi096 ceph-mon[32753]: pgmap v3725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:30 smithi194 ceph-mon[33017]: pgmap v3725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:32 smithi033 ceph-mon[27845]: pgmap v3726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:32 smithi096 ceph-mon[32753]: pgmap v3726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:32 smithi194 ceph-mon[33017]: pgmap v3726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5791.771947 secs 2024-04-26T13:27:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5791.771947 secs 2024-04-26T13:27:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5791.771947 secs 2024-04-26T13:27:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:34 smithi033 ceph-mon[27845]: pgmap v3727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5794.519112 secs 2024-04-26T13:27:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:34 smithi096 ceph-mon[32753]: pgmap v3727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5794.519112 secs 2024-04-26T13:27:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:34 smithi194 ceph-mon[33017]: pgmap v3727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5794.519112 secs 2024-04-26T13:27:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:36 smithi033 ceph-mon[27845]: pgmap v3728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:36 smithi096 ceph-mon[32753]: pgmap v3728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:36 smithi194 ceph-mon[33017]: pgmap v3728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:36.914+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:36.914+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:36.914+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:27:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:38 smithi033 ceph-mon[27845]: pgmap v3729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5796.772031 secs 2024-04-26T13:27:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:38 smithi096 ceph-mon[32753]: pgmap v3729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5796.772031 secs 2024-04-26T13:27:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:38 smithi194 ceph-mon[33017]: pgmap v3729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5796.772031 secs 2024-04-26T13:27:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5799.519198 secs 2024-04-26T13:27:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5799.519198 secs 2024-04-26T13:27:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5799.519198 secs 2024-04-26T13:27:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:40 smithi033 ceph-mon[27845]: pgmap v3730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:40 smithi096 ceph-mon[32753]: pgmap v3730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:40 smithi194 ceph-mon[33017]: pgmap v3730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:42 smithi033 ceph-mon[27845]: pgmap v3731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:42 smithi096 ceph-mon[32753]: pgmap v3731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:42 smithi194 ceph-mon[33017]: pgmap v3731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5801.772118 secs 2024-04-26T13:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5801.772118 secs 2024-04-26T13:27:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5801.772118 secs 2024-04-26T13:27:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:44 smithi194 ceph-mon[33017]: pgmap v3732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5804.519319 secs 2024-04-26T13:27:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:44 smithi033 ceph-mon[27845]: pgmap v3732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5804.519319 secs 2024-04-26T13:27:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:44 smithi096 ceph-mon[32753]: pgmap v3732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5804.519319 secs 2024-04-26T13:27:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:46 smithi194 ceph-mon[33017]: pgmap v3733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:46 smithi033 ceph-mon[27845]: pgmap v3733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:46 smithi096 ceph-mon[32753]: pgmap v3733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:48 smithi194 ceph-mon[33017]: pgmap v3734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5806.772179 secs 2024-04-26T13:27:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:48 smithi033 ceph-mon[27845]: pgmap v3734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5806.772179 secs 2024-04-26T13:27:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:48 smithi096 ceph-mon[32753]: pgmap v3734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5806.772179 secs 2024-04-26T13:27:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5809.519451 secs 2024-04-26T13:27:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5809.519451 secs 2024-04-26T13:27:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5809.519451 secs 2024-04-26T13:27:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:50 smithi194 ceph-mon[33017]: pgmap v3735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:50 smithi033 ceph-mon[27845]: pgmap v3735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:50 smithi096 ceph-mon[32753]: pgmap v3735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:51.921+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:51.921+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:27:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:27:51.921+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:27:52.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:52 smithi096 ceph-mon[32753]: pgmap v3736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:52 smithi194 ceph-mon[33017]: pgmap v3736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:52.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:52 smithi033 ceph-mon[27845]: pgmap v3736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:27:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5811.772264 secs 2024-04-26T13:27:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5811.772264 secs 2024-04-26T13:27:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5811.772264 secs 2024-04-26T13:27:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:54 smithi194 ceph-mon[33017]: pgmap v3737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5814.519574 secs 2024-04-26T13:27:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:54 smithi033 ceph-mon[27845]: pgmap v3737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5814.519574 secs 2024-04-26T13:27:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:54 smithi096 ceph-mon[32753]: pgmap v3737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:27:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5814.519574 secs 2024-04-26T13:27:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:56 smithi194 ceph-mon[33017]: pgmap v3738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:56 smithi033 ceph-mon[27845]: pgmap v3738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:56 smithi096 ceph-mon[32753]: pgmap v3738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:58 smithi033 ceph-mon[27845]: pgmap v3739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5816.772349 secs 2024-04-26T13:27:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:58 smithi194 ceph-mon[33017]: pgmap v3739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5816.772349 secs 2024-04-26T13:27:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:58 smithi096 ceph-mon[32753]: pgmap v3739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:27:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5816.772349 secs 2024-04-26T13:27:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 5819.519701 secs 2024-04-26T13:27:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:27:59 smithi194 ceph-mon[33017]: slow request 5761.247489 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T13:27:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 5819.519701 secs 2024-04-26T13:27:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:27:59 smithi033 ceph-mon[27845]: slow request 5761.247489 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T13:27:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 5819.519701 secs 2024-04-26T13:27:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:27:59 smithi096 ceph-mon[32753]: slow request 5761.247489 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T13:28:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:00 smithi194 ceph-mon[33017]: pgmap v3740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:00 smithi033 ceph-mon[27845]: pgmap v3740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:00 smithi096 ceph-mon[32753]: pgmap v3740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:02 smithi194 ceph-mon[33017]: pgmap v3741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:02 smithi033 ceph-mon[27845]: pgmap v3741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:02 smithi096 ceph-mon[32753]: pgmap v3741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5821.772425 secs 2024-04-26T13:28:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5821.772425 secs 2024-04-26T13:28:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5821.772425 secs 2024-04-26T13:28:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:04 smithi194 ceph-mon[33017]: pgmap v3742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5824.519823 secs 2024-04-26T13:28:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:04 smithi033 ceph-mon[27845]: pgmap v3742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5824.519823 secs 2024-04-26T13:28:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:04 smithi096 ceph-mon[32753]: pgmap v3742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5824.519823 secs 2024-04-26T13:28:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:06 smithi194 ceph-mon[33017]: pgmap v3743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:06 smithi033 ceph-mon[27845]: pgmap v3743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:06 smithi096 ceph-mon[32753]: pgmap v3743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:06.934+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:06.934+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:06.934+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:28:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:08 smithi194 ceph-mon[33017]: pgmap v3744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5826.772501 secs 2024-04-26T13:28:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:08 smithi033 ceph-mon[27845]: pgmap v3744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5826.772501 secs 2024-04-26T13:28:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:08 smithi096 ceph-mon[32753]: pgmap v3744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5826.772501 secs 2024-04-26T13:28:09.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5829.519922 secs 2024-04-26T13:28:09.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:09 smithi033 ceph-mon[27845]: pgmap v3745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:09.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:28:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5829.519922 secs 2024-04-26T13:28:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:09 smithi194 ceph-mon[33017]: pgmap v3745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:28:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5829.519922 secs 2024-04-26T13:28:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:09 smithi096 ceph-mon[32753]: pgmap v3745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:28:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:28:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:28:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:10 smithi194 ceph-mon[33017]: pgmap v3746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:28:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:10 smithi194 ceph-mon[33017]: pgmap v3747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:28:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:28:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:28:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:10 smithi033 ceph-mon[27845]: pgmap v3746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:28:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:10 smithi033 ceph-mon[27845]: pgmap v3747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:10 smithi096 ceph-mon[32753]: pgmap v3746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:10 smithi096 ceph-mon[32753]: pgmap v3747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:28:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:13 smithi033 ceph-mon[27845]: pgmap v3748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:13 smithi096 ceph-mon[32753]: pgmap v3748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:13 smithi194 ceph-mon[33017]: pgmap v3748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5831.772585 secs 2024-04-26T13:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5834.520075 secs 2024-04-26T13:28:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5831.772585 secs 2024-04-26T13:28:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5834.520075 secs 2024-04-26T13:28:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5831.772585 secs 2024-04-26T13:28:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5834.520075 secs 2024-04-26T13:28:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:15 smithi033 ceph-mon[27845]: pgmap v3749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:28:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:15 smithi096 ceph-mon[32753]: pgmap v3749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:28:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:15 smithi194 ceph-mon[33017]: pgmap v3749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:28:17.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:17 smithi194 ceph-mon[33017]: pgmap v3750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:17 smithi033 ceph-mon[27845]: pgmap v3750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:17 smithi096 ceph-mon[32753]: pgmap v3750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5836.772680 secs 2024-04-26T13:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:19 smithi033 ceph-mon[27845]: pgmap v3751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5839.520128 secs 2024-04-26T13:28:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5836.772680 secs 2024-04-26T13:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:19 smithi096 ceph-mon[32753]: pgmap v3751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5839.520128 secs 2024-04-26T13:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5836.772680 secs 2024-04-26T13:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:19 smithi194 ceph-mon[33017]: pgmap v3751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5839.520128 secs 2024-04-26T13:28:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:21 smithi033 ceph-mon[27845]: pgmap v3752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:28:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:21 smithi096 ceph-mon[32753]: pgmap v3752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:28:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:21 smithi194 ceph-mon[33017]: pgmap v3752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:28:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:21.950+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:21.950+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:21.950+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:28:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:23 smithi033 ceph-mon[27845]: pgmap v3753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5841.772765 secs 2024-04-26T13:28:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:23 smithi096 ceph-mon[32753]: pgmap v3753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5841.772765 secs 2024-04-26T13:28:23.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:23 smithi194 ceph-mon[33017]: pgmap v3753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:23.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5841.772765 secs 2024-04-26T13:28:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5844.520198 secs 2024-04-26T13:28:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5844.520198 secs 2024-04-26T13:28:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5844.520198 secs 2024-04-26T13:28:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:25 smithi033 ceph-mon[27845]: pgmap v3754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:25 smithi096 ceph-mon[32753]: pgmap v3754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:25 smithi194 ceph-mon[33017]: pgmap v3754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:27 smithi033 ceph-mon[27845]: pgmap v3755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:27 smithi096 ceph-mon[32753]: pgmap v3755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:27 smithi194 ceph-mon[33017]: pgmap v3755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5846.772851 secs 2024-04-26T13:28:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5846.772851 secs 2024-04-26T13:28:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5846.772851 secs 2024-04-26T13:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:29 smithi033 ceph-mon[27845]: pgmap v3756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5849.520320 secs 2024-04-26T13:28:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:29 smithi096 ceph-mon[32753]: pgmap v3756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5849.520320 secs 2024-04-26T13:28:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:29 smithi194 ceph-mon[33017]: pgmap v3756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5849.520320 secs 2024-04-26T13:28:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:31 smithi033 ceph-mon[27845]: pgmap v3757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:31 smithi096 ceph-mon[32753]: pgmap v3757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:31 smithi194 ceph-mon[33017]: pgmap v3757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:33 smithi096 ceph-mon[32753]: pgmap v3758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5851.772930 secs 2024-04-26T13:28:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:33 smithi194 ceph-mon[33017]: pgmap v3758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5851.772930 secs 2024-04-26T13:28:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:33 smithi033 ceph-mon[27845]: pgmap v3758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5851.772930 secs 2024-04-26T13:28:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5854.520445 secs 2024-04-26T13:28:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5854.520445 secs 2024-04-26T13:28:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5854.520445 secs 2024-04-26T13:28:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:35 smithi194 ceph-mon[33017]: pgmap v3759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:35 smithi033 ceph-mon[27845]: pgmap v3759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:35 smithi096 ceph-mon[32753]: pgmap v3759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:36.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:36.959+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:36.959+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:36.959+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:28:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:37 smithi194 ceph-mon[33017]: pgmap v3760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:37 smithi033 ceph-mon[27845]: pgmap v3760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:37 smithi096 ceph-mon[32753]: pgmap v3760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:37.785+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T13:28:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:37.785+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:28:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:37.786+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T13:28:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:37.786+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:28:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5856.772995 secs 2024-04-26T13:28:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5856.772995 secs 2024-04-26T13:28:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5856.772995 secs 2024-04-26T13:28:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:39 smithi033 ceph-mon[27845]: pgmap v3761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5859.520563 secs 2024-04-26T13:28:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:39 smithi194 ceph-mon[33017]: pgmap v3761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5859.520563 secs 2024-04-26T13:28:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:39 smithi096 ceph-mon[32753]: pgmap v3761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5859.520563 secs 2024-04-26T13:28:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:41 smithi194 ceph-mon[33017]: pgmap v3762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:41 smithi033 ceph-mon[27845]: pgmap v3762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:41 smithi096 ceph-mon[32753]: pgmap v3762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:43 smithi194 ceph-mon[33017]: pgmap v3763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5861.773082 secs 2024-04-26T13:28:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:43 smithi033 ceph-mon[27845]: pgmap v3763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5861.773082 secs 2024-04-26T13:28:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:43 smithi096 ceph-mon[32753]: pgmap v3763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5861.773082 secs 2024-04-26T13:28:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5864.520678 secs 2024-04-26T13:28:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:44 smithi194 ceph-mon[33017]: pgmap v3764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5864.520678 secs 2024-04-26T13:28:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:44 smithi033 ceph-mon[27845]: pgmap v3764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5864.520678 secs 2024-04-26T13:28:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:44 smithi096 ceph-mon[32753]: pgmap v3764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:47 smithi033 ceph-mon[27845]: pgmap v3765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:47 smithi096 ceph-mon[32753]: pgmap v3765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:47 smithi194 ceph-mon[33017]: pgmap v3765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5866.773167 secs 2024-04-26T13:28:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:49 smithi033 ceph-mon[27845]: pgmap v3766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5869.520822 secs 2024-04-26T13:28:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5866.773167 secs 2024-04-26T13:28:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:49 smithi096 ceph-mon[32753]: pgmap v3766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5869.520822 secs 2024-04-26T13:28:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5866.773167 secs 2024-04-26T13:28:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:49 smithi194 ceph-mon[33017]: pgmap v3766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5869.520822 secs 2024-04-26T13:28:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:51 smithi033 ceph-mon[27845]: pgmap v3767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:51 smithi096 ceph-mon[32753]: pgmap v3767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:51 smithi194 ceph-mon[33017]: pgmap v3767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:51.971+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:51.971+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:28:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:28:51.971+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:28:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:53 smithi033 ceph-mon[27845]: pgmap v3768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5871.773251 secs 2024-04-26T13:28:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:53 smithi096 ceph-mon[32753]: pgmap v3768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5871.773251 secs 2024-04-26T13:28:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:53 smithi194 ceph-mon[33017]: pgmap v3768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5871.773251 secs 2024-04-26T13:28:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5874.520919 secs 2024-04-26T13:28:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5874.520919 secs 2024-04-26T13:28:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5874.520919 secs 2024-04-26T13:28:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:55 smithi033 ceph-mon[27845]: pgmap v3769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:55 smithi096 ceph-mon[32753]: pgmap v3769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:55 smithi194 ceph-mon[33017]: pgmap v3769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:57 smithi033 ceph-mon[27845]: pgmap v3770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:57 smithi096 ceph-mon[32753]: pgmap v3770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:57 smithi194 ceph-mon[33017]: pgmap v3770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:28:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5876.773339 secs 2024-04-26T13:28:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5876.773339 secs 2024-04-26T13:28:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5876.773339 secs 2024-04-26T13:28:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:59 smithi033 ceph-mon[27845]: pgmap v3771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:28:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5879.521070 secs 2024-04-26T13:28:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:59 smithi096 ceph-mon[32753]: pgmap v3771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:28:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5879.521070 secs 2024-04-26T13:28:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:59 smithi194 ceph-mon[33017]: pgmap v3771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:28:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:28:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5879.521070 secs 2024-04-26T13:29:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:01 smithi033 ceph-mon[27845]: pgmap v3772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:01 smithi096 ceph-mon[32753]: pgmap v3772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:01 smithi194 ceph-mon[33017]: pgmap v3772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:03 smithi096 ceph-mon[32753]: pgmap v3773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5881.773422 secs 2024-04-26T13:29:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:03 smithi194 ceph-mon[33017]: pgmap v3773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5881.773422 secs 2024-04-26T13:29:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:03 smithi033 ceph-mon[27845]: pgmap v3773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5881.773422 secs 2024-04-26T13:29:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5884.521192 secs 2024-04-26T13:29:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5884.521192 secs 2024-04-26T13:29:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5884.521192 secs 2024-04-26T13:29:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:05 smithi194 ceph-mon[33017]: pgmap v3774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:05 smithi033 ceph-mon[27845]: pgmap v3774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:05 smithi096 ceph-mon[32753]: pgmap v3774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:06.986+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:06.986+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:06.986+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:29:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:07 smithi194 ceph-mon[33017]: pgmap v3775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:07 smithi033 ceph-mon[27845]: pgmap v3775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:07 smithi096 ceph-mon[32753]: pgmap v3775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5886.773505 secs 2024-04-26T13:29:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5886.773505 secs 2024-04-26T13:29:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5886.773505 secs 2024-04-26T13:29:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:09 smithi033 ceph-mon[27845]: pgmap v3776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:09.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5889.521318 secs 2024-04-26T13:29:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:09 smithi194 ceph-mon[33017]: pgmap v3776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5889.521318 secs 2024-04-26T13:29:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:09 smithi096 ceph-mon[32753]: pgmap v3776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5889.521318 secs 2024-04-26T13:29:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:11 smithi033 ceph-mon[27845]: pgmap v3777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:29:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:11 smithi194 ceph-mon[33017]: pgmap v3777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:29:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:11 smithi096 ceph-mon[32753]: pgmap v3777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:29:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:29:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:29:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:29:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:29:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:29:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:29:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:29:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:29:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:29:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:13 smithi194 ceph-mon[33017]: pgmap v3778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:13 smithi194 ceph-mon[33017]: pgmap v3779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5891.773591 secs 2024-04-26T13:29:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:13 smithi033 ceph-mon[27845]: pgmap v3778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:13 smithi033 ceph-mon[27845]: pgmap v3779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5891.773591 secs 2024-04-26T13:29:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:13 smithi096 ceph-mon[32753]: pgmap v3778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:13 smithi096 ceph-mon[32753]: pgmap v3779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5891.773591 secs 2024-04-26T13:29:15.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5894.521439 secs 2024-04-26T13:29:15.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:14 smithi033 ceph-mon[27845]: pgmap v3780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5894.521439 secs 2024-04-26T13:29:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:14 smithi194 ceph-mon[33017]: pgmap v3780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5894.521439 secs 2024-04-26T13:29:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:14 smithi096 ceph-mon[32753]: pgmap v3780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:16 smithi194 ceph-mon[33017]: pgmap v3781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:16 smithi033 ceph-mon[27845]: pgmap v3781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:16 smithi096 ceph-mon[32753]: pgmap v3781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5896.773652 secs 2024-04-26T13:29:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:18 smithi194 ceph-mon[33017]: pgmap v3782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5896.773652 secs 2024-04-26T13:29:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:18 smithi033 ceph-mon[27845]: pgmap v3782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5896.773652 secs 2024-04-26T13:29:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:18 smithi096 ceph-mon[32753]: pgmap v3782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5899.521562 secs 2024-04-26T13:29:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5899.521562 secs 2024-04-26T13:29:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5899.521562 secs 2024-04-26T13:29:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:20 smithi194 ceph-mon[33017]: pgmap v3783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:20 smithi033 ceph-mon[27845]: pgmap v3783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:20 smithi096 ceph-mon[32753]: pgmap v3783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:29:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:22.002+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:22.002+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:22.002+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:29:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:22.793+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:29:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:22.793+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T13:29:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:22.794+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:29:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:22.794+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T13:29:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:22 smithi096 ceph-mon[32753]: pgmap v3784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:29:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:22 smithi194 ceph-mon[33017]: pgmap v3784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:29:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:22 smithi033 ceph-mon[27845]: pgmap v3784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:29:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5901.773751 secs 2024-04-26T13:29:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5901.773751 secs 2024-04-26T13:29:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5901.773751 secs 2024-04-26T13:29:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5904.521642 secs 2024-04-26T13:29:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:24 smithi194 ceph-mon[33017]: pgmap v3785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5904.521642 secs 2024-04-26T13:29:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:24 smithi033 ceph-mon[27845]: pgmap v3785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5904.521642 secs 2024-04-26T13:29:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:24 smithi096 ceph-mon[32753]: pgmap v3785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:26 smithi194 ceph-mon[33017]: pgmap v3786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:26 smithi033 ceph-mon[27845]: pgmap v3786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:26 smithi096 ceph-mon[32753]: pgmap v3786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5906.773837 secs 2024-04-26T13:29:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:28 smithi194 ceph-mon[33017]: pgmap v3787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5909.521767 secs 2024-04-26T13:29:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5906.773837 secs 2024-04-26T13:29:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:28 smithi033 ceph-mon[27845]: pgmap v3787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5909.521767 secs 2024-04-26T13:29:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5906.773837 secs 2024-04-26T13:29:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:28 smithi096 ceph-mon[32753]: pgmap v3787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5909.521767 secs 2024-04-26T13:29:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:30 smithi194 ceph-mon[33017]: pgmap v3788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:30 smithi033 ceph-mon[27845]: pgmap v3788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:30 smithi096 ceph-mon[32753]: pgmap v3788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:32 smithi194 ceph-mon[33017]: pgmap v3789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:32 smithi033 ceph-mon[27845]: pgmap v3789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:32 smithi096 ceph-mon[32753]: pgmap v3789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5911.773920 secs 2024-04-26T13:29:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5914.521922 secs 2024-04-26T13:29:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5911.773920 secs 2024-04-26T13:29:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5914.521922 secs 2024-04-26T13:29:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5911.773920 secs 2024-04-26T13:29:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5914.521922 secs 2024-04-26T13:29:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:35 smithi194 ceph-mon[33017]: pgmap v3790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:35 smithi033 ceph-mon[27845]: pgmap v3790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:35 smithi096 ceph-mon[32753]: pgmap v3790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:37.017+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:37.017+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:37.017+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:29:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:37 smithi194 ceph-mon[33017]: pgmap v3791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:37 smithi033 ceph-mon[27845]: pgmap v3791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:37 smithi096 ceph-mon[32753]: pgmap v3791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5916.774026 secs 2024-04-26T13:29:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:39 smithi194 ceph-mon[33017]: pgmap v3792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5919.522021 secs 2024-04-26T13:29:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5916.774026 secs 2024-04-26T13:29:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:39 smithi033 ceph-mon[27845]: pgmap v3792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5919.522021 secs 2024-04-26T13:29:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5916.774026 secs 2024-04-26T13:29:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:39 smithi096 ceph-mon[32753]: pgmap v3792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5919.522021 secs 2024-04-26T13:29:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:41 smithi194 ceph-mon[33017]: pgmap v3793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:41 smithi033 ceph-mon[27845]: pgmap v3793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:41 smithi096 ceph-mon[32753]: pgmap v3793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:43 smithi194 ceph-mon[33017]: pgmap v3794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:43 smithi033 ceph-mon[27845]: pgmap v3794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:43 smithi096 ceph-mon[32753]: pgmap v3794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5921.774111 secs 2024-04-26T13:29:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5924.522176 secs 2024-04-26T13:29:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5921.774111 secs 2024-04-26T13:29:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5924.522176 secs 2024-04-26T13:29:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5921.774111 secs 2024-04-26T13:29:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5924.522176 secs 2024-04-26T13:29:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:45 smithi194 ceph-mon[33017]: pgmap v3795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:45 smithi033 ceph-mon[27845]: pgmap v3795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:45 smithi096 ceph-mon[32753]: pgmap v3795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:47 smithi194 ceph-mon[33017]: pgmap v3796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:47 smithi033 ceph-mon[27845]: pgmap v3796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:47 smithi096 ceph-mon[32753]: pgmap v3796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5926.774169 secs 2024-04-26T13:29:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:49 smithi194 ceph-mon[33017]: pgmap v3797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5929.522305 secs 2024-04-26T13:29:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5926.774169 secs 2024-04-26T13:29:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:49 smithi033 ceph-mon[27845]: pgmap v3797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5929.522305 secs 2024-04-26T13:29:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5926.774169 secs 2024-04-26T13:29:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:49 smithi096 ceph-mon[32753]: pgmap v3797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5929.522305 secs 2024-04-26T13:29:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:51 smithi194 ceph-mon[33017]: pgmap v3798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:51 smithi033 ceph-mon[27845]: pgmap v3798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:51 smithi096 ceph-mon[32753]: pgmap v3798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:52.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:52.033+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:52.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:52.033+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:29:52.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:29:52.033+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:29:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:53 smithi194 ceph-mon[33017]: pgmap v3799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:53 smithi033 ceph-mon[27845]: pgmap v3799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:53 smithi096 ceph-mon[32753]: pgmap v3799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5931.774273 secs 2024-04-26T13:29:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5934.522370 secs 2024-04-26T13:29:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5931.774273 secs 2024-04-26T13:29:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5934.522370 secs 2024-04-26T13:29:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5931.774273 secs 2024-04-26T13:29:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5934.522370 secs 2024-04-26T13:29:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:55 smithi194 ceph-mon[33017]: pgmap v3800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:55 smithi033 ceph-mon[27845]: pgmap v3800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:55 smithi096 ceph-mon[32753]: pgmap v3800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:57 smithi194 ceph-mon[33017]: pgmap v3801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:57 smithi033 ceph-mon[27845]: pgmap v3801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:57 smithi096 ceph-mon[32753]: pgmap v3801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:29:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5936.774357 secs 2024-04-26T13:29:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:59 smithi194 ceph-mon[33017]: pgmap v3802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:29:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5939.522451 secs 2024-04-26T13:29:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5936.774357 secs 2024-04-26T13:29:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:59 smithi033 ceph-mon[27845]: pgmap v3802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:29:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5939.522451 secs 2024-04-26T13:29:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5936.774357 secs 2024-04-26T13:29:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:59 smithi096 ceph-mon[32753]: pgmap v3802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:29:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:29:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5939.522451 secs 2024-04-26T13:30:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:30:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:30:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:30:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:01 smithi194 ceph-mon[33017]: pgmap v3803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:01 smithi033 ceph-mon[27845]: pgmap v3803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:01 smithi096 ceph-mon[32753]: pgmap v3803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:03 smithi033 ceph-mon[27845]: pgmap v3804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:03 smithi096 ceph-mon[32753]: pgmap v3804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:03 smithi194 ceph-mon[33017]: pgmap v3804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5941.774441 secs 2024-04-26T13:30:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5944.522576 secs 2024-04-26T13:30:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5941.774441 secs 2024-04-26T13:30:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5944.522576 secs 2024-04-26T13:30:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5941.774441 secs 2024-04-26T13:30:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5944.522576 secs 2024-04-26T13:30:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:05 smithi033 ceph-mon[27845]: pgmap v3805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:05 smithi096 ceph-mon[32753]: pgmap v3805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:05 smithi194 ceph-mon[33017]: pgmap v3805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:07.038+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:07.038+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:07.038+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:30:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:07 smithi033 ceph-mon[27845]: pgmap v3806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:07 smithi096 ceph-mon[32753]: pgmap v3806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:07 smithi194 ceph-mon[33017]: pgmap v3806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5946.774523 secs 2024-04-26T13:30:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:09 smithi033 ceph-mon[27845]: pgmap v3807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5949.522736 secs 2024-04-26T13:30:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5946.774523 secs 2024-04-26T13:30:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:09 smithi096 ceph-mon[32753]: pgmap v3807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5949.522736 secs 2024-04-26T13:30:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5946.774523 secs 2024-04-26T13:30:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:09 smithi194 ceph-mon[33017]: pgmap v3807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5949.522736 secs 2024-04-26T13:30:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:11 smithi033 ceph-mon[27845]: pgmap v3808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:11 smithi096 ceph-mon[32753]: pgmap v3808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:11 smithi194 ceph-mon[33017]: pgmap v3808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:30:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:30:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:30:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:13 smithi033 ceph-mon[27845]: pgmap v3809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5951.774608 secs 2024-04-26T13:30:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:13 smithi096 ceph-mon[32753]: pgmap v3809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5951.774608 secs 2024-04-26T13:30:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:13 smithi194 ceph-mon[33017]: pgmap v3809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5951.774608 secs 2024-04-26T13:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5954.522857 secs 2024-04-26T13:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:14 smithi033 ceph-mon[27845]: pgmap v3810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:14 smithi033 ceph-mon[27845]: pgmap v3811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:30:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5954.522857 secs 2024-04-26T13:30:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:30:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:30:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:14 smithi096 ceph-mon[32753]: pgmap v3810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:14 smithi096 ceph-mon[32753]: pgmap v3811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:30:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5954.522857 secs 2024-04-26T13:30:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:30:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:30:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:14 smithi194 ceph-mon[33017]: pgmap v3810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:14 smithi194 ceph-mon[33017]: pgmap v3811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:30:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:15 smithi033 ceph-mon[27845]: pgmap v3812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:15 smithi096 ceph-mon[32753]: pgmap v3812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:15 smithi194 ceph-mon[33017]: pgmap v3812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:17.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:17 smithi194 ceph-mon[33017]: pgmap v3813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:17 smithi033 ceph-mon[27845]: pgmap v3813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:17 smithi096 ceph-mon[32753]: pgmap v3813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5956.774693 secs 2024-04-26T13:30:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5956.774693 secs 2024-04-26T13:30:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5956.774693 secs 2024-04-26T13:30:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5959.522979 secs 2024-04-26T13:30:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:19 smithi033 ceph-mon[27845]: pgmap v3814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T13:30:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5959.522979 secs 2024-04-26T13:30:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:19 smithi096 ceph-mon[32753]: pgmap v3814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T13:30:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5959.522979 secs 2024-04-26T13:30:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:19 smithi194 ceph-mon[33017]: pgmap v3814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T13:30:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:21 smithi033 ceph-mon[27845]: pgmap v3815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:21 smithi096 ceph-mon[32753]: pgmap v3815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:21 smithi194 ceph-mon[33017]: pgmap v3815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:30:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:22.045+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:22.046+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:22.046+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:30:23.487 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5961.774774 secs 2024-04-26T13:30:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5961.774774 secs 2024-04-26T13:30:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5961.774774 secs 2024-04-26T13:30:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5964.523102 secs 2024-04-26T13:30:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:24 smithi194 ceph-mon[33017]: pgmap v3816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:30:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5964.523102 secs 2024-04-26T13:30:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:24 smithi033 ceph-mon[27845]: pgmap v3816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:30:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5964.523102 secs 2024-04-26T13:30:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:24 smithi096 ceph-mon[32753]: pgmap v3816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:30:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:25 smithi033 ceph-mon[27845]: pgmap v3817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:25 smithi096 ceph-mon[32753]: pgmap v3817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:25 smithi194 ceph-mon[33017]: pgmap v3817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:27 smithi033 ceph-mon[27845]: pgmap v3818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:30:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:27 smithi096 ceph-mon[32753]: pgmap v3818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:30:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:27 smithi194 ceph-mon[33017]: pgmap v3818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:30:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5966.774860 secs 2024-04-26T13:30:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5966.774860 secs 2024-04-26T13:30:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5966.774860 secs 2024-04-26T13:30:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5969.523246 secs 2024-04-26T13:30:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:29 smithi033 ceph-mon[27845]: pgmap v3819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5969.523246 secs 2024-04-26T13:30:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:29 smithi096 ceph-mon[32753]: pgmap v3819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5969.523246 secs 2024-04-26T13:30:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:29 smithi194 ceph-mon[33017]: pgmap v3819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:31 smithi033 ceph-mon[27845]: pgmap v3820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:30:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:31 smithi096 ceph-mon[32753]: pgmap v3820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:30:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:31 smithi194 ceph-mon[33017]: pgmap v3820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:30:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5971.774941 secs 2024-04-26T13:30:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5971.774941 secs 2024-04-26T13:30:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5971.774941 secs 2024-04-26T13:30:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5974.523371 secs 2024-04-26T13:30:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:34 smithi194 ceph-mon[33017]: pgmap v3821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:30:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5974.523371 secs 2024-04-26T13:30:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:34 smithi033 ceph-mon[27845]: pgmap v3821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:30:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5974.523371 secs 2024-04-26T13:30:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:34 smithi096 ceph-mon[32753]: pgmap v3821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:30:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:35 smithi033 ceph-mon[27845]: pgmap v3822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:35 smithi096 ceph-mon[32753]: pgmap v3822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:35 smithi194 ceph-mon[33017]: pgmap v3822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:30:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:37.059+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:37.059+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:37.059+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:30:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:37 smithi033 ceph-mon[27845]: pgmap v3823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:37 smithi096 ceph-mon[32753]: pgmap v3823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:37 smithi194 ceph-mon[33017]: pgmap v3823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5976.775029 secs 2024-04-26T13:30:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5976.775029 secs 2024-04-26T13:30:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5976.775029 secs 2024-04-26T13:30:39.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5979.523468 secs 2024-04-26T13:30:39.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:39 smithi033 ceph-mon[27845]: pgmap v3824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5979.523468 secs 2024-04-26T13:30:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:39 smithi096 ceph-mon[32753]: pgmap v3824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5979.523468 secs 2024-04-26T13:30:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:39 smithi194 ceph-mon[33017]: pgmap v3824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:41 smithi033 ceph-mon[27845]: pgmap v3825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:41 smithi096 ceph-mon[32753]: pgmap v3825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:41 smithi194 ceph-mon[33017]: pgmap v3825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5981.775117 secs 2024-04-26T13:30:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5981.775117 secs 2024-04-26T13:30:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5981.775117 secs 2024-04-26T13:30:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5984.523587 secs 2024-04-26T13:30:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:44 smithi194 ceph-mon[33017]: pgmap v3826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5984.523587 secs 2024-04-26T13:30:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:44 smithi033 ceph-mon[27845]: pgmap v3826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5984.523587 secs 2024-04-26T13:30:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:44 smithi096 ceph-mon[32753]: pgmap v3826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:45 smithi033 ceph-mon[27845]: pgmap v3827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:45 smithi096 ceph-mon[32753]: pgmap v3827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:45 smithi194 ceph-mon[33017]: pgmap v3827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:47.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:47 smithi194 ceph-mon[33017]: pgmap v3828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:47 smithi033 ceph-mon[27845]: pgmap v3828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:47 smithi096 ceph-mon[32753]: pgmap v3828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5986.775177 secs 2024-04-26T13:30:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5986.775177 secs 2024-04-26T13:30:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5986.775177 secs 2024-04-26T13:30:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5989.523705 secs 2024-04-26T13:30:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:49 smithi033 ceph-mon[27845]: pgmap v3829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5989.523705 secs 2024-04-26T13:30:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:49 smithi096 ceph-mon[32753]: pgmap v3829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5989.523705 secs 2024-04-26T13:30:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:49 smithi194 ceph-mon[33017]: pgmap v3829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:51 smithi033 ceph-mon[27845]: pgmap v3830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:51 smithi096 ceph-mon[32753]: pgmap v3830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:51 smithi194 ceph-mon[33017]: pgmap v3830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:52.075+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:52.075+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:30:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:30:52.075+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:30:53.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5991.775236 secs 2024-04-26T13:30:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5991.775236 secs 2024-04-26T13:30:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5991.775236 secs 2024-04-26T13:30:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5994.523825 secs 2024-04-26T13:30:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:54 smithi194 ceph-mon[33017]: pgmap v3831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5994.523825 secs 2024-04-26T13:30:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:54 smithi033 ceph-mon[27845]: pgmap v3831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5994.523825 secs 2024-04-26T13:30:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:54 smithi096 ceph-mon[32753]: pgmap v3831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:55 smithi033 ceph-mon[27845]: pgmap v3832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:55 smithi096 ceph-mon[32753]: pgmap v3832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:55 smithi194 ceph-mon[33017]: pgmap v3832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:57 smithi033 ceph-mon[27845]: pgmap v3833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:57 smithi096 ceph-mon[32753]: pgmap v3833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:57 smithi194 ceph-mon[33017]: pgmap v3833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:30:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 5996.775305 secs 2024-04-26T13:30:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 5996.775305 secs 2024-04-26T13:30:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 5996.775305 secs 2024-04-26T13:30:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 5999.523945 secs 2024-04-26T13:30:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:30:59 smithi033 ceph-mon[27845]: pgmap v3834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 5999.523945 secs 2024-04-26T13:30:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:30:59 smithi096 ceph-mon[32753]: pgmap v3834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:30:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 5999.523945 secs 2024-04-26T13:30:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:30:59 smithi194 ceph-mon[33017]: pgmap v3834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:01 smithi033 ceph-mon[27845]: pgmap v3835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:01 smithi096 ceph-mon[32753]: pgmap v3835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:01 smithi194 ceph-mon[33017]: pgmap v3835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6001.775365 secs 2024-04-26T13:31:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6001.775365 secs 2024-04-26T13:31:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6001.775365 secs 2024-04-26T13:31:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6004.524058 secs 2024-04-26T13:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:04 smithi194 ceph-mon[33017]: pgmap v3836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6004.524058 secs 2024-04-26T13:31:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:04 smithi033 ceph-mon[27845]: pgmap v3836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6004.524058 secs 2024-04-26T13:31:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:04 smithi096 ceph-mon[32753]: pgmap v3836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:05 smithi033 ceph-mon[27845]: pgmap v3837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:05 smithi096 ceph-mon[32753]: pgmap v3837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:05 smithi194 ceph-mon[33017]: pgmap v3837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:07.090+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:07.090+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:07.090+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:31:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:07 smithi033 ceph-mon[27845]: pgmap v3838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:07 smithi096 ceph-mon[32753]: pgmap v3838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:07 smithi194 ceph-mon[33017]: pgmap v3838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6006.775424 secs 2024-04-26T13:31:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6006.775424 secs 2024-04-26T13:31:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6006.775424 secs 2024-04-26T13:31:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6009.524175 secs 2024-04-26T13:31:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:09 smithi033 ceph-mon[27845]: pgmap v3839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6009.524175 secs 2024-04-26T13:31:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:09 smithi096 ceph-mon[32753]: pgmap v3839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6009.524175 secs 2024-04-26T13:31:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:09 smithi194 ceph-mon[33017]: pgmap v3839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:11 smithi033 ceph-mon[27845]: pgmap v3840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:11 smithi096 ceph-mon[32753]: pgmap v3840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:11 smithi194 ceph-mon[33017]: pgmap v3840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6011.775480 secs 2024-04-26T13:31:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6011.775480 secs 2024-04-26T13:31:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6011.775480 secs 2024-04-26T13:31:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6014.524293 secs 2024-04-26T13:31:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:14 smithi194 ceph-mon[33017]: pgmap v3841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:31:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6014.524293 secs 2024-04-26T13:31:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:14 smithi033 ceph-mon[27845]: pgmap v3841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:31:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6014.524293 secs 2024-04-26T13:31:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:14 smithi096 ceph-mon[32753]: pgmap v3841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: pgmap v3842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:31:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:31:16.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: pgmap v3842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:31:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: pgmap v3842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:31:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:16 smithi033 ceph-mon[27845]: pgmap v3843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:31:17.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:16 smithi033 ceph-mon[27845]: pgmap v3844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:16 smithi096 ceph-mon[32753]: pgmap v3843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:31:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:16 smithi096 ceph-mon[32753]: pgmap v3844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:17.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:16 smithi194 ceph-mon[33017]: pgmap v3843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:31:17.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:16 smithi194 ceph-mon[33017]: pgmap v3844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6016.775549 secs 2024-04-26T13:31:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:18 smithi096 ceph-mon[32753]: pgmap v3845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:31:19.215 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6016.775549 secs 2024-04-26T13:31:19.215 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:18 smithi033 ceph-mon[27845]: pgmap v3845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:31:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6016.775549 secs 2024-04-26T13:31:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:18 smithi194 ceph-mon[33017]: pgmap v3845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:31:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6019.524386 secs 2024-04-26T13:31:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6019.524386 secs 2024-04-26T13:31:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6019.524386 secs 2024-04-26T13:31:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:20 smithi194 ceph-mon[33017]: pgmap v3846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:20 smithi033 ceph-mon[27845]: pgmap v3846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:20 smithi096 ceph-mon[32753]: pgmap v3846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:22.102+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:22.102+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:22.102+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:31:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:22 smithi096 ceph-mon[32753]: pgmap v3847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:22 smithi194 ceph-mon[33017]: pgmap v3847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:22 smithi033 ceph-mon[27845]: pgmap v3847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:31:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6021.775650 secs 2024-04-26T13:31:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6021.775650 secs 2024-04-26T13:31:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6021.775650 secs 2024-04-26T13:31:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6024.524528 secs 2024-04-26T13:31:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:24 smithi194 ceph-mon[33017]: pgmap v3848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:31:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6024.524528 secs 2024-04-26T13:31:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:24 smithi033 ceph-mon[27845]: pgmap v3848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:31:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6024.524528 secs 2024-04-26T13:31:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:24 smithi096 ceph-mon[32753]: pgmap v3848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:31:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:26 smithi194 ceph-mon[33017]: pgmap v3849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:31:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:26 smithi033 ceph-mon[27845]: pgmap v3849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:31:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:26 smithi096 ceph-mon[32753]: pgmap v3849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:31:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6026.775752 secs 2024-04-26T13:31:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:28 smithi194 ceph-mon[33017]: pgmap v3850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6026.775752 secs 2024-04-26T13:31:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:28 smithi033 ceph-mon[27845]: pgmap v3850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6026.775752 secs 2024-04-26T13:31:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:28 smithi096 ceph-mon[32753]: pgmap v3850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6029.524666 secs 2024-04-26T13:31:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6029.524666 secs 2024-04-26T13:31:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6029.524666 secs 2024-04-26T13:31:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:30 smithi194 ceph-mon[33017]: pgmap v3851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:30 smithi033 ceph-mon[27845]: pgmap v3851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:30 smithi096 ceph-mon[32753]: pgmap v3851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:32 smithi194 ceph-mon[33017]: pgmap v3852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:32 smithi033 ceph-mon[27845]: pgmap v3852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:32 smithi096 ceph-mon[32753]: pgmap v3852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6031.775809 secs 2024-04-26T13:31:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6031.775809 secs 2024-04-26T13:31:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6031.775809 secs 2024-04-26T13:31:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6034.524804 secs 2024-04-26T13:31:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:34 smithi194 ceph-mon[33017]: pgmap v3853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6034.524804 secs 2024-04-26T13:31:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:34 smithi033 ceph-mon[27845]: pgmap v3853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6034.524804 secs 2024-04-26T13:31:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:34 smithi096 ceph-mon[32753]: pgmap v3853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:37.117+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:37.117+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:37.117+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:31:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:37 smithi194 ceph-mon[33017]: pgmap v3854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:37 smithi033 ceph-mon[27845]: pgmap v3854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:37 smithi096 ceph-mon[32753]: pgmap v3854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6036.775894 secs 2024-04-26T13:31:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:39 smithi194 ceph-mon[33017]: pgmap v3855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6039.524920 secs 2024-04-26T13:31:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6036.775894 secs 2024-04-26T13:31:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:39 smithi033 ceph-mon[27845]: pgmap v3855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6039.524920 secs 2024-04-26T13:31:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6036.775894 secs 2024-04-26T13:31:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:39 smithi096 ceph-mon[32753]: pgmap v3855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6039.524920 secs 2024-04-26T13:31:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:41 smithi194 ceph-mon[33017]: pgmap v3856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:41 smithi033 ceph-mon[27845]: pgmap v3856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:41 smithi096 ceph-mon[32753]: pgmap v3856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:43 smithi194 ceph-mon[33017]: pgmap v3857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:43 smithi033 ceph-mon[27845]: pgmap v3857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:43 smithi096 ceph-mon[32753]: pgmap v3857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6041.775995 secs 2024-04-26T13:31:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6044.525012 secs 2024-04-26T13:31:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6041.775995 secs 2024-04-26T13:31:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6044.525012 secs 2024-04-26T13:31:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6041.775995 secs 2024-04-26T13:31:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6044.525012 secs 2024-04-26T13:31:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:45 smithi194 ceph-mon[33017]: pgmap v3858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:45 smithi033 ceph-mon[27845]: pgmap v3858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:45 smithi096 ceph-mon[32753]: pgmap v3858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:47 smithi194 ceph-mon[33017]: pgmap v3859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:47 smithi033 ceph-mon[27845]: pgmap v3859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:47 smithi096 ceph-mon[32753]: pgmap v3859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6046.776056 secs 2024-04-26T13:31:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:49 smithi194 ceph-mon[33017]: pgmap v3860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6049.525150 secs 2024-04-26T13:31:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6046.776056 secs 2024-04-26T13:31:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:49 smithi033 ceph-mon[27845]: pgmap v3860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6049.525150 secs 2024-04-26T13:31:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6046.776056 secs 2024-04-26T13:31:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:49 smithi096 ceph-mon[32753]: pgmap v3860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6049.525150 secs 2024-04-26T13:31:51.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:51 smithi033 ceph-mon[27845]: pgmap v3861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:51 smithi194 ceph-mon[33017]: pgmap v3861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:51 smithi096 ceph-mon[32753]: pgmap v3861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:52.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:52.131+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:52.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:52.132+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:31:52.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:31:52.132+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:31:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:53 smithi194 ceph-mon[33017]: pgmap v3862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:53 smithi033 ceph-mon[27845]: pgmap v3862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:53 smithi096 ceph-mon[32753]: pgmap v3862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6051.776144 secs 2024-04-26T13:31:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6054.525264 secs 2024-04-26T13:31:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6051.776144 secs 2024-04-26T13:31:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6054.525264 secs 2024-04-26T13:31:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6051.776144 secs 2024-04-26T13:31:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6054.525264 secs 2024-04-26T13:31:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:55 smithi194 ceph-mon[33017]: pgmap v3863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:55 smithi033 ceph-mon[27845]: pgmap v3863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:55 smithi096 ceph-mon[32753]: pgmap v3863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:57 smithi194 ceph-mon[33017]: pgmap v3864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:57 smithi033 ceph-mon[27845]: pgmap v3864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:57 smithi096 ceph-mon[32753]: pgmap v3864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:31:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6056.776203 secs 2024-04-26T13:31:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:59 smithi194 ceph-mon[33017]: pgmap v3865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6059.525382 secs 2024-04-26T13:31:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6056.776203 secs 2024-04-26T13:31:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:59 smithi033 ceph-mon[27845]: pgmap v3865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6059.525382 secs 2024-04-26T13:31:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6056.776203 secs 2024-04-26T13:31:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:59 smithi096 ceph-mon[32753]: pgmap v3865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:31:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6059.525382 secs 2024-04-26T13:32:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:01 smithi194 ceph-mon[33017]: pgmap v3866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:01 smithi033 ceph-mon[27845]: pgmap v3866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:01 smithi096 ceph-mon[32753]: pgmap v3866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:03 smithi194 ceph-mon[33017]: pgmap v3867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:03 smithi033 ceph-mon[27845]: pgmap v3867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:03 smithi096 ceph-mon[32753]: pgmap v3867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6061.776272 secs 2024-04-26T13:32:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6064.525480 secs 2024-04-26T13:32:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6061.776272 secs 2024-04-26T13:32:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6064.525480 secs 2024-04-26T13:32:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6061.776272 secs 2024-04-26T13:32:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6064.525480 secs 2024-04-26T13:32:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:05 smithi194 ceph-mon[33017]: pgmap v3868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:05 smithi033 ceph-mon[27845]: pgmap v3868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:05 smithi096 ceph-mon[32753]: pgmap v3868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:07.142+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:07.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:07.142+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:07.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:07.142+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:32:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:07 smithi033 ceph-mon[27845]: pgmap v3869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:07 smithi096 ceph-mon[32753]: pgmap v3869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:07 smithi194 ceph-mon[33017]: pgmap v3869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6066.776333 secs 2024-04-26T13:32:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:09 smithi033 ceph-mon[27845]: pgmap v3870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6069.525599 secs 2024-04-26T13:32:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6066.776333 secs 2024-04-26T13:32:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:09 smithi096 ceph-mon[32753]: pgmap v3870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6069.525599 secs 2024-04-26T13:32:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6066.776333 secs 2024-04-26T13:32:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:09 smithi194 ceph-mon[33017]: pgmap v3870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6069.525599 secs 2024-04-26T13:32:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:11 smithi033 ceph-mon[27845]: pgmap v3871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:11 smithi096 ceph-mon[32753]: pgmap v3871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:11 smithi194 ceph-mon[33017]: pgmap v3871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:13 smithi033 ceph-mon[27845]: pgmap v3872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:13 smithi096 ceph-mon[32753]: pgmap v3872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:13 smithi194 ceph-mon[33017]: pgmap v3872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6071.776414 secs 2024-04-26T13:32:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6074.525670 secs 2024-04-26T13:32:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6071.776414 secs 2024-04-26T13:32:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6074.525670 secs 2024-04-26T13:32:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6071.776414 secs 2024-04-26T13:32:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6074.525670 secs 2024-04-26T13:32:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:15 smithi033 ceph-mon[27845]: pgmap v3873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:15 smithi096 ceph-mon[32753]: pgmap v3873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:15 smithi194 ceph-mon[33017]: pgmap v3873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:17.404 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:17 smithi194 ceph-mon[33017]: pgmap v3874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:32:17.437 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:17 smithi096 ceph-mon[32753]: pgmap v3874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:17.437 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:32:17.438 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:17 smithi033 ceph-mon[27845]: pgmap v3874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:17.438 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:32:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:32:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:32:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:32:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:32:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:32:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:32:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:32:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:32:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:32:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:19 smithi033 ceph-mon[27845]: pgmap v3875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6076.776501 secs 2024-04-26T13:32:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:19 smithi033 ceph-mon[27845]: pgmap v3876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6079.525787 secs 2024-04-26T13:32:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:19 smithi096 ceph-mon[32753]: pgmap v3875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6076.776501 secs 2024-04-26T13:32:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:19 smithi096 ceph-mon[32753]: pgmap v3876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6079.525787 secs 2024-04-26T13:32:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:19 smithi194 ceph-mon[33017]: pgmap v3875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6076.776501 secs 2024-04-26T13:32:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:19 smithi194 ceph-mon[33017]: pgmap v3876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6079.525787 secs 2024-04-26T13:32:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:21 smithi033 ceph-mon[27845]: pgmap v3877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:21 smithi096 ceph-mon[32753]: pgmap v3877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:21 smithi194 ceph-mon[33017]: pgmap v3877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:22.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:22.147+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:22.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:22.147+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:22.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:22.147+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:32:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:22.825+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:32:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:22.825+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T13:32:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:22.826+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:32:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:22.826+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:32:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:23 smithi033 ceph-mon[27845]: pgmap v3878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6081.776587 secs 2024-04-26T13:32:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:23 smithi096 ceph-mon[32753]: pgmap v3878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6081.776587 secs 2024-04-26T13:32:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:23 smithi194 ceph-mon[33017]: pgmap v3878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6081.776587 secs 2024-04-26T13:32:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6084.525918 secs 2024-04-26T13:32:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6084.525918 secs 2024-04-26T13:32:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6084.525918 secs 2024-04-26T13:32:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:25 smithi033 ceph-mon[27845]: pgmap v3879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:25 smithi096 ceph-mon[32753]: pgmap v3879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:25 smithi194 ceph-mon[33017]: pgmap v3879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:27 smithi096 ceph-mon[32753]: pgmap v3880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:27 smithi194 ceph-mon[33017]: pgmap v3880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:27 smithi033 ceph-mon[27845]: pgmap v3880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:32:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6086.776672 secs 2024-04-26T13:32:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6086.776672 secs 2024-04-26T13:32:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6086.776672 secs 2024-04-26T13:32:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:29 smithi194 ceph-mon[33017]: pgmap v3881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:32:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6089.526018 secs 2024-04-26T13:32:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:29 smithi033 ceph-mon[27845]: pgmap v3881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:32:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6089.526018 secs 2024-04-26T13:32:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:29 smithi096 ceph-mon[32753]: pgmap v3881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:32:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6089.526018 secs 2024-04-26T13:32:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:31 smithi194 ceph-mon[33017]: pgmap v3882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:31 smithi033 ceph-mon[27845]: pgmap v3882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:31 smithi096 ceph-mon[32753]: pgmap v3882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:33 smithi194 ceph-mon[33017]: pgmap v3883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6091.776758 secs 2024-04-26T13:32:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:33 smithi033 ceph-mon[27845]: pgmap v3883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6091.776758 secs 2024-04-26T13:32:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:33 smithi096 ceph-mon[32753]: pgmap v3883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6091.776758 secs 2024-04-26T13:32:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6094.526161 secs 2024-04-26T13:32:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6094.526161 secs 2024-04-26T13:32:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6094.526161 secs 2024-04-26T13:32:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:35 smithi194 ceph-mon[33017]: pgmap v3884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:35 smithi033 ceph-mon[27845]: pgmap v3884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:35 smithi096 ceph-mon[32753]: pgmap v3884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:37.155+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:37.155+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:37.156+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:32:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:37 smithi194 ceph-mon[33017]: pgmap v3885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:37 smithi033 ceph-mon[27845]: pgmap v3885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:37 smithi096 ceph-mon[32753]: pgmap v3885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6096.776841 secs 2024-04-26T13:32:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6096.776841 secs 2024-04-26T13:32:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6096.776841 secs 2024-04-26T13:32:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:39 smithi194 ceph-mon[33017]: pgmap v3886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6099.526277 secs 2024-04-26T13:32:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:39 smithi033 ceph-mon[27845]: pgmap v3886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6099.526277 secs 2024-04-26T13:32:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:39 smithi096 ceph-mon[32753]: pgmap v3886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6099.526277 secs 2024-04-26T13:32:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:41 smithi194 ceph-mon[33017]: pgmap v3887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:41 smithi033 ceph-mon[27845]: pgmap v3887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:41 smithi096 ceph-mon[32753]: pgmap v3887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:43.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:43 smithi194 ceph-mon[33017]: pgmap v3888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:43.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6101.776924 secs 2024-04-26T13:32:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:43 smithi033 ceph-mon[27845]: pgmap v3888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6101.776924 secs 2024-04-26T13:32:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:43 smithi096 ceph-mon[32753]: pgmap v3888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6101.776924 secs 2024-04-26T13:32:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6104.526391 secs 2024-04-26T13:32:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6104.526391 secs 2024-04-26T13:32:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6104.526391 secs 2024-04-26T13:32:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:45 smithi194 ceph-mon[33017]: pgmap v3889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:45 smithi033 ceph-mon[27845]: pgmap v3889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:45 smithi096 ceph-mon[32753]: pgmap v3889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:47 smithi194 ceph-mon[33017]: pgmap v3890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:47 smithi033 ceph-mon[27845]: pgmap v3890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:47 smithi096 ceph-mon[32753]: pgmap v3890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6106.777027 secs 2024-04-26T13:32:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6106.777027 secs 2024-04-26T13:32:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6106.777027 secs 2024-04-26T13:32:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:49 smithi194 ceph-mon[33017]: pgmap v3891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6109.526514 secs 2024-04-26T13:32:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:49 smithi033 ceph-mon[27845]: pgmap v3891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6109.526514 secs 2024-04-26T13:32:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:49 smithi096 ceph-mon[32753]: pgmap v3891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6109.526514 secs 2024-04-26T13:32:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:51 smithi194 ceph-mon[33017]: pgmap v3892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:51 smithi033 ceph-mon[27845]: pgmap v3892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:51 smithi096 ceph-mon[32753]: pgmap v3892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:52.168+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:52.168+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:32:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:32:52.168+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:32:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:53 smithi194 ceph-mon[33017]: pgmap v3893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6111.777117 secs 2024-04-26T13:32:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:53 smithi033 ceph-mon[27845]: pgmap v3893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6111.777117 secs 2024-04-26T13:32:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:53 smithi096 ceph-mon[32753]: pgmap v3893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6111.777117 secs 2024-04-26T13:32:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6114.526628 secs 2024-04-26T13:32:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6114.526628 secs 2024-04-26T13:32:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6114.526628 secs 2024-04-26T13:32:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:55 smithi194 ceph-mon[33017]: pgmap v3894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:55 smithi033 ceph-mon[27845]: pgmap v3894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:55 smithi096 ceph-mon[32753]: pgmap v3894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:57 smithi194 ceph-mon[33017]: pgmap v3895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:57 smithi033 ceph-mon[27845]: pgmap v3895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:57 smithi096 ceph-mon[32753]: pgmap v3895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:32:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6116.777200 secs 2024-04-26T13:32:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6116.777200 secs 2024-04-26T13:32:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6116.777200 secs 2024-04-26T13:32:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:59 smithi194 ceph-mon[33017]: pgmap v3896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6119.526773 secs 2024-04-26T13:32:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:59 smithi033 ceph-mon[27845]: pgmap v3896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6119.526773 secs 2024-04-26T13:32:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:59 smithi096 ceph-mon[32753]: pgmap v3896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:32:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6119.526773 secs 2024-04-26T13:33:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:01 smithi194 ceph-mon[33017]: pgmap v3897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:01 smithi033 ceph-mon[27845]: pgmap v3897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:01 smithi096 ceph-mon[32753]: pgmap v3897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:03 smithi194 ceph-mon[33017]: pgmap v3898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6121.777284 secs 2024-04-26T13:33:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:03 smithi033 ceph-mon[27845]: pgmap v3898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6121.777284 secs 2024-04-26T13:33:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:03 smithi096 ceph-mon[32753]: pgmap v3898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6121.777284 secs 2024-04-26T13:33:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6124.526892 secs 2024-04-26T13:33:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6124.526892 secs 2024-04-26T13:33:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6124.526892 secs 2024-04-26T13:33:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:05 smithi194 ceph-mon[33017]: pgmap v3899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:05 smithi033 ceph-mon[27845]: pgmap v3899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:05 smithi096 ceph-mon[32753]: pgmap v3899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:06 smithi033 ceph-mon[27845]: pgmap v3900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:06 smithi096 ceph-mon[32753]: pgmap v3900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:06 smithi194 ceph-mon[33017]: pgmap v3900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:07.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:07.168+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:07.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:07.168+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:07.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:07.168+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:33:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6126.777348 secs 2024-04-26T13:33:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:08 smithi033 ceph-mon[27845]: pgmap v3901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6126.777348 secs 2024-04-26T13:33:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:08 smithi096 ceph-mon[32753]: pgmap v3901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6126.777348 secs 2024-04-26T13:33:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:08 smithi194 ceph-mon[33017]: pgmap v3901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6129.526987 secs 2024-04-26T13:33:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6129.526987 secs 2024-04-26T13:33:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6129.526987 secs 2024-04-26T13:33:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:10 smithi033 ceph-mon[27845]: pgmap v3902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:10 smithi096 ceph-mon[32753]: pgmap v3902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:10 smithi194 ceph-mon[33017]: pgmap v3902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:12 smithi033 ceph-mon[27845]: pgmap v3903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:12 smithi096 ceph-mon[32753]: pgmap v3903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:12 smithi194 ceph-mon[33017]: pgmap v3903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6131.777434 secs 2024-04-26T13:33:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6131.777434 secs 2024-04-26T13:33:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6131.777434 secs 2024-04-26T13:33:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6134.527126 secs 2024-04-26T13:33:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:14 smithi033 ceph-mon[27845]: pgmap v3904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6134.527126 secs 2024-04-26T13:33:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:14 smithi096 ceph-mon[32753]: pgmap v3904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6134.527126 secs 2024-04-26T13:33:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:14 smithi194 ceph-mon[33017]: pgmap v3904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:16 smithi033 ceph-mon[27845]: pgmap v3905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:16 smithi096 ceph-mon[32753]: pgmap v3905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:16 smithi194 ceph-mon[33017]: pgmap v3905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6136.777520 secs 2024-04-26T13:33:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:18 smithi033 ceph-mon[27845]: pgmap v3906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:33:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6136.777520 secs 2024-04-26T13:33:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:18 smithi096 ceph-mon[32753]: pgmap v3906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:33:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6136.777520 secs 2024-04-26T13:33:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:18 smithi194 ceph-mon[33017]: pgmap v3906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:33:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6139.527196 secs 2024-04-26T13:33:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6139.527196 secs 2024-04-26T13:33:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6139.527196 secs 2024-04-26T13:33:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:20 smithi194 ceph-mon[33017]: pgmap v3907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:33:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:33:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:20 smithi194 ceph-mon[33017]: pgmap v3908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:33:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:20 smithi194 ceph-mon[33017]: pgmap v3909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:20 smithi033 ceph-mon[27845]: pgmap v3907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:33:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:33:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:20 smithi033 ceph-mon[27845]: pgmap v3908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:33:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:20 smithi033 ceph-mon[27845]: pgmap v3909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:20 smithi096 ceph-mon[32753]: pgmap v3907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:33:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:33:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:20 smithi096 ceph-mon[32753]: pgmap v3908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:33:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:20 smithi096 ceph-mon[32753]: pgmap v3909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:33:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:22.184+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:22.184+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:22.184+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:33:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:23 smithi033 ceph-mon[27845]: pgmap v3910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6141.777595 secs 2024-04-26T13:33:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:23 smithi096 ceph-mon[32753]: pgmap v3910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6141.777595 secs 2024-04-26T13:33:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:23 smithi194 ceph-mon[33017]: pgmap v3910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6141.777595 secs 2024-04-26T13:33:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6144.527309 secs 2024-04-26T13:33:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6144.527309 secs 2024-04-26T13:33:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6144.527309 secs 2024-04-26T13:33:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:25 smithi096 ceph-mon[32753]: pgmap v3911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:33:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:25 smithi194 ceph-mon[33017]: pgmap v3911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:33:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:25 smithi033 ceph-mon[27845]: pgmap v3911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:33:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:27 smithi194 ceph-mon[33017]: pgmap v3912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:27 smithi033 ceph-mon[27845]: pgmap v3912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:27 smithi096 ceph-mon[32753]: pgmap v3912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:33:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6146.777677 secs 2024-04-26T13:33:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6146.777677 secs 2024-04-26T13:33:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6146.777677 secs 2024-04-26T13:33:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6149.527425 secs 2024-04-26T13:33:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:29 smithi194 ceph-mon[33017]: pgmap v3913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:33:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6149.527425 secs 2024-04-26T13:33:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:29 smithi033 ceph-mon[27845]: pgmap v3913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:33:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6149.527425 secs 2024-04-26T13:33:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:29 smithi096 ceph-mon[32753]: pgmap v3913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:33:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:31 smithi194 ceph-mon[33017]: pgmap v3914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:33:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:31 smithi033 ceph-mon[27845]: pgmap v3914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:33:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:31 smithi096 ceph-mon[32753]: pgmap v3914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:33:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:33 smithi194 ceph-mon[33017]: pgmap v3915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6151.777763 secs 2024-04-26T13:33:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:33 smithi033 ceph-mon[27845]: pgmap v3915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6151.777763 secs 2024-04-26T13:33:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:33 smithi096 ceph-mon[32753]: pgmap v3915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6151.777763 secs 2024-04-26T13:33:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6154.527492 secs 2024-04-26T13:33:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6154.527492 secs 2024-04-26T13:33:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6154.527492 secs 2024-04-26T13:33:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:35 smithi194 ceph-mon[33017]: pgmap v3916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:35 smithi033 ceph-mon[27845]: pgmap v3916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:35 smithi096 ceph-mon[32753]: pgmap v3916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:37.185+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:37.185+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:37.185+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:33:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:37 smithi194 ceph-mon[33017]: pgmap v3917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:37 smithi033 ceph-mon[27845]: pgmap v3917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:37 smithi096 ceph-mon[32753]: pgmap v3917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6156.777848 secs 2024-04-26T13:33:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6156.777848 secs 2024-04-26T13:33:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6156.777848 secs 2024-04-26T13:33:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6159.527626 secs 2024-04-26T13:33:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:39 smithi033 ceph-mon[27845]: pgmap v3918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6159.527626 secs 2024-04-26T13:33:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:39 smithi194 ceph-mon[33017]: pgmap v3918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6159.527626 secs 2024-04-26T13:33:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:39 smithi096 ceph-mon[32753]: pgmap v3918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:41 smithi194 ceph-mon[33017]: pgmap v3919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:41 smithi033 ceph-mon[27845]: pgmap v3919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:41 smithi096 ceph-mon[32753]: pgmap v3919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:43 smithi194 ceph-mon[33017]: pgmap v3920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6161.777907 secs 2024-04-26T13:33:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:43 smithi033 ceph-mon[27845]: pgmap v3920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6161.777907 secs 2024-04-26T13:33:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:43 smithi096 ceph-mon[32753]: pgmap v3920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6161.777907 secs 2024-04-26T13:33:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6164.527742 secs 2024-04-26T13:33:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6164.527742 secs 2024-04-26T13:33:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6164.527742 secs 2024-04-26T13:33:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:45 smithi194 ceph-mon[33017]: pgmap v3921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:45 smithi033 ceph-mon[27845]: pgmap v3921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:45 smithi096 ceph-mon[32753]: pgmap v3921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:47 smithi194 ceph-mon[33017]: pgmap v3922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:47 smithi033 ceph-mon[27845]: pgmap v3922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:47 smithi096 ceph-mon[32753]: pgmap v3922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6166.777997 secs 2024-04-26T13:33:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6166.777997 secs 2024-04-26T13:33:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6166.777997 secs 2024-04-26T13:33:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6169.527874 secs 2024-04-26T13:33:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:49 smithi194 ceph-mon[33017]: pgmap v3923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6169.527874 secs 2024-04-26T13:33:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:49 smithi033 ceph-mon[27845]: pgmap v3923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6169.527874 secs 2024-04-26T13:33:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:49 smithi096 ceph-mon[32753]: pgmap v3923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:51 smithi194 ceph-mon[33017]: pgmap v3924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:51 smithi033 ceph-mon[27845]: pgmap v3924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:51 smithi096 ceph-mon[32753]: pgmap v3924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:52.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:52.191+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:52.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:52.191+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:33:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:33:52.191+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:33:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:53 smithi194 ceph-mon[33017]: pgmap v3925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6171.778084 secs 2024-04-26T13:33:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:53 smithi033 ceph-mon[27845]: pgmap v3925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6171.778084 secs 2024-04-26T13:33:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:53 smithi096 ceph-mon[32753]: pgmap v3925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6171.778084 secs 2024-04-26T13:33:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6174.527989 secs 2024-04-26T13:33:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6174.527989 secs 2024-04-26T13:33:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6174.527989 secs 2024-04-26T13:33:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:55 smithi194 ceph-mon[33017]: pgmap v3926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:55 smithi033 ceph-mon[27845]: pgmap v3926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:55 smithi096 ceph-mon[32753]: pgmap v3926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:57 smithi194 ceph-mon[33017]: pgmap v3927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:57 smithi033 ceph-mon[27845]: pgmap v3927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:57 smithi096 ceph-mon[32753]: pgmap v3927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:33:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6176.778170 secs 2024-04-26T13:33:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6176.778170 secs 2024-04-26T13:33:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6176.778170 secs 2024-04-26T13:33:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6179.528120 secs 2024-04-26T13:33:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:33:59 smithi194 ceph-mon[33017]: pgmap v3928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6179.528120 secs 2024-04-26T13:33:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:33:59 smithi033 ceph-mon[27845]: pgmap v3928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:33:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6179.528120 secs 2024-04-26T13:33:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:33:59 smithi096 ceph-mon[32753]: pgmap v3928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:01 smithi194 ceph-mon[33017]: pgmap v3929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:01 smithi033 ceph-mon[27845]: pgmap v3929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:01 smithi096 ceph-mon[32753]: pgmap v3929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:03 smithi194 ceph-mon[33017]: pgmap v3930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6181.778308 secs 2024-04-26T13:34:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:03 smithi033 ceph-mon[27845]: pgmap v3930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6181.778308 secs 2024-04-26T13:34:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:03 smithi096 ceph-mon[32753]: pgmap v3930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6181.778308 secs 2024-04-26T13:34:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6184.528186 secs 2024-04-26T13:34:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6184.528186 secs 2024-04-26T13:34:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6184.528186 secs 2024-04-26T13:34:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:05 smithi033 ceph-mon[27845]: pgmap v3931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:05 smithi096 ceph-mon[32753]: pgmap v3931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:05 smithi194 ceph-mon[33017]: pgmap v3931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:07.204+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:07.204+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:07.204+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:34:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:07 smithi033 ceph-mon[27845]: pgmap v3932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:07 smithi096 ceph-mon[32753]: pgmap v3932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:07 smithi194 ceph-mon[33017]: pgmap v3932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6186.778386 secs 2024-04-26T13:34:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:08 smithi033 ceph-mon[27845]: pgmap v3933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6186.778386 secs 2024-04-26T13:34:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:08 smithi096 ceph-mon[32753]: pgmap v3933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6186.778386 secs 2024-04-26T13:34:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:08 smithi194 ceph-mon[33017]: pgmap v3933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6189.528301 secs 2024-04-26T13:34:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6189.528301 secs 2024-04-26T13:34:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6189.528301 secs 2024-04-26T13:34:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:10 smithi033 ceph-mon[27845]: pgmap v3934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:10 smithi096 ceph-mon[32753]: pgmap v3934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:10 smithi194 ceph-mon[33017]: pgmap v3934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:13 smithi194 ceph-mon[33017]: pgmap v3935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6191.778471 secs 2024-04-26T13:34:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:13 smithi033 ceph-mon[27845]: pgmap v3935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6191.778471 secs 2024-04-26T13:34:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:13 smithi096 ceph-mon[32753]: pgmap v3935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6191.778471 secs 2024-04-26T13:34:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6194.528380 secs 2024-04-26T13:34:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6194.528380 secs 2024-04-26T13:34:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6194.528380 secs 2024-04-26T13:34:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:15 smithi194 ceph-mon[33017]: pgmap v3936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:15 smithi033 ceph-mon[27845]: pgmap v3936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:15 smithi096 ceph-mon[32753]: pgmap v3936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:17 smithi194 ceph-mon[33017]: pgmap v3937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:17 smithi033 ceph-mon[27845]: pgmap v3937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:17 smithi096 ceph-mon[32753]: pgmap v3937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:18.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6196.778555 secs 2024-04-26T13:34:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6196.778555 secs 2024-04-26T13:34:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6196.778555 secs 2024-04-26T13:34:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6199.528460 secs 2024-04-26T13:34:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:19 smithi194 ceph-mon[33017]: pgmap v3938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6199.528460 secs 2024-04-26T13:34:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:19 smithi033 ceph-mon[27845]: pgmap v3938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6199.528460 secs 2024-04-26T13:34:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:19 smithi096 ceph-mon[32753]: pgmap v3938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:21 smithi194 ceph-mon[33017]: pgmap v3939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:34:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:21 smithi033 ceph-mon[27845]: pgmap v3939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:34:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:21 smithi096 ceph-mon[32753]: pgmap v3939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:34:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:22.220+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:22.220+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:22.220+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:34:22.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:34:22.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:34:22.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:34:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:34:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:34:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:34:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:34:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:34:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:34:23.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:23 smithi096 ceph-mon[32753]: pgmap v3940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:23.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:23 smithi096 ceph-mon[32753]: pgmap v3941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:34:23.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6201.778618 secs 2024-04-26T13:34:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:23 smithi194 ceph-mon[33017]: pgmap v3940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:23 smithi194 ceph-mon[33017]: pgmap v3941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:34:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6201.778618 secs 2024-04-26T13:34:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:23 smithi033 ceph-mon[27845]: pgmap v3940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:23 smithi033 ceph-mon[27845]: pgmap v3941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:34:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6201.778618 secs 2024-04-26T13:34:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6204.528548 secs 2024-04-26T13:34:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6204.528548 secs 2024-04-26T13:34:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6204.528548 secs 2024-04-26T13:34:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:25 smithi194 ceph-mon[33017]: pgmap v3942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:34:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:25 smithi033 ceph-mon[27845]: pgmap v3942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:34:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:25 smithi096 ceph-mon[32753]: pgmap v3942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:34:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:27 smithi194 ceph-mon[33017]: pgmap v3943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:27 smithi033 ceph-mon[27845]: pgmap v3943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:27 smithi096 ceph-mon[32753]: pgmap v3943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6206.778702 secs 2024-04-26T13:34:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6206.778702 secs 2024-04-26T13:34:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6206.778702 secs 2024-04-26T13:34:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:29 smithi194 ceph-mon[33017]: pgmap v3944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6209.528675 secs 2024-04-26T13:34:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:29 smithi033 ceph-mon[27845]: pgmap v3944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6209.528675 secs 2024-04-26T13:34:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:29 smithi096 ceph-mon[32753]: pgmap v3944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6209.528675 secs 2024-04-26T13:34:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:31 smithi194 ceph-mon[33017]: pgmap v3945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:31 smithi033 ceph-mon[27845]: pgmap v3945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:31 smithi096 ceph-mon[32753]: pgmap v3945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:34:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:33 smithi194 ceph-mon[33017]: pgmap v3946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:34:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6211.778793 secs 2024-04-26T13:34:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:33 smithi033 ceph-mon[27845]: pgmap v3946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:34:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6211.778793 secs 2024-04-26T13:34:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:33 smithi096 ceph-mon[32753]: pgmap v3946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:34:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6211.778793 secs 2024-04-26T13:34:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6214.528819 secs 2024-04-26T13:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6214.528819 secs 2024-04-26T13:34:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6214.528819 secs 2024-04-26T13:34:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:35 smithi194 ceph-mon[33017]: pgmap v3947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:34:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:35 smithi033 ceph-mon[27845]: pgmap v3947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:34:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:35 smithi096 ceph-mon[32753]: pgmap v3947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:34:37.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:37.236+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:37.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:37.236+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:37.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:37.236+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:34:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:37 smithi194 ceph-mon[33017]: pgmap v3948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:34:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:37 smithi033 ceph-mon[27845]: pgmap v3948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:34:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:37 smithi096 ceph-mon[32753]: pgmap v3948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:34:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6216.778880 secs 2024-04-26T13:34:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6216.778880 secs 2024-04-26T13:34:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6216.778880 secs 2024-04-26T13:34:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:39 smithi194 ceph-mon[33017]: pgmap v3949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6219.528935 secs 2024-04-26T13:34:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:39 smithi033 ceph-mon[27845]: pgmap v3949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6219.528935 secs 2024-04-26T13:34:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:39 smithi096 ceph-mon[32753]: pgmap v3949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6219.528935 secs 2024-04-26T13:34:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:41 smithi194 ceph-mon[33017]: pgmap v3950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:41 smithi033 ceph-mon[27845]: pgmap v3950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:41 smithi096 ceph-mon[32753]: pgmap v3950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:43 smithi194 ceph-mon[33017]: pgmap v3951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6221.778989 secs 2024-04-26T13:34:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:43 smithi033 ceph-mon[27845]: pgmap v3951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6221.778989 secs 2024-04-26T13:34:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:43 smithi096 ceph-mon[32753]: pgmap v3951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6221.778989 secs 2024-04-26T13:34:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6224.529001 secs 2024-04-26T13:34:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6224.529001 secs 2024-04-26T13:34:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6224.529001 secs 2024-04-26T13:34:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:45 smithi033 ceph-mon[27845]: pgmap v3952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:45 smithi096 ceph-mon[32753]: pgmap v3952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:45 smithi194 ceph-mon[33017]: pgmap v3952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:47 smithi033 ceph-mon[27845]: pgmap v3953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:47 smithi096 ceph-mon[32753]: pgmap v3953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:47 smithi194 ceph-mon[33017]: pgmap v3953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6226.779063 secs 2024-04-26T13:34:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6226.779063 secs 2024-04-26T13:34:48.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6226.779063 secs 2024-04-26T13:34:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:49 smithi033 ceph-mon[27845]: pgmap v3954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6229.529139 secs 2024-04-26T13:34:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:49 smithi096 ceph-mon[32753]: pgmap v3954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6229.529139 secs 2024-04-26T13:34:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:49 smithi194 ceph-mon[33017]: pgmap v3954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6229.529139 secs 2024-04-26T13:34:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:50 smithi033 ceph-mon[27845]: pgmap v3955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:50 smithi096 ceph-mon[32753]: pgmap v3955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:50 smithi194 ceph-mon[33017]: pgmap v3955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:52.243+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:52.243+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:34:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:34:52.243+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:34:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:52 smithi194 ceph-mon[33017]: pgmap v3956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:52 smithi033 ceph-mon[27845]: pgmap v3956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:52 smithi096 ceph-mon[32753]: pgmap v3956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6231.779149 secs 2024-04-26T13:34:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6231.779149 secs 2024-04-26T13:34:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6231.779149 secs 2024-04-26T13:34:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6234.529263 secs 2024-04-26T13:34:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:54 smithi194 ceph-mon[33017]: pgmap v3957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6234.529263 secs 2024-04-26T13:34:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:54 smithi033 ceph-mon[27845]: pgmap v3957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6234.529263 secs 2024-04-26T13:34:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:54 smithi096 ceph-mon[32753]: pgmap v3957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:56 smithi194 ceph-mon[33017]: pgmap v3958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:56 smithi033 ceph-mon[27845]: pgmap v3958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:56 smithi096 ceph-mon[32753]: pgmap v3958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:34:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6236.779256 secs 2024-04-26T13:34:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:58 smithi194 ceph-mon[33017]: pgmap v3959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6236.779256 secs 2024-04-26T13:34:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:58 smithi033 ceph-mon[27845]: pgmap v3959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:34:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6236.779256 secs 2024-04-26T13:34:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:58 smithi096 ceph-mon[32753]: pgmap v3959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6239.529355 secs 2024-04-26T13:35:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6239.529355 secs 2024-04-26T13:35:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6239.529355 secs 2024-04-26T13:35:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:00 smithi194 ceph-mon[33017]: pgmap v3960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:00 smithi033 ceph-mon[27845]: pgmap v3960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:00 smithi096 ceph-mon[32753]: pgmap v3960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:02 smithi033 ceph-mon[27845]: pgmap v3961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:02 smithi096 ceph-mon[32753]: pgmap v3961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:02 smithi194 ceph-mon[33017]: pgmap v3961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6241.779302 secs 2024-04-26T13:35:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6241.779302 secs 2024-04-26T13:35:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6241.779302 secs 2024-04-26T13:35:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6244.529474 secs 2024-04-26T13:35:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:04 smithi033 ceph-mon[27845]: pgmap v3962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6244.529474 secs 2024-04-26T13:35:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:04 smithi096 ceph-mon[32753]: pgmap v3962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6244.529474 secs 2024-04-26T13:35:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:04 smithi194 ceph-mon[33017]: pgmap v3962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:06 smithi033 ceph-mon[27845]: pgmap v3963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:06 smithi096 ceph-mon[32753]: pgmap v3963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:07.258+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:07.258+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:07.258+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:35:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:06 smithi194 ceph-mon[33017]: pgmap v3963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6246.779389 secs 2024-04-26T13:35:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:08 smithi033 ceph-mon[27845]: pgmap v3964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6246.779389 secs 2024-04-26T13:35:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:08 smithi096 ceph-mon[32753]: pgmap v3964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6246.779389 secs 2024-04-26T13:35:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:08 smithi194 ceph-mon[33017]: pgmap v3964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6249.529608 secs 2024-04-26T13:35:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6249.529608 secs 2024-04-26T13:35:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6249.529608 secs 2024-04-26T13:35:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:10 smithi033 ceph-mon[27845]: pgmap v3965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:10 smithi096 ceph-mon[32753]: pgmap v3965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:10 smithi194 ceph-mon[33017]: pgmap v3965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:12 smithi033 ceph-mon[27845]: pgmap v3966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:12 smithi096 ceph-mon[32753]: pgmap v3966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:12 smithi194 ceph-mon[33017]: pgmap v3966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6251.779448 secs 2024-04-26T13:35:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6251.779448 secs 2024-04-26T13:35:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6251.779448 secs 2024-04-26T13:35:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6254.529728 secs 2024-04-26T13:35:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:14 smithi033 ceph-mon[27845]: pgmap v3967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6254.529728 secs 2024-04-26T13:35:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:14 smithi096 ceph-mon[32753]: pgmap v3967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6254.529728 secs 2024-04-26T13:35:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:14 smithi194 ceph-mon[33017]: pgmap v3967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:16 smithi033 ceph-mon[27845]: pgmap v3968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:16 smithi096 ceph-mon[32753]: pgmap v3968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:16 smithi194 ceph-mon[33017]: pgmap v3968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6256.779533 secs 2024-04-26T13:35:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:18 smithi033 ceph-mon[27845]: pgmap v3969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6256.779533 secs 2024-04-26T13:35:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:18 smithi096 ceph-mon[32753]: pgmap v3969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6256.779533 secs 2024-04-26T13:35:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:18 smithi194 ceph-mon[33017]: pgmap v3969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6259.529850 secs 2024-04-26T13:35:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6259.529850 secs 2024-04-26T13:35:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6259.529850 secs 2024-04-26T13:35:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:20 smithi194 ceph-mon[33017]: pgmap v3970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:20 smithi033 ceph-mon[27845]: pgmap v3970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:20 smithi096 ceph-mon[32753]: pgmap v3970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:22.273+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:22.273+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:22.273+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:35:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:22.855+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T13:35:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:22.855+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:35:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:22.855+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T13:35:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:22.855+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:35:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:22 smithi096 ceph-mon[32753]: pgmap v3971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:35:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:22 smithi194 ceph-mon[33017]: pgmap v3971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:35:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:22 smithi033 ceph-mon[27845]: pgmap v3971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:35:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6261.779760 secs 2024-04-26T13:35:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6261.779760 secs 2024-04-26T13:35:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6261.779760 secs 2024-04-26T13:35:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6264.529922 secs 2024-04-26T13:35:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:24 smithi194 ceph-mon[33017]: pgmap v3972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6264.529922 secs 2024-04-26T13:35:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:24 smithi033 ceph-mon[27845]: pgmap v3972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6264.529922 secs 2024-04-26T13:35:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:24 smithi096 ceph-mon[32753]: pgmap v3972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:26 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:35:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:26 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:35:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:26 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:35:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:26 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:35:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:26 smithi194 ceph-mon[33017]: pgmap v3973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:26 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:35:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:26 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:35:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:26 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:35:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:26 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:35:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:26 smithi033 ceph-mon[27845]: pgmap v3973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:26 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:35:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:26 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:35:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:26 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:35:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:26 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:35:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:26 smithi096 ceph-mon[32753]: pgmap v3973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:29.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6266.779859 secs 2024-04-26T13:35:29.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: pgmap v3974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:29.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: pgmap v3975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: pgmap v3976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:35:29.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6266.779859 secs 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: pgmap v3974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: pgmap v3975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: pgmap v3976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6266.779859 secs 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: pgmap v3974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: pgmap v3975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: pgmap v3976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:35:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:35:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6269.530039 secs 2024-04-26T13:35:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6269.530039 secs 2024-04-26T13:35:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6269.530039 secs 2024-04-26T13:35:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:30 smithi194 ceph-mon[33017]: pgmap v3977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:30 smithi033 ceph-mon[27845]: pgmap v3977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:30 smithi096 ceph-mon[32753]: pgmap v3977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:33 smithi033 ceph-mon[27845]: pgmap v3978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6271.779961 secs 2024-04-26T13:35:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:33 smithi096 ceph-mon[32753]: pgmap v3978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6271.779961 secs 2024-04-26T13:35:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:33 smithi194 ceph-mon[33017]: pgmap v3978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6271.779961 secs 2024-04-26T13:35:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6274.530136 secs 2024-04-26T13:35:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6274.530136 secs 2024-04-26T13:35:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6274.530136 secs 2024-04-26T13:35:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:35 smithi033 ceph-mon[27845]: pgmap v3979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:35 smithi096 ceph-mon[32753]: pgmap v3979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:35 smithi194 ceph-mon[33017]: pgmap v3979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:36 smithi033 ceph-mon[27845]: pgmap v3980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:36 smithi096 ceph-mon[32753]: pgmap v3980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:36 smithi194 ceph-mon[33017]: pgmap v3980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:35:37.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:37.289+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:37.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:37.290+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:37.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:37.290+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:35:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6276.780003 secs 2024-04-26T13:35:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6276.780003 secs 2024-04-26T13:35:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6276.780003 secs 2024-04-26T13:35:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6279.530262 secs 2024-04-26T13:35:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:39 smithi194 ceph-mon[33017]: pgmap v3981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:35:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6279.530262 secs 2024-04-26T13:35:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:39 smithi033 ceph-mon[27845]: pgmap v3981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:35:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6279.530262 secs 2024-04-26T13:35:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:39 smithi096 ceph-mon[32753]: pgmap v3981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:35:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:41 smithi033 ceph-mon[27845]: pgmap v3982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:41 smithi096 ceph-mon[32753]: pgmap v3982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:41 smithi194 ceph-mon[33017]: pgmap v3982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:43 smithi033 ceph-mon[27845]: pgmap v3983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6281.780064 secs 2024-04-26T13:35:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:43 smithi096 ceph-mon[32753]: pgmap v3983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6281.780064 secs 2024-04-26T13:35:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:43 smithi194 ceph-mon[33017]: pgmap v3983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6281.780064 secs 2024-04-26T13:35:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6284.530349 secs 2024-04-26T13:35:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6284.530349 secs 2024-04-26T13:35:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6284.530349 secs 2024-04-26T13:35:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:45 smithi033 ceph-mon[27845]: pgmap v3984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:45 smithi096 ceph-mon[32753]: pgmap v3984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:45 smithi194 ceph-mon[33017]: pgmap v3984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:46 smithi033 ceph-mon[27845]: pgmap v3985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:46 smithi096 ceph-mon[32753]: pgmap v3985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:46 smithi194 ceph-mon[33017]: pgmap v3985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6286.780147 secs 2024-04-26T13:35:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6286.780147 secs 2024-04-26T13:35:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6286.780147 secs 2024-04-26T13:35:49.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6289.530470 secs 2024-04-26T13:35:49.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:49 smithi194 ceph-mon[33017]: pgmap v3986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6289.530470 secs 2024-04-26T13:35:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:49 smithi033 ceph-mon[27845]: pgmap v3986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6289.530470 secs 2024-04-26T13:35:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:49 smithi096 ceph-mon[32753]: pgmap v3986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:51 smithi033 ceph-mon[27845]: pgmap v3987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:51 smithi096 ceph-mon[32753]: pgmap v3987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:51 smithi194 ceph-mon[33017]: pgmap v3987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:52.301+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:52.301+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:35:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:35:52.301+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:35:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:53 smithi033 ceph-mon[27845]: pgmap v3988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6291.780231 secs 2024-04-26T13:35:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:53 smithi096 ceph-mon[32753]: pgmap v3988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6291.780231 secs 2024-04-26T13:35:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:53 smithi194 ceph-mon[33017]: pgmap v3988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6291.780231 secs 2024-04-26T13:35:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6294.530584 secs 2024-04-26T13:35:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:54 smithi033 ceph-mon[27845]: pgmap v3989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6294.530584 secs 2024-04-26T13:35:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:54 smithi096 ceph-mon[32753]: pgmap v3989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6294.530584 secs 2024-04-26T13:35:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:54 smithi194 ceph-mon[33017]: pgmap v3989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:35:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:57 smithi033 ceph-mon[27845]: pgmap v3990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:57 smithi096 ceph-mon[32753]: pgmap v3990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:57 smithi194 ceph-mon[33017]: pgmap v3990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6296.780319 secs 2024-04-26T13:35:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6296.780319 secs 2024-04-26T13:35:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6296.780319 secs 2024-04-26T13:35:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6299.530726 secs 2024-04-26T13:35:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:35:59 smithi033 ceph-mon[27845]: pgmap v3991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6299.530726 secs 2024-04-26T13:35:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:35:59 smithi096 ceph-mon[32753]: pgmap v3991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:35:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6299.530726 secs 2024-04-26T13:35:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:35:59 smithi194 ceph-mon[33017]: pgmap v3991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:00.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:00 smithi033 ceph-mon[27845]: pgmap v3992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:00 smithi194 ceph-mon[33017]: pgmap v3992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:00 smithi096 ceph-mon[32753]: pgmap v3992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:03 smithi033 ceph-mon[27845]: pgmap v3993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6301.780381 secs 2024-04-26T13:36:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:03 smithi096 ceph-mon[32753]: pgmap v3993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6301.780381 secs 2024-04-26T13:36:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:03 smithi194 ceph-mon[33017]: pgmap v3993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6301.780381 secs 2024-04-26T13:36:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6304.530847 secs 2024-04-26T13:36:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6304.530847 secs 2024-04-26T13:36:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6304.530847 secs 2024-04-26T13:36:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:05 smithi033 ceph-mon[27845]: pgmap v3994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:05 smithi096 ceph-mon[32753]: pgmap v3994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:05 smithi194 ceph-mon[33017]: pgmap v3994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:06 smithi033 ceph-mon[27845]: pgmap v3995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:06 smithi096 ceph-mon[32753]: pgmap v3995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:06 smithi194 ceph-mon[33017]: pgmap v3995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:07.314+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:07.314+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:07.314+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:36:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6306.780462 secs 2024-04-26T13:36:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6306.780462 secs 2024-04-26T13:36:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6306.780462 secs 2024-04-26T13:36:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6309.530963 secs 2024-04-26T13:36:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:09 smithi194 ceph-mon[33017]: pgmap v3996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6309.530963 secs 2024-04-26T13:36:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:09 smithi033 ceph-mon[27845]: pgmap v3996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6309.530963 secs 2024-04-26T13:36:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:09 smithi096 ceph-mon[32753]: pgmap v3996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:11 smithi033 ceph-mon[27845]: pgmap v3997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:11 smithi096 ceph-mon[32753]: pgmap v3997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:11 smithi194 ceph-mon[33017]: pgmap v3997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:12 smithi033 ceph-mon[27845]: pgmap v3998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:12 smithi096 ceph-mon[32753]: pgmap v3998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:12 smithi194 ceph-mon[33017]: pgmap v3998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6311.780548 secs 2024-04-26T13:36:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6311.780548 secs 2024-04-26T13:36:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6311.780548 secs 2024-04-26T13:36:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6314.531079 secs 2024-04-26T13:36:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:14 smithi033 ceph-mon[27845]: pgmap v3999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6314.531079 secs 2024-04-26T13:36:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:14 smithi096 ceph-mon[32753]: pgmap v3999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6314.531079 secs 2024-04-26T13:36:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:14 smithi194 ceph-mon[33017]: pgmap v3999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:16 smithi194 ceph-mon[33017]: pgmap v4000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:16 smithi033 ceph-mon[27845]: pgmap v4000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:16 smithi096 ceph-mon[32753]: pgmap v4000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6316.780634 secs 2024-04-26T13:36:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6316.780634 secs 2024-04-26T13:36:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6316.780634 secs 2024-04-26T13:36:19.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6319.531205 secs 2024-04-26T13:36:19.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:19 smithi194 ceph-mon[33017]: pgmap v4001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6319.531205 secs 2024-04-26T13:36:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:19 smithi033 ceph-mon[27845]: pgmap v4001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6319.531205 secs 2024-04-26T13:36:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:19 smithi096 ceph-mon[32753]: pgmap v4001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:20 smithi194 ceph-mon[33017]: pgmap v4002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:20 smithi033 ceph-mon[27845]: pgmap v4002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:20 smithi096 ceph-mon[32753]: pgmap v4002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:22.329+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:22.329+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:22.330+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:36:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:22 smithi194 ceph-mon[33017]: pgmap v4003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:22 smithi033 ceph-mon[27845]: pgmap v4003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:22 smithi096 ceph-mon[32753]: pgmap v4003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:23.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6321.780717 secs 2024-04-26T13:36:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6321.780717 secs 2024-04-26T13:36:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6321.780717 secs 2024-04-26T13:36:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6324.531340 secs 2024-04-26T13:36:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:24 smithi194 ceph-mon[33017]: pgmap v4004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6324.531340 secs 2024-04-26T13:36:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:24 smithi033 ceph-mon[27845]: pgmap v4004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6324.531340 secs 2024-04-26T13:36:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:24 smithi096 ceph-mon[32753]: pgmap v4004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:26 smithi194 ceph-mon[33017]: pgmap v4005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:26 smithi033 ceph-mon[27845]: pgmap v4005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:26 smithi096 ceph-mon[32753]: pgmap v4005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6326.780798 secs 2024-04-26T13:36:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6326.780798 secs 2024-04-26T13:36:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6326.780798 secs 2024-04-26T13:36:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6329.531461 secs 2024-04-26T13:36:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:29 smithi194 ceph-mon[33017]: pgmap v4006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:36:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6329.531461 secs 2024-04-26T13:36:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:29 smithi033 ceph-mon[27845]: pgmap v4006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:36:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6329.531461 secs 2024-04-26T13:36:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:29 smithi096 ceph-mon[32753]: pgmap v4006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:36:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:36:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:36:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:36:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:36:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:36:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:36:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:36:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:36:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:36:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:31 smithi194 ceph-mon[33017]: pgmap v4007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:36:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:31 smithi194 ceph-mon[33017]: pgmap v4008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:31 smithi033 ceph-mon[27845]: pgmap v4007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:36:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:31 smithi033 ceph-mon[27845]: pgmap v4008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:31 smithi096 ceph-mon[32753]: pgmap v4007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:36:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:31 smithi096 ceph-mon[32753]: pgmap v4008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:33 smithi194 ceph-mon[33017]: pgmap v4009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6331.780879 secs 2024-04-26T13:36:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:33 smithi033 ceph-mon[27845]: pgmap v4009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6331.780879 secs 2024-04-26T13:36:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:33 smithi096 ceph-mon[32753]: pgmap v4009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6331.780879 secs 2024-04-26T13:36:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6334.531536 secs 2024-04-26T13:36:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6334.531536 secs 2024-04-26T13:36:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6334.531536 secs 2024-04-26T13:36:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:35 smithi194 ceph-mon[33017]: pgmap v4010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:36:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:35 smithi033 ceph-mon[27845]: pgmap v4010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:36:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:35 smithi096 ceph-mon[32753]: pgmap v4010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:36:37.346 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:37.346+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:37.346 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:37.346+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:37.346 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:37.346+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:36:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:37 smithi194 ceph-mon[33017]: pgmap v4011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:37 smithi033 ceph-mon[27845]: pgmap v4011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:37 smithi096 ceph-mon[32753]: pgmap v4011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6336.780986 secs 2024-04-26T13:36:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6336.780986 secs 2024-04-26T13:36:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6336.780986 secs 2024-04-26T13:36:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:39 smithi194 ceph-mon[33017]: pgmap v4012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6339.531686 secs 2024-04-26T13:36:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:39 smithi033 ceph-mon[27845]: pgmap v4012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6339.531686 secs 2024-04-26T13:36:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:39 smithi096 ceph-mon[32753]: pgmap v4012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:36:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6339.531686 secs 2024-04-26T13:36:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:41 smithi194 ceph-mon[33017]: pgmap v4013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:36:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:41 smithi033 ceph-mon[27845]: pgmap v4013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:36:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:41 smithi096 ceph-mon[32753]: pgmap v4013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:36:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:43 smithi194 ceph-mon[33017]: pgmap v4014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:43.641 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6341.781005 secs 2024-04-26T13:36:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:43 smithi033 ceph-mon[27845]: pgmap v4014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6341.781005 secs 2024-04-26T13:36:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:43 smithi096 ceph-mon[32753]: pgmap v4014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6341.781005 secs 2024-04-26T13:36:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6344.531787 secs 2024-04-26T13:36:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6344.531787 secs 2024-04-26T13:36:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6344.531787 secs 2024-04-26T13:36:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:45 smithi194 ceph-mon[33017]: pgmap v4015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:45 smithi033 ceph-mon[27845]: pgmap v4015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:45 smithi096 ceph-mon[32753]: pgmap v4015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:47 smithi194 ceph-mon[33017]: pgmap v4016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:47 smithi033 ceph-mon[27845]: pgmap v4016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:47 smithi096 ceph-mon[32753]: pgmap v4016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6346.781091 secs 2024-04-26T13:36:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6346.781091 secs 2024-04-26T13:36:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6346.781091 secs 2024-04-26T13:36:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:49 smithi194 ceph-mon[33017]: pgmap v4017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6349.531941 secs 2024-04-26T13:36:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:49 smithi033 ceph-mon[27845]: pgmap v4017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6349.531941 secs 2024-04-26T13:36:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:49 smithi096 ceph-mon[32753]: pgmap v4017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6349.531941 secs 2024-04-26T13:36:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:51 smithi194 ceph-mon[33017]: pgmap v4018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:51 smithi033 ceph-mon[27845]: pgmap v4018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:51 smithi096 ceph-mon[32753]: pgmap v4018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:52.356+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:52.356+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:36:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:52.356+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:36:52.871 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:52.872+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:36:52.871 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:52.872+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T13:36:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:52.872+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:36:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:36:52.872+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:36:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:53 smithi194 ceph-mon[33017]: pgmap v4019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6351.781177 secs 2024-04-26T13:36:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:53 smithi033 ceph-mon[27845]: pgmap v4019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6351.781177 secs 2024-04-26T13:36:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:53 smithi096 ceph-mon[32753]: pgmap v4019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6351.781177 secs 2024-04-26T13:36:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6354.532078 secs 2024-04-26T13:36:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6354.532078 secs 2024-04-26T13:36:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6354.532078 secs 2024-04-26T13:36:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:55 smithi033 ceph-mon[27845]: pgmap v4020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:55 smithi096 ceph-mon[32753]: pgmap v4020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:55 smithi194 ceph-mon[33017]: pgmap v4020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:57 smithi033 ceph-mon[27845]: pgmap v4021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:57 smithi096 ceph-mon[32753]: pgmap v4021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:57 smithi194 ceph-mon[33017]: pgmap v4021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:36:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6356.781241 secs 2024-04-26T13:36:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6356.781241 secs 2024-04-26T13:36:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6356.781241 secs 2024-04-26T13:36:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:59 smithi033 ceph-mon[27845]: pgmap v4022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6359.532193 secs 2024-04-26T13:36:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:59 smithi096 ceph-mon[32753]: pgmap v4022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6359.532193 secs 2024-04-26T13:36:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:59 smithi194 ceph-mon[33017]: pgmap v4022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:36:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6359.532193 secs 2024-04-26T13:37:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:01 smithi033 ceph-mon[27845]: pgmap v4023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:01 smithi096 ceph-mon[32753]: pgmap v4023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:01 smithi194 ceph-mon[33017]: pgmap v4023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:03 smithi033 ceph-mon[27845]: pgmap v4024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6361.781305 secs 2024-04-26T13:37:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:03 smithi096 ceph-mon[32753]: pgmap v4024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6361.781305 secs 2024-04-26T13:37:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:03 smithi194 ceph-mon[33017]: pgmap v4024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6361.781305 secs 2024-04-26T13:37:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6364.532286 secs 2024-04-26T13:37:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6364.532286 secs 2024-04-26T13:37:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6364.532286 secs 2024-04-26T13:37:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:05 smithi033 ceph-mon[27845]: pgmap v4025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:05 smithi096 ceph-mon[32753]: pgmap v4025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:05 smithi194 ceph-mon[33017]: pgmap v4025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:06 smithi033 ceph-mon[27845]: pgmap v4026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:06 smithi096 ceph-mon[32753]: pgmap v4026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:06 smithi194 ceph-mon[33017]: pgmap v4026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:07.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:07.371+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:07.371+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:37:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6366.781364 secs 2024-04-26T13:37:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:09 smithi033 ceph-mon[27845]: pgmap v4027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6369.532423 secs 2024-04-26T13:37:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6366.781364 secs 2024-04-26T13:37:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:09 smithi096 ceph-mon[32753]: pgmap v4027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6369.532423 secs 2024-04-26T13:37:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6366.781364 secs 2024-04-26T13:37:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:09 smithi194 ceph-mon[33017]: pgmap v4027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6369.532423 secs 2024-04-26T13:37:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:11 smithi033 ceph-mon[27845]: pgmap v4028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:11 smithi096 ceph-mon[32753]: pgmap v4028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:11 smithi194 ceph-mon[33017]: pgmap v4028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:13 smithi033 ceph-mon[27845]: pgmap v4029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:13 smithi096 ceph-mon[32753]: pgmap v4029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:13 smithi194 ceph-mon[33017]: pgmap v4029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6371.781444 secs 2024-04-26T13:37:14.482 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6374.532486 secs 2024-04-26T13:37:14.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6371.781444 secs 2024-04-26T13:37:14.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6374.532486 secs 2024-04-26T13:37:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6371.781444 secs 2024-04-26T13:37:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6374.532486 secs 2024-04-26T13:37:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:15 smithi033 ceph-mon[27845]: pgmap v4030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:15 smithi096 ceph-mon[32753]: pgmap v4030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:15 smithi194 ceph-mon[33017]: pgmap v4030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:17 smithi033 ceph-mon[27845]: pgmap v4031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:17 smithi096 ceph-mon[32753]: pgmap v4031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:17 smithi194 ceph-mon[33017]: pgmap v4031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6376.781529 secs 2024-04-26T13:37:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:19 smithi033 ceph-mon[27845]: pgmap v4032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6379.532621 secs 2024-04-26T13:37:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6376.781529 secs 2024-04-26T13:37:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:19 smithi096 ceph-mon[32753]: pgmap v4032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6379.532621 secs 2024-04-26T13:37:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6376.781529 secs 2024-04-26T13:37:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:19 smithi194 ceph-mon[33017]: pgmap v4032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6379.532621 secs 2024-04-26T13:37:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:21 smithi033 ceph-mon[27845]: pgmap v4033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:21 smithi096 ceph-mon[32753]: pgmap v4033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:21 smithi194 ceph-mon[33017]: pgmap v4033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:22.386+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:22.386+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:22.386+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:37:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:23 smithi096 ceph-mon[32753]: pgmap v4034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:23 smithi194 ceph-mon[33017]: pgmap v4034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:23 smithi033 ceph-mon[27845]: pgmap v4034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6381.781612 secs 2024-04-26T13:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6384.532739 secs 2024-04-26T13:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6381.781612 secs 2024-04-26T13:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6384.532739 secs 2024-04-26T13:37:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6381.781612 secs 2024-04-26T13:37:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6384.532739 secs 2024-04-26T13:37:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:25 smithi194 ceph-mon[33017]: pgmap v4035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:25 smithi033 ceph-mon[27845]: pgmap v4035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:25 smithi096 ceph-mon[32753]: pgmap v4035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:27 smithi194 ceph-mon[33017]: pgmap v4036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:27 smithi033 ceph-mon[27845]: pgmap v4036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:27 smithi096 ceph-mon[32753]: pgmap v4036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6386.781697 secs 2024-04-26T13:37:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6386.781697 secs 2024-04-26T13:37:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6386.781697 secs 2024-04-26T13:37:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:29 smithi194 ceph-mon[33017]: pgmap v4037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6389.532858 secs 2024-04-26T13:37:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:29 smithi033 ceph-mon[27845]: pgmap v4037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6389.532858 secs 2024-04-26T13:37:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:29 smithi096 ceph-mon[32753]: pgmap v4037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6389.532858 secs 2024-04-26T13:37:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:31 smithi194 ceph-mon[33017]: pgmap v4038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:37:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:31 smithi033 ceph-mon[27845]: pgmap v4038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:37:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:31 smithi096 ceph-mon[32753]: pgmap v4038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:37:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:37:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:37:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:37:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:37:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:37:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:37:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:37:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:37:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:37:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:33 smithi194 ceph-mon[33017]: pgmap v4039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:33 smithi194 ceph-mon[33017]: pgmap v4040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6391.781800 secs 2024-04-26T13:37:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:33 smithi033 ceph-mon[27845]: pgmap v4039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:33 smithi033 ceph-mon[27845]: pgmap v4040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6391.781800 secs 2024-04-26T13:37:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:33 smithi096 ceph-mon[32753]: pgmap v4039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:33 smithi096 ceph-mon[32753]: pgmap v4040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6391.781800 secs 2024-04-26T13:37:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6394.532921 secs 2024-04-26T13:37:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6394.532921 secs 2024-04-26T13:37:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6394.532921 secs 2024-04-26T13:37:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:35 smithi194 ceph-mon[33017]: pgmap v4041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:35 smithi033 ceph-mon[27845]: pgmap v4041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:35 smithi096 ceph-mon[32753]: pgmap v4041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:37.389+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:37.389+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:37.389+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:37:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:37 smithi194 ceph-mon[33017]: pgmap v4042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:37 smithi033 ceph-mon[27845]: pgmap v4042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:37 smithi096 ceph-mon[32753]: pgmap v4042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6396.781882 secs 2024-04-26T13:37:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6396.781882 secs 2024-04-26T13:37:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6396.781882 secs 2024-04-26T13:37:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:39 smithi194 ceph-mon[33017]: pgmap v4043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6399.533032 secs 2024-04-26T13:37:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:39 smithi033 ceph-mon[27845]: pgmap v4043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6399.533032 secs 2024-04-26T13:37:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:39 smithi096 ceph-mon[32753]: pgmap v4043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6399.533032 secs 2024-04-26T13:37:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:41 smithi194 ceph-mon[33017]: pgmap v4044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:41 smithi033 ceph-mon[27845]: pgmap v4044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:41 smithi096 ceph-mon[32753]: pgmap v4044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:37:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:43 smithi194 ceph-mon[33017]: pgmap v4045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:37:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6401.781973 secs 2024-04-26T13:37:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:43 smithi033 ceph-mon[27845]: pgmap v4045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:37:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6401.781973 secs 2024-04-26T13:37:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:43 smithi096 ceph-mon[32753]: pgmap v4045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:37:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6401.781973 secs 2024-04-26T13:37:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6404.533179 secs 2024-04-26T13:37:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6404.533179 secs 2024-04-26T13:37:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6404.533179 secs 2024-04-26T13:37:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:45 smithi194 ceph-mon[33017]: pgmap v4046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:45 smithi033 ceph-mon[27845]: pgmap v4046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:45 smithi096 ceph-mon[32753]: pgmap v4046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:47 smithi194 ceph-mon[33017]: pgmap v4047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:47 smithi033 ceph-mon[27845]: pgmap v4047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:47 smithi096 ceph-mon[32753]: pgmap v4047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6406.782052 secs 2024-04-26T13:37:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6406.782052 secs 2024-04-26T13:37:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6406.782052 secs 2024-04-26T13:37:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:49 smithi194 ceph-mon[33017]: pgmap v4048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6409.533293 secs 2024-04-26T13:37:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:49 smithi033 ceph-mon[27845]: pgmap v4048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6409.533293 secs 2024-04-26T13:37:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:49 smithi096 ceph-mon[32753]: pgmap v4048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6409.533293 secs 2024-04-26T13:37:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:51 smithi194 ceph-mon[33017]: pgmap v4049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:51 smithi033 ceph-mon[27845]: pgmap v4049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:51 smithi096 ceph-mon[32753]: pgmap v4049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:52.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:52.398+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:52.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:52.398+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:37:52.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:37:52.398+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:37:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:53 smithi194 ceph-mon[33017]: pgmap v4050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6411.782137 secs 2024-04-26T13:37:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:53 smithi033 ceph-mon[27845]: pgmap v4050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6411.782137 secs 2024-04-26T13:37:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:53 smithi096 ceph-mon[32753]: pgmap v4050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6411.782137 secs 2024-04-26T13:37:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6414.533425 secs 2024-04-26T13:37:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6414.533425 secs 2024-04-26T13:37:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6414.533425 secs 2024-04-26T13:37:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:55 smithi194 ceph-mon[33017]: pgmap v4051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:55 smithi033 ceph-mon[27845]: pgmap v4051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:55 smithi096 ceph-mon[32753]: pgmap v4051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:57 smithi194 ceph-mon[33017]: pgmap v4052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:57 smithi033 ceph-mon[27845]: pgmap v4052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:57 smithi096 ceph-mon[32753]: pgmap v4052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:37:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6416.782226 secs 2024-04-26T13:37:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6416.782226 secs 2024-04-26T13:37:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6416.782226 secs 2024-04-26T13:37:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:59 smithi033 ceph-mon[27845]: pgmap v4053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6419.533532 secs 2024-04-26T13:37:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:59 smithi096 ceph-mon[32753]: pgmap v4053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6419.533532 secs 2024-04-26T13:37:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:59 smithi194 ceph-mon[33017]: pgmap v4053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:37:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6419.533532 secs 2024-04-26T13:38:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:01 smithi033 ceph-mon[27845]: pgmap v4054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:01 smithi096 ceph-mon[32753]: pgmap v4054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:01 smithi194 ceph-mon[33017]: pgmap v4054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:03 smithi033 ceph-mon[27845]: pgmap v4055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6421.782318 secs 2024-04-26T13:38:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:03 smithi096 ceph-mon[32753]: pgmap v4055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6421.782318 secs 2024-04-26T13:38:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:03 smithi194 ceph-mon[33017]: pgmap v4055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6421.782318 secs 2024-04-26T13:38:04.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6424.533679 secs 2024-04-26T13:38:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6424.533679 secs 2024-04-26T13:38:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6424.533679 secs 2024-04-26T13:38:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:05 smithi033 ceph-mon[27845]: pgmap v4056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:05 smithi096 ceph-mon[32753]: pgmap v4056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:05 smithi194 ceph-mon[33017]: pgmap v4056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:07.410+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:07.410+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:07.410+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:38:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:07 smithi033 ceph-mon[27845]: pgmap v4057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:07 smithi096 ceph-mon[32753]: pgmap v4057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:07 smithi194 ceph-mon[33017]: pgmap v4057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6426.782381 secs 2024-04-26T13:38:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6426.782381 secs 2024-04-26T13:38:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6426.782381 secs 2024-04-26T13:38:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:09 smithi033 ceph-mon[27845]: pgmap v4058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6429.533807 secs 2024-04-26T13:38:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:09 smithi096 ceph-mon[32753]: pgmap v4058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6429.533807 secs 2024-04-26T13:38:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:09 smithi194 ceph-mon[33017]: pgmap v4058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6429.533807 secs 2024-04-26T13:38:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:11 smithi033 ceph-mon[27845]: pgmap v4059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:11 smithi096 ceph-mon[32753]: pgmap v4059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:11 smithi194 ceph-mon[33017]: pgmap v4059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:13 smithi033 ceph-mon[27845]: pgmap v4060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6431.782441 secs 2024-04-26T13:38:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:13 smithi096 ceph-mon[32753]: pgmap v4060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6431.782441 secs 2024-04-26T13:38:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:13 smithi194 ceph-mon[33017]: pgmap v4060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6431.782441 secs 2024-04-26T13:38:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6434.533949 secs 2024-04-26T13:38:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6434.533949 secs 2024-04-26T13:38:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6434.533949 secs 2024-04-26T13:38:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:15 smithi033 ceph-mon[27845]: pgmap v4061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:15 smithi096 ceph-mon[32753]: pgmap v4061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:15 smithi194 ceph-mon[33017]: pgmap v4061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:16 smithi096 ceph-mon[32753]: pgmap v4062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:16 smithi194 ceph-mon[33017]: pgmap v4062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:16.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:16 smithi033 ceph-mon[27845]: pgmap v4062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6436.782525 secs 2024-04-26T13:38:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:18 smithi194 ceph-mon[33017]: pgmap v4063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6436.782525 secs 2024-04-26T13:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:18 smithi033 ceph-mon[27845]: pgmap v4063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6436.782525 secs 2024-04-26T13:38:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:18 smithi096 ceph-mon[32753]: pgmap v4063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:20.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6439.534072 secs 2024-04-26T13:38:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6439.534072 secs 2024-04-26T13:38:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6439.534072 secs 2024-04-26T13:38:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:20 smithi194 ceph-mon[33017]: pgmap v4064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:20 smithi033 ceph-mon[27845]: pgmap v4064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:20 smithi096 ceph-mon[32753]: pgmap v4064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:22.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:22.426+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:22.426+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:22.426+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:38:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:22.886+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T13:38:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:22.886+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:38:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:22.886+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T13:38:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:22.886+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:38:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:22 smithi194 ceph-mon[33017]: pgmap v4065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:22 smithi033 ceph-mon[27845]: pgmap v4065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:22 smithi096 ceph-mon[32753]: pgmap v4065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6441.782610 secs 2024-04-26T13:38:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6441.782610 secs 2024-04-26T13:38:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6441.782610 secs 2024-04-26T13:38:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6444.534218 secs 2024-04-26T13:38:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:24 smithi194 ceph-mon[33017]: pgmap v4066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6444.534218 secs 2024-04-26T13:38:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:24 smithi033 ceph-mon[27845]: pgmap v4066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6444.534218 secs 2024-04-26T13:38:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:24 smithi096 ceph-mon[32753]: pgmap v4066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:26 smithi194 ceph-mon[33017]: pgmap v4067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:26 smithi033 ceph-mon[27845]: pgmap v4067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:26 smithi096 ceph-mon[32753]: pgmap v4067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6446.782694 secs 2024-04-26T13:38:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:28 smithi194 ceph-mon[33017]: pgmap v4068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6446.782694 secs 2024-04-26T13:38:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:28 smithi033 ceph-mon[27845]: pgmap v4068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6446.782694 secs 2024-04-26T13:38:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:28 smithi096 ceph-mon[32753]: pgmap v4068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6449.534319 secs 2024-04-26T13:38:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6449.534319 secs 2024-04-26T13:38:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6449.534319 secs 2024-04-26T13:38:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:30 smithi033 ceph-mon[27845]: pgmap v4069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:30 smithi096 ceph-mon[32753]: pgmap v4069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:30 smithi194 ceph-mon[33017]: pgmap v4069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:32 smithi033 ceph-mon[27845]: pgmap v4070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:38:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:32 smithi096 ceph-mon[32753]: pgmap v4070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:38:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:32 smithi194 ceph-mon[33017]: pgmap v4070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:38:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6451.782789 secs 2024-04-26T13:38:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:38:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:38:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:33 smithi033 ceph-mon[27845]: pgmap v4071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:33 smithi033 ceph-mon[27845]: pgmap v4072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:38:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6451.782789 secs 2024-04-26T13:38:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:38:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:38:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:33 smithi096 ceph-mon[32753]: pgmap v4071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:33 smithi096 ceph-mon[32753]: pgmap v4072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6451.782789 secs 2024-04-26T13:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:33 smithi194 ceph-mon[33017]: pgmap v4071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:33 smithi194 ceph-mon[33017]: pgmap v4072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:38:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6454.534436 secs 2024-04-26T13:38:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6454.534436 secs 2024-04-26T13:38:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6454.534436 secs 2024-04-26T13:38:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:35 smithi033 ceph-mon[27845]: pgmap v4073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:35 smithi096 ceph-mon[32753]: pgmap v4073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:35 smithi194 ceph-mon[33017]: pgmap v4073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:37.440+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:37.440+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:37.440+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:38:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:37 smithi194 ceph-mon[33017]: pgmap v4074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:37 smithi033 ceph-mon[27845]: pgmap v4074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:37 smithi096 ceph-mon[32753]: pgmap v4074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6456.782877 secs 2024-04-26T13:38:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6456.782877 secs 2024-04-26T13:38:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6456.782877 secs 2024-04-26T13:38:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6459.534523 secs 2024-04-26T13:38:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:39 smithi194 ceph-mon[33017]: pgmap v4075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6459.534523 secs 2024-04-26T13:38:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:39 smithi033 ceph-mon[27845]: pgmap v4075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6459.534523 secs 2024-04-26T13:38:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:39 smithi096 ceph-mon[32753]: pgmap v4075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:41.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:41 smithi033 ceph-mon[27845]: pgmap v4076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:41 smithi194 ceph-mon[33017]: pgmap v4076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:41 smithi096 ceph-mon[32753]: pgmap v4076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:38:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6461.782954 secs 2024-04-26T13:38:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6461.782954 secs 2024-04-26T13:38:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6461.782954 secs 2024-04-26T13:38:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6464.534664 secs 2024-04-26T13:38:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:44 smithi194 ceph-mon[33017]: pgmap v4077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:38:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6464.534664 secs 2024-04-26T13:38:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:44 smithi033 ceph-mon[27845]: pgmap v4077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:38:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6464.534664 secs 2024-04-26T13:38:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:44 smithi096 ceph-mon[32753]: pgmap v4077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:38:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:45 smithi194 ceph-mon[33017]: pgmap v4078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:45 smithi033 ceph-mon[27845]: pgmap v4078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:45 smithi096 ceph-mon[32753]: pgmap v4078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:47 smithi194 ceph-mon[33017]: pgmap v4079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:47 smithi033 ceph-mon[27845]: pgmap v4079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:47 smithi096 ceph-mon[32753]: pgmap v4079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6466.783044 secs 2024-04-26T13:38:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6466.783044 secs 2024-04-26T13:38:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6466.783044 secs 2024-04-26T13:38:49.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6469.534793 secs 2024-04-26T13:38:49.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:49 smithi194 ceph-mon[33017]: pgmap v4080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6469.534793 secs 2024-04-26T13:38:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:49 smithi033 ceph-mon[27845]: pgmap v4080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6469.534793 secs 2024-04-26T13:38:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:49 smithi096 ceph-mon[32753]: pgmap v4080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:38:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:51 smithi033 ceph-mon[27845]: pgmap v4081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:51 smithi194 ceph-mon[33017]: pgmap v4081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:51 smithi096 ceph-mon[32753]: pgmap v4081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:52.450+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:52.450+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:38:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:38:52.450+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:38:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6471.783105 secs 2024-04-26T13:38:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6471.783105 secs 2024-04-26T13:38:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6471.783105 secs 2024-04-26T13:38:54.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6474.534917 secs 2024-04-26T13:38:54.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:54 smithi096 ceph-mon[32753]: pgmap v4082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6474.534917 secs 2024-04-26T13:38:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:54 smithi194 ceph-mon[33017]: pgmap v4082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6474.534917 secs 2024-04-26T13:38:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:54 smithi033 ceph-mon[27845]: pgmap v4082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:38:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:55 smithi194 ceph-mon[33017]: pgmap v4083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:38:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:55 smithi033 ceph-mon[27845]: pgmap v4083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:38:55.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:55 smithi096 ceph-mon[32753]: pgmap v4083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:38:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:57 smithi194 ceph-mon[33017]: pgmap v4084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:38:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:57 smithi033 ceph-mon[27845]: pgmap v4084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:38:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:57 smithi096 ceph-mon[32753]: pgmap v4084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:38:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6476.783198 secs 2024-04-26T13:38:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6476.783198 secs 2024-04-26T13:38:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6476.783198 secs 2024-04-26T13:38:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6479.535059 secs 2024-04-26T13:38:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:38:59 smithi194 ceph-mon[33017]: pgmap v4085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:38:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6479.535059 secs 2024-04-26T13:38:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:38:59 smithi033 ceph-mon[27845]: pgmap v4085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:38:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6479.535059 secs 2024-04-26T13:38:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:38:59 smithi096 ceph-mon[32753]: pgmap v4085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:39:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:01 smithi194 ceph-mon[33017]: pgmap v4086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:39:01.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:01 smithi096 ceph-mon[32753]: pgmap v4086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:39:01.934 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:01 smithi033 ceph-mon[27845]: pgmap v4086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:39:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6481.783259 secs 2024-04-26T13:39:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6481.783259 secs 2024-04-26T13:39:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6481.783259 secs 2024-04-26T13:39:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6484.535159 secs 2024-04-26T13:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:04 smithi194 ceph-mon[33017]: pgmap v4087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:39:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6484.535159 secs 2024-04-26T13:39:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:04 smithi033 ceph-mon[27845]: pgmap v4087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:39:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6484.535159 secs 2024-04-26T13:39:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:04 smithi096 ceph-mon[32753]: pgmap v4087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:39:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:05 smithi194 ceph-mon[33017]: pgmap v4088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:39:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:05 smithi033 ceph-mon[27845]: pgmap v4088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:39:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:05 smithi096 ceph-mon[32753]: pgmap v4088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:07.458+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:07.458+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:07.458+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:07 smithi194 ceph-mon[33017]: pgmap v4089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:07 smithi033 ceph-mon[27845]: pgmap v4089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:07 smithi096 ceph-mon[32753]: pgmap v4089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6486.783341 secs 2024-04-26T13:39:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6486.783341 secs 2024-04-26T13:39:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6486.783341 secs 2024-04-26T13:39:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6489.535284 secs 2024-04-26T13:39:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:09 smithi194 ceph-mon[33017]: pgmap v4090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6489.535284 secs 2024-04-26T13:39:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:09 smithi033 ceph-mon[27845]: pgmap v4090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6489.535284 secs 2024-04-26T13:39:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:09 smithi096 ceph-mon[32753]: pgmap v4090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:11.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:11 smithi033 ceph-mon[27845]: pgmap v4091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:11 smithi194 ceph-mon[33017]: pgmap v4091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:11 smithi096 ceph-mon[32753]: pgmap v4091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6491.783398 secs 2024-04-26T13:39:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6491.783398 secs 2024-04-26T13:39:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6491.783398 secs 2024-04-26T13:39:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6494.535410 secs 2024-04-26T13:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:14 smithi194 ceph-mon[33017]: pgmap v4092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6494.535410 secs 2024-04-26T13:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:14 smithi033 ceph-mon[27845]: pgmap v4092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6494.535410 secs 2024-04-26T13:39:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:14 smithi096 ceph-mon[32753]: pgmap v4092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:15 smithi194 ceph-mon[33017]: pgmap v4093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:15 smithi033 ceph-mon[27845]: pgmap v4093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:15 smithi096 ceph-mon[32753]: pgmap v4093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:17 smithi194 ceph-mon[33017]: pgmap v4094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:17 smithi033 ceph-mon[27845]: pgmap v4094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:17 smithi096 ceph-mon[32753]: pgmap v4094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6496.783475 secs 2024-04-26T13:39:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6496.783475 secs 2024-04-26T13:39:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6496.783475 secs 2024-04-26T13:39:19.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6499.535494 secs 2024-04-26T13:39:19.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:19 smithi194 ceph-mon[33017]: pgmap v4095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6499.535494 secs 2024-04-26T13:39:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:19 smithi033 ceph-mon[27845]: pgmap v4095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6499.535494 secs 2024-04-26T13:39:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:19 smithi096 ceph-mon[32753]: pgmap v4095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:21 smithi194 ceph-mon[33017]: pgmap v4096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:21 smithi033 ceph-mon[27845]: pgmap v4096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:21 smithi096 ceph-mon[32753]: pgmap v4096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:22.473+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:22.473+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:22.473+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:39:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6501.783557 secs 2024-04-26T13:39:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6501.783557 secs 2024-04-26T13:39:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6501.783557 secs 2024-04-26T13:39:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6504.535617 secs 2024-04-26T13:39:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:24 smithi096 ceph-mon[32753]: pgmap v4097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6504.535617 secs 2024-04-26T13:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:24 smithi194 ceph-mon[33017]: pgmap v4097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6504.535617 secs 2024-04-26T13:39:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:24 smithi033 ceph-mon[27845]: pgmap v4097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:25 smithi194 ceph-mon[33017]: pgmap v4098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:25 smithi033 ceph-mon[27845]: pgmap v4098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:25 smithi096 ceph-mon[32753]: pgmap v4098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:27 smithi194 ceph-mon[33017]: pgmap v4099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:27.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:27 smithi033 ceph-mon[27845]: pgmap v4099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:27 smithi096 ceph-mon[32753]: pgmap v4099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6506.783642 secs 2024-04-26T13:39:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6506.783642 secs 2024-04-26T13:39:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6506.783642 secs 2024-04-26T13:39:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6509.535740 secs 2024-04-26T13:39:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:29 smithi194 ceph-mon[33017]: pgmap v4100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6509.535740 secs 2024-04-26T13:39:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:29 smithi033 ceph-mon[27845]: pgmap v4100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6509.535740 secs 2024-04-26T13:39:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:29 smithi096 ceph-mon[32753]: pgmap v4100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:31 smithi194 ceph-mon[33017]: pgmap v4101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:31 smithi033 ceph-mon[27845]: pgmap v4101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:31 smithi096 ceph-mon[32753]: pgmap v4101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6511.783727 secs 2024-04-26T13:39:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6511.783727 secs 2024-04-26T13:39:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6511.783727 secs 2024-04-26T13:39:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6514.535862 secs 2024-04-26T13:39:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:34 smithi194 ceph-mon[33017]: pgmap v4102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:39:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6514.535862 secs 2024-04-26T13:39:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:34 smithi033 ceph-mon[27845]: pgmap v4102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:39:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6514.535862 secs 2024-04-26T13:39:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:34 smithi096 ceph-mon[32753]: pgmap v4102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:39:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:39:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:39:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:39:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:39:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:39:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:39:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:39:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:39:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:39:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:36 smithi194 ceph-mon[33017]: pgmap v4103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:39:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:36 smithi194 ceph-mon[33017]: pgmap v4104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:36 smithi033 ceph-mon[27845]: pgmap v4103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:39:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:36 smithi033 ceph-mon[27845]: pgmap v4104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:36 smithi096 ceph-mon[32753]: pgmap v4103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:39:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:36 smithi096 ceph-mon[32753]: pgmap v4104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:37.485+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:37.485+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:37.485+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:39:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:38 smithi194 ceph-mon[33017]: pgmap v4105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6516.783809 secs 2024-04-26T13:39:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:38 smithi033 ceph-mon[27845]: pgmap v4105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6516.783809 secs 2024-04-26T13:39:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:38 smithi096 ceph-mon[32753]: pgmap v4105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6516.783809 secs 2024-04-26T13:39:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6519.535986 secs 2024-04-26T13:39:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6519.535986 secs 2024-04-26T13:39:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6519.535986 secs 2024-04-26T13:39:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:40 smithi194 ceph-mon[33017]: pgmap v4106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:39:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:40 smithi033 ceph-mon[27845]: pgmap v4106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:39:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:40 smithi096 ceph-mon[32753]: pgmap v4106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:39:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:42 smithi194 ceph-mon[33017]: pgmap v4107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:42 smithi033 ceph-mon[27845]: pgmap v4107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:42 smithi096 ceph-mon[32753]: pgmap v4107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6521.783895 secs 2024-04-26T13:39:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6521.783895 secs 2024-04-26T13:39:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6521.783895 secs 2024-04-26T13:39:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6524.536135 secs 2024-04-26T13:39:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:44 smithi194 ceph-mon[33017]: pgmap v4108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6524.536135 secs 2024-04-26T13:39:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:44 smithi033 ceph-mon[27845]: pgmap v4108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6524.536135 secs 2024-04-26T13:39:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:44 smithi096 ceph-mon[32753]: pgmap v4108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:39:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:46 smithi194 ceph-mon[33017]: pgmap v4109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:39:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:46 smithi033 ceph-mon[27845]: pgmap v4109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:39:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:46 smithi096 ceph-mon[32753]: pgmap v4109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:39:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:48 smithi194 ceph-mon[33017]: pgmap v4110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6526.783978 secs 2024-04-26T13:39:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:48 smithi033 ceph-mon[27845]: pgmap v4110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6526.783978 secs 2024-04-26T13:39:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:48 smithi096 ceph-mon[32753]: pgmap v4110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6526.783978 secs 2024-04-26T13:39:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6529.536230 secs 2024-04-26T13:39:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6529.536230 secs 2024-04-26T13:39:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6529.536230 secs 2024-04-26T13:39:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:50 smithi194 ceph-mon[33017]: pgmap v4111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:50 smithi033 ceph-mon[27845]: pgmap v4111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:50 smithi096 ceph-mon[32753]: pgmap v4111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:52.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:52.501+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:52.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:52.501+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:39:52.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:39:52.501+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:39:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:52 smithi194 ceph-mon[33017]: pgmap v4112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:52 smithi033 ceph-mon[27845]: pgmap v4112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:52 smithi096 ceph-mon[32753]: pgmap v4112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6531.784064 secs 2024-04-26T13:39:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6531.784064 secs 2024-04-26T13:39:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6531.784064 secs 2024-04-26T13:39:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6534.536353 secs 2024-04-26T13:39:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:54 smithi194 ceph-mon[33017]: pgmap v4113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6534.536353 secs 2024-04-26T13:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:54 smithi033 ceph-mon[27845]: pgmap v4113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6534.536353 secs 2024-04-26T13:39:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:54 smithi096 ceph-mon[32753]: pgmap v4113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:56 smithi033 ceph-mon[27845]: pgmap v4114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:56 smithi096 ceph-mon[32753]: pgmap v4114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:56 smithi194 ceph-mon[33017]: pgmap v4114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:39:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:58 smithi033 ceph-mon[27845]: pgmap v4115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6536.784123 secs 2024-04-26T13:39:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:58 smithi096 ceph-mon[32753]: pgmap v4115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6536.784123 secs 2024-04-26T13:39:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:58 smithi194 ceph-mon[33017]: pgmap v4115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:39:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6536.784123 secs 2024-04-26T13:39:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6539.536478 secs 2024-04-26T13:39:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6539.536478 secs 2024-04-26T13:39:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6539.536478 secs 2024-04-26T13:40:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:00 smithi033 ceph-mon[27845]: pgmap v4116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:40:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:00 smithi096 ceph-mon[32753]: pgmap v4116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:40:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:00 smithi194 ceph-mon[33017]: pgmap v4116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:40:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:02 smithi033 ceph-mon[27845]: pgmap v4117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:02 smithi096 ceph-mon[32753]: pgmap v4117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:02 smithi194 ceph-mon[33017]: pgmap v4117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6541.784185 secs 2024-04-26T13:40:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6541.784185 secs 2024-04-26T13:40:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6541.784185 secs 2024-04-26T13:40:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6544.536565 secs 2024-04-26T13:40:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:04 smithi033 ceph-mon[27845]: pgmap v4118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6544.536565 secs 2024-04-26T13:40:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:04 smithi096 ceph-mon[32753]: pgmap v4118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6544.536565 secs 2024-04-26T13:40:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:04 smithi194 ceph-mon[33017]: pgmap v4118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:06 smithi033 ceph-mon[27845]: pgmap v4119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:06 smithi096 ceph-mon[32753]: pgmap v4119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:06 smithi194 ceph-mon[33017]: pgmap v4119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:07.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:07.510+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:07.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:07.510+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:07.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:07.510+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:40:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:08 smithi033 ceph-mon[27845]: pgmap v4120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6546.784303 secs 2024-04-26T13:40:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:08 smithi096 ceph-mon[32753]: pgmap v4120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6546.784303 secs 2024-04-26T13:40:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:08 smithi194 ceph-mon[33017]: pgmap v4120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6546.784303 secs 2024-04-26T13:40:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6549.536706 secs 2024-04-26T13:40:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6549.536706 secs 2024-04-26T13:40:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6549.536706 secs 2024-04-26T13:40:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:10 smithi033 ceph-mon[27845]: pgmap v4121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:10 smithi096 ceph-mon[32753]: pgmap v4121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:10 smithi194 ceph-mon[33017]: pgmap v4121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:12 smithi033 ceph-mon[27845]: pgmap v4122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:12 smithi096 ceph-mon[32753]: pgmap v4122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:12 smithi194 ceph-mon[33017]: pgmap v4122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6551.784389 secs 2024-04-26T13:40:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6551.784389 secs 2024-04-26T13:40:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6551.784389 secs 2024-04-26T13:40:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6554.536844 secs 2024-04-26T13:40:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:14 smithi194 ceph-mon[33017]: pgmap v4123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6554.536844 secs 2024-04-26T13:40:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:14 smithi033 ceph-mon[27845]: pgmap v4123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6554.536844 secs 2024-04-26T13:40:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:14 smithi096 ceph-mon[32753]: pgmap v4123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:15 smithi194 ceph-mon[33017]: pgmap v4124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:15 smithi033 ceph-mon[27845]: pgmap v4124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:15 smithi096 ceph-mon[32753]: pgmap v4124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:18 smithi194 ceph-mon[33017]: pgmap v4125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:18 smithi033 ceph-mon[27845]: pgmap v4125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:18 smithi096 ceph-mon[32753]: pgmap v4125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:19.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6556.784467 secs 2024-04-26T13:40:19.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6559.536971 secs 2024-04-26T13:40:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6556.784467 secs 2024-04-26T13:40:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6559.536971 secs 2024-04-26T13:40:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6556.784467 secs 2024-04-26T13:40:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6559.536971 secs 2024-04-26T13:40:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:20 smithi194 ceph-mon[33017]: pgmap v4126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:20 smithi033 ceph-mon[27845]: pgmap v4126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:20 smithi096 ceph-mon[32753]: pgmap v4126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:22.522+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:22.522+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:22.522+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:40:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:22 smithi194 ceph-mon[33017]: pgmap v4127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:22 smithi033 ceph-mon[27845]: pgmap v4127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:22 smithi096 ceph-mon[32753]: pgmap v4127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6561.784553 secs 2024-04-26T13:40:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6561.784553 secs 2024-04-26T13:40:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6561.784553 secs 2024-04-26T13:40:24.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6564.537095 secs 2024-04-26T13:40:24.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:24 smithi096 ceph-mon[32753]: pgmap v4128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6564.537095 secs 2024-04-26T13:40:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:24 smithi194 ceph-mon[33017]: pgmap v4128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6564.537095 secs 2024-04-26T13:40:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:24 smithi033 ceph-mon[27845]: pgmap v4128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:26 smithi194 ceph-mon[33017]: pgmap v4129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:26 smithi033 ceph-mon[27845]: pgmap v4129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:26 smithi096 ceph-mon[32753]: pgmap v4129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:28 smithi194 ceph-mon[33017]: pgmap v4130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6566.784637 secs 2024-04-26T13:40:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:28 smithi033 ceph-mon[27845]: pgmap v4130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6566.784637 secs 2024-04-26T13:40:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:28 smithi096 ceph-mon[32753]: pgmap v4130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6566.784637 secs 2024-04-26T13:40:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6569.537219 secs 2024-04-26T13:40:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6569.537219 secs 2024-04-26T13:40:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6569.537219 secs 2024-04-26T13:40:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:30 smithi194 ceph-mon[33017]: pgmap v4131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:30 smithi033 ceph-mon[27845]: pgmap v4131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:30 smithi096 ceph-mon[32753]: pgmap v4131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:32 smithi194 ceph-mon[33017]: pgmap v4132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:32 smithi033 ceph-mon[27845]: pgmap v4132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:32 smithi096 ceph-mon[32753]: pgmap v4132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6571.784725 secs 2024-04-26T13:40:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6571.784725 secs 2024-04-26T13:40:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6571.784725 secs 2024-04-26T13:40:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6574.537345 secs 2024-04-26T13:40:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:34 smithi194 ceph-mon[33017]: pgmap v4133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6574.537345 secs 2024-04-26T13:40:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:34 smithi033 ceph-mon[27845]: pgmap v4133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6574.537345 secs 2024-04-26T13:40:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:34 smithi096 ceph-mon[32753]: pgmap v4133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:36 smithi194 ceph-mon[33017]: pgmap v4134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:40:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:36 smithi033 ceph-mon[27845]: pgmap v4134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:40:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:36 smithi096 ceph-mon[32753]: pgmap v4134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:40:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:37.535+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:37.535+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:37.535+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:40:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:40:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:40:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:40:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:40:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:40:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:40:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:40:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:40:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:40:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:38 smithi194 ceph-mon[33017]: pgmap v4135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:38 smithi194 ceph-mon[33017]: pgmap v4136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6576.784810 secs 2024-04-26T13:40:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:38 smithi033 ceph-mon[27845]: pgmap v4135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:38 smithi033 ceph-mon[27845]: pgmap v4136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6576.784810 secs 2024-04-26T13:40:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:38 smithi096 ceph-mon[32753]: pgmap v4135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:38 smithi096 ceph-mon[32753]: pgmap v4136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6576.784810 secs 2024-04-26T13:40:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6579.537469 secs 2024-04-26T13:40:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6579.537469 secs 2024-04-26T13:40:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6579.537469 secs 2024-04-26T13:40:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:40 smithi194 ceph-mon[33017]: pgmap v4137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:40 smithi033 ceph-mon[27845]: pgmap v4137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:40 smithi096 ceph-mon[32753]: pgmap v4137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:42 smithi194 ceph-mon[33017]: pgmap v4138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:42 smithi033 ceph-mon[27845]: pgmap v4138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:42 smithi096 ceph-mon[32753]: pgmap v4138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6581.784894 secs 2024-04-26T13:40:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6581.784894 secs 2024-04-26T13:40:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6581.784894 secs 2024-04-26T13:40:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:44 smithi194 ceph-mon[33017]: pgmap v4139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6584.537589 secs 2024-04-26T13:40:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:44 smithi033 ceph-mon[27845]: pgmap v4139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6584.537589 secs 2024-04-26T13:40:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:44 smithi096 ceph-mon[32753]: pgmap v4139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6584.537589 secs 2024-04-26T13:40:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:46 smithi194 ceph-mon[33017]: pgmap v4140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:46 smithi033 ceph-mon[27845]: pgmap v4140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:46 smithi096 ceph-mon[32753]: pgmap v4140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:40:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:48 smithi194 ceph-mon[33017]: pgmap v4141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:40:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6586.784951 secs 2024-04-26T13:40:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:48 smithi033 ceph-mon[27845]: pgmap v4141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:40:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6586.784951 secs 2024-04-26T13:40:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:48 smithi096 ceph-mon[32753]: pgmap v4141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:40:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6586.784951 secs 2024-04-26T13:40:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6589.537733 secs 2024-04-26T13:40:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6589.537733 secs 2024-04-26T13:40:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6589.537733 secs 2024-04-26T13:40:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:50 smithi194 ceph-mon[33017]: pgmap v4142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:50 smithi033 ceph-mon[27845]: pgmap v4142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:50 smithi096 ceph-mon[32753]: pgmap v4142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:52.542+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:52.542+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:40:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:40:52.542+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:40:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:52 smithi194 ceph-mon[33017]: pgmap v4143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:52 smithi033 ceph-mon[27845]: pgmap v4143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:52 smithi096 ceph-mon[32753]: pgmap v4143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6591.785037 secs 2024-04-26T13:40:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6591.785037 secs 2024-04-26T13:40:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6591.785037 secs 2024-04-26T13:40:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:54 smithi033 ceph-mon[27845]: pgmap v4144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6594.537878 secs 2024-04-26T13:40:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:54 smithi096 ceph-mon[32753]: pgmap v4144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6594.537878 secs 2024-04-26T13:40:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:54 smithi194 ceph-mon[33017]: pgmap v4144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6594.537878 secs 2024-04-26T13:40:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:56 smithi033 ceph-mon[27845]: pgmap v4145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:56 smithi096 ceph-mon[32753]: pgmap v4145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:56 smithi194 ceph-mon[33017]: pgmap v4145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:40:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:58 smithi033 ceph-mon[27845]: pgmap v4146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6596.785127 secs 2024-04-26T13:40:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:58 smithi096 ceph-mon[32753]: pgmap v4146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6596.785127 secs 2024-04-26T13:40:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:58 smithi194 ceph-mon[33017]: pgmap v4146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:40:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6596.785127 secs 2024-04-26T13:40:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:40:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6599.538006 secs 2024-04-26T13:40:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:40:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6599.538006 secs 2024-04-26T13:40:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:40:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6599.538006 secs 2024-04-26T13:41:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:00 smithi033 ceph-mon[27845]: pgmap v4147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:00 smithi096 ceph-mon[32753]: pgmap v4147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:00 smithi194 ceph-mon[33017]: pgmap v4147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:02 smithi033 ceph-mon[27845]: pgmap v4148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:02 smithi096 ceph-mon[32753]: pgmap v4148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:02 smithi194 ceph-mon[33017]: pgmap v4148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6601.785213 secs 2024-04-26T13:41:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6601.785213 secs 2024-04-26T13:41:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6601.785213 secs 2024-04-26T13:41:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:04 smithi033 ceph-mon[27845]: pgmap v4149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6604.538138 secs 2024-04-26T13:41:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:04 smithi096 ceph-mon[32753]: pgmap v4149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6604.538138 secs 2024-04-26T13:41:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:04 smithi194 ceph-mon[33017]: pgmap v4149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6604.538138 secs 2024-04-26T13:41:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:06 smithi033 ceph-mon[27845]: pgmap v4150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:06 smithi096 ceph-mon[32753]: pgmap v4150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:06 smithi194 ceph-mon[33017]: pgmap v4150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:07.556+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:07.556+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:07.556+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:41:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:08 smithi033 ceph-mon[27845]: pgmap v4151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6606.785297 secs 2024-04-26T13:41:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:08 smithi096 ceph-mon[32753]: pgmap v4151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6606.785297 secs 2024-04-26T13:41:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:08 smithi194 ceph-mon[33017]: pgmap v4151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6606.785297 secs 2024-04-26T13:41:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6609.538224 secs 2024-04-26T13:41:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6609.538224 secs 2024-04-26T13:41:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6609.538224 secs 2024-04-26T13:41:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:10 smithi194 ceph-mon[33017]: pgmap v4152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:10.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:10 smithi033 ceph-mon[27845]: pgmap v4152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:10 smithi096 ceph-mon[32753]: pgmap v4152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:12.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:12 smithi033 ceph-mon[27845]: pgmap v4153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:12 smithi194 ceph-mon[33017]: pgmap v4153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:12 smithi096 ceph-mon[32753]: pgmap v4153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6611.785397 secs 2024-04-26T13:41:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6611.785397 secs 2024-04-26T13:41:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6611.785397 secs 2024-04-26T13:41:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:14 smithi194 ceph-mon[33017]: pgmap v4154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6614.538299 secs 2024-04-26T13:41:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:14 smithi033 ceph-mon[27845]: pgmap v4154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6614.538299 secs 2024-04-26T13:41:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:14 smithi096 ceph-mon[32753]: pgmap v4154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6614.538299 secs 2024-04-26T13:41:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:15 smithi194 ceph-mon[33017]: pgmap v4155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:15 smithi033 ceph-mon[27845]: pgmap v4155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:15 smithi096 ceph-mon[32753]: pgmap v4155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:17 smithi033 ceph-mon[27845]: pgmap v4156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:17 smithi096 ceph-mon[32753]: pgmap v4156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:17 smithi194 ceph-mon[33017]: pgmap v4156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6616.785479 secs 2024-04-26T13:41:19.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6616.785479 secs 2024-04-26T13:41:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6616.785479 secs 2024-04-26T13:41:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6619.538422 secs 2024-04-26T13:41:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:19 smithi194 ceph-mon[33017]: pgmap v4157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6619.538422 secs 2024-04-26T13:41:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:19 smithi033 ceph-mon[27845]: pgmap v4157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6619.538422 secs 2024-04-26T13:41:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:19 smithi096 ceph-mon[32753]: pgmap v4157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 msgr2=0x7feee40b3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 msgr2=0x7feee40b3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 0x7feee40b3000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 0x7feee40b3000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0032000 comp rx=0 tx=0).stop 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 msgr2=0x7feee40b3000 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 msgr2=0x7feee40b3000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b0570 0x7feee40b3000 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.724+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 0x7feee40b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T13:41:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.725+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 0x7feee40b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T13:41:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:21.726+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 0x7feee40b3540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0048000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T13:41:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:21 smithi194 ceph-mon[33017]: pgmap v4158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:21 smithi033 ceph-mon[27845]: pgmap v4158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:21 smithi096 ceph-mon[32753]: pgmap v4158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:22.556+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:22.556+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:41:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:22.556+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:22.916+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:41:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:22.916+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T13:41:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:22.917+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:41:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:22.917+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:41:24.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6621.785566 secs 2024-04-26T13:41:24.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:23 smithi096 ceph-mon[32753]: pgmap v4159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6621.785566 secs 2024-04-26T13:41:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:23 smithi194 ceph-mon[33017]: pgmap v4159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6621.785566 secs 2024-04-26T13:41:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:23 smithi033 ceph-mon[27845]: pgmap v4159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6624.538550 secs 2024-04-26T13:41:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6624.538550 secs 2024-04-26T13:41:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6624.538550 secs 2024-04-26T13:41:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:25 smithi194 ceph-mon[33017]: pgmap v4160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:25 smithi033 ceph-mon[27845]: pgmap v4160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:25 smithi096 ceph-mon[32753]: pgmap v4160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:27 smithi194 ceph-mon[33017]: pgmap v4161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:27 smithi033 ceph-mon[27845]: pgmap v4161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:27 smithi096 ceph-mon[32753]: pgmap v4161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6626.785649 secs 2024-04-26T13:41:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6626.785649 secs 2024-04-26T13:41:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6626.785649 secs 2024-04-26T13:41:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6629.538678 secs 2024-04-26T13:41:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:29 smithi194 ceph-mon[33017]: pgmap v4162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6629.538678 secs 2024-04-26T13:41:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:29 smithi033 ceph-mon[27845]: pgmap v4162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6629.538678 secs 2024-04-26T13:41:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:29 smithi096 ceph-mon[32753]: pgmap v4162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:31 smithi194 ceph-mon[33017]: pgmap v4163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:31 smithi033 ceph-mon[27845]: pgmap v4163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:31 smithi096 ceph-mon[32753]: pgmap v4163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6631.785737 secs 2024-04-26T13:41:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:33 smithi194 ceph-mon[33017]: pgmap v4164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6631.785737 secs 2024-04-26T13:41:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:33 smithi033 ceph-mon[27845]: pgmap v4164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6631.785737 secs 2024-04-26T13:41:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:33 smithi096 ceph-mon[32753]: pgmap v4164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6634.538804 secs 2024-04-26T13:41:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6634.538804 secs 2024-04-26T13:41:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6634.538804 secs 2024-04-26T13:41:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:35 smithi194 ceph-mon[33017]: pgmap v4165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:35 smithi033 ceph-mon[27845]: pgmap v4165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:35 smithi096 ceph-mon[32753]: pgmap v4165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:37.572+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:37.572+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:41:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:37.572+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:38 smithi194 ceph-mon[33017]: pgmap v4166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:41:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:38 smithi033 ceph-mon[27845]: pgmap v4166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:41:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:38 smithi096 ceph-mon[32753]: pgmap v4166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:41:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6636.785830 secs 2024-04-26T13:41:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:41:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:41:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6636.785830 secs 2024-04-26T13:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:41:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6636.785830 secs 2024-04-26T13:41:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:41:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:41:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:41:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6639.538923 secs 2024-04-26T13:41:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:40 smithi194 ceph-mon[33017]: pgmap v4167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:40 smithi194 ceph-mon[33017]: pgmap v4168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6639.538923 secs 2024-04-26T13:41:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:40 smithi033 ceph-mon[27845]: pgmap v4167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:40 smithi033 ceph-mon[27845]: pgmap v4168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6639.538923 secs 2024-04-26T13:41:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:40 smithi096 ceph-mon[32753]: pgmap v4167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:40 smithi096 ceph-mon[32753]: pgmap v4168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:42.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:42 smithi033 ceph-mon[27845]: pgmap v4169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:42 smithi194 ceph-mon[33017]: pgmap v4169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:42 smithi096 ceph-mon[32753]: pgmap v4169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6641.785911 secs 2024-04-26T13:41:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:44 smithi194 ceph-mon[33017]: pgmap v4170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6641.785911 secs 2024-04-26T13:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:44 smithi033 ceph-mon[27845]: pgmap v4170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6641.785911 secs 2024-04-26T13:41:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:44 smithi096 ceph-mon[32753]: pgmap v4170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6644.539067 secs 2024-04-26T13:41:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6644.539067 secs 2024-04-26T13:41:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6644.539067 secs 2024-04-26T13:41:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:46 smithi194 ceph-mon[33017]: pgmap v4171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:46 smithi033 ceph-mon[27845]: pgmap v4171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:46 smithi096 ceph-mon[32753]: pgmap v4171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:48 smithi194 ceph-mon[33017]: pgmap v4172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:48 smithi033 ceph-mon[27845]: pgmap v4172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:48 smithi096 ceph-mon[32753]: pgmap v4172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:41:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6646.785996 secs 2024-04-26T13:41:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6646.785996 secs 2024-04-26T13:41:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6646.785996 secs 2024-04-26T13:41:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6649.539203 secs 2024-04-26T13:41:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:50 smithi194 ceph-mon[33017]: pgmap v4173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:41:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6649.539203 secs 2024-04-26T13:41:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:50 smithi033 ceph-mon[27845]: pgmap v4173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:41:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6649.539203 secs 2024-04-26T13:41:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:50 smithi096 ceph-mon[32753]: pgmap v4173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:41:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:52 smithi194 ceph-mon[33017]: pgmap v4174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:52 smithi033 ceph-mon[27845]: pgmap v4174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:52 smithi096 ceph-mon[32753]: pgmap v4174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:52.588+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:52.588+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:41:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:41:52.588+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:41:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6651.786057 secs 2024-04-26T13:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:54 smithi194 ceph-mon[33017]: pgmap v4175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6651.786057 secs 2024-04-26T13:41:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:54 smithi033 ceph-mon[27845]: pgmap v4175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6651.786057 secs 2024-04-26T13:41:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:54 smithi096 ceph-mon[32753]: pgmap v4175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6654.539329 secs 2024-04-26T13:41:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6654.539329 secs 2024-04-26T13:41:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6654.539329 secs 2024-04-26T13:41:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:56 smithi194 ceph-mon[33017]: pgmap v4176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:56 smithi033 ceph-mon[27845]: pgmap v4176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:56 smithi096 ceph-mon[32753]: pgmap v4176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:41:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:58 smithi194 ceph-mon[33017]: pgmap v4177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:58 smithi033 ceph-mon[27845]: pgmap v4177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:58 smithi096 ceph-mon[32753]: pgmap v4177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:41:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:41:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6656.786139 secs 2024-04-26T13:41:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:41:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6656.786139 secs 2024-04-26T13:41:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:41:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6656.786139 secs 2024-04-26T13:42:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6659.539465 secs 2024-04-26T13:42:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:00 smithi194 ceph-mon[33017]: pgmap v4178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6659.539465 secs 2024-04-26T13:42:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:00 smithi033 ceph-mon[27845]: pgmap v4178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6659.539465 secs 2024-04-26T13:42:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:00 smithi096 ceph-mon[32753]: pgmap v4178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:02 smithi194 ceph-mon[33017]: pgmap v4179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:02 smithi033 ceph-mon[27845]: pgmap v4179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:02 smithi096 ceph-mon[32753]: pgmap v4179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6661.786222 secs 2024-04-26T13:42:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:04 smithi033 ceph-mon[27845]: pgmap v4180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6664.539531 secs 2024-04-26T13:42:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6661.786222 secs 2024-04-26T13:42:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:04 smithi096 ceph-mon[32753]: pgmap v4180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6664.539531 secs 2024-04-26T13:42:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6661.786222 secs 2024-04-26T13:42:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:04 smithi194 ceph-mon[33017]: pgmap v4180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6664.539531 secs 2024-04-26T13:42:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:06 smithi033 ceph-mon[27845]: pgmap v4181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:06 smithi096 ceph-mon[32753]: pgmap v4181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:06 smithi194 ceph-mon[33017]: pgmap v4181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:07.601 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:07.602+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:07.602+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:07.602+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:42:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:08 smithi033 ceph-mon[27845]: pgmap v4182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:08 smithi096 ceph-mon[32753]: pgmap v4182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:08 smithi194 ceph-mon[33017]: pgmap v4182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6666.786305 secs 2024-04-26T13:42:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6669.539675 secs 2024-04-26T13:42:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6666.786305 secs 2024-04-26T13:42:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6669.539675 secs 2024-04-26T13:42:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6666.786305 secs 2024-04-26T13:42:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6669.539675 secs 2024-04-26T13:42:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:10 smithi033 ceph-mon[27845]: pgmap v4183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:10 smithi096 ceph-mon[32753]: pgmap v4183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:10 smithi194 ceph-mon[33017]: pgmap v4183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:12 smithi033 ceph-mon[27845]: pgmap v4184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:12 smithi096 ceph-mon[32753]: pgmap v4184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:12 smithi194 ceph-mon[33017]: pgmap v4184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6671.786388 secs 2024-04-26T13:42:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:14 smithi033 ceph-mon[27845]: pgmap v4185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6674.539737 secs 2024-04-26T13:42:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6671.786388 secs 2024-04-26T13:42:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:14 smithi096 ceph-mon[32753]: pgmap v4185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6674.539737 secs 2024-04-26T13:42:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6671.786388 secs 2024-04-26T13:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:14 smithi194 ceph-mon[33017]: pgmap v4185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6674.539737 secs 2024-04-26T13:42:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:16 smithi033 ceph-mon[27845]: pgmap v4186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:16 smithi096 ceph-mon[32753]: pgmap v4186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:16 smithi194 ceph-mon[33017]: pgmap v4186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:18 smithi033 ceph-mon[27845]: pgmap v4187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:18 smithi096 ceph-mon[32753]: pgmap v4187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:18 smithi194 ceph-mon[33017]: pgmap v4187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6676.786472 secs 2024-04-26T13:42:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6679.539863 secs 2024-04-26T13:42:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6676.786472 secs 2024-04-26T13:42:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6679.539863 secs 2024-04-26T13:42:19.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6676.786472 secs 2024-04-26T13:42:19.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6679.539863 secs 2024-04-26T13:42:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:20 smithi033 ceph-mon[27845]: pgmap v4188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:20 smithi096 ceph-mon[32753]: pgmap v4188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:20 smithi194 ceph-mon[33017]: pgmap v4188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:22 smithi033 ceph-mon[27845]: pgmap v4189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:22 smithi096 ceph-mon[32753]: pgmap v4189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:22 smithi194 ceph-mon[33017]: pgmap v4189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:22.611+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:22.611+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:22.611+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6681.786548 secs 2024-04-26T13:42:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:24 smithi033 ceph-mon[27845]: pgmap v4190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6684.539985 secs 2024-04-26T13:42:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6681.786548 secs 2024-04-26T13:42:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:24 smithi096 ceph-mon[32753]: pgmap v4190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6684.539985 secs 2024-04-26T13:42:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6681.786548 secs 2024-04-26T13:42:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:24 smithi194 ceph-mon[33017]: pgmap v4190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6684.539985 secs 2024-04-26T13:42:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:26 smithi096 ceph-mon[32753]: pgmap v4191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:26 smithi194 ceph-mon[33017]: pgmap v4191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:26.686 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:26 smithi033 ceph-mon[27845]: pgmap v4191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:28 smithi194 ceph-mon[33017]: pgmap v4192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:28 smithi033 ceph-mon[27845]: pgmap v4192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:28 smithi096 ceph-mon[32753]: pgmap v4192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6686.786632 secs 2024-04-26T13:42:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6689.540137 secs 2024-04-26T13:42:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6686.786632 secs 2024-04-26T13:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6689.540137 secs 2024-04-26T13:42:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6686.786632 secs 2024-04-26T13:42:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6689.540137 secs 2024-04-26T13:42:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:30 smithi194 ceph-mon[33017]: pgmap v4193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:30 smithi033 ceph-mon[27845]: pgmap v4193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:30 smithi096 ceph-mon[32753]: pgmap v4193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:32 smithi194 ceph-mon[33017]: pgmap v4194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:32 smithi033 ceph-mon[27845]: pgmap v4194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:32 smithi096 ceph-mon[32753]: pgmap v4194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6691.786715 secs 2024-04-26T13:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:34 smithi194 ceph-mon[33017]: pgmap v4195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6694.540258 secs 2024-04-26T13:42:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6691.786715 secs 2024-04-26T13:42:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:34 smithi033 ceph-mon[27845]: pgmap v4195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6694.540258 secs 2024-04-26T13:42:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6691.786715 secs 2024-04-26T13:42:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:34 smithi096 ceph-mon[32753]: pgmap v4195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6694.540258 secs 2024-04-26T13:42:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:36 smithi194 ceph-mon[33017]: pgmap v4196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:36 smithi033 ceph-mon[27845]: pgmap v4196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:36 smithi096 ceph-mon[32753]: pgmap v4196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:37.620+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:37.620+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:37.620+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:42:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:38 smithi194 ceph-mon[33017]: pgmap v4197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6696.786789 secs 2024-04-26T13:42:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:38 smithi033 ceph-mon[27845]: pgmap v4197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6696.786789 secs 2024-04-26T13:42:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:38 smithi096 ceph-mon[32753]: pgmap v4197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6696.786789 secs 2024-04-26T13:42:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6699.540350 secs 2024-04-26T13:42:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:42:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6699.540350 secs 2024-04-26T13:42:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:42:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6699.540350 secs 2024-04-26T13:42:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:42:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:40 smithi194 ceph-mon[33017]: pgmap v4198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:40 smithi033 ceph-mon[27845]: pgmap v4198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:40 smithi096 ceph-mon[32753]: pgmap v4198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:42:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:42:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:41 smithi194 ceph-mon[33017]: pgmap v4199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:42:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:41 smithi194 ceph-mon[33017]: pgmap v4200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:42:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:42:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:42:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:41 smithi033 ceph-mon[27845]: pgmap v4199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:42:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:41 smithi033 ceph-mon[27845]: pgmap v4200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:42:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:42:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:42:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:41 smithi096 ceph-mon[32753]: pgmap v4199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:42:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:41 smithi096 ceph-mon[32753]: pgmap v4200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:42:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:43 smithi194 ceph-mon[33017]: pgmap v4201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6701.786872 secs 2024-04-26T13:42:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:43 smithi033 ceph-mon[27845]: pgmap v4201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6701.786872 secs 2024-04-26T13:42:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:43 smithi096 ceph-mon[32753]: pgmap v4201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6701.786872 secs 2024-04-26T13:42:44.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6704.540456 secs 2024-04-26T13:42:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6704.540456 secs 2024-04-26T13:42:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6704.540456 secs 2024-04-26T13:42:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:45 smithi194 ceph-mon[33017]: pgmap v4202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:42:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:45 smithi033 ceph-mon[27845]: pgmap v4202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:42:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:45 smithi096 ceph-mon[32753]: pgmap v4202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:42:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:47 smithi194 ceph-mon[33017]: pgmap v4203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:47 smithi033 ceph-mon[27845]: pgmap v4203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:47 smithi096 ceph-mon[32753]: pgmap v4203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6706.786957 secs 2024-04-26T13:42:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6706.786957 secs 2024-04-26T13:42:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6706.786957 secs 2024-04-26T13:42:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6709.540585 secs 2024-04-26T13:42:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:49 smithi194 ceph-mon[33017]: pgmap v4204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6709.540585 secs 2024-04-26T13:42:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:49 smithi033 ceph-mon[27845]: pgmap v4204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6709.540585 secs 2024-04-26T13:42:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:49 smithi096 ceph-mon[32753]: pgmap v4204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:42:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:51 smithi194 ceph-mon[33017]: pgmap v4205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:42:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:51 smithi033 ceph-mon[27845]: pgmap v4205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:42:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:51 smithi096 ceph-mon[32753]: pgmap v4205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:42:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:52.629+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:52.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:52.629+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:42:52.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:42:52.629+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:42:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:53 smithi194 ceph-mon[33017]: pgmap v4206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6711.787040 secs 2024-04-26T13:42:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:53 smithi033 ceph-mon[27845]: pgmap v4206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6711.787040 secs 2024-04-26T13:42:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:53 smithi096 ceph-mon[32753]: pgmap v4206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6711.787040 secs 2024-04-26T13:42:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6714.540731 secs 2024-04-26T13:42:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6714.540731 secs 2024-04-26T13:42:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6714.540731 secs 2024-04-26T13:42:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:55 smithi033 ceph-mon[27845]: pgmap v4207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:55 smithi096 ceph-mon[32753]: pgmap v4207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:55 smithi194 ceph-mon[33017]: pgmap v4207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:57 smithi033 ceph-mon[27845]: pgmap v4208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:57 smithi096 ceph-mon[32753]: pgmap v4208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:57 smithi194 ceph-mon[33017]: pgmap v4208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:42:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6716.787125 secs 2024-04-26T13:42:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6716.787125 secs 2024-04-26T13:42:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6716.787125 secs 2024-04-26T13:42:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6719.540856 secs 2024-04-26T13:42:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:42:59 smithi033 ceph-mon[27845]: pgmap v4209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6719.540856 secs 2024-04-26T13:42:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:42:59 smithi096 ceph-mon[32753]: pgmap v4209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:42:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6719.540856 secs 2024-04-26T13:42:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:42:59 smithi194 ceph-mon[33017]: pgmap v4209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:01 smithi033 ceph-mon[27845]: pgmap v4210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:01 smithi096 ceph-mon[32753]: pgmap v4210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:01 smithi194 ceph-mon[33017]: pgmap v4210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:03 smithi033 ceph-mon[27845]: pgmap v4211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6721.787213 secs 2024-04-26T13:43:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:03 smithi096 ceph-mon[32753]: pgmap v4211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6721.787213 secs 2024-04-26T13:43:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:03 smithi194 ceph-mon[33017]: pgmap v4211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6721.787213 secs 2024-04-26T13:43:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6724.540978 secs 2024-04-26T13:43:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6724.540978 secs 2024-04-26T13:43:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6724.540978 secs 2024-04-26T13:43:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:05 smithi033 ceph-mon[27845]: pgmap v4212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:05 smithi096 ceph-mon[32753]: pgmap v4212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:05 smithi194 ceph-mon[33017]: pgmap v4212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:07.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:07.644+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:07.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:07.644+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:43:07.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:07.644+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:07 smithi033 ceph-mon[27845]: pgmap v4213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:07 smithi096 ceph-mon[32753]: pgmap v4213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:07 smithi194 ceph-mon[33017]: pgmap v4213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6726.787312 secs 2024-04-26T13:43:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6726.787312 secs 2024-04-26T13:43:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6726.787312 secs 2024-04-26T13:43:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6729.541138 secs 2024-04-26T13:43:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:09 smithi033 ceph-mon[27845]: pgmap v4214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6729.541138 secs 2024-04-26T13:43:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:09 smithi096 ceph-mon[32753]: pgmap v4214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6729.541138 secs 2024-04-26T13:43:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:09 smithi194 ceph-mon[33017]: pgmap v4214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:11 smithi096 ceph-mon[32753]: pgmap v4215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:11 smithi194 ceph-mon[33017]: pgmap v4215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:11 smithi033 ceph-mon[27845]: pgmap v4215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:13.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:13 smithi033 ceph-mon[27845]: pgmap v4216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:13.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6731.787372 secs 2024-04-26T13:43:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:13 smithi194 ceph-mon[33017]: pgmap v4216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6731.787372 secs 2024-04-26T13:43:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:13 smithi096 ceph-mon[32753]: pgmap v4216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6731.787372 secs 2024-04-26T13:43:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6734.541256 secs 2024-04-26T13:43:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6734.541256 secs 2024-04-26T13:43:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6734.541256 secs 2024-04-26T13:43:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:15 smithi194 ceph-mon[33017]: pgmap v4217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:15 smithi033 ceph-mon[27845]: pgmap v4217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:15 smithi096 ceph-mon[32753]: pgmap v4217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:17 smithi194 ceph-mon[33017]: pgmap v4218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:17 smithi033 ceph-mon[27845]: pgmap v4218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:17 smithi096 ceph-mon[32753]: pgmap v4218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6736.787457 secs 2024-04-26T13:43:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6736.787457 secs 2024-04-26T13:43:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6736.787457 secs 2024-04-26T13:43:19.753 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6739.541320 secs 2024-04-26T13:43:19.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:19 smithi194 ceph-mon[33017]: pgmap v4219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6739.541320 secs 2024-04-26T13:43:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:19 smithi033 ceph-mon[27845]: pgmap v4219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6739.541320 secs 2024-04-26T13:43:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:19 smithi096 ceph-mon[32753]: pgmap v4219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:21 smithi194 ceph-mon[33017]: pgmap v4220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:21 smithi033 ceph-mon[27845]: pgmap v4220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:21 smithi096 ceph-mon[32753]: pgmap v4220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:22.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:22.658+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:22.658+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:22.658+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:43:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:23 smithi194 ceph-mon[33017]: pgmap v4221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6741.787541 secs 2024-04-26T13:43:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:23 smithi033 ceph-mon[27845]: pgmap v4221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6741.787541 secs 2024-04-26T13:43:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:23 smithi096 ceph-mon[32753]: pgmap v4221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6741.787541 secs 2024-04-26T13:43:24.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6744.541467 secs 2024-04-26T13:43:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6744.541467 secs 2024-04-26T13:43:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6744.541467 secs 2024-04-26T13:43:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:25 smithi194 ceph-mon[33017]: pgmap v4222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:25 smithi033 ceph-mon[27845]: pgmap v4222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:25 smithi096 ceph-mon[32753]: pgmap v4222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:26 smithi194 ceph-mon[33017]: pgmap v4223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:26.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:26 smithi033 ceph-mon[27845]: pgmap v4223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:26 smithi096 ceph-mon[32753]: pgmap v4223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6746.787624 secs 2024-04-26T13:43:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6746.787624 secs 2024-04-26T13:43:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6746.787624 secs 2024-04-26T13:43:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6749.541579 secs 2024-04-26T13:43:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:29 smithi194 ceph-mon[33017]: pgmap v4224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6749.541579 secs 2024-04-26T13:43:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:29 smithi033 ceph-mon[27845]: pgmap v4224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6749.541579 secs 2024-04-26T13:43:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:29 smithi096 ceph-mon[32753]: pgmap v4224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:31 smithi194 ceph-mon[33017]: pgmap v4225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:31 smithi033 ceph-mon[27845]: pgmap v4225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:31 smithi096 ceph-mon[32753]: pgmap v4225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:33 smithi194 ceph-mon[33017]: pgmap v4226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6751.787705 secs 2024-04-26T13:43:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6754.541705 secs 2024-04-26T13:43:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:33 smithi033 ceph-mon[27845]: pgmap v4226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6751.787705 secs 2024-04-26T13:43:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6754.541705 secs 2024-04-26T13:43:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:33 smithi096 ceph-mon[32753]: pgmap v4226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6751.787705 secs 2024-04-26T13:43:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6754.541705 secs 2024-04-26T13:43:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:35 smithi194 ceph-mon[33017]: pgmap v4227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:35 smithi033 ceph-mon[27845]: pgmap v4227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:35 smithi096 ceph-mon[32753]: pgmap v4227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:37 smithi194 ceph-mon[33017]: pgmap v4228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:37.672+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:37.672+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:43:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:37.672+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:37 smithi033 ceph-mon[27845]: pgmap v4228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:37 smithi096 ceph-mon[32753]: pgmap v4228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6756.787787 secs 2024-04-26T13:43:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6759.541816 secs 2024-04-26T13:43:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6756.787787 secs 2024-04-26T13:43:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6759.541816 secs 2024-04-26T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6756.787787 secs 2024-04-26T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6759.541816 secs 2024-04-26T13:43:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:39 smithi033 ceph-mon[27845]: pgmap v4229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:39 smithi096 ceph-mon[32753]: pgmap v4229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:39 smithi194 ceph-mon[33017]: pgmap v4229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:41 smithi033 ceph-mon[27845]: pgmap v4230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:43:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:41 smithi096 ceph-mon[32753]: pgmap v4230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:43:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:41 smithi194 ceph-mon[33017]: pgmap v4230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:43:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:42 smithi194 ceph-mon[33017]: pgmap v4231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:42 smithi033 ceph-mon[27845]: pgmap v4231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:42 smithi096 ceph-mon[32753]: pgmap v4231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6761.787872 secs 2024-04-26T13:43:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:43:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:43:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:43 smithi033 ceph-mon[27845]: pgmap v4232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:43:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:43 smithi033 ceph-mon[27845]: pgmap v4233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:43.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:43.594 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6764.541955 secs 2024-04-26T13:43:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6761.787872 secs 2024-04-26T13:43:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:43:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:43:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:43 smithi194 ceph-mon[33017]: pgmap v4232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:43:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:43 smithi194 ceph-mon[33017]: pgmap v4233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6764.541955 secs 2024-04-26T13:43:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6761.787872 secs 2024-04-26T13:43:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:43:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:43:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:43 smithi096 ceph-mon[32753]: pgmap v4232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:43:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:43 smithi096 ceph-mon[32753]: pgmap v4233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:43:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6764.541955 secs 2024-04-26T13:43:46.230 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:45 smithi033 ceph-mon[27845]: pgmap v4234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:45 smithi194 ceph-mon[33017]: pgmap v4234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:45 smithi096 ceph-mon[32753]: pgmap v4234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:47 smithi194 ceph-mon[33017]: pgmap v4235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:47 smithi033 ceph-mon[27845]: pgmap v4235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:47 smithi096 ceph-mon[32753]: pgmap v4235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:49.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6766.787954 secs 2024-04-26T13:43:49.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6769.542094 secs 2024-04-26T13:43:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6766.787954 secs 2024-04-26T13:43:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6769.542094 secs 2024-04-26T13:43:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6766.787954 secs 2024-04-26T13:43:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6769.542094 secs 2024-04-26T13:43:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:49 smithi194 ceph-mon[33017]: pgmap v4236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:43:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:49 smithi033 ceph-mon[27845]: pgmap v4236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:43:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:49 smithi096 ceph-mon[32753]: pgmap v4236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:43:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:51 smithi194 ceph-mon[33017]: pgmap v4237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:51 smithi033 ceph-mon[27845]: pgmap v4237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:51 smithi096 ceph-mon[32753]: pgmap v4237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:43:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:52.683+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:52.683+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:43:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:43:52.683+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:43:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6771.788036 secs 2024-04-26T13:43:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:54 smithi194 ceph-mon[33017]: pgmap v4238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:43:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6774.542222 secs 2024-04-26T13:43:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6771.788036 secs 2024-04-26T13:43:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:54 smithi033 ceph-mon[27845]: pgmap v4238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:43:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6774.542222 secs 2024-04-26T13:43:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6771.788036 secs 2024-04-26T13:43:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:54 smithi096 ceph-mon[32753]: pgmap v4238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:43:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6774.542222 secs 2024-04-26T13:43:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:56 smithi194 ceph-mon[33017]: pgmap v4239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:56 smithi033 ceph-mon[27845]: pgmap v4239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:56 smithi096 ceph-mon[32753]: pgmap v4239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:43:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:58 smithi194 ceph-mon[33017]: pgmap v4240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:58 smithi033 ceph-mon[27845]: pgmap v4240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:58.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:58 smithi096 ceph-mon[32753]: pgmap v4240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:43:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6776.788117 secs 2024-04-26T13:43:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6779.542321 secs 2024-04-26T13:43:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6776.788117 secs 2024-04-26T13:43:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6779.542321 secs 2024-04-26T13:43:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6776.788117 secs 2024-04-26T13:43:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6779.542321 secs 2024-04-26T13:44:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:00 smithi194 ceph-mon[33017]: pgmap v4241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:00 smithi033 ceph-mon[27845]: pgmap v4241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:00 smithi096 ceph-mon[32753]: pgmap v4241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:02 smithi194 ceph-mon[33017]: pgmap v4242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:02 smithi033 ceph-mon[27845]: pgmap v4242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:02 smithi096 ceph-mon[32753]: pgmap v4242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6781.788177 secs 2024-04-26T13:44:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:04 smithi194 ceph-mon[33017]: pgmap v4243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6784.542442 secs 2024-04-26T13:44:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6781.788177 secs 2024-04-26T13:44:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:04 smithi033 ceph-mon[27845]: pgmap v4243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6784.542442 secs 2024-04-26T13:44:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6781.788177 secs 2024-04-26T13:44:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:04 smithi096 ceph-mon[32753]: pgmap v4243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6784.542442 secs 2024-04-26T13:44:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:06 smithi194 ceph-mon[33017]: pgmap v4244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:06 smithi033 ceph-mon[27845]: pgmap v4244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:06 smithi096 ceph-mon[32753]: pgmap v4244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:07.697+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:07.697+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:07.697+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:44:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:08 smithi194 ceph-mon[33017]: pgmap v4245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:08 smithi033 ceph-mon[27845]: pgmap v4245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:08 smithi096 ceph-mon[32753]: pgmap v4245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6786.788276 secs 2024-04-26T13:44:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6789.542572 secs 2024-04-26T13:44:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6786.788276 secs 2024-04-26T13:44:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6789.542572 secs 2024-04-26T13:44:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6786.788276 secs 2024-04-26T13:44:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6789.542572 secs 2024-04-26T13:44:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:10 smithi194 ceph-mon[33017]: pgmap v4246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:10 smithi033 ceph-mon[27845]: pgmap v4246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:10 smithi096 ceph-mon[32753]: pgmap v4246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:12 smithi194 ceph-mon[33017]: pgmap v4247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:12 smithi033 ceph-mon[27845]: pgmap v4247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:12 smithi096 ceph-mon[32753]: pgmap v4247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6791.788360 secs 2024-04-26T13:44:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:14 smithi194 ceph-mon[33017]: pgmap v4248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6794.542702 secs 2024-04-26T13:44:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6791.788360 secs 2024-04-26T13:44:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:14 smithi033 ceph-mon[27845]: pgmap v4248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6794.542702 secs 2024-04-26T13:44:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6791.788360 secs 2024-04-26T13:44:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:14 smithi096 ceph-mon[32753]: pgmap v4248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6794.542702 secs 2024-04-26T13:44:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:16 smithi194 ceph-mon[33017]: pgmap v4249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:16 smithi033 ceph-mon[27845]: pgmap v4249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:16 smithi096 ceph-mon[32753]: pgmap v4249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:18.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:18 smithi033 ceph-mon[27845]: pgmap v4250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:18 smithi194 ceph-mon[33017]: pgmap v4250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:18 smithi096 ceph-mon[32753]: pgmap v4250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6796.788441 secs 2024-04-26T13:44:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6799.542841 secs 2024-04-26T13:44:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6796.788441 secs 2024-04-26T13:44:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6799.542841 secs 2024-04-26T13:44:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6796.788441 secs 2024-04-26T13:44:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6799.542841 secs 2024-04-26T13:44:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:20 smithi194 ceph-mon[33017]: pgmap v4251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:20 smithi033 ceph-mon[27845]: pgmap v4251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:20 smithi096 ceph-mon[32753]: pgmap v4251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:22 smithi194 ceph-mon[33017]: pgmap v4252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:22 smithi033 ceph-mon[27845]: pgmap v4252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:22 smithi096 ceph-mon[32753]: pgmap v4252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:22.712+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:22.712+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:22.712+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:44:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:22.946+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:44:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:22.946+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T13:44:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:22.947+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:44:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:22.947+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:44:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6801.788526 secs 2024-04-26T13:44:24.604 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:24 smithi033 ceph-mon[27845]: pgmap v4253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:24.604 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6804.542925 secs 2024-04-26T13:44:24.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6801.788526 secs 2024-04-26T13:44:24.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:24 smithi194 ceph-mon[33017]: pgmap v4253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:24.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6804.542925 secs 2024-04-26T13:44:24.605 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6801.788526 secs 2024-04-26T13:44:24.605 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:24 smithi096 ceph-mon[32753]: pgmap v4253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:24.605 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6804.542925 secs 2024-04-26T13:44:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:26 smithi033 ceph-mon[27845]: pgmap v4254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:26 smithi096 ceph-mon[32753]: pgmap v4254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:26 smithi194 ceph-mon[33017]: pgmap v4254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:28 smithi033 ceph-mon[27845]: pgmap v4255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:28 smithi096 ceph-mon[32753]: pgmap v4255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:28 smithi194 ceph-mon[33017]: pgmap v4255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6806.788607 secs 2024-04-26T13:44:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6809.543061 secs 2024-04-26T13:44:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6806.788607 secs 2024-04-26T13:44:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6809.543061 secs 2024-04-26T13:44:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6806.788607 secs 2024-04-26T13:44:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6809.543061 secs 2024-04-26T13:44:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:30 smithi033 ceph-mon[27845]: pgmap v4256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:30 smithi096 ceph-mon[32753]: pgmap v4256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:30 smithi194 ceph-mon[33017]: pgmap v4256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:32 smithi033 ceph-mon[27845]: pgmap v4257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:32 smithi096 ceph-mon[32753]: pgmap v4257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:32 smithi194 ceph-mon[33017]: pgmap v4257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6814.543186 secs 2024-04-26T13:44:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6814.543186 secs 2024-04-26T13:44:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6814.543186 secs 2024-04-26T13:44:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6811.788688 secs 2024-04-26T13:44:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:34 smithi033 ceph-mon[27845]: pgmap v4258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6811.788688 secs 2024-04-26T13:44:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:34 smithi096 ceph-mon[32753]: pgmap v4258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6811.788688 secs 2024-04-26T13:44:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:34 smithi194 ceph-mon[33017]: pgmap v4258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:36 smithi033 ceph-mon[27845]: pgmap v4259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:36 smithi096 ceph-mon[32753]: pgmap v4259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:36 smithi194 ceph-mon[33017]: pgmap v4259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:37.716+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:37.716+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:37.716+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:44:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:38 smithi033 ceph-mon[27845]: pgmap v4260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6819.543252 secs 2024-04-26T13:44:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:38 smithi096 ceph-mon[32753]: pgmap v4260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6819.543252 secs 2024-04-26T13:44:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:38 smithi194 ceph-mon[33017]: pgmap v4260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6819.543252 secs 2024-04-26T13:44:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6816.788772 secs 2024-04-26T13:44:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6816.788772 secs 2024-04-26T13:44:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6816.788772 secs 2024-04-26T13:44:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:40 smithi096 ceph-mon[32753]: pgmap v4261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:40 smithi194 ceph-mon[33017]: pgmap v4261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:40 smithi033 ceph-mon[27845]: pgmap v4261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:42 smithi194 ceph-mon[33017]: pgmap v4262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:42 smithi033 ceph-mon[27845]: pgmap v4262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:42 smithi096 ceph-mon[32753]: pgmap v4262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:44:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6824.543363 secs 2024-04-26T13:44:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6824.543363 secs 2024-04-26T13:44:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6824.543363 secs 2024-04-26T13:44:44.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6821.788833 secs 2024-04-26T13:44:44.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:44 smithi033 ceph-mon[27845]: pgmap v4263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:44.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:44:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6821.788833 secs 2024-04-26T13:44:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:44 smithi194 ceph-mon[33017]: pgmap v4263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:44:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6821.788833 secs 2024-04-26T13:44:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:44 smithi096 ceph-mon[32753]: pgmap v4263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:44:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:44:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:44:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:44:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:44:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:44:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:44:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:44:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:44:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:44:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:46 smithi194 ceph-mon[33017]: pgmap v4264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:44:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:46 smithi194 ceph-mon[33017]: pgmap v4265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:46 smithi033 ceph-mon[27845]: pgmap v4264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:44:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:46 smithi033 ceph-mon[27845]: pgmap v4265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:46 smithi096 ceph-mon[32753]: pgmap v4264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:44:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:46 smithi096 ceph-mon[32753]: pgmap v4265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:48 smithi194 ceph-mon[33017]: pgmap v4266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6829.543479 secs 2024-04-26T13:44:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:48 smithi033 ceph-mon[27845]: pgmap v4266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6829.543479 secs 2024-04-26T13:44:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:48 smithi096 ceph-mon[32753]: pgmap v4266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6829.543479 secs 2024-04-26T13:44:49.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6826.788915 secs 2024-04-26T13:44:49.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6826.788915 secs 2024-04-26T13:44:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6826.788915 secs 2024-04-26T13:44:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:50 smithi194 ceph-mon[33017]: pgmap v4267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:50 smithi033 ceph-mon[27845]: pgmap v4267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:50 smithi096 ceph-mon[32753]: pgmap v4267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:52 smithi194 ceph-mon[33017]: pgmap v4268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:52 smithi033 ceph-mon[27845]: pgmap v4268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:52 smithi096 ceph-mon[32753]: pgmap v4268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:52.723+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:52.724+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:44:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:44:52.724+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:44:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6831.788995 secs 2024-04-26T13:44:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6834.543562 secs 2024-04-26T13:44:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6831.788995 secs 2024-04-26T13:44:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6834.543562 secs 2024-04-26T13:44:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6831.788995 secs 2024-04-26T13:44:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6834.543562 secs 2024-04-26T13:44:54.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:54 smithi096 ceph-mon[32753]: pgmap v4269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:54 smithi194 ceph-mon[33017]: pgmap v4269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:54 smithi033 ceph-mon[27845]: pgmap v4269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:44:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:56 smithi194 ceph-mon[33017]: pgmap v4270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:44:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:56 smithi033 ceph-mon[27845]: pgmap v4270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:44:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:56 smithi096 ceph-mon[32753]: pgmap v4270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:44:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:58 smithi194 ceph-mon[33017]: pgmap v4271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6836.789079 secs 2024-04-26T13:44:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:44:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6839.543700 secs 2024-04-26T13:44:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:58 smithi033 ceph-mon[27845]: pgmap v4271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6836.789079 secs 2024-04-26T13:44:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:44:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6839.543700 secs 2024-04-26T13:44:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:58 smithi096 ceph-mon[32753]: pgmap v4271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:44:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6836.789079 secs 2024-04-26T13:44:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:44:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6839.543700 secs 2024-04-26T13:45:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:00 smithi194 ceph-mon[33017]: pgmap v4272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:00 smithi033 ceph-mon[27845]: pgmap v4272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:00 smithi096 ceph-mon[32753]: pgmap v4272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:02 smithi194 ceph-mon[33017]: pgmap v4273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:02 smithi033 ceph-mon[27845]: pgmap v4273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:02 smithi096 ceph-mon[32753]: pgmap v4273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6841.789177 secs 2024-04-26T13:45:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6844.543832 secs 2024-04-26T13:45:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6841.789177 secs 2024-04-26T13:45:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6844.543832 secs 2024-04-26T13:45:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6841.789177 secs 2024-04-26T13:45:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6844.543832 secs 2024-04-26T13:45:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:04 smithi194 ceph-mon[33017]: pgmap v4274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:04 smithi033 ceph-mon[27845]: pgmap v4274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:04 smithi096 ceph-mon[32753]: pgmap v4274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:06 smithi194 ceph-mon[33017]: pgmap v4275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:06 smithi033 ceph-mon[27845]: pgmap v4275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:06 smithi096 ceph-mon[32753]: pgmap v4275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:07.739+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:07.739+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:07.739+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:45:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:07.955+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:45:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:07.955+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T13:45:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:07.956+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:45:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:07.956+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:45:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:08 smithi194 ceph-mon[33017]: pgmap v4276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6846.789260 secs 2024-04-26T13:45:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6849.543944 secs 2024-04-26T13:45:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:08 smithi033 ceph-mon[27845]: pgmap v4276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6846.789260 secs 2024-04-26T13:45:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6849.543944 secs 2024-04-26T13:45:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:08 smithi096 ceph-mon[32753]: pgmap v4276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6846.789260 secs 2024-04-26T13:45:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6849.543944 secs 2024-04-26T13:45:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:10 smithi194 ceph-mon[33017]: pgmap v4277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:10 smithi033 ceph-mon[27845]: pgmap v4277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:10 smithi096 ceph-mon[32753]: pgmap v4277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:12 smithi194 ceph-mon[33017]: pgmap v4278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:12 smithi033 ceph-mon[27845]: pgmap v4278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:12 smithi096 ceph-mon[32753]: pgmap v4278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6851.789350 secs 2024-04-26T13:45:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6854.544081 secs 2024-04-26T13:45:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6851.789350 secs 2024-04-26T13:45:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6854.544081 secs 2024-04-26T13:45:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6851.789350 secs 2024-04-26T13:45:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6854.544081 secs 2024-04-26T13:45:14.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:14 smithi033 ceph-mon[27845]: pgmap v4279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:14 smithi194 ceph-mon[33017]: pgmap v4279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:14 smithi096 ceph-mon[32753]: pgmap v4279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:45:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:16 smithi194 ceph-mon[33017]: pgmap v4280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:16 smithi033 ceph-mon[27845]: pgmap v4280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:16 smithi096 ceph-mon[32753]: pgmap v4280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:45:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:18 smithi194 ceph-mon[33017]: pgmap v4281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6856.789433 secs 2024-04-26T13:45:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6859.544171 secs 2024-04-26T13:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:18 smithi033 ceph-mon[27845]: pgmap v4281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6856.789433 secs 2024-04-26T13:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6859.544171 secs 2024-04-26T13:45:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:18 smithi096 ceph-mon[32753]: pgmap v4281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6856.789433 secs 2024-04-26T13:45:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6859.544171 secs 2024-04-26T13:45:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:20 smithi194 ceph-mon[33017]: pgmap v4282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:20 smithi033 ceph-mon[27845]: pgmap v4282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:20 smithi096 ceph-mon[32753]: pgmap v4282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:22 smithi194 ceph-mon[33017]: pgmap v4283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:22 smithi033 ceph-mon[27845]: pgmap v4283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:22 smithi096 ceph-mon[32753]: pgmap v4283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:22.746+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:22.746+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:22.746+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:45:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6861.789520 secs 2024-04-26T13:45:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6864.544283 secs 2024-04-26T13:45:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6861.789520 secs 2024-04-26T13:45:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6864.544283 secs 2024-04-26T13:45:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6861.789520 secs 2024-04-26T13:45:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6864.544283 secs 2024-04-26T13:45:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:24 smithi194 ceph-mon[33017]: pgmap v4284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:24 smithi033 ceph-mon[27845]: pgmap v4284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:24 smithi096 ceph-mon[32753]: pgmap v4284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:26 smithi033 ceph-mon[27845]: pgmap v4285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:26 smithi096 ceph-mon[32753]: pgmap v4285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:26 smithi194 ceph-mon[33017]: pgmap v4285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:28 smithi033 ceph-mon[27845]: pgmap v4286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6866.789621 secs 2024-04-26T13:45:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6869.544403 secs 2024-04-26T13:45:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:28 smithi096 ceph-mon[32753]: pgmap v4286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6866.789621 secs 2024-04-26T13:45:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6869.544403 secs 2024-04-26T13:45:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:28 smithi194 ceph-mon[33017]: pgmap v4286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6866.789621 secs 2024-04-26T13:45:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6869.544403 secs 2024-04-26T13:45:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:30 smithi033 ceph-mon[27845]: pgmap v4287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:30 smithi096 ceph-mon[32753]: pgmap v4287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:30 smithi194 ceph-mon[33017]: pgmap v4287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:32 smithi033 ceph-mon[27845]: pgmap v4288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:32 smithi096 ceph-mon[32753]: pgmap v4288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:32 smithi194 ceph-mon[33017]: pgmap v4288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6871.789710 secs 2024-04-26T13:45:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6874.544522 secs 2024-04-26T13:45:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6871.789710 secs 2024-04-26T13:45:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6874.544522 secs 2024-04-26T13:45:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6871.789710 secs 2024-04-26T13:45:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6874.544522 secs 2024-04-26T13:45:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:34 smithi033 ceph-mon[27845]: pgmap v4289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:34 smithi096 ceph-mon[32753]: pgmap v4289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:34 smithi194 ceph-mon[33017]: pgmap v4289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:36 smithi033 ceph-mon[27845]: pgmap v4290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:36 smithi096 ceph-mon[32753]: pgmap v4290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:36 smithi194 ceph-mon[33017]: pgmap v4290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:37.749+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:37.750+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:37.750+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:45:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:38 smithi033 ceph-mon[27845]: pgmap v4291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6876.789795 secs 2024-04-26T13:45:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6879.544638 secs 2024-04-26T13:45:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:38 smithi096 ceph-mon[32753]: pgmap v4291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6876.789795 secs 2024-04-26T13:45:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6879.544638 secs 2024-04-26T13:45:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:38 smithi194 ceph-mon[33017]: pgmap v4291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6876.789795 secs 2024-04-26T13:45:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6879.544638 secs 2024-04-26T13:45:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:40 smithi033 ceph-mon[27845]: pgmap v4292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:40 smithi096 ceph-mon[32753]: pgmap v4292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:40 smithi194 ceph-mon[33017]: pgmap v4292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:42 smithi033 ceph-mon[27845]: pgmap v4293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:42 smithi096 ceph-mon[32753]: pgmap v4293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:42 smithi194 ceph-mon[33017]: pgmap v4293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6881.789878 secs 2024-04-26T13:45:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6884.544754 secs 2024-04-26T13:45:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6881.789878 secs 2024-04-26T13:45:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6884.544754 secs 2024-04-26T13:45:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6881.789878 secs 2024-04-26T13:45:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6884.544754 secs 2024-04-26T13:45:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:44 smithi033 ceph-mon[27845]: pgmap v4294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:44 smithi096 ceph-mon[32753]: pgmap v4294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:44 smithi194 ceph-mon[33017]: pgmap v4294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:45:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:45:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:45:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:46 smithi033 ceph-mon[27845]: pgmap v4295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:46 smithi096 ceph-mon[32753]: pgmap v4295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:46 smithi194 ceph-mon[33017]: pgmap v4295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: pgmap v4296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6886.789943 secs 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6889.544849 secs 2024-04-26T13:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: pgmap v4296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6886.789943 secs 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6889.544849 secs 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: pgmap v4296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6886.789943 secs 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6889.544849 secs 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:48.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:49 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:49 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:49 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:49 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:49 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:49 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:49 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:49 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:49 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:49 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:49 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:49 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:50 smithi194 ceph-mon[33017]: pgmap v4297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:50 smithi033 ceph-mon[27845]: pgmap v4297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:50 smithi096 ceph-mon[32753]: pgmap v4297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: pgmap v4298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: pgmap v4299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: pgmap v4300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: pgmap v4298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:45:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:45:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:45:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: pgmap v4299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:45:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: pgmap v4300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: pgmap v4298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:45:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:45:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: pgmap v4299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:45:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: pgmap v4300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:45:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:52.763+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:52.763+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:45:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:45:52.763+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:45:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6891.790013 secs 2024-04-26T13:45:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:53 smithi033 ceph-mon[27845]: pgmap v4301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:45:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6894.544967 secs 2024-04-26T13:45:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6891.790013 secs 2024-04-26T13:45:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:53 smithi096 ceph-mon[32753]: pgmap v4301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:45:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6894.544967 secs 2024-04-26T13:45:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6891.790013 secs 2024-04-26T13:45:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:53 smithi194 ceph-mon[33017]: pgmap v4301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:45:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6894.544967 secs 2024-04-26T13:45:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:55 smithi033 ceph-mon[27845]: pgmap v4302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:55 smithi096 ceph-mon[32753]: pgmap v4302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:55 smithi194 ceph-mon[33017]: pgmap v4302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:57 smithi033 ceph-mon[27845]: pgmap v4303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:57 smithi096 ceph-mon[32753]: pgmap v4303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:57 smithi194 ceph-mon[33017]: pgmap v4303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:45:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6896.790095 secs 2024-04-26T13:45:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6899.545086 secs 2024-04-26T13:45:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6896.790095 secs 2024-04-26T13:45:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6899.545086 secs 2024-04-26T13:45:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6896.790095 secs 2024-04-26T13:45:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6899.545086 secs 2024-04-26T13:46:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:45:59 smithi033 ceph-mon[27845]: pgmap v4304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:46:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:45:59 smithi096 ceph-mon[32753]: pgmap v4304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:46:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:45:59 smithi194 ceph-mon[33017]: pgmap v4304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:46:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:01 smithi033 ceph-mon[27845]: pgmap v4305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:46:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:01 smithi096 ceph-mon[32753]: pgmap v4305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:46:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:01 smithi194 ceph-mon[33017]: pgmap v4305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:46:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6901.790156 secs 2024-04-26T13:46:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:03 smithi033 ceph-mon[27845]: pgmap v4306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6904.545204 secs 2024-04-26T13:46:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6901.790156 secs 2024-04-26T13:46:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:03 smithi096 ceph-mon[32753]: pgmap v4306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6904.545204 secs 2024-04-26T13:46:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6901.790156 secs 2024-04-26T13:46:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:03 smithi194 ceph-mon[33017]: pgmap v4306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6904.545204 secs 2024-04-26T13:46:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:05 smithi033 ceph-mon[27845]: pgmap v4307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:05 smithi096 ceph-mon[32753]: pgmap v4307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:05 smithi194 ceph-mon[33017]: pgmap v4307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:07.779+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:07.779+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:07.779+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:46:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:07 smithi033 ceph-mon[27845]: pgmap v4308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:07 smithi096 ceph-mon[32753]: pgmap v4308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:07 smithi194 ceph-mon[33017]: pgmap v4308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6906.790258 secs 2024-04-26T13:46:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6909.545328 secs 2024-04-26T13:46:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6906.790258 secs 2024-04-26T13:46:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6909.545328 secs 2024-04-26T13:46:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6906.790258 secs 2024-04-26T13:46:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6909.545328 secs 2024-04-26T13:46:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:09 smithi033 ceph-mon[27845]: pgmap v4309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:09 smithi096 ceph-mon[32753]: pgmap v4309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:09 smithi194 ceph-mon[33017]: pgmap v4309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:11 smithi033 ceph-mon[27845]: pgmap v4310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:12.262 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:11 smithi096 ceph-mon[32753]: pgmap v4310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:11 smithi194 ceph-mon[33017]: pgmap v4310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6911.790318 secs 2024-04-26T13:46:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:13 smithi096 ceph-mon[32753]: pgmap v4311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6914.545446 secs 2024-04-26T13:46:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6911.790318 secs 2024-04-26T13:46:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:13 smithi194 ceph-mon[33017]: pgmap v4311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6914.545446 secs 2024-04-26T13:46:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6911.790318 secs 2024-04-26T13:46:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:13 smithi033 ceph-mon[27845]: pgmap v4311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6914.545446 secs 2024-04-26T13:46:16.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:15 smithi033 ceph-mon[27845]: pgmap v4312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:15 smithi194 ceph-mon[33017]: pgmap v4312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:15 smithi096 ceph-mon[32753]: pgmap v4312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:17 smithi194 ceph-mon[33017]: pgmap v4313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:17 smithi033 ceph-mon[27845]: pgmap v4313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:17 smithi096 ceph-mon[32753]: pgmap v4313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6916.790401 secs 2024-04-26T13:46:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6919.545595 secs 2024-04-26T13:46:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6916.790401 secs 2024-04-26T13:46:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6919.545595 secs 2024-04-26T13:46:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6916.790401 secs 2024-04-26T13:46:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6919.545595 secs 2024-04-26T13:46:20.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:19 smithi194 ceph-mon[33017]: pgmap v4314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:19 smithi033 ceph-mon[27845]: pgmap v4314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:19 smithi096 ceph-mon[32753]: pgmap v4314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:21 smithi033 ceph-mon[27845]: pgmap v4315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:21 smithi194 ceph-mon[33017]: pgmap v4315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:21 smithi096 ceph-mon[32753]: pgmap v4315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:22.794+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:22.794+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:22.794+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:46:24.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6921.790483 secs 2024-04-26T13:46:24.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:23 smithi096 ceph-mon[32753]: pgmap v4316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:24.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6924.545727 secs 2024-04-26T13:46:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6921.790483 secs 2024-04-26T13:46:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:23 smithi194 ceph-mon[33017]: pgmap v4316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6924.545727 secs 2024-04-26T13:46:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6921.790483 secs 2024-04-26T13:46:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:23 smithi033 ceph-mon[27845]: pgmap v4316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6924.545727 secs 2024-04-26T13:46:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:25 smithi194 ceph-mon[33017]: pgmap v4317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:25 smithi033 ceph-mon[27845]: pgmap v4317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:25 smithi096 ceph-mon[32753]: pgmap v4317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:27 smithi194 ceph-mon[33017]: pgmap v4318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:27 smithi033 ceph-mon[27845]: pgmap v4318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:27 smithi096 ceph-mon[32753]: pgmap v4318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6926.790565 secs 2024-04-26T13:46:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6929.545846 secs 2024-04-26T13:46:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6926.790565 secs 2024-04-26T13:46:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6929.545846 secs 2024-04-26T13:46:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6926.790565 secs 2024-04-26T13:46:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6929.545846 secs 2024-04-26T13:46:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:29 smithi194 ceph-mon[33017]: pgmap v4319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:29 smithi033 ceph-mon[27845]: pgmap v4319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:29 smithi096 ceph-mon[32753]: pgmap v4319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:31 smithi194 ceph-mon[33017]: pgmap v4320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:31 smithi033 ceph-mon[27845]: pgmap v4320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:31 smithi096 ceph-mon[32753]: pgmap v4320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6931.790626 secs 2024-04-26T13:46:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:33 smithi194 ceph-mon[33017]: pgmap v4321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6934.545994 secs 2024-04-26T13:46:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6931.790626 secs 2024-04-26T13:46:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:33 smithi033 ceph-mon[27845]: pgmap v4321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6934.545994 secs 2024-04-26T13:46:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6931.790626 secs 2024-04-26T13:46:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:33 smithi096 ceph-mon[32753]: pgmap v4321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6934.545994 secs 2024-04-26T13:46:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:36 smithi194 ceph-mon[33017]: pgmap v4322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:36 smithi033 ceph-mon[27845]: pgmap v4322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:36 smithi096 ceph-mon[32753]: pgmap v4322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:37.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:37.810+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:37.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:37.810+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:37.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:37.810+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:46:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:38 smithi194 ceph-mon[33017]: pgmap v4323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:38 smithi033 ceph-mon[27845]: pgmap v4323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:38 smithi096 ceph-mon[32753]: pgmap v4323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6936.790711 secs 2024-04-26T13:46:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6939.546136 secs 2024-04-26T13:46:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6936.790711 secs 2024-04-26T13:46:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6939.546136 secs 2024-04-26T13:46:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6936.790711 secs 2024-04-26T13:46:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6939.546136 secs 2024-04-26T13:46:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:40 smithi194 ceph-mon[33017]: pgmap v4324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:40 smithi033 ceph-mon[27845]: pgmap v4324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:40 smithi096 ceph-mon[32753]: pgmap v4324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:42 smithi194 ceph-mon[33017]: pgmap v4325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:42 smithi033 ceph-mon[27845]: pgmap v4325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:42 smithi096 ceph-mon[32753]: pgmap v4325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6941.790795 secs 2024-04-26T13:46:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:44 smithi194 ceph-mon[33017]: pgmap v4326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6944.546244 secs 2024-04-26T13:46:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6941.790795 secs 2024-04-26T13:46:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:44 smithi033 ceph-mon[27845]: pgmap v4326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6944.546244 secs 2024-04-26T13:46:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6941.790795 secs 2024-04-26T13:46:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:44 smithi096 ceph-mon[32753]: pgmap v4326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6944.546244 secs 2024-04-26T13:46:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:46 smithi194 ceph-mon[33017]: pgmap v4327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:46 smithi033 ceph-mon[27845]: pgmap v4327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:46 smithi096 ceph-mon[32753]: pgmap v4327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:48 smithi194 ceph-mon[33017]: pgmap v4328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:48 smithi033 ceph-mon[27845]: pgmap v4328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:48 smithi096 ceph-mon[32753]: pgmap v4328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6946.790879 secs 2024-04-26T13:46:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6949.546364 secs 2024-04-26T13:46:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6946.790879 secs 2024-04-26T13:46:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6949.546364 secs 2024-04-26T13:46:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6946.790879 secs 2024-04-26T13:46:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6949.546364 secs 2024-04-26T13:46:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:50 smithi194 ceph-mon[33017]: pgmap v4329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:50 smithi033 ceph-mon[27845]: pgmap v4329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:50 smithi096 ceph-mon[32753]: pgmap v4329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:52 smithi194 ceph-mon[33017]: pgmap v4330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:46:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:52 smithi033 ceph-mon[27845]: pgmap v4330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:46:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:52 smithi096 ceph-mon[32753]: pgmap v4330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:46:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:46:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:52.811+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:52.811+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:46:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:46:52.811+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:46:53.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:46:53.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:46:53.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:53 smithi033 ceph-mon[27845]: pgmap v4331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:53.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:53 smithi033 ceph-mon[27845]: pgmap v4332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:53.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:46:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:46:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:46:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:53 smithi194 ceph-mon[33017]: pgmap v4331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:53 smithi194 ceph-mon[33017]: pgmap v4332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:46:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:46:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:46:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:53 smithi096 ceph-mon[32753]: pgmap v4331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:46:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:53 smithi096 ceph-mon[32753]: pgmap v4332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:46:54.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6951.790970 secs 2024-04-26T13:46:54.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6954.546481 secs 2024-04-26T13:46:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6951.790970 secs 2024-04-26T13:46:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6954.546481 secs 2024-04-26T13:46:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6951.790970 secs 2024-04-26T13:46:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6954.546481 secs 2024-04-26T13:46:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:55 smithi194 ceph-mon[33017]: pgmap v4333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:55 smithi033 ceph-mon[27845]: pgmap v4333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:55 smithi096 ceph-mon[32753]: pgmap v4333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:56 smithi194 ceph-mon[33017]: pgmap v4334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:56 smithi033 ceph-mon[27845]: pgmap v4334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:56 smithi096 ceph-mon[32753]: pgmap v4334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6959.546610 secs 2024-04-26T13:46:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6959.546610 secs 2024-04-26T13:46:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6959.546610 secs 2024-04-26T13:46:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6956.791055 secs 2024-04-26T13:46:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:46:59 smithi194 ceph-mon[33017]: pgmap v4335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6956.791055 secs 2024-04-26T13:46:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:46:59 smithi033 ceph-mon[27845]: pgmap v4335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:46:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6956.791055 secs 2024-04-26T13:46:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:46:59 smithi096 ceph-mon[32753]: pgmap v4335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:00 smithi194 ceph-mon[33017]: pgmap v4336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:00.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:00 smithi033 ceph-mon[27845]: pgmap v4336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:00 smithi096 ceph-mon[32753]: pgmap v4336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:02 smithi194 ceph-mon[33017]: pgmap v4337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:47:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:02 smithi033 ceph-mon[27845]: pgmap v4337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:47:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:02 smithi096 ceph-mon[32753]: pgmap v4337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:47:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6961.791115 secs 2024-04-26T13:47:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6964.546725 secs 2024-04-26T13:47:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6961.791115 secs 2024-04-26T13:47:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6964.546725 secs 2024-04-26T13:47:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6961.791115 secs 2024-04-26T13:47:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6964.546725 secs 2024-04-26T13:47:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:04 smithi194 ceph-mon[33017]: pgmap v4338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:04 smithi033 ceph-mon[27845]: pgmap v4338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:04 smithi096 ceph-mon[32753]: pgmap v4338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:06 smithi194 ceph-mon[33017]: pgmap v4339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:06 smithi033 ceph-mon[27845]: pgmap v4339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:06 smithi096 ceph-mon[32753]: pgmap v4339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:07.821+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:07.821+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:07.821+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:47:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6969.546841 secs 2024-04-26T13:47:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6969.546841 secs 2024-04-26T13:47:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6969.546841 secs 2024-04-26T13:47:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6966.791202 secs 2024-04-26T13:47:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:09 smithi033 ceph-mon[27845]: pgmap v4340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6966.791202 secs 2024-04-26T13:47:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:09 smithi194 ceph-mon[33017]: pgmap v4340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6966.791202 secs 2024-04-26T13:47:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:09 smithi096 ceph-mon[32753]: pgmap v4340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:10 smithi194 ceph-mon[33017]: pgmap v4341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:10 smithi033 ceph-mon[27845]: pgmap v4341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:10 smithi096 ceph-mon[32753]: pgmap v4341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:12 smithi194 ceph-mon[33017]: pgmap v4342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:12 smithi033 ceph-mon[27845]: pgmap v4342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:12 smithi096 ceph-mon[32753]: pgmap v4342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6971.791265 secs 2024-04-26T13:47:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6974.546964 secs 2024-04-26T13:47:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6971.791265 secs 2024-04-26T13:47:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6974.546964 secs 2024-04-26T13:47:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6971.791265 secs 2024-04-26T13:47:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6974.546964 secs 2024-04-26T13:47:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:14 smithi194 ceph-mon[33017]: pgmap v4343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:14 smithi033 ceph-mon[27845]: pgmap v4343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:14 smithi096 ceph-mon[32753]: pgmap v4343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:16 smithi194 ceph-mon[33017]: pgmap v4344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:16.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:16 smithi033 ceph-mon[27845]: pgmap v4344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:16 smithi096 ceph-mon[32753]: pgmap v4344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6979.547082 secs 2024-04-26T13:47:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6979.547082 secs 2024-04-26T13:47:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6979.547082 secs 2024-04-26T13:47:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6976.791323 secs 2024-04-26T13:47:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:19 smithi194 ceph-mon[33017]: pgmap v4345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6976.791323 secs 2024-04-26T13:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:19 smithi033 ceph-mon[27845]: pgmap v4345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6976.791323 secs 2024-04-26T13:47:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:19 smithi096 ceph-mon[32753]: pgmap v4345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:20 smithi194 ceph-mon[33017]: pgmap v4346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:20 smithi033 ceph-mon[27845]: pgmap v4346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:20 smithi096 ceph-mon[32753]: pgmap v4346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:22.829+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:22.829+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:22.829+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:47:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:22 smithi194 ceph-mon[33017]: pgmap v4347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:22 smithi033 ceph-mon[27845]: pgmap v4347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:22 smithi096 ceph-mon[32753]: pgmap v4347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:22.976+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T13:47:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:22.976+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:47:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:22.976+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T13:47:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:22.977+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:47:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6981.791409 secs 2024-04-26T13:47:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6984.547208 secs 2024-04-26T13:47:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6981.791409 secs 2024-04-26T13:47:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6984.547208 secs 2024-04-26T13:47:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6981.791409 secs 2024-04-26T13:47:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6984.547208 secs 2024-04-26T13:47:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:24 smithi194 ceph-mon[33017]: pgmap v4348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:24 smithi033 ceph-mon[27845]: pgmap v4348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:24 smithi096 ceph-mon[32753]: pgmap v4348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:26 smithi194 ceph-mon[33017]: pgmap v4349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:26 smithi033 ceph-mon[27845]: pgmap v4349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:26 smithi096 ceph-mon[32753]: pgmap v4349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6989.547299 secs 2024-04-26T13:47:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6989.547299 secs 2024-04-26T13:47:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6989.547299 secs 2024-04-26T13:47:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6986.791497 secs 2024-04-26T13:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:29 smithi194 ceph-mon[33017]: pgmap v4350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6986.791497 secs 2024-04-26T13:47:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:29 smithi033 ceph-mon[27845]: pgmap v4350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6986.791497 secs 2024-04-26T13:47:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:29 smithi096 ceph-mon[32753]: pgmap v4350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:30 smithi194 ceph-mon[33017]: pgmap v4351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:30.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:30 smithi033 ceph-mon[27845]: pgmap v4351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:30 smithi096 ceph-mon[32753]: pgmap v4351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:32 smithi194 ceph-mon[33017]: pgmap v4352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:32 smithi033 ceph-mon[27845]: pgmap v4352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:32 smithi096 ceph-mon[32753]: pgmap v4352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6991.791579 secs 2024-04-26T13:47:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6994.547466 secs 2024-04-26T13:47:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6991.791579 secs 2024-04-26T13:47:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6994.547466 secs 2024-04-26T13:47:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6991.791579 secs 2024-04-26T13:47:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6994.547466 secs 2024-04-26T13:47:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:34 smithi194 ceph-mon[33017]: pgmap v4353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:34 smithi033 ceph-mon[27845]: pgmap v4353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:34 smithi096 ceph-mon[32753]: pgmap v4353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:47:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:36 smithi194 ceph-mon[33017]: pgmap v4354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:36 smithi033 ceph-mon[27845]: pgmap v4354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:36 smithi096 ceph-mon[32753]: pgmap v4354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:37.840+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:37.840+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:37.840+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:47:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 6999.547601 secs 2024-04-26T13:47:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 6999.547601 secs 2024-04-26T13:47:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 6999.547601 secs 2024-04-26T13:47:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 6996.791661 secs 2024-04-26T13:47:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:39 smithi194 ceph-mon[33017]: pgmap v4355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 6996.791661 secs 2024-04-26T13:47:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:39 smithi033 ceph-mon[27845]: pgmap v4355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 6996.791661 secs 2024-04-26T13:47:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:39 smithi096 ceph-mon[32753]: pgmap v4355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:40 smithi194 ceph-mon[33017]: pgmap v4356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:40 smithi033 ceph-mon[27845]: pgmap v4356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:40 smithi096 ceph-mon[32753]: pgmap v4356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:42 smithi194 ceph-mon[33017]: pgmap v4357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:42 smithi033 ceph-mon[27845]: pgmap v4357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:42 smithi096 ceph-mon[32753]: pgmap v4357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7001.791747 secs 2024-04-26T13:47:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7004.547731 secs 2024-04-26T13:47:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7001.791747 secs 2024-04-26T13:47:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7004.547731 secs 2024-04-26T13:47:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7001.791747 secs 2024-04-26T13:47:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7004.547731 secs 2024-04-26T13:47:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:44 smithi194 ceph-mon[33017]: pgmap v4358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:44 smithi033 ceph-mon[27845]: pgmap v4358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:44 smithi096 ceph-mon[32753]: pgmap v4358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:46 smithi194 ceph-mon[33017]: pgmap v4359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:46 smithi033 ceph-mon[27845]: pgmap v4359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:46 smithi096 ceph-mon[32753]: pgmap v4359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7009.547845 secs 2024-04-26T13:47:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7009.547845 secs 2024-04-26T13:47:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7009.547845 secs 2024-04-26T13:47:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7006.791827 secs 2024-04-26T13:47:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:49 smithi194 ceph-mon[33017]: pgmap v4360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7006.791827 secs 2024-04-26T13:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:49 smithi033 ceph-mon[27845]: pgmap v4360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7006.791827 secs 2024-04-26T13:47:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:49 smithi096 ceph-mon[32753]: pgmap v4360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:47:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:50 smithi194 ceph-mon[33017]: pgmap v4361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:50 smithi033 ceph-mon[27845]: pgmap v4361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:50 smithi096 ceph-mon[32753]: pgmap v4361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:47:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:52 smithi194 ceph-mon[33017]: pgmap v4362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:52.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:52.855+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:52.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:52.855+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:47:52.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:47:52.855+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:47:52.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:52 smithi033 ceph-mon[27845]: pgmap v4362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:52 smithi096 ceph-mon[32753]: pgmap v4362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7011.791912 secs 2024-04-26T13:47:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:47:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7014.547947 secs 2024-04-26T13:47:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7011.791912 secs 2024-04-26T13:47:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:47:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7014.547947 secs 2024-04-26T13:47:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7011.791912 secs 2024-04-26T13:47:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:47:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7014.547947 secs 2024-04-26T13:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:54 smithi033 ceph-mon[27845]: pgmap v4363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:54 smithi033 ceph-mon[27845]: pgmap v4364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:47:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:47:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:47:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:54 smithi096 ceph-mon[32753]: pgmap v4363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:54 smithi096 ceph-mon[32753]: pgmap v4364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:47:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:47:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:54 smithi194 ceph-mon[33017]: pgmap v4363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:54 smithi194 ceph-mon[33017]: pgmap v4364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:47:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:57 smithi194 ceph-mon[33017]: pgmap v4365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:57 smithi033 ceph-mon[27845]: pgmap v4365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:57 smithi096 ceph-mon[32753]: pgmap v4365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7019.548065 secs 2024-04-26T13:47:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7019.548065 secs 2024-04-26T13:47:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7019.548065 secs 2024-04-26T13:47:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7016.792013 secs 2024-04-26T13:47:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:47:59 smithi194 ceph-mon[33017]: pgmap v4366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7016.792013 secs 2024-04-26T13:47:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:47:59 smithi033 ceph-mon[27845]: pgmap v4366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:47:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7016.792013 secs 2024-04-26T13:47:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:47:59 smithi096 ceph-mon[32753]: pgmap v4366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:01 smithi194 ceph-mon[33017]: pgmap v4367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:01 smithi033 ceph-mon[27845]: pgmap v4367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:01 smithi096 ceph-mon[32753]: pgmap v4367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:03 smithi194 ceph-mon[33017]: pgmap v4368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7024.548199 secs 2024-04-26T13:48:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:03 smithi033 ceph-mon[27845]: pgmap v4368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7024.548199 secs 2024-04-26T13:48:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:03 smithi096 ceph-mon[32753]: pgmap v4368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7024.548199 secs 2024-04-26T13:48:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7021.792100 secs 2024-04-26T13:48:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7021.792100 secs 2024-04-26T13:48:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7021.792100 secs 2024-04-26T13:48:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:05 smithi194 ceph-mon[33017]: pgmap v4369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:48:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:05 smithi033 ceph-mon[27845]: pgmap v4369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:48:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:05 smithi096 ceph-mon[32753]: pgmap v4369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:48:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:07 smithi194 ceph-mon[33017]: pgmap v4370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:07 smithi033 ceph-mon[27845]: pgmap v4370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:07 smithi096 ceph-mon[32753]: pgmap v4370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:07.867+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:07.867+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:07.867+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:48:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7026.792190 secs 2024-04-26T13:48:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7029.548333 secs 2024-04-26T13:48:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7026.792190 secs 2024-04-26T13:48:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7029.548333 secs 2024-04-26T13:48:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7026.792190 secs 2024-04-26T13:48:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7029.548333 secs 2024-04-26T13:48:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:09 smithi194 ceph-mon[33017]: pgmap v4371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:09 smithi033 ceph-mon[27845]: pgmap v4371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:09 smithi096 ceph-mon[32753]: pgmap v4371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:11 smithi194 ceph-mon[33017]: pgmap v4372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:11 smithi033 ceph-mon[27845]: pgmap v4372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:11 smithi096 ceph-mon[32753]: pgmap v4372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:13 smithi194 ceph-mon[33017]: pgmap v4373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7031.792294 secs 2024-04-26T13:48:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7034.548421 secs 2024-04-26T13:48:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:13 smithi033 ceph-mon[27845]: pgmap v4373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7031.792294 secs 2024-04-26T13:48:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7034.548421 secs 2024-04-26T13:48:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:13 smithi096 ceph-mon[32753]: pgmap v4373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7031.792294 secs 2024-04-26T13:48:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7034.548421 secs 2024-04-26T13:48:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:15 smithi194 ceph-mon[33017]: pgmap v4374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:15 smithi033 ceph-mon[27845]: pgmap v4374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:15 smithi096 ceph-mon[32753]: pgmap v4374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:17 smithi194 ceph-mon[33017]: pgmap v4375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:17 smithi033 ceph-mon[27845]: pgmap v4375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:17 smithi096 ceph-mon[32753]: pgmap v4375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7036.792376 secs 2024-04-26T13:48:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7039.548525 secs 2024-04-26T13:48:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7036.792376 secs 2024-04-26T13:48:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7039.548525 secs 2024-04-26T13:48:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7036.792376 secs 2024-04-26T13:48:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7039.548525 secs 2024-04-26T13:48:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:19 smithi194 ceph-mon[33017]: pgmap v4376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:19 smithi033 ceph-mon[27845]: pgmap v4376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:19 smithi096 ceph-mon[32753]: pgmap v4376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:21 smithi194 ceph-mon[33017]: pgmap v4377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:21 smithi033 ceph-mon[27845]: pgmap v4377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:21 smithi096 ceph-mon[32753]: pgmap v4377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:22.881+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:22.882+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:22.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:22.882+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:48:23.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:23 smithi194 ceph-mon[33017]: pgmap v4378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:23.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7041.792438 secs 2024-04-26T13:48:23.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7044.548591 secs 2024-04-26T13:48:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:23 smithi033 ceph-mon[27845]: pgmap v4378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7041.792438 secs 2024-04-26T13:48:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7044.548591 secs 2024-04-26T13:48:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:23 smithi096 ceph-mon[32753]: pgmap v4378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7041.792438 secs 2024-04-26T13:48:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7044.548591 secs 2024-04-26T13:48:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:25 smithi194 ceph-mon[33017]: pgmap v4379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:25 smithi033 ceph-mon[27845]: pgmap v4379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:25 smithi096 ceph-mon[32753]: pgmap v4379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:27 smithi194 ceph-mon[33017]: pgmap v4380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:27 smithi033 ceph-mon[27845]: pgmap v4380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:27 smithi096 ceph-mon[32753]: pgmap v4380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7046.792535 secs 2024-04-26T13:48:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7049.548710 secs 2024-04-26T13:48:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7046.792535 secs 2024-04-26T13:48:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7049.548710 secs 2024-04-26T13:48:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7046.792535 secs 2024-04-26T13:48:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7049.548710 secs 2024-04-26T13:48:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:29 smithi194 ceph-mon[33017]: pgmap v4381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:29 smithi033 ceph-mon[27845]: pgmap v4381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:29 smithi096 ceph-mon[32753]: pgmap v4381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:31 smithi194 ceph-mon[33017]: pgmap v4382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:31 smithi033 ceph-mon[27845]: pgmap v4382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:31 smithi096 ceph-mon[32753]: pgmap v4382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:33 smithi194 ceph-mon[33017]: pgmap v4383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7051.792635 secs 2024-04-26T13:48:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7054.548850 secs 2024-04-26T13:48:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:33 smithi033 ceph-mon[27845]: pgmap v4383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7051.792635 secs 2024-04-26T13:48:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7054.548850 secs 2024-04-26T13:48:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:33 smithi096 ceph-mon[32753]: pgmap v4383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7051.792635 secs 2024-04-26T13:48:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7054.548850 secs 2024-04-26T13:48:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:35 smithi194 ceph-mon[33017]: pgmap v4384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:35 smithi033 ceph-mon[27845]: pgmap v4384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:35 smithi096 ceph-mon[32753]: pgmap v4384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:37 smithi194 ceph-mon[33017]: pgmap v4385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:37 smithi033 ceph-mon[27845]: pgmap v4385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:37 smithi096 ceph-mon[32753]: pgmap v4385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:37.898+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:37.898+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:37.898+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:48:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7056.792736 secs 2024-04-26T13:48:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7059.548997 secs 2024-04-26T13:48:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7056.792736 secs 2024-04-26T13:48:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7059.548997 secs 2024-04-26T13:48:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7056.792736 secs 2024-04-26T13:48:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7059.548997 secs 2024-04-26T13:48:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:39 smithi033 ceph-mon[27845]: pgmap v4386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:39 smithi096 ceph-mon[32753]: pgmap v4386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:39 smithi194 ceph-mon[33017]: pgmap v4386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:41 smithi033 ceph-mon[27845]: pgmap v4387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:41 smithi096 ceph-mon[32753]: pgmap v4387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:41 smithi194 ceph-mon[33017]: pgmap v4387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:43 smithi033 ceph-mon[27845]: pgmap v4388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7061.792836 secs 2024-04-26T13:48:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7064.549130 secs 2024-04-26T13:48:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:43 smithi096 ceph-mon[32753]: pgmap v4388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7061.792836 secs 2024-04-26T13:48:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7064.549130 secs 2024-04-26T13:48:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:43 smithi194 ceph-mon[33017]: pgmap v4388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7061.792836 secs 2024-04-26T13:48:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7064.549130 secs 2024-04-26T13:48:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:45 smithi033 ceph-mon[27845]: pgmap v4389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:45 smithi096 ceph-mon[32753]: pgmap v4389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:45 smithi194 ceph-mon[33017]: pgmap v4389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:47 smithi033 ceph-mon[27845]: pgmap v4390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:47 smithi096 ceph-mon[32753]: pgmap v4390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:47 smithi194 ceph-mon[33017]: pgmap v4390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7066.792935 secs 2024-04-26T13:48:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7069.549248 secs 2024-04-26T13:48:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7066.792935 secs 2024-04-26T13:48:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7069.549248 secs 2024-04-26T13:48:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7066.792935 secs 2024-04-26T13:48:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7069.549248 secs 2024-04-26T13:48:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:49 smithi033 ceph-mon[27845]: pgmap v4391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:49 smithi096 ceph-mon[32753]: pgmap v4391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:49 smithi194 ceph-mon[33017]: pgmap v4391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:51 smithi033 ceph-mon[27845]: pgmap v4392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:51 smithi096 ceph-mon[32753]: pgmap v4392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:51 smithi194 ceph-mon[33017]: pgmap v4392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:48:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:52.914+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:52.914+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:48:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:48:52.914+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:48:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:53 smithi033 ceph-mon[27845]: pgmap v4393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7071.793004 secs 2024-04-26T13:48:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7074.549364 secs 2024-04-26T13:48:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:53 smithi096 ceph-mon[32753]: pgmap v4393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7071.793004 secs 2024-04-26T13:48:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7074.549364 secs 2024-04-26T13:48:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:53 smithi194 ceph-mon[33017]: pgmap v4393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7071.793004 secs 2024-04-26T13:48:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7074.549364 secs 2024-04-26T13:48:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:55 smithi033 ceph-mon[27845]: pgmap v4394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:48:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:55 smithi096 ceph-mon[32753]: pgmap v4394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:48:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:55 smithi194 ceph-mon[33017]: pgmap v4394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:48:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:48:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:48:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:48:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:48:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:48:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:48:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:48:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:48:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:48:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:48:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:57 smithi096 ceph-mon[32753]: pgmap v4395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:48:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:57 smithi096 ceph-mon[32753]: pgmap v4396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:57 smithi194 ceph-mon[33017]: pgmap v4395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:48:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:57 smithi194 ceph-mon[33017]: pgmap v4396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:57 smithi033 ceph-mon[27845]: pgmap v4395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:48:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:57 smithi033 ceph-mon[27845]: pgmap v4396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7076.793059 secs 2024-04-26T13:48:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7079.549484 secs 2024-04-26T13:48:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7076.793059 secs 2024-04-26T13:48:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7079.549484 secs 2024-04-26T13:48:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7076.793059 secs 2024-04-26T13:48:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7079.549484 secs 2024-04-26T13:48:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:48:59 smithi194 ceph-mon[33017]: pgmap v4397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:48:59 smithi033 ceph-mon[27845]: pgmap v4397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:48:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:48:59 smithi096 ceph-mon[32753]: pgmap v4397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:01 smithi194 ceph-mon[33017]: pgmap v4398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:01 smithi033 ceph-mon[27845]: pgmap v4398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:01 smithi096 ceph-mon[32753]: pgmap v4398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:03 smithi194 ceph-mon[33017]: pgmap v4399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7081.793120 secs 2024-04-26T13:49:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7084.549599 secs 2024-04-26T13:49:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:03 smithi033 ceph-mon[27845]: pgmap v4399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7081.793120 secs 2024-04-26T13:49:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7084.549599 secs 2024-04-26T13:49:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:03 smithi096 ceph-mon[32753]: pgmap v4399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7081.793120 secs 2024-04-26T13:49:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7084.549599 secs 2024-04-26T13:49:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:05 smithi194 ceph-mon[33017]: pgmap v4400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:05 smithi033 ceph-mon[27845]: pgmap v4400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:05 smithi096 ceph-mon[32753]: pgmap v4400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:49:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:07 smithi194 ceph-mon[33017]: pgmap v4401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:49:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:07 smithi033 ceph-mon[27845]: pgmap v4401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:49:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:07.929+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:07.929+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:07.929+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:49:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:07 smithi096 ceph-mon[32753]: pgmap v4401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:49:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7086.793204 secs 2024-04-26T13:49:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7089.549695 secs 2024-04-26T13:49:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7086.793204 secs 2024-04-26T13:49:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7089.549695 secs 2024-04-26T13:49:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7086.793204 secs 2024-04-26T13:49:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7089.549695 secs 2024-04-26T13:49:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:09 smithi194 ceph-mon[33017]: pgmap v4402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:09 smithi033 ceph-mon[27845]: pgmap v4402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:09 smithi096 ceph-mon[32753]: pgmap v4402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:11 smithi194 ceph-mon[33017]: pgmap v4403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:11.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:11 smithi033 ceph-mon[27845]: pgmap v4403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:11 smithi096 ceph-mon[32753]: pgmap v4403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:13 smithi194 ceph-mon[33017]: pgmap v4404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7091.793286 secs 2024-04-26T13:49:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7094.549814 secs 2024-04-26T13:49:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:13 smithi033 ceph-mon[27845]: pgmap v4404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7091.793286 secs 2024-04-26T13:49:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7094.549814 secs 2024-04-26T13:49:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:13 smithi096 ceph-mon[32753]: pgmap v4404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7091.793286 secs 2024-04-26T13:49:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7094.549814 secs 2024-04-26T13:49:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:15 smithi194 ceph-mon[33017]: pgmap v4405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:15 smithi033 ceph-mon[27845]: pgmap v4405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:15 smithi096 ceph-mon[32753]: pgmap v4405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:17 smithi194 ceph-mon[33017]: pgmap v4406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:17 smithi033 ceph-mon[27845]: pgmap v4406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:17 smithi096 ceph-mon[32753]: pgmap v4406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7096.793348 secs 2024-04-26T13:49:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7099.549955 secs 2024-04-26T13:49:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7096.793348 secs 2024-04-26T13:49:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7099.549955 secs 2024-04-26T13:49:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7096.793348 secs 2024-04-26T13:49:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7099.549955 secs 2024-04-26T13:49:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:19 smithi194 ceph-mon[33017]: pgmap v4407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:19 smithi033 ceph-mon[27845]: pgmap v4407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:19 smithi096 ceph-mon[32753]: pgmap v4407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:21 smithi194 ceph-mon[33017]: pgmap v4408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:21 smithi033 ceph-mon[27845]: pgmap v4408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:21 smithi096 ceph-mon[32753]: pgmap v4408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:22.943+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:22.943+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:22.943+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:49:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:23 smithi194 ceph-mon[33017]: pgmap v4409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7101.793439 secs 2024-04-26T13:49:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7104.550061 secs 2024-04-26T13:49:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:23 smithi033 ceph-mon[27845]: pgmap v4409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7101.793439 secs 2024-04-26T13:49:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7104.550061 secs 2024-04-26T13:49:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:23 smithi096 ceph-mon[32753]: pgmap v4409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7101.793439 secs 2024-04-26T13:49:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7104.550061 secs 2024-04-26T13:49:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:25 smithi194 ceph-mon[33017]: pgmap v4410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:25 smithi033 ceph-mon[27845]: pgmap v4410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:25 smithi096 ceph-mon[32753]: pgmap v4410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:27 smithi194 ceph-mon[33017]: pgmap v4411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:27 smithi033 ceph-mon[27845]: pgmap v4411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:27 smithi096 ceph-mon[32753]: pgmap v4411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7106.793549 secs 2024-04-26T13:49:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7109.550188 secs 2024-04-26T13:49:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7106.793549 secs 2024-04-26T13:49:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7109.550188 secs 2024-04-26T13:49:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7106.793549 secs 2024-04-26T13:49:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7109.550188 secs 2024-04-26T13:49:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:29 smithi194 ceph-mon[33017]: pgmap v4412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:29 smithi033 ceph-mon[27845]: pgmap v4412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:29 smithi096 ceph-mon[32753]: pgmap v4412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:31 smithi194 ceph-mon[33017]: pgmap v4413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:31 smithi033 ceph-mon[27845]: pgmap v4413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:31 smithi096 ceph-mon[32753]: pgmap v4413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:33 smithi194 ceph-mon[33017]: pgmap v4414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7111.793637 secs 2024-04-26T13:49:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7114.550305 secs 2024-04-26T13:49:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:33 smithi033 ceph-mon[27845]: pgmap v4414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7111.793637 secs 2024-04-26T13:49:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7114.550305 secs 2024-04-26T13:49:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:33 smithi096 ceph-mon[32753]: pgmap v4414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7111.793637 secs 2024-04-26T13:49:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7114.550305 secs 2024-04-26T13:49:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:35 smithi194 ceph-mon[33017]: pgmap v4415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:35 smithi033 ceph-mon[27845]: pgmap v4415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:35 smithi096 ceph-mon[32753]: pgmap v4415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:36 smithi194 ceph-mon[33017]: pgmap v4416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:36 smithi033 ceph-mon[27845]: pgmap v4416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:36 smithi096 ceph-mon[32753]: pgmap v4416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:37.959+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:37.959+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:37.959+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:49:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7116.793737 secs 2024-04-26T13:49:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:38 smithi033 ceph-mon[27845]: pgmap v4417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7119.550458 secs 2024-04-26T13:49:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7116.793737 secs 2024-04-26T13:49:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:38 smithi096 ceph-mon[32753]: pgmap v4417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7119.550458 secs 2024-04-26T13:49:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7116.793737 secs 2024-04-26T13:49:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:38 smithi194 ceph-mon[33017]: pgmap v4417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7119.550458 secs 2024-04-26T13:49:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:40 smithi033 ceph-mon[27845]: pgmap v4418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:40 smithi096 ceph-mon[32753]: pgmap v4418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:40 smithi194 ceph-mon[33017]: pgmap v4418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:42 smithi033 ceph-mon[27845]: pgmap v4419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:42 smithi096 ceph-mon[32753]: pgmap v4419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:42 smithi194 ceph-mon[33017]: pgmap v4419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7121.793838 secs 2024-04-26T13:49:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7124.550585 secs 2024-04-26T13:49:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7121.793838 secs 2024-04-26T13:49:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7124.550585 secs 2024-04-26T13:49:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7121.793838 secs 2024-04-26T13:49:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7124.550585 secs 2024-04-26T13:49:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:44 smithi033 ceph-mon[27845]: pgmap v4420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:44 smithi096 ceph-mon[32753]: pgmap v4420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:44 smithi194 ceph-mon[33017]: pgmap v4420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:47.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:46 smithi033 ceph-mon[27845]: pgmap v4421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:46 smithi096 ceph-mon[32753]: pgmap v4421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:46 smithi194 ceph-mon[33017]: pgmap v4421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7126.793932 secs 2024-04-26T13:49:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:48 smithi096 ceph-mon[32753]: pgmap v4422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7129.550714 secs 2024-04-26T13:49:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7126.793932 secs 2024-04-26T13:49:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:48 smithi194 ceph-mon[33017]: pgmap v4422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7129.550714 secs 2024-04-26T13:49:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7126.793932 secs 2024-04-26T13:49:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:48 smithi033 ceph-mon[27845]: pgmap v4422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7129.550714 secs 2024-04-26T13:49:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:50 smithi194 ceph-mon[33017]: pgmap v4423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:50 smithi033 ceph-mon[27845]: pgmap v4423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:50 smithi096 ceph-mon[32753]: pgmap v4423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:52.975+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:52.975+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:49:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:49:52.975+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:49:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:52 smithi194 ceph-mon[33017]: pgmap v4424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:52 smithi033 ceph-mon[27845]: pgmap v4424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:52 smithi096 ceph-mon[32753]: pgmap v4424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7131.794028 secs 2024-04-26T13:49:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7134.550862 secs 2024-04-26T13:49:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7131.794028 secs 2024-04-26T13:49:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7134.550862 secs 2024-04-26T13:49:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7131.794028 secs 2024-04-26T13:49:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7134.550862 secs 2024-04-26T13:49:55.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:54 smithi096 ceph-mon[32753]: pgmap v4425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:54 smithi194 ceph-mon[33017]: pgmap v4425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:54 smithi033 ceph-mon[27845]: pgmap v4425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:56 smithi194 ceph-mon[33017]: pgmap v4426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:49:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:56 smithi033 ceph-mon[27845]: pgmap v4426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:57.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:49:57.513 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:56 smithi096 ceph-mon[32753]: pgmap v4426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:49:57.513 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:49:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:49:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:49:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:57 smithi194 ceph-mon[33017]: pgmap v4427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:49:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:49:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:49:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:57 smithi033 ceph-mon[27845]: pgmap v4427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:49:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:49:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:49:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:57 smithi096 ceph-mon[32753]: pgmap v4427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:49:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:49:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7136.794115 secs 2024-04-26T13:49:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7139.551001 secs 2024-04-26T13:49:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7136.794115 secs 2024-04-26T13:49:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7139.551001 secs 2024-04-26T13:49:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7136.794115 secs 2024-04-26T13:49:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7139.551001 secs 2024-04-26T13:50:00.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:49:59 smithi033 ceph-mon[27845]: pgmap v4428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:50:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:49:59 smithi194 ceph-mon[33017]: pgmap v4428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:50:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:49:59 smithi096 ceph-mon[32753]: pgmap v4428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:50:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:50:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:50:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T13:50:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:01 smithi194 ceph-mon[33017]: pgmap v4429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:01 smithi033 ceph-mon[27845]: pgmap v4429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:01 smithi096 ceph-mon[32753]: pgmap v4429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7144.551139 secs 2024-04-26T13:50:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7144.551139 secs 2024-04-26T13:50:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7144.551139 secs 2024-04-26T13:50:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7141.794201 secs 2024-04-26T13:50:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:04 smithi194 ceph-mon[33017]: pgmap v4430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7141.794201 secs 2024-04-26T13:50:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:04 smithi033 ceph-mon[27845]: pgmap v4430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7141.794201 secs 2024-04-26T13:50:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:04 smithi096 ceph-mon[32753]: pgmap v4430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:06 smithi194 ceph-mon[33017]: pgmap v4431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:50:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:06 smithi033 ceph-mon[27845]: pgmap v4431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:50:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:06 smithi096 ceph-mon[32753]: pgmap v4431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:50:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:07.982+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:07.982+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:50:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:07.982+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:08 smithi194 ceph-mon[33017]: pgmap v4432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7146.794261 secs 2024-04-26T13:50:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7149.551266 secs 2024-04-26T13:50:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:08 smithi033 ceph-mon[27845]: pgmap v4432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7146.794261 secs 2024-04-26T13:50:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7149.551266 secs 2024-04-26T13:50:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:08 smithi096 ceph-mon[32753]: pgmap v4432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7146.794261 secs 2024-04-26T13:50:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7149.551266 secs 2024-04-26T13:50:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:10 smithi194 ceph-mon[33017]: pgmap v4433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:10.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:10 smithi033 ceph-mon[27845]: pgmap v4433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:10 smithi096 ceph-mon[32753]: pgmap v4433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:12 smithi194 ceph-mon[33017]: pgmap v4434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:12 smithi033 ceph-mon[27845]: pgmap v4434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:12 smithi096 ceph-mon[32753]: pgmap v4434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7151.794350 secs 2024-04-26T13:50:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7154.551398 secs 2024-04-26T13:50:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7151.794350 secs 2024-04-26T13:50:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7154.551398 secs 2024-04-26T13:50:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7151.794350 secs 2024-04-26T13:50:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7154.551398 secs 2024-04-26T13:50:14.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:14 smithi033 ceph-mon[27845]: pgmap v4435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:14 smithi194 ceph-mon[33017]: pgmap v4435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:14 smithi096 ceph-mon[32753]: pgmap v4435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:16 smithi194 ceph-mon[33017]: pgmap v4436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:16.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:16 smithi033 ceph-mon[27845]: pgmap v4436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:16 smithi096 ceph-mon[32753]: pgmap v4436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:18 smithi194 ceph-mon[33017]: pgmap v4437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7156.794413 secs 2024-04-26T13:50:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7159.551517 secs 2024-04-26T13:50:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:18 smithi033 ceph-mon[27845]: pgmap v4437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7156.794413 secs 2024-04-26T13:50:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7159.551517 secs 2024-04-26T13:50:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:18 smithi096 ceph-mon[32753]: pgmap v4437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7156.794413 secs 2024-04-26T13:50:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7159.551517 secs 2024-04-26T13:50:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:20 smithi194 ceph-mon[33017]: pgmap v4438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:20 smithi033 ceph-mon[27845]: pgmap v4438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:20 smithi096 ceph-mon[32753]: pgmap v4438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:22 smithi194 ceph-mon[33017]: pgmap v4439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:22 smithi033 ceph-mon[27845]: pgmap v4439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:22 smithi096 ceph-mon[32753]: pgmap v4439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:22.989+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:22.989+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:22.989+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:50:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:23.006+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T13:50:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:23.006+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:50:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:23.007+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T13:50:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:23.007+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:50:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7161.794510 secs 2024-04-26T13:50:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7164.551640 secs 2024-04-26T13:50:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7161.794510 secs 2024-04-26T13:50:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7164.551640 secs 2024-04-26T13:50:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7161.794510 secs 2024-04-26T13:50:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7164.551640 secs 2024-04-26T13:50:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:24 smithi194 ceph-mon[33017]: pgmap v4440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:24 smithi033 ceph-mon[27845]: pgmap v4440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:24 smithi096 ceph-mon[32753]: pgmap v4440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:25 smithi194 ceph-mon[33017]: pgmap v4441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:25 smithi033 ceph-mon[27845]: pgmap v4441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:25 smithi096 ceph-mon[32753]: pgmap v4441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:28 smithi194 ceph-mon[33017]: pgmap v4442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7166.794609 secs 2024-04-26T13:50:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7169.551736 secs 2024-04-26T13:50:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:28 smithi033 ceph-mon[27845]: pgmap v4442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7166.794609 secs 2024-04-26T13:50:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7169.551736 secs 2024-04-26T13:50:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:28 smithi096 ceph-mon[32753]: pgmap v4442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7166.794609 secs 2024-04-26T13:50:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7169.551736 secs 2024-04-26T13:50:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:30 smithi194 ceph-mon[33017]: pgmap v4443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:30 smithi033 ceph-mon[27845]: pgmap v4443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:30 smithi096 ceph-mon[32753]: pgmap v4443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:32 smithi194 ceph-mon[33017]: pgmap v4444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:32 smithi033 ceph-mon[27845]: pgmap v4444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:32 smithi096 ceph-mon[32753]: pgmap v4444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7171.794670 secs 2024-04-26T13:50:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7174.551857 secs 2024-04-26T13:50:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7171.794670 secs 2024-04-26T13:50:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7174.551857 secs 2024-04-26T13:50:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7171.794670 secs 2024-04-26T13:50:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7174.551857 secs 2024-04-26T13:50:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:34 smithi194 ceph-mon[33017]: pgmap v4445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:34 smithi033 ceph-mon[27845]: pgmap v4445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:34 smithi096 ceph-mon[32753]: pgmap v4445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:36 smithi194 ceph-mon[33017]: pgmap v4446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:36 smithi033 ceph-mon[27845]: pgmap v4446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:36 smithi096 ceph-mon[32753]: pgmap v4446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:38.001+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:38.001+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:38.001+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:50:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:38 smithi194 ceph-mon[33017]: pgmap v4447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7176.794771 secs 2024-04-26T13:50:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7179.551976 secs 2024-04-26T13:50:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:38 smithi033 ceph-mon[27845]: pgmap v4447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7176.794771 secs 2024-04-26T13:50:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7179.551976 secs 2024-04-26T13:50:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:38 smithi096 ceph-mon[32753]: pgmap v4447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7176.794771 secs 2024-04-26T13:50:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7179.551976 secs 2024-04-26T13:50:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:40 smithi194 ceph-mon[33017]: pgmap v4448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:40 smithi033 ceph-mon[27845]: pgmap v4448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:40 smithi096 ceph-mon[32753]: pgmap v4448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:42 smithi194 ceph-mon[33017]: pgmap v4449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:42 smithi033 ceph-mon[27845]: pgmap v4449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:42 smithi096 ceph-mon[32753]: pgmap v4449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7181.794869 secs 2024-04-26T13:50:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7184.552091 secs 2024-04-26T13:50:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:43 smithi194 ceph-mon[33017]: pgmap v4450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7181.794869 secs 2024-04-26T13:50:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7184.552091 secs 2024-04-26T13:50:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:43 smithi033 ceph-mon[27845]: pgmap v4450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7181.794869 secs 2024-04-26T13:50:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7184.552091 secs 2024-04-26T13:50:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:43 smithi096 ceph-mon[32753]: pgmap v4450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:46 smithi194 ceph-mon[33017]: pgmap v4451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:46 smithi033 ceph-mon[27845]: pgmap v4451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:46 smithi096 ceph-mon[32753]: pgmap v4451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:48 smithi194 ceph-mon[33017]: pgmap v4452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7186.794976 secs 2024-04-26T13:50:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7189.552211 secs 2024-04-26T13:50:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:48 smithi033 ceph-mon[27845]: pgmap v4452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7186.794976 secs 2024-04-26T13:50:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7189.552211 secs 2024-04-26T13:50:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:48 smithi096 ceph-mon[32753]: pgmap v4452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7186.794976 secs 2024-04-26T13:50:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7189.552211 secs 2024-04-26T13:50:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:50 smithi194 ceph-mon[33017]: pgmap v4453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:50 smithi033 ceph-mon[27845]: pgmap v4453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:50 smithi096 ceph-mon[32753]: pgmap v4453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:52 smithi194 ceph-mon[33017]: pgmap v4454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:52.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:52 smithi033 ceph-mon[27845]: pgmap v4454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:52 smithi096 ceph-mon[32753]: pgmap v4454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:53.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:53.017+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:53.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:53.017+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:50:53.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:50:53.017+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:50:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7191.795036 secs 2024-04-26T13:50:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7194.552336 secs 2024-04-26T13:50:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7191.795036 secs 2024-04-26T13:50:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7194.552336 secs 2024-04-26T13:50:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7191.795036 secs 2024-04-26T13:50:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7194.552336 secs 2024-04-26T13:50:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:54 smithi194 ceph-mon[33017]: pgmap v4455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:54 smithi033 ceph-mon[27845]: pgmap v4455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:54 smithi096 ceph-mon[32753]: pgmap v4455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:56 smithi194 ceph-mon[33017]: pgmap v4456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:56.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:56 smithi033 ceph-mon[27845]: pgmap v4456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:56 smithi096 ceph-mon[32753]: pgmap v4456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:50:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:58 smithi194 ceph-mon[33017]: pgmap v4457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:58.958 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7196.795135 secs 2024-04-26T13:50:58.959 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:50:58.959 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7199.552431 secs 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:58 smithi096 ceph-mon[32753]: pgmap v4457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7196.795135 secs 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7199.552431 secs 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:58 smithi033 ceph-mon[27845]: pgmap v4457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7196.795135 secs 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:50:58.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7199.552431 secs 2024-04-26T13:50:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:50:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:50:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:50:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:50:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:50:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:50:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:50:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:50:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:50:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:50:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:50:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:51:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:00 smithi194 ceph-mon[33017]: pgmap v4458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:00 smithi194 ceph-mon[33017]: pgmap v4459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:51:00.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:00 smithi033 ceph-mon[27845]: pgmap v4458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:00.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:00 smithi033 ceph-mon[27845]: pgmap v4459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:51:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:00 smithi096 ceph-mon[32753]: pgmap v4458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:00 smithi096 ceph-mon[32753]: pgmap v4459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T13:51:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:01 smithi194 ceph-mon[33017]: pgmap v4460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:01 smithi033 ceph-mon[27845]: pgmap v4460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:01 smithi096 ceph-mon[32753]: pgmap v4460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7204.552543 secs 2024-04-26T13:51:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7204.552543 secs 2024-04-26T13:51:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7204.552543 secs 2024-04-26T13:51:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7201.795196 secs 2024-04-26T13:51:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:04 smithi194 ceph-mon[33017]: pgmap v4461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7201.795196 secs 2024-04-26T13:51:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:04 smithi033 ceph-mon[27845]: pgmap v4461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7201.795196 secs 2024-04-26T13:51:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:04 smithi096 ceph-mon[32753]: pgmap v4461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:06 smithi194 ceph-mon[33017]: pgmap v4462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:06 smithi033 ceph-mon[27845]: pgmap v4462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:06 smithi096 ceph-mon[32753]: pgmap v4462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:08.024+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:08.024+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:08.024+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:51:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:08 smithi194 ceph-mon[33017]: pgmap v4463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7206.795273 secs 2024-04-26T13:51:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7209.552660 secs 2024-04-26T13:51:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:08 smithi033 ceph-mon[27845]: pgmap v4463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7206.795273 secs 2024-04-26T13:51:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7209.552660 secs 2024-04-26T13:51:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:08 smithi096 ceph-mon[32753]: pgmap v4463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:51:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7206.795273 secs 2024-04-26T13:51:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7209.552660 secs 2024-04-26T13:51:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:10 smithi194 ceph-mon[33017]: pgmap v4464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:51:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:10 smithi033 ceph-mon[27845]: pgmap v4464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:51:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:10 smithi096 ceph-mon[32753]: pgmap v4464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:51:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:12 smithi194 ceph-mon[33017]: pgmap v4465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:12 smithi033 ceph-mon[27845]: pgmap v4465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:12 smithi096 ceph-mon[32753]: pgmap v4465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7211.795350 secs 2024-04-26T13:51:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7214.552776 secs 2024-04-26T13:51:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7211.795350 secs 2024-04-26T13:51:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7214.552776 secs 2024-04-26T13:51:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7211.795350 secs 2024-04-26T13:51:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7214.552776 secs 2024-04-26T13:51:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:14 smithi194 ceph-mon[33017]: pgmap v4466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:14 smithi033 ceph-mon[27845]: pgmap v4466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:14 smithi096 ceph-mon[32753]: pgmap v4466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:16 smithi033 ceph-mon[27845]: pgmap v4467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:16 smithi194 ceph-mon[33017]: pgmap v4467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:16 smithi096 ceph-mon[32753]: pgmap v4467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:18 smithi194 ceph-mon[33017]: pgmap v4468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7216.795435 secs 2024-04-26T13:51:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7219.552901 secs 2024-04-26T13:51:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:18 smithi033 ceph-mon[27845]: pgmap v4468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7216.795435 secs 2024-04-26T13:51:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7219.552901 secs 2024-04-26T13:51:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:18 smithi096 ceph-mon[32753]: pgmap v4468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7216.795435 secs 2024-04-26T13:51:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7219.552901 secs 2024-04-26T13:51:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:20 smithi194 ceph-mon[33017]: pgmap v4469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:20 smithi033 ceph-mon[27845]: pgmap v4469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:20 smithi096 ceph-mon[32753]: pgmap v4469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:22 smithi194 ceph-mon[33017]: pgmap v4470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:22 smithi033 ceph-mon[27845]: pgmap v4470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:22 smithi096 ceph-mon[32753]: pgmap v4470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:23.027+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:23.027+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:23.027+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:51:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7221.795518 secs 2024-04-26T13:51:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7224.553023 secs 2024-04-26T13:51:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7221.795518 secs 2024-04-26T13:51:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7224.553023 secs 2024-04-26T13:51:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7221.795518 secs 2024-04-26T13:51:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7224.553023 secs 2024-04-26T13:51:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:24 smithi194 ceph-mon[33017]: pgmap v4471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:24 smithi033 ceph-mon[27845]: pgmap v4471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:24 smithi096 ceph-mon[32753]: pgmap v4471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:26.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:26 smithi096 ceph-mon[32753]: pgmap v4472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:26 smithi194 ceph-mon[33017]: pgmap v4472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:26 smithi033 ceph-mon[27845]: pgmap v4472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:28 smithi194 ceph-mon[33017]: pgmap v4473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7226.795601 secs 2024-04-26T13:51:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7229.553174 secs 2024-04-26T13:51:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:28 smithi033 ceph-mon[27845]: pgmap v4473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7226.795601 secs 2024-04-26T13:51:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7229.553174 secs 2024-04-26T13:51:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:28 smithi096 ceph-mon[32753]: pgmap v4473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7226.795601 secs 2024-04-26T13:51:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7229.553174 secs 2024-04-26T13:51:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:30 smithi194 ceph-mon[33017]: pgmap v4474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:30.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:30 smithi033 ceph-mon[27845]: pgmap v4474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:30 smithi096 ceph-mon[32753]: pgmap v4474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:32 smithi194 ceph-mon[33017]: pgmap v4475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:32 smithi033 ceph-mon[27845]: pgmap v4475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:32 smithi096 ceph-mon[32753]: pgmap v4475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7231.795684 secs 2024-04-26T13:51:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7234.553291 secs 2024-04-26T13:51:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7231.795684 secs 2024-04-26T13:51:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7234.553291 secs 2024-04-26T13:51:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7231.795684 secs 2024-04-26T13:51:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7234.553291 secs 2024-04-26T13:51:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:34 smithi194 ceph-mon[33017]: pgmap v4476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:34 smithi033 ceph-mon[27845]: pgmap v4476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:34 smithi096 ceph-mon[32753]: pgmap v4476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:36 smithi194 ceph-mon[33017]: pgmap v4477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:36 smithi033 ceph-mon[27845]: pgmap v4477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:36 smithi096 ceph-mon[32753]: pgmap v4477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:38.039+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:38.039+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:38.039+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:51:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:38 smithi194 ceph-mon[33017]: pgmap v4478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7236.795770 secs 2024-04-26T13:51:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7239.553402 secs 2024-04-26T13:51:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:38 smithi033 ceph-mon[27845]: pgmap v4478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7236.795770 secs 2024-04-26T13:51:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7239.553402 secs 2024-04-26T13:51:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:38 smithi096 ceph-mon[32753]: pgmap v4478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7236.795770 secs 2024-04-26T13:51:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7239.553402 secs 2024-04-26T13:51:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:40 smithi194 ceph-mon[33017]: pgmap v4479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:40 smithi033 ceph-mon[27845]: pgmap v4479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:40 smithi096 ceph-mon[32753]: pgmap v4479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:42 smithi194 ceph-mon[33017]: pgmap v4480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:42 smithi033 ceph-mon[27845]: pgmap v4480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:42 smithi096 ceph-mon[32753]: pgmap v4480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7241.795853 secs 2024-04-26T13:51:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7244.553523 secs 2024-04-26T13:51:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7241.795853 secs 2024-04-26T13:51:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7244.553523 secs 2024-04-26T13:51:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7241.795853 secs 2024-04-26T13:51:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7244.553523 secs 2024-04-26T13:51:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:44 smithi194 ceph-mon[33017]: pgmap v4481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:44 smithi033 ceph-mon[27845]: pgmap v4481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:44 smithi096 ceph-mon[32753]: pgmap v4481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:46 smithi194 ceph-mon[33017]: pgmap v4482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:46.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:46 smithi033 ceph-mon[27845]: pgmap v4482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:46 smithi096 ceph-mon[32753]: pgmap v4482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:48 smithi194 ceph-mon[33017]: pgmap v4483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7246.795937 secs 2024-04-26T13:51:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7249.553655 secs 2024-04-26T13:51:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:48 smithi033 ceph-mon[27845]: pgmap v4483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7246.795937 secs 2024-04-26T13:51:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7249.553655 secs 2024-04-26T13:51:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:48 smithi096 ceph-mon[32753]: pgmap v4483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7246.795937 secs 2024-04-26T13:51:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7249.553655 secs 2024-04-26T13:51:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:50 smithi194 ceph-mon[33017]: pgmap v4484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:50 smithi033 ceph-mon[27845]: pgmap v4484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:50 smithi096 ceph-mon[32753]: pgmap v4484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:52 smithi194 ceph-mon[33017]: pgmap v4485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:52 smithi033 ceph-mon[27845]: pgmap v4485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:52 smithi096 ceph-mon[32753]: pgmap v4485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:53.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:53.050+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:53.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:53.050+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:51:53.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:51:53.050+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:51:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7251.796020 secs 2024-04-26T13:51:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7254.553774 secs 2024-04-26T13:51:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7251.796020 secs 2024-04-26T13:51:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7254.553774 secs 2024-04-26T13:51:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7251.796020 secs 2024-04-26T13:51:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7254.553774 secs 2024-04-26T13:51:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:54 smithi194 ceph-mon[33017]: pgmap v4486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:54 smithi033 ceph-mon[27845]: pgmap v4486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:54 smithi096 ceph-mon[32753]: pgmap v4486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:55 smithi033 ceph-mon[27845]: pgmap v4487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:55 smithi096 ceph-mon[32753]: pgmap v4487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:55 smithi194 ceph-mon[33017]: pgmap v4487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:51:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:58 smithi194 ceph-mon[33017]: pgmap v4488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7256.796106 secs 2024-04-26T13:51:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:51:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7259.553893 secs 2024-04-26T13:51:58.634 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:58 smithi096 ceph-mon[32753]: pgmap v4488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:58.634 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7256.796106 secs 2024-04-26T13:51:58.634 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:51:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7259.553893 secs 2024-04-26T13:51:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:58 smithi033 ceph-mon[27845]: pgmap v4488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:51:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7256.796106 secs 2024-04-26T13:51:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:51:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7259.553893 secs 2024-04-26T13:52:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:00 smithi194 ceph-mon[33017]: pgmap v4489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:52:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:00 smithi033 ceph-mon[27845]: pgmap v4489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:52:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:00 smithi096 ceph-mon[32753]: pgmap v4489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:52:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:52:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:52:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:52:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:52:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:52:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:52:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:52:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:52:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:52:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:02 smithi194 ceph-mon[33017]: pgmap v4490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:52:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:02 smithi194 ceph-mon[33017]: pgmap v4491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:02 smithi033 ceph-mon[27845]: pgmap v4490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:52:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:02 smithi033 ceph-mon[27845]: pgmap v4491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:02 smithi096 ceph-mon[32753]: pgmap v4490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T13:52:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:02 smithi096 ceph-mon[32753]: pgmap v4491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7261.796189 secs 2024-04-26T13:52:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7264.554012 secs 2024-04-26T13:52:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7261.796189 secs 2024-04-26T13:52:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7264.554012 secs 2024-04-26T13:52:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7261.796189 secs 2024-04-26T13:52:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7264.554012 secs 2024-04-26T13:52:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:04 smithi194 ceph-mon[33017]: pgmap v4492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:04 smithi033 ceph-mon[27845]: pgmap v4492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:04 smithi096 ceph-mon[32753]: pgmap v4492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:06 smithi194 ceph-mon[33017]: pgmap v4493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:06 smithi033 ceph-mon[27845]: pgmap v4493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:06 smithi096 ceph-mon[32753]: pgmap v4493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:08.066+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:08.066+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:08.066+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:52:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:08 smithi194 ceph-mon[33017]: pgmap v4494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7266.796289 secs 2024-04-26T13:52:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7269.554102 secs 2024-04-26T13:52:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:08 smithi033 ceph-mon[27845]: pgmap v4494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7266.796289 secs 2024-04-26T13:52:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7269.554102 secs 2024-04-26T13:52:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:08 smithi096 ceph-mon[32753]: pgmap v4494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7266.796289 secs 2024-04-26T13:52:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7269.554102 secs 2024-04-26T13:52:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:10 smithi194 ceph-mon[33017]: pgmap v4495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:10 smithi033 ceph-mon[27845]: pgmap v4495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:10 smithi096 ceph-mon[32753]: pgmap v4495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:52:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:12 smithi194 ceph-mon[33017]: pgmap v4496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:52:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:12 smithi033 ceph-mon[27845]: pgmap v4496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:52:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:12 smithi096 ceph-mon[32753]: pgmap v4496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:52:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7271.796374 secs 2024-04-26T13:52:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7274.554219 secs 2024-04-26T13:52:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7271.796374 secs 2024-04-26T13:52:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7274.554219 secs 2024-04-26T13:52:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7271.796374 secs 2024-04-26T13:52:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7274.554219 secs 2024-04-26T13:52:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:14 smithi194 ceph-mon[33017]: pgmap v4497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:14 smithi033 ceph-mon[27845]: pgmap v4497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:14 smithi096 ceph-mon[32753]: pgmap v4497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:16 smithi194 ceph-mon[33017]: pgmap v4498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:16 smithi033 ceph-mon[27845]: pgmap v4498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:16 smithi096 ceph-mon[32753]: pgmap v4498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:18.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:18 smithi033 ceph-mon[27845]: pgmap v4499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:18.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7276.796459 secs 2024-04-26T13:52:18.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7279.554340 secs 2024-04-26T13:52:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:18 smithi096 ceph-mon[32753]: pgmap v4499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7276.796459 secs 2024-04-26T13:52:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7279.554340 secs 2024-04-26T13:52:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:18 smithi194 ceph-mon[33017]: pgmap v4499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7276.796459 secs 2024-04-26T13:52:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7279.554340 secs 2024-04-26T13:52:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:20 smithi033 ceph-mon[27845]: pgmap v4500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:20 smithi096 ceph-mon[32753]: pgmap v4500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:20 smithi194 ceph-mon[33017]: pgmap v4500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:22.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:22 smithi194 ceph-mon[33017]: pgmap v4501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:22 smithi033 ceph-mon[27845]: pgmap v4501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:22 smithi096 ceph-mon[32753]: pgmap v4501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:23.074+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:23.074+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:23.074+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:52:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7281.796540 secs 2024-04-26T13:52:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7284.554462 secs 2024-04-26T13:52:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7281.796540 secs 2024-04-26T13:52:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7284.554462 secs 2024-04-26T13:52:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7281.796540 secs 2024-04-26T13:52:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7284.554462 secs 2024-04-26T13:52:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:24 smithi033 ceph-mon[27845]: pgmap v4502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:24 smithi096 ceph-mon[32753]: pgmap v4502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:24 smithi194 ceph-mon[33017]: pgmap v4502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:26 smithi033 ceph-mon[27845]: pgmap v4503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:26 smithi096 ceph-mon[32753]: pgmap v4503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:26 smithi194 ceph-mon[33017]: pgmap v4503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:28 smithi033 ceph-mon[27845]: pgmap v4504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7286.796624 secs 2024-04-26T13:52:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7289.554601 secs 2024-04-26T13:52:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:28 smithi096 ceph-mon[32753]: pgmap v4504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7286.796624 secs 2024-04-26T13:52:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7289.554601 secs 2024-04-26T13:52:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:28 smithi194 ceph-mon[33017]: pgmap v4504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7286.796624 secs 2024-04-26T13:52:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7289.554601 secs 2024-04-26T13:52:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:30 smithi033 ceph-mon[27845]: pgmap v4505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:30 smithi096 ceph-mon[32753]: pgmap v4505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:30 smithi194 ceph-mon[33017]: pgmap v4505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:32 smithi033 ceph-mon[27845]: pgmap v4506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:32 smithi096 ceph-mon[32753]: pgmap v4506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:32 smithi194 ceph-mon[33017]: pgmap v4506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7291.796704 secs 2024-04-26T13:52:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7294.554716 secs 2024-04-26T13:52:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7291.796704 secs 2024-04-26T13:52:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7294.554716 secs 2024-04-26T13:52:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7291.796704 secs 2024-04-26T13:52:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7294.554716 secs 2024-04-26T13:52:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:34 smithi033 ceph-mon[27845]: pgmap v4507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:34 smithi096 ceph-mon[32753]: pgmap v4507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:34 smithi194 ceph-mon[33017]: pgmap v4507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:36 smithi033 ceph-mon[27845]: pgmap v4508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:36 smithi096 ceph-mon[32753]: pgmap v4508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:36 smithi194 ceph-mon[33017]: pgmap v4508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:38.088+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:38.088+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:52:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:38.088+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:38 smithi096 ceph-mon[32753]: pgmap v4509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7296.796788 secs 2024-04-26T13:52:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7299.554833 secs 2024-04-26T13:52:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:38 smithi194 ceph-mon[33017]: pgmap v4509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7296.796788 secs 2024-04-26T13:52:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7299.554833 secs 2024-04-26T13:52:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:38 smithi033 ceph-mon[27845]: pgmap v4509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7296.796788 secs 2024-04-26T13:52:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7299.554833 secs 2024-04-26T13:52:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:40 smithi096 ceph-mon[32753]: pgmap v4510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:40 smithi194 ceph-mon[33017]: pgmap v4510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:40 smithi033 ceph-mon[27845]: pgmap v4510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:42 smithi194 ceph-mon[33017]: pgmap v4511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:42 smithi033 ceph-mon[27845]: pgmap v4511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:42 smithi096 ceph-mon[32753]: pgmap v4511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7301.796882 secs 2024-04-26T13:52:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7304.554947 secs 2024-04-26T13:52:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7301.796882 secs 2024-04-26T13:52:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7304.554947 secs 2024-04-26T13:52:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7301.796882 secs 2024-04-26T13:52:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7304.554947 secs 2024-04-26T13:52:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:44 smithi194 ceph-mon[33017]: pgmap v4512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:44 smithi033 ceph-mon[27845]: pgmap v4512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:44 smithi096 ceph-mon[32753]: pgmap v4512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:46 smithi194 ceph-mon[33017]: pgmap v4513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:46 smithi033 ceph-mon[27845]: pgmap v4513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:46 smithi096 ceph-mon[32753]: pgmap v4513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:48.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:48 smithi033 ceph-mon[27845]: pgmap v4514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:48.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7306.796990 secs 2024-04-26T13:52:48.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7309.555065 secs 2024-04-26T13:52:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:48 smithi194 ceph-mon[33017]: pgmap v4514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7306.796990 secs 2024-04-26T13:52:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7309.555065 secs 2024-04-26T13:52:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:48 smithi096 ceph-mon[32753]: pgmap v4514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7306.796990 secs 2024-04-26T13:52:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7309.555065 secs 2024-04-26T13:52:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:50 smithi194 ceph-mon[33017]: pgmap v4515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:50 smithi033 ceph-mon[27845]: pgmap v4515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:50 smithi096 ceph-mon[32753]: pgmap v4515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:52 smithi194 ceph-mon[33017]: pgmap v4516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:52.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:52 smithi033 ceph-mon[27845]: pgmap v4516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:52 smithi096 ceph-mon[32753]: pgmap v4516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:53.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:53.090+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:53.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:53.090+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:52:53.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:52:53.090+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:52:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7311.797061 secs 2024-04-26T13:52:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7314.555130 secs 2024-04-26T13:52:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7311.797061 secs 2024-04-26T13:52:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7314.555130 secs 2024-04-26T13:52:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7311.797061 secs 2024-04-26T13:52:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7314.555130 secs 2024-04-26T13:52:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:54 smithi194 ceph-mon[33017]: pgmap v4517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:54 smithi033 ceph-mon[27845]: pgmap v4517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:54 smithi096 ceph-mon[32753]: pgmap v4517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:56 smithi194 ceph-mon[33017]: pgmap v4518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:56.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:56 smithi033 ceph-mon[27845]: pgmap v4518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:56 smithi096 ceph-mon[32753]: pgmap v4518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:52:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:58 smithi194 ceph-mon[33017]: pgmap v4519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7316.797120 secs 2024-04-26T13:52:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:52:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7319.555266 secs 2024-04-26T13:52:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:58 smithi033 ceph-mon[27845]: pgmap v4519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7316.797120 secs 2024-04-26T13:52:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:52:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7319.555266 secs 2024-04-26T13:52:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:58 smithi096 ceph-mon[32753]: pgmap v4519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:52:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7316.797120 secs 2024-04-26T13:52:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:52:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7319.555266 secs 2024-04-26T13:53:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:00 smithi194 ceph-mon[33017]: pgmap v4520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:00 smithi033 ceph-mon[27845]: pgmap v4520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:00 smithi096 ceph-mon[32753]: pgmap v4520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:53:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:53:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:53:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:02 smithi194 ceph-mon[33017]: pgmap v4521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:02.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:02 smithi033 ceph-mon[27845]: pgmap v4521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:02 smithi096 ceph-mon[32753]: pgmap v4521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7321.797179 secs 2024-04-26T13:53:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:53:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:53:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:03 smithi194 ceph-mon[33017]: pgmap v4522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:03 smithi194 ceph-mon[33017]: pgmap v4523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:53:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7324.555379 secs 2024-04-26T13:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7321.797179 secs 2024-04-26T13:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:03 smithi033 ceph-mon[27845]: pgmap v4522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:03 smithi033 ceph-mon[27845]: pgmap v4523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7324.555379 secs 2024-04-26T13:53:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7321.797179 secs 2024-04-26T13:53:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:53:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:53:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:03 smithi096 ceph-mon[32753]: pgmap v4522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:03 smithi096 ceph-mon[32753]: pgmap v4523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:53:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7324.555379 secs 2024-04-26T13:53:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:04 smithi033 ceph-mon[27845]: pgmap v4524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:04 smithi096 ceph-mon[32753]: pgmap v4524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:04 smithi194 ceph-mon[33017]: pgmap v4524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:06 smithi033 ceph-mon[27845]: pgmap v4525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:06 smithi096 ceph-mon[32753]: pgmap v4525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:06 smithi194 ceph-mon[33017]: pgmap v4525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:08.105+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:08.105+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:08.105+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:53:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7329.555499 secs 2024-04-26T13:53:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7329.555499 secs 2024-04-26T13:53:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7329.555499 secs 2024-04-26T13:53:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7326.797284 secs 2024-04-26T13:53:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:09 smithi194 ceph-mon[33017]: pgmap v4526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7326.797284 secs 2024-04-26T13:53:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:09 smithi033 ceph-mon[27845]: pgmap v4526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7326.797284 secs 2024-04-26T13:53:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:09 smithi096 ceph-mon[32753]: pgmap v4526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:10.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:10 smithi033 ceph-mon[27845]: pgmap v4527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:10 smithi096 ceph-mon[32753]: pgmap v4527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:10 smithi194 ceph-mon[33017]: pgmap v4527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:53:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:12 smithi033 ceph-mon[27845]: pgmap v4528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:53:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:12 smithi096 ceph-mon[32753]: pgmap v4528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:53:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:12 smithi194 ceph-mon[33017]: pgmap v4528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:53:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7331.797385 secs 2024-04-26T13:53:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7334.555609 secs 2024-04-26T13:53:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7331.797385 secs 2024-04-26T13:53:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7334.555609 secs 2024-04-26T13:53:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7331.797385 secs 2024-04-26T13:53:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7334.555609 secs 2024-04-26T13:53:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:16 smithi194 ceph-mon[33017]: pgmap v4529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:16.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:16 smithi033 ceph-mon[27845]: pgmap v4529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:16 smithi096 ceph-mon[32753]: pgmap v4529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:18 smithi194 ceph-mon[33017]: pgmap v4530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7336.797474 secs 2024-04-26T13:53:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7339.555725 secs 2024-04-26T13:53:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:18 smithi033 ceph-mon[27845]: pgmap v4530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7336.797474 secs 2024-04-26T13:53:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7339.555725 secs 2024-04-26T13:53:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:18 smithi096 ceph-mon[32753]: pgmap v4530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7336.797474 secs 2024-04-26T13:53:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7339.555725 secs 2024-04-26T13:53:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:20 smithi194 ceph-mon[33017]: pgmap v4531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:20 smithi033 ceph-mon[27845]: pgmap v4531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:20 smithi096 ceph-mon[32753]: pgmap v4531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:22 smithi194 ceph-mon[33017]: pgmap v4532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:22 smithi033 ceph-mon[27845]: pgmap v4532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:22 smithi096 ceph-mon[32753]: pgmap v4532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.038+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T13:53:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.038+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T13:53:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.038+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T13:53:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.038+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T13:53:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.038+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x564385821900 2024-04-26T13:53:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.038+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T13:53:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.038+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:53:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.039+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:53:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.115+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.115+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:23.115+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:53:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7341.797574 secs 2024-04-26T13:53:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7344.555846 secs 2024-04-26T13:53:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7341.797574 secs 2024-04-26T13:53:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7344.555846 secs 2024-04-26T13:53:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7341.797574 secs 2024-04-26T13:53:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7344.555846 secs 2024-04-26T13:53:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:24 smithi194 ceph-mon[33017]: pgmap v4533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:24 smithi033 ceph-mon[27845]: pgmap v4533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:24 smithi096 ceph-mon[32753]: pgmap v4533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:26 smithi194 ceph-mon[33017]: pgmap v4534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:26.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:26 smithi033 ceph-mon[27845]: pgmap v4534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:26 smithi096 ceph-mon[32753]: pgmap v4534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:28 smithi194 ceph-mon[33017]: pgmap v4535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7346.797677 secs 2024-04-26T13:53:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7349.555966 secs 2024-04-26T13:53:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:28 smithi033 ceph-mon[27845]: pgmap v4535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7346.797677 secs 2024-04-26T13:53:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7349.555966 secs 2024-04-26T13:53:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:28 smithi096 ceph-mon[32753]: pgmap v4535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7346.797677 secs 2024-04-26T13:53:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7349.555966 secs 2024-04-26T13:53:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:30 smithi194 ceph-mon[33017]: pgmap v4536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:30.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:30 smithi033 ceph-mon[27845]: pgmap v4536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:30 smithi096 ceph-mon[32753]: pgmap v4536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:32 smithi194 ceph-mon[33017]: pgmap v4537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:32 smithi033 ceph-mon[27845]: pgmap v4537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:32 smithi096 ceph-mon[32753]: pgmap v4537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7351.797765 secs 2024-04-26T13:53:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7354.556073 secs 2024-04-26T13:53:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7351.797765 secs 2024-04-26T13:53:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7354.556073 secs 2024-04-26T13:53:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7351.797765 secs 2024-04-26T13:53:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7354.556073 secs 2024-04-26T13:53:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:34 smithi194 ceph-mon[33017]: pgmap v4538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:34 smithi033 ceph-mon[27845]: pgmap v4538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:34 smithi096 ceph-mon[32753]: pgmap v4538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:36 smithi194 ceph-mon[33017]: pgmap v4539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:36 smithi033 ceph-mon[27845]: pgmap v4539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:36 smithi096 ceph-mon[32753]: pgmap v4539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:38.127+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:38.127+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:53:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:38.127+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:38 smithi194 ceph-mon[33017]: pgmap v4540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7356.797866 secs 2024-04-26T13:53:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7359.556194 secs 2024-04-26T13:53:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:38 smithi033 ceph-mon[27845]: pgmap v4540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7356.797866 secs 2024-04-26T13:53:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7359.556194 secs 2024-04-26T13:53:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:38 smithi096 ceph-mon[32753]: pgmap v4540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7356.797866 secs 2024-04-26T13:53:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7359.556194 secs 2024-04-26T13:53:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:40 smithi194 ceph-mon[33017]: pgmap v4541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:40 smithi033 ceph-mon[27845]: pgmap v4541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:40 smithi096 ceph-mon[32753]: pgmap v4541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:42 smithi194 ceph-mon[33017]: pgmap v4542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:42 smithi033 ceph-mon[27845]: pgmap v4542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:42.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:42 smithi096 ceph-mon[32753]: pgmap v4542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7361.797975 secs 2024-04-26T13:53:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7364.556314 secs 2024-04-26T13:53:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7361.797975 secs 2024-04-26T13:53:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7364.556314 secs 2024-04-26T13:53:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7361.797975 secs 2024-04-26T13:53:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7364.556314 secs 2024-04-26T13:53:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:44 smithi194 ceph-mon[33017]: pgmap v4543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:44 smithi033 ceph-mon[27845]: pgmap v4543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:44 smithi096 ceph-mon[32753]: pgmap v4543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:46 smithi194 ceph-mon[33017]: pgmap v4544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:46.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:46 smithi033 ceph-mon[27845]: pgmap v4544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:46 smithi096 ceph-mon[32753]: pgmap v4544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:48 smithi194 ceph-mon[33017]: pgmap v4545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7366.798060 secs 2024-04-26T13:53:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7369.556441 secs 2024-04-26T13:53:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:48 smithi033 ceph-mon[27845]: pgmap v4545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7366.798060 secs 2024-04-26T13:53:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7369.556441 secs 2024-04-26T13:53:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:48 smithi096 ceph-mon[32753]: pgmap v4545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7366.798060 secs 2024-04-26T13:53:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7369.556441 secs 2024-04-26T13:53:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:50 smithi033 ceph-mon[27845]: pgmap v4546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:50 smithi096 ceph-mon[32753]: pgmap v4546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:50 smithi194 ceph-mon[33017]: pgmap v4546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:52.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:52 smithi194 ceph-mon[33017]: pgmap v4547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:52.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:52 smithi033 ceph-mon[27845]: pgmap v4547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:52 smithi096 ceph-mon[32753]: pgmap v4547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:53.144+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:53.144+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:53:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:53:53.144+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:53:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7371.798147 secs 2024-04-26T13:53:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7374.556557 secs 2024-04-26T13:53:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7371.798147 secs 2024-04-26T13:53:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7374.556557 secs 2024-04-26T13:53:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7371.798147 secs 2024-04-26T13:53:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7374.556557 secs 2024-04-26T13:53:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:54 smithi033 ceph-mon[27845]: pgmap v4548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:54 smithi096 ceph-mon[32753]: pgmap v4548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:54 smithi194 ceph-mon[33017]: pgmap v4548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:55 smithi033 ceph-mon[27845]: pgmap v4549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:55 smithi096 ceph-mon[32753]: pgmap v4549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:55 smithi194 ceph-mon[33017]: pgmap v4549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:53:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:57 smithi033 ceph-mon[27845]: pgmap v4550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:57 smithi096 ceph-mon[32753]: pgmap v4550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:57 smithi194 ceph-mon[33017]: pgmap v4550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7376.798230 secs 2024-04-26T13:53:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7379.556709 secs 2024-04-26T13:53:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7376.798230 secs 2024-04-26T13:53:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7379.556709 secs 2024-04-26T13:53:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7376.798230 secs 2024-04-26T13:53:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7379.556709 secs 2024-04-26T13:53:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:53:59 smithi033 ceph-mon[27845]: pgmap v4551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:53:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:53:59 smithi096 ceph-mon[32753]: pgmap v4551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:53:59 smithi194 ceph-mon[33017]: pgmap v4551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:01 smithi033 ceph-mon[27845]: pgmap v4552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:01 smithi096 ceph-mon[32753]: pgmap v4552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:01 smithi194 ceph-mon[33017]: pgmap v4552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7381.798316 secs 2024-04-26T13:54:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:03 smithi033 ceph-mon[27845]: pgmap v4553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:54:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7384.556804 secs 2024-04-26T13:54:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7381.798316 secs 2024-04-26T13:54:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:03 smithi096 ceph-mon[32753]: pgmap v4553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:54:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7384.556804 secs 2024-04-26T13:54:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7381.798316 secs 2024-04-26T13:54:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:03 smithi194 ceph-mon[33017]: pgmap v4553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:54:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7384.556804 secs 2024-04-26T13:54:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:05 smithi033 ceph-mon[27845]: pgmap v4554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:54:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:54:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:05 smithi033 ceph-mon[27845]: pgmap v4555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:54:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:05 smithi033 ceph-mon[27845]: pgmap v4556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:05 smithi096 ceph-mon[32753]: pgmap v4554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:54:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:54:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:05 smithi096 ceph-mon[32753]: pgmap v4555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:54:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:05 smithi096 ceph-mon[32753]: pgmap v4556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:05 smithi194 ceph-mon[33017]: pgmap v4554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:54:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:54:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:05 smithi194 ceph-mon[33017]: pgmap v4555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T13:54:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:05 smithi194 ceph-mon[33017]: pgmap v4556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:54:08.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:08.156+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:08.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:08.156+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:08.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:08.156+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:54:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:08 smithi194 ceph-mon[33017]: pgmap v4557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7389.556923 secs 2024-04-26T13:54:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:08 smithi033 ceph-mon[27845]: pgmap v4557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7389.556923 secs 2024-04-26T13:54:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:08 smithi096 ceph-mon[32753]: pgmap v4557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7389.556923 secs 2024-04-26T13:54:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7386.798404 secs 2024-04-26T13:54:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7386.798404 secs 2024-04-26T13:54:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7386.798404 secs 2024-04-26T13:54:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:10 smithi194 ceph-mon[33017]: pgmap v4558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:54:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:10 smithi033 ceph-mon[27845]: pgmap v4558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:54:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:10 smithi096 ceph-mon[32753]: pgmap v4558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:54:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:12 smithi194 ceph-mon[33017]: pgmap v4559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:12 smithi033 ceph-mon[27845]: pgmap v4559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:12 smithi096 ceph-mon[32753]: pgmap v4559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T13:54:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7391.798498 secs 2024-04-26T13:54:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7394.557064 secs 2024-04-26T13:54:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7391.798498 secs 2024-04-26T13:54:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7394.557064 secs 2024-04-26T13:54:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7391.798498 secs 2024-04-26T13:54:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7394.557064 secs 2024-04-26T13:54:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:14 smithi194 ceph-mon[33017]: pgmap v4560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:54:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:14 smithi033 ceph-mon[27845]: pgmap v4560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:54:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:14 smithi096 ceph-mon[32753]: pgmap v4560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:54:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:16 smithi194 ceph-mon[33017]: pgmap v4561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:54:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:16 smithi033 ceph-mon[27845]: pgmap v4561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:54:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:16 smithi096 ceph-mon[32753]: pgmap v4561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:54:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:18 smithi194 ceph-mon[33017]: pgmap v4562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7396.798597 secs 2024-04-26T13:54:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7399.557162 secs 2024-04-26T13:54:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:18 smithi033 ceph-mon[27845]: pgmap v4562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7396.798597 secs 2024-04-26T13:54:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7399.557162 secs 2024-04-26T13:54:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:18 smithi096 ceph-mon[32753]: pgmap v4562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7396.798597 secs 2024-04-26T13:54:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7399.557162 secs 2024-04-26T13:54:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:20 smithi194 ceph-mon[33017]: pgmap v4563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:20 smithi033 ceph-mon[27845]: pgmap v4563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:20 smithi096 ceph-mon[32753]: pgmap v4563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:22 smithi194 ceph-mon[33017]: pgmap v4564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:22 smithi033 ceph-mon[27845]: pgmap v4564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:22 smithi096 ceph-mon[32753]: pgmap v4564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:23.162+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:23.162+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:23.162+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:54:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7401.798696 secs 2024-04-26T13:54:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7404.557228 secs 2024-04-26T13:54:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7401.798696 secs 2024-04-26T13:54:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7404.557228 secs 2024-04-26T13:54:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7401.798696 secs 2024-04-26T13:54:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7404.557228 secs 2024-04-26T13:54:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:24 smithi194 ceph-mon[33017]: pgmap v4565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:24 smithi033 ceph-mon[27845]: pgmap v4565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:24 smithi096 ceph-mon[32753]: pgmap v4565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:26 smithi194 ceph-mon[33017]: pgmap v4566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:26 smithi033 ceph-mon[27845]: pgmap v4566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:26 smithi096 ceph-mon[32753]: pgmap v4566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:28 smithi194 ceph-mon[33017]: pgmap v4567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7406.798797 secs 2024-04-26T13:54:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7409.557312 secs 2024-04-26T13:54:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:28 smithi033 ceph-mon[27845]: pgmap v4567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7406.798797 secs 2024-04-26T13:54:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7409.557312 secs 2024-04-26T13:54:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:28 smithi096 ceph-mon[32753]: pgmap v4567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7406.798797 secs 2024-04-26T13:54:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7409.557312 secs 2024-04-26T13:54:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:30 smithi033 ceph-mon[27845]: pgmap v4568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:30 smithi096 ceph-mon[32753]: pgmap v4568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:30.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:30 smithi194 ceph-mon[33017]: pgmap v4568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:32 smithi033 ceph-mon[27845]: pgmap v4569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:32 smithi096 ceph-mon[32753]: pgmap v4569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:32 smithi194 ceph-mon[33017]: pgmap v4569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7411.798900 secs 2024-04-26T13:54:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7414.557433 secs 2024-04-26T13:54:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7411.798900 secs 2024-04-26T13:54:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7414.557433 secs 2024-04-26T13:54:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7411.798900 secs 2024-04-26T13:54:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7414.557433 secs 2024-04-26T13:54:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:34 smithi033 ceph-mon[27845]: pgmap v4570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:34 smithi096 ceph-mon[32753]: pgmap v4570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:34 smithi194 ceph-mon[33017]: pgmap v4570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:36 smithi033 ceph-mon[27845]: pgmap v4571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:36 smithi096 ceph-mon[32753]: pgmap v4571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:36 smithi194 ceph-mon[33017]: pgmap v4571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:38.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:38.173+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:38.173+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:38.173+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:54:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:38 smithi033 ceph-mon[27845]: pgmap v4572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7416.799002 secs 2024-04-26T13:54:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7419.557570 secs 2024-04-26T13:54:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:38 smithi096 ceph-mon[32753]: pgmap v4572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7416.799002 secs 2024-04-26T13:54:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7419.557570 secs 2024-04-26T13:54:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:38 smithi194 ceph-mon[33017]: pgmap v4572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7416.799002 secs 2024-04-26T13:54:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7419.557570 secs 2024-04-26T13:54:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:40 smithi033 ceph-mon[27845]: pgmap v4573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:40 smithi096 ceph-mon[32753]: pgmap v4573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:40 smithi194 ceph-mon[33017]: pgmap v4573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:42 smithi033 ceph-mon[27845]: pgmap v4574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:42 smithi096 ceph-mon[32753]: pgmap v4574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:42 smithi194 ceph-mon[33017]: pgmap v4574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7421.799102 secs 2024-04-26T13:54:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7424.557690 secs 2024-04-26T13:54:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7421.799102 secs 2024-04-26T13:54:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7424.557690 secs 2024-04-26T13:54:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7421.799102 secs 2024-04-26T13:54:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7424.557690 secs 2024-04-26T13:54:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:44 smithi033 ceph-mon[27845]: pgmap v4575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:44 smithi096 ceph-mon[32753]: pgmap v4575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:44 smithi194 ceph-mon[33017]: pgmap v4575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:54:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:46 smithi033 ceph-mon[27845]: pgmap v4576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:46 smithi096 ceph-mon[32753]: pgmap v4576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:46 smithi194 ceph-mon[33017]: pgmap v4576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:54:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:48 smithi033 ceph-mon[27845]: pgmap v4577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7426.799207 secs 2024-04-26T13:54:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7429.557800 secs 2024-04-26T13:54:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:48 smithi096 ceph-mon[32753]: pgmap v4577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7426.799207 secs 2024-04-26T13:54:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7429.557800 secs 2024-04-26T13:54:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:48 smithi194 ceph-mon[33017]: pgmap v4577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7426.799207 secs 2024-04-26T13:54:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7429.557800 secs 2024-04-26T13:54:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:50 smithi033 ceph-mon[27845]: pgmap v4578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:50 smithi096 ceph-mon[32753]: pgmap v4578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:50 smithi194 ceph-mon[33017]: pgmap v4578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:52 smithi096 ceph-mon[32753]: pgmap v4579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:52 smithi194 ceph-mon[33017]: pgmap v4579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:52.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:52 smithi033 ceph-mon[27845]: pgmap v4579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:53.183+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:53.184+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:54:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:54:53.184+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:54:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7431.799310 secs 2024-04-26T13:54:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7434.557917 secs 2024-04-26T13:54:53.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7431.799310 secs 2024-04-26T13:54:53.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7434.557917 secs 2024-04-26T13:54:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7431.799310 secs 2024-04-26T13:54:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7434.557917 secs 2024-04-26T13:54:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:54 smithi194 ceph-mon[33017]: pgmap v4580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:54 smithi033 ceph-mon[27845]: pgmap v4580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:54 smithi096 ceph-mon[32753]: pgmap v4580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:56 smithi194 ceph-mon[33017]: pgmap v4581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:56 smithi033 ceph-mon[27845]: pgmap v4581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:56 smithi096 ceph-mon[32753]: pgmap v4581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:54:58.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:58 smithi096 ceph-mon[32753]: pgmap v4582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:58.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7436.799371 secs 2024-04-26T13:54:58.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:54:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7439.558029 secs 2024-04-26T13:54:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:58 smithi194 ceph-mon[33017]: pgmap v4582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7436.799371 secs 2024-04-26T13:54:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:54:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7439.558029 secs 2024-04-26T13:54:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:58 smithi033 ceph-mon[27845]: pgmap v4582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:54:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7436.799371 secs 2024-04-26T13:54:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:54:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7439.558029 secs 2024-04-26T13:55:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:00 smithi194 ceph-mon[33017]: pgmap v4583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:55:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:00 smithi033 ceph-mon[27845]: pgmap v4583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:55:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:00 smithi096 ceph-mon[32753]: pgmap v4583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:55:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:02 smithi194 ceph-mon[33017]: pgmap v4584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:02.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:02 smithi033 ceph-mon[27845]: pgmap v4584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:02 smithi096 ceph-mon[32753]: pgmap v4584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7441.799475 secs 2024-04-26T13:55:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7444.558146 secs 2024-04-26T13:55:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7441.799475 secs 2024-04-26T13:55:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7444.558146 secs 2024-04-26T13:55:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7441.799475 secs 2024-04-26T13:55:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7444.558146 secs 2024-04-26T13:55:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:04 smithi194 ceph-mon[33017]: pgmap v4585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:04 smithi033 ceph-mon[27845]: pgmap v4585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:04 smithi096 ceph-mon[32753]: pgmap v4585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:06 smithi194 ceph-mon[33017]: pgmap v4586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:55:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:06 smithi033 ceph-mon[27845]: pgmap v4586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:55:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:06 smithi096 ceph-mon[32753]: pgmap v4586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:55:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:55:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:55:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:55:07.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:55:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:55:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:55:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:55:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:55:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:55:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:08.199+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:08.199+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:08.199+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:55:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:08 smithi194 ceph-mon[33017]: pgmap v4587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:08 smithi194 ceph-mon[33017]: pgmap v4588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7446.799576 secs 2024-04-26T13:55:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7449.558263 secs 2024-04-26T13:55:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:08 smithi033 ceph-mon[27845]: pgmap v4587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:08 smithi033 ceph-mon[27845]: pgmap v4588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7446.799576 secs 2024-04-26T13:55:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7449.558263 secs 2024-04-26T13:55:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:08 smithi096 ceph-mon[32753]: pgmap v4587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:08 smithi096 ceph-mon[32753]: pgmap v4588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7446.799576 secs 2024-04-26T13:55:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7449.558263 secs 2024-04-26T13:55:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:10 smithi194 ceph-mon[33017]: pgmap v4589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:55:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:10 smithi033 ceph-mon[27845]: pgmap v4589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:55:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:10 smithi096 ceph-mon[32753]: pgmap v4589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:55:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:12 smithi194 ceph-mon[33017]: pgmap v4590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:12 smithi033 ceph-mon[27845]: pgmap v4590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:12 smithi096 ceph-mon[32753]: pgmap v4590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7451.799677 secs 2024-04-26T13:55:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7454.558383 secs 2024-04-26T13:55:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7451.799677 secs 2024-04-26T13:55:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7454.558383 secs 2024-04-26T13:55:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7451.799677 secs 2024-04-26T13:55:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7454.558383 secs 2024-04-26T13:55:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:14 smithi194 ceph-mon[33017]: pgmap v4591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:14 smithi033 ceph-mon[27845]: pgmap v4591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:14 smithi096 ceph-mon[32753]: pgmap v4591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:16 smithi194 ceph-mon[33017]: pgmap v4592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:16 smithi033 ceph-mon[27845]: pgmap v4592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:16 smithi096 ceph-mon[32753]: pgmap v4592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:55:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:18 smithi194 ceph-mon[33017]: pgmap v4593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:55:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7456.799779 secs 2024-04-26T13:55:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7459.558508 secs 2024-04-26T13:55:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:18 smithi033 ceph-mon[27845]: pgmap v4593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:55:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7456.799779 secs 2024-04-26T13:55:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7459.558508 secs 2024-04-26T13:55:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:18 smithi096 ceph-mon[32753]: pgmap v4593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:55:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7456.799779 secs 2024-04-26T13:55:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7459.558508 secs 2024-04-26T13:55:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:20 smithi194 ceph-mon[33017]: pgmap v4594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:20 smithi033 ceph-mon[27845]: pgmap v4594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:20 smithi096 ceph-mon[32753]: pgmap v4594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:22 smithi194 ceph-mon[33017]: pgmap v4595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:22 smithi033 ceph-mon[27845]: pgmap v4595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:22 smithi096 ceph-mon[32753]: pgmap v4595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:23.215+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:23.215+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:23.215+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7461.799881 secs 2024-04-26T13:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7464.558644 secs 2024-04-26T13:55:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7461.799881 secs 2024-04-26T13:55:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7464.558644 secs 2024-04-26T13:55:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7461.799881 secs 2024-04-26T13:55:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7464.558644 secs 2024-04-26T13:55:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:24 smithi194 ceph-mon[33017]: pgmap v4596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:24 smithi033 ceph-mon[27845]: pgmap v4596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:24 smithi096 ceph-mon[32753]: pgmap v4596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:26 smithi194 ceph-mon[33017]: pgmap v4597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:26.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:26 smithi033 ceph-mon[27845]: pgmap v4597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:26 smithi096 ceph-mon[32753]: pgmap v4597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:28 smithi194 ceph-mon[33017]: pgmap v4598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7466.799991 secs 2024-04-26T13:55:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7469.558780 secs 2024-04-26T13:55:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:28 smithi033 ceph-mon[27845]: pgmap v4598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7466.799991 secs 2024-04-26T13:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7469.558780 secs 2024-04-26T13:55:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:28 smithi096 ceph-mon[32753]: pgmap v4598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7466.799991 secs 2024-04-26T13:55:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7469.558780 secs 2024-04-26T13:55:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:30 smithi194 ceph-mon[33017]: pgmap v4599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:30.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:30 smithi033 ceph-mon[27845]: pgmap v4599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:30 smithi096 ceph-mon[32753]: pgmap v4599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:32 smithi194 ceph-mon[33017]: pgmap v4600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:32 smithi033 ceph-mon[27845]: pgmap v4600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:32 smithi096 ceph-mon[32753]: pgmap v4600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7471.800068 secs 2024-04-26T13:55:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7474.558877 secs 2024-04-26T13:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7471.800068 secs 2024-04-26T13:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7474.558877 secs 2024-04-26T13:55:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7471.800068 secs 2024-04-26T13:55:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7474.558877 secs 2024-04-26T13:55:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:34 smithi194 ceph-mon[33017]: pgmap v4601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:34 smithi033 ceph-mon[27845]: pgmap v4601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:34 smithi096 ceph-mon[32753]: pgmap v4601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:36 smithi194 ceph-mon[33017]: pgmap v4602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:36 smithi033 ceph-mon[27845]: pgmap v4602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:36 smithi096 ceph-mon[32753]: pgmap v4602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:38.228+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:38.228+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:38.228+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:55:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:38 smithi194 ceph-mon[33017]: pgmap v4603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7476.800140 secs 2024-04-26T13:55:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7479.559029 secs 2024-04-26T13:55:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:38 smithi033 ceph-mon[27845]: pgmap v4603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7476.800140 secs 2024-04-26T13:55:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7479.559029 secs 2024-04-26T13:55:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:38 smithi096 ceph-mon[32753]: pgmap v4603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7476.800140 secs 2024-04-26T13:55:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7479.559029 secs 2024-04-26T13:55:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:40 smithi033 ceph-mon[27845]: pgmap v4604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:40 smithi096 ceph-mon[32753]: pgmap v4604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:40 smithi194 ceph-mon[33017]: pgmap v4604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:42 smithi033 ceph-mon[27845]: pgmap v4605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:42 smithi096 ceph-mon[32753]: pgmap v4605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:42 smithi194 ceph-mon[33017]: pgmap v4605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7481.800203 secs 2024-04-26T13:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7484.559177 secs 2024-04-26T13:55:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7481.800203 secs 2024-04-26T13:55:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7484.559177 secs 2024-04-26T13:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7481.800203 secs 2024-04-26T13:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7484.559177 secs 2024-04-26T13:55:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:44 smithi033 ceph-mon[27845]: pgmap v4606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:44 smithi096 ceph-mon[32753]: pgmap v4606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:44 smithi194 ceph-mon[33017]: pgmap v4606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:46.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:46 smithi033 ceph-mon[27845]: pgmap v4607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:46 smithi096 ceph-mon[32753]: pgmap v4607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:46 smithi194 ceph-mon[33017]: pgmap v4607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:48 smithi033 ceph-mon[27845]: pgmap v4608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7486.800264 secs 2024-04-26T13:55:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7489.559291 secs 2024-04-26T13:55:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:48 smithi096 ceph-mon[32753]: pgmap v4608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7486.800264 secs 2024-04-26T13:55:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7489.559291 secs 2024-04-26T13:55:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:48 smithi194 ceph-mon[33017]: pgmap v4608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7486.800264 secs 2024-04-26T13:55:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7489.559291 secs 2024-04-26T13:55:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:50 smithi033 ceph-mon[27845]: pgmap v4609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:50 smithi096 ceph-mon[32753]: pgmap v4609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:50 smithi194 ceph-mon[33017]: pgmap v4609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:51 smithi033 ceph-mon[27845]: pgmap v4610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:51 smithi096 ceph-mon[32753]: pgmap v4610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:52.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:51 smithi194 ceph-mon[33017]: pgmap v4610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:53.242+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:53.242+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:55:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:55:53.242+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:55:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7491.800326 secs 2024-04-26T13:55:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:53 smithi194 ceph-mon[33017]: pgmap v4611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7494.559358 secs 2024-04-26T13:55:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7491.800326 secs 2024-04-26T13:55:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:53 smithi033 ceph-mon[27845]: pgmap v4611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7494.559358 secs 2024-04-26T13:55:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7491.800326 secs 2024-04-26T13:55:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:53 smithi096 ceph-mon[32753]: pgmap v4611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7494.559358 secs 2024-04-26T13:55:56.335 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:55 smithi033 ceph-mon[27845]: pgmap v4612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:55 smithi194 ceph-mon[33017]: pgmap v4612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:55 smithi096 ceph-mon[32753]: pgmap v4612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:55:58.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:57 smithi096 ceph-mon[32753]: pgmap v4613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:57 smithi194 ceph-mon[33017]: pgmap v4613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:57 smithi033 ceph-mon[27845]: pgmap v4613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:55:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7496.800385 secs 2024-04-26T13:55:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7499.559483 secs 2024-04-26T13:55:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7496.800385 secs 2024-04-26T13:55:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7499.559483 secs 2024-04-26T13:55:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7496.800385 secs 2024-04-26T13:55:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7499.559483 secs 2024-04-26T13:56:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:55:59 smithi194 ceph-mon[33017]: pgmap v4614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:55:59 smithi033 ceph-mon[27845]: pgmap v4614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:55:59 smithi096 ceph-mon[32753]: pgmap v4614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:02 smithi194 ceph-mon[33017]: pgmap v4615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:02 smithi033 ceph-mon[27845]: pgmap v4615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:02 smithi096 ceph-mon[32753]: pgmap v4615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7501.800435 secs 2024-04-26T13:56:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:04 smithi194 ceph-mon[33017]: pgmap v4616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7504.559596 secs 2024-04-26T13:56:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7501.800435 secs 2024-04-26T13:56:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:04 smithi033 ceph-mon[27845]: pgmap v4616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7504.559596 secs 2024-04-26T13:56:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7501.800435 secs 2024-04-26T13:56:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:04 smithi096 ceph-mon[32753]: pgmap v4616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7504.559596 secs 2024-04-26T13:56:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:06 smithi194 ceph-mon[33017]: pgmap v4617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:06 smithi033 ceph-mon[27845]: pgmap v4617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:06 smithi096 ceph-mon[32753]: pgmap v4617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:08.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:08.257+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:08.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:08.257+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:08.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:08.257+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:56:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:08 smithi194 ceph-mon[33017]: pgmap v4618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:56:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:08 smithi033 ceph-mon[27845]: pgmap v4618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:08 smithi096 ceph-mon[32753]: pgmap v4618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7506.800525 secs 2024-04-26T13:56:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7509.559686 secs 2024-04-26T13:56:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7506.800525 secs 2024-04-26T13:56:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7509.559686 secs 2024-04-26T13:56:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7506.800525 secs 2024-04-26T13:56:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7509.559686 secs 2024-04-26T13:56:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:10 smithi194 ceph-mon[33017]: pgmap v4619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:10 smithi033 ceph-mon[27845]: pgmap v4619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:10 smithi096 ceph-mon[32753]: pgmap v4619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:12.298 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:12 smithi194 ceph-mon[33017]: pgmap v4620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:12.298 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:12.298 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:12 smithi033 ceph-mon[27845]: pgmap v4620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:12 smithi096 ceph-mon[32753]: pgmap v4620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7511.800587 secs 2024-04-26T13:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:14 smithi194 ceph-mon[33017]: pgmap v4621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7514.559779 secs 2024-04-26T13:56:14.368 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7511.800587 secs 2024-04-26T13:56:14.368 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:14 smithi033 ceph-mon[27845]: pgmap v4621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:14.368 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7514.559779 secs 2024-04-26T13:56:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7511.800587 secs 2024-04-26T13:56:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:14 smithi096 ceph-mon[32753]: pgmap v4621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7514.559779 secs 2024-04-26T13:56:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:56:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:56:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:56:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:15.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T13:56:15.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T13:56:16.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:16 smithi033 ceph-mon[27845]: pgmap v4622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:16 smithi194 ceph-mon[33017]: pgmap v4622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:16 smithi096 ceph-mon[32753]: pgmap v4622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:18.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:18 smithi033 ceph-mon[27845]: pgmap v4623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:18 smithi194 ceph-mon[33017]: pgmap v4623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:18 smithi096 ceph-mon[32753]: pgmap v4623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7516.800670 secs 2024-04-26T13:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7519.559879 secs 2024-04-26T13:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7516.800670 secs 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7519.559879 secs 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7516.800670 secs 2024-04-26T13:56:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7519.559879 secs 2024-04-26T13:56:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: pgmap v4624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:56:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: pgmap v4625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: pgmap v4626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T13:56:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: pgmap v4624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:56:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:56:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:56:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:56:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: pgmap v4625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: pgmap v4626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T13:56:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: pgmap v4624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T13:56:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T13:56:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T13:56:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:56:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:56:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: pgmap v4625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: pgmap v4626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T13:56:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:56:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 msgr2=0x7feee40b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T13:56:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 msgr2=0x7feee40b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T13:56:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 0x7feee40b3540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0048000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T13:56:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 0x7feee40b3540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0048000 comp rx=0 tx=0).stop 2024-04-26T13:56:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 msgr2=0x7feee40b3540 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T13:56:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 msgr2=0x7feee40b3540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T13:56:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2960 0x7feee40b3540 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T13:56:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.727+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T13:56:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.728+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T13:56:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:21.730+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3a80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef004e000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T13:56:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:22 smithi033 ceph-mon[27845]: pgmap v4627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:22 smithi194 ceph-mon[33017]: pgmap v4627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:22 smithi096 ceph-mon[32753]: pgmap v4627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:23.067+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:56:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:23.067+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:56:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:23.068+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:56:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:23.068+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:56:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:23.258+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:23.258+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:23.258+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:56:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7521.800756 secs 2024-04-26T13:56:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7524.560012 secs 2024-04-26T13:56:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7521.800756 secs 2024-04-26T13:56:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7524.560012 secs 2024-04-26T13:56:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7521.800756 secs 2024-04-26T13:56:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7524.560012 secs 2024-04-26T13:56:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:24 smithi096 ceph-mon[32753]: pgmap v4628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:56:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:24 smithi194 ceph-mon[33017]: pgmap v4628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:56:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:24 smithi033 ceph-mon[27845]: pgmap v4628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T13:56:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:26 smithi194 ceph-mon[33017]: pgmap v4629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:26 smithi033 ceph-mon[27845]: pgmap v4629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:26 smithi096 ceph-mon[32753]: pgmap v4629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:28 smithi194 ceph-mon[33017]: pgmap v4630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7526.800839 secs 2024-04-26T13:56:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7529.560128 secs 2024-04-26T13:56:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:28 smithi033 ceph-mon[27845]: pgmap v4630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7526.800839 secs 2024-04-26T13:56:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7529.560128 secs 2024-04-26T13:56:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:28 smithi096 ceph-mon[32753]: pgmap v4630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7526.800839 secs 2024-04-26T13:56:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7529.560128 secs 2024-04-26T13:56:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:30 smithi194 ceph-mon[33017]: pgmap v4631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:56:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:30 smithi033 ceph-mon[27845]: pgmap v4631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:56:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:30 smithi096 ceph-mon[32753]: pgmap v4631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:56:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:32 smithi194 ceph-mon[33017]: pgmap v4632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:56:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:32 smithi033 ceph-mon[27845]: pgmap v4632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:56:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:32 smithi096 ceph-mon[32753]: pgmap v4632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:56:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7531.800927 secs 2024-04-26T13:56:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7534.560241 secs 2024-04-26T13:56:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7531.800927 secs 2024-04-26T13:56:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7534.560241 secs 2024-04-26T13:56:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7531.800927 secs 2024-04-26T13:56:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7534.560241 secs 2024-04-26T13:56:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:34 smithi033 ceph-mon[27845]: pgmap v4633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:34 smithi096 ceph-mon[32753]: pgmap v4633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:34 smithi194 ceph-mon[33017]: pgmap v4633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:36 smithi033 ceph-mon[27845]: pgmap v4634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:36 smithi096 ceph-mon[32753]: pgmap v4634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:36 smithi194 ceph-mon[33017]: pgmap v4634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:38.258+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:38.258+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:38.258+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:56:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:38 smithi033 ceph-mon[27845]: pgmap v4635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:56:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7536.801026 secs 2024-04-26T13:56:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7539.560351 secs 2024-04-26T13:56:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:38 smithi096 ceph-mon[32753]: pgmap v4635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:56:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7536.801026 secs 2024-04-26T13:56:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7539.560351 secs 2024-04-26T13:56:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:38 smithi194 ceph-mon[33017]: pgmap v4635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:56:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7536.801026 secs 2024-04-26T13:56:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7539.560351 secs 2024-04-26T13:56:40.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:40 smithi033 ceph-mon[27845]: pgmap v4636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:56:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:40 smithi096 ceph-mon[32753]: pgmap v4636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:56:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:40 smithi194 ceph-mon[33017]: pgmap v4636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:56:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:42 smithi033 ceph-mon[27845]: pgmap v4637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:56:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:42 smithi096 ceph-mon[32753]: pgmap v4637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:56:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:42 smithi194 ceph-mon[33017]: pgmap v4637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T13:56:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7541.801110 secs 2024-04-26T13:56:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7544.560460 secs 2024-04-26T13:56:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7541.801110 secs 2024-04-26T13:56:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7544.560460 secs 2024-04-26T13:56:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7541.801110 secs 2024-04-26T13:56:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7544.560460 secs 2024-04-26T13:56:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:44 smithi033 ceph-mon[27845]: pgmap v4638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:56:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:44 smithi096 ceph-mon[32753]: pgmap v4638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:56:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:44 smithi194 ceph-mon[33017]: pgmap v4638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T13:56:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:46 smithi033 ceph-mon[27845]: pgmap v4639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:46 smithi096 ceph-mon[32753]: pgmap v4639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:46 smithi194 ceph-mon[33017]: pgmap v4639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T13:56:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:48 smithi033 ceph-mon[27845]: pgmap v4640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7546.801193 secs 2024-04-26T13:56:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7549.560583 secs 2024-04-26T13:56:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:48 smithi096 ceph-mon[32753]: pgmap v4640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7546.801193 secs 2024-04-26T13:56:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7549.560583 secs 2024-04-26T13:56:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:48 smithi194 ceph-mon[33017]: pgmap v4640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7546.801193 secs 2024-04-26T13:56:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7549.560583 secs 2024-04-26T13:56:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:50 smithi033 ceph-mon[27845]: pgmap v4641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:56:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:50 smithi096 ceph-mon[32753]: pgmap v4641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:56:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:50 smithi194 ceph-mon[33017]: pgmap v4641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:56:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:52 smithi033 ceph-mon[27845]: pgmap v4642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:56:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:52 smithi096 ceph-mon[32753]: pgmap v4642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:56:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:52 smithi194 ceph-mon[33017]: pgmap v4642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:56:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:53.274+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:53.274+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:56:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:56:53.274+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:56:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7551.801283 secs 2024-04-26T13:56:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7554.560714 secs 2024-04-26T13:56:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7551.801283 secs 2024-04-26T13:56:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7554.560714 secs 2024-04-26T13:56:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7551.801283 secs 2024-04-26T13:56:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7554.560714 secs 2024-04-26T13:56:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:54 smithi096 ceph-mon[32753]: pgmap v4643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:54 smithi194 ceph-mon[33017]: pgmap v4643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:54 smithi033 ceph-mon[27845]: pgmap v4643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:56 smithi194 ceph-mon[33017]: pgmap v4644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:56.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:56 smithi033 ceph-mon[27845]: pgmap v4644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:56 smithi096 ceph-mon[32753]: pgmap v4644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:56:58.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:58 smithi096 ceph-mon[32753]: pgmap v4645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:58.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7556.801344 secs 2024-04-26T13:56:58.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:56:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7559.560851 secs 2024-04-26T13:56:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:58 smithi194 ceph-mon[33017]: pgmap v4645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7556.801344 secs 2024-04-26T13:56:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:56:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7559.560851 secs 2024-04-26T13:56:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:58 smithi033 ceph-mon[27845]: pgmap v4645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:56:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7556.801344 secs 2024-04-26T13:56:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:56:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7559.560851 secs 2024-04-26T13:57:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:00 smithi194 ceph-mon[33017]: pgmap v4646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:00.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:00 smithi033 ceph-mon[27845]: pgmap v4646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:00 smithi096 ceph-mon[32753]: pgmap v4646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:02 smithi194 ceph-mon[33017]: pgmap v4647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:02.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:02 smithi033 ceph-mon[27845]: pgmap v4647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:02 smithi096 ceph-mon[32753]: pgmap v4647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7561.801430 secs 2024-04-26T13:57:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7564.560984 secs 2024-04-26T13:57:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7561.801430 secs 2024-04-26T13:57:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7564.560984 secs 2024-04-26T13:57:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7561.801430 secs 2024-04-26T13:57:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7564.560984 secs 2024-04-26T13:57:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:04 smithi194 ceph-mon[33017]: pgmap v4648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:04 smithi033 ceph-mon[27845]: pgmap v4648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:04 smithi096 ceph-mon[32753]: pgmap v4648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:06 smithi194 ceph-mon[33017]: pgmap v4649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:06 smithi033 ceph-mon[27845]: pgmap v4649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:06 smithi096 ceph-mon[32753]: pgmap v4649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:08.280+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:08.280+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:08.280+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:57:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:08 smithi194 ceph-mon[33017]: pgmap v4650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7566.801507 secs 2024-04-26T13:57:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7569.561102 secs 2024-04-26T13:57:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:08 smithi033 ceph-mon[27845]: pgmap v4650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7566.801507 secs 2024-04-26T13:57:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7569.561102 secs 2024-04-26T13:57:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:08 smithi096 ceph-mon[32753]: pgmap v4650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7566.801507 secs 2024-04-26T13:57:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7569.561102 secs 2024-04-26T13:57:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:10 smithi194 ceph-mon[33017]: pgmap v4651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:10.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:10 smithi033 ceph-mon[27845]: pgmap v4651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:10 smithi096 ceph-mon[32753]: pgmap v4651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:12 smithi194 ceph-mon[33017]: pgmap v4652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:12 smithi033 ceph-mon[27845]: pgmap v4652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:12 smithi096 ceph-mon[32753]: pgmap v4652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7571.801590 secs 2024-04-26T13:57:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7574.561193 secs 2024-04-26T13:57:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7571.801590 secs 2024-04-26T13:57:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7574.561193 secs 2024-04-26T13:57:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7571.801590 secs 2024-04-26T13:57:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7574.561193 secs 2024-04-26T13:57:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:14 smithi033 ceph-mon[27845]: pgmap v4653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:14 smithi096 ceph-mon[32753]: pgmap v4653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:14 smithi194 ceph-mon[33017]: pgmap v4653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:16.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:16 smithi033 ceph-mon[27845]: pgmap v4654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:16 smithi096 ceph-mon[32753]: pgmap v4654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:16 smithi194 ceph-mon[33017]: pgmap v4654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:18 smithi033 ceph-mon[27845]: pgmap v4655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7576.801671 secs 2024-04-26T13:57:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7579.561285 secs 2024-04-26T13:57:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:18 smithi096 ceph-mon[32753]: pgmap v4655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7576.801671 secs 2024-04-26T13:57:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7579.561285 secs 2024-04-26T13:57:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:18 smithi194 ceph-mon[33017]: pgmap v4655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7576.801671 secs 2024-04-26T13:57:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7579.561285 secs 2024-04-26T13:57:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:57:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:57:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:57:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:20 smithi033 ceph-mon[27845]: pgmap v4656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:20 smithi096 ceph-mon[32753]: pgmap v4656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:20 smithi194 ceph-mon[33017]: pgmap v4656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T13:57:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:57:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:57:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:57:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:57:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:57:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:57:22.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:57:22.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:57:22.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:57:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:22 smithi033 ceph-mon[27845]: pgmap v4657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:22 smithi033 ceph-mon[27845]: pgmap v4658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:22 smithi096 ceph-mon[32753]: pgmap v4657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:22 smithi096 ceph-mon[32753]: pgmap v4658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:22 smithi194 ceph-mon[33017]: pgmap v4657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:22 smithi194 ceph-mon[33017]: pgmap v4658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:23.295+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:23.296+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:23.296+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:57:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7581.801755 secs 2024-04-26T13:57:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7584.561401 secs 2024-04-26T13:57:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7581.801755 secs 2024-04-26T13:57:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7584.561401 secs 2024-04-26T13:57:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7581.801755 secs 2024-04-26T13:57:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7584.561401 secs 2024-04-26T13:57:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:24 smithi033 ceph-mon[27845]: pgmap v4659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:24 smithi096 ceph-mon[32753]: pgmap v4659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:24 smithi194 ceph-mon[33017]: pgmap v4659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:25 smithi033 ceph-mon[27845]: pgmap v4660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:25 smithi096 ceph-mon[32753]: pgmap v4660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:25 smithi194 ceph-mon[33017]: pgmap v4660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:28 smithi194 ceph-mon[33017]: pgmap v4661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:28 smithi033 ceph-mon[27845]: pgmap v4661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:28 smithi096 ceph-mon[32753]: pgmap v4661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:57:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7586.801838 secs 2024-04-26T13:57:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7589.561502 secs 2024-04-26T13:57:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7586.801838 secs 2024-04-26T13:57:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7589.561502 secs 2024-04-26T13:57:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7586.801838 secs 2024-04-26T13:57:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7589.561502 secs 2024-04-26T13:57:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:30 smithi194 ceph-mon[33017]: pgmap v4662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T13:57:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:30 smithi033 ceph-mon[27845]: pgmap v4662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T13:57:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:30 smithi096 ceph-mon[32753]: pgmap v4662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T13:57:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:32 smithi194 ceph-mon[33017]: pgmap v4663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:57:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:32 smithi033 ceph-mon[27845]: pgmap v4663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:57:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:32 smithi096 ceph-mon[32753]: pgmap v4663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:57:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7591.801922 secs 2024-04-26T13:57:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:34 smithi194 ceph-mon[33017]: pgmap v4664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7594.561601 secs 2024-04-26T13:57:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7591.801922 secs 2024-04-26T13:57:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:34 smithi033 ceph-mon[27845]: pgmap v4664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7594.561601 secs 2024-04-26T13:57:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7591.801922 secs 2024-04-26T13:57:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:34 smithi096 ceph-mon[32753]: pgmap v4664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7594.561601 secs 2024-04-26T13:57:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:36 smithi194 ceph-mon[33017]: pgmap v4665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:36 smithi033 ceph-mon[27845]: pgmap v4665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:36 smithi096 ceph-mon[32753]: pgmap v4665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:38.306+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:38.306+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:57:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:38.306+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:38 smithi194 ceph-mon[33017]: pgmap v4666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:38 smithi033 ceph-mon[27845]: pgmap v4666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:38 smithi096 ceph-mon[32753]: pgmap v4666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7596.802005 secs 2024-04-26T13:57:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7599.561737 secs 2024-04-26T13:57:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7596.802005 secs 2024-04-26T13:57:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7599.561737 secs 2024-04-26T13:57:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7596.802005 secs 2024-04-26T13:57:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7599.561737 secs 2024-04-26T13:57:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:40 smithi194 ceph-mon[33017]: pgmap v4667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:40 smithi033 ceph-mon[27845]: pgmap v4667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:40 smithi096 ceph-mon[32753]: pgmap v4667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:42 smithi194 ceph-mon[33017]: pgmap v4668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:42 smithi033 ceph-mon[27845]: pgmap v4668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:42 smithi096 ceph-mon[32753]: pgmap v4668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7601.802091 secs 2024-04-26T13:57:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:44 smithi033 ceph-mon[27845]: pgmap v4669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7604.561853 secs 2024-04-26T13:57:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7601.802091 secs 2024-04-26T13:57:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:44 smithi096 ceph-mon[32753]: pgmap v4669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7604.561853 secs 2024-04-26T13:57:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7601.802091 secs 2024-04-26T13:57:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:44 smithi194 ceph-mon[33017]: pgmap v4669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7604.561853 secs 2024-04-26T13:57:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:46 smithi033 ceph-mon[27845]: pgmap v4670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:46 smithi096 ceph-mon[32753]: pgmap v4670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:46 smithi194 ceph-mon[33017]: pgmap v4670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:48 smithi033 ceph-mon[27845]: pgmap v4671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:48 smithi096 ceph-mon[32753]: pgmap v4671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:48 smithi194 ceph-mon[33017]: pgmap v4671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7606.802177 secs 2024-04-26T13:57:49.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7609.561972 secs 2024-04-26T13:57:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7606.802177 secs 2024-04-26T13:57:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7609.561972 secs 2024-04-26T13:57:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7606.802177 secs 2024-04-26T13:57:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7609.561972 secs 2024-04-26T13:57:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:50 smithi033 ceph-mon[27845]: pgmap v4672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:50 smithi096 ceph-mon[32753]: pgmap v4672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:50 smithi194 ceph-mon[33017]: pgmap v4672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:52.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:52 smithi194 ceph-mon[33017]: pgmap v4673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:52 smithi033 ceph-mon[27845]: pgmap v4673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:52 smithi096 ceph-mon[32753]: pgmap v4673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:53.322+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:53.322+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:57:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:57:53.322+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:57:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7611.802284 secs 2024-04-26T13:57:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:54 smithi033 ceph-mon[27845]: pgmap v4674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7614.562103 secs 2024-04-26T13:57:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7611.802284 secs 2024-04-26T13:57:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:54 smithi096 ceph-mon[32753]: pgmap v4674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7614.562103 secs 2024-04-26T13:57:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7611.802284 secs 2024-04-26T13:57:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:54 smithi194 ceph-mon[33017]: pgmap v4674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7614.562103 secs 2024-04-26T13:57:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:56 smithi033 ceph-mon[27845]: pgmap v4675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:56 smithi096 ceph-mon[32753]: pgmap v4675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:56 smithi194 ceph-mon[33017]: pgmap v4675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:57:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:58 smithi033 ceph-mon[27845]: pgmap v4676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:58 smithi096 ceph-mon[32753]: pgmap v4676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:58 smithi194 ceph-mon[33017]: pgmap v4676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:57:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7616.802366 secs 2024-04-26T13:57:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7619.562224 secs 2024-04-26T13:57:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7616.802366 secs 2024-04-26T13:57:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7619.562224 secs 2024-04-26T13:57:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7616.802366 secs 2024-04-26T13:57:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7619.562224 secs 2024-04-26T13:58:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:00 smithi033 ceph-mon[27845]: pgmap v4677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:00 smithi096 ceph-mon[32753]: pgmap v4677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:00 smithi194 ceph-mon[33017]: pgmap v4677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:02 smithi033 ceph-mon[27845]: pgmap v4678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:02 smithi096 ceph-mon[32753]: pgmap v4678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:02 smithi194 ceph-mon[33017]: pgmap v4678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7621.802450 secs 2024-04-26T13:58:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:04 smithi033 ceph-mon[27845]: pgmap v4679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7624.562333 secs 2024-04-26T13:58:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7621.802450 secs 2024-04-26T13:58:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:04 smithi096 ceph-mon[32753]: pgmap v4679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7624.562333 secs 2024-04-26T13:58:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7621.802450 secs 2024-04-26T13:58:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:04 smithi194 ceph-mon[33017]: pgmap v4679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7624.562333 secs 2024-04-26T13:58:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:06 smithi096 ceph-mon[32753]: pgmap v4680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:06 smithi194 ceph-mon[33017]: pgmap v4680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:06 smithi033 ceph-mon[27845]: pgmap v4680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:08.338+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:08.339+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:08.339+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:58:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:08 smithi096 ceph-mon[32753]: pgmap v4681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:08 smithi194 ceph-mon[33017]: pgmap v4681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:08 smithi033 ceph-mon[27845]: pgmap v4681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7626.802513 secs 2024-04-26T13:58:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7629.562396 secs 2024-04-26T13:58:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7626.802513 secs 2024-04-26T13:58:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7629.562396 secs 2024-04-26T13:58:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7626.802513 secs 2024-04-26T13:58:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7629.562396 secs 2024-04-26T13:58:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:10 smithi194 ceph-mon[33017]: pgmap v4682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:10 smithi033 ceph-mon[27845]: pgmap v4682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:10 smithi096 ceph-mon[32753]: pgmap v4682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:12 smithi194 ceph-mon[33017]: pgmap v4683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:12 smithi033 ceph-mon[27845]: pgmap v4683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:12 smithi096 ceph-mon[32753]: pgmap v4683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7631.802598 secs 2024-04-26T13:58:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:14 smithi194 ceph-mon[33017]: pgmap v4684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7634.562491 secs 2024-04-26T13:58:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7631.802598 secs 2024-04-26T13:58:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:14 smithi033 ceph-mon[27845]: pgmap v4684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7634.562491 secs 2024-04-26T13:58:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7631.802598 secs 2024-04-26T13:58:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:14 smithi096 ceph-mon[32753]: pgmap v4684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7634.562491 secs 2024-04-26T13:58:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:16 smithi194 ceph-mon[33017]: pgmap v4685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:16 smithi033 ceph-mon[27845]: pgmap v4685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:16 smithi096 ceph-mon[32753]: pgmap v4685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:18 smithi194 ceph-mon[33017]: pgmap v4686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:18 smithi033 ceph-mon[27845]: pgmap v4686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:18 smithi096 ceph-mon[32753]: pgmap v4686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7636.802682 secs 2024-04-26T13:58:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7639.562587 secs 2024-04-26T13:58:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7636.802682 secs 2024-04-26T13:58:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7639.562587 secs 2024-04-26T13:58:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7636.802682 secs 2024-04-26T13:58:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7639.562587 secs 2024-04-26T13:58:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:20 smithi194 ceph-mon[33017]: pgmap v4687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:20 smithi033 ceph-mon[27845]: pgmap v4687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:20 smithi096 ceph-mon[32753]: pgmap v4687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:22 smithi194 ceph-mon[33017]: pgmap v4688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:58:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:22 smithi033 ceph-mon[27845]: pgmap v4688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:58:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:22 smithi096 ceph-mon[32753]: pgmap v4688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:58:23.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:23.351+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:23.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:23.351+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:23.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:23.351+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:58:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:58:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:58:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:23 smithi194 ceph-mon[33017]: pgmap v4689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:23 smithi194 ceph-mon[33017]: pgmap v4690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:58:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:58:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:58:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:23 smithi033 ceph-mon[27845]: pgmap v4689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:23 smithi033 ceph-mon[27845]: pgmap v4690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:58:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:58:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:58:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:23 smithi096 ceph-mon[32753]: pgmap v4689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:23 smithi096 ceph-mon[32753]: pgmap v4690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:58:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7641.802765 secs 2024-04-26T13:58:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7644.562710 secs 2024-04-26T13:58:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7641.802765 secs 2024-04-26T13:58:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7644.562710 secs 2024-04-26T13:58:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7641.802765 secs 2024-04-26T13:58:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7644.562710 secs 2024-04-26T13:58:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:25 smithi194 ceph-mon[33017]: pgmap v4691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:25 smithi033 ceph-mon[27845]: pgmap v4691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:25 smithi096 ceph-mon[32753]: pgmap v4691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:27 smithi194 ceph-mon[33017]: pgmap v4692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:27 smithi033 ceph-mon[27845]: pgmap v4692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:27 smithi096 ceph-mon[32753]: pgmap v4692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7646.802850 secs 2024-04-26T13:58:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7649.562826 secs 2024-04-26T13:58:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7646.802850 secs 2024-04-26T13:58:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7649.562826 secs 2024-04-26T13:58:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7646.802850 secs 2024-04-26T13:58:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7649.562826 secs 2024-04-26T13:58:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:29 smithi194 ceph-mon[33017]: pgmap v4693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:29 smithi033 ceph-mon[27845]: pgmap v4693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:29 smithi096 ceph-mon[32753]: pgmap v4693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:31 smithi194 ceph-mon[33017]: pgmap v4694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:31 smithi033 ceph-mon[27845]: pgmap v4694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:31 smithi096 ceph-mon[32753]: pgmap v4694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:58:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7651.802936 secs 2024-04-26T13:58:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:33 smithi194 ceph-mon[33017]: pgmap v4695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:58:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7654.562943 secs 2024-04-26T13:58:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7651.802936 secs 2024-04-26T13:58:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:33 smithi033 ceph-mon[27845]: pgmap v4695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:58:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7654.562943 secs 2024-04-26T13:58:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7651.802936 secs 2024-04-26T13:58:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:33 smithi096 ceph-mon[32753]: pgmap v4695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:58:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7654.562943 secs 2024-04-26T13:58:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:35 smithi194 ceph-mon[33017]: pgmap v4696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:35 smithi033 ceph-mon[27845]: pgmap v4696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:35 smithi096 ceph-mon[32753]: pgmap v4696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:37 smithi194 ceph-mon[33017]: pgmap v4697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:37 smithi033 ceph-mon[27845]: pgmap v4697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:37 smithi096 ceph-mon[32753]: pgmap v4697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:38.362+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:38.362+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:38.362+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:58:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7656.802997 secs 2024-04-26T13:58:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7659.563056 secs 2024-04-26T13:58:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7656.802997 secs 2024-04-26T13:58:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7659.563056 secs 2024-04-26T13:58:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7656.802997 secs 2024-04-26T13:58:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7659.563056 secs 2024-04-26T13:58:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:39 smithi194 ceph-mon[33017]: pgmap v4698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:39 smithi033 ceph-mon[27845]: pgmap v4698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:39 smithi096 ceph-mon[32753]: pgmap v4698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:41 smithi194 ceph-mon[33017]: pgmap v4699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:41 smithi033 ceph-mon[27845]: pgmap v4699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:41 smithi096 ceph-mon[32753]: pgmap v4699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7661.803088 secs 2024-04-26T13:58:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:43 smithi194 ceph-mon[33017]: pgmap v4700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7664.563172 secs 2024-04-26T13:58:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7661.803088 secs 2024-04-26T13:58:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:43 smithi033 ceph-mon[27845]: pgmap v4700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7664.563172 secs 2024-04-26T13:58:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7661.803088 secs 2024-04-26T13:58:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:43 smithi096 ceph-mon[32753]: pgmap v4700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7664.563172 secs 2024-04-26T13:58:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:45 smithi194 ceph-mon[33017]: pgmap v4701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:45 smithi033 ceph-mon[27845]: pgmap v4701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:45 smithi096 ceph-mon[32753]: pgmap v4701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:47 smithi194 ceph-mon[33017]: pgmap v4702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:47 smithi033 ceph-mon[27845]: pgmap v4702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:47 smithi096 ceph-mon[32753]: pgmap v4702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7666.803149 secs 2024-04-26T13:58:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7669.563285 secs 2024-04-26T13:58:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7666.803149 secs 2024-04-26T13:58:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7669.563285 secs 2024-04-26T13:58:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7666.803149 secs 2024-04-26T13:58:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7669.563285 secs 2024-04-26T13:58:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:49 smithi194 ceph-mon[33017]: pgmap v4703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:49 smithi033 ceph-mon[27845]: pgmap v4703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:49 smithi096 ceph-mon[32753]: pgmap v4703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:51 smithi194 ceph-mon[33017]: pgmap v4704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:51 smithi033 ceph-mon[27845]: pgmap v4704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:51 smithi096 ceph-mon[32753]: pgmap v4704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:53.378+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:53.378+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:58:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:58:53.378+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:58:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7671.803231 secs 2024-04-26T13:58:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:53 smithi194 ceph-mon[33017]: pgmap v4705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7674.563395 secs 2024-04-26T13:58:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7671.803231 secs 2024-04-26T13:58:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:53 smithi033 ceph-mon[27845]: pgmap v4705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7674.563395 secs 2024-04-26T13:58:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7671.803231 secs 2024-04-26T13:58:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:53 smithi096 ceph-mon[32753]: pgmap v4705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7674.563395 secs 2024-04-26T13:58:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:55 smithi194 ceph-mon[33017]: pgmap v4706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:55 smithi033 ceph-mon[27845]: pgmap v4706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:55 smithi096 ceph-mon[32753]: pgmap v4706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:58:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:57 smithi194 ceph-mon[33017]: pgmap v4707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:57 smithi033 ceph-mon[27845]: pgmap v4707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:57 smithi096 ceph-mon[32753]: pgmap v4707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7676.803320 secs 2024-04-26T13:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7679.563513 secs 2024-04-26T13:58:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7676.803320 secs 2024-04-26T13:58:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7679.563513 secs 2024-04-26T13:58:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7676.803320 secs 2024-04-26T13:58:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7679.563513 secs 2024-04-26T13:59:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:58:59 smithi194 ceph-mon[33017]: pgmap v4708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:58:59 smithi033 ceph-mon[27845]: pgmap v4708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:58:59 smithi096 ceph-mon[32753]: pgmap v4708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:01 smithi033 ceph-mon[27845]: pgmap v4709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:01 smithi096 ceph-mon[32753]: pgmap v4709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:01 smithi194 ceph-mon[33017]: pgmap v4709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7681.803399 secs 2024-04-26T13:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:03 smithi033 ceph-mon[27845]: pgmap v4710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7684.563632 secs 2024-04-26T13:59:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7681.803399 secs 2024-04-26T13:59:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:03 smithi096 ceph-mon[32753]: pgmap v4710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7684.563632 secs 2024-04-26T13:59:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7681.803399 secs 2024-04-26T13:59:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:03 smithi194 ceph-mon[33017]: pgmap v4710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7684.563632 secs 2024-04-26T13:59:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:05 smithi033 ceph-mon[27845]: pgmap v4711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:05 smithi096 ceph-mon[32753]: pgmap v4711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:05 smithi194 ceph-mon[33017]: pgmap v4711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:07 smithi033 ceph-mon[27845]: pgmap v4712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:07 smithi096 ceph-mon[32753]: pgmap v4712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:07 smithi194 ceph-mon[33017]: pgmap v4712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:08.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:08.380+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:08.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:08.380+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:08.380+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:59:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7686.803472 secs 2024-04-26T13:59:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7689.563766 secs 2024-04-26T13:59:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7686.803472 secs 2024-04-26T13:59:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7689.563766 secs 2024-04-26T13:59:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7686.803472 secs 2024-04-26T13:59:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7689.563766 secs 2024-04-26T13:59:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:09 smithi033 ceph-mon[27845]: pgmap v4713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:09 smithi096 ceph-mon[32753]: pgmap v4713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:09 smithi194 ceph-mon[33017]: pgmap v4713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:11 smithi033 ceph-mon[27845]: pgmap v4714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:11 smithi096 ceph-mon[32753]: pgmap v4714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:11 smithi194 ceph-mon[33017]: pgmap v4714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7691.803549 secs 2024-04-26T13:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:13 smithi033 ceph-mon[27845]: pgmap v4715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7694.563900 secs 2024-04-26T13:59:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7691.803549 secs 2024-04-26T13:59:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:13 smithi096 ceph-mon[32753]: pgmap v4715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7694.563900 secs 2024-04-26T13:59:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7691.803549 secs 2024-04-26T13:59:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:13 smithi194 ceph-mon[33017]: pgmap v4715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7694.563900 secs 2024-04-26T13:59:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:15 smithi033 ceph-mon[27845]: pgmap v4716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:15 smithi096 ceph-mon[32753]: pgmap v4716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:15 smithi194 ceph-mon[33017]: pgmap v4716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:17 smithi033 ceph-mon[27845]: pgmap v4717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:17 smithi096 ceph-mon[32753]: pgmap v4717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:17 smithi194 ceph-mon[33017]: pgmap v4717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7696.803631 secs 2024-04-26T13:59:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7699.564042 secs 2024-04-26T13:59:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7696.803631 secs 2024-04-26T13:59:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7699.564042 secs 2024-04-26T13:59:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7696.803631 secs 2024-04-26T13:59:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7699.564042 secs 2024-04-26T13:59:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:19 smithi033 ceph-mon[27845]: pgmap v4718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:19 smithi096 ceph-mon[32753]: pgmap v4718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:19 smithi194 ceph-mon[33017]: pgmap v4718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:21 smithi033 ceph-mon[27845]: pgmap v4719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:21 smithi096 ceph-mon[32753]: pgmap v4719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:21 smithi194 ceph-mon[33017]: pgmap v4719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:23.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:23.098+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T13:59:23.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:23.098+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T13:59:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:23.099+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T13:59:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:23.099+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T13:59:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:23.396+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:23.396+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:23.396+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:59:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7701.803714 secs 2024-04-26T13:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:23 smithi096 ceph-mon[32753]: pgmap v4720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7704.564146 secs 2024-04-26T13:59:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7701.803714 secs 2024-04-26T13:59:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:23 smithi194 ceph-mon[33017]: pgmap v4720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:59:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7704.564146 secs 2024-04-26T13:59:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7701.803714 secs 2024-04-26T13:59:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:23 smithi033 ceph-mon[27845]: pgmap v4720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T13:59:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7704.564146 secs 2024-04-26T13:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:24 smithi194 ceph-mon[33017]: pgmap v4721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:24 smithi194 ceph-mon[33017]: pgmap v4722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T13:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:59:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:59:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:59:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:24 smithi033 ceph-mon[27845]: pgmap v4721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:24 smithi033 ceph-mon[27845]: pgmap v4722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T13:59:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:59:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T13:59:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T13:59:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:24 smithi096 ceph-mon[32753]: pgmap v4721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:24 smithi096 ceph-mon[32753]: pgmap v4722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T13:59:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T13:59:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:27 smithi194 ceph-mon[33017]: pgmap v4723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:27.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:27 smithi033 ceph-mon[27845]: pgmap v4723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:27 smithi096 ceph-mon[32753]: pgmap v4723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7706.803794 secs 2024-04-26T13:59:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7709.564263 secs 2024-04-26T13:59:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7706.803794 secs 2024-04-26T13:59:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7709.564263 secs 2024-04-26T13:59:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7706.803794 secs 2024-04-26T13:59:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7709.564263 secs 2024-04-26T13:59:29.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:29 smithi096 ceph-mon[32753]: pgmap v4724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:29 smithi194 ceph-mon[33017]: pgmap v4724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:29 smithi033 ceph-mon[27845]: pgmap v4724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:31 smithi194 ceph-mon[33017]: pgmap v4725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:31 smithi033 ceph-mon[27845]: pgmap v4725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:31 smithi096 ceph-mon[32753]: pgmap v4725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:33 smithi194 ceph-mon[33017]: pgmap v4726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7711.803876 secs 2024-04-26T13:59:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7714.564326 secs 2024-04-26T13:59:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:33 smithi033 ceph-mon[27845]: pgmap v4726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7711.803876 secs 2024-04-26T13:59:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7714.564326 secs 2024-04-26T13:59:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:33 smithi096 ceph-mon[32753]: pgmap v4726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7711.803876 secs 2024-04-26T13:59:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7714.564326 secs 2024-04-26T13:59:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:35 smithi194 ceph-mon[33017]: pgmap v4727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:59:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:35 smithi033 ceph-mon[27845]: pgmap v4727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:59:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:35 smithi096 ceph-mon[32753]: pgmap v4727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T13:59:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:37 smithi194 ceph-mon[33017]: pgmap v4728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:37 smithi033 ceph-mon[27845]: pgmap v4728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:37 smithi096 ceph-mon[32753]: pgmap v4728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:38.404+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:38.404+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:59:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:38.404+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7716.803962 secs 2024-04-26T13:59:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7719.564451 secs 2024-04-26T13:59:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7716.803962 secs 2024-04-26T13:59:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7719.564451 secs 2024-04-26T13:59:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7716.803962 secs 2024-04-26T13:59:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7719.564451 secs 2024-04-26T13:59:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:39 smithi194 ceph-mon[33017]: pgmap v4729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:39 smithi033 ceph-mon[27845]: pgmap v4729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:39 smithi096 ceph-mon[32753]: pgmap v4729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:41 smithi194 ceph-mon[33017]: pgmap v4730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:41 smithi033 ceph-mon[27845]: pgmap v4730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:41 smithi096 ceph-mon[32753]: pgmap v4730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:43 smithi194 ceph-mon[33017]: pgmap v4731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7721.804045 secs 2024-04-26T13:59:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7724.564564 secs 2024-04-26T13:59:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:43 smithi033 ceph-mon[27845]: pgmap v4731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7721.804045 secs 2024-04-26T13:59:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7724.564564 secs 2024-04-26T13:59:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:43 smithi096 ceph-mon[32753]: pgmap v4731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7721.804045 secs 2024-04-26T13:59:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7724.564564 secs 2024-04-26T13:59:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:45 smithi194 ceph-mon[33017]: pgmap v4732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:45 smithi033 ceph-mon[27845]: pgmap v4732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:45 smithi096 ceph-mon[32753]: pgmap v4732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:47 smithi194 ceph-mon[33017]: pgmap v4733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:47 smithi033 ceph-mon[27845]: pgmap v4733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:47 smithi096 ceph-mon[32753]: pgmap v4733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7726.804129 secs 2024-04-26T13:59:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7729.564680 secs 2024-04-26T13:59:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7726.804129 secs 2024-04-26T13:59:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7729.564680 secs 2024-04-26T13:59:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7726.804129 secs 2024-04-26T13:59:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7729.564680 secs 2024-04-26T13:59:49.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:49 smithi033 ceph-mon[27845]: pgmap v4734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:49 smithi194 ceph-mon[33017]: pgmap v4734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:49 smithi096 ceph-mon[32753]: pgmap v4734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:51 smithi194 ceph-mon[33017]: pgmap v4735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:51 smithi033 ceph-mon[27845]: pgmap v4735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:51 smithi096 ceph-mon[32753]: pgmap v4735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T13:59:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:53.412+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:53.412+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T13:59:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T13:59:53.412+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T13:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:53 smithi194 ceph-mon[33017]: pgmap v4736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7731.804231 secs 2024-04-26T13:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7734.564800 secs 2024-04-26T13:59:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:53 smithi033 ceph-mon[27845]: pgmap v4736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7731.804231 secs 2024-04-26T13:59:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7734.564800 secs 2024-04-26T13:59:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:53 smithi096 ceph-mon[32753]: pgmap v4736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7731.804231 secs 2024-04-26T13:59:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7734.564800 secs 2024-04-26T13:59:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:55 smithi194 ceph-mon[33017]: pgmap v4737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:55 smithi033 ceph-mon[27845]: pgmap v4737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:55 smithi096 ceph-mon[32753]: pgmap v4737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:57 smithi194 ceph-mon[33017]: pgmap v4738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:57 smithi033 ceph-mon[27845]: pgmap v4738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:57 smithi096 ceph-mon[32753]: pgmap v4738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T13:59:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7736.804325 secs 2024-04-26T13:59:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7739.564882 secs 2024-04-26T13:59:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7736.804325 secs 2024-04-26T13:59:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7739.564882 secs 2024-04-26T13:59:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7736.804325 secs 2024-04-26T13:59:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7739.564882 secs 2024-04-26T13:59:59.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 13:59:59 smithi096 ceph-mon[32753]: pgmap v4739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 13:59:59 smithi194 ceph-mon[33017]: pgmap v4739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T13:59:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 13:59:59 smithi033 ceph-mon[27845]: pgmap v4739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:00:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:00:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:00:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:01 smithi194 ceph-mon[33017]: pgmap v4740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:01.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:01 smithi096 ceph-mon[32753]: pgmap v4740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:01.935 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:01 smithi033 ceph-mon[27845]: pgmap v4740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:03 smithi194 ceph-mon[33017]: pgmap v4741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7741.804382 secs 2024-04-26T14:00:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7744.565016 secs 2024-04-26T14:00:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:03 smithi033 ceph-mon[27845]: pgmap v4741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7741.804382 secs 2024-04-26T14:00:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7744.565016 secs 2024-04-26T14:00:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:03 smithi096 ceph-mon[32753]: pgmap v4741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7741.804382 secs 2024-04-26T14:00:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7744.565016 secs 2024-04-26T14:00:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:05 smithi194 ceph-mon[33017]: pgmap v4742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:05 smithi033 ceph-mon[27845]: pgmap v4742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:05 smithi096 ceph-mon[32753]: pgmap v4742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:07 smithi194 ceph-mon[33017]: pgmap v4743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:07.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:07 smithi033 ceph-mon[27845]: pgmap v4743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:07 smithi096 ceph-mon[32753]: pgmap v4743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:08.423+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:08.423+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:00:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:08.423+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7746.804438 secs 2024-04-26T14:00:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7749.565141 secs 2024-04-26T14:00:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7746.804438 secs 2024-04-26T14:00:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7749.565141 secs 2024-04-26T14:00:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7746.804438 secs 2024-04-26T14:00:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7749.565141 secs 2024-04-26T14:00:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:09 smithi033 ceph-mon[27845]: pgmap v4744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:09 smithi096 ceph-mon[32753]: pgmap v4744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:09 smithi194 ceph-mon[33017]: pgmap v4744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:11.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:11 smithi033 ceph-mon[27845]: pgmap v4745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:11 smithi096 ceph-mon[32753]: pgmap v4745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:11 smithi194 ceph-mon[33017]: pgmap v4745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:13 smithi033 ceph-mon[27845]: pgmap v4746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7751.804495 secs 2024-04-26T14:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7754.565263 secs 2024-04-26T14:00:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:13 smithi096 ceph-mon[32753]: pgmap v4746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7751.804495 secs 2024-04-26T14:00:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7754.565263 secs 2024-04-26T14:00:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:13 smithi194 ceph-mon[33017]: pgmap v4746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7751.804495 secs 2024-04-26T14:00:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7754.565263 secs 2024-04-26T14:00:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:15 smithi033 ceph-mon[27845]: pgmap v4747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:15 smithi096 ceph-mon[32753]: pgmap v4747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:15 smithi194 ceph-mon[33017]: pgmap v4747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:17 smithi033 ceph-mon[27845]: pgmap v4748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:17 smithi096 ceph-mon[32753]: pgmap v4748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:17 smithi194 ceph-mon[33017]: pgmap v4748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7756.804553 secs 2024-04-26T14:00:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7759.565375 secs 2024-04-26T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7756.804553 secs 2024-04-26T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7759.565375 secs 2024-04-26T14:00:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7756.804553 secs 2024-04-26T14:00:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7759.565375 secs 2024-04-26T14:00:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:19 smithi033 ceph-mon[27845]: pgmap v4749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:19 smithi096 ceph-mon[32753]: pgmap v4749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:19 smithi194 ceph-mon[33017]: pgmap v4749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:21 smithi033 ceph-mon[27845]: pgmap v4750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:21 smithi096 ceph-mon[32753]: pgmap v4750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:21 smithi194 ceph-mon[33017]: pgmap v4750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:23.438+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:23.438+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:23.438+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:00:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:23 smithi033 ceph-mon[27845]: pgmap v4751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7761.804608 secs 2024-04-26T14:00:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7764.565450 secs 2024-04-26T14:00:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:23 smithi096 ceph-mon[32753]: pgmap v4751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7761.804608 secs 2024-04-26T14:00:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7764.565450 secs 2024-04-26T14:00:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:23 smithi194 ceph-mon[33017]: pgmap v4751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7761.804608 secs 2024-04-26T14:00:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7764.565450 secs 2024-04-26T14:00:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:25 smithi033 ceph-mon[27845]: pgmap v4752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:00:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:25 smithi096 ceph-mon[32753]: pgmap v4752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:00:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:25 smithi194 ceph-mon[33017]: pgmap v4752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:00:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:00:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:00:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:00:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:00:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:00:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:00:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:00:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:00:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:00:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:27 smithi096 ceph-mon[32753]: pgmap v4753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T14:00:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:27 smithi096 ceph-mon[32753]: pgmap v4754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:27 smithi194 ceph-mon[33017]: pgmap v4753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T14:00:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:27 smithi194 ceph-mon[33017]: pgmap v4754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:27 smithi033 ceph-mon[27845]: pgmap v4753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T14:00:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:27 smithi033 ceph-mon[27845]: pgmap v4754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7766.804682 secs 2024-04-26T14:00:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7769.565568 secs 2024-04-26T14:00:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7766.804682 secs 2024-04-26T14:00:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7769.565568 secs 2024-04-26T14:00:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7766.804682 secs 2024-04-26T14:00:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7769.565568 secs 2024-04-26T14:00:30.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:29 smithi096 ceph-mon[32753]: pgmap v4755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:29 smithi194 ceph-mon[33017]: pgmap v4755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:29 smithi033 ceph-mon[27845]: pgmap v4755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:30 smithi194 ceph-mon[33017]: pgmap v4756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:30 smithi033 ceph-mon[27845]: pgmap v4756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:30 smithi096 ceph-mon[32753]: pgmap v4756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:33 smithi194 ceph-mon[33017]: pgmap v4757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:33 smithi033 ceph-mon[27845]: pgmap v4757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:33 smithi096 ceph-mon[32753]: pgmap v4757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:00:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7771.804724 secs 2024-04-26T14:00:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7774.565720 secs 2024-04-26T14:00:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7771.804724 secs 2024-04-26T14:00:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7774.565720 secs 2024-04-26T14:00:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7771.804724 secs 2024-04-26T14:00:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7774.565720 secs 2024-04-26T14:00:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:35 smithi033 ceph-mon[27845]: pgmap v4758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:00:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:35 smithi096 ceph-mon[32753]: pgmap v4758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:00:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:35 smithi194 ceph-mon[33017]: pgmap v4758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:00:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:37 smithi033 ceph-mon[27845]: pgmap v4759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:00:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:37 smithi096 ceph-mon[32753]: pgmap v4759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:00:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:37 smithi194 ceph-mon[33017]: pgmap v4759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:00:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:38.451+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:38.451+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:38.451+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:00:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7776.804781 secs 2024-04-26T14:00:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:39 smithi033 ceph-mon[27845]: pgmap v4760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7779.565857 secs 2024-04-26T14:00:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7776.804781 secs 2024-04-26T14:00:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:39 smithi096 ceph-mon[32753]: pgmap v4760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7779.565857 secs 2024-04-26T14:00:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7776.804781 secs 2024-04-26T14:00:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:39 smithi194 ceph-mon[33017]: pgmap v4760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7779.565857 secs 2024-04-26T14:00:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:41 smithi033 ceph-mon[27845]: pgmap v4761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:41 smithi096 ceph-mon[32753]: pgmap v4761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:41 smithi194 ceph-mon[33017]: pgmap v4761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:43 smithi033 ceph-mon[27845]: pgmap v4762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:43 smithi096 ceph-mon[32753]: pgmap v4762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:43 smithi194 ceph-mon[33017]: pgmap v4762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7781.804838 secs 2024-04-26T14:00:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7784.565958 secs 2024-04-26T14:00:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7781.804838 secs 2024-04-26T14:00:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7784.565958 secs 2024-04-26T14:00:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7781.804838 secs 2024-04-26T14:00:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7784.565958 secs 2024-04-26T14:00:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:45 smithi033 ceph-mon[27845]: pgmap v4763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:45 smithi096 ceph-mon[32753]: pgmap v4763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:45.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:45 smithi194 ceph-mon[33017]: pgmap v4763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:47 smithi033 ceph-mon[27845]: pgmap v4764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:47 smithi096 ceph-mon[32753]: pgmap v4764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:47 smithi194 ceph-mon[33017]: pgmap v4764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7786.804896 secs 2024-04-26T14:00:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:49 smithi033 ceph-mon[27845]: pgmap v4765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7789.566088 secs 2024-04-26T14:00:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7786.804896 secs 2024-04-26T14:00:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:49 smithi096 ceph-mon[32753]: pgmap v4765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7789.566088 secs 2024-04-26T14:00:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7786.804896 secs 2024-04-26T14:00:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:49 smithi194 ceph-mon[33017]: pgmap v4765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7789.566088 secs 2024-04-26T14:00:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:51 smithi033 ceph-mon[27845]: pgmap v4766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:51 smithi096 ceph-mon[32753]: pgmap v4766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:51 smithi194 ceph-mon[33017]: pgmap v4766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:53.409 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:53 smithi194 ceph-mon[33017]: pgmap v4767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:53 smithi033 ceph-mon[27845]: pgmap v4767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:53 smithi096 ceph-mon[32753]: pgmap v4767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:53.467+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:53.467+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:00:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:00:53.467+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:00:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7791.804972 secs 2024-04-26T14:00:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7794.566204 secs 2024-04-26T14:00:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7791.804972 secs 2024-04-26T14:00:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7794.566204 secs 2024-04-26T14:00:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7791.804972 secs 2024-04-26T14:00:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7794.566204 secs 2024-04-26T14:00:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:55 smithi033 ceph-mon[27845]: pgmap v4768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:55 smithi096 ceph-mon[32753]: pgmap v4768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:55.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:55 smithi194 ceph-mon[33017]: pgmap v4768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:57 smithi096 ceph-mon[32753]: pgmap v4769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:57 smithi194 ceph-mon[33017]: pgmap v4769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:57 smithi033 ceph-mon[27845]: pgmap v4769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:00:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7796.804996 secs 2024-04-26T14:00:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:59 smithi096 ceph-mon[32753]: pgmap v4770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7799.566299 secs 2024-04-26T14:00:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7796.804996 secs 2024-04-26T14:00:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:59 smithi194 ceph-mon[33017]: pgmap v4770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7799.566299 secs 2024-04-26T14:00:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7796.804996 secs 2024-04-26T14:00:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:59 smithi033 ceph-mon[27845]: pgmap v4770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:00:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7799.566299 secs 2024-04-26T14:01:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:01 smithi194 ceph-mon[33017]: pgmap v4771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:01 smithi033 ceph-mon[27845]: pgmap v4771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:01 smithi096 ceph-mon[32753]: pgmap v4771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:03 smithi194 ceph-mon[33017]: pgmap v4772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:03 smithi033 ceph-mon[27845]: pgmap v4772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:03 smithi096 ceph-mon[32753]: pgmap v4772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7801.805058 secs 2024-04-26T14:01:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7804.566415 secs 2024-04-26T14:01:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7801.805058 secs 2024-04-26T14:01:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7804.566415 secs 2024-04-26T14:01:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7801.805058 secs 2024-04-26T14:01:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7804.566415 secs 2024-04-26T14:01:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:05 smithi194 ceph-mon[33017]: pgmap v4773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:05 smithi033 ceph-mon[27845]: pgmap v4773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:05 smithi096 ceph-mon[32753]: pgmap v4773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:07 smithi194 ceph-mon[33017]: pgmap v4774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:07 smithi033 ceph-mon[27845]: pgmap v4774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:07 smithi096 ceph-mon[32753]: pgmap v4774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:08.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:08.480+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:08.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:08.480+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:01:08.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:08.480+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7806.805137 secs 2024-04-26T14:01:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:09 smithi194 ceph-mon[33017]: pgmap v4775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7809.566493 secs 2024-04-26T14:01:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7806.805137 secs 2024-04-26T14:01:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:09 smithi033 ceph-mon[27845]: pgmap v4775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7809.566493 secs 2024-04-26T14:01:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7806.805137 secs 2024-04-26T14:01:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:09 smithi096 ceph-mon[32753]: pgmap v4775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7809.566493 secs 2024-04-26T14:01:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:11 smithi194 ceph-mon[33017]: pgmap v4776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:11 smithi033 ceph-mon[27845]: pgmap v4776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:11 smithi096 ceph-mon[32753]: pgmap v4776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:13 smithi194 ceph-mon[33017]: pgmap v4777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:13 smithi033 ceph-mon[27845]: pgmap v4777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:13 smithi096 ceph-mon[32753]: pgmap v4777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7811.805220 secs 2024-04-26T14:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7814.566574 secs 2024-04-26T14:01:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7811.805220 secs 2024-04-26T14:01:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7814.566574 secs 2024-04-26T14:01:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7811.805220 secs 2024-04-26T14:01:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7814.566574 secs 2024-04-26T14:01:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:15 smithi096 ceph-mon[32753]: pgmap v4778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:15 smithi194 ceph-mon[33017]: pgmap v4778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:15 smithi033 ceph-mon[27845]: pgmap v4778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:17 smithi194 ceph-mon[33017]: pgmap v4779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:17 smithi033 ceph-mon[27845]: pgmap v4779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:17 smithi096 ceph-mon[32753]: pgmap v4779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7816.805278 secs 2024-04-26T14:01:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7819.566697 secs 2024-04-26T14:01:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7816.805278 secs 2024-04-26T14:01:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7819.566697 secs 2024-04-26T14:01:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7816.805278 secs 2024-04-26T14:01:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7819.566697 secs 2024-04-26T14:01:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:19 smithi194 ceph-mon[33017]: pgmap v4780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:19 smithi033 ceph-mon[27845]: pgmap v4780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:19 smithi096 ceph-mon[32753]: pgmap v4780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:21 smithi194 ceph-mon[33017]: pgmap v4781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:21 smithi033 ceph-mon[27845]: pgmap v4781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:21 smithi096 ceph-mon[32753]: pgmap v4781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:23.490+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:23.490+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:23.490+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:01:24.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:23 smithi033 ceph-mon[27845]: pgmap v4782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:24.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7821.805360 secs 2024-04-26T14:01:24.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7824.566811 secs 2024-04-26T14:01:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:23 smithi194 ceph-mon[33017]: pgmap v4782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7821.805360 secs 2024-04-26T14:01:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7824.566811 secs 2024-04-26T14:01:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:23 smithi096 ceph-mon[32753]: pgmap v4782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7821.805360 secs 2024-04-26T14:01:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7824.566811 secs 2024-04-26T14:01:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:24 smithi194 ceph-mon[33017]: pgmap v4783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:24 smithi033 ceph-mon[27845]: pgmap v4783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:24 smithi096 ceph-mon[32753]: pgmap v4783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:27 smithi033 ceph-mon[27845]: pgmap v4784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:01:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:27 smithi096 ceph-mon[32753]: pgmap v4784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:01:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:27 smithi194 ceph-mon[33017]: pgmap v4784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7826.805443 secs 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7829.566929 secs 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:01:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7826.805443 secs 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7829.566929 secs 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:01:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7826.805443 secs 2024-04-26T14:01:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7829.566929 secs 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:01:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:29 smithi194 ceph-mon[33017]: pgmap v4785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:29 smithi194 ceph-mon[33017]: pgmap v4786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:01:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:29 smithi194 ceph-mon[33017]: pgmap v4787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:01:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:29 smithi033 ceph-mon[27845]: pgmap v4785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:29 smithi033 ceph-mon[27845]: pgmap v4786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:01:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:29 smithi033 ceph-mon[27845]: pgmap v4787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:01:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:29 smithi096 ceph-mon[32753]: pgmap v4785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:29 smithi096 ceph-mon[32753]: pgmap v4786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:01:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:29 smithi096 ceph-mon[32753]: pgmap v4787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:01:31.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:30 smithi096 ceph-mon[32753]: pgmap v4788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:30 smithi194 ceph-mon[33017]: pgmap v4788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:30 smithi033 ceph-mon[27845]: pgmap v4788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:33 smithi194 ceph-mon[33017]: pgmap v4789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7831.805526 secs 2024-04-26T14:01:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7834.567043 secs 2024-04-26T14:01:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:33 smithi033 ceph-mon[27845]: pgmap v4789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7831.805526 secs 2024-04-26T14:01:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7834.567043 secs 2024-04-26T14:01:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:33 smithi096 ceph-mon[32753]: pgmap v4789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7831.805526 secs 2024-04-26T14:01:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7834.567043 secs 2024-04-26T14:01:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:35 smithi194 ceph-mon[33017]: pgmap v4790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:35 smithi033 ceph-mon[27845]: pgmap v4790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:35 smithi096 ceph-mon[32753]: pgmap v4790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:37 smithi033 ceph-mon[27845]: pgmap v4791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:37 smithi096 ceph-mon[32753]: pgmap v4791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:37 smithi194 ceph-mon[33017]: pgmap v4791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:01:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:38.120+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:01:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:38.120+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:01:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:38.120+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T14:01:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:38.121+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T14:01:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:38.497+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:38.497+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:38.497+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:01:38.870 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7836.805614 secs 2024-04-26T14:01:38.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7839.567152 secs 2024-04-26T14:01:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7836.805614 secs 2024-04-26T14:01:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7839.567152 secs 2024-04-26T14:01:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7836.805614 secs 2024-04-26T14:01:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7839.567152 secs 2024-04-26T14:01:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:39 smithi033 ceph-mon[27845]: pgmap v4792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:01:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:39 smithi096 ceph-mon[32753]: pgmap v4792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:01:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:39 smithi194 ceph-mon[33017]: pgmap v4792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:01:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:41 smithi033 ceph-mon[27845]: pgmap v4793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:41 smithi096 ceph-mon[32753]: pgmap v4793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:41 smithi194 ceph-mon[33017]: pgmap v4793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:43 smithi033 ceph-mon[27845]: pgmap v4794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7841.805697 secs 2024-04-26T14:01:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7844.567224 secs 2024-04-26T14:01:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:43 smithi096 ceph-mon[32753]: pgmap v4794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7841.805697 secs 2024-04-26T14:01:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7844.567224 secs 2024-04-26T14:01:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:43 smithi194 ceph-mon[33017]: pgmap v4794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7841.805697 secs 2024-04-26T14:01:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7844.567224 secs 2024-04-26T14:01:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:45 smithi033 ceph-mon[27845]: pgmap v4795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:45 smithi096 ceph-mon[32753]: pgmap v4795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:45 smithi194 ceph-mon[33017]: pgmap v4795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:47 smithi033 ceph-mon[27845]: pgmap v4796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:47 smithi096 ceph-mon[32753]: pgmap v4796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:47 smithi194 ceph-mon[33017]: pgmap v4796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7846.805778 secs 2024-04-26T14:01:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7849.567302 secs 2024-04-26T14:01:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7846.805778 secs 2024-04-26T14:01:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7849.567302 secs 2024-04-26T14:01:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7846.805778 secs 2024-04-26T14:01:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7849.567302 secs 2024-04-26T14:01:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:49 smithi033 ceph-mon[27845]: pgmap v4797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:49 smithi096 ceph-mon[32753]: pgmap v4797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:49 smithi194 ceph-mon[33017]: pgmap v4797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:51 smithi033 ceph-mon[27845]: pgmap v4798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:51 smithi096 ceph-mon[32753]: pgmap v4798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:51 smithi194 ceph-mon[33017]: pgmap v4798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:53.513+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:53.513+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:01:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:01:53.513+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:01:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:53 smithi033 ceph-mon[27845]: pgmap v4799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7851.805860 secs 2024-04-26T14:01:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7854.567419 secs 2024-04-26T14:01:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:53 smithi096 ceph-mon[32753]: pgmap v4799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7851.805860 secs 2024-04-26T14:01:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7854.567419 secs 2024-04-26T14:01:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:53 smithi194 ceph-mon[33017]: pgmap v4799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7851.805860 secs 2024-04-26T14:01:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7854.567419 secs 2024-04-26T14:01:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:55 smithi033 ceph-mon[27845]: pgmap v4800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:55 smithi096 ceph-mon[32753]: pgmap v4800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:55 smithi194 ceph-mon[33017]: pgmap v4800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:01:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:57 smithi033 ceph-mon[27845]: pgmap v4801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:57 smithi096 ceph-mon[32753]: pgmap v4801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:57 smithi194 ceph-mon[33017]: pgmap v4801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:01:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7856.805942 secs 2024-04-26T14:01:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7859.567556 secs 2024-04-26T14:01:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7856.805942 secs 2024-04-26T14:01:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7859.567556 secs 2024-04-26T14:01:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7856.805942 secs 2024-04-26T14:01:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7859.567556 secs 2024-04-26T14:02:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:01:59 smithi194 ceph-mon[33017]: pgmap v4802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:01:59 smithi033 ceph-mon[27845]: pgmap v4802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:01:59 smithi096 ceph-mon[32753]: pgmap v4802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:01.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:01 smithi096 ceph-mon[32753]: pgmap v4803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:01 smithi194 ceph-mon[33017]: pgmap v4803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:01 smithi033 ceph-mon[27845]: pgmap v4803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:03 smithi194 ceph-mon[33017]: pgmap v4804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7861.806023 secs 2024-04-26T14:02:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7864.567676 secs 2024-04-26T14:02:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:03 smithi033 ceph-mon[27845]: pgmap v4804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7861.806023 secs 2024-04-26T14:02:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7864.567676 secs 2024-04-26T14:02:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:03 smithi096 ceph-mon[32753]: pgmap v4804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7861.806023 secs 2024-04-26T14:02:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7864.567676 secs 2024-04-26T14:02:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:05 smithi194 ceph-mon[33017]: pgmap v4805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:05 smithi033 ceph-mon[27845]: pgmap v4805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:05 smithi096 ceph-mon[32753]: pgmap v4805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:07 smithi194 ceph-mon[33017]: pgmap v4806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:07 smithi033 ceph-mon[27845]: pgmap v4806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:07 smithi096 ceph-mon[32753]: pgmap v4806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:08.519+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:08.519+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:08.519+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:02:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7866.806086 secs 2024-04-26T14:02:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7869.567794 secs 2024-04-26T14:02:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7866.806086 secs 2024-04-26T14:02:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7869.567794 secs 2024-04-26T14:02:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7866.806086 secs 2024-04-26T14:02:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7869.567794 secs 2024-04-26T14:02:10.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:09 smithi033 ceph-mon[27845]: pgmap v4807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:09 smithi194 ceph-mon[33017]: pgmap v4807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:09 smithi096 ceph-mon[32753]: pgmap v4807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:10 smithi194 ceph-mon[33017]: pgmap v4808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:10 smithi033 ceph-mon[27845]: pgmap v4808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:10 smithi096 ceph-mon[32753]: pgmap v4808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:13 smithi033 ceph-mon[27845]: pgmap v4809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7871.806141 secs 2024-04-26T14:02:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7874.567907 secs 2024-04-26T14:02:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:13 smithi096 ceph-mon[32753]: pgmap v4809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7871.806141 secs 2024-04-26T14:02:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7874.567907 secs 2024-04-26T14:02:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:13 smithi194 ceph-mon[33017]: pgmap v4809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7871.806141 secs 2024-04-26T14:02:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7874.567907 secs 2024-04-26T14:02:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:15 smithi194 ceph-mon[33017]: pgmap v4810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:15 smithi033 ceph-mon[27845]: pgmap v4810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:15 smithi096 ceph-mon[32753]: pgmap v4810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:17 smithi194 ceph-mon[33017]: pgmap v4811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:17 smithi033 ceph-mon[27845]: pgmap v4811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:17 smithi096 ceph-mon[32753]: pgmap v4811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7876.806220 secs 2024-04-26T14:02:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7879.568027 secs 2024-04-26T14:02:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7876.806220 secs 2024-04-26T14:02:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7879.568027 secs 2024-04-26T14:02:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7876.806220 secs 2024-04-26T14:02:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7879.568027 secs 2024-04-26T14:02:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:19 smithi194 ceph-mon[33017]: pgmap v4812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:19 smithi033 ceph-mon[27845]: pgmap v4812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:19 smithi096 ceph-mon[32753]: pgmap v4812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:20 smithi194 ceph-mon[33017]: pgmap v4813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:20 smithi033 ceph-mon[27845]: pgmap v4813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:20 smithi096 ceph-mon[32753]: pgmap v4813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:23.129+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:02:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:23.129+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:02:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:23.130+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:02:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:23.130+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:02:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:23.534+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:23.534+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:23.534+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:02:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:23 smithi033 ceph-mon[27845]: pgmap v4814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7881.806281 secs 2024-04-26T14:02:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7884.568143 secs 2024-04-26T14:02:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:23 smithi096 ceph-mon[32753]: pgmap v4814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7881.806281 secs 2024-04-26T14:02:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7884.568143 secs 2024-04-26T14:02:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:23 smithi194 ceph-mon[33017]: pgmap v4814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7881.806281 secs 2024-04-26T14:02:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7884.568143 secs 2024-04-26T14:02:25.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:25 smithi194 ceph-mon[33017]: pgmap v4815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:25 smithi033 ceph-mon[27845]: pgmap v4815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:25 smithi096 ceph-mon[32753]: pgmap v4815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:27.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:27 smithi033 ceph-mon[27845]: pgmap v4816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:27 smithi096 ceph-mon[32753]: pgmap v4816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:27 smithi194 ceph-mon[33017]: pgmap v4816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7886.806362 secs 2024-04-26T14:02:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7889.568265 secs 2024-04-26T14:02:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7886.806362 secs 2024-04-26T14:02:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7889.568265 secs 2024-04-26T14:02:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7886.806362 secs 2024-04-26T14:02:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7889.568265 secs 2024-04-26T14:02:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:29 smithi033 ceph-mon[27845]: pgmap v4817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:02:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:29 smithi096 ceph-mon[32753]: pgmap v4817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:02:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:29 smithi194 ceph-mon[33017]: pgmap v4817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:02:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:02:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:02:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:02:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:02:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:02:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:02:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:02:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:02:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:02:31.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:31 smithi033 ceph-mon[27845]: pgmap v4818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:02:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:31 smithi033 ceph-mon[27845]: pgmap v4819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:31 smithi096 ceph-mon[32753]: pgmap v4818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:02:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:31 smithi096 ceph-mon[32753]: pgmap v4819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:31 smithi194 ceph-mon[33017]: pgmap v4818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:02:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:31 smithi194 ceph-mon[33017]: pgmap v4819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:33 smithi033 ceph-mon[27845]: pgmap v4820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7891.806444 secs 2024-04-26T14:02:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7894.568374 secs 2024-04-26T14:02:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:33 smithi096 ceph-mon[32753]: pgmap v4820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7891.806444 secs 2024-04-26T14:02:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7894.568374 secs 2024-04-26T14:02:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:33 smithi194 ceph-mon[33017]: pgmap v4820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7891.806444 secs 2024-04-26T14:02:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7894.568374 secs 2024-04-26T14:02:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:35 smithi033 ceph-mon[27845]: pgmap v4821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:02:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:35 smithi096 ceph-mon[32753]: pgmap v4821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:02:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:35 smithi194 ceph-mon[33017]: pgmap v4821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:02:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:37 smithi033 ceph-mon[27845]: pgmap v4822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:37 smithi096 ceph-mon[32753]: pgmap v4822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:37 smithi194 ceph-mon[33017]: pgmap v4822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:38.549+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:38.549+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:38.549+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:02:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7896.806526 secs 2024-04-26T14:02:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7899.568452 secs 2024-04-26T14:02:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7896.806526 secs 2024-04-26T14:02:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7899.568452 secs 2024-04-26T14:02:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7896.806526 secs 2024-04-26T14:02:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7899.568452 secs 2024-04-26T14:02:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:39 smithi194 ceph-mon[33017]: pgmap v4823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:39 smithi033 ceph-mon[27845]: pgmap v4823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:39 smithi096 ceph-mon[32753]: pgmap v4823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:02:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:41 smithi194 ceph-mon[33017]: pgmap v4824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:02:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:41 smithi033 ceph-mon[27845]: pgmap v4824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:02:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:41 smithi096 ceph-mon[32753]: pgmap v4824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:02:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:43 smithi194 ceph-mon[33017]: pgmap v4825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7901.806607 secs 2024-04-26T14:02:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7904.568543 secs 2024-04-26T14:02:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:43 smithi033 ceph-mon[27845]: pgmap v4825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7901.806607 secs 2024-04-26T14:02:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7904.568543 secs 2024-04-26T14:02:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:43 smithi096 ceph-mon[32753]: pgmap v4825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7901.806607 secs 2024-04-26T14:02:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7904.568543 secs 2024-04-26T14:02:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:45 smithi194 ceph-mon[33017]: pgmap v4826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:45 smithi033 ceph-mon[27845]: pgmap v4826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:45 smithi096 ceph-mon[32753]: pgmap v4826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:47 smithi194 ceph-mon[33017]: pgmap v4827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:02:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:47 smithi033 ceph-mon[27845]: pgmap v4827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:02:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:47 smithi096 ceph-mon[32753]: pgmap v4827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:02:49.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7906.806667 secs 2024-04-26T14:02:49.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7909.568661 secs 2024-04-26T14:02:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7906.806667 secs 2024-04-26T14:02:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7909.568661 secs 2024-04-26T14:02:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7906.806667 secs 2024-04-26T14:02:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7909.568661 secs 2024-04-26T14:02:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:49 smithi194 ceph-mon[33017]: pgmap v4828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:49 smithi033 ceph-mon[27845]: pgmap v4828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:49 smithi096 ceph-mon[32753]: pgmap v4828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:02:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:51 smithi194 ceph-mon[33017]: pgmap v4829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:02:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:51 smithi033 ceph-mon[27845]: pgmap v4829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:02:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:51 smithi096 ceph-mon[32753]: pgmap v4829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:02:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:53.564+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:53.564+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:02:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:02:53.564+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:02:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:53 smithi194 ceph-mon[33017]: pgmap v4830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7911.806729 secs 2024-04-26T14:02:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7914.568776 secs 2024-04-26T14:02:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:53 smithi033 ceph-mon[27845]: pgmap v4830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7911.806729 secs 2024-04-26T14:02:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7914.568776 secs 2024-04-26T14:02:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:53 smithi096 ceph-mon[32753]: pgmap v4830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7911.806729 secs 2024-04-26T14:02:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7914.568776 secs 2024-04-26T14:02:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:55 smithi194 ceph-mon[33017]: pgmap v4831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:55 smithi033 ceph-mon[27845]: pgmap v4831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:55 smithi096 ceph-mon[32753]: pgmap v4831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:56 smithi194 ceph-mon[33017]: pgmap v4832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:56 smithi033 ceph-mon[27845]: pgmap v4832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:56 smithi096 ceph-mon[32753]: pgmap v4832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:02:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7916.806814 secs 2024-04-26T14:02:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7919.568900 secs 2024-04-26T14:02:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:02:59 smithi194 ceph-mon[33017]: pgmap v4833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7916.806814 secs 2024-04-26T14:02:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7919.568900 secs 2024-04-26T14:02:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:02:59 smithi033 ceph-mon[27845]: pgmap v4833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:02:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7916.806814 secs 2024-04-26T14:02:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7919.568900 secs 2024-04-26T14:02:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:02:59 smithi096 ceph-mon[32753]: pgmap v4833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:01 smithi194 ceph-mon[33017]: pgmap v4834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:01 smithi033 ceph-mon[27845]: pgmap v4834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:01 smithi096 ceph-mon[32753]: pgmap v4834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:03 smithi194 ceph-mon[33017]: pgmap v4835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7924.568975 secs 2024-04-26T14:03:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:03 smithi033 ceph-mon[27845]: pgmap v4835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7924.568975 secs 2024-04-26T14:03:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:03 smithi096 ceph-mon[32753]: pgmap v4835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7924.568975 secs 2024-04-26T14:03:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7921.806899 secs 2024-04-26T14:03:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7921.806899 secs 2024-04-26T14:03:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7921.806899 secs 2024-04-26T14:03:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:05 smithi194 ceph-mon[33017]: pgmap v4836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:05 smithi033 ceph-mon[27845]: pgmap v4836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:05 smithi096 ceph-mon[32753]: pgmap v4836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:07 smithi194 ceph-mon[33017]: pgmap v4837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:07 smithi033 ceph-mon[27845]: pgmap v4837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:07 smithi096 ceph-mon[32753]: pgmap v4837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:08.581+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:08.581+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:08.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:08.581+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:03:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7929.569134 secs 2024-04-26T14:03:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7929.569134 secs 2024-04-26T14:03:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7929.569134 secs 2024-04-26T14:03:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7926.806999 secs 2024-04-26T14:03:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:09 smithi194 ceph-mon[33017]: pgmap v4838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7926.806999 secs 2024-04-26T14:03:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:09 smithi033 ceph-mon[27845]: pgmap v4838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7926.806999 secs 2024-04-26T14:03:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:09 smithi096 ceph-mon[32753]: pgmap v4838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:11 smithi194 ceph-mon[33017]: pgmap v4839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:11 smithi033 ceph-mon[27845]: pgmap v4839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:11 smithi096 ceph-mon[32753]: pgmap v4839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:13 smithi194 ceph-mon[33017]: pgmap v4840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7934.569180 secs 2024-04-26T14:03:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:13 smithi033 ceph-mon[27845]: pgmap v4840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7934.569180 secs 2024-04-26T14:03:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:13 smithi096 ceph-mon[32753]: pgmap v4840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7934.569180 secs 2024-04-26T14:03:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7931.807063 secs 2024-04-26T14:03:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7931.807063 secs 2024-04-26T14:03:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7931.807063 secs 2024-04-26T14:03:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:15 smithi194 ceph-mon[33017]: pgmap v4841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:15 smithi033 ceph-mon[27845]: pgmap v4841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:15 smithi096 ceph-mon[32753]: pgmap v4841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:17 smithi194 ceph-mon[33017]: pgmap v4842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:17 smithi033 ceph-mon[27845]: pgmap v4842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:17 smithi096 ceph-mon[32753]: pgmap v4842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7936.807123 secs 2024-04-26T14:03:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7939.569297 secs 2024-04-26T14:03:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7936.807123 secs 2024-04-26T14:03:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7939.569297 secs 2024-04-26T14:03:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7936.807123 secs 2024-04-26T14:03:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7939.569297 secs 2024-04-26T14:03:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:19 smithi033 ceph-mon[27845]: pgmap v4843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:19 smithi096 ceph-mon[32753]: pgmap v4843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:19 smithi194 ceph-mon[33017]: pgmap v4843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:21 smithi033 ceph-mon[27845]: pgmap v4844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:21 smithi096 ceph-mon[32753]: pgmap v4844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:21 smithi194 ceph-mon[33017]: pgmap v4844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:23.594+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:23.594+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:03:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:23.594+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:23 smithi033 ceph-mon[27845]: pgmap v4845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7941.807182 secs 2024-04-26T14:03:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7944.569361 secs 2024-04-26T14:03:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:23 smithi096 ceph-mon[32753]: pgmap v4845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7941.807182 secs 2024-04-26T14:03:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7944.569361 secs 2024-04-26T14:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:23 smithi194 ceph-mon[33017]: pgmap v4845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7941.807182 secs 2024-04-26T14:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7944.569361 secs 2024-04-26T14:03:25.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:25 smithi194 ceph-mon[33017]: pgmap v4846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:25 smithi033 ceph-mon[27845]: pgmap v4846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:25 smithi096 ceph-mon[32753]: pgmap v4846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:27 smithi033 ceph-mon[27845]: pgmap v4847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:27 smithi096 ceph-mon[32753]: pgmap v4847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:27 smithi194 ceph-mon[33017]: pgmap v4847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7946.807262 secs 2024-04-26T14:03:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7949.569468 secs 2024-04-26T14:03:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7946.807262 secs 2024-04-26T14:03:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7949.569468 secs 2024-04-26T14:03:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7946.807262 secs 2024-04-26T14:03:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7949.569468 secs 2024-04-26T14:03:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:29 smithi033 ceph-mon[27845]: pgmap v4848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:29 smithi096 ceph-mon[32753]: pgmap v4848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:29 smithi194 ceph-mon[33017]: pgmap v4848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:31 smithi033 ceph-mon[27845]: pgmap v4849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:03:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:31 smithi096 ceph-mon[32753]: pgmap v4849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:03:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:31 smithi194 ceph-mon[33017]: pgmap v4849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:03:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:03:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:03:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:03:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:03:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:03:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:03:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:03:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:03:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:03:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:33 smithi033 ceph-mon[27845]: pgmap v4850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:33 smithi033 ceph-mon[27845]: pgmap v4851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7951.807347 secs 2024-04-26T14:03:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7954.569584 secs 2024-04-26T14:03:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:33 smithi096 ceph-mon[32753]: pgmap v4850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:33 smithi096 ceph-mon[32753]: pgmap v4851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7951.807347 secs 2024-04-26T14:03:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7954.569584 secs 2024-04-26T14:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:33 smithi194 ceph-mon[33017]: pgmap v4850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:33 smithi194 ceph-mon[33017]: pgmap v4851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7951.807347 secs 2024-04-26T14:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7954.569584 secs 2024-04-26T14:03:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:35 smithi033 ceph-mon[27845]: pgmap v4852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:03:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:35 smithi096 ceph-mon[32753]: pgmap v4852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:03:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:35 smithi194 ceph-mon[33017]: pgmap v4852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:03:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:37 smithi033 ceph-mon[27845]: pgmap v4853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:37 smithi096 ceph-mon[32753]: pgmap v4853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:37 smithi194 ceph-mon[33017]: pgmap v4853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:38.598+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:38.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:38.598+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:03:38.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:38.598+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7956.807425 secs 2024-04-26T14:03:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7959.569701 secs 2024-04-26T14:03:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7956.807425 secs 2024-04-26T14:03:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7959.569701 secs 2024-04-26T14:03:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7956.807425 secs 2024-04-26T14:03:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7959.569701 secs 2024-04-26T14:03:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:39 smithi194 ceph-mon[33017]: pgmap v4854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:39 smithi033 ceph-mon[27845]: pgmap v4854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:39 smithi096 ceph-mon[32753]: pgmap v4854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:41 smithi194 ceph-mon[33017]: pgmap v4855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:41 smithi033 ceph-mon[27845]: pgmap v4855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:41 smithi096 ceph-mon[32753]: pgmap v4855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:03:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:43 smithi033 ceph-mon[27845]: pgmap v4856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:03:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7961.807506 secs 2024-04-26T14:03:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7964.569841 secs 2024-04-26T14:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:43 smithi194 ceph-mon[33017]: pgmap v4856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7961.807506 secs 2024-04-26T14:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7964.569841 secs 2024-04-26T14:03:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:43 smithi096 ceph-mon[32753]: pgmap v4856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:03:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7961.807506 secs 2024-04-26T14:03:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7964.569841 secs 2024-04-26T14:03:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:45 smithi194 ceph-mon[33017]: pgmap v4857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:45 smithi033 ceph-mon[27845]: pgmap v4857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:45 smithi096 ceph-mon[32753]: pgmap v4857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:47 smithi194 ceph-mon[33017]: pgmap v4858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:47 smithi033 ceph-mon[27845]: pgmap v4858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:47 smithi096 ceph-mon[32753]: pgmap v4858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7966.807564 secs 2024-04-26T14:03:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7969.569976 secs 2024-04-26T14:03:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7966.807564 secs 2024-04-26T14:03:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7969.569976 secs 2024-04-26T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7966.807564 secs 2024-04-26T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7969.569976 secs 2024-04-26T14:03:49.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:49 smithi033 ceph-mon[27845]: pgmap v4859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:49 smithi194 ceph-mon[33017]: pgmap v4859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:49 smithi096 ceph-mon[32753]: pgmap v4859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:51 smithi194 ceph-mon[33017]: pgmap v4860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:51 smithi033 ceph-mon[27845]: pgmap v4860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:51 smithi096 ceph-mon[32753]: pgmap v4860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:53.604+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:53.604+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:03:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:03:53.604+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:03:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:53 smithi194 ceph-mon[33017]: pgmap v4861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7971.807624 secs 2024-04-26T14:03:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7974.570090 secs 2024-04-26T14:03:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:53 smithi033 ceph-mon[27845]: pgmap v4861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7971.807624 secs 2024-04-26T14:03:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7974.570090 secs 2024-04-26T14:03:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:53 smithi096 ceph-mon[32753]: pgmap v4861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7971.807624 secs 2024-04-26T14:03:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7974.570090 secs 2024-04-26T14:03:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:55 smithi194 ceph-mon[33017]: pgmap v4862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:55 smithi033 ceph-mon[27845]: pgmap v4862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:55 smithi096 ceph-mon[32753]: pgmap v4862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:57 smithi194 ceph-mon[33017]: pgmap v4863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:57 smithi033 ceph-mon[27845]: pgmap v4863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:57 smithi096 ceph-mon[32753]: pgmap v4863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:03:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7976.807707 secs 2024-04-26T14:03:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7979.570208 secs 2024-04-26T14:03:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7976.807707 secs 2024-04-26T14:03:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7979.570208 secs 2024-04-26T14:03:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7976.807707 secs 2024-04-26T14:03:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7979.570208 secs 2024-04-26T14:03:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:03:59 smithi194 ceph-mon[33017]: pgmap v4864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:03:59 smithi033 ceph-mon[27845]: pgmap v4864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:03:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:03:59 smithi096 ceph-mon[32753]: pgmap v4864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:01 smithi194 ceph-mon[33017]: pgmap v4865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:01 smithi033 ceph-mon[27845]: pgmap v4865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:01 smithi096 ceph-mon[32753]: pgmap v4865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:03 smithi194 ceph-mon[33017]: pgmap v4866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7981.807797 secs 2024-04-26T14:04:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7984.570323 secs 2024-04-26T14:04:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:03 smithi033 ceph-mon[27845]: pgmap v4866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7981.807797 secs 2024-04-26T14:04:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7984.570323 secs 2024-04-26T14:04:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:03 smithi096 ceph-mon[32753]: pgmap v4866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7981.807797 secs 2024-04-26T14:04:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7984.570323 secs 2024-04-26T14:04:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:05 smithi194 ceph-mon[33017]: pgmap v4867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:05 smithi033 ceph-mon[27845]: pgmap v4867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:05 smithi096 ceph-mon[32753]: pgmap v4867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:07 smithi194 ceph-mon[33017]: pgmap v4868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:07 smithi033 ceph-mon[27845]: pgmap v4868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:07 smithi096 ceph-mon[32753]: pgmap v4868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:08.620+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:08.620+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:08.620+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:04:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7986.807878 secs 2024-04-26T14:04:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7989.570403 secs 2024-04-26T14:04:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7986.807878 secs 2024-04-26T14:04:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7989.570403 secs 2024-04-26T14:04:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7986.807878 secs 2024-04-26T14:04:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7989.570403 secs 2024-04-26T14:04:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:09 smithi194 ceph-mon[33017]: pgmap v4869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:09 smithi033 ceph-mon[27845]: pgmap v4869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:09 smithi096 ceph-mon[32753]: pgmap v4869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:11 smithi194 ceph-mon[33017]: pgmap v4870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:11.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:11 smithi033 ceph-mon[27845]: pgmap v4870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:11 smithi096 ceph-mon[32753]: pgmap v4870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:13 smithi194 ceph-mon[33017]: pgmap v4871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7991.807989 secs 2024-04-26T14:04:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7994.570538 secs 2024-04-26T14:04:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:13 smithi033 ceph-mon[27845]: pgmap v4871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7991.807989 secs 2024-04-26T14:04:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7994.570538 secs 2024-04-26T14:04:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:13 smithi096 ceph-mon[32753]: pgmap v4871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7991.807989 secs 2024-04-26T14:04:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7994.570538 secs 2024-04-26T14:04:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:15 smithi194 ceph-mon[33017]: pgmap v4872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:15 smithi033 ceph-mon[27845]: pgmap v4872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:15 smithi096 ceph-mon[32753]: pgmap v4872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:17 smithi194 ceph-mon[33017]: pgmap v4873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:17 smithi033 ceph-mon[27845]: pgmap v4873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:17 smithi096 ceph-mon[32753]: pgmap v4873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 7996.808005 secs 2024-04-26T14:04:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 7999.570661 secs 2024-04-26T14:04:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 7996.808005 secs 2024-04-26T14:04:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 7999.570661 secs 2024-04-26T14:04:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 7996.808005 secs 2024-04-26T14:04:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 7999.570661 secs 2024-04-26T14:04:19.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:19 smithi033 ceph-mon[27845]: pgmap v4874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:19 smithi194 ceph-mon[33017]: pgmap v4874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:19 smithi096 ceph-mon[32753]: pgmap v4874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:21 smithi194 ceph-mon[33017]: pgmap v4875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:21 smithi033 ceph-mon[27845]: pgmap v4875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:21 smithi096 ceph-mon[32753]: pgmap v4875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:23.636 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:23.636+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:23.636 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:23.636+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:23.636 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:23.636+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:04:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:23 smithi194 ceph-mon[33017]: pgmap v4876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8001.808067 secs 2024-04-26T14:04:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8004.570795 secs 2024-04-26T14:04:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:23 smithi033 ceph-mon[27845]: pgmap v4876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8001.808067 secs 2024-04-26T14:04:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8004.570795 secs 2024-04-26T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:23 smithi096 ceph-mon[32753]: pgmap v4876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8001.808067 secs 2024-04-26T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8004.570795 secs 2024-04-26T14:04:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:25 smithi194 ceph-mon[33017]: pgmap v4877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:25 smithi033 ceph-mon[27845]: pgmap v4877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:25 smithi096 ceph-mon[32753]: pgmap v4877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:27 smithi194 ceph-mon[33017]: pgmap v4878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:27 smithi033 ceph-mon[27845]: pgmap v4878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:27 smithi096 ceph-mon[32753]: pgmap v4878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8006.808127 secs 2024-04-26T14:04:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8009.570892 secs 2024-04-26T14:04:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8006.808127 secs 2024-04-26T14:04:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8009.570892 secs 2024-04-26T14:04:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8006.808127 secs 2024-04-26T14:04:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8009.570892 secs 2024-04-26T14:04:29.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:29 smithi033 ceph-mon[27845]: pgmap v4879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:29 smithi096 ceph-mon[32753]: pgmap v4879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:29 smithi194 ceph-mon[33017]: pgmap v4879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:31 smithi033 ceph-mon[27845]: pgmap v4880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:31 smithi096 ceph-mon[32753]: pgmap v4880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:31 smithi194 ceph-mon[33017]: pgmap v4880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:04:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:04:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:04:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:33 smithi033 ceph-mon[27845]: pgmap v4881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8011.808191 secs 2024-04-26T14:04:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8014.570953 secs 2024-04-26T14:04:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:33 smithi096 ceph-mon[32753]: pgmap v4881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8011.808191 secs 2024-04-26T14:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8014.570953 secs 2024-04-26T14:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:33 smithi194 ceph-mon[33017]: pgmap v4881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8011.808191 secs 2024-04-26T14:04:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8014.570953 secs 2024-04-26T14:04:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:04:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:04:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:04:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:04:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:04:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:04:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:04:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:35 smithi033 ceph-mon[27845]: pgmap v4882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:35 smithi033 ceph-mon[27845]: pgmap v4883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T14:04:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:35 smithi033 ceph-mon[27845]: pgmap v4884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:35 smithi096 ceph-mon[32753]: pgmap v4882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:35 smithi096 ceph-mon[32753]: pgmap v4883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T14:04:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:35 smithi096 ceph-mon[32753]: pgmap v4884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:35 smithi194 ceph-mon[33017]: pgmap v4882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:35 smithi194 ceph-mon[33017]: pgmap v4883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T14:04:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:35 smithi194 ceph-mon[33017]: pgmap v4884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:37 smithi033 ceph-mon[27845]: pgmap v4885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:37 smithi096 ceph-mon[32753]: pgmap v4885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:37 smithi194 ceph-mon[33017]: pgmap v4885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:38.640+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:38.640+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:38.640+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:04:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8016.808250 secs 2024-04-26T14:04:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8019.571071 secs 2024-04-26T14:04:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8016.808250 secs 2024-04-26T14:04:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8019.571071 secs 2024-04-26T14:04:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8016.808250 secs 2024-04-26T14:04:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8019.571071 secs 2024-04-26T14:04:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:39 smithi033 ceph-mon[27845]: pgmap v4886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:04:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:39 smithi096 ceph-mon[32753]: pgmap v4886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:04:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:39 smithi194 ceph-mon[33017]: pgmap v4886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:04:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:41 smithi033 ceph-mon[27845]: pgmap v4887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:41 smithi096 ceph-mon[32753]: pgmap v4887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:41 smithi194 ceph-mon[33017]: pgmap v4887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:43 smithi033 ceph-mon[27845]: pgmap v4888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8021.808312 secs 2024-04-26T14:04:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8024.571209 secs 2024-04-26T14:04:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:43 smithi096 ceph-mon[32753]: pgmap v4888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8021.808312 secs 2024-04-26T14:04:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8024.571209 secs 2024-04-26T14:04:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:43 smithi194 ceph-mon[33017]: pgmap v4888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:04:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8021.808312 secs 2024-04-26T14:04:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8024.571209 secs 2024-04-26T14:04:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:45 smithi033 ceph-mon[27845]: pgmap v4889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:04:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:45 smithi096 ceph-mon[32753]: pgmap v4889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:04:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:45 smithi194 ceph-mon[33017]: pgmap v4889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:04:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:47 smithi096 ceph-mon[32753]: pgmap v4890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:47 smithi194 ceph-mon[33017]: pgmap v4890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:47 smithi033 ceph-mon[27845]: pgmap v4890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:49.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8026.808369 secs 2024-04-26T14:04:49.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8029.571301 secs 2024-04-26T14:04:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8026.808369 secs 2024-04-26T14:04:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8029.571301 secs 2024-04-26T14:04:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8026.808369 secs 2024-04-26T14:04:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8029.571301 secs 2024-04-26T14:04:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:49 smithi194 ceph-mon[33017]: pgmap v4891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:49 smithi033 ceph-mon[27845]: pgmap v4891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:49 smithi096 ceph-mon[32753]: pgmap v4891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:51 smithi194 ceph-mon[33017]: pgmap v4892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:51 smithi033 ceph-mon[27845]: pgmap v4892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:51 smithi096 ceph-mon[32753]: pgmap v4892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:53.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:53.647+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:53.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:53.647+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:04:53.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:04:53.647+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:04:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:53 smithi194 ceph-mon[33017]: pgmap v4893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8031.808453 secs 2024-04-26T14:04:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8034.571384 secs 2024-04-26T14:04:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:53 smithi033 ceph-mon[27845]: pgmap v4893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8031.808453 secs 2024-04-26T14:04:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8034.571384 secs 2024-04-26T14:04:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:53 smithi096 ceph-mon[32753]: pgmap v4893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8031.808453 secs 2024-04-26T14:04:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8034.571384 secs 2024-04-26T14:04:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:55 smithi194 ceph-mon[33017]: pgmap v4894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:55 smithi033 ceph-mon[27845]: pgmap v4894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:55 smithi096 ceph-mon[32753]: pgmap v4894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:04:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:57 smithi194 ceph-mon[33017]: pgmap v4895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:57 smithi033 ceph-mon[27845]: pgmap v4895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:57 smithi096 ceph-mon[32753]: pgmap v4895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:04:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8036.808533 secs 2024-04-26T14:04:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8039.571500 secs 2024-04-26T14:04:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8036.808533 secs 2024-04-26T14:04:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8039.571500 secs 2024-04-26T14:04:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8036.808533 secs 2024-04-26T14:04:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8039.571500 secs 2024-04-26T14:05:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:04:59 smithi194 ceph-mon[33017]: pgmap v4896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:04:59 smithi033 ceph-mon[27845]: pgmap v4896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:04:59 smithi096 ceph-mon[32753]: pgmap v4896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:01.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:01 smithi096 ceph-mon[32753]: pgmap v4897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:01 smithi194 ceph-mon[33017]: pgmap v4897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:01 smithi033 ceph-mon[27845]: pgmap v4897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:03 smithi194 ceph-mon[33017]: pgmap v4898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8041.808594 secs 2024-04-26T14:05:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8044.571628 secs 2024-04-26T14:05:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:03 smithi033 ceph-mon[27845]: pgmap v4898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8041.808594 secs 2024-04-26T14:05:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8044.571628 secs 2024-04-26T14:05:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:03 smithi096 ceph-mon[32753]: pgmap v4898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8041.808594 secs 2024-04-26T14:05:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8044.571628 secs 2024-04-26T14:05:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:05 smithi194 ceph-mon[33017]: pgmap v4899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:05 smithi033 ceph-mon[27845]: pgmap v4899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:05 smithi096 ceph-mon[32753]: pgmap v4899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:07 smithi194 ceph-mon[33017]: pgmap v4900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:07 smithi033 ceph-mon[27845]: pgmap v4900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:07 smithi096 ceph-mon[32753]: pgmap v4900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:08.649+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:08.649+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:08.649+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:05:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8046.808673 secs 2024-04-26T14:05:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8049.571745 secs 2024-04-26T14:05:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8046.808673 secs 2024-04-26T14:05:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8049.571745 secs 2024-04-26T14:05:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8046.808673 secs 2024-04-26T14:05:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8049.571745 secs 2024-04-26T14:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:09 smithi194 ceph-mon[33017]: pgmap v4901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:09 smithi033 ceph-mon[27845]: pgmap v4901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:09 smithi096 ceph-mon[32753]: pgmap v4901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:10 smithi194 ceph-mon[33017]: pgmap v4902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:10 smithi033 ceph-mon[27845]: pgmap v4902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:10 smithi096 ceph-mon[32753]: pgmap v4902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8051.808756 secs 2024-04-26T14:05:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:13 smithi033 ceph-mon[27845]: pgmap v4903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8054.571866 secs 2024-04-26T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8051.808756 secs 2024-04-26T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:13 smithi096 ceph-mon[32753]: pgmap v4903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8054.571866 secs 2024-04-26T14:05:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8051.808756 secs 2024-04-26T14:05:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:13 smithi194 ceph-mon[33017]: pgmap v4903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:05:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8054.571866 secs 2024-04-26T14:05:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:15 smithi033 ceph-mon[27845]: pgmap v4904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:15 smithi096 ceph-mon[32753]: pgmap v4904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:15 smithi194 ceph-mon[33017]: pgmap v4904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:05:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:17 smithi033 ceph-mon[27845]: pgmap v4905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:17 smithi096 ceph-mon[32753]: pgmap v4905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:17 smithi194 ceph-mon[33017]: pgmap v4905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8056.808838 secs 2024-04-26T14:05:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8059.571985 secs 2024-04-26T14:05:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8056.808838 secs 2024-04-26T14:05:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8059.571985 secs 2024-04-26T14:05:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8056.808838 secs 2024-04-26T14:05:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8059.571985 secs 2024-04-26T14:05:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:19 smithi033 ceph-mon[27845]: pgmap v4906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:19 smithi096 ceph-mon[32753]: pgmap v4906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:19 smithi194 ceph-mon[33017]: pgmap v4906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:21 smithi033 ceph-mon[27845]: pgmap v4907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:21 smithi096 ceph-mon[32753]: pgmap v4907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:21 smithi194 ceph-mon[33017]: pgmap v4907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:23.161+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:05:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:23.161+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:05:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:23.161+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:05:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:23.161+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:05:23.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:23.650+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:23.650+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:05:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:23.650+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8061.808920 secs 2024-04-26T14:05:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:23 smithi033 ceph-mon[27845]: pgmap v4908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8064.572099 secs 2024-04-26T14:05:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8061.808920 secs 2024-04-26T14:05:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:23 smithi096 ceph-mon[32753]: pgmap v4908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8064.572099 secs 2024-04-26T14:05:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8061.808920 secs 2024-04-26T14:05:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:23 smithi194 ceph-mon[33017]: pgmap v4908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8064.572099 secs 2024-04-26T14:05:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:25 smithi033 ceph-mon[27845]: pgmap v4909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:25 smithi096 ceph-mon[32753]: pgmap v4909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:25 smithi194 ceph-mon[33017]: pgmap v4909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:27 smithi033 ceph-mon[27845]: pgmap v4910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:27 smithi096 ceph-mon[32753]: pgmap v4910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:27 smithi194 ceph-mon[33017]: pgmap v4910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8066.809020 secs 2024-04-26T14:05:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8069.572164 secs 2024-04-26T14:05:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8066.809020 secs 2024-04-26T14:05:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8069.572164 secs 2024-04-26T14:05:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8066.809020 secs 2024-04-26T14:05:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8069.572164 secs 2024-04-26T14:05:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:29 smithi194 ceph-mon[33017]: pgmap v4911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:29 smithi033 ceph-mon[27845]: pgmap v4911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:29 smithi096 ceph-mon[32753]: pgmap v4911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:31.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:31 smithi096 ceph-mon[32753]: pgmap v4912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:31 smithi194 ceph-mon[33017]: pgmap v4912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:31 smithi033 ceph-mon[27845]: pgmap v4912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8071.809102 secs 2024-04-26T14:05:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:33 smithi194 ceph-mon[33017]: pgmap v4913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8074.572272 secs 2024-04-26T14:05:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8071.809102 secs 2024-04-26T14:05:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:33 smithi033 ceph-mon[27845]: pgmap v4913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8074.572272 secs 2024-04-26T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8071.809102 secs 2024-04-26T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:33 smithi096 ceph-mon[32753]: pgmap v4913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8074.572272 secs 2024-04-26T14:05:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:35 smithi194 ceph-mon[33017]: pgmap v4914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:05:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:35 smithi033 ceph-mon[27845]: pgmap v4914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:05:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:35 smithi096 ceph-mon[32753]: pgmap v4914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:05:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:05:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:05:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:05:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:05:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:05:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:05:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:37 smithi194 ceph-mon[33017]: pgmap v4915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:37 smithi194 ceph-mon[33017]: pgmap v4916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:37 smithi033 ceph-mon[27845]: pgmap v4915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:37 smithi033 ceph-mon[27845]: pgmap v4916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:37 smithi096 ceph-mon[32753]: pgmap v4915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:37 smithi096 ceph-mon[32753]: pgmap v4916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:38.660+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:38.660+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:38.660+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:05:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8076.809191 secs 2024-04-26T14:05:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8079.572385 secs 2024-04-26T14:05:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8076.809191 secs 2024-04-26T14:05:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8079.572385 secs 2024-04-26T14:05:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8076.809191 secs 2024-04-26T14:05:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8079.572385 secs 2024-04-26T14:05:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:39 smithi194 ceph-mon[33017]: pgmap v4917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:39 smithi033 ceph-mon[27845]: pgmap v4917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:39 smithi096 ceph-mon[32753]: pgmap v4917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:41 smithi194 ceph-mon[33017]: pgmap v4918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:41 smithi033 ceph-mon[27845]: pgmap v4918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:41 smithi096 ceph-mon[32753]: pgmap v4918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:43 smithi194 ceph-mon[33017]: pgmap v4919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8081.809249 secs 2024-04-26T14:05:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8084.572521 secs 2024-04-26T14:05:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:43 smithi033 ceph-mon[27845]: pgmap v4919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8081.809249 secs 2024-04-26T14:05:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8084.572521 secs 2024-04-26T14:05:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:43 smithi096 ceph-mon[32753]: pgmap v4919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:05:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8081.809249 secs 2024-04-26T14:05:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8084.572521 secs 2024-04-26T14:05:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:45 smithi194 ceph-mon[33017]: pgmap v4920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:05:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:45 smithi033 ceph-mon[27845]: pgmap v4920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:05:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:45 smithi096 ceph-mon[32753]: pgmap v4920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:05:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:47 smithi194 ceph-mon[33017]: pgmap v4921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:05:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:47 smithi033 ceph-mon[27845]: pgmap v4921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:05:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:47 smithi096 ceph-mon[32753]: pgmap v4921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:05:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8086.809311 secs 2024-04-26T14:05:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8089.572640 secs 2024-04-26T14:05:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8086.809311 secs 2024-04-26T14:05:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8089.572640 secs 2024-04-26T14:05:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8086.809311 secs 2024-04-26T14:05:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8089.572640 secs 2024-04-26T14:05:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:49 smithi194 ceph-mon[33017]: pgmap v4922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:49 smithi033 ceph-mon[27845]: pgmap v4922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:49 smithi096 ceph-mon[32753]: pgmap v4922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:51 smithi194 ceph-mon[33017]: pgmap v4923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:51 smithi033 ceph-mon[27845]: pgmap v4923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:51 smithi096 ceph-mon[32753]: pgmap v4923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:52 smithi194 ceph-mon[33017]: pgmap v4924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:53.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:52 smithi033 ceph-mon[27845]: pgmap v4924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:52 smithi096 ceph-mon[32753]: pgmap v4924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:53.673+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:53.673+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:05:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:05:53.673+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:05:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8091.809372 secs 2024-04-26T14:05:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8094.572765 secs 2024-04-26T14:05:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8091.809372 secs 2024-04-26T14:05:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8094.572765 secs 2024-04-26T14:05:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8091.809372 secs 2024-04-26T14:05:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8094.572765 secs 2024-04-26T14:05:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:54 smithi194 ceph-mon[33017]: pgmap v4925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:54 smithi033 ceph-mon[27845]: pgmap v4925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:54 smithi096 ceph-mon[32753]: pgmap v4925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:57 smithi194 ceph-mon[33017]: pgmap v4926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:57 smithi033 ceph-mon[27845]: pgmap v4926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:57 smithi096 ceph-mon[32753]: pgmap v4926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8096.809430 secs 2024-04-26T14:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8099.572862 secs 2024-04-26T14:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:05:59 smithi194 ceph-mon[33017]: pgmap v4927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8096.809430 secs 2024-04-26T14:05:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8099.572862 secs 2024-04-26T14:05:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:05:59 smithi033 ceph-mon[27845]: pgmap v4927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:05:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8096.809430 secs 2024-04-26T14:05:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8099.572862 secs 2024-04-26T14:05:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:05:59 smithi096 ceph-mon[32753]: pgmap v4927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:01 smithi194 ceph-mon[33017]: pgmap v4928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:01 smithi033 ceph-mon[27845]: pgmap v4928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:01 smithi096 ceph-mon[32753]: pgmap v4928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:03 smithi194 ceph-mon[33017]: pgmap v4929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8104.572980 secs 2024-04-26T14:06:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:03 smithi033 ceph-mon[27845]: pgmap v4929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8104.572980 secs 2024-04-26T14:06:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:03 smithi096 ceph-mon[32753]: pgmap v4929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8104.572980 secs 2024-04-26T14:06:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8101.809488 secs 2024-04-26T14:06:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8101.809488 secs 2024-04-26T14:06:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8101.809488 secs 2024-04-26T14:06:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:05 smithi194 ceph-mon[33017]: pgmap v4930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:05 smithi033 ceph-mon[27845]: pgmap v4930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:05 smithi096 ceph-mon[32753]: pgmap v4930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:07 smithi194 ceph-mon[33017]: pgmap v4931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:07 smithi033 ceph-mon[27845]: pgmap v4931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:07 smithi096 ceph-mon[32753]: pgmap v4931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8109.573100 secs 2024-04-26T14:06:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8109.573100 secs 2024-04-26T14:06:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8109.573100 secs 2024-04-26T14:06:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:08.689+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:08.689+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:08.689+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:06:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8106.809549 secs 2024-04-26T14:06:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:09 smithi194 ceph-mon[33017]: pgmap v4932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8106.809549 secs 2024-04-26T14:06:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:09 smithi033 ceph-mon[27845]: pgmap v4932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8106.809549 secs 2024-04-26T14:06:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:09 smithi096 ceph-mon[32753]: pgmap v4932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:11 smithi194 ceph-mon[33017]: pgmap v4933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:11 smithi033 ceph-mon[27845]: pgmap v4933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:11 smithi096 ceph-mon[32753]: pgmap v4933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:13 smithi194 ceph-mon[33017]: pgmap v4934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8114.573252 secs 2024-04-26T14:06:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:13 smithi033 ceph-mon[27845]: pgmap v4934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8114.573252 secs 2024-04-26T14:06:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:13 smithi096 ceph-mon[32753]: pgmap v4934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8114.573252 secs 2024-04-26T14:06:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8111.809610 secs 2024-04-26T14:06:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8111.809610 secs 2024-04-26T14:06:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8111.809610 secs 2024-04-26T14:06:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:15 smithi033 ceph-mon[27845]: pgmap v4935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:15 smithi096 ceph-mon[32753]: pgmap v4935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:15 smithi194 ceph-mon[33017]: pgmap v4935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:17 smithi033 ceph-mon[27845]: pgmap v4936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:17 smithi096 ceph-mon[32753]: pgmap v4936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:17 smithi194 ceph-mon[33017]: pgmap v4936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8116.809670 secs 2024-04-26T14:06:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8119.573327 secs 2024-04-26T14:06:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8116.809670 secs 2024-04-26T14:06:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8119.573327 secs 2024-04-26T14:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8116.809670 secs 2024-04-26T14:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8119.573327 secs 2024-04-26T14:06:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:19 smithi033 ceph-mon[27845]: pgmap v4937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:19 smithi096 ceph-mon[32753]: pgmap v4937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:19 smithi194 ceph-mon[33017]: pgmap v4937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:21 smithi033 ceph-mon[27845]: pgmap v4938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:21 smithi096 ceph-mon[32753]: pgmap v4938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:21 smithi194 ceph-mon[33017]: pgmap v4938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:23 smithi033 ceph-mon[27845]: pgmap v4939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8121.809729 secs 2024-04-26T14:06:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8124.573421 secs 2024-04-26T14:06:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:23 smithi096 ceph-mon[32753]: pgmap v4939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8121.809729 secs 2024-04-26T14:06:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8124.573421 secs 2024-04-26T14:06:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:23.704+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:23.704+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:06:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:23.704+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:23 smithi194 ceph-mon[33017]: pgmap v4939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8121.809729 secs 2024-04-26T14:06:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8124.573421 secs 2024-04-26T14:06:25.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:25 smithi194 ceph-mon[33017]: pgmap v4940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:25 smithi033 ceph-mon[27845]: pgmap v4940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:25 smithi096 ceph-mon[32753]: pgmap v4940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:27 smithi033 ceph-mon[27845]: pgmap v4941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:27 smithi096 ceph-mon[32753]: pgmap v4941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:27 smithi194 ceph-mon[33017]: pgmap v4941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8126.809783 secs 2024-04-26T14:06:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8129.573539 secs 2024-04-26T14:06:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8126.809783 secs 2024-04-26T14:06:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8129.573539 secs 2024-04-26T14:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8126.809783 secs 2024-04-26T14:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8129.573539 secs 2024-04-26T14:06:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:29 smithi033 ceph-mon[27845]: pgmap v4942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:29 smithi096 ceph-mon[32753]: pgmap v4942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:29 smithi194 ceph-mon[33017]: pgmap v4942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:31 smithi033 ceph-mon[27845]: pgmap v4943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:31 smithi096 ceph-mon[32753]: pgmap v4943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:31 smithi194 ceph-mon[33017]: pgmap v4943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:33 smithi096 ceph-mon[32753]: pgmap v4944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8131.809843 secs 2024-04-26T14:06:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8134.573650 secs 2024-04-26T14:06:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:33 smithi194 ceph-mon[33017]: pgmap v4944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8131.809843 secs 2024-04-26T14:06:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8134.573650 secs 2024-04-26T14:06:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:33 smithi033 ceph-mon[27845]: pgmap v4944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8131.809843 secs 2024-04-26T14:06:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8134.573650 secs 2024-04-26T14:06:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:35 smithi033 ceph-mon[27845]: pgmap v4945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:35 smithi194 ceph-mon[33017]: pgmap v4945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:35 smithi096 ceph-mon[32753]: pgmap v4945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:37 smithi194 ceph-mon[33017]: pgmap v4946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:37 smithi033 ceph-mon[27845]: pgmap v4946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:37.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:06:37.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:37 smithi096 ceph-mon[32753]: pgmap v4946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:06:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:38.719+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:38.719+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:38.719+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:06:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8136.809907 secs 2024-04-26T14:06:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8139.573718 secs 2024-04-26T14:06:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8136.809907 secs 2024-04-26T14:06:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8139.573718 secs 2024-04-26T14:06:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8136.809907 secs 2024-04-26T14:06:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8139.573718 secs 2024-04-26T14:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:39 smithi194 ceph-mon[33017]: pgmap v4947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:39 smithi033 ceph-mon[27845]: pgmap v4947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:39 smithi096 ceph-mon[32753]: pgmap v4947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:41 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:06:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:41 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:06:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:41 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:06:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:41 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:06:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:41 smithi194 ceph-mon[33017]: pgmap v4948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:41 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:06:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:41 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:06:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:41 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:06:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:41 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:06:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:41 smithi033 ceph-mon[27845]: pgmap v4948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:41 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:06:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:41 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:06:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:41 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:06:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:41 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:06:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:41 smithi096 ceph-mon[32753]: pgmap v4948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: pgmap v4949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8141.809995 secs 2024-04-26T14:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8144.573865 secs 2024-04-26T14:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: pgmap v4949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8141.809995 secs 2024-04-26T14:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8144.573865 secs 2024-04-26T14:06:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: pgmap v4949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8141.809995 secs 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:06:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8144.573865 secs 2024-04-26T14:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:44 smithi194 ceph-mon[33017]: pgmap v4950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:44 smithi194 ceph-mon[33017]: pgmap v4951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:44 smithi033 ceph-mon[27845]: pgmap v4950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:44 smithi033 ceph-mon[27845]: pgmap v4951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:44 smithi096 ceph-mon[32753]: pgmap v4950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:44 smithi096 ceph-mon[32753]: pgmap v4951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:46 smithi194 ceph-mon[33017]: pgmap v4952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:46 smithi033 ceph-mon[27845]: pgmap v4952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:46 smithi096 ceph-mon[32753]: pgmap v4952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:48 smithi194 ceph-mon[33017]: pgmap v4953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8146.810077 secs 2024-04-26T14:06:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8149.573981 secs 2024-04-26T14:06:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:48 smithi033 ceph-mon[27845]: pgmap v4953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8146.810077 secs 2024-04-26T14:06:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8149.573981 secs 2024-04-26T14:06:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:48 smithi096 ceph-mon[32753]: pgmap v4953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8146.810077 secs 2024-04-26T14:06:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8149.573981 secs 2024-04-26T14:06:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:50 smithi194 ceph-mon[33017]: pgmap v4954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:06:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:50 smithi033 ceph-mon[27845]: pgmap v4954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:06:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:50 smithi096 ceph-mon[32753]: pgmap v4954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:06:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:52 smithi194 ceph-mon[33017]: pgmap v4955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:52 smithi033 ceph-mon[27845]: pgmap v4955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:52 smithi096 ceph-mon[32753]: pgmap v4955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:06:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8151.810159 secs 2024-04-26T14:06:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8154.574100 secs 2024-04-26T14:06:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:53.732+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:53.732+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:06:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:06:53.732+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:06:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8151.810159 secs 2024-04-26T14:06:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8154.574100 secs 2024-04-26T14:06:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8151.810159 secs 2024-04-26T14:06:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8154.574100 secs 2024-04-26T14:06:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:54 smithi194 ceph-mon[33017]: pgmap v4956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:06:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:54 smithi033 ceph-mon[27845]: pgmap v4956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:06:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:54 smithi096 ceph-mon[32753]: pgmap v4956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:06:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:56 smithi194 ceph-mon[33017]: pgmap v4957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:56 smithi033 ceph-mon[27845]: pgmap v4957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:56 smithi096 ceph-mon[32753]: pgmap v4957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:06:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:58 smithi194 ceph-mon[33017]: pgmap v4958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8156.810265 secs 2024-04-26T14:06:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:06:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8159.574192 secs 2024-04-26T14:06:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:58 smithi033 ceph-mon[27845]: pgmap v4958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8156.810265 secs 2024-04-26T14:06:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:06:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8159.574192 secs 2024-04-26T14:06:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:58 smithi096 ceph-mon[32753]: pgmap v4958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:06:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8156.810265 secs 2024-04-26T14:06:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:06:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8159.574192 secs 2024-04-26T14:07:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:00 smithi194 ceph-mon[33017]: pgmap v4959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:00 smithi033 ceph-mon[27845]: pgmap v4959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:00 smithi096 ceph-mon[32753]: pgmap v4959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:02 smithi194 ceph-mon[33017]: pgmap v4960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:02.856 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:02 smithi033 ceph-mon[27845]: pgmap v4960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:02 smithi096 ceph-mon[32753]: pgmap v4960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8161.810347 secs 2024-04-26T14:07:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8164.574312 secs 2024-04-26T14:07:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8161.810347 secs 2024-04-26T14:07:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8164.574312 secs 2024-04-26T14:07:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8161.810347 secs 2024-04-26T14:07:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8164.574312 secs 2024-04-26T14:07:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:04 smithi194 ceph-mon[33017]: pgmap v4961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:04 smithi033 ceph-mon[27845]: pgmap v4961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:04 smithi096 ceph-mon[32753]: pgmap v4961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:06 smithi194 ceph-mon[33017]: pgmap v4962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:06 smithi033 ceph-mon[27845]: pgmap v4962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:06 smithi096 ceph-mon[32753]: pgmap v4962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:08.742+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:08.742+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:08.742+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:07:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:08 smithi194 ceph-mon[33017]: pgmap v4963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8166.810409 secs 2024-04-26T14:07:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8169.574458 secs 2024-04-26T14:07:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:08 smithi033 ceph-mon[27845]: pgmap v4963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8166.810409 secs 2024-04-26T14:07:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8169.574458 secs 2024-04-26T14:07:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:08 smithi096 ceph-mon[32753]: pgmap v4963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8166.810409 secs 2024-04-26T14:07:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8169.574458 secs 2024-04-26T14:07:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:10 smithi194 ceph-mon[33017]: pgmap v4964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:10 smithi033 ceph-mon[27845]: pgmap v4964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:10 smithi096 ceph-mon[32753]: pgmap v4964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:12 smithi194 ceph-mon[33017]: pgmap v4965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:12 smithi033 ceph-mon[27845]: pgmap v4965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:12 smithi096 ceph-mon[32753]: pgmap v4965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8171.810471 secs 2024-04-26T14:07:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8174.574540 secs 2024-04-26T14:07:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8171.810471 secs 2024-04-26T14:07:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8174.574540 secs 2024-04-26T14:07:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8171.810471 secs 2024-04-26T14:07:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8174.574540 secs 2024-04-26T14:07:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:14 smithi194 ceph-mon[33017]: pgmap v4966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:14 smithi033 ceph-mon[27845]: pgmap v4966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:14 smithi096 ceph-mon[32753]: pgmap v4966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:16 smithi194 ceph-mon[33017]: pgmap v4967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:16 smithi033 ceph-mon[27845]: pgmap v4967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:16 smithi096 ceph-mon[32753]: pgmap v4967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:18 smithi194 ceph-mon[33017]: pgmap v4968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8176.810555 secs 2024-04-26T14:07:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8179.574670 secs 2024-04-26T14:07:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:18 smithi033 ceph-mon[27845]: pgmap v4968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8176.810555 secs 2024-04-26T14:07:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8179.574670 secs 2024-04-26T14:07:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:18 smithi096 ceph-mon[32753]: pgmap v4968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8176.810555 secs 2024-04-26T14:07:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8179.574670 secs 2024-04-26T14:07:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:20 smithi194 ceph-mon[33017]: pgmap v4969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:20 smithi033 ceph-mon[27845]: pgmap v4969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:20 smithi096 ceph-mon[32753]: pgmap v4969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:22 smithi194 ceph-mon[33017]: pgmap v4970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:22 smithi033 ceph-mon[27845]: pgmap v4970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:22 smithi096 ceph-mon[32753]: pgmap v4970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:23.757+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:23.757+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:23.757+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:07:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8181.810640 secs 2024-04-26T14:07:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8184.574795 secs 2024-04-26T14:07:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8181.810640 secs 2024-04-26T14:07:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8184.574795 secs 2024-04-26T14:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8181.810640 secs 2024-04-26T14:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8184.574795 secs 2024-04-26T14:07:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:24 smithi033 ceph-mon[27845]: pgmap v4971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:24 smithi096 ceph-mon[32753]: pgmap v4971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:24 smithi194 ceph-mon[33017]: pgmap v4971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:26.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:26 smithi194 ceph-mon[33017]: pgmap v4972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:26.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:26 smithi033 ceph-mon[27845]: pgmap v4972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:26 smithi096 ceph-mon[32753]: pgmap v4972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:28 smithi033 ceph-mon[27845]: pgmap v4973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8186.810722 secs 2024-04-26T14:07:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8189.574916 secs 2024-04-26T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:28 smithi096 ceph-mon[32753]: pgmap v4973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8186.810722 secs 2024-04-26T14:07:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8189.574916 secs 2024-04-26T14:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:28 smithi194 ceph-mon[33017]: pgmap v4973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8186.810722 secs 2024-04-26T14:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8189.574916 secs 2024-04-26T14:07:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:30 smithi033 ceph-mon[27845]: pgmap v4974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:30 smithi096 ceph-mon[32753]: pgmap v4974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:30 smithi194 ceph-mon[33017]: pgmap v4974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:32 smithi033 ceph-mon[27845]: pgmap v4975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:32 smithi096 ceph-mon[32753]: pgmap v4975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:32 smithi194 ceph-mon[33017]: pgmap v4975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8191.810781 secs 2024-04-26T14:07:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8194.575046 secs 2024-04-26T14:07:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8191.810781 secs 2024-04-26T14:07:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8194.575046 secs 2024-04-26T14:07:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8191.810781 secs 2024-04-26T14:07:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8194.575046 secs 2024-04-26T14:07:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:34 smithi033 ceph-mon[27845]: pgmap v4976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:34 smithi096 ceph-mon[32753]: pgmap v4976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:34 smithi194 ceph-mon[33017]: pgmap v4976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:36 smithi033 ceph-mon[27845]: pgmap v4977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:36.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:36 smithi096 ceph-mon[32753]: pgmap v4977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:36 smithi194 ceph-mon[33017]: pgmap v4977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:38.772+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:38.772+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:38.772+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:07:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:38 smithi033 ceph-mon[27845]: pgmap v4978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8196.810865 secs 2024-04-26T14:07:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8199.575101 secs 2024-04-26T14:07:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:38 smithi096 ceph-mon[32753]: pgmap v4978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8196.810865 secs 2024-04-26T14:07:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8199.575101 secs 2024-04-26T14:07:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:38 smithi194 ceph-mon[33017]: pgmap v4978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8196.810865 secs 2024-04-26T14:07:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8199.575101 secs 2024-04-26T14:07:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:40 smithi033 ceph-mon[27845]: pgmap v4979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:40 smithi096 ceph-mon[32753]: pgmap v4979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:40 smithi194 ceph-mon[33017]: pgmap v4979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:42 smithi033 ceph-mon[27845]: pgmap v4980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:42 smithi096 ceph-mon[32753]: pgmap v4980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:42 smithi194 ceph-mon[33017]: pgmap v4980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:07:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8201.810949 secs 2024-04-26T14:07:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8204.575194 secs 2024-04-26T14:07:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:07:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8201.810949 secs 2024-04-26T14:07:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8204.575194 secs 2024-04-26T14:07:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8201.810949 secs 2024-04-26T14:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8204.575194 secs 2024-04-26T14:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:07:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:44 smithi194 ceph-mon[33017]: pgmap v4981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:07:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:07:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:44 smithi033 ceph-mon[27845]: pgmap v4981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:44 smithi096 ceph-mon[32753]: pgmap v4981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:07:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:07:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:45 smithi194 ceph-mon[33017]: pgmap v4982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:07:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:45 smithi194 ceph-mon[33017]: pgmap v4983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:45 smithi033 ceph-mon[27845]: pgmap v4982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:07:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:45 smithi033 ceph-mon[27845]: pgmap v4983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:45 smithi096 ceph-mon[32753]: pgmap v4982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:07:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:45 smithi096 ceph-mon[32753]: pgmap v4983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:47 smithi194 ceph-mon[33017]: pgmap v4984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:47 smithi033 ceph-mon[27845]: pgmap v4984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:47 smithi096 ceph-mon[32753]: pgmap v4984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8206.811041 secs 2024-04-26T14:07:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8209.575276 secs 2024-04-26T14:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8206.811041 secs 2024-04-26T14:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8209.575276 secs 2024-04-26T14:07:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8206.811041 secs 2024-04-26T14:07:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8209.575276 secs 2024-04-26T14:07:50.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:49 smithi033 ceph-mon[27845]: pgmap v4985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:49 smithi194 ceph-mon[33017]: pgmap v4985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:49 smithi096 ceph-mon[32753]: pgmap v4985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:51 smithi194 ceph-mon[33017]: pgmap v4986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:51 smithi033 ceph-mon[27845]: pgmap v4986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:51 smithi096 ceph-mon[32753]: pgmap v4986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:53.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:53.788+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:53.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:53.788+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:07:53.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:07:53.788+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:07:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:53 smithi194 ceph-mon[33017]: pgmap v4987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8211.811103 secs 2024-04-26T14:07:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8214.575372 secs 2024-04-26T14:07:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:53 smithi033 ceph-mon[27845]: pgmap v4987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8211.811103 secs 2024-04-26T14:07:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8214.575372 secs 2024-04-26T14:07:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:53 smithi096 ceph-mon[32753]: pgmap v4987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:07:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8211.811103 secs 2024-04-26T14:07:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8214.575372 secs 2024-04-26T14:07:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:55 smithi194 ceph-mon[33017]: pgmap v4988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:07:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:55 smithi033 ceph-mon[27845]: pgmap v4988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:07:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:55 smithi096 ceph-mon[32753]: pgmap v4988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:07:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:57 smithi194 ceph-mon[33017]: pgmap v4989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:57 smithi033 ceph-mon[27845]: pgmap v4989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:57 smithi096 ceph-mon[32753]: pgmap v4989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:07:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8216.811187 secs 2024-04-26T14:07:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8219.575493 secs 2024-04-26T14:07:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8216.811187 secs 2024-04-26T14:07:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8219.575493 secs 2024-04-26T14:07:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8216.811187 secs 2024-04-26T14:07:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8219.575493 secs 2024-04-26T14:08:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:07:59 smithi194 ceph-mon[33017]: pgmap v4990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:07:59 smithi033 ceph-mon[27845]: pgmap v4990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:07:59 smithi096 ceph-mon[32753]: pgmap v4990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:01 smithi194 ceph-mon[33017]: pgmap v4991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:01 smithi033 ceph-mon[27845]: pgmap v4991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:01 smithi096 ceph-mon[32753]: pgmap v4991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8221.811282 secs 2024-04-26T14:08:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:03 smithi194 ceph-mon[33017]: pgmap v4992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8224.575562 secs 2024-04-26T14:08:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8221.811282 secs 2024-04-26T14:08:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:03 smithi033 ceph-mon[27845]: pgmap v4992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8224.575562 secs 2024-04-26T14:08:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8221.811282 secs 2024-04-26T14:08:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:03 smithi096 ceph-mon[32753]: pgmap v4992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8224.575562 secs 2024-04-26T14:08:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:05 smithi194 ceph-mon[33017]: pgmap v4993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:05 smithi033 ceph-mon[27845]: pgmap v4993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:05 smithi096 ceph-mon[32753]: pgmap v4993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:07 smithi194 ceph-mon[33017]: pgmap v4994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:07 smithi033 ceph-mon[27845]: pgmap v4994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:07 smithi096 ceph-mon[32753]: pgmap v4994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:08.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:08.790+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:08.790+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:08:09.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8226.811367 secs 2024-04-26T14:08:09.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8229.575682 secs 2024-04-26T14:08:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8226.811367 secs 2024-04-26T14:08:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8229.575682 secs 2024-04-26T14:08:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8226.811367 secs 2024-04-26T14:08:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8229.575682 secs 2024-04-26T14:08:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:09 smithi194 ceph-mon[33017]: pgmap v4995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:09 smithi033 ceph-mon[27845]: pgmap v4995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:09 smithi096 ceph-mon[32753]: pgmap v4995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:11 smithi194 ceph-mon[33017]: pgmap v4996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:11 smithi033 ceph-mon[27845]: pgmap v4996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:11 smithi096 ceph-mon[32753]: pgmap v4996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8231.811452 secs 2024-04-26T14:08:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:13 smithi194 ceph-mon[33017]: pgmap v4997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8234.575813 secs 2024-04-26T14:08:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8231.811452 secs 2024-04-26T14:08:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:13 smithi033 ceph-mon[27845]: pgmap v4997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8234.575813 secs 2024-04-26T14:08:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8231.811452 secs 2024-04-26T14:08:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:13 smithi096 ceph-mon[32753]: pgmap v4997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8234.575813 secs 2024-04-26T14:08:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:15 smithi194 ceph-mon[33017]: pgmap v4998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:15 smithi033 ceph-mon[27845]: pgmap v4998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:15 smithi096 ceph-mon[32753]: pgmap v4998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:17 smithi194 ceph-mon[33017]: pgmap v4999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:17 smithi033 ceph-mon[27845]: pgmap v4999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:17 smithi096 ceph-mon[32753]: pgmap v4999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8236.811538 secs 2024-04-26T14:08:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8239.575909 secs 2024-04-26T14:08:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8236.811538 secs 2024-04-26T14:08:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8239.575909 secs 2024-04-26T14:08:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8236.811538 secs 2024-04-26T14:08:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8239.575909 secs 2024-04-26T14:08:20.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:19 smithi033 ceph-mon[27845]: pgmap v5000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:19 smithi194 ceph-mon[33017]: pgmap v5000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:19 smithi096 ceph-mon[32753]: pgmap v5000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:20 smithi194 ceph-mon[33017]: pgmap v5001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:20 smithi033 ceph-mon[27845]: pgmap v5001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:20 smithi096 ceph-mon[32753]: pgmap v5001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:23.192+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:08:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:23.192+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:08:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:23.192+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:08:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:23.192+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:08:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:23.797+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:23.797+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:23.797+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:08:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8241.811597 secs 2024-04-26T14:08:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:23 smithi033 ceph-mon[27845]: pgmap v5002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8244.576030 secs 2024-04-26T14:08:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8241.811597 secs 2024-04-26T14:08:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:23 smithi096 ceph-mon[32753]: pgmap v5002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8244.576030 secs 2024-04-26T14:08:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8241.811597 secs 2024-04-26T14:08:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:23 smithi194 ceph-mon[33017]: pgmap v5002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8244.576030 secs 2024-04-26T14:08:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:25 smithi033 ceph-mon[27845]: pgmap v5003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:25 smithi096 ceph-mon[32753]: pgmap v5003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:25 smithi194 ceph-mon[33017]: pgmap v5003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:27.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:27 smithi194 ceph-mon[33017]: pgmap v5004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:27 smithi033 ceph-mon[27845]: pgmap v5004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:27 smithi096 ceph-mon[32753]: pgmap v5004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8246.811686 secs 2024-04-26T14:08:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8249.576149 secs 2024-04-26T14:08:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8246.811686 secs 2024-04-26T14:08:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8249.576149 secs 2024-04-26T14:08:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8246.811686 secs 2024-04-26T14:08:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8249.576149 secs 2024-04-26T14:08:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:29 smithi033 ceph-mon[27845]: pgmap v5005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:29 smithi096 ceph-mon[32753]: pgmap v5005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:29 smithi194 ceph-mon[33017]: pgmap v5005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:31 smithi033 ceph-mon[27845]: pgmap v5006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:31 smithi096 ceph-mon[32753]: pgmap v5006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:31 smithi194 ceph-mon[33017]: pgmap v5006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8251.811746 secs 2024-04-26T14:08:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:33 smithi033 ceph-mon[27845]: pgmap v5007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8254.576267 secs 2024-04-26T14:08:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8251.811746 secs 2024-04-26T14:08:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:33 smithi096 ceph-mon[32753]: pgmap v5007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8254.576267 secs 2024-04-26T14:08:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8251.811746 secs 2024-04-26T14:08:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:33 smithi194 ceph-mon[33017]: pgmap v5007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8254.576267 secs 2024-04-26T14:08:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:35 smithi096 ceph-mon[32753]: pgmap v5008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:36.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:35 smithi194 ceph-mon[33017]: pgmap v5008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:35 smithi033 ceph-mon[27845]: pgmap v5008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:37 smithi096 ceph-mon[32753]: pgmap v5009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:37 smithi194 ceph-mon[33017]: pgmap v5009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:37 smithi033 ceph-mon[27845]: pgmap v5009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:38.797+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:38.797+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:38.797+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:08:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8256.811803 secs 2024-04-26T14:08:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8259.576385 secs 2024-04-26T14:08:39.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8256.811803 secs 2024-04-26T14:08:39.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8259.576385 secs 2024-04-26T14:08:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8256.811803 secs 2024-04-26T14:08:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8259.576385 secs 2024-04-26T14:08:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:39 smithi194 ceph-mon[33017]: pgmap v5010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:39 smithi033 ceph-mon[27845]: pgmap v5010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:39 smithi096 ceph-mon[32753]: pgmap v5010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:41 smithi194 ceph-mon[33017]: pgmap v5011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:41 smithi033 ceph-mon[27845]: pgmap v5011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:41 smithi096 ceph-mon[32753]: pgmap v5011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8261.811864 secs 2024-04-26T14:08:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:43 smithi194 ceph-mon[33017]: pgmap v5012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8264.576531 secs 2024-04-26T14:08:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8261.811864 secs 2024-04-26T14:08:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:43 smithi033 ceph-mon[27845]: pgmap v5012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8264.576531 secs 2024-04-26T14:08:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8261.811864 secs 2024-04-26T14:08:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:43 smithi096 ceph-mon[32753]: pgmap v5012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8264.576531 secs 2024-04-26T14:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:45 smithi194 ceph-mon[33017]: pgmap v5013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:08:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:45 smithi033 ceph-mon[27845]: pgmap v5013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:08:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:45 smithi096 ceph-mon[32753]: pgmap v5013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:08:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:08:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:08:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:08:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:08:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:08:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:08:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:08:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:08:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:08:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:08:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:47 smithi194 ceph-mon[33017]: pgmap v5014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:47 smithi194 ceph-mon[33017]: pgmap v5015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:47 smithi033 ceph-mon[27845]: pgmap v5014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:48.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:47 smithi033 ceph-mon[27845]: pgmap v5015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:47 smithi096 ceph-mon[32753]: pgmap v5014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:08:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:47 smithi096 ceph-mon[32753]: pgmap v5015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8266.811950 secs 2024-04-26T14:08:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8269.576682 secs 2024-04-26T14:08:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8266.811950 secs 2024-04-26T14:08:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8269.576682 secs 2024-04-26T14:08:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8266.811950 secs 2024-04-26T14:08:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8269.576682 secs 2024-04-26T14:08:50.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:49 smithi033 ceph-mon[27845]: pgmap v5016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:49 smithi194 ceph-mon[33017]: pgmap v5016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:49 smithi096 ceph-mon[32753]: pgmap v5016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:51 smithi194 ceph-mon[33017]: pgmap v5017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:51 smithi033 ceph-mon[27845]: pgmap v5017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:51 smithi096 ceph-mon[32753]: pgmap v5017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:53.814+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:53.814+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:08:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:08:53.814+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:08:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:53 smithi194 ceph-mon[33017]: pgmap v5018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8271.812033 secs 2024-04-26T14:08:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8274.576817 secs 2024-04-26T14:08:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:53 smithi033 ceph-mon[27845]: pgmap v5018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8271.812033 secs 2024-04-26T14:08:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8274.576817 secs 2024-04-26T14:08:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:53 smithi096 ceph-mon[32753]: pgmap v5018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:08:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8271.812033 secs 2024-04-26T14:08:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8274.576817 secs 2024-04-26T14:08:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:55 smithi194 ceph-mon[33017]: pgmap v5019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:08:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:55 smithi033 ceph-mon[27845]: pgmap v5019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:08:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:55 smithi096 ceph-mon[32753]: pgmap v5019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:08:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:57 smithi194 ceph-mon[33017]: pgmap v5020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:08:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:57 smithi033 ceph-mon[27845]: pgmap v5020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:08:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:57 smithi096 ceph-mon[32753]: pgmap v5020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:08:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8276.812118 secs 2024-04-26T14:08:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8279.576947 secs 2024-04-26T14:08:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8276.812118 secs 2024-04-26T14:08:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8279.576947 secs 2024-04-26T14:08:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8276.812118 secs 2024-04-26T14:08:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8279.576947 secs 2024-04-26T14:09:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:08:59 smithi194 ceph-mon[33017]: pgmap v5021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:08:59 smithi033 ceph-mon[27845]: pgmap v5021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:08:59 smithi096 ceph-mon[32753]: pgmap v5021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:01 smithi194 ceph-mon[33017]: pgmap v5022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:01 smithi033 ceph-mon[27845]: pgmap v5022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:01 smithi096 ceph-mon[32753]: pgmap v5022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:03 smithi194 ceph-mon[33017]: pgmap v5023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8281.812202 secs 2024-04-26T14:09:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8284.577062 secs 2024-04-26T14:09:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:03 smithi033 ceph-mon[27845]: pgmap v5023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8281.812202 secs 2024-04-26T14:09:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8284.577062 secs 2024-04-26T14:09:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:03 smithi096 ceph-mon[32753]: pgmap v5023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8281.812202 secs 2024-04-26T14:09:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8284.577062 secs 2024-04-26T14:09:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:05 smithi194 ceph-mon[33017]: pgmap v5024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:05 smithi033 ceph-mon[27845]: pgmap v5024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:05 smithi096 ceph-mon[32753]: pgmap v5024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:07 smithi194 ceph-mon[33017]: pgmap v5025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:07 smithi033 ceph-mon[27845]: pgmap v5025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:07 smithi096 ceph-mon[32753]: pgmap v5025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:08.829+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:08.829+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:08.829+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:09:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8286.812263 secs 2024-04-26T14:09:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8289.577179 secs 2024-04-26T14:09:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8286.812263 secs 2024-04-26T14:09:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8289.577179 secs 2024-04-26T14:09:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8286.812263 secs 2024-04-26T14:09:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8289.577179 secs 2024-04-26T14:09:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:09 smithi194 ceph-mon[33017]: pgmap v5026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:09 smithi033 ceph-mon[27845]: pgmap v5026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:09 smithi096 ceph-mon[32753]: pgmap v5026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:10 smithi194 ceph-mon[33017]: pgmap v5027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:10 smithi033 ceph-mon[27845]: pgmap v5027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:10 smithi096 ceph-mon[32753]: pgmap v5027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:13 smithi194 ceph-mon[33017]: pgmap v5028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8294.577296 secs 2024-04-26T14:09:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:13 smithi033 ceph-mon[27845]: pgmap v5028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8294.577296 secs 2024-04-26T14:09:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:13 smithi096 ceph-mon[32753]: pgmap v5028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8294.577296 secs 2024-04-26T14:09:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8291.812347 secs 2024-04-26T14:09:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8291.812347 secs 2024-04-26T14:09:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8291.812347 secs 2024-04-26T14:09:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:15 smithi033 ceph-mon[27845]: pgmap v5029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:15 smithi096 ceph-mon[32753]: pgmap v5029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:15 smithi194 ceph-mon[33017]: pgmap v5029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:17 smithi033 ceph-mon[27845]: pgmap v5030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:17 smithi096 ceph-mon[32753]: pgmap v5030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:17 smithi194 ceph-mon[33017]: pgmap v5030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8299.577408 secs 2024-04-26T14:09:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8299.577408 secs 2024-04-26T14:09:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8299.577408 secs 2024-04-26T14:09:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8296.812409 secs 2024-04-26T14:09:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:19 smithi033 ceph-mon[27845]: pgmap v5031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8296.812409 secs 2024-04-26T14:09:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:19 smithi096 ceph-mon[32753]: pgmap v5031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8296.812409 secs 2024-04-26T14:09:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:19 smithi194 ceph-mon[33017]: pgmap v5031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:21 smithi033 ceph-mon[27845]: pgmap v5032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:21 smithi096 ceph-mon[32753]: pgmap v5032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:21 smithi194 ceph-mon[33017]: pgmap v5032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:23 smithi033 ceph-mon[27845]: pgmap v5033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8301.812491 secs 2024-04-26T14:09:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8304.577523 secs 2024-04-26T14:09:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:23 smithi096 ceph-mon[32753]: pgmap v5033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8301.812491 secs 2024-04-26T14:09:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8304.577523 secs 2024-04-26T14:09:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:23.834+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:23.834+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:23.834+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:09:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:23 smithi194 ceph-mon[33017]: pgmap v5033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8301.812491 secs 2024-04-26T14:09:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8304.577523 secs 2024-04-26T14:09:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:25 smithi033 ceph-mon[27845]: pgmap v5034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:25 smithi096 ceph-mon[32753]: pgmap v5034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:25 smithi194 ceph-mon[33017]: pgmap v5034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:27.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:27 smithi194 ceph-mon[33017]: pgmap v5035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:27 smithi033 ceph-mon[27845]: pgmap v5035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:27 smithi096 ceph-mon[32753]: pgmap v5035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8306.812572 secs 2024-04-26T14:09:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8309.577621 secs 2024-04-26T14:09:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8306.812572 secs 2024-04-26T14:09:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8309.577621 secs 2024-04-26T14:09:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8306.812572 secs 2024-04-26T14:09:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8309.577621 secs 2024-04-26T14:09:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:29 smithi033 ceph-mon[27845]: pgmap v5036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:29 smithi096 ceph-mon[32753]: pgmap v5036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:29 smithi194 ceph-mon[33017]: pgmap v5036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:31 smithi033 ceph-mon[27845]: pgmap v5037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:31 smithi096 ceph-mon[32753]: pgmap v5037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:31 smithi194 ceph-mon[33017]: pgmap v5037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:33 smithi033 ceph-mon[27845]: pgmap v5038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8311.812655 secs 2024-04-26T14:09:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8314.577736 secs 2024-04-26T14:09:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:33 smithi096 ceph-mon[32753]: pgmap v5038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8311.812655 secs 2024-04-26T14:09:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8314.577736 secs 2024-04-26T14:09:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:33 smithi194 ceph-mon[33017]: pgmap v5038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8311.812655 secs 2024-04-26T14:09:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8314.577736 secs 2024-04-26T14:09:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:35 smithi096 ceph-mon[32753]: pgmap v5039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:35 smithi194 ceph-mon[33017]: pgmap v5039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:35 smithi033 ceph-mon[27845]: pgmap v5039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:37 smithi194 ceph-mon[33017]: pgmap v5040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:37 smithi033 ceph-mon[27845]: pgmap v5040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:37 smithi096 ceph-mon[32753]: pgmap v5040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:38.847+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:38.847+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:38.847+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:09:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8316.812715 secs 2024-04-26T14:09:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8319.577871 secs 2024-04-26T14:09:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8316.812715 secs 2024-04-26T14:09:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8319.577871 secs 2024-04-26T14:09:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8316.812715 secs 2024-04-26T14:09:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8319.577871 secs 2024-04-26T14:09:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:39 smithi194 ceph-mon[33017]: pgmap v5041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:39 smithi033 ceph-mon[27845]: pgmap v5041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:39 smithi096 ceph-mon[32753]: pgmap v5041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:41 smithi194 ceph-mon[33017]: pgmap v5042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:41 smithi033 ceph-mon[27845]: pgmap v5042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:41 smithi096 ceph-mon[32753]: pgmap v5042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:43 smithi194 ceph-mon[33017]: pgmap v5043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8321.812796 secs 2024-04-26T14:09:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8324.577991 secs 2024-04-26T14:09:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:43 smithi033 ceph-mon[27845]: pgmap v5043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8321.812796 secs 2024-04-26T14:09:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8324.577991 secs 2024-04-26T14:09:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:43 smithi096 ceph-mon[32753]: pgmap v5043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8321.812796 secs 2024-04-26T14:09:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8324.577991 secs 2024-04-26T14:09:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:45 smithi194 ceph-mon[33017]: pgmap v5044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:45 smithi033 ceph-mon[27845]: pgmap v5044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:45 smithi096 ceph-mon[32753]: pgmap v5044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:47 smithi194 ceph-mon[33017]: pgmap v5045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:09:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:47 smithi033 ceph-mon[27845]: pgmap v5045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:09:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:47 smithi096 ceph-mon[32753]: pgmap v5045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:09:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:09:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8326.812876 secs 2024-04-26T14:09:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:09:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:09:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:09:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8329.578136 secs 2024-04-26T14:09:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8326.812876 secs 2024-04-26T14:09:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:09:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:09:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:09:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8329.578136 secs 2024-04-26T14:09:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8326.812876 secs 2024-04-26T14:09:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:09:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:09:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:09:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8329.578136 secs 2024-04-26T14:09:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:49 smithi194 ceph-mon[33017]: pgmap v5046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:49 smithi194 ceph-mon[33017]: pgmap v5047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:49 smithi033 ceph-mon[27845]: pgmap v5046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:49 smithi033 ceph-mon[27845]: pgmap v5047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:49 smithi096 ceph-mon[32753]: pgmap v5046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:09:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:49 smithi096 ceph-mon[32753]: pgmap v5047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:51 smithi194 ceph-mon[33017]: pgmap v5048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:51 smithi033 ceph-mon[27845]: pgmap v5048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:51 smithi096 ceph-mon[32753]: pgmap v5048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:53.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:53.206+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:09:53.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:53.206+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:09:53.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:53.207+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T14:09:53.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:53.207+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T14:09:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:53 smithi194 ceph-mon[33017]: pgmap v5049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8331.812959 secs 2024-04-26T14:09:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8334.578230 secs 2024-04-26T14:09:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:53.863+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:53.864+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:09:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:09:53.864+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:09:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:53 smithi033 ceph-mon[27845]: pgmap v5049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8331.812959 secs 2024-04-26T14:09:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8334.578230 secs 2024-04-26T14:09:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:53 smithi096 ceph-mon[32753]: pgmap v5049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8331.812959 secs 2024-04-26T14:09:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8334.578230 secs 2024-04-26T14:09:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:55 smithi194 ceph-mon[33017]: pgmap v5050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:09:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:55 smithi033 ceph-mon[27845]: pgmap v5050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:09:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:55 smithi096 ceph-mon[32753]: pgmap v5050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:09:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:57 smithi194 ceph-mon[33017]: pgmap v5051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:57 smithi033 ceph-mon[27845]: pgmap v5051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:57 smithi096 ceph-mon[32753]: pgmap v5051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:09:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8336.813044 secs 2024-04-26T14:09:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8339.578346 secs 2024-04-26T14:09:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8336.813044 secs 2024-04-26T14:09:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8339.578346 secs 2024-04-26T14:09:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8336.813044 secs 2024-04-26T14:09:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8339.578346 secs 2024-04-26T14:09:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:09:59 smithi194 ceph-mon[33017]: pgmap v5052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:09:59 smithi033 ceph-mon[27845]: pgmap v5052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:09:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:09:59 smithi096 ceph-mon[32753]: pgmap v5052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:10:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:10:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:10:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:10:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:01 smithi194 ceph-mon[33017]: pgmap v5053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:01 smithi033 ceph-mon[27845]: pgmap v5053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:01 smithi096 ceph-mon[32753]: pgmap v5053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:03 smithi194 ceph-mon[33017]: pgmap v5054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8341.813102 secs 2024-04-26T14:10:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8344.578471 secs 2024-04-26T14:10:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:03 smithi033 ceph-mon[27845]: pgmap v5054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8341.813102 secs 2024-04-26T14:10:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8344.578471 secs 2024-04-26T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:03 smithi096 ceph-mon[32753]: pgmap v5054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8341.813102 secs 2024-04-26T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8344.578471 secs 2024-04-26T14:10:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:05 smithi194 ceph-mon[33017]: pgmap v5055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:05 smithi033 ceph-mon[27845]: pgmap v5055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:05 smithi096 ceph-mon[32753]: pgmap v5055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:07 smithi194 ceph-mon[33017]: pgmap v5056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:07 smithi033 ceph-mon[27845]: pgmap v5056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:07 smithi096 ceph-mon[32753]: pgmap v5056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8346.813193 secs 2024-04-26T14:10:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8349.578592 secs 2024-04-26T14:10:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:08.872+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:08.872+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:10:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:08.872+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8346.813193 secs 2024-04-26T14:10:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8349.578592 secs 2024-04-26T14:10:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8346.813193 secs 2024-04-26T14:10:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8349.578592 secs 2024-04-26T14:10:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:09 smithi194 ceph-mon[33017]: pgmap v5057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:09 smithi033 ceph-mon[27845]: pgmap v5057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:09 smithi096 ceph-mon[32753]: pgmap v5057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:11 smithi194 ceph-mon[33017]: pgmap v5058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:11 smithi033 ceph-mon[27845]: pgmap v5058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:11 smithi096 ceph-mon[32753]: pgmap v5058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:13 smithi194 ceph-mon[33017]: pgmap v5059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8351.813253 secs 2024-04-26T14:10:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8354.578713 secs 2024-04-26T14:10:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:13 smithi033 ceph-mon[27845]: pgmap v5059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8351.813253 secs 2024-04-26T14:10:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8354.578713 secs 2024-04-26T14:10:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:13 smithi096 ceph-mon[32753]: pgmap v5059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8351.813253 secs 2024-04-26T14:10:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8354.578713 secs 2024-04-26T14:10:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:15 smithi194 ceph-mon[33017]: pgmap v5060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:15 smithi033 ceph-mon[27845]: pgmap v5060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:15 smithi096 ceph-mon[32753]: pgmap v5060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:17 smithi194 ceph-mon[33017]: pgmap v5061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:17 smithi033 ceph-mon[27845]: pgmap v5061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:17 smithi096 ceph-mon[32753]: pgmap v5061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8356.813313 secs 2024-04-26T14:10:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8359.578850 secs 2024-04-26T14:10:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8356.813313 secs 2024-04-26T14:10:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8359.578850 secs 2024-04-26T14:10:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8356.813313 secs 2024-04-26T14:10:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8359.578850 secs 2024-04-26T14:10:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:19 smithi194 ceph-mon[33017]: pgmap v5062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:19 smithi033 ceph-mon[27845]: pgmap v5062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:19 smithi096 ceph-mon[32753]: pgmap v5062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:21 smithi194 ceph-mon[33017]: pgmap v5063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:21 smithi033 ceph-mon[27845]: pgmap v5063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:21 smithi096 ceph-mon[32753]: pgmap v5063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:23 smithi194 ceph-mon[33017]: pgmap v5064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8361.813395 secs 2024-04-26T14:10:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8364.578972 secs 2024-04-26T14:10:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:23.879+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:23.879+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:23.879+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:10:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:23 smithi033 ceph-mon[27845]: pgmap v5064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8361.813395 secs 2024-04-26T14:10:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8364.578972 secs 2024-04-26T14:10:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:23 smithi096 ceph-mon[32753]: pgmap v5064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8361.813395 secs 2024-04-26T14:10:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8364.578972 secs 2024-04-26T14:10:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:25 smithi033 ceph-mon[27845]: pgmap v5065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:25 smithi096 ceph-mon[32753]: pgmap v5065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:25 smithi194 ceph-mon[33017]: pgmap v5065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:27.904 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:27 smithi194 ceph-mon[33017]: pgmap v5066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:27 smithi033 ceph-mon[27845]: pgmap v5066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:27 smithi096 ceph-mon[32753]: pgmap v5066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8366.813475 secs 2024-04-26T14:10:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8369.579094 secs 2024-04-26T14:10:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8366.813475 secs 2024-04-26T14:10:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8369.579094 secs 2024-04-26T14:10:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8366.813475 secs 2024-04-26T14:10:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8369.579094 secs 2024-04-26T14:10:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:29 smithi033 ceph-mon[27845]: pgmap v5067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:29 smithi096 ceph-mon[32753]: pgmap v5067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:29 smithi194 ceph-mon[33017]: pgmap v5067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:31 smithi033 ceph-mon[27845]: pgmap v5068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:31 smithi096 ceph-mon[32753]: pgmap v5068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:31 smithi194 ceph-mon[33017]: pgmap v5068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:33 smithi033 ceph-mon[27845]: pgmap v5069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8371.813553 secs 2024-04-26T14:10:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8374.579218 secs 2024-04-26T14:10:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:33 smithi096 ceph-mon[32753]: pgmap v5069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8371.813553 secs 2024-04-26T14:10:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8374.579218 secs 2024-04-26T14:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:33 smithi194 ceph-mon[33017]: pgmap v5069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8371.813553 secs 2024-04-26T14:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8374.579218 secs 2024-04-26T14:10:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:35 smithi033 ceph-mon[27845]: pgmap v5070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:35 smithi096 ceph-mon[32753]: pgmap v5070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:35 smithi194 ceph-mon[33017]: pgmap v5070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:37 smithi033 ceph-mon[27845]: pgmap v5071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:37 smithi096 ceph-mon[32753]: pgmap v5071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:37 smithi194 ceph-mon[33017]: pgmap v5071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:38.893+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:38.893+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:38.893+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:10:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8376.813614 secs 2024-04-26T14:10:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8379.579338 secs 2024-04-26T14:10:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8376.813614 secs 2024-04-26T14:10:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8379.579338 secs 2024-04-26T14:10:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8376.813614 secs 2024-04-26T14:10:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8379.579338 secs 2024-04-26T14:10:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:39 smithi033 ceph-mon[27845]: pgmap v5072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:39 smithi096 ceph-mon[32753]: pgmap v5072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:40.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:39 smithi194 ceph-mon[33017]: pgmap v5072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:41 smithi033 ceph-mon[27845]: pgmap v5073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:41 smithi096 ceph-mon[32753]: pgmap v5073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:41 smithi194 ceph-mon[33017]: pgmap v5073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:43 smithi033 ceph-mon[27845]: pgmap v5074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8381.813694 secs 2024-04-26T14:10:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8384.579463 secs 2024-04-26T14:10:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:43 smithi096 ceph-mon[32753]: pgmap v5074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8381.813694 secs 2024-04-26T14:10:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8384.579463 secs 2024-04-26T14:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:43 smithi194 ceph-mon[33017]: pgmap v5074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8381.813694 secs 2024-04-26T14:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8384.579463 secs 2024-04-26T14:10:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:45 smithi096 ceph-mon[32753]: pgmap v5075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:45 smithi194 ceph-mon[33017]: pgmap v5075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:45 smithi033 ceph-mon[27845]: pgmap v5075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:47 smithi096 ceph-mon[32753]: pgmap v5076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:47 smithi194 ceph-mon[33017]: pgmap v5076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:47 smithi033 ceph-mon[27845]: pgmap v5076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:10:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8386.813776 secs 2024-04-26T14:10:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8389.579616 secs 2024-04-26T14:10:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:10:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8386.813776 secs 2024-04-26T14:10:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8389.579616 secs 2024-04-26T14:10:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:10:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8386.813776 secs 2024-04-26T14:10:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8389.579616 secs 2024-04-26T14:10:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:10:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:49 smithi194 ceph-mon[33017]: pgmap v5077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:49 smithi033 ceph-mon[27845]: pgmap v5077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:49 smithi096 ceph-mon[32753]: pgmap v5077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:10:51.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:10:51.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:10:51.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:50 smithi033 ceph-mon[27845]: pgmap v5078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:10:51.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:50 smithi033 ceph-mon[27845]: pgmap v5079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:51.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:10:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:10:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:10:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:50 smithi194 ceph-mon[33017]: pgmap v5078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:10:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:50 smithi194 ceph-mon[33017]: pgmap v5079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:10:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:10:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:10:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:50 smithi096 ceph-mon[32753]: pgmap v5078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:10:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:50 smithi096 ceph-mon[32753]: pgmap v5079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:10:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:52 smithi194 ceph-mon[33017]: pgmap v5080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:52 smithi033 ceph-mon[27845]: pgmap v5080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:52 smithi096 ceph-mon[32753]: pgmap v5080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:53.905+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:53.905+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:10:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:10:53.905+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8391.813860 secs 2024-04-26T14:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8394.579749 secs 2024-04-26T14:10:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8391.813860 secs 2024-04-26T14:10:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8394.579749 secs 2024-04-26T14:10:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8391.813860 secs 2024-04-26T14:10:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8394.579749 secs 2024-04-26T14:10:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:54 smithi194 ceph-mon[33017]: pgmap v5081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:54 smithi033 ceph-mon[27845]: pgmap v5081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:54 smithi096 ceph-mon[32753]: pgmap v5081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:56 smithi194 ceph-mon[33017]: pgmap v5082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:56 smithi033 ceph-mon[27845]: pgmap v5082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:56 smithi096 ceph-mon[32753]: pgmap v5082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8396.813942 secs 2024-04-26T14:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:58 smithi194 ceph-mon[33017]: pgmap v5083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:10:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8399.579868 secs 2024-04-26T14:10:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8396.813942 secs 2024-04-26T14:10:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:58 smithi033 ceph-mon[27845]: pgmap v5083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:10:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8399.579868 secs 2024-04-26T14:10:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8396.813942 secs 2024-04-26T14:10:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:58 smithi096 ceph-mon[32753]: pgmap v5083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:10:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:10:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8399.579868 secs 2024-04-26T14:11:01.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:00 smithi194 ceph-mon[33017]: pgmap v5084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:11:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:00 smithi033 ceph-mon[27845]: pgmap v5084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:11:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:00 smithi096 ceph-mon[32753]: pgmap v5084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:11:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:02 smithi194 ceph-mon[33017]: pgmap v5085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:02 smithi033 ceph-mon[27845]: pgmap v5085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:02 smithi096 ceph-mon[32753]: pgmap v5085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8401.814032 secs 2024-04-26T14:11:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8404.579989 secs 2024-04-26T14:11:04.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8401.814032 secs 2024-04-26T14:11:04.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8404.579989 secs 2024-04-26T14:11:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8401.814032 secs 2024-04-26T14:11:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8404.579989 secs 2024-04-26T14:11:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:04 smithi194 ceph-mon[33017]: pgmap v5086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:04 smithi033 ceph-mon[27845]: pgmap v5086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:04 smithi096 ceph-mon[32753]: pgmap v5086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:05 smithi194 ceph-mon[33017]: pgmap v5087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:05 smithi033 ceph-mon[27845]: pgmap v5087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:05 smithi096 ceph-mon[32753]: pgmap v5087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:08.919+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:08.919+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:11:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:08.919+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8406.814092 secs 2024-04-26T14:11:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:08 smithi194 ceph-mon[33017]: pgmap v5088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8409.580134 secs 2024-04-26T14:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8406.814092 secs 2024-04-26T14:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:08 smithi033 ceph-mon[27845]: pgmap v5088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8409.580134 secs 2024-04-26T14:11:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8406.814092 secs 2024-04-26T14:11:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:08 smithi096 ceph-mon[32753]: pgmap v5088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8409.580134 secs 2024-04-26T14:11:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:10 smithi194 ceph-mon[33017]: pgmap v5089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:10 smithi033 ceph-mon[27845]: pgmap v5089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:10 smithi096 ceph-mon[32753]: pgmap v5089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:12 smithi194 ceph-mon[33017]: pgmap v5090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:12 smithi033 ceph-mon[27845]: pgmap v5090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:12 smithi096 ceph-mon[32753]: pgmap v5090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8411.814177 secs 2024-04-26T14:11:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8414.580250 secs 2024-04-26T14:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8411.814177 secs 2024-04-26T14:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8414.580250 secs 2024-04-26T14:11:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8411.814177 secs 2024-04-26T14:11:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8414.580250 secs 2024-04-26T14:11:15.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:14 smithi033 ceph-mon[27845]: pgmap v5091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:14 smithi194 ceph-mon[33017]: pgmap v5091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:14 smithi096 ceph-mon[32753]: pgmap v5091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:16 smithi194 ceph-mon[33017]: pgmap v5092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:16 smithi033 ceph-mon[27845]: pgmap v5092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:16 smithi096 ceph-mon[32753]: pgmap v5092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8416.814253 secs 2024-04-26T14:11:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:18 smithi194 ceph-mon[33017]: pgmap v5093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8419.580376 secs 2024-04-26T14:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8416.814253 secs 2024-04-26T14:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:18 smithi033 ceph-mon[27845]: pgmap v5093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8419.580376 secs 2024-04-26T14:11:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8416.814253 secs 2024-04-26T14:11:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:18 smithi096 ceph-mon[32753]: pgmap v5093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8419.580376 secs 2024-04-26T14:11:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:20 smithi194 ceph-mon[33017]: pgmap v5094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:20 smithi033 ceph-mon[27845]: pgmap v5094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:20 smithi096 ceph-mon[32753]: pgmap v5094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T14:11:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T14:11:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3a80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef004e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T14:11:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3a80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef004e000 comp rx=0 tx=0).stop 2024-04-26T14:11:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3a80 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T14:11:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T14:11:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3a80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T14:11:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.731+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T14:11:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.732+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T14:11:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:21.734+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3fc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0054000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T14:11:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:21 smithi033 ceph-mon[27845]: pgmap v5095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:21 smithi096 ceph-mon[32753]: pgmap v5095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:21 smithi194 ceph-mon[33017]: pgmap v5095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:23.222+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:11:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:23.222+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:11:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:23.222+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:11:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:23.222+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:11:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8421.814313 secs 2024-04-26T14:11:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8424.580436 secs 2024-04-26T14:11:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8421.814313 secs 2024-04-26T14:11:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8424.580436 secs 2024-04-26T14:11:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8421.814313 secs 2024-04-26T14:11:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8424.580436 secs 2024-04-26T14:11:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:23.922+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:23.922+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:11:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:23.922+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:24 smithi033 ceph-mon[27845]: pgmap v5096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:24 smithi096 ceph-mon[32753]: pgmap v5096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:24 smithi194 ceph-mon[33017]: pgmap v5096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:26 smithi194 ceph-mon[33017]: pgmap v5097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:26 smithi033 ceph-mon[27845]: pgmap v5097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:26 smithi096 ceph-mon[32753]: pgmap v5097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8426.814372 secs 2024-04-26T14:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:28 smithi194 ceph-mon[33017]: pgmap v5098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8429.580514 secs 2024-04-26T14:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8426.814372 secs 2024-04-26T14:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:28 smithi033 ceph-mon[27845]: pgmap v5098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8429.580514 secs 2024-04-26T14:11:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8426.814372 secs 2024-04-26T14:11:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:28 smithi096 ceph-mon[32753]: pgmap v5098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8429.580514 secs 2024-04-26T14:11:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:30 smithi194 ceph-mon[33017]: pgmap v5099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:30 smithi033 ceph-mon[27845]: pgmap v5099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:30 smithi096 ceph-mon[32753]: pgmap v5099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:32 smithi194 ceph-mon[33017]: pgmap v5100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:32 smithi033 ceph-mon[27845]: pgmap v5100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:32 smithi096 ceph-mon[32753]: pgmap v5100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8431.814452 secs 2024-04-26T14:11:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8434.580605 secs 2024-04-26T14:11:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8431.814452 secs 2024-04-26T14:11:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8434.580605 secs 2024-04-26T14:11:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8431.814452 secs 2024-04-26T14:11:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8434.580605 secs 2024-04-26T14:11:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:34 smithi194 ceph-mon[33017]: pgmap v5101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:34 smithi033 ceph-mon[27845]: pgmap v5101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:34 smithi096 ceph-mon[32753]: pgmap v5101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:36 smithi194 ceph-mon[33017]: pgmap v5102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:36 smithi033 ceph-mon[27845]: pgmap v5102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:36 smithi096 ceph-mon[32753]: pgmap v5102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:37 smithi033 ceph-mon[27845]: pgmap v5103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:37 smithi096 ceph-mon[32753]: pgmap v5103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:37 smithi194 ceph-mon[33017]: pgmap v5103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:38.931+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:38.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:38.931+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:38.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:38.931+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:11:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8436.814534 secs 2024-04-26T14:11:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8439.580687 secs 2024-04-26T14:11:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8436.814534 secs 2024-04-26T14:11:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8439.580687 secs 2024-04-26T14:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8436.814534 secs 2024-04-26T14:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8439.580687 secs 2024-04-26T14:11:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:39 smithi033 ceph-mon[27845]: pgmap v5104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:39 smithi096 ceph-mon[32753]: pgmap v5104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:39 smithi194 ceph-mon[33017]: pgmap v5104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:42 smithi194 ceph-mon[33017]: pgmap v5105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:42 smithi033 ceph-mon[27845]: pgmap v5105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:42 smithi096 ceph-mon[32753]: pgmap v5105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:44.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8441.814613 secs 2024-04-26T14:11:44.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8444.580800 secs 2024-04-26T14:11:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8441.814613 secs 2024-04-26T14:11:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8444.580800 secs 2024-04-26T14:11:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8441.814613 secs 2024-04-26T14:11:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8444.580800 secs 2024-04-26T14:11:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:44 smithi194 ceph-mon[33017]: pgmap v5106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:44 smithi033 ceph-mon[27845]: pgmap v5106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:44 smithi096 ceph-mon[32753]: pgmap v5106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:11:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:46 smithi194 ceph-mon[33017]: pgmap v5107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:46 smithi033 ceph-mon[27845]: pgmap v5107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:46 smithi096 ceph-mon[32753]: pgmap v5107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:11:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8446.814695 secs 2024-04-26T14:11:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:48 smithi194 ceph-mon[33017]: pgmap v5108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8449.580927 secs 2024-04-26T14:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8446.814695 secs 2024-04-26T14:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:48 smithi033 ceph-mon[27845]: pgmap v5108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8449.580927 secs 2024-04-26T14:11:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8446.814695 secs 2024-04-26T14:11:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:48 smithi096 ceph-mon[32753]: pgmap v5108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8449.580927 secs 2024-04-26T14:11:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:50 smithi194 ceph-mon[33017]: pgmap v5109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:11:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:50 smithi033 ceph-mon[27845]: pgmap v5109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:11:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:50 smithi096 ceph-mon[32753]: pgmap v5109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:11:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:11:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:11:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:11:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:11:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:11:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:11:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:11:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:52 smithi033 ceph-mon[27845]: pgmap v5110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:52 smithi033 ceph-mon[27845]: pgmap v5111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:52 smithi096 ceph-mon[32753]: pgmap v5110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:52 smithi096 ceph-mon[32753]: pgmap v5111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:52 smithi194 ceph-mon[33017]: pgmap v5110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:11:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:52 smithi194 ceph-mon[33017]: pgmap v5111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:53.933+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:53.933+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:11:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:11:53.933+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:11:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8451.814774 secs 2024-04-26T14:11:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8454.581028 secs 2024-04-26T14:11:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:53 smithi033 ceph-mon[27845]: pgmap v5112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8451.814774 secs 2024-04-26T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8454.581028 secs 2024-04-26T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:53 smithi096 ceph-mon[32753]: pgmap v5112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8451.814774 secs 2024-04-26T14:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8454.581028 secs 2024-04-26T14:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:53 smithi194 ceph-mon[33017]: pgmap v5112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:56 smithi194 ceph-mon[33017]: pgmap v5113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:56 smithi033 ceph-mon[27845]: pgmap v5113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:56 smithi096 ceph-mon[32753]: pgmap v5113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:58 smithi194 ceph-mon[33017]: pgmap v5114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8456.814832 secs 2024-04-26T14:11:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:11:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8459.581149 secs 2024-04-26T14:11:58.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:58 smithi033 ceph-mon[27845]: pgmap v5114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:58.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8456.814832 secs 2024-04-26T14:11:58.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:11:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8459.581149 secs 2024-04-26T14:11:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:58 smithi096 ceph-mon[32753]: pgmap v5114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:11:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8456.814832 secs 2024-04-26T14:11:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:11:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8459.581149 secs 2024-04-26T14:12:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:00 smithi194 ceph-mon[33017]: pgmap v5115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:00 smithi033 ceph-mon[27845]: pgmap v5115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:00 smithi096 ceph-mon[32753]: pgmap v5115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:02 smithi194 ceph-mon[33017]: pgmap v5116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:12:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:02 smithi033 ceph-mon[27845]: pgmap v5116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:12:02.937 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:02 smithi096 ceph-mon[32753]: pgmap v5116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:12:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8461.814908 secs 2024-04-26T14:12:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8464.581272 secs 2024-04-26T14:12:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8461.814908 secs 2024-04-26T14:12:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8464.581272 secs 2024-04-26T14:12:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8461.814908 secs 2024-04-26T14:12:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8464.581272 secs 2024-04-26T14:12:04.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:04 smithi096 ceph-mon[32753]: pgmap v5117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:04 smithi194 ceph-mon[33017]: pgmap v5117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:04 smithi033 ceph-mon[27845]: pgmap v5117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:06 smithi194 ceph-mon[33017]: pgmap v5118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:06 smithi033 ceph-mon[27845]: pgmap v5118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:06 smithi096 ceph-mon[32753]: pgmap v5118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:08 smithi194 ceph-mon[33017]: pgmap v5119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8466.815014 secs 2024-04-26T14:12:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8469.581366 secs 2024-04-26T14:12:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:08 smithi033 ceph-mon[27845]: pgmap v5119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8466.815014 secs 2024-04-26T14:12:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8469.581366 secs 2024-04-26T14:12:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:08 smithi096 ceph-mon[32753]: pgmap v5119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8466.815014 secs 2024-04-26T14:12:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8469.581366 secs 2024-04-26T14:12:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:08.940+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:08.940+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:08.940+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:12:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:10 smithi194 ceph-mon[33017]: pgmap v5120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:10 smithi033 ceph-mon[27845]: pgmap v5120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:10 smithi096 ceph-mon[32753]: pgmap v5120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:12 smithi194 ceph-mon[33017]: pgmap v5121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:12 smithi033 ceph-mon[27845]: pgmap v5121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:12 smithi096 ceph-mon[32753]: pgmap v5121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8471.815101 secs 2024-04-26T14:12:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8474.581453 secs 2024-04-26T14:12:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8471.815101 secs 2024-04-26T14:12:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8474.581453 secs 2024-04-26T14:12:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8471.815101 secs 2024-04-26T14:12:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8474.581453 secs 2024-04-26T14:12:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:14 smithi194 ceph-mon[33017]: pgmap v5122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:14 smithi033 ceph-mon[27845]: pgmap v5122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:14 smithi096 ceph-mon[32753]: pgmap v5122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:16 smithi194 ceph-mon[33017]: pgmap v5123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:16 smithi033 ceph-mon[27845]: pgmap v5123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:16 smithi096 ceph-mon[32753]: pgmap v5123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:18 smithi194 ceph-mon[33017]: pgmap v5124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8476.815191 secs 2024-04-26T14:12:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8479.581577 secs 2024-04-26T14:12:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:18 smithi033 ceph-mon[27845]: pgmap v5124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8476.815191 secs 2024-04-26T14:12:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8479.581577 secs 2024-04-26T14:12:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:18 smithi096 ceph-mon[32753]: pgmap v5124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8476.815191 secs 2024-04-26T14:12:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8479.581577 secs 2024-04-26T14:12:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:20 smithi194 ceph-mon[33017]: pgmap v5125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:20 smithi033 ceph-mon[27845]: pgmap v5125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:20 smithi096 ceph-mon[32753]: pgmap v5125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:22 smithi194 ceph-mon[33017]: pgmap v5126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:22 smithi033 ceph-mon[27845]: pgmap v5126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:22 smithi096 ceph-mon[32753]: pgmap v5126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8481.815251 secs 2024-04-26T14:12:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8484.581718 secs 2024-04-26T14:12:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8481.815251 secs 2024-04-26T14:12:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8484.581718 secs 2024-04-26T14:12:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8481.815251 secs 2024-04-26T14:12:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8484.581718 secs 2024-04-26T14:12:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:23.941+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:23.941+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:23.941+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:12:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:24 smithi194 ceph-mon[33017]: pgmap v5127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:24 smithi033 ceph-mon[27845]: pgmap v5127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:24 smithi096 ceph-mon[32753]: pgmap v5127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:26 smithi194 ceph-mon[33017]: pgmap v5128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:26 smithi033 ceph-mon[27845]: pgmap v5128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:26 smithi096 ceph-mon[32753]: pgmap v5128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:28 smithi194 ceph-mon[33017]: pgmap v5129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8486.815311 secs 2024-04-26T14:12:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8489.581838 secs 2024-04-26T14:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:28 smithi033 ceph-mon[27845]: pgmap v5129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8486.815311 secs 2024-04-26T14:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8489.581838 secs 2024-04-26T14:12:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:28 smithi096 ceph-mon[32753]: pgmap v5129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8486.815311 secs 2024-04-26T14:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8489.581838 secs 2024-04-26T14:12:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:30 smithi194 ceph-mon[33017]: pgmap v5130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:30 smithi033 ceph-mon[27845]: pgmap v5130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:30 smithi096 ceph-mon[32753]: pgmap v5130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:32 smithi194 ceph-mon[33017]: pgmap v5131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:32 smithi033 ceph-mon[27845]: pgmap v5131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:32 smithi096 ceph-mon[32753]: pgmap v5131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8491.815395 secs 2024-04-26T14:12:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8494.581979 secs 2024-04-26T14:12:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8491.815395 secs 2024-04-26T14:12:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8494.581979 secs 2024-04-26T14:12:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8491.815395 secs 2024-04-26T14:12:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8494.581979 secs 2024-04-26T14:12:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:34 smithi194 ceph-mon[33017]: pgmap v5132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:34 smithi033 ceph-mon[27845]: pgmap v5132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:34 smithi096 ceph-mon[32753]: pgmap v5132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:36 smithi194 ceph-mon[33017]: pgmap v5133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:36 smithi033 ceph-mon[27845]: pgmap v5133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:36 smithi096 ceph-mon[32753]: pgmap v5133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:38 smithi033 ceph-mon[27845]: pgmap v5134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8496.815478 secs 2024-04-26T14:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8499.582124 secs 2024-04-26T14:12:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:38 smithi096 ceph-mon[32753]: pgmap v5134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8496.815478 secs 2024-04-26T14:12:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8499.582124 secs 2024-04-26T14:12:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:38.955+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:38.955+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:38.955+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:12:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:38 smithi194 ceph-mon[33017]: pgmap v5134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8496.815478 secs 2024-04-26T14:12:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8499.582124 secs 2024-04-26T14:12:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:40 smithi033 ceph-mon[27845]: pgmap v5135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:40 smithi096 ceph-mon[32753]: pgmap v5135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:40 smithi194 ceph-mon[33017]: pgmap v5135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:42 smithi033 ceph-mon[27845]: pgmap v5136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:42 smithi096 ceph-mon[32753]: pgmap v5136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:42 smithi194 ceph-mon[33017]: pgmap v5136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8501.815563 secs 2024-04-26T14:12:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8504.582185 secs 2024-04-26T14:12:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8501.815563 secs 2024-04-26T14:12:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8504.582185 secs 2024-04-26T14:12:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8501.815563 secs 2024-04-26T14:12:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8504.582185 secs 2024-04-26T14:12:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:44 smithi033 ceph-mon[27845]: pgmap v5137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:44 smithi096 ceph-mon[32753]: pgmap v5137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:44 smithi194 ceph-mon[33017]: pgmap v5137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:46 smithi033 ceph-mon[27845]: pgmap v5138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:46 smithi096 ceph-mon[32753]: pgmap v5138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:46 smithi194 ceph-mon[33017]: pgmap v5138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:48 smithi033 ceph-mon[27845]: pgmap v5139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8506.815647 secs 2024-04-26T14:12:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8509.582295 secs 2024-04-26T14:12:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:48 smithi096 ceph-mon[32753]: pgmap v5139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8506.815647 secs 2024-04-26T14:12:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8509.582295 secs 2024-04-26T14:12:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:48 smithi194 ceph-mon[33017]: pgmap v5139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8506.815647 secs 2024-04-26T14:12:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8509.582295 secs 2024-04-26T14:12:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:50 smithi033 ceph-mon[27845]: pgmap v5140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:50 smithi096 ceph-mon[32753]: pgmap v5140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:50 smithi194 ceph-mon[33017]: pgmap v5140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:12:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:52 smithi033 ceph-mon[27845]: pgmap v5141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:12:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:52 smithi096 ceph-mon[32753]: pgmap v5141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:12:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:52 smithi194 ceph-mon[33017]: pgmap v5141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:12:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8511.815732 secs 2024-04-26T14:12:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:12:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:12:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8514.582343 secs 2024-04-26T14:12:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8511.815732 secs 2024-04-26T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:12:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:12:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8514.582343 secs 2024-04-26T14:12:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:12:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:53.961+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:53.961+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:12:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:12:53.961+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8511.815732 secs 2024-04-26T14:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8514.582343 secs 2024-04-26T14:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:12:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:54 smithi033 ceph-mon[27845]: pgmap v5142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:54 smithi033 ceph-mon[27845]: pgmap v5143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:54 smithi096 ceph-mon[32753]: pgmap v5142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:54 smithi096 ceph-mon[32753]: pgmap v5143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:54 smithi194 ceph-mon[33017]: pgmap v5142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:12:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:54 smithi194 ceph-mon[33017]: pgmap v5143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:56 smithi033 ceph-mon[27845]: pgmap v5144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:56 smithi096 ceph-mon[32753]: pgmap v5144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:56 smithi194 ceph-mon[33017]: pgmap v5144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:58 smithi033 ceph-mon[27845]: pgmap v5145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8516.815815 secs 2024-04-26T14:12:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:12:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8519.582463 secs 2024-04-26T14:12:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:58 smithi096 ceph-mon[32753]: pgmap v5145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8516.815815 secs 2024-04-26T14:12:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:12:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8519.582463 secs 2024-04-26T14:12:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:58 smithi194 ceph-mon[33017]: pgmap v5145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:12:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8516.815815 secs 2024-04-26T14:12:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:12:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8519.582463 secs 2024-04-26T14:13:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:00 smithi096 ceph-mon[32753]: pgmap v5146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:13:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:00 smithi194 ceph-mon[33017]: pgmap v5146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:13:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:00 smithi033 ceph-mon[27845]: pgmap v5146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:13:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:02 smithi194 ceph-mon[33017]: pgmap v5147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:02 smithi033 ceph-mon[27845]: pgmap v5147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:02 smithi096 ceph-mon[32753]: pgmap v5147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8521.815896 secs 2024-04-26T14:13:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8524.582546 secs 2024-04-26T14:13:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8521.815896 secs 2024-04-26T14:13:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8524.582546 secs 2024-04-26T14:13:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8521.815896 secs 2024-04-26T14:13:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8524.582546 secs 2024-04-26T14:13:04.951 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:04 smithi033 ceph-mon[27845]: pgmap v5148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:13:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:04 smithi194 ceph-mon[33017]: pgmap v5148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:13:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:04 smithi096 ceph-mon[32753]: pgmap v5148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:13:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:06 smithi194 ceph-mon[33017]: pgmap v5149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:06 smithi033 ceph-mon[27845]: pgmap v5149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:06 smithi096 ceph-mon[32753]: pgmap v5149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:08.962+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:08.962+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:08.962+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:13:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:08 smithi194 ceph-mon[33017]: pgmap v5150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8526.815982 secs 2024-04-26T14:13:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8529.582651 secs 2024-04-26T14:13:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:08 smithi033 ceph-mon[27845]: pgmap v5150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8526.815982 secs 2024-04-26T14:13:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8529.582651 secs 2024-04-26T14:13:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:08 smithi096 ceph-mon[32753]: pgmap v5150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8526.815982 secs 2024-04-26T14:13:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8529.582651 secs 2024-04-26T14:13:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:10 smithi194 ceph-mon[33017]: pgmap v5151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:10 smithi033 ceph-mon[27845]: pgmap v5151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:10 smithi096 ceph-mon[32753]: pgmap v5151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:12 smithi194 ceph-mon[33017]: pgmap v5152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:12 smithi033 ceph-mon[27845]: pgmap v5152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:12 smithi096 ceph-mon[32753]: pgmap v5152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8531.816039 secs 2024-04-26T14:13:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8534.582789 secs 2024-04-26T14:13:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8531.816039 secs 2024-04-26T14:13:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8534.582789 secs 2024-04-26T14:13:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8531.816039 secs 2024-04-26T14:13:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8534.582789 secs 2024-04-26T14:13:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:14 smithi194 ceph-mon[33017]: pgmap v5153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:14 smithi033 ceph-mon[27845]: pgmap v5153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:14 smithi096 ceph-mon[32753]: pgmap v5153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:15 smithi194 ceph-mon[33017]: pgmap v5154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:15 smithi033 ceph-mon[27845]: pgmap v5154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:15 smithi096 ceph-mon[32753]: pgmap v5154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:18 smithi194 ceph-mon[33017]: pgmap v5155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:18 smithi033 ceph-mon[27845]: pgmap v5155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:18 smithi096 ceph-mon[32753]: pgmap v5155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8536.816123 secs 2024-04-26T14:13:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8539.582912 secs 2024-04-26T14:13:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8536.816123 secs 2024-04-26T14:13:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8539.582912 secs 2024-04-26T14:13:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8536.816123 secs 2024-04-26T14:13:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8539.582912 secs 2024-04-26T14:13:20.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:20 smithi033 ceph-mon[27845]: pgmap v5156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:20 smithi194 ceph-mon[33017]: pgmap v5156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:20 smithi096 ceph-mon[32753]: pgmap v5156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:22 smithi194 ceph-mon[33017]: pgmap v5157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:22 smithi033 ceph-mon[27845]: pgmap v5157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:22 smithi096 ceph-mon[32753]: pgmap v5157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:23.970+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:23.971+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:23.971+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:13:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8541.816213 secs 2024-04-26T14:13:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8544.583066 secs 2024-04-26T14:13:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:24 smithi194 ceph-mon[33017]: pgmap v5158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8541.816213 secs 2024-04-26T14:13:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8544.583066 secs 2024-04-26T14:13:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:24 smithi033 ceph-mon[27845]: pgmap v5158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8541.816213 secs 2024-04-26T14:13:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8544.583066 secs 2024-04-26T14:13:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:24 smithi096 ceph-mon[32753]: pgmap v5158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:26 smithi194 ceph-mon[33017]: pgmap v5159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:26 smithi033 ceph-mon[27845]: pgmap v5159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:26 smithi096 ceph-mon[32753]: pgmap v5159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:28 smithi194 ceph-mon[33017]: pgmap v5160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:28 smithi033 ceph-mon[27845]: pgmap v5160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:28 smithi096 ceph-mon[32753]: pgmap v5160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8546.816291 secs 2024-04-26T14:13:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8549.583217 secs 2024-04-26T14:13:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8546.816291 secs 2024-04-26T14:13:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8549.583217 secs 2024-04-26T14:13:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8546.816291 secs 2024-04-26T14:13:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8549.583217 secs 2024-04-26T14:13:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:30 smithi194 ceph-mon[33017]: pgmap v5161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:30 smithi033 ceph-mon[27845]: pgmap v5161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:30 smithi096 ceph-mon[32753]: pgmap v5161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:32 smithi194 ceph-mon[33017]: pgmap v5162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:32 smithi033 ceph-mon[27845]: pgmap v5162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:32 smithi096 ceph-mon[32753]: pgmap v5162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8551.816376 secs 2024-04-26T14:13:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8554.583334 secs 2024-04-26T14:13:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:34 smithi194 ceph-mon[33017]: pgmap v5163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8551.816376 secs 2024-04-26T14:13:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8554.583334 secs 2024-04-26T14:13:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:34 smithi033 ceph-mon[27845]: pgmap v5163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8551.816376 secs 2024-04-26T14:13:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8554.583334 secs 2024-04-26T14:13:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:34 smithi096 ceph-mon[32753]: pgmap v5163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:36 smithi194 ceph-mon[33017]: pgmap v5164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:36 smithi033 ceph-mon[27845]: pgmap v5164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:36 smithi096 ceph-mon[32753]: pgmap v5164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:38 smithi194 ceph-mon[33017]: pgmap v5165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:38 smithi033 ceph-mon[27845]: pgmap v5165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:38 smithi096 ceph-mon[32753]: pgmap v5165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:38.986+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:38.986+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:38.986+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:13:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8556.816435 secs 2024-04-26T14:13:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8559.583450 secs 2024-04-26T14:13:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8556.816435 secs 2024-04-26T14:13:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8559.583450 secs 2024-04-26T14:13:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8556.816435 secs 2024-04-26T14:13:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8559.583450 secs 2024-04-26T14:13:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:40 smithi194 ceph-mon[33017]: pgmap v5166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:40 smithi033 ceph-mon[27845]: pgmap v5166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:40 smithi096 ceph-mon[32753]: pgmap v5166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:42 smithi194 ceph-mon[33017]: pgmap v5167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:42 smithi033 ceph-mon[27845]: pgmap v5167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:42 smithi096 ceph-mon[32753]: pgmap v5167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8561.816521 secs 2024-04-26T14:13:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8564.583523 secs 2024-04-26T14:13:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:44 smithi194 ceph-mon[33017]: pgmap v5168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8561.816521 secs 2024-04-26T14:13:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8564.583523 secs 2024-04-26T14:13:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:44 smithi033 ceph-mon[27845]: pgmap v5168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8561.816521 secs 2024-04-26T14:13:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8564.583523 secs 2024-04-26T14:13:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:44 smithi096 ceph-mon[32753]: pgmap v5168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:45 smithi194 ceph-mon[33017]: pgmap v5169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:45 smithi033 ceph-mon[27845]: pgmap v5169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:45 smithi096 ceph-mon[32753]: pgmap v5169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:48 smithi194 ceph-mon[33017]: pgmap v5170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:48 smithi033 ceph-mon[27845]: pgmap v5170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:48 smithi096 ceph-mon[32753]: pgmap v5170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8566.816607 secs 2024-04-26T14:13:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8569.583620 secs 2024-04-26T14:13:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8566.816607 secs 2024-04-26T14:13:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8569.583620 secs 2024-04-26T14:13:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8566.816607 secs 2024-04-26T14:13:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8569.583620 secs 2024-04-26T14:13:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:50 smithi194 ceph-mon[33017]: pgmap v5171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:50 smithi033 ceph-mon[27845]: pgmap v5171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:50 smithi096 ceph-mon[32753]: pgmap v5171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:52 smithi033 ceph-mon[27845]: pgmap v5172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:52 smithi194 ceph-mon[33017]: pgmap v5172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:52 smithi096 ceph-mon[32753]: pgmap v5172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:13:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:53.997+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:53.997+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:13:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:13:53.997+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:13:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8571.816691 secs 2024-04-26T14:13:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8574.583756 secs 2024-04-26T14:13:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:54 smithi194 ceph-mon[33017]: pgmap v5173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:13:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8571.816691 secs 2024-04-26T14:13:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8574.583756 secs 2024-04-26T14:13:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:54 smithi033 ceph-mon[27845]: pgmap v5173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:13:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8571.816691 secs 2024-04-26T14:13:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8574.583756 secs 2024-04-26T14:13:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:54 smithi096 ceph-mon[32753]: pgmap v5173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:13:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:13:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:13:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:13:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:55 smithi194 ceph-mon[33017]: pgmap v5174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:13:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:55 smithi194 ceph-mon[33017]: pgmap v5175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:13:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:13:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:13:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:55 smithi033 ceph-mon[27845]: pgmap v5174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:13:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:55 smithi033 ceph-mon[27845]: pgmap v5175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:13:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:13:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:13:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:55 smithi096 ceph-mon[32753]: pgmap v5174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:13:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:55 smithi096 ceph-mon[32753]: pgmap v5175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:13:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:56 smithi194 ceph-mon[33017]: pgmap v5176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:56 smithi033 ceph-mon[27845]: pgmap v5176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:56 smithi096 ceph-mon[32753]: pgmap v5176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8576.816774 secs 2024-04-26T14:13:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8579.583874 secs 2024-04-26T14:13:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8576.816774 secs 2024-04-26T14:13:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8579.583874 secs 2024-04-26T14:13:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8576.816774 secs 2024-04-26T14:13:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8579.583874 secs 2024-04-26T14:13:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:13:59 smithi194 ceph-mon[33017]: pgmap v5177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:13:59 smithi033 ceph-mon[27845]: pgmap v5177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:13:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:13:59 smithi096 ceph-mon[32753]: pgmap v5177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:14:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:00 smithi194 ceph-mon[33017]: pgmap v5178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:14:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:00 smithi033 ceph-mon[27845]: pgmap v5178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:14:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:00 smithi096 ceph-mon[32753]: pgmap v5178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:14:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:02 smithi194 ceph-mon[33017]: pgmap v5179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:14:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:02 smithi033 ceph-mon[27845]: pgmap v5179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:14:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:02 smithi096 ceph-mon[32753]: pgmap v5179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:14:04.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8581.816860 secs 2024-04-26T14:14:04.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8584.584006 secs 2024-04-26T14:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8581.816860 secs 2024-04-26T14:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8584.584006 secs 2024-04-26T14:14:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8581.816860 secs 2024-04-26T14:14:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8584.584006 secs 2024-04-26T14:14:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:04 smithi194 ceph-mon[33017]: pgmap v5180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:14:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:04 smithi033 ceph-mon[27845]: pgmap v5180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:14:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:04 smithi096 ceph-mon[32753]: pgmap v5180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:14:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:07 smithi194 ceph-mon[33017]: pgmap v5181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:07 smithi033 ceph-mon[27845]: pgmap v5181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:07 smithi096 ceph-mon[32753]: pgmap v5181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:09.005+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:09.005+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:14:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:09.005+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8586.816921 secs 2024-04-26T14:14:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8589.584169 secs 2024-04-26T14:14:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8586.816921 secs 2024-04-26T14:14:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8589.584169 secs 2024-04-26T14:14:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8586.816921 secs 2024-04-26T14:14:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8589.584169 secs 2024-04-26T14:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:09 smithi194 ceph-mon[33017]: pgmap v5182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:09 smithi033 ceph-mon[27845]: pgmap v5182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:09 smithi096 ceph-mon[32753]: pgmap v5182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:11 smithi194 ceph-mon[33017]: pgmap v5183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:11 smithi033 ceph-mon[27845]: pgmap v5183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:11 smithi096 ceph-mon[32753]: pgmap v5183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8591.817003 secs 2024-04-26T14:14:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:13 smithi194 ceph-mon[33017]: pgmap v5184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8594.584251 secs 2024-04-26T14:14:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8591.817003 secs 2024-04-26T14:14:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:13 smithi033 ceph-mon[27845]: pgmap v5184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8594.584251 secs 2024-04-26T14:14:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8591.817003 secs 2024-04-26T14:14:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:13 smithi096 ceph-mon[32753]: pgmap v5184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8594.584251 secs 2024-04-26T14:14:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:15 smithi194 ceph-mon[33017]: pgmap v5185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:15 smithi033 ceph-mon[27845]: pgmap v5185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:15 smithi096 ceph-mon[32753]: pgmap v5185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:17 smithi194 ceph-mon[33017]: pgmap v5186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:17 smithi033 ceph-mon[27845]: pgmap v5186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:17 smithi096 ceph-mon[32753]: pgmap v5186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8596.817065 secs 2024-04-26T14:14:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8599.584375 secs 2024-04-26T14:14:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8596.817065 secs 2024-04-26T14:14:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8599.584375 secs 2024-04-26T14:14:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8596.817065 secs 2024-04-26T14:14:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8599.584375 secs 2024-04-26T14:14:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:19 smithi194 ceph-mon[33017]: pgmap v5187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:19 smithi033 ceph-mon[27845]: pgmap v5187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:19 smithi096 ceph-mon[32753]: pgmap v5187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:22.255 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:21 smithi033 ceph-mon[27845]: pgmap v5188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:21 smithi194 ceph-mon[33017]: pgmap v5188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:21 smithi096 ceph-mon[32753]: pgmap v5188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:23.254+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:14:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:23.254+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:14:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:23.254+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T14:14:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:23.254+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T14:14:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:24.020+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:24.020+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:24.020+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:14:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8601.817150 secs 2024-04-26T14:14:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:23 smithi194 ceph-mon[33017]: pgmap v5189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8604.584454 secs 2024-04-26T14:14:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8601.817150 secs 2024-04-26T14:14:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:23 smithi033 ceph-mon[27845]: pgmap v5189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8604.584454 secs 2024-04-26T14:14:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8601.817150 secs 2024-04-26T14:14:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:23 smithi096 ceph-mon[32753]: pgmap v5189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8604.584454 secs 2024-04-26T14:14:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:26 smithi194 ceph-mon[33017]: pgmap v5190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:26 smithi033 ceph-mon[27845]: pgmap v5190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:26 smithi096 ceph-mon[32753]: pgmap v5190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:28 smithi194 ceph-mon[33017]: pgmap v5191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:28 smithi033 ceph-mon[27845]: pgmap v5191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:28 smithi096 ceph-mon[32753]: pgmap v5191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8606.817235 secs 2024-04-26T14:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8609.584579 secs 2024-04-26T14:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8606.817235 secs 2024-04-26T14:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8609.584579 secs 2024-04-26T14:14:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8606.817235 secs 2024-04-26T14:14:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8609.584579 secs 2024-04-26T14:14:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:30 smithi194 ceph-mon[33017]: pgmap v5192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:30 smithi033 ceph-mon[27845]: pgmap v5192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:30 smithi096 ceph-mon[32753]: pgmap v5192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:32 smithi194 ceph-mon[33017]: pgmap v5193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:32 smithi033 ceph-mon[27845]: pgmap v5193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:32 smithi096 ceph-mon[32753]: pgmap v5193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8611.817323 secs 2024-04-26T14:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:34 smithi194 ceph-mon[33017]: pgmap v5194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8614.584708 secs 2024-04-26T14:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8611.817323 secs 2024-04-26T14:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:34 smithi033 ceph-mon[27845]: pgmap v5194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8614.584708 secs 2024-04-26T14:14:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8611.817323 secs 2024-04-26T14:14:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:34 smithi096 ceph-mon[32753]: pgmap v5194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8614.584708 secs 2024-04-26T14:14:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:36 smithi194 ceph-mon[33017]: pgmap v5195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:36 smithi033 ceph-mon[27845]: pgmap v5195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:36 smithi096 ceph-mon[32753]: pgmap v5195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:38 smithi194 ceph-mon[33017]: pgmap v5196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:38 smithi033 ceph-mon[27845]: pgmap v5196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:38 smithi096 ceph-mon[32753]: pgmap v5196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:39.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:39.025+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:39.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:39.025+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:39.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:39.025+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8616.817409 secs 2024-04-26T14:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8619.584771 secs 2024-04-26T14:14:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8616.817409 secs 2024-04-26T14:14:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8619.584771 secs 2024-04-26T14:14:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8616.817409 secs 2024-04-26T14:14:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8619.584771 secs 2024-04-26T14:14:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:40 smithi194 ceph-mon[33017]: pgmap v5197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:40 smithi033 ceph-mon[27845]: pgmap v5197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:40 smithi096 ceph-mon[32753]: pgmap v5197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:42 smithi194 ceph-mon[33017]: pgmap v5198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:42 smithi033 ceph-mon[27845]: pgmap v5198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:42 smithi096 ceph-mon[32753]: pgmap v5198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8621.817496 secs 2024-04-26T14:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:44 smithi194 ceph-mon[33017]: pgmap v5199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8624.584898 secs 2024-04-26T14:14:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8621.817496 secs 2024-04-26T14:14:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:44 smithi033 ceph-mon[27845]: pgmap v5199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8624.584898 secs 2024-04-26T14:14:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8621.817496 secs 2024-04-26T14:14:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:44 smithi096 ceph-mon[32753]: pgmap v5199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8624.584898 secs 2024-04-26T14:14:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:46 smithi194 ceph-mon[33017]: pgmap v5200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:46 smithi033 ceph-mon[27845]: pgmap v5200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:46 smithi096 ceph-mon[32753]: pgmap v5200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:48 smithi194 ceph-mon[33017]: pgmap v5201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:48 smithi033 ceph-mon[27845]: pgmap v5201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:48 smithi096 ceph-mon[32753]: pgmap v5201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8626.817578 secs 2024-04-26T14:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8629.585024 secs 2024-04-26T14:14:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8626.817578 secs 2024-04-26T14:14:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8629.585024 secs 2024-04-26T14:14:49.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8626.817578 secs 2024-04-26T14:14:49.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8629.585024 secs 2024-04-26T14:14:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:50 smithi194 ceph-mon[33017]: pgmap v5202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:50 smithi033 ceph-mon[27845]: pgmap v5202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:50 smithi096 ceph-mon[32753]: pgmap v5202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:52 smithi194 ceph-mon[33017]: pgmap v5203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:52 smithi033 ceph-mon[27845]: pgmap v5203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:52 smithi096 ceph-mon[32753]: pgmap v5203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:54.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:54.031+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:14:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:14:54.031+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8631.817661 secs 2024-04-26T14:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:54 smithi194 ceph-mon[33017]: pgmap v5204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8634.585152 secs 2024-04-26T14:14:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8631.817661 secs 2024-04-26T14:14:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:54 smithi033 ceph-mon[27845]: pgmap v5204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8634.585152 secs 2024-04-26T14:14:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8631.817661 secs 2024-04-26T14:14:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:54 smithi096 ceph-mon[32753]: pgmap v5204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8634.585152 secs 2024-04-26T14:14:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:56 smithi033 ceph-mon[27845]: pgmap v5205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:14:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:56 smithi096 ceph-mon[32753]: pgmap v5205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:14:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:56 smithi194 ceph-mon[33017]: pgmap v5205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:14:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:14:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:58 smithi033 ceph-mon[27845]: pgmap v5206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:14:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:14:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:58 smithi033 ceph-mon[27845]: pgmap v5207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:14:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:58 smithi033 ceph-mon[27845]: pgmap v5208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:14:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:58 smithi096 ceph-mon[32753]: pgmap v5206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:14:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:14:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:58 smithi096 ceph-mon[32753]: pgmap v5207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:14:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:58 smithi096 ceph-mon[32753]: pgmap v5208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:14:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:58.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:58 smithi194 ceph-mon[33017]: pgmap v5206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:14:58.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:14:58.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:14:58.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:58 smithi194 ceph-mon[33017]: pgmap v5207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:14:58.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:58 smithi194 ceph-mon[33017]: pgmap v5208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:14:58.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:14:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8636.817749 secs 2024-04-26T14:14:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8639.585245 secs 2024-04-26T14:14:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8636.817749 secs 2024-04-26T14:14:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8639.585245 secs 2024-04-26T14:14:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8636.817749 secs 2024-04-26T14:14:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8639.585245 secs 2024-04-26T14:15:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:00 smithi033 ceph-mon[27845]: pgmap v5209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:00 smithi096 ceph-mon[32753]: pgmap v5209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:00 smithi194 ceph-mon[33017]: pgmap v5209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:02 smithi194 ceph-mon[33017]: pgmap v5210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:02 smithi033 ceph-mon[27845]: pgmap v5210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:02 smithi096 ceph-mon[32753]: pgmap v5210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8641.817834 secs 2024-04-26T14:15:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8644.585358 secs 2024-04-26T14:15:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8641.817834 secs 2024-04-26T14:15:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8644.585358 secs 2024-04-26T14:15:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8641.817834 secs 2024-04-26T14:15:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8644.585358 secs 2024-04-26T14:15:04.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:04 smithi096 ceph-mon[32753]: pgmap v5211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:15:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:04 smithi194 ceph-mon[33017]: pgmap v5211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:15:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:04 smithi033 ceph-mon[27845]: pgmap v5211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:15:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:06 smithi194 ceph-mon[33017]: pgmap v5212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:06 smithi033 ceph-mon[27845]: pgmap v5212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:06 smithi096 ceph-mon[32753]: pgmap v5212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:15:08.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:08 smithi033 ceph-mon[27845]: pgmap v5213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:15:08.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8646.817917 secs 2024-04-26T14:15:08.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8649.585482 secs 2024-04-26T14:15:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:08 smithi194 ceph-mon[33017]: pgmap v5213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:15:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8646.817917 secs 2024-04-26T14:15:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8649.585482 secs 2024-04-26T14:15:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:08 smithi096 ceph-mon[32753]: pgmap v5213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:15:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8646.817917 secs 2024-04-26T14:15:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8649.585482 secs 2024-04-26T14:15:09.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:09.046+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:09.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:09.047+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:09.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:09.047+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:15:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:10 smithi194 ceph-mon[33017]: pgmap v5214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:10 smithi033 ceph-mon[27845]: pgmap v5214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:10 smithi096 ceph-mon[32753]: pgmap v5214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:12 smithi194 ceph-mon[33017]: pgmap v5215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:12 smithi033 ceph-mon[27845]: pgmap v5215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:12 smithi096 ceph-mon[32753]: pgmap v5215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8651.818007 secs 2024-04-26T14:15:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8654.585609 secs 2024-04-26T14:15:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8651.818007 secs 2024-04-26T14:15:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8654.585609 secs 2024-04-26T14:15:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8651.818007 secs 2024-04-26T14:15:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8654.585609 secs 2024-04-26T14:15:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:14 smithi194 ceph-mon[33017]: pgmap v5216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:14 smithi033 ceph-mon[27845]: pgmap v5216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:14 smithi096 ceph-mon[32753]: pgmap v5216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:16 smithi033 ceph-mon[27845]: pgmap v5217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:16 smithi096 ceph-mon[32753]: pgmap v5217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:16 smithi194 ceph-mon[33017]: pgmap v5217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:18 smithi033 ceph-mon[27845]: pgmap v5218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8656.818094 secs 2024-04-26T14:15:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8659.585740 secs 2024-04-26T14:15:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:18 smithi096 ceph-mon[32753]: pgmap v5218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8656.818094 secs 2024-04-26T14:15:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8659.585740 secs 2024-04-26T14:15:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:18 smithi194 ceph-mon[33017]: pgmap v5218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8656.818094 secs 2024-04-26T14:15:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8659.585740 secs 2024-04-26T14:15:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:20 smithi033 ceph-mon[27845]: pgmap v5219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:20 smithi096 ceph-mon[32753]: pgmap v5219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:20 smithi194 ceph-mon[33017]: pgmap v5219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:22 smithi033 ceph-mon[27845]: pgmap v5220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:22.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:22 smithi096 ceph-mon[32753]: pgmap v5220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:22 smithi194 ceph-mon[33017]: pgmap v5220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8661.818196 secs 2024-04-26T14:15:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8664.585848 secs 2024-04-26T14:15:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8661.818196 secs 2024-04-26T14:15:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8664.585848 secs 2024-04-26T14:15:24.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:24.061+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:24.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:24.061+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:24.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:24.061+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:15:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8661.818196 secs 2024-04-26T14:15:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8664.585848 secs 2024-04-26T14:15:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:24 smithi033 ceph-mon[27845]: pgmap v5221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:24 smithi096 ceph-mon[32753]: pgmap v5221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:24 smithi194 ceph-mon[33017]: pgmap v5221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:26 smithi033 ceph-mon[27845]: pgmap v5222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:26 smithi096 ceph-mon[32753]: pgmap v5222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:26 smithi194 ceph-mon[33017]: pgmap v5222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:28.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:28 smithi194 ceph-mon[33017]: pgmap v5223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:28.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8666.818259 secs 2024-04-26T14:15:28.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8669.585965 secs 2024-04-26T14:15:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:28 smithi033 ceph-mon[27845]: pgmap v5223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8666.818259 secs 2024-04-26T14:15:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8669.585965 secs 2024-04-26T14:15:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:28 smithi096 ceph-mon[32753]: pgmap v5223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8666.818259 secs 2024-04-26T14:15:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8669.585965 secs 2024-04-26T14:15:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:30 smithi033 ceph-mon[27845]: pgmap v5224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:30 smithi096 ceph-mon[32753]: pgmap v5224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:30 smithi194 ceph-mon[33017]: pgmap v5224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:32 smithi033 ceph-mon[27845]: pgmap v5225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:32 smithi096 ceph-mon[32753]: pgmap v5225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:32 smithi194 ceph-mon[33017]: pgmap v5225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8671.818316 secs 2024-04-26T14:15:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8674.586099 secs 2024-04-26T14:15:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8671.818316 secs 2024-04-26T14:15:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8674.586099 secs 2024-04-26T14:15:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8671.818316 secs 2024-04-26T14:15:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8674.586099 secs 2024-04-26T14:15:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:34 smithi033 ceph-mon[27845]: pgmap v5226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:34 smithi096 ceph-mon[32753]: pgmap v5226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:34 smithi194 ceph-mon[33017]: pgmap v5226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:15:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:36 smithi033 ceph-mon[27845]: pgmap v5227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:36 smithi096 ceph-mon[32753]: pgmap v5227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:36 smithi194 ceph-mon[33017]: pgmap v5227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:15:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:38 smithi096 ceph-mon[32753]: pgmap v5228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8676.818377 secs 2024-04-26T14:15:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8679.586185 secs 2024-04-26T14:15:39.071 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:39.070+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:39.071 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:39.070+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:39.071 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:39.070+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:15:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:38 smithi194 ceph-mon[33017]: pgmap v5228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8676.818377 secs 2024-04-26T14:15:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8679.586185 secs 2024-04-26T14:15:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:38 smithi033 ceph-mon[27845]: pgmap v5228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8676.818377 secs 2024-04-26T14:15:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8679.586185 secs 2024-04-26T14:15:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:40 smithi194 ceph-mon[33017]: pgmap v5229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:40 smithi033 ceph-mon[27845]: pgmap v5229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:40 smithi096 ceph-mon[32753]: pgmap v5229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:42 smithi194 ceph-mon[33017]: pgmap v5230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:42 smithi033 ceph-mon[27845]: pgmap v5230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:42 smithi096 ceph-mon[32753]: pgmap v5230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8681.818437 secs 2024-04-26T14:15:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8684.586305 secs 2024-04-26T14:15:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8681.818437 secs 2024-04-26T14:15:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8684.586305 secs 2024-04-26T14:15:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8681.818437 secs 2024-04-26T14:15:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8684.586305 secs 2024-04-26T14:15:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:44 smithi194 ceph-mon[33017]: pgmap v5231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:44 smithi033 ceph-mon[27845]: pgmap v5231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:44 smithi096 ceph-mon[32753]: pgmap v5231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:46 smithi194 ceph-mon[33017]: pgmap v5232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:46 smithi033 ceph-mon[27845]: pgmap v5232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:46 smithi096 ceph-mon[32753]: pgmap v5232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:48 smithi194 ceph-mon[33017]: pgmap v5233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:49.165 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8686.818522 secs 2024-04-26T14:15:49.166 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8689.586431 secs 2024-04-26T14:15:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:48 smithi033 ceph-mon[27845]: pgmap v5233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8686.818522 secs 2024-04-26T14:15:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8689.586431 secs 2024-04-26T14:15:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:48 smithi096 ceph-mon[32753]: pgmap v5233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8686.818522 secs 2024-04-26T14:15:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8689.586431 secs 2024-04-26T14:15:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:50 smithi194 ceph-mon[33017]: pgmap v5234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:50 smithi033 ceph-mon[27845]: pgmap v5234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:50 smithi096 ceph-mon[32753]: pgmap v5234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:52.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:52 smithi033 ceph-mon[27845]: pgmap v5235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:52 smithi194 ceph-mon[33017]: pgmap v5235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:52 smithi096 ceph-mon[32753]: pgmap v5235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:54.078+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:54.079+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:15:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:15:54.079+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:15:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8691.818605 secs 2024-04-26T14:15:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8694.586559 secs 2024-04-26T14:15:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8691.818605 secs 2024-04-26T14:15:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8694.586559 secs 2024-04-26T14:15:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8691.818605 secs 2024-04-26T14:15:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8694.586559 secs 2024-04-26T14:15:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:54 smithi194 ceph-mon[33017]: pgmap v5236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:54 smithi033 ceph-mon[27845]: pgmap v5236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:54 smithi096 ceph-mon[32753]: pgmap v5236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:56 smithi194 ceph-mon[33017]: pgmap v5237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:56 smithi033 ceph-mon[27845]: pgmap v5237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:56 smithi096 ceph-mon[32753]: pgmap v5237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:15:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:58 smithi194 ceph-mon[33017]: pgmap v5238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8696.818687 secs 2024-04-26T14:15:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:15:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8699.586653 secs 2024-04-26T14:15:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:58 smithi033 ceph-mon[27845]: pgmap v5238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8696.818687 secs 2024-04-26T14:15:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:15:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8699.586653 secs 2024-04-26T14:15:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:58 smithi096 ceph-mon[32753]: pgmap v5238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:15:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8696.818687 secs 2024-04-26T14:15:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:15:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8699.586653 secs 2024-04-26T14:16:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:16:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:16:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:15:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:16:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:16:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:16:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:15:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:16:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:16:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:16:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:15:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:16:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:00 smithi194 ceph-mon[33017]: pgmap v5239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:00 smithi194 ceph-mon[33017]: pgmap v5240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T14:16:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:00 smithi033 ceph-mon[27845]: pgmap v5239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:00 smithi033 ceph-mon[27845]: pgmap v5240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T14:16:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:00 smithi096 ceph-mon[32753]: pgmap v5239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:00 smithi096 ceph-mon[32753]: pgmap v5240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T14:16:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:02 smithi194 ceph-mon[33017]: pgmap v5241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:02 smithi033 ceph-mon[27845]: pgmap v5241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:02 smithi096 ceph-mon[32753]: pgmap v5241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8701.818773 secs 2024-04-26T14:16:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8704.586780 secs 2024-04-26T14:16:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8701.818773 secs 2024-04-26T14:16:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8704.586780 secs 2024-04-26T14:16:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8701.818773 secs 2024-04-26T14:16:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8704.586780 secs 2024-04-26T14:16:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:04 smithi194 ceph-mon[33017]: pgmap v5242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:04 smithi033 ceph-mon[27845]: pgmap v5242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:04 smithi096 ceph-mon[32753]: pgmap v5242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:06 smithi194 ceph-mon[33017]: pgmap v5243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:06 smithi033 ceph-mon[27845]: pgmap v5243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:06 smithi096 ceph-mon[32753]: pgmap v5243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:09.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:09.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:16:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:09.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:08 smithi194 ceph-mon[33017]: pgmap v5244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8706.818858 secs 2024-04-26T14:16:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8709.586910 secs 2024-04-26T14:16:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:08 smithi033 ceph-mon[27845]: pgmap v5244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8706.818858 secs 2024-04-26T14:16:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8709.586910 secs 2024-04-26T14:16:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:08 smithi096 ceph-mon[32753]: pgmap v5244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:16:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8706.818858 secs 2024-04-26T14:16:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8709.586910 secs 2024-04-26T14:16:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:10 smithi194 ceph-mon[33017]: pgmap v5245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:16:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:10 smithi033 ceph-mon[27845]: pgmap v5245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:16:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:10 smithi096 ceph-mon[32753]: pgmap v5245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:16:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:12 smithi194 ceph-mon[33017]: pgmap v5246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:12 smithi033 ceph-mon[27845]: pgmap v5246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:12 smithi096 ceph-mon[32753]: pgmap v5246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8711.818917 secs 2024-04-26T14:16:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8714.587033 secs 2024-04-26T14:16:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8711.818917 secs 2024-04-26T14:16:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8714.587033 secs 2024-04-26T14:16:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8711.818917 secs 2024-04-26T14:16:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8714.587033 secs 2024-04-26T14:16:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:14 smithi194 ceph-mon[33017]: pgmap v5247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:14 smithi033 ceph-mon[27845]: pgmap v5247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:14 smithi096 ceph-mon[32753]: pgmap v5247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:16 smithi194 ceph-mon[33017]: pgmap v5248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:16 smithi033 ceph-mon[27845]: pgmap v5248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:16 smithi096 ceph-mon[32753]: pgmap v5248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:18 smithi194 ceph-mon[33017]: pgmap v5249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8716.818978 secs 2024-04-26T14:16:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8719.587094 secs 2024-04-26T14:16:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:18 smithi033 ceph-mon[27845]: pgmap v5249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8716.818978 secs 2024-04-26T14:16:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8719.587094 secs 2024-04-26T14:16:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:18 smithi096 ceph-mon[32753]: pgmap v5249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8716.818978 secs 2024-04-26T14:16:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8719.587094 secs 2024-04-26T14:16:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:20 smithi194 ceph-mon[33017]: pgmap v5250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:20 smithi033 ceph-mon[27845]: pgmap v5250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:20 smithi096 ceph-mon[32753]: pgmap v5250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:22 smithi194 ceph-mon[33017]: pgmap v5251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:22 smithi033 ceph-mon[27845]: pgmap v5251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:22 smithi096 ceph-mon[32753]: pgmap v5251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:24.102+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:24.102+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:16:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:24.102+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8721.819040 secs 2024-04-26T14:16:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8724.587214 secs 2024-04-26T14:16:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8721.819040 secs 2024-04-26T14:16:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8724.587214 secs 2024-04-26T14:16:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8721.819040 secs 2024-04-26T14:16:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8724.587214 secs 2024-04-26T14:16:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:24 smithi033 ceph-mon[27845]: pgmap v5252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:24 smithi096 ceph-mon[32753]: pgmap v5252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:24 smithi194 ceph-mon[33017]: pgmap v5252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:26 smithi033 ceph-mon[27845]: pgmap v5253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:26 smithi096 ceph-mon[32753]: pgmap v5253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:26 smithi194 ceph-mon[33017]: pgmap v5253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:29.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:28 smithi194 ceph-mon[33017]: pgmap v5254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:29.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8726.819124 secs 2024-04-26T14:16:29.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8729.587335 secs 2024-04-26T14:16:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:28 smithi033 ceph-mon[27845]: pgmap v5254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8726.819124 secs 2024-04-26T14:16:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8729.587335 secs 2024-04-26T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:28 smithi096 ceph-mon[32753]: pgmap v5254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8726.819124 secs 2024-04-26T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8729.587335 secs 2024-04-26T14:16:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:30 smithi033 ceph-mon[27845]: pgmap v5255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:30 smithi096 ceph-mon[32753]: pgmap v5255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:30 smithi194 ceph-mon[33017]: pgmap v5255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:32 smithi033 ceph-mon[27845]: pgmap v5256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:32 smithi096 ceph-mon[32753]: pgmap v5256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:32 smithi194 ceph-mon[33017]: pgmap v5256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8731.819181 secs 2024-04-26T14:16:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8734.587480 secs 2024-04-26T14:16:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8731.819181 secs 2024-04-26T14:16:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8734.587480 secs 2024-04-26T14:16:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8731.819181 secs 2024-04-26T14:16:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8734.587480 secs 2024-04-26T14:16:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:34 smithi033 ceph-mon[27845]: pgmap v5257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:34 smithi096 ceph-mon[32753]: pgmap v5257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:34 smithi194 ceph-mon[33017]: pgmap v5257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:36 smithi033 ceph-mon[27845]: pgmap v5258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:36 smithi096 ceph-mon[32753]: pgmap v5258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:36 smithi194 ceph-mon[33017]: pgmap v5258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:39.114+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:39.114+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:39.114+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:16:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:38 smithi033 ceph-mon[27845]: pgmap v5259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8736.819259 secs 2024-04-26T14:16:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8739.587591 secs 2024-04-26T14:16:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:38 smithi096 ceph-mon[32753]: pgmap v5259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8736.819259 secs 2024-04-26T14:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8739.587591 secs 2024-04-26T14:16:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:38 smithi194 ceph-mon[33017]: pgmap v5259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8736.819259 secs 2024-04-26T14:16:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8739.587591 secs 2024-04-26T14:16:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:40 smithi033 ceph-mon[27845]: pgmap v5260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:40 smithi096 ceph-mon[32753]: pgmap v5260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:40 smithi194 ceph-mon[33017]: pgmap v5260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:42 smithi033 ceph-mon[27845]: pgmap v5261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:42 smithi096 ceph-mon[32753]: pgmap v5261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:42 smithi194 ceph-mon[33017]: pgmap v5261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8741.819341 secs 2024-04-26T14:16:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8744.587715 secs 2024-04-26T14:16:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8741.819341 secs 2024-04-26T14:16:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8744.587715 secs 2024-04-26T14:16:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8741.819341 secs 2024-04-26T14:16:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8744.587715 secs 2024-04-26T14:16:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:44 smithi033 ceph-mon[27845]: pgmap v5262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:44 smithi096 ceph-mon[32753]: pgmap v5262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:44 smithi194 ceph-mon[33017]: pgmap v5262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:45 smithi194 ceph-mon[33017]: pgmap v5263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:45 smithi033 ceph-mon[27845]: pgmap v5263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:45 smithi096 ceph-mon[32753]: pgmap v5263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:48 smithi194 ceph-mon[33017]: pgmap v5264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:48 smithi033 ceph-mon[27845]: pgmap v5264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:48 smithi096 ceph-mon[32753]: pgmap v5264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8746.819423 secs 2024-04-26T14:16:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8749.587846 secs 2024-04-26T14:16:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8746.819423 secs 2024-04-26T14:16:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8749.587846 secs 2024-04-26T14:16:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8746.819423 secs 2024-04-26T14:16:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8749.587846 secs 2024-04-26T14:16:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:50 smithi194 ceph-mon[33017]: pgmap v5265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:50 smithi033 ceph-mon[27845]: pgmap v5265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:50 smithi096 ceph-mon[32753]: pgmap v5265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:52 smithi194 ceph-mon[33017]: pgmap v5266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:52 smithi033 ceph-mon[27845]: pgmap v5266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:52 smithi096 ceph-mon[32753]: pgmap v5266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:54.131+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:54.131+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:16:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:16:54.131+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:16:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8751.819507 secs 2024-04-26T14:16:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8754.587960 secs 2024-04-26T14:16:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:54 smithi194 ceph-mon[33017]: pgmap v5267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8751.819507 secs 2024-04-26T14:16:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8754.587960 secs 2024-04-26T14:16:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:54 smithi033 ceph-mon[27845]: pgmap v5267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8751.819507 secs 2024-04-26T14:16:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8754.587960 secs 2024-04-26T14:16:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:54 smithi096 ceph-mon[32753]: pgmap v5267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:56 smithi194 ceph-mon[33017]: pgmap v5268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:56 smithi033 ceph-mon[27845]: pgmap v5268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:56 smithi096 ceph-mon[32753]: pgmap v5268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:16:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:58 smithi194 ceph-mon[33017]: pgmap v5269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:58 smithi033 ceph-mon[27845]: pgmap v5269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:58 smithi096 ceph-mon[32753]: pgmap v5269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:16:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8756.819566 secs 2024-04-26T14:16:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8759.588078 secs 2024-04-26T14:16:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8756.819566 secs 2024-04-26T14:16:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8759.588078 secs 2024-04-26T14:16:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8756.819566 secs 2024-04-26T14:16:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8759.588078 secs 2024-04-26T14:17:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:00 smithi194 ceph-mon[33017]: pgmap v5270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:17:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:00 smithi033 ceph-mon[27845]: pgmap v5270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:17:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:00 smithi096 ceph-mon[32753]: pgmap v5270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:17:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:02 smithi194 ceph-mon[33017]: pgmap v5271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:02 smithi033 ceph-mon[27845]: pgmap v5271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:02 smithi096 ceph-mon[32753]: pgmap v5271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8761.819662 secs 2024-04-26T14:17:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:17:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:17:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:17:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:17:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8764.588193 secs 2024-04-26T14:17:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: pgmap v5272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:04.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:04.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8761.819662 secs 2024-04-26T14:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8764.588193 secs 2024-04-26T14:17:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: pgmap v5272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8761.819662 secs 2024-04-26T14:17:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:17:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:17:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:17:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:17:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8764.588193 secs 2024-04-26T14:17:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: pgmap v5272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: pgmap v5273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:17:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:17:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:17:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: pgmap v5274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:17:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: pgmap v5275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: pgmap v5273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:17:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:17:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:17:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:17:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: pgmap v5274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:17:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: pgmap v5275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: pgmap v5273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:17:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:17:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: pgmap v5274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:17:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: pgmap v5275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:17:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8766.819744 secs 2024-04-26T14:17:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:08 smithi194 ceph-mon[33017]: pgmap v5276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8769.588290 secs 2024-04-26T14:17:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:09.134+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:09.134+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:09.134+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:17:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8766.819744 secs 2024-04-26T14:17:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:08 smithi033 ceph-mon[27845]: pgmap v5276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8769.588290 secs 2024-04-26T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8766.819744 secs 2024-04-26T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:08 smithi096 ceph-mon[32753]: pgmap v5276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8769.588290 secs 2024-04-26T14:17:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:10 smithi194 ceph-mon[33017]: pgmap v5277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:10 smithi033 ceph-mon[27845]: pgmap v5277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:10 smithi096 ceph-mon[32753]: pgmap v5277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:12 smithi194 ceph-mon[33017]: pgmap v5278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:12 smithi033 ceph-mon[27845]: pgmap v5278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:12 smithi096 ceph-mon[32753]: pgmap v5278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:17:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8771.819825 secs 2024-04-26T14:17:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8774.588418 secs 2024-04-26T14:17:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8771.819825 secs 2024-04-26T14:17:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8774.588418 secs 2024-04-26T14:17:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8771.819825 secs 2024-04-26T14:17:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8774.588418 secs 2024-04-26T14:17:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:14 smithi194 ceph-mon[33017]: pgmap v5279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:17:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:14 smithi033 ceph-mon[27845]: pgmap v5279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:17:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:14 smithi096 ceph-mon[32753]: pgmap v5279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:17:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:16 smithi194 ceph-mon[33017]: pgmap v5280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:17:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:16 smithi033 ceph-mon[27845]: pgmap v5280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:17:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:16 smithi096 ceph-mon[32753]: pgmap v5280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:17:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8776.819909 secs 2024-04-26T14:17:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:18 smithi194 ceph-mon[33017]: pgmap v5281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8779.588536 secs 2024-04-26T14:17:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8776.819909 secs 2024-04-26T14:17:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:18 smithi033 ceph-mon[27845]: pgmap v5281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8779.588536 secs 2024-04-26T14:17:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8776.819909 secs 2024-04-26T14:17:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:18 smithi096 ceph-mon[32753]: pgmap v5281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8779.588536 secs 2024-04-26T14:17:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:20 smithi194 ceph-mon[33017]: pgmap v5282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:20 smithi033 ceph-mon[27845]: pgmap v5282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:20 smithi096 ceph-mon[32753]: pgmap v5282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:22 smithi194 ceph-mon[33017]: pgmap v5283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:22 smithi033 ceph-mon[27845]: pgmap v5283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:22 smithi096 ceph-mon[32753]: pgmap v5283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:23.284+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:17:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:23.284+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:17:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:23.285+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:17:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:23.285+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:17:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8781.820001 secs 2024-04-26T14:17:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8784.588667 secs 2024-04-26T14:17:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:24.145+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:24.145+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:24.145+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:17:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8781.820001 secs 2024-04-26T14:17:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8784.588667 secs 2024-04-26T14:17:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8781.820001 secs 2024-04-26T14:17:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8784.588667 secs 2024-04-26T14:17:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:24 smithi194 ceph-mon[33017]: pgmap v5284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:24 smithi033 ceph-mon[27845]: pgmap v5284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:24 smithi096 ceph-mon[32753]: pgmap v5284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:26 smithi194 ceph-mon[33017]: pgmap v5285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:26 smithi033 ceph-mon[27845]: pgmap v5285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:26 smithi096 ceph-mon[32753]: pgmap v5285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8786.820087 secs 2024-04-26T14:17:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:28 smithi194 ceph-mon[33017]: pgmap v5286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8789.588797 secs 2024-04-26T14:17:29.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8786.820087 secs 2024-04-26T14:17:29.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:28 smithi033 ceph-mon[27845]: pgmap v5286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:29.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8789.588797 secs 2024-04-26T14:17:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8786.820087 secs 2024-04-26T14:17:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:28 smithi096 ceph-mon[32753]: pgmap v5286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8789.588797 secs 2024-04-26T14:17:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:30 smithi194 ceph-mon[33017]: pgmap v5287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:30 smithi033 ceph-mon[27845]: pgmap v5287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:30 smithi096 ceph-mon[32753]: pgmap v5287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:32 smithi194 ceph-mon[33017]: pgmap v5288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:32 smithi033 ceph-mon[27845]: pgmap v5288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:32 smithi096 ceph-mon[32753]: pgmap v5288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8791.820175 secs 2024-04-26T14:17:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8794.588930 secs 2024-04-26T14:17:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8791.820175 secs 2024-04-26T14:17:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8794.588930 secs 2024-04-26T14:17:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8791.820175 secs 2024-04-26T14:17:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8794.588930 secs 2024-04-26T14:17:35.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:34 smithi033 ceph-mon[27845]: pgmap v5289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:34 smithi194 ceph-mon[33017]: pgmap v5289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:34 smithi096 ceph-mon[32753]: pgmap v5289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:36 smithi194 ceph-mon[33017]: pgmap v5290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:36 smithi033 ceph-mon[27845]: pgmap v5290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:36 smithi096 ceph-mon[32753]: pgmap v5290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8796.820236 secs 2024-04-26T14:17:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:38 smithi194 ceph-mon[33017]: pgmap v5291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8799.589050 secs 2024-04-26T14:17:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:39.160+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:39.160+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:39.160+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:17:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8796.820236 secs 2024-04-26T14:17:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:38 smithi033 ceph-mon[27845]: pgmap v5291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8799.589050 secs 2024-04-26T14:17:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8796.820236 secs 2024-04-26T14:17:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:38 smithi096 ceph-mon[32753]: pgmap v5291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8799.589050 secs 2024-04-26T14:17:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:40 smithi033 ceph-mon[27845]: pgmap v5292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:40 smithi096 ceph-mon[32753]: pgmap v5292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:40 smithi194 ceph-mon[33017]: pgmap v5292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:42 smithi033 ceph-mon[27845]: pgmap v5293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:42 smithi096 ceph-mon[32753]: pgmap v5293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:42 smithi194 ceph-mon[33017]: pgmap v5293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8801.820322 secs 2024-04-26T14:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8804.589152 secs 2024-04-26T14:17:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8801.820322 secs 2024-04-26T14:17:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8804.589152 secs 2024-04-26T14:17:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8801.820322 secs 2024-04-26T14:17:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8804.589152 secs 2024-04-26T14:17:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:44 smithi033 ceph-mon[27845]: pgmap v5294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:44 smithi096 ceph-mon[32753]: pgmap v5294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:44 smithi194 ceph-mon[33017]: pgmap v5294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:46 smithi033 ceph-mon[27845]: pgmap v5295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:46 smithi096 ceph-mon[32753]: pgmap v5295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:46 smithi194 ceph-mon[33017]: pgmap v5295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8806.820411 secs 2024-04-26T14:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:48 smithi033 ceph-mon[27845]: pgmap v5296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8809.589277 secs 2024-04-26T14:17:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8806.820411 secs 2024-04-26T14:17:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:48 smithi096 ceph-mon[32753]: pgmap v5296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8809.589277 secs 2024-04-26T14:17:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8806.820411 secs 2024-04-26T14:17:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:48 smithi194 ceph-mon[33017]: pgmap v5296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8809.589277 secs 2024-04-26T14:17:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:50 smithi033 ceph-mon[27845]: pgmap v5297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:50 smithi096 ceph-mon[32753]: pgmap v5297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:50 smithi194 ceph-mon[33017]: pgmap v5297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:52 smithi033 ceph-mon[27845]: pgmap v5298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:52 smithi096 ceph-mon[32753]: pgmap v5298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:52 smithi194 ceph-mon[33017]: pgmap v5298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8811.820499 secs 2024-04-26T14:17:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8814.589395 secs 2024-04-26T14:17:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:54.176+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:54.176+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:17:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:17:54.176+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:17:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8811.820499 secs 2024-04-26T14:17:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8814.589395 secs 2024-04-26T14:17:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8811.820499 secs 2024-04-26T14:17:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8814.589395 secs 2024-04-26T14:17:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:54 smithi033 ceph-mon[27845]: pgmap v5299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:54 smithi096 ceph-mon[32753]: pgmap v5299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:54 smithi194 ceph-mon[33017]: pgmap v5299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:56 smithi033 ceph-mon[27845]: pgmap v5300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:56 smithi096 ceph-mon[32753]: pgmap v5300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:56 smithi194 ceph-mon[33017]: pgmap v5300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8816.820584 secs 2024-04-26T14:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:58 smithi033 ceph-mon[27845]: pgmap v5301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:17:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8819.589522 secs 2024-04-26T14:17:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8816.820584 secs 2024-04-26T14:17:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:58 smithi096 ceph-mon[32753]: pgmap v5301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:17:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8819.589522 secs 2024-04-26T14:17:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8816.820584 secs 2024-04-26T14:17:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:58 smithi194 ceph-mon[33017]: pgmap v5301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:17:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:17:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8819.589522 secs 2024-04-26T14:18:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:00 smithi033 ceph-mon[27845]: pgmap v5302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:00 smithi096 ceph-mon[32753]: pgmap v5302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:00 smithi194 ceph-mon[33017]: pgmap v5302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:02 smithi096 ceph-mon[32753]: pgmap v5303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:02 smithi194 ceph-mon[33017]: pgmap v5303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:02 smithi033 ceph-mon[27845]: pgmap v5303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8821.820668 secs 2024-04-26T14:18:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8824.589637 secs 2024-04-26T14:18:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8821.820668 secs 2024-04-26T14:18:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8824.589637 secs 2024-04-26T14:18:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8821.820668 secs 2024-04-26T14:18:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8824.589637 secs 2024-04-26T14:18:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:04 smithi194 ceph-mon[33017]: pgmap v5304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:04 smithi033 ceph-mon[27845]: pgmap v5304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:04 smithi096 ceph-mon[32753]: pgmap v5304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:05 smithi194 ceph-mon[33017]: pgmap v5305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:05 smithi033 ceph-mon[27845]: pgmap v5305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:05 smithi096 ceph-mon[32753]: pgmap v5305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:18:07.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:18:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:08.292+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:08.292+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:08.293+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:08.293+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:18:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:18:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:18:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:07 smithi194 ceph-mon[33017]: pgmap v5306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:07 smithi194 ceph-mon[33017]: pgmap v5307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:18:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:18:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:18:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:07 smithi033 ceph-mon[27845]: pgmap v5306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:07 smithi033 ceph-mon[27845]: pgmap v5307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:18:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:18:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:18:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:07 smithi096 ceph-mon[32753]: pgmap v5306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:07 smithi096 ceph-mon[32753]: pgmap v5307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:18:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:09.192+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:09.192+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:09.192+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:18:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8826.820754 secs 2024-04-26T14:18:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8829.589779 secs 2024-04-26T14:18:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8826.820754 secs 2024-04-26T14:18:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8829.589779 secs 2024-04-26T14:18:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8826.820754 secs 2024-04-26T14:18:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8829.589779 secs 2024-04-26T14:18:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:09 smithi194 ceph-mon[33017]: pgmap v5308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:09 smithi033 ceph-mon[27845]: pgmap v5308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:09 smithi096 ceph-mon[32753]: pgmap v5308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:12 smithi194 ceph-mon[33017]: pgmap v5309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:12 smithi033 ceph-mon[27845]: pgmap v5309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:12 smithi096 ceph-mon[32753]: pgmap v5309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8831.820842 secs 2024-04-26T14:18:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8834.589935 secs 2024-04-26T14:18:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8831.820842 secs 2024-04-26T14:18:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8834.589935 secs 2024-04-26T14:18:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8831.820842 secs 2024-04-26T14:18:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8834.589935 secs 2024-04-26T14:18:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:14 smithi194 ceph-mon[33017]: pgmap v5310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:14 smithi033 ceph-mon[27845]: pgmap v5310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:14 smithi096 ceph-mon[32753]: pgmap v5310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:16 smithi194 ceph-mon[33017]: pgmap v5311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:16 smithi033 ceph-mon[27845]: pgmap v5311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:16 smithi096 ceph-mon[32753]: pgmap v5311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:18:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:18 smithi194 ceph-mon[33017]: pgmap v5312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:18:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8836.820927 secs 2024-04-26T14:18:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8839.590060 secs 2024-04-26T14:18:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:18 smithi033 ceph-mon[27845]: pgmap v5312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:18:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8836.820927 secs 2024-04-26T14:18:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8839.590060 secs 2024-04-26T14:18:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:18 smithi096 ceph-mon[32753]: pgmap v5312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:18:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8836.820927 secs 2024-04-26T14:18:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8839.590060 secs 2024-04-26T14:18:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:20 smithi194 ceph-mon[33017]: pgmap v5313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:20 smithi033 ceph-mon[27845]: pgmap v5313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:20 smithi096 ceph-mon[32753]: pgmap v5313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:22 smithi194 ceph-mon[33017]: pgmap v5314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:22 smithi033 ceph-mon[27845]: pgmap v5314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:22 smithi096 ceph-mon[32753]: pgmap v5314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8841.821007 secs 2024-04-26T14:18:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8844.590196 secs 2024-04-26T14:18:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8841.821007 secs 2024-04-26T14:18:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8844.590196 secs 2024-04-26T14:18:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8841.821007 secs 2024-04-26T14:18:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8844.590196 secs 2024-04-26T14:18:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:24.193+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:24.193+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:24.193+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:18:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:24 smithi194 ceph-mon[33017]: pgmap v5315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:24 smithi033 ceph-mon[27845]: pgmap v5315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:24 smithi096 ceph-mon[32753]: pgmap v5315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:26 smithi194 ceph-mon[33017]: pgmap v5316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:26 smithi033 ceph-mon[27845]: pgmap v5316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:26 smithi096 ceph-mon[32753]: pgmap v5316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:28 smithi194 ceph-mon[33017]: pgmap v5317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8846.821069 secs 2024-04-26T14:18:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8849.590318 secs 2024-04-26T14:18:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:28 smithi033 ceph-mon[27845]: pgmap v5317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8846.821069 secs 2024-04-26T14:18:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8849.590318 secs 2024-04-26T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:28 smithi096 ceph-mon[32753]: pgmap v5317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8846.821069 secs 2024-04-26T14:18:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8849.590318 secs 2024-04-26T14:18:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:30 smithi194 ceph-mon[33017]: pgmap v5318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:30 smithi033 ceph-mon[27845]: pgmap v5318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:30 smithi096 ceph-mon[32753]: pgmap v5318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:32 smithi194 ceph-mon[33017]: pgmap v5319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:32 smithi033 ceph-mon[27845]: pgmap v5319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:32 smithi096 ceph-mon[32753]: pgmap v5319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8851.821128 secs 2024-04-26T14:18:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8854.590439 secs 2024-04-26T14:18:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8851.821128 secs 2024-04-26T14:18:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8854.590439 secs 2024-04-26T14:18:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8851.821128 secs 2024-04-26T14:18:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8854.590439 secs 2024-04-26T14:18:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:34 smithi194 ceph-mon[33017]: pgmap v5320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:34 smithi033 ceph-mon[27845]: pgmap v5320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:34 smithi096 ceph-mon[32753]: pgmap v5320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:36 smithi194 ceph-mon[33017]: pgmap v5321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:36 smithi033 ceph-mon[27845]: pgmap v5321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:36 smithi096 ceph-mon[32753]: pgmap v5321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:38 smithi033 ceph-mon[27845]: pgmap v5322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8856.821212 secs 2024-04-26T14:18:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8859.590567 secs 2024-04-26T14:18:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:38 smithi096 ceph-mon[32753]: pgmap v5322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8856.821212 secs 2024-04-26T14:18:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8859.590567 secs 2024-04-26T14:18:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:38 smithi194 ceph-mon[33017]: pgmap v5322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8856.821212 secs 2024-04-26T14:18:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8859.590567 secs 2024-04-26T14:18:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:39.208+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:39.208+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:39.208+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:40 smithi033 ceph-mon[27845]: pgmap v5323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:40 smithi096 ceph-mon[32753]: pgmap v5323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:40 smithi194 ceph-mon[33017]: pgmap v5323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:42 smithi033 ceph-mon[27845]: pgmap v5324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:42 smithi096 ceph-mon[32753]: pgmap v5324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:42 smithi194 ceph-mon[33017]: pgmap v5324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8861.821315 secs 2024-04-26T14:18:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8864.590690 secs 2024-04-26T14:18:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8861.821315 secs 2024-04-26T14:18:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8864.590690 secs 2024-04-26T14:18:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8861.821315 secs 2024-04-26T14:18:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8864.590690 secs 2024-04-26T14:18:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:44 smithi033 ceph-mon[27845]: pgmap v5325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:44 smithi096 ceph-mon[32753]: pgmap v5325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:44 smithi194 ceph-mon[33017]: pgmap v5325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:46 smithi033 ceph-mon[27845]: pgmap v5326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:46 smithi096 ceph-mon[32753]: pgmap v5326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:46 smithi194 ceph-mon[33017]: pgmap v5326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:48 smithi033 ceph-mon[27845]: pgmap v5327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8866.821414 secs 2024-04-26T14:18:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8869.590835 secs 2024-04-26T14:18:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:48 smithi096 ceph-mon[32753]: pgmap v5327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8866.821414 secs 2024-04-26T14:18:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8869.590835 secs 2024-04-26T14:18:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:48 smithi194 ceph-mon[33017]: pgmap v5327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8866.821414 secs 2024-04-26T14:18:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8869.590835 secs 2024-04-26T14:18:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:50 smithi033 ceph-mon[27845]: pgmap v5328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:50 smithi096 ceph-mon[32753]: pgmap v5328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:50 smithi194 ceph-mon[33017]: pgmap v5328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:52 smithi033 ceph-mon[27845]: pgmap v5329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:52 smithi096 ceph-mon[32753]: pgmap v5329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:52 smithi194 ceph-mon[33017]: pgmap v5329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8871.821509 secs 2024-04-26T14:18:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8874.590993 secs 2024-04-26T14:18:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8871.821509 secs 2024-04-26T14:18:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8874.590993 secs 2024-04-26T14:18:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8871.821509 secs 2024-04-26T14:18:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8874.590993 secs 2024-04-26T14:18:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:54.214+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:54.214+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:18:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:18:54.214+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:18:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:54 smithi033 ceph-mon[27845]: pgmap v5330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:54 smithi096 ceph-mon[32753]: pgmap v5330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:54 smithi194 ceph-mon[33017]: pgmap v5330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:56 smithi033 ceph-mon[27845]: pgmap v5331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:56 smithi096 ceph-mon[32753]: pgmap v5331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:56 smithi194 ceph-mon[33017]: pgmap v5331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:18:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:58 smithi033 ceph-mon[27845]: pgmap v5332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8876.821609 secs 2024-04-26T14:18:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:18:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8879.591138 secs 2024-04-26T14:18:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:58 smithi096 ceph-mon[32753]: pgmap v5332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8876.821609 secs 2024-04-26T14:18:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:18:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8879.591138 secs 2024-04-26T14:18:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:58 smithi194 ceph-mon[33017]: pgmap v5332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:18:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8876.821609 secs 2024-04-26T14:18:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:18:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8879.591138 secs 2024-04-26T14:19:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:00 smithi096 ceph-mon[32753]: pgmap v5333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:00 smithi194 ceph-mon[33017]: pgmap v5333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:00 smithi033 ceph-mon[27845]: pgmap v5333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:02 smithi194 ceph-mon[33017]: pgmap v5334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:02 smithi033 ceph-mon[27845]: pgmap v5334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:02 smithi096 ceph-mon[32753]: pgmap v5334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8881.821711 secs 2024-04-26T14:19:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8884.591262 secs 2024-04-26T14:19:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8881.821711 secs 2024-04-26T14:19:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8884.591262 secs 2024-04-26T14:19:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8881.821711 secs 2024-04-26T14:19:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8884.591262 secs 2024-04-26T14:19:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:04 smithi194 ceph-mon[33017]: pgmap v5335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:04 smithi033 ceph-mon[27845]: pgmap v5335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:04 smithi096 ceph-mon[32753]: pgmap v5335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:06 smithi194 ceph-mon[33017]: pgmap v5336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:06 smithi033 ceph-mon[27845]: pgmap v5336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:06 smithi096 ceph-mon[32753]: pgmap v5336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:08 smithi194 ceph-mon[33017]: pgmap v5337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8886.821814 secs 2024-04-26T14:19:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:19:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8889.591351 secs 2024-04-26T14:19:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:08 smithi033 ceph-mon[27845]: pgmap v5337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8886.821814 secs 2024-04-26T14:19:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:19:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8889.591351 secs 2024-04-26T14:19:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:08 smithi096 ceph-mon[32753]: pgmap v5337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8886.821814 secs 2024-04-26T14:19:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:19:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8889.591351 secs 2024-04-26T14:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:09.229+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:09.229+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:09.229+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:19:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:19:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:19:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:19:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:19:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:19:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:19:10.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:19:10.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:19:10.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:19:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:10 smithi194 ceph-mon[33017]: pgmap v5338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:10 smithi194 ceph-mon[33017]: pgmap v5339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:19:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:10 smithi033 ceph-mon[27845]: pgmap v5338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:10 smithi033 ceph-mon[27845]: pgmap v5339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:19:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:10 smithi096 ceph-mon[32753]: pgmap v5338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:10 smithi096 ceph-mon[32753]: pgmap v5339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:19:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:12 smithi194 ceph-mon[33017]: pgmap v5340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:12 smithi033 ceph-mon[27845]: pgmap v5340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:12 smithi096 ceph-mon[32753]: pgmap v5340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8891.821908 secs 2024-04-26T14:19:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8894.591493 secs 2024-04-26T14:19:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8891.821908 secs 2024-04-26T14:19:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8894.591493 secs 2024-04-26T14:19:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8891.821908 secs 2024-04-26T14:19:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8894.591493 secs 2024-04-26T14:19:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:14 smithi194 ceph-mon[33017]: pgmap v5341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:14 smithi033 ceph-mon[27845]: pgmap v5341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:15.214 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:14 smithi096 ceph-mon[32753]: pgmap v5341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:16 smithi194 ceph-mon[33017]: pgmap v5342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:16 smithi033 ceph-mon[27845]: pgmap v5342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:16 smithi096 ceph-mon[32753]: pgmap v5342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:18 smithi194 ceph-mon[33017]: pgmap v5343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8896.821994 secs 2024-04-26T14:19:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8899.591617 secs 2024-04-26T14:19:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:18 smithi033 ceph-mon[27845]: pgmap v5343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8896.821994 secs 2024-04-26T14:19:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8899.591617 secs 2024-04-26T14:19:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:18 smithi096 ceph-mon[32753]: pgmap v5343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:19:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8896.821994 secs 2024-04-26T14:19:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8899.591617 secs 2024-04-26T14:19:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:20 smithi194 ceph-mon[33017]: pgmap v5344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:19:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:20 smithi033 ceph-mon[27845]: pgmap v5344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:19:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:20 smithi096 ceph-mon[32753]: pgmap v5344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:19:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:22 smithi194 ceph-mon[33017]: pgmap v5345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:22 smithi033 ceph-mon[27845]: pgmap v5345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:22 smithi096 ceph-mon[32753]: pgmap v5345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8901.822091 secs 2024-04-26T14:19:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8904.591682 secs 2024-04-26T14:19:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8901.822091 secs 2024-04-26T14:19:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8904.591682 secs 2024-04-26T14:19:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8901.822091 secs 2024-04-26T14:19:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8904.591682 secs 2024-04-26T14:19:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:24.244+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:24.244+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:24.244+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:19:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:24 smithi194 ceph-mon[33017]: pgmap v5346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:24 smithi033 ceph-mon[27845]: pgmap v5346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:24 smithi096 ceph-mon[32753]: pgmap v5346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:26 smithi194 ceph-mon[33017]: pgmap v5347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:26 smithi033 ceph-mon[27845]: pgmap v5347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:26 smithi096 ceph-mon[32753]: pgmap v5347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:28 smithi194 ceph-mon[33017]: pgmap v5348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8906.822151 secs 2024-04-26T14:19:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8909.591746 secs 2024-04-26T14:19:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:28 smithi033 ceph-mon[27845]: pgmap v5348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8906.822151 secs 2024-04-26T14:19:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8909.591746 secs 2024-04-26T14:19:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:28 smithi096 ceph-mon[32753]: pgmap v5348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8906.822151 secs 2024-04-26T14:19:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8909.591746 secs 2024-04-26T14:19:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:30 smithi194 ceph-mon[33017]: pgmap v5349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:30 smithi033 ceph-mon[27845]: pgmap v5349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:30 smithi096 ceph-mon[32753]: pgmap v5349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:32 smithi194 ceph-mon[33017]: pgmap v5350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:32 smithi033 ceph-mon[27845]: pgmap v5350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:32 smithi096 ceph-mon[32753]: pgmap v5350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:34.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8911.822256 secs 2024-04-26T14:19:34.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8914.591889 secs 2024-04-26T14:19:34.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8911.822256 secs 2024-04-26T14:19:34.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8914.591889 secs 2024-04-26T14:19:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8911.822256 secs 2024-04-26T14:19:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8914.591889 secs 2024-04-26T14:19:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:34 smithi194 ceph-mon[33017]: pgmap v5351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:34 smithi033 ceph-mon[27845]: pgmap v5351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:34 smithi096 ceph-mon[32753]: pgmap v5351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:36 smithi194 ceph-mon[33017]: pgmap v5352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:36 smithi033 ceph-mon[27845]: pgmap v5352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:36 smithi096 ceph-mon[32753]: pgmap v5352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:38 smithi194 ceph-mon[33017]: pgmap v5353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8916.822323 secs 2024-04-26T14:19:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8919.592014 secs 2024-04-26T14:19:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:38 smithi033 ceph-mon[27845]: pgmap v5353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8916.822323 secs 2024-04-26T14:19:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8919.592014 secs 2024-04-26T14:19:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:38 smithi096 ceph-mon[32753]: pgmap v5353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8916.822323 secs 2024-04-26T14:19:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8919.592014 secs 2024-04-26T14:19:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:39.249+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:39.249+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:39.249+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:19:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:40 smithi194 ceph-mon[33017]: pgmap v5354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:40 smithi033 ceph-mon[27845]: pgmap v5354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:40 smithi096 ceph-mon[32753]: pgmap v5354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:42 smithi194 ceph-mon[33017]: pgmap v5355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:42 smithi033 ceph-mon[27845]: pgmap v5355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:42 smithi096 ceph-mon[32753]: pgmap v5355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8921.822407 secs 2024-04-26T14:19:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8924.592158 secs 2024-04-26T14:19:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8921.822407 secs 2024-04-26T14:19:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8924.592158 secs 2024-04-26T14:19:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8921.822407 secs 2024-04-26T14:19:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8924.592158 secs 2024-04-26T14:19:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:44 smithi194 ceph-mon[33017]: pgmap v5356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:44 smithi033 ceph-mon[27845]: pgmap v5356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:44 smithi096 ceph-mon[32753]: pgmap v5356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:46 smithi194 ceph-mon[33017]: pgmap v5357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:46 smithi033 ceph-mon[27845]: pgmap v5357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:46 smithi096 ceph-mon[32753]: pgmap v5357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:48 smithi194 ceph-mon[33017]: pgmap v5358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8926.822490 secs 2024-04-26T14:19:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8929.592269 secs 2024-04-26T14:19:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:48 smithi033 ceph-mon[27845]: pgmap v5358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8926.822490 secs 2024-04-26T14:19:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8929.592269 secs 2024-04-26T14:19:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:48 smithi096 ceph-mon[32753]: pgmap v5358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8926.822490 secs 2024-04-26T14:19:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8929.592269 secs 2024-04-26T14:19:51.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:50 smithi033 ceph-mon[27845]: pgmap v5359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:50 smithi096 ceph-mon[32753]: pgmap v5359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:50 smithi194 ceph-mon[33017]: pgmap v5359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:52 smithi033 ceph-mon[27845]: pgmap v5360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:52 smithi096 ceph-mon[32753]: pgmap v5360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:52 smithi194 ceph-mon[33017]: pgmap v5360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8931.822574 secs 2024-04-26T14:19:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8934.592393 secs 2024-04-26T14:19:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8931.822574 secs 2024-04-26T14:19:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8934.592393 secs 2024-04-26T14:19:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:54.265+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:54.266+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:19:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:19:54.266+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:19:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8931.822574 secs 2024-04-26T14:19:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8934.592393 secs 2024-04-26T14:19:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:54 smithi033 ceph-mon[27845]: pgmap v5361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:54 smithi096 ceph-mon[32753]: pgmap v5361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:54 smithi194 ceph-mon[33017]: pgmap v5361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:56 smithi033 ceph-mon[27845]: pgmap v5362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:56 smithi096 ceph-mon[32753]: pgmap v5362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:56 smithi194 ceph-mon[33017]: pgmap v5362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:19:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:58 smithi033 ceph-mon[27845]: pgmap v5363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8936.822657 secs 2024-04-26T14:19:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:19:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8939.592510 secs 2024-04-26T14:19:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:58 smithi096 ceph-mon[32753]: pgmap v5363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8936.822657 secs 2024-04-26T14:19:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:19:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8939.592510 secs 2024-04-26T14:19:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:58 smithi194 ceph-mon[33017]: pgmap v5363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:19:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8936.822657 secs 2024-04-26T14:19:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:19:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8939.592510 secs 2024-04-26T14:20:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:00 smithi033 ceph-mon[27845]: pgmap v5364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:00 smithi096 ceph-mon[32753]: pgmap v5364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:20:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:00 smithi194 ceph-mon[33017]: pgmap v5364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:20:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:02 smithi033 ceph-mon[27845]: pgmap v5365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:02 smithi096 ceph-mon[32753]: pgmap v5365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:02 smithi194 ceph-mon[33017]: pgmap v5365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8941.822740 secs 2024-04-26T14:20:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8944.592588 secs 2024-04-26T14:20:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8941.822740 secs 2024-04-26T14:20:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8944.592588 secs 2024-04-26T14:20:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8941.822740 secs 2024-04-26T14:20:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8944.592588 secs 2024-04-26T14:20:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:04 smithi033 ceph-mon[27845]: pgmap v5366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:04 smithi096 ceph-mon[32753]: pgmap v5366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:04 smithi194 ceph-mon[33017]: pgmap v5366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:06 smithi096 ceph-mon[32753]: pgmap v5367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:06 smithi194 ceph-mon[33017]: pgmap v5367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:06 smithi033 ceph-mon[27845]: pgmap v5367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:09.281+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:09.281+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:09.281+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:20:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:08 smithi194 ceph-mon[33017]: pgmap v5368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8946.822828 secs 2024-04-26T14:20:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8949.592705 secs 2024-04-26T14:20:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:08 smithi033 ceph-mon[27845]: pgmap v5368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8946.822828 secs 2024-04-26T14:20:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8949.592705 secs 2024-04-26T14:20:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:08 smithi096 ceph-mon[32753]: pgmap v5368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8946.822828 secs 2024-04-26T14:20:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8949.592705 secs 2024-04-26T14:20:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:20:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:20:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:20:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:10 smithi194 ceph-mon[33017]: pgmap v5369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:20:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:20:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:20:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:10 smithi033 ceph-mon[27845]: pgmap v5369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:20:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:20:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:20:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:10 smithi096 ceph-mon[32753]: pgmap v5369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:20:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:20:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:20:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:11 smithi194 ceph-mon[33017]: pgmap v5370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:20:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:11 smithi194 ceph-mon[33017]: pgmap v5371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:11 smithi033 ceph-mon[27845]: pgmap v5370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:20:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:11 smithi033 ceph-mon[27845]: pgmap v5371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:11 smithi096 ceph-mon[32753]: pgmap v5370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:20:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:11 smithi096 ceph-mon[32753]: pgmap v5371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:13 smithi194 ceph-mon[33017]: pgmap v5372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:13 smithi033 ceph-mon[27845]: pgmap v5372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:13 smithi096 ceph-mon[32753]: pgmap v5372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8951.822909 secs 2024-04-26T14:20:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8954.592825 secs 2024-04-26T14:20:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8951.822909 secs 2024-04-26T14:20:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8954.592825 secs 2024-04-26T14:20:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8951.822909 secs 2024-04-26T14:20:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8954.592825 secs 2024-04-26T14:20:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:15 smithi194 ceph-mon[33017]: pgmap v5373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:15 smithi033 ceph-mon[27845]: pgmap v5373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:15 smithi096 ceph-mon[32753]: pgmap v5373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:17 smithi194 ceph-mon[33017]: pgmap v5374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:17 smithi033 ceph-mon[27845]: pgmap v5374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:17 smithi096 ceph-mon[32753]: pgmap v5374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8956.823010 secs 2024-04-26T14:20:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8959.592950 secs 2024-04-26T14:20:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8956.823010 secs 2024-04-26T14:20:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8959.592950 secs 2024-04-26T14:20:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8956.823010 secs 2024-04-26T14:20:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8959.592950 secs 2024-04-26T14:20:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:19 smithi194 ceph-mon[33017]: pgmap v5375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:20.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:19 smithi033 ceph-mon[27845]: pgmap v5375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:19 smithi096 ceph-mon[32753]: pgmap v5375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:20:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:21 smithi194 ceph-mon[33017]: pgmap v5376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:20:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:21 smithi033 ceph-mon[27845]: pgmap v5376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:20:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:21 smithi096 ceph-mon[32753]: pgmap v5376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:20:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:23.315+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:20:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:23.315+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:20:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:23.315+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T14:20:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:23.315+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T14:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8961.823102 secs 2024-04-26T14:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:23 smithi194 ceph-mon[33017]: pgmap v5377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8964.593043 secs 2024-04-26T14:20:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8961.823102 secs 2024-04-26T14:20:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:23 smithi033 ceph-mon[27845]: pgmap v5377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8964.593043 secs 2024-04-26T14:20:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8961.823102 secs 2024-04-26T14:20:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:23 smithi096 ceph-mon[32753]: pgmap v5377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8964.593043 secs 2024-04-26T14:20:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:24.292+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:24.292+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:24.292+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:20:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:25 smithi194 ceph-mon[33017]: pgmap v5378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:25 smithi033 ceph-mon[27845]: pgmap v5378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:25 smithi096 ceph-mon[32753]: pgmap v5378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:27 smithi194 ceph-mon[33017]: pgmap v5379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:27 smithi033 ceph-mon[27845]: pgmap v5379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:27 smithi096 ceph-mon[32753]: pgmap v5379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8966.823161 secs 2024-04-26T14:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8969.593164 secs 2024-04-26T14:20:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8966.823161 secs 2024-04-26T14:20:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8969.593164 secs 2024-04-26T14:20:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8966.823161 secs 2024-04-26T14:20:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8969.593164 secs 2024-04-26T14:20:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:29 smithi194 ceph-mon[33017]: pgmap v5380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:29 smithi033 ceph-mon[27845]: pgmap v5380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:29 smithi096 ceph-mon[32753]: pgmap v5380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:31 smithi194 ceph-mon[33017]: pgmap v5381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:31 smithi033 ceph-mon[27845]: pgmap v5381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:31 smithi096 ceph-mon[32753]: pgmap v5381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8971.823224 secs 2024-04-26T14:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:33 smithi194 ceph-mon[33017]: pgmap v5382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8974.593262 secs 2024-04-26T14:20:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8971.823224 secs 2024-04-26T14:20:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:33 smithi033 ceph-mon[27845]: pgmap v5382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8974.593262 secs 2024-04-26T14:20:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8971.823224 secs 2024-04-26T14:20:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:33 smithi096 ceph-mon[32753]: pgmap v5382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:20:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8974.593262 secs 2024-04-26T14:20:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:35 smithi033 ceph-mon[27845]: pgmap v5383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:35 smithi096 ceph-mon[32753]: pgmap v5383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:35 smithi194 ceph-mon[33017]: pgmap v5383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:20:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:37 smithi033 ceph-mon[27845]: pgmap v5384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:37 smithi096 ceph-mon[32753]: pgmap v5384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:37 smithi194 ceph-mon[33017]: pgmap v5384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8976.823306 secs 2024-04-26T14:20:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8979.593360 secs 2024-04-26T14:20:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8976.823306 secs 2024-04-26T14:20:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8979.593360 secs 2024-04-26T14:20:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:39.305+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:39.305+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:39.305+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:20:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8976.823306 secs 2024-04-26T14:20:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8979.593360 secs 2024-04-26T14:20:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:39 smithi033 ceph-mon[27845]: pgmap v5385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:39 smithi096 ceph-mon[32753]: pgmap v5385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:39 smithi194 ceph-mon[33017]: pgmap v5385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:41 smithi033 ceph-mon[27845]: pgmap v5386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:41 smithi096 ceph-mon[32753]: pgmap v5386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:41 smithi194 ceph-mon[33017]: pgmap v5386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8981.823383 secs 2024-04-26T14:20:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:43 smithi194 ceph-mon[33017]: pgmap v5387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8984.593470 secs 2024-04-26T14:20:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8981.823383 secs 2024-04-26T14:20:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:43 smithi033 ceph-mon[27845]: pgmap v5387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8984.593470 secs 2024-04-26T14:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8981.823383 secs 2024-04-26T14:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:43 smithi096 ceph-mon[32753]: pgmap v5387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8984.593470 secs 2024-04-26T14:20:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:45 smithi194 ceph-mon[33017]: pgmap v5388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:45 smithi033 ceph-mon[27845]: pgmap v5388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:45 smithi096 ceph-mon[32753]: pgmap v5388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:47 smithi194 ceph-mon[33017]: pgmap v5389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:47 smithi033 ceph-mon[27845]: pgmap v5389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:47 smithi096 ceph-mon[32753]: pgmap v5389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8986.823463 secs 2024-04-26T14:20:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8989.593606 secs 2024-04-26T14:20:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8986.823463 secs 2024-04-26T14:20:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8989.593606 secs 2024-04-26T14:20:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8986.823463 secs 2024-04-26T14:20:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8989.593606 secs 2024-04-26T14:20:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:49 smithi194 ceph-mon[33017]: pgmap v5390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:49 smithi033 ceph-mon[27845]: pgmap v5390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:49 smithi096 ceph-mon[32753]: pgmap v5390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:51 smithi194 ceph-mon[33017]: pgmap v5391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:51 smithi033 ceph-mon[27845]: pgmap v5391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:51 smithi096 ceph-mon[32753]: pgmap v5391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8991.823562 secs 2024-04-26T14:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:53 smithi194 ceph-mon[33017]: pgmap v5392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8994.593679 secs 2024-04-26T14:20:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8991.823562 secs 2024-04-26T14:20:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:53 smithi033 ceph-mon[27845]: pgmap v5392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8994.593679 secs 2024-04-26T14:20:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8991.823562 secs 2024-04-26T14:20:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:53 smithi096 ceph-mon[32753]: pgmap v5392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8994.593679 secs 2024-04-26T14:20:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:54.308+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:54.308+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:20:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:20:54.308+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:20:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:55 smithi194 ceph-mon[33017]: pgmap v5393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:55 smithi033 ceph-mon[27845]: pgmap v5393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:55 smithi096 ceph-mon[32753]: pgmap v5393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:20:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:57 smithi194 ceph-mon[33017]: pgmap v5394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:57 smithi033 ceph-mon[27845]: pgmap v5394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:57 smithi096 ceph-mon[32753]: pgmap v5394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:20:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 8996.823642 secs 2024-04-26T14:20:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 8999.593794 secs 2024-04-26T14:20:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 8996.823642 secs 2024-04-26T14:20:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 8999.593794 secs 2024-04-26T14:20:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 8996.823642 secs 2024-04-26T14:20:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 8999.593794 secs 2024-04-26T14:21:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:20:59 smithi033 ceph-mon[27845]: pgmap v5395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:20:59 smithi096 ceph-mon[32753]: pgmap v5395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:20:59 smithi194 ceph-mon[33017]: pgmap v5395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:01 smithi033 ceph-mon[27845]: pgmap v5396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:01 smithi096 ceph-mon[32753]: pgmap v5396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:01 smithi194 ceph-mon[33017]: pgmap v5396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:04.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9001.823724 secs 2024-04-26T14:21:04.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:03 smithi194 ceph-mon[33017]: pgmap v5397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:04.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9004.593910 secs 2024-04-26T14:21:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9001.823724 secs 2024-04-26T14:21:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:03 smithi033 ceph-mon[27845]: pgmap v5397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9004.593910 secs 2024-04-26T14:21:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9001.823724 secs 2024-04-26T14:21:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:03 smithi096 ceph-mon[32753]: pgmap v5397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9004.593910 secs 2024-04-26T14:21:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:05 smithi033 ceph-mon[27845]: pgmap v5398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:05 smithi096 ceph-mon[32753]: pgmap v5398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:05 smithi194 ceph-mon[33017]: pgmap v5398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:07 smithi033 ceph-mon[27845]: pgmap v5399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:07 smithi096 ceph-mon[32753]: pgmap v5399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:07 smithi194 ceph-mon[33017]: pgmap v5399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9006.823808 secs 2024-04-26T14:21:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9009.594033 secs 2024-04-26T14:21:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9006.823808 secs 2024-04-26T14:21:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9009.594033 secs 2024-04-26T14:21:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:09.316+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:09.316+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:09.316+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:21:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9006.823808 secs 2024-04-26T14:21:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9009.594033 secs 2024-04-26T14:21:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:09 smithi033 ceph-mon[27845]: pgmap v5400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:09 smithi096 ceph-mon[32753]: pgmap v5400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:09 smithi194 ceph-mon[33017]: pgmap v5400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:11 smithi033 ceph-mon[27845]: pgmap v5401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:21:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:11 smithi096 ceph-mon[32753]: pgmap v5401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:21:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:11 smithi194 ceph-mon[33017]: pgmap v5401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:21:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:21:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:21:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:21:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:21:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:21:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:21:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:21:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:21:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:13 smithi033 ceph-mon[27845]: pgmap v5402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:13 smithi033 ceph-mon[27845]: pgmap v5403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9011.823893 secs 2024-04-26T14:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9014.594175 secs 2024-04-26T14:21:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:13 smithi096 ceph-mon[32753]: pgmap v5402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:13 smithi096 ceph-mon[32753]: pgmap v5403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9011.823893 secs 2024-04-26T14:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9014.594175 secs 2024-04-26T14:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:13 smithi194 ceph-mon[33017]: pgmap v5402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:13 smithi194 ceph-mon[33017]: pgmap v5403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9011.823893 secs 2024-04-26T14:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9014.594175 secs 2024-04-26T14:21:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:15 smithi033 ceph-mon[27845]: pgmap v5404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:21:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:15 smithi096 ceph-mon[32753]: pgmap v5404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:21:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:15 smithi194 ceph-mon[33017]: pgmap v5404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:21:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:17 smithi033 ceph-mon[27845]: pgmap v5405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:17 smithi096 ceph-mon[32753]: pgmap v5405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:17 smithi194 ceph-mon[33017]: pgmap v5405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9016.823977 secs 2024-04-26T14:21:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9019.594271 secs 2024-04-26T14:21:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9016.823977 secs 2024-04-26T14:21:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9019.594271 secs 2024-04-26T14:21:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9016.823977 secs 2024-04-26T14:21:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9019.594271 secs 2024-04-26T14:21:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:19 smithi194 ceph-mon[33017]: pgmap v5406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:19 smithi033 ceph-mon[27845]: pgmap v5406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:19 smithi096 ceph-mon[32753]: pgmap v5406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:22.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:21 smithi033 ceph-mon[27845]: pgmap v5407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:21 smithi194 ceph-mon[33017]: pgmap v5407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:21 smithi096 ceph-mon[32753]: pgmap v5407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:24.328+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:24.328+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:24.328+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:21:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:23 smithi194 ceph-mon[33017]: pgmap v5408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:21:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9021.824033 secs 2024-04-26T14:21:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9024.594375 secs 2024-04-26T14:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:23 smithi033 ceph-mon[27845]: pgmap v5408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9021.824033 secs 2024-04-26T14:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9024.594375 secs 2024-04-26T14:21:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:23 smithi096 ceph-mon[32753]: pgmap v5408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9021.824033 secs 2024-04-26T14:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9024.594375 secs 2024-04-26T14:21:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:25 smithi194 ceph-mon[33017]: pgmap v5409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:25 smithi033 ceph-mon[27845]: pgmap v5409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:25 smithi096 ceph-mon[32753]: pgmap v5409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:27 smithi194 ceph-mon[33017]: pgmap v5410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:27 smithi033 ceph-mon[27845]: pgmap v5410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:27 smithi096 ceph-mon[32753]: pgmap v5410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9026.824121 secs 2024-04-26T14:21:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9029.594508 secs 2024-04-26T14:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9026.824121 secs 2024-04-26T14:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9029.594508 secs 2024-04-26T14:21:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9026.824121 secs 2024-04-26T14:21:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9029.594508 secs 2024-04-26T14:21:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:29 smithi194 ceph-mon[33017]: pgmap v5411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:29 smithi033 ceph-mon[27845]: pgmap v5411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:29 smithi096 ceph-mon[32753]: pgmap v5411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:31 smithi194 ceph-mon[33017]: pgmap v5412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:31 smithi033 ceph-mon[27845]: pgmap v5412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:31 smithi096 ceph-mon[32753]: pgmap v5412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:33 smithi194 ceph-mon[33017]: pgmap v5413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:33 smithi033 ceph-mon[27845]: pgmap v5413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:33 smithi096 ceph-mon[32753]: pgmap v5413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9031.824210 secs 2024-04-26T14:21:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9034.594638 secs 2024-04-26T14:21:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9031.824210 secs 2024-04-26T14:21:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9034.594638 secs 2024-04-26T14:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9031.824210 secs 2024-04-26T14:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9034.594638 secs 2024-04-26T14:21:35.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:35 smithi096 ceph-mon[32753]: pgmap v5414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:35 smithi194 ceph-mon[33017]: pgmap v5414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:35 smithi033 ceph-mon[27845]: pgmap v5414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:37 smithi194 ceph-mon[33017]: pgmap v5415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:37 smithi033 ceph-mon[27845]: pgmap v5415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:37 smithi096 ceph-mon[32753]: pgmap v5415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9036.824269 secs 2024-04-26T14:21:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9039.594765 secs 2024-04-26T14:21:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9036.824269 secs 2024-04-26T14:21:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9039.594765 secs 2024-04-26T14:21:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9036.824269 secs 2024-04-26T14:21:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9039.594765 secs 2024-04-26T14:21:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:39.342+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:39.342+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:39.342+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:21:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:39 smithi194 ceph-mon[33017]: pgmap v5416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:39 smithi033 ceph-mon[27845]: pgmap v5416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:39 smithi096 ceph-mon[32753]: pgmap v5416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:41 smithi194 ceph-mon[33017]: pgmap v5417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:41 smithi033 ceph-mon[27845]: pgmap v5417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:41 smithi096 ceph-mon[32753]: pgmap v5417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:43 smithi194 ceph-mon[33017]: pgmap v5418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:43.944 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9041.824344 secs 2024-04-26T14:21:43.944 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9044.594893 secs 2024-04-26T14:21:43.945 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:43 smithi096 ceph-mon[32753]: pgmap v5418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:43.945 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9041.824344 secs 2024-04-26T14:21:43.945 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9044.594893 secs 2024-04-26T14:21:43.946 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:43 smithi033 ceph-mon[27845]: pgmap v5418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:43.946 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9041.824344 secs 2024-04-26T14:21:43.946 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9044.594893 secs 2024-04-26T14:21:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:45 smithi194 ceph-mon[33017]: pgmap v5419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:45 smithi033 ceph-mon[27845]: pgmap v5419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:45 smithi096 ceph-mon[32753]: pgmap v5419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:47 smithi194 ceph-mon[33017]: pgmap v5420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:47 smithi033 ceph-mon[27845]: pgmap v5420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:47 smithi096 ceph-mon[32753]: pgmap v5420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9046.824428 secs 2024-04-26T14:21:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9049.595010 secs 2024-04-26T14:21:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9046.824428 secs 2024-04-26T14:21:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9049.595010 secs 2024-04-26T14:21:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9046.824428 secs 2024-04-26T14:21:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9049.595010 secs 2024-04-26T14:21:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:49 smithi194 ceph-mon[33017]: pgmap v5421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:49 smithi033 ceph-mon[27845]: pgmap v5421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:49 smithi096 ceph-mon[32753]: pgmap v5421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:51 smithi194 ceph-mon[33017]: pgmap v5422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:51 smithi033 ceph-mon[27845]: pgmap v5422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:51 smithi096 ceph-mon[32753]: pgmap v5422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:21:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:53 smithi033 ceph-mon[27845]: pgmap v5423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9051.824512 secs 2024-04-26T14:21:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9054.595125 secs 2024-04-26T14:21:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:53 smithi194 ceph-mon[33017]: pgmap v5423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9051.824512 secs 2024-04-26T14:21:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9054.595125 secs 2024-04-26T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:53 smithi096 ceph-mon[32753]: pgmap v5423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9051.824512 secs 2024-04-26T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9054.595125 secs 2024-04-26T14:21:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:54.355+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:54.356+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:21:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:21:54.356+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:21:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:55 smithi194 ceph-mon[33017]: pgmap v5424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:55 smithi033 ceph-mon[27845]: pgmap v5424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:55 smithi096 ceph-mon[32753]: pgmap v5424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:57 smithi194 ceph-mon[33017]: pgmap v5425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:57 smithi033 ceph-mon[27845]: pgmap v5425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:57 smithi096 ceph-mon[32753]: pgmap v5425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:21:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9056.824596 secs 2024-04-26T14:21:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9059.595248 secs 2024-04-26T14:21:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9056.824596 secs 2024-04-26T14:21:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9059.595248 secs 2024-04-26T14:21:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9056.824596 secs 2024-04-26T14:21:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9059.595248 secs 2024-04-26T14:21:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:21:59 smithi194 ceph-mon[33017]: pgmap v5426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:21:59 smithi033 ceph-mon[27845]: pgmap v5426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:21:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:21:59 smithi096 ceph-mon[32753]: pgmap v5426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:01 smithi194 ceph-mon[33017]: pgmap v5427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:01 smithi033 ceph-mon[27845]: pgmap v5427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:01 smithi096 ceph-mon[32753]: pgmap v5427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:03 smithi194 ceph-mon[33017]: pgmap v5428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9061.824680 secs 2024-04-26T14:22:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9064.595323 secs 2024-04-26T14:22:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:03 smithi033 ceph-mon[27845]: pgmap v5428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9061.824680 secs 2024-04-26T14:22:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9064.595323 secs 2024-04-26T14:22:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:03 smithi096 ceph-mon[32753]: pgmap v5428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9061.824680 secs 2024-04-26T14:22:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9064.595323 secs 2024-04-26T14:22:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:05 smithi194 ceph-mon[33017]: pgmap v5429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:05 smithi033 ceph-mon[27845]: pgmap v5429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:05 smithi096 ceph-mon[32753]: pgmap v5429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:07 smithi194 ceph-mon[33017]: pgmap v5430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:22:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:07 smithi033 ceph-mon[27845]: pgmap v5430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:22:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:07 smithi096 ceph-mon[32753]: pgmap v5430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:22:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9066.824762 secs 2024-04-26T14:22:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9069.595446 secs 2024-04-26T14:22:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9066.824762 secs 2024-04-26T14:22:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9069.595446 secs 2024-04-26T14:22:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9066.824762 secs 2024-04-26T14:22:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9069.595446 secs 2024-04-26T14:22:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:09.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:09.371+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:22:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:09.371+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:09 smithi194 ceph-mon[33017]: pgmap v5431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:09 smithi033 ceph-mon[27845]: pgmap v5431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:09 smithi096 ceph-mon[32753]: pgmap v5431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:11 smithi194 ceph-mon[33017]: pgmap v5432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:11 smithi033 ceph-mon[27845]: pgmap v5432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:11 smithi096 ceph-mon[32753]: pgmap v5432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:13 smithi194 ceph-mon[33017]: pgmap v5433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9071.824847 secs 2024-04-26T14:22:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:22:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9074.595543 secs 2024-04-26T14:22:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:13 smithi033 ceph-mon[27845]: pgmap v5433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9071.824847 secs 2024-04-26T14:22:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:22:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9074.595543 secs 2024-04-26T14:22:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:13 smithi096 ceph-mon[32753]: pgmap v5433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9071.824847 secs 2024-04-26T14:22:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:22:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9074.595543 secs 2024-04-26T14:22:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:22:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:22:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:22:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:22:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:22:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:22:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:22:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:22:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:22:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:15 smithi194 ceph-mon[33017]: pgmap v5434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:15 smithi194 ceph-mon[33017]: pgmap v5435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:22:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:15 smithi033 ceph-mon[27845]: pgmap v5434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:15 smithi033 ceph-mon[27845]: pgmap v5435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:22:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:15 smithi096 ceph-mon[32753]: pgmap v5434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:15 smithi096 ceph-mon[32753]: pgmap v5435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:22:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:17 smithi194 ceph-mon[33017]: pgmap v5436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:17 smithi033 ceph-mon[27845]: pgmap v5436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:17 smithi096 ceph-mon[32753]: pgmap v5436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9076.824930 secs 2024-04-26T14:22:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9079.595684 secs 2024-04-26T14:22:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9076.824930 secs 2024-04-26T14:22:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9079.595684 secs 2024-04-26T14:22:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9076.824930 secs 2024-04-26T14:22:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9079.595684 secs 2024-04-26T14:22:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:19 smithi194 ceph-mon[33017]: pgmap v5437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:19 smithi033 ceph-mon[27845]: pgmap v5437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:19 smithi096 ceph-mon[32753]: pgmap v5437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:21 smithi194 ceph-mon[33017]: pgmap v5438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:21 smithi033 ceph-mon[27845]: pgmap v5438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:21 smithi096 ceph-mon[32753]: pgmap v5438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:23 smithi194 ceph-mon[33017]: pgmap v5439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9081.825025 secs 2024-04-26T14:22:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9084.595809 secs 2024-04-26T14:22:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:23 smithi033 ceph-mon[27845]: pgmap v5439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9081.825025 secs 2024-04-26T14:22:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9084.595809 secs 2024-04-26T14:22:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:23 smithi096 ceph-mon[32753]: pgmap v5439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:22:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9081.825025 secs 2024-04-26T14:22:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9084.595809 secs 2024-04-26T14:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:24.386+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:24.386+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:24.386+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:22:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:25 smithi194 ceph-mon[33017]: pgmap v5440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:22:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:25 smithi033 ceph-mon[27845]: pgmap v5440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:22:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:25 smithi096 ceph-mon[32753]: pgmap v5440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:22:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:27 smithi194 ceph-mon[33017]: pgmap v5441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:27 smithi033 ceph-mon[27845]: pgmap v5441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:27 smithi096 ceph-mon[32753]: pgmap v5441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9086.825083 secs 2024-04-26T14:22:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9089.595933 secs 2024-04-26T14:22:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9086.825083 secs 2024-04-26T14:22:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9089.595933 secs 2024-04-26T14:22:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9086.825083 secs 2024-04-26T14:22:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9089.595933 secs 2024-04-26T14:22:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:29 smithi194 ceph-mon[33017]: pgmap v5442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:29 smithi033 ceph-mon[27845]: pgmap v5442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:29.969 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:29 smithi096 ceph-mon[32753]: pgmap v5442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:31 smithi033 ceph-mon[27845]: pgmap v5443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:31 smithi096 ceph-mon[32753]: pgmap v5443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:31 smithi194 ceph-mon[33017]: pgmap v5443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:33 smithi033 ceph-mon[27845]: pgmap v5444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9091.825158 secs 2024-04-26T14:22:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9094.596057 secs 2024-04-26T14:22:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:33 smithi096 ceph-mon[32753]: pgmap v5444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9091.825158 secs 2024-04-26T14:22:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9094.596057 secs 2024-04-26T14:22:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:33 smithi194 ceph-mon[33017]: pgmap v5444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9091.825158 secs 2024-04-26T14:22:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9094.596057 secs 2024-04-26T14:22:35.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:35 smithi194 ceph-mon[33017]: pgmap v5445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:35 smithi033 ceph-mon[27845]: pgmap v5445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:35 smithi096 ceph-mon[32753]: pgmap v5445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:37 smithi033 ceph-mon[27845]: pgmap v5446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:37 smithi096 ceph-mon[32753]: pgmap v5446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:37 smithi194 ceph-mon[33017]: pgmap v5446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9096.825242 secs 2024-04-26T14:22:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9099.596184 secs 2024-04-26T14:22:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9096.825242 secs 2024-04-26T14:22:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9099.596184 secs 2024-04-26T14:22:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9096.825242 secs 2024-04-26T14:22:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9099.596184 secs 2024-04-26T14:22:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:39.400+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:39.401+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:39.401+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:22:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:39 smithi033 ceph-mon[27845]: pgmap v5447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:39 smithi096 ceph-mon[32753]: pgmap v5447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:39 smithi194 ceph-mon[33017]: pgmap v5447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:41 smithi033 ceph-mon[27845]: pgmap v5448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:41 smithi096 ceph-mon[32753]: pgmap v5448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:41 smithi194 ceph-mon[33017]: pgmap v5448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:43 smithi033 ceph-mon[27845]: pgmap v5449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9101.825326 secs 2024-04-26T14:22:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9104.596302 secs 2024-04-26T14:22:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:43 smithi096 ceph-mon[32753]: pgmap v5449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9101.825326 secs 2024-04-26T14:22:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9104.596302 secs 2024-04-26T14:22:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:43 smithi194 ceph-mon[33017]: pgmap v5449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9101.825326 secs 2024-04-26T14:22:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9104.596302 secs 2024-04-26T14:22:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:45 smithi033 ceph-mon[27845]: pgmap v5450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:45 smithi096 ceph-mon[32753]: pgmap v5450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:45 smithi194 ceph-mon[33017]: pgmap v5450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:47 smithi033 ceph-mon[27845]: pgmap v5451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:47 smithi096 ceph-mon[32753]: pgmap v5451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:47 smithi194 ceph-mon[33017]: pgmap v5451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9106.825410 secs 2024-04-26T14:22:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9109.596428 secs 2024-04-26T14:22:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9106.825410 secs 2024-04-26T14:22:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9109.596428 secs 2024-04-26T14:22:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9106.825410 secs 2024-04-26T14:22:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9109.596428 secs 2024-04-26T14:22:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:49 smithi033 ceph-mon[27845]: pgmap v5452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:49 smithi096 ceph-mon[32753]: pgmap v5452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:49 smithi194 ceph-mon[33017]: pgmap v5452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:51 smithi033 ceph-mon[27845]: pgmap v5453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:51 smithi096 ceph-mon[32753]: pgmap v5453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:51 smithi194 ceph-mon[33017]: pgmap v5453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:53 smithi096 ceph-mon[32753]: pgmap v5454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9111.825494 secs 2024-04-26T14:22:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9114.596553 secs 2024-04-26T14:22:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:53 smithi194 ceph-mon[33017]: pgmap v5454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9111.825494 secs 2024-04-26T14:22:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9114.596553 secs 2024-04-26T14:22:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:53 smithi033 ceph-mon[27845]: pgmap v5454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9111.825494 secs 2024-04-26T14:22:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9114.596553 secs 2024-04-26T14:22:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:54.408+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:54.408+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:22:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:22:54.408+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:22:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:55 smithi194 ceph-mon[33017]: pgmap v5455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:55 smithi033 ceph-mon[27845]: pgmap v5455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:55 smithi096 ceph-mon[32753]: pgmap v5455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:22:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:57 smithi194 ceph-mon[33017]: pgmap v5456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:57 smithi033 ceph-mon[27845]: pgmap v5456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:57 smithi096 ceph-mon[32753]: pgmap v5456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:22:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9116.825580 secs 2024-04-26T14:22:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9119.596678 secs 2024-04-26T14:22:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9116.825580 secs 2024-04-26T14:22:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9119.596678 secs 2024-04-26T14:22:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9116.825580 secs 2024-04-26T14:22:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9119.596678 secs 2024-04-26T14:23:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:22:59 smithi194 ceph-mon[33017]: pgmap v5457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:22:59 smithi033 ceph-mon[27845]: pgmap v5457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:22:59 smithi096 ceph-mon[32753]: pgmap v5457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:01 smithi194 ceph-mon[33017]: pgmap v5458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:01 smithi033 ceph-mon[27845]: pgmap v5458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:01 smithi096 ceph-mon[32753]: pgmap v5458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:03 smithi194 ceph-mon[33017]: pgmap v5459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9121.825664 secs 2024-04-26T14:23:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9124.596829 secs 2024-04-26T14:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:03 smithi033 ceph-mon[27845]: pgmap v5459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9121.825664 secs 2024-04-26T14:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9124.596829 secs 2024-04-26T14:23:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:03 smithi096 ceph-mon[32753]: pgmap v5459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9121.825664 secs 2024-04-26T14:23:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9124.596829 secs 2024-04-26T14:23:05.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:05 smithi096 ceph-mon[32753]: pgmap v5460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:05 smithi194 ceph-mon[33017]: pgmap v5460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:05 smithi033 ceph-mon[27845]: pgmap v5460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:07 smithi194 ceph-mon[33017]: pgmap v5461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:07 smithi033 ceph-mon[27845]: pgmap v5461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:07 smithi096 ceph-mon[32753]: pgmap v5461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9126.825749 secs 2024-04-26T14:23:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9129.596948 secs 2024-04-26T14:23:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9126.825749 secs 2024-04-26T14:23:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9129.596948 secs 2024-04-26T14:23:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9126.825749 secs 2024-04-26T14:23:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9129.596948 secs 2024-04-26T14:23:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:09.409+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:09.409+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:09.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:09.409+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:23:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:09 smithi194 ceph-mon[33017]: pgmap v5462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:09 smithi033 ceph-mon[27845]: pgmap v5462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:09 smithi096 ceph-mon[32753]: pgmap v5462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:12.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:11 smithi194 ceph-mon[33017]: pgmap v5463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:11 smithi033 ceph-mon[27845]: pgmap v5463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:11 smithi096 ceph-mon[32753]: pgmap v5463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:13 smithi194 ceph-mon[33017]: pgmap v5464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9131.825833 secs 2024-04-26T14:23:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9134.597087 secs 2024-04-26T14:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:13 smithi033 ceph-mon[27845]: pgmap v5464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9131.825833 secs 2024-04-26T14:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9134.597087 secs 2024-04-26T14:23:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:13 smithi096 ceph-mon[32753]: pgmap v5464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9131.825833 secs 2024-04-26T14:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9134.597087 secs 2024-04-26T14:23:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:23:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:23:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:23:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:15 smithi194 ceph-mon[33017]: pgmap v5465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:23:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:23:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:23:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:15 smithi033 ceph-mon[27845]: pgmap v5465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:23:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:23:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:23:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:15 smithi096 ceph-mon[32753]: pgmap v5465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:23:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:16 smithi194 ceph-mon[33017]: pgmap v5466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:23:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:16 smithi194 ceph-mon[33017]: pgmap v5467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:16 smithi033 ceph-mon[27845]: pgmap v5466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:23:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:16 smithi033 ceph-mon[27845]: pgmap v5467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:16 smithi096 ceph-mon[32753]: pgmap v5466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:23:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:16 smithi096 ceph-mon[32753]: pgmap v5467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9136.825919 secs 2024-04-26T14:23:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:18 smithi194 ceph-mon[33017]: pgmap v5468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9139.597192 secs 2024-04-26T14:23:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9136.825919 secs 2024-04-26T14:23:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:18 smithi033 ceph-mon[27845]: pgmap v5468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9139.597192 secs 2024-04-26T14:23:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9136.825919 secs 2024-04-26T14:23:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:18 smithi096 ceph-mon[32753]: pgmap v5468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9139.597192 secs 2024-04-26T14:23:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:20 smithi194 ceph-mon[33017]: pgmap v5469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:20 smithi033 ceph-mon[27845]: pgmap v5469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:20 smithi096 ceph-mon[32753]: pgmap v5469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:22 smithi194 ceph-mon[33017]: pgmap v5470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:22 smithi033 ceph-mon[27845]: pgmap v5470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:22 smithi096 ceph-mon[32753]: pgmap v5470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:23.347+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:23:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:23.347+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:23:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:23.348+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:23:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:23.348+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:23:24.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9141.825979 secs 2024-04-26T14:23:24.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9144.597295 secs 2024-04-26T14:23:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9141.825979 secs 2024-04-26T14:23:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9144.597295 secs 2024-04-26T14:23:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9141.825979 secs 2024-04-26T14:23:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9144.597295 secs 2024-04-26T14:23:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:24.418+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:24.418+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:24.418+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:23:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:24 smithi194 ceph-mon[33017]: pgmap v5471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:24 smithi033 ceph-mon[27845]: pgmap v5471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:24 smithi096 ceph-mon[32753]: pgmap v5471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:23:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:26 smithi194 ceph-mon[33017]: pgmap v5472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:23:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:26 smithi033 ceph-mon[27845]: pgmap v5472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:23:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:26 smithi096 ceph-mon[32753]: pgmap v5472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:23:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9146.826066 secs 2024-04-26T14:23:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:28 smithi194 ceph-mon[33017]: pgmap v5473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9149.597384 secs 2024-04-26T14:23:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9146.826066 secs 2024-04-26T14:23:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:28 smithi033 ceph-mon[27845]: pgmap v5473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9149.597384 secs 2024-04-26T14:23:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9146.826066 secs 2024-04-26T14:23:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:28 smithi096 ceph-mon[32753]: pgmap v5473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9149.597384 secs 2024-04-26T14:23:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:30 smithi194 ceph-mon[33017]: pgmap v5474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:30 smithi033 ceph-mon[27845]: pgmap v5474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:30 smithi096 ceph-mon[32753]: pgmap v5474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:32 smithi194 ceph-mon[33017]: pgmap v5475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:32 smithi033 ceph-mon[27845]: pgmap v5475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:32 smithi096 ceph-mon[32753]: pgmap v5475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9151.826150 secs 2024-04-26T14:23:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9154.597510 secs 2024-04-26T14:23:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9151.826150 secs 2024-04-26T14:23:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9154.597510 secs 2024-04-26T14:23:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9151.826150 secs 2024-04-26T14:23:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9154.597510 secs 2024-04-26T14:23:35.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:34 smithi096 ceph-mon[32753]: pgmap v5476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:34 smithi194 ceph-mon[33017]: pgmap v5476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:34 smithi033 ceph-mon[27845]: pgmap v5476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:36 smithi194 ceph-mon[33017]: pgmap v5477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:36 smithi033 ceph-mon[27845]: pgmap v5477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:36 smithi096 ceph-mon[32753]: pgmap v5477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9156.826235 secs 2024-04-26T14:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:38 smithi194 ceph-mon[33017]: pgmap v5478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9159.597634 secs 2024-04-26T14:23:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9156.826235 secs 2024-04-26T14:23:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:38 smithi033 ceph-mon[27845]: pgmap v5478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9159.597634 secs 2024-04-26T14:23:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9156.826235 secs 2024-04-26T14:23:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:38 smithi096 ceph-mon[32753]: pgmap v5478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9159.597634 secs 2024-04-26T14:23:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:39.433+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:39.433+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:39.433+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:23:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:40 smithi194 ceph-mon[33017]: pgmap v5479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:40 smithi033 ceph-mon[27845]: pgmap v5479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:40 smithi096 ceph-mon[32753]: pgmap v5479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:42 smithi194 ceph-mon[33017]: pgmap v5480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:43.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:42 smithi033 ceph-mon[27845]: pgmap v5480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:42 smithi096 ceph-mon[32753]: pgmap v5480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9161.826318 secs 2024-04-26T14:23:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9164.597742 secs 2024-04-26T14:23:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9161.826318 secs 2024-04-26T14:23:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9164.597742 secs 2024-04-26T14:23:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9161.826318 secs 2024-04-26T14:23:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9164.597742 secs 2024-04-26T14:23:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:44 smithi033 ceph-mon[27845]: pgmap v5481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:44 smithi096 ceph-mon[32753]: pgmap v5481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:44 smithi194 ceph-mon[33017]: pgmap v5481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:46 smithi033 ceph-mon[27845]: pgmap v5482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:46 smithi096 ceph-mon[32753]: pgmap v5482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:46 smithi194 ceph-mon[33017]: pgmap v5482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9166.826402 secs 2024-04-26T14:23:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:48 smithi033 ceph-mon[27845]: pgmap v5483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9169.597886 secs 2024-04-26T14:23:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9166.826402 secs 2024-04-26T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:48 smithi096 ceph-mon[32753]: pgmap v5483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9169.597886 secs 2024-04-26T14:23:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9166.826402 secs 2024-04-26T14:23:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:48 smithi194 ceph-mon[33017]: pgmap v5483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9169.597886 secs 2024-04-26T14:23:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:50 smithi033 ceph-mon[27845]: pgmap v5484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:50 smithi096 ceph-mon[32753]: pgmap v5484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:50 smithi194 ceph-mon[33017]: pgmap v5484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:52 smithi033 ceph-mon[27845]: pgmap v5485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:52 smithi096 ceph-mon[32753]: pgmap v5485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:52 smithi194 ceph-mon[33017]: pgmap v5485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9171.826485 secs 2024-04-26T14:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9174.598008 secs 2024-04-26T14:23:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9171.826485 secs 2024-04-26T14:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9174.598008 secs 2024-04-26T14:23:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9171.826485 secs 2024-04-26T14:23:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9174.598008 secs 2024-04-26T14:23:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:54.449+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:54.449+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:23:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:23:54.449+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:23:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:54 smithi033 ceph-mon[27845]: pgmap v5486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:54 smithi096 ceph-mon[32753]: pgmap v5486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:54 smithi194 ceph-mon[33017]: pgmap v5486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:56 smithi033 ceph-mon[27845]: pgmap v5487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:56 smithi096 ceph-mon[32753]: pgmap v5487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:56 smithi194 ceph-mon[33017]: pgmap v5487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:23:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9176.826569 secs 2024-04-26T14:23:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:58 smithi033 ceph-mon[27845]: pgmap v5488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:23:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9179.598120 secs 2024-04-26T14:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9176.826569 secs 2024-04-26T14:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:58 smithi096 ceph-mon[32753]: pgmap v5488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:23:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9179.598120 secs 2024-04-26T14:23:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9176.826569 secs 2024-04-26T14:23:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:58 smithi194 ceph-mon[33017]: pgmap v5488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:23:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:23:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9179.598120 secs 2024-04-26T14:24:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:00 smithi033 ceph-mon[27845]: pgmap v5489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:00 smithi096 ceph-mon[32753]: pgmap v5489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:00 smithi194 ceph-mon[33017]: pgmap v5489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:02 smithi033 ceph-mon[27845]: pgmap v5490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:02 smithi096 ceph-mon[32753]: pgmap v5490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:02 smithi194 ceph-mon[33017]: pgmap v5490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9181.826652 secs 2024-04-26T14:24:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9184.598245 secs 2024-04-26T14:24:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9181.826652 secs 2024-04-26T14:24:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9184.598245 secs 2024-04-26T14:24:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9181.826652 secs 2024-04-26T14:24:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9184.598245 secs 2024-04-26T14:24:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:04 smithi096 ceph-mon[32753]: pgmap v5491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:05.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:04 smithi194 ceph-mon[33017]: pgmap v5491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:04 smithi033 ceph-mon[27845]: pgmap v5491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:06 smithi096 ceph-mon[32753]: pgmap v5492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:06 smithi194 ceph-mon[33017]: pgmap v5492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:06 smithi033 ceph-mon[27845]: pgmap v5492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9186.826734 secs 2024-04-26T14:24:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:08 smithi194 ceph-mon[33017]: pgmap v5493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9189.598369 secs 2024-04-26T14:24:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9186.826734 secs 2024-04-26T14:24:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:08 smithi033 ceph-mon[27845]: pgmap v5493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9189.598369 secs 2024-04-26T14:24:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9186.826734 secs 2024-04-26T14:24:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:08 smithi096 ceph-mon[32753]: pgmap v5493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9189.598369 secs 2024-04-26T14:24:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:09.464+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:09.465+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:09.465+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:24:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:10 smithi194 ceph-mon[33017]: pgmap v5494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:10 smithi033 ceph-mon[27845]: pgmap v5494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:10 smithi096 ceph-mon[32753]: pgmap v5494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:12 smithi194 ceph-mon[33017]: pgmap v5495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:12 smithi033 ceph-mon[27845]: pgmap v5495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:12 smithi096 ceph-mon[32753]: pgmap v5495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9191.826816 secs 2024-04-26T14:24:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9194.598451 secs 2024-04-26T14:24:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9191.826816 secs 2024-04-26T14:24:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9194.598451 secs 2024-04-26T14:24:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9191.826816 secs 2024-04-26T14:24:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9194.598451 secs 2024-04-26T14:24:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:14 smithi194 ceph-mon[33017]: pgmap v5496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:14 smithi033 ceph-mon[27845]: pgmap v5496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:14 smithi096 ceph-mon[32753]: pgmap v5496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:16 smithi194 ceph-mon[33017]: pgmap v5497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:24:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:16 smithi033 ceph-mon[27845]: pgmap v5497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:24:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:16 smithi096 ceph-mon[32753]: pgmap v5497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:24:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:24:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:24:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:24:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:24:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:24:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:24:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:24:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:24:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:24:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:18 smithi194 ceph-mon[33017]: pgmap v5498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:18 smithi194 ceph-mon[33017]: pgmap v5499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:18 smithi033 ceph-mon[27845]: pgmap v5498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:18 smithi033 ceph-mon[27845]: pgmap v5499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:18 smithi096 ceph-mon[32753]: pgmap v5498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:18 smithi096 ceph-mon[32753]: pgmap v5499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9196.826881 secs 2024-04-26T14:24:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9199.598597 secs 2024-04-26T14:24:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9196.826881 secs 2024-04-26T14:24:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9199.598597 secs 2024-04-26T14:24:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9196.826881 secs 2024-04-26T14:24:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9199.598597 secs 2024-04-26T14:24:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:20 smithi194 ceph-mon[33017]: pgmap v5500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:20 smithi033 ceph-mon[27845]: pgmap v5500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:20 smithi096 ceph-mon[32753]: pgmap v5500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:22 smithi194 ceph-mon[33017]: pgmap v5501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:22 smithi033 ceph-mon[27845]: pgmap v5501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:22 smithi096 ceph-mon[32753]: pgmap v5501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:24.475+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:24.475+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:24.475+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:24:24.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9201.826970 secs 2024-04-26T14:24:24.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9204.598738 secs 2024-04-26T14:24:24.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:24 smithi033 ceph-mon[27845]: pgmap v5502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9201.826970 secs 2024-04-26T14:24:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9204.598738 secs 2024-04-26T14:24:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:24 smithi194 ceph-mon[33017]: pgmap v5502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9201.826970 secs 2024-04-26T14:24:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9204.598738 secs 2024-04-26T14:24:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:24 smithi096 ceph-mon[32753]: pgmap v5502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:26 smithi194 ceph-mon[33017]: pgmap v5503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:26 smithi033 ceph-mon[27845]: pgmap v5503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:26 smithi096 ceph-mon[32753]: pgmap v5503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:28 smithi194 ceph-mon[33017]: pgmap v5504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:24:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9206.827060 secs 2024-04-26T14:24:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9209.598865 secs 2024-04-26T14:24:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:28 smithi033 ceph-mon[27845]: pgmap v5504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:24:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9206.827060 secs 2024-04-26T14:24:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9209.598865 secs 2024-04-26T14:24:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:28 smithi096 ceph-mon[32753]: pgmap v5504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:24:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9206.827060 secs 2024-04-26T14:24:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9209.598865 secs 2024-04-26T14:24:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:30 smithi194 ceph-mon[33017]: pgmap v5505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:30 smithi033 ceph-mon[27845]: pgmap v5505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:30 smithi096 ceph-mon[32753]: pgmap v5505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:32 smithi194 ceph-mon[33017]: pgmap v5506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:32 smithi033 ceph-mon[27845]: pgmap v5506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:32 smithi096 ceph-mon[32753]: pgmap v5506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9211.827144 secs 2024-04-26T14:24:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9214.599001 secs 2024-04-26T14:24:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9211.827144 secs 2024-04-26T14:24:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9214.599001 secs 2024-04-26T14:24:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9211.827144 secs 2024-04-26T14:24:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9214.599001 secs 2024-04-26T14:24:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:34 smithi194 ceph-mon[33017]: pgmap v5507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:34 smithi033 ceph-mon[27845]: pgmap v5507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:34 smithi096 ceph-mon[32753]: pgmap v5507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:36 smithi194 ceph-mon[33017]: pgmap v5508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:36 smithi033 ceph-mon[27845]: pgmap v5508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:36 smithi096 ceph-mon[32753]: pgmap v5508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:38 smithi194 ceph-mon[33017]: pgmap v5509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9216.827205 secs 2024-04-26T14:24:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9219.599102 secs 2024-04-26T14:24:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:38 smithi033 ceph-mon[27845]: pgmap v5509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9216.827205 secs 2024-04-26T14:24:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9219.599102 secs 2024-04-26T14:24:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:38 smithi096 ceph-mon[32753]: pgmap v5509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9216.827205 secs 2024-04-26T14:24:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9219.599102 secs 2024-04-26T14:24:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:39.482+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:39.482+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:39.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:39.482+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:24:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:40 smithi194 ceph-mon[33017]: pgmap v5510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:40 smithi033 ceph-mon[27845]: pgmap v5510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:40 smithi096 ceph-mon[32753]: pgmap v5510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:42 smithi194 ceph-mon[33017]: pgmap v5511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:42 smithi033 ceph-mon[27845]: pgmap v5511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:42 smithi096 ceph-mon[32753]: pgmap v5511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:24:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9221.827284 secs 2024-04-26T14:24:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9224.599193 secs 2024-04-26T14:24:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9221.827284 secs 2024-04-26T14:24:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9224.599193 secs 2024-04-26T14:24:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9221.827284 secs 2024-04-26T14:24:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9224.599193 secs 2024-04-26T14:24:44.849 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:44 smithi033 ceph-mon[27845]: pgmap v5512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:44 smithi194 ceph-mon[33017]: pgmap v5512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:44 smithi096 ceph-mon[32753]: pgmap v5512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:46 smithi194 ceph-mon[33017]: pgmap v5513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:46 smithi033 ceph-mon[27845]: pgmap v5513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:46 smithi096 ceph-mon[32753]: pgmap v5513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:24:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:48 smithi194 ceph-mon[33017]: pgmap v5514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9226.827366 secs 2024-04-26T14:24:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9229.599319 secs 2024-04-26T14:24:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:48 smithi033 ceph-mon[27845]: pgmap v5514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9226.827366 secs 2024-04-26T14:24:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9229.599319 secs 2024-04-26T14:24:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:48 smithi096 ceph-mon[32753]: pgmap v5514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:24:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9226.827366 secs 2024-04-26T14:24:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9229.599319 secs 2024-04-26T14:24:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:50 smithi194 ceph-mon[33017]: pgmap v5515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:24:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:50 smithi033 ceph-mon[27845]: pgmap v5515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:24:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:50 smithi096 ceph-mon[32753]: pgmap v5515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:24:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:52 smithi194 ceph-mon[33017]: pgmap v5516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:24:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:52 smithi033 ceph-mon[27845]: pgmap v5516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:24:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:52 smithi096 ceph-mon[32753]: pgmap v5516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:24:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9231.827448 secs 2024-04-26T14:24:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9234.599450 secs 2024-04-26T14:24:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9231.827448 secs 2024-04-26T14:24:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9234.599450 secs 2024-04-26T14:24:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9231.827448 secs 2024-04-26T14:24:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9234.599450 secs 2024-04-26T14:24:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:54.499+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:54.499+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:24:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:24:54.499+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:24:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:54 smithi194 ceph-mon[33017]: pgmap v5517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:54 smithi033 ceph-mon[27845]: pgmap v5517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:54 smithi096 ceph-mon[32753]: pgmap v5517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:56 smithi194 ceph-mon[33017]: pgmap v5518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:24:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:56 smithi033 ceph-mon[27845]: pgmap v5518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:24:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:56 smithi096 ceph-mon[32753]: pgmap v5518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:24:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:58 smithi194 ceph-mon[33017]: pgmap v5519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9236.827530 secs 2024-04-26T14:24:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:24:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9239.599576 secs 2024-04-26T14:24:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:58 smithi033 ceph-mon[27845]: pgmap v5519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9236.827530 secs 2024-04-26T14:24:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:24:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9239.599576 secs 2024-04-26T14:24:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:58 smithi096 ceph-mon[32753]: pgmap v5519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:24:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9236.827530 secs 2024-04-26T14:24:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:24:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9239.599576 secs 2024-04-26T14:25:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:00 smithi194 ceph-mon[33017]: pgmap v5520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:25:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:00 smithi033 ceph-mon[27845]: pgmap v5520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:25:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:00 smithi096 ceph-mon[32753]: pgmap v5520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:25:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:02 smithi194 ceph-mon[33017]: pgmap v5521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:02 smithi033 ceph-mon[27845]: pgmap v5521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:02 smithi096 ceph-mon[32753]: pgmap v5521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9241.827614 secs 2024-04-26T14:25:03.955 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9244.599713 secs 2024-04-26T14:25:03.955 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9241.827614 secs 2024-04-26T14:25:03.955 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9244.599713 secs 2024-04-26T14:25:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9241.827614 secs 2024-04-26T14:25:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9244.599713 secs 2024-04-26T14:25:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:04 smithi033 ceph-mon[27845]: pgmap v5522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:04 smithi096 ceph-mon[32753]: pgmap v5522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:04 smithi194 ceph-mon[33017]: pgmap v5522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:06.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:06 smithi194 ceph-mon[33017]: pgmap v5523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:06 smithi033 ceph-mon[27845]: pgmap v5523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:06 smithi096 ceph-mon[32753]: pgmap v5523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:08 smithi033 ceph-mon[27845]: pgmap v5524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9246.827696 secs 2024-04-26T14:25:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9249.599847 secs 2024-04-26T14:25:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:08 smithi096 ceph-mon[32753]: pgmap v5524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9246.827696 secs 2024-04-26T14:25:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9249.599847 secs 2024-04-26T14:25:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:08 smithi194 ceph-mon[33017]: pgmap v5524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9246.827696 secs 2024-04-26T14:25:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9249.599847 secs 2024-04-26T14:25:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:09.502+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:09.502+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:25:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:09.502+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:10 smithi033 ceph-mon[27845]: pgmap v5525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:10 smithi096 ceph-mon[32753]: pgmap v5525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:10 smithi194 ceph-mon[33017]: pgmap v5525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:12 smithi033 ceph-mon[27845]: pgmap v5526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:12 smithi096 ceph-mon[32753]: pgmap v5526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:12 smithi194 ceph-mon[33017]: pgmap v5526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9251.827776 secs 2024-04-26T14:25:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9254.599942 secs 2024-04-26T14:25:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9251.827776 secs 2024-04-26T14:25:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9254.599942 secs 2024-04-26T14:25:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9251.827776 secs 2024-04-26T14:25:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9254.599942 secs 2024-04-26T14:25:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:14 smithi033 ceph-mon[27845]: pgmap v5527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:14 smithi096 ceph-mon[32753]: pgmap v5527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:14 smithi194 ceph-mon[33017]: pgmap v5527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:16 smithi033 ceph-mon[27845]: pgmap v5528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:16 smithi194 ceph-mon[33017]: pgmap v5528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:16 smithi096 ceph-mon[32753]: pgmap v5528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:18 smithi194 ceph-mon[33017]: pgmap v5529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:25:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:18 smithi033 ceph-mon[27845]: pgmap v5529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:25:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:18 smithi096 ceph-mon[32753]: pgmap v5529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:25:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9256.827860 secs 2024-04-26T14:25:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9259.600013 secs 2024-04-26T14:25:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9256.827860 secs 2024-04-26T14:25:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9259.600013 secs 2024-04-26T14:25:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9256.827860 secs 2024-04-26T14:25:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9259.600013 secs 2024-04-26T14:25:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:20 smithi033 ceph-mon[27845]: pgmap v5530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:25:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:25:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:20 smithi096 ceph-mon[32753]: pgmap v5530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:25:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:25:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:20 smithi194 ceph-mon[33017]: pgmap v5530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:25:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:25:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:25:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:25:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:21 smithi033 ceph-mon[27845]: pgmap v5531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:25:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:21 smithi033 ceph-mon[27845]: pgmap v5532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:21 smithi096 ceph-mon[32753]: pgmap v5531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:25:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:21 smithi096 ceph-mon[32753]: pgmap v5532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:21 smithi194 ceph-mon[33017]: pgmap v5531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:25:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:21 smithi194 ceph-mon[33017]: pgmap v5532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:23 smithi033 ceph-mon[27845]: pgmap v5533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9261.827942 secs 2024-04-26T14:25:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9264.600083 secs 2024-04-26T14:25:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:23 smithi096 ceph-mon[32753]: pgmap v5533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9261.827942 secs 2024-04-26T14:25:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9264.600083 secs 2024-04-26T14:25:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:23 smithi194 ceph-mon[33017]: pgmap v5533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9261.827942 secs 2024-04-26T14:25:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9264.600083 secs 2024-04-26T14:25:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:24.509+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:24.509+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:24.509+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:25:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:25 smithi033 ceph-mon[27845]: pgmap v5534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:25:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:25 smithi096 ceph-mon[32753]: pgmap v5534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:25:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:25 smithi194 ceph-mon[33017]: pgmap v5534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:25:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:27 smithi033 ceph-mon[27845]: pgmap v5535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:27 smithi096 ceph-mon[32753]: pgmap v5535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:27 smithi194 ceph-mon[33017]: pgmap v5535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:25:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9266.828028 secs 2024-04-26T14:25:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9269.600220 secs 2024-04-26T14:25:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9266.828028 secs 2024-04-26T14:25:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9269.600220 secs 2024-04-26T14:25:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9266.828028 secs 2024-04-26T14:25:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9269.600220 secs 2024-04-26T14:25:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:29 smithi033 ceph-mon[27845]: pgmap v5536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:29 smithi096 ceph-mon[32753]: pgmap v5536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:29 smithi194 ceph-mon[33017]: pgmap v5536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:31 smithi096 ceph-mon[32753]: pgmap v5537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:25:31.951 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:31 smithi033 ceph-mon[27845]: pgmap v5537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:25:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:31 smithi194 ceph-mon[33017]: pgmap v5537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:25:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:33 smithi194 ceph-mon[33017]: pgmap v5538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9271.828114 secs 2024-04-26T14:25:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9274.600294 secs 2024-04-26T14:25:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:33 smithi033 ceph-mon[27845]: pgmap v5538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9271.828114 secs 2024-04-26T14:25:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9274.600294 secs 2024-04-26T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:33 smithi096 ceph-mon[32753]: pgmap v5538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9271.828114 secs 2024-04-26T14:25:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9274.600294 secs 2024-04-26T14:25:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:35 smithi194 ceph-mon[33017]: pgmap v5539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:35 smithi033 ceph-mon[27845]: pgmap v5539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:35 smithi096 ceph-mon[32753]: pgmap v5539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:37.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:37 smithi096 ceph-mon[32753]: pgmap v5540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:37 smithi194 ceph-mon[33017]: pgmap v5540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:37 smithi033 ceph-mon[27845]: pgmap v5540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9276.828204 secs 2024-04-26T14:25:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9279.600372 secs 2024-04-26T14:25:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9276.828204 secs 2024-04-26T14:25:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9279.600372 secs 2024-04-26T14:25:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9276.828204 secs 2024-04-26T14:25:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9279.600372 secs 2024-04-26T14:25:39.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:39.520+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:39.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:39.520+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:25:39.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:39.520+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:39 smithi194 ceph-mon[33017]: pgmap v5541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:39 smithi033 ceph-mon[27845]: pgmap v5541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:39 smithi096 ceph-mon[32753]: pgmap v5541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:41 smithi194 ceph-mon[33017]: pgmap v5542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:41 smithi033 ceph-mon[27845]: pgmap v5542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:41 smithi096 ceph-mon[32753]: pgmap v5542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:43 smithi194 ceph-mon[33017]: pgmap v5543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9281.828286 secs 2024-04-26T14:25:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9284.600478 secs 2024-04-26T14:25:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:43 smithi033 ceph-mon[27845]: pgmap v5543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9281.828286 secs 2024-04-26T14:25:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9284.600478 secs 2024-04-26T14:25:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:43 smithi096 ceph-mon[32753]: pgmap v5543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9281.828286 secs 2024-04-26T14:25:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9284.600478 secs 2024-04-26T14:25:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:45 smithi194 ceph-mon[33017]: pgmap v5544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:45 smithi033 ceph-mon[27845]: pgmap v5544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:45 smithi096 ceph-mon[32753]: pgmap v5544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:47 smithi194 ceph-mon[33017]: pgmap v5545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:47 smithi033 ceph-mon[27845]: pgmap v5545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:47 smithi096 ceph-mon[32753]: pgmap v5545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9286.828370 secs 2024-04-26T14:25:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9289.600553 secs 2024-04-26T14:25:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9286.828370 secs 2024-04-26T14:25:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9289.600553 secs 2024-04-26T14:25:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9286.828370 secs 2024-04-26T14:25:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9289.600553 secs 2024-04-26T14:25:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:49 smithi194 ceph-mon[33017]: pgmap v5546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:49 smithi033 ceph-mon[27845]: pgmap v5546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:49 smithi096 ceph-mon[32753]: pgmap v5546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:51 smithi194 ceph-mon[33017]: pgmap v5547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:51 smithi033 ceph-mon[27845]: pgmap v5547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:51 smithi096 ceph-mon[32753]: pgmap v5547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:53 smithi194 ceph-mon[33017]: pgmap v5548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9291.828453 secs 2024-04-26T14:25:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9294.600668 secs 2024-04-26T14:25:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:53 smithi033 ceph-mon[27845]: pgmap v5548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9291.828453 secs 2024-04-26T14:25:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9294.600668 secs 2024-04-26T14:25:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:53 smithi096 ceph-mon[32753]: pgmap v5548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9291.828453 secs 2024-04-26T14:25:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9294.600668 secs 2024-04-26T14:25:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:54.536+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:54.537+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:25:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:25:54.537+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:25:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:55 smithi194 ceph-mon[33017]: pgmap v5549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:55 smithi033 ceph-mon[27845]: pgmap v5549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:55 smithi096 ceph-mon[32753]: pgmap v5549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:25:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:57 smithi194 ceph-mon[33017]: pgmap v5550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:57 smithi033 ceph-mon[27845]: pgmap v5550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:57 smithi096 ceph-mon[32753]: pgmap v5550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:25:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9296.828538 secs 2024-04-26T14:25:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9299.600768 secs 2024-04-26T14:25:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9296.828538 secs 2024-04-26T14:25:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9299.600768 secs 2024-04-26T14:25:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9296.828538 secs 2024-04-26T14:25:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9299.600768 secs 2024-04-26T14:26:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:25:59 smithi194 ceph-mon[33017]: pgmap v5551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:25:59 smithi033 ceph-mon[27845]: pgmap v5551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:25:59 smithi096 ceph-mon[32753]: pgmap v5551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:01 smithi194 ceph-mon[33017]: pgmap v5552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:01 smithi033 ceph-mon[27845]: pgmap v5552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:01 smithi096 ceph-mon[32753]: pgmap v5552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:03 smithi194 ceph-mon[33017]: pgmap v5553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9301.828619 secs 2024-04-26T14:26:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9304.600884 secs 2024-04-26T14:26:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:03 smithi033 ceph-mon[27845]: pgmap v5553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9301.828619 secs 2024-04-26T14:26:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9304.600884 secs 2024-04-26T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:03 smithi096 ceph-mon[32753]: pgmap v5553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9301.828619 secs 2024-04-26T14:26:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9304.600884 secs 2024-04-26T14:26:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:05 smithi194 ceph-mon[33017]: pgmap v5554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:05 smithi033 ceph-mon[27845]: pgmap v5554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:05 smithi096 ceph-mon[32753]: pgmap v5554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:07 smithi194 ceph-mon[33017]: pgmap v5555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:07 smithi033 ceph-mon[27845]: pgmap v5555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:07 smithi096 ceph-mon[32753]: pgmap v5555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9306.828701 secs 2024-04-26T14:26:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9309.601003 secs 2024-04-26T14:26:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9306.828701 secs 2024-04-26T14:26:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9309.601003 secs 2024-04-26T14:26:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9306.828701 secs 2024-04-26T14:26:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9309.601003 secs 2024-04-26T14:26:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:09.552+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:09.552+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:09.552+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:26:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:09 smithi194 ceph-mon[33017]: pgmap v5556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:09 smithi033 ceph-mon[27845]: pgmap v5556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:09 smithi096 ceph-mon[32753]: pgmap v5556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:11 smithi194 ceph-mon[33017]: pgmap v5557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:11 smithi033 ceph-mon[27845]: pgmap v5557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:11 smithi096 ceph-mon[32753]: pgmap v5557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:13 smithi194 ceph-mon[33017]: pgmap v5558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9311.828782 secs 2024-04-26T14:26:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9314.601135 secs 2024-04-26T14:26:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:13 smithi033 ceph-mon[27845]: pgmap v5558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9311.828782 secs 2024-04-26T14:26:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9314.601135 secs 2024-04-26T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:13 smithi096 ceph-mon[32753]: pgmap v5558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9311.828782 secs 2024-04-26T14:26:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9314.601135 secs 2024-04-26T14:26:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:15 smithi194 ceph-mon[33017]: pgmap v5559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:15 smithi033 ceph-mon[27845]: pgmap v5559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:15 smithi096 ceph-mon[32753]: pgmap v5559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:17 smithi194 ceph-mon[33017]: pgmap v5560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:17 smithi033 ceph-mon[27845]: pgmap v5560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:17 smithi096 ceph-mon[32753]: pgmap v5560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9316.828866 secs 2024-04-26T14:26:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9319.601200 secs 2024-04-26T14:26:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9316.828866 secs 2024-04-26T14:26:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9319.601200 secs 2024-04-26T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9316.828866 secs 2024-04-26T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9319.601200 secs 2024-04-26T14:26:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:19 smithi194 ceph-mon[33017]: pgmap v5561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:19 smithi033 ceph-mon[27845]: pgmap v5561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:19 smithi096 ceph-mon[32753]: pgmap v5561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:26:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:26:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T14:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T14:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3fc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0054000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T14:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3fc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0054000 comp rx=0 tx=0).stop 2024-04-26T14:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3fc0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T14:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T14:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3fc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T14:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T14:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.734+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T14:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:21.736+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0057000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T14:26:22.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:21 smithi096 ceph-mon[32753]: pgmap v5562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:22.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:21 smithi194 ceph-mon[33017]: pgmap v5562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:21 smithi033 ceph-mon[27845]: pgmap v5562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.378+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:26:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.378+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:26:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.378+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T14:26:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.379+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T14:26:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.379+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x564385821900 2024-04-26T14:26:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.379+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:26:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.379+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:26:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:23.379+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:26:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:23 smithi033 ceph-mon[27845]: pgmap v5563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9321.828927 secs 2024-04-26T14:26:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9324.601262 secs 2024-04-26T14:26:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:23 smithi096 ceph-mon[32753]: pgmap v5563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9321.828927 secs 2024-04-26T14:26:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9324.601262 secs 2024-04-26T14:26:24.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:23 smithi194 ceph-mon[33017]: pgmap v5563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:24.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9321.828927 secs 2024-04-26T14:26:24.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9324.601262 secs 2024-04-26T14:26:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:24.555+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:24.555+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:24.555+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:26:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:25 smithi194 ceph-mon[33017]: pgmap v5564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:25 smithi033 ceph-mon[27845]: pgmap v5564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:25 smithi096 ceph-mon[32753]: pgmap v5564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:27 smithi194 ceph-mon[33017]: pgmap v5565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:26:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:26:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:27 smithi033 ceph-mon[27845]: pgmap v5565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:26:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:26:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:27 smithi096 ceph-mon[32753]: pgmap v5565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:26:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:26:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:26:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9326.829006 secs 2024-04-26T14:26:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:28 smithi194 ceph-mon[33017]: pgmap v5566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:28 smithi194 ceph-mon[33017]: pgmap v5567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9329.601374 secs 2024-04-26T14:26:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9326.829006 secs 2024-04-26T14:26:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:28 smithi033 ceph-mon[27845]: pgmap v5566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:28 smithi033 ceph-mon[27845]: pgmap v5567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9329.601374 secs 2024-04-26T14:26:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9326.829006 secs 2024-04-26T14:26:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:28 smithi096 ceph-mon[32753]: pgmap v5566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:28 smithi096 ceph-mon[32753]: pgmap v5567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9329.601374 secs 2024-04-26T14:26:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:30 smithi194 ceph-mon[33017]: pgmap v5568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:30 smithi033 ceph-mon[27845]: pgmap v5568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:30 smithi096 ceph-mon[32753]: pgmap v5568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:32 smithi194 ceph-mon[33017]: pgmap v5569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:32 smithi033 ceph-mon[27845]: pgmap v5569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:32 smithi096 ceph-mon[32753]: pgmap v5569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9331.829099 secs 2024-04-26T14:26:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9334.601485 secs 2024-04-26T14:26:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9331.829099 secs 2024-04-26T14:26:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9334.601485 secs 2024-04-26T14:26:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9331.829099 secs 2024-04-26T14:26:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9334.601485 secs 2024-04-26T14:26:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:34 smithi194 ceph-mon[33017]: pgmap v5570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:34 smithi033 ceph-mon[27845]: pgmap v5570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:34 smithi096 ceph-mon[32753]: pgmap v5570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:36 smithi194 ceph-mon[33017]: pgmap v5571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:36 smithi033 ceph-mon[27845]: pgmap v5571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:36 smithi096 ceph-mon[32753]: pgmap v5571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:26:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9336.829160 secs 2024-04-26T14:26:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:38 smithi194 ceph-mon[33017]: pgmap v5572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:26:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9339.601573 secs 2024-04-26T14:26:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9336.829160 secs 2024-04-26T14:26:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:38 smithi033 ceph-mon[27845]: pgmap v5572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:26:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9339.601573 secs 2024-04-26T14:26:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9336.829160 secs 2024-04-26T14:26:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:38 smithi096 ceph-mon[32753]: pgmap v5572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:26:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9339.601573 secs 2024-04-26T14:26:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:39.569+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:39.569+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:39.569+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:26:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:40 smithi194 ceph-mon[33017]: pgmap v5573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:40 smithi033 ceph-mon[27845]: pgmap v5573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:40 smithi096 ceph-mon[32753]: pgmap v5573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:42 smithi194 ceph-mon[33017]: pgmap v5574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:42 smithi033 ceph-mon[27845]: pgmap v5574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:42 smithi096 ceph-mon[32753]: pgmap v5574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9341.829244 secs 2024-04-26T14:26:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9344.601713 secs 2024-04-26T14:26:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9341.829244 secs 2024-04-26T14:26:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9344.601713 secs 2024-04-26T14:26:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9341.829244 secs 2024-04-26T14:26:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9344.601713 secs 2024-04-26T14:26:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:44 smithi194 ceph-mon[33017]: pgmap v5575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:44 smithi033 ceph-mon[27845]: pgmap v5575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:44 smithi096 ceph-mon[32753]: pgmap v5575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:46 smithi033 ceph-mon[27845]: pgmap v5576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:46 smithi096 ceph-mon[32753]: pgmap v5576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:46 smithi194 ceph-mon[33017]: pgmap v5576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9346.829329 secs 2024-04-26T14:26:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:48 smithi033 ceph-mon[27845]: pgmap v5577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9349.601841 secs 2024-04-26T14:26:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9346.829329 secs 2024-04-26T14:26:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:48 smithi096 ceph-mon[32753]: pgmap v5577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9349.601841 secs 2024-04-26T14:26:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9346.829329 secs 2024-04-26T14:26:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:48 smithi194 ceph-mon[33017]: pgmap v5577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9349.601841 secs 2024-04-26T14:26:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:50 smithi033 ceph-mon[27845]: pgmap v5578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:50 smithi096 ceph-mon[32753]: pgmap v5578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:50 smithi194 ceph-mon[33017]: pgmap v5578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:52 smithi033 ceph-mon[27845]: pgmap v5579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:52 smithi096 ceph-mon[32753]: pgmap v5579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:52 smithi194 ceph-mon[33017]: pgmap v5579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9351.829413 secs 2024-04-26T14:26:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9354.601973 secs 2024-04-26T14:26:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9351.829413 secs 2024-04-26T14:26:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9354.601973 secs 2024-04-26T14:26:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9351.829413 secs 2024-04-26T14:26:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9354.601973 secs 2024-04-26T14:26:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:54.585+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:54.585+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:26:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:26:54.585+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:26:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:54 smithi033 ceph-mon[27845]: pgmap v5580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:54 smithi096 ceph-mon[32753]: pgmap v5580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:54 smithi194 ceph-mon[33017]: pgmap v5580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:56 smithi033 ceph-mon[27845]: pgmap v5581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:56 smithi096 ceph-mon[32753]: pgmap v5581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:56 smithi194 ceph-mon[33017]: pgmap v5581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:26:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9356.829497 secs 2024-04-26T14:26:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:58 smithi033 ceph-mon[27845]: pgmap v5582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:26:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9359.602094 secs 2024-04-26T14:26:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9356.829497 secs 2024-04-26T14:26:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:58 smithi096 ceph-mon[32753]: pgmap v5582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:26:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9359.602094 secs 2024-04-26T14:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9356.829497 secs 2024-04-26T14:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:58 smithi194 ceph-mon[33017]: pgmap v5582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:26:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9359.602094 secs 2024-04-26T14:27:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:00 smithi033 ceph-mon[27845]: pgmap v5583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:00 smithi096 ceph-mon[32753]: pgmap v5583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:00 smithi194 ceph-mon[33017]: pgmap v5583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:02 smithi033 ceph-mon[27845]: pgmap v5584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:02 smithi096 ceph-mon[32753]: pgmap v5584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:02 smithi194 ceph-mon[33017]: pgmap v5584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9361.829559 secs 2024-04-26T14:27:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9364.602205 secs 2024-04-26T14:27:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9361.829559 secs 2024-04-26T14:27:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9364.602205 secs 2024-04-26T14:27:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9361.829559 secs 2024-04-26T14:27:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9364.602205 secs 2024-04-26T14:27:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:04 smithi096 ceph-mon[32753]: pgmap v5585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:04 smithi194 ceph-mon[33017]: pgmap v5585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:04 smithi033 ceph-mon[27845]: pgmap v5585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:06 smithi096 ceph-mon[32753]: pgmap v5586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:06 smithi194 ceph-mon[33017]: pgmap v5586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:06 smithi033 ceph-mon[27845]: pgmap v5586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9366.829658 secs 2024-04-26T14:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:08 smithi194 ceph-mon[33017]: pgmap v5587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9369.602317 secs 2024-04-26T14:27:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9366.829658 secs 2024-04-26T14:27:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:08 smithi033 ceph-mon[27845]: pgmap v5587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9369.602317 secs 2024-04-26T14:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9366.829658 secs 2024-04-26T14:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:08 smithi096 ceph-mon[32753]: pgmap v5587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9369.602317 secs 2024-04-26T14:27:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:09.589+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:09.589+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:09.589+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:27:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:10 smithi194 ceph-mon[33017]: pgmap v5588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:10 smithi033 ceph-mon[27845]: pgmap v5588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:10 smithi096 ceph-mon[32753]: pgmap v5588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:12 smithi194 ceph-mon[33017]: pgmap v5589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:12 smithi033 ceph-mon[27845]: pgmap v5589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:12 smithi096 ceph-mon[32753]: pgmap v5589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9371.829760 secs 2024-04-26T14:27:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9374.602389 secs 2024-04-26T14:27:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9371.829760 secs 2024-04-26T14:27:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9374.602389 secs 2024-04-26T14:27:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9371.829760 secs 2024-04-26T14:27:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9374.602389 secs 2024-04-26T14:27:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:14 smithi194 ceph-mon[33017]: pgmap v5590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:14 smithi033 ceph-mon[27845]: pgmap v5590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:14 smithi096 ceph-mon[32753]: pgmap v5590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:16 smithi194 ceph-mon[33017]: pgmap v5591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:16 smithi033 ceph-mon[27845]: pgmap v5591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:16 smithi096 ceph-mon[32753]: pgmap v5591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9376.829847 secs 2024-04-26T14:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:18 smithi194 ceph-mon[33017]: pgmap v5592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9379.602506 secs 2024-04-26T14:27:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9376.829847 secs 2024-04-26T14:27:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:18 smithi033 ceph-mon[27845]: pgmap v5592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9379.602506 secs 2024-04-26T14:27:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9376.829847 secs 2024-04-26T14:27:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:18 smithi096 ceph-mon[32753]: pgmap v5592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9379.602506 secs 2024-04-26T14:27:21.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:20 smithi033 ceph-mon[27845]: pgmap v5593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:20 smithi194 ceph-mon[33017]: pgmap v5593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:20 smithi096 ceph-mon[32753]: pgmap v5593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:22 smithi194 ceph-mon[33017]: pgmap v5594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:22 smithi033 ceph-mon[27845]: pgmap v5594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:22 smithi096 ceph-mon[32753]: pgmap v5594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9381.829933 secs 2024-04-26T14:27:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9384.602626 secs 2024-04-26T14:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9381.829933 secs 2024-04-26T14:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9384.602626 secs 2024-04-26T14:27:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9381.829933 secs 2024-04-26T14:27:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9384.602626 secs 2024-04-26T14:27:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:24.605+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:24.605+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:27:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:24.605+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:25.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:24 smithi033 ceph-mon[27845]: pgmap v5595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:24 smithi194 ceph-mon[33017]: pgmap v5595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:24 smithi096 ceph-mon[32753]: pgmap v5595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:26 smithi194 ceph-mon[33017]: pgmap v5596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:26 smithi033 ceph-mon[27845]: pgmap v5596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:26 smithi096 ceph-mon[32753]: pgmap v5596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9386.830027 secs 2024-04-26T14:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: pgmap v5597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9389.602698 secs 2024-04-26T14:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9386.830027 secs 2024-04-26T14:27:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: pgmap v5597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:27:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9389.602698 secs 2024-04-26T14:27:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9386.830027 secs 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: pgmap v5597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9389.602698 secs 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:30 smithi194 ceph-mon[33017]: pgmap v5598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:30 smithi033 ceph-mon[27845]: pgmap v5598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:30 smithi096 ceph-mon[32753]: pgmap v5598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:32.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:32 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:27:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:32 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:27:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:32 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:27:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:32 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:27:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:32 smithi194 ceph-mon[33017]: pgmap v5599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:32 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:27:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:32 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:27:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:32 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:27:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:32 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:27:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:32 smithi033 ceph-mon[27845]: pgmap v5599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:32 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:27:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:32 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:27:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:32 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:27:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:32 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:27:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:32 smithi096 ceph-mon[32753]: pgmap v5599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9391.830090 secs 2024-04-26T14:27:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9394.602834 secs 2024-04-26T14:27:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9391.830090 secs 2024-04-26T14:27:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9394.602834 secs 2024-04-26T14:27:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9391.830090 secs 2024-04-26T14:27:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9394.602834 secs 2024-04-26T14:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: pgmap v5600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: pgmap v5601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: pgmap v5602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: pgmap v5600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: pgmap v5601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: pgmap v5602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:27:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: pgmap v5600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:27:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:27:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: pgmap v5601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:27:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: pgmap v5602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:27:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:27:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:36 smithi194 ceph-mon[33017]: pgmap v5603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:36 smithi033 ceph-mon[27845]: pgmap v5603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:36 smithi096 ceph-mon[32753]: pgmap v5603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:39.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9396.830149 secs 2024-04-26T14:27:39.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:38 smithi096 ceph-mon[32753]: pgmap v5604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:27:39.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9399.602989 secs 2024-04-26T14:27:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9396.830149 secs 2024-04-26T14:27:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:38 smithi194 ceph-mon[33017]: pgmap v5604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:27:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9399.602989 secs 2024-04-26T14:27:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9396.830149 secs 2024-04-26T14:27:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:38 smithi033 ceph-mon[27845]: pgmap v5604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:27:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9399.602989 secs 2024-04-26T14:27:39.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:39.605+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:39.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:39.605+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:27:39.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:39.605+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:40 smithi194 ceph-mon[33017]: pgmap v5605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:40 smithi033 ceph-mon[27845]: pgmap v5605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:40 smithi096 ceph-mon[32753]: pgmap v5605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:42 smithi194 ceph-mon[33017]: pgmap v5606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:42 smithi033 ceph-mon[27845]: pgmap v5606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:42 smithi096 ceph-mon[32753]: pgmap v5606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9401.830208 secs 2024-04-26T14:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9404.603123 secs 2024-04-26T14:27:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9401.830208 secs 2024-04-26T14:27:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9404.603123 secs 2024-04-26T14:27:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9401.830208 secs 2024-04-26T14:27:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9404.603123 secs 2024-04-26T14:27:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:44 smithi194 ceph-mon[33017]: pgmap v5607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:27:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:44 smithi033 ceph-mon[27845]: pgmap v5607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:27:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:44 smithi096 ceph-mon[32753]: pgmap v5607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:27:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:46 smithi194 ceph-mon[33017]: pgmap v5608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:46 smithi033 ceph-mon[27845]: pgmap v5608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:47 smithi096 ceph-mon[32753]: pgmap v5608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9406.830276 secs 2024-04-26T14:27:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:49 smithi194 ceph-mon[33017]: pgmap v5609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9409.603124 secs 2024-04-26T14:27:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9406.830276 secs 2024-04-26T14:27:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:49 smithi033 ceph-mon[27845]: pgmap v5609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9409.603124 secs 2024-04-26T14:27:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9406.830276 secs 2024-04-26T14:27:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:49 smithi096 ceph-mon[32753]: pgmap v5609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9409.603124 secs 2024-04-26T14:27:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:51 smithi194 ceph-mon[33017]: pgmap v5610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:51 smithi033 ceph-mon[27845]: pgmap v5610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:51 smithi096 ceph-mon[32753]: pgmap v5610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:53 smithi194 ceph-mon[33017]: pgmap v5611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:53 smithi033 ceph-mon[27845]: pgmap v5611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:53 smithi096 ceph-mon[32753]: pgmap v5611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9411.830355 secs 2024-04-26T14:27:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9414.603241 secs 2024-04-26T14:27:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9411.830355 secs 2024-04-26T14:27:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9414.603241 secs 2024-04-26T14:27:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9411.830355 secs 2024-04-26T14:27:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9414.603241 secs 2024-04-26T14:27:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:54.621+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:54.621+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:27:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:27:54.621+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:27:55.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:55 smithi033 ceph-mon[27845]: pgmap v5612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:55 smithi194 ceph-mon[33017]: pgmap v5612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:55 smithi096 ceph-mon[32753]: pgmap v5612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:57 smithi194 ceph-mon[33017]: pgmap v5613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:57 smithi033 ceph-mon[27845]: pgmap v5613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:57 smithi096 ceph-mon[32753]: pgmap v5613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:27:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9416.830436 secs 2024-04-26T14:27:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:59 smithi194 ceph-mon[33017]: pgmap v5614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9419.603344 secs 2024-04-26T14:27:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9416.830436 secs 2024-04-26T14:27:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:59 smithi033 ceph-mon[27845]: pgmap v5614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9419.603344 secs 2024-04-26T14:27:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9416.830436 secs 2024-04-26T14:27:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:59 smithi096 ceph-mon[32753]: pgmap v5614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:27:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9419.603344 secs 2024-04-26T14:28:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:01 smithi194 ceph-mon[33017]: pgmap v5615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:01 smithi033 ceph-mon[27845]: pgmap v5615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:01 smithi096 ceph-mon[32753]: pgmap v5615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:03 smithi194 ceph-mon[33017]: pgmap v5616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:03 smithi033 ceph-mon[27845]: pgmap v5616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:03 smithi096 ceph-mon[32753]: pgmap v5616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9421.830495 secs 2024-04-26T14:28:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9424.603459 secs 2024-04-26T14:28:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9421.830495 secs 2024-04-26T14:28:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9424.603459 secs 2024-04-26T14:28:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9421.830495 secs 2024-04-26T14:28:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9424.603459 secs 2024-04-26T14:28:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:05 smithi194 ceph-mon[33017]: pgmap v5617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:05 smithi033 ceph-mon[27845]: pgmap v5617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:05 smithi096 ceph-mon[32753]: pgmap v5617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:06 smithi194 ceph-mon[33017]: pgmap v5618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:06 smithi033 ceph-mon[27845]: pgmap v5618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:06 smithi096 ceph-mon[32753]: pgmap v5618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9426.830579 secs 2024-04-26T14:28:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:08 smithi194 ceph-mon[33017]: pgmap v5619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9429.603581 secs 2024-04-26T14:28:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9426.830579 secs 2024-04-26T14:28:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:08 smithi033 ceph-mon[27845]: pgmap v5619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9429.603581 secs 2024-04-26T14:28:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9426.830579 secs 2024-04-26T14:28:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:08 smithi096 ceph-mon[32753]: pgmap v5619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9429.603581 secs 2024-04-26T14:28:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:09.637+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:09.637+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:28:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:09.637+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:10 smithi194 ceph-mon[33017]: pgmap v5620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:10 smithi033 ceph-mon[27845]: pgmap v5620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:10 smithi096 ceph-mon[32753]: pgmap v5620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:12 smithi194 ceph-mon[33017]: pgmap v5621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:12 smithi033 ceph-mon[27845]: pgmap v5621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:12 smithi096 ceph-mon[32753]: pgmap v5621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9431.830660 secs 2024-04-26T14:28:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9434.603718 secs 2024-04-26T14:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9431.830660 secs 2024-04-26T14:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9434.603718 secs 2024-04-26T14:28:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9431.830660 secs 2024-04-26T14:28:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9434.603718 secs 2024-04-26T14:28:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:15 smithi194 ceph-mon[33017]: pgmap v5622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:15 smithi033 ceph-mon[27845]: pgmap v5622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:15 smithi096 ceph-mon[32753]: pgmap v5622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:17 smithi194 ceph-mon[33017]: pgmap v5623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:17 smithi033 ceph-mon[27845]: pgmap v5623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:17 smithi096 ceph-mon[32753]: pgmap v5623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9436.830743 secs 2024-04-26T14:28:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:19 smithi194 ceph-mon[33017]: pgmap v5624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9439.603857 secs 2024-04-26T14:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9436.830743 secs 2024-04-26T14:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:19 smithi033 ceph-mon[27845]: pgmap v5624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9439.603857 secs 2024-04-26T14:28:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9436.830743 secs 2024-04-26T14:28:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:19 smithi096 ceph-mon[32753]: pgmap v5624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9439.603857 secs 2024-04-26T14:28:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:21 smithi194 ceph-mon[33017]: pgmap v5625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:21 smithi033 ceph-mon[27845]: pgmap v5625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:21 smithi096 ceph-mon[32753]: pgmap v5625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:23.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:23 smithi033 ceph-mon[27845]: pgmap v5626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:23 smithi194 ceph-mon[33017]: pgmap v5626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:23 smithi096 ceph-mon[32753]: pgmap v5626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9441.830844 secs 2024-04-26T14:28:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9444.603991 secs 2024-04-26T14:28:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9441.830844 secs 2024-04-26T14:28:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9444.603991 secs 2024-04-26T14:28:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9441.830844 secs 2024-04-26T14:28:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9444.603991 secs 2024-04-26T14:28:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:24.640+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:24.640+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:28:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:24.640+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:25.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:25 smithi033 ceph-mon[27845]: pgmap v5627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:25 smithi194 ceph-mon[33017]: pgmap v5627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:25 smithi096 ceph-mon[32753]: pgmap v5627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:27 smithi194 ceph-mon[33017]: pgmap v5628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:27 smithi033 ceph-mon[27845]: pgmap v5628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:27 smithi096 ceph-mon[32753]: pgmap v5628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9446.830934 secs 2024-04-26T14:28:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:29 smithi194 ceph-mon[33017]: pgmap v5629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9449.604140 secs 2024-04-26T14:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9446.830934 secs 2024-04-26T14:28:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:29 smithi033 ceph-mon[27845]: pgmap v5629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9449.604140 secs 2024-04-26T14:28:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9446.830934 secs 2024-04-26T14:28:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:29 smithi096 ceph-mon[32753]: pgmap v5629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9449.604140 secs 2024-04-26T14:28:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:31 smithi194 ceph-mon[33017]: pgmap v5630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:31 smithi033 ceph-mon[27845]: pgmap v5630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:31 smithi096 ceph-mon[32753]: pgmap v5630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:33 smithi194 ceph-mon[33017]: pgmap v5631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:33 smithi033 ceph-mon[27845]: pgmap v5631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:33 smithi096 ceph-mon[32753]: pgmap v5631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9451.831026 secs 2024-04-26T14:28:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9454.604258 secs 2024-04-26T14:28:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9451.831026 secs 2024-04-26T14:28:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9454.604258 secs 2024-04-26T14:28:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9451.831026 secs 2024-04-26T14:28:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9454.604258 secs 2024-04-26T14:28:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:35 smithi194 ceph-mon[33017]: pgmap v5632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:28:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:35 smithi033 ceph-mon[27845]: pgmap v5632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:28:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:35 smithi096 ceph-mon[32753]: pgmap v5632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:28:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:28:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:28:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:28:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:28:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:28:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:28:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:28:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:28:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:28:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:37 smithi194 ceph-mon[33017]: pgmap v5633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:28:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:37 smithi194 ceph-mon[33017]: pgmap v5634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:37 smithi033 ceph-mon[27845]: pgmap v5633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:28:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:37 smithi033 ceph-mon[27845]: pgmap v5634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:37 smithi096 ceph-mon[32753]: pgmap v5633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:28:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:37 smithi096 ceph-mon[32753]: pgmap v5634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:38 smithi194 ceph-mon[33017]: pgmap v5635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:38 smithi033 ceph-mon[27845]: pgmap v5635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:38 smithi096 ceph-mon[32753]: pgmap v5635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9456.831086 secs 2024-04-26T14:28:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9459.604377 secs 2024-04-26T14:28:39.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9456.831086 secs 2024-04-26T14:28:39.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9459.604377 secs 2024-04-26T14:28:39.438 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9456.831086 secs 2024-04-26T14:28:39.438 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9459.604377 secs 2024-04-26T14:28:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:39.644+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:39.645+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:28:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:39.645+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:40 smithi033 ceph-mon[27845]: pgmap v5636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:40 smithi096 ceph-mon[32753]: pgmap v5636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:40 smithi194 ceph-mon[33017]: pgmap v5636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:42 smithi194 ceph-mon[33017]: pgmap v5637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:42 smithi033 ceph-mon[27845]: pgmap v5637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:42 smithi096 ceph-mon[32753]: pgmap v5637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9461.831147 secs 2024-04-26T14:28:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9464.604495 secs 2024-04-26T14:28:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9461.831147 secs 2024-04-26T14:28:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9464.604495 secs 2024-04-26T14:28:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9461.831147 secs 2024-04-26T14:28:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9464.604495 secs 2024-04-26T14:28:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:44 smithi194 ceph-mon[33017]: pgmap v5638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:44 smithi033 ceph-mon[27845]: pgmap v5638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:44 smithi096 ceph-mon[32753]: pgmap v5638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:28:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:46 smithi194 ceph-mon[33017]: pgmap v5639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:28:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:46 smithi033 ceph-mon[27845]: pgmap v5639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:28:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:46 smithi096 ceph-mon[32753]: pgmap v5639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:28:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9466.831236 secs 2024-04-26T14:28:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:48 smithi194 ceph-mon[33017]: pgmap v5640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9469.604646 secs 2024-04-26T14:28:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9466.831236 secs 2024-04-26T14:28:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:48 smithi033 ceph-mon[27845]: pgmap v5640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9469.604646 secs 2024-04-26T14:28:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9466.831236 secs 2024-04-26T14:28:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:48 smithi096 ceph-mon[32753]: pgmap v5640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9469.604646 secs 2024-04-26T14:28:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:50 smithi194 ceph-mon[33017]: pgmap v5641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:50 smithi033 ceph-mon[27845]: pgmap v5641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:50 smithi096 ceph-mon[32753]: pgmap v5641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:52 smithi194 ceph-mon[33017]: pgmap v5642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:52 smithi033 ceph-mon[27845]: pgmap v5642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:52 smithi096 ceph-mon[32753]: pgmap v5642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9471.831336 secs 2024-04-26T14:28:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9474.604769 secs 2024-04-26T14:28:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9471.831336 secs 2024-04-26T14:28:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9474.604769 secs 2024-04-26T14:28:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9471.831336 secs 2024-04-26T14:28:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9474.604769 secs 2024-04-26T14:28:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:54.661+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:54.661+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:28:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:28:54.661+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:28:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:54 smithi194 ceph-mon[33017]: pgmap v5643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:54 smithi033 ceph-mon[27845]: pgmap v5643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:54 smithi096 ceph-mon[32753]: pgmap v5643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:56 smithi194 ceph-mon[33017]: pgmap v5644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:56 smithi033 ceph-mon[27845]: pgmap v5644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:56 smithi096 ceph-mon[32753]: pgmap v5644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:28:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9476.831395 secs 2024-04-26T14:28:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:58 smithi194 ceph-mon[33017]: pgmap v5645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:28:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9479.604903 secs 2024-04-26T14:28:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9476.831395 secs 2024-04-26T14:28:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:58 smithi033 ceph-mon[27845]: pgmap v5645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:28:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9479.604903 secs 2024-04-26T14:28:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9476.831395 secs 2024-04-26T14:28:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:58 smithi096 ceph-mon[32753]: pgmap v5645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:28:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:28:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9479.604903 secs 2024-04-26T14:29:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:00 smithi194 ceph-mon[33017]: pgmap v5646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:00 smithi033 ceph-mon[27845]: pgmap v5646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:00 smithi096 ceph-mon[32753]: pgmap v5646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:02 smithi194 ceph-mon[33017]: pgmap v5647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:02 smithi033 ceph-mon[27845]: pgmap v5647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:02 smithi096 ceph-mon[32753]: pgmap v5647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9481.831480 secs 2024-04-26T14:29:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9484.605006 secs 2024-04-26T14:29:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9481.831480 secs 2024-04-26T14:29:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9484.605006 secs 2024-04-26T14:29:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9481.831480 secs 2024-04-26T14:29:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9484.605006 secs 2024-04-26T14:29:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:04 smithi194 ceph-mon[33017]: pgmap v5648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:04 smithi033 ceph-mon[27845]: pgmap v5648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:04 smithi096 ceph-mon[32753]: pgmap v5648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:06 smithi194 ceph-mon[33017]: pgmap v5649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:06 smithi033 ceph-mon[27845]: pgmap v5649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:06 smithi096 ceph-mon[32753]: pgmap v5649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9486.831560 secs 2024-04-26T14:29:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:08 smithi194 ceph-mon[33017]: pgmap v5650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9489.605146 secs 2024-04-26T14:29:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9486.831560 secs 2024-04-26T14:29:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:08 smithi033 ceph-mon[27845]: pgmap v5650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9489.605146 secs 2024-04-26T14:29:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9486.831560 secs 2024-04-26T14:29:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:08 smithi096 ceph-mon[32753]: pgmap v5650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9489.605146 secs 2024-04-26T14:29:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:09.670+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:09.670+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:29:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:09.670+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:10 smithi194 ceph-mon[33017]: pgmap v5651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:10 smithi033 ceph-mon[27845]: pgmap v5651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:10 smithi096 ceph-mon[32753]: pgmap v5651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:12 smithi194 ceph-mon[33017]: pgmap v5652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:12 smithi033 ceph-mon[27845]: pgmap v5652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:12 smithi096 ceph-mon[32753]: pgmap v5652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9491.831644 secs 2024-04-26T14:29:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9494.605276 secs 2024-04-26T14:29:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9491.831644 secs 2024-04-26T14:29:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9494.605276 secs 2024-04-26T14:29:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9491.831644 secs 2024-04-26T14:29:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9494.605276 secs 2024-04-26T14:29:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:14 smithi194 ceph-mon[33017]: pgmap v5653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:14 smithi033 ceph-mon[27845]: pgmap v5653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:14 smithi096 ceph-mon[32753]: pgmap v5653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:16 smithi194 ceph-mon[33017]: pgmap v5654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:16 smithi033 ceph-mon[27845]: pgmap v5654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:16 smithi096 ceph-mon[32753]: pgmap v5654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9496.831728 secs 2024-04-26T14:29:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:18 smithi194 ceph-mon[33017]: pgmap v5655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9499.605388 secs 2024-04-26T14:29:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9496.831728 secs 2024-04-26T14:29:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:18 smithi033 ceph-mon[27845]: pgmap v5655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9499.605388 secs 2024-04-26T14:29:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9496.831728 secs 2024-04-26T14:29:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:18 smithi096 ceph-mon[32753]: pgmap v5655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9499.605388 secs 2024-04-26T14:29:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:20 smithi194 ceph-mon[33017]: pgmap v5656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:20 smithi033 ceph-mon[27845]: pgmap v5656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:20 smithi096 ceph-mon[32753]: pgmap v5656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:29:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:22 smithi194 ceph-mon[33017]: pgmap v5657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:22 smithi033 ceph-mon[27845]: pgmap v5657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:22 smithi096 ceph-mon[32753]: pgmap v5657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:23.408+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:29:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:23.408+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:29:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:23.409+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006610 con 0x564385821900 2024-04-26T14:29:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:23.409+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:29:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9501.831811 secs 2024-04-26T14:29:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9504.605492 secs 2024-04-26T14:29:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9501.831811 secs 2024-04-26T14:29:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9504.605492 secs 2024-04-26T14:29:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9501.831811 secs 2024-04-26T14:29:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9504.605492 secs 2024-04-26T14:29:24.681 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:24.681+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:24.681+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:29:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:24.681+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:25.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:24 smithi033 ceph-mon[27845]: pgmap v5658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:24 smithi194 ceph-mon[33017]: pgmap v5658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:24 smithi096 ceph-mon[32753]: pgmap v5658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:26 smithi194 ceph-mon[33017]: pgmap v5659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:26 smithi033 ceph-mon[27845]: pgmap v5659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:26 smithi096 ceph-mon[32753]: pgmap v5659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9506.831911 secs 2024-04-26T14:29:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:28 smithi194 ceph-mon[33017]: pgmap v5660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9509.605592 secs 2024-04-26T14:29:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9506.831911 secs 2024-04-26T14:29:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:28 smithi033 ceph-mon[27845]: pgmap v5660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9509.605592 secs 2024-04-26T14:29:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9506.831911 secs 2024-04-26T14:29:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:28 smithi096 ceph-mon[32753]: pgmap v5660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9509.605592 secs 2024-04-26T14:29:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:30 smithi194 ceph-mon[33017]: pgmap v5661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:30 smithi033 ceph-mon[27845]: pgmap v5661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:30 smithi096 ceph-mon[32753]: pgmap v5661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:32 smithi194 ceph-mon[33017]: pgmap v5662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:32 smithi033 ceph-mon[27845]: pgmap v5662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:32 smithi096 ceph-mon[32753]: pgmap v5662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9511.832007 secs 2024-04-26T14:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9514.605662 secs 2024-04-26T14:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9511.832007 secs 2024-04-26T14:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9514.605662 secs 2024-04-26T14:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9511.832007 secs 2024-04-26T14:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9514.605662 secs 2024-04-26T14:29:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:34 smithi194 ceph-mon[33017]: pgmap v5663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:34 smithi033 ceph-mon[27845]: pgmap v5663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:34 smithi096 ceph-mon[32753]: pgmap v5663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:36 smithi194 ceph-mon[33017]: pgmap v5664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:29:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:36 smithi033 ceph-mon[27845]: pgmap v5664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:29:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:36 smithi096 ceph-mon[32753]: pgmap v5664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:29:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:29:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:29:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:29:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:29:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:29:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:29:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:29:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:29:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:29:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:38 smithi194 ceph-mon[33017]: pgmap v5665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:38 smithi194 ceph-mon[33017]: pgmap v5666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9516.832071 secs 2024-04-26T14:29:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9519.605816 secs 2024-04-26T14:29:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:38 smithi033 ceph-mon[27845]: pgmap v5665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:38 smithi033 ceph-mon[27845]: pgmap v5666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9516.832071 secs 2024-04-26T14:29:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9519.605816 secs 2024-04-26T14:29:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:38 smithi096 ceph-mon[32753]: pgmap v5665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:38 smithi096 ceph-mon[32753]: pgmap v5666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9516.832071 secs 2024-04-26T14:29:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9519.605816 secs 2024-04-26T14:29:39.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:39.696+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:39.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:39.696+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:29:39.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:39.696+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:40 smithi194 ceph-mon[33017]: pgmap v5667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:40 smithi033 ceph-mon[27845]: pgmap v5667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:40 smithi096 ceph-mon[32753]: pgmap v5667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:29:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:42 smithi194 ceph-mon[33017]: pgmap v5668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:42 smithi033 ceph-mon[27845]: pgmap v5668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:42 smithi096 ceph-mon[32753]: pgmap v5668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9521.832156 secs 2024-04-26T14:29:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9524.605930 secs 2024-04-26T14:29:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9521.832156 secs 2024-04-26T14:29:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9524.605930 secs 2024-04-26T14:29:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9521.832156 secs 2024-04-26T14:29:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9524.605930 secs 2024-04-26T14:29:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:44 smithi194 ceph-mon[33017]: pgmap v5669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:44 smithi033 ceph-mon[27845]: pgmap v5669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:44 smithi096 ceph-mon[32753]: pgmap v5669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:46 smithi033 ceph-mon[27845]: pgmap v5670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:46 smithi096 ceph-mon[32753]: pgmap v5670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:46 smithi194 ceph-mon[33017]: pgmap v5670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:48 smithi033 ceph-mon[27845]: pgmap v5671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:29:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9526.832219 secs 2024-04-26T14:29:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9529.606051 secs 2024-04-26T14:29:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:48 smithi096 ceph-mon[32753]: pgmap v5671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:29:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9526.832219 secs 2024-04-26T14:29:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9529.606051 secs 2024-04-26T14:29:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:48 smithi194 ceph-mon[33017]: pgmap v5671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:29:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9526.832219 secs 2024-04-26T14:29:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9529.606051 secs 2024-04-26T14:29:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:50 smithi033 ceph-mon[27845]: pgmap v5672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:50 smithi096 ceph-mon[32753]: pgmap v5672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:50 smithi194 ceph-mon[33017]: pgmap v5672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:52 smithi033 ceph-mon[27845]: pgmap v5673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:52 smithi096 ceph-mon[32753]: pgmap v5673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:52 smithi194 ceph-mon[33017]: pgmap v5673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:29:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9531.832303 secs 2024-04-26T14:29:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9534.606170 secs 2024-04-26T14:29:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9531.832303 secs 2024-04-26T14:29:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9534.606170 secs 2024-04-26T14:29:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9531.832303 secs 2024-04-26T14:29:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9534.606170 secs 2024-04-26T14:29:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:54.711+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:54.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:54.711+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:29:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:29:54.711+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:29:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:54 smithi033 ceph-mon[27845]: pgmap v5674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:54 smithi096 ceph-mon[32753]: pgmap v5674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:54 smithi194 ceph-mon[33017]: pgmap v5674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:56 smithi033 ceph-mon[27845]: pgmap v5675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:56 smithi096 ceph-mon[32753]: pgmap v5675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:56 smithi194 ceph-mon[33017]: pgmap v5675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:58 smithi033 ceph-mon[27845]: pgmap v5676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9536.832385 secs 2024-04-26T14:29:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:29:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9539.606274 secs 2024-04-26T14:29:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:58 smithi096 ceph-mon[32753]: pgmap v5676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9536.832385 secs 2024-04-26T14:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:29:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9539.606274 secs 2024-04-26T14:29:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:58 smithi194 ceph-mon[33017]: pgmap v5676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:29:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9536.832385 secs 2024-04-26T14:29:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:29:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9539.606274 secs 2024-04-26T14:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:00 smithi033 ceph-mon[27845]: pgmap v5677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:30:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:00 smithi096 ceph-mon[32753]: pgmap v5677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:30:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:00 smithi194 ceph-mon[33017]: pgmap v5677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:30:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:02 smithi033 ceph-mon[27845]: pgmap v5678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:02 smithi096 ceph-mon[32753]: pgmap v5678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:02 smithi194 ceph-mon[33017]: pgmap v5678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9541.832469 secs 2024-04-26T14:30:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9544.606371 secs 2024-04-26T14:30:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9541.832469 secs 2024-04-26T14:30:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9544.606371 secs 2024-04-26T14:30:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9541.832469 secs 2024-04-26T14:30:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9544.606371 secs 2024-04-26T14:30:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:04 smithi033 ceph-mon[27845]: pgmap v5679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:04 smithi096 ceph-mon[32753]: pgmap v5679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:04 smithi194 ceph-mon[33017]: pgmap v5679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:06 smithi033 ceph-mon[27845]: pgmap v5680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:06 smithi096 ceph-mon[32753]: pgmap v5680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:06 smithi194 ceph-mon[33017]: pgmap v5680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:08 smithi096 ceph-mon[32753]: pgmap v5681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9546.832528 secs 2024-04-26T14:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9549.606489 secs 2024-04-26T14:30:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:08 smithi194 ceph-mon[33017]: pgmap v5681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9546.832528 secs 2024-04-26T14:30:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9549.606489 secs 2024-04-26T14:30:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:08 smithi033 ceph-mon[27845]: pgmap v5681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9546.832528 secs 2024-04-26T14:30:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9549.606489 secs 2024-04-26T14:30:09.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:09.727+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:09.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:09.727+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:30:09.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:09.727+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:10 smithi194 ceph-mon[33017]: pgmap v5682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:10 smithi033 ceph-mon[27845]: pgmap v5682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:10 smithi096 ceph-mon[32753]: pgmap v5682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:12 smithi194 ceph-mon[33017]: pgmap v5683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:12 smithi033 ceph-mon[27845]: pgmap v5683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:12 smithi096 ceph-mon[32753]: pgmap v5683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9551.832614 secs 2024-04-26T14:30:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9554.606632 secs 2024-04-26T14:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9551.832614 secs 2024-04-26T14:30:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9554.606632 secs 2024-04-26T14:30:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9551.832614 secs 2024-04-26T14:30:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9554.606632 secs 2024-04-26T14:30:15.210 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:14 smithi096 ceph-mon[32753]: pgmap v5684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:14 smithi194 ceph-mon[33017]: pgmap v5684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:14 smithi033 ceph-mon[27845]: pgmap v5684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:16 smithi194 ceph-mon[33017]: pgmap v5685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:16 smithi033 ceph-mon[27845]: pgmap v5685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:16 smithi096 ceph-mon[32753]: pgmap v5685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:18 smithi194 ceph-mon[33017]: pgmap v5686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9556.832699 secs 2024-04-26T14:30:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9559.606750 secs 2024-04-26T14:30:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:18 smithi033 ceph-mon[27845]: pgmap v5686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9556.832699 secs 2024-04-26T14:30:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9559.606750 secs 2024-04-26T14:30:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:18 smithi096 ceph-mon[32753]: pgmap v5686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9556.832699 secs 2024-04-26T14:30:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9559.606750 secs 2024-04-26T14:30:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:20 smithi194 ceph-mon[33017]: pgmap v5687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:20 smithi033 ceph-mon[27845]: pgmap v5687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:20 smithi096 ceph-mon[32753]: pgmap v5687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:22 smithi194 ceph-mon[33017]: pgmap v5688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:22 smithi033 ceph-mon[27845]: pgmap v5688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:22 smithi096 ceph-mon[32753]: pgmap v5688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9561.832757 secs 2024-04-26T14:30:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9564.606860 secs 2024-04-26T14:30:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9561.832757 secs 2024-04-26T14:30:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9564.606860 secs 2024-04-26T14:30:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9561.832757 secs 2024-04-26T14:30:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9564.606860 secs 2024-04-26T14:30:24.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:24.741+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:24.741+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:30:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:24.741+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:25.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:24 smithi033 ceph-mon[27845]: pgmap v5689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:24 smithi194 ceph-mon[33017]: pgmap v5689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:24 smithi096 ceph-mon[32753]: pgmap v5689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:27 smithi194 ceph-mon[33017]: pgmap v5690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:27 smithi033 ceph-mon[27845]: pgmap v5690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:27 smithi096 ceph-mon[32753]: pgmap v5690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:29 smithi194 ceph-mon[33017]: pgmap v5691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9566.832841 secs 2024-04-26T14:30:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9569.606989 secs 2024-04-26T14:30:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:29 smithi033 ceph-mon[27845]: pgmap v5691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9566.832841 secs 2024-04-26T14:30:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9569.606989 secs 2024-04-26T14:30:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:29 smithi096 ceph-mon[32753]: pgmap v5691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9566.832841 secs 2024-04-26T14:30:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9569.606989 secs 2024-04-26T14:30:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:31 smithi194 ceph-mon[33017]: pgmap v5692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:31 smithi033 ceph-mon[27845]: pgmap v5692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:31 smithi096 ceph-mon[32753]: pgmap v5692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:33 smithi194 ceph-mon[33017]: pgmap v5693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:33 smithi033 ceph-mon[27845]: pgmap v5693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:33 smithi096 ceph-mon[32753]: pgmap v5693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9571.832926 secs 2024-04-26T14:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9574.607121 secs 2024-04-26T14:30:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9571.832926 secs 2024-04-26T14:30:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9574.607121 secs 2024-04-26T14:30:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9571.832926 secs 2024-04-26T14:30:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9574.607121 secs 2024-04-26T14:30:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:35 smithi194 ceph-mon[33017]: pgmap v5694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:35 smithi033 ceph-mon[27845]: pgmap v5694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:35 smithi096 ceph-mon[32753]: pgmap v5694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:37 smithi194 ceph-mon[33017]: pgmap v5695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:37 smithi033 ceph-mon[27845]: pgmap v5695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:37 smithi096 ceph-mon[32753]: pgmap v5695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:30:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:30:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:39 smithi194 ceph-mon[33017]: pgmap v5696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9576.833010 secs 2024-04-26T14:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9579.607195 secs 2024-04-26T14:30:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:39 smithi033 ceph-mon[27845]: pgmap v5696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9576.833010 secs 2024-04-26T14:30:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9579.607195 secs 2024-04-26T14:30:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:39 smithi096 ceph-mon[32753]: pgmap v5696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9576.833010 secs 2024-04-26T14:30:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9579.607195 secs 2024-04-26T14:30:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:39.755+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:39.755+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:30:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:39.755+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:40.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:30:40.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:30:40.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:40 smithi096 ceph-mon[32753]: pgmap v5697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:40.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:40 smithi096 ceph-mon[32753]: pgmap v5698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:30:40.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:40 smithi194 ceph-mon[33017]: pgmap v5697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:40 smithi194 ceph-mon[33017]: pgmap v5698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:30:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:30:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:30:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:40 smithi033 ceph-mon[27845]: pgmap v5697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:40 smithi033 ceph-mon[27845]: pgmap v5698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:30:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:30:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:41 smithi033 ceph-mon[27845]: pgmap v5699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:41 smithi096 ceph-mon[32753]: pgmap v5699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:41 smithi194 ceph-mon[33017]: pgmap v5699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:43 smithi033 ceph-mon[27845]: pgmap v5700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:43 smithi096 ceph-mon[32753]: pgmap v5700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:43 smithi194 ceph-mon[33017]: pgmap v5700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9581.833028 secs 2024-04-26T14:30:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9584.607263 secs 2024-04-26T14:30:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9581.833028 secs 2024-04-26T14:30:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9584.607263 secs 2024-04-26T14:30:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9581.833028 secs 2024-04-26T14:30:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9584.607263 secs 2024-04-26T14:30:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:45 smithi033 ceph-mon[27845]: pgmap v5701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:45.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:45 smithi096 ceph-mon[32753]: pgmap v5701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:45 smithi194 ceph-mon[33017]: pgmap v5701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:47 smithi033 ceph-mon[27845]: pgmap v5702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:47 smithi096 ceph-mon[32753]: pgmap v5702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:47 smithi194 ceph-mon[33017]: pgmap v5702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:30:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9586.833089 secs 2024-04-26T14:30:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9589.607379 secs 2024-04-26T14:30:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9586.833089 secs 2024-04-26T14:30:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9589.607379 secs 2024-04-26T14:30:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9586.833089 secs 2024-04-26T14:30:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9589.607379 secs 2024-04-26T14:30:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:49 smithi194 ceph-mon[33017]: pgmap v5703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:30:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:49 smithi033 ceph-mon[27845]: pgmap v5703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:30:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:49 smithi096 ceph-mon[32753]: pgmap v5703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:30:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:51 smithi033 ceph-mon[27845]: pgmap v5704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:51 smithi096 ceph-mon[32753]: pgmap v5704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:51 smithi194 ceph-mon[33017]: pgmap v5704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:53.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:53 smithi194 ceph-mon[33017]: pgmap v5705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:53 smithi033 ceph-mon[27845]: pgmap v5705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:53 smithi096 ceph-mon[32753]: pgmap v5705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9591.833169 secs 2024-04-26T14:30:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9594.607478 secs 2024-04-26T14:30:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9591.833169 secs 2024-04-26T14:30:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9594.607478 secs 2024-04-26T14:30:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9591.833169 secs 2024-04-26T14:30:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9594.607478 secs 2024-04-26T14:30:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:54.764+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:54.764+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:30:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:30:54.764+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:30:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:55 smithi033 ceph-mon[27845]: pgmap v5706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:55 smithi096 ceph-mon[32753]: pgmap v5706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:55 smithi194 ceph-mon[33017]: pgmap v5706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:30:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:57 smithi033 ceph-mon[27845]: pgmap v5707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:57 smithi096 ceph-mon[32753]: pgmap v5707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:57 smithi194 ceph-mon[33017]: pgmap v5707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9596.833251 secs 2024-04-26T14:30:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9599.607622 secs 2024-04-26T14:30:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9596.833251 secs 2024-04-26T14:30:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9599.607622 secs 2024-04-26T14:30:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9596.833251 secs 2024-04-26T14:30:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9599.607622 secs 2024-04-26T14:30:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:30:59 smithi194 ceph-mon[33017]: pgmap v5708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:30:59 smithi033 ceph-mon[27845]: pgmap v5708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:30:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:30:59 smithi096 ceph-mon[32753]: pgmap v5708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:01 smithi033 ceph-mon[27845]: pgmap v5709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:01 smithi096 ceph-mon[32753]: pgmap v5709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:01 smithi194 ceph-mon[33017]: pgmap v5709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:03 smithi033 ceph-mon[27845]: pgmap v5710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:03 smithi096 ceph-mon[32753]: pgmap v5710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:03 smithi194 ceph-mon[33017]: pgmap v5710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9601.833333 secs 2024-04-26T14:31:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9604.607751 secs 2024-04-26T14:31:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9601.833333 secs 2024-04-26T14:31:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9604.607751 secs 2024-04-26T14:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9601.833333 secs 2024-04-26T14:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9604.607751 secs 2024-04-26T14:31:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:05 smithi033 ceph-mon[27845]: pgmap v5711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:05 smithi096 ceph-mon[32753]: pgmap v5711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:05 smithi194 ceph-mon[33017]: pgmap v5711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:07 smithi033 ceph-mon[27845]: pgmap v5712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:07 smithi096 ceph-mon[32753]: pgmap v5712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:07 smithi194 ceph-mon[33017]: pgmap v5712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9606.833418 secs 2024-04-26T14:31:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9609.607872 secs 2024-04-26T14:31:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9606.833418 secs 2024-04-26T14:31:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9609.607872 secs 2024-04-26T14:31:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9606.833418 secs 2024-04-26T14:31:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9609.607872 secs 2024-04-26T14:31:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:09.780+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:09.780+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:31:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:09.780+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:09 smithi194 ceph-mon[33017]: pgmap v5713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:09 smithi033 ceph-mon[27845]: pgmap v5713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:09 smithi096 ceph-mon[32753]: pgmap v5713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:11 smithi033 ceph-mon[27845]: pgmap v5714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:11 smithi096 ceph-mon[32753]: pgmap v5714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:11 smithi194 ceph-mon[33017]: pgmap v5714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:13 smithi033 ceph-mon[27845]: pgmap v5715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:13 smithi096 ceph-mon[32753]: pgmap v5715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:13 smithi194 ceph-mon[33017]: pgmap v5715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9611.833504 secs 2024-04-26T14:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9614.607997 secs 2024-04-26T14:31:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9611.833504 secs 2024-04-26T14:31:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9614.607997 secs 2024-04-26T14:31:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9611.833504 secs 2024-04-26T14:31:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9614.607997 secs 2024-04-26T14:31:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:15 smithi033 ceph-mon[27845]: pgmap v5716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:15 smithi096 ceph-mon[32753]: pgmap v5716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:15 smithi194 ceph-mon[33017]: pgmap v5716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:17 smithi033 ceph-mon[27845]: pgmap v5717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:17 smithi096 ceph-mon[32753]: pgmap v5717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:17 smithi194 ceph-mon[33017]: pgmap v5717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9616.833590 secs 2024-04-26T14:31:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9619.608072 secs 2024-04-26T14:31:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9616.833590 secs 2024-04-26T14:31:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9619.608072 secs 2024-04-26T14:31:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9616.833590 secs 2024-04-26T14:31:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9619.608072 secs 2024-04-26T14:31:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:19 smithi194 ceph-mon[33017]: pgmap v5718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:19 smithi033 ceph-mon[27845]: pgmap v5718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:19 smithi096 ceph-mon[32753]: pgmap v5718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:21 smithi033 ceph-mon[27845]: pgmap v5719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:21 smithi096 ceph-mon[32753]: pgmap v5719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:21 smithi194 ceph-mon[33017]: pgmap v5719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:23 smithi033 ceph-mon[27845]: pgmap v5720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:23 smithi096 ceph-mon[32753]: pgmap v5720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:23 smithi194 ceph-mon[33017]: pgmap v5720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9621.833649 secs 2024-04-26T14:31:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9624.608200 secs 2024-04-26T14:31:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9621.833649 secs 2024-04-26T14:31:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9624.608200 secs 2024-04-26T14:31:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9621.833649 secs 2024-04-26T14:31:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9624.608200 secs 2024-04-26T14:31:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:24.788+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:24.788+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:31:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:24.788+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:25 smithi096 ceph-mon[32753]: pgmap v5721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:25 smithi194 ceph-mon[33017]: pgmap v5721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:25 smithi033 ceph-mon[27845]: pgmap v5721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:27 smithi033 ceph-mon[27845]: pgmap v5722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:27 smithi096 ceph-mon[32753]: pgmap v5722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:27 smithi194 ceph-mon[33017]: pgmap v5722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9626.833735 secs 2024-04-26T14:31:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9629.608329 secs 2024-04-26T14:31:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9626.833735 secs 2024-04-26T14:31:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9629.608329 secs 2024-04-26T14:31:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9626.833735 secs 2024-04-26T14:31:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9629.608329 secs 2024-04-26T14:31:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:29 smithi194 ceph-mon[33017]: pgmap v5723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:29 smithi033 ceph-mon[27845]: pgmap v5723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:29 smithi096 ceph-mon[32753]: pgmap v5723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:31 smithi033 ceph-mon[27845]: pgmap v5724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:31 smithi096 ceph-mon[32753]: pgmap v5724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:31 smithi194 ceph-mon[33017]: pgmap v5724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:33 smithi033 ceph-mon[27845]: pgmap v5725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:33.443 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:33 smithi096 ceph-mon[32753]: pgmap v5725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:33 smithi194 ceph-mon[33017]: pgmap v5725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9631.833819 secs 2024-04-26T14:31:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9634.608461 secs 2024-04-26T14:31:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9631.833819 secs 2024-04-26T14:31:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9634.608461 secs 2024-04-26T14:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9631.833819 secs 2024-04-26T14:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9634.608461 secs 2024-04-26T14:31:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:35 smithi096 ceph-mon[32753]: pgmap v5726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:35 smithi194 ceph-mon[33017]: pgmap v5726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:35 smithi033 ceph-mon[27845]: pgmap v5726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:37 smithi033 ceph-mon[27845]: pgmap v5727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:37 smithi096 ceph-mon[32753]: pgmap v5727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:37 smithi194 ceph-mon[33017]: pgmap v5727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9636.833903 secs 2024-04-26T14:31:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9639.608576 secs 2024-04-26T14:31:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9636.833903 secs 2024-04-26T14:31:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9639.608576 secs 2024-04-26T14:31:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9636.833903 secs 2024-04-26T14:31:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9639.608576 secs 2024-04-26T14:31:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:39.802+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:39.802+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:31:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:39.803+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:39 smithi194 ceph-mon[33017]: pgmap v5728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:31:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:39 smithi033 ceph-mon[27845]: pgmap v5728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:31:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:39 smithi096 ceph-mon[32753]: pgmap v5728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:31:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: pgmap v5729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:31:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:31:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: pgmap v5729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:31:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: pgmap v5729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:31:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:31:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:31:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:42 smithi194 ceph-mon[33017]: pgmap v5730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:31:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:42 smithi194 ceph-mon[33017]: pgmap v5731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:42 smithi033 ceph-mon[27845]: pgmap v5730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:31:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:42 smithi033 ceph-mon[27845]: pgmap v5731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:42 smithi096 ceph-mon[32753]: pgmap v5730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:31:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:42 smithi096 ceph-mon[32753]: pgmap v5731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9641.833999 secs 2024-04-26T14:31:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9644.608717 secs 2024-04-26T14:31:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9641.833999 secs 2024-04-26T14:31:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9644.608717 secs 2024-04-26T14:31:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9641.833999 secs 2024-04-26T14:31:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9644.608717 secs 2024-04-26T14:31:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:44 smithi194 ceph-mon[33017]: pgmap v5732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:31:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:44 smithi033 ceph-mon[27845]: pgmap v5732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:31:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:44 smithi096 ceph-mon[32753]: pgmap v5732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:31:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:46 smithi194 ceph-mon[33017]: pgmap v5733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:46 smithi033 ceph-mon[27845]: pgmap v5733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:46 smithi096 ceph-mon[32753]: pgmap v5733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9646.834060 secs 2024-04-26T14:31:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:48 smithi194 ceph-mon[33017]: pgmap v5734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9649.608861 secs 2024-04-26T14:31:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9646.834060 secs 2024-04-26T14:31:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:48 smithi033 ceph-mon[27845]: pgmap v5734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9649.608861 secs 2024-04-26T14:31:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9646.834060 secs 2024-04-26T14:31:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:48 smithi096 ceph-mon[32753]: pgmap v5734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9649.608861 secs 2024-04-26T14:31:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:50 smithi194 ceph-mon[33017]: pgmap v5735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:50 smithi033 ceph-mon[27845]: pgmap v5735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:50 smithi096 ceph-mon[32753]: pgmap v5735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:31:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:52 smithi194 ceph-mon[33017]: pgmap v5736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:31:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:52 smithi033 ceph-mon[27845]: pgmap v5736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:31:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:52 smithi096 ceph-mon[32753]: pgmap v5736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:31:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9651.834147 secs 2024-04-26T14:31:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9654.608985 secs 2024-04-26T14:31:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9651.834147 secs 2024-04-26T14:31:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9654.608985 secs 2024-04-26T14:31:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9651.834147 secs 2024-04-26T14:31:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9654.608985 secs 2024-04-26T14:31:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:54.809+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:54.809+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:31:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:31:54.809+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:31:55.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:54 smithi033 ceph-mon[27845]: pgmap v5737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:54 smithi194 ceph-mon[33017]: pgmap v5737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:54 smithi096 ceph-mon[32753]: pgmap v5737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:56 smithi194 ceph-mon[33017]: pgmap v5738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:56 smithi033 ceph-mon[27845]: pgmap v5738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:56 smithi096 ceph-mon[32753]: pgmap v5738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:31:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9656.834236 secs 2024-04-26T14:31:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:58 smithi194 ceph-mon[33017]: pgmap v5739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:31:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9659.609129 secs 2024-04-26T14:31:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9656.834236 secs 2024-04-26T14:31:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:58 smithi033 ceph-mon[27845]: pgmap v5739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:31:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9659.609129 secs 2024-04-26T14:31:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9656.834236 secs 2024-04-26T14:31:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:58 smithi096 ceph-mon[32753]: pgmap v5739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:31:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:31:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9659.609129 secs 2024-04-26T14:32:01.071 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:00 smithi033 ceph-mon[27845]: pgmap v5740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:00 smithi194 ceph-mon[33017]: pgmap v5740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:00 smithi096 ceph-mon[32753]: pgmap v5740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:02 smithi194 ceph-mon[33017]: pgmap v5741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:02 smithi033 ceph-mon[27845]: pgmap v5741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:02 smithi096 ceph-mon[32753]: pgmap v5741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9661.834321 secs 2024-04-26T14:32:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9664.609226 secs 2024-04-26T14:32:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9661.834321 secs 2024-04-26T14:32:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9664.609226 secs 2024-04-26T14:32:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9661.834321 secs 2024-04-26T14:32:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9664.609226 secs 2024-04-26T14:32:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:04 smithi194 ceph-mon[33017]: pgmap v5742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:04 smithi033 ceph-mon[27845]: pgmap v5742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:04 smithi096 ceph-mon[32753]: pgmap v5742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:06 smithi194 ceph-mon[33017]: pgmap v5743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:06 smithi033 ceph-mon[27845]: pgmap v5743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:06 smithi096 ceph-mon[32753]: pgmap v5743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9666.834405 secs 2024-04-26T14:32:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:08 smithi194 ceph-mon[33017]: pgmap v5744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9669.609356 secs 2024-04-26T14:32:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9666.834405 secs 2024-04-26T14:32:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:08 smithi033 ceph-mon[27845]: pgmap v5744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9669.609356 secs 2024-04-26T14:32:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9666.834405 secs 2024-04-26T14:32:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:08 smithi096 ceph-mon[32753]: pgmap v5744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9669.609356 secs 2024-04-26T14:32:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:09.825+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:09.825+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:32:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:09.825+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:10 smithi194 ceph-mon[33017]: pgmap v5745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:10 smithi033 ceph-mon[27845]: pgmap v5745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:10 smithi096 ceph-mon[32753]: pgmap v5745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:12 smithi194 ceph-mon[33017]: pgmap v5746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:12 smithi033 ceph-mon[27845]: pgmap v5746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:12 smithi096 ceph-mon[32753]: pgmap v5746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9671.834483 secs 2024-04-26T14:32:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9674.609476 secs 2024-04-26T14:32:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9671.834483 secs 2024-04-26T14:32:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9674.609476 secs 2024-04-26T14:32:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9671.834483 secs 2024-04-26T14:32:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9674.609476 secs 2024-04-26T14:32:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:14 smithi194 ceph-mon[33017]: pgmap v5747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:14 smithi033 ceph-mon[27845]: pgmap v5747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:14 smithi096 ceph-mon[32753]: pgmap v5747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:16 smithi194 ceph-mon[33017]: pgmap v5748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:16 smithi033 ceph-mon[27845]: pgmap v5748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:16 smithi096 ceph-mon[32753]: pgmap v5748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9676.834564 secs 2024-04-26T14:32:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:18 smithi194 ceph-mon[33017]: pgmap v5749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9679.609596 secs 2024-04-26T14:32:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9676.834564 secs 2024-04-26T14:32:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:18 smithi033 ceph-mon[27845]: pgmap v5749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9679.609596 secs 2024-04-26T14:32:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9676.834564 secs 2024-04-26T14:32:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:18 smithi096 ceph-mon[32753]: pgmap v5749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9679.609596 secs 2024-04-26T14:32:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:20 smithi194 ceph-mon[33017]: pgmap v5750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:20 smithi033 ceph-mon[27845]: pgmap v5750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:20 smithi096 ceph-mon[32753]: pgmap v5750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:22 smithi194 ceph-mon[33017]: pgmap v5751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:22 smithi033 ceph-mon[27845]: pgmap v5751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:22 smithi096 ceph-mon[32753]: pgmap v5751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:23.438+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:32:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:23.438+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:32:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:23.438+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:32:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:23.438+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x7fef00077780 2024-04-26T14:32:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9681.834648 secs 2024-04-26T14:32:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9684.609716 secs 2024-04-26T14:32:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9681.834648 secs 2024-04-26T14:32:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9684.609716 secs 2024-04-26T14:32:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9681.834648 secs 2024-04-26T14:32:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9684.609716 secs 2024-04-26T14:32:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:24.840+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:24.840+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:24.840+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:32:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:24 smithi194 ceph-mon[33017]: pgmap v5752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:24 smithi033 ceph-mon[27845]: pgmap v5752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:24 smithi096 ceph-mon[32753]: pgmap v5752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:26 smithi194 ceph-mon[33017]: pgmap v5753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:26 smithi033 ceph-mon[27845]: pgmap v5753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:26 smithi096 ceph-mon[32753]: pgmap v5753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9686.834734 secs 2024-04-26T14:32:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:28 smithi194 ceph-mon[33017]: pgmap v5754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9689.609848 secs 2024-04-26T14:32:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9686.834734 secs 2024-04-26T14:32:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:28 smithi033 ceph-mon[27845]: pgmap v5754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9689.609848 secs 2024-04-26T14:32:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9686.834734 secs 2024-04-26T14:32:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:28 smithi096 ceph-mon[32753]: pgmap v5754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9689.609848 secs 2024-04-26T14:32:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:30 smithi033 ceph-mon[27845]: pgmap v5755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:30 smithi096 ceph-mon[32753]: pgmap v5755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:30 smithi194 ceph-mon[33017]: pgmap v5755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:32 smithi033 ceph-mon[27845]: pgmap v5756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:32 smithi096 ceph-mon[32753]: pgmap v5756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:32 smithi194 ceph-mon[33017]: pgmap v5756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9691.834823 secs 2024-04-26T14:32:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9694.609986 secs 2024-04-26T14:32:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9691.834823 secs 2024-04-26T14:32:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9694.609986 secs 2024-04-26T14:32:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9691.834823 secs 2024-04-26T14:32:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9694.609986 secs 2024-04-26T14:32:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:34 smithi033 ceph-mon[27845]: pgmap v5757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:34 smithi096 ceph-mon[32753]: pgmap v5757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:34 smithi194 ceph-mon[33017]: pgmap v5757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:36 smithi033 ceph-mon[27845]: pgmap v5758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:36 smithi096 ceph-mon[32753]: pgmap v5758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:36 smithi194 ceph-mon[33017]: pgmap v5758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9696.834907 secs 2024-04-26T14:32:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:38 smithi033 ceph-mon[27845]: pgmap v5759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9699.610118 secs 2024-04-26T14:32:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9696.834907 secs 2024-04-26T14:32:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:38 smithi096 ceph-mon[32753]: pgmap v5759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9699.610118 secs 2024-04-26T14:32:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9696.834907 secs 2024-04-26T14:32:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:38 smithi194 ceph-mon[33017]: pgmap v5759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9699.610118 secs 2024-04-26T14:32:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:39.849+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:39.850+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:32:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:39.850+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:40 smithi033 ceph-mon[27845]: pgmap v5760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:40 smithi096 ceph-mon[32753]: pgmap v5760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:40 smithi194 ceph-mon[33017]: pgmap v5760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:43.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:42 smithi194 ceph-mon[33017]: pgmap v5761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:43.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:32:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:42 smithi033 ceph-mon[27845]: pgmap v5761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:32:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:42 smithi096 ceph-mon[32753]: pgmap v5761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:32:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9701.835001 secs 2024-04-26T14:32:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9704.610245 secs 2024-04-26T14:32:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:32:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:32:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:32:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9701.835001 secs 2024-04-26T14:32:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9704.610245 secs 2024-04-26T14:32:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:32:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:32:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:32:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9701.835001 secs 2024-04-26T14:32:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9704.610245 secs 2024-04-26T14:32:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:32:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:32:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:32:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:44 smithi033 ceph-mon[27845]: pgmap v5762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:44 smithi033 ceph-mon[27845]: pgmap v5763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:44 smithi096 ceph-mon[32753]: pgmap v5762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:44 smithi096 ceph-mon[32753]: pgmap v5763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:44 smithi194 ceph-mon[33017]: pgmap v5762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:45.428 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:44 smithi194 ceph-mon[33017]: pgmap v5763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:46 smithi033 ceph-mon[27845]: pgmap v5764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:46 smithi096 ceph-mon[32753]: pgmap v5764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:46 smithi194 ceph-mon[33017]: pgmap v5764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:48 smithi033 ceph-mon[27845]: pgmap v5765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9706.835061 secs 2024-04-26T14:32:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9709.610376 secs 2024-04-26T14:32:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:48 smithi096 ceph-mon[32753]: pgmap v5765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9706.835061 secs 2024-04-26T14:32:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9709.610376 secs 2024-04-26T14:32:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:48 smithi194 ceph-mon[33017]: pgmap v5765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9706.835061 secs 2024-04-26T14:32:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9709.610376 secs 2024-04-26T14:32:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:50 smithi033 ceph-mon[27845]: pgmap v5766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:50 smithi096 ceph-mon[32753]: pgmap v5766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:50 smithi194 ceph-mon[33017]: pgmap v5766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:52 smithi096 ceph-mon[32753]: pgmap v5767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:52 smithi194 ceph-mon[33017]: pgmap v5767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:52 smithi033 ceph-mon[27845]: pgmap v5767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:32:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9711.835123 secs 2024-04-26T14:32:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9714.610525 secs 2024-04-26T14:32:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9711.835123 secs 2024-04-26T14:32:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9714.610525 secs 2024-04-26T14:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9711.835123 secs 2024-04-26T14:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9714.610525 secs 2024-04-26T14:32:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:54.864+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:54.864+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:32:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:32:54.864+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:32:55.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:54 smithi033 ceph-mon[27845]: pgmap v5768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:32:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:54 smithi194 ceph-mon[33017]: pgmap v5768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:32:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:54 smithi096 ceph-mon[32753]: pgmap v5768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:32:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:56 smithi194 ceph-mon[33017]: pgmap v5769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:56 smithi033 ceph-mon[27845]: pgmap v5769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:56 smithi096 ceph-mon[32753]: pgmap v5769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:32:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:58 smithi194 ceph-mon[33017]: pgmap v5770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9716.835183 secs 2024-04-26T14:32:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:32:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9719.610639 secs 2024-04-26T14:32:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:58 smithi033 ceph-mon[27845]: pgmap v5770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9716.835183 secs 2024-04-26T14:32:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:32:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9719.610639 secs 2024-04-26T14:32:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:58 smithi096 ceph-mon[32753]: pgmap v5770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9716.835183 secs 2024-04-26T14:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:32:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9719.610639 secs 2024-04-26T14:33:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:00 smithi194 ceph-mon[33017]: pgmap v5771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:00 smithi033 ceph-mon[27845]: pgmap v5771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:00 smithi096 ceph-mon[32753]: pgmap v5771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:03.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:02 smithi033 ceph-mon[27845]: pgmap v5772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:02 smithi194 ceph-mon[33017]: pgmap v5772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:02 smithi096 ceph-mon[32753]: pgmap v5772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9721.835269 secs 2024-04-26T14:33:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9724.610768 secs 2024-04-26T14:33:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9721.835269 secs 2024-04-26T14:33:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9724.610768 secs 2024-04-26T14:33:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9721.835269 secs 2024-04-26T14:33:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9724.610768 secs 2024-04-26T14:33:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:04 smithi194 ceph-mon[33017]: pgmap v5773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:04 smithi033 ceph-mon[27845]: pgmap v5773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:04 smithi096 ceph-mon[32753]: pgmap v5773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:06 smithi194 ceph-mon[33017]: pgmap v5774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:06 smithi033 ceph-mon[27845]: pgmap v5774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:06 smithi096 ceph-mon[32753]: pgmap v5774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:08 smithi194 ceph-mon[33017]: pgmap v5775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9726.835348 secs 2024-04-26T14:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9729.610894 secs 2024-04-26T14:33:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:08 smithi033 ceph-mon[27845]: pgmap v5775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9726.835348 secs 2024-04-26T14:33:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9729.610894 secs 2024-04-26T14:33:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:08 smithi096 ceph-mon[32753]: pgmap v5775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9726.835348 secs 2024-04-26T14:33:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9729.610894 secs 2024-04-26T14:33:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:09.880+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:09.881+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:09.881+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:33:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:10 smithi194 ceph-mon[33017]: pgmap v5776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:10 smithi033 ceph-mon[27845]: pgmap v5776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:10 smithi096 ceph-mon[32753]: pgmap v5776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:12 smithi194 ceph-mon[33017]: pgmap v5777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:12 smithi033 ceph-mon[27845]: pgmap v5777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:12 smithi096 ceph-mon[32753]: pgmap v5777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9731.835429 secs 2024-04-26T14:33:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9734.611012 secs 2024-04-26T14:33:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9731.835429 secs 2024-04-26T14:33:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9734.611012 secs 2024-04-26T14:33:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9731.835429 secs 2024-04-26T14:33:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9734.611012 secs 2024-04-26T14:33:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:15 smithi194 ceph-mon[33017]: pgmap v5778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:15 smithi033 ceph-mon[27845]: pgmap v5778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:15 smithi096 ceph-mon[32753]: pgmap v5778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:17 smithi194 ceph-mon[33017]: pgmap v5779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:17 smithi033 ceph-mon[27845]: pgmap v5779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:17 smithi096 ceph-mon[32753]: pgmap v5779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:19 smithi194 ceph-mon[33017]: pgmap v5780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9736.835512 secs 2024-04-26T14:33:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9739.611148 secs 2024-04-26T14:33:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:19 smithi033 ceph-mon[27845]: pgmap v5780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9736.835512 secs 2024-04-26T14:33:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9739.611148 secs 2024-04-26T14:33:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:19 smithi096 ceph-mon[32753]: pgmap v5780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9736.835512 secs 2024-04-26T14:33:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9739.611148 secs 2024-04-26T14:33:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:21 smithi194 ceph-mon[33017]: pgmap v5781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:21 smithi033 ceph-mon[27845]: pgmap v5781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:21 smithi096 ceph-mon[32753]: pgmap v5781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:23 smithi194 ceph-mon[33017]: pgmap v5782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:23 smithi033 ceph-mon[27845]: pgmap v5782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:23 smithi096 ceph-mon[32753]: pgmap v5782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9741.835593 secs 2024-04-26T14:33:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9744.611266 secs 2024-04-26T14:33:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9741.835593 secs 2024-04-26T14:33:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9744.611266 secs 2024-04-26T14:33:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9741.835593 secs 2024-04-26T14:33:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9744.611266 secs 2024-04-26T14:33:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:24.896+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:24.896+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:33:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:24.896+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:25.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:25 smithi033 ceph-mon[27845]: pgmap v5783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:25 smithi194 ceph-mon[33017]: pgmap v5783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:25 smithi096 ceph-mon[32753]: pgmap v5783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:27 smithi194 ceph-mon[33017]: pgmap v5784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:27 smithi033 ceph-mon[27845]: pgmap v5784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:27 smithi096 ceph-mon[32753]: pgmap v5784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:29 smithi194 ceph-mon[33017]: pgmap v5785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9746.835676 secs 2024-04-26T14:33:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9749.611383 secs 2024-04-26T14:33:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:29 smithi033 ceph-mon[27845]: pgmap v5785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9746.835676 secs 2024-04-26T14:33:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9749.611383 secs 2024-04-26T14:33:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:29 smithi096 ceph-mon[32753]: pgmap v5785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9746.835676 secs 2024-04-26T14:33:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9749.611383 secs 2024-04-26T14:33:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:31 smithi194 ceph-mon[33017]: pgmap v5786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:31 smithi033 ceph-mon[27845]: pgmap v5786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:31 smithi096 ceph-mon[32753]: pgmap v5786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:33 smithi194 ceph-mon[33017]: pgmap v5787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:33 smithi033 ceph-mon[27845]: pgmap v5787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:33 smithi096 ceph-mon[32753]: pgmap v5787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9751.835759 secs 2024-04-26T14:33:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9754.611510 secs 2024-04-26T14:33:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9751.835759 secs 2024-04-26T14:33:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9754.611510 secs 2024-04-26T14:33:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9751.835759 secs 2024-04-26T14:33:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9754.611510 secs 2024-04-26T14:33:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:35 smithi194 ceph-mon[33017]: pgmap v5788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:35 smithi033 ceph-mon[27845]: pgmap v5788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:35 smithi096 ceph-mon[32753]: pgmap v5788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:37 smithi194 ceph-mon[33017]: pgmap v5789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:37 smithi033 ceph-mon[27845]: pgmap v5789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:37 smithi096 ceph-mon[32753]: pgmap v5789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:39 smithi194 ceph-mon[33017]: pgmap v5790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9756.835819 secs 2024-04-26T14:33:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9759.611663 secs 2024-04-26T14:33:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:39 smithi033 ceph-mon[27845]: pgmap v5790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9756.835819 secs 2024-04-26T14:33:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9759.611663 secs 2024-04-26T14:33:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:39 smithi096 ceph-mon[32753]: pgmap v5790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9756.835819 secs 2024-04-26T14:33:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9759.611663 secs 2024-04-26T14:33:39.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:39.905+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:39.905+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:33:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:39.905+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:41 smithi194 ceph-mon[33017]: pgmap v5791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:41 smithi033 ceph-mon[27845]: pgmap v5791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:41 smithi096 ceph-mon[32753]: pgmap v5791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:33:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:43 smithi194 ceph-mon[33017]: pgmap v5792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:43 smithi033 ceph-mon[27845]: pgmap v5792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:43 smithi096 ceph-mon[32753]: pgmap v5792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9761.835902 secs 2024-04-26T14:33:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9764.611803 secs 2024-04-26T14:33:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:33:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9761.835902 secs 2024-04-26T14:33:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9764.611803 secs 2024-04-26T14:33:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:33:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9761.835902 secs 2024-04-26T14:33:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9764.611803 secs 2024-04-26T14:33:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:33:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:45 smithi033 ceph-mon[27845]: pgmap v5793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:33:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:33:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:33:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:45 smithi096 ceph-mon[32753]: pgmap v5793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:33:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:33:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:33:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:45 smithi194 ceph-mon[33017]: pgmap v5793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:33:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:33:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:33:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:46 smithi033 ceph-mon[27845]: pgmap v5794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:33:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:46 smithi033 ceph-mon[27845]: pgmap v5795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:46 smithi096 ceph-mon[32753]: pgmap v5794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:33:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:46 smithi096 ceph-mon[32753]: pgmap v5795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:46 smithi194 ceph-mon[33017]: pgmap v5794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:33:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:46 smithi194 ceph-mon[33017]: pgmap v5795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:48 smithi033 ceph-mon[27845]: pgmap v5796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:48 smithi096 ceph-mon[32753]: pgmap v5796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:48 smithi194 ceph-mon[33017]: pgmap v5796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9766.835988 secs 2024-04-26T14:33:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9769.611932 secs 2024-04-26T14:33:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9766.835988 secs 2024-04-26T14:33:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9769.611932 secs 2024-04-26T14:33:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9766.835988 secs 2024-04-26T14:33:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9769.611932 secs 2024-04-26T14:33:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:50 smithi033 ceph-mon[27845]: pgmap v5797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:33:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:50 smithi096 ceph-mon[32753]: pgmap v5797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:33:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:50 smithi194 ceph-mon[33017]: pgmap v5797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:33:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:52 smithi033 ceph-mon[27845]: pgmap v5798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:52 smithi096 ceph-mon[32753]: pgmap v5798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:52 smithi194 ceph-mon[33017]: pgmap v5798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:53 smithi194 ceph-mon[33017]: pgmap v5799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:53 smithi033 ceph-mon[27845]: pgmap v5799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:53 smithi096 ceph-mon[32753]: pgmap v5799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:33:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9771.836073 secs 2024-04-26T14:33:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9774.612035 secs 2024-04-26T14:33:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9771.836073 secs 2024-04-26T14:33:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9774.612035 secs 2024-04-26T14:33:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9771.836073 secs 2024-04-26T14:33:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9774.612035 secs 2024-04-26T14:33:54.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:54.911+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:54.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:54.911+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:33:54.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:33:54.911+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:33:55.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:55 smithi033 ceph-mon[27845]: pgmap v5800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:33:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:55 smithi194 ceph-mon[33017]: pgmap v5800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:33:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:55 smithi096 ceph-mon[32753]: pgmap v5800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:33:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:58 smithi194 ceph-mon[33017]: pgmap v5801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:58 smithi033 ceph-mon[27845]: pgmap v5801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:58 smithi096 ceph-mon[32753]: pgmap v5801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:33:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9776.836154 secs 2024-04-26T14:33:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:33:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9779.612159 secs 2024-04-26T14:33:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9776.836154 secs 2024-04-26T14:33:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:33:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9779.612159 secs 2024-04-26T14:33:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9776.836154 secs 2024-04-26T14:33:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:33:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9779.612159 secs 2024-04-26T14:34:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:00 smithi194 ceph-mon[33017]: pgmap v5802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:00 smithi033 ceph-mon[27845]: pgmap v5802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:00 smithi096 ceph-mon[32753]: pgmap v5802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:01 smithi194 ceph-mon[33017]: pgmap v5803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:01 smithi033 ceph-mon[27845]: pgmap v5803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:01 smithi096 ceph-mon[32753]: pgmap v5803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9781.836259 secs 2024-04-26T14:34:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:04 smithi194 ceph-mon[33017]: pgmap v5804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9784.612281 secs 2024-04-26T14:34:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9781.836259 secs 2024-04-26T14:34:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:04 smithi033 ceph-mon[27845]: pgmap v5804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9784.612281 secs 2024-04-26T14:34:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9781.836259 secs 2024-04-26T14:34:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:04 smithi096 ceph-mon[32753]: pgmap v5804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9784.612281 secs 2024-04-26T14:34:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:06 smithi194 ceph-mon[33017]: pgmap v5805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:06 smithi033 ceph-mon[27845]: pgmap v5805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:06 smithi096 ceph-mon[32753]: pgmap v5805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:08 smithi194 ceph-mon[33017]: pgmap v5806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:08 smithi033 ceph-mon[27845]: pgmap v5806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:08 smithi096 ceph-mon[32753]: pgmap v5806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9786.836320 secs 2024-04-26T14:34:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9789.612377 secs 2024-04-26T14:34:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9786.836320 secs 2024-04-26T14:34:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9789.612377 secs 2024-04-26T14:34:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9786.836320 secs 2024-04-26T14:34:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9789.612377 secs 2024-04-26T14:34:09.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:09.926+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:09.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:09.926+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:34:09.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:09.926+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:10 smithi194 ceph-mon[33017]: pgmap v5807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:10 smithi033 ceph-mon[27845]: pgmap v5807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:10 smithi096 ceph-mon[32753]: pgmap v5807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:12 smithi194 ceph-mon[33017]: pgmap v5808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:12 smithi033 ceph-mon[27845]: pgmap v5808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:12 smithi096 ceph-mon[32753]: pgmap v5808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9791.836404 secs 2024-04-26T14:34:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:14 smithi194 ceph-mon[33017]: pgmap v5809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9794.612522 secs 2024-04-26T14:34:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9791.836404 secs 2024-04-26T14:34:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:14 smithi033 ceph-mon[27845]: pgmap v5809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9794.612522 secs 2024-04-26T14:34:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9791.836404 secs 2024-04-26T14:34:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:14 smithi096 ceph-mon[32753]: pgmap v5809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9794.612522 secs 2024-04-26T14:34:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:16 smithi194 ceph-mon[33017]: pgmap v5810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:16 smithi033 ceph-mon[27845]: pgmap v5810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:16 smithi096 ceph-mon[32753]: pgmap v5810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:18 smithi194 ceph-mon[33017]: pgmap v5811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:18 smithi033 ceph-mon[27845]: pgmap v5811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:18 smithi096 ceph-mon[32753]: pgmap v5811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9796.836487 secs 2024-04-26T14:34:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9799.612659 secs 2024-04-26T14:34:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9796.836487 secs 2024-04-26T14:34:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9799.612659 secs 2024-04-26T14:34:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9796.836487 secs 2024-04-26T14:34:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9799.612659 secs 2024-04-26T14:34:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:20 smithi033 ceph-mon[27845]: pgmap v5812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:20 smithi096 ceph-mon[32753]: pgmap v5812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:20 smithi194 ceph-mon[33017]: pgmap v5812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:22 smithi033 ceph-mon[27845]: pgmap v5813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:22.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:22 smithi096 ceph-mon[32753]: pgmap v5813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:22 smithi194 ceph-mon[33017]: pgmap v5813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9801.836575 secs 2024-04-26T14:34:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:24 smithi033 ceph-mon[27845]: pgmap v5814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9804.612799 secs 2024-04-26T14:34:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9801.836575 secs 2024-04-26T14:34:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:24 smithi096 ceph-mon[32753]: pgmap v5814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9804.612799 secs 2024-04-26T14:34:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9801.836575 secs 2024-04-26T14:34:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:24 smithi194 ceph-mon[33017]: pgmap v5814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9804.612799 secs 2024-04-26T14:34:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:24.941+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:24.941+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:34:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:24.941+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:26 smithi033 ceph-mon[27845]: pgmap v5815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:26 smithi096 ceph-mon[32753]: pgmap v5815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:26 smithi194 ceph-mon[33017]: pgmap v5815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:28 smithi033 ceph-mon[27845]: pgmap v5816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:28 smithi096 ceph-mon[32753]: pgmap v5816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:28 smithi194 ceph-mon[33017]: pgmap v5816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9806.836654 secs 2024-04-26T14:34:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9809.612941 secs 2024-04-26T14:34:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9806.836654 secs 2024-04-26T14:34:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9809.612941 secs 2024-04-26T14:34:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9806.836654 secs 2024-04-26T14:34:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9809.612941 secs 2024-04-26T14:34:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:30 smithi033 ceph-mon[27845]: pgmap v5817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:30 smithi096 ceph-mon[32753]: pgmap v5817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:30 smithi194 ceph-mon[33017]: pgmap v5817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:32 smithi033 ceph-mon[27845]: pgmap v5818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:32 smithi096 ceph-mon[32753]: pgmap v5818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:32 smithi194 ceph-mon[33017]: pgmap v5818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:33 smithi194 ceph-mon[33017]: pgmap v5819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:33 smithi033 ceph-mon[27845]: pgmap v5819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:33 smithi096 ceph-mon[32753]: pgmap v5819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9811.836733 secs 2024-04-26T14:34:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9814.613067 secs 2024-04-26T14:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9811.836733 secs 2024-04-26T14:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9814.613067 secs 2024-04-26T14:34:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9811.836733 secs 2024-04-26T14:34:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9814.613067 secs 2024-04-26T14:34:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:35 smithi194 ceph-mon[33017]: pgmap v5820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:35 smithi033 ceph-mon[27845]: pgmap v5820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:35 smithi096 ceph-mon[32753]: pgmap v5820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:38 smithi194 ceph-mon[33017]: pgmap v5821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:38 smithi033 ceph-mon[27845]: pgmap v5821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:38 smithi096 ceph-mon[32753]: pgmap v5821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:38.460+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:34:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:38.460+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:34:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:38.461+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:34:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:38.461+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:34:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9816.836791 secs 2024-04-26T14:34:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9819.613167 secs 2024-04-26T14:34:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9816.836791 secs 2024-04-26T14:34:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9819.613167 secs 2024-04-26T14:34:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9816.836791 secs 2024-04-26T14:34:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9819.613167 secs 2024-04-26T14:34:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:39.957+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:39.957+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:34:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:39.957+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:40 smithi194 ceph-mon[33017]: pgmap v5822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:40 smithi033 ceph-mon[27845]: pgmap v5822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:40 smithi096 ceph-mon[32753]: pgmap v5822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:42 smithi194 ceph-mon[33017]: pgmap v5823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:42 smithi033 ceph-mon[27845]: pgmap v5823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:42 smithi096 ceph-mon[32753]: pgmap v5823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9821.836874 secs 2024-04-26T14:34:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:44 smithi194 ceph-mon[33017]: pgmap v5824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9824.613253 secs 2024-04-26T14:34:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9821.836874 secs 2024-04-26T14:34:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:44 smithi033 ceph-mon[27845]: pgmap v5824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9824.613253 secs 2024-04-26T14:34:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9821.836874 secs 2024-04-26T14:34:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:44 smithi096 ceph-mon[32753]: pgmap v5824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:44.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9824.613253 secs 2024-04-26T14:34:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:46 smithi194 ceph-mon[33017]: pgmap v5825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:34:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:46 smithi033 ceph-mon[27845]: pgmap v5825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:34:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:46 smithi096 ceph-mon[32753]: pgmap v5825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:34:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:34:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:34:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:34:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:34:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:34:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:34:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:34:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:34:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:34:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:34:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:48 smithi033 ceph-mon[27845]: pgmap v5826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:48 smithi033 ceph-mon[27845]: pgmap v5827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:48 smithi096 ceph-mon[32753]: pgmap v5826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:48 smithi096 ceph-mon[32753]: pgmap v5827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:48 smithi194 ceph-mon[33017]: pgmap v5826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:34:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:48 smithi194 ceph-mon[33017]: pgmap v5827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9826.836960 secs 2024-04-26T14:34:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9829.613383 secs 2024-04-26T14:34:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9826.836960 secs 2024-04-26T14:34:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9829.613383 secs 2024-04-26T14:34:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9826.836960 secs 2024-04-26T14:34:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9829.613383 secs 2024-04-26T14:34:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:50 smithi033 ceph-mon[27845]: pgmap v5828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:50 smithi096 ceph-mon[32753]: pgmap v5828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:50 smithi194 ceph-mon[33017]: pgmap v5828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:52 smithi033 ceph-mon[27845]: pgmap v5829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:52 smithi096 ceph-mon[32753]: pgmap v5829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:52 smithi194 ceph-mon[33017]: pgmap v5829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9831.837043 secs 2024-04-26T14:34:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:54 smithi033 ceph-mon[27845]: pgmap v5830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9834.613511 secs 2024-04-26T14:34:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9831.837043 secs 2024-04-26T14:34:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:54 smithi096 ceph-mon[32753]: pgmap v5830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9834.613511 secs 2024-04-26T14:34:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9831.837043 secs 2024-04-26T14:34:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:54 smithi194 ceph-mon[33017]: pgmap v5830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9834.613511 secs 2024-04-26T14:34:54.968 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:54.967+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:54.968 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:54.967+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:34:54.968 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:34:54.967+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:34:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:56 smithi033 ceph-mon[27845]: pgmap v5831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:56 smithi096 ceph-mon[32753]: pgmap v5831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:56 smithi194 ceph-mon[33017]: pgmap v5831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:34:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:58 smithi033 ceph-mon[27845]: pgmap v5832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:34:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:58 smithi096 ceph-mon[32753]: pgmap v5832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:34:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:58 smithi194 ceph-mon[33017]: pgmap v5832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:34:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9836.837126 secs 2024-04-26T14:34:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9839.613655 secs 2024-04-26T14:34:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9836.837126 secs 2024-04-26T14:34:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9839.613655 secs 2024-04-26T14:34:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9836.837126 secs 2024-04-26T14:34:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9839.613655 secs 2024-04-26T14:35:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:00 smithi033 ceph-mon[27845]: pgmap v5833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:00 smithi096 ceph-mon[32753]: pgmap v5833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:00 smithi194 ceph-mon[33017]: pgmap v5833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:01 smithi194 ceph-mon[33017]: pgmap v5834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:01 smithi033 ceph-mon[27845]: pgmap v5834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:01 smithi096 ceph-mon[32753]: pgmap v5834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9841.837211 secs 2024-04-26T14:35:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:03 smithi194 ceph-mon[33017]: pgmap v5835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9844.613773 secs 2024-04-26T14:35:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9841.837211 secs 2024-04-26T14:35:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:03 smithi033 ceph-mon[27845]: pgmap v5835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9844.613773 secs 2024-04-26T14:35:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9841.837211 secs 2024-04-26T14:35:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:03 smithi096 ceph-mon[32753]: pgmap v5835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9844.613773 secs 2024-04-26T14:35:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:05 smithi194 ceph-mon[33017]: pgmap v5836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:05 smithi033 ceph-mon[27845]: pgmap v5836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:05 smithi096 ceph-mon[32753]: pgmap v5836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:07.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:07 smithi033 ceph-mon[27845]: pgmap v5837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:07 smithi194 ceph-mon[33017]: pgmap v5837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:07 smithi096 ceph-mon[32753]: pgmap v5837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9846.837295 secs 2024-04-26T14:35:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9849.613894 secs 2024-04-26T14:35:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9846.837295 secs 2024-04-26T14:35:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9849.613894 secs 2024-04-26T14:35:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9846.837295 secs 2024-04-26T14:35:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9849.613894 secs 2024-04-26T14:35:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:09.984+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:09.984+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:35:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:09.984+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:09 smithi194 ceph-mon[33017]: pgmap v5838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:09 smithi033 ceph-mon[27845]: pgmap v5838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:09 smithi096 ceph-mon[32753]: pgmap v5838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:11 smithi194 ceph-mon[33017]: pgmap v5839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:11 smithi033 ceph-mon[27845]: pgmap v5839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:11 smithi096 ceph-mon[32753]: pgmap v5839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9851.837379 secs 2024-04-26T14:35:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:13 smithi194 ceph-mon[33017]: pgmap v5840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9854.614015 secs 2024-04-26T14:35:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9851.837379 secs 2024-04-26T14:35:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:13 smithi033 ceph-mon[27845]: pgmap v5840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9854.614015 secs 2024-04-26T14:35:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9851.837379 secs 2024-04-26T14:35:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:13 smithi096 ceph-mon[32753]: pgmap v5840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9854.614015 secs 2024-04-26T14:35:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:15 smithi194 ceph-mon[33017]: pgmap v5841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:15 smithi033 ceph-mon[27845]: pgmap v5841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:15 smithi096 ceph-mon[32753]: pgmap v5841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:17 smithi194 ceph-mon[33017]: pgmap v5842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:17 smithi033 ceph-mon[27845]: pgmap v5842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:17 smithi096 ceph-mon[32753]: pgmap v5842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9856.837460 secs 2024-04-26T14:35:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9859.614169 secs 2024-04-26T14:35:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9856.837460 secs 2024-04-26T14:35:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9859.614169 secs 2024-04-26T14:35:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9856.837460 secs 2024-04-26T14:35:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9859.614169 secs 2024-04-26T14:35:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:19 smithi194 ceph-mon[33017]: pgmap v5843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:19 smithi033 ceph-mon[27845]: pgmap v5843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:19 smithi096 ceph-mon[32753]: pgmap v5843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:21 smithi194 ceph-mon[33017]: pgmap v5844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:21 smithi033 ceph-mon[27845]: pgmap v5844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:21 smithi096 ceph-mon[32753]: pgmap v5844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:23.468+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:35:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:23.468+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:35:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:23.469+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:35:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:23.469+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T14:35:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9861.837559 secs 2024-04-26T14:35:24.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:23 smithi194 ceph-mon[33017]: pgmap v5845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:24.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9864.614288 secs 2024-04-26T14:35:24.358 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9861.837559 secs 2024-04-26T14:35:24.358 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:23 smithi096 ceph-mon[32753]: pgmap v5845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:24.358 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9864.614288 secs 2024-04-26T14:35:24.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9861.837559 secs 2024-04-26T14:35:24.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:23 smithi033 ceph-mon[27845]: pgmap v5845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:24.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9864.614288 secs 2024-04-26T14:35:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:24.999+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:24.999+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:35:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:24.999+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:25 smithi194 ceph-mon[33017]: pgmap v5846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:25 smithi033 ceph-mon[27845]: pgmap v5846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:25 smithi096 ceph-mon[32753]: pgmap v5846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:27 smithi194 ceph-mon[33017]: pgmap v5847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:27 smithi033 ceph-mon[27845]: pgmap v5847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:27 smithi096 ceph-mon[32753]: pgmap v5847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9866.837642 secs 2024-04-26T14:35:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9869.614416 secs 2024-04-26T14:35:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9866.837642 secs 2024-04-26T14:35:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9869.614416 secs 2024-04-26T14:35:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9866.837642 secs 2024-04-26T14:35:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9869.614416 secs 2024-04-26T14:35:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:29 smithi194 ceph-mon[33017]: pgmap v5848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:29 smithi033 ceph-mon[27845]: pgmap v5848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:29 smithi096 ceph-mon[32753]: pgmap v5848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:31 smithi194 ceph-mon[33017]: pgmap v5849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:31 smithi033 ceph-mon[27845]: pgmap v5849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:31 smithi096 ceph-mon[32753]: pgmap v5849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9871.837739 secs 2024-04-26T14:35:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:33 smithi194 ceph-mon[33017]: pgmap v5850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9874.614534 secs 2024-04-26T14:35:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9871.837739 secs 2024-04-26T14:35:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:33 smithi033 ceph-mon[27845]: pgmap v5850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9874.614534 secs 2024-04-26T14:35:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9871.837739 secs 2024-04-26T14:35:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:33 smithi096 ceph-mon[32753]: pgmap v5850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9874.614534 secs 2024-04-26T14:35:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:35 smithi194 ceph-mon[33017]: pgmap v5851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:35 smithi033 ceph-mon[27845]: pgmap v5851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:35 smithi096 ceph-mon[32753]: pgmap v5851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:35:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:37 smithi194 ceph-mon[33017]: pgmap v5852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:37 smithi033 ceph-mon[27845]: pgmap v5852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:37 smithi096 ceph-mon[32753]: pgmap v5852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9876.837819 secs 2024-04-26T14:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9879.614671 secs 2024-04-26T14:35:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9876.837819 secs 2024-04-26T14:35:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9879.614671 secs 2024-04-26T14:35:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9876.837819 secs 2024-04-26T14:35:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9879.614671 secs 2024-04-26T14:35:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:40.015+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:40.015+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:35:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:40.015+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:39 smithi194 ceph-mon[33017]: pgmap v5853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:39 smithi033 ceph-mon[27845]: pgmap v5853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:39 smithi096 ceph-mon[32753]: pgmap v5853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:35:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:41 smithi194 ceph-mon[33017]: pgmap v5854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:41 smithi033 ceph-mon[27845]: pgmap v5854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:41 smithi096 ceph-mon[32753]: pgmap v5854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9881.837900 secs 2024-04-26T14:35:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:43 smithi194 ceph-mon[33017]: pgmap v5855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9884.614747 secs 2024-04-26T14:35:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9881.837900 secs 2024-04-26T14:35:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:43 smithi033 ceph-mon[27845]: pgmap v5855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9884.614747 secs 2024-04-26T14:35:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9881.837900 secs 2024-04-26T14:35:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:43 smithi096 ceph-mon[32753]: pgmap v5855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9884.614747 secs 2024-04-26T14:35:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:45 smithi033 ceph-mon[27845]: pgmap v5856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:45 smithi096 ceph-mon[32753]: pgmap v5856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:45 smithi194 ceph-mon[33017]: pgmap v5856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:35:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:47 smithi033 ceph-mon[27845]: pgmap v5857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:35:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:47 smithi096 ceph-mon[32753]: pgmap v5857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:35:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:47 smithi194 ceph-mon[33017]: pgmap v5857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:35:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9886.838007 secs 2024-04-26T14:35:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9889.614863 secs 2024-04-26T14:35:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9886.838007 secs 2024-04-26T14:35:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9889.614863 secs 2024-04-26T14:35:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9886.838007 secs 2024-04-26T14:35:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9889.614863 secs 2024-04-26T14:35:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:49 smithi033 ceph-mon[27845]: pgmap v5858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:35:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:35:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:49 smithi096 ceph-mon[32753]: pgmap v5858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:35:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:35:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:49 smithi194 ceph-mon[33017]: pgmap v5858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:35:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:35:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:35:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:35:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:50 smithi033 ceph-mon[27845]: pgmap v5859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T14:35:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:50 smithi033 ceph-mon[27845]: pgmap v5860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:35:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:50 smithi096 ceph-mon[32753]: pgmap v5859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T14:35:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:50 smithi096 ceph-mon[32753]: pgmap v5860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:35:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:50 smithi194 ceph-mon[33017]: pgmap v5859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T14:35:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:50 smithi194 ceph-mon[33017]: pgmap v5860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:35:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:52 smithi033 ceph-mon[27845]: pgmap v5861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:52 smithi096 ceph-mon[32753]: pgmap v5861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:52 smithi194 ceph-mon[33017]: pgmap v5861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9891.838090 secs 2024-04-26T14:35:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9894.614987 secs 2024-04-26T14:35:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9891.838090 secs 2024-04-26T14:35:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9894.614987 secs 2024-04-26T14:35:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9891.838090 secs 2024-04-26T14:35:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9894.614987 secs 2024-04-26T14:35:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:55.028+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:55.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:55.028+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:35:55.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:35:55.028+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:35:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:54 smithi033 ceph-mon[27845]: pgmap v5862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:35:55.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:54 smithi096 ceph-mon[32753]: pgmap v5862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:35:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:54 smithi194 ceph-mon[33017]: pgmap v5862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:35:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:56 smithi096 ceph-mon[32753]: pgmap v5863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:56 smithi194 ceph-mon[33017]: pgmap v5863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:56 smithi033 ceph-mon[27845]: pgmap v5863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:59.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:58 smithi033 ceph-mon[27845]: pgmap v5864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:59.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9896.838173 secs 2024-04-26T14:35:59.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:35:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9899.615131 secs 2024-04-26T14:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:58 smithi194 ceph-mon[33017]: pgmap v5864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9896.838173 secs 2024-04-26T14:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:35:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9899.615131 secs 2024-04-26T14:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:58 smithi096 ceph-mon[32753]: pgmap v5864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9896.838173 secs 2024-04-26T14:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:35:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9899.615131 secs 2024-04-26T14:36:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:00 smithi194 ceph-mon[33017]: pgmap v5865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:36:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:00 smithi033 ceph-mon[27845]: pgmap v5865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:36:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:00 smithi096 ceph-mon[32753]: pgmap v5865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:36:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:02 smithi194 ceph-mon[33017]: pgmap v5866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:02 smithi033 ceph-mon[27845]: pgmap v5866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:02 smithi096 ceph-mon[32753]: pgmap v5866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9901.838275 secs 2024-04-26T14:36:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9904.615198 secs 2024-04-26T14:36:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9901.838275 secs 2024-04-26T14:36:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9904.615198 secs 2024-04-26T14:36:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9901.838275 secs 2024-04-26T14:36:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9904.615198 secs 2024-04-26T14:36:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:04 smithi194 ceph-mon[33017]: pgmap v5867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:04 smithi033 ceph-mon[27845]: pgmap v5867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:04 smithi096 ceph-mon[32753]: pgmap v5867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:06 smithi194 ceph-mon[33017]: pgmap v5868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:06 smithi033 ceph-mon[27845]: pgmap v5868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:06 smithi096 ceph-mon[32753]: pgmap v5868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:08 smithi194 ceph-mon[33017]: pgmap v5869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9906.838360 secs 2024-04-26T14:36:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9909.615307 secs 2024-04-26T14:36:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:08 smithi033 ceph-mon[27845]: pgmap v5869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9906.838360 secs 2024-04-26T14:36:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9909.615307 secs 2024-04-26T14:36:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:08 smithi096 ceph-mon[32753]: pgmap v5869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9906.838360 secs 2024-04-26T14:36:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9909.615307 secs 2024-04-26T14:36:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:10.044+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:10.044+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:10.044+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:36:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:10 smithi194 ceph-mon[33017]: pgmap v5870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:10 smithi033 ceph-mon[27845]: pgmap v5870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:10 smithi096 ceph-mon[32753]: pgmap v5870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:12 smithi194 ceph-mon[33017]: pgmap v5871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:12 smithi033 ceph-mon[27845]: pgmap v5871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:12 smithi096 ceph-mon[32753]: pgmap v5871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9911.838445 secs 2024-04-26T14:36:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9914.615451 secs 2024-04-26T14:36:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9911.838445 secs 2024-04-26T14:36:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9914.615451 secs 2024-04-26T14:36:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9911.838445 secs 2024-04-26T14:36:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9914.615451 secs 2024-04-26T14:36:15.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:14 smithi194 ceph-mon[33017]: pgmap v5872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:15.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:14 smithi096 ceph-mon[32753]: pgmap v5872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:14 smithi033 ceph-mon[27845]: pgmap v5872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:16 smithi194 ceph-mon[33017]: pgmap v5873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:16 smithi033 ceph-mon[27845]: pgmap v5873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:16 smithi096 ceph-mon[32753]: pgmap v5873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:19 smithi194 ceph-mon[33017]: pgmap v5874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9916.838526 secs 2024-04-26T14:36:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9919.615575 secs 2024-04-26T14:36:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:19 smithi033 ceph-mon[27845]: pgmap v5874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9916.838526 secs 2024-04-26T14:36:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9919.615575 secs 2024-04-26T14:36:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:19 smithi096 ceph-mon[32753]: pgmap v5874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9916.838526 secs 2024-04-26T14:36:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9919.615575 secs 2024-04-26T14:36:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:21 smithi194 ceph-mon[33017]: pgmap v5875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:21 smithi033 ceph-mon[27845]: pgmap v5875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:21 smithi096 ceph-mon[32753]: pgmap v5875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:23 smithi194 ceph-mon[33017]: pgmap v5876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:23 smithi033 ceph-mon[27845]: pgmap v5876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:23 smithi096 ceph-mon[32753]: pgmap v5876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9921.838616 secs 2024-04-26T14:36:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9924.615716 secs 2024-04-26T14:36:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9921.838616 secs 2024-04-26T14:36:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9924.615716 secs 2024-04-26T14:36:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9921.838616 secs 2024-04-26T14:36:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9924.615716 secs 2024-04-26T14:36:25.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:25.053+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:25.053+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:25.053+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:25 smithi194 ceph-mon[33017]: pgmap v5877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:25 smithi033 ceph-mon[27845]: pgmap v5877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:25 smithi096 ceph-mon[32753]: pgmap v5877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:27 smithi194 ceph-mon[33017]: pgmap v5878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:27 smithi033 ceph-mon[27845]: pgmap v5878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:27 smithi096 ceph-mon[32753]: pgmap v5878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:29 smithi194 ceph-mon[33017]: pgmap v5879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9926.838705 secs 2024-04-26T14:36:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9929.615838 secs 2024-04-26T14:36:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:29 smithi033 ceph-mon[27845]: pgmap v5879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9926.838705 secs 2024-04-26T14:36:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9929.615838 secs 2024-04-26T14:36:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:29 smithi096 ceph-mon[32753]: pgmap v5879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9926.838705 secs 2024-04-26T14:36:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9929.615838 secs 2024-04-26T14:36:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:31 smithi194 ceph-mon[33017]: pgmap v5880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:31 smithi033 ceph-mon[27845]: pgmap v5880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:31 smithi096 ceph-mon[32753]: pgmap v5880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:33 smithi194 ceph-mon[33017]: pgmap v5881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:33 smithi033 ceph-mon[27845]: pgmap v5881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:33 smithi096 ceph-mon[32753]: pgmap v5881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9931.838789 secs 2024-04-26T14:36:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9934.615968 secs 2024-04-26T14:36:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9931.838789 secs 2024-04-26T14:36:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9934.615968 secs 2024-04-26T14:36:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9931.838789 secs 2024-04-26T14:36:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9934.615968 secs 2024-04-26T14:36:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:35 smithi194 ceph-mon[33017]: pgmap v5882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:35 smithi033 ceph-mon[27845]: pgmap v5882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:35 smithi096 ceph-mon[32753]: pgmap v5882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:37 smithi194 ceph-mon[33017]: pgmap v5883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:37 smithi033 ceph-mon[27845]: pgmap v5883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:37 smithi096 ceph-mon[32753]: pgmap v5883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:39 smithi194 ceph-mon[33017]: pgmap v5884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9936.838872 secs 2024-04-26T14:36:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9939.616091 secs 2024-04-26T14:36:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:39 smithi033 ceph-mon[27845]: pgmap v5884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9936.838872 secs 2024-04-26T14:36:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9939.616091 secs 2024-04-26T14:36:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:39 smithi096 ceph-mon[32753]: pgmap v5884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9936.838872 secs 2024-04-26T14:36:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9939.616091 secs 2024-04-26T14:36:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:40.064+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:40.064+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:40.064+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:36:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:41 smithi194 ceph-mon[33017]: pgmap v5885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:41 smithi033 ceph-mon[27845]: pgmap v5885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:41 smithi096 ceph-mon[32753]: pgmap v5885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:43 smithi194 ceph-mon[33017]: pgmap v5886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:43 smithi033 ceph-mon[27845]: pgmap v5886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:43 smithi096 ceph-mon[32753]: pgmap v5886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9941.838957 secs 2024-04-26T14:36:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9944.616145 secs 2024-04-26T14:36:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9941.838957 secs 2024-04-26T14:36:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9944.616145 secs 2024-04-26T14:36:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9941.838957 secs 2024-04-26T14:36:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9944.616145 secs 2024-04-26T14:36:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:45 smithi194 ceph-mon[33017]: pgmap v5887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:45 smithi033 ceph-mon[27845]: pgmap v5887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:45 smithi096 ceph-mon[32753]: pgmap v5887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:47 smithi033 ceph-mon[27845]: pgmap v5888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:47 smithi096 ceph-mon[32753]: pgmap v5888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:47 smithi194 ceph-mon[33017]: pgmap v5888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:36:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:49 smithi033 ceph-mon[27845]: pgmap v5889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9946.839046 secs 2024-04-26T14:36:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9949.616241 secs 2024-04-26T14:36:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:49 smithi096 ceph-mon[32753]: pgmap v5889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9946.839046 secs 2024-04-26T14:36:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9949.616241 secs 2024-04-26T14:36:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:49 smithi194 ceph-mon[33017]: pgmap v5889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9946.839046 secs 2024-04-26T14:36:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9949.616241 secs 2024-04-26T14:36:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:36:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:36:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:36:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:51 smithi033 ceph-mon[27845]: pgmap v5890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:36:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:36:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:36:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:51 smithi096 ceph-mon[32753]: pgmap v5890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:36:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:36:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:36:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:51 smithi194 ceph-mon[33017]: pgmap v5890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:36:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:36:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:36:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:36:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:52 smithi033 ceph-mon[27845]: pgmap v5891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-26T14:36:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:52 smithi033 ceph-mon[27845]: pgmap v5892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:52 smithi096 ceph-mon[32753]: pgmap v5891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-26T14:36:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:52 smithi096 ceph-mon[32753]: pgmap v5892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:52 smithi194 ceph-mon[33017]: pgmap v5891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-26T14:36:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:52 smithi194 ceph-mon[33017]: pgmap v5892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9951.839132 secs 2024-04-26T14:36:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:54 smithi033 ceph-mon[27845]: pgmap v5893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9954.616338 secs 2024-04-26T14:36:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9951.839132 secs 2024-04-26T14:36:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:54 smithi096 ceph-mon[32753]: pgmap v5893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9954.616338 secs 2024-04-26T14:36:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9951.839132 secs 2024-04-26T14:36:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:54 smithi194 ceph-mon[33017]: pgmap v5893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9954.616338 secs 2024-04-26T14:36:55.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:55.078+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:55.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:55.078+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:36:55.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:36:55.078+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:36:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:56 smithi033 ceph-mon[27845]: pgmap v5894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:56 smithi096 ceph-mon[32753]: pgmap v5894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:56 smithi194 ceph-mon[33017]: pgmap v5894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:58 smithi033 ceph-mon[27845]: pgmap v5895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:58 smithi096 ceph-mon[32753]: pgmap v5895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:58 smithi194 ceph-mon[33017]: pgmap v5895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:36:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9956.839215 secs 2024-04-26T14:36:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9959.616452 secs 2024-04-26T14:36:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9956.839215 secs 2024-04-26T14:36:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9959.616452 secs 2024-04-26T14:36:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9956.839215 secs 2024-04-26T14:36:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9959.616452 secs 2024-04-26T14:37:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:00 smithi033 ceph-mon[27845]: pgmap v5896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:37:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:00 smithi096 ceph-mon[32753]: pgmap v5896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:37:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:00 smithi194 ceph-mon[33017]: pgmap v5896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:37:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:02 smithi033 ceph-mon[27845]: pgmap v5897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:37:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:02 smithi096 ceph-mon[32753]: pgmap v5897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:37:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:02 smithi194 ceph-mon[33017]: pgmap v5897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9961.839299 secs 2024-04-26T14:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:04 smithi096 ceph-mon[32753]: pgmap v5898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9964.616570 secs 2024-04-26T14:37:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9961.839299 secs 2024-04-26T14:37:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:04 smithi194 ceph-mon[33017]: pgmap v5898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9964.616570 secs 2024-04-26T14:37:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9961.839299 secs 2024-04-26T14:37:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:04 smithi033 ceph-mon[27845]: pgmap v5898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9964.616570 secs 2024-04-26T14:37:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:06 smithi194 ceph-mon[33017]: pgmap v5899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:06 smithi033 ceph-mon[27845]: pgmap v5899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:06 smithi096 ceph-mon[32753]: pgmap v5899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:07 smithi194 ceph-mon[33017]: pgmap v5900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:07 smithi033 ceph-mon[27845]: pgmap v5900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:07 smithi096 ceph-mon[32753]: pgmap v5900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9966.839361 secs 2024-04-26T14:37:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9969.616689 secs 2024-04-26T14:37:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9966.839361 secs 2024-04-26T14:37:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9969.616689 secs 2024-04-26T14:37:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9966.839361 secs 2024-04-26T14:37:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9969.616689 secs 2024-04-26T14:37:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:09 smithi033 ceph-mon[27845]: pgmap v5901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:09 smithi096 ceph-mon[32753]: pgmap v5901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:10.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:10.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:10.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:10.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:10.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:10.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:37:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:09 smithi194 ceph-mon[33017]: pgmap v5901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:12 smithi033 ceph-mon[27845]: pgmap v5902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:12 smithi194 ceph-mon[33017]: pgmap v5902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:12 smithi096 ceph-mon[32753]: pgmap v5902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9971.839445 secs 2024-04-26T14:37:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:14 smithi194 ceph-mon[33017]: pgmap v5903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9974.616824 secs 2024-04-26T14:37:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9971.839445 secs 2024-04-26T14:37:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:14 smithi033 ceph-mon[27845]: pgmap v5903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9974.616824 secs 2024-04-26T14:37:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9971.839445 secs 2024-04-26T14:37:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:14 smithi096 ceph-mon[32753]: pgmap v5903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9974.616824 secs 2024-04-26T14:37:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:16 smithi194 ceph-mon[33017]: pgmap v5904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:16 smithi033 ceph-mon[27845]: pgmap v5904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:16 smithi096 ceph-mon[32753]: pgmap v5904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:18 smithi194 ceph-mon[33017]: pgmap v5905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:18 smithi033 ceph-mon[27845]: pgmap v5905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:18 smithi096 ceph-mon[32753]: pgmap v5905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9976.839505 secs 2024-04-26T14:37:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9979.616952 secs 2024-04-26T14:37:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9976.839505 secs 2024-04-26T14:37:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9979.616952 secs 2024-04-26T14:37:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9976.839505 secs 2024-04-26T14:37:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9979.616952 secs 2024-04-26T14:37:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:20 smithi194 ceph-mon[33017]: pgmap v5906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:20 smithi033 ceph-mon[27845]: pgmap v5906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:20 smithi096 ceph-mon[32753]: pgmap v5906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:22 smithi194 ceph-mon[33017]: pgmap v5907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:22 smithi033 ceph-mon[27845]: pgmap v5907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:22 smithi096 ceph-mon[32753]: pgmap v5907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9981.839593 secs 2024-04-26T14:37:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:24 smithi194 ceph-mon[33017]: pgmap v5908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9984.617075 secs 2024-04-26T14:37:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9981.839593 secs 2024-04-26T14:37:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:24 smithi033 ceph-mon[27845]: pgmap v5908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9984.617075 secs 2024-04-26T14:37:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9981.839593 secs 2024-04-26T14:37:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:24 smithi096 ceph-mon[32753]: pgmap v5908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9984.617075 secs 2024-04-26T14:37:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:25.110+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:25.110+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:25.110+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:37:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:26 smithi194 ceph-mon[33017]: pgmap v5909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:26 smithi033 ceph-mon[27845]: pgmap v5909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:26 smithi096 ceph-mon[32753]: pgmap v5909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:28 smithi194 ceph-mon[33017]: pgmap v5910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:28 smithi033 ceph-mon[27845]: pgmap v5910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:28 smithi096 ceph-mon[32753]: pgmap v5910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9986.839680 secs 2024-04-26T14:37:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9989.617204 secs 2024-04-26T14:37:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9986.839680 secs 2024-04-26T14:37:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9989.617204 secs 2024-04-26T14:37:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9986.839680 secs 2024-04-26T14:37:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9989.617204 secs 2024-04-26T14:37:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:30 smithi033 ceph-mon[27845]: pgmap v5911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:30 smithi096 ceph-mon[32753]: pgmap v5911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:30 smithi194 ceph-mon[33017]: pgmap v5911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:32 smithi033 ceph-mon[27845]: pgmap v5912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:32 smithi096 ceph-mon[32753]: pgmap v5912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:32 smithi194 ceph-mon[33017]: pgmap v5912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9991.839762 secs 2024-04-26T14:37:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:34 smithi033 ceph-mon[27845]: pgmap v5913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9994.617313 secs 2024-04-26T14:37:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9991.839762 secs 2024-04-26T14:37:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:34 smithi096 ceph-mon[32753]: pgmap v5913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9994.617313 secs 2024-04-26T14:37:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9991.839762 secs 2024-04-26T14:37:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:34 smithi194 ceph-mon[33017]: pgmap v5913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9994.617313 secs 2024-04-26T14:37:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:36 smithi033 ceph-mon[27845]: pgmap v5914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:36 smithi096 ceph-mon[32753]: pgmap v5914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:36 smithi194 ceph-mon[33017]: pgmap v5914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:38 smithi033 ceph-mon[27845]: pgmap v5915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:38 smithi096 ceph-mon[32753]: pgmap v5915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:38 smithi194 ceph-mon[33017]: pgmap v5915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 9996.839851 secs 2024-04-26T14:37:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 9999.617433 secs 2024-04-26T14:37:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 9996.839851 secs 2024-04-26T14:37:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 9999.617433 secs 2024-04-26T14:37:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 9996.839851 secs 2024-04-26T14:37:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 9999.617433 secs 2024-04-26T14:37:40.122 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:40.122+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:40.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:40.122+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:40.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:40.122+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:37:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:40 smithi033 ceph-mon[27845]: pgmap v5916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:40 smithi096 ceph-mon[32753]: pgmap v5916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:40 smithi194 ceph-mon[33017]: pgmap v5916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:42 smithi033 ceph-mon[27845]: pgmap v5917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:42 smithi096 ceph-mon[32753]: pgmap v5917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:42 smithi194 ceph-mon[33017]: pgmap v5917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10001.839937 secs 2024-04-26T14:37:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:44 smithi033 ceph-mon[27845]: pgmap v5918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10004.617590 secs 2024-04-26T14:37:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10001.839937 secs 2024-04-26T14:37:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:44 smithi096 ceph-mon[32753]: pgmap v5918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10004.617590 secs 2024-04-26T14:37:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10001.839937 secs 2024-04-26T14:37:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:44 smithi194 ceph-mon[33017]: pgmap v5918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10004.617590 secs 2024-04-26T14:37:46.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:46 smithi194 ceph-mon[33017]: pgmap v5919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:46 smithi033 ceph-mon[27845]: pgmap v5919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:46 smithi096 ceph-mon[32753]: pgmap v5919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:48 smithi033 ceph-mon[27845]: pgmap v5920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:48 smithi096 ceph-mon[32753]: pgmap v5920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:48 smithi194 ceph-mon[33017]: pgmap v5920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10006.840025 secs 2024-04-26T14:37:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10009.617710 secs 2024-04-26T14:37:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10006.840025 secs 2024-04-26T14:37:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10009.617710 secs 2024-04-26T14:37:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10006.840025 secs 2024-04-26T14:37:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10009.617710 secs 2024-04-26T14:37:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:50 smithi033 ceph-mon[27845]: pgmap v5921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:50 smithi096 ceph-mon[32753]: pgmap v5921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:50 smithi194 ceph-mon[33017]: pgmap v5921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:52 smithi096 ceph-mon[32753]: pgmap v5922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:37:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:52 smithi194 ceph-mon[33017]: pgmap v5922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:37:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:52 smithi033 ceph-mon[27845]: pgmap v5922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:37:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10011.840085 secs 2024-04-26T14:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:54 smithi096 ceph-mon[32753]: pgmap v5923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10014.617774 secs 2024-04-26T14:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10011.840085 secs 2024-04-26T14:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:54 smithi194 ceph-mon[33017]: pgmap v5923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10014.617774 secs 2024-04-26T14:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10011.840085 secs 2024-04-26T14:37:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:54 smithi033 ceph-mon[27845]: pgmap v5923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10014.617774 secs 2024-04-26T14:37:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:55.136+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:55.136+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:37:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:37:55.136+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:37:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:56 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:37:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:56 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:37:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:56 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:37:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:56 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:37:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:56 smithi194 ceph-mon[33017]: pgmap v5924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:56 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:37:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:56 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:37:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:56 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:37:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:56 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:37:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:56 smithi033 ceph-mon[27845]: pgmap v5924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:56 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:37:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:56 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:37:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:56 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:37:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:56 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:37:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:56 smithi096 ceph-mon[32753]: pgmap v5924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:37:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:57 smithi194 ceph-mon[33017]: pgmap v5925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:57 smithi033 ceph-mon[27845]: pgmap v5925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:57 smithi096 ceph-mon[32753]: pgmap v5925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:37:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: pgmap v5926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: pgmap v5927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: pgmap v5926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: pgmap v5927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:37:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:37:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:37:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: pgmap v5926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:37:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: pgmap v5927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:37:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:37:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10016.840149 secs 2024-04-26T14:37:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10019.617890 secs 2024-04-26T14:37:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10016.840149 secs 2024-04-26T14:37:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10019.617890 secs 2024-04-26T14:37:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10016.840149 secs 2024-04-26T14:37:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10019.617890 secs 2024-04-26T14:38:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:00 smithi194 ceph-mon[33017]: pgmap v5928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:00 smithi033 ceph-mon[27845]: pgmap v5928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:00 smithi096 ceph-mon[32753]: pgmap v5928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:02 smithi194 ceph-mon[33017]: pgmap v5929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:02 smithi033 ceph-mon[27845]: pgmap v5929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:02 smithi096 ceph-mon[32753]: pgmap v5929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10021.840205 secs 2024-04-26T14:38:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10024.618021 secs 2024-04-26T14:38:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:04 smithi194 ceph-mon[33017]: pgmap v5930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:38:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10021.840205 secs 2024-04-26T14:38:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10024.618021 secs 2024-04-26T14:38:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:04 smithi033 ceph-mon[27845]: pgmap v5930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:38:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10021.840205 secs 2024-04-26T14:38:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10024.618021 secs 2024-04-26T14:38:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:04 smithi096 ceph-mon[32753]: pgmap v5930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:38:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:06 smithi194 ceph-mon[33017]: pgmap v5931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:06 smithi033 ceph-mon[27845]: pgmap v5931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:06 smithi096 ceph-mon[32753]: pgmap v5931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:38:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:08 smithi194 ceph-mon[33017]: pgmap v5932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:38:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:08 smithi033 ceph-mon[27845]: pgmap v5932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:38:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:08 smithi096 ceph-mon[32753]: pgmap v5932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:38:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10026.840264 secs 2024-04-26T14:38:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10029.618166 secs 2024-04-26T14:38:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10026.840264 secs 2024-04-26T14:38:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10029.618166 secs 2024-04-26T14:38:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10026.840264 secs 2024-04-26T14:38:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10029.618166 secs 2024-04-26T14:38:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:10.140+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:10.140+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:10.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:10.140+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:38:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:10 smithi194 ceph-mon[33017]: pgmap v5933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:10 smithi033 ceph-mon[27845]: pgmap v5933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:10 smithi096 ceph-mon[32753]: pgmap v5933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:12 smithi194 ceph-mon[33017]: pgmap v5934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:12 smithi033 ceph-mon[27845]: pgmap v5934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:12 smithi096 ceph-mon[32753]: pgmap v5934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10031.840324 secs 2024-04-26T14:38:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10034.618286 secs 2024-04-26T14:38:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:14 smithi033 ceph-mon[27845]: pgmap v5935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10031.840324 secs 2024-04-26T14:38:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10034.618286 secs 2024-04-26T14:38:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:14 smithi194 ceph-mon[33017]: pgmap v5935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10031.840324 secs 2024-04-26T14:38:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10034.618286 secs 2024-04-26T14:38:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:14 smithi096 ceph-mon[32753]: pgmap v5935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:16.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:16 smithi096 ceph-mon[32753]: pgmap v5936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:16 smithi194 ceph-mon[33017]: pgmap v5936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:16 smithi033 ceph-mon[27845]: pgmap v5936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:18 smithi194 ceph-mon[33017]: pgmap v5937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:18 smithi033 ceph-mon[27845]: pgmap v5937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:18 smithi096 ceph-mon[32753]: pgmap v5937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10036.840414 secs 2024-04-26T14:38:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10039.618406 secs 2024-04-26T14:38:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10036.840414 secs 2024-04-26T14:38:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10039.618406 secs 2024-04-26T14:38:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10036.840414 secs 2024-04-26T14:38:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10039.618406 secs 2024-04-26T14:38:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:20 smithi194 ceph-mon[33017]: pgmap v5938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:20 smithi033 ceph-mon[27845]: pgmap v5938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:20 smithi096 ceph-mon[32753]: pgmap v5938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:22 smithi194 ceph-mon[33017]: pgmap v5939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:22 smithi033 ceph-mon[27845]: pgmap v5939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:22 smithi096 ceph-mon[32753]: pgmap v5939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:23.498+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:38:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:23.498+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:38:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:23.499+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:38:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:23.499+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x7fef00077780 2024-04-26T14:38:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10041.840503 secs 2024-04-26T14:38:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10044.618479 secs 2024-04-26T14:38:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:24 smithi194 ceph-mon[33017]: pgmap v5940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10041.840503 secs 2024-04-26T14:38:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10044.618479 secs 2024-04-26T14:38:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:24 smithi033 ceph-mon[27845]: pgmap v5940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10041.840503 secs 2024-04-26T14:38:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10044.618479 secs 2024-04-26T14:38:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:24 smithi096 ceph-mon[32753]: pgmap v5940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:25.145+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:25.146+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:25.146+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:38:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:26 smithi194 ceph-mon[33017]: pgmap v5941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:26 smithi033 ceph-mon[27845]: pgmap v5941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:26 smithi096 ceph-mon[32753]: pgmap v5941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:28 smithi194 ceph-mon[33017]: pgmap v5942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:28 smithi033 ceph-mon[27845]: pgmap v5942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:28 smithi096 ceph-mon[32753]: pgmap v5942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:29.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10046.840589 secs 2024-04-26T14:38:29.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10049.618594 secs 2024-04-26T14:38:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10046.840589 secs 2024-04-26T14:38:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10049.618594 secs 2024-04-26T14:38:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10046.840589 secs 2024-04-26T14:38:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10049.618594 secs 2024-04-26T14:38:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:30 smithi194 ceph-mon[33017]: pgmap v5943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:30 smithi033 ceph-mon[27845]: pgmap v5943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:30 smithi096 ceph-mon[32753]: pgmap v5943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:32 smithi194 ceph-mon[33017]: pgmap v5944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:32 smithi033 ceph-mon[27845]: pgmap v5944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:32 smithi096 ceph-mon[32753]: pgmap v5944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10051.840676 secs 2024-04-26T14:38:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10051.840676 secs 2024-04-26T14:38:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10051.840676 secs 2024-04-26T14:38:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10054.618690 secs 2024-04-26T14:38:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:34 smithi194 ceph-mon[33017]: pgmap v5945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10054.618690 secs 2024-04-26T14:38:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:34 smithi033 ceph-mon[27845]: pgmap v5945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10054.618690 secs 2024-04-26T14:38:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:34 smithi096 ceph-mon[32753]: pgmap v5945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:36 smithi194 ceph-mon[33017]: pgmap v5946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:36 smithi033 ceph-mon[27845]: pgmap v5946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:36 smithi096 ceph-mon[32753]: pgmap v5946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:38 smithi194 ceph-mon[33017]: pgmap v5947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10056.840739 secs 2024-04-26T14:38:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:38 smithi033 ceph-mon[27845]: pgmap v5947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10056.840739 secs 2024-04-26T14:38:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:38 smithi096 ceph-mon[32753]: pgmap v5947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10056.840739 secs 2024-04-26T14:38:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10059.618773 secs 2024-04-26T14:38:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10059.618773 secs 2024-04-26T14:38:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10059.618773 secs 2024-04-26T14:38:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:40.160+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:40.160+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:38:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:40.160+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:40 smithi033 ceph-mon[27845]: pgmap v5948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:40 smithi096 ceph-mon[32753]: pgmap v5948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:40 smithi194 ceph-mon[33017]: pgmap v5948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:42 smithi033 ceph-mon[27845]: pgmap v5949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:42 smithi096 ceph-mon[32753]: pgmap v5949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:42 smithi194 ceph-mon[33017]: pgmap v5949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10061.840826 secs 2024-04-26T14:38:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10064.618887 secs 2024-04-26T14:38:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10061.840826 secs 2024-04-26T14:38:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10064.618887 secs 2024-04-26T14:38:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10061.840826 secs 2024-04-26T14:38:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10064.618887 secs 2024-04-26T14:38:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:44 smithi033 ceph-mon[27845]: pgmap v5950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:44 smithi096 ceph-mon[32753]: pgmap v5950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:44 smithi194 ceph-mon[33017]: pgmap v5950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:46.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:46 smithi194 ceph-mon[33017]: pgmap v5951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:46 smithi033 ceph-mon[27845]: pgmap v5951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:46 smithi096 ceph-mon[32753]: pgmap v5951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:48 smithi033 ceph-mon[27845]: pgmap v5952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10066.840885 secs 2024-04-26T14:38:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10069.619032 secs 2024-04-26T14:38:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:48 smithi096 ceph-mon[32753]: pgmap v5952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10066.840885 secs 2024-04-26T14:38:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10069.619032 secs 2024-04-26T14:38:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:48 smithi194 ceph-mon[33017]: pgmap v5952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10066.840885 secs 2024-04-26T14:38:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10069.619032 secs 2024-04-26T14:38:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:50 smithi033 ceph-mon[27845]: pgmap v5953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:50 smithi096 ceph-mon[32753]: pgmap v5953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:50 smithi194 ceph-mon[33017]: pgmap v5953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:52 smithi033 ceph-mon[27845]: pgmap v5954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:52 smithi096 ceph-mon[32753]: pgmap v5954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:52 smithi194 ceph-mon[33017]: pgmap v5954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10071.840997 secs 2024-04-26T14:38:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10074.619148 secs 2024-04-26T14:38:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10071.840997 secs 2024-04-26T14:38:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10074.619148 secs 2024-04-26T14:38:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10071.840997 secs 2024-04-26T14:38:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10074.619148 secs 2024-04-26T14:38:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:54 smithi033 ceph-mon[27845]: pgmap v5955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:54 smithi096 ceph-mon[32753]: pgmap v5955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:54 smithi194 ceph-mon[33017]: pgmap v5955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:55.175+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:55.175+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:38:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:38:55.175+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:38:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:56 smithi033 ceph-mon[27845]: pgmap v5956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:56 smithi096 ceph-mon[32753]: pgmap v5956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:56 smithi194 ceph-mon[33017]: pgmap v5956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:38:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:58 smithi096 ceph-mon[32753]: pgmap v5957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10076.841087 secs 2024-04-26T14:38:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:38:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10079.619240 secs 2024-04-26T14:38:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:58 smithi194 ceph-mon[33017]: pgmap v5957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10076.841087 secs 2024-04-26T14:38:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:38:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10079.619240 secs 2024-04-26T14:38:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:58 smithi033 ceph-mon[27845]: pgmap v5957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:38:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10076.841087 secs 2024-04-26T14:38:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:38:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10079.619240 secs 2024-04-26T14:39:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:39:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:39:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:59 smithi033 ceph-mon[27845]: pgmap v5958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:59 smithi033 ceph-mon[27845]: pgmap v5959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:39:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:38:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:39:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:39:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:39:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:59 smithi194 ceph-mon[33017]: pgmap v5958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:59 smithi194 ceph-mon[33017]: pgmap v5959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:39:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:38:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:39:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:39:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:39:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:59 smithi096 ceph-mon[32753]: pgmap v5958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:59 smithi096 ceph-mon[32753]: pgmap v5959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:39:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:38:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:39:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:02 smithi033 ceph-mon[27845]: pgmap v5960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:02 smithi096 ceph-mon[32753]: pgmap v5960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:02 smithi194 ceph-mon[33017]: pgmap v5960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10081.841178 secs 2024-04-26T14:39:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:04 smithi033 ceph-mon[27845]: pgmap v5961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10084.619364 secs 2024-04-26T14:39:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10081.841178 secs 2024-04-26T14:39:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:04 smithi096 ceph-mon[32753]: pgmap v5961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10084.619364 secs 2024-04-26T14:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10081.841178 secs 2024-04-26T14:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:04 smithi194 ceph-mon[33017]: pgmap v5961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10084.619364 secs 2024-04-26T14:39:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:06 smithi033 ceph-mon[27845]: pgmap v5962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:06 smithi096 ceph-mon[32753]: pgmap v5962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:06 smithi194 ceph-mon[33017]: pgmap v5962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:08 smithi033 ceph-mon[27845]: pgmap v5963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:08 smithi096 ceph-mon[32753]: pgmap v5963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:08 smithi194 ceph-mon[33017]: pgmap v5963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:39:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10086.841264 secs 2024-04-26T14:39:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10089.619466 secs 2024-04-26T14:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10086.841264 secs 2024-04-26T14:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10089.619466 secs 2024-04-26T14:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10086.841264 secs 2024-04-26T14:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10089.619466 secs 2024-04-26T14:39:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:10.190+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:10.190+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:10.190+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:39:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:10 smithi096 ceph-mon[32753]: pgmap v5964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:39:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:10 smithi194 ceph-mon[33017]: pgmap v5964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:39:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:10 smithi033 ceph-mon[27845]: pgmap v5964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:39:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:12 smithi194 ceph-mon[33017]: pgmap v5965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:12 smithi033 ceph-mon[27845]: pgmap v5965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:12 smithi096 ceph-mon[32753]: pgmap v5965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10091.841327 secs 2024-04-26T14:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:14 smithi194 ceph-mon[33017]: pgmap v5966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10094.619550 secs 2024-04-26T14:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10091.841327 secs 2024-04-26T14:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:14 smithi033 ceph-mon[27845]: pgmap v5966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10094.619550 secs 2024-04-26T14:39:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10091.841327 secs 2024-04-26T14:39:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:14 smithi096 ceph-mon[32753]: pgmap v5966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10094.619550 secs 2024-04-26T14:39:16.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:16 smithi096 ceph-mon[32753]: pgmap v5967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:16 smithi194 ceph-mon[33017]: pgmap v5967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:16 smithi033 ceph-mon[27845]: pgmap v5967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:18 smithi194 ceph-mon[33017]: pgmap v5968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:18 smithi033 ceph-mon[27845]: pgmap v5968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:18 smithi096 ceph-mon[32753]: pgmap v5968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10096.841409 secs 2024-04-26T14:39:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10099.619668 secs 2024-04-26T14:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10096.841409 secs 2024-04-26T14:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10099.619668 secs 2024-04-26T14:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10096.841409 secs 2024-04-26T14:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10099.619668 secs 2024-04-26T14:39:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:20 smithi194 ceph-mon[33017]: pgmap v5969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:20 smithi033 ceph-mon[27845]: pgmap v5969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:20 smithi096 ceph-mon[32753]: pgmap v5969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:21 smithi194 ceph-mon[33017]: pgmap v5970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:21 smithi033 ceph-mon[27845]: pgmap v5970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:21 smithi096 ceph-mon[32753]: pgmap v5970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10101.841492 secs 2024-04-26T14:39:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:24 smithi033 ceph-mon[27845]: pgmap v5971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10104.619796 secs 2024-04-26T14:39:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10101.841492 secs 2024-04-26T14:39:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:24 smithi096 ceph-mon[32753]: pgmap v5971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10104.619796 secs 2024-04-26T14:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10101.841492 secs 2024-04-26T14:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:24 smithi194 ceph-mon[33017]: pgmap v5971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10104.619796 secs 2024-04-26T14:39:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:25.207+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:25.207+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:25.207+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:39:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:26 smithi033 ceph-mon[27845]: pgmap v5972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:26 smithi096 ceph-mon[32753]: pgmap v5972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:26 smithi194 ceph-mon[33017]: pgmap v5972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:28 smithi033 ceph-mon[27845]: pgmap v5973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:28 smithi096 ceph-mon[32753]: pgmap v5973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:28 smithi194 ceph-mon[33017]: pgmap v5973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10106.841578 secs 2024-04-26T14:39:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10109.619943 secs 2024-04-26T14:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10106.841578 secs 2024-04-26T14:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10109.619943 secs 2024-04-26T14:39:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10106.841578 secs 2024-04-26T14:39:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10109.619943 secs 2024-04-26T14:39:30.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:30 smithi033 ceph-mon[27845]: pgmap v5974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:30 smithi194 ceph-mon[33017]: pgmap v5974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:30 smithi096 ceph-mon[32753]: pgmap v5974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:32 smithi194 ceph-mon[33017]: pgmap v5975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:32 smithi033 ceph-mon[27845]: pgmap v5975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:32 smithi096 ceph-mon[32753]: pgmap v5975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10111.841679 secs 2024-04-26T14:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:34 smithi194 ceph-mon[33017]: pgmap v5976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10114.620067 secs 2024-04-26T14:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10111.841679 secs 2024-04-26T14:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:34 smithi033 ceph-mon[27845]: pgmap v5976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10114.620067 secs 2024-04-26T14:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10111.841679 secs 2024-04-26T14:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:34 smithi096 ceph-mon[32753]: pgmap v5976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10114.620067 secs 2024-04-26T14:39:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:36 smithi194 ceph-mon[33017]: pgmap v5977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:36 smithi033 ceph-mon[27845]: pgmap v5977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:36 smithi096 ceph-mon[32753]: pgmap v5977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:38 smithi194 ceph-mon[33017]: pgmap v5978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:38 smithi033 ceph-mon[27845]: pgmap v5978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:38 smithi096 ceph-mon[32753]: pgmap v5978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10116.841784 secs 2024-04-26T14:39:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10119.620196 secs 2024-04-26T14:39:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10116.841784 secs 2024-04-26T14:39:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10119.620196 secs 2024-04-26T14:39:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10116.841784 secs 2024-04-26T14:39:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10119.620196 secs 2024-04-26T14:39:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:40.222+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:40.222+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:39:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:40.222+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:40 smithi194 ceph-mon[33017]: pgmap v5979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:40 smithi033 ceph-mon[27845]: pgmap v5979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:40 smithi096 ceph-mon[32753]: pgmap v5979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:41 smithi194 ceph-mon[33017]: pgmap v5980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:41 smithi033 ceph-mon[27845]: pgmap v5980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:41 smithi096 ceph-mon[32753]: pgmap v5980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10121.841874 secs 2024-04-26T14:39:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:44 smithi033 ceph-mon[27845]: pgmap v5981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10124.620321 secs 2024-04-26T14:39:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10121.841874 secs 2024-04-26T14:39:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:44 smithi096 ceph-mon[32753]: pgmap v5981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10124.620321 secs 2024-04-26T14:39:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10121.841874 secs 2024-04-26T14:39:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:44 smithi194 ceph-mon[33017]: pgmap v5981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10124.620321 secs 2024-04-26T14:39:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:46 smithi096 ceph-mon[32753]: pgmap v5982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:46 smithi194 ceph-mon[33017]: pgmap v5982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:46 smithi033 ceph-mon[27845]: pgmap v5982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:48 smithi096 ceph-mon[32753]: pgmap v5983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:48 smithi194 ceph-mon[33017]: pgmap v5983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:48 smithi033 ceph-mon[27845]: pgmap v5983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10126.841979 secs 2024-04-26T14:39:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10129.620405 secs 2024-04-26T14:39:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10126.841979 secs 2024-04-26T14:39:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10129.620405 secs 2024-04-26T14:39:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10126.841979 secs 2024-04-26T14:39:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10129.620405 secs 2024-04-26T14:39:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:50 smithi194 ceph-mon[33017]: pgmap v5984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:50 smithi033 ceph-mon[27845]: pgmap v5984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:50 smithi096 ceph-mon[32753]: pgmap v5984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:52 smithi194 ceph-mon[33017]: pgmap v5985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:52 smithi033 ceph-mon[27845]: pgmap v5985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:52 smithi096 ceph-mon[32753]: pgmap v5985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10131.842004 secs 2024-04-26T14:39:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:54 smithi194 ceph-mon[33017]: pgmap v5986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10134.620526 secs 2024-04-26T14:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10131.842004 secs 2024-04-26T14:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:54 smithi033 ceph-mon[27845]: pgmap v5986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10134.620526 secs 2024-04-26T14:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10131.842004 secs 2024-04-26T14:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:54 smithi096 ceph-mon[32753]: pgmap v5986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10134.620526 secs 2024-04-26T14:39:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:55.229+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:55.229+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:39:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:39:55.229+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:39:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:56 smithi194 ceph-mon[33017]: pgmap v5987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:56 smithi033 ceph-mon[27845]: pgmap v5987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:56 smithi096 ceph-mon[32753]: pgmap v5987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:39:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:58 smithi194 ceph-mon[33017]: pgmap v5988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:58 smithi033 ceph-mon[27845]: pgmap v5988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:58 smithi096 ceph-mon[32753]: pgmap v5988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:39:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10136.842065 secs 2024-04-26T14:39:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10139.620655 secs 2024-04-26T14:39:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10136.842065 secs 2024-04-26T14:39:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10139.620655 secs 2024-04-26T14:39:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10136.842065 secs 2024-04-26T14:39:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10139.620655 secs 2024-04-26T14:40:00.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:00 smithi033 ceph-mon[27845]: pgmap v5989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:00.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:40:00.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:00 smithi194 ceph-mon[33017]: pgmap v5989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:00 smithi096 ceph-mon[32753]: pgmap v5989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:40:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:40:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:40:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:01 smithi033 ceph-mon[27845]: pgmap v5990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:40:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:40:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:40:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:01 smithi194 ceph-mon[33017]: pgmap v5990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:40:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:40:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:40:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:01 smithi096 ceph-mon[32753]: pgmap v5990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:40:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10141.842123 secs 2024-04-26T14:40:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:03 smithi194 ceph-mon[33017]: pgmap v5991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10144.620780 secs 2024-04-26T14:40:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10141.842123 secs 2024-04-26T14:40:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:03 smithi033 ceph-mon[27845]: pgmap v5991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10144.620780 secs 2024-04-26T14:40:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10141.842123 secs 2024-04-26T14:40:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:03 smithi096 ceph-mon[32753]: pgmap v5991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10144.620780 secs 2024-04-26T14:40:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:05 smithi194 ceph-mon[33017]: pgmap v5992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:05 smithi033 ceph-mon[27845]: pgmap v5992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:05 smithi096 ceph-mon[32753]: pgmap v5992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:07 smithi194 ceph-mon[33017]: pgmap v5993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:07 smithi033 ceph-mon[27845]: pgmap v5993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:07 smithi096 ceph-mon[32753]: pgmap v5993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10146.842212 secs 2024-04-26T14:40:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10149.620908 secs 2024-04-26T14:40:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10146.842212 secs 2024-04-26T14:40:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10149.620908 secs 2024-04-26T14:40:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10146.842212 secs 2024-04-26T14:40:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10149.620908 secs 2024-04-26T14:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:09 smithi033 ceph-mon[27845]: pgmap v5994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:09 smithi096 ceph-mon[32753]: pgmap v5994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:10.231+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:10.231+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:40:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:10.231+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:09 smithi194 ceph-mon[33017]: pgmap v5994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:11 smithi033 ceph-mon[27845]: pgmap v5995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:11 smithi096 ceph-mon[32753]: pgmap v5995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:11 smithi194 ceph-mon[33017]: pgmap v5995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:40:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10151.842297 secs 2024-04-26T14:40:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:13 smithi033 ceph-mon[27845]: pgmap v5996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10154.621038 secs 2024-04-26T14:40:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10151.842297 secs 2024-04-26T14:40:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:13 smithi096 ceph-mon[32753]: pgmap v5996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10154.621038 secs 2024-04-26T14:40:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10151.842297 secs 2024-04-26T14:40:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:13 smithi194 ceph-mon[33017]: pgmap v5996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10154.621038 secs 2024-04-26T14:40:16.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:15 smithi194 ceph-mon[33017]: pgmap v5997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:15 smithi033 ceph-mon[27845]: pgmap v5997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:16.284 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:15 smithi096 ceph-mon[32753]: pgmap v5997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:17 smithi033 ceph-mon[27845]: pgmap v5998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:17 smithi096 ceph-mon[32753]: pgmap v5998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:17 smithi194 ceph-mon[33017]: pgmap v5998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10156.842381 secs 2024-04-26T14:40:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10159.621096 secs 2024-04-26T14:40:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10156.842381 secs 2024-04-26T14:40:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10159.621096 secs 2024-04-26T14:40:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10156.842381 secs 2024-04-26T14:40:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10159.621096 secs 2024-04-26T14:40:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:19 smithi033 ceph-mon[27845]: pgmap v5999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:19 smithi096 ceph-mon[32753]: pgmap v5999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:19 smithi194 ceph-mon[33017]: pgmap v5999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:21 smithi033 ceph-mon[27845]: pgmap v6000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:21 smithi096 ceph-mon[32753]: pgmap v6000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:21 smithi194 ceph-mon[33017]: pgmap v6000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10161.842465 secs 2024-04-26T14:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:23 smithi033 ceph-mon[27845]: pgmap v6001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10164.621224 secs 2024-04-26T14:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10161.842465 secs 2024-04-26T14:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:23 smithi096 ceph-mon[32753]: pgmap v6001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10164.621224 secs 2024-04-26T14:40:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10161.842465 secs 2024-04-26T14:40:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:23 smithi194 ceph-mon[33017]: pgmap v6001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10164.621224 secs 2024-04-26T14:40:25.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:25.241+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:25.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:25.241+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:25.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:25.241+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:40:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:25 smithi033 ceph-mon[27845]: pgmap v6002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:25 smithi096 ceph-mon[32753]: pgmap v6002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:25 smithi194 ceph-mon[33017]: pgmap v6002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:27 smithi033 ceph-mon[27845]: pgmap v6003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:27 smithi096 ceph-mon[32753]: pgmap v6003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:27 smithi194 ceph-mon[33017]: pgmap v6003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10166.842551 secs 2024-04-26T14:40:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10169.621350 secs 2024-04-26T14:40:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10166.842551 secs 2024-04-26T14:40:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10169.621350 secs 2024-04-26T14:40:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10166.842551 secs 2024-04-26T14:40:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10169.621350 secs 2024-04-26T14:40:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:29 smithi096 ceph-mon[32753]: pgmap v6004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:29 smithi194 ceph-mon[33017]: pgmap v6004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:29 smithi033 ceph-mon[27845]: pgmap v6004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:31 smithi096 ceph-mon[32753]: pgmap v6005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:31 smithi194 ceph-mon[33017]: pgmap v6005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:31 smithi033 ceph-mon[27845]: pgmap v6005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10171.842636 secs 2024-04-26T14:40:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:33 smithi194 ceph-mon[33017]: pgmap v6006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10174.621485 secs 2024-04-26T14:40:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10171.842636 secs 2024-04-26T14:40:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:33 smithi033 ceph-mon[27845]: pgmap v6006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10174.621485 secs 2024-04-26T14:40:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10171.842636 secs 2024-04-26T14:40:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:33 smithi096 ceph-mon[32753]: pgmap v6006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10174.621485 secs 2024-04-26T14:40:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:35 smithi194 ceph-mon[33017]: pgmap v6007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:35 smithi033 ceph-mon[27845]: pgmap v6007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:35 smithi096 ceph-mon[32753]: pgmap v6007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:37 smithi194 ceph-mon[33017]: pgmap v6008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:37 smithi033 ceph-mon[27845]: pgmap v6008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:37 smithi096 ceph-mon[32753]: pgmap v6008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10176.842721 secs 2024-04-26T14:40:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10179.621575 secs 2024-04-26T14:40:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10176.842721 secs 2024-04-26T14:40:39.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10179.621575 secs 2024-04-26T14:40:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10176.842721 secs 2024-04-26T14:40:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10179.621575 secs 2024-04-26T14:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:40.256+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:40.256+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:40.256+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:40:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:39 smithi194 ceph-mon[33017]: pgmap v6009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:39 smithi033 ceph-mon[27845]: pgmap v6009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:39 smithi096 ceph-mon[32753]: pgmap v6009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:41 smithi194 ceph-mon[33017]: pgmap v6010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:41 smithi033 ceph-mon[27845]: pgmap v6010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:41 smithi096 ceph-mon[32753]: pgmap v6010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10181.842779 secs 2024-04-26T14:40:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:43 smithi194 ceph-mon[33017]: pgmap v6011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10184.621657 secs 2024-04-26T14:40:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10181.842779 secs 2024-04-26T14:40:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:43 smithi033 ceph-mon[27845]: pgmap v6011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10184.621657 secs 2024-04-26T14:40:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10181.842779 secs 2024-04-26T14:40:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:43 smithi096 ceph-mon[32753]: pgmap v6011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10184.621657 secs 2024-04-26T14:40:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:45 smithi194 ceph-mon[33017]: pgmap v6012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:45 smithi033 ceph-mon[27845]: pgmap v6012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:45 smithi096 ceph-mon[32753]: pgmap v6012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:48.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:47 smithi096 ceph-mon[32753]: pgmap v6013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:47 smithi194 ceph-mon[33017]: pgmap v6013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:47 smithi033 ceph-mon[27845]: pgmap v6013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10186.842837 secs 2024-04-26T14:40:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10189.621800 secs 2024-04-26T14:40:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10186.842837 secs 2024-04-26T14:40:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10189.621800 secs 2024-04-26T14:40:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10186.842837 secs 2024-04-26T14:40:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10189.621800 secs 2024-04-26T14:40:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:49 smithi194 ceph-mon[33017]: pgmap v6014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:49 smithi033 ceph-mon[27845]: pgmap v6014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:49 smithi096 ceph-mon[32753]: pgmap v6014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:51 smithi033 ceph-mon[27845]: pgmap v6015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:51 smithi096 ceph-mon[32753]: pgmap v6015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:51 smithi194 ceph-mon[33017]: pgmap v6015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10191.842919 secs 2024-04-26T14:40:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:53 smithi033 ceph-mon[27845]: pgmap v6016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10194.621924 secs 2024-04-26T14:40:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10191.842919 secs 2024-04-26T14:40:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:53 smithi096 ceph-mon[32753]: pgmap v6016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10194.621924 secs 2024-04-26T14:40:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10191.842919 secs 2024-04-26T14:40:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:53 smithi194 ceph-mon[33017]: pgmap v6016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10194.621924 secs 2024-04-26T14:40:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:55.268+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:55.268+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:40:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:40:55.268+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:40:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:55 smithi033 ceph-mon[27845]: pgmap v6017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:55 smithi096 ceph-mon[32753]: pgmap v6017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:55 smithi194 ceph-mon[33017]: pgmap v6017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:40:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:57 smithi033 ceph-mon[27845]: pgmap v6018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:57 smithi096 ceph-mon[32753]: pgmap v6018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:57 smithi194 ceph-mon[33017]: pgmap v6018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:40:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10196.843010 secs 2024-04-26T14:40:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10199.622058 secs 2024-04-26T14:40:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10196.843010 secs 2024-04-26T14:40:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10199.622058 secs 2024-04-26T14:40:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10196.843010 secs 2024-04-26T14:40:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10199.622058 secs 2024-04-26T14:41:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:40:59 smithi033 ceph-mon[27845]: pgmap v6019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:40:59 smithi096 ceph-mon[32753]: pgmap v6019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:40:59 smithi194 ceph-mon[33017]: pgmap v6019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:01 smithi033 ceph-mon[27845]: pgmap v6020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:41:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:01 smithi096 ceph-mon[32753]: pgmap v6020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:41:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:01 smithi194 ceph-mon[33017]: pgmap v6020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:41:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:41:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:41:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:41:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:41:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:41:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:41:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:41:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:41:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:41:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:03 smithi033 ceph-mon[27845]: pgmap v6021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:03 smithi033 ceph-mon[27845]: pgmap v6022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10201.843099 secs 2024-04-26T14:41:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10204.622186 secs 2024-04-26T14:41:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:03 smithi096 ceph-mon[32753]: pgmap v6021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:03 smithi096 ceph-mon[32753]: pgmap v6022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10201.843099 secs 2024-04-26T14:41:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10204.622186 secs 2024-04-26T14:41:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:03 smithi194 ceph-mon[33017]: pgmap v6021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:03 smithi194 ceph-mon[33017]: pgmap v6022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10201.843099 secs 2024-04-26T14:41:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10204.622186 secs 2024-04-26T14:41:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:05 smithi033 ceph-mon[27845]: pgmap v6023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:05 smithi096 ceph-mon[32753]: pgmap v6023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:05 smithi194 ceph-mon[33017]: pgmap v6023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:07 smithi033 ceph-mon[27845]: pgmap v6024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:07 smithi096 ceph-mon[32753]: pgmap v6024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:07 smithi194 ceph-mon[33017]: pgmap v6024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10206.843184 secs 2024-04-26T14:41:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10209.622344 secs 2024-04-26T14:41:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10206.843184 secs 2024-04-26T14:41:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10209.622344 secs 2024-04-26T14:41:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10206.843184 secs 2024-04-26T14:41:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10209.622344 secs 2024-04-26T14:41:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:10.282+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:10.282+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:10.282+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:41:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:09 smithi194 ceph-mon[33017]: pgmap v6025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:09 smithi033 ceph-mon[27845]: pgmap v6025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:09 smithi096 ceph-mon[32753]: pgmap v6025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:11 smithi194 ceph-mon[33017]: pgmap v6026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:11 smithi033 ceph-mon[27845]: pgmap v6026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:11 smithi096 ceph-mon[32753]: pgmap v6026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:41:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:13 smithi194 ceph-mon[33017]: pgmap v6027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:41:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10211.843270 secs 2024-04-26T14:41:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10214.622444 secs 2024-04-26T14:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:13 smithi033 ceph-mon[27845]: pgmap v6027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10211.843270 secs 2024-04-26T14:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10214.622444 secs 2024-04-26T14:41:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:13 smithi096 ceph-mon[32753]: pgmap v6027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:41:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10211.843270 secs 2024-04-26T14:41:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10214.622444 secs 2024-04-26T14:41:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:15 smithi194 ceph-mon[33017]: pgmap v6028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:15 smithi033 ceph-mon[27845]: pgmap v6028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:15 smithi096 ceph-mon[32753]: pgmap v6028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:18.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:17 smithi096 ceph-mon[32753]: pgmap v6029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:17 smithi194 ceph-mon[33017]: pgmap v6029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:17 smithi033 ceph-mon[27845]: pgmap v6029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10216.843352 secs 2024-04-26T14:41:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10219.622513 secs 2024-04-26T14:41:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10216.843352 secs 2024-04-26T14:41:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10219.622513 secs 2024-04-26T14:41:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10216.843352 secs 2024-04-26T14:41:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10219.622513 secs 2024-04-26T14:41:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:19 smithi194 ceph-mon[33017]: pgmap v6030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:19 smithi033 ceph-mon[27845]: pgmap v6030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:19 smithi096 ceph-mon[32753]: pgmap v6030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T14:41:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T14:41:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0057000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T14:41:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0057000 comp rx=0 tx=0).stop 2024-04-26T14:41:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T14:41:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T14:41:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T14:41:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.736+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T14:41:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.737+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T14:41:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:21.739+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3cc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef004f040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T14:41:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:21 smithi194 ceph-mon[33017]: pgmap v6031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:21 smithi033 ceph-mon[27845]: pgmap v6031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:21 smithi096 ceph-mon[32753]: pgmap v6031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:23.529+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:41:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:23.529+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:41:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:23.529+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:41:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:23.529+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:41:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:23 smithi194 ceph-mon[33017]: pgmap v6032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10221.843409 secs 2024-04-26T14:41:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10224.622635 secs 2024-04-26T14:41:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:23 smithi033 ceph-mon[27845]: pgmap v6032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10221.843409 secs 2024-04-26T14:41:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10224.622635 secs 2024-04-26T14:41:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:23 smithi096 ceph-mon[32753]: pgmap v6032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10221.843409 secs 2024-04-26T14:41:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10224.622635 secs 2024-04-26T14:41:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:25.286+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:25.286+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:25.286+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:41:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:25 smithi194 ceph-mon[33017]: pgmap v6033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:25 smithi033 ceph-mon[27845]: pgmap v6033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:25 smithi096 ceph-mon[32753]: pgmap v6033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:27 smithi194 ceph-mon[33017]: pgmap v6034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:27 smithi033 ceph-mon[27845]: pgmap v6034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:27 smithi096 ceph-mon[32753]: pgmap v6034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10226.843493 secs 2024-04-26T14:41:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10229.622782 secs 2024-04-26T14:41:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10226.843493 secs 2024-04-26T14:41:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10229.622782 secs 2024-04-26T14:41:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10226.843493 secs 2024-04-26T14:41:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10229.622782 secs 2024-04-26T14:41:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:30 smithi033 ceph-mon[27845]: pgmap v6035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:30 smithi194 ceph-mon[33017]: pgmap v6035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:30 smithi096 ceph-mon[32753]: pgmap v6035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:32 smithi194 ceph-mon[33017]: pgmap v6036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:32 smithi033 ceph-mon[27845]: pgmap v6036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:32 smithi096 ceph-mon[32753]: pgmap v6036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:34 smithi194 ceph-mon[33017]: pgmap v6037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10231.843582 secs 2024-04-26T14:41:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10234.622879 secs 2024-04-26T14:41:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:34 smithi033 ceph-mon[27845]: pgmap v6037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10231.843582 secs 2024-04-26T14:41:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10234.622879 secs 2024-04-26T14:41:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:34 smithi096 ceph-mon[32753]: pgmap v6037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10231.843582 secs 2024-04-26T14:41:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10234.622879 secs 2024-04-26T14:41:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:36 smithi194 ceph-mon[33017]: pgmap v6038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:36 smithi033 ceph-mon[27845]: pgmap v6038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:36 smithi096 ceph-mon[32753]: pgmap v6038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:38 smithi194 ceph-mon[33017]: pgmap v6039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:38 smithi033 ceph-mon[27845]: pgmap v6039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:38 smithi096 ceph-mon[32753]: pgmap v6039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10236.843658 secs 2024-04-26T14:41:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10239.623024 secs 2024-04-26T14:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10236.843658 secs 2024-04-26T14:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10239.623024 secs 2024-04-26T14:41:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10236.843658 secs 2024-04-26T14:41:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10239.623024 secs 2024-04-26T14:41:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:40.290+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:40.290+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:40.290+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:41:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:40 smithi194 ceph-mon[33017]: pgmap v6040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:40 smithi033 ceph-mon[27845]: pgmap v6040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:40 smithi096 ceph-mon[32753]: pgmap v6040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:42 smithi194 ceph-mon[33017]: pgmap v6041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:42 smithi033 ceph-mon[27845]: pgmap v6041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:42 smithi096 ceph-mon[32753]: pgmap v6041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:44 smithi194 ceph-mon[33017]: pgmap v6042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10241.843742 secs 2024-04-26T14:41:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10244.623149 secs 2024-04-26T14:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:44 smithi033 ceph-mon[27845]: pgmap v6042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10241.843742 secs 2024-04-26T14:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10244.623149 secs 2024-04-26T14:41:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:44 smithi096 ceph-mon[32753]: pgmap v6042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10241.843742 secs 2024-04-26T14:41:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10244.623149 secs 2024-04-26T14:41:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:46 smithi194 ceph-mon[33017]: pgmap v6043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:46 smithi033 ceph-mon[27845]: pgmap v6043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:46 smithi096 ceph-mon[32753]: pgmap v6043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:48 smithi194 ceph-mon[33017]: pgmap v6044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:48 smithi033 ceph-mon[27845]: pgmap v6044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:48 smithi096 ceph-mon[32753]: pgmap v6044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10246.843818 secs 2024-04-26T14:41:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10249.623236 secs 2024-04-26T14:41:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10246.843818 secs 2024-04-26T14:41:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10249.623236 secs 2024-04-26T14:41:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10246.843818 secs 2024-04-26T14:41:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10249.623236 secs 2024-04-26T14:41:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:50 smithi194 ceph-mon[33017]: pgmap v6045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:50 smithi033 ceph-mon[27845]: pgmap v6045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:50 smithi096 ceph-mon[32753]: pgmap v6045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:52 smithi033 ceph-mon[27845]: pgmap v6046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:52 smithi194 ceph-mon[33017]: pgmap v6046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:52 smithi096 ceph-mon[32753]: pgmap v6046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:54 smithi194 ceph-mon[33017]: pgmap v6047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10251.843904 secs 2024-04-26T14:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10254.623360 secs 2024-04-26T14:41:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:54 smithi033 ceph-mon[27845]: pgmap v6047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10251.843904 secs 2024-04-26T14:41:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10254.623360 secs 2024-04-26T14:41:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:54 smithi096 ceph-mon[32753]: pgmap v6047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10251.843904 secs 2024-04-26T14:41:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10254.623360 secs 2024-04-26T14:41:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:55.305+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:55.305+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:41:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:41:55.305+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:41:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:56 smithi194 ceph-mon[33017]: pgmap v6048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:56 smithi033 ceph-mon[27845]: pgmap v6048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:56 smithi096 ceph-mon[32753]: pgmap v6048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:41:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:58 smithi194 ceph-mon[33017]: pgmap v6049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:58 smithi033 ceph-mon[27845]: pgmap v6049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:58 smithi096 ceph-mon[32753]: pgmap v6049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:41:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10256.843998 secs 2024-04-26T14:41:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:41:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10259.623447 secs 2024-04-26T14:41:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10256.843998 secs 2024-04-26T14:41:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:41:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10259.623447 secs 2024-04-26T14:41:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10256.843998 secs 2024-04-26T14:41:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:41:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10259.623447 secs 2024-04-26T14:42:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:00 smithi194 ceph-mon[33017]: pgmap v6050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:00 smithi033 ceph-mon[27845]: pgmap v6050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:00 smithi096 ceph-mon[32753]: pgmap v6050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:02 smithi033 ceph-mon[27845]: pgmap v6051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:02 smithi096 ceph-mon[32753]: pgmap v6051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:02 smithi194 ceph-mon[33017]: pgmap v6051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:42:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:42:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:42:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:04 smithi033 ceph-mon[27845]: pgmap v6052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10261.844080 secs 2024-04-26T14:42:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10264.623538 secs 2024-04-26T14:42:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:04 smithi096 ceph-mon[32753]: pgmap v6052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10261.844080 secs 2024-04-26T14:42:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10264.623538 secs 2024-04-26T14:42:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:04 smithi194 ceph-mon[33017]: pgmap v6052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10261.844080 secs 2024-04-26T14:42:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10264.623538 secs 2024-04-26T14:42:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:42:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:42:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:05 smithi033 ceph-mon[27845]: pgmap v6053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:05 smithi033 ceph-mon[27845]: pgmap v6054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:42:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:05 smithi096 ceph-mon[32753]: pgmap v6053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:05 smithi096 ceph-mon[32753]: pgmap v6054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:42:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:42:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:42:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:05 smithi194 ceph-mon[33017]: pgmap v6053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:05 smithi194 ceph-mon[33017]: pgmap v6054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:42:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:42:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:07 smithi194 ceph-mon[33017]: pgmap v6055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:07 smithi033 ceph-mon[27845]: pgmap v6055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:07 smithi096 ceph-mon[32753]: pgmap v6055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10266.844162 secs 2024-04-26T14:42:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:09 smithi194 ceph-mon[33017]: pgmap v6056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10269.623669 secs 2024-04-26T14:42:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10266.844162 secs 2024-04-26T14:42:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:09 smithi033 ceph-mon[27845]: pgmap v6056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10269.623669 secs 2024-04-26T14:42:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10266.844162 secs 2024-04-26T14:42:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:09 smithi096 ceph-mon[32753]: pgmap v6056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10269.623669 secs 2024-04-26T14:42:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:10.311+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:10.311+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:10.311+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:42:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:11 smithi194 ceph-mon[33017]: pgmap v6057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:11 smithi033 ceph-mon[27845]: pgmap v6057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:11 smithi096 ceph-mon[32753]: pgmap v6057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:13 smithi194 ceph-mon[33017]: pgmap v6058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:13 smithi033 ceph-mon[27845]: pgmap v6058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:13 smithi096 ceph-mon[32753]: pgmap v6058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10271.844249 secs 2024-04-26T14:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10274.623793 secs 2024-04-26T14:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10271.844249 secs 2024-04-26T14:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10274.623793 secs 2024-04-26T14:42:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10271.844249 secs 2024-04-26T14:42:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10274.623793 secs 2024-04-26T14:42:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:15 smithi194 ceph-mon[33017]: pgmap v6059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:42:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:15 smithi033 ceph-mon[27845]: pgmap v6059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:42:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:15 smithi096 ceph-mon[32753]: pgmap v6059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:42:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:17 smithi194 ceph-mon[33017]: pgmap v6060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:17 smithi033 ceph-mon[27845]: pgmap v6060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:17 smithi096 ceph-mon[32753]: pgmap v6060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10276.844334 secs 2024-04-26T14:42:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:19 smithi194 ceph-mon[33017]: pgmap v6061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10279.623918 secs 2024-04-26T14:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10276.844334 secs 2024-04-26T14:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:19 smithi033 ceph-mon[27845]: pgmap v6061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10279.623918 secs 2024-04-26T14:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10276.844334 secs 2024-04-26T14:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:19 smithi096 ceph-mon[32753]: pgmap v6061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10279.623918 secs 2024-04-26T14:42:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:21 smithi194 ceph-mon[33017]: pgmap v6062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:21 smithi033 ceph-mon[27845]: pgmap v6062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:21 smithi096 ceph-mon[32753]: pgmap v6062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:23 smithi033 ceph-mon[27845]: pgmap v6063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:23 smithi194 ceph-mon[33017]: pgmap v6063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:23 smithi096 ceph-mon[32753]: pgmap v6063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10281.844435 secs 2024-04-26T14:42:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10284.624039 secs 2024-04-26T14:42:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:24 smithi194 ceph-mon[33017]: pgmap v6064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10281.844435 secs 2024-04-26T14:42:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10284.624039 secs 2024-04-26T14:42:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:24 smithi033 ceph-mon[27845]: pgmap v6064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10281.844435 secs 2024-04-26T14:42:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10284.624039 secs 2024-04-26T14:42:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:24 smithi096 ceph-mon[32753]: pgmap v6064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:25.327+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:25.327+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:25.327+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:42:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:27 smithi194 ceph-mon[33017]: pgmap v6065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:27 smithi033 ceph-mon[27845]: pgmap v6065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:27 smithi096 ceph-mon[32753]: pgmap v6065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10286.844532 secs 2024-04-26T14:42:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:29 smithi194 ceph-mon[33017]: pgmap v6066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10289.624171 secs 2024-04-26T14:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10286.844532 secs 2024-04-26T14:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:29 smithi033 ceph-mon[27845]: pgmap v6066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10289.624171 secs 2024-04-26T14:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10286.844532 secs 2024-04-26T14:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:29 smithi096 ceph-mon[32753]: pgmap v6066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10289.624171 secs 2024-04-26T14:42:31.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:31 smithi033 ceph-mon[27845]: pgmap v6067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:31 smithi194 ceph-mon[33017]: pgmap v6067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:31 smithi096 ceph-mon[32753]: pgmap v6067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:33 smithi194 ceph-mon[33017]: pgmap v6068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:33 smithi033 ceph-mon[27845]: pgmap v6068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:33 smithi096 ceph-mon[32753]: pgmap v6068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10291.844629 secs 2024-04-26T14:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10294.624293 secs 2024-04-26T14:42:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10291.844629 secs 2024-04-26T14:42:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10294.624293 secs 2024-04-26T14:42:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10291.844629 secs 2024-04-26T14:42:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10294.624293 secs 2024-04-26T14:42:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:35 smithi194 ceph-mon[33017]: pgmap v6069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:35 smithi033 ceph-mon[27845]: pgmap v6069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:35 smithi096 ceph-mon[32753]: pgmap v6069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:37 smithi194 ceph-mon[33017]: pgmap v6070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:37 smithi033 ceph-mon[27845]: pgmap v6070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:37 smithi096 ceph-mon[32753]: pgmap v6070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10296.844731 secs 2024-04-26T14:42:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:39 smithi033 ceph-mon[27845]: pgmap v6071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10299.624379 secs 2024-04-26T14:42:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10296.844731 secs 2024-04-26T14:42:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:39 smithi194 ceph-mon[33017]: pgmap v6071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10299.624379 secs 2024-04-26T14:42:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10296.844731 secs 2024-04-26T14:42:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:39 smithi096 ceph-mon[32753]: pgmap v6071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10299.624379 secs 2024-04-26T14:42:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:40.333+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:40.333+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:40.333+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:42:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:40 smithi194 ceph-mon[33017]: pgmap v6072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:40 smithi033 ceph-mon[27845]: pgmap v6072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:40 smithi096 ceph-mon[32753]: pgmap v6072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:43 smithi194 ceph-mon[33017]: pgmap v6073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:43 smithi033 ceph-mon[27845]: pgmap v6073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:43 smithi096 ceph-mon[32753]: pgmap v6073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10301.844818 secs 2024-04-26T14:42:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10304.624463 secs 2024-04-26T14:42:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10301.844818 secs 2024-04-26T14:42:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10304.624463 secs 2024-04-26T14:42:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10301.844818 secs 2024-04-26T14:42:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10304.624463 secs 2024-04-26T14:42:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:45 smithi194 ceph-mon[33017]: pgmap v6074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:45 smithi033 ceph-mon[27845]: pgmap v6074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:45 smithi096 ceph-mon[32753]: pgmap v6074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:47 smithi194 ceph-mon[33017]: pgmap v6075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:47 smithi033 ceph-mon[27845]: pgmap v6075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:47 smithi096 ceph-mon[32753]: pgmap v6075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10306.844918 secs 2024-04-26T14:42:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:49 smithi194 ceph-mon[33017]: pgmap v6076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10309.624568 secs 2024-04-26T14:42:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10306.844918 secs 2024-04-26T14:42:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:49 smithi033 ceph-mon[27845]: pgmap v6076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10309.624568 secs 2024-04-26T14:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10306.844918 secs 2024-04-26T14:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:49 smithi096 ceph-mon[32753]: pgmap v6076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10309.624568 secs 2024-04-26T14:42:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:51 smithi194 ceph-mon[33017]: pgmap v6077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:51 smithi033 ceph-mon[27845]: pgmap v6077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:51 smithi096 ceph-mon[32753]: pgmap v6077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:53.545+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:42:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:53.545+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:42:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:53.545+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:42:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:53.545+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:42:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:53 smithi194 ceph-mon[33017]: pgmap v6078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:53 smithi033 ceph-mon[27845]: pgmap v6078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:53 smithi096 ceph-mon[32753]: pgmap v6078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10311.845020 secs 2024-04-26T14:42:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10314.624698 secs 2024-04-26T14:42:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10311.845020 secs 2024-04-26T14:42:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10314.624698 secs 2024-04-26T14:42:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10311.845020 secs 2024-04-26T14:42:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10314.624698 secs 2024-04-26T14:42:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:55.341+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:55.341+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:42:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:42:55.341+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:42:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:55 smithi194 ceph-mon[33017]: pgmap v6079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:55 smithi033 ceph-mon[27845]: pgmap v6079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:55 smithi096 ceph-mon[32753]: pgmap v6079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:56 smithi194 ceph-mon[33017]: pgmap v6080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:56 smithi033 ceph-mon[27845]: pgmap v6080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:56 smithi096 ceph-mon[32753]: pgmap v6080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:42:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10316.845118 secs 2024-04-26T14:42:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:59 smithi194 ceph-mon[33017]: pgmap v6081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10319.624815 secs 2024-04-26T14:42:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10316.845118 secs 2024-04-26T14:42:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:59 smithi033 ceph-mon[27845]: pgmap v6081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10319.624815 secs 2024-04-26T14:42:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10316.845118 secs 2024-04-26T14:42:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:59 smithi096 ceph-mon[32753]: pgmap v6081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:42:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10319.624815 secs 2024-04-26T14:43:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:01 smithi194 ceph-mon[33017]: pgmap v6082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:01 smithi033 ceph-mon[27845]: pgmap v6082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:01 smithi096 ceph-mon[32753]: pgmap v6082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:03 smithi194 ceph-mon[33017]: pgmap v6083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:03 smithi033 ceph-mon[27845]: pgmap v6083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:03 smithi096 ceph-mon[32753]: pgmap v6083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10321.845202 secs 2024-04-26T14:43:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10324.624888 secs 2024-04-26T14:43:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10321.845202 secs 2024-04-26T14:43:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10324.624888 secs 2024-04-26T14:43:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10321.845202 secs 2024-04-26T14:43:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10324.624888 secs 2024-04-26T14:43:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:05 smithi194 ceph-mon[33017]: pgmap v6084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:43:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:05 smithi033 ceph-mon[27845]: pgmap v6084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:43:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:05 smithi096 ceph-mon[32753]: pgmap v6084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:43:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:43:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:43:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:43:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:43:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:43:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:43:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:43:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:43:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:43:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:07 smithi194 ceph-mon[33017]: pgmap v6085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:43:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:07 smithi194 ceph-mon[33017]: pgmap v6086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:07 smithi033 ceph-mon[27845]: pgmap v6085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:43:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:07 smithi033 ceph-mon[27845]: pgmap v6086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:07 smithi096 ceph-mon[32753]: pgmap v6085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:43:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:07 smithi096 ceph-mon[32753]: pgmap v6086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:08 smithi194 ceph-mon[33017]: pgmap v6087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:08 smithi033 ceph-mon[27845]: pgmap v6087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:08 smithi096 ceph-mon[32753]: pgmap v6087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10326.845302 secs 2024-04-26T14:43:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10329.625015 secs 2024-04-26T14:43:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10326.845302 secs 2024-04-26T14:43:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10329.625015 secs 2024-04-26T14:43:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10326.845302 secs 2024-04-26T14:43:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10329.625015 secs 2024-04-26T14:43:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:10.349+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:10.349+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:10.349+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:43:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:10 smithi194 ceph-mon[33017]: pgmap v6088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:10 smithi033 ceph-mon[27845]: pgmap v6088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:10 smithi096 ceph-mon[32753]: pgmap v6088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:12 smithi194 ceph-mon[33017]: pgmap v6089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:12 smithi033 ceph-mon[27845]: pgmap v6089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:12 smithi096 ceph-mon[32753]: pgmap v6089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10331.845387 secs 2024-04-26T14:43:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10334.625160 secs 2024-04-26T14:43:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10331.845387 secs 2024-04-26T14:43:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10334.625160 secs 2024-04-26T14:43:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10331.845387 secs 2024-04-26T14:43:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10334.625160 secs 2024-04-26T14:43:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:14 smithi194 ceph-mon[33017]: pgmap v6090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:14 smithi033 ceph-mon[27845]: pgmap v6090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:14 smithi096 ceph-mon[32753]: pgmap v6090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:43:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:16 smithi194 ceph-mon[33017]: pgmap v6091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:43:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:16 smithi033 ceph-mon[27845]: pgmap v6091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:16 smithi096 ceph-mon[32753]: pgmap v6091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:43:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10336.845488 secs 2024-04-26T14:43:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:18 smithi033 ceph-mon[27845]: pgmap v6092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10339.625283 secs 2024-04-26T14:43:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10336.845488 secs 2024-04-26T14:43:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:18 smithi096 ceph-mon[32753]: pgmap v6092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10339.625283 secs 2024-04-26T14:43:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10336.845488 secs 2024-04-26T14:43:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:18 smithi194 ceph-mon[33017]: pgmap v6092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10339.625283 secs 2024-04-26T14:43:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:20 smithi033 ceph-mon[27845]: pgmap v6093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:20 smithi096 ceph-mon[32753]: pgmap v6093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:20 smithi194 ceph-mon[33017]: pgmap v6093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:22 smithi033 ceph-mon[27845]: pgmap v6094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:22 smithi096 ceph-mon[32753]: pgmap v6094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:22 smithi194 ceph-mon[33017]: pgmap v6094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10341.845588 secs 2024-04-26T14:43:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10344.625380 secs 2024-04-26T14:43:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10341.845588 secs 2024-04-26T14:43:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10344.625380 secs 2024-04-26T14:43:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10341.845588 secs 2024-04-26T14:43:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10344.625380 secs 2024-04-26T14:43:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:24 smithi033 ceph-mon[27845]: pgmap v6095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:24 smithi096 ceph-mon[32753]: pgmap v6095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:24 smithi194 ceph-mon[33017]: pgmap v6095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:25.366 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:25.366+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:25.366 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:25.366+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:25.366 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:25.366+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:43:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:26 smithi033 ceph-mon[27845]: pgmap v6096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:26 smithi096 ceph-mon[32753]: pgmap v6096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:26 smithi194 ceph-mon[33017]: pgmap v6096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10346.845690 secs 2024-04-26T14:43:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:28 smithi033 ceph-mon[27845]: pgmap v6097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10349.625481 secs 2024-04-26T14:43:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10346.845690 secs 2024-04-26T14:43:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:28 smithi096 ceph-mon[32753]: pgmap v6097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10349.625481 secs 2024-04-26T14:43:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10346.845690 secs 2024-04-26T14:43:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:28 smithi194 ceph-mon[33017]: pgmap v6097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10349.625481 secs 2024-04-26T14:43:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:30 smithi033 ceph-mon[27845]: pgmap v6098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:30 smithi096 ceph-mon[32753]: pgmap v6098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:30 smithi194 ceph-mon[33017]: pgmap v6098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:32 smithi033 ceph-mon[27845]: pgmap v6099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:32 smithi096 ceph-mon[32753]: pgmap v6099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:32 smithi194 ceph-mon[33017]: pgmap v6099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10351.845793 secs 2024-04-26T14:43:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10354.625580 secs 2024-04-26T14:43:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10351.845793 secs 2024-04-26T14:43:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10354.625580 secs 2024-04-26T14:43:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10351.845793 secs 2024-04-26T14:43:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10354.625580 secs 2024-04-26T14:43:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:34 smithi033 ceph-mon[27845]: pgmap v6100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:34 smithi096 ceph-mon[32753]: pgmap v6100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:34 smithi194 ceph-mon[33017]: pgmap v6100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:36 smithi033 ceph-mon[27845]: pgmap v6101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:36 smithi096 ceph-mon[32753]: pgmap v6101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:36 smithi194 ceph-mon[33017]: pgmap v6101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10356.845892 secs 2024-04-26T14:43:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:38 smithi033 ceph-mon[27845]: pgmap v6102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10359.625706 secs 2024-04-26T14:43:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10356.845892 secs 2024-04-26T14:43:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:38 smithi096 ceph-mon[32753]: pgmap v6102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10359.625706 secs 2024-04-26T14:43:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10356.845892 secs 2024-04-26T14:43:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:38 smithi194 ceph-mon[33017]: pgmap v6102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10359.625706 secs 2024-04-26T14:43:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:40.380+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:40.381+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:40.381+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:43:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:40 smithi096 ceph-mon[32753]: pgmap v6103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:40 smithi194 ceph-mon[33017]: pgmap v6103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:40 smithi033 ceph-mon[27845]: pgmap v6103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:42 smithi096 ceph-mon[32753]: pgmap v6104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:42 smithi194 ceph-mon[33017]: pgmap v6104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:42 smithi033 ceph-mon[27845]: pgmap v6104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10361.845993 secs 2024-04-26T14:43:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10364.625829 secs 2024-04-26T14:43:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10361.845993 secs 2024-04-26T14:43:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10364.625829 secs 2024-04-26T14:43:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10361.845993 secs 2024-04-26T14:43:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10364.625829 secs 2024-04-26T14:43:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:44 smithi194 ceph-mon[33017]: pgmap v6105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:44 smithi033 ceph-mon[27845]: pgmap v6105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:44 smithi096 ceph-mon[32753]: pgmap v6105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:46 smithi194 ceph-mon[33017]: pgmap v6106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:46 smithi033 ceph-mon[27845]: pgmap v6106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:46 smithi096 ceph-mon[32753]: pgmap v6106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10366.846083 secs 2024-04-26T14:43:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:48 smithi194 ceph-mon[33017]: pgmap v6107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10369.625968 secs 2024-04-26T14:43:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10366.846083 secs 2024-04-26T14:43:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:48 smithi033 ceph-mon[27845]: pgmap v6107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10369.625968 secs 2024-04-26T14:43:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10366.846083 secs 2024-04-26T14:43:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:48 smithi096 ceph-mon[32753]: pgmap v6107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10369.625968 secs 2024-04-26T14:43:51.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:50 smithi096 ceph-mon[32753]: pgmap v6108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:50 smithi194 ceph-mon[33017]: pgmap v6108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:50 smithi033 ceph-mon[27845]: pgmap v6108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:52 smithi194 ceph-mon[33017]: pgmap v6109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:52 smithi033 ceph-mon[27845]: pgmap v6109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:52 smithi096 ceph-mon[32753]: pgmap v6109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10371.846160 secs 2024-04-26T14:43:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10374.626097 secs 2024-04-26T14:43:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10371.846160 secs 2024-04-26T14:43:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10374.626097 secs 2024-04-26T14:43:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10371.846160 secs 2024-04-26T14:43:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10374.626097 secs 2024-04-26T14:43:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:54 smithi194 ceph-mon[33017]: pgmap v6110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:55.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:55.382+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:55.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:55.382+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:43:55.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:43:55.382+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:43:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:54 smithi033 ceph-mon[27845]: pgmap v6110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:54 smithi096 ceph-mon[32753]: pgmap v6110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:56 smithi194 ceph-mon[33017]: pgmap v6111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:56 smithi033 ceph-mon[27845]: pgmap v6111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:56 smithi096 ceph-mon[32753]: pgmap v6111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:43:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10376.846244 secs 2024-04-26T14:43:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:58 smithi194 ceph-mon[33017]: pgmap v6112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:43:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10379.626220 secs 2024-04-26T14:43:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10376.846244 secs 2024-04-26T14:43:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:58 smithi033 ceph-mon[27845]: pgmap v6112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:43:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10379.626220 secs 2024-04-26T14:43:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10376.846244 secs 2024-04-26T14:43:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:58 smithi096 ceph-mon[32753]: pgmap v6112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:43:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:43:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10379.626220 secs 2024-04-26T14:44:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:00 smithi194 ceph-mon[33017]: pgmap v6113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:00 smithi033 ceph-mon[27845]: pgmap v6113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:00 smithi096 ceph-mon[32753]: pgmap v6113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:03 smithi194 ceph-mon[33017]: pgmap v6114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:03 smithi033 ceph-mon[27845]: pgmap v6114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:03 smithi096 ceph-mon[32753]: pgmap v6114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10381.846326 secs 2024-04-26T14:44:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10384.626316 secs 2024-04-26T14:44:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10381.846326 secs 2024-04-26T14:44:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10384.626316 secs 2024-04-26T14:44:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10381.846326 secs 2024-04-26T14:44:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10384.626316 secs 2024-04-26T14:44:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:05 smithi194 ceph-mon[33017]: pgmap v6115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:05 smithi033 ceph-mon[27845]: pgmap v6115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:05 smithi096 ceph-mon[32753]: pgmap v6115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:07 smithi194 ceph-mon[33017]: pgmap v6116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:44:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:07 smithi033 ceph-mon[27845]: pgmap v6116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:44:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:07 smithi096 ceph-mon[32753]: pgmap v6116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:44:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:44:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:44:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:44:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:44:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:44:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:44:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:44:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:44:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:44:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:09 smithi194 ceph-mon[33017]: pgmap v6117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:09 smithi194 ceph-mon[33017]: pgmap v6118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10386.846389 secs 2024-04-26T14:44:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10389.626427 secs 2024-04-26T14:44:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:09 smithi033 ceph-mon[27845]: pgmap v6117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:09 smithi033 ceph-mon[27845]: pgmap v6118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10386.846389 secs 2024-04-26T14:44:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10389.626427 secs 2024-04-26T14:44:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:09 smithi096 ceph-mon[32753]: pgmap v6117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:09 smithi096 ceph-mon[32753]: pgmap v6118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10386.846389 secs 2024-04-26T14:44:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10389.626427 secs 2024-04-26T14:44:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:10.389+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:10.389+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:10.389+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:44:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:11 smithi194 ceph-mon[33017]: pgmap v6119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:11 smithi033 ceph-mon[27845]: pgmap v6119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:11 smithi096 ceph-mon[32753]: pgmap v6119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:13 smithi194 ceph-mon[33017]: pgmap v6120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:13 smithi033 ceph-mon[27845]: pgmap v6120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:13 smithi096 ceph-mon[32753]: pgmap v6120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10391.846488 secs 2024-04-26T14:44:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10394.626565 secs 2024-04-26T14:44:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10391.846488 secs 2024-04-26T14:44:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10394.626565 secs 2024-04-26T14:44:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10391.846488 secs 2024-04-26T14:44:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10394.626565 secs 2024-04-26T14:44:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:15 smithi194 ceph-mon[33017]: pgmap v6121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:15 smithi033 ceph-mon[27845]: pgmap v6121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:15 smithi096 ceph-mon[32753]: pgmap v6121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:17 smithi194 ceph-mon[33017]: pgmap v6122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:17 smithi033 ceph-mon[27845]: pgmap v6122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:17 smithi096 ceph-mon[32753]: pgmap v6122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:44:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:19 smithi194 ceph-mon[33017]: pgmap v6123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:44:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10396.846590 secs 2024-04-26T14:44:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10399.626739 secs 2024-04-26T14:44:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:19 smithi033 ceph-mon[27845]: pgmap v6123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:44:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10396.846590 secs 2024-04-26T14:44:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10399.626739 secs 2024-04-26T14:44:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:19 smithi096 ceph-mon[32753]: pgmap v6123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:44:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10396.846590 secs 2024-04-26T14:44:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10399.626739 secs 2024-04-26T14:44:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:21 smithi194 ceph-mon[33017]: pgmap v6124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:21 smithi033 ceph-mon[27845]: pgmap v6124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:21 smithi096 ceph-mon[32753]: pgmap v6124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:23 smithi194 ceph-mon[33017]: pgmap v6125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:23 smithi033 ceph-mon[27845]: pgmap v6125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:23 smithi096 ceph-mon[32753]: pgmap v6125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:23.561+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:44:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:23.561+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:44:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:23.561+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006610 con 0x564385821900 2024-04-26T14:44:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:23.562+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:44:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10401.846689 secs 2024-04-26T14:44:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10404.626866 secs 2024-04-26T14:44:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10401.846689 secs 2024-04-26T14:44:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10404.626866 secs 2024-04-26T14:44:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10401.846689 secs 2024-04-26T14:44:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10404.626866 secs 2024-04-26T14:44:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:25 smithi194 ceph-mon[33017]: pgmap v6126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:25.401+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:25.401+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:25.401+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:44:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:25 smithi033 ceph-mon[27845]: pgmap v6126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:25 smithi096 ceph-mon[32753]: pgmap v6126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:27 smithi194 ceph-mon[33017]: pgmap v6127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:27 smithi033 ceph-mon[27845]: pgmap v6127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:27 smithi096 ceph-mon[32753]: pgmap v6127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:29 smithi194 ceph-mon[33017]: pgmap v6128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10406.846791 secs 2024-04-26T14:44:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10409.626980 secs 2024-04-26T14:44:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:29 smithi033 ceph-mon[27845]: pgmap v6128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10406.846791 secs 2024-04-26T14:44:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10409.626980 secs 2024-04-26T14:44:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:29 smithi096 ceph-mon[32753]: pgmap v6128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:44:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10406.846791 secs 2024-04-26T14:44:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10409.626980 secs 2024-04-26T14:44:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:31 smithi033 ceph-mon[27845]: pgmap v6129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:31 smithi096 ceph-mon[32753]: pgmap v6129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:31 smithi194 ceph-mon[33017]: pgmap v6129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:44:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:33 smithi033 ceph-mon[27845]: pgmap v6130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:33 smithi096 ceph-mon[32753]: pgmap v6130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:33 smithi194 ceph-mon[33017]: pgmap v6130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10411.846889 secs 2024-04-26T14:44:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10414.627133 secs 2024-04-26T14:44:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10411.846889 secs 2024-04-26T14:44:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10414.627133 secs 2024-04-26T14:44:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10411.846889 secs 2024-04-26T14:44:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10414.627133 secs 2024-04-26T14:44:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:35 smithi033 ceph-mon[27845]: pgmap v6131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:35 smithi096 ceph-mon[32753]: pgmap v6131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:35 smithi194 ceph-mon[33017]: pgmap v6131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:37 smithi033 ceph-mon[27845]: pgmap v6132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:37 smithi096 ceph-mon[32753]: pgmap v6132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:37 smithi194 ceph-mon[33017]: pgmap v6132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:39 smithi033 ceph-mon[27845]: pgmap v6133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10416.846991 secs 2024-04-26T14:44:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10419.627244 secs 2024-04-26T14:44:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:39 smithi096 ceph-mon[32753]: pgmap v6133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10416.846991 secs 2024-04-26T14:44:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10419.627244 secs 2024-04-26T14:44:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:39 smithi194 ceph-mon[33017]: pgmap v6133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10416.846991 secs 2024-04-26T14:44:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10419.627244 secs 2024-04-26T14:44:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:40.416+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:40.416+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:40.416+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:44:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:41 smithi033 ceph-mon[27845]: pgmap v6134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:41 smithi096 ceph-mon[32753]: pgmap v6134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:41 smithi194 ceph-mon[33017]: pgmap v6134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:43 smithi033 ceph-mon[27845]: pgmap v6135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:43 smithi096 ceph-mon[32753]: pgmap v6135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:43 smithi194 ceph-mon[33017]: pgmap v6135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10421.847050 secs 2024-04-26T14:44:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10424.627323 secs 2024-04-26T14:44:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10421.847050 secs 2024-04-26T14:44:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10424.627323 secs 2024-04-26T14:44:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10421.847050 secs 2024-04-26T14:44:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10424.627323 secs 2024-04-26T14:44:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:45 smithi033 ceph-mon[27845]: pgmap v6136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:45 smithi096 ceph-mon[32753]: pgmap v6136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:45 smithi194 ceph-mon[33017]: pgmap v6136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:47 smithi096 ceph-mon[32753]: pgmap v6137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:47 smithi194 ceph-mon[33017]: pgmap v6137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:47 smithi033 ceph-mon[27845]: pgmap v6137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:49 smithi096 ceph-mon[32753]: pgmap v6138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10426.847111 secs 2024-04-26T14:44:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10429.627458 secs 2024-04-26T14:44:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:49 smithi194 ceph-mon[33017]: pgmap v6138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10426.847111 secs 2024-04-26T14:44:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10429.627458 secs 2024-04-26T14:44:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:49 smithi033 ceph-mon[27845]: pgmap v6138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10426.847111 secs 2024-04-26T14:44:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10429.627458 secs 2024-04-26T14:44:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:51 smithi194 ceph-mon[33017]: pgmap v6139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:51 smithi033 ceph-mon[27845]: pgmap v6139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:51 smithi096 ceph-mon[32753]: pgmap v6139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:53 smithi194 ceph-mon[33017]: pgmap v6140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:53 smithi033 ceph-mon[27845]: pgmap v6140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:53 smithi096 ceph-mon[32753]: pgmap v6140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10431.847192 secs 2024-04-26T14:44:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10434.627523 secs 2024-04-26T14:44:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10431.847192 secs 2024-04-26T14:44:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10434.627523 secs 2024-04-26T14:44:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10431.847192 secs 2024-04-26T14:44:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10434.627523 secs 2024-04-26T14:44:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:55.430+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:55.430+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:44:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:44:55.430+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:44:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:55 smithi194 ceph-mon[33017]: pgmap v6141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:55 smithi033 ceph-mon[27845]: pgmap v6141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:55 smithi096 ceph-mon[32753]: pgmap v6141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:57 smithi194 ceph-mon[33017]: pgmap v6142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:57 smithi033 ceph-mon[27845]: pgmap v6142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:57 smithi096 ceph-mon[32753]: pgmap v6142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:44:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:59 smithi194 ceph-mon[33017]: pgmap v6143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10436.847253 secs 2024-04-26T14:44:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:44:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10439.627669 secs 2024-04-26T14:44:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:59 smithi033 ceph-mon[27845]: pgmap v6143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10436.847253 secs 2024-04-26T14:44:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:44:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10439.627669 secs 2024-04-26T14:44:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:59 smithi096 ceph-mon[32753]: pgmap v6143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:44:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10436.847253 secs 2024-04-26T14:44:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:44:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10439.627669 secs 2024-04-26T14:45:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:01 smithi194 ceph-mon[33017]: pgmap v6144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:01 smithi033 ceph-mon[27845]: pgmap v6144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:01 smithi096 ceph-mon[32753]: pgmap v6144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:03.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:03 smithi033 ceph-mon[27845]: pgmap v6145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:03 smithi194 ceph-mon[33017]: pgmap v6145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:03 smithi096 ceph-mon[32753]: pgmap v6145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10441.847354 secs 2024-04-26T14:45:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10444.627795 secs 2024-04-26T14:45:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10441.847354 secs 2024-04-26T14:45:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10444.627795 secs 2024-04-26T14:45:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10441.847354 secs 2024-04-26T14:45:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10444.627795 secs 2024-04-26T14:45:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:05 smithi194 ceph-mon[33017]: pgmap v6146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:05 smithi033 ceph-mon[27845]: pgmap v6146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:05 smithi096 ceph-mon[32753]: pgmap v6146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:07 smithi194 ceph-mon[33017]: pgmap v6147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:07 smithi033 ceph-mon[27845]: pgmap v6147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:07 smithi096 ceph-mon[32753]: pgmap v6147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:45:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:45:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:45:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:09 smithi194 ceph-mon[33017]: pgmap v6148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10446.847415 secs 2024-04-26T14:45:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10449.627860 secs 2024-04-26T14:45:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:45:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:45:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:45:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:09 smithi033 ceph-mon[27845]: pgmap v6148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10446.847415 secs 2024-04-26T14:45:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10449.627860 secs 2024-04-26T14:45:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:45:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:45:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:09 smithi096 ceph-mon[32753]: pgmap v6148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10446.847415 secs 2024-04-26T14:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10449.627860 secs 2024-04-26T14:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:45:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:10.444+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:10.444+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:45:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:10.444+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:10 smithi194 ceph-mon[33017]: pgmap v6149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:10 smithi194 ceph-mon[33017]: pgmap v6150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:45:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:10 smithi033 ceph-mon[27845]: pgmap v6149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:10 smithi033 ceph-mon[27845]: pgmap v6150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:45:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:10 smithi096 ceph-mon[32753]: pgmap v6149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:10 smithi096 ceph-mon[32753]: pgmap v6150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T14:45:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:12 smithi194 ceph-mon[33017]: pgmap v6151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:12 smithi033 ceph-mon[27845]: pgmap v6151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:12 smithi096 ceph-mon[32753]: pgmap v6151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10451.847475 secs 2024-04-26T14:45:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:14 smithi033 ceph-mon[27845]: pgmap v6152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10454.627987 secs 2024-04-26T14:45:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10451.847475 secs 2024-04-26T14:45:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:14 smithi194 ceph-mon[33017]: pgmap v6152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10454.627987 secs 2024-04-26T14:45:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10451.847475 secs 2024-04-26T14:45:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:14 smithi096 ceph-mon[32753]: pgmap v6152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10454.627987 secs 2024-04-26T14:45:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:16 smithi194 ceph-mon[33017]: pgmap v6153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:16 smithi033 ceph-mon[27845]: pgmap v6153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:16 smithi096 ceph-mon[32753]: pgmap v6153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:18 smithi194 ceph-mon[33017]: pgmap v6154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:18 smithi033 ceph-mon[27845]: pgmap v6154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:18 smithi096 ceph-mon[32753]: pgmap v6154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:45:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10456.847559 secs 2024-04-26T14:45:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10459.628120 secs 2024-04-26T14:45:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10456.847559 secs 2024-04-26T14:45:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10459.628120 secs 2024-04-26T14:45:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10456.847559 secs 2024-04-26T14:45:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10459.628120 secs 2024-04-26T14:45:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:20 smithi194 ceph-mon[33017]: pgmap v6155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:45:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:20 smithi033 ceph-mon[27845]: pgmap v6155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:45:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:20 smithi096 ceph-mon[32753]: pgmap v6155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:45:21.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:21 smithi096 ceph-mon[32753]: pgmap v6156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:21 smithi194 ceph-mon[33017]: pgmap v6156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:21 smithi033 ceph-mon[27845]: pgmap v6156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10461.847641 secs 2024-04-26T14:45:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:24 smithi033 ceph-mon[27845]: pgmap v6157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10464.628210 secs 2024-04-26T14:45:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10461.847641 secs 2024-04-26T14:45:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:24 smithi096 ceph-mon[32753]: pgmap v6157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10464.628210 secs 2024-04-26T14:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10461.847641 secs 2024-04-26T14:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:24 smithi194 ceph-mon[33017]: pgmap v6157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10464.628210 secs 2024-04-26T14:45:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:25.449+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:25.449+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:45:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:25.449+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:26 smithi033 ceph-mon[27845]: pgmap v6158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:26 smithi096 ceph-mon[32753]: pgmap v6158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:26 smithi194 ceph-mon[33017]: pgmap v6158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:28 smithi033 ceph-mon[27845]: pgmap v6159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:28 smithi096 ceph-mon[32753]: pgmap v6159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:28 smithi194 ceph-mon[33017]: pgmap v6159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10466.847729 secs 2024-04-26T14:45:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10469.628331 secs 2024-04-26T14:45:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10466.847729 secs 2024-04-26T14:45:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10469.628331 secs 2024-04-26T14:45:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10466.847729 secs 2024-04-26T14:45:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10469.628331 secs 2024-04-26T14:45:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:30 smithi033 ceph-mon[27845]: pgmap v6160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:30 smithi096 ceph-mon[32753]: pgmap v6160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:30 smithi194 ceph-mon[33017]: pgmap v6160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:32 smithi033 ceph-mon[27845]: pgmap v6161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:32 smithi096 ceph-mon[32753]: pgmap v6161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:32 smithi194 ceph-mon[33017]: pgmap v6161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10471.847810 secs 2024-04-26T14:45:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10471.847810 secs 2024-04-26T14:45:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10471.847810 secs 2024-04-26T14:45:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:34 smithi096 ceph-mon[32753]: pgmap v6162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10474.628431 secs 2024-04-26T14:45:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:34 smithi194 ceph-mon[33017]: pgmap v6162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10474.628431 secs 2024-04-26T14:45:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:34 smithi033 ceph-mon[27845]: pgmap v6162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:45:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10474.628431 secs 2024-04-26T14:45:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:36 smithi096 ceph-mon[32753]: pgmap v6163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:36 smithi194 ceph-mon[33017]: pgmap v6163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:36 smithi033 ceph-mon[27845]: pgmap v6163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:45:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:38 smithi194 ceph-mon[33017]: pgmap v6164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10476.847890 secs 2024-04-26T14:45:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:38 smithi033 ceph-mon[27845]: pgmap v6164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10476.847890 secs 2024-04-26T14:45:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:38 smithi096 ceph-mon[32753]: pgmap v6164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10476.847890 secs 2024-04-26T14:45:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10479.628545 secs 2024-04-26T14:45:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10479.628545 secs 2024-04-26T14:45:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10479.628545 secs 2024-04-26T14:45:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:40.461+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:40.462+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:40.462+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:45:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:40 smithi194 ceph-mon[33017]: pgmap v6165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:40 smithi033 ceph-mon[27845]: pgmap v6165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:40 smithi096 ceph-mon[32753]: pgmap v6165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:42 smithi194 ceph-mon[33017]: pgmap v6166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:42 smithi033 ceph-mon[27845]: pgmap v6166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:42 smithi096 ceph-mon[32753]: pgmap v6166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10481.847974 secs 2024-04-26T14:45:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10481.847974 secs 2024-04-26T14:45:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10481.847974 secs 2024-04-26T14:45:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:44 smithi194 ceph-mon[33017]: pgmap v6167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10484.628661 secs 2024-04-26T14:45:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:44 smithi033 ceph-mon[27845]: pgmap v6167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10484.628661 secs 2024-04-26T14:45:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:44 smithi096 ceph-mon[32753]: pgmap v6167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10484.628661 secs 2024-04-26T14:45:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:46 smithi194 ceph-mon[33017]: pgmap v6168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:46 smithi033 ceph-mon[27845]: pgmap v6168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:46 smithi096 ceph-mon[32753]: pgmap v6168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:48 smithi194 ceph-mon[33017]: pgmap v6169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10486.848047 secs 2024-04-26T14:45:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:48 smithi033 ceph-mon[27845]: pgmap v6169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10486.848047 secs 2024-04-26T14:45:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:48 smithi096 ceph-mon[32753]: pgmap v6169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10486.848047 secs 2024-04-26T14:45:49.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10489.628778 secs 2024-04-26T14:45:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10489.628778 secs 2024-04-26T14:45:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10489.628778 secs 2024-04-26T14:45:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:50 smithi194 ceph-mon[33017]: pgmap v6170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:50 smithi033 ceph-mon[27845]: pgmap v6170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:50 smithi096 ceph-mon[32753]: pgmap v6170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:52 smithi194 ceph-mon[33017]: pgmap v6171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:52 smithi033 ceph-mon[27845]: pgmap v6171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:52 smithi096 ceph-mon[32753]: pgmap v6171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10491.848125 secs 2024-04-26T14:45:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10491.848125 secs 2024-04-26T14:45:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10491.848125 secs 2024-04-26T14:45:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:54 smithi194 ceph-mon[33017]: pgmap v6172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10494.628875 secs 2024-04-26T14:45:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:54 smithi033 ceph-mon[27845]: pgmap v6172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10494.628875 secs 2024-04-26T14:45:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:54 smithi096 ceph-mon[32753]: pgmap v6172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10494.628875 secs 2024-04-26T14:45:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:55.464+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:55.464+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:45:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:45:55.464+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:45:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:56 smithi194 ceph-mon[33017]: pgmap v6173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:56 smithi033 ceph-mon[27845]: pgmap v6173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:56 smithi096 ceph-mon[32753]: pgmap v6173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:45:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:58 smithi194 ceph-mon[33017]: pgmap v6174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10496.848211 secs 2024-04-26T14:45:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:58 smithi033 ceph-mon[27845]: pgmap v6174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10496.848211 secs 2024-04-26T14:45:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:58 smithi096 ceph-mon[32753]: pgmap v6174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:45:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10496.848211 secs 2024-04-26T14:45:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10499.628995 secs 2024-04-26T14:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10499.628995 secs 2024-04-26T14:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10499.628995 secs 2024-04-26T14:46:00.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:00 smithi033 ceph-mon[27845]: pgmap v6175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:00 smithi194 ceph-mon[33017]: pgmap v6175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:00 smithi096 ceph-mon[32753]: pgmap v6175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:02 smithi194 ceph-mon[33017]: pgmap v6176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:02 smithi033 ceph-mon[27845]: pgmap v6176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:02 smithi096 ceph-mon[32753]: pgmap v6176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:03.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10501.848282 secs 2024-04-26T14:46:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10501.848282 secs 2024-04-26T14:46:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10501.848282 secs 2024-04-26T14:46:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:04 smithi194 ceph-mon[33017]: pgmap v6177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10504.629134 secs 2024-04-26T14:46:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:04 smithi033 ceph-mon[27845]: pgmap v6177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10504.629134 secs 2024-04-26T14:46:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:04 smithi096 ceph-mon[32753]: pgmap v6177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10504.629134 secs 2024-04-26T14:46:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:05 smithi194 ceph-mon[33017]: pgmap v6178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:05 smithi033 ceph-mon[27845]: pgmap v6178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:05 smithi096 ceph-mon[32753]: pgmap v6178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:08 smithi033 ceph-mon[27845]: pgmap v6179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10506.848367 secs 2024-04-26T14:46:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:08 smithi096 ceph-mon[32753]: pgmap v6179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10506.848367 secs 2024-04-26T14:46:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:08 smithi194 ceph-mon[33017]: pgmap v6179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10506.848367 secs 2024-04-26T14:46:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10509.629252 secs 2024-04-26T14:46:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10509.629252 secs 2024-04-26T14:46:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10509.629252 secs 2024-04-26T14:46:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:10 smithi033 ceph-mon[27845]: pgmap v6180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:46:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:10 smithi096 ceph-mon[32753]: pgmap v6180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:46:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:10.478+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:10.478+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:46:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:10.478+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:10 smithi194 ceph-mon[33017]: pgmap v6180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:46:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:11 smithi194 ceph-mon[33017]: pgmap v6181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:11 smithi033 ceph-mon[27845]: pgmap v6181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:11 smithi096 ceph-mon[32753]: pgmap v6181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:46:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:46:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:12 smithi033 ceph-mon[27845]: pgmap v6182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:46:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:12 smithi033 ceph-mon[27845]: pgmap v6183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:46:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:46:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:12 smithi096 ceph-mon[32753]: pgmap v6182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:46:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:12 smithi096 ceph-mon[32753]: pgmap v6183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:46:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:46:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:12 smithi194 ceph-mon[33017]: pgmap v6182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:46:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:12 smithi194 ceph-mon[33017]: pgmap v6183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:46:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10511.848452 secs 2024-04-26T14:46:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10511.848452 secs 2024-04-26T14:46:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10511.848452 secs 2024-04-26T14:46:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10514.629368 secs 2024-04-26T14:46:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:14 smithi033 ceph-mon[27845]: pgmap v6184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:46:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10514.629368 secs 2024-04-26T14:46:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:14 smithi096 ceph-mon[32753]: pgmap v6184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:46:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10514.629368 secs 2024-04-26T14:46:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:14 smithi194 ceph-mon[33017]: pgmap v6184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:46:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:16 smithi194 ceph-mon[33017]: pgmap v6185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:16 smithi033 ceph-mon[27845]: pgmap v6185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:16 smithi096 ceph-mon[32753]: pgmap v6185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10516.848541 secs 2024-04-26T14:46:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10516.848541 secs 2024-04-26T14:46:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10516.848541 secs 2024-04-26T14:46:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:18 smithi194 ceph-mon[33017]: pgmap v6186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10519.629480 secs 2024-04-26T14:46:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:18 smithi033 ceph-mon[27845]: pgmap v6186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10519.629480 secs 2024-04-26T14:46:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:18 smithi096 ceph-mon[32753]: pgmap v6186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10519.629480 secs 2024-04-26T14:46:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:20 smithi194 ceph-mon[33017]: pgmap v6187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:20 smithi033 ceph-mon[27845]: pgmap v6187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:20 smithi096 ceph-mon[32753]: pgmap v6187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T14:46:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:22 smithi194 ceph-mon[33017]: pgmap v6188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:46:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10521.848653 secs 2024-04-26T14:46:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:22 smithi033 ceph-mon[27845]: pgmap v6188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:46:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10521.848653 secs 2024-04-26T14:46:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:22 smithi096 ceph-mon[32753]: pgmap v6188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:46:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10521.848653 secs 2024-04-26T14:46:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10524.629595 secs 2024-04-26T14:46:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10524.629595 secs 2024-04-26T14:46:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10524.629595 secs 2024-04-26T14:46:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:24 smithi194 ceph-mon[33017]: pgmap v6189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:24 smithi033 ceph-mon[27845]: pgmap v6189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:24 smithi096 ceph-mon[32753]: pgmap v6189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:25.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:25.492+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:25.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:25.492+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:25.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:25.492+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:46:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:26 smithi194 ceph-mon[33017]: pgmap v6190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:26 smithi033 ceph-mon[27845]: pgmap v6190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:26 smithi096 ceph-mon[32753]: pgmap v6190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10526.848753 secs 2024-04-26T14:46:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10526.848753 secs 2024-04-26T14:46:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10526.848753 secs 2024-04-26T14:46:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:28 smithi194 ceph-mon[33017]: pgmap v6191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10529.629719 secs 2024-04-26T14:46:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:28 smithi033 ceph-mon[27845]: pgmap v6191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10529.629719 secs 2024-04-26T14:46:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:28 smithi096 ceph-mon[32753]: pgmap v6191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10529.629719 secs 2024-04-26T14:46:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:30 smithi194 ceph-mon[33017]: pgmap v6192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:30 smithi033 ceph-mon[27845]: pgmap v6192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:30 smithi096 ceph-mon[32753]: pgmap v6192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:32 smithi194 ceph-mon[33017]: pgmap v6193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10531.848819 secs 2024-04-26T14:46:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:32 smithi033 ceph-mon[27845]: pgmap v6193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10531.848819 secs 2024-04-26T14:46:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:32 smithi096 ceph-mon[32753]: pgmap v6193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10531.848819 secs 2024-04-26T14:46:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10534.629836 secs 2024-04-26T14:46:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10534.629836 secs 2024-04-26T14:46:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10534.629836 secs 2024-04-26T14:46:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:34 smithi194 ceph-mon[33017]: pgmap v6194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:34 smithi033 ceph-mon[27845]: pgmap v6194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:34 smithi096 ceph-mon[32753]: pgmap v6194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:36 smithi194 ceph-mon[33017]: pgmap v6195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:36 smithi033 ceph-mon[27845]: pgmap v6195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:36 smithi096 ceph-mon[32753]: pgmap v6195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10536.848903 secs 2024-04-26T14:46:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10536.848903 secs 2024-04-26T14:46:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10536.848903 secs 2024-04-26T14:46:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:38 smithi033 ceph-mon[27845]: pgmap v6196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10539.629958 secs 2024-04-26T14:46:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:38 smithi096 ceph-mon[32753]: pgmap v6196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10539.629958 secs 2024-04-26T14:46:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:38 smithi194 ceph-mon[33017]: pgmap v6196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10539.629958 secs 2024-04-26T14:46:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:40.508+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:40.508+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:40.508+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:46:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:40 smithi033 ceph-mon[27845]: pgmap v6197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:40 smithi096 ceph-mon[32753]: pgmap v6197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:40 smithi194 ceph-mon[33017]: pgmap v6197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:42 smithi033 ceph-mon[27845]: pgmap v6198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10541.848988 secs 2024-04-26T14:46:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:42 smithi096 ceph-mon[32753]: pgmap v6198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10541.848988 secs 2024-04-26T14:46:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:42 smithi194 ceph-mon[33017]: pgmap v6198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10541.848988 secs 2024-04-26T14:46:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10544.630049 secs 2024-04-26T14:46:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10544.630049 secs 2024-04-26T14:46:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10544.630049 secs 2024-04-26T14:46:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:44 smithi033 ceph-mon[27845]: pgmap v6199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:44 smithi096 ceph-mon[32753]: pgmap v6199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:44 smithi194 ceph-mon[33017]: pgmap v6199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:46 smithi033 ceph-mon[27845]: pgmap v6200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:46 smithi096 ceph-mon[32753]: pgmap v6200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:46 smithi194 ceph-mon[33017]: pgmap v6200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10546.849098 secs 2024-04-26T14:46:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10546.849098 secs 2024-04-26T14:46:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10546.849098 secs 2024-04-26T14:46:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:48 smithi033 ceph-mon[27845]: pgmap v6201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10549.630142 secs 2024-04-26T14:46:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:48 smithi096 ceph-mon[32753]: pgmap v6201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10549.630142 secs 2024-04-26T14:46:49.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:48 smithi194 ceph-mon[33017]: pgmap v6201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:49.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10549.630142 secs 2024-04-26T14:46:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:50 smithi033 ceph-mon[27845]: pgmap v6202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:50 smithi096 ceph-mon[32753]: pgmap v6202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:50 smithi194 ceph-mon[33017]: pgmap v6202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:52 smithi033 ceph-mon[27845]: pgmap v6203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10551.849188 secs 2024-04-26T14:46:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:52 smithi096 ceph-mon[32753]: pgmap v6203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10551.849188 secs 2024-04-26T14:46:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:52 smithi194 ceph-mon[33017]: pgmap v6203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10551.849188 secs 2024-04-26T14:46:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10554.630239 secs 2024-04-26T14:46:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10554.630239 secs 2024-04-26T14:46:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10554.630239 secs 2024-04-26T14:46:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:54 smithi033 ceph-mon[27845]: pgmap v6204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:54 smithi096 ceph-mon[32753]: pgmap v6204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:54 smithi194 ceph-mon[33017]: pgmap v6204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:55.517+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:55.517+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:46:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:46:55.517+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:46:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:56 smithi096 ceph-mon[32753]: pgmap v6205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:56 smithi194 ceph-mon[33017]: pgmap v6205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:56 smithi033 ceph-mon[27845]: pgmap v6205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:46:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10556.849249 secs 2024-04-26T14:46:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10556.849249 secs 2024-04-26T14:46:58.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10556.849249 secs 2024-04-26T14:46:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:58 smithi194 ceph-mon[33017]: pgmap v6206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:46:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10559.630359 secs 2024-04-26T14:46:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:58 smithi033 ceph-mon[27845]: pgmap v6206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:46:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10559.630359 secs 2024-04-26T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:58 smithi096 ceph-mon[32753]: pgmap v6206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:46:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10559.630359 secs 2024-04-26T14:47:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:00 smithi194 ceph-mon[33017]: pgmap v6207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:00 smithi033 ceph-mon[27845]: pgmap v6207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:00 smithi096 ceph-mon[32753]: pgmap v6207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:02 smithi194 ceph-mon[33017]: pgmap v6208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10561.849312 secs 2024-04-26T14:47:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:02 smithi033 ceph-mon[27845]: pgmap v6208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10561.849312 secs 2024-04-26T14:47:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:02 smithi096 ceph-mon[32753]: pgmap v6208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10561.849312 secs 2024-04-26T14:47:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10564.630461 secs 2024-04-26T14:47:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10564.630461 secs 2024-04-26T14:47:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10564.630461 secs 2024-04-26T14:47:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:04 smithi194 ceph-mon[33017]: pgmap v6209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:04 smithi033 ceph-mon[27845]: pgmap v6209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:04 smithi096 ceph-mon[32753]: pgmap v6209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:06 smithi194 ceph-mon[33017]: pgmap v6210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:06 smithi033 ceph-mon[27845]: pgmap v6210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:06 smithi096 ceph-mon[32753]: pgmap v6210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10566.849401 secs 2024-04-26T14:47:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10566.849401 secs 2024-04-26T14:47:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10566.849401 secs 2024-04-26T14:47:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:08 smithi194 ceph-mon[33017]: pgmap v6211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10569.630606 secs 2024-04-26T14:47:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:08 smithi033 ceph-mon[27845]: pgmap v6211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10569.630606 secs 2024-04-26T14:47:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:08 smithi096 ceph-mon[32753]: pgmap v6211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10569.630606 secs 2024-04-26T14:47:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:10.531+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:10.531+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:10.531+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:47:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:10 smithi194 ceph-mon[33017]: pgmap v6212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:10 smithi033 ceph-mon[27845]: pgmap v6212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:10 smithi096 ceph-mon[32753]: pgmap v6212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:12 smithi194 ceph-mon[33017]: pgmap v6213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:47:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10571.849461 secs 2024-04-26T14:47:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:12 smithi033 ceph-mon[27845]: pgmap v6213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:47:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10571.849461 secs 2024-04-26T14:47:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:12 smithi096 ceph-mon[32753]: pgmap v6213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:47:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10571.849461 secs 2024-04-26T14:47:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10574.630717 secs 2024-04-26T14:47:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:47:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:47:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:47:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10574.630717 secs 2024-04-26T14:47:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:47:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:47:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:47:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10574.630717 secs 2024-04-26T14:47:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:47:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:47:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:47:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:15 smithi194 ceph-mon[33017]: pgmap v6214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:15 smithi194 ceph-mon[33017]: pgmap v6215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:15 smithi033 ceph-mon[27845]: pgmap v6214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:15 smithi033 ceph-mon[27845]: pgmap v6215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:15 smithi096 ceph-mon[32753]: pgmap v6214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:15 smithi096 ceph-mon[32753]: pgmap v6215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:17 smithi194 ceph-mon[33017]: pgmap v6216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:17 smithi033 ceph-mon[27845]: pgmap v6216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:17 smithi096 ceph-mon[32753]: pgmap v6216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10576.849543 secs 2024-04-26T14:47:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10576.849543 secs 2024-04-26T14:47:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10576.849543 secs 2024-04-26T14:47:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:19 smithi194 ceph-mon[33017]: pgmap v6217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10579.630836 secs 2024-04-26T14:47:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:19 smithi033 ceph-mon[27845]: pgmap v6217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10579.630836 secs 2024-04-26T14:47:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:19 smithi096 ceph-mon[32753]: pgmap v6217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10579.630836 secs 2024-04-26T14:47:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:21 smithi194 ceph-mon[33017]: pgmap v6218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:21 smithi033 ceph-mon[27845]: pgmap v6218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:21 smithi096 ceph-mon[32753]: pgmap v6218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:23 smithi194 ceph-mon[33017]: pgmap v6219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10581.849615 secs 2024-04-26T14:47:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:23 smithi033 ceph-mon[27845]: pgmap v6219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10581.849615 secs 2024-04-26T14:47:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:23 smithi096 ceph-mon[32753]: pgmap v6219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:47:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10581.849615 secs 2024-04-26T14:47:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:23.592+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:47:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:23.592+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:47:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:23.592+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:47:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:23.592+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:47:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10584.630933 secs 2024-04-26T14:47:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10584.630933 secs 2024-04-26T14:47:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10584.630933 secs 2024-04-26T14:47:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:25 smithi194 ceph-mon[33017]: pgmap v6220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:47:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:25 smithi033 ceph-mon[27845]: pgmap v6220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:47:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:25 smithi096 ceph-mon[32753]: pgmap v6220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:47:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:25.547+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:25.547+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:25.547+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:47:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:27 smithi194 ceph-mon[33017]: pgmap v6221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:27 smithi033 ceph-mon[27845]: pgmap v6221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:27 smithi096 ceph-mon[32753]: pgmap v6221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10586.849699 secs 2024-04-26T14:47:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10586.849699 secs 2024-04-26T14:47:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10586.849699 secs 2024-04-26T14:47:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:29 smithi194 ceph-mon[33017]: pgmap v6222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10589.631073 secs 2024-04-26T14:47:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:29 smithi033 ceph-mon[27845]: pgmap v6222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10589.631073 secs 2024-04-26T14:47:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:29 smithi096 ceph-mon[32753]: pgmap v6222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10589.631073 secs 2024-04-26T14:47:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:31 smithi194 ceph-mon[33017]: pgmap v6223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:31 smithi033 ceph-mon[27845]: pgmap v6223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:31 smithi096 ceph-mon[32753]: pgmap v6223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:33 smithi033 ceph-mon[27845]: pgmap v6224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10591.849782 secs 2024-04-26T14:47:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:33 smithi194 ceph-mon[33017]: pgmap v6224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10591.849782 secs 2024-04-26T14:47:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:33 smithi096 ceph-mon[32753]: pgmap v6224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10591.849782 secs 2024-04-26T14:47:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10594.631172 secs 2024-04-26T14:47:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10594.631172 secs 2024-04-26T14:47:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10594.631172 secs 2024-04-26T14:47:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:35 smithi194 ceph-mon[33017]: pgmap v6225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:35 smithi033 ceph-mon[27845]: pgmap v6225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:35 smithi096 ceph-mon[32753]: pgmap v6225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:37 smithi194 ceph-mon[33017]: pgmap v6226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:37 smithi033 ceph-mon[27845]: pgmap v6226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:37 smithi096 ceph-mon[32753]: pgmap v6226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10596.849867 secs 2024-04-26T14:47:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10596.849867 secs 2024-04-26T14:47:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10596.849867 secs 2024-04-26T14:47:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:39 smithi033 ceph-mon[27845]: pgmap v6227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10599.631242 secs 2024-04-26T14:47:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:39 smithi096 ceph-mon[32753]: pgmap v6227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10599.631242 secs 2024-04-26T14:47:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:39 smithi194 ceph-mon[33017]: pgmap v6227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10599.631242 secs 2024-04-26T14:47:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:40.549+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:40.549+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:40.549+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:47:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:41 smithi033 ceph-mon[27845]: pgmap v6228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:41 smithi096 ceph-mon[32753]: pgmap v6228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:41 smithi194 ceph-mon[33017]: pgmap v6228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:43 smithi033 ceph-mon[27845]: pgmap v6229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10601.849947 secs 2024-04-26T14:47:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:43 smithi096 ceph-mon[32753]: pgmap v6229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10601.849947 secs 2024-04-26T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:43 smithi194 ceph-mon[33017]: pgmap v6229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10601.849947 secs 2024-04-26T14:47:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10604.631361 secs 2024-04-26T14:47:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10604.631361 secs 2024-04-26T14:47:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10604.631361 secs 2024-04-26T14:47:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:45 smithi033 ceph-mon[27845]: pgmap v6230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:45 smithi096 ceph-mon[32753]: pgmap v6230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:45 smithi194 ceph-mon[33017]: pgmap v6230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:47 smithi033 ceph-mon[27845]: pgmap v6231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:47 smithi096 ceph-mon[32753]: pgmap v6231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:47 smithi194 ceph-mon[33017]: pgmap v6231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10606.850030 secs 2024-04-26T14:47:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10606.850030 secs 2024-04-26T14:47:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10606.850030 secs 2024-04-26T14:47:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:49 smithi033 ceph-mon[27845]: pgmap v6232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10609.631456 secs 2024-04-26T14:47:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:49 smithi096 ceph-mon[32753]: pgmap v6232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10609.631456 secs 2024-04-26T14:47:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:49 smithi194 ceph-mon[33017]: pgmap v6232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10609.631456 secs 2024-04-26T14:47:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:51 smithi033 ceph-mon[27845]: pgmap v6233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:51 smithi096 ceph-mon[32753]: pgmap v6233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:51 smithi194 ceph-mon[33017]: pgmap v6233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:53 smithi033 ceph-mon[27845]: pgmap v6234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10611.850111 secs 2024-04-26T14:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:53 smithi096 ceph-mon[32753]: pgmap v6234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10611.850111 secs 2024-04-26T14:47:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:53 smithi194 ceph-mon[33017]: pgmap v6234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10611.850111 secs 2024-04-26T14:47:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10614.631574 secs 2024-04-26T14:47:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10614.631574 secs 2024-04-26T14:47:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10614.631574 secs 2024-04-26T14:47:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:55 smithi096 ceph-mon[32753]: pgmap v6235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:55.561+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:55.561+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:47:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:47:55.561+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:47:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:55 smithi194 ceph-mon[33017]: pgmap v6235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:55 smithi033 ceph-mon[27845]: pgmap v6235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:57 smithi096 ceph-mon[32753]: pgmap v6236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:57 smithi194 ceph-mon[33017]: pgmap v6236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:57 smithi033 ceph-mon[27845]: pgmap v6236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:47:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10616.850186 secs 2024-04-26T14:47:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10616.850186 secs 2024-04-26T14:47:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10616.850186 secs 2024-04-26T14:47:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:59 smithi194 ceph-mon[33017]: pgmap v6237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:47:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10619.631649 secs 2024-04-26T14:47:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:59 smithi033 ceph-mon[27845]: pgmap v6237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:47:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10619.631649 secs 2024-04-26T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:59 smithi096 ceph-mon[32753]: pgmap v6237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:47:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10619.631649 secs 2024-04-26T14:48:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:01 smithi194 ceph-mon[33017]: pgmap v6238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:01 smithi033 ceph-mon[27845]: pgmap v6238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:01 smithi096 ceph-mon[32753]: pgmap v6238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:03.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:03 smithi033 ceph-mon[27845]: pgmap v6239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:03.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10621.850276 secs 2024-04-26T14:48:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:03 smithi194 ceph-mon[33017]: pgmap v6239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10621.850276 secs 2024-04-26T14:48:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:03 smithi096 ceph-mon[32753]: pgmap v6239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10621.850276 secs 2024-04-26T14:48:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10624.631758 secs 2024-04-26T14:48:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10624.631758 secs 2024-04-26T14:48:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10624.631758 secs 2024-04-26T14:48:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:05 smithi194 ceph-mon[33017]: pgmap v6240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:05 smithi033 ceph-mon[27845]: pgmap v6240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:05 smithi096 ceph-mon[32753]: pgmap v6240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:07 smithi194 ceph-mon[33017]: pgmap v6241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:07 smithi033 ceph-mon[27845]: pgmap v6241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:07 smithi096 ceph-mon[32753]: pgmap v6241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10626.850375 secs 2024-04-26T14:48:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10626.850375 secs 2024-04-26T14:48:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10626.850375 secs 2024-04-26T14:48:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:09 smithi194 ceph-mon[33017]: pgmap v6242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10629.631899 secs 2024-04-26T14:48:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:09 smithi033 ceph-mon[27845]: pgmap v6242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10629.631899 secs 2024-04-26T14:48:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:09 smithi096 ceph-mon[32753]: pgmap v6242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10629.631899 secs 2024-04-26T14:48:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:10.573+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:10.573+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:10.573+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:48:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:11 smithi194 ceph-mon[33017]: pgmap v6243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:11 smithi033 ceph-mon[27845]: pgmap v6243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:11 smithi096 ceph-mon[32753]: pgmap v6243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:13 smithi194 ceph-mon[33017]: pgmap v6244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10631.850474 secs 2024-04-26T14:48:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:13 smithi033 ceph-mon[27845]: pgmap v6244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10631.850474 secs 2024-04-26T14:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:13 smithi096 ceph-mon[32753]: pgmap v6244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10631.850474 secs 2024-04-26T14:48:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10634.632017 secs 2024-04-26T14:48:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:48:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10634.632017 secs 2024-04-26T14:48:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:48:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10634.632017 secs 2024-04-26T14:48:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:48:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:15 smithi033 ceph-mon[27845]: pgmap v6245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.450 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:15 smithi194 ceph-mon[33017]: pgmap v6245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:15.450 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.450 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.450 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.450 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:15 smithi096 ceph-mon[32753]: pgmap v6245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: pgmap v6246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: pgmap v6246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: pgmap v6246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:18 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:48:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:18 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:48:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:18 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:48:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:18 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:48:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10636.850557 secs 2024-04-26T14:48:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:18 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:48:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:18 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:48:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:18 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:48:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:18 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:48:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10636.850557 secs 2024-04-26T14:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:18 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:18 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:18 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:18 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10636.850557 secs 2024-04-26T14:48:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:19 smithi194 ceph-mon[33017]: pgmap v6247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10639.632170 secs 2024-04-26T14:48:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:19 smithi033 ceph-mon[27845]: pgmap v6247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10639.632170 secs 2024-04-26T14:48:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:19 smithi096 ceph-mon[32753]: pgmap v6247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10639.632170 secs 2024-04-26T14:48:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:48:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:48:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:48:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:48:20.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:48:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:48:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:48:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:48:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:21 smithi194 ceph-mon[33017]: pgmap v6248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:21 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:48:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:21 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:48:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:21 smithi194 ceph-mon[33017]: pgmap v6249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:48:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:21 smithi194 ceph-mon[33017]: pgmap v6250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:21 smithi033 ceph-mon[27845]: pgmap v6248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:21 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:48:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:21 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:48:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:21 smithi033 ceph-mon[27845]: pgmap v6249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:48:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:21 smithi033 ceph-mon[27845]: pgmap v6250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:21 smithi096 ceph-mon[32753]: pgmap v6248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:21 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:48:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:21 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:48:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:21 smithi096 ceph-mon[32753]: pgmap v6249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:48:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:21 smithi096 ceph-mon[32753]: pgmap v6250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:23 smithi194 ceph-mon[33017]: pgmap v6251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10641.850633 secs 2024-04-26T14:48:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:23 smithi033 ceph-mon[27845]: pgmap v6251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10641.850633 secs 2024-04-26T14:48:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:23 smithi096 ceph-mon[32753]: pgmap v6251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10641.850633 secs 2024-04-26T14:48:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10644.632287 secs 2024-04-26T14:48:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10644.632287 secs 2024-04-26T14:48:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10644.632287 secs 2024-04-26T14:48:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:25.578+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:25.578+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:25.578+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:48:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:25 smithi194 ceph-mon[33017]: pgmap v6252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:48:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:25 smithi033 ceph-mon[27845]: pgmap v6252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:48:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:25 smithi096 ceph-mon[32753]: pgmap v6252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:48:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:27 smithi194 ceph-mon[33017]: pgmap v6253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:27 smithi033 ceph-mon[27845]: pgmap v6253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:27 smithi096 ceph-mon[32753]: pgmap v6253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:48:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10646.850733 secs 2024-04-26T14:48:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10646.850733 secs 2024-04-26T14:48:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10646.850733 secs 2024-04-26T14:48:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:29 smithi194 ceph-mon[33017]: pgmap v6254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:48:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10649.632412 secs 2024-04-26T14:48:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:29 smithi033 ceph-mon[27845]: pgmap v6254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:48:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10649.632412 secs 2024-04-26T14:48:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:29 smithi096 ceph-mon[32753]: pgmap v6254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:48:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10649.632412 secs 2024-04-26T14:48:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:30 smithi194 ceph-mon[33017]: pgmap v6255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:48:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:30 smithi033 ceph-mon[27845]: pgmap v6255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:48:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:30 smithi096 ceph-mon[32753]: pgmap v6255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:48:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:32 smithi194 ceph-mon[33017]: pgmap v6256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10651.850835 secs 2024-04-26T14:48:33.379 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:32 smithi033 ceph-mon[27845]: pgmap v6256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:33.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10651.850835 secs 2024-04-26T14:48:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:32 smithi096 ceph-mon[32753]: pgmap v6256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10651.850835 secs 2024-04-26T14:48:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10654.632530 secs 2024-04-26T14:48:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10654.632530 secs 2024-04-26T14:48:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10654.632530 secs 2024-04-26T14:48:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:34 smithi033 ceph-mon[27845]: pgmap v6257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:34 smithi194 ceph-mon[33017]: pgmap v6257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:34 smithi096 ceph-mon[32753]: pgmap v6257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:36 smithi194 ceph-mon[33017]: pgmap v6258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:36 smithi033 ceph-mon[27845]: pgmap v6258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:36 smithi096 ceph-mon[32753]: pgmap v6258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10656.850935 secs 2024-04-26T14:48:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10656.850935 secs 2024-04-26T14:48:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10656.850935 secs 2024-04-26T14:48:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:39 smithi194 ceph-mon[33017]: pgmap v6259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10659.632661 secs 2024-04-26T14:48:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:39 smithi033 ceph-mon[27845]: pgmap v6259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10659.632661 secs 2024-04-26T14:48:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:39 smithi096 ceph-mon[32753]: pgmap v6259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10659.632661 secs 2024-04-26T14:48:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:40.580+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:40.580+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:40.580+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:48:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:41 smithi194 ceph-mon[33017]: pgmap v6260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:41 smithi033 ceph-mon[27845]: pgmap v6260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:41 smithi096 ceph-mon[32753]: pgmap v6260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:43 smithi194 ceph-mon[33017]: pgmap v6261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10661.851023 secs 2024-04-26T14:48:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:43 smithi033 ceph-mon[27845]: pgmap v6261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10661.851023 secs 2024-04-26T14:48:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:43 smithi096 ceph-mon[32753]: pgmap v6261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10661.851023 secs 2024-04-26T14:48:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10664.632736 secs 2024-04-26T14:48:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10664.632736 secs 2024-04-26T14:48:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10664.632736 secs 2024-04-26T14:48:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:45 smithi194 ceph-mon[33017]: pgmap v6262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:45 smithi033 ceph-mon[27845]: pgmap v6262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:45 smithi096 ceph-mon[32753]: pgmap v6262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:47 smithi194 ceph-mon[33017]: pgmap v6263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:47 smithi033 ceph-mon[27845]: pgmap v6263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:47 smithi096 ceph-mon[32753]: pgmap v6263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10666.851084 secs 2024-04-26T14:48:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10666.851084 secs 2024-04-26T14:48:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10666.851084 secs 2024-04-26T14:48:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:49 smithi194 ceph-mon[33017]: pgmap v6264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10669.632854 secs 2024-04-26T14:48:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:49 smithi033 ceph-mon[27845]: pgmap v6264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10669.632854 secs 2024-04-26T14:48:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:49 smithi096 ceph-mon[32753]: pgmap v6264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10669.632854 secs 2024-04-26T14:48:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:51 smithi194 ceph-mon[33017]: pgmap v6265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:51 smithi033 ceph-mon[27845]: pgmap v6265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:51 smithi096 ceph-mon[32753]: pgmap v6265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:53 smithi194 ceph-mon[33017]: pgmap v6266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10671.851183 secs 2024-04-26T14:48:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:53 smithi033 ceph-mon[27845]: pgmap v6266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10671.851183 secs 2024-04-26T14:48:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:53 smithi096 ceph-mon[32753]: pgmap v6266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10671.851183 secs 2024-04-26T14:48:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10674.633003 secs 2024-04-26T14:48:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10674.633003 secs 2024-04-26T14:48:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10674.633003 secs 2024-04-26T14:48:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:55 smithi194 ceph-mon[33017]: pgmap v6267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:55 smithi033 ceph-mon[27845]: pgmap v6267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:55 smithi096 ceph-mon[32753]: pgmap v6267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:55.594+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:55.594+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:48:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:48:55.594+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:48:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:57 smithi194 ceph-mon[33017]: pgmap v6268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:57 smithi033 ceph-mon[27845]: pgmap v6268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:57 smithi096 ceph-mon[32753]: pgmap v6268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:48:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10676.851286 secs 2024-04-26T14:48:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10676.851286 secs 2024-04-26T14:48:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10676.851286 secs 2024-04-26T14:48:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:59 smithi194 ceph-mon[33017]: pgmap v6269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10679.633138 secs 2024-04-26T14:48:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:59 smithi033 ceph-mon[27845]: pgmap v6269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10679.633138 secs 2024-04-26T14:48:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:59 smithi096 ceph-mon[32753]: pgmap v6269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:48:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10679.633138 secs 2024-04-26T14:49:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:01 smithi194 ceph-mon[33017]: pgmap v6270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:01 smithi033 ceph-mon[27845]: pgmap v6270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:01 smithi096 ceph-mon[32753]: pgmap v6270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:03 smithi033 ceph-mon[27845]: pgmap v6271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10681.851353 secs 2024-04-26T14:49:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:03 smithi096 ceph-mon[32753]: pgmap v6271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10681.851353 secs 2024-04-26T14:49:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:03 smithi194 ceph-mon[33017]: pgmap v6271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10681.851353 secs 2024-04-26T14:49:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10684.633263 secs 2024-04-26T14:49:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10684.633263 secs 2024-04-26T14:49:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10684.633263 secs 2024-04-26T14:49:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:05 smithi033 ceph-mon[27845]: pgmap v6272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:05 smithi096 ceph-mon[32753]: pgmap v6272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:05 smithi194 ceph-mon[33017]: pgmap v6272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:07 smithi033 ceph-mon[27845]: pgmap v6273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:07 smithi096 ceph-mon[32753]: pgmap v6273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:07 smithi194 ceph-mon[33017]: pgmap v6273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10686.851441 secs 2024-04-26T14:49:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10686.851441 secs 2024-04-26T14:49:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10686.851441 secs 2024-04-26T14:49:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:09 smithi033 ceph-mon[27845]: pgmap v6274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10689.633359 secs 2024-04-26T14:49:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:09 smithi096 ceph-mon[32753]: pgmap v6274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10689.633359 secs 2024-04-26T14:49:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:09 smithi194 ceph-mon[33017]: pgmap v6274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10689.633359 secs 2024-04-26T14:49:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:10.608+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:10.608+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:10.608+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:49:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:11 smithi033 ceph-mon[27845]: pgmap v6275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:11 smithi096 ceph-mon[32753]: pgmap v6275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:11 smithi194 ceph-mon[33017]: pgmap v6275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:13 smithi033 ceph-mon[27845]: pgmap v6276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10691.851540 secs 2024-04-26T14:49:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:13 smithi096 ceph-mon[32753]: pgmap v6276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10691.851540 secs 2024-04-26T14:49:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:13 smithi194 ceph-mon[33017]: pgmap v6276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10691.851540 secs 2024-04-26T14:49:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10694.633500 secs 2024-04-26T14:49:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10694.633500 secs 2024-04-26T14:49:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10694.633500 secs 2024-04-26T14:49:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:15 smithi033 ceph-mon[27845]: pgmap v6277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:15 smithi096 ceph-mon[32753]: pgmap v6277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:15 smithi194 ceph-mon[33017]: pgmap v6277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:16 smithi194 ceph-mon[33017]: pgmap v6278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:16 smithi033 ceph-mon[27845]: pgmap v6278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:16 smithi096 ceph-mon[32753]: pgmap v6278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10696.851643 secs 2024-04-26T14:49:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10696.851643 secs 2024-04-26T14:49:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10696.851643 secs 2024-04-26T14:49:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:18 smithi033 ceph-mon[27845]: pgmap v6279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10699.633637 secs 2024-04-26T14:49:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:18 smithi096 ceph-mon[32753]: pgmap v6279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10699.633637 secs 2024-04-26T14:49:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:18 smithi194 ceph-mon[33017]: pgmap v6279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10699.633637 secs 2024-04-26T14:49:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:20 smithi194 ceph-mon[33017]: pgmap v6280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:49:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:20 smithi033 ceph-mon[27845]: pgmap v6280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:49:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:21 smithi096 ceph-mon[32753]: pgmap v6280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:49:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:49:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:49:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:49:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:49:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:49:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:49:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:49:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:49:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:49:23.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:23 smithi096 ceph-mon[32753]: pgmap v6281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:23.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:23 smithi096 ceph-mon[32753]: pgmap v6282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:23.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10701.851738 secs 2024-04-26T14:49:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:23 smithi194 ceph-mon[33017]: pgmap v6281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:23 smithi194 ceph-mon[33017]: pgmap v6282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10701.851738 secs 2024-04-26T14:49:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:23 smithi033 ceph-mon[27845]: pgmap v6281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:23 smithi033 ceph-mon[27845]: pgmap v6282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10701.851738 secs 2024-04-26T14:49:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10704.633755 secs 2024-04-26T14:49:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10704.633755 secs 2024-04-26T14:49:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10704.633755 secs 2024-04-26T14:49:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:25 smithi194 ceph-mon[33017]: pgmap v6283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:25 smithi033 ceph-mon[27845]: pgmap v6283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:25 smithi096 ceph-mon[32753]: pgmap v6283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:25.617+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:25.617+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:25.617+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:49:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:27 smithi194 ceph-mon[33017]: pgmap v6284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:27 smithi033 ceph-mon[27845]: pgmap v6284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:27 smithi096 ceph-mon[32753]: pgmap v6284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10706.851841 secs 2024-04-26T14:49:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10706.851841 secs 2024-04-26T14:49:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10706.851841 secs 2024-04-26T14:49:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:29 smithi194 ceph-mon[33017]: pgmap v6285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10709.633859 secs 2024-04-26T14:49:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:29 smithi033 ceph-mon[27845]: pgmap v6285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10709.633859 secs 2024-04-26T14:49:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:29 smithi096 ceph-mon[32753]: pgmap v6285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10709.633859 secs 2024-04-26T14:49:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:31 smithi194 ceph-mon[33017]: pgmap v6286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:31 smithi033 ceph-mon[27845]: pgmap v6286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:31 smithi096 ceph-mon[32753]: pgmap v6286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:49:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:33 smithi194 ceph-mon[33017]: pgmap v6287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:49:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10711.851941 secs 2024-04-26T14:49:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:33 smithi033 ceph-mon[27845]: pgmap v6287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:49:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10711.851941 secs 2024-04-26T14:49:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:33 smithi096 ceph-mon[32753]: pgmap v6287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:49:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10711.851941 secs 2024-04-26T14:49:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10714.633980 secs 2024-04-26T14:49:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10714.633980 secs 2024-04-26T14:49:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10714.633980 secs 2024-04-26T14:49:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:35 smithi194 ceph-mon[33017]: pgmap v6288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:35 smithi033 ceph-mon[27845]: pgmap v6288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:35 smithi096 ceph-mon[32753]: pgmap v6288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:37.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:37 smithi033 ceph-mon[27845]: pgmap v6289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:37 smithi194 ceph-mon[33017]: pgmap v6289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:37 smithi096 ceph-mon[32753]: pgmap v6289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10716.852044 secs 2024-04-26T14:49:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10716.852044 secs 2024-04-26T14:49:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10716.852044 secs 2024-04-26T14:49:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:39 smithi194 ceph-mon[33017]: pgmap v6290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10719.634099 secs 2024-04-26T14:49:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:39 smithi033 ceph-mon[27845]: pgmap v6290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10719.634099 secs 2024-04-26T14:49:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:39 smithi096 ceph-mon[32753]: pgmap v6290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10719.634099 secs 2024-04-26T14:49:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:40.633+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:40.633+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:40.633+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:49:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:41 smithi194 ceph-mon[33017]: pgmap v6291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:41 smithi033 ceph-mon[27845]: pgmap v6291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:41 smithi096 ceph-mon[32753]: pgmap v6291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:43 smithi194 ceph-mon[33017]: pgmap v6292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10721.852133 secs 2024-04-26T14:49:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:43 smithi033 ceph-mon[27845]: pgmap v6292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10721.852133 secs 2024-04-26T14:49:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:43 smithi096 ceph-mon[32753]: pgmap v6292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10721.852133 secs 2024-04-26T14:49:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10724.634211 secs 2024-04-26T14:49:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10724.634211 secs 2024-04-26T14:49:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10724.634211 secs 2024-04-26T14:49:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:45 smithi033 ceph-mon[27845]: pgmap v6293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:45 smithi096 ceph-mon[32753]: pgmap v6293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:45 smithi194 ceph-mon[33017]: pgmap v6293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:47 smithi033 ceph-mon[27845]: pgmap v6294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:47 smithi096 ceph-mon[32753]: pgmap v6294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:47 smithi194 ceph-mon[33017]: pgmap v6294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10726.852236 secs 2024-04-26T14:49:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10726.852236 secs 2024-04-26T14:49:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10726.852236 secs 2024-04-26T14:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:49 smithi033 ceph-mon[27845]: pgmap v6295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10729.634287 secs 2024-04-26T14:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:49 smithi096 ceph-mon[32753]: pgmap v6295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10729.634287 secs 2024-04-26T14:49:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:49 smithi194 ceph-mon[33017]: pgmap v6295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10729.634287 secs 2024-04-26T14:49:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:51 smithi033 ceph-mon[27845]: pgmap v6296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:51 smithi096 ceph-mon[32753]: pgmap v6296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:51 smithi194 ceph-mon[33017]: pgmap v6296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:53 smithi033 ceph-mon[27845]: pgmap v6297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10731.852333 secs 2024-04-26T14:49:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:53 smithi096 ceph-mon[32753]: pgmap v6297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10731.852333 secs 2024-04-26T14:49:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:53 smithi194 ceph-mon[33017]: pgmap v6297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10731.852333 secs 2024-04-26T14:49:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10734.634354 secs 2024-04-26T14:49:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10734.634354 secs 2024-04-26T14:49:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10734.634354 secs 2024-04-26T14:49:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:55 smithi033 ceph-mon[27845]: pgmap v6298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:55 smithi096 ceph-mon[32753]: pgmap v6298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:55 smithi194 ceph-mon[33017]: pgmap v6298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:55.641+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:55.641+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:49:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:49:55.641+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:49:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:57 smithi033 ceph-mon[27845]: pgmap v6299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:57 smithi096 ceph-mon[32753]: pgmap v6299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:57 smithi194 ceph-mon[33017]: pgmap v6299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:49:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10736.852435 secs 2024-04-26T14:49:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10736.852435 secs 2024-04-26T14:49:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10736.852435 secs 2024-04-26T14:49:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:59 smithi033 ceph-mon[27845]: pgmap v6300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10739.634472 secs 2024-04-26T14:49:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:59 smithi096 ceph-mon[32753]: pgmap v6300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10739.634472 secs 2024-04-26T14:49:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:59 smithi194 ceph-mon[33017]: pgmap v6300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:49:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10739.634472 secs 2024-04-26T14:50:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:50:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:50:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T14:50:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:01 smithi194 ceph-mon[33017]: pgmap v6301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:01 smithi033 ceph-mon[27845]: pgmap v6301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:01 smithi096 ceph-mon[32753]: pgmap v6301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:03.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:03 smithi033 ceph-mon[27845]: pgmap v6302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:03.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10741.852496 secs 2024-04-26T14:50:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:03 smithi194 ceph-mon[33017]: pgmap v6302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10741.852496 secs 2024-04-26T14:50:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:03 smithi096 ceph-mon[32753]: pgmap v6302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10741.852496 secs 2024-04-26T14:50:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10744.634618 secs 2024-04-26T14:50:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10744.634618 secs 2024-04-26T14:50:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10744.634618 secs 2024-04-26T14:50:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:05 smithi194 ceph-mon[33017]: pgmap v6303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:05 smithi033 ceph-mon[27845]: pgmap v6303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:05 smithi096 ceph-mon[32753]: pgmap v6303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:07 smithi194 ceph-mon[33017]: pgmap v6304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:07 smithi033 ceph-mon[27845]: pgmap v6304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:07 smithi096 ceph-mon[32753]: pgmap v6304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10746.852581 secs 2024-04-26T14:50:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10746.852581 secs 2024-04-26T14:50:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10746.852581 secs 2024-04-26T14:50:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:09 smithi033 ceph-mon[27845]: pgmap v6305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10749.634733 secs 2024-04-26T14:50:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:09 smithi194 ceph-mon[33017]: pgmap v6305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10749.634733 secs 2024-04-26T14:50:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:09 smithi096 ceph-mon[32753]: pgmap v6305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10749.634733 secs 2024-04-26T14:50:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:10.643+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:10.643+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:10.643+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:50:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:11 smithi194 ceph-mon[33017]: pgmap v6306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:11 smithi033 ceph-mon[27845]: pgmap v6306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:11 smithi096 ceph-mon[32753]: pgmap v6306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:13 smithi194 ceph-mon[33017]: pgmap v6307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10751.852665 secs 2024-04-26T14:50:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:13 smithi033 ceph-mon[27845]: pgmap v6307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10751.852665 secs 2024-04-26T14:50:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:13 smithi096 ceph-mon[32753]: pgmap v6307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10751.852665 secs 2024-04-26T14:50:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10754.634839 secs 2024-04-26T14:50:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10754.634839 secs 2024-04-26T14:50:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10754.634839 secs 2024-04-26T14:50:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:15 smithi194 ceph-mon[33017]: pgmap v6308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:15 smithi033 ceph-mon[27845]: pgmap v6308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:15 smithi096 ceph-mon[32753]: pgmap v6308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:17 smithi194 ceph-mon[33017]: pgmap v6309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:17 smithi033 ceph-mon[27845]: pgmap v6309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:17 smithi096 ceph-mon[32753]: pgmap v6309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10756.852746 secs 2024-04-26T14:50:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10756.852746 secs 2024-04-26T14:50:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10756.852746 secs 2024-04-26T14:50:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:19 smithi194 ceph-mon[33017]: pgmap v6310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10759.634951 secs 2024-04-26T14:50:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:19 smithi033 ceph-mon[27845]: pgmap v6310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10759.634951 secs 2024-04-26T14:50:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:19 smithi096 ceph-mon[32753]: pgmap v6310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10759.634951 secs 2024-04-26T14:50:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:21 smithi194 ceph-mon[33017]: pgmap v6311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:21 smithi033 ceph-mon[27845]: pgmap v6311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:21 smithi096 ceph-mon[32753]: pgmap v6311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:50:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:50:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:50:23.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:23 smithi096 ceph-mon[32753]: pgmap v6312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:23.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10761.852830 secs 2024-04-26T14:50:23.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:50:23.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:50:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:23 smithi194 ceph-mon[33017]: pgmap v6312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10761.852830 secs 2024-04-26T14:50:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:50:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:50:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:23.623+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:50:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:23.623+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:50:23.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:23.623+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:50:23.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:23.623+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:50:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:23 smithi033 ceph-mon[27845]: pgmap v6312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10761.852830 secs 2024-04-26T14:50:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:50:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:50:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10764.635070 secs 2024-04-26T14:50:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:24 smithi194 ceph-mon[33017]: pgmap v6313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:24 smithi194 ceph-mon[33017]: pgmap v6314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:50:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10764.635070 secs 2024-04-26T14:50:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:24 smithi033 ceph-mon[27845]: pgmap v6313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:24 smithi033 ceph-mon[27845]: pgmap v6314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:50:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10764.635070 secs 2024-04-26T14:50:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:24 smithi096 ceph-mon[32753]: pgmap v6313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:24 smithi096 ceph-mon[32753]: pgmap v6314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:25.654+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:25.654+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:25.654+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:50:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:26 smithi194 ceph-mon[33017]: pgmap v6315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:26 smithi033 ceph-mon[27845]: pgmap v6315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:26 smithi096 ceph-mon[32753]: pgmap v6315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:28 smithi194 ceph-mon[33017]: pgmap v6316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10766.852913 secs 2024-04-26T14:50:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:28 smithi033 ceph-mon[27845]: pgmap v6316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10766.852913 secs 2024-04-26T14:50:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:28 smithi096 ceph-mon[32753]: pgmap v6316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10766.852913 secs 2024-04-26T14:50:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10769.635188 secs 2024-04-26T14:50:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10769.635188 secs 2024-04-26T14:50:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10769.635188 secs 2024-04-26T14:50:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:30 smithi194 ceph-mon[33017]: pgmap v6317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:50:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:30 smithi033 ceph-mon[27845]: pgmap v6317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:50:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:30 smithi096 ceph-mon[32753]: pgmap v6317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:50:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:32 smithi194 ceph-mon[33017]: pgmap v6318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:32 smithi033 ceph-mon[27845]: pgmap v6318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:32 smithi096 ceph-mon[32753]: pgmap v6318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:33.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10771.853009 secs 2024-04-26T14:50:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10771.853009 secs 2024-04-26T14:50:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10771.853009 secs 2024-04-26T14:50:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10774.635248 secs 2024-04-26T14:50:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:34 smithi194 ceph-mon[33017]: pgmap v6319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:50:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10774.635248 secs 2024-04-26T14:50:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:34 smithi033 ceph-mon[27845]: pgmap v6319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10774.635248 secs 2024-04-26T14:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:34 smithi096 ceph-mon[32753]: pgmap v6319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:50:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:36 smithi194 ceph-mon[33017]: pgmap v6320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:36 smithi033 ceph-mon[27845]: pgmap v6320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:36 smithi096 ceph-mon[32753]: pgmap v6320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:38 smithi194 ceph-mon[33017]: pgmap v6321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10776.853095 secs 2024-04-26T14:50:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:38 smithi033 ceph-mon[27845]: pgmap v6321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10776.853095 secs 2024-04-26T14:50:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:38 smithi096 ceph-mon[32753]: pgmap v6321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10776.853095 secs 2024-04-26T14:50:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10779.635369 secs 2024-04-26T14:50:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10779.635369 secs 2024-04-26T14:50:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10779.635369 secs 2024-04-26T14:50:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:40.656+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:40.656+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:40.656+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:50:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:40 smithi033 ceph-mon[27845]: pgmap v6322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:50:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:40 smithi096 ceph-mon[32753]: pgmap v6322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:50:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:40 smithi194 ceph-mon[33017]: pgmap v6322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:50:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:41 smithi033 ceph-mon[27845]: pgmap v6323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:41 smithi096 ceph-mon[32753]: pgmap v6323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:41 smithi194 ceph-mon[33017]: pgmap v6323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10781.853176 secs 2024-04-26T14:50:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10781.853176 secs 2024-04-26T14:50:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10781.853176 secs 2024-04-26T14:50:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:43 smithi194 ceph-mon[33017]: pgmap v6324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:43 smithi033 ceph-mon[27845]: pgmap v6324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:43 smithi096 ceph-mon[32753]: pgmap v6324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10784.635477 secs 2024-04-26T14:50:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10784.635477 secs 2024-04-26T14:50:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10784.635477 secs 2024-04-26T14:50:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:45 smithi194 ceph-mon[33017]: pgmap v6325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:45 smithi033 ceph-mon[27845]: pgmap v6325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:45 smithi096 ceph-mon[32753]: pgmap v6325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:48 smithi194 ceph-mon[33017]: pgmap v6326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10786.853262 secs 2024-04-26T14:50:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:48 smithi033 ceph-mon[27845]: pgmap v6326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10786.853262 secs 2024-04-26T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:48 smithi096 ceph-mon[32753]: pgmap v6326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10786.853262 secs 2024-04-26T14:50:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10789.635593 secs 2024-04-26T14:50:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10789.635593 secs 2024-04-26T14:50:49.785 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10789.635593 secs 2024-04-26T14:50:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:50 smithi033 ceph-mon[27845]: pgmap v6327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:50:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:50 smithi096 ceph-mon[32753]: pgmap v6327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:50:50.709 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:50 smithi194 ceph-mon[33017]: pgmap v6327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:50:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:51 smithi033 ceph-mon[27845]: pgmap v6328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:51 smithi096 ceph-mon[32753]: pgmap v6328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:51 smithi194 ceph-mon[33017]: pgmap v6328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:50:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10791.853346 secs 2024-04-26T14:50:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10791.853346 secs 2024-04-26T14:50:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10791.853346 secs 2024-04-26T14:50:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:53 smithi194 ceph-mon[33017]: pgmap v6329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:53 smithi033 ceph-mon[27845]: pgmap v6329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:53 smithi096 ceph-mon[32753]: pgmap v6329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10794.635708 secs 2024-04-26T14:50:55.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10794.635708 secs 2024-04-26T14:50:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10794.635708 secs 2024-04-26T14:50:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:55.671+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:55.671+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:50:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:50:55.671+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:50:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:55 smithi194 ceph-mon[33017]: pgmap v6330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:55 smithi033 ceph-mon[27845]: pgmap v6330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:55 smithi096 ceph-mon[32753]: pgmap v6330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:50:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:58 smithi194 ceph-mon[33017]: pgmap v6331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10796.853426 secs 2024-04-26T14:50:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:58 smithi033 ceph-mon[27845]: pgmap v6331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10796.853426 secs 2024-04-26T14:50:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:58 smithi096 ceph-mon[32753]: pgmap v6331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:50:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10796.853426 secs 2024-04-26T14:50:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10799.635800 secs 2024-04-26T14:50:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10799.635800 secs 2024-04-26T14:50:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10799.635800 secs 2024-04-26T14:51:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:00 smithi194 ceph-mon[33017]: pgmap v6332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:00 smithi033 ceph-mon[27845]: pgmap v6332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:00 smithi096 ceph-mon[32753]: pgmap v6332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:01 smithi033 ceph-mon[27845]: pgmap v6333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:01 smithi096 ceph-mon[32753]: pgmap v6333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:01 smithi194 ceph-mon[33017]: pgmap v6333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10801.853508 secs 2024-04-26T14:51:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10801.853508 secs 2024-04-26T14:51:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10801.853508 secs 2024-04-26T14:51:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:03 smithi194 ceph-mon[33017]: pgmap v6334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:03 smithi033 ceph-mon[27845]: pgmap v6334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:03 smithi096 ceph-mon[32753]: pgmap v6334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10804.635923 secs 2024-04-26T14:51:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10804.635923 secs 2024-04-26T14:51:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10804.635923 secs 2024-04-26T14:51:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:05 smithi194 ceph-mon[33017]: pgmap v6335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:05 smithi033 ceph-mon[27845]: pgmap v6335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:05 smithi096 ceph-mon[32753]: pgmap v6335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:08.631+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:51:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:08.631+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:51:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:08.631+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:51:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:08.631+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:51:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:08 smithi033 ceph-mon[27845]: pgmap v6336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10806.853594 secs 2024-04-26T14:51:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:08 smithi096 ceph-mon[32753]: pgmap v6336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10806.853594 secs 2024-04-26T14:51:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:08 smithi194 ceph-mon[33017]: pgmap v6336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10806.853594 secs 2024-04-26T14:51:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10809.636015 secs 2024-04-26T14:51:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10809.636015 secs 2024-04-26T14:51:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10809.636015 secs 2024-04-26T14:51:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:10 smithi033 ceph-mon[27845]: pgmap v6337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:10 smithi096 ceph-mon[32753]: pgmap v6337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:10.684+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:10.684+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:10.685+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:51:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:10 smithi194 ceph-mon[33017]: pgmap v6337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:11 smithi033 ceph-mon[27845]: pgmap v6338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:11 smithi096 ceph-mon[32753]: pgmap v6338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:11 smithi194 ceph-mon[33017]: pgmap v6338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10811.853683 secs 2024-04-26T14:51:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10811.853683 secs 2024-04-26T14:51:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10811.853683 secs 2024-04-26T14:51:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:13 smithi194 ceph-mon[33017]: pgmap v6339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:13 smithi033 ceph-mon[27845]: pgmap v6339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:13 smithi096 ceph-mon[32753]: pgmap v6339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10814.636131 secs 2024-04-26T14:51:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10814.636131 secs 2024-04-26T14:51:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10814.636131 secs 2024-04-26T14:51:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:15 smithi194 ceph-mon[33017]: pgmap v6340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:15 smithi033 ceph-mon[27845]: pgmap v6340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:15 smithi096 ceph-mon[32753]: pgmap v6340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:18 smithi033 ceph-mon[27845]: pgmap v6341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10816.853771 secs 2024-04-26T14:51:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:18 smithi096 ceph-mon[32753]: pgmap v6341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10816.853771 secs 2024-04-26T14:51:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:18 smithi194 ceph-mon[33017]: pgmap v6341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10816.853771 secs 2024-04-26T14:51:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10819.636217 secs 2024-04-26T14:51:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10819.636217 secs 2024-04-26T14:51:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10819.636217 secs 2024-04-26T14:51:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:20 smithi033 ceph-mon[27845]: pgmap v6342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:20 smithi096 ceph-mon[32753]: pgmap v6342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:20 smithi194 ceph-mon[33017]: pgmap v6342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T14:51:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:21 smithi033 ceph-mon[27845]: pgmap v6343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:21 smithi096 ceph-mon[32753]: pgmap v6343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:21 smithi194 ceph-mon[33017]: pgmap v6343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10821.853859 secs 2024-04-26T14:51:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10821.853859 secs 2024-04-26T14:51:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10821.853859 secs 2024-04-26T14:51:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:23 smithi194 ceph-mon[33017]: pgmap v6344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:51:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:23 smithi033 ceph-mon[27845]: pgmap v6344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:51:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:23 smithi096 ceph-mon[32753]: pgmap v6344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:51:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10824.636333 secs 2024-04-26T14:51:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10824.636333 secs 2024-04-26T14:51:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10824.636333 secs 2024-04-26T14:51:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:25.689+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:25.689+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:25.689+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:51:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:51:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:51:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:25 smithi194 ceph-mon[33017]: pgmap v6345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:51:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:51:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:51:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:51:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:25 smithi033 ceph-mon[27845]: pgmap v6345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:51:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:51:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:51:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:51:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:25 smithi096 ceph-mon[32753]: pgmap v6345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:51:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:51:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:28 smithi194 ceph-mon[33017]: pgmap v6346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10826.853946 secs 2024-04-26T14:51:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:28 smithi033 ceph-mon[27845]: pgmap v6346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10826.853946 secs 2024-04-26T14:51:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:28 smithi096 ceph-mon[32753]: pgmap v6346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10826.853946 secs 2024-04-26T14:51:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10829.636434 secs 2024-04-26T14:51:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10829.636434 secs 2024-04-26T14:51:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10829.636434 secs 2024-04-26T14:51:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:30 smithi194 ceph-mon[33017]: pgmap v6347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:30 smithi033 ceph-mon[27845]: pgmap v6347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:30 smithi096 ceph-mon[32753]: pgmap v6347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:32 smithi194 ceph-mon[33017]: pgmap v6348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:32 smithi033 ceph-mon[27845]: pgmap v6348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:32 smithi096 ceph-mon[32753]: pgmap v6348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:51:33.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10831.854035 secs 2024-04-26T14:51:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10831.854035 secs 2024-04-26T14:51:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10831.854035 secs 2024-04-26T14:51:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:34 smithi194 ceph-mon[33017]: pgmap v6349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10834.636571 secs 2024-04-26T14:51:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:34 smithi033 ceph-mon[27845]: pgmap v6349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10834.636571 secs 2024-04-26T14:51:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:34 smithi096 ceph-mon[32753]: pgmap v6349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10834.636571 secs 2024-04-26T14:51:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:36 smithi194 ceph-mon[33017]: pgmap v6350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:51:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:36 smithi033 ceph-mon[27845]: pgmap v6350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:51:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:36 smithi096 ceph-mon[32753]: pgmap v6350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:51:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:38 smithi194 ceph-mon[33017]: pgmap v6351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10836.854115 secs 2024-04-26T14:51:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:38 smithi033 ceph-mon[27845]: pgmap v6351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10836.854115 secs 2024-04-26T14:51:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:38 smithi096 ceph-mon[32753]: pgmap v6351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10836.854115 secs 2024-04-26T14:51:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10839.636636 secs 2024-04-26T14:51:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10839.636636 secs 2024-04-26T14:51:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10839.636636 secs 2024-04-26T14:51:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:40 smithi194 ceph-mon[33017]: pgmap v6352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:40 smithi033 ceph-mon[27845]: pgmap v6352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:40 smithi096 ceph-mon[32753]: pgmap v6352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:40.705+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:40.706+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:40.706+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:51:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:42 smithi194 ceph-mon[33017]: pgmap v6353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:42 smithi033 ceph-mon[27845]: pgmap v6353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:42 smithi096 ceph-mon[32753]: pgmap v6353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10841.854204 secs 2024-04-26T14:51:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10841.854204 secs 2024-04-26T14:51:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10841.854204 secs 2024-04-26T14:51:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:44 smithi194 ceph-mon[33017]: pgmap v6354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10844.636756 secs 2024-04-26T14:51:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:44 smithi033 ceph-mon[27845]: pgmap v6354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10844.636756 secs 2024-04-26T14:51:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:44 smithi096 ceph-mon[32753]: pgmap v6354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10844.636756 secs 2024-04-26T14:51:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:46 smithi194 ceph-mon[33017]: pgmap v6355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:46 smithi033 ceph-mon[27845]: pgmap v6355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:46 smithi096 ceph-mon[32753]: pgmap v6355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:48 smithi194 ceph-mon[33017]: pgmap v6356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10846.854291 secs 2024-04-26T14:51:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:48 smithi033 ceph-mon[27845]: pgmap v6356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10846.854291 secs 2024-04-26T14:51:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:48 smithi096 ceph-mon[32753]: pgmap v6356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10846.854291 secs 2024-04-26T14:51:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10849.636893 secs 2024-04-26T14:51:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10849.636893 secs 2024-04-26T14:51:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10849.636893 secs 2024-04-26T14:51:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:50 smithi033 ceph-mon[27845]: pgmap v6357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:50 smithi096 ceph-mon[32753]: pgmap v6357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:50 smithi194 ceph-mon[33017]: pgmap v6357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:52 smithi033 ceph-mon[27845]: pgmap v6358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:52 smithi096 ceph-mon[32753]: pgmap v6358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:52 smithi194 ceph-mon[33017]: pgmap v6358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10851.854376 secs 2024-04-26T14:51:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10851.854376 secs 2024-04-26T14:51:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10851.854376 secs 2024-04-26T14:51:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:54 smithi033 ceph-mon[27845]: pgmap v6359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10854.637035 secs 2024-04-26T14:51:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:54 smithi096 ceph-mon[32753]: pgmap v6359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10854.637035 secs 2024-04-26T14:51:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:54 smithi194 ceph-mon[33017]: pgmap v6359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10854.637035 secs 2024-04-26T14:51:55.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:55.711+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:55.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:55.711+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:51:55.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:51:55.712+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:51:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:56 smithi033 ceph-mon[27845]: pgmap v6360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:56 smithi096 ceph-mon[32753]: pgmap v6360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:56 smithi194 ceph-mon[33017]: pgmap v6360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:51:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:58 smithi033 ceph-mon[27845]: pgmap v6361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10856.854472 secs 2024-04-26T14:51:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:58 smithi096 ceph-mon[32753]: pgmap v6361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10856.854472 secs 2024-04-26T14:51:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:58 smithi194 ceph-mon[33017]: pgmap v6361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:51:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10856.854472 secs 2024-04-26T14:51:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:51:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10859.637175 secs 2024-04-26T14:51:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:51:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10859.637175 secs 2024-04-26T14:51:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:51:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10859.637175 secs 2024-04-26T14:52:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:00 smithi033 ceph-mon[27845]: pgmap v6362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:00 smithi096 ceph-mon[32753]: pgmap v6362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:00 smithi194 ceph-mon[33017]: pgmap v6362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:02 smithi033 ceph-mon[27845]: pgmap v6363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:02 smithi096 ceph-mon[32753]: pgmap v6363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:02 smithi194 ceph-mon[33017]: pgmap v6363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10861.854561 secs 2024-04-26T14:52:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10861.854561 secs 2024-04-26T14:52:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10861.854561 secs 2024-04-26T14:52:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:04 smithi033 ceph-mon[27845]: pgmap v6364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10864.637293 secs 2024-04-26T14:52:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:04 smithi096 ceph-mon[32753]: pgmap v6364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10864.637293 secs 2024-04-26T14:52:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:04 smithi194 ceph-mon[33017]: pgmap v6364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10864.637293 secs 2024-04-26T14:52:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:06 smithi096 ceph-mon[32753]: pgmap v6365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:06 smithi194 ceph-mon[33017]: pgmap v6365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:06 smithi033 ceph-mon[27845]: pgmap v6365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:08 smithi194 ceph-mon[33017]: pgmap v6366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10866.854662 secs 2024-04-26T14:52:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:08 smithi033 ceph-mon[27845]: pgmap v6366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10866.854662 secs 2024-04-26T14:52:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:08 smithi096 ceph-mon[32753]: pgmap v6366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10866.854662 secs 2024-04-26T14:52:09.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10869.637407 secs 2024-04-26T14:52:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10869.637407 secs 2024-04-26T14:52:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10869.637407 secs 2024-04-26T14:52:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:10 smithi194 ceph-mon[33017]: pgmap v6367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:10 smithi033 ceph-mon[27845]: pgmap v6367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:10 smithi096 ceph-mon[32753]: pgmap v6367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:10.726+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:10.726+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:10.726+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:52:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:12 smithi194 ceph-mon[33017]: pgmap v6368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:12 smithi033 ceph-mon[27845]: pgmap v6368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:12 smithi096 ceph-mon[32753]: pgmap v6368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10871.854764 secs 2024-04-26T14:52:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10871.854764 secs 2024-04-26T14:52:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10871.854764 secs 2024-04-26T14:52:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:14 smithi194 ceph-mon[33017]: pgmap v6369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10874.637466 secs 2024-04-26T14:52:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:14 smithi033 ceph-mon[27845]: pgmap v6369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10874.637466 secs 2024-04-26T14:52:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:14 smithi096 ceph-mon[32753]: pgmap v6369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10874.637466 secs 2024-04-26T14:52:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:16 smithi194 ceph-mon[33017]: pgmap v6370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:16 smithi033 ceph-mon[27845]: pgmap v6370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:16 smithi096 ceph-mon[32753]: pgmap v6370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:18 smithi194 ceph-mon[33017]: pgmap v6371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:18.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10876.854866 secs 2024-04-26T14:52:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:18 smithi033 ceph-mon[27845]: pgmap v6371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10876.854866 secs 2024-04-26T14:52:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:18 smithi096 ceph-mon[32753]: pgmap v6371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10876.854866 secs 2024-04-26T14:52:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10879.637591 secs 2024-04-26T14:52:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10879.637591 secs 2024-04-26T14:52:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10879.637591 secs 2024-04-26T14:52:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:20 smithi194 ceph-mon[33017]: pgmap v6372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:20 smithi033 ceph-mon[27845]: pgmap v6372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:20 smithi096 ceph-mon[32753]: pgmap v6372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:22 smithi194 ceph-mon[33017]: pgmap v6373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:22 smithi033 ceph-mon[27845]: pgmap v6373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:22 smithi096 ceph-mon[32753]: pgmap v6373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:23.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10881.854974 secs 2024-04-26T14:52:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10881.854974 secs 2024-04-26T14:52:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10881.854974 secs 2024-04-26T14:52:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:24 smithi194 ceph-mon[33017]: pgmap v6374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10884.637726 secs 2024-04-26T14:52:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:24 smithi033 ceph-mon[27845]: pgmap v6374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10884.637726 secs 2024-04-26T14:52:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:24 smithi096 ceph-mon[32753]: pgmap v6374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10884.637726 secs 2024-04-26T14:52:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:25.740+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:25.740+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:25.740+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:52:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:26 smithi194 ceph-mon[33017]: pgmap v6375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:52:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:26 smithi033 ceph-mon[27845]: pgmap v6375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:52:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:26 smithi096 ceph-mon[32753]: pgmap v6375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:52:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:52:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:52:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:52:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:52:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:52:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:52:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:52:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:52:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:52:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:28 smithi194 ceph-mon[33017]: pgmap v6376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:28 smithi194 ceph-mon[33017]: pgmap v6377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10886.855060 secs 2024-04-26T14:52:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:28 smithi033 ceph-mon[27845]: pgmap v6376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:28 smithi033 ceph-mon[27845]: pgmap v6377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10886.855060 secs 2024-04-26T14:52:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:28 smithi096 ceph-mon[32753]: pgmap v6376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:28 smithi096 ceph-mon[32753]: pgmap v6377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10886.855060 secs 2024-04-26T14:52:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10889.637863 secs 2024-04-26T14:52:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10889.637863 secs 2024-04-26T14:52:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10889.637863 secs 2024-04-26T14:52:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:30 smithi194 ceph-mon[33017]: pgmap v6378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:30 smithi033 ceph-mon[27845]: pgmap v6378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:30 smithi096 ceph-mon[32753]: pgmap v6378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:32 smithi194 ceph-mon[33017]: pgmap v6379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:32 smithi033 ceph-mon[27845]: pgmap v6379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:32 smithi096 ceph-mon[32753]: pgmap v6379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:33.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10891.855151 secs 2024-04-26T14:52:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10891.855151 secs 2024-04-26T14:52:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10891.855151 secs 2024-04-26T14:52:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:34 smithi194 ceph-mon[33017]: pgmap v6380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10894.637982 secs 2024-04-26T14:52:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:34 smithi033 ceph-mon[27845]: pgmap v6380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10894.637982 secs 2024-04-26T14:52:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:34 smithi096 ceph-mon[32753]: pgmap v6380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10894.637982 secs 2024-04-26T14:52:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:36 smithi194 ceph-mon[33017]: pgmap v6381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:36 smithi033 ceph-mon[27845]: pgmap v6381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:36 smithi096 ceph-mon[32753]: pgmap v6381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:52:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:38 smithi194 ceph-mon[33017]: pgmap v6382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:52:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10896.855213 secs 2024-04-26T14:52:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:38 smithi033 ceph-mon[27845]: pgmap v6382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:52:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10896.855213 secs 2024-04-26T14:52:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:38 smithi096 ceph-mon[32753]: pgmap v6382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:52:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10896.855213 secs 2024-04-26T14:52:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10899.638099 secs 2024-04-26T14:52:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10899.638099 secs 2024-04-26T14:52:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10899.638099 secs 2024-04-26T14:52:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:40 smithi194 ceph-mon[33017]: pgmap v6383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:40 smithi033 ceph-mon[27845]: pgmap v6383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:40 smithi096 ceph-mon[32753]: pgmap v6383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:40.751+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:40.751+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:52:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:40.751+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:42 smithi194 ceph-mon[33017]: pgmap v6384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:42 smithi033 ceph-mon[27845]: pgmap v6384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:42 smithi096 ceph-mon[32753]: pgmap v6384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10901.855315 secs 2024-04-26T14:52:43.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10901.855315 secs 2024-04-26T14:52:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10901.855315 secs 2024-04-26T14:52:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:44 smithi194 ceph-mon[33017]: pgmap v6385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10904.638209 secs 2024-04-26T14:52:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:44 smithi033 ceph-mon[27845]: pgmap v6385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10904.638209 secs 2024-04-26T14:52:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:44 smithi096 ceph-mon[32753]: pgmap v6385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10904.638209 secs 2024-04-26T14:52:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:46 smithi033 ceph-mon[27845]: pgmap v6386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:46 smithi096 ceph-mon[32753]: pgmap v6386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:46 smithi194 ceph-mon[33017]: pgmap v6386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:48 smithi033 ceph-mon[27845]: pgmap v6387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10906.855417 secs 2024-04-26T14:52:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:48 smithi096 ceph-mon[32753]: pgmap v6387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10906.855417 secs 2024-04-26T14:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:48 smithi194 ceph-mon[33017]: pgmap v6387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10906.855417 secs 2024-04-26T14:52:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10909.638330 secs 2024-04-26T14:52:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10909.638330 secs 2024-04-26T14:52:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10909.638330 secs 2024-04-26T14:52:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:50 smithi033 ceph-mon[27845]: pgmap v6388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:50 smithi096 ceph-mon[32753]: pgmap v6388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:50 smithi194 ceph-mon[33017]: pgmap v6388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:52 smithi033 ceph-mon[27845]: pgmap v6389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:52 smithi096 ceph-mon[32753]: pgmap v6389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:52 smithi194 ceph-mon[33017]: pgmap v6389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10911.855516 secs 2024-04-26T14:52:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10911.855516 secs 2024-04-26T14:52:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10911.855516 secs 2024-04-26T14:52:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:54 smithi033 ceph-mon[27845]: pgmap v6390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10914.638446 secs 2024-04-26T14:52:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:54 smithi096 ceph-mon[32753]: pgmap v6390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10914.638446 secs 2024-04-26T14:52:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:54 smithi194 ceph-mon[33017]: pgmap v6390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10914.638446 secs 2024-04-26T14:52:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:55 smithi033 ceph-mon[27845]: pgmap v6391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:55 smithi096 ceph-mon[32753]: pgmap v6391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:55.762+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:55.763+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:52:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:52:55.763+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:52:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:55 smithi194 ceph-mon[33017]: pgmap v6391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:52:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:57 smithi194 ceph-mon[33017]: pgmap v6392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10916.855617 secs 2024-04-26T14:52:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:57 smithi033 ceph-mon[27845]: pgmap v6392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10916.855617 secs 2024-04-26T14:52:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:57 smithi096 ceph-mon[32753]: pgmap v6392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:52:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10916.855617 secs 2024-04-26T14:53:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10919.638526 secs 2024-04-26T14:53:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:52:59 smithi194 ceph-mon[33017]: pgmap v6393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:00.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10919.638526 secs 2024-04-26T14:53:00.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:52:59 smithi033 ceph-mon[27845]: pgmap v6393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10919.638526 secs 2024-04-26T14:53:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:52:59 smithi096 ceph-mon[32753]: pgmap v6393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:01 smithi194 ceph-mon[33017]: pgmap v6394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:01 smithi033 ceph-mon[27845]: pgmap v6394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:01 smithi096 ceph-mon[32753]: pgmap v6394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:03.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10921.855719 secs 2024-04-26T14:53:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10921.855719 secs 2024-04-26T14:53:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10921.855719 secs 2024-04-26T14:53:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:03 smithi194 ceph-mon[33017]: pgmap v6395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:03 smithi033 ceph-mon[27845]: pgmap v6395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:03 smithi096 ceph-mon[32753]: pgmap v6395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10924.638642 secs 2024-04-26T14:53:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10924.638642 secs 2024-04-26T14:53:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10924.638642 secs 2024-04-26T14:53:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:05 smithi194 ceph-mon[33017]: pgmap v6396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:05 smithi033 ceph-mon[27845]: pgmap v6396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:05 smithi096 ceph-mon[32753]: pgmap v6396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:07 smithi033 ceph-mon[27845]: pgmap v6397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10926.855819 secs 2024-04-26T14:53:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:07 smithi096 ceph-mon[32753]: pgmap v6397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10926.855819 secs 2024-04-26T14:53:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:07 smithi194 ceph-mon[33017]: pgmap v6397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10926.855819 secs 2024-04-26T14:53:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10929.638767 secs 2024-04-26T14:53:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10929.638767 secs 2024-04-26T14:53:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10929.638767 secs 2024-04-26T14:53:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:09 smithi194 ceph-mon[33017]: pgmap v6398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:09 smithi033 ceph-mon[27845]: pgmap v6398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:09 smithi096 ceph-mon[32753]: pgmap v6398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:10.768+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:10.768+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:10.768+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:53:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:11 smithi194 ceph-mon[33017]: pgmap v6399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:11 smithi033 ceph-mon[27845]: pgmap v6399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:11 smithi096 ceph-mon[32753]: pgmap v6399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10931.855920 secs 2024-04-26T14:53:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10931.855920 secs 2024-04-26T14:53:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10931.855920 secs 2024-04-26T14:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:13 smithi033 ceph-mon[27845]: pgmap v6400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10934.638885 secs 2024-04-26T14:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:13 smithi194 ceph-mon[33017]: pgmap v6400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10934.638885 secs 2024-04-26T14:53:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:13 smithi096 ceph-mon[32753]: pgmap v6400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10934.638885 secs 2024-04-26T14:53:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:15 smithi194 ceph-mon[33017]: pgmap v6401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:15 smithi033 ceph-mon[27845]: pgmap v6401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:15 smithi096 ceph-mon[32753]: pgmap v6401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:17 smithi194 ceph-mon[33017]: pgmap v6402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10936.856024 secs 2024-04-26T14:53:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:17 smithi033 ceph-mon[27845]: pgmap v6402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10936.856024 secs 2024-04-26T14:53:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:17 smithi096 ceph-mon[32753]: pgmap v6402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10936.856024 secs 2024-04-26T14:53:19.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10939.638983 secs 2024-04-26T14:53:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10939.638983 secs 2024-04-26T14:53:19.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10939.638983 secs 2024-04-26T14:53:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:19 smithi194 ceph-mon[33017]: pgmap v6403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:19 smithi033 ceph-mon[27845]: pgmap v6403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:19 smithi096 ceph-mon[32753]: pgmap v6403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:21 smithi194 ceph-mon[33017]: pgmap v6404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:53:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:21 smithi033 ceph-mon[27845]: pgmap v6404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:53:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:21 smithi096 ceph-mon[32753]: pgmap v6404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:53:23.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10941.856123 secs 2024-04-26T14:53:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10941.856123 secs 2024-04-26T14:53:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10941.856123 secs 2024-04-26T14:53:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:23.652+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:53:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:23.652+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:53:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:23.652+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:53:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:23.652+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:53:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:23 smithi194 ceph-mon[33017]: pgmap v6405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10944.639074 secs 2024-04-26T14:53:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:23 smithi033 ceph-mon[27845]: pgmap v6405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10944.639074 secs 2024-04-26T14:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:23 smithi096 ceph-mon[32753]: pgmap v6405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10944.639074 secs 2024-04-26T14:53:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:25.775+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:25.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:25.775+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:25.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:25.775+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:53:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:26 smithi194 ceph-mon[33017]: pgmap v6406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:53:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:26 smithi033 ceph-mon[27845]: pgmap v6406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:53:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:26 smithi096 ceph-mon[32753]: pgmap v6406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:53:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:53:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:53:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:53:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:28 smithi194 ceph-mon[33017]: pgmap v6407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10946.856211 secs 2024-04-26T14:53:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:28 smithi033 ceph-mon[27845]: pgmap v6407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10946.856211 secs 2024-04-26T14:53:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:28 smithi096 ceph-mon[32753]: pgmap v6407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10946.856211 secs 2024-04-26T14:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10949.639190 secs 2024-04-26T14:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:29 smithi194 ceph-mon[33017]: pgmap v6408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:29 smithi194 ceph-mon[33017]: pgmap v6409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10949.639190 secs 2024-04-26T14:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:29 smithi033 ceph-mon[27845]: pgmap v6408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:29 smithi033 ceph-mon[27845]: pgmap v6409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10949.639190 secs 2024-04-26T14:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:29 smithi096 ceph-mon[32753]: pgmap v6408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:29 smithi096 ceph-mon[32753]: pgmap v6409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T14:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:53:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:31 smithi033 ceph-mon[27845]: pgmap v6410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:31 smithi096 ceph-mon[32753]: pgmap v6410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:31 smithi194 ceph-mon[33017]: pgmap v6410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:33 smithi033 ceph-mon[27845]: pgmap v6411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10951.856292 secs 2024-04-26T14:53:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:33 smithi096 ceph-mon[32753]: pgmap v6411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10951.856292 secs 2024-04-26T14:53:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:33 smithi194 ceph-mon[33017]: pgmap v6411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10951.856292 secs 2024-04-26T14:53:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10954.639282 secs 2024-04-26T14:53:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10954.639282 secs 2024-04-26T14:53:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10954.639282 secs 2024-04-26T14:53:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:35 smithi033 ceph-mon[27845]: pgmap v6412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:53:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:35 smithi096 ceph-mon[32753]: pgmap v6412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:53:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:35 smithi194 ceph-mon[33017]: pgmap v6412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T14:53:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:37 smithi033 ceph-mon[27845]: pgmap v6413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:37 smithi096 ceph-mon[32753]: pgmap v6413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:37 smithi194 ceph-mon[33017]: pgmap v6413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10956.856395 secs 2024-04-26T14:53:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:38 smithi033 ceph-mon[27845]: pgmap v6414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:53:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10956.856395 secs 2024-04-26T14:53:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:38 smithi096 ceph-mon[32753]: pgmap v6414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:53:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10956.856395 secs 2024-04-26T14:53:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:38 smithi194 ceph-mon[33017]: pgmap v6414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:53:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10959.639393 secs 2024-04-26T14:53:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10959.639393 secs 2024-04-26T14:53:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10959.639393 secs 2024-04-26T14:53:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:40 smithi033 ceph-mon[27845]: pgmap v6415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:40 smithi096 ceph-mon[32753]: pgmap v6415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:40.790+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:40.790+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:40.790+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:53:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:40 smithi194 ceph-mon[33017]: pgmap v6415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:43 smithi033 ceph-mon[27845]: pgmap v6416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10961.856496 secs 2024-04-26T14:53:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:43 smithi096 ceph-mon[32753]: pgmap v6416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10961.856496 secs 2024-04-26T14:53:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:43 smithi194 ceph-mon[33017]: pgmap v6416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10961.856496 secs 2024-04-26T14:53:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10964.639486 secs 2024-04-26T14:53:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10964.639486 secs 2024-04-26T14:53:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10964.639486 secs 2024-04-26T14:53:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:45 smithi033 ceph-mon[27845]: pgmap v6417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:45 smithi096 ceph-mon[32753]: pgmap v6417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:45 smithi194 ceph-mon[33017]: pgmap v6417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:47 smithi033 ceph-mon[27845]: pgmap v6418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:47 smithi096 ceph-mon[32753]: pgmap v6418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:47 smithi194 ceph-mon[33017]: pgmap v6418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10966.856598 secs 2024-04-26T14:53:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10966.856598 secs 2024-04-26T14:53:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10966.856598 secs 2024-04-26T14:53:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10969.639610 secs 2024-04-26T14:53:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:49 smithi033 ceph-mon[27845]: pgmap v6419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10969.639610 secs 2024-04-26T14:53:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:49 smithi096 ceph-mon[32753]: pgmap v6419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10969.639610 secs 2024-04-26T14:53:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:49 smithi194 ceph-mon[33017]: pgmap v6419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:50 smithi033 ceph-mon[27845]: pgmap v6420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:50 smithi096 ceph-mon[32753]: pgmap v6420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:50 smithi194 ceph-mon[33017]: pgmap v6420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:53:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:53 smithi033 ceph-mon[27845]: pgmap v6421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10971.856702 secs 2024-04-26T14:53:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:53 smithi096 ceph-mon[32753]: pgmap v6421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10971.856702 secs 2024-04-26T14:53:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:53 smithi194 ceph-mon[33017]: pgmap v6421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10971.856702 secs 2024-04-26T14:53:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10974.639726 secs 2024-04-26T14:53:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10974.639726 secs 2024-04-26T14:53:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10974.639726 secs 2024-04-26T14:53:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:55 smithi033 ceph-mon[27845]: pgmap v6422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:55 smithi096 ceph-mon[32753]: pgmap v6422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:55.798+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:55.798+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:53:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:53:55.798+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:53:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:55 smithi194 ceph-mon[33017]: pgmap v6422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:56 smithi194 ceph-mon[33017]: pgmap v6423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:56 smithi033 ceph-mon[27845]: pgmap v6423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:56 smithi096 ceph-mon[32753]: pgmap v6423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:53:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10976.856801 secs 2024-04-26T14:53:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10976.856801 secs 2024-04-26T14:53:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10976.856801 secs 2024-04-26T14:53:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:58 smithi194 ceph-mon[33017]: pgmap v6424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:58 smithi033 ceph-mon[27845]: pgmap v6424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:53:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:58 smithi096 ceph-mon[32753]: pgmap v6424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10979.639845 secs 2024-04-26T14:54:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10979.639845 secs 2024-04-26T14:54:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10979.639845 secs 2024-04-26T14:54:01.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:00 smithi033 ceph-mon[27845]: pgmap v6425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:00 smithi194 ceph-mon[33017]: pgmap v6425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:00 smithi096 ceph-mon[32753]: pgmap v6425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:03 smithi033 ceph-mon[27845]: pgmap v6426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10981.856903 secs 2024-04-26T14:54:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:03 smithi096 ceph-mon[32753]: pgmap v6426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10981.856903 secs 2024-04-26T14:54:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:03 smithi194 ceph-mon[33017]: pgmap v6426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10981.856903 secs 2024-04-26T14:54:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10984.639907 secs 2024-04-26T14:54:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10984.639907 secs 2024-04-26T14:54:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10984.639907 secs 2024-04-26T14:54:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:05 smithi033 ceph-mon[27845]: pgmap v6427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:05 smithi096 ceph-mon[32753]: pgmap v6427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:05 smithi194 ceph-mon[33017]: pgmap v6427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:06 smithi194 ceph-mon[33017]: pgmap v6428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:07.100 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:06 smithi096 ceph-mon[32753]: pgmap v6428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:07.100 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:06 smithi033 ceph-mon[27845]: pgmap v6428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10986.857007 secs 2024-04-26T14:54:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10986.857007 secs 2024-04-26T14:54:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10986.857007 secs 2024-04-26T14:54:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:08 smithi194 ceph-mon[33017]: pgmap v6429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:08 smithi033 ceph-mon[27845]: pgmap v6429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:08 smithi096 ceph-mon[32753]: pgmap v6429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10989.640042 secs 2024-04-26T14:54:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10989.640042 secs 2024-04-26T14:54:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10989.640042 secs 2024-04-26T14:54:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:10.814+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:10.814+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:10.814+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:54:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:10 smithi194 ceph-mon[33017]: pgmap v6430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:10 smithi033 ceph-mon[27845]: pgmap v6430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:10 smithi096 ceph-mon[32753]: pgmap v6430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:13 smithi033 ceph-mon[27845]: pgmap v6431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10991.857105 secs 2024-04-26T14:54:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:13 smithi096 ceph-mon[32753]: pgmap v6431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10991.857105 secs 2024-04-26T14:54:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:13 smithi194 ceph-mon[33017]: pgmap v6431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10991.857105 secs 2024-04-26T14:54:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10994.640162 secs 2024-04-26T14:54:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10994.640162 secs 2024-04-26T14:54:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10994.640162 secs 2024-04-26T14:54:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:15 smithi033 ceph-mon[27845]: pgmap v6432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:15 smithi096 ceph-mon[32753]: pgmap v6432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:15 smithi194 ceph-mon[33017]: pgmap v6432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:16 smithi033 ceph-mon[27845]: pgmap v6433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:16 smithi096 ceph-mon[32753]: pgmap v6433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:16 smithi194 ceph-mon[33017]: pgmap v6433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 10996.857187 secs 2024-04-26T14:54:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 10996.857187 secs 2024-04-26T14:54:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 10996.857187 secs 2024-04-26T14:54:19.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:18 smithi194 ceph-mon[33017]: pgmap v6434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:18 smithi033 ceph-mon[27845]: pgmap v6434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:18 smithi096 ceph-mon[32753]: pgmap v6434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 10999.640300 secs 2024-04-26T14:54:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 10999.640300 secs 2024-04-26T14:54:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 10999.640300 secs 2024-04-26T14:54:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:20 smithi194 ceph-mon[33017]: pgmap v6435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:20 smithi033 ceph-mon[27845]: pgmap v6435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:20 smithi096 ceph-mon[32753]: pgmap v6435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:23 smithi033 ceph-mon[27845]: pgmap v6436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11001.857287 secs 2024-04-26T14:54:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:23 smithi096 ceph-mon[32753]: pgmap v6436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11001.857287 secs 2024-04-26T14:54:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:23 smithi194 ceph-mon[33017]: pgmap v6436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11001.857287 secs 2024-04-26T14:54:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11004.640408 secs 2024-04-26T14:54:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11004.640408 secs 2024-04-26T14:54:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11004.640408 secs 2024-04-26T14:54:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:25 smithi033 ceph-mon[27845]: pgmap v6437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:25 smithi096 ceph-mon[32753]: pgmap v6437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:25.828+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:25.828+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:25.828+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:54:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:25 smithi194 ceph-mon[33017]: pgmap v6437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:26 smithi033 ceph-mon[27845]: pgmap v6438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:26 smithi096 ceph-mon[32753]: pgmap v6438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:26 smithi194 ceph-mon[33017]: pgmap v6438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11006.857388 secs 2024-04-26T14:54:28.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11006.857388 secs 2024-04-26T14:54:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11006.857388 secs 2024-04-26T14:54:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:28 smithi194 ceph-mon[33017]: pgmap v6439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:54:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:28 smithi033 ceph-mon[27845]: pgmap v6439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:54:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:28 smithi096 ceph-mon[32753]: pgmap v6439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:54:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11009.640527 secs 2024-04-26T14:54:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11009.640527 secs 2024-04-26T14:54:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11009.640527 secs 2024-04-26T14:54:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:54:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:54:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:30 smithi194 ceph-mon[33017]: pgmap v6440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:54:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:30 smithi194 ceph-mon[33017]: pgmap v6441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:54:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:54:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:54:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:30 smithi033 ceph-mon[27845]: pgmap v6440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:54:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:30 smithi033 ceph-mon[27845]: pgmap v6441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:54:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:54:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:54:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:30 smithi096 ceph-mon[32753]: pgmap v6440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:54:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:30 smithi096 ceph-mon[32753]: pgmap v6441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:54:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:33 smithi194 ceph-mon[33017]: pgmap v6442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11011.857489 secs 2024-04-26T14:54:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:33 smithi033 ceph-mon[27845]: pgmap v6442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11011.857489 secs 2024-04-26T14:54:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:33 smithi096 ceph-mon[32753]: pgmap v6442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11011.857489 secs 2024-04-26T14:54:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11014.640643 secs 2024-04-26T14:54:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11014.640643 secs 2024-04-26T14:54:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11014.640643 secs 2024-04-26T14:54:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:35 smithi194 ceph-mon[33017]: pgmap v6443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:54:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:35 smithi033 ceph-mon[27845]: pgmap v6443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:54:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:35 smithi096 ceph-mon[32753]: pgmap v6443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:54:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:37 smithi194 ceph-mon[33017]: pgmap v6444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:37 smithi033 ceph-mon[27845]: pgmap v6444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:37 smithi096 ceph-mon[32753]: pgmap v6444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11016.857591 secs 2024-04-26T14:54:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11016.857591 secs 2024-04-26T14:54:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11016.857591 secs 2024-04-26T14:54:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:39 smithi194 ceph-mon[33017]: pgmap v6445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11019.640778 secs 2024-04-26T14:54:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:39 smithi033 ceph-mon[27845]: pgmap v6445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11019.640778 secs 2024-04-26T14:54:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:39 smithi096 ceph-mon[32753]: pgmap v6445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:54:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11019.640778 secs 2024-04-26T14:54:40.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:40.840+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:40.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:40.840+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:40.841 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:40.840+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:54:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:41 smithi194 ceph-mon[33017]: pgmap v6446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:54:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:41 smithi033 ceph-mon[27845]: pgmap v6446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:54:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:41 smithi096 ceph-mon[32753]: pgmap v6446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:54:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:43 smithi194 ceph-mon[33017]: pgmap v6447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11021.857695 secs 2024-04-26T14:54:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:43 smithi033 ceph-mon[27845]: pgmap v6447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11021.857695 secs 2024-04-26T14:54:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:43 smithi096 ceph-mon[32753]: pgmap v6447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11021.857695 secs 2024-04-26T14:54:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11024.640894 secs 2024-04-26T14:54:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11024.640894 secs 2024-04-26T14:54:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11024.640894 secs 2024-04-26T14:54:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:45 smithi033 ceph-mon[27845]: pgmap v6448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:45 smithi096 ceph-mon[32753]: pgmap v6448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:45 smithi194 ceph-mon[33017]: pgmap v6448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:47 smithi033 ceph-mon[27845]: pgmap v6449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:47 smithi096 ceph-mon[32753]: pgmap v6449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:47 smithi194 ceph-mon[33017]: pgmap v6449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11026.857798 secs 2024-04-26T14:54:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11026.857798 secs 2024-04-26T14:54:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11026.857798 secs 2024-04-26T14:54:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:49 smithi033 ceph-mon[27845]: pgmap v6450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11029.641029 secs 2024-04-26T14:54:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:49 smithi096 ceph-mon[32753]: pgmap v6450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11029.641029 secs 2024-04-26T14:54:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:49 smithi194 ceph-mon[33017]: pgmap v6450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11029.641029 secs 2024-04-26T14:54:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:51 smithi033 ceph-mon[27845]: pgmap v6451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:51 smithi096 ceph-mon[32753]: pgmap v6451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:51 smithi194 ceph-mon[33017]: pgmap v6451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:53 smithi033 ceph-mon[27845]: pgmap v6452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11031.857895 secs 2024-04-26T14:54:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:53 smithi096 ceph-mon[32753]: pgmap v6452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11031.857895 secs 2024-04-26T14:54:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:53 smithi194 ceph-mon[33017]: pgmap v6452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11031.857895 secs 2024-04-26T14:54:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11034.641141 secs 2024-04-26T14:54:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11034.641141 secs 2024-04-26T14:54:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11034.641141 secs 2024-04-26T14:54:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:55 smithi033 ceph-mon[27845]: pgmap v6453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:55 smithi096 ceph-mon[32753]: pgmap v6453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:55 smithi194 ceph-mon[33017]: pgmap v6453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:55.856+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:55.856+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:54:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:54:55.856+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:54:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:57 smithi033 ceph-mon[27845]: pgmap v6454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:57 smithi096 ceph-mon[32753]: pgmap v6454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:57 smithi194 ceph-mon[33017]: pgmap v6454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:54:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11036.857998 secs 2024-04-26T14:54:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11036.857998 secs 2024-04-26T14:54:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11036.857998 secs 2024-04-26T14:54:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:59 smithi096 ceph-mon[32753]: pgmap v6455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11039.641206 secs 2024-04-26T14:54:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:59 smithi194 ceph-mon[33017]: pgmap v6455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11039.641206 secs 2024-04-26T14:54:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:59 smithi033 ceph-mon[27845]: pgmap v6455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:54:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11039.641206 secs 2024-04-26T14:55:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:01 smithi194 ceph-mon[33017]: pgmap v6456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:01 smithi033 ceph-mon[27845]: pgmap v6456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:01 smithi096 ceph-mon[32753]: pgmap v6456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:03 smithi033 ceph-mon[27845]: pgmap v6457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11041.858099 secs 2024-04-26T14:55:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:03 smithi194 ceph-mon[33017]: pgmap v6457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11041.858099 secs 2024-04-26T14:55:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:03 smithi096 ceph-mon[32753]: pgmap v6457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11041.858099 secs 2024-04-26T14:55:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11044.641317 secs 2024-04-26T14:55:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11044.641317 secs 2024-04-26T14:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11044.641317 secs 2024-04-26T14:55:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:05 smithi194 ceph-mon[33017]: pgmap v6458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:05 smithi033 ceph-mon[27845]: pgmap v6458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:05 smithi096 ceph-mon[32753]: pgmap v6458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:07 smithi194 ceph-mon[33017]: pgmap v6459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:07 smithi033 ceph-mon[27845]: pgmap v6459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:07 smithi096 ceph-mon[32753]: pgmap v6459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11046.858203 secs 2024-04-26T14:55:08.993 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11046.858203 secs 2024-04-26T14:55:08.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11046.858203 secs 2024-04-26T14:55:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:09 smithi194 ceph-mon[33017]: pgmap v6460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11049.641399 secs 2024-04-26T14:55:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:09 smithi033 ceph-mon[27845]: pgmap v6460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11049.641399 secs 2024-04-26T14:55:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:09 smithi096 ceph-mon[32753]: pgmap v6460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11049.641399 secs 2024-04-26T14:55:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:10.867+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:10.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:10.867+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:10.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:10.867+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:55:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:11 smithi194 ceph-mon[33017]: pgmap v6461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:11 smithi033 ceph-mon[27845]: pgmap v6461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:11 smithi096 ceph-mon[32753]: pgmap v6461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:13 smithi194 ceph-mon[33017]: pgmap v6462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11051.858260 secs 2024-04-26T14:55:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:13 smithi033 ceph-mon[27845]: pgmap v6462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11051.858260 secs 2024-04-26T14:55:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:13 smithi096 ceph-mon[32753]: pgmap v6462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11051.858260 secs 2024-04-26T14:55:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11054.641498 secs 2024-04-26T14:55:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11054.641498 secs 2024-04-26T14:55:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11054.641498 secs 2024-04-26T14:55:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:15 smithi194 ceph-mon[33017]: pgmap v6463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:15 smithi033 ceph-mon[27845]: pgmap v6463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:15 smithi096 ceph-mon[32753]: pgmap v6463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:17 smithi194 ceph-mon[33017]: pgmap v6464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:17 smithi033 ceph-mon[27845]: pgmap v6464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:17 smithi096 ceph-mon[32753]: pgmap v6464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:18.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11056.858341 secs 2024-04-26T14:55:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11056.858341 secs 2024-04-26T14:55:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11056.858341 secs 2024-04-26T14:55:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:19 smithi194 ceph-mon[33017]: pgmap v6465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11059.641562 secs 2024-04-26T14:55:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:19 smithi033 ceph-mon[27845]: pgmap v6465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11059.641562 secs 2024-04-26T14:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:19 smithi096 ceph-mon[32753]: pgmap v6465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11059.641562 secs 2024-04-26T14:55:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:21 smithi194 ceph-mon[33017]: pgmap v6466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:21 smithi033 ceph-mon[27845]: pgmap v6466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:21 smithi096 ceph-mon[32753]: pgmap v6466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:23 smithi194 ceph-mon[33017]: pgmap v6467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11061.858426 secs 2024-04-26T14:55:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:23 smithi033 ceph-mon[27845]: pgmap v6467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11061.858426 secs 2024-04-26T14:55:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:23 smithi096 ceph-mon[32753]: pgmap v6467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11061.858426 secs 2024-04-26T14:55:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11064.641708 secs 2024-04-26T14:55:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11064.641708 secs 2024-04-26T14:55:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11064.641708 secs 2024-04-26T14:55:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:25 smithi194 ceph-mon[33017]: pgmap v6468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:25 smithi033 ceph-mon[27845]: pgmap v6468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:25 smithi096 ceph-mon[32753]: pgmap v6468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T14:55:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:25.879+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:25.879+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:25.879+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:55:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:27 smithi194 ceph-mon[33017]: pgmap v6469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:27 smithi033 ceph-mon[27845]: pgmap v6469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:27 smithi096 ceph-mon[32753]: pgmap v6469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:55:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11066.858512 secs 2024-04-26T14:55:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11066.858512 secs 2024-04-26T14:55:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11066.858512 secs 2024-04-26T14:55:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:29 smithi194 ceph-mon[33017]: pgmap v6470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11069.641823 secs 2024-04-26T14:55:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:29 smithi033 ceph-mon[27845]: pgmap v6470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11069.641823 secs 2024-04-26T14:55:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:29 smithi096 ceph-mon[32753]: pgmap v6470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11069.641823 secs 2024-04-26T14:55:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:31 smithi194 ceph-mon[33017]: pgmap v6471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:55:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:31 smithi033 ceph-mon[27845]: pgmap v6471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:55:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:31 smithi096 ceph-mon[32753]: pgmap v6471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:55:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:55:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:55:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:55:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:55:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:55:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:55:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:55:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:55:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:55:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:33 smithi194 ceph-mon[33017]: pgmap v6472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:33 smithi194 ceph-mon[33017]: pgmap v6473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11071.858590 secs 2024-04-26T14:55:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:33 smithi033 ceph-mon[27845]: pgmap v6472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:33 smithi033 ceph-mon[27845]: pgmap v6473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11071.858590 secs 2024-04-26T14:55:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:33 smithi096 ceph-mon[32753]: pgmap v6472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:33 smithi096 ceph-mon[32753]: pgmap v6473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11071.858590 secs 2024-04-26T14:55:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11074.641944 secs 2024-04-26T14:55:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11074.641944 secs 2024-04-26T14:55:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11074.641944 secs 2024-04-26T14:55:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:35 smithi194 ceph-mon[33017]: pgmap v6474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:35 smithi033 ceph-mon[27845]: pgmap v6474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:35 smithi096 ceph-mon[32753]: pgmap v6474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:37 smithi194 ceph-mon[33017]: pgmap v6475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:37 smithi033 ceph-mon[27845]: pgmap v6475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:37 smithi096 ceph-mon[32753]: pgmap v6475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11076.858673 secs 2024-04-26T14:55:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11076.858673 secs 2024-04-26T14:55:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11076.858673 secs 2024-04-26T14:55:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:39 smithi033 ceph-mon[27845]: pgmap v6476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11079.642064 secs 2024-04-26T14:55:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:39 smithi096 ceph-mon[32753]: pgmap v6476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11079.642064 secs 2024-04-26T14:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:39 smithi194 ceph-mon[33017]: pgmap v6476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11079.642064 secs 2024-04-26T14:55:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:40.884+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:40.884+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:40.884+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:55:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:41 smithi033 ceph-mon[27845]: pgmap v6477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:41 smithi096 ceph-mon[32753]: pgmap v6477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:41 smithi194 ceph-mon[33017]: pgmap v6477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:55:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:43 smithi033 ceph-mon[27845]: pgmap v6478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:55:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11081.858755 secs 2024-04-26T14:55:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:43 smithi096 ceph-mon[32753]: pgmap v6478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:55:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11081.858755 secs 2024-04-26T14:55:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:43 smithi194 ceph-mon[33017]: pgmap v6478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:55:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11081.858755 secs 2024-04-26T14:55:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11084.642189 secs 2024-04-26T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11084.642189 secs 2024-04-26T14:55:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11084.642189 secs 2024-04-26T14:55:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:45 smithi033 ceph-mon[27845]: pgmap v6479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:45 smithi096 ceph-mon[32753]: pgmap v6479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:45 smithi194 ceph-mon[33017]: pgmap v6479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:47 smithi033 ceph-mon[27845]: pgmap v6480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:47 smithi096 ceph-mon[32753]: pgmap v6480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:47 smithi194 ceph-mon[33017]: pgmap v6480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11086.858838 secs 2024-04-26T14:55:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11086.858838 secs 2024-04-26T14:55:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11086.858838 secs 2024-04-26T14:55:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:49 smithi033 ceph-mon[27845]: pgmap v6481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11089.642257 secs 2024-04-26T14:55:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:49 smithi096 ceph-mon[32753]: pgmap v6481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11089.642257 secs 2024-04-26T14:55:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:49 smithi194 ceph-mon[33017]: pgmap v6481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11089.642257 secs 2024-04-26T14:55:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:51 smithi033 ceph-mon[27845]: pgmap v6482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:51 smithi096 ceph-mon[32753]: pgmap v6482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:51 smithi194 ceph-mon[33017]: pgmap v6482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:53 smithi033 ceph-mon[27845]: pgmap v6483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11091.858923 secs 2024-04-26T14:55:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:53 smithi096 ceph-mon[32753]: pgmap v6483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11091.858923 secs 2024-04-26T14:55:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:53 smithi194 ceph-mon[33017]: pgmap v6483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11091.858923 secs 2024-04-26T14:55:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11094.642372 secs 2024-04-26T14:55:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11094.642372 secs 2024-04-26T14:55:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11094.642372 secs 2024-04-26T14:55:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:55 smithi096 ceph-mon[32753]: pgmap v6484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:55 smithi194 ceph-mon[33017]: pgmap v6484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:55.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:55.900+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:55.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:55.900+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:55:55.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:55:55.900+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:55:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:55 smithi033 ceph-mon[27845]: pgmap v6484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:56 smithi194 ceph-mon[33017]: pgmap v6485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:56 smithi033 ceph-mon[27845]: pgmap v6485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:56 smithi096 ceph-mon[32753]: pgmap v6485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:55:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11096.859026 secs 2024-04-26T14:55:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11096.859026 secs 2024-04-26T14:55:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11096.859026 secs 2024-04-26T14:55:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:58 smithi194 ceph-mon[33017]: pgmap v6486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:58 smithi033 ceph-mon[27845]: pgmap v6486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:55:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:58 smithi096 ceph-mon[32753]: pgmap v6486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:55:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11099.642490 secs 2024-04-26T14:56:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:55:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11099.642490 secs 2024-04-26T14:56:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:55:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11099.642490 secs 2024-04-26T14:56:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:00 smithi194 ceph-mon[33017]: pgmap v6487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:00 smithi033 ceph-mon[27845]: pgmap v6487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:00 smithi096 ceph-mon[32753]: pgmap v6487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:02 smithi194 ceph-mon[33017]: pgmap v6488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11101.859128 secs 2024-04-26T14:56:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:02 smithi033 ceph-mon[27845]: pgmap v6488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11101.859128 secs 2024-04-26T14:56:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:02 smithi096 ceph-mon[32753]: pgmap v6488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11101.859128 secs 2024-04-26T14:56:04.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11104.642614 secs 2024-04-26T14:56:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11104.642614 secs 2024-04-26T14:56:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11104.642614 secs 2024-04-26T14:56:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:04 smithi194 ceph-mon[33017]: pgmap v6489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:04 smithi033 ceph-mon[27845]: pgmap v6489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:04 smithi096 ceph-mon[32753]: pgmap v6489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:06 smithi194 ceph-mon[33017]: pgmap v6490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:06 smithi033 ceph-mon[27845]: pgmap v6490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:06 smithi096 ceph-mon[32753]: pgmap v6490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11106.859188 secs 2024-04-26T14:56:08.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11106.859188 secs 2024-04-26T14:56:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11106.859188 secs 2024-04-26T14:56:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:08 smithi194 ceph-mon[33017]: pgmap v6491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11109.642753 secs 2024-04-26T14:56:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:08 smithi033 ceph-mon[27845]: pgmap v6491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11109.642753 secs 2024-04-26T14:56:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:08 smithi096 ceph-mon[32753]: pgmap v6491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11109.642753 secs 2024-04-26T14:56:10.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:10.913+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:10.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:10.913+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:10.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:10.913+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:56:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:10 smithi194 ceph-mon[33017]: pgmap v6492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:10 smithi033 ceph-mon[27845]: pgmap v6492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:10 smithi096 ceph-mon[32753]: pgmap v6492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:12 smithi194 ceph-mon[33017]: pgmap v6493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11111.859288 secs 2024-04-26T14:56:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:12 smithi033 ceph-mon[27845]: pgmap v6493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11111.859288 secs 2024-04-26T14:56:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:12 smithi096 ceph-mon[32753]: pgmap v6493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11111.859288 secs 2024-04-26T14:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11114.642866 secs 2024-04-26T14:56:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11114.642866 secs 2024-04-26T14:56:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11114.642866 secs 2024-04-26T14:56:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:14 smithi194 ceph-mon[33017]: pgmap v6494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:14 smithi033 ceph-mon[27845]: pgmap v6494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:14 smithi096 ceph-mon[32753]: pgmap v6494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:16 smithi194 ceph-mon[33017]: pgmap v6495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:16 smithi033 ceph-mon[27845]: pgmap v6495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:16 smithi096 ceph-mon[32753]: pgmap v6495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11116.859376 secs 2024-04-26T14:56:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11116.859376 secs 2024-04-26T14:56:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11116.859376 secs 2024-04-26T14:56:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:18 smithi194 ceph-mon[33017]: pgmap v6496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11119.643003 secs 2024-04-26T14:56:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:18 smithi033 ceph-mon[27845]: pgmap v6496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11119.643003 secs 2024-04-26T14:56:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:18 smithi096 ceph-mon[32753]: pgmap v6496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11119.643003 secs 2024-04-26T14:56:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:20 smithi194 ceph-mon[33017]: pgmap v6497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:20 smithi033 ceph-mon[27845]: pgmap v6497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:20 smithi096 ceph-mon[32753]: pgmap v6497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3cc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef004f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3cc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef004f040 comp rx=0 tx=0).stop 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3cc0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 msgr2=0x7feee40b3cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b33a0 0x7feee40b3cc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T14:56:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.740+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T14:56:21.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.741+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T14:56:21.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:21.743+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0062040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T14:56:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:22 smithi194 ceph-mon[33017]: pgmap v6498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11121.859436 secs 2024-04-26T14:56:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:22 smithi033 ceph-mon[27845]: pgmap v6498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11121.859436 secs 2024-04-26T14:56:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:22 smithi096 ceph-mon[32753]: pgmap v6498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11121.859436 secs 2024-04-26T14:56:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:23.683+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:56:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:23.683+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:56:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:23.683+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:56:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:23.684+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:56:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11124.643136 secs 2024-04-26T14:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11124.643136 secs 2024-04-26T14:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11124.643136 secs 2024-04-26T14:56:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:24 smithi033 ceph-mon[27845]: pgmap v6499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:24 smithi096 ceph-mon[32753]: pgmap v6499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:24 smithi194 ceph-mon[33017]: pgmap v6499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:25.914+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:25.915+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:56:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:25.915+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:26 smithi033 ceph-mon[27845]: pgmap v6500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:26 smithi096 ceph-mon[32753]: pgmap v6500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:26 smithi194 ceph-mon[33017]: pgmap v6500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11126.859522 secs 2024-04-26T14:56:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11126.859522 secs 2024-04-26T14:56:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11126.859522 secs 2024-04-26T14:56:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:28 smithi033 ceph-mon[27845]: pgmap v6501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11129.643252 secs 2024-04-26T14:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:28 smithi096 ceph-mon[32753]: pgmap v6501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11129.643252 secs 2024-04-26T14:56:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:28 smithi194 ceph-mon[33017]: pgmap v6501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11129.643252 secs 2024-04-26T14:56:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:30 smithi033 ceph-mon[27845]: pgmap v6502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:30 smithi096 ceph-mon[32753]: pgmap v6502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:30 smithi194 ceph-mon[33017]: pgmap v6502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:32 smithi033 ceph-mon[27845]: pgmap v6503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:56:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11131.859614 secs 2024-04-26T14:56:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:32 smithi096 ceph-mon[32753]: pgmap v6503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:56:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11131.859614 secs 2024-04-26T14:56:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:32 smithi194 ceph-mon[33017]: pgmap v6503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:56:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11131.859614 secs 2024-04-26T14:56:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11134.643319 secs 2024-04-26T14:56:34.402 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:34.402 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:34.402 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11134.643319 secs 2024-04-26T14:56:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11134.643319 secs 2024-04-26T14:56:35.146 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:35 smithi194 ceph-mon[33017]: pgmap v6504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:35.396 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:35 smithi033 ceph-mon[27845]: pgmap v6504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:35 smithi096 ceph-mon[32753]: pgmap v6504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:37.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:37 smithi033 ceph-mon[27845]: pgmap v6505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:37.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:37 smithi194 ceph-mon[33017]: pgmap v6505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:37 smithi096 ceph-mon[32753]: pgmap v6505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:37.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:37.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11136.859696 secs 2024-04-26T14:56:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11136.859696 secs 2024-04-26T14:56:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11136.859696 secs 2024-04-26T14:56:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:39 smithi194 ceph-mon[33017]: pgmap v6506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11139.643439 secs 2024-04-26T14:56:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:39 smithi033 ceph-mon[27845]: pgmap v6506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11139.643439 secs 2024-04-26T14:56:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:39 smithi096 ceph-mon[32753]: pgmap v6506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11139.643439 secs 2024-04-26T14:56:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:40.931+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:40.931+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:40.931+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: pgmap v6507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: pgmap v6508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: pgmap v6509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: pgmap v6507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:56:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:56:41.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: pgmap v6508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:56:41.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: pgmap v6509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:41.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: pgmap v6507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: pgmap v6508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: pgmap v6509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:56:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:43 smithi194 ceph-mon[33017]: pgmap v6510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11141.859780 secs 2024-04-26T14:56:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:43 smithi033 ceph-mon[27845]: pgmap v6510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11141.859780 secs 2024-04-26T14:56:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:43 smithi096 ceph-mon[32753]: pgmap v6510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11141.859780 secs 2024-04-26T14:56:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11144.643553 secs 2024-04-26T14:56:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11144.643553 secs 2024-04-26T14:56:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11144.643553 secs 2024-04-26T14:56:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:45 smithi194 ceph-mon[33017]: pgmap v6511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:56:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:45 smithi033 ceph-mon[27845]: pgmap v6511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:56:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:45 smithi096 ceph-mon[32753]: pgmap v6511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:56:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:47 smithi194 ceph-mon[33017]: pgmap v6512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:47 smithi033 ceph-mon[27845]: pgmap v6512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:47 smithi096 ceph-mon[32753]: pgmap v6512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:56:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11146.859862 secs 2024-04-26T14:56:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11146.859862 secs 2024-04-26T14:56:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11146.859862 secs 2024-04-26T14:56:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:49 smithi194 ceph-mon[33017]: pgmap v6513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:56:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11149.643678 secs 2024-04-26T14:56:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:49 smithi033 ceph-mon[27845]: pgmap v6513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:56:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11149.643678 secs 2024-04-26T14:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:49 smithi096 ceph-mon[32753]: pgmap v6513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11149.643678 secs 2024-04-26T14:56:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:51 smithi033 ceph-mon[27845]: pgmap v6514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:56:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:51 smithi096 ceph-mon[32753]: pgmap v6514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:56:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:51 smithi194 ceph-mon[33017]: pgmap v6514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:56:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:53 smithi033 ceph-mon[27845]: pgmap v6515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11151.859946 secs 2024-04-26T14:56:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:53 smithi096 ceph-mon[32753]: pgmap v6515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11151.859946 secs 2024-04-26T14:56:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:53 smithi194 ceph-mon[33017]: pgmap v6515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11151.859946 secs 2024-04-26T14:56:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11154.643823 secs 2024-04-26T14:56:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11154.643823 secs 2024-04-26T14:56:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11154.643823 secs 2024-04-26T14:56:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:55 smithi033 ceph-mon[27845]: pgmap v6516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:55 smithi096 ceph-mon[32753]: pgmap v6516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:55 smithi194 ceph-mon[33017]: pgmap v6516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:55.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:55.940+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:55.940+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:56:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:56:55.940+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:56:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:57 smithi033 ceph-mon[27845]: pgmap v6517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:57 smithi096 ceph-mon[32753]: pgmap v6517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:57 smithi194 ceph-mon[33017]: pgmap v6517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:56:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11156.860006 secs 2024-04-26T14:56:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11156.860006 secs 2024-04-26T14:56:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11156.860006 secs 2024-04-26T14:56:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:59 smithi033 ceph-mon[27845]: pgmap v6518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:56:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11159.643944 secs 2024-04-26T14:56:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:59 smithi096 ceph-mon[32753]: pgmap v6518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:56:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11159.643944 secs 2024-04-26T14:56:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:59 smithi194 ceph-mon[33017]: pgmap v6518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:56:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:56:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11159.643944 secs 2024-04-26T14:57:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:01 smithi033 ceph-mon[27845]: pgmap v6519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:01 smithi096 ceph-mon[32753]: pgmap v6519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:01 smithi194 ceph-mon[33017]: pgmap v6519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:03 smithi033 ceph-mon[27845]: pgmap v6520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11161.860086 secs 2024-04-26T14:57:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:03 smithi096 ceph-mon[32753]: pgmap v6520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11161.860086 secs 2024-04-26T14:57:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:03 smithi194 ceph-mon[33017]: pgmap v6520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11161.860086 secs 2024-04-26T14:57:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11164.644060 secs 2024-04-26T14:57:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11164.644060 secs 2024-04-26T14:57:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11164.644060 secs 2024-04-26T14:57:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:05 smithi033 ceph-mon[27845]: pgmap v6521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:05 smithi096 ceph-mon[32753]: pgmap v6521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:05 smithi194 ceph-mon[33017]: pgmap v6521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:07 smithi096 ceph-mon[32753]: pgmap v6522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:07 smithi194 ceph-mon[33017]: pgmap v6522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:07 smithi033 ceph-mon[27845]: pgmap v6522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11166.860171 secs 2024-04-26T14:57:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11166.860171 secs 2024-04-26T14:57:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11166.860171 secs 2024-04-26T14:57:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:09 smithi194 ceph-mon[33017]: pgmap v6523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11169.644177 secs 2024-04-26T14:57:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:09 smithi033 ceph-mon[27845]: pgmap v6523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11169.644177 secs 2024-04-26T14:57:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:09 smithi096 ceph-mon[32753]: pgmap v6523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11169.644177 secs 2024-04-26T14:57:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:10.952+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:10.952+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:10.952+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:57:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:11 smithi194 ceph-mon[33017]: pgmap v6524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:11 smithi033 ceph-mon[27845]: pgmap v6524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:11 smithi096 ceph-mon[32753]: pgmap v6524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:13 smithi194 ceph-mon[33017]: pgmap v6525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11171.860261 secs 2024-04-26T14:57:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:13 smithi033 ceph-mon[27845]: pgmap v6525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11171.860261 secs 2024-04-26T14:57:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:13 smithi096 ceph-mon[32753]: pgmap v6525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11171.860261 secs 2024-04-26T14:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11174.644317 secs 2024-04-26T14:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11174.644317 secs 2024-04-26T14:57:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11174.644317 secs 2024-04-26T14:57:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:15 smithi194 ceph-mon[33017]: pgmap v6526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:15 smithi033 ceph-mon[27845]: pgmap v6526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:15 smithi096 ceph-mon[32753]: pgmap v6526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:17 smithi194 ceph-mon[33017]: pgmap v6527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:17 smithi033 ceph-mon[27845]: pgmap v6527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:17 smithi096 ceph-mon[32753]: pgmap v6527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11176.860361 secs 2024-04-26T14:57:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11176.860361 secs 2024-04-26T14:57:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11176.860361 secs 2024-04-26T14:57:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:19 smithi194 ceph-mon[33017]: pgmap v6528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11179.644434 secs 2024-04-26T14:57:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:19 smithi033 ceph-mon[27845]: pgmap v6528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11179.644434 secs 2024-04-26T14:57:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:19 smithi096 ceph-mon[32753]: pgmap v6528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11179.644434 secs 2024-04-26T14:57:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:21 smithi194 ceph-mon[33017]: pgmap v6529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:21 smithi033 ceph-mon[27845]: pgmap v6529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:21 smithi096 ceph-mon[32753]: pgmap v6529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:23 smithi194 ceph-mon[33017]: pgmap v6530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11181.860441 secs 2024-04-26T14:57:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:23 smithi033 ceph-mon[27845]: pgmap v6530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11181.860441 secs 2024-04-26T14:57:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:23 smithi096 ceph-mon[32753]: pgmap v6530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11181.860441 secs 2024-04-26T14:57:24.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11184.644546 secs 2024-04-26T14:57:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11184.644546 secs 2024-04-26T14:57:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11184.644546 secs 2024-04-26T14:57:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:25 smithi194 ceph-mon[33017]: pgmap v6531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:25 smithi033 ceph-mon[27845]: pgmap v6531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:25 smithi096 ceph-mon[32753]: pgmap v6531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:25.957+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:25.957+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:25.957+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:57:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:27 smithi194 ceph-mon[33017]: pgmap v6532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:27 smithi033 ceph-mon[27845]: pgmap v6532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:27 smithi096 ceph-mon[32753]: pgmap v6532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11186.860531 secs 2024-04-26T14:57:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11186.860531 secs 2024-04-26T14:57:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11186.860531 secs 2024-04-26T14:57:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:29 smithi194 ceph-mon[33017]: pgmap v6533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11189.644698 secs 2024-04-26T14:57:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:29 smithi033 ceph-mon[27845]: pgmap v6533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11189.644698 secs 2024-04-26T14:57:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:29 smithi096 ceph-mon[32753]: pgmap v6533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11189.644698 secs 2024-04-26T14:57:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:31 smithi194 ceph-mon[33017]: pgmap v6534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:31 smithi033 ceph-mon[27845]: pgmap v6534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:31 smithi096 ceph-mon[32753]: pgmap v6534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:33 smithi194 ceph-mon[33017]: pgmap v6535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11191.860615 secs 2024-04-26T14:57:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:33 smithi033 ceph-mon[27845]: pgmap v6535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11191.860615 secs 2024-04-26T14:57:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:33 smithi096 ceph-mon[32753]: pgmap v6535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11191.860615 secs 2024-04-26T14:57:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11194.644852 secs 2024-04-26T14:57:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11194.644852 secs 2024-04-26T14:57:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11194.644852 secs 2024-04-26T14:57:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:35 smithi033 ceph-mon[27845]: pgmap v6536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:35 smithi096 ceph-mon[32753]: pgmap v6536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:35 smithi194 ceph-mon[33017]: pgmap v6536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:37 smithi033 ceph-mon[27845]: pgmap v6537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:37 smithi096 ceph-mon[32753]: pgmap v6537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:37 smithi194 ceph-mon[33017]: pgmap v6537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11196.860697 secs 2024-04-26T14:57:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11196.860697 secs 2024-04-26T14:57:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11196.860697 secs 2024-04-26T14:57:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:39 smithi033 ceph-mon[27845]: pgmap v6538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11199.644972 secs 2024-04-26T14:57:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:39 smithi096 ceph-mon[32753]: pgmap v6538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11199.644972 secs 2024-04-26T14:57:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:39 smithi194 ceph-mon[33017]: pgmap v6538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11199.644972 secs 2024-04-26T14:57:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:40.972+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:40.972+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:40.972+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:57:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:41 smithi033 ceph-mon[27845]: pgmap v6539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:57:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:41 smithi096 ceph-mon[32753]: pgmap v6539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:57:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:41 smithi194 ceph-mon[33017]: pgmap v6539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:57:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:57:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:57:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:57:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:57:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:57:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:57:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:57:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:57:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:57:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:43 smithi033 ceph-mon[27845]: pgmap v6540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:43 smithi033 ceph-mon[27845]: pgmap v6541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11201.860782 secs 2024-04-26T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:43 smithi096 ceph-mon[32753]: pgmap v6540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:43 smithi096 ceph-mon[32753]: pgmap v6541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11201.860782 secs 2024-04-26T14:57:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:43 smithi194 ceph-mon[33017]: pgmap v6540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:43 smithi194 ceph-mon[33017]: pgmap v6541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11201.860782 secs 2024-04-26T14:57:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11204.645083 secs 2024-04-26T14:57:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11204.645083 secs 2024-04-26T14:57:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11204.645083 secs 2024-04-26T14:57:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:45 smithi033 ceph-mon[27845]: pgmap v6542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:45 smithi096 ceph-mon[32753]: pgmap v6542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:45 smithi194 ceph-mon[33017]: pgmap v6542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:47 smithi033 ceph-mon[27845]: pgmap v6543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:47 smithi096 ceph-mon[32753]: pgmap v6543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:47 smithi194 ceph-mon[33017]: pgmap v6543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11206.860865 secs 2024-04-26T14:57:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11206.860865 secs 2024-04-26T14:57:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11206.860865 secs 2024-04-26T14:57:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:49 smithi096 ceph-mon[32753]: pgmap v6544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11209.645201 secs 2024-04-26T14:57:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:49 smithi194 ceph-mon[33017]: pgmap v6544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11209.645201 secs 2024-04-26T14:57:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:49 smithi033 ceph-mon[27845]: pgmap v6544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11209.645201 secs 2024-04-26T14:57:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:51 smithi194 ceph-mon[33017]: pgmap v6545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:51 smithi033 ceph-mon[27845]: pgmap v6545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:51 smithi096 ceph-mon[32753]: pgmap v6545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:57:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:53 smithi194 ceph-mon[33017]: pgmap v6546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:57:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11211.860947 secs 2024-04-26T14:57:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:53 smithi033 ceph-mon[27845]: pgmap v6546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:57:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11211.860947 secs 2024-04-26T14:57:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:53 smithi096 ceph-mon[32753]: pgmap v6546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:57:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11211.860947 secs 2024-04-26T14:57:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11214.645302 secs 2024-04-26T14:57:54.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11214.645302 secs 2024-04-26T14:57:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11214.645302 secs 2024-04-26T14:57:55.594 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:55 smithi096 ceph-mon[32753]: pgmap v6547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:55 smithi194 ceph-mon[33017]: pgmap v6547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:55 smithi033 ceph-mon[27845]: pgmap v6547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:55.988+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:55.988+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:57:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:57:55.988+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:57:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:57 smithi194 ceph-mon[33017]: pgmap v6548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:57.994 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:57 smithi096 ceph-mon[32753]: pgmap v6548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:57.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:57 smithi033 ceph-mon[27845]: pgmap v6548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:57:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11216.861031 secs 2024-04-26T14:57:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11216.861031 secs 2024-04-26T14:57:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11216.861031 secs 2024-04-26T14:57:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:59 smithi194 ceph-mon[33017]: pgmap v6549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11219.645368 secs 2024-04-26T14:57:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:59 smithi033 ceph-mon[27845]: pgmap v6549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11219.645368 secs 2024-04-26T14:57:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:59 smithi096 ceph-mon[32753]: pgmap v6549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:57:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11219.645368 secs 2024-04-26T14:58:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:01 smithi194 ceph-mon[33017]: pgmap v6550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:01 smithi033 ceph-mon[27845]: pgmap v6550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:01 smithi096 ceph-mon[32753]: pgmap v6550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:03 smithi194 ceph-mon[33017]: pgmap v6551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11221.861092 secs 2024-04-26T14:58:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:03 smithi033 ceph-mon[27845]: pgmap v6551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11221.861092 secs 2024-04-26T14:58:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:03 smithi096 ceph-mon[32753]: pgmap v6551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11221.861092 secs 2024-04-26T14:58:04.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11224.645437 secs 2024-04-26T14:58:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11224.645437 secs 2024-04-26T14:58:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11224.645437 secs 2024-04-26T14:58:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:05 smithi194 ceph-mon[33017]: pgmap v6552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:05 smithi033 ceph-mon[27845]: pgmap v6552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:05 smithi096 ceph-mon[32753]: pgmap v6552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:07 smithi194 ceph-mon[33017]: pgmap v6553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:07 smithi033 ceph-mon[27845]: pgmap v6553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:07 smithi096 ceph-mon[32753]: pgmap v6553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11226.861152 secs 2024-04-26T14:58:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11226.861152 secs 2024-04-26T14:58:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11226.861152 secs 2024-04-26T14:58:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:09 smithi194 ceph-mon[33017]: pgmap v6554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11229.645553 secs 2024-04-26T14:58:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:09 smithi033 ceph-mon[27845]: pgmap v6554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11229.645553 secs 2024-04-26T14:58:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:09 smithi096 ceph-mon[32753]: pgmap v6554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11229.645553 secs 2024-04-26T14:58:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:11.000+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:11.000+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:11.000+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:58:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:11 smithi194 ceph-mon[33017]: pgmap v6555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:11 smithi033 ceph-mon[27845]: pgmap v6555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:11 smithi096 ceph-mon[32753]: pgmap v6555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:13 smithi194 ceph-mon[33017]: pgmap v6556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11231.861239 secs 2024-04-26T14:58:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:13 smithi033 ceph-mon[27845]: pgmap v6556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11231.861239 secs 2024-04-26T14:58:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:13 smithi096 ceph-mon[32753]: pgmap v6556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11231.861239 secs 2024-04-26T14:58:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11234.645674 secs 2024-04-26T14:58:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11234.645674 secs 2024-04-26T14:58:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11234.645674 secs 2024-04-26T14:58:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:15 smithi194 ceph-mon[33017]: pgmap v6557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:15 smithi033 ceph-mon[27845]: pgmap v6557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:15 smithi096 ceph-mon[32753]: pgmap v6557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:17 smithi194 ceph-mon[33017]: pgmap v6558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:17 smithi033 ceph-mon[27845]: pgmap v6558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:17 smithi096 ceph-mon[32753]: pgmap v6558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11236.861299 secs 2024-04-26T14:58:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11236.861299 secs 2024-04-26T14:58:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11236.861299 secs 2024-04-26T14:58:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:19 smithi194 ceph-mon[33017]: pgmap v6559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11239.645792 secs 2024-04-26T14:58:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:19 smithi033 ceph-mon[27845]: pgmap v6559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11239.645792 secs 2024-04-26T14:58:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:19 smithi096 ceph-mon[32753]: pgmap v6559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11239.645792 secs 2024-04-26T14:58:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:21 smithi194 ceph-mon[33017]: pgmap v6560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:21 smithi033 ceph-mon[27845]: pgmap v6560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:21 smithi096 ceph-mon[32753]: pgmap v6560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:23 smithi194 ceph-mon[33017]: pgmap v6561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11241.861382 secs 2024-04-26T14:58:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:23 smithi033 ceph-mon[27845]: pgmap v6561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11241.861382 secs 2024-04-26T14:58:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:23 smithi096 ceph-mon[32753]: pgmap v6561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11241.861382 secs 2024-04-26T14:58:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11244.645899 secs 2024-04-26T14:58:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11244.645899 secs 2024-04-26T14:58:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11244.645899 secs 2024-04-26T14:58:25.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:25 smithi033 ceph-mon[27845]: pgmap v6562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:25 smithi194 ceph-mon[33017]: pgmap v6562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:25 smithi096 ceph-mon[32753]: pgmap v6562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:26.008+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:26.008+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:26.008+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:58:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:27 smithi194 ceph-mon[33017]: pgmap v6563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:27 smithi033 ceph-mon[27845]: pgmap v6563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:27 smithi096 ceph-mon[32753]: pgmap v6563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11246.861465 secs 2024-04-26T14:58:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11246.861465 secs 2024-04-26T14:58:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11246.861465 secs 2024-04-26T14:58:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:29 smithi194 ceph-mon[33017]: pgmap v6564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11249.645998 secs 2024-04-26T14:58:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:29 smithi033 ceph-mon[27845]: pgmap v6564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11249.645998 secs 2024-04-26T14:58:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:29 smithi096 ceph-mon[32753]: pgmap v6564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11249.645998 secs 2024-04-26T14:58:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:31 smithi194 ceph-mon[33017]: pgmap v6565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:31 smithi033 ceph-mon[27845]: pgmap v6565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:31 smithi096 ceph-mon[32753]: pgmap v6565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:33 smithi033 ceph-mon[27845]: pgmap v6566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11251.861546 secs 2024-04-26T14:58:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:33 smithi096 ceph-mon[32753]: pgmap v6566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11251.861546 secs 2024-04-26T14:58:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:33 smithi194 ceph-mon[33017]: pgmap v6566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11251.861546 secs 2024-04-26T14:58:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11254.646129 secs 2024-04-26T14:58:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11254.646129 secs 2024-04-26T14:58:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11254.646129 secs 2024-04-26T14:58:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:35 smithi033 ceph-mon[27845]: pgmap v6567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:35 smithi096 ceph-mon[32753]: pgmap v6567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:35 smithi194 ceph-mon[33017]: pgmap v6567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:37 smithi033 ceph-mon[27845]: pgmap v6568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:37 smithi096 ceph-mon[32753]: pgmap v6568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:37 smithi194 ceph-mon[33017]: pgmap v6568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11256.861630 secs 2024-04-26T14:58:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11256.861630 secs 2024-04-26T14:58:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11256.861630 secs 2024-04-26T14:58:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:39 smithi033 ceph-mon[27845]: pgmap v6569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11259.646229 secs 2024-04-26T14:58:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:39 smithi096 ceph-mon[32753]: pgmap v6569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11259.646229 secs 2024-04-26T14:58:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:39 smithi194 ceph-mon[33017]: pgmap v6569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11259.646229 secs 2024-04-26T14:58:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:41.010+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:41.010+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:41.010+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:58:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:41 smithi033 ceph-mon[27845]: pgmap v6570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:41 smithi096 ceph-mon[32753]: pgmap v6570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:41 smithi194 ceph-mon[33017]: pgmap v6570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:42.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:58:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:58:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:58:43.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:43 smithi096 ceph-mon[32753]: pgmap v6571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:43.660 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.660 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.660 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.660 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.660 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11261.861698 secs 2024-04-26T14:58:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:43 smithi033 ceph-mon[27845]: pgmap v6571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11261.861698 secs 2024-04-26T14:58:43.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:43 smithi194 ceph-mon[33017]: pgmap v6571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:43.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:43.778 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11261.861698 secs 2024-04-26T14:58:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11264.646297 secs 2024-04-26T14:58:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11264.646297 secs 2024-04-26T14:58:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11264.646297 secs 2024-04-26T14:58:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: pgmap v6572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: pgmap v6572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: pgmap v6572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:45.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T14:58:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T14:58:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:46.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:47 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:58:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:47 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:58:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:47 smithi096 ceph-mon[32753]: pgmap v6573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:47 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:58:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:47 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:58:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:47 smithi194 ceph-mon[33017]: pgmap v6573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:47 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T14:58:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:47 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T14:58:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:47 smithi033 ceph-mon[27845]: pgmap v6573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:58:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11266.861754 secs 2024-04-26T14:58:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:58:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11266.861754 secs 2024-04-26T14:58:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:58:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11266.861754 secs 2024-04-26T14:58:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T14:58:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:58:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:58:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:58:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:49 smithi194 ceph-mon[33017]: pgmap v6574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:49 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:58:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:49 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:58:49.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:49 smithi194 ceph-mon[33017]: pgmap v6575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:58:49.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:49 smithi194 ceph-mon[33017]: pgmap v6576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:58:49.756 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11269.646438 secs 2024-04-26T14:58:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:49 smithi033 ceph-mon[27845]: pgmap v6574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:49 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:58:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:49 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:58:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:49 smithi033 ceph-mon[27845]: pgmap v6575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:58:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:49 smithi033 ceph-mon[27845]: pgmap v6576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:58:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11269.646438 secs 2024-04-26T14:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:49 smithi096 ceph-mon[32753]: pgmap v6574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:49 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T14:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:49 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T14:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:49 smithi096 ceph-mon[32753]: pgmap v6575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T14:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:49 smithi096 ceph-mon[32753]: pgmap v6576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11269.646438 secs 2024-04-26T14:58:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:51 smithi194 ceph-mon[33017]: pgmap v6577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:51 smithi033 ceph-mon[27845]: pgmap v6577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:51 smithi096 ceph-mon[32753]: pgmap v6577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:53 smithi194 ceph-mon[33017]: pgmap v6578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11271.861838 secs 2024-04-26T14:58:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:53 smithi033 ceph-mon[27845]: pgmap v6578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11271.861838 secs 2024-04-26T14:58:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:53 smithi096 ceph-mon[32753]: pgmap v6578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11271.861838 secs 2024-04-26T14:58:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11274.646581 secs 2024-04-26T14:58:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11274.646581 secs 2024-04-26T14:58:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11274.646581 secs 2024-04-26T14:58:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:55 smithi194 ceph-mon[33017]: pgmap v6579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:58:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:55 smithi033 ceph-mon[27845]: pgmap v6579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:58:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:55 smithi096 ceph-mon[32753]: pgmap v6579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T14:58:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:56.026+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:56.026+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:58:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:58:56.026+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:58:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:56 smithi194 ceph-mon[33017]: pgmap v6580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:56 smithi033 ceph-mon[27845]: pgmap v6580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:56 smithi096 ceph-mon[32753]: pgmap v6580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T14:58:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11276.861922 secs 2024-04-26T14:58:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11276.861922 secs 2024-04-26T14:58:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11276.861922 secs 2024-04-26T14:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:58 smithi194 ceph-mon[33017]: pgmap v6581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:58:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:58 smithi033 ceph-mon[27845]: pgmap v6581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:58:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:58 smithi096 ceph-mon[32753]: pgmap v6581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T14:59:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11279.646707 secs 2024-04-26T14:59:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11279.646707 secs 2024-04-26T14:59:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11279.646707 secs 2024-04-26T14:59:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:00 smithi194 ceph-mon[33017]: pgmap v6582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:00 smithi033 ceph-mon[27845]: pgmap v6582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:00 smithi096 ceph-mon[32753]: pgmap v6582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:03 smithi194 ceph-mon[33017]: pgmap v6583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11281.862003 secs 2024-04-26T14:59:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:03 smithi033 ceph-mon[27845]: pgmap v6583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11281.862003 secs 2024-04-26T14:59:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:03 smithi096 ceph-mon[32753]: pgmap v6583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11281.862003 secs 2024-04-26T14:59:04.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11284.646826 secs 2024-04-26T14:59:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11284.646826 secs 2024-04-26T14:59:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11284.646826 secs 2024-04-26T14:59:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:05 smithi194 ceph-mon[33017]: pgmap v6584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:05 smithi033 ceph-mon[27845]: pgmap v6584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:05 smithi096 ceph-mon[32753]: pgmap v6584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:07 smithi194 ceph-mon[33017]: pgmap v6585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:07 smithi033 ceph-mon[27845]: pgmap v6585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:07 smithi096 ceph-mon[32753]: pgmap v6585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11286.862086 secs 2024-04-26T14:59:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11286.862086 secs 2024-04-26T14:59:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11286.862086 secs 2024-04-26T14:59:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:09 smithi194 ceph-mon[33017]: pgmap v6586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11289.646949 secs 2024-04-26T14:59:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:09 smithi033 ceph-mon[27845]: pgmap v6586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11289.646949 secs 2024-04-26T14:59:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:09 smithi096 ceph-mon[32753]: pgmap v6586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11289.646949 secs 2024-04-26T14:59:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:11.039+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:11.039+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:11.039+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:59:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:11 smithi194 ceph-mon[33017]: pgmap v6587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:11 smithi033 ceph-mon[27845]: pgmap v6587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:11 smithi096 ceph-mon[32753]: pgmap v6587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:13 smithi033 ceph-mon[27845]: pgmap v6588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11291.862175 secs 2024-04-26T14:59:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:13 smithi096 ceph-mon[32753]: pgmap v6588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11291.862175 secs 2024-04-26T14:59:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:13 smithi194 ceph-mon[33017]: pgmap v6588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11291.862175 secs 2024-04-26T14:59:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11294.647073 secs 2024-04-26T14:59:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11294.647073 secs 2024-04-26T14:59:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11294.647073 secs 2024-04-26T14:59:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:15 smithi033 ceph-mon[27845]: pgmap v6589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:15 smithi096 ceph-mon[32753]: pgmap v6589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:15 smithi194 ceph-mon[33017]: pgmap v6589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:17 smithi033 ceph-mon[27845]: pgmap v6590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:17 smithi096 ceph-mon[32753]: pgmap v6590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:17 smithi194 ceph-mon[33017]: pgmap v6590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11296.862262 secs 2024-04-26T14:59:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11296.862262 secs 2024-04-26T14:59:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11296.862262 secs 2024-04-26T14:59:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:19 smithi033 ceph-mon[27845]: pgmap v6591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11299.647222 secs 2024-04-26T14:59:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:19 smithi096 ceph-mon[32753]: pgmap v6591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11299.647222 secs 2024-04-26T14:59:19.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:19 smithi194 ceph-mon[33017]: pgmap v6591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:19.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11299.647222 secs 2024-04-26T14:59:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:21 smithi033 ceph-mon[27845]: pgmap v6592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:21 smithi096 ceph-mon[32753]: pgmap v6592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:21 smithi194 ceph-mon[33017]: pgmap v6592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:23 smithi033 ceph-mon[27845]: pgmap v6593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11301.862344 secs 2024-04-26T14:59:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:23 smithi096 ceph-mon[32753]: pgmap v6593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11301.862344 secs 2024-04-26T14:59:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:23 smithi194 ceph-mon[33017]: pgmap v6593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11301.862344 secs 2024-04-26T14:59:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.713+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T14:59:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.713+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T14:59:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.714+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T14:59:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.714+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T14:59:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.714+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x564385821900 2024-04-26T14:59:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.714+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T14:59:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.714+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T14:59:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:23.714+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T14:59:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11304.647338 secs 2024-04-26T14:59:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11304.647338 secs 2024-04-26T14:59:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11304.647338 secs 2024-04-26T14:59:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:25 smithi033 ceph-mon[27845]: pgmap v6594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:25 smithi096 ceph-mon[32753]: pgmap v6594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:25 smithi194 ceph-mon[33017]: pgmap v6594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:26.050+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:26.050+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:26.050+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:59:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:27 smithi033 ceph-mon[27845]: pgmap v6595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:27 smithi096 ceph-mon[32753]: pgmap v6595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:27 smithi194 ceph-mon[33017]: pgmap v6595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11306.862401 secs 2024-04-26T14:59:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11306.862401 secs 2024-04-26T14:59:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11306.862401 secs 2024-04-26T14:59:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:29 smithi194 ceph-mon[33017]: pgmap v6596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11309.647418 secs 2024-04-26T14:59:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:29 smithi033 ceph-mon[27845]: pgmap v6596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11309.647418 secs 2024-04-26T14:59:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:29 smithi096 ceph-mon[32753]: pgmap v6596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11309.647418 secs 2024-04-26T14:59:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:31 smithi194 ceph-mon[33017]: pgmap v6597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:31 smithi033 ceph-mon[27845]: pgmap v6597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:31 smithi096 ceph-mon[32753]: pgmap v6597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:33 smithi194 ceph-mon[33017]: pgmap v6598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11311.862483 secs 2024-04-26T14:59:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:33 smithi033 ceph-mon[27845]: pgmap v6598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11311.862483 secs 2024-04-26T14:59:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:33 smithi096 ceph-mon[32753]: pgmap v6598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11311.862483 secs 2024-04-26T14:59:34.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11314.647535 secs 2024-04-26T14:59:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11314.647535 secs 2024-04-26T14:59:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11314.647535 secs 2024-04-26T14:59:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:35 smithi194 ceph-mon[33017]: pgmap v6599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:35 smithi033 ceph-mon[27845]: pgmap v6599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:35 smithi096 ceph-mon[32753]: pgmap v6599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:37 smithi194 ceph-mon[33017]: pgmap v6600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:37 smithi033 ceph-mon[27845]: pgmap v6600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:37 smithi096 ceph-mon[32753]: pgmap v6600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11316.862564 secs 2024-04-26T14:59:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11316.862564 secs 2024-04-26T14:59:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11316.862564 secs 2024-04-26T14:59:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:39 smithi194 ceph-mon[33017]: pgmap v6601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11319.647641 secs 2024-04-26T14:59:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:39 smithi033 ceph-mon[27845]: pgmap v6601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11319.647641 secs 2024-04-26T14:59:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:39 smithi096 ceph-mon[32753]: pgmap v6601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11319.647641 secs 2024-04-26T14:59:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:41.064+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:41.064+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:41.064+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:59:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:41 smithi194 ceph-mon[33017]: pgmap v6602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:41 smithi033 ceph-mon[27845]: pgmap v6602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:41 smithi096 ceph-mon[32753]: pgmap v6602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:43 smithi194 ceph-mon[33017]: pgmap v6603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11321.862646 secs 2024-04-26T14:59:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:43 smithi033 ceph-mon[27845]: pgmap v6603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11321.862646 secs 2024-04-26T14:59:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:43 smithi096 ceph-mon[32753]: pgmap v6603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11321.862646 secs 2024-04-26T14:59:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11324.647770 secs 2024-04-26T14:59:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11324.647770 secs 2024-04-26T14:59:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11324.647770 secs 2024-04-26T14:59:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:45 smithi194 ceph-mon[33017]: pgmap v6604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:45 smithi033 ceph-mon[27845]: pgmap v6604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:45 smithi096 ceph-mon[32753]: pgmap v6604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:47 smithi194 ceph-mon[33017]: pgmap v6605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:47 smithi033 ceph-mon[27845]: pgmap v6605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:47 smithi096 ceph-mon[32753]: pgmap v6605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T14:59:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11326.862725 secs 2024-04-26T14:59:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11326.862725 secs 2024-04-26T14:59:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11326.862725 secs 2024-04-26T14:59:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:49 smithi194 ceph-mon[33017]: pgmap v6606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11329.647892 secs 2024-04-26T14:59:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:59:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:49 smithi033 ceph-mon[27845]: pgmap v6606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11329.647892 secs 2024-04-26T14:59:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:59:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:49 smithi096 ceph-mon[32753]: pgmap v6606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T14:59:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11329.647892 secs 2024-04-26T14:59:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T14:59:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:59:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:59:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:59:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:59:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:59:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:59:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T14:59:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T14:59:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T14:59:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:51 smithi194 ceph-mon[33017]: pgmap v6607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:59:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:51 smithi194 ceph-mon[33017]: pgmap v6608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:51 smithi033 ceph-mon[27845]: pgmap v6607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:59:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:51 smithi033 ceph-mon[27845]: pgmap v6608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:51 smithi096 ceph-mon[32753]: pgmap v6607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T14:59:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:51 smithi096 ceph-mon[32753]: pgmap v6608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:53 smithi194 ceph-mon[33017]: pgmap v6609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11331.862807 secs 2024-04-26T14:59:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:53 smithi033 ceph-mon[27845]: pgmap v6609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11331.862807 secs 2024-04-26T14:59:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:53 smithi096 ceph-mon[32753]: pgmap v6609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11331.862807 secs 2024-04-26T14:59:54.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11334.648011 secs 2024-04-26T14:59:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11334.648011 secs 2024-04-26T14:59:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11334.648011 secs 2024-04-26T14:59:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:55 smithi194 ceph-mon[33017]: pgmap v6610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:55 smithi033 ceph-mon[27845]: pgmap v6610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:55 smithi096 ceph-mon[32753]: pgmap v6610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:56.080+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:56.080+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T14:59:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T14:59:56.080+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T14:59:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:57 smithi194 ceph-mon[33017]: pgmap v6611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:57 smithi033 ceph-mon[27845]: pgmap v6611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:57 smithi096 ceph-mon[32753]: pgmap v6611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11336.862890 secs 2024-04-26T14:59:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11336.862890 secs 2024-04-26T14:59:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11336.862890 secs 2024-04-26T14:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:59 smithi194 ceph-mon[33017]: pgmap v6612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 14:59:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11339.648155 secs 2024-04-26T14:59:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:59 smithi033 ceph-mon[27845]: pgmap v6612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 14:59:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11339.648155 secs 2024-04-26T14:59:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:59 smithi096 ceph-mon[32753]: pgmap v6612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T14:59:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 14:59:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11339.648155 secs 2024-04-26T15:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:00:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:00:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:01 smithi194 ceph-mon[33017]: pgmap v6613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:00:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:01 smithi033 ceph-mon[27845]: pgmap v6613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:00:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:01 smithi096 ceph-mon[32753]: pgmap v6613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:00:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:03 smithi194 ceph-mon[33017]: pgmap v6614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11341.862978 secs 2024-04-26T15:00:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:03 smithi033 ceph-mon[27845]: pgmap v6614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11341.862978 secs 2024-04-26T15:00:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:03 smithi096 ceph-mon[32753]: pgmap v6614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11341.862978 secs 2024-04-26T15:00:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11344.648245 secs 2024-04-26T15:00:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11344.648245 secs 2024-04-26T15:00:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11344.648245 secs 2024-04-26T15:00:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:05 smithi033 ceph-mon[27845]: pgmap v6615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:05 smithi096 ceph-mon[32753]: pgmap v6615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:05 smithi194 ceph-mon[33017]: pgmap v6615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:07 smithi033 ceph-mon[27845]: pgmap v6616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:07 smithi096 ceph-mon[32753]: pgmap v6616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:07 smithi194 ceph-mon[33017]: pgmap v6616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11346.863063 secs 2024-04-26T15:00:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11346.863063 secs 2024-04-26T15:00:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11346.863063 secs 2024-04-26T15:00:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:09 smithi033 ceph-mon[27845]: pgmap v6617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11349.648333 secs 2024-04-26T15:00:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:09 smithi096 ceph-mon[32753]: pgmap v6617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11349.648333 secs 2024-04-26T15:00:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:09 smithi194 ceph-mon[33017]: pgmap v6617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11349.648333 secs 2024-04-26T15:00:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:11.082+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:11.082+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:11.082+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:00:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:11 smithi033 ceph-mon[27845]: pgmap v6618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:11 smithi096 ceph-mon[32753]: pgmap v6618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:11 smithi194 ceph-mon[33017]: pgmap v6618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:13 smithi033 ceph-mon[27845]: pgmap v6619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11351.863120 secs 2024-04-26T15:00:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:13 smithi096 ceph-mon[32753]: pgmap v6619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11351.863120 secs 2024-04-26T15:00:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:13 smithi194 ceph-mon[33017]: pgmap v6619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11351.863120 secs 2024-04-26T15:00:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11354.648455 secs 2024-04-26T15:00:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11354.648455 secs 2024-04-26T15:00:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11354.648455 secs 2024-04-26T15:00:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:15 smithi033 ceph-mon[27845]: pgmap v6620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:15 smithi096 ceph-mon[32753]: pgmap v6620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:15 smithi194 ceph-mon[33017]: pgmap v6620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:17 smithi033 ceph-mon[27845]: pgmap v6621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:17 smithi096 ceph-mon[32753]: pgmap v6621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:17 smithi194 ceph-mon[33017]: pgmap v6621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11356.863182 secs 2024-04-26T15:00:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11356.863182 secs 2024-04-26T15:00:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11356.863182 secs 2024-04-26T15:00:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:19 smithi033 ceph-mon[27845]: pgmap v6622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11359.648521 secs 2024-04-26T15:00:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:19 smithi096 ceph-mon[32753]: pgmap v6622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11359.648521 secs 2024-04-26T15:00:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:19 smithi194 ceph-mon[33017]: pgmap v6622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11359.648521 secs 2024-04-26T15:00:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:21 smithi033 ceph-mon[27845]: pgmap v6623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:21 smithi096 ceph-mon[32753]: pgmap v6623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:21 smithi194 ceph-mon[33017]: pgmap v6623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:23 smithi096 ceph-mon[32753]: pgmap v6624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11361.863265 secs 2024-04-26T15:00:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:23 smithi194 ceph-mon[33017]: pgmap v6624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11361.863265 secs 2024-04-26T15:00:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:23 smithi033 ceph-mon[27845]: pgmap v6624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11361.863265 secs 2024-04-26T15:00:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11364.648644 secs 2024-04-26T15:00:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11364.648644 secs 2024-04-26T15:00:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11364.648644 secs 2024-04-26T15:00:25.852 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:25 smithi096 ceph-mon[32753]: pgmap v6625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:25 smithi194 ceph-mon[33017]: pgmap v6625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:25 smithi033 ceph-mon[27845]: pgmap v6625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:26.090+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:26.090+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:26.090+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:00:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:27 smithi194 ceph-mon[33017]: pgmap v6626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:27 smithi033 ceph-mon[27845]: pgmap v6626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:27 smithi096 ceph-mon[32753]: pgmap v6626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11366.863354 secs 2024-04-26T15:00:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11366.863354 secs 2024-04-26T15:00:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11366.863354 secs 2024-04-26T15:00:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:29 smithi194 ceph-mon[33017]: pgmap v6627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11369.648768 secs 2024-04-26T15:00:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:29 smithi033 ceph-mon[27845]: pgmap v6627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11369.648768 secs 2024-04-26T15:00:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:29 smithi096 ceph-mon[32753]: pgmap v6627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11369.648768 secs 2024-04-26T15:00:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:31 smithi194 ceph-mon[33017]: pgmap v6628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:31 smithi033 ceph-mon[27845]: pgmap v6628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:31 smithi096 ceph-mon[32753]: pgmap v6628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:33 smithi194 ceph-mon[33017]: pgmap v6629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11371.863436 secs 2024-04-26T15:00:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:33 smithi033 ceph-mon[27845]: pgmap v6629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11371.863436 secs 2024-04-26T15:00:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:33 smithi096 ceph-mon[32753]: pgmap v6629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11371.863436 secs 2024-04-26T15:00:34.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11374.648896 secs 2024-04-26T15:00:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11374.648896 secs 2024-04-26T15:00:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11374.648896 secs 2024-04-26T15:00:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:35 smithi194 ceph-mon[33017]: pgmap v6630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:35 smithi033 ceph-mon[27845]: pgmap v6630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:35 smithi096 ceph-mon[32753]: pgmap v6630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:00:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:37 smithi194 ceph-mon[33017]: pgmap v6631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:37 smithi033 ceph-mon[27845]: pgmap v6631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:37 smithi096 ceph-mon[32753]: pgmap v6631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:00:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11376.863519 secs 2024-04-26T15:00:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:38 smithi194 ceph-mon[33017]: pgmap v6632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11376.863519 secs 2024-04-26T15:00:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:38 smithi033 ceph-mon[27845]: pgmap v6632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11376.863519 secs 2024-04-26T15:00:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:38 smithi096 ceph-mon[32753]: pgmap v6632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11379.649019 secs 2024-04-26T15:00:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11379.649019 secs 2024-04-26T15:00:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11379.649019 secs 2024-04-26T15:00:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:40 smithi194 ceph-mon[33017]: pgmap v6633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:40 smithi033 ceph-mon[27845]: pgmap v6633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:40 smithi096 ceph-mon[32753]: pgmap v6633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:41.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:41.103+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:41.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:41.103+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:41.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:41.103+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:00:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:42 smithi194 ceph-mon[33017]: pgmap v6634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:42 smithi033 ceph-mon[27845]: pgmap v6634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:42 smithi096 ceph-mon[32753]: pgmap v6634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11381.863578 secs 2024-04-26T15:00:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11381.863578 secs 2024-04-26T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11381.863578 secs 2024-04-26T15:00:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11384.649139 secs 2024-04-26T15:00:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:44 smithi194 ceph-mon[33017]: pgmap v6635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11384.649139 secs 2024-04-26T15:00:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:44 smithi033 ceph-mon[27845]: pgmap v6635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11384.649139 secs 2024-04-26T15:00:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:44 smithi096 ceph-mon[32753]: pgmap v6635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:46 smithi194 ceph-mon[33017]: pgmap v6636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:46 smithi033 ceph-mon[27845]: pgmap v6636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:46 smithi096 ceph-mon[32753]: pgmap v6636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11386.863659 secs 2024-04-26T15:00:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11386.863659 secs 2024-04-26T15:00:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11386.863659 secs 2024-04-26T15:00:49.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:48 smithi194 ceph-mon[33017]: pgmap v6637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:48 smithi033 ceph-mon[27845]: pgmap v6637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:48 smithi096 ceph-mon[32753]: pgmap v6637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11389.649276 secs 2024-04-26T15:00:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11389.649276 secs 2024-04-26T15:00:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11389.649276 secs 2024-04-26T15:00:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:50 smithi194 ceph-mon[33017]: pgmap v6638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:00:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:50 smithi033 ceph-mon[27845]: pgmap v6638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:00:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:50 smithi096 ceph-mon[32753]: pgmap v6638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:00:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:00:52.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:00:52.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:00:52.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:51 smithi194 ceph-mon[33017]: pgmap v6639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:52.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:51 smithi194 ceph-mon[33017]: pgmap v6640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:52.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:00:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:00:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:00:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:51 smithi033 ceph-mon[27845]: pgmap v6639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:51 smithi033 ceph-mon[27845]: pgmap v6640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:00:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:00:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:00:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:51 smithi096 ceph-mon[32753]: pgmap v6639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:00:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:51 smithi096 ceph-mon[32753]: pgmap v6640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:00:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11391.863742 secs 2024-04-26T15:00:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11391.863742 secs 2024-04-26T15:00:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11391.863742 secs 2024-04-26T15:00:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:53 smithi096 ceph-mon[32753]: pgmap v6641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:53 smithi194 ceph-mon[33017]: pgmap v6641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:53 smithi033 ceph-mon[27845]: pgmap v6641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11394.649413 secs 2024-04-26T15:00:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11394.649413 secs 2024-04-26T15:00:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11394.649413 secs 2024-04-26T15:00:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:55 smithi194 ceph-mon[33017]: pgmap v6642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:56.110+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:56.110+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:00:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:00:56.110+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:00:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:55 smithi033 ceph-mon[27845]: pgmap v6642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:55 smithi096 ceph-mon[32753]: pgmap v6642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:58 smithi033 ceph-mon[27845]: pgmap v6643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11396.863826 secs 2024-04-26T15:00:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:58 smithi096 ceph-mon[32753]: pgmap v6643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11396.863826 secs 2024-04-26T15:00:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:58 smithi194 ceph-mon[33017]: pgmap v6643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:00:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11396.863826 secs 2024-04-26T15:00:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11399.649540 secs 2024-04-26T15:00:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11399.649540 secs 2024-04-26T15:00:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11399.649540 secs 2024-04-26T15:01:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:00 smithi033 ceph-mon[27845]: pgmap v6644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:01:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:00 smithi096 ceph-mon[32753]: pgmap v6644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:01:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:00 smithi194 ceph-mon[33017]: pgmap v6644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:01:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:02 smithi033 ceph-mon[27845]: pgmap v6645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:01:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:02 smithi096 ceph-mon[32753]: pgmap v6645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:01:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:02 smithi194 ceph-mon[33017]: pgmap v6645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:01:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11401.863912 secs 2024-04-26T15:01:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11401.863912 secs 2024-04-26T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11401.863912 secs 2024-04-26T15:01:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11404.649664 secs 2024-04-26T15:01:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:04 smithi096 ceph-mon[32753]: pgmap v6646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11404.649664 secs 2024-04-26T15:01:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:04 smithi194 ceph-mon[33017]: pgmap v6646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11404.649664 secs 2024-04-26T15:01:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:04 smithi033 ceph-mon[27845]: pgmap v6646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:06 smithi096 ceph-mon[32753]: pgmap v6647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:06 smithi194 ceph-mon[33017]: pgmap v6647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:06 smithi033 ceph-mon[27845]: pgmap v6647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:08 smithi194 ceph-mon[33017]: pgmap v6648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11406.863995 secs 2024-04-26T15:01:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:08 smithi033 ceph-mon[27845]: pgmap v6648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11406.863995 secs 2024-04-26T15:01:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:08 smithi096 ceph-mon[32753]: pgmap v6648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11406.863995 secs 2024-04-26T15:01:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11409.649768 secs 2024-04-26T15:01:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11409.649768 secs 2024-04-26T15:01:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11409.649768 secs 2024-04-26T15:01:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:10 smithi194 ceph-mon[33017]: pgmap v6649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:10 smithi033 ceph-mon[27845]: pgmap v6649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:10 smithi096 ceph-mon[32753]: pgmap v6649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:11.127+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:11.127+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:11.127+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:01:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:12 smithi194 ceph-mon[33017]: pgmap v6650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:12 smithi033 ceph-mon[27845]: pgmap v6650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:12 smithi096 ceph-mon[32753]: pgmap v6650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11411.864055 secs 2024-04-26T15:01:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11411.864055 secs 2024-04-26T15:01:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11411.864055 secs 2024-04-26T15:01:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11414.649869 secs 2024-04-26T15:01:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:14 smithi194 ceph-mon[33017]: pgmap v6651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11414.649869 secs 2024-04-26T15:01:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:14 smithi033 ceph-mon[27845]: pgmap v6651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11414.649869 secs 2024-04-26T15:01:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:14 smithi096 ceph-mon[32753]: pgmap v6651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:16 smithi194 ceph-mon[33017]: pgmap v6652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:16 smithi033 ceph-mon[27845]: pgmap v6652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:16 smithi096 ceph-mon[32753]: pgmap v6652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:18.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:18 smithi194 ceph-mon[33017]: pgmap v6653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:18.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11416.864139 secs 2024-04-26T15:01:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:18 smithi033 ceph-mon[27845]: pgmap v6653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11416.864139 secs 2024-04-26T15:01:18.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:18 smithi096 ceph-mon[32753]: pgmap v6653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:18.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11416.864139 secs 2024-04-26T15:01:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11419.649994 secs 2024-04-26T15:01:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11419.649994 secs 2024-04-26T15:01:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11419.649994 secs 2024-04-26T15:01:20.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:20 smithi194 ceph-mon[33017]: pgmap v6654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:20 smithi033 ceph-mon[27845]: pgmap v6654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:20 smithi096 ceph-mon[32753]: pgmap v6654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:21 smithi194 ceph-mon[33017]: pgmap v6655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:21 smithi033 ceph-mon[27845]: pgmap v6655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:21 smithi096 ceph-mon[32753]: pgmap v6655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11421.864221 secs 2024-04-26T15:01:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11421.864221 secs 2024-04-26T15:01:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11421.864221 secs 2024-04-26T15:01:24.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:23 smithi096 ceph-mon[32753]: pgmap v6656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:23 smithi194 ceph-mon[33017]: pgmap v6656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:23 smithi033 ceph-mon[27845]: pgmap v6656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11424.650126 secs 2024-04-26T15:01:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11424.650126 secs 2024-04-26T15:01:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11424.650126 secs 2024-04-26T15:01:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:25 smithi194 ceph-mon[33017]: pgmap v6657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:26.139+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:26.139+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:26.139+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:01:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:25 smithi033 ceph-mon[27845]: pgmap v6657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:25 smithi096 ceph-mon[32753]: pgmap v6657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:28 smithi033 ceph-mon[27845]: pgmap v6658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11426.864284 secs 2024-04-26T15:01:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:28 smithi096 ceph-mon[32753]: pgmap v6658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11426.864284 secs 2024-04-26T15:01:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:28 smithi194 ceph-mon[33017]: pgmap v6658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11426.864284 secs 2024-04-26T15:01:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11429.650250 secs 2024-04-26T15:01:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11429.650250 secs 2024-04-26T15:01:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11429.650250 secs 2024-04-26T15:01:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:30 smithi033 ceph-mon[27845]: pgmap v6659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:30 smithi096 ceph-mon[32753]: pgmap v6659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:30 smithi194 ceph-mon[33017]: pgmap v6659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:32 smithi096 ceph-mon[32753]: pgmap v6660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:32 smithi194 ceph-mon[33017]: pgmap v6660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:32 smithi033 ceph-mon[27845]: pgmap v6660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11431.864368 secs 2024-04-26T15:01:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11431.864368 secs 2024-04-26T15:01:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11431.864368 secs 2024-04-26T15:01:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11434.650332 secs 2024-04-26T15:01:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:34 smithi033 ceph-mon[27845]: pgmap v6661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11434.650332 secs 2024-04-26T15:01:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:34 smithi194 ceph-mon[33017]: pgmap v6661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11434.650332 secs 2024-04-26T15:01:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:34 smithi096 ceph-mon[32753]: pgmap v6661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:36 smithi194 ceph-mon[33017]: pgmap v6662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:36 smithi033 ceph-mon[27845]: pgmap v6662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:36 smithi096 ceph-mon[32753]: pgmap v6662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:38 smithi194 ceph-mon[33017]: pgmap v6663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11436.864447 secs 2024-04-26T15:01:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:38 smithi033 ceph-mon[27845]: pgmap v6663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11436.864447 secs 2024-04-26T15:01:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:38 smithi096 ceph-mon[32753]: pgmap v6663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11436.864447 secs 2024-04-26T15:01:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11439.650460 secs 2024-04-26T15:01:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11439.650460 secs 2024-04-26T15:01:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11439.650460 secs 2024-04-26T15:01:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:40 smithi194 ceph-mon[33017]: pgmap v6664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:40 smithi033 ceph-mon[27845]: pgmap v6664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:40 smithi096 ceph-mon[32753]: pgmap v6664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:41.140+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:41.140+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:41.140+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:01:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:42 smithi194 ceph-mon[33017]: pgmap v6665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:42 smithi033 ceph-mon[27845]: pgmap v6665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:42 smithi096 ceph-mon[32753]: pgmap v6665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11441.864534 secs 2024-04-26T15:01:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11441.864534 secs 2024-04-26T15:01:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11441.864534 secs 2024-04-26T15:01:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11444.650558 secs 2024-04-26T15:01:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:44 smithi194 ceph-mon[33017]: pgmap v6666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11444.650558 secs 2024-04-26T15:01:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:44 smithi033 ceph-mon[27845]: pgmap v6666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11444.650558 secs 2024-04-26T15:01:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:44 smithi096 ceph-mon[32753]: pgmap v6666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:46 smithi194 ceph-mon[33017]: pgmap v6667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:46 smithi033 ceph-mon[27845]: pgmap v6667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:46 smithi096 ceph-mon[32753]: pgmap v6667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:48 smithi194 ceph-mon[33017]: pgmap v6668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11446.864594 secs 2024-04-26T15:01:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:48 smithi033 ceph-mon[27845]: pgmap v6668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11446.864594 secs 2024-04-26T15:01:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:48 smithi096 ceph-mon[32753]: pgmap v6668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11446.864594 secs 2024-04-26T15:01:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11449.650682 secs 2024-04-26T15:01:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11449.650682 secs 2024-04-26T15:01:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11449.650682 secs 2024-04-26T15:01:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:50 smithi194 ceph-mon[33017]: pgmap v6669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:50 smithi033 ceph-mon[27845]: pgmap v6669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:50 smithi096 ceph-mon[32753]: pgmap v6669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:01:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:51 smithi194 ceph-mon[33017]: pgmap v6670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:51 smithi033 ceph-mon[27845]: pgmap v6670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:51 smithi096 ceph-mon[32753]: pgmap v6670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:01:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:01:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:01:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11451.864668 secs 2024-04-26T15:01:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:53 smithi194 ceph-mon[33017]: pgmap v6671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11451.864668 secs 2024-04-26T15:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:53 smithi033 ceph-mon[27845]: pgmap v6671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11451.864668 secs 2024-04-26T15:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:53 smithi096 ceph-mon[32753]: pgmap v6671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:01:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11454.650793 secs 2024-04-26T15:01:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:01:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:01:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: pgmap v6672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:01:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: pgmap v6673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:01:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11454.650793 secs 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: pgmap v6672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: pgmap v6673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11454.650793 secs 2024-04-26T15:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: pgmap v6672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:01:54.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: pgmap v6673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:01:54.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:01:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:56.151+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:56.151+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:01:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:01:56.151+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:01:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:56 smithi194 ceph-mon[33017]: pgmap v6674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:01:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:56 smithi033 ceph-mon[27845]: pgmap v6674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:01:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:56 smithi096 ceph-mon[32753]: pgmap v6674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:01:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11456.864750 secs 2024-04-26T15:01:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11456.864750 secs 2024-04-26T15:01:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11456.864750 secs 2024-04-26T15:01:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:58 smithi194 ceph-mon[33017]: pgmap v6675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:01:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:01:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11459.650940 secs 2024-04-26T15:01:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:58 smithi033 ceph-mon[27845]: pgmap v6675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:01:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:01:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11459.650940 secs 2024-04-26T15:01:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:58 smithi096 ceph-mon[32753]: pgmap v6675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:01:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:01:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11459.650940 secs 2024-04-26T15:02:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:00 smithi194 ceph-mon[33017]: pgmap v6676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:02:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:00 smithi033 ceph-mon[27845]: pgmap v6676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:02:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:00 smithi096 ceph-mon[32753]: pgmap v6676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:02:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:02 smithi194 ceph-mon[33017]: pgmap v6677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:02:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11461.864830 secs 2024-04-26T15:02:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:02 smithi033 ceph-mon[27845]: pgmap v6677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:02:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11461.864830 secs 2024-04-26T15:02:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:02 smithi096 ceph-mon[32753]: pgmap v6677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:02:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11461.864830 secs 2024-04-26T15:02:04.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11464.651011 secs 2024-04-26T15:02:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11464.651011 secs 2024-04-26T15:02:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11464.651011 secs 2024-04-26T15:02:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:05 smithi194 ceph-mon[33017]: pgmap v6678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:02:05.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:05 smithi033 ceph-mon[27845]: pgmap v6678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:02:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:05 smithi096 ceph-mon[32753]: pgmap v6678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:02:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:07 smithi194 ceph-mon[33017]: pgmap v6679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:07 smithi033 ceph-mon[27845]: pgmap v6679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:07 smithi096 ceph-mon[32753]: pgmap v6679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11466.864935 secs 2024-04-26T15:02:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11466.864935 secs 2024-04-26T15:02:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11466.864935 secs 2024-04-26T15:02:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:09 smithi194 ceph-mon[33017]: pgmap v6680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11469.651126 secs 2024-04-26T15:02:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:09 smithi033 ceph-mon[27845]: pgmap v6680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11469.651126 secs 2024-04-26T15:02:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:09 smithi096 ceph-mon[32753]: pgmap v6680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11469.651126 secs 2024-04-26T15:02:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:11.164+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:11.164+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:11.164+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:02:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:11 smithi194 ceph-mon[33017]: pgmap v6681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:11 smithi033 ceph-mon[27845]: pgmap v6681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:11 smithi096 ceph-mon[32753]: pgmap v6681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:13 smithi194 ceph-mon[33017]: pgmap v6682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11471.865016 secs 2024-04-26T15:02:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:13 smithi033 ceph-mon[27845]: pgmap v6682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:13.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11471.865016 secs 2024-04-26T15:02:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:13 smithi096 ceph-mon[32753]: pgmap v6682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11471.865016 secs 2024-04-26T15:02:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11474.651253 secs 2024-04-26T15:02:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11474.651253 secs 2024-04-26T15:02:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11474.651253 secs 2024-04-26T15:02:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:15 smithi194 ceph-mon[33017]: pgmap v6683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:15.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:15 smithi033 ceph-mon[27845]: pgmap v6683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:15 smithi096 ceph-mon[32753]: pgmap v6683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:17 smithi194 ceph-mon[33017]: pgmap v6684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:17 smithi033 ceph-mon[27845]: pgmap v6684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:17 smithi096 ceph-mon[32753]: pgmap v6684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11476.865075 secs 2024-04-26T15:02:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11476.865075 secs 2024-04-26T15:02:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11476.865075 secs 2024-04-26T15:02:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:19 smithi194 ceph-mon[33017]: pgmap v6685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11479.651398 secs 2024-04-26T15:02:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:19 smithi033 ceph-mon[27845]: pgmap v6685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11479.651398 secs 2024-04-26T15:02:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:19 smithi096 ceph-mon[32753]: pgmap v6685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11479.651398 secs 2024-04-26T15:02:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:21 smithi194 ceph-mon[33017]: pgmap v6686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:21 smithi033 ceph-mon[27845]: pgmap v6686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:21 smithi096 ceph-mon[32753]: pgmap v6686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:23 smithi194 ceph-mon[33017]: pgmap v6687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11481.865158 secs 2024-04-26T15:02:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:23 smithi033 ceph-mon[27845]: pgmap v6687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11481.865158 secs 2024-04-26T15:02:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:23 smithi096 ceph-mon[32753]: pgmap v6687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11481.865158 secs 2024-04-26T15:02:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:23.745+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:02:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:23.745+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:02:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:23.745+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:02:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:23.745+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:02:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11484.651522 secs 2024-04-26T15:02:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11484.651522 secs 2024-04-26T15:02:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11484.651522 secs 2024-04-26T15:02:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:25 smithi194 ceph-mon[33017]: pgmap v6688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:25 smithi033 ceph-mon[27845]: pgmap v6688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:25 smithi096 ceph-mon[32753]: pgmap v6688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:26.168+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:26.168+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:26.168+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:02:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:27 smithi194 ceph-mon[33017]: pgmap v6689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:27 smithi033 ceph-mon[27845]: pgmap v6689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:27 smithi096 ceph-mon[32753]: pgmap v6689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11486.865219 secs 2024-04-26T15:02:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11486.865219 secs 2024-04-26T15:02:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11486.865219 secs 2024-04-26T15:02:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:29 smithi033 ceph-mon[27845]: pgmap v6690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11489.651596 secs 2024-04-26T15:02:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:29 smithi096 ceph-mon[32753]: pgmap v6690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11489.651596 secs 2024-04-26T15:02:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:29 smithi194 ceph-mon[33017]: pgmap v6690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11489.651596 secs 2024-04-26T15:02:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:31 smithi033 ceph-mon[27845]: pgmap v6691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:31 smithi096 ceph-mon[32753]: pgmap v6691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:31 smithi194 ceph-mon[33017]: pgmap v6691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:33 smithi033 ceph-mon[27845]: pgmap v6692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11491.865300 secs 2024-04-26T15:02:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:33 smithi096 ceph-mon[32753]: pgmap v6692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11491.865300 secs 2024-04-26T15:02:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:33 smithi194 ceph-mon[33017]: pgmap v6692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11491.865300 secs 2024-04-26T15:02:34.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11494.651722 secs 2024-04-26T15:02:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11494.651722 secs 2024-04-26T15:02:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11494.651722 secs 2024-04-26T15:02:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:35 smithi033 ceph-mon[27845]: pgmap v6693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:35 smithi096 ceph-mon[32753]: pgmap v6693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:35 smithi194 ceph-mon[33017]: pgmap v6693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:37 smithi033 ceph-mon[27845]: pgmap v6694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:37 smithi096 ceph-mon[32753]: pgmap v6694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:37 smithi194 ceph-mon[33017]: pgmap v6694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11496.865380 secs 2024-04-26T15:02:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11496.865380 secs 2024-04-26T15:02:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11496.865380 secs 2024-04-26T15:02:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:39 smithi033 ceph-mon[27845]: pgmap v6695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11499.651842 secs 2024-04-26T15:02:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:39 smithi096 ceph-mon[32753]: pgmap v6695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11499.651842 secs 2024-04-26T15:02:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:39 smithi194 ceph-mon[33017]: pgmap v6695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11499.651842 secs 2024-04-26T15:02:41.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:41.179+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:41.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:41.179+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:41.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:41.179+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:02:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:41 smithi033 ceph-mon[27845]: pgmap v6696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:41 smithi096 ceph-mon[32753]: pgmap v6696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:41 smithi194 ceph-mon[33017]: pgmap v6696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:43 smithi033 ceph-mon[27845]: pgmap v6697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11501.865467 secs 2024-04-26T15:02:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:43 smithi096 ceph-mon[32753]: pgmap v6697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11501.865467 secs 2024-04-26T15:02:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:43 smithi194 ceph-mon[33017]: pgmap v6697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11501.865467 secs 2024-04-26T15:02:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11504.651983 secs 2024-04-26T15:02:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11504.651983 secs 2024-04-26T15:02:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11504.651983 secs 2024-04-26T15:02:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:45 smithi096 ceph-mon[32753]: pgmap v6698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:45.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:45 smithi194 ceph-mon[33017]: pgmap v6698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:45 smithi033 ceph-mon[27845]: pgmap v6698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:47 smithi194 ceph-mon[33017]: pgmap v6699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:47 smithi033 ceph-mon[27845]: pgmap v6699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:47 smithi096 ceph-mon[32753]: pgmap v6699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11506.865549 secs 2024-04-26T15:02:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11506.865549 secs 2024-04-26T15:02:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11506.865549 secs 2024-04-26T15:02:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:49 smithi194 ceph-mon[33017]: pgmap v6700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11509.652129 secs 2024-04-26T15:02:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:49 smithi033 ceph-mon[27845]: pgmap v6700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11509.652129 secs 2024-04-26T15:02:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:49 smithi096 ceph-mon[32753]: pgmap v6700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11509.652129 secs 2024-04-26T15:02:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:51 smithi194 ceph-mon[33017]: pgmap v6701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:51 smithi033 ceph-mon[27845]: pgmap v6701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:51 smithi096 ceph-mon[32753]: pgmap v6701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:02:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:53 smithi194 ceph-mon[33017]: pgmap v6702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11511.865630 secs 2024-04-26T15:02:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:53 smithi033 ceph-mon[27845]: pgmap v6702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11511.865630 secs 2024-04-26T15:02:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:53 smithi096 ceph-mon[32753]: pgmap v6702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11511.865630 secs 2024-04-26T15:02:54.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11514.652155 secs 2024-04-26T15:02:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11514.652155 secs 2024-04-26T15:02:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11514.652155 secs 2024-04-26T15:02:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:55 smithi194 ceph-mon[33017]: pgmap v6703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:02:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:55 smithi033 ceph-mon[27845]: pgmap v6703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:02:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:55 smithi096 ceph-mon[32753]: pgmap v6703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:02:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:02:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:56.189+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:56.189+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:02:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:02:56.189+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:02:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:02:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:02:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:02:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:02:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:02:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:02:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:02:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:02:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:02:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:57 smithi194 ceph-mon[33017]: pgmap v6704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:02:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:57 smithi194 ceph-mon[33017]: pgmap v6705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:02:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:57 smithi033 ceph-mon[27845]: pgmap v6704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:02:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:57 smithi033 ceph-mon[27845]: pgmap v6705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:02:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:57 smithi096 ceph-mon[32753]: pgmap v6704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:02:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:57 smithi096 ceph-mon[32753]: pgmap v6705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:02:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11516.865713 secs 2024-04-26T15:02:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11516.865713 secs 2024-04-26T15:02:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11516.865713 secs 2024-04-26T15:02:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:59 smithi194 ceph-mon[33017]: pgmap v6706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:02:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11519.652273 secs 2024-04-26T15:02:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:59 smithi033 ceph-mon[27845]: pgmap v6706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:02:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11519.652273 secs 2024-04-26T15:02:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:59 smithi096 ceph-mon[32753]: pgmap v6706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:02:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11519.652273 secs 2024-04-26T15:03:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:01 smithi194 ceph-mon[33017]: pgmap v6707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:01 smithi033 ceph-mon[27845]: pgmap v6707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:01 smithi096 ceph-mon[32753]: pgmap v6707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:03 smithi194 ceph-mon[33017]: pgmap v6708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:03 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 11521.865798 secs 2024-04-26T15:03:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:03 smithi194 ceph-mon[33017]: slow request 11521.865797 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T15:03:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:03 smithi033 ceph-mon[27845]: pgmap v6708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:03 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 11521.865798 secs 2024-04-26T15:03:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:03 smithi033 ceph-mon[27845]: slow request 11521.865797 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T15:03:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:03 smithi096 ceph-mon[32753]: pgmap v6708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:03 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 11521.865798 secs 2024-04-26T15:03:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:03 smithi096 ceph-mon[32753]: slow request 11521.865797 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T15:03:04.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:04 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 11524.652416 secs 2024-04-26T15:03:04.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:04 smithi033 ceph-mon[27845]: slow request 11524.652416 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T15:03:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:04 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 11524.652416 secs 2024-04-26T15:03:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:04 smithi194 ceph-mon[33017]: slow request 11524.652416 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T15:03:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:04 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 11524.652416 secs 2024-04-26T15:03:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:04 smithi096 ceph-mon[32753]: slow request 11524.652416 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T15:03:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:05 smithi194 ceph-mon[33017]: pgmap v6709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:05 smithi033 ceph-mon[27845]: pgmap v6709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:05 smithi096 ceph-mon[32753]: pgmap v6709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:07 smithi194 ceph-mon[33017]: pgmap v6710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:03:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:07 smithi033 ceph-mon[27845]: pgmap v6710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:03:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:07 smithi096 ceph-mon[32753]: pgmap v6710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:03:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11526.865878 secs 2024-04-26T15:03:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11526.865878 secs 2024-04-26T15:03:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11526.865878 secs 2024-04-26T15:03:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:09 smithi194 ceph-mon[33017]: pgmap v6711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11529.652532 secs 2024-04-26T15:03:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:09 smithi033 ceph-mon[27845]: pgmap v6711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11529.652532 secs 2024-04-26T15:03:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:09 smithi096 ceph-mon[32753]: pgmap v6711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11529.652532 secs 2024-04-26T15:03:11.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:11.205+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:11.205+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:11.205+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:03:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:11 smithi194 ceph-mon[33017]: pgmap v6712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:11 smithi033 ceph-mon[27845]: pgmap v6712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:11 smithi096 ceph-mon[32753]: pgmap v6712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:13 smithi194 ceph-mon[33017]: pgmap v6713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11531.865962 secs 2024-04-26T15:03:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:13 smithi033 ceph-mon[27845]: pgmap v6713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11531.865962 secs 2024-04-26T15:03:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:13 smithi096 ceph-mon[32753]: pgmap v6713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11531.865962 secs 2024-04-26T15:03:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11534.652651 secs 2024-04-26T15:03:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11534.652651 secs 2024-04-26T15:03:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11534.652651 secs 2024-04-26T15:03:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:15 smithi194 ceph-mon[33017]: pgmap v6714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:15 smithi033 ceph-mon[27845]: pgmap v6714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:15 smithi096 ceph-mon[32753]: pgmap v6714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:17 smithi194 ceph-mon[33017]: pgmap v6715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:17 smithi033 ceph-mon[27845]: pgmap v6715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:17 smithi096 ceph-mon[32753]: pgmap v6715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11536.866043 secs 2024-04-26T15:03:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11536.866043 secs 2024-04-26T15:03:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11536.866043 secs 2024-04-26T15:03:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:19 smithi194 ceph-mon[33017]: pgmap v6716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11539.652775 secs 2024-04-26T15:03:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:19 smithi033 ceph-mon[27845]: pgmap v6716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11539.652775 secs 2024-04-26T15:03:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:19 smithi096 ceph-mon[32753]: pgmap v6716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11539.652775 secs 2024-04-26T15:03:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:21 smithi194 ceph-mon[33017]: pgmap v6717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:21 smithi033 ceph-mon[27845]: pgmap v6717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:21 smithi096 ceph-mon[32753]: pgmap v6717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:23 smithi194 ceph-mon[33017]: pgmap v6718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11541.866125 secs 2024-04-26T15:03:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:23 smithi033 ceph-mon[27845]: pgmap v6718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11541.866125 secs 2024-04-26T15:03:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:23 smithi096 ceph-mon[32753]: pgmap v6718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11541.866125 secs 2024-04-26T15:03:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11544.652895 secs 2024-04-26T15:03:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11544.652895 secs 2024-04-26T15:03:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11544.652895 secs 2024-04-26T15:03:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:25 smithi033 ceph-mon[27845]: pgmap v6719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:25 smithi096 ceph-mon[32753]: pgmap v6719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:25 smithi194 ceph-mon[33017]: pgmap v6719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:26.212+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:26.212+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:26.212+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:03:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:27 smithi033 ceph-mon[27845]: pgmap v6720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:27 smithi096 ceph-mon[32753]: pgmap v6720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:27 smithi194 ceph-mon[33017]: pgmap v6720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11546.866186 secs 2024-04-26T15:03:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11546.866186 secs 2024-04-26T15:03:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11546.866186 secs 2024-04-26T15:03:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:29 smithi033 ceph-mon[27845]: pgmap v6721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11549.653021 secs 2024-04-26T15:03:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:29 smithi096 ceph-mon[32753]: pgmap v6721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11549.653021 secs 2024-04-26T15:03:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:29 smithi194 ceph-mon[33017]: pgmap v6721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11549.653021 secs 2024-04-26T15:03:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:31 smithi033 ceph-mon[27845]: pgmap v6722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:31 smithi096 ceph-mon[32753]: pgmap v6722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:31 smithi194 ceph-mon[33017]: pgmap v6722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:33 smithi033 ceph-mon[27845]: pgmap v6723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11551.866246 secs 2024-04-26T15:03:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:33 smithi096 ceph-mon[32753]: pgmap v6723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11551.866246 secs 2024-04-26T15:03:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:33 smithi194 ceph-mon[33017]: pgmap v6723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11551.866246 secs 2024-04-26T15:03:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11554.653171 secs 2024-04-26T15:03:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11554.653171 secs 2024-04-26T15:03:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11554.653171 secs 2024-04-26T15:03:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:35 smithi033 ceph-mon[27845]: pgmap v6724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:35 smithi096 ceph-mon[32753]: pgmap v6724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:35 smithi194 ceph-mon[33017]: pgmap v6724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:37 smithi033 ceph-mon[27845]: pgmap v6725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:37 smithi096 ceph-mon[32753]: pgmap v6725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:37 smithi194 ceph-mon[33017]: pgmap v6725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11556.866329 secs 2024-04-26T15:03:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11556.866329 secs 2024-04-26T15:03:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11556.866329 secs 2024-04-26T15:03:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:39 smithi033 ceph-mon[27845]: pgmap v6726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11559.653296 secs 2024-04-26T15:03:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:39 smithi096 ceph-mon[32753]: pgmap v6726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11559.653296 secs 2024-04-26T15:03:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:39 smithi194 ceph-mon[33017]: pgmap v6726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11559.653296 secs 2024-04-26T15:03:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:41.222+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:41.223+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:41.223+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:03:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:41 smithi096 ceph-mon[32753]: pgmap v6727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:41 smithi194 ceph-mon[33017]: pgmap v6727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:41 smithi033 ceph-mon[27845]: pgmap v6727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:43 smithi194 ceph-mon[33017]: pgmap v6728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11561.866416 secs 2024-04-26T15:03:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:43 smithi033 ceph-mon[27845]: pgmap v6728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11561.866416 secs 2024-04-26T15:03:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:43 smithi096 ceph-mon[32753]: pgmap v6728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11561.866416 secs 2024-04-26T15:03:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11564.653403 secs 2024-04-26T15:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11564.653403 secs 2024-04-26T15:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11564.653403 secs 2024-04-26T15:03:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:45 smithi194 ceph-mon[33017]: pgmap v6729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:45 smithi033 ceph-mon[27845]: pgmap v6729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:45 smithi096 ceph-mon[32753]: pgmap v6729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:47 smithi194 ceph-mon[33017]: pgmap v6730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:47 smithi033 ceph-mon[27845]: pgmap v6730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:47 smithi096 ceph-mon[32753]: pgmap v6730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11566.866498 secs 2024-04-26T15:03:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11566.866498 secs 2024-04-26T15:03:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11566.866498 secs 2024-04-26T15:03:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:49 smithi194 ceph-mon[33017]: pgmap v6731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11569.653520 secs 2024-04-26T15:03:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:49 smithi033 ceph-mon[27845]: pgmap v6731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11569.653520 secs 2024-04-26T15:03:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:49 smithi096 ceph-mon[32753]: pgmap v6731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11569.653520 secs 2024-04-26T15:03:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:51 smithi194 ceph-mon[33017]: pgmap v6732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:51 smithi033 ceph-mon[27845]: pgmap v6732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:51 smithi096 ceph-mon[32753]: pgmap v6732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:53 smithi194 ceph-mon[33017]: pgmap v6733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11571.866559 secs 2024-04-26T15:03:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:53 smithi033 ceph-mon[27845]: pgmap v6733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11571.866559 secs 2024-04-26T15:03:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:53 smithi096 ceph-mon[32753]: pgmap v6733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11571.866559 secs 2024-04-26T15:03:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11574.653642 secs 2024-04-26T15:03:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11574.653642 secs 2024-04-26T15:03:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11574.653642 secs 2024-04-26T15:03:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:55 smithi194 ceph-mon[33017]: pgmap v6734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:55 smithi033 ceph-mon[27845]: pgmap v6734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:55 smithi096 ceph-mon[32753]: pgmap v6734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:56.223+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:56.223+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:03:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:03:56.223+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:03:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:03:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:03:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:03:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:57 smithi194 ceph-mon[33017]: pgmap v6735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:03:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:03:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:03:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:57 smithi033 ceph-mon[27845]: pgmap v6735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:03:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:03:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:03:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:57 smithi096 ceph-mon[32753]: pgmap v6735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:03:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:03:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:03:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:03:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:58 smithi194 ceph-mon[33017]: pgmap v6736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:58 smithi194 ceph-mon[33017]: pgmap v6737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11576.866643 secs 2024-04-26T15:03:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:58 smithi033 ceph-mon[27845]: pgmap v6736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:58 smithi033 ceph-mon[27845]: pgmap v6737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11576.866643 secs 2024-04-26T15:03:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:58 smithi096 ceph-mon[32753]: pgmap v6736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:03:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:58 smithi096 ceph-mon[32753]: pgmap v6737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:03:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11576.866643 secs 2024-04-26T15:03:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 11579.653705 secs 2024-04-26T15:03:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:03:59 smithi194 ceph-mon[33017]: slow request 11521.381494 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T15:03:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 11579.653705 secs 2024-04-26T15:03:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:03:59 smithi033 ceph-mon[27845]: slow request 11521.381494 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T15:03:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 11579.653705 secs 2024-04-26T15:03:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:03:59 smithi096 ceph-mon[32753]: slow request 11521.381494 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T15:04:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:00 smithi194 ceph-mon[33017]: pgmap v6738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:00 smithi033 ceph-mon[27845]: pgmap v6738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:00 smithi096 ceph-mon[32753]: pgmap v6738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:01 smithi194 ceph-mon[33017]: pgmap v6739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:01 smithi033 ceph-mon[27845]: pgmap v6739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:01 smithi096 ceph-mon[32753]: pgmap v6739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11581.866724 secs 2024-04-26T15:04:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11581.866724 secs 2024-04-26T15:04:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11581.866724 secs 2024-04-26T15:04:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:03 smithi033 ceph-mon[27845]: pgmap v6740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:03 smithi194 ceph-mon[33017]: pgmap v6740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:03 smithi096 ceph-mon[32753]: pgmap v6740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11584.653827 secs 2024-04-26T15:04:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11584.653827 secs 2024-04-26T15:04:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11584.653827 secs 2024-04-26T15:04:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:05 smithi194 ceph-mon[33017]: pgmap v6741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:05 smithi033 ceph-mon[27845]: pgmap v6741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:05 smithi096 ceph-mon[32753]: pgmap v6741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:04:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:08 smithi033 ceph-mon[27845]: pgmap v6742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:04:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11586.866808 secs 2024-04-26T15:04:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:08 smithi096 ceph-mon[32753]: pgmap v6742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:04:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11586.866808 secs 2024-04-26T15:04:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:08 smithi194 ceph-mon[33017]: pgmap v6742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:04:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11586.866808 secs 2024-04-26T15:04:09.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11589.653947 secs 2024-04-26T15:04:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11589.653947 secs 2024-04-26T15:04:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11589.653947 secs 2024-04-26T15:04:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:10 smithi033 ceph-mon[27845]: pgmap v6743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:10 smithi096 ceph-mon[32753]: pgmap v6743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:10 smithi194 ceph-mon[33017]: pgmap v6743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:11.233+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:11.233+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:11.233+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:04:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:12 smithi033 ceph-mon[27845]: pgmap v6744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:12 smithi096 ceph-mon[32753]: pgmap v6744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:12 smithi194 ceph-mon[33017]: pgmap v6744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11591.866867 secs 2024-04-26T15:04:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11591.866867 secs 2024-04-26T15:04:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11591.866867 secs 2024-04-26T15:04:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11594.654073 secs 2024-04-26T15:04:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:14 smithi033 ceph-mon[27845]: pgmap v6745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11594.654073 secs 2024-04-26T15:04:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:14 smithi096 ceph-mon[32753]: pgmap v6745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11594.654073 secs 2024-04-26T15:04:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:14 smithi194 ceph-mon[33017]: pgmap v6745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:16 smithi033 ceph-mon[27845]: pgmap v6746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:16 smithi096 ceph-mon[32753]: pgmap v6746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:16 smithi194 ceph-mon[33017]: pgmap v6746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:18 smithi033 ceph-mon[27845]: pgmap v6747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11596.866957 secs 2024-04-26T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:18 smithi096 ceph-mon[32753]: pgmap v6747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11596.866957 secs 2024-04-26T15:04:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:18 smithi194 ceph-mon[33017]: pgmap v6747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11596.866957 secs 2024-04-26T15:04:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11599.654213 secs 2024-04-26T15:04:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11599.654213 secs 2024-04-26T15:04:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11599.654213 secs 2024-04-26T15:04:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:20 smithi194 ceph-mon[33017]: pgmap v6748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:20 smithi033 ceph-mon[27845]: pgmap v6748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:20 smithi096 ceph-mon[32753]: pgmap v6748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:22 smithi194 ceph-mon[33017]: pgmap v6749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:22 smithi033 ceph-mon[27845]: pgmap v6749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:22 smithi096 ceph-mon[32753]: pgmap v6749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11601.867019 secs 2024-04-26T15:04:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11601.867019 secs 2024-04-26T15:04:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11601.867019 secs 2024-04-26T15:04:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11604.654275 secs 2024-04-26T15:04:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:24 smithi096 ceph-mon[32753]: pgmap v6750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11604.654275 secs 2024-04-26T15:04:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:24 smithi194 ceph-mon[33017]: pgmap v6750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11604.654275 secs 2024-04-26T15:04:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:24 smithi033 ceph-mon[27845]: pgmap v6750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:26.243+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:26.244+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:04:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:26.244+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:26 smithi194 ceph-mon[33017]: pgmap v6751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:26 smithi033 ceph-mon[27845]: pgmap v6751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:26 smithi096 ceph-mon[32753]: pgmap v6751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:28 smithi194 ceph-mon[33017]: pgmap v6752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11606.867079 secs 2024-04-26T15:04:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:28 smithi033 ceph-mon[27845]: pgmap v6752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11606.867079 secs 2024-04-26T15:04:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:28 smithi096 ceph-mon[32753]: pgmap v6752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11606.867079 secs 2024-04-26T15:04:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11609.654363 secs 2024-04-26T15:04:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11609.654363 secs 2024-04-26T15:04:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11609.654363 secs 2024-04-26T15:04:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:30 smithi194 ceph-mon[33017]: pgmap v6753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:30 smithi033 ceph-mon[27845]: pgmap v6753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:30 smithi096 ceph-mon[32753]: pgmap v6753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:32 smithi194 ceph-mon[33017]: pgmap v6754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:32 smithi033 ceph-mon[27845]: pgmap v6754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:32 smithi096 ceph-mon[32753]: pgmap v6754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11611.867162 secs 2024-04-26T15:04:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11611.867162 secs 2024-04-26T15:04:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11611.867162 secs 2024-04-26T15:04:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11614.654479 secs 2024-04-26T15:04:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:34 smithi033 ceph-mon[27845]: pgmap v6755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11614.654479 secs 2024-04-26T15:04:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:34 smithi194 ceph-mon[33017]: pgmap v6755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11614.654479 secs 2024-04-26T15:04:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:34 smithi096 ceph-mon[32753]: pgmap v6755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:36 smithi194 ceph-mon[33017]: pgmap v6756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:36.912 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:36 smithi033 ceph-mon[27845]: pgmap v6756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:36 smithi096 ceph-mon[32753]: pgmap v6756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:38 smithi194 ceph-mon[33017]: pgmap v6757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11616.867226 secs 2024-04-26T15:04:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:38 smithi033 ceph-mon[27845]: pgmap v6757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11616.867226 secs 2024-04-26T15:04:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:38 smithi096 ceph-mon[32753]: pgmap v6757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11616.867226 secs 2024-04-26T15:04:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11619.654605 secs 2024-04-26T15:04:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11619.654605 secs 2024-04-26T15:04:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11619.654605 secs 2024-04-26T15:04:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:40 smithi194 ceph-mon[33017]: pgmap v6758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:40 smithi033 ceph-mon[27845]: pgmap v6758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:40 smithi096 ceph-mon[32753]: pgmap v6758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:41.259+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:41.259+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:04:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:41.259+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:42 smithi194 ceph-mon[33017]: pgmap v6759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:42 smithi033 ceph-mon[27845]: pgmap v6759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:42 smithi096 ceph-mon[32753]: pgmap v6759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11621.867290 secs 2024-04-26T15:04:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11621.867290 secs 2024-04-26T15:04:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11621.867290 secs 2024-04-26T15:04:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11624.654729 secs 2024-04-26T15:04:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:44 smithi194 ceph-mon[33017]: pgmap v6760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11624.654729 secs 2024-04-26T15:04:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:44 smithi033 ceph-mon[27845]: pgmap v6760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11624.654729 secs 2024-04-26T15:04:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:44 smithi096 ceph-mon[32753]: pgmap v6760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:45 smithi194 ceph-mon[33017]: pgmap v6761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:45 smithi033 ceph-mon[27845]: pgmap v6761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:45 smithi096 ceph-mon[32753]: pgmap v6761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:48 smithi033 ceph-mon[27845]: pgmap v6762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11626.867374 secs 2024-04-26T15:04:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:48 smithi096 ceph-mon[32753]: pgmap v6762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11626.867374 secs 2024-04-26T15:04:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:48 smithi194 ceph-mon[33017]: pgmap v6762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11626.867374 secs 2024-04-26T15:04:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11629.654871 secs 2024-04-26T15:04:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11629.654871 secs 2024-04-26T15:04:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11629.654871 secs 2024-04-26T15:04:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:50 smithi033 ceph-mon[27845]: pgmap v6763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:50 smithi096 ceph-mon[32753]: pgmap v6763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:50 smithi194 ceph-mon[33017]: pgmap v6763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:52 smithi033 ceph-mon[27845]: pgmap v6764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:52 smithi096 ceph-mon[32753]: pgmap v6764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:52 smithi194 ceph-mon[33017]: pgmap v6764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11631.867435 secs 2024-04-26T15:04:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11631.867435 secs 2024-04-26T15:04:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11631.867435 secs 2024-04-26T15:04:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11634.654993 secs 2024-04-26T15:04:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:54 smithi096 ceph-mon[32753]: pgmap v6765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11634.654993 secs 2024-04-26T15:04:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:54 smithi194 ceph-mon[33017]: pgmap v6765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11634.654993 secs 2024-04-26T15:04:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:54 smithi033 ceph-mon[27845]: pgmap v6765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:56.261+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:56.261+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:04:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:04:56.261+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:04:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:56 smithi194 ceph-mon[33017]: pgmap v6766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:56 smithi033 ceph-mon[27845]: pgmap v6766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:56 smithi096 ceph-mon[32753]: pgmap v6766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:04:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:58 smithi194 ceph-mon[33017]: pgmap v6767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11636.867493 secs 2024-04-26T15:04:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:04:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:58 smithi033 ceph-mon[27845]: pgmap v6767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11636.867493 secs 2024-04-26T15:04:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:58 smithi096 ceph-mon[32753]: pgmap v6767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11636.867493 secs 2024-04-26T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:04:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11639.655134 secs 2024-04-26T15:04:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:04:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:04:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:04:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:04:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11639.655134 secs 2024-04-26T15:04:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:04:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:04:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:04:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11639.655134 secs 2024-04-26T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:04:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:05:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:00 smithi194 ceph-mon[33017]: pgmap v6768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:00 smithi194 ceph-mon[33017]: pgmap v6769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:05:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:00 smithi033 ceph-mon[27845]: pgmap v6768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:00 smithi033 ceph-mon[27845]: pgmap v6769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:05:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:00 smithi096 ceph-mon[32753]: pgmap v6768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:00 smithi096 ceph-mon[32753]: pgmap v6769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:05:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:02 smithi194 ceph-mon[33017]: pgmap v6770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:02 smithi033 ceph-mon[27845]: pgmap v6770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:02 smithi096 ceph-mon[32753]: pgmap v6770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11641.867551 secs 2024-04-26T15:05:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11641.867551 secs 2024-04-26T15:05:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11641.867551 secs 2024-04-26T15:05:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:04 smithi033 ceph-mon[27845]: pgmap v6771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11644.655274 secs 2024-04-26T15:05:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:04 smithi194 ceph-mon[33017]: pgmap v6771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11644.655274 secs 2024-04-26T15:05:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:04 smithi096 ceph-mon[32753]: pgmap v6771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11644.655274 secs 2024-04-26T15:05:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:06 smithi194 ceph-mon[33017]: pgmap v6772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:06 smithi033 ceph-mon[27845]: pgmap v6772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:06 smithi096 ceph-mon[32753]: pgmap v6772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:08 smithi194 ceph-mon[33017]: pgmap v6773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11646.867594 secs 2024-04-26T15:05:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:08 smithi033 ceph-mon[27845]: pgmap v6773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11646.867594 secs 2024-04-26T15:05:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:08 smithi096 ceph-mon[32753]: pgmap v6773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:05:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11646.867594 secs 2024-04-26T15:05:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11649.655358 secs 2024-04-26T15:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11649.655358 secs 2024-04-26T15:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11649.655358 secs 2024-04-26T15:05:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:10 smithi194 ceph-mon[33017]: pgmap v6774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:05:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:10 smithi033 ceph-mon[27845]: pgmap v6774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:05:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:10 smithi096 ceph-mon[32753]: pgmap v6774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:05:11.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:11.277+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:11.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:11.277+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:11.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:11.277+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:05:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:12 smithi194 ceph-mon[33017]: pgmap v6775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:12 smithi033 ceph-mon[27845]: pgmap v6775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:12 smithi096 ceph-mon[32753]: pgmap v6775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11651.867680 secs 2024-04-26T15:05:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11651.867680 secs 2024-04-26T15:05:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11651.867680 secs 2024-04-26T15:05:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:14 smithi194 ceph-mon[33017]: pgmap v6776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11654.655477 secs 2024-04-26T15:05:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:14 smithi033 ceph-mon[27845]: pgmap v6776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11654.655477 secs 2024-04-26T15:05:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:14 smithi096 ceph-mon[32753]: pgmap v6776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11654.655477 secs 2024-04-26T15:05:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:16 smithi194 ceph-mon[33017]: pgmap v6777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:16 smithi033 ceph-mon[27845]: pgmap v6777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:16 smithi096 ceph-mon[32753]: pgmap v6777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:18 smithi194 ceph-mon[33017]: pgmap v6778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11656.867763 secs 2024-04-26T15:05:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:18 smithi033 ceph-mon[27845]: pgmap v6778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11656.867763 secs 2024-04-26T15:05:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:18 smithi096 ceph-mon[32753]: pgmap v6778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11656.867763 secs 2024-04-26T15:05:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11659.655602 secs 2024-04-26T15:05:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11659.655602 secs 2024-04-26T15:05:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11659.655602 secs 2024-04-26T15:05:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:20 smithi194 ceph-mon[33017]: pgmap v6779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:20 smithi033 ceph-mon[27845]: pgmap v6779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:20 smithi096 ceph-mon[32753]: pgmap v6779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:21 smithi194 ceph-mon[33017]: pgmap v6780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:21 smithi033 ceph-mon[27845]: pgmap v6780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:21 smithi096 ceph-mon[32753]: pgmap v6780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11661.867845 secs 2024-04-26T15:05:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11661.867845 secs 2024-04-26T15:05:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11661.867845 secs 2024-04-26T15:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:23.776+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006610 con 0x564385821900 2024-04-26T15:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:23.776+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:23.777+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:23.777+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006610 con 0x7fef00077780 2024-04-26T15:05:24.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:23 smithi096 ceph-mon[32753]: pgmap v6781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:23 smithi194 ceph-mon[33017]: pgmap v6781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:23 smithi033 ceph-mon[27845]: pgmap v6781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11664.655723 secs 2024-04-26T15:05:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11664.655723 secs 2024-04-26T15:05:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11664.655723 secs 2024-04-26T15:05:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:25 smithi194 ceph-mon[33017]: pgmap v6782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:25 smithi033 ceph-mon[27845]: pgmap v6782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:25 smithi096 ceph-mon[32753]: pgmap v6782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:26.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:26.285+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:26.285+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:26.285+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:05:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:28 smithi033 ceph-mon[27845]: pgmap v6783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11666.867930 secs 2024-04-26T15:05:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:28 smithi096 ceph-mon[32753]: pgmap v6783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11666.867930 secs 2024-04-26T15:05:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:28 smithi194 ceph-mon[33017]: pgmap v6783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11666.867930 secs 2024-04-26T15:05:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11669.655863 secs 2024-04-26T15:05:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11669.655863 secs 2024-04-26T15:05:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11669.655863 secs 2024-04-26T15:05:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:30 smithi033 ceph-mon[27845]: pgmap v6784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:30 smithi096 ceph-mon[32753]: pgmap v6784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:30 smithi194 ceph-mon[33017]: pgmap v6784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:32 smithi033 ceph-mon[27845]: pgmap v6785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:32 smithi096 ceph-mon[32753]: pgmap v6785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:32 smithi194 ceph-mon[33017]: pgmap v6785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11671.867997 secs 2024-04-26T15:05:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11671.867997 secs 2024-04-26T15:05:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11671.867997 secs 2024-04-26T15:05:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:34 smithi033 ceph-mon[27845]: pgmap v6786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11674.656010 secs 2024-04-26T15:05:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:34 smithi096 ceph-mon[32753]: pgmap v6786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11674.656010 secs 2024-04-26T15:05:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:34 smithi194 ceph-mon[33017]: pgmap v6786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11674.656010 secs 2024-04-26T15:05:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:36 smithi033 ceph-mon[27845]: pgmap v6787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:36 smithi096 ceph-mon[32753]: pgmap v6787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:36 smithi194 ceph-mon[33017]: pgmap v6787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:38 smithi033 ceph-mon[27845]: pgmap v6788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11676.868080 secs 2024-04-26T15:05:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:38 smithi096 ceph-mon[32753]: pgmap v6788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11676.868080 secs 2024-04-26T15:05:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:38 smithi194 ceph-mon[33017]: pgmap v6788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11676.868080 secs 2024-04-26T15:05:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11679.656152 secs 2024-04-26T15:05:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11679.656152 secs 2024-04-26T15:05:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11679.656152 secs 2024-04-26T15:05:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:40 smithi033 ceph-mon[27845]: pgmap v6789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:40 smithi096 ceph-mon[32753]: pgmap v6789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:40 smithi194 ceph-mon[33017]: pgmap v6789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:41.295+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:41.295+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:41.295+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:05:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:42 smithi096 ceph-mon[32753]: pgmap v6790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:42 smithi194 ceph-mon[33017]: pgmap v6790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:42 smithi033 ceph-mon[27845]: pgmap v6790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11681.868163 secs 2024-04-26T15:05:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11681.868163 secs 2024-04-26T15:05:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11681.868163 secs 2024-04-26T15:05:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:44 smithi194 ceph-mon[33017]: pgmap v6791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11684.656245 secs 2024-04-26T15:05:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:44 smithi033 ceph-mon[27845]: pgmap v6791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11684.656245 secs 2024-04-26T15:05:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:44 smithi096 ceph-mon[32753]: pgmap v6791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11684.656245 secs 2024-04-26T15:05:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:46 smithi194 ceph-mon[33017]: pgmap v6792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:46.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:46 smithi033 ceph-mon[27845]: pgmap v6792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:46 smithi096 ceph-mon[32753]: pgmap v6792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:48 smithi194 ceph-mon[33017]: pgmap v6793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11686.868247 secs 2024-04-26T15:05:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:48 smithi033 ceph-mon[27845]: pgmap v6793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11686.868247 secs 2024-04-26T15:05:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:48 smithi096 ceph-mon[32753]: pgmap v6793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11686.868247 secs 2024-04-26T15:05:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11689.656371 secs 2024-04-26T15:05:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11689.656371 secs 2024-04-26T15:05:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11689.656371 secs 2024-04-26T15:05:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:50 smithi194 ceph-mon[33017]: pgmap v6794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:50 smithi033 ceph-mon[27845]: pgmap v6794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:50 smithi096 ceph-mon[32753]: pgmap v6794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:52 smithi194 ceph-mon[33017]: pgmap v6795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:52 smithi033 ceph-mon[27845]: pgmap v6795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:52 smithi096 ceph-mon[32753]: pgmap v6795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:05:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11691.868330 secs 2024-04-26T15:05:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11691.868330 secs 2024-04-26T15:05:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11691.868330 secs 2024-04-26T15:05:54.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:54 smithi096 ceph-mon[32753]: pgmap v6796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:54.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11694.656431 secs 2024-04-26T15:05:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:54 smithi194 ceph-mon[33017]: pgmap v6796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11694.656431 secs 2024-04-26T15:05:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:54 smithi033 ceph-mon[27845]: pgmap v6796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:05:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11694.656431 secs 2024-04-26T15:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:56.310+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:56.310+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:05:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:05:56.310+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:05:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:56 smithi194 ceph-mon[33017]: pgmap v6797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:05:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:56 smithi033 ceph-mon[27845]: pgmap v6797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:05:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:56 smithi096 ceph-mon[32753]: pgmap v6797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:05:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:58 smithi194 ceph-mon[33017]: pgmap v6798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:05:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11696.868409 secs 2024-04-26T15:05:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:58 smithi033 ceph-mon[27845]: pgmap v6798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:05:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11696.868409 secs 2024-04-26T15:05:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:58 smithi096 ceph-mon[32753]: pgmap v6798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:05:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11696.868409 secs 2024-04-26T15:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11699.656549 secs 2024-04-26T15:05:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11699.656549 secs 2024-04-26T15:05:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11699.656549 secs 2024-04-26T15:06:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:00 smithi194 ceph-mon[33017]: pgmap v6799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:06:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:06:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:00 smithi033 ceph-mon[27845]: pgmap v6799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:06:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:06:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:00 smithi096 ceph-mon[32753]: pgmap v6799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:06:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:06:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:06:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:06:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:06:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:06:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:06:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:06:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:06:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:06:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:06:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:02 smithi194 ceph-mon[33017]: pgmap v6800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T15:06:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:02 smithi194 ceph-mon[33017]: pgmap v6801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:06:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:02 smithi033 ceph-mon[27845]: pgmap v6800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T15:06:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:02 smithi033 ceph-mon[27845]: pgmap v6801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:06:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:02 smithi096 ceph-mon[32753]: pgmap v6800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T15:06:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:02 smithi096 ceph-mon[32753]: pgmap v6801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:06:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11701.868471 secs 2024-04-26T15:06:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11701.868471 secs 2024-04-26T15:06:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11701.868471 secs 2024-04-26T15:06:04.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:04 smithi033 ceph-mon[27845]: pgmap v6802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:06:04.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11704.656672 secs 2024-04-26T15:06:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:04 smithi194 ceph-mon[33017]: pgmap v6802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:06:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11704.656672 secs 2024-04-26T15:06:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:04 smithi096 ceph-mon[32753]: pgmap v6802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:06:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11704.656672 secs 2024-04-26T15:06:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:06 smithi194 ceph-mon[33017]: pgmap v6803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:06 smithi033 ceph-mon[27845]: pgmap v6803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:06 smithi096 ceph-mon[32753]: pgmap v6803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:08 smithi194 ceph-mon[33017]: pgmap v6804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11706.868554 secs 2024-04-26T15:06:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:08 smithi033 ceph-mon[27845]: pgmap v6804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11706.868554 secs 2024-04-26T15:06:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:08 smithi096 ceph-mon[32753]: pgmap v6804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11706.868554 secs 2024-04-26T15:06:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11709.656801 secs 2024-04-26T15:06:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11709.656801 secs 2024-04-26T15:06:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11709.656801 secs 2024-04-26T15:06:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:10 smithi033 ceph-mon[27845]: pgmap v6805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:10 smithi194 ceph-mon[33017]: pgmap v6805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:10 smithi096 ceph-mon[32753]: pgmap v6805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:06:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:11.326+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:11.326+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:11.326+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:06:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:12 smithi194 ceph-mon[33017]: pgmap v6806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:06:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:12 smithi033 ceph-mon[27845]: pgmap v6806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:06:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:12 smithi096 ceph-mon[32753]: pgmap v6806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:06:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11711.868638 secs 2024-04-26T15:06:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11711.868638 secs 2024-04-26T15:06:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11711.868638 secs 2024-04-26T15:06:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:14 smithi194 ceph-mon[33017]: pgmap v6807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11714.656940 secs 2024-04-26T15:06:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:14 smithi033 ceph-mon[27845]: pgmap v6807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11714.656940 secs 2024-04-26T15:06:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:14 smithi096 ceph-mon[32753]: pgmap v6807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11714.656940 secs 2024-04-26T15:06:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:16 smithi194 ceph-mon[33017]: pgmap v6808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:16 smithi033 ceph-mon[27845]: pgmap v6808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:16 smithi096 ceph-mon[32753]: pgmap v6808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:18 smithi194 ceph-mon[33017]: pgmap v6809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11716.868721 secs 2024-04-26T15:06:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:18 smithi033 ceph-mon[27845]: pgmap v6809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11716.868721 secs 2024-04-26T15:06:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:18 smithi096 ceph-mon[32753]: pgmap v6809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11716.868721 secs 2024-04-26T15:06:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11719.657061 secs 2024-04-26T15:06:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11719.657061 secs 2024-04-26T15:06:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11719.657061 secs 2024-04-26T15:06:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:20 smithi194 ceph-mon[33017]: pgmap v6810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:20 smithi033 ceph-mon[27845]: pgmap v6810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:20 smithi096 ceph-mon[32753]: pgmap v6810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:22 smithi033 ceph-mon[27845]: pgmap v6811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:22 smithi096 ceph-mon[32753]: pgmap v6811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:22 smithi194 ceph-mon[33017]: pgmap v6811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11721.868823 secs 2024-04-26T15:06:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11721.868823 secs 2024-04-26T15:06:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11721.868823 secs 2024-04-26T15:06:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:24 smithi033 ceph-mon[27845]: pgmap v6812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11724.657212 secs 2024-04-26T15:06:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:24 smithi096 ceph-mon[32753]: pgmap v6812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11724.657212 secs 2024-04-26T15:06:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:24 smithi194 ceph-mon[33017]: pgmap v6812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11724.657212 secs 2024-04-26T15:06:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:26.339+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:26.339+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:26.339+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:06:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:26 smithi033 ceph-mon[27845]: pgmap v6813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:26 smithi096 ceph-mon[32753]: pgmap v6813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:26 smithi194 ceph-mon[33017]: pgmap v6813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:28 smithi033 ceph-mon[27845]: pgmap v6814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11726.868908 secs 2024-04-26T15:06:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:28 smithi096 ceph-mon[32753]: pgmap v6814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11726.868908 secs 2024-04-26T15:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:28 smithi194 ceph-mon[33017]: pgmap v6814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11726.868908 secs 2024-04-26T15:06:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11729.657334 secs 2024-04-26T15:06:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11729.657334 secs 2024-04-26T15:06:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11729.657334 secs 2024-04-26T15:06:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:30 smithi033 ceph-mon[27845]: pgmap v6815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:30 smithi096 ceph-mon[32753]: pgmap v6815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:30 smithi194 ceph-mon[33017]: pgmap v6815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:32 smithi033 ceph-mon[27845]: pgmap v6816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:32 smithi096 ceph-mon[32753]: pgmap v6816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:32 smithi194 ceph-mon[33017]: pgmap v6816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11731.868991 secs 2024-04-26T15:06:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11731.868991 secs 2024-04-26T15:06:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11731.868991 secs 2024-04-26T15:06:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:34 smithi033 ceph-mon[27845]: pgmap v6817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11734.657464 secs 2024-04-26T15:06:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:34 smithi096 ceph-mon[32753]: pgmap v6817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11734.657464 secs 2024-04-26T15:06:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:34 smithi194 ceph-mon[33017]: pgmap v6817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11734.657464 secs 2024-04-26T15:06:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:36 smithi033 ceph-mon[27845]: pgmap v6818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:36 smithi096 ceph-mon[32753]: pgmap v6818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:36 smithi194 ceph-mon[33017]: pgmap v6818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:38 smithi033 ceph-mon[27845]: pgmap v6819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11736.869075 secs 2024-04-26T15:06:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:38 smithi096 ceph-mon[32753]: pgmap v6819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11736.869075 secs 2024-04-26T15:06:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:38 smithi194 ceph-mon[33017]: pgmap v6819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11736.869075 secs 2024-04-26T15:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11739.657572 secs 2024-04-26T15:06:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11739.657572 secs 2024-04-26T15:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11739.657572 secs 2024-04-26T15:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:40 smithi194 ceph-mon[33017]: pgmap v6820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:40 smithi033 ceph-mon[27845]: pgmap v6820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:40 smithi096 ceph-mon[32753]: pgmap v6820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:41.354+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:41.355+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:41.355+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:06:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:42 smithi194 ceph-mon[33017]: pgmap v6821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:42 smithi033 ceph-mon[27845]: pgmap v6821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:42 smithi096 ceph-mon[32753]: pgmap v6821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11741.869159 secs 2024-04-26T15:06:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11741.869159 secs 2024-04-26T15:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11741.869159 secs 2024-04-26T15:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:44 smithi194 ceph-mon[33017]: pgmap v6822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11744.657707 secs 2024-04-26T15:06:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:44 smithi033 ceph-mon[27845]: pgmap v6822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11744.657707 secs 2024-04-26T15:06:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:44 smithi096 ceph-mon[32753]: pgmap v6822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11744.657707 secs 2024-04-26T15:06:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:46 smithi194 ceph-mon[33017]: pgmap v6823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:46 smithi033 ceph-mon[27845]: pgmap v6823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:46 smithi096 ceph-mon[32753]: pgmap v6823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:48 smithi194 ceph-mon[33017]: pgmap v6824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11746.869243 secs 2024-04-26T15:06:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:48 smithi033 ceph-mon[27845]: pgmap v6824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11746.869243 secs 2024-04-26T15:06:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:48 smithi096 ceph-mon[32753]: pgmap v6824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11746.869243 secs 2024-04-26T15:06:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11749.657829 secs 2024-04-26T15:06:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11749.657829 secs 2024-04-26T15:06:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11749.657829 secs 2024-04-26T15:06:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:50 smithi194 ceph-mon[33017]: pgmap v6825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:50 smithi033 ceph-mon[27845]: pgmap v6825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:50 smithi096 ceph-mon[32753]: pgmap v6825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:52 smithi194 ceph-mon[33017]: pgmap v6826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:52 smithi033 ceph-mon[27845]: pgmap v6826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:52 smithi096 ceph-mon[32753]: pgmap v6826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11751.869328 secs 2024-04-26T15:06:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11751.869328 secs 2024-04-26T15:06:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11751.869328 secs 2024-04-26T15:06:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:54 smithi096 ceph-mon[32753]: pgmap v6827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:54.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11754.657950 secs 2024-04-26T15:06:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:54 smithi194 ceph-mon[33017]: pgmap v6827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11754.657950 secs 2024-04-26T15:06:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:54 smithi033 ceph-mon[27845]: pgmap v6827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11754.657950 secs 2024-04-26T15:06:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:56.368+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:56.368+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:06:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:06:56.368+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:06:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:56 smithi194 ceph-mon[33017]: pgmap v6828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:56 smithi033 ceph-mon[27845]: pgmap v6828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:56 smithi096 ceph-mon[32753]: pgmap v6828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:06:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:58 smithi194 ceph-mon[33017]: pgmap v6829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11756.869412 secs 2024-04-26T15:06:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:58 smithi033 ceph-mon[27845]: pgmap v6829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11756.869412 secs 2024-04-26T15:06:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:58 smithi096 ceph-mon[32753]: pgmap v6829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:06:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11756.869412 secs 2024-04-26T15:06:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11759.658073 secs 2024-04-26T15:06:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11759.658073 secs 2024-04-26T15:06:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11759.658073 secs 2024-04-26T15:07:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:00 smithi194 ceph-mon[33017]: pgmap v6830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:00 smithi033 ceph-mon[27845]: pgmap v6830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:00 smithi096 ceph-mon[32753]: pgmap v6830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:07:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:07:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:07:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:02 smithi194 ceph-mon[33017]: pgmap v6831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:02 smithi033 ceph-mon[27845]: pgmap v6831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:02.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:02.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:02 smithi096 ceph-mon[32753]: pgmap v6831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11761.869472 secs 2024-04-26T15:07:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:07:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:07:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11761.869472 secs 2024-04-26T15:07:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:07:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:07:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11761.869472 secs 2024-04-26T15:07:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:07:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:07:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:07:04.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:04 smithi033 ceph-mon[27845]: pgmap v6832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:04.829 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11764.658199 secs 2024-04-26T15:07:04.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:04 smithi033 ceph-mon[27845]: pgmap v6833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:07:04.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:04 smithi033 ceph-mon[27845]: pgmap v6834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:07:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:04 smithi194 ceph-mon[33017]: pgmap v6832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11764.658199 secs 2024-04-26T15:07:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:04 smithi194 ceph-mon[33017]: pgmap v6833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:07:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:04 smithi194 ceph-mon[33017]: pgmap v6834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:04 smithi096 ceph-mon[32753]: pgmap v6832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11764.658199 secs 2024-04-26T15:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:04 smithi096 ceph-mon[32753]: pgmap v6833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:04 smithi096 ceph-mon[32753]: pgmap v6834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:07:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:06 smithi194 ceph-mon[33017]: pgmap v6835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:06 smithi033 ceph-mon[27845]: pgmap v6835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:06 smithi096 ceph-mon[32753]: pgmap v6835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:08 smithi194 ceph-mon[33017]: pgmap v6836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11766.869556 secs 2024-04-26T15:07:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:08 smithi033 ceph-mon[27845]: pgmap v6836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11766.869556 secs 2024-04-26T15:07:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:08 smithi096 ceph-mon[32753]: pgmap v6836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11766.869556 secs 2024-04-26T15:07:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11769.658319 secs 2024-04-26T15:07:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11769.658319 secs 2024-04-26T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11769.658319 secs 2024-04-26T15:07:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:10 smithi194 ceph-mon[33017]: pgmap v6837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:10 smithi033 ceph-mon[27845]: pgmap v6837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:10 smithi096 ceph-mon[32753]: pgmap v6837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:11.373+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:11.373+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:11.373+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:07:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:11 smithi033 ceph-mon[27845]: pgmap v6838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:11 smithi096 ceph-mon[32753]: pgmap v6838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:11 smithi194 ceph-mon[33017]: pgmap v6838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:07:13.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11771.869642 secs 2024-04-26T15:07:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11771.869642 secs 2024-04-26T15:07:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11771.869642 secs 2024-04-26T15:07:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:13 smithi194 ceph-mon[33017]: pgmap v6839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:07:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:13 smithi033 ceph-mon[27845]: pgmap v6839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:07:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:13 smithi096 ceph-mon[32753]: pgmap v6839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:07:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11774.658439 secs 2024-04-26T15:07:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11774.658439 secs 2024-04-26T15:07:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11774.658439 secs 2024-04-26T15:07:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:15 smithi194 ceph-mon[33017]: pgmap v6840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:15 smithi033 ceph-mon[27845]: pgmap v6840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:15 smithi096 ceph-mon[32753]: pgmap v6840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:18 smithi194 ceph-mon[33017]: pgmap v6841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11776.869724 secs 2024-04-26T15:07:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:18 smithi033 ceph-mon[27845]: pgmap v6841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11776.869724 secs 2024-04-26T15:07:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:18 smithi096 ceph-mon[32753]: pgmap v6841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11776.869724 secs 2024-04-26T15:07:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11779.658570 secs 2024-04-26T15:07:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11779.658570 secs 2024-04-26T15:07:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11779.658570 secs 2024-04-26T15:07:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:20 smithi194 ceph-mon[33017]: pgmap v6842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:20 smithi033 ceph-mon[27845]: pgmap v6842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:20 smithi096 ceph-mon[32753]: pgmap v6842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:22 smithi194 ceph-mon[33017]: pgmap v6843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:22 smithi033 ceph-mon[27845]: pgmap v6843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:22 smithi096 ceph-mon[32753]: pgmap v6843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11781.869786 secs 2024-04-26T15:07:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11781.869786 secs 2024-04-26T15:07:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11781.869786 secs 2024-04-26T15:07:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11784.658692 secs 2024-04-26T15:07:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:24 smithi096 ceph-mon[32753]: pgmap v6844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11784.658692 secs 2024-04-26T15:07:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:24 smithi194 ceph-mon[33017]: pgmap v6844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11784.658692 secs 2024-04-26T15:07:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:24 smithi033 ceph-mon[27845]: pgmap v6844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:26.377+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:26.377+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:26.377+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:07:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:26 smithi194 ceph-mon[33017]: pgmap v6845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:26 smithi033 ceph-mon[27845]: pgmap v6845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:26 smithi096 ceph-mon[32753]: pgmap v6845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:28 smithi194 ceph-mon[33017]: pgmap v6846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11786.869864 secs 2024-04-26T15:07:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:28 smithi033 ceph-mon[27845]: pgmap v6846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11786.869864 secs 2024-04-26T15:07:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:28 smithi096 ceph-mon[32753]: pgmap v6846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11786.869864 secs 2024-04-26T15:07:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11789.658810 secs 2024-04-26T15:07:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11789.658810 secs 2024-04-26T15:07:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11789.658810 secs 2024-04-26T15:07:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:30 smithi194 ceph-mon[33017]: pgmap v6847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:30 smithi033 ceph-mon[27845]: pgmap v6847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:30 smithi096 ceph-mon[32753]: pgmap v6847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:32 smithi194 ceph-mon[33017]: pgmap v6848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:32 smithi033 ceph-mon[27845]: pgmap v6848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:32 smithi096 ceph-mon[32753]: pgmap v6848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11791.869949 secs 2024-04-26T15:07:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11791.869949 secs 2024-04-26T15:07:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11791.869949 secs 2024-04-26T15:07:34.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11794.658951 secs 2024-04-26T15:07:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:34 smithi033 ceph-mon[27845]: pgmap v6849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11794.658951 secs 2024-04-26T15:07:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:34 smithi194 ceph-mon[33017]: pgmap v6849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11794.658951 secs 2024-04-26T15:07:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:34 smithi096 ceph-mon[32753]: pgmap v6849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:36 smithi194 ceph-mon[33017]: pgmap v6850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:36 smithi033 ceph-mon[27845]: pgmap v6850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:36 smithi096 ceph-mon[32753]: pgmap v6850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:38.799+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:07:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:38.799+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:07:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:38.799+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:07:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:38.800+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:07:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:38 smithi194 ceph-mon[33017]: pgmap v6851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11796.870004 secs 2024-04-26T15:07:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:38 smithi033 ceph-mon[27845]: pgmap v6851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11796.870004 secs 2024-04-26T15:07:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:38 smithi096 ceph-mon[32753]: pgmap v6851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11796.870004 secs 2024-04-26T15:07:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11799.659093 secs 2024-04-26T15:07:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11799.659093 secs 2024-04-26T15:07:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11799.659093 secs 2024-04-26T15:07:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:40 smithi194 ceph-mon[33017]: pgmap v6852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:40 smithi033 ceph-mon[27845]: pgmap v6852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:40 smithi096 ceph-mon[32753]: pgmap v6852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:41.393+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:41.393+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:41.393+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:07:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:41 smithi194 ceph-mon[33017]: pgmap v6853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:41 smithi033 ceph-mon[27845]: pgmap v6853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:41 smithi096 ceph-mon[32753]: pgmap v6853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11801.870115 secs 2024-04-26T15:07:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11801.870115 secs 2024-04-26T15:07:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11801.870115 secs 2024-04-26T15:07:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:43 smithi194 ceph-mon[33017]: pgmap v6854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:43 smithi033 ceph-mon[27845]: pgmap v6854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:43 smithi096 ceph-mon[32753]: pgmap v6854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11804.659222 secs 2024-04-26T15:07:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11804.659222 secs 2024-04-26T15:07:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11804.659222 secs 2024-04-26T15:07:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:45 smithi194 ceph-mon[33017]: pgmap v6855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:45 smithi033 ceph-mon[27845]: pgmap v6855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:45 smithi096 ceph-mon[32753]: pgmap v6855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:48 smithi194 ceph-mon[33017]: pgmap v6856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11806.870180 secs 2024-04-26T15:07:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:48 smithi033 ceph-mon[27845]: pgmap v6856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11806.870180 secs 2024-04-26T15:07:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:48 smithi096 ceph-mon[32753]: pgmap v6856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11806.870180 secs 2024-04-26T15:07:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11809.659345 secs 2024-04-26T15:07:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11809.659345 secs 2024-04-26T15:07:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11809.659345 secs 2024-04-26T15:07:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:50 smithi194 ceph-mon[33017]: pgmap v6857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:50 smithi033 ceph-mon[27845]: pgmap v6857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:50 smithi096 ceph-mon[32753]: pgmap v6857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:52 smithi194 ceph-mon[33017]: pgmap v6858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:52 smithi033 ceph-mon[27845]: pgmap v6858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:52 smithi096 ceph-mon[32753]: pgmap v6858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11811.870241 secs 2024-04-26T15:07:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11811.870241 secs 2024-04-26T15:07:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11811.870241 secs 2024-04-26T15:07:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11814.659417 secs 2024-04-26T15:07:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:54 smithi096 ceph-mon[32753]: pgmap v6859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11814.659417 secs 2024-04-26T15:07:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:54 smithi194 ceph-mon[33017]: pgmap v6859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11814.659417 secs 2024-04-26T15:07:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:54 smithi033 ceph-mon[27845]: pgmap v6859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:56.401+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:56.401+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:07:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:07:56.401+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:07:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:56 smithi194 ceph-mon[33017]: pgmap v6860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:56 smithi033 ceph-mon[27845]: pgmap v6860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:56 smithi096 ceph-mon[32753]: pgmap v6860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:07:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:58 smithi194 ceph-mon[33017]: pgmap v6861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11816.870327 secs 2024-04-26T15:07:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:58 smithi033 ceph-mon[27845]: pgmap v6861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11816.870327 secs 2024-04-26T15:07:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:58 smithi096 ceph-mon[32753]: pgmap v6861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:07:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11816.870327 secs 2024-04-26T15:07:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:07:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11819.659481 secs 2024-04-26T15:07:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:07:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11819.659481 secs 2024-04-26T15:07:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:07:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11819.659481 secs 2024-04-26T15:08:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:00 smithi194 ceph-mon[33017]: pgmap v6862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:00 smithi033 ceph-mon[27845]: pgmap v6862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:00 smithi096 ceph-mon[32753]: pgmap v6862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:02 smithi194 ceph-mon[33017]: pgmap v6863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:02 smithi033 ceph-mon[27845]: pgmap v6863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:02 smithi096 ceph-mon[32753]: pgmap v6863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11821.870409 secs 2024-04-26T15:08:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11821.870409 secs 2024-04-26T15:08:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11821.870409 secs 2024-04-26T15:08:04.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11824.659603 secs 2024-04-26T15:08:04.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:04 smithi033 ceph-mon[27845]: pgmap v6864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:04.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:08:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11824.659603 secs 2024-04-26T15:08:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:04 smithi194 ceph-mon[33017]: pgmap v6864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:08:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11824.659603 secs 2024-04-26T15:08:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:04 smithi096 ceph-mon[32753]: pgmap v6864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:08:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:08:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:08:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:08:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:08:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:08:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:08:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:08:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:08:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:08:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:06 smithi194 ceph-mon[33017]: pgmap v6865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:08:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:06 smithi194 ceph-mon[33017]: pgmap v6866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:06 smithi033 ceph-mon[27845]: pgmap v6865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:08:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:06 smithi033 ceph-mon[27845]: pgmap v6866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:06 smithi096 ceph-mon[32753]: pgmap v6865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:08:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:06 smithi096 ceph-mon[32753]: pgmap v6866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:07 smithi194 ceph-mon[33017]: pgmap v6867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:07 smithi033 ceph-mon[27845]: pgmap v6867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:07 smithi096 ceph-mon[32753]: pgmap v6867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11826.870470 secs 2024-04-26T15:08:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11826.870470 secs 2024-04-26T15:08:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11826.870470 secs 2024-04-26T15:08:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11829.659674 secs 2024-04-26T15:08:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:09 smithi194 ceph-mon[33017]: pgmap v6868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:08:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11829.659674 secs 2024-04-26T15:08:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:09 smithi033 ceph-mon[27845]: pgmap v6868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:08:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11829.659674 secs 2024-04-26T15:08:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:09 smithi096 ceph-mon[32753]: pgmap v6868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:08:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:11.407+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:11.407+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:08:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:11.407+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:12 smithi033 ceph-mon[27845]: pgmap v6869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:12 smithi096 ceph-mon[32753]: pgmap v6869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:12 smithi194 ceph-mon[33017]: pgmap v6869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11831.870553 secs 2024-04-26T15:08:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11831.870553 secs 2024-04-26T15:08:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11831.870553 secs 2024-04-26T15:08:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:14 smithi033 ceph-mon[27845]: pgmap v6870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11834.659824 secs 2024-04-26T15:08:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:14 smithi096 ceph-mon[32753]: pgmap v6870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11834.659824 secs 2024-04-26T15:08:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:14 smithi194 ceph-mon[33017]: pgmap v6870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:08:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11834.659824 secs 2024-04-26T15:08:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:16 smithi033 ceph-mon[27845]: pgmap v6871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:08:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:16 smithi096 ceph-mon[32753]: pgmap v6871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:08:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:16 smithi194 ceph-mon[33017]: pgmap v6871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:08:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:18 smithi194 ceph-mon[33017]: pgmap v6872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11836.870633 secs 2024-04-26T15:08:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:18 smithi033 ceph-mon[27845]: pgmap v6872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11836.870633 secs 2024-04-26T15:08:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:18 smithi096 ceph-mon[32753]: pgmap v6872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11836.870633 secs 2024-04-26T15:08:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11839.659970 secs 2024-04-26T15:08:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11839.659970 secs 2024-04-26T15:08:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11839.659970 secs 2024-04-26T15:08:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:20 smithi194 ceph-mon[33017]: pgmap v6873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:20 smithi033 ceph-mon[27845]: pgmap v6873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:20 smithi096 ceph-mon[32753]: pgmap v6873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:22 smithi194 ceph-mon[33017]: pgmap v6874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:22 smithi033 ceph-mon[27845]: pgmap v6874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:22 smithi096 ceph-mon[32753]: pgmap v6874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11841.870717 secs 2024-04-26T15:08:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11841.870717 secs 2024-04-26T15:08:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11841.870717 secs 2024-04-26T15:08:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:23.806+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:08:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:23.807+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:08:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:23.807+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:08:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:23.808+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:08:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:24 smithi096 ceph-mon[32753]: pgmap v6875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11844.660062 secs 2024-04-26T15:08:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:24 smithi194 ceph-mon[33017]: pgmap v6875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11844.660062 secs 2024-04-26T15:08:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:24 smithi033 ceph-mon[27845]: pgmap v6875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11844.660062 secs 2024-04-26T15:08:26.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:26.411+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:26.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:26.411+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:26.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:26.411+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:08:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:26 smithi194 ceph-mon[33017]: pgmap v6876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:26 smithi033 ceph-mon[27845]: pgmap v6876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:26 smithi096 ceph-mon[32753]: pgmap v6876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:28 smithi194 ceph-mon[33017]: pgmap v6877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11846.870795 secs 2024-04-26T15:08:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:28 smithi033 ceph-mon[27845]: pgmap v6877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11846.870795 secs 2024-04-26T15:08:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:28 smithi096 ceph-mon[32753]: pgmap v6877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11846.870795 secs 2024-04-26T15:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11849.660198 secs 2024-04-26T15:08:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11849.660198 secs 2024-04-26T15:08:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11849.660198 secs 2024-04-26T15:08:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:30 smithi194 ceph-mon[33017]: pgmap v6878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:30 smithi033 ceph-mon[27845]: pgmap v6878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:30 smithi096 ceph-mon[32753]: pgmap v6878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:32 smithi194 ceph-mon[33017]: pgmap v6879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:32 smithi033 ceph-mon[27845]: pgmap v6879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:32 smithi096 ceph-mon[32753]: pgmap v6879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11851.870873 secs 2024-04-26T15:08:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11851.870873 secs 2024-04-26T15:08:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11851.870873 secs 2024-04-26T15:08:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:34 smithi194 ceph-mon[33017]: pgmap v6880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11854.660276 secs 2024-04-26T15:08:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:34 smithi033 ceph-mon[27845]: pgmap v6880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11854.660276 secs 2024-04-26T15:08:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:34 smithi096 ceph-mon[32753]: pgmap v6880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11854.660276 secs 2024-04-26T15:08:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:36 smithi194 ceph-mon[33017]: pgmap v6881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:36 smithi033 ceph-mon[27845]: pgmap v6881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:36 smithi096 ceph-mon[32753]: pgmap v6881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:38 smithi033 ceph-mon[27845]: pgmap v6882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11856.870958 secs 2024-04-26T15:08:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:38 smithi096 ceph-mon[32753]: pgmap v6882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11856.870958 secs 2024-04-26T15:08:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:38 smithi194 ceph-mon[33017]: pgmap v6882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11856.870958 secs 2024-04-26T15:08:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11859.660397 secs 2024-04-26T15:08:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11859.660397 secs 2024-04-26T15:08:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11859.660397 secs 2024-04-26T15:08:40.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:40 smithi033 ceph-mon[27845]: pgmap v6883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:40 smithi096 ceph-mon[32753]: pgmap v6883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:40 smithi194 ceph-mon[33017]: pgmap v6883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:08:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:41.427+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:41.427+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:08:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:41.427+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:42 smithi033 ceph-mon[27845]: pgmap v6884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:42 smithi096 ceph-mon[32753]: pgmap v6884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:42 smithi194 ceph-mon[33017]: pgmap v6884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:08:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11861.871035 secs 2024-04-26T15:08:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11861.871035 secs 2024-04-26T15:08:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11861.871035 secs 2024-04-26T15:08:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:44 smithi033 ceph-mon[27845]: pgmap v6885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11864.660520 secs 2024-04-26T15:08:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:44 smithi096 ceph-mon[32753]: pgmap v6885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11864.660520 secs 2024-04-26T15:08:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:44 smithi194 ceph-mon[33017]: pgmap v6885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11864.660520 secs 2024-04-26T15:08:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:46 smithi033 ceph-mon[27845]: pgmap v6886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:46 smithi096 ceph-mon[32753]: pgmap v6886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:46 smithi194 ceph-mon[33017]: pgmap v6886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:48 smithi033 ceph-mon[27845]: pgmap v6887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11866.871120 secs 2024-04-26T15:08:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:48 smithi096 ceph-mon[32753]: pgmap v6887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11866.871120 secs 2024-04-26T15:08:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:48 smithi194 ceph-mon[33017]: pgmap v6887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11866.871120 secs 2024-04-26T15:08:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11869.660593 secs 2024-04-26T15:08:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11869.660593 secs 2024-04-26T15:08:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11869.660593 secs 2024-04-26T15:08:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:50 smithi033 ceph-mon[27845]: pgmap v6888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:50 smithi096 ceph-mon[32753]: pgmap v6888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:50 smithi194 ceph-mon[33017]: pgmap v6888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:52 smithi033 ceph-mon[27845]: pgmap v6889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:52 smithi096 ceph-mon[32753]: pgmap v6889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:52 smithi194 ceph-mon[33017]: pgmap v6889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11871.871182 secs 2024-04-26T15:08:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11871.871182 secs 2024-04-26T15:08:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11871.871182 secs 2024-04-26T15:08:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:54 smithi096 ceph-mon[32753]: pgmap v6890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11874.660716 secs 2024-04-26T15:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:54 smithi194 ceph-mon[33017]: pgmap v6890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11874.660716 secs 2024-04-26T15:08:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:54 smithi033 ceph-mon[27845]: pgmap v6890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11874.660716 secs 2024-04-26T15:08:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:56.442+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:56.443+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:08:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:08:56.443+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:08:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:56 smithi194 ceph-mon[33017]: pgmap v6891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:56 smithi033 ceph-mon[27845]: pgmap v6891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:56 smithi096 ceph-mon[32753]: pgmap v6891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:08:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:58 smithi194 ceph-mon[33017]: pgmap v6892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11876.871245 secs 2024-04-26T15:08:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:58 smithi033 ceph-mon[27845]: pgmap v6892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11876.871245 secs 2024-04-26T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:58 smithi096 ceph-mon[32753]: pgmap v6892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11876.871245 secs 2024-04-26T15:08:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11879.660838 secs 2024-04-26T15:08:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11879.660838 secs 2024-04-26T15:08:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11879.660838 secs 2024-04-26T15:09:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:00 smithi194 ceph-mon[33017]: pgmap v6893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:00 smithi033 ceph-mon[27845]: pgmap v6893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:00 smithi096 ceph-mon[32753]: pgmap v6893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:02 smithi194 ceph-mon[33017]: pgmap v6894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:02 smithi033 ceph-mon[27845]: pgmap v6894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:02 smithi096 ceph-mon[32753]: pgmap v6894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11881.871328 secs 2024-04-26T15:09:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11881.871328 secs 2024-04-26T15:09:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11881.871328 secs 2024-04-26T15:09:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:04 smithi033 ceph-mon[27845]: pgmap v6895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11884.660902 secs 2024-04-26T15:09:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:04 smithi194 ceph-mon[33017]: pgmap v6895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11884.660902 secs 2024-04-26T15:09:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:04 smithi096 ceph-mon[32753]: pgmap v6895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11884.660902 secs 2024-04-26T15:09:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:09:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:09:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:09:06.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:06 smithi033 ceph-mon[27845]: pgmap v6896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:06.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.822 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:06 smithi194 ceph-mon[33017]: pgmap v6896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:06.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.892 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:06 smithi096 ceph-mon[32753]: pgmap v6896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:06.892 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.892 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.892 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:06.893 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:08 smithi194 ceph-mon[33017]: pgmap v6897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11886.871398 secs 2024-04-26T15:09:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:08 smithi033 ceph-mon[27845]: pgmap v6897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11886.871398 secs 2024-04-26T15:09:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:08 smithi096 ceph-mon[32753]: pgmap v6897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11886.871398 secs 2024-04-26T15:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11889.660966 secs 2024-04-26T15:09:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11889.660966 secs 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11889.660966 secs 2024-04-26T15:09:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:09.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:10 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:10 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:10 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:10 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:10 smithi194 ceph-mon[33017]: pgmap v6898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:10 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:09:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:10 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:09:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:10 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:09:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:10 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:09:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:10 smithi033 ceph-mon[27845]: pgmap v6898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:10 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:10 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:10 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:10 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:10 smithi096 ceph-mon[32753]: pgmap v6898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:11.457+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:11.457+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:11.457+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: pgmap v6899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:09:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:09:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:09:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:09:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: pgmap v6900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:09:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: pgmap v6901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:12.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: pgmap v6899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: pgmap v6900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: pgmap v6901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:12.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: pgmap v6899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: pgmap v6900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: pgmap v6901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:09:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11891.871482 secs 2024-04-26T15:09:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11891.871482 secs 2024-04-26T15:09:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11891.871482 secs 2024-04-26T15:09:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11894.661087 secs 2024-04-26T15:09:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:14 smithi194 ceph-mon[33017]: pgmap v6902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:09:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11894.661087 secs 2024-04-26T15:09:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:14 smithi033 ceph-mon[27845]: pgmap v6902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:09:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11894.661087 secs 2024-04-26T15:09:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:14 smithi096 ceph-mon[32753]: pgmap v6902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:09:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:15 smithi033 ceph-mon[27845]: pgmap v6903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:15 smithi096 ceph-mon[32753]: pgmap v6903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:15 smithi194 ceph-mon[33017]: pgmap v6903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:17.816 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:17 smithi033 ceph-mon[27845]: pgmap v6904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:17 smithi096 ceph-mon[32753]: pgmap v6904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:17 smithi194 ceph-mon[33017]: pgmap v6904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11896.871542 secs 2024-04-26T15:09:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11896.871542 secs 2024-04-26T15:09:19.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11896.871542 secs 2024-04-26T15:09:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11899.661217 secs 2024-04-26T15:09:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:19 smithi033 ceph-mon[27845]: pgmap v6905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11899.661217 secs 2024-04-26T15:09:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:19 smithi096 ceph-mon[32753]: pgmap v6905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:20.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11899.661217 secs 2024-04-26T15:09:20.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:19 smithi194 ceph-mon[33017]: pgmap v6905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:09:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:21 smithi033 ceph-mon[27845]: pgmap v6906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:09:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:21 smithi096 ceph-mon[32753]: pgmap v6906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:09:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:21 smithi194 ceph-mon[33017]: pgmap v6906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:09:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11901.871625 secs 2024-04-26T15:09:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11901.871625 secs 2024-04-26T15:09:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11901.871625 secs 2024-04-26T15:09:24.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:23 smithi096 ceph-mon[32753]: pgmap v6907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:23 smithi194 ceph-mon[33017]: pgmap v6907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:23 smithi033 ceph-mon[27845]: pgmap v6907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11904.661345 secs 2024-04-26T15:09:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11904.661345 secs 2024-04-26T15:09:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11904.661345 secs 2024-04-26T15:09:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:25 smithi194 ceph-mon[33017]: pgmap v6908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:25 smithi033 ceph-mon[27845]: pgmap v6908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:25 smithi096 ceph-mon[32753]: pgmap v6908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:26.462+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:26.462+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:26.462+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:09:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:27 smithi033 ceph-mon[27845]: pgmap v6909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:27 smithi096 ceph-mon[32753]: pgmap v6909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:27 smithi194 ceph-mon[33017]: pgmap v6909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11906.871705 secs 2024-04-26T15:09:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11906.871705 secs 2024-04-26T15:09:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11906.871705 secs 2024-04-26T15:09:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11909.661475 secs 2024-04-26T15:09:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:29 smithi033 ceph-mon[27845]: pgmap v6910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11909.661475 secs 2024-04-26T15:09:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:29 smithi096 ceph-mon[32753]: pgmap v6910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11909.661475 secs 2024-04-26T15:09:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:29 smithi194 ceph-mon[33017]: pgmap v6910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:31 smithi033 ceph-mon[27845]: pgmap v6911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:31 smithi096 ceph-mon[32753]: pgmap v6911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:32.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:31 smithi194 ceph-mon[33017]: pgmap v6911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11911.871788 secs 2024-04-26T15:09:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11911.871788 secs 2024-04-26T15:09:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11911.871788 secs 2024-04-26T15:09:34.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:33 smithi033 ceph-mon[27845]: pgmap v6912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:33 smithi194 ceph-mon[33017]: pgmap v6912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:33 smithi096 ceph-mon[32753]: pgmap v6912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11914.661571 secs 2024-04-26T15:09:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11914.661571 secs 2024-04-26T15:09:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11914.661571 secs 2024-04-26T15:09:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:35 smithi194 ceph-mon[33017]: pgmap v6913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:35 smithi033 ceph-mon[27845]: pgmap v6913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:35 smithi096 ceph-mon[32753]: pgmap v6913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:37 smithi033 ceph-mon[27845]: pgmap v6914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:37 smithi096 ceph-mon[32753]: pgmap v6914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:37 smithi194 ceph-mon[33017]: pgmap v6914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11916.871871 secs 2024-04-26T15:09:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11916.871871 secs 2024-04-26T15:09:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11916.871871 secs 2024-04-26T15:09:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11919.661698 secs 2024-04-26T15:09:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:39 smithi033 ceph-mon[27845]: pgmap v6915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11919.661698 secs 2024-04-26T15:09:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:39 smithi096 ceph-mon[32753]: pgmap v6915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11919.661698 secs 2024-04-26T15:09:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:39 smithi194 ceph-mon[33017]: pgmap v6915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:41.478+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:41.478+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:41.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:41.478+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:09:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:41 smithi033 ceph-mon[27845]: pgmap v6916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:41 smithi096 ceph-mon[32753]: pgmap v6916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:41 smithi194 ceph-mon[33017]: pgmap v6916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11921.871952 secs 2024-04-26T15:09:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11921.871952 secs 2024-04-26T15:09:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11921.871952 secs 2024-04-26T15:09:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:43 smithi194 ceph-mon[33017]: pgmap v6917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:43 smithi033 ceph-mon[27845]: pgmap v6917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:43 smithi096 ceph-mon[32753]: pgmap v6917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11924.661852 secs 2024-04-26T15:09:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11924.661852 secs 2024-04-26T15:09:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11924.661852 secs 2024-04-26T15:09:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:45 smithi194 ceph-mon[33017]: pgmap v6918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:45 smithi033 ceph-mon[27845]: pgmap v6918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:45 smithi096 ceph-mon[32753]: pgmap v6918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:47 smithi033 ceph-mon[27845]: pgmap v6919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:47 smithi096 ceph-mon[32753]: pgmap v6919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:47 smithi194 ceph-mon[33017]: pgmap v6919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11926.872036 secs 2024-04-26T15:09:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11926.872036 secs 2024-04-26T15:09:49.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11926.872036 secs 2024-04-26T15:09:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11929.661976 secs 2024-04-26T15:09:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:49 smithi033 ceph-mon[27845]: pgmap v6920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11929.661976 secs 2024-04-26T15:09:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:49 smithi096 ceph-mon[32753]: pgmap v6920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11929.661976 secs 2024-04-26T15:09:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:49 smithi194 ceph-mon[33017]: pgmap v6920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:51 smithi033 ceph-mon[27845]: pgmap v6921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:51 smithi096 ceph-mon[32753]: pgmap v6921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:51 smithi194 ceph-mon[33017]: pgmap v6921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11931.872097 secs 2024-04-26T15:09:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11931.872097 secs 2024-04-26T15:09:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11931.872097 secs 2024-04-26T15:09:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:53 smithi096 ceph-mon[32753]: pgmap v6922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:53 smithi194 ceph-mon[33017]: pgmap v6922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:53 smithi033 ceph-mon[27845]: pgmap v6922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11934.662102 secs 2024-04-26T15:09:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11934.662102 secs 2024-04-26T15:09:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11934.662102 secs 2024-04-26T15:09:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:55 smithi194 ceph-mon[33017]: pgmap v6923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:55 smithi033 ceph-mon[27845]: pgmap v6923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:55 smithi096 ceph-mon[32753]: pgmap v6923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:56.486+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:56.487+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:09:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:09:56.487+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:09:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:57 smithi033 ceph-mon[27845]: pgmap v6924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:57 smithi096 ceph-mon[32753]: pgmap v6924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:57 smithi194 ceph-mon[33017]: pgmap v6924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:09:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11936.872159 secs 2024-04-26T15:09:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11936.872159 secs 2024-04-26T15:09:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11936.872159 secs 2024-04-26T15:09:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11939.662224 secs 2024-04-26T15:09:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:09:59 smithi033 ceph-mon[27845]: pgmap v6925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:09:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11939.662224 secs 2024-04-26T15:09:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:09:59 smithi096 ceph-mon[32753]: pgmap v6925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11939.662224 secs 2024-04-26T15:10:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:09:59 smithi194 ceph-mon[33017]: pgmap v6925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:10:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:10:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:01 smithi096 ceph-mon[32753]: pgmap v6926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:01 smithi194 ceph-mon[33017]: pgmap v6926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:01 smithi033 ceph-mon[27845]: pgmap v6926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11941.872215 secs 2024-04-26T15:10:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11941.872215 secs 2024-04-26T15:10:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11941.872215 secs 2024-04-26T15:10:04.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:03 smithi033 ceph-mon[27845]: pgmap v6927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:03 smithi194 ceph-mon[33017]: pgmap v6927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:03 smithi096 ceph-mon[32753]: pgmap v6927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11944.662346 secs 2024-04-26T15:10:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11944.662346 secs 2024-04-26T15:10:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11944.662346 secs 2024-04-26T15:10:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:05 smithi194 ceph-mon[33017]: pgmap v6928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:05 smithi033 ceph-mon[27845]: pgmap v6928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:05 smithi096 ceph-mon[32753]: pgmap v6928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:07 smithi033 ceph-mon[27845]: pgmap v6929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:07 smithi096 ceph-mon[32753]: pgmap v6929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:07 smithi194 ceph-mon[33017]: pgmap v6929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11946.872276 secs 2024-04-26T15:10:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11946.872276 secs 2024-04-26T15:10:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11946.872276 secs 2024-04-26T15:10:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11949.662433 secs 2024-04-26T15:10:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:09 smithi033 ceph-mon[27845]: pgmap v6930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11949.662433 secs 2024-04-26T15:10:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:09 smithi096 ceph-mon[32753]: pgmap v6930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11949.662433 secs 2024-04-26T15:10:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:09 smithi194 ceph-mon[33017]: pgmap v6930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:11.496+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:11.496+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:10:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:11.496+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:11 smithi033 ceph-mon[27845]: pgmap v6931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:11 smithi096 ceph-mon[32753]: pgmap v6931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:11 smithi194 ceph-mon[33017]: pgmap v6931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:10:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:10:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:10:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11951.872362 secs 2024-04-26T15:10:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:10:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:10:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:13 smithi033 ceph-mon[27845]: pgmap v6932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:13 smithi033 ceph-mon[27845]: pgmap v6933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:10:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11951.872362 secs 2024-04-26T15:10:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:10:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:10:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:13 smithi096 ceph-mon[32753]: pgmap v6932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:13 smithi096 ceph-mon[32753]: pgmap v6933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11951.872362 secs 2024-04-26T15:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:13 smithi194 ceph-mon[33017]: pgmap v6932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:13 smithi194 ceph-mon[33017]: pgmap v6933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:10:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11954.662545 secs 2024-04-26T15:10:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11954.662545 secs 2024-04-26T15:10:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11954.662545 secs 2024-04-26T15:10:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:15 smithi096 ceph-mon[32753]: pgmap v6934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:15 smithi194 ceph-mon[33017]: pgmap v6934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:15 smithi033 ceph-mon[27845]: pgmap v6934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:18 smithi194 ceph-mon[33017]: pgmap v6935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11956.872445 secs 2024-04-26T15:10:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:18 smithi033 ceph-mon[27845]: pgmap v6935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11956.872445 secs 2024-04-26T15:10:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:18 smithi096 ceph-mon[32753]: pgmap v6935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11956.872445 secs 2024-04-26T15:10:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11959.662686 secs 2024-04-26T15:10:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11959.662686 secs 2024-04-26T15:10:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11959.662686 secs 2024-04-26T15:10:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:20 smithi194 ceph-mon[33017]: pgmap v6936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:10:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:20 smithi033 ceph-mon[27845]: pgmap v6936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:10:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:20 smithi096 ceph-mon[32753]: pgmap v6936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:10:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:22 smithi194 ceph-mon[33017]: pgmap v6937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:22 smithi033 ceph-mon[27845]: pgmap v6937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:22 smithi096 ceph-mon[32753]: pgmap v6937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:10:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11961.872528 secs 2024-04-26T15:10:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11961.872528 secs 2024-04-26T15:10:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11961.872528 secs 2024-04-26T15:10:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11964.662806 secs 2024-04-26T15:10:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:24 smithi194 ceph-mon[33017]: pgmap v6938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:10:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11964.662806 secs 2024-04-26T15:10:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:24 smithi033 ceph-mon[27845]: pgmap v6938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:10:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11964.662806 secs 2024-04-26T15:10:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:24 smithi096 ceph-mon[32753]: pgmap v6938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:10:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:26.512+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:26.513+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:26.513+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:10:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:26 smithi033 ceph-mon[27845]: pgmap v6939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:26 smithi096 ceph-mon[32753]: pgmap v6939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:26 smithi194 ceph-mon[33017]: pgmap v6939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:28 smithi033 ceph-mon[27845]: pgmap v6940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11966.872613 secs 2024-04-26T15:10:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:28 smithi096 ceph-mon[32753]: pgmap v6940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11966.872613 secs 2024-04-26T15:10:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:28 smithi194 ceph-mon[33017]: pgmap v6940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11966.872613 secs 2024-04-26T15:10:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11969.662929 secs 2024-04-26T15:10:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11969.662929 secs 2024-04-26T15:10:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11969.662929 secs 2024-04-26T15:10:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:30 smithi033 ceph-mon[27845]: pgmap v6941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:30 smithi096 ceph-mon[32753]: pgmap v6941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:30 smithi194 ceph-mon[33017]: pgmap v6941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:32 smithi033 ceph-mon[27845]: pgmap v6942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:32 smithi096 ceph-mon[32753]: pgmap v6942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:32 smithi194 ceph-mon[33017]: pgmap v6942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11971.872697 secs 2024-04-26T15:10:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11971.872697 secs 2024-04-26T15:10:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11971.872697 secs 2024-04-26T15:10:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11974.663031 secs 2024-04-26T15:10:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:34 smithi033 ceph-mon[27845]: pgmap v6943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11974.663031 secs 2024-04-26T15:10:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:34 smithi096 ceph-mon[32753]: pgmap v6943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11974.663031 secs 2024-04-26T15:10:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:34 smithi194 ceph-mon[33017]: pgmap v6943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:36 smithi033 ceph-mon[27845]: pgmap v6944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:36 smithi096 ceph-mon[32753]: pgmap v6944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:36 smithi194 ceph-mon[33017]: pgmap v6944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:38 smithi033 ceph-mon[27845]: pgmap v6945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11976.872780 secs 2024-04-26T15:10:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:38 smithi096 ceph-mon[32753]: pgmap v6945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11976.872780 secs 2024-04-26T15:10:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:38 smithi194 ceph-mon[33017]: pgmap v6945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11976.872780 secs 2024-04-26T15:10:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11979.663179 secs 2024-04-26T15:10:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11979.663179 secs 2024-04-26T15:10:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11979.663179 secs 2024-04-26T15:10:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:40 smithi096 ceph-mon[32753]: pgmap v6946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:40 smithi194 ceph-mon[33017]: pgmap v6946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:40 smithi033 ceph-mon[27845]: pgmap v6946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:41.523 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:41.523+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:41.523+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:41.523+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:10:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:42 smithi096 ceph-mon[32753]: pgmap v6947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:42 smithi194 ceph-mon[33017]: pgmap v6947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:42 smithi033 ceph-mon[27845]: pgmap v6947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11981.872865 secs 2024-04-26T15:10:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11981.872865 secs 2024-04-26T15:10:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11981.872865 secs 2024-04-26T15:10:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11984.663296 secs 2024-04-26T15:10:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:44 smithi194 ceph-mon[33017]: pgmap v6948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11984.663296 secs 2024-04-26T15:10:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:44 smithi033 ceph-mon[27845]: pgmap v6948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11984.663296 secs 2024-04-26T15:10:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:44 smithi096 ceph-mon[32753]: pgmap v6948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:46 smithi194 ceph-mon[33017]: pgmap v6949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:46 smithi033 ceph-mon[27845]: pgmap v6949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:46 smithi096 ceph-mon[32753]: pgmap v6949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:48 smithi194 ceph-mon[33017]: pgmap v6950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11986.872951 secs 2024-04-26T15:10:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:48 smithi033 ceph-mon[27845]: pgmap v6950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11986.872951 secs 2024-04-26T15:10:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:48 smithi096 ceph-mon[32753]: pgmap v6950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11986.872951 secs 2024-04-26T15:10:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11989.663404 secs 2024-04-26T15:10:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11989.663404 secs 2024-04-26T15:10:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11989.663404 secs 2024-04-26T15:10:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:50 smithi194 ceph-mon[33017]: pgmap v6951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:50 smithi033 ceph-mon[27845]: pgmap v6951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:50 smithi096 ceph-mon[32753]: pgmap v6951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:52 smithi194 ceph-mon[33017]: pgmap v6952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:52 smithi033 ceph-mon[27845]: pgmap v6952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:52 smithi096 ceph-mon[32753]: pgmap v6952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11991.873057 secs 2024-04-26T15:10:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11991.873057 secs 2024-04-26T15:10:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11991.873057 secs 2024-04-26T15:10:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11994.663523 secs 2024-04-26T15:10:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:54 smithi096 ceph-mon[32753]: pgmap v6953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11994.663523 secs 2024-04-26T15:10:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:54 smithi194 ceph-mon[33017]: pgmap v6953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11994.663523 secs 2024-04-26T15:10:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:54 smithi033 ceph-mon[27845]: pgmap v6953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:56.532+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:56.532+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:10:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:10:56.532+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:10:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:56 smithi194 ceph-mon[33017]: pgmap v6954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:56 smithi033 ceph-mon[27845]: pgmap v6954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:56 smithi096 ceph-mon[32753]: pgmap v6954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:10:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:58 smithi194 ceph-mon[33017]: pgmap v6955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 11996.873138 secs 2024-04-26T15:10:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:58 smithi033 ceph-mon[27845]: pgmap v6955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 11996.873138 secs 2024-04-26T15:10:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:58 smithi096 ceph-mon[32753]: pgmap v6955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:10:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 11996.873138 secs 2024-04-26T15:10:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:10:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 11999.663648 secs 2024-04-26T15:10:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:10:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 11999.663648 secs 2024-04-26T15:10:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:10:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 11999.663648 secs 2024-04-26T15:11:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:00 smithi194 ceph-mon[33017]: pgmap v6956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:00 smithi033 ceph-mon[27845]: pgmap v6956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:00 smithi096 ceph-mon[32753]: pgmap v6956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:02 smithi194 ceph-mon[33017]: pgmap v6957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:02 smithi033 ceph-mon[27845]: pgmap v6957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:02 smithi096 ceph-mon[32753]: pgmap v6957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12001.873200 secs 2024-04-26T15:11:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12001.873200 secs 2024-04-26T15:11:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12001.873200 secs 2024-04-26T15:11:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12004.663714 secs 2024-04-26T15:11:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:04 smithi033 ceph-mon[27845]: pgmap v6958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12004.663714 secs 2024-04-26T15:11:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:04 smithi194 ceph-mon[33017]: pgmap v6958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12004.663714 secs 2024-04-26T15:11:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:04 smithi096 ceph-mon[32753]: pgmap v6958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:06 smithi194 ceph-mon[33017]: pgmap v6959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:06 smithi033 ceph-mon[27845]: pgmap v6959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:06 smithi096 ceph-mon[32753]: pgmap v6959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:08 smithi194 ceph-mon[33017]: pgmap v6960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12006.873289 secs 2024-04-26T15:11:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:08 smithi033 ceph-mon[27845]: pgmap v6960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12006.873289 secs 2024-04-26T15:11:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:08 smithi096 ceph-mon[32753]: pgmap v6960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12006.873289 secs 2024-04-26T15:11:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12009.663829 secs 2024-04-26T15:11:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12009.663829 secs 2024-04-26T15:11:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12009.663829 secs 2024-04-26T15:11:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:10 smithi194 ceph-mon[33017]: pgmap v6961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:10 smithi033 ceph-mon[27845]: pgmap v6961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:10 smithi096 ceph-mon[32753]: pgmap v6961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:11.547+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:11.547+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:11.547+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:11:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:12 smithi194 ceph-mon[33017]: pgmap v6962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:12 smithi033 ceph-mon[27845]: pgmap v6962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:12 smithi096 ceph-mon[32753]: pgmap v6962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12011.873348 secs 2024-04-26T15:11:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12011.873348 secs 2024-04-26T15:11:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12011.873348 secs 2024-04-26T15:11:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12014.663924 secs 2024-04-26T15:11:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:14 smithi194 ceph-mon[33017]: pgmap v6963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:11:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12014.663924 secs 2024-04-26T15:11:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:14 smithi033 ceph-mon[27845]: pgmap v6963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12014.663924 secs 2024-04-26T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:14 smithi096 ceph-mon[32753]: pgmap v6963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:11:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:11:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:11:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:11:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:11:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:11:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:11:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:11:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:11:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:11:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:16 smithi194 ceph-mon[33017]: pgmap v6964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:11:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:16 smithi194 ceph-mon[33017]: pgmap v6965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:16 smithi033 ceph-mon[27845]: pgmap v6964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:11:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:16 smithi033 ceph-mon[27845]: pgmap v6965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:16 smithi096 ceph-mon[32753]: pgmap v6964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:11:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:16 smithi096 ceph-mon[32753]: pgmap v6965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:18 smithi194 ceph-mon[33017]: pgmap v6966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12016.873407 secs 2024-04-26T15:11:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:18 smithi033 ceph-mon[27845]: pgmap v6966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12016.873407 secs 2024-04-26T15:11:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:18 smithi096 ceph-mon[32753]: pgmap v6966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12016.873407 secs 2024-04-26T15:11:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12019.664044 secs 2024-04-26T15:11:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12019.664044 secs 2024-04-26T15:11:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12019.664044 secs 2024-04-26T15:11:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:20 smithi033 ceph-mon[27845]: pgmap v6967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:11:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:20 smithi096 ceph-mon[32753]: pgmap v6967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:11:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:20 smithi194 ceph-mon[33017]: pgmap v6967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:11:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0062040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0062040 comp rx=0 tx=0).stop 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3780 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.743+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T15:11:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.744+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T15:11:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:21.745+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T15:11:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:22 smithi033 ceph-mon[27845]: pgmap v6968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:22 smithi096 ceph-mon[32753]: pgmap v6968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:22 smithi194 ceph-mon[33017]: pgmap v6968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:23.836+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:11:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:23.836+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:11:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:23.836+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:11:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:23.837+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:11:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12021.873490 secs 2024-04-26T15:11:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12021.873490 secs 2024-04-26T15:11:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12021.873490 secs 2024-04-26T15:11:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:24 smithi033 ceph-mon[27845]: pgmap v6969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12024.664179 secs 2024-04-26T15:11:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:24 smithi096 ceph-mon[32753]: pgmap v6969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12024.664179 secs 2024-04-26T15:11:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:24 smithi194 ceph-mon[33017]: pgmap v6969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:11:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12024.664179 secs 2024-04-26T15:11:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:25 smithi033 ceph-mon[27845]: pgmap v6970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:11:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:25 smithi096 ceph-mon[32753]: pgmap v6970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:11:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:25 smithi194 ceph-mon[33017]: pgmap v6970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:11:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:26.554+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:26.554+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:11:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:26.554+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:27 smithi194 ceph-mon[33017]: pgmap v6971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12026.873571 secs 2024-04-26T15:11:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:27 smithi033 ceph-mon[27845]: pgmap v6971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12026.873571 secs 2024-04-26T15:11:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:27 smithi096 ceph-mon[32753]: pgmap v6971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12026.873571 secs 2024-04-26T15:11:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12029.664321 secs 2024-04-26T15:11:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12029.664321 secs 2024-04-26T15:11:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12029.664321 secs 2024-04-26T15:11:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:30 smithi194 ceph-mon[33017]: pgmap v6972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:30 smithi033 ceph-mon[27845]: pgmap v6972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:30 smithi096 ceph-mon[32753]: pgmap v6972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:32 smithi194 ceph-mon[33017]: pgmap v6973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:32 smithi033 ceph-mon[27845]: pgmap v6973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:32 smithi096 ceph-mon[32753]: pgmap v6973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12031.873655 secs 2024-04-26T15:11:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12031.873655 secs 2024-04-26T15:11:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12031.873655 secs 2024-04-26T15:11:34.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:34 smithi033 ceph-mon[27845]: pgmap v6974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:34.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12034.664441 secs 2024-04-26T15:11:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:34 smithi194 ceph-mon[33017]: pgmap v6974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12034.664441 secs 2024-04-26T15:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:34 smithi096 ceph-mon[32753]: pgmap v6974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12034.664441 secs 2024-04-26T15:11:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:36 smithi194 ceph-mon[33017]: pgmap v6975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:36 smithi033 ceph-mon[27845]: pgmap v6975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:36 smithi096 ceph-mon[32753]: pgmap v6975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:38 smithi194 ceph-mon[33017]: pgmap v6976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12036.873741 secs 2024-04-26T15:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:38 smithi033 ceph-mon[27845]: pgmap v6976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12036.873741 secs 2024-04-26T15:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:38 smithi096 ceph-mon[32753]: pgmap v6976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12036.873741 secs 2024-04-26T15:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12039.664560 secs 2024-04-26T15:11:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12039.664560 secs 2024-04-26T15:11:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12039.664560 secs 2024-04-26T15:11:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:40 smithi194 ceph-mon[33017]: pgmap v6977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:40 smithi033 ceph-mon[27845]: pgmap v6977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:40 smithi096 ceph-mon[32753]: pgmap v6977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:41.565+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:41.565+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:41.565+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:11:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:42 smithi194 ceph-mon[33017]: pgmap v6978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:42 smithi033 ceph-mon[27845]: pgmap v6978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:42 smithi096 ceph-mon[32753]: pgmap v6978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12041.873824 secs 2024-04-26T15:11:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12041.873824 secs 2024-04-26T15:11:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12041.873824 secs 2024-04-26T15:11:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:44 smithi194 ceph-mon[33017]: pgmap v6979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12044.664693 secs 2024-04-26T15:11:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:44 smithi033 ceph-mon[27845]: pgmap v6979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:44.449 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12044.664693 secs 2024-04-26T15:11:44.449 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:44 smithi096 ceph-mon[32753]: pgmap v6979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:44.449 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12044.664693 secs 2024-04-26T15:11:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:46 smithi194 ceph-mon[33017]: pgmap v6980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:46 smithi033 ceph-mon[27845]: pgmap v6980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:46 smithi096 ceph-mon[32753]: pgmap v6980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:48 smithi194 ceph-mon[33017]: pgmap v6981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12046.873904 secs 2024-04-26T15:11:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:48 smithi033 ceph-mon[27845]: pgmap v6981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12046.873904 secs 2024-04-26T15:11:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:48 smithi096 ceph-mon[32753]: pgmap v6981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12046.873904 secs 2024-04-26T15:11:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12049.664812 secs 2024-04-26T15:11:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12049.664812 secs 2024-04-26T15:11:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12049.664812 secs 2024-04-26T15:11:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:50 smithi194 ceph-mon[33017]: pgmap v6982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:50 smithi033 ceph-mon[27845]: pgmap v6982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:50 smithi096 ceph-mon[32753]: pgmap v6982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:52 smithi033 ceph-mon[27845]: pgmap v6983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:52 smithi096 ceph-mon[32753]: pgmap v6983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:52 smithi194 ceph-mon[33017]: pgmap v6983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12051.873988 secs 2024-04-26T15:11:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12051.873988 secs 2024-04-26T15:11:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12051.873988 secs 2024-04-26T15:11:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:54 smithi033 ceph-mon[27845]: pgmap v6984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:54.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12054.664900 secs 2024-04-26T15:11:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:54 smithi096 ceph-mon[32753]: pgmap v6984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12054.664900 secs 2024-04-26T15:11:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:54 smithi194 ceph-mon[33017]: pgmap v6984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12054.664900 secs 2024-04-26T15:11:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:56.566+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:56.566+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:11:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:11:56.566+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:11:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:56 smithi033 ceph-mon[27845]: pgmap v6985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:56 smithi096 ceph-mon[32753]: pgmap v6985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:56 smithi194 ceph-mon[33017]: pgmap v6985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:11:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:58 smithi033 ceph-mon[27845]: pgmap v6986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12056.874077 secs 2024-04-26T15:11:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:58 smithi096 ceph-mon[32753]: pgmap v6986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12056.874077 secs 2024-04-26T15:11:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:58 smithi194 ceph-mon[33017]: pgmap v6986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:11:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12056.874077 secs 2024-04-26T15:11:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12059.665025 secs 2024-04-26T15:11:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12059.665025 secs 2024-04-26T15:11:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12059.665025 secs 2024-04-26T15:12:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:00 smithi033 ceph-mon[27845]: pgmap v6987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:00 smithi096 ceph-mon[32753]: pgmap v6987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:00 smithi194 ceph-mon[33017]: pgmap v6987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:02 smithi033 ceph-mon[27845]: pgmap v6988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:02 smithi096 ceph-mon[32753]: pgmap v6988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:02 smithi194 ceph-mon[33017]: pgmap v6988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12061.874166 secs 2024-04-26T15:12:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12061.874166 secs 2024-04-26T15:12:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12061.874166 secs 2024-04-26T15:12:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:04 smithi033 ceph-mon[27845]: pgmap v6989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12064.665138 secs 2024-04-26T15:12:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:04 smithi096 ceph-mon[32753]: pgmap v6989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12064.665138 secs 2024-04-26T15:12:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:04 smithi194 ceph-mon[33017]: pgmap v6989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12064.665138 secs 2024-04-26T15:12:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:06 smithi033 ceph-mon[27845]: pgmap v6990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:06 smithi096 ceph-mon[32753]: pgmap v6990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:06 smithi194 ceph-mon[33017]: pgmap v6990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:08 smithi096 ceph-mon[32753]: pgmap v6991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12066.874253 secs 2024-04-26T15:12:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:08 smithi194 ceph-mon[33017]: pgmap v6991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12066.874253 secs 2024-04-26T15:12:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:08 smithi033 ceph-mon[27845]: pgmap v6991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12066.874253 secs 2024-04-26T15:12:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12069.665255 secs 2024-04-26T15:12:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12069.665255 secs 2024-04-26T15:12:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12069.665255 secs 2024-04-26T15:12:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:10 smithi194 ceph-mon[33017]: pgmap v6992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:10 smithi033 ceph-mon[27845]: pgmap v6992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:10 smithi096 ceph-mon[32753]: pgmap v6992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:11.581+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:11.581+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:11.581+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:12:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:12 smithi194 ceph-mon[33017]: pgmap v6993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:12 smithi033 ceph-mon[27845]: pgmap v6993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:12 smithi096 ceph-mon[32753]: pgmap v6993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12071.874339 secs 2024-04-26T15:12:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12071.874339 secs 2024-04-26T15:12:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12071.874339 secs 2024-04-26T15:12:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:14 smithi194 ceph-mon[33017]: pgmap v6994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12074.665371 secs 2024-04-26T15:12:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:14 smithi033 ceph-mon[27845]: pgmap v6994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12074.665371 secs 2024-04-26T15:12:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:14 smithi096 ceph-mon[32753]: pgmap v6994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12074.665371 secs 2024-04-26T15:12:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:12:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:12:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:12:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:16 smithi194 ceph-mon[33017]: pgmap v6995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:16 smithi033 ceph-mon[27845]: pgmap v6995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:16 smithi096 ceph-mon[32753]: pgmap v6995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:12:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:12:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:17 smithi194 ceph-mon[33017]: pgmap v6996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:17 smithi194 ceph-mon[33017]: pgmap v6997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:12:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:12:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:12:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:17 smithi033 ceph-mon[27845]: pgmap v6996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:17 smithi033 ceph-mon[27845]: pgmap v6997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:12:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:12:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:12:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:17 smithi096 ceph-mon[32753]: pgmap v6996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:17 smithi096 ceph-mon[32753]: pgmap v6997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:12:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12076.874425 secs 2024-04-26T15:12:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12076.874425 secs 2024-04-26T15:12:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12076.874425 secs 2024-04-26T15:12:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12079.665486 secs 2024-04-26T15:12:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:19 smithi194 ceph-mon[33017]: pgmap v6998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12079.665486 secs 2024-04-26T15:12:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:19 smithi033 ceph-mon[27845]: pgmap v6998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12079.665486 secs 2024-04-26T15:12:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:19 smithi096 ceph-mon[32753]: pgmap v6998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:21 smithi033 ceph-mon[27845]: pgmap v6999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:21 smithi096 ceph-mon[32753]: pgmap v6999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:21 smithi194 ceph-mon[33017]: pgmap v6999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:23 smithi033 ceph-mon[27845]: pgmap v7000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12081.874513 secs 2024-04-26T15:12:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:23 smithi096 ceph-mon[32753]: pgmap v7000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12081.874513 secs 2024-04-26T15:12:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:23 smithi194 ceph-mon[33017]: pgmap v7000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12081.874513 secs 2024-04-26T15:12:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12084.665598 secs 2024-04-26T15:12:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12084.665598 secs 2024-04-26T15:12:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12084.665598 secs 2024-04-26T15:12:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:25 smithi033 ceph-mon[27845]: pgmap v7001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:25 smithi096 ceph-mon[32753]: pgmap v7001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:25 smithi194 ceph-mon[33017]: pgmap v7001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:12:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:26.597+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:26.597+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:26.597+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:12:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:27 smithi033 ceph-mon[27845]: pgmap v7002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:12:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:27 smithi096 ceph-mon[32753]: pgmap v7002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:12:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:27 smithi194 ceph-mon[33017]: pgmap v7002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:12:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12086.874571 secs 2024-04-26T15:12:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:28 smithi033 ceph-mon[27845]: pgmap v7003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12086.874571 secs 2024-04-26T15:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:28 smithi096 ceph-mon[32753]: pgmap v7003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12086.874571 secs 2024-04-26T15:12:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:28 smithi194 ceph-mon[33017]: pgmap v7003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12089.665720 secs 2024-04-26T15:12:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12089.665720 secs 2024-04-26T15:12:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12089.665720 secs 2024-04-26T15:12:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:30 smithi033 ceph-mon[27845]: pgmap v7004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:30 smithi096 ceph-mon[32753]: pgmap v7004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:30 smithi194 ceph-mon[33017]: pgmap v7004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:33 smithi033 ceph-mon[27845]: pgmap v7005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12091.874661 secs 2024-04-26T15:12:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:33 smithi096 ceph-mon[32753]: pgmap v7005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12091.874661 secs 2024-04-26T15:12:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:33 smithi194 ceph-mon[33017]: pgmap v7005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12091.874661 secs 2024-04-26T15:12:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12094.665833 secs 2024-04-26T15:12:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12094.665833 secs 2024-04-26T15:12:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12094.665833 secs 2024-04-26T15:12:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:35 smithi033 ceph-mon[27845]: pgmap v7006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:35 smithi096 ceph-mon[32753]: pgmap v7006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:35 smithi194 ceph-mon[33017]: pgmap v7006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:37 smithi033 ceph-mon[27845]: pgmap v7007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:37 smithi096 ceph-mon[32753]: pgmap v7007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:37 smithi194 ceph-mon[33017]: pgmap v7007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12096.874746 secs 2024-04-26T15:12:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12096.874746 secs 2024-04-26T15:12:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12096.874746 secs 2024-04-26T15:12:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12099.665961 secs 2024-04-26T15:12:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:39 smithi033 ceph-mon[27845]: pgmap v7008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12099.665961 secs 2024-04-26T15:12:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:39 smithi096 ceph-mon[32753]: pgmap v7008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12099.665961 secs 2024-04-26T15:12:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:39 smithi194 ceph-mon[33017]: pgmap v7008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:40 smithi033 ceph-mon[27845]: pgmap v7009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:40 smithi096 ceph-mon[32753]: pgmap v7009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:40 smithi194 ceph-mon[33017]: pgmap v7009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:41.605+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:41.605+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:41.605+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:12:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:43 smithi033 ceph-mon[27845]: pgmap v7010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12101.874834 secs 2024-04-26T15:12:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:43 smithi096 ceph-mon[32753]: pgmap v7010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12101.874834 secs 2024-04-26T15:12:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:43 smithi194 ceph-mon[33017]: pgmap v7010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12101.874834 secs 2024-04-26T15:12:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12104.666100 secs 2024-04-26T15:12:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12104.666100 secs 2024-04-26T15:12:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12104.666100 secs 2024-04-26T15:12:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:45 smithi033 ceph-mon[27845]: pgmap v7011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:45 smithi096 ceph-mon[32753]: pgmap v7011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:45 smithi194 ceph-mon[33017]: pgmap v7011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:46 smithi033 ceph-mon[27845]: pgmap v7012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:46 smithi096 ceph-mon[32753]: pgmap v7012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:46 smithi194 ceph-mon[33017]: pgmap v7012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:47 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12106.874945 secs 2024-04-26T15:12:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:47 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12106.874945 secs 2024-04-26T15:12:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:47 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12106.874945 secs 2024-04-26T15:12:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:48 smithi194 ceph-mon[33017]: pgmap v7013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:48 smithi033 ceph-mon[27845]: pgmap v7013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:48 smithi096 ceph-mon[32753]: pgmap v7013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12109.666219 secs 2024-04-26T15:12:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12109.666219 secs 2024-04-26T15:12:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12109.666219 secs 2024-04-26T15:12:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:50 smithi033 ceph-mon[27845]: pgmap v7014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:50 smithi096 ceph-mon[32753]: pgmap v7014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:50 smithi194 ceph-mon[33017]: pgmap v7014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:53 smithi033 ceph-mon[27845]: pgmap v7015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12111.875037 secs 2024-04-26T15:12:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:53 smithi096 ceph-mon[32753]: pgmap v7015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12111.875037 secs 2024-04-26T15:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:53 smithi194 ceph-mon[33017]: pgmap v7015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12111.875037 secs 2024-04-26T15:12:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12114.666333 secs 2024-04-26T15:12:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12114.666333 secs 2024-04-26T15:12:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12114.666333 secs 2024-04-26T15:12:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:55 smithi033 ceph-mon[27845]: pgmap v7016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:55 smithi096 ceph-mon[32753]: pgmap v7016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:55 smithi194 ceph-mon[33017]: pgmap v7016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:12:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:56.621+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:56.621+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:12:56.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:12:56.621+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:12:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:56 smithi033 ceph-mon[27845]: pgmap v7017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:56 smithi096 ceph-mon[32753]: pgmap v7017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:56 smithi194 ceph-mon[33017]: pgmap v7017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12116.875125 secs 2024-04-26T15:12:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12116.875125 secs 2024-04-26T15:12:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12116.875125 secs 2024-04-26T15:12:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:58 smithi194 ceph-mon[33017]: pgmap v7018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:58 smithi033 ceph-mon[27845]: pgmap v7018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:12:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:58 smithi096 ceph-mon[32753]: pgmap v7018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:12:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12119.666463 secs 2024-04-26T15:13:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:12:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12119.666463 secs 2024-04-26T15:13:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:12:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12119.666463 secs 2024-04-26T15:13:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:00 smithi033 ceph-mon[27845]: pgmap v7019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:00 smithi096 ceph-mon[32753]: pgmap v7019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:00 smithi194 ceph-mon[33017]: pgmap v7019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:03 smithi033 ceph-mon[27845]: pgmap v7020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12121.875185 secs 2024-04-26T15:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:03 smithi096 ceph-mon[32753]: pgmap v7020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12121.875185 secs 2024-04-26T15:13:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:03 smithi194 ceph-mon[33017]: pgmap v7020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12121.875185 secs 2024-04-26T15:13:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12124.666578 secs 2024-04-26T15:13:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12124.666578 secs 2024-04-26T15:13:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12124.666578 secs 2024-04-26T15:13:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:05 smithi033 ceph-mon[27845]: pgmap v7021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:05 smithi096 ceph-mon[32753]: pgmap v7021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:05 smithi194 ceph-mon[33017]: pgmap v7021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:06 smithi033 ceph-mon[27845]: pgmap v7022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:06 smithi096 ceph-mon[32753]: pgmap v7022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:06 smithi194 ceph-mon[33017]: pgmap v7022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12126.875289 secs 2024-04-26T15:13:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12126.875289 secs 2024-04-26T15:13:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12126.875289 secs 2024-04-26T15:13:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:08 smithi194 ceph-mon[33017]: pgmap v7023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:08 smithi033 ceph-mon[27845]: pgmap v7023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:08 smithi096 ceph-mon[32753]: pgmap v7023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12129.666712 secs 2024-04-26T15:13:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12129.666712 secs 2024-04-26T15:13:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12129.666712 secs 2024-04-26T15:13:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:10 smithi033 ceph-mon[27845]: pgmap v7024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:10 smithi096 ceph-mon[32753]: pgmap v7024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:10 smithi194 ceph-mon[33017]: pgmap v7024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:11.623+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:11.623+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:11.623+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:13:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:13 smithi033 ceph-mon[27845]: pgmap v7025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12131.875376 secs 2024-04-26T15:13:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:13 smithi096 ceph-mon[32753]: pgmap v7025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12131.875376 secs 2024-04-26T15:13:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:13 smithi194 ceph-mon[33017]: pgmap v7025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12131.875376 secs 2024-04-26T15:13:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12134.666835 secs 2024-04-26T15:13:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12134.666835 secs 2024-04-26T15:13:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12134.666835 secs 2024-04-26T15:13:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:15 smithi033 ceph-mon[27845]: pgmap v7026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:15 smithi096 ceph-mon[32753]: pgmap v7026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:15 smithi194 ceph-mon[33017]: pgmap v7026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:17 smithi033 ceph-mon[27845]: pgmap v7027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:13:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:17 smithi096 ceph-mon[32753]: pgmap v7027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:13:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:17 smithi194 ceph-mon[33017]: pgmap v7027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:13:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12136.875480 secs 2024-04-26T15:13:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:13:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:13:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:13:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12136.875480 secs 2024-04-26T15:13:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:13:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:13:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:13:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12136.875480 secs 2024-04-26T15:13:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:13:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:13:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:13:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:19 smithi033 ceph-mon[27845]: pgmap v7028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:19 smithi033 ceph-mon[27845]: pgmap v7029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12139.666972 secs 2024-04-26T15:13:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:19 smithi096 ceph-mon[32753]: pgmap v7028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:19 smithi096 ceph-mon[32753]: pgmap v7029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12139.666972 secs 2024-04-26T15:13:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:19 smithi194 ceph-mon[33017]: pgmap v7028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:19 smithi194 ceph-mon[33017]: pgmap v7029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12139.666972 secs 2024-04-26T15:13:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:21 smithi033 ceph-mon[27845]: pgmap v7030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:21 smithi096 ceph-mon[32753]: pgmap v7030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:21 smithi194 ceph-mon[33017]: pgmap v7030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:23 smithi096 ceph-mon[32753]: pgmap v7031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12141.875582 secs 2024-04-26T15:13:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:23 smithi194 ceph-mon[33017]: pgmap v7031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12141.875582 secs 2024-04-26T15:13:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:23 smithi033 ceph-mon[27845]: pgmap v7031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12141.875582 secs 2024-04-26T15:13:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12144.667064 secs 2024-04-26T15:13:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12144.667064 secs 2024-04-26T15:13:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12144.667064 secs 2024-04-26T15:13:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:25 smithi194 ceph-mon[33017]: pgmap v7032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:13:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:25 smithi033 ceph-mon[27845]: pgmap v7032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:13:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:25 smithi096 ceph-mon[32753]: pgmap v7032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:13:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:26.632+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:26.632+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:26.632+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:13:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:27 smithi194 ceph-mon[33017]: pgmap v7033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:27 smithi033 ceph-mon[27845]: pgmap v7033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:27 smithi096 ceph-mon[32753]: pgmap v7033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:13:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12146.875681 secs 2024-04-26T15:13:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12146.875681 secs 2024-04-26T15:13:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12146.875681 secs 2024-04-26T15:13:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:29 smithi194 ceph-mon[33017]: pgmap v7034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:13:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12149.667178 secs 2024-04-26T15:13:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:29 smithi033 ceph-mon[27845]: pgmap v7034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:13:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12149.667178 secs 2024-04-26T15:13:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:29 smithi096 ceph-mon[32753]: pgmap v7034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:13:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12149.667178 secs 2024-04-26T15:13:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:31 smithi194 ceph-mon[33017]: pgmap v7035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:31 smithi033 ceph-mon[27845]: pgmap v7035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:31 smithi096 ceph-mon[32753]: pgmap v7035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:33 smithi194 ceph-mon[33017]: pgmap v7036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12151.875779 secs 2024-04-26T15:13:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:33 smithi033 ceph-mon[27845]: pgmap v7036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12151.875779 secs 2024-04-26T15:13:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:33 smithi096 ceph-mon[32753]: pgmap v7036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12151.875779 secs 2024-04-26T15:13:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12154.667293 secs 2024-04-26T15:13:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12154.667293 secs 2024-04-26T15:13:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12154.667293 secs 2024-04-26T15:13:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:35 smithi194 ceph-mon[33017]: pgmap v7037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:35 smithi033 ceph-mon[27845]: pgmap v7037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:35 smithi096 ceph-mon[32753]: pgmap v7037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:37 smithi194 ceph-mon[33017]: pgmap v7038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:37 smithi033 ceph-mon[27845]: pgmap v7038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:37 smithi096 ceph-mon[32753]: pgmap v7038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12156.875880 secs 2024-04-26T15:13:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12156.875880 secs 2024-04-26T15:13:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12156.875880 secs 2024-04-26T15:13:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:39 smithi194 ceph-mon[33017]: pgmap v7039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12159.667357 secs 2024-04-26T15:13:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:39 smithi033 ceph-mon[27845]: pgmap v7039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12159.667357 secs 2024-04-26T15:13:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:39 smithi096 ceph-mon[32753]: pgmap v7039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12159.667357 secs 2024-04-26T15:13:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:41.634+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:41.635+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:41.635+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:13:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:41 smithi194 ceph-mon[33017]: pgmap v7040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:41 smithi033 ceph-mon[27845]: pgmap v7040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:41 smithi096 ceph-mon[32753]: pgmap v7040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:43 smithi194 ceph-mon[33017]: pgmap v7041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12161.875990 secs 2024-04-26T15:13:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:43 smithi033 ceph-mon[27845]: pgmap v7041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12161.875990 secs 2024-04-26T15:13:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:43 smithi096 ceph-mon[32753]: pgmap v7041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12161.875990 secs 2024-04-26T15:13:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12164.667429 secs 2024-04-26T15:13:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12164.667429 secs 2024-04-26T15:13:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12164.667429 secs 2024-04-26T15:13:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:45 smithi194 ceph-mon[33017]: pgmap v7042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:45 smithi033 ceph-mon[27845]: pgmap v7042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:45 smithi096 ceph-mon[32753]: pgmap v7042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:47 smithi194 ceph-mon[33017]: pgmap v7043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:47 smithi033 ceph-mon[27845]: pgmap v7043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:47 smithi096 ceph-mon[32753]: pgmap v7043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12166.876091 secs 2024-04-26T15:13:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12166.876091 secs 2024-04-26T15:13:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12166.876091 secs 2024-04-26T15:13:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12169.667523 secs 2024-04-26T15:13:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:49 smithi194 ceph-mon[33017]: pgmap v7044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12169.667523 secs 2024-04-26T15:13:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:49 smithi033 ceph-mon[27845]: pgmap v7044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12169.667523 secs 2024-04-26T15:13:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:49 smithi096 ceph-mon[32753]: pgmap v7044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:51 smithi194 ceph-mon[33017]: pgmap v7045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:51 smithi033 ceph-mon[27845]: pgmap v7045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:51 smithi096 ceph-mon[32753]: pgmap v7045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:53 smithi194 ceph-mon[33017]: pgmap v7046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12171.876178 secs 2024-04-26T15:13:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:53 smithi033 ceph-mon[27845]: pgmap v7046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12171.876178 secs 2024-04-26T15:13:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:53 smithi096 ceph-mon[32753]: pgmap v7046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12171.876178 secs 2024-04-26T15:13:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12174.667638 secs 2024-04-26T15:13:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12174.667638 secs 2024-04-26T15:13:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12174.667638 secs 2024-04-26T15:13:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:55 smithi194 ceph-mon[33017]: pgmap v7047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:55 smithi033 ceph-mon[27845]: pgmap v7047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:55 smithi096 ceph-mon[32753]: pgmap v7047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:56.641+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:56.641+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:13:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:13:56.641+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:13:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:57 smithi194 ceph-mon[33017]: pgmap v7048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:57 smithi033 ceph-mon[27845]: pgmap v7048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:57 smithi096 ceph-mon[32753]: pgmap v7048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:13:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12176.876266 secs 2024-04-26T15:13:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12176.876266 secs 2024-04-26T15:13:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12176.876266 secs 2024-04-26T15:13:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12179.667753 secs 2024-04-26T15:13:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:13:59 smithi194 ceph-mon[33017]: pgmap v7049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12179.667753 secs 2024-04-26T15:13:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:13:59 smithi033 ceph-mon[27845]: pgmap v7049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:13:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12179.667753 secs 2024-04-26T15:13:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:13:59 smithi096 ceph-mon[32753]: pgmap v7049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:01 smithi194 ceph-mon[33017]: pgmap v7050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:01 smithi033 ceph-mon[27845]: pgmap v7050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:01 smithi096 ceph-mon[32753]: pgmap v7050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:03 smithi194 ceph-mon[33017]: pgmap v7051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12181.876366 secs 2024-04-26T15:14:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:03 smithi033 ceph-mon[27845]: pgmap v7051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12181.876366 secs 2024-04-26T15:14:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:03 smithi096 ceph-mon[32753]: pgmap v7051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12181.876366 secs 2024-04-26T15:14:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12184.667869 secs 2024-04-26T15:14:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12184.667869 secs 2024-04-26T15:14:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12184.667869 secs 2024-04-26T15:14:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:05 smithi033 ceph-mon[27845]: pgmap v7052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:05 smithi096 ceph-mon[32753]: pgmap v7052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:05 smithi194 ceph-mon[33017]: pgmap v7052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:07 smithi033 ceph-mon[27845]: pgmap v7053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:07 smithi096 ceph-mon[32753]: pgmap v7053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:07 smithi194 ceph-mon[33017]: pgmap v7053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12186.876467 secs 2024-04-26T15:14:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12186.876467 secs 2024-04-26T15:14:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12186.876467 secs 2024-04-26T15:14:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12189.668003 secs 2024-04-26T15:14:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:09 smithi033 ceph-mon[27845]: pgmap v7054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12189.668003 secs 2024-04-26T15:14:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:09 smithi096 ceph-mon[32753]: pgmap v7054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12189.668003 secs 2024-04-26T15:14:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:09 smithi194 ceph-mon[33017]: pgmap v7054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:11.647+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:11.647+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:11.647+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:14:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:11 smithi033 ceph-mon[27845]: pgmap v7055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:11 smithi096 ceph-mon[32753]: pgmap v7055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:11 smithi194 ceph-mon[33017]: pgmap v7055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:13 smithi033 ceph-mon[27845]: pgmap v7056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12191.876570 secs 2024-04-26T15:14:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:13 smithi096 ceph-mon[32753]: pgmap v7056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12191.876570 secs 2024-04-26T15:14:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:13 smithi194 ceph-mon[33017]: pgmap v7056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12191.876570 secs 2024-04-26T15:14:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12194.668138 secs 2024-04-26T15:14:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12194.668138 secs 2024-04-26T15:14:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12194.668138 secs 2024-04-26T15:14:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:15 smithi033 ceph-mon[27845]: pgmap v7057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:15 smithi096 ceph-mon[32753]: pgmap v7057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:15 smithi194 ceph-mon[33017]: pgmap v7057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:16 smithi033 ceph-mon[27845]: pgmap v7058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:16 smithi096 ceph-mon[32753]: pgmap v7058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:16 smithi194 ceph-mon[33017]: pgmap v7058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12196.876628 secs 2024-04-26T15:14:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12196.876628 secs 2024-04-26T15:14:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12196.876628 secs 2024-04-26T15:14:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:18 smithi033 ceph-mon[27845]: pgmap v7059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:18 smithi096 ceph-mon[32753]: pgmap v7059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:14:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:18 smithi194 ceph-mon[33017]: pgmap v7059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:14:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12199.668259 secs 2024-04-26T15:14:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12199.668259 secs 2024-04-26T15:14:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12199.668259 secs 2024-04-26T15:14:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:14:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:14:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:20 smithi033 ceph-mon[27845]: pgmap v7060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:14:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:20 smithi033 ceph-mon[27845]: pgmap v7061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:21.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:14:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:14:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:14:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:20 smithi096 ceph-mon[32753]: pgmap v7060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:14:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:20 smithi096 ceph-mon[32753]: pgmap v7061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:14:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:14:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:14:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:20 smithi194 ceph-mon[33017]: pgmap v7060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:14:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:20 smithi194 ceph-mon[33017]: pgmap v7061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:14:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:22 smithi194 ceph-mon[33017]: pgmap v7062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12201.876708 secs 2024-04-26T15:14:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:22 smithi033 ceph-mon[27845]: pgmap v7062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12201.876708 secs 2024-04-26T15:14:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:22 smithi096 ceph-mon[32753]: pgmap v7062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12201.876708 secs 2024-04-26T15:14:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:23.866+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:14:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:23.866+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:14:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:23.867+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:14:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:23.867+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:14:24.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12204.668375 secs 2024-04-26T15:14:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12204.668375 secs 2024-04-26T15:14:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12204.668375 secs 2024-04-26T15:14:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:24 smithi194 ceph-mon[33017]: pgmap v7063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:24 smithi033 ceph-mon[27845]: pgmap v7063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:24 smithi096 ceph-mon[32753]: pgmap v7063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:26.662+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:26.662+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:14:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:26.662+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:26 smithi194 ceph-mon[33017]: pgmap v7064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:26 smithi033 ceph-mon[27845]: pgmap v7064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:26 smithi096 ceph-mon[32753]: pgmap v7064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:28.272 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12206.876793 secs 2024-04-26T15:14:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12206.876793 secs 2024-04-26T15:14:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12206.876793 secs 2024-04-26T15:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:28 smithi194 ceph-mon[33017]: pgmap v7065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12209.668500 secs 2024-04-26T15:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:28 smithi033 ceph-mon[27845]: pgmap v7065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12209.668500 secs 2024-04-26T15:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:28 smithi096 ceph-mon[32753]: pgmap v7065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12209.668500 secs 2024-04-26T15:14:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:30 smithi194 ceph-mon[33017]: pgmap v7066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:14:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:30 smithi033 ceph-mon[27845]: pgmap v7066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:14:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:30 smithi096 ceph-mon[32753]: pgmap v7066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:14:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:33 smithi194 ceph-mon[33017]: pgmap v7067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12211.876878 secs 2024-04-26T15:14:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:33 smithi033 ceph-mon[27845]: pgmap v7067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:33.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12211.876878 secs 2024-04-26T15:14:33.511 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:33 smithi096 ceph-mon[32753]: pgmap v7067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:33.511 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12211.876878 secs 2024-04-26T15:14:34.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12214.668636 secs 2024-04-26T15:14:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12214.668636 secs 2024-04-26T15:14:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12214.668636 secs 2024-04-26T15:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:35 smithi194 ceph-mon[33017]: pgmap v7068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:35 smithi033 ceph-mon[27845]: pgmap v7068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:35 smithi096 ceph-mon[32753]: pgmap v7068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:37 smithi194 ceph-mon[33017]: pgmap v7069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:37 smithi033 ceph-mon[27845]: pgmap v7069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:37 smithi096 ceph-mon[32753]: pgmap v7069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12216.876970 secs 2024-04-26T15:14:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12216.876970 secs 2024-04-26T15:14:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12216.876970 secs 2024-04-26T15:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:39 smithi194 ceph-mon[33017]: pgmap v7070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12219.668790 secs 2024-04-26T15:14:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:39 smithi033 ceph-mon[27845]: pgmap v7070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12219.668790 secs 2024-04-26T15:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:39 smithi096 ceph-mon[32753]: pgmap v7070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12219.668790 secs 2024-04-26T15:14:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:41 smithi194 ceph-mon[33017]: pgmap v7071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:41 smithi033 ceph-mon[27845]: pgmap v7071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:41 smithi096 ceph-mon[32753]: pgmap v7071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:41.679+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:41.679+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:14:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:41.679+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:43 smithi194 ceph-mon[33017]: pgmap v7072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12221.877052 secs 2024-04-26T15:14:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:43 smithi033 ceph-mon[27845]: pgmap v7072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12221.877052 secs 2024-04-26T15:14:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:43 smithi096 ceph-mon[32753]: pgmap v7072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12221.877052 secs 2024-04-26T15:14:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12224.668907 secs 2024-04-26T15:14:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12224.668907 secs 2024-04-26T15:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12224.668907 secs 2024-04-26T15:14:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:45 smithi194 ceph-mon[33017]: pgmap v7073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:45 smithi033 ceph-mon[27845]: pgmap v7073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:45 smithi096 ceph-mon[32753]: pgmap v7073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:47 smithi194 ceph-mon[33017]: pgmap v7074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:47 smithi033 ceph-mon[27845]: pgmap v7074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:47 smithi096 ceph-mon[32753]: pgmap v7074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12226.877115 secs 2024-04-26T15:14:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12226.877115 secs 2024-04-26T15:14:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12226.877115 secs 2024-04-26T15:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:49 smithi194 ceph-mon[33017]: pgmap v7075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12229.669029 secs 2024-04-26T15:14:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:49 smithi033 ceph-mon[27845]: pgmap v7075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12229.669029 secs 2024-04-26T15:14:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:49 smithi096 ceph-mon[32753]: pgmap v7075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12229.669029 secs 2024-04-26T15:14:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:51 smithi194 ceph-mon[33017]: pgmap v7076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:51 smithi033 ceph-mon[27845]: pgmap v7076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:51 smithi096 ceph-mon[32753]: pgmap v7076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:53 smithi194 ceph-mon[33017]: pgmap v7077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12231.877178 secs 2024-04-26T15:14:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:53 smithi033 ceph-mon[27845]: pgmap v7077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12231.877178 secs 2024-04-26T15:14:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:53 smithi096 ceph-mon[32753]: pgmap v7077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12231.877178 secs 2024-04-26T15:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12234.669151 secs 2024-04-26T15:14:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12234.669151 secs 2024-04-26T15:14:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12234.669151 secs 2024-04-26T15:14:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:55 smithi194 ceph-mon[33017]: pgmap v7078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:55 smithi033 ceph-mon[27845]: pgmap v7078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:55 smithi096 ceph-mon[32753]: pgmap v7078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:56.689+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:56.689+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:14:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:14:56.689+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:14:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:57 smithi033 ceph-mon[27845]: pgmap v7079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:57 smithi096 ceph-mon[32753]: pgmap v7079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:57 smithi194 ceph-mon[33017]: pgmap v7079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:14:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12236.877266 secs 2024-04-26T15:14:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12236.877266 secs 2024-04-26T15:14:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12236.877266 secs 2024-04-26T15:14:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:59 smithi033 ceph-mon[27845]: pgmap v7080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12239.669281 secs 2024-04-26T15:14:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:59 smithi096 ceph-mon[32753]: pgmap v7080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12239.669281 secs 2024-04-26T15:14:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:59 smithi194 ceph-mon[33017]: pgmap v7080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:14:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12239.669281 secs 2024-04-26T15:15:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:01 smithi033 ceph-mon[27845]: pgmap v7081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:01 smithi096 ceph-mon[32753]: pgmap v7081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:01 smithi194 ceph-mon[33017]: pgmap v7081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:03 smithi033 ceph-mon[27845]: pgmap v7082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12241.877349 secs 2024-04-26T15:15:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:03 smithi096 ceph-mon[32753]: pgmap v7082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12241.877349 secs 2024-04-26T15:15:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:03 smithi194 ceph-mon[33017]: pgmap v7082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12241.877349 secs 2024-04-26T15:15:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12244.669402 secs 2024-04-26T15:15:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12244.669402 secs 2024-04-26T15:15:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12244.669402 secs 2024-04-26T15:15:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:05 smithi033 ceph-mon[27845]: pgmap v7083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:05 smithi096 ceph-mon[32753]: pgmap v7083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:05 smithi194 ceph-mon[33017]: pgmap v7083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:07 smithi033 ceph-mon[27845]: pgmap v7084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:07 smithi096 ceph-mon[32753]: pgmap v7084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:07 smithi194 ceph-mon[33017]: pgmap v7084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12246.877435 secs 2024-04-26T15:15:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12246.877435 secs 2024-04-26T15:15:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12246.877435 secs 2024-04-26T15:15:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:09 smithi033 ceph-mon[27845]: pgmap v7085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12249.669470 secs 2024-04-26T15:15:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:09 smithi096 ceph-mon[32753]: pgmap v7085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12249.669470 secs 2024-04-26T15:15:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:09 smithi194 ceph-mon[33017]: pgmap v7085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12249.669470 secs 2024-04-26T15:15:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:11 smithi033 ceph-mon[27845]: pgmap v7086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:11 smithi096 ceph-mon[32753]: pgmap v7086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:11 smithi194 ceph-mon[33017]: pgmap v7086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:11.699+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:11.699+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:15:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:11.699+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:13 smithi033 ceph-mon[27845]: pgmap v7087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12251.877517 secs 2024-04-26T15:15:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:13 smithi096 ceph-mon[32753]: pgmap v7087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12251.877517 secs 2024-04-26T15:15:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:13 smithi194 ceph-mon[33017]: pgmap v7087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12251.877517 secs 2024-04-26T15:15:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12254.669581 secs 2024-04-26T15:15:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12254.669581 secs 2024-04-26T15:15:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12254.669581 secs 2024-04-26T15:15:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:15 smithi194 ceph-mon[33017]: pgmap v7088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:15 smithi033 ceph-mon[27845]: pgmap v7088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:15 smithi096 ceph-mon[32753]: pgmap v7088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:17 smithi194 ceph-mon[33017]: pgmap v7089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:17 smithi033 ceph-mon[27845]: pgmap v7089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:17 smithi096 ceph-mon[32753]: pgmap v7089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12256.877601 secs 2024-04-26T15:15:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12256.877601 secs 2024-04-26T15:15:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12256.877601 secs 2024-04-26T15:15:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:19 smithi194 ceph-mon[33017]: pgmap v7090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12259.669699 secs 2024-04-26T15:15:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:19 smithi033 ceph-mon[27845]: pgmap v7090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12259.669699 secs 2024-04-26T15:15:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:19 smithi096 ceph-mon[32753]: pgmap v7090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12259.669699 secs 2024-04-26T15:15:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:21 smithi194 ceph-mon[33017]: pgmap v7091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:15:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:15:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:21 smithi033 ceph-mon[27845]: pgmap v7091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:15:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:15:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:21 smithi096 ceph-mon[32753]: pgmap v7091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:15:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:15:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:15:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:15:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:22 smithi194 ceph-mon[33017]: pgmap v7092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:15:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:22 smithi194 ceph-mon[33017]: pgmap v7093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:15:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:15:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:15:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:22 smithi033 ceph-mon[27845]: pgmap v7092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:15:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:22 smithi033 ceph-mon[27845]: pgmap v7093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:15:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:15:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:15:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:22 smithi096 ceph-mon[32753]: pgmap v7092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:15:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:22 smithi096 ceph-mon[32753]: pgmap v7093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:15:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12261.877683 secs 2024-04-26T15:15:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12261.877683 secs 2024-04-26T15:15:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12261.877683 secs 2024-04-26T15:15:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12264.669834 secs 2024-04-26T15:15:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:24 smithi096 ceph-mon[32753]: pgmap v7094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12264.669834 secs 2024-04-26T15:15:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:24 smithi194 ceph-mon[33017]: pgmap v7094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12264.669834 secs 2024-04-26T15:15:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:24 smithi033 ceph-mon[27845]: pgmap v7094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:25 smithi194 ceph-mon[33017]: pgmap v7095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:25 smithi033 ceph-mon[27845]: pgmap v7095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:25 smithi096 ceph-mon[32753]: pgmap v7095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:26.713+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:26.713+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:26.713+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:15:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:27 smithi194 ceph-mon[33017]: pgmap v7096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:27 smithi033 ceph-mon[27845]: pgmap v7096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:27 smithi096 ceph-mon[32753]: pgmap v7096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12266.877762 secs 2024-04-26T15:15:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12266.877762 secs 2024-04-26T15:15:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12266.877762 secs 2024-04-26T15:15:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12269.669950 secs 2024-04-26T15:15:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:29 smithi194 ceph-mon[33017]: pgmap v7097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12269.669950 secs 2024-04-26T15:15:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:29 smithi033 ceph-mon[27845]: pgmap v7097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12269.669950 secs 2024-04-26T15:15:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:29 smithi096 ceph-mon[32753]: pgmap v7097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:15:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:31 smithi194 ceph-mon[33017]: pgmap v7098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:31 smithi033 ceph-mon[27845]: pgmap v7098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:31 smithi096 ceph-mon[32753]: pgmap v7098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:15:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12271.877843 secs 2024-04-26T15:15:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12271.877843 secs 2024-04-26T15:15:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12271.877843 secs 2024-04-26T15:15:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:33 smithi033 ceph-mon[27845]: pgmap v7099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:15:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:33 smithi096 ceph-mon[32753]: pgmap v7099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:15:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:33 smithi194 ceph-mon[33017]: pgmap v7099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:15:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12274.670063 secs 2024-04-26T15:15:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12274.670063 secs 2024-04-26T15:15:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12274.670063 secs 2024-04-26T15:15:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:35 smithi033 ceph-mon[27845]: pgmap v7100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:15:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:35 smithi096 ceph-mon[32753]: pgmap v7100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:15:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:35 smithi194 ceph-mon[33017]: pgmap v7100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:15:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:37 smithi194 ceph-mon[33017]: pgmap v7101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:37 smithi033 ceph-mon[27845]: pgmap v7101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:37 smithi096 ceph-mon[32753]: pgmap v7101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12276.877928 secs 2024-04-26T15:15:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12276.877928 secs 2024-04-26T15:15:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12276.877928 secs 2024-04-26T15:15:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12279.670179 secs 2024-04-26T15:15:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:39 smithi194 ceph-mon[33017]: pgmap v7102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12279.670179 secs 2024-04-26T15:15:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:39 smithi033 ceph-mon[27845]: pgmap v7102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12279.670179 secs 2024-04-26T15:15:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:39 smithi096 ceph-mon[32753]: pgmap v7102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:41.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:41.729+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:41.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:41.729+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:41.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:41.729+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:15:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:41 smithi194 ceph-mon[33017]: pgmap v7103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:41 smithi033 ceph-mon[27845]: pgmap v7103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:41 smithi096 ceph-mon[32753]: pgmap v7103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12281.878024 secs 2024-04-26T15:15:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12281.878024 secs 2024-04-26T15:15:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12281.878024 secs 2024-04-26T15:15:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:43 smithi033 ceph-mon[27845]: pgmap v7104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:43 smithi096 ceph-mon[32753]: pgmap v7104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:43 smithi194 ceph-mon[33017]: pgmap v7104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12284.670295 secs 2024-04-26T15:15:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12284.670295 secs 2024-04-26T15:15:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12284.670295 secs 2024-04-26T15:15:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:45 smithi033 ceph-mon[27845]: pgmap v7105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:45 smithi096 ceph-mon[32753]: pgmap v7105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:45 smithi194 ceph-mon[33017]: pgmap v7105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:47 smithi194 ceph-mon[33017]: pgmap v7106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:47 smithi033 ceph-mon[27845]: pgmap v7106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:47 smithi096 ceph-mon[32753]: pgmap v7106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:49.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12286.878108 secs 2024-04-26T15:15:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12286.878108 secs 2024-04-26T15:15:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12286.878108 secs 2024-04-26T15:15:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12289.670417 secs 2024-04-26T15:15:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:49 smithi194 ceph-mon[33017]: pgmap v7107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12289.670417 secs 2024-04-26T15:15:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:49 smithi033 ceph-mon[27845]: pgmap v7107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12289.670417 secs 2024-04-26T15:15:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:49 smithi096 ceph-mon[32753]: pgmap v7107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:51 smithi194 ceph-mon[33017]: pgmap v7108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:51 smithi033 ceph-mon[27845]: pgmap v7108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:51 smithi096 ceph-mon[32753]: pgmap v7108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12291.878197 secs 2024-04-26T15:15:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12291.878197 secs 2024-04-26T15:15:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12291.878197 secs 2024-04-26T15:15:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:53.882+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:15:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:53.882+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:15:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:53.883+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:15:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:53.883+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:15:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:53 smithi033 ceph-mon[27845]: pgmap v7109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:53 smithi096 ceph-mon[32753]: pgmap v7109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:53 smithi194 ceph-mon[33017]: pgmap v7109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12294.670536 secs 2024-04-26T15:15:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12294.670536 secs 2024-04-26T15:15:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12294.670536 secs 2024-04-26T15:15:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:55 smithi033 ceph-mon[27845]: pgmap v7110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:55 smithi096 ceph-mon[32753]: pgmap v7110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:55 smithi194 ceph-mon[33017]: pgmap v7110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:15:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:56.743+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:56.743+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:15:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:15:56.743+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:15:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:57 smithi194 ceph-mon[33017]: pgmap v7111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:57 smithi033 ceph-mon[27845]: pgmap v7111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:57 smithi096 ceph-mon[32753]: pgmap v7111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:15:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12296.878280 secs 2024-04-26T15:15:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12296.878280 secs 2024-04-26T15:15:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12296.878280 secs 2024-04-26T15:16:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12299.670656 secs 2024-04-26T15:16:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:15:59 smithi194 ceph-mon[33017]: pgmap v7112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12299.670656 secs 2024-04-26T15:16:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:15:59 smithi033 ceph-mon[27845]: pgmap v7112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12299.670656 secs 2024-04-26T15:16:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:15:59 smithi096 ceph-mon[32753]: pgmap v7112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:02 smithi194 ceph-mon[33017]: pgmap v7113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:02 smithi033 ceph-mon[27845]: pgmap v7113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:02 smithi096 ceph-mon[32753]: pgmap v7113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12301.878363 secs 2024-04-26T15:16:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12301.878363 secs 2024-04-26T15:16:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12301.878363 secs 2024-04-26T15:16:04.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12304.670790 secs 2024-04-26T15:16:04.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:04 smithi033 ceph-mon[27845]: pgmap v7114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12304.670790 secs 2024-04-26T15:16:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:04 smithi194 ceph-mon[33017]: pgmap v7114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12304.670790 secs 2024-04-26T15:16:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:04 smithi096 ceph-mon[32753]: pgmap v7114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:05 smithi194 ceph-mon[33017]: pgmap v7115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:05 smithi033 ceph-mon[27845]: pgmap v7115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:05 smithi096 ceph-mon[32753]: pgmap v7115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:07 smithi194 ceph-mon[33017]: pgmap v7116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:07 smithi033 ceph-mon[27845]: pgmap v7116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:07 smithi096 ceph-mon[32753]: pgmap v7116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12306.878422 secs 2024-04-26T15:16:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12306.878422 secs 2024-04-26T15:16:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12306.878422 secs 2024-04-26T15:16:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12309.670908 secs 2024-04-26T15:16:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:09 smithi194 ceph-mon[33017]: pgmap v7117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12309.670908 secs 2024-04-26T15:16:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:09 smithi033 ceph-mon[27845]: pgmap v7117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12309.670908 secs 2024-04-26T15:16:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:09 smithi096 ceph-mon[32753]: pgmap v7117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:11.752+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:11.752+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:11.752+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:16:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:11 smithi194 ceph-mon[33017]: pgmap v7118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:11 smithi033 ceph-mon[27845]: pgmap v7118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:11 smithi096 ceph-mon[32753]: pgmap v7118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12311.878504 secs 2024-04-26T15:16:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12311.878504 secs 2024-04-26T15:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12311.878504 secs 2024-04-26T15:16:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:13 smithi033 ceph-mon[27845]: pgmap v7119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:13 smithi096 ceph-mon[32753]: pgmap v7119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:13 smithi194 ceph-mon[33017]: pgmap v7119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12314.671019 secs 2024-04-26T15:16:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12314.671019 secs 2024-04-26T15:16:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12314.671019 secs 2024-04-26T15:16:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:15 smithi033 ceph-mon[27845]: pgmap v7120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:15 smithi096 ceph-mon[32753]: pgmap v7120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:15 smithi194 ceph-mon[33017]: pgmap v7120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:17 smithi194 ceph-mon[33017]: pgmap v7121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:17 smithi033 ceph-mon[27845]: pgmap v7121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:17 smithi096 ceph-mon[32753]: pgmap v7121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:19.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12316.878589 secs 2024-04-26T15:16:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12316.878589 secs 2024-04-26T15:16:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12316.878589 secs 2024-04-26T15:16:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12319.671136 secs 2024-04-26T15:16:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:19 smithi194 ceph-mon[33017]: pgmap v7122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12319.671136 secs 2024-04-26T15:16:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:19 smithi033 ceph-mon[27845]: pgmap v7122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12319.671136 secs 2024-04-26T15:16:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:19 smithi096 ceph-mon[32753]: pgmap v7122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:21 smithi194 ceph-mon[33017]: pgmap v7123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:21 smithi033 ceph-mon[27845]: pgmap v7123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:21 smithi096 ceph-mon[32753]: pgmap v7123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:16:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:16:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:16:24.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12321.878674 secs 2024-04-26T15:16:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:16:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:16:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:23 smithi096 ceph-mon[32753]: pgmap v7124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:23 smithi096 ceph-mon[32753]: pgmap v7125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:24.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:16:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12321.878674 secs 2024-04-26T15:16:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:16:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:16:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:23 smithi194 ceph-mon[33017]: pgmap v7124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:23 smithi194 ceph-mon[33017]: pgmap v7125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:16:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12321.878674 secs 2024-04-26T15:16:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:16:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:16:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:23 smithi033 ceph-mon[27845]: pgmap v7124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:23 smithi033 ceph-mon[27845]: pgmap v7125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:16:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12324.671185 secs 2024-04-26T15:16:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12324.671185 secs 2024-04-26T15:16:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12324.671185 secs 2024-04-26T15:16:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:25 smithi194 ceph-mon[33017]: pgmap v7126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:25 smithi033 ceph-mon[27845]: pgmap v7126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:25 smithi096 ceph-mon[32753]: pgmap v7126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:26.761+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:26.761+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:16:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:26.761+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:28 smithi033 ceph-mon[27845]: pgmap v7127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12326.878759 secs 2024-04-26T15:16:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:28 smithi096 ceph-mon[32753]: pgmap v7127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12326.878759 secs 2024-04-26T15:16:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:28 smithi194 ceph-mon[33017]: pgmap v7127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12326.878759 secs 2024-04-26T15:16:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12329.671265 secs 2024-04-26T15:16:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12329.671265 secs 2024-04-26T15:16:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12329.671265 secs 2024-04-26T15:16:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:30 smithi096 ceph-mon[32753]: pgmap v7128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:16:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:30 smithi194 ceph-mon[33017]: pgmap v7128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:16:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:30 smithi033 ceph-mon[27845]: pgmap v7128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:16:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:32 smithi194 ceph-mon[33017]: pgmap v7129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:32 smithi033 ceph-mon[27845]: pgmap v7129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:32 smithi096 ceph-mon[32753]: pgmap v7129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:16:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12331.878842 secs 2024-04-26T15:16:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12331.878842 secs 2024-04-26T15:16:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12331.878842 secs 2024-04-26T15:16:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12334.671370 secs 2024-04-26T15:16:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:34 smithi194 ceph-mon[33017]: pgmap v7130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:16:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12334.671370 secs 2024-04-26T15:16:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:34 smithi033 ceph-mon[27845]: pgmap v7130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:16:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12334.671370 secs 2024-04-26T15:16:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:34 smithi096 ceph-mon[32753]: pgmap v7130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:16:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:36 smithi194 ceph-mon[33017]: pgmap v7131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:36 smithi033 ceph-mon[27845]: pgmap v7131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:36 smithi096 ceph-mon[32753]: pgmap v7131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:38 smithi194 ceph-mon[33017]: pgmap v7132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12336.878929 secs 2024-04-26T15:16:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:38 smithi033 ceph-mon[27845]: pgmap v7132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12336.878929 secs 2024-04-26T15:16:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:38 smithi096 ceph-mon[32753]: pgmap v7132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12336.878929 secs 2024-04-26T15:16:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12339.671478 secs 2024-04-26T15:16:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12339.671478 secs 2024-04-26T15:16:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12339.671478 secs 2024-04-26T15:16:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:40 smithi194 ceph-mon[33017]: pgmap v7133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:40 smithi033 ceph-mon[27845]: pgmap v7133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:40 smithi096 ceph-mon[32753]: pgmap v7133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:41.768+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:41.768+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:41.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:41.768+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:16:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:42 smithi194 ceph-mon[33017]: pgmap v7134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:42 smithi033 ceph-mon[27845]: pgmap v7134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:42 smithi096 ceph-mon[32753]: pgmap v7134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12341.879016 secs 2024-04-26T15:16:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12341.879016 secs 2024-04-26T15:16:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12341.879016 secs 2024-04-26T15:16:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12344.671599 secs 2024-04-26T15:16:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:44 smithi194 ceph-mon[33017]: pgmap v7135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12344.671599 secs 2024-04-26T15:16:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:44 smithi033 ceph-mon[27845]: pgmap v7135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12344.671599 secs 2024-04-26T15:16:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:44 smithi096 ceph-mon[32753]: pgmap v7135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:46 smithi194 ceph-mon[33017]: pgmap v7136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:47.104 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:46 smithi096 ceph-mon[32753]: pgmap v7136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:47.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:46 smithi033 ceph-mon[27845]: pgmap v7136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:48 smithi194 ceph-mon[33017]: pgmap v7137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12346.879104 secs 2024-04-26T15:16:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:48 smithi033 ceph-mon[27845]: pgmap v7137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12346.879104 secs 2024-04-26T15:16:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:48 smithi096 ceph-mon[32753]: pgmap v7137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12346.879104 secs 2024-04-26T15:16:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12349.671711 secs 2024-04-26T15:16:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12349.671711 secs 2024-04-26T15:16:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12349.671711 secs 2024-04-26T15:16:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:50 smithi194 ceph-mon[33017]: pgmap v7138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:50 smithi033 ceph-mon[27845]: pgmap v7138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:50 smithi096 ceph-mon[32753]: pgmap v7138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:52 smithi194 ceph-mon[33017]: pgmap v7139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:52 smithi033 ceph-mon[27845]: pgmap v7139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:52 smithi096 ceph-mon[32753]: pgmap v7139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12351.879162 secs 2024-04-26T15:16:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12351.879162 secs 2024-04-26T15:16:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12351.879162 secs 2024-04-26T15:16:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12354.671829 secs 2024-04-26T15:16:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:54 smithi096 ceph-mon[32753]: pgmap v7140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12354.671829 secs 2024-04-26T15:16:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:54 smithi194 ceph-mon[33017]: pgmap v7140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12354.671829 secs 2024-04-26T15:16:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:54 smithi033 ceph-mon[27845]: pgmap v7140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:56.777+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:56.778+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:16:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:16:56.778+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:16:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:56 smithi194 ceph-mon[33017]: pgmap v7141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:56 smithi033 ceph-mon[27845]: pgmap v7141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:56 smithi096 ceph-mon[32753]: pgmap v7141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:16:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:58 smithi194 ceph-mon[33017]: pgmap v7142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12356.879223 secs 2024-04-26T15:16:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:58 smithi033 ceph-mon[27845]: pgmap v7142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12356.879223 secs 2024-04-26T15:16:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:58 smithi096 ceph-mon[32753]: pgmap v7142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:16:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12356.879223 secs 2024-04-26T15:16:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12359.671944 secs 2024-04-26T15:16:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12359.671944 secs 2024-04-26T15:16:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12359.671944 secs 2024-04-26T15:17:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:00 smithi194 ceph-mon[33017]: pgmap v7143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:00 smithi033 ceph-mon[27845]: pgmap v7143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:00 smithi096 ceph-mon[32753]: pgmap v7143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:02 smithi194 ceph-mon[33017]: pgmap v7144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:02 smithi033 ceph-mon[27845]: pgmap v7144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:02 smithi096 ceph-mon[32753]: pgmap v7144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12361.879284 secs 2024-04-26T15:17:03.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12361.879284 secs 2024-04-26T15:17:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12361.879284 secs 2024-04-26T15:17:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12364.672060 secs 2024-04-26T15:17:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:04 smithi194 ceph-mon[33017]: pgmap v7145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12364.672060 secs 2024-04-26T15:17:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:04 smithi033 ceph-mon[27845]: pgmap v7145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12364.672060 secs 2024-04-26T15:17:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:04 smithi096 ceph-mon[32753]: pgmap v7145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:06 smithi194 ceph-mon[33017]: pgmap v7146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:06 smithi033 ceph-mon[27845]: pgmap v7146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:06 smithi096 ceph-mon[32753]: pgmap v7146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:08 smithi194 ceph-mon[33017]: pgmap v7147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12366.879345 secs 2024-04-26T15:17:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:08 smithi033 ceph-mon[27845]: pgmap v7147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12366.879345 secs 2024-04-26T15:17:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:08 smithi096 ceph-mon[32753]: pgmap v7147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12366.879345 secs 2024-04-26T15:17:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12369.672163 secs 2024-04-26T15:17:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12369.672163 secs 2024-04-26T15:17:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12369.672163 secs 2024-04-26T15:17:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:10 smithi194 ceph-mon[33017]: pgmap v7148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:10 smithi033 ceph-mon[27845]: pgmap v7148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:10 smithi096 ceph-mon[32753]: pgmap v7148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:11.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:11.786+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:11.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:11.786+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:11.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:11.786+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:17:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:12 smithi033 ceph-mon[27845]: pgmap v7149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:12 smithi096 ceph-mon[32753]: pgmap v7149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:12 smithi194 ceph-mon[33017]: pgmap v7149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12371.879406 secs 2024-04-26T15:17:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12371.879406 secs 2024-04-26T15:17:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12371.879406 secs 2024-04-26T15:17:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12374.672235 secs 2024-04-26T15:17:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:14 smithi033 ceph-mon[27845]: pgmap v7150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12374.672235 secs 2024-04-26T15:17:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:14 smithi096 ceph-mon[32753]: pgmap v7150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12374.672235 secs 2024-04-26T15:17:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:14 smithi194 ceph-mon[33017]: pgmap v7150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:16 smithi033 ceph-mon[27845]: pgmap v7151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:16 smithi096 ceph-mon[32753]: pgmap v7151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:16 smithi194 ceph-mon[33017]: pgmap v7151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:18 smithi033 ceph-mon[27845]: pgmap v7152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12376.879492 secs 2024-04-26T15:17:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:18 smithi096 ceph-mon[32753]: pgmap v7152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12376.879492 secs 2024-04-26T15:17:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:18 smithi194 ceph-mon[33017]: pgmap v7152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12376.879492 secs 2024-04-26T15:17:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12379.672297 secs 2024-04-26T15:17:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12379.672297 secs 2024-04-26T15:17:20.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12379.672297 secs 2024-04-26T15:17:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:20 smithi033 ceph-mon[27845]: pgmap v7153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:20 smithi096 ceph-mon[32753]: pgmap v7153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:20 smithi194 ceph-mon[33017]: pgmap v7153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:22 smithi033 ceph-mon[27845]: pgmap v7154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:22 smithi096 ceph-mon[32753]: pgmap v7154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:22 smithi194 ceph-mon[33017]: pgmap v7154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:23.897+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:17:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:23.897+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:17:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:23.898+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:17:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:23.898+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:17:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12381.879553 secs 2024-04-26T15:17:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12381.879553 secs 2024-04-26T15:17:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12381.879553 secs 2024-04-26T15:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12384.672404 secs 2024-04-26T15:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:24 smithi033 ceph-mon[27845]: pgmap v7155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12384.672404 secs 2024-04-26T15:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:24 smithi096 ceph-mon[32753]: pgmap v7155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12384.672404 secs 2024-04-26T15:17:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:24 smithi194 ceph-mon[33017]: pgmap v7155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:17:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:26.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:26.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:26.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:26.789+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:26.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:26.789+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:17:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:26 smithi096 ceph-mon[32753]: pgmap v7156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:17:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:17:26.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:26 smithi194 ceph-mon[33017]: pgmap v7156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:17:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:17:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:26 smithi033 ceph-mon[27845]: pgmap v7156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:17:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:17:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:17:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:27 smithi194 ceph-mon[33017]: pgmap v7157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:17:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:27 smithi194 ceph-mon[33017]: pgmap v7158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:27 smithi033 ceph-mon[27845]: pgmap v7157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:17:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:27 smithi033 ceph-mon[27845]: pgmap v7158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:27 smithi096 ceph-mon[32753]: pgmap v7157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:17:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:27 smithi096 ceph-mon[32753]: pgmap v7158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12386.879636 secs 2024-04-26T15:17:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12386.879636 secs 2024-04-26T15:17:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12386.879636 secs 2024-04-26T15:17:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:29 smithi194 ceph-mon[33017]: pgmap v7159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:17:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12389.672509 secs 2024-04-26T15:17:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:29 smithi033 ceph-mon[27845]: pgmap v7159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:17:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12389.672509 secs 2024-04-26T15:17:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:29 smithi096 ceph-mon[32753]: pgmap v7159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:17:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12389.672509 secs 2024-04-26T15:17:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:30 smithi194 ceph-mon[33017]: pgmap v7160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:30 smithi033 ceph-mon[27845]: pgmap v7160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:30 smithi096 ceph-mon[32753]: pgmap v7160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:33 smithi194 ceph-mon[33017]: pgmap v7161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12391.879720 secs 2024-04-26T15:17:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:33 smithi033 ceph-mon[27845]: pgmap v7161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12391.879720 secs 2024-04-26T15:17:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:33 smithi096 ceph-mon[32753]: pgmap v7161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:17:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12391.879720 secs 2024-04-26T15:17:34.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12394.672580 secs 2024-04-26T15:17:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12394.672580 secs 2024-04-26T15:17:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12394.672580 secs 2024-04-26T15:17:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:35 smithi194 ceph-mon[33017]: pgmap v7162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:17:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:35 smithi033 ceph-mon[27845]: pgmap v7162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:17:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:35 smithi096 ceph-mon[32753]: pgmap v7162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:17:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:37 smithi194 ceph-mon[33017]: pgmap v7163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:17:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:37 smithi033 ceph-mon[27845]: pgmap v7163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:17:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:37 smithi096 ceph-mon[32753]: pgmap v7163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:17:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12396.879822 secs 2024-04-26T15:17:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12396.879822 secs 2024-04-26T15:17:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12396.879822 secs 2024-04-26T15:17:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:39 smithi194 ceph-mon[33017]: pgmap v7164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12399.672702 secs 2024-04-26T15:17:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:39 smithi033 ceph-mon[27845]: pgmap v7164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12399.672702 secs 2024-04-26T15:17:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:39 smithi096 ceph-mon[32753]: pgmap v7164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12399.672702 secs 2024-04-26T15:17:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:41 smithi194 ceph-mon[33017]: pgmap v7165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:41 smithi033 ceph-mon[27845]: pgmap v7165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:41 smithi096 ceph-mon[32753]: pgmap v7165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:41.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:41.805+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:41.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:41.805+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:41.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:41.805+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:17:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:43 smithi194 ceph-mon[33017]: pgmap v7166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12401.879930 secs 2024-04-26T15:17:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:43 smithi033 ceph-mon[27845]: pgmap v7166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12401.879930 secs 2024-04-26T15:17:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:43 smithi096 ceph-mon[32753]: pgmap v7166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12401.879930 secs 2024-04-26T15:17:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12404.672775 secs 2024-04-26T15:17:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12404.672775 secs 2024-04-26T15:17:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12404.672775 secs 2024-04-26T15:17:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:45 smithi194 ceph-mon[33017]: pgmap v7167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:45 smithi033 ceph-mon[27845]: pgmap v7167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:45 smithi096 ceph-mon[32753]: pgmap v7167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:47 smithi033 ceph-mon[27845]: pgmap v7168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:47 smithi096 ceph-mon[32753]: pgmap v7168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:47 smithi194 ceph-mon[33017]: pgmap v7168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12406.880016 secs 2024-04-26T15:17:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12406.880016 secs 2024-04-26T15:17:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12406.880016 secs 2024-04-26T15:17:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:49 smithi033 ceph-mon[27845]: pgmap v7169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12409.672918 secs 2024-04-26T15:17:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:49 smithi096 ceph-mon[32753]: pgmap v7169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12409.672918 secs 2024-04-26T15:17:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:49 smithi194 ceph-mon[33017]: pgmap v7169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12409.672918 secs 2024-04-26T15:17:51.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:51 smithi194 ceph-mon[33017]: pgmap v7170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:51 smithi033 ceph-mon[27845]: pgmap v7170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:51 smithi096 ceph-mon[32753]: pgmap v7170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:53 smithi033 ceph-mon[27845]: pgmap v7171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12411.880096 secs 2024-04-26T15:17:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:53 smithi096 ceph-mon[32753]: pgmap v7171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12411.880096 secs 2024-04-26T15:17:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:53 smithi194 ceph-mon[33017]: pgmap v7171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12411.880096 secs 2024-04-26T15:17:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12414.673033 secs 2024-04-26T15:17:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12414.673033 secs 2024-04-26T15:17:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12414.673033 secs 2024-04-26T15:17:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:55 smithi033 ceph-mon[27845]: pgmap v7172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:55 smithi096 ceph-mon[32753]: pgmap v7172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:55 smithi194 ceph-mon[33017]: pgmap v7172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:56.814+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:56.814+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:17:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:17:56.814+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:17:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:57 smithi033 ceph-mon[27845]: pgmap v7173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:57 smithi096 ceph-mon[32753]: pgmap v7173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:57 smithi194 ceph-mon[33017]: pgmap v7173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:17:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12416.880155 secs 2024-04-26T15:17:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12416.880155 secs 2024-04-26T15:17:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12416.880155 secs 2024-04-26T15:17:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:59 smithi033 ceph-mon[27845]: pgmap v7174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12419.673154 secs 2024-04-26T15:17:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:59 smithi096 ceph-mon[32753]: pgmap v7174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12419.673154 secs 2024-04-26T15:17:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:59 smithi194 ceph-mon[33017]: pgmap v7174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:17:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12419.673154 secs 2024-04-26T15:18:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:01 smithi033 ceph-mon[27845]: pgmap v7175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:01 smithi096 ceph-mon[32753]: pgmap v7175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:01 smithi194 ceph-mon[33017]: pgmap v7175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:03 smithi033 ceph-mon[27845]: pgmap v7176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12421.880218 secs 2024-04-26T15:18:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:03 smithi096 ceph-mon[32753]: pgmap v7176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12421.880218 secs 2024-04-26T15:18:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:03 smithi194 ceph-mon[33017]: pgmap v7176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12421.880218 secs 2024-04-26T15:18:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12424.673253 secs 2024-04-26T15:18:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12424.673253 secs 2024-04-26T15:18:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12424.673253 secs 2024-04-26T15:18:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:05 smithi096 ceph-mon[32753]: pgmap v7177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:05 smithi194 ceph-mon[33017]: pgmap v7177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:05 smithi033 ceph-mon[27845]: pgmap v7177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:07 smithi194 ceph-mon[33017]: pgmap v7178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:07 smithi033 ceph-mon[27845]: pgmap v7178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:07 smithi096 ceph-mon[32753]: pgmap v7178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12426.880302 secs 2024-04-26T15:18:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12426.880302 secs 2024-04-26T15:18:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12426.880302 secs 2024-04-26T15:18:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:09 smithi194 ceph-mon[33017]: pgmap v7179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12429.673373 secs 2024-04-26T15:18:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:09 smithi033 ceph-mon[27845]: pgmap v7179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12429.673373 secs 2024-04-26T15:18:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:09 smithi096 ceph-mon[32753]: pgmap v7179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12429.673373 secs 2024-04-26T15:18:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:11 smithi194 ceph-mon[33017]: pgmap v7180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:11 smithi033 ceph-mon[27845]: pgmap v7180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:11 smithi096 ceph-mon[32753]: pgmap v7180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:11.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:11.819+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:11.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:11.819+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:11.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:11.819+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:18:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:13 smithi194 ceph-mon[33017]: pgmap v7181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12431.880362 secs 2024-04-26T15:18:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:13 smithi033 ceph-mon[27845]: pgmap v7181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12431.880362 secs 2024-04-26T15:18:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:13 smithi096 ceph-mon[32753]: pgmap v7181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12431.880362 secs 2024-04-26T15:18:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12434.673490 secs 2024-04-26T15:18:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12434.673490 secs 2024-04-26T15:18:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12434.673490 secs 2024-04-26T15:18:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:15 smithi194 ceph-mon[33017]: pgmap v7182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:15 smithi033 ceph-mon[27845]: pgmap v7182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:15 smithi096 ceph-mon[32753]: pgmap v7182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:17 smithi194 ceph-mon[33017]: pgmap v7183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:17 smithi033 ceph-mon[27845]: pgmap v7183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:17 smithi096 ceph-mon[32753]: pgmap v7183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12436.880446 secs 2024-04-26T15:18:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12436.880446 secs 2024-04-26T15:18:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12436.880446 secs 2024-04-26T15:18:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:19 smithi194 ceph-mon[33017]: pgmap v7184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12439.673581 secs 2024-04-26T15:18:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:19 smithi033 ceph-mon[27845]: pgmap v7184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12439.673581 secs 2024-04-26T15:18:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:19 smithi096 ceph-mon[32753]: pgmap v7184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12439.673581 secs 2024-04-26T15:18:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:21 smithi194 ceph-mon[33017]: pgmap v7185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:21 smithi033 ceph-mon[27845]: pgmap v7185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:21 smithi096 ceph-mon[32753]: pgmap v7185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:23 smithi194 ceph-mon[33017]: pgmap v7186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12441.880506 secs 2024-04-26T15:18:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:23 smithi033 ceph-mon[27845]: pgmap v7186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12441.880506 secs 2024-04-26T15:18:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:23 smithi096 ceph-mon[32753]: pgmap v7186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12441.880506 secs 2024-04-26T15:18:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12444.673697 secs 2024-04-26T15:18:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12444.673697 secs 2024-04-26T15:18:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12444.673697 secs 2024-04-26T15:18:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:25 smithi194 ceph-mon[33017]: pgmap v7187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:25 smithi033 ceph-mon[27845]: pgmap v7187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:25 smithi096 ceph-mon[32753]: pgmap v7187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:26.826+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:26.826+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:26.826+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:18:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:27 smithi194 ceph-mon[33017]: pgmap v7188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:18:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:27 smithi033 ceph-mon[27845]: pgmap v7188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:18:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:27 smithi096 ceph-mon[32753]: pgmap v7188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:18:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12446.880588 secs 2024-04-26T15:18:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:18:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:18:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:28 smithi194 ceph-mon[33017]: pgmap v7189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:28 smithi194 ceph-mon[33017]: pgmap v7190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:18:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12446.880588 secs 2024-04-26T15:18:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:18:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:18:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:28 smithi033 ceph-mon[27845]: pgmap v7189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:28 smithi033 ceph-mon[27845]: pgmap v7190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:18:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12446.880588 secs 2024-04-26T15:18:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:18:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:18:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:28 smithi096 ceph-mon[32753]: pgmap v7189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:28 smithi096 ceph-mon[32753]: pgmap v7190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:18:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12449.673814 secs 2024-04-26T15:18:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12449.673814 secs 2024-04-26T15:18:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12449.673814 secs 2024-04-26T15:18:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:30 smithi194 ceph-mon[33017]: pgmap v7191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:30 smithi033 ceph-mon[27845]: pgmap v7191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:30 smithi096 ceph-mon[32753]: pgmap v7191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:31 smithi194 ceph-mon[33017]: pgmap v7192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:31 smithi033 ceph-mon[27845]: pgmap v7192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:31 smithi096 ceph-mon[32753]: pgmap v7192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12451.880672 secs 2024-04-26T15:18:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12451.880672 secs 2024-04-26T15:18:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12451.880672 secs 2024-04-26T15:18:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:33 smithi033 ceph-mon[27845]: pgmap v7193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:33 smithi096 ceph-mon[32753]: pgmap v7193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:33 smithi194 ceph-mon[33017]: pgmap v7193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12454.673945 secs 2024-04-26T15:18:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12454.673945 secs 2024-04-26T15:18:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12454.673945 secs 2024-04-26T15:18:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:35 smithi033 ceph-mon[27845]: pgmap v7194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:35 smithi096 ceph-mon[32753]: pgmap v7194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:35 smithi194 ceph-mon[33017]: pgmap v7194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:18:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:37 smithi194 ceph-mon[33017]: pgmap v7195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:18:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:37 smithi033 ceph-mon[27845]: pgmap v7195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:18:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:37 smithi096 ceph-mon[32753]: pgmap v7195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:18:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12456.880759 secs 2024-04-26T15:18:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12456.880759 secs 2024-04-26T15:18:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12456.880759 secs 2024-04-26T15:18:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12459.674060 secs 2024-04-26T15:18:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:39 smithi194 ceph-mon[33017]: pgmap v7196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12459.674060 secs 2024-04-26T15:18:40.235 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:39 smithi033 ceph-mon[27845]: pgmap v7196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:40.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12459.674060 secs 2024-04-26T15:18:40.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:39 smithi096 ceph-mon[32753]: pgmap v7196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:41.843+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:41.843+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:41.843+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:18:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:41 smithi194 ceph-mon[33017]: pgmap v7197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:41 smithi033 ceph-mon[27845]: pgmap v7197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:41 smithi096 ceph-mon[32753]: pgmap v7197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12461.880839 secs 2024-04-26T15:18:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12461.880839 secs 2024-04-26T15:18:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12461.880839 secs 2024-04-26T15:18:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:43 smithi033 ceph-mon[27845]: pgmap v7198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:43 smithi096 ceph-mon[32753]: pgmap v7198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:43 smithi194 ceph-mon[33017]: pgmap v7198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12464.674178 secs 2024-04-26T15:18:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12464.674178 secs 2024-04-26T15:18:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12464.674178 secs 2024-04-26T15:18:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:45 smithi033 ceph-mon[27845]: pgmap v7199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:45 smithi096 ceph-mon[32753]: pgmap v7199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:45 smithi194 ceph-mon[33017]: pgmap v7199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:47 smithi194 ceph-mon[33017]: pgmap v7200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:47 smithi033 ceph-mon[27845]: pgmap v7200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:47 smithi096 ceph-mon[32753]: pgmap v7200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12466.880924 secs 2024-04-26T15:18:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12466.880924 secs 2024-04-26T15:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12466.880924 secs 2024-04-26T15:18:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12469.674295 secs 2024-04-26T15:18:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:49 smithi194 ceph-mon[33017]: pgmap v7201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12469.674295 secs 2024-04-26T15:18:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:49 smithi033 ceph-mon[27845]: pgmap v7201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12469.674295 secs 2024-04-26T15:18:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:49 smithi096 ceph-mon[32753]: pgmap v7201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:51 smithi194 ceph-mon[33017]: pgmap v7202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:51 smithi033 ceph-mon[27845]: pgmap v7202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:51 smithi096 ceph-mon[32753]: pgmap v7202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12471.881024 secs 2024-04-26T15:18:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12471.881024 secs 2024-04-26T15:18:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12471.881024 secs 2024-04-26T15:18:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:53 smithi033 ceph-mon[27845]: pgmap v7203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:53 smithi096 ceph-mon[32753]: pgmap v7203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:53 smithi194 ceph-mon[33017]: pgmap v7203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12474.674407 secs 2024-04-26T15:18:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12474.674407 secs 2024-04-26T15:18:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12474.674407 secs 2024-04-26T15:18:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:55 smithi033 ceph-mon[27845]: pgmap v7204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:55 smithi096 ceph-mon[32753]: pgmap v7204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:55 smithi194 ceph-mon[33017]: pgmap v7204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:18:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:56.855+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:56.855+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:18:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:18:56.855+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:18:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:57 smithi194 ceph-mon[33017]: pgmap v7205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:57 smithi033 ceph-mon[27845]: pgmap v7205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:57 smithi096 ceph-mon[32753]: pgmap v7205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:18:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12476.881085 secs 2024-04-26T15:18:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12476.881085 secs 2024-04-26T15:18:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12476.881085 secs 2024-04-26T15:19:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12479.674528 secs 2024-04-26T15:19:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:18:59 smithi194 ceph-mon[33017]: pgmap v7206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12479.674528 secs 2024-04-26T15:19:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:18:59 smithi033 ceph-mon[27845]: pgmap v7206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12479.674528 secs 2024-04-26T15:19:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:18:59 smithi096 ceph-mon[32753]: pgmap v7206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:01 smithi194 ceph-mon[33017]: pgmap v7207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:01 smithi033 ceph-mon[27845]: pgmap v7207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:01 smithi096 ceph-mon[32753]: pgmap v7207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12481.881146 secs 2024-04-26T15:19:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12481.881146 secs 2024-04-26T15:19:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12481.881146 secs 2024-04-26T15:19:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:03 smithi033 ceph-mon[27845]: pgmap v7208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:03 smithi096 ceph-mon[32753]: pgmap v7208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:03 smithi194 ceph-mon[33017]: pgmap v7208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12484.674622 secs 2024-04-26T15:19:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12484.674622 secs 2024-04-26T15:19:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12484.674622 secs 2024-04-26T15:19:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:05 smithi033 ceph-mon[27845]: pgmap v7209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:05 smithi096 ceph-mon[32753]: pgmap v7209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:05 smithi194 ceph-mon[33017]: pgmap v7209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:07 smithi194 ceph-mon[33017]: pgmap v7210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:07 smithi033 ceph-mon[27845]: pgmap v7210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:07 smithi096 ceph-mon[32753]: pgmap v7210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12486.881233 secs 2024-04-26T15:19:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12486.881233 secs 2024-04-26T15:19:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12486.881233 secs 2024-04-26T15:19:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12489.674740 secs 2024-04-26T15:19:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:09 smithi194 ceph-mon[33017]: pgmap v7211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12489.674740 secs 2024-04-26T15:19:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:09 smithi033 ceph-mon[27845]: pgmap v7211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12489.674740 secs 2024-04-26T15:19:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:09 smithi096 ceph-mon[32753]: pgmap v7211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:11.871+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:11.871+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:19:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:11.871+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:11 smithi194 ceph-mon[33017]: pgmap v7212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:11 smithi033 ceph-mon[27845]: pgmap v7212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:11 smithi096 ceph-mon[32753]: pgmap v7212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12491.881315 secs 2024-04-26T15:19:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12491.881315 secs 2024-04-26T15:19:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12491.881315 secs 2024-04-26T15:19:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:13 smithi033 ceph-mon[27845]: pgmap v7213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:13 smithi096 ceph-mon[32753]: pgmap v7213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:13 smithi194 ceph-mon[33017]: pgmap v7213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12494.674861 secs 2024-04-26T15:19:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12494.674861 secs 2024-04-26T15:19:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12494.674861 secs 2024-04-26T15:19:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:15 smithi033 ceph-mon[27845]: pgmap v7214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:15 smithi096 ceph-mon[32753]: pgmap v7214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:15 smithi194 ceph-mon[33017]: pgmap v7214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:17 smithi194 ceph-mon[33017]: pgmap v7215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:17 smithi033 ceph-mon[27845]: pgmap v7215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:17 smithi096 ceph-mon[32753]: pgmap v7215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12496.881398 secs 2024-04-26T15:19:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12496.881398 secs 2024-04-26T15:19:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12496.881398 secs 2024-04-26T15:19:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12499.674999 secs 2024-04-26T15:19:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:19 smithi194 ceph-mon[33017]: pgmap v7216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12499.674999 secs 2024-04-26T15:19:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:19 smithi033 ceph-mon[27845]: pgmap v7216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12499.674999 secs 2024-04-26T15:19:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:19 smithi096 ceph-mon[32753]: pgmap v7216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:21 smithi194 ceph-mon[33017]: pgmap v7217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:21 smithi033 ceph-mon[27845]: pgmap v7217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:21 smithi096 ceph-mon[32753]: pgmap v7217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12501.881460 secs 2024-04-26T15:19:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12501.881460 secs 2024-04-26T15:19:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12501.881460 secs 2024-04-26T15:19:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:23 smithi033 ceph-mon[27845]: pgmap v7218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:23 smithi096 ceph-mon[32753]: pgmap v7218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:23 smithi194 ceph-mon[33017]: pgmap v7218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12504.675131 secs 2024-04-26T15:19:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12504.675131 secs 2024-04-26T15:19:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12504.675131 secs 2024-04-26T15:19:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:25 smithi033 ceph-mon[27845]: pgmap v7219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:25 smithi096 ceph-mon[32753]: pgmap v7219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:25 smithi194 ceph-mon[33017]: pgmap v7219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:26.884+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:26.884+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:26.884+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:19:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:27 smithi194 ceph-mon[33017]: pgmap v7220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:27 smithi033 ceph-mon[27845]: pgmap v7220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:27 smithi096 ceph-mon[32753]: pgmap v7220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12506.881546 secs 2024-04-26T15:19:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:19:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12506.881546 secs 2024-04-26T15:19:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:19:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12506.881546 secs 2024-04-26T15:19:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12509.675199 secs 2024-04-26T15:19:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:29 smithi194 ceph-mon[33017]: pgmap v7221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12509.675199 secs 2024-04-26T15:19:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:29 smithi033 ceph-mon[27845]: pgmap v7221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12509.675199 secs 2024-04-26T15:19:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:29 smithi096 ceph-mon[32753]: pgmap v7221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:31.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:31.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:32 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:19:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:32 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:19:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:32 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:19:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:32 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:19:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:32 smithi033 ceph-mon[27845]: pgmap v7222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:32 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:19:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:32 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:19:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:32 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:19:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:32 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:19:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:32 smithi096 ceph-mon[32753]: pgmap v7222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:32 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:19:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:32 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:19:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:32 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:19:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:32 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:19:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:32 smithi194 ceph-mon[33017]: pgmap v7222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12511.881607 secs 2024-04-26T15:19:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:33 smithi194 ceph-mon[33017]: pgmap v7223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12511.881607 secs 2024-04-26T15:19:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:33 smithi033 ceph-mon[27845]: pgmap v7223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12511.881607 secs 2024-04-26T15:19:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:33 smithi096 ceph-mon[32753]: pgmap v7223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:35.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12514.675313 secs 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: pgmap v7224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:35.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12514.675313 secs 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:19:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:19:35.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:19:35.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:19:35.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: pgmap v7224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:35.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12514.675313 secs 2024-04-26T15:19:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: pgmap v7224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:19:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:37 smithi194 ceph-mon[33017]: pgmap v7225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:37 smithi033 ceph-mon[27845]: pgmap v7225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:37 smithi096 ceph-mon[32753]: pgmap v7225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12516.881687 secs 2024-04-26T15:19:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12516.881687 secs 2024-04-26T15:19:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12516.881687 secs 2024-04-26T15:19:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:39 smithi033 ceph-mon[27845]: pgmap v7226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12519.675414 secs 2024-04-26T15:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:39 smithi096 ceph-mon[32753]: pgmap v7226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12519.675414 secs 2024-04-26T15:19:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:39 smithi194 ceph-mon[33017]: pgmap v7226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12519.675414 secs 2024-04-26T15:19:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:41 smithi033 ceph-mon[27845]: pgmap v7227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:41 smithi096 ceph-mon[32753]: pgmap v7227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:41 smithi194 ceph-mon[33017]: pgmap v7227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:41.900+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:41.900+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:19:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:41.900+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:43 smithi033 ceph-mon[27845]: pgmap v7228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12521.881770 secs 2024-04-26T15:19:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:43 smithi096 ceph-mon[32753]: pgmap v7228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12521.881770 secs 2024-04-26T15:19:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:43 smithi194 ceph-mon[33017]: pgmap v7228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12521.881770 secs 2024-04-26T15:19:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12524.675478 secs 2024-04-26T15:19:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12524.675478 secs 2024-04-26T15:19:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12524.675478 secs 2024-04-26T15:19:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:45 smithi033 ceph-mon[27845]: pgmap v7229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:45.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:45 smithi096 ceph-mon[32753]: pgmap v7229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:45 smithi194 ceph-mon[33017]: pgmap v7229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:19:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:47 smithi033 ceph-mon[27845]: pgmap v7230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:47 smithi096 ceph-mon[32753]: pgmap v7230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:47 smithi194 ceph-mon[33017]: pgmap v7230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12526.881859 secs 2024-04-26T15:19:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12526.881859 secs 2024-04-26T15:19:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12526.881859 secs 2024-04-26T15:19:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:49 smithi033 ceph-mon[27845]: pgmap v7231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12529.675598 secs 2024-04-26T15:19:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:49 smithi096 ceph-mon[32753]: pgmap v7231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12529.675598 secs 2024-04-26T15:19:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:49 smithi194 ceph-mon[33017]: pgmap v7231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12529.675598 secs 2024-04-26T15:19:51.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:51 smithi194 ceph-mon[33017]: pgmap v7232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:51 smithi033 ceph-mon[27845]: pgmap v7232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:51 smithi096 ceph-mon[32753]: pgmap v7232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:53 smithi033 ceph-mon[27845]: pgmap v7233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12531.881948 secs 2024-04-26T15:19:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:53 smithi096 ceph-mon[32753]: pgmap v7233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12531.881948 secs 2024-04-26T15:19:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:53 smithi194 ceph-mon[33017]: pgmap v7233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12531.881948 secs 2024-04-26T15:19:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12534.675712 secs 2024-04-26T15:19:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12534.675712 secs 2024-04-26T15:19:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12534.675712 secs 2024-04-26T15:19:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:55 smithi033 ceph-mon[27845]: pgmap v7234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:55 smithi096 ceph-mon[32753]: pgmap v7234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:55 smithi194 ceph-mon[33017]: pgmap v7234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:56.910+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:56.911+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:19:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:19:56.911+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:19:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:57 smithi096 ceph-mon[32753]: pgmap v7235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:57 smithi194 ceph-mon[33017]: pgmap v7235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:57 smithi033 ceph-mon[27845]: pgmap v7235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:19:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12536.882037 secs 2024-04-26T15:19:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12536.882037 secs 2024-04-26T15:19:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12536.882037 secs 2024-04-26T15:19:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:59 smithi194 ceph-mon[33017]: pgmap v7236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:19:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12539.675833 secs 2024-04-26T15:19:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:59 smithi033 ceph-mon[27845]: pgmap v7236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:19:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12539.675833 secs 2024-04-26T15:19:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:59 smithi096 ceph-mon[32753]: pgmap v7236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:19:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:19:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12539.675833 secs 2024-04-26T15:20:00.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:20:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:20:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:20:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:01 smithi194 ceph-mon[33017]: pgmap v7237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:01 smithi033 ceph-mon[27845]: pgmap v7237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:01 smithi096 ceph-mon[32753]: pgmap v7237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:03 smithi194 ceph-mon[33017]: pgmap v7238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12541.882122 secs 2024-04-26T15:20:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:03 smithi033 ceph-mon[27845]: pgmap v7238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12541.882122 secs 2024-04-26T15:20:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:03 smithi096 ceph-mon[32753]: pgmap v7238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12541.882122 secs 2024-04-26T15:20:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12544.675951 secs 2024-04-26T15:20:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12544.675951 secs 2024-04-26T15:20:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12544.675951 secs 2024-04-26T15:20:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:05 smithi096 ceph-mon[32753]: pgmap v7239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:05.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:05 smithi033 ceph-mon[27845]: pgmap v7239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:05 smithi194 ceph-mon[33017]: pgmap v7239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:07 smithi194 ceph-mon[33017]: pgmap v7240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:07 smithi033 ceph-mon[27845]: pgmap v7240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:07 smithi096 ceph-mon[32753]: pgmap v7240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12546.882185 secs 2024-04-26T15:20:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12546.882185 secs 2024-04-26T15:20:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12546.882185 secs 2024-04-26T15:20:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:09 smithi194 ceph-mon[33017]: pgmap v7241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12549.676103 secs 2024-04-26T15:20:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:09 smithi033 ceph-mon[27845]: pgmap v7241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12549.676103 secs 2024-04-26T15:20:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:09 smithi096 ceph-mon[32753]: pgmap v7241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12549.676103 secs 2024-04-26T15:20:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:11 smithi194 ceph-mon[33017]: pgmap v7242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:11 smithi033 ceph-mon[27845]: pgmap v7242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:11.926+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:11.926+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:11.926+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:20:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:11 smithi096 ceph-mon[32753]: pgmap v7242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:13 smithi194 ceph-mon[33017]: pgmap v7243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12551.882291 secs 2024-04-26T15:20:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:13 smithi033 ceph-mon[27845]: pgmap v7243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12551.882291 secs 2024-04-26T15:20:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:13 smithi096 ceph-mon[32753]: pgmap v7243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12551.882291 secs 2024-04-26T15:20:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12554.676222 secs 2024-04-26T15:20:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12554.676222 secs 2024-04-26T15:20:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12554.676222 secs 2024-04-26T15:20:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:15 smithi194 ceph-mon[33017]: pgmap v7244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:15 smithi033 ceph-mon[27845]: pgmap v7244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:15 smithi096 ceph-mon[32753]: pgmap v7244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:17 smithi033 ceph-mon[27845]: pgmap v7245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:17 smithi096 ceph-mon[32753]: pgmap v7245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:17 smithi194 ceph-mon[33017]: pgmap v7245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12556.882384 secs 2024-04-26T15:20:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12556.882384 secs 2024-04-26T15:20:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12556.882384 secs 2024-04-26T15:20:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:19 smithi033 ceph-mon[27845]: pgmap v7246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12559.676307 secs 2024-04-26T15:20:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:19 smithi096 ceph-mon[32753]: pgmap v7246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12559.676307 secs 2024-04-26T15:20:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:19 smithi194 ceph-mon[33017]: pgmap v7246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12559.676307 secs 2024-04-26T15:20:21.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:21 smithi194 ceph-mon[33017]: pgmap v7247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:21 smithi033 ceph-mon[27845]: pgmap v7247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:21 smithi096 ceph-mon[32753]: pgmap v7247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:23 smithi033 ceph-mon[27845]: pgmap v7248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12561.882473 secs 2024-04-26T15:20:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:23.929+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:20:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:23.929+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:20:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:23.929+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:20:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:23.929+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:20:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:23 smithi096 ceph-mon[32753]: pgmap v7248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12561.882473 secs 2024-04-26T15:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:23 smithi194 ceph-mon[33017]: pgmap v7248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12561.882473 secs 2024-04-26T15:20:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12564.676424 secs 2024-04-26T15:20:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12564.676424 secs 2024-04-26T15:20:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12564.676424 secs 2024-04-26T15:20:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:25 smithi033 ceph-mon[27845]: pgmap v7249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:25 smithi096 ceph-mon[32753]: pgmap v7249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:25 smithi194 ceph-mon[33017]: pgmap v7249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:26.941+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:26.941+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:26.941+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:20:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:27 smithi033 ceph-mon[27845]: pgmap v7250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:27 smithi096 ceph-mon[32753]: pgmap v7250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:27 smithi194 ceph-mon[33017]: pgmap v7250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12566.882556 secs 2024-04-26T15:20:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12566.882556 secs 2024-04-26T15:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12566.882556 secs 2024-04-26T15:20:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:29 smithi033 ceph-mon[27845]: pgmap v7251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12569.676542 secs 2024-04-26T15:20:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:29 smithi096 ceph-mon[32753]: pgmap v7251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12569.676542 secs 2024-04-26T15:20:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:29 smithi194 ceph-mon[33017]: pgmap v7251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12569.676542 secs 2024-04-26T15:20:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:31 smithi033 ceph-mon[27845]: pgmap v7252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:31 smithi096 ceph-mon[32753]: pgmap v7252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:31 smithi194 ceph-mon[33017]: pgmap v7252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:33 smithi096 ceph-mon[32753]: pgmap v7253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12571.882632 secs 2024-04-26T15:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:33 smithi194 ceph-mon[33017]: pgmap v7253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12571.882632 secs 2024-04-26T15:20:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:33 smithi033 ceph-mon[27845]: pgmap v7253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12571.882632 secs 2024-04-26T15:20:35.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12574.676657 secs 2024-04-26T15:20:35.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12574.676657 secs 2024-04-26T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:20:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12574.676657 secs 2024-04-26T15:20:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:20:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:35 smithi194 ceph-mon[33017]: pgmap v7254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:35 smithi033 ceph-mon[27845]: pgmap v7254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:35 smithi096 ceph-mon[32753]: pgmap v7254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:20:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:20:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:20:37.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:20:37.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:20:37.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:20:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:20:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:20:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:20:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:37 smithi194 ceph-mon[33017]: pgmap v7255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:20:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:37 smithi194 ceph-mon[33017]: pgmap v7256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:37 smithi033 ceph-mon[27845]: pgmap v7255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:20:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:37 smithi033 ceph-mon[27845]: pgmap v7256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:37 smithi096 ceph-mon[32753]: pgmap v7255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:20:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:37 smithi096 ceph-mon[32753]: pgmap v7256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12576.882719 secs 2024-04-26T15:20:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12576.882719 secs 2024-04-26T15:20:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12576.882719 secs 2024-04-26T15:20:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:39 smithi194 ceph-mon[33017]: pgmap v7257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12579.676774 secs 2024-04-26T15:20:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:39 smithi033 ceph-mon[27845]: pgmap v7257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12579.676774 secs 2024-04-26T15:20:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:39 smithi096 ceph-mon[32753]: pgmap v7257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12579.676774 secs 2024-04-26T15:20:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:41.954+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:41.954+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:41.954+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:20:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:41 smithi194 ceph-mon[33017]: pgmap v7258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:41 smithi033 ceph-mon[27845]: pgmap v7258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:41 smithi096 ceph-mon[32753]: pgmap v7258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:43 smithi194 ceph-mon[33017]: pgmap v7259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12581.882802 secs 2024-04-26T15:20:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:43 smithi033 ceph-mon[27845]: pgmap v7259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12581.882802 secs 2024-04-26T15:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:43 smithi096 ceph-mon[32753]: pgmap v7259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12581.882802 secs 2024-04-26T15:20:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12584.676892 secs 2024-04-26T15:20:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12584.676892 secs 2024-04-26T15:20:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12584.676892 secs 2024-04-26T15:20:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:45 smithi194 ceph-mon[33017]: pgmap v7260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:45 smithi033 ceph-mon[27845]: pgmap v7260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:45 smithi096 ceph-mon[32753]: pgmap v7260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:47 smithi194 ceph-mon[33017]: pgmap v7261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:20:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:47 smithi033 ceph-mon[27845]: pgmap v7261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:20:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:47 smithi096 ceph-mon[32753]: pgmap v7261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:20:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12586.882890 secs 2024-04-26T15:20:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:48 smithi194 ceph-mon[33017]: pgmap v7262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12586.882890 secs 2024-04-26T15:20:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:48 smithi033 ceph-mon[27845]: pgmap v7262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12586.882890 secs 2024-04-26T15:20:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:48 smithi096 ceph-mon[32753]: pgmap v7262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12589.677014 secs 2024-04-26T15:20:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12589.677014 secs 2024-04-26T15:20:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12589.677014 secs 2024-04-26T15:20:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:50 smithi194 ceph-mon[33017]: pgmap v7263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:50 smithi033 ceph-mon[27845]: pgmap v7263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:50 smithi096 ceph-mon[32753]: pgmap v7263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:52 smithi033 ceph-mon[27845]: pgmap v7264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12591.882991 secs 2024-04-26T15:20:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:52 smithi096 ceph-mon[32753]: pgmap v7264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12591.882991 secs 2024-04-26T15:20:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:52 smithi194 ceph-mon[33017]: pgmap v7264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12591.882991 secs 2024-04-26T15:20:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12594.677127 secs 2024-04-26T15:20:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:54 smithi033 ceph-mon[27845]: pgmap v7265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12594.677127 secs 2024-04-26T15:20:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:54 smithi096 ceph-mon[32753]: pgmap v7265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12594.677127 secs 2024-04-26T15:20:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:54 smithi194 ceph-mon[33017]: pgmap v7265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:56.961+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:56.961+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:20:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:20:56.961+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:20:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:56 smithi033 ceph-mon[27845]: pgmap v7266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:56 smithi096 ceph-mon[32753]: pgmap v7266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:56 smithi194 ceph-mon[33017]: pgmap v7266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:20:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12596.883053 secs 2024-04-26T15:20:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12596.883053 secs 2024-04-26T15:20:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12596.883053 secs 2024-04-26T15:20:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:58 smithi033 ceph-mon[27845]: pgmap v7267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:58 smithi096 ceph-mon[32753]: pgmap v7267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:20:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:58 smithi194 ceph-mon[33017]: pgmap v7267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:20:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12599.677262 secs 2024-04-26T15:21:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:20:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12599.677262 secs 2024-04-26T15:21:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:20:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12599.677262 secs 2024-04-26T15:21:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:00 smithi033 ceph-mon[27845]: pgmap v7268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:00 smithi096 ceph-mon[32753]: pgmap v7268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:00 smithi194 ceph-mon[33017]: pgmap v7268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:02 smithi033 ceph-mon[27845]: pgmap v7269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12601.883131 secs 2024-04-26T15:21:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:02 smithi096 ceph-mon[32753]: pgmap v7269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12601.883131 secs 2024-04-26T15:21:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:02 smithi194 ceph-mon[33017]: pgmap v7269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12601.883131 secs 2024-04-26T15:21:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12604.677397 secs 2024-04-26T15:21:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:04 smithi033 ceph-mon[27845]: pgmap v7270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12604.677397 secs 2024-04-26T15:21:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:04 smithi096 ceph-mon[32753]: pgmap v7270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12604.677397 secs 2024-04-26T15:21:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:04 smithi194 ceph-mon[33017]: pgmap v7270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:06 smithi033 ceph-mon[27845]: pgmap v7271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:06 smithi096 ceph-mon[32753]: pgmap v7271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:06 smithi194 ceph-mon[33017]: pgmap v7271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12606.883190 secs 2024-04-26T15:21:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12606.883190 secs 2024-04-26T15:21:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12606.883190 secs 2024-04-26T15:21:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:08 smithi194 ceph-mon[33017]: pgmap v7272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:08 smithi033 ceph-mon[27845]: pgmap v7272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:08 smithi096 ceph-mon[32753]: pgmap v7272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12609.677513 secs 2024-04-26T15:21:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12609.677513 secs 2024-04-26T15:21:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12609.677513 secs 2024-04-26T15:21:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:10 smithi194 ceph-mon[33017]: pgmap v7273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:10 smithi033 ceph-mon[27845]: pgmap v7273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:10 smithi096 ceph-mon[32753]: pgmap v7273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:11.976+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:11.976+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:11.976+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:21:13.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:12 smithi033 ceph-mon[27845]: pgmap v7274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:13.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12611.883246 secs 2024-04-26T15:21:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:12 smithi194 ceph-mon[33017]: pgmap v7274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12611.883246 secs 2024-04-26T15:21:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:12 smithi096 ceph-mon[32753]: pgmap v7274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12611.883246 secs 2024-04-26T15:21:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12614.677633 secs 2024-04-26T15:21:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:14 smithi194 ceph-mon[33017]: pgmap v7275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12614.677633 secs 2024-04-26T15:21:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:14 smithi033 ceph-mon[27845]: pgmap v7275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12614.677633 secs 2024-04-26T15:21:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:14 smithi096 ceph-mon[32753]: pgmap v7275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:17 smithi194 ceph-mon[33017]: pgmap v7276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:17 smithi033 ceph-mon[27845]: pgmap v7276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:17 smithi096 ceph-mon[32753]: pgmap v7276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12616.883336 secs 2024-04-26T15:21:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12616.883336 secs 2024-04-26T15:21:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12616.883336 secs 2024-04-26T15:21:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:19 smithi194 ceph-mon[33017]: pgmap v7277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12619.677756 secs 2024-04-26T15:21:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:19 smithi033 ceph-mon[27845]: pgmap v7277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12619.677756 secs 2024-04-26T15:21:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:19 smithi096 ceph-mon[32753]: pgmap v7277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12619.677756 secs 2024-04-26T15:21:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:21 smithi194 ceph-mon[33017]: pgmap v7278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:21 smithi033 ceph-mon[27845]: pgmap v7278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:21 smithi096 ceph-mon[32753]: pgmap v7278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:23 smithi033 ceph-mon[27845]: pgmap v7279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12621.883417 secs 2024-04-26T15:21:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:23 smithi096 ceph-mon[32753]: pgmap v7279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12621.883417 secs 2024-04-26T15:21:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:23 smithi194 ceph-mon[33017]: pgmap v7279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12621.883417 secs 2024-04-26T15:21:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12624.677886 secs 2024-04-26T15:21:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12624.677886 secs 2024-04-26T15:21:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12624.677886 secs 2024-04-26T15:21:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:25 smithi033 ceph-mon[27845]: pgmap v7280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:25 smithi096 ceph-mon[32753]: pgmap v7280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:25 smithi194 ceph-mon[33017]: pgmap v7280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:26.990+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:26.990+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:26.990+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:21:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:27 smithi033 ceph-mon[27845]: pgmap v7281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:27 smithi096 ceph-mon[32753]: pgmap v7281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:27 smithi194 ceph-mon[33017]: pgmap v7281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12626.883502 secs 2024-04-26T15:21:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12626.883502 secs 2024-04-26T15:21:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12626.883502 secs 2024-04-26T15:21:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:29 smithi033 ceph-mon[27845]: pgmap v7282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12629.678005 secs 2024-04-26T15:21:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:29 smithi096 ceph-mon[32753]: pgmap v7282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12629.678005 secs 2024-04-26T15:21:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:29 smithi194 ceph-mon[33017]: pgmap v7282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12629.678005 secs 2024-04-26T15:21:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:31 smithi033 ceph-mon[27845]: pgmap v7283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:31 smithi096 ceph-mon[32753]: pgmap v7283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:31 smithi194 ceph-mon[33017]: pgmap v7283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:33 smithi033 ceph-mon[27845]: pgmap v7284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12631.883585 secs 2024-04-26T15:21:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:33 smithi096 ceph-mon[32753]: pgmap v7284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12631.883585 secs 2024-04-26T15:21:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:33 smithi194 ceph-mon[33017]: pgmap v7284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12631.883585 secs 2024-04-26T15:21:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12634.678140 secs 2024-04-26T15:21:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12634.678140 secs 2024-04-26T15:21:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12634.678140 secs 2024-04-26T15:21:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:35 smithi033 ceph-mon[27845]: pgmap v7285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:35 smithi096 ceph-mon[32753]: pgmap v7285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:35 smithi194 ceph-mon[33017]: pgmap v7285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:21:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:21:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:21:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:37 smithi194 ceph-mon[33017]: pgmap v7286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:21:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:21:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:21:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:37 smithi033 ceph-mon[27845]: pgmap v7286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:21:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:21:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:21:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:37 smithi096 ceph-mon[32753]: pgmap v7286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:21:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:21:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:21:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:38 smithi194 ceph-mon[33017]: pgmap v7287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:38 smithi194 ceph-mon[33017]: pgmap v7288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12636.883648 secs 2024-04-26T15:21:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:38 smithi033 ceph-mon[27845]: pgmap v7287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:38 smithi033 ceph-mon[27845]: pgmap v7288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12636.883648 secs 2024-04-26T15:21:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:38 smithi096 ceph-mon[32753]: pgmap v7287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:38 smithi096 ceph-mon[32753]: pgmap v7288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12636.883648 secs 2024-04-26T15:21:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12639.678257 secs 2024-04-26T15:21:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12639.678257 secs 2024-04-26T15:21:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12639.678257 secs 2024-04-26T15:21:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:40 smithi194 ceph-mon[33017]: pgmap v7289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:40 smithi033 ceph-mon[27845]: pgmap v7289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:40 smithi096 ceph-mon[32753]: pgmap v7289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:42.003+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:42.003+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:42.003+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:21:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:42 smithi194 ceph-mon[33017]: pgmap v7290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:42 smithi033 ceph-mon[27845]: pgmap v7290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:42 smithi096 ceph-mon[32753]: pgmap v7290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12641.883730 secs 2024-04-26T15:21:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12641.883730 secs 2024-04-26T15:21:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12641.883730 secs 2024-04-26T15:21:45.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12644.678374 secs 2024-04-26T15:21:45.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:44 smithi033 ceph-mon[27845]: pgmap v7291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12644.678374 secs 2024-04-26T15:21:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:44 smithi194 ceph-mon[33017]: pgmap v7291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12644.678374 secs 2024-04-26T15:21:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:44 smithi096 ceph-mon[32753]: pgmap v7291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:46 smithi194 ceph-mon[33017]: pgmap v7292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:46 smithi033 ceph-mon[27845]: pgmap v7292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:46 smithi096 ceph-mon[32753]: pgmap v7292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:21:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:48 smithi194 ceph-mon[33017]: pgmap v7293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:21:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12646.883793 secs 2024-04-26T15:21:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:48 smithi033 ceph-mon[27845]: pgmap v7293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:21:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12646.883793 secs 2024-04-26T15:21:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:48 smithi096 ceph-mon[32753]: pgmap v7293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:21:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12646.883793 secs 2024-04-26T15:21:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12649.678493 secs 2024-04-26T15:21:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12649.678493 secs 2024-04-26T15:21:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12649.678493 secs 2024-04-26T15:21:51.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:50 smithi194 ceph-mon[33017]: pgmap v7294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:50 smithi033 ceph-mon[27845]: pgmap v7294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:50 smithi096 ceph-mon[32753]: pgmap v7294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:52 smithi194 ceph-mon[33017]: pgmap v7295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:52 smithi033 ceph-mon[27845]: pgmap v7295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:52 smithi096 ceph-mon[32753]: pgmap v7295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12651.883875 secs 2024-04-26T15:21:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12651.883875 secs 2024-04-26T15:21:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12651.883875 secs 2024-04-26T15:21:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12654.678616 secs 2024-04-26T15:21:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:54 smithi194 ceph-mon[33017]: pgmap v7296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12654.678616 secs 2024-04-26T15:21:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:54 smithi033 ceph-mon[27845]: pgmap v7296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12654.678616 secs 2024-04-26T15:21:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:54 smithi096 ceph-mon[32753]: pgmap v7296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:55 smithi194 ceph-mon[33017]: pgmap v7297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:55 smithi033 ceph-mon[27845]: pgmap v7297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:55 smithi096 ceph-mon[32753]: pgmap v7297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:21:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:57.016+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:57.016+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:21:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:21:57.016+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:21:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:58 smithi194 ceph-mon[33017]: pgmap v7298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12656.883938 secs 2024-04-26T15:21:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:58 smithi033 ceph-mon[27845]: pgmap v7298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12656.883938 secs 2024-04-26T15:21:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:58 smithi096 ceph-mon[32753]: pgmap v7298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:21:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12656.883938 secs 2024-04-26T15:21:59.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:21:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12659.678754 secs 2024-04-26T15:21:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:21:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12659.678754 secs 2024-04-26T15:22:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:21:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12659.678754 secs 2024-04-26T15:22:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:00 smithi033 ceph-mon[27845]: pgmap v7299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:00 smithi096 ceph-mon[32753]: pgmap v7299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:00 smithi194 ceph-mon[33017]: pgmap v7299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:02 smithi033 ceph-mon[27845]: pgmap v7300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:02 smithi096 ceph-mon[32753]: pgmap v7300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:02 smithi194 ceph-mon[33017]: pgmap v7300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12661.884027 secs 2024-04-26T15:22:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12661.884027 secs 2024-04-26T15:22:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12661.884027 secs 2024-04-26T15:22:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12664.678868 secs 2024-04-26T15:22:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:04 smithi033 ceph-mon[27845]: pgmap v7301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12664.678868 secs 2024-04-26T15:22:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:04 smithi096 ceph-mon[32753]: pgmap v7301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12664.678868 secs 2024-04-26T15:22:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:04 smithi194 ceph-mon[33017]: pgmap v7301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:06 smithi033 ceph-mon[27845]: pgmap v7302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:06 smithi096 ceph-mon[32753]: pgmap v7302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:06 smithi194 ceph-mon[33017]: pgmap v7302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:08 smithi033 ceph-mon[27845]: pgmap v7303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12666.884110 secs 2024-04-26T15:22:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:08 smithi096 ceph-mon[32753]: pgmap v7303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12666.884110 secs 2024-04-26T15:22:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:08 smithi194 ceph-mon[33017]: pgmap v7303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12666.884110 secs 2024-04-26T15:22:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12669.678990 secs 2024-04-26T15:22:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12669.678990 secs 2024-04-26T15:22:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12669.678990 secs 2024-04-26T15:22:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:10 smithi033 ceph-mon[27845]: pgmap v7304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:10 smithi096 ceph-mon[32753]: pgmap v7304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:10 smithi194 ceph-mon[33017]: pgmap v7304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:12.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:12.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:12.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:12.032+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:12.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:12.032+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:22:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:12 smithi033 ceph-mon[27845]: pgmap v7305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:12 smithi096 ceph-mon[32753]: pgmap v7305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:12 smithi194 ceph-mon[33017]: pgmap v7305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12671.884193 secs 2024-04-26T15:22:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12671.884193 secs 2024-04-26T15:22:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12671.884193 secs 2024-04-26T15:22:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12674.679131 secs 2024-04-26T15:22:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:14 smithi033 ceph-mon[27845]: pgmap v7306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12674.679131 secs 2024-04-26T15:22:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:14 smithi096 ceph-mon[32753]: pgmap v7306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12674.679131 secs 2024-04-26T15:22:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:14 smithi194 ceph-mon[33017]: pgmap v7306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:16 smithi033 ceph-mon[27845]: pgmap v7307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:16 smithi096 ceph-mon[32753]: pgmap v7307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:16 smithi194 ceph-mon[33017]: pgmap v7307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:18 smithi096 ceph-mon[32753]: pgmap v7308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12676.884252 secs 2024-04-26T15:22:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:18 smithi194 ceph-mon[33017]: pgmap v7308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12676.884252 secs 2024-04-26T15:22:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:18 smithi033 ceph-mon[27845]: pgmap v7308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12676.884252 secs 2024-04-26T15:22:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12679.679231 secs 2024-04-26T15:22:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12679.679231 secs 2024-04-26T15:22:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12679.679231 secs 2024-04-26T15:22:21.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:20 smithi194 ceph-mon[33017]: pgmap v7309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:20 smithi033 ceph-mon[27845]: pgmap v7309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:20 smithi096 ceph-mon[32753]: pgmap v7309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:22 smithi194 ceph-mon[33017]: pgmap v7310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:22 smithi033 ceph-mon[27845]: pgmap v7310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:22 smithi096 ceph-mon[32753]: pgmap v7310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:24.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12681.884327 secs 2024-04-26T15:22:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12681.884327 secs 2024-04-26T15:22:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12681.884327 secs 2024-04-26T15:22:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12684.679292 secs 2024-04-26T15:22:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:24 smithi194 ceph-mon[33017]: pgmap v7311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12684.679292 secs 2024-04-26T15:22:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:24 smithi033 ceph-mon[27845]: pgmap v7311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12684.679292 secs 2024-04-26T15:22:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:24 smithi096 ceph-mon[32753]: pgmap v7311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:27.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:27.047+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:27.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:27.047+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:27.047+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:22:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:26 smithi194 ceph-mon[33017]: pgmap v7312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:26 smithi033 ceph-mon[27845]: pgmap v7312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:26 smithi096 ceph-mon[32753]: pgmap v7312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:28 smithi194 ceph-mon[33017]: pgmap v7313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12686.884389 secs 2024-04-26T15:22:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:28 smithi033 ceph-mon[27845]: pgmap v7313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12686.884389 secs 2024-04-26T15:22:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:28 smithi096 ceph-mon[32753]: pgmap v7313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12686.884389 secs 2024-04-26T15:22:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12689.679410 secs 2024-04-26T15:22:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12689.679410 secs 2024-04-26T15:22:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12689.679410 secs 2024-04-26T15:22:31.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:30 smithi033 ceph-mon[27845]: pgmap v7314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:30 smithi194 ceph-mon[33017]: pgmap v7314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:30 smithi096 ceph-mon[32753]: pgmap v7314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:32 smithi194 ceph-mon[33017]: pgmap v7315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:32 smithi033 ceph-mon[27845]: pgmap v7315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:32 smithi096 ceph-mon[32753]: pgmap v7315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12691.884466 secs 2024-04-26T15:22:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12691.884466 secs 2024-04-26T15:22:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12691.884466 secs 2024-04-26T15:22:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12694.679486 secs 2024-04-26T15:22:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:34 smithi194 ceph-mon[33017]: pgmap v7316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12694.679486 secs 2024-04-26T15:22:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:34 smithi033 ceph-mon[27845]: pgmap v7316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12694.679486 secs 2024-04-26T15:22:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:34 smithi096 ceph-mon[32753]: pgmap v7316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:36 smithi194 ceph-mon[33017]: pgmap v7317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:36 smithi033 ceph-mon[27845]: pgmap v7317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:36 smithi096 ceph-mon[32753]: pgmap v7317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:37 smithi194 ceph-mon[33017]: pgmap v7318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:37 smithi033 ceph-mon[27845]: pgmap v7318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:37 smithi096 ceph-mon[32753]: pgmap v7318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12696.884551 secs 2024-04-26T15:22:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:22:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12696.884551 secs 2024-04-26T15:22:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:22:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12696.884551 secs 2024-04-26T15:22:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:22:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12699.679575 secs 2024-04-26T15:22:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:22:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:22:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:39 smithi194 ceph-mon[33017]: pgmap v7319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:22:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:39 smithi194 ceph-mon[33017]: pgmap v7320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T15:22:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:22:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12699.679575 secs 2024-04-26T15:22:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:22:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:22:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:39 smithi033 ceph-mon[27845]: pgmap v7319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:22:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:39 smithi033 ceph-mon[27845]: pgmap v7320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T15:22:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:22:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12699.679575 secs 2024-04-26T15:22:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:22:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:22:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:39 smithi096 ceph-mon[32753]: pgmap v7319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:22:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:39 smithi096 ceph-mon[32753]: pgmap v7320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T15:22:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:22:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:42.061+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:42.061+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:42.061+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:22:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:42 smithi194 ceph-mon[33017]: pgmap v7321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:42 smithi033 ceph-mon[27845]: pgmap v7321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:42 smithi096 ceph-mon[32753]: pgmap v7321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12701.884631 secs 2024-04-26T15:22:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12701.884631 secs 2024-04-26T15:22:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12701.884631 secs 2024-04-26T15:22:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12704.679690 secs 2024-04-26T15:22:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:44 smithi194 ceph-mon[33017]: pgmap v7322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12704.679690 secs 2024-04-26T15:22:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:44 smithi033 ceph-mon[27845]: pgmap v7322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12704.679690 secs 2024-04-26T15:22:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:44 smithi096 ceph-mon[32753]: pgmap v7322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:46 smithi194 ceph-mon[33017]: pgmap v7323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:46 smithi033 ceph-mon[27845]: pgmap v7323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:46 smithi096 ceph-mon[32753]: pgmap v7323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:48 smithi194 ceph-mon[33017]: pgmap v7324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12706.884691 secs 2024-04-26T15:22:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:48 smithi033 ceph-mon[27845]: pgmap v7324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12706.884691 secs 2024-04-26T15:22:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:48 smithi096 ceph-mon[32753]: pgmap v7324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12706.884691 secs 2024-04-26T15:22:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12709.679826 secs 2024-04-26T15:22:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12709.679826 secs 2024-04-26T15:22:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12709.679826 secs 2024-04-26T15:22:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:50 smithi194 ceph-mon[33017]: pgmap v7325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:22:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:50 smithi033 ceph-mon[27845]: pgmap v7325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:22:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:50 smithi096 ceph-mon[32753]: pgmap v7325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:22:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:52 smithi194 ceph-mon[33017]: pgmap v7326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:52 smithi033 ceph-mon[27845]: pgmap v7326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:52 smithi096 ceph-mon[32753]: pgmap v7326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:22:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12711.884763 secs 2024-04-26T15:22:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12711.884763 secs 2024-04-26T15:22:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12711.884763 secs 2024-04-26T15:22:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12714.679940 secs 2024-04-26T15:22:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:54 smithi033 ceph-mon[27845]: pgmap v7327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12714.679940 secs 2024-04-26T15:22:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:54 smithi096 ceph-mon[32753]: pgmap v7327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12714.679940 secs 2024-04-26T15:22:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:54 smithi194 ceph-mon[33017]: pgmap v7327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:56 smithi033 ceph-mon[27845]: pgmap v7328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:56 smithi096 ceph-mon[32753]: pgmap v7328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:56 smithi194 ceph-mon[33017]: pgmap v7328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:22:57.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:57.069+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:57.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:57.069+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:22:57.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:22:57.069+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:22:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:58 smithi033 ceph-mon[27845]: pgmap v7329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12716.884848 secs 2024-04-26T15:22:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:58 smithi096 ceph-mon[32753]: pgmap v7329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12716.884848 secs 2024-04-26T15:22:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:58 smithi194 ceph-mon[33017]: pgmap v7329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:22:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12716.884848 secs 2024-04-26T15:22:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12719.680074 secs 2024-04-26T15:22:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12719.680074 secs 2024-04-26T15:22:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12719.680074 secs 2024-04-26T15:23:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:00 smithi033 ceph-mon[27845]: pgmap v7330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:00 smithi096 ceph-mon[32753]: pgmap v7330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:00 smithi194 ceph-mon[33017]: pgmap v7330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:02 smithi033 ceph-mon[27845]: pgmap v7331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:23:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:02 smithi096 ceph-mon[32753]: pgmap v7331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:23:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:02 smithi194 ceph-mon[33017]: pgmap v7331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:23:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12721.884930 secs 2024-04-26T15:23:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12721.884930 secs 2024-04-26T15:23:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12721.884930 secs 2024-04-26T15:23:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12724.680188 secs 2024-04-26T15:23:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:04 smithi033 ceph-mon[27845]: pgmap v7332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12724.680188 secs 2024-04-26T15:23:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:04 smithi096 ceph-mon[32753]: pgmap v7332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12724.680188 secs 2024-04-26T15:23:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:04 smithi194 ceph-mon[33017]: pgmap v7332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:06 smithi033 ceph-mon[27845]: pgmap v7333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:23:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:06 smithi096 ceph-mon[32753]: pgmap v7333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:23:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:06 smithi194 ceph-mon[33017]: pgmap v7333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:23:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:08 smithi033 ceph-mon[27845]: pgmap v7334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12726.885013 secs 2024-04-26T15:23:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:08 smithi096 ceph-mon[32753]: pgmap v7334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12726.885013 secs 2024-04-26T15:23:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:08 smithi194 ceph-mon[33017]: pgmap v7334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12726.885013 secs 2024-04-26T15:23:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12729.680308 secs 2024-04-26T15:23:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12729.680308 secs 2024-04-26T15:23:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12729.680308 secs 2024-04-26T15:23:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:10 smithi096 ceph-mon[32753]: pgmap v7335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:23:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:10 smithi194 ceph-mon[33017]: pgmap v7335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:23:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:10 smithi033 ceph-mon[27845]: pgmap v7335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:23:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:12.084+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:12.084+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:12.084+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:23:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:12 smithi194 ceph-mon[33017]: pgmap v7336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:23:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:12 smithi033 ceph-mon[27845]: pgmap v7336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:23:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:12 smithi096 ceph-mon[32753]: pgmap v7336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:23:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12731.885097 secs 2024-04-26T15:23:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12731.885097 secs 2024-04-26T15:23:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12731.885097 secs 2024-04-26T15:23:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12734.680414 secs 2024-04-26T15:23:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:14 smithi194 ceph-mon[33017]: pgmap v7337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12734.680414 secs 2024-04-26T15:23:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:14 smithi033 ceph-mon[27845]: pgmap v7337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12734.680414 secs 2024-04-26T15:23:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:14 smithi096 ceph-mon[32753]: pgmap v7337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:16 smithi194 ceph-mon[33017]: pgmap v7338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:16 smithi033 ceph-mon[27845]: pgmap v7338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:16 smithi096 ceph-mon[32753]: pgmap v7338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:18 smithi194 ceph-mon[33017]: pgmap v7339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12736.885186 secs 2024-04-26T15:23:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:18 smithi033 ceph-mon[27845]: pgmap v7339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12736.885186 secs 2024-04-26T15:23:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:18 smithi096 ceph-mon[32753]: pgmap v7339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12736.885186 secs 2024-04-26T15:23:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12739.680537 secs 2024-04-26T15:23:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12739.680537 secs 2024-04-26T15:23:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12739.680537 secs 2024-04-26T15:23:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:20 smithi194 ceph-mon[33017]: pgmap v7340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:20 smithi033 ceph-mon[27845]: pgmap v7340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:20 smithi096 ceph-mon[32753]: pgmap v7340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:22 smithi194 ceph-mon[33017]: pgmap v7341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:22 smithi033 ceph-mon[27845]: pgmap v7341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:22 smithi096 ceph-mon[32753]: pgmap v7341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12741.885267 secs 2024-04-26T15:23:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12741.885267 secs 2024-04-26T15:23:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12741.885267 secs 2024-04-26T15:23:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:23.961+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:23:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:23.961+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:23:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:23.961+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:23:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:23.962+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:23:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12744.680648 secs 2024-04-26T15:23:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:24 smithi096 ceph-mon[32753]: pgmap v7342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12744.680648 secs 2024-04-26T15:23:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:24 smithi194 ceph-mon[33017]: pgmap v7342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12744.680648 secs 2024-04-26T15:23:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:24 smithi033 ceph-mon[27845]: pgmap v7342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:26 smithi194 ceph-mon[33017]: pgmap v7343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:26 smithi033 ceph-mon[27845]: pgmap v7343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:26 smithi096 ceph-mon[32753]: pgmap v7343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:27.100+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:27.100+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:27.100+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:23:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:28 smithi194 ceph-mon[33017]: pgmap v7344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12746.885350 secs 2024-04-26T15:23:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:28 smithi033 ceph-mon[27845]: pgmap v7344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12746.885350 secs 2024-04-26T15:23:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:28 smithi096 ceph-mon[32753]: pgmap v7344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12746.885350 secs 2024-04-26T15:23:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12749.680771 secs 2024-04-26T15:23:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12749.680771 secs 2024-04-26T15:23:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12749.680771 secs 2024-04-26T15:23:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:30 smithi194 ceph-mon[33017]: pgmap v7345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:30 smithi033 ceph-mon[27845]: pgmap v7345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:30 smithi096 ceph-mon[32753]: pgmap v7345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:32 smithi194 ceph-mon[33017]: pgmap v7346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:32 smithi033 ceph-mon[27845]: pgmap v7346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:32 smithi096 ceph-mon[32753]: pgmap v7346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12751.885432 secs 2024-04-26T15:23:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12751.885432 secs 2024-04-26T15:23:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12751.885432 secs 2024-04-26T15:23:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12754.680886 secs 2024-04-26T15:23:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:34 smithi194 ceph-mon[33017]: pgmap v7347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12754.680886 secs 2024-04-26T15:23:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:34 smithi033 ceph-mon[27845]: pgmap v7347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12754.680886 secs 2024-04-26T15:23:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:34 smithi096 ceph-mon[32753]: pgmap v7347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:36 smithi194 ceph-mon[33017]: pgmap v7348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:36 smithi033 ceph-mon[27845]: pgmap v7348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:36 smithi096 ceph-mon[32753]: pgmap v7348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:38 smithi194 ceph-mon[33017]: pgmap v7349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12756.885515 secs 2024-04-26T15:23:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:38 smithi033 ceph-mon[27845]: pgmap v7349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12756.885515 secs 2024-04-26T15:23:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:38 smithi096 ceph-mon[32753]: pgmap v7349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12756.885515 secs 2024-04-26T15:23:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12759.681002 secs 2024-04-26T15:23:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12759.681002 secs 2024-04-26T15:23:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12759.681002 secs 2024-04-26T15:23:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:40 smithi194 ceph-mon[33017]: pgmap v7350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:23:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:40 smithi033 ceph-mon[27845]: pgmap v7350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:23:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:40 smithi096 ceph-mon[32753]: pgmap v7350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:23:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:23:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:23:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:23:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:23:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:23:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:23:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:23:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:23:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:23:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:23:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:42.114+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:42.114+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:42.114+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:23:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:42 smithi194 ceph-mon[33017]: pgmap v7351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T15:23:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:42 smithi194 ceph-mon[33017]: pgmap v7352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:42 smithi033 ceph-mon[27845]: pgmap v7351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T15:23:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:42 smithi033 ceph-mon[27845]: pgmap v7352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:42 smithi096 ceph-mon[32753]: pgmap v7351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T15:23:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:42 smithi096 ceph-mon[32753]: pgmap v7352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12761.885600 secs 2024-04-26T15:23:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12761.885600 secs 2024-04-26T15:23:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12761.885600 secs 2024-04-26T15:23:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:44 smithi194 ceph-mon[33017]: pgmap v7353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12764.681100 secs 2024-04-26T15:23:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:44 smithi033 ceph-mon[27845]: pgmap v7353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12764.681100 secs 2024-04-26T15:23:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:44 smithi096 ceph-mon[32753]: pgmap v7353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12764.681100 secs 2024-04-26T15:23:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:46 smithi194 ceph-mon[33017]: pgmap v7354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:46 smithi033 ceph-mon[27845]: pgmap v7354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:46 smithi096 ceph-mon[32753]: pgmap v7354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:48 smithi194 ceph-mon[33017]: pgmap v7355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12766.885685 secs 2024-04-26T15:23:48.912 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:48 smithi033 ceph-mon[27845]: pgmap v7355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:48.912 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12766.885685 secs 2024-04-26T15:23:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:48 smithi096 ceph-mon[32753]: pgmap v7355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:23:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12766.885685 secs 2024-04-26T15:23:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12769.681217 secs 2024-04-26T15:23:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12769.681217 secs 2024-04-26T15:23:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12769.681217 secs 2024-04-26T15:23:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:50 smithi033 ceph-mon[27845]: pgmap v7356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:50 smithi096 ceph-mon[32753]: pgmap v7356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:50 smithi194 ceph-mon[33017]: pgmap v7356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:23:52.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:52 smithi194 ceph-mon[33017]: pgmap v7357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:23:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:52 smithi033 ceph-mon[27845]: pgmap v7357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:23:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:52 smithi096 ceph-mon[32753]: pgmap v7357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:23:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12771.885771 secs 2024-04-26T15:23:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12771.885771 secs 2024-04-26T15:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12771.885771 secs 2024-04-26T15:23:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:54 smithi033 ceph-mon[27845]: pgmap v7358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12774.681283 secs 2024-04-26T15:23:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:54 smithi096 ceph-mon[32753]: pgmap v7358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12774.681283 secs 2024-04-26T15:23:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:54 smithi194 ceph-mon[33017]: pgmap v7358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12774.681283 secs 2024-04-26T15:23:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:56 smithi033 ceph-mon[27845]: pgmap v7359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:56 smithi096 ceph-mon[32753]: pgmap v7359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:56 smithi194 ceph-mon[33017]: pgmap v7359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:23:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:57.128+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:57.128+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:23:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:23:57.128+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:23:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:58 smithi033 ceph-mon[27845]: pgmap v7360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12776.885855 secs 2024-04-26T15:23:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:58 smithi096 ceph-mon[32753]: pgmap v7360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12776.885855 secs 2024-04-26T15:23:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:58 smithi194 ceph-mon[33017]: pgmap v7360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:23:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12776.885855 secs 2024-04-26T15:23:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:23:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12779.681410 secs 2024-04-26T15:23:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:23:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12779.681410 secs 2024-04-26T15:24:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:23:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12779.681410 secs 2024-04-26T15:24:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:00 smithi033 ceph-mon[27845]: pgmap v7361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:00 smithi096 ceph-mon[32753]: pgmap v7361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:00 smithi194 ceph-mon[33017]: pgmap v7361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:02 smithi033 ceph-mon[27845]: pgmap v7362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:02 smithi096 ceph-mon[32753]: pgmap v7362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:02 smithi194 ceph-mon[33017]: pgmap v7362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12781.885937 secs 2024-04-26T15:24:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12781.885937 secs 2024-04-26T15:24:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12781.885937 secs 2024-04-26T15:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:04 smithi096 ceph-mon[32753]: pgmap v7363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12784.681519 secs 2024-04-26T15:24:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:04 smithi194 ceph-mon[33017]: pgmap v7363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12784.681519 secs 2024-04-26T15:24:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:04 smithi033 ceph-mon[27845]: pgmap v7363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12784.681519 secs 2024-04-26T15:24:06.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:06 smithi033 ceph-mon[27845]: pgmap v7364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:06 smithi194 ceph-mon[33017]: pgmap v7364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:06 smithi096 ceph-mon[32753]: pgmap v7364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:08.969+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:24:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:08.969+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:24:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:08.970+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:24:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:08.970+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:24:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:08 smithi194 ceph-mon[33017]: pgmap v7365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12786.886018 secs 2024-04-26T15:24:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:08 smithi033 ceph-mon[27845]: pgmap v7365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12786.886018 secs 2024-04-26T15:24:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:08 smithi096 ceph-mon[32753]: pgmap v7365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12786.886018 secs 2024-04-26T15:24:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12789.681655 secs 2024-04-26T15:24:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12789.681655 secs 2024-04-26T15:24:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12789.681655 secs 2024-04-26T15:24:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:10 smithi194 ceph-mon[33017]: pgmap v7366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:10 smithi033 ceph-mon[27845]: pgmap v7366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:10 smithi096 ceph-mon[32753]: pgmap v7366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:12.140+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:12.140+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:12.140+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:24:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:12 smithi194 ceph-mon[33017]: pgmap v7367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:12 smithi033 ceph-mon[27845]: pgmap v7367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:12 smithi096 ceph-mon[32753]: pgmap v7367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12791.886100 secs 2024-04-26T15:24:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12791.886100 secs 2024-04-26T15:24:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12791.886100 secs 2024-04-26T15:24:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:14 smithi194 ceph-mon[33017]: pgmap v7368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12794.681789 secs 2024-04-26T15:24:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:14 smithi033 ceph-mon[27845]: pgmap v7368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12794.681789 secs 2024-04-26T15:24:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:14 smithi096 ceph-mon[32753]: pgmap v7368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12794.681789 secs 2024-04-26T15:24:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:16 smithi194 ceph-mon[33017]: pgmap v7369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:16 smithi033 ceph-mon[27845]: pgmap v7369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:16 smithi096 ceph-mon[32753]: pgmap v7369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:18 smithi194 ceph-mon[33017]: pgmap v7370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12796.886173 secs 2024-04-26T15:24:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:18 smithi033 ceph-mon[27845]: pgmap v7370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12796.886173 secs 2024-04-26T15:24:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:18 smithi096 ceph-mon[32753]: pgmap v7370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12796.886173 secs 2024-04-26T15:24:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12799.681907 secs 2024-04-26T15:24:20.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12799.681907 secs 2024-04-26T15:24:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12799.681907 secs 2024-04-26T15:24:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:20 smithi194 ceph-mon[33017]: pgmap v7371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:20 smithi033 ceph-mon[27845]: pgmap v7371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:20 smithi096 ceph-mon[32753]: pgmap v7371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:22 smithi194 ceph-mon[33017]: pgmap v7372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:22 smithi033 ceph-mon[27845]: pgmap v7372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:22 smithi096 ceph-mon[32753]: pgmap v7372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:24.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12801.886247 secs 2024-04-26T15:24:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12801.886247 secs 2024-04-26T15:24:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12801.886247 secs 2024-04-26T15:24:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:24 smithi194 ceph-mon[33017]: pgmap v7373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12804.682001 secs 2024-04-26T15:24:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:24 smithi033 ceph-mon[27845]: pgmap v7373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12804.682001 secs 2024-04-26T15:24:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:24 smithi096 ceph-mon[32753]: pgmap v7373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12804.682001 secs 2024-04-26T15:24:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:26 smithi194 ceph-mon[33017]: pgmap v7374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:27.156+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:27.156+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:27.156+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:24:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:26 smithi033 ceph-mon[27845]: pgmap v7374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:26 smithi096 ceph-mon[32753]: pgmap v7374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:28 smithi194 ceph-mon[33017]: pgmap v7375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:29.137 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12806.886337 secs 2024-04-26T15:24:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:28 smithi033 ceph-mon[27845]: pgmap v7375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12806.886337 secs 2024-04-26T15:24:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:28 smithi096 ceph-mon[32753]: pgmap v7375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12806.886337 secs 2024-04-26T15:24:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12809.682135 secs 2024-04-26T15:24:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12809.682135 secs 2024-04-26T15:24:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12809.682135 secs 2024-04-26T15:24:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:30 smithi194 ceph-mon[33017]: pgmap v7376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:30 smithi033 ceph-mon[27845]: pgmap v7376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:30 smithi096 ceph-mon[32753]: pgmap v7376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:31 smithi194 ceph-mon[33017]: pgmap v7377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:31 smithi033 ceph-mon[27845]: pgmap v7377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:31 smithi096 ceph-mon[32753]: pgmap v7377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12811.886449 secs 2024-04-26T15:24:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12811.886449 secs 2024-04-26T15:24:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12811.886449 secs 2024-04-26T15:24:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:33 smithi194 ceph-mon[33017]: pgmap v7378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:33 smithi033 ceph-mon[27845]: pgmap v7378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:33 smithi096 ceph-mon[32753]: pgmap v7378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12814.682255 secs 2024-04-26T15:24:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12814.682255 secs 2024-04-26T15:24:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12814.682255 secs 2024-04-26T15:24:36.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:35 smithi033 ceph-mon[27845]: pgmap v7379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:35 smithi194 ceph-mon[33017]: pgmap v7379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:35 smithi096 ceph-mon[32753]: pgmap v7379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:37 smithi194 ceph-mon[33017]: pgmap v7380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12816.886548 secs 2024-04-26T15:24:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:37 smithi033 ceph-mon[27845]: pgmap v7380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12816.886548 secs 2024-04-26T15:24:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:37 smithi096 ceph-mon[32753]: pgmap v7380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12816.886548 secs 2024-04-26T15:24:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12819.682380 secs 2024-04-26T15:24:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:39 smithi194 ceph-mon[33017]: pgmap v7381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12819.682380 secs 2024-04-26T15:24:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:39 smithi033 ceph-mon[27845]: pgmap v7381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12819.682380 secs 2024-04-26T15:24:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:39 smithi096 ceph-mon[32753]: pgmap v7381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:42.170+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:42.170+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:42.170+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:24:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:41 smithi194 ceph-mon[33017]: pgmap v7382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:24:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:41 smithi033 ceph-mon[27845]: pgmap v7382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:24:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:41 smithi096 ceph-mon[32753]: pgmap v7382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:24:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:24:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:24:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:43 smithi194 ceph-mon[33017]: pgmap v7383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:43 smithi194 ceph-mon[33017]: pgmap v7384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:24:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12821.886650 secs 2024-04-26T15:24:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:24:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:24:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:43 smithi033 ceph-mon[27845]: pgmap v7383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:43 smithi033 ceph-mon[27845]: pgmap v7384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:24:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12821.886650 secs 2024-04-26T15:24:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:24:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:24:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:42 smithi096 ceph-mon[32753]: pgmap v7383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:42 smithi096 ceph-mon[32753]: pgmap v7384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:24:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12821.886650 secs 2024-04-26T15:24:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12824.682497 secs 2024-04-26T15:24:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12824.682497 secs 2024-04-26T15:24:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12824.682497 secs 2024-04-26T15:24:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:45 smithi194 ceph-mon[33017]: pgmap v7385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:45 smithi033 ceph-mon[27845]: pgmap v7385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:45 smithi096 ceph-mon[32753]: pgmap v7385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:47 smithi194 ceph-mon[33017]: pgmap v7386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:47 smithi033 ceph-mon[27845]: pgmap v7386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:47 smithi096 ceph-mon[32753]: pgmap v7386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12826.886749 secs 2024-04-26T15:24:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12826.886749 secs 2024-04-26T15:24:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12826.886749 secs 2024-04-26T15:24:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12829.682615 secs 2024-04-26T15:24:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:49 smithi194 ceph-mon[33017]: pgmap v7387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12829.682615 secs 2024-04-26T15:24:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:49 smithi033 ceph-mon[27845]: pgmap v7387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12829.682615 secs 2024-04-26T15:24:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:49 smithi096 ceph-mon[32753]: pgmap v7387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:51 smithi194 ceph-mon[33017]: pgmap v7388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:51 smithi033 ceph-mon[27845]: pgmap v7388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:51 smithi096 ceph-mon[32753]: pgmap v7388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:24:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:53 smithi194 ceph-mon[33017]: pgmap v7389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:24:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12831.886869 secs 2024-04-26T15:24:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:53 smithi033 ceph-mon[27845]: pgmap v7389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:24:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12831.886869 secs 2024-04-26T15:24:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:53 smithi096 ceph-mon[32753]: pgmap v7389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:24:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12831.886869 secs 2024-04-26T15:24:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12834.682734 secs 2024-04-26T15:24:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12834.682734 secs 2024-04-26T15:24:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12834.682734 secs 2024-04-26T15:24:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:55 smithi033 ceph-mon[27845]: pgmap v7390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:55 smithi096 ceph-mon[32753]: pgmap v7390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:55 smithi194 ceph-mon[33017]: pgmap v7390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:24:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:57.182+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:57.182+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:24:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:24:57.182+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:24:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:57 smithi033 ceph-mon[27845]: pgmap v7391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:57 smithi096 ceph-mon[32753]: pgmap v7391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:57 smithi194 ceph-mon[33017]: pgmap v7391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12836.886978 secs 2024-04-26T15:24:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12836.886978 secs 2024-04-26T15:24:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12836.886978 secs 2024-04-26T15:24:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12839.682851 secs 2024-04-26T15:24:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:24:59 smithi033 ceph-mon[27845]: pgmap v7392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:24:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12839.682851 secs 2024-04-26T15:24:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:24:59 smithi096 ceph-mon[32753]: pgmap v7392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12839.682851 secs 2024-04-26T15:25:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:24:59 smithi194 ceph-mon[33017]: pgmap v7392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:01 smithi033 ceph-mon[27845]: pgmap v7393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:01 smithi096 ceph-mon[32753]: pgmap v7393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:01 smithi194 ceph-mon[33017]: pgmap v7393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:03 smithi033 ceph-mon[27845]: pgmap v7394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12841.887086 secs 2024-04-26T15:25:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:03 smithi096 ceph-mon[32753]: pgmap v7394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12841.887086 secs 2024-04-26T15:25:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:03 smithi194 ceph-mon[33017]: pgmap v7394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12841.887086 secs 2024-04-26T15:25:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12844.682964 secs 2024-04-26T15:25:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12844.682964 secs 2024-04-26T15:25:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12844.682964 secs 2024-04-26T15:25:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:05 smithi033 ceph-mon[27845]: pgmap v7395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:05 smithi096 ceph-mon[32753]: pgmap v7395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:05 smithi194 ceph-mon[33017]: pgmap v7395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:07 smithi033 ceph-mon[27845]: pgmap v7396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:07 smithi096 ceph-mon[32753]: pgmap v7396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:07 smithi194 ceph-mon[33017]: pgmap v7396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12846.887174 secs 2024-04-26T15:25:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12846.887174 secs 2024-04-26T15:25:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12846.887174 secs 2024-04-26T15:25:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12849.683079 secs 2024-04-26T15:25:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:09 smithi033 ceph-mon[27845]: pgmap v7397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12849.683079 secs 2024-04-26T15:25:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:09 smithi096 ceph-mon[32753]: pgmap v7397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12849.683079 secs 2024-04-26T15:25:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:09 smithi194 ceph-mon[33017]: pgmap v7397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:11 smithi033 ceph-mon[27845]: pgmap v7398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:11 smithi096 ceph-mon[32753]: pgmap v7398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:11 smithi194 ceph-mon[33017]: pgmap v7398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:12.191+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:12.191+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:12.191+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:25:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:13 smithi096 ceph-mon[32753]: pgmap v7399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12851.887255 secs 2024-04-26T15:25:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:13 smithi194 ceph-mon[33017]: pgmap v7399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12851.887255 secs 2024-04-26T15:25:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:13 smithi033 ceph-mon[27845]: pgmap v7399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12851.887255 secs 2024-04-26T15:25:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12854.683192 secs 2024-04-26T15:25:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12854.683192 secs 2024-04-26T15:25:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12854.683192 secs 2024-04-26T15:25:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:15 smithi194 ceph-mon[33017]: pgmap v7400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:15 smithi033 ceph-mon[27845]: pgmap v7400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:15 smithi096 ceph-mon[32753]: pgmap v7400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:17 smithi194 ceph-mon[33017]: pgmap v7401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:17 smithi033 ceph-mon[27845]: pgmap v7401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:17 smithi096 ceph-mon[32753]: pgmap v7401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12856.887341 secs 2024-04-26T15:25:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12856.887341 secs 2024-04-26T15:25:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12856.887341 secs 2024-04-26T15:25:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12859.683309 secs 2024-04-26T15:25:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:19 smithi194 ceph-mon[33017]: pgmap v7402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12859.683309 secs 2024-04-26T15:25:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:19 smithi033 ceph-mon[27845]: pgmap v7402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12859.683309 secs 2024-04-26T15:25:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:19 smithi096 ceph-mon[32753]: pgmap v7402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:22.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:21 smithi033 ceph-mon[27845]: pgmap v7403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:21 smithi194 ceph-mon[33017]: pgmap v7403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:21 smithi096 ceph-mon[32753]: pgmap v7403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:23 smithi194 ceph-mon[33017]: pgmap v7404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12861.887428 secs 2024-04-26T15:25:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:23 smithi033 ceph-mon[27845]: pgmap v7404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12861.887428 secs 2024-04-26T15:25:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:23 smithi096 ceph-mon[32753]: pgmap v7404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12861.887428 secs 2024-04-26T15:25:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12864.683424 secs 2024-04-26T15:25:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12864.683424 secs 2024-04-26T15:25:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12864.683424 secs 2024-04-26T15:25:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:25 smithi194 ceph-mon[33017]: pgmap v7405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:25 smithi033 ceph-mon[27845]: pgmap v7405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:25 smithi096 ceph-mon[32753]: pgmap v7405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:27.197+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:27.197+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:27.197+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:25:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:27 smithi194 ceph-mon[33017]: pgmap v7406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:27 smithi033 ceph-mon[27845]: pgmap v7406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:27 smithi096 ceph-mon[32753]: pgmap v7406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12866.887515 secs 2024-04-26T15:25:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12866.887515 secs 2024-04-26T15:25:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12866.887515 secs 2024-04-26T15:25:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12869.683555 secs 2024-04-26T15:25:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:29 smithi194 ceph-mon[33017]: pgmap v7407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12869.683555 secs 2024-04-26T15:25:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:29 smithi033 ceph-mon[27845]: pgmap v7407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12869.683555 secs 2024-04-26T15:25:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:29 smithi096 ceph-mon[32753]: pgmap v7407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:31 smithi194 ceph-mon[33017]: pgmap v7408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:31 smithi033 ceph-mon[27845]: pgmap v7408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:31 smithi096 ceph-mon[32753]: pgmap v7408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:33 smithi194 ceph-mon[33017]: pgmap v7409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12871.887600 secs 2024-04-26T15:25:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:33 smithi033 ceph-mon[27845]: pgmap v7409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12871.887600 secs 2024-04-26T15:25:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:33 smithi096 ceph-mon[32753]: pgmap v7409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12871.887600 secs 2024-04-26T15:25:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12874.683661 secs 2024-04-26T15:25:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12874.683661 secs 2024-04-26T15:25:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12874.683661 secs 2024-04-26T15:25:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:35 smithi194 ceph-mon[33017]: pgmap v7410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:35 smithi033 ceph-mon[27845]: pgmap v7410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:35 smithi096 ceph-mon[32753]: pgmap v7410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:37 smithi194 ceph-mon[33017]: pgmap v7411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:37 smithi033 ceph-mon[27845]: pgmap v7411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:37 smithi096 ceph-mon[32753]: pgmap v7411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12876.887659 secs 2024-04-26T15:25:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12876.887659 secs 2024-04-26T15:25:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12876.887659 secs 2024-04-26T15:25:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12879.683779 secs 2024-04-26T15:25:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:39 smithi194 ceph-mon[33017]: pgmap v7412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12879.683779 secs 2024-04-26T15:25:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:39 smithi033 ceph-mon[27845]: pgmap v7412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12879.683779 secs 2024-04-26T15:25:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:39 smithi096 ceph-mon[32753]: pgmap v7412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:41 smithi194 ceph-mon[33017]: pgmap v7413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:41 smithi033 ceph-mon[27845]: pgmap v7413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:41 smithi096 ceph-mon[32753]: pgmap v7413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:25:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:42.212+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:42.212+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:42.212+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:25:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:42 smithi194 ceph-mon[33017]: pgmap v7414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:42 smithi033 ceph-mon[27845]: pgmap v7414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:42 smithi096 ceph-mon[32753]: pgmap v7414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12881.887744 secs 2024-04-26T15:25:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:25:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12881.887744 secs 2024-04-26T15:25:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:25:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12881.887744 secs 2024-04-26T15:25:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:25:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12884.683865 secs 2024-04-26T15:25:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:25:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:25:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:44 smithi194 ceph-mon[33017]: pgmap v7415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:44 smithi194 ceph-mon[33017]: pgmap v7416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:25:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:25:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12884.683865 secs 2024-04-26T15:25:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:25:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:25:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:44 smithi033 ceph-mon[27845]: pgmap v7415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:44 smithi033 ceph-mon[27845]: pgmap v7416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:25:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:25:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12884.683865 secs 2024-04-26T15:25:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:44 smithi096 ceph-mon[32753]: pgmap v7415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:44 smithi096 ceph-mon[32753]: pgmap v7416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:25:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:47 smithi194 ceph-mon[33017]: pgmap v7417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:47 smithi033 ceph-mon[27845]: pgmap v7417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:47 smithi096 ceph-mon[32753]: pgmap v7417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12886.887831 secs 2024-04-26T15:25:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12886.887831 secs 2024-04-26T15:25:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12886.887831 secs 2024-04-26T15:25:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:49 smithi194 ceph-mon[33017]: pgmap v7418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12889.683999 secs 2024-04-26T15:25:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:49 smithi033 ceph-mon[27845]: pgmap v7418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12889.683999 secs 2024-04-26T15:25:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:49 smithi096 ceph-mon[32753]: pgmap v7418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12889.683999 secs 2024-04-26T15:25:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:51 smithi194 ceph-mon[33017]: pgmap v7419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:51 smithi033 ceph-mon[27845]: pgmap v7419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:51 smithi096 ceph-mon[32753]: pgmap v7419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:53 smithi194 ceph-mon[33017]: pgmap v7420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12891.887916 secs 2024-04-26T15:25:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:53 smithi033 ceph-mon[27845]: pgmap v7420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12891.887916 secs 2024-04-26T15:25:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:53 smithi096 ceph-mon[32753]: pgmap v7420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:25:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12891.887916 secs 2024-04-26T15:25:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12894.684130 secs 2024-04-26T15:25:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12894.684130 secs 2024-04-26T15:25:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12894.684130 secs 2024-04-26T15:25:55.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:55 smithi096 ceph-mon[32753]: pgmap v7421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:25:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:55 smithi194 ceph-mon[33017]: pgmap v7421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:25:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:55 smithi033 ceph-mon[27845]: pgmap v7421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:25:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:57.215+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:57.215+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:25:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:25:57.215+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:25:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:57 smithi194 ceph-mon[33017]: pgmap v7422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:25:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:57 smithi033 ceph-mon[27845]: pgmap v7422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:25:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:57 smithi096 ceph-mon[32753]: pgmap v7422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:25:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12896.888009 secs 2024-04-26T15:25:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12896.888009 secs 2024-04-26T15:25:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12896.888009 secs 2024-04-26T15:25:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:59 smithi194 ceph-mon[33017]: pgmap v7423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:25:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12899.684245 secs 2024-04-26T15:25:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:59 smithi033 ceph-mon[27845]: pgmap v7423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:25:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12899.684245 secs 2024-04-26T15:25:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:59 smithi096 ceph-mon[32753]: pgmap v7423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:25:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:25:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12899.684245 secs 2024-04-26T15:26:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:01 smithi194 ceph-mon[33017]: pgmap v7424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:01 smithi033 ceph-mon[27845]: pgmap v7424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:01 smithi096 ceph-mon[32753]: pgmap v7424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:03 smithi194 ceph-mon[33017]: pgmap v7425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12901.888092 secs 2024-04-26T15:26:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:03 smithi033 ceph-mon[27845]: pgmap v7425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12901.888092 secs 2024-04-26T15:26:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:03 smithi096 ceph-mon[32753]: pgmap v7425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12901.888092 secs 2024-04-26T15:26:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12904.684358 secs 2024-04-26T15:26:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12904.684358 secs 2024-04-26T15:26:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12904.684358 secs 2024-04-26T15:26:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:05 smithi194 ceph-mon[33017]: pgmap v7426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:05 smithi033 ceph-mon[27845]: pgmap v7426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:05 smithi096 ceph-mon[32753]: pgmap v7426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:07 smithi194 ceph-mon[33017]: pgmap v7427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:07 smithi033 ceph-mon[27845]: pgmap v7427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:07 smithi096 ceph-mon[32753]: pgmap v7427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12906.888177 secs 2024-04-26T15:26:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12906.888177 secs 2024-04-26T15:26:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12906.888177 secs 2024-04-26T15:26:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:09 smithi194 ceph-mon[33017]: pgmap v7428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12909.684473 secs 2024-04-26T15:26:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:09 smithi033 ceph-mon[27845]: pgmap v7428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12909.684473 secs 2024-04-26T15:26:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:09 smithi096 ceph-mon[32753]: pgmap v7428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12909.684473 secs 2024-04-26T15:26:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:11 smithi194 ceph-mon[33017]: pgmap v7429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:11 smithi033 ceph-mon[27845]: pgmap v7429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:11 smithi096 ceph-mon[32753]: pgmap v7429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:12.232+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:12.232+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:26:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:12.232+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:13 smithi194 ceph-mon[33017]: pgmap v7430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12911.888262 secs 2024-04-26T15:26:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:13 smithi033 ceph-mon[27845]: pgmap v7430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12911.888262 secs 2024-04-26T15:26:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:13 smithi096 ceph-mon[32753]: pgmap v7430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12911.888262 secs 2024-04-26T15:26:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12914.684590 secs 2024-04-26T15:26:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12914.684590 secs 2024-04-26T15:26:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12914.684590 secs 2024-04-26T15:26:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:15 smithi194 ceph-mon[33017]: pgmap v7431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:15 smithi033 ceph-mon[27845]: pgmap v7431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:15 smithi096 ceph-mon[32753]: pgmap v7431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:17 smithi033 ceph-mon[27845]: pgmap v7432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:17 smithi096 ceph-mon[32753]: pgmap v7432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:17 smithi194 ceph-mon[33017]: pgmap v7432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12916.888324 secs 2024-04-26T15:26:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12916.888324 secs 2024-04-26T15:26:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12916.888324 secs 2024-04-26T15:26:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:19 smithi033 ceph-mon[27845]: pgmap v7433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12919.684707 secs 2024-04-26T15:26:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:19 smithi096 ceph-mon[32753]: pgmap v7433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12919.684707 secs 2024-04-26T15:26:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:19 smithi194 ceph-mon[33017]: pgmap v7433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12919.684707 secs 2024-04-26T15:26:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:21 smithi033 ceph-mon[27845]: pgmap v7434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:21 smithi096 ceph-mon[32753]: pgmap v7434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T15:26:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T15:26:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T15:26:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062040 comp rx=0 tx=0).stop 2024-04-26T15:26:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3cc0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T15:26:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T15:26:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3cc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T15:26:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.746+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T15:26:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.747+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T15:26:21.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:21.749+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0067000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T15:26:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:21 smithi194 ceph-mon[33017]: pgmap v7434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:23.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:23 smithi194 ceph-mon[33017]: pgmap v7435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:23.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12921.888408 secs 2024-04-26T15:26:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:23 smithi033 ceph-mon[27845]: pgmap v7435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12921.888408 secs 2024-04-26T15:26:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:23 smithi096 ceph-mon[32753]: pgmap v7435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12921.888408 secs 2024-04-26T15:26:23.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:23.992+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:26:23.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:23.992+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:26:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:23.993+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:26:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:23.993+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x7fef00077780 2024-04-26T15:26:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12924.684841 secs 2024-04-26T15:26:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12924.684841 secs 2024-04-26T15:26:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12924.684841 secs 2024-04-26T15:26:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:25 smithi033 ceph-mon[27845]: pgmap v7436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:25 smithi096 ceph-mon[32753]: pgmap v7436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:25 smithi194 ceph-mon[33017]: pgmap v7436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:27.238+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:27.238+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:27.238+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:27 smithi033 ceph-mon[27845]: pgmap v7437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:27 smithi096 ceph-mon[32753]: pgmap v7437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:27 smithi194 ceph-mon[33017]: pgmap v7437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12926.888497 secs 2024-04-26T15:26:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12926.888497 secs 2024-04-26T15:26:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12926.888497 secs 2024-04-26T15:26:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:29 smithi033 ceph-mon[27845]: pgmap v7438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12929.684960 secs 2024-04-26T15:26:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:29 smithi096 ceph-mon[32753]: pgmap v7438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12929.684960 secs 2024-04-26T15:26:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:29 smithi194 ceph-mon[33017]: pgmap v7438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12929.684960 secs 2024-04-26T15:26:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:31 smithi033 ceph-mon[27845]: pgmap v7439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:31 smithi096 ceph-mon[32753]: pgmap v7439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:31 smithi194 ceph-mon[33017]: pgmap v7439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:33 smithi033 ceph-mon[27845]: pgmap v7440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12931.888593 secs 2024-04-26T15:26:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:33 smithi096 ceph-mon[32753]: pgmap v7440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12931.888593 secs 2024-04-26T15:26:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:33 smithi194 ceph-mon[33017]: pgmap v7440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12931.888593 secs 2024-04-26T15:26:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12934.685075 secs 2024-04-26T15:26:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12934.685075 secs 2024-04-26T15:26:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12934.685075 secs 2024-04-26T15:26:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:35 smithi096 ceph-mon[32753]: pgmap v7441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:35 smithi194 ceph-mon[33017]: pgmap v7441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:35 smithi033 ceph-mon[27845]: pgmap v7441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:37.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:37 smithi033 ceph-mon[27845]: pgmap v7442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:37 smithi194 ceph-mon[33017]: pgmap v7442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:37 smithi096 ceph-mon[32753]: pgmap v7442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12936.888694 secs 2024-04-26T15:26:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12936.888694 secs 2024-04-26T15:26:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12936.888694 secs 2024-04-26T15:26:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:39 smithi194 ceph-mon[33017]: pgmap v7443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12939.685172 secs 2024-04-26T15:26:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:39 smithi033 ceph-mon[27845]: pgmap v7443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12939.685172 secs 2024-04-26T15:26:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:39 smithi096 ceph-mon[32753]: pgmap v7443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12939.685172 secs 2024-04-26T15:26:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:41 smithi194 ceph-mon[33017]: pgmap v7444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:41 smithi033 ceph-mon[27845]: pgmap v7444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:41 smithi096 ceph-mon[32753]: pgmap v7444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:42.246+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:42.247+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:42.247+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:26:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:43 smithi194 ceph-mon[33017]: pgmap v7445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12941.888794 secs 2024-04-26T15:26:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:43 smithi033 ceph-mon[27845]: pgmap v7445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12941.888794 secs 2024-04-26T15:26:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:43 smithi096 ceph-mon[32753]: pgmap v7445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12941.888794 secs 2024-04-26T15:26:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12944.685320 secs 2024-04-26T15:26:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:26:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12944.685320 secs 2024-04-26T15:26:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:26:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12944.685320 secs 2024-04-26T15:26:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:26:45.797 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:45 smithi194 ceph-mon[33017]: pgmap v7446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:45.804 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:45 smithi096 ceph-mon[32753]: pgmap v7446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:45 smithi033 ceph-mon[27845]: pgmap v7446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:47.846 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:47 smithi033 ceph-mon[27845]: pgmap v7447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:47 smithi194 ceph-mon[33017]: pgmap v7447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:47 smithi096 ceph-mon[32753]: pgmap v7447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:48.597 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12946.888896 secs 2024-04-26T15:26:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12946.888896 secs 2024-04-26T15:26:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12946.888896 secs 2024-04-26T15:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:49 smithi194 ceph-mon[33017]: pgmap v7448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12949.685432 secs 2024-04-26T15:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:49 smithi033 ceph-mon[27845]: pgmap v7448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12949.685432 secs 2024-04-26T15:26:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:49 smithi096 ceph-mon[32753]: pgmap v7448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12949.685432 secs 2024-04-26T15:26:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:51 smithi194 ceph-mon[33017]: pgmap v7449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:51 smithi033 ceph-mon[27845]: pgmap v7449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:51 smithi096 ceph-mon[32753]: pgmap v7449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:26:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:26:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:26:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: pgmap v7450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: pgmap v7451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:52 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12951.889001 secs 2024-04-26T15:26:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:26:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:26:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: pgmap v7450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: pgmap v7451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12951.889001 secs 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: pgmap v7450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: pgmap v7451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:26:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12951.889001 secs 2024-04-26T15:26:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12954.685556 secs 2024-04-26T15:26:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:54 smithi194 ceph-mon[33017]: pgmap v7452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12954.685556 secs 2024-04-26T15:26:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:54 smithi033 ceph-mon[27845]: pgmap v7452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12954.685556 secs 2024-04-26T15:26:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:54 smithi096 ceph-mon[32753]: pgmap v7452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:57.263+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:57.263+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:26:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:26:57.263+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:26:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:56 smithi194 ceph-mon[33017]: pgmap v7453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:57.356 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:56 smithi033 ceph-mon[27845]: pgmap v7453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:56 smithi096 ceph-mon[32753]: pgmap v7453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12956.889091 secs 2024-04-26T15:26:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12956.889091 secs 2024-04-26T15:26:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12956.889091 secs 2024-04-26T15:26:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:26:59 smithi096 ceph-mon[32753]: pgmap v7454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:59.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:26:59 smithi194 ceph-mon[33017]: pgmap v7454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:26:59 smithi033 ceph-mon[27845]: pgmap v7454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:27:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12959.685681 secs 2024-04-26T15:27:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12959.685681 secs 2024-04-26T15:27:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12959.685681 secs 2024-04-26T15:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:01 smithi194 ceph-mon[33017]: pgmap v7455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:27:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:01 smithi033 ceph-mon[27845]: pgmap v7455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:27:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:01 smithi096 ceph-mon[32753]: pgmap v7455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:27:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:03 smithi194 ceph-mon[33017]: pgmap v7456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:27:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12961.889183 secs 2024-04-26T15:27:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:03 smithi033 ceph-mon[27845]: pgmap v7456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:27:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12961.889183 secs 2024-04-26T15:27:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:03 smithi096 ceph-mon[32753]: pgmap v7456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:27:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12961.889183 secs 2024-04-26T15:27:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12964.685752 secs 2024-04-26T15:27:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:05 smithi194 ceph-mon[33017]: pgmap v7457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12964.685752 secs 2024-04-26T15:27:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:05 smithi033 ceph-mon[27845]: pgmap v7457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12964.685752 secs 2024-04-26T15:27:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:05 smithi096 ceph-mon[32753]: pgmap v7457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:07 smithi194 ceph-mon[33017]: pgmap v7458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:07 smithi033 ceph-mon[27845]: pgmap v7458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:07 smithi096 ceph-mon[32753]: pgmap v7458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12966.889245 secs 2024-04-26T15:27:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12966.889245 secs 2024-04-26T15:27:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12966.889245 secs 2024-04-26T15:27:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:09 smithi194 ceph-mon[33017]: pgmap v7459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:09 smithi033 ceph-mon[27845]: pgmap v7459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:09 smithi096 ceph-mon[32753]: pgmap v7459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12969.685874 secs 2024-04-26T15:27:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12969.685874 secs 2024-04-26T15:27:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12969.685874 secs 2024-04-26T15:27:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:11 smithi194 ceph-mon[33017]: pgmap v7460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:11 smithi033 ceph-mon[27845]: pgmap v7460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:11 smithi096 ceph-mon[32753]: pgmap v7460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:12.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:12.278+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:12.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:12.278+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:12.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:12.278+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:27:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:13 smithi194 ceph-mon[33017]: pgmap v7461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12971.889330 secs 2024-04-26T15:27:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:13 smithi033 ceph-mon[27845]: pgmap v7461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12971.889330 secs 2024-04-26T15:27:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:13 smithi096 ceph-mon[32753]: pgmap v7461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12971.889330 secs 2024-04-26T15:27:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12974.686008 secs 2024-04-26T15:27:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12974.686008 secs 2024-04-26T15:27:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12974.686008 secs 2024-04-26T15:27:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:15 smithi194 ceph-mon[33017]: pgmap v7462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:15 smithi033 ceph-mon[27845]: pgmap v7462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:15 smithi096 ceph-mon[32753]: pgmap v7462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:17 smithi194 ceph-mon[33017]: pgmap v7463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:17 smithi033 ceph-mon[27845]: pgmap v7463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:17 smithi096 ceph-mon[32753]: pgmap v7463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12976.889419 secs 2024-04-26T15:27:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12976.889419 secs 2024-04-26T15:27:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12976.889419 secs 2024-04-26T15:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:19 smithi194 ceph-mon[33017]: pgmap v7464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12979.686171 secs 2024-04-26T15:27:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:19 smithi033 ceph-mon[27845]: pgmap v7464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12979.686171 secs 2024-04-26T15:27:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:19 smithi096 ceph-mon[32753]: pgmap v7464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12979.686171 secs 2024-04-26T15:27:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:21 smithi194 ceph-mon[33017]: pgmap v7465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:21 smithi033 ceph-mon[27845]: pgmap v7465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:21 smithi096 ceph-mon[32753]: pgmap v7465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:23 smithi033 ceph-mon[27845]: pgmap v7466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12981.889503 secs 2024-04-26T15:27:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:23 smithi096 ceph-mon[32753]: pgmap v7466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12981.889503 secs 2024-04-26T15:27:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:23 smithi194 ceph-mon[33017]: pgmap v7466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12981.889503 secs 2024-04-26T15:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12984.686289 secs 2024-04-26T15:27:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12984.686289 secs 2024-04-26T15:27:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12984.686289 secs 2024-04-26T15:27:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:25 smithi033 ceph-mon[27845]: pgmap v7467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:25 smithi096 ceph-mon[32753]: pgmap v7467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:25 smithi194 ceph-mon[33017]: pgmap v7467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:27.285+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:27.286+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:27.286+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:27:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:27 smithi033 ceph-mon[27845]: pgmap v7468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:27 smithi096 ceph-mon[32753]: pgmap v7468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:27 smithi194 ceph-mon[33017]: pgmap v7468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:28.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12986.889589 secs 2024-04-26T15:27:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12986.889589 secs 2024-04-26T15:27:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12986.889589 secs 2024-04-26T15:27:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:29 smithi033 ceph-mon[27845]: pgmap v7469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12989.686406 secs 2024-04-26T15:27:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:29 smithi096 ceph-mon[32753]: pgmap v7469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12989.686406 secs 2024-04-26T15:27:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:29 smithi194 ceph-mon[33017]: pgmap v7469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12989.686406 secs 2024-04-26T15:27:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:31 smithi033 ceph-mon[27845]: pgmap v7470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:31 smithi096 ceph-mon[32753]: pgmap v7470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:31 smithi194 ceph-mon[33017]: pgmap v7470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:33 smithi033 ceph-mon[27845]: pgmap v7471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:33.438 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12991.889674 secs 2024-04-26T15:27:33.438 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:33 smithi096 ceph-mon[32753]: pgmap v7471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:33.438 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12991.889674 secs 2024-04-26T15:27:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:33 smithi194 ceph-mon[33017]: pgmap v7471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12991.889674 secs 2024-04-26T15:27:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12994.686533 secs 2024-04-26T15:27:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12994.686533 secs 2024-04-26T15:27:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12994.686533 secs 2024-04-26T15:27:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:35 smithi033 ceph-mon[27845]: pgmap v7472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:35 smithi096 ceph-mon[32753]: pgmap v7472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:35 smithi194 ceph-mon[33017]: pgmap v7472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:37 smithi194 ceph-mon[33017]: pgmap v7473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:38.129 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:37 smithi033 ceph-mon[27845]: pgmap v7473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:37 smithi096 ceph-mon[32753]: pgmap v7473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:39.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 12996.889762 secs 2024-04-26T15:27:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 12996.889762 secs 2024-04-26T15:27:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 12996.889762 secs 2024-04-26T15:27:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:39 smithi194 ceph-mon[33017]: pgmap v7474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 12999.686678 secs 2024-04-26T15:27:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:39 smithi033 ceph-mon[27845]: pgmap v7474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 12999.686678 secs 2024-04-26T15:27:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:39 smithi096 ceph-mon[32753]: pgmap v7474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 12999.686678 secs 2024-04-26T15:27:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:41 smithi194 ceph-mon[33017]: pgmap v7475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:41 smithi033 ceph-mon[27845]: pgmap v7475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:41 smithi096 ceph-mon[32753]: pgmap v7475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:42.296+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:42.296+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:42.296+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:27:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:43 smithi194 ceph-mon[33017]: pgmap v7476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13001.889848 secs 2024-04-26T15:27:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:43 smithi033 ceph-mon[27845]: pgmap v7476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13001.889848 secs 2024-04-26T15:27:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:43 smithi096 ceph-mon[32753]: pgmap v7476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13001.889848 secs 2024-04-26T15:27:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13004.686739 secs 2024-04-26T15:27:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13004.686739 secs 2024-04-26T15:27:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13004.686739 secs 2024-04-26T15:27:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:45 smithi194 ceph-mon[33017]: pgmap v7477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:45 smithi033 ceph-mon[27845]: pgmap v7477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:45 smithi096 ceph-mon[32753]: pgmap v7477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:47 smithi194 ceph-mon[33017]: pgmap v7478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:47 smithi033 ceph-mon[27845]: pgmap v7478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:47 smithi096 ceph-mon[32753]: pgmap v7478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13006.889934 secs 2024-04-26T15:27:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13006.889934 secs 2024-04-26T15:27:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13006.889934 secs 2024-04-26T15:27:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:49 smithi194 ceph-mon[33017]: pgmap v7479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13009.686858 secs 2024-04-26T15:27:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:49 smithi033 ceph-mon[27845]: pgmap v7479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13009.686858 secs 2024-04-26T15:27:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:49 smithi096 ceph-mon[32753]: pgmap v7479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13009.686858 secs 2024-04-26T15:27:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:51 smithi194 ceph-mon[33017]: pgmap v7480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:51 smithi033 ceph-mon[27845]: pgmap v7480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:51 smithi096 ceph-mon[32753]: pgmap v7480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:27:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:27:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:27:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:27:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:53 smithi194 ceph-mon[33017]: pgmap v7481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13011.890012 secs 2024-04-26T15:27:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:53 smithi033 ceph-mon[27845]: pgmap v7481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13011.890012 secs 2024-04-26T15:27:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:53 smithi096 ceph-mon[32753]: pgmap v7481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13011.890012 secs 2024-04-26T15:27:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13014.686948 secs 2024-04-26T15:27:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:27:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:27:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13014.686948 secs 2024-04-26T15:27:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:27:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:27:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13014.686948 secs 2024-04-26T15:27:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:27:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:27:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:27:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:55 smithi194 ceph-mon[33017]: pgmap v7482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:55 smithi194 ceph-mon[33017]: pgmap v7483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T15:27:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:55 smithi194 ceph-mon[33017]: pgmap v7484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:27:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:55 smithi033 ceph-mon[27845]: pgmap v7482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:55 smithi033 ceph-mon[27845]: pgmap v7483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T15:27:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:55 smithi033 ceph-mon[27845]: pgmap v7484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:27:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:55 smithi096 ceph-mon[32753]: pgmap v7482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:27:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:55 smithi096 ceph-mon[32753]: pgmap v7483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T15:27:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:55 smithi096 ceph-mon[32753]: pgmap v7484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:27:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:57.311+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:57.311+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:27:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:27:57.311+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:27:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:57 smithi194 ceph-mon[33017]: pgmap v7485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:27:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:57 smithi033 ceph-mon[27845]: pgmap v7485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:27:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:57 smithi096 ceph-mon[32753]: pgmap v7485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:27:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13016.890097 secs 2024-04-26T15:27:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:58 smithi194 ceph-mon[33017]: pgmap v7486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:27:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13016.890097 secs 2024-04-26T15:27:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:58 smithi033 ceph-mon[27845]: pgmap v7486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:27:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13016.890097 secs 2024-04-26T15:27:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:58 smithi096 ceph-mon[32753]: pgmap v7486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:28:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13019.687012 secs 2024-04-26T15:28:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13019.687012 secs 2024-04-26T15:28:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13019.687012 secs 2024-04-26T15:28:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:00 smithi194 ceph-mon[33017]: pgmap v7487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:28:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:00 smithi033 ceph-mon[27845]: pgmap v7487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:28:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:00 smithi096 ceph-mon[32753]: pgmap v7487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:28:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:03 smithi033 ceph-mon[27845]: pgmap v7488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:28:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13021.890159 secs 2024-04-26T15:28:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:03 smithi096 ceph-mon[32753]: pgmap v7488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:28:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13021.890159 secs 2024-04-26T15:28:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:03 smithi194 ceph-mon[33017]: pgmap v7488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:28:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13021.890159 secs 2024-04-26T15:28:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13024.687127 secs 2024-04-26T15:28:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13024.687127 secs 2024-04-26T15:28:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13024.687127 secs 2024-04-26T15:28:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:05 smithi033 ceph-mon[27845]: pgmap v7489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:28:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:05 smithi096 ceph-mon[32753]: pgmap v7489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:28:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:05 smithi194 ceph-mon[33017]: pgmap v7489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:28:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:07 smithi033 ceph-mon[27845]: pgmap v7490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:07 smithi096 ceph-mon[32753]: pgmap v7490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:07 smithi194 ceph-mon[33017]: pgmap v7490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13026.890265 secs 2024-04-26T15:28:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13026.890265 secs 2024-04-26T15:28:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13026.890265 secs 2024-04-26T15:28:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13029.687236 secs 2024-04-26T15:28:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:09 smithi033 ceph-mon[27845]: pgmap v7491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13029.687236 secs 2024-04-26T15:28:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:09 smithi096 ceph-mon[32753]: pgmap v7491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13029.687236 secs 2024-04-26T15:28:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:09 smithi194 ceph-mon[33017]: pgmap v7491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:11 smithi096 ceph-mon[32753]: pgmap v7492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:11 smithi194 ceph-mon[33017]: pgmap v7492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:11 smithi033 ceph-mon[27845]: pgmap v7492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:12.314+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:12.314+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:12.314+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:28:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:13 smithi194 ceph-mon[33017]: pgmap v7493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13031.890322 secs 2024-04-26T15:28:14.127 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:13 smithi033 ceph-mon[27845]: pgmap v7493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:14.127 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13031.890322 secs 2024-04-26T15:28:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:13 smithi096 ceph-mon[32753]: pgmap v7493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13031.890322 secs 2024-04-26T15:28:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13034.687311 secs 2024-04-26T15:28:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13034.687311 secs 2024-04-26T15:28:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13034.687311 secs 2024-04-26T15:28:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:15 smithi194 ceph-mon[33017]: pgmap v7494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:15 smithi033 ceph-mon[27845]: pgmap v7494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:15 smithi096 ceph-mon[32753]: pgmap v7494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:17 smithi194 ceph-mon[33017]: pgmap v7495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:17 smithi033 ceph-mon[27845]: pgmap v7495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:17 smithi096 ceph-mon[32753]: pgmap v7495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13036.890408 secs 2024-04-26T15:28:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13036.890408 secs 2024-04-26T15:28:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13036.890408 secs 2024-04-26T15:28:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13039.687429 secs 2024-04-26T15:28:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:19 smithi194 ceph-mon[33017]: pgmap v7496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13039.687429 secs 2024-04-26T15:28:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:19 smithi033 ceph-mon[27845]: pgmap v7496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13039.687429 secs 2024-04-26T15:28:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:19 smithi096 ceph-mon[32753]: pgmap v7496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:21 smithi194 ceph-mon[33017]: pgmap v7497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:21 smithi033 ceph-mon[27845]: pgmap v7497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:21 smithi096 ceph-mon[32753]: pgmap v7497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:23 smithi194 ceph-mon[33017]: pgmap v7498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13041.890495 secs 2024-04-26T15:28:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:23 smithi033 ceph-mon[27845]: pgmap v7498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13041.890495 secs 2024-04-26T15:28:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:23 smithi096 ceph-mon[32753]: pgmap v7498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13041.890495 secs 2024-04-26T15:28:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13044.687552 secs 2024-04-26T15:28:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13044.687552 secs 2024-04-26T15:28:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13044.687552 secs 2024-04-26T15:28:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:25 smithi194 ceph-mon[33017]: pgmap v7499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:25 smithi033 ceph-mon[27845]: pgmap v7499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:25 smithi096 ceph-mon[32753]: pgmap v7499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:27.329+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:27.329+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:27.329+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:28:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:27 smithi194 ceph-mon[33017]: pgmap v7500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:27 smithi033 ceph-mon[27845]: pgmap v7500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:27 smithi096 ceph-mon[32753]: pgmap v7500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13046.890582 secs 2024-04-26T15:28:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13046.890582 secs 2024-04-26T15:28:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13046.890582 secs 2024-04-26T15:28:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13049.687676 secs 2024-04-26T15:28:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:29 smithi033 ceph-mon[27845]: pgmap v7501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13049.687676 secs 2024-04-26T15:28:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:29 smithi194 ceph-mon[33017]: pgmap v7501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13049.687676 secs 2024-04-26T15:28:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:29 smithi096 ceph-mon[32753]: pgmap v7501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:31 smithi194 ceph-mon[33017]: pgmap v7502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:31 smithi033 ceph-mon[27845]: pgmap v7502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:31 smithi096 ceph-mon[32753]: pgmap v7502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:33 smithi194 ceph-mon[33017]: pgmap v7503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13051.890668 secs 2024-04-26T15:28:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:33 smithi033 ceph-mon[27845]: pgmap v7503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13051.890668 secs 2024-04-26T15:28:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:33 smithi096 ceph-mon[32753]: pgmap v7503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13051.890668 secs 2024-04-26T15:28:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13054.687812 secs 2024-04-26T15:28:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13054.687812 secs 2024-04-26T15:28:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13054.687812 secs 2024-04-26T15:28:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:35 smithi194 ceph-mon[33017]: pgmap v7504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:35 smithi033 ceph-mon[27845]: pgmap v7504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:35 smithi096 ceph-mon[32753]: pgmap v7504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:37 smithi194 ceph-mon[33017]: pgmap v7505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:37 smithi033 ceph-mon[27845]: pgmap v7505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:37 smithi096 ceph-mon[32753]: pgmap v7505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:39.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13056.890731 secs 2024-04-26T15:28:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13056.890731 secs 2024-04-26T15:28:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13056.890731 secs 2024-04-26T15:28:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13059.687937 secs 2024-04-26T15:28:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:39 smithi194 ceph-mon[33017]: pgmap v7506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13059.687937 secs 2024-04-26T15:28:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:39 smithi033 ceph-mon[27845]: pgmap v7506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13059.687937 secs 2024-04-26T15:28:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:39 smithi096 ceph-mon[32753]: pgmap v7506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:41 smithi194 ceph-mon[33017]: pgmap v7507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:41 smithi033 ceph-mon[27845]: pgmap v7507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:41 smithi096 ceph-mon[32753]: pgmap v7507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:42.341+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:42.341+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:42.341+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:28:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:43 smithi194 ceph-mon[33017]: pgmap v7508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13061.890841 secs 2024-04-26T15:28:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:43 smithi033 ceph-mon[27845]: pgmap v7508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13061.890841 secs 2024-04-26T15:28:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:43 smithi096 ceph-mon[32753]: pgmap v7508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13061.890841 secs 2024-04-26T15:28:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13064.688054 secs 2024-04-26T15:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13064.688054 secs 2024-04-26T15:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13064.688054 secs 2024-04-26T15:28:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:45 smithi194 ceph-mon[33017]: pgmap v7509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:45 smithi033 ceph-mon[27845]: pgmap v7509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:45 smithi096 ceph-mon[32753]: pgmap v7509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:47 smithi194 ceph-mon[33017]: pgmap v7510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:47 smithi033 ceph-mon[27845]: pgmap v7510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:47 smithi096 ceph-mon[32753]: pgmap v7510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13066.890944 secs 2024-04-26T15:28:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13066.890944 secs 2024-04-26T15:28:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13066.890944 secs 2024-04-26T15:28:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13069.688190 secs 2024-04-26T15:28:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:49 smithi194 ceph-mon[33017]: pgmap v7511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13069.688190 secs 2024-04-26T15:28:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:49 smithi033 ceph-mon[27845]: pgmap v7511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13069.688190 secs 2024-04-26T15:28:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:49 smithi096 ceph-mon[32753]: pgmap v7511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:50 smithi194 ceph-mon[33017]: pgmap v7512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:50 smithi033 ceph-mon[27845]: pgmap v7512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:50 smithi096 ceph-mon[32753]: pgmap v7512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:53 smithi033 ceph-mon[27845]: pgmap v7513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13071.891002 secs 2024-04-26T15:28:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:53 smithi096 ceph-mon[32753]: pgmap v7513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13071.891002 secs 2024-04-26T15:28:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:53 smithi194 ceph-mon[33017]: pgmap v7513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13071.891002 secs 2024-04-26T15:28:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13074.688312 secs 2024-04-26T15:28:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13074.688312 secs 2024-04-26T15:28:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13074.688312 secs 2024-04-26T15:28:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:55 smithi096 ceph-mon[32753]: pgmap v7514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:28:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:55 smithi194 ceph-mon[33017]: pgmap v7514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:28:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:55 smithi033 ceph-mon[27845]: pgmap v7514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:28:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:28:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:28:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:28:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:28:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:28:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:28:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:28:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:28:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:28:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:28:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:57.351+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:57.351+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:28:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:28:57.351+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:28:58.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:57 smithi096 ceph-mon[32753]: pgmap v7515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:58.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:57 smithi096 ceph-mon[32753]: pgmap v7516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:28:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:57 smithi194 ceph-mon[33017]: pgmap v7515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:57 smithi194 ceph-mon[33017]: pgmap v7516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:28:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:57 smithi033 ceph-mon[27845]: pgmap v7515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:28:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:57 smithi033 ceph-mon[27845]: pgmap v7516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:28:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13076.891065 secs 2024-04-26T15:28:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13076.891065 secs 2024-04-26T15:28:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13076.891065 secs 2024-04-26T15:29:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13079.688394 secs 2024-04-26T15:29:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:28:59 smithi194 ceph-mon[33017]: pgmap v7517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13079.688394 secs 2024-04-26T15:29:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:28:59 smithi033 ceph-mon[27845]: pgmap v7517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13079.688394 secs 2024-04-26T15:29:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:28:59 smithi096 ceph-mon[32753]: pgmap v7517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:01 smithi194 ceph-mon[33017]: pgmap v7518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:01 smithi033 ceph-mon[27845]: pgmap v7518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:01 smithi096 ceph-mon[32753]: pgmap v7518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:03 smithi194 ceph-mon[33017]: pgmap v7519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13081.891123 secs 2024-04-26T15:29:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:03 smithi033 ceph-mon[27845]: pgmap v7519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13081.891123 secs 2024-04-26T15:29:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:03 smithi096 ceph-mon[32753]: pgmap v7519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13081.891123 secs 2024-04-26T15:29:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13084.688470 secs 2024-04-26T15:29:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13084.688470 secs 2024-04-26T15:29:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13084.688470 secs 2024-04-26T15:29:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:05 smithi194 ceph-mon[33017]: pgmap v7520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T15:29:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:05 smithi033 ceph-mon[27845]: pgmap v7520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T15:29:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:05 smithi096 ceph-mon[32753]: pgmap v7520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T15:29:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:07 smithi194 ceph-mon[33017]: pgmap v7521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:29:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:07 smithi033 ceph-mon[27845]: pgmap v7521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:29:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:07 smithi096 ceph-mon[32753]: pgmap v7521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:29:09.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13086.891183 secs 2024-04-26T15:29:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13086.891183 secs 2024-04-26T15:29:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13086.891183 secs 2024-04-26T15:29:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13089.688614 secs 2024-04-26T15:29:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:09 smithi194 ceph-mon[33017]: pgmap v7522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13089.688614 secs 2024-04-26T15:29:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:09 smithi033 ceph-mon[27845]: pgmap v7522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13089.688614 secs 2024-04-26T15:29:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:09 smithi096 ceph-mon[32753]: pgmap v7522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:11 smithi194 ceph-mon[33017]: pgmap v7523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:11 smithi033 ceph-mon[27845]: pgmap v7523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:11 smithi096 ceph-mon[32753]: pgmap v7523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:12.361 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:12.362+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:12.361 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:12.362+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:12.362 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:12.362+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:29:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:13 smithi194 ceph-mon[33017]: pgmap v7524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13091.891237 secs 2024-04-26T15:29:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:13 smithi033 ceph-mon[27845]: pgmap v7524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13091.891237 secs 2024-04-26T15:29:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:13 smithi096 ceph-mon[32753]: pgmap v7524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13091.891237 secs 2024-04-26T15:29:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13094.688759 secs 2024-04-26T15:29:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13094.688759 secs 2024-04-26T15:29:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13094.688759 secs 2024-04-26T15:29:16.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:15 smithi033 ceph-mon[27845]: pgmap v7525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:15 smithi194 ceph-mon[33017]: pgmap v7525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:15 smithi096 ceph-mon[32753]: pgmap v7525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:17 smithi194 ceph-mon[33017]: pgmap v7526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:17 smithi033 ceph-mon[27845]: pgmap v7526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:17 smithi096 ceph-mon[32753]: pgmap v7526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13096.891320 secs 2024-04-26T15:29:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13096.891320 secs 2024-04-26T15:29:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13096.891320 secs 2024-04-26T15:29:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13099.688902 secs 2024-04-26T15:29:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:19 smithi194 ceph-mon[33017]: pgmap v7527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13099.688902 secs 2024-04-26T15:29:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:19 smithi033 ceph-mon[27845]: pgmap v7527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13099.688902 secs 2024-04-26T15:29:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:19 smithi096 ceph-mon[32753]: pgmap v7527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:21 smithi194 ceph-mon[33017]: pgmap v7528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:21 smithi033 ceph-mon[27845]: pgmap v7528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:21 smithi096 ceph-mon[32753]: pgmap v7528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:24.023+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:29:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:24.023+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:29:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:24.024+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:29:24.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:24.024+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:29:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:23 smithi194 ceph-mon[33017]: pgmap v7529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13101.891408 secs 2024-04-26T15:29:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:23 smithi033 ceph-mon[27845]: pgmap v7529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13101.891408 secs 2024-04-26T15:29:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:23 smithi096 ceph-mon[32753]: pgmap v7529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13101.891408 secs 2024-04-26T15:29:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13104.689037 secs 2024-04-26T15:29:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13104.689037 secs 2024-04-26T15:29:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13104.689037 secs 2024-04-26T15:29:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:25 smithi194 ceph-mon[33017]: pgmap v7530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:25 smithi033 ceph-mon[27845]: pgmap v7530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:25 smithi096 ceph-mon[32753]: pgmap v7530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:27.368+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:27.368+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:27.368+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:29:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:27 smithi194 ceph-mon[33017]: pgmap v7531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:27 smithi033 ceph-mon[27845]: pgmap v7531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:27 smithi096 ceph-mon[32753]: pgmap v7531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:29.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13106.891490 secs 2024-04-26T15:29:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13106.891490 secs 2024-04-26T15:29:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13106.891490 secs 2024-04-26T15:29:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13109.689166 secs 2024-04-26T15:29:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:29 smithi194 ceph-mon[33017]: pgmap v7532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13109.689166 secs 2024-04-26T15:29:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:29 smithi033 ceph-mon[27845]: pgmap v7532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13109.689166 secs 2024-04-26T15:29:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:29 smithi096 ceph-mon[32753]: pgmap v7532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:31 smithi194 ceph-mon[33017]: pgmap v7533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:31 smithi033 ceph-mon[27845]: pgmap v7533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:31 smithi096 ceph-mon[32753]: pgmap v7533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:33 smithi194 ceph-mon[33017]: pgmap v7534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13111.891550 secs 2024-04-26T15:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:33 smithi033 ceph-mon[27845]: pgmap v7534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13111.891550 secs 2024-04-26T15:29:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:33 smithi096 ceph-mon[32753]: pgmap v7534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13111.891550 secs 2024-04-26T15:29:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13114.689247 secs 2024-04-26T15:29:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13114.689247 secs 2024-04-26T15:29:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13114.689247 secs 2024-04-26T15:29:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:35 smithi033 ceph-mon[27845]: pgmap v7535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:35 smithi096 ceph-mon[32753]: pgmap v7535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:35 smithi194 ceph-mon[33017]: pgmap v7535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:36 smithi033 ceph-mon[27845]: pgmap v7536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:36 smithi096 ceph-mon[32753]: pgmap v7536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:36 smithi194 ceph-mon[33017]: pgmap v7536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13116.891632 secs 2024-04-26T15:29:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13116.891632 secs 2024-04-26T15:29:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13116.891632 secs 2024-04-26T15:29:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:38 smithi033 ceph-mon[27845]: pgmap v7537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:38 smithi194 ceph-mon[33017]: pgmap v7537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:38 smithi096 ceph-mon[32753]: pgmap v7537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13119.689359 secs 2024-04-26T15:29:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13119.689359 secs 2024-04-26T15:29:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13119.689359 secs 2024-04-26T15:29:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:40 smithi194 ceph-mon[33017]: pgmap v7538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:40 smithi033 ceph-mon[27845]: pgmap v7538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:40 smithi096 ceph-mon[32753]: pgmap v7538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:42.370+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:42.371+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:42.371+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:29:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:43 smithi194 ceph-mon[33017]: pgmap v7539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13121.891715 secs 2024-04-26T15:29:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:43 smithi033 ceph-mon[27845]: pgmap v7539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13121.891715 secs 2024-04-26T15:29:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:43 smithi096 ceph-mon[32753]: pgmap v7539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13121.891715 secs 2024-04-26T15:29:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13124.689474 secs 2024-04-26T15:29:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13124.689474 secs 2024-04-26T15:29:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13124.689474 secs 2024-04-26T15:29:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:45 smithi033 ceph-mon[27845]: pgmap v7540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:45 smithi096 ceph-mon[32753]: pgmap v7540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:45 smithi194 ceph-mon[33017]: pgmap v7540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:47 smithi033 ceph-mon[27845]: pgmap v7541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:47 smithi096 ceph-mon[32753]: pgmap v7541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:47 smithi194 ceph-mon[33017]: pgmap v7541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13126.891798 secs 2024-04-26T15:29:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13126.891798 secs 2024-04-26T15:29:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13126.891798 secs 2024-04-26T15:29:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13129.689584 secs 2024-04-26T15:29:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:49 smithi033 ceph-mon[27845]: pgmap v7542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13129.689584 secs 2024-04-26T15:29:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:49 smithi096 ceph-mon[32753]: pgmap v7542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13129.689584 secs 2024-04-26T15:29:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:49 smithi194 ceph-mon[33017]: pgmap v7542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:51 smithi033 ceph-mon[27845]: pgmap v7543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:51 smithi096 ceph-mon[32753]: pgmap v7543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:51 smithi194 ceph-mon[33017]: pgmap v7543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:29:53.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:53 smithi194 ceph-mon[33017]: pgmap v7544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:53.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13131.891880 secs 2024-04-26T15:29:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:53 smithi033 ceph-mon[27845]: pgmap v7544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13131.891880 secs 2024-04-26T15:29:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:53 smithi096 ceph-mon[32753]: pgmap v7544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13131.891880 secs 2024-04-26T15:29:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13134.689735 secs 2024-04-26T15:29:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13134.689735 secs 2024-04-26T15:29:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13134.689735 secs 2024-04-26T15:29:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:55 smithi033 ceph-mon[27845]: pgmap v7545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:55 smithi096 ceph-mon[32753]: pgmap v7545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:55 smithi194 ceph-mon[33017]: pgmap v7545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:29:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:57.385+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:57.385+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:29:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:29:57.385+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:29:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:57 smithi033 ceph-mon[27845]: pgmap v7546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:29:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:57 smithi096 ceph-mon[32753]: pgmap v7546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:29:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:57 smithi194 ceph-mon[33017]: pgmap v7546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:29:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:29:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13136.891946 secs 2024-04-26T15:29:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13136.891946 secs 2024-04-26T15:29:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13136.891946 secs 2024-04-26T15:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13139.689843 secs 2024-04-26T15:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:59 smithi033 ceph-mon[27845]: pgmap v7547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:29:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13139.689843 secs 2024-04-26T15:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:59 smithi096 ceph-mon[32753]: pgmap v7547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:29:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13139.689843 secs 2024-04-26T15:29:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:59 smithi194 ceph-mon[33017]: pgmap v7547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:29:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:29:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:29:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:30:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:30:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:01 smithi194 ceph-mon[33017]: pgmap v7548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:30:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:30:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:30:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:30:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:01 smithi033 ceph-mon[27845]: pgmap v7548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:30:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:30:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:30:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:30:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:01 smithi096 ceph-mon[32753]: pgmap v7548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: pgmap v7549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: pgmap v7550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: pgmap v7551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T15:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13141.892035 secs 2024-04-26T15:30:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: pgmap v7549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:30:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:30:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:30:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:30:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: pgmap v7550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:30:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: pgmap v7551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T15:30:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13141.892035 secs 2024-04-26T15:30:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: pgmap v7549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: pgmap v7550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:30:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: pgmap v7551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T15:30:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:30:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13141.892035 secs 2024-04-26T15:30:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13144.689986 secs 2024-04-26T15:30:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13144.689986 secs 2024-04-26T15:30:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13144.689986 secs 2024-04-26T15:30:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:05 smithi194 ceph-mon[33017]: pgmap v7552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:05 smithi033 ceph-mon[27845]: pgmap v7552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:05 smithi096 ceph-mon[32753]: pgmap v7552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:07 smithi033 ceph-mon[27845]: pgmap v7553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:07 smithi096 ceph-mon[32753]: pgmap v7553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:07 smithi194 ceph-mon[33017]: pgmap v7553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13146.892124 secs 2024-04-26T15:30:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13146.892124 secs 2024-04-26T15:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13146.892124 secs 2024-04-26T15:30:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13149.690089 secs 2024-04-26T15:30:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:09 smithi033 ceph-mon[27845]: pgmap v7554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:30:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13149.690089 secs 2024-04-26T15:30:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:09 smithi096 ceph-mon[32753]: pgmap v7554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:30:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13149.690089 secs 2024-04-26T15:30:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:09 smithi194 ceph-mon[33017]: pgmap v7554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:30:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:11 smithi033 ceph-mon[27845]: pgmap v7555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:11 smithi096 ceph-mon[32753]: pgmap v7555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:11 smithi194 ceph-mon[33017]: pgmap v7555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:30:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:12.391+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:12.391+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:12.391+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:30:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:13 smithi033 ceph-mon[27845]: pgmap v7556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:30:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13151.892208 secs 2024-04-26T15:30:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:13 smithi096 ceph-mon[32753]: pgmap v7556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:30:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13151.892208 secs 2024-04-26T15:30:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:13 smithi194 ceph-mon[33017]: pgmap v7556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:30:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13151.892208 secs 2024-04-26T15:30:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13154.690213 secs 2024-04-26T15:30:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13154.690213 secs 2024-04-26T15:30:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13154.690213 secs 2024-04-26T15:30:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:15 smithi033 ceph-mon[27845]: pgmap v7557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:15 smithi096 ceph-mon[32753]: pgmap v7557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:15 smithi194 ceph-mon[33017]: pgmap v7557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:17 smithi194 ceph-mon[33017]: pgmap v7558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:17 smithi033 ceph-mon[27845]: pgmap v7558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:17 smithi096 ceph-mon[32753]: pgmap v7558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13156.892297 secs 2024-04-26T15:30:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13156.892297 secs 2024-04-26T15:30:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13156.892297 secs 2024-04-26T15:30:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13159.690349 secs 2024-04-26T15:30:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:19 smithi194 ceph-mon[33017]: pgmap v7559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13159.690349 secs 2024-04-26T15:30:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:19 smithi033 ceph-mon[27845]: pgmap v7559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13159.690349 secs 2024-04-26T15:30:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:19 smithi096 ceph-mon[32753]: pgmap v7559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:21 smithi194 ceph-mon[33017]: pgmap v7560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:21 smithi033 ceph-mon[27845]: pgmap v7560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:21 smithi096 ceph-mon[32753]: pgmap v7560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:23 smithi194 ceph-mon[33017]: pgmap v7561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13161.892383 secs 2024-04-26T15:30:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:23 smithi033 ceph-mon[27845]: pgmap v7561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13161.892383 secs 2024-04-26T15:30:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:23 smithi096 ceph-mon[32753]: pgmap v7561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13161.892383 secs 2024-04-26T15:30:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13164.690491 secs 2024-04-26T15:30:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13164.690491 secs 2024-04-26T15:30:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13164.690491 secs 2024-04-26T15:30:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:25 smithi194 ceph-mon[33017]: pgmap v7562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:25 smithi033 ceph-mon[27845]: pgmap v7562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:25 smithi096 ceph-mon[32753]: pgmap v7562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:27.398+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:27.398+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:27.398+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:30:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:27 smithi194 ceph-mon[33017]: pgmap v7563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:27 smithi033 ceph-mon[27845]: pgmap v7563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:27 smithi096 ceph-mon[32753]: pgmap v7563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13166.892466 secs 2024-04-26T15:30:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13166.892466 secs 2024-04-26T15:30:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13166.892466 secs 2024-04-26T15:30:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13169.690609 secs 2024-04-26T15:30:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:29 smithi194 ceph-mon[33017]: pgmap v7564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13169.690609 secs 2024-04-26T15:30:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:29 smithi033 ceph-mon[27845]: pgmap v7564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13169.690609 secs 2024-04-26T15:30:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:29 smithi096 ceph-mon[32753]: pgmap v7564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:31 smithi194 ceph-mon[33017]: pgmap v7565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:31 smithi033 ceph-mon[27845]: pgmap v7565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:31 smithi096 ceph-mon[32753]: pgmap v7565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:33 smithi033 ceph-mon[27845]: pgmap v7566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13171.892547 secs 2024-04-26T15:30:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:33 smithi096 ceph-mon[32753]: pgmap v7566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13171.892547 secs 2024-04-26T15:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:33 smithi194 ceph-mon[33017]: pgmap v7566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13171.892547 secs 2024-04-26T15:30:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13174.690730 secs 2024-04-26T15:30:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13174.690730 secs 2024-04-26T15:30:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13174.690730 secs 2024-04-26T15:30:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:35 smithi033 ceph-mon[27845]: pgmap v7567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:35 smithi096 ceph-mon[32753]: pgmap v7567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:35 smithi194 ceph-mon[33017]: pgmap v7567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:36 smithi033 ceph-mon[27845]: pgmap v7568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:36 smithi096 ceph-mon[32753]: pgmap v7568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:36 smithi194 ceph-mon[33017]: pgmap v7568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13176.892632 secs 2024-04-26T15:30:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13176.892632 secs 2024-04-26T15:30:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13176.892632 secs 2024-04-26T15:30:39.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:38 smithi033 ceph-mon[27845]: pgmap v7569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:38 smithi194 ceph-mon[33017]: pgmap v7569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:38 smithi096 ceph-mon[32753]: pgmap v7569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13179.690793 secs 2024-04-26T15:30:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13179.690793 secs 2024-04-26T15:30:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13179.690793 secs 2024-04-26T15:30:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:40 smithi194 ceph-mon[33017]: pgmap v7570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:40 smithi033 ceph-mon[27845]: pgmap v7570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:40 smithi096 ceph-mon[32753]: pgmap v7570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:42.412+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:42.412+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:30:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:42.412+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:43 smithi033 ceph-mon[27845]: pgmap v7571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13181.892721 secs 2024-04-26T15:30:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:43 smithi096 ceph-mon[32753]: pgmap v7571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13181.892721 secs 2024-04-26T15:30:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:43 smithi194 ceph-mon[33017]: pgmap v7571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13181.892721 secs 2024-04-26T15:30:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13184.690910 secs 2024-04-26T15:30:44.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13184.690910 secs 2024-04-26T15:30:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13184.690910 secs 2024-04-26T15:30:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:45 smithi033 ceph-mon[27845]: pgmap v7572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:45 smithi096 ceph-mon[32753]: pgmap v7572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:45 smithi194 ceph-mon[33017]: pgmap v7572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:47 smithi033 ceph-mon[27845]: pgmap v7573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:47 smithi096 ceph-mon[32753]: pgmap v7573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:47 smithi194 ceph-mon[33017]: pgmap v7573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13186.892807 secs 2024-04-26T15:30:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13186.892807 secs 2024-04-26T15:30:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13186.892807 secs 2024-04-26T15:30:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13189.691001 secs 2024-04-26T15:30:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:49 smithi194 ceph-mon[33017]: pgmap v7574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13189.691001 secs 2024-04-26T15:30:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:49 smithi033 ceph-mon[27845]: pgmap v7574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13189.691001 secs 2024-04-26T15:30:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:49 smithi096 ceph-mon[32753]: pgmap v7574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:51 smithi194 ceph-mon[33017]: pgmap v7575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:51 smithi033 ceph-mon[27845]: pgmap v7575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:51 smithi096 ceph-mon[32753]: pgmap v7575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:53 smithi194 ceph-mon[33017]: pgmap v7576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13191.892892 secs 2024-04-26T15:30:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:53 smithi033 ceph-mon[27845]: pgmap v7576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13191.892892 secs 2024-04-26T15:30:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:53 smithi096 ceph-mon[32753]: pgmap v7576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13191.892892 secs 2024-04-26T15:30:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13194.691144 secs 2024-04-26T15:30:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13194.691144 secs 2024-04-26T15:30:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13194.691144 secs 2024-04-26T15:30:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:55 smithi194 ceph-mon[33017]: pgmap v7577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:55 smithi033 ceph-mon[27845]: pgmap v7577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:55 smithi096 ceph-mon[32753]: pgmap v7577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:30:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:57.427+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:57.427+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:30:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:30:57.427+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:30:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:57 smithi194 ceph-mon[33017]: pgmap v7578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:57 smithi033 ceph-mon[27845]: pgmap v7578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:57 smithi096 ceph-mon[32753]: pgmap v7578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:30:59.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13196.892998 secs 2024-04-26T15:30:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13196.892998 secs 2024-04-26T15:30:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13196.892998 secs 2024-04-26T15:31:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13199.691223 secs 2024-04-26T15:31:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:30:59 smithi194 ceph-mon[33017]: pgmap v7579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13199.691223 secs 2024-04-26T15:31:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:30:59 smithi033 ceph-mon[27845]: pgmap v7579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13199.691223 secs 2024-04-26T15:31:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:30:59 smithi096 ceph-mon[32753]: pgmap v7579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:01 smithi194 ceph-mon[33017]: pgmap v7580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:01 smithi033 ceph-mon[27845]: pgmap v7580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:01 smithi096 ceph-mon[32753]: pgmap v7580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:03 smithi194 ceph-mon[33017]: pgmap v7581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13201.893084 secs 2024-04-26T15:31:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:31:04.112 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:03 smithi033 ceph-mon[27845]: pgmap v7581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:04.113 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13201.893084 secs 2024-04-26T15:31:04.113 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:31:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:03 smithi096 ceph-mon[32753]: pgmap v7581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13201.893084 secs 2024-04-26T15:31:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:31:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13204.691316 secs 2024-04-26T15:31:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:31:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:31:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:31:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13204.691316 secs 2024-04-26T15:31:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:31:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:31:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:31:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13204.691316 secs 2024-04-26T15:31:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:31:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:31:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:31:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:05 smithi194 ceph-mon[33017]: pgmap v7582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:05 smithi194 ceph-mon[33017]: pgmap v7583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T15:31:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:05 smithi033 ceph-mon[27845]: pgmap v7582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:05 smithi033 ceph-mon[27845]: pgmap v7583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T15:31:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:05 smithi096 ceph-mon[32753]: pgmap v7582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:05 smithi096 ceph-mon[32753]: pgmap v7583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T15:31:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:07 smithi194 ceph-mon[33017]: pgmap v7584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:07 smithi033 ceph-mon[27845]: pgmap v7584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:07 smithi096 ceph-mon[32753]: pgmap v7584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:09.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13206.893146 secs 2024-04-26T15:31:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13206.893146 secs 2024-04-26T15:31:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13206.893146 secs 2024-04-26T15:31:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13209.691380 secs 2024-04-26T15:31:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:09 smithi194 ceph-mon[33017]: pgmap v7585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13209.691380 secs 2024-04-26T15:31:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:09 smithi033 ceph-mon[27845]: pgmap v7585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13209.691380 secs 2024-04-26T15:31:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:09 smithi096 ceph-mon[32753]: pgmap v7585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:11 smithi194 ceph-mon[33017]: pgmap v7586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:11 smithi033 ceph-mon[27845]: pgmap v7586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:11 smithi096 ceph-mon[32753]: pgmap v7586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:12.428+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:12.428+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:31:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:12.428+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:13 smithi194 ceph-mon[33017]: pgmap v7587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13211.893203 secs 2024-04-26T15:31:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:13 smithi033 ceph-mon[27845]: pgmap v7587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13211.893203 secs 2024-04-26T15:31:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:13 smithi096 ceph-mon[32753]: pgmap v7587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13211.893203 secs 2024-04-26T15:31:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13214.691498 secs 2024-04-26T15:31:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13214.691498 secs 2024-04-26T15:31:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13214.691498 secs 2024-04-26T15:31:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:15 smithi194 ceph-mon[33017]: pgmap v7588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:31:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:15 smithi033 ceph-mon[27845]: pgmap v7588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:31:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:15 smithi096 ceph-mon[32753]: pgmap v7588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:31:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:17 smithi194 ceph-mon[33017]: pgmap v7589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:17 smithi033 ceph-mon[27845]: pgmap v7589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:17 smithi096 ceph-mon[32753]: pgmap v7589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13216.893292 secs 2024-04-26T15:31:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13216.893292 secs 2024-04-26T15:31:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13216.893292 secs 2024-04-26T15:31:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13219.691633 secs 2024-04-26T15:31:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:19 smithi194 ceph-mon[33017]: pgmap v7590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13219.691633 secs 2024-04-26T15:31:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:19 smithi033 ceph-mon[27845]: pgmap v7590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13219.691633 secs 2024-04-26T15:31:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:19 smithi096 ceph-mon[32753]: pgmap v7590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:21 smithi194 ceph-mon[33017]: pgmap v7591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:21 smithi033 ceph-mon[27845]: pgmap v7591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:21 smithi096 ceph-mon[32753]: pgmap v7591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:23 smithi194 ceph-mon[33017]: pgmap v7592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13221.893355 secs 2024-04-26T15:31:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:23 smithi033 ceph-mon[27845]: pgmap v7592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13221.893355 secs 2024-04-26T15:31:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:23 smithi096 ceph-mon[32753]: pgmap v7592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13221.893355 secs 2024-04-26T15:31:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13224.691746 secs 2024-04-26T15:31:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13224.691746 secs 2024-04-26T15:31:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13224.691746 secs 2024-04-26T15:31:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:25 smithi194 ceph-mon[33017]: pgmap v7593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:25 smithi033 ceph-mon[27845]: pgmap v7593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:25 smithi096 ceph-mon[32753]: pgmap v7593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:27.433+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:27.433+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:27.433+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:31:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:27 smithi194 ceph-mon[33017]: pgmap v7594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:27 smithi033 ceph-mon[27845]: pgmap v7594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:27 smithi096 ceph-mon[32753]: pgmap v7594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13226.893417 secs 2024-04-26T15:31:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13226.893417 secs 2024-04-26T15:31:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13226.893417 secs 2024-04-26T15:31:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13229.691865 secs 2024-04-26T15:31:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:29 smithi033 ceph-mon[27845]: pgmap v7595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13229.691865 secs 2024-04-26T15:31:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:29 smithi096 ceph-mon[32753]: pgmap v7595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13229.691865 secs 2024-04-26T15:31:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:29 smithi194 ceph-mon[33017]: pgmap v7595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:31 smithi033 ceph-mon[27845]: pgmap v7596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:31 smithi096 ceph-mon[32753]: pgmap v7596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:31 smithi194 ceph-mon[33017]: pgmap v7596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:33 smithi033 ceph-mon[27845]: pgmap v7597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13231.893503 secs 2024-04-26T15:31:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:33 smithi096 ceph-mon[32753]: pgmap v7597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13231.893503 secs 2024-04-26T15:31:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:33 smithi194 ceph-mon[33017]: pgmap v7597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13231.893503 secs 2024-04-26T15:31:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13234.691985 secs 2024-04-26T15:31:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13234.691985 secs 2024-04-26T15:31:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13234.691985 secs 2024-04-26T15:31:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:35 smithi033 ceph-mon[27845]: pgmap v7598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:35 smithi096 ceph-mon[32753]: pgmap v7598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:36.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:35 smithi194 ceph-mon[33017]: pgmap v7598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:36 smithi033 ceph-mon[27845]: pgmap v7599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:36 smithi096 ceph-mon[32753]: pgmap v7599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:36 smithi194 ceph-mon[33017]: pgmap v7599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13236.893593 secs 2024-04-26T15:31:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13236.893593 secs 2024-04-26T15:31:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13236.893593 secs 2024-04-26T15:31:39.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:38 smithi033 ceph-mon[27845]: pgmap v7600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:38 smithi194 ceph-mon[33017]: pgmap v7600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:38 smithi096 ceph-mon[32753]: pgmap v7600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13239.692123 secs 2024-04-26T15:31:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13239.692123 secs 2024-04-26T15:31:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13239.692123 secs 2024-04-26T15:31:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:40 smithi194 ceph-mon[33017]: pgmap v7601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:40 smithi033 ceph-mon[27845]: pgmap v7601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:40 smithi096 ceph-mon[32753]: pgmap v7601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:31:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:42.444+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:42.444+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:42.444+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:31:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:43 smithi033 ceph-mon[27845]: pgmap v7602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13241.893681 secs 2024-04-26T15:31:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:43 smithi096 ceph-mon[32753]: pgmap v7602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13241.893681 secs 2024-04-26T15:31:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:43 smithi194 ceph-mon[33017]: pgmap v7602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13241.893681 secs 2024-04-26T15:31:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13244.692233 secs 2024-04-26T15:31:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13244.692233 secs 2024-04-26T15:31:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13244.692233 secs 2024-04-26T15:31:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:45 smithi033 ceph-mon[27845]: pgmap v7603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:45 smithi096 ceph-mon[32753]: pgmap v7603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:45 smithi194 ceph-mon[33017]: pgmap v7603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:31:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:47 smithi033 ceph-mon[27845]: pgmap v7604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:47 smithi096 ceph-mon[32753]: pgmap v7604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:47 smithi194 ceph-mon[33017]: pgmap v7604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:31:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13246.893740 secs 2024-04-26T15:31:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13246.893740 secs 2024-04-26T15:31:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13246.893740 secs 2024-04-26T15:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13249.692367 secs 2024-04-26T15:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:49 smithi033 ceph-mon[27845]: pgmap v7605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13249.692367 secs 2024-04-26T15:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:49 smithi096 ceph-mon[32753]: pgmap v7605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13249.692367 secs 2024-04-26T15:31:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:49 smithi194 ceph-mon[33017]: pgmap v7605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:31:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:51 smithi033 ceph-mon[27845]: pgmap v7606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:31:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:51 smithi096 ceph-mon[32753]: pgmap v7606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:31:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:51 smithi194 ceph-mon[33017]: pgmap v7606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:31:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:53 smithi033 ceph-mon[27845]: pgmap v7607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:31:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13251.893823 secs 2024-04-26T15:31:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:53 smithi096 ceph-mon[32753]: pgmap v7607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:31:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13251.893823 secs 2024-04-26T15:31:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:53 smithi194 ceph-mon[33017]: pgmap v7607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:31:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13251.893823 secs 2024-04-26T15:31:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13254.692462 secs 2024-04-26T15:31:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13254.692462 secs 2024-04-26T15:31:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13254.692462 secs 2024-04-26T15:31:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:55 smithi194 ceph-mon[33017]: pgmap v7608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:31:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:55 smithi033 ceph-mon[27845]: pgmap v7608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:31:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:55 smithi096 ceph-mon[32753]: pgmap v7608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:31:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:57.453+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:57.453+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:31:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:31:57.453+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:31:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:57 smithi194 ceph-mon[33017]: pgmap v7609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:31:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:57 smithi033 ceph-mon[27845]: pgmap v7609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:31:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:57 smithi096 ceph-mon[32753]: pgmap v7609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:31:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13256.893910 secs 2024-04-26T15:31:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13256.893910 secs 2024-04-26T15:31:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13256.893910 secs 2024-04-26T15:31:59.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13259.692589 secs 2024-04-26T15:31:59.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:31:59 smithi096 ceph-mon[32753]: pgmap v7610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:31:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13259.692589 secs 2024-04-26T15:31:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:31:59 smithi194 ceph-mon[33017]: pgmap v7610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:31:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13259.692589 secs 2024-04-26T15:31:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:31:59 smithi033 ceph-mon[27845]: pgmap v7610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:32:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:01 smithi194 ceph-mon[33017]: pgmap v7611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:32:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:01 smithi033 ceph-mon[27845]: pgmap v7611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:32:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:01 smithi096 ceph-mon[32753]: pgmap v7611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:32:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:03 smithi194 ceph-mon[33017]: pgmap v7612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13261.893992 secs 2024-04-26T15:32:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:03 smithi033 ceph-mon[27845]: pgmap v7612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13261.893992 secs 2024-04-26T15:32:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:03 smithi096 ceph-mon[32753]: pgmap v7612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13261.893992 secs 2024-04-26T15:32:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13264.692685 secs 2024-04-26T15:32:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13264.692685 secs 2024-04-26T15:32:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13264.692685 secs 2024-04-26T15:32:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:05 smithi194 ceph-mon[33017]: pgmap v7613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:32:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:32:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:05 smithi033 ceph-mon[27845]: pgmap v7613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:32:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:32:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:05 smithi096 ceph-mon[32753]: pgmap v7613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:32:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: pgmap v7614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:32:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:32:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: pgmap v7614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:32:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: pgmap v7614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:32:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:32:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:32:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:32:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:08 smithi194 ceph-mon[33017]: pgmap v7615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:32:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:08 smithi194 ceph-mon[33017]: pgmap v7616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13266.894056 secs 2024-04-26T15:32:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:08 smithi033 ceph-mon[27845]: pgmap v7615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:32:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:08 smithi033 ceph-mon[27845]: pgmap v7616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13266.894056 secs 2024-04-26T15:32:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:08 smithi096 ceph-mon[32753]: pgmap v7615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:32:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:08 smithi096 ceph-mon[32753]: pgmap v7616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13266.894056 secs 2024-04-26T15:32:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13269.692821 secs 2024-04-26T15:32:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13269.692821 secs 2024-04-26T15:32:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13269.692821 secs 2024-04-26T15:32:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:10 smithi033 ceph-mon[27845]: pgmap v7617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:32:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:10 smithi096 ceph-mon[32753]: pgmap v7617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:32:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:10 smithi194 ceph-mon[33017]: pgmap v7617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:32:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:12 smithi033 ceph-mon[27845]: pgmap v7618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:12 smithi096 ceph-mon[32753]: pgmap v7618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:12.465+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:12.465+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:12.465+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:32:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:12 smithi194 ceph-mon[33017]: pgmap v7618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13271.894116 secs 2024-04-26T15:32:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13271.894116 secs 2024-04-26T15:32:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13271.894116 secs 2024-04-26T15:32:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:14 smithi033 ceph-mon[27845]: pgmap v7619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T15:32:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13274.692955 secs 2024-04-26T15:32:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:14 smithi096 ceph-mon[32753]: pgmap v7619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T15:32:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13274.692955 secs 2024-04-26T15:32:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:14 smithi194 ceph-mon[33017]: pgmap v7619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T15:32:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13274.692955 secs 2024-04-26T15:32:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:16 smithi033 ceph-mon[27845]: pgmap v7620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:16 smithi096 ceph-mon[32753]: pgmap v7620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:16 smithi194 ceph-mon[33017]: pgmap v7620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:32:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:18 smithi033 ceph-mon[27845]: pgmap v7621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:32:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13276.894178 secs 2024-04-26T15:32:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:18 smithi096 ceph-mon[32753]: pgmap v7621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:32:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13276.894178 secs 2024-04-26T15:32:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:18 smithi194 ceph-mon[33017]: pgmap v7621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:32:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13276.894178 secs 2024-04-26T15:32:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13279.693088 secs 2024-04-26T15:32:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13279.693088 secs 2024-04-26T15:32:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13279.693088 secs 2024-04-26T15:32:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:20 smithi033 ceph-mon[27845]: pgmap v7622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:20.552 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:20 smithi096 ceph-mon[32753]: pgmap v7622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:20 smithi194 ceph-mon[33017]: pgmap v7622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:22 smithi033 ceph-mon[27845]: pgmap v7623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:22 smithi096 ceph-mon[32753]: pgmap v7623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:22 smithi194 ceph-mon[33017]: pgmap v7623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13281.894239 secs 2024-04-26T15:32:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13281.894239 secs 2024-04-26T15:32:23.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13281.894239 secs 2024-04-26T15:32:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.053+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:32:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.053+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T15:32:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.054+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T15:32:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.054+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x564385821900 2024-04-26T15:32:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.054+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T15:32:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.054+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:32:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.054+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:32:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:24.055+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:32:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:24 smithi033 ceph-mon[27845]: pgmap v7624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13284.693199 secs 2024-04-26T15:32:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:24 smithi096 ceph-mon[32753]: pgmap v7624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13284.693199 secs 2024-04-26T15:32:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:24 smithi194 ceph-mon[33017]: pgmap v7624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13284.693199 secs 2024-04-26T15:32:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:26 smithi096 ceph-mon[32753]: pgmap v7625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:26 smithi194 ceph-mon[33017]: pgmap v7625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:26 smithi033 ceph-mon[27845]: pgmap v7625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:27.471+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:27.471+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:32:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:27.471+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:28 smithi194 ceph-mon[33017]: pgmap v7626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13286.894300 secs 2024-04-26T15:32:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:28 smithi033 ceph-mon[27845]: pgmap v7626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13286.894300 secs 2024-04-26T15:32:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:28 smithi096 ceph-mon[32753]: pgmap v7626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13286.894300 secs 2024-04-26T15:32:29.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13289.693319 secs 2024-04-26T15:32:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13289.693319 secs 2024-04-26T15:32:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13289.693319 secs 2024-04-26T15:32:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:30 smithi194 ceph-mon[33017]: pgmap v7627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:30 smithi033 ceph-mon[27845]: pgmap v7627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:30 smithi096 ceph-mon[32753]: pgmap v7627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:32 smithi194 ceph-mon[33017]: pgmap v7628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:32 smithi033 ceph-mon[27845]: pgmap v7628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:32 smithi096 ceph-mon[32753]: pgmap v7628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13291.894384 secs 2024-04-26T15:32:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13291.894384 secs 2024-04-26T15:32:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13291.894384 secs 2024-04-26T15:32:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:34 smithi194 ceph-mon[33017]: pgmap v7629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13294.693440 secs 2024-04-26T15:32:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:34 smithi033 ceph-mon[27845]: pgmap v7629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13294.693440 secs 2024-04-26T15:32:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:34 smithi096 ceph-mon[32753]: pgmap v7629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13294.693440 secs 2024-04-26T15:32:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:36 smithi194 ceph-mon[33017]: pgmap v7630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:36 smithi033 ceph-mon[27845]: pgmap v7630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:36 smithi096 ceph-mon[32753]: pgmap v7630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:38.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:38 smithi033 ceph-mon[27845]: pgmap v7631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:38.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13296.894468 secs 2024-04-26T15:32:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:38 smithi194 ceph-mon[33017]: pgmap v7631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13296.894468 secs 2024-04-26T15:32:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:38 smithi096 ceph-mon[32753]: pgmap v7631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13296.894468 secs 2024-04-26T15:32:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13299.693539 secs 2024-04-26T15:32:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13299.693539 secs 2024-04-26T15:32:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13299.693539 secs 2024-04-26T15:32:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:40 smithi194 ceph-mon[33017]: pgmap v7632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:40 smithi033 ceph-mon[27845]: pgmap v7632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:40 smithi096 ceph-mon[32753]: pgmap v7632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:42.472+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:42.473+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:42.473+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:32:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:42 smithi194 ceph-mon[33017]: pgmap v7633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:42 smithi033 ceph-mon[27845]: pgmap v7633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:42 smithi096 ceph-mon[32753]: pgmap v7633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13301.894552 secs 2024-04-26T15:32:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13301.894552 secs 2024-04-26T15:32:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13301.894552 secs 2024-04-26T15:32:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:44 smithi194 ceph-mon[33017]: pgmap v7634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13304.693605 secs 2024-04-26T15:32:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:44 smithi033 ceph-mon[27845]: pgmap v7634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13304.693605 secs 2024-04-26T15:32:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:44 smithi096 ceph-mon[32753]: pgmap v7634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13304.693605 secs 2024-04-26T15:32:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:46 smithi194 ceph-mon[33017]: pgmap v7635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:46 smithi033 ceph-mon[27845]: pgmap v7635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:46 smithi096 ceph-mon[32753]: pgmap v7635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:48 smithi194 ceph-mon[33017]: pgmap v7636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13306.894634 secs 2024-04-26T15:32:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:48 smithi033 ceph-mon[27845]: pgmap v7636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13306.894634 secs 2024-04-26T15:32:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:48 smithi096 ceph-mon[32753]: pgmap v7636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13306.894634 secs 2024-04-26T15:32:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13309.693721 secs 2024-04-26T15:32:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13309.693721 secs 2024-04-26T15:32:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13309.693721 secs 2024-04-26T15:32:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:50 smithi194 ceph-mon[33017]: pgmap v7637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:50 smithi033 ceph-mon[27845]: pgmap v7637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:50 smithi096 ceph-mon[32753]: pgmap v7637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:52 smithi194 ceph-mon[33017]: pgmap v7638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:52 smithi033 ceph-mon[27845]: pgmap v7638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:52 smithi096 ceph-mon[32753]: pgmap v7638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13311.894693 secs 2024-04-26T15:32:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13311.894693 secs 2024-04-26T15:32:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13311.894693 secs 2024-04-26T15:32:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:54 smithi194 ceph-mon[33017]: pgmap v7639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13314.693842 secs 2024-04-26T15:32:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:54 smithi033 ceph-mon[27845]: pgmap v7639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13314.693842 secs 2024-04-26T15:32:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:54 smithi096 ceph-mon[32753]: pgmap v7639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13314.693842 secs 2024-04-26T15:32:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:56 smithi194 ceph-mon[33017]: pgmap v7640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:56 smithi033 ceph-mon[27845]: pgmap v7640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:56 smithi096 ceph-mon[32753]: pgmap v7640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:32:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:57.487+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:57.487+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:32:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:32:57.487+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:32:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:58 smithi194 ceph-mon[33017]: pgmap v7641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13316.894775 secs 2024-04-26T15:32:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:58 smithi033 ceph-mon[27845]: pgmap v7641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13316.894775 secs 2024-04-26T15:32:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:58 smithi096 ceph-mon[32753]: pgmap v7641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:32:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13316.894775 secs 2024-04-26T15:32:59.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13319.693963 secs 2024-04-26T15:32:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13319.693963 secs 2024-04-26T15:32:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13319.693963 secs 2024-04-26T15:33:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:00 smithi194 ceph-mon[33017]: pgmap v7642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:00 smithi033 ceph-mon[27845]: pgmap v7642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:00 smithi096 ceph-mon[32753]: pgmap v7642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:02 smithi194 ceph-mon[33017]: pgmap v7643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:02 smithi033 ceph-mon[27845]: pgmap v7643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:02 smithi096 ceph-mon[32753]: pgmap v7643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13321.894859 secs 2024-04-26T15:33:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13321.894859 secs 2024-04-26T15:33:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13321.894859 secs 2024-04-26T15:33:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:04 smithi194 ceph-mon[33017]: pgmap v7644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13324.694081 secs 2024-04-26T15:33:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:04 smithi033 ceph-mon[27845]: pgmap v7644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13324.694081 secs 2024-04-26T15:33:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:04 smithi096 ceph-mon[32753]: pgmap v7644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13324.694081 secs 2024-04-26T15:33:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:06 smithi194 ceph-mon[33017]: pgmap v7645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:06 smithi033 ceph-mon[27845]: pgmap v7645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:06 smithi096 ceph-mon[32753]: pgmap v7645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:33:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:33:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:33:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:08 smithi033 ceph-mon[27845]: pgmap v7646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13326.894922 secs 2024-04-26T15:33:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:08 smithi096 ceph-mon[32753]: pgmap v7646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13326.894922 secs 2024-04-26T15:33:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:08 smithi194 ceph-mon[33017]: pgmap v7646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13326.894922 secs 2024-04-26T15:33:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13329.694155 secs 2024-04-26T15:33:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:33:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:33:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:09 smithi033 ceph-mon[27845]: pgmap v7647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:09 smithi033 ceph-mon[27845]: pgmap v7648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:33:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13329.694155 secs 2024-04-26T15:33:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:33:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:33:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:09 smithi096 ceph-mon[32753]: pgmap v7647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:09 smithi096 ceph-mon[32753]: pgmap v7648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:33:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13329.694155 secs 2024-04-26T15:33:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:33:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:33:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:09 smithi194 ceph-mon[33017]: pgmap v7647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:09 smithi194 ceph-mon[33017]: pgmap v7648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:33:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:11 smithi033 ceph-mon[27845]: pgmap v7649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:11 smithi096 ceph-mon[32753]: pgmap v7649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:11 smithi194 ceph-mon[33017]: pgmap v7649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:12.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:12.496+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:12.496+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:12.496+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:33:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:13 smithi096 ceph-mon[32753]: pgmap v7650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13331.895004 secs 2024-04-26T15:33:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:13 smithi194 ceph-mon[33017]: pgmap v7650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13331.895004 secs 2024-04-26T15:33:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:13 smithi033 ceph-mon[27845]: pgmap v7650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13331.895004 secs 2024-04-26T15:33:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13334.694275 secs 2024-04-26T15:33:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13334.694275 secs 2024-04-26T15:33:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13334.694275 secs 2024-04-26T15:33:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:15 smithi194 ceph-mon[33017]: pgmap v7651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:15 smithi033 ceph-mon[27845]: pgmap v7651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:15 smithi096 ceph-mon[32753]: pgmap v7651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:17 smithi033 ceph-mon[27845]: pgmap v7652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:17 smithi096 ceph-mon[32753]: pgmap v7652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:17 smithi194 ceph-mon[33017]: pgmap v7652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:33:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13336.895065 secs 2024-04-26T15:33:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13336.895065 secs 2024-04-26T15:33:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13336.895065 secs 2024-04-26T15:33:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13339.694398 secs 2024-04-26T15:33:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:19 smithi033 ceph-mon[27845]: pgmap v7653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:33:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13339.694398 secs 2024-04-26T15:33:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:19 smithi096 ceph-mon[32753]: pgmap v7653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:33:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13339.694398 secs 2024-04-26T15:33:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:19 smithi194 ceph-mon[33017]: pgmap v7653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:33:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:21 smithi033 ceph-mon[27845]: pgmap v7654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:21 smithi096 ceph-mon[32753]: pgmap v7654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:21 smithi194 ceph-mon[33017]: pgmap v7654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:24.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:23 smithi194 ceph-mon[33017]: pgmap v7655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:24.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13341.895127 secs 2024-04-26T15:33:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:23 smithi033 ceph-mon[27845]: pgmap v7655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13341.895127 secs 2024-04-26T15:33:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:23 smithi096 ceph-mon[32753]: pgmap v7655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13341.895127 secs 2024-04-26T15:33:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13344.694486 secs 2024-04-26T15:33:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13344.694486 secs 2024-04-26T15:33:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13344.694486 secs 2024-04-26T15:33:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:25 smithi033 ceph-mon[27845]: pgmap v7656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:25 smithi096 ceph-mon[32753]: pgmap v7656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:25 smithi194 ceph-mon[33017]: pgmap v7656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:27.507+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:27.507+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:27.507+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:33:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:27 smithi033 ceph-mon[27845]: pgmap v7657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:27 smithi096 ceph-mon[32753]: pgmap v7657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:27 smithi194 ceph-mon[33017]: pgmap v7657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13346.895189 secs 2024-04-26T15:33:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13346.895189 secs 2024-04-26T15:33:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13346.895189 secs 2024-04-26T15:33:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13349.694598 secs 2024-04-26T15:33:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:29 smithi033 ceph-mon[27845]: pgmap v7658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13349.694598 secs 2024-04-26T15:33:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:29 smithi096 ceph-mon[32753]: pgmap v7658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13349.694598 secs 2024-04-26T15:33:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:29 smithi194 ceph-mon[33017]: pgmap v7658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:31 smithi033 ceph-mon[27845]: pgmap v7659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:31 smithi096 ceph-mon[32753]: pgmap v7659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:31 smithi194 ceph-mon[33017]: pgmap v7659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:33 smithi033 ceph-mon[27845]: pgmap v7660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13351.895271 secs 2024-04-26T15:33:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:33 smithi096 ceph-mon[32753]: pgmap v7660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13351.895271 secs 2024-04-26T15:33:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:33 smithi194 ceph-mon[33017]: pgmap v7660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13351.895271 secs 2024-04-26T15:33:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13354.694731 secs 2024-04-26T15:33:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13354.694731 secs 2024-04-26T15:33:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13354.694731 secs 2024-04-26T15:33:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:35 smithi033 ceph-mon[27845]: pgmap v7661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:35 smithi096 ceph-mon[32753]: pgmap v7661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:35 smithi194 ceph-mon[33017]: pgmap v7661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:36 smithi033 ceph-mon[27845]: pgmap v7662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:36 smithi096 ceph-mon[32753]: pgmap v7662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:36 smithi194 ceph-mon[33017]: pgmap v7662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13356.895334 secs 2024-04-26T15:33:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13356.895334 secs 2024-04-26T15:33:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13356.895334 secs 2024-04-26T15:33:39.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:38 smithi033 ceph-mon[27845]: pgmap v7663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:38 smithi194 ceph-mon[33017]: pgmap v7663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:38 smithi096 ceph-mon[32753]: pgmap v7663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:40.271 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13359.694866 secs 2024-04-26T15:33:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13359.694866 secs 2024-04-26T15:33:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13359.694866 secs 2024-04-26T15:33:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:40 smithi194 ceph-mon[33017]: pgmap v7664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:40 smithi033 ceph-mon[27845]: pgmap v7664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:40 smithi096 ceph-mon[32753]: pgmap v7664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:42.520+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:42.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:42.520+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:42.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:42.520+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:33:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:43 smithi194 ceph-mon[33017]: pgmap v7665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13361.895395 secs 2024-04-26T15:33:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:43 smithi033 ceph-mon[27845]: pgmap v7665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13361.895395 secs 2024-04-26T15:33:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:43 smithi096 ceph-mon[32753]: pgmap v7665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13361.895395 secs 2024-04-26T15:33:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13364.694976 secs 2024-04-26T15:33:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13364.694976 secs 2024-04-26T15:33:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13364.694976 secs 2024-04-26T15:33:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:45 smithi194 ceph-mon[33017]: pgmap v7666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:45 smithi033 ceph-mon[27845]: pgmap v7666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:45 smithi096 ceph-mon[32753]: pgmap v7666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:47 smithi194 ceph-mon[33017]: pgmap v7667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:47 smithi033 ceph-mon[27845]: pgmap v7667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:47 smithi096 ceph-mon[32753]: pgmap v7667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13366.895455 secs 2024-04-26T15:33:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13366.895455 secs 2024-04-26T15:33:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13366.895455 secs 2024-04-26T15:33:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13369.695078 secs 2024-04-26T15:33:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:49 smithi194 ceph-mon[33017]: pgmap v7668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13369.695078 secs 2024-04-26T15:33:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:49 smithi033 ceph-mon[27845]: pgmap v7668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13369.695078 secs 2024-04-26T15:33:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:49 smithi096 ceph-mon[32753]: pgmap v7668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:51 smithi194 ceph-mon[33017]: pgmap v7669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:51 smithi033 ceph-mon[27845]: pgmap v7669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:51 smithi096 ceph-mon[32753]: pgmap v7669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:53 smithi194 ceph-mon[33017]: pgmap v7670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13371.895539 secs 2024-04-26T15:33:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:53 smithi033 ceph-mon[27845]: pgmap v7670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13371.895539 secs 2024-04-26T15:33:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:53 smithi096 ceph-mon[32753]: pgmap v7670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13371.895539 secs 2024-04-26T15:33:55.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13374.695181 secs 2024-04-26T15:33:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13374.695181 secs 2024-04-26T15:33:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13374.695181 secs 2024-04-26T15:33:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:55 smithi194 ceph-mon[33017]: pgmap v7671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:56.142 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:55 smithi033 ceph-mon[27845]: pgmap v7671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:55 smithi096 ceph-mon[32753]: pgmap v7671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:33:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:57.535+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:57.535+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:33:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:33:57.535+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:33:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:57 smithi194 ceph-mon[33017]: pgmap v7672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:57 smithi033 ceph-mon[27845]: pgmap v7672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:57 smithi096 ceph-mon[32753]: pgmap v7672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:33:59.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13376.895624 secs 2024-04-26T15:33:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13376.895624 secs 2024-04-26T15:33:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13376.895624 secs 2024-04-26T15:34:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13379.695291 secs 2024-04-26T15:34:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:33:59 smithi194 ceph-mon[33017]: pgmap v7673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13379.695291 secs 2024-04-26T15:34:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:33:59 smithi033 ceph-mon[27845]: pgmap v7673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13379.695291 secs 2024-04-26T15:34:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:33:59 smithi096 ceph-mon[32753]: pgmap v7673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:01 smithi194 ceph-mon[33017]: pgmap v7674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:01 smithi033 ceph-mon[27845]: pgmap v7674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:01 smithi096 ceph-mon[32753]: pgmap v7674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:03 smithi194 ceph-mon[33017]: pgmap v7675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13381.895706 secs 2024-04-26T15:34:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:03 smithi033 ceph-mon[27845]: pgmap v7675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13381.895706 secs 2024-04-26T15:34:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:03 smithi096 ceph-mon[32753]: pgmap v7675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13381.895706 secs 2024-04-26T15:34:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13384.695397 secs 2024-04-26T15:34:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13384.695397 secs 2024-04-26T15:34:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13384.695397 secs 2024-04-26T15:34:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:05 smithi194 ceph-mon[33017]: pgmap v7676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:05 smithi033 ceph-mon[27845]: pgmap v7676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:05 smithi096 ceph-mon[32753]: pgmap v7676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:07 smithi194 ceph-mon[33017]: pgmap v7677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:07 smithi033 ceph-mon[27845]: pgmap v7677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:07 smithi096 ceph-mon[32753]: pgmap v7677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:09.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13386.895791 secs 2024-04-26T15:34:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13386.895791 secs 2024-04-26T15:34:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13386.895791 secs 2024-04-26T15:34:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13389.695519 secs 2024-04-26T15:34:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:09 smithi194 ceph-mon[33017]: pgmap v7678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:34:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13389.695519 secs 2024-04-26T15:34:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:09 smithi033 ceph-mon[27845]: pgmap v7678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:34:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13389.695519 secs 2024-04-26T15:34:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:09 smithi096 ceph-mon[32753]: pgmap v7678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:34:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:34:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:34:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:34:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:34:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:34:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:34:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:34:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:34:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:34:12.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:11 smithi033 ceph-mon[27845]: pgmap v7679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:34:12.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:11 smithi033 ceph-mon[27845]: pgmap v7680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:11 smithi194 ceph-mon[33017]: pgmap v7679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:34:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:11 smithi194 ceph-mon[33017]: pgmap v7680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:11 smithi096 ceph-mon[32753]: pgmap v7679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:34:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:11 smithi096 ceph-mon[32753]: pgmap v7680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:12.538+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:12.538+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:34:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:12.538+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:13 smithi194 ceph-mon[33017]: pgmap v7681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13391.895876 secs 2024-04-26T15:34:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:13 smithi033 ceph-mon[27845]: pgmap v7681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13391.895876 secs 2024-04-26T15:34:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:13 smithi096 ceph-mon[32753]: pgmap v7681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13391.895876 secs 2024-04-26T15:34:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13394.695626 secs 2024-04-26T15:34:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13394.695626 secs 2024-04-26T15:34:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13394.695626 secs 2024-04-26T15:34:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:15 smithi194 ceph-mon[33017]: pgmap v7682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T15:34:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:15 smithi033 ceph-mon[27845]: pgmap v7682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T15:34:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:15 smithi096 ceph-mon[32753]: pgmap v7682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T15:34:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:17 smithi194 ceph-mon[33017]: pgmap v7683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:17 smithi033 ceph-mon[27845]: pgmap v7683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:17 smithi096 ceph-mon[32753]: pgmap v7683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13396.895961 secs 2024-04-26T15:34:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13396.895961 secs 2024-04-26T15:34:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13396.895961 secs 2024-04-26T15:34:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13399.695762 secs 2024-04-26T15:34:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:19 smithi033 ceph-mon[27845]: pgmap v7684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13399.695762 secs 2024-04-26T15:34:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:19 smithi096 ceph-mon[32753]: pgmap v7684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13399.695762 secs 2024-04-26T15:34:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:19 smithi194 ceph-mon[33017]: pgmap v7684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:21 smithi033 ceph-mon[27845]: pgmap v7685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:34:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:21 smithi096 ceph-mon[32753]: pgmap v7685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:34:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:21 smithi194 ceph-mon[33017]: pgmap v7685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:34:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:23 smithi033 ceph-mon[27845]: pgmap v7686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13401.896008 secs 2024-04-26T15:34:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:23 smithi096 ceph-mon[32753]: pgmap v7686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13401.896008 secs 2024-04-26T15:34:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:23 smithi194 ceph-mon[33017]: pgmap v7686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13401.896008 secs 2024-04-26T15:34:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13404.695877 secs 2024-04-26T15:34:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13404.695877 secs 2024-04-26T15:34:25.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13404.695877 secs 2024-04-26T15:34:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:25 smithi033 ceph-mon[27845]: pgmap v7687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:25 smithi096 ceph-mon[32753]: pgmap v7687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:25 smithi194 ceph-mon[33017]: pgmap v7687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:27.541+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:27.542+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:27.542+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:34:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:27 smithi033 ceph-mon[27845]: pgmap v7688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:27 smithi096 ceph-mon[32753]: pgmap v7688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:27 smithi194 ceph-mon[33017]: pgmap v7688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13406.896099 secs 2024-04-26T15:34:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13406.896099 secs 2024-04-26T15:34:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13406.896099 secs 2024-04-26T15:34:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13409.696012 secs 2024-04-26T15:34:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:29 smithi033 ceph-mon[27845]: pgmap v7689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13409.696012 secs 2024-04-26T15:34:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:29 smithi096 ceph-mon[32753]: pgmap v7689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13409.696012 secs 2024-04-26T15:34:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:29 smithi194 ceph-mon[33017]: pgmap v7689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:31 smithi033 ceph-mon[27845]: pgmap v7690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:31 smithi096 ceph-mon[32753]: pgmap v7690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:31 smithi194 ceph-mon[33017]: pgmap v7690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:33 smithi033 ceph-mon[27845]: pgmap v7691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13411.896190 secs 2024-04-26T15:34:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:33 smithi096 ceph-mon[32753]: pgmap v7691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13411.896190 secs 2024-04-26T15:34:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:33 smithi194 ceph-mon[33017]: pgmap v7691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13411.896190 secs 2024-04-26T15:34:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13414.696174 secs 2024-04-26T15:34:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13414.696174 secs 2024-04-26T15:34:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13414.696174 secs 2024-04-26T15:34:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:35 smithi033 ceph-mon[27845]: pgmap v7692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:35 smithi096 ceph-mon[32753]: pgmap v7692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:35 smithi194 ceph-mon[33017]: pgmap v7692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:36 smithi096 ceph-mon[32753]: pgmap v7693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:36 smithi194 ceph-mon[33017]: pgmap v7693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:36 smithi033 ceph-mon[27845]: pgmap v7693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13416.896274 secs 2024-04-26T15:34:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13416.896274 secs 2024-04-26T15:34:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13416.896274 secs 2024-04-26T15:34:39.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:38 smithi033 ceph-mon[27845]: pgmap v7694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:38 smithi194 ceph-mon[33017]: pgmap v7694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:38 smithi096 ceph-mon[32753]: pgmap v7694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13419.696285 secs 2024-04-26T15:34:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13419.696285 secs 2024-04-26T15:34:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13419.696285 secs 2024-04-26T15:34:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:40 smithi194 ceph-mon[33017]: pgmap v7695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:40 smithi033 ceph-mon[27845]: pgmap v7695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:40 smithi096 ceph-mon[32753]: pgmap v7695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:42.555+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:42.555+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:42.555+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:34:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:43 smithi096 ceph-mon[32753]: pgmap v7696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13421.896361 secs 2024-04-26T15:34:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:43 smithi194 ceph-mon[33017]: pgmap v7696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13421.896361 secs 2024-04-26T15:34:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:43 smithi033 ceph-mon[27845]: pgmap v7696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13421.896361 secs 2024-04-26T15:34:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13424.696400 secs 2024-04-26T15:34:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13424.696400 secs 2024-04-26T15:34:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13424.696400 secs 2024-04-26T15:34:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:45 smithi194 ceph-mon[33017]: pgmap v7697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:45 smithi033 ceph-mon[27845]: pgmap v7697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:45 smithi096 ceph-mon[32753]: pgmap v7697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:47 smithi194 ceph-mon[33017]: pgmap v7698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:47 smithi033 ceph-mon[27845]: pgmap v7698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:47 smithi096 ceph-mon[32753]: pgmap v7698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13426.896447 secs 2024-04-26T15:34:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13426.896447 secs 2024-04-26T15:34:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13426.896447 secs 2024-04-26T15:34:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13429.696479 secs 2024-04-26T15:34:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:49 smithi194 ceph-mon[33017]: pgmap v7699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13429.696479 secs 2024-04-26T15:34:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:49 smithi033 ceph-mon[27845]: pgmap v7699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13429.696479 secs 2024-04-26T15:34:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:49 smithi096 ceph-mon[32753]: pgmap v7699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:51 smithi194 ceph-mon[33017]: pgmap v7700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:51 smithi033 ceph-mon[27845]: pgmap v7700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:51 smithi096 ceph-mon[32753]: pgmap v7700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:34:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:53 smithi194 ceph-mon[33017]: pgmap v7701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13431.896532 secs 2024-04-26T15:34:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:53 smithi033 ceph-mon[27845]: pgmap v7701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13431.896532 secs 2024-04-26T15:34:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:53 smithi096 ceph-mon[32753]: pgmap v7701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13431.896532 secs 2024-04-26T15:34:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13434.696597 secs 2024-04-26T15:34:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13434.696597 secs 2024-04-26T15:34:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13434.696597 secs 2024-04-26T15:34:55.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:55 smithi194 ceph-mon[33017]: pgmap v7702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:55 smithi033 ceph-mon[27845]: pgmap v7702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:55 smithi096 ceph-mon[32753]: pgmap v7702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:34:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:57.570+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:57.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:57.570+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:34:57.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:34:57.570+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:34:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:57 smithi194 ceph-mon[33017]: pgmap v7703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:57 smithi033 ceph-mon[27845]: pgmap v7703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:57 smithi096 ceph-mon[32753]: pgmap v7703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:34:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13436.896621 secs 2024-04-26T15:34:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13436.896621 secs 2024-04-26T15:34:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13436.896621 secs 2024-04-26T15:34:59.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13439.696671 secs 2024-04-26T15:34:59.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:34:59 smithi096 ceph-mon[32753]: pgmap v7704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13439.696671 secs 2024-04-26T15:34:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:34:59 smithi194 ceph-mon[33017]: pgmap v7704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:34:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13439.696671 secs 2024-04-26T15:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:34:59 smithi033 ceph-mon[27845]: pgmap v7704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:01 smithi194 ceph-mon[33017]: pgmap v7705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:01 smithi033 ceph-mon[27845]: pgmap v7705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:01 smithi096 ceph-mon[32753]: pgmap v7705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:03 smithi194 ceph-mon[33017]: pgmap v7706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13441.896708 secs 2024-04-26T15:35:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:03 smithi033 ceph-mon[27845]: pgmap v7706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13441.896708 secs 2024-04-26T15:35:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:03 smithi096 ceph-mon[32753]: pgmap v7706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13441.896708 secs 2024-04-26T15:35:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13444.696783 secs 2024-04-26T15:35:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13444.696783 secs 2024-04-26T15:35:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13444.696783 secs 2024-04-26T15:35:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:05 smithi194 ceph-mon[33017]: pgmap v7707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:05 smithi033 ceph-mon[27845]: pgmap v7707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:05 smithi096 ceph-mon[32753]: pgmap v7707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:07 smithi194 ceph-mon[33017]: pgmap v7708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:35:07.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:07 smithi033 ceph-mon[27845]: pgmap v7708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:35:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:07 smithi096 ceph-mon[32753]: pgmap v7708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:35:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13446.896789 secs 2024-04-26T15:35:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13446.896789 secs 2024-04-26T15:35:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13446.896789 secs 2024-04-26T15:35:09.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13449.696898 secs 2024-04-26T15:35:09.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:09 smithi033 ceph-mon[27845]: pgmap v7709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13449.696898 secs 2024-04-26T15:35:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:09 smithi194 ceph-mon[33017]: pgmap v7709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13449.696898 secs 2024-04-26T15:35:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:09 smithi096 ceph-mon[32753]: pgmap v7709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:11 smithi194 ceph-mon[33017]: pgmap v7710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:35:11.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:11 smithi033 ceph-mon[27845]: pgmap v7710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:35:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:11 smithi096 ceph-mon[32753]: pgmap v7710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:35:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:12.583+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:12.583+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:12.583+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:35:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:35:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:35:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:35:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:35:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:35:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:35:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:35:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:35:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:35:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:13 smithi194 ceph-mon[33017]: pgmap v7711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:13 smithi194 ceph-mon[33017]: pgmap v7712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13451.896876 secs 2024-04-26T15:35:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:13 smithi033 ceph-mon[27845]: pgmap v7711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:13 smithi033 ceph-mon[27845]: pgmap v7712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13451.896876 secs 2024-04-26T15:35:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:13 smithi096 ceph-mon[32753]: pgmap v7711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:13 smithi096 ceph-mon[32753]: pgmap v7712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13451.896876 secs 2024-04-26T15:35:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13454.697012 secs 2024-04-26T15:35:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13454.697012 secs 2024-04-26T15:35:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13454.697012 secs 2024-04-26T15:35:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:15 smithi194 ceph-mon[33017]: pgmap v7713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T15:35:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:15 smithi033 ceph-mon[27845]: pgmap v7713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T15:35:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:15 smithi096 ceph-mon[32753]: pgmap v7713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T15:35:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:17 smithi194 ceph-mon[33017]: pgmap v7714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:17 smithi033 ceph-mon[27845]: pgmap v7714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:17 smithi096 ceph-mon[32753]: pgmap v7714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13456.896970 secs 2024-04-26T15:35:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13456.896970 secs 2024-04-26T15:35:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13456.896970 secs 2024-04-26T15:35:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:19 smithi194 ceph-mon[33017]: pgmap v7715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13459.697074 secs 2024-04-26T15:35:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:19 smithi033 ceph-mon[27845]: pgmap v7715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13459.697074 secs 2024-04-26T15:35:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:19 smithi096 ceph-mon[32753]: pgmap v7715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13459.697074 secs 2024-04-26T15:35:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:21 smithi194 ceph-mon[33017]: pgmap v7716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:21 smithi033 ceph-mon[27845]: pgmap v7716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:21 smithi096 ceph-mon[32753]: pgmap v7716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:35:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:23 smithi033 ceph-mon[27845]: pgmap v7717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:35:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13461.897028 secs 2024-04-26T15:35:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:23 smithi096 ceph-mon[32753]: pgmap v7717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:35:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13461.897028 secs 2024-04-26T15:35:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:24.084+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:35:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:24.084+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T15:35:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:24.085+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:35:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:24.085+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0020 con 0x7fef00077780 2024-04-26T15:35:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:23 smithi194 ceph-mon[33017]: pgmap v7717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:35:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13461.897028 secs 2024-04-26T15:35:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13464.697164 secs 2024-04-26T15:35:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13464.697164 secs 2024-04-26T15:35:25.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13464.697164 secs 2024-04-26T15:35:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:25 smithi033 ceph-mon[27845]: pgmap v7718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:25 smithi096 ceph-mon[32753]: pgmap v7718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:25 smithi194 ceph-mon[33017]: pgmap v7718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:27.593+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:27.593+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:27.593+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:35:27.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:27 smithi033 ceph-mon[27845]: pgmap v7719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:27 smithi096 ceph-mon[32753]: pgmap v7719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:27 smithi194 ceph-mon[33017]: pgmap v7719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13466.897118 secs 2024-04-26T15:35:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13466.897118 secs 2024-04-26T15:35:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13466.897118 secs 2024-04-26T15:35:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:29 smithi033 ceph-mon[27845]: pgmap v7720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13469.697280 secs 2024-04-26T15:35:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:29 smithi096 ceph-mon[32753]: pgmap v7720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13469.697280 secs 2024-04-26T15:35:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:29 smithi194 ceph-mon[33017]: pgmap v7720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13469.697280 secs 2024-04-26T15:35:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:31 smithi033 ceph-mon[27845]: pgmap v7721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:31 smithi096 ceph-mon[32753]: pgmap v7721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:31 smithi194 ceph-mon[33017]: pgmap v7721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:33 smithi033 ceph-mon[27845]: pgmap v7722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13471.897180 secs 2024-04-26T15:35:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:33 smithi096 ceph-mon[32753]: pgmap v7722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13471.897180 secs 2024-04-26T15:35:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:33 smithi194 ceph-mon[33017]: pgmap v7722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13471.897180 secs 2024-04-26T15:35:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13474.697353 secs 2024-04-26T15:35:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13474.697353 secs 2024-04-26T15:35:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13474.697353 secs 2024-04-26T15:35:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:35 smithi033 ceph-mon[27845]: pgmap v7723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:35 smithi096 ceph-mon[32753]: pgmap v7723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:35 smithi194 ceph-mon[33017]: pgmap v7723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:37 smithi033 ceph-mon[27845]: pgmap v7724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:37 smithi096 ceph-mon[32753]: pgmap v7724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:37 smithi194 ceph-mon[33017]: pgmap v7724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13476.897266 secs 2024-04-26T15:35:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13476.897266 secs 2024-04-26T15:35:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13476.897266 secs 2024-04-26T15:35:40.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:39 smithi033 ceph-mon[27845]: pgmap v7725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:40.099 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13479.697476 secs 2024-04-26T15:35:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:39 smithi194 ceph-mon[33017]: pgmap v7725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13479.697476 secs 2024-04-26T15:35:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:39 smithi096 ceph-mon[32753]: pgmap v7725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13479.697476 secs 2024-04-26T15:35:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:41 smithi194 ceph-mon[33017]: pgmap v7726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:41 smithi033 ceph-mon[27845]: pgmap v7726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:41 smithi096 ceph-mon[32753]: pgmap v7726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:42.603+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:42.603+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:42.603 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:42.603+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:35:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:43 smithi194 ceph-mon[33017]: pgmap v7727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13481.897325 secs 2024-04-26T15:35:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:43 smithi033 ceph-mon[27845]: pgmap v7727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13481.897325 secs 2024-04-26T15:35:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:43 smithi096 ceph-mon[32753]: pgmap v7727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13481.897325 secs 2024-04-26T15:35:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13484.697595 secs 2024-04-26T15:35:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13484.697595 secs 2024-04-26T15:35:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13484.697595 secs 2024-04-26T15:35:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:45 smithi194 ceph-mon[33017]: pgmap v7728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:45 smithi033 ceph-mon[27845]: pgmap v7728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:45 smithi096 ceph-mon[32753]: pgmap v7728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:47 smithi194 ceph-mon[33017]: pgmap v7729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:47 smithi033 ceph-mon[27845]: pgmap v7729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:47 smithi096 ceph-mon[32753]: pgmap v7729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13486.897385 secs 2024-04-26T15:35:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13486.897385 secs 2024-04-26T15:35:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13486.897385 secs 2024-04-26T15:35:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:49 smithi194 ceph-mon[33017]: pgmap v7730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13489.697732 secs 2024-04-26T15:35:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:49 smithi033 ceph-mon[27845]: pgmap v7730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13489.697732 secs 2024-04-26T15:35:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:49 smithi096 ceph-mon[32753]: pgmap v7730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13489.697732 secs 2024-04-26T15:35:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:51 smithi194 ceph-mon[33017]: pgmap v7731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:51 smithi033 ceph-mon[27845]: pgmap v7731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:51 smithi096 ceph-mon[32753]: pgmap v7731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:53 smithi194 ceph-mon[33017]: pgmap v7732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13491.897444 secs 2024-04-26T15:35:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:53 smithi033 ceph-mon[27845]: pgmap v7732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13491.897444 secs 2024-04-26T15:35:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:53 smithi096 ceph-mon[32753]: pgmap v7732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13491.897444 secs 2024-04-26T15:35:55.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13494.697847 secs 2024-04-26T15:35:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13494.697847 secs 2024-04-26T15:35:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13494.697847 secs 2024-04-26T15:35:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:55 smithi194 ceph-mon[33017]: pgmap v7733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:55 smithi033 ceph-mon[27845]: pgmap v7733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:55 smithi096 ceph-mon[32753]: pgmap v7733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:35:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:57.605+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:57.605+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:35:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:35:57.605+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:35:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:57 smithi194 ceph-mon[33017]: pgmap v7734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:57 smithi033 ceph-mon[27845]: pgmap v7734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:57 smithi096 ceph-mon[32753]: pgmap v7734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:35:59.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13496.897528 secs 2024-04-26T15:35:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13496.897528 secs 2024-04-26T15:35:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13496.897528 secs 2024-04-26T15:36:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:59 smithi194 ceph-mon[33017]: pgmap v7735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13499.697963 secs 2024-04-26T15:36:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:59 smithi033 ceph-mon[27845]: pgmap v7735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13499.697963 secs 2024-04-26T15:36:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:59 smithi096 ceph-mon[32753]: pgmap v7735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13499.697963 secs 2024-04-26T15:36:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:01 smithi194 ceph-mon[33017]: pgmap v7736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:01 smithi033 ceph-mon[27845]: pgmap v7736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:01 smithi096 ceph-mon[32753]: pgmap v7736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:03 smithi194 ceph-mon[33017]: pgmap v7737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13501.897610 secs 2024-04-26T15:36:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:03 smithi033 ceph-mon[27845]: pgmap v7737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13501.897610 secs 2024-04-26T15:36:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:03 smithi096 ceph-mon[32753]: pgmap v7737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13501.897610 secs 2024-04-26T15:36:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13504.698096 secs 2024-04-26T15:36:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13504.698096 secs 2024-04-26T15:36:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13504.698096 secs 2024-04-26T15:36:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:05 smithi194 ceph-mon[33017]: pgmap v7738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:05 smithi033 ceph-mon[27845]: pgmap v7738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:05 smithi096 ceph-mon[32753]: pgmap v7738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:07 smithi194 ceph-mon[33017]: pgmap v7739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:07 smithi033 ceph-mon[27845]: pgmap v7739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:07 smithi096 ceph-mon[32753]: pgmap v7739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13506.897671 secs 2024-04-26T15:36:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13506.897671 secs 2024-04-26T15:36:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13506.897671 secs 2024-04-26T15:36:10.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:09 smithi033 ceph-mon[27845]: pgmap v7740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:10.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13509.698182 secs 2024-04-26T15:36:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:09 smithi194 ceph-mon[33017]: pgmap v7740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13509.698182 secs 2024-04-26T15:36:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:09 smithi096 ceph-mon[32753]: pgmap v7740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13509.698182 secs 2024-04-26T15:36:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:11 smithi194 ceph-mon[33017]: pgmap v7741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:11 smithi033 ceph-mon[27845]: pgmap v7741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:11 smithi096 ceph-mon[32753]: pgmap v7741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:12.615+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:12.615+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:12.615+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:36:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:36:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:36:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:36:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:13 smithi194 ceph-mon[33017]: pgmap v7742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13511.897743 secs 2024-04-26T15:36:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:36:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:36:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:36:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:13 smithi033 ceph-mon[27845]: pgmap v7742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13511.897743 secs 2024-04-26T15:36:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:36:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:36:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:36:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:13 smithi096 ceph-mon[32753]: pgmap v7742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13511.897743 secs 2024-04-26T15:36:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:36:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:36:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:36:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13514.698238 secs 2024-04-26T15:36:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:14 smithi194 ceph-mon[33017]: pgmap v7743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:14 smithi194 ceph-mon[33017]: pgmap v7744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13514.698238 secs 2024-04-26T15:36:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:14 smithi033 ceph-mon[27845]: pgmap v7743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:14 smithi033 ceph-mon[27845]: pgmap v7744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13514.698238 secs 2024-04-26T15:36:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:14 smithi096 ceph-mon[32753]: pgmap v7743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:14 smithi096 ceph-mon[32753]: pgmap v7744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:16 smithi194 ceph-mon[33017]: pgmap v7745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:16 smithi033 ceph-mon[27845]: pgmap v7745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:16 smithi096 ceph-mon[32753]: pgmap v7745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13516.897825 secs 2024-04-26T15:36:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:18 smithi194 ceph-mon[33017]: pgmap v7746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13516.897825 secs 2024-04-26T15:36:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:18 smithi033 ceph-mon[27845]: pgmap v7746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13516.897825 secs 2024-04-26T15:36:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:18 smithi096 ceph-mon[32753]: pgmap v7746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13519.698314 secs 2024-04-26T15:36:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13519.698314 secs 2024-04-26T15:36:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13519.698314 secs 2024-04-26T15:36:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:20 smithi033 ceph-mon[27845]: pgmap v7747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:20 smithi096 ceph-mon[32753]: pgmap v7747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:20 smithi194 ceph-mon[33017]: pgmap v7747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:22 smithi033 ceph-mon[27845]: pgmap v7748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:22 smithi096 ceph-mon[32753]: pgmap v7748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:22 smithi194 ceph-mon[33017]: pgmap v7748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:36:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13521.897905 secs 2024-04-26T15:36:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13521.897905 secs 2024-04-26T15:36:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13521.897905 secs 2024-04-26T15:36:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13524.698376 secs 2024-04-26T15:36:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:24 smithi033 ceph-mon[27845]: pgmap v7749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:36:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13524.698376 secs 2024-04-26T15:36:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:24 smithi096 ceph-mon[32753]: pgmap v7749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:36:25.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13524.698376 secs 2024-04-26T15:36:25.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:24 smithi194 ceph-mon[33017]: pgmap v7749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:36:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:26 smithi033 ceph-mon[27845]: pgmap v7750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:26 smithi096 ceph-mon[32753]: pgmap v7750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:26 smithi194 ceph-mon[33017]: pgmap v7750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:27.628+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:27.629+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:27.629+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:36:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13526.897989 secs 2024-04-26T15:36:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:28 smithi033 ceph-mon[27845]: pgmap v7751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13526.897989 secs 2024-04-26T15:36:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:28 smithi096 ceph-mon[32753]: pgmap v7751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13526.897989 secs 2024-04-26T15:36:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:28 smithi194 ceph-mon[33017]: pgmap v7751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13529.698499 secs 2024-04-26T15:36:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13529.698499 secs 2024-04-26T15:36:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13529.698499 secs 2024-04-26T15:36:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:30 smithi033 ceph-mon[27845]: pgmap v7752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:30 smithi096 ceph-mon[32753]: pgmap v7752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:30 smithi194 ceph-mon[33017]: pgmap v7752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:32 smithi033 ceph-mon[27845]: pgmap v7753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:32 smithi096 ceph-mon[32753]: pgmap v7753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:32 smithi194 ceph-mon[33017]: pgmap v7753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13531.898069 secs 2024-04-26T15:36:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13531.898069 secs 2024-04-26T15:36:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13531.898069 secs 2024-04-26T15:36:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13534.698621 secs 2024-04-26T15:36:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:34 smithi033 ceph-mon[27845]: pgmap v7754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13534.698621 secs 2024-04-26T15:36:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:34 smithi096 ceph-mon[32753]: pgmap v7754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13534.698621 secs 2024-04-26T15:36:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:34 smithi194 ceph-mon[33017]: pgmap v7754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:36:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:36 smithi096 ceph-mon[32753]: pgmap v7755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:36 smithi194 ceph-mon[33017]: pgmap v7755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:36 smithi033 ceph-mon[27845]: pgmap v7755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:36:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13536.898129 secs 2024-04-26T15:36:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:38 smithi096 ceph-mon[32753]: pgmap v7756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13536.898129 secs 2024-04-26T15:36:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:38 smithi194 ceph-mon[33017]: pgmap v7756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13536.898129 secs 2024-04-26T15:36:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:38 smithi033 ceph-mon[27845]: pgmap v7756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:40.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13539.698737 secs 2024-04-26T15:36:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13539.698737 secs 2024-04-26T15:36:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13539.698737 secs 2024-04-26T15:36:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:40 smithi194 ceph-mon[33017]: pgmap v7757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:40 smithi033 ceph-mon[27845]: pgmap v7757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:40 smithi096 ceph-mon[32753]: pgmap v7757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:41 smithi194 ceph-mon[33017]: pgmap v7758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:41 smithi033 ceph-mon[27845]: pgmap v7758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:41 smithi096 ceph-mon[32753]: pgmap v7758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:42.633+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:42.633+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:36:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:42.633+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13541.898216 secs 2024-04-26T15:36:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13541.898216 secs 2024-04-26T15:36:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13541.898216 secs 2024-04-26T15:36:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:43 smithi194 ceph-mon[33017]: pgmap v7759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:43 smithi033 ceph-mon[27845]: pgmap v7759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:43 smithi096 ceph-mon[32753]: pgmap v7759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13544.698854 secs 2024-04-26T15:36:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13544.698854 secs 2024-04-26T15:36:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13544.698854 secs 2024-04-26T15:36:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:45 smithi194 ceph-mon[33017]: pgmap v7760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:45 smithi033 ceph-mon[27845]: pgmap v7760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:45 smithi096 ceph-mon[32753]: pgmap v7760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13546.898304 secs 2024-04-26T15:36:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:48 smithi033 ceph-mon[27845]: pgmap v7761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13546.898304 secs 2024-04-26T15:36:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:48 smithi096 ceph-mon[32753]: pgmap v7761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13546.898304 secs 2024-04-26T15:36:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:48 smithi194 ceph-mon[33017]: pgmap v7761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:50.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13549.698989 secs 2024-04-26T15:36:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13549.698989 secs 2024-04-26T15:36:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13549.698989 secs 2024-04-26T15:36:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:50 smithi033 ceph-mon[27845]: pgmap v7762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:50 smithi096 ceph-mon[32753]: pgmap v7762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:50 smithi194 ceph-mon[33017]: pgmap v7762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:53.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:52 smithi033 ceph-mon[27845]: pgmap v7763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:52 smithi096 ceph-mon[32753]: pgmap v7763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:52 smithi194 ceph-mon[33017]: pgmap v7763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13551.898392 secs 2024-04-26T15:36:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13551.898392 secs 2024-04-26T15:36:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13551.898392 secs 2024-04-26T15:36:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13554.699129 secs 2024-04-26T15:36:55.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:54 smithi033 ceph-mon[27845]: pgmap v7764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13554.699129 secs 2024-04-26T15:36:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:54 smithi096 ceph-mon[32753]: pgmap v7764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:55.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13554.699129 secs 2024-04-26T15:36:55.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:54 smithi194 ceph-mon[33017]: pgmap v7764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:56 smithi033 ceph-mon[27845]: pgmap v7765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:56 smithi096 ceph-mon[32753]: pgmap v7765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:56 smithi194 ceph-mon[33017]: pgmap v7765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:36:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:57.643+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:57.643+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:36:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:36:57.643+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:36:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13556.898491 secs 2024-04-26T15:36:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:58 smithi033 ceph-mon[27845]: pgmap v7766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13556.898491 secs 2024-04-26T15:36:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:58 smithi096 ceph-mon[32753]: pgmap v7766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:36:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13556.898491 secs 2024-04-26T15:36:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:58 smithi194 ceph-mon[33017]: pgmap v7766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13559.699195 secs 2024-04-26T15:37:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13559.699195 secs 2024-04-26T15:37:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13559.699195 secs 2024-04-26T15:37:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:00 smithi194 ceph-mon[33017]: pgmap v7767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:00 smithi033 ceph-mon[27845]: pgmap v7767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:00 smithi096 ceph-mon[32753]: pgmap v7767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:02 smithi194 ceph-mon[33017]: pgmap v7768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:02 smithi033 ceph-mon[27845]: pgmap v7768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:02 smithi096 ceph-mon[32753]: pgmap v7768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13561.898576 secs 2024-04-26T15:37:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:03 smithi194 ceph-mon[33017]: pgmap v7769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13561.898576 secs 2024-04-26T15:37:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:03 smithi033 ceph-mon[27845]: pgmap v7769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13561.898576 secs 2024-04-26T15:37:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:03 smithi096 ceph-mon[32753]: pgmap v7769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13564.699333 secs 2024-04-26T15:37:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13564.699333 secs 2024-04-26T15:37:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13564.699333 secs 2024-04-26T15:37:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:05 smithi194 ceph-mon[33017]: pgmap v7770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:05 smithi033 ceph-mon[27845]: pgmap v7770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:05 smithi096 ceph-mon[32753]: pgmap v7770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13566.898662 secs 2024-04-26T15:37:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:08 smithi033 ceph-mon[27845]: pgmap v7771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13566.898662 secs 2024-04-26T15:37:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:08 smithi096 ceph-mon[32753]: pgmap v7771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13566.898662 secs 2024-04-26T15:37:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:08 smithi194 ceph-mon[33017]: pgmap v7771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13569.699434 secs 2024-04-26T15:37:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13569.699434 secs 2024-04-26T15:37:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13569.699434 secs 2024-04-26T15:37:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:10 smithi033 ceph-mon[27845]: pgmap v7772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:10 smithi096 ceph-mon[32753]: pgmap v7772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:10 smithi194 ceph-mon[33017]: pgmap v7772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:12.654+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:12.654+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:12.654+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:37:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:12 smithi033 ceph-mon[27845]: pgmap v7773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:12 smithi096 ceph-mon[32753]: pgmap v7773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:12 smithi194 ceph-mon[33017]: pgmap v7773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13571.898740 secs 2024-04-26T15:37:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13571.898740 secs 2024-04-26T15:37:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13571.898740 secs 2024-04-26T15:37:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13574.699562 secs 2024-04-26T15:37:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:14 smithi096 ceph-mon[32753]: pgmap v7774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:37:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13574.699562 secs 2024-04-26T15:37:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:14 smithi194 ceph-mon[33017]: pgmap v7774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:37:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13574.699562 secs 2024-04-26T15:37:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:14 smithi033 ceph-mon[27845]: pgmap v7774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:37:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:37:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:37:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:37:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:37:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:37:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:37:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:37:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:37:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:37:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:16 smithi194 ceph-mon[33017]: pgmap v7775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:37:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:16 smithi194 ceph-mon[33017]: pgmap v7776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:16 smithi033 ceph-mon[27845]: pgmap v7775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:37:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:16 smithi033 ceph-mon[27845]: pgmap v7776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:16 smithi096 ceph-mon[32753]: pgmap v7775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:37:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:16 smithi096 ceph-mon[32753]: pgmap v7776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:17 smithi033 ceph-mon[27845]: pgmap v7777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13576.898828 secs 2024-04-26T15:37:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:17 smithi194 ceph-mon[33017]: pgmap v7777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13576.898828 secs 2024-04-26T15:37:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:17 smithi096 ceph-mon[32753]: pgmap v7777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13576.898828 secs 2024-04-26T15:37:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13579.699719 secs 2024-04-26T15:37:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13579.699719 secs 2024-04-26T15:37:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13579.699719 secs 2024-04-26T15:37:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:20 smithi194 ceph-mon[33017]: pgmap v7778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:20 smithi033 ceph-mon[27845]: pgmap v7778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:20 smithi096 ceph-mon[32753]: pgmap v7778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:22 smithi194 ceph-mon[33017]: pgmap v7779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:22 smithi033 ceph-mon[27845]: pgmap v7779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:22 smithi096 ceph-mon[32753]: pgmap v7779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13581.898912 secs 2024-04-26T15:37:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13581.898912 secs 2024-04-26T15:37:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13581.898912 secs 2024-04-26T15:37:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13584.699842 secs 2024-04-26T15:37:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:24 smithi194 ceph-mon[33017]: pgmap v7780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13584.699842 secs 2024-04-26T15:37:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:24 smithi033 ceph-mon[27845]: pgmap v7780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13584.699842 secs 2024-04-26T15:37:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:24 smithi096 ceph-mon[32753]: pgmap v7780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:37:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:26 smithi194 ceph-mon[33017]: pgmap v7781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:37:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:26 smithi033 ceph-mon[27845]: pgmap v7781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:37:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:26 smithi096 ceph-mon[32753]: pgmap v7781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:37:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:27.658+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:27.659+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:27.659+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:37:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:28 smithi194 ceph-mon[33017]: pgmap v7782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13586.898979 secs 2024-04-26T15:37:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:28 smithi033 ceph-mon[27845]: pgmap v7782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13586.898979 secs 2024-04-26T15:37:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:28 smithi096 ceph-mon[32753]: pgmap v7782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13586.898979 secs 2024-04-26T15:37:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13589.699971 secs 2024-04-26T15:37:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13589.699971 secs 2024-04-26T15:37:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13589.699971 secs 2024-04-26T15:37:30.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:30 smithi096 ceph-mon[32753]: pgmap v7783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:30 smithi194 ceph-mon[33017]: pgmap v7783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:30.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:30 smithi033 ceph-mon[27845]: pgmap v7783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:32 smithi194 ceph-mon[33017]: pgmap v7784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:32 smithi033 ceph-mon[27845]: pgmap v7784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:32 smithi096 ceph-mon[32753]: pgmap v7784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13591.899042 secs 2024-04-26T15:37:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13591.899042 secs 2024-04-26T15:37:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13591.899042 secs 2024-04-26T15:37:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13594.700090 secs 2024-04-26T15:37:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:34 smithi194 ceph-mon[33017]: pgmap v7785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13594.700090 secs 2024-04-26T15:37:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:34 smithi033 ceph-mon[27845]: pgmap v7785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13594.700090 secs 2024-04-26T15:37:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:34 smithi096 ceph-mon[32753]: pgmap v7785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:36 smithi033 ceph-mon[27845]: pgmap v7786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:36 smithi096 ceph-mon[32753]: pgmap v7786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:36 smithi194 ceph-mon[33017]: pgmap v7786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:38 smithi033 ceph-mon[27845]: pgmap v7787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13596.899102 secs 2024-04-26T15:37:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:38 smithi096 ceph-mon[32753]: pgmap v7787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13596.899102 secs 2024-04-26T15:37:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:38 smithi194 ceph-mon[33017]: pgmap v7787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13596.899102 secs 2024-04-26T15:37:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13599.700233 secs 2024-04-26T15:37:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13599.700233 secs 2024-04-26T15:37:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13599.700233 secs 2024-04-26T15:37:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:40 smithi033 ceph-mon[27845]: pgmap v7788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:40 smithi096 ceph-mon[32753]: pgmap v7788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:40 smithi194 ceph-mon[33017]: pgmap v7788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:42.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:42.673+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:42.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:42.673+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:42.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:42.673+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:37:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:42 smithi033 ceph-mon[27845]: pgmap v7789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:42 smithi096 ceph-mon[32753]: pgmap v7789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:42 smithi194 ceph-mon[33017]: pgmap v7789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13601.899187 secs 2024-04-26T15:37:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13601.899187 secs 2024-04-26T15:37:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13601.899187 secs 2024-04-26T15:37:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13604.700358 secs 2024-04-26T15:37:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:44 smithi033 ceph-mon[27845]: pgmap v7790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13604.700358 secs 2024-04-26T15:37:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:44 smithi096 ceph-mon[32753]: pgmap v7790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13604.700358 secs 2024-04-26T15:37:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:44 smithi194 ceph-mon[33017]: pgmap v7790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:46.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:46 smithi033 ceph-mon[27845]: pgmap v7791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:46 smithi096 ceph-mon[32753]: pgmap v7791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:46 smithi194 ceph-mon[33017]: pgmap v7791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:48 smithi033 ceph-mon[27845]: pgmap v7792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13606.899263 secs 2024-04-26T15:37:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:48 smithi096 ceph-mon[32753]: pgmap v7792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13606.899263 secs 2024-04-26T15:37:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:48 smithi194 ceph-mon[33017]: pgmap v7792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13606.899263 secs 2024-04-26T15:37:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13609.700474 secs 2024-04-26T15:37:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13609.700474 secs 2024-04-26T15:37:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13609.700474 secs 2024-04-26T15:37:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:50 smithi033 ceph-mon[27845]: pgmap v7793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:50 smithi096 ceph-mon[32753]: pgmap v7793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:50 smithi194 ceph-mon[33017]: pgmap v7793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:52 smithi033 ceph-mon[27845]: pgmap v7794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:52 smithi096 ceph-mon[32753]: pgmap v7794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:52 smithi194 ceph-mon[33017]: pgmap v7794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13611.899324 secs 2024-04-26T15:37:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13611.899324 secs 2024-04-26T15:37:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13611.899324 secs 2024-04-26T15:37:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13614.700597 secs 2024-04-26T15:37:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:54 smithi194 ceph-mon[33017]: pgmap v7795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13614.700597 secs 2024-04-26T15:37:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:54 smithi033 ceph-mon[27845]: pgmap v7795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13614.700597 secs 2024-04-26T15:37:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:54 smithi096 ceph-mon[32753]: pgmap v7795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:56 smithi194 ceph-mon[33017]: pgmap v7796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:56 smithi033 ceph-mon[27845]: pgmap v7796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:56 smithi096 ceph-mon[32753]: pgmap v7796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:37:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:57.689+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:57.689+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:37:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:37:57.689+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:37:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:58 smithi194 ceph-mon[33017]: pgmap v7797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13616.899408 secs 2024-04-26T15:37:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:58 smithi033 ceph-mon[27845]: pgmap v7797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13616.899408 secs 2024-04-26T15:37:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:58 smithi096 ceph-mon[32753]: pgmap v7797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:37:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13616.899408 secs 2024-04-26T15:38:00.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13619.700734 secs 2024-04-26T15:38:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13619.700734 secs 2024-04-26T15:38:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13619.700734 secs 2024-04-26T15:38:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:00 smithi194 ceph-mon[33017]: pgmap v7798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:01.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:00 smithi033 ceph-mon[27845]: pgmap v7798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:00 smithi096 ceph-mon[32753]: pgmap v7798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:02 smithi194 ceph-mon[33017]: pgmap v7799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:02 smithi033 ceph-mon[27845]: pgmap v7799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:02 smithi096 ceph-mon[32753]: pgmap v7799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13621.899493 secs 2024-04-26T15:38:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13621.899493 secs 2024-04-26T15:38:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13621.899493 secs 2024-04-26T15:38:04.983 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13624.700868 secs 2024-04-26T15:38:04.983 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:04 smithi033 ceph-mon[27845]: pgmap v7800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13624.700868 secs 2024-04-26T15:38:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:04 smithi194 ceph-mon[33017]: pgmap v7800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13624.700868 secs 2024-04-26T15:38:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:04 smithi096 ceph-mon[32753]: pgmap v7800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:06 smithi194 ceph-mon[33017]: pgmap v7801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:06 smithi033 ceph-mon[27845]: pgmap v7801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:06 smithi096 ceph-mon[32753]: pgmap v7801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:08 smithi194 ceph-mon[33017]: pgmap v7802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13626.899577 secs 2024-04-26T15:38:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:08 smithi033 ceph-mon[27845]: pgmap v7802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13626.899577 secs 2024-04-26T15:38:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:08 smithi096 ceph-mon[32753]: pgmap v7802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13626.899577 secs 2024-04-26T15:38:10.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13629.700999 secs 2024-04-26T15:38:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13629.700999 secs 2024-04-26T15:38:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13629.700999 secs 2024-04-26T15:38:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:10 smithi194 ceph-mon[33017]: pgmap v7803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:10 smithi033 ceph-mon[27845]: pgmap v7803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:10 smithi096 ceph-mon[32753]: pgmap v7803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:12.700+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:12.700+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:38:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:12.700+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:12 smithi194 ceph-mon[33017]: pgmap v7804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:12 smithi033 ceph-mon[27845]: pgmap v7804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:12 smithi096 ceph-mon[32753]: pgmap v7804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13631.899639 secs 2024-04-26T15:38:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13631.899639 secs 2024-04-26T15:38:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13631.899639 secs 2024-04-26T15:38:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13634.701143 secs 2024-04-26T15:38:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:14 smithi194 ceph-mon[33017]: pgmap v7805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13634.701143 secs 2024-04-26T15:38:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:14 smithi033 ceph-mon[27845]: pgmap v7805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13634.701143 secs 2024-04-26T15:38:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:14 smithi096 ceph-mon[32753]: pgmap v7805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:16 smithi194 ceph-mon[33017]: pgmap v7806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:38:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:16 smithi033 ceph-mon[27845]: pgmap v7806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:38:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:16 smithi096 ceph-mon[32753]: pgmap v7806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:38:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:18 smithi194 ceph-mon[33017]: pgmap v7807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13636.899725 secs 2024-04-26T15:38:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:38:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:38:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:18 smithi033 ceph-mon[27845]: pgmap v7807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13636.899725 secs 2024-04-26T15:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:38:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:18 smithi096 ceph-mon[32753]: pgmap v7807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13636.899725 secs 2024-04-26T15:38:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:38:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:38:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:38:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:19 smithi194 ceph-mon[33017]: pgmap v7808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:38:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:19 smithi194 ceph-mon[33017]: pgmap v7809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:38:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13639.701265 secs 2024-04-26T15:38:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:19 smithi033 ceph-mon[27845]: pgmap v7808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:38:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:19 smithi033 ceph-mon[27845]: pgmap v7809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:38:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13639.701265 secs 2024-04-26T15:38:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:19 smithi096 ceph-mon[32753]: pgmap v7808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:38:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:19 smithi096 ceph-mon[32753]: pgmap v7809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:38:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13639.701265 secs 2024-04-26T15:38:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:21 smithi194 ceph-mon[33017]: pgmap v7810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:21 smithi033 ceph-mon[27845]: pgmap v7810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:21 smithi096 ceph-mon[32753]: pgmap v7810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:23 smithi194 ceph-mon[33017]: pgmap v7811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13641.899813 secs 2024-04-26T15:38:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:24.115+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:38:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:24.115+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:38:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:24.116+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:38:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:24.116+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:38:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:23 smithi033 ceph-mon[27845]: pgmap v7811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13641.899813 secs 2024-04-26T15:38:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:23 smithi096 ceph-mon[32753]: pgmap v7811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13641.899813 secs 2024-04-26T15:38:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13644.701322 secs 2024-04-26T15:38:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13644.701322 secs 2024-04-26T15:38:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13644.701322 secs 2024-04-26T15:38:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:25 smithi194 ceph-mon[33017]: pgmap v7812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:38:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:25 smithi033 ceph-mon[27845]: pgmap v7812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:38:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:25 smithi096 ceph-mon[32753]: pgmap v7812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:38:27.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:27.704+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:27.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:27.705+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:27.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:27.705+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:38:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:27 smithi194 ceph-mon[33017]: pgmap v7813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:27 smithi033 ceph-mon[27845]: pgmap v7813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:27 smithi096 ceph-mon[32753]: pgmap v7813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:38:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13646.899896 secs 2024-04-26T15:38:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13646.899896 secs 2024-04-26T15:38:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13646.899896 secs 2024-04-26T15:38:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:29 smithi194 ceph-mon[33017]: pgmap v7814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:38:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13649.701441 secs 2024-04-26T15:38:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:29 smithi033 ceph-mon[27845]: pgmap v7814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:38:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13649.701441 secs 2024-04-26T15:38:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:29 smithi096 ceph-mon[32753]: pgmap v7814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:38:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13649.701441 secs 2024-04-26T15:38:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:31 smithi194 ceph-mon[33017]: pgmap v7815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:31 smithi033 ceph-mon[27845]: pgmap v7815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:31 smithi096 ceph-mon[32753]: pgmap v7815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:33 smithi194 ceph-mon[33017]: pgmap v7816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13651.899997 secs 2024-04-26T15:38:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:33 smithi033 ceph-mon[27845]: pgmap v7816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13651.899997 secs 2024-04-26T15:38:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:33 smithi096 ceph-mon[32753]: pgmap v7816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13651.899997 secs 2024-04-26T15:38:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13654.701538 secs 2024-04-26T15:38:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13654.701538 secs 2024-04-26T15:38:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13654.701538 secs 2024-04-26T15:38:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:35 smithi033 ceph-mon[27845]: pgmap v7817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:35 smithi096 ceph-mon[32753]: pgmap v7817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:35 smithi194 ceph-mon[33017]: pgmap v7817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:37 smithi033 ceph-mon[27845]: pgmap v7818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:37 smithi096 ceph-mon[32753]: pgmap v7818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:37 smithi194 ceph-mon[33017]: pgmap v7818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13656.900055 secs 2024-04-26T15:38:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13656.900055 secs 2024-04-26T15:38:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13656.900055 secs 2024-04-26T15:38:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:39 smithi033 ceph-mon[27845]: pgmap v7819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13659.701679 secs 2024-04-26T15:38:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:39 smithi096 ceph-mon[32753]: pgmap v7819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13659.701679 secs 2024-04-26T15:38:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:39 smithi194 ceph-mon[33017]: pgmap v7819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13659.701679 secs 2024-04-26T15:38:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:41 smithi033 ceph-mon[27845]: pgmap v7820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:41 smithi096 ceph-mon[32753]: pgmap v7820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:41 smithi194 ceph-mon[33017]: pgmap v7820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:42.720+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:42.720+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:42.720+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:38:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:43 smithi033 ceph-mon[27845]: pgmap v7821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13661.900114 secs 2024-04-26T15:38:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:43 smithi096 ceph-mon[32753]: pgmap v7821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13661.900114 secs 2024-04-26T15:38:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:43 smithi194 ceph-mon[33017]: pgmap v7821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13661.900114 secs 2024-04-26T15:38:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13664.701745 secs 2024-04-26T15:38:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13664.701745 secs 2024-04-26T15:38:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13664.701745 secs 2024-04-26T15:38:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:45 smithi033 ceph-mon[27845]: pgmap v7822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:45 smithi096 ceph-mon[32753]: pgmap v7822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:45 smithi194 ceph-mon[33017]: pgmap v7822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:47 smithi033 ceph-mon[27845]: pgmap v7823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:47 smithi096 ceph-mon[32753]: pgmap v7823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:47 smithi194 ceph-mon[33017]: pgmap v7823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13666.900173 secs 2024-04-26T15:38:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13666.900173 secs 2024-04-26T15:38:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13666.900173 secs 2024-04-26T15:38:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:49 smithi096 ceph-mon[32753]: pgmap v7824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13669.701862 secs 2024-04-26T15:38:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:49 smithi194 ceph-mon[33017]: pgmap v7824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13669.701862 secs 2024-04-26T15:38:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:49 smithi033 ceph-mon[27845]: pgmap v7824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13669.701862 secs 2024-04-26T15:38:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:51 smithi096 ceph-mon[32753]: pgmap v7825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:52.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:51 smithi033 ceph-mon[27845]: pgmap v7825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:51 smithi194 ceph-mon[33017]: pgmap v7825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:53 smithi194 ceph-mon[33017]: pgmap v7826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13671.900251 secs 2024-04-26T15:38:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:53 smithi033 ceph-mon[27845]: pgmap v7826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13671.900251 secs 2024-04-26T15:38:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:53 smithi096 ceph-mon[32753]: pgmap v7826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13671.900251 secs 2024-04-26T15:38:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13674.701962 secs 2024-04-26T15:38:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13674.701962 secs 2024-04-26T15:38:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13674.701962 secs 2024-04-26T15:38:56.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:55 smithi194 ceph-mon[33017]: pgmap v7827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:55 smithi033 ceph-mon[27845]: pgmap v7827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:55 smithi096 ceph-mon[32753]: pgmap v7827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:38:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:57.726+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:57.726+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:38:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:38:57.726+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:38:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:57 smithi194 ceph-mon[33017]: pgmap v7828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:57 smithi033 ceph-mon[27845]: pgmap v7828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:57 smithi096 ceph-mon[32753]: pgmap v7828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:38:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13676.900334 secs 2024-04-26T15:38:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13676.900334 secs 2024-04-26T15:38:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13676.900334 secs 2024-04-26T15:39:00.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:59 smithi096 ceph-mon[32753]: pgmap v7829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:00.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13679.702076 secs 2024-04-26T15:39:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:59 smithi194 ceph-mon[33017]: pgmap v7829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13679.702076 secs 2024-04-26T15:39:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:59 smithi033 ceph-mon[27845]: pgmap v7829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13679.702076 secs 2024-04-26T15:39:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:00 smithi194 ceph-mon[33017]: pgmap v7830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:00 smithi033 ceph-mon[27845]: pgmap v7830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:00 smithi096 ceph-mon[32753]: pgmap v7830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:03 smithi194 ceph-mon[33017]: pgmap v7831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13681.900393 secs 2024-04-26T15:39:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:03 smithi033 ceph-mon[27845]: pgmap v7831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13681.900393 secs 2024-04-26T15:39:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:03 smithi096 ceph-mon[32753]: pgmap v7831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13681.900393 secs 2024-04-26T15:39:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13684.702189 secs 2024-04-26T15:39:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13684.702189 secs 2024-04-26T15:39:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13684.702189 secs 2024-04-26T15:39:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:05 smithi194 ceph-mon[33017]: pgmap v7832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:05 smithi033 ceph-mon[27845]: pgmap v7832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:05 smithi096 ceph-mon[32753]: pgmap v7832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:07.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:07 smithi194 ceph-mon[33017]: pgmap v7833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:07 smithi033 ceph-mon[27845]: pgmap v7833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:07 smithi096 ceph-mon[32753]: pgmap v7833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13686.900476 secs 2024-04-26T15:39:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13686.900476 secs 2024-04-26T15:39:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13686.900476 secs 2024-04-26T15:39:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:09 smithi194 ceph-mon[33017]: pgmap v7834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13689.702304 secs 2024-04-26T15:39:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:09 smithi033 ceph-mon[27845]: pgmap v7834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13689.702304 secs 2024-04-26T15:39:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:09 smithi096 ceph-mon[32753]: pgmap v7834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13689.702304 secs 2024-04-26T15:39:11.477 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:11 smithi033 ceph-mon[27845]: pgmap v7835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:11 smithi194 ceph-mon[33017]: pgmap v7835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:11 smithi096 ceph-mon[32753]: pgmap v7835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:12.739+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:12.739+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:39:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:12.739+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:13 smithi194 ceph-mon[33017]: pgmap v7836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13691.900561 secs 2024-04-26T15:39:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:13 smithi033 ceph-mon[27845]: pgmap v7836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13691.900561 secs 2024-04-26T15:39:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:13 smithi096 ceph-mon[32753]: pgmap v7836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13691.900561 secs 2024-04-26T15:39:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13694.702449 secs 2024-04-26T15:39:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13694.702449 secs 2024-04-26T15:39:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13694.702449 secs 2024-04-26T15:39:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:15 smithi194 ceph-mon[33017]: pgmap v7837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:15 smithi033 ceph-mon[27845]: pgmap v7837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:15 smithi096 ceph-mon[32753]: pgmap v7837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:17.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:17 smithi194 ceph-mon[33017]: pgmap v7838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:17 smithi033 ceph-mon[27845]: pgmap v7838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:17 smithi096 ceph-mon[32753]: pgmap v7838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13696.900622 secs 2024-04-26T15:39:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13696.900622 secs 2024-04-26T15:39:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13696.900622 secs 2024-04-26T15:39:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:19 smithi194 ceph-mon[33017]: pgmap v7839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13699.702516 secs 2024-04-26T15:39:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:39:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:19 smithi033 ceph-mon[27845]: pgmap v7839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13699.702516 secs 2024-04-26T15:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:19 smithi096 ceph-mon[32753]: pgmap v7839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13699.702516 secs 2024-04-26T15:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:39:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:39:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:39:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:20 smithi194 ceph-mon[33017]: pgmap v7840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:39:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:20 smithi194 ceph-mon[33017]: pgmap v7841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:39:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:39:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:39:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:20 smithi033 ceph-mon[27845]: pgmap v7840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:39:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:20 smithi033 ceph-mon[27845]: pgmap v7841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:39:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:39:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:39:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:20 smithi096 ceph-mon[32753]: pgmap v7840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:39:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:20 smithi096 ceph-mon[32753]: pgmap v7841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:39:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:22 smithi194 ceph-mon[33017]: pgmap v7842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:22 smithi033 ceph-mon[27845]: pgmap v7842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:22 smithi096 ceph-mon[32753]: pgmap v7842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13701.900705 secs 2024-04-26T15:39:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13701.900705 secs 2024-04-26T15:39:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13701.900705 secs 2024-04-26T15:39:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13704.702612 secs 2024-04-26T15:39:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:24 smithi194 ceph-mon[33017]: pgmap v7843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13704.702612 secs 2024-04-26T15:39:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:24 smithi033 ceph-mon[27845]: pgmap v7843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13704.702612 secs 2024-04-26T15:39:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:24 smithi096 ceph-mon[32753]: pgmap v7843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:26 smithi194 ceph-mon[33017]: pgmap v7844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:26 smithi033 ceph-mon[27845]: pgmap v7844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:26 smithi096 ceph-mon[32753]: pgmap v7844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:27.741+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:27.741+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:27.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:27.741+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:39:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13706.900789 secs 2024-04-26T15:39:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:28 smithi194 ceph-mon[33017]: pgmap v7845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13706.900789 secs 2024-04-26T15:39:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:28 smithi033 ceph-mon[27845]: pgmap v7845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13706.900789 secs 2024-04-26T15:39:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:28 smithi096 ceph-mon[32753]: pgmap v7845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:39:30.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13709.702729 secs 2024-04-26T15:39:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13709.702729 secs 2024-04-26T15:39:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13709.702729 secs 2024-04-26T15:39:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:30 smithi194 ceph-mon[33017]: pgmap v7846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:39:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:30 smithi033 ceph-mon[27845]: pgmap v7846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:39:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:30 smithi096 ceph-mon[32753]: pgmap v7846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:39:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:32 smithi194 ceph-mon[33017]: pgmap v7847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:32 smithi033 ceph-mon[27845]: pgmap v7847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:32 smithi096 ceph-mon[32753]: pgmap v7847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13711.900874 secs 2024-04-26T15:39:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13711.900874 secs 2024-04-26T15:39:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13711.900874 secs 2024-04-26T15:39:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13714.702836 secs 2024-04-26T15:39:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:34 smithi194 ceph-mon[33017]: pgmap v7848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13714.702836 secs 2024-04-26T15:39:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:34 smithi033 ceph-mon[27845]: pgmap v7848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13714.702836 secs 2024-04-26T15:39:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:34 smithi096 ceph-mon[32753]: pgmap v7848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:36 smithi194 ceph-mon[33017]: pgmap v7849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:36 smithi033 ceph-mon[27845]: pgmap v7849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:36 smithi096 ceph-mon[32753]: pgmap v7849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13716.900958 secs 2024-04-26T15:39:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:38 smithi194 ceph-mon[33017]: pgmap v7850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13716.900958 secs 2024-04-26T15:39:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:38 smithi033 ceph-mon[27845]: pgmap v7850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13716.900958 secs 2024-04-26T15:39:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:38 smithi096 ceph-mon[32753]: pgmap v7850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13719.702954 secs 2024-04-26T15:39:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13719.702954 secs 2024-04-26T15:39:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13719.702954 secs 2024-04-26T15:39:41.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:40 smithi033 ceph-mon[27845]: pgmap v7851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:40 smithi194 ceph-mon[33017]: pgmap v7851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:40 smithi096 ceph-mon[32753]: pgmap v7851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:42.756+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:42.756+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:42.756+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:39:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:42 smithi194 ceph-mon[33017]: pgmap v7852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:42 smithi033 ceph-mon[27845]: pgmap v7852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:42 smithi096 ceph-mon[32753]: pgmap v7852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13721.901043 secs 2024-04-26T15:39:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13721.901043 secs 2024-04-26T15:39:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13721.901043 secs 2024-04-26T15:39:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13724.703069 secs 2024-04-26T15:39:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:44 smithi194 ceph-mon[33017]: pgmap v7853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13724.703069 secs 2024-04-26T15:39:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:44 smithi033 ceph-mon[27845]: pgmap v7853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13724.703069 secs 2024-04-26T15:39:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:44 smithi096 ceph-mon[32753]: pgmap v7853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:46 smithi194 ceph-mon[33017]: pgmap v7854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:47.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:46 smithi033 ceph-mon[27845]: pgmap v7854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:46 smithi096 ceph-mon[32753]: pgmap v7854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13726.901127 secs 2024-04-26T15:39:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:48 smithi194 ceph-mon[33017]: pgmap v7855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13726.901127 secs 2024-04-26T15:39:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:48 smithi033 ceph-mon[27845]: pgmap v7855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13726.901127 secs 2024-04-26T15:39:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:48 smithi096 ceph-mon[32753]: pgmap v7855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13729.703190 secs 2024-04-26T15:39:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13729.703190 secs 2024-04-26T15:39:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13729.703190 secs 2024-04-26T15:39:51.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:50 smithi033 ceph-mon[27845]: pgmap v7856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:50 smithi096 ceph-mon[32753]: pgmap v7856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:50 smithi194 ceph-mon[33017]: pgmap v7856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:52 smithi033 ceph-mon[27845]: pgmap v7857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:52 smithi096 ceph-mon[32753]: pgmap v7857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:52 smithi194 ceph-mon[33017]: pgmap v7857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13731.901189 secs 2024-04-26T15:39:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13731.901189 secs 2024-04-26T15:39:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13731.901189 secs 2024-04-26T15:39:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13734.703298 secs 2024-04-26T15:39:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:54 smithi033 ceph-mon[27845]: pgmap v7858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13734.703298 secs 2024-04-26T15:39:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:54 smithi096 ceph-mon[32753]: pgmap v7858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13734.703298 secs 2024-04-26T15:39:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:54 smithi194 ceph-mon[33017]: pgmap v7858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:57.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:56 smithi194 ceph-mon[33017]: pgmap v7859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:56 smithi033 ceph-mon[27845]: pgmap v7859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:56 smithi096 ceph-mon[32753]: pgmap v7859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:39:57.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:57.772+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:57.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:57.772+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:39:57.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:39:57.772+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:39:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13736.901251 secs 2024-04-26T15:39:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:58 smithi033 ceph-mon[27845]: pgmap v7860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13736.901251 secs 2024-04-26T15:39:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:58 smithi096 ceph-mon[32753]: pgmap v7860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:39:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13736.901251 secs 2024-04-26T15:39:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:58 smithi194 ceph-mon[33017]: pgmap v7860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13739.703408 secs 2024-04-26T15:40:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13739.703408 secs 2024-04-26T15:40:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13739.703408 secs 2024-04-26T15:40:01.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:00 smithi033 ceph-mon[27845]: pgmap v7861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:40:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:00 smithi096 ceph-mon[32753]: pgmap v7861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:40:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:00 smithi194 ceph-mon[33017]: pgmap v7861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:40:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:02 smithi033 ceph-mon[27845]: pgmap v7862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:02 smithi096 ceph-mon[32753]: pgmap v7862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:02 smithi194 ceph-mon[33017]: pgmap v7862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13741.901310 secs 2024-04-26T15:40:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13741.901310 secs 2024-04-26T15:40:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13741.901310 secs 2024-04-26T15:40:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13744.703542 secs 2024-04-26T15:40:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:04 smithi194 ceph-mon[33017]: pgmap v7863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13744.703542 secs 2024-04-26T15:40:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:04 smithi033 ceph-mon[27845]: pgmap v7863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13744.703542 secs 2024-04-26T15:40:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:04 smithi096 ceph-mon[32753]: pgmap v7863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:06 smithi194 ceph-mon[33017]: pgmap v7864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:06 smithi033 ceph-mon[27845]: pgmap v7864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:06 smithi096 ceph-mon[32753]: pgmap v7864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13746.901371 secs 2024-04-26T15:40:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:08 smithi194 ceph-mon[33017]: pgmap v7865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13746.901371 secs 2024-04-26T15:40:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:08 smithi033 ceph-mon[27845]: pgmap v7865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13746.901371 secs 2024-04-26T15:40:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:08 smithi096 ceph-mon[32753]: pgmap v7865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13749.703664 secs 2024-04-26T15:40:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13749.703664 secs 2024-04-26T15:40:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13749.703664 secs 2024-04-26T15:40:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:10 smithi194 ceph-mon[33017]: pgmap v7866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:10 smithi033 ceph-mon[27845]: pgmap v7866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:10 smithi096 ceph-mon[32753]: pgmap v7866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:12.781+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:12.781+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:12.781+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:40:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:12 smithi194 ceph-mon[33017]: pgmap v7867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13751.901431 secs 2024-04-26T15:40:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:12 smithi033 ceph-mon[27845]: pgmap v7867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13751.901431 secs 2024-04-26T15:40:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:12 smithi096 ceph-mon[32753]: pgmap v7867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13751.901431 secs 2024-04-26T15:40:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13754.703783 secs 2024-04-26T15:40:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:14 smithi194 ceph-mon[33017]: pgmap v7868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13754.703783 secs 2024-04-26T15:40:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:14 smithi033 ceph-mon[27845]: pgmap v7868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13754.703783 secs 2024-04-26T15:40:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:14 smithi096 ceph-mon[32753]: pgmap v7868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:16 smithi194 ceph-mon[33017]: pgmap v7869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:16 smithi033 ceph-mon[27845]: pgmap v7869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:16 smithi096 ceph-mon[32753]: pgmap v7869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13756.901530 secs 2024-04-26T15:40:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13756.901530 secs 2024-04-26T15:40:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13756.901530 secs 2024-04-26T15:40:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:18 smithi194 ceph-mon[33017]: pgmap v7870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:18 smithi033 ceph-mon[27845]: pgmap v7870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:18 smithi096 ceph-mon[32753]: pgmap v7870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13759.703920 secs 2024-04-26T15:40:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:20 smithi194 ceph-mon[33017]: pgmap v7871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13759.703920 secs 2024-04-26T15:40:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:20 smithi033 ceph-mon[27845]: pgmap v7871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13759.703920 secs 2024-04-26T15:40:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:20 smithi096 ceph-mon[32753]: pgmap v7871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:21.338 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:40:21.339 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.339 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.339 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.339 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:40:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:40:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:22 smithi194 ceph-mon[33017]: pgmap v7872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:22 smithi033 ceph-mon[27845]: pgmap v7872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:22 smithi096 ceph-mon[32753]: pgmap v7872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13761.901624 secs 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13761.901624 secs 2024-04-26T15:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13761.901624 secs 2024-04-26T15:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13764.704014 secs 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: pgmap v7873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13764.704014 secs 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: pgmap v7873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13764.704014 secs 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: pgmap v7873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: pgmap v7874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:40:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: pgmap v7874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:40:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:40:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: pgmap v7874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:40:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:40:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:40:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:40:27.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:27.797+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:27.797+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:40:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:27.797+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:27 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:40:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:27 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:40:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:27 smithi194 ceph-mon[33017]: pgmap v7875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:40:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:27 smithi194 ceph-mon[33017]: pgmap v7876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:27 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:40:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:27 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:40:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:27 smithi033 ceph-mon[27845]: pgmap v7875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:40:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:27 smithi033 ceph-mon[27845]: pgmap v7876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:27 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:40:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:27 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:40:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:27 smithi096 ceph-mon[32753]: pgmap v7875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:40:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:27 smithi096 ceph-mon[32753]: pgmap v7876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13766.901706 secs 2024-04-26T15:40:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13766.901706 secs 2024-04-26T15:40:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13766.901706 secs 2024-04-26T15:40:30.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:29 smithi096 ceph-mon[32753]: pgmap v7877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:40:30.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13769.704148 secs 2024-04-26T15:40:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:29 smithi194 ceph-mon[33017]: pgmap v7877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:40:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13769.704148 secs 2024-04-26T15:40:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:29 smithi033 ceph-mon[27845]: pgmap v7877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:40:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13769.704148 secs 2024-04-26T15:40:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:31 smithi194 ceph-mon[33017]: pgmap v7878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:31 smithi033 ceph-mon[27845]: pgmap v7878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:31 smithi096 ceph-mon[32753]: pgmap v7878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:33 smithi194 ceph-mon[33017]: pgmap v7879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13771.901789 secs 2024-04-26T15:40:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:33 smithi033 ceph-mon[27845]: pgmap v7879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13771.901789 secs 2024-04-26T15:40:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:33 smithi096 ceph-mon[32753]: pgmap v7879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13771.901789 secs 2024-04-26T15:40:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13774.704286 secs 2024-04-26T15:40:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13774.704286 secs 2024-04-26T15:40:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13774.704286 secs 2024-04-26T15:40:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:35 smithi194 ceph-mon[33017]: pgmap v7880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:40:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:35 smithi033 ceph-mon[27845]: pgmap v7880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:40:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:35 smithi096 ceph-mon[32753]: pgmap v7880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:40:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:37 smithi194 ceph-mon[33017]: pgmap v7881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:40:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:37 smithi033 ceph-mon[27845]: pgmap v7881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:40:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:37 smithi096 ceph-mon[32753]: pgmap v7881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:40:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13776.901875 secs 2024-04-26T15:40:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:39.139+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:40:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:39.139+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:40:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:39.140+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:40:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:39.140+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:40:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13776.901875 secs 2024-04-26T15:40:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13776.901875 secs 2024-04-26T15:40:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:39 smithi194 ceph-mon[33017]: pgmap v7882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13779.704413 secs 2024-04-26T15:40:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:39 smithi033 ceph-mon[27845]: pgmap v7882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13779.704413 secs 2024-04-26T15:40:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:39 smithi096 ceph-mon[32753]: pgmap v7882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13779.704413 secs 2024-04-26T15:40:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:41 smithi194 ceph-mon[33017]: pgmap v7883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:41 smithi033 ceph-mon[27845]: pgmap v7883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:41 smithi096 ceph-mon[32753]: pgmap v7883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:42.807+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:42.807+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:40:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:42.807+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:43 smithi194 ceph-mon[33017]: pgmap v7884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13781.901958 secs 2024-04-26T15:40:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:43 smithi033 ceph-mon[27845]: pgmap v7884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13781.901958 secs 2024-04-26T15:40:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:43 smithi096 ceph-mon[32753]: pgmap v7884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13781.901958 secs 2024-04-26T15:40:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13784.704481 secs 2024-04-26T15:40:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13784.704481 secs 2024-04-26T15:40:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13784.704481 secs 2024-04-26T15:40:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:45 smithi194 ceph-mon[33017]: pgmap v7885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:45 smithi033 ceph-mon[27845]: pgmap v7885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:45 smithi096 ceph-mon[32753]: pgmap v7885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:40:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:47 smithi194 ceph-mon[33017]: pgmap v7886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:47 smithi033 ceph-mon[27845]: pgmap v7886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:47 smithi096 ceph-mon[32753]: pgmap v7886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:40:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13786.902058 secs 2024-04-26T15:40:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13786.902058 secs 2024-04-26T15:40:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13786.902058 secs 2024-04-26T15:40:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:49 smithi033 ceph-mon[27845]: pgmap v7887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13789.704598 secs 2024-04-26T15:40:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:49 smithi096 ceph-mon[32753]: pgmap v7887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13789.704598 secs 2024-04-26T15:40:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:49 smithi194 ceph-mon[33017]: pgmap v7887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13789.704598 secs 2024-04-26T15:40:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:51 smithi033 ceph-mon[27845]: pgmap v7888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:51 smithi096 ceph-mon[32753]: pgmap v7888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:51 smithi194 ceph-mon[33017]: pgmap v7888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:53 smithi033 ceph-mon[27845]: pgmap v7889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13791.902157 secs 2024-04-26T15:40:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:53 smithi096 ceph-mon[32753]: pgmap v7889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13791.902157 secs 2024-04-26T15:40:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:53 smithi194 ceph-mon[33017]: pgmap v7889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13791.902157 secs 2024-04-26T15:40:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13794.704709 secs 2024-04-26T15:40:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13794.704709 secs 2024-04-26T15:40:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13794.704709 secs 2024-04-26T15:40:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:55 smithi033 ceph-mon[27845]: pgmap v7890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:55 smithi096 ceph-mon[32753]: pgmap v7890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:55 smithi194 ceph-mon[33017]: pgmap v7890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:40:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:57.815+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:57.815+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:40:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:40:57.815+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:40:58.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:57 smithi194 ceph-mon[33017]: pgmap v7891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:57 smithi033 ceph-mon[27845]: pgmap v7891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:57 smithi096 ceph-mon[32753]: pgmap v7891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:40:59.170 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13796.902219 secs 2024-04-26T15:40:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13796.902219 secs 2024-04-26T15:40:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13796.902219 secs 2024-04-26T15:41:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:59 smithi033 ceph-mon[27845]: pgmap v7892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:40:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13799.704805 secs 2024-04-26T15:41:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:59 smithi096 ceph-mon[32753]: pgmap v7892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:40:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13799.704805 secs 2024-04-26T15:41:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:59 smithi194 ceph-mon[33017]: pgmap v7892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:40:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13799.704805 secs 2024-04-26T15:41:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:01 smithi033 ceph-mon[27845]: pgmap v7893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:01 smithi096 ceph-mon[32753]: pgmap v7893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:01 smithi194 ceph-mon[33017]: pgmap v7893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:03 smithi033 ceph-mon[27845]: pgmap v7894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13801.902315 secs 2024-04-26T15:41:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:03 smithi096 ceph-mon[32753]: pgmap v7894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13801.902315 secs 2024-04-26T15:41:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:03 smithi194 ceph-mon[33017]: pgmap v7894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13801.902315 secs 2024-04-26T15:41:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13804.704946 secs 2024-04-26T15:41:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13804.704946 secs 2024-04-26T15:41:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13804.704946 secs 2024-04-26T15:41:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:05 smithi096 ceph-mon[32753]: pgmap v7895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:05 smithi194 ceph-mon[33017]: pgmap v7895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:05 smithi033 ceph-mon[27845]: pgmap v7895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:07 smithi194 ceph-mon[33017]: pgmap v7896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:07 smithi033 ceph-mon[27845]: pgmap v7896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:07 smithi096 ceph-mon[32753]: pgmap v7896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13806.902356 secs 2024-04-26T15:41:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13806.902356 secs 2024-04-26T15:41:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13806.902356 secs 2024-04-26T15:41:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:09 smithi194 ceph-mon[33017]: pgmap v7897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13809.705064 secs 2024-04-26T15:41:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:09 smithi033 ceph-mon[27845]: pgmap v7897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13809.705064 secs 2024-04-26T15:41:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:09 smithi096 ceph-mon[32753]: pgmap v7897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13809.705064 secs 2024-04-26T15:41:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:11 smithi033 ceph-mon[27845]: pgmap v7898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:11 smithi194 ceph-mon[33017]: pgmap v7898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:11 smithi096 ceph-mon[32753]: pgmap v7898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:12.821+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:12.821+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:41:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:12.821+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:13 smithi194 ceph-mon[33017]: pgmap v7899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13811.902419 secs 2024-04-26T15:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:13 smithi033 ceph-mon[27845]: pgmap v7899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13811.902419 secs 2024-04-26T15:41:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:13 smithi096 ceph-mon[32753]: pgmap v7899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13811.902419 secs 2024-04-26T15:41:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13814.705210 secs 2024-04-26T15:41:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13814.705210 secs 2024-04-26T15:41:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13814.705210 secs 2024-04-26T15:41:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:15 smithi194 ceph-mon[33017]: pgmap v7900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:15 smithi033 ceph-mon[27845]: pgmap v7900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:15 smithi096 ceph-mon[32753]: pgmap v7900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:17 smithi194 ceph-mon[33017]: pgmap v7901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13816.902503 secs 2024-04-26T15:41:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:17 smithi033 ceph-mon[27845]: pgmap v7901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13816.902503 secs 2024-04-26T15:41:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:17 smithi096 ceph-mon[32753]: pgmap v7901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13816.902503 secs 2024-04-26T15:41:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:19 smithi194 ceph-mon[33017]: pgmap v7902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13819.705340 secs 2024-04-26T15:41:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:19 smithi033 ceph-mon[27845]: pgmap v7902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13819.705340 secs 2024-04-26T15:41:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:19 smithi096 ceph-mon[32753]: pgmap v7902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13819.705340 secs 2024-04-26T15:41:21.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T15:41:21.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T15:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0067000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T15:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0067000 comp rx=0 tx=0).stop 2024-04-26T15:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T15:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T15:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T15:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.749+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T15:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.750+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T15:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:21.751+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0038050 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T15:41:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:21 smithi194 ceph-mon[33017]: pgmap v7903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:21 smithi033 ceph-mon[27845]: pgmap v7903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:21 smithi096 ceph-mon[32753]: pgmap v7903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13821.902581 secs 2024-04-26T15:41:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13821.902581 secs 2024-04-26T15:41:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13821.902581 secs 2024-04-26T15:41:24.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:24.147+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:41:24.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:24.147+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:41:24.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:24.147+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:41:24.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:24.147+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:41:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:24 smithi194 ceph-mon[33017]: pgmap v7904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:24 smithi033 ceph-mon[27845]: pgmap v7904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:24 smithi096 ceph-mon[32753]: pgmap v7904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13824.705429 secs 2024-04-26T15:41:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:25 smithi194 ceph-mon[33017]: pgmap v7905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13824.705429 secs 2024-04-26T15:41:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:25 smithi033 ceph-mon[27845]: pgmap v7905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13824.705429 secs 2024-04-26T15:41:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:25 smithi096 ceph-mon[32753]: pgmap v7905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:27 smithi033 ceph-mon[27845]: pgmap v7906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:41:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:27 smithi096 ceph-mon[32753]: pgmap v7906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:41:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:27 smithi194 ceph-mon[33017]: pgmap v7906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:41:27.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:27.829+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:27.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:27.829+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:27.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:27.829+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13826.902665 secs 2024-04-26T15:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:28 smithi033 ceph-mon[27845]: pgmap v7907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:28 smithi033 ceph-mon[27845]: pgmap v7908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13826.902665 secs 2024-04-26T15:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:28 smithi096 ceph-mon[32753]: pgmap v7907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:28 smithi096 ceph-mon[32753]: pgmap v7908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:41:28.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13826.902665 secs 2024-04-26T15:41:28.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:41:28.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:41:28.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:28 smithi194 ceph-mon[33017]: pgmap v7907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:28.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:28 smithi194 ceph-mon[33017]: pgmap v7908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:28.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:41:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13829.705510 secs 2024-04-26T15:41:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13829.705510 secs 2024-04-26T15:41:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13829.705510 secs 2024-04-26T15:41:30.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:30 smithi096 ceph-mon[32753]: pgmap v7909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:30 smithi194 ceph-mon[33017]: pgmap v7909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:30 smithi033 ceph-mon[27845]: pgmap v7909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:32 smithi194 ceph-mon[33017]: pgmap v7910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:32 smithi033 ceph-mon[27845]: pgmap v7910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:32 smithi096 ceph-mon[32753]: pgmap v7910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13831.902720 secs 2024-04-26T15:41:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13831.902720 secs 2024-04-26T15:41:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13831.902720 secs 2024-04-26T15:41:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13834.705584 secs 2024-04-26T15:41:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:34 smithi033 ceph-mon[27845]: pgmap v7911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13834.705584 secs 2024-04-26T15:41:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:34 smithi096 ceph-mon[32753]: pgmap v7911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13834.705584 secs 2024-04-26T15:41:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:34 smithi194 ceph-mon[33017]: pgmap v7911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:36 smithi033 ceph-mon[27845]: pgmap v7912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:36 smithi096 ceph-mon[32753]: pgmap v7912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:36 smithi194 ceph-mon[33017]: pgmap v7912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:41:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13836.902803 secs 2024-04-26T15:41:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:38 smithi033 ceph-mon[27845]: pgmap v7913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:41:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13836.902803 secs 2024-04-26T15:41:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:38 smithi096 ceph-mon[32753]: pgmap v7913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:41:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13836.902803 secs 2024-04-26T15:41:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:38 smithi194 ceph-mon[33017]: pgmap v7913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:41:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13839.705657 secs 2024-04-26T15:41:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13839.705657 secs 2024-04-26T15:41:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13839.705657 secs 2024-04-26T15:41:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:40 smithi033 ceph-mon[27845]: pgmap v7914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:40 smithi096 ceph-mon[32753]: pgmap v7914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:40 smithi194 ceph-mon[33017]: pgmap v7914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:42.842+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:42.842+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:42.842+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:41:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:42 smithi033 ceph-mon[27845]: pgmap v7915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:42 smithi096 ceph-mon[32753]: pgmap v7915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:42 smithi194 ceph-mon[33017]: pgmap v7915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13841.902887 secs 2024-04-26T15:41:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13841.902887 secs 2024-04-26T15:41:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13841.902887 secs 2024-04-26T15:41:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13844.705716 secs 2024-04-26T15:41:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:44 smithi033 ceph-mon[27845]: pgmap v7916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13844.705716 secs 2024-04-26T15:41:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:44 smithi096 ceph-mon[32753]: pgmap v7916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13844.705716 secs 2024-04-26T15:41:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:44 smithi194 ceph-mon[33017]: pgmap v7916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:47.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:46 smithi033 ceph-mon[27845]: pgmap v7917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:46 smithi096 ceph-mon[32753]: pgmap v7917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:46 smithi194 ceph-mon[33017]: pgmap v7917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:49.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13846.902992 secs 2024-04-26T15:41:49.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:48 smithi033 ceph-mon[27845]: pgmap v7918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13846.902992 secs 2024-04-26T15:41:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:48 smithi096 ceph-mon[32753]: pgmap v7918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13846.902992 secs 2024-04-26T15:41:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:48 smithi194 ceph-mon[33017]: pgmap v7918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13849.705837 secs 2024-04-26T15:41:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13849.705837 secs 2024-04-26T15:41:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13849.705837 secs 2024-04-26T15:41:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:50 smithi096 ceph-mon[32753]: pgmap v7919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:50 smithi194 ceph-mon[33017]: pgmap v7919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:50 smithi033 ceph-mon[27845]: pgmap v7919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:52 smithi194 ceph-mon[33017]: pgmap v7920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:52 smithi033 ceph-mon[27845]: pgmap v7920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:52 smithi096 ceph-mon[32753]: pgmap v7920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13851.903079 secs 2024-04-26T15:41:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13851.903079 secs 2024-04-26T15:41:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13851.903079 secs 2024-04-26T15:41:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13854.705937 secs 2024-04-26T15:41:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:54 smithi194 ceph-mon[33017]: pgmap v7921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13854.705937 secs 2024-04-26T15:41:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:54 smithi033 ceph-mon[27845]: pgmap v7921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13854.705937 secs 2024-04-26T15:41:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:54 smithi096 ceph-mon[32753]: pgmap v7921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:56 smithi194 ceph-mon[33017]: pgmap v7922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:56 smithi033 ceph-mon[27845]: pgmap v7922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:56 smithi096 ceph-mon[32753]: pgmap v7922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:41:57.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:57.858+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:57.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:57.858+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:41:57.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:41:57.858+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:41:59.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13856.903141 secs 2024-04-26T15:41:59.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:58 smithi033 ceph-mon[27845]: pgmap v7923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13856.903141 secs 2024-04-26T15:41:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:58 smithi194 ceph-mon[33017]: pgmap v7923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:41:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13856.903141 secs 2024-04-26T15:41:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:58 smithi096 ceph-mon[32753]: pgmap v7923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:00.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:41:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13859.706063 secs 2024-04-26T15:42:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:41:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13859.706063 secs 2024-04-26T15:42:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:41:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13859.706063 secs 2024-04-26T15:42:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:00 smithi194 ceph-mon[33017]: pgmap v7924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:00 smithi033 ceph-mon[27845]: pgmap v7924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:01.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:00 smithi096 ceph-mon[32753]: pgmap v7924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:02 smithi194 ceph-mon[33017]: pgmap v7925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:02 smithi033 ceph-mon[27845]: pgmap v7925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:02 smithi096 ceph-mon[32753]: pgmap v7925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13861.903203 secs 2024-04-26T15:42:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13861.903203 secs 2024-04-26T15:42:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13861.903203 secs 2024-04-26T15:42:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13864.706188 secs 2024-04-26T15:42:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:04 smithi194 ceph-mon[33017]: pgmap v7926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13864.706188 secs 2024-04-26T15:42:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:04 smithi033 ceph-mon[27845]: pgmap v7926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13864.706188 secs 2024-04-26T15:42:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:04 smithi096 ceph-mon[32753]: pgmap v7926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:06 smithi194 ceph-mon[33017]: pgmap v7927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:06 smithi033 ceph-mon[27845]: pgmap v7927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:06 smithi096 ceph-mon[32753]: pgmap v7927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13866.903301 secs 2024-04-26T15:42:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13866.903301 secs 2024-04-26T15:42:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13866.903301 secs 2024-04-26T15:42:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:09 smithi194 ceph-mon[33017]: pgmap v7928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:09 smithi033 ceph-mon[27845]: pgmap v7928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:09 smithi096 ceph-mon[32753]: pgmap v7928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13869.706323 secs 2024-04-26T15:42:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:10 smithi194 ceph-mon[33017]: pgmap v7929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13869.706323 secs 2024-04-26T15:42:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:10 smithi033 ceph-mon[27845]: pgmap v7929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13869.706323 secs 2024-04-26T15:42:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:10 smithi096 ceph-mon[32753]: pgmap v7929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:12.872+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:12.872+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:12.872+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:42:13.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:12 smithi033 ceph-mon[27845]: pgmap v7930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:12 smithi096 ceph-mon[32753]: pgmap v7930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:12 smithi194 ceph-mon[33017]: pgmap v7930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13871.903385 secs 2024-04-26T15:42:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13871.903385 secs 2024-04-26T15:42:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13871.903385 secs 2024-04-26T15:42:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13874.706434 secs 2024-04-26T15:42:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:14 smithi033 ceph-mon[27845]: pgmap v7931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13874.706434 secs 2024-04-26T15:42:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:14 smithi096 ceph-mon[32753]: pgmap v7931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13874.706434 secs 2024-04-26T15:42:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:14 smithi194 ceph-mon[33017]: pgmap v7931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:16 smithi033 ceph-mon[27845]: pgmap v7932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:16 smithi096 ceph-mon[32753]: pgmap v7932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:16 smithi194 ceph-mon[33017]: pgmap v7932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13876.903469 secs 2024-04-26T15:42:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:18 smithi033 ceph-mon[27845]: pgmap v7933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13876.903469 secs 2024-04-26T15:42:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:18 smithi096 ceph-mon[32753]: pgmap v7933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13876.903469 secs 2024-04-26T15:42:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:18 smithi194 ceph-mon[33017]: pgmap v7933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13879.706562 secs 2024-04-26T15:42:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13879.706562 secs 2024-04-26T15:42:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13879.706562 secs 2024-04-26T15:42:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:20 smithi033 ceph-mon[27845]: pgmap v7934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:20 smithi096 ceph-mon[32753]: pgmap v7934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:20 smithi194 ceph-mon[33017]: pgmap v7934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:22 smithi033 ceph-mon[27845]: pgmap v7935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:22 smithi096 ceph-mon[32753]: pgmap v7935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:22 smithi194 ceph-mon[33017]: pgmap v7935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13881.903553 secs 2024-04-26T15:42:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13881.903553 secs 2024-04-26T15:42:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13881.903553 secs 2024-04-26T15:42:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13884.706682 secs 2024-04-26T15:42:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:24 smithi033 ceph-mon[27845]: pgmap v7936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13884.706682 secs 2024-04-26T15:42:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:24 smithi096 ceph-mon[32753]: pgmap v7936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13884.706682 secs 2024-04-26T15:42:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:24 smithi194 ceph-mon[33017]: pgmap v7936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:26 smithi096 ceph-mon[32753]: pgmap v7937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:26 smithi194 ceph-mon[33017]: pgmap v7937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:26 smithi033 ceph-mon[27845]: pgmap v7937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:27.884+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:27.884+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:42:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:27.884+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13886.903637 secs 2024-04-26T15:42:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:28 smithi096 ceph-mon[32753]: pgmap v7938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:42:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13886.903637 secs 2024-04-26T15:42:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:28 smithi194 ceph-mon[33017]: pgmap v7938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:42:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13886.903637 secs 2024-04-26T15:42:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:28 smithi033 ceph-mon[27845]: pgmap v7938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:42:30.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13889.706774 secs 2024-04-26T15:42:30.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:42:30.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:42:30.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:42:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13889.706774 secs 2024-04-26T15:42:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:42:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:42:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:42:30.353 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13889.706774 secs 2024-04-26T15:42:30.353 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:42:30.353 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:42:30.353 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:42:31.310 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:30 smithi096 ceph-mon[32753]: pgmap v7939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:31.310 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:30 smithi096 ceph-mon[32753]: pgmap v7940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:42:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:30 smithi194 ceph-mon[33017]: pgmap v7939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:30 smithi194 ceph-mon[33017]: pgmap v7940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:42:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:30 smithi033 ceph-mon[27845]: pgmap v7939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:30 smithi033 ceph-mon[27845]: pgmap v7940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:42:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:32 smithi194 ceph-mon[33017]: pgmap v7941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:32 smithi033 ceph-mon[27845]: pgmap v7941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:32 smithi096 ceph-mon[32753]: pgmap v7941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13891.903722 secs 2024-04-26T15:42:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13891.903722 secs 2024-04-26T15:42:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13891.903722 secs 2024-04-26T15:42:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13894.706872 secs 2024-04-26T15:42:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:34 smithi194 ceph-mon[33017]: pgmap v7942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13894.706872 secs 2024-04-26T15:42:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:34 smithi033 ceph-mon[27845]: pgmap v7942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13894.706872 secs 2024-04-26T15:42:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:34 smithi096 ceph-mon[32753]: pgmap v7942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:36 smithi194 ceph-mon[33017]: pgmap v7943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:36 smithi033 ceph-mon[27845]: pgmap v7943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:36 smithi096 ceph-mon[32753]: pgmap v7943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:38 smithi194 ceph-mon[33017]: pgmap v7944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13896.903807 secs 2024-04-26T15:42:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:38 smithi033 ceph-mon[27845]: pgmap v7944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13896.903807 secs 2024-04-26T15:42:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:38 smithi096 ceph-mon[32753]: pgmap v7944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13896.903807 secs 2024-04-26T15:42:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13899.707006 secs 2024-04-26T15:42:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13899.707006 secs 2024-04-26T15:42:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13899.707006 secs 2024-04-26T15:42:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:40 smithi096 ceph-mon[32753]: pgmap v7945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:42:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:40 smithi194 ceph-mon[33017]: pgmap v7945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:42:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:40 smithi033 ceph-mon[27845]: pgmap v7945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:42:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:42.900+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:42.900+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:42:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:42.900+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:42 smithi194 ceph-mon[33017]: pgmap v7946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:42:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13901.903894 secs 2024-04-26T15:42:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:42 smithi033 ceph-mon[27845]: pgmap v7946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:42:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13901.903894 secs 2024-04-26T15:42:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:42 smithi096 ceph-mon[32753]: pgmap v7946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:42:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13901.903894 secs 2024-04-26T15:42:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13904.707155 secs 2024-04-26T15:42:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:45 smithi194 ceph-mon[33017]: pgmap v7947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13904.707155 secs 2024-04-26T15:42:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:45 smithi033 ceph-mon[27845]: pgmap v7947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13904.707155 secs 2024-04-26T15:42:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:44 smithi096 ceph-mon[32753]: pgmap v7947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:47 smithi194 ceph-mon[33017]: pgmap v7948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:47 smithi033 ceph-mon[27845]: pgmap v7948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:47 smithi096 ceph-mon[32753]: pgmap v7948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13906.903994 secs 2024-04-26T15:42:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13906.903994 secs 2024-04-26T15:42:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13906.903994 secs 2024-04-26T15:42:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:49 smithi194 ceph-mon[33017]: pgmap v7949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:49 smithi033 ceph-mon[27845]: pgmap v7949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:49 smithi096 ceph-mon[32753]: pgmap v7949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13909.707296 secs 2024-04-26T15:42:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13909.707296 secs 2024-04-26T15:42:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13909.707296 secs 2024-04-26T15:42:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:51 smithi194 ceph-mon[33017]: pgmap v7950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:51 smithi033 ceph-mon[27845]: pgmap v7950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:51 smithi096 ceph-mon[32753]: pgmap v7950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:42:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:52 smithi194 ceph-mon[33017]: pgmap v7951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:42:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:52 smithi033 ceph-mon[27845]: pgmap v7951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:42:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:52 smithi096 ceph-mon[32753]: pgmap v7951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:42:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13911.904081 secs 2024-04-26T15:42:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13911.904081 secs 2024-04-26T15:42:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13911.904081 secs 2024-04-26T15:42:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:54 smithi194 ceph-mon[33017]: pgmap v7952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:54 smithi033 ceph-mon[27845]: pgmap v7952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:54 smithi096 ceph-mon[32753]: pgmap v7952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13914.707413 secs 2024-04-26T15:42:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13914.707413 secs 2024-04-26T15:42:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13914.707413 secs 2024-04-26T15:42:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:56 smithi194 ceph-mon[33017]: pgmap v7953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:56 smithi033 ceph-mon[27845]: pgmap v7953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:56 smithi096 ceph-mon[32753]: pgmap v7953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:42:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:57.915+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:57.915+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:42:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:42:57.915+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:42:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:58 smithi194 ceph-mon[33017]: pgmap v7954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13916.904171 secs 2024-04-26T15:42:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:58 smithi033 ceph-mon[27845]: pgmap v7954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13916.904171 secs 2024-04-26T15:42:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:58 smithi096 ceph-mon[32753]: pgmap v7954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:42:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13916.904171 secs 2024-04-26T15:42:59.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13919.707529 secs 2024-04-26T15:42:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13919.707529 secs 2024-04-26T15:42:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13919.707529 secs 2024-04-26T15:43:00.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:00 smithi096 ceph-mon[32753]: pgmap v7955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:00 smithi194 ceph-mon[33017]: pgmap v7955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:00.910 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:00 smithi033 ceph-mon[27845]: pgmap v7955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:02 smithi194 ceph-mon[33017]: pgmap v7956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:02.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:02 smithi033 ceph-mon[27845]: pgmap v7956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:02 smithi096 ceph-mon[32753]: pgmap v7956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13921.904277 secs 2024-04-26T15:43:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13921.904277 secs 2024-04-26T15:43:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13921.904277 secs 2024-04-26T15:43:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13924.707642 secs 2024-04-26T15:43:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:04 smithi194 ceph-mon[33017]: pgmap v7957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13924.707642 secs 2024-04-26T15:43:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:04 smithi033 ceph-mon[27845]: pgmap v7957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13924.707642 secs 2024-04-26T15:43:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:04 smithi096 ceph-mon[32753]: pgmap v7957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:06 smithi194 ceph-mon[33017]: pgmap v7958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:06 smithi033 ceph-mon[27845]: pgmap v7958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:06 smithi096 ceph-mon[32753]: pgmap v7958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:08 smithi194 ceph-mon[33017]: pgmap v7959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13926.904322 secs 2024-04-26T15:43:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:08 smithi033 ceph-mon[27845]: pgmap v7959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13926.904322 secs 2024-04-26T15:43:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:08 smithi096 ceph-mon[32753]: pgmap v7959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13926.904322 secs 2024-04-26T15:43:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13929.707739 secs 2024-04-26T15:43:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13929.707739 secs 2024-04-26T15:43:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13929.707739 secs 2024-04-26T15:43:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:10 smithi194 ceph-mon[33017]: pgmap v7960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:10.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:10 smithi033 ceph-mon[27845]: pgmap v7960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:10 smithi096 ceph-mon[32753]: pgmap v7960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:12 smithi194 ceph-mon[33017]: pgmap v7961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:12.917+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:12.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:12.917+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:12.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:12.917+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:43:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:12 smithi033 ceph-mon[27845]: pgmap v7961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:12 smithi096 ceph-mon[32753]: pgmap v7961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13931.904411 secs 2024-04-26T15:43:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13931.904411 secs 2024-04-26T15:43:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13931.904411 secs 2024-04-26T15:43:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13934.707874 secs 2024-04-26T15:43:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:14 smithi033 ceph-mon[27845]: pgmap v7962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13934.707874 secs 2024-04-26T15:43:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:14 smithi096 ceph-mon[32753]: pgmap v7962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13934.707874 secs 2024-04-26T15:43:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:14 smithi194 ceph-mon[33017]: pgmap v7962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:16.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:16 smithi033 ceph-mon[27845]: pgmap v7963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:16 smithi096 ceph-mon[32753]: pgmap v7963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:16 smithi194 ceph-mon[33017]: pgmap v7963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:18 smithi033 ceph-mon[27845]: pgmap v7964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13936.904506 secs 2024-04-26T15:43:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:18 smithi096 ceph-mon[32753]: pgmap v7964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13936.904506 secs 2024-04-26T15:43:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:18 smithi194 ceph-mon[33017]: pgmap v7964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13936.904506 secs 2024-04-26T15:43:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13939.707970 secs 2024-04-26T15:43:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13939.707970 secs 2024-04-26T15:43:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13939.707970 secs 2024-04-26T15:43:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:20 smithi033 ceph-mon[27845]: pgmap v7965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:20 smithi096 ceph-mon[32753]: pgmap v7965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:20 smithi194 ceph-mon[33017]: pgmap v7965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:22 smithi033 ceph-mon[27845]: pgmap v7966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:22 smithi096 ceph-mon[32753]: pgmap v7966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:22 smithi194 ceph-mon[33017]: pgmap v7966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:43:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13941.904610 secs 2024-04-26T15:43:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13941.904610 secs 2024-04-26T15:43:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13941.904610 secs 2024-04-26T15:43:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13944.708085 secs 2024-04-26T15:43:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:24 smithi033 ceph-mon[27845]: pgmap v7967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13944.708085 secs 2024-04-26T15:43:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:24 smithi096 ceph-mon[32753]: pgmap v7967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13944.708085 secs 2024-04-26T15:43:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:24 smithi194 ceph-mon[33017]: pgmap v7967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:26.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:26 smithi033 ceph-mon[27845]: pgmap v7968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:26 smithi096 ceph-mon[32753]: pgmap v7968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:26 smithi194 ceph-mon[33017]: pgmap v7968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:27.933+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:27.933+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:43:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:27.933+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:28 smithi096 ceph-mon[32753]: pgmap v7969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13946.904691 secs 2024-04-26T15:43:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:28 smithi194 ceph-mon[33017]: pgmap v7969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13946.904691 secs 2024-04-26T15:43:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:28 smithi033 ceph-mon[27845]: pgmap v7969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13946.904691 secs 2024-04-26T15:43:30.003 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13949.708219 secs 2024-04-26T15:43:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13949.708219 secs 2024-04-26T15:43:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13949.708219 secs 2024-04-26T15:43:31.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:30 smithi096 ceph-mon[32753]: pgmap v7970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:31.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:43:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:30 smithi194 ceph-mon[33017]: pgmap v7970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:43:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:30 smithi033 ceph-mon[27845]: pgmap v7970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:43:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:43:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:43:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:43:32.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:43:32.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:43:32.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:43:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:43:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:43:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:43:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:32 smithi194 ceph-mon[33017]: pgmap v7971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-26T15:43:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:32 smithi194 ceph-mon[33017]: pgmap v7972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:32 smithi033 ceph-mon[27845]: pgmap v7971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-26T15:43:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:32 smithi033 ceph-mon[27845]: pgmap v7972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:32 smithi096 ceph-mon[32753]: pgmap v7971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-26T15:43:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:32 smithi096 ceph-mon[32753]: pgmap v7972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13951.904780 secs 2024-04-26T15:43:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13951.904780 secs 2024-04-26T15:43:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13951.904780 secs 2024-04-26T15:43:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:34 smithi194 ceph-mon[33017]: pgmap v7973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13954.708338 secs 2024-04-26T15:43:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:34 smithi033 ceph-mon[27845]: pgmap v7973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13954.708338 secs 2024-04-26T15:43:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:34 smithi096 ceph-mon[32753]: pgmap v7973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13954.708338 secs 2024-04-26T15:43:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:36 smithi194 ceph-mon[33017]: pgmap v7974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:36 smithi033 ceph-mon[27845]: pgmap v7974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:36 smithi096 ceph-mon[32753]: pgmap v7974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:38 smithi194 ceph-mon[33017]: pgmap v7975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13956.904864 secs 2024-04-26T15:43:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:38 smithi033 ceph-mon[27845]: pgmap v7975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13956.904864 secs 2024-04-26T15:43:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:38 smithi096 ceph-mon[32753]: pgmap v7975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:43:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13956.904864 secs 2024-04-26T15:43:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13959.708419 secs 2024-04-26T15:43:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13959.708419 secs 2024-04-26T15:43:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13959.708419 secs 2024-04-26T15:43:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:40 smithi194 ceph-mon[33017]: pgmap v7976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T15:43:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:40 smithi033 ceph-mon[27845]: pgmap v7976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T15:43:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:40 smithi096 ceph-mon[32753]: pgmap v7976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T15:43:42.943 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:42.944+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:42.943 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:42.944+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:43:42.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:42.944+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:42 smithi194 ceph-mon[33017]: pgmap v7977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:43:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:42 smithi033 ceph-mon[27845]: pgmap v7977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:43:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:42 smithi096 ceph-mon[32753]: pgmap v7977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:43:44.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13961.904953 secs 2024-04-26T15:43:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13961.904953 secs 2024-04-26T15:43:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13961.904953 secs 2024-04-26T15:43:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:44 smithi194 ceph-mon[33017]: pgmap v7978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13964.708539 secs 2024-04-26T15:43:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:44 smithi033 ceph-mon[27845]: pgmap v7978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13964.708539 secs 2024-04-26T15:43:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:44 smithi096 ceph-mon[32753]: pgmap v7978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13964.708539 secs 2024-04-26T15:43:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:46 smithi194 ceph-mon[33017]: pgmap v7979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:47.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:46 smithi033 ceph-mon[27845]: pgmap v7979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:46 smithi096 ceph-mon[32753]: pgmap v7979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:48 smithi194 ceph-mon[33017]: pgmap v7980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13966.905041 secs 2024-04-26T15:43:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:48 smithi033 ceph-mon[27845]: pgmap v7980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13966.905041 secs 2024-04-26T15:43:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:48 smithi096 ceph-mon[32753]: pgmap v7980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13966.905041 secs 2024-04-26T15:43:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13969.708604 secs 2024-04-26T15:43:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13969.708604 secs 2024-04-26T15:43:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13969.708604 secs 2024-04-26T15:43:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:50 smithi194 ceph-mon[33017]: pgmap v7981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:51.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:50 smithi033 ceph-mon[27845]: pgmap v7981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:50 smithi096 ceph-mon[32753]: pgmap v7981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:52 smithi194 ceph-mon[33017]: pgmap v7982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:53.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:52 smithi033 ceph-mon[27845]: pgmap v7982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:52 smithi096 ceph-mon[32753]: pgmap v7982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13971.905126 secs 2024-04-26T15:43:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13971.905126 secs 2024-04-26T15:43:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13971.905126 secs 2024-04-26T15:43:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:54 smithi194 ceph-mon[33017]: pgmap v7983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13974.708720 secs 2024-04-26T15:43:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:54 smithi033 ceph-mon[27845]: pgmap v7983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13974.708720 secs 2024-04-26T15:43:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:54 smithi096 ceph-mon[32753]: pgmap v7983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13974.708720 secs 2024-04-26T15:43:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:56 smithi194 ceph-mon[33017]: pgmap v7984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:56 smithi033 ceph-mon[27845]: pgmap v7984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:56 smithi096 ceph-mon[32753]: pgmap v7984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:43:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:57.947+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:57.947+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:43:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:43:57.947+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:43:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:58 smithi194 ceph-mon[33017]: pgmap v7985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13976.905214 secs 2024-04-26T15:43:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:58 smithi033 ceph-mon[27845]: pgmap v7985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13976.905214 secs 2024-04-26T15:43:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:58 smithi096 ceph-mon[32753]: pgmap v7985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:43:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13976.905214 secs 2024-04-26T15:44:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13979.708855 secs 2024-04-26T15:44:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13979.708855 secs 2024-04-26T15:44:00.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13979.708855 secs 2024-04-26T15:44:01.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:00 smithi096 ceph-mon[32753]: pgmap v7986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:00 smithi194 ceph-mon[33017]: pgmap v7986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:01.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:00 smithi033 ceph-mon[27845]: pgmap v7986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:02 smithi194 ceph-mon[33017]: pgmap v7987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:02 smithi033 ceph-mon[27845]: pgmap v7987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:03.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:02 smithi096 ceph-mon[32753]: pgmap v7987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13981.905313 secs 2024-04-26T15:44:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13981.905313 secs 2024-04-26T15:44:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13981.905313 secs 2024-04-26T15:44:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:04 smithi194 ceph-mon[33017]: pgmap v7988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13984.708970 secs 2024-04-26T15:44:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:04 smithi033 ceph-mon[27845]: pgmap v7988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13984.708970 secs 2024-04-26T15:44:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:04 smithi096 ceph-mon[32753]: pgmap v7988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13984.708970 secs 2024-04-26T15:44:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:06 smithi194 ceph-mon[33017]: pgmap v7989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:06 smithi033 ceph-mon[27845]: pgmap v7989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:07.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:06 smithi096 ceph-mon[32753]: pgmap v7989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:08 smithi033 ceph-mon[27845]: pgmap v7990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13986.905397 secs 2024-04-26T15:44:09.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:08 smithi096 ceph-mon[32753]: pgmap v7990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13986.905397 secs 2024-04-26T15:44:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:08 smithi194 ceph-mon[33017]: pgmap v7990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13986.905397 secs 2024-04-26T15:44:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13989.709090 secs 2024-04-26T15:44:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13989.709090 secs 2024-04-26T15:44:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13989.709090 secs 2024-04-26T15:44:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:10 smithi033 ceph-mon[27845]: pgmap v7991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:11.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:10 smithi096 ceph-mon[32753]: pgmap v7991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:10 smithi194 ceph-mon[33017]: pgmap v7991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:12.960+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:12.960+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:12.960+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:44:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:12 smithi033 ceph-mon[27845]: pgmap v7992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:13.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:12 smithi096 ceph-mon[32753]: pgmap v7992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:12 smithi194 ceph-mon[33017]: pgmap v7992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13991.905457 secs 2024-04-26T15:44:14.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13991.905457 secs 2024-04-26T15:44:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13991.905457 secs 2024-04-26T15:44:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:14 smithi033 ceph-mon[27845]: pgmap v7993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13994.709224 secs 2024-04-26T15:44:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:14 smithi096 ceph-mon[32753]: pgmap v7993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13994.709224 secs 2024-04-26T15:44:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:14 smithi194 ceph-mon[33017]: pgmap v7993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13994.709224 secs 2024-04-26T15:44:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:16 smithi033 ceph-mon[27845]: pgmap v7994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:16 smithi096 ceph-mon[32753]: pgmap v7994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:16 smithi194 ceph-mon[33017]: pgmap v7994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:18 smithi033 ceph-mon[27845]: pgmap v7995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 13996.905544 secs 2024-04-26T15:44:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:18 smithi096 ceph-mon[32753]: pgmap v7995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 13996.905544 secs 2024-04-26T15:44:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:18 smithi194 ceph-mon[33017]: pgmap v7995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 13996.905544 secs 2024-04-26T15:44:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 13999.709303 secs 2024-04-26T15:44:20.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 13999.709303 secs 2024-04-26T15:44:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 13999.709303 secs 2024-04-26T15:44:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:20 smithi033 ceph-mon[27845]: pgmap v7996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:20 smithi096 ceph-mon[32753]: pgmap v7996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:20 smithi194 ceph-mon[33017]: pgmap v7996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:22 smithi033 ceph-mon[27845]: pgmap v7997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:23.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:22 smithi096 ceph-mon[32753]: pgmap v7997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:22 smithi194 ceph-mon[33017]: pgmap v7997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:24.179+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:44:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:24.179+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0063a0 con 0x564385821900 2024-04-26T15:44:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:24.179+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:44:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:24.179+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:44:24.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14001.905627 secs 2024-04-26T15:44:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14001.905627 secs 2024-04-26T15:44:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14001.905627 secs 2024-04-26T15:44:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:24 smithi096 ceph-mon[32753]: pgmap v7998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14004.709417 secs 2024-04-26T15:44:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:24 smithi194 ceph-mon[33017]: pgmap v7998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14004.709417 secs 2024-04-26T15:44:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:24 smithi033 ceph-mon[27845]: pgmap v7998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14004.709417 secs 2024-04-26T15:44:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:26 smithi194 ceph-mon[33017]: pgmap v7999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:26 smithi033 ceph-mon[27845]: pgmap v7999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:26 smithi096 ceph-mon[32753]: pgmap v7999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:27.968+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:27.968+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:44:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:27.968+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:28 smithi194 ceph-mon[33017]: pgmap v8000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14006.905707 secs 2024-04-26T15:44:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:28 smithi033 ceph-mon[27845]: pgmap v8000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14006.905707 secs 2024-04-26T15:44:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:28 smithi096 ceph-mon[32753]: pgmap v8000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14006.905707 secs 2024-04-26T15:44:30.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14009.709539 secs 2024-04-26T15:44:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14009.709539 secs 2024-04-26T15:44:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14009.709539 secs 2024-04-26T15:44:31.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:30 smithi096 ceph-mon[32753]: pgmap v8001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:30 smithi194 ceph-mon[33017]: pgmap v8001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:30 smithi033 ceph-mon[27845]: pgmap v8001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:44:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:44:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:44:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:32 smithi194 ceph-mon[33017]: pgmap v8002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:44:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:44:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:44:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:32 smithi033 ceph-mon[27845]: pgmap v8002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:44:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:44:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:44:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:32 smithi096 ceph-mon[32753]: pgmap v8002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:44:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:44:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:44:34.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14011.905788 secs 2024-04-26T15:44:34.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:33 smithi033 ceph-mon[27845]: pgmap v8003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:34.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:33 smithi033 ceph-mon[27845]: pgmap v8004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14011.905788 secs 2024-04-26T15:44:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:33 smithi194 ceph-mon[33017]: pgmap v8003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:33 smithi194 ceph-mon[33017]: pgmap v8004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14011.905788 secs 2024-04-26T15:44:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:33 smithi096 ceph-mon[32753]: pgmap v8003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:33 smithi096 ceph-mon[32753]: pgmap v8004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14014.709655 secs 2024-04-26T15:44:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14014.709655 secs 2024-04-26T15:44:35.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14014.709655 secs 2024-04-26T15:44:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:35 smithi194 ceph-mon[33017]: pgmap v8005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:35 smithi033 ceph-mon[27845]: pgmap v8005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:35 smithi096 ceph-mon[32753]: pgmap v8005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:38 smithi194 ceph-mon[33017]: pgmap v8006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14016.905861 secs 2024-04-26T15:44:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:38 smithi033 ceph-mon[27845]: pgmap v8006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14016.905861 secs 2024-04-26T15:44:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:38 smithi096 ceph-mon[32753]: pgmap v8006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14016.905861 secs 2024-04-26T15:44:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14019.709805 secs 2024-04-26T15:44:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:40 smithi194 ceph-mon[33017]: pgmap v8007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14019.709805 secs 2024-04-26T15:44:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:40 smithi033 ceph-mon[27845]: pgmap v8007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14019.709805 secs 2024-04-26T15:44:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:40 smithi096 ceph-mon[32753]: pgmap v8007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:42 smithi194 ceph-mon[33017]: pgmap v8008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:42 smithi033 ceph-mon[27845]: pgmap v8008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:42 smithi096 ceph-mon[32753]: pgmap v8008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:44:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:42.974+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:42.974+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:42.974+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:44:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14021.905949 secs 2024-04-26T15:44:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14021.905949 secs 2024-04-26T15:44:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14021.905949 secs 2024-04-26T15:44:44.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:44 smithi033 ceph-mon[27845]: pgmap v8009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:44:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:44 smithi194 ceph-mon[33017]: pgmap v8009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:44:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:44 smithi096 ceph-mon[32753]: pgmap v8009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:44:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14024.709919 secs 2024-04-26T15:44:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14024.709919 secs 2024-04-26T15:44:45.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14024.709919 secs 2024-04-26T15:44:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:46 smithi194 ceph-mon[33017]: pgmap v8010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:46 smithi033 ceph-mon[27845]: pgmap v8010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:46 smithi096 ceph-mon[32753]: pgmap v8010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:48 smithi194 ceph-mon[33017]: pgmap v8011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14026.906055 secs 2024-04-26T15:44:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:48 smithi033 ceph-mon[27845]: pgmap v8011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14026.906055 secs 2024-04-26T15:44:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:48 smithi096 ceph-mon[32753]: pgmap v8011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14026.906055 secs 2024-04-26T15:44:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:49 smithi194 ceph-mon[33017]: pgmap v8012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:49 smithi033 ceph-mon[27845]: pgmap v8012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:49 smithi096 ceph-mon[32753]: pgmap v8012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14029.709985 secs 2024-04-26T15:44:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14029.709985 secs 2024-04-26T15:44:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14029.709985 secs 2024-04-26T15:44:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:51 smithi194 ceph-mon[33017]: pgmap v8013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:51 smithi033 ceph-mon[27845]: pgmap v8013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:51 smithi096 ceph-mon[32753]: pgmap v8013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14031.906138 secs 2024-04-26T15:44:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:53 smithi194 ceph-mon[33017]: pgmap v8014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14031.906138 secs 2024-04-26T15:44:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:53 smithi033 ceph-mon[27845]: pgmap v8014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:54.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14031.906138 secs 2024-04-26T15:44:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:53 smithi096 ceph-mon[32753]: pgmap v8014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14034.710099 secs 2024-04-26T15:44:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14034.710099 secs 2024-04-26T15:44:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14034.710099 secs 2024-04-26T15:44:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:55 smithi194 ceph-mon[33017]: pgmap v8015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:55 smithi033 ceph-mon[27845]: pgmap v8015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:56.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:55 smithi096 ceph-mon[32753]: pgmap v8015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:44:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:57.990+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:57.990+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:44:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:44:57.990+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:44:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:57 smithi194 ceph-mon[33017]: pgmap v8016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:57 smithi033 ceph-mon[27845]: pgmap v8016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:58.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:57 smithi096 ceph-mon[32753]: pgmap v8016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:44:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14036.906237 secs 2024-04-26T15:44:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14036.906237 secs 2024-04-26T15:44:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14036.906237 secs 2024-04-26T15:45:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14039.710180 secs 2024-04-26T15:45:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:44:59 smithi194 ceph-mon[33017]: pgmap v8017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14039.710180 secs 2024-04-26T15:45:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:44:59 smithi033 ceph-mon[27845]: pgmap v8017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14039.710180 secs 2024-04-26T15:45:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:44:59 smithi096 ceph-mon[32753]: pgmap v8017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:01 smithi194 ceph-mon[33017]: pgmap v8018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:01 smithi033 ceph-mon[27845]: pgmap v8018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:02.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:01 smithi096 ceph-mon[32753]: pgmap v8018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14041.906314 secs 2024-04-26T15:45:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:03 smithi194 ceph-mon[33017]: pgmap v8019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14041.906314 secs 2024-04-26T15:45:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:03 smithi033 ceph-mon[27845]: pgmap v8019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14041.906314 secs 2024-04-26T15:45:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:03 smithi096 ceph-mon[32753]: pgmap v8019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14044.710296 secs 2024-04-26T15:45:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14044.710296 secs 2024-04-26T15:45:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14044.710296 secs 2024-04-26T15:45:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:05 smithi194 ceph-mon[33017]: pgmap v8020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:05 smithi033 ceph-mon[27845]: pgmap v8020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:05 smithi096 ceph-mon[32753]: pgmap v8020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:07 smithi033 ceph-mon[27845]: pgmap v8021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:08.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:07 smithi096 ceph-mon[32753]: pgmap v8021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:07 smithi194 ceph-mon[33017]: pgmap v8021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14046.906396 secs 2024-04-26T15:45:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14046.906396 secs 2024-04-26T15:45:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14046.906396 secs 2024-04-26T15:45:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14049.710406 secs 2024-04-26T15:45:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:09 smithi033 ceph-mon[27845]: pgmap v8022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:10.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14049.710406 secs 2024-04-26T15:45:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:09 smithi096 ceph-mon[32753]: pgmap v8022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14049.710406 secs 2024-04-26T15:45:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:09 smithi194 ceph-mon[33017]: pgmap v8022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:11 smithi033 ceph-mon[27845]: pgmap v8023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:12.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:11 smithi096 ceph-mon[32753]: pgmap v8023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:11 smithi194 ceph-mon[33017]: pgmap v8023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:13.004+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:13.004+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:13.004+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:45:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14051.906479 secs 2024-04-26T15:45:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:13 smithi033 ceph-mon[27845]: pgmap v8024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14051.906479 secs 2024-04-26T15:45:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:13 smithi096 ceph-mon[32753]: pgmap v8024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14051.906479 secs 2024-04-26T15:45:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:13 smithi194 ceph-mon[33017]: pgmap v8024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14054.710526 secs 2024-04-26T15:45:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14054.710526 secs 2024-04-26T15:45:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14054.710526 secs 2024-04-26T15:45:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:15 smithi194 ceph-mon[33017]: pgmap v8025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:15 smithi033 ceph-mon[27845]: pgmap v8025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:15.680 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:15 smithi096 ceph-mon[32753]: pgmap v8025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:17 smithi194 ceph-mon[33017]: pgmap v8026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:17 smithi033 ceph-mon[27845]: pgmap v8026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:17 smithi096 ceph-mon[32753]: pgmap v8026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14056.906567 secs 2024-04-26T15:45:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14056.906567 secs 2024-04-26T15:45:19.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14056.906567 secs 2024-04-26T15:45:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14059.710635 secs 2024-04-26T15:45:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:19 smithi194 ceph-mon[33017]: pgmap v8027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14059.710635 secs 2024-04-26T15:45:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:19 smithi033 ceph-mon[27845]: pgmap v8027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:20.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14059.710635 secs 2024-04-26T15:45:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:19 smithi096 ceph-mon[32753]: pgmap v8027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:21 smithi194 ceph-mon[33017]: pgmap v8028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:21 smithi033 ceph-mon[27845]: pgmap v8028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:22.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:21 smithi096 ceph-mon[32753]: pgmap v8028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14061.906651 secs 2024-04-26T15:45:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:23 smithi194 ceph-mon[33017]: pgmap v8029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14061.906651 secs 2024-04-26T15:45:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:23 smithi033 ceph-mon[27845]: pgmap v8029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14061.906651 secs 2024-04-26T15:45:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:23 smithi096 ceph-mon[32753]: pgmap v8029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14064.710750 secs 2024-04-26T15:45:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14064.710750 secs 2024-04-26T15:45:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14064.710750 secs 2024-04-26T15:45:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:25 smithi194 ceph-mon[33017]: pgmap v8030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:25 smithi033 ceph-mon[27845]: pgmap v8030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:26.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:25 smithi096 ceph-mon[32753]: pgmap v8030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:28.015+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:28.015+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:28.015+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:45:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:27 smithi194 ceph-mon[33017]: pgmap v8031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:27 smithi033 ceph-mon[27845]: pgmap v8031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:28.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:27 smithi096 ceph-mon[32753]: pgmap v8031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14066.906739 secs 2024-04-26T15:45:29.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14066.906739 secs 2024-04-26T15:45:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14066.906739 secs 2024-04-26T15:45:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14069.710842 secs 2024-04-26T15:45:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:29 smithi033 ceph-mon[27845]: pgmap v8032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:30.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14069.710842 secs 2024-04-26T15:45:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:29 smithi096 ceph-mon[32753]: pgmap v8032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:30.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14069.710842 secs 2024-04-26T15:45:30.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:29 smithi194 ceph-mon[33017]: pgmap v8032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:31 smithi033 ceph-mon[27845]: pgmap v8033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:31 smithi096 ceph-mon[32753]: pgmap v8033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:31 smithi194 ceph-mon[33017]: pgmap v8033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:45:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14071.906821 secs 2024-04-26T15:45:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:33 smithi033 ceph-mon[27845]: pgmap v8034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:45:34.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14071.906821 secs 2024-04-26T15:45:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:33 smithi096 ceph-mon[32753]: pgmap v8034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:45:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14071.906821 secs 2024-04-26T15:45:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:33 smithi194 ceph-mon[33017]: pgmap v8034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:45:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14074.710928 secs 2024-04-26T15:45:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:45:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:45:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:45:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14074.710928 secs 2024-04-26T15:45:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:45:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:45:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:45:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14074.710928 secs 2024-04-26T15:45:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:45:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:45:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:45:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:35 smithi033 ceph-mon[27845]: pgmap v8035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:45:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:35 smithi033 ceph-mon[27845]: pgmap v8036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:36.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:35 smithi096 ceph-mon[32753]: pgmap v8035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:45:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:35 smithi096 ceph-mon[32753]: pgmap v8036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:35 smithi194 ceph-mon[33017]: pgmap v8035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:45:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:35 smithi194 ceph-mon[33017]: pgmap v8036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:37 smithi033 ceph-mon[27845]: pgmap v8037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:38.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:37 smithi096 ceph-mon[32753]: pgmap v8037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:37 smithi194 ceph-mon[33017]: pgmap v8037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14076.906911 secs 2024-04-26T15:45:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14076.906911 secs 2024-04-26T15:45:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14076.906911 secs 2024-04-26T15:45:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14079.711040 secs 2024-04-26T15:45:40.324 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:39 smithi033 ceph-mon[27845]: pgmap v8038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:40.325 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14079.711040 secs 2024-04-26T15:45:40.325 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:39 smithi096 ceph-mon[32753]: pgmap v8038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14079.711040 secs 2024-04-26T15:45:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:39 smithi194 ceph-mon[33017]: pgmap v8038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:41 smithi033 ceph-mon[27845]: pgmap v8039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:41 smithi096 ceph-mon[32753]: pgmap v8039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:41 smithi194 ceph-mon[33017]: pgmap v8039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:43.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:43.032+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:43.032+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:45:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:43 smithi033 ceph-mon[27845]: pgmap v8040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14081.907010 secs 2024-04-26T15:45:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:43 smithi096 ceph-mon[32753]: pgmap v8040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14081.907010 secs 2024-04-26T15:45:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:43 smithi194 ceph-mon[33017]: pgmap v8040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:45:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14081.907010 secs 2024-04-26T15:45:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14084.711156 secs 2024-04-26T15:45:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14084.711156 secs 2024-04-26T15:45:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14084.711156 secs 2024-04-26T15:45:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:45 smithi096 ceph-mon[32753]: pgmap v8041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:45:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:45 smithi194 ceph-mon[33017]: pgmap v8041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:45:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:45 smithi033 ceph-mon[27845]: pgmap v8041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:45:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:47 smithi194 ceph-mon[33017]: pgmap v8042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:47 smithi033 ceph-mon[27845]: pgmap v8042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:48.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:47 smithi096 ceph-mon[32753]: pgmap v8042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14086.907092 secs 2024-04-26T15:45:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14086.907092 secs 2024-04-26T15:45:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14086.907092 secs 2024-04-26T15:45:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14089.711295 secs 2024-04-26T15:45:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:49 smithi194 ceph-mon[33017]: pgmap v8043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14089.711295 secs 2024-04-26T15:45:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:49 smithi033 ceph-mon[27845]: pgmap v8043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:50.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14089.711295 secs 2024-04-26T15:45:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:49 smithi096 ceph-mon[32753]: pgmap v8043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:51 smithi033 ceph-mon[27845]: pgmap v8044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:45:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:51 smithi194 ceph-mon[33017]: pgmap v8044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:45:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:51 smithi096 ceph-mon[32753]: pgmap v8044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:45:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:53 smithi194 ceph-mon[33017]: pgmap v8045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14091.907179 secs 2024-04-26T15:45:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:53 smithi033 ceph-mon[27845]: pgmap v8045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14091.907179 secs 2024-04-26T15:45:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:53 smithi096 ceph-mon[32753]: pgmap v8045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:45:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14091.907179 secs 2024-04-26T15:45:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14094.711373 secs 2024-04-26T15:45:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14094.711373 secs 2024-04-26T15:45:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14094.711373 secs 2024-04-26T15:45:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:55 smithi194 ceph-mon[33017]: pgmap v8046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:45:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:55 smithi033 ceph-mon[27845]: pgmap v8046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:45:56.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:55 smithi096 ceph-mon[32753]: pgmap v8046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:45:58.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:58.045+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:58.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:58.045+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:45:58.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:45:58.045+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:45:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:57 smithi194 ceph-mon[33017]: pgmap v8047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:57 smithi033 ceph-mon[27845]: pgmap v8047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:58.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:57 smithi096 ceph-mon[32753]: pgmap v8047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:45:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14096.907237 secs 2024-04-26T15:45:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14096.907237 secs 2024-04-26T15:45:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14096.907237 secs 2024-04-26T15:46:00.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14099.711451 secs 2024-04-26T15:46:00.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:45:59 smithi194 ceph-mon[33017]: pgmap v8048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14099.711451 secs 2024-04-26T15:46:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:45:59 smithi033 ceph-mon[27845]: pgmap v8048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:00.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14099.711451 secs 2024-04-26T15:46:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:45:59 smithi096 ceph-mon[32753]: pgmap v8048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:01 smithi194 ceph-mon[33017]: pgmap v8049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:01 smithi033 ceph-mon[27845]: pgmap v8049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:01 smithi096 ceph-mon[32753]: pgmap v8049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:04 smithi194 ceph-mon[33017]: pgmap v8050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14101.907320 secs 2024-04-26T15:46:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:03 smithi033 ceph-mon[27845]: pgmap v8050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14101.907320 secs 2024-04-26T15:46:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:03 smithi096 ceph-mon[32753]: pgmap v8050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14101.907320 secs 2024-04-26T15:46:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14104.711589 secs 2024-04-26T15:46:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14104.711589 secs 2024-04-26T15:46:05.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14104.711589 secs 2024-04-26T15:46:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:06 smithi194 ceph-mon[33017]: pgmap v8051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:06 smithi033 ceph-mon[27845]: pgmap v8051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:06.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:06 smithi096 ceph-mon[32753]: pgmap v8051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:08 smithi194 ceph-mon[33017]: pgmap v8052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14106.907382 secs 2024-04-26T15:46:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:08 smithi033 ceph-mon[27845]: pgmap v8052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14106.907382 secs 2024-04-26T15:46:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:08 smithi096 ceph-mon[32753]: pgmap v8052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14106.907382 secs 2024-04-26T15:46:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14109.711718 secs 2024-04-26T15:46:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:10 smithi194 ceph-mon[33017]: pgmap v8053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14109.711718 secs 2024-04-26T15:46:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:10 smithi033 ceph-mon[27845]: pgmap v8053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:10.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14109.711718 secs 2024-04-26T15:46:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:10 smithi096 ceph-mon[32753]: pgmap v8053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:12 smithi194 ceph-mon[33017]: pgmap v8054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:12 smithi033 ceph-mon[27845]: pgmap v8054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:12.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:12 smithi096 ceph-mon[32753]: pgmap v8054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:13.056+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:13.056+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:13.056+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:46:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14111.907443 secs 2024-04-26T15:46:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14111.907443 secs 2024-04-26T15:46:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14111.907443 secs 2024-04-26T15:46:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:14 smithi194 ceph-mon[33017]: pgmap v8055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:14 smithi033 ceph-mon[27845]: pgmap v8055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:14.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:14 smithi096 ceph-mon[32753]: pgmap v8055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:15.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14114.711833 secs 2024-04-26T15:46:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14114.711833 secs 2024-04-26T15:46:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14114.711833 secs 2024-04-26T15:46:16.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:16 smithi194 ceph-mon[33017]: pgmap v8056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:16 smithi096 ceph-mon[32753]: pgmap v8056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:16.631 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:16 smithi033 ceph-mon[27845]: pgmap v8056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:18 smithi194 ceph-mon[33017]: pgmap v8057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14116.907530 secs 2024-04-26T15:46:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:18 smithi033 ceph-mon[27845]: pgmap v8057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14116.907530 secs 2024-04-26T15:46:18.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:18 smithi096 ceph-mon[32753]: pgmap v8057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14116.907530 secs 2024-04-26T15:46:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:19 smithi033 ceph-mon[27845]: pgmap v8058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:19.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:19 smithi096 ceph-mon[32753]: pgmap v8058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:19 smithi194 ceph-mon[33017]: pgmap v8058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14119.711967 secs 2024-04-26T15:46:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14119.711967 secs 2024-04-26T15:46:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14119.711967 secs 2024-04-26T15:46:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:21 smithi033 ceph-mon[27845]: pgmap v8059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:21 smithi096 ceph-mon[32753]: pgmap v8059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:21 smithi194 ceph-mon[33017]: pgmap v8059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:23 smithi194 ceph-mon[33017]: pgmap v8060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14121.907618 secs 2024-04-26T15:46:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:23 smithi033 ceph-mon[27845]: pgmap v8060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14121.907618 secs 2024-04-26T15:46:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:23 smithi096 ceph-mon[32753]: pgmap v8060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14121.907618 secs 2024-04-26T15:46:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14124.712144 secs 2024-04-26T15:46:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14124.712144 secs 2024-04-26T15:46:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14124.712144 secs 2024-04-26T15:46:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:25 smithi194 ceph-mon[33017]: pgmap v8061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:25 smithi033 ceph-mon[27845]: pgmap v8061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:25 smithi096 ceph-mon[32753]: pgmap v8061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:27 smithi194 ceph-mon[33017]: pgmap v8062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:27.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:27 smithi033 ceph-mon[27845]: pgmap v8062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:27 smithi096 ceph-mon[32753]: pgmap v8062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:28.067+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:28.067+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:46:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:28.067+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14126.907705 secs 2024-04-26T15:46:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14126.907705 secs 2024-04-26T15:46:28.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14126.907705 secs 2024-04-26T15:46:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14129.712211 secs 2024-04-26T15:46:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:29 smithi194 ceph-mon[33017]: pgmap v8063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14129.712211 secs 2024-04-26T15:46:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:29 smithi033 ceph-mon[27845]: pgmap v8063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:29.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14129.712211 secs 2024-04-26T15:46:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:29 smithi096 ceph-mon[32753]: pgmap v8063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:31 smithi194 ceph-mon[33017]: pgmap v8064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:31.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:31 smithi033 ceph-mon[27845]: pgmap v8064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:31 smithi096 ceph-mon[32753]: pgmap v8064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:33 smithi194 ceph-mon[33017]: pgmap v8065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14131.907791 secs 2024-04-26T15:46:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:33 smithi033 ceph-mon[27845]: pgmap v8065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14131.907791 secs 2024-04-26T15:46:33.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:33 smithi096 ceph-mon[32753]: pgmap v8065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14131.907791 secs 2024-04-26T15:46:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14134.712330 secs 2024-04-26T15:46:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14134.712330 secs 2024-04-26T15:46:34.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14134.712330 secs 2024-04-26T15:46:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:35 smithi194 ceph-mon[33017]: pgmap v8066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:46:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:35 smithi033 ceph-mon[27845]: pgmap v8066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:46:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:35 smithi096 ceph-mon[32753]: pgmap v8066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:46:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:46:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:46:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:46:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:46:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:46:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:46:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:46:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:46:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:46:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:37 smithi033 ceph-mon[27845]: pgmap v8067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:37 smithi033 ceph-mon[27845]: pgmap v8068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:46:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:37 smithi096 ceph-mon[32753]: pgmap v8067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:37 smithi096 ceph-mon[32753]: pgmap v8068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:46:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:37 smithi194 ceph-mon[33017]: pgmap v8067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:37 smithi194 ceph-mon[33017]: pgmap v8068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:46:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14136.907882 secs 2024-04-26T15:46:38.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14136.907882 secs 2024-04-26T15:46:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14136.907882 secs 2024-04-26T15:46:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:39 smithi033 ceph-mon[27845]: pgmap v8069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:46:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14139.712396 secs 2024-04-26T15:46:39.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:39 smithi096 ceph-mon[32753]: pgmap v8069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:46:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14139.712396 secs 2024-04-26T15:46:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:39 smithi194 ceph-mon[33017]: pgmap v8069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:46:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14139.712396 secs 2024-04-26T15:46:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:41 smithi033 ceph-mon[27845]: pgmap v8070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:46:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:41 smithi096 ceph-mon[32753]: pgmap v8070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:46:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:41 smithi194 ceph-mon[33017]: pgmap v8070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:46:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:43.077+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:43.077+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:46:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:43.077+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:43 smithi033 ceph-mon[27845]: pgmap v8071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:46:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14141.907977 secs 2024-04-26T15:46:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:43 smithi096 ceph-mon[32753]: pgmap v8071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:46:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14141.907977 secs 2024-04-26T15:46:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:43 smithi194 ceph-mon[33017]: pgmap v8071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T15:46:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14141.907977 secs 2024-04-26T15:46:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14144.712521 secs 2024-04-26T15:46:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14144.712521 secs 2024-04-26T15:46:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14144.712521 secs 2024-04-26T15:46:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:45 smithi033 ceph-mon[27845]: pgmap v8072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 950 B/s rd, 1 op/s 2024-04-26T15:46:45.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:45 smithi096 ceph-mon[32753]: pgmap v8072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 950 B/s rd, 1 op/s 2024-04-26T15:46:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:45 smithi194 ceph-mon[33017]: pgmap v8072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 950 B/s rd, 1 op/s 2024-04-26T15:46:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:47 smithi033 ceph-mon[27845]: pgmap v8073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:46:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:47 smithi096 ceph-mon[32753]: pgmap v8073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:46:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:47 smithi194 ceph-mon[33017]: pgmap v8073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:46:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14146.908065 secs 2024-04-26T15:46:48.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14146.908065 secs 2024-04-26T15:46:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14146.908065 secs 2024-04-26T15:46:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:49 smithi033 ceph-mon[27845]: pgmap v8074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:46:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14149.712607 secs 2024-04-26T15:46:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:49 smithi096 ceph-mon[32753]: pgmap v8074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:46:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14149.712607 secs 2024-04-26T15:46:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:49 smithi194 ceph-mon[33017]: pgmap v8074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:46:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14149.712607 secs 2024-04-26T15:46:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:51 smithi033 ceph-mon[27845]: pgmap v8075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:46:51.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:51 smithi096 ceph-mon[32753]: pgmap v8075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:46:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:51 smithi194 ceph-mon[33017]: pgmap v8075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:46:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:53 smithi033 ceph-mon[27845]: pgmap v8076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14151.908123 secs 2024-04-26T15:46:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:53 smithi096 ceph-mon[32753]: pgmap v8076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14151.908123 secs 2024-04-26T15:46:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:53 smithi194 ceph-mon[33017]: pgmap v8076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14151.908123 secs 2024-04-26T15:46:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14154.712733 secs 2024-04-26T15:46:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14154.712733 secs 2024-04-26T15:46:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14154.712733 secs 2024-04-26T15:46:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:55 smithi194 ceph-mon[33017]: pgmap v8077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:55 smithi033 ceph-mon[27845]: pgmap v8077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:55 smithi096 ceph-mon[32753]: pgmap v8077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:46:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:58.093+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:58.093+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:46:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:46:58.093+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:46:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:57 smithi194 ceph-mon[33017]: pgmap v8078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:57 smithi033 ceph-mon[27845]: pgmap v8078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:57 smithi096 ceph-mon[32753]: pgmap v8078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:46:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14156.908208 secs 2024-04-26T15:46:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14156.908208 secs 2024-04-26T15:46:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14156.908208 secs 2024-04-26T15:47:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:59 smithi194 ceph-mon[33017]: pgmap v8079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:46:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14159.712889 secs 2024-04-26T15:47:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:59 smithi033 ceph-mon[27845]: pgmap v8079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:46:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14159.712889 secs 2024-04-26T15:47:00.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:59 smithi096 ceph-mon[32753]: pgmap v8079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:46:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14159.712889 secs 2024-04-26T15:47:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:01 smithi194 ceph-mon[33017]: pgmap v8080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:01 smithi033 ceph-mon[27845]: pgmap v8080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:01 smithi096 ceph-mon[32753]: pgmap v8080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:03.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:03 smithi096 ceph-mon[32753]: pgmap v8081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:03.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14161.908294 secs 2024-04-26T15:47:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:03 smithi194 ceph-mon[33017]: pgmap v8081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14161.908294 secs 2024-04-26T15:47:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:03 smithi033 ceph-mon[27845]: pgmap v8081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14161.908294 secs 2024-04-26T15:47:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14164.713011 secs 2024-04-26T15:47:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14164.713011 secs 2024-04-26T15:47:05.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14164.713011 secs 2024-04-26T15:47:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:05 smithi194 ceph-mon[33017]: pgmap v8082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:05 smithi033 ceph-mon[27845]: pgmap v8082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:06.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:05 smithi096 ceph-mon[32753]: pgmap v8082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:07 smithi194 ceph-mon[33017]: pgmap v8083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:07 smithi033 ceph-mon[27845]: pgmap v8083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:07 smithi096 ceph-mon[32753]: pgmap v8083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14166.908381 secs 2024-04-26T15:47:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14166.908381 secs 2024-04-26T15:47:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14166.908381 secs 2024-04-26T15:47:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:09 smithi194 ceph-mon[33017]: pgmap v8084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14169.713082 secs 2024-04-26T15:47:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:09 smithi033 ceph-mon[27845]: pgmap v8084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14169.713082 secs 2024-04-26T15:47:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:09 smithi096 ceph-mon[32753]: pgmap v8084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14169.713082 secs 2024-04-26T15:47:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:11 smithi194 ceph-mon[33017]: pgmap v8085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:11 smithi033 ceph-mon[27845]: pgmap v8085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:11 smithi096 ceph-mon[32753]: pgmap v8085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:13.099+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:13.099+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:47:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:13.099+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:13 smithi194 ceph-mon[33017]: pgmap v8086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14171.908465 secs 2024-04-26T15:47:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:13 smithi033 ceph-mon[27845]: pgmap v8086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14171.908465 secs 2024-04-26T15:47:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:13 smithi096 ceph-mon[32753]: pgmap v8086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14171.908465 secs 2024-04-26T15:47:15.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14174.713192 secs 2024-04-26T15:47:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14174.713192 secs 2024-04-26T15:47:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14174.713192 secs 2024-04-26T15:47:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:15 smithi194 ceph-mon[33017]: pgmap v8087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:15 smithi033 ceph-mon[27845]: pgmap v8087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:15 smithi096 ceph-mon[32753]: pgmap v8087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:17 smithi194 ceph-mon[33017]: pgmap v8088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:17 smithi033 ceph-mon[27845]: pgmap v8088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:17 smithi096 ceph-mon[32753]: pgmap v8088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14176.908551 secs 2024-04-26T15:47:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14176.908551 secs 2024-04-26T15:47:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14176.908551 secs 2024-04-26T15:47:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:19 smithi194 ceph-mon[33017]: pgmap v8089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14179.713318 secs 2024-04-26T15:47:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:19 smithi033 ceph-mon[27845]: pgmap v8089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14179.713318 secs 2024-04-26T15:47:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:19 smithi096 ceph-mon[32753]: pgmap v8089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14179.713318 secs 2024-04-26T15:47:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:21 smithi194 ceph-mon[33017]: pgmap v8090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:21 smithi033 ceph-mon[27845]: pgmap v8090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:21 smithi096 ceph-mon[32753]: pgmap v8090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:23 smithi194 ceph-mon[33017]: pgmap v8091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14181.908640 secs 2024-04-26T15:47:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:23 smithi033 ceph-mon[27845]: pgmap v8091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14181.908640 secs 2024-04-26T15:47:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:23 smithi096 ceph-mon[32753]: pgmap v8091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14181.908640 secs 2024-04-26T15:47:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:24.208+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T15:47:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:24.208+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:47:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:24.208+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006630 con 0x564385821900 2024-04-26T15:47:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:24.208+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:47:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14184.713434 secs 2024-04-26T15:47:25.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14184.713434 secs 2024-04-26T15:47:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14184.713434 secs 2024-04-26T15:47:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:25 smithi194 ceph-mon[33017]: pgmap v8092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:25 smithi033 ceph-mon[27845]: pgmap v8092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:26.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:25 smithi096 ceph-mon[32753]: pgmap v8092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:27 smithi194 ceph-mon[33017]: pgmap v8093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:28.112+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:28.112+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:28.112+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:47:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:27 smithi033 ceph-mon[27845]: pgmap v8093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:27 smithi096 ceph-mon[32753]: pgmap v8093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14186.908727 secs 2024-04-26T15:47:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14186.908727 secs 2024-04-26T15:47:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14186.908727 secs 2024-04-26T15:47:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:29 smithi194 ceph-mon[33017]: pgmap v8094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14189.713531 secs 2024-04-26T15:47:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:29 smithi033 ceph-mon[27845]: pgmap v8094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14189.713531 secs 2024-04-26T15:47:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:29 smithi096 ceph-mon[32753]: pgmap v8094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14189.713531 secs 2024-04-26T15:47:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:31 smithi194 ceph-mon[33017]: pgmap v8095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:31 smithi033 ceph-mon[27845]: pgmap v8095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:32.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:31 smithi096 ceph-mon[32753]: pgmap v8095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:33 smithi194 ceph-mon[33017]: pgmap v8096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14191.908814 secs 2024-04-26T15:47:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:33 smithi033 ceph-mon[27845]: pgmap v8096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14191.908814 secs 2024-04-26T15:47:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:33 smithi096 ceph-mon[32753]: pgmap v8096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14191.908814 secs 2024-04-26T15:47:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14194.713633 secs 2024-04-26T15:47:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14194.713633 secs 2024-04-26T15:47:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14194.713633 secs 2024-04-26T15:47:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:35 smithi033 ceph-mon[27845]: pgmap v8097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:36.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:35 smithi096 ceph-mon[32753]: pgmap v8097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:35 smithi194 ceph-mon[33017]: pgmap v8097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:47:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:47:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:47:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:37 smithi033 ceph-mon[27845]: pgmap v8098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:37 smithi096 ceph-mon[32753]: pgmap v8098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:37 smithi194 ceph-mon[33017]: pgmap v8098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14196.908898 secs 2024-04-26T15:47:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:47:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:47:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:38 smithi033 ceph-mon[27845]: pgmap v8099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:38 smithi033 ceph-mon[27845]: pgmap v8100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:47:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14196.908898 secs 2024-04-26T15:47:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:47:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:47:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:38 smithi096 ceph-mon[32753]: pgmap v8099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:38 smithi096 ceph-mon[32753]: pgmap v8100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:47:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14196.908898 secs 2024-04-26T15:47:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:47:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:47:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:38 smithi194 ceph-mon[33017]: pgmap v8099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:38 smithi194 ceph-mon[33017]: pgmap v8100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:47:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14199.713756 secs 2024-04-26T15:47:40.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14199.713756 secs 2024-04-26T15:47:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14199.713756 secs 2024-04-26T15:47:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:40 smithi033 ceph-mon[27845]: pgmap v8101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:40 smithi096 ceph-mon[32753]: pgmap v8101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:40 smithi194 ceph-mon[33017]: pgmap v8101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:43.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:43.118+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:43.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:43.118+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:43.118+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:47:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:42 smithi194 ceph-mon[33017]: pgmap v8102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:42 smithi033 ceph-mon[27845]: pgmap v8102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:42 smithi096 ceph-mon[32753]: pgmap v8102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14201.908985 secs 2024-04-26T15:47:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14201.908985 secs 2024-04-26T15:47:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14201.908985 secs 2024-04-26T15:47:45.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14204.713869 secs 2024-04-26T15:47:45.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:44 smithi033 ceph-mon[27845]: pgmap v8103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14204.713869 secs 2024-04-26T15:47:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:44 smithi194 ceph-mon[33017]: pgmap v8103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14204.713869 secs 2024-04-26T15:47:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:44 smithi096 ceph-mon[32753]: pgmap v8103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:46 smithi194 ceph-mon[33017]: pgmap v8104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:46 smithi033 ceph-mon[27845]: pgmap v8104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:46 smithi096 ceph-mon[32753]: pgmap v8104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:47:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14206.909075 secs 2024-04-26T15:47:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:48 smithi194 ceph-mon[33017]: pgmap v8105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:47:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14206.909075 secs 2024-04-26T15:47:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:48 smithi033 ceph-mon[27845]: pgmap v8105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:47:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14206.909075 secs 2024-04-26T15:47:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:48 smithi096 ceph-mon[32753]: pgmap v8105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:47:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14209.713984 secs 2024-04-26T15:47:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14209.713984 secs 2024-04-26T15:47:50.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14209.713984 secs 2024-04-26T15:47:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:50 smithi194 ceph-mon[33017]: pgmap v8106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:50 smithi033 ceph-mon[27845]: pgmap v8106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:50 smithi096 ceph-mon[32753]: pgmap v8106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:53 smithi194 ceph-mon[33017]: pgmap v8107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:52 smithi033 ceph-mon[27845]: pgmap v8107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:53.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:53 smithi096 ceph-mon[32753]: pgmap v8107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14211.909160 secs 2024-04-26T15:47:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14211.909160 secs 2024-04-26T15:47:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14211.909160 secs 2024-04-26T15:47:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14214.714144 secs 2024-04-26T15:47:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:55 smithi194 ceph-mon[33017]: pgmap v8108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14214.714144 secs 2024-04-26T15:47:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:55 smithi033 ceph-mon[27845]: pgmap v8108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14214.714144 secs 2024-04-26T15:47:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:55 smithi096 ceph-mon[32753]: pgmap v8108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:57 smithi194 ceph-mon[33017]: pgmap v8109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:57 smithi033 ceph-mon[27845]: pgmap v8109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:57 smithi096 ceph-mon[32753]: pgmap v8109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:47:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:58.132+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:58.132+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:47:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:47:58.132+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:47:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14216.909221 secs 2024-04-26T15:47:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14216.909221 secs 2024-04-26T15:47:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14216.909221 secs 2024-04-26T15:47:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:47:59 smithi194 ceph-mon[33017]: pgmap v8110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:47:59 smithi033 ceph-mon[27845]: pgmap v8110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:47:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:47:59 smithi096 ceph-mon[32753]: pgmap v8110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14219.714259 secs 2024-04-26T15:48:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14219.714259 secs 2024-04-26T15:48:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14219.714259 secs 2024-04-26T15:48:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:01 smithi194 ceph-mon[33017]: pgmap v8111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:01 smithi033 ceph-mon[27845]: pgmap v8111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:01.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:01 smithi096 ceph-mon[32753]: pgmap v8111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:03 smithi194 ceph-mon[33017]: pgmap v8112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14221.909318 secs 2024-04-26T15:48:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:03 smithi033 ceph-mon[27845]: pgmap v8112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14221.909318 secs 2024-04-26T15:48:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:03 smithi096 ceph-mon[32753]: pgmap v8112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14221.909318 secs 2024-04-26T15:48:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14224.714354 secs 2024-04-26T15:48:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:05 smithi194 ceph-mon[33017]: pgmap v8113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14224.714354 secs 2024-04-26T15:48:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:05 smithi033 ceph-mon[27845]: pgmap v8113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:05.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14224.714354 secs 2024-04-26T15:48:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:05 smithi096 ceph-mon[32753]: pgmap v8113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:07 smithi194 ceph-mon[33017]: pgmap v8114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:07 smithi033 ceph-mon[27845]: pgmap v8114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:07.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:07 smithi096 ceph-mon[32753]: pgmap v8114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14226.909377 secs 2024-04-26T15:48:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14226.909377 secs 2024-04-26T15:48:08.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14226.909377 secs 2024-04-26T15:48:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:09 smithi194 ceph-mon[33017]: pgmap v8115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:09 smithi033 ceph-mon[27845]: pgmap v8115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:09 smithi096 ceph-mon[32753]: pgmap v8115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14229.714510 secs 2024-04-26T15:48:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14229.714510 secs 2024-04-26T15:48:10.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14229.714510 secs 2024-04-26T15:48:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:11 smithi194 ceph-mon[33017]: pgmap v8116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:11 smithi033 ceph-mon[27845]: pgmap v8116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:11.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:11 smithi096 ceph-mon[32753]: pgmap v8116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:12 smithi033 ceph-mon[27845]: pgmap v8117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:12.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:12 smithi096 ceph-mon[32753]: pgmap v8117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:12 smithi194 ceph-mon[33017]: pgmap v8117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:13.147+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:13.147+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:13.147+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:48:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14231.909463 secs 2024-04-26T15:48:13.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14231.909463 secs 2024-04-26T15:48:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14231.909463 secs 2024-04-26T15:48:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:14 smithi033 ceph-mon[27845]: pgmap v8118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:14.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:14 smithi096 ceph-mon[32753]: pgmap v8118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:14 smithi194 ceph-mon[33017]: pgmap v8118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14234.714602 secs 2024-04-26T15:48:15.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14234.714602 secs 2024-04-26T15:48:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14234.714602 secs 2024-04-26T15:48:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:16 smithi033 ceph-mon[27845]: pgmap v8119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:16.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:16 smithi096 ceph-mon[32753]: pgmap v8119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:16 smithi194 ceph-mon[33017]: pgmap v8119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14236.909550 secs 2024-04-26T15:48:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:18 smithi194 ceph-mon[33017]: pgmap v8120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14236.909550 secs 2024-04-26T15:48:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:18 smithi033 ceph-mon[27845]: pgmap v8120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:19.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14236.909550 secs 2024-04-26T15:48:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:18 smithi096 ceph-mon[32753]: pgmap v8120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14239.714720 secs 2024-04-26T15:48:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14239.714720 secs 2024-04-26T15:48:20.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14239.714720 secs 2024-04-26T15:48:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:20 smithi194 ceph-mon[33017]: pgmap v8121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:20 smithi033 ceph-mon[27845]: pgmap v8121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:21.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:20 smithi096 ceph-mon[32753]: pgmap v8121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:22 smithi194 ceph-mon[33017]: pgmap v8122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:22 smithi033 ceph-mon[27845]: pgmap v8122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:23.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:22 smithi096 ceph-mon[32753]: pgmap v8122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14241.909632 secs 2024-04-26T15:48:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14241.909632 secs 2024-04-26T15:48:24.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14241.909632 secs 2024-04-26T15:48:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14244.714854 secs 2024-04-26T15:48:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:25 smithi194 ceph-mon[33017]: pgmap v8123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14244.714854 secs 2024-04-26T15:48:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:25 smithi033 ceph-mon[27845]: pgmap v8123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14244.714854 secs 2024-04-26T15:48:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:25 smithi096 ceph-mon[32753]: pgmap v8123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:27 smithi194 ceph-mon[33017]: pgmap v8124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:27 smithi033 ceph-mon[27845]: pgmap v8124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:27.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:27 smithi096 ceph-mon[32753]: pgmap v8124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:28.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:28.156+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:28.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:28.156+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:48:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:28.156+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14246.909721 secs 2024-04-26T15:48:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:29 smithi194 ceph-mon[33017]: pgmap v8125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14246.909721 secs 2024-04-26T15:48:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:29 smithi033 ceph-mon[27845]: pgmap v8125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:29.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14246.909721 secs 2024-04-26T15:48:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:29 smithi096 ceph-mon[32753]: pgmap v8125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14249.714954 secs 2024-04-26T15:48:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14249.714954 secs 2024-04-26T15:48:30.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14249.714954 secs 2024-04-26T15:48:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:31 smithi194 ceph-mon[33017]: pgmap v8126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:31 smithi033 ceph-mon[27845]: pgmap v8126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:31.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:31 smithi096 ceph-mon[32753]: pgmap v8126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:33 smithi194 ceph-mon[33017]: pgmap v8127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14251.909805 secs 2024-04-26T15:48:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:33 smithi033 ceph-mon[27845]: pgmap v8127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14251.909805 secs 2024-04-26T15:48:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:33 smithi096 ceph-mon[32753]: pgmap v8127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14251.909805 secs 2024-04-26T15:48:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14254.715092 secs 2024-04-26T15:48:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:35 smithi194 ceph-mon[33017]: pgmap v8128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14254.715092 secs 2024-04-26T15:48:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:35 smithi033 ceph-mon[27845]: pgmap v8128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:35.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14254.715092 secs 2024-04-26T15:48:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:35 smithi096 ceph-mon[32753]: pgmap v8128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:37 smithi194 ceph-mon[33017]: pgmap v8129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:37 smithi033 ceph-mon[27845]: pgmap v8129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:37 smithi096 ceph-mon[32753]: pgmap v8129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14256.909892 secs 2024-04-26T15:48:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14256.909892 secs 2024-04-26T15:48:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14256.909892 secs 2024-04-26T15:48:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:39 smithi194 ceph-mon[33017]: pgmap v8130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:48:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:39 smithi033 ceph-mon[27845]: pgmap v8130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:48:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:39 smithi096 ceph-mon[32753]: pgmap v8130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:48:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14259.715158 secs 2024-04-26T15:48:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:40 smithi033 ceph-mon[27845]: pgmap v8131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14259.715158 secs 2024-04-26T15:48:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:40 smithi096 ceph-mon[32753]: pgmap v8131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14259.715158 secs 2024-04-26T15:48:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:40 smithi194 ceph-mon[33017]: pgmap v8131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:48:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:48:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:41 smithi033 ceph-mon[27845]: pgmap v8132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:48:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:41 smithi033 ceph-mon[27845]: pgmap v8133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:48:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:48:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:41 smithi096 ceph-mon[32753]: pgmap v8132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:48:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:41 smithi096 ceph-mon[32753]: pgmap v8133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:48:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:48:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:41 smithi194 ceph-mon[33017]: pgmap v8132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:48:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:41 smithi194 ceph-mon[33017]: pgmap v8133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:48:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:43.166+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:43.166+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:43.166+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:48:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:43 smithi194 ceph-mon[33017]: pgmap v8134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14261.909982 secs 2024-04-26T15:48:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:43 smithi033 ceph-mon[27845]: pgmap v8134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14261.909982 secs 2024-04-26T15:48:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:43 smithi096 ceph-mon[32753]: pgmap v8134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14261.909982 secs 2024-04-26T15:48:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14264.715272 secs 2024-04-26T15:48:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14264.715272 secs 2024-04-26T15:48:44.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14264.715272 secs 2024-04-26T15:48:45.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:45 smithi033 ceph-mon[27845]: pgmap v8135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:45 smithi194 ceph-mon[33017]: pgmap v8135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:45 smithi096 ceph-mon[32753]: pgmap v8135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:47 smithi194 ceph-mon[33017]: pgmap v8136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:47 smithi033 ceph-mon[27845]: pgmap v8136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:47.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:47 smithi096 ceph-mon[32753]: pgmap v8136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:48:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14266.910068 secs 2024-04-26T15:48:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14266.910068 secs 2024-04-26T15:48:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14266.910068 secs 2024-04-26T15:48:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14269.715394 secs 2024-04-26T15:48:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:49 smithi194 ceph-mon[33017]: pgmap v8137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:48:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14269.715394 secs 2024-04-26T15:48:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:49 smithi033 ceph-mon[27845]: pgmap v8137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:48:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14269.715394 secs 2024-04-26T15:48:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:49 smithi096 ceph-mon[32753]: pgmap v8137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:48:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:51 smithi194 ceph-mon[33017]: pgmap v8138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:48:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:51 smithi033 ceph-mon[27845]: pgmap v8138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:48:51.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:51 smithi096 ceph-mon[32753]: pgmap v8138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:48:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:53 smithi194 ceph-mon[33017]: pgmap v8139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14271.910154 secs 2024-04-26T15:48:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:53 smithi033 ceph-mon[27845]: pgmap v8139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14271.910154 secs 2024-04-26T15:48:53.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:53 smithi096 ceph-mon[32753]: pgmap v8139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14271.910154 secs 2024-04-26T15:48:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:54.224+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:48:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:54.224+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T15:48:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:54.225+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:48:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:54.225+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:48:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14274.715488 secs 2024-04-26T15:48:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14274.715488 secs 2024-04-26T15:48:54.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14274.715488 secs 2024-04-26T15:48:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:55 smithi194 ceph-mon[33017]: pgmap v8140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:55 smithi033 ceph-mon[27845]: pgmap v8140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:55 smithi096 ceph-mon[32753]: pgmap v8140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:48:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:57 smithi194 ceph-mon[33017]: pgmap v8141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:57 smithi033 ceph-mon[27845]: pgmap v8141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:57 smithi096 ceph-mon[32753]: pgmap v8141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:58.178+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:58.178+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:48:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:48:58.178+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:48:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14276.910242 secs 2024-04-26T15:48:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14276.910242 secs 2024-04-26T15:48:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14276.910242 secs 2024-04-26T15:48:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14279.715591 secs 2024-04-26T15:48:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:48:59 smithi033 ceph-mon[27845]: pgmap v8142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:48:59.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14279.715591 secs 2024-04-26T15:48:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:48:59 smithi096 ceph-mon[32753]: pgmap v8142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14279.715591 secs 2024-04-26T15:49:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:48:59 smithi194 ceph-mon[33017]: pgmap v8142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:01.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:01 smithi194 ceph-mon[33017]: pgmap v8143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:01 smithi033 ceph-mon[27845]: pgmap v8143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:01 smithi096 ceph-mon[32753]: pgmap v8143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:03 smithi033 ceph-mon[27845]: pgmap v8144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14281.910323 secs 2024-04-26T15:49:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:03 smithi096 ceph-mon[32753]: pgmap v8144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14281.910323 secs 2024-04-26T15:49:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:03 smithi194 ceph-mon[33017]: pgmap v8144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14281.910323 secs 2024-04-26T15:49:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14284.715707 secs 2024-04-26T15:49:04.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14284.715707 secs 2024-04-26T15:49:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14284.715707 secs 2024-04-26T15:49:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:05 smithi033 ceph-mon[27845]: pgmap v8145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:05.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:05 smithi096 ceph-mon[32753]: pgmap v8145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:05 smithi194 ceph-mon[33017]: pgmap v8145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:07.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:07 smithi033 ceph-mon[27845]: pgmap v8146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:07.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:07 smithi096 ceph-mon[32753]: pgmap v8146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:07 smithi194 ceph-mon[33017]: pgmap v8146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14286.910409 secs 2024-04-26T15:49:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14286.910409 secs 2024-04-26T15:49:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14286.910409 secs 2024-04-26T15:49:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14289.715820 secs 2024-04-26T15:49:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:09 smithi033 ceph-mon[27845]: pgmap v8147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:09.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14289.715820 secs 2024-04-26T15:49:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:09 smithi096 ceph-mon[32753]: pgmap v8147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14289.715820 secs 2024-04-26T15:49:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:09 smithi194 ceph-mon[33017]: pgmap v8147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:11.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:11 smithi033 ceph-mon[27845]: pgmap v8148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:11.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:11 smithi096 ceph-mon[32753]: pgmap v8148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:11 smithi194 ceph-mon[33017]: pgmap v8148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:13.182+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:13.182+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:13.182+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:49:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:13 smithi033 ceph-mon[27845]: pgmap v8149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14291.910471 secs 2024-04-26T15:49:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:13 smithi096 ceph-mon[32753]: pgmap v8149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14291.910471 secs 2024-04-26T15:49:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:13 smithi194 ceph-mon[33017]: pgmap v8149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14291.910471 secs 2024-04-26T15:49:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14294.715934 secs 2024-04-26T15:49:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14294.715934 secs 2024-04-26T15:49:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14294.715934 secs 2024-04-26T15:49:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:15 smithi096 ceph-mon[32753]: pgmap v8150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:15 smithi194 ceph-mon[33017]: pgmap v8150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:15 smithi033 ceph-mon[27845]: pgmap v8150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:17 smithi194 ceph-mon[33017]: pgmap v8151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:17 smithi033 ceph-mon[27845]: pgmap v8151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:18.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:17 smithi096 ceph-mon[32753]: pgmap v8151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14296.910557 secs 2024-04-26T15:49:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14296.910557 secs 2024-04-26T15:49:19.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14296.910557 secs 2024-04-26T15:49:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14299.716054 secs 2024-04-26T15:49:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:19 smithi194 ceph-mon[33017]: pgmap v8152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14299.716054 secs 2024-04-26T15:49:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:19 smithi033 ceph-mon[27845]: pgmap v8152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14299.716054 secs 2024-04-26T15:49:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:19 smithi096 ceph-mon[32753]: pgmap v8152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:21 smithi194 ceph-mon[33017]: pgmap v8153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:21 smithi033 ceph-mon[27845]: pgmap v8153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:21 smithi096 ceph-mon[32753]: pgmap v8153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:23 smithi194 ceph-mon[33017]: pgmap v8154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14301.910644 secs 2024-04-26T15:49:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:23 smithi033 ceph-mon[27845]: pgmap v8154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14301.910644 secs 2024-04-26T15:49:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:23 smithi096 ceph-mon[32753]: pgmap v8154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14301.910644 secs 2024-04-26T15:49:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14304.716169 secs 2024-04-26T15:49:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14304.716169 secs 2024-04-26T15:49:25.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14304.716169 secs 2024-04-26T15:49:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:25 smithi194 ceph-mon[33017]: pgmap v8155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:26.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:25 smithi033 ceph-mon[27845]: pgmap v8155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:25 smithi096 ceph-mon[32753]: pgmap v8155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:27 smithi194 ceph-mon[33017]: pgmap v8156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:27 smithi033 ceph-mon[27845]: pgmap v8156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:28.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:27 smithi096 ceph-mon[32753]: pgmap v8156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:28.197+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:28.197+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:28.197+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:49:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14306.910732 secs 2024-04-26T15:49:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14306.910732 secs 2024-04-26T15:49:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14306.910732 secs 2024-04-26T15:49:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14309.716305 secs 2024-04-26T15:49:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:29 smithi194 ceph-mon[33017]: pgmap v8157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14309.716305 secs 2024-04-26T15:49:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:29 smithi033 ceph-mon[27845]: pgmap v8157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14309.716305 secs 2024-04-26T15:49:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:29 smithi096 ceph-mon[32753]: pgmap v8157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:31 smithi194 ceph-mon[33017]: pgmap v8158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:31 smithi033 ceph-mon[27845]: pgmap v8158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:32.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:31 smithi096 ceph-mon[32753]: pgmap v8158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:33 smithi194 ceph-mon[33017]: pgmap v8159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14311.910819 secs 2024-04-26T15:49:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:33 smithi033 ceph-mon[27845]: pgmap v8159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14311.910819 secs 2024-04-26T15:49:34.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:33 smithi096 ceph-mon[32753]: pgmap v8159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14311.910819 secs 2024-04-26T15:49:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14314.716391 secs 2024-04-26T15:49:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14314.716391 secs 2024-04-26T15:49:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14314.716391 secs 2024-04-26T15:49:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:35 smithi194 ceph-mon[33017]: pgmap v8160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:35 smithi033 ceph-mon[27845]: pgmap v8160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:36.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:35 smithi096 ceph-mon[32753]: pgmap v8160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:37 smithi194 ceph-mon[33017]: pgmap v8161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:37 smithi033 ceph-mon[27845]: pgmap v8161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:37 smithi096 ceph-mon[32753]: pgmap v8161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14316.910905 secs 2024-04-26T15:49:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14316.910905 secs 2024-04-26T15:49:39.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14316.910905 secs 2024-04-26T15:49:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14319.716469 secs 2024-04-26T15:49:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:39 smithi194 ceph-mon[33017]: pgmap v8162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14319.716469 secs 2024-04-26T15:49:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:39 smithi033 ceph-mon[27845]: pgmap v8162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:40.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14319.716469 secs 2024-04-26T15:49:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:39 smithi096 ceph-mon[32753]: pgmap v8162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:41 smithi194 ceph-mon[33017]: pgmap v8163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:49:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:41 smithi033 ceph-mon[27845]: pgmap v8163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:49:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:41 smithi096 ceph-mon[32753]: pgmap v8163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:49:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:49:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:49:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:49:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:49:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:49:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:49:43.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:49:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:49:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:49:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:43.211+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:43.212+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:43.212+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:49:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:43 smithi194 ceph-mon[33017]: pgmap v8164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:43 smithi194 ceph-mon[33017]: pgmap v8165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14321.910990 secs 2024-04-26T15:49:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:43 smithi033 ceph-mon[27845]: pgmap v8164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:43 smithi033 ceph-mon[27845]: pgmap v8165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14321.910990 secs 2024-04-26T15:49:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:43 smithi096 ceph-mon[32753]: pgmap v8164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:43 smithi096 ceph-mon[32753]: pgmap v8165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14321.910990 secs 2024-04-26T15:49:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14324.716590 secs 2024-04-26T15:49:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14324.716590 secs 2024-04-26T15:49:45.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14324.716590 secs 2024-04-26T15:49:46.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:45 smithi033 ceph-mon[27845]: pgmap v8166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:49:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:45 smithi194 ceph-mon[33017]: pgmap v8166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:49:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:45 smithi096 ceph-mon[32753]: pgmap v8166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:49:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:47 smithi194 ceph-mon[33017]: pgmap v8167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:47 smithi033 ceph-mon[27845]: pgmap v8167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:48.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:47 smithi096 ceph-mon[32753]: pgmap v8167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14326.911078 secs 2024-04-26T15:49:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14326.911078 secs 2024-04-26T15:49:49.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14326.911078 secs 2024-04-26T15:49:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:49 smithi194 ceph-mon[33017]: pgmap v8168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14329.716700 secs 2024-04-26T15:49:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:49 smithi033 ceph-mon[27845]: pgmap v8168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14329.716700 secs 2024-04-26T15:49:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:49 smithi096 ceph-mon[32753]: pgmap v8168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14329.716700 secs 2024-04-26T15:49:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:51 smithi194 ceph-mon[33017]: pgmap v8169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:51 smithi033 ceph-mon[27845]: pgmap v8169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:51 smithi096 ceph-mon[32753]: pgmap v8169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:49:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:53 smithi194 ceph-mon[33017]: pgmap v8170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:49:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14331.911167 secs 2024-04-26T15:49:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:53 smithi033 ceph-mon[27845]: pgmap v8170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:49:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14331.911167 secs 2024-04-26T15:49:54.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:53 smithi096 ceph-mon[32753]: pgmap v8170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:49:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14331.911167 secs 2024-04-26T15:49:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14334.716808 secs 2024-04-26T15:49:55.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14334.716808 secs 2024-04-26T15:49:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14334.716808 secs 2024-04-26T15:49:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:55 smithi033 ceph-mon[27845]: pgmap v8171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:56.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:55 smithi096 ceph-mon[32753]: pgmap v8171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:55 smithi194 ceph-mon[33017]: pgmap v8171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:49:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:57 smithi033 ceph-mon[27845]: pgmap v8172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:58.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:57 smithi096 ceph-mon[32753]: pgmap v8172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:58.227+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:58.227+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:49:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:49:58.227+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:49:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:57 smithi194 ceph-mon[33017]: pgmap v8172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:49:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14336.911228 secs 2024-04-26T15:49:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14336.911228 secs 2024-04-26T15:49:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14336.911228 secs 2024-04-26T15:50:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:59 smithi033 ceph-mon[27845]: pgmap v8173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14339.716944 secs 2024-04-26T15:50:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:59 smithi096 ceph-mon[32753]: pgmap v8173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14339.716944 secs 2024-04-26T15:50:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:59 smithi194 ceph-mon[33017]: pgmap v8173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14339.716944 secs 2024-04-26T15:50:01.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:50:01.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:50:01.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T15:50:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:01 smithi033 ceph-mon[27845]: pgmap v8174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:01 smithi096 ceph-mon[32753]: pgmap v8174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:01 smithi194 ceph-mon[33017]: pgmap v8174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:03 smithi033 ceph-mon[27845]: pgmap v8175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14341.911316 secs 2024-04-26T15:50:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:03 smithi096 ceph-mon[32753]: pgmap v8175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14341.911316 secs 2024-04-26T15:50:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:03 smithi194 ceph-mon[33017]: pgmap v8175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14341.911316 secs 2024-04-26T15:50:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14344.717077 secs 2024-04-26T15:50:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14344.717077 secs 2024-04-26T15:50:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14344.717077 secs 2024-04-26T15:50:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:05 smithi033 ceph-mon[27845]: pgmap v8176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:06.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:05 smithi096 ceph-mon[32753]: pgmap v8176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:05 smithi194 ceph-mon[33017]: pgmap v8176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:07 smithi096 ceph-mon[32753]: pgmap v8177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:07 smithi194 ceph-mon[33017]: pgmap v8177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:07 smithi033 ceph-mon[27845]: pgmap v8177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14346.911402 secs 2024-04-26T15:50:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14346.911402 secs 2024-04-26T15:50:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14346.911402 secs 2024-04-26T15:50:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:09 smithi194 ceph-mon[33017]: pgmap v8178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14349.717158 secs 2024-04-26T15:50:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:09 smithi033 ceph-mon[27845]: pgmap v8178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14349.717158 secs 2024-04-26T15:50:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:09 smithi096 ceph-mon[32753]: pgmap v8178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14349.717158 secs 2024-04-26T15:50:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:11 smithi194 ceph-mon[33017]: pgmap v8179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:11 smithi033 ceph-mon[27845]: pgmap v8179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:12.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:11 smithi096 ceph-mon[32753]: pgmap v8179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:13.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:13.240+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:13.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:13.240+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:13.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:13.240+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:50:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:13 smithi194 ceph-mon[33017]: pgmap v8180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14351.911488 secs 2024-04-26T15:50:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:13 smithi033 ceph-mon[27845]: pgmap v8180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14351.911488 secs 2024-04-26T15:50:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:13 smithi096 ceph-mon[32753]: pgmap v8180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14351.911488 secs 2024-04-26T15:50:15.209 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14354.717269 secs 2024-04-26T15:50:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14354.717269 secs 2024-04-26T15:50:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14354.717269 secs 2024-04-26T15:50:16.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:15 smithi033 ceph-mon[27845]: pgmap v8181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:15 smithi194 ceph-mon[33017]: pgmap v8181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:15 smithi096 ceph-mon[32753]: pgmap v8181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:17 smithi194 ceph-mon[33017]: pgmap v8182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:17 smithi033 ceph-mon[27845]: pgmap v8182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:17 smithi096 ceph-mon[32753]: pgmap v8182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14356.911577 secs 2024-04-26T15:50:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14356.911577 secs 2024-04-26T15:50:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14356.911577 secs 2024-04-26T15:50:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:19 smithi194 ceph-mon[33017]: pgmap v8183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14359.717379 secs 2024-04-26T15:50:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:19 smithi033 ceph-mon[27845]: pgmap v8183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14359.717379 secs 2024-04-26T15:50:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:19 smithi096 ceph-mon[32753]: pgmap v8183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14359.717379 secs 2024-04-26T15:50:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:21 smithi194 ceph-mon[33017]: pgmap v8184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:21 smithi033 ceph-mon[27845]: pgmap v8184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:21 smithi096 ceph-mon[32753]: pgmap v8184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:24.238+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:50:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:24.238+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T15:50:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:24.239+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:50:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:24.239+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:50:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:23 smithi194 ceph-mon[33017]: pgmap v8185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14361.911677 secs 2024-04-26T15:50:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:23 smithi033 ceph-mon[27845]: pgmap v8185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14361.911677 secs 2024-04-26T15:50:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:23 smithi096 ceph-mon[32753]: pgmap v8185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14361.911677 secs 2024-04-26T15:50:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14364.717501 secs 2024-04-26T15:50:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14364.717501 secs 2024-04-26T15:50:25.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14364.717501 secs 2024-04-26T15:50:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:26 smithi194 ceph-mon[33017]: pgmap v8186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:26 smithi033 ceph-mon[27845]: pgmap v8186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:26 smithi096 ceph-mon[32753]: pgmap v8186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:28.249+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:28.249+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:28.249+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:50:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:28 smithi194 ceph-mon[33017]: pgmap v8187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:28 smithi033 ceph-mon[27845]: pgmap v8187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:28 smithi096 ceph-mon[32753]: pgmap v8187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14366.911764 secs 2024-04-26T15:50:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14366.911764 secs 2024-04-26T15:50:29.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14366.911764 secs 2024-04-26T15:50:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:30 smithi194 ceph-mon[33017]: pgmap v8188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14369.717581 secs 2024-04-26T15:50:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:30 smithi033 ceph-mon[27845]: pgmap v8188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14369.717581 secs 2024-04-26T15:50:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:30 smithi096 ceph-mon[32753]: pgmap v8188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14369.717581 secs 2024-04-26T15:50:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:32 smithi194 ceph-mon[33017]: pgmap v8189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:32 smithi033 ceph-mon[27845]: pgmap v8189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:32.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:32 smithi096 ceph-mon[32753]: pgmap v8189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:33.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14371.911847 secs 2024-04-26T15:50:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14371.911847 secs 2024-04-26T15:50:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14371.911847 secs 2024-04-26T15:50:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:34 smithi194 ceph-mon[33017]: pgmap v8190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:34 smithi033 ceph-mon[27845]: pgmap v8190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:34.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:34 smithi096 ceph-mon[32753]: pgmap v8190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14374.717719 secs 2024-04-26T15:50:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14374.717719 secs 2024-04-26T15:50:35.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14374.717719 secs 2024-04-26T15:50:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:36 smithi194 ceph-mon[33017]: pgmap v8191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:36 smithi033 ceph-mon[27845]: pgmap v8191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:36 smithi096 ceph-mon[32753]: pgmap v8191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:38 smithi194 ceph-mon[33017]: pgmap v8192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14376.911936 secs 2024-04-26T15:50:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:38 smithi033 ceph-mon[27845]: pgmap v8192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14376.911936 secs 2024-04-26T15:50:38.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:38 smithi096 ceph-mon[32753]: pgmap v8192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14376.911936 secs 2024-04-26T15:50:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:40 smithi194 ceph-mon[33017]: pgmap v8193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14379.717861 secs 2024-04-26T15:50:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:40 smithi033 ceph-mon[27845]: pgmap v8193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14379.717861 secs 2024-04-26T15:50:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:40 smithi096 ceph-mon[32753]: pgmap v8193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14379.717861 secs 2024-04-26T15:50:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:42 smithi194 ceph-mon[33017]: pgmap v8194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:42 smithi033 ceph-mon[27845]: pgmap v8194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:42.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:42 smithi096 ceph-mon[32753]: pgmap v8194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:43.265+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:43.265+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:43.265+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:50:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:50:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14381.912002 secs 2024-04-26T15:50:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:50:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14381.912002 secs 2024-04-26T15:50:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:50:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14381.912002 secs 2024-04-26T15:50:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:44 smithi033 ceph-mon[27845]: pgmap v8195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:44 smithi096 ceph-mon[32753]: pgmap v8195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:44 smithi194 ceph-mon[33017]: pgmap v8195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14384.717949 secs 2024-04-26T15:50:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14384.717949 secs 2024-04-26T15:50:45.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14384.717949 secs 2024-04-26T15:50:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: pgmap v8196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: pgmap v8196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: pgmap v8196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:47 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:50:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:47 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:50:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:47 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:50:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:47 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:50:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:47 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:50:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:47 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:50:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:47 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:50:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:47 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:50:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:47 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T15:50:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:47 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T15:50:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:47 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T15:50:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:47 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T15:50:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:48 smithi033 ceph-mon[27845]: pgmap v8197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14386.912087 secs 2024-04-26T15:50:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:48 smithi096 ceph-mon[32753]: pgmap v8197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14386.912087 secs 2024-04-26T15:50:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:48 smithi194 ceph-mon[33017]: pgmap v8197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:50:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14386.912087 secs 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:50:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:50:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:50:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:50 smithi033 ceph-mon[27845]: pgmap v8198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14389.718086 secs 2024-04-26T15:50:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:50 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:50:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:50 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:50:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:50 smithi033 ceph-mon[27845]: pgmap v8199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:50:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:50 smithi033 ceph-mon[27845]: pgmap v8200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:50:50.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:50 smithi096 ceph-mon[32753]: pgmap v8198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14389.718086 secs 2024-04-26T15:50:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:50 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:50:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:50 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:50:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:50 smithi096 ceph-mon[32753]: pgmap v8199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:50:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:50 smithi096 ceph-mon[32753]: pgmap v8200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:50:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:50 smithi194 ceph-mon[33017]: pgmap v8198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:50:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14389.718086 secs 2024-04-26T15:50:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:50 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T15:50:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:50 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T15:50:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:50 smithi194 ceph-mon[33017]: pgmap v8199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:50:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:50 smithi194 ceph-mon[33017]: pgmap v8200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:50:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:51 smithi033 ceph-mon[27845]: pgmap v8201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:51.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:51 smithi096 ceph-mon[32753]: pgmap v8201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:51 smithi194 ceph-mon[33017]: pgmap v8201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14391.912170 secs 2024-04-26T15:50:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:53 smithi033 ceph-mon[27845]: pgmap v8202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:54.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14391.912170 secs 2024-04-26T15:50:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:53 smithi096 ceph-mon[32753]: pgmap v8202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14391.912170 secs 2024-04-26T15:50:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:53 smithi194 ceph-mon[33017]: pgmap v8202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14394.718202 secs 2024-04-26T15:50:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14394.718202 secs 2024-04-26T15:50:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14394.718202 secs 2024-04-26T15:50:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:55 smithi033 ceph-mon[27845]: pgmap v8203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:55 smithi096 ceph-mon[32753]: pgmap v8203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:55 smithi194 ceph-mon[33017]: pgmap v8203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:57 smithi033 ceph-mon[27845]: pgmap v8204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:58.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:57 smithi096 ceph-mon[32753]: pgmap v8204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:58.279+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:58.279+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:50:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:50:58.279+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:50:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:57 smithi194 ceph-mon[33017]: pgmap v8204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:50:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14396.912229 secs 2024-04-26T15:50:59.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14396.912229 secs 2024-04-26T15:50:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14396.912229 secs 2024-04-26T15:51:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14399.718314 secs 2024-04-26T15:51:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:50:59 smithi033 ceph-mon[27845]: pgmap v8205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:51:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14399.718314 secs 2024-04-26T15:51:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:50:59 smithi096 ceph-mon[32753]: pgmap v8205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:51:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14399.718314 secs 2024-04-26T15:51:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:50:59 smithi194 ceph-mon[33017]: pgmap v8205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:51:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:01 smithi033 ceph-mon[27845]: pgmap v8206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:01 smithi096 ceph-mon[32753]: pgmap v8206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:01 smithi194 ceph-mon[33017]: pgmap v8206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14401.912302 secs 2024-04-26T15:51:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:03 smithi194 ceph-mon[33017]: pgmap v8207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14401.912302 secs 2024-04-26T15:51:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:03 smithi033 ceph-mon[27845]: pgmap v8207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:03.680 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14401.912302 secs 2024-04-26T15:51:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:03 smithi096 ceph-mon[32753]: pgmap v8207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14404.718390 secs 2024-04-26T15:51:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14404.718390 secs 2024-04-26T15:51:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14404.718390 secs 2024-04-26T15:51:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:05 smithi194 ceph-mon[33017]: pgmap v8208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:05 smithi033 ceph-mon[27845]: pgmap v8208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:05 smithi096 ceph-mon[32753]: pgmap v8208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:07 smithi033 ceph-mon[27845]: pgmap v8209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:07 smithi096 ceph-mon[32753]: pgmap v8209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:07 smithi194 ceph-mon[33017]: pgmap v8209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14406.912363 secs 2024-04-26T15:51:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14406.912363 secs 2024-04-26T15:51:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14406.912363 secs 2024-04-26T15:51:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14409.718553 secs 2024-04-26T15:51:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:09 smithi033 ceph-mon[27845]: pgmap v8210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14409.718553 secs 2024-04-26T15:51:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:09 smithi096 ceph-mon[32753]: pgmap v8210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14409.718553 secs 2024-04-26T15:51:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:09 smithi194 ceph-mon[33017]: pgmap v8210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:11 smithi033 ceph-mon[27845]: pgmap v8211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:12.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:11 smithi096 ceph-mon[32753]: pgmap v8211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:11 smithi194 ceph-mon[33017]: pgmap v8211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:13.293+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:13.293+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:13.293+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:51:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14411.912451 secs 2024-04-26T15:51:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:13 smithi033 ceph-mon[27845]: pgmap v8212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14411.912451 secs 2024-04-26T15:51:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:13 smithi096 ceph-mon[32753]: pgmap v8212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14411.912451 secs 2024-04-26T15:51:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:13 smithi194 ceph-mon[33017]: pgmap v8212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14414.718669 secs 2024-04-26T15:51:15.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14414.718669 secs 2024-04-26T15:51:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14414.718669 secs 2024-04-26T15:51:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:15 smithi033 ceph-mon[27845]: pgmap v8213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:15 smithi096 ceph-mon[32753]: pgmap v8213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:15 smithi194 ceph-mon[33017]: pgmap v8213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:17 smithi033 ceph-mon[27845]: pgmap v8214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:17 smithi096 ceph-mon[32753]: pgmap v8214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:17 smithi194 ceph-mon[33017]: pgmap v8214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14416.912540 secs 2024-04-26T15:51:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14416.912540 secs 2024-04-26T15:51:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14416.912540 secs 2024-04-26T15:51:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14419.718808 secs 2024-04-26T15:51:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:19 smithi194 ceph-mon[33017]: pgmap v8215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14419.718808 secs 2024-04-26T15:51:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:19 smithi033 ceph-mon[27845]: pgmap v8215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14419.718808 secs 2024-04-26T15:51:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:19 smithi096 ceph-mon[32753]: pgmap v8215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:21 smithi194 ceph-mon[33017]: pgmap v8216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:21 smithi033 ceph-mon[27845]: pgmap v8216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:21 smithi096 ceph-mon[32753]: pgmap v8216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14421.912622 secs 2024-04-26T15:51:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:23 smithi194 ceph-mon[33017]: pgmap v8217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14421.912622 secs 2024-04-26T15:51:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:23 smithi033 ceph-mon[27845]: pgmap v8217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14421.912622 secs 2024-04-26T15:51:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:23 smithi096 ceph-mon[32753]: pgmap v8217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14424.718944 secs 2024-04-26T15:51:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14424.718944 secs 2024-04-26T15:51:25.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14424.718944 secs 2024-04-26T15:51:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:25 smithi194 ceph-mon[33017]: pgmap v8218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:25 smithi033 ceph-mon[27845]: pgmap v8218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:26.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:25 smithi096 ceph-mon[32753]: pgmap v8218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:28.308+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:28.308+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:51:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:28.308+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:27 smithi194 ceph-mon[33017]: pgmap v8219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:27 smithi033 ceph-mon[27845]: pgmap v8219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:28.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:27 smithi096 ceph-mon[32753]: pgmap v8219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14426.912711 secs 2024-04-26T15:51:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14426.912711 secs 2024-04-26T15:51:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14426.912711 secs 2024-04-26T15:51:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14429.719061 secs 2024-04-26T15:51:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:29 smithi194 ceph-mon[33017]: pgmap v8220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14429.719061 secs 2024-04-26T15:51:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:29 smithi033 ceph-mon[27845]: pgmap v8220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14429.719061 secs 2024-04-26T15:51:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:29 smithi096 ceph-mon[32753]: pgmap v8220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:31 smithi194 ceph-mon[33017]: pgmap v8221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:31 smithi033 ceph-mon[27845]: pgmap v8221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:32.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:31 smithi096 ceph-mon[32753]: pgmap v8221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14431.912794 secs 2024-04-26T15:51:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:33 smithi194 ceph-mon[33017]: pgmap v8222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14431.912794 secs 2024-04-26T15:51:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:33 smithi033 ceph-mon[27845]: pgmap v8222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14431.912794 secs 2024-04-26T15:51:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:33 smithi096 ceph-mon[32753]: pgmap v8222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14434.719177 secs 2024-04-26T15:51:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14434.719177 secs 2024-04-26T15:51:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14434.719177 secs 2024-04-26T15:51:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:36 smithi194 ceph-mon[33017]: pgmap v8223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:36 smithi033 ceph-mon[27845]: pgmap v8223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:36.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:36 smithi096 ceph-mon[32753]: pgmap v8223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:38 smithi194 ceph-mon[33017]: pgmap v8224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:38 smithi033 ceph-mon[27845]: pgmap v8224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:38 smithi096 ceph-mon[32753]: pgmap v8224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14436.912883 secs 2024-04-26T15:51:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14436.912883 secs 2024-04-26T15:51:39.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14436.912883 secs 2024-04-26T15:51:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14439.719289 secs 2024-04-26T15:51:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:40 smithi194 ceph-mon[33017]: pgmap v8225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14439.719289 secs 2024-04-26T15:51:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:40 smithi033 ceph-mon[27845]: pgmap v8225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14439.719289 secs 2024-04-26T15:51:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:40 smithi096 ceph-mon[32753]: pgmap v8225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:42 smithi194 ceph-mon[33017]: pgmap v8226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:42 smithi033 ceph-mon[27845]: pgmap v8226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:42.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:42 smithi096 ceph-mon[32753]: pgmap v8226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:43.324+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:43.324+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:43.324+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:51:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14441.912977 secs 2024-04-26T15:51:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:44 smithi194 ceph-mon[33017]: pgmap v8227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14441.912977 secs 2024-04-26T15:51:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:44 smithi033 ceph-mon[27845]: pgmap v8227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14441.912977 secs 2024-04-26T15:51:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:44 smithi096 ceph-mon[32753]: pgmap v8227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14444.719407 secs 2024-04-26T15:51:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14444.719407 secs 2024-04-26T15:51:45.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14444.719407 secs 2024-04-26T15:51:46.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:46 smithi033 ceph-mon[27845]: pgmap v8228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:46 smithi194 ceph-mon[33017]: pgmap v8228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:46.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:46 smithi096 ceph-mon[32753]: pgmap v8228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:51:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:48 smithi194 ceph-mon[33017]: pgmap v8229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14446.913064 secs 2024-04-26T15:51:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:48 smithi033 ceph-mon[27845]: pgmap v8229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14446.913064 secs 2024-04-26T15:51:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:48 smithi096 ceph-mon[32753]: pgmap v8229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14446.913064 secs 2024-04-26T15:51:50.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14449.719486 secs 2024-04-26T15:51:50.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:50 smithi033 ceph-mon[27845]: pgmap v8230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:50.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:51:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14449.719486 secs 2024-04-26T15:51:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:50 smithi194 ceph-mon[33017]: pgmap v8230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:51:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14449.719486 secs 2024-04-26T15:51:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:50 smithi096 ceph-mon[32753]: pgmap v8230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:51:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:51:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:51:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:51:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:51:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:51:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:51:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:51:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:51:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:51:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:51:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:52 smithi194 ceph-mon[33017]: pgmap v8231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:51:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:52 smithi194 ceph-mon[33017]: pgmap v8232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:52 smithi033 ceph-mon[27845]: pgmap v8231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:51:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:52 smithi033 ceph-mon[27845]: pgmap v8232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:52 smithi096 ceph-mon[32753]: pgmap v8231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:51:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:52 smithi096 ceph-mon[32753]: pgmap v8232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14451.913152 secs 2024-04-26T15:51:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14451.913152 secs 2024-04-26T15:51:53.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14451.913152 secs 2024-04-26T15:51:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:54 smithi194 ceph-mon[33017]: pgmap v8233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:54 smithi033 ceph-mon[27845]: pgmap v8233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:54 smithi096 ceph-mon[32753]: pgmap v8233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14454.719601 secs 2024-04-26T15:51:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14454.719601 secs 2024-04-26T15:51:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14454.719601 secs 2024-04-26T15:51:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:56 smithi194 ceph-mon[33017]: pgmap v8234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:56 smithi033 ceph-mon[27845]: pgmap v8234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:56 smithi096 ceph-mon[32753]: pgmap v8234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:58.337+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:58.337+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:51:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:51:58.337+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:51:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:58 smithi194 ceph-mon[33017]: pgmap v8235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14456.913241 secs 2024-04-26T15:51:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:58 smithi033 ceph-mon[27845]: pgmap v8235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14456.913241 secs 2024-04-26T15:51:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:58 smithi096 ceph-mon[32753]: pgmap v8235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:51:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14456.913241 secs 2024-04-26T15:52:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14459.719737 secs 2024-04-26T15:52:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:00 smithi033 ceph-mon[27845]: pgmap v8236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:00.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14459.719737 secs 2024-04-26T15:52:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:00 smithi096 ceph-mon[32753]: pgmap v8236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14459.719737 secs 2024-04-26T15:52:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:00 smithi194 ceph-mon[33017]: pgmap v8236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:02.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:02 smithi194 ceph-mon[33017]: pgmap v8237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:52:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:02 smithi033 ceph-mon[27845]: pgmap v8237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:52:02.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:02 smithi096 ceph-mon[32753]: pgmap v8237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:52:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14461.913328 secs 2024-04-26T15:52:03.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14461.913328 secs 2024-04-26T15:52:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14461.913328 secs 2024-04-26T15:52:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:04 smithi033 ceph-mon[27845]: pgmap v8238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:04 smithi096 ceph-mon[32753]: pgmap v8238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:04 smithi194 ceph-mon[33017]: pgmap v8238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14464.719811 secs 2024-04-26T15:52:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14464.719811 secs 2024-04-26T15:52:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14464.719811 secs 2024-04-26T15:52:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:06 smithi033 ceph-mon[27845]: pgmap v8239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:06 smithi096 ceph-mon[32753]: pgmap v8239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:06 smithi194 ceph-mon[33017]: pgmap v8239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:07 smithi033 ceph-mon[27845]: pgmap v8240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:07 smithi096 ceph-mon[32753]: pgmap v8240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:07 smithi194 ceph-mon[33017]: pgmap v8240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14466.913417 secs 2024-04-26T15:52:08.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14466.913417 secs 2024-04-26T15:52:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14466.913417 secs 2024-04-26T15:52:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:09 smithi033 ceph-mon[27845]: pgmap v8241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:09 smithi096 ceph-mon[32753]: pgmap v8241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:09 smithi194 ceph-mon[33017]: pgmap v8241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14469.719909 secs 2024-04-26T15:52:10.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14469.719909 secs 2024-04-26T15:52:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14469.719909 secs 2024-04-26T15:52:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:11 smithi033 ceph-mon[27845]: pgmap v8242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:11 smithi096 ceph-mon[32753]: pgmap v8242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:11 smithi194 ceph-mon[33017]: pgmap v8242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:13.352+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:13.352+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:13.352+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:52:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:13 smithi033 ceph-mon[27845]: pgmap v8243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14471.913501 secs 2024-04-26T15:52:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:13 smithi096 ceph-mon[32753]: pgmap v8243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14471.913501 secs 2024-04-26T15:52:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:13 smithi194 ceph-mon[33017]: pgmap v8243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14471.913501 secs 2024-04-26T15:52:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14474.720007 secs 2024-04-26T15:52:14.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14474.720007 secs 2024-04-26T15:52:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14474.720007 secs 2024-04-26T15:52:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:15 smithi033 ceph-mon[27845]: pgmap v8244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:15.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:15 smithi096 ceph-mon[32753]: pgmap v8244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:15 smithi194 ceph-mon[33017]: pgmap v8244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:17 smithi033 ceph-mon[27845]: pgmap v8245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:17.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:17 smithi096 ceph-mon[32753]: pgmap v8245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:17 smithi194 ceph-mon[33017]: pgmap v8245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14476.913589 secs 2024-04-26T15:52:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14476.913589 secs 2024-04-26T15:52:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14476.913589 secs 2024-04-26T15:52:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14479.720127 secs 2024-04-26T15:52:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:19 smithi033 ceph-mon[27845]: pgmap v8246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14479.720127 secs 2024-04-26T15:52:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:19 smithi096 ceph-mon[32753]: pgmap v8246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14479.720127 secs 2024-04-26T15:52:20.136 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:19 smithi194 ceph-mon[33017]: pgmap v8246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:21 smithi033 ceph-mon[27845]: pgmap v8247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:21 smithi096 ceph-mon[32753]: pgmap v8247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:21 smithi194 ceph-mon[33017]: pgmap v8247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:23 smithi033 ceph-mon[27845]: pgmap v8248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14481.913676 secs 2024-04-26T15:52:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:23 smithi096 ceph-mon[32753]: pgmap v8248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14481.913676 secs 2024-04-26T15:52:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:23 smithi194 ceph-mon[33017]: pgmap v8248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14481.913676 secs 2024-04-26T15:52:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14484.720173 secs 2024-04-26T15:52:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14484.720173 secs 2024-04-26T15:52:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14484.720173 secs 2024-04-26T15:52:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:25 smithi096 ceph-mon[32753]: pgmap v8249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:25 smithi194 ceph-mon[33017]: pgmap v8249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:25 smithi033 ceph-mon[27845]: pgmap v8249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:27 smithi194 ceph-mon[33017]: pgmap v8250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:27 smithi033 ceph-mon[27845]: pgmap v8250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:27 smithi096 ceph-mon[32753]: pgmap v8250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:28.356+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:28.356+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:28.356+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:52:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14486.913778 secs 2024-04-26T15:52:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14486.913778 secs 2024-04-26T15:52:29.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14486.913778 secs 2024-04-26T15:52:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14489.720273 secs 2024-04-26T15:52:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:29 smithi194 ceph-mon[33017]: pgmap v8251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14489.720273 secs 2024-04-26T15:52:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:29 smithi033 ceph-mon[27845]: pgmap v8251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14489.720273 secs 2024-04-26T15:52:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:29 smithi096 ceph-mon[32753]: pgmap v8251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:31 smithi194 ceph-mon[33017]: pgmap v8252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:31 smithi033 ceph-mon[27845]: pgmap v8252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:31 smithi096 ceph-mon[32753]: pgmap v8252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:33 smithi194 ceph-mon[33017]: pgmap v8253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14491.913879 secs 2024-04-26T15:52:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:33 smithi033 ceph-mon[27845]: pgmap v8253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14491.913879 secs 2024-04-26T15:52:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:33 smithi096 ceph-mon[32753]: pgmap v8253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14491.913879 secs 2024-04-26T15:52:35.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14494.720388 secs 2024-04-26T15:52:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14494.720388 secs 2024-04-26T15:52:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14494.720388 secs 2024-04-26T15:52:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:35 smithi194 ceph-mon[33017]: pgmap v8254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:35 smithi033 ceph-mon[27845]: pgmap v8254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:35 smithi096 ceph-mon[32753]: pgmap v8254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:37 smithi194 ceph-mon[33017]: pgmap v8255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:37 smithi033 ceph-mon[27845]: pgmap v8255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:37 smithi096 ceph-mon[32753]: pgmap v8255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14496.913973 secs 2024-04-26T15:52:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14496.913973 secs 2024-04-26T15:52:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14496.913973 secs 2024-04-26T15:52:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14499.720506 secs 2024-04-26T15:52:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:39 smithi194 ceph-mon[33017]: pgmap v8256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14499.720506 secs 2024-04-26T15:52:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:39 smithi033 ceph-mon[27845]: pgmap v8256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14499.720506 secs 2024-04-26T15:52:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:39 smithi096 ceph-mon[32753]: pgmap v8256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:41 smithi194 ceph-mon[33017]: pgmap v8257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:41 smithi033 ceph-mon[27845]: pgmap v8257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:41 smithi096 ceph-mon[32753]: pgmap v8257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:43.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:43.372+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:43.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:43.373+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:43.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:43.373+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:52:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:43 smithi194 ceph-mon[33017]: pgmap v8258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14501.914053 secs 2024-04-26T15:52:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:43 smithi033 ceph-mon[27845]: pgmap v8258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14501.914053 secs 2024-04-26T15:52:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:43 smithi096 ceph-mon[32753]: pgmap v8258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14501.914053 secs 2024-04-26T15:52:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14504.720641 secs 2024-04-26T15:52:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14504.720641 secs 2024-04-26T15:52:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14504.720641 secs 2024-04-26T15:52:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:45 smithi194 ceph-mon[33017]: pgmap v8259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:45 smithi033 ceph-mon[27845]: pgmap v8259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:46.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:45 smithi096 ceph-mon[32753]: pgmap v8259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:47 smithi194 ceph-mon[33017]: pgmap v8260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:47 smithi033 ceph-mon[27845]: pgmap v8260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:47 smithi096 ceph-mon[32753]: pgmap v8260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14506.914138 secs 2024-04-26T15:52:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14506.914138 secs 2024-04-26T15:52:49.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14506.914138 secs 2024-04-26T15:52:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14509.720779 secs 2024-04-26T15:52:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:49 smithi194 ceph-mon[33017]: pgmap v8261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14509.720779 secs 2024-04-26T15:52:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:49 smithi033 ceph-mon[27845]: pgmap v8261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14509.720779 secs 2024-04-26T15:52:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:49 smithi096 ceph-mon[32753]: pgmap v8261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:51 smithi194 ceph-mon[33017]: pgmap v8262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:52:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:51 smithi033 ceph-mon[27845]: pgmap v8262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:52:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:51 smithi096 ceph-mon[32753]: pgmap v8262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:52:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:52:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:52:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:52:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:52:53.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:52:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:52:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:52:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:52:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:52:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:52:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:53 smithi194 ceph-mon[33017]: pgmap v8263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:53 smithi194 ceph-mon[33017]: pgmap v8264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14511.914238 secs 2024-04-26T15:52:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:53 smithi033 ceph-mon[27845]: pgmap v8263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:53 smithi033 ceph-mon[27845]: pgmap v8264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14511.914238 secs 2024-04-26T15:52:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:53 smithi096 ceph-mon[32753]: pgmap v8263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:52:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:53 smithi096 ceph-mon[32753]: pgmap v8264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14511.914238 secs 2024-04-26T15:52:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14514.720856 secs 2024-04-26T15:52:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14514.720856 secs 2024-04-26T15:52:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14514.720856 secs 2024-04-26T15:52:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:55 smithi194 ceph-mon[33017]: pgmap v8265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:52:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:55 smithi033 ceph-mon[27845]: pgmap v8265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:52:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:55 smithi096 ceph-mon[32753]: pgmap v8265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:52:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:57 smithi194 ceph-mon[33017]: pgmap v8266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:57 smithi033 ceph-mon[27845]: pgmap v8266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:57 smithi096 ceph-mon[32753]: pgmap v8266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:52:58.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:58.385+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:58.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:58.385+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:52:58.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:52:58.385+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:52:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14516.914320 secs 2024-04-26T15:52:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14516.914320 secs 2024-04-26T15:52:59.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14516.914320 secs 2024-04-26T15:53:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:59 smithi194 ceph-mon[33017]: pgmap v8267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:52:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14519.721006 secs 2024-04-26T15:53:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:59 smithi033 ceph-mon[27845]: pgmap v8267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:52:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14519.721006 secs 2024-04-26T15:53:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:59 smithi096 ceph-mon[32753]: pgmap v8267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:52:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14519.721006 secs 2024-04-26T15:53:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:01 smithi194 ceph-mon[33017]: pgmap v8268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:01 smithi033 ceph-mon[27845]: pgmap v8268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:01 smithi096 ceph-mon[32753]: pgmap v8268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:03 smithi194 ceph-mon[33017]: pgmap v8269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:53:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14521.914400 secs 2024-04-26T15:53:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:03 smithi033 ceph-mon[27845]: pgmap v8269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:53:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14521.914400 secs 2024-04-26T15:53:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:03 smithi096 ceph-mon[32753]: pgmap v8269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:53:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14521.914400 secs 2024-04-26T15:53:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14524.721135 secs 2024-04-26T15:53:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14524.721135 secs 2024-04-26T15:53:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14524.721135 secs 2024-04-26T15:53:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:05 smithi033 ceph-mon[27845]: pgmap v8270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:05 smithi096 ceph-mon[32753]: pgmap v8270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:05 smithi194 ceph-mon[33017]: pgmap v8270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:08.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:07 smithi033 ceph-mon[27845]: pgmap v8271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:07 smithi096 ceph-mon[32753]: pgmap v8271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:07 smithi194 ceph-mon[33017]: pgmap v8271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14526.914482 secs 2024-04-26T15:53:09.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14526.914482 secs 2024-04-26T15:53:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14526.914482 secs 2024-04-26T15:53:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:09 smithi033 ceph-mon[27845]: pgmap v8272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14529.721198 secs 2024-04-26T15:53:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:09 smithi096 ceph-mon[32753]: pgmap v8272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14529.721198 secs 2024-04-26T15:53:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:09 smithi194 ceph-mon[33017]: pgmap v8272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14529.721198 secs 2024-04-26T15:53:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:11 smithi033 ceph-mon[27845]: pgmap v8273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:12.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:11 smithi096 ceph-mon[32753]: pgmap v8273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:11 smithi194 ceph-mon[33017]: pgmap v8273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:13.390+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:13.390+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:13.390+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:53:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:13 smithi033 ceph-mon[27845]: pgmap v8274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14531.914566 secs 2024-04-26T15:53:14.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:13 smithi096 ceph-mon[32753]: pgmap v8274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14531.914566 secs 2024-04-26T15:53:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:13 smithi194 ceph-mon[33017]: pgmap v8274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14531.914566 secs 2024-04-26T15:53:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14534.721320 secs 2024-04-26T15:53:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14534.721320 secs 2024-04-26T15:53:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14534.721320 secs 2024-04-26T15:53:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:15 smithi033 ceph-mon[27845]: pgmap v8275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:15 smithi096 ceph-mon[32753]: pgmap v8275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:15 smithi194 ceph-mon[33017]: pgmap v8275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:17 smithi033 ceph-mon[27845]: pgmap v8276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:17 smithi096 ceph-mon[32753]: pgmap v8276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:17 smithi194 ceph-mon[33017]: pgmap v8276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14536.914649 secs 2024-04-26T15:53:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14536.914649 secs 2024-04-26T15:53:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14536.914649 secs 2024-04-26T15:53:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:19 smithi033 ceph-mon[27845]: pgmap v8277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14539.721399 secs 2024-04-26T15:53:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:19 smithi096 ceph-mon[32753]: pgmap v8277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14539.721399 secs 2024-04-26T15:53:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:19 smithi194 ceph-mon[33017]: pgmap v8277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14539.721399 secs 2024-04-26T15:53:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:21 smithi033 ceph-mon[27845]: pgmap v8278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:21 smithi096 ceph-mon[32753]: pgmap v8278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:21 smithi194 ceph-mon[33017]: pgmap v8278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:23 smithi096 ceph-mon[32753]: pgmap v8279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14541.914731 secs 2024-04-26T15:53:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:24.269+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:53:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:24.269+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T15:53:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:24.269+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:53:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:24.270+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:53:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:23 smithi194 ceph-mon[33017]: pgmap v8279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14541.914731 secs 2024-04-26T15:53:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:23 smithi033 ceph-mon[27845]: pgmap v8279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14541.914731 secs 2024-04-26T15:53:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14544.721512 secs 2024-04-26T15:53:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14544.721512 secs 2024-04-26T15:53:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14544.721512 secs 2024-04-26T15:53:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:25 smithi194 ceph-mon[33017]: pgmap v8280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:25 smithi033 ceph-mon[27845]: pgmap v8280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:25 smithi096 ceph-mon[32753]: pgmap v8280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:27 smithi194 ceph-mon[33017]: pgmap v8281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:28.403+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:28.403+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:28.403+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:53:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:27 smithi033 ceph-mon[27845]: pgmap v8281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:27 smithi096 ceph-mon[32753]: pgmap v8281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14546.914815 secs 2024-04-26T15:53:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14546.914815 secs 2024-04-26T15:53:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14546.914815 secs 2024-04-26T15:53:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:29 smithi194 ceph-mon[33017]: pgmap v8282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14549.721615 secs 2024-04-26T15:53:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:29 smithi033 ceph-mon[27845]: pgmap v8282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14549.721615 secs 2024-04-26T15:53:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:29 smithi096 ceph-mon[32753]: pgmap v8282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14549.721615 secs 2024-04-26T15:53:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:31 smithi194 ceph-mon[33017]: pgmap v8283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:31 smithi033 ceph-mon[27845]: pgmap v8283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:31 smithi096 ceph-mon[32753]: pgmap v8283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:33 smithi194 ceph-mon[33017]: pgmap v8284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14551.914899 secs 2024-04-26T15:53:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:33 smithi033 ceph-mon[27845]: pgmap v8284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14551.914899 secs 2024-04-26T15:53:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:33 smithi096 ceph-mon[32753]: pgmap v8284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14551.914899 secs 2024-04-26T15:53:35.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14554.721690 secs 2024-04-26T15:53:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14554.721690 secs 2024-04-26T15:53:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14554.721690 secs 2024-04-26T15:53:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:35 smithi194 ceph-mon[33017]: pgmap v8285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:35 smithi033 ceph-mon[27845]: pgmap v8285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:35 smithi096 ceph-mon[32753]: pgmap v8285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:37 smithi194 ceph-mon[33017]: pgmap v8286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:37 smithi033 ceph-mon[27845]: pgmap v8286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:37 smithi096 ceph-mon[32753]: pgmap v8286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14556.915003 secs 2024-04-26T15:53:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14556.915003 secs 2024-04-26T15:53:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14556.915003 secs 2024-04-26T15:53:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:40 smithi194 ceph-mon[33017]: pgmap v8287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14559.721807 secs 2024-04-26T15:53:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:40 smithi033 ceph-mon[27845]: pgmap v8287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14559.721807 secs 2024-04-26T15:53:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:40 smithi096 ceph-mon[32753]: pgmap v8287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14559.721807 secs 2024-04-26T15:53:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:42 smithi194 ceph-mon[33017]: pgmap v8288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:42 smithi033 ceph-mon[27845]: pgmap v8288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:42 smithi096 ceph-mon[32753]: pgmap v8288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:43.415+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:43.415+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:43.415+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:53:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:44 smithi194 ceph-mon[33017]: pgmap v8289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14561.915064 secs 2024-04-26T15:53:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:44 smithi033 ceph-mon[27845]: pgmap v8289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14561.915064 secs 2024-04-26T15:53:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:44 smithi096 ceph-mon[32753]: pgmap v8289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14561.915064 secs 2024-04-26T15:53:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14564.721942 secs 2024-04-26T15:53:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14564.721942 secs 2024-04-26T15:53:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14564.721942 secs 2024-04-26T15:53:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:46 smithi194 ceph-mon[33017]: pgmap v8290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:46 smithi033 ceph-mon[27845]: pgmap v8290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:46 smithi096 ceph-mon[32753]: pgmap v8290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:53:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:48 smithi194 ceph-mon[33017]: pgmap v8291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:48 smithi033 ceph-mon[27845]: pgmap v8291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:48 smithi096 ceph-mon[32753]: pgmap v8291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14566.915152 secs 2024-04-26T15:53:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14566.915152 secs 2024-04-26T15:53:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14566.915152 secs 2024-04-26T15:53:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:50 smithi194 ceph-mon[33017]: pgmap v8292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14569.722005 secs 2024-04-26T15:53:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:50 smithi033 ceph-mon[27845]: pgmap v8292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14569.722005 secs 2024-04-26T15:53:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:50 smithi096 ceph-mon[32753]: pgmap v8292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14569.722005 secs 2024-04-26T15:53:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:52 smithi194 ceph-mon[33017]: pgmap v8293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:52 smithi033 ceph-mon[27845]: pgmap v8293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:52 smithi096 ceph-mon[32753]: pgmap v8293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:53:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:53:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14571.915221 secs 2024-04-26T15:53:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:53:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14571.915221 secs 2024-04-26T15:53:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:53:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14571.915221 secs 2024-04-26T15:53:54.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:54 smithi033 ceph-mon[27845]: pgmap v8294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:54.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14574.722091 secs 2024-04-26T15:53:54.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:53:54.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:53:54.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:53:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:54 smithi194 ceph-mon[33017]: pgmap v8294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14574.722091 secs 2024-04-26T15:53:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:53:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:53:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:53:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:54 smithi096 ceph-mon[32753]: pgmap v8294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14574.722091 secs 2024-04-26T15:53:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:53:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:53:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:53:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:55 smithi194 ceph-mon[33017]: pgmap v8295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:55 smithi194 ceph-mon[33017]: pgmap v8296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:53:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:55 smithi033 ceph-mon[27845]: pgmap v8295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:55 smithi033 ceph-mon[27845]: pgmap v8296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:53:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:55 smithi096 ceph-mon[32753]: pgmap v8295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:53:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:55 smithi096 ceph-mon[32753]: pgmap v8296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:53:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:57 smithi194 ceph-mon[33017]: pgmap v8297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:57 smithi033 ceph-mon[27845]: pgmap v8297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:57 smithi096 ceph-mon[32753]: pgmap v8297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14576.915303 secs 2024-04-26T15:53:58.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:58.418+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:58.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:58.419+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:53:58.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:53:58.419+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:53:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14576.915303 secs 2024-04-26T15:53:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14576.915303 secs 2024-04-26T15:53:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:59 smithi194 ceph-mon[33017]: pgmap v8298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14579.722212 secs 2024-04-26T15:53:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:59 smithi033 ceph-mon[27845]: pgmap v8298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14579.722212 secs 2024-04-26T15:53:59.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:59 smithi096 ceph-mon[32753]: pgmap v8298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:53:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14579.722212 secs 2024-04-26T15:54:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:01 smithi194 ceph-mon[33017]: pgmap v8299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:01 smithi033 ceph-mon[27845]: pgmap v8299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:01.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:01 smithi096 ceph-mon[32753]: pgmap v8299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:03 smithi194 ceph-mon[33017]: pgmap v8300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14581.915376 secs 2024-04-26T15:54:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:03 smithi033 ceph-mon[27845]: pgmap v8300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14581.915376 secs 2024-04-26T15:54:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:03 smithi096 ceph-mon[32753]: pgmap v8300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14581.915376 secs 2024-04-26T15:54:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14584.722334 secs 2024-04-26T15:54:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14584.722334 secs 2024-04-26T15:54:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14584.722334 secs 2024-04-26T15:54:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:05 smithi033 ceph-mon[27845]: pgmap v8301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:54:05.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:05 smithi096 ceph-mon[32753]: pgmap v8301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:54:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:05 smithi194 ceph-mon[33017]: pgmap v8301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:54:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:07 smithi033 ceph-mon[27845]: pgmap v8302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:07 smithi096 ceph-mon[32753]: pgmap v8302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:07 smithi194 ceph-mon[33017]: pgmap v8302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14586.915461 secs 2024-04-26T15:54:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14586.915461 secs 2024-04-26T15:54:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14586.915461 secs 2024-04-26T15:54:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:09 smithi033 ceph-mon[27845]: pgmap v8303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14589.722448 secs 2024-04-26T15:54:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:09 smithi096 ceph-mon[32753]: pgmap v8303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14589.722448 secs 2024-04-26T15:54:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:09 smithi194 ceph-mon[33017]: pgmap v8303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14589.722448 secs 2024-04-26T15:54:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:11 smithi033 ceph-mon[27845]: pgmap v8304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:11 smithi096 ceph-mon[32753]: pgmap v8304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:11 smithi194 ceph-mon[33017]: pgmap v8304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:13 smithi033 ceph-mon[27845]: pgmap v8305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14591.915546 secs 2024-04-26T15:54:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:13 smithi096 ceph-mon[32753]: pgmap v8305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14591.915546 secs 2024-04-26T15:54:13.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:13.434+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:13.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:13.434+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:13.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:13.434+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:54:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:13 smithi194 ceph-mon[33017]: pgmap v8305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14591.915546 secs 2024-04-26T15:54:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14594.722570 secs 2024-04-26T15:54:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14594.722570 secs 2024-04-26T15:54:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14594.722570 secs 2024-04-26T15:54:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:15 smithi033 ceph-mon[27845]: pgmap v8306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:15 smithi096 ceph-mon[32753]: pgmap v8306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:15 smithi194 ceph-mon[33017]: pgmap v8306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:17 smithi033 ceph-mon[27845]: pgmap v8307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:17 smithi096 ceph-mon[32753]: pgmap v8307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:17 smithi194 ceph-mon[33017]: pgmap v8307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14596.915628 secs 2024-04-26T15:54:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14596.915628 secs 2024-04-26T15:54:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14596.915628 secs 2024-04-26T15:54:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:19 smithi033 ceph-mon[27845]: pgmap v8308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14599.722688 secs 2024-04-26T15:54:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:19 smithi096 ceph-mon[32753]: pgmap v8308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14599.722688 secs 2024-04-26T15:54:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:19 smithi194 ceph-mon[33017]: pgmap v8308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14599.722688 secs 2024-04-26T15:54:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:20 smithi096 ceph-mon[32753]: pgmap v8309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:20 smithi194 ceph-mon[33017]: pgmap v8309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:20 smithi033 ceph-mon[27845]: pgmap v8309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:22 smithi194 ceph-mon[33017]: pgmap v8310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:22 smithi033 ceph-mon[27845]: pgmap v8310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:22 smithi096 ceph-mon[32753]: pgmap v8310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14601.915707 secs 2024-04-26T15:54:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14604.722823 secs 2024-04-26T15:54:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14601.915707 secs 2024-04-26T15:54:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14604.722823 secs 2024-04-26T15:54:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14601.915707 secs 2024-04-26T15:54:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14604.722823 secs 2024-04-26T15:54:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:24 smithi194 ceph-mon[33017]: pgmap v8311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:24 smithi033 ceph-mon[27845]: pgmap v8311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:24 smithi096 ceph-mon[32753]: pgmap v8311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:26 smithi194 ceph-mon[33017]: pgmap v8312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:26 smithi033 ceph-mon[27845]: pgmap v8312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:26 smithi096 ceph-mon[32753]: pgmap v8312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:28.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:28.448+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:28.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:28.448+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:28.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:28.448+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:54:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14606.915791 secs 2024-04-26T15:54:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:29 smithi194 ceph-mon[33017]: pgmap v8313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14609.722946 secs 2024-04-26T15:54:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14606.915791 secs 2024-04-26T15:54:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:29 smithi033 ceph-mon[27845]: pgmap v8313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14609.722946 secs 2024-04-26T15:54:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14606.915791 secs 2024-04-26T15:54:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:29 smithi096 ceph-mon[32753]: pgmap v8313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14609.722946 secs 2024-04-26T15:54:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:31 smithi194 ceph-mon[33017]: pgmap v8314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:31 smithi033 ceph-mon[27845]: pgmap v8314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:31 smithi096 ceph-mon[32753]: pgmap v8314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:33 smithi194 ceph-mon[33017]: pgmap v8315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:33 smithi033 ceph-mon[27845]: pgmap v8315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:33 smithi096 ceph-mon[32753]: pgmap v8315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14611.915873 secs 2024-04-26T15:54:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14614.723060 secs 2024-04-26T15:54:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14611.915873 secs 2024-04-26T15:54:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14614.723060 secs 2024-04-26T15:54:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14611.915873 secs 2024-04-26T15:54:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14614.723060 secs 2024-04-26T15:54:35.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:35 smithi096 ceph-mon[32753]: pgmap v8316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:35 smithi194 ceph-mon[33017]: pgmap v8316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:35 smithi033 ceph-mon[27845]: pgmap v8316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:37 smithi194 ceph-mon[33017]: pgmap v8317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:37 smithi033 ceph-mon[27845]: pgmap v8317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:37 smithi096 ceph-mon[32753]: pgmap v8317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14616.915955 secs 2024-04-26T15:54:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:39 smithi194 ceph-mon[33017]: pgmap v8318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14619.723188 secs 2024-04-26T15:54:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14616.915955 secs 2024-04-26T15:54:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:39 smithi033 ceph-mon[27845]: pgmap v8318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14619.723188 secs 2024-04-26T15:54:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14616.915955 secs 2024-04-26T15:54:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:39 smithi096 ceph-mon[32753]: pgmap v8318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14619.723188 secs 2024-04-26T15:54:41.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:41 smithi033 ceph-mon[27845]: pgmap v8319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:41 smithi194 ceph-mon[33017]: pgmap v8319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:41 smithi096 ceph-mon[32753]: pgmap v8319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:43 smithi194 ceph-mon[33017]: pgmap v8320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:43 smithi033 ceph-mon[27845]: pgmap v8320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:43 smithi096 ceph-mon[32753]: pgmap v8320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:43.456+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:43.456+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:43.456+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:54:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14621.916039 secs 2024-04-26T15:54:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14624.723282 secs 2024-04-26T15:54:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14621.916039 secs 2024-04-26T15:54:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14624.723282 secs 2024-04-26T15:54:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14621.916039 secs 2024-04-26T15:54:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14624.723282 secs 2024-04-26T15:54:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:45 smithi194 ceph-mon[33017]: pgmap v8321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:45 smithi033 ceph-mon[27845]: pgmap v8321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:45 smithi096 ceph-mon[32753]: pgmap v8321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:47 smithi194 ceph-mon[33017]: pgmap v8322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:47 smithi033 ceph-mon[27845]: pgmap v8322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:47 smithi096 ceph-mon[32753]: pgmap v8322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:48.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14626.916122 secs 2024-04-26T15:54:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14626.916122 secs 2024-04-26T15:54:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14626.916122 secs 2024-04-26T15:54:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:49 smithi194 ceph-mon[33017]: pgmap v8323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14629.723404 secs 2024-04-26T15:54:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:49 smithi033 ceph-mon[27845]: pgmap v8323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14629.723404 secs 2024-04-26T15:54:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:49 smithi096 ceph-mon[32753]: pgmap v8323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14629.723404 secs 2024-04-26T15:54:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:51 smithi194 ceph-mon[33017]: pgmap v8324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:51 smithi033 ceph-mon[27845]: pgmap v8324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:51 smithi096 ceph-mon[32753]: pgmap v8324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:54:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:53 smithi194 ceph-mon[33017]: pgmap v8325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14631.916207 secs 2024-04-26T15:54:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:53 smithi033 ceph-mon[27845]: pgmap v8325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14631.916207 secs 2024-04-26T15:54:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:53 smithi096 ceph-mon[32753]: pgmap v8325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14631.916207 secs 2024-04-26T15:54:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14634.723523 secs 2024-04-26T15:54:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14634.723523 secs 2024-04-26T15:54:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14634.723523 secs 2024-04-26T15:54:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:55 smithi194 ceph-mon[33017]: pgmap v8326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:54:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:55 smithi033 ceph-mon[27845]: pgmap v8326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:54:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:55 smithi096 ceph-mon[32753]: pgmap v8326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:54:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:54:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:54:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:54:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:54:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:54:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:54:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:54:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:54:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:54:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:54:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:57 smithi033 ceph-mon[27845]: pgmap v8327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:54:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:57 smithi033 ceph-mon[27845]: pgmap v8328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:57 smithi096 ceph-mon[32753]: pgmap v8327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:54:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:57 smithi096 ceph-mon[32753]: pgmap v8328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:57 smithi194 ceph-mon[33017]: pgmap v8327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:54:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:57 smithi194 ceph-mon[33017]: pgmap v8328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14636.916291 secs 2024-04-26T15:54:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14636.916291 secs 2024-04-26T15:54:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:58.473+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:58.473+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:54:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:54:58.473+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:54:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14636.916291 secs 2024-04-26T15:54:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:59 smithi033 ceph-mon[27845]: pgmap v8329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14639.723589 secs 2024-04-26T15:54:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:59 smithi096 ceph-mon[32753]: pgmap v8329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14639.723589 secs 2024-04-26T15:54:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:59 smithi194 ceph-mon[33017]: pgmap v8329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:54:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14639.723589 secs 2024-04-26T15:55:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:01 smithi033 ceph-mon[27845]: pgmap v8330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:01 smithi096 ceph-mon[32753]: pgmap v8330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:01 smithi194 ceph-mon[33017]: pgmap v8330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:03 smithi033 ceph-mon[27845]: pgmap v8331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14641.916371 secs 2024-04-26T15:55:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:03 smithi096 ceph-mon[32753]: pgmap v8331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14641.916371 secs 2024-04-26T15:55:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:03 smithi194 ceph-mon[33017]: pgmap v8331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14641.916371 secs 2024-04-26T15:55:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14644.723737 secs 2024-04-26T15:55:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14644.723737 secs 2024-04-26T15:55:04.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14644.723737 secs 2024-04-26T15:55:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:05 smithi033 ceph-mon[27845]: pgmap v8332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:05 smithi096 ceph-mon[32753]: pgmap v8332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:05 smithi194 ceph-mon[33017]: pgmap v8332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:07 smithi033 ceph-mon[27845]: pgmap v8333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:55:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:07 smithi096 ceph-mon[32753]: pgmap v8333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:55:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:07 smithi194 ceph-mon[33017]: pgmap v8333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:55:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14646.916454 secs 2024-04-26T15:55:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14646.916454 secs 2024-04-26T15:55:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14646.916454 secs 2024-04-26T15:55:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:09 smithi033 ceph-mon[27845]: pgmap v8334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14649.723882 secs 2024-04-26T15:55:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:09 smithi096 ceph-mon[32753]: pgmap v8334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14649.723882 secs 2024-04-26T15:55:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:09 smithi194 ceph-mon[33017]: pgmap v8334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14649.723882 secs 2024-04-26T15:55:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:10 smithi194 ceph-mon[33017]: pgmap v8335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:10 smithi033 ceph-mon[27845]: pgmap v8335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:10 smithi096 ceph-mon[32753]: pgmap v8335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:12 smithi194 ceph-mon[33017]: pgmap v8336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:12 smithi033 ceph-mon[27845]: pgmap v8336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:12 smithi096 ceph-mon[32753]: pgmap v8336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:13.488+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:13.488+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:13.488+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14651.916535 secs 2024-04-26T15:55:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14654.723998 secs 2024-04-26T15:55:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14651.916535 secs 2024-04-26T15:55:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14654.723998 secs 2024-04-26T15:55:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14651.916535 secs 2024-04-26T15:55:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14654.723998 secs 2024-04-26T15:55:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:14 smithi033 ceph-mon[27845]: pgmap v8337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:14 smithi096 ceph-mon[32753]: pgmap v8337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:14 smithi194 ceph-mon[33017]: pgmap v8337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:16.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:16 smithi033 ceph-mon[27845]: pgmap v8338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:16 smithi096 ceph-mon[32753]: pgmap v8338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:16 smithi194 ceph-mon[33017]: pgmap v8338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:18 smithi033 ceph-mon[27845]: pgmap v8339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14656.916618 secs 2024-04-26T15:55:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14659.724165 secs 2024-04-26T15:55:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:18 smithi096 ceph-mon[32753]: pgmap v8339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14656.916618 secs 2024-04-26T15:55:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14659.724165 secs 2024-04-26T15:55:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:18 smithi194 ceph-mon[33017]: pgmap v8339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14656.916618 secs 2024-04-26T15:55:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14659.724165 secs 2024-04-26T15:55:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:20 smithi033 ceph-mon[27845]: pgmap v8340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:20 smithi096 ceph-mon[32753]: pgmap v8340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:20 smithi194 ceph-mon[33017]: pgmap v8340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:22 smithi033 ceph-mon[27845]: pgmap v8341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:22 smithi096 ceph-mon[32753]: pgmap v8341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:22 smithi194 ceph-mon[33017]: pgmap v8341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14661.916700 secs 2024-04-26T15:55:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14664.724285 secs 2024-04-26T15:55:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14661.916700 secs 2024-04-26T15:55:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14664.724285 secs 2024-04-26T15:55:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14661.916700 secs 2024-04-26T15:55:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14664.724285 secs 2024-04-26T15:55:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:24 smithi033 ceph-mon[27845]: pgmap v8342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:24 smithi096 ceph-mon[32753]: pgmap v8342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:24 smithi194 ceph-mon[33017]: pgmap v8342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:26.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:26 smithi033 ceph-mon[27845]: pgmap v8343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:26 smithi096 ceph-mon[32753]: pgmap v8343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:26 smithi194 ceph-mon[33017]: pgmap v8343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:28.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:28.503+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:28.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:28.503+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:55:28.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:28.503+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:28 smithi033 ceph-mon[27845]: pgmap v8344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14666.916785 secs 2024-04-26T15:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14669.724410 secs 2024-04-26T15:55:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:28 smithi096 ceph-mon[32753]: pgmap v8344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14666.916785 secs 2024-04-26T15:55:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14669.724410 secs 2024-04-26T15:55:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:28 smithi194 ceph-mon[33017]: pgmap v8344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14666.916785 secs 2024-04-26T15:55:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14669.724410 secs 2024-04-26T15:55:30.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:30 smithi033 ceph-mon[27845]: pgmap v8345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:30 smithi096 ceph-mon[32753]: pgmap v8345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:30 smithi194 ceph-mon[33017]: pgmap v8345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:32 smithi033 ceph-mon[27845]: pgmap v8346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:32 smithi096 ceph-mon[32753]: pgmap v8346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:32 smithi194 ceph-mon[33017]: pgmap v8346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14671.916870 secs 2024-04-26T15:55:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14674.724525 secs 2024-04-26T15:55:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14671.916870 secs 2024-04-26T15:55:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14674.724525 secs 2024-04-26T15:55:34.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14671.916870 secs 2024-04-26T15:55:34.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14674.724525 secs 2024-04-26T15:55:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:34 smithi096 ceph-mon[32753]: pgmap v8347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:34 smithi194 ceph-mon[33017]: pgmap v8347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:34 smithi033 ceph-mon[27845]: pgmap v8347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:36 smithi096 ceph-mon[32753]: pgmap v8348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:36 smithi194 ceph-mon[33017]: pgmap v8348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:36 smithi033 ceph-mon[27845]: pgmap v8348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:38 smithi194 ceph-mon[33017]: pgmap v8349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14676.916954 secs 2024-04-26T15:55:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14679.724642 secs 2024-04-26T15:55:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:38 smithi033 ceph-mon[27845]: pgmap v8349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14676.916954 secs 2024-04-26T15:55:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14679.724642 secs 2024-04-26T15:55:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:38 smithi096 ceph-mon[32753]: pgmap v8349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:55:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14676.916954 secs 2024-04-26T15:55:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14679.724642 secs 2024-04-26T15:55:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:40 smithi194 ceph-mon[33017]: pgmap v8350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:40 smithi033 ceph-mon[27845]: pgmap v8350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:40 smithi096 ceph-mon[32753]: pgmap v8350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:55:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:42 smithi194 ceph-mon[33017]: pgmap v8351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:42 smithi033 ceph-mon[27845]: pgmap v8351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:42 smithi096 ceph-mon[32753]: pgmap v8351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:43.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:43.519+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:43.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:43.519+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:43.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:43.519+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:55:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14681.917004 secs 2024-04-26T15:55:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14684.724756 secs 2024-04-26T15:55:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14681.917004 secs 2024-04-26T15:55:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14684.724756 secs 2024-04-26T15:55:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14681.917004 secs 2024-04-26T15:55:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14684.724756 secs 2024-04-26T15:55:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:44 smithi194 ceph-mon[33017]: pgmap v8352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:44 smithi033 ceph-mon[27845]: pgmap v8352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:44 smithi096 ceph-mon[32753]: pgmap v8352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:46 smithi194 ceph-mon[33017]: pgmap v8353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:47.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:46 smithi033 ceph-mon[27845]: pgmap v8353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:46 smithi096 ceph-mon[32753]: pgmap v8353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:48 smithi033 ceph-mon[27845]: pgmap v8354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14686.917085 secs 2024-04-26T15:55:48.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14689.724876 secs 2024-04-26T15:55:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:48 smithi194 ceph-mon[33017]: pgmap v8354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14686.917085 secs 2024-04-26T15:55:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14689.724876 secs 2024-04-26T15:55:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:48 smithi096 ceph-mon[32753]: pgmap v8354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14686.917085 secs 2024-04-26T15:55:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14689.724876 secs 2024-04-26T15:55:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:50 smithi194 ceph-mon[33017]: pgmap v8355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:51.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:50 smithi033 ceph-mon[27845]: pgmap v8355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:50 smithi096 ceph-mon[32753]: pgmap v8355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:52 smithi194 ceph-mon[33017]: pgmap v8356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:52 smithi033 ceph-mon[27845]: pgmap v8356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:52 smithi096 ceph-mon[32753]: pgmap v8356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14691.917164 secs 2024-04-26T15:55:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14694.725015 secs 2024-04-26T15:55:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14691.917164 secs 2024-04-26T15:55:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14694.725015 secs 2024-04-26T15:55:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14691.917164 secs 2024-04-26T15:55:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14694.725015 secs 2024-04-26T15:55:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:54 smithi194 ceph-mon[33017]: pgmap v8357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:54 smithi033 ceph-mon[27845]: pgmap v8357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:54 smithi096 ceph-mon[32753]: pgmap v8357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:56 smithi194 ceph-mon[33017]: pgmap v8358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:55:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:56 smithi033 ceph-mon[27845]: pgmap v8358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:55:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:56 smithi096 ceph-mon[32753]: pgmap v8358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:55:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:55:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:55:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:55:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:55:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:55:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:55:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:55:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:55:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:55:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:55:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:58.527+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:58.527+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:55:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:55:58.527+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:55:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:58 smithi194 ceph-mon[33017]: pgmap v8359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:58 smithi194 ceph-mon[33017]: pgmap v8360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14696.917225 secs 2024-04-26T15:55:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:55:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14699.725193 secs 2024-04-26T15:55:59.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:58 smithi033 ceph-mon[27845]: pgmap v8359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:59.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:58 smithi033 ceph-mon[27845]: pgmap v8360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:59.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14696.917225 secs 2024-04-26T15:55:59.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:55:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14699.725193 secs 2024-04-26T15:55:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:58 smithi096 ceph-mon[32753]: pgmap v8359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:55:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:58 smithi096 ceph-mon[32753]: pgmap v8360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:55:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14696.917225 secs 2024-04-26T15:55:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:55:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14699.725193 secs 2024-04-26T15:56:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:00 smithi194 ceph-mon[33017]: pgmap v8361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:56:01.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:00 smithi033 ceph-mon[27845]: pgmap v8361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:56:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:00 smithi096 ceph-mon[32753]: pgmap v8361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:56:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:02 smithi194 ceph-mon[33017]: pgmap v8362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:02 smithi033 ceph-mon[27845]: pgmap v8362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:02 smithi096 ceph-mon[32753]: pgmap v8362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14701.917325 secs 2024-04-26T15:56:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14704.725319 secs 2024-04-26T15:56:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14701.917325 secs 2024-04-26T15:56:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14704.725319 secs 2024-04-26T15:56:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14701.917325 secs 2024-04-26T15:56:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14704.725319 secs 2024-04-26T15:56:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:04 smithi194 ceph-mon[33017]: pgmap v8363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:04 smithi033 ceph-mon[27845]: pgmap v8363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:04 smithi096 ceph-mon[32753]: pgmap v8363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:06 smithi194 ceph-mon[33017]: pgmap v8364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:06 smithi033 ceph-mon[27845]: pgmap v8364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:06 smithi096 ceph-mon[32753]: pgmap v8364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:56:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:08 smithi194 ceph-mon[33017]: pgmap v8365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:56:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14706.917391 secs 2024-04-26T15:56:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14709.725451 secs 2024-04-26T15:56:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:08 smithi033 ceph-mon[27845]: pgmap v8365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:56:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14706.917391 secs 2024-04-26T15:56:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14709.725451 secs 2024-04-26T15:56:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:08 smithi096 ceph-mon[32753]: pgmap v8365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:56:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14706.917391 secs 2024-04-26T15:56:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14709.725451 secs 2024-04-26T15:56:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:10 smithi194 ceph-mon[33017]: pgmap v8366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:10 smithi033 ceph-mon[27845]: pgmap v8366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:10 smithi096 ceph-mon[32753]: pgmap v8366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:12 smithi194 ceph-mon[33017]: pgmap v8367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:12 smithi033 ceph-mon[27845]: pgmap v8367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:12 smithi096 ceph-mon[32753]: pgmap v8367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:13.542+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:13.542+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:13.542+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:56:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14711.917474 secs 2024-04-26T15:56:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14714.725521 secs 2024-04-26T15:56:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14711.917474 secs 2024-04-26T15:56:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14714.725521 secs 2024-04-26T15:56:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14711.917474 secs 2024-04-26T15:56:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14714.725521 secs 2024-04-26T15:56:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:14 smithi194 ceph-mon[33017]: pgmap v8368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:14 smithi033 ceph-mon[27845]: pgmap v8368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:14 smithi096 ceph-mon[32753]: pgmap v8368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:16 smithi194 ceph-mon[33017]: pgmap v8369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:16 smithi033 ceph-mon[27845]: pgmap v8369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:16 smithi096 ceph-mon[32753]: pgmap v8369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:18 smithi194 ceph-mon[33017]: pgmap v8370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14716.917557 secs 2024-04-26T15:56:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14719.725638 secs 2024-04-26T15:56:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:18 smithi033 ceph-mon[27845]: pgmap v8370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14716.917557 secs 2024-04-26T15:56:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14719.725638 secs 2024-04-26T15:56:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:18 smithi096 ceph-mon[32753]: pgmap v8370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14716.917557 secs 2024-04-26T15:56:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14719.725638 secs 2024-04-26T15:56:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:20 smithi194 ceph-mon[33017]: pgmap v8371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:20 smithi033 ceph-mon[27845]: pgmap v8371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:20 smithi096 ceph-mon[32753]: pgmap v8371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T15:56:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T15:56:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0038050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T15:56:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0038050 comp rx=0 tx=0).stop 2024-04-26T15:56:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T15:56:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T15:56:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T15:56:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.752+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T15:56:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.753+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T15:56:21.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:21.755+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0062040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T15:56:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:22 smithi194 ceph-mon[33017]: pgmap v8372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:22 smithi033 ceph-mon[27845]: pgmap v8372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:22 smithi096 ceph-mon[32753]: pgmap v8372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14721.917642 secs 2024-04-26T15:56:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14724.725753 secs 2024-04-26T15:56:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14721.917642 secs 2024-04-26T15:56:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14724.725753 secs 2024-04-26T15:56:24.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:24.301+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:56:24.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:24.301+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T15:56:24.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:24.302+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T15:56:24.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:24.302+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:56:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14721.917642 secs 2024-04-26T15:56:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14724.725753 secs 2024-04-26T15:56:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:24 smithi033 ceph-mon[27845]: pgmap v8373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:24 smithi096 ceph-mon[32753]: pgmap v8373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:24 smithi194 ceph-mon[33017]: pgmap v8373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:26 smithi033 ceph-mon[27845]: pgmap v8374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:26 smithi096 ceph-mon[32753]: pgmap v8374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:26 smithi194 ceph-mon[33017]: pgmap v8374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:28.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:28.542+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:28.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:28.542+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:56:28.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:28.542+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:28 smithi033 ceph-mon[27845]: pgmap v8375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14726.917725 secs 2024-04-26T15:56:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14729.725868 secs 2024-04-26T15:56:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:28 smithi096 ceph-mon[32753]: pgmap v8375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14726.917725 secs 2024-04-26T15:56:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14729.725868 secs 2024-04-26T15:56:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:28 smithi194 ceph-mon[33017]: pgmap v8375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14726.917725 secs 2024-04-26T15:56:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14729.725868 secs 2024-04-26T15:56:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:30 smithi033 ceph-mon[27845]: pgmap v8376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:30 smithi096 ceph-mon[32753]: pgmap v8376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:30 smithi194 ceph-mon[33017]: pgmap v8376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:32 smithi033 ceph-mon[27845]: pgmap v8377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:32 smithi096 ceph-mon[32753]: pgmap v8377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:32 smithi194 ceph-mon[33017]: pgmap v8377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14731.917807 secs 2024-04-26T15:56:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14734.725999 secs 2024-04-26T15:56:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14731.917807 secs 2024-04-26T15:56:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14734.725999 secs 2024-04-26T15:56:34.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14731.917807 secs 2024-04-26T15:56:34.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14734.725999 secs 2024-04-26T15:56:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:34 smithi033 ceph-mon[27845]: pgmap v8378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:34 smithi096 ceph-mon[32753]: pgmap v8378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:34 smithi194 ceph-mon[33017]: pgmap v8378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T15:56:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:36 smithi033 ceph-mon[27845]: pgmap v8379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:36 smithi096 ceph-mon[32753]: pgmap v8379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:36 smithi194 ceph-mon[33017]: pgmap v8379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T15:56:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:38 smithi033 ceph-mon[27845]: pgmap v8380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14736.917898 secs 2024-04-26T15:56:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14739.726154 secs 2024-04-26T15:56:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:38 smithi096 ceph-mon[32753]: pgmap v8380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14736.917898 secs 2024-04-26T15:56:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14739.726154 secs 2024-04-26T15:56:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:38 smithi194 ceph-mon[33017]: pgmap v8380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14736.917898 secs 2024-04-26T15:56:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14739.726154 secs 2024-04-26T15:56:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:40 smithi033 ceph-mon[27845]: pgmap v8381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:40 smithi096 ceph-mon[32753]: pgmap v8381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:40 smithi194 ceph-mon[33017]: pgmap v8381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:42 smithi033 ceph-mon[27845]: pgmap v8382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:42 smithi096 ceph-mon[32753]: pgmap v8382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:42 smithi194 ceph-mon[33017]: pgmap v8382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:43.553+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:43.553+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:56:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:43.553+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14741.917982 secs 2024-04-26T15:56:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14744.726280 secs 2024-04-26T15:56:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14741.917982 secs 2024-04-26T15:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14744.726280 secs 2024-04-26T15:56:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14741.917982 secs 2024-04-26T15:56:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14744.726280 secs 2024-04-26T15:56:45.231 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:44 smithi033 ceph-mon[27845]: pgmap v8383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:44 smithi194 ceph-mon[33017]: pgmap v8383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:44 smithi096 ceph-mon[32753]: pgmap v8383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:46 smithi194 ceph-mon[33017]: pgmap v8384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:46 smithi033 ceph-mon[27845]: pgmap v8384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:46 smithi096 ceph-mon[32753]: pgmap v8384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:49.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:48 smithi033 ceph-mon[27845]: pgmap v8385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:49.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14746.918068 secs 2024-04-26T15:56:49.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14749.726416 secs 2024-04-26T15:56:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:48 smithi194 ceph-mon[33017]: pgmap v8385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14746.918068 secs 2024-04-26T15:56:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14749.726416 secs 2024-04-26T15:56:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:48 smithi096 ceph-mon[32753]: pgmap v8385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14746.918068 secs 2024-04-26T15:56:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14749.726416 secs 2024-04-26T15:56:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:50 smithi194 ceph-mon[33017]: pgmap v8386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:50 smithi033 ceph-mon[27845]: pgmap v8386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:50 smithi096 ceph-mon[32753]: pgmap v8386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:52 smithi194 ceph-mon[33017]: pgmap v8387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:52 smithi033 ceph-mon[27845]: pgmap v8387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:52 smithi096 ceph-mon[32753]: pgmap v8387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14751.918147 secs 2024-04-26T15:56:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14754.726540 secs 2024-04-26T15:56:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14751.918147 secs 2024-04-26T15:56:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14754.726540 secs 2024-04-26T15:56:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14751.918147 secs 2024-04-26T15:56:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14754.726540 secs 2024-04-26T15:56:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:54 smithi194 ceph-mon[33017]: pgmap v8388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:54 smithi033 ceph-mon[27845]: pgmap v8388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:54 smithi096 ceph-mon[32753]: pgmap v8388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:56 smithi194 ceph-mon[33017]: pgmap v8389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:56 smithi033 ceph-mon[27845]: pgmap v8389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:56 smithi096 ceph-mon[32753]: pgmap v8389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:56:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:56:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:56:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:56:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:58.569+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:58.569+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:56:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:56:58.569+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:56:59.177 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:58 smithi194 ceph-mon[33017]: pgmap v8390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:59.178 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14756.918214 secs 2024-04-26T15:56:59.178 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:56:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14759.726607 secs 2024-04-26T15:56:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:58 smithi096 ceph-mon[32753]: pgmap v8390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14756.918214 secs 2024-04-26T15:56:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:56:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14759.726607 secs 2024-04-26T15:56:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:58 smithi033 ceph-mon[27845]: pgmap v8390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:56:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14756.918214 secs 2024-04-26T15:56:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:56:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14759.726607 secs 2024-04-26T15:57:01.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:00 smithi033 ceph-mon[27845]: pgmap v8391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:01.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:00 smithi096 ceph-mon[32753]: pgmap v8391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:01.347 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:00 smithi194 ceph-mon[33017]: pgmap v8391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:02 smithi194 ceph-mon[33017]: pgmap v8392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:02 smithi033 ceph-mon[27845]: pgmap v8392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:02 smithi096 ceph-mon[32753]: pgmap v8392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14761.918320 secs 2024-04-26T15:57:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14764.726724 secs 2024-04-26T15:57:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14761.918320 secs 2024-04-26T15:57:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14764.726724 secs 2024-04-26T15:57:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14761.918320 secs 2024-04-26T15:57:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14764.726724 secs 2024-04-26T15:57:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:04 smithi194 ceph-mon[33017]: pgmap v8393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:57:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:57:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:04 smithi033 ceph-mon[27845]: pgmap v8393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:57:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:57:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:04 smithi096 ceph-mon[32753]: pgmap v8393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:57:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:57:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:57:06.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:05 smithi096 ceph-mon[32753]: pgmap v8394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:57:06.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:05 smithi096 ceph-mon[32753]: pgmap v8395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:05 smithi194 ceph-mon[33017]: pgmap v8394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:57:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:05 smithi194 ceph-mon[33017]: pgmap v8395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:05 smithi033 ceph-mon[27845]: pgmap v8394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T15:57:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:05 smithi033 ceph-mon[27845]: pgmap v8395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:07 smithi194 ceph-mon[33017]: pgmap v8396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:07 smithi033 ceph-mon[27845]: pgmap v8396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:07 smithi096 ceph-mon[32753]: pgmap v8396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14766.918402 secs 2024-04-26T15:57:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14769.726879 secs 2024-04-26T15:57:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14766.918402 secs 2024-04-26T15:57:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14769.726879 secs 2024-04-26T15:57:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14766.918402 secs 2024-04-26T15:57:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14769.726879 secs 2024-04-26T15:57:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:09.308+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T15:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:09.308+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:09.309+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006630 con 0x564385821900 2024-04-26T15:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:09.309+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T15:57:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:09 smithi194 ceph-mon[33017]: pgmap v8397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:09 smithi033 ceph-mon[27845]: pgmap v8397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:09 smithi096 ceph-mon[32753]: pgmap v8397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:11 smithi194 ceph-mon[33017]: pgmap v8398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:11 smithi033 ceph-mon[27845]: pgmap v8398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:11 smithi096 ceph-mon[32753]: pgmap v8398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:13.585+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:13.585+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:13.585+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:57:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:13 smithi033 ceph-mon[27845]: pgmap v8399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14771.918487 secs 2024-04-26T15:57:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14774.726992 secs 2024-04-26T15:57:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:13 smithi096 ceph-mon[32753]: pgmap v8399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14771.918487 secs 2024-04-26T15:57:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14774.726992 secs 2024-04-26T15:57:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:13 smithi194 ceph-mon[33017]: pgmap v8399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:57:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14771.918487 secs 2024-04-26T15:57:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14774.726992 secs 2024-04-26T15:57:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:15 smithi194 ceph-mon[33017]: pgmap v8400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:57:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:15 smithi033 ceph-mon[27845]: pgmap v8400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:57:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:15 smithi096 ceph-mon[32753]: pgmap v8400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:57:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:17 smithi194 ceph-mon[33017]: pgmap v8401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:17 smithi033 ceph-mon[27845]: pgmap v8401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:17 smithi096 ceph-mon[32753]: pgmap v8401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14776.918570 secs 2024-04-26T15:57:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14779.727103 secs 2024-04-26T15:57:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14776.918570 secs 2024-04-26T15:57:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14779.727103 secs 2024-04-26T15:57:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14776.918570 secs 2024-04-26T15:57:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14779.727103 secs 2024-04-26T15:57:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:19 smithi194 ceph-mon[33017]: pgmap v8402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:19 smithi033 ceph-mon[27845]: pgmap v8402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:19 smithi096 ceph-mon[32753]: pgmap v8402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:21 smithi194 ceph-mon[33017]: pgmap v8403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:21 smithi033 ceph-mon[27845]: pgmap v8403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:21 smithi096 ceph-mon[32753]: pgmap v8403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:23 smithi194 ceph-mon[33017]: pgmap v8404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14781.918654 secs 2024-04-26T15:57:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14784.727224 secs 2024-04-26T15:57:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:23 smithi033 ceph-mon[27845]: pgmap v8404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14781.918654 secs 2024-04-26T15:57:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14784.727224 secs 2024-04-26T15:57:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:23 smithi096 ceph-mon[32753]: pgmap v8404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14781.918654 secs 2024-04-26T15:57:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14784.727224 secs 2024-04-26T15:57:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:25 smithi194 ceph-mon[33017]: pgmap v8405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:25 smithi033 ceph-mon[27845]: pgmap v8405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:25 smithi096 ceph-mon[32753]: pgmap v8405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:28.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:27 smithi194 ceph-mon[33017]: pgmap v8406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:27 smithi033 ceph-mon[27845]: pgmap v8406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:27 smithi096 ceph-mon[32753]: pgmap v8406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:28.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:28.592+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:28.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:28.592+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:57:28.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:28.592+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14786.918733 secs 2024-04-26T15:57:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14789.727342 secs 2024-04-26T15:57:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14786.918733 secs 2024-04-26T15:57:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14789.727342 secs 2024-04-26T15:57:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14786.918733 secs 2024-04-26T15:57:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14789.727342 secs 2024-04-26T15:57:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:29 smithi194 ceph-mon[33017]: pgmap v8407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:29 smithi033 ceph-mon[27845]: pgmap v8407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:29 smithi096 ceph-mon[32753]: pgmap v8407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:31 smithi194 ceph-mon[33017]: pgmap v8408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:31 smithi033 ceph-mon[27845]: pgmap v8408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:31 smithi096 ceph-mon[32753]: pgmap v8408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:33 smithi194 ceph-mon[33017]: pgmap v8409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14791.918815 secs 2024-04-26T15:57:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14794.727458 secs 2024-04-26T15:57:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:33 smithi033 ceph-mon[27845]: pgmap v8409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14791.918815 secs 2024-04-26T15:57:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14794.727458 secs 2024-04-26T15:57:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:33 smithi096 ceph-mon[32753]: pgmap v8409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14791.918815 secs 2024-04-26T15:57:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14794.727458 secs 2024-04-26T15:57:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:35 smithi194 ceph-mon[33017]: pgmap v8410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:35 smithi033 ceph-mon[27845]: pgmap v8410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:35 smithi096 ceph-mon[32753]: pgmap v8410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:37 smithi194 ceph-mon[33017]: pgmap v8411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:37 smithi033 ceph-mon[27845]: pgmap v8411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:37 smithi096 ceph-mon[32753]: pgmap v8411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14796.918898 secs 2024-04-26T15:57:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14799.727576 secs 2024-04-26T15:57:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14796.918898 secs 2024-04-26T15:57:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14799.727576 secs 2024-04-26T15:57:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14796.918898 secs 2024-04-26T15:57:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14799.727576 secs 2024-04-26T15:57:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:39 smithi194 ceph-mon[33017]: pgmap v8412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:39 smithi033 ceph-mon[27845]: pgmap v8412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:39 smithi096 ceph-mon[32753]: pgmap v8412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:41 smithi194 ceph-mon[33017]: pgmap v8413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:41 smithi033 ceph-mon[27845]: pgmap v8413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:41 smithi096 ceph-mon[32753]: pgmap v8413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:43.599+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:43.599+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:57:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:43.599+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:43 smithi194 ceph-mon[33017]: pgmap v8414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14801.918995 secs 2024-04-26T15:57:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14804.727707 secs 2024-04-26T15:57:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:43 smithi033 ceph-mon[27845]: pgmap v8414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14801.918995 secs 2024-04-26T15:57:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14804.727707 secs 2024-04-26T15:57:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:43 smithi096 ceph-mon[32753]: pgmap v8414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14801.918995 secs 2024-04-26T15:57:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14804.727707 secs 2024-04-26T15:57:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:45 smithi194 ceph-mon[33017]: pgmap v8415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:45 smithi033 ceph-mon[27845]: pgmap v8415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:45 smithi096 ceph-mon[32753]: pgmap v8415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:47 smithi194 ceph-mon[33017]: pgmap v8416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:47 smithi033 ceph-mon[27845]: pgmap v8416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:47 smithi096 ceph-mon[32753]: pgmap v8416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14806.919036 secs 2024-04-26T15:57:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14809.727853 secs 2024-04-26T15:57:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14806.919036 secs 2024-04-26T15:57:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14809.727853 secs 2024-04-26T15:57:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14806.919036 secs 2024-04-26T15:57:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14809.727853 secs 2024-04-26T15:57:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:49 smithi194 ceph-mon[33017]: pgmap v8417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:49 smithi033 ceph-mon[27845]: pgmap v8417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:49 smithi096 ceph-mon[32753]: pgmap v8417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:51 smithi194 ceph-mon[33017]: pgmap v8418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:51 smithi033 ceph-mon[27845]: pgmap v8418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:51 smithi096 ceph-mon[32753]: pgmap v8418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:53 smithi194 ceph-mon[33017]: pgmap v8419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14811.919127 secs 2024-04-26T15:57:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14814.727953 secs 2024-04-26T15:57:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:53 smithi033 ceph-mon[27845]: pgmap v8419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14811.919127 secs 2024-04-26T15:57:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14814.727953 secs 2024-04-26T15:57:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:53 smithi096 ceph-mon[32753]: pgmap v8419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14811.919127 secs 2024-04-26T15:57:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14814.727953 secs 2024-04-26T15:57:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:55 smithi194 ceph-mon[33017]: pgmap v8420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:55 smithi033 ceph-mon[27845]: pgmap v8420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:55 smithi096 ceph-mon[32753]: pgmap v8420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:57:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:57 smithi194 ceph-mon[33017]: pgmap v8421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:57 smithi033 ceph-mon[27845]: pgmap v8421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:57 smithi096 ceph-mon[32753]: pgmap v8421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:57:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:58.607+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:58.607+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:57:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:57:58.607+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:57:59.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14816.919229 secs 2024-04-26T15:57:59.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14819.728074 secs 2024-04-26T15:57:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14816.919229 secs 2024-04-26T15:57:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14819.728074 secs 2024-04-26T15:57:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14816.919229 secs 2024-04-26T15:57:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14819.728074 secs 2024-04-26T15:58:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:57:59 smithi033 ceph-mon[27845]: pgmap v8422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:57:59 smithi096 ceph-mon[32753]: pgmap v8422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:57:59 smithi194 ceph-mon[33017]: pgmap v8422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:01 smithi033 ceph-mon[27845]: pgmap v8423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:01 smithi096 ceph-mon[32753]: pgmap v8423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:01 smithi194 ceph-mon[33017]: pgmap v8423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:03 smithi033 ceph-mon[27845]: pgmap v8424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14821.919289 secs 2024-04-26T15:58:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14824.728193 secs 2024-04-26T15:58:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:03 smithi096 ceph-mon[32753]: pgmap v8424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14821.919289 secs 2024-04-26T15:58:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14824.728193 secs 2024-04-26T15:58:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:03 smithi194 ceph-mon[33017]: pgmap v8424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14821.919289 secs 2024-04-26T15:58:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14824.728193 secs 2024-04-26T15:58:06.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:05 smithi194 ceph-mon[33017]: pgmap v8425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:06.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:58:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:05 smithi033 ceph-mon[27845]: pgmap v8425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:58:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:05 smithi096 ceph-mon[32753]: pgmap v8425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:58:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:58:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:58:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:58:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:58:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:58:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:58:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:58:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:58:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:58:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:07 smithi033 ceph-mon[27845]: pgmap v8426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:07 smithi033 ceph-mon[27845]: pgmap v8427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:07 smithi096 ceph-mon[32753]: pgmap v8426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:07 smithi096 ceph-mon[32753]: pgmap v8427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:07 smithi194 ceph-mon[33017]: pgmap v8426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:07 smithi194 ceph-mon[33017]: pgmap v8427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14826.919380 secs 2024-04-26T15:58:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14829.728304 secs 2024-04-26T15:58:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14826.919380 secs 2024-04-26T15:58:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14829.728304 secs 2024-04-26T15:58:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14826.919380 secs 2024-04-26T15:58:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14829.728304 secs 2024-04-26T15:58:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:09 smithi033 ceph-mon[27845]: pgmap v8428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:09 smithi096 ceph-mon[32753]: pgmap v8428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:09 smithi194 ceph-mon[33017]: pgmap v8428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:11 smithi033 ceph-mon[27845]: pgmap v8429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:11 smithi096 ceph-mon[32753]: pgmap v8429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:11 smithi194 ceph-mon[33017]: pgmap v8429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:13.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:13.616+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:13.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:13.616+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:13.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:13.616+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:58:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:13 smithi033 ceph-mon[27845]: pgmap v8430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14831.919464 secs 2024-04-26T15:58:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14834.728362 secs 2024-04-26T15:58:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:13 smithi096 ceph-mon[32753]: pgmap v8430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14831.919464 secs 2024-04-26T15:58:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14834.728362 secs 2024-04-26T15:58:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:13 smithi194 ceph-mon[33017]: pgmap v8430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14831.919464 secs 2024-04-26T15:58:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14834.728362 secs 2024-04-26T15:58:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:15 smithi033 ceph-mon[27845]: pgmap v8431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:58:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:15 smithi096 ceph-mon[32753]: pgmap v8431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:58:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:15 smithi194 ceph-mon[33017]: pgmap v8431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T15:58:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:17 smithi096 ceph-mon[32753]: pgmap v8432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:58:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:17 smithi194 ceph-mon[33017]: pgmap v8432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:58:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:17 smithi033 ceph-mon[27845]: pgmap v8432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:58:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14836.919550 secs 2024-04-26T15:58:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14839.728475 secs 2024-04-26T15:58:19.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14836.919550 secs 2024-04-26T15:58:19.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14839.728475 secs 2024-04-26T15:58:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14836.919550 secs 2024-04-26T15:58:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14839.728475 secs 2024-04-26T15:58:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:19 smithi194 ceph-mon[33017]: pgmap v8433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:19 smithi033 ceph-mon[27845]: pgmap v8433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:19 smithi096 ceph-mon[32753]: pgmap v8433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:21 smithi194 ceph-mon[33017]: pgmap v8434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:21 smithi033 ceph-mon[27845]: pgmap v8434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:21 smithi096 ceph-mon[32753]: pgmap v8434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:23 smithi194 ceph-mon[33017]: pgmap v8435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14841.919628 secs 2024-04-26T15:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14844.728569 secs 2024-04-26T15:58:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:23 smithi033 ceph-mon[27845]: pgmap v8435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14841.919628 secs 2024-04-26T15:58:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14844.728569 secs 2024-04-26T15:58:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:23 smithi096 ceph-mon[32753]: pgmap v8435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14841.919628 secs 2024-04-26T15:58:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14844.728569 secs 2024-04-26T15:58:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:25 smithi194 ceph-mon[33017]: pgmap v8436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:25 smithi033 ceph-mon[27845]: pgmap v8436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:25 smithi096 ceph-mon[32753]: pgmap v8436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:27 smithi194 ceph-mon[33017]: pgmap v8437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:27 smithi033 ceph-mon[27845]: pgmap v8437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:27 smithi096 ceph-mon[32753]: pgmap v8437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:28.627+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:28.627+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:28.627+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:58:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14846.919708 secs 2024-04-26T15:58:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14849.728670 secs 2024-04-26T15:58:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14846.919708 secs 2024-04-26T15:58:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14849.728670 secs 2024-04-26T15:58:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14846.919708 secs 2024-04-26T15:58:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14849.728670 secs 2024-04-26T15:58:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:29 smithi194 ceph-mon[33017]: pgmap v8438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:29 smithi033 ceph-mon[27845]: pgmap v8438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:29 smithi096 ceph-mon[32753]: pgmap v8438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:31 smithi194 ceph-mon[33017]: pgmap v8439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:31 smithi033 ceph-mon[27845]: pgmap v8439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:31 smithi096 ceph-mon[32753]: pgmap v8439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:33 smithi194 ceph-mon[33017]: pgmap v8440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14851.919813 secs 2024-04-26T15:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14854.728786 secs 2024-04-26T15:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:33 smithi033 ceph-mon[27845]: pgmap v8440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14851.919813 secs 2024-04-26T15:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14854.728786 secs 2024-04-26T15:58:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:33 smithi096 ceph-mon[32753]: pgmap v8440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14851.919813 secs 2024-04-26T15:58:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14854.728786 secs 2024-04-26T15:58:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:35 smithi194 ceph-mon[33017]: pgmap v8441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:35 smithi033 ceph-mon[27845]: pgmap v8441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:35 smithi096 ceph-mon[32753]: pgmap v8441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:37 smithi194 ceph-mon[33017]: pgmap v8442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:37 smithi033 ceph-mon[27845]: pgmap v8442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:37 smithi096 ceph-mon[32753]: pgmap v8442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14856.919896 secs 2024-04-26T15:58:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14859.728903 secs 2024-04-26T15:58:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14856.919896 secs 2024-04-26T15:58:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14859.728903 secs 2024-04-26T15:58:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14856.919896 secs 2024-04-26T15:58:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14859.728903 secs 2024-04-26T15:58:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:40 smithi194 ceph-mon[33017]: pgmap v8443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:40 smithi033 ceph-mon[27845]: pgmap v8443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:40 smithi096 ceph-mon[32753]: pgmap v8443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:42 smithi194 ceph-mon[33017]: pgmap v8444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:42 smithi033 ceph-mon[27845]: pgmap v8444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:42 smithi096 ceph-mon[32753]: pgmap v8444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:43.639+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:43.639+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:58:43.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:43.639+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:44 smithi194 ceph-mon[33017]: pgmap v8445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14861.919982 secs 2024-04-26T15:58:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14864.729020 secs 2024-04-26T15:58:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:44 smithi033 ceph-mon[27845]: pgmap v8445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14861.919982 secs 2024-04-26T15:58:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14864.729020 secs 2024-04-26T15:58:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:44 smithi096 ceph-mon[32753]: pgmap v8445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14861.919982 secs 2024-04-26T15:58:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14864.729020 secs 2024-04-26T15:58:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:46 smithi194 ceph-mon[33017]: pgmap v8446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:46 smithi033 ceph-mon[27845]: pgmap v8446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:46 smithi096 ceph-mon[32753]: pgmap v8446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:48 smithi194 ceph-mon[33017]: pgmap v8447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:48 smithi033 ceph-mon[27845]: pgmap v8447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:48 smithi096 ceph-mon[32753]: pgmap v8447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14866.920057 secs 2024-04-26T15:58:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14869.729132 secs 2024-04-26T15:58:49.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14866.920057 secs 2024-04-26T15:58:49.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14869.729132 secs 2024-04-26T15:58:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14866.920057 secs 2024-04-26T15:58:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14869.729132 secs 2024-04-26T15:58:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:50 smithi194 ceph-mon[33017]: pgmap v8448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:50 smithi033 ceph-mon[27845]: pgmap v8448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:50 smithi096 ceph-mon[32753]: pgmap v8448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:52 smithi194 ceph-mon[33017]: pgmap v8449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:52 smithi033 ceph-mon[27845]: pgmap v8449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:52 smithi096 ceph-mon[32753]: pgmap v8449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:58:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:54 smithi194 ceph-mon[33017]: pgmap v8450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14871.920118 secs 2024-04-26T15:58:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14874.729252 secs 2024-04-26T15:58:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:54 smithi033 ceph-mon[27845]: pgmap v8450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14871.920118 secs 2024-04-26T15:58:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14874.729252 secs 2024-04-26T15:58:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:54 smithi096 ceph-mon[32753]: pgmap v8450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:58:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14871.920118 secs 2024-04-26T15:58:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14874.729252 secs 2024-04-26T15:58:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:56 smithi194 ceph-mon[33017]: pgmap v8451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:56 smithi033 ceph-mon[27845]: pgmap v8451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:56 smithi096 ceph-mon[32753]: pgmap v8451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:58:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:58 smithi194 ceph-mon[33017]: pgmap v8452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:58 smithi033 ceph-mon[27845]: pgmap v8452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:58 smithi096 ceph-mon[32753]: pgmap v8452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:58:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:58.651+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:58.651+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:58:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:58:58.651+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:58:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14876.920179 secs 2024-04-26T15:58:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14879.729389 secs 2024-04-26T15:58:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14876.920179 secs 2024-04-26T15:58:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14879.729389 secs 2024-04-26T15:58:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14876.920179 secs 2024-04-26T15:58:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14879.729389 secs 2024-04-26T15:59:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:00 smithi194 ceph-mon[33017]: pgmap v8453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:00 smithi033 ceph-mon[27845]: pgmap v8453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:00 smithi096 ceph-mon[32753]: pgmap v8453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:02 smithi194 ceph-mon[33017]: pgmap v8454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:02 smithi033 ceph-mon[27845]: pgmap v8454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:02 smithi096 ceph-mon[32753]: pgmap v8454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14881.920256 secs 2024-04-26T15:59:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14881.920256 secs 2024-04-26T15:59:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14881.920256 secs 2024-04-26T15:59:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:04 smithi194 ceph-mon[33017]: pgmap v8455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14884.729523 secs 2024-04-26T15:59:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:04 smithi033 ceph-mon[27845]: pgmap v8455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14884.729523 secs 2024-04-26T15:59:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:04 smithi096 ceph-mon[32753]: pgmap v8455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14884.729523 secs 2024-04-26T15:59:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:06 smithi033 ceph-mon[27845]: pgmap v8456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:59:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:06 smithi096 ceph-mon[32753]: pgmap v8456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:59:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:06 smithi194 ceph-mon[33017]: pgmap v8456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T15:59:07.325 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:59:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:59:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T15:59:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:08 smithi033 ceph-mon[27845]: pgmap v8457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:59:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14886.920343 secs 2024-04-26T15:59:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:08 smithi096 ceph-mon[32753]: pgmap v8457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:59:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14886.920343 secs 2024-04-26T15:59:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:08 smithi194 ceph-mon[33017]: pgmap v8457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T15:59:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14886.920343 secs 2024-04-26T15:59:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14889.729623 secs 2024-04-26T15:59:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:59:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:59:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14889.729623 secs 2024-04-26T15:59:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:59:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:59:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14889.729623 secs 2024-04-26T15:59:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T15:59:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T15:59:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T15:59:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:10 smithi033 ceph-mon[27845]: pgmap v8458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:10 smithi033 ceph-mon[27845]: pgmap v8459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:59:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:10 smithi033 ceph-mon[27845]: pgmap v8460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:59:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:10 smithi096 ceph-mon[32753]: pgmap v8458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:10 smithi096 ceph-mon[32753]: pgmap v8459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:59:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:10 smithi096 ceph-mon[32753]: pgmap v8460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:59:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:10 smithi194 ceph-mon[33017]: pgmap v8458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:10 smithi194 ceph-mon[33017]: pgmap v8459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T15:59:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:10 smithi194 ceph-mon[33017]: pgmap v8460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T15:59:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:12 smithi033 ceph-mon[27845]: pgmap v8461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:12 smithi096 ceph-mon[32753]: pgmap v8461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:12 smithi194 ceph-mon[33017]: pgmap v8461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14891.920433 secs 2024-04-26T15:59:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14891.920433 secs 2024-04-26T15:59:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14891.920433 secs 2024-04-26T15:59:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:13.656+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:13.656+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:59:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:13.656+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:14 smithi033 ceph-mon[27845]: pgmap v8462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T15:59:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14894.729772 secs 2024-04-26T15:59:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:14 smithi096 ceph-mon[32753]: pgmap v8462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T15:59:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14894.729772 secs 2024-04-26T15:59:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:14 smithi194 ceph-mon[33017]: pgmap v8462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T15:59:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14894.729772 secs 2024-04-26T15:59:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:16 smithi033 ceph-mon[27845]: pgmap v8463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:16 smithi096 ceph-mon[32753]: pgmap v8463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:16 smithi194 ceph-mon[33017]: pgmap v8463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:18 smithi033 ceph-mon[27845]: pgmap v8464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14896.920530 secs 2024-04-26T15:59:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:18 smithi096 ceph-mon[32753]: pgmap v8464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14896.920530 secs 2024-04-26T15:59:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:18 smithi194 ceph-mon[33017]: pgmap v8464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T15:59:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14896.920530 secs 2024-04-26T15:59:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14899.729894 secs 2024-04-26T15:59:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14899.729894 secs 2024-04-26T15:59:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14899.729894 secs 2024-04-26T15:59:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:20 smithi033 ceph-mon[27845]: pgmap v8465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:59:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:20 smithi096 ceph-mon[32753]: pgmap v8465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:59:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:20 smithi194 ceph-mon[33017]: pgmap v8465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T15:59:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:22 smithi096 ceph-mon[32753]: pgmap v8466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:22 smithi194 ceph-mon[33017]: pgmap v8466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:22 smithi033 ceph-mon[27845]: pgmap v8466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14901.920614 secs 2024-04-26T15:59:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14901.920614 secs 2024-04-26T15:59:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14901.920614 secs 2024-04-26T15:59:24.330 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:24.331+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:59:24.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:24.331+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T15:59:24.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:24.332+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0004740 con 0x7fef00077780 2024-04-26T15:59:24.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:24.332+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T15:59:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:24 smithi194 ceph-mon[33017]: pgmap v8467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14904.730009 secs 2024-04-26T15:59:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:24 smithi033 ceph-mon[27845]: pgmap v8467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14904.730009 secs 2024-04-26T15:59:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:24 smithi096 ceph-mon[32753]: pgmap v8467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14904.730009 secs 2024-04-26T15:59:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:26 smithi194 ceph-mon[33017]: pgmap v8468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:26 smithi033 ceph-mon[27845]: pgmap v8468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:26 smithi096 ceph-mon[32753]: pgmap v8468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:28 smithi194 ceph-mon[33017]: pgmap v8469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14906.920695 secs 2024-04-26T15:59:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:28.672+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:28.672+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:59:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:28.672+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:28 smithi033 ceph-mon[27845]: pgmap v8469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14906.920695 secs 2024-04-26T15:59:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:28 smithi096 ceph-mon[32753]: pgmap v8469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14906.920695 secs 2024-04-26T15:59:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14909.730131 secs 2024-04-26T15:59:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14909.730131 secs 2024-04-26T15:59:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14909.730131 secs 2024-04-26T15:59:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:30 smithi194 ceph-mon[33017]: pgmap v8470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:30 smithi033 ceph-mon[27845]: pgmap v8470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:30 smithi096 ceph-mon[32753]: pgmap v8470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:32 smithi194 ceph-mon[33017]: pgmap v8471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:32 smithi033 ceph-mon[27845]: pgmap v8471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:32 smithi096 ceph-mon[32753]: pgmap v8471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14911.920778 secs 2024-04-26T15:59:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14911.920778 secs 2024-04-26T15:59:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14911.920778 secs 2024-04-26T15:59:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:34 smithi194 ceph-mon[33017]: pgmap v8472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14914.730254 secs 2024-04-26T15:59:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:34 smithi033 ceph-mon[27845]: pgmap v8472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14914.730254 secs 2024-04-26T15:59:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:34 smithi096 ceph-mon[32753]: pgmap v8472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14914.730254 secs 2024-04-26T15:59:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:35 smithi194 ceph-mon[33017]: pgmap v8473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:35 smithi033 ceph-mon[27845]: pgmap v8473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:35 smithi096 ceph-mon[32753]: pgmap v8473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:37 smithi033 ceph-mon[27845]: pgmap v8474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:37 smithi096 ceph-mon[32753]: pgmap v8474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:38.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:37 smithi194 ceph-mon[33017]: pgmap v8474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14916.920863 secs 2024-04-26T15:59:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14919.730384 secs 2024-04-26T15:59:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14916.920863 secs 2024-04-26T15:59:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14919.730384 secs 2024-04-26T15:59:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14916.920863 secs 2024-04-26T15:59:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14919.730384 secs 2024-04-26T15:59:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:39 smithi033 ceph-mon[27845]: pgmap v8475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:39 smithi096 ceph-mon[32753]: pgmap v8475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:39 smithi194 ceph-mon[33017]: pgmap v8475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:41 smithi033 ceph-mon[27845]: pgmap v8476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:41 smithi096 ceph-mon[32753]: pgmap v8476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:41 smithi194 ceph-mon[33017]: pgmap v8476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:43.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:43.683+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:43.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:43.683+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:43.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:43.683+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:59:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14921.920950 secs 2024-04-26T15:59:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:43 smithi096 ceph-mon[32753]: pgmap v8477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14924.730479 secs 2024-04-26T15:59:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14921.920950 secs 2024-04-26T15:59:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:43 smithi194 ceph-mon[33017]: pgmap v8477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14924.730479 secs 2024-04-26T15:59:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14921.920950 secs 2024-04-26T15:59:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:43 smithi033 ceph-mon[27845]: pgmap v8477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14924.730479 secs 2024-04-26T15:59:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:45 smithi096 ceph-mon[32753]: pgmap v8478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:45 smithi194 ceph-mon[33017]: pgmap v8478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:45 smithi033 ceph-mon[27845]: pgmap v8478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:47 smithi194 ceph-mon[33017]: pgmap v8479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:47 smithi033 ceph-mon[27845]: pgmap v8479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:47 smithi096 ceph-mon[32753]: pgmap v8479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:49.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14926.921037 secs 2024-04-26T15:59:49.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14929.730598 secs 2024-04-26T15:59:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14926.921037 secs 2024-04-26T15:59:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14929.730598 secs 2024-04-26T15:59:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14926.921037 secs 2024-04-26T15:59:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14929.730598 secs 2024-04-26T15:59:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:49 smithi194 ceph-mon[33017]: pgmap v8480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:49 smithi033 ceph-mon[27845]: pgmap v8480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:49 smithi096 ceph-mon[32753]: pgmap v8480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:51 smithi194 ceph-mon[33017]: pgmap v8481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:51 smithi033 ceph-mon[27845]: pgmap v8481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:51 smithi096 ceph-mon[32753]: pgmap v8481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14931.921094 secs 2024-04-26T15:59:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:53 smithi194 ceph-mon[33017]: pgmap v8482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14934.730734 secs 2024-04-26T15:59:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14931.921094 secs 2024-04-26T15:59:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:53 smithi033 ceph-mon[27845]: pgmap v8482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14934.730734 secs 2024-04-26T15:59:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14931.921094 secs 2024-04-26T15:59:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:53 smithi096 ceph-mon[32753]: pgmap v8482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14934.730734 secs 2024-04-26T15:59:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:55 smithi194 ceph-mon[33017]: pgmap v8483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:55 smithi033 ceph-mon[27845]: pgmap v8483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:55 smithi096 ceph-mon[32753]: pgmap v8483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T15:59:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:57 smithi194 ceph-mon[33017]: pgmap v8484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:57 smithi033 ceph-mon[27845]: pgmap v8484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:57 smithi096 ceph-mon[32753]: pgmap v8484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T15:59:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:58.699+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:58.699+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T15:59:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T15:59:58.699+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T15:59:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14936.921156 secs 2024-04-26T15:59:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14939.730841 secs 2024-04-26T15:59:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14936.921156 secs 2024-04-26T15:59:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14939.730841 secs 2024-04-26T15:59:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14936.921156 secs 2024-04-26T15:59:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14939.730841 secs 2024-04-26T16:00:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 15:59:59 smithi194 ceph-mon[33017]: pgmap v8485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 15:59:59 smithi033 ceph-mon[27845]: pgmap v8485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 15:59:59 smithi096 ceph-mon[32753]: pgmap v8485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:00:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:00:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:00:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:02 smithi194 ceph-mon[33017]: pgmap v8486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:02 smithi033 ceph-mon[27845]: pgmap v8486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:02 smithi096 ceph-mon[32753]: pgmap v8486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14941.921263 secs 2024-04-26T16:00:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:04 smithi194 ceph-mon[33017]: pgmap v8487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14944.730968 secs 2024-04-26T16:00:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14941.921263 secs 2024-04-26T16:00:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:04 smithi033 ceph-mon[27845]: pgmap v8487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14944.730968 secs 2024-04-26T16:00:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14941.921263 secs 2024-04-26T16:00:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:04 smithi096 ceph-mon[32753]: pgmap v8487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14944.730968 secs 2024-04-26T16:00:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:06 smithi194 ceph-mon[33017]: pgmap v8488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:06 smithi033 ceph-mon[27845]: pgmap v8488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:06 smithi096 ceph-mon[32753]: pgmap v8488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:08 smithi194 ceph-mon[33017]: pgmap v8489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:08 smithi033 ceph-mon[27845]: pgmap v8489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:08 smithi096 ceph-mon[32753]: pgmap v8489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:09.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14946.921361 secs 2024-04-26T16:00:09.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14949.731088 secs 2024-04-26T16:00:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14946.921361 secs 2024-04-26T16:00:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14949.731088 secs 2024-04-26T16:00:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14946.921361 secs 2024-04-26T16:00:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14949.731088 secs 2024-04-26T16:00:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:10 smithi194 ceph-mon[33017]: pgmap v8490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:00:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:10 smithi033 ceph-mon[27845]: pgmap v8490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:00:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:10 smithi096 ceph-mon[32753]: pgmap v8490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:00:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:00:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:00:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:00:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:00:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:00:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:00:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:00:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:00:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:00:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:12 smithi194 ceph-mon[33017]: pgmap v8491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:00:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:12 smithi194 ceph-mon[33017]: pgmap v8492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:12 smithi033 ceph-mon[27845]: pgmap v8491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:00:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:12 smithi033 ceph-mon[27845]: pgmap v8492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:12 smithi096 ceph-mon[32753]: pgmap v8491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:00:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:12 smithi096 ceph-mon[32753]: pgmap v8492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:13.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:13.714+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:13.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:13.714+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:00:13.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:13.714+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:14 smithi194 ceph-mon[33017]: pgmap v8493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14951.921445 secs 2024-04-26T16:00:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14954.731206 secs 2024-04-26T16:00:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:14 smithi033 ceph-mon[27845]: pgmap v8493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14951.921445 secs 2024-04-26T16:00:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14954.731206 secs 2024-04-26T16:00:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:14 smithi096 ceph-mon[32753]: pgmap v8493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14951.921445 secs 2024-04-26T16:00:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14954.731206 secs 2024-04-26T16:00:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:16 smithi194 ceph-mon[33017]: pgmap v8494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:16 smithi033 ceph-mon[27845]: pgmap v8494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:16 smithi096 ceph-mon[32753]: pgmap v8494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:18 smithi194 ceph-mon[33017]: pgmap v8495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:18 smithi033 ceph-mon[27845]: pgmap v8495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:18 smithi096 ceph-mon[32753]: pgmap v8495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:19.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14956.921503 secs 2024-04-26T16:00:19.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14959.731327 secs 2024-04-26T16:00:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14956.921503 secs 2024-04-26T16:00:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14959.731327 secs 2024-04-26T16:00:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14956.921503 secs 2024-04-26T16:00:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14959.731327 secs 2024-04-26T16:00:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:20 smithi194 ceph-mon[33017]: pgmap v8496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:20 smithi033 ceph-mon[27845]: pgmap v8496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:20 smithi096 ceph-mon[32753]: pgmap v8496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:00:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:22 smithi194 ceph-mon[33017]: pgmap v8497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:00:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:22 smithi033 ceph-mon[27845]: pgmap v8497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:00:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:22 smithi096 ceph-mon[32753]: pgmap v8497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:00:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14961.921587 secs 2024-04-26T16:00:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14961.921587 secs 2024-04-26T16:00:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14961.921587 secs 2024-04-26T16:00:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:24 smithi033 ceph-mon[27845]: pgmap v8498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14964.731435 secs 2024-04-26T16:00:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:24 smithi194 ceph-mon[33017]: pgmap v8498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14964.731435 secs 2024-04-26T16:00:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:24 smithi096 ceph-mon[32753]: pgmap v8498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14964.731435 secs 2024-04-26T16:00:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:26 smithi033 ceph-mon[27845]: pgmap v8499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:26 smithi096 ceph-mon[32753]: pgmap v8499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:26 smithi194 ceph-mon[33017]: pgmap v8499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:28 smithi033 ceph-mon[27845]: pgmap v8500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14966.921672 secs 2024-04-26T16:00:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:28 smithi096 ceph-mon[32753]: pgmap v8500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14966.921672 secs 2024-04-26T16:00:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:28 smithi194 ceph-mon[33017]: pgmap v8500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14966.921672 secs 2024-04-26T16:00:28.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:28.729+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:28.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:28.729+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:28.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:28.729+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:00:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14969.731547 secs 2024-04-26T16:00:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14969.731547 secs 2024-04-26T16:00:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14969.731547 secs 2024-04-26T16:00:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:30 smithi033 ceph-mon[27845]: pgmap v8501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:30 smithi096 ceph-mon[32753]: pgmap v8501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:30 smithi194 ceph-mon[33017]: pgmap v8501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:32 smithi033 ceph-mon[27845]: pgmap v8502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:32 smithi096 ceph-mon[32753]: pgmap v8502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:32 smithi194 ceph-mon[33017]: pgmap v8502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14971.921761 secs 2024-04-26T16:00:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14971.921761 secs 2024-04-26T16:00:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14971.921761 secs 2024-04-26T16:00:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:34 smithi033 ceph-mon[27845]: pgmap v8503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14974.731678 secs 2024-04-26T16:00:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:34 smithi096 ceph-mon[32753]: pgmap v8503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14974.731678 secs 2024-04-26T16:00:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:34 smithi194 ceph-mon[33017]: pgmap v8503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14974.731678 secs 2024-04-26T16:00:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:36 smithi033 ceph-mon[27845]: pgmap v8504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:36 smithi096 ceph-mon[32753]: pgmap v8504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:36 smithi194 ceph-mon[33017]: pgmap v8504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:38.404 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:38 smithi194 ceph-mon[33017]: pgmap v8505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:38.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14976.921847 secs 2024-04-26T16:00:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:38 smithi033 ceph-mon[27845]: pgmap v8505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14976.921847 secs 2024-04-26T16:00:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:38 smithi096 ceph-mon[32753]: pgmap v8505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14976.921847 secs 2024-04-26T16:00:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14979.731815 secs 2024-04-26T16:00:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14979.731815 secs 2024-04-26T16:00:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14979.731815 secs 2024-04-26T16:00:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:40 smithi033 ceph-mon[27845]: pgmap v8506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:40 smithi096 ceph-mon[32753]: pgmap v8506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:40 smithi194 ceph-mon[33017]: pgmap v8506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:42 smithi033 ceph-mon[27845]: pgmap v8507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:42 smithi096 ceph-mon[32753]: pgmap v8507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:42 smithi194 ceph-mon[33017]: pgmap v8507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14981.921931 secs 2024-04-26T16:00:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14981.921931 secs 2024-04-26T16:00:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14981.921931 secs 2024-04-26T16:00:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:43.738+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:43.738+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:43.738+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:00:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:44 smithi194 ceph-mon[33017]: pgmap v8508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14984.731930 secs 2024-04-26T16:00:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:44 smithi033 ceph-mon[27845]: pgmap v8508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14984.731930 secs 2024-04-26T16:00:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:44 smithi096 ceph-mon[32753]: pgmap v8508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14984.731930 secs 2024-04-26T16:00:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:46 smithi194 ceph-mon[33017]: pgmap v8509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:46 smithi033 ceph-mon[27845]: pgmap v8509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:46 smithi096 ceph-mon[32753]: pgmap v8509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:48 smithi194 ceph-mon[33017]: pgmap v8510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14986.922014 secs 2024-04-26T16:00:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:48 smithi033 ceph-mon[27845]: pgmap v8510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14986.922014 secs 2024-04-26T16:00:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:48 smithi096 ceph-mon[32753]: pgmap v8510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14986.922014 secs 2024-04-26T16:00:49.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14989.732030 secs 2024-04-26T16:00:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14989.732030 secs 2024-04-26T16:00:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14989.732030 secs 2024-04-26T16:00:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:50 smithi194 ceph-mon[33017]: pgmap v8511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:50 smithi033 ceph-mon[27845]: pgmap v8511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:50 smithi096 ceph-mon[32753]: pgmap v8511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:52 smithi194 ceph-mon[33017]: pgmap v8512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:52 smithi033 ceph-mon[27845]: pgmap v8512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:52 smithi096 ceph-mon[32753]: pgmap v8512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14991.922120 secs 2024-04-26T16:00:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14991.922120 secs 2024-04-26T16:00:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14991.922120 secs 2024-04-26T16:00:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:54 smithi194 ceph-mon[33017]: pgmap v8513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14994.732151 secs 2024-04-26T16:00:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:54 smithi033 ceph-mon[27845]: pgmap v8513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14994.732151 secs 2024-04-26T16:00:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:54 smithi096 ceph-mon[32753]: pgmap v8513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14994.732151 secs 2024-04-26T16:00:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:56 smithi194 ceph-mon[33017]: pgmap v8514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:56 smithi033 ceph-mon[27845]: pgmap v8514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:56 smithi096 ceph-mon[32753]: pgmap v8514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:00:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:58 smithi194 ceph-mon[33017]: pgmap v8515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:58.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 14996.922208 secs 2024-04-26T16:00:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:58 smithi033 ceph-mon[27845]: pgmap v8515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 14996.922208 secs 2024-04-26T16:00:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:58 smithi096 ceph-mon[32753]: pgmap v8515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:00:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 14996.922208 secs 2024-04-26T16:00:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:58.746+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:58.746+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:00:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:00:58.746+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:00:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 14999.732275 secs 2024-04-26T16:00:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 14999.732275 secs 2024-04-26T16:00:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 14999.732275 secs 2024-04-26T16:01:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:00 smithi194 ceph-mon[33017]: pgmap v8516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:00 smithi033 ceph-mon[27845]: pgmap v8516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:00 smithi096 ceph-mon[32753]: pgmap v8516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:01 smithi033 ceph-mon[27845]: pgmap v8517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:01 smithi194 ceph-mon[33017]: pgmap v8517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:01 smithi096 ceph-mon[32753]: pgmap v8517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:03 smithi033 ceph-mon[27845]: pgmap v8518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15001.922294 secs 2024-04-26T16:01:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15004.732338 secs 2024-04-26T16:01:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:03 smithi096 ceph-mon[32753]: pgmap v8518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15001.922294 secs 2024-04-26T16:01:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15004.732338 secs 2024-04-26T16:01:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:03 smithi194 ceph-mon[33017]: pgmap v8518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15001.922294 secs 2024-04-26T16:01:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15004.732338 secs 2024-04-26T16:01:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:05 smithi033 ceph-mon[27845]: pgmap v8519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:05 smithi096 ceph-mon[32753]: pgmap v8519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:05 smithi194 ceph-mon[33017]: pgmap v8519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:07.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:07 smithi033 ceph-mon[27845]: pgmap v8520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:07 smithi096 ceph-mon[32753]: pgmap v8520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:07 smithi194 ceph-mon[33017]: pgmap v8520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15006.922378 secs 2024-04-26T16:01:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15009.732435 secs 2024-04-26T16:01:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15006.922378 secs 2024-04-26T16:01:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15009.732435 secs 2024-04-26T16:01:09.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15006.922378 secs 2024-04-26T16:01:09.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15009.732435 secs 2024-04-26T16:01:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:09 smithi033 ceph-mon[27845]: pgmap v8521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:09 smithi096 ceph-mon[32753]: pgmap v8521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:09 smithi194 ceph-mon[33017]: pgmap v8521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:11 smithi033 ceph-mon[27845]: pgmap v8522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:01:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:11 smithi096 ceph-mon[32753]: pgmap v8522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:01:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:11 smithi194 ceph-mon[33017]: pgmap v8522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:01:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:13.761+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:13.761+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:13.761+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:01:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:13 smithi096 ceph-mon[32753]: pgmap v8523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15011.922463 secs 2024-04-26T16:01:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15014.732526 secs 2024-04-26T16:01:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:13 smithi194 ceph-mon[33017]: pgmap v8523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15011.922463 secs 2024-04-26T16:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15014.732526 secs 2024-04-26T16:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:13 smithi033 ceph-mon[27845]: pgmap v8523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15011.922463 secs 2024-04-26T16:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15014.732526 secs 2024-04-26T16:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:01:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:15 smithi194 ceph-mon[33017]: pgmap v8524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:01:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:01:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:01:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:01:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:15 smithi033 ceph-mon[27845]: pgmap v8524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:15 smithi096 ceph-mon[32753]: pgmap v8524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: pgmap v8525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:01:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:01:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: pgmap v8526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:01:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: pgmap v8527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: pgmap v8525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:01:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: pgmap v8526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:01:18.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: pgmap v8527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:18.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: pgmap v8525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: pgmap v8526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: pgmap v8527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:01:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15016.922520 secs 2024-04-26T16:01:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15019.732643 secs 2024-04-26T16:01:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15016.922520 secs 2024-04-26T16:01:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15019.732643 secs 2024-04-26T16:01:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15016.922520 secs 2024-04-26T16:01:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15019.732643 secs 2024-04-26T16:01:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:19 smithi194 ceph-mon[33017]: pgmap v8528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:01:20.113 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:19 smithi033 ceph-mon[27845]: pgmap v8528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:01:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:19 smithi096 ceph-mon[32753]: pgmap v8528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:01:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:22 smithi194 ceph-mon[33017]: pgmap v8529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:22 smithi033 ceph-mon[27845]: pgmap v8529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:22 smithi096 ceph-mon[32753]: pgmap v8529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15021.922578 secs 2024-04-26T16:01:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:24 smithi194 ceph-mon[33017]: pgmap v8530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:01:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15024.732712 secs 2024-04-26T16:01:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15021.922578 secs 2024-04-26T16:01:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:24 smithi033 ceph-mon[27845]: pgmap v8530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:01:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15024.732712 secs 2024-04-26T16:01:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15021.922578 secs 2024-04-26T16:01:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:24 smithi096 ceph-mon[32753]: pgmap v8530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:01:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15024.732712 secs 2024-04-26T16:01:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:26 smithi194 ceph-mon[33017]: pgmap v8531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:26 smithi033 ceph-mon[27845]: pgmap v8531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:26 smithi096 ceph-mon[32753]: pgmap v8531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:01:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:28 smithi194 ceph-mon[33017]: pgmap v8532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:01:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15026.922658 secs 2024-04-26T16:01:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:28 smithi033 ceph-mon[27845]: pgmap v8532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:01:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15026.922658 secs 2024-04-26T16:01:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:28 smithi096 ceph-mon[32753]: pgmap v8532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:01:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15026.922658 secs 2024-04-26T16:01:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:28.772+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:28.772+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:01:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:28.772+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15029.732779 secs 2024-04-26T16:01:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15029.732779 secs 2024-04-26T16:01:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15029.732779 secs 2024-04-26T16:01:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:30 smithi033 ceph-mon[27845]: pgmap v8533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:30 smithi096 ceph-mon[32753]: pgmap v8533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:30 smithi194 ceph-mon[33017]: pgmap v8533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:32 smithi033 ceph-mon[27845]: pgmap v8534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:32 smithi096 ceph-mon[32753]: pgmap v8534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:32 smithi194 ceph-mon[33017]: pgmap v8534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15031.922739 secs 2024-04-26T16:01:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15031.922739 secs 2024-04-26T16:01:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15031.922739 secs 2024-04-26T16:01:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:34 smithi033 ceph-mon[27845]: pgmap v8535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15034.732894 secs 2024-04-26T16:01:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:34 smithi096 ceph-mon[32753]: pgmap v8535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15034.732894 secs 2024-04-26T16:01:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:34 smithi194 ceph-mon[33017]: pgmap v8535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15034.732894 secs 2024-04-26T16:01:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:36 smithi033 ceph-mon[27845]: pgmap v8536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:36 smithi096 ceph-mon[32753]: pgmap v8536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:36 smithi194 ceph-mon[33017]: pgmap v8536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:38 smithi033 ceph-mon[27845]: pgmap v8537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15036.922825 secs 2024-04-26T16:01:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:38 smithi096 ceph-mon[32753]: pgmap v8537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15036.922825 secs 2024-04-26T16:01:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:38 smithi194 ceph-mon[33017]: pgmap v8537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15036.922825 secs 2024-04-26T16:01:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15039.732973 secs 2024-04-26T16:01:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15039.732973 secs 2024-04-26T16:01:39.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15039.732973 secs 2024-04-26T16:01:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:40 smithi033 ceph-mon[27845]: pgmap v8538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:40 smithi096 ceph-mon[32753]: pgmap v8538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:40 smithi194 ceph-mon[33017]: pgmap v8538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:42 smithi033 ceph-mon[27845]: pgmap v8539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:42 smithi096 ceph-mon[32753]: pgmap v8539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:42 smithi194 ceph-mon[33017]: pgmap v8539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:43.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15041.922908 secs 2024-04-26T16:01:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15041.922908 secs 2024-04-26T16:01:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15041.922908 secs 2024-04-26T16:01:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:43.782+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:43.782+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:01:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:43.782+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:44 smithi096 ceph-mon[32753]: pgmap v8540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15044.733088 secs 2024-04-26T16:01:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:44 smithi194 ceph-mon[33017]: pgmap v8540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15044.733088 secs 2024-04-26T16:01:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:44 smithi033 ceph-mon[27845]: pgmap v8540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15044.733088 secs 2024-04-26T16:01:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:46 smithi194 ceph-mon[33017]: pgmap v8541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:46 smithi033 ceph-mon[27845]: pgmap v8541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:46 smithi096 ceph-mon[32753]: pgmap v8541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:48 smithi194 ceph-mon[33017]: pgmap v8542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15046.923007 secs 2024-04-26T16:01:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:48 smithi033 ceph-mon[27845]: pgmap v8542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15046.923007 secs 2024-04-26T16:01:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:48 smithi096 ceph-mon[32753]: pgmap v8542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15046.923007 secs 2024-04-26T16:01:49.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15049.733202 secs 2024-04-26T16:01:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15049.733202 secs 2024-04-26T16:01:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15049.733202 secs 2024-04-26T16:01:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:50 smithi194 ceph-mon[33017]: pgmap v8543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:50 smithi033 ceph-mon[27845]: pgmap v8543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:50 smithi096 ceph-mon[32753]: pgmap v8543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:52 smithi194 ceph-mon[33017]: pgmap v8544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:52 smithi033 ceph-mon[27845]: pgmap v8544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:52 smithi096 ceph-mon[32753]: pgmap v8544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15051.923091 secs 2024-04-26T16:01:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15051.923091 secs 2024-04-26T16:01:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15051.923091 secs 2024-04-26T16:01:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:54 smithi194 ceph-mon[33017]: pgmap v8545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15054.733318 secs 2024-04-26T16:01:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:54 smithi033 ceph-mon[27845]: pgmap v8545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15054.733318 secs 2024-04-26T16:01:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:54 smithi096 ceph-mon[32753]: pgmap v8545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15054.733318 secs 2024-04-26T16:01:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:56 smithi194 ceph-mon[33017]: pgmap v8546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:56 smithi033 ceph-mon[27845]: pgmap v8546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:56 smithi096 ceph-mon[32753]: pgmap v8546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:01:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:58 smithi033 ceph-mon[27845]: pgmap v8547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15056.923172 secs 2024-04-26T16:01:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:58 smithi194 ceph-mon[33017]: pgmap v8547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15056.923172 secs 2024-04-26T16:01:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:58 smithi096 ceph-mon[32753]: pgmap v8547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:01:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15056.923172 secs 2024-04-26T16:01:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:58.795+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:58.795+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:01:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:01:58.795+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:01:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15059.733439 secs 2024-04-26T16:01:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15059.733439 secs 2024-04-26T16:01:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15059.733439 secs 2024-04-26T16:02:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:00 smithi194 ceph-mon[33017]: pgmap v8548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:00 smithi033 ceph-mon[27845]: pgmap v8548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:00 smithi096 ceph-mon[32753]: pgmap v8548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:02 smithi194 ceph-mon[33017]: pgmap v8549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:02 smithi033 ceph-mon[27845]: pgmap v8549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:02 smithi096 ceph-mon[32753]: pgmap v8549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15061.923268 secs 2024-04-26T16:02:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15061.923268 secs 2024-04-26T16:02:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15061.923268 secs 2024-04-26T16:02:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:04 smithi194 ceph-mon[33017]: pgmap v8550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15064.733556 secs 2024-04-26T16:02:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:04 smithi033 ceph-mon[27845]: pgmap v8550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15064.733556 secs 2024-04-26T16:02:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:04 smithi096 ceph-mon[32753]: pgmap v8550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15064.733556 secs 2024-04-26T16:02:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:05 smithi194 ceph-mon[33017]: pgmap v8551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:05 smithi033 ceph-mon[27845]: pgmap v8551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:05 smithi096 ceph-mon[32753]: pgmap v8551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:08.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:08 smithi096 ceph-mon[32753]: pgmap v8552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:08.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15066.923349 secs 2024-04-26T16:02:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:08 smithi033 ceph-mon[27845]: pgmap v8552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15066.923349 secs 2024-04-26T16:02:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:08 smithi194 ceph-mon[33017]: pgmap v8552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15066.923349 secs 2024-04-26T16:02:09.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15069.733674 secs 2024-04-26T16:02:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15069.733674 secs 2024-04-26T16:02:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15069.733674 secs 2024-04-26T16:02:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:10 smithi033 ceph-mon[27845]: pgmap v8553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:10 smithi096 ceph-mon[32753]: pgmap v8553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:10.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:10 smithi194 ceph-mon[33017]: pgmap v8553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:12 smithi033 ceph-mon[27845]: pgmap v8554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:12 smithi096 ceph-mon[32753]: pgmap v8554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:12 smithi194 ceph-mon[33017]: pgmap v8554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15071.923406 secs 2024-04-26T16:02:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15071.923406 secs 2024-04-26T16:02:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15071.923406 secs 2024-04-26T16:02:13.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:13.807+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:13.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:13.807+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:13.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:13.807+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:02:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:14 smithi033 ceph-mon[27845]: pgmap v8555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15074.733808 secs 2024-04-26T16:02:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:14 smithi096 ceph-mon[32753]: pgmap v8555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15074.733808 secs 2024-04-26T16:02:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:14 smithi194 ceph-mon[33017]: pgmap v8555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15074.733808 secs 2024-04-26T16:02:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:16 smithi033 ceph-mon[27845]: pgmap v8556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:16 smithi096 ceph-mon[32753]: pgmap v8556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:16 smithi194 ceph-mon[33017]: pgmap v8556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:18 smithi033 ceph-mon[27845]: pgmap v8557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:02:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15076.923467 secs 2024-04-26T16:02:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:18 smithi096 ceph-mon[32753]: pgmap v8557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:02:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15076.923467 secs 2024-04-26T16:02:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:18 smithi194 ceph-mon[33017]: pgmap v8557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:02:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15076.923467 secs 2024-04-26T16:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15079.733896 secs 2024-04-26T16:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15079.733896 secs 2024-04-26T16:02:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15079.733896 secs 2024-04-26T16:02:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:20 smithi033 ceph-mon[27845]: pgmap v8558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:02:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:02:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:20 smithi096 ceph-mon[32753]: pgmap v8558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:02:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:02:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:20 smithi194 ceph-mon[33017]: pgmap v8558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:02:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:02:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:02:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:21 smithi033 ceph-mon[27845]: pgmap v8559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T16:02:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:21 smithi033 ceph-mon[27845]: pgmap v8560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:21 smithi096 ceph-mon[32753]: pgmap v8559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T16:02:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:21 smithi096 ceph-mon[32753]: pgmap v8560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:21 smithi194 ceph-mon[33017]: pgmap v8559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T16:02:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:21 smithi194 ceph-mon[33017]: pgmap v8560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:23 smithi033 ceph-mon[27845]: pgmap v8561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15081.923523 secs 2024-04-26T16:02:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:23 smithi096 ceph-mon[32753]: pgmap v8561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15081.923523 secs 2024-04-26T16:02:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:23 smithi194 ceph-mon[33017]: pgmap v8561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15081.923523 secs 2024-04-26T16:02:24.360 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:24.361+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T16:02:24.360 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:24.361+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T16:02:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:24.362+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T16:02:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:24.362+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:02:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15084.733990 secs 2024-04-26T16:02:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15084.733990 secs 2024-04-26T16:02:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15084.733990 secs 2024-04-26T16:02:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:25 smithi096 ceph-mon[32753]: pgmap v8562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:02:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:25 smithi194 ceph-mon[33017]: pgmap v8562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:02:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:25 smithi033 ceph-mon[27845]: pgmap v8562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:02:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:27 smithi096 ceph-mon[32753]: pgmap v8563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:27 smithi194 ceph-mon[33017]: pgmap v8563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:27 smithi033 ceph-mon[27845]: pgmap v8563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15086.923581 secs 2024-04-26T16:02:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15086.923581 secs 2024-04-26T16:02:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15086.923581 secs 2024-04-26T16:02:28.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:28.822+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:28.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:28.822+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:02:28.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:28.822+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:29 smithi194 ceph-mon[33017]: pgmap v8564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15089.734139 secs 2024-04-26T16:02:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:29 smithi033 ceph-mon[27845]: pgmap v8564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15089.734139 secs 2024-04-26T16:02:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:29 smithi096 ceph-mon[32753]: pgmap v8564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:02:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15089.734139 secs 2024-04-26T16:02:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:31 smithi194 ceph-mon[33017]: pgmap v8565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:02:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:31 smithi033 ceph-mon[27845]: pgmap v8565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:02:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:31 smithi096 ceph-mon[32753]: pgmap v8565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:02:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:33 smithi194 ceph-mon[33017]: pgmap v8566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15091.923638 secs 2024-04-26T16:02:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:33 smithi033 ceph-mon[27845]: pgmap v8566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15091.923638 secs 2024-04-26T16:02:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:33 smithi096 ceph-mon[32753]: pgmap v8566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15091.923638 secs 2024-04-26T16:02:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15094.734229 secs 2024-04-26T16:02:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15094.734229 secs 2024-04-26T16:02:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15094.734229 secs 2024-04-26T16:02:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:35 smithi194 ceph-mon[33017]: pgmap v8567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:35 smithi033 ceph-mon[27845]: pgmap v8567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:35 smithi096 ceph-mon[32753]: pgmap v8567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:37 smithi194 ceph-mon[33017]: pgmap v8568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:37 smithi033 ceph-mon[27845]: pgmap v8568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:37 smithi096 ceph-mon[32753]: pgmap v8568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:38.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15096.923695 secs 2024-04-26T16:02:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15096.923695 secs 2024-04-26T16:02:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15096.923695 secs 2024-04-26T16:02:39.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:39 smithi194 ceph-mon[33017]: pgmap v8569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:39.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15099.734375 secs 2024-04-26T16:02:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:39 smithi033 ceph-mon[27845]: pgmap v8569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15099.734375 secs 2024-04-26T16:02:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:39 smithi096 ceph-mon[32753]: pgmap v8569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15099.734375 secs 2024-04-26T16:02:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:41 smithi194 ceph-mon[33017]: pgmap v8570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:41 smithi033 ceph-mon[27845]: pgmap v8570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:41 smithi096 ceph-mon[32753]: pgmap v8570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:43 smithi194 ceph-mon[33017]: pgmap v8571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15101.923754 secs 2024-04-26T16:02:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:43 smithi033 ceph-mon[27845]: pgmap v8571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15101.923754 secs 2024-04-26T16:02:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:43 smithi096 ceph-mon[32753]: pgmap v8571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15101.923754 secs 2024-04-26T16:02:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:43.837+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:43.837+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:02:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:43.837+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15104.734498 secs 2024-04-26T16:02:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15104.734498 secs 2024-04-26T16:02:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15104.734498 secs 2024-04-26T16:02:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:45 smithi194 ceph-mon[33017]: pgmap v8572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:45 smithi033 ceph-mon[27845]: pgmap v8572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:45 smithi096 ceph-mon[32753]: pgmap v8572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:47 smithi194 ceph-mon[33017]: pgmap v8573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:47 smithi033 ceph-mon[27845]: pgmap v8573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:47 smithi096 ceph-mon[32753]: pgmap v8573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15106.923810 secs 2024-04-26T16:02:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15106.923810 secs 2024-04-26T16:02:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15106.923810 secs 2024-04-26T16:02:49.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:49 smithi033 ceph-mon[27845]: pgmap v8574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:49.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15109.734634 secs 2024-04-26T16:02:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:49 smithi194 ceph-mon[33017]: pgmap v8574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15109.734634 secs 2024-04-26T16:02:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:49 smithi096 ceph-mon[32753]: pgmap v8574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15109.734634 secs 2024-04-26T16:02:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:50 smithi194 ceph-mon[33017]: pgmap v8575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:50 smithi033 ceph-mon[27845]: pgmap v8575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:50 smithi096 ceph-mon[32753]: pgmap v8575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:52 smithi096 ceph-mon[32753]: pgmap v8576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:52 smithi194 ceph-mon[33017]: pgmap v8576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:52 smithi033 ceph-mon[27845]: pgmap v8576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15111.923869 secs 2024-04-26T16:02:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15114.734748 secs 2024-04-26T16:02:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15111.923869 secs 2024-04-26T16:02:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15114.734748 secs 2024-04-26T16:02:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15111.923869 secs 2024-04-26T16:02:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15114.734748 secs 2024-04-26T16:02:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:54 smithi194 ceph-mon[33017]: pgmap v8577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:54 smithi033 ceph-mon[27845]: pgmap v8577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:54 smithi096 ceph-mon[32753]: pgmap v8577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:56 smithi194 ceph-mon[33017]: pgmap v8578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:56 smithi033 ceph-mon[27845]: pgmap v8578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:56 smithi096 ceph-mon[32753]: pgmap v8578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:02:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:58.848+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:58.848+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:02:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:02:58.848+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:02:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:58 smithi194 ceph-mon[33017]: pgmap v8579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15116.923926 secs 2024-04-26T16:02:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:02:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15119.734870 secs 2024-04-26T16:02:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:58 smithi033 ceph-mon[27845]: pgmap v8579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:59.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15116.923926 secs 2024-04-26T16:02:59.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:02:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15119.734870 secs 2024-04-26T16:02:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:58 smithi096 ceph-mon[32753]: pgmap v8579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:02:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15116.923926 secs 2024-04-26T16:02:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:02:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15119.734870 secs 2024-04-26T16:03:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:00 smithi194 ceph-mon[33017]: pgmap v8580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:00 smithi033 ceph-mon[27845]: pgmap v8580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:00 smithi096 ceph-mon[32753]: pgmap v8580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:02 smithi194 ceph-mon[33017]: pgmap v8581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:02 smithi033 ceph-mon[27845]: pgmap v8581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:02 smithi096 ceph-mon[32753]: pgmap v8581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15121.923976 secs 2024-04-26T16:03:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15124.735011 secs 2024-04-26T16:03:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15121.923976 secs 2024-04-26T16:03:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15124.735011 secs 2024-04-26T16:03:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15121.923976 secs 2024-04-26T16:03:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15124.735011 secs 2024-04-26T16:03:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:04 smithi194 ceph-mon[33017]: pgmap v8582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:04 smithi033 ceph-mon[27845]: pgmap v8582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:04 smithi096 ceph-mon[32753]: pgmap v8582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:06 smithi194 ceph-mon[33017]: pgmap v8583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:06 smithi033 ceph-mon[27845]: pgmap v8583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:06 smithi096 ceph-mon[32753]: pgmap v8583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:08.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:08 smithi096 ceph-mon[32753]: pgmap v8584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:08.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15126.924034 secs 2024-04-26T16:03:08.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15129.735133 secs 2024-04-26T16:03:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:08 smithi194 ceph-mon[33017]: pgmap v8584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15126.924034 secs 2024-04-26T16:03:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15129.735133 secs 2024-04-26T16:03:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:08 smithi033 ceph-mon[27845]: pgmap v8584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15126.924034 secs 2024-04-26T16:03:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15129.735133 secs 2024-04-26T16:03:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:10 smithi194 ceph-mon[33017]: pgmap v8585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:10 smithi033 ceph-mon[27845]: pgmap v8585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:10 smithi096 ceph-mon[32753]: pgmap v8585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:12 smithi194 ceph-mon[33017]: pgmap v8586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:12 smithi033 ceph-mon[27845]: pgmap v8586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:12 smithi096 ceph-mon[32753]: pgmap v8586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:13.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:13.859+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:13.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:13.859+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:03:13.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:13.859+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15131.924103 secs 2024-04-26T16:03:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15134.735243 secs 2024-04-26T16:03:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15131.924103 secs 2024-04-26T16:03:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15134.735243 secs 2024-04-26T16:03:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15131.924103 secs 2024-04-26T16:03:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15134.735243 secs 2024-04-26T16:03:15.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:14 smithi033 ceph-mon[27845]: pgmap v8587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:14 smithi194 ceph-mon[33017]: pgmap v8587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:14 smithi096 ceph-mon[32753]: pgmap v8587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:16 smithi194 ceph-mon[33017]: pgmap v8588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:16 smithi033 ceph-mon[27845]: pgmap v8588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:16 smithi096 ceph-mon[32753]: pgmap v8588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:18 smithi194 ceph-mon[33017]: pgmap v8589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15136.924181 secs 2024-04-26T16:03:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15139.735361 secs 2024-04-26T16:03:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:18 smithi033 ceph-mon[27845]: pgmap v8589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15136.924181 secs 2024-04-26T16:03:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15139.735361 secs 2024-04-26T16:03:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:18 smithi096 ceph-mon[32753]: pgmap v8589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15136.924181 secs 2024-04-26T16:03:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15139.735361 secs 2024-04-26T16:03:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:20 smithi194 ceph-mon[33017]: pgmap v8590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:03:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:20 smithi033 ceph-mon[27845]: pgmap v8590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:03:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:20 smithi096 ceph-mon[32753]: pgmap v8590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:03:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:03:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:03:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:21 smithi194 ceph-mon[33017]: pgmap v8591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:21 smithi194 ceph-mon[33017]: pgmap v8592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:03:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:03:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:03:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:21 smithi033 ceph-mon[27845]: pgmap v8591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:21 smithi033 ceph-mon[27845]: pgmap v8592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:03:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:03:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:03:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:21 smithi096 ceph-mon[32753]: pgmap v8591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:21 smithi096 ceph-mon[32753]: pgmap v8592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:03:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15141.924277 secs 2024-04-26T16:03:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15141.924277 secs 2024-04-26T16:03:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15141.924277 secs 2024-04-26T16:03:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:24 smithi033 ceph-mon[27845]: pgmap v8593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15144.735422 secs 2024-04-26T16:03:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:24 smithi096 ceph-mon[32753]: pgmap v8593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15144.735422 secs 2024-04-26T16:03:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:24 smithi194 ceph-mon[33017]: pgmap v8593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15144.735422 secs 2024-04-26T16:03:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:25 smithi194 ceph-mon[33017]: pgmap v8594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:25 smithi033 ceph-mon[27845]: pgmap v8594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:25 smithi096 ceph-mon[32753]: pgmap v8594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:27 smithi194 ceph-mon[33017]: pgmap v8595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:27 smithi033 ceph-mon[27845]: pgmap v8595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:27 smithi096 ceph-mon[32753]: pgmap v8595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:03:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15146.924336 secs 2024-04-26T16:03:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15146.924336 secs 2024-04-26T16:03:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15146.924336 secs 2024-04-26T16:03:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:28.866+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:28.867+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:28.867+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:03:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15149.735512 secs 2024-04-26T16:03:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:29 smithi194 ceph-mon[33017]: pgmap v8596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:03:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15149.735512 secs 2024-04-26T16:03:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:29 smithi033 ceph-mon[27845]: pgmap v8596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:03:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15149.735512 secs 2024-04-26T16:03:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:29 smithi096 ceph-mon[32753]: pgmap v8596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:03:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:31 smithi194 ceph-mon[33017]: pgmap v8597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:03:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:31 smithi033 ceph-mon[27845]: pgmap v8597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:03:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:31 smithi096 ceph-mon[32753]: pgmap v8597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15151.924439 secs 2024-04-26T16:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:33 smithi194 ceph-mon[33017]: pgmap v8598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15154.735629 secs 2024-04-26T16:03:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15151.924439 secs 2024-04-26T16:03:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:33 smithi033 ceph-mon[27845]: pgmap v8598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15154.735629 secs 2024-04-26T16:03:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15151.924439 secs 2024-04-26T16:03:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:33 smithi096 ceph-mon[32753]: pgmap v8598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15154.735629 secs 2024-04-26T16:03:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:35 smithi194 ceph-mon[33017]: pgmap v8599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:35 smithi033 ceph-mon[27845]: pgmap v8599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:35 smithi096 ceph-mon[32753]: pgmap v8599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:37 smithi194 ceph-mon[33017]: pgmap v8600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:37 smithi033 ceph-mon[27845]: pgmap v8600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:37 smithi096 ceph-mon[32753]: pgmap v8600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:38.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15156.924499 secs 2024-04-26T16:03:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15156.924499 secs 2024-04-26T16:03:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15156.924499 secs 2024-04-26T16:03:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15159.735771 secs 2024-04-26T16:03:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:39 smithi194 ceph-mon[33017]: pgmap v8601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15159.735771 secs 2024-04-26T16:03:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:39 smithi033 ceph-mon[27845]: pgmap v8601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15159.735771 secs 2024-04-26T16:03:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:39 smithi096 ceph-mon[32753]: pgmap v8601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:41 smithi194 ceph-mon[33017]: pgmap v8602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:41 smithi033 ceph-mon[27845]: pgmap v8602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:41 smithi096 ceph-mon[32753]: pgmap v8602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15161.924597 secs 2024-04-26T16:03:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15161.924597 secs 2024-04-26T16:03:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15161.924597 secs 2024-04-26T16:03:43.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:43.881+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:43.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:43.881+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:03:43.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:43.881+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:44 smithi033 ceph-mon[27845]: pgmap v8603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15164.735900 secs 2024-04-26T16:03:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:44 smithi096 ceph-mon[32753]: pgmap v8603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15164.735900 secs 2024-04-26T16:03:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:44 smithi194 ceph-mon[33017]: pgmap v8603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15164.735900 secs 2024-04-26T16:03:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:45 smithi194 ceph-mon[33017]: pgmap v8604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:45 smithi033 ceph-mon[27845]: pgmap v8604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:45 smithi096 ceph-mon[32753]: pgmap v8604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:47 smithi194 ceph-mon[33017]: pgmap v8605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:47 smithi033 ceph-mon[27845]: pgmap v8605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:47 smithi096 ceph-mon[32753]: pgmap v8605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15166.924656 secs 2024-04-26T16:03:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15166.924656 secs 2024-04-26T16:03:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15166.924656 secs 2024-04-26T16:03:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15169.736014 secs 2024-04-26T16:03:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:49 smithi194 ceph-mon[33017]: pgmap v8606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15169.736014 secs 2024-04-26T16:03:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:49 smithi033 ceph-mon[27845]: pgmap v8606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15169.736014 secs 2024-04-26T16:03:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:49 smithi096 ceph-mon[32753]: pgmap v8606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:51 smithi194 ceph-mon[33017]: pgmap v8607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:51 smithi033 ceph-mon[27845]: pgmap v8607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:51 smithi096 ceph-mon[32753]: pgmap v8607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15171.924738 secs 2024-04-26T16:03:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15171.924738 secs 2024-04-26T16:03:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15171.924738 secs 2024-04-26T16:03:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:54 smithi033 ceph-mon[27845]: pgmap v8608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15174.736160 secs 2024-04-26T16:03:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:54 smithi096 ceph-mon[32753]: pgmap v8608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15174.736160 secs 2024-04-26T16:03:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:54 smithi194 ceph-mon[33017]: pgmap v8608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15174.736160 secs 2024-04-26T16:03:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:55 smithi194 ceph-mon[33017]: pgmap v8609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:55 smithi033 ceph-mon[27845]: pgmap v8609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:55 smithi096 ceph-mon[32753]: pgmap v8609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:03:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:57 smithi194 ceph-mon[33017]: pgmap v8610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:57 smithi033 ceph-mon[27845]: pgmap v8610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:57 smithi096 ceph-mon[32753]: pgmap v8610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15176.924798 secs 2024-04-26T16:03:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15176.924798 secs 2024-04-26T16:03:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15176.924798 secs 2024-04-26T16:03:58.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:58.892+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:58.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:58.892+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:03:58.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:03:58.892+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:03:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15179.736227 secs 2024-04-26T16:03:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:03:59 smithi194 ceph-mon[33017]: pgmap v8611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15179.736227 secs 2024-04-26T16:03:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:03:59 smithi033 ceph-mon[27845]: pgmap v8611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:03:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15179.736227 secs 2024-04-26T16:03:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:03:59 smithi096 ceph-mon[32753]: pgmap v8611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:01 smithi194 ceph-mon[33017]: pgmap v8612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:01 smithi033 ceph-mon[27845]: pgmap v8612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:01 smithi096 ceph-mon[32753]: pgmap v8612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15181.924883 secs 2024-04-26T16:04:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15181.924883 secs 2024-04-26T16:04:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15181.924883 secs 2024-04-26T16:04:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:04 smithi033 ceph-mon[27845]: pgmap v8613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15184.736286 secs 2024-04-26T16:04:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:04 smithi096 ceph-mon[32753]: pgmap v8613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15184.736286 secs 2024-04-26T16:04:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:04 smithi194 ceph-mon[33017]: pgmap v8613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15184.736286 secs 2024-04-26T16:04:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:05 smithi194 ceph-mon[33017]: pgmap v8614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:05 smithi033 ceph-mon[27845]: pgmap v8614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:05 smithi096 ceph-mon[32753]: pgmap v8614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:07 smithi194 ceph-mon[33017]: pgmap v8615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:07 smithi033 ceph-mon[27845]: pgmap v8615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:07 smithi096 ceph-mon[32753]: pgmap v8615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:08.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15186.924982 secs 2024-04-26T16:04:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15186.924982 secs 2024-04-26T16:04:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15186.924982 secs 2024-04-26T16:04:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15189.736378 secs 2024-04-26T16:04:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:09 smithi194 ceph-mon[33017]: pgmap v8616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15189.736378 secs 2024-04-26T16:04:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:09 smithi033 ceph-mon[27845]: pgmap v8616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15189.736378 secs 2024-04-26T16:04:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:09 smithi096 ceph-mon[32753]: pgmap v8616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:11 smithi194 ceph-mon[33017]: pgmap v8617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:11 smithi033 ceph-mon[27845]: pgmap v8617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:11 smithi096 ceph-mon[32753]: pgmap v8617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15191.925070 secs 2024-04-26T16:04:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15191.925070 secs 2024-04-26T16:04:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15191.925070 secs 2024-04-26T16:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:13.904+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:13.904+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:13.904+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:14 smithi096 ceph-mon[32753]: pgmap v8618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15194.736489 secs 2024-04-26T16:04:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:14 smithi194 ceph-mon[33017]: pgmap v8618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15194.736489 secs 2024-04-26T16:04:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:14 smithi033 ceph-mon[27845]: pgmap v8618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15194.736489 secs 2024-04-26T16:04:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:15 smithi194 ceph-mon[33017]: pgmap v8619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:15 smithi033 ceph-mon[27845]: pgmap v8619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:15 smithi096 ceph-mon[32753]: pgmap v8619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:17 smithi194 ceph-mon[33017]: pgmap v8620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:17 smithi033 ceph-mon[27845]: pgmap v8620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:17 smithi096 ceph-mon[32753]: pgmap v8620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15196.925133 secs 2024-04-26T16:04:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15196.925133 secs 2024-04-26T16:04:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15196.925133 secs 2024-04-26T16:04:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15199.736608 secs 2024-04-26T16:04:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:19 smithi194 ceph-mon[33017]: pgmap v8621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15199.736608 secs 2024-04-26T16:04:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:19 smithi033 ceph-mon[27845]: pgmap v8621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15199.736608 secs 2024-04-26T16:04:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:19 smithi096 ceph-mon[32753]: pgmap v8621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:21 smithi194 ceph-mon[33017]: pgmap v8622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:21 smithi033 ceph-mon[27845]: pgmap v8622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:21 smithi096 ceph-mon[32753]: pgmap v8622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:04:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:04:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:04:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15201.925194 secs 2024-04-26T16:04:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:04:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:04:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:23 smithi194 ceph-mon[33017]: pgmap v8623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:23 smithi194 ceph-mon[33017]: pgmap v8624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:04:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15201.925194 secs 2024-04-26T16:04:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:04:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:04:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:23 smithi033 ceph-mon[27845]: pgmap v8623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:23 smithi033 ceph-mon[27845]: pgmap v8624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:04:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15201.925194 secs 2024-04-26T16:04:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:04:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:04:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:23 smithi096 ceph-mon[32753]: pgmap v8623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:23 smithi096 ceph-mon[32753]: pgmap v8624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:04:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15204.736694 secs 2024-04-26T16:04:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15204.736694 secs 2024-04-26T16:04:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15204.736694 secs 2024-04-26T16:04:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:25 smithi033 ceph-mon[27845]: pgmap v8625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:25 smithi096 ceph-mon[32753]: pgmap v8625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:25 smithi194 ceph-mon[33017]: pgmap v8625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:27 smithi194 ceph-mon[33017]: pgmap v8626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:27 smithi033 ceph-mon[27845]: pgmap v8626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:27 smithi096 ceph-mon[32753]: pgmap v8626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:28.919+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:28.919+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:04:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:28.919+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15206.925283 secs 2024-04-26T16:04:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15209.736818 secs 2024-04-26T16:04:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15206.925283 secs 2024-04-26T16:04:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15209.736818 secs 2024-04-26T16:04:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15206.925283 secs 2024-04-26T16:04:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15209.736818 secs 2024-04-26T16:04:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:29 smithi194 ceph-mon[33017]: pgmap v8627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:04:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:29 smithi033 ceph-mon[27845]: pgmap v8627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:04:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:29 smithi096 ceph-mon[32753]: pgmap v8627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:04:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:31 smithi194 ceph-mon[33017]: pgmap v8628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:31 smithi033 ceph-mon[27845]: pgmap v8628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:31 smithi096 ceph-mon[32753]: pgmap v8628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:04:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15211.925373 secs 2024-04-26T16:04:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:33 smithi194 ceph-mon[33017]: pgmap v8629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:04:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15214.736951 secs 2024-04-26T16:04:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15211.925373 secs 2024-04-26T16:04:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:33 smithi033 ceph-mon[27845]: pgmap v8629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:04:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15214.736951 secs 2024-04-26T16:04:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15211.925373 secs 2024-04-26T16:04:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:33 smithi096 ceph-mon[32753]: pgmap v8629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:04:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15214.736951 secs 2024-04-26T16:04:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:35 smithi194 ceph-mon[33017]: pgmap v8630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:35 smithi033 ceph-mon[27845]: pgmap v8630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:35 smithi096 ceph-mon[32753]: pgmap v8630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:37 smithi194 ceph-mon[33017]: pgmap v8631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:37 smithi033 ceph-mon[27845]: pgmap v8631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:37 smithi096 ceph-mon[32753]: pgmap v8631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15216.925456 secs 2024-04-26T16:04:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15219.737070 secs 2024-04-26T16:04:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15216.925456 secs 2024-04-26T16:04:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15219.737070 secs 2024-04-26T16:04:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15216.925456 secs 2024-04-26T16:04:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15219.737070 secs 2024-04-26T16:04:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:40 smithi194 ceph-mon[33017]: pgmap v8632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:40 smithi033 ceph-mon[27845]: pgmap v8632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:40 smithi096 ceph-mon[32753]: pgmap v8632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:42 smithi194 ceph-mon[33017]: pgmap v8633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:42 smithi033 ceph-mon[27845]: pgmap v8633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:42 smithi096 ceph-mon[32753]: pgmap v8633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:43.924+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:43.924+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:43.924+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:04:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15221.925544 secs 2024-04-26T16:04:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:44 smithi194 ceph-mon[33017]: pgmap v8634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15224.737187 secs 2024-04-26T16:04:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15221.925544 secs 2024-04-26T16:04:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:44 smithi033 ceph-mon[27845]: pgmap v8634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15224.737187 secs 2024-04-26T16:04:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15221.925544 secs 2024-04-26T16:04:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:44 smithi096 ceph-mon[32753]: pgmap v8634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:04:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15224.737187 secs 2024-04-26T16:04:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:46 smithi194 ceph-mon[33017]: pgmap v8635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:46 smithi033 ceph-mon[27845]: pgmap v8635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:46 smithi096 ceph-mon[32753]: pgmap v8635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:04:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:48 smithi194 ceph-mon[33017]: pgmap v8636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:48 smithi033 ceph-mon[27845]: pgmap v8636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:48 smithi096 ceph-mon[32753]: pgmap v8636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:49.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15226.925643 secs 2024-04-26T16:04:49.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15229.737283 secs 2024-04-26T16:04:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15226.925643 secs 2024-04-26T16:04:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15229.737283 secs 2024-04-26T16:04:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15226.925643 secs 2024-04-26T16:04:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15229.737283 secs 2024-04-26T16:04:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:50 smithi194 ceph-mon[33017]: pgmap v8637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:50 smithi033 ceph-mon[27845]: pgmap v8637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:50 smithi096 ceph-mon[32753]: pgmap v8637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:52 smithi194 ceph-mon[33017]: pgmap v8638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:52 smithi033 ceph-mon[27845]: pgmap v8638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:52 smithi096 ceph-mon[32753]: pgmap v8638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15231.925738 secs 2024-04-26T16:04:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:54 smithi194 ceph-mon[33017]: pgmap v8639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15234.737397 secs 2024-04-26T16:04:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15231.925738 secs 2024-04-26T16:04:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:54 smithi033 ceph-mon[27845]: pgmap v8639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15234.737397 secs 2024-04-26T16:04:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15231.925738 secs 2024-04-26T16:04:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:54 smithi096 ceph-mon[32753]: pgmap v8639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15234.737397 secs 2024-04-26T16:04:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:56 smithi194 ceph-mon[33017]: pgmap v8640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:56 smithi033 ceph-mon[27845]: pgmap v8640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:56 smithi096 ceph-mon[32753]: pgmap v8640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:04:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:58 smithi194 ceph-mon[33017]: pgmap v8641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:58 smithi033 ceph-mon[27845]: pgmap v8641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:58 smithi096 ceph-mon[32753]: pgmap v8641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:04:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:58.939+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:58.939+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:04:58.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:04:58.939+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:04:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15236.925801 secs 2024-04-26T16:04:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:04:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15239.737520 secs 2024-04-26T16:04:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15236.925801 secs 2024-04-26T16:04:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:04:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15239.737520 secs 2024-04-26T16:04:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15236.925801 secs 2024-04-26T16:04:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:04:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15239.737520 secs 2024-04-26T16:05:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:00 smithi194 ceph-mon[33017]: pgmap v8642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:00 smithi033 ceph-mon[27845]: pgmap v8642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:00 smithi096 ceph-mon[32753]: pgmap v8642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:02 smithi194 ceph-mon[33017]: pgmap v8643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:02 smithi033 ceph-mon[27845]: pgmap v8643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:02 smithi096 ceph-mon[32753]: pgmap v8643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15241.925885 secs 2024-04-26T16:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:04 smithi194 ceph-mon[33017]: pgmap v8644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15244.737631 secs 2024-04-26T16:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15241.925885 secs 2024-04-26T16:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:04 smithi033 ceph-mon[27845]: pgmap v8644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15244.737631 secs 2024-04-26T16:05:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15241.925885 secs 2024-04-26T16:05:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:04 smithi096 ceph-mon[32753]: pgmap v8644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15244.737631 secs 2024-04-26T16:05:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:06 smithi194 ceph-mon[33017]: pgmap v8645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:06 smithi033 ceph-mon[27845]: pgmap v8645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:06 smithi096 ceph-mon[32753]: pgmap v8645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:08 smithi194 ceph-mon[33017]: pgmap v8646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:08 smithi033 ceph-mon[27845]: pgmap v8646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:08 smithi096 ceph-mon[32753]: pgmap v8646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15246.925974 secs 2024-04-26T16:05:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15249.737748 secs 2024-04-26T16:05:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15246.925974 secs 2024-04-26T16:05:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15249.737748 secs 2024-04-26T16:05:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15246.925974 secs 2024-04-26T16:05:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15249.737748 secs 2024-04-26T16:05:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:10 smithi033 ceph-mon[27845]: pgmap v8647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:10 smithi096 ceph-mon[32753]: pgmap v8647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:10 smithi194 ceph-mon[33017]: pgmap v8647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:12.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:12 smithi194 ceph-mon[33017]: pgmap v8648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:12 smithi033 ceph-mon[27845]: pgmap v8648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:12 smithi096 ceph-mon[32753]: pgmap v8648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:13.955+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:13.955+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:13.955+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:05:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15251.926059 secs 2024-04-26T16:05:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:14 smithi033 ceph-mon[27845]: pgmap v8649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15254.737860 secs 2024-04-26T16:05:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15251.926059 secs 2024-04-26T16:05:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:14 smithi096 ceph-mon[32753]: pgmap v8649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15254.737860 secs 2024-04-26T16:05:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15251.926059 secs 2024-04-26T16:05:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:14 smithi194 ceph-mon[33017]: pgmap v8649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15254.737860 secs 2024-04-26T16:05:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:16 smithi033 ceph-mon[27845]: pgmap v8650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:16 smithi096 ceph-mon[32753]: pgmap v8650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:16 smithi194 ceph-mon[33017]: pgmap v8650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:18 smithi033 ceph-mon[27845]: pgmap v8651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15256.926145 secs 2024-04-26T16:05:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:18 smithi096 ceph-mon[32753]: pgmap v8651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15256.926145 secs 2024-04-26T16:05:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:18 smithi194 ceph-mon[33017]: pgmap v8651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15256.926145 secs 2024-04-26T16:05:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15259.738005 secs 2024-04-26T16:05:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15259.738005 secs 2024-04-26T16:05:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15259.738005 secs 2024-04-26T16:05:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:20 smithi033 ceph-mon[27845]: pgmap v8652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:20 smithi096 ceph-mon[32753]: pgmap v8652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:20 smithi194 ceph-mon[33017]: pgmap v8652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:22 smithi033 ceph-mon[27845]: pgmap v8653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:22 smithi096 ceph-mon[32753]: pgmap v8653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:22 smithi194 ceph-mon[33017]: pgmap v8653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15261.926248 secs 2024-04-26T16:05:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15261.926248 secs 2024-04-26T16:05:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15261.926248 secs 2024-04-26T16:05:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.392+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T16:05:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.392+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T16:05:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.392+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feecc006630 con 0x564385821900 2024-04-26T16:05:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.392+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x564385821900 2024-04-26T16:05:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.393+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T16:05:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.393+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:05:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.393+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0020 con 0x564385821900 2024-04-26T16:05:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:24.394+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:05:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:24 smithi033 ceph-mon[27845]: pgmap v8654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:05:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15264.738127 secs 2024-04-26T16:05:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:24 smithi096 ceph-mon[32753]: pgmap v8654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:05:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15264.738127 secs 2024-04-26T16:05:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:24 smithi194 ceph-mon[33017]: pgmap v8654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:05:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15264.738127 secs 2024-04-26T16:05:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:05:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:05:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:05:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:05:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:05:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:05:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:05:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:05:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:05:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:26 smithi033 ceph-mon[27845]: pgmap v8655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:05:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:26 smithi033 ceph-mon[27845]: pgmap v8656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:26 smithi096 ceph-mon[32753]: pgmap v8655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:05:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:26 smithi096 ceph-mon[32753]: pgmap v8656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:26.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:26 smithi194 ceph-mon[33017]: pgmap v8655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:05:26.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:26 smithi194 ceph-mon[33017]: pgmap v8656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:28 smithi096 ceph-mon[32753]: pgmap v8657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15266.926308 secs 2024-04-26T16:05:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:28 smithi194 ceph-mon[33017]: pgmap v8657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15266.926308 secs 2024-04-26T16:05:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:28 smithi033 ceph-mon[27845]: pgmap v8657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15266.926308 secs 2024-04-26T16:05:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:28.971+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:28.971+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:05:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:28.971+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15269.738246 secs 2024-04-26T16:05:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15269.738246 secs 2024-04-26T16:05:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15269.738246 secs 2024-04-26T16:05:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:30 smithi194 ceph-mon[33017]: pgmap v8658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:30 smithi033 ceph-mon[27845]: pgmap v8658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:30 smithi096 ceph-mon[32753]: pgmap v8658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:32 smithi194 ceph-mon[33017]: pgmap v8659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:32 smithi033 ceph-mon[27845]: pgmap v8659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:32 smithi096 ceph-mon[32753]: pgmap v8659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15271.926369 secs 2024-04-26T16:05:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15271.926369 secs 2024-04-26T16:05:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15271.926369 secs 2024-04-26T16:05:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:34 smithi194 ceph-mon[33017]: pgmap v8660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15274.738311 secs 2024-04-26T16:05:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:34 smithi033 ceph-mon[27845]: pgmap v8660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15274.738311 secs 2024-04-26T16:05:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:34 smithi096 ceph-mon[32753]: pgmap v8660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:05:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15274.738311 secs 2024-04-26T16:05:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:36 smithi194 ceph-mon[33017]: pgmap v8661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:05:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:36 smithi033 ceph-mon[27845]: pgmap v8661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:05:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:36 smithi096 ceph-mon[32753]: pgmap v8661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:05:38.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:38 smithi096 ceph-mon[32753]: pgmap v8662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:38.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15276.926428 secs 2024-04-26T16:05:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:38 smithi194 ceph-mon[33017]: pgmap v8662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15276.926428 secs 2024-04-26T16:05:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:38 smithi033 ceph-mon[27845]: pgmap v8662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15276.926428 secs 2024-04-26T16:05:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15279.738432 secs 2024-04-26T16:05:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15279.738432 secs 2024-04-26T16:05:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15279.738432 secs 2024-04-26T16:05:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:40 smithi194 ceph-mon[33017]: pgmap v8663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:40 smithi033 ceph-mon[27845]: pgmap v8663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:40 smithi096 ceph-mon[32753]: pgmap v8663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:42 smithi194 ceph-mon[33017]: pgmap v8664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:42 smithi033 ceph-mon[27845]: pgmap v8664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:42 smithi096 ceph-mon[32753]: pgmap v8664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15281.926489 secs 2024-04-26T16:05:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15281.926489 secs 2024-04-26T16:05:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15281.926489 secs 2024-04-26T16:05:43.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:43.981+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:43.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:43.981+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:43.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:43.981+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:05:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:44 smithi194 ceph-mon[33017]: pgmap v8665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15284.738562 secs 2024-04-26T16:05:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:44 smithi033 ceph-mon[27845]: pgmap v8665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15284.738562 secs 2024-04-26T16:05:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:44 smithi096 ceph-mon[32753]: pgmap v8665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15284.738562 secs 2024-04-26T16:05:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:46 smithi194 ceph-mon[33017]: pgmap v8666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:46 smithi033 ceph-mon[27845]: pgmap v8666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:46 smithi096 ceph-mon[32753]: pgmap v8666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:48 smithi194 ceph-mon[33017]: pgmap v8667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15286.926576 secs 2024-04-26T16:05:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:48 smithi033 ceph-mon[27845]: pgmap v8667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15286.926576 secs 2024-04-26T16:05:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:48 smithi096 ceph-mon[32753]: pgmap v8667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15286.926576 secs 2024-04-26T16:05:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15289.738699 secs 2024-04-26T16:05:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15289.738699 secs 2024-04-26T16:05:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15289.738699 secs 2024-04-26T16:05:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:50 smithi194 ceph-mon[33017]: pgmap v8668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:50 smithi033 ceph-mon[27845]: pgmap v8668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:50 smithi096 ceph-mon[32753]: pgmap v8668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:52 smithi033 ceph-mon[27845]: pgmap v8669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:52 smithi194 ceph-mon[33017]: pgmap v8669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:52 smithi096 ceph-mon[32753]: pgmap v8669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15291.926660 secs 2024-04-26T16:05:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15291.926660 secs 2024-04-26T16:05:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15291.926660 secs 2024-04-26T16:05:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:54 smithi194 ceph-mon[33017]: pgmap v8670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15294.738858 secs 2024-04-26T16:05:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:54 smithi033 ceph-mon[27845]: pgmap v8670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15294.738858 secs 2024-04-26T16:05:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:54 smithi096 ceph-mon[32753]: pgmap v8670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15294.738858 secs 2024-04-26T16:05:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:56 smithi194 ceph-mon[33017]: pgmap v8671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:56 smithi033 ceph-mon[27845]: pgmap v8671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:56 smithi096 ceph-mon[32753]: pgmap v8671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:05:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:58 smithi194 ceph-mon[33017]: pgmap v8672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15296.926719 secs 2024-04-26T16:05:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:58 smithi033 ceph-mon[27845]: pgmap v8672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15296.926719 secs 2024-04-26T16:05:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:58 smithi096 ceph-mon[32753]: pgmap v8672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15296.926719 secs 2024-04-26T16:05:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:58.996+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:58.996+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:05:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:05:58.996+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15299.738961 secs 2024-04-26T16:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:05:59 smithi194 ceph-mon[33017]: pgmap v8673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15299.738961 secs 2024-04-26T16:05:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:05:59 smithi033 ceph-mon[27845]: pgmap v8673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:05:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15299.738961 secs 2024-04-26T16:05:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:05:59 smithi096 ceph-mon[32753]: pgmap v8673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:01 smithi033 ceph-mon[27845]: pgmap v8674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:01 smithi096 ceph-mon[32753]: pgmap v8674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:01 smithi194 ceph-mon[33017]: pgmap v8674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:03 smithi033 ceph-mon[27845]: pgmap v8675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15301.926778 secs 2024-04-26T16:06:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15304.739058 secs 2024-04-26T16:06:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:03 smithi096 ceph-mon[32753]: pgmap v8675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15301.926778 secs 2024-04-26T16:06:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15304.739058 secs 2024-04-26T16:06:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:03 smithi194 ceph-mon[33017]: pgmap v8675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15301.926778 secs 2024-04-26T16:06:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15304.739058 secs 2024-04-26T16:06:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:05 smithi033 ceph-mon[27845]: pgmap v8676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:05 smithi096 ceph-mon[32753]: pgmap v8676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:05 smithi194 ceph-mon[33017]: pgmap v8676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:07 smithi033 ceph-mon[27845]: pgmap v8677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:07 smithi096 ceph-mon[32753]: pgmap v8677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:07 smithi194 ceph-mon[33017]: pgmap v8677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15306.926862 secs 2024-04-26T16:06:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15309.739192 secs 2024-04-26T16:06:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15306.926862 secs 2024-04-26T16:06:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15309.739192 secs 2024-04-26T16:06:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15306.926862 secs 2024-04-26T16:06:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15309.739192 secs 2024-04-26T16:06:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:09 smithi194 ceph-mon[33017]: pgmap v8678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:09 smithi033 ceph-mon[27845]: pgmap v8678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:09 smithi096 ceph-mon[32753]: pgmap v8678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:11 smithi194 ceph-mon[33017]: pgmap v8679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:11 smithi033 ceph-mon[27845]: pgmap v8679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:11 smithi096 ceph-mon[32753]: pgmap v8679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:14.012+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:14.012+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:06:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:14.012+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:13 smithi194 ceph-mon[33017]: pgmap v8680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15311.926946 secs 2024-04-26T16:06:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15314.739319 secs 2024-04-26T16:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:13 smithi033 ceph-mon[27845]: pgmap v8680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15311.926946 secs 2024-04-26T16:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15314.739319 secs 2024-04-26T16:06:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:13 smithi096 ceph-mon[32753]: pgmap v8680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15311.926946 secs 2024-04-26T16:06:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15314.739319 secs 2024-04-26T16:06:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:15 smithi194 ceph-mon[33017]: pgmap v8681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:15 smithi033 ceph-mon[27845]: pgmap v8681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:15 smithi096 ceph-mon[32753]: pgmap v8681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:17 smithi194 ceph-mon[33017]: pgmap v8682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:17 smithi033 ceph-mon[27845]: pgmap v8682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:17 smithi096 ceph-mon[32753]: pgmap v8682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15316.927029 secs 2024-04-26T16:06:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15319.739454 secs 2024-04-26T16:06:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15316.927029 secs 2024-04-26T16:06:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15319.739454 secs 2024-04-26T16:06:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15316.927029 secs 2024-04-26T16:06:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15319.739454 secs 2024-04-26T16:06:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:19 smithi194 ceph-mon[33017]: pgmap v8683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:19 smithi033 ceph-mon[27845]: pgmap v8683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:19 smithi096 ceph-mon[32753]: pgmap v8683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:21 smithi194 ceph-mon[33017]: pgmap v8684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:21 smithi033 ceph-mon[27845]: pgmap v8684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:21 smithi096 ceph-mon[32753]: pgmap v8684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:23 smithi194 ceph-mon[33017]: pgmap v8685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15321.927116 secs 2024-04-26T16:06:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15324.739575 secs 2024-04-26T16:06:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:23 smithi033 ceph-mon[27845]: pgmap v8685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15321.927116 secs 2024-04-26T16:06:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15324.739575 secs 2024-04-26T16:06:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:23 smithi096 ceph-mon[32753]: pgmap v8685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15321.927116 secs 2024-04-26T16:06:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15324.739575 secs 2024-04-26T16:06:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:25 smithi194 ceph-mon[33017]: pgmap v8686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:06:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:25 smithi033 ceph-mon[27845]: pgmap v8686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:06:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:25 smithi096 ceph-mon[32753]: pgmap v8686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:06:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:06:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:06:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:26 smithi194 ceph-mon[33017]: pgmap v8687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:26 smithi194 ceph-mon[33017]: pgmap v8688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:06:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:06:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:06:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:26 smithi033 ceph-mon[27845]: pgmap v8687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:26 smithi033 ceph-mon[27845]: pgmap v8688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:06:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:06:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:06:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:26 smithi096 ceph-mon[32753]: pgmap v8687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:26 smithi096 ceph-mon[32753]: pgmap v8688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:06:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15326.927175 secs 2024-04-26T16:06:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15326.927175 secs 2024-04-26T16:06:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15326.927175 secs 2024-04-26T16:06:29.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:29.017+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:29.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:29.017+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:06:29.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:29.017+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:29 smithi194 ceph-mon[33017]: pgmap v8689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15329.739676 secs 2024-04-26T16:06:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:29 smithi033 ceph-mon[27845]: pgmap v8689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15329.739676 secs 2024-04-26T16:06:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:29 smithi096 ceph-mon[32753]: pgmap v8689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15329.739676 secs 2024-04-26T16:06:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:31 smithi194 ceph-mon[33017]: pgmap v8690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:31 smithi033 ceph-mon[27845]: pgmap v8690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:31 smithi096 ceph-mon[32753]: pgmap v8690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:33 smithi194 ceph-mon[33017]: pgmap v8691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15331.927264 secs 2024-04-26T16:06:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:33 smithi033 ceph-mon[27845]: pgmap v8691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15331.927264 secs 2024-04-26T16:06:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:33 smithi096 ceph-mon[32753]: pgmap v8691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:06:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15331.927264 secs 2024-04-26T16:06:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15334.739758 secs 2024-04-26T16:06:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15334.739758 secs 2024-04-26T16:06:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15334.739758 secs 2024-04-26T16:06:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:35 smithi194 ceph-mon[33017]: pgmap v8692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:06:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:35 smithi033 ceph-mon[27845]: pgmap v8692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:06:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:35 smithi096 ceph-mon[32753]: pgmap v8692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:06:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:37 smithi194 ceph-mon[33017]: pgmap v8693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:06:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:37 smithi033 ceph-mon[27845]: pgmap v8693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:06:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:37 smithi096 ceph-mon[32753]: pgmap v8693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:06:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15336.927322 secs 2024-04-26T16:06:38.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15336.927322 secs 2024-04-26T16:06:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15336.927322 secs 2024-04-26T16:06:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:39 smithi194 ceph-mon[33017]: pgmap v8694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15339.739855 secs 2024-04-26T16:06:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:39 smithi033 ceph-mon[27845]: pgmap v8694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15339.739855 secs 2024-04-26T16:06:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:39 smithi096 ceph-mon[32753]: pgmap v8694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15339.739855 secs 2024-04-26T16:06:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:41 smithi194 ceph-mon[33017]: pgmap v8695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:41 smithi033 ceph-mon[27845]: pgmap v8695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:41 smithi096 ceph-mon[32753]: pgmap v8695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:42 smithi194 ceph-mon[33017]: pgmap v8696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:42 smithi033 ceph-mon[27845]: pgmap v8696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:42 smithi096 ceph-mon[32753]: pgmap v8696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15341.927384 secs 2024-04-26T16:06:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15341.927384 secs 2024-04-26T16:06:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15341.927384 secs 2024-04-26T16:06:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:44.018+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:44.018+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:44.018+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:06:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15344.739973 secs 2024-04-26T16:06:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:44 smithi194 ceph-mon[33017]: pgmap v8697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15344.739973 secs 2024-04-26T16:06:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:44 smithi033 ceph-mon[27845]: pgmap v8697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15344.739973 secs 2024-04-26T16:06:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:44 smithi096 ceph-mon[32753]: pgmap v8697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:47 smithi194 ceph-mon[33017]: pgmap v8698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:47 smithi033 ceph-mon[27845]: pgmap v8698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:47 smithi096 ceph-mon[32753]: pgmap v8698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15346.927446 secs 2024-04-26T16:06:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15346.927446 secs 2024-04-26T16:06:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15346.927446 secs 2024-04-26T16:06:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:49 smithi194 ceph-mon[33017]: pgmap v8699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15349.740041 secs 2024-04-26T16:06:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:49 smithi033 ceph-mon[27845]: pgmap v8699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15349.740041 secs 2024-04-26T16:06:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:49 smithi096 ceph-mon[32753]: pgmap v8699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15349.740041 secs 2024-04-26T16:06:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:51 smithi194 ceph-mon[33017]: pgmap v8700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:51 smithi033 ceph-mon[27845]: pgmap v8700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:51 smithi096 ceph-mon[32753]: pgmap v8700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:53 smithi194 ceph-mon[33017]: pgmap v8701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15351.927529 secs 2024-04-26T16:06:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:53 smithi033 ceph-mon[27845]: pgmap v8701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15351.927529 secs 2024-04-26T16:06:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:53 smithi096 ceph-mon[32753]: pgmap v8701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15351.927529 secs 2024-04-26T16:06:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15354.740159 secs 2024-04-26T16:06:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15354.740159 secs 2024-04-26T16:06:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15354.740159 secs 2024-04-26T16:06:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:55 smithi194 ceph-mon[33017]: pgmap v8702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:55 smithi033 ceph-mon[27845]: pgmap v8702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:55 smithi096 ceph-mon[32753]: pgmap v8702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:56 smithi194 ceph-mon[33017]: pgmap v8703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:56 smithi033 ceph-mon[27845]: pgmap v8703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:56 smithi096 ceph-mon[32753]: pgmap v8703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:06:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15356.927590 secs 2024-04-26T16:06:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15356.927590 secs 2024-04-26T16:06:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15356.927590 secs 2024-04-26T16:06:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:59.028+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:59.028+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:06:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:06:59.028+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:06:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:59 smithi194 ceph-mon[33017]: pgmap v8704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15359.740278 secs 2024-04-26T16:06:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:59 smithi033 ceph-mon[27845]: pgmap v8704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15359.740278 secs 2024-04-26T16:06:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:59 smithi096 ceph-mon[32753]: pgmap v8704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:06:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15359.740278 secs 2024-04-26T16:07:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:01 smithi194 ceph-mon[33017]: pgmap v8705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:01 smithi033 ceph-mon[27845]: pgmap v8705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:01 smithi096 ceph-mon[32753]: pgmap v8705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:03 smithi194 ceph-mon[33017]: pgmap v8706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15361.927672 secs 2024-04-26T16:07:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:03 smithi033 ceph-mon[27845]: pgmap v8706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15361.927672 secs 2024-04-26T16:07:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:03 smithi096 ceph-mon[32753]: pgmap v8706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15361.927672 secs 2024-04-26T16:07:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15364.740395 secs 2024-04-26T16:07:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15364.740395 secs 2024-04-26T16:07:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15364.740395 secs 2024-04-26T16:07:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:05 smithi194 ceph-mon[33017]: pgmap v8707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:05 smithi033 ceph-mon[27845]: pgmap v8707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:05 smithi096 ceph-mon[32753]: pgmap v8707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:07 smithi194 ceph-mon[33017]: pgmap v8708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:07 smithi033 ceph-mon[27845]: pgmap v8708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:07 smithi096 ceph-mon[32753]: pgmap v8708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:08.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15366.927756 secs 2024-04-26T16:07:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15366.927756 secs 2024-04-26T16:07:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15366.927756 secs 2024-04-26T16:07:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:09 smithi194 ceph-mon[33017]: pgmap v8709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15369.740488 secs 2024-04-26T16:07:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:09 smithi033 ceph-mon[27845]: pgmap v8709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15369.740488 secs 2024-04-26T16:07:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:09 smithi096 ceph-mon[32753]: pgmap v8709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:07:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15369.740488 secs 2024-04-26T16:07:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:10 smithi194 ceph-mon[33017]: pgmap v8710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:10 smithi033 ceph-mon[27845]: pgmap v8710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:10 smithi096 ceph-mon[32753]: pgmap v8710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:07:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:13 smithi194 ceph-mon[33017]: pgmap v8711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15371.927849 secs 2024-04-26T16:07:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:13 smithi033 ceph-mon[27845]: pgmap v8711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15371.927849 secs 2024-04-26T16:07:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:13 smithi096 ceph-mon[32753]: pgmap v8711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15371.927849 secs 2024-04-26T16:07:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:14.039+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:14.039+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:14.039+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:07:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15374.740569 secs 2024-04-26T16:07:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15374.740569 secs 2024-04-26T16:07:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15374.740569 secs 2024-04-26T16:07:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:15 smithi194 ceph-mon[33017]: pgmap v8712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:15 smithi033 ceph-mon[27845]: pgmap v8712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:15 smithi096 ceph-mon[32753]: pgmap v8712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:17 smithi194 ceph-mon[33017]: pgmap v8713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:17 smithi033 ceph-mon[27845]: pgmap v8713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:17 smithi096 ceph-mon[32753]: pgmap v8713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15376.927934 secs 2024-04-26T16:07:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15376.927934 secs 2024-04-26T16:07:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15376.927934 secs 2024-04-26T16:07:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:19 smithi194 ceph-mon[33017]: pgmap v8714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15379.740709 secs 2024-04-26T16:07:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:19 smithi033 ceph-mon[27845]: pgmap v8714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15379.740709 secs 2024-04-26T16:07:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:19 smithi096 ceph-mon[32753]: pgmap v8714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15379.740709 secs 2024-04-26T16:07:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:20 smithi194 ceph-mon[33017]: pgmap v8715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:20 smithi033 ceph-mon[27845]: pgmap v8715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:20 smithi096 ceph-mon[32753]: pgmap v8715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:23 smithi194 ceph-mon[33017]: pgmap v8716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15381.928025 secs 2024-04-26T16:07:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:23 smithi033 ceph-mon[27845]: pgmap v8716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15381.928025 secs 2024-04-26T16:07:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:23 smithi096 ceph-mon[32753]: pgmap v8716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15381.928025 secs 2024-04-26T16:07:24.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15384.740827 secs 2024-04-26T16:07:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15384.740827 secs 2024-04-26T16:07:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15384.740827 secs 2024-04-26T16:07:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:25 smithi194 ceph-mon[33017]: pgmap v8717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:25 smithi033 ceph-mon[27845]: pgmap v8717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:25 smithi096 ceph-mon[32753]: pgmap v8717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:27 smithi194 ceph-mon[33017]: pgmap v8718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:27 smithi033 ceph-mon[27845]: pgmap v8718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:27 smithi096 ceph-mon[32753]: pgmap v8718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15386.928083 secs 2024-04-26T16:07:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:07:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:07:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:07:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15386.928083 secs 2024-04-26T16:07:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:07:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:07:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:07:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15386.928083 secs 2024-04-26T16:07:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:07:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:07:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:07:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:29.054+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:29.054+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:07:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:29.054+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:29 smithi194 ceph-mon[33017]: pgmap v8719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:29 smithi194 ceph-mon[33017]: pgmap v8720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15389.740963 secs 2024-04-26T16:07:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:29 smithi033 ceph-mon[27845]: pgmap v8719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:29 smithi033 ceph-mon[27845]: pgmap v8720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15389.740963 secs 2024-04-26T16:07:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:29 smithi096 ceph-mon[32753]: pgmap v8719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:29 smithi096 ceph-mon[32753]: pgmap v8720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15389.740963 secs 2024-04-26T16:07:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:30 smithi194 ceph-mon[33017]: pgmap v8721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:30 smithi033 ceph-mon[27845]: pgmap v8721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:30 smithi096 ceph-mon[32753]: pgmap v8721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:32 smithi194 ceph-mon[33017]: pgmap v8722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:32 smithi033 ceph-mon[27845]: pgmap v8722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:32 smithi096 ceph-mon[32753]: pgmap v8722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15391.928164 secs 2024-04-26T16:07:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15394.741083 secs 2024-04-26T16:07:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15391.928164 secs 2024-04-26T16:07:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15394.741083 secs 2024-04-26T16:07:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15391.928164 secs 2024-04-26T16:07:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15394.741083 secs 2024-04-26T16:07:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:34 smithi194 ceph-mon[33017]: pgmap v8723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:34 smithi033 ceph-mon[27845]: pgmap v8723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:34 smithi096 ceph-mon[32753]: pgmap v8723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:36 smithi194 ceph-mon[33017]: pgmap v8724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:36 smithi033 ceph-mon[27845]: pgmap v8724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:36 smithi096 ceph-mon[32753]: pgmap v8724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:07:39.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15396.928267 secs 2024-04-26T16:07:39.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:38 smithi096 ceph-mon[32753]: pgmap v8725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:07:39.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15399.741201 secs 2024-04-26T16:07:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15396.928267 secs 2024-04-26T16:07:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:38 smithi194 ceph-mon[33017]: pgmap v8725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:07:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15399.741201 secs 2024-04-26T16:07:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15396.928267 secs 2024-04-26T16:07:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:38 smithi033 ceph-mon[27845]: pgmap v8725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:07:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15399.741201 secs 2024-04-26T16:07:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:40 smithi194 ceph-mon[33017]: pgmap v8726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:40 smithi033 ceph-mon[27845]: pgmap v8726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:40 smithi096 ceph-mon[32753]: pgmap v8726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:42 smithi194 ceph-mon[33017]: pgmap v8727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:42 smithi033 ceph-mon[27845]: pgmap v8727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:42 smithi096 ceph-mon[32753]: pgmap v8727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:44.063+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:44.063+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:07:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:44.063+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15401.928349 secs 2024-04-26T16:07:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15404.741316 secs 2024-04-26T16:07:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15401.928349 secs 2024-04-26T16:07:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15404.741316 secs 2024-04-26T16:07:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15401.928349 secs 2024-04-26T16:07:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15404.741316 secs 2024-04-26T16:07:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:44 smithi194 ceph-mon[33017]: pgmap v8728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:44 smithi033 ceph-mon[27845]: pgmap v8728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:44 smithi096 ceph-mon[32753]: pgmap v8728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:46 smithi194 ceph-mon[33017]: pgmap v8729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:46 smithi033 ceph-mon[27845]: pgmap v8729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:46 smithi096 ceph-mon[32753]: pgmap v8729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15406.928431 secs 2024-04-26T16:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:48 smithi194 ceph-mon[33017]: pgmap v8730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15409.741398 secs 2024-04-26T16:07:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15406.928431 secs 2024-04-26T16:07:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:48 smithi033 ceph-mon[27845]: pgmap v8730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15409.741398 secs 2024-04-26T16:07:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15406.928431 secs 2024-04-26T16:07:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:48 smithi096 ceph-mon[32753]: pgmap v8730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15409.741398 secs 2024-04-26T16:07:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:50 smithi194 ceph-mon[33017]: pgmap v8731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:50 smithi033 ceph-mon[27845]: pgmap v8731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:50 smithi096 ceph-mon[32753]: pgmap v8731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:53 smithi194 ceph-mon[33017]: pgmap v8732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:53 smithi033 ceph-mon[27845]: pgmap v8732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:53 smithi096 ceph-mon[32753]: pgmap v8732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15411.928510 secs 2024-04-26T16:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15414.741515 secs 2024-04-26T16:07:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15411.928510 secs 2024-04-26T16:07:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15414.741515 secs 2024-04-26T16:07:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15411.928510 secs 2024-04-26T16:07:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15414.741515 secs 2024-04-26T16:07:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:55 smithi194 ceph-mon[33017]: pgmap v8733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:55 smithi033 ceph-mon[27845]: pgmap v8733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:55 smithi096 ceph-mon[32753]: pgmap v8733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:57 smithi194 ceph-mon[33017]: pgmap v8734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:57 smithi033 ceph-mon[27845]: pgmap v8734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:57 smithi096 ceph-mon[32753]: pgmap v8734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:07:59.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:59.079+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:59.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:59.079+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:07:59.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:07:59.079+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:07:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15416.928593 secs 2024-04-26T16:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:59 smithi194 ceph-mon[33017]: pgmap v8735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:07:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15419.741615 secs 2024-04-26T16:07:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15416.928593 secs 2024-04-26T16:07:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:59 smithi033 ceph-mon[27845]: pgmap v8735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:07:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15419.741615 secs 2024-04-26T16:07:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15416.928593 secs 2024-04-26T16:07:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:59 smithi096 ceph-mon[32753]: pgmap v8735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:07:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:07:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15419.741615 secs 2024-04-26T16:08:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:01 smithi194 ceph-mon[33017]: pgmap v8736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:01 smithi033 ceph-mon[27845]: pgmap v8736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:01 smithi096 ceph-mon[32753]: pgmap v8736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:03 smithi194 ceph-mon[33017]: pgmap v8737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:03 smithi033 ceph-mon[27845]: pgmap v8737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:03 smithi096 ceph-mon[32753]: pgmap v8737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15421.928675 secs 2024-04-26T16:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15424.741731 secs 2024-04-26T16:08:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15421.928675 secs 2024-04-26T16:08:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15424.741731 secs 2024-04-26T16:08:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15421.928675 secs 2024-04-26T16:08:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15424.741731 secs 2024-04-26T16:08:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:05 smithi194 ceph-mon[33017]: pgmap v8738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:05 smithi033 ceph-mon[27845]: pgmap v8738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:05 smithi096 ceph-mon[32753]: pgmap v8738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:07 smithi194 ceph-mon[33017]: pgmap v8739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:07 smithi033 ceph-mon[27845]: pgmap v8739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:07 smithi096 ceph-mon[32753]: pgmap v8739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15426.928736 secs 2024-04-26T16:08:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:09 smithi194 ceph-mon[33017]: pgmap v8740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15429.741864 secs 2024-04-26T16:08:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15426.928736 secs 2024-04-26T16:08:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:09 smithi033 ceph-mon[27845]: pgmap v8740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15429.741864 secs 2024-04-26T16:08:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15426.928736 secs 2024-04-26T16:08:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:09 smithi096 ceph-mon[32753]: pgmap v8740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15429.741864 secs 2024-04-26T16:08:11.318 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:11 smithi033 ceph-mon[27845]: pgmap v8741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:11 smithi194 ceph-mon[33017]: pgmap v8741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:11 smithi096 ceph-mon[32753]: pgmap v8741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:13 smithi194 ceph-mon[33017]: pgmap v8742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:13 smithi033 ceph-mon[27845]: pgmap v8742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:13 smithi096 ceph-mon[32753]: pgmap v8742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:14.095+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:14.095+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:14.095+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:08:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15431.928815 secs 2024-04-26T16:08:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15434.741979 secs 2024-04-26T16:08:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15431.928815 secs 2024-04-26T16:08:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15434.741979 secs 2024-04-26T16:08:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15431.928815 secs 2024-04-26T16:08:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15434.741979 secs 2024-04-26T16:08:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:15 smithi194 ceph-mon[33017]: pgmap v8743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:15 smithi033 ceph-mon[27845]: pgmap v8743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:15 smithi096 ceph-mon[32753]: pgmap v8743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:17 smithi194 ceph-mon[33017]: pgmap v8744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:17 smithi033 ceph-mon[27845]: pgmap v8744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:17 smithi096 ceph-mon[32753]: pgmap v8744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15436.928874 secs 2024-04-26T16:08:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:19 smithi194 ceph-mon[33017]: pgmap v8745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15439.742098 secs 2024-04-26T16:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15436.928874 secs 2024-04-26T16:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:19 smithi033 ceph-mon[27845]: pgmap v8745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15439.742098 secs 2024-04-26T16:08:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15436.928874 secs 2024-04-26T16:08:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:19 smithi096 ceph-mon[32753]: pgmap v8745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15439.742098 secs 2024-04-26T16:08:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:21 smithi194 ceph-mon[33017]: pgmap v8746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:21 smithi033 ceph-mon[27845]: pgmap v8746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:21 smithi096 ceph-mon[32753]: pgmap v8746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:23 smithi033 ceph-mon[27845]: pgmap v8747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:23 smithi096 ceph-mon[32753]: pgmap v8747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:23 smithi194 ceph-mon[33017]: pgmap v8747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:24.422+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:08:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:24.422+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:08:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:24.423+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:08:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:24.423+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:08:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15441.928959 secs 2024-04-26T16:08:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15444.742250 secs 2024-04-26T16:08:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15441.928959 secs 2024-04-26T16:08:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15444.742250 secs 2024-04-26T16:08:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15441.928959 secs 2024-04-26T16:08:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15444.742250 secs 2024-04-26T16:08:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:25 smithi033 ceph-mon[27845]: pgmap v8748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:25 smithi096 ceph-mon[32753]: pgmap v8748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:25 smithi194 ceph-mon[33017]: pgmap v8748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:27 smithi033 ceph-mon[27845]: pgmap v8749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:27 smithi096 ceph-mon[32753]: pgmap v8749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:27 smithi194 ceph-mon[33017]: pgmap v8749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:29.106+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:29.106+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:29.106+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:08:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15446.929042 secs 2024-04-26T16:08:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:29 smithi033 ceph-mon[27845]: pgmap v8750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15449.742349 secs 2024-04-26T16:08:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15446.929042 secs 2024-04-26T16:08:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:29 smithi096 ceph-mon[32753]: pgmap v8750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15449.742349 secs 2024-04-26T16:08:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15446.929042 secs 2024-04-26T16:08:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:29 smithi194 ceph-mon[33017]: pgmap v8750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15449.742349 secs 2024-04-26T16:08:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:08:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:08:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:08:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:08:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:08:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:08:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:08:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:08:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:08:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:31 smithi033 ceph-mon[27845]: pgmap v8751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:08:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:31 smithi033 ceph-mon[27845]: pgmap v8752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:31 smithi096 ceph-mon[32753]: pgmap v8751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:08:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:31 smithi096 ceph-mon[32753]: pgmap v8752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:31 smithi194 ceph-mon[33017]: pgmap v8751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:08:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:31 smithi194 ceph-mon[33017]: pgmap v8752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:33 smithi033 ceph-mon[27845]: pgmap v8753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15451.929104 secs 2024-04-26T16:08:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:33 smithi096 ceph-mon[32753]: pgmap v8753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15451.929104 secs 2024-04-26T16:08:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:33 smithi194 ceph-mon[33017]: pgmap v8753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15451.929104 secs 2024-04-26T16:08:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15454.742464 secs 2024-04-26T16:08:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15454.742464 secs 2024-04-26T16:08:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15454.742464 secs 2024-04-26T16:08:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:35 smithi033 ceph-mon[27845]: pgmap v8754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:35 smithi096 ceph-mon[32753]: pgmap v8754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:35 smithi194 ceph-mon[33017]: pgmap v8754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:37 smithi033 ceph-mon[27845]: pgmap v8755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:37 smithi096 ceph-mon[32753]: pgmap v8755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:37 smithi194 ceph-mon[33017]: pgmap v8755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15456.929192 secs 2024-04-26T16:08:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15456.929192 secs 2024-04-26T16:08:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15456.929192 secs 2024-04-26T16:08:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:39 smithi096 ceph-mon[32753]: pgmap v8756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15459.742584 secs 2024-04-26T16:08:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:39 smithi194 ceph-mon[33017]: pgmap v8756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15459.742584 secs 2024-04-26T16:08:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:39 smithi033 ceph-mon[27845]: pgmap v8756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:08:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15459.742584 secs 2024-04-26T16:08:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:41 smithi194 ceph-mon[33017]: pgmap v8757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:08:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:41 smithi033 ceph-mon[27845]: pgmap v8757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:08:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:41 smithi096 ceph-mon[32753]: pgmap v8757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:08:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:43 smithi033 ceph-mon[27845]: pgmap v8758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15461.929281 secs 2024-04-26T16:08:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:43 smithi194 ceph-mon[33017]: pgmap v8758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15461.929281 secs 2024-04-26T16:08:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:43 smithi096 ceph-mon[32753]: pgmap v8758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15461.929281 secs 2024-04-26T16:08:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:44.112+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:44.112+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:08:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:44.112+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15464.742699 secs 2024-04-26T16:08:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15464.742699 secs 2024-04-26T16:08:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15464.742699 secs 2024-04-26T16:08:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:45 smithi194 ceph-mon[33017]: pgmap v8759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:45 smithi033 ceph-mon[27845]: pgmap v8759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:45 smithi096 ceph-mon[32753]: pgmap v8759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:47 smithi194 ceph-mon[33017]: pgmap v8760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:47 smithi033 ceph-mon[27845]: pgmap v8760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:47 smithi096 ceph-mon[32753]: pgmap v8760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15466.929341 secs 2024-04-26T16:08:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15466.929341 secs 2024-04-26T16:08:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15466.929341 secs 2024-04-26T16:08:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:49 smithi194 ceph-mon[33017]: pgmap v8761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15469.742820 secs 2024-04-26T16:08:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:49 smithi033 ceph-mon[27845]: pgmap v8761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15469.742820 secs 2024-04-26T16:08:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:49 smithi096 ceph-mon[32753]: pgmap v8761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15469.742820 secs 2024-04-26T16:08:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:51 smithi194 ceph-mon[33017]: pgmap v8762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:51 smithi033 ceph-mon[27845]: pgmap v8762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:51 smithi096 ceph-mon[32753]: pgmap v8762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:53 smithi194 ceph-mon[33017]: pgmap v8763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15471.929423 secs 2024-04-26T16:08:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:53 smithi033 ceph-mon[27845]: pgmap v8763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15471.929423 secs 2024-04-26T16:08:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:53 smithi096 ceph-mon[32753]: pgmap v8763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15471.929423 secs 2024-04-26T16:08:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15474.742951 secs 2024-04-26T16:08:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15474.742951 secs 2024-04-26T16:08:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15474.742951 secs 2024-04-26T16:08:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:55 smithi194 ceph-mon[33017]: pgmap v8764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:55 smithi033 ceph-mon[27845]: pgmap v8764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:55 smithi096 ceph-mon[32753]: pgmap v8764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:57 smithi194 ceph-mon[33017]: pgmap v8765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:57 smithi033 ceph-mon[27845]: pgmap v8765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:57 smithi096 ceph-mon[32753]: pgmap v8765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:08:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15476.929484 secs 2024-04-26T16:08:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15476.929484 secs 2024-04-26T16:08:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15476.929484 secs 2024-04-26T16:08:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:59.126+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:59.126+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:08:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:08:59.126+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:08:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:59 smithi194 ceph-mon[33017]: pgmap v8766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15479.743089 secs 2024-04-26T16:08:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:59 smithi033 ceph-mon[27845]: pgmap v8766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15479.743089 secs 2024-04-26T16:08:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:59 smithi096 ceph-mon[32753]: pgmap v8766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:08:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15479.743089 secs 2024-04-26T16:09:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:01 smithi194 ceph-mon[33017]: pgmap v8767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:01 smithi033 ceph-mon[27845]: pgmap v8767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:01 smithi096 ceph-mon[32753]: pgmap v8767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:03 smithi194 ceph-mon[33017]: pgmap v8768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15481.929568 secs 2024-04-26T16:09:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:03 smithi033 ceph-mon[27845]: pgmap v8768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15481.929568 secs 2024-04-26T16:09:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:03 smithi096 ceph-mon[32753]: pgmap v8768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15481.929568 secs 2024-04-26T16:09:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15484.743206 secs 2024-04-26T16:09:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15484.743206 secs 2024-04-26T16:09:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15484.743206 secs 2024-04-26T16:09:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:05 smithi194 ceph-mon[33017]: pgmap v8769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:05 smithi033 ceph-mon[27845]: pgmap v8769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:05 smithi096 ceph-mon[32753]: pgmap v8769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:07 smithi194 ceph-mon[33017]: pgmap v8770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:07 smithi033 ceph-mon[27845]: pgmap v8770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:07 smithi096 ceph-mon[32753]: pgmap v8770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15486.929651 secs 2024-04-26T16:09:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15486.929651 secs 2024-04-26T16:09:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15486.929651 secs 2024-04-26T16:09:09.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:09 smithi096 ceph-mon[32753]: pgmap v8771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:09.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15489.743277 secs 2024-04-26T16:09:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:09 smithi194 ceph-mon[33017]: pgmap v8771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15489.743277 secs 2024-04-26T16:09:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:09 smithi033 ceph-mon[27845]: pgmap v8771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15489.743277 secs 2024-04-26T16:09:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:11 smithi194 ceph-mon[33017]: pgmap v8772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:11 smithi033 ceph-mon[27845]: pgmap v8772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:11 smithi096 ceph-mon[32753]: pgmap v8772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:13.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:13 smithi194 ceph-mon[33017]: pgmap v8773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:13.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15491.929734 secs 2024-04-26T16:09:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:13 smithi033 ceph-mon[27845]: pgmap v8773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15491.929734 secs 2024-04-26T16:09:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:13 smithi096 ceph-mon[32753]: pgmap v8773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15491.929734 secs 2024-04-26T16:09:14.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:14.141+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:14.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:14.141+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:14.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:14.141+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:09:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15494.743391 secs 2024-04-26T16:09:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15494.743391 secs 2024-04-26T16:09:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15494.743391 secs 2024-04-26T16:09:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:15 smithi194 ceph-mon[33017]: pgmap v8774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:15 smithi033 ceph-mon[27845]: pgmap v8774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:15 smithi096 ceph-mon[32753]: pgmap v8774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:17 smithi194 ceph-mon[33017]: pgmap v8775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:17 smithi033 ceph-mon[27845]: pgmap v8775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:17 smithi096 ceph-mon[32753]: pgmap v8775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:18 smithi096 ceph-mon[32753]: pgmap v8776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15496.929817 secs 2024-04-26T16:09:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15499.743510 secs 2024-04-26T16:09:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:18 smithi194 ceph-mon[33017]: pgmap v8776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15496.929817 secs 2024-04-26T16:09:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15499.743510 secs 2024-04-26T16:09:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:18 smithi033 ceph-mon[27845]: pgmap v8776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15496.929817 secs 2024-04-26T16:09:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15499.743510 secs 2024-04-26T16:09:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:20 smithi194 ceph-mon[33017]: pgmap v8777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:20 smithi033 ceph-mon[27845]: pgmap v8777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:20 smithi096 ceph-mon[32753]: pgmap v8777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:22 smithi194 ceph-mon[33017]: pgmap v8778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:22 smithi033 ceph-mon[27845]: pgmap v8778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:22 smithi096 ceph-mon[32753]: pgmap v8778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:24.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15501.929901 secs 2024-04-26T16:09:24.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15504.743622 secs 2024-04-26T16:09:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15501.929901 secs 2024-04-26T16:09:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15504.743622 secs 2024-04-26T16:09:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15501.929901 secs 2024-04-26T16:09:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15504.743622 secs 2024-04-26T16:09:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:24 smithi194 ceph-mon[33017]: pgmap v8779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:24 smithi033 ceph-mon[27845]: pgmap v8779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:24 smithi096 ceph-mon[32753]: pgmap v8779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:26 smithi194 ceph-mon[33017]: pgmap v8780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:26 smithi033 ceph-mon[27845]: pgmap v8780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:26 smithi096 ceph-mon[32753]: pgmap v8780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:28 smithi194 ceph-mon[33017]: pgmap v8781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15506.930135 secs 2024-04-26T16:09:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15509.743743 secs 2024-04-26T16:09:29.138 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:28 smithi033 ceph-mon[27845]: pgmap v8781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:29.138 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15506.930135 secs 2024-04-26T16:09:29.138 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15509.743743 secs 2024-04-26T16:09:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:29.145+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:29.145+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:09:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:29.145+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:28 smithi096 ceph-mon[32753]: pgmap v8781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15506.930135 secs 2024-04-26T16:09:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15509.743743 secs 2024-04-26T16:09:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:30 smithi194 ceph-mon[33017]: pgmap v8782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:30 smithi033 ceph-mon[27845]: pgmap v8782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:30 smithi096 ceph-mon[32753]: pgmap v8782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:32 smithi194 ceph-mon[33017]: pgmap v8783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:32 smithi194 ceph-mon[33017]: pgmap v8784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:09:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:32 smithi194 ceph-mon[33017]: pgmap v8785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:32 smithi033 ceph-mon[27845]: pgmap v8783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:32 smithi033 ceph-mon[27845]: pgmap v8784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:09:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:32 smithi033 ceph-mon[27845]: pgmap v8785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:32 smithi096 ceph-mon[32753]: pgmap v8783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:32 smithi096 ceph-mon[32753]: pgmap v8784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:09:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:32 smithi096 ceph-mon[32753]: pgmap v8785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:09:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15511.930226 secs 2024-04-26T16:09:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15514.743883 secs 2024-04-26T16:09:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15511.930226 secs 2024-04-26T16:09:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15514.743883 secs 2024-04-26T16:09:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15511.930226 secs 2024-04-26T16:09:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15514.743883 secs 2024-04-26T16:09:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:34 smithi194 ceph-mon[33017]: pgmap v8786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:09:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:34 smithi033 ceph-mon[27845]: pgmap v8786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:09:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:34 smithi096 ceph-mon[32753]: pgmap v8786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:09:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:37 smithi033 ceph-mon[27845]: pgmap v8787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:37 smithi096 ceph-mon[32753]: pgmap v8787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:37 smithi194 ceph-mon[33017]: pgmap v8787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15516.930315 secs 2024-04-26T16:09:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15516.930315 secs 2024-04-26T16:09:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15516.930315 secs 2024-04-26T16:09:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:39 smithi096 ceph-mon[32753]: pgmap v8788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:09:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15519.743965 secs 2024-04-26T16:09:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:39 smithi194 ceph-mon[33017]: pgmap v8788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:09:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15519.743965 secs 2024-04-26T16:09:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:39 smithi033 ceph-mon[27845]: pgmap v8788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:09:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15519.743965 secs 2024-04-26T16:09:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:41 smithi194 ceph-mon[33017]: pgmap v8789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:41 smithi033 ceph-mon[27845]: pgmap v8789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:41 smithi096 ceph-mon[32753]: pgmap v8789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:09:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:43 smithi194 ceph-mon[33017]: pgmap v8790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:09:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15521.930405 secs 2024-04-26T16:09:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:43 smithi033 ceph-mon[27845]: pgmap v8790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:09:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15521.930405 secs 2024-04-26T16:09:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:43 smithi096 ceph-mon[32753]: pgmap v8790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:09:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15521.930405 secs 2024-04-26T16:09:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:44.155+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:44.155+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:09:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:44.155+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15524.744115 secs 2024-04-26T16:09:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15524.744115 secs 2024-04-26T16:09:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15524.744115 secs 2024-04-26T16:09:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:45 smithi033 ceph-mon[27845]: pgmap v8791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:45 smithi194 ceph-mon[33017]: pgmap v8791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:45 smithi096 ceph-mon[32753]: pgmap v8791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:47 smithi194 ceph-mon[33017]: pgmap v8792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:47 smithi033 ceph-mon[27845]: pgmap v8792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:47 smithi096 ceph-mon[32753]: pgmap v8792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15526.930465 secs 2024-04-26T16:09:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15526.930465 secs 2024-04-26T16:09:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15526.930465 secs 2024-04-26T16:09:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:49 smithi194 ceph-mon[33017]: pgmap v8793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15529.744240 secs 2024-04-26T16:09:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:49 smithi033 ceph-mon[27845]: pgmap v8793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15529.744240 secs 2024-04-26T16:09:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:49 smithi096 ceph-mon[32753]: pgmap v8793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15529.744240 secs 2024-04-26T16:09:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:51 smithi194 ceph-mon[33017]: pgmap v8794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:51 smithi033 ceph-mon[27845]: pgmap v8794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:51 smithi096 ceph-mon[32753]: pgmap v8794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:53 smithi194 ceph-mon[33017]: pgmap v8795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15531.930549 secs 2024-04-26T16:09:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:53 smithi033 ceph-mon[27845]: pgmap v8795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15531.930549 secs 2024-04-26T16:09:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:53 smithi096 ceph-mon[32753]: pgmap v8795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15531.930549 secs 2024-04-26T16:09:54.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15534.744356 secs 2024-04-26T16:09:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15534.744356 secs 2024-04-26T16:09:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15534.744356 secs 2024-04-26T16:09:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:55 smithi194 ceph-mon[33017]: pgmap v8796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:55 smithi033 ceph-mon[27845]: pgmap v8796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:55 smithi096 ceph-mon[32753]: pgmap v8796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:57 smithi194 ceph-mon[33017]: pgmap v8797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:57 smithi033 ceph-mon[27845]: pgmap v8797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:57 smithi096 ceph-mon[32753]: pgmap v8797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:09:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15536.930633 secs 2024-04-26T16:09:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15536.930633 secs 2024-04-26T16:09:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15536.930633 secs 2024-04-26T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:59.161+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:59.161+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:09:59.161+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:09:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:59 smithi194 ceph-mon[33017]: pgmap v8798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15539.744455 secs 2024-04-26T16:09:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:59 smithi033 ceph-mon[27845]: pgmap v8798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15539.744455 secs 2024-04-26T16:09:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:59 smithi096 ceph-mon[32753]: pgmap v8798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:09:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15539.744455 secs 2024-04-26T16:10:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:10:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:10:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:10:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:01 smithi194 ceph-mon[33017]: pgmap v8799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:01 smithi033 ceph-mon[27845]: pgmap v8799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:01.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:01 smithi096 ceph-mon[32753]: pgmap v8799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:03 smithi194 ceph-mon[33017]: pgmap v8800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15541.930721 secs 2024-04-26T16:10:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:03 smithi033 ceph-mon[27845]: pgmap v8800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15541.930721 secs 2024-04-26T16:10:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:03 smithi096 ceph-mon[32753]: pgmap v8800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15541.930721 secs 2024-04-26T16:10:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15544.744553 secs 2024-04-26T16:10:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15544.744553 secs 2024-04-26T16:10:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15544.744553 secs 2024-04-26T16:10:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:05 smithi194 ceph-mon[33017]: pgmap v8801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:05 smithi033 ceph-mon[27845]: pgmap v8801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:05 smithi096 ceph-mon[32753]: pgmap v8801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:07 smithi194 ceph-mon[33017]: pgmap v8802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:07 smithi033 ceph-mon[27845]: pgmap v8802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:07 smithi096 ceph-mon[32753]: pgmap v8802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15546.930827 secs 2024-04-26T16:10:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:08 smithi194 ceph-mon[33017]: pgmap v8803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15549.744677 secs 2024-04-26T16:10:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15546.930827 secs 2024-04-26T16:10:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:08 smithi033 ceph-mon[27845]: pgmap v8803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15549.744677 secs 2024-04-26T16:10:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15546.930827 secs 2024-04-26T16:10:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:08 smithi096 ceph-mon[32753]: pgmap v8803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15549.744677 secs 2024-04-26T16:10:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:11 smithi194 ceph-mon[33017]: pgmap v8804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:11 smithi033 ceph-mon[27845]: pgmap v8804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:11 smithi096 ceph-mon[32753]: pgmap v8804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:13 smithi194 ceph-mon[33017]: pgmap v8805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15551.930935 secs 2024-04-26T16:10:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:13 smithi033 ceph-mon[27845]: pgmap v8805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15551.930935 secs 2024-04-26T16:10:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:13 smithi096 ceph-mon[32753]: pgmap v8805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15551.930935 secs 2024-04-26T16:10:14.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:14.178+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:14.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:14.178+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:14.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:14.178+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:10:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15554.744790 secs 2024-04-26T16:10:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15554.744790 secs 2024-04-26T16:10:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15554.744790 secs 2024-04-26T16:10:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:15 smithi194 ceph-mon[33017]: pgmap v8806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:15 smithi033 ceph-mon[27845]: pgmap v8806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:15 smithi096 ceph-mon[32753]: pgmap v8806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:17 smithi194 ceph-mon[33017]: pgmap v8807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:17 smithi033 ceph-mon[27845]: pgmap v8807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:17 smithi096 ceph-mon[32753]: pgmap v8807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15556.931003 secs 2024-04-26T16:10:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15556.931003 secs 2024-04-26T16:10:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15556.931003 secs 2024-04-26T16:10:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:19 smithi194 ceph-mon[33017]: pgmap v8808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15559.744919 secs 2024-04-26T16:10:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:19 smithi033 ceph-mon[27845]: pgmap v8808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15559.744919 secs 2024-04-26T16:10:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:19 smithi096 ceph-mon[32753]: pgmap v8808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15559.744919 secs 2024-04-26T16:10:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:21 smithi194 ceph-mon[33017]: pgmap v8809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:21 smithi033 ceph-mon[27845]: pgmap v8809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:21 smithi096 ceph-mon[32753]: pgmap v8809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:23 smithi194 ceph-mon[33017]: pgmap v8810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15561.931090 secs 2024-04-26T16:10:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:23 smithi033 ceph-mon[27845]: pgmap v8810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15561.931090 secs 2024-04-26T16:10:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:23 smithi096 ceph-mon[32753]: pgmap v8810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15561.931090 secs 2024-04-26T16:10:24.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15564.745033 secs 2024-04-26T16:10:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15564.745033 secs 2024-04-26T16:10:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15564.745033 secs 2024-04-26T16:10:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:25 smithi194 ceph-mon[33017]: pgmap v8811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:25 smithi033 ceph-mon[27845]: pgmap v8811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:25 smithi096 ceph-mon[32753]: pgmap v8811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:27 smithi194 ceph-mon[33017]: pgmap v8812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:27 smithi033 ceph-mon[27845]: pgmap v8812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:27 smithi096 ceph-mon[32753]: pgmap v8812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15566.931152 secs 2024-04-26T16:10:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15566.931152 secs 2024-04-26T16:10:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15566.931152 secs 2024-04-26T16:10:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:29.188+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:29.188+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:29.188+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:10:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:29 smithi194 ceph-mon[33017]: pgmap v8813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15569.745148 secs 2024-04-26T16:10:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:29 smithi033 ceph-mon[27845]: pgmap v8813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15569.745148 secs 2024-04-26T16:10:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:29 smithi096 ceph-mon[32753]: pgmap v8813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15569.745148 secs 2024-04-26T16:10:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:31 smithi033 ceph-mon[27845]: pgmap v8814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:31 smithi194 ceph-mon[33017]: pgmap v8814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:31 smithi096 ceph-mon[32753]: pgmap v8814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:33 smithi194 ceph-mon[33017]: pgmap v8815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15571.931241 secs 2024-04-26T16:10:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:33 smithi033 ceph-mon[27845]: pgmap v8815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15571.931241 secs 2024-04-26T16:10:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:33 smithi096 ceph-mon[32753]: pgmap v8815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15571.931241 secs 2024-04-26T16:10:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15574.745237 secs 2024-04-26T16:10:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:10:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15574.745237 secs 2024-04-26T16:10:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:10:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15574.745237 secs 2024-04-26T16:10:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:10:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:35 smithi194 ceph-mon[33017]: pgmap v8816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:35 smithi194 ceph-mon[33017]: pgmap v8817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:35 smithi033 ceph-mon[27845]: pgmap v8816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:35 smithi033 ceph-mon[27845]: pgmap v8817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:35 smithi096 ceph-mon[32753]: pgmap v8816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:35 smithi096 ceph-mon[32753]: pgmap v8817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:37 smithi194 ceph-mon[33017]: pgmap v8818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:37 smithi033 ceph-mon[27845]: pgmap v8818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:37 smithi096 ceph-mon[32753]: pgmap v8818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15576.931302 secs 2024-04-26T16:10:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15576.931302 secs 2024-04-26T16:10:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15576.931302 secs 2024-04-26T16:10:39.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:39 smithi096 ceph-mon[32753]: pgmap v8819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:39.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15579.745307 secs 2024-04-26T16:10:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:39 smithi194 ceph-mon[33017]: pgmap v8819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15579.745307 secs 2024-04-26T16:10:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:39 smithi033 ceph-mon[27845]: pgmap v8819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15579.745307 secs 2024-04-26T16:10:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:41 smithi194 ceph-mon[33017]: pgmap v8820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:41 smithi033 ceph-mon[27845]: pgmap v8820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:41 smithi096 ceph-mon[32753]: pgmap v8820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:43 smithi194 ceph-mon[33017]: pgmap v8821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15581.931379 secs 2024-04-26T16:10:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:43 smithi033 ceph-mon[27845]: pgmap v8821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15581.931379 secs 2024-04-26T16:10:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:43 smithi096 ceph-mon[32753]: pgmap v8821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:10:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15581.931379 secs 2024-04-26T16:10:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:44.203+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:44.203+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:44.203+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:10:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15584.745424 secs 2024-04-26T16:10:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15584.745424 secs 2024-04-26T16:10:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15584.745424 secs 2024-04-26T16:10:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:45 smithi194 ceph-mon[33017]: pgmap v8822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:10:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:45 smithi033 ceph-mon[27845]: pgmap v8822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:10:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:45 smithi096 ceph-mon[32753]: pgmap v8822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:10:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:47 smithi194 ceph-mon[33017]: pgmap v8823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:47 smithi033 ceph-mon[27845]: pgmap v8823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:47 smithi096 ceph-mon[32753]: pgmap v8823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15586.931440 secs 2024-04-26T16:10:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15586.931440 secs 2024-04-26T16:10:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15586.931440 secs 2024-04-26T16:10:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:49 smithi033 ceph-mon[27845]: pgmap v8824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15589.745543 secs 2024-04-26T16:10:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:49 smithi096 ceph-mon[32753]: pgmap v8824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15589.745543 secs 2024-04-26T16:10:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:49 smithi194 ceph-mon[33017]: pgmap v8824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15589.745543 secs 2024-04-26T16:10:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:50 smithi033 ceph-mon[27845]: pgmap v8825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:50 smithi096 ceph-mon[32753]: pgmap v8825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:50 smithi194 ceph-mon[33017]: pgmap v8825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:52 smithi194 ceph-mon[33017]: pgmap v8826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:53.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:52 smithi033 ceph-mon[27845]: pgmap v8826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:52 smithi096 ceph-mon[32753]: pgmap v8826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:54.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15591.931523 secs 2024-04-26T16:10:54.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15594.745692 secs 2024-04-26T16:10:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15591.931523 secs 2024-04-26T16:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15594.745692 secs 2024-04-26T16:10:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15591.931523 secs 2024-04-26T16:10:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15594.745692 secs 2024-04-26T16:10:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:54 smithi194 ceph-mon[33017]: pgmap v8827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:54 smithi033 ceph-mon[27845]: pgmap v8827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:54 smithi096 ceph-mon[32753]: pgmap v8827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:56 smithi194 ceph-mon[33017]: pgmap v8828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:56 smithi033 ceph-mon[27845]: pgmap v8828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:56 smithi096 ceph-mon[32753]: pgmap v8828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15596.931607 secs 2024-04-26T16:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:58 smithi194 ceph-mon[33017]: pgmap v8829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:10:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15599.745793 secs 2024-04-26T16:10:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15596.931607 secs 2024-04-26T16:10:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:58 smithi033 ceph-mon[27845]: pgmap v8829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:10:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15599.745793 secs 2024-04-26T16:10:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15596.931607 secs 2024-04-26T16:10:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:58 smithi096 ceph-mon[32753]: pgmap v8829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:10:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:10:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15599.745793 secs 2024-04-26T16:10:59.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:59.207+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:59.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:59.207+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:10:59.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:10:59.207+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:11:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:00 smithi194 ceph-mon[33017]: pgmap v8830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:01.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:00 smithi033 ceph-mon[27845]: pgmap v8830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:00 smithi096 ceph-mon[32753]: pgmap v8830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:02 smithi194 ceph-mon[33017]: pgmap v8831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:02 smithi033 ceph-mon[27845]: pgmap v8831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:02 smithi096 ceph-mon[32753]: pgmap v8831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15601.931706 secs 2024-04-26T16:11:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15604.745914 secs 2024-04-26T16:11:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15601.931706 secs 2024-04-26T16:11:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15604.745914 secs 2024-04-26T16:11:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15601.931706 secs 2024-04-26T16:11:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15604.745914 secs 2024-04-26T16:11:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:04 smithi194 ceph-mon[33017]: pgmap v8832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:04 smithi033 ceph-mon[27845]: pgmap v8832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:04 smithi096 ceph-mon[32753]: pgmap v8832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:06 smithi194 ceph-mon[33017]: pgmap v8833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:06 smithi033 ceph-mon[27845]: pgmap v8833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:06 smithi096 ceph-mon[32753]: pgmap v8833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15606.931766 secs 2024-04-26T16:11:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:08 smithi194 ceph-mon[33017]: pgmap v8834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15609.746033 secs 2024-04-26T16:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15606.931766 secs 2024-04-26T16:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:08 smithi033 ceph-mon[27845]: pgmap v8834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15609.746033 secs 2024-04-26T16:11:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15606.931766 secs 2024-04-26T16:11:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:08 smithi096 ceph-mon[32753]: pgmap v8834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15609.746033 secs 2024-04-26T16:11:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:10 smithi194 ceph-mon[33017]: pgmap v8835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:10 smithi033 ceph-mon[27845]: pgmap v8835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:10 smithi096 ceph-mon[32753]: pgmap v8835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:12 smithi194 ceph-mon[33017]: pgmap v8836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:12 smithi033 ceph-mon[27845]: pgmap v8836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:12 smithi096 ceph-mon[32753]: pgmap v8836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15611.931849 secs 2024-04-26T16:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15614.746172 secs 2024-04-26T16:11:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15611.931849 secs 2024-04-26T16:11:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15614.746172 secs 2024-04-26T16:11:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:14.220+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:14.221+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:14.221+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:11:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15611.931849 secs 2024-04-26T16:11:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15614.746172 secs 2024-04-26T16:11:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:14 smithi033 ceph-mon[27845]: pgmap v8837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:14 smithi096 ceph-mon[32753]: pgmap v8837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:14 smithi194 ceph-mon[33017]: pgmap v8837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:16 smithi033 ceph-mon[27845]: pgmap v8838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:16 smithi096 ceph-mon[32753]: pgmap v8838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:16 smithi194 ceph-mon[33017]: pgmap v8838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15616.931933 secs 2024-04-26T16:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:18 smithi033 ceph-mon[27845]: pgmap v8839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15619.746294 secs 2024-04-26T16:11:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15616.931933 secs 2024-04-26T16:11:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:18 smithi096 ceph-mon[32753]: pgmap v8839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15619.746294 secs 2024-04-26T16:11:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15616.931933 secs 2024-04-26T16:11:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:18 smithi194 ceph-mon[33017]: pgmap v8839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15619.746294 secs 2024-04-26T16:11:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:20 smithi033 ceph-mon[27845]: pgmap v8840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:20 smithi096 ceph-mon[32753]: pgmap v8840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:20 smithi194 ceph-mon[33017]: pgmap v8840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:21.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T16:11:21.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T16:11:21.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0062040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T16:11:21.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0062040 comp rx=0 tx=0).stop 2024-04-26T16:11:21.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T16:11:21.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:11:21.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:21.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.755+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:21.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.756+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:21.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:21.757+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0022040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:11:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:22 smithi033 ceph-mon[27845]: pgmap v8841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:22 smithi096 ceph-mon[32753]: pgmap v8841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:22 smithi194 ceph-mon[33017]: pgmap v8841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15621.932021 secs 2024-04-26T16:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15624.746392 secs 2024-04-26T16:11:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15621.932021 secs 2024-04-26T16:11:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15624.746392 secs 2024-04-26T16:11:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15621.932021 secs 2024-04-26T16:11:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15624.746392 secs 2024-04-26T16:11:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:24.453+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T16:11:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:24.453+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T16:11:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:24.454+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T16:11:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:24.454+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:11:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:24 smithi033 ceph-mon[27845]: pgmap v8842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:24 smithi096 ceph-mon[32753]: pgmap v8842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:24 smithi194 ceph-mon[33017]: pgmap v8842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:26 smithi033 ceph-mon[27845]: pgmap v8843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:26 smithi096 ceph-mon[32753]: pgmap v8843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:26 smithi194 ceph-mon[33017]: pgmap v8843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15626.932104 secs 2024-04-26T16:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:28 smithi033 ceph-mon[27845]: pgmap v8844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15629.746486 secs 2024-04-26T16:11:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15626.932104 secs 2024-04-26T16:11:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:28 smithi096 ceph-mon[32753]: pgmap v8844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15629.746486 secs 2024-04-26T16:11:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:29.228+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:29.228+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:11:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:29.228+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15626.932104 secs 2024-04-26T16:11:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:28 smithi194 ceph-mon[33017]: pgmap v8844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15629.746486 secs 2024-04-26T16:11:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:30 smithi033 ceph-mon[27845]: pgmap v8845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:30 smithi096 ceph-mon[32753]: pgmap v8845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:30 smithi194 ceph-mon[33017]: pgmap v8845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:32 smithi033 ceph-mon[27845]: pgmap v8846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:32 smithi096 ceph-mon[32753]: pgmap v8846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:32 smithi194 ceph-mon[33017]: pgmap v8846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15631.932190 secs 2024-04-26T16:11:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15634.746605 secs 2024-04-26T16:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15631.932190 secs 2024-04-26T16:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15634.746605 secs 2024-04-26T16:11:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15631.932190 secs 2024-04-26T16:11:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15634.746605 secs 2024-04-26T16:11:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:34 smithi194 ceph-mon[33017]: pgmap v8847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:34 smithi096 ceph-mon[32753]: pgmap v8847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:35.397 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:35.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.415 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:34 smithi033 ceph-mon[27845]: pgmap v8847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:35.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:35.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:35.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:36 smithi194 ceph-mon[33017]: pgmap v8848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:36 smithi033 ceph-mon[27845]: pgmap v8848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:36 smithi096 ceph-mon[32753]: pgmap v8848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:11:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:11:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:11:38.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:37 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:11:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:11:38.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:37 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:11:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:11:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:37 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15636.932253 secs 2024-04-26T16:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:38 smithi194 ceph-mon[33017]: pgmap v8849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15639.746746 secs 2024-04-26T16:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15636.932253 secs 2024-04-26T16:11:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:38 smithi033 ceph-mon[27845]: pgmap v8849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15639.746746 secs 2024-04-26T16:11:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15636.932253 secs 2024-04-26T16:11:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:38 smithi096 ceph-mon[32753]: pgmap v8849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15639.746746 secs 2024-04-26T16:11:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:41.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: pgmap v8850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:41.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:11:41.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:11:41.236 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:41.236 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:41.236 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: pgmap v8851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:11:41.236 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: pgmap v8852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:41.236 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: pgmap v8850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:11:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:11:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: pgmap v8851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:11:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: pgmap v8852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: pgmap v8850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: pgmap v8851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: pgmap v8852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:11:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:43 smithi033 ceph-mon[27845]: pgmap v8853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15641.932338 secs 2024-04-26T16:11:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:43 smithi096 ceph-mon[32753]: pgmap v8853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15641.932338 secs 2024-04-26T16:11:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:43 smithi194 ceph-mon[33017]: pgmap v8853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15641.932338 secs 2024-04-26T16:11:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:44.232+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:44.232+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:11:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:44.232+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15644.746889 secs 2024-04-26T16:11:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15644.746889 secs 2024-04-26T16:11:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15644.746889 secs 2024-04-26T16:11:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:45 smithi194 ceph-mon[33017]: pgmap v8854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:11:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:45 smithi033 ceph-mon[27845]: pgmap v8854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:11:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:45 smithi096 ceph-mon[32753]: pgmap v8854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:11:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:47 smithi194 ceph-mon[33017]: pgmap v8855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:47 smithi033 ceph-mon[27845]: pgmap v8855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:47 smithi096 ceph-mon[32753]: pgmap v8855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:11:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15646.932399 secs 2024-04-26T16:11:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15646.932399 secs 2024-04-26T16:11:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15646.932399 secs 2024-04-26T16:11:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:49 smithi194 ceph-mon[33017]: pgmap v8856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:11:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15649.746957 secs 2024-04-26T16:11:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:49 smithi033 ceph-mon[27845]: pgmap v8856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:11:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15649.746957 secs 2024-04-26T16:11:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:49 smithi096 ceph-mon[32753]: pgmap v8856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:11:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15649.746957 secs 2024-04-26T16:11:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:51 smithi194 ceph-mon[33017]: pgmap v8857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:11:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:51 smithi033 ceph-mon[27845]: pgmap v8857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:11:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:51 smithi096 ceph-mon[32753]: pgmap v8857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:11:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:53 smithi194 ceph-mon[33017]: pgmap v8858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15651.932484 secs 2024-04-26T16:11:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:53 smithi033 ceph-mon[27845]: pgmap v8858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15651.932484 secs 2024-04-26T16:11:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:53 smithi096 ceph-mon[32753]: pgmap v8858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15651.932484 secs 2024-04-26T16:11:54.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15654.747058 secs 2024-04-26T16:11:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15654.747058 secs 2024-04-26T16:11:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15654.747058 secs 2024-04-26T16:11:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:55 smithi194 ceph-mon[33017]: pgmap v8859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:55 smithi033 ceph-mon[27845]: pgmap v8859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:55 smithi096 ceph-mon[32753]: pgmap v8859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:57 smithi194 ceph-mon[33017]: pgmap v8860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:57 smithi033 ceph-mon[27845]: pgmap v8860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:57 smithi096 ceph-mon[32753]: pgmap v8860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:11:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15656.932546 secs 2024-04-26T16:11:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15656.932546 secs 2024-04-26T16:11:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15656.932546 secs 2024-04-26T16:11:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:59.247+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:59.247+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:11:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:11:59.247+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:11:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:59 smithi194 ceph-mon[33017]: pgmap v8861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15659.747181 secs 2024-04-26T16:11:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:59 smithi033 ceph-mon[27845]: pgmap v8861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15659.747181 secs 2024-04-26T16:11:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:59 smithi096 ceph-mon[32753]: pgmap v8861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:11:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15659.747181 secs 2024-04-26T16:12:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:01 smithi194 ceph-mon[33017]: pgmap v8862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:01 smithi033 ceph-mon[27845]: pgmap v8862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:01 smithi096 ceph-mon[32753]: pgmap v8862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:03 smithi194 ceph-mon[33017]: pgmap v8863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15661.932632 secs 2024-04-26T16:12:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:03 smithi033 ceph-mon[27845]: pgmap v8863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15661.932632 secs 2024-04-26T16:12:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:03 smithi096 ceph-mon[32753]: pgmap v8863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15661.932632 secs 2024-04-26T16:12:04.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15664.747268 secs 2024-04-26T16:12:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15664.747268 secs 2024-04-26T16:12:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15664.747268 secs 2024-04-26T16:12:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:05 smithi194 ceph-mon[33017]: pgmap v8864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:05 smithi033 ceph-mon[27845]: pgmap v8864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:05 smithi096 ceph-mon[32753]: pgmap v8864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:07 smithi194 ceph-mon[33017]: pgmap v8865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:07 smithi033 ceph-mon[27845]: pgmap v8865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:07 smithi096 ceph-mon[32753]: pgmap v8865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15666.932715 secs 2024-04-26T16:12:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15666.932715 secs 2024-04-26T16:12:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15666.932715 secs 2024-04-26T16:12:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:09 smithi194 ceph-mon[33017]: pgmap v8866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15669.747389 secs 2024-04-26T16:12:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:09 smithi033 ceph-mon[27845]: pgmap v8866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15669.747389 secs 2024-04-26T16:12:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:09 smithi096 ceph-mon[32753]: pgmap v8866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15669.747389 secs 2024-04-26T16:12:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:11 smithi194 ceph-mon[33017]: pgmap v8867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:11 smithi033 ceph-mon[27845]: pgmap v8867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:11 smithi096 ceph-mon[32753]: pgmap v8867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:13 smithi194 ceph-mon[33017]: pgmap v8868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15671.932782 secs 2024-04-26T16:12:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:13 smithi033 ceph-mon[27845]: pgmap v8868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15671.932782 secs 2024-04-26T16:12:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:13 smithi096 ceph-mon[32753]: pgmap v8868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15671.932782 secs 2024-04-26T16:12:14.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:14.263+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:14.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:14.263+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:14.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:14.263+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:12:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15674.747454 secs 2024-04-26T16:12:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15674.747454 secs 2024-04-26T16:12:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15674.747454 secs 2024-04-26T16:12:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:15 smithi194 ceph-mon[33017]: pgmap v8869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:15 smithi033 ceph-mon[27845]: pgmap v8869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:15 smithi096 ceph-mon[32753]: pgmap v8869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:17 smithi194 ceph-mon[33017]: pgmap v8870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:17 smithi033 ceph-mon[27845]: pgmap v8870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:17 smithi096 ceph-mon[32753]: pgmap v8870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15676.932864 secs 2024-04-26T16:12:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15676.932864 secs 2024-04-26T16:12:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15676.932864 secs 2024-04-26T16:12:19.361 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:19 smithi033 ceph-mon[27845]: pgmap v8871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:19 smithi194 ceph-mon[33017]: pgmap v8871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15679.747552 secs 2024-04-26T16:12:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:19 smithi096 ceph-mon[32753]: pgmap v8871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15679.747552 secs 2024-04-26T16:12:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15679.747552 secs 2024-04-26T16:12:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:21 smithi194 ceph-mon[33017]: pgmap v8872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:21 smithi033 ceph-mon[27845]: pgmap v8872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:21 smithi096 ceph-mon[32753]: pgmap v8872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:23 smithi194 ceph-mon[33017]: pgmap v8873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15681.932948 secs 2024-04-26T16:12:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:23 smithi033 ceph-mon[27845]: pgmap v8873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15681.932948 secs 2024-04-26T16:12:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:23 smithi096 ceph-mon[32753]: pgmap v8873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15681.932948 secs 2024-04-26T16:12:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15684.747632 secs 2024-04-26T16:12:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15684.747632 secs 2024-04-26T16:12:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15684.747632 secs 2024-04-26T16:12:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:25 smithi033 ceph-mon[27845]: pgmap v8874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:25 smithi096 ceph-mon[32753]: pgmap v8874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:25 smithi194 ceph-mon[33017]: pgmap v8874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:27 smithi033 ceph-mon[27845]: pgmap v8875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:27 smithi096 ceph-mon[32753]: pgmap v8875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:27 smithi194 ceph-mon[33017]: pgmap v8875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15686.933010 secs 2024-04-26T16:12:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15686.933010 secs 2024-04-26T16:12:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15686.933010 secs 2024-04-26T16:12:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:29.279+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:29.279+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:29.279+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:12:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:29 smithi033 ceph-mon[27845]: pgmap v8876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15689.747756 secs 2024-04-26T16:12:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:29 smithi096 ceph-mon[32753]: pgmap v8876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15689.747756 secs 2024-04-26T16:12:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:29 smithi194 ceph-mon[33017]: pgmap v8876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15689.747756 secs 2024-04-26T16:12:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:31 smithi033 ceph-mon[27845]: pgmap v8877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:31 smithi096 ceph-mon[32753]: pgmap v8877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:31 smithi194 ceph-mon[33017]: pgmap v8877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:32 smithi033 ceph-mon[27845]: pgmap v8878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:32 smithi096 ceph-mon[32753]: pgmap v8878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:32 smithi194 ceph-mon[33017]: pgmap v8878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15691.933073 secs 2024-04-26T16:12:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15691.933073 secs 2024-04-26T16:12:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15691.933073 secs 2024-04-26T16:12:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15694.747871 secs 2024-04-26T16:12:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:34 smithi033 ceph-mon[27845]: pgmap v8879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15694.747871 secs 2024-04-26T16:12:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:34 smithi096 ceph-mon[32753]: pgmap v8879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15694.747871 secs 2024-04-26T16:12:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:34 smithi194 ceph-mon[33017]: pgmap v8879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:37 smithi194 ceph-mon[33017]: pgmap v8880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:37 smithi033 ceph-mon[27845]: pgmap v8880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:37 smithi096 ceph-mon[32753]: pgmap v8880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15696.933132 secs 2024-04-26T16:12:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15696.933132 secs 2024-04-26T16:12:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15696.933132 secs 2024-04-26T16:12:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:39 smithi194 ceph-mon[33017]: pgmap v8881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15699.747971 secs 2024-04-26T16:12:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:39 smithi033 ceph-mon[27845]: pgmap v8881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15699.747971 secs 2024-04-26T16:12:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:39 smithi096 ceph-mon[32753]: pgmap v8881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15699.747971 secs 2024-04-26T16:12:41.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:41 smithi096 ceph-mon[32753]: pgmap v8882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:41.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:41 smithi194 ceph-mon[33017]: pgmap v8882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:41 smithi033 ceph-mon[27845]: pgmap v8882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:12:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:12:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:12:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:43 smithi194 ceph-mon[33017]: pgmap v8883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:43 smithi194 ceph-mon[33017]: pgmap v8884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15701.933195 secs 2024-04-26T16:12:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:43 smithi033 ceph-mon[27845]: pgmap v8883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:43 smithi033 ceph-mon[27845]: pgmap v8884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15701.933195 secs 2024-04-26T16:12:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:43 smithi096 ceph-mon[32753]: pgmap v8883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:43 smithi096 ceph-mon[32753]: pgmap v8884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15701.933195 secs 2024-04-26T16:12:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:44.290+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:44.290+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:12:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:44.290+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15704.748090 secs 2024-04-26T16:12:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15704.748090 secs 2024-04-26T16:12:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15704.748090 secs 2024-04-26T16:12:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:45 smithi194 ceph-mon[33017]: pgmap v8885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:45 smithi033 ceph-mon[27845]: pgmap v8885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:45 smithi096 ceph-mon[32753]: pgmap v8885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:47 smithi194 ceph-mon[33017]: pgmap v8886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:47 smithi033 ceph-mon[27845]: pgmap v8886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:47 smithi096 ceph-mon[32753]: pgmap v8886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15706.933272 secs 2024-04-26T16:12:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15706.933272 secs 2024-04-26T16:12:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15706.933272 secs 2024-04-26T16:12:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:49 smithi194 ceph-mon[33017]: pgmap v8887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15709.748241 secs 2024-04-26T16:12:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:49 smithi033 ceph-mon[27845]: pgmap v8887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15709.748241 secs 2024-04-26T16:12:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:49 smithi096 ceph-mon[32753]: pgmap v8887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15709.748241 secs 2024-04-26T16:12:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:51 smithi194 ceph-mon[33017]: pgmap v8888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:51 smithi033 ceph-mon[27845]: pgmap v8888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:51 smithi096 ceph-mon[32753]: pgmap v8888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:12:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:53 smithi194 ceph-mon[33017]: pgmap v8889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:12:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15711.933355 secs 2024-04-26T16:12:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:53 smithi033 ceph-mon[27845]: pgmap v8889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:12:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15711.933355 secs 2024-04-26T16:12:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:53 smithi096 ceph-mon[32753]: pgmap v8889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:12:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15711.933355 secs 2024-04-26T16:12:54.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15714.748356 secs 2024-04-26T16:12:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15714.748356 secs 2024-04-26T16:12:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15714.748356 secs 2024-04-26T16:12:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:55 smithi194 ceph-mon[33017]: pgmap v8890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:55 smithi033 ceph-mon[27845]: pgmap v8890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:55 smithi096 ceph-mon[32753]: pgmap v8890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:57 smithi194 ceph-mon[33017]: pgmap v8891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:57 smithi033 ceph-mon[27845]: pgmap v8891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:57 smithi096 ceph-mon[32753]: pgmap v8891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:12:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15716.933415 secs 2024-04-26T16:12:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15716.933415 secs 2024-04-26T16:12:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15716.933415 secs 2024-04-26T16:12:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:59.305+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:59.305+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:12:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:12:59.305+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:12:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:59 smithi194 ceph-mon[33017]: pgmap v8892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:12:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15719.748480 secs 2024-04-26T16:12:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:59 smithi033 ceph-mon[27845]: pgmap v8892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:12:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15719.748480 secs 2024-04-26T16:12:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:59 smithi096 ceph-mon[32753]: pgmap v8892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:12:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:12:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15719.748480 secs 2024-04-26T16:13:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:01 smithi194 ceph-mon[33017]: pgmap v8893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:01 smithi033 ceph-mon[27845]: pgmap v8893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:01 smithi096 ceph-mon[32753]: pgmap v8893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:03 smithi194 ceph-mon[33017]: pgmap v8894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15721.933498 secs 2024-04-26T16:13:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:03 smithi033 ceph-mon[27845]: pgmap v8894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15721.933498 secs 2024-04-26T16:13:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:03 smithi096 ceph-mon[32753]: pgmap v8894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15721.933498 secs 2024-04-26T16:13:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15724.748548 secs 2024-04-26T16:13:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15724.748548 secs 2024-04-26T16:13:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15724.748548 secs 2024-04-26T16:13:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:05 smithi194 ceph-mon[33017]: pgmap v8895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:05 smithi033 ceph-mon[27845]: pgmap v8895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:05 smithi096 ceph-mon[32753]: pgmap v8895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:07 smithi194 ceph-mon[33017]: pgmap v8896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:07 smithi033 ceph-mon[27845]: pgmap v8896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:07 smithi096 ceph-mon[32753]: pgmap v8896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15726.933582 secs 2024-04-26T16:13:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15726.933582 secs 2024-04-26T16:13:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15726.933582 secs 2024-04-26T16:13:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:09 smithi194 ceph-mon[33017]: pgmap v8897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15729.748665 secs 2024-04-26T16:13:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:09 smithi033 ceph-mon[27845]: pgmap v8897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15729.748665 secs 2024-04-26T16:13:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:09 smithi096 ceph-mon[32753]: pgmap v8897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15729.748665 secs 2024-04-26T16:13:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:11 smithi033 ceph-mon[27845]: pgmap v8898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:11 smithi096 ceph-mon[32753]: pgmap v8898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:11 smithi194 ceph-mon[33017]: pgmap v8898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:13 smithi033 ceph-mon[27845]: pgmap v8899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15731.933671 secs 2024-04-26T16:13:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:13 smithi096 ceph-mon[32753]: pgmap v8899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15731.933671 secs 2024-04-26T16:13:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:13 smithi194 ceph-mon[33017]: pgmap v8899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15731.933671 secs 2024-04-26T16:13:14.320 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:14.321+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:14.321+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:13:14.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:14.321+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15734.748779 secs 2024-04-26T16:13:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15734.748779 secs 2024-04-26T16:13:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15734.748779 secs 2024-04-26T16:13:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:15 smithi033 ceph-mon[27845]: pgmap v8900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:15 smithi096 ceph-mon[32753]: pgmap v8900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:15 smithi194 ceph-mon[33017]: pgmap v8900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:17 smithi033 ceph-mon[27845]: pgmap v8901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:17 smithi096 ceph-mon[32753]: pgmap v8901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:17 smithi194 ceph-mon[33017]: pgmap v8901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15736.933754 secs 2024-04-26T16:13:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:18 smithi194 ceph-mon[33017]: pgmap v8902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15739.748898 secs 2024-04-26T16:13:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15736.933754 secs 2024-04-26T16:13:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:18 smithi033 ceph-mon[27845]: pgmap v8902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15739.748898 secs 2024-04-26T16:13:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15736.933754 secs 2024-04-26T16:13:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:18 smithi096 ceph-mon[32753]: pgmap v8902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15739.748898 secs 2024-04-26T16:13:21.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:20 smithi033 ceph-mon[27845]: pgmap v8903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:20 smithi096 ceph-mon[32753]: pgmap v8903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:20 smithi194 ceph-mon[33017]: pgmap v8903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:22 smithi033 ceph-mon[27845]: pgmap v8904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:22 smithi096 ceph-mon[32753]: pgmap v8904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:22 smithi194 ceph-mon[33017]: pgmap v8904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15741.933835 secs 2024-04-26T16:13:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15744.749013 secs 2024-04-26T16:13:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15741.933835 secs 2024-04-26T16:13:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15744.749013 secs 2024-04-26T16:13:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15741.933835 secs 2024-04-26T16:13:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15744.749013 secs 2024-04-26T16:13:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:24 smithi194 ceph-mon[33017]: pgmap v8905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:24 smithi033 ceph-mon[27845]: pgmap v8905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:24 smithi096 ceph-mon[32753]: pgmap v8905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:26 smithi194 ceph-mon[33017]: pgmap v8906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:26 smithi033 ceph-mon[27845]: pgmap v8906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:26 smithi096 ceph-mon[32753]: pgmap v8906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:29.333+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:29.333+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:29.333+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:13:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15746.933933 secs 2024-04-26T16:13:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:28 smithi194 ceph-mon[33017]: pgmap v8907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15749.749140 secs 2024-04-26T16:13:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15746.933933 secs 2024-04-26T16:13:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:28 smithi033 ceph-mon[27845]: pgmap v8907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15749.749140 secs 2024-04-26T16:13:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15746.933933 secs 2024-04-26T16:13:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:28 smithi096 ceph-mon[32753]: pgmap v8907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15749.749140 secs 2024-04-26T16:13:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:30 smithi194 ceph-mon[33017]: pgmap v8908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:30 smithi033 ceph-mon[27845]: pgmap v8908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:30 smithi096 ceph-mon[32753]: pgmap v8908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:32 smithi194 ceph-mon[33017]: pgmap v8909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:32 smithi033 ceph-mon[27845]: pgmap v8909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:32 smithi096 ceph-mon[32753]: pgmap v8909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15751.934005 secs 2024-04-26T16:13:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15754.749196 secs 2024-04-26T16:13:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15751.934005 secs 2024-04-26T16:13:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15754.749196 secs 2024-04-26T16:13:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15751.934005 secs 2024-04-26T16:13:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15754.749196 secs 2024-04-26T16:13:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:34 smithi194 ceph-mon[33017]: pgmap v8910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:34 smithi033 ceph-mon[27845]: pgmap v8910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:34 smithi096 ceph-mon[32753]: pgmap v8910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:36 smithi194 ceph-mon[33017]: pgmap v8911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:13:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:36 smithi033 ceph-mon[27845]: pgmap v8911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:13:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:36 smithi096 ceph-mon[32753]: pgmap v8911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:13:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15756.934085 secs 2024-04-26T16:13:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:38 smithi194 ceph-mon[33017]: pgmap v8912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15759.749317 secs 2024-04-26T16:13:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15756.934085 secs 2024-04-26T16:13:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:38 smithi033 ceph-mon[27845]: pgmap v8912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15759.749317 secs 2024-04-26T16:13:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15756.934085 secs 2024-04-26T16:13:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:38 smithi096 ceph-mon[32753]: pgmap v8912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15759.749317 secs 2024-04-26T16:13:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:39.475+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:13:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:39.475+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T16:13:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:39.476+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:13:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:39.476+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:13:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:40 smithi194 ceph-mon[33017]: pgmap v8913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:13:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:40 smithi033 ceph-mon[27845]: pgmap v8913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:13:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:40 smithi096 ceph-mon[32753]: pgmap v8913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:13:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:42 smithi194 ceph-mon[33017]: pgmap v8914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:42 smithi033 ceph-mon[27845]: pgmap v8914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:42 smithi096 ceph-mon[32753]: pgmap v8914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:44.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:44.349+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:44.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:44.349+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:44.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:44.349+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:13:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15761.934152 secs 2024-04-26T16:13:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15764.749441 secs 2024-04-26T16:13:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:13:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15761.934152 secs 2024-04-26T16:13:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15764.749441 secs 2024-04-26T16:13:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:13:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15761.934152 secs 2024-04-26T16:13:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15764.749441 secs 2024-04-26T16:13:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:13:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:45 smithi194 ceph-mon[33017]: pgmap v8915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:45 smithi194 ceph-mon[33017]: pgmap v8916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:13:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:45 smithi033 ceph-mon[27845]: pgmap v8915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:45 smithi033 ceph-mon[27845]: pgmap v8916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:13:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:45 smithi096 ceph-mon[32753]: pgmap v8915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:13:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:45 smithi096 ceph-mon[32753]: pgmap v8916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:13:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:47 smithi194 ceph-mon[33017]: pgmap v8917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:47 smithi033 ceph-mon[27845]: pgmap v8917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:47 smithi096 ceph-mon[32753]: pgmap v8917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:49 smithi194 ceph-mon[33017]: pgmap v8918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15766.934214 secs 2024-04-26T16:13:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15769.749562 secs 2024-04-26T16:13:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:49 smithi033 ceph-mon[27845]: pgmap v8918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15766.934214 secs 2024-04-26T16:13:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15769.749562 secs 2024-04-26T16:13:49.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:49 smithi096 ceph-mon[32753]: pgmap v8918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:49.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15766.934214 secs 2024-04-26T16:13:49.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15769.749562 secs 2024-04-26T16:13:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:51 smithi194 ceph-mon[33017]: pgmap v8919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:51 smithi033 ceph-mon[27845]: pgmap v8919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:51 smithi096 ceph-mon[32753]: pgmap v8919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:53.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:53 smithi033 ceph-mon[27845]: pgmap v8920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:53 smithi194 ceph-mon[33017]: pgmap v8920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:53 smithi096 ceph-mon[32753]: pgmap v8920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:13:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15771.934296 secs 2024-04-26T16:13:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15774.749680 secs 2024-04-26T16:13:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15771.934296 secs 2024-04-26T16:13:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15774.749680 secs 2024-04-26T16:13:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15771.934296 secs 2024-04-26T16:13:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15774.749680 secs 2024-04-26T16:13:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:55 smithi194 ceph-mon[33017]: pgmap v8921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:13:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:55 smithi033 ceph-mon[27845]: pgmap v8921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:13:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:55 smithi096 ceph-mon[32753]: pgmap v8921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:13:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:57 smithi194 ceph-mon[33017]: pgmap v8922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:57 smithi033 ceph-mon[27845]: pgmap v8922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:57 smithi096 ceph-mon[32753]: pgmap v8922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:13:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:59 smithi194 ceph-mon[33017]: pgmap v8923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15776.934379 secs 2024-04-26T16:13:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:13:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15779.749799 secs 2024-04-26T16:13:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:59.364+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:59.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:59.364+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:13:59.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:13:59.364+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:13:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:59 smithi033 ceph-mon[27845]: pgmap v8923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15776.934379 secs 2024-04-26T16:13:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:13:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15779.749799 secs 2024-04-26T16:13:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:59 smithi096 ceph-mon[32753]: pgmap v8923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:13:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15776.934379 secs 2024-04-26T16:13:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:13:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15779.749799 secs 2024-04-26T16:14:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:01 smithi194 ceph-mon[33017]: pgmap v8924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:01 smithi033 ceph-mon[27845]: pgmap v8924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:01 smithi096 ceph-mon[32753]: pgmap v8924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:03 smithi194 ceph-mon[33017]: pgmap v8925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:03 smithi033 ceph-mon[27845]: pgmap v8925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:03 smithi096 ceph-mon[32753]: pgmap v8925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15781.934464 secs 2024-04-26T16:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15784.749910 secs 2024-04-26T16:14:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15781.934464 secs 2024-04-26T16:14:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15784.749910 secs 2024-04-26T16:14:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15781.934464 secs 2024-04-26T16:14:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15784.749910 secs 2024-04-26T16:14:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:05 smithi194 ceph-mon[33017]: pgmap v8926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:05 smithi033 ceph-mon[27845]: pgmap v8926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:05 smithi096 ceph-mon[32753]: pgmap v8926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:07 smithi194 ceph-mon[33017]: pgmap v8927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:07 smithi033 ceph-mon[27845]: pgmap v8927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:07 smithi096 ceph-mon[32753]: pgmap v8927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:09 smithi194 ceph-mon[33017]: pgmap v8928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15786.934542 secs 2024-04-26T16:14:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15789.750028 secs 2024-04-26T16:14:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:09 smithi033 ceph-mon[27845]: pgmap v8928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15786.934542 secs 2024-04-26T16:14:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15789.750028 secs 2024-04-26T16:14:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:09 smithi096 ceph-mon[32753]: pgmap v8928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15786.934542 secs 2024-04-26T16:14:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15789.750028 secs 2024-04-26T16:14:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:11 smithi194 ceph-mon[33017]: pgmap v8929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:11 smithi033 ceph-mon[27845]: pgmap v8929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:11 smithi096 ceph-mon[32753]: pgmap v8929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:13 smithi194 ceph-mon[33017]: pgmap v8930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:13 smithi033 ceph-mon[27845]: pgmap v8930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:13 smithi096 ceph-mon[32753]: pgmap v8930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:14.374+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:14.374+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:14.374+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:14:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15791.934622 secs 2024-04-26T16:14:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15794.750086 secs 2024-04-26T16:14:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15791.934622 secs 2024-04-26T16:14:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15794.750086 secs 2024-04-26T16:14:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15791.934622 secs 2024-04-26T16:14:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15794.750086 secs 2024-04-26T16:14:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:15 smithi033 ceph-mon[27845]: pgmap v8931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:15 smithi096 ceph-mon[32753]: pgmap v8931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:15 smithi194 ceph-mon[33017]: pgmap v8931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:17 smithi033 ceph-mon[27845]: pgmap v8932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:17 smithi096 ceph-mon[32753]: pgmap v8932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:17 smithi194 ceph-mon[33017]: pgmap v8932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:19 smithi033 ceph-mon[27845]: pgmap v8933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15796.934707 secs 2024-04-26T16:14:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15799.750171 secs 2024-04-26T16:14:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:19 smithi096 ceph-mon[32753]: pgmap v8933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15796.934707 secs 2024-04-26T16:14:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15799.750171 secs 2024-04-26T16:14:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:19 smithi194 ceph-mon[33017]: pgmap v8933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15796.934707 secs 2024-04-26T16:14:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15799.750171 secs 2024-04-26T16:14:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:21 smithi033 ceph-mon[27845]: pgmap v8934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:21 smithi096 ceph-mon[32753]: pgmap v8934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:21 smithi194 ceph-mon[33017]: pgmap v8934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:23 smithi033 ceph-mon[27845]: pgmap v8935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:23 smithi096 ceph-mon[32753]: pgmap v8935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:23 smithi194 ceph-mon[33017]: pgmap v8935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15801.934790 secs 2024-04-26T16:14:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15804.750286 secs 2024-04-26T16:14:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15801.934790 secs 2024-04-26T16:14:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15804.750286 secs 2024-04-26T16:14:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:24.483+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:14:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:24.483+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T16:14:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:24.484+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T16:14:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:24.484+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:14:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15801.934790 secs 2024-04-26T16:14:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15804.750286 secs 2024-04-26T16:14:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:25 smithi033 ceph-mon[27845]: pgmap v8936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:25 smithi096 ceph-mon[32753]: pgmap v8936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:25 smithi194 ceph-mon[33017]: pgmap v8936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:27 smithi033 ceph-mon[27845]: pgmap v8937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:27 smithi096 ceph-mon[32753]: pgmap v8937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:27 smithi194 ceph-mon[33017]: pgmap v8937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:29.376+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:29.376+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:29.376+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:29 smithi033 ceph-mon[27845]: pgmap v8938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15806.934872 secs 2024-04-26T16:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15809.750407 secs 2024-04-26T16:14:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:29 smithi096 ceph-mon[32753]: pgmap v8938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15806.934872 secs 2024-04-26T16:14:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15809.750407 secs 2024-04-26T16:14:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:29 smithi194 ceph-mon[33017]: pgmap v8938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15806.934872 secs 2024-04-26T16:14:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15809.750407 secs 2024-04-26T16:14:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:31 smithi033 ceph-mon[27845]: pgmap v8939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:31 smithi096 ceph-mon[32753]: pgmap v8939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:31 smithi194 ceph-mon[33017]: pgmap v8939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:33 smithi033 ceph-mon[27845]: pgmap v8940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:33 smithi096 ceph-mon[32753]: pgmap v8940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:33 smithi194 ceph-mon[33017]: pgmap v8940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15811.934950 secs 2024-04-26T16:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15814.750521 secs 2024-04-26T16:14:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15811.934950 secs 2024-04-26T16:14:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15814.750521 secs 2024-04-26T16:14:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15811.934950 secs 2024-04-26T16:14:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15814.750521 secs 2024-04-26T16:14:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:35 smithi096 ceph-mon[32753]: pgmap v8941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:35 smithi194 ceph-mon[33017]: pgmap v8941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:35 smithi033 ceph-mon[27845]: pgmap v8941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:37 smithi096 ceph-mon[32753]: pgmap v8942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:37 smithi194 ceph-mon[33017]: pgmap v8942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:37 smithi033 ceph-mon[27845]: pgmap v8942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15816.935008 secs 2024-04-26T16:14:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15816.935008 secs 2024-04-26T16:14:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15816.935008 secs 2024-04-26T16:14:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:39 smithi194 ceph-mon[33017]: pgmap v8943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15819.750647 secs 2024-04-26T16:14:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:39 smithi033 ceph-mon[27845]: pgmap v8943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15819.750647 secs 2024-04-26T16:14:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:39 smithi096 ceph-mon[32753]: pgmap v8943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15819.750647 secs 2024-04-26T16:14:41.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:41 smithi096 ceph-mon[32753]: pgmap v8944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:41 smithi194 ceph-mon[33017]: pgmap v8944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:41 smithi033 ceph-mon[27845]: pgmap v8944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:14:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:43 smithi194 ceph-mon[33017]: pgmap v8945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15821.935068 secs 2024-04-26T16:14:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:43 smithi033 ceph-mon[27845]: pgmap v8945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15821.935068 secs 2024-04-26T16:14:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:43 smithi096 ceph-mon[32753]: pgmap v8945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15821.935068 secs 2024-04-26T16:14:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:44.381+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:44.381+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:44.381+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:14:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15824.750766 secs 2024-04-26T16:14:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15824.750766 secs 2024-04-26T16:14:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15824.750766 secs 2024-04-26T16:14:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:45 smithi194 ceph-mon[33017]: pgmap v8946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:45 smithi033 ceph-mon[27845]: pgmap v8946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:45 smithi096 ceph-mon[32753]: pgmap v8946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:46 smithi194 ceph-mon[33017]: pgmap v8947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:14:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:46 smithi194 ceph-mon[33017]: pgmap v8948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:14:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:46 smithi033 ceph-mon[27845]: pgmap v8947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:14:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:46 smithi033 ceph-mon[27845]: pgmap v8948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:14:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:46 smithi096 ceph-mon[32753]: pgmap v8947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:14:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:46 smithi096 ceph-mon[32753]: pgmap v8948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:14:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:48 smithi194 ceph-mon[33017]: pgmap v8949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15826.935122 secs 2024-04-26T16:14:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:48 smithi033 ceph-mon[27845]: pgmap v8949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15826.935122 secs 2024-04-26T16:14:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:48 smithi096 ceph-mon[32753]: pgmap v8949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15826.935122 secs 2024-04-26T16:14:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15829.750908 secs 2024-04-26T16:14:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15829.750908 secs 2024-04-26T16:14:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15829.750908 secs 2024-04-26T16:14:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:50 smithi033 ceph-mon[27845]: pgmap v8950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:14:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:50 smithi096 ceph-mon[32753]: pgmap v8950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:14:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:50 smithi194 ceph-mon[33017]: pgmap v8950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:14:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:52 smithi033 ceph-mon[27845]: pgmap v8951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:52 smithi096 ceph-mon[32753]: pgmap v8951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:52 smithi194 ceph-mon[33017]: pgmap v8951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15831.935184 secs 2024-04-26T16:14:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15831.935184 secs 2024-04-26T16:14:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15831.935184 secs 2024-04-26T16:14:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:54 smithi033 ceph-mon[27845]: pgmap v8952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15834.751050 secs 2024-04-26T16:14:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:54 smithi096 ceph-mon[32753]: pgmap v8952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15834.751050 secs 2024-04-26T16:14:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:54 smithi194 ceph-mon[33017]: pgmap v8952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:14:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15834.751050 secs 2024-04-26T16:14:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:56 smithi033 ceph-mon[27845]: pgmap v8953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:14:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:56 smithi096 ceph-mon[32753]: pgmap v8953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:14:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:56 smithi194 ceph-mon[33017]: pgmap v8953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:14:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:58 smithi033 ceph-mon[27845]: pgmap v8954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15836.935269 secs 2024-04-26T16:14:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:58 smithi096 ceph-mon[32753]: pgmap v8954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15836.935269 secs 2024-04-26T16:14:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:58 smithi194 ceph-mon[33017]: pgmap v8954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:14:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15836.935269 secs 2024-04-26T16:14:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:59.395+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:59.395+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:14:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:14:59.395+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:14:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15839.751175 secs 2024-04-26T16:14:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15839.751175 secs 2024-04-26T16:14:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15839.751175 secs 2024-04-26T16:15:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:00 smithi033 ceph-mon[27845]: pgmap v8955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:00 smithi096 ceph-mon[32753]: pgmap v8955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:00 smithi194 ceph-mon[33017]: pgmap v8955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:02 smithi033 ceph-mon[27845]: pgmap v8956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:02 smithi096 ceph-mon[32753]: pgmap v8956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:02 smithi194 ceph-mon[33017]: pgmap v8956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15841.935330 secs 2024-04-26T16:15:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15841.935330 secs 2024-04-26T16:15:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15841.935330 secs 2024-04-26T16:15:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:04 smithi033 ceph-mon[27845]: pgmap v8957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15844.751313 secs 2024-04-26T16:15:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:04 smithi096 ceph-mon[32753]: pgmap v8957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15844.751313 secs 2024-04-26T16:15:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:04 smithi194 ceph-mon[33017]: pgmap v8957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15844.751313 secs 2024-04-26T16:15:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:05 smithi096 ceph-mon[32753]: pgmap v8958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:05 smithi194 ceph-mon[33017]: pgmap v8958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:05 smithi033 ceph-mon[27845]: pgmap v8958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:08 smithi033 ceph-mon[27845]: pgmap v8959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15846.935414 secs 2024-04-26T16:15:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:08 smithi096 ceph-mon[32753]: pgmap v8959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15846.935414 secs 2024-04-26T16:15:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:08 smithi194 ceph-mon[33017]: pgmap v8959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15846.935414 secs 2024-04-26T16:15:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15849.751431 secs 2024-04-26T16:15:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15849.751431 secs 2024-04-26T16:15:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15849.751431 secs 2024-04-26T16:15:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:10 smithi033 ceph-mon[27845]: pgmap v8960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:10 smithi096 ceph-mon[32753]: pgmap v8960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:10 smithi194 ceph-mon[33017]: pgmap v8960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:12.477 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:12 smithi033 ceph-mon[27845]: pgmap v8961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:12 smithi096 ceph-mon[32753]: pgmap v8961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:12 smithi194 ceph-mon[33017]: pgmap v8961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15851.935498 secs 2024-04-26T16:15:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15851.935498 secs 2024-04-26T16:15:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15851.935498 secs 2024-04-26T16:15:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:14.412+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:14.412+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:15:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:14.412+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:14 smithi033 ceph-mon[27845]: pgmap v8962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15854.751521 secs 2024-04-26T16:15:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:14 smithi096 ceph-mon[32753]: pgmap v8962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15854.751521 secs 2024-04-26T16:15:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:14 smithi194 ceph-mon[33017]: pgmap v8962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15854.751521 secs 2024-04-26T16:15:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:16 smithi033 ceph-mon[27845]: pgmap v8963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:16 smithi096 ceph-mon[32753]: pgmap v8963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:16 smithi194 ceph-mon[33017]: pgmap v8963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:18 smithi033 ceph-mon[27845]: pgmap v8964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15856.935575 secs 2024-04-26T16:15:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:18 smithi096 ceph-mon[32753]: pgmap v8964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15856.935575 secs 2024-04-26T16:15:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:18 smithi194 ceph-mon[33017]: pgmap v8964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:18.929 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15856.935575 secs 2024-04-26T16:15:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15859.751650 secs 2024-04-26T16:15:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15859.751650 secs 2024-04-26T16:15:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15859.751650 secs 2024-04-26T16:15:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:20 smithi033 ceph-mon[27845]: pgmap v8965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:20 smithi096 ceph-mon[32753]: pgmap v8965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:20 smithi194 ceph-mon[33017]: pgmap v8965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:22 smithi033 ceph-mon[27845]: pgmap v8966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:22 smithi096 ceph-mon[32753]: pgmap v8966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:22 smithi194 ceph-mon[33017]: pgmap v8966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15861.935663 secs 2024-04-26T16:15:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15861.935663 secs 2024-04-26T16:15:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15861.935663 secs 2024-04-26T16:15:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:24 smithi096 ceph-mon[32753]: pgmap v8967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15864.751786 secs 2024-04-26T16:15:24.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:24 smithi033 ceph-mon[27845]: pgmap v8967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:24.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15864.751786 secs 2024-04-26T16:15:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:24 smithi194 ceph-mon[33017]: pgmap v8967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15864.751786 secs 2024-04-26T16:15:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:25 smithi194 ceph-mon[33017]: pgmap v8968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:25 smithi033 ceph-mon[27845]: pgmap v8968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:25 smithi096 ceph-mon[32753]: pgmap v8968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:28 smithi033 ceph-mon[27845]: pgmap v8969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15866.935744 secs 2024-04-26T16:15:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:28 smithi096 ceph-mon[32753]: pgmap v8969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15866.935744 secs 2024-04-26T16:15:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:28 smithi194 ceph-mon[33017]: pgmap v8969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15866.935744 secs 2024-04-26T16:15:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:29.425+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:29.425+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:15:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:29.425+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15869.751912 secs 2024-04-26T16:15:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15869.751912 secs 2024-04-26T16:15:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15869.751912 secs 2024-04-26T16:15:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:30 smithi033 ceph-mon[27845]: pgmap v8970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:30 smithi096 ceph-mon[32753]: pgmap v8970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:30 smithi194 ceph-mon[33017]: pgmap v8970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:32 smithi033 ceph-mon[27845]: pgmap v8971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:32 smithi096 ceph-mon[32753]: pgmap v8971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:32 smithi194 ceph-mon[33017]: pgmap v8971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15871.935827 secs 2024-04-26T16:15:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15871.935827 secs 2024-04-26T16:15:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15871.935827 secs 2024-04-26T16:15:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:34 smithi033 ceph-mon[27845]: pgmap v8972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15874.752032 secs 2024-04-26T16:15:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:34 smithi096 ceph-mon[32753]: pgmap v8972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15874.752032 secs 2024-04-26T16:15:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:34 smithi194 ceph-mon[33017]: pgmap v8972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15874.752032 secs 2024-04-26T16:15:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:36 smithi033 ceph-mon[27845]: pgmap v8973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:36 smithi096 ceph-mon[32753]: pgmap v8973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:36 smithi194 ceph-mon[33017]: pgmap v8973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:38 smithi033 ceph-mon[27845]: pgmap v8974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15876.935911 secs 2024-04-26T16:15:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:38 smithi096 ceph-mon[32753]: pgmap v8974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15876.935911 secs 2024-04-26T16:15:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:38 smithi194 ceph-mon[33017]: pgmap v8974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15876.935911 secs 2024-04-26T16:15:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15879.752192 secs 2024-04-26T16:15:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15879.752192 secs 2024-04-26T16:15:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15879.752192 secs 2024-04-26T16:15:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:40 smithi033 ceph-mon[27845]: pgmap v8975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:40 smithi096 ceph-mon[32753]: pgmap v8975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:40 smithi194 ceph-mon[33017]: pgmap v8975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:42 smithi096 ceph-mon[32753]: pgmap v8976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:42 smithi194 ceph-mon[33017]: pgmap v8976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:42.895 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:42 smithi033 ceph-mon[27845]: pgmap v8976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15881.936013 secs 2024-04-26T16:15:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:43 smithi194 ceph-mon[33017]: pgmap v8977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15881.936013 secs 2024-04-26T16:15:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:43 smithi033 ceph-mon[27845]: pgmap v8977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15881.936013 secs 2024-04-26T16:15:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:43 smithi096 ceph-mon[32753]: pgmap v8977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:44.440+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:44.440+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:44.440+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:15:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15884.752271 secs 2024-04-26T16:15:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15884.752271 secs 2024-04-26T16:15:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15884.752271 secs 2024-04-26T16:15:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:45 smithi194 ceph-mon[33017]: pgmap v8978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:45 smithi033 ceph-mon[27845]: pgmap v8978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:45 smithi096 ceph-mon[32753]: pgmap v8978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:15:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:47 smithi194 ceph-mon[33017]: pgmap v8979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:47 smithi194 ceph-mon[33017]: pgmap v8980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:15:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:47 smithi033 ceph-mon[27845]: pgmap v8979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:47 smithi033 ceph-mon[27845]: pgmap v8980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:15:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:47 smithi096 ceph-mon[32753]: pgmap v8979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:15:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:47 smithi096 ceph-mon[32753]: pgmap v8980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:15:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15886.936076 secs 2024-04-26T16:15:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15886.936076 secs 2024-04-26T16:15:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15886.936076 secs 2024-04-26T16:15:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15889.752416 secs 2024-04-26T16:15:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:49 smithi194 ceph-mon[33017]: pgmap v8981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:15:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15889.752416 secs 2024-04-26T16:15:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:49 smithi033 ceph-mon[27845]: pgmap v8981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:15:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15889.752416 secs 2024-04-26T16:15:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:49 smithi096 ceph-mon[32753]: pgmap v8981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:15:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:52 smithi194 ceph-mon[33017]: pgmap v8982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:52 smithi033 ceph-mon[27845]: pgmap v8982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:52 smithi096 ceph-mon[32753]: pgmap v8982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15891.936166 secs 2024-04-26T16:15:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:54 smithi194 ceph-mon[33017]: pgmap v8983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15894.752547 secs 2024-04-26T16:15:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15891.936166 secs 2024-04-26T16:15:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:54 smithi033 ceph-mon[27845]: pgmap v8983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15894.752547 secs 2024-04-26T16:15:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15891.936166 secs 2024-04-26T16:15:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:54 smithi096 ceph-mon[32753]: pgmap v8983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15894.752547 secs 2024-04-26T16:15:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:56 smithi194 ceph-mon[33017]: pgmap v8984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:56 smithi033 ceph-mon[27845]: pgmap v8984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:56 smithi096 ceph-mon[32753]: pgmap v8984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:15:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:58 smithi194 ceph-mon[33017]: pgmap v8985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:15:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:58 smithi033 ceph-mon[27845]: pgmap v8985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:15:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:58 smithi096 ceph-mon[32753]: pgmap v8985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:15:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15896.936243 secs 2024-04-26T16:15:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:15:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15899.752671 secs 2024-04-26T16:15:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15896.936243 secs 2024-04-26T16:15:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:15:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15899.752671 secs 2024-04-26T16:15:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15896.936243 secs 2024-04-26T16:15:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:15:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15899.752671 secs 2024-04-26T16:15:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:59.454+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:59.454+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:15:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:15:59.454+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:16:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:00 smithi194 ceph-mon[33017]: pgmap v8986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:00 smithi033 ceph-mon[27845]: pgmap v8986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:00 smithi096 ceph-mon[32753]: pgmap v8986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:02 smithi194 ceph-mon[33017]: pgmap v8987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:02 smithi033 ceph-mon[27845]: pgmap v8987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:02 smithi096 ceph-mon[32753]: pgmap v8987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15901.936302 secs 2024-04-26T16:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:04 smithi194 ceph-mon[33017]: pgmap v8988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15904.752799 secs 2024-04-26T16:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15901.936302 secs 2024-04-26T16:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:04 smithi033 ceph-mon[27845]: pgmap v8988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15904.752799 secs 2024-04-26T16:16:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15901.936302 secs 2024-04-26T16:16:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:04 smithi096 ceph-mon[32753]: pgmap v8988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15904.752799 secs 2024-04-26T16:16:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:06 smithi194 ceph-mon[33017]: pgmap v8989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:06 smithi033 ceph-mon[27845]: pgmap v8989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:06 smithi096 ceph-mon[32753]: pgmap v8989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:08 smithi194 ceph-mon[33017]: pgmap v8990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:08 smithi033 ceph-mon[27845]: pgmap v8990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:08 smithi096 ceph-mon[32753]: pgmap v8990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15906.936363 secs 2024-04-26T16:16:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15909.752925 secs 2024-04-26T16:16:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15906.936363 secs 2024-04-26T16:16:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15909.752925 secs 2024-04-26T16:16:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15906.936363 secs 2024-04-26T16:16:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15909.752925 secs 2024-04-26T16:16:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:10 smithi194 ceph-mon[33017]: pgmap v8991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:10 smithi033 ceph-mon[27845]: pgmap v8991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:10 smithi096 ceph-mon[32753]: pgmap v8991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:12 smithi194 ceph-mon[33017]: pgmap v8992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:12 smithi033 ceph-mon[27845]: pgmap v8992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:12 smithi096 ceph-mon[32753]: pgmap v8992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15911.936425 secs 2024-04-26T16:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:14 smithi194 ceph-mon[33017]: pgmap v8993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15914.753051 secs 2024-04-26T16:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15911.936425 secs 2024-04-26T16:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:14 smithi033 ceph-mon[27845]: pgmap v8993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15914.753051 secs 2024-04-26T16:16:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15911.936425 secs 2024-04-26T16:16:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:14 smithi096 ceph-mon[32753]: pgmap v8993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15914.753051 secs 2024-04-26T16:16:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:14.470+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:14.470+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:16:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:14.470+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:16 smithi033 ceph-mon[27845]: pgmap v8994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:16 smithi096 ceph-mon[32753]: pgmap v8994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:16 smithi194 ceph-mon[33017]: pgmap v8994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:18 smithi033 ceph-mon[27845]: pgmap v8995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:18 smithi096 ceph-mon[32753]: pgmap v8995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:18 smithi194 ceph-mon[33017]: pgmap v8995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15916.936506 secs 2024-04-26T16:16:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15919.753173 secs 2024-04-26T16:16:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15916.936506 secs 2024-04-26T16:16:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15919.753173 secs 2024-04-26T16:16:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15916.936506 secs 2024-04-26T16:16:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15919.753173 secs 2024-04-26T16:16:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:20 smithi033 ceph-mon[27845]: pgmap v8996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:20 smithi096 ceph-mon[32753]: pgmap v8996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:20 smithi194 ceph-mon[33017]: pgmap v8996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:22 smithi033 ceph-mon[27845]: pgmap v8997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:22 smithi096 ceph-mon[32753]: pgmap v8997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:22 smithi194 ceph-mon[33017]: pgmap v8997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15921.936586 secs 2024-04-26T16:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:24 smithi033 ceph-mon[27845]: pgmap v8998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15924.753298 secs 2024-04-26T16:16:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15921.936586 secs 2024-04-26T16:16:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:24 smithi096 ceph-mon[32753]: pgmap v8998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15924.753298 secs 2024-04-26T16:16:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15921.936586 secs 2024-04-26T16:16:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:24 smithi194 ceph-mon[33017]: pgmap v8998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:16:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15924.753298 secs 2024-04-26T16:16:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:26 smithi033 ceph-mon[27845]: pgmap v8999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:26 smithi096 ceph-mon[32753]: pgmap v8999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:26 smithi194 ceph-mon[33017]: pgmap v8999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:16:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:28 smithi033 ceph-mon[27845]: pgmap v9000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:28 smithi096 ceph-mon[32753]: pgmap v9000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:28 smithi194 ceph-mon[33017]: pgmap v9000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15926.936667 secs 2024-04-26T16:16:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15929.753430 secs 2024-04-26T16:16:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15926.936667 secs 2024-04-26T16:16:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15929.753430 secs 2024-04-26T16:16:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:29.478+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:29.478+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:29.478+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:16:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15926.936667 secs 2024-04-26T16:16:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15929.753430 secs 2024-04-26T16:16:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:30 smithi033 ceph-mon[27845]: pgmap v9001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:30 smithi096 ceph-mon[32753]: pgmap v9001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:30 smithi194 ceph-mon[33017]: pgmap v9001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:32 smithi033 ceph-mon[27845]: pgmap v9002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:32 smithi096 ceph-mon[32753]: pgmap v9002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:32 smithi194 ceph-mon[33017]: pgmap v9002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15931.936750 secs 2024-04-26T16:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:34 smithi033 ceph-mon[27845]: pgmap v9003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15934.753500 secs 2024-04-26T16:16:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15931.936750 secs 2024-04-26T16:16:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:34 smithi096 ceph-mon[32753]: pgmap v9003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15934.753500 secs 2024-04-26T16:16:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15931.936750 secs 2024-04-26T16:16:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:34 smithi194 ceph-mon[33017]: pgmap v9003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15934.753500 secs 2024-04-26T16:16:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:36 smithi033 ceph-mon[27845]: pgmap v9004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:36 smithi096 ceph-mon[32753]: pgmap v9004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:36 smithi194 ceph-mon[33017]: pgmap v9004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:38 smithi096 ceph-mon[32753]: pgmap v9005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:38 smithi194 ceph-mon[33017]: pgmap v9005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:38 smithi033 ceph-mon[27845]: pgmap v9005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15936.936831 secs 2024-04-26T16:16:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15939.753626 secs 2024-04-26T16:16:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15936.936831 secs 2024-04-26T16:16:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15939.753626 secs 2024-04-26T16:16:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15936.936831 secs 2024-04-26T16:16:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15939.753626 secs 2024-04-26T16:16:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:40 smithi194 ceph-mon[33017]: pgmap v9006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:40 smithi033 ceph-mon[27845]: pgmap v9006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:40 smithi096 ceph-mon[32753]: pgmap v9006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:42.489 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:42 smithi096 ceph-mon[32753]: pgmap v9007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:42 smithi194 ceph-mon[33017]: pgmap v9007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:42 smithi033 ceph-mon[27845]: pgmap v9007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15941.936909 secs 2024-04-26T16:16:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15941.936909 secs 2024-04-26T16:16:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15941.936909 secs 2024-04-26T16:16:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:44.488+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:44.488+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:16:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:44.488+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:44 smithi194 ceph-mon[33017]: pgmap v9008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15944.753748 secs 2024-04-26T16:16:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:44 smithi033 ceph-mon[27845]: pgmap v9008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15944.753748 secs 2024-04-26T16:16:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:44 smithi096 ceph-mon[32753]: pgmap v9008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15944.753748 secs 2024-04-26T16:16:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:46 smithi194 ceph-mon[33017]: pgmap v9009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:46 smithi033 ceph-mon[27845]: pgmap v9009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:46 smithi096 ceph-mon[32753]: pgmap v9009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:16:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:48 smithi194 ceph-mon[33017]: pgmap v9010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15946.936981 secs 2024-04-26T16:16:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:48 smithi033 ceph-mon[27845]: pgmap v9010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15946.936981 secs 2024-04-26T16:16:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:48 smithi096 ceph-mon[32753]: pgmap v9010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15946.936981 secs 2024-04-26T16:16:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15949.753863 secs 2024-04-26T16:16:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:16:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15949.753863 secs 2024-04-26T16:16:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:16:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15949.753863 secs 2024-04-26T16:16:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:16:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:50 smithi194 ceph-mon[33017]: pgmap v9011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:50 smithi194 ceph-mon[33017]: pgmap v9012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:50 smithi033 ceph-mon[27845]: pgmap v9011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:50 smithi033 ceph-mon[27845]: pgmap v9012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:50 smithi096 ceph-mon[32753]: pgmap v9011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:16:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:50 smithi096 ceph-mon[32753]: pgmap v9012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:52 smithi194 ceph-mon[33017]: pgmap v9013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:52 smithi033 ceph-mon[27845]: pgmap v9013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:52 smithi096 ceph-mon[32753]: pgmap v9013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15951.937008 secs 2024-04-26T16:16:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15951.937008 secs 2024-04-26T16:16:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15951.937008 secs 2024-04-26T16:16:54.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:54 smithi033 ceph-mon[27845]: pgmap v9014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:54.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15954.753983 secs 2024-04-26T16:16:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:54 smithi194 ceph-mon[33017]: pgmap v9014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15954.753983 secs 2024-04-26T16:16:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:54 smithi096 ceph-mon[32753]: pgmap v9014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15954.753983 secs 2024-04-26T16:16:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:56 smithi194 ceph-mon[33017]: pgmap v9015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:56 smithi033 ceph-mon[27845]: pgmap v9015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:56 smithi096 ceph-mon[32753]: pgmap v9015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:58 smithi194 ceph-mon[33017]: pgmap v9016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15956.937091 secs 2024-04-26T16:16:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:58 smithi033 ceph-mon[27845]: pgmap v9016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15956.937091 secs 2024-04-26T16:16:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:58 smithi096 ceph-mon[32753]: pgmap v9016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:16:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15956.937091 secs 2024-04-26T16:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:59.495+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:59.495+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:16:59.495+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:16:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15959.754151 secs 2024-04-26T16:16:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15959.754151 secs 2024-04-26T16:16:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15959.754151 secs 2024-04-26T16:17:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:00 smithi194 ceph-mon[33017]: pgmap v9017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:17:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:00 smithi033 ceph-mon[27845]: pgmap v9017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:17:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:00 smithi096 ceph-mon[32753]: pgmap v9017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:17:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:02 smithi194 ceph-mon[33017]: pgmap v9018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:02 smithi033 ceph-mon[27845]: pgmap v9018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:02 smithi096 ceph-mon[32753]: pgmap v9018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15961.937152 secs 2024-04-26T16:17:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15961.937152 secs 2024-04-26T16:17:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15961.937152 secs 2024-04-26T16:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:04 smithi194 ceph-mon[33017]: pgmap v9019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15964.754265 secs 2024-04-26T16:17:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:04 smithi033 ceph-mon[27845]: pgmap v9019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15964.754265 secs 2024-04-26T16:17:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:04 smithi096 ceph-mon[32753]: pgmap v9019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15964.754265 secs 2024-04-26T16:17:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:06 smithi194 ceph-mon[33017]: pgmap v9020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:06 smithi033 ceph-mon[27845]: pgmap v9020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:06 smithi096 ceph-mon[32753]: pgmap v9020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:08 smithi194 ceph-mon[33017]: pgmap v9021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15966.937208 secs 2024-04-26T16:17:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:08 smithi033 ceph-mon[27845]: pgmap v9021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15966.937208 secs 2024-04-26T16:17:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:08 smithi096 ceph-mon[32753]: pgmap v9021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15966.937208 secs 2024-04-26T16:17:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15969.754334 secs 2024-04-26T16:17:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15969.754334 secs 2024-04-26T16:17:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15969.754334 secs 2024-04-26T16:17:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:10 smithi194 ceph-mon[33017]: pgmap v9022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:10 smithi033 ceph-mon[27845]: pgmap v9022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:10 smithi096 ceph-mon[32753]: pgmap v9022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:12 smithi194 ceph-mon[33017]: pgmap v9023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:12 smithi033 ceph-mon[27845]: pgmap v9023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:12 smithi096 ceph-mon[32753]: pgmap v9023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15971.937270 secs 2024-04-26T16:17:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15971.937270 secs 2024-04-26T16:17:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15971.937270 secs 2024-04-26T16:17:14.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:14.506+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:14.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:14.506+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:17:14.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:14.506+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:14 smithi194 ceph-mon[33017]: pgmap v9024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15974.754457 secs 2024-04-26T16:17:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:14 smithi033 ceph-mon[27845]: pgmap v9024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15974.754457 secs 2024-04-26T16:17:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:14 smithi096 ceph-mon[32753]: pgmap v9024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15974.754457 secs 2024-04-26T16:17:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:16 smithi194 ceph-mon[33017]: pgmap v9025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:16 smithi033 ceph-mon[27845]: pgmap v9025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:16 smithi096 ceph-mon[32753]: pgmap v9025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:18 smithi033 ceph-mon[27845]: pgmap v9026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15976.937352 secs 2024-04-26T16:17:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:18 smithi096 ceph-mon[32753]: pgmap v9026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15976.937352 secs 2024-04-26T16:17:18.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:18 smithi194 ceph-mon[33017]: pgmap v9026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:18.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15976.937352 secs 2024-04-26T16:17:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15979.754536 secs 2024-04-26T16:17:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15979.754536 secs 2024-04-26T16:17:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15979.754536 secs 2024-04-26T16:17:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:20 smithi033 ceph-mon[27845]: pgmap v9027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:20 smithi096 ceph-mon[32753]: pgmap v9027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:20 smithi194 ceph-mon[33017]: pgmap v9027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:22 smithi033 ceph-mon[27845]: pgmap v9028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:22 smithi096 ceph-mon[32753]: pgmap v9028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:22 smithi194 ceph-mon[33017]: pgmap v9028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15981.937411 secs 2024-04-26T16:17:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15981.937411 secs 2024-04-26T16:17:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15981.937411 secs 2024-04-26T16:17:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:24.514+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:17:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:24.514+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:17:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:24.514+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:17:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:24.514+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T16:17:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:24 smithi033 ceph-mon[27845]: pgmap v9029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15984.754651 secs 2024-04-26T16:17:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:24 smithi096 ceph-mon[32753]: pgmap v9029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15984.754651 secs 2024-04-26T16:17:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:24 smithi194 ceph-mon[33017]: pgmap v9029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15984.754651 secs 2024-04-26T16:17:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:26 smithi033 ceph-mon[27845]: pgmap v9030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:26 smithi096 ceph-mon[32753]: pgmap v9030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:26 smithi194 ceph-mon[33017]: pgmap v9030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:28 smithi033 ceph-mon[27845]: pgmap v9031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15986.937495 secs 2024-04-26T16:17:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:28 smithi096 ceph-mon[32753]: pgmap v9031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15986.937495 secs 2024-04-26T16:17:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:28 smithi194 ceph-mon[33017]: pgmap v9031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15986.937495 secs 2024-04-26T16:17:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:29.521+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:29.521+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:17:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:29.521+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15989.754750 secs 2024-04-26T16:17:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15989.754750 secs 2024-04-26T16:17:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15989.754750 secs 2024-04-26T16:17:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:30 smithi033 ceph-mon[27845]: pgmap v9032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:30 smithi096 ceph-mon[32753]: pgmap v9032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:30 smithi194 ceph-mon[33017]: pgmap v9032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:32 smithi033 ceph-mon[27845]: pgmap v9033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:32 smithi096 ceph-mon[32753]: pgmap v9033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:32 smithi194 ceph-mon[33017]: pgmap v9033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15991.937579 secs 2024-04-26T16:17:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15991.937579 secs 2024-04-26T16:17:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15991.937579 secs 2024-04-26T16:17:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:34 smithi096 ceph-mon[32753]: pgmap v9034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15994.754868 secs 2024-04-26T16:17:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:34 smithi194 ceph-mon[33017]: pgmap v9034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15994.754868 secs 2024-04-26T16:17:34.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:34 smithi033 ceph-mon[27845]: pgmap v9034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:34.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15994.754868 secs 2024-04-26T16:17:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:36 smithi194 ceph-mon[33017]: pgmap v9035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:36 smithi033 ceph-mon[27845]: pgmap v9035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:36 smithi096 ceph-mon[32753]: pgmap v9035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:38 smithi194 ceph-mon[33017]: pgmap v9036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 15996.937640 secs 2024-04-26T16:17:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:38 smithi033 ceph-mon[27845]: pgmap v9036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 15996.937640 secs 2024-04-26T16:17:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:38 smithi096 ceph-mon[32753]: pgmap v9036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 15996.937640 secs 2024-04-26T16:17:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 15999.754987 secs 2024-04-26T16:17:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 15999.754987 secs 2024-04-26T16:17:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 15999.754987 secs 2024-04-26T16:17:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:40 smithi194 ceph-mon[33017]: pgmap v9037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:40 smithi033 ceph-mon[27845]: pgmap v9037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:40 smithi096 ceph-mon[32753]: pgmap v9037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:41 smithi194 ceph-mon[33017]: pgmap v9038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:41 smithi033 ceph-mon[27845]: pgmap v9038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:41 smithi096 ceph-mon[32753]: pgmap v9038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:43 smithi194 ceph-mon[33017]: pgmap v9039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16001.937724 secs 2024-04-26T16:17:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16004.755100 secs 2024-04-26T16:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:43 smithi033 ceph-mon[27845]: pgmap v9039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16001.937724 secs 2024-04-26T16:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16004.755100 secs 2024-04-26T16:17:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:43 smithi096 ceph-mon[32753]: pgmap v9039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16001.937724 secs 2024-04-26T16:17:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16004.755100 secs 2024-04-26T16:17:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:44.533+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:44.533+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:17:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:44.533+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:45 smithi194 ceph-mon[33017]: pgmap v9040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:45 smithi033 ceph-mon[27845]: pgmap v9040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:45 smithi096 ceph-mon[32753]: pgmap v9040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:17:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:47 smithi194 ceph-mon[33017]: pgmap v9041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:47 smithi033 ceph-mon[27845]: pgmap v9041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:47 smithi096 ceph-mon[32753]: pgmap v9041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16006.937782 secs 2024-04-26T16:17:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16009.755237 secs 2024-04-26T16:17:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16006.937782 secs 2024-04-26T16:17:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16009.755237 secs 2024-04-26T16:17:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16006.937782 secs 2024-04-26T16:17:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16009.755237 secs 2024-04-26T16:17:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:49 smithi194 ceph-mon[33017]: pgmap v9042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:49 smithi033 ceph-mon[27845]: pgmap v9042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:49 smithi096 ceph-mon[32753]: pgmap v9042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:17:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:50 smithi194 ceph-mon[33017]: pgmap v9043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:17:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:50 smithi194 ceph-mon[33017]: pgmap v9044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:17:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:50 smithi033 ceph-mon[27845]: pgmap v9043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:17:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:50 smithi033 ceph-mon[27845]: pgmap v9044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:17:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:50 smithi096 ceph-mon[32753]: pgmap v9043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:17:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:50 smithi096 ceph-mon[32753]: pgmap v9044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:17:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:53 smithi033 ceph-mon[27845]: pgmap v9045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16011.937868 secs 2024-04-26T16:17:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:53 smithi096 ceph-mon[32753]: pgmap v9045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16011.937868 secs 2024-04-26T16:17:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:53 smithi194 ceph-mon[33017]: pgmap v9045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16011.937868 secs 2024-04-26T16:17:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16014.755372 secs 2024-04-26T16:17:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16014.755372 secs 2024-04-26T16:17:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16014.755372 secs 2024-04-26T16:17:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:55 smithi033 ceph-mon[27845]: pgmap v9046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:55 smithi096 ceph-mon[32753]: pgmap v9046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:55 smithi194 ceph-mon[33017]: pgmap v9046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:57 smithi033 ceph-mon[27845]: pgmap v9047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:57 smithi096 ceph-mon[32753]: pgmap v9047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:57 smithi194 ceph-mon[33017]: pgmap v9047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16016.937954 secs 2024-04-26T16:17:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16016.937954 secs 2024-04-26T16:17:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16016.937954 secs 2024-04-26T16:17:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:59.548+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:59.548+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:17:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:17:59.548+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:17:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:59 smithi033 ceph-mon[27845]: pgmap v9048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16019.755453 secs 2024-04-26T16:17:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:59 smithi096 ceph-mon[32753]: pgmap v9048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16019.755453 secs 2024-04-26T16:17:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:59 smithi194 ceph-mon[33017]: pgmap v9048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:17:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16019.755453 secs 2024-04-26T16:18:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:01 smithi033 ceph-mon[27845]: pgmap v9049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:18:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:01 smithi096 ceph-mon[32753]: pgmap v9049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:18:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:01 smithi194 ceph-mon[33017]: pgmap v9049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:18:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:03 smithi033 ceph-mon[27845]: pgmap v9050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16021.938015 secs 2024-04-26T16:18:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:03 smithi096 ceph-mon[32753]: pgmap v9050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16021.938015 secs 2024-04-26T16:18:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:03 smithi194 ceph-mon[33017]: pgmap v9050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16021.938015 secs 2024-04-26T16:18:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16024.755568 secs 2024-04-26T16:18:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16024.755568 secs 2024-04-26T16:18:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16024.755568 secs 2024-04-26T16:18:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:05 smithi096 ceph-mon[32753]: pgmap v9051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:05 smithi194 ceph-mon[33017]: pgmap v9051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:05 smithi033 ceph-mon[27845]: pgmap v9051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:07 smithi194 ceph-mon[33017]: pgmap v9052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:07 smithi033 ceph-mon[27845]: pgmap v9052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:07 smithi096 ceph-mon[32753]: pgmap v9052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16026.938078 secs 2024-04-26T16:18:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16026.938078 secs 2024-04-26T16:18:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16026.938078 secs 2024-04-26T16:18:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16029.755634 secs 2024-04-26T16:18:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:09 smithi194 ceph-mon[33017]: pgmap v9053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16029.755634 secs 2024-04-26T16:18:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:09 smithi033 ceph-mon[27845]: pgmap v9053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16029.755634 secs 2024-04-26T16:18:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:09 smithi096 ceph-mon[32753]: pgmap v9053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:11 smithi194 ceph-mon[33017]: pgmap v9054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:11 smithi033 ceph-mon[27845]: pgmap v9054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:11 smithi096 ceph-mon[32753]: pgmap v9054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:12 smithi194 ceph-mon[33017]: pgmap v9055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:12 smithi033 ceph-mon[27845]: pgmap v9055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:12 smithi096 ceph-mon[32753]: pgmap v9055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:13.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16031.938140 secs 2024-04-26T16:18:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16031.938140 secs 2024-04-26T16:18:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16031.938140 secs 2024-04-26T16:18:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:14.563+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:14.563+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:14.563+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:18:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16034.755696 secs 2024-04-26T16:18:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:14 smithi194 ceph-mon[33017]: pgmap v9056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16034.755696 secs 2024-04-26T16:18:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:14 smithi033 ceph-mon[27845]: pgmap v9056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16034.755696 secs 2024-04-26T16:18:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:14 smithi096 ceph-mon[32753]: pgmap v9056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:17 smithi033 ceph-mon[27845]: pgmap v9057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:17 smithi096 ceph-mon[32753]: pgmap v9057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:17 smithi194 ceph-mon[33017]: pgmap v9057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16036.938203 secs 2024-04-26T16:18:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16036.938203 secs 2024-04-26T16:18:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16036.938203 secs 2024-04-26T16:18:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16039.755762 secs 2024-04-26T16:18:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:19 smithi033 ceph-mon[27845]: pgmap v9058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16039.755762 secs 2024-04-26T16:18:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:19 smithi096 ceph-mon[32753]: pgmap v9058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16039.755762 secs 2024-04-26T16:18:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:19 smithi194 ceph-mon[33017]: pgmap v9058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:21 smithi033 ceph-mon[27845]: pgmap v9059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:21 smithi096 ceph-mon[32753]: pgmap v9059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:21 smithi194 ceph-mon[33017]: pgmap v9059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:23 smithi033 ceph-mon[27845]: pgmap v9060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16041.938263 secs 2024-04-26T16:18:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:23 smithi096 ceph-mon[32753]: pgmap v9060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16041.938263 secs 2024-04-26T16:18:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:23 smithi194 ceph-mon[33017]: pgmap v9060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16041.938263 secs 2024-04-26T16:18:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16044.755893 secs 2024-04-26T16:18:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16044.755893 secs 2024-04-26T16:18:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16044.755893 secs 2024-04-26T16:18:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:25 smithi096 ceph-mon[32753]: pgmap v9061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:25 smithi194 ceph-mon[33017]: pgmap v9061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:25 smithi033 ceph-mon[27845]: pgmap v9061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:27.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:27 smithi096 ceph-mon[32753]: pgmap v9062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:27 smithi194 ceph-mon[33017]: pgmap v9062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:27 smithi033 ceph-mon[27845]: pgmap v9062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16046.938347 secs 2024-04-26T16:18:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16046.938347 secs 2024-04-26T16:18:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16046.938347 secs 2024-04-26T16:18:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:29.579+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:29.579+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:18:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:29.579+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16049.756009 secs 2024-04-26T16:18:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:29 smithi194 ceph-mon[33017]: pgmap v9063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16049.756009 secs 2024-04-26T16:18:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:29 smithi033 ceph-mon[27845]: pgmap v9063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16049.756009 secs 2024-04-26T16:18:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:29 smithi096 ceph-mon[32753]: pgmap v9063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:31 smithi194 ceph-mon[33017]: pgmap v9064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:31 smithi033 ceph-mon[27845]: pgmap v9064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:31 smithi096 ceph-mon[32753]: pgmap v9064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:33 smithi194 ceph-mon[33017]: pgmap v9065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16051.938428 secs 2024-04-26T16:18:33.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:33 smithi033 ceph-mon[27845]: pgmap v9065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:33.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16051.938428 secs 2024-04-26T16:18:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:33 smithi096 ceph-mon[32753]: pgmap v9065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16051.938428 secs 2024-04-26T16:18:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16054.756127 secs 2024-04-26T16:18:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16054.756127 secs 2024-04-26T16:18:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16054.756127 secs 2024-04-26T16:18:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:35 smithi194 ceph-mon[33017]: pgmap v9066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:35 smithi033 ceph-mon[27845]: pgmap v9066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:35 smithi096 ceph-mon[32753]: pgmap v9066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:37 smithi194 ceph-mon[33017]: pgmap v9067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:37 smithi033 ceph-mon[27845]: pgmap v9067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:37 smithi096 ceph-mon[32753]: pgmap v9067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16056.938512 secs 2024-04-26T16:18:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:38 smithi194 ceph-mon[33017]: pgmap v9068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16056.938512 secs 2024-04-26T16:18:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:38 smithi033 ceph-mon[27845]: pgmap v9068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16056.938512 secs 2024-04-26T16:18:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:38 smithi096 ceph-mon[32753]: pgmap v9068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16059.756252 secs 2024-04-26T16:18:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16059.756252 secs 2024-04-26T16:18:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16059.756252 secs 2024-04-26T16:18:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:40 smithi194 ceph-mon[33017]: pgmap v9069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:40 smithi033 ceph-mon[27845]: pgmap v9069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:40 smithi096 ceph-mon[32753]: pgmap v9069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:43 smithi033 ceph-mon[27845]: pgmap v9070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16061.938595 secs 2024-04-26T16:18:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:43 smithi096 ceph-mon[32753]: pgmap v9070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16061.938595 secs 2024-04-26T16:18:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:43 smithi194 ceph-mon[33017]: pgmap v9070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16061.938595 secs 2024-04-26T16:18:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:44.589+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:44.589+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:18:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:44.589+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16064.756399 secs 2024-04-26T16:18:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16064.756399 secs 2024-04-26T16:18:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16064.756399 secs 2024-04-26T16:18:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:45 smithi033 ceph-mon[27845]: pgmap v9071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:45 smithi096 ceph-mon[32753]: pgmap v9071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:45 smithi194 ceph-mon[33017]: pgmap v9071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:47 smithi033 ceph-mon[27845]: pgmap v9072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:47 smithi096 ceph-mon[32753]: pgmap v9072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:47 smithi194 ceph-mon[33017]: pgmap v9072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16066.938680 secs 2024-04-26T16:18:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16066.938680 secs 2024-04-26T16:18:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16066.938680 secs 2024-04-26T16:18:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16069.756518 secs 2024-04-26T16:18:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:49 smithi033 ceph-mon[27845]: pgmap v9073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16069.756518 secs 2024-04-26T16:18:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:49 smithi096 ceph-mon[32753]: pgmap v9073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16069.756518 secs 2024-04-26T16:18:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:49 smithi194 ceph-mon[33017]: pgmap v9073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:51 smithi194 ceph-mon[33017]: pgmap v9074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:51 smithi033 ceph-mon[27845]: pgmap v9074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.940 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:51 smithi096 ceph-mon[32753]: pgmap v9074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:18:51.940 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:18:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:18:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:18:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:53 smithi194 ceph-mon[33017]: pgmap v9075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:53 smithi194 ceph-mon[33017]: pgmap v9076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16071.938764 secs 2024-04-26T16:18:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:53 smithi033 ceph-mon[27845]: pgmap v9075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:53 smithi033 ceph-mon[27845]: pgmap v9076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16071.938764 secs 2024-04-26T16:18:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:53 smithi096 ceph-mon[32753]: pgmap v9075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:18:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:53 smithi096 ceph-mon[32753]: pgmap v9076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16071.938764 secs 2024-04-26T16:18:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16074.756651 secs 2024-04-26T16:18:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16074.756651 secs 2024-04-26T16:18:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16074.756651 secs 2024-04-26T16:18:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:55 smithi194 ceph-mon[33017]: pgmap v9077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:18:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:55 smithi033 ceph-mon[27845]: pgmap v9077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:18:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:55 smithi096 ceph-mon[32753]: pgmap v9077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:18:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:57 smithi194 ceph-mon[33017]: pgmap v9078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:57 smithi033 ceph-mon[27845]: pgmap v9078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:57 smithi096 ceph-mon[32753]: pgmap v9078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16076.938822 secs 2024-04-26T16:18:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:58 smithi194 ceph-mon[33017]: pgmap v9079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16076.938822 secs 2024-04-26T16:18:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:58 smithi033 ceph-mon[27845]: pgmap v9079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16076.938822 secs 2024-04-26T16:18:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:58 smithi096 ceph-mon[32753]: pgmap v9079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:18:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:59.599+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:59.599+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:18:59.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:18:59.599+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:18:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:18:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16079.756787 secs 2024-04-26T16:18:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:18:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16079.756787 secs 2024-04-26T16:18:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:18:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16079.756787 secs 2024-04-26T16:19:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:00 smithi194 ceph-mon[33017]: pgmap v9080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:19:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:00 smithi033 ceph-mon[27845]: pgmap v9080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:19:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:00 smithi096 ceph-mon[32753]: pgmap v9080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:19:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:03 smithi194 ceph-mon[33017]: pgmap v9081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:19:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:03 smithi033 ceph-mon[27845]: pgmap v9081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:19:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:03 smithi096 ceph-mon[32753]: pgmap v9081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:19:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16081.938905 secs 2024-04-26T16:19:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16084.756869 secs 2024-04-26T16:19:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16081.938905 secs 2024-04-26T16:19:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16084.756869 secs 2024-04-26T16:19:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16081.938905 secs 2024-04-26T16:19:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16084.756869 secs 2024-04-26T16:19:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:05 smithi194 ceph-mon[33017]: pgmap v9082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:05 smithi033 ceph-mon[27845]: pgmap v9082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:05 smithi096 ceph-mon[32753]: pgmap v9082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:07 smithi194 ceph-mon[33017]: pgmap v9083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:07 smithi033 ceph-mon[27845]: pgmap v9083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:07 smithi096 ceph-mon[32753]: pgmap v9083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16086.938986 secs 2024-04-26T16:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:09 smithi194 ceph-mon[33017]: pgmap v9084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16089.756989 secs 2024-04-26T16:19:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16086.938986 secs 2024-04-26T16:19:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:09 smithi033 ceph-mon[27845]: pgmap v9084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16089.756989 secs 2024-04-26T16:19:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16086.938986 secs 2024-04-26T16:19:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:09 smithi096 ceph-mon[32753]: pgmap v9084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16089.756989 secs 2024-04-26T16:19:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:11 smithi194 ceph-mon[33017]: pgmap v9085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:11 smithi033 ceph-mon[27845]: pgmap v9085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:11 smithi096 ceph-mon[32753]: pgmap v9085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:13 smithi194 ceph-mon[33017]: pgmap v9086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:13 smithi033 ceph-mon[27845]: pgmap v9086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:13 smithi096 ceph-mon[32753]: pgmap v9086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16091.939068 secs 2024-04-26T16:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16094.757061 secs 2024-04-26T16:19:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16091.939068 secs 2024-04-26T16:19:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16094.757061 secs 2024-04-26T16:19:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16091.939068 secs 2024-04-26T16:19:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16094.757061 secs 2024-04-26T16:19:14.615 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:14.616+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:14.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:14.616+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:14.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:14.616+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:19:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:15 smithi194 ceph-mon[33017]: pgmap v9087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:15 smithi033 ceph-mon[27845]: pgmap v9087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:15 smithi096 ceph-mon[32753]: pgmap v9087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:17 smithi033 ceph-mon[27845]: pgmap v9088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:17 smithi096 ceph-mon[32753]: pgmap v9088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:17 smithi194 ceph-mon[33017]: pgmap v9088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16096.939131 secs 2024-04-26T16:19:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:19 smithi033 ceph-mon[27845]: pgmap v9089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16099.757162 secs 2024-04-26T16:19:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16096.939131 secs 2024-04-26T16:19:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:19 smithi096 ceph-mon[32753]: pgmap v9089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16099.757162 secs 2024-04-26T16:19:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16096.939131 secs 2024-04-26T16:19:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:19 smithi194 ceph-mon[33017]: pgmap v9089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16099.757162 secs 2024-04-26T16:19:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:21 smithi033 ceph-mon[27845]: pgmap v9090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:21 smithi096 ceph-mon[32753]: pgmap v9090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:21 smithi194 ceph-mon[33017]: pgmap v9090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:23 smithi033 ceph-mon[27845]: pgmap v9091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:23 smithi096 ceph-mon[32753]: pgmap v9091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:23 smithi194 ceph-mon[33017]: pgmap v9091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16101.939212 secs 2024-04-26T16:19:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16104.757281 secs 2024-04-26T16:19:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16101.939212 secs 2024-04-26T16:19:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16104.757281 secs 2024-04-26T16:19:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16101.939212 secs 2024-04-26T16:19:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16104.757281 secs 2024-04-26T16:19:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:25 smithi033 ceph-mon[27845]: pgmap v9092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:25 smithi096 ceph-mon[32753]: pgmap v9092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:25 smithi194 ceph-mon[33017]: pgmap v9092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:27 smithi033 ceph-mon[27845]: pgmap v9093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:27 smithi096 ceph-mon[32753]: pgmap v9093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:27 smithi194 ceph-mon[33017]: pgmap v9093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16106.939284 secs 2024-04-26T16:19:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:29 smithi033 ceph-mon[27845]: pgmap v9094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16109.757401 secs 2024-04-26T16:19:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16106.939284 secs 2024-04-26T16:19:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:29 smithi096 ceph-mon[32753]: pgmap v9094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16109.757401 secs 2024-04-26T16:19:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16106.939284 secs 2024-04-26T16:19:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:29 smithi194 ceph-mon[33017]: pgmap v9094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16109.757401 secs 2024-04-26T16:19:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:29.630+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:29.630+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:19:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:29.630+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:31 smithi033 ceph-mon[27845]: pgmap v9095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:31 smithi096 ceph-mon[32753]: pgmap v9095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:31 smithi194 ceph-mon[33017]: pgmap v9095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:33 smithi033 ceph-mon[27845]: pgmap v9096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:33 smithi096 ceph-mon[32753]: pgmap v9096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:33 smithi194 ceph-mon[33017]: pgmap v9096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16111.939369 secs 2024-04-26T16:19:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16114.757514 secs 2024-04-26T16:19:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16111.939369 secs 2024-04-26T16:19:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16114.757514 secs 2024-04-26T16:19:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16111.939369 secs 2024-04-26T16:19:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16114.757514 secs 2024-04-26T16:19:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:35 smithi033 ceph-mon[27845]: pgmap v9097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:35 smithi096 ceph-mon[32753]: pgmap v9097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:35 smithi194 ceph-mon[33017]: pgmap v9097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:37 smithi096 ceph-mon[32753]: pgmap v9098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:37 smithi194 ceph-mon[33017]: pgmap v9098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:37 smithi033 ceph-mon[27845]: pgmap v9098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:19:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16116.939453 secs 2024-04-26T16:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:39 smithi096 ceph-mon[32753]: pgmap v9099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16119.757632 secs 2024-04-26T16:19:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16116.939453 secs 2024-04-26T16:19:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:39 smithi194 ceph-mon[33017]: pgmap v9099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16119.757632 secs 2024-04-26T16:19:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16116.939453 secs 2024-04-26T16:19:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:39 smithi033 ceph-mon[27845]: pgmap v9099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16119.757632 secs 2024-04-26T16:19:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:41 smithi194 ceph-mon[33017]: pgmap v9100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:41 smithi033 ceph-mon[27845]: pgmap v9100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:41 smithi096 ceph-mon[32753]: pgmap v9100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:43.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:43 smithi096 ceph-mon[32753]: pgmap v9101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:43 smithi194 ceph-mon[33017]: pgmap v9101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:43 smithi033 ceph-mon[27845]: pgmap v9101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16121.939513 secs 2024-04-26T16:19:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16124.757778 secs 2024-04-26T16:19:44.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:44.645+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:44.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:44.645+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:19:44.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:44.645+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16121.939513 secs 2024-04-26T16:19:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16124.757778 secs 2024-04-26T16:19:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16121.939513 secs 2024-04-26T16:19:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16124.757778 secs 2024-04-26T16:19:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:45 smithi194 ceph-mon[33017]: pgmap v9102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:45 smithi033 ceph-mon[27845]: pgmap v9102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:45 smithi096 ceph-mon[32753]: pgmap v9102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:47 smithi194 ceph-mon[33017]: pgmap v9103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:47 smithi033 ceph-mon[27845]: pgmap v9103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:47 smithi096 ceph-mon[32753]: pgmap v9103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16126.939596 secs 2024-04-26T16:19:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:49 smithi194 ceph-mon[33017]: pgmap v9104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16129.757904 secs 2024-04-26T16:19:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16126.939596 secs 2024-04-26T16:19:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:49 smithi033 ceph-mon[27845]: pgmap v9104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16129.757904 secs 2024-04-26T16:19:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16126.939596 secs 2024-04-26T16:19:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:49 smithi096 ceph-mon[32753]: pgmap v9104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:19:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16129.757904 secs 2024-04-26T16:19:51.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:51 smithi033 ceph-mon[27845]: pgmap v9105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:51 smithi194 ceph-mon[33017]: pgmap v9105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:51 smithi096 ceph-mon[32753]: pgmap v9105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:19:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:53 smithi194 ceph-mon[33017]: pgmap v9106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16131.939687 secs 2024-04-26T16:19:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:53 smithi033 ceph-mon[27845]: pgmap v9106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16131.939687 secs 2024-04-26T16:19:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:53 smithi096 ceph-mon[32753]: pgmap v9106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16131.939687 secs 2024-04-26T16:19:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:54.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:54.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16134.758024 secs 2024-04-26T16:19:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16134.758024 secs 2024-04-26T16:19:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16134.758024 secs 2024-04-26T16:19:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:55 smithi194 ceph-mon[33017]: pgmap v9107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:55 smithi033 ceph-mon[27845]: pgmap v9107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:55 smithi096 ceph-mon[32753]: pgmap v9107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:19:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:19:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:56 smithi194 ceph-mon[33017]: pgmap v9108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T16:19:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:56 smithi194 ceph-mon[33017]: pgmap v9109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:19:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:56 smithi033 ceph-mon[27845]: pgmap v9108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T16:19:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:56 smithi033 ceph-mon[27845]: pgmap v9109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:19:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:56 smithi096 ceph-mon[32753]: pgmap v9108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T16:19:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:56 smithi096 ceph-mon[32753]: pgmap v9109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:19:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:58 smithi194 ceph-mon[33017]: pgmap v9110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:19:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16136.939767 secs 2024-04-26T16:19:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:58 smithi033 ceph-mon[27845]: pgmap v9110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:19:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16136.939767 secs 2024-04-26T16:19:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:58 smithi096 ceph-mon[32753]: pgmap v9110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:19:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16136.939767 secs 2024-04-26T16:19:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:19:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16139.758171 secs 2024-04-26T16:19:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:59.660+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:59.660+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:19:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:19:59.660+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:19:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:19:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16139.758171 secs 2024-04-26T16:19:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:19:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16139.758171 secs 2024-04-26T16:20:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:00 smithi194 ceph-mon[33017]: pgmap v9111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:20:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:20:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:00 smithi033 ceph-mon[27845]: pgmap v9111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:20:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:20:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:00 smithi096 ceph-mon[32753]: pgmap v9111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:20:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:20:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:02 smithi194 ceph-mon[33017]: pgmap v9112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:20:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:02 smithi033 ceph-mon[27845]: pgmap v9112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:20:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:02 smithi096 ceph-mon[32753]: pgmap v9112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:20:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16141.939851 secs 2024-04-26T16:20:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16141.939851 secs 2024-04-26T16:20:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16141.939851 secs 2024-04-26T16:20:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:04 smithi194 ceph-mon[33017]: pgmap v9113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:20:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16144.758285 secs 2024-04-26T16:20:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:04 smithi033 ceph-mon[27845]: pgmap v9113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:20:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16144.758285 secs 2024-04-26T16:20:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:04 smithi096 ceph-mon[32753]: pgmap v9113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:20:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16144.758285 secs 2024-04-26T16:20:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:06 smithi194 ceph-mon[33017]: pgmap v9114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:20:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:06 smithi033 ceph-mon[27845]: pgmap v9114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:20:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:06 smithi096 ceph-mon[32753]: pgmap v9114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:20:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:08 smithi194 ceph-mon[33017]: pgmap v9115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16146.939911 secs 2024-04-26T16:20:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:08 smithi033 ceph-mon[27845]: pgmap v9115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16146.939911 secs 2024-04-26T16:20:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:08 smithi096 ceph-mon[32753]: pgmap v9115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16146.939911 secs 2024-04-26T16:20:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16149.758400 secs 2024-04-26T16:20:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16149.758400 secs 2024-04-26T16:20:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16149.758400 secs 2024-04-26T16:20:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:10 smithi194 ceph-mon[33017]: pgmap v9116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:10 smithi033 ceph-mon[27845]: pgmap v9116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:10 smithi096 ceph-mon[32753]: pgmap v9116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:12 smithi194 ceph-mon[33017]: pgmap v9117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:12 smithi033 ceph-mon[27845]: pgmap v9117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:12 smithi096 ceph-mon[32753]: pgmap v9117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16151.940016 secs 2024-04-26T16:20:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16151.940016 secs 2024-04-26T16:20:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16151.940016 secs 2024-04-26T16:20:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:14 smithi194 ceph-mon[33017]: pgmap v9118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16154.758516 secs 2024-04-26T16:20:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:14.670+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:14.670+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:14.670+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:20:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:14 smithi033 ceph-mon[27845]: pgmap v9118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16154.758516 secs 2024-04-26T16:20:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:14 smithi096 ceph-mon[32753]: pgmap v9118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16154.758516 secs 2024-04-26T16:20:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:16 smithi194 ceph-mon[33017]: pgmap v9119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:16 smithi033 ceph-mon[27845]: pgmap v9119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:16 smithi096 ceph-mon[32753]: pgmap v9119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:18 smithi194 ceph-mon[33017]: pgmap v9120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16156.940077 secs 2024-04-26T16:20:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:18 smithi033 ceph-mon[27845]: pgmap v9120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16156.940077 secs 2024-04-26T16:20:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:18 smithi096 ceph-mon[32753]: pgmap v9120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16156.940077 secs 2024-04-26T16:20:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16159.758634 secs 2024-04-26T16:20:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16159.758634 secs 2024-04-26T16:20:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16159.758634 secs 2024-04-26T16:20:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:20 smithi033 ceph-mon[27845]: pgmap v9121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:20 smithi096 ceph-mon[32753]: pgmap v9121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:20 smithi194 ceph-mon[33017]: pgmap v9121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:22 smithi033 ceph-mon[27845]: pgmap v9122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:22 smithi096 ceph-mon[32753]: pgmap v9122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:22 smithi194 ceph-mon[33017]: pgmap v9122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16161.940166 secs 2024-04-26T16:20:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16161.940166 secs 2024-04-26T16:20:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16161.940166 secs 2024-04-26T16:20:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:24.543+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:20:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:24.544+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:20:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:24.544+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:20:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:24.544+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:20:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:24 smithi033 ceph-mon[27845]: pgmap v9123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16164.758760 secs 2024-04-26T16:20:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:24 smithi096 ceph-mon[32753]: pgmap v9123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16164.758760 secs 2024-04-26T16:20:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:24 smithi194 ceph-mon[33017]: pgmap v9123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16164.758760 secs 2024-04-26T16:20:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:26 smithi033 ceph-mon[27845]: pgmap v9124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:26 smithi096 ceph-mon[32753]: pgmap v9124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:26 smithi194 ceph-mon[33017]: pgmap v9124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:28 smithi033 ceph-mon[27845]: pgmap v9125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16166.940251 secs 2024-04-26T16:20:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:28 smithi096 ceph-mon[32753]: pgmap v9125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16166.940251 secs 2024-04-26T16:20:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:28 smithi194 ceph-mon[33017]: pgmap v9125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16166.940251 secs 2024-04-26T16:20:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16169.758903 secs 2024-04-26T16:20:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16169.758903 secs 2024-04-26T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:29.685+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:29.685+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:29.685+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16169.758903 secs 2024-04-26T16:20:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:30 smithi033 ceph-mon[27845]: pgmap v9126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:30 smithi096 ceph-mon[32753]: pgmap v9126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:30 smithi194 ceph-mon[33017]: pgmap v9126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:32 smithi033 ceph-mon[27845]: pgmap v9127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:32 smithi096 ceph-mon[32753]: pgmap v9127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:32 smithi194 ceph-mon[33017]: pgmap v9127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16171.940366 secs 2024-04-26T16:20:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16171.940366 secs 2024-04-26T16:20:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16171.940366 secs 2024-04-26T16:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:34 smithi194 ceph-mon[33017]: pgmap v9128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16174.759079 secs 2024-04-26T16:20:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:34 smithi033 ceph-mon[27845]: pgmap v9128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16174.759079 secs 2024-04-26T16:20:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:34 smithi096 ceph-mon[32753]: pgmap v9128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16174.759079 secs 2024-04-26T16:20:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:36 smithi194 ceph-mon[33017]: pgmap v9129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:36 smithi033 ceph-mon[27845]: pgmap v9129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:36 smithi096 ceph-mon[32753]: pgmap v9129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:38 smithi033 ceph-mon[27845]: pgmap v9130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16176.940444 secs 2024-04-26T16:20:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:38 smithi194 ceph-mon[33017]: pgmap v9130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16176.940444 secs 2024-04-26T16:20:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:38 smithi096 ceph-mon[32753]: pgmap v9130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16176.940444 secs 2024-04-26T16:20:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16179.759200 secs 2024-04-26T16:20:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16179.759200 secs 2024-04-26T16:20:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16179.759200 secs 2024-04-26T16:20:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:40 smithi194 ceph-mon[33017]: pgmap v9131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:40 smithi033 ceph-mon[27845]: pgmap v9131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:40 smithi096 ceph-mon[32753]: pgmap v9131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:42 smithi194 ceph-mon[33017]: pgmap v9132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:42 smithi033 ceph-mon[27845]: pgmap v9132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:42 smithi096 ceph-mon[32753]: pgmap v9132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:43.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16181.940525 secs 2024-04-26T16:20:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16181.940525 secs 2024-04-26T16:20:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16181.940525 secs 2024-04-26T16:20:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:44.701+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:44.701+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:20:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:44.701+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:44 smithi194 ceph-mon[33017]: pgmap v9133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16184.759308 secs 2024-04-26T16:20:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:44 smithi033 ceph-mon[27845]: pgmap v9133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16184.759308 secs 2024-04-26T16:20:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:44 smithi096 ceph-mon[32753]: pgmap v9133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16184.759308 secs 2024-04-26T16:20:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:46 smithi194 ceph-mon[33017]: pgmap v9134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:46 smithi033 ceph-mon[27845]: pgmap v9134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:46 smithi096 ceph-mon[32753]: pgmap v9134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:48 smithi194 ceph-mon[33017]: pgmap v9135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16186.940607 secs 2024-04-26T16:20:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:48 smithi033 ceph-mon[27845]: pgmap v9135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16186.940607 secs 2024-04-26T16:20:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:48 smithi096 ceph-mon[32753]: pgmap v9135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16186.940607 secs 2024-04-26T16:20:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16189.759420 secs 2024-04-26T16:20:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16189.759420 secs 2024-04-26T16:20:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16189.759420 secs 2024-04-26T16:20:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:50 smithi194 ceph-mon[33017]: pgmap v9136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:50 smithi033 ceph-mon[27845]: pgmap v9136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:50 smithi096 ceph-mon[32753]: pgmap v9136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:51 smithi194 ceph-mon[33017]: pgmap v9137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:51 smithi033 ceph-mon[27845]: pgmap v9137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:51 smithi096 ceph-mon[32753]: pgmap v9137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:53 smithi033 ceph-mon[27845]: pgmap v9138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16191.940690 secs 2024-04-26T16:20:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16194.759513 secs 2024-04-26T16:20:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:53 smithi096 ceph-mon[32753]: pgmap v9138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16191.940690 secs 2024-04-26T16:20:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16194.759513 secs 2024-04-26T16:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:53 smithi194 ceph-mon[33017]: pgmap v9138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16191.940690 secs 2024-04-26T16:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16194.759513 secs 2024-04-26T16:20:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:55 smithi033 ceph-mon[27845]: pgmap v9139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:55 smithi096 ceph-mon[32753]: pgmap v9139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:55 smithi194 ceph-mon[33017]: pgmap v9139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:20:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:56 smithi033 ceph-mon[27845]: pgmap v9140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:56 smithi033 ceph-mon[27845]: pgmap v9141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:20:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:20:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:56 smithi096 ceph-mon[32753]: pgmap v9140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:56 smithi096 ceph-mon[32753]: pgmap v9141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:20:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:20:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:56 smithi194 ceph-mon[33017]: pgmap v9140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:20:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:56 smithi194 ceph-mon[33017]: pgmap v9141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:20:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:20:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16196.940771 secs 2024-04-26T16:20:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16196.940771 secs 2024-04-26T16:20:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16196.940771 secs 2024-04-26T16:20:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:59 smithi194 ceph-mon[33017]: pgmap v9142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:20:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:20:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16199.759657 secs 2024-04-26T16:20:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:59 smithi033 ceph-mon[27845]: pgmap v9142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:20:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:20:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16199.759657 secs 2024-04-26T16:20:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:59 smithi096 ceph-mon[32753]: pgmap v9142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:20:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:20:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16199.759657 secs 2024-04-26T16:20:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:59.715+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:59.715+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:20:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:20:59.715+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:21:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:01 smithi033 ceph-mon[27845]: pgmap v9143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:21:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:01 smithi096 ceph-mon[32753]: pgmap v9143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:21:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:01 smithi194 ceph-mon[33017]: pgmap v9143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:21:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:03 smithi033 ceph-mon[27845]: pgmap v9144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:21:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16201.940856 secs 2024-04-26T16:21:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:03 smithi096 ceph-mon[32753]: pgmap v9144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:21:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16201.940856 secs 2024-04-26T16:21:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:03 smithi194 ceph-mon[33017]: pgmap v9144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:21:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16201.940856 secs 2024-04-26T16:21:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16204.759791 secs 2024-04-26T16:21:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16204.759791 secs 2024-04-26T16:21:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16204.759791 secs 2024-04-26T16:21:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:05 smithi033 ceph-mon[27845]: pgmap v9145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:21:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:05 smithi096 ceph-mon[32753]: pgmap v9145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:21:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:05 smithi194 ceph-mon[33017]: pgmap v9145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:21:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:07 smithi033 ceph-mon[27845]: pgmap v9146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:21:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:07 smithi096 ceph-mon[32753]: pgmap v9146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:21:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:07 smithi194 ceph-mon[33017]: pgmap v9146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:21:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16206.940946 secs 2024-04-26T16:21:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16206.940946 secs 2024-04-26T16:21:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16206.940946 secs 2024-04-26T16:21:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:09 smithi033 ceph-mon[27845]: pgmap v9147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16209.759933 secs 2024-04-26T16:21:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:09 smithi096 ceph-mon[32753]: pgmap v9147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16209.759933 secs 2024-04-26T16:21:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:09 smithi194 ceph-mon[33017]: pgmap v9147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16209.759933 secs 2024-04-26T16:21:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:11 smithi033 ceph-mon[27845]: pgmap v9148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:11 smithi096 ceph-mon[32753]: pgmap v9148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:11 smithi194 ceph-mon[33017]: pgmap v9148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:13 smithi033 ceph-mon[27845]: pgmap v9149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16211.941007 secs 2024-04-26T16:21:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:13 smithi096 ceph-mon[32753]: pgmap v9149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16211.941007 secs 2024-04-26T16:21:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:13 smithi194 ceph-mon[33017]: pgmap v9149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16211.941007 secs 2024-04-26T16:21:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16214.760079 secs 2024-04-26T16:21:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16214.760079 secs 2024-04-26T16:21:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:14.722+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:14.722+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:21:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:14.722+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16214.760079 secs 2024-04-26T16:21:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:15 smithi033 ceph-mon[27845]: pgmap v9150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:15 smithi096 ceph-mon[32753]: pgmap v9150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:15 smithi194 ceph-mon[33017]: pgmap v9150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:17 smithi033 ceph-mon[27845]: pgmap v9151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:17 smithi096 ceph-mon[32753]: pgmap v9151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:17 smithi194 ceph-mon[33017]: pgmap v9151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16216.941069 secs 2024-04-26T16:21:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:18 smithi194 ceph-mon[33017]: pgmap v9152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16219.760200 secs 2024-04-26T16:21:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16216.941069 secs 2024-04-26T16:21:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:18 smithi033 ceph-mon[27845]: pgmap v9152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16219.760200 secs 2024-04-26T16:21:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16216.941069 secs 2024-04-26T16:21:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:18 smithi096 ceph-mon[32753]: pgmap v9152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16219.760200 secs 2024-04-26T16:21:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:21 smithi033 ceph-mon[27845]: pgmap v9153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:21 smithi096 ceph-mon[32753]: pgmap v9153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:21 smithi194 ceph-mon[33017]: pgmap v9153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:23 smithi033 ceph-mon[27845]: pgmap v9154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16221.941153 secs 2024-04-26T16:21:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:23 smithi096 ceph-mon[32753]: pgmap v9154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16221.941153 secs 2024-04-26T16:21:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:23 smithi194 ceph-mon[33017]: pgmap v9154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16221.941153 secs 2024-04-26T16:21:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16224.760317 secs 2024-04-26T16:21:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16224.760317 secs 2024-04-26T16:21:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16224.760317 secs 2024-04-26T16:21:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:25 smithi033 ceph-mon[27845]: pgmap v9155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:25 smithi096 ceph-mon[32753]: pgmap v9155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:25 smithi194 ceph-mon[33017]: pgmap v9155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:27 smithi033 ceph-mon[27845]: pgmap v9156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:27 smithi096 ceph-mon[32753]: pgmap v9156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:27 smithi194 ceph-mon[33017]: pgmap v9156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16226.941257 secs 2024-04-26T16:21:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16226.941257 secs 2024-04-26T16:21:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16226.941257 secs 2024-04-26T16:21:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:29 smithi033 ceph-mon[27845]: pgmap v9157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16229.760379 secs 2024-04-26T16:21:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:29 smithi096 ceph-mon[32753]: pgmap v9157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16229.760379 secs 2024-04-26T16:21:29.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:29.738+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:29.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:29.739+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:29.739+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:21:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:29 smithi194 ceph-mon[33017]: pgmap v9157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16229.760379 secs 2024-04-26T16:21:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:31 smithi033 ceph-mon[27845]: pgmap v9158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:31 smithi096 ceph-mon[32753]: pgmap v9158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:31 smithi194 ceph-mon[33017]: pgmap v9158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:33 smithi033 ceph-mon[27845]: pgmap v9159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16231.941341 secs 2024-04-26T16:21:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:33 smithi096 ceph-mon[32753]: pgmap v9159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16231.941341 secs 2024-04-26T16:21:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:33 smithi194 ceph-mon[33017]: pgmap v9159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16231.941341 secs 2024-04-26T16:21:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16234.760494 secs 2024-04-26T16:21:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16234.760494 secs 2024-04-26T16:21:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16234.760494 secs 2024-04-26T16:21:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:35 smithi096 ceph-mon[32753]: pgmap v9160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:35 smithi194 ceph-mon[33017]: pgmap v9160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:35 smithi033 ceph-mon[27845]: pgmap v9160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:37 smithi096 ceph-mon[32753]: pgmap v9161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:37 smithi194 ceph-mon[33017]: pgmap v9161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:37 smithi033 ceph-mon[27845]: pgmap v9161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16236.941425 secs 2024-04-26T16:21:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16236.941425 secs 2024-04-26T16:21:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16236.941425 secs 2024-04-26T16:21:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:39 smithi194 ceph-mon[33017]: pgmap v9162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16239.760613 secs 2024-04-26T16:21:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:39 smithi033 ceph-mon[27845]: pgmap v9162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16239.760613 secs 2024-04-26T16:21:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:39 smithi096 ceph-mon[32753]: pgmap v9162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16239.760613 secs 2024-04-26T16:21:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:41 smithi194 ceph-mon[33017]: pgmap v9163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:41 smithi033 ceph-mon[27845]: pgmap v9163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:41 smithi096 ceph-mon[32753]: pgmap v9163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:43.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:43 smithi096 ceph-mon[32753]: pgmap v9164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:43.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16241.941508 secs 2024-04-26T16:21:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:43 smithi194 ceph-mon[33017]: pgmap v9164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16241.941508 secs 2024-04-26T16:21:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:43 smithi033 ceph-mon[27845]: pgmap v9164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16241.941508 secs 2024-04-26T16:21:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:44.746+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:44.746+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:21:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:44.746+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16244.760723 secs 2024-04-26T16:21:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16244.760723 secs 2024-04-26T16:21:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16244.760723 secs 2024-04-26T16:21:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:45 smithi194 ceph-mon[33017]: pgmap v9165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:45 smithi033 ceph-mon[27845]: pgmap v9165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:45 smithi096 ceph-mon[32753]: pgmap v9165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:47 smithi194 ceph-mon[33017]: pgmap v9166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:47 smithi033 ceph-mon[27845]: pgmap v9166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:47 smithi096 ceph-mon[32753]: pgmap v9166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16246.941590 secs 2024-04-26T16:21:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16246.941590 secs 2024-04-26T16:21:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16246.941590 secs 2024-04-26T16:21:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:49 smithi194 ceph-mon[33017]: pgmap v9167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16249.760844 secs 2024-04-26T16:21:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:49 smithi033 ceph-mon[27845]: pgmap v9167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16249.760844 secs 2024-04-26T16:21:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:49 smithi096 ceph-mon[32753]: pgmap v9167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16249.760844 secs 2024-04-26T16:21:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:51 smithi194 ceph-mon[33017]: pgmap v9168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:51 smithi033 ceph-mon[27845]: pgmap v9168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:51 smithi096 ceph-mon[32753]: pgmap v9168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:53 smithi194 ceph-mon[33017]: pgmap v9169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16251.941672 secs 2024-04-26T16:21:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:53 smithi033 ceph-mon[27845]: pgmap v9169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16251.941672 secs 2024-04-26T16:21:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:53 smithi096 ceph-mon[32753]: pgmap v9169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16251.941672 secs 2024-04-26T16:21:54.558 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:54.558+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:21:54.558 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:54.558+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:21:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:54.559+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:21:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:54.559+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:21:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16254.760962 secs 2024-04-26T16:21:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16254.760962 secs 2024-04-26T16:21:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16254.760962 secs 2024-04-26T16:21:55.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:55 smithi033 ceph-mon[27845]: pgmap v9170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:55 smithi194 ceph-mon[33017]: pgmap v9170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:55 smithi096 ceph-mon[32753]: pgmap v9170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:21:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:56 smithi194 ceph-mon[33017]: pgmap v9171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:56 smithi033 ceph-mon[27845]: pgmap v9171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:56 smithi096 ceph-mon[32753]: pgmap v9171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:21:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16256.941740 secs 2024-04-26T16:21:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:58 smithi194 ceph-mon[33017]: pgmap v9172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:21:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16259.761024 secs 2024-04-26T16:21:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16256.941740 secs 2024-04-26T16:21:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:58 smithi033 ceph-mon[27845]: pgmap v9172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:21:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16259.761024 secs 2024-04-26T16:21:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16256.941740 secs 2024-04-26T16:21:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:58 smithi096 ceph-mon[32753]: pgmap v9172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:21:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:21:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16259.761024 secs 2024-04-26T16:21:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:59.761+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:59.761+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:21:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:21:59.761+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:22:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:22:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:22:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:22:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:01 smithi194 ceph-mon[33017]: pgmap v9173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:22:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:22:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:22:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:22:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:01 smithi033 ceph-mon[27845]: pgmap v9173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:22:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:22:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:22:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:22:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:01 smithi096 ceph-mon[32753]: pgmap v9173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: pgmap v9174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16261.941823 secs 2024-04-26T16:22:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: pgmap v9174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16261.941823 secs 2024-04-26T16:22:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: pgmap v9174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16261.941823 secs 2024-04-26T16:22:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:22:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:04 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:22:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:04 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:22:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:04 smithi194 ceph-mon[33017]: pgmap v9175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:22:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:04 smithi194 ceph-mon[33017]: pgmap v9176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:22:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16264.761145 secs 2024-04-26T16:22:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:04 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:22:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:04 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:22:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:04 smithi033 ceph-mon[27845]: pgmap v9175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:22:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:04 smithi033 ceph-mon[27845]: pgmap v9176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:22:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16264.761145 secs 2024-04-26T16:22:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:04 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:22:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:04 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:22:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:04 smithi096 ceph-mon[32753]: pgmap v9175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:22:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:04 smithi096 ceph-mon[32753]: pgmap v9176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:22:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16264.761145 secs 2024-04-26T16:22:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:06 smithi194 ceph-mon[33017]: pgmap v9177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:06 smithi033 ceph-mon[27845]: pgmap v9177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:06 smithi096 ceph-mon[32753]: pgmap v9177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:08 smithi194 ceph-mon[33017]: pgmap v9178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16266.941907 secs 2024-04-26T16:22:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:08 smithi033 ceph-mon[27845]: pgmap v9178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16266.941907 secs 2024-04-26T16:22:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:08 smithi096 ceph-mon[32753]: pgmap v9178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16266.941907 secs 2024-04-26T16:22:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16269.761276 secs 2024-04-26T16:22:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16269.761276 secs 2024-04-26T16:22:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16269.761276 secs 2024-04-26T16:22:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:10 smithi194 ceph-mon[33017]: pgmap v9179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:22:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:10 smithi033 ceph-mon[27845]: pgmap v9179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:22:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:10 smithi096 ceph-mon[32753]: pgmap v9179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:22:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:12 smithi194 ceph-mon[33017]: pgmap v9180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:12 smithi033 ceph-mon[27845]: pgmap v9180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:12 smithi096 ceph-mon[32753]: pgmap v9180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:22:13.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16271.942009 secs 2024-04-26T16:22:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16271.942009 secs 2024-04-26T16:22:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16271.942009 secs 2024-04-26T16:22:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:14 smithi194 ceph-mon[33017]: pgmap v9181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:22:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16274.761340 secs 2024-04-26T16:22:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:14 smithi033 ceph-mon[27845]: pgmap v9181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:22:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16274.761340 secs 2024-04-26T16:22:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:14 smithi096 ceph-mon[32753]: pgmap v9181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:22:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16274.761340 secs 2024-04-26T16:22:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:14.775+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:14.775+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:14.775+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:22:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:16 smithi194 ceph-mon[33017]: pgmap v9182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:16 smithi033 ceph-mon[27845]: pgmap v9182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:16 smithi096 ceph-mon[32753]: pgmap v9182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:18 smithi194 ceph-mon[33017]: pgmap v9183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16276.942093 secs 2024-04-26T16:22:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:18 smithi033 ceph-mon[27845]: pgmap v9183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16276.942093 secs 2024-04-26T16:22:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:18 smithi096 ceph-mon[32753]: pgmap v9183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16276.942093 secs 2024-04-26T16:22:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16279.761459 secs 2024-04-26T16:22:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16279.761459 secs 2024-04-26T16:22:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16279.761459 secs 2024-04-26T16:22:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:20 smithi194 ceph-mon[33017]: pgmap v9184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:20 smithi033 ceph-mon[27845]: pgmap v9184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:20 smithi096 ceph-mon[32753]: pgmap v9184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:22 smithi033 ceph-mon[27845]: pgmap v9185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:22 smithi096 ceph-mon[32753]: pgmap v9185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:22 smithi194 ceph-mon[33017]: pgmap v9185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16281.942156 secs 2024-04-26T16:22:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16281.942156 secs 2024-04-26T16:22:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16281.942156 secs 2024-04-26T16:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:24 smithi033 ceph-mon[27845]: pgmap v9186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16284.761582 secs 2024-04-26T16:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:24 smithi096 ceph-mon[32753]: pgmap v9186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16284.761582 secs 2024-04-26T16:22:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:24 smithi194 ceph-mon[33017]: pgmap v9186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16284.761582 secs 2024-04-26T16:22:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:26 smithi194 ceph-mon[33017]: pgmap v9187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:26 smithi033 ceph-mon[27845]: pgmap v9187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:26 smithi096 ceph-mon[32753]: pgmap v9187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:28 smithi033 ceph-mon[27845]: pgmap v9188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16286.942241 secs 2024-04-26T16:22:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:28 smithi096 ceph-mon[32753]: pgmap v9188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16286.942241 secs 2024-04-26T16:22:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:28 smithi194 ceph-mon[33017]: pgmap v9188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16286.942241 secs 2024-04-26T16:22:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16289.761708 secs 2024-04-26T16:22:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16289.761708 secs 2024-04-26T16:22:29.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:29.779+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:29.779+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:22:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:29.779+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16289.761708 secs 2024-04-26T16:22:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:30 smithi033 ceph-mon[27845]: pgmap v9189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:30 smithi096 ceph-mon[32753]: pgmap v9189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:30 smithi194 ceph-mon[33017]: pgmap v9189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:32 smithi033 ceph-mon[27845]: pgmap v9190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:32 smithi096 ceph-mon[32753]: pgmap v9190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:32 smithi194 ceph-mon[33017]: pgmap v9190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16291.942302 secs 2024-04-26T16:22:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16291.942302 secs 2024-04-26T16:22:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16291.942302 secs 2024-04-26T16:22:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:34 smithi033 ceph-mon[27845]: pgmap v9191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16294.761838 secs 2024-04-26T16:22:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:34 smithi096 ceph-mon[32753]: pgmap v9191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16294.761838 secs 2024-04-26T16:22:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:34 smithi194 ceph-mon[33017]: pgmap v9191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16294.761838 secs 2024-04-26T16:22:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:36 smithi033 ceph-mon[27845]: pgmap v9192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:36 smithi096 ceph-mon[32753]: pgmap v9192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:36 smithi194 ceph-mon[33017]: pgmap v9192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:38 smithi033 ceph-mon[27845]: pgmap v9193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16296.942387 secs 2024-04-26T16:22:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:38 smithi096 ceph-mon[32753]: pgmap v9193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16296.942387 secs 2024-04-26T16:22:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:38 smithi194 ceph-mon[33017]: pgmap v9193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16296.942387 secs 2024-04-26T16:22:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16299.761925 secs 2024-04-26T16:22:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16299.761925 secs 2024-04-26T16:22:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16299.761925 secs 2024-04-26T16:22:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:40 smithi194 ceph-mon[33017]: pgmap v9194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:40 smithi033 ceph-mon[27845]: pgmap v9194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:40 smithi096 ceph-mon[32753]: pgmap v9194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:42 smithi033 ceph-mon[27845]: pgmap v9195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:42 smithi194 ceph-mon[33017]: pgmap v9195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:42 smithi096 ceph-mon[32753]: pgmap v9195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:43.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16301.942470 secs 2024-04-26T16:22:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16301.942470 secs 2024-04-26T16:22:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16301.942470 secs 2024-04-26T16:22:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:44 smithi194 ceph-mon[33017]: pgmap v9196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16304.762040 secs 2024-04-26T16:22:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:44.790+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:44.790+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:44.790+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:22:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:44 smithi033 ceph-mon[27845]: pgmap v9196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16304.762040 secs 2024-04-26T16:22:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:44 smithi096 ceph-mon[32753]: pgmap v9196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16304.762040 secs 2024-04-26T16:22:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:46 smithi194 ceph-mon[33017]: pgmap v9197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:46 smithi033 ceph-mon[27845]: pgmap v9197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:46 smithi096 ceph-mon[32753]: pgmap v9197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:48 smithi194 ceph-mon[33017]: pgmap v9198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16306.942554 secs 2024-04-26T16:22:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:48 smithi033 ceph-mon[27845]: pgmap v9198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16306.942554 secs 2024-04-26T16:22:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:48 smithi096 ceph-mon[32753]: pgmap v9198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16306.942554 secs 2024-04-26T16:22:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16309.762155 secs 2024-04-26T16:22:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16309.762155 secs 2024-04-26T16:22:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16309.762155 secs 2024-04-26T16:22:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:50 smithi194 ceph-mon[33017]: pgmap v9199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:50 smithi033 ceph-mon[27845]: pgmap v9199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:50 smithi096 ceph-mon[32753]: pgmap v9199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:52 smithi194 ceph-mon[33017]: pgmap v9200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:52 smithi033 ceph-mon[27845]: pgmap v9200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:52 smithi096 ceph-mon[32753]: pgmap v9200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16311.942633 secs 2024-04-26T16:22:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16311.942633 secs 2024-04-26T16:22:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16311.942633 secs 2024-04-26T16:22:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:54 smithi194 ceph-mon[33017]: pgmap v9201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16314.762216 secs 2024-04-26T16:22:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:54 smithi033 ceph-mon[27845]: pgmap v9201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16314.762216 secs 2024-04-26T16:22:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:54 smithi096 ceph-mon[32753]: pgmap v9201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16314.762216 secs 2024-04-26T16:22:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:56 smithi194 ceph-mon[33017]: pgmap v9202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:56 smithi033 ceph-mon[27845]: pgmap v9202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:56 smithi096 ceph-mon[32753]: pgmap v9202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:22:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:58 smithi194 ceph-mon[33017]: pgmap v9203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16316.942719 secs 2024-04-26T16:22:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:58 smithi033 ceph-mon[27845]: pgmap v9203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16316.942719 secs 2024-04-26T16:22:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:58 smithi096 ceph-mon[32753]: pgmap v9203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:22:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16316.942719 secs 2024-04-26T16:22:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:59.805+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:59.805+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:22:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:22:59.805+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:22:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16319.762350 secs 2024-04-26T16:22:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16319.762350 secs 2024-04-26T16:22:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16319.762350 secs 2024-04-26T16:23:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:00 smithi194 ceph-mon[33017]: pgmap v9204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:00 smithi033 ceph-mon[27845]: pgmap v9204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:00 smithi096 ceph-mon[32753]: pgmap v9204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:01 smithi194 ceph-mon[33017]: pgmap v9205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:01 smithi033 ceph-mon[27845]: pgmap v9205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:01 smithi096 ceph-mon[32753]: pgmap v9205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16321.942800 secs 2024-04-26T16:23:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:03 smithi194 ceph-mon[33017]: pgmap v9206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16324.762450 secs 2024-04-26T16:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16321.942800 secs 2024-04-26T16:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:03 smithi033 ceph-mon[27845]: pgmap v9206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16324.762450 secs 2024-04-26T16:23:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16321.942800 secs 2024-04-26T16:23:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:03 smithi096 ceph-mon[32753]: pgmap v9206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16324.762450 secs 2024-04-26T16:23:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:04 smithi194 ceph-mon[33017]: pgmap v9207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:23:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:04 smithi194 ceph-mon[33017]: pgmap v9208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:23:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:04 smithi033 ceph-mon[27845]: pgmap v9207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:23:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:04 smithi033 ceph-mon[27845]: pgmap v9208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:23:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:04 smithi096 ceph-mon[32753]: pgmap v9207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:23:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:04 smithi096 ceph-mon[32753]: pgmap v9208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:23:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:07 smithi096 ceph-mon[32753]: pgmap v9209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:07 smithi194 ceph-mon[33017]: pgmap v9209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:07 smithi033 ceph-mon[27845]: pgmap v9209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16326.942884 secs 2024-04-26T16:23:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16326.942884 secs 2024-04-26T16:23:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16326.942884 secs 2024-04-26T16:23:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16329.762591 secs 2024-04-26T16:23:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:09 smithi194 ceph-mon[33017]: pgmap v9210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16329.762591 secs 2024-04-26T16:23:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:09 smithi033 ceph-mon[27845]: pgmap v9210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16329.762591 secs 2024-04-26T16:23:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:09 smithi096 ceph-mon[32753]: pgmap v9210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:11 smithi194 ceph-mon[33017]: pgmap v9211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:11 smithi033 ceph-mon[27845]: pgmap v9211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:11 smithi096 ceph-mon[32753]: pgmap v9211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:13.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:13 smithi096 ceph-mon[32753]: pgmap v9212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:13.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16331.942971 secs 2024-04-26T16:23:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:13 smithi194 ceph-mon[33017]: pgmap v9212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16331.942971 secs 2024-04-26T16:23:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:13 smithi033 ceph-mon[27845]: pgmap v9212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:23:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16331.942971 secs 2024-04-26T16:23:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16334.762719 secs 2024-04-26T16:23:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:14.815+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:14.815+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:14.815+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:23:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16334.762719 secs 2024-04-26T16:23:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16334.762719 secs 2024-04-26T16:23:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:15 smithi194 ceph-mon[33017]: pgmap v9213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:23:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:15 smithi033 ceph-mon[27845]: pgmap v9213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:23:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:15 smithi096 ceph-mon[32753]: pgmap v9213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:23:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:17 smithi194 ceph-mon[33017]: pgmap v9214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:17 smithi033 ceph-mon[27845]: pgmap v9214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:17 smithi096 ceph-mon[32753]: pgmap v9214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16336.943025 secs 2024-04-26T16:23:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16336.943025 secs 2024-04-26T16:23:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16336.943025 secs 2024-04-26T16:23:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16339.762863 secs 2024-04-26T16:23:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:19 smithi194 ceph-mon[33017]: pgmap v9215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16339.762863 secs 2024-04-26T16:23:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:19 smithi033 ceph-mon[27845]: pgmap v9215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16339.762863 secs 2024-04-26T16:23:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:19 smithi096 ceph-mon[32753]: pgmap v9215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:21 smithi194 ceph-mon[33017]: pgmap v9216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:21 smithi033 ceph-mon[27845]: pgmap v9216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:21 smithi096 ceph-mon[32753]: pgmap v9216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:23 smithi194 ceph-mon[33017]: pgmap v9217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16341.943083 secs 2024-04-26T16:23:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:23 smithi033 ceph-mon[27845]: pgmap v9217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16341.943083 secs 2024-04-26T16:23:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:23 smithi096 ceph-mon[32753]: pgmap v9217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16341.943083 secs 2024-04-26T16:23:24.573 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:24.573+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:23:24.573 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:24.573+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T16:23:24.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:24.574+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:23:24.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:24.574+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:23:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16344.762978 secs 2024-04-26T16:23:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16344.762978 secs 2024-04-26T16:23:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16344.762978 secs 2024-04-26T16:23:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:25 smithi194 ceph-mon[33017]: pgmap v9218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:25 smithi033 ceph-mon[27845]: pgmap v9218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:25 smithi096 ceph-mon[32753]: pgmap v9218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:27 smithi194 ceph-mon[33017]: pgmap v9219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:27 smithi033 ceph-mon[27845]: pgmap v9219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:27 smithi096 ceph-mon[32753]: pgmap v9219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16346.943143 secs 2024-04-26T16:23:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16346.943143 secs 2024-04-26T16:23:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16346.943143 secs 2024-04-26T16:23:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16349.763043 secs 2024-04-26T16:23:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:29 smithi033 ceph-mon[27845]: pgmap v9220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:29.824+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:29.824+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:29.824+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:23:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16349.763043 secs 2024-04-26T16:23:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:29 smithi194 ceph-mon[33017]: pgmap v9220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16349.763043 secs 2024-04-26T16:23:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:29 smithi096 ceph-mon[32753]: pgmap v9220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:30 smithi194 ceph-mon[33017]: pgmap v9221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:30 smithi033 ceph-mon[27845]: pgmap v9221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:30 smithi096 ceph-mon[32753]: pgmap v9221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:33 smithi194 ceph-mon[33017]: pgmap v9222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16351.943205 secs 2024-04-26T16:23:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:33 smithi033 ceph-mon[27845]: pgmap v9222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16351.943205 secs 2024-04-26T16:23:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:33 smithi096 ceph-mon[32753]: pgmap v9222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16351.943205 secs 2024-04-26T16:23:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16354.763158 secs 2024-04-26T16:23:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16354.763158 secs 2024-04-26T16:23:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16354.763158 secs 2024-04-26T16:23:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:35 smithi194 ceph-mon[33017]: pgmap v9223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:35 smithi033 ceph-mon[27845]: pgmap v9223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:35 smithi096 ceph-mon[32753]: pgmap v9223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:37 smithi194 ceph-mon[33017]: pgmap v9224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:37 smithi033 ceph-mon[27845]: pgmap v9224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:37 smithi096 ceph-mon[32753]: pgmap v9224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16356.943265 secs 2024-04-26T16:23:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16356.943265 secs 2024-04-26T16:23:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16356.943265 secs 2024-04-26T16:23:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16359.763223 secs 2024-04-26T16:23:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:39 smithi194 ceph-mon[33017]: pgmap v9225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16359.763223 secs 2024-04-26T16:23:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:39 smithi033 ceph-mon[27845]: pgmap v9225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16359.763223 secs 2024-04-26T16:23:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:39 smithi096 ceph-mon[32753]: pgmap v9225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:41 smithi194 ceph-mon[33017]: pgmap v9226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:41 smithi033 ceph-mon[27845]: pgmap v9226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:41 smithi096 ceph-mon[32753]: pgmap v9226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:43.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:43 smithi096 ceph-mon[32753]: pgmap v9227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:43.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16361.943349 secs 2024-04-26T16:23:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:43 smithi194 ceph-mon[33017]: pgmap v9227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16361.943349 secs 2024-04-26T16:23:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:43 smithi033 ceph-mon[27845]: pgmap v9227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16361.943349 secs 2024-04-26T16:23:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16364.763324 secs 2024-04-26T16:23:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:44.835+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:44.835+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:44.835+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:23:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16364.763324 secs 2024-04-26T16:23:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16364.763324 secs 2024-04-26T16:23:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:45 smithi194 ceph-mon[33017]: pgmap v9228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:45 smithi033 ceph-mon[27845]: pgmap v9228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:45 smithi096 ceph-mon[32753]: pgmap v9228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:47 smithi194 ceph-mon[33017]: pgmap v9229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:47 smithi033 ceph-mon[27845]: pgmap v9229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:47 smithi096 ceph-mon[32753]: pgmap v9229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16366.943432 secs 2024-04-26T16:23:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16366.943432 secs 2024-04-26T16:23:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16366.943432 secs 2024-04-26T16:23:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16369.763446 secs 2024-04-26T16:23:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:49 smithi194 ceph-mon[33017]: pgmap v9230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16369.763446 secs 2024-04-26T16:23:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:49 smithi033 ceph-mon[27845]: pgmap v9230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16369.763446 secs 2024-04-26T16:23:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:49 smithi096 ceph-mon[32753]: pgmap v9230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:51 smithi194 ceph-mon[33017]: pgmap v9231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:51 smithi033 ceph-mon[27845]: pgmap v9231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:51 smithi096 ceph-mon[32753]: pgmap v9231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:52 smithi194 ceph-mon[33017]: pgmap v9232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:52 smithi033 ceph-mon[27845]: pgmap v9232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:52 smithi096 ceph-mon[32753]: pgmap v9232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16371.943513 secs 2024-04-26T16:23:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16371.943513 secs 2024-04-26T16:23:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16371.943513 secs 2024-04-26T16:23:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16374.763561 secs 2024-04-26T16:23:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:54 smithi194 ceph-mon[33017]: pgmap v9233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16374.763561 secs 2024-04-26T16:23:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:54 smithi033 ceph-mon[27845]: pgmap v9233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16374.763561 secs 2024-04-26T16:23:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:54 smithi096 ceph-mon[32753]: pgmap v9233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:23:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:57 smithi194 ceph-mon[33017]: pgmap v9234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:57 smithi033 ceph-mon[27845]: pgmap v9234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:57 smithi096 ceph-mon[32753]: pgmap v9234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16376.943603 secs 2024-04-26T16:23:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16376.943603 secs 2024-04-26T16:23:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16376.943603 secs 2024-04-26T16:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:59.849+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:59.849+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:23:59.849+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:23:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16379.763684 secs 2024-04-26T16:23:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:23:59 smithi194 ceph-mon[33017]: pgmap v9235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16379.763684 secs 2024-04-26T16:23:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:23:59 smithi033 ceph-mon[27845]: pgmap v9235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:23:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16379.763684 secs 2024-04-26T16:23:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:23:59 smithi096 ceph-mon[32753]: pgmap v9235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:01 smithi194 ceph-mon[33017]: pgmap v9236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:01 smithi033 ceph-mon[27845]: pgmap v9236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:01 smithi096 ceph-mon[32753]: pgmap v9236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:03 smithi194 ceph-mon[33017]: pgmap v9237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16381.943687 secs 2024-04-26T16:24:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:03 smithi033 ceph-mon[27845]: pgmap v9237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16381.943687 secs 2024-04-26T16:24:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:03 smithi096 ceph-mon[32753]: pgmap v9237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16381.943687 secs 2024-04-26T16:24:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16384.763802 secs 2024-04-26T16:24:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16384.763802 secs 2024-04-26T16:24:04.948 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16384.763802 secs 2024-04-26T16:24:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:05 smithi194 ceph-mon[33017]: pgmap v9238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:05 smithi033 ceph-mon[27845]: pgmap v9238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:05 smithi096 ceph-mon[32753]: pgmap v9238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:24:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:24:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:24:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:07 smithi194 ceph-mon[33017]: pgmap v9239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:07 smithi194 ceph-mon[33017]: pgmap v9240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:07 smithi033 ceph-mon[27845]: pgmap v9239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:07 smithi033 ceph-mon[27845]: pgmap v9240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:07 smithi096 ceph-mon[32753]: pgmap v9239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:07 smithi096 ceph-mon[32753]: pgmap v9240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16386.943768 secs 2024-04-26T16:24:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16386.943768 secs 2024-04-26T16:24:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16386.943768 secs 2024-04-26T16:24:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:09 smithi194 ceph-mon[33017]: pgmap v9241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16389.763942 secs 2024-04-26T16:24:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:09 smithi033 ceph-mon[27845]: pgmap v9241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16389.763942 secs 2024-04-26T16:24:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:09 smithi096 ceph-mon[32753]: pgmap v9241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16389.763942 secs 2024-04-26T16:24:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:11 smithi194 ceph-mon[33017]: pgmap v9242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:11 smithi033 ceph-mon[27845]: pgmap v9242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:11 smithi096 ceph-mon[32753]: pgmap v9242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:13 smithi194 ceph-mon[33017]: pgmap v9243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16391.943847 secs 2024-04-26T16:24:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:13 smithi033 ceph-mon[27845]: pgmap v9243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16391.943847 secs 2024-04-26T16:24:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:13 smithi096 ceph-mon[32753]: pgmap v9243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:24:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16391.943847 secs 2024-04-26T16:24:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16394.764058 secs 2024-04-26T16:24:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:14.855+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:14.855+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:14.855+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:24:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16394.764058 secs 2024-04-26T16:24:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16394.764058 secs 2024-04-26T16:24:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:15 smithi194 ceph-mon[33017]: pgmap v9244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:24:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:15 smithi033 ceph-mon[27845]: pgmap v9244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:24:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:15 smithi096 ceph-mon[32753]: pgmap v9244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:24:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:16 smithi194 ceph-mon[33017]: pgmap v9245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:24:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:16 smithi033 ceph-mon[27845]: pgmap v9245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:24:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:16 smithi096 ceph-mon[32753]: pgmap v9245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:24:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16396.943928 secs 2024-04-26T16:24:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:19 smithi194 ceph-mon[33017]: pgmap v9246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16399.764200 secs 2024-04-26T16:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16396.943928 secs 2024-04-26T16:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:19 smithi033 ceph-mon[27845]: pgmap v9246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16399.764200 secs 2024-04-26T16:24:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16396.943928 secs 2024-04-26T16:24:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:19 smithi096 ceph-mon[32753]: pgmap v9246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16399.764200 secs 2024-04-26T16:24:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:21 smithi033 ceph-mon[27845]: pgmap v9247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:21 smithi096 ceph-mon[32753]: pgmap v9247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:21 smithi194 ceph-mon[33017]: pgmap v9247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:23 smithi033 ceph-mon[27845]: pgmap v9248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:23 smithi096 ceph-mon[32753]: pgmap v9248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:23 smithi194 ceph-mon[33017]: pgmap v9248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16401.943999 secs 2024-04-26T16:24:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16404.764261 secs 2024-04-26T16:24:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16401.943999 secs 2024-04-26T16:24:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16404.764261 secs 2024-04-26T16:24:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16401.943999 secs 2024-04-26T16:24:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16404.764261 secs 2024-04-26T16:24:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:25 smithi033 ceph-mon[27845]: pgmap v9249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:25 smithi096 ceph-mon[32753]: pgmap v9249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:25 smithi194 ceph-mon[33017]: pgmap v9249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:27 smithi033 ceph-mon[27845]: pgmap v9250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:27 smithi096 ceph-mon[32753]: pgmap v9250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:27 smithi194 ceph-mon[33017]: pgmap v9250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16406.944081 secs 2024-04-26T16:24:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:29 smithi033 ceph-mon[27845]: pgmap v9251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16409.764342 secs 2024-04-26T16:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16406.944081 secs 2024-04-26T16:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:29 smithi096 ceph-mon[32753]: pgmap v9251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16409.764342 secs 2024-04-26T16:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16406.944081 secs 2024-04-26T16:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:29 smithi194 ceph-mon[33017]: pgmap v9251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16409.764342 secs 2024-04-26T16:24:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:29.868+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:29.868+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:29.868+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:24:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:31 smithi033 ceph-mon[27845]: pgmap v9252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:31 smithi096 ceph-mon[32753]: pgmap v9252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:31 smithi194 ceph-mon[33017]: pgmap v9252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:33 smithi033 ceph-mon[27845]: pgmap v9253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:33 smithi096 ceph-mon[32753]: pgmap v9253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:33 smithi194 ceph-mon[33017]: pgmap v9253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16411.944165 secs 2024-04-26T16:24:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16414.764456 secs 2024-04-26T16:24:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16411.944165 secs 2024-04-26T16:24:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16414.764456 secs 2024-04-26T16:24:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16411.944165 secs 2024-04-26T16:24:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16414.764456 secs 2024-04-26T16:24:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:35 smithi033 ceph-mon[27845]: pgmap v9254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:35 smithi096 ceph-mon[32753]: pgmap v9254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:35 smithi194 ceph-mon[33017]: pgmap v9254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:37 smithi033 ceph-mon[27845]: pgmap v9255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:37 smithi096 ceph-mon[32753]: pgmap v9255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:37 smithi194 ceph-mon[33017]: pgmap v9255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16416.944228 secs 2024-04-26T16:24:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:39 smithi033 ceph-mon[27845]: pgmap v9256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16419.764574 secs 2024-04-26T16:24:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16416.944228 secs 2024-04-26T16:24:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:39 smithi096 ceph-mon[32753]: pgmap v9256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16419.764574 secs 2024-04-26T16:24:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16416.944228 secs 2024-04-26T16:24:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:39 smithi194 ceph-mon[33017]: pgmap v9256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16419.764574 secs 2024-04-26T16:24:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:41 smithi033 ceph-mon[27845]: pgmap v9257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:41 smithi096 ceph-mon[32753]: pgmap v9257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:41 smithi194 ceph-mon[33017]: pgmap v9257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:43 smithi096 ceph-mon[32753]: pgmap v9258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:43 smithi194 ceph-mon[33017]: pgmap v9258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:43 smithi033 ceph-mon[27845]: pgmap v9258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16421.944309 secs 2024-04-26T16:24:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16424.764692 secs 2024-04-26T16:24:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16421.944309 secs 2024-04-26T16:24:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16424.764692 secs 2024-04-26T16:24:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16421.944309 secs 2024-04-26T16:24:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16424.764692 secs 2024-04-26T16:24:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:44.877+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:44.877+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:44.877+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:24:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:45 smithi194 ceph-mon[33017]: pgmap v9259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:45 smithi033 ceph-mon[27845]: pgmap v9259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:45 smithi096 ceph-mon[32753]: pgmap v9259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:47 smithi194 ceph-mon[33017]: pgmap v9260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:47 smithi033 ceph-mon[27845]: pgmap v9260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:47 smithi096 ceph-mon[32753]: pgmap v9260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16426.944393 secs 2024-04-26T16:24:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:49 smithi194 ceph-mon[33017]: pgmap v9261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16429.764828 secs 2024-04-26T16:24:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16426.944393 secs 2024-04-26T16:24:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:49 smithi033 ceph-mon[27845]: pgmap v9261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16429.764828 secs 2024-04-26T16:24:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16426.944393 secs 2024-04-26T16:24:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:49 smithi096 ceph-mon[32753]: pgmap v9261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16429.764828 secs 2024-04-26T16:24:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:51 smithi194 ceph-mon[33017]: pgmap v9262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:51 smithi033 ceph-mon[27845]: pgmap v9262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:51 smithi096 ceph-mon[32753]: pgmap v9262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:53 smithi194 ceph-mon[33017]: pgmap v9263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:53 smithi033 ceph-mon[27845]: pgmap v9263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:53 smithi096 ceph-mon[32753]: pgmap v9263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16431.944473 secs 2024-04-26T16:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16434.764943 secs 2024-04-26T16:24:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16431.944473 secs 2024-04-26T16:24:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16434.764943 secs 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16431.944473 secs 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16434.764943 secs 2024-04-26T16:24:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:55 smithi194 ceph-mon[33017]: pgmap v9264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:55 smithi033 ceph-mon[27845]: pgmap v9264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:55 smithi096 ceph-mon[32753]: pgmap v9264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:57.477 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:57 smithi033 ceph-mon[27845]: pgmap v9265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:57 smithi194 ceph-mon[33017]: pgmap v9265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:57 smithi096 ceph-mon[32753]: pgmap v9265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:24:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16436.944557 secs 2024-04-26T16:24:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:59 smithi194 ceph-mon[33017]: pgmap v9266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16439.765062 secs 2024-04-26T16:24:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16436.944557 secs 2024-04-26T16:24:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:59 smithi033 ceph-mon[27845]: pgmap v9266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16439.765062 secs 2024-04-26T16:24:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16436.944557 secs 2024-04-26T16:24:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:59 smithi096 ceph-mon[32753]: pgmap v9266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:24:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16439.765062 secs 2024-04-26T16:24:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:59.883+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:59.883+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:24:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:24:59.883+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:25:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:01 smithi194 ceph-mon[33017]: pgmap v9267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:01 smithi033 ceph-mon[27845]: pgmap v9267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:01 smithi096 ceph-mon[32753]: pgmap v9267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:03 smithi194 ceph-mon[33017]: pgmap v9268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:03 smithi033 ceph-mon[27845]: pgmap v9268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:03 smithi096 ceph-mon[32753]: pgmap v9268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16441.944638 secs 2024-04-26T16:25:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16444.765174 secs 2024-04-26T16:25:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16441.944638 secs 2024-04-26T16:25:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16444.765174 secs 2024-04-26T16:25:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16441.944638 secs 2024-04-26T16:25:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16444.765174 secs 2024-04-26T16:25:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:05 smithi194 ceph-mon[33017]: pgmap v9269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:05 smithi033 ceph-mon[27845]: pgmap v9269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:05 smithi096 ceph-mon[32753]: pgmap v9269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:07 smithi194 ceph-mon[33017]: pgmap v9270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:07 smithi033 ceph-mon[27845]: pgmap v9270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:07 smithi096 ceph-mon[32753]: pgmap v9270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16446.944721 secs 2024-04-26T16:25:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:25:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16446.944721 secs 2024-04-26T16:25:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:25:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16446.944721 secs 2024-04-26T16:25:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:25:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:09 smithi194 ceph-mon[33017]: pgmap v9271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:09 smithi194 ceph-mon[33017]: pgmap v9272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16449.765292 secs 2024-04-26T16:25:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:09 smithi033 ceph-mon[27845]: pgmap v9271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:09 smithi033 ceph-mon[27845]: pgmap v9272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16449.765292 secs 2024-04-26T16:25:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:09 smithi096 ceph-mon[32753]: pgmap v9271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:09 smithi096 ceph-mon[32753]: pgmap v9272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16449.765292 secs 2024-04-26T16:25:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:11 smithi194 ceph-mon[33017]: pgmap v9273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:11 smithi033 ceph-mon[27845]: pgmap v9273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:11 smithi096 ceph-mon[32753]: pgmap v9273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:13 smithi194 ceph-mon[33017]: pgmap v9274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16451.944805 secs 2024-04-26T16:25:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:13 smithi033 ceph-mon[27845]: pgmap v9274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16451.944805 secs 2024-04-26T16:25:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:13 smithi096 ceph-mon[32753]: pgmap v9274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16451.944805 secs 2024-04-26T16:25:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16454.765365 secs 2024-04-26T16:25:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16454.765365 secs 2024-04-26T16:25:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16454.765365 secs 2024-04-26T16:25:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:14.895+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:25:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:14.895+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:25:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:14.895+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:25:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:15 smithi194 ceph-mon[33017]: pgmap v9275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:15 smithi033 ceph-mon[27845]: pgmap v9275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:15 smithi096 ceph-mon[32753]: pgmap v9275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:17 smithi194 ceph-mon[33017]: pgmap v9276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:17 smithi033 ceph-mon[27845]: pgmap v9276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:17 smithi096 ceph-mon[32753]: pgmap v9276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:25:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16456.944888 secs 2024-04-26T16:25:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:19 smithi033 ceph-mon[27845]: pgmap v9277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:25:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16459.765496 secs 2024-04-26T16:25:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16456.944888 secs 2024-04-26T16:25:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:19 smithi194 ceph-mon[33017]: pgmap v9277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:25:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16459.765496 secs 2024-04-26T16:25:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16456.944888 secs 2024-04-26T16:25:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:19 smithi096 ceph-mon[32753]: pgmap v9277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:25:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16459.765496 secs 2024-04-26T16:25:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:21 smithi194 ceph-mon[33017]: pgmap v9278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:21 smithi033 ceph-mon[27845]: pgmap v9278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:21 smithi096 ceph-mon[32753]: pgmap v9278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:23 smithi194 ceph-mon[33017]: pgmap v9279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:23 smithi033 ceph-mon[27845]: pgmap v9279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:23 smithi096 ceph-mon[32753]: pgmap v9279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16461.944994 secs 2024-04-26T16:25:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16464.765560 secs 2024-04-26T16:25:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16461.944994 secs 2024-04-26T16:25:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16464.765560 secs 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16461.944994 secs 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16464.765560 secs 2024-04-26T16:25:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:25 smithi194 ceph-mon[33017]: pgmap v9280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:25 smithi033 ceph-mon[27845]: pgmap v9280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:25 smithi096 ceph-mon[32753]: pgmap v9280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:27 smithi194 ceph-mon[33017]: pgmap v9281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:27 smithi033 ceph-mon[27845]: pgmap v9281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:27 smithi096 ceph-mon[32753]: pgmap v9281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16466.945084 secs 2024-04-26T16:25:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:29 smithi194 ceph-mon[33017]: pgmap v9282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16469.765683 secs 2024-04-26T16:25:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16466.945084 secs 2024-04-26T16:25:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:29 smithi033 ceph-mon[27845]: pgmap v9282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16469.765683 secs 2024-04-26T16:25:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16466.945084 secs 2024-04-26T16:25:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:29 smithi096 ceph-mon[32753]: pgmap v9282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16469.765683 secs 2024-04-26T16:25:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:29.911+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:25:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:29.912+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:25:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:29.912+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:25:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:31 smithi033 ceph-mon[27845]: pgmap v9283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:31 smithi096 ceph-mon[32753]: pgmap v9283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:31 smithi194 ceph-mon[33017]: pgmap v9283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:33 smithi033 ceph-mon[27845]: pgmap v9284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:33 smithi096 ceph-mon[32753]: pgmap v9284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:33 smithi194 ceph-mon[33017]: pgmap v9284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16471.945164 secs 2024-04-26T16:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16474.765801 secs 2024-04-26T16:25:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16471.945164 secs 2024-04-26T16:25:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16474.765801 secs 2024-04-26T16:25:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16471.945164 secs 2024-04-26T16:25:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16474.765801 secs 2024-04-26T16:25:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:35 smithi033 ceph-mon[27845]: pgmap v9285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:35 smithi096 ceph-mon[32753]: pgmap v9285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:35 smithi194 ceph-mon[33017]: pgmap v9285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:37 smithi033 ceph-mon[27845]: pgmap v9286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:37 smithi096 ceph-mon[32753]: pgmap v9286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:37 smithi194 ceph-mon[33017]: pgmap v9286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16476.945222 secs 2024-04-26T16:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:39 smithi033 ceph-mon[27845]: pgmap v9287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16479.765918 secs 2024-04-26T16:25:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16476.945222 secs 2024-04-26T16:25:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:39 smithi096 ceph-mon[32753]: pgmap v9287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16479.765918 secs 2024-04-26T16:25:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16476.945222 secs 2024-04-26T16:25:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:39 smithi194 ceph-mon[33017]: pgmap v9287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16479.765918 secs 2024-04-26T16:25:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:41 smithi033 ceph-mon[27845]: pgmap v9288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:41 smithi096 ceph-mon[32753]: pgmap v9288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:41 smithi194 ceph-mon[33017]: pgmap v9288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:43 smithi033 ceph-mon[27845]: pgmap v9289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:43 smithi096 ceph-mon[32753]: pgmap v9289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:43 smithi194 ceph-mon[33017]: pgmap v9289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16481.945282 secs 2024-04-26T16:25:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16484.765986 secs 2024-04-26T16:25:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16481.945282 secs 2024-04-26T16:25:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16484.765986 secs 2024-04-26T16:25:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16481.945282 secs 2024-04-26T16:25:44.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16484.765986 secs 2024-04-26T16:25:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:44.924+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:25:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:44.924+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:25:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:44.924+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:25:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:45 smithi033 ceph-mon[27845]: pgmap v9290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:45 smithi096 ceph-mon[32753]: pgmap v9290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:45 smithi194 ceph-mon[33017]: pgmap v9290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:47 smithi033 ceph-mon[27845]: pgmap v9291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:47 smithi096 ceph-mon[32753]: pgmap v9291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:47 smithi194 ceph-mon[33017]: pgmap v9291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16486.945344 secs 2024-04-26T16:25:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:49 smithi033 ceph-mon[27845]: pgmap v9292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16489.766130 secs 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16486.945344 secs 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:49 smithi096 ceph-mon[32753]: pgmap v9292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16489.766130 secs 2024-04-26T16:25:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16486.945344 secs 2024-04-26T16:25:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:49 smithi194 ceph-mon[33017]: pgmap v9292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16489.766130 secs 2024-04-26T16:25:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:51 smithi033 ceph-mon[27845]: pgmap v9293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:51 smithi096 ceph-mon[32753]: pgmap v9293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:51 smithi194 ceph-mon[33017]: pgmap v9293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:53 smithi033 ceph-mon[27845]: pgmap v9294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:53 smithi096 ceph-mon[32753]: pgmap v9294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:53 smithi194 ceph-mon[33017]: pgmap v9294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16491.945426 secs 2024-04-26T16:25:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16494.766207 secs 2024-04-26T16:25:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16491.945426 secs 2024-04-26T16:25:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16494.766207 secs 2024-04-26T16:25:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16491.945426 secs 2024-04-26T16:25:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16494.766207 secs 2024-04-26T16:25:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:55 smithi194 ceph-mon[33017]: pgmap v9295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:55 smithi033 ceph-mon[27845]: pgmap v9295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:55 smithi096 ceph-mon[32753]: pgmap v9295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:57.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:57 smithi033 ceph-mon[27845]: pgmap v9296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:57 smithi194 ceph-mon[33017]: pgmap v9296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:57 smithi096 ceph-mon[32753]: pgmap v9296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:25:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16496.945511 secs 2024-04-26T16:25:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:59 smithi194 ceph-mon[33017]: pgmap v9297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:25:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16499.766348 secs 2024-04-26T16:25:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16496.945511 secs 2024-04-26T16:25:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:59 smithi033 ceph-mon[27845]: pgmap v9297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:25:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16499.766348 secs 2024-04-26T16:25:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16496.945511 secs 2024-04-26T16:25:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:59 smithi096 ceph-mon[32753]: pgmap v9297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:25:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:25:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16499.766348 secs 2024-04-26T16:25:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:59.933+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:25:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:59.933+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:25:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:25:59.933+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:01 smithi194 ceph-mon[33017]: pgmap v9298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:01 smithi033 ceph-mon[27845]: pgmap v9298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:01 smithi096 ceph-mon[32753]: pgmap v9298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:03 smithi194 ceph-mon[33017]: pgmap v9299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:03 smithi033 ceph-mon[27845]: pgmap v9299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:03 smithi096 ceph-mon[32753]: pgmap v9299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16501.945593 secs 2024-04-26T16:26:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16504.766433 secs 2024-04-26T16:26:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16501.945593 secs 2024-04-26T16:26:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16504.766433 secs 2024-04-26T16:26:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16501.945593 secs 2024-04-26T16:26:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16504.766433 secs 2024-04-26T16:26:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:05 smithi194 ceph-mon[33017]: pgmap v9300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:05 smithi033 ceph-mon[27845]: pgmap v9300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:05 smithi096 ceph-mon[32753]: pgmap v9300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:07 smithi194 ceph-mon[33017]: pgmap v9301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:07 smithi033 ceph-mon[27845]: pgmap v9301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:07 smithi096 ceph-mon[32753]: pgmap v9301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16506.945677 secs 2024-04-26T16:26:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:09 smithi194 ceph-mon[33017]: pgmap v9302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16509.766524 secs 2024-04-26T16:26:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16506.945677 secs 2024-04-26T16:26:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:09 smithi033 ceph-mon[27845]: pgmap v9302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16509.766524 secs 2024-04-26T16:26:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16506.945677 secs 2024-04-26T16:26:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:09 smithi096 ceph-mon[32753]: pgmap v9302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16509.766524 secs 2024-04-26T16:26:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:10 smithi194 ceph-mon[33017]: pgmap v9303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:26:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:10 smithi194 ceph-mon[33017]: pgmap v9304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:26:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:10 smithi033 ceph-mon[27845]: pgmap v9303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:26:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:10 smithi033 ceph-mon[27845]: pgmap v9304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:26:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:10 smithi096 ceph-mon[32753]: pgmap v9303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:26:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:10 smithi096 ceph-mon[32753]: pgmap v9304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:26:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:12 smithi194 ceph-mon[33017]: pgmap v9305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:12 smithi033 ceph-mon[27845]: pgmap v9305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:12 smithi096 ceph-mon[32753]: pgmap v9305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:13.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16511.945757 secs 2024-04-26T16:26:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16511.945757 secs 2024-04-26T16:26:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16511.945757 secs 2024-04-26T16:26:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16514.766659 secs 2024-04-26T16:26:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:14 smithi194 ceph-mon[33017]: pgmap v9306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16514.766659 secs 2024-04-26T16:26:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:14 smithi033 ceph-mon[27845]: pgmap v9306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16514.766659 secs 2024-04-26T16:26:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:14 smithi096 ceph-mon[32753]: pgmap v9306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:14.948+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:14.948+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:26:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:14.948+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:16 smithi194 ceph-mon[33017]: pgmap v9307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:16 smithi033 ceph-mon[27845]: pgmap v9307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:16 smithi096 ceph-mon[32753]: pgmap v9307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:18 smithi194 ceph-mon[33017]: pgmap v9308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16516.945839 secs 2024-04-26T16:26:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:18 smithi033 ceph-mon[27845]: pgmap v9308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16516.945839 secs 2024-04-26T16:26:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:18 smithi096 ceph-mon[32753]: pgmap v9308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:26:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16516.945839 secs 2024-04-26T16:26:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16519.766755 secs 2024-04-26T16:26:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16519.766755 secs 2024-04-26T16:26:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16519.766755 secs 2024-04-26T16:26:20.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:20 smithi033 ceph-mon[27845]: pgmap v9309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:26:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:20 smithi194 ceph-mon[33017]: pgmap v9309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:26:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:20 smithi096 ceph-mon[32753]: pgmap v9309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:26:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T16:26:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T16:26:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0022040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T16:26:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0022040 comp rx=0 tx=0).stop 2024-04-26T16:26:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T16:26:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:26:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.757+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.758+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:21.759+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0075070 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:22 smithi194 ceph-mon[33017]: pgmap v9310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:22 smithi033 ceph-mon[27845]: pgmap v9310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:22 smithi096 ceph-mon[32753]: pgmap v9310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16521.945923 secs 2024-04-26T16:26:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16521.945923 secs 2024-04-26T16:26:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16521.945923 secs 2024-04-26T16:26:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:24.605+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:26:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:24.605+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:26:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:24.606+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:26:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:24.606+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:26:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16524.766869 secs 2024-04-26T16:26:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:24 smithi194 ceph-mon[33017]: pgmap v9311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16524.766869 secs 2024-04-26T16:26:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:24 smithi033 ceph-mon[27845]: pgmap v9311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16524.766869 secs 2024-04-26T16:26:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:24 smithi096 ceph-mon[32753]: pgmap v9311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:26 smithi194 ceph-mon[33017]: pgmap v9312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:26 smithi033 ceph-mon[27845]: pgmap v9312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:26 smithi096 ceph-mon[32753]: pgmap v9312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:28 smithi194 ceph-mon[33017]: pgmap v9313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16526.946025 secs 2024-04-26T16:26:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:28 smithi033 ceph-mon[27845]: pgmap v9313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16526.946025 secs 2024-04-26T16:26:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:28 smithi096 ceph-mon[32753]: pgmap v9313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16526.946025 secs 2024-04-26T16:26:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16529.767008 secs 2024-04-26T16:26:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:29 smithi194 ceph-mon[33017]: pgmap v9314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16529.767008 secs 2024-04-26T16:26:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:29 smithi033 ceph-mon[27845]: pgmap v9314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16529.767008 secs 2024-04-26T16:26:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:29 smithi096 ceph-mon[32753]: pgmap v9314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:29.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:29.958+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:29.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:29.958+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:29.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:29.958+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:26:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:32 smithi194 ceph-mon[33017]: pgmap v9315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:32 smithi033 ceph-mon[27845]: pgmap v9315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:32 smithi096 ceph-mon[32753]: pgmap v9315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16531.946116 secs 2024-04-26T16:26:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16531.946116 secs 2024-04-26T16:26:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16531.946116 secs 2024-04-26T16:26:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16534.767158 secs 2024-04-26T16:26:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:34 smithi194 ceph-mon[33017]: pgmap v9316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16534.767158 secs 2024-04-26T16:26:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:34 smithi033 ceph-mon[27845]: pgmap v9316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16534.767158 secs 2024-04-26T16:26:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:34 smithi096 ceph-mon[32753]: pgmap v9316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:36 smithi194 ceph-mon[33017]: pgmap v9317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:36 smithi033 ceph-mon[27845]: pgmap v9317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:36 smithi096 ceph-mon[32753]: pgmap v9317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:38 smithi194 ceph-mon[33017]: pgmap v9318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16536.946205 secs 2024-04-26T16:26:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:38 smithi033 ceph-mon[27845]: pgmap v9318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16536.946205 secs 2024-04-26T16:26:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:38 smithi096 ceph-mon[32753]: pgmap v9318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16536.946205 secs 2024-04-26T16:26:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16539.767225 secs 2024-04-26T16:26:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16539.767225 secs 2024-04-26T16:26:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16539.767225 secs 2024-04-26T16:26:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:40 smithi194 ceph-mon[33017]: pgmap v9319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:40 smithi033 ceph-mon[27845]: pgmap v9319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:40 smithi096 ceph-mon[32753]: pgmap v9319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:42 smithi194 ceph-mon[33017]: pgmap v9320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:42 smithi033 ceph-mon[27845]: pgmap v9320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:42 smithi096 ceph-mon[32753]: pgmap v9320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16541.946286 secs 2024-04-26T16:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16544.767355 secs 2024-04-26T16:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:44 smithi033 ceph-mon[27845]: pgmap v9321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16541.946286 secs 2024-04-26T16:26:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16544.767355 secs 2024-04-26T16:26:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:44 smithi096 ceph-mon[32753]: pgmap v9321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16541.946286 secs 2024-04-26T16:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16544.767355 secs 2024-04-26T16:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:44 smithi194 ceph-mon[33017]: pgmap v9321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:44.973+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:44.974+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:44.974+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:26:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:46 smithi194 ceph-mon[33017]: pgmap v9322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:46 smithi033 ceph-mon[27845]: pgmap v9322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:46 smithi096 ceph-mon[32753]: pgmap v9322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:48 smithi194 ceph-mon[33017]: pgmap v9323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16546.946370 secs 2024-04-26T16:26:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:48 smithi033 ceph-mon[27845]: pgmap v9323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16546.946370 secs 2024-04-26T16:26:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:48 smithi096 ceph-mon[32753]: pgmap v9323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16546.946370 secs 2024-04-26T16:26:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16549.767474 secs 2024-04-26T16:26:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16549.767474 secs 2024-04-26T16:26:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16549.767474 secs 2024-04-26T16:26:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:50 smithi194 ceph-mon[33017]: pgmap v9324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:50 smithi033 ceph-mon[27845]: pgmap v9324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:50 smithi096 ceph-mon[32753]: pgmap v9324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:52 smithi194 ceph-mon[33017]: pgmap v9325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:52 smithi033 ceph-mon[27845]: pgmap v9325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:52 smithi096 ceph-mon[32753]: pgmap v9325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16551.946451 secs 2024-04-26T16:26:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16551.946451 secs 2024-04-26T16:26:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16551.946451 secs 2024-04-26T16:26:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16554.767571 secs 2024-04-26T16:26:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:54 smithi194 ceph-mon[33017]: pgmap v9326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16554.767571 secs 2024-04-26T16:26:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:54 smithi033 ceph-mon[27845]: pgmap v9326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16554.767571 secs 2024-04-26T16:26:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:54 smithi096 ceph-mon[32753]: pgmap v9326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:56 smithi194 ceph-mon[33017]: pgmap v9327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:56 smithi033 ceph-mon[27845]: pgmap v9327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:56 smithi096 ceph-mon[32753]: pgmap v9327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:26:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:58 smithi194 ceph-mon[33017]: pgmap v9328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16556.946532 secs 2024-04-26T16:26:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:58 smithi033 ceph-mon[27845]: pgmap v9328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16556.946532 secs 2024-04-26T16:26:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:58 smithi096 ceph-mon[32753]: pgmap v9328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:26:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16556.946532 secs 2024-04-26T16:26:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:26:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16559.767690 secs 2024-04-26T16:26:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:26:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16559.767690 secs 2024-04-26T16:26:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:26:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16559.767690 secs 2024-04-26T16:26:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:59.989+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:26:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:59.989+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:26:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:26:59.989+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:27:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:00 smithi194 ceph-mon[33017]: pgmap v9329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:00 smithi033 ceph-mon[27845]: pgmap v9329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:00 smithi096 ceph-mon[32753]: pgmap v9329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:01 smithi194 ceph-mon[33017]: pgmap v9330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:01 smithi033 ceph-mon[27845]: pgmap v9330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:01 smithi096 ceph-mon[32753]: pgmap v9330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16561.946612 secs 2024-04-26T16:27:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16561.946612 secs 2024-04-26T16:27:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16561.946612 secs 2024-04-26T16:27:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16564.767822 secs 2024-04-26T16:27:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:04 smithi194 ceph-mon[33017]: pgmap v9331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16564.767822 secs 2024-04-26T16:27:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:04 smithi033 ceph-mon[27845]: pgmap v9331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16564.767822 secs 2024-04-26T16:27:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:04 smithi096 ceph-mon[32753]: pgmap v9331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:06 smithi194 ceph-mon[33017]: pgmap v9332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:06 smithi033 ceph-mon[27845]: pgmap v9332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:06 smithi096 ceph-mon[32753]: pgmap v9332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:08 smithi194 ceph-mon[33017]: pgmap v9333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16566.946719 secs 2024-04-26T16:27:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:08 smithi033 ceph-mon[27845]: pgmap v9333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16566.946719 secs 2024-04-26T16:27:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:08 smithi096 ceph-mon[32753]: pgmap v9333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16566.946719 secs 2024-04-26T16:27:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16569.767959 secs 2024-04-26T16:27:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16569.767959 secs 2024-04-26T16:27:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16569.767959 secs 2024-04-26T16:27:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:10 smithi194 ceph-mon[33017]: pgmap v9334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:10 smithi033 ceph-mon[27845]: pgmap v9334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:10 smithi096 ceph-mon[32753]: pgmap v9334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:12.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:12 smithi096 ceph-mon[32753]: pgmap v9335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:12.666 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:12 smithi194 ceph-mon[33017]: pgmap v9335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:12 smithi033 ceph-mon[27845]: pgmap v9335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:13.627 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16571.946805 secs 2024-04-26T16:27:13.687 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16571.946805 secs 2024-04-26T16:27:13.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16571.946805 secs 2024-04-26T16:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16574.768023 secs 2024-04-26T16:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:14 smithi194 ceph-mon[33017]: pgmap v9336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16574.768023 secs 2024-04-26T16:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:14 smithi033 ceph-mon[27845]: pgmap v9336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16574.768023 secs 2024-04-26T16:27:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:14 smithi096 ceph-mon[32753]: pgmap v9336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:27:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:14.991+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:27:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:14.992+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:27:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:14.992+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:27:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:15 smithi194 ceph-mon[33017]: pgmap v9337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:15.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:15 smithi033 ceph-mon[27845]: pgmap v9337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:15 smithi096 ceph-mon[32753]: pgmap v9337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:27:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:17 smithi033 ceph-mon[27845]: pgmap v9338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T16:27:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:17 smithi033 ceph-mon[27845]: pgmap v9339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:17 smithi096 ceph-mon[32753]: pgmap v9338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T16:27:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:17 smithi096 ceph-mon[32753]: pgmap v9339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:17 smithi194 ceph-mon[33017]: pgmap v9338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T16:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:17 smithi194 ceph-mon[33017]: pgmap v9339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:27:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16576.946893 secs 2024-04-26T16:27:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16579.768144 secs 2024-04-26T16:27:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16576.946893 secs 2024-04-26T16:27:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16579.768144 secs 2024-04-26T16:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16576.946893 secs 2024-04-26T16:27:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16579.768144 secs 2024-04-26T16:27:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:19 smithi033 ceph-mon[27845]: pgmap v9340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T16:27:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:19 smithi096 ceph-mon[32753]: pgmap v9340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T16:27:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:19 smithi194 ceph-mon[33017]: pgmap v9340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T16:27:22.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:21 smithi033 ceph-mon[27845]: pgmap v9341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:21 smithi194 ceph-mon[33017]: pgmap v9341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:21 smithi096 ceph-mon[32753]: pgmap v9341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16581.946986 secs 2024-04-26T16:27:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:23 smithi194 ceph-mon[33017]: pgmap v9342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16584.768237 secs 2024-04-26T16:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16581.946986 secs 2024-04-26T16:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:23 smithi033 ceph-mon[27845]: pgmap v9342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16584.768237 secs 2024-04-26T16:27:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16581.946986 secs 2024-04-26T16:27:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:23 smithi096 ceph-mon[32753]: pgmap v9342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16584.768237 secs 2024-04-26T16:27:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:25 smithi194 ceph-mon[33017]: pgmap v9343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:25 smithi033 ceph-mon[27845]: pgmap v9343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:25 smithi096 ceph-mon[32753]: pgmap v9343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:27 smithi033 ceph-mon[27845]: pgmap v9344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:27:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:27 smithi194 ceph-mon[33017]: pgmap v9344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:27:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:27 smithi096 ceph-mon[32753]: pgmap v9344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:27:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16586.947045 secs 2024-04-26T16:27:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16589.768303 secs 2024-04-26T16:27:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16586.947045 secs 2024-04-26T16:27:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16589.768303 secs 2024-04-26T16:27:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16586.947045 secs 2024-04-26T16:27:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16589.768303 secs 2024-04-26T16:27:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:30.008+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:27:30.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:30.008+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:27:30.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:30.008+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:27:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:29 smithi194 ceph-mon[33017]: pgmap v9345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:29 smithi033 ceph-mon[27845]: pgmap v9345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:29 smithi096 ceph-mon[32753]: pgmap v9345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:31 smithi194 ceph-mon[33017]: pgmap v9346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:31 smithi033 ceph-mon[27845]: pgmap v9346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:31 smithi096 ceph-mon[32753]: pgmap v9346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16591.947106 secs 2024-04-26T16:27:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:33 smithi194 ceph-mon[33017]: pgmap v9347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16594.768442 secs 2024-04-26T16:27:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16591.947106 secs 2024-04-26T16:27:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:33 smithi033 ceph-mon[27845]: pgmap v9347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16594.768442 secs 2024-04-26T16:27:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16591.947106 secs 2024-04-26T16:27:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:33 smithi096 ceph-mon[32753]: pgmap v9347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16594.768442 secs 2024-04-26T16:27:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:35 smithi194 ceph-mon[33017]: pgmap v9348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:35 smithi033 ceph-mon[27845]: pgmap v9348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:35 smithi096 ceph-mon[32753]: pgmap v9348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:37 smithi194 ceph-mon[33017]: pgmap v9349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:37 smithi033 ceph-mon[27845]: pgmap v9349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:37 smithi096 ceph-mon[32753]: pgmap v9349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16596.947166 secs 2024-04-26T16:27:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16599.768565 secs 2024-04-26T16:27:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16596.947166 secs 2024-04-26T16:27:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16599.768565 secs 2024-04-26T16:27:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16596.947166 secs 2024-04-26T16:27:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16599.768565 secs 2024-04-26T16:27:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:40 smithi194 ceph-mon[33017]: pgmap v9350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:40 smithi033 ceph-mon[27845]: pgmap v9350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:40 smithi096 ceph-mon[32753]: pgmap v9350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:42 smithi194 ceph-mon[33017]: pgmap v9351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:42 smithi033 ceph-mon[27845]: pgmap v9351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:42 smithi096 ceph-mon[32753]: pgmap v9351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16601.947226 secs 2024-04-26T16:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:44 smithi194 ceph-mon[33017]: pgmap v9352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16604.768682 secs 2024-04-26T16:27:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16601.947226 secs 2024-04-26T16:27:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:44 smithi033 ceph-mon[27845]: pgmap v9352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16604.768682 secs 2024-04-26T16:27:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16601.947226 secs 2024-04-26T16:27:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:44 smithi096 ceph-mon[32753]: pgmap v9352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16604.768682 secs 2024-04-26T16:27:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:45.023+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:27:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:45.023+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:27:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:27:45.023+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:27:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:46 smithi194 ceph-mon[33017]: pgmap v9353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:46 smithi033 ceph-mon[27845]: pgmap v9353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:46 smithi096 ceph-mon[32753]: pgmap v9353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:48 smithi194 ceph-mon[33017]: pgmap v9354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:48 smithi033 ceph-mon[27845]: pgmap v9354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:48 smithi096 ceph-mon[32753]: pgmap v9354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16606.947285 secs 2024-04-26T16:27:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16609.768819 secs 2024-04-26T16:27:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16606.947285 secs 2024-04-26T16:27:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16609.768819 secs 2024-04-26T16:27:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16606.947285 secs 2024-04-26T16:27:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16609.768819 secs 2024-04-26T16:27:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:50 smithi194 ceph-mon[33017]: pgmap v9355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:50 smithi033 ceph-mon[27845]: pgmap v9355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:50 smithi096 ceph-mon[32753]: pgmap v9355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:52 smithi194 ceph-mon[33017]: pgmap v9356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:52 smithi033 ceph-mon[27845]: pgmap v9356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:52 smithi096 ceph-mon[32753]: pgmap v9356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16611.947345 secs 2024-04-26T16:27:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:54 smithi194 ceph-mon[33017]: pgmap v9357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16614.768935 secs 2024-04-26T16:27:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16611.947345 secs 2024-04-26T16:27:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:54 smithi033 ceph-mon[27845]: pgmap v9357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16614.768935 secs 2024-04-26T16:27:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16611.947345 secs 2024-04-26T16:27:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:54 smithi096 ceph-mon[32753]: pgmap v9357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16614.768935 secs 2024-04-26T16:27:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:56 smithi194 ceph-mon[33017]: pgmap v9358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:56 smithi033 ceph-mon[27845]: pgmap v9358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:56 smithi096 ceph-mon[32753]: pgmap v9358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:27:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:58 smithi194 ceph-mon[33017]: pgmap v9359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:58 smithi033 ceph-mon[27845]: pgmap v9359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:58 smithi096 ceph-mon[32753]: pgmap v9359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:27:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16616.947402 secs 2024-04-26T16:27:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16619.769048 secs 2024-04-26T16:27:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16616.947402 secs 2024-04-26T16:27:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16619.769048 secs 2024-04-26T16:27:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16616.947402 secs 2024-04-26T16:27:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16619.769048 secs 2024-04-26T16:28:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:00.037+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:00.037+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:00.037+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:28:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:00 smithi194 ceph-mon[33017]: pgmap v9360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:00 smithi033 ceph-mon[27845]: pgmap v9360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:00 smithi096 ceph-mon[32753]: pgmap v9360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:02 smithi194 ceph-mon[33017]: pgmap v9361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:02 smithi033 ceph-mon[27845]: pgmap v9361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:02 smithi096 ceph-mon[32753]: pgmap v9361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16621.947485 secs 2024-04-26T16:28:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:04 smithi194 ceph-mon[33017]: pgmap v9362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16624.769201 secs 2024-04-26T16:28:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16621.947485 secs 2024-04-26T16:28:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:04 smithi033 ceph-mon[27845]: pgmap v9362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16624.769201 secs 2024-04-26T16:28:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16621.947485 secs 2024-04-26T16:28:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:04 smithi096 ceph-mon[32753]: pgmap v9362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16624.769201 secs 2024-04-26T16:28:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:06 smithi194 ceph-mon[33017]: pgmap v9363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:06 smithi033 ceph-mon[27845]: pgmap v9363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:06 smithi096 ceph-mon[32753]: pgmap v9363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:08 smithi194 ceph-mon[33017]: pgmap v9364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:08 smithi033 ceph-mon[27845]: pgmap v9364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:08 smithi096 ceph-mon[32753]: pgmap v9364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16626.947570 secs 2024-04-26T16:28:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16629.769324 secs 2024-04-26T16:28:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16626.947570 secs 2024-04-26T16:28:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16629.769324 secs 2024-04-26T16:28:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16626.947570 secs 2024-04-26T16:28:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16629.769324 secs 2024-04-26T16:28:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:10 smithi033 ceph-mon[27845]: pgmap v9365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:10 smithi096 ceph-mon[32753]: pgmap v9365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:10 smithi194 ceph-mon[33017]: pgmap v9365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:12 smithi033 ceph-mon[27845]: pgmap v9366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:12 smithi096 ceph-mon[32753]: pgmap v9366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:12 smithi194 ceph-mon[33017]: pgmap v9366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:14.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16631.947649 secs 2024-04-26T16:28:14.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:14 smithi194 ceph-mon[33017]: pgmap v9367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:14.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16634.769437 secs 2024-04-26T16:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16631.947649 secs 2024-04-26T16:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:14 smithi033 ceph-mon[27845]: pgmap v9367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16634.769437 secs 2024-04-26T16:28:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16631.947649 secs 2024-04-26T16:28:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:14 smithi096 ceph-mon[32753]: pgmap v9367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16634.769437 secs 2024-04-26T16:28:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:15.047+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:15.047+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:15.047+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:28:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:16 smithi033 ceph-mon[27845]: pgmap v9368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:16 smithi096 ceph-mon[32753]: pgmap v9368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:16.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:16 smithi194 ceph-mon[33017]: pgmap v9368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:18 smithi033 ceph-mon[27845]: pgmap v9369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:18 smithi096 ceph-mon[32753]: pgmap v9369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:18 smithi194 ceph-mon[33017]: pgmap v9369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16636.947737 secs 2024-04-26T16:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16639.769559 secs 2024-04-26T16:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:28:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16636.947737 secs 2024-04-26T16:28:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16639.769559 secs 2024-04-26T16:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16636.947737 secs 2024-04-26T16:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16639.769559 secs 2024-04-26T16:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:28:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:20 smithi033 ceph-mon[27845]: pgmap v9370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:20 smithi033 ceph-mon[27845]: pgmap v9371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:20 smithi096 ceph-mon[32753]: pgmap v9370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:20 smithi096 ceph-mon[32753]: pgmap v9371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:20 smithi194 ceph-mon[33017]: pgmap v9370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:20 smithi194 ceph-mon[33017]: pgmap v9371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:22 smithi033 ceph-mon[27845]: pgmap v9372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:22 smithi096 ceph-mon[32753]: pgmap v9372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:22 smithi194 ceph-mon[33017]: pgmap v9372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:24 smithi033 ceph-mon[27845]: pgmap v9373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16641.947818 secs 2024-04-26T16:28:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16644.769703 secs 2024-04-26T16:28:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:24 smithi096 ceph-mon[32753]: pgmap v9373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16641.947818 secs 2024-04-26T16:28:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16644.769703 secs 2024-04-26T16:28:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:24 smithi194 ceph-mon[33017]: pgmap v9373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16641.947818 secs 2024-04-26T16:28:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16644.769703 secs 2024-04-26T16:28:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:26 smithi033 ceph-mon[27845]: pgmap v9374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:26 smithi096 ceph-mon[32753]: pgmap v9374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:26 smithi194 ceph-mon[33017]: pgmap v9374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:28 smithi033 ceph-mon[27845]: pgmap v9375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:28 smithi096 ceph-mon[32753]: pgmap v9375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:28 smithi194 ceph-mon[33017]: pgmap v9375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16646.947902 secs 2024-04-26T16:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16649.769842 secs 2024-04-26T16:28:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16646.947902 secs 2024-04-26T16:28:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16649.769842 secs 2024-04-26T16:28:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16646.947902 secs 2024-04-26T16:28:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16649.769842 secs 2024-04-26T16:28:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:30.063+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:30.063+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:28:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:30.063+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:30 smithi096 ceph-mon[32753]: pgmap v9376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:28:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:30 smithi194 ceph-mon[33017]: pgmap v9376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:28:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:30 smithi033 ceph-mon[27845]: pgmap v9376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:28:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:32 smithi096 ceph-mon[32753]: pgmap v9377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:32 smithi194 ceph-mon[33017]: pgmap v9377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:32 smithi033 ceph-mon[27845]: pgmap v9377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:34 smithi194 ceph-mon[33017]: pgmap v9378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16651.948003 secs 2024-04-26T16:28:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16654.769988 secs 2024-04-26T16:28:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:34 smithi033 ceph-mon[27845]: pgmap v9378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16651.948003 secs 2024-04-26T16:28:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16654.769988 secs 2024-04-26T16:28:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:34 smithi096 ceph-mon[32753]: pgmap v9378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16651.948003 secs 2024-04-26T16:28:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16654.769988 secs 2024-04-26T16:28:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:36 smithi194 ceph-mon[33017]: pgmap v9379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:36 smithi033 ceph-mon[27845]: pgmap v9379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:36 smithi096 ceph-mon[32753]: pgmap v9379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:38 smithi194 ceph-mon[33017]: pgmap v9380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:38 smithi033 ceph-mon[27845]: pgmap v9380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:38 smithi096 ceph-mon[32753]: pgmap v9380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16656.948104 secs 2024-04-26T16:28:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16659.770136 secs 2024-04-26T16:28:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16656.948104 secs 2024-04-26T16:28:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16659.770136 secs 2024-04-26T16:28:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16656.948104 secs 2024-04-26T16:28:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16659.770136 secs 2024-04-26T16:28:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:40 smithi033 ceph-mon[27845]: pgmap v9381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:40 smithi194 ceph-mon[33017]: pgmap v9381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:40 smithi096 ceph-mon[32753]: pgmap v9381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:28:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:42 smithi194 ceph-mon[33017]: pgmap v9382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:42 smithi033 ceph-mon[27845]: pgmap v9382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:42 smithi096 ceph-mon[32753]: pgmap v9382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:28:44.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:44 smithi096 ceph-mon[32753]: pgmap v9383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:44.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16661.948193 secs 2024-04-26T16:28:44.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16664.770250 secs 2024-04-26T16:28:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:44 smithi194 ceph-mon[33017]: pgmap v9383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16661.948193 secs 2024-04-26T16:28:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16664.770250 secs 2024-04-26T16:28:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:44 smithi033 ceph-mon[27845]: pgmap v9383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16661.948193 secs 2024-04-26T16:28:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16664.770250 secs 2024-04-26T16:28:45.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:45.074+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:45.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:45.074+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:28:45.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:28:45.074+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:28:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:46 smithi194 ceph-mon[33017]: pgmap v9384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:46 smithi033 ceph-mon[27845]: pgmap v9384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:46 smithi096 ceph-mon[32753]: pgmap v9384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:48 smithi194 ceph-mon[33017]: pgmap v9385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:48 smithi033 ceph-mon[27845]: pgmap v9385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:48 smithi096 ceph-mon[32753]: pgmap v9385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16666.948293 secs 2024-04-26T16:28:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16669.770399 secs 2024-04-26T16:28:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16666.948293 secs 2024-04-26T16:28:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16669.770399 secs 2024-04-26T16:28:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16666.948293 secs 2024-04-26T16:28:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16669.770399 secs 2024-04-26T16:28:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:50 smithi194 ceph-mon[33017]: pgmap v9386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:50 smithi033 ceph-mon[27845]: pgmap v9386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:50 smithi096 ceph-mon[32753]: pgmap v9386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:52 smithi194 ceph-mon[33017]: pgmap v9387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:52 smithi033 ceph-mon[27845]: pgmap v9387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:52 smithi096 ceph-mon[32753]: pgmap v9387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:54 smithi194 ceph-mon[33017]: pgmap v9388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16671.948380 secs 2024-04-26T16:28:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16674.770530 secs 2024-04-26T16:28:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:54 smithi033 ceph-mon[27845]: pgmap v9388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16671.948380 secs 2024-04-26T16:28:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16674.770530 secs 2024-04-26T16:28:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:54 smithi096 ceph-mon[32753]: pgmap v9388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16671.948380 secs 2024-04-26T16:28:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16674.770530 secs 2024-04-26T16:28:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:56 smithi194 ceph-mon[33017]: pgmap v9389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:56 smithi033 ceph-mon[27845]: pgmap v9389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:56 smithi096 ceph-mon[32753]: pgmap v9389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:28:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:58 smithi194 ceph-mon[33017]: pgmap v9390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:58 smithi033 ceph-mon[27845]: pgmap v9390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:58 smithi096 ceph-mon[32753]: pgmap v9390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:28:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16676.948467 secs 2024-04-26T16:28:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:28:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16679.770667 secs 2024-04-26T16:28:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16676.948467 secs 2024-04-26T16:28:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:28:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16679.770667 secs 2024-04-26T16:28:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16676.948467 secs 2024-04-26T16:28:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:28:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16679.770667 secs 2024-04-26T16:29:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:00.088+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:00.088+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:00.088+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:29:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:00 smithi194 ceph-mon[33017]: pgmap v9391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:00 smithi033 ceph-mon[27845]: pgmap v9391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:00 smithi096 ceph-mon[32753]: pgmap v9391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:02 smithi194 ceph-mon[33017]: pgmap v9392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:02 smithi033 ceph-mon[27845]: pgmap v9392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:02 smithi096 ceph-mon[32753]: pgmap v9392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16681.948554 secs 2024-04-26T16:29:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16681.948554 secs 2024-04-26T16:29:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16681.948554 secs 2024-04-26T16:29:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:04 smithi194 ceph-mon[33017]: pgmap v9393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16684.770794 secs 2024-04-26T16:29:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:04 smithi033 ceph-mon[27845]: pgmap v9393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16684.770794 secs 2024-04-26T16:29:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:04 smithi096 ceph-mon[32753]: pgmap v9393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16684.770794 secs 2024-04-26T16:29:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:06 smithi194 ceph-mon[33017]: pgmap v9394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:06 smithi033 ceph-mon[27845]: pgmap v9394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:06 smithi096 ceph-mon[32753]: pgmap v9394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:08 smithi194 ceph-mon[33017]: pgmap v9395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16686.948655 secs 2024-04-26T16:29:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:08 smithi033 ceph-mon[27845]: pgmap v9395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16686.948655 secs 2024-04-26T16:29:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:08 smithi096 ceph-mon[32753]: pgmap v9395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16686.948655 secs 2024-04-26T16:29:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16689.770912 secs 2024-04-26T16:29:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16689.770912 secs 2024-04-26T16:29:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16689.770912 secs 2024-04-26T16:29:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:10 smithi194 ceph-mon[33017]: pgmap v9396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:10 smithi033 ceph-mon[27845]: pgmap v9396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:10 smithi096 ceph-mon[32753]: pgmap v9396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:12 smithi194 ceph-mon[33017]: pgmap v9397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:12 smithi033 ceph-mon[27845]: pgmap v9397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:12 smithi096 ceph-mon[32753]: pgmap v9397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16691.948756 secs 2024-04-26T16:29:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16691.948756 secs 2024-04-26T16:29:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16691.948756 secs 2024-04-26T16:29:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:14 smithi194 ceph-mon[33017]: pgmap v9398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16694.770981 secs 2024-04-26T16:29:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:14 smithi033 ceph-mon[27845]: pgmap v9398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16694.770981 secs 2024-04-26T16:29:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:14 smithi096 ceph-mon[32753]: pgmap v9398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16694.770981 secs 2024-04-26T16:29:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:15.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:15.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:15.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:29:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:16 smithi194 ceph-mon[33017]: pgmap v9399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:16 smithi033 ceph-mon[27845]: pgmap v9399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:16 smithi096 ceph-mon[32753]: pgmap v9399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:18 smithi194 ceph-mon[33017]: pgmap v9400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16696.948859 secs 2024-04-26T16:29:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:18 smithi033 ceph-mon[27845]: pgmap v9400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16696.948859 secs 2024-04-26T16:29:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:18 smithi096 ceph-mon[32753]: pgmap v9400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16696.948859 secs 2024-04-26T16:29:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16699.771101 secs 2024-04-26T16:29:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16699.771101 secs 2024-04-26T16:29:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16699.771101 secs 2024-04-26T16:29:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:20 smithi033 ceph-mon[27845]: pgmap v9401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:29:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:20 smithi096 ceph-mon[32753]: pgmap v9401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:29:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:20 smithi194 ceph-mon[33017]: pgmap v9401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:29:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:21 smithi033 ceph-mon[27845]: pgmap v9402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:29:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:21 smithi033 ceph-mon[27845]: pgmap v9403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:21 smithi096 ceph-mon[32753]: pgmap v9402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:29:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:21 smithi096 ceph-mon[32753]: pgmap v9403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:21 smithi194 ceph-mon[33017]: pgmap v9402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:29:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:21 smithi194 ceph-mon[33017]: pgmap v9403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:23 smithi033 ceph-mon[27845]: pgmap v9404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16701.948962 secs 2024-04-26T16:29:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:23 smithi096 ceph-mon[32753]: pgmap v9404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16701.948962 secs 2024-04-26T16:29:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:23 smithi194 ceph-mon[33017]: pgmap v9404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16701.948962 secs 2024-04-26T16:29:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:24.636+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:29:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:24.636+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:29:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:24.636+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T16:29:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:24.636+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:29:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16704.771224 secs 2024-04-26T16:29:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16704.771224 secs 2024-04-26T16:29:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16704.771224 secs 2024-04-26T16:29:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:25 smithi033 ceph-mon[27845]: pgmap v9405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T16:29:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:25 smithi096 ceph-mon[32753]: pgmap v9405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T16:29:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:25 smithi194 ceph-mon[33017]: pgmap v9405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T16:29:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:27 smithi033 ceph-mon[27845]: pgmap v9406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:27 smithi096 ceph-mon[32753]: pgmap v9406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:27 smithi194 ceph-mon[33017]: pgmap v9406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16706.949005 secs 2024-04-26T16:29:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16706.949005 secs 2024-04-26T16:29:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16706.949005 secs 2024-04-26T16:29:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:29 smithi033 ceph-mon[27845]: pgmap v9407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16709.771345 secs 2024-04-26T16:29:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:29 smithi096 ceph-mon[32753]: pgmap v9407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16709.771345 secs 2024-04-26T16:29:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:29 smithi194 ceph-mon[33017]: pgmap v9407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16709.771345 secs 2024-04-26T16:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:30.104+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:30.104+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:30.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:30.104+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:29:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:31 smithi033 ceph-mon[27845]: pgmap v9408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:29:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:31 smithi096 ceph-mon[32753]: pgmap v9408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:29:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:31 smithi194 ceph-mon[33017]: pgmap v9408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:29:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:33 smithi033 ceph-mon[27845]: pgmap v9409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16711.949068 secs 2024-04-26T16:29:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:33 smithi096 ceph-mon[32753]: pgmap v9409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16711.949068 secs 2024-04-26T16:29:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:33 smithi194 ceph-mon[33017]: pgmap v9409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16711.949068 secs 2024-04-26T16:29:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16714.771461 secs 2024-04-26T16:29:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16714.771461 secs 2024-04-26T16:29:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16714.771461 secs 2024-04-26T16:29:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:35 smithi096 ceph-mon[32753]: pgmap v9410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:35 smithi194 ceph-mon[33017]: pgmap v9410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:35 smithi033 ceph-mon[27845]: pgmap v9410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:37 smithi194 ceph-mon[33017]: pgmap v9411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:37 smithi033 ceph-mon[27845]: pgmap v9411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:37 smithi096 ceph-mon[32753]: pgmap v9411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16716.949129 secs 2024-04-26T16:29:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16716.949129 secs 2024-04-26T16:29:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16716.949129 secs 2024-04-26T16:29:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:39 smithi194 ceph-mon[33017]: pgmap v9412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16719.771583 secs 2024-04-26T16:29:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:39 smithi033 ceph-mon[27845]: pgmap v9412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16719.771583 secs 2024-04-26T16:29:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:39 smithi096 ceph-mon[32753]: pgmap v9412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16719.771583 secs 2024-04-26T16:29:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:41 smithi194 ceph-mon[33017]: pgmap v9413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:41 smithi033 ceph-mon[27845]: pgmap v9413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:41 smithi096 ceph-mon[32753]: pgmap v9413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:43 smithi194 ceph-mon[33017]: pgmap v9414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16721.949189 secs 2024-04-26T16:29:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:43 smithi033 ceph-mon[27845]: pgmap v9414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16721.949189 secs 2024-04-26T16:29:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:43 smithi096 ceph-mon[32753]: pgmap v9414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16721.949189 secs 2024-04-26T16:29:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16724.771700 secs 2024-04-26T16:29:44.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16724.771700 secs 2024-04-26T16:29:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16724.771700 secs 2024-04-26T16:29:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:45.110+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:45.111+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:29:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:29:45.111+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:29:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:45 smithi194 ceph-mon[33017]: pgmap v9415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:45 smithi033 ceph-mon[27845]: pgmap v9415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:45 smithi096 ceph-mon[32753]: pgmap v9415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:47 smithi194 ceph-mon[33017]: pgmap v9416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:47 smithi033 ceph-mon[27845]: pgmap v9416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:47 smithi096 ceph-mon[32753]: pgmap v9416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16726.949275 secs 2024-04-26T16:29:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16726.949275 secs 2024-04-26T16:29:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16726.949275 secs 2024-04-26T16:29:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:49 smithi194 ceph-mon[33017]: pgmap v9417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16729.771783 secs 2024-04-26T16:29:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:49 smithi033 ceph-mon[27845]: pgmap v9417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16729.771783 secs 2024-04-26T16:29:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:49 smithi096 ceph-mon[32753]: pgmap v9417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16729.771783 secs 2024-04-26T16:29:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:51 smithi194 ceph-mon[33017]: pgmap v9418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:51 smithi033 ceph-mon[27845]: pgmap v9418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:51 smithi096 ceph-mon[32753]: pgmap v9418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:29:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:53 smithi194 ceph-mon[33017]: pgmap v9419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16731.949360 secs 2024-04-26T16:29:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:53 smithi033 ceph-mon[27845]: pgmap v9419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16731.949360 secs 2024-04-26T16:29:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:53 smithi096 ceph-mon[32753]: pgmap v9419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16731.949360 secs 2024-04-26T16:29:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16734.771895 secs 2024-04-26T16:29:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16734.771895 secs 2024-04-26T16:29:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16734.771895 secs 2024-04-26T16:29:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:55 smithi194 ceph-mon[33017]: pgmap v9420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:55 smithi033 ceph-mon[27845]: pgmap v9420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:55 smithi096 ceph-mon[32753]: pgmap v9420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:29:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:57 smithi194 ceph-mon[33017]: pgmap v9421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:57 smithi033 ceph-mon[27845]: pgmap v9421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:57 smithi096 ceph-mon[32753]: pgmap v9421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:29:58.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16736.949444 secs 2024-04-26T16:29:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16736.949444 secs 2024-04-26T16:29:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16736.949444 secs 2024-04-26T16:29:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:59 smithi194 ceph-mon[33017]: pgmap v9422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:29:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16739.772015 secs 2024-04-26T16:29:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:59 smithi033 ceph-mon[27845]: pgmap v9422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:29:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16739.772015 secs 2024-04-26T16:29:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:59 smithi096 ceph-mon[32753]: pgmap v9422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:29:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:29:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16739.772015 secs 2024-04-26T16:30:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:00.121+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:00.121+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:00.121+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:30:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:30:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:30:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:30:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:01 smithi194 ceph-mon[33017]: pgmap v9423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:01 smithi033 ceph-mon[27845]: pgmap v9423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:01 smithi096 ceph-mon[32753]: pgmap v9423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:03 smithi194 ceph-mon[33017]: pgmap v9424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16741.949501 secs 2024-04-26T16:30:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:03 smithi033 ceph-mon[27845]: pgmap v9424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16741.949501 secs 2024-04-26T16:30:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:03 smithi096 ceph-mon[32753]: pgmap v9424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16741.949501 secs 2024-04-26T16:30:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16744.772119 secs 2024-04-26T16:30:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16744.772119 secs 2024-04-26T16:30:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16744.772119 secs 2024-04-26T16:30:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:05 smithi194 ceph-mon[33017]: pgmap v9425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:30:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:05 smithi033 ceph-mon[27845]: pgmap v9425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:30:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:05 smithi096 ceph-mon[32753]: pgmap v9425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:30:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:07 smithi194 ceph-mon[33017]: pgmap v9426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:30:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:07 smithi033 ceph-mon[27845]: pgmap v9426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:30:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:07 smithi096 ceph-mon[32753]: pgmap v9426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:30:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16746.949560 secs 2024-04-26T16:30:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16746.949560 secs 2024-04-26T16:30:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16746.949560 secs 2024-04-26T16:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:09.643+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:09.643+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:09.643+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T16:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:09.643+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:30:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:09 smithi194 ceph-mon[33017]: pgmap v9427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16749.772193 secs 2024-04-26T16:30:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:09 smithi033 ceph-mon[27845]: pgmap v9427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16749.772193 secs 2024-04-26T16:30:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:09 smithi096 ceph-mon[32753]: pgmap v9427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16749.772193 secs 2024-04-26T16:30:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:11 smithi194 ceph-mon[33017]: pgmap v9428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:11 smithi033 ceph-mon[27845]: pgmap v9428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:11 smithi096 ceph-mon[32753]: pgmap v9428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:13 smithi194 ceph-mon[33017]: pgmap v9429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:13.969 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16751.949641 secs 2024-04-26T16:30:13.970 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:13 smithi096 ceph-mon[32753]: pgmap v9429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:13.971 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16751.949641 secs 2024-04-26T16:30:13.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:13 smithi033 ceph-mon[27845]: pgmap v9429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:13.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16751.949641 secs 2024-04-26T16:30:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16754.772310 secs 2024-04-26T16:30:14.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16754.772310 secs 2024-04-26T16:30:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16754.772310 secs 2024-04-26T16:30:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:15.129+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:15.130+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:15.130+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:30:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:15 smithi033 ceph-mon[27845]: pgmap v9430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:30:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:15 smithi096 ceph-mon[32753]: pgmap v9430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:30:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:15 smithi194 ceph-mon[33017]: pgmap v9430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:30:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:16 smithi033 ceph-mon[27845]: pgmap v9431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:30:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:16 smithi096 ceph-mon[32753]: pgmap v9431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:30:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:16 smithi194 ceph-mon[33017]: pgmap v9431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:30:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16756.949729 secs 2024-04-26T16:30:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16756.949729 secs 2024-04-26T16:30:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16756.949729 secs 2024-04-26T16:30:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:19 smithi033 ceph-mon[27845]: pgmap v9432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16759.772423 secs 2024-04-26T16:30:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:19 smithi096 ceph-mon[32753]: pgmap v9432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16759.772423 secs 2024-04-26T16:30:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:19 smithi194 ceph-mon[33017]: pgmap v9432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16759.772423 secs 2024-04-26T16:30:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:21 smithi033 ceph-mon[27845]: pgmap v9433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:21 smithi096 ceph-mon[32753]: pgmap v9433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:21 smithi194 ceph-mon[33017]: pgmap v9433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:22.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:22.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:23 smithi194 ceph-mon[33017]: pgmap v9434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16761.949780 secs 2024-04-26T16:30:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:23 smithi033 ceph-mon[27845]: pgmap v9434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16761.949780 secs 2024-04-26T16:30:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:23 smithi096 ceph-mon[32753]: pgmap v9434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16761.949780 secs 2024-04-26T16:30:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:30:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:24 smithi194 ceph-mon[33017]: pgmap v9435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:30:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:24 smithi194 ceph-mon[33017]: pgmap v9436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T16:30:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16764.772498 secs 2024-04-26T16:30:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:24 smithi033 ceph-mon[27845]: pgmap v9435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:30:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:24 smithi033 ceph-mon[27845]: pgmap v9436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T16:30:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16764.772498 secs 2024-04-26T16:30:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:24 smithi096 ceph-mon[32753]: pgmap v9435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:30:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:24 smithi096 ceph-mon[32753]: pgmap v9436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T16:30:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16764.772498 secs 2024-04-26T16:30:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:26 smithi194 ceph-mon[33017]: pgmap v9437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:26 smithi033 ceph-mon[27845]: pgmap v9437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:26 smithi096 ceph-mon[32753]: pgmap v9437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:28 smithi194 ceph-mon[33017]: pgmap v9438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16766.949842 secs 2024-04-26T16:30:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:28 smithi033 ceph-mon[27845]: pgmap v9438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16766.949842 secs 2024-04-26T16:30:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:28 smithi096 ceph-mon[32753]: pgmap v9438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16766.949842 secs 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16769.772659 secs 2024-04-26T16:30:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16769.772659 secs 2024-04-26T16:30:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16769.772659 secs 2024-04-26T16:30:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:30.131+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:30.132+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:30.132+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:30 smithi194 ceph-mon[33017]: pgmap v9439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:30:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:30 smithi033 ceph-mon[27845]: pgmap v9439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:30:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:30 smithi096 ceph-mon[32753]: pgmap v9439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:30:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:32 smithi033 ceph-mon[27845]: pgmap v9440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:32 smithi096 ceph-mon[32753]: pgmap v9440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:32 smithi194 ceph-mon[33017]: pgmap v9440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:30:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16771.949903 secs 2024-04-26T16:30:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16774.772800 secs 2024-04-26T16:30:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16771.949903 secs 2024-04-26T16:30:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16774.772800 secs 2024-04-26T16:30:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16771.949903 secs 2024-04-26T16:30:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16774.772800 secs 2024-04-26T16:30:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:34 smithi033 ceph-mon[27845]: pgmap v9441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:30:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:34 smithi096 ceph-mon[32753]: pgmap v9441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:30:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:34 smithi194 ceph-mon[33017]: pgmap v9441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:30:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:35 smithi033 ceph-mon[27845]: pgmap v9442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:35 smithi096 ceph-mon[32753]: pgmap v9442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:35 smithi194 ceph-mon[33017]: pgmap v9442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:38 smithi194 ceph-mon[33017]: pgmap v9443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:38 smithi033 ceph-mon[27845]: pgmap v9443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:38 smithi096 ceph-mon[32753]: pgmap v9443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16776.949969 secs 2024-04-26T16:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16779.772931 secs 2024-04-26T16:30:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16776.949969 secs 2024-04-26T16:30:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16779.772931 secs 2024-04-26T16:30:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16776.949969 secs 2024-04-26T16:30:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16779.772931 secs 2024-04-26T16:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:40 smithi194 ceph-mon[33017]: pgmap v9444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:40 smithi033 ceph-mon[27845]: pgmap v9444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:40 smithi096 ceph-mon[32753]: pgmap v9444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:42 smithi194 ceph-mon[33017]: pgmap v9445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:42 smithi033 ceph-mon[27845]: pgmap v9445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:42 smithi096 ceph-mon[32753]: pgmap v9445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16781.950016 secs 2024-04-26T16:30:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:44 smithi194 ceph-mon[33017]: pgmap v9446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16784.773058 secs 2024-04-26T16:30:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16781.950016 secs 2024-04-26T16:30:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:44 smithi033 ceph-mon[27845]: pgmap v9446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16784.773058 secs 2024-04-26T16:30:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16781.950016 secs 2024-04-26T16:30:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:44 smithi096 ceph-mon[32753]: pgmap v9446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16784.773058 secs 2024-04-26T16:30:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:45.147+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:45.147+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:30:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:30:45.147+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:30:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:46 smithi194 ceph-mon[33017]: pgmap v9447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:46 smithi033 ceph-mon[27845]: pgmap v9447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:46 smithi096 ceph-mon[32753]: pgmap v9447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:48 smithi194 ceph-mon[33017]: pgmap v9448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:48 smithi033 ceph-mon[27845]: pgmap v9448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:48 smithi096 ceph-mon[32753]: pgmap v9448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16786.950099 secs 2024-04-26T16:30:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16789.773204 secs 2024-04-26T16:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16786.950099 secs 2024-04-26T16:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16789.773204 secs 2024-04-26T16:30:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16786.950099 secs 2024-04-26T16:30:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16789.773204 secs 2024-04-26T16:30:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:50 smithi194 ceph-mon[33017]: pgmap v9449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:50 smithi033 ceph-mon[27845]: pgmap v9449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:50 smithi096 ceph-mon[32753]: pgmap v9449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:52 smithi194 ceph-mon[33017]: pgmap v9450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:52 smithi033 ceph-mon[27845]: pgmap v9450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:52 smithi096 ceph-mon[32753]: pgmap v9450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16791.950154 secs 2024-04-26T16:30:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:54 smithi194 ceph-mon[33017]: pgmap v9451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16794.773338 secs 2024-04-26T16:30:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16791.950154 secs 2024-04-26T16:30:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:54 smithi033 ceph-mon[27845]: pgmap v9451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16794.773338 secs 2024-04-26T16:30:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16791.950154 secs 2024-04-26T16:30:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:54 smithi096 ceph-mon[32753]: pgmap v9451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16794.773338 secs 2024-04-26T16:30:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:56 smithi194 ceph-mon[33017]: pgmap v9452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:56 smithi033 ceph-mon[27845]: pgmap v9452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:56 smithi096 ceph-mon[32753]: pgmap v9452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:30:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:58 smithi194 ceph-mon[33017]: pgmap v9453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:58 smithi033 ceph-mon[27845]: pgmap v9453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:58 smithi096 ceph-mon[32753]: pgmap v9453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16796.950237 secs 2024-04-26T16:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:30:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16799.773399 secs 2024-04-26T16:30:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16796.950237 secs 2024-04-26T16:30:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:30:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16799.773399 secs 2024-04-26T16:30:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16796.950237 secs 2024-04-26T16:30:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:30:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16799.773399 secs 2024-04-26T16:31:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:00.163+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:00.163+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:00.163+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:31:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:00 smithi033 ceph-mon[27845]: pgmap v9454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:00 smithi096 ceph-mon[32753]: pgmap v9454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:00 smithi194 ceph-mon[33017]: pgmap v9454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:02 smithi033 ceph-mon[27845]: pgmap v9455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:02 smithi096 ceph-mon[32753]: pgmap v9455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:02 smithi194 ceph-mon[33017]: pgmap v9455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16801.950317 secs 2024-04-26T16:31:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:04 smithi033 ceph-mon[27845]: pgmap v9456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16804.773522 secs 2024-04-26T16:31:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16801.950317 secs 2024-04-26T16:31:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:04 smithi096 ceph-mon[32753]: pgmap v9456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16804.773522 secs 2024-04-26T16:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16801.950317 secs 2024-04-26T16:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:04 smithi194 ceph-mon[33017]: pgmap v9456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16804.773522 secs 2024-04-26T16:31:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:06 smithi033 ceph-mon[27845]: pgmap v9457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:06 smithi096 ceph-mon[32753]: pgmap v9457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:06 smithi194 ceph-mon[33017]: pgmap v9457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:08 smithi033 ceph-mon[27845]: pgmap v9458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:08 smithi096 ceph-mon[32753]: pgmap v9458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:08 smithi194 ceph-mon[33017]: pgmap v9458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16806.950404 secs 2024-04-26T16:31:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16809.773639 secs 2024-04-26T16:31:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16806.950404 secs 2024-04-26T16:31:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16809.773639 secs 2024-04-26T16:31:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16806.950404 secs 2024-04-26T16:31:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16809.773639 secs 2024-04-26T16:31:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:10 smithi033 ceph-mon[27845]: pgmap v9459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:10 smithi096 ceph-mon[32753]: pgmap v9459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:10 smithi194 ceph-mon[33017]: pgmap v9459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:12 smithi033 ceph-mon[27845]: pgmap v9460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:12 smithi096 ceph-mon[32753]: pgmap v9460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:12 smithi194 ceph-mon[33017]: pgmap v9460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:14.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16811.950492 secs 2024-04-26T16:31:14.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:14 smithi194 ceph-mon[33017]: pgmap v9461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:14.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16814.773745 secs 2024-04-26T16:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16811.950492 secs 2024-04-26T16:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:14 smithi033 ceph-mon[27845]: pgmap v9461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16814.773745 secs 2024-04-26T16:31:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16811.950492 secs 2024-04-26T16:31:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:14 smithi096 ceph-mon[32753]: pgmap v9461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16814.773745 secs 2024-04-26T16:31:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:15.176+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:15.176+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:31:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:15.176+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:16 smithi033 ceph-mon[27845]: pgmap v9462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:16 smithi096 ceph-mon[32753]: pgmap v9462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:16 smithi194 ceph-mon[33017]: pgmap v9462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:18 smithi033 ceph-mon[27845]: pgmap v9463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:18 smithi096 ceph-mon[32753]: pgmap v9463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:18 smithi194 ceph-mon[33017]: pgmap v9463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16816.950556 secs 2024-04-26T16:31:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16819.773864 secs 2024-04-26T16:31:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16816.950556 secs 2024-04-26T16:31:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16819.773864 secs 2024-04-26T16:31:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16816.950556 secs 2024-04-26T16:31:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16819.773864 secs 2024-04-26T16:31:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:20 smithi096 ceph-mon[32753]: pgmap v9464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:20 smithi194 ceph-mon[33017]: pgmap v9464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:20 smithi033 ceph-mon[27845]: pgmap v9464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:22 smithi096 ceph-mon[32753]: pgmap v9465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:22 smithi194 ceph-mon[33017]: pgmap v9465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:22 smithi033 ceph-mon[27845]: pgmap v9465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16821.950618 secs 2024-04-26T16:31:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:24 smithi194 ceph-mon[33017]: pgmap v9466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16824.774026 secs 2024-04-26T16:31:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16821.950618 secs 2024-04-26T16:31:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:24 smithi033 ceph-mon[27845]: pgmap v9466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16824.774026 secs 2024-04-26T16:31:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16821.950618 secs 2024-04-26T16:31:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:24 smithi096 ceph-mon[32753]: pgmap v9466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16824.774026 secs 2024-04-26T16:31:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:31:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:31:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:31:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:26 smithi194 ceph-mon[33017]: pgmap v9467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:31:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:26 smithi194 ceph-mon[33017]: pgmap v9468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:26 smithi033 ceph-mon[27845]: pgmap v9467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:31:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:26 smithi033 ceph-mon[27845]: pgmap v9468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:26 smithi096 ceph-mon[32753]: pgmap v9467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:31:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:26 smithi096 ceph-mon[32753]: pgmap v9468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:28.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:28 smithi033 ceph-mon[27845]: pgmap v9469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:28 smithi194 ceph-mon[33017]: pgmap v9469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:28 smithi096 ceph-mon[32753]: pgmap v9469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16826.950703 secs 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16829.774145 secs 2024-04-26T16:31:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16826.950703 secs 2024-04-26T16:31:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16829.774145 secs 2024-04-26T16:31:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16826.950703 secs 2024-04-26T16:31:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16829.774145 secs 2024-04-26T16:31:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:30.192+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:30.192+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:31:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:30.192+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:30 smithi194 ceph-mon[33017]: pgmap v9470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:30 smithi033 ceph-mon[27845]: pgmap v9470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:30 smithi096 ceph-mon[32753]: pgmap v9470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:32 smithi033 ceph-mon[27845]: pgmap v9471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:32 smithi194 ceph-mon[33017]: pgmap v9471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:32 smithi096 ceph-mon[32753]: pgmap v9471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:34 smithi194 ceph-mon[33017]: pgmap v9472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16831.950788 secs 2024-04-26T16:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16834.774277 secs 2024-04-26T16:31:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:34 smithi033 ceph-mon[27845]: pgmap v9472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16831.950788 secs 2024-04-26T16:31:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16834.774277 secs 2024-04-26T16:31:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:34 smithi096 ceph-mon[32753]: pgmap v9472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:31:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16831.950788 secs 2024-04-26T16:31:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16834.774277 secs 2024-04-26T16:31:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:36 smithi194 ceph-mon[33017]: pgmap v9473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:31:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:36 smithi033 ceph-mon[27845]: pgmap v9473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:31:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:36 smithi096 ceph-mon[32753]: pgmap v9473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:31:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:38 smithi194 ceph-mon[33017]: pgmap v9474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:38 smithi033 ceph-mon[27845]: pgmap v9474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:38 smithi096 ceph-mon[32753]: pgmap v9474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16836.950871 secs 2024-04-26T16:31:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16839.774397 secs 2024-04-26T16:31:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16836.950871 secs 2024-04-26T16:31:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16839.774397 secs 2024-04-26T16:31:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16836.950871 secs 2024-04-26T16:31:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16839.774397 secs 2024-04-26T16:31:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:40 smithi194 ceph-mon[33017]: pgmap v9475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:40 smithi033 ceph-mon[27845]: pgmap v9475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:40 smithi096 ceph-mon[32753]: pgmap v9475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:42 smithi194 ceph-mon[33017]: pgmap v9476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:42 smithi033 ceph-mon[27845]: pgmap v9476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:42 smithi096 ceph-mon[32753]: pgmap v9476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:44 smithi194 ceph-mon[33017]: pgmap v9477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16841.950933 secs 2024-04-26T16:31:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16844.774510 secs 2024-04-26T16:31:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:44 smithi033 ceph-mon[27845]: pgmap v9477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16841.950933 secs 2024-04-26T16:31:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16844.774510 secs 2024-04-26T16:31:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:44 smithi096 ceph-mon[32753]: pgmap v9477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16841.950933 secs 2024-04-26T16:31:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16844.774510 secs 2024-04-26T16:31:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:45.194+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:45.195+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:31:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:31:45.195+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:31:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:46 smithi194 ceph-mon[33017]: pgmap v9478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:46 smithi033 ceph-mon[27845]: pgmap v9478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:46 smithi096 ceph-mon[32753]: pgmap v9478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:48 smithi194 ceph-mon[33017]: pgmap v9479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:48 smithi033 ceph-mon[27845]: pgmap v9479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:48 smithi096 ceph-mon[32753]: pgmap v9479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16846.951000 secs 2024-04-26T16:31:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16849.774662 secs 2024-04-26T16:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16846.951000 secs 2024-04-26T16:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16849.774662 secs 2024-04-26T16:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16846.951000 secs 2024-04-26T16:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16849.774662 secs 2024-04-26T16:31:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:50 smithi194 ceph-mon[33017]: pgmap v9480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:50 smithi033 ceph-mon[27845]: pgmap v9480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:50 smithi096 ceph-mon[32753]: pgmap v9480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:52 smithi194 ceph-mon[33017]: pgmap v9481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:52 smithi033 ceph-mon[27845]: pgmap v9481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:52 smithi096 ceph-mon[32753]: pgmap v9481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16851.951058 secs 2024-04-26T16:31:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16851.951058 secs 2024-04-26T16:31:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16851.951058 secs 2024-04-26T16:31:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:54 smithi194 ceph-mon[33017]: pgmap v9482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16854.774844 secs 2024-04-26T16:31:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:54 smithi033 ceph-mon[27845]: pgmap v9482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16854.774844 secs 2024-04-26T16:31:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:54 smithi096 ceph-mon[32753]: pgmap v9482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16854.774844 secs 2024-04-26T16:31:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:56 smithi194 ceph-mon[33017]: pgmap v9483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:56 smithi033 ceph-mon[27845]: pgmap v9483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:56 smithi096 ceph-mon[32753]: pgmap v9483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:31:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:58 smithi194 ceph-mon[33017]: pgmap v9484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16856.951118 secs 2024-04-26T16:31:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:58 smithi033 ceph-mon[27845]: pgmap v9484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16856.951118 secs 2024-04-26T16:31:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:58 smithi096 ceph-mon[32753]: pgmap v9484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:31:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16856.951118 secs 2024-04-26T16:31:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16859.774998 secs 2024-04-26T16:31:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16859.774998 secs 2024-04-26T16:31:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16859.774998 secs 2024-04-26T16:32:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:00.201+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:00.201+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:32:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:00.201+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:00 smithi194 ceph-mon[33017]: pgmap v9485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:00 smithi033 ceph-mon[27845]: pgmap v9485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:00 smithi096 ceph-mon[32753]: pgmap v9485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:02 smithi194 ceph-mon[33017]: pgmap v9486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:02 smithi033 ceph-mon[27845]: pgmap v9486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:02 smithi096 ceph-mon[32753]: pgmap v9486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16861.951180 secs 2024-04-26T16:32:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16861.951180 secs 2024-04-26T16:32:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16861.951180 secs 2024-04-26T16:32:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:04 smithi194 ceph-mon[33017]: pgmap v9487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16864.775081 secs 2024-04-26T16:32:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:04 smithi033 ceph-mon[27845]: pgmap v9487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16864.775081 secs 2024-04-26T16:32:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:04 smithi096 ceph-mon[32753]: pgmap v9487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16864.775081 secs 2024-04-26T16:32:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:06 smithi194 ceph-mon[33017]: pgmap v9488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:06 smithi033 ceph-mon[27845]: pgmap v9488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:06 smithi096 ceph-mon[32753]: pgmap v9488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:08 smithi194 ceph-mon[33017]: pgmap v9489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16866.951235 secs 2024-04-26T16:32:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:08 smithi033 ceph-mon[27845]: pgmap v9489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16866.951235 secs 2024-04-26T16:32:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:08 smithi096 ceph-mon[32753]: pgmap v9489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16866.951235 secs 2024-04-26T16:32:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16869.775197 secs 2024-04-26T16:32:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16869.775197 secs 2024-04-26T16:32:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16869.775197 secs 2024-04-26T16:32:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:10 smithi033 ceph-mon[27845]: pgmap v9490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:10 smithi096 ceph-mon[32753]: pgmap v9490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:10 smithi194 ceph-mon[33017]: pgmap v9490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:12 smithi033 ceph-mon[27845]: pgmap v9491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:12 smithi096 ceph-mon[32753]: pgmap v9491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:12 smithi194 ceph-mon[33017]: pgmap v9491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16871.951318 secs 2024-04-26T16:32:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16871.951318 secs 2024-04-26T16:32:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16871.951318 secs 2024-04-26T16:32:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:14 smithi033 ceph-mon[27845]: pgmap v9492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16874.775308 secs 2024-04-26T16:32:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:14 smithi096 ceph-mon[32753]: pgmap v9492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16874.775308 secs 2024-04-26T16:32:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:14 smithi194 ceph-mon[33017]: pgmap v9492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16874.775308 secs 2024-04-26T16:32:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:15.204+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:15.204+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:15.204+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:32:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:16 smithi033 ceph-mon[27845]: pgmap v9493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:16 smithi096 ceph-mon[32753]: pgmap v9493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:16 smithi194 ceph-mon[33017]: pgmap v9493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:18.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:18 smithi033 ceph-mon[27845]: pgmap v9494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:18.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16876.951401 secs 2024-04-26T16:32:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:18 smithi096 ceph-mon[32753]: pgmap v9494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16876.951401 secs 2024-04-26T16:32:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:18 smithi194 ceph-mon[33017]: pgmap v9494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16876.951401 secs 2024-04-26T16:32:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16879.775392 secs 2024-04-26T16:32:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16879.775392 secs 2024-04-26T16:32:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16879.775392 secs 2024-04-26T16:32:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:20 smithi033 ceph-mon[27845]: pgmap v9495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:20 smithi096 ceph-mon[32753]: pgmap v9495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:20 smithi194 ceph-mon[33017]: pgmap v9495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:22 smithi033 ceph-mon[27845]: pgmap v9496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:22 smithi096 ceph-mon[32753]: pgmap v9496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:22 smithi194 ceph-mon[33017]: pgmap v9496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16881.951486 secs 2024-04-26T16:32:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16881.951486 secs 2024-04-26T16:32:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16881.951486 secs 2024-04-26T16:32:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:24.665+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:32:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:24.665+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:32:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:24.665+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T16:32:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:24.666+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:32:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:24 smithi096 ceph-mon[32753]: pgmap v9497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16884.775501 secs 2024-04-26T16:32:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:24 smithi194 ceph-mon[33017]: pgmap v9497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16884.775501 secs 2024-04-26T16:32:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:24 smithi033 ceph-mon[27845]: pgmap v9497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16884.775501 secs 2024-04-26T16:32:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:26 smithi033 ceph-mon[27845]: pgmap v9498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:26 smithi096 ceph-mon[32753]: pgmap v9498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:26 smithi194 ceph-mon[33017]: pgmap v9498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:28 smithi194 ceph-mon[33017]: pgmap v9499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16886.951556 secs 2024-04-26T16:32:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:28 smithi033 ceph-mon[27845]: pgmap v9499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16886.951556 secs 2024-04-26T16:32:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:28 smithi096 ceph-mon[32753]: pgmap v9499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16886.951556 secs 2024-04-26T16:32:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16889.775589 secs 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16889.775589 secs 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16889.775589 secs 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:30.219+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:30.219+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:32:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:30.219+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:30 smithi033 ceph-mon[27845]: pgmap v9500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:30 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:32:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:30 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:32:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:30 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:32:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:30 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:32:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:30 smithi096 ceph-mon[32753]: pgmap v9500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:30 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:32:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:30 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:32:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:30 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:32:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:30 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:32:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:30 smithi194 ceph-mon[33017]: pgmap v9500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:30 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:32:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:30 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:32:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:30 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:32:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:30 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:32:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: pgmap v9501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:32.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:32.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: pgmap v9501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: pgmap v9501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:32:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:33 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:32:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:33 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:32:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:33 smithi033 ceph-mon[27845]: pgmap v9502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T16:32:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:33 smithi033 ceph-mon[27845]: pgmap v9503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16891.951642 secs 2024-04-26T16:32:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16894.775741 secs 2024-04-26T16:32:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:33 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:32:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:33 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:32:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:33 smithi096 ceph-mon[32753]: pgmap v9502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T16:32:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:33 smithi096 ceph-mon[32753]: pgmap v9503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16891.951642 secs 2024-04-26T16:32:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16894.775741 secs 2024-04-26T16:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:33 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:33 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:33 smithi194 ceph-mon[33017]: pgmap v9502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T16:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:33 smithi194 ceph-mon[33017]: pgmap v9503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16891.951642 secs 2024-04-26T16:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16894.775741 secs 2024-04-26T16:32:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:35 smithi033 ceph-mon[27845]: pgmap v9504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T16:32:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:35 smithi096 ceph-mon[32753]: pgmap v9504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T16:32:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:35 smithi194 ceph-mon[33017]: pgmap v9504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T16:32:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:36 smithi194 ceph-mon[33017]: pgmap v9505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:36 smithi033 ceph-mon[27845]: pgmap v9505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:36 smithi096 ceph-mon[32753]: pgmap v9505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16896.951703 secs 2024-04-26T16:32:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:38 smithi194 ceph-mon[33017]: pgmap v9506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16899.775862 secs 2024-04-26T16:32:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16896.951703 secs 2024-04-26T16:32:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:38 smithi033 ceph-mon[27845]: pgmap v9506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16899.775862 secs 2024-04-26T16:32:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16896.951703 secs 2024-04-26T16:32:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:38 smithi096 ceph-mon[32753]: pgmap v9506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16899.775862 secs 2024-04-26T16:32:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:40 smithi194 ceph-mon[33017]: pgmap v9507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:40 smithi033 ceph-mon[27845]: pgmap v9507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:40 smithi096 ceph-mon[32753]: pgmap v9507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T16:32:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:42 smithi194 ceph-mon[33017]: pgmap v9508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:32:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:42 smithi033 ceph-mon[27845]: pgmap v9508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:32:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:42 smithi096 ceph-mon[32753]: pgmap v9508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:32:44.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16901.951781 secs 2024-04-26T16:32:44.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16904.775984 secs 2024-04-26T16:32:44.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16901.951781 secs 2024-04-26T16:32:44.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16904.775984 secs 2024-04-26T16:32:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16901.951781 secs 2024-04-26T16:32:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16904.775984 secs 2024-04-26T16:32:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:45.232+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:45.232+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:32:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:32:45.232+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:32:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:44 smithi194 ceph-mon[33017]: pgmap v9509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:44 smithi033 ceph-mon[27845]: pgmap v9509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:44 smithi096 ceph-mon[32753]: pgmap v9509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:46 smithi194 ceph-mon[33017]: pgmap v9510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:46 smithi033 ceph-mon[27845]: pgmap v9510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:46 smithi096 ceph-mon[32753]: pgmap v9510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16906.951862 secs 2024-04-26T16:32:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:48 smithi194 ceph-mon[33017]: pgmap v9511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16909.776126 secs 2024-04-26T16:32:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16906.951862 secs 2024-04-26T16:32:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:48 smithi033 ceph-mon[27845]: pgmap v9511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16909.776126 secs 2024-04-26T16:32:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16906.951862 secs 2024-04-26T16:32:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:48 smithi096 ceph-mon[32753]: pgmap v9511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16909.776126 secs 2024-04-26T16:32:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:50 smithi194 ceph-mon[33017]: pgmap v9512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:51.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:50 smithi033 ceph-mon[27845]: pgmap v9512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:50 smithi096 ceph-mon[32753]: pgmap v9512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:52 smithi194 ceph-mon[33017]: pgmap v9513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:52 smithi033 ceph-mon[27845]: pgmap v9513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:52 smithi096 ceph-mon[32753]: pgmap v9513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16911.951944 secs 2024-04-26T16:32:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16914.776231 secs 2024-04-26T16:32:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16911.951944 secs 2024-04-26T16:32:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16914.776231 secs 2024-04-26T16:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16911.951944 secs 2024-04-26T16:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16914.776231 secs 2024-04-26T16:32:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:55 smithi194 ceph-mon[33017]: pgmap v9514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:55 smithi033 ceph-mon[27845]: pgmap v9514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:55 smithi096 ceph-mon[32753]: pgmap v9514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:57 smithi194 ceph-mon[33017]: pgmap v9515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:57 smithi033 ceph-mon[27845]: pgmap v9515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:57 smithi096 ceph-mon[32753]: pgmap v9515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:32:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16916.952006 secs 2024-04-26T16:32:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:59 smithi194 ceph-mon[33017]: pgmap v9516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16919.776341 secs 2024-04-26T16:32:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16916.952006 secs 2024-04-26T16:32:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:59 smithi033 ceph-mon[27845]: pgmap v9516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16919.776341 secs 2024-04-26T16:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16916.952006 secs 2024-04-26T16:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:59 smithi096 ceph-mon[32753]: pgmap v9516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16919.776341 secs 2024-04-26T16:33:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:00.247+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:00.247+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:33:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:00.247+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:01 smithi194 ceph-mon[33017]: pgmap v9517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:01 smithi033 ceph-mon[27845]: pgmap v9517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:01 smithi096 ceph-mon[32753]: pgmap v9517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:03 smithi194 ceph-mon[33017]: pgmap v9518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:03 smithi033 ceph-mon[27845]: pgmap v9518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:03 smithi096 ceph-mon[32753]: pgmap v9518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16921.952065 secs 2024-04-26T16:33:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16924.776466 secs 2024-04-26T16:33:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16921.952065 secs 2024-04-26T16:33:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16924.776466 secs 2024-04-26T16:33:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16921.952065 secs 2024-04-26T16:33:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16924.776466 secs 2024-04-26T16:33:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:05 smithi033 ceph-mon[27845]: pgmap v9519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:05 smithi194 ceph-mon[33017]: pgmap v9519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:05 smithi096 ceph-mon[32753]: pgmap v9519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:07 smithi194 ceph-mon[33017]: pgmap v9520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:07 smithi033 ceph-mon[27845]: pgmap v9520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:07 smithi096 ceph-mon[32753]: pgmap v9520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16926.952147 secs 2024-04-26T16:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:09 smithi194 ceph-mon[33017]: pgmap v9521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16929.776529 secs 2024-04-26T16:33:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16926.952147 secs 2024-04-26T16:33:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:09 smithi033 ceph-mon[27845]: pgmap v9521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16929.776529 secs 2024-04-26T16:33:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16926.952147 secs 2024-04-26T16:33:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:09 smithi096 ceph-mon[32753]: pgmap v9521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16929.776529 secs 2024-04-26T16:33:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:11 smithi194 ceph-mon[33017]: pgmap v9522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:11 smithi033 ceph-mon[27845]: pgmap v9522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:11 smithi096 ceph-mon[32753]: pgmap v9522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:13 smithi194 ceph-mon[33017]: pgmap v9523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:13 smithi033 ceph-mon[27845]: pgmap v9523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:13 smithi096 ceph-mon[32753]: pgmap v9523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:14.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16931.952228 secs 2024-04-26T16:33:14.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16934.776648 secs 2024-04-26T16:33:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16931.952228 secs 2024-04-26T16:33:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16934.776648 secs 2024-04-26T16:33:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16931.952228 secs 2024-04-26T16:33:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16934.776648 secs 2024-04-26T16:33:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:15.255+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:15.255+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:15.255+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:33:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:15 smithi194 ceph-mon[33017]: pgmap v9524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:15 smithi033 ceph-mon[27845]: pgmap v9524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:15 smithi096 ceph-mon[32753]: pgmap v9524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:17 smithi194 ceph-mon[33017]: pgmap v9525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:17 smithi033 ceph-mon[27845]: pgmap v9525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:17 smithi096 ceph-mon[32753]: pgmap v9525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16936.952310 secs 2024-04-26T16:33:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:19 smithi194 ceph-mon[33017]: pgmap v9526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16939.776785 secs 2024-04-26T16:33:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16936.952310 secs 2024-04-26T16:33:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:19 smithi033 ceph-mon[27845]: pgmap v9526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16939.776785 secs 2024-04-26T16:33:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16936.952310 secs 2024-04-26T16:33:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:19 smithi096 ceph-mon[32753]: pgmap v9526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16939.776785 secs 2024-04-26T16:33:21.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:21 smithi033 ceph-mon[27845]: pgmap v9527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:21 smithi194 ceph-mon[33017]: pgmap v9527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:21 smithi096 ceph-mon[32753]: pgmap v9527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:23 smithi194 ceph-mon[33017]: pgmap v9528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:23 smithi033 ceph-mon[27845]: pgmap v9528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:23 smithi096 ceph-mon[32753]: pgmap v9528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16941.952399 secs 2024-04-26T16:33:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16944.776904 secs 2024-04-26T16:33:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16941.952399 secs 2024-04-26T16:33:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16944.776904 secs 2024-04-26T16:33:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16941.952399 secs 2024-04-26T16:33:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16944.776904 secs 2024-04-26T16:33:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:25 smithi194 ceph-mon[33017]: pgmap v9529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:25 smithi033 ceph-mon[27845]: pgmap v9529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:25 smithi096 ceph-mon[32753]: pgmap v9529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:27 smithi194 ceph-mon[33017]: pgmap v9530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:27 smithi033 ceph-mon[27845]: pgmap v9530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:27 smithi096 ceph-mon[32753]: pgmap v9530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16946.952481 secs 2024-04-26T16:33:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:29 smithi194 ceph-mon[33017]: pgmap v9531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16949.776985 secs 2024-04-26T16:33:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16946.952481 secs 2024-04-26T16:33:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:29 smithi033 ceph-mon[27845]: pgmap v9531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16949.776985 secs 2024-04-26T16:33:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16946.952481 secs 2024-04-26T16:33:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:29 smithi096 ceph-mon[32753]: pgmap v9531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16949.776985 secs 2024-04-26T16:33:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:30.270+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:30.270+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:30.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:30.270+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:33:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:31 smithi194 ceph-mon[33017]: pgmap v9532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:31 smithi033 ceph-mon[27845]: pgmap v9532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:31 smithi096 ceph-mon[32753]: pgmap v9532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:33 smithi194 ceph-mon[33017]: pgmap v9533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16951.952574 secs 2024-04-26T16:33:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:33 smithi033 ceph-mon[27845]: pgmap v9533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16951.952574 secs 2024-04-26T16:33:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:33 smithi096 ceph-mon[32753]: pgmap v9533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16951.952574 secs 2024-04-26T16:33:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16954.777130 secs 2024-04-26T16:33:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:34 smithi194 ceph-mon[33017]: pgmap v9534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:34 smithi194 ceph-mon[33017]: pgmap v9535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:33:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16954.777130 secs 2024-04-26T16:33:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:34 smithi033 ceph-mon[27845]: pgmap v9534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:34 smithi033 ceph-mon[27845]: pgmap v9535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:33:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16954.777130 secs 2024-04-26T16:33:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:34 smithi096 ceph-mon[32753]: pgmap v9534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:34 smithi096 ceph-mon[32753]: pgmap v9535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:33:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:36 smithi033 ceph-mon[27845]: pgmap v9536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:36 smithi096 ceph-mon[32753]: pgmap v9536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:36 smithi194 ceph-mon[33017]: pgmap v9536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:38 smithi033 ceph-mon[27845]: pgmap v9537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16956.952657 secs 2024-04-26T16:33:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16959.777268 secs 2024-04-26T16:33:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:38 smithi096 ceph-mon[32753]: pgmap v9537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16956.952657 secs 2024-04-26T16:33:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16959.777268 secs 2024-04-26T16:33:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:38 smithi194 ceph-mon[33017]: pgmap v9537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16956.952657 secs 2024-04-26T16:33:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16959.777268 secs 2024-04-26T16:33:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:40 smithi033 ceph-mon[27845]: pgmap v9538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:40 smithi096 ceph-mon[32753]: pgmap v9538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:40 smithi194 ceph-mon[33017]: pgmap v9538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:42 smithi033 ceph-mon[27845]: pgmap v9539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:42 smithi096 ceph-mon[32753]: pgmap v9539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:42 smithi194 ceph-mon[33017]: pgmap v9539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:33:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16961.952717 secs 2024-04-26T16:33:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16964.777386 secs 2024-04-26T16:33:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16961.952717 secs 2024-04-26T16:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16964.777386 secs 2024-04-26T16:33:44.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16961.952717 secs 2024-04-26T16:33:44.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16964.777386 secs 2024-04-26T16:33:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:44 smithi033 ceph-mon[27845]: pgmap v9540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:33:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:44 smithi096 ceph-mon[32753]: pgmap v9540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:33:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:44 smithi194 ceph-mon[33017]: pgmap v9540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:33:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:45.279+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:45.279+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:33:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:33:45.279+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:33:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:45 smithi033 ceph-mon[27845]: pgmap v9541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:45 smithi096 ceph-mon[32753]: pgmap v9541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:45 smithi194 ceph-mon[33017]: pgmap v9541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:48 smithi033 ceph-mon[27845]: pgmap v9542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16966.952804 secs 2024-04-26T16:33:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16969.777505 secs 2024-04-26T16:33:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:48 smithi096 ceph-mon[32753]: pgmap v9542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16966.952804 secs 2024-04-26T16:33:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16969.777505 secs 2024-04-26T16:33:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:48 smithi194 ceph-mon[33017]: pgmap v9542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16966.952804 secs 2024-04-26T16:33:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16969.777505 secs 2024-04-26T16:33:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:50 smithi033 ceph-mon[27845]: pgmap v9543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:50 smithi096 ceph-mon[32753]: pgmap v9543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:50 smithi194 ceph-mon[33017]: pgmap v9543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:52 smithi033 ceph-mon[27845]: pgmap v9544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:52 smithi096 ceph-mon[32753]: pgmap v9544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:52 smithi194 ceph-mon[33017]: pgmap v9544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16971.952889 secs 2024-04-26T16:33:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16974.777631 secs 2024-04-26T16:33:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16971.952889 secs 2024-04-26T16:33:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16974.777631 secs 2024-04-26T16:33:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16971.952889 secs 2024-04-26T16:33:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16974.777631 secs 2024-04-26T16:33:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:54 smithi033 ceph-mon[27845]: pgmap v9545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:54 smithi096 ceph-mon[32753]: pgmap v9545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:54 smithi194 ceph-mon[33017]: pgmap v9545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:55 smithi033 ceph-mon[27845]: pgmap v9546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:55 smithi096 ceph-mon[32753]: pgmap v9546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:55 smithi194 ceph-mon[33017]: pgmap v9546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:33:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:58 smithi033 ceph-mon[27845]: pgmap v9547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16976.952994 secs 2024-04-26T16:33:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:33:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16979.777766 secs 2024-04-26T16:33:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:58 smithi096 ceph-mon[32753]: pgmap v9547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16976.952994 secs 2024-04-26T16:33:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:33:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16979.777766 secs 2024-04-26T16:33:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:58 smithi194 ceph-mon[33017]: pgmap v9547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:33:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16976.952994 secs 2024-04-26T16:33:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:33:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16979.777766 secs 2024-04-26T16:34:00.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:00.294+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:00.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:00.294+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:34:00.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:00.294+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:00 smithi033 ceph-mon[27845]: pgmap v9548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:00 smithi096 ceph-mon[32753]: pgmap v9548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:00 smithi194 ceph-mon[33017]: pgmap v9548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:02 smithi033 ceph-mon[27845]: pgmap v9549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:02 smithi096 ceph-mon[32753]: pgmap v9549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:02 smithi194 ceph-mon[33017]: pgmap v9549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16981.953057 secs 2024-04-26T16:34:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16984.777885 secs 2024-04-26T16:34:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16981.953057 secs 2024-04-26T16:34:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16984.777885 secs 2024-04-26T16:34:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16981.953057 secs 2024-04-26T16:34:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16984.777885 secs 2024-04-26T16:34:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:04 smithi033 ceph-mon[27845]: pgmap v9550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:04 smithi096 ceph-mon[32753]: pgmap v9550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:04 smithi194 ceph-mon[33017]: pgmap v9550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:05 smithi096 ceph-mon[32753]: pgmap v9551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:05 smithi194 ceph-mon[33017]: pgmap v9551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:05 smithi033 ceph-mon[27845]: pgmap v9551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:08 smithi033 ceph-mon[27845]: pgmap v9552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16986.953159 secs 2024-04-26T16:34:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16989.777966 secs 2024-04-26T16:34:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:08 smithi096 ceph-mon[32753]: pgmap v9552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16986.953159 secs 2024-04-26T16:34:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16989.777966 secs 2024-04-26T16:34:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:08 smithi194 ceph-mon[33017]: pgmap v9552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16986.953159 secs 2024-04-26T16:34:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16989.777966 secs 2024-04-26T16:34:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:10 smithi033 ceph-mon[27845]: pgmap v9553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:10 smithi096 ceph-mon[32753]: pgmap v9553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:10 smithi194 ceph-mon[33017]: pgmap v9553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:12 smithi033 ceph-mon[27845]: pgmap v9554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:12 smithi096 ceph-mon[32753]: pgmap v9554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:12 smithi194 ceph-mon[33017]: pgmap v9554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16991.953247 secs 2024-04-26T16:34:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16994.778040 secs 2024-04-26T16:34:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:13 smithi033 ceph-mon[27845]: pgmap v9555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16991.953247 secs 2024-04-26T16:34:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16994.778040 secs 2024-04-26T16:34:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:13 smithi096 ceph-mon[32753]: pgmap v9555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:14.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16991.953247 secs 2024-04-26T16:34:14.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16994.778040 secs 2024-04-26T16:34:14.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:13 smithi194 ceph-mon[33017]: pgmap v9555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:15.297+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:15.297+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:34:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:15.297+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:16 smithi033 ceph-mon[27845]: pgmap v9556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:16 smithi096 ceph-mon[32753]: pgmap v9556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:16 smithi194 ceph-mon[33017]: pgmap v9556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:18 smithi033 ceph-mon[27845]: pgmap v9557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 16996.953352 secs 2024-04-26T16:34:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 16999.778145 secs 2024-04-26T16:34:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:18 smithi096 ceph-mon[32753]: pgmap v9557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 16996.953352 secs 2024-04-26T16:34:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 16999.778145 secs 2024-04-26T16:34:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:18 smithi194 ceph-mon[33017]: pgmap v9557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 16996.953352 secs 2024-04-26T16:34:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 16999.778145 secs 2024-04-26T16:34:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:19 smithi194 ceph-mon[33017]: pgmap v9558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:19 smithi033 ceph-mon[27845]: pgmap v9558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:19 smithi096 ceph-mon[32753]: pgmap v9558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:22 smithi033 ceph-mon[27845]: pgmap v9559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:22 smithi096 ceph-mon[32753]: pgmap v9559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:22 smithi194 ceph-mon[33017]: pgmap v9559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17001.953439 secs 2024-04-26T16:34:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17004.778258 secs 2024-04-26T16:34:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17001.953439 secs 2024-04-26T16:34:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17004.778258 secs 2024-04-26T16:34:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17001.953439 secs 2024-04-26T16:34:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17004.778258 secs 2024-04-26T16:34:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:24 smithi033 ceph-mon[27845]: pgmap v9560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:24 smithi096 ceph-mon[32753]: pgmap v9560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:24 smithi194 ceph-mon[33017]: pgmap v9560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:25 smithi096 ceph-mon[32753]: pgmap v9561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:25 smithi194 ceph-mon[33017]: pgmap v9561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:25 smithi033 ceph-mon[27845]: pgmap v9561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:28 smithi033 ceph-mon[27845]: pgmap v9562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17006.953539 secs 2024-04-26T16:34:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17009.778378 secs 2024-04-26T16:34:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:28 smithi096 ceph-mon[32753]: pgmap v9562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17006.953539 secs 2024-04-26T16:34:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17009.778378 secs 2024-04-26T16:34:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:28 smithi194 ceph-mon[33017]: pgmap v9562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17006.953539 secs 2024-04-26T16:34:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17009.778378 secs 2024-04-26T16:34:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:30.312+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:30.312+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:30.312+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:34:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:30 smithi033 ceph-mon[27845]: pgmap v9563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:30 smithi096 ceph-mon[32753]: pgmap v9563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:30 smithi194 ceph-mon[33017]: pgmap v9563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:31 smithi033 ceph-mon[27845]: pgmap v9564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:31 smithi096 ceph-mon[32753]: pgmap v9564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:31 smithi194 ceph-mon[33017]: pgmap v9564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17011.953638 secs 2024-04-26T16:34:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17011.953638 secs 2024-04-26T16:34:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17011.953638 secs 2024-04-26T16:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17014.778498 secs 2024-04-26T16:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:34 smithi033 ceph-mon[27845]: pgmap v9565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17014.778498 secs 2024-04-26T16:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:34 smithi096 ceph-mon[32753]: pgmap v9565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17014.778498 secs 2024-04-26T16:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:34 smithi194 ceph-mon[33017]: pgmap v9565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:34:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:34:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:34:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:36 smithi033 ceph-mon[27845]: pgmap v9566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:34:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:36 smithi033 ceph-mon[27845]: pgmap v9567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:36 smithi096 ceph-mon[32753]: pgmap v9566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:34:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:36 smithi096 ceph-mon[32753]: pgmap v9567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:36 smithi194 ceph-mon[33017]: pgmap v9566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:34:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:36 smithi194 ceph-mon[33017]: pgmap v9567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:38 smithi033 ceph-mon[27845]: pgmap v9568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17016.953738 secs 2024-04-26T16:34:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:38 smithi096 ceph-mon[32753]: pgmap v9568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17016.953738 secs 2024-04-26T16:34:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:38 smithi194 ceph-mon[33017]: pgmap v9568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17016.953738 secs 2024-04-26T16:34:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17019.778603 secs 2024-04-26T16:34:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17019.778603 secs 2024-04-26T16:34:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17019.778603 secs 2024-04-26T16:34:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:40 smithi033 ceph-mon[27845]: pgmap v9569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:34:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:40 smithi096 ceph-mon[32753]: pgmap v9569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:34:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:40 smithi194 ceph-mon[33017]: pgmap v9569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:34:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:42 smithi033 ceph-mon[27845]: pgmap v9570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:42 smithi096 ceph-mon[32753]: pgmap v9570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:42 smithi194 ceph-mon[33017]: pgmap v9570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17021.953838 secs 2024-04-26T16:34:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17021.953838 secs 2024-04-26T16:34:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17021.953838 secs 2024-04-26T16:34:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:44 smithi096 ceph-mon[32753]: pgmap v9571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17024.778697 secs 2024-04-26T16:34:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:44 smithi194 ceph-mon[33017]: pgmap v9571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:44.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17024.778697 secs 2024-04-26T16:34:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:44 smithi033 ceph-mon[27845]: pgmap v9571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:34:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17024.778697 secs 2024-04-26T16:34:45.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:45.321+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:45.321+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:34:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:34:45.321+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:34:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:46 smithi096 ceph-mon[32753]: pgmap v9572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:34:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:46 smithi194 ceph-mon[33017]: pgmap v9572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:34:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:46 smithi033 ceph-mon[27845]: pgmap v9572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:34:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:48 smithi194 ceph-mon[33017]: pgmap v9573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17026.953898 secs 2024-04-26T16:34:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:48 smithi033 ceph-mon[27845]: pgmap v9573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17026.953898 secs 2024-04-26T16:34:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:48 smithi096 ceph-mon[32753]: pgmap v9573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17026.953898 secs 2024-04-26T16:34:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17029.778814 secs 2024-04-26T16:34:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17029.778814 secs 2024-04-26T16:34:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17029.778814 secs 2024-04-26T16:34:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:50 smithi194 ceph-mon[33017]: pgmap v9574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:50 smithi033 ceph-mon[27845]: pgmap v9574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:50 smithi096 ceph-mon[32753]: pgmap v9574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:52 smithi194 ceph-mon[33017]: pgmap v9575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:52 smithi033 ceph-mon[27845]: pgmap v9575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:52 smithi096 ceph-mon[32753]: pgmap v9575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17031.953957 secs 2024-04-26T16:34:53.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17031.953957 secs 2024-04-26T16:34:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17031.953957 secs 2024-04-26T16:34:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:54 smithi194 ceph-mon[33017]: pgmap v9576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17034.778950 secs 2024-04-26T16:34:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:54 smithi033 ceph-mon[27845]: pgmap v9576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17034.778950 secs 2024-04-26T16:34:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:54 smithi096 ceph-mon[32753]: pgmap v9576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17034.778950 secs 2024-04-26T16:34:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:56 smithi194 ceph-mon[33017]: pgmap v9577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:56 smithi033 ceph-mon[27845]: pgmap v9577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:56 smithi096 ceph-mon[32753]: pgmap v9577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:58 smithi194 ceph-mon[33017]: pgmap v9578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17036.954040 secs 2024-04-26T16:34:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:58 smithi033 ceph-mon[27845]: pgmap v9578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17036.954040 secs 2024-04-26T16:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:58 smithi096 ceph-mon[32753]: pgmap v9578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17036.954040 secs 2024-04-26T16:34:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17039.779088 secs 2024-04-26T16:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17039.779088 secs 2024-04-26T16:34:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17039.779088 secs 2024-04-26T16:35:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:00.337+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:00.337+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:35:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:00.337+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:00.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:00 smithi033 ceph-mon[27845]: pgmap v9579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:00 smithi194 ceph-mon[33017]: pgmap v9579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:00 smithi096 ceph-mon[32753]: pgmap v9579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:02 smithi194 ceph-mon[33017]: pgmap v9580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:02 smithi033 ceph-mon[27845]: pgmap v9580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:02 smithi096 ceph-mon[32753]: pgmap v9580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17041.954135 secs 2024-04-26T16:35:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17041.954135 secs 2024-04-26T16:35:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17041.954135 secs 2024-04-26T16:35:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:04 smithi194 ceph-mon[33017]: pgmap v9581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17044.779201 secs 2024-04-26T16:35:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:04 smithi033 ceph-mon[27845]: pgmap v9581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17044.779201 secs 2024-04-26T16:35:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:04 smithi096 ceph-mon[32753]: pgmap v9581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17044.779201 secs 2024-04-26T16:35:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:06 smithi194 ceph-mon[33017]: pgmap v9582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:06 smithi033 ceph-mon[27845]: pgmap v9582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:06 smithi096 ceph-mon[32753]: pgmap v9582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:08 smithi194 ceph-mon[33017]: pgmap v9583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17046.954218 secs 2024-04-26T16:35:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:08 smithi033 ceph-mon[27845]: pgmap v9583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17046.954218 secs 2024-04-26T16:35:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:08 smithi096 ceph-mon[32753]: pgmap v9583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17046.954218 secs 2024-04-26T16:35:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17049.779269 secs 2024-04-26T16:35:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17049.779269 secs 2024-04-26T16:35:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17049.779269 secs 2024-04-26T16:35:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:10 smithi194 ceph-mon[33017]: pgmap v9584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:10 smithi033 ceph-mon[27845]: pgmap v9584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:10 smithi096 ceph-mon[32753]: pgmap v9584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:12 smithi194 ceph-mon[33017]: pgmap v9585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:12 smithi033 ceph-mon[27845]: pgmap v9585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:12 smithi096 ceph-mon[32753]: pgmap v9585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17051.954308 secs 2024-04-26T16:35:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17051.954308 secs 2024-04-26T16:35:13.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17051.954308 secs 2024-04-26T16:35:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:14 smithi194 ceph-mon[33017]: pgmap v9586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17054.779380 secs 2024-04-26T16:35:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:14 smithi033 ceph-mon[27845]: pgmap v9586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17054.779380 secs 2024-04-26T16:35:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:14 smithi096 ceph-mon[32753]: pgmap v9586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17054.779380 secs 2024-04-26T16:35:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:15.353+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:15.353+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:15.353+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:35:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:16 smithi194 ceph-mon[33017]: pgmap v9587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:16 smithi033 ceph-mon[27845]: pgmap v9587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:16 smithi096 ceph-mon[32753]: pgmap v9587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:18.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:18 smithi194 ceph-mon[33017]: pgmap v9588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:18.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17056.954386 secs 2024-04-26T16:35:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:18 smithi033 ceph-mon[27845]: pgmap v9588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17056.954386 secs 2024-04-26T16:35:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:18 smithi096 ceph-mon[32753]: pgmap v9588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17056.954386 secs 2024-04-26T16:35:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17059.779453 secs 2024-04-26T16:35:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17059.779453 secs 2024-04-26T16:35:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17059.779453 secs 2024-04-26T16:35:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:20 smithi194 ceph-mon[33017]: pgmap v9589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:20 smithi033 ceph-mon[27845]: pgmap v9589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:20 smithi096 ceph-mon[32753]: pgmap v9589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:22 smithi194 ceph-mon[33017]: pgmap v9590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:22 smithi033 ceph-mon[27845]: pgmap v9590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:22 smithi096 ceph-mon[32753]: pgmap v9590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17061.954468 secs 2024-04-26T16:35:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17061.954468 secs 2024-04-26T16:35:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17061.954468 secs 2024-04-26T16:35:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:24.695+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:35:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:24.695+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T16:35:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:24.696+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:35:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:24.696+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:35:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:24 smithi194 ceph-mon[33017]: pgmap v9591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17064.779569 secs 2024-04-26T16:35:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:24 smithi033 ceph-mon[27845]: pgmap v9591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17064.779569 secs 2024-04-26T16:35:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:24 smithi096 ceph-mon[32753]: pgmap v9591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17064.779569 secs 2024-04-26T16:35:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:26 smithi194 ceph-mon[33017]: pgmap v9592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:26 smithi033 ceph-mon[27845]: pgmap v9592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:26 smithi096 ceph-mon[32753]: pgmap v9592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:28 smithi033 ceph-mon[27845]: pgmap v9593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17066.954551 secs 2024-04-26T16:35:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:28 smithi096 ceph-mon[32753]: pgmap v9593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17066.954551 secs 2024-04-26T16:35:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:28 smithi194 ceph-mon[33017]: pgmap v9593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17066.954551 secs 2024-04-26T16:35:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17069.779688 secs 2024-04-26T16:35:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17069.779688 secs 2024-04-26T16:35:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17069.779688 secs 2024-04-26T16:35:30.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:30.367+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:30.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:30.367+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:30.367+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:35:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:30 smithi033 ceph-mon[27845]: pgmap v9594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:30 smithi096 ceph-mon[32753]: pgmap v9594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:30 smithi194 ceph-mon[33017]: pgmap v9594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:32 smithi033 ceph-mon[27845]: pgmap v9595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:32 smithi096 ceph-mon[32753]: pgmap v9595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:32 smithi194 ceph-mon[33017]: pgmap v9595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17071.954632 secs 2024-04-26T16:35:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17071.954632 secs 2024-04-26T16:35:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17071.954632 secs 2024-04-26T16:35:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:34 smithi033 ceph-mon[27845]: pgmap v9596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17074.779797 secs 2024-04-26T16:35:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:34 smithi096 ceph-mon[32753]: pgmap v9596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17074.779797 secs 2024-04-26T16:35:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:34 smithi194 ceph-mon[33017]: pgmap v9596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17074.779797 secs 2024-04-26T16:35:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:36 smithi033 ceph-mon[27845]: pgmap v9597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:36 smithi096 ceph-mon[32753]: pgmap v9597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:36 smithi194 ceph-mon[33017]: pgmap v9597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:35:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:35:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:35:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:38 smithi033 ceph-mon[27845]: pgmap v9598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:38 smithi033 ceph-mon[27845]: pgmap v9599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17076.954716 secs 2024-04-26T16:35:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17079.779915 secs 2024-04-26T16:35:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:38 smithi096 ceph-mon[32753]: pgmap v9598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:38 smithi096 ceph-mon[32753]: pgmap v9599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17076.954716 secs 2024-04-26T16:35:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17079.779915 secs 2024-04-26T16:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:38 smithi194 ceph-mon[33017]: pgmap v9598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:38 smithi194 ceph-mon[33017]: pgmap v9599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17076.954716 secs 2024-04-26T16:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17079.779915 secs 2024-04-26T16:35:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:40 smithi033 ceph-mon[27845]: pgmap v9600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:35:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:40 smithi096 ceph-mon[32753]: pgmap v9600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:35:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:40 smithi194 ceph-mon[33017]: pgmap v9600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:35:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:42 smithi033 ceph-mon[27845]: pgmap v9601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:42 smithi096 ceph-mon[32753]: pgmap v9601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:42 smithi194 ceph-mon[33017]: pgmap v9601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:44.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17081.954800 secs 2024-04-26T16:35:44.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17084.780032 secs 2024-04-26T16:35:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17081.954800 secs 2024-04-26T16:35:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17084.780032 secs 2024-04-26T16:35:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17081.954800 secs 2024-04-26T16:35:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17084.780032 secs 2024-04-26T16:35:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:44 smithi194 ceph-mon[33017]: pgmap v9602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:44 smithi033 ceph-mon[27845]: pgmap v9602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:44 smithi096 ceph-mon[32753]: pgmap v9602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:45.381+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:45.382+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:35:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:35:45.382+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:35:46.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:45 smithi096 ceph-mon[32753]: pgmap v9603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:45 smithi194 ceph-mon[33017]: pgmap v9603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:45 smithi033 ceph-mon[27845]: pgmap v9603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:35:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:48 smithi194 ceph-mon[33017]: pgmap v9604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:35:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:48 smithi033 ceph-mon[27845]: pgmap v9604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:35:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:48 smithi096 ceph-mon[32753]: pgmap v9604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:35:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17086.954885 secs 2024-04-26T16:35:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17089.780152 secs 2024-04-26T16:35:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17086.954885 secs 2024-04-26T16:35:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17089.780152 secs 2024-04-26T16:35:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17086.954885 secs 2024-04-26T16:35:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17089.780152 secs 2024-04-26T16:35:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:50 smithi194 ceph-mon[33017]: pgmap v9605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:50 smithi033 ceph-mon[27845]: pgmap v9605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:50 smithi096 ceph-mon[32753]: pgmap v9605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:52 smithi194 ceph-mon[33017]: pgmap v9606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:52 smithi033 ceph-mon[27845]: pgmap v9606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:52 smithi096 ceph-mon[32753]: pgmap v9606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17091.954991 secs 2024-04-26T16:35:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:54 smithi194 ceph-mon[33017]: pgmap v9607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17094.780290 secs 2024-04-26T16:35:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17091.954991 secs 2024-04-26T16:35:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:54 smithi033 ceph-mon[27845]: pgmap v9607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17094.780290 secs 2024-04-26T16:35:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17091.954991 secs 2024-04-26T16:35:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:54 smithi096 ceph-mon[32753]: pgmap v9607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17094.780290 secs 2024-04-26T16:35:56.343 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:56 smithi033 ceph-mon[27845]: pgmap v9608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:56 smithi194 ceph-mon[33017]: pgmap v9608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:56 smithi096 ceph-mon[32753]: pgmap v9608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:35:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:58 smithi194 ceph-mon[33017]: pgmap v9609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:58 smithi033 ceph-mon[27845]: pgmap v9609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:58 smithi096 ceph-mon[32753]: pgmap v9609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17096.955082 secs 2024-04-26T16:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17099.780406 secs 2024-04-26T16:35:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17096.955082 secs 2024-04-26T16:35:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17099.780406 secs 2024-04-26T16:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17096.955082 secs 2024-04-26T16:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17099.780406 secs 2024-04-26T16:36:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:00.391+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:00.391+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:36:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:00.391+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:00 smithi033 ceph-mon[27845]: pgmap v9610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:00 smithi096 ceph-mon[32753]: pgmap v9610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:00 smithi194 ceph-mon[33017]: pgmap v9610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:02 smithi033 ceph-mon[27845]: pgmap v9611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:02 smithi096 ceph-mon[32753]: pgmap v9611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:02 smithi194 ceph-mon[33017]: pgmap v9611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17101.955167 secs 2024-04-26T16:36:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:04 smithi033 ceph-mon[27845]: pgmap v9612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17104.780530 secs 2024-04-26T16:36:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17101.955167 secs 2024-04-26T16:36:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:04 smithi096 ceph-mon[32753]: pgmap v9612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17104.780530 secs 2024-04-26T16:36:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17101.955167 secs 2024-04-26T16:36:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:04 smithi194 ceph-mon[33017]: pgmap v9612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17104.780530 secs 2024-04-26T16:36:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:06 smithi033 ceph-mon[27845]: pgmap v9613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:06 smithi096 ceph-mon[32753]: pgmap v9613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:06 smithi194 ceph-mon[33017]: pgmap v9613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:08 smithi033 ceph-mon[27845]: pgmap v9614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:08 smithi096 ceph-mon[32753]: pgmap v9614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:08 smithi194 ceph-mon[33017]: pgmap v9614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17106.955268 secs 2024-04-26T16:36:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17109.780611 secs 2024-04-26T16:36:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17106.955268 secs 2024-04-26T16:36:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17109.780611 secs 2024-04-26T16:36:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17106.955268 secs 2024-04-26T16:36:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17109.780611 secs 2024-04-26T16:36:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:10 smithi033 ceph-mon[27845]: pgmap v9615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:10 smithi096 ceph-mon[32753]: pgmap v9615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:10 smithi194 ceph-mon[33017]: pgmap v9615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:12 smithi033 ceph-mon[27845]: pgmap v9616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:12 smithi096 ceph-mon[32753]: pgmap v9616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:12 smithi194 ceph-mon[33017]: pgmap v9616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:14.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17111.955356 secs 2024-04-26T16:36:14.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:14 smithi194 ceph-mon[33017]: pgmap v9617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:14.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17114.780745 secs 2024-04-26T16:36:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17111.955356 secs 2024-04-26T16:36:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:14 smithi033 ceph-mon[27845]: pgmap v9617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17114.780745 secs 2024-04-26T16:36:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17111.955356 secs 2024-04-26T16:36:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:14 smithi096 ceph-mon[32753]: pgmap v9617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17114.780745 secs 2024-04-26T16:36:15.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:15.404+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:15.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:15.404+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:36:15.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:15.404+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:16 smithi033 ceph-mon[27845]: pgmap v9618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:16 smithi096 ceph-mon[32753]: pgmap v9618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:16 smithi194 ceph-mon[33017]: pgmap v9618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:18 smithi033 ceph-mon[27845]: pgmap v9619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:18 smithi096 ceph-mon[32753]: pgmap v9619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:18 smithi194 ceph-mon[33017]: pgmap v9619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17116.955457 secs 2024-04-26T16:36:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17119.780884 secs 2024-04-26T16:36:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17116.955457 secs 2024-04-26T16:36:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17119.780884 secs 2024-04-26T16:36:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17116.955457 secs 2024-04-26T16:36:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17119.780884 secs 2024-04-26T16:36:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:20 smithi194 ceph-mon[33017]: pgmap v9620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:20 smithi033 ceph-mon[27845]: pgmap v9620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:20 smithi096 ceph-mon[32753]: pgmap v9620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:22 smithi194 ceph-mon[33017]: pgmap v9621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:22 smithi033 ceph-mon[27845]: pgmap v9621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:22 smithi096 ceph-mon[32753]: pgmap v9621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17121.955557 secs 2024-04-26T16:36:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17121.955557 secs 2024-04-26T16:36:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17121.955557 secs 2024-04-26T16:36:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:24 smithi033 ceph-mon[27845]: pgmap v9622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17124.780981 secs 2024-04-26T16:36:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:24 smithi096 ceph-mon[32753]: pgmap v9622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17124.780981 secs 2024-04-26T16:36:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:24 smithi194 ceph-mon[33017]: pgmap v9622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17124.780981 secs 2024-04-26T16:36:26.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:26 smithi033 ceph-mon[27845]: pgmap v9623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:26 smithi096 ceph-mon[32753]: pgmap v9623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:26 smithi194 ceph-mon[33017]: pgmap v9623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:28 smithi033 ceph-mon[27845]: pgmap v9624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17126.955657 secs 2024-04-26T16:36:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:28 smithi096 ceph-mon[32753]: pgmap v9624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17126.955657 secs 2024-04-26T16:36:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:28 smithi194 ceph-mon[33017]: pgmap v9624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17126.955657 secs 2024-04-26T16:36:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17129.781094 secs 2024-04-26T16:36:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17129.781094 secs 2024-04-26T16:36:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17129.781094 secs 2024-04-26T16:36:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:30.416+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:30.416+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:36:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:30.416+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:30 smithi033 ceph-mon[27845]: pgmap v9625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:30 smithi096 ceph-mon[32753]: pgmap v9625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:30 smithi194 ceph-mon[33017]: pgmap v9625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:32 smithi033 ceph-mon[27845]: pgmap v9626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:32.695 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:32 smithi096 ceph-mon[32753]: pgmap v9626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:32 smithi194 ceph-mon[33017]: pgmap v9626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17131.955755 secs 2024-04-26T16:36:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17131.955755 secs 2024-04-26T16:36:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17131.955755 secs 2024-04-26T16:36:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:34 smithi033 ceph-mon[27845]: pgmap v9627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17134.781184 secs 2024-04-26T16:36:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:34 smithi096 ceph-mon[32753]: pgmap v9627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17134.781184 secs 2024-04-26T16:36:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:34 smithi194 ceph-mon[33017]: pgmap v9627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17134.781184 secs 2024-04-26T16:36:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:36 smithi033 ceph-mon[27845]: pgmap v9628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:36 smithi096 ceph-mon[32753]: pgmap v9628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:36 smithi194 ceph-mon[33017]: pgmap v9628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:38 smithi033 ceph-mon[27845]: pgmap v9629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17136.955821 secs 2024-04-26T16:36:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:38 smithi096 ceph-mon[32753]: pgmap v9629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17136.955821 secs 2024-04-26T16:36:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:38 smithi194 ceph-mon[33017]: pgmap v9629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17136.955821 secs 2024-04-26T16:36:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17139.781280 secs 2024-04-26T16:36:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:36:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17139.781280 secs 2024-04-26T16:36:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:36:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17139.781280 secs 2024-04-26T16:36:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:36:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:40 smithi033 ceph-mon[27845]: pgmap v9630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:40 smithi033 ceph-mon[27845]: pgmap v9631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:40 smithi096 ceph-mon[32753]: pgmap v9630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:40 smithi096 ceph-mon[32753]: pgmap v9631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:40 smithi194 ceph-mon[33017]: pgmap v9630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:40 smithi194 ceph-mon[33017]: pgmap v9631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:42 smithi096 ceph-mon[32753]: pgmap v9632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:42 smithi194 ceph-mon[33017]: pgmap v9632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:42 smithi033 ceph-mon[27845]: pgmap v9632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17141.955880 secs 2024-04-26T16:36:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17141.955880 secs 2024-04-26T16:36:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17141.955880 secs 2024-04-26T16:36:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:44 smithi194 ceph-mon[33017]: pgmap v9633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17144.781418 secs 2024-04-26T16:36:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:44 smithi033 ceph-mon[27845]: pgmap v9633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17144.781418 secs 2024-04-26T16:36:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:44 smithi096 ceph-mon[32753]: pgmap v9633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17144.781418 secs 2024-04-26T16:36:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:45.431+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:45.431+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:36:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:36:45.431+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:36:46.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:46 smithi096 ceph-mon[32753]: pgmap v9634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:46 smithi194 ceph-mon[33017]: pgmap v9634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:46 smithi033 ceph-mon[27845]: pgmap v9634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:48 smithi194 ceph-mon[33017]: pgmap v9635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17146.955938 secs 2024-04-26T16:36:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:48 smithi033 ceph-mon[27845]: pgmap v9635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17146.955938 secs 2024-04-26T16:36:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:48 smithi096 ceph-mon[32753]: pgmap v9635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:36:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17146.955938 secs 2024-04-26T16:36:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17149.781569 secs 2024-04-26T16:36:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17149.781569 secs 2024-04-26T16:36:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17149.781569 secs 2024-04-26T16:36:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:50 smithi194 ceph-mon[33017]: pgmap v9636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:36:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:50 smithi033 ceph-mon[27845]: pgmap v9636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:36:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:50 smithi096 ceph-mon[32753]: pgmap v9636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:36:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:52 smithi194 ceph-mon[33017]: pgmap v9637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:52 smithi033 ceph-mon[27845]: pgmap v9637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:52 smithi096 ceph-mon[32753]: pgmap v9637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17151.955980 secs 2024-04-26T16:36:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17151.955980 secs 2024-04-26T16:36:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17151.955980 secs 2024-04-26T16:36:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:54 smithi194 ceph-mon[33017]: pgmap v9638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17154.781708 secs 2024-04-26T16:36:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:54 smithi033 ceph-mon[27845]: pgmap v9638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17154.781708 secs 2024-04-26T16:36:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:54 smithi096 ceph-mon[32753]: pgmap v9638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17154.781708 secs 2024-04-26T16:36:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:56 smithi194 ceph-mon[33017]: pgmap v9639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:56 smithi033 ceph-mon[27845]: pgmap v9639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:56 smithi096 ceph-mon[32753]: pgmap v9639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:36:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:58 smithi194 ceph-mon[33017]: pgmap v9640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17156.956040 secs 2024-04-26T16:36:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:58 smithi033 ceph-mon[27845]: pgmap v9640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17156.956040 secs 2024-04-26T16:36:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:58 smithi096 ceph-mon[32753]: pgmap v9640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:36:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17156.956040 secs 2024-04-26T16:36:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17159.781805 secs 2024-04-26T16:36:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17159.781805 secs 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17159.781805 secs 2024-04-26T16:37:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:00.440+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:00.440+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:00.440+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:37:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:00 smithi194 ceph-mon[33017]: pgmap v9641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:00 smithi033 ceph-mon[27845]: pgmap v9641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:00 smithi096 ceph-mon[32753]: pgmap v9641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:02 smithi194 ceph-mon[33017]: pgmap v9642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:02 smithi033 ceph-mon[27845]: pgmap v9642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:02 smithi096 ceph-mon[32753]: pgmap v9642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17161.956100 secs 2024-04-26T16:37:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17161.956100 secs 2024-04-26T16:37:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17161.956100 secs 2024-04-26T16:37:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:04 smithi194 ceph-mon[33017]: pgmap v9643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17164.781916 secs 2024-04-26T16:37:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:04 smithi033 ceph-mon[27845]: pgmap v9643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17164.781916 secs 2024-04-26T16:37:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:04 smithi096 ceph-mon[32753]: pgmap v9643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17164.781916 secs 2024-04-26T16:37:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:06 smithi194 ceph-mon[33017]: pgmap v9644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:06 smithi033 ceph-mon[27845]: pgmap v9644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:06 smithi096 ceph-mon[32753]: pgmap v9644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:08 smithi194 ceph-mon[33017]: pgmap v9645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17166.956158 secs 2024-04-26T16:37:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:08 smithi033 ceph-mon[27845]: pgmap v9645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17166.956158 secs 2024-04-26T16:37:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:08 smithi096 ceph-mon[32753]: pgmap v9645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17166.956158 secs 2024-04-26T16:37:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17169.782070 secs 2024-04-26T16:37:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17169.782070 secs 2024-04-26T16:37:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17169.782070 secs 2024-04-26T16:37:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:10 smithi194 ceph-mon[33017]: pgmap v9646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:10 smithi033 ceph-mon[27845]: pgmap v9646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:10 smithi096 ceph-mon[32753]: pgmap v9646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:12 smithi194 ceph-mon[33017]: pgmap v9647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:12 smithi033 ceph-mon[27845]: pgmap v9647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:12 smithi096 ceph-mon[32753]: pgmap v9647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17171.956220 secs 2024-04-26T16:37:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17171.956220 secs 2024-04-26T16:37:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17171.956220 secs 2024-04-26T16:37:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:14 smithi194 ceph-mon[33017]: pgmap v9648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17174.782150 secs 2024-04-26T16:37:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:14 smithi033 ceph-mon[27845]: pgmap v9648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17174.782150 secs 2024-04-26T16:37:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:14 smithi096 ceph-mon[32753]: pgmap v9648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17174.782150 secs 2024-04-26T16:37:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:15.446+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:15.446+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:15.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:15.446+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:37:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:16 smithi194 ceph-mon[33017]: pgmap v9649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:16 smithi033 ceph-mon[27845]: pgmap v9649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:16 smithi096 ceph-mon[32753]: pgmap v9649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:18 smithi194 ceph-mon[33017]: pgmap v9650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17176.956309 secs 2024-04-26T16:37:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:18 smithi033 ceph-mon[27845]: pgmap v9650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17176.956309 secs 2024-04-26T16:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:18 smithi096 ceph-mon[32753]: pgmap v9650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17176.956309 secs 2024-04-26T16:37:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17179.782237 secs 2024-04-26T16:37:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17179.782237 secs 2024-04-26T16:37:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17179.782237 secs 2024-04-26T16:37:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:20 smithi194 ceph-mon[33017]: pgmap v9651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:20 smithi033 ceph-mon[27845]: pgmap v9651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:20 smithi096 ceph-mon[32753]: pgmap v9651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:22 smithi194 ceph-mon[33017]: pgmap v9652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:22 smithi033 ceph-mon[27845]: pgmap v9652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:22 smithi096 ceph-mon[32753]: pgmap v9652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17181.956369 secs 2024-04-26T16:37:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17181.956369 secs 2024-04-26T16:37:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17181.956369 secs 2024-04-26T16:37:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:24 smithi033 ceph-mon[27845]: pgmap v9653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17184.782375 secs 2024-04-26T16:37:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:24 smithi096 ceph-mon[32753]: pgmap v9653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17184.782375 secs 2024-04-26T16:37:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:24 smithi194 ceph-mon[33017]: pgmap v9653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17184.782375 secs 2024-04-26T16:37:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:26 smithi033 ceph-mon[27845]: pgmap v9654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:26 smithi096 ceph-mon[32753]: pgmap v9654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:26 smithi194 ceph-mon[33017]: pgmap v9654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:37:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:28 smithi033 ceph-mon[27845]: pgmap v9655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17186.956453 secs 2024-04-26T16:37:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:28 smithi096 ceph-mon[32753]: pgmap v9655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17186.956453 secs 2024-04-26T16:37:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:28 smithi194 ceph-mon[33017]: pgmap v9655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17186.956453 secs 2024-04-26T16:37:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17189.782487 secs 2024-04-26T16:37:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17189.782487 secs 2024-04-26T16:37:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17189.782487 secs 2024-04-26T16:37:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:30.460+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:30.460+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:37:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:30.460+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:30 smithi033 ceph-mon[27845]: pgmap v9656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:30 smithi096 ceph-mon[32753]: pgmap v9656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:30 smithi194 ceph-mon[33017]: pgmap v9656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:32 smithi033 ceph-mon[27845]: pgmap v9657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:37:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:32 smithi096 ceph-mon[32753]: pgmap v9657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:37:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:32 smithi194 ceph-mon[33017]: pgmap v9657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:37:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17191.956535 secs 2024-04-26T16:37:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17191.956535 secs 2024-04-26T16:37:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17191.956535 secs 2024-04-26T16:37:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:34 smithi033 ceph-mon[27845]: pgmap v9658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17194.782609 secs 2024-04-26T16:37:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:34 smithi096 ceph-mon[32753]: pgmap v9658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17194.782609 secs 2024-04-26T16:37:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:34 smithi194 ceph-mon[33017]: pgmap v9658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17194.782609 secs 2024-04-26T16:37:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:36 smithi033 ceph-mon[27845]: pgmap v9659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:37:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:36 smithi096 ceph-mon[32753]: pgmap v9659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:37:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:36 smithi194 ceph-mon[33017]: pgmap v9659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:37:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:38 smithi033 ceph-mon[27845]: pgmap v9660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17196.956618 secs 2024-04-26T16:37:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:38 smithi096 ceph-mon[32753]: pgmap v9660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17196.956618 secs 2024-04-26T16:37:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:38 smithi194 ceph-mon[33017]: pgmap v9660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17196.956618 secs 2024-04-26T16:37:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17199.782729 secs 2024-04-26T16:37:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17199.782729 secs 2024-04-26T16:37:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17199.782729 secs 2024-04-26T16:37:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:40 smithi194 ceph-mon[33017]: pgmap v9661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:37:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:40 smithi033 ceph-mon[27845]: pgmap v9661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:37:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:40 smithi096 ceph-mon[32753]: pgmap v9661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:37:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:41 smithi194 ceph-mon[33017]: pgmap v9662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T16:37:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:41 smithi194 ceph-mon[33017]: pgmap v9663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:41 smithi033 ceph-mon[27845]: pgmap v9662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T16:37:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:41 smithi033 ceph-mon[27845]: pgmap v9663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:41 smithi096 ceph-mon[32753]: pgmap v9662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T16:37:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:41 smithi096 ceph-mon[32753]: pgmap v9663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:43 smithi194 ceph-mon[33017]: pgmap v9664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17201.956703 secs 2024-04-26T16:37:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17204.782854 secs 2024-04-26T16:37:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:43 smithi033 ceph-mon[27845]: pgmap v9664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17201.956703 secs 2024-04-26T16:37:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17204.782854 secs 2024-04-26T16:37:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:43 smithi096 ceph-mon[32753]: pgmap v9664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17201.956703 secs 2024-04-26T16:37:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17204.782854 secs 2024-04-26T16:37:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:45.470+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:45.470+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:37:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:37:45.470+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:37:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:45 smithi194 ceph-mon[33017]: pgmap v9665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s 2024-04-26T16:37:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:45 smithi033 ceph-mon[27845]: pgmap v9665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s 2024-04-26T16:37:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:45 smithi096 ceph-mon[32753]: pgmap v9665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s 2024-04-26T16:37:47.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:46 smithi096 ceph-mon[32753]: pgmap v9666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:46 smithi194 ceph-mon[33017]: pgmap v9666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:46 smithi033 ceph-mon[27845]: pgmap v9666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17206.956788 secs 2024-04-26T16:37:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17206.956788 secs 2024-04-26T16:37:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17206.956788 secs 2024-04-26T16:37:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17209.782951 secs 2024-04-26T16:37:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:49 smithi033 ceph-mon[27845]: pgmap v9667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17209.782951 secs 2024-04-26T16:37:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:49 smithi096 ceph-mon[32753]: pgmap v9667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17209.782951 secs 2024-04-26T16:37:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:49 smithi194 ceph-mon[33017]: pgmap v9667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:51 smithi194 ceph-mon[33017]: pgmap v9668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:37:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:51 smithi033 ceph-mon[27845]: pgmap v9668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:37:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:51 smithi096 ceph-mon[32753]: pgmap v9668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:37:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:53 smithi194 ceph-mon[33017]: pgmap v9669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17211.956849 secs 2024-04-26T16:37:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:53 smithi033 ceph-mon[27845]: pgmap v9669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17211.956849 secs 2024-04-26T16:37:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:53 smithi096 ceph-mon[32753]: pgmap v9669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:37:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17211.956849 secs 2024-04-26T16:37:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17214.783069 secs 2024-04-26T16:37:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17214.783069 secs 2024-04-26T16:37:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17214.783069 secs 2024-04-26T16:37:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:55 smithi194 ceph-mon[33017]: pgmap v9670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:55 smithi033 ceph-mon[27845]: pgmap v9670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:55 smithi096 ceph-mon[32753]: pgmap v9670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:37:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:57 smithi194 ceph-mon[33017]: pgmap v9671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:57 smithi033 ceph-mon[27845]: pgmap v9671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:57 smithi096 ceph-mon[32753]: pgmap v9671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:37:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17216.956912 secs 2024-04-26T16:37:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17216.956912 secs 2024-04-26T16:37:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17216.956912 secs 2024-04-26T16:37:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17219.783185 secs 2024-04-26T16:37:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:37:59 smithi194 ceph-mon[33017]: pgmap v9672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17219.783185 secs 2024-04-26T16:37:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:37:59 smithi033 ceph-mon[27845]: pgmap v9672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:37:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17219.783185 secs 2024-04-26T16:37:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:37:59 smithi096 ceph-mon[32753]: pgmap v9672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:00.476+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:00.476+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:38:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:00.476+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:01 smithi194 ceph-mon[33017]: pgmap v9673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:01 smithi033 ceph-mon[27845]: pgmap v9673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:01 smithi096 ceph-mon[32753]: pgmap v9673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:03 smithi194 ceph-mon[33017]: pgmap v9674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17221.956993 secs 2024-04-26T16:38:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:03 smithi033 ceph-mon[27845]: pgmap v9674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17221.956993 secs 2024-04-26T16:38:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:03 smithi096 ceph-mon[32753]: pgmap v9674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17221.956993 secs 2024-04-26T16:38:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17224.783310 secs 2024-04-26T16:38:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17224.783310 secs 2024-04-26T16:38:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17224.783310 secs 2024-04-26T16:38:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:05 smithi194 ceph-mon[33017]: pgmap v9675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:05 smithi033 ceph-mon[27845]: pgmap v9675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:05 smithi096 ceph-mon[32753]: pgmap v9675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:07 smithi194 ceph-mon[33017]: pgmap v9676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:07 smithi033 ceph-mon[27845]: pgmap v9676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:07 smithi096 ceph-mon[32753]: pgmap v9676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17226.957055 secs 2024-04-26T16:38:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17226.957055 secs 2024-04-26T16:38:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17226.957055 secs 2024-04-26T16:38:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17229.783449 secs 2024-04-26T16:38:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:09 smithi194 ceph-mon[33017]: pgmap v9677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17229.783449 secs 2024-04-26T16:38:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:09 smithi033 ceph-mon[27845]: pgmap v9677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17229.783449 secs 2024-04-26T16:38:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:09 smithi096 ceph-mon[32753]: pgmap v9677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:11 smithi194 ceph-mon[33017]: pgmap v9678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:11 smithi033 ceph-mon[27845]: pgmap v9678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:11 smithi096 ceph-mon[32753]: pgmap v9678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:13 smithi194 ceph-mon[33017]: pgmap v9679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17231.957114 secs 2024-04-26T16:38:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:13 smithi033 ceph-mon[27845]: pgmap v9679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17231.957114 secs 2024-04-26T16:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:13 smithi096 ceph-mon[32753]: pgmap v9679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17231.957114 secs 2024-04-26T16:38:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17234.783560 secs 2024-04-26T16:38:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17234.783560 secs 2024-04-26T16:38:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17234.783560 secs 2024-04-26T16:38:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:15.490+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:15.490+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:15.490+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:38:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:15 smithi194 ceph-mon[33017]: pgmap v9680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:15 smithi033 ceph-mon[27845]: pgmap v9680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:15 smithi096 ceph-mon[32753]: pgmap v9680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:17 smithi194 ceph-mon[33017]: pgmap v9681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:17 smithi033 ceph-mon[27845]: pgmap v9681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:17 smithi096 ceph-mon[32753]: pgmap v9681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17236.957197 secs 2024-04-26T16:38:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17236.957197 secs 2024-04-26T16:38:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17236.957197 secs 2024-04-26T16:38:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17239.783693 secs 2024-04-26T16:38:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:19 smithi194 ceph-mon[33017]: pgmap v9682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17239.783693 secs 2024-04-26T16:38:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:19 smithi033 ceph-mon[27845]: pgmap v9682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17239.783693 secs 2024-04-26T16:38:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:19 smithi096 ceph-mon[32753]: pgmap v9682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:21 smithi194 ceph-mon[33017]: pgmap v9683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:21 smithi033 ceph-mon[27845]: pgmap v9683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:21 smithi096 ceph-mon[32753]: pgmap v9683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:23 smithi033 ceph-mon[27845]: pgmap v9684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17241.957281 secs 2024-04-26T16:38:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:23 smithi096 ceph-mon[32753]: pgmap v9684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17241.957281 secs 2024-04-26T16:38:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:23 smithi194 ceph-mon[33017]: pgmap v9684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17241.957281 secs 2024-04-26T16:38:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.725+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T16:38:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.725+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:38:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.726+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T16:38:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.726+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T16:38:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.726+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T16:38:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.726+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T16:38:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.726+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T16:38:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:24.727+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T16:38:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17244.783839 secs 2024-04-26T16:38:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17244.783839 secs 2024-04-26T16:38:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17244.783839 secs 2024-04-26T16:38:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:25 smithi033 ceph-mon[27845]: pgmap v9685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:25 smithi096 ceph-mon[32753]: pgmap v9685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:25 smithi194 ceph-mon[33017]: pgmap v9685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:27 smithi033 ceph-mon[27845]: pgmap v9686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:27 smithi096 ceph-mon[32753]: pgmap v9686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:27 smithi194 ceph-mon[33017]: pgmap v9686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17246.957369 secs 2024-04-26T16:38:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17246.957369 secs 2024-04-26T16:38:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17246.957369 secs 2024-04-26T16:38:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17249.783974 secs 2024-04-26T16:38:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:29 smithi033 ceph-mon[27845]: pgmap v9687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17249.783974 secs 2024-04-26T16:38:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:29 smithi096 ceph-mon[32753]: pgmap v9687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17249.783974 secs 2024-04-26T16:38:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:29 smithi194 ceph-mon[33017]: pgmap v9687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:30.503+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:30.503+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:38:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:30.503+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:31 smithi033 ceph-mon[27845]: pgmap v9688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:31 smithi096 ceph-mon[32753]: pgmap v9688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:31 smithi194 ceph-mon[33017]: pgmap v9688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:33 smithi033 ceph-mon[27845]: pgmap v9689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17251.957451 secs 2024-04-26T16:38:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:33 smithi096 ceph-mon[32753]: pgmap v9689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17251.957451 secs 2024-04-26T16:38:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:33 smithi194 ceph-mon[33017]: pgmap v9689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17251.957451 secs 2024-04-26T16:38:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17254.784127 secs 2024-04-26T16:38:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17254.784127 secs 2024-04-26T16:38:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17254.784127 secs 2024-04-26T16:38:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:35 smithi033 ceph-mon[27845]: pgmap v9690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:35 smithi096 ceph-mon[32753]: pgmap v9690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:35 smithi194 ceph-mon[33017]: pgmap v9690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:37 smithi033 ceph-mon[27845]: pgmap v9691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:37 smithi096 ceph-mon[32753]: pgmap v9691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:37 smithi194 ceph-mon[33017]: pgmap v9691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17256.957533 secs 2024-04-26T16:38:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17256.957533 secs 2024-04-26T16:38:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17256.957533 secs 2024-04-26T16:38:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17259.784241 secs 2024-04-26T16:38:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:39 smithi096 ceph-mon[32753]: pgmap v9692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17259.784241 secs 2024-04-26T16:38:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:39 smithi194 ceph-mon[33017]: pgmap v9692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17259.784241 secs 2024-04-26T16:38:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:39 smithi033 ceph-mon[27845]: pgmap v9692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:41 smithi096 ceph-mon[32753]: pgmap v9693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:41 smithi194 ceph-mon[33017]: pgmap v9693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:41 smithi033 ceph-mon[27845]: pgmap v9693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:38:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:38:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:38:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:43 smithi194 ceph-mon[33017]: pgmap v9694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:43 smithi194 ceph-mon[33017]: pgmap v9695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17261.957617 secs 2024-04-26T16:38:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17264.784357 secs 2024-04-26T16:38:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:43 smithi033 ceph-mon[27845]: pgmap v9694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:43 smithi033 ceph-mon[27845]: pgmap v9695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17261.957617 secs 2024-04-26T16:38:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17264.784357 secs 2024-04-26T16:38:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:43 smithi096 ceph-mon[32753]: pgmap v9694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:43 smithi096 ceph-mon[32753]: pgmap v9695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17261.957617 secs 2024-04-26T16:38:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17264.784357 secs 2024-04-26T16:38:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:45.507+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:45.507+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:38:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:38:45.507+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:38:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:45 smithi194 ceph-mon[33017]: pgmap v9696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T16:38:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:45 smithi033 ceph-mon[27845]: pgmap v9696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T16:38:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:45 smithi096 ceph-mon[32753]: pgmap v9696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T16:38:47.983 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:47 smithi096 ceph-mon[32753]: pgmap v9697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:47 smithi194 ceph-mon[33017]: pgmap v9697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:47 smithi033 ceph-mon[27845]: pgmap v9697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17266.957701 secs 2024-04-26T16:38:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17269.784459 secs 2024-04-26T16:38:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17266.957701 secs 2024-04-26T16:38:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17269.784459 secs 2024-04-26T16:38:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17266.957701 secs 2024-04-26T16:38:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17269.784459 secs 2024-04-26T16:38:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:49 smithi194 ceph-mon[33017]: pgmap v9698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:49 smithi033 ceph-mon[27845]: pgmap v9698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:49 smithi096 ceph-mon[32753]: pgmap v9698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:50 smithi194 ceph-mon[33017]: pgmap v9699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:50 smithi033 ceph-mon[27845]: pgmap v9699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:50 smithi096 ceph-mon[32753]: pgmap v9699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:38:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:53 smithi194 ceph-mon[33017]: pgmap v9700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:38:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:53 smithi033 ceph-mon[27845]: pgmap v9700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:38:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:53 smithi096 ceph-mon[32753]: pgmap v9700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:38:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17271.957784 secs 2024-04-26T16:38:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17274.784588 secs 2024-04-26T16:38:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17271.957784 secs 2024-04-26T16:38:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17274.784588 secs 2024-04-26T16:38:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17271.957784 secs 2024-04-26T16:38:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17274.784588 secs 2024-04-26T16:38:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:55 smithi194 ceph-mon[33017]: pgmap v9701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:55 smithi033 ceph-mon[27845]: pgmap v9701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:55 smithi096 ceph-mon[32753]: pgmap v9701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:57 smithi194 ceph-mon[33017]: pgmap v9702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:57 smithi033 ceph-mon[27845]: pgmap v9702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:57 smithi096 ceph-mon[32753]: pgmap v9702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17276.957864 secs 2024-04-26T16:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:59 smithi194 ceph-mon[33017]: pgmap v9703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17279.784732 secs 2024-04-26T16:38:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17276.957864 secs 2024-04-26T16:38:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:59 smithi033 ceph-mon[27845]: pgmap v9703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17279.784732 secs 2024-04-26T16:38:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17276.957864 secs 2024-04-26T16:38:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:59 smithi096 ceph-mon[32753]: pgmap v9703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:38:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17279.784732 secs 2024-04-26T16:39:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:00.520+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:00.520+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:00.521+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:39:01.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:01 smithi033 ceph-mon[27845]: pgmap v9704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:01 smithi194 ceph-mon[33017]: pgmap v9704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:01 smithi096 ceph-mon[32753]: pgmap v9704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:03 smithi194 ceph-mon[33017]: pgmap v9705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:03 smithi033 ceph-mon[27845]: pgmap v9705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:03 smithi096 ceph-mon[32753]: pgmap v9705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17281.957923 secs 2024-04-26T16:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17284.784881 secs 2024-04-26T16:39:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17281.957923 secs 2024-04-26T16:39:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17284.784881 secs 2024-04-26T16:39:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17281.957923 secs 2024-04-26T16:39:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17284.784881 secs 2024-04-26T16:39:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:05 smithi194 ceph-mon[33017]: pgmap v9706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:05 smithi033 ceph-mon[27845]: pgmap v9706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:05 smithi096 ceph-mon[32753]: pgmap v9706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:07 smithi194 ceph-mon[33017]: pgmap v9707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:07 smithi033 ceph-mon[27845]: pgmap v9707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:07 smithi096 ceph-mon[32753]: pgmap v9707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17286.957985 secs 2024-04-26T16:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:09 smithi194 ceph-mon[33017]: pgmap v9708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17289.785007 secs 2024-04-26T16:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17286.957985 secs 2024-04-26T16:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:09 smithi033 ceph-mon[27845]: pgmap v9708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17289.785007 secs 2024-04-26T16:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17286.957985 secs 2024-04-26T16:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:09 smithi096 ceph-mon[32753]: pgmap v9708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17289.785007 secs 2024-04-26T16:39:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:11 smithi194 ceph-mon[33017]: pgmap v9709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:11 smithi033 ceph-mon[27845]: pgmap v9709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:11 smithi096 ceph-mon[32753]: pgmap v9709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:13 smithi194 ceph-mon[33017]: pgmap v9710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:13 smithi033 ceph-mon[27845]: pgmap v9710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:13 smithi096 ceph-mon[32753]: pgmap v9710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17291.958040 secs 2024-04-26T16:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17294.785152 secs 2024-04-26T16:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17291.958040 secs 2024-04-26T16:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17294.785152 secs 2024-04-26T16:39:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17291.958040 secs 2024-04-26T16:39:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17294.785152 secs 2024-04-26T16:39:15.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:15.536+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:15.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:15.536+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:39:15.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:15.536+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:15 smithi194 ceph-mon[33017]: pgmap v9711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:15 smithi033 ceph-mon[27845]: pgmap v9711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:15 smithi096 ceph-mon[32753]: pgmap v9711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:17 smithi194 ceph-mon[33017]: pgmap v9712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:17 smithi033 ceph-mon[27845]: pgmap v9712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:17 smithi096 ceph-mon[32753]: pgmap v9712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17296.958102 secs 2024-04-26T16:39:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:19 smithi033 ceph-mon[27845]: pgmap v9713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17299.785280 secs 2024-04-26T16:39:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17296.958102 secs 2024-04-26T16:39:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:19 smithi194 ceph-mon[33017]: pgmap v9713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17299.785280 secs 2024-04-26T16:39:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17296.958102 secs 2024-04-26T16:39:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:19 smithi096 ceph-mon[32753]: pgmap v9713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17299.785280 secs 2024-04-26T16:39:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:21 smithi194 ceph-mon[33017]: pgmap v9714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:21 smithi033 ceph-mon[27845]: pgmap v9714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:21 smithi096 ceph-mon[32753]: pgmap v9714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:23 smithi194 ceph-mon[33017]: pgmap v9715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:23 smithi033 ceph-mon[27845]: pgmap v9715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:23 smithi096 ceph-mon[32753]: pgmap v9715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17301.958163 secs 2024-04-26T16:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17304.785417 secs 2024-04-26T16:39:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17301.958163 secs 2024-04-26T16:39:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17304.785417 secs 2024-04-26T16:39:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17301.958163 secs 2024-04-26T16:39:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17304.785417 secs 2024-04-26T16:39:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:25 smithi194 ceph-mon[33017]: pgmap v9716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:25 smithi033 ceph-mon[27845]: pgmap v9716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:25 smithi096 ceph-mon[32753]: pgmap v9716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:27 smithi194 ceph-mon[33017]: pgmap v9717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:27 smithi033 ceph-mon[27845]: pgmap v9717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:27 smithi096 ceph-mon[32753]: pgmap v9717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17306.958240 secs 2024-04-26T16:39:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:29 smithi194 ceph-mon[33017]: pgmap v9718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17309.785559 secs 2024-04-26T16:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17306.958240 secs 2024-04-26T16:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:29 smithi033 ceph-mon[27845]: pgmap v9718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17309.785559 secs 2024-04-26T16:39:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17306.958240 secs 2024-04-26T16:39:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:29 smithi096 ceph-mon[32753]: pgmap v9718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17309.785559 secs 2024-04-26T16:39:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:30.551+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:30.551+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:39:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:30.551+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:31 smithi194 ceph-mon[33017]: pgmap v9719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:31 smithi033 ceph-mon[27845]: pgmap v9719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:31 smithi096 ceph-mon[32753]: pgmap v9719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:33 smithi194 ceph-mon[33017]: pgmap v9720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:33 smithi033 ceph-mon[27845]: pgmap v9720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:33 smithi096 ceph-mon[32753]: pgmap v9720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17311.958295 secs 2024-04-26T16:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17314.786058 secs 2024-04-26T16:39:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17311.958295 secs 2024-04-26T16:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17314.786058 secs 2024-04-26T16:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17311.958295 secs 2024-04-26T16:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17314.786058 secs 2024-04-26T16:39:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:35 smithi194 ceph-mon[33017]: pgmap v9721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:35 smithi033 ceph-mon[27845]: pgmap v9721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:35 smithi096 ceph-mon[32753]: pgmap v9721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:37 smithi194 ceph-mon[33017]: pgmap v9722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:37 smithi033 ceph-mon[27845]: pgmap v9722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:37 smithi096 ceph-mon[32753]: pgmap v9722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17316.958377 secs 2024-04-26T16:39:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17316.958377 secs 2024-04-26T16:39:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17316.958377 secs 2024-04-26T16:39:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:39 smithi033 ceph-mon[27845]: pgmap v9723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17319.786179 secs 2024-04-26T16:39:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:39 smithi096 ceph-mon[32753]: pgmap v9723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17319.786179 secs 2024-04-26T16:39:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:39 smithi194 ceph-mon[33017]: pgmap v9723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17319.786179 secs 2024-04-26T16:39:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:41 smithi033 ceph-mon[27845]: pgmap v9724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:41 smithi096 ceph-mon[32753]: pgmap v9724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:41 smithi194 ceph-mon[33017]: pgmap v9724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:43 smithi033 ceph-mon[27845]: pgmap v9725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17321.958447 secs 2024-04-26T16:39:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:43 smithi096 ceph-mon[32753]: pgmap v9725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17321.958447 secs 2024-04-26T16:39:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:43 smithi194 ceph-mon[33017]: pgmap v9725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17321.958447 secs 2024-04-26T16:39:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17324.786302 secs 2024-04-26T16:39:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:39:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17324.786302 secs 2024-04-26T16:39:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:39:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17324.786302 secs 2024-04-26T16:39:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:39:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:45.566+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:45.566+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:39:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:39:45.566+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:39:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:45 smithi033 ceph-mon[27845]: pgmap v9726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:45 smithi033 ceph-mon[27845]: pgmap v9727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:45 smithi096 ceph-mon[32753]: pgmap v9726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:45 smithi096 ceph-mon[32753]: pgmap v9727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:45.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:45 smithi194 ceph-mon[33017]: pgmap v9726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:45.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:45 smithi194 ceph-mon[33017]: pgmap v9727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:47 smithi033 ceph-mon[27845]: pgmap v9728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:47 smithi096 ceph-mon[32753]: pgmap v9728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:47 smithi194 ceph-mon[33017]: pgmap v9728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17326.958504 secs 2024-04-26T16:39:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17326.958504 secs 2024-04-26T16:39:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17326.958504 secs 2024-04-26T16:39:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:49 smithi033 ceph-mon[27845]: pgmap v9729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17329.786433 secs 2024-04-26T16:39:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:49 smithi096 ceph-mon[32753]: pgmap v9729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17329.786433 secs 2024-04-26T16:39:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:49 smithi194 ceph-mon[33017]: pgmap v9729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17329.786433 secs 2024-04-26T16:39:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:51 smithi033 ceph-mon[27845]: pgmap v9730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:51 smithi096 ceph-mon[32753]: pgmap v9730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:51 smithi194 ceph-mon[33017]: pgmap v9730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:53 smithi033 ceph-mon[27845]: pgmap v9731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17331.958564 secs 2024-04-26T16:39:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:53 smithi096 ceph-mon[32753]: pgmap v9731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17331.958564 secs 2024-04-26T16:39:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:53 smithi194 ceph-mon[33017]: pgmap v9731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:39:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17331.958564 secs 2024-04-26T16:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17334.786516 secs 2024-04-26T16:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17334.786516 secs 2024-04-26T16:39:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17334.786516 secs 2024-04-26T16:39:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:55 smithi033 ceph-mon[27845]: pgmap v9732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:39:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:55 smithi096 ceph-mon[32753]: pgmap v9732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:39:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:55 smithi194 ceph-mon[33017]: pgmap v9732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:39:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:57 smithi096 ceph-mon[32753]: pgmap v9733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:57 smithi194 ceph-mon[33017]: pgmap v9733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:57 smithi033 ceph-mon[27845]: pgmap v9733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:39:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17336.958646 secs 2024-04-26T16:39:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17336.958646 secs 2024-04-26T16:39:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17336.958646 secs 2024-04-26T16:39:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:59 smithi194 ceph-mon[33017]: pgmap v9734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17339.786630 secs 2024-04-26T16:39:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:59 smithi033 ceph-mon[27845]: pgmap v9734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17339.786630 secs 2024-04-26T16:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:59 smithi096 ceph-mon[32753]: pgmap v9734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17339.786630 secs 2024-04-26T16:40:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:00.578+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:00.578+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:40:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:00.578+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:40:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:40:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:40:01.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:01 smithi033 ceph-mon[27845]: pgmap v9735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:01 smithi194 ceph-mon[33017]: pgmap v9735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:01 smithi096 ceph-mon[32753]: pgmap v9735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:03 smithi194 ceph-mon[33017]: pgmap v9736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17341.958728 secs 2024-04-26T16:40:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:03 smithi033 ceph-mon[27845]: pgmap v9736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17341.958728 secs 2024-04-26T16:40:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:03 smithi096 ceph-mon[32753]: pgmap v9736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17341.958728 secs 2024-04-26T16:40:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17344.786744 secs 2024-04-26T16:40:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17344.786744 secs 2024-04-26T16:40:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17344.786744 secs 2024-04-26T16:40:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:05 smithi194 ceph-mon[33017]: pgmap v9737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:05 smithi033 ceph-mon[27845]: pgmap v9737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:05 smithi096 ceph-mon[32753]: pgmap v9737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:07 smithi194 ceph-mon[33017]: pgmap v9738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:07 smithi033 ceph-mon[27845]: pgmap v9738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:07 smithi096 ceph-mon[32753]: pgmap v9738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17346.958810 secs 2024-04-26T16:40:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17346.958810 secs 2024-04-26T16:40:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17346.958810 secs 2024-04-26T16:40:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:09 smithi194 ceph-mon[33017]: pgmap v9739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17349.786856 secs 2024-04-26T16:40:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:09 smithi033 ceph-mon[27845]: pgmap v9739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17349.786856 secs 2024-04-26T16:40:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:09 smithi096 ceph-mon[32753]: pgmap v9739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17349.786856 secs 2024-04-26T16:40:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:11 smithi194 ceph-mon[33017]: pgmap v9740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:11 smithi033 ceph-mon[27845]: pgmap v9740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:11 smithi096 ceph-mon[32753]: pgmap v9740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:13 smithi194 ceph-mon[33017]: pgmap v9741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17351.958896 secs 2024-04-26T16:40:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:13 smithi033 ceph-mon[27845]: pgmap v9741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17351.958896 secs 2024-04-26T16:40:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:13 smithi096 ceph-mon[32753]: pgmap v9741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17351.958896 secs 2024-04-26T16:40:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17354.786981 secs 2024-04-26T16:40:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17354.786981 secs 2024-04-26T16:40:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17354.786981 secs 2024-04-26T16:40:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:15.593+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:15.594+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:15.594+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:40:15.769 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:15 smithi194 ceph-mon[33017]: pgmap v9742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:15 smithi033 ceph-mon[27845]: pgmap v9742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:15 smithi096 ceph-mon[32753]: pgmap v9742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:17 smithi194 ceph-mon[33017]: pgmap v9743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:17 smithi033 ceph-mon[27845]: pgmap v9743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:17 smithi096 ceph-mon[32753]: pgmap v9743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17356.958999 secs 2024-04-26T16:40:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17356.958999 secs 2024-04-26T16:40:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17356.958999 secs 2024-04-26T16:40:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:19 smithi194 ceph-mon[33017]: pgmap v9744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17359.787080 secs 2024-04-26T16:40:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:19 smithi033 ceph-mon[27845]: pgmap v9744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17359.787080 secs 2024-04-26T16:40:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:19 smithi096 ceph-mon[32753]: pgmap v9744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17359.787080 secs 2024-04-26T16:40:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:21 smithi194 ceph-mon[33017]: pgmap v9745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:21 smithi033 ceph-mon[27845]: pgmap v9745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:21 smithi096 ceph-mon[32753]: pgmap v9745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:23 smithi194 ceph-mon[33017]: pgmap v9746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17361.959085 secs 2024-04-26T16:40:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:23 smithi033 ceph-mon[27845]: pgmap v9746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17361.959085 secs 2024-04-26T16:40:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:23 smithi096 ceph-mon[32753]: pgmap v9746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17361.959085 secs 2024-04-26T16:40:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17364.787176 secs 2024-04-26T16:40:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17364.787176 secs 2024-04-26T16:40:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17364.787176 secs 2024-04-26T16:40:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:25 smithi194 ceph-mon[33017]: pgmap v9747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:25 smithi033 ceph-mon[27845]: pgmap v9747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:25 smithi096 ceph-mon[32753]: pgmap v9747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:27 smithi194 ceph-mon[33017]: pgmap v9748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:27 smithi033 ceph-mon[27845]: pgmap v9748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:27 smithi096 ceph-mon[32753]: pgmap v9748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17366.959171 secs 2024-04-26T16:40:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17366.959171 secs 2024-04-26T16:40:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17366.959171 secs 2024-04-26T16:40:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:29 smithi194 ceph-mon[33017]: pgmap v9749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17369.787254 secs 2024-04-26T16:40:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:29 smithi033 ceph-mon[27845]: pgmap v9749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17369.787254 secs 2024-04-26T16:40:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:29 smithi096 ceph-mon[32753]: pgmap v9749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17369.787254 secs 2024-04-26T16:40:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:30.602+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:30.602+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:30.602+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:40:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:31 smithi194 ceph-mon[33017]: pgmap v9750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:31 smithi033 ceph-mon[27845]: pgmap v9750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:31 smithi096 ceph-mon[32753]: pgmap v9750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:33 smithi194 ceph-mon[33017]: pgmap v9751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17371.959270 secs 2024-04-26T16:40:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:33 smithi033 ceph-mon[27845]: pgmap v9751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17371.959270 secs 2024-04-26T16:40:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:33 smithi096 ceph-mon[32753]: pgmap v9751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17371.959270 secs 2024-04-26T16:40:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17374.787348 secs 2024-04-26T16:40:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17374.787348 secs 2024-04-26T16:40:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17374.787348 secs 2024-04-26T16:40:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:35 smithi194 ceph-mon[33017]: pgmap v9752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:35 smithi033 ceph-mon[27845]: pgmap v9752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:35 smithi096 ceph-mon[32753]: pgmap v9752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:37 smithi033 ceph-mon[27845]: pgmap v9753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:37 smithi096 ceph-mon[32753]: pgmap v9753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:37 smithi194 ceph-mon[33017]: pgmap v9753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17376.959331 secs 2024-04-26T16:40:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17376.959331 secs 2024-04-26T16:40:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17376.959331 secs 2024-04-26T16:40:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:39 smithi033 ceph-mon[27845]: pgmap v9754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17379.787464 secs 2024-04-26T16:40:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:39 smithi096 ceph-mon[32753]: pgmap v9754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17379.787464 secs 2024-04-26T16:40:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:39 smithi194 ceph-mon[33017]: pgmap v9754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17379.787464 secs 2024-04-26T16:40:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:41 smithi033 ceph-mon[27845]: pgmap v9755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:41 smithi096 ceph-mon[32753]: pgmap v9755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:41 smithi194 ceph-mon[33017]: pgmap v9755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:43 smithi033 ceph-mon[27845]: pgmap v9756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17381.959415 secs 2024-04-26T16:40:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:43 smithi096 ceph-mon[32753]: pgmap v9756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17381.959415 secs 2024-04-26T16:40:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:43 smithi194 ceph-mon[33017]: pgmap v9756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17381.959415 secs 2024-04-26T16:40:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17384.787557 secs 2024-04-26T16:40:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17384.787557 secs 2024-04-26T16:40:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:45.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17384.787557 secs 2024-04-26T16:40:45.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:45.616+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:45.616+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:40:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:40:45.616+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:40:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:45 smithi033 ceph-mon[27845]: pgmap v9757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:45 smithi096 ceph-mon[32753]: pgmap v9757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:45 smithi194 ceph-mon[33017]: pgmap v9757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:40:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:47 smithi033 ceph-mon[27845]: pgmap v9758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:47 smithi033 ceph-mon[27845]: pgmap v9759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:47 smithi033 ceph-mon[27845]: pgmap v9760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:47 smithi096 ceph-mon[32753]: pgmap v9758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:47 smithi096 ceph-mon[32753]: pgmap v9759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:40:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:47 smithi096 ceph-mon[32753]: pgmap v9760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:47 smithi194 ceph-mon[33017]: pgmap v9758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:40:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:47 smithi194 ceph-mon[33017]: pgmap v9759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:40:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:47 smithi194 ceph-mon[33017]: pgmap v9760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17386.959498 secs 2024-04-26T16:40:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17386.959498 secs 2024-04-26T16:40:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17386.959498 secs 2024-04-26T16:40:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17389.787637 secs 2024-04-26T16:40:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:49 smithi033 ceph-mon[27845]: pgmap v9761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17389.787637 secs 2024-04-26T16:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:49 smithi096 ceph-mon[32753]: pgmap v9761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:40:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17389.787637 secs 2024-04-26T16:40:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:49 smithi194 ceph-mon[33017]: pgmap v9761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:40:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:51 smithi033 ceph-mon[27845]: pgmap v9762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:51 smithi096 ceph-mon[32753]: pgmap v9762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:51 smithi194 ceph-mon[33017]: pgmap v9762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:53 smithi096 ceph-mon[32753]: pgmap v9763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17391.959558 secs 2024-04-26T16:40:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:53 smithi194 ceph-mon[33017]: pgmap v9763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17391.959558 secs 2024-04-26T16:40:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:53 smithi033 ceph-mon[27845]: pgmap v9763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:40:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17391.959558 secs 2024-04-26T16:40:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17394.787705 secs 2024-04-26T16:40:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17394.787705 secs 2024-04-26T16:40:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17394.787705 secs 2024-04-26T16:40:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:55 smithi194 ceph-mon[33017]: pgmap v9764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:55 smithi033 ceph-mon[27845]: pgmap v9764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:55 smithi096 ceph-mon[32753]: pgmap v9764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:40:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:57 smithi194 ceph-mon[33017]: pgmap v9765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:40:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:57 smithi033 ceph-mon[27845]: pgmap v9765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:40:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:57 smithi096 ceph-mon[32753]: pgmap v9765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:40:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17396.959619 secs 2024-04-26T16:40:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17399.787865 secs 2024-04-26T16:40:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17396.959619 secs 2024-04-26T16:40:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17399.787865 secs 2024-04-26T16:40:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17396.959619 secs 2024-04-26T16:40:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17399.787865 secs 2024-04-26T16:41:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:40:59 smithi194 ceph-mon[33017]: pgmap v9766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:40:59 smithi033 ceph-mon[27845]: pgmap v9766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:40:59 smithi096 ceph-mon[32753]: pgmap v9766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:00.619+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:00.619+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:00.620 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:00.619+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:41:01.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:01 smithi033 ceph-mon[27845]: pgmap v9767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:01 smithi194 ceph-mon[33017]: pgmap v9767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:01 smithi096 ceph-mon[32753]: pgmap v9767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:03 smithi194 ceph-mon[33017]: pgmap v9768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17401.959680 secs 2024-04-26T16:41:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17404.788003 secs 2024-04-26T16:41:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:03 smithi033 ceph-mon[27845]: pgmap v9768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17401.959680 secs 2024-04-26T16:41:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17404.788003 secs 2024-04-26T16:41:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:03 smithi096 ceph-mon[32753]: pgmap v9768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17401.959680 secs 2024-04-26T16:41:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17404.788003 secs 2024-04-26T16:41:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:05 smithi194 ceph-mon[33017]: pgmap v9769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:05 smithi033 ceph-mon[27845]: pgmap v9769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:05 smithi096 ceph-mon[32753]: pgmap v9769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:07 smithi194 ceph-mon[33017]: pgmap v9770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:08.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:07 smithi033 ceph-mon[27845]: pgmap v9770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:07 smithi096 ceph-mon[32753]: pgmap v9770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17406.959739 secs 2024-04-26T16:41:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17409.788163 secs 2024-04-26T16:41:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17406.959739 secs 2024-04-26T16:41:09.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17409.788163 secs 2024-04-26T16:41:09.264 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17406.959739 secs 2024-04-26T16:41:09.264 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17409.788163 secs 2024-04-26T16:41:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:09 smithi194 ceph-mon[33017]: pgmap v9771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:09 smithi033 ceph-mon[27845]: pgmap v9771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:09 smithi096 ceph-mon[32753]: pgmap v9771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:10 smithi194 ceph-mon[33017]: pgmap v9772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:10 smithi033 ceph-mon[27845]: pgmap v9772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:10 smithi096 ceph-mon[32753]: pgmap v9772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:13 smithi194 ceph-mon[33017]: pgmap v9773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17411.959824 secs 2024-04-26T16:41:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:13 smithi033 ceph-mon[27845]: pgmap v9773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17411.959824 secs 2024-04-26T16:41:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:13 smithi096 ceph-mon[32753]: pgmap v9773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17411.959824 secs 2024-04-26T16:41:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17414.788259 secs 2024-04-26T16:41:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17414.788259 secs 2024-04-26T16:41:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17414.788259 secs 2024-04-26T16:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:15.632+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:15.632+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:15.632+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:41:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:15 smithi194 ceph-mon[33017]: pgmap v9774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:15 smithi033 ceph-mon[27845]: pgmap v9774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:15 smithi096 ceph-mon[32753]: pgmap v9774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:17 smithi194 ceph-mon[33017]: pgmap v9775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:17 smithi033 ceph-mon[27845]: pgmap v9775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:17 smithi096 ceph-mon[32753]: pgmap v9775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17416.959906 secs 2024-04-26T16:41:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17416.959906 secs 2024-04-26T16:41:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17416.959906 secs 2024-04-26T16:41:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17419.788327 secs 2024-04-26T16:41:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:19 smithi194 ceph-mon[33017]: pgmap v9776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17419.788327 secs 2024-04-26T16:41:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:19 smithi033 ceph-mon[27845]: pgmap v9776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17419.788327 secs 2024-04-26T16:41:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:19 smithi096 ceph-mon[32753]: pgmap v9776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:21.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0075070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feef000f880 tx=0x7feef0075070 comp rx=0 tx=0).stop 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.760+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.761+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:21.762+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062070 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:21 smithi194 ceph-mon[33017]: pgmap v9777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:21 smithi033 ceph-mon[27845]: pgmap v9777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:21 smithi096 ceph-mon[32753]: pgmap v9777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:23 smithi194 ceph-mon[33017]: pgmap v9778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17421.959993 secs 2024-04-26T16:41:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:23 smithi033 ceph-mon[27845]: pgmap v9778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17421.959993 secs 2024-04-26T16:41:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:23 smithi096 ceph-mon[32753]: pgmap v9778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17421.959993 secs 2024-04-26T16:41:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:24.755+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:41:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:24.755+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:24.756+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T16:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:24.756+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:41:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17424.788458 secs 2024-04-26T16:41:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17424.788458 secs 2024-04-26T16:41:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17424.788458 secs 2024-04-26T16:41:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:25 smithi194 ceph-mon[33017]: pgmap v9779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:25 smithi033 ceph-mon[27845]: pgmap v9779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:25 smithi096 ceph-mon[32753]: pgmap v9779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:27 smithi194 ceph-mon[33017]: pgmap v9780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:27 smithi033 ceph-mon[27845]: pgmap v9780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:27 smithi096 ceph-mon[32753]: pgmap v9780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17426.960054 secs 2024-04-26T16:41:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17426.960054 secs 2024-04-26T16:41:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17426.960054 secs 2024-04-26T16:41:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17429.788577 secs 2024-04-26T16:41:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:29 smithi033 ceph-mon[27845]: pgmap v9781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17429.788577 secs 2024-04-26T16:41:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:29 smithi096 ceph-mon[32753]: pgmap v9781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17429.788577 secs 2024-04-26T16:41:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:29 smithi194 ceph-mon[33017]: pgmap v9781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:30.638+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:30.638+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:41:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:30.638+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:31 smithi033 ceph-mon[27845]: pgmap v9782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:31 smithi096 ceph-mon[32753]: pgmap v9782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:31 smithi194 ceph-mon[33017]: pgmap v9782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:33 smithi033 ceph-mon[27845]: pgmap v9783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17431.960118 secs 2024-04-26T16:41:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:33 smithi096 ceph-mon[32753]: pgmap v9783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17431.960118 secs 2024-04-26T16:41:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:33 smithi194 ceph-mon[33017]: pgmap v9783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17431.960118 secs 2024-04-26T16:41:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17434.788722 secs 2024-04-26T16:41:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17434.788722 secs 2024-04-26T16:41:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17434.788722 secs 2024-04-26T16:41:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:35 smithi033 ceph-mon[27845]: pgmap v9784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:35.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:35 smithi096 ceph-mon[32753]: pgmap v9784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:35 smithi194 ceph-mon[33017]: pgmap v9784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:37 smithi033 ceph-mon[27845]: pgmap v9785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:41:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:37 smithi096 ceph-mon[32753]: pgmap v9785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:41:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:37 smithi194 ceph-mon[33017]: pgmap v9785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:41:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17436.960206 secs 2024-04-26T16:41:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17436.960206 secs 2024-04-26T16:41:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17436.960206 secs 2024-04-26T16:41:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17439.788866 secs 2024-04-26T16:41:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:39 smithi033 ceph-mon[27845]: pgmap v9786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17439.788866 secs 2024-04-26T16:41:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:39 smithi096 ceph-mon[32753]: pgmap v9786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17439.788866 secs 2024-04-26T16:41:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:39 smithi194 ceph-mon[33017]: pgmap v9786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:41 smithi033 ceph-mon[27845]: pgmap v9787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:41 smithi096 ceph-mon[32753]: pgmap v9787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:41 smithi194 ceph-mon[33017]: pgmap v9787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:41:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:43 smithi033 ceph-mon[27845]: pgmap v9788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17441.960309 secs 2024-04-26T16:41:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:43 smithi096 ceph-mon[32753]: pgmap v9788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17441.960309 secs 2024-04-26T16:41:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:43 smithi194 ceph-mon[33017]: pgmap v9788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17441.960309 secs 2024-04-26T16:41:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17444.788994 secs 2024-04-26T16:41:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17444.788994 secs 2024-04-26T16:41:45.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17444.788994 secs 2024-04-26T16:41:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:45.641+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:45.641+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:41:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:41:45.641+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:41:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:45 smithi096 ceph-mon[32753]: pgmap v9789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:45 smithi194 ceph-mon[33017]: pgmap v9789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:45 smithi033 ceph-mon[27845]: pgmap v9789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:47 smithi096 ceph-mon[32753]: pgmap v9790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:41:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:47 smithi194 ceph-mon[33017]: pgmap v9790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:41:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:47 smithi033 ceph-mon[27845]: pgmap v9790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:41:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17446.960369 secs 2024-04-26T16:41:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:41:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17446.960369 secs 2024-04-26T16:41:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:41:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17446.960369 secs 2024-04-26T16:41:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:41:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:49 smithi194 ceph-mon[33017]: pgmap v9791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:49 smithi194 ceph-mon[33017]: pgmap v9792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17449.789138 secs 2024-04-26T16:41:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:49 smithi033 ceph-mon[27845]: pgmap v9791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:49 smithi033 ceph-mon[27845]: pgmap v9792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17449.789138 secs 2024-04-26T16:41:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:49 smithi096 ceph-mon[32753]: pgmap v9791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:41:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:49 smithi096 ceph-mon[32753]: pgmap v9792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17449.789138 secs 2024-04-26T16:41:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:51 smithi194 ceph-mon[33017]: pgmap v9793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:51 smithi033 ceph-mon[27845]: pgmap v9793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:51 smithi096 ceph-mon[32753]: pgmap v9793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:53 smithi194 ceph-mon[33017]: pgmap v9794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17451.960475 secs 2024-04-26T16:41:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:53 smithi033 ceph-mon[27845]: pgmap v9794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17451.960475 secs 2024-04-26T16:41:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:53 smithi096 ceph-mon[32753]: pgmap v9794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17451.960475 secs 2024-04-26T16:41:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17454.789252 secs 2024-04-26T16:41:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17454.789252 secs 2024-04-26T16:41:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17454.789252 secs 2024-04-26T16:41:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:55 smithi194 ceph-mon[33017]: pgmap v9795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T16:41:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:55 smithi033 ceph-mon[27845]: pgmap v9795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T16:41:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:55 smithi096 ceph-mon[32753]: pgmap v9795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T16:41:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:57 smithi194 ceph-mon[33017]: pgmap v9796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:57 smithi033 ceph-mon[27845]: pgmap v9796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:57 smithi096 ceph-mon[32753]: pgmap v9796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:41:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17456.960555 secs 2024-04-26T16:41:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17459.789332 secs 2024-04-26T16:41:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17456.960555 secs 2024-04-26T16:41:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17459.789332 secs 2024-04-26T16:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17456.960555 secs 2024-04-26T16:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17459.789332 secs 2024-04-26T16:42:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:41:59 smithi194 ceph-mon[33017]: pgmap v9797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:42:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:41:59 smithi033 ceph-mon[27845]: pgmap v9797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:42:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:41:59 smithi096 ceph-mon[32753]: pgmap v9797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:42:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:00.657+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:00.657+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:42:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:00.657+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:01 smithi194 ceph-mon[33017]: pgmap v9798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:01 smithi033 ceph-mon[27845]: pgmap v9798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:01 smithi096 ceph-mon[32753]: pgmap v9798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:03 smithi194 ceph-mon[33017]: pgmap v9799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17461.960637 secs 2024-04-26T16:42:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17464.789451 secs 2024-04-26T16:42:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:03 smithi033 ceph-mon[27845]: pgmap v9799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17461.960637 secs 2024-04-26T16:42:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17464.789451 secs 2024-04-26T16:42:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:03 smithi096 ceph-mon[32753]: pgmap v9799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17461.960637 secs 2024-04-26T16:42:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17464.789451 secs 2024-04-26T16:42:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:05 smithi194 ceph-mon[33017]: pgmap v9800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:05 smithi033 ceph-mon[27845]: pgmap v9800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:05 smithi096 ceph-mon[32753]: pgmap v9800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:07 smithi194 ceph-mon[33017]: pgmap v9801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:07 smithi033 ceph-mon[27845]: pgmap v9801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:07 smithi096 ceph-mon[32753]: pgmap v9801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17466.960721 secs 2024-04-26T16:42:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:08 smithi194 ceph-mon[33017]: pgmap v9802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17469.789574 secs 2024-04-26T16:42:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17466.960721 secs 2024-04-26T16:42:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:08 smithi033 ceph-mon[27845]: pgmap v9802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17469.789574 secs 2024-04-26T16:42:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17466.960721 secs 2024-04-26T16:42:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:08 smithi096 ceph-mon[32753]: pgmap v9802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17469.789574 secs 2024-04-26T16:42:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:11 smithi194 ceph-mon[33017]: pgmap v9803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:11 smithi033 ceph-mon[27845]: pgmap v9803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:11 smithi096 ceph-mon[32753]: pgmap v9803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:13 smithi194 ceph-mon[33017]: pgmap v9804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:13 smithi033 ceph-mon[27845]: pgmap v9804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:13 smithi096 ceph-mon[32753]: pgmap v9804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17471.960798 secs 2024-04-26T16:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17474.789693 secs 2024-04-26T16:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17471.960798 secs 2024-04-26T16:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17474.789693 secs 2024-04-26T16:42:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17471.960798 secs 2024-04-26T16:42:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17474.789693 secs 2024-04-26T16:42:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:15 smithi194 ceph-mon[33017]: pgmap v9805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:15.672+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:15.672+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:42:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:15.672+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:15 smithi033 ceph-mon[27845]: pgmap v9805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:15 smithi096 ceph-mon[32753]: pgmap v9805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:17 smithi194 ceph-mon[33017]: pgmap v9806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:17 smithi033 ceph-mon[27845]: pgmap v9806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:17 smithi096 ceph-mon[32753]: pgmap v9806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17476.960857 secs 2024-04-26T16:42:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:19 smithi194 ceph-mon[33017]: pgmap v9807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17479.789813 secs 2024-04-26T16:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17476.960857 secs 2024-04-26T16:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:19 smithi033 ceph-mon[27845]: pgmap v9807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17479.789813 secs 2024-04-26T16:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17476.960857 secs 2024-04-26T16:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:19 smithi096 ceph-mon[32753]: pgmap v9807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17479.789813 secs 2024-04-26T16:42:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:21 smithi194 ceph-mon[33017]: pgmap v9808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:21 smithi033 ceph-mon[27845]: pgmap v9808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:21 smithi096 ceph-mon[32753]: pgmap v9808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:23 smithi194 ceph-mon[33017]: pgmap v9809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:23 smithi033 ceph-mon[27845]: pgmap v9809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:23 smithi096 ceph-mon[32753]: pgmap v9809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17481.960941 secs 2024-04-26T16:42:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17484.789966 secs 2024-04-26T16:42:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17481.960941 secs 2024-04-26T16:42:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17484.789966 secs 2024-04-26T16:42:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17481.960941 secs 2024-04-26T16:42:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17484.789966 secs 2024-04-26T16:42:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:25 smithi194 ceph-mon[33017]: pgmap v9810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:25 smithi033 ceph-mon[27845]: pgmap v9810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:25 smithi096 ceph-mon[32753]: pgmap v9810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:27 smithi194 ceph-mon[33017]: pgmap v9811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:27 smithi033 ceph-mon[27845]: pgmap v9811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:27 smithi096 ceph-mon[32753]: pgmap v9811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17486.960977 secs 2024-04-26T16:42:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:29 smithi194 ceph-mon[33017]: pgmap v9812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17489.790129 secs 2024-04-26T16:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17486.960977 secs 2024-04-26T16:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:29 smithi033 ceph-mon[27845]: pgmap v9812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17489.790129 secs 2024-04-26T16:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17486.960977 secs 2024-04-26T16:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:29 smithi096 ceph-mon[32753]: pgmap v9812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17489.790129 secs 2024-04-26T16:42:30.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:30.685+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:30.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:30.685+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:30.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:30.685+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:42:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:31 smithi194 ceph-mon[33017]: pgmap v9813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:31 smithi033 ceph-mon[27845]: pgmap v9813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:31 smithi096 ceph-mon[32753]: pgmap v9813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:33 smithi194 ceph-mon[33017]: pgmap v9814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:33 smithi033 ceph-mon[27845]: pgmap v9814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:33 smithi096 ceph-mon[32753]: pgmap v9814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17491.961066 secs 2024-04-26T16:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17494.790207 secs 2024-04-26T16:42:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17491.961066 secs 2024-04-26T16:42:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17494.790207 secs 2024-04-26T16:42:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17491.961066 secs 2024-04-26T16:42:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17494.790207 secs 2024-04-26T16:42:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:35 smithi194 ceph-mon[33017]: pgmap v9815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:35 smithi033 ceph-mon[27845]: pgmap v9815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:35 smithi096 ceph-mon[32753]: pgmap v9815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:37 smithi194 ceph-mon[33017]: pgmap v9816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:37 smithi033 ceph-mon[27845]: pgmap v9816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:37 smithi096 ceph-mon[32753]: pgmap v9816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17496.961125 secs 2024-04-26T16:42:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:39 smithi194 ceph-mon[33017]: pgmap v9817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17499.790276 secs 2024-04-26T16:42:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17496.961125 secs 2024-04-26T16:42:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:39 smithi033 ceph-mon[27845]: pgmap v9817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17499.790276 secs 2024-04-26T16:42:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17496.961125 secs 2024-04-26T16:42:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:39 smithi096 ceph-mon[32753]: pgmap v9817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17499.790276 secs 2024-04-26T16:42:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:41 smithi194 ceph-mon[33017]: pgmap v9818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:41 smithi033 ceph-mon[27845]: pgmap v9818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:41 smithi096 ceph-mon[32753]: pgmap v9818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:43 smithi194 ceph-mon[33017]: pgmap v9819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:43 smithi033 ceph-mon[27845]: pgmap v9819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:43 smithi096 ceph-mon[32753]: pgmap v9819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17501.961215 secs 2024-04-26T16:42:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17504.790413 secs 2024-04-26T16:42:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17501.961215 secs 2024-04-26T16:42:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17504.790413 secs 2024-04-26T16:42:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17501.961215 secs 2024-04-26T16:42:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17504.790413 secs 2024-04-26T16:42:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:45 smithi033 ceph-mon[27845]: pgmap v9820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:45 smithi096 ceph-mon[32753]: pgmap v9820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:45.700+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:45.700+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:42:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:42:45.700+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:42:45.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:45 smithi194 ceph-mon[33017]: pgmap v9820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:47 smithi033 ceph-mon[27845]: pgmap v9821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:47 smithi096 ceph-mon[32753]: pgmap v9821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:47 smithi194 ceph-mon[33017]: pgmap v9821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17506.961299 secs 2024-04-26T16:42:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17506.961299 secs 2024-04-26T16:42:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17506.961299 secs 2024-04-26T16:42:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:49 smithi033 ceph-mon[27845]: pgmap v9822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17509.790556 secs 2024-04-26T16:42:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:49 smithi096 ceph-mon[32753]: pgmap v9822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17509.790556 secs 2024-04-26T16:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:49 smithi194 ceph-mon[33017]: pgmap v9822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17509.790556 secs 2024-04-26T16:42:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:51 smithi033 ceph-mon[27845]: pgmap v9823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:51 smithi096 ceph-mon[32753]: pgmap v9823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:51 smithi194 ceph-mon[33017]: pgmap v9823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:42:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:52.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: pgmap v9824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17511.961385 secs 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: pgmap v9824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17511.961385 secs 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:42:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:42:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:42:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:42:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: pgmap v9824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17511.961385 secs 2024-04-26T16:42:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17514.790624 secs 2024-04-26T16:42:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17514.790624 secs 2024-04-26T16:42:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17514.790624 secs 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: pgmap v9825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:42:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:42:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: pgmap v9826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T16:42:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: pgmap v9827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:42:55.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: pgmap v9825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:42:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:42:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: pgmap v9826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T16:42:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: pgmap v9827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:42:55.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: pgmap v9825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:42:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: pgmap v9826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: pgmap v9827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:42:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:42:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:56 smithi194 ceph-mon[33017]: pgmap v9828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:42:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:56 smithi033 ceph-mon[27845]: pgmap v9828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:42:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:56 smithi096 ceph-mon[32753]: pgmap v9828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:42:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17516.961481 secs 2024-04-26T16:42:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17516.961481 secs 2024-04-26T16:42:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17516.961481 secs 2024-04-26T16:42:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17519.790782 secs 2024-04-26T16:42:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:42:59 smithi033 ceph-mon[27845]: pgmap v9829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:42:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17519.790782 secs 2024-04-26T16:42:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:42:59 smithi096 ceph-mon[32753]: pgmap v9829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:42:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17519.790782 secs 2024-04-26T16:42:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:42:59 smithi194 ceph-mon[33017]: pgmap v9829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:43:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:00.708+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:00.708+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:00.708+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:43:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:00 smithi194 ceph-mon[33017]: pgmap v9830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:43:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:00 smithi033 ceph-mon[27845]: pgmap v9830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:43:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:00 smithi096 ceph-mon[32753]: pgmap v9830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:43:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:02 smithi194 ceph-mon[33017]: pgmap v9831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:43:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:02 smithi033 ceph-mon[27845]: pgmap v9831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:43:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:02 smithi096 ceph-mon[32753]: pgmap v9831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:43:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17521.961566 secs 2024-04-26T16:43:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17524.790850 secs 2024-04-26T16:43:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17521.961566 secs 2024-04-26T16:43:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17524.790850 secs 2024-04-26T16:43:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17521.961566 secs 2024-04-26T16:43:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17524.790850 secs 2024-04-26T16:43:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:04 smithi194 ceph-mon[33017]: pgmap v9832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:43:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:04 smithi033 ceph-mon[27845]: pgmap v9832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:43:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:04 smithi096 ceph-mon[32753]: pgmap v9832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:43:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:06 smithi194 ceph-mon[33017]: pgmap v9833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:06 smithi033 ceph-mon[27845]: pgmap v9833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:06 smithi096 ceph-mon[32753]: pgmap v9833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17526.961665 secs 2024-04-26T16:43:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17526.961665 secs 2024-04-26T16:43:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17526.961665 secs 2024-04-26T16:43:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17529.790967 secs 2024-04-26T16:43:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:09 smithi096 ceph-mon[32753]: pgmap v9834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17529.790967 secs 2024-04-26T16:43:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:09 smithi194 ceph-mon[33017]: pgmap v9834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17529.790967 secs 2024-04-26T16:43:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:09 smithi033 ceph-mon[27845]: pgmap v9834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:10 smithi194 ceph-mon[33017]: pgmap v9835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:10 smithi033 ceph-mon[27845]: pgmap v9835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:10 smithi096 ceph-mon[32753]: pgmap v9835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:13.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:12 smithi033 ceph-mon[27845]: pgmap v9836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:12 smithi194 ceph-mon[33017]: pgmap v9836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:12 smithi096 ceph-mon[32753]: pgmap v9836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17531.961767 secs 2024-04-26T16:43:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17534.791062 secs 2024-04-26T16:43:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17531.961767 secs 2024-04-26T16:43:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17534.791062 secs 2024-04-26T16:43:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17531.961767 secs 2024-04-26T16:43:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17534.791062 secs 2024-04-26T16:43:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:14 smithi194 ceph-mon[33017]: pgmap v9837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:14 smithi033 ceph-mon[27845]: pgmap v9837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:14 smithi096 ceph-mon[32753]: pgmap v9837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:15.716+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:15.716+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:43:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:15.716+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:16 smithi194 ceph-mon[33017]: pgmap v9838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:16 smithi033 ceph-mon[27845]: pgmap v9838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:16 smithi096 ceph-mon[32753]: pgmap v9838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17536.961826 secs 2024-04-26T16:43:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17536.961826 secs 2024-04-26T16:43:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17536.961826 secs 2024-04-26T16:43:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17539.791168 secs 2024-04-26T16:43:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:19 smithi194 ceph-mon[33017]: pgmap v9839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:19.889 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17539.791168 secs 2024-04-26T16:43:19.889 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:19 smithi096 ceph-mon[32753]: pgmap v9839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17539.791168 secs 2024-04-26T16:43:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:19 smithi033 ceph-mon[27845]: pgmap v9839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:20 smithi194 ceph-mon[33017]: pgmap v9840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:20 smithi033 ceph-mon[27845]: pgmap v9840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:20 smithi096 ceph-mon[32753]: pgmap v9840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:22 smithi194 ceph-mon[33017]: pgmap v9841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:22 smithi033 ceph-mon[27845]: pgmap v9841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:22 smithi096 ceph-mon[32753]: pgmap v9841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17541.961928 secs 2024-04-26T16:43:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17544.791233 secs 2024-04-26T16:43:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17541.961928 secs 2024-04-26T16:43:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17544.791233 secs 2024-04-26T16:43:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17541.961928 secs 2024-04-26T16:43:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17544.791233 secs 2024-04-26T16:43:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:24 smithi194 ceph-mon[33017]: pgmap v9842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:24 smithi033 ceph-mon[27845]: pgmap v9842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:24 smithi096 ceph-mon[32753]: pgmap v9842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:26 smithi194 ceph-mon[33017]: pgmap v9843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:26 smithi033 ceph-mon[27845]: pgmap v9843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:26 smithi096 ceph-mon[32753]: pgmap v9843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17546.962005 secs 2024-04-26T16:43:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17546.962005 secs 2024-04-26T16:43:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17546.962005 secs 2024-04-26T16:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17549.791372 secs 2024-04-26T16:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:29 smithi096 ceph-mon[32753]: pgmap v9844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17549.791372 secs 2024-04-26T16:43:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:29 smithi194 ceph-mon[33017]: pgmap v9844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17549.791372 secs 2024-04-26T16:43:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:29 smithi033 ceph-mon[27845]: pgmap v9844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:30.732+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:30.732+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:30.732+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:43:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:30 smithi194 ceph-mon[33017]: pgmap v9845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:30 smithi033 ceph-mon[27845]: pgmap v9845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:30 smithi096 ceph-mon[32753]: pgmap v9845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:32 smithi194 ceph-mon[33017]: pgmap v9846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:32 smithi033 ceph-mon[27845]: pgmap v9846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:32 smithi096 ceph-mon[32753]: pgmap v9846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17551.962100 secs 2024-04-26T16:43:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17554.791488 secs 2024-04-26T16:43:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17551.962100 secs 2024-04-26T16:43:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17554.791488 secs 2024-04-26T16:43:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17551.962100 secs 2024-04-26T16:43:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17554.791488 secs 2024-04-26T16:43:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:34 smithi194 ceph-mon[33017]: pgmap v9847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:34 smithi033 ceph-mon[27845]: pgmap v9847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:34 smithi096 ceph-mon[32753]: pgmap v9847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:36 smithi194 ceph-mon[33017]: pgmap v9848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:36 smithi033 ceph-mon[27845]: pgmap v9848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:36 smithi096 ceph-mon[32753]: pgmap v9848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17556.962225 secs 2024-04-26T16:43:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17556.962225 secs 2024-04-26T16:43:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17556.962225 secs 2024-04-26T16:43:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17559.791629 secs 2024-04-26T16:43:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:39 smithi096 ceph-mon[32753]: pgmap v9849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17559.791629 secs 2024-04-26T16:43:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:39 smithi194 ceph-mon[33017]: pgmap v9849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17559.791629 secs 2024-04-26T16:43:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:39 smithi033 ceph-mon[27845]: pgmap v9849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:40 smithi194 ceph-mon[33017]: pgmap v9850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:40 smithi033 ceph-mon[27845]: pgmap v9850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:40 smithi096 ceph-mon[32753]: pgmap v9850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:42 smithi194 ceph-mon[33017]: pgmap v9851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:42 smithi033 ceph-mon[27845]: pgmap v9851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:42 smithi096 ceph-mon[32753]: pgmap v9851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17561.962332 secs 2024-04-26T16:43:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17564.791770 secs 2024-04-26T16:43:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17561.962332 secs 2024-04-26T16:43:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17564.791770 secs 2024-04-26T16:43:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17561.962332 secs 2024-04-26T16:43:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17564.791770 secs 2024-04-26T16:43:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:44 smithi194 ceph-mon[33017]: pgmap v9852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:44 smithi033 ceph-mon[27845]: pgmap v9852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:44 smithi096 ceph-mon[32753]: pgmap v9852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:45.738+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:45.738+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:43:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:43:45.738+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:43:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:46 smithi194 ceph-mon[33017]: pgmap v9853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:46 smithi033 ceph-mon[27845]: pgmap v9853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:46 smithi096 ceph-mon[32753]: pgmap v9853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17566.962442 secs 2024-04-26T16:43:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17566.962442 secs 2024-04-26T16:43:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17566.962442 secs 2024-04-26T16:43:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17569.791866 secs 2024-04-26T16:43:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:49 smithi096 ceph-mon[32753]: pgmap v9854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17569.791866 secs 2024-04-26T16:43:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:49 smithi194 ceph-mon[33017]: pgmap v9854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17569.791866 secs 2024-04-26T16:43:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:49 smithi033 ceph-mon[27845]: pgmap v9854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:50 smithi194 ceph-mon[33017]: pgmap v9855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:50 smithi033 ceph-mon[27845]: pgmap v9855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:50 smithi096 ceph-mon[32753]: pgmap v9855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:52 smithi194 ceph-mon[33017]: pgmap v9856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:52 smithi033 ceph-mon[27845]: pgmap v9856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:52 smithi096 ceph-mon[32753]: pgmap v9856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17571.962532 secs 2024-04-26T16:43:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17574.791966 secs 2024-04-26T16:43:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17571.962532 secs 2024-04-26T16:43:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17574.791966 secs 2024-04-26T16:43:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17571.962532 secs 2024-04-26T16:43:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17574.791966 secs 2024-04-26T16:43:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:54 smithi194 ceph-mon[33017]: pgmap v9857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:54 smithi033 ceph-mon[27845]: pgmap v9857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:54 smithi096 ceph-mon[32753]: pgmap v9857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:43:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:56 smithi033 ceph-mon[27845]: pgmap v9858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:56 smithi033 ceph-mon[27845]: pgmap v9859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:43:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:43:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:56 smithi096 ceph-mon[32753]: pgmap v9858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:56 smithi096 ceph-mon[32753]: pgmap v9859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:43:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:43:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:56 smithi194 ceph-mon[33017]: pgmap v9858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:43:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:56 smithi194 ceph-mon[33017]: pgmap v9859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:43:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:43:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17576.962633 secs 2024-04-26T16:43:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17576.962633 secs 2024-04-26T16:43:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17576.962633 secs 2024-04-26T16:43:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17579.792097 secs 2024-04-26T16:43:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:43:59 smithi194 ceph-mon[33017]: pgmap v9860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:43:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17579.792097 secs 2024-04-26T16:43:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:43:59 smithi033 ceph-mon[27845]: pgmap v9860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:43:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17579.792097 secs 2024-04-26T16:43:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:43:59 smithi096 ceph-mon[32753]: pgmap v9860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:00.750+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:00.750+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:00.750+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:44:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:01 smithi194 ceph-mon[33017]: pgmap v9861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:01 smithi033 ceph-mon[27845]: pgmap v9861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:01 smithi096 ceph-mon[32753]: pgmap v9861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:03 smithi194 ceph-mon[33017]: pgmap v9862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17581.962728 secs 2024-04-26T16:44:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:03 smithi033 ceph-mon[27845]: pgmap v9862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17581.962728 secs 2024-04-26T16:44:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:03 smithi096 ceph-mon[32753]: pgmap v9862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17581.962728 secs 2024-04-26T16:44:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17584.792227 secs 2024-04-26T16:44:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17584.792227 secs 2024-04-26T16:44:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17584.792227 secs 2024-04-26T16:44:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:05 smithi194 ceph-mon[33017]: pgmap v9863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:44:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:05 smithi033 ceph-mon[27845]: pgmap v9863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:44:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:05 smithi096 ceph-mon[32753]: pgmap v9863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:44:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:07 smithi194 ceph-mon[33017]: pgmap v9864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:44:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:07 smithi033 ceph-mon[27845]: pgmap v9864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:44:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:07 smithi096 ceph-mon[32753]: pgmap v9864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:44:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17586.962828 secs 2024-04-26T16:44:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17586.962828 secs 2024-04-26T16:44:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17586.962828 secs 2024-04-26T16:44:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17589.792347 secs 2024-04-26T16:44:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:09 smithi194 ceph-mon[33017]: pgmap v9865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17589.792347 secs 2024-04-26T16:44:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:09 smithi033 ceph-mon[27845]: pgmap v9865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17589.792347 secs 2024-04-26T16:44:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:09 smithi096 ceph-mon[32753]: pgmap v9865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:11 smithi194 ceph-mon[33017]: pgmap v9866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:11 smithi033 ceph-mon[27845]: pgmap v9866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:11 smithi096 ceph-mon[32753]: pgmap v9866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:13 smithi194 ceph-mon[33017]: pgmap v9867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17591.962930 secs 2024-04-26T16:44:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:13 smithi033 ceph-mon[27845]: pgmap v9867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17591.962930 secs 2024-04-26T16:44:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:13 smithi096 ceph-mon[32753]: pgmap v9867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17591.962930 secs 2024-04-26T16:44:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17594.792469 secs 2024-04-26T16:44:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17594.792469 secs 2024-04-26T16:44:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17594.792469 secs 2024-04-26T16:44:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:15.757+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:15.757+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:44:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:15.757+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:15 smithi194 ceph-mon[33017]: pgmap v9868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:15 smithi033 ceph-mon[27845]: pgmap v9868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:15 smithi096 ceph-mon[32753]: pgmap v9868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:17 smithi194 ceph-mon[33017]: pgmap v9869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:17 smithi033 ceph-mon[27845]: pgmap v9869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:17 smithi096 ceph-mon[32753]: pgmap v9869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:18.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17596.963033 secs 2024-04-26T16:44:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17596.963033 secs 2024-04-26T16:44:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17596.963033 secs 2024-04-26T16:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17599.792588 secs 2024-04-26T16:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:19 smithi194 ceph-mon[33017]: pgmap v9870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17599.792588 secs 2024-04-26T16:44:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:19 smithi033 ceph-mon[27845]: pgmap v9870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17599.792588 secs 2024-04-26T16:44:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:19 smithi096 ceph-mon[32753]: pgmap v9870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:21 smithi194 ceph-mon[33017]: pgmap v9871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:21 smithi033 ceph-mon[27845]: pgmap v9871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:21 smithi096 ceph-mon[32753]: pgmap v9871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:23 smithi194 ceph-mon[33017]: pgmap v9872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17601.963124 secs 2024-04-26T16:44:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:23 smithi033 ceph-mon[27845]: pgmap v9872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17601.963124 secs 2024-04-26T16:44:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:23 smithi096 ceph-mon[32753]: pgmap v9872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17601.963124 secs 2024-04-26T16:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:24.785+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:24.785+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:24.785+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:44:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:24.786+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:44:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17604.792713 secs 2024-04-26T16:44:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17604.792713 secs 2024-04-26T16:44:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17604.792713 secs 2024-04-26T16:44:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:25 smithi194 ceph-mon[33017]: pgmap v9873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:25 smithi033 ceph-mon[27845]: pgmap v9873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:25 smithi096 ceph-mon[32753]: pgmap v9873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:27 smithi194 ceph-mon[33017]: pgmap v9874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:27 smithi033 ceph-mon[27845]: pgmap v9874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:27 smithi096 ceph-mon[32753]: pgmap v9874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17606.963213 secs 2024-04-26T16:44:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17606.963213 secs 2024-04-26T16:44:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17606.963213 secs 2024-04-26T16:44:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17609.792846 secs 2024-04-26T16:44:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:29 smithi194 ceph-mon[33017]: pgmap v9875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17609.792846 secs 2024-04-26T16:44:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:29 smithi033 ceph-mon[27845]: pgmap v9875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17609.792846 secs 2024-04-26T16:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:29 smithi096 ceph-mon[32753]: pgmap v9875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:30.760+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:30.761+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:30.761+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:44:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:31 smithi194 ceph-mon[33017]: pgmap v9876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:31 smithi033 ceph-mon[27845]: pgmap v9876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:31 smithi096 ceph-mon[32753]: pgmap v9876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:33 smithi194 ceph-mon[33017]: pgmap v9877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17611.963314 secs 2024-04-26T16:44:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:33 smithi033 ceph-mon[27845]: pgmap v9877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17611.963314 secs 2024-04-26T16:44:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:33 smithi096 ceph-mon[32753]: pgmap v9877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17611.963314 secs 2024-04-26T16:44:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17614.792965 secs 2024-04-26T16:44:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17614.792965 secs 2024-04-26T16:44:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17614.792965 secs 2024-04-26T16:44:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:35 smithi033 ceph-mon[27845]: pgmap v9878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:35 smithi096 ceph-mon[32753]: pgmap v9878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:35 smithi194 ceph-mon[33017]: pgmap v9878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:37 smithi033 ceph-mon[27845]: pgmap v9879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:37 smithi096 ceph-mon[32753]: pgmap v9879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:37 smithi194 ceph-mon[33017]: pgmap v9879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17616.963414 secs 2024-04-26T16:44:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17616.963414 secs 2024-04-26T16:44:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17616.963414 secs 2024-04-26T16:44:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17619.793099 secs 2024-04-26T16:44:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:39 smithi033 ceph-mon[27845]: pgmap v9880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17619.793099 secs 2024-04-26T16:44:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:39 smithi096 ceph-mon[32753]: pgmap v9880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17619.793099 secs 2024-04-26T16:44:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:39 smithi194 ceph-mon[33017]: pgmap v9880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:41 smithi033 ceph-mon[27845]: pgmap v9881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:41 smithi096 ceph-mon[32753]: pgmap v9881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:41 smithi194 ceph-mon[33017]: pgmap v9881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:43 smithi033 ceph-mon[27845]: pgmap v9882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17621.963491 secs 2024-04-26T16:44:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:43 smithi096 ceph-mon[32753]: pgmap v9882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17621.963491 secs 2024-04-26T16:44:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:43 smithi194 ceph-mon[33017]: pgmap v9882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17621.963491 secs 2024-04-26T16:44:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17624.793223 secs 2024-04-26T16:44:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17624.793223 secs 2024-04-26T16:44:45.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17624.793223 secs 2024-04-26T16:44:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:45.774+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:45.774+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:44:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:44:45.774+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:44:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:45 smithi033 ceph-mon[27845]: pgmap v9883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:45 smithi096 ceph-mon[32753]: pgmap v9883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:45 smithi194 ceph-mon[33017]: pgmap v9883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:47 smithi033 ceph-mon[27845]: pgmap v9884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:47 smithi096 ceph-mon[32753]: pgmap v9884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:47 smithi194 ceph-mon[33017]: pgmap v9884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17626.963593 secs 2024-04-26T16:44:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17626.963593 secs 2024-04-26T16:44:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17626.963593 secs 2024-04-26T16:44:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17629.793341 secs 2024-04-26T16:44:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:49 smithi033 ceph-mon[27845]: pgmap v9885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17629.793341 secs 2024-04-26T16:44:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:49 smithi096 ceph-mon[32753]: pgmap v9885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17629.793341 secs 2024-04-26T16:44:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:49 smithi194 ceph-mon[33017]: pgmap v9885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:51 smithi096 ceph-mon[32753]: pgmap v9886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:51 smithi194 ceph-mon[33017]: pgmap v9886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:51 smithi033 ceph-mon[27845]: pgmap v9886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:44:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:53 smithi194 ceph-mon[33017]: pgmap v9887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17631.963693 secs 2024-04-26T16:44:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:53 smithi033 ceph-mon[27845]: pgmap v9887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17631.963693 secs 2024-04-26T16:44:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:53 smithi096 ceph-mon[32753]: pgmap v9887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:44:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17631.963693 secs 2024-04-26T16:44:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17634.793453 secs 2024-04-26T16:44:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17634.793453 secs 2024-04-26T16:44:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17634.793453 secs 2024-04-26T16:44:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:55 smithi194 ceph-mon[33017]: pgmap v9888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:55 smithi033 ceph-mon[27845]: pgmap v9888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:55 smithi096 ceph-mon[32753]: pgmap v9888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:57 smithi194 ceph-mon[33017]: pgmap v9889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:44:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:57 smithi033 ceph-mon[27845]: pgmap v9889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:44:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:57 smithi096 ceph-mon[32753]: pgmap v9889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:44:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:44:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17636.963795 secs 2024-04-26T16:44:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:44:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:44:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:44:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17636.963795 secs 2024-04-26T16:44:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:44:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:44:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17636.963795 secs 2024-04-26T16:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:45:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:59 smithi194 ceph-mon[33017]: pgmap v9890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:59 smithi194 ceph-mon[33017]: pgmap v9891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:44:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17639.793551 secs 2024-04-26T16:45:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:59 smithi033 ceph-mon[27845]: pgmap v9890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:59 smithi033 ceph-mon[27845]: pgmap v9891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:44:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17639.793551 secs 2024-04-26T16:45:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:59 smithi096 ceph-mon[32753]: pgmap v9890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:59 smithi096 ceph-mon[32753]: pgmap v9891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:44:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17639.793551 secs 2024-04-26T16:45:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:00.790+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:00.791+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:00.791+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:45:01.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:01 smithi033 ceph-mon[27845]: pgmap v9892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:01 smithi194 ceph-mon[33017]: pgmap v9892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:01 smithi096 ceph-mon[32753]: pgmap v9892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:03 smithi194 ceph-mon[33017]: pgmap v9893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17641.963860 secs 2024-04-26T16:45:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:03 smithi033 ceph-mon[27845]: pgmap v9893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17641.963860 secs 2024-04-26T16:45:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:03 smithi096 ceph-mon[32753]: pgmap v9893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17641.963860 secs 2024-04-26T16:45:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17644.793630 secs 2024-04-26T16:45:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17644.793630 secs 2024-04-26T16:45:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17644.793630 secs 2024-04-26T16:45:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:05 smithi194 ceph-mon[33017]: pgmap v9894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:05 smithi033 ceph-mon[27845]: pgmap v9894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:05 smithi096 ceph-mon[32753]: pgmap v9894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:07 smithi194 ceph-mon[33017]: pgmap v9895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:07 smithi033 ceph-mon[27845]: pgmap v9895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:07 smithi096 ceph-mon[32753]: pgmap v9895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:45:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17646.963959 secs 2024-04-26T16:45:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17649.793752 secs 2024-04-26T16:45:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17646.963959 secs 2024-04-26T16:45:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17649.793752 secs 2024-04-26T16:45:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17646.963959 secs 2024-04-26T16:45:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17649.793752 secs 2024-04-26T16:45:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:09 smithi194 ceph-mon[33017]: pgmap v9896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:45:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:09 smithi033 ceph-mon[27845]: pgmap v9896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:45:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:09 smithi096 ceph-mon[32753]: pgmap v9896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:45:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:11 smithi194 ceph-mon[33017]: pgmap v9897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:11 smithi033 ceph-mon[27845]: pgmap v9897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:11 smithi096 ceph-mon[32753]: pgmap v9897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:13 smithi194 ceph-mon[33017]: pgmap v9898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17651.964055 secs 2024-04-26T16:45:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17654.793872 secs 2024-04-26T16:45:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:13 smithi033 ceph-mon[27845]: pgmap v9898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17651.964055 secs 2024-04-26T16:45:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17654.793872 secs 2024-04-26T16:45:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:13 smithi096 ceph-mon[32753]: pgmap v9898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17651.964055 secs 2024-04-26T16:45:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17654.793872 secs 2024-04-26T16:45:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:15.803+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:15.804+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:15.804+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:45:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:15 smithi194 ceph-mon[33017]: pgmap v9899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:15 smithi033 ceph-mon[27845]: pgmap v9899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:15 smithi096 ceph-mon[32753]: pgmap v9899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:17 smithi194 ceph-mon[33017]: pgmap v9900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:17 smithi033 ceph-mon[27845]: pgmap v9900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:17 smithi096 ceph-mon[32753]: pgmap v9900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:19.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17656.964150 secs 2024-04-26T16:45:19.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17659.793949 secs 2024-04-26T16:45:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17656.964150 secs 2024-04-26T16:45:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17659.793949 secs 2024-04-26T16:45:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17656.964150 secs 2024-04-26T16:45:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17659.793949 secs 2024-04-26T16:45:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:19 smithi194 ceph-mon[33017]: pgmap v9901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:19 smithi033 ceph-mon[27845]: pgmap v9901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:19 smithi096 ceph-mon[32753]: pgmap v9901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:21 smithi194 ceph-mon[33017]: pgmap v9902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:21 smithi033 ceph-mon[27845]: pgmap v9902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:21 smithi096 ceph-mon[32753]: pgmap v9902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:22 smithi194 ceph-mon[33017]: pgmap v9903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:22 smithi033 ceph-mon[27845]: pgmap v9903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:22 smithi096 ceph-mon[32753]: pgmap v9903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17661.964237 secs 2024-04-26T16:45:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17664.794135 secs 2024-04-26T16:45:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17661.964237 secs 2024-04-26T16:45:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17664.794135 secs 2024-04-26T16:45:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17661.964237 secs 2024-04-26T16:45:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17664.794135 secs 2024-04-26T16:45:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:24 smithi033 ceph-mon[27845]: pgmap v9904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:24 smithi096 ceph-mon[32753]: pgmap v9904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:24 smithi194 ceph-mon[33017]: pgmap v9904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:27 smithi033 ceph-mon[27845]: pgmap v9905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:27 smithi096 ceph-mon[32753]: pgmap v9905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:27 smithi194 ceph-mon[33017]: pgmap v9905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17666.964319 secs 2024-04-26T16:45:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:29 smithi033 ceph-mon[27845]: pgmap v9906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17669.794268 secs 2024-04-26T16:45:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17666.964319 secs 2024-04-26T16:45:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:29 smithi096 ceph-mon[32753]: pgmap v9906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17669.794268 secs 2024-04-26T16:45:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17666.964319 secs 2024-04-26T16:45:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:29 smithi194 ceph-mon[33017]: pgmap v9906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17669.794268 secs 2024-04-26T16:45:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:30.818+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:30.818+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:30.818+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:45:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:31 smithi033 ceph-mon[27845]: pgmap v9907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:31 smithi096 ceph-mon[32753]: pgmap v9907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:31 smithi194 ceph-mon[33017]: pgmap v9907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:33 smithi033 ceph-mon[27845]: pgmap v9908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:33 smithi096 ceph-mon[32753]: pgmap v9908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:33 smithi194 ceph-mon[33017]: pgmap v9908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17671.964404 secs 2024-04-26T16:45:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17674.794368 secs 2024-04-26T16:45:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17671.964404 secs 2024-04-26T16:45:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17674.794368 secs 2024-04-26T16:45:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17671.964404 secs 2024-04-26T16:45:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17674.794368 secs 2024-04-26T16:45:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:35 smithi033 ceph-mon[27845]: pgmap v9909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:35 smithi096 ceph-mon[32753]: pgmap v9909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:35 smithi194 ceph-mon[33017]: pgmap v9909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:45:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:37 smithi033 ceph-mon[27845]: pgmap v9910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:37 smithi096 ceph-mon[32753]: pgmap v9910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:37 smithi194 ceph-mon[33017]: pgmap v9910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:45:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17676.964487 secs 2024-04-26T16:45:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:39 smithi096 ceph-mon[32753]: pgmap v9911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17679.794494 secs 2024-04-26T16:45:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17676.964487 secs 2024-04-26T16:45:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:39 smithi194 ceph-mon[33017]: pgmap v9911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17679.794494 secs 2024-04-26T16:45:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17676.964487 secs 2024-04-26T16:45:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:39 smithi033 ceph-mon[27845]: pgmap v9911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17679.794494 secs 2024-04-26T16:45:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:41 smithi096 ceph-mon[32753]: pgmap v9912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:41 smithi194 ceph-mon[33017]: pgmap v9912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:41 smithi033 ceph-mon[27845]: pgmap v9912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:43 smithi194 ceph-mon[33017]: pgmap v9913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:43 smithi033 ceph-mon[27845]: pgmap v9913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:43 smithi096 ceph-mon[32753]: pgmap v9913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17681.964570 secs 2024-04-26T16:45:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17684.794623 secs 2024-04-26T16:45:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17681.964570 secs 2024-04-26T16:45:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17684.794623 secs 2024-04-26T16:45:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17681.964570 secs 2024-04-26T16:45:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17684.794623 secs 2024-04-26T16:45:45.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:45 smithi194 ceph-mon[33017]: pgmap v9914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:45 smithi033 ceph-mon[27845]: pgmap v9914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:45 smithi096 ceph-mon[32753]: pgmap v9914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:45.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:45.833+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:45.833+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:45:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:45:45.833+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:45:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:47 smithi194 ceph-mon[33017]: pgmap v9915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:47 smithi033 ceph-mon[27845]: pgmap v9915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:47 smithi096 ceph-mon[32753]: pgmap v9915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:49.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17686.964655 secs 2024-04-26T16:45:49.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:49 smithi096 ceph-mon[32753]: pgmap v9916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:49.485 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17689.794755 secs 2024-04-26T16:45:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17686.964655 secs 2024-04-26T16:45:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:49 smithi194 ceph-mon[33017]: pgmap v9916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17689.794755 secs 2024-04-26T16:45:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17686.964655 secs 2024-04-26T16:45:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:49 smithi033 ceph-mon[27845]: pgmap v9916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17689.794755 secs 2024-04-26T16:45:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:51 smithi194 ceph-mon[33017]: pgmap v9917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:51 smithi033 ceph-mon[27845]: pgmap v9917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:51 smithi096 ceph-mon[32753]: pgmap v9917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:53 smithi194 ceph-mon[33017]: pgmap v9918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:53 smithi033 ceph-mon[27845]: pgmap v9918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:53 smithi096 ceph-mon[32753]: pgmap v9918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17691.964743 secs 2024-04-26T16:45:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17694.794823 secs 2024-04-26T16:45:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17691.964743 secs 2024-04-26T16:45:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17694.794823 secs 2024-04-26T16:45:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17691.964743 secs 2024-04-26T16:45:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17694.794823 secs 2024-04-26T16:45:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:55 smithi194 ceph-mon[33017]: pgmap v9919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:55 smithi033 ceph-mon[27845]: pgmap v9919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:55 smithi096 ceph-mon[32753]: pgmap v9919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:57 smithi194 ceph-mon[33017]: pgmap v9920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:57 smithi033 ceph-mon[27845]: pgmap v9920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:57 smithi096 ceph-mon[32753]: pgmap v9920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:45:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17696.964829 secs 2024-04-26T16:45:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:59 smithi194 ceph-mon[33017]: pgmap v9921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17699.794973 secs 2024-04-26T16:45:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:45:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17696.964829 secs 2024-04-26T16:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:59 smithi033 ceph-mon[27845]: pgmap v9921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17699.794973 secs 2024-04-26T16:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:45:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17696.964829 secs 2024-04-26T16:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:59 smithi096 ceph-mon[32753]: pgmap v9921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17699.794973 secs 2024-04-26T16:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:45:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:46:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:46:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:46:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:00 smithi194 ceph-mon[33017]: pgmap v9922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:46:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:00 smithi194 ceph-mon[33017]: pgmap v9923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:46:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:46:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:46:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:00 smithi033 ceph-mon[27845]: pgmap v9922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:46:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:00 smithi033 ceph-mon[27845]: pgmap v9923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:00 smithi096 ceph-mon[32753]: pgmap v9922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:00 smithi096 ceph-mon[32753]: pgmap v9923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:46:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:00.842+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:00.842+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:00.842+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:46:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:02 smithi194 ceph-mon[33017]: pgmap v9924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:02 smithi033 ceph-mon[27845]: pgmap v9924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:02 smithi096 ceph-mon[32753]: pgmap v9924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:04.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17701.964912 secs 2024-04-26T16:46:04.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17704.795153 secs 2024-04-26T16:46:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17701.964912 secs 2024-04-26T16:46:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17704.795153 secs 2024-04-26T16:46:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17701.964912 secs 2024-04-26T16:46:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17704.795153 secs 2024-04-26T16:46:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:04 smithi194 ceph-mon[33017]: pgmap v9925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:04 smithi033 ceph-mon[27845]: pgmap v9925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:04 smithi096 ceph-mon[32753]: pgmap v9925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:06 smithi194 ceph-mon[33017]: pgmap v9926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:06 smithi033 ceph-mon[27845]: pgmap v9926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:06 smithi096 ceph-mon[32753]: pgmap v9926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:07 smithi194 ceph-mon[33017]: pgmap v9927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:07 smithi033 ceph-mon[27845]: pgmap v9927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:07 smithi096 ceph-mon[32753]: pgmap v9927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:46:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17706.964996 secs 2024-04-26T16:46:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17709.795270 secs 2024-04-26T16:46:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17706.964996 secs 2024-04-26T16:46:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17709.795270 secs 2024-04-26T16:46:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17706.964996 secs 2024-04-26T16:46:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17709.795270 secs 2024-04-26T16:46:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:09 smithi194 ceph-mon[33017]: pgmap v9928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:46:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:09 smithi033 ceph-mon[27845]: pgmap v9928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:46:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:09 smithi096 ceph-mon[32753]: pgmap v9928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:46:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:12 smithi194 ceph-mon[33017]: pgmap v9929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:12 smithi033 ceph-mon[27845]: pgmap v9929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:12 smithi096 ceph-mon[32753]: pgmap v9929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17711.965081 secs 2024-04-26T16:46:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17714.795395 secs 2024-04-26T16:46:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17711.965081 secs 2024-04-26T16:46:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17714.795395 secs 2024-04-26T16:46:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17711.965081 secs 2024-04-26T16:46:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17714.795395 secs 2024-04-26T16:46:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:14 smithi194 ceph-mon[33017]: pgmap v9930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:14 smithi033 ceph-mon[27845]: pgmap v9930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:14 smithi096 ceph-mon[32753]: pgmap v9930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:15.843+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:15.843+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:15.843+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:46:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:15 smithi194 ceph-mon[33017]: pgmap v9931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:15 smithi033 ceph-mon[27845]: pgmap v9931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:15 smithi096 ceph-mon[32753]: pgmap v9931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17716.965170 secs 2024-04-26T16:46:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:18 smithi194 ceph-mon[33017]: pgmap v9932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17719.795521 secs 2024-04-26T16:46:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17716.965170 secs 2024-04-26T16:46:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:18 smithi033 ceph-mon[27845]: pgmap v9932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17719.795521 secs 2024-04-26T16:46:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17716.965170 secs 2024-04-26T16:46:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:18 smithi096 ceph-mon[32753]: pgmap v9932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17719.795521 secs 2024-04-26T16:46:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:20 smithi194 ceph-mon[33017]: pgmap v9933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:20 smithi033 ceph-mon[27845]: pgmap v9933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:20 smithi096 ceph-mon[32753]: pgmap v9933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:22 smithi194 ceph-mon[33017]: pgmap v9934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:22 smithi033 ceph-mon[27845]: pgmap v9934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:22 smithi096 ceph-mon[32753]: pgmap v9934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17721.965254 secs 2024-04-26T16:46:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17724.795651 secs 2024-04-26T16:46:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17721.965254 secs 2024-04-26T16:46:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17724.795651 secs 2024-04-26T16:46:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17721.965254 secs 2024-04-26T16:46:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17724.795651 secs 2024-04-26T16:46:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:24 smithi194 ceph-mon[33017]: pgmap v9935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:24 smithi033 ceph-mon[27845]: pgmap v9935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:24 smithi096 ceph-mon[32753]: pgmap v9935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:25 smithi194 ceph-mon[33017]: pgmap v9936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:25 smithi033 ceph-mon[27845]: pgmap v9936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:25 smithi096 ceph-mon[32753]: pgmap v9936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17726.965313 secs 2024-04-26T16:46:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:28 smithi194 ceph-mon[33017]: pgmap v9937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17729.795781 secs 2024-04-26T16:46:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17726.965313 secs 2024-04-26T16:46:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:28 smithi033 ceph-mon[27845]: pgmap v9937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17729.795781 secs 2024-04-26T16:46:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17726.965313 secs 2024-04-26T16:46:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:28 smithi096 ceph-mon[32753]: pgmap v9937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17729.795781 secs 2024-04-26T16:46:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:30.858+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:30.858+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:30.858+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:46:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:30 smithi194 ceph-mon[33017]: pgmap v9938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:30 smithi033 ceph-mon[27845]: pgmap v9938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:30 smithi096 ceph-mon[32753]: pgmap v9938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:32 smithi194 ceph-mon[33017]: pgmap v9939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:32 smithi033 ceph-mon[27845]: pgmap v9939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:32 smithi096 ceph-mon[32753]: pgmap v9939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:34.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17731.965389 secs 2024-04-26T16:46:34.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17734.795906 secs 2024-04-26T16:46:34.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:33 smithi033 ceph-mon[27845]: pgmap v9940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17731.965389 secs 2024-04-26T16:46:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17734.795906 secs 2024-04-26T16:46:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:33 smithi194 ceph-mon[33017]: pgmap v9940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17731.965389 secs 2024-04-26T16:46:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17734.795906 secs 2024-04-26T16:46:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:33 smithi096 ceph-mon[32753]: pgmap v9940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:36 smithi194 ceph-mon[33017]: pgmap v9941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:36 smithi033 ceph-mon[27845]: pgmap v9941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:36 smithi096 ceph-mon[32753]: pgmap v9941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17736.965450 secs 2024-04-26T16:46:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:38 smithi194 ceph-mon[33017]: pgmap v9942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17739.796053 secs 2024-04-26T16:46:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17736.965450 secs 2024-04-26T16:46:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:38 smithi033 ceph-mon[27845]: pgmap v9942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17739.796053 secs 2024-04-26T16:46:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17736.965450 secs 2024-04-26T16:46:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:38 smithi096 ceph-mon[32753]: pgmap v9942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17739.796053 secs 2024-04-26T16:46:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:39.807+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:46:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:39.807+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:46:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:39.807+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:46:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:39.807+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:46:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:40 smithi194 ceph-mon[33017]: pgmap v9943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:40 smithi033 ceph-mon[27845]: pgmap v9943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:40 smithi096 ceph-mon[32753]: pgmap v9943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:41 smithi194 ceph-mon[33017]: pgmap v9944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:41 smithi033 ceph-mon[27845]: pgmap v9944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:41 smithi096 ceph-mon[32753]: pgmap v9944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17741.965510 secs 2024-04-26T16:46:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17741.965510 secs 2024-04-26T16:46:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17741.965510 secs 2024-04-26T16:46:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17744.796190 secs 2024-04-26T16:46:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:44 smithi194 ceph-mon[33017]: pgmap v9945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17744.796190 secs 2024-04-26T16:46:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:44 smithi033 ceph-mon[27845]: pgmap v9945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17744.796190 secs 2024-04-26T16:46:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:44 smithi096 ceph-mon[32753]: pgmap v9945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:45.873+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:45.873+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:46:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:46:45.873+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:46:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:46 smithi194 ceph-mon[33017]: pgmap v9946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:46 smithi033 ceph-mon[27845]: pgmap v9946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:46 smithi096 ceph-mon[32753]: pgmap v9946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:46:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17746.965593 secs 2024-04-26T16:46:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:48 smithi194 ceph-mon[33017]: pgmap v9947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17749.796315 secs 2024-04-26T16:46:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17746.965593 secs 2024-04-26T16:46:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:48 smithi033 ceph-mon[27845]: pgmap v9947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17749.796315 secs 2024-04-26T16:46:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17746.965593 secs 2024-04-26T16:46:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:48 smithi096 ceph-mon[32753]: pgmap v9947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:46:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17749.796315 secs 2024-04-26T16:46:50.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:49 smithi096 ceph-mon[32753]: pgmap v9948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:46:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:49 smithi194 ceph-mon[33017]: pgmap v9948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:46:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:49 smithi033 ceph-mon[27845]: pgmap v9948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:46:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:52 smithi194 ceph-mon[33017]: pgmap v9949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:52 smithi033 ceph-mon[27845]: pgmap v9949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:52 smithi096 ceph-mon[32753]: pgmap v9949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17751.965681 secs 2024-04-26T16:46:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17754.796443 secs 2024-04-26T16:46:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17751.965681 secs 2024-04-26T16:46:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17754.796443 secs 2024-04-26T16:46:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17751.965681 secs 2024-04-26T16:46:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17754.796443 secs 2024-04-26T16:46:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:54 smithi194 ceph-mon[33017]: pgmap v9950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:54 smithi033 ceph-mon[27845]: pgmap v9950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:54 smithi096 ceph-mon[32753]: pgmap v9950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:55 smithi194 ceph-mon[33017]: pgmap v9951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:46:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:55 smithi033 ceph-mon[27845]: pgmap v9951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:46:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:55 smithi096 ceph-mon[32753]: pgmap v9951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:46:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17756.965742 secs 2024-04-26T16:46:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:58 smithi194 ceph-mon[33017]: pgmap v9952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17759.796571 secs 2024-04-26T16:46:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17756.965742 secs 2024-04-26T16:46:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:58 smithi033 ceph-mon[27845]: pgmap v9952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17759.796571 secs 2024-04-26T16:46:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17756.965742 secs 2024-04-26T16:46:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:58 smithi096 ceph-mon[32753]: pgmap v9952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:46:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17759.796571 secs 2024-04-26T16:47:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:46:59 smithi033 ceph-mon[27845]: pgmap v9953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:47:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:46:59 smithi096 ceph-mon[32753]: pgmap v9953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:47:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:46:59 smithi194 ceph-mon[33017]: pgmap v9953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:47:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:00.888+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:00.888+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:47:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:00.888+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:47:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:47:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:01 smithi033 ceph-mon[27845]: pgmap v9954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:01 smithi033 ceph-mon[27845]: pgmap v9955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:47:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:47:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:47:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:01 smithi096 ceph-mon[32753]: pgmap v9954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:01 smithi096 ceph-mon[32753]: pgmap v9955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:47:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:47:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:47:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:01 smithi194 ceph-mon[33017]: pgmap v9954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:01 smithi194 ceph-mon[33017]: pgmap v9955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:47:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17761.965824 secs 2024-04-26T16:47:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17761.965824 secs 2024-04-26T16:47:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17761.965824 secs 2024-04-26T16:47:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17764.796704 secs 2024-04-26T16:47:04.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:04 smithi033 ceph-mon[27845]: pgmap v9956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17764.796704 secs 2024-04-26T16:47:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:04 smithi194 ceph-mon[33017]: pgmap v9956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17764.796704 secs 2024-04-26T16:47:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:04 smithi096 ceph-mon[32753]: pgmap v9956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:06 smithi194 ceph-mon[33017]: pgmap v9957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:06 smithi033 ceph-mon[27845]: pgmap v9957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:06 smithi096 ceph-mon[32753]: pgmap v9957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:08 smithi194 ceph-mon[33017]: pgmap v9958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17766.965911 secs 2024-04-26T16:47:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:08 smithi033 ceph-mon[27845]: pgmap v9958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17766.965911 secs 2024-04-26T16:47:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:08 smithi096 ceph-mon[32753]: pgmap v9958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17766.965911 secs 2024-04-26T16:47:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17769.796826 secs 2024-04-26T16:47:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17769.796826 secs 2024-04-26T16:47:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17769.796826 secs 2024-04-26T16:47:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:10 smithi194 ceph-mon[33017]: pgmap v9959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:10 smithi033 ceph-mon[27845]: pgmap v9959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:10 smithi096 ceph-mon[32753]: pgmap v9959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:47:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:12 smithi194 ceph-mon[33017]: pgmap v9960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:47:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:12 smithi033 ceph-mon[27845]: pgmap v9960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:47:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:12 smithi096 ceph-mon[32753]: pgmap v9960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:47:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17771.966018 secs 2024-04-26T16:47:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17771.966018 secs 2024-04-26T16:47:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17771.966018 secs 2024-04-26T16:47:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17774.796954 secs 2024-04-26T16:47:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:14 smithi194 ceph-mon[33017]: pgmap v9961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17774.796954 secs 2024-04-26T16:47:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:14 smithi033 ceph-mon[27845]: pgmap v9961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17774.796954 secs 2024-04-26T16:47:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:14 smithi096 ceph-mon[32753]: pgmap v9961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:15.891+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:15.891+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:47:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:15.891+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:16 smithi194 ceph-mon[33017]: pgmap v9962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:16 smithi033 ceph-mon[27845]: pgmap v9962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:16 smithi096 ceph-mon[32753]: pgmap v9962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:18 smithi194 ceph-mon[33017]: pgmap v9963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17776.966080 secs 2024-04-26T16:47:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:18 smithi033 ceph-mon[27845]: pgmap v9963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17776.966080 secs 2024-04-26T16:47:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:18 smithi096 ceph-mon[32753]: pgmap v9963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17776.966080 secs 2024-04-26T16:47:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17779.797086 secs 2024-04-26T16:47:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17779.797086 secs 2024-04-26T16:47:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17779.797086 secs 2024-04-26T16:47:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:20 smithi194 ceph-mon[33017]: pgmap v9964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:20 smithi033 ceph-mon[27845]: pgmap v9964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:20 smithi096 ceph-mon[32753]: pgmap v9964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:22 smithi194 ceph-mon[33017]: pgmap v9965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:22 smithi033 ceph-mon[27845]: pgmap v9965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:22 smithi096 ceph-mon[32753]: pgmap v9965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17781.966163 secs 2024-04-26T16:47:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17781.966163 secs 2024-04-26T16:47:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17781.966163 secs 2024-04-26T16:47:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:24.815+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:47:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:24.815+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:47:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:24.815+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:47:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:24.815+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:47:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17784.797212 secs 2024-04-26T16:47:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:24 smithi033 ceph-mon[27845]: pgmap v9966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17784.797212 secs 2024-04-26T16:47:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:24 smithi096 ceph-mon[32753]: pgmap v9966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17784.797212 secs 2024-04-26T16:47:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:24 smithi194 ceph-mon[33017]: pgmap v9966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:26 smithi033 ceph-mon[27845]: pgmap v9967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:26 smithi096 ceph-mon[32753]: pgmap v9967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:26 smithi194 ceph-mon[33017]: pgmap v9967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:28 smithi033 ceph-mon[27845]: pgmap v9968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17786.966246 secs 2024-04-26T16:47:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:28 smithi096 ceph-mon[32753]: pgmap v9968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17786.966246 secs 2024-04-26T16:47:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:28 smithi194 ceph-mon[33017]: pgmap v9968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17786.966246 secs 2024-04-26T16:47:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17789.797333 secs 2024-04-26T16:47:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17789.797333 secs 2024-04-26T16:47:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17789.797333 secs 2024-04-26T16:47:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:30.906+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:30.906+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:47:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:30.906+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:30 smithi033 ceph-mon[27845]: pgmap v9969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:30 smithi096 ceph-mon[32753]: pgmap v9969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:30 smithi194 ceph-mon[33017]: pgmap v9969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:32 smithi033 ceph-mon[27845]: pgmap v9970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:32 smithi096 ceph-mon[32753]: pgmap v9970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:32 smithi194 ceph-mon[33017]: pgmap v9970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17791.966306 secs 2024-04-26T16:47:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17791.966306 secs 2024-04-26T16:47:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17791.966306 secs 2024-04-26T16:47:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17794.797462 secs 2024-04-26T16:47:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:34 smithi033 ceph-mon[27845]: pgmap v9971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17794.797462 secs 2024-04-26T16:47:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:34 smithi096 ceph-mon[32753]: pgmap v9971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17794.797462 secs 2024-04-26T16:47:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:34 smithi194 ceph-mon[33017]: pgmap v9971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:36 smithi033 ceph-mon[27845]: pgmap v9972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:36 smithi096 ceph-mon[32753]: pgmap v9972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:36 smithi194 ceph-mon[33017]: pgmap v9972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:38 smithi033 ceph-mon[27845]: pgmap v9973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17796.966401 secs 2024-04-26T16:47:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:38 smithi096 ceph-mon[32753]: pgmap v9973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17796.966401 secs 2024-04-26T16:47:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:38 smithi194 ceph-mon[33017]: pgmap v9973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17796.966401 secs 2024-04-26T16:47:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17799.797544 secs 2024-04-26T16:47:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17799.797544 secs 2024-04-26T16:47:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17799.797544 secs 2024-04-26T16:47:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:40 smithi194 ceph-mon[33017]: pgmap v9974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:40 smithi033 ceph-mon[27845]: pgmap v9974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:40 smithi096 ceph-mon[32753]: pgmap v9974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:42 smithi194 ceph-mon[33017]: pgmap v9975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:42 smithi033 ceph-mon[27845]: pgmap v9975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:42 smithi096 ceph-mon[32753]: pgmap v9975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17801.966484 secs 2024-04-26T16:47:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17801.966484 secs 2024-04-26T16:47:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17801.966484 secs 2024-04-26T16:47:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17804.797679 secs 2024-04-26T16:47:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:44 smithi194 ceph-mon[33017]: pgmap v9976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17804.797679 secs 2024-04-26T16:47:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:44 smithi033 ceph-mon[27845]: pgmap v9976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17804.797679 secs 2024-04-26T16:47:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:44 smithi096 ceph-mon[32753]: pgmap v9976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:45.920+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:45.920+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:47:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:47:45.920+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:47:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:46 smithi194 ceph-mon[33017]: pgmap v9977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:46 smithi033 ceph-mon[27845]: pgmap v9977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:46 smithi096 ceph-mon[32753]: pgmap v9977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:48 smithi194 ceph-mon[33017]: pgmap v9978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17806.966569 secs 2024-04-26T16:47:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:48 smithi033 ceph-mon[27845]: pgmap v9978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17806.966569 secs 2024-04-26T16:47:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:48 smithi096 ceph-mon[32753]: pgmap v9978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17806.966569 secs 2024-04-26T16:47:50.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17809.797803 secs 2024-04-26T16:47:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17809.797803 secs 2024-04-26T16:47:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17809.797803 secs 2024-04-26T16:47:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:50 smithi194 ceph-mon[33017]: pgmap v9979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:50 smithi033 ceph-mon[27845]: pgmap v9979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:50 smithi096 ceph-mon[32753]: pgmap v9979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:52 smithi194 ceph-mon[33017]: pgmap v9980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:53.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:52 smithi033 ceph-mon[27845]: pgmap v9980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:52 smithi096 ceph-mon[32753]: pgmap v9980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17811.966647 secs 2024-04-26T16:47:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17811.966647 secs 2024-04-26T16:47:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17811.966647 secs 2024-04-26T16:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17814.797926 secs 2024-04-26T16:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:54 smithi194 ceph-mon[33017]: pgmap v9981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17814.797926 secs 2024-04-26T16:47:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:54 smithi033 ceph-mon[27845]: pgmap v9981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17814.797926 secs 2024-04-26T16:47:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:54 smithi096 ceph-mon[32753]: pgmap v9981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:56 smithi194 ceph-mon[33017]: pgmap v9982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:56 smithi033 ceph-mon[27845]: pgmap v9982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:56 smithi096 ceph-mon[32753]: pgmap v9982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:47:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:58 smithi194 ceph-mon[33017]: pgmap v9983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17816.966728 secs 2024-04-26T16:47:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:58 smithi033 ceph-mon[27845]: pgmap v9983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17816.966728 secs 2024-04-26T16:47:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:58 smithi096 ceph-mon[32753]: pgmap v9983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:47:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17816.966728 secs 2024-04-26T16:48:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:47:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17819.798037 secs 2024-04-26T16:48:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:47:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17819.798037 secs 2024-04-26T16:48:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:47:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17819.798037 secs 2024-04-26T16:48:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:00.931+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:00.931+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:00.931+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:48:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:00 smithi194 ceph-mon[33017]: pgmap v9984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:00 smithi033 ceph-mon[27845]: pgmap v9984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:00 smithi096 ceph-mon[32753]: pgmap v9984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:48:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:48:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:48:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:02 smithi194 ceph-mon[33017]: pgmap v9985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:02 smithi033 ceph-mon[27845]: pgmap v9985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:02 smithi096 ceph-mon[32753]: pgmap v9985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17821.966811 secs 2024-04-26T16:48:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:48:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:48:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:48:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17824.798168 secs 2024-04-26T16:48:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17821.966811 secs 2024-04-26T16:48:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:48:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:48:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:48:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17824.798168 secs 2024-04-26T16:48:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17821.966811 secs 2024-04-26T16:48:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:48:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:48:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:48:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17824.798168 secs 2024-04-26T16:48:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:04 smithi194 ceph-mon[33017]: pgmap v9986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:04 smithi194 ceph-mon[33017]: pgmap v9987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:04 smithi033 ceph-mon[27845]: pgmap v9986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:04 smithi033 ceph-mon[27845]: pgmap v9987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:04 smithi096 ceph-mon[32753]: pgmap v9986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:04 smithi096 ceph-mon[32753]: pgmap v9987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:06 smithi194 ceph-mon[33017]: pgmap v9988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:06 smithi033 ceph-mon[27845]: pgmap v9988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:06 smithi096 ceph-mon[32753]: pgmap v9988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:08 smithi194 ceph-mon[33017]: pgmap v9989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17826.966894 secs 2024-04-26T16:48:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17829.798288 secs 2024-04-26T16:48:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:08 smithi033 ceph-mon[27845]: pgmap v9989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17826.966894 secs 2024-04-26T16:48:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17829.798288 secs 2024-04-26T16:48:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:08 smithi096 ceph-mon[32753]: pgmap v9989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17826.966894 secs 2024-04-26T16:48:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17829.798288 secs 2024-04-26T16:48:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:10 smithi194 ceph-mon[33017]: pgmap v9990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:48:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:10 smithi033 ceph-mon[27845]: pgmap v9990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:48:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:10 smithi096 ceph-mon[32753]: pgmap v9990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:48:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:12 smithi194 ceph-mon[33017]: pgmap v9991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:12 smithi033 ceph-mon[27845]: pgmap v9991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:12 smithi096 ceph-mon[32753]: pgmap v9991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:48:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17831.966978 secs 2024-04-26T16:48:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17834.798417 secs 2024-04-26T16:48:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17831.966978 secs 2024-04-26T16:48:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17834.798417 secs 2024-04-26T16:48:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17831.966978 secs 2024-04-26T16:48:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17834.798417 secs 2024-04-26T16:48:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:14 smithi194 ceph-mon[33017]: pgmap v9992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:48:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:14 smithi033 ceph-mon[27845]: pgmap v9992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:48:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:14 smithi096 ceph-mon[32753]: pgmap v9992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:48:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:15.940+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:15.940+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:15.940+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:48:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:16 smithi194 ceph-mon[33017]: pgmap v9993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:16 smithi033 ceph-mon[27845]: pgmap v9993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:16 smithi096 ceph-mon[32753]: pgmap v9993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:18 smithi194 ceph-mon[33017]: pgmap v9994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17836.967005 secs 2024-04-26T16:48:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17839.798494 secs 2024-04-26T16:48:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:18 smithi033 ceph-mon[27845]: pgmap v9994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17836.967005 secs 2024-04-26T16:48:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17839.798494 secs 2024-04-26T16:48:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:18 smithi096 ceph-mon[32753]: pgmap v9994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17836.967005 secs 2024-04-26T16:48:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17839.798494 secs 2024-04-26T16:48:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:19 smithi033 ceph-mon[27845]: pgmap v9995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:19 smithi096 ceph-mon[32753]: pgmap v9995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:19 smithi194 ceph-mon[33017]: pgmap v9995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:22 smithi033 ceph-mon[27845]: pgmap v9996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:22 smithi096 ceph-mon[32753]: pgmap v9996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:22 smithi194 ceph-mon[33017]: pgmap v9996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17841.967088 secs 2024-04-26T16:48:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:24 smithi033 ceph-mon[27845]: pgmap v9997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17844.798613 secs 2024-04-26T16:48:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17841.967088 secs 2024-04-26T16:48:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:24 smithi096 ceph-mon[32753]: pgmap v9997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17844.798613 secs 2024-04-26T16:48:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17841.967088 secs 2024-04-26T16:48:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:24 smithi194 ceph-mon[33017]: pgmap v9997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17844.798613 secs 2024-04-26T16:48:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:26 smithi033 ceph-mon[27845]: pgmap v9998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:26 smithi096 ceph-mon[32753]: pgmap v9998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:26 smithi194 ceph-mon[33017]: pgmap v9998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:28 smithi033 ceph-mon[27845]: pgmap v9999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:28 smithi096 ceph-mon[32753]: pgmap v9999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:28 smithi194 ceph-mon[33017]: pgmap v9999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17846.967148 secs 2024-04-26T16:48:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17849.798705 secs 2024-04-26T16:48:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17846.967148 secs 2024-04-26T16:48:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17849.798705 secs 2024-04-26T16:48:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17846.967148 secs 2024-04-26T16:48:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17849.798705 secs 2024-04-26T16:48:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:30 smithi033 ceph-mon[27845]: pgmap v10000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:30 smithi096 ceph-mon[32753]: pgmap v10000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:30 smithi194 ceph-mon[33017]: pgmap v10000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:30.945+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:30.945+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:30.945+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:48:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:32 smithi033 ceph-mon[27845]: pgmap v10001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:32 smithi096 ceph-mon[32753]: pgmap v10001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:32 smithi194 ceph-mon[33017]: pgmap v10001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17851.967230 secs 2024-04-26T16:48:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:34 smithi096 ceph-mon[32753]: pgmap v10002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17854.798825 secs 2024-04-26T16:48:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17851.967230 secs 2024-04-26T16:48:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:34 smithi194 ceph-mon[33017]: pgmap v10002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17854.798825 secs 2024-04-26T16:48:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17851.967230 secs 2024-04-26T16:48:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:34 smithi033 ceph-mon[27845]: pgmap v10002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17854.798825 secs 2024-04-26T16:48:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:36 smithi194 ceph-mon[33017]: pgmap v10003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:36 smithi033 ceph-mon[27845]: pgmap v10003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:36 smithi096 ceph-mon[32753]: pgmap v10003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:38 smithi194 ceph-mon[33017]: pgmap v10004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:38 smithi033 ceph-mon[27845]: pgmap v10004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:38 smithi096 ceph-mon[32753]: pgmap v10004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17856.967291 secs 2024-04-26T16:48:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17859.798961 secs 2024-04-26T16:48:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17856.967291 secs 2024-04-26T16:48:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17859.798961 secs 2024-04-26T16:48:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17856.967291 secs 2024-04-26T16:48:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17859.798961 secs 2024-04-26T16:48:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:40 smithi194 ceph-mon[33017]: pgmap v10005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:40 smithi033 ceph-mon[27845]: pgmap v10005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:40 smithi096 ceph-mon[32753]: pgmap v10005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:42 smithi194 ceph-mon[33017]: pgmap v10006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:42 smithi033 ceph-mon[27845]: pgmap v10006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:42 smithi096 ceph-mon[32753]: pgmap v10006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17861.967351 secs 2024-04-26T16:48:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:44 smithi194 ceph-mon[33017]: pgmap v10007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17864.799098 secs 2024-04-26T16:48:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17861.967351 secs 2024-04-26T16:48:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:44 smithi033 ceph-mon[27845]: pgmap v10007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17864.799098 secs 2024-04-26T16:48:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17861.967351 secs 2024-04-26T16:48:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:44 smithi096 ceph-mon[32753]: pgmap v10007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17864.799098 secs 2024-04-26T16:48:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:45.953+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:45.953+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:48:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:48:45.953+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:48:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:46 smithi194 ceph-mon[33017]: pgmap v10008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:46 smithi033 ceph-mon[27845]: pgmap v10008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:46 smithi096 ceph-mon[32753]: pgmap v10008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:48 smithi194 ceph-mon[33017]: pgmap v10009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:48 smithi033 ceph-mon[27845]: pgmap v10009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:48 smithi096 ceph-mon[32753]: pgmap v10009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17866.967409 secs 2024-04-26T16:48:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17869.799235 secs 2024-04-26T16:48:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17866.967409 secs 2024-04-26T16:48:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17869.799235 secs 2024-04-26T16:48:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17866.967409 secs 2024-04-26T16:48:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17869.799235 secs 2024-04-26T16:48:50.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:50 smithi096 ceph-mon[32753]: pgmap v10010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:50 smithi194 ceph-mon[33017]: pgmap v10010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:50 smithi033 ceph-mon[27845]: pgmap v10010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:52 smithi194 ceph-mon[33017]: pgmap v10011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:52 smithi033 ceph-mon[27845]: pgmap v10011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:52 smithi096 ceph-mon[32753]: pgmap v10011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17871.967492 secs 2024-04-26T16:48:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:54 smithi194 ceph-mon[33017]: pgmap v10012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17874.799351 secs 2024-04-26T16:48:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17871.967492 secs 2024-04-26T16:48:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:54 smithi033 ceph-mon[27845]: pgmap v10012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17874.799351 secs 2024-04-26T16:48:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17871.967492 secs 2024-04-26T16:48:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:54 smithi096 ceph-mon[32753]: pgmap v10012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17874.799351 secs 2024-04-26T16:48:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:56 smithi194 ceph-mon[33017]: pgmap v10013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:56 smithi033 ceph-mon[27845]: pgmap v10013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:56 smithi096 ceph-mon[32753]: pgmap v10013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:48:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:58 smithi194 ceph-mon[33017]: pgmap v10014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:58 smithi033 ceph-mon[27845]: pgmap v10014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:58 smithi096 ceph-mon[32753]: pgmap v10014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:48:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17876.967575 secs 2024-04-26T16:48:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17879.799504 secs 2024-04-26T16:48:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17876.967575 secs 2024-04-26T16:48:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17879.799504 secs 2024-04-26T16:48:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17876.967575 secs 2024-04-26T16:48:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17879.799504 secs 2024-04-26T16:49:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:00 smithi194 ceph-mon[33017]: pgmap v10015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:00 smithi033 ceph-mon[27845]: pgmap v10015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:00 smithi096 ceph-mon[32753]: pgmap v10015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:00.969+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:00.969+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:00.969+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:49:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:02 smithi194 ceph-mon[33017]: pgmap v10016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:02 smithi033 ceph-mon[27845]: pgmap v10016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:02 smithi096 ceph-mon[32753]: pgmap v10016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17881.967635 secs 2024-04-26T16:49:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:04 smithi194 ceph-mon[33017]: pgmap v10017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17884.799570 secs 2024-04-26T16:49:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17881.967635 secs 2024-04-26T16:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:04 smithi033 ceph-mon[27845]: pgmap v10017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17884.799570 secs 2024-04-26T16:49:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:49:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17881.967635 secs 2024-04-26T16:49:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:04 smithi096 ceph-mon[32753]: pgmap v10017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17884.799570 secs 2024-04-26T16:49:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:49:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:49:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:49:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:05 smithi194 ceph-mon[33017]: pgmap v10018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:49:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:05 smithi194 ceph-mon[33017]: pgmap v10019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:49:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:49:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:49:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:05 smithi033 ceph-mon[27845]: pgmap v10018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:49:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:05 smithi033 ceph-mon[27845]: pgmap v10019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:49:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:49:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:49:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:05 smithi096 ceph-mon[32753]: pgmap v10018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:49:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:05 smithi096 ceph-mon[32753]: pgmap v10019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:49:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:06 smithi033 ceph-mon[27845]: pgmap v10020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:06 smithi096 ceph-mon[32753]: pgmap v10020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:06 smithi194 ceph-mon[33017]: pgmap v10020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17886.967717 secs 2024-04-26T16:49:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17886.967717 secs 2024-04-26T16:49:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17886.967717 secs 2024-04-26T16:49:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17889.799714 secs 2024-04-26T16:49:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:09 smithi194 ceph-mon[33017]: pgmap v10021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17889.799714 secs 2024-04-26T16:49:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:09 smithi033 ceph-mon[27845]: pgmap v10021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17889.799714 secs 2024-04-26T16:49:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:09 smithi096 ceph-mon[32753]: pgmap v10021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:10 smithi033 ceph-mon[27845]: pgmap v10022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:10 smithi096 ceph-mon[32753]: pgmap v10022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:10 smithi194 ceph-mon[33017]: pgmap v10022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:12 smithi033 ceph-mon[27845]: pgmap v10023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:12 smithi096 ceph-mon[32753]: pgmap v10023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:12 smithi194 ceph-mon[33017]: pgmap v10023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:49:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17891.967797 secs 2024-04-26T16:49:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17894.799863 secs 2024-04-26T16:49:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17891.967797 secs 2024-04-26T16:49:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17894.799863 secs 2024-04-26T16:49:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17891.967797 secs 2024-04-26T16:49:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17894.799863 secs 2024-04-26T16:49:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:14 smithi033 ceph-mon[27845]: pgmap v10024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:49:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:14 smithi096 ceph-mon[32753]: pgmap v10024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:49:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:14 smithi194 ceph-mon[33017]: pgmap v10024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:49:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:15.985+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:15.985+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:15.985+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:49:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:16 smithi033 ceph-mon[27845]: pgmap v10025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:16 smithi096 ceph-mon[32753]: pgmap v10025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:16 smithi194 ceph-mon[33017]: pgmap v10025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17896.967881 secs 2024-04-26T16:49:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17896.967881 secs 2024-04-26T16:49:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17896.967881 secs 2024-04-26T16:49:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17899.800004 secs 2024-04-26T16:49:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:19 smithi033 ceph-mon[27845]: pgmap v10026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17899.800004 secs 2024-04-26T16:49:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:19 smithi096 ceph-mon[32753]: pgmap v10026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17899.800004 secs 2024-04-26T16:49:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:19 smithi194 ceph-mon[33017]: pgmap v10026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:20 smithi033 ceph-mon[27845]: pgmap v10027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:20 smithi096 ceph-mon[32753]: pgmap v10027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:20 smithi194 ceph-mon[33017]: pgmap v10027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:22 smithi033 ceph-mon[27845]: pgmap v10028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:22 smithi096 ceph-mon[32753]: pgmap v10028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:22 smithi194 ceph-mon[33017]: pgmap v10028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17901.967967 secs 2024-04-26T16:49:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17904.800145 secs 2024-04-26T16:49:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17901.967967 secs 2024-04-26T16:49:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17904.800145 secs 2024-04-26T16:49:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17901.967967 secs 2024-04-26T16:49:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17904.800145 secs 2024-04-26T16:49:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:24 smithi033 ceph-mon[27845]: pgmap v10029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:24 smithi096 ceph-mon[32753]: pgmap v10029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:24 smithi194 ceph-mon[33017]: pgmap v10029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:26 smithi033 ceph-mon[27845]: pgmap v10030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:26 smithi096 ceph-mon[32753]: pgmap v10030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:26 smithi194 ceph-mon[33017]: pgmap v10030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17906.968046 secs 2024-04-26T16:49:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17906.968046 secs 2024-04-26T16:49:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17906.968046 secs 2024-04-26T16:49:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17909.800282 secs 2024-04-26T16:49:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:29 smithi194 ceph-mon[33017]: pgmap v10031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17909.800282 secs 2024-04-26T16:49:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:29 smithi033 ceph-mon[27845]: pgmap v10031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17909.800282 secs 2024-04-26T16:49:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:29 smithi096 ceph-mon[32753]: pgmap v10031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:30.986+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:30.986+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:30.986+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:49:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:30 smithi033 ceph-mon[27845]: pgmap v10032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:30 smithi096 ceph-mon[32753]: pgmap v10032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:30 smithi194 ceph-mon[33017]: pgmap v10032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:32 smithi033 ceph-mon[27845]: pgmap v10033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:32 smithi096 ceph-mon[32753]: pgmap v10033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:32 smithi194 ceph-mon[33017]: pgmap v10033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17911.968132 secs 2024-04-26T16:49:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17914.800365 secs 2024-04-26T16:49:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17911.968132 secs 2024-04-26T16:49:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17914.800365 secs 2024-04-26T16:49:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17911.968132 secs 2024-04-26T16:49:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17914.800365 secs 2024-04-26T16:49:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:34 smithi033 ceph-mon[27845]: pgmap v10034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:34 smithi096 ceph-mon[32753]: pgmap v10034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:34 smithi194 ceph-mon[33017]: pgmap v10034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:37.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:36 smithi033 ceph-mon[27845]: pgmap v10035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:36 smithi096 ceph-mon[32753]: pgmap v10035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:36 smithi194 ceph-mon[33017]: pgmap v10035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17916.968220 secs 2024-04-26T16:49:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17916.968220 secs 2024-04-26T16:49:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17916.968220 secs 2024-04-26T16:49:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17919.800483 secs 2024-04-26T16:49:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:39 smithi194 ceph-mon[33017]: pgmap v10036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17919.800483 secs 2024-04-26T16:49:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:39 smithi033 ceph-mon[27845]: pgmap v10036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17919.800483 secs 2024-04-26T16:49:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:39 smithi096 ceph-mon[32753]: pgmap v10036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:40 smithi033 ceph-mon[27845]: pgmap v10037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:40 smithi096 ceph-mon[32753]: pgmap v10037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:40 smithi194 ceph-mon[33017]: pgmap v10037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:42 smithi033 ceph-mon[27845]: pgmap v10038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:42 smithi096 ceph-mon[32753]: pgmap v10038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:42 smithi194 ceph-mon[33017]: pgmap v10038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17921.968281 secs 2024-04-26T16:49:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17924.800598 secs 2024-04-26T16:49:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17921.968281 secs 2024-04-26T16:49:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17924.800598 secs 2024-04-26T16:49:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17921.968281 secs 2024-04-26T16:49:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17924.800598 secs 2024-04-26T16:49:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:44 smithi033 ceph-mon[27845]: pgmap v10039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:44 smithi096 ceph-mon[32753]: pgmap v10039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:44 smithi194 ceph-mon[33017]: pgmap v10039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:46.000+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:46.000+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:49:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:49:46.000+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:49:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:46 smithi033 ceph-mon[27845]: pgmap v10040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:46 smithi096 ceph-mon[32753]: pgmap v10040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:46 smithi194 ceph-mon[33017]: pgmap v10040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17926.968341 secs 2024-04-26T16:49:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17926.968341 secs 2024-04-26T16:49:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17926.968341 secs 2024-04-26T16:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17929.800737 secs 2024-04-26T16:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:49 smithi194 ceph-mon[33017]: pgmap v10041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17929.800737 secs 2024-04-26T16:49:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:49 smithi033 ceph-mon[27845]: pgmap v10041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17929.800737 secs 2024-04-26T16:49:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:49 smithi096 ceph-mon[32753]: pgmap v10041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:50 smithi033 ceph-mon[27845]: pgmap v10042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:50 smithi096 ceph-mon[32753]: pgmap v10042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:50 smithi194 ceph-mon[33017]: pgmap v10042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:52 smithi033 ceph-mon[27845]: pgmap v10043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:52 smithi096 ceph-mon[32753]: pgmap v10043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:52 smithi194 ceph-mon[33017]: pgmap v10043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17931.968402 secs 2024-04-26T16:49:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17934.800877 secs 2024-04-26T16:49:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17931.968402 secs 2024-04-26T16:49:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17934.800877 secs 2024-04-26T16:49:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17931.968402 secs 2024-04-26T16:49:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17934.800877 secs 2024-04-26T16:49:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:54 smithi033 ceph-mon[27845]: pgmap v10044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:54 smithi096 ceph-mon[32753]: pgmap v10044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:54 smithi194 ceph-mon[33017]: pgmap v10044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:49:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:56 smithi033 ceph-mon[27845]: pgmap v10045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:56 smithi096 ceph-mon[32753]: pgmap v10045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:56 smithi194 ceph-mon[33017]: pgmap v10045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17936.968463 secs 2024-04-26T16:49:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17936.968463 secs 2024-04-26T16:49:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17936.968463 secs 2024-04-26T16:49:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17939.801015 secs 2024-04-26T16:49:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:49:59 smithi194 ceph-mon[33017]: pgmap v10046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17939.801015 secs 2024-04-26T16:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:49:59 smithi033 ceph-mon[27845]: pgmap v10046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:49:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17939.801015 secs 2024-04-26T16:49:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:49:59 smithi096 ceph-mon[32753]: pgmap v10046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:50:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:50:00.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T16:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:01.009+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:01.009+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:01.009+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:01 smithi194 ceph-mon[33017]: pgmap v10047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:01 smithi033 ceph-mon[27845]: pgmap v10047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:01 smithi096 ceph-mon[32753]: pgmap v10047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:02 smithi033 ceph-mon[27845]: pgmap v10048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:02 smithi096 ceph-mon[32753]: pgmap v10048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:02 smithi194 ceph-mon[33017]: pgmap v10048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17941.968543 secs 2024-04-26T16:50:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17944.801159 secs 2024-04-26T16:50:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17941.968543 secs 2024-04-26T16:50:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17944.801159 secs 2024-04-26T16:50:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17941.968543 secs 2024-04-26T16:50:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17944.801159 secs 2024-04-26T16:50:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:04 smithi033 ceph-mon[27845]: pgmap v10049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:04 smithi096 ceph-mon[32753]: pgmap v10049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:04 smithi194 ceph-mon[33017]: pgmap v10049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:50:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:50:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:50:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:50:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:50:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:06 smithi194 ceph-mon[33017]: pgmap v10050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:06 smithi194 ceph-mon[33017]: pgmap v10051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:50:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:50:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:06 smithi033 ceph-mon[27845]: pgmap v10050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:06 smithi033 ceph-mon[27845]: pgmap v10051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:06 smithi096 ceph-mon[32753]: pgmap v10050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:06 smithi096 ceph-mon[32753]: pgmap v10051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:50:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17946.968605 secs 2024-04-26T16:50:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17946.968605 secs 2024-04-26T16:50:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17946.968605 secs 2024-04-26T16:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17949.801278 secs 2024-04-26T16:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:09 smithi194 ceph-mon[33017]: pgmap v10052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17949.801278 secs 2024-04-26T16:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:09 smithi033 ceph-mon[27845]: pgmap v10052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17949.801278 secs 2024-04-26T16:50:09.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:09 smithi096 ceph-mon[32753]: pgmap v10052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:11 smithi194 ceph-mon[33017]: pgmap v10053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:11 smithi033 ceph-mon[27845]: pgmap v10053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:11 smithi096 ceph-mon[32753]: pgmap v10053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:13 smithi194 ceph-mon[33017]: pgmap v10054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17951.968686 secs 2024-04-26T16:50:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:13 smithi033 ceph-mon[27845]: pgmap v10054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17951.968686 secs 2024-04-26T16:50:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:13 smithi096 ceph-mon[32753]: pgmap v10054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17951.968686 secs 2024-04-26T16:50:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17954.801398 secs 2024-04-26T16:50:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17954.801398 secs 2024-04-26T16:50:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17954.801398 secs 2024-04-26T16:50:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:15 smithi194 ceph-mon[33017]: pgmap v10055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:15 smithi033 ceph-mon[27845]: pgmap v10055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:15 smithi096 ceph-mon[32753]: pgmap v10055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:50:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:16.026+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:16.026+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:16.026+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:50:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:17 smithi194 ceph-mon[33017]: pgmap v10056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:50:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:17 smithi033 ceph-mon[27845]: pgmap v10056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:50:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:17 smithi096 ceph-mon[32753]: pgmap v10056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:50:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17956.968769 secs 2024-04-26T16:50:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17956.968769 secs 2024-04-26T16:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17956.968769 secs 2024-04-26T16:50:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17959.801514 secs 2024-04-26T16:50:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:19 smithi194 ceph-mon[33017]: pgmap v10057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17959.801514 secs 2024-04-26T16:50:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:19 smithi033 ceph-mon[27845]: pgmap v10057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17959.801514 secs 2024-04-26T16:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:19 smithi096 ceph-mon[32753]: pgmap v10057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:21 smithi194 ceph-mon[33017]: pgmap v10058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:21 smithi033 ceph-mon[27845]: pgmap v10058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:21 smithi096 ceph-mon[32753]: pgmap v10058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:23 smithi194 ceph-mon[33017]: pgmap v10059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17961.968852 secs 2024-04-26T16:50:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:23 smithi033 ceph-mon[27845]: pgmap v10059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17961.968852 secs 2024-04-26T16:50:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:23 smithi096 ceph-mon[32753]: pgmap v10059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17961.968852 secs 2024-04-26T16:50:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:24.845+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:50:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:24.845+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:50:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:24.846+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:50:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:24.846+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:50:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17964.801629 secs 2024-04-26T16:50:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17964.801629 secs 2024-04-26T16:50:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17964.801629 secs 2024-04-26T16:50:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:25 smithi194 ceph-mon[33017]: pgmap v10060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:25 smithi033 ceph-mon[27845]: pgmap v10060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:25 smithi096 ceph-mon[32753]: pgmap v10060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:27 smithi194 ceph-mon[33017]: pgmap v10061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:27 smithi033 ceph-mon[27845]: pgmap v10061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:27 smithi096 ceph-mon[32753]: pgmap v10061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17966.968935 secs 2024-04-26T16:50:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17966.968935 secs 2024-04-26T16:50:28.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17966.968935 secs 2024-04-26T16:50:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17969.801759 secs 2024-04-26T16:50:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:29 smithi194 ceph-mon[33017]: pgmap v10062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17969.801759 secs 2024-04-26T16:50:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:29 smithi033 ceph-mon[27845]: pgmap v10062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17969.801759 secs 2024-04-26T16:50:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:29 smithi096 ceph-mon[32753]: pgmap v10062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:31.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:31.032+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:31.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:31.032+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:50:31.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:31.032+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:31 smithi033 ceph-mon[27845]: pgmap v10063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:31 smithi096 ceph-mon[32753]: pgmap v10063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:31 smithi194 ceph-mon[33017]: pgmap v10063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:33 smithi033 ceph-mon[27845]: pgmap v10064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17971.968998 secs 2024-04-26T16:50:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:33 smithi096 ceph-mon[32753]: pgmap v10064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17971.968998 secs 2024-04-26T16:50:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:33 smithi194 ceph-mon[33017]: pgmap v10064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17971.968998 secs 2024-04-26T16:50:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17974.801918 secs 2024-04-26T16:50:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17974.801918 secs 2024-04-26T16:50:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17974.801918 secs 2024-04-26T16:50:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:35 smithi033 ceph-mon[27845]: pgmap v10065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:35 smithi096 ceph-mon[32753]: pgmap v10065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:35 smithi194 ceph-mon[33017]: pgmap v10065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:37 smithi033 ceph-mon[27845]: pgmap v10066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:37 smithi096 ceph-mon[32753]: pgmap v10066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:37 smithi194 ceph-mon[33017]: pgmap v10066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17976.969058 secs 2024-04-26T16:50:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17976.969058 secs 2024-04-26T16:50:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17976.969058 secs 2024-04-26T16:50:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17979.802048 secs 2024-04-26T16:50:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:39 smithi033 ceph-mon[27845]: pgmap v10067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17979.802048 secs 2024-04-26T16:50:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:39 smithi096 ceph-mon[32753]: pgmap v10067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17979.802048 secs 2024-04-26T16:50:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:39 smithi194 ceph-mon[33017]: pgmap v10067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:41 smithi033 ceph-mon[27845]: pgmap v10068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:41 smithi096 ceph-mon[32753]: pgmap v10068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:41 smithi194 ceph-mon[33017]: pgmap v10068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:43 smithi033 ceph-mon[27845]: pgmap v10069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17981.969144 secs 2024-04-26T16:50:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:43 smithi096 ceph-mon[32753]: pgmap v10069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17981.969144 secs 2024-04-26T16:50:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:43 smithi194 ceph-mon[33017]: pgmap v10069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17981.969144 secs 2024-04-26T16:50:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17984.802176 secs 2024-04-26T16:50:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17984.802176 secs 2024-04-26T16:50:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17984.802176 secs 2024-04-26T16:50:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:45 smithi033 ceph-mon[27845]: pgmap v10070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:45 smithi096 ceph-mon[32753]: pgmap v10070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:46.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:46.048+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:46.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:46.048+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:50:46.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:50:46.048+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:50:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:45 smithi194 ceph-mon[33017]: pgmap v10070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:47 smithi033 ceph-mon[27845]: pgmap v10071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:47 smithi096 ceph-mon[32753]: pgmap v10071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:47 smithi194 ceph-mon[33017]: pgmap v10071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:49.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17986.969206 secs 2024-04-26T16:50:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17986.969206 secs 2024-04-26T16:50:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17986.969206 secs 2024-04-26T16:50:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17989.802274 secs 2024-04-26T16:50:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:49 smithi194 ceph-mon[33017]: pgmap v10072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17989.802274 secs 2024-04-26T16:50:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:49 smithi033 ceph-mon[27845]: pgmap v10072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17989.802274 secs 2024-04-26T16:50:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:49 smithi096 ceph-mon[32753]: pgmap v10072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:51.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:51 smithi096 ceph-mon[32753]: pgmap v10073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:51 smithi194 ceph-mon[33017]: pgmap v10073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:51 smithi033 ceph-mon[27845]: pgmap v10073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:53 smithi194 ceph-mon[33017]: pgmap v10074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17991.969268 secs 2024-04-26T16:50:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:53 smithi033 ceph-mon[27845]: pgmap v10074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17991.969268 secs 2024-04-26T16:50:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:53 smithi096 ceph-mon[32753]: pgmap v10074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17991.969268 secs 2024-04-26T16:50:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17994.802393 secs 2024-04-26T16:50:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17994.802393 secs 2024-04-26T16:50:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17994.802393 secs 2024-04-26T16:50:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:55 smithi194 ceph-mon[33017]: pgmap v10075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:55 smithi033 ceph-mon[27845]: pgmap v10075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:55 smithi096 ceph-mon[32753]: pgmap v10075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:50:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:57 smithi194 ceph-mon[33017]: pgmap v10076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:57 smithi033 ceph-mon[27845]: pgmap v10076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:57 smithi096 ceph-mon[32753]: pgmap v10076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:50:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 17996.969329 secs 2024-04-26T16:50:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 17996.969329 secs 2024-04-26T16:50:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 17996.969329 secs 2024-04-26T16:51:00.018 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 17999.802509 secs 2024-04-26T16:51:00.018 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:50:59 smithi033 ceph-mon[27845]: pgmap v10077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 17999.802509 secs 2024-04-26T16:51:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:50:59 smithi194 ceph-mon[33017]: pgmap v10077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 17999.802509 secs 2024-04-26T16:51:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:50:59 smithi096 ceph-mon[32753]: pgmap v10077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:01.050+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:01.050+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:01.050+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:51:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:01 smithi194 ceph-mon[33017]: pgmap v10078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:01 smithi033 ceph-mon[27845]: pgmap v10078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:01 smithi096 ceph-mon[32753]: pgmap v10078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:03 smithi194 ceph-mon[33017]: pgmap v10079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18001.969413 secs 2024-04-26T16:51:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:03 smithi033 ceph-mon[27845]: pgmap v10079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18001.969413 secs 2024-04-26T16:51:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:03 smithi096 ceph-mon[32753]: pgmap v10079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18001.969413 secs 2024-04-26T16:51:05.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18004.802591 secs 2024-04-26T16:51:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18004.802591 secs 2024-04-26T16:51:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18004.802591 secs 2024-04-26T16:51:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:05 smithi194 ceph-mon[33017]: pgmap v10080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:05 smithi033 ceph-mon[27845]: pgmap v10080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:05 smithi096 ceph-mon[32753]: pgmap v10080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:07 smithi194 ceph-mon[33017]: pgmap v10081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:51:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:07 smithi033 ceph-mon[27845]: pgmap v10081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:51:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:07 smithi096 ceph-mon[32753]: pgmap v10081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:51:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18006.969475 secs 2024-04-26T16:51:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18006.969475 secs 2024-04-26T16:51:09.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18006.969475 secs 2024-04-26T16:51:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18009.802681 secs 2024-04-26T16:51:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:09 smithi194 ceph-mon[33017]: pgmap v10082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:51:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:51:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18009.802681 secs 2024-04-26T16:51:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:09 smithi033 ceph-mon[27845]: pgmap v10082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:51:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:51:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18009.802681 secs 2024-04-26T16:51:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:09 smithi096 ceph-mon[32753]: pgmap v10082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:51:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:51:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:51:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:10 smithi194 ceph-mon[33017]: pgmap v10083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:51:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:10 smithi194 ceph-mon[33017]: pgmap v10084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:51:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:10 smithi033 ceph-mon[27845]: pgmap v10083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:51:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:10 smithi033 ceph-mon[27845]: pgmap v10084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:51:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:10 smithi096 ceph-mon[32753]: pgmap v10083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:51:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:10 smithi096 ceph-mon[32753]: pgmap v10084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:51:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:12 smithi194 ceph-mon[33017]: pgmap v10085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:12 smithi033 ceph-mon[27845]: pgmap v10085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:12 smithi096 ceph-mon[32753]: pgmap v10085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18011.969557 secs 2024-04-26T16:51:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18011.969557 secs 2024-04-26T16:51:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18011.969557 secs 2024-04-26T16:51:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:14 smithi194 ceph-mon[33017]: pgmap v10086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:51:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18014.802823 secs 2024-04-26T16:51:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:14 smithi033 ceph-mon[27845]: pgmap v10086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:51:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18014.802823 secs 2024-04-26T16:51:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:14 smithi096 ceph-mon[32753]: pgmap v10086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:51:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18014.802823 secs 2024-04-26T16:51:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:16.066+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:16.066+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:16.066+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:51:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:16 smithi194 ceph-mon[33017]: pgmap v10087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:16 smithi033 ceph-mon[27845]: pgmap v10087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:16 smithi096 ceph-mon[32753]: pgmap v10087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:18 smithi194 ceph-mon[33017]: pgmap v10088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18016.969642 secs 2024-04-26T16:51:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18019.802957 secs 2024-04-26T16:51:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:18 smithi033 ceph-mon[27845]: pgmap v10088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18016.969642 secs 2024-04-26T16:51:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18019.802957 secs 2024-04-26T16:51:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:18 smithi096 ceph-mon[32753]: pgmap v10088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T16:51:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18016.969642 secs 2024-04-26T16:51:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18019.802957 secs 2024-04-26T16:51:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:20 smithi194 ceph-mon[33017]: pgmap v10089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:51:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:20 smithi033 ceph-mon[27845]: pgmap v10089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:51:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:20 smithi096 ceph-mon[32753]: pgmap v10089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:51:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:22 smithi194 ceph-mon[33017]: pgmap v10090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:22 smithi033 ceph-mon[27845]: pgmap v10090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:22 smithi096 ceph-mon[32753]: pgmap v10090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18021.969724 secs 2024-04-26T16:51:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18024.803077 secs 2024-04-26T16:51:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18021.969724 secs 2024-04-26T16:51:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18024.803077 secs 2024-04-26T16:51:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18021.969724 secs 2024-04-26T16:51:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18024.803077 secs 2024-04-26T16:51:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:24 smithi194 ceph-mon[33017]: pgmap v10091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:24 smithi033 ceph-mon[27845]: pgmap v10091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:24 smithi096 ceph-mon[32753]: pgmap v10091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:26 smithi194 ceph-mon[33017]: pgmap v10092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:26 smithi033 ceph-mon[27845]: pgmap v10092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:26 smithi096 ceph-mon[32753]: pgmap v10092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:28 smithi194 ceph-mon[33017]: pgmap v10093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18026.969807 secs 2024-04-26T16:51:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18029.803199 secs 2024-04-26T16:51:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:28 smithi033 ceph-mon[27845]: pgmap v10093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18026.969807 secs 2024-04-26T16:51:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18029.803199 secs 2024-04-26T16:51:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:28 smithi096 ceph-mon[32753]: pgmap v10093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18026.969807 secs 2024-04-26T16:51:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18029.803199 secs 2024-04-26T16:51:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:31.075+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:31.075+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:31.075+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:51:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:30 smithi194 ceph-mon[33017]: pgmap v10094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:30 smithi033 ceph-mon[27845]: pgmap v10094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:30 smithi096 ceph-mon[32753]: pgmap v10094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:32 smithi033 ceph-mon[27845]: pgmap v10095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:32 smithi096 ceph-mon[32753]: pgmap v10095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:32 smithi194 ceph-mon[33017]: pgmap v10095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18031.969890 secs 2024-04-26T16:51:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18034.803331 secs 2024-04-26T16:51:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18031.969890 secs 2024-04-26T16:51:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18034.803331 secs 2024-04-26T16:51:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18031.969890 secs 2024-04-26T16:51:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18034.803331 secs 2024-04-26T16:51:35.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:34 smithi033 ceph-mon[27845]: pgmap v10096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:34 smithi096 ceph-mon[32753]: pgmap v10096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:34 smithi194 ceph-mon[33017]: pgmap v10096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:36 smithi033 ceph-mon[27845]: pgmap v10097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:36 smithi096 ceph-mon[32753]: pgmap v10097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:36 smithi194 ceph-mon[33017]: pgmap v10097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:38 smithi033 ceph-mon[27845]: pgmap v10098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18036.969994 secs 2024-04-26T16:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18039.803406 secs 2024-04-26T16:51:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:38 smithi096 ceph-mon[32753]: pgmap v10098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18036.969994 secs 2024-04-26T16:51:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18039.803406 secs 2024-04-26T16:51:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:38 smithi194 ceph-mon[33017]: pgmap v10098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18036.969994 secs 2024-04-26T16:51:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18039.803406 secs 2024-04-26T16:51:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:39 smithi096 ceph-mon[32753]: pgmap v10099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:39 smithi194 ceph-mon[33017]: pgmap v10099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:39 smithi033 ceph-mon[27845]: pgmap v10099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:42 smithi033 ceph-mon[27845]: pgmap v10100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:42 smithi096 ceph-mon[32753]: pgmap v10100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:42 smithi194 ceph-mon[33017]: pgmap v10100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18041.970055 secs 2024-04-26T16:51:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:44 smithi033 ceph-mon[27845]: pgmap v10101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18044.803529 secs 2024-04-26T16:51:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18041.970055 secs 2024-04-26T16:51:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:44 smithi096 ceph-mon[32753]: pgmap v10101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18044.803529 secs 2024-04-26T16:51:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18041.970055 secs 2024-04-26T16:51:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:44 smithi194 ceph-mon[33017]: pgmap v10101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18044.803529 secs 2024-04-26T16:51:46.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:46.076+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:46.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:46.076+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:51:46.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:51:46.076+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:51:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:46 smithi033 ceph-mon[27845]: pgmap v10102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:46 smithi096 ceph-mon[32753]: pgmap v10102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:46 smithi194 ceph-mon[33017]: pgmap v10102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:48 smithi033 ceph-mon[27845]: pgmap v10103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:48 smithi096 ceph-mon[32753]: pgmap v10103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:48 smithi194 ceph-mon[33017]: pgmap v10103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18046.970116 secs 2024-04-26T16:51:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18049.803659 secs 2024-04-26T16:51:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18046.970116 secs 2024-04-26T16:51:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18049.803659 secs 2024-04-26T16:51:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18046.970116 secs 2024-04-26T16:51:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18049.803659 secs 2024-04-26T16:51:50.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:50 smithi194 ceph-mon[33017]: pgmap v10104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:50 smithi033 ceph-mon[27845]: pgmap v10104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:50 smithi096 ceph-mon[32753]: pgmap v10104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:52.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:52 smithi096 ceph-mon[32753]: pgmap v10105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:52 smithi194 ceph-mon[33017]: pgmap v10105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:52 smithi033 ceph-mon[27845]: pgmap v10105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18051.970177 secs 2024-04-26T16:51:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:54 smithi194 ceph-mon[33017]: pgmap v10106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18054.803790 secs 2024-04-26T16:51:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18051.970177 secs 2024-04-26T16:51:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:54 smithi033 ceph-mon[27845]: pgmap v10106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18054.803790 secs 2024-04-26T16:51:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18051.970177 secs 2024-04-26T16:51:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:54 smithi096 ceph-mon[32753]: pgmap v10106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18054.803790 secs 2024-04-26T16:51:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:56 smithi194 ceph-mon[33017]: pgmap v10107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:56 smithi033 ceph-mon[27845]: pgmap v10107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:56 smithi096 ceph-mon[32753]: pgmap v10107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:51:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:58 smithi194 ceph-mon[33017]: pgmap v10108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:58 smithi033 ceph-mon[27845]: pgmap v10108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:58 smithi096 ceph-mon[32753]: pgmap v10108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:51:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18056.970236 secs 2024-04-26T16:51:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:51:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18059.803864 secs 2024-04-26T16:51:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18056.970236 secs 2024-04-26T16:51:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:51:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18059.803864 secs 2024-04-26T16:51:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18056.970236 secs 2024-04-26T16:51:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:51:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18059.803864 secs 2024-04-26T16:52:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:00 smithi194 ceph-mon[33017]: pgmap v10109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:00 smithi033 ceph-mon[27845]: pgmap v10109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:00 smithi096 ceph-mon[32753]: pgmap v10109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:01.081+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:01.081+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:01.081+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:52:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:02 smithi194 ceph-mon[33017]: pgmap v10110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:02 smithi033 ceph-mon[27845]: pgmap v10110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:02 smithi096 ceph-mon[32753]: pgmap v10110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18061.970320 secs 2024-04-26T16:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:04 smithi194 ceph-mon[33017]: pgmap v10111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18064.804010 secs 2024-04-26T16:52:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18061.970320 secs 2024-04-26T16:52:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:04 smithi033 ceph-mon[27845]: pgmap v10111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18064.804010 secs 2024-04-26T16:52:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18061.970320 secs 2024-04-26T16:52:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:04 smithi096 ceph-mon[32753]: pgmap v10111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18064.804010 secs 2024-04-26T16:52:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:06 smithi194 ceph-mon[33017]: pgmap v10112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:06 smithi033 ceph-mon[27845]: pgmap v10112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:06 smithi096 ceph-mon[32753]: pgmap v10112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:08 smithi194 ceph-mon[33017]: pgmap v10113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:08 smithi033 ceph-mon[27845]: pgmap v10113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:08 smithi096 ceph-mon[32753]: pgmap v10113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18066.970402 secs 2024-04-26T16:52:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18069.804168 secs 2024-04-26T16:52:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18066.970402 secs 2024-04-26T16:52:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18069.804168 secs 2024-04-26T16:52:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18066.970402 secs 2024-04-26T16:52:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18069.804168 secs 2024-04-26T16:52:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:10 smithi194 ceph-mon[33017]: pgmap v10114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:52:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:10 smithi033 ceph-mon[27845]: pgmap v10114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:52:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:10 smithi096 ceph-mon[32753]: pgmap v10114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:52:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:52:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:52:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:11 smithi194 ceph-mon[33017]: pgmap v10115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:52:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:11 smithi194 ceph-mon[33017]: pgmap v10116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:52:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:52:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:52:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:11 smithi033 ceph-mon[27845]: pgmap v10115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:52:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:11 smithi033 ceph-mon[27845]: pgmap v10116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:52:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:52:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:52:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:11 smithi096 ceph-mon[32753]: pgmap v10115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:52:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:11 smithi096 ceph-mon[32753]: pgmap v10116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:52:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:13 smithi194 ceph-mon[33017]: pgmap v10117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18071.970484 secs 2024-04-26T16:52:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:13 smithi033 ceph-mon[27845]: pgmap v10117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18071.970484 secs 2024-04-26T16:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:13 smithi096 ceph-mon[32753]: pgmap v10117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18071.970484 secs 2024-04-26T16:52:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18074.804308 secs 2024-04-26T16:52:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18074.804308 secs 2024-04-26T16:52:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18074.804308 secs 2024-04-26T16:52:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:16.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:16.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:16.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:52:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:15 smithi194 ceph-mon[33017]: pgmap v10118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:15 smithi033 ceph-mon[27845]: pgmap v10118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:15 smithi096 ceph-mon[32753]: pgmap v10118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:17 smithi194 ceph-mon[33017]: pgmap v10119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:17 smithi033 ceph-mon[27845]: pgmap v10119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:17 smithi096 ceph-mon[32753]: pgmap v10119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18076.970568 secs 2024-04-26T16:52:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18079.804378 secs 2024-04-26T16:52:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18076.970568 secs 2024-04-26T16:52:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18079.804378 secs 2024-04-26T16:52:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18076.970568 secs 2024-04-26T16:52:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18079.804378 secs 2024-04-26T16:52:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:19 smithi194 ceph-mon[33017]: pgmap v10120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:19 smithi033 ceph-mon[27845]: pgmap v10120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:19 smithi096 ceph-mon[32753]: pgmap v10120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:52:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:21 smithi194 ceph-mon[33017]: pgmap v10121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:52:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:21 smithi033 ceph-mon[27845]: pgmap v10121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:52:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:21 smithi096 ceph-mon[32753]: pgmap v10121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:52:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18081.970656 secs 2024-04-26T16:52:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:23 smithi194 ceph-mon[33017]: pgmap v10122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18084.804534 secs 2024-04-26T16:52:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18081.970656 secs 2024-04-26T16:52:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:23 smithi033 ceph-mon[27845]: pgmap v10122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18084.804534 secs 2024-04-26T16:52:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18081.970656 secs 2024-04-26T16:52:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:23 smithi096 ceph-mon[32753]: pgmap v10122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18084.804534 secs 2024-04-26T16:52:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:25 smithi194 ceph-mon[33017]: pgmap v10123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:25 smithi033 ceph-mon[27845]: pgmap v10123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:25 smithi096 ceph-mon[32753]: pgmap v10123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:27 smithi194 ceph-mon[33017]: pgmap v10124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:27 smithi033 ceph-mon[27845]: pgmap v10124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:27 smithi096 ceph-mon[32753]: pgmap v10124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18086.970737 secs 2024-04-26T16:52:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18089.804621 secs 2024-04-26T16:52:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18086.970737 secs 2024-04-26T16:52:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18089.804621 secs 2024-04-26T16:52:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18086.970737 secs 2024-04-26T16:52:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18089.804621 secs 2024-04-26T16:52:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:29 smithi194 ceph-mon[33017]: pgmap v10125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:29 smithi033 ceph-mon[27845]: pgmap v10125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:29 smithi096 ceph-mon[32753]: pgmap v10125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:31.101+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:31.101+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:52:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:31.101+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:31 smithi033 ceph-mon[27845]: pgmap v10126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:31 smithi096 ceph-mon[32753]: pgmap v10126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:31 smithi194 ceph-mon[33017]: pgmap v10126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18091.970820 secs 2024-04-26T16:52:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:33 smithi033 ceph-mon[27845]: pgmap v10127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18094.804754 secs 2024-04-26T16:52:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18091.970820 secs 2024-04-26T16:52:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:33 smithi096 ceph-mon[32753]: pgmap v10127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18094.804754 secs 2024-04-26T16:52:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18091.970820 secs 2024-04-26T16:52:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:33 smithi194 ceph-mon[33017]: pgmap v10127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18094.804754 secs 2024-04-26T16:52:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:35 smithi033 ceph-mon[27845]: pgmap v10128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:35 smithi096 ceph-mon[32753]: pgmap v10128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:35 smithi194 ceph-mon[33017]: pgmap v10128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:37 smithi033 ceph-mon[27845]: pgmap v10129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:37 smithi096 ceph-mon[32753]: pgmap v10129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:37 smithi194 ceph-mon[33017]: pgmap v10129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18096.970906 secs 2024-04-26T16:52:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18099.804891 secs 2024-04-26T16:52:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18096.970906 secs 2024-04-26T16:52:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18099.804891 secs 2024-04-26T16:52:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18096.970906 secs 2024-04-26T16:52:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18099.804891 secs 2024-04-26T16:52:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:39 smithi033 ceph-mon[27845]: pgmap v10130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:39 smithi096 ceph-mon[32753]: pgmap v10130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:39 smithi194 ceph-mon[33017]: pgmap v10130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:41 smithi033 ceph-mon[27845]: pgmap v10131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:41 smithi096 ceph-mon[32753]: pgmap v10131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:41 smithi194 ceph-mon[33017]: pgmap v10131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:52:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:42 smithi033 ceph-mon[27845]: pgmap v10132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:42 smithi096 ceph-mon[32753]: pgmap v10132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:42 smithi194 ceph-mon[33017]: pgmap v10132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18101.970985 secs 2024-04-26T16:52:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18104.804991 secs 2024-04-26T16:52:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18101.970985 secs 2024-04-26T16:52:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18104.804991 secs 2024-04-26T16:52:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18101.970985 secs 2024-04-26T16:52:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18104.804991 secs 2024-04-26T16:52:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:44 smithi033 ceph-mon[27845]: pgmap v10133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:44 smithi096 ceph-mon[32753]: pgmap v10133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:44 smithi194 ceph-mon[33017]: pgmap v10133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:46.104+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:46.105+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:52:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:52:46.105+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:52:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:47 smithi194 ceph-mon[33017]: pgmap v10134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:47 smithi033 ceph-mon[27845]: pgmap v10134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:47 smithi096 ceph-mon[32753]: pgmap v10134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:49.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18106.971074 secs 2024-04-26T16:52:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18106.971074 secs 2024-04-26T16:52:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18106.971074 secs 2024-04-26T16:52:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18109.805130 secs 2024-04-26T16:52:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:49 smithi194 ceph-mon[33017]: pgmap v10135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18109.805130 secs 2024-04-26T16:52:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:49 smithi033 ceph-mon[27845]: pgmap v10135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18109.805130 secs 2024-04-26T16:52:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:49 smithi096 ceph-mon[32753]: pgmap v10135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:51 smithi194 ceph-mon[33017]: pgmap v10136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:52.293 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:51 smithi096 ceph-mon[32753]: pgmap v10136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:52.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:51 smithi033 ceph-mon[27845]: pgmap v10136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18111.971154 secs 2024-04-26T16:52:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:53 smithi194 ceph-mon[33017]: pgmap v10137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18114.805247 secs 2024-04-26T16:52:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18111.971154 secs 2024-04-26T16:52:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:53 smithi033 ceph-mon[27845]: pgmap v10137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18114.805247 secs 2024-04-26T16:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18111.971154 secs 2024-04-26T16:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:53 smithi096 ceph-mon[32753]: pgmap v10137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:52:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18114.805247 secs 2024-04-26T16:52:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:55 smithi194 ceph-mon[33017]: pgmap v10138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:55 smithi033 ceph-mon[27845]: pgmap v10138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:55 smithi096 ceph-mon[32753]: pgmap v10138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:52:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:57 smithi194 ceph-mon[33017]: pgmap v10139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:57 smithi033 ceph-mon[27845]: pgmap v10139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:57 smithi096 ceph-mon[32753]: pgmap v10139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:52:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18116.971241 secs 2024-04-26T16:52:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18119.805336 secs 2024-04-26T16:52:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18116.971241 secs 2024-04-26T16:52:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18119.805336 secs 2024-04-26T16:52:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18116.971241 secs 2024-04-26T16:52:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18119.805336 secs 2024-04-26T16:53:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:52:59 smithi194 ceph-mon[33017]: pgmap v10140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:52:59 smithi033 ceph-mon[27845]: pgmap v10140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:52:59 smithi096 ceph-mon[32753]: pgmap v10140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:01.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:01.119+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:01.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:01.119+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:01.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:01.119+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:53:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:01 smithi194 ceph-mon[33017]: pgmap v10141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:01 smithi033 ceph-mon[27845]: pgmap v10141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:01 smithi096 ceph-mon[32753]: pgmap v10141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18121.971301 secs 2024-04-26T16:53:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:03 smithi194 ceph-mon[33017]: pgmap v10142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18124.805393 secs 2024-04-26T16:53:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18121.971301 secs 2024-04-26T16:53:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:03 smithi033 ceph-mon[27845]: pgmap v10142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18124.805393 secs 2024-04-26T16:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18121.971301 secs 2024-04-26T16:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:03 smithi096 ceph-mon[32753]: pgmap v10142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18124.805393 secs 2024-04-26T16:53:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:05 smithi033 ceph-mon[27845]: pgmap v10143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:05 smithi096 ceph-mon[32753]: pgmap v10143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:05 smithi194 ceph-mon[33017]: pgmap v10143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:07 smithi033 ceph-mon[27845]: pgmap v10144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:07 smithi096 ceph-mon[32753]: pgmap v10144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:07 smithi194 ceph-mon[33017]: pgmap v10144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18126.971385 secs 2024-04-26T16:53:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18129.805510 secs 2024-04-26T16:53:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18126.971385 secs 2024-04-26T16:53:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18129.805510 secs 2024-04-26T16:53:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18126.971385 secs 2024-04-26T16:53:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18129.805510 secs 2024-04-26T16:53:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:09 smithi033 ceph-mon[27845]: pgmap v10145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:09 smithi096 ceph-mon[32753]: pgmap v10145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:09 smithi194 ceph-mon[33017]: pgmap v10145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:11 smithi033 ceph-mon[27845]: pgmap v10146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:53:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:11 smithi096 ceph-mon[32753]: pgmap v10146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:11 smithi194 ceph-mon[33017]: pgmap v10146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:53:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18131.971467 secs 2024-04-26T16:53:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:13 smithi033 ceph-mon[27845]: pgmap v10147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18134.805576 secs 2024-04-26T16:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18131.971467 secs 2024-04-26T16:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:13 smithi096 ceph-mon[32753]: pgmap v10147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18134.805576 secs 2024-04-26T16:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18131.971467 secs 2024-04-26T16:53:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:13 smithi194 ceph-mon[33017]: pgmap v10147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18134.805576 secs 2024-04-26T16:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:15.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:16.135+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:16.135+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:16.136+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:53:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:53:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:53:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:53:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:53:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:15 smithi033 ceph-mon[27845]: pgmap v10148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:15 smithi096 ceph-mon[32753]: pgmap v10148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T16:53:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T16:53:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T16:53:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T16:53:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:15 smithi194 ceph-mon[33017]: pgmap v10148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: pgmap v10149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:53:18.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:53:18.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: pgmap v10149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:53:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:53:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: pgmap v10149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:53:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:18 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:53:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:18 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:53:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:18 smithi033 ceph-mon[27845]: pgmap v10150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:53:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:18 smithi033 ceph-mon[27845]: pgmap v10151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18136.971548 secs 2024-04-26T16:53:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18139.805721 secs 2024-04-26T16:53:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:18 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:53:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:18 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:53:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:18 smithi096 ceph-mon[32753]: pgmap v10150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:53:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:18 smithi096 ceph-mon[32753]: pgmap v10151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18136.971548 secs 2024-04-26T16:53:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18139.805721 secs 2024-04-26T16:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:18 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T16:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:18 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T16:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:18 smithi194 ceph-mon[33017]: pgmap v10150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T16:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:18 smithi194 ceph-mon[33017]: pgmap v10151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18136.971548 secs 2024-04-26T16:53:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18139.805721 secs 2024-04-26T16:53:21.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:20 smithi194 ceph-mon[33017]: pgmap v10152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:53:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:20 smithi033 ceph-mon[27845]: pgmap v10152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:53:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:20 smithi096 ceph-mon[32753]: pgmap v10152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:53:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:21 smithi033 ceph-mon[27845]: pgmap v10153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:21 smithi096 ceph-mon[32753]: pgmap v10153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:21 smithi194 ceph-mon[33017]: pgmap v10153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18141.971631 secs 2024-04-26T16:53:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:24 smithi194 ceph-mon[33017]: pgmap v10154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:53:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18144.805844 secs 2024-04-26T16:53:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18141.971631 secs 2024-04-26T16:53:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:24 smithi033 ceph-mon[27845]: pgmap v10154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:53:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18144.805844 secs 2024-04-26T16:53:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18141.971631 secs 2024-04-26T16:53:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:24 smithi096 ceph-mon[32753]: pgmap v10154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T16:53:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18144.805844 secs 2024-04-26T16:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:24.876+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:53:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:24.876+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:24.877+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:53:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:24.877+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:53:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:26 smithi194 ceph-mon[33017]: pgmap v10155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:26 smithi033 ceph-mon[27845]: pgmap v10155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:26 smithi096 ceph-mon[32753]: pgmap v10155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:53:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:28 smithi194 ceph-mon[33017]: pgmap v10156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:53:28.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:28 smithi033 ceph-mon[27845]: pgmap v10156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:53:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:28 smithi096 ceph-mon[32753]: pgmap v10156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:53:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18146.971719 secs 2024-04-26T16:53:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18149.805962 secs 2024-04-26T16:53:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18146.971719 secs 2024-04-26T16:53:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18149.805962 secs 2024-04-26T16:53:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18146.971719 secs 2024-04-26T16:53:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18149.805962 secs 2024-04-26T16:53:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:30 smithi194 ceph-mon[33017]: pgmap v10157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:30 smithi033 ceph-mon[27845]: pgmap v10157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:30 smithi096 ceph-mon[32753]: pgmap v10157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:31.151+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:31.151+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:31.151+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:53:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:32 smithi194 ceph-mon[33017]: pgmap v10158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:32 smithi033 ceph-mon[27845]: pgmap v10158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:32 smithi096 ceph-mon[32753]: pgmap v10158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18151.971777 secs 2024-04-26T16:53:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:34 smithi194 ceph-mon[33017]: pgmap v10159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18154.806082 secs 2024-04-26T16:53:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18151.971777 secs 2024-04-26T16:53:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:34 smithi033 ceph-mon[27845]: pgmap v10159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18154.806082 secs 2024-04-26T16:53:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18151.971777 secs 2024-04-26T16:53:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:34 smithi096 ceph-mon[32753]: pgmap v10159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18154.806082 secs 2024-04-26T16:53:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:36 smithi194 ceph-mon[33017]: pgmap v10160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:36 smithi033 ceph-mon[27845]: pgmap v10160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:36 smithi096 ceph-mon[32753]: pgmap v10160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:38 smithi033 ceph-mon[27845]: pgmap v10161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:38 smithi194 ceph-mon[33017]: pgmap v10161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:38 smithi096 ceph-mon[32753]: pgmap v10161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18156.971836 secs 2024-04-26T16:53:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18159.806170 secs 2024-04-26T16:53:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18156.971836 secs 2024-04-26T16:53:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18159.806170 secs 2024-04-26T16:53:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18156.971836 secs 2024-04-26T16:53:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18159.806170 secs 2024-04-26T16:53:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:40 smithi194 ceph-mon[33017]: pgmap v10162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:40 smithi033 ceph-mon[27845]: pgmap v10162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:40 smithi096 ceph-mon[32753]: pgmap v10162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:42 smithi194 ceph-mon[33017]: pgmap v10163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:42 smithi033 ceph-mon[27845]: pgmap v10163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:42 smithi096 ceph-mon[32753]: pgmap v10163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18161.971919 secs 2024-04-26T16:53:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:44 smithi194 ceph-mon[33017]: pgmap v10164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18164.806291 secs 2024-04-26T16:53:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18161.971919 secs 2024-04-26T16:53:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:44 smithi033 ceph-mon[27845]: pgmap v10164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18164.806291 secs 2024-04-26T16:53:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18161.971919 secs 2024-04-26T16:53:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:44 smithi096 ceph-mon[32753]: pgmap v10164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18164.806291 secs 2024-04-26T16:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:46.167+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:46.167+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:53:46.167+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:53:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:46 smithi194 ceph-mon[33017]: pgmap v10165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:46 smithi033 ceph-mon[27845]: pgmap v10165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:46 smithi096 ceph-mon[32753]: pgmap v10165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:48 smithi194 ceph-mon[33017]: pgmap v10166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:48 smithi033 ceph-mon[27845]: pgmap v10166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:48 smithi096 ceph-mon[32753]: pgmap v10166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18166.972003 secs 2024-04-26T16:53:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18169.806411 secs 2024-04-26T16:53:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18166.972003 secs 2024-04-26T16:53:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18169.806411 secs 2024-04-26T16:53:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18166.972003 secs 2024-04-26T16:53:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18169.806411 secs 2024-04-26T16:53:51.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:50 smithi033 ceph-mon[27845]: pgmap v10167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:50 smithi194 ceph-mon[33017]: pgmap v10167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:50 smithi096 ceph-mon[32753]: pgmap v10167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:52 smithi194 ceph-mon[33017]: pgmap v10168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:52 smithi033 ceph-mon[27845]: pgmap v10168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:52 smithi096 ceph-mon[32753]: pgmap v10168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18171.972061 secs 2024-04-26T16:53:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18171.972061 secs 2024-04-26T16:53:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18171.972061 secs 2024-04-26T16:53:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:54 smithi194 ceph-mon[33017]: pgmap v10169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18174.806530 secs 2024-04-26T16:53:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:54 smithi033 ceph-mon[27845]: pgmap v10169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18174.806530 secs 2024-04-26T16:53:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:54 smithi096 ceph-mon[32753]: pgmap v10169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18174.806530 secs 2024-04-26T16:53:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:56 smithi194 ceph-mon[33017]: pgmap v10170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:56 smithi033 ceph-mon[27845]: pgmap v10170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:56 smithi096 ceph-mon[32753]: pgmap v10170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:53:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:58 smithi194 ceph-mon[33017]: pgmap v10171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18176.972122 secs 2024-04-26T16:53:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:58 smithi033 ceph-mon[27845]: pgmap v10171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18176.972122 secs 2024-04-26T16:53:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:58 smithi096 ceph-mon[32753]: pgmap v10171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:53:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18176.972122 secs 2024-04-26T16:54:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18179.806648 secs 2024-04-26T16:54:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18179.806648 secs 2024-04-26T16:54:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18179.806648 secs 2024-04-26T16:54:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:00 smithi194 ceph-mon[33017]: pgmap v10172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:00 smithi033 ceph-mon[27845]: pgmap v10172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:01.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:01.181+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:01.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:01.181+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:01.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:01.181+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:54:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:00 smithi096 ceph-mon[32753]: pgmap v10172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:02 smithi194 ceph-mon[33017]: pgmap v10173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:02 smithi033 ceph-mon[27845]: pgmap v10173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:02 smithi096 ceph-mon[32753]: pgmap v10173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18181.972203 secs 2024-04-26T16:54:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18181.972203 secs 2024-04-26T16:54:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18181.972203 secs 2024-04-26T16:54:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:04 smithi194 ceph-mon[33017]: pgmap v10174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18184.806775 secs 2024-04-26T16:54:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:04 smithi033 ceph-mon[27845]: pgmap v10174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18184.806775 secs 2024-04-26T16:54:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:04 smithi096 ceph-mon[32753]: pgmap v10174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18184.806775 secs 2024-04-26T16:54:07.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:06 smithi033 ceph-mon[27845]: pgmap v10175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:06 smithi194 ceph-mon[33017]: pgmap v10175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:06 smithi096 ceph-mon[32753]: pgmap v10175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:08 smithi194 ceph-mon[33017]: pgmap v10176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18186.972265 secs 2024-04-26T16:54:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18189.806886 secs 2024-04-26T16:54:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:08 smithi033 ceph-mon[27845]: pgmap v10176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18186.972265 secs 2024-04-26T16:54:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18189.806886 secs 2024-04-26T16:54:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:08 smithi096 ceph-mon[32753]: pgmap v10176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18186.972265 secs 2024-04-26T16:54:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18189.806886 secs 2024-04-26T16:54:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:10 smithi194 ceph-mon[33017]: pgmap v10177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:10 smithi033 ceph-mon[27845]: pgmap v10177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:10 smithi096 ceph-mon[32753]: pgmap v10177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:12 smithi194 ceph-mon[33017]: pgmap v10178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:12 smithi033 ceph-mon[27845]: pgmap v10178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:12 smithi096 ceph-mon[32753]: pgmap v10178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18191.972329 secs 2024-04-26T16:54:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18194.807004 secs 2024-04-26T16:54:14.193 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18191.972329 secs 2024-04-26T16:54:14.193 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18194.807004 secs 2024-04-26T16:54:14.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18191.972329 secs 2024-04-26T16:54:14.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18194.807004 secs 2024-04-26T16:54:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:14 smithi194 ceph-mon[33017]: pgmap v10179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:14 smithi033 ceph-mon[27845]: pgmap v10179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:14 smithi096 ceph-mon[32753]: pgmap v10179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:16.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:16.189+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:16.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:16.189+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:16.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:16.189+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:54:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:16 smithi194 ceph-mon[33017]: pgmap v10180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:16 smithi033 ceph-mon[27845]: pgmap v10180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:16 smithi096 ceph-mon[32753]: pgmap v10180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:54:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:54:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:54:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:18 smithi033 ceph-mon[27845]: pgmap v10181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18196.972393 secs 2024-04-26T16:54:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18199.807158 secs 2024-04-26T16:54:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:18 smithi096 ceph-mon[32753]: pgmap v10181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18196.972393 secs 2024-04-26T16:54:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18199.807158 secs 2024-04-26T16:54:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:18 smithi194 ceph-mon[33017]: pgmap v10181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18196.972393 secs 2024-04-26T16:54:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18199.807158 secs 2024-04-26T16:54:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:54:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:54:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:19 smithi033 ceph-mon[27845]: pgmap v10182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:19 smithi033 ceph-mon[27845]: pgmap v10183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:54:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:54:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:54:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:19 smithi096 ceph-mon[32753]: pgmap v10182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:19 smithi096 ceph-mon[32753]: pgmap v10183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:54:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:54:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:54:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:19 smithi194 ceph-mon[33017]: pgmap v10182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:19 smithi194 ceph-mon[33017]: pgmap v10183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:54:21.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:20 smithi033 ceph-mon[27845]: pgmap v10184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:21.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:20 smithi194 ceph-mon[33017]: pgmap v10184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:20 smithi096 ceph-mon[32753]: pgmap v10184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:22 smithi194 ceph-mon[33017]: pgmap v10185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:22 smithi033 ceph-mon[27845]: pgmap v10185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:22 smithi096 ceph-mon[32753]: pgmap v10185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:24.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18201.972450 secs 2024-04-26T16:54:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18204.807228 secs 2024-04-26T16:54:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18201.972450 secs 2024-04-26T16:54:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18204.807228 secs 2024-04-26T16:54:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18201.972450 secs 2024-04-26T16:54:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18204.807228 secs 2024-04-26T16:54:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:24 smithi194 ceph-mon[33017]: pgmap v10186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:24 smithi033 ceph-mon[27845]: pgmap v10186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:24 smithi096 ceph-mon[32753]: pgmap v10186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:26 smithi194 ceph-mon[33017]: pgmap v10187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:26 smithi033 ceph-mon[27845]: pgmap v10187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:26 smithi096 ceph-mon[32753]: pgmap v10187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:54:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18206.972509 secs 2024-04-26T16:54:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18206.972509 secs 2024-04-26T16:54:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18206.972509 secs 2024-04-26T16:54:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18209.807309 secs 2024-04-26T16:54:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:29 smithi194 ceph-mon[33017]: pgmap v10188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:54:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18209.807309 secs 2024-04-26T16:54:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:29 smithi033 ceph-mon[27845]: pgmap v10188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:54:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18209.807309 secs 2024-04-26T16:54:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:29 smithi096 ceph-mon[32753]: pgmap v10188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:54:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:31.200+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:31.201+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:31.201+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:54:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:30 smithi194 ceph-mon[33017]: pgmap v10189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:30 smithi033 ceph-mon[27845]: pgmap v10189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:30 smithi096 ceph-mon[32753]: pgmap v10189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:32 smithi194 ceph-mon[33017]: pgmap v10190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:32 smithi033 ceph-mon[27845]: pgmap v10190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:32 smithi096 ceph-mon[32753]: pgmap v10190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18211.972566 secs 2024-04-26T16:54:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18214.807428 secs 2024-04-26T16:54:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18211.972566 secs 2024-04-26T16:54:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18214.807428 secs 2024-04-26T16:54:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18211.972566 secs 2024-04-26T16:54:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18214.807428 secs 2024-04-26T16:54:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:34 smithi194 ceph-mon[33017]: pgmap v10191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:34 smithi033 ceph-mon[27845]: pgmap v10191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:34 smithi096 ceph-mon[32753]: pgmap v10191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:37.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:36 smithi033 ceph-mon[27845]: pgmap v10192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:36 smithi194 ceph-mon[33017]: pgmap v10192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:36 smithi096 ceph-mon[32753]: pgmap v10192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18216.972627 secs 2024-04-26T16:54:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18216.972627 secs 2024-04-26T16:54:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18216.972627 secs 2024-04-26T16:54:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18219.807507 secs 2024-04-26T16:54:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:39 smithi194 ceph-mon[33017]: pgmap v10193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18219.807507 secs 2024-04-26T16:54:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:39 smithi033 ceph-mon[27845]: pgmap v10193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18219.807507 secs 2024-04-26T16:54:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:39 smithi096 ceph-mon[32753]: pgmap v10193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:40 smithi194 ceph-mon[33017]: pgmap v10194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:40 smithi033 ceph-mon[27845]: pgmap v10194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:40 smithi096 ceph-mon[32753]: pgmap v10194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:42 smithi194 ceph-mon[33017]: pgmap v10195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:42 smithi033 ceph-mon[27845]: pgmap v10195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:42 smithi096 ceph-mon[32753]: pgmap v10195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18221.972686 secs 2024-04-26T16:54:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18224.807581 secs 2024-04-26T16:54:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18221.972686 secs 2024-04-26T16:54:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18224.807581 secs 2024-04-26T16:54:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18221.972686 secs 2024-04-26T16:54:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18224.807581 secs 2024-04-26T16:54:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:44 smithi194 ceph-mon[33017]: pgmap v10196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:44 smithi033 ceph-mon[27845]: pgmap v10196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:44 smithi096 ceph-mon[32753]: pgmap v10196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:46.210+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:46.210+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:54:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:46.210+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:54:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:46 smithi194 ceph-mon[33017]: pgmap v10197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:46 smithi033 ceph-mon[27845]: pgmap v10197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:46 smithi096 ceph-mon[32753]: pgmap v10197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18226.972744 secs 2024-04-26T16:54:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18226.972744 secs 2024-04-26T16:54:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18226.972744 secs 2024-04-26T16:54:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18229.807695 secs 2024-04-26T16:54:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:49 smithi194 ceph-mon[33017]: pgmap v10198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18229.807695 secs 2024-04-26T16:54:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:49 smithi033 ceph-mon[27845]: pgmap v10198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18229.807695 secs 2024-04-26T16:54:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:49 smithi096 ceph-mon[32753]: pgmap v10198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:50 smithi194 ceph-mon[33017]: pgmap v10199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:50 smithi033 ceph-mon[27845]: pgmap v10199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:50 smithi096 ceph-mon[32753]: pgmap v10199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:52 smithi194 ceph-mon[33017]: pgmap v10200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:52 smithi033 ceph-mon[27845]: pgmap v10200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:52 smithi096 ceph-mon[32753]: pgmap v10200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:54.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18231.972824 secs 2024-04-26T16:54:54.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18234.807833 secs 2024-04-26T16:54:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18231.972824 secs 2024-04-26T16:54:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18234.807833 secs 2024-04-26T16:54:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18231.972824 secs 2024-04-26T16:54:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18234.807833 secs 2024-04-26T16:54:54.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:54.892+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:54:54.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:54.892+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:54:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:54.893+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T16:54:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:54:54.893+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:54:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:54 smithi194 ceph-mon[33017]: pgmap v10201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:54 smithi033 ceph-mon[27845]: pgmap v10201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:54 smithi096 ceph-mon[32753]: pgmap v10201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:54:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:56 smithi194 ceph-mon[33017]: pgmap v10202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:56 smithi033 ceph-mon[27845]: pgmap v10202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:56 smithi096 ceph-mon[32753]: pgmap v10202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18236.972908 secs 2024-04-26T16:54:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18236.972908 secs 2024-04-26T16:54:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18236.972908 secs 2024-04-26T16:54:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18239.807946 secs 2024-04-26T16:54:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:54:59 smithi194 ceph-mon[33017]: pgmap v10203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18239.807946 secs 2024-04-26T16:54:59.946 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:54:59 smithi033 ceph-mon[27845]: pgmap v10203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:54:59.947 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18239.807946 secs 2024-04-26T16:54:59.947 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:54:59 smithi096 ceph-mon[32753]: pgmap v10203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:01.225+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:01.225+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:55:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:01.225+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:00 smithi194 ceph-mon[33017]: pgmap v10204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:00 smithi033 ceph-mon[27845]: pgmap v10204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:00 smithi096 ceph-mon[32753]: pgmap v10204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:02 smithi194 ceph-mon[33017]: pgmap v10205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:02 smithi033 ceph-mon[27845]: pgmap v10205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:02 smithi096 ceph-mon[32753]: pgmap v10205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18241.973008 secs 2024-04-26T16:55:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18244.808033 secs 2024-04-26T16:55:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18241.973008 secs 2024-04-26T16:55:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18244.808033 secs 2024-04-26T16:55:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18241.973008 secs 2024-04-26T16:55:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18244.808033 secs 2024-04-26T16:55:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:04 smithi194 ceph-mon[33017]: pgmap v10206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:04 smithi033 ceph-mon[27845]: pgmap v10206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:04 smithi096 ceph-mon[32753]: pgmap v10206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:07.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:06 smithi033 ceph-mon[27845]: pgmap v10207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:06 smithi194 ceph-mon[33017]: pgmap v10207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:06 smithi096 ceph-mon[32753]: pgmap v10207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18246.973070 secs 2024-04-26T16:55:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18246.973070 secs 2024-04-26T16:55:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18246.973070 secs 2024-04-26T16:55:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18249.808148 secs 2024-04-26T16:55:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:09 smithi194 ceph-mon[33017]: pgmap v10208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18249.808148 secs 2024-04-26T16:55:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:09 smithi033 ceph-mon[27845]: pgmap v10208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18249.808148 secs 2024-04-26T16:55:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:09 smithi096 ceph-mon[32753]: pgmap v10208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:10 smithi194 ceph-mon[33017]: pgmap v10209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:10 smithi033 ceph-mon[27845]: pgmap v10209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:10 smithi096 ceph-mon[32753]: pgmap v10209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:12 smithi194 ceph-mon[33017]: pgmap v10210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:12 smithi033 ceph-mon[27845]: pgmap v10210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:12 smithi096 ceph-mon[32753]: pgmap v10210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18251.973152 secs 2024-04-26T16:55:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18254.808308 secs 2024-04-26T16:55:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18251.973152 secs 2024-04-26T16:55:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18254.808308 secs 2024-04-26T16:55:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18251.973152 secs 2024-04-26T16:55:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18254.808308 secs 2024-04-26T16:55:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:14 smithi194 ceph-mon[33017]: pgmap v10211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:14 smithi033 ceph-mon[27845]: pgmap v10211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:14 smithi096 ceph-mon[32753]: pgmap v10211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:16.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:16.235+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:16.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:16.235+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:16.236 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:16.235+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:55:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:16 smithi194 ceph-mon[33017]: pgmap v10212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:16 smithi033 ceph-mon[27845]: pgmap v10212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:16 smithi096 ceph-mon[32753]: pgmap v10212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18256.973234 secs 2024-04-26T16:55:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18256.973234 secs 2024-04-26T16:55:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18256.973234 secs 2024-04-26T16:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18259.808448 secs 2024-04-26T16:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:19 smithi194 ceph-mon[33017]: pgmap v10213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:55:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18259.808448 secs 2024-04-26T16:55:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:19 smithi033 ceph-mon[27845]: pgmap v10213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:55:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18259.808448 secs 2024-04-26T16:55:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:19 smithi096 ceph-mon[32753]: pgmap v10213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:55:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:55:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:55:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:55:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:55:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:55:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:55:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:55:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:55:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:55:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:21 smithi194 ceph-mon[33017]: pgmap v10214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:55:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:21 smithi194 ceph-mon[33017]: pgmap v10215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:21 smithi033 ceph-mon[27845]: pgmap v10214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:55:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:21 smithi033 ceph-mon[27845]: pgmap v10215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:21 smithi096 ceph-mon[32753]: pgmap v10214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T16:55:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:21 smithi096 ceph-mon[32753]: pgmap v10215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:23 smithi194 ceph-mon[33017]: pgmap v10216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18261.973295 secs 2024-04-26T16:55:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:23 smithi033 ceph-mon[27845]: pgmap v10216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18261.973295 secs 2024-04-26T16:55:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:23 smithi096 ceph-mon[32753]: pgmap v10216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18261.973295 secs 2024-04-26T16:55:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18264.808567 secs 2024-04-26T16:55:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18264.808567 secs 2024-04-26T16:55:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18264.808567 secs 2024-04-26T16:55:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:25 smithi033 ceph-mon[27845]: pgmap v10217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:25 smithi096 ceph-mon[32753]: pgmap v10217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:25 smithi194 ceph-mon[33017]: pgmap v10217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:27 smithi033 ceph-mon[27845]: pgmap v10218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:27 smithi096 ceph-mon[32753]: pgmap v10218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:27 smithi194 ceph-mon[33017]: pgmap v10218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18266.973356 secs 2024-04-26T16:55:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18266.973356 secs 2024-04-26T16:55:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18266.973356 secs 2024-04-26T16:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18269.808711 secs 2024-04-26T16:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:29 smithi033 ceph-mon[27845]: pgmap v10219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18269.808711 secs 2024-04-26T16:55:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:29 smithi096 ceph-mon[32753]: pgmap v10219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18269.808711 secs 2024-04-26T16:55:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:29 smithi194 ceph-mon[33017]: pgmap v10219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:55:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:31.245+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:31.245+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:31.245+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:55:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:31 smithi033 ceph-mon[27845]: pgmap v10220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:55:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:31 smithi096 ceph-mon[32753]: pgmap v10220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:55:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:31 smithi194 ceph-mon[33017]: pgmap v10220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:33 smithi033 ceph-mon[27845]: pgmap v10221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18271.973416 secs 2024-04-26T16:55:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:33 smithi096 ceph-mon[32753]: pgmap v10221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18271.973416 secs 2024-04-26T16:55:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:33 smithi194 ceph-mon[33017]: pgmap v10221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18271.973416 secs 2024-04-26T16:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18274.808829 secs 2024-04-26T16:55:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18274.808829 secs 2024-04-26T16:55:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18274.808829 secs 2024-04-26T16:55:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:35 smithi033 ceph-mon[27845]: pgmap v10222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:35 smithi096 ceph-mon[32753]: pgmap v10222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:35 smithi194 ceph-mon[33017]: pgmap v10222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:37 smithi033 ceph-mon[27845]: pgmap v10223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:37 smithi096 ceph-mon[32753]: pgmap v10223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:37 smithi194 ceph-mon[33017]: pgmap v10223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18276.973475 secs 2024-04-26T16:55:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18276.973475 secs 2024-04-26T16:55:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18276.973475 secs 2024-04-26T16:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18279.808892 secs 2024-04-26T16:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:39 smithi033 ceph-mon[27845]: pgmap v10224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18279.808892 secs 2024-04-26T16:55:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:39 smithi096 ceph-mon[32753]: pgmap v10224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18279.808892 secs 2024-04-26T16:55:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:39 smithi194 ceph-mon[33017]: pgmap v10224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:41 smithi033 ceph-mon[27845]: pgmap v10225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:41 smithi096 ceph-mon[32753]: pgmap v10225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:41 smithi194 ceph-mon[33017]: pgmap v10225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:43 smithi033 ceph-mon[27845]: pgmap v10226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18281.973556 secs 2024-04-26T16:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:43 smithi096 ceph-mon[32753]: pgmap v10226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18281.973556 secs 2024-04-26T16:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:43 smithi194 ceph-mon[33017]: pgmap v10226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18281.973556 secs 2024-04-26T16:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18284.809010 secs 2024-04-26T16:55:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18284.809010 secs 2024-04-26T16:55:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18284.809010 secs 2024-04-26T16:55:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:45 smithi194 ceph-mon[33017]: pgmap v10227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:45 smithi033 ceph-mon[27845]: pgmap v10227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:45 smithi096 ceph-mon[32753]: pgmap v10227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:46.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:46.259+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:46.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:46.259+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:55:46.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:55:46.259+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:55:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:47 smithi194 ceph-mon[33017]: pgmap v10228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:47 smithi033 ceph-mon[27845]: pgmap v10228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:47 smithi096 ceph-mon[32753]: pgmap v10228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18286.973640 secs 2024-04-26T16:55:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18286.973640 secs 2024-04-26T16:55:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18286.973640 secs 2024-04-26T16:55:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18289.809135 secs 2024-04-26T16:55:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:49 smithi194 ceph-mon[33017]: pgmap v10229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18289.809135 secs 2024-04-26T16:55:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:49 smithi033 ceph-mon[27845]: pgmap v10229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18289.809135 secs 2024-04-26T16:55:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:49 smithi096 ceph-mon[32753]: pgmap v10229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:51 smithi194 ceph-mon[33017]: pgmap v10230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:51 smithi033 ceph-mon[27845]: pgmap v10230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:51 smithi096 ceph-mon[32753]: pgmap v10230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:53 smithi194 ceph-mon[33017]: pgmap v10231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18291.973722 secs 2024-04-26T16:55:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:53 smithi033 ceph-mon[27845]: pgmap v10231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18291.973722 secs 2024-04-26T16:55:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:53 smithi096 ceph-mon[32753]: pgmap v10231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18291.973722 secs 2024-04-26T16:55:55.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18294.809257 secs 2024-04-26T16:55:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18294.809257 secs 2024-04-26T16:55:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18294.809257 secs 2024-04-26T16:55:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:55 smithi194 ceph-mon[33017]: pgmap v10232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:55 smithi033 ceph-mon[27845]: pgmap v10232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:55 smithi096 ceph-mon[32753]: pgmap v10232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:55:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:57 smithi194 ceph-mon[33017]: pgmap v10233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:57 smithi033 ceph-mon[27845]: pgmap v10233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:57 smithi096 ceph-mon[32753]: pgmap v10233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:55:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18296.973804 secs 2024-04-26T16:55:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18296.973804 secs 2024-04-26T16:55:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18296.973804 secs 2024-04-26T16:56:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18299.809355 secs 2024-04-26T16:56:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:55:59 smithi194 ceph-mon[33017]: pgmap v10234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18299.809355 secs 2024-04-26T16:56:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:55:59 smithi033 ceph-mon[27845]: pgmap v10234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18299.809355 secs 2024-04-26T16:56:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:55:59 smithi096 ceph-mon[32753]: pgmap v10234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:01.266+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:01.266+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:01.266+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:56:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:01 smithi194 ceph-mon[33017]: pgmap v10235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:01 smithi033 ceph-mon[27845]: pgmap v10235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:01 smithi096 ceph-mon[32753]: pgmap v10235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:03 smithi194 ceph-mon[33017]: pgmap v10236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18301.973884 secs 2024-04-26T16:56:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:03 smithi033 ceph-mon[27845]: pgmap v10236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18301.973884 secs 2024-04-26T16:56:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:03 smithi096 ceph-mon[32753]: pgmap v10236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18301.973884 secs 2024-04-26T16:56:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18304.809479 secs 2024-04-26T16:56:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18304.809479 secs 2024-04-26T16:56:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18304.809479 secs 2024-04-26T16:56:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:05 smithi194 ceph-mon[33017]: pgmap v10237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:05 smithi033 ceph-mon[27845]: pgmap v10237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:05 smithi096 ceph-mon[32753]: pgmap v10237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:07 smithi194 ceph-mon[33017]: pgmap v10238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:07 smithi033 ceph-mon[27845]: pgmap v10238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:07 smithi096 ceph-mon[32753]: pgmap v10238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18306.973988 secs 2024-04-26T16:56:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18306.973988 secs 2024-04-26T16:56:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18306.973988 secs 2024-04-26T16:56:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18309.809595 secs 2024-04-26T16:56:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:09 smithi194 ceph-mon[33017]: pgmap v10239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18309.809595 secs 2024-04-26T16:56:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:09 smithi033 ceph-mon[27845]: pgmap v10239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18309.809595 secs 2024-04-26T16:56:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:09 smithi096 ceph-mon[32753]: pgmap v10239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:11 smithi194 ceph-mon[33017]: pgmap v10240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:11 smithi033 ceph-mon[27845]: pgmap v10240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:11 smithi096 ceph-mon[32753]: pgmap v10240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:13 smithi194 ceph-mon[33017]: pgmap v10241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18311.974071 secs 2024-04-26T16:56:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:13 smithi033 ceph-mon[27845]: pgmap v10241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18311.974071 secs 2024-04-26T16:56:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:13 smithi096 ceph-mon[32753]: pgmap v10241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18311.974071 secs 2024-04-26T16:56:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18314.809711 secs 2024-04-26T16:56:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18314.809711 secs 2024-04-26T16:56:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18314.809711 secs 2024-04-26T16:56:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:15 smithi194 ceph-mon[33017]: pgmap v10242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:15 smithi033 ceph-mon[27845]: pgmap v10242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:15 smithi096 ceph-mon[32753]: pgmap v10242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:16.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:16.272+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:16.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:16.272+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:56:16.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:16.272+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:17 smithi194 ceph-mon[33017]: pgmap v10243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:17 smithi033 ceph-mon[27845]: pgmap v10243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:17 smithi096 ceph-mon[32753]: pgmap v10243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18316.974154 secs 2024-04-26T16:56:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18316.974154 secs 2024-04-26T16:56:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18316.974154 secs 2024-04-26T16:56:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18319.809828 secs 2024-04-26T16:56:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:19 smithi194 ceph-mon[33017]: pgmap v10244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18319.809828 secs 2024-04-26T16:56:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:19 smithi033 ceph-mon[27845]: pgmap v10244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18319.809828 secs 2024-04-26T16:56:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:19 smithi096 ceph-mon[32753]: pgmap v10244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:56:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:56:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:56:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T16:56:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T16:56:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T16:56:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062070 comp rx=0 tx=0).stop 2024-04-26T16:56:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T16:56:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:56:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:56:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.762+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 0x7feee40b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:56:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.763+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 0x7feee40b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:56:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:21.764+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 0x7feee40b3710 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0022070 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:56:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:21 smithi194 ceph-mon[33017]: pgmap v10245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:21 smithi033 ceph-mon[27845]: pgmap v10245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:21 smithi096 ceph-mon[32753]: pgmap v10245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:56:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:56:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:56:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:56:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:56:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:56:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:56:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:56:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:56:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:23 smithi033 ceph-mon[27845]: pgmap v10246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:23 smithi033 ceph-mon[27845]: pgmap v10247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18321.974235 secs 2024-04-26T16:56:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18324.809949 secs 2024-04-26T16:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:23 smithi096 ceph-mon[32753]: pgmap v10246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:23 smithi096 ceph-mon[32753]: pgmap v10247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18321.974235 secs 2024-04-26T16:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18324.809949 secs 2024-04-26T16:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:23 smithi194 ceph-mon[33017]: pgmap v10246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:23 smithi194 ceph-mon[33017]: pgmap v10247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18321.974235 secs 2024-04-26T16:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18324.809949 secs 2024-04-26T16:56:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:24.908+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:56:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:24.908+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:56:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:24.908+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T16:56:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:24.908+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:56:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:25 smithi033 ceph-mon[27845]: pgmap v10248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:56:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:25 smithi096 ceph-mon[32753]: pgmap v10248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:56:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:25 smithi194 ceph-mon[33017]: pgmap v10248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T16:56:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:27 smithi033 ceph-mon[27845]: pgmap v10249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:27 smithi096 ceph-mon[32753]: pgmap v10249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:27 smithi194 ceph-mon[33017]: pgmap v10249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18326.974296 secs 2024-04-26T16:56:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18329.810069 secs 2024-04-26T16:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18326.974296 secs 2024-04-26T16:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18329.810069 secs 2024-04-26T16:56:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18326.974296 secs 2024-04-26T16:56:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18329.810069 secs 2024-04-26T16:56:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:29 smithi033 ceph-mon[27845]: pgmap v10250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:29 smithi096 ceph-mon[32753]: pgmap v10250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:29 smithi194 ceph-mon[33017]: pgmap v10250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:31.280+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:31.280+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:31.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:31.280+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:56:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:31 smithi033 ceph-mon[27845]: pgmap v10251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:31 smithi096 ceph-mon[32753]: pgmap v10251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:31 smithi194 ceph-mon[33017]: pgmap v10251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:56:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:33 smithi033 ceph-mon[27845]: pgmap v10252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:56:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18331.974354 secs 2024-04-26T16:56:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18334.810225 secs 2024-04-26T16:56:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:33 smithi096 ceph-mon[32753]: pgmap v10252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:56:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18331.974354 secs 2024-04-26T16:56:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18334.810225 secs 2024-04-26T16:56:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:33 smithi194 ceph-mon[33017]: pgmap v10252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:56:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18331.974354 secs 2024-04-26T16:56:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18334.810225 secs 2024-04-26T16:56:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:35 smithi033 ceph-mon[27845]: pgmap v10253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:35 smithi096 ceph-mon[32753]: pgmap v10253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:35 smithi194 ceph-mon[33017]: pgmap v10253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:37 smithi033 ceph-mon[27845]: pgmap v10254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:37 smithi096 ceph-mon[32753]: pgmap v10254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:37 smithi194 ceph-mon[33017]: pgmap v10254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18336.974412 secs 2024-04-26T16:56:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18339.810348 secs 2024-04-26T16:56:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18336.974412 secs 2024-04-26T16:56:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18339.810348 secs 2024-04-26T16:56:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18336.974412 secs 2024-04-26T16:56:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18339.810348 secs 2024-04-26T16:56:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:39 smithi033 ceph-mon[27845]: pgmap v10255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:39 smithi096 ceph-mon[32753]: pgmap v10255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:39 smithi194 ceph-mon[33017]: pgmap v10255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:41 smithi033 ceph-mon[27845]: pgmap v10256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:41 smithi096 ceph-mon[32753]: pgmap v10256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:41 smithi194 ceph-mon[33017]: pgmap v10256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:43 smithi096 ceph-mon[32753]: pgmap v10257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18341.974494 secs 2024-04-26T16:56:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18344.810468 secs 2024-04-26T16:56:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:43 smithi194 ceph-mon[33017]: pgmap v10257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18341.974494 secs 2024-04-26T16:56:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18344.810468 secs 2024-04-26T16:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:43 smithi033 ceph-mon[27845]: pgmap v10257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18341.974494 secs 2024-04-26T16:56:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18344.810468 secs 2024-04-26T16:56:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:46.296+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:46.296+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:56:46.297 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:56:46.296+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:56:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:45 smithi194 ceph-mon[33017]: pgmap v10258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:45 smithi033 ceph-mon[27845]: pgmap v10258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:45 smithi096 ceph-mon[32753]: pgmap v10258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:47 smithi194 ceph-mon[33017]: pgmap v10259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:47 smithi033 ceph-mon[27845]: pgmap v10259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:47 smithi096 ceph-mon[32753]: pgmap v10259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18346.974578 secs 2024-04-26T16:56:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18349.810588 secs 2024-04-26T16:56:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18346.974578 secs 2024-04-26T16:56:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18349.810588 secs 2024-04-26T16:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18346.974578 secs 2024-04-26T16:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18349.810588 secs 2024-04-26T16:56:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:49 smithi194 ceph-mon[33017]: pgmap v10260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:49 smithi033 ceph-mon[27845]: pgmap v10260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:49 smithi096 ceph-mon[32753]: pgmap v10260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:51.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:50 smithi194 ceph-mon[33017]: pgmap v10261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:50 smithi033 ceph-mon[27845]: pgmap v10261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:50 smithi096 ceph-mon[32753]: pgmap v10261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:53 smithi194 ceph-mon[33017]: pgmap v10262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:53 smithi033 ceph-mon[27845]: pgmap v10262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:53 smithi096 ceph-mon[32753]: pgmap v10262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18351.974638 secs 2024-04-26T16:56:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18354.810707 secs 2024-04-26T16:56:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18351.974638 secs 2024-04-26T16:56:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18354.810707 secs 2024-04-26T16:56:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18351.974638 secs 2024-04-26T16:56:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18354.810707 secs 2024-04-26T16:56:55.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:55 smithi096 ceph-mon[32753]: pgmap v10263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:55 smithi194 ceph-mon[33017]: pgmap v10263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:55 smithi033 ceph-mon[27845]: pgmap v10263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:57 smithi194 ceph-mon[33017]: pgmap v10264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:57 smithi033 ceph-mon[27845]: pgmap v10264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:57 smithi096 ceph-mon[32753]: pgmap v10264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:56:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18356.974723 secs 2024-04-26T16:56:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:59 smithi194 ceph-mon[33017]: pgmap v10265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:56:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18359.810826 secs 2024-04-26T16:56:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18356.974723 secs 2024-04-26T16:56:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:59 smithi033 ceph-mon[27845]: pgmap v10265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:56:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18359.810826 secs 2024-04-26T16:56:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18356.974723 secs 2024-04-26T16:56:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:59 smithi096 ceph-mon[32753]: pgmap v10265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:56:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:56:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18359.810826 secs 2024-04-26T16:57:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:01.311+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:01.311+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:01.311+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:57:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:01 smithi194 ceph-mon[33017]: pgmap v10266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:01 smithi033 ceph-mon[27845]: pgmap v10266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:01 smithi096 ceph-mon[32753]: pgmap v10266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:03 smithi194 ceph-mon[33017]: pgmap v10267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:03 smithi033 ceph-mon[27845]: pgmap v10267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:03 smithi096 ceph-mon[32753]: pgmap v10267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18361.974810 secs 2024-04-26T16:57:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18364.810947 secs 2024-04-26T16:57:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18361.974810 secs 2024-04-26T16:57:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18364.810947 secs 2024-04-26T16:57:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18361.974810 secs 2024-04-26T16:57:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18364.810947 secs 2024-04-26T16:57:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:05 smithi194 ceph-mon[33017]: pgmap v10268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:05 smithi033 ceph-mon[27845]: pgmap v10268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:05 smithi096 ceph-mon[32753]: pgmap v10268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:07 smithi194 ceph-mon[33017]: pgmap v10269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:07 smithi033 ceph-mon[27845]: pgmap v10269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:07 smithi096 ceph-mon[32753]: pgmap v10269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18366.974893 secs 2024-04-26T16:57:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:09 smithi194 ceph-mon[33017]: pgmap v10270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18369.811063 secs 2024-04-26T16:57:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18366.974893 secs 2024-04-26T16:57:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:09 smithi033 ceph-mon[27845]: pgmap v10270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18369.811063 secs 2024-04-26T16:57:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18366.974893 secs 2024-04-26T16:57:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:09 smithi096 ceph-mon[32753]: pgmap v10270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18369.811063 secs 2024-04-26T16:57:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:11 smithi194 ceph-mon[33017]: pgmap v10271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:11 smithi033 ceph-mon[27845]: pgmap v10271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:11 smithi096 ceph-mon[32753]: pgmap v10271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:13 smithi194 ceph-mon[33017]: pgmap v10272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:13 smithi033 ceph-mon[27845]: pgmap v10272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:13 smithi096 ceph-mon[32753]: pgmap v10272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18371.974954 secs 2024-04-26T16:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18374.811193 secs 2024-04-26T16:57:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18371.974954 secs 2024-04-26T16:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18374.811193 secs 2024-04-26T16:57:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18371.974954 secs 2024-04-26T16:57:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18374.811193 secs 2024-04-26T16:57:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:15 smithi194 ceph-mon[33017]: pgmap v10273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:15 smithi033 ceph-mon[27845]: pgmap v10273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:15 smithi096 ceph-mon[32753]: pgmap v10273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:16.326+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:16.326+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:16.326+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:57:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:17 smithi194 ceph-mon[33017]: pgmap v10274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:17 smithi033 ceph-mon[27845]: pgmap v10274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:17 smithi096 ceph-mon[32753]: pgmap v10274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18376.975015 secs 2024-04-26T16:57:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:19 smithi194 ceph-mon[33017]: pgmap v10275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18379.811310 secs 2024-04-26T16:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18376.975015 secs 2024-04-26T16:57:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:19 smithi033 ceph-mon[27845]: pgmap v10275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18379.811310 secs 2024-04-26T16:57:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18376.975015 secs 2024-04-26T16:57:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:19 smithi096 ceph-mon[32753]: pgmap v10275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18379.811310 secs 2024-04-26T16:57:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:21 smithi194 ceph-mon[33017]: pgmap v10276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:21 smithi033 ceph-mon[27845]: pgmap v10276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:21 smithi096 ceph-mon[32753]: pgmap v10276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:23 smithi194 ceph-mon[33017]: pgmap v10277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:57:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:23 smithi033 ceph-mon[27845]: pgmap v10277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:57:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:23 smithi096 ceph-mon[32753]: pgmap v10277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:57:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18381.975084 secs 2024-04-26T16:57:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18384.811422 secs 2024-04-26T16:57:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18381.975084 secs 2024-04-26T16:57:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18384.811422 secs 2024-04-26T16:57:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18381.975084 secs 2024-04-26T16:57:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18384.811422 secs 2024-04-26T16:57:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:25 smithi033 ceph-mon[27845]: pgmap v10278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:25 smithi096 ceph-mon[32753]: pgmap v10278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:25.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:25 smithi194 ceph-mon[33017]: pgmap v10278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:27.792 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:27 smithi096 ceph-mon[32753]: pgmap v10279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:57:27.794 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:27 smithi194 ceph-mon[33017]: pgmap v10279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:57:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:27 smithi033 ceph-mon[27845]: pgmap v10279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T16:57:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18386.975165 secs 2024-04-26T16:57:29.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18386.975165 secs 2024-04-26T16:57:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18386.975165 secs 2024-04-26T16:57:29.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:29 smithi194 ceph-mon[33017]: pgmap v10280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:57:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:29 smithi033 ceph-mon[27845]: pgmap v10280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:57:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18389.811540 secs 2024-04-26T16:57:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18389.811540 secs 2024-04-26T16:57:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:29 smithi096 ceph-mon[32753]: pgmap v10280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T16:57:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18389.811540 secs 2024-04-26T16:57:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:57:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:57:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:30 smithi194 ceph-mon[33017]: pgmap v10281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T16:57:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:30 smithi194 ceph-mon[33017]: pgmap v10282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:30 smithi033 ceph-mon[27845]: pgmap v10281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T16:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:30 smithi033 ceph-mon[27845]: pgmap v10282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:57:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:57:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:30 smithi096 ceph-mon[32753]: pgmap v10281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T16:57:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:30 smithi096 ceph-mon[32753]: pgmap v10282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:31.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:57:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:31.331+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:31.331+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:57:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:31.331+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:32 smithi096 ceph-mon[32753]: pgmap v10283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:32 smithi194 ceph-mon[33017]: pgmap v10283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:32 smithi033 ceph-mon[27845]: pgmap v10283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18391.975227 secs 2024-04-26T16:57:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18394.811662 secs 2024-04-26T16:57:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18391.975227 secs 2024-04-26T16:57:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18394.811662 secs 2024-04-26T16:57:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18391.975227 secs 2024-04-26T16:57:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18394.811662 secs 2024-04-26T16:57:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:34 smithi194 ceph-mon[33017]: pgmap v10284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:34 smithi033 ceph-mon[27845]: pgmap v10284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:34 smithi096 ceph-mon[32753]: pgmap v10284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T16:57:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:36 smithi194 ceph-mon[33017]: pgmap v10285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:57:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:36 smithi096 ceph-mon[32753]: pgmap v10285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:57:37.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:36 smithi033 ceph-mon[27845]: pgmap v10285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:57:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18396.975287 secs 2024-04-26T16:57:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:38 smithi194 ceph-mon[33017]: pgmap v10286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:57:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18399.811815 secs 2024-04-26T16:57:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18396.975287 secs 2024-04-26T16:57:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:38 smithi033 ceph-mon[27845]: pgmap v10286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:57:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18399.811815 secs 2024-04-26T16:57:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18396.975287 secs 2024-04-26T16:57:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:38 smithi096 ceph-mon[32753]: pgmap v10286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:57:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18399.811815 secs 2024-04-26T16:57:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:40 smithi033 ceph-mon[27845]: pgmap v10287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:57:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:40 smithi194 ceph-mon[33017]: pgmap v10287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:57:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:40 smithi096 ceph-mon[32753]: pgmap v10287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:57:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:42 smithi194 ceph-mon[33017]: pgmap v10288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:42 smithi033 ceph-mon[27845]: pgmap v10288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:42 smithi096 ceph-mon[32753]: pgmap v10288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18401.975348 secs 2024-04-26T16:57:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18404.811935 secs 2024-04-26T16:57:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18401.975348 secs 2024-04-26T16:57:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18404.811935 secs 2024-04-26T16:57:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18401.975348 secs 2024-04-26T16:57:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18404.811935 secs 2024-04-26T16:57:45.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:44 smithi033 ceph-mon[27845]: pgmap v10289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:44 smithi194 ceph-mon[33017]: pgmap v10289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:44 smithi096 ceph-mon[32753]: pgmap v10289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:46.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:46.337+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:46.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:46.337+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:57:46.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:57:46.337+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:57:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:46 smithi194 ceph-mon[33017]: pgmap v10290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:46 smithi033 ceph-mon[27845]: pgmap v10290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:46 smithi096 ceph-mon[32753]: pgmap v10290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:49.351 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18406.975436 secs 2024-04-26T16:57:49.352 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:48 smithi033 ceph-mon[27845]: pgmap v10291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:49.352 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18409.811998 secs 2024-04-26T16:57:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18406.975436 secs 2024-04-26T16:57:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:48 smithi194 ceph-mon[33017]: pgmap v10291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18409.811998 secs 2024-04-26T16:57:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18406.975436 secs 2024-04-26T16:57:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:48 smithi096 ceph-mon[32753]: pgmap v10291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18409.811998 secs 2024-04-26T16:57:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:50 smithi194 ceph-mon[33017]: pgmap v10292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:50 smithi033 ceph-mon[27845]: pgmap v10292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:50 smithi096 ceph-mon[32753]: pgmap v10292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:53.327 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:52 smithi194 ceph-mon[33017]: pgmap v10293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:52 smithi033 ceph-mon[27845]: pgmap v10293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:52 smithi096 ceph-mon[32753]: pgmap v10293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:54.251 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18411.975509 secs 2024-04-26T16:57:54.251 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18414.812089 secs 2024-04-26T16:57:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18411.975509 secs 2024-04-26T16:57:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18414.812089 secs 2024-04-26T16:57:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18411.975509 secs 2024-04-26T16:57:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18414.812089 secs 2024-04-26T16:57:55.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:54 smithi096 ceph-mon[32753]: pgmap v10294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:54 smithi194 ceph-mon[33017]: pgmap v10294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:54 smithi033 ceph-mon[27845]: pgmap v10294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:56 smithi194 ceph-mon[33017]: pgmap v10295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:56 smithi033 ceph-mon[27845]: pgmap v10295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:56 smithi096 ceph-mon[32753]: pgmap v10295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:57:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:57 smithi194 ceph-mon[33017]: pgmap v10296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:57 smithi033 ceph-mon[27845]: pgmap v10296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:57 smithi096 ceph-mon[32753]: pgmap v10296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:57:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18416.975554 secs 2024-04-26T16:57:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18419.812206 secs 2024-04-26T16:57:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18416.975554 secs 2024-04-26T16:57:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18419.812206 secs 2024-04-26T16:57:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18416.975554 secs 2024-04-26T16:57:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18419.812206 secs 2024-04-26T16:58:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:57:59 smithi033 ceph-mon[27845]: pgmap v10297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:57:59 smithi194 ceph-mon[33017]: pgmap v10297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:57:59 smithi096 ceph-mon[32753]: pgmap v10297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:01.351+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:01.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:01.351+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:01.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:01.351+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:58:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:02 smithi194 ceph-mon[33017]: pgmap v10298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:02 smithi033 ceph-mon[27845]: pgmap v10298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:02 smithi096 ceph-mon[32753]: pgmap v10298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18421.975641 secs 2024-04-26T16:58:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18424.812289 secs 2024-04-26T16:58:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18421.975641 secs 2024-04-26T16:58:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18424.812289 secs 2024-04-26T16:58:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18421.975641 secs 2024-04-26T16:58:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18424.812289 secs 2024-04-26T16:58:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:04 smithi194 ceph-mon[33017]: pgmap v10299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:04 smithi033 ceph-mon[27845]: pgmap v10299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:04 smithi096 ceph-mon[32753]: pgmap v10299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:05 smithi194 ceph-mon[33017]: pgmap v10300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:05 smithi033 ceph-mon[27845]: pgmap v10300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:05 smithi096 ceph-mon[32753]: pgmap v10300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18426.975701 secs 2024-04-26T16:58:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:08 smithi194 ceph-mon[33017]: pgmap v10301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18429.812408 secs 2024-04-26T16:58:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18426.975701 secs 2024-04-26T16:58:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:08 smithi033 ceph-mon[27845]: pgmap v10301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18429.812408 secs 2024-04-26T16:58:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18426.975701 secs 2024-04-26T16:58:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:08 smithi096 ceph-mon[32753]: pgmap v10301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18429.812408 secs 2024-04-26T16:58:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:10 smithi194 ceph-mon[33017]: pgmap v10302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:10 smithi033 ceph-mon[27845]: pgmap v10302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:10 smithi096 ceph-mon[32753]: pgmap v10302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:12 smithi194 ceph-mon[33017]: pgmap v10303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:12 smithi033 ceph-mon[27845]: pgmap v10303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:12 smithi096 ceph-mon[32753]: pgmap v10303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18431.975764 secs 2024-04-26T16:58:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18434.812491 secs 2024-04-26T16:58:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18431.975764 secs 2024-04-26T16:58:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18434.812491 secs 2024-04-26T16:58:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18431.975764 secs 2024-04-26T16:58:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18434.812491 secs 2024-04-26T16:58:15.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:14 smithi033 ceph-mon[27845]: pgmap v10304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:14 smithi194 ceph-mon[33017]: pgmap v10304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:14 smithi096 ceph-mon[32753]: pgmap v10304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:15 smithi194 ceph-mon[33017]: pgmap v10305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:16.354+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:16.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:16.354+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:16.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:16.354+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:58:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:15 smithi033 ceph-mon[27845]: pgmap v10305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:15 smithi096 ceph-mon[32753]: pgmap v10305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18436.975824 secs 2024-04-26T16:58:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:18 smithi194 ceph-mon[33017]: pgmap v10306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18439.812629 secs 2024-04-26T16:58:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18436.975824 secs 2024-04-26T16:58:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:18 smithi033 ceph-mon[27845]: pgmap v10306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18439.812629 secs 2024-04-26T16:58:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18436.975824 secs 2024-04-26T16:58:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:18 smithi096 ceph-mon[32753]: pgmap v10306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18439.812629 secs 2024-04-26T16:58:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:20 smithi194 ceph-mon[33017]: pgmap v10307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:20 smithi033 ceph-mon[27845]: pgmap v10307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:20 smithi096 ceph-mon[32753]: pgmap v10307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:22 smithi194 ceph-mon[33017]: pgmap v10308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:22 smithi033 ceph-mon[27845]: pgmap v10308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:22 smithi096 ceph-mon[32753]: pgmap v10308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18441.975876 secs 2024-04-26T16:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18444.812766 secs 2024-04-26T16:58:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18441.975876 secs 2024-04-26T16:58:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18444.812766 secs 2024-04-26T16:58:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18441.975876 secs 2024-04-26T16:58:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18444.812766 secs 2024-04-26T16:58:25.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:24 smithi096 ceph-mon[32753]: pgmap v10309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:24 smithi194 ceph-mon[33017]: pgmap v10309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:24 smithi033 ceph-mon[27845]: pgmap v10309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:25 smithi194 ceph-mon[33017]: pgmap v10310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:25 smithi033 ceph-mon[27845]: pgmap v10310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:25 smithi096 ceph-mon[32753]: pgmap v10310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18446.975934 secs 2024-04-26T16:58:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:28 smithi194 ceph-mon[33017]: pgmap v10311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18449.812876 secs 2024-04-26T16:58:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18446.975934 secs 2024-04-26T16:58:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:28 smithi033 ceph-mon[27845]: pgmap v10311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18449.812876 secs 2024-04-26T16:58:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18446.975934 secs 2024-04-26T16:58:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:28 smithi096 ceph-mon[32753]: pgmap v10311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18449.812876 secs 2024-04-26T16:58:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:30 smithi194 ceph-mon[33017]: pgmap v10312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:58:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:31.370+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:31.370+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:31.370+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:58:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:30 smithi033 ceph-mon[27845]: pgmap v10312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:58:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:30 smithi096 ceph-mon[32753]: pgmap v10312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:58:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:58:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:58:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:58:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:58:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:58:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:58:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:58:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:58:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:58:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:32 smithi194 ceph-mon[33017]: pgmap v10313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:32 smithi194 ceph-mon[33017]: pgmap v10314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:32 smithi033 ceph-mon[27845]: pgmap v10313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:32 smithi033 ceph-mon[27845]: pgmap v10314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:32 smithi096 ceph-mon[32753]: pgmap v10313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:32 smithi096 ceph-mon[32753]: pgmap v10314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18451.975990 secs 2024-04-26T16:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18454.813033 secs 2024-04-26T16:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:33 smithi194 ceph-mon[33017]: pgmap v10315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18451.975990 secs 2024-04-26T16:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18454.813033 secs 2024-04-26T16:58:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:33 smithi033 ceph-mon[27845]: pgmap v10315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18451.975990 secs 2024-04-26T16:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18454.813033 secs 2024-04-26T16:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:33 smithi096 ceph-mon[32753]: pgmap v10315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:36 smithi033 ceph-mon[27845]: pgmap v10316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:36 smithi096 ceph-mon[32753]: pgmap v10316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:36 smithi194 ceph-mon[33017]: pgmap v10316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:38 smithi033 ceph-mon[27845]: pgmap v10317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18456.976053 secs 2024-04-26T16:58:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:38 smithi096 ceph-mon[32753]: pgmap v10317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18456.976053 secs 2024-04-26T16:58:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:38 smithi194 ceph-mon[33017]: pgmap v10317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18456.976053 secs 2024-04-26T16:58:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18459.813150 secs 2024-04-26T16:58:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18459.813150 secs 2024-04-26T16:58:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18459.813150 secs 2024-04-26T16:58:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:40 smithi033 ceph-mon[27845]: pgmap v10318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:40 smithi096 ceph-mon[32753]: pgmap v10318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:40 smithi194 ceph-mon[33017]: pgmap v10318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:58:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:42 smithi033 ceph-mon[27845]: pgmap v10319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:58:42.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:42 smithi096 ceph-mon[32753]: pgmap v10319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:58:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:42 smithi194 ceph-mon[33017]: pgmap v10319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:58:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18461.976113 secs 2024-04-26T16:58:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18461.976113 secs 2024-04-26T16:58:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18461.976113 secs 2024-04-26T16:58:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18464.813254 secs 2024-04-26T16:58:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:44 smithi033 ceph-mon[27845]: pgmap v10320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18464.813254 secs 2024-04-26T16:58:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:44 smithi096 ceph-mon[32753]: pgmap v10320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18464.813254 secs 2024-04-26T16:58:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:44 smithi194 ceph-mon[33017]: pgmap v10320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:46.378+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:46.378+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:58:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:58:46.378+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:58:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:46 smithi033 ceph-mon[27845]: pgmap v10321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:46 smithi096 ceph-mon[32753]: pgmap v10321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:46 smithi194 ceph-mon[33017]: pgmap v10321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:48 smithi033 ceph-mon[27845]: pgmap v10322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18466.976175 secs 2024-04-26T16:58:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:48 smithi096 ceph-mon[32753]: pgmap v10322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18466.976175 secs 2024-04-26T16:58:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:48 smithi194 ceph-mon[33017]: pgmap v10322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18466.976175 secs 2024-04-26T16:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18469.813407 secs 2024-04-26T16:58:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18469.813407 secs 2024-04-26T16:58:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18469.813407 secs 2024-04-26T16:58:50.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:50 smithi096 ceph-mon[32753]: pgmap v10323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:50.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:50 smithi033 ceph-mon[27845]: pgmap v10323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:50 smithi194 ceph-mon[33017]: pgmap v10323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:52 smithi194 ceph-mon[33017]: pgmap v10324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:52 smithi033 ceph-mon[27845]: pgmap v10324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:52 smithi096 ceph-mon[32753]: pgmap v10324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:54.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18471.976234 secs 2024-04-26T16:58:54.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18471.976234 secs 2024-04-26T16:58:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18471.976234 secs 2024-04-26T16:58:55.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18474.813484 secs 2024-04-26T16:58:55.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:54 smithi096 ceph-mon[32753]: pgmap v10325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18474.813484 secs 2024-04-26T16:58:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:54 smithi194 ceph-mon[33017]: pgmap v10325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18474.813484 secs 2024-04-26T16:58:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:54 smithi033 ceph-mon[27845]: pgmap v10325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:56 smithi194 ceph-mon[33017]: pgmap v10326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:56 smithi033 ceph-mon[27845]: pgmap v10326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:56 smithi096 ceph-mon[32753]: pgmap v10326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:58 smithi194 ceph-mon[33017]: pgmap v10327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18476.976296 secs 2024-04-26T16:58:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:58 smithi033 ceph-mon[27845]: pgmap v10327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18476.976296 secs 2024-04-26T16:58:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:58 smithi096 ceph-mon[32753]: pgmap v10327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:58:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18476.976296 secs 2024-04-26T16:59:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18479.813579 secs 2024-04-26T16:59:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18479.813579 secs 2024-04-26T16:59:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18479.813579 secs 2024-04-26T16:59:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:00 smithi194 ceph-mon[33017]: pgmap v10328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:00 smithi033 ceph-mon[27845]: pgmap v10328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:00 smithi096 ceph-mon[32753]: pgmap v10328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:01.394+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:01.394+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:59:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:01.394+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:02 smithi194 ceph-mon[33017]: pgmap v10329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:02 smithi033 ceph-mon[27845]: pgmap v10329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:02 smithi096 ceph-mon[32753]: pgmap v10329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18481.976356 secs 2024-04-26T16:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18481.976356 secs 2024-04-26T16:59:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18481.976356 secs 2024-04-26T16:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18484.813716 secs 2024-04-26T16:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:04 smithi194 ceph-mon[33017]: pgmap v10330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18484.813716 secs 2024-04-26T16:59:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:04 smithi033 ceph-mon[27845]: pgmap v10330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18484.813716 secs 2024-04-26T16:59:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:04 smithi096 ceph-mon[32753]: pgmap v10330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:06 smithi194 ceph-mon[33017]: pgmap v10331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:06 smithi033 ceph-mon[27845]: pgmap v10331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:06 smithi096 ceph-mon[32753]: pgmap v10331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:08 smithi194 ceph-mon[33017]: pgmap v10332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18486.976419 secs 2024-04-26T16:59:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:08 smithi033 ceph-mon[27845]: pgmap v10332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18486.976419 secs 2024-04-26T16:59:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:08 smithi096 ceph-mon[32753]: pgmap v10332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18486.976419 secs 2024-04-26T16:59:10.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18489.813834 secs 2024-04-26T16:59:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18489.813834 secs 2024-04-26T16:59:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18489.813834 secs 2024-04-26T16:59:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:10 smithi194 ceph-mon[33017]: pgmap v10333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:10 smithi033 ceph-mon[27845]: pgmap v10333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:10 smithi096 ceph-mon[32753]: pgmap v10333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:12 smithi194 ceph-mon[33017]: pgmap v10334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:12 smithi033 ceph-mon[27845]: pgmap v10334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:12 smithi096 ceph-mon[32753]: pgmap v10334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18491.976503 secs 2024-04-26T16:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18491.976503 secs 2024-04-26T16:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18491.976503 secs 2024-04-26T16:59:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18494.813947 secs 2024-04-26T16:59:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:14 smithi194 ceph-mon[33017]: pgmap v10335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18494.813947 secs 2024-04-26T16:59:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:14 smithi033 ceph-mon[27845]: pgmap v10335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18494.813947 secs 2024-04-26T16:59:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:14 smithi096 ceph-mon[32753]: pgmap v10335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:16.409+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:16.409+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:16.409+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:59:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:16 smithi194 ceph-mon[33017]: pgmap v10336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:16 smithi033 ceph-mon[27845]: pgmap v10336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:16 smithi096 ceph-mon[32753]: pgmap v10336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:18 smithi194 ceph-mon[33017]: pgmap v10337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18496.976562 secs 2024-04-26T16:59:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:18 smithi033 ceph-mon[27845]: pgmap v10337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18496.976562 secs 2024-04-26T16:59:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:18 smithi096 ceph-mon[32753]: pgmap v10337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18496.976562 secs 2024-04-26T16:59:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18499.814080 secs 2024-04-26T16:59:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18499.814080 secs 2024-04-26T16:59:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18499.814080 secs 2024-04-26T16:59:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:20 smithi194 ceph-mon[33017]: pgmap v10338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:20 smithi033 ceph-mon[27845]: pgmap v10338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:20 smithi096 ceph-mon[32753]: pgmap v10338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:22 smithi194 ceph-mon[33017]: pgmap v10339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:22 smithi033 ceph-mon[27845]: pgmap v10339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:22 smithi096 ceph-mon[32753]: pgmap v10339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18501.976643 secs 2024-04-26T16:59:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18501.976643 secs 2024-04-26T16:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18501.976643 secs 2024-04-26T16:59:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:24.935+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T16:59:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:24.935+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T16:59:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:24.935+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T16:59:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:24.935+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T16:59:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18504.814199 secs 2024-04-26T16:59:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:24 smithi096 ceph-mon[32753]: pgmap v10340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18504.814199 secs 2024-04-26T16:59:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:24 smithi194 ceph-mon[33017]: pgmap v10340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18504.814199 secs 2024-04-26T16:59:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:24 smithi033 ceph-mon[27845]: pgmap v10340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:26 smithi194 ceph-mon[33017]: pgmap v10341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:26 smithi033 ceph-mon[27845]: pgmap v10341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:26 smithi096 ceph-mon[32753]: pgmap v10341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:28 smithi194 ceph-mon[33017]: pgmap v10342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18506.976701 secs 2024-04-26T16:59:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:28 smithi033 ceph-mon[27845]: pgmap v10342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18506.976701 secs 2024-04-26T16:59:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:28 smithi096 ceph-mon[32753]: pgmap v10342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18506.976701 secs 2024-04-26T16:59:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18509.814320 secs 2024-04-26T16:59:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18509.814320 secs 2024-04-26T16:59:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18509.814320 secs 2024-04-26T16:59:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:30 smithi194 ceph-mon[33017]: pgmap v10343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:30 smithi033 ceph-mon[27845]: pgmap v10343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:30 smithi096 ceph-mon[32753]: pgmap v10343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:31.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:31.424+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:31.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:31.425+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:31.425 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:31.425+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:59:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:32 smithi194 ceph-mon[33017]: pgmap v10344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:59:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:32 smithi033 ceph-mon[27845]: pgmap v10344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:59:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:32 smithi096 ceph-mon[32753]: pgmap v10344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:33.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:59:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18511.976766 secs 2024-04-26T16:59:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:59:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:59:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18511.976766 secs 2024-04-26T16:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:59:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18511.976766 secs 2024-04-26T16:59:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:59:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:59:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T16:59:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:34 smithi033 ceph-mon[27845]: pgmap v10345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:34 smithi033 ceph-mon[27845]: pgmap v10346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18514.814474 secs 2024-04-26T16:59:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:34 smithi096 ceph-mon[32753]: pgmap v10345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:34 smithi096 ceph-mon[32753]: pgmap v10346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18514.814474 secs 2024-04-26T16:59:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:34 smithi194 ceph-mon[33017]: pgmap v10345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:34 smithi194 ceph-mon[33017]: pgmap v10346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18514.814474 secs 2024-04-26T16:59:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:36 smithi033 ceph-mon[27845]: pgmap v10347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:36 smithi096 ceph-mon[32753]: pgmap v10347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:36 smithi194 ceph-mon[33017]: pgmap v10347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:38 smithi033 ceph-mon[27845]: pgmap v10348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18516.976826 secs 2024-04-26T16:59:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18519.814569 secs 2024-04-26T16:59:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:38 smithi096 ceph-mon[32753]: pgmap v10348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18516.976826 secs 2024-04-26T16:59:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18519.814569 secs 2024-04-26T16:59:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:38 smithi194 ceph-mon[33017]: pgmap v10348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18516.976826 secs 2024-04-26T16:59:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18519.814569 secs 2024-04-26T16:59:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:40 smithi033 ceph-mon[27845]: pgmap v10349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T16:59:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:40 smithi096 ceph-mon[32753]: pgmap v10349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T16:59:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:40 smithi194 ceph-mon[33017]: pgmap v10349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T16:59:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:42 smithi033 ceph-mon[27845]: pgmap v10350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:42 smithi096 ceph-mon[32753]: pgmap v10350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:42 smithi194 ceph-mon[33017]: pgmap v10350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18521.976885 secs 2024-04-26T16:59:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18524.814686 secs 2024-04-26T16:59:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18521.976885 secs 2024-04-26T16:59:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18524.814686 secs 2024-04-26T16:59:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18521.976885 secs 2024-04-26T16:59:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18524.814686 secs 2024-04-26T16:59:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:44 smithi033 ceph-mon[27845]: pgmap v10351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:59:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:44 smithi096 ceph-mon[32753]: pgmap v10351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:59:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:44 smithi194 ceph-mon[33017]: pgmap v10351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T16:59:46.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:46.427+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:46.427+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T16:59:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T16:59:46.427+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T16:59:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:46 smithi033 ceph-mon[27845]: pgmap v10352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:46 smithi096 ceph-mon[32753]: pgmap v10352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:46 smithi194 ceph-mon[33017]: pgmap v10352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:48 smithi033 ceph-mon[27845]: pgmap v10353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18526.976930 secs 2024-04-26T16:59:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18529.814783 secs 2024-04-26T16:59:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:48 smithi096 ceph-mon[32753]: pgmap v10353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18526.976930 secs 2024-04-26T16:59:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18529.814783 secs 2024-04-26T16:59:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:48 smithi194 ceph-mon[33017]: pgmap v10353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18526.976930 secs 2024-04-26T16:59:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18529.814783 secs 2024-04-26T16:59:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:50 smithi033 ceph-mon[27845]: pgmap v10354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:59:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:50 smithi096 ceph-mon[32753]: pgmap v10354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:59:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:50 smithi194 ceph-mon[33017]: pgmap v10354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T16:59:53.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:52 smithi033 ceph-mon[27845]: pgmap v10355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:52 smithi096 ceph-mon[32753]: pgmap v10355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:52 smithi194 ceph-mon[33017]: pgmap v10355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T16:59:54.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18531.976976 secs 2024-04-26T16:59:54.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18534.814903 secs 2024-04-26T16:59:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18531.976976 secs 2024-04-26T16:59:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18534.814903 secs 2024-04-26T16:59:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18531.976976 secs 2024-04-26T16:59:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18534.814903 secs 2024-04-26T16:59:55.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:54 smithi096 ceph-mon[32753]: pgmap v10356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:54 smithi194 ceph-mon[33017]: pgmap v10356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:54 smithi033 ceph-mon[27845]: pgmap v10356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:56 smithi194 ceph-mon[33017]: pgmap v10357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:56 smithi033 ceph-mon[27845]: pgmap v10357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:56 smithi096 ceph-mon[32753]: pgmap v10357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T16:59:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:58 smithi194 ceph-mon[33017]: pgmap v10358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18536.977038 secs 2024-04-26T16:59:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 16:59:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18539.815021 secs 2024-04-26T16:59:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:58 smithi033 ceph-mon[27845]: pgmap v10358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18536.977038 secs 2024-04-26T16:59:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 16:59:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18539.815021 secs 2024-04-26T16:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:58 smithi096 ceph-mon[32753]: pgmap v10358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T16:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18536.977038 secs 2024-04-26T16:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 16:59:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18539.815021 secs 2024-04-26T17:00:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:00 smithi194 ceph-mon[33017]: pgmap v10359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:00:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:00 smithi033 ceph-mon[27845]: pgmap v10359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:00:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:00 smithi096 ceph-mon[32753]: pgmap v10359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:00:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:01.443+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:01.443+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:01.443+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:00:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:02 smithi194 ceph-mon[33017]: pgmap v10360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:02 smithi033 ceph-mon[27845]: pgmap v10360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:03.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:02 smithi096 ceph-mon[32753]: pgmap v10360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:04.272 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18541.977091 secs 2024-04-26T17:00:04.272 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18544.815143 secs 2024-04-26T17:00:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18541.977091 secs 2024-04-26T17:00:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18544.815143 secs 2024-04-26T17:00:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18541.977091 secs 2024-04-26T17:00:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18544.815143 secs 2024-04-26T17:00:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:04 smithi194 ceph-mon[33017]: pgmap v10361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:04 smithi033 ceph-mon[27845]: pgmap v10361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:04 smithi096 ceph-mon[32753]: pgmap v10361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:05 smithi194 ceph-mon[33017]: pgmap v10362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:05 smithi033 ceph-mon[27845]: pgmap v10362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:05 smithi096 ceph-mon[32753]: pgmap v10362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:08 smithi033 ceph-mon[27845]: pgmap v10363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:08 smithi096 ceph-mon[32753]: pgmap v10363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:08 smithi194 ceph-mon[33017]: pgmap v10363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18546.977151 secs 2024-04-26T17:00:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18549.815209 secs 2024-04-26T17:00:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18546.977151 secs 2024-04-26T17:00:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18549.815209 secs 2024-04-26T17:00:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18546.977151 secs 2024-04-26T17:00:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18549.815209 secs 2024-04-26T17:00:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:10 smithi033 ceph-mon[27845]: pgmap v10364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:10 smithi096 ceph-mon[32753]: pgmap v10364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:10 smithi194 ceph-mon[33017]: pgmap v10364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:12 smithi033 ceph-mon[27845]: pgmap v10365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:12 smithi096 ceph-mon[32753]: pgmap v10365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:12 smithi194 ceph-mon[33017]: pgmap v10365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18551.977210 secs 2024-04-26T17:00:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:14 smithi033 ceph-mon[27845]: pgmap v10366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18554.815271 secs 2024-04-26T17:00:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18551.977210 secs 2024-04-26T17:00:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:14 smithi096 ceph-mon[32753]: pgmap v10366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18554.815271 secs 2024-04-26T17:00:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18551.977210 secs 2024-04-26T17:00:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:14 smithi194 ceph-mon[33017]: pgmap v10366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18554.815271 secs 2024-04-26T17:00:16.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:16.457+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:16.457+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:16.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:16.457+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:00:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:16 smithi033 ceph-mon[27845]: pgmap v10367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:16 smithi096 ceph-mon[32753]: pgmap v10367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:16 smithi194 ceph-mon[33017]: pgmap v10367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:18 smithi033 ceph-mon[27845]: pgmap v10368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:18 smithi096 ceph-mon[32753]: pgmap v10368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:18 smithi194 ceph-mon[33017]: pgmap v10368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18556.977273 secs 2024-04-26T17:00:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18559.815391 secs 2024-04-26T17:00:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18556.977273 secs 2024-04-26T17:00:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18559.815391 secs 2024-04-26T17:00:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18556.977273 secs 2024-04-26T17:00:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18559.815391 secs 2024-04-26T17:00:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:20 smithi033 ceph-mon[27845]: pgmap v10369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:20 smithi096 ceph-mon[32753]: pgmap v10369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:20 smithi194 ceph-mon[33017]: pgmap v10369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:00:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:22 smithi033 ceph-mon[27845]: pgmap v10370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:22 smithi096 ceph-mon[32753]: pgmap v10370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:22 smithi194 ceph-mon[33017]: pgmap v10370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18561.977335 secs 2024-04-26T17:00:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:24 smithi096 ceph-mon[32753]: pgmap v10371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18564.815465 secs 2024-04-26T17:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18561.977335 secs 2024-04-26T17:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:24 smithi194 ceph-mon[33017]: pgmap v10371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18564.815465 secs 2024-04-26T17:00:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18561.977335 secs 2024-04-26T17:00:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:24 smithi033 ceph-mon[27845]: pgmap v10371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18564.815465 secs 2024-04-26T17:00:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:26 smithi096 ceph-mon[32753]: pgmap v10372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:26 smithi194 ceph-mon[33017]: pgmap v10372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:26 smithi033 ceph-mon[27845]: pgmap v10372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:28 smithi194 ceph-mon[33017]: pgmap v10373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:28 smithi033 ceph-mon[27845]: pgmap v10373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:28 smithi096 ceph-mon[32753]: pgmap v10373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18566.977396 secs 2024-04-26T17:00:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18569.815578 secs 2024-04-26T17:00:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18566.977396 secs 2024-04-26T17:00:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18569.815578 secs 2024-04-26T17:00:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18566.977396 secs 2024-04-26T17:00:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18569.815578 secs 2024-04-26T17:00:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:30 smithi194 ceph-mon[33017]: pgmap v10374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:30 smithi033 ceph-mon[27845]: pgmap v10374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:30 smithi096 ceph-mon[32753]: pgmap v10374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:31.472+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:31.472+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:00:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:31.472+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:32 smithi194 ceph-mon[33017]: pgmap v10375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:32 smithi033 ceph-mon[27845]: pgmap v10375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:32 smithi096 ceph-mon[32753]: pgmap v10375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18571.977458 secs 2024-04-26T17:00:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:34 smithi194 ceph-mon[33017]: pgmap v10376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18574.815695 secs 2024-04-26T17:00:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:00:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18571.977458 secs 2024-04-26T17:00:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:34 smithi033 ceph-mon[27845]: pgmap v10376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18574.815695 secs 2024-04-26T17:00:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:00:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18571.977458 secs 2024-04-26T17:00:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:34 smithi096 ceph-mon[32753]: pgmap v10376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18574.815695 secs 2024-04-26T17:00:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:00:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:00:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:00:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:00:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:00:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:00:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:00:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:00:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:00:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:00:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:36 smithi194 ceph-mon[33017]: pgmap v10377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:00:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:36 smithi194 ceph-mon[33017]: pgmap v10378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:36 smithi033 ceph-mon[27845]: pgmap v10377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:00:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:36 smithi033 ceph-mon[27845]: pgmap v10378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:36 smithi096 ceph-mon[32753]: pgmap v10377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:00:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:36 smithi096 ceph-mon[32753]: pgmap v10378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:38 smithi194 ceph-mon[33017]: pgmap v10379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18576.977543 secs 2024-04-26T17:00:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:38 smithi033 ceph-mon[27845]: pgmap v10379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18576.977543 secs 2024-04-26T17:00:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:38 smithi096 ceph-mon[32753]: pgmap v10379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18576.977543 secs 2024-04-26T17:00:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18579.815813 secs 2024-04-26T17:00:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18579.815813 secs 2024-04-26T17:00:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18579.815813 secs 2024-04-26T17:00:40.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:40 smithi033 ceph-mon[27845]: pgmap v10380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:00:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:40 smithi194 ceph-mon[33017]: pgmap v10380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:00:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:40 smithi096 ceph-mon[32753]: pgmap v10380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:00:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:42 smithi194 ceph-mon[33017]: pgmap v10381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:42 smithi033 ceph-mon[27845]: pgmap v10381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:42 smithi096 ceph-mon[32753]: pgmap v10381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18581.977627 secs 2024-04-26T17:00:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18581.977627 secs 2024-04-26T17:00:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18581.977627 secs 2024-04-26T17:00:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:44 smithi194 ceph-mon[33017]: pgmap v10382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18584.815930 secs 2024-04-26T17:00:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:44 smithi033 ceph-mon[27845]: pgmap v10382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18584.815930 secs 2024-04-26T17:00:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:44 smithi096 ceph-mon[32753]: pgmap v10382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:00:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18584.815930 secs 2024-04-26T17:00:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:46.488+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:46.488+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:00:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:00:46.488+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:00:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:46 smithi194 ceph-mon[33017]: pgmap v10383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:00:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:46 smithi033 ceph-mon[27845]: pgmap v10383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:00:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:46 smithi096 ceph-mon[32753]: pgmap v10383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:00:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:48 smithi194 ceph-mon[33017]: pgmap v10384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18586.977709 secs 2024-04-26T17:00:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:48 smithi033 ceph-mon[27845]: pgmap v10384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18586.977709 secs 2024-04-26T17:00:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:48 smithi096 ceph-mon[32753]: pgmap v10384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18586.977709 secs 2024-04-26T17:00:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18589.816074 secs 2024-04-26T17:00:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18589.816074 secs 2024-04-26T17:00:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18589.816074 secs 2024-04-26T17:00:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:50 smithi194 ceph-mon[33017]: pgmap v10385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:50.863 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:50 smithi033 ceph-mon[27845]: pgmap v10385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:50 smithi096 ceph-mon[32753]: pgmap v10385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:52 smithi194 ceph-mon[33017]: pgmap v10386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:52 smithi033 ceph-mon[27845]: pgmap v10386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:52 smithi096 ceph-mon[32753]: pgmap v10386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18591.977798 secs 2024-04-26T17:00:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18591.977798 secs 2024-04-26T17:00:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18591.977798 secs 2024-04-26T17:00:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:54 smithi194 ceph-mon[33017]: pgmap v10387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18594.816204 secs 2024-04-26T17:00:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:54 smithi033 ceph-mon[27845]: pgmap v10387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18594.816204 secs 2024-04-26T17:00:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:54 smithi096 ceph-mon[32753]: pgmap v10387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18594.816204 secs 2024-04-26T17:00:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:56 smithi194 ceph-mon[33017]: pgmap v10388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:56 smithi033 ceph-mon[27845]: pgmap v10388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:56 smithi096 ceph-mon[32753]: pgmap v10388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:00:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:58 smithi194 ceph-mon[33017]: pgmap v10389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18596.977882 secs 2024-04-26T17:00:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:58 smithi033 ceph-mon[27845]: pgmap v10389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18596.977882 secs 2024-04-26T17:00:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:58 smithi096 ceph-mon[32753]: pgmap v10389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:00:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18596.977882 secs 2024-04-26T17:00:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18599.816284 secs 2024-04-26T17:00:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18599.816284 secs 2024-04-26T17:00:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18599.816284 secs 2024-04-26T17:01:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:00 smithi194 ceph-mon[33017]: pgmap v10390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:00 smithi033 ceph-mon[27845]: pgmap v10390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:00 smithi096 ceph-mon[32753]: pgmap v10390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:01.490+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:01.490+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:01:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:01.490+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:02 smithi194 ceph-mon[33017]: pgmap v10391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:02 smithi033 ceph-mon[27845]: pgmap v10391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:02 smithi096 ceph-mon[32753]: pgmap v10391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18601.977974 secs 2024-04-26T17:01:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18601.977974 secs 2024-04-26T17:01:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18601.977974 secs 2024-04-26T17:01:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:04 smithi194 ceph-mon[33017]: pgmap v10392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18604.816363 secs 2024-04-26T17:01:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:04 smithi033 ceph-mon[27845]: pgmap v10392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18604.816363 secs 2024-04-26T17:01:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:04 smithi096 ceph-mon[32753]: pgmap v10392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18604.816363 secs 2024-04-26T17:01:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:06 smithi194 ceph-mon[33017]: pgmap v10393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:06 smithi033 ceph-mon[27845]: pgmap v10393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:06 smithi096 ceph-mon[32753]: pgmap v10393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:08 smithi194 ceph-mon[33017]: pgmap v10394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18606.978002 secs 2024-04-26T17:01:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:08 smithi033 ceph-mon[27845]: pgmap v10394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18606.978002 secs 2024-04-26T17:01:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:08 smithi096 ceph-mon[32753]: pgmap v10394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18606.978002 secs 2024-04-26T17:01:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18609.816521 secs 2024-04-26T17:01:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18609.816521 secs 2024-04-26T17:01:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18609.816521 secs 2024-04-26T17:01:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:10 smithi033 ceph-mon[27845]: pgmap v10395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:10 smithi096 ceph-mon[32753]: pgmap v10395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:10 smithi194 ceph-mon[33017]: pgmap v10395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:12 smithi033 ceph-mon[27845]: pgmap v10396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:12 smithi096 ceph-mon[32753]: pgmap v10396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:12 smithi194 ceph-mon[33017]: pgmap v10396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18611.978063 secs 2024-04-26T17:01:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18611.978063 secs 2024-04-26T17:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18611.978063 secs 2024-04-26T17:01:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:14 smithi033 ceph-mon[27845]: pgmap v10397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18614.816648 secs 2024-04-26T17:01:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:14 smithi096 ceph-mon[32753]: pgmap v10397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18614.816648 secs 2024-04-26T17:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:14 smithi194 ceph-mon[33017]: pgmap v10397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18614.816648 secs 2024-04-26T17:01:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:16.503+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:16.503+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:16.503+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:01:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:16 smithi033 ceph-mon[27845]: pgmap v10398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:16 smithi096 ceph-mon[32753]: pgmap v10398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:16 smithi194 ceph-mon[33017]: pgmap v10398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:18 smithi033 ceph-mon[27845]: pgmap v10399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18616.978125 secs 2024-04-26T17:01:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:18 smithi096 ceph-mon[32753]: pgmap v10399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18616.978125 secs 2024-04-26T17:01:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:18 smithi194 ceph-mon[33017]: pgmap v10399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18616.978125 secs 2024-04-26T17:01:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18619.816731 secs 2024-04-26T17:01:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18619.816731 secs 2024-04-26T17:01:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18619.816731 secs 2024-04-26T17:01:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:20 smithi194 ceph-mon[33017]: pgmap v10400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:20 smithi033 ceph-mon[27845]: pgmap v10400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:20 smithi096 ceph-mon[32753]: pgmap v10400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:22 smithi194 ceph-mon[33017]: pgmap v10401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:22 smithi033 ceph-mon[27845]: pgmap v10401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:22 smithi096 ceph-mon[32753]: pgmap v10401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:24.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18621.978204 secs 2024-04-26T17:01:24.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18624.816812 secs 2024-04-26T17:01:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18621.978204 secs 2024-04-26T17:01:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18624.816812 secs 2024-04-26T17:01:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18621.978204 secs 2024-04-26T17:01:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18624.816812 secs 2024-04-26T17:01:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:24 smithi194 ceph-mon[33017]: pgmap v10402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:24 smithi033 ceph-mon[27845]: pgmap v10402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:24 smithi096 ceph-mon[32753]: pgmap v10402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:27.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:26 smithi096 ceph-mon[32753]: pgmap v10403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:26 smithi194 ceph-mon[33017]: pgmap v10403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:26 smithi033 ceph-mon[27845]: pgmap v10403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:28 smithi194 ceph-mon[33017]: pgmap v10404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18626.978264 secs 2024-04-26T17:01:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18629.816968 secs 2024-04-26T17:01:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:28 smithi033 ceph-mon[27845]: pgmap v10404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18626.978264 secs 2024-04-26T17:01:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18629.816968 secs 2024-04-26T17:01:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:28 smithi096 ceph-mon[32753]: pgmap v10404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18626.978264 secs 2024-04-26T17:01:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18629.816968 secs 2024-04-26T17:01:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:30 smithi194 ceph-mon[33017]: pgmap v10405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:30 smithi033 ceph-mon[27845]: pgmap v10405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:30 smithi096 ceph-mon[32753]: pgmap v10405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:31.512+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:31.512+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:31.512+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:01:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:32 smithi194 ceph-mon[33017]: pgmap v10406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:32 smithi033 ceph-mon[27845]: pgmap v10406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:32 smithi096 ceph-mon[32753]: pgmap v10406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18631.978343 secs 2024-04-26T17:01:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18634.817053 secs 2024-04-26T17:01:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18631.978343 secs 2024-04-26T17:01:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18634.817053 secs 2024-04-26T17:01:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18631.978343 secs 2024-04-26T17:01:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18634.817053 secs 2024-04-26T17:01:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:34 smithi194 ceph-mon[33017]: pgmap v10407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:34 smithi033 ceph-mon[27845]: pgmap v10407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:34 smithi096 ceph-mon[32753]: pgmap v10407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:01:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:01:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:01:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:37 smithi194 ceph-mon[33017]: pgmap v10408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:01:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:37 smithi033 ceph-mon[27845]: pgmap v10408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:01:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:37 smithi096 ceph-mon[32753]: pgmap v10408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:01:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:38 smithi194 ceph-mon[33017]: pgmap v10409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:01:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:01:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:01:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:38 smithi194 ceph-mon[33017]: pgmap v10410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:01:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:38 smithi194 ceph-mon[33017]: pgmap v10411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:38 smithi033 ceph-mon[27845]: pgmap v10409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:01:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:01:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:01:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:38 smithi033 ceph-mon[27845]: pgmap v10410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:01:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:38 smithi033 ceph-mon[27845]: pgmap v10411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:38 smithi096 ceph-mon[32753]: pgmap v10409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:01:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:01:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:01:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:38 smithi096 ceph-mon[32753]: pgmap v10410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:01:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:38 smithi096 ceph-mon[32753]: pgmap v10411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:01:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18636.978401 secs 2024-04-26T17:01:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18639.817219 secs 2024-04-26T17:01:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18636.978401 secs 2024-04-26T17:01:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18639.817219 secs 2024-04-26T17:01:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18636.978401 secs 2024-04-26T17:01:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18639.817219 secs 2024-04-26T17:01:40.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:40 smithi033 ceph-mon[27845]: pgmap v10412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:40 smithi194 ceph-mon[33017]: pgmap v10412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:40 smithi096 ceph-mon[32753]: pgmap v10412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:42 smithi194 ceph-mon[33017]: pgmap v10413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:42 smithi033 ceph-mon[27845]: pgmap v10413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:42 smithi096 ceph-mon[32753]: pgmap v10413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:01:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18641.978459 secs 2024-04-26T17:01:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18641.978459 secs 2024-04-26T17:01:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18641.978459 secs 2024-04-26T17:01:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18644.817344 secs 2024-04-26T17:01:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:44 smithi194 ceph-mon[33017]: pgmap v10414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:01:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18644.817344 secs 2024-04-26T17:01:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:44 smithi033 ceph-mon[27845]: pgmap v10414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:01:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18644.817344 secs 2024-04-26T17:01:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:44 smithi096 ceph-mon[32753]: pgmap v10414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:01:46.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:46.527+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:46.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:46.527+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:01:46.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:01:46.527+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:01:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:46 smithi194 ceph-mon[33017]: pgmap v10415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:01:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:46 smithi033 ceph-mon[27845]: pgmap v10415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:01:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:46 smithi096 ceph-mon[32753]: pgmap v10415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:01:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:48 smithi194 ceph-mon[33017]: pgmap v10416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:01:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18646.978517 secs 2024-04-26T17:01:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:48 smithi033 ceph-mon[27845]: pgmap v10416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:01:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18646.978517 secs 2024-04-26T17:01:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:48 smithi096 ceph-mon[32753]: pgmap v10416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:01:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18646.978517 secs 2024-04-26T17:01:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18649.817463 secs 2024-04-26T17:01:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18649.817463 secs 2024-04-26T17:01:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18649.817463 secs 2024-04-26T17:01:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:50 smithi194 ceph-mon[33017]: pgmap v10417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:51.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:50 smithi033 ceph-mon[27845]: pgmap v10417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:50 smithi096 ceph-mon[32753]: pgmap v10417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:52 smithi194 ceph-mon[33017]: pgmap v10418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:52 smithi033 ceph-mon[27845]: pgmap v10418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:53.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:52 smithi096 ceph-mon[32753]: pgmap v10418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18651.978578 secs 2024-04-26T17:01:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18651.978578 secs 2024-04-26T17:01:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18651.978578 secs 2024-04-26T17:01:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18654.817608 secs 2024-04-26T17:01:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:54 smithi194 ceph-mon[33017]: pgmap v10419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18654.817608 secs 2024-04-26T17:01:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:54 smithi033 ceph-mon[27845]: pgmap v10419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18654.817608 secs 2024-04-26T17:01:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:54 smithi096 ceph-mon[32753]: pgmap v10419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:57.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:56 smithi194 ceph-mon[33017]: pgmap v10420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:56 smithi033 ceph-mon[27845]: pgmap v10420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:56 smithi096 ceph-mon[32753]: pgmap v10420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:01:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:58 smithi194 ceph-mon[33017]: pgmap v10421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18656.978636 secs 2024-04-26T17:01:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:58 smithi033 ceph-mon[27845]: pgmap v10421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18656.978636 secs 2024-04-26T17:01:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:58 smithi096 ceph-mon[32753]: pgmap v10421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:01:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18656.978636 secs 2024-04-26T17:02:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18659.817732 secs 2024-04-26T17:02:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18659.817732 secs 2024-04-26T17:02:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18659.817732 secs 2024-04-26T17:02:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:00 smithi194 ceph-mon[33017]: pgmap v10422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:00 smithi033 ceph-mon[27845]: pgmap v10422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:00 smithi096 ceph-mon[32753]: pgmap v10422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:01.542+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:01.543+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:01.543+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:02:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:02 smithi194 ceph-mon[33017]: pgmap v10423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:02 smithi033 ceph-mon[27845]: pgmap v10423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:02 smithi096 ceph-mon[32753]: pgmap v10423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18661.978686 secs 2024-04-26T17:02:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18661.978686 secs 2024-04-26T17:02:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18661.978686 secs 2024-04-26T17:02:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18664.817857 secs 2024-04-26T17:02:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:04 smithi194 ceph-mon[33017]: pgmap v10424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18664.817857 secs 2024-04-26T17:02:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:04 smithi033 ceph-mon[27845]: pgmap v10424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18664.817857 secs 2024-04-26T17:02:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:04 smithi096 ceph-mon[32753]: pgmap v10424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:06 smithi194 ceph-mon[33017]: pgmap v10425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:06 smithi033 ceph-mon[27845]: pgmap v10425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:06 smithi096 ceph-mon[32753]: pgmap v10425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:08 smithi194 ceph-mon[33017]: pgmap v10426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18666.978745 secs 2024-04-26T17:02:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:08 smithi033 ceph-mon[27845]: pgmap v10426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18666.978745 secs 2024-04-26T17:02:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:08 smithi096 ceph-mon[32753]: pgmap v10426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18666.978745 secs 2024-04-26T17:02:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18669.817987 secs 2024-04-26T17:02:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18669.817987 secs 2024-04-26T17:02:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18669.817987 secs 2024-04-26T17:02:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:10 smithi194 ceph-mon[33017]: pgmap v10427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:10 smithi033 ceph-mon[27845]: pgmap v10427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:10 smithi096 ceph-mon[32753]: pgmap v10427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:12 smithi194 ceph-mon[33017]: pgmap v10428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:12 smithi033 ceph-mon[27845]: pgmap v10428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:12 smithi096 ceph-mon[32753]: pgmap v10428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18671.978796 secs 2024-04-26T17:02:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18671.978796 secs 2024-04-26T17:02:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18671.978796 secs 2024-04-26T17:02:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18674.818133 secs 2024-04-26T17:02:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:14 smithi033 ceph-mon[27845]: pgmap v10429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18674.818133 secs 2024-04-26T17:02:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:14 smithi096 ceph-mon[32753]: pgmap v10429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18674.818133 secs 2024-04-26T17:02:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:14 smithi194 ceph-mon[33017]: pgmap v10429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:16.557+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:16.557+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:16.557+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:02:17.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:16 smithi033 ceph-mon[27845]: pgmap v10430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:16 smithi096 ceph-mon[32753]: pgmap v10430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:16 smithi194 ceph-mon[33017]: pgmap v10430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:18 smithi033 ceph-mon[27845]: pgmap v10431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18676.978854 secs 2024-04-26T17:02:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:18 smithi096 ceph-mon[32753]: pgmap v10431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18676.978854 secs 2024-04-26T17:02:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:18 smithi194 ceph-mon[33017]: pgmap v10431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18676.978854 secs 2024-04-26T17:02:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18679.818276 secs 2024-04-26T17:02:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18679.818276 secs 2024-04-26T17:02:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18679.818276 secs 2024-04-26T17:02:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:20 smithi033 ceph-mon[27845]: pgmap v10432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:20 smithi096 ceph-mon[32753]: pgmap v10432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:20 smithi194 ceph-mon[33017]: pgmap v10432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:22 smithi033 ceph-mon[27845]: pgmap v10433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:22 smithi096 ceph-mon[32753]: pgmap v10433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:22 smithi194 ceph-mon[33017]: pgmap v10433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18681.978911 secs 2024-04-26T17:02:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18681.978911 secs 2024-04-26T17:02:24.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18681.978911 secs 2024-04-26T17:02:24.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:24.965+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:02:24.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:24.965+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T17:02:24.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:24.965+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:02:24.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:24.965+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:02:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18684.818359 secs 2024-04-26T17:02:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:24 smithi033 ceph-mon[27845]: pgmap v10434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18684.818359 secs 2024-04-26T17:02:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:24 smithi096 ceph-mon[32753]: pgmap v10434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18684.818359 secs 2024-04-26T17:02:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:24 smithi194 ceph-mon[33017]: pgmap v10434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:26 smithi033 ceph-mon[27845]: pgmap v10435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:26 smithi096 ceph-mon[32753]: pgmap v10435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:26 smithi194 ceph-mon[33017]: pgmap v10435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:28 smithi033 ceph-mon[27845]: pgmap v10436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18686.978978 secs 2024-04-26T17:02:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18689.818486 secs 2024-04-26T17:02:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:28 smithi096 ceph-mon[32753]: pgmap v10436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18686.978978 secs 2024-04-26T17:02:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18689.818486 secs 2024-04-26T17:02:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:28 smithi194 ceph-mon[33017]: pgmap v10436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18686.978978 secs 2024-04-26T17:02:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18689.818486 secs 2024-04-26T17:02:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:30 smithi033 ceph-mon[27845]: pgmap v10437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:30 smithi096 ceph-mon[32753]: pgmap v10437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:30 smithi194 ceph-mon[33017]: pgmap v10437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:31.574+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:31.574+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:31.574+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:02:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:32 smithi033 ceph-mon[27845]: pgmap v10438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:32 smithi096 ceph-mon[32753]: pgmap v10438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:32 smithi194 ceph-mon[33017]: pgmap v10438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18691.979031 secs 2024-04-26T17:02:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18694.818621 secs 2024-04-26T17:02:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18691.979031 secs 2024-04-26T17:02:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18694.818621 secs 2024-04-26T17:02:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18691.979031 secs 2024-04-26T17:02:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18694.818621 secs 2024-04-26T17:02:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:34 smithi194 ceph-mon[33017]: pgmap v10439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:34 smithi033 ceph-mon[27845]: pgmap v10439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:34 smithi096 ceph-mon[32753]: pgmap v10439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:36 smithi194 ceph-mon[33017]: pgmap v10440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:36 smithi033 ceph-mon[27845]: pgmap v10440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:36 smithi096 ceph-mon[32753]: pgmap v10440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:38 smithi194 ceph-mon[33017]: pgmap v10441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18696.979091 secs 2024-04-26T17:02:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:02:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18699.818706 secs 2024-04-26T17:02:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:38 smithi033 ceph-mon[27845]: pgmap v10441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18696.979091 secs 2024-04-26T17:02:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:02:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18699.818706 secs 2024-04-26T17:02:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:38 smithi096 ceph-mon[32753]: pgmap v10441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18696.979091 secs 2024-04-26T17:02:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:02:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18699.818706 secs 2024-04-26T17:02:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: pgmap v10442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:02:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:02:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: pgmap v10442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:02:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:02:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: pgmap v10442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:02:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:02:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:02:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:41 smithi194 ceph-mon[33017]: pgmap v10443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:02:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:41 smithi194 ceph-mon[33017]: pgmap v10444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:41 smithi033 ceph-mon[27845]: pgmap v10443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:02:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:41 smithi033 ceph-mon[27845]: pgmap v10444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:41 smithi096 ceph-mon[32753]: pgmap v10443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:02:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:41 smithi096 ceph-mon[32753]: pgmap v10444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:43 smithi194 ceph-mon[33017]: pgmap v10445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:43 smithi033 ceph-mon[27845]: pgmap v10445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:43 smithi096 ceph-mon[32753]: pgmap v10445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18701.979176 secs 2024-04-26T17:02:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18704.818825 secs 2024-04-26T17:02:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18701.979176 secs 2024-04-26T17:02:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18704.818825 secs 2024-04-26T17:02:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18701.979176 secs 2024-04-26T17:02:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18704.818825 secs 2024-04-26T17:02:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:45 smithi194 ceph-mon[33017]: pgmap v10446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:02:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:45 smithi033 ceph-mon[27845]: pgmap v10446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:02:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:45 smithi096 ceph-mon[32753]: pgmap v10446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:02:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:46.584+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:46.584+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:02:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:02:46.584+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:02:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:47 smithi194 ceph-mon[33017]: pgmap v10447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:47 smithi033 ceph-mon[27845]: pgmap v10447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:47 smithi096 ceph-mon[32753]: pgmap v10447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:02:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18706.979275 secs 2024-04-26T17:02:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:49 smithi194 ceph-mon[33017]: pgmap v10448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:02:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18709.818954 secs 2024-04-26T17:02:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18706.979275 secs 2024-04-26T17:02:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:49 smithi033 ceph-mon[27845]: pgmap v10448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:02:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18709.818954 secs 2024-04-26T17:02:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18706.979275 secs 2024-04-26T17:02:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:49 smithi096 ceph-mon[32753]: pgmap v10448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:02:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18709.818954 secs 2024-04-26T17:02:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:51 smithi194 ceph-mon[33017]: pgmap v10449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:02:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:51 smithi033 ceph-mon[27845]: pgmap v10449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:02:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:51 smithi096 ceph-mon[32753]: pgmap v10449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:02:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:53 smithi194 ceph-mon[33017]: pgmap v10450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:53 smithi033 ceph-mon[27845]: pgmap v10450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:53 smithi096 ceph-mon[32753]: pgmap v10450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18711.979355 secs 2024-04-26T17:02:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18714.819115 secs 2024-04-26T17:02:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18711.979355 secs 2024-04-26T17:02:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18714.819115 secs 2024-04-26T17:02:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18711.979355 secs 2024-04-26T17:02:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18714.819115 secs 2024-04-26T17:02:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:55 smithi194 ceph-mon[33017]: pgmap v10451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:55 smithi033 ceph-mon[27845]: pgmap v10451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:55 smithi096 ceph-mon[32753]: pgmap v10451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:57 smithi194 ceph-mon[33017]: pgmap v10452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:57.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:57 smithi033 ceph-mon[27845]: pgmap v10452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:57 smithi096 ceph-mon[32753]: pgmap v10452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:02:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18716.979437 secs 2024-04-26T17:02:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:59 smithi194 ceph-mon[33017]: pgmap v10453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18719.819230 secs 2024-04-26T17:02:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18716.979437 secs 2024-04-26T17:02:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:59 smithi033 ceph-mon[27845]: pgmap v10453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18719.819230 secs 2024-04-26T17:02:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18716.979437 secs 2024-04-26T17:02:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:59 smithi096 ceph-mon[32753]: pgmap v10453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:02:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18719.819230 secs 2024-04-26T17:03:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:01.598+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:01.598+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:01.598+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:03:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:01 smithi194 ceph-mon[33017]: pgmap v10454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:01 smithi033 ceph-mon[27845]: pgmap v10454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:01 smithi096 ceph-mon[32753]: pgmap v10454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:03 smithi194 ceph-mon[33017]: pgmap v10455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:03 smithi033 ceph-mon[27845]: pgmap v10455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:03 smithi096 ceph-mon[32753]: pgmap v10455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18721.979519 secs 2024-04-26T17:03:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18724.819332 secs 2024-04-26T17:03:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18721.979519 secs 2024-04-26T17:03:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18724.819332 secs 2024-04-26T17:03:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18721.979519 secs 2024-04-26T17:03:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18724.819332 secs 2024-04-26T17:03:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:05 smithi194 ceph-mon[33017]: pgmap v10456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:05 smithi033 ceph-mon[27845]: pgmap v10456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:05 smithi096 ceph-mon[32753]: pgmap v10456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:07 smithi194 ceph-mon[33017]: pgmap v10457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:07 smithi033 ceph-mon[27845]: pgmap v10457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:07 smithi096 ceph-mon[32753]: pgmap v10457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18726.979603 secs 2024-04-26T17:03:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:09 smithi194 ceph-mon[33017]: pgmap v10458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18729.819438 secs 2024-04-26T17:03:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18726.979603 secs 2024-04-26T17:03:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:09 smithi033 ceph-mon[27845]: pgmap v10458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18729.819438 secs 2024-04-26T17:03:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18726.979603 secs 2024-04-26T17:03:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:09 smithi096 ceph-mon[32753]: pgmap v10458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18729.819438 secs 2024-04-26T17:03:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:09.972+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:03:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:09.972+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T17:03:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:09.972+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:03:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:09.972+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:03:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:11 smithi194 ceph-mon[33017]: pgmap v10459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:11 smithi033 ceph-mon[27845]: pgmap v10459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:11 smithi096 ceph-mon[32753]: pgmap v10459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:13 smithi194 ceph-mon[33017]: pgmap v10460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:13 smithi033 ceph-mon[27845]: pgmap v10460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:13 smithi096 ceph-mon[32753]: pgmap v10460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18731.979687 secs 2024-04-26T17:03:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18734.819564 secs 2024-04-26T17:03:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18731.979687 secs 2024-04-26T17:03:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18734.819564 secs 2024-04-26T17:03:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18731.979687 secs 2024-04-26T17:03:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18734.819564 secs 2024-04-26T17:03:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:15 smithi194 ceph-mon[33017]: pgmap v10461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:15 smithi033 ceph-mon[27845]: pgmap v10461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:15 smithi096 ceph-mon[32753]: pgmap v10461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:16.604+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:16.604+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:16.604+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:03:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:17 smithi194 ceph-mon[33017]: pgmap v10462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:17 smithi033 ceph-mon[27845]: pgmap v10462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:17 smithi096 ceph-mon[32753]: pgmap v10462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18736.979771 secs 2024-04-26T17:03:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:19 smithi194 ceph-mon[33017]: pgmap v10463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18739.819731 secs 2024-04-26T17:03:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18736.979771 secs 2024-04-26T17:03:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:19 smithi033 ceph-mon[27845]: pgmap v10463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18739.819731 secs 2024-04-26T17:03:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18736.979771 secs 2024-04-26T17:03:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:19 smithi096 ceph-mon[32753]: pgmap v10463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18739.819731 secs 2024-04-26T17:03:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:21 smithi194 ceph-mon[33017]: pgmap v10464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:21 smithi033 ceph-mon[27845]: pgmap v10464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:21 smithi096 ceph-mon[32753]: pgmap v10464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:23 smithi194 ceph-mon[33017]: pgmap v10465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:23 smithi033 ceph-mon[27845]: pgmap v10465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:23 smithi096 ceph-mon[32753]: pgmap v10465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18741.979854 secs 2024-04-26T17:03:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18744.819877 secs 2024-04-26T17:03:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18741.979854 secs 2024-04-26T17:03:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18744.819877 secs 2024-04-26T17:03:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18741.979854 secs 2024-04-26T17:03:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18744.819877 secs 2024-04-26T17:03:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:25 smithi033 ceph-mon[27845]: pgmap v10466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:25 smithi096 ceph-mon[32753]: pgmap v10466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:25 smithi194 ceph-mon[33017]: pgmap v10466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:27 smithi033 ceph-mon[27845]: pgmap v10467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:27 smithi096 ceph-mon[32753]: pgmap v10467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:27 smithi194 ceph-mon[33017]: pgmap v10467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18746.979970 secs 2024-04-26T17:03:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:29 smithi033 ceph-mon[27845]: pgmap v10468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18749.819984 secs 2024-04-26T17:03:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18746.979970 secs 2024-04-26T17:03:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:29 smithi096 ceph-mon[32753]: pgmap v10468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18749.819984 secs 2024-04-26T17:03:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18746.979970 secs 2024-04-26T17:03:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:29 smithi194 ceph-mon[33017]: pgmap v10468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18749.819984 secs 2024-04-26T17:03:31.615 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:31.615+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:31.615+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:03:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:31.615+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:31 smithi033 ceph-mon[27845]: pgmap v10469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:31 smithi096 ceph-mon[32753]: pgmap v10469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:31 smithi194 ceph-mon[33017]: pgmap v10469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:33 smithi033 ceph-mon[27845]: pgmap v10470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:33 smithi096 ceph-mon[32753]: pgmap v10470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:33 smithi194 ceph-mon[33017]: pgmap v10470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18751.980045 secs 2024-04-26T17:03:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18754.820060 secs 2024-04-26T17:03:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18751.980045 secs 2024-04-26T17:03:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18754.820060 secs 2024-04-26T17:03:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18751.980045 secs 2024-04-26T17:03:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18754.820060 secs 2024-04-26T17:03:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:35 smithi033 ceph-mon[27845]: pgmap v10471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:35 smithi096 ceph-mon[32753]: pgmap v10471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:35 smithi194 ceph-mon[33017]: pgmap v10471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:37 smithi033 ceph-mon[27845]: pgmap v10472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:37 smithi096 ceph-mon[32753]: pgmap v10472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:37 smithi194 ceph-mon[33017]: pgmap v10472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18756.980106 secs 2024-04-26T17:03:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:39 smithi033 ceph-mon[27845]: pgmap v10473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18759.820180 secs 2024-04-26T17:03:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18756.980106 secs 2024-04-26T17:03:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:39 smithi096 ceph-mon[32753]: pgmap v10473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18759.820180 secs 2024-04-26T17:03:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18756.980106 secs 2024-04-26T17:03:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:39 smithi194 ceph-mon[33017]: pgmap v10473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18759.820180 secs 2024-04-26T17:03:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:41 smithi033 ceph-mon[27845]: pgmap v10474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:03:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:41 smithi096 ceph-mon[32753]: pgmap v10474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:03:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:41 smithi194 ceph-mon[33017]: pgmap v10474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:43 smithi033 ceph-mon[27845]: pgmap v10475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:43 smithi096 ceph-mon[32753]: pgmap v10475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:43 smithi194 ceph-mon[33017]: pgmap v10475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18761.980173 secs 2024-04-26T17:03:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18764.820250 secs 2024-04-26T17:03:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:03:44.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18761.980173 secs 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18764.820250 secs 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:03:44.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18761.980173 secs 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18764.820250 secs 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:03:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:03:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:45 smithi194 ceph-mon[33017]: pgmap v10476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:45 smithi033 ceph-mon[27845]: pgmap v10476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:45 smithi096 ceph-mon[32753]: pgmap v10476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:03:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:46.631+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:46.631+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:03:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:03:46.631+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: pgmap v10477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:03:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:03:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:03:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:03:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: pgmap v10478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:03:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: pgmap v10479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: pgmap v10477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:03:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:03:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:03:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:03:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: pgmap v10478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:03:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: pgmap v10479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:47.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: pgmap v10477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:03:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:03:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:03:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:03:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:03:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: pgmap v10478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:03:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: pgmap v10479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:47.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:03:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18766.980234 secs 2024-04-26T17:03:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18766.980234 secs 2024-04-26T17:03:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18766.980234 secs 2024-04-26T17:03:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18769.820388 secs 2024-04-26T17:03:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:49 smithi194 ceph-mon[33017]: pgmap v10480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:03:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18769.820388 secs 2024-04-26T17:03:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:49 smithi033 ceph-mon[27845]: pgmap v10480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:03:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18769.820388 secs 2024-04-26T17:03:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:49 smithi096 ceph-mon[32753]: pgmap v10480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:03:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:51 smithi194 ceph-mon[33017]: pgmap v10481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:51 smithi033 ceph-mon[27845]: pgmap v10481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:51 smithi096 ceph-mon[32753]: pgmap v10481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:53 smithi194 ceph-mon[33017]: pgmap v10482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18771.980295 secs 2024-04-26T17:03:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:53 smithi033 ceph-mon[27845]: pgmap v10482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18771.980295 secs 2024-04-26T17:03:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:53 smithi096 ceph-mon[32753]: pgmap v10482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18771.980295 secs 2024-04-26T17:03:55.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18774.820518 secs 2024-04-26T17:03:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18774.820518 secs 2024-04-26T17:03:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18774.820518 secs 2024-04-26T17:03:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:55 smithi194 ceph-mon[33017]: pgmap v10483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:55 smithi033 ceph-mon[27845]: pgmap v10483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:55 smithi096 ceph-mon[32753]: pgmap v10483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:03:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:57 smithi194 ceph-mon[33017]: pgmap v10484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:03:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:57 smithi033 ceph-mon[27845]: pgmap v10484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:03:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:57 smithi096 ceph-mon[32753]: pgmap v10484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:03:59.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18776.980356 secs 2024-04-26T17:03:59.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18776.980356 secs 2024-04-26T17:03:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18776.980356 secs 2024-04-26T17:04:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18779.820585 secs 2024-04-26T17:04:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:03:59 smithi194 ceph-mon[33017]: pgmap v10485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18779.820585 secs 2024-04-26T17:04:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:03:59 smithi033 ceph-mon[27845]: pgmap v10485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18779.820585 secs 2024-04-26T17:04:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:03:59 smithi096 ceph-mon[32753]: pgmap v10485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:01.632+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:01.632+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:04:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:01.632+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:01 smithi194 ceph-mon[33017]: pgmap v10486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:01 smithi033 ceph-mon[27845]: pgmap v10486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:01 smithi096 ceph-mon[32753]: pgmap v10486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:03 smithi194 ceph-mon[33017]: pgmap v10487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18781.980416 secs 2024-04-26T17:04:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:03 smithi033 ceph-mon[27845]: pgmap v10487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18781.980416 secs 2024-04-26T17:04:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:03 smithi096 ceph-mon[32753]: pgmap v10487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18781.980416 secs 2024-04-26T17:04:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18784.820724 secs 2024-04-26T17:04:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18784.820724 secs 2024-04-26T17:04:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18784.820724 secs 2024-04-26T17:04:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:05 smithi194 ceph-mon[33017]: pgmap v10488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:05 smithi033 ceph-mon[27845]: pgmap v10488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:05 smithi096 ceph-mon[32753]: pgmap v10488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:07 smithi194 ceph-mon[33017]: pgmap v10489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:07 smithi033 ceph-mon[27845]: pgmap v10489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:07 smithi096 ceph-mon[32753]: pgmap v10489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18786.980524 secs 2024-04-26T17:04:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18786.980524 secs 2024-04-26T17:04:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18786.980524 secs 2024-04-26T17:04:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18789.820861 secs 2024-04-26T17:04:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:09 smithi194 ceph-mon[33017]: pgmap v10490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18789.820861 secs 2024-04-26T17:04:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:09 smithi033 ceph-mon[27845]: pgmap v10490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18789.820861 secs 2024-04-26T17:04:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:09 smithi096 ceph-mon[32753]: pgmap v10490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:11 smithi194 ceph-mon[33017]: pgmap v10491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:11 smithi033 ceph-mon[27845]: pgmap v10491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:11 smithi096 ceph-mon[32753]: pgmap v10491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:13 smithi194 ceph-mon[33017]: pgmap v10492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18791.980605 secs 2024-04-26T17:04:14.127 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:13 smithi033 ceph-mon[27845]: pgmap v10492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:14.127 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18791.980605 secs 2024-04-26T17:04:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:13 smithi096 ceph-mon[32753]: pgmap v10492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18791.980605 secs 2024-04-26T17:04:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18794.820985 secs 2024-04-26T17:04:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18794.820985 secs 2024-04-26T17:04:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18794.820985 secs 2024-04-26T17:04:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:15 smithi194 ceph-mon[33017]: pgmap v10493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:15 smithi033 ceph-mon[27845]: pgmap v10493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:15 smithi096 ceph-mon[32753]: pgmap v10493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:16.646+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:16.646+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:16.646+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:04:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:17 smithi194 ceph-mon[33017]: pgmap v10494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:17 smithi033 ceph-mon[27845]: pgmap v10494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:17 smithi096 ceph-mon[32753]: pgmap v10494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18796.980689 secs 2024-04-26T17:04:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18796.980689 secs 2024-04-26T17:04:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18796.980689 secs 2024-04-26T17:04:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18799.821080 secs 2024-04-26T17:04:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:19 smithi194 ceph-mon[33017]: pgmap v10495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18799.821080 secs 2024-04-26T17:04:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:19 smithi033 ceph-mon[27845]: pgmap v10495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18799.821080 secs 2024-04-26T17:04:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:19 smithi096 ceph-mon[32753]: pgmap v10495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:21 smithi194 ceph-mon[33017]: pgmap v10496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:21 smithi033 ceph-mon[27845]: pgmap v10496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:21 smithi096 ceph-mon[32753]: pgmap v10496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:23 smithi194 ceph-mon[33017]: pgmap v10497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18801.980770 secs 2024-04-26T17:04:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:23 smithi033 ceph-mon[27845]: pgmap v10497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18801.980770 secs 2024-04-26T17:04:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:23 smithi096 ceph-mon[32753]: pgmap v10497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18801.980770 secs 2024-04-26T17:04:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18804.821163 secs 2024-04-26T17:04:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18804.821163 secs 2024-04-26T17:04:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18804.821163 secs 2024-04-26T17:04:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:25 smithi194 ceph-mon[33017]: pgmap v10498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:26.116 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:25 smithi096 ceph-mon[32753]: pgmap v10498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:25 smithi033 ceph-mon[27845]: pgmap v10498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:27 smithi194 ceph-mon[33017]: pgmap v10499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:27 smithi033 ceph-mon[27845]: pgmap v10499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:27 smithi096 ceph-mon[32753]: pgmap v10499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18806.980853 secs 2024-04-26T17:04:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18806.980853 secs 2024-04-26T17:04:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18806.980853 secs 2024-04-26T17:04:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18809.821283 secs 2024-04-26T17:04:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:29 smithi194 ceph-mon[33017]: pgmap v10500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18809.821283 secs 2024-04-26T17:04:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:29 smithi033 ceph-mon[27845]: pgmap v10500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18809.821283 secs 2024-04-26T17:04:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:29 smithi096 ceph-mon[32753]: pgmap v10500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:31.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:31.656+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:31.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:31.656+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:31.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:31.656+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:04:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:31 smithi033 ceph-mon[27845]: pgmap v10501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:31 smithi096 ceph-mon[32753]: pgmap v10501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:31 smithi194 ceph-mon[33017]: pgmap v10501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:33 smithi033 ceph-mon[27845]: pgmap v10502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18811.980933 secs 2024-04-26T17:04:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:33 smithi096 ceph-mon[32753]: pgmap v10502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18811.980933 secs 2024-04-26T17:04:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:33 smithi194 ceph-mon[33017]: pgmap v10502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18811.980933 secs 2024-04-26T17:04:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18814.821399 secs 2024-04-26T17:04:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18814.821399 secs 2024-04-26T17:04:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18814.821399 secs 2024-04-26T17:04:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:35 smithi194 ceph-mon[33017]: pgmap v10503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:36.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:35 smithi096 ceph-mon[32753]: pgmap v10503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:35 smithi033 ceph-mon[27845]: pgmap v10503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:37 smithi033 ceph-mon[27845]: pgmap v10504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:37 smithi096 ceph-mon[32753]: pgmap v10504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:37 smithi194 ceph-mon[33017]: pgmap v10504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18816.980996 secs 2024-04-26T17:04:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18816.980996 secs 2024-04-26T17:04:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18816.980996 secs 2024-04-26T17:04:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18819.821538 secs 2024-04-26T17:04:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:39 smithi033 ceph-mon[27845]: pgmap v10505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18819.821538 secs 2024-04-26T17:04:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:39 smithi096 ceph-mon[32753]: pgmap v10505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18819.821538 secs 2024-04-26T17:04:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:39 smithi194 ceph-mon[33017]: pgmap v10505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:41 smithi033 ceph-mon[27845]: pgmap v10506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:41 smithi096 ceph-mon[32753]: pgmap v10506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:41 smithi194 ceph-mon[33017]: pgmap v10506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:43 smithi033 ceph-mon[27845]: pgmap v10507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18821.981078 secs 2024-04-26T17:04:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18824.821681 secs 2024-04-26T17:04:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:43 smithi096 ceph-mon[32753]: pgmap v10507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18821.981078 secs 2024-04-26T17:04:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18824.821681 secs 2024-04-26T17:04:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:43 smithi194 ceph-mon[33017]: pgmap v10507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18821.981078 secs 2024-04-26T17:04:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18824.821681 secs 2024-04-26T17:04:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:45 smithi033 ceph-mon[27845]: pgmap v10508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:45 smithi096 ceph-mon[32753]: pgmap v10508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:45 smithi194 ceph-mon[33017]: pgmap v10508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:04:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:46.661+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:46.661+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:04:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:04:46.661+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:04:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:47 smithi096 ceph-mon[32753]: pgmap v10509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:04:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:47 smithi194 ceph-mon[33017]: pgmap v10509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:04:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:47 smithi033 ceph-mon[27845]: pgmap v10509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:04:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18826.981169 secs 2024-04-26T17:04:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:04:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:04:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:04:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18829.821781 secs 2024-04-26T17:04:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18826.981169 secs 2024-04-26T17:04:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:04:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:04:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:04:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18829.821781 secs 2024-04-26T17:04:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18826.981169 secs 2024-04-26T17:04:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:04:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:04:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:04:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18829.821781 secs 2024-04-26T17:04:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:49 smithi194 ceph-mon[33017]: pgmap v10510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:49 smithi194 ceph-mon[33017]: pgmap v10511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:49 smithi033 ceph-mon[27845]: pgmap v10510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:49 smithi033 ceph-mon[27845]: pgmap v10511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:49 smithi096 ceph-mon[32753]: pgmap v10510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:04:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:49 smithi096 ceph-mon[32753]: pgmap v10511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:51 smithi194 ceph-mon[33017]: pgmap v10512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:51 smithi033 ceph-mon[27845]: pgmap v10512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:51 smithi096 ceph-mon[32753]: pgmap v10512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:53 smithi194 ceph-mon[33017]: pgmap v10513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18831.981227 secs 2024-04-26T17:04:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18834.821899 secs 2024-04-26T17:04:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:53 smithi033 ceph-mon[27845]: pgmap v10513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18831.981227 secs 2024-04-26T17:04:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18834.821899 secs 2024-04-26T17:04:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:53 smithi096 ceph-mon[32753]: pgmap v10513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18831.981227 secs 2024-04-26T17:04:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18834.821899 secs 2024-04-26T17:04:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:55 smithi194 ceph-mon[33017]: pgmap v10514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:04:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:55 smithi033 ceph-mon[27845]: pgmap v10514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:04:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:55 smithi096 ceph-mon[32753]: pgmap v10514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:04:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:57 smithi194 ceph-mon[33017]: pgmap v10515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:57 smithi033 ceph-mon[27845]: pgmap v10515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:57 smithi096 ceph-mon[32753]: pgmap v10515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:04:59.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18836.981288 secs 2024-04-26T17:04:59.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18839.822035 secs 2024-04-26T17:04:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18836.981288 secs 2024-04-26T17:04:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18839.822035 secs 2024-04-26T17:04:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18836.981288 secs 2024-04-26T17:04:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18839.822035 secs 2024-04-26T17:05:00.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:04:59 smithi033 ceph-mon[27845]: pgmap v10516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:05:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:04:59 smithi194 ceph-mon[33017]: pgmap v10516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:05:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:04:59 smithi096 ceph-mon[32753]: pgmap v10516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:05:01.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:01.674+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:01.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:01.674+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:05:01.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:01.674+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:01 smithi194 ceph-mon[33017]: pgmap v10517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:01 smithi033 ceph-mon[27845]: pgmap v10517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:01 smithi096 ceph-mon[32753]: pgmap v10517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:03 smithi194 ceph-mon[33017]: pgmap v10518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18841.981367 secs 2024-04-26T17:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18844.822156 secs 2024-04-26T17:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:03 smithi033 ceph-mon[27845]: pgmap v10518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18841.981367 secs 2024-04-26T17:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18844.822156 secs 2024-04-26T17:05:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:03 smithi096 ceph-mon[32753]: pgmap v10518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18841.981367 secs 2024-04-26T17:05:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18844.822156 secs 2024-04-26T17:05:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:05 smithi194 ceph-mon[33017]: pgmap v10519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:05 smithi033 ceph-mon[27845]: pgmap v10519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:05 smithi096 ceph-mon[32753]: pgmap v10519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:07 smithi194 ceph-mon[33017]: pgmap v10520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:07 smithi033 ceph-mon[27845]: pgmap v10520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:07 smithi096 ceph-mon[32753]: pgmap v10520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18846.981475 secs 2024-04-26T17:05:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18849.822274 secs 2024-04-26T17:05:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18846.981475 secs 2024-04-26T17:05:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18849.822274 secs 2024-04-26T17:05:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18846.981475 secs 2024-04-26T17:05:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18849.822274 secs 2024-04-26T17:05:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:10 smithi194 ceph-mon[33017]: pgmap v10521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:10 smithi033 ceph-mon[27845]: pgmap v10521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:10 smithi096 ceph-mon[32753]: pgmap v10521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:12 smithi194 ceph-mon[33017]: pgmap v10522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:12 smithi033 ceph-mon[27845]: pgmap v10522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:12 smithi096 ceph-mon[32753]: pgmap v10522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:14 smithi194 ceph-mon[33017]: pgmap v10523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18851.981574 secs 2024-04-26T17:05:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18854.822397 secs 2024-04-26T17:05:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:14 smithi033 ceph-mon[27845]: pgmap v10523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18851.981574 secs 2024-04-26T17:05:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18854.822397 secs 2024-04-26T17:05:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:14 smithi096 ceph-mon[32753]: pgmap v10523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18851.981574 secs 2024-04-26T17:05:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18854.822397 secs 2024-04-26T17:05:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:16 smithi194 ceph-mon[33017]: pgmap v10524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:16 smithi033 ceph-mon[27845]: pgmap v10524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:16 smithi096 ceph-mon[32753]: pgmap v10524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:16.674+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:16.674+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:16.674+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:05:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:18 smithi194 ceph-mon[33017]: pgmap v10525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:18 smithi033 ceph-mon[27845]: pgmap v10525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:18 smithi096 ceph-mon[32753]: pgmap v10525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18856.981655 secs 2024-04-26T17:05:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18859.822518 secs 2024-04-26T17:05:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18856.981655 secs 2024-04-26T17:05:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18859.822518 secs 2024-04-26T17:05:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18856.981655 secs 2024-04-26T17:05:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18859.822518 secs 2024-04-26T17:05:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:20 smithi194 ceph-mon[33017]: pgmap v10526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:20 smithi033 ceph-mon[27845]: pgmap v10526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:20 smithi096 ceph-mon[32753]: pgmap v10526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:22 smithi194 ceph-mon[33017]: pgmap v10527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:22 smithi033 ceph-mon[27845]: pgmap v10527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:22 smithi096 ceph-mon[32753]: pgmap v10527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:23 smithi033 ceph-mon[27845]: pgmap v10528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:23 smithi096 ceph-mon[32753]: pgmap v10528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:23 smithi194 ceph-mon[33017]: pgmap v10528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18861.981741 secs 2024-04-26T17:05:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18864.822656 secs 2024-04-26T17:05:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18861.981741 secs 2024-04-26T17:05:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18864.822656 secs 2024-04-26T17:05:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18861.981741 secs 2024-04-26T17:05:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18864.822656 secs 2024-04-26T17:05:24.996 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:24.994+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T17:05:24.996 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:24.994+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:05:24.996 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:24.995+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T17:05:24.996 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:24.995+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:05:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:25 smithi033 ceph-mon[27845]: pgmap v10529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:25 smithi096 ceph-mon[32753]: pgmap v10529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:25 smithi194 ceph-mon[33017]: pgmap v10529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:27 smithi033 ceph-mon[27845]: pgmap v10530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:27 smithi096 ceph-mon[32753]: pgmap v10530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:27 smithi194 ceph-mon[33017]: pgmap v10530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18866.981841 secs 2024-04-26T17:05:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:29 smithi033 ceph-mon[27845]: pgmap v10531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18869.822781 secs 2024-04-26T17:05:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18866.981841 secs 2024-04-26T17:05:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:29 smithi096 ceph-mon[32753]: pgmap v10531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18869.822781 secs 2024-04-26T17:05:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18866.981841 secs 2024-04-26T17:05:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:29 smithi194 ceph-mon[33017]: pgmap v10531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18869.822781 secs 2024-04-26T17:05:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:31 smithi033 ceph-mon[27845]: pgmap v10532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:31.677+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:31.677+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:05:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:31.677+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:31 smithi096 ceph-mon[32753]: pgmap v10532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:31 smithi194 ceph-mon[33017]: pgmap v10532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:33 smithi033 ceph-mon[27845]: pgmap v10533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:33 smithi096 ceph-mon[32753]: pgmap v10533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:33 smithi194 ceph-mon[33017]: pgmap v10533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18871.981929 secs 2024-04-26T17:05:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18874.822850 secs 2024-04-26T17:05:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18871.981929 secs 2024-04-26T17:05:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18874.822850 secs 2024-04-26T17:05:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18871.981929 secs 2024-04-26T17:05:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18874.822850 secs 2024-04-26T17:05:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:35 smithi033 ceph-mon[27845]: pgmap v10534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:35 smithi096 ceph-mon[32753]: pgmap v10534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:35 smithi194 ceph-mon[33017]: pgmap v10534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:37 smithi033 ceph-mon[27845]: pgmap v10535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:37 smithi096 ceph-mon[32753]: pgmap v10535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:37 smithi194 ceph-mon[33017]: pgmap v10535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18876.982031 secs 2024-04-26T17:05:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:39 smithi033 ceph-mon[27845]: pgmap v10536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18879.822983 secs 2024-04-26T17:05:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18876.982031 secs 2024-04-26T17:05:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:39 smithi096 ceph-mon[32753]: pgmap v10536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18879.822983 secs 2024-04-26T17:05:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18876.982031 secs 2024-04-26T17:05:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:39 smithi194 ceph-mon[33017]: pgmap v10536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18879.822983 secs 2024-04-26T17:05:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:41 smithi033 ceph-mon[27845]: pgmap v10537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:41 smithi096 ceph-mon[32753]: pgmap v10537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:41 smithi194 ceph-mon[33017]: pgmap v10537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:05:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:43 smithi033 ceph-mon[27845]: pgmap v10538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:43 smithi096 ceph-mon[32753]: pgmap v10538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:43 smithi194 ceph-mon[33017]: pgmap v10538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18881.982133 secs 2024-04-26T17:05:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18884.823092 secs 2024-04-26T17:05:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18881.982133 secs 2024-04-26T17:05:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18884.823092 secs 2024-04-26T17:05:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18881.982133 secs 2024-04-26T17:05:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18884.823092 secs 2024-04-26T17:05:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:45 smithi194 ceph-mon[33017]: pgmap v10539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:45 smithi033 ceph-mon[27845]: pgmap v10539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:45 smithi096 ceph-mon[32753]: pgmap v10539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:46.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:46.693+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:46.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:46.693+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:05:46.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:05:46.694+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:05:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:47 smithi033 ceph-mon[27845]: pgmap v10540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:47 smithi194 ceph-mon[33017]: pgmap v10540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:47.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:47 smithi096 ceph-mon[32753]: pgmap v10540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18886.982193 secs 2024-04-26T17:05:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:49 smithi194 ceph-mon[33017]: pgmap v10541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18889.823208 secs 2024-04-26T17:05:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:05:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18886.982193 secs 2024-04-26T17:05:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:49 smithi033 ceph-mon[27845]: pgmap v10541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18889.823208 secs 2024-04-26T17:05:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:05:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18886.982193 secs 2024-04-26T17:05:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:49 smithi096 ceph-mon[32753]: pgmap v10541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:05:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18889.823208 secs 2024-04-26T17:05:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:05:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:05:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:05:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:50 smithi194 ceph-mon[33017]: pgmap v10542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:05:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:50 smithi194 ceph-mon[33017]: pgmap v10543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:05:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:05:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:05:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:50 smithi033 ceph-mon[27845]: pgmap v10542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:05:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:50 smithi033 ceph-mon[27845]: pgmap v10543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:05:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:05:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:05:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:50 smithi096 ceph-mon[32753]: pgmap v10542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:05:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:50 smithi096 ceph-mon[32753]: pgmap v10543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:05:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:52 smithi033 ceph-mon[27845]: pgmap v10544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:52 smithi096 ceph-mon[32753]: pgmap v10544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:52 smithi194 ceph-mon[33017]: pgmap v10544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18891.982252 secs 2024-04-26T17:05:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18891.982252 secs 2024-04-26T17:05:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18891.982252 secs 2024-04-26T17:05:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18894.823302 secs 2024-04-26T17:05:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:54 smithi194 ceph-mon[33017]: pgmap v10545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:05:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18894.823302 secs 2024-04-26T17:05:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:54 smithi033 ceph-mon[27845]: pgmap v10545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:05:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18894.823302 secs 2024-04-26T17:05:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:54 smithi096 ceph-mon[32753]: pgmap v10545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:05:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:56 smithi033 ceph-mon[27845]: pgmap v10546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:56 smithi096 ceph-mon[32753]: pgmap v10546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:56 smithi194 ceph-mon[33017]: pgmap v10546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:58.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:58 smithi194 ceph-mon[33017]: pgmap v10547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:58 smithi033 ceph-mon[27845]: pgmap v10547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:58 smithi096 ceph-mon[32753]: pgmap v10547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:05:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18896.982335 secs 2024-04-26T17:05:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18899.823425 secs 2024-04-26T17:05:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18896.982335 secs 2024-04-26T17:05:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18899.823425 secs 2024-04-26T17:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18896.982335 secs 2024-04-26T17:05:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18899.823425 secs 2024-04-26T17:06:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:00 smithi033 ceph-mon[27845]: pgmap v10548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:06:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:00 smithi096 ceph-mon[32753]: pgmap v10548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:06:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:00 smithi194 ceph-mon[33017]: pgmap v10548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:06:01.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:01.696+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:01.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:01.696+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:01.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:01.696+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:06:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:02 smithi033 ceph-mon[27845]: pgmap v10549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:02 smithi096 ceph-mon[32753]: pgmap v10549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:02 smithi194 ceph-mon[33017]: pgmap v10549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18901.982393 secs 2024-04-26T17:06:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18901.982393 secs 2024-04-26T17:06:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18901.982393 secs 2024-04-26T17:06:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18904.823541 secs 2024-04-26T17:06:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:04 smithi194 ceph-mon[33017]: pgmap v10550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18904.823541 secs 2024-04-26T17:06:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:04 smithi033 ceph-mon[27845]: pgmap v10550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18904.823541 secs 2024-04-26T17:06:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:04 smithi096 ceph-mon[32753]: pgmap v10550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:06 smithi033 ceph-mon[27845]: pgmap v10551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:06 smithi096 ceph-mon[32753]: pgmap v10551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:06 smithi194 ceph-mon[33017]: pgmap v10551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:08 smithi033 ceph-mon[27845]: pgmap v10552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:08 smithi096 ceph-mon[32753]: pgmap v10552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:08 smithi194 ceph-mon[33017]: pgmap v10552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18906.982477 secs 2024-04-26T17:06:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18909.823656 secs 2024-04-26T17:06:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18906.982477 secs 2024-04-26T17:06:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18909.823656 secs 2024-04-26T17:06:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18906.982477 secs 2024-04-26T17:06:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18909.823656 secs 2024-04-26T17:06:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:10 smithi033 ceph-mon[27845]: pgmap v10553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:10 smithi096 ceph-mon[32753]: pgmap v10553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:10 smithi194 ceph-mon[33017]: pgmap v10553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:12 smithi033 ceph-mon[27845]: pgmap v10554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:12 smithi096 ceph-mon[32753]: pgmap v10554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:12 smithi194 ceph-mon[33017]: pgmap v10554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18911.982547 secs 2024-04-26T17:06:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18911.982547 secs 2024-04-26T17:06:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18911.982547 secs 2024-04-26T17:06:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18914.823778 secs 2024-04-26T17:06:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:14 smithi194 ceph-mon[33017]: pgmap v10555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18914.823778 secs 2024-04-26T17:06:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:14 smithi033 ceph-mon[27845]: pgmap v10555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18914.823778 secs 2024-04-26T17:06:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:14 smithi096 ceph-mon[32753]: pgmap v10555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:16 smithi033 ceph-mon[27845]: pgmap v10556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:16 smithi096 ceph-mon[32753]: pgmap v10556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:16 smithi194 ceph-mon[33017]: pgmap v10556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:16.711+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:16.711+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:06:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:16.711+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:18 smithi033 ceph-mon[27845]: pgmap v10557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:18 smithi096 ceph-mon[32753]: pgmap v10557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:18 smithi194 ceph-mon[33017]: pgmap v10557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18916.982626 secs 2024-04-26T17:06:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18919.823895 secs 2024-04-26T17:06:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18916.982626 secs 2024-04-26T17:06:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18919.823895 secs 2024-04-26T17:06:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18916.982626 secs 2024-04-26T17:06:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18919.823895 secs 2024-04-26T17:06:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:20 smithi033 ceph-mon[27845]: pgmap v10558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:20 smithi096 ceph-mon[32753]: pgmap v10558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:20 smithi194 ceph-mon[33017]: pgmap v10558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:22 smithi033 ceph-mon[27845]: pgmap v10559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:22 smithi096 ceph-mon[32753]: pgmap v10559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:22 smithi194 ceph-mon[33017]: pgmap v10559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18921.982702 secs 2024-04-26T17:06:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18921.982702 secs 2024-04-26T17:06:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18921.982702 secs 2024-04-26T17:06:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18924.824042 secs 2024-04-26T17:06:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:24 smithi194 ceph-mon[33017]: pgmap v10560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18924.824042 secs 2024-04-26T17:06:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:24 smithi033 ceph-mon[27845]: pgmap v10560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18924.824042 secs 2024-04-26T17:06:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:24 smithi096 ceph-mon[32753]: pgmap v10560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:26 smithi033 ceph-mon[27845]: pgmap v10561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:26 smithi096 ceph-mon[32753]: pgmap v10561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:26 smithi194 ceph-mon[33017]: pgmap v10561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:28.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:28 smithi194 ceph-mon[33017]: pgmap v10562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:28 smithi033 ceph-mon[27845]: pgmap v10562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:28 smithi096 ceph-mon[32753]: pgmap v10562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18926.982787 secs 2024-04-26T17:06:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18929.824195 secs 2024-04-26T17:06:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18926.982787 secs 2024-04-26T17:06:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18929.824195 secs 2024-04-26T17:06:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18926.982787 secs 2024-04-26T17:06:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18929.824195 secs 2024-04-26T17:06:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:30 smithi033 ceph-mon[27845]: pgmap v10563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:30 smithi096 ceph-mon[32753]: pgmap v10563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:30 smithi194 ceph-mon[33017]: pgmap v10563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:31.724+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:31.724+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:06:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:31.724+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:32 smithi033 ceph-mon[27845]: pgmap v10564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:32 smithi096 ceph-mon[32753]: pgmap v10564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:32 smithi194 ceph-mon[33017]: pgmap v10564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18931.982870 secs 2024-04-26T17:06:33.904 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18931.982870 secs 2024-04-26T17:06:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18931.982870 secs 2024-04-26T17:06:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18934.824317 secs 2024-04-26T17:06:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:34 smithi194 ceph-mon[33017]: pgmap v10565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18934.824317 secs 2024-04-26T17:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:34 smithi033 ceph-mon[27845]: pgmap v10565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18934.824317 secs 2024-04-26T17:06:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:34 smithi096 ceph-mon[32753]: pgmap v10565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:36 smithi033 ceph-mon[27845]: pgmap v10566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:36 smithi096 ceph-mon[32753]: pgmap v10566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:36 smithi194 ceph-mon[33017]: pgmap v10566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:38 smithi033 ceph-mon[27845]: pgmap v10567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:38 smithi096 ceph-mon[32753]: pgmap v10567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:38 smithi194 ceph-mon[33017]: pgmap v10567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18936.982953 secs 2024-04-26T17:06:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18939.824433 secs 2024-04-26T17:06:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18936.982953 secs 2024-04-26T17:06:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18939.824433 secs 2024-04-26T17:06:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18936.982953 secs 2024-04-26T17:06:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18939.824433 secs 2024-04-26T17:06:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:40 smithi033 ceph-mon[27845]: pgmap v10568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:40 smithi096 ceph-mon[32753]: pgmap v10568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:40 smithi194 ceph-mon[33017]: pgmap v10568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:42 smithi033 ceph-mon[27845]: pgmap v10569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:42 smithi096 ceph-mon[32753]: pgmap v10569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:42 smithi194 ceph-mon[33017]: pgmap v10569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18941.983018 secs 2024-04-26T17:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18941.983018 secs 2024-04-26T17:06:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18941.983018 secs 2024-04-26T17:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18944.824534 secs 2024-04-26T17:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:44 smithi194 ceph-mon[33017]: pgmap v10570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18944.824534 secs 2024-04-26T17:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:44 smithi033 ceph-mon[27845]: pgmap v10570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18944.824534 secs 2024-04-26T17:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:44 smithi096 ceph-mon[32753]: pgmap v10570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:46 smithi033 ceph-mon[27845]: pgmap v10571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:46 smithi096 ceph-mon[32753]: pgmap v10571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:46 smithi194 ceph-mon[33017]: pgmap v10571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:46.729+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:46.730+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:06:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:06:46.730+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:06:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:48 smithi033 ceph-mon[27845]: pgmap v10572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:48 smithi096 ceph-mon[32753]: pgmap v10572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:48 smithi194 ceph-mon[33017]: pgmap v10572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18946.983078 secs 2024-04-26T17:06:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18949.824648 secs 2024-04-26T17:06:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18946.983078 secs 2024-04-26T17:06:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18949.824648 secs 2024-04-26T17:06:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18946.983078 secs 2024-04-26T17:06:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18949.824648 secs 2024-04-26T17:06:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:50 smithi033 ceph-mon[27845]: pgmap v10573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:50 smithi096 ceph-mon[32753]: pgmap v10573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:50 smithi194 ceph-mon[33017]: pgmap v10573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:06:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:06:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:06:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:06:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:06:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:06:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:52 smithi033 ceph-mon[27845]: pgmap v10574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:52 smithi033 ceph-mon[27845]: pgmap v10575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:06:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:06:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:06:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:52 smithi096 ceph-mon[32753]: pgmap v10574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:52 smithi096 ceph-mon[32753]: pgmap v10575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:06:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:06:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:06:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:52 smithi194 ceph-mon[33017]: pgmap v10574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:06:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:52 smithi194 ceph-mon[33017]: pgmap v10575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:06:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18951.983135 secs 2024-04-26T17:06:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18951.983135 secs 2024-04-26T17:06:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18951.983135 secs 2024-04-26T17:06:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18954.824776 secs 2024-04-26T17:06:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:54 smithi194 ceph-mon[33017]: pgmap v10576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18954.824776 secs 2024-04-26T17:06:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:54 smithi033 ceph-mon[27845]: pgmap v10576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18954.824776 secs 2024-04-26T17:06:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:54 smithi096 ceph-mon[32753]: pgmap v10576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:56 smithi194 ceph-mon[33017]: pgmap v10577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:56 smithi033 ceph-mon[27845]: pgmap v10577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:56 smithi096 ceph-mon[32753]: pgmap v10577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:58 smithi194 ceph-mon[33017]: pgmap v10578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18956.983220 secs 2024-04-26T17:06:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:58 smithi033 ceph-mon[27845]: pgmap v10578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18956.983220 secs 2024-04-26T17:06:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:58 smithi096 ceph-mon[32753]: pgmap v10578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:06:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18956.983220 secs 2024-04-26T17:07:00.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18959.824910 secs 2024-04-26T17:07:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18959.824910 secs 2024-04-26T17:07:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18959.824910 secs 2024-04-26T17:07:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:00 smithi194 ceph-mon[33017]: pgmap v10579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:00 smithi033 ceph-mon[27845]: pgmap v10579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:00 smithi096 ceph-mon[32753]: pgmap v10579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:01.735+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:01.735+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:01.735+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:07:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:02 smithi194 ceph-mon[33017]: pgmap v10580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:07:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:02 smithi033 ceph-mon[27845]: pgmap v10580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:07:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:02 smithi096 ceph-mon[32753]: pgmap v10580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:07:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18961.983308 secs 2024-04-26T17:07:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18961.983308 secs 2024-04-26T17:07:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18961.983308 secs 2024-04-26T17:07:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18964.825037 secs 2024-04-26T17:07:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:04 smithi194 ceph-mon[33017]: pgmap v10581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:05.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18964.825037 secs 2024-04-26T17:07:05.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:04 smithi033 ceph-mon[27845]: pgmap v10581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18964.825037 secs 2024-04-26T17:07:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:04 smithi096 ceph-mon[32753]: pgmap v10581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:06 smithi194 ceph-mon[33017]: pgmap v10582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:06 smithi033 ceph-mon[27845]: pgmap v10582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:06 smithi096 ceph-mon[32753]: pgmap v10582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:08 smithi194 ceph-mon[33017]: pgmap v10583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18966.983392 secs 2024-04-26T17:07:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:08 smithi033 ceph-mon[27845]: pgmap v10583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18966.983392 secs 2024-04-26T17:07:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:08 smithi096 ceph-mon[32753]: pgmap v10583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18966.983392 secs 2024-04-26T17:07:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18969.825172 secs 2024-04-26T17:07:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18969.825172 secs 2024-04-26T17:07:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18969.825172 secs 2024-04-26T17:07:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:10 smithi194 ceph-mon[33017]: pgmap v10584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:10 smithi033 ceph-mon[27845]: pgmap v10584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:10 smithi096 ceph-mon[32753]: pgmap v10584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:12 smithi194 ceph-mon[33017]: pgmap v10585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:12 smithi033 ceph-mon[27845]: pgmap v10585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:12 smithi096 ceph-mon[32753]: pgmap v10585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:14.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18971.983476 secs 2024-04-26T17:07:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18971.983476 secs 2024-04-26T17:07:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18971.983476 secs 2024-04-26T17:07:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18974.825301 secs 2024-04-26T17:07:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:14 smithi194 ceph-mon[33017]: pgmap v10586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18974.825301 secs 2024-04-26T17:07:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:14 smithi033 ceph-mon[27845]: pgmap v10586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18974.825301 secs 2024-04-26T17:07:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:14 smithi096 ceph-mon[32753]: pgmap v10586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:16.749+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:16.749+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:07:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:16.749+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:16 smithi194 ceph-mon[33017]: pgmap v10587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:16 smithi033 ceph-mon[27845]: pgmap v10587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:16 smithi096 ceph-mon[32753]: pgmap v10587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:18 smithi194 ceph-mon[33017]: pgmap v10588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18976.983556 secs 2024-04-26T17:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:18 smithi033 ceph-mon[27845]: pgmap v10588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18976.983556 secs 2024-04-26T17:07:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:18 smithi096 ceph-mon[32753]: pgmap v10588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18976.983556 secs 2024-04-26T17:07:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18979.825428 secs 2024-04-26T17:07:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18979.825428 secs 2024-04-26T17:07:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18979.825428 secs 2024-04-26T17:07:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:20 smithi194 ceph-mon[33017]: pgmap v10589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:20 smithi033 ceph-mon[27845]: pgmap v10589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:20 smithi096 ceph-mon[32753]: pgmap v10589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:22 smithi194 ceph-mon[33017]: pgmap v10590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:22 smithi033 ceph-mon[27845]: pgmap v10590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:22 smithi096 ceph-mon[32753]: pgmap v10590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18981.983636 secs 2024-04-26T17:07:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18981.983636 secs 2024-04-26T17:07:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18981.983636 secs 2024-04-26T17:07:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18984.825489 secs 2024-04-26T17:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:24 smithi194 ceph-mon[33017]: pgmap v10591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18984.825489 secs 2024-04-26T17:07:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:24 smithi033 ceph-mon[27845]: pgmap v10591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18984.825489 secs 2024-04-26T17:07:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:24 smithi096 ceph-mon[32753]: pgmap v10591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:26 smithi194 ceph-mon[33017]: pgmap v10592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:26 smithi033 ceph-mon[27845]: pgmap v10592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:26 smithi096 ceph-mon[32753]: pgmap v10592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:28 smithi194 ceph-mon[33017]: pgmap v10593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18986.983722 secs 2024-04-26T17:07:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:28 smithi033 ceph-mon[27845]: pgmap v10593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18986.983722 secs 2024-04-26T17:07:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:28 smithi096 ceph-mon[32753]: pgmap v10593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18986.983722 secs 2024-04-26T17:07:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18989.825606 secs 2024-04-26T17:07:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18989.825606 secs 2024-04-26T17:07:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18989.825606 secs 2024-04-26T17:07:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:30 smithi194 ceph-mon[33017]: pgmap v10594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:30 smithi033 ceph-mon[27845]: pgmap v10594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:30 smithi096 ceph-mon[32753]: pgmap v10594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:31.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:31.751+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:31.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:31.751+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:31.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:31.751+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:07:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:32 smithi033 ceph-mon[27845]: pgmap v10595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:32 smithi096 ceph-mon[32753]: pgmap v10595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:32 smithi194 ceph-mon[33017]: pgmap v10595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18991.983804 secs 2024-04-26T17:07:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18991.983804 secs 2024-04-26T17:07:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18991.983804 secs 2024-04-26T17:07:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18994.825724 secs 2024-04-26T17:07:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:34 smithi033 ceph-mon[27845]: pgmap v10596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18994.825724 secs 2024-04-26T17:07:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:34 smithi096 ceph-mon[32753]: pgmap v10596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18994.825724 secs 2024-04-26T17:07:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:34 smithi194 ceph-mon[33017]: pgmap v10596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:36 smithi033 ceph-mon[27845]: pgmap v10597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:36 smithi096 ceph-mon[32753]: pgmap v10597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:36 smithi194 ceph-mon[33017]: pgmap v10597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:38 smithi033 ceph-mon[27845]: pgmap v10598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:39.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 18996.983881 secs 2024-04-26T17:07:39.304 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:38 smithi096 ceph-mon[32753]: pgmap v10598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:39.305 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 18996.983881 secs 2024-04-26T17:07:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:38 smithi194 ceph-mon[33017]: pgmap v10598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 18996.983881 secs 2024-04-26T17:07:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 18999.825864 secs 2024-04-26T17:07:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 18999.825864 secs 2024-04-26T17:07:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 18999.825864 secs 2024-04-26T17:07:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:40 smithi033 ceph-mon[27845]: pgmap v10599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:40 smithi096 ceph-mon[32753]: pgmap v10599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:40 smithi194 ceph-mon[33017]: pgmap v10599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:42 smithi033 ceph-mon[27845]: pgmap v10600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:42 smithi096 ceph-mon[32753]: pgmap v10600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:42 smithi194 ceph-mon[33017]: pgmap v10600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19001.983960 secs 2024-04-26T17:07:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19001.983960 secs 2024-04-26T17:07:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19001.983960 secs 2024-04-26T17:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19004.826006 secs 2024-04-26T17:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:44 smithi033 ceph-mon[27845]: pgmap v10601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19004.826006 secs 2024-04-26T17:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:44 smithi096 ceph-mon[32753]: pgmap v10601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19004.826006 secs 2024-04-26T17:07:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:44 smithi194 ceph-mon[33017]: pgmap v10601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:46.754+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:46.754+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:07:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:07:46.754+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:07:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:46 smithi033 ceph-mon[27845]: pgmap v10602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:46 smithi096 ceph-mon[32753]: pgmap v10602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:46 smithi194 ceph-mon[33017]: pgmap v10602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:48 smithi033 ceph-mon[27845]: pgmap v10603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19006.984042 secs 2024-04-26T17:07:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:48 smithi096 ceph-mon[32753]: pgmap v10603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19006.984042 secs 2024-04-26T17:07:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:48 smithi194 ceph-mon[33017]: pgmap v10603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19006.984042 secs 2024-04-26T17:07:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19009.826147 secs 2024-04-26T17:07:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19009.826147 secs 2024-04-26T17:07:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19009.826147 secs 2024-04-26T17:07:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:50 smithi096 ceph-mon[32753]: pgmap v10604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:51.215 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:50 smithi033 ceph-mon[27845]: pgmap v10604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:50 smithi194 ceph-mon[33017]: pgmap v10604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:07:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:52 smithi194 ceph-mon[33017]: pgmap v10605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:07:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:52 smithi033 ceph-mon[27845]: pgmap v10605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:07:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:52 smithi096 ceph-mon[32753]: pgmap v10605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19011.984134 secs 2024-04-26T17:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:07:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19011.984134 secs 2024-04-26T17:07:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:07:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:07:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:07:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19011.984134 secs 2024-04-26T17:07:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:07:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:07:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:07:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:54 smithi194 ceph-mon[33017]: pgmap v10606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:54 smithi194 ceph-mon[33017]: pgmap v10607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19014.826215 secs 2024-04-26T17:07:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:54 smithi033 ceph-mon[27845]: pgmap v10606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:54 smithi033 ceph-mon[27845]: pgmap v10607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19014.826215 secs 2024-04-26T17:07:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:54 smithi096 ceph-mon[32753]: pgmap v10606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:07:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:54 smithi096 ceph-mon[32753]: pgmap v10607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19014.826215 secs 2024-04-26T17:07:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:56 smithi194 ceph-mon[33017]: pgmap v10608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:56 smithi033 ceph-mon[27845]: pgmap v10608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:56 smithi096 ceph-mon[32753]: pgmap v10608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:58 smithi194 ceph-mon[33017]: pgmap v10609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19016.984215 secs 2024-04-26T17:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:07:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19019.826279 secs 2024-04-26T17:07:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:58 smithi033 ceph-mon[27845]: pgmap v10609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19016.984215 secs 2024-04-26T17:07:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:07:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19019.826279 secs 2024-04-26T17:07:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:58 smithi096 ceph-mon[32753]: pgmap v10609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:07:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19016.984215 secs 2024-04-26T17:07:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:07:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19019.826279 secs 2024-04-26T17:08:01.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:00 smithi096 ceph-mon[32753]: pgmap v10610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:00 smithi194 ceph-mon[33017]: pgmap v10610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:00 smithi033 ceph-mon[27845]: pgmap v10610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:01.770+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:01.770+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:01.770+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:08:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:02 smithi194 ceph-mon[33017]: pgmap v10611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:02 smithi033 ceph-mon[27845]: pgmap v10611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:02 smithi096 ceph-mon[32753]: pgmap v10611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19021.984304 secs 2024-04-26T17:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19024.826357 secs 2024-04-26T17:08:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19021.984304 secs 2024-04-26T17:08:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19024.826357 secs 2024-04-26T17:08:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19021.984304 secs 2024-04-26T17:08:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19024.826357 secs 2024-04-26T17:08:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:04 smithi194 ceph-mon[33017]: pgmap v10612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:08:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:04 smithi033 ceph-mon[27845]: pgmap v10612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:08:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:04 smithi096 ceph-mon[32753]: pgmap v10612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:08:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:06 smithi194 ceph-mon[33017]: pgmap v10613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:08:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:06 smithi033 ceph-mon[27845]: pgmap v10613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:08:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:06 smithi096 ceph-mon[32753]: pgmap v10613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:08:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:08 smithi194 ceph-mon[33017]: pgmap v10614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19026.984399 secs 2024-04-26T17:08:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19029.826456 secs 2024-04-26T17:08:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:08 smithi033 ceph-mon[27845]: pgmap v10614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19026.984399 secs 2024-04-26T17:08:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19029.826456 secs 2024-04-26T17:08:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:08 smithi096 ceph-mon[32753]: pgmap v10614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19026.984399 secs 2024-04-26T17:08:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19029.826456 secs 2024-04-26T17:08:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:10 smithi194 ceph-mon[33017]: pgmap v10615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:08:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:10 smithi033 ceph-mon[27845]: pgmap v10615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:08:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:10 smithi096 ceph-mon[32753]: pgmap v10615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:08:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:13 smithi194 ceph-mon[33017]: pgmap v10616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:08:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:13 smithi033 ceph-mon[27845]: pgmap v10616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:08:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:13 smithi096 ceph-mon[32753]: pgmap v10616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:08:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19031.984495 secs 2024-04-26T17:08:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19034.826554 secs 2024-04-26T17:08:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19031.984495 secs 2024-04-26T17:08:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19034.826554 secs 2024-04-26T17:08:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19031.984495 secs 2024-04-26T17:08:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19034.826554 secs 2024-04-26T17:08:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:15 smithi194 ceph-mon[33017]: pgmap v10617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:15 smithi033 ceph-mon[27845]: pgmap v10617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:15 smithi096 ceph-mon[32753]: pgmap v10617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:08:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:16.777+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:16.778+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:16.778+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:08:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:17 smithi194 ceph-mon[33017]: pgmap v10618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:08:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:17 smithi033 ceph-mon[27845]: pgmap v10618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:08:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:17 smithi096 ceph-mon[32753]: pgmap v10618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:19 smithi194 ceph-mon[33017]: pgmap v10619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19036.984556 secs 2024-04-26T17:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19039.826673 secs 2024-04-26T17:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:19 smithi033 ceph-mon[27845]: pgmap v10619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19036.984556 secs 2024-04-26T17:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19039.826673 secs 2024-04-26T17:08:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:19 smithi096 ceph-mon[32753]: pgmap v10619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19036.984556 secs 2024-04-26T17:08:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19039.826673 secs 2024-04-26T17:08:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:20 smithi033 ceph-mon[27845]: pgmap v10620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:20 smithi096 ceph-mon[32753]: pgmap v10620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:20 smithi194 ceph-mon[33017]: pgmap v10620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:22 smithi033 ceph-mon[27845]: pgmap v10621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:22 smithi096 ceph-mon[32753]: pgmap v10621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:22 smithi194 ceph-mon[33017]: pgmap v10621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19041.984640 secs 2024-04-26T17:08:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:24 smithi033 ceph-mon[27845]: pgmap v10622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19044.826784 secs 2024-04-26T17:08:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19041.984640 secs 2024-04-26T17:08:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:24 smithi096 ceph-mon[32753]: pgmap v10622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19044.826784 secs 2024-04-26T17:08:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19041.984640 secs 2024-04-26T17:08:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:24 smithi194 ceph-mon[33017]: pgmap v10622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19044.826784 secs 2024-04-26T17:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:25.027+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T17:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:25.027+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:25.027+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T17:08:25.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:25.027+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:08:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:26 smithi033 ceph-mon[27845]: pgmap v10623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:26 smithi096 ceph-mon[32753]: pgmap v10623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:26 smithi194 ceph-mon[33017]: pgmap v10623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:28 smithi033 ceph-mon[27845]: pgmap v10624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:28 smithi096 ceph-mon[32753]: pgmap v10624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:28 smithi194 ceph-mon[33017]: pgmap v10624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19046.984726 secs 2024-04-26T17:08:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19049.826847 secs 2024-04-26T17:08:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19046.984726 secs 2024-04-26T17:08:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19049.826847 secs 2024-04-26T17:08:29.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19046.984726 secs 2024-04-26T17:08:29.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19049.826847 secs 2024-04-26T17:08:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:30 smithi033 ceph-mon[27845]: pgmap v10625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:30 smithi096 ceph-mon[32753]: pgmap v10625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:30 smithi194 ceph-mon[33017]: pgmap v10625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:31.793+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:31.793+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:31.793+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:08:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:32 smithi096 ceph-mon[32753]: pgmap v10626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:32 smithi194 ceph-mon[33017]: pgmap v10626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:32 smithi033 ceph-mon[27845]: pgmap v10626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19051.984809 secs 2024-04-26T17:08:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:34 smithi096 ceph-mon[32753]: pgmap v10627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19054.826965 secs 2024-04-26T17:08:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19051.984809 secs 2024-04-26T17:08:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:34 smithi194 ceph-mon[33017]: pgmap v10627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19054.826965 secs 2024-04-26T17:08:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19051.984809 secs 2024-04-26T17:08:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:34 smithi033 ceph-mon[27845]: pgmap v10627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19054.826965 secs 2024-04-26T17:08:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:36 smithi194 ceph-mon[33017]: pgmap v10628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:36 smithi033 ceph-mon[27845]: pgmap v10628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:36 smithi096 ceph-mon[32753]: pgmap v10628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:38 smithi033 ceph-mon[27845]: pgmap v10629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:38 smithi194 ceph-mon[33017]: pgmap v10629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:38 smithi096 ceph-mon[32753]: pgmap v10629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19056.984890 secs 2024-04-26T17:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19059.827082 secs 2024-04-26T17:08:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19056.984890 secs 2024-04-26T17:08:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19059.827082 secs 2024-04-26T17:08:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19056.984890 secs 2024-04-26T17:08:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19059.827082 secs 2024-04-26T17:08:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:40 smithi194 ceph-mon[33017]: pgmap v10630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:40 smithi033 ceph-mon[27845]: pgmap v10630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:40 smithi096 ceph-mon[32753]: pgmap v10630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:42 smithi194 ceph-mon[33017]: pgmap v10631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:42 smithi033 ceph-mon[27845]: pgmap v10631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:42 smithi096 ceph-mon[32753]: pgmap v10631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:44.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19061.984995 secs 2024-04-26T17:08:44.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:44 smithi033 ceph-mon[27845]: pgmap v10632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:44.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19064.827200 secs 2024-04-26T17:08:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19061.984995 secs 2024-04-26T17:08:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:44 smithi194 ceph-mon[33017]: pgmap v10632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19064.827200 secs 2024-04-26T17:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19061.984995 secs 2024-04-26T17:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:44 smithi096 ceph-mon[32753]: pgmap v10632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19064.827200 secs 2024-04-26T17:08:46.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:46.801+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:46.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:46.802+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:08:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:08:46.802+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:08:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:46 smithi194 ceph-mon[33017]: pgmap v10633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:46 smithi033 ceph-mon[27845]: pgmap v10633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:46 smithi096 ceph-mon[32753]: pgmap v10633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:08:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:48 smithi194 ceph-mon[33017]: pgmap v10634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:48 smithi033 ceph-mon[27845]: pgmap v10634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:48 smithi096 ceph-mon[32753]: pgmap v10634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19066.985085 secs 2024-04-26T17:08:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19069.827279 secs 2024-04-26T17:08:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19066.985085 secs 2024-04-26T17:08:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19069.827279 secs 2024-04-26T17:08:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19066.985085 secs 2024-04-26T17:08:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19069.827279 secs 2024-04-26T17:08:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:50 smithi194 ceph-mon[33017]: pgmap v10635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:50 smithi033 ceph-mon[27845]: pgmap v10635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:50 smithi096 ceph-mon[32753]: pgmap v10635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:52 smithi194 ceph-mon[33017]: pgmap v10636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:52 smithi033 ceph-mon[27845]: pgmap v10636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:52 smithi096 ceph-mon[32753]: pgmap v10636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19071.985185 secs 2024-04-26T17:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:54 smithi194 ceph-mon[33017]: pgmap v10637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19074.827402 secs 2024-04-26T17:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19071.985185 secs 2024-04-26T17:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:54 smithi033 ceph-mon[27845]: pgmap v10637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19074.827402 secs 2024-04-26T17:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:08:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19071.985185 secs 2024-04-26T17:08:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:54 smithi096 ceph-mon[32753]: pgmap v10637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:08:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19074.827402 secs 2024-04-26T17:08:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:08:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:08:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:08:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:55 smithi194 ceph-mon[33017]: pgmap v10638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:08:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:55 smithi194 ceph-mon[33017]: pgmap v10639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:08:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:08:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:08:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:55 smithi033 ceph-mon[27845]: pgmap v10638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:08:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:55 smithi033 ceph-mon[27845]: pgmap v10639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:08:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:08:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:08:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:55 smithi096 ceph-mon[32753]: pgmap v10638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:08:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:55 smithi096 ceph-mon[32753]: pgmap v10639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:08:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:57 smithi033 ceph-mon[27845]: pgmap v10640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:57 smithi096 ceph-mon[32753]: pgmap v10640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:57 smithi194 ceph-mon[33017]: pgmap v10640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:08:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19076.985290 secs 2024-04-26T17:08:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19076.985290 secs 2024-04-26T17:08:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19076.985290 secs 2024-04-26T17:08:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19079.827554 secs 2024-04-26T17:08:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:08:59 smithi194 ceph-mon[33017]: pgmap v10641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:08:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19079.827554 secs 2024-04-26T17:08:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:08:59 smithi033 ceph-mon[27845]: pgmap v10641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:08:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19079.827554 secs 2024-04-26T17:08:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:08:59 smithi096 ceph-mon[32753]: pgmap v10641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:09:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:01 smithi033 ceph-mon[27845]: pgmap v10642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:01 smithi096 ceph-mon[32753]: pgmap v10642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:01 smithi194 ceph-mon[33017]: pgmap v10642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:01.803+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:01.803+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:01.803+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:09:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:03 smithi033 ceph-mon[27845]: pgmap v10643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:03 smithi096 ceph-mon[32753]: pgmap v10643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:03 smithi194 ceph-mon[33017]: pgmap v10643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19081.985372 secs 2024-04-26T17:09:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19084.827672 secs 2024-04-26T17:09:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19081.985372 secs 2024-04-26T17:09:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19084.827672 secs 2024-04-26T17:09:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19081.985372 secs 2024-04-26T17:09:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19084.827672 secs 2024-04-26T17:09:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:05 smithi033 ceph-mon[27845]: pgmap v10644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:09:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:05 smithi096 ceph-mon[32753]: pgmap v10644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:09:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:05 smithi194 ceph-mon[33017]: pgmap v10644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:09:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:07 smithi033 ceph-mon[27845]: pgmap v10645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:07 smithi096 ceph-mon[32753]: pgmap v10645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:07 smithi194 ceph-mon[33017]: pgmap v10645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19086.985456 secs 2024-04-26T17:09:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19086.985456 secs 2024-04-26T17:09:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19086.985456 secs 2024-04-26T17:09:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19089.827783 secs 2024-04-26T17:09:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:09 smithi194 ceph-mon[33017]: pgmap v10646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19089.827783 secs 2024-04-26T17:09:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:09 smithi033 ceph-mon[27845]: pgmap v10646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19089.827783 secs 2024-04-26T17:09:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:09 smithi096 ceph-mon[32753]: pgmap v10646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:11 smithi033 ceph-mon[27845]: pgmap v10647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:11 smithi096 ceph-mon[32753]: pgmap v10647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:11 smithi194 ceph-mon[33017]: pgmap v10647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:13 smithi033 ceph-mon[27845]: pgmap v10648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:13 smithi096 ceph-mon[32753]: pgmap v10648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:13 smithi194 ceph-mon[33017]: pgmap v10648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19091.985516 secs 2024-04-26T17:09:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19094.827905 secs 2024-04-26T17:09:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19091.985516 secs 2024-04-26T17:09:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19094.827905 secs 2024-04-26T17:09:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19091.985516 secs 2024-04-26T17:09:14.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19094.827905 secs 2024-04-26T17:09:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:15 smithi033 ceph-mon[27845]: pgmap v10649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:15 smithi096 ceph-mon[32753]: pgmap v10649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:15 smithi194 ceph-mon[33017]: pgmap v10649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:16.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:16.816+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:16.816+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:16.816+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:09:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:17 smithi033 ceph-mon[27845]: pgmap v10650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:17 smithi096 ceph-mon[32753]: pgmap v10650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:17 smithi194 ceph-mon[33017]: pgmap v10650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19096.985602 secs 2024-04-26T17:09:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19096.985602 secs 2024-04-26T17:09:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19096.985602 secs 2024-04-26T17:09:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19099.828040 secs 2024-04-26T17:09:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:19 smithi194 ceph-mon[33017]: pgmap v10651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19099.828040 secs 2024-04-26T17:09:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:19 smithi033 ceph-mon[27845]: pgmap v10651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19099.828040 secs 2024-04-26T17:09:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:19 smithi096 ceph-mon[32753]: pgmap v10651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:21 smithi033 ceph-mon[27845]: pgmap v10652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:21 smithi096 ceph-mon[32753]: pgmap v10652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:21 smithi194 ceph-mon[33017]: pgmap v10652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:23 smithi033 ceph-mon[27845]: pgmap v10653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:23 smithi096 ceph-mon[32753]: pgmap v10653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:23.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:23 smithi194 ceph-mon[33017]: pgmap v10653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19101.985683 secs 2024-04-26T17:09:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19104.828102 secs 2024-04-26T17:09:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19101.985683 secs 2024-04-26T17:09:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19104.828102 secs 2024-04-26T17:09:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19101.985683 secs 2024-04-26T17:09:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19104.828102 secs 2024-04-26T17:09:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:25 smithi033 ceph-mon[27845]: pgmap v10654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:25 smithi096 ceph-mon[32753]: pgmap v10654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:25 smithi194 ceph-mon[33017]: pgmap v10654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:27 smithi033 ceph-mon[27845]: pgmap v10655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:27 smithi096 ceph-mon[32753]: pgmap v10655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:27 smithi194 ceph-mon[33017]: pgmap v10655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19106.985772 secs 2024-04-26T17:09:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19106.985772 secs 2024-04-26T17:09:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19106.985772 secs 2024-04-26T17:09:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19109.828220 secs 2024-04-26T17:09:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:29 smithi194 ceph-mon[33017]: pgmap v10656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19109.828220 secs 2024-04-26T17:09:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:29 smithi033 ceph-mon[27845]: pgmap v10656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19109.828220 secs 2024-04-26T17:09:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:29 smithi096 ceph-mon[32753]: pgmap v10656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:31 smithi033 ceph-mon[27845]: pgmap v10657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:31 smithi096 ceph-mon[32753]: pgmap v10657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:31 smithi194 ceph-mon[33017]: pgmap v10657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:31.822+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:31.822+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:09:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:31.822+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:33 smithi033 ceph-mon[27845]: pgmap v10658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:33 smithi096 ceph-mon[32753]: pgmap v10658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:33 smithi194 ceph-mon[33017]: pgmap v10658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19111.985858 secs 2024-04-26T17:09:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19114.828341 secs 2024-04-26T17:09:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19111.985858 secs 2024-04-26T17:09:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19114.828341 secs 2024-04-26T17:09:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19111.985858 secs 2024-04-26T17:09:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19114.828341 secs 2024-04-26T17:09:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:35 smithi033 ceph-mon[27845]: pgmap v10659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:35 smithi096 ceph-mon[32753]: pgmap v10659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:35 smithi194 ceph-mon[33017]: pgmap v10659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:37 smithi033 ceph-mon[27845]: pgmap v10660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:37 smithi096 ceph-mon[32753]: pgmap v10660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:37 smithi194 ceph-mon[33017]: pgmap v10660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19116.985946 secs 2024-04-26T17:09:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19116.985946 secs 2024-04-26T17:09:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19116.985946 secs 2024-04-26T17:09:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19119.828403 secs 2024-04-26T17:09:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:39 smithi194 ceph-mon[33017]: pgmap v10661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19119.828403 secs 2024-04-26T17:09:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:39 smithi033 ceph-mon[27845]: pgmap v10661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19119.828403 secs 2024-04-26T17:09:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:39 smithi096 ceph-mon[32753]: pgmap v10661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:41 smithi033 ceph-mon[27845]: pgmap v10662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:41 smithi096 ceph-mon[32753]: pgmap v10662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:41 smithi194 ceph-mon[33017]: pgmap v10662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:43 smithi033 ceph-mon[27845]: pgmap v10663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:43 smithi096 ceph-mon[32753]: pgmap v10663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:43 smithi194 ceph-mon[33017]: pgmap v10663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19121.986011 secs 2024-04-26T17:09:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19124.828523 secs 2024-04-26T17:09:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19121.986011 secs 2024-04-26T17:09:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19124.828523 secs 2024-04-26T17:09:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19121.986011 secs 2024-04-26T17:09:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19124.828523 secs 2024-04-26T17:09:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:45 smithi033 ceph-mon[27845]: pgmap v10664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:45 smithi096 ceph-mon[32753]: pgmap v10664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:45 smithi194 ceph-mon[33017]: pgmap v10664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:46.830+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:46.830+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:09:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:09:46.830+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:09:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:47 smithi033 ceph-mon[27845]: pgmap v10665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:47 smithi096 ceph-mon[32753]: pgmap v10665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:47 smithi194 ceph-mon[33017]: pgmap v10665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19126.986099 secs 2024-04-26T17:09:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19126.986099 secs 2024-04-26T17:09:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19126.986099 secs 2024-04-26T17:09:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19129.828642 secs 2024-04-26T17:09:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:49 smithi194 ceph-mon[33017]: pgmap v10666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19129.828642 secs 2024-04-26T17:09:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:49 smithi033 ceph-mon[27845]: pgmap v10666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19129.828642 secs 2024-04-26T17:09:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:49 smithi096 ceph-mon[32753]: pgmap v10666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:51 smithi033 ceph-mon[27845]: pgmap v10667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:51 smithi096 ceph-mon[32753]: pgmap v10667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:51 smithi194 ceph-mon[33017]: pgmap v10667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:53 smithi033 ceph-mon[27845]: pgmap v10668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:53 smithi096 ceph-mon[32753]: pgmap v10668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:53 smithi194 ceph-mon[33017]: pgmap v10668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19131.986161 secs 2024-04-26T17:09:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19134.828762 secs 2024-04-26T17:09:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19131.986161 secs 2024-04-26T17:09:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19134.828762 secs 2024-04-26T17:09:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19131.986161 secs 2024-04-26T17:09:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19134.828762 secs 2024-04-26T17:09:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:55 smithi033 ceph-mon[27845]: pgmap v10669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:55 smithi096 ceph-mon[32753]: pgmap v10669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:55 smithi194 ceph-mon[33017]: pgmap v10669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:09:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:09:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:09:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:09:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:09:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:09:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:57 smithi194 ceph-mon[33017]: pgmap v10670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:57 smithi194 ceph-mon[33017]: pgmap v10671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:09:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:09:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:09:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:57 smithi033 ceph-mon[27845]: pgmap v10670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:57 smithi033 ceph-mon[27845]: pgmap v10671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:09:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:09:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:09:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:57 smithi096 ceph-mon[32753]: pgmap v10670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:09:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:57 smithi096 ceph-mon[32753]: pgmap v10671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:09:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19136.986224 secs 2024-04-26T17:09:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19136.986224 secs 2024-04-26T17:09:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19136.986224 secs 2024-04-26T17:09:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19139.828848 secs 2024-04-26T17:09:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:09:59 smithi194 ceph-mon[33017]: pgmap v10672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19139.828848 secs 2024-04-26T17:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:09:59 smithi033 ceph-mon[27845]: pgmap v10672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:09:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19139.828848 secs 2024-04-26T17:09:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:09:59 smithi096 ceph-mon[32753]: pgmap v10672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:10:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:10:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:10:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:01.836+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:01.836+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:10:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:01.836+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:01 smithi194 ceph-mon[33017]: pgmap v10673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:01 smithi033 ceph-mon[27845]: pgmap v10673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:01 smithi096 ceph-mon[32753]: pgmap v10673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:03 smithi194 ceph-mon[33017]: pgmap v10674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19141.986286 secs 2024-04-26T17:10:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:03 smithi033 ceph-mon[27845]: pgmap v10674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19141.986286 secs 2024-04-26T17:10:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:03 smithi096 ceph-mon[32753]: pgmap v10674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19141.986286 secs 2024-04-26T17:10:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19144.828972 secs 2024-04-26T17:10:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19144.828972 secs 2024-04-26T17:10:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19144.828972 secs 2024-04-26T17:10:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:05 smithi194 ceph-mon[33017]: pgmap v10675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:05 smithi033 ceph-mon[27845]: pgmap v10675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:05 smithi096 ceph-mon[32753]: pgmap v10675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:10:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:07 smithi194 ceph-mon[33017]: pgmap v10676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:10:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:07 smithi033 ceph-mon[27845]: pgmap v10676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:10:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:07 smithi096 ceph-mon[32753]: pgmap v10676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:10:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19146.986348 secs 2024-04-26T17:10:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19146.986348 secs 2024-04-26T17:10:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19146.986348 secs 2024-04-26T17:10:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19149.829093 secs 2024-04-26T17:10:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:09 smithi194 ceph-mon[33017]: pgmap v10677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19149.829093 secs 2024-04-26T17:10:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:09 smithi033 ceph-mon[27845]: pgmap v10677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19149.829093 secs 2024-04-26T17:10:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:09 smithi096 ceph-mon[32753]: pgmap v10677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:12.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:11 smithi033 ceph-mon[27845]: pgmap v10678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:11 smithi194 ceph-mon[33017]: pgmap v10678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:11 smithi096 ceph-mon[32753]: pgmap v10678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:14.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:13 smithi033 ceph-mon[27845]: pgmap v10679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:14.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19151.986434 secs 2024-04-26T17:10:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:13 smithi194 ceph-mon[33017]: pgmap v10679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19151.986434 secs 2024-04-26T17:10:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:13 smithi096 ceph-mon[32753]: pgmap v10679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19151.986434 secs 2024-04-26T17:10:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19154.829215 secs 2024-04-26T17:10:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19154.829215 secs 2024-04-26T17:10:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19154.829215 secs 2024-04-26T17:10:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:15 smithi194 ceph-mon[33017]: pgmap v10680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:15 smithi033 ceph-mon[27845]: pgmap v10680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:15 smithi096 ceph-mon[32753]: pgmap v10680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:16.839+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:16.839+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:10:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:16.839+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:17 smithi194 ceph-mon[33017]: pgmap v10681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:17 smithi033 ceph-mon[27845]: pgmap v10681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:17 smithi096 ceph-mon[32753]: pgmap v10681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19156.986517 secs 2024-04-26T17:10:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19156.986517 secs 2024-04-26T17:10:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19156.986517 secs 2024-04-26T17:10:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19159.829327 secs 2024-04-26T17:10:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:19 smithi194 ceph-mon[33017]: pgmap v10682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:20.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19159.829327 secs 2024-04-26T17:10:20.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:19 smithi033 ceph-mon[27845]: pgmap v10682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19159.829327 secs 2024-04-26T17:10:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:19 smithi096 ceph-mon[32753]: pgmap v10682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:21 smithi194 ceph-mon[33017]: pgmap v10683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:21 smithi033 ceph-mon[27845]: pgmap v10683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:21 smithi096 ceph-mon[32753]: pgmap v10683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:23 smithi194 ceph-mon[33017]: pgmap v10684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19161.986603 secs 2024-04-26T17:10:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:23 smithi033 ceph-mon[27845]: pgmap v10684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19161.986603 secs 2024-04-26T17:10:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:23 smithi096 ceph-mon[32753]: pgmap v10684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19161.986603 secs 2024-04-26T17:10:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19164.829456 secs 2024-04-26T17:10:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19164.829456 secs 2024-04-26T17:10:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19164.829456 secs 2024-04-26T17:10:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:25 smithi194 ceph-mon[33017]: pgmap v10685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:25 smithi033 ceph-mon[27845]: pgmap v10685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:25 smithi096 ceph-mon[32753]: pgmap v10685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:27 smithi194 ceph-mon[33017]: pgmap v10686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:27 smithi033 ceph-mon[27845]: pgmap v10686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:27 smithi096 ceph-mon[32753]: pgmap v10686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19166.986689 secs 2024-04-26T17:10:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19166.986689 secs 2024-04-26T17:10:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19166.986689 secs 2024-04-26T17:10:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19169.829572 secs 2024-04-26T17:10:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:29 smithi194 ceph-mon[33017]: pgmap v10687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19169.829572 secs 2024-04-26T17:10:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:29 smithi033 ceph-mon[27845]: pgmap v10687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19169.829572 secs 2024-04-26T17:10:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:29 smithi096 ceph-mon[32753]: pgmap v10687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:31.854+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:31.854+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:31.854+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:10:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:31 smithi194 ceph-mon[33017]: pgmap v10688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:31 smithi033 ceph-mon[27845]: pgmap v10688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:31 smithi096 ceph-mon[32753]: pgmap v10688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:33 smithi033 ceph-mon[27845]: pgmap v10689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19171.986774 secs 2024-04-26T17:10:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:33 smithi096 ceph-mon[32753]: pgmap v10689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19171.986774 secs 2024-04-26T17:10:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:33 smithi194 ceph-mon[33017]: pgmap v10689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19171.986774 secs 2024-04-26T17:10:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19174.829644 secs 2024-04-26T17:10:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19174.829644 secs 2024-04-26T17:10:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19174.829644 secs 2024-04-26T17:10:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:35 smithi033 ceph-mon[27845]: pgmap v10690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:35 smithi096 ceph-mon[32753]: pgmap v10690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:35 smithi194 ceph-mon[33017]: pgmap v10690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:37 smithi033 ceph-mon[27845]: pgmap v10691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:37 smithi096 ceph-mon[32753]: pgmap v10691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:37 smithi194 ceph-mon[33017]: pgmap v10691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19176.986858 secs 2024-04-26T17:10:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19176.986858 secs 2024-04-26T17:10:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19176.986858 secs 2024-04-26T17:10:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19179.829763 secs 2024-04-26T17:10:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:39 smithi194 ceph-mon[33017]: pgmap v10692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19179.829763 secs 2024-04-26T17:10:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:39 smithi033 ceph-mon[27845]: pgmap v10692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19179.829763 secs 2024-04-26T17:10:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:39 smithi096 ceph-mon[32753]: pgmap v10692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:10:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:41 smithi033 ceph-mon[27845]: pgmap v10693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:41 smithi096 ceph-mon[32753]: pgmap v10693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:41 smithi194 ceph-mon[33017]: pgmap v10693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:10:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:43 smithi033 ceph-mon[27845]: pgmap v10694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19181.986943 secs 2024-04-26T17:10:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:43 smithi096 ceph-mon[32753]: pgmap v10694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19181.986943 secs 2024-04-26T17:10:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:43 smithi194 ceph-mon[33017]: pgmap v10694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19181.986943 secs 2024-04-26T17:10:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19184.829884 secs 2024-04-26T17:10:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19184.829884 secs 2024-04-26T17:10:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19184.829884 secs 2024-04-26T17:10:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:45 smithi033 ceph-mon[27845]: pgmap v10695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:45 smithi096 ceph-mon[32753]: pgmap v10695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:45 smithi194 ceph-mon[33017]: pgmap v10695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:46.871+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:46.871+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:10:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:10:46.871+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:10:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:47 smithi033 ceph-mon[27845]: pgmap v10696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:47 smithi096 ceph-mon[32753]: pgmap v10696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:47 smithi194 ceph-mon[33017]: pgmap v10696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19186.987026 secs 2024-04-26T17:10:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19186.987026 secs 2024-04-26T17:10:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19186.987026 secs 2024-04-26T17:10:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19189.829995 secs 2024-04-26T17:10:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:49 smithi096 ceph-mon[32753]: pgmap v10697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19189.829995 secs 2024-04-26T17:10:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:49 smithi194 ceph-mon[33017]: pgmap v10697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19189.829995 secs 2024-04-26T17:10:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:49 smithi033 ceph-mon[27845]: pgmap v10697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:51 smithi194 ceph-mon[33017]: pgmap v10698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:51 smithi033 ceph-mon[27845]: pgmap v10698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:51 smithi096 ceph-mon[32753]: pgmap v10698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:53 smithi194 ceph-mon[33017]: pgmap v10699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19191.987089 secs 2024-04-26T17:10:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:53 smithi033 ceph-mon[27845]: pgmap v10699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19191.987089 secs 2024-04-26T17:10:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:53 smithi096 ceph-mon[32753]: pgmap v10699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19191.987089 secs 2024-04-26T17:10:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19194.830132 secs 2024-04-26T17:10:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19194.830132 secs 2024-04-26T17:10:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19194.830132 secs 2024-04-26T17:10:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:55 smithi194 ceph-mon[33017]: pgmap v10700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:55 smithi033 ceph-mon[27845]: pgmap v10700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:55 smithi096 ceph-mon[32753]: pgmap v10700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:10:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:10:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:10:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:10:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:57 smithi194 ceph-mon[33017]: pgmap v10701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:57 smithi033 ceph-mon[27845]: pgmap v10701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:57 smithi096 ceph-mon[32753]: pgmap v10701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:10:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19196.987148 secs 2024-04-26T17:10:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:10:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:10:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:10:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19199.830212 secs 2024-04-26T17:10:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19196.987148 secs 2024-04-26T17:10:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:10:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:10:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:10:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19199.830212 secs 2024-04-26T17:10:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19196.987148 secs 2024-04-26T17:10:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:10:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:10:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:10:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19199.830212 secs 2024-04-26T17:11:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:59 smithi194 ceph-mon[33017]: pgmap v10702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:10:59 smithi194 ceph-mon[33017]: pgmap v10703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:59 smithi033 ceph-mon[27845]: pgmap v10702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:10:59 smithi033 ceph-mon[27845]: pgmap v10703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:59 smithi096 ceph-mon[32753]: pgmap v10702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:10:59 smithi096 ceph-mon[32753]: pgmap v10703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:01.886+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:01.886+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:11:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:01.886+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:01 smithi194 ceph-mon[33017]: pgmap v10704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:01 smithi033 ceph-mon[27845]: pgmap v10704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:01 smithi096 ceph-mon[32753]: pgmap v10704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:03 smithi194 ceph-mon[33017]: pgmap v10705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19201.987231 secs 2024-04-26T17:11:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19204.830330 secs 2024-04-26T17:11:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:03 smithi033 ceph-mon[27845]: pgmap v10705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19201.987231 secs 2024-04-26T17:11:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19204.830330 secs 2024-04-26T17:11:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:03 smithi096 ceph-mon[32753]: pgmap v10705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19201.987231 secs 2024-04-26T17:11:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19204.830330 secs 2024-04-26T17:11:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:06 smithi194 ceph-mon[33017]: pgmap v10706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:11:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:06 smithi033 ceph-mon[27845]: pgmap v10706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:11:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:06 smithi096 ceph-mon[32753]: pgmap v10706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:11:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:08 smithi194 ceph-mon[33017]: pgmap v10707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:08 smithi033 ceph-mon[27845]: pgmap v10707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:08 smithi096 ceph-mon[32753]: pgmap v10707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19206.987315 secs 2024-04-26T17:11:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19209.830424 secs 2024-04-26T17:11:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19206.987315 secs 2024-04-26T17:11:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19209.830424 secs 2024-04-26T17:11:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19206.987315 secs 2024-04-26T17:11:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19209.830424 secs 2024-04-26T17:11:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:10 smithi194 ceph-mon[33017]: pgmap v10708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:11:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:10 smithi033 ceph-mon[27845]: pgmap v10708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:11:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:10 smithi096 ceph-mon[32753]: pgmap v10708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:11:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:12 smithi194 ceph-mon[33017]: pgmap v10709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:12 smithi033 ceph-mon[27845]: pgmap v10709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:12 smithi096 ceph-mon[32753]: pgmap v10709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:14.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:14 smithi033 ceph-mon[27845]: pgmap v10710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:14.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19211.987399 secs 2024-04-26T17:11:14.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19214.830526 secs 2024-04-26T17:11:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:14 smithi194 ceph-mon[33017]: pgmap v10710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19211.987399 secs 2024-04-26T17:11:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19214.830526 secs 2024-04-26T17:11:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:14 smithi096 ceph-mon[32753]: pgmap v10710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19211.987399 secs 2024-04-26T17:11:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19214.830526 secs 2024-04-26T17:11:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:16 smithi194 ceph-mon[33017]: pgmap v10711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:16 smithi033 ceph-mon[27845]: pgmap v10711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:16 smithi096 ceph-mon[32753]: pgmap v10711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:16.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:16.891+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:16.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:16.891+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:11:16.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:16.891+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:18 smithi194 ceph-mon[33017]: pgmap v10712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:18 smithi033 ceph-mon[27845]: pgmap v10712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:18 smithi096 ceph-mon[32753]: pgmap v10712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19216.987469 secs 2024-04-26T17:11:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19219.830662 secs 2024-04-26T17:11:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19216.987469 secs 2024-04-26T17:11:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19219.830662 secs 2024-04-26T17:11:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19216.987469 secs 2024-04-26T17:11:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19219.830662 secs 2024-04-26T17:11:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:20 smithi194 ceph-mon[33017]: pgmap v10713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:20 smithi033 ceph-mon[27845]: pgmap v10713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:20 smithi096 ceph-mon[32753]: pgmap v10713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T17:11:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T17:11:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 0x7feee40b3710 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0022070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T17:11:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 0x7feee40b3710 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0022070 comp rx=0 tx=0).stop 2024-04-26T17:11:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T17:11:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T17:11:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40af500 0x7feee40b3710 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:11:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.764+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:11:21.766 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.765+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:11:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:21.767+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0058040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:11:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:22 smithi194 ceph-mon[33017]: pgmap v10714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:22 smithi033 ceph-mon[27845]: pgmap v10714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:22 smithi096 ceph-mon[32753]: pgmap v10714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:24 smithi194 ceph-mon[33017]: pgmap v10715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19221.987551 secs 2024-04-26T17:11:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19224.830779 secs 2024-04-26T17:11:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:24 smithi033 ceph-mon[27845]: pgmap v10715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19221.987551 secs 2024-04-26T17:11:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19224.830779 secs 2024-04-26T17:11:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:24 smithi096 ceph-mon[32753]: pgmap v10715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19221.987551 secs 2024-04-26T17:11:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19224.830779 secs 2024-04-26T17:11:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.057+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T17:11:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.057+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:11:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.058+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T17:11:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.058+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T17:11:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.058+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:11:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.058+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:11:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.058+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006630 con 0x564385821900 2024-04-26T17:11:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:25.059+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc004740 con 0x7fef00077780 2024-04-26T17:11:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:26 smithi194 ceph-mon[33017]: pgmap v10716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:26 smithi033 ceph-mon[27845]: pgmap v10716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:26 smithi096 ceph-mon[32753]: pgmap v10716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:28 smithi194 ceph-mon[33017]: pgmap v10717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:28 smithi033 ceph-mon[27845]: pgmap v10717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:28 smithi096 ceph-mon[32753]: pgmap v10717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:29.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19226.987645 secs 2024-04-26T17:11:29.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19229.830888 secs 2024-04-26T17:11:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19226.987645 secs 2024-04-26T17:11:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19229.830888 secs 2024-04-26T17:11:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19226.987645 secs 2024-04-26T17:11:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19229.830888 secs 2024-04-26T17:11:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:30 smithi194 ceph-mon[33017]: pgmap v10718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:30 smithi033 ceph-mon[27845]: pgmap v10718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:30 smithi096 ceph-mon[32753]: pgmap v10718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:31.901+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:31.901+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:31.901+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:11:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:32 smithi194 ceph-mon[33017]: pgmap v10719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:32 smithi033 ceph-mon[27845]: pgmap v10719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:32 smithi096 ceph-mon[32753]: pgmap v10719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:34 smithi194 ceph-mon[33017]: pgmap v10720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19231.987742 secs 2024-04-26T17:11:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19234.831012 secs 2024-04-26T17:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:34 smithi033 ceph-mon[27845]: pgmap v10720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19231.987742 secs 2024-04-26T17:11:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19234.831012 secs 2024-04-26T17:11:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:34 smithi096 ceph-mon[32753]: pgmap v10720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19231.987742 secs 2024-04-26T17:11:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19234.831012 secs 2024-04-26T17:11:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:36 smithi033 ceph-mon[27845]: pgmap v10721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:36 smithi096 ceph-mon[32753]: pgmap v10721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:36 smithi194 ceph-mon[33017]: pgmap v10721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:38 smithi033 ceph-mon[27845]: pgmap v10722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:38 smithi096 ceph-mon[32753]: pgmap v10722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:38 smithi194 ceph-mon[33017]: pgmap v10722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19236.987844 secs 2024-04-26T17:11:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19239.831130 secs 2024-04-26T17:11:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19236.987844 secs 2024-04-26T17:11:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19239.831130 secs 2024-04-26T17:11:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19236.987844 secs 2024-04-26T17:11:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19239.831130 secs 2024-04-26T17:11:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:40 smithi033 ceph-mon[27845]: pgmap v10723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:40 smithi096 ceph-mon[32753]: pgmap v10723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:40 smithi194 ceph-mon[33017]: pgmap v10723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:41 smithi033 ceph-mon[27845]: pgmap v10724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:41 smithi096 ceph-mon[32753]: pgmap v10724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:41 smithi194 ceph-mon[33017]: pgmap v10724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:43 smithi033 ceph-mon[27845]: pgmap v10725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:43 smithi096 ceph-mon[32753]: pgmap v10725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:43 smithi194 ceph-mon[33017]: pgmap v10725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19241.987946 secs 2024-04-26T17:11:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19244.831241 secs 2024-04-26T17:11:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19241.987946 secs 2024-04-26T17:11:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19244.831241 secs 2024-04-26T17:11:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19241.987946 secs 2024-04-26T17:11:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19244.831241 secs 2024-04-26T17:11:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:45 smithi033 ceph-mon[27845]: pgmap v10726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:45 smithi194 ceph-mon[33017]: pgmap v10726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:45 smithi096 ceph-mon[32753]: pgmap v10726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:46.917+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:46.917+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:11:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:11:46.917+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:11:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:47 smithi194 ceph-mon[33017]: pgmap v10727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:47 smithi033 ceph-mon[27845]: pgmap v10727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:47 smithi096 ceph-mon[32753]: pgmap v10727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19246.988002 secs 2024-04-26T17:11:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:49 smithi194 ceph-mon[33017]: pgmap v10728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19249.831318 secs 2024-04-26T17:11:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19246.988002 secs 2024-04-26T17:11:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:49 smithi033 ceph-mon[27845]: pgmap v10728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19249.831318 secs 2024-04-26T17:11:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19246.988002 secs 2024-04-26T17:11:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:49 smithi096 ceph-mon[32753]: pgmap v10728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19249.831318 secs 2024-04-26T17:11:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:51 smithi194 ceph-mon[33017]: pgmap v10729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:51 smithi033 ceph-mon[27845]: pgmap v10729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:51 smithi096 ceph-mon[32753]: pgmap v10729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:11:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:53 smithi194 ceph-mon[33017]: pgmap v10730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:53 smithi033 ceph-mon[27845]: pgmap v10730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:53 smithi096 ceph-mon[32753]: pgmap v10730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19251.988089 secs 2024-04-26T17:11:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19254.831428 secs 2024-04-26T17:11:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19251.988089 secs 2024-04-26T17:11:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19254.831428 secs 2024-04-26T17:11:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19251.988089 secs 2024-04-26T17:11:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19254.831428 secs 2024-04-26T17:11:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:55 smithi194 ceph-mon[33017]: pgmap v10731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:55 smithi033 ceph-mon[27845]: pgmap v10731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:55 smithi096 ceph-mon[32753]: pgmap v10731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:57 smithi194 ceph-mon[33017]: pgmap v10732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:57 smithi033 ceph-mon[27845]: pgmap v10732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:57 smithi096 ceph-mon[32753]: pgmap v10732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:11:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19256.988147 secs 2024-04-26T17:11:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:59 smithi194 ceph-mon[33017]: pgmap v10733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19259.831500 secs 2024-04-26T17:11:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:11:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:11:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19256.988147 secs 2024-04-26T17:11:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:59 smithi033 ceph-mon[27845]: pgmap v10733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19259.831500 secs 2024-04-26T17:11:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:11:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:11:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19256.988147 secs 2024-04-26T17:11:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:59 smithi096 ceph-mon[32753]: pgmap v10733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:11:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19259.831500 secs 2024-04-26T17:11:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:11:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:12:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:01 smithi033 ceph-mon[27845]: pgmap v10734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:12:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:12:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:01 smithi033 ceph-mon[27845]: pgmap v10735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:12:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:01 smithi033 ceph-mon[27845]: pgmap v10736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:01 smithi096 ceph-mon[32753]: pgmap v10734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:12:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:12:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:01 smithi096 ceph-mon[32753]: pgmap v10735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:12:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:01 smithi096 ceph-mon[32753]: pgmap v10736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:01.932+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:01.932+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:01.932+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:12:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:01 smithi194 ceph-mon[33017]: pgmap v10734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:12:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:12:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:01 smithi194 ceph-mon[33017]: pgmap v10735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:12:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:01 smithi194 ceph-mon[33017]: pgmap v10736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:12:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19261.988245 secs 2024-04-26T17:12:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:04 smithi194 ceph-mon[33017]: pgmap v10737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19264.831625 secs 2024-04-26T17:12:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19261.988245 secs 2024-04-26T17:12:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:04 smithi033 ceph-mon[27845]: pgmap v10737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19264.831625 secs 2024-04-26T17:12:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19261.988245 secs 2024-04-26T17:12:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:04 smithi096 ceph-mon[32753]: pgmap v10737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19264.831625 secs 2024-04-26T17:12:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:06 smithi194 ceph-mon[33017]: pgmap v10738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:06 smithi033 ceph-mon[27845]: pgmap v10738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:06 smithi096 ceph-mon[32753]: pgmap v10738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:08 smithi033 ceph-mon[27845]: pgmap v10739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:08 smithi096 ceph-mon[32753]: pgmap v10739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:08 smithi194 ceph-mon[33017]: pgmap v10739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:12:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19266.988306 secs 2024-04-26T17:12:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19269.831775 secs 2024-04-26T17:12:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19266.988306 secs 2024-04-26T17:12:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19269.831775 secs 2024-04-26T17:12:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19266.988306 secs 2024-04-26T17:12:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19269.831775 secs 2024-04-26T17:12:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:10 smithi033 ceph-mon[27845]: pgmap v10740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:12:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:10 smithi096 ceph-mon[32753]: pgmap v10740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:12:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:10 smithi194 ceph-mon[33017]: pgmap v10740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:12:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:12 smithi033 ceph-mon[27845]: pgmap v10741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:12:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:12 smithi096 ceph-mon[32753]: pgmap v10741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:12:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:12 smithi194 ceph-mon[33017]: pgmap v10741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:12:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19271.988392 secs 2024-04-26T17:12:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:14 smithi033 ceph-mon[27845]: pgmap v10742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19274.831925 secs 2024-04-26T17:12:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19271.988392 secs 2024-04-26T17:12:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:14 smithi096 ceph-mon[32753]: pgmap v10742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19274.831925 secs 2024-04-26T17:12:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19271.988392 secs 2024-04-26T17:12:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:14 smithi194 ceph-mon[33017]: pgmap v10742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19274.831925 secs 2024-04-26T17:12:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:16 smithi033 ceph-mon[27845]: pgmap v10743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:16 smithi096 ceph-mon[32753]: pgmap v10743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:16 smithi194 ceph-mon[33017]: pgmap v10743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:16.947+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:16.947+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:16.947+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:12:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:18 smithi033 ceph-mon[27845]: pgmap v10744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:18 smithi096 ceph-mon[32753]: pgmap v10744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:18 smithi194 ceph-mon[33017]: pgmap v10744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19276.988493 secs 2024-04-26T17:12:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19279.832045 secs 2024-04-26T17:12:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19276.988493 secs 2024-04-26T17:12:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19279.832045 secs 2024-04-26T17:12:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19276.988493 secs 2024-04-26T17:12:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19279.832045 secs 2024-04-26T17:12:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:20 smithi033 ceph-mon[27845]: pgmap v10745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:20 smithi096 ceph-mon[32753]: pgmap v10745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:20 smithi194 ceph-mon[33017]: pgmap v10745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:21 smithi033 ceph-mon[27845]: pgmap v10746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:21 smithi096 ceph-mon[32753]: pgmap v10746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:21 smithi194 ceph-mon[33017]: pgmap v10746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19281.988592 secs 2024-04-26T17:12:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:24 smithi033 ceph-mon[27845]: pgmap v10747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19284.832165 secs 2024-04-26T17:12:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19281.988592 secs 2024-04-26T17:12:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:24 smithi096 ceph-mon[32753]: pgmap v10747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19284.832165 secs 2024-04-26T17:12:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19281.988592 secs 2024-04-26T17:12:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:24 smithi194 ceph-mon[33017]: pgmap v10747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19284.832165 secs 2024-04-26T17:12:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:26 smithi033 ceph-mon[27845]: pgmap v10748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:26 smithi096 ceph-mon[32753]: pgmap v10748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:26 smithi194 ceph-mon[33017]: pgmap v10748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:28 smithi033 ceph-mon[27845]: pgmap v10749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:28 smithi096 ceph-mon[32753]: pgmap v10749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:28 smithi194 ceph-mon[33017]: pgmap v10749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19286.988689 secs 2024-04-26T17:12:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19289.832282 secs 2024-04-26T17:12:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19286.988689 secs 2024-04-26T17:12:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19289.832282 secs 2024-04-26T17:12:29.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19286.988689 secs 2024-04-26T17:12:29.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19289.832282 secs 2024-04-26T17:12:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:30 smithi033 ceph-mon[27845]: pgmap v10750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:30 smithi096 ceph-mon[32753]: pgmap v10750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:30 smithi194 ceph-mon[33017]: pgmap v10750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:31.961+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:31.961+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:31.961+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:12:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:32 smithi033 ceph-mon[27845]: pgmap v10751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:32 smithi096 ceph-mon[32753]: pgmap v10751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:32 smithi194 ceph-mon[33017]: pgmap v10751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19291.988746 secs 2024-04-26T17:12:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:34 smithi033 ceph-mon[27845]: pgmap v10752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19294.832362 secs 2024-04-26T17:12:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19291.988746 secs 2024-04-26T17:12:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:34 smithi096 ceph-mon[32753]: pgmap v10752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19294.832362 secs 2024-04-26T17:12:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19291.988746 secs 2024-04-26T17:12:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:34 smithi194 ceph-mon[33017]: pgmap v10752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19294.832362 secs 2024-04-26T17:12:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:36 smithi033 ceph-mon[27845]: pgmap v10753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:36 smithi096 ceph-mon[32753]: pgmap v10753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:36 smithi194 ceph-mon[33017]: pgmap v10753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:37 smithi194 ceph-mon[33017]: pgmap v10754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:37 smithi033 ceph-mon[27845]: pgmap v10754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:37 smithi096 ceph-mon[32753]: pgmap v10754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19296.988826 secs 2024-04-26T17:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19296.988826 secs 2024-04-26T17:12:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19296.988826 secs 2024-04-26T17:12:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19299.832452 secs 2024-04-26T17:12:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:39 smithi033 ceph-mon[27845]: pgmap v10755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19299.832452 secs 2024-04-26T17:12:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:39 smithi096 ceph-mon[32753]: pgmap v10755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19299.832452 secs 2024-04-26T17:12:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:39 smithi194 ceph-mon[33017]: pgmap v10755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:42 smithi033 ceph-mon[27845]: pgmap v10756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:42 smithi096 ceph-mon[32753]: pgmap v10756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:42 smithi194 ceph-mon[33017]: pgmap v10756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19301.988908 secs 2024-04-26T17:12:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:44 smithi033 ceph-mon[27845]: pgmap v10757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19304.832587 secs 2024-04-26T17:12:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19301.988908 secs 2024-04-26T17:12:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:44 smithi096 ceph-mon[32753]: pgmap v10757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19304.832587 secs 2024-04-26T17:12:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19301.988908 secs 2024-04-26T17:12:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:44 smithi194 ceph-mon[33017]: pgmap v10757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19304.832587 secs 2024-04-26T17:12:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:46 smithi033 ceph-mon[27845]: pgmap v10758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:46 smithi096 ceph-mon[32753]: pgmap v10758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:46 smithi194 ceph-mon[33017]: pgmap v10758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:46.978+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:46.978+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:12:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:12:46.978+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:12:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:48 smithi033 ceph-mon[27845]: pgmap v10759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:48 smithi096 ceph-mon[32753]: pgmap v10759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:48 smithi194 ceph-mon[33017]: pgmap v10759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19306.988992 secs 2024-04-26T17:12:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19309.832692 secs 2024-04-26T17:12:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19306.988992 secs 2024-04-26T17:12:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19309.832692 secs 2024-04-26T17:12:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19306.988992 secs 2024-04-26T17:12:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19309.832692 secs 2024-04-26T17:12:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:50 smithi033 ceph-mon[27845]: pgmap v10760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:50 smithi096 ceph-mon[32753]: pgmap v10760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:50 smithi194 ceph-mon[33017]: pgmap v10760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:52 smithi033 ceph-mon[27845]: pgmap v10761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:52 smithi096 ceph-mon[32753]: pgmap v10761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:52 smithi194 ceph-mon[33017]: pgmap v10761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:53 smithi033 ceph-mon[27845]: pgmap v10762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:53 smithi096 ceph-mon[32753]: pgmap v10762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:53 smithi194 ceph-mon[33017]: pgmap v10762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19311.989072 secs 2024-04-26T17:12:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19314.832815 secs 2024-04-26T17:12:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19311.989072 secs 2024-04-26T17:12:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19314.832815 secs 2024-04-26T17:12:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19311.989072 secs 2024-04-26T17:12:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19314.832815 secs 2024-04-26T17:12:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:55 smithi033 ceph-mon[27845]: pgmap v10763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:55 smithi096 ceph-mon[32753]: pgmap v10763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:55 smithi194 ceph-mon[33017]: pgmap v10763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:12:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:58 smithi033 ceph-mon[27845]: pgmap v10764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:58 smithi096 ceph-mon[32753]: pgmap v10764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:58 smithi194 ceph-mon[33017]: pgmap v10764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:12:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19316.989134 secs 2024-04-26T17:12:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:12:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19319.832936 secs 2024-04-26T17:12:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19316.989134 secs 2024-04-26T17:12:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:12:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19319.832936 secs 2024-04-26T17:12:59.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19316.989134 secs 2024-04-26T17:12:59.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:12:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19319.832936 secs 2024-04-26T17:13:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:00 smithi033 ceph-mon[27845]: pgmap v10765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:00 smithi096 ceph-mon[32753]: pgmap v10765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:00 smithi194 ceph-mon[33017]: pgmap v10765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:01.987+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:01.987+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:01.987+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:13:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:02 smithi033 ceph-mon[27845]: pgmap v10766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:13:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:02 smithi096 ceph-mon[32753]: pgmap v10766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:13:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:02 smithi194 ceph-mon[33017]: pgmap v10766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:13:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:13:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:13:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:13:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:13:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:13:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:13:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:13:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:13:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:04 smithi033 ceph-mon[27845]: pgmap v10767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:04 smithi033 ceph-mon[27845]: pgmap v10768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19321.989216 secs 2024-04-26T17:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19324.833061 secs 2024-04-26T17:13:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:04 smithi096 ceph-mon[32753]: pgmap v10767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:04 smithi096 ceph-mon[32753]: pgmap v10768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19321.989216 secs 2024-04-26T17:13:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19324.833061 secs 2024-04-26T17:13:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:04 smithi194 ceph-mon[33017]: pgmap v10767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:04 smithi194 ceph-mon[33017]: pgmap v10768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19321.989216 secs 2024-04-26T17:13:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19324.833061 secs 2024-04-26T17:13:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:05 smithi033 ceph-mon[27845]: pgmap v10769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:05 smithi096 ceph-mon[32753]: pgmap v10769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:05 smithi194 ceph-mon[33017]: pgmap v10769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:07 smithi194 ceph-mon[33017]: pgmap v10770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:07 smithi033 ceph-mon[27845]: pgmap v10770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:07 smithi096 ceph-mon[32753]: pgmap v10770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19326.989274 secs 2024-04-26T17:13:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19326.989274 secs 2024-04-26T17:13:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19326.989274 secs 2024-04-26T17:13:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19329.833177 secs 2024-04-26T17:13:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:09 smithi194 ceph-mon[33017]: pgmap v10771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19329.833177 secs 2024-04-26T17:13:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:09 smithi033 ceph-mon[27845]: pgmap v10771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19329.833177 secs 2024-04-26T17:13:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:09 smithi096 ceph-mon[32753]: pgmap v10771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:11 smithi194 ceph-mon[33017]: pgmap v10772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:11 smithi033 ceph-mon[27845]: pgmap v10772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:11 smithi096 ceph-mon[32753]: pgmap v10772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:13:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:13 smithi194 ceph-mon[33017]: pgmap v10773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:13:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19331.989357 secs 2024-04-26T17:13:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:13 smithi033 ceph-mon[27845]: pgmap v10773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:13:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19331.989357 secs 2024-04-26T17:13:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:13 smithi096 ceph-mon[32753]: pgmap v10773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:13:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19331.989357 secs 2024-04-26T17:13:15.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19334.833300 secs 2024-04-26T17:13:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19334.833300 secs 2024-04-26T17:13:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19334.833300 secs 2024-04-26T17:13:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:15 smithi194 ceph-mon[33017]: pgmap v10774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:15 smithi033 ceph-mon[27845]: pgmap v10774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:15 smithi096 ceph-mon[32753]: pgmap v10774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:17.003+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:17.003+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:17.003+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:13:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:17 smithi194 ceph-mon[33017]: pgmap v10775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:17 smithi033 ceph-mon[27845]: pgmap v10775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:17 smithi096 ceph-mon[32753]: pgmap v10775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19336.989444 secs 2024-04-26T17:13:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19336.989444 secs 2024-04-26T17:13:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19336.989444 secs 2024-04-26T17:13:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19339.833365 secs 2024-04-26T17:13:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:19 smithi194 ceph-mon[33017]: pgmap v10776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19339.833365 secs 2024-04-26T17:13:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:19 smithi033 ceph-mon[27845]: pgmap v10776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19339.833365 secs 2024-04-26T17:13:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:19 smithi096 ceph-mon[32753]: pgmap v10776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:21 smithi194 ceph-mon[33017]: pgmap v10777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:21 smithi033 ceph-mon[27845]: pgmap v10777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:21 smithi096 ceph-mon[32753]: pgmap v10777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:23 smithi194 ceph-mon[33017]: pgmap v10778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19341.989524 secs 2024-04-26T17:13:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:23 smithi033 ceph-mon[27845]: pgmap v10778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19341.989524 secs 2024-04-26T17:13:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:23 smithi096 ceph-mon[32753]: pgmap v10778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19341.989524 secs 2024-04-26T17:13:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19344.833482 secs 2024-04-26T17:13:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19344.833482 secs 2024-04-26T17:13:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19344.833482 secs 2024-04-26T17:13:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:25 smithi033 ceph-mon[27845]: pgmap v10779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:25 smithi096 ceph-mon[32753]: pgmap v10779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:25 smithi194 ceph-mon[33017]: pgmap v10779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:27 smithi033 ceph-mon[27845]: pgmap v10780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:27 smithi096 ceph-mon[32753]: pgmap v10780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:27 smithi194 ceph-mon[33017]: pgmap v10780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19346.989608 secs 2024-04-26T17:13:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19346.989608 secs 2024-04-26T17:13:29.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19346.989608 secs 2024-04-26T17:13:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19349.833600 secs 2024-04-26T17:13:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:29 smithi033 ceph-mon[27845]: pgmap v10781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19349.833600 secs 2024-04-26T17:13:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:29 smithi096 ceph-mon[32753]: pgmap v10781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19349.833600 secs 2024-04-26T17:13:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:29 smithi194 ceph-mon[33017]: pgmap v10781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:32.009+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:32.009+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:32.009+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:13:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:31 smithi033 ceph-mon[27845]: pgmap v10782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:31 smithi096 ceph-mon[32753]: pgmap v10782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:31 smithi194 ceph-mon[33017]: pgmap v10782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:33 smithi033 ceph-mon[27845]: pgmap v10783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19351.989692 secs 2024-04-26T17:13:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:33 smithi096 ceph-mon[32753]: pgmap v10783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19351.989692 secs 2024-04-26T17:13:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:33 smithi194 ceph-mon[33017]: pgmap v10783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19351.989692 secs 2024-04-26T17:13:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19354.833717 secs 2024-04-26T17:13:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19354.833717 secs 2024-04-26T17:13:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19354.833717 secs 2024-04-26T17:13:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:35 smithi033 ceph-mon[27845]: pgmap v10784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:35 smithi096 ceph-mon[32753]: pgmap v10784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:35 smithi194 ceph-mon[33017]: pgmap v10784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:37 smithi033 ceph-mon[27845]: pgmap v10785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:37 smithi096 ceph-mon[32753]: pgmap v10785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:37 smithi194 ceph-mon[33017]: pgmap v10785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19356.989770 secs 2024-04-26T17:13:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19356.989770 secs 2024-04-26T17:13:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19356.989770 secs 2024-04-26T17:13:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19359.833842 secs 2024-04-26T17:13:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:39 smithi033 ceph-mon[27845]: pgmap v10786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19359.833842 secs 2024-04-26T17:13:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:39 smithi096 ceph-mon[32753]: pgmap v10786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19359.833842 secs 2024-04-26T17:13:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:39 smithi194 ceph-mon[33017]: pgmap v10786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:41 smithi033 ceph-mon[27845]: pgmap v10787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:41 smithi096 ceph-mon[32753]: pgmap v10787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:41 smithi194 ceph-mon[33017]: pgmap v10787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:43 smithi033 ceph-mon[27845]: pgmap v10788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19361.989852 secs 2024-04-26T17:13:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:43 smithi096 ceph-mon[32753]: pgmap v10788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19361.989852 secs 2024-04-26T17:13:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:43 smithi194 ceph-mon[33017]: pgmap v10788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19361.989852 secs 2024-04-26T17:13:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19364.833965 secs 2024-04-26T17:13:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19364.833965 secs 2024-04-26T17:13:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19364.833965 secs 2024-04-26T17:13:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:45 smithi194 ceph-mon[33017]: pgmap v10789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:45 smithi033 ceph-mon[27845]: pgmap v10789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:45 smithi096 ceph-mon[32753]: pgmap v10789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:47.023+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:47.023+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:13:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:13:47.023+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:13:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:47 smithi194 ceph-mon[33017]: pgmap v10790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:47 smithi033 ceph-mon[27845]: pgmap v10790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:47 smithi096 ceph-mon[32753]: pgmap v10790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:49.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19366.989932 secs 2024-04-26T17:13:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19366.989932 secs 2024-04-26T17:13:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19366.989932 secs 2024-04-26T17:13:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19369.834082 secs 2024-04-26T17:13:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:49 smithi194 ceph-mon[33017]: pgmap v10791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19369.834082 secs 2024-04-26T17:13:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:49 smithi033 ceph-mon[27845]: pgmap v10791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19369.834082 secs 2024-04-26T17:13:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:49 smithi096 ceph-mon[32753]: pgmap v10791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:51 smithi194 ceph-mon[33017]: pgmap v10792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:51 smithi033 ceph-mon[27845]: pgmap v10792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:51 smithi096 ceph-mon[32753]: pgmap v10792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:53 smithi194 ceph-mon[33017]: pgmap v10793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19371.990015 secs 2024-04-26T17:13:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:53 smithi033 ceph-mon[27845]: pgmap v10793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19371.990015 secs 2024-04-26T17:13:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:53 smithi096 ceph-mon[32753]: pgmap v10793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19371.990015 secs 2024-04-26T17:13:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19374.834197 secs 2024-04-26T17:13:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19374.834197 secs 2024-04-26T17:13:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19374.834197 secs 2024-04-26T17:13:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:55 smithi194 ceph-mon[33017]: pgmap v10794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:55 smithi033 ceph-mon[27845]: pgmap v10794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:55 smithi096 ceph-mon[32753]: pgmap v10794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:13:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:57 smithi194 ceph-mon[33017]: pgmap v10795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:57 smithi033 ceph-mon[27845]: pgmap v10795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:57 smithi096 ceph-mon[32753]: pgmap v10795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:13:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19376.990093 secs 2024-04-26T17:13:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19376.990093 secs 2024-04-26T17:13:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19376.990093 secs 2024-04-26T17:14:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19379.834348 secs 2024-04-26T17:14:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:13:59 smithi194 ceph-mon[33017]: pgmap v10796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19379.834348 secs 2024-04-26T17:14:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:13:59 smithi033 ceph-mon[27845]: pgmap v10796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19379.834348 secs 2024-04-26T17:14:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:13:59 smithi096 ceph-mon[32753]: pgmap v10796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:02.038+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:02.038+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:02.038+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:14:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:02 smithi194 ceph-mon[33017]: pgmap v10797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:02 smithi033 ceph-mon[27845]: pgmap v10797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:02 smithi096 ceph-mon[32753]: pgmap v10797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: pgmap v10798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19381.990174 secs 2024-04-26T17:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19384.834433 secs 2024-04-26T17:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: pgmap v10798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:04.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19381.990174 secs 2024-04-26T17:14:04.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:14:04.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19384.834433 secs 2024-04-26T17:14:04.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.408 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.408 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.408 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: pgmap v10798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:04.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19381.990174 secs 2024-04-26T17:14:04.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:14:04.411 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19384.834433 secs 2024-04-26T17:14:04.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:04.412 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:06 smithi194 ceph-mon[33017]: pgmap v10799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:06 smithi033 ceph-mon[27845]: pgmap v10799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:06 smithi096 ceph-mon[32753]: pgmap v10799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:08 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:14:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:08 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:14:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:08 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:14:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:08 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:14:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:08 smithi194 ceph-mon[33017]: pgmap v10800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:08 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:14:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:08 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:14:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:08 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:14:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:08 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:14:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:08 smithi033 ceph-mon[27845]: pgmap v10800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:08 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:14:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:08 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:14:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:08 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:14:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:08 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:14:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:08 smithi096 ceph-mon[32753]: pgmap v10800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19386.990259 secs 2024-04-26T17:14:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19389.834533 secs 2024-04-26T17:14:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19386.990259 secs 2024-04-26T17:14:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19389.834533 secs 2024-04-26T17:14:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19386.990259 secs 2024-04-26T17:14:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19389.834533 secs 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: pgmap v10801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:14:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:14:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:14:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:14:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: pgmap v10802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:14:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: pgmap v10803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:14:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: pgmap v10801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:14:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:14:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:14:10.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:14:10.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: pgmap v10802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:14:10.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: pgmap v10803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:14:10.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: pgmap v10801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:14:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:14:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:14:10.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:14:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:14:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: pgmap v10802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:14:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: pgmap v10803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:14:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:14:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:11 smithi194 ceph-mon[33017]: pgmap v10804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:11 smithi033 ceph-mon[27845]: pgmap v10804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:11 smithi096 ceph-mon[32753]: pgmap v10804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:13 smithi194 ceph-mon[33017]: pgmap v10805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:14:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:13 smithi033 ceph-mon[27845]: pgmap v10805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:14:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:13 smithi096 ceph-mon[32753]: pgmap v10805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:14:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19391.990340 secs 2024-04-26T17:14:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19394.834671 secs 2024-04-26T17:14:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19391.990340 secs 2024-04-26T17:14:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19394.834671 secs 2024-04-26T17:14:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19391.990340 secs 2024-04-26T17:14:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19394.834671 secs 2024-04-26T17:14:15.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:15 smithi033 ceph-mon[27845]: pgmap v10806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:15 smithi194 ceph-mon[33017]: pgmap v10806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:15 smithi096 ceph-mon[32753]: pgmap v10806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:17.054+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:17.054+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:17.054+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:14:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:17 smithi194 ceph-mon[33017]: pgmap v10807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:17 smithi033 ceph-mon[27845]: pgmap v10807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:17 smithi096 ceph-mon[32753]: pgmap v10807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:14:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19396.990423 secs 2024-04-26T17:14:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19396.990423 secs 2024-04-26T17:14:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19396.990423 secs 2024-04-26T17:14:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19399.834804 secs 2024-04-26T17:14:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:19 smithi033 ceph-mon[27845]: pgmap v10808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:14:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19399.834804 secs 2024-04-26T17:14:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:19 smithi096 ceph-mon[32753]: pgmap v10808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:14:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19399.834804 secs 2024-04-26T17:14:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:19 smithi194 ceph-mon[33017]: pgmap v10808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:14:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:21 smithi194 ceph-mon[33017]: pgmap v10809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:21 smithi033 ceph-mon[27845]: pgmap v10809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:21 smithi096 ceph-mon[32753]: pgmap v10809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:23 smithi194 ceph-mon[33017]: pgmap v10810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:23 smithi033 ceph-mon[27845]: pgmap v10810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:23 smithi096 ceph-mon[32753]: pgmap v10810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19401.990506 secs 2024-04-26T17:14:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19404.834942 secs 2024-04-26T17:14:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19401.990506 secs 2024-04-26T17:14:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19404.834942 secs 2024-04-26T17:14:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19401.990506 secs 2024-04-26T17:14:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19404.834942 secs 2024-04-26T17:14:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:25.089+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:14:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:25.089+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T17:14:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:25.090+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 156 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T17:14:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:25.090+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 156 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:14:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:25 smithi194 ceph-mon[33017]: pgmap v10811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:25 smithi033 ceph-mon[27845]: pgmap v10811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:25 smithi096 ceph-mon[32753]: pgmap v10811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:27 smithi194 ceph-mon[33017]: pgmap v10812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:27 smithi033 ceph-mon[27845]: pgmap v10812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:27 smithi096 ceph-mon[32753]: pgmap v10812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19406.990588 secs 2024-04-26T17:14:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19406.990588 secs 2024-04-26T17:14:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19406.990588 secs 2024-04-26T17:14:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19409.835056 secs 2024-04-26T17:14:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:29 smithi033 ceph-mon[27845]: pgmap v10813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19409.835056 secs 2024-04-26T17:14:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:29 smithi096 ceph-mon[32753]: pgmap v10813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19409.835056 secs 2024-04-26T17:14:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:29 smithi194 ceph-mon[33017]: pgmap v10813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:31 smithi194 ceph-mon[33017]: pgmap v10814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:31 smithi033 ceph-mon[27845]: pgmap v10814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:31 smithi096 ceph-mon[32753]: pgmap v10814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:32.054+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:32.054+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:32.054+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:14:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:33 smithi194 ceph-mon[33017]: pgmap v10815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:33 smithi033 ceph-mon[27845]: pgmap v10815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:33 smithi096 ceph-mon[32753]: pgmap v10815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19411.990668 secs 2024-04-26T17:14:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19414.835178 secs 2024-04-26T17:14:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19411.990668 secs 2024-04-26T17:14:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19414.835178 secs 2024-04-26T17:14:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19411.990668 secs 2024-04-26T17:14:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19414.835178 secs 2024-04-26T17:14:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:35 smithi194 ceph-mon[33017]: pgmap v10816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:35 smithi033 ceph-mon[27845]: pgmap v10816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:35 smithi096 ceph-mon[32753]: pgmap v10816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:37.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:37 smithi096 ceph-mon[32753]: pgmap v10817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:37 smithi194 ceph-mon[33017]: pgmap v10817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:37 smithi033 ceph-mon[27845]: pgmap v10817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19416.990761 secs 2024-04-26T17:14:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19416.990761 secs 2024-04-26T17:14:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19416.990761 secs 2024-04-26T17:14:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19419.835269 secs 2024-04-26T17:14:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:39 smithi033 ceph-mon[27845]: pgmap v10818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19419.835269 secs 2024-04-26T17:14:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:39 smithi096 ceph-mon[32753]: pgmap v10818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19419.835269 secs 2024-04-26T17:14:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:39 smithi194 ceph-mon[33017]: pgmap v10818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:41 smithi194 ceph-mon[33017]: pgmap v10819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:41 smithi033 ceph-mon[27845]: pgmap v10819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:41 smithi096 ceph-mon[32753]: pgmap v10819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:43 smithi194 ceph-mon[33017]: pgmap v10820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:43 smithi033 ceph-mon[27845]: pgmap v10820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:43 smithi096 ceph-mon[32753]: pgmap v10820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19421.990842 secs 2024-04-26T17:14:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19424.835396 secs 2024-04-26T17:14:44.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19421.990842 secs 2024-04-26T17:14:44.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19424.835396 secs 2024-04-26T17:14:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19421.990842 secs 2024-04-26T17:14:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19424.835396 secs 2024-04-26T17:14:45.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:45 smithi033 ceph-mon[27845]: pgmap v10821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:45 smithi194 ceph-mon[33017]: pgmap v10821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:45 smithi096 ceph-mon[32753]: pgmap v10821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:47.059+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:47.059+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:14:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:14:47.059+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:14:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:47 smithi194 ceph-mon[33017]: pgmap v10822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:47 smithi033 ceph-mon[27845]: pgmap v10822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:47 smithi096 ceph-mon[32753]: pgmap v10822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19426.990921 secs 2024-04-26T17:14:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19426.990921 secs 2024-04-26T17:14:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19426.990921 secs 2024-04-26T17:14:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19429.835513 secs 2024-04-26T17:14:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:49 smithi033 ceph-mon[27845]: pgmap v10823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19429.835513 secs 2024-04-26T17:14:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:49 smithi096 ceph-mon[32753]: pgmap v10823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19429.835513 secs 2024-04-26T17:14:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:49 smithi194 ceph-mon[33017]: pgmap v10823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:51 smithi194 ceph-mon[33017]: pgmap v10824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:51 smithi033 ceph-mon[27845]: pgmap v10824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:51 smithi096 ceph-mon[32753]: pgmap v10824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:53 smithi194 ceph-mon[33017]: pgmap v10825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:53 smithi033 ceph-mon[27845]: pgmap v10825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:53 smithi096 ceph-mon[32753]: pgmap v10825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19431.990995 secs 2024-04-26T17:14:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19434.835596 secs 2024-04-26T17:14:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19431.990995 secs 2024-04-26T17:14:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19434.835596 secs 2024-04-26T17:14:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19431.990995 secs 2024-04-26T17:14:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19434.835596 secs 2024-04-26T17:14:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:55 smithi194 ceph-mon[33017]: pgmap v10826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:55 smithi033 ceph-mon[27845]: pgmap v10826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:55 smithi096 ceph-mon[32753]: pgmap v10826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:14:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:57 smithi194 ceph-mon[33017]: pgmap v10827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:57 smithi033 ceph-mon[27845]: pgmap v10827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:57 smithi096 ceph-mon[32753]: pgmap v10827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19436.991054 secs 2024-04-26T17:14:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19436.991054 secs 2024-04-26T17:14:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19436.991054 secs 2024-04-26T17:14:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19439.835733 secs 2024-04-26T17:14:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:14:59 smithi033 ceph-mon[27845]: pgmap v10828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:14:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19439.835733 secs 2024-04-26T17:14:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:14:59 smithi096 ceph-mon[32753]: pgmap v10828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19439.835733 secs 2024-04-26T17:15:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:14:59 smithi194 ceph-mon[33017]: pgmap v10828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:01 smithi194 ceph-mon[33017]: pgmap v10829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:01 smithi033 ceph-mon[27845]: pgmap v10829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:01 smithi096 ceph-mon[32753]: pgmap v10829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:02.075+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:02.075+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:02.075+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:15:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:03 smithi194 ceph-mon[33017]: pgmap v10830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:03 smithi033 ceph-mon[27845]: pgmap v10830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:03 smithi096 ceph-mon[32753]: pgmap v10830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19441.991144 secs 2024-04-26T17:15:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19444.835865 secs 2024-04-26T17:15:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19441.991144 secs 2024-04-26T17:15:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19444.835865 secs 2024-04-26T17:15:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19441.991144 secs 2024-04-26T17:15:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19444.835865 secs 2024-04-26T17:15:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:05 smithi194 ceph-mon[33017]: pgmap v10831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:05 smithi033 ceph-mon[27845]: pgmap v10831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:05 smithi096 ceph-mon[32753]: pgmap v10831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:07.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:07 smithi096 ceph-mon[32753]: pgmap v10832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:07 smithi194 ceph-mon[33017]: pgmap v10832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:07 smithi033 ceph-mon[27845]: pgmap v10832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19446.991207 secs 2024-04-26T17:15:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19446.991207 secs 2024-04-26T17:15:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19446.991207 secs 2024-04-26T17:15:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19449.835998 secs 2024-04-26T17:15:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:09 smithi033 ceph-mon[27845]: pgmap v10833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:15:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19449.835998 secs 2024-04-26T17:15:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:09 smithi096 ceph-mon[32753]: pgmap v10833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:15:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19449.835998 secs 2024-04-26T17:15:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:09 smithi194 ceph-mon[33017]: pgmap v10833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:15:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:15:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:15:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:15:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:15:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:15:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:15:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:15:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:15:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:15:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:11 smithi194 ceph-mon[33017]: pgmap v10834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:15:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:11 smithi194 ceph-mon[33017]: pgmap v10835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:11 smithi033 ceph-mon[27845]: pgmap v10834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:15:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:11 smithi033 ceph-mon[27845]: pgmap v10835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:11 smithi096 ceph-mon[32753]: pgmap v10834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:15:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:11 smithi096 ceph-mon[32753]: pgmap v10835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19451.991279 secs 2024-04-26T17:15:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:13 smithi033 ceph-mon[27845]: pgmap v10836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19451.991279 secs 2024-04-26T17:15:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:13 smithi096 ceph-mon[32753]: pgmap v10836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19451.991279 secs 2024-04-26T17:15:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:13 smithi194 ceph-mon[33017]: pgmap v10836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19454.836140 secs 2024-04-26T17:15:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19454.836140 secs 2024-04-26T17:15:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19454.836140 secs 2024-04-26T17:15:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:15 smithi033 ceph-mon[27845]: pgmap v10837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:15 smithi096 ceph-mon[32753]: pgmap v10837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:15 smithi194 ceph-mon[33017]: pgmap v10837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:17.088+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:17.089+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:17.089+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:15:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:17 smithi033 ceph-mon[27845]: pgmap v10838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:17 smithi096 ceph-mon[32753]: pgmap v10838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:17 smithi194 ceph-mon[33017]: pgmap v10838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19456.991360 secs 2024-04-26T17:15:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19456.991360 secs 2024-04-26T17:15:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19456.991360 secs 2024-04-26T17:15:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19459.836263 secs 2024-04-26T17:15:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:19 smithi033 ceph-mon[27845]: pgmap v10839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19459.836263 secs 2024-04-26T17:15:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:19 smithi096 ceph-mon[32753]: pgmap v10839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19459.836263 secs 2024-04-26T17:15:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:19 smithi194 ceph-mon[33017]: pgmap v10839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:15:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:21 smithi033 ceph-mon[27845]: pgmap v10840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:15:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:21 smithi096 ceph-mon[32753]: pgmap v10840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:15:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:21 smithi194 ceph-mon[33017]: pgmap v10840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:15:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19461.991438 secs 2024-04-26T17:15:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:23 smithi033 ceph-mon[27845]: pgmap v10841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19461.991438 secs 2024-04-26T17:15:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:23 smithi096 ceph-mon[32753]: pgmap v10841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19461.991438 secs 2024-04-26T17:15:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:23 smithi194 ceph-mon[33017]: pgmap v10841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19464.836377 secs 2024-04-26T17:15:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19464.836377 secs 2024-04-26T17:15:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19464.836377 secs 2024-04-26T17:15:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:25 smithi033 ceph-mon[27845]: pgmap v10842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:25 smithi096 ceph-mon[32753]: pgmap v10842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:25 smithi194 ceph-mon[33017]: pgmap v10842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:27 smithi033 ceph-mon[27845]: pgmap v10843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:27 smithi096 ceph-mon[32753]: pgmap v10843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:27 smithi194 ceph-mon[33017]: pgmap v10843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19466.991520 secs 2024-04-26T17:15:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19466.991520 secs 2024-04-26T17:15:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19466.991520 secs 2024-04-26T17:15:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19469.836494 secs 2024-04-26T17:15:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:29 smithi096 ceph-mon[32753]: pgmap v10844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19469.836494 secs 2024-04-26T17:15:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:29 smithi194 ceph-mon[33017]: pgmap v10844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19469.836494 secs 2024-04-26T17:15:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:29 smithi033 ceph-mon[27845]: pgmap v10844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:32.103+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:32.103+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:32.103+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:15:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:31 smithi194 ceph-mon[33017]: pgmap v10845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:31 smithi033 ceph-mon[27845]: pgmap v10845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:31 smithi096 ceph-mon[32753]: pgmap v10845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19471.991602 secs 2024-04-26T17:15:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:33 smithi194 ceph-mon[33017]: pgmap v10846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19471.991602 secs 2024-04-26T17:15:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:33 smithi033 ceph-mon[27845]: pgmap v10846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19471.991602 secs 2024-04-26T17:15:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:33 smithi096 ceph-mon[32753]: pgmap v10846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19474.836618 secs 2024-04-26T17:15:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19474.836618 secs 2024-04-26T17:15:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19474.836618 secs 2024-04-26T17:15:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:35 smithi194 ceph-mon[33017]: pgmap v10847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:35 smithi033 ceph-mon[27845]: pgmap v10847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:35 smithi096 ceph-mon[32753]: pgmap v10847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:38.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:37 smithi096 ceph-mon[32753]: pgmap v10848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:37 smithi194 ceph-mon[33017]: pgmap v10848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:37 smithi033 ceph-mon[27845]: pgmap v10848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19476.991678 secs 2024-04-26T17:15:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19476.991678 secs 2024-04-26T17:15:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19476.991678 secs 2024-04-26T17:15:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19479.836754 secs 2024-04-26T17:15:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:39 smithi194 ceph-mon[33017]: pgmap v10849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19479.836754 secs 2024-04-26T17:15:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:39 smithi033 ceph-mon[27845]: pgmap v10849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19479.836754 secs 2024-04-26T17:15:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:39 smithi096 ceph-mon[32753]: pgmap v10849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:41 smithi194 ceph-mon[33017]: pgmap v10850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:41 smithi033 ceph-mon[27845]: pgmap v10850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:41 smithi096 ceph-mon[32753]: pgmap v10850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19481.991764 secs 2024-04-26T17:15:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:43 smithi194 ceph-mon[33017]: pgmap v10851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19481.991764 secs 2024-04-26T17:15:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:43 smithi033 ceph-mon[27845]: pgmap v10851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19481.991764 secs 2024-04-26T17:15:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:43 smithi096 ceph-mon[32753]: pgmap v10851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:45.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19484.836890 secs 2024-04-26T17:15:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19484.836890 secs 2024-04-26T17:15:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19484.836890 secs 2024-04-26T17:15:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:45 smithi194 ceph-mon[33017]: pgmap v10852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:45 smithi033 ceph-mon[27845]: pgmap v10852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:45 smithi096 ceph-mon[32753]: pgmap v10852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:47.109+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:47.109+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:15:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:15:47.109+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:15:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:48 smithi194 ceph-mon[33017]: pgmap v10853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:48 smithi033 ceph-mon[27845]: pgmap v10853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:48 smithi096 ceph-mon[32753]: pgmap v10853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19486.991843 secs 2024-04-26T17:15:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19486.991843 secs 2024-04-26T17:15:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19486.991843 secs 2024-04-26T17:15:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19489.837024 secs 2024-04-26T17:15:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:50 smithi194 ceph-mon[33017]: pgmap v10854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19489.837024 secs 2024-04-26T17:15:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:50 smithi033 ceph-mon[27845]: pgmap v10854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19489.837024 secs 2024-04-26T17:15:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:50 smithi096 ceph-mon[32753]: pgmap v10854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:52 smithi194 ceph-mon[33017]: pgmap v10855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:52 smithi033 ceph-mon[27845]: pgmap v10855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:52 smithi096 ceph-mon[32753]: pgmap v10855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19491.991902 secs 2024-04-26T17:15:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:54 smithi194 ceph-mon[33017]: pgmap v10856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19494.837178 secs 2024-04-26T17:15:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19491.991902 secs 2024-04-26T17:15:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:54 smithi033 ceph-mon[27845]: pgmap v10856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19494.837178 secs 2024-04-26T17:15:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19491.991902 secs 2024-04-26T17:15:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:54 smithi096 ceph-mon[32753]: pgmap v10856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19494.837178 secs 2024-04-26T17:15:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:56 smithi194 ceph-mon[33017]: pgmap v10857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:56 smithi033 ceph-mon[27845]: pgmap v10857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:56 smithi096 ceph-mon[32753]: pgmap v10857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:15:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:58 smithi194 ceph-mon[33017]: pgmap v10858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:58 smithi033 ceph-mon[27845]: pgmap v10858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:58 smithi096 ceph-mon[32753]: pgmap v10858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:15:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19496.992003 secs 2024-04-26T17:15:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:15:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19499.837298 secs 2024-04-26T17:15:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19496.992003 secs 2024-04-26T17:15:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:15:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19499.837298 secs 2024-04-26T17:15:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19496.992003 secs 2024-04-26T17:15:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:15:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19499.837298 secs 2024-04-26T17:16:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:00 smithi194 ceph-mon[33017]: pgmap v10859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:00 smithi033 ceph-mon[27845]: pgmap v10859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:00 smithi096 ceph-mon[32753]: pgmap v10859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:02.126 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:02.125+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:02.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:02.125+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:16:02.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:02.125+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:02 smithi194 ceph-mon[33017]: pgmap v10860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:02 smithi033 ceph-mon[27845]: pgmap v10860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:02 smithi096 ceph-mon[32753]: pgmap v10860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19501.992092 secs 2024-04-26T17:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:04 smithi194 ceph-mon[33017]: pgmap v10861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19504.837418 secs 2024-04-26T17:16:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19501.992092 secs 2024-04-26T17:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:04 smithi033 ceph-mon[27845]: pgmap v10861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19504.837418 secs 2024-04-26T17:16:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19501.992092 secs 2024-04-26T17:16:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:04 smithi096 ceph-mon[32753]: pgmap v10861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19504.837418 secs 2024-04-26T17:16:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:06 smithi194 ceph-mon[33017]: pgmap v10862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:06 smithi033 ceph-mon[27845]: pgmap v10862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:06 smithi096 ceph-mon[32753]: pgmap v10862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:08 smithi194 ceph-mon[33017]: pgmap v10863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:08 smithi033 ceph-mon[27845]: pgmap v10863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:08 smithi096 ceph-mon[32753]: pgmap v10863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19506.992169 secs 2024-04-26T17:16:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19509.837571 secs 2024-04-26T17:16:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19506.992169 secs 2024-04-26T17:16:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19509.837571 secs 2024-04-26T17:16:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19506.992169 secs 2024-04-26T17:16:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19509.837571 secs 2024-04-26T17:16:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:10 smithi033 ceph-mon[27845]: pgmap v10864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:10 smithi194 ceph-mon[33017]: pgmap v10864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:10 smithi096 ceph-mon[32753]: pgmap v10864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:12 smithi194 ceph-mon[33017]: pgmap v10865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:16:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:12 smithi033 ceph-mon[27845]: pgmap v10865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:16:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:12 smithi096 ceph-mon[32753]: pgmap v10865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:16:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:16:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:16:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:16:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:16:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:16:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:16:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:16:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:16:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:16:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:14 smithi194 ceph-mon[33017]: pgmap v10866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:14 smithi194 ceph-mon[33017]: pgmap v10867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19511.992234 secs 2024-04-26T17:16:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19514.837692 secs 2024-04-26T17:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:14 smithi033 ceph-mon[27845]: pgmap v10866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:14 smithi033 ceph-mon[27845]: pgmap v10867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19511.992234 secs 2024-04-26T17:16:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19514.837692 secs 2024-04-26T17:16:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:14 smithi096 ceph-mon[32753]: pgmap v10866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:14 smithi096 ceph-mon[32753]: pgmap v10867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19511.992234 secs 2024-04-26T17:16:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19514.837692 secs 2024-04-26T17:16:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:16 smithi033 ceph-mon[27845]: pgmap v10868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:16 smithi096 ceph-mon[32753]: pgmap v10868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:16 smithi194 ceph-mon[33017]: pgmap v10868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:17.140+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:17.140+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:17.141+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:16:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:18 smithi033 ceph-mon[27845]: pgmap v10869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:18 smithi096 ceph-mon[32753]: pgmap v10869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:18 smithi194 ceph-mon[33017]: pgmap v10869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19516.992322 secs 2024-04-26T17:16:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19519.837818 secs 2024-04-26T17:16:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19516.992322 secs 2024-04-26T17:16:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19519.837818 secs 2024-04-26T17:16:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19516.992322 secs 2024-04-26T17:16:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19519.837818 secs 2024-04-26T17:16:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:20 smithi033 ceph-mon[27845]: pgmap v10870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:20 smithi096 ceph-mon[32753]: pgmap v10870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:20 smithi194 ceph-mon[33017]: pgmap v10870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:22 smithi033 ceph-mon[27845]: pgmap v10871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:22 smithi096 ceph-mon[32753]: pgmap v10871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:22 smithi194 ceph-mon[33017]: pgmap v10871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:16:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:24 smithi033 ceph-mon[27845]: pgmap v10872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19521.992419 secs 2024-04-26T17:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19524.837937 secs 2024-04-26T17:16:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:24 smithi096 ceph-mon[32753]: pgmap v10872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:16:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19521.992419 secs 2024-04-26T17:16:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19524.837937 secs 2024-04-26T17:16:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:24 smithi194 ceph-mon[33017]: pgmap v10872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:16:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19521.992419 secs 2024-04-26T17:16:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19524.837937 secs 2024-04-26T17:16:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:26 smithi033 ceph-mon[27845]: pgmap v10873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:26 smithi096 ceph-mon[32753]: pgmap v10873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:26 smithi194 ceph-mon[33017]: pgmap v10873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:28 smithi033 ceph-mon[27845]: pgmap v10874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:28 smithi096 ceph-mon[32753]: pgmap v10874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:28 smithi194 ceph-mon[33017]: pgmap v10874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19526.992522 secs 2024-04-26T17:16:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19529.838073 secs 2024-04-26T17:16:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19526.992522 secs 2024-04-26T17:16:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19529.838073 secs 2024-04-26T17:16:29.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19526.992522 secs 2024-04-26T17:16:29.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19529.838073 secs 2024-04-26T17:16:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:30 smithi033 ceph-mon[27845]: pgmap v10875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:30 smithi096 ceph-mon[32753]: pgmap v10875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:30 smithi194 ceph-mon[33017]: pgmap v10875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:32.151+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:32.151+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:16:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:32.151+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:32 smithi033 ceph-mon[27845]: pgmap v10876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:32 smithi096 ceph-mon[32753]: pgmap v10876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:32 smithi194 ceph-mon[33017]: pgmap v10876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:34 smithi033 ceph-mon[27845]: pgmap v10877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19531.992623 secs 2024-04-26T17:16:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19534.838196 secs 2024-04-26T17:16:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:34 smithi096 ceph-mon[32753]: pgmap v10877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19531.992623 secs 2024-04-26T17:16:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19534.838196 secs 2024-04-26T17:16:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:34 smithi194 ceph-mon[33017]: pgmap v10877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19531.992623 secs 2024-04-26T17:16:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19534.838196 secs 2024-04-26T17:16:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:35 smithi194 ceph-mon[33017]: pgmap v10878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:35 smithi033 ceph-mon[27845]: pgmap v10878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:35 smithi096 ceph-mon[32753]: pgmap v10878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:37 smithi194 ceph-mon[33017]: pgmap v10879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:37 smithi033 ceph-mon[27845]: pgmap v10879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:37 smithi096 ceph-mon[32753]: pgmap v10879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19536.992724 secs 2024-04-26T17:16:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19536.992724 secs 2024-04-26T17:16:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19536.992724 secs 2024-04-26T17:16:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19539.838313 secs 2024-04-26T17:16:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:39 smithi194 ceph-mon[33017]: pgmap v10880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19539.838313 secs 2024-04-26T17:16:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:39 smithi033 ceph-mon[27845]: pgmap v10880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19539.838313 secs 2024-04-26T17:16:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:39 smithi096 ceph-mon[32753]: pgmap v10880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:41 smithi033 ceph-mon[27845]: pgmap v10881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:41 smithi096 ceph-mon[32753]: pgmap v10881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:41 smithi194 ceph-mon[33017]: pgmap v10881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:43 smithi033 ceph-mon[27845]: pgmap v10882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19541.992810 secs 2024-04-26T17:16:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:43 smithi096 ceph-mon[32753]: pgmap v10882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19541.992810 secs 2024-04-26T17:16:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:43 smithi194 ceph-mon[33017]: pgmap v10882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19541.992810 secs 2024-04-26T17:16:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19544.838376 secs 2024-04-26T17:16:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19544.838376 secs 2024-04-26T17:16:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19544.838376 secs 2024-04-26T17:16:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:45 smithi033 ceph-mon[27845]: pgmap v10883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:45 smithi096 ceph-mon[32753]: pgmap v10883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:45 smithi194 ceph-mon[33017]: pgmap v10883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:47.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:47.156+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:47.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:47.156+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:16:47.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:16:47.156+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:16:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:47 smithi033 ceph-mon[27845]: pgmap v10884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:47 smithi096 ceph-mon[32753]: pgmap v10884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:47 smithi194 ceph-mon[33017]: pgmap v10884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19546.992919 secs 2024-04-26T17:16:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19546.992919 secs 2024-04-26T17:16:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19546.992919 secs 2024-04-26T17:16:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19549.838502 secs 2024-04-26T17:16:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:49 smithi033 ceph-mon[27845]: pgmap v10885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19549.838502 secs 2024-04-26T17:16:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:49 smithi096 ceph-mon[32753]: pgmap v10885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19549.838502 secs 2024-04-26T17:16:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:49 smithi194 ceph-mon[33017]: pgmap v10885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:51 smithi033 ceph-mon[27845]: pgmap v10886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:51 smithi096 ceph-mon[32753]: pgmap v10886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:51 smithi194 ceph-mon[33017]: pgmap v10886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:16:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:53 smithi033 ceph-mon[27845]: pgmap v10887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19551.993010 secs 2024-04-26T17:16:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:53 smithi096 ceph-mon[32753]: pgmap v10887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19551.993010 secs 2024-04-26T17:16:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:53 smithi194 ceph-mon[33017]: pgmap v10887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:16:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19551.993010 secs 2024-04-26T17:16:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19554.838565 secs 2024-04-26T17:16:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19554.838565 secs 2024-04-26T17:16:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19554.838565 secs 2024-04-26T17:16:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:55 smithi033 ceph-mon[27845]: pgmap v10888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:16:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:55 smithi096 ceph-mon[32753]: pgmap v10888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:16:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:55 smithi194 ceph-mon[33017]: pgmap v10888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:16:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:57 smithi033 ceph-mon[27845]: pgmap v10889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:16:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:57 smithi096 ceph-mon[32753]: pgmap v10889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:16:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:57 smithi194 ceph-mon[33017]: pgmap v10889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:16:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19556.993090 secs 2024-04-26T17:16:59.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19556.993090 secs 2024-04-26T17:16:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19556.993090 secs 2024-04-26T17:16:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19559.838682 secs 2024-04-26T17:16:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:16:59 smithi096 ceph-mon[32753]: pgmap v10890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19559.838682 secs 2024-04-26T17:17:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:16:59 smithi194 ceph-mon[33017]: pgmap v10890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19559.838682 secs 2024-04-26T17:17:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:16:59 smithi033 ceph-mon[27845]: pgmap v10890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:01 smithi194 ceph-mon[33017]: pgmap v10891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:02.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:02.168+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:02.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:02.168+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:02.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:02.168+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:17:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:01 smithi033 ceph-mon[27845]: pgmap v10891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:01 smithi096 ceph-mon[32753]: pgmap v10891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:03 smithi194 ceph-mon[33017]: pgmap v10892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19561.993148 secs 2024-04-26T17:17:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:03 smithi033 ceph-mon[27845]: pgmap v10892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19561.993148 secs 2024-04-26T17:17:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:03 smithi096 ceph-mon[32753]: pgmap v10892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19561.993148 secs 2024-04-26T17:17:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19564.838824 secs 2024-04-26T17:17:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19564.838824 secs 2024-04-26T17:17:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19564.838824 secs 2024-04-26T17:17:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:05 smithi194 ceph-mon[33017]: pgmap v10893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:05 smithi033 ceph-mon[27845]: pgmap v10893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:05 smithi096 ceph-mon[32753]: pgmap v10893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:07 smithi194 ceph-mon[33017]: pgmap v10894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:07 smithi033 ceph-mon[27845]: pgmap v10894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:07 smithi096 ceph-mon[32753]: pgmap v10894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:09.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19566.993210 secs 2024-04-26T17:17:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19566.993210 secs 2024-04-26T17:17:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19566.993210 secs 2024-04-26T17:17:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19569.838938 secs 2024-04-26T17:17:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:09 smithi194 ceph-mon[33017]: pgmap v10895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19569.838938 secs 2024-04-26T17:17:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:09 smithi033 ceph-mon[27845]: pgmap v10895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19569.838938 secs 2024-04-26T17:17:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:09 smithi096 ceph-mon[32753]: pgmap v10895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:11 smithi194 ceph-mon[33017]: pgmap v10896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:11 smithi033 ceph-mon[27845]: pgmap v10896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:11 smithi096 ceph-mon[32753]: pgmap v10896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:13 smithi194 ceph-mon[33017]: pgmap v10897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19571.993271 secs 2024-04-26T17:17:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:17:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:13 smithi033 ceph-mon[27845]: pgmap v10897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19571.993271 secs 2024-04-26T17:17:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:17:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:13 smithi096 ceph-mon[32753]: pgmap v10897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19571.993271 secs 2024-04-26T17:17:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:17:15.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19574.839036 secs 2024-04-26T17:17:15.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:17:15.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:17:15.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:17:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19574.839036 secs 2024-04-26T17:17:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:17:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:17:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:17:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19574.839036 secs 2024-04-26T17:17:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:17:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:17:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:17:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:15 smithi194 ceph-mon[33017]: pgmap v10898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:15 smithi194 ceph-mon[33017]: pgmap v10899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T17:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:15 smithi033 ceph-mon[27845]: pgmap v10898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:15 smithi033 ceph-mon[27845]: pgmap v10899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T17:17:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:15 smithi096 ceph-mon[32753]: pgmap v10898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:15 smithi096 ceph-mon[32753]: pgmap v10899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T17:17:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:17.181+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:17.181+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:17.182+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:17:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:17 smithi194 ceph-mon[33017]: pgmap v10900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:17 smithi033 ceph-mon[27845]: pgmap v10900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:17 smithi096 ceph-mon[32753]: pgmap v10900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19576.993332 secs 2024-04-26T17:17:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19576.993332 secs 2024-04-26T17:17:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19576.993332 secs 2024-04-26T17:17:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:19 smithi194 ceph-mon[33017]: pgmap v10901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19579.839181 secs 2024-04-26T17:17:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:19 smithi033 ceph-mon[27845]: pgmap v10901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19579.839181 secs 2024-04-26T17:17:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:19 smithi096 ceph-mon[32753]: pgmap v10901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19579.839181 secs 2024-04-26T17:17:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:21 smithi194 ceph-mon[33017]: pgmap v10902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:17:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:21 smithi033 ceph-mon[27845]: pgmap v10902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:17:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:21 smithi096 ceph-mon[32753]: pgmap v10902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:17:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:23 smithi194 ceph-mon[33017]: pgmap v10903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:17:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19581.993411 secs 2024-04-26T17:17:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:23 smithi033 ceph-mon[27845]: pgmap v10903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:17:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19581.993411 secs 2024-04-26T17:17:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:23 smithi096 ceph-mon[32753]: pgmap v10903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:17:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19581.993411 secs 2024-04-26T17:17:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19584.839297 secs 2024-04-26T17:17:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:25.122+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:17:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:25.122+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:17:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:25.123+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 157 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006630 con 0x564385821900 2024-04-26T17:17:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:25.123+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 157 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:17:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19584.839297 secs 2024-04-26T17:17:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19584.839297 secs 2024-04-26T17:17:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:25 smithi194 ceph-mon[33017]: pgmap v10904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:17:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:25 smithi033 ceph-mon[27845]: pgmap v10904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:17:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:25 smithi096 ceph-mon[32753]: pgmap v10904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:17:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:27 smithi194 ceph-mon[33017]: pgmap v10905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:27 smithi033 ceph-mon[27845]: pgmap v10905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:27 smithi096 ceph-mon[32753]: pgmap v10905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19586.993494 secs 2024-04-26T17:17:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19586.993494 secs 2024-04-26T17:17:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19586.993494 secs 2024-04-26T17:17:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:29 smithi194 ceph-mon[33017]: pgmap v10906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19589.839362 secs 2024-04-26T17:17:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:29 smithi033 ceph-mon[27845]: pgmap v10906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19589.839362 secs 2024-04-26T17:17:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:29 smithi096 ceph-mon[32753]: pgmap v10906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:17:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19589.839362 secs 2024-04-26T17:17:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:31 smithi194 ceph-mon[33017]: pgmap v10907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:32.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:31 smithi033 ceph-mon[27845]: pgmap v10907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:31 smithi096 ceph-mon[32753]: pgmap v10907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:17:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:32.196+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:32.196+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:17:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:32.196+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:33 smithi194 ceph-mon[33017]: pgmap v10908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19591.993576 secs 2024-04-26T17:17:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:33 smithi033 ceph-mon[27845]: pgmap v10908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19591.993576 secs 2024-04-26T17:17:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:33 smithi096 ceph-mon[32753]: pgmap v10908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19591.993576 secs 2024-04-26T17:17:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19594.839472 secs 2024-04-26T17:17:35.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19594.839472 secs 2024-04-26T17:17:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19594.839472 secs 2024-04-26T17:17:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:35 smithi194 ceph-mon[33017]: pgmap v10909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:36.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:35 smithi033 ceph-mon[27845]: pgmap v10909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:35 smithi096 ceph-mon[32753]: pgmap v10909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:37 smithi194 ceph-mon[33017]: pgmap v10910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:37 smithi033 ceph-mon[27845]: pgmap v10910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:37 smithi096 ceph-mon[32753]: pgmap v10910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19596.993659 secs 2024-04-26T17:17:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19596.993659 secs 2024-04-26T17:17:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19596.993659 secs 2024-04-26T17:17:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:39 smithi194 ceph-mon[33017]: pgmap v10911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19599.839586 secs 2024-04-26T17:17:40.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:39 smithi033 ceph-mon[27845]: pgmap v10911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:40.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19599.839586 secs 2024-04-26T17:17:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:39 smithi096 ceph-mon[32753]: pgmap v10911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19599.839586 secs 2024-04-26T17:17:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:41 smithi194 ceph-mon[33017]: pgmap v10912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:42.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:41 smithi033 ceph-mon[27845]: pgmap v10912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:41 smithi096 ceph-mon[32753]: pgmap v10912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:43 smithi033 ceph-mon[27845]: pgmap v10913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19601.993737 secs 2024-04-26T17:17:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:43 smithi096 ceph-mon[32753]: pgmap v10913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19601.993737 secs 2024-04-26T17:17:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:43 smithi194 ceph-mon[33017]: pgmap v10913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19601.993737 secs 2024-04-26T17:17:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19604.839684 secs 2024-04-26T17:17:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19604.839684 secs 2024-04-26T17:17:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19604.839684 secs 2024-04-26T17:17:46.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:45 smithi033 ceph-mon[27845]: pgmap v10914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:17:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:45 smithi096 ceph-mon[32753]: pgmap v10914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:17:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:45 smithi194 ceph-mon[33017]: pgmap v10914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:17:47.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:47.200+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:47.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:47.200+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:17:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:17:47.200+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:17:48.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:47 smithi033 ceph-mon[27845]: pgmap v10915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:47 smithi096 ceph-mon[32753]: pgmap v10915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:47 smithi194 ceph-mon[33017]: pgmap v10915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19606.993821 secs 2024-04-26T17:17:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19606.993821 secs 2024-04-26T17:17:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19606.993821 secs 2024-04-26T17:17:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:49 smithi033 ceph-mon[27845]: pgmap v10916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19609.839780 secs 2024-04-26T17:17:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:49 smithi096 ceph-mon[32753]: pgmap v10916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19609.839780 secs 2024-04-26T17:17:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:49 smithi194 ceph-mon[33017]: pgmap v10916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19609.839780 secs 2024-04-26T17:17:52.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:51 smithi033 ceph-mon[27845]: pgmap v10917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:51 smithi096 ceph-mon[32753]: pgmap v10917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:51 smithi194 ceph-mon[33017]: pgmap v10917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:17:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:53 smithi033 ceph-mon[27845]: pgmap v10918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19611.993906 secs 2024-04-26T17:17:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:53 smithi096 ceph-mon[32753]: pgmap v10918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19611.993906 secs 2024-04-26T17:17:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:53 smithi194 ceph-mon[33017]: pgmap v10918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:17:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19611.993906 secs 2024-04-26T17:17:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19614.839919 secs 2024-04-26T17:17:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19614.839919 secs 2024-04-26T17:17:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19614.839919 secs 2024-04-26T17:17:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:55 smithi033 ceph-mon[27845]: pgmap v10919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:17:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:55 smithi096 ceph-mon[32753]: pgmap v10919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:17:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:55 smithi194 ceph-mon[33017]: pgmap v10919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:17:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:57 smithi033 ceph-mon[27845]: pgmap v10920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:57 smithi096 ceph-mon[32753]: pgmap v10920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:57 smithi194 ceph-mon[33017]: pgmap v10920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:17:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19616.993991 secs 2024-04-26T17:17:59.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19616.993991 secs 2024-04-26T17:17:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19616.993991 secs 2024-04-26T17:18:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:59 smithi096 ceph-mon[32753]: pgmap v10921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19619.840036 secs 2024-04-26T17:18:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:59 smithi194 ceph-mon[33017]: pgmap v10921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19619.840036 secs 2024-04-26T17:18:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:59 smithi033 ceph-mon[27845]: pgmap v10921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19619.840036 secs 2024-04-26T17:18:02.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:02.213+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:02.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:02.213+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:02.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:02.213+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:18:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:01 smithi194 ceph-mon[33017]: pgmap v10922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:01 smithi033 ceph-mon[27845]: pgmap v10922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:01 smithi096 ceph-mon[32753]: pgmap v10922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:03 smithi194 ceph-mon[33017]: pgmap v10923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19621.994049 secs 2024-04-26T17:18:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:03 smithi033 ceph-mon[27845]: pgmap v10923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19621.994049 secs 2024-04-26T17:18:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:03 smithi096 ceph-mon[32753]: pgmap v10923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19621.994049 secs 2024-04-26T17:18:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19624.840177 secs 2024-04-26T17:18:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19624.840177 secs 2024-04-26T17:18:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19624.840177 secs 2024-04-26T17:18:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:05 smithi194 ceph-mon[33017]: pgmap v10924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:18:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:05 smithi033 ceph-mon[27845]: pgmap v10924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:18:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:05 smithi096 ceph-mon[32753]: pgmap v10924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:18:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:07 smithi194 ceph-mon[33017]: pgmap v10925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:07 smithi033 ceph-mon[27845]: pgmap v10925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:07 smithi096 ceph-mon[32753]: pgmap v10925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:09.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19626.994134 secs 2024-04-26T17:18:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19626.994134 secs 2024-04-26T17:18:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19626.994134 secs 2024-04-26T17:18:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:09 smithi194 ceph-mon[33017]: pgmap v10926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19629.840240 secs 2024-04-26T17:18:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:09 smithi033 ceph-mon[27845]: pgmap v10926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19629.840240 secs 2024-04-26T17:18:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:09 smithi096 ceph-mon[32753]: pgmap v10926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19629.840240 secs 2024-04-26T17:18:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:11 smithi194 ceph-mon[33017]: pgmap v10927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:11 smithi033 ceph-mon[27845]: pgmap v10927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:11 smithi096 ceph-mon[32753]: pgmap v10927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:14.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:13 smithi033 ceph-mon[27845]: pgmap v10928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:14.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19631.994216 secs 2024-04-26T17:18:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:13 smithi194 ceph-mon[33017]: pgmap v10928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19631.994216 secs 2024-04-26T17:18:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:13 smithi096 ceph-mon[32753]: pgmap v10928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19631.994216 secs 2024-04-26T17:18:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19634.840359 secs 2024-04-26T17:18:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:18:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19634.840359 secs 2024-04-26T17:18:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:18:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19634.840359 secs 2024-04-26T17:18:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:18:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:16 smithi194 ceph-mon[33017]: pgmap v10929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:18:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:18:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:18:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:18:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:16 smithi033 ceph-mon[27845]: pgmap v10929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:18:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:18:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:18:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:18:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:16 smithi096 ceph-mon[32753]: pgmap v10929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:18:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:18:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:18:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:18:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:17.228+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:17.228+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:17.228+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:18:17.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:17 smithi033 ceph-mon[27845]: pgmap v10930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:17.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:17 smithi033 ceph-mon[27845]: pgmap v10931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:17 smithi194 ceph-mon[33017]: pgmap v10930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:17 smithi194 ceph-mon[33017]: pgmap v10931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:17 smithi096 ceph-mon[32753]: pgmap v10930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:17 smithi096 ceph-mon[32753]: pgmap v10931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19636.994277 secs 2024-04-26T17:18:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:19 smithi194 ceph-mon[33017]: pgmap v10932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19636.994277 secs 2024-04-26T17:18:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:19 smithi033 ceph-mon[27845]: pgmap v10932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19636.994277 secs 2024-04-26T17:18:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:19 smithi096 ceph-mon[32753]: pgmap v10932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19639.840431 secs 2024-04-26T17:18:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19639.840431 secs 2024-04-26T17:18:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19639.840431 secs 2024-04-26T17:18:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:21 smithi194 ceph-mon[33017]: pgmap v10933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:21 smithi033 ceph-mon[27845]: pgmap v10933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:21 smithi096 ceph-mon[32753]: pgmap v10933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:23 smithi194 ceph-mon[33017]: pgmap v10934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:23 smithi033 ceph-mon[27845]: pgmap v10934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:23 smithi096 ceph-mon[32753]: pgmap v10934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:18:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19641.994360 secs 2024-04-26T17:18:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19644.840517 secs 2024-04-26T17:18:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19641.994360 secs 2024-04-26T17:18:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19644.840517 secs 2024-04-26T17:18:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19641.994360 secs 2024-04-26T17:18:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19644.840517 secs 2024-04-26T17:18:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:25 smithi194 ceph-mon[33017]: pgmap v10935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:18:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:25 smithi033 ceph-mon[27845]: pgmap v10935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:18:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:25 smithi096 ceph-mon[32753]: pgmap v10935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:18:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:27 smithi194 ceph-mon[33017]: pgmap v10936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:18:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:27 smithi033 ceph-mon[27845]: pgmap v10936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:18:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:27 smithi096 ceph-mon[32753]: pgmap v10936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:18:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19646.994421 secs 2024-04-26T17:18:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:29 smithi194 ceph-mon[33017]: pgmap v10937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19649.840616 secs 2024-04-26T17:18:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19646.994421 secs 2024-04-26T17:18:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:29 smithi033 ceph-mon[27845]: pgmap v10937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19649.840616 secs 2024-04-26T17:18:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19646.994421 secs 2024-04-26T17:18:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:29 smithi096 ceph-mon[32753]: pgmap v10937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19649.840616 secs 2024-04-26T17:18:31.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:31 smithi194 ceph-mon[33017]: pgmap v10938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:31 smithi033 ceph-mon[27845]: pgmap v10938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:31 smithi096 ceph-mon[32753]: pgmap v10938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:32.230+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:32.230+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:32.230+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:18:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:33 smithi194 ceph-mon[33017]: pgmap v10939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:33 smithi033 ceph-mon[27845]: pgmap v10939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:33 smithi096 ceph-mon[32753]: pgmap v10939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19651.994504 secs 2024-04-26T17:18:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19654.840717 secs 2024-04-26T17:18:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19651.994504 secs 2024-04-26T17:18:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19654.840717 secs 2024-04-26T17:18:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19651.994504 secs 2024-04-26T17:18:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19654.840717 secs 2024-04-26T17:18:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:35 smithi194 ceph-mon[33017]: pgmap v10940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:35 smithi033 ceph-mon[27845]: pgmap v10940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:35 smithi096 ceph-mon[32753]: pgmap v10940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:37 smithi194 ceph-mon[33017]: pgmap v10941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:37 smithi033 ceph-mon[27845]: pgmap v10941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:37 smithi096 ceph-mon[32753]: pgmap v10941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19656.994592 secs 2024-04-26T17:18:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:39 smithi194 ceph-mon[33017]: pgmap v10942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19659.840838 secs 2024-04-26T17:18:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19656.994592 secs 2024-04-26T17:18:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:39 smithi033 ceph-mon[27845]: pgmap v10942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19659.840838 secs 2024-04-26T17:18:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19656.994592 secs 2024-04-26T17:18:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:39 smithi096 ceph-mon[32753]: pgmap v10942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19659.840838 secs 2024-04-26T17:18:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:41 smithi033 ceph-mon[27845]: pgmap v10943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:41 smithi096 ceph-mon[32753]: pgmap v10943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:41 smithi194 ceph-mon[33017]: pgmap v10943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:43 smithi033 ceph-mon[27845]: pgmap v10944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:43 smithi096 ceph-mon[32753]: pgmap v10944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:43 smithi194 ceph-mon[33017]: pgmap v10944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19661.994675 secs 2024-04-26T17:18:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19664.840964 secs 2024-04-26T17:18:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19661.994675 secs 2024-04-26T17:18:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19664.840964 secs 2024-04-26T17:18:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19661.994675 secs 2024-04-26T17:18:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19664.840964 secs 2024-04-26T17:18:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:45 smithi033 ceph-mon[27845]: pgmap v10945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:45 smithi096 ceph-mon[32753]: pgmap v10945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:45 smithi194 ceph-mon[33017]: pgmap v10945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:47.235+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:47.235+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:18:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:18:47.235+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:18:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:47 smithi033 ceph-mon[27845]: pgmap v10946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:47 smithi096 ceph-mon[32753]: pgmap v10946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:47 smithi194 ceph-mon[33017]: pgmap v10946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19666.994735 secs 2024-04-26T17:18:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:49 smithi033 ceph-mon[27845]: pgmap v10947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19669.841080 secs 2024-04-26T17:18:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19666.994735 secs 2024-04-26T17:18:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:49 smithi096 ceph-mon[32753]: pgmap v10947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19669.841080 secs 2024-04-26T17:18:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19666.994735 secs 2024-04-26T17:18:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:49 smithi194 ceph-mon[33017]: pgmap v10947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19669.841080 secs 2024-04-26T17:18:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:51 smithi033 ceph-mon[27845]: pgmap v10948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:51 smithi096 ceph-mon[32753]: pgmap v10948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:51 smithi194 ceph-mon[33017]: pgmap v10948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:53 smithi033 ceph-mon[27845]: pgmap v10949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:53 smithi096 ceph-mon[32753]: pgmap v10949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:53 smithi194 ceph-mon[33017]: pgmap v10949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19671.994813 secs 2024-04-26T17:18:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19674.841176 secs 2024-04-26T17:18:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19671.994813 secs 2024-04-26T17:18:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19674.841176 secs 2024-04-26T17:18:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19671.994813 secs 2024-04-26T17:18:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19674.841176 secs 2024-04-26T17:18:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:55 smithi033 ceph-mon[27845]: pgmap v10950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:55 smithi096 ceph-mon[32753]: pgmap v10950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:55 smithi194 ceph-mon[33017]: pgmap v10950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:57 smithi033 ceph-mon[27845]: pgmap v10951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:57 smithi096 ceph-mon[32753]: pgmap v10951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:57 smithi194 ceph-mon[33017]: pgmap v10951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:18:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19676.994895 secs 2024-04-26T17:18:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:59 smithi033 ceph-mon[27845]: pgmap v10952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:18:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19679.841252 secs 2024-04-26T17:18:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19676.994895 secs 2024-04-26T17:18:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:59 smithi096 ceph-mon[32753]: pgmap v10952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:18:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19679.841252 secs 2024-04-26T17:18:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19676.994895 secs 2024-04-26T17:18:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:59 smithi194 ceph-mon[33017]: pgmap v10952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:18:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:18:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19679.841252 secs 2024-04-26T17:19:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:01 smithi033 ceph-mon[27845]: pgmap v10953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:01 smithi096 ceph-mon[32753]: pgmap v10953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:01 smithi194 ceph-mon[33017]: pgmap v10953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:02.251+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:02.252+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:19:02.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:02.252+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:03 smithi096 ceph-mon[32753]: pgmap v10954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:03 smithi194 ceph-mon[33017]: pgmap v10954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:03 smithi033 ceph-mon[27845]: pgmap v10954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19681.994980 secs 2024-04-26T17:19:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19684.841372 secs 2024-04-26T17:19:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19681.994980 secs 2024-04-26T17:19:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19684.841372 secs 2024-04-26T17:19:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19681.994980 secs 2024-04-26T17:19:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19684.841372 secs 2024-04-26T17:19:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:05 smithi194 ceph-mon[33017]: pgmap v10955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:05 smithi033 ceph-mon[27845]: pgmap v10955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:05 smithi096 ceph-mon[32753]: pgmap v10955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:07 smithi194 ceph-mon[33017]: pgmap v10956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:07 smithi033 ceph-mon[27845]: pgmap v10956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:07 smithi096 ceph-mon[32753]: pgmap v10956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:09.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19686.995070 secs 2024-04-26T17:19:09.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:09 smithi096 ceph-mon[32753]: pgmap v10957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:09.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19689.841493 secs 2024-04-26T17:19:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19686.995070 secs 2024-04-26T17:19:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:09 smithi194 ceph-mon[33017]: pgmap v10957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19689.841493 secs 2024-04-26T17:19:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19686.995070 secs 2024-04-26T17:19:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:09 smithi033 ceph-mon[27845]: pgmap v10957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19689.841493 secs 2024-04-26T17:19:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:11 smithi194 ceph-mon[33017]: pgmap v10958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:11 smithi033 ceph-mon[27845]: pgmap v10958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:11 smithi096 ceph-mon[32753]: pgmap v10958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:13 smithi194 ceph-mon[33017]: pgmap v10959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:13 smithi033 ceph-mon[27845]: pgmap v10959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:13 smithi096 ceph-mon[32753]: pgmap v10959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19691.995137 secs 2024-04-26T17:19:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19694.841612 secs 2024-04-26T17:19:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19691.995137 secs 2024-04-26T17:19:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19694.841612 secs 2024-04-26T17:19:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19691.995137 secs 2024-04-26T17:19:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19694.841612 secs 2024-04-26T17:19:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:15 smithi194 ceph-mon[33017]: pgmap v10960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:15 smithi033 ceph-mon[27845]: pgmap v10960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:15 smithi096 ceph-mon[32753]: pgmap v10960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:16 smithi194 ceph-mon[33017]: pgmap v10961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:19:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:16 smithi033 ceph-mon[27845]: pgmap v10961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:19:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:16 smithi096 ceph-mon[32753]: pgmap v10961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:19:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:17.267+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:17.267+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:19:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:17.267+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:19:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:19:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:19:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:19:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:19:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:19:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:19:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:19:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:19:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:18 smithi033 ceph-mon[27845]: pgmap v10962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:18 smithi033 ceph-mon[27845]: pgmap v10963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19696.995202 secs 2024-04-26T17:19:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:18 smithi096 ceph-mon[32753]: pgmap v10962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:18 smithi096 ceph-mon[32753]: pgmap v10963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19696.995202 secs 2024-04-26T17:19:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:18 smithi194 ceph-mon[33017]: pgmap v10962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:18 smithi194 ceph-mon[33017]: pgmap v10963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19696.995202 secs 2024-04-26T17:19:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19699.841714 secs 2024-04-26T17:19:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19699.841714 secs 2024-04-26T17:19:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19699.841714 secs 2024-04-26T17:19:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:20 smithi033 ceph-mon[27845]: pgmap v10964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:20 smithi096 ceph-mon[32753]: pgmap v10964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:20 smithi194 ceph-mon[33017]: pgmap v10964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:22 smithi033 ceph-mon[27845]: pgmap v10965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:22 smithi096 ceph-mon[32753]: pgmap v10965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:22 smithi194 ceph-mon[33017]: pgmap v10965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19701.995261 secs 2024-04-26T17:19:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19701.995261 secs 2024-04-26T17:19:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19701.995261 secs 2024-04-26T17:19:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19704.841834 secs 2024-04-26T17:19:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:24 smithi033 ceph-mon[27845]: pgmap v10966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19704.841834 secs 2024-04-26T17:19:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:24 smithi096 ceph-mon[32753]: pgmap v10966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19704.841834 secs 2024-04-26T17:19:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:24 smithi194 ceph-mon[33017]: pgmap v10966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:26.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:26 smithi033 ceph-mon[27845]: pgmap v10967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:26 smithi096 ceph-mon[32753]: pgmap v10967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:26 smithi194 ceph-mon[33017]: pgmap v10967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:19:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:28 smithi033 ceph-mon[27845]: pgmap v10968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:19:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19706.995322 secs 2024-04-26T17:19:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:28 smithi096 ceph-mon[32753]: pgmap v10968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:19:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19706.995322 secs 2024-04-26T17:19:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:28 smithi194 ceph-mon[33017]: pgmap v10968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:19:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19706.995322 secs 2024-04-26T17:19:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19709.841930 secs 2024-04-26T17:19:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19709.841930 secs 2024-04-26T17:19:30.003 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19709.841930 secs 2024-04-26T17:19:30.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:30 smithi033 ceph-mon[27845]: pgmap v10969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:30 smithi096 ceph-mon[32753]: pgmap v10969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:30 smithi194 ceph-mon[33017]: pgmap v10969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:32.267+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:32.267+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:19:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:32.267+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:32 smithi033 ceph-mon[27845]: pgmap v10970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:32 smithi096 ceph-mon[32753]: pgmap v10970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:32 smithi194 ceph-mon[33017]: pgmap v10970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19711.995382 secs 2024-04-26T17:19:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19711.995382 secs 2024-04-26T17:19:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19711.995382 secs 2024-04-26T17:19:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19714.842048 secs 2024-04-26T17:19:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:34 smithi194 ceph-mon[33017]: pgmap v10971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19714.842048 secs 2024-04-26T17:19:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:34 smithi033 ceph-mon[27845]: pgmap v10971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19714.842048 secs 2024-04-26T17:19:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:34 smithi096 ceph-mon[32753]: pgmap v10971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:36 smithi194 ceph-mon[33017]: pgmap v10972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:37.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:36 smithi033 ceph-mon[27845]: pgmap v10972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:36 smithi096 ceph-mon[32753]: pgmap v10972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:38 smithi194 ceph-mon[33017]: pgmap v10973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19716.995438 secs 2024-04-26T17:19:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:38 smithi033 ceph-mon[27845]: pgmap v10973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19716.995438 secs 2024-04-26T17:19:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:38 smithi096 ceph-mon[32753]: pgmap v10973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19716.995438 secs 2024-04-26T17:19:40.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19719.842166 secs 2024-04-26T17:19:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19719.842166 secs 2024-04-26T17:19:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:40.146+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:19:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:40.146+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:19:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:40.146+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 158 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:19:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:40.146+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 158 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:19:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19719.842166 secs 2024-04-26T17:19:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:40 smithi194 ceph-mon[33017]: pgmap v10974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:40 smithi033 ceph-mon[27845]: pgmap v10974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:40 smithi096 ceph-mon[32753]: pgmap v10974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:42 smithi194 ceph-mon[33017]: pgmap v10975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:42 smithi033 ceph-mon[27845]: pgmap v10975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:42 smithi096 ceph-mon[32753]: pgmap v10975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19721.995520 secs 2024-04-26T17:19:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19721.995520 secs 2024-04-26T17:19:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19721.995520 secs 2024-04-26T17:19:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19724.842282 secs 2024-04-26T17:19:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:44 smithi194 ceph-mon[33017]: pgmap v10976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19724.842282 secs 2024-04-26T17:19:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:44 smithi033 ceph-mon[27845]: pgmap v10976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19724.842282 secs 2024-04-26T17:19:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:44 smithi096 ceph-mon[32753]: pgmap v10976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:46 smithi194 ceph-mon[33017]: pgmap v10977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:47.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:46 smithi033 ceph-mon[27845]: pgmap v10977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:46 smithi096 ceph-mon[32753]: pgmap v10977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:47.283+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:47.283 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:47.283+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:19:47.283 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:19:47.283+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:19:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:48 smithi194 ceph-mon[33017]: pgmap v10978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19726.995600 secs 2024-04-26T17:19:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:48 smithi033 ceph-mon[27845]: pgmap v10978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19726.995600 secs 2024-04-26T17:19:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:48 smithi096 ceph-mon[32753]: pgmap v10978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19726.995600 secs 2024-04-26T17:19:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19729.842396 secs 2024-04-26T17:19:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19729.842396 secs 2024-04-26T17:19:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19729.842396 secs 2024-04-26T17:19:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:50 smithi194 ceph-mon[33017]: pgmap v10979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:51.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:50 smithi033 ceph-mon[27845]: pgmap v10979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:50 smithi096 ceph-mon[32753]: pgmap v10979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:52 smithi194 ceph-mon[33017]: pgmap v10980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:53.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:52 smithi033 ceph-mon[27845]: pgmap v10980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:52 smithi096 ceph-mon[32753]: pgmap v10980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19731.995659 secs 2024-04-26T17:19:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19731.995659 secs 2024-04-26T17:19:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19731.995659 secs 2024-04-26T17:19:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19734.842508 secs 2024-04-26T17:19:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:54 smithi194 ceph-mon[33017]: pgmap v10981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19734.842508 secs 2024-04-26T17:19:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:54 smithi033 ceph-mon[27845]: pgmap v10981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19734.842508 secs 2024-04-26T17:19:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:54 smithi096 ceph-mon[32753]: pgmap v10981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:56 smithi194 ceph-mon[33017]: pgmap v10982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:56 smithi033 ceph-mon[27845]: pgmap v10982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:56 smithi096 ceph-mon[32753]: pgmap v10982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:19:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:58 smithi194 ceph-mon[33017]: pgmap v10983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19736.995736 secs 2024-04-26T17:19:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:58 smithi033 ceph-mon[27845]: pgmap v10983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19736.995736 secs 2024-04-26T17:19:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:58 smithi096 ceph-mon[32753]: pgmap v10983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:19:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19736.995736 secs 2024-04-26T17:20:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:19:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19739.842628 secs 2024-04-26T17:20:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:19:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19739.842628 secs 2024-04-26T17:20:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:19:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19739.842628 secs 2024-04-26T17:20:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:00 smithi194 ceph-mon[33017]: pgmap v10984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:20:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:00 smithi033 ceph-mon[27845]: pgmap v10984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:20:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:00 smithi096 ceph-mon[32753]: pgmap v10984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:20:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:02.288+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:02.288+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:20:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:02.288+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:02 smithi194 ceph-mon[33017]: pgmap v10985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:02 smithi033 ceph-mon[27845]: pgmap v10985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:02 smithi096 ceph-mon[32753]: pgmap v10985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19741.995817 secs 2024-04-26T17:20:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19741.995817 secs 2024-04-26T17:20:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19741.995817 secs 2024-04-26T17:20:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19744.842750 secs 2024-04-26T17:20:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:04 smithi033 ceph-mon[27845]: pgmap v10986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19744.842750 secs 2024-04-26T17:20:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:04 smithi194 ceph-mon[33017]: pgmap v10986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19744.842750 secs 2024-04-26T17:20:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:04 smithi096 ceph-mon[32753]: pgmap v10986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:06 smithi194 ceph-mon[33017]: pgmap v10987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:06 smithi033 ceph-mon[27845]: pgmap v10987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:06 smithi096 ceph-mon[32753]: pgmap v10987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:08 smithi194 ceph-mon[33017]: pgmap v10988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19746.995898 secs 2024-04-26T17:20:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:08 smithi033 ceph-mon[27845]: pgmap v10988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19746.995898 secs 2024-04-26T17:20:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:08 smithi096 ceph-mon[32753]: pgmap v10988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19746.995898 secs 2024-04-26T17:20:10.082 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19749.842867 secs 2024-04-26T17:20:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19749.842867 secs 2024-04-26T17:20:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19749.842867 secs 2024-04-26T17:20:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:10 smithi194 ceph-mon[33017]: pgmap v10989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:10 smithi033 ceph-mon[27845]: pgmap v10989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:10 smithi096 ceph-mon[32753]: pgmap v10989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:12 smithi194 ceph-mon[33017]: pgmap v10990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:12 smithi033 ceph-mon[27845]: pgmap v10990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:12 smithi096 ceph-mon[32753]: pgmap v10990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19751.996000 secs 2024-04-26T17:20:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19751.996000 secs 2024-04-26T17:20:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19751.996000 secs 2024-04-26T17:20:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19754.842984 secs 2024-04-26T17:20:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:14 smithi194 ceph-mon[33017]: pgmap v10991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19754.842984 secs 2024-04-26T17:20:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:14 smithi033 ceph-mon[27845]: pgmap v10991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19754.842984 secs 2024-04-26T17:20:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:14 smithi096 ceph-mon[32753]: pgmap v10991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:16 smithi033 ceph-mon[27845]: pgmap v10992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:16 smithi096 ceph-mon[32753]: pgmap v10992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:17.301+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:17.301+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:17.301+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:20:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:16 smithi194 ceph-mon[33017]: pgmap v10992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:18 smithi033 ceph-mon[27845]: pgmap v10993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19756.996061 secs 2024-04-26T17:20:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:20:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:18 smithi096 ceph-mon[32753]: pgmap v10993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19756.996061 secs 2024-04-26T17:20:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:20:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:18 smithi194 ceph-mon[33017]: pgmap v10993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19756.996061 secs 2024-04-26T17:20:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19759.843078 secs 2024-04-26T17:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:19 smithi033 ceph-mon[27845]: pgmap v10994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:19 smithi033 ceph-mon[27845]: pgmap v10995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:20:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19759.843078 secs 2024-04-26T17:20:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:20:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:20:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:19 smithi096 ceph-mon[32753]: pgmap v10994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:19 smithi096 ceph-mon[32753]: pgmap v10995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:20:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:20:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19759.843078 secs 2024-04-26T17:20:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:20:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:20:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:19 smithi194 ceph-mon[33017]: pgmap v10994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:19 smithi194 ceph-mon[33017]: pgmap v10995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:20:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:20:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:21 smithi194 ceph-mon[33017]: pgmap v10996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:21 smithi033 ceph-mon[27845]: pgmap v10996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:21 smithi096 ceph-mon[32753]: pgmap v10996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:23 smithi194 ceph-mon[33017]: pgmap v10997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:23 smithi033 ceph-mon[27845]: pgmap v10997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:23 smithi096 ceph-mon[32753]: pgmap v10997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19761.996140 secs 2024-04-26T17:20:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19764.843200 secs 2024-04-26T17:20:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19761.996140 secs 2024-04-26T17:20:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19764.843200 secs 2024-04-26T17:20:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19761.996140 secs 2024-04-26T17:20:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19764.843200 secs 2024-04-26T17:20:25.153 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:25.154+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:20:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:25.154+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:20:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:25.155+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 159 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:20:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:25.155+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 159 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:20:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:25 smithi194 ceph-mon[33017]: pgmap v10998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:25 smithi033 ceph-mon[27845]: pgmap v10998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:25 smithi096 ceph-mon[32753]: pgmap v10998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:27 smithi194 ceph-mon[33017]: pgmap v10999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:27 smithi033 ceph-mon[27845]: pgmap v10999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:27 smithi096 ceph-mon[32753]: pgmap v10999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:20:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19766.996225 secs 2024-04-26T17:20:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19766.996225 secs 2024-04-26T17:20:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19766.996225 secs 2024-04-26T17:20:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19769.843339 secs 2024-04-26T17:20:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:29 smithi033 ceph-mon[27845]: pgmap v11000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:20:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19769.843339 secs 2024-04-26T17:20:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:29 smithi096 ceph-mon[32753]: pgmap v11000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:20:30.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19769.843339 secs 2024-04-26T17:20:30.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:29 smithi194 ceph-mon[33017]: pgmap v11000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:20:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:31 smithi194 ceph-mon[33017]: pgmap v11001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:31 smithi033 ceph-mon[27845]: pgmap v11001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:31 smithi096 ceph-mon[32753]: pgmap v11001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:32.301+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:32.301+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:32.301+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:20:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:33 smithi194 ceph-mon[33017]: pgmap v11002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:33 smithi033 ceph-mon[27845]: pgmap v11002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:33 smithi096 ceph-mon[32753]: pgmap v11002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19771.996284 secs 2024-04-26T17:20:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19774.843459 secs 2024-04-26T17:20:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19771.996284 secs 2024-04-26T17:20:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19774.843459 secs 2024-04-26T17:20:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19771.996284 secs 2024-04-26T17:20:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19774.843459 secs 2024-04-26T17:20:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:35 smithi194 ceph-mon[33017]: pgmap v11003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:35.857 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:35 smithi096 ceph-mon[32753]: pgmap v11003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:35.858 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:35 smithi033 ceph-mon[27845]: pgmap v11003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:37 smithi194 ceph-mon[33017]: pgmap v11004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:37 smithi033 ceph-mon[27845]: pgmap v11004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:37 smithi096 ceph-mon[32753]: pgmap v11004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19776.996363 secs 2024-04-26T17:20:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19776.996363 secs 2024-04-26T17:20:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19776.996363 secs 2024-04-26T17:20:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19779.843536 secs 2024-04-26T17:20:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:39 smithi033 ceph-mon[27845]: pgmap v11005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19779.843536 secs 2024-04-26T17:20:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:39 smithi096 ceph-mon[32753]: pgmap v11005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19779.843536 secs 2024-04-26T17:20:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:39 smithi194 ceph-mon[33017]: pgmap v11005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:41 smithi194 ceph-mon[33017]: pgmap v11006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:41 smithi033 ceph-mon[27845]: pgmap v11006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:41 smithi096 ceph-mon[32753]: pgmap v11006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:43 smithi194 ceph-mon[33017]: pgmap v11007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:43 smithi033 ceph-mon[27845]: pgmap v11007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:43 smithi096 ceph-mon[32753]: pgmap v11007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19781.996447 secs 2024-04-26T17:20:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19784.843651 secs 2024-04-26T17:20:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19781.996447 secs 2024-04-26T17:20:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19784.843651 secs 2024-04-26T17:20:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19781.996447 secs 2024-04-26T17:20:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19784.843651 secs 2024-04-26T17:20:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:45 smithi194 ceph-mon[33017]: pgmap v11008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:45 smithi033 ceph-mon[27845]: pgmap v11008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:45 smithi096 ceph-mon[32753]: pgmap v11008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:47.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:47.316+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:47.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:47.317+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:20:47.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:20:47.317+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:20:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:47 smithi194 ceph-mon[33017]: pgmap v11009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:47 smithi033 ceph-mon[27845]: pgmap v11009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:47 smithi096 ceph-mon[32753]: pgmap v11009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19786.996539 secs 2024-04-26T17:20:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19786.996539 secs 2024-04-26T17:20:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19786.996539 secs 2024-04-26T17:20:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19789.843766 secs 2024-04-26T17:20:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:49 smithi033 ceph-mon[27845]: pgmap v11010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:20:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19789.843766 secs 2024-04-26T17:20:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:49 smithi096 ceph-mon[32753]: pgmap v11010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:20:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19789.843766 secs 2024-04-26T17:20:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:49 smithi194 ceph-mon[33017]: pgmap v11010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:20:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:51 smithi194 ceph-mon[33017]: pgmap v11011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:51 smithi033 ceph-mon[27845]: pgmap v11011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:51 smithi096 ceph-mon[32753]: pgmap v11011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:53 smithi194 ceph-mon[33017]: pgmap v11012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:53 smithi033 ceph-mon[27845]: pgmap v11012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:53 smithi096 ceph-mon[32753]: pgmap v11012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19791.996621 secs 2024-04-26T17:20:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19794.843911 secs 2024-04-26T17:20:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19791.996621 secs 2024-04-26T17:20:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19794.843911 secs 2024-04-26T17:20:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19791.996621 secs 2024-04-26T17:20:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19794.843911 secs 2024-04-26T17:20:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:55 smithi194 ceph-mon[33017]: pgmap v11013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:55 smithi033 ceph-mon[27845]: pgmap v11013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:55 smithi096 ceph-mon[32753]: pgmap v11013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:20:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:57 smithi194 ceph-mon[33017]: pgmap v11014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:57 smithi033 ceph-mon[27845]: pgmap v11014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:57 smithi096 ceph-mon[32753]: pgmap v11014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:20:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19796.996702 secs 2024-04-26T17:20:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19796.996702 secs 2024-04-26T17:20:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19796.996702 secs 2024-04-26T17:20:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19799.844026 secs 2024-04-26T17:20:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:20:59 smithi033 ceph-mon[27845]: pgmap v11015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:20:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19799.844026 secs 2024-04-26T17:20:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:20:59 smithi096 ceph-mon[32753]: pgmap v11015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:00.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19799.844026 secs 2024-04-26T17:21:00.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:20:59 smithi194 ceph-mon[33017]: pgmap v11015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:01 smithi194 ceph-mon[33017]: pgmap v11016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:01 smithi033 ceph-mon[27845]: pgmap v11016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:01 smithi096 ceph-mon[32753]: pgmap v11016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:02.332+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:02.332+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:02.332+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:21:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:03 smithi194 ceph-mon[33017]: pgmap v11017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:03 smithi033 ceph-mon[27845]: pgmap v11017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:03 smithi096 ceph-mon[32753]: pgmap v11017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19801.996804 secs 2024-04-26T17:21:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19804.844148 secs 2024-04-26T17:21:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19801.996804 secs 2024-04-26T17:21:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19804.844148 secs 2024-04-26T17:21:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19801.996804 secs 2024-04-26T17:21:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19804.844148 secs 2024-04-26T17:21:05.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:05 smithi033 ceph-mon[27845]: pgmap v11018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:05.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:05 smithi194 ceph-mon[33017]: pgmap v11018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:05 smithi096 ceph-mon[32753]: pgmap v11018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:07 smithi194 ceph-mon[33017]: pgmap v11019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:07 smithi033 ceph-mon[27845]: pgmap v11019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:07 smithi096 ceph-mon[32753]: pgmap v11019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19806.996873 secs 2024-04-26T17:21:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19806.996873 secs 2024-04-26T17:21:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19806.996873 secs 2024-04-26T17:21:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19809.844267 secs 2024-04-26T17:21:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:09 smithi033 ceph-mon[27845]: pgmap v11020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19809.844267 secs 2024-04-26T17:21:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:09 smithi096 ceph-mon[32753]: pgmap v11020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19809.844267 secs 2024-04-26T17:21:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:09 smithi194 ceph-mon[33017]: pgmap v11020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:11 smithi194 ceph-mon[33017]: pgmap v11021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:11 smithi033 ceph-mon[27845]: pgmap v11021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:11 smithi096 ceph-mon[32753]: pgmap v11021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:13 smithi194 ceph-mon[33017]: pgmap v11022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:13 smithi033 ceph-mon[27845]: pgmap v11022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:13 smithi096 ceph-mon[32753]: pgmap v11022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19811.996956 secs 2024-04-26T17:21:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19814.844348 secs 2024-04-26T17:21:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19811.996956 secs 2024-04-26T17:21:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19814.844348 secs 2024-04-26T17:21:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19811.996956 secs 2024-04-26T17:21:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19814.844348 secs 2024-04-26T17:21:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:16 smithi194 ceph-mon[33017]: pgmap v11023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:16 smithi033 ceph-mon[27845]: pgmap v11023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:16 smithi096 ceph-mon[32753]: pgmap v11023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:17.337+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:17.337+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:21:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:17.337+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:18 smithi194 ceph-mon[33017]: pgmap v11024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:18 smithi033 ceph-mon[27845]: pgmap v11024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:18 smithi096 ceph-mon[32753]: pgmap v11024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19816.997044 secs 2024-04-26T17:21:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19819.844423 secs 2024-04-26T17:21:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19816.997044 secs 2024-04-26T17:21:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19819.844423 secs 2024-04-26T17:21:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19816.997044 secs 2024-04-26T17:21:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19819.844423 secs 2024-04-26T17:21:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:20 smithi033 ceph-mon[27845]: pgmap v11025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:21:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:20 smithi096 ceph-mon[32753]: pgmap v11025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:21:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:20 smithi194 ceph-mon[33017]: pgmap v11025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:21:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:21:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:21:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:21:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:21 smithi033 ceph-mon[27845]: pgmap v11026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:21 smithi033 ceph-mon[27845]: pgmap v11027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:21:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:21:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:21:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:21 smithi096 ceph-mon[32753]: pgmap v11026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:21 smithi096 ceph-mon[32753]: pgmap v11027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:21:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:21:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:21:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:21 smithi194 ceph-mon[33017]: pgmap v11026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:21 smithi194 ceph-mon[33017]: pgmap v11027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:21:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19821.997131 secs 2024-04-26T17:21:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:24 smithi194 ceph-mon[33017]: pgmap v11028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19821.997131 secs 2024-04-26T17:21:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:24 smithi033 ceph-mon[27845]: pgmap v11028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19821.997131 secs 2024-04-26T17:21:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:24 smithi096 ceph-mon[32753]: pgmap v11028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19824.844539 secs 2024-04-26T17:21:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19824.844539 secs 2024-04-26T17:21:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19824.844539 secs 2024-04-26T17:21:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:26 smithi194 ceph-mon[33017]: pgmap v11029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:26 smithi033 ceph-mon[27845]: pgmap v11029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:26 smithi096 ceph-mon[32753]: pgmap v11029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:28 smithi194 ceph-mon[33017]: pgmap v11030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:28 smithi033 ceph-mon[27845]: pgmap v11030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:28 smithi096 ceph-mon[32753]: pgmap v11030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:21:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19826.997218 secs 2024-04-26T17:21:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19826.997218 secs 2024-04-26T17:21:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19826.997218 secs 2024-04-26T17:21:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19829.844604 secs 2024-04-26T17:21:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:30 smithi194 ceph-mon[33017]: pgmap v11031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:21:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19829.844604 secs 2024-04-26T17:21:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:30 smithi033 ceph-mon[27845]: pgmap v11031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:21:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19829.844604 secs 2024-04-26T17:21:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:30 smithi096 ceph-mon[32753]: pgmap v11031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:21:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:32.351+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:32.351+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:21:32.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:32.351+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:32 smithi194 ceph-mon[33017]: pgmap v11032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:21:32.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:32 smithi033 ceph-mon[27845]: pgmap v11032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:21:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:32 smithi096 ceph-mon[32753]: pgmap v11032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:21:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19831.997305 secs 2024-04-26T17:21:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:34 smithi194 ceph-mon[33017]: pgmap v11033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19831.997305 secs 2024-04-26T17:21:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:34 smithi033 ceph-mon[27845]: pgmap v11033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19831.997305 secs 2024-04-26T17:21:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:34 smithi096 ceph-mon[32753]: pgmap v11033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19834.844724 secs 2024-04-26T17:21:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19834.844724 secs 2024-04-26T17:21:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19834.844724 secs 2024-04-26T17:21:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:36 smithi194 ceph-mon[33017]: pgmap v11034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:36 smithi033 ceph-mon[27845]: pgmap v11034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:36 smithi096 ceph-mon[32753]: pgmap v11034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:38 smithi194 ceph-mon[33017]: pgmap v11035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:38 smithi033 ceph-mon[27845]: pgmap v11035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:38 smithi096 ceph-mon[32753]: pgmap v11035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19836.997391 secs 2024-04-26T17:21:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19839.844818 secs 2024-04-26T17:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19836.997391 secs 2024-04-26T17:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19839.844818 secs 2024-04-26T17:21:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19836.997391 secs 2024-04-26T17:21:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19839.844818 secs 2024-04-26T17:21:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:40 smithi194 ceph-mon[33017]: pgmap v11036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:40 smithi033 ceph-mon[27845]: pgmap v11036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:40 smithi096 ceph-mon[32753]: pgmap v11036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:42 smithi194 ceph-mon[33017]: pgmap v11037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:42 smithi033 ceph-mon[27845]: pgmap v11037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:42 smithi096 ceph-mon[32753]: pgmap v11037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19841.997451 secs 2024-04-26T17:21:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:44 smithi194 ceph-mon[33017]: pgmap v11038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19844.844955 secs 2024-04-26T17:21:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19841.997451 secs 2024-04-26T17:21:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:44 smithi033 ceph-mon[27845]: pgmap v11038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19844.844955 secs 2024-04-26T17:21:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19841.997451 secs 2024-04-26T17:21:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:44 smithi096 ceph-mon[32753]: pgmap v11038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19844.844955 secs 2024-04-26T17:21:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:46 smithi033 ceph-mon[27845]: pgmap v11039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:46 smithi096 ceph-mon[32753]: pgmap v11039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:46 smithi194 ceph-mon[33017]: pgmap v11039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:47.366+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:47.367+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:21:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:21:47.367+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:21:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:48 smithi033 ceph-mon[27845]: pgmap v11040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:48 smithi096 ceph-mon[32753]: pgmap v11040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:48 smithi194 ceph-mon[33017]: pgmap v11040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19846.997546 secs 2024-04-26T17:21:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19849.845072 secs 2024-04-26T17:21:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19846.997546 secs 2024-04-26T17:21:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19849.845072 secs 2024-04-26T17:21:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19846.997546 secs 2024-04-26T17:21:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19849.845072 secs 2024-04-26T17:21:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:50 smithi033 ceph-mon[27845]: pgmap v11041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:50 smithi096 ceph-mon[32753]: pgmap v11041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:50 smithi194 ceph-mon[33017]: pgmap v11041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:52 smithi033 ceph-mon[27845]: pgmap v11042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:52 smithi096 ceph-mon[32753]: pgmap v11042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:52 smithi194 ceph-mon[33017]: pgmap v11042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19851.997646 secs 2024-04-26T17:21:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:54 smithi033 ceph-mon[27845]: pgmap v11043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19854.845196 secs 2024-04-26T17:21:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19851.997646 secs 2024-04-26T17:21:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:54 smithi096 ceph-mon[32753]: pgmap v11043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19854.845196 secs 2024-04-26T17:21:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19851.997646 secs 2024-04-26T17:21:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:54 smithi194 ceph-mon[33017]: pgmap v11043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19854.845196 secs 2024-04-26T17:21:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:56 smithi033 ceph-mon[27845]: pgmap v11044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:56 smithi096 ceph-mon[32753]: pgmap v11044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:56 smithi194 ceph-mon[33017]: pgmap v11044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:21:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:58 smithi033 ceph-mon[27845]: pgmap v11045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:58 smithi096 ceph-mon[32753]: pgmap v11045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:58 smithi194 ceph-mon[33017]: pgmap v11045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:21:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19856.997749 secs 2024-04-26T17:21:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:21:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19859.845262 secs 2024-04-26T17:21:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19856.997749 secs 2024-04-26T17:21:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:21:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19859.845262 secs 2024-04-26T17:21:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19856.997749 secs 2024-04-26T17:21:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:21:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19859.845262 secs 2024-04-26T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:00 smithi033 ceph-mon[27845]: pgmap v11046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:00 smithi096 ceph-mon[32753]: pgmap v11046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:00.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:00 smithi194 ceph-mon[33017]: pgmap v11046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:02.369+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:02.369+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:02.369+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:22:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:02 smithi033 ceph-mon[27845]: pgmap v11047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:02 smithi096 ceph-mon[32753]: pgmap v11047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:02 smithi194 ceph-mon[33017]: pgmap v11047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19861.997846 secs 2024-04-26T17:22:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:04 smithi096 ceph-mon[32753]: pgmap v11048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19864.845331 secs 2024-04-26T17:22:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19861.997846 secs 2024-04-26T17:22:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:04 smithi194 ceph-mon[33017]: pgmap v11048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19864.845331 secs 2024-04-26T17:22:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19861.997846 secs 2024-04-26T17:22:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:04 smithi033 ceph-mon[27845]: pgmap v11048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19864.845331 secs 2024-04-26T17:22:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:06 smithi194 ceph-mon[33017]: pgmap v11049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:06 smithi033 ceph-mon[27845]: pgmap v11049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:06 smithi096 ceph-mon[32753]: pgmap v11049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:08.477 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:08 smithi033 ceph-mon[27845]: pgmap v11050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:08 smithi194 ceph-mon[33017]: pgmap v11050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:08 smithi096 ceph-mon[32753]: pgmap v11050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19866.997953 secs 2024-04-26T17:22:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19869.845401 secs 2024-04-26T17:22:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19866.997953 secs 2024-04-26T17:22:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19869.845401 secs 2024-04-26T17:22:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19866.997953 secs 2024-04-26T17:22:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19869.845401 secs 2024-04-26T17:22:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:10 smithi194 ceph-mon[33017]: pgmap v11051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:10 smithi033 ceph-mon[27845]: pgmap v11051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:10 smithi096 ceph-mon[32753]: pgmap v11051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:12.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:12 smithi096 ceph-mon[32753]: pgmap v11052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:12 smithi194 ceph-mon[33017]: pgmap v11052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:12 smithi033 ceph-mon[27845]: pgmap v11052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19871.998050 secs 2024-04-26T17:22:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:14 smithi194 ceph-mon[33017]: pgmap v11053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19874.845548 secs 2024-04-26T17:22:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19871.998050 secs 2024-04-26T17:22:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:14 smithi033 ceph-mon[27845]: pgmap v11053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19874.845548 secs 2024-04-26T17:22:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19871.998050 secs 2024-04-26T17:22:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:14 smithi096 ceph-mon[32753]: pgmap v11053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19874.845548 secs 2024-04-26T17:22:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:16 smithi194 ceph-mon[33017]: pgmap v11054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:16 smithi033 ceph-mon[27845]: pgmap v11054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:16 smithi096 ceph-mon[32753]: pgmap v11054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:17.383+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:17.383+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:17.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:17.383+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:22:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:18 smithi194 ceph-mon[33017]: pgmap v11055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:18 smithi033 ceph-mon[27845]: pgmap v11055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:18 smithi096 ceph-mon[32753]: pgmap v11055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:19.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19876.998138 secs 2024-04-26T17:22:19.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19879.845669 secs 2024-04-26T17:22:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19876.998138 secs 2024-04-26T17:22:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19879.845669 secs 2024-04-26T17:22:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19876.998138 secs 2024-04-26T17:22:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19879.845669 secs 2024-04-26T17:22:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:20 smithi194 ceph-mon[33017]: pgmap v11056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:20 smithi033 ceph-mon[27845]: pgmap v11056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:20 smithi096 ceph-mon[32753]: pgmap v11056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:22 smithi194 ceph-mon[33017]: pgmap v11057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:22:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:22 smithi033 ceph-mon[27845]: pgmap v11057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:22:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:22 smithi096 ceph-mon[32753]: pgmap v11057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:22:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19881.998226 secs 2024-04-26T17:22:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:24 smithi194 ceph-mon[33017]: pgmap v11058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19884.845806 secs 2024-04-26T17:22:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:22:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:22:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19881.998226 secs 2024-04-26T17:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:24 smithi033 ceph-mon[27845]: pgmap v11058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19884.845806 secs 2024-04-26T17:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:22:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19881.998226 secs 2024-04-26T17:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:24 smithi096 ceph-mon[32753]: pgmap v11058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19884.845806 secs 2024-04-26T17:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:22:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:22:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:25 smithi194 ceph-mon[33017]: pgmap v11059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:22:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:25 smithi194 ceph-mon[33017]: pgmap v11060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:22:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:25 smithi033 ceph-mon[27845]: pgmap v11059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:22:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:25 smithi033 ceph-mon[27845]: pgmap v11060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:22:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:25 smithi096 ceph-mon[32753]: pgmap v11059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:22:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:25 smithi096 ceph-mon[32753]: pgmap v11060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:22:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:27 smithi194 ceph-mon[33017]: pgmap v11061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:27 smithi033 ceph-mon[27845]: pgmap v11061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:27 smithi096 ceph-mon[32753]: pgmap v11061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:29 smithi194 ceph-mon[33017]: pgmap v11062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19886.998289 secs 2024-04-26T17:22:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19889.845922 secs 2024-04-26T17:22:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:29 smithi033 ceph-mon[27845]: pgmap v11062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19886.998289 secs 2024-04-26T17:22:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19889.845922 secs 2024-04-26T17:22:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:29 smithi096 ceph-mon[32753]: pgmap v11062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19886.998289 secs 2024-04-26T17:22:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19889.845922 secs 2024-04-26T17:22:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:30 smithi194 ceph-mon[33017]: pgmap v11063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:30 smithi033 ceph-mon[27845]: pgmap v11063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:30 smithi096 ceph-mon[32753]: pgmap v11063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:32.399+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:32.399+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:32.399+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:22:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:32 smithi033 ceph-mon[27845]: pgmap v11064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:32 smithi096 ceph-mon[32753]: pgmap v11064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:32 smithi194 ceph-mon[33017]: pgmap v11064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:22:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19891.998348 secs 2024-04-26T17:22:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19891.998348 secs 2024-04-26T17:22:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19891.998348 secs 2024-04-26T17:22:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19894.846071 secs 2024-04-26T17:22:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:34 smithi033 ceph-mon[27845]: pgmap v11065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:22:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19894.846071 secs 2024-04-26T17:22:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:34 smithi096 ceph-mon[32753]: pgmap v11065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:22:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19894.846071 secs 2024-04-26T17:22:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:34 smithi194 ceph-mon[33017]: pgmap v11065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:22:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:36 smithi033 ceph-mon[27845]: pgmap v11066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:36 smithi096 ceph-mon[32753]: pgmap v11066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:36 smithi194 ceph-mon[33017]: pgmap v11066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:38 smithi033 ceph-mon[27845]: pgmap v11067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19896.998438 secs 2024-04-26T17:22:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:38 smithi096 ceph-mon[32753]: pgmap v11067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19896.998438 secs 2024-04-26T17:22:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:38 smithi194 ceph-mon[33017]: pgmap v11067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19896.998438 secs 2024-04-26T17:22:40.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19899.846126 secs 2024-04-26T17:22:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19899.846126 secs 2024-04-26T17:22:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19899.846126 secs 2024-04-26T17:22:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:40 smithi194 ceph-mon[33017]: pgmap v11068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:40 smithi033 ceph-mon[27845]: pgmap v11068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:40 smithi096 ceph-mon[32753]: pgmap v11068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:42.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:42 smithi096 ceph-mon[32753]: pgmap v11069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:42 smithi194 ceph-mon[33017]: pgmap v11069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:43.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:42 smithi033 ceph-mon[27845]: pgmap v11069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19901.998526 secs 2024-04-26T17:22:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19901.998526 secs 2024-04-26T17:22:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19901.998526 secs 2024-04-26T17:22:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19904.846277 secs 2024-04-26T17:22:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:44 smithi194 ceph-mon[33017]: pgmap v11070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19904.846277 secs 2024-04-26T17:22:45.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:44 smithi033 ceph-mon[27845]: pgmap v11070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19904.846277 secs 2024-04-26T17:22:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:44 smithi096 ceph-mon[32753]: pgmap v11070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:46 smithi194 ceph-mon[33017]: pgmap v11071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:47.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:46 smithi033 ceph-mon[27845]: pgmap v11071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:46 smithi096 ceph-mon[32753]: pgmap v11071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:47.408+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:47.408+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:22:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:22:47.408+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:22:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:48 smithi194 ceph-mon[33017]: pgmap v11072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19906.998624 secs 2024-04-26T17:22:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:48 smithi033 ceph-mon[27845]: pgmap v11072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19906.998624 secs 2024-04-26T17:22:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:48 smithi096 ceph-mon[32753]: pgmap v11072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19906.998624 secs 2024-04-26T17:22:50.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19909.846392 secs 2024-04-26T17:22:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19909.846392 secs 2024-04-26T17:22:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19909.846392 secs 2024-04-26T17:22:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:50 smithi194 ceph-mon[33017]: pgmap v11073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:51.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:50 smithi033 ceph-mon[27845]: pgmap v11073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:50 smithi096 ceph-mon[32753]: pgmap v11073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:52 smithi194 ceph-mon[33017]: pgmap v11074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:53.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:52 smithi033 ceph-mon[27845]: pgmap v11074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:52 smithi096 ceph-mon[32753]: pgmap v11074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19911.998722 secs 2024-04-26T17:22:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19911.998722 secs 2024-04-26T17:22:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19911.998722 secs 2024-04-26T17:22:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19914.846512 secs 2024-04-26T17:22:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:54 smithi194 ceph-mon[33017]: pgmap v11075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19914.846512 secs 2024-04-26T17:22:55.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:54 smithi033 ceph-mon[27845]: pgmap v11075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19914.846512 secs 2024-04-26T17:22:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:54 smithi096 ceph-mon[32753]: pgmap v11075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:56 smithi194 ceph-mon[33017]: pgmap v11076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:56 smithi033 ceph-mon[27845]: pgmap v11076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:56 smithi096 ceph-mon[32753]: pgmap v11076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:22:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:58 smithi194 ceph-mon[33017]: pgmap v11077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19916.998782 secs 2024-04-26T17:22:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:58 smithi033 ceph-mon[27845]: pgmap v11077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19916.998782 secs 2024-04-26T17:22:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:58 smithi096 ceph-mon[32753]: pgmap v11077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:22:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19916.998782 secs 2024-04-26T17:23:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19919.846662 secs 2024-04-26T17:23:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19919.846662 secs 2024-04-26T17:23:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19919.846662 secs 2024-04-26T17:23:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:00 smithi194 ceph-mon[33017]: pgmap v11078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:01.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:00 smithi033 ceph-mon[27845]: pgmap v11078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:00 smithi096 ceph-mon[32753]: pgmap v11078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:02.422+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:02.422+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:23:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:02.422+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:02 smithi194 ceph-mon[33017]: pgmap v11079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:02 smithi033 ceph-mon[27845]: pgmap v11079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:02 smithi096 ceph-mon[32753]: pgmap v11079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19921.998859 secs 2024-04-26T17:23:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19921.998859 secs 2024-04-26T17:23:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19921.998859 secs 2024-04-26T17:23:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19924.846779 secs 2024-04-26T17:23:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:04 smithi194 ceph-mon[33017]: pgmap v11080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19924.846779 secs 2024-04-26T17:23:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:04 smithi033 ceph-mon[27845]: pgmap v11080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19924.846779 secs 2024-04-26T17:23:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:04 smithi096 ceph-mon[32753]: pgmap v11080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:06 smithi194 ceph-mon[33017]: pgmap v11081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:06 smithi033 ceph-mon[27845]: pgmap v11081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:06 smithi096 ceph-mon[32753]: pgmap v11081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:08 smithi194 ceph-mon[33017]: pgmap v11082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19926.998938 secs 2024-04-26T17:23:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:08 smithi033 ceph-mon[27845]: pgmap v11082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19926.998938 secs 2024-04-26T17:23:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:08 smithi096 ceph-mon[32753]: pgmap v11082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19926.998938 secs 2024-04-26T17:23:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19929.846898 secs 2024-04-26T17:23:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19929.846898 secs 2024-04-26T17:23:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19929.846898 secs 2024-04-26T17:23:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:10 smithi194 ceph-mon[33017]: pgmap v11083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:10 smithi033 ceph-mon[27845]: pgmap v11083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:10 smithi096 ceph-mon[32753]: pgmap v11083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:12 smithi194 ceph-mon[33017]: pgmap v11084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:12 smithi033 ceph-mon[27845]: pgmap v11084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:12 smithi096 ceph-mon[32753]: pgmap v11084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19931.999020 secs 2024-04-26T17:23:14.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19931.999020 secs 2024-04-26T17:23:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19931.999020 secs 2024-04-26T17:23:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19934.847034 secs 2024-04-26T17:23:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:14 smithi194 ceph-mon[33017]: pgmap v11085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19934.847034 secs 2024-04-26T17:23:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:14 smithi033 ceph-mon[27845]: pgmap v11085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19934.847034 secs 2024-04-26T17:23:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:14 smithi096 ceph-mon[32753]: pgmap v11085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:16 smithi194 ceph-mon[33017]: pgmap v11086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:16 smithi033 ceph-mon[27845]: pgmap v11086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:16 smithi096 ceph-mon[32753]: pgmap v11086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:17.433+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:17.433+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:17.433+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:23:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:18 smithi194 ceph-mon[33017]: pgmap v11087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19936.999081 secs 2024-04-26T17:23:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:18 smithi033 ceph-mon[27845]: pgmap v11087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19936.999081 secs 2024-04-26T17:23:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:18 smithi096 ceph-mon[32753]: pgmap v11087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19936.999081 secs 2024-04-26T17:23:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19939.847135 secs 2024-04-26T17:23:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19939.847135 secs 2024-04-26T17:23:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19939.847135 secs 2024-04-26T17:23:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:20 smithi033 ceph-mon[27845]: pgmap v11088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:20 smithi096 ceph-mon[32753]: pgmap v11088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:20 smithi194 ceph-mon[33017]: pgmap v11088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:22 smithi033 ceph-mon[27845]: pgmap v11089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:22 smithi096 ceph-mon[32753]: pgmap v11089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:22 smithi194 ceph-mon[33017]: pgmap v11089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19941.999165 secs 2024-04-26T17:23:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19941.999165 secs 2024-04-26T17:23:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19941.999165 secs 2024-04-26T17:23:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19944.847253 secs 2024-04-26T17:23:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:24 smithi033 ceph-mon[27845]: pgmap v11090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:23:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19944.847253 secs 2024-04-26T17:23:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:24 smithi096 ceph-mon[32753]: pgmap v11090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:25.184+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:25.184+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:23:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:25.185+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 160 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:23:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:25.185+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 160 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:23:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19944.847253 secs 2024-04-26T17:23:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:24 smithi194 ceph-mon[33017]: pgmap v11090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:23:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:23:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:23:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:23:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:23:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:23:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:23:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:23:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:23:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:23:27.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:26 smithi033 ceph-mon[27845]: pgmap v11091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:23:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:26 smithi033 ceph-mon[27845]: pgmap v11092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:26 smithi096 ceph-mon[32753]: pgmap v11091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:23:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:26 smithi096 ceph-mon[32753]: pgmap v11092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:26 smithi194 ceph-mon[33017]: pgmap v11091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:23:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:26 smithi194 ceph-mon[33017]: pgmap v11092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:28 smithi033 ceph-mon[27845]: pgmap v11093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19946.999225 secs 2024-04-26T17:23:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:28 smithi096 ceph-mon[32753]: pgmap v11093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19946.999225 secs 2024-04-26T17:23:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:28 smithi194 ceph-mon[33017]: pgmap v11093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19946.999225 secs 2024-04-26T17:23:30.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19949.847334 secs 2024-04-26T17:23:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19949.847334 secs 2024-04-26T17:23:30.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19949.847334 secs 2024-04-26T17:23:31.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:30 smithi033 ceph-mon[27845]: pgmap v11094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T17:23:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:30 smithi096 ceph-mon[32753]: pgmap v11094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T17:23:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:30 smithi194 ceph-mon[33017]: pgmap v11094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T17:23:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:32.448+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:32.448+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:32.448+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:23:33.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:32 smithi033 ceph-mon[27845]: pgmap v11095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:32 smithi096 ceph-mon[32753]: pgmap v11095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:32 smithi194 ceph-mon[33017]: pgmap v11095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19951.999308 secs 2024-04-26T17:23:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19951.999308 secs 2024-04-26T17:23:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19951.999308 secs 2024-04-26T17:23:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:34 smithi096 ceph-mon[32753]: pgmap v11096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19954.847414 secs 2024-04-26T17:23:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:34 smithi194 ceph-mon[33017]: pgmap v11096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19954.847414 secs 2024-04-26T17:23:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:34 smithi033 ceph-mon[27845]: pgmap v11096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19954.847414 secs 2024-04-26T17:23:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:36 smithi096 ceph-mon[32753]: pgmap v11097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:23:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:36 smithi194 ceph-mon[33017]: pgmap v11097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:23:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:36 smithi033 ceph-mon[27845]: pgmap v11097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:23:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:38 smithi194 ceph-mon[33017]: pgmap v11098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19956.999370 secs 2024-04-26T17:23:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:38 smithi033 ceph-mon[27845]: pgmap v11098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19956.999370 secs 2024-04-26T17:23:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:38 smithi096 ceph-mon[32753]: pgmap v11098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19956.999370 secs 2024-04-26T17:23:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19959.847533 secs 2024-04-26T17:23:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19959.847533 secs 2024-04-26T17:23:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19959.847533 secs 2024-04-26T17:23:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:40 smithi194 ceph-mon[33017]: pgmap v11099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:23:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:40 smithi033 ceph-mon[27845]: pgmap v11099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:23:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:40 smithi096 ceph-mon[32753]: pgmap v11099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:23:43.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:42 smithi096 ceph-mon[32753]: pgmap v11100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:42 smithi194 ceph-mon[33017]: pgmap v11100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:42 smithi033 ceph-mon[27845]: pgmap v11100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19961.999452 secs 2024-04-26T17:23:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19961.999452 secs 2024-04-26T17:23:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19961.999452 secs 2024-04-26T17:23:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:44 smithi194 ceph-mon[33017]: pgmap v11101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19964.847670 secs 2024-04-26T17:23:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:44 smithi033 ceph-mon[27845]: pgmap v11101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19964.847670 secs 2024-04-26T17:23:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:44 smithi096 ceph-mon[32753]: pgmap v11101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19964.847670 secs 2024-04-26T17:23:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:46 smithi194 ceph-mon[33017]: pgmap v11102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:46 smithi033 ceph-mon[27845]: pgmap v11102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:46 smithi096 ceph-mon[32753]: pgmap v11102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:47.457+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:47.457+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:23:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:23:47.457+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:23:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:48 smithi194 ceph-mon[33017]: pgmap v11103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19966.999536 secs 2024-04-26T17:23:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:48 smithi033 ceph-mon[27845]: pgmap v11103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19966.999536 secs 2024-04-26T17:23:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:48 smithi096 ceph-mon[32753]: pgmap v11103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19966.999536 secs 2024-04-26T17:23:50.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19969.847788 secs 2024-04-26T17:23:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19969.847788 secs 2024-04-26T17:23:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19969.847788 secs 2024-04-26T17:23:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:50 smithi194 ceph-mon[33017]: pgmap v11104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:50 smithi033 ceph-mon[27845]: pgmap v11104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:50 smithi096 ceph-mon[32753]: pgmap v11104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:52 smithi194 ceph-mon[33017]: pgmap v11105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:52 smithi033 ceph-mon[27845]: pgmap v11105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:52 smithi096 ceph-mon[32753]: pgmap v11105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:23:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19971.999623 secs 2024-04-26T17:23:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19971.999623 secs 2024-04-26T17:23:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19971.999623 secs 2024-04-26T17:23:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:55 smithi194 ceph-mon[33017]: pgmap v11106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19974.847904 secs 2024-04-26T17:23:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:55 smithi033 ceph-mon[27845]: pgmap v11106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19974.847904 secs 2024-04-26T17:23:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:55 smithi096 ceph-mon[32753]: pgmap v11106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19974.847904 secs 2024-04-26T17:23:57.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:57 smithi033 ceph-mon[27845]: pgmap v11107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:57 smithi194 ceph-mon[33017]: pgmap v11107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:57 smithi096 ceph-mon[32753]: pgmap v11107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:23:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:59 smithi194 ceph-mon[33017]: pgmap v11108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:23:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19976.999682 secs 2024-04-26T17:23:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:59 smithi033 ceph-mon[27845]: pgmap v11108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:23:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19976.999682 secs 2024-04-26T17:23:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:59 smithi096 ceph-mon[32753]: pgmap v11108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:23:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:23:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19976.999682 secs 2024-04-26T17:24:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19979.848015 secs 2024-04-26T17:24:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19979.848015 secs 2024-04-26T17:24:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19979.848015 secs 2024-04-26T17:24:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:01 smithi194 ceph-mon[33017]: pgmap v11109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:24:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:01 smithi033 ceph-mon[27845]: pgmap v11109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:24:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:01 smithi096 ceph-mon[32753]: pgmap v11109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:24:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:02.461+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:02.461+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:02.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:02.461+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:24:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:03 smithi194 ceph-mon[33017]: pgmap v11110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:24:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:03 smithi033 ceph-mon[27845]: pgmap v11110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:24:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:03 smithi096 ceph-mon[32753]: pgmap v11110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:24:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19981.999765 secs 2024-04-26T17:24:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19981.999765 secs 2024-04-26T17:24:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19981.999765 secs 2024-04-26T17:24:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:05 smithi194 ceph-mon[33017]: pgmap v11111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19984.848131 secs 2024-04-26T17:24:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:05 smithi033 ceph-mon[27845]: pgmap v11111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19984.848131 secs 2024-04-26T17:24:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:05 smithi096 ceph-mon[32753]: pgmap v11111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19984.848131 secs 2024-04-26T17:24:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:07 smithi194 ceph-mon[33017]: pgmap v11112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:07 smithi033 ceph-mon[27845]: pgmap v11112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:07 smithi096 ceph-mon[32753]: pgmap v11112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:09 smithi194 ceph-mon[33017]: pgmap v11113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19986.999847 secs 2024-04-26T17:24:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:09 smithi033 ceph-mon[27845]: pgmap v11113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19986.999847 secs 2024-04-26T17:24:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:09 smithi096 ceph-mon[32753]: pgmap v11113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19986.999847 secs 2024-04-26T17:24:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19989.848215 secs 2024-04-26T17:24:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19989.848215 secs 2024-04-26T17:24:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19989.848215 secs 2024-04-26T17:24:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:11 smithi194 ceph-mon[33017]: pgmap v11114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:11 smithi033 ceph-mon[27845]: pgmap v11114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:11 smithi096 ceph-mon[32753]: pgmap v11114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:13 smithi194 ceph-mon[33017]: pgmap v11115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:24:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:13 smithi033 ceph-mon[27845]: pgmap v11115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:24:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:13 smithi096 ceph-mon[32753]: pgmap v11115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:24:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19991.999905 secs 2024-04-26T17:24:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19991.999905 secs 2024-04-26T17:24:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19991.999905 secs 2024-04-26T17:24:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:15 smithi194 ceph-mon[33017]: pgmap v11116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19994.848335 secs 2024-04-26T17:24:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:15 smithi033 ceph-mon[27845]: pgmap v11116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19994.848335 secs 2024-04-26T17:24:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:15 smithi096 ceph-mon[32753]: pgmap v11116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19994.848335 secs 2024-04-26T17:24:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:17 smithi194 ceph-mon[33017]: pgmap v11117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:17 smithi033 ceph-mon[27845]: pgmap v11117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:17 smithi096 ceph-mon[32753]: pgmap v11117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:17.475+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:17.475+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:17.475+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:24:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:19 smithi033 ceph-mon[27845]: pgmap v11118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 19997.000002 secs 2024-04-26T17:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 19999.848448 secs 2024-04-26T17:24:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:19 smithi096 ceph-mon[32753]: pgmap v11118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 19997.000002 secs 2024-04-26T17:24:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 19999.848448 secs 2024-04-26T17:24:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:19 smithi194 ceph-mon[33017]: pgmap v11118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 19997.000002 secs 2024-04-26T17:24:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 19999.848448 secs 2024-04-26T17:24:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:21 smithi033 ceph-mon[27845]: pgmap v11119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:21 smithi096 ceph-mon[32753]: pgmap v11119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:21 smithi194 ceph-mon[33017]: pgmap v11119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:23 smithi033 ceph-mon[27845]: pgmap v11120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:23 smithi096 ceph-mon[32753]: pgmap v11120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:23 smithi194 ceph-mon[33017]: pgmap v11120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20002.000061 secs 2024-04-26T17:24:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20004.848568 secs 2024-04-26T17:24:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20002.000061 secs 2024-04-26T17:24:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20004.848568 secs 2024-04-26T17:24:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20002.000061 secs 2024-04-26T17:24:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20004.848568 secs 2024-04-26T17:24:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:25 smithi033 ceph-mon[27845]: pgmap v11121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:25 smithi096 ceph-mon[32753]: pgmap v11121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:25 smithi194 ceph-mon[33017]: pgmap v11121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:24:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:24:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:24:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:27 smithi194 ceph-mon[33017]: pgmap v11122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:27 smithi033 ceph-mon[27845]: pgmap v11122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:27 smithi096 ceph-mon[32753]: pgmap v11122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: pgmap v11123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20007.000131 secs 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20009.848635 secs 2024-04-26T17:24:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: pgmap v11123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20007.000131 secs 2024-04-26T17:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20009.848635 secs 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: pgmap v11123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20007.000131 secs 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20009.848635 secs 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:30 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:24:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:30 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:24:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:30 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:24:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:30 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:24:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:30 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:24:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:30 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:24:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:30 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:24:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:30 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:24:30.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:30 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:24:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:30 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:24:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:30 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:24:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:30 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:24:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:31 smithi194 ceph-mon[33017]: pgmap v11124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:31 smithi033 ceph-mon[27845]: pgmap v11124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:31 smithi096 ceph-mon[32753]: pgmap v11124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:32.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:32.488+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:32.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:32.488+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:32.488 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:32.488+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:24:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:24:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:24:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:24:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:24:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:24:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:33 smithi194 ceph-mon[33017]: pgmap v11125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:33 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:24:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:33 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:24:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:33 smithi194 ceph-mon[33017]: pgmap v11126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:24:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:33 smithi194 ceph-mon[33017]: pgmap v11127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:33 smithi033 ceph-mon[27845]: pgmap v11125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:33 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:24:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:33 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:24:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:33 smithi033 ceph-mon[27845]: pgmap v11126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:24:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:33 smithi033 ceph-mon[27845]: pgmap v11127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:33 smithi096 ceph-mon[32753]: pgmap v11125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:33 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:24:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:33 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:24:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:33 smithi096 ceph-mon[32753]: pgmap v11126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:24:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:33 smithi096 ceph-mon[32753]: pgmap v11127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20012.000215 secs 2024-04-26T17:24:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20014.848759 secs 2024-04-26T17:24:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20012.000215 secs 2024-04-26T17:24:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20014.848759 secs 2024-04-26T17:24:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20012.000215 secs 2024-04-26T17:24:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20014.848759 secs 2024-04-26T17:24:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:35 smithi194 ceph-mon[33017]: pgmap v11128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:24:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:35 smithi033 ceph-mon[27845]: pgmap v11128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:24:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:35 smithi096 ceph-mon[32753]: pgmap v11128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:24:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:37 smithi194 ceph-mon[33017]: pgmap v11129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:37 smithi033 ceph-mon[27845]: pgmap v11129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:37 smithi096 ceph-mon[32753]: pgmap v11129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20017.000278 secs 2024-04-26T17:24:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:39 smithi194 ceph-mon[33017]: pgmap v11130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:24:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20019.848825 secs 2024-04-26T17:24:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20017.000278 secs 2024-04-26T17:24:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:39 smithi033 ceph-mon[27845]: pgmap v11130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:24:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20019.848825 secs 2024-04-26T17:24:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20017.000278 secs 2024-04-26T17:24:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:39 smithi096 ceph-mon[32753]: pgmap v11130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:24:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20019.848825 secs 2024-04-26T17:24:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:41 smithi194 ceph-mon[33017]: pgmap v11131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:41 smithi033 ceph-mon[27845]: pgmap v11131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:41 smithi096 ceph-mon[32753]: pgmap v11131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:24:43.497 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:43 smithi096 ceph-mon[32753]: pgmap v11132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:24:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:43 smithi194 ceph-mon[33017]: pgmap v11132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:24:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:43 smithi033 ceph-mon[27845]: pgmap v11132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:24:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20022.000361 secs 2024-04-26T17:24:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20024.848945 secs 2024-04-26T17:24:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20022.000361 secs 2024-04-26T17:24:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20024.848945 secs 2024-04-26T17:24:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20022.000361 secs 2024-04-26T17:24:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20024.848945 secs 2024-04-26T17:24:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:45 smithi194 ceph-mon[33017]: pgmap v11133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:45 smithi033 ceph-mon[27845]: pgmap v11133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:45 smithi096 ceph-mon[32753]: pgmap v11133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:47.502+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:47.502+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:24:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:24:47.502+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:24:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:47 smithi194 ceph-mon[33017]: pgmap v11134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:47 smithi033 ceph-mon[27845]: pgmap v11134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:47 smithi096 ceph-mon[32753]: pgmap v11134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20027.000444 secs 2024-04-26T17:24:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:49 smithi194 ceph-mon[33017]: pgmap v11135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20029.849068 secs 2024-04-26T17:24:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20027.000444 secs 2024-04-26T17:24:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:49 smithi033 ceph-mon[27845]: pgmap v11135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20029.849068 secs 2024-04-26T17:24:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20027.000444 secs 2024-04-26T17:24:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:49 smithi096 ceph-mon[32753]: pgmap v11135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20029.849068 secs 2024-04-26T17:24:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:51 smithi194 ceph-mon[33017]: pgmap v11136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:51 smithi033 ceph-mon[27845]: pgmap v11136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:51 smithi096 ceph-mon[32753]: pgmap v11136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:53 smithi194 ceph-mon[33017]: pgmap v11137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:53 smithi033 ceph-mon[27845]: pgmap v11137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:53 smithi096 ceph-mon[32753]: pgmap v11137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20032.000526 secs 2024-04-26T17:24:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20034.849173 secs 2024-04-26T17:24:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20032.000526 secs 2024-04-26T17:24:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20034.849173 secs 2024-04-26T17:24:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20032.000526 secs 2024-04-26T17:24:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20034.849173 secs 2024-04-26T17:24:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:55 smithi194 ceph-mon[33017]: pgmap v11138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:55 smithi033 ceph-mon[27845]: pgmap v11138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:55 smithi096 ceph-mon[32753]: pgmap v11138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:57 smithi194 ceph-mon[33017]: pgmap v11139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:57 smithi033 ceph-mon[27845]: pgmap v11139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:57 smithi096 ceph-mon[32753]: pgmap v11139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:24:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20037.000610 secs 2024-04-26T17:24:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:59 smithi194 ceph-mon[33017]: pgmap v11140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20039.849242 secs 2024-04-26T17:24:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20037.000610 secs 2024-04-26T17:24:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:59 smithi033 ceph-mon[27845]: pgmap v11140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20039.849242 secs 2024-04-26T17:24:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20037.000610 secs 2024-04-26T17:24:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:59 smithi096 ceph-mon[32753]: pgmap v11140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:24:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20039.849242 secs 2024-04-26T17:25:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:01 smithi194 ceph-mon[33017]: pgmap v11141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:01 smithi033 ceph-mon[27845]: pgmap v11141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:01 smithi096 ceph-mon[32753]: pgmap v11141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:02.518+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:02.518+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:02.518+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:25:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:03 smithi194 ceph-mon[33017]: pgmap v11142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:03 smithi033 ceph-mon[27845]: pgmap v11142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:03 smithi096 ceph-mon[32753]: pgmap v11142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20042.000696 secs 2024-04-26T17:25:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20044.849310 secs 2024-04-26T17:25:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:04 smithi194 ceph-mon[33017]: pgmap v11143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20042.000696 secs 2024-04-26T17:25:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20044.849310 secs 2024-04-26T17:25:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:04 smithi033 ceph-mon[27845]: pgmap v11143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20042.000696 secs 2024-04-26T17:25:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20044.849310 secs 2024-04-26T17:25:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:04 smithi096 ceph-mon[32753]: pgmap v11143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:06 smithi194 ceph-mon[33017]: pgmap v11144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:07.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:06 smithi033 ceph-mon[27845]: pgmap v11144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:06 smithi096 ceph-mon[32753]: pgmap v11144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20047.000757 secs 2024-04-26T17:25:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:08 smithi033 ceph-mon[27845]: pgmap v11145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20047.000757 secs 2024-04-26T17:25:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:08 smithi096 ceph-mon[32753]: pgmap v11145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20047.000757 secs 2024-04-26T17:25:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:08 smithi194 ceph-mon[33017]: pgmap v11145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20049.849427 secs 2024-04-26T17:25:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20049.849427 secs 2024-04-26T17:25:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20049.849427 secs 2024-04-26T17:25:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:10 smithi033 ceph-mon[27845]: pgmap v11146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:10 smithi096 ceph-mon[32753]: pgmap v11146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:10 smithi194 ceph-mon[33017]: pgmap v11146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:12 smithi033 ceph-mon[27845]: pgmap v11147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:12 smithi096 ceph-mon[32753]: pgmap v11147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:12 smithi194 ceph-mon[33017]: pgmap v11147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20052.000842 secs 2024-04-26T17:25:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20052.000842 secs 2024-04-26T17:25:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20052.000842 secs 2024-04-26T17:25:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20054.849560 secs 2024-04-26T17:25:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:14 smithi033 ceph-mon[27845]: pgmap v11148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20054.849560 secs 2024-04-26T17:25:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:14 smithi096 ceph-mon[32753]: pgmap v11148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20054.849560 secs 2024-04-26T17:25:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:14 smithi194 ceph-mon[33017]: pgmap v11148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:17.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:16 smithi033 ceph-mon[27845]: pgmap v11149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:16 smithi096 ceph-mon[32753]: pgmap v11149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:16 smithi194 ceph-mon[33017]: pgmap v11149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:17.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:17.530+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:17.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:17.530+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:17.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:17.530+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:25:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20057.000927 secs 2024-04-26T17:25:19.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:18 smithi033 ceph-mon[27845]: pgmap v11150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20057.000927 secs 2024-04-26T17:25:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:18 smithi096 ceph-mon[32753]: pgmap v11150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20057.000927 secs 2024-04-26T17:25:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:18 smithi194 ceph-mon[33017]: pgmap v11150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20059.849679 secs 2024-04-26T17:25:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20059.849679 secs 2024-04-26T17:25:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20059.849679 secs 2024-04-26T17:25:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:20 smithi033 ceph-mon[27845]: pgmap v11151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:20 smithi096 ceph-mon[32753]: pgmap v11151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:20 smithi194 ceph-mon[33017]: pgmap v11151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:23.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:22 smithi033 ceph-mon[27845]: pgmap v11152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:22 smithi096 ceph-mon[32753]: pgmap v11152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:22 smithi194 ceph-mon[33017]: pgmap v11152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20062.001007 secs 2024-04-26T17:25:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20062.001007 secs 2024-04-26T17:25:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20062.001007 secs 2024-04-26T17:25:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20064.849799 secs 2024-04-26T17:25:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:24 smithi033 ceph-mon[27845]: pgmap v11153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20064.849799 secs 2024-04-26T17:25:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:24 smithi096 ceph-mon[32753]: pgmap v11153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20064.849799 secs 2024-04-26T17:25:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:24 smithi194 ceph-mon[33017]: pgmap v11153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:26 smithi096 ceph-mon[32753]: pgmap v11154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:26 smithi194 ceph-mon[33017]: pgmap v11154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:26 smithi033 ceph-mon[27845]: pgmap v11154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20067.001095 secs 2024-04-26T17:25:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:28 smithi096 ceph-mon[32753]: pgmap v11155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20067.001095 secs 2024-04-26T17:25:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:28 smithi194 ceph-mon[33017]: pgmap v11155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20067.001095 secs 2024-04-26T17:25:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:28 smithi033 ceph-mon[27845]: pgmap v11155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20069.849926 secs 2024-04-26T17:25:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20069.849926 secs 2024-04-26T17:25:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20069.849926 secs 2024-04-26T17:25:31.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:30 smithi194 ceph-mon[33017]: pgmap v11156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:30 smithi033 ceph-mon[27845]: pgmap v11156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:30 smithi096 ceph-mon[32753]: pgmap v11156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:32.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:32.532+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:32.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:32.533+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:32.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:32.533+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:25:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:32 smithi194 ceph-mon[33017]: pgmap v11157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:25:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:32 smithi033 ceph-mon[27845]: pgmap v11157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:25:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:32 smithi096 ceph-mon[32753]: pgmap v11157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:25:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20072.001187 secs 2024-04-26T17:25:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:25:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:25:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20072.001187 secs 2024-04-26T17:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:25:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20072.001187 secs 2024-04-26T17:25:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:25:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:25:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:25:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20074.850057 secs 2024-04-26T17:25:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:34 smithi194 ceph-mon[33017]: pgmap v11158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:34 smithi194 ceph-mon[33017]: pgmap v11159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20074.850057 secs 2024-04-26T17:25:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:34 smithi033 ceph-mon[27845]: pgmap v11158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:34 smithi033 ceph-mon[27845]: pgmap v11159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20074.850057 secs 2024-04-26T17:25:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:34 smithi096 ceph-mon[32753]: pgmap v11158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:34 smithi096 ceph-mon[32753]: pgmap v11159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:36 smithi194 ceph-mon[33017]: pgmap v11160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:36 smithi033 ceph-mon[27845]: pgmap v11160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:36 smithi096 ceph-mon[32753]: pgmap v11160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:38 smithi194 ceph-mon[33017]: pgmap v11161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20077.001264 secs 2024-04-26T17:25:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:38 smithi033 ceph-mon[27845]: pgmap v11161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20077.001264 secs 2024-04-26T17:25:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:38 smithi096 ceph-mon[32753]: pgmap v11161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:39.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20077.001264 secs 2024-04-26T17:25:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20079.850182 secs 2024-04-26T17:25:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20079.850182 secs 2024-04-26T17:25:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20079.850182 secs 2024-04-26T17:25:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:40 smithi194 ceph-mon[33017]: pgmap v11162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:41.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:40 smithi033 ceph-mon[27845]: pgmap v11162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:40 smithi096 ceph-mon[32753]: pgmap v11162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:42 smithi194 ceph-mon[33017]: pgmap v11163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:42 smithi033 ceph-mon[27845]: pgmap v11163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:42 smithi096 ceph-mon[32753]: pgmap v11163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:25:44.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20082.001347 secs 2024-04-26T17:25:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20082.001347 secs 2024-04-26T17:25:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20082.001347 secs 2024-04-26T17:25:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20084.850338 secs 2024-04-26T17:25:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:45 smithi194 ceph-mon[33017]: pgmap v11164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:25:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20084.850338 secs 2024-04-26T17:25:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:45 smithi033 ceph-mon[27845]: pgmap v11164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:25:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20084.850338 secs 2024-04-26T17:25:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:45 smithi096 ceph-mon[32753]: pgmap v11164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:25:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:47 smithi194 ceph-mon[33017]: pgmap v11165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:47 smithi033 ceph-mon[27845]: pgmap v11165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:47 smithi096 ceph-mon[32753]: pgmap v11165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:47.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:47.543+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:47.543+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:25:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:25:47.543+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:25:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:49 smithi194 ceph-mon[33017]: pgmap v11166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20087.001435 secs 2024-04-26T17:25:49.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:49 smithi033 ceph-mon[27845]: pgmap v11166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20087.001435 secs 2024-04-26T17:25:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:49 smithi096 ceph-mon[32753]: pgmap v11166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20087.001435 secs 2024-04-26T17:25:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20089.850417 secs 2024-04-26T17:25:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20089.850417 secs 2024-04-26T17:25:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20089.850417 secs 2024-04-26T17:25:51.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:51 smithi033 ceph-mon[27845]: pgmap v11167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:51 smithi194 ceph-mon[33017]: pgmap v11167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:51 smithi096 ceph-mon[32753]: pgmap v11167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:25:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:53 smithi194 ceph-mon[33017]: pgmap v11168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:53 smithi033 ceph-mon[27845]: pgmap v11168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:53 smithi096 ceph-mon[32753]: pgmap v11168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:54.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20092.001519 secs 2024-04-26T17:25:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20092.001519 secs 2024-04-26T17:25:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20092.001519 secs 2024-04-26T17:25:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20094.850540 secs 2024-04-26T17:25:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:55 smithi194 ceph-mon[33017]: pgmap v11169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20094.850540 secs 2024-04-26T17:25:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:55 smithi033 ceph-mon[27845]: pgmap v11169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20094.850540 secs 2024-04-26T17:25:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:55 smithi096 ceph-mon[32753]: pgmap v11169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:25:57.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:57 smithi194 ceph-mon[33017]: pgmap v11170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:25:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:57 smithi033 ceph-mon[27845]: pgmap v11170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:25:57.445 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:57 smithi096 ceph-mon[32753]: pgmap v11170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:25:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:59 smithi194 ceph-mon[33017]: pgmap v11171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:25:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:25:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20097.001580 secs 2024-04-26T17:25:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:59 smithi033 ceph-mon[27845]: pgmap v11171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:25:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:25:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20097.001580 secs 2024-04-26T17:25:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:59 smithi096 ceph-mon[32753]: pgmap v11171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:25:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:25:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20097.001580 secs 2024-04-26T17:26:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20099.850639 secs 2024-04-26T17:26:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20099.850639 secs 2024-04-26T17:26:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20099.850639 secs 2024-04-26T17:26:01.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:01 smithi194 ceph-mon[33017]: pgmap v11172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:26:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:01 smithi033 ceph-mon[27845]: pgmap v11172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:26:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:01 smithi096 ceph-mon[32753]: pgmap v11172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:26:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:02.550+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:02.550+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:02.550+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:26:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:03 smithi194 ceph-mon[33017]: pgmap v11173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:26:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:03 smithi033 ceph-mon[27845]: pgmap v11173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:26:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:03 smithi096 ceph-mon[32753]: pgmap v11173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:26:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20102.001665 secs 2024-04-26T17:26:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20102.001665 secs 2024-04-26T17:26:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20102.001665 secs 2024-04-26T17:26:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20104.850763 secs 2024-04-26T17:26:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:05 smithi194 ceph-mon[33017]: pgmap v11174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:26:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20104.850763 secs 2024-04-26T17:26:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:05 smithi033 ceph-mon[27845]: pgmap v11174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:26:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20104.850763 secs 2024-04-26T17:26:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:05 smithi096 ceph-mon[32753]: pgmap v11174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:26:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:07 smithi194 ceph-mon[33017]: pgmap v11175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:26:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:07 smithi033 ceph-mon[27845]: pgmap v11175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:26:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:07 smithi096 ceph-mon[32753]: pgmap v11175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:26:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:09 smithi194 ceph-mon[33017]: pgmap v11176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20107.001749 secs 2024-04-26T17:26:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:09 smithi033 ceph-mon[27845]: pgmap v11176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20107.001749 secs 2024-04-26T17:26:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:09 smithi096 ceph-mon[32753]: pgmap v11176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20107.001749 secs 2024-04-26T17:26:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20109.850885 secs 2024-04-26T17:26:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20109.850885 secs 2024-04-26T17:26:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20109.850885 secs 2024-04-26T17:26:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:11 smithi033 ceph-mon[27845]: pgmap v11177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:11 smithi096 ceph-mon[32753]: pgmap v11177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:11 smithi194 ceph-mon[33017]: pgmap v11177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:13 smithi033 ceph-mon[27845]: pgmap v11178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:13 smithi096 ceph-mon[32753]: pgmap v11178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:13 smithi194 ceph-mon[33017]: pgmap v11178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20112.001834 secs 2024-04-26T17:26:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20114.851035 secs 2024-04-26T17:26:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20112.001834 secs 2024-04-26T17:26:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20114.851035 secs 2024-04-26T17:26:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20112.001834 secs 2024-04-26T17:26:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20114.851035 secs 2024-04-26T17:26:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:15 smithi033 ceph-mon[27845]: pgmap v11179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:15 smithi096 ceph-mon[32753]: pgmap v11179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:15 smithi194 ceph-mon[33017]: pgmap v11179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:17 smithi033 ceph-mon[27845]: pgmap v11180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:17 smithi096 ceph-mon[32753]: pgmap v11180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:17.560+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:17.560+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:17.560+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:26:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:17 smithi194 ceph-mon[33017]: pgmap v11180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:19 smithi033 ceph-mon[27845]: pgmap v11181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20117.001921 secs 2024-04-26T17:26:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20119.851160 secs 2024-04-26T17:26:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:19 smithi096 ceph-mon[32753]: pgmap v11181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20117.001921 secs 2024-04-26T17:26:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20119.851160 secs 2024-04-26T17:26:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:19 smithi194 ceph-mon[33017]: pgmap v11181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20117.001921 secs 2024-04-26T17:26:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20119.851160 secs 2024-04-26T17:26:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:21 smithi033 ceph-mon[27845]: pgmap v11182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:21 smithi096 ceph-mon[32753]: pgmap v11182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:21 smithi194 ceph-mon[33017]: pgmap v11182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T17:26:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T17:26:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0058040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T17:26:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0058040 comp rx=0 tx=0).stop 2024-04-26T17:26:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T17:26:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 msgr2=0x7feee40b3c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T17:26:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3330 0x7feee40b3c50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:26:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.768+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:26:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.769+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:26:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:21.771+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0074000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:26:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:23 smithi033 ceph-mon[27845]: pgmap v11183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:23 smithi096 ceph-mon[32753]: pgmap v11183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:23 smithi194 ceph-mon[33017]: pgmap v11183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20122.002007 secs 2024-04-26T17:26:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20124.851247 secs 2024-04-26T17:26:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20122.002007 secs 2024-04-26T17:26:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20124.851247 secs 2024-04-26T17:26:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20122.002007 secs 2024-04-26T17:26:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20124.851247 secs 2024-04-26T17:26:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:25.216+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:26:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:25.216+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:26:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:25.217+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 161 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006630 con 0x564385821900 2024-04-26T17:26:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:25.217+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 161 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:26:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:25 smithi033 ceph-mon[27845]: pgmap v11184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:25 smithi096 ceph-mon[32753]: pgmap v11184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:25 smithi194 ceph-mon[33017]: pgmap v11184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:27 smithi033 ceph-mon[27845]: pgmap v11185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:27 smithi096 ceph-mon[32753]: pgmap v11185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:27 smithi194 ceph-mon[33017]: pgmap v11185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:29 smithi096 ceph-mon[32753]: pgmap v11186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20127.002083 secs 2024-04-26T17:26:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20129.851347 secs 2024-04-26T17:26:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:29 smithi194 ceph-mon[33017]: pgmap v11186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20127.002083 secs 2024-04-26T17:26:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20129.851347 secs 2024-04-26T17:26:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:29 smithi033 ceph-mon[27845]: pgmap v11186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20127.002083 secs 2024-04-26T17:26:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20129.851347 secs 2024-04-26T17:26:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:31 smithi096 ceph-mon[32753]: pgmap v11187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:31 smithi194 ceph-mon[33017]: pgmap v11187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:31 smithi033 ceph-mon[27845]: pgmap v11187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:32.570 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:32.571+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:32.571+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:32.571+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:26:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:33 smithi033 ceph-mon[27845]: pgmap v11188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:33 smithi194 ceph-mon[33017]: pgmap v11188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:33 smithi096 ceph-mon[32753]: pgmap v11188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20132.002168 secs 2024-04-26T17:26:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20134.851430 secs 2024-04-26T17:26:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:26:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20132.002168 secs 2024-04-26T17:26:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20134.851430 secs 2024-04-26T17:26:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:26:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20132.002168 secs 2024-04-26T17:26:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20134.851430 secs 2024-04-26T17:26:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:26:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:35 smithi194 ceph-mon[33017]: pgmap v11189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:26:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:26:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:26:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:35 smithi033 ceph-mon[27845]: pgmap v11189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:26:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:26:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:26:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:35 smithi096 ceph-mon[32753]: pgmap v11189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:26:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:26:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:26:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:36 smithi194 ceph-mon[33017]: pgmap v11190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:26:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:36 smithi194 ceph-mon[33017]: pgmap v11191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:36 smithi033 ceph-mon[27845]: pgmap v11190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:26:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:36 smithi033 ceph-mon[27845]: pgmap v11191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:36 smithi096 ceph-mon[32753]: pgmap v11190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:26:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:36 smithi096 ceph-mon[32753]: pgmap v11191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:38 smithi194 ceph-mon[33017]: pgmap v11192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:38 smithi033 ceph-mon[27845]: pgmap v11192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:38 smithi096 ceph-mon[32753]: pgmap v11192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20137.002277 secs 2024-04-26T17:26:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20139.851561 secs 2024-04-26T17:26:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20137.002277 secs 2024-04-26T17:26:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20139.851561 secs 2024-04-26T17:26:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20137.002277 secs 2024-04-26T17:26:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20139.851561 secs 2024-04-26T17:26:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:40 smithi194 ceph-mon[33017]: pgmap v11193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:26:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:40 smithi033 ceph-mon[27845]: pgmap v11193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:26:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:40 smithi096 ceph-mon[32753]: pgmap v11193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:26:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:42 smithi194 ceph-mon[33017]: pgmap v11194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:42 smithi033 ceph-mon[27845]: pgmap v11194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:42 smithi096 ceph-mon[32753]: pgmap v11194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20142.002364 secs 2024-04-26T17:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:44 smithi194 ceph-mon[33017]: pgmap v11195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20144.851645 secs 2024-04-26T17:26:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20142.002364 secs 2024-04-26T17:26:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:44 smithi033 ceph-mon[27845]: pgmap v11195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20144.851645 secs 2024-04-26T17:26:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20142.002364 secs 2024-04-26T17:26:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:44 smithi096 ceph-mon[32753]: pgmap v11195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:26:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20144.851645 secs 2024-04-26T17:26:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:46 smithi194 ceph-mon[33017]: pgmap v11196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:26:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:46 smithi033 ceph-mon[27845]: pgmap v11196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:26:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:46 smithi096 ceph-mon[32753]: pgmap v11196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:26:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:47.585+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:47.585+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:26:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:26:47.585+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:26:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:48 smithi194 ceph-mon[33017]: pgmap v11197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:48 smithi033 ceph-mon[27845]: pgmap v11197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:48 smithi096 ceph-mon[32753]: pgmap v11197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20147.002450 secs 2024-04-26T17:26:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20149.851721 secs 2024-04-26T17:26:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20147.002450 secs 2024-04-26T17:26:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20149.851721 secs 2024-04-26T17:26:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20147.002450 secs 2024-04-26T17:26:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20149.851721 secs 2024-04-26T17:26:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:50 smithi194 ceph-mon[33017]: pgmap v11198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:50 smithi033 ceph-mon[27845]: pgmap v11198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:50 smithi096 ceph-mon[32753]: pgmap v11198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:52 smithi194 ceph-mon[33017]: pgmap v11199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:52 smithi033 ceph-mon[27845]: pgmap v11199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:52 smithi096 ceph-mon[32753]: pgmap v11199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20152.002537 secs 2024-04-26T17:26:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:54 smithi194 ceph-mon[33017]: pgmap v11200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20154.851870 secs 2024-04-26T17:26:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20152.002537 secs 2024-04-26T17:26:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:54 smithi033 ceph-mon[27845]: pgmap v11200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20154.851870 secs 2024-04-26T17:26:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20152.002537 secs 2024-04-26T17:26:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:54 smithi096 ceph-mon[32753]: pgmap v11200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20154.851870 secs 2024-04-26T17:26:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:56 smithi194 ceph-mon[33017]: pgmap v11201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:56 smithi033 ceph-mon[27845]: pgmap v11201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:56 smithi096 ceph-mon[32753]: pgmap v11201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:26:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:58 smithi194 ceph-mon[33017]: pgmap v11202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:58 smithi033 ceph-mon[27845]: pgmap v11202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:58 smithi096 ceph-mon[32753]: pgmap v11202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:26:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20157.002623 secs 2024-04-26T17:26:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:26:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20159.851996 secs 2024-04-26T17:26:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20157.002623 secs 2024-04-26T17:26:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:26:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20159.851996 secs 2024-04-26T17:26:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20157.002623 secs 2024-04-26T17:26:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:26:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20159.851996 secs 2024-04-26T17:27:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:00 smithi194 ceph-mon[33017]: pgmap v11203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:00 smithi033 ceph-mon[27845]: pgmap v11203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:00 smithi096 ceph-mon[32753]: pgmap v11203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:02.600+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:02.600+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:27:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:02.600+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:02 smithi194 ceph-mon[33017]: pgmap v11204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:02 smithi033 ceph-mon[27845]: pgmap v11204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:02 smithi096 ceph-mon[32753]: pgmap v11204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20162.002710 secs 2024-04-26T17:27:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:04 smithi194 ceph-mon[33017]: pgmap v11205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20164.852069 secs 2024-04-26T17:27:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20162.002710 secs 2024-04-26T17:27:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:04 smithi033 ceph-mon[27845]: pgmap v11205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20164.852069 secs 2024-04-26T17:27:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20162.002710 secs 2024-04-26T17:27:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:04 smithi096 ceph-mon[32753]: pgmap v11205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20164.852069 secs 2024-04-26T17:27:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:06 smithi194 ceph-mon[33017]: pgmap v11206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:06 smithi033 ceph-mon[27845]: pgmap v11206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:06 smithi096 ceph-mon[32753]: pgmap v11206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:08 smithi194 ceph-mon[33017]: pgmap v11207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:08 smithi033 ceph-mon[27845]: pgmap v11207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:08 smithi096 ceph-mon[32753]: pgmap v11207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20167.002795 secs 2024-04-26T17:27:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20169.852200 secs 2024-04-26T17:27:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20167.002795 secs 2024-04-26T17:27:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20169.852200 secs 2024-04-26T17:27:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20167.002795 secs 2024-04-26T17:27:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20169.852200 secs 2024-04-26T17:27:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:10 smithi194 ceph-mon[33017]: pgmap v11208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:10 smithi033 ceph-mon[27845]: pgmap v11208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:10 smithi096 ceph-mon[32753]: pgmap v11208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:11 smithi194 ceph-mon[33017]: pgmap v11209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:11 smithi033 ceph-mon[27845]: pgmap v11209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:11 smithi096 ceph-mon[32753]: pgmap v11209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20172.002882 secs 2024-04-26T17:27:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:14 smithi194 ceph-mon[33017]: pgmap v11210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20174.852341 secs 2024-04-26T17:27:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20172.002882 secs 2024-04-26T17:27:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:14 smithi033 ceph-mon[27845]: pgmap v11210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20174.852341 secs 2024-04-26T17:27:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20172.002882 secs 2024-04-26T17:27:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:14 smithi096 ceph-mon[32753]: pgmap v11210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20174.852341 secs 2024-04-26T17:27:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:16 smithi194 ceph-mon[33017]: pgmap v11211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:16 smithi033 ceph-mon[27845]: pgmap v11211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:16 smithi096 ceph-mon[32753]: pgmap v11211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:17.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:17.608+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:17.608+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:17.608+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:27:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:18 smithi194 ceph-mon[33017]: pgmap v11212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:18 smithi033 ceph-mon[27845]: pgmap v11212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:18 smithi096 ceph-mon[32753]: pgmap v11212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20177.002995 secs 2024-04-26T17:27:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20179.852409 secs 2024-04-26T17:27:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20177.002995 secs 2024-04-26T17:27:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20179.852409 secs 2024-04-26T17:27:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20177.002995 secs 2024-04-26T17:27:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20179.852409 secs 2024-04-26T17:27:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:20 smithi194 ceph-mon[33017]: pgmap v11213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:20 smithi033 ceph-mon[27845]: pgmap v11213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:20 smithi096 ceph-mon[32753]: pgmap v11213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:22 smithi194 ceph-mon[33017]: pgmap v11214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:22 smithi033 ceph-mon[27845]: pgmap v11214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:22 smithi096 ceph-mon[32753]: pgmap v11214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20182.003081 secs 2024-04-26T17:27:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:24 smithi194 ceph-mon[33017]: pgmap v11215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20184.852534 secs 2024-04-26T17:27:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20182.003081 secs 2024-04-26T17:27:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:24 smithi033 ceph-mon[27845]: pgmap v11215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20184.852534 secs 2024-04-26T17:27:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20182.003081 secs 2024-04-26T17:27:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:24 smithi096 ceph-mon[32753]: pgmap v11215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20184.852534 secs 2024-04-26T17:27:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:26 smithi194 ceph-mon[33017]: pgmap v11216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:26 smithi033 ceph-mon[27845]: pgmap v11216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:26 smithi096 ceph-mon[32753]: pgmap v11216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:28 smithi194 ceph-mon[33017]: pgmap v11217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:28 smithi033 ceph-mon[27845]: pgmap v11217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:28 smithi096 ceph-mon[32753]: pgmap v11217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20187.003139 secs 2024-04-26T17:27:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20189.852671 secs 2024-04-26T17:27:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20187.003139 secs 2024-04-26T17:27:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20189.852671 secs 2024-04-26T17:27:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20187.003139 secs 2024-04-26T17:27:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20189.852671 secs 2024-04-26T17:27:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:30 smithi194 ceph-mon[33017]: pgmap v11218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:30 smithi033 ceph-mon[27845]: pgmap v11218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:30 smithi096 ceph-mon[32753]: pgmap v11218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:32 smithi194 ceph-mon[33017]: pgmap v11219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:32.624+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:32.624+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:32.624+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:27:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:32 smithi033 ceph-mon[27845]: pgmap v11219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:32 smithi096 ceph-mon[32753]: pgmap v11219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20192.003232 secs 2024-04-26T17:27:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:34 smithi194 ceph-mon[33017]: pgmap v11220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20194.852806 secs 2024-04-26T17:27:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20192.003232 secs 2024-04-26T17:27:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:34 smithi033 ceph-mon[27845]: pgmap v11220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20194.852806 secs 2024-04-26T17:27:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20192.003232 secs 2024-04-26T17:27:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:34 smithi096 ceph-mon[32753]: pgmap v11220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20194.852806 secs 2024-04-26T17:27:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:36 smithi194 ceph-mon[33017]: pgmap v11221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:27:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:36 smithi033 ceph-mon[27845]: pgmap v11221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:27:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:36 smithi096 ceph-mon[32753]: pgmap v11221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:27:38.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:38 smithi194 ceph-mon[33017]: pgmap v11222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:38 smithi033 ceph-mon[27845]: pgmap v11222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:38 smithi096 ceph-mon[32753]: pgmap v11222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20197.003320 secs 2024-04-26T17:27:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20199.852901 secs 2024-04-26T17:27:39.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20197.003320 secs 2024-04-26T17:27:39.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20199.852901 secs 2024-04-26T17:27:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20197.003320 secs 2024-04-26T17:27:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20199.852901 secs 2024-04-26T17:27:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:40 smithi194 ceph-mon[33017]: pgmap v11223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:40 smithi033 ceph-mon[27845]: pgmap v11223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:40 smithi096 ceph-mon[32753]: pgmap v11223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:27:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:42 smithi194 ceph-mon[33017]: pgmap v11224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:42 smithi033 ceph-mon[27845]: pgmap v11224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:42 smithi096 ceph-mon[32753]: pgmap v11224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:43 smithi033 ceph-mon[27845]: pgmap v11225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:43 smithi033 ceph-mon[27845]: pgmap v11226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:27:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:27:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:43 smithi096 ceph-mon[32753]: pgmap v11225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:27:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:43 smithi096 ceph-mon[32753]: pgmap v11226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:27:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:27:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:43 smithi194 ceph-mon[33017]: pgmap v11225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:27:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:43 smithi194 ceph-mon[33017]: pgmap v11226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:27:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20202.003404 secs 2024-04-26T17:27:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20204.853043 secs 2024-04-26T17:27:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20202.003404 secs 2024-04-26T17:27:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20204.853043 secs 2024-04-26T17:27:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20202.003404 secs 2024-04-26T17:27:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20204.853043 secs 2024-04-26T17:27:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:45 smithi033 ceph-mon[27845]: pgmap v11227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:45 smithi096 ceph-mon[32753]: pgmap v11227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:45 smithi194 ceph-mon[33017]: pgmap v11227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:47.630+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:47.630+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:27:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:47.630+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:27:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:47 smithi194 ceph-mon[33017]: pgmap v11228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:47 smithi033 ceph-mon[27845]: pgmap v11228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:47 smithi096 ceph-mon[32753]: pgmap v11228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:49.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20207.003490 secs 2024-04-26T17:27:49.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:49 smithi194 ceph-mon[33017]: pgmap v11229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:49.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20209.853137 secs 2024-04-26T17:27:49.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20207.003490 secs 2024-04-26T17:27:49.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:49 smithi096 ceph-mon[32753]: pgmap v11229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:49.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20209.853137 secs 2024-04-26T17:27:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20207.003490 secs 2024-04-26T17:27:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:49 smithi033 ceph-mon[27845]: pgmap v11229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20209.853137 secs 2024-04-26T17:27:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:51 smithi194 ceph-mon[33017]: pgmap v11230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:51 smithi033 ceph-mon[27845]: pgmap v11230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:51 smithi096 ceph-mon[32753]: pgmap v11230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:27:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:53 smithi194 ceph-mon[33017]: pgmap v11231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:27:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:53 smithi033 ceph-mon[27845]: pgmap v11231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:27:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:53 smithi096 ceph-mon[32753]: pgmap v11231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:27:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20212.003573 secs 2024-04-26T17:27:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20214.853253 secs 2024-04-26T17:27:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20212.003573 secs 2024-04-26T17:27:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20214.853253 secs 2024-04-26T17:27:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20212.003573 secs 2024-04-26T17:27:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20214.853253 secs 2024-04-26T17:27:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:55.231+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:27:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:55.231+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:27:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:55.231+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 162 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006630 con 0x564385821900 2024-04-26T17:27:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:27:55.231+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 162 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:27:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:55 smithi194 ceph-mon[33017]: pgmap v11232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:27:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:55 smithi033 ceph-mon[27845]: pgmap v11232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:27:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:55 smithi096 ceph-mon[32753]: pgmap v11232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:27:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:57 smithi194 ceph-mon[33017]: pgmap v11233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:27:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:57 smithi033 ceph-mon[27845]: pgmap v11233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:27:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:57 smithi096 ceph-mon[32753]: pgmap v11233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:27:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20217.003659 secs 2024-04-26T17:27:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:59 smithi194 ceph-mon[33017]: pgmap v11234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:27:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20219.853369 secs 2024-04-26T17:27:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20217.003659 secs 2024-04-26T17:27:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:59 smithi033 ceph-mon[27845]: pgmap v11234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:27:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20219.853369 secs 2024-04-26T17:27:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20217.003659 secs 2024-04-26T17:27:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:59 smithi096 ceph-mon[32753]: pgmap v11234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:27:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20219.853369 secs 2024-04-26T17:28:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:01 smithi194 ceph-mon[33017]: pgmap v11235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:28:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:01 smithi033 ceph-mon[27845]: pgmap v11235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:28:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:01 smithi096 ceph-mon[32753]: pgmap v11235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:28:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:02.642+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:02.642+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:02.642+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:28:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:03 smithi194 ceph-mon[33017]: pgmap v11236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:28:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:03 smithi033 ceph-mon[27845]: pgmap v11236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:28:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:03 smithi096 ceph-mon[32753]: pgmap v11236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:28:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20222.003743 secs 2024-04-26T17:28:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20224.853484 secs 2024-04-26T17:28:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20222.003743 secs 2024-04-26T17:28:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20224.853484 secs 2024-04-26T17:28:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20222.003743 secs 2024-04-26T17:28:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20224.853484 secs 2024-04-26T17:28:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:05 smithi194 ceph-mon[33017]: pgmap v11237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:28:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:05 smithi033 ceph-mon[27845]: pgmap v11237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:28:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:05 smithi096 ceph-mon[32753]: pgmap v11237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:28:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:07 smithi194 ceph-mon[33017]: pgmap v11238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:07.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:07 smithi033 ceph-mon[27845]: pgmap v11238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:07 smithi096 ceph-mon[32753]: pgmap v11238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20227.003830 secs 2024-04-26T17:28:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20229.853599 secs 2024-04-26T17:28:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:09 smithi194 ceph-mon[33017]: pgmap v11239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:09.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20227.003830 secs 2024-04-26T17:28:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20229.853599 secs 2024-04-26T17:28:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:09 smithi033 ceph-mon[27845]: pgmap v11239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20227.003830 secs 2024-04-26T17:28:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20229.853599 secs 2024-04-26T17:28:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:09 smithi096 ceph-mon[32753]: pgmap v11239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:11 smithi194 ceph-mon[33017]: pgmap v11240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:11.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:11 smithi033 ceph-mon[27845]: pgmap v11240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:11 smithi096 ceph-mon[32753]: pgmap v11240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:13 smithi194 ceph-mon[33017]: pgmap v11241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:13 smithi033 ceph-mon[27845]: pgmap v11241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:13 smithi096 ceph-mon[32753]: pgmap v11241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:14.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20232.003914 secs 2024-04-26T17:28:14.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20234.853714 secs 2024-04-26T17:28:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20232.003914 secs 2024-04-26T17:28:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20234.853714 secs 2024-04-26T17:28:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20232.003914 secs 2024-04-26T17:28:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20234.853714 secs 2024-04-26T17:28:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:15 smithi194 ceph-mon[33017]: pgmap v11242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:15 smithi033 ceph-mon[27845]: pgmap v11242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:15 smithi096 ceph-mon[32753]: pgmap v11242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:17.658+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:17.658+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:17.658+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:28:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:17 smithi194 ceph-mon[33017]: pgmap v11243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:17.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:17 smithi033 ceph-mon[27845]: pgmap v11243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:17 smithi096 ceph-mon[32753]: pgmap v11243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20237.004017 secs 2024-04-26T17:28:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20239.853839 secs 2024-04-26T17:28:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:19 smithi194 ceph-mon[33017]: pgmap v11244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:19.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20237.004017 secs 2024-04-26T17:28:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20239.853839 secs 2024-04-26T17:28:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:19 smithi033 ceph-mon[27845]: pgmap v11244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20237.004017 secs 2024-04-26T17:28:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20239.853839 secs 2024-04-26T17:28:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:19 smithi096 ceph-mon[32753]: pgmap v11244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:21 smithi194 ceph-mon[33017]: pgmap v11245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:21 smithi033 ceph-mon[27845]: pgmap v11245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:21 smithi096 ceph-mon[32753]: pgmap v11245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:23 smithi194 ceph-mon[33017]: pgmap v11246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:23.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:23 smithi033 ceph-mon[27845]: pgmap v11246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:23 smithi096 ceph-mon[32753]: pgmap v11246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20242.004108 secs 2024-04-26T17:28:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20244.853977 secs 2024-04-26T17:28:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20242.004108 secs 2024-04-26T17:28:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20244.853977 secs 2024-04-26T17:28:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20242.004108 secs 2024-04-26T17:28:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20244.853977 secs 2024-04-26T17:28:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:25 smithi033 ceph-mon[27845]: pgmap v11247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:25 smithi096 ceph-mon[32753]: pgmap v11247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:25 smithi194 ceph-mon[33017]: pgmap v11247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:27.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:27 smithi033 ceph-mon[27845]: pgmap v11248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:27 smithi096 ceph-mon[32753]: pgmap v11248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:27 smithi194 ceph-mon[33017]: pgmap v11248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20247.004168 secs 2024-04-26T17:28:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20249.854103 secs 2024-04-26T17:28:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:29 smithi033 ceph-mon[27845]: pgmap v11249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20247.004168 secs 2024-04-26T17:28:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20249.854103 secs 2024-04-26T17:28:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:29 smithi096 ceph-mon[32753]: pgmap v11249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20247.004168 secs 2024-04-26T17:28:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20249.854103 secs 2024-04-26T17:28:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:29 smithi194 ceph-mon[33017]: pgmap v11249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:31.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:31 smithi033 ceph-mon[27845]: pgmap v11250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:31 smithi096 ceph-mon[32753]: pgmap v11250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:31 smithi194 ceph-mon[33017]: pgmap v11250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:32.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:32.661+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:32.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:32.661+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:28:32.661 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:32.661+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:33.904 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:33 smithi194 ceph-mon[33017]: pgmap v11251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:33 smithi033 ceph-mon[27845]: pgmap v11251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:33 smithi096 ceph-mon[32753]: pgmap v11251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20252.004227 secs 2024-04-26T17:28:34.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20254.854220 secs 2024-04-26T17:28:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20252.004227 secs 2024-04-26T17:28:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20254.854220 secs 2024-04-26T17:28:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20252.004227 secs 2024-04-26T17:28:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20254.854220 secs 2024-04-26T17:28:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:35 smithi033 ceph-mon[27845]: pgmap v11252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:35 smithi096 ceph-mon[32753]: pgmap v11252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:35 smithi194 ceph-mon[33017]: pgmap v11252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:37 smithi033 ceph-mon[27845]: pgmap v11253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:37 smithi096 ceph-mon[32753]: pgmap v11253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:37 smithi194 ceph-mon[33017]: pgmap v11253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20257.004289 secs 2024-04-26T17:28:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20257.004289 secs 2024-04-26T17:28:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20257.004289 secs 2024-04-26T17:28:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20259.854311 secs 2024-04-26T17:28:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:39 smithi033 ceph-mon[27845]: pgmap v11254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20259.854311 secs 2024-04-26T17:28:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:39 smithi096 ceph-mon[32753]: pgmap v11254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20259.854311 secs 2024-04-26T17:28:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:39 smithi194 ceph-mon[33017]: pgmap v11254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:41 smithi033 ceph-mon[27845]: pgmap v11255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:41 smithi096 ceph-mon[32753]: pgmap v11255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:41 smithi194 ceph-mon[33017]: pgmap v11255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:43 smithi096 ceph-mon[32753]: pgmap v11256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:28:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20262.004377 secs 2024-04-26T17:28:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:43 smithi194 ceph-mon[33017]: pgmap v11256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:28:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20262.004377 secs 2024-04-26T17:28:44.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:43 smithi033 ceph-mon[27845]: pgmap v11256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:28:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20262.004377 secs 2024-04-26T17:28:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20264.854437 secs 2024-04-26T17:28:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:28:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:28:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:44 smithi194 ceph-mon[33017]: pgmap v11257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:44 smithi194 ceph-mon[33017]: pgmap v11258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:28:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20264.854437 secs 2024-04-26T17:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:44 smithi033 ceph-mon[27845]: pgmap v11257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:44 smithi033 ceph-mon[27845]: pgmap v11258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:28:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20264.854437 secs 2024-04-26T17:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:44 smithi096 ceph-mon[32753]: pgmap v11257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:44 smithi096 ceph-mon[32753]: pgmap v11258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:28:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:28:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:47 smithi194 ceph-mon[33017]: pgmap v11259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:47.674+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:47.674+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:28:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:28:47.674+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:28:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:47 smithi033 ceph-mon[27845]: pgmap v11259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:47 smithi096 ceph-mon[32753]: pgmap v11259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20267.004460 secs 2024-04-26T17:28:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:49 smithi194 ceph-mon[33017]: pgmap v11260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20269.854506 secs 2024-04-26T17:28:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20267.004460 secs 2024-04-26T17:28:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:49 smithi033 ceph-mon[27845]: pgmap v11260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20269.854506 secs 2024-04-26T17:28:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20267.004460 secs 2024-04-26T17:28:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:49 smithi096 ceph-mon[32753]: pgmap v11260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20269.854506 secs 2024-04-26T17:28:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:51 smithi194 ceph-mon[33017]: pgmap v11261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:51 smithi033 ceph-mon[27845]: pgmap v11261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:51 smithi096 ceph-mon[32753]: pgmap v11261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:53 smithi194 ceph-mon[33017]: pgmap v11262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:53.609 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:53 smithi033 ceph-mon[27845]: pgmap v11262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:53 smithi096 ceph-mon[32753]: pgmap v11262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:28:54.316 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20272.004517 secs 2024-04-26T17:28:54.316 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20274.854625 secs 2024-04-26T17:28:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20272.004517 secs 2024-04-26T17:28:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20274.854625 secs 2024-04-26T17:28:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20272.004517 secs 2024-04-26T17:28:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20274.854625 secs 2024-04-26T17:28:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:55 smithi194 ceph-mon[33017]: pgmap v11263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:28:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:55 smithi033 ceph-mon[27845]: pgmap v11263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:28:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:55 smithi096 ceph-mon[32753]: pgmap v11263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:28:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:57 smithi194 ceph-mon[33017]: pgmap v11264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:57 smithi033 ceph-mon[27845]: pgmap v11264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:57 smithi096 ceph-mon[32753]: pgmap v11264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:28:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20277.004603 secs 2024-04-26T17:28:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:59 smithi194 ceph-mon[33017]: pgmap v11265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:28:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20279.854762 secs 2024-04-26T17:28:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20277.004603 secs 2024-04-26T17:28:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:59 smithi033 ceph-mon[27845]: pgmap v11265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:28:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20279.854762 secs 2024-04-26T17:28:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20277.004603 secs 2024-04-26T17:28:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:59 smithi096 ceph-mon[32753]: pgmap v11265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:28:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:28:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20279.854762 secs 2024-04-26T17:29:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:01 smithi194 ceph-mon[33017]: pgmap v11266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:01 smithi033 ceph-mon[27845]: pgmap v11266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:01 smithi096 ceph-mon[32753]: pgmap v11266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:02.691+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:02.691+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:02.691+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:29:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:03 smithi194 ceph-mon[33017]: pgmap v11267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:03 smithi033 ceph-mon[27845]: pgmap v11267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:03 smithi096 ceph-mon[32753]: pgmap v11267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20282.004687 secs 2024-04-26T17:29:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20284.854898 secs 2024-04-26T17:29:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20282.004687 secs 2024-04-26T17:29:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20284.854898 secs 2024-04-26T17:29:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20282.004687 secs 2024-04-26T17:29:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20284.854898 secs 2024-04-26T17:29:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:05 smithi194 ceph-mon[33017]: pgmap v11268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:05 smithi033 ceph-mon[27845]: pgmap v11268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:05 smithi096 ceph-mon[32753]: pgmap v11268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:07 smithi194 ceph-mon[33017]: pgmap v11269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:07 smithi033 ceph-mon[27845]: pgmap v11269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:07 smithi096 ceph-mon[32753]: pgmap v11269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20287.004772 secs 2024-04-26T17:29:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:09 smithi194 ceph-mon[33017]: pgmap v11270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20289.855015 secs 2024-04-26T17:29:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20287.004772 secs 2024-04-26T17:29:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:09 smithi033 ceph-mon[27845]: pgmap v11270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20289.855015 secs 2024-04-26T17:29:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20287.004772 secs 2024-04-26T17:29:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:09 smithi096 ceph-mon[32753]: pgmap v11270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20289.855015 secs 2024-04-26T17:29:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:11 smithi194 ceph-mon[33017]: pgmap v11271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:11 smithi033 ceph-mon[27845]: pgmap v11271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:11 smithi096 ceph-mon[32753]: pgmap v11271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:13 smithi194 ceph-mon[33017]: pgmap v11272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:13.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:13 smithi033 ceph-mon[27845]: pgmap v11272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:13 smithi096 ceph-mon[32753]: pgmap v11272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:14.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20292.004856 secs 2024-04-26T17:29:14.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20294.855134 secs 2024-04-26T17:29:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20292.004856 secs 2024-04-26T17:29:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20294.855134 secs 2024-04-26T17:29:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20292.004856 secs 2024-04-26T17:29:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20294.855134 secs 2024-04-26T17:29:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:15 smithi194 ceph-mon[33017]: pgmap v11273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:15 smithi033 ceph-mon[27845]: pgmap v11273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:15 smithi096 ceph-mon[32753]: pgmap v11273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:17 smithi194 ceph-mon[33017]: pgmap v11274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:17 smithi033 ceph-mon[27845]: pgmap v11274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:17 smithi096 ceph-mon[32753]: pgmap v11274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:17.699+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:17.699+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:17.699+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:29:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20297.004914 secs 2024-04-26T17:29:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:19 smithi194 ceph-mon[33017]: pgmap v11275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20299.855253 secs 2024-04-26T17:29:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20297.004914 secs 2024-04-26T17:29:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:19 smithi033 ceph-mon[27845]: pgmap v11275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20299.855253 secs 2024-04-26T17:29:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20297.004914 secs 2024-04-26T17:29:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:19 smithi096 ceph-mon[32753]: pgmap v11275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20299.855253 secs 2024-04-26T17:29:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:20 smithi194 ceph-mon[33017]: pgmap v11276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:20 smithi033 ceph-mon[27845]: pgmap v11276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:20 smithi096 ceph-mon[32753]: pgmap v11276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:23 smithi194 ceph-mon[33017]: pgmap v11277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:23.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:23 smithi033 ceph-mon[27845]: pgmap v11277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:23 smithi096 ceph-mon[32753]: pgmap v11277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20302.004979 secs 2024-04-26T17:29:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20304.855338 secs 2024-04-26T17:29:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20302.004979 secs 2024-04-26T17:29:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20304.855338 secs 2024-04-26T17:29:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20302.004979 secs 2024-04-26T17:29:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20304.855338 secs 2024-04-26T17:29:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:25.247+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:29:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:25.247+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:29:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:25.247+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 163 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:29:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:25.248+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 163 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:29:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:25 smithi194 ceph-mon[33017]: pgmap v11278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:25 smithi033 ceph-mon[27845]: pgmap v11278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:25 smithi096 ceph-mon[32753]: pgmap v11278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:27 smithi194 ceph-mon[33017]: pgmap v11279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:27 smithi033 ceph-mon[27845]: pgmap v11279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:27 smithi096 ceph-mon[32753]: pgmap v11279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20307.005004 secs 2024-04-26T17:29:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:29 smithi194 ceph-mon[33017]: pgmap v11280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20309.855455 secs 2024-04-26T17:29:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20307.005004 secs 2024-04-26T17:29:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:29 smithi033 ceph-mon[27845]: pgmap v11280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20309.855455 secs 2024-04-26T17:29:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20307.005004 secs 2024-04-26T17:29:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:29 smithi096 ceph-mon[32753]: pgmap v11280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20309.855455 secs 2024-04-26T17:29:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:31 smithi194 ceph-mon[33017]: pgmap v11281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:31 smithi033 ceph-mon[27845]: pgmap v11281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:31 smithi096 ceph-mon[32753]: pgmap v11281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:32.707+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:32.707+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:32.707+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:29:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:33 smithi194 ceph-mon[33017]: pgmap v11282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:33 smithi033 ceph-mon[27845]: pgmap v11282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:33 smithi096 ceph-mon[32753]: pgmap v11282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20312.005088 secs 2024-04-26T17:29:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20314.855551 secs 2024-04-26T17:29:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20312.005088 secs 2024-04-26T17:29:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20314.855551 secs 2024-04-26T17:29:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20312.005088 secs 2024-04-26T17:29:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20314.855551 secs 2024-04-26T17:29:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:35 smithi194 ceph-mon[33017]: pgmap v11283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:35 smithi033 ceph-mon[27845]: pgmap v11283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:35 smithi096 ceph-mon[32753]: pgmap v11283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:37 smithi194 ceph-mon[33017]: pgmap v11284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:37 smithi033 ceph-mon[27845]: pgmap v11284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:37 smithi096 ceph-mon[32753]: pgmap v11284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20317.005176 secs 2024-04-26T17:29:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:39 smithi194 ceph-mon[33017]: pgmap v11285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20319.855686 secs 2024-04-26T17:29:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20317.005176 secs 2024-04-26T17:29:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:39 smithi033 ceph-mon[27845]: pgmap v11285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20319.855686 secs 2024-04-26T17:29:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20317.005176 secs 2024-04-26T17:29:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:39 smithi096 ceph-mon[32753]: pgmap v11285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20319.855686 secs 2024-04-26T17:29:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:41 smithi194 ceph-mon[33017]: pgmap v11286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:41 smithi033 ceph-mon[27845]: pgmap v11286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:41 smithi096 ceph-mon[32753]: pgmap v11286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:29:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:43 smithi194 ceph-mon[33017]: pgmap v11287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:43 smithi033 ceph-mon[27845]: pgmap v11287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:43 smithi096 ceph-mon[32753]: pgmap v11287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:44.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20322.005237 secs 2024-04-26T17:29:44.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20324.855808 secs 2024-04-26T17:29:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20322.005237 secs 2024-04-26T17:29:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20324.855808 secs 2024-04-26T17:29:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20322.005237 secs 2024-04-26T17:29:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20324.855808 secs 2024-04-26T17:29:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:45 smithi194 ceph-mon[33017]: pgmap v11288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:29:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:45 smithi033 ceph-mon[27845]: pgmap v11288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:29:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:45 smithi096 ceph-mon[32753]: pgmap v11288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:29:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:29:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:29:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:29:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:29:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:29:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:29:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:29:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:29:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:29:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:47 smithi194 ceph-mon[33017]: pgmap v11289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:29:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:47 smithi194 ceph-mon[33017]: pgmap v11290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:47 smithi033 ceph-mon[27845]: pgmap v11289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:29:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:47 smithi033 ceph-mon[27845]: pgmap v11290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:47 smithi096 ceph-mon[32753]: pgmap v11289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:29:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:47 smithi096 ceph-mon[32753]: pgmap v11290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:47.721+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:47.721+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:29:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:29:47.721+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:29:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20327.005321 secs 2024-04-26T17:29:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:49 smithi194 ceph-mon[33017]: pgmap v11291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20329.855905 secs 2024-04-26T17:29:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20327.005321 secs 2024-04-26T17:29:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:49 smithi033 ceph-mon[27845]: pgmap v11291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20329.855905 secs 2024-04-26T17:29:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20327.005321 secs 2024-04-26T17:29:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:49 smithi096 ceph-mon[32753]: pgmap v11291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20329.855905 secs 2024-04-26T17:29:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:51 smithi194 ceph-mon[33017]: pgmap v11292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:51 smithi033 ceph-mon[27845]: pgmap v11292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:51 smithi096 ceph-mon[32753]: pgmap v11292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:53 smithi194 ceph-mon[33017]: pgmap v11293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:53 smithi033 ceph-mon[27845]: pgmap v11293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:53 smithi096 ceph-mon[32753]: pgmap v11293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:54.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20332.005406 secs 2024-04-26T17:29:54.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20334.856044 secs 2024-04-26T17:29:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20332.005406 secs 2024-04-26T17:29:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20334.856044 secs 2024-04-26T17:29:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20332.005406 secs 2024-04-26T17:29:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20334.856044 secs 2024-04-26T17:29:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:55 smithi194 ceph-mon[33017]: pgmap v11294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:55 smithi033 ceph-mon[27845]: pgmap v11294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:55 smithi096 ceph-mon[32753]: pgmap v11294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:29:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:57 smithi194 ceph-mon[33017]: pgmap v11295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:29:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:57 smithi033 ceph-mon[27845]: pgmap v11295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:29:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:57 smithi096 ceph-mon[32753]: pgmap v11295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:29:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20337.005469 secs 2024-04-26T17:29:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:59 smithi194 ceph-mon[33017]: pgmap v11296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:29:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20339.856187 secs 2024-04-26T17:29:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20337.005469 secs 2024-04-26T17:29:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:59 smithi033 ceph-mon[27845]: pgmap v11296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:29:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20339.856187 secs 2024-04-26T17:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20337.005469 secs 2024-04-26T17:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:59 smithi096 ceph-mon[32753]: pgmap v11296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:29:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:29:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20339.856187 secs 2024-04-26T17:30:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:30:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:30:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:30:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:01 smithi033 ceph-mon[27845]: pgmap v11297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:01 smithi096 ceph-mon[32753]: pgmap v11297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:01 smithi194 ceph-mon[33017]: pgmap v11297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:02 smithi033 ceph-mon[27845]: pgmap v11298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:02 smithi096 ceph-mon[32753]: pgmap v11298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:02.737+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:02.737+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:02.737+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:30:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:02 smithi194 ceph-mon[33017]: pgmap v11298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20342.005557 secs 2024-04-26T17:30:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20342.005557 secs 2024-04-26T17:30:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20342.005557 secs 2024-04-26T17:30:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20344.856296 secs 2024-04-26T17:30:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:04 smithi194 ceph-mon[33017]: pgmap v11299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20344.856296 secs 2024-04-26T17:30:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:04 smithi033 ceph-mon[27845]: pgmap v11299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20344.856296 secs 2024-04-26T17:30:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:04 smithi096 ceph-mon[32753]: pgmap v11299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:06 smithi194 ceph-mon[33017]: pgmap v11300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:06 smithi033 ceph-mon[27845]: pgmap v11300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:06 smithi096 ceph-mon[32753]: pgmap v11300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20347.005642 secs 2024-04-26T17:30:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:08 smithi194 ceph-mon[33017]: pgmap v11301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20347.005642 secs 2024-04-26T17:30:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:08 smithi033 ceph-mon[27845]: pgmap v11301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20347.005642 secs 2024-04-26T17:30:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:08 smithi096 ceph-mon[32753]: pgmap v11301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20349.856412 secs 2024-04-26T17:30:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20349.856412 secs 2024-04-26T17:30:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20349.856412 secs 2024-04-26T17:30:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:10 smithi194 ceph-mon[33017]: pgmap v11302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:10 smithi033 ceph-mon[27845]: pgmap v11302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:10 smithi096 ceph-mon[32753]: pgmap v11302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:12 smithi194 ceph-mon[33017]: pgmap v11303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:12 smithi033 ceph-mon[27845]: pgmap v11303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:12 smithi096 ceph-mon[32753]: pgmap v11303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:14.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20352.005721 secs 2024-04-26T17:30:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20352.005721 secs 2024-04-26T17:30:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20352.005721 secs 2024-04-26T17:30:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20354.856529 secs 2024-04-26T17:30:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:14 smithi194 ceph-mon[33017]: pgmap v11304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20354.856529 secs 2024-04-26T17:30:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:14 smithi033 ceph-mon[27845]: pgmap v11304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20354.856529 secs 2024-04-26T17:30:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:14 smithi096 ceph-mon[32753]: pgmap v11304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:17.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:16 smithi194 ceph-mon[33017]: pgmap v11305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:16 smithi033 ceph-mon[27845]: pgmap v11305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:16 smithi096 ceph-mon[32753]: pgmap v11305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:17.746+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:17.746+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:17.746+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:30:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20357.005805 secs 2024-04-26T17:30:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:18 smithi194 ceph-mon[33017]: pgmap v11306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20357.005805 secs 2024-04-26T17:30:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:18 smithi033 ceph-mon[27845]: pgmap v11306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20357.005805 secs 2024-04-26T17:30:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:18 smithi096 ceph-mon[32753]: pgmap v11306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20359.856640 secs 2024-04-26T17:30:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20359.856640 secs 2024-04-26T17:30:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20359.856640 secs 2024-04-26T17:30:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:20 smithi194 ceph-mon[33017]: pgmap v11307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:20 smithi033 ceph-mon[27845]: pgmap v11307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:20 smithi096 ceph-mon[32753]: pgmap v11307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:23 smithi194 ceph-mon[33017]: pgmap v11308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:23 smithi033 ceph-mon[27845]: pgmap v11308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:23 smithi096 ceph-mon[32753]: pgmap v11308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20362.005891 secs 2024-04-26T17:30:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20362.005891 secs 2024-04-26T17:30:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20362.005891 secs 2024-04-26T17:30:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20364.856761 secs 2024-04-26T17:30:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:25 smithi194 ceph-mon[33017]: pgmap v11309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20364.856761 secs 2024-04-26T17:30:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:25 smithi033 ceph-mon[27845]: pgmap v11309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20364.856761 secs 2024-04-26T17:30:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:25 smithi096 ceph-mon[32753]: pgmap v11309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:27.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:27 smithi194 ceph-mon[33017]: pgmap v11310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:27 smithi033 ceph-mon[27845]: pgmap v11310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:27 smithi096 ceph-mon[32753]: pgmap v11310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20367.005994 secs 2024-04-26T17:30:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:29 smithi194 ceph-mon[33017]: pgmap v11311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20367.005994 secs 2024-04-26T17:30:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:29 smithi033 ceph-mon[27845]: pgmap v11311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20367.005994 secs 2024-04-26T17:30:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:29 smithi096 ceph-mon[32753]: pgmap v11311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20369.856892 secs 2024-04-26T17:30:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20369.856892 secs 2024-04-26T17:30:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20369.856892 secs 2024-04-26T17:30:31.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:31 smithi194 ceph-mon[33017]: pgmap v11312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:31.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:31 smithi033 ceph-mon[27845]: pgmap v11312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:31 smithi096 ceph-mon[32753]: pgmap v11312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:32.758+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:32.758+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:32.758+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:30:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:33 smithi194 ceph-mon[33017]: pgmap v11313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:33.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:33 smithi033 ceph-mon[27845]: pgmap v11313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:33 smithi096 ceph-mon[32753]: pgmap v11313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20372.006084 secs 2024-04-26T17:30:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20372.006084 secs 2024-04-26T17:30:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20372.006084 secs 2024-04-26T17:30:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20374.857023 secs 2024-04-26T17:30:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:35 smithi194 ceph-mon[33017]: pgmap v11314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20374.857023 secs 2024-04-26T17:30:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:35 smithi033 ceph-mon[27845]: pgmap v11314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20374.857023 secs 2024-04-26T17:30:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:35 smithi096 ceph-mon[32753]: pgmap v11314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:37.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:37 smithi194 ceph-mon[33017]: pgmap v11315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:37.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:37 smithi033 ceph-mon[27845]: pgmap v11315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:37 smithi096 ceph-mon[32753]: pgmap v11315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20377.006167 secs 2024-04-26T17:30:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:39 smithi194 ceph-mon[33017]: pgmap v11316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20377.006167 secs 2024-04-26T17:30:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:39 smithi033 ceph-mon[27845]: pgmap v11316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20377.006167 secs 2024-04-26T17:30:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:39 smithi096 ceph-mon[32753]: pgmap v11316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20379.857217 secs 2024-04-26T17:30:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20379.857217 secs 2024-04-26T17:30:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20379.857217 secs 2024-04-26T17:30:41.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:41 smithi033 ceph-mon[27845]: pgmap v11317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:41.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:41 smithi194 ceph-mon[33017]: pgmap v11317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:41 smithi096 ceph-mon[32753]: pgmap v11317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:43 smithi194 ceph-mon[33017]: pgmap v11318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:43 smithi033 ceph-mon[27845]: pgmap v11318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:43 smithi096 ceph-mon[32753]: pgmap v11318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20382.006254 secs 2024-04-26T17:30:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20382.006254 secs 2024-04-26T17:30:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20382.006254 secs 2024-04-26T17:30:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20384.857405 secs 2024-04-26T17:30:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:45 smithi194 ceph-mon[33017]: pgmap v11319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20384.857405 secs 2024-04-26T17:30:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:45 smithi033 ceph-mon[27845]: pgmap v11319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20384.857405 secs 2024-04-26T17:30:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:45 smithi096 ceph-mon[32753]: pgmap v11319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:47 smithi194 ceph-mon[33017]: pgmap v11320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:47.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:30:47.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:47 smithi033 ceph-mon[27845]: pgmap v11320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:30:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:47 smithi096 ceph-mon[32753]: pgmap v11320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:30:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:30:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:47.772+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:47.772+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:30:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:30:47.772+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:30:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:30:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:30:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:30:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:30:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:30:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:30:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:30:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:30:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:49 smithi033 ceph-mon[27845]: pgmap v11321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:49 smithi033 ceph-mon[27845]: pgmap v11322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20387.006344 secs 2024-04-26T17:30:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:49 smithi096 ceph-mon[32753]: pgmap v11321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:49 smithi096 ceph-mon[32753]: pgmap v11322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20387.006344 secs 2024-04-26T17:30:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:49 smithi194 ceph-mon[33017]: pgmap v11321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:30:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:49 smithi194 ceph-mon[33017]: pgmap v11322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20387.006344 secs 2024-04-26T17:30:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20389.857519 secs 2024-04-26T17:30:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20389.857519 secs 2024-04-26T17:30:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20389.857519 secs 2024-04-26T17:30:51.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:51 smithi033 ceph-mon[27845]: pgmap v11323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:51 smithi096 ceph-mon[32753]: pgmap v11323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:51 smithi194 ceph-mon[33017]: pgmap v11323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:53.399 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:53 smithi194 ceph-mon[33017]: pgmap v11324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:53 smithi033 ceph-mon[27845]: pgmap v11324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:53 smithi096 ceph-mon[32753]: pgmap v11324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20392.006428 secs 2024-04-26T17:30:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20392.006428 secs 2024-04-26T17:30:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20392.006428 secs 2024-04-26T17:30:55.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20394.857601 secs 2024-04-26T17:30:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:55 smithi033 ceph-mon[27845]: pgmap v11325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20394.857601 secs 2024-04-26T17:30:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:55 smithi096 ceph-mon[32753]: pgmap v11325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20394.857601 secs 2024-04-26T17:30:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:55 smithi194 ceph-mon[33017]: pgmap v11325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:57.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:57 smithi033 ceph-mon[27845]: pgmap v11326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:57 smithi096 ceph-mon[32753]: pgmap v11326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:57 smithi194 ceph-mon[33017]: pgmap v11326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:59 smithi033 ceph-mon[27845]: pgmap v11327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20397.006511 secs 2024-04-26T17:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:30:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20399.857740 secs 2024-04-26T17:30:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:59 smithi096 ceph-mon[32753]: pgmap v11327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:30:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20397.006511 secs 2024-04-26T17:30:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:30:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20399.857740 secs 2024-04-26T17:30:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:59 smithi194 ceph-mon[33017]: pgmap v11327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:30:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20397.006511 secs 2024-04-26T17:30:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:30:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20399.857740 secs 2024-04-26T17:31:01.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:01 smithi033 ceph-mon[27845]: pgmap v11328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:01 smithi096 ceph-mon[32753]: pgmap v11328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:01 smithi194 ceph-mon[33017]: pgmap v11328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:02.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:02.789+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:02.789+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:31:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:03 smithi033 ceph-mon[27845]: pgmap v11329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:03 smithi096 ceph-mon[32753]: pgmap v11329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:03 smithi194 ceph-mon[33017]: pgmap v11329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20402.006593 secs 2024-04-26T17:31:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20404.857864 secs 2024-04-26T17:31:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20402.006593 secs 2024-04-26T17:31:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20404.857864 secs 2024-04-26T17:31:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20402.006593 secs 2024-04-26T17:31:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20404.857864 secs 2024-04-26T17:31:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:05 smithi194 ceph-mon[33017]: pgmap v11330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:05 smithi033 ceph-mon[27845]: pgmap v11330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:05 smithi096 ceph-mon[32753]: pgmap v11330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:07 smithi194 ceph-mon[33017]: pgmap v11331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:07 smithi033 ceph-mon[27845]: pgmap v11331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:07 smithi096 ceph-mon[32753]: pgmap v11331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:09 smithi194 ceph-mon[33017]: pgmap v11332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20407.006675 secs 2024-04-26T17:31:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20409.857981 secs 2024-04-26T17:31:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:09 smithi033 ceph-mon[27845]: pgmap v11332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20407.006675 secs 2024-04-26T17:31:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20409.857981 secs 2024-04-26T17:31:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:09 smithi096 ceph-mon[32753]: pgmap v11332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20407.006675 secs 2024-04-26T17:31:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20409.857981 secs 2024-04-26T17:31:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:11 smithi194 ceph-mon[33017]: pgmap v11333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:11 smithi033 ceph-mon[27845]: pgmap v11333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:11 smithi096 ceph-mon[32753]: pgmap v11333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:13 smithi033 ceph-mon[27845]: pgmap v11334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:13 smithi194 ceph-mon[33017]: pgmap v11334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:13 smithi096 ceph-mon[32753]: pgmap v11334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:14.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20412.006758 secs 2024-04-26T17:31:14.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20414.858134 secs 2024-04-26T17:31:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20412.006758 secs 2024-04-26T17:31:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20414.858134 secs 2024-04-26T17:31:14.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20412.006758 secs 2024-04-26T17:31:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20414.858134 secs 2024-04-26T17:31:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:15 smithi194 ceph-mon[33017]: pgmap v11335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:15 smithi033 ceph-mon[27845]: pgmap v11335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:15 smithi096 ceph-mon[32753]: pgmap v11335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:17 smithi194 ceph-mon[33017]: pgmap v11336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:17.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:17 smithi033 ceph-mon[27845]: pgmap v11336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:17 smithi096 ceph-mon[32753]: pgmap v11336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:17.802+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:17.802+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:17.802+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:31:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:19 smithi194 ceph-mon[33017]: pgmap v11337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20417.006841 secs 2024-04-26T17:31:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20419.858246 secs 2024-04-26T17:31:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:19 smithi033 ceph-mon[27845]: pgmap v11337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20417.006841 secs 2024-04-26T17:31:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20419.858246 secs 2024-04-26T17:31:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:19 smithi096 ceph-mon[32753]: pgmap v11337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20417.006841 secs 2024-04-26T17:31:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20419.858246 secs 2024-04-26T17:31:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:21 smithi194 ceph-mon[33017]: pgmap v11338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:21 smithi033 ceph-mon[27845]: pgmap v11338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:21 smithi096 ceph-mon[32753]: pgmap v11338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:23 smithi194 ceph-mon[33017]: pgmap v11339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:23 smithi033 ceph-mon[27845]: pgmap v11339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:23 smithi096 ceph-mon[32753]: pgmap v11339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20422.006925 secs 2024-04-26T17:31:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20424.858371 secs 2024-04-26T17:31:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20422.006925 secs 2024-04-26T17:31:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20424.858371 secs 2024-04-26T17:31:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20422.006925 secs 2024-04-26T17:31:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20424.858371 secs 2024-04-26T17:31:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:25 smithi194 ceph-mon[33017]: pgmap v11340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:25 smithi033 ceph-mon[27845]: pgmap v11340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:25 smithi096 ceph-mon[32753]: pgmap v11340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:27 smithi194 ceph-mon[33017]: pgmap v11341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:27 smithi033 ceph-mon[27845]: pgmap v11341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:27 smithi096 ceph-mon[32753]: pgmap v11341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:29 smithi194 ceph-mon[33017]: pgmap v11342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20427.007024 secs 2024-04-26T17:31:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20429.858484 secs 2024-04-26T17:31:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:29 smithi033 ceph-mon[27845]: pgmap v11342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20427.007024 secs 2024-04-26T17:31:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20429.858484 secs 2024-04-26T17:31:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:29 smithi096 ceph-mon[32753]: pgmap v11342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20427.007024 secs 2024-04-26T17:31:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20429.858484 secs 2024-04-26T17:31:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:31 smithi194 ceph-mon[33017]: pgmap v11343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:31 smithi033 ceph-mon[27845]: pgmap v11343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:31 smithi096 ceph-mon[32753]: pgmap v11343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:32.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:32.804+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:32.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:32.804+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:32.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:32.804+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:31:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:33 smithi194 ceph-mon[33017]: pgmap v11344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:33 smithi033 ceph-mon[27845]: pgmap v11344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:33 smithi096 ceph-mon[32753]: pgmap v11344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20432.007112 secs 2024-04-26T17:31:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20434.858609 secs 2024-04-26T17:31:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20432.007112 secs 2024-04-26T17:31:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20434.858609 secs 2024-04-26T17:31:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20432.007112 secs 2024-04-26T17:31:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20434.858609 secs 2024-04-26T17:31:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:35 smithi194 ceph-mon[33017]: pgmap v11345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:35 smithi033 ceph-mon[27845]: pgmap v11345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:35 smithi096 ceph-mon[32753]: pgmap v11345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:37 smithi194 ceph-mon[33017]: pgmap v11346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:37 smithi033 ceph-mon[27845]: pgmap v11346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:37 smithi096 ceph-mon[32753]: pgmap v11346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:39 smithi194 ceph-mon[33017]: pgmap v11347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20437.007198 secs 2024-04-26T17:31:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20439.858752 secs 2024-04-26T17:31:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:39 smithi033 ceph-mon[27845]: pgmap v11347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20437.007198 secs 2024-04-26T17:31:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20439.858752 secs 2024-04-26T17:31:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:39 smithi096 ceph-mon[32753]: pgmap v11347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20437.007198 secs 2024-04-26T17:31:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20439.858752 secs 2024-04-26T17:31:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:41 smithi194 ceph-mon[33017]: pgmap v11348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:41 smithi033 ceph-mon[27845]: pgmap v11348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:41 smithi096 ceph-mon[32753]: pgmap v11348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:43 smithi194 ceph-mon[33017]: pgmap v11349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:43 smithi033 ceph-mon[27845]: pgmap v11349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:43 smithi096 ceph-mon[32753]: pgmap v11349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:44.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20442.007280 secs 2024-04-26T17:31:44.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20444.858892 secs 2024-04-26T17:31:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20442.007280 secs 2024-04-26T17:31:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20444.858892 secs 2024-04-26T17:31:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20442.007280 secs 2024-04-26T17:31:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20444.858892 secs 2024-04-26T17:31:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:45 smithi194 ceph-mon[33017]: pgmap v11350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:45 smithi033 ceph-mon[27845]: pgmap v11350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:45 smithi096 ceph-mon[32753]: pgmap v11350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:47 smithi194 ceph-mon[33017]: pgmap v11351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:47 smithi033 ceph-mon[27845]: pgmap v11351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:47 smithi096 ceph-mon[32753]: pgmap v11351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:31:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:47.818+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:47.818+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:31:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:31:47.818+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:31:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:31:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:31:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:31:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:49 smithi194 ceph-mon[33017]: pgmap v11352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20447.007368 secs 2024-04-26T17:31:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20449.858996 secs 2024-04-26T17:31:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:31:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:31:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:31:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:49 smithi033 ceph-mon[27845]: pgmap v11352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20447.007368 secs 2024-04-26T17:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20449.858996 secs 2024-04-26T17:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:49 smithi096 ceph-mon[32753]: pgmap v11352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20447.007368 secs 2024-04-26T17:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20449.858996 secs 2024-04-26T17:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:31:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:31:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:50 smithi194 ceph-mon[33017]: pgmap v11353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:50 smithi194 ceph-mon[33017]: pgmap v11354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:31:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:50 smithi033 ceph-mon[27845]: pgmap v11353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:50 smithi033 ceph-mon[27845]: pgmap v11354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:31:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:50 smithi096 ceph-mon[32753]: pgmap v11353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:50 smithi096 ceph-mon[32753]: pgmap v11354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:31:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:52 smithi194 ceph-mon[33017]: pgmap v11355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:52 smithi033 ceph-mon[27845]: pgmap v11355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:52 smithi096 ceph-mon[32753]: pgmap v11355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20452.007428 secs 2024-04-26T17:31:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:54 smithi033 ceph-mon[27845]: pgmap v11356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20454.859145 secs 2024-04-26T17:31:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20452.007428 secs 2024-04-26T17:31:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:54 smithi096 ceph-mon[32753]: pgmap v11356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20454.859145 secs 2024-04-26T17:31:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20452.007428 secs 2024-04-26T17:31:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:54 smithi194 ceph-mon[33017]: pgmap v11356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20454.859145 secs 2024-04-26T17:31:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:56 smithi033 ceph-mon[27845]: pgmap v11357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:56 smithi096 ceph-mon[32753]: pgmap v11357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:56 smithi194 ceph-mon[33017]: pgmap v11357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:58 smithi033 ceph-mon[27845]: pgmap v11358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:58 smithi096 ceph-mon[32753]: pgmap v11358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:58 smithi194 ceph-mon[33017]: pgmap v11358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:31:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20457.007527 secs 2024-04-26T17:31:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20459.859275 secs 2024-04-26T17:31:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20457.007527 secs 2024-04-26T17:31:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20459.859275 secs 2024-04-26T17:31:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20457.007527 secs 2024-04-26T17:31:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20459.859275 secs 2024-04-26T17:32:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:00 smithi033 ceph-mon[27845]: pgmap v11359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:32:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:00 smithi096 ceph-mon[32753]: pgmap v11359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:32:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:00 smithi194 ceph-mon[33017]: pgmap v11359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:32:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:02 smithi033 ceph-mon[27845]: pgmap v11360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:02 smithi096 ceph-mon[32753]: pgmap v11360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:02.824+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:02.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:02.824+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:02.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:02.824+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:32:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:02 smithi194 ceph-mon[33017]: pgmap v11360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:04.654 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20462.007625 secs 2024-04-26T17:32:04.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:04 smithi194 ceph-mon[33017]: pgmap v11361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:04.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20464.859410 secs 2024-04-26T17:32:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20462.007625 secs 2024-04-26T17:32:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:04 smithi033 ceph-mon[27845]: pgmap v11361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20464.859410 secs 2024-04-26T17:32:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20462.007625 secs 2024-04-26T17:32:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:04 smithi096 ceph-mon[32753]: pgmap v11361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20464.859410 secs 2024-04-26T17:32:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:05 smithi033 ceph-mon[27845]: pgmap v11362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:05 smithi096 ceph-mon[32753]: pgmap v11362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:05 smithi194 ceph-mon[33017]: pgmap v11362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:08 smithi033 ceph-mon[27845]: pgmap v11363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:08 smithi096 ceph-mon[32753]: pgmap v11363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:08 smithi194 ceph-mon[33017]: pgmap v11363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20467.007683 secs 2024-04-26T17:32:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20469.859531 secs 2024-04-26T17:32:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20467.007683 secs 2024-04-26T17:32:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20469.859531 secs 2024-04-26T17:32:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20467.007683 secs 2024-04-26T17:32:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20469.859531 secs 2024-04-26T17:32:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:10 smithi033 ceph-mon[27845]: pgmap v11364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:10 smithi096 ceph-mon[32753]: pgmap v11364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:10 smithi194 ceph-mon[33017]: pgmap v11364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:11 smithi033 ceph-mon[27845]: pgmap v11365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:11 smithi096 ceph-mon[32753]: pgmap v11365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:11 smithi194 ceph-mon[33017]: pgmap v11365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20472.007773 secs 2024-04-26T17:32:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:14 smithi033 ceph-mon[27845]: pgmap v11366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20474.859651 secs 2024-04-26T17:32:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20472.007773 secs 2024-04-26T17:32:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:14 smithi096 ceph-mon[32753]: pgmap v11366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20474.859651 secs 2024-04-26T17:32:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20472.007773 secs 2024-04-26T17:32:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:14 smithi194 ceph-mon[33017]: pgmap v11366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20474.859651 secs 2024-04-26T17:32:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:15 smithi194 ceph-mon[33017]: pgmap v11367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:15 smithi033 ceph-mon[27845]: pgmap v11367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:15 smithi096 ceph-mon[32753]: pgmap v11367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:17.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:17.828+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:17.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:17.828+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:17.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:17.828+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:32:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:18 smithi033 ceph-mon[27845]: pgmap v11368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:18 smithi096 ceph-mon[32753]: pgmap v11368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:18 smithi194 ceph-mon[33017]: pgmap v11368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20477.007858 secs 2024-04-26T17:32:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20479.859770 secs 2024-04-26T17:32:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20477.007858 secs 2024-04-26T17:32:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20479.859770 secs 2024-04-26T17:32:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20477.007858 secs 2024-04-26T17:32:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20479.859770 secs 2024-04-26T17:32:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:20 smithi033 ceph-mon[27845]: pgmap v11369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:20 smithi096 ceph-mon[32753]: pgmap v11369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:20 smithi194 ceph-mon[33017]: pgmap v11369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:21 smithi033 ceph-mon[27845]: pgmap v11370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:21 smithi096 ceph-mon[32753]: pgmap v11370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:21 smithi194 ceph-mon[33017]: pgmap v11370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20482.007941 secs 2024-04-26T17:32:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:24 smithi033 ceph-mon[27845]: pgmap v11371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20484.859891 secs 2024-04-26T17:32:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20482.007941 secs 2024-04-26T17:32:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:24 smithi096 ceph-mon[32753]: pgmap v11371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20484.859891 secs 2024-04-26T17:32:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20482.007941 secs 2024-04-26T17:32:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:24 smithi194 ceph-mon[33017]: pgmap v11371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20484.859891 secs 2024-04-26T17:32:25.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:25.278+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:32:25.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:25.278+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:32:25.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:25.278+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 164 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:32:25.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:25.278+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 164 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:32:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:25 smithi033 ceph-mon[27845]: pgmap v11372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:25 smithi096 ceph-mon[32753]: pgmap v11372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:25 smithi194 ceph-mon[33017]: pgmap v11372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:28 smithi033 ceph-mon[27845]: pgmap v11373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:28 smithi096 ceph-mon[32753]: pgmap v11373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:28 smithi194 ceph-mon[33017]: pgmap v11373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20487.008024 secs 2024-04-26T17:32:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20489.860010 secs 2024-04-26T17:32:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:29 smithi033 ceph-mon[27845]: pgmap v11374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20487.008024 secs 2024-04-26T17:32:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20489.860010 secs 2024-04-26T17:32:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:29 smithi096 ceph-mon[32753]: pgmap v11374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20487.008024 secs 2024-04-26T17:32:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20489.860010 secs 2024-04-26T17:32:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:29 smithi194 ceph-mon[33017]: pgmap v11374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:31 smithi194 ceph-mon[33017]: pgmap v11375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:31.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:31 smithi033 ceph-mon[27845]: pgmap v11375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:31 smithi096 ceph-mon[32753]: pgmap v11375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:32.840+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:32.840+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:32.840+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:32:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:33 smithi194 ceph-mon[33017]: pgmap v11376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:33 smithi033 ceph-mon[27845]: pgmap v11376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:33 smithi096 ceph-mon[32753]: pgmap v11376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:34.753 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20492.008107 secs 2024-04-26T17:32:34.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20494.860133 secs 2024-04-26T17:32:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20492.008107 secs 2024-04-26T17:32:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20494.860133 secs 2024-04-26T17:32:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20492.008107 secs 2024-04-26T17:32:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20494.860133 secs 2024-04-26T17:32:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:35 smithi194 ceph-mon[33017]: pgmap v11377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:35 smithi033 ceph-mon[27845]: pgmap v11377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:35 smithi096 ceph-mon[32753]: pgmap v11377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:37 smithi194 ceph-mon[33017]: pgmap v11378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:37 smithi033 ceph-mon[27845]: pgmap v11378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:37 smithi096 ceph-mon[32753]: pgmap v11378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20497.008198 secs 2024-04-26T17:32:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20497.008198 secs 2024-04-26T17:32:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20497.008198 secs 2024-04-26T17:32:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20499.860253 secs 2024-04-26T17:32:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:39 smithi194 ceph-mon[33017]: pgmap v11379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:40.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20499.860253 secs 2024-04-26T17:32:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:39 smithi033 ceph-mon[27845]: pgmap v11379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20499.860253 secs 2024-04-26T17:32:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:39 smithi096 ceph-mon[32753]: pgmap v11379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:41 smithi194 ceph-mon[33017]: pgmap v11380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:41 smithi033 ceph-mon[27845]: pgmap v11380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:41 smithi096 ceph-mon[32753]: pgmap v11380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:43 smithi194 ceph-mon[33017]: pgmap v11381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:43 smithi033 ceph-mon[27845]: pgmap v11381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:43 smithi096 ceph-mon[32753]: pgmap v11381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:44.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20502.008281 secs 2024-04-26T17:32:44.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20504.860377 secs 2024-04-26T17:32:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20502.008281 secs 2024-04-26T17:32:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20504.860377 secs 2024-04-26T17:32:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20502.008281 secs 2024-04-26T17:32:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20504.860377 secs 2024-04-26T17:32:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:45 smithi033 ceph-mon[27845]: pgmap v11382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:45 smithi194 ceph-mon[33017]: pgmap v11382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:45 smithi096 ceph-mon[32753]: pgmap v11382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:32:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:47.844+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:47.844+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:32:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:32:47.844+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:32:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:47 smithi194 ceph-mon[33017]: pgmap v11383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:47 smithi033 ceph-mon[27845]: pgmap v11383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:47 smithi096 ceph-mon[32753]: pgmap v11383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:32:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20507.008366 secs 2024-04-26T17:32:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20507.008366 secs 2024-04-26T17:32:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20507.008366 secs 2024-04-26T17:32:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20509.860486 secs 2024-04-26T17:32:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:49 smithi194 ceph-mon[33017]: pgmap v11384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:32:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20509.860486 secs 2024-04-26T17:32:50.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:49 smithi033 ceph-mon[27845]: pgmap v11384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:32:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20509.860486 secs 2024-04-26T17:32:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:49 smithi096 ceph-mon[32753]: pgmap v11384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:32:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:51 smithi194 ceph-mon[33017]: pgmap v11385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:51 smithi033 ceph-mon[27845]: pgmap v11385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:51 smithi096 ceph-mon[32753]: pgmap v11385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: pgmap v11386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: pgmap v11387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20512.008429 secs 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: pgmap v11386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: pgmap v11387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20512.008429 secs 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: pgmap v11386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: pgmap v11387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:32:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20512.008429 secs 2024-04-26T17:32:55.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20514.860626 secs 2024-04-26T17:32:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20514.860626 secs 2024-04-26T17:32:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20514.860626 secs 2024-04-26T17:32:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:55 smithi194 ceph-mon[33017]: pgmap v11388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:55 smithi033 ceph-mon[27845]: pgmap v11388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:55 smithi096 ceph-mon[32753]: pgmap v11388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:57 smithi194 ceph-mon[33017]: pgmap v11389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:57 smithi033 ceph-mon[27845]: pgmap v11389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:57 smithi096 ceph-mon[32753]: pgmap v11389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:32:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20517.008514 secs 2024-04-26T17:32:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20517.008514 secs 2024-04-26T17:32:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20517.008514 secs 2024-04-26T17:33:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20519.860763 secs 2024-04-26T17:33:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:32:59 smithi194 ceph-mon[33017]: pgmap v11390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:33:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20519.860763 secs 2024-04-26T17:33:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:32:59 smithi033 ceph-mon[27845]: pgmap v11390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:33:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20519.860763 secs 2024-04-26T17:33:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:32:59 smithi096 ceph-mon[32753]: pgmap v11390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:33:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:01 smithi194 ceph-mon[33017]: pgmap v11391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:01 smithi033 ceph-mon[27845]: pgmap v11391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:01 smithi096 ceph-mon[32753]: pgmap v11391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:02.860+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:02.860+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:33:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:02.860+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:03 smithi194 ceph-mon[33017]: pgmap v11392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:33:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20522.008596 secs 2024-04-26T17:33:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:03 smithi033 ceph-mon[27845]: pgmap v11392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:33:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20522.008596 secs 2024-04-26T17:33:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:03 smithi096 ceph-mon[32753]: pgmap v11392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:33:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20522.008596 secs 2024-04-26T17:33:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20524.860878 secs 2024-04-26T17:33:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20524.860878 secs 2024-04-26T17:33:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20524.860878 secs 2024-04-26T17:33:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:05 smithi194 ceph-mon[33017]: pgmap v11393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:06.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:05 smithi033 ceph-mon[27845]: pgmap v11393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:05 smithi096 ceph-mon[32753]: pgmap v11393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:07 smithi194 ceph-mon[33017]: pgmap v11394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:08.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:07 smithi033 ceph-mon[27845]: pgmap v11394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:07 smithi096 ceph-mon[32753]: pgmap v11394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20527.008669 secs 2024-04-26T17:33:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20527.008669 secs 2024-04-26T17:33:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20527.008669 secs 2024-04-26T17:33:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20529.861012 secs 2024-04-26T17:33:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:09 smithi033 ceph-mon[27845]: pgmap v11395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20529.861012 secs 2024-04-26T17:33:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:09 smithi096 ceph-mon[32753]: pgmap v11395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20529.861012 secs 2024-04-26T17:33:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:09 smithi194 ceph-mon[33017]: pgmap v11395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:11 smithi033 ceph-mon[27845]: pgmap v11396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:11 smithi096 ceph-mon[32753]: pgmap v11396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:11 smithi194 ceph-mon[33017]: pgmap v11396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:13 smithi033 ceph-mon[27845]: pgmap v11397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:14.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20532.008753 secs 2024-04-26T17:33:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:13 smithi096 ceph-mon[32753]: pgmap v11397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20532.008753 secs 2024-04-26T17:33:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:13 smithi194 ceph-mon[33017]: pgmap v11397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20532.008753 secs 2024-04-26T17:33:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20534.861125 secs 2024-04-26T17:33:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20534.861125 secs 2024-04-26T17:33:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20534.861125 secs 2024-04-26T17:33:16.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:15 smithi033 ceph-mon[27845]: pgmap v11398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:15 smithi096 ceph-mon[32753]: pgmap v11398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:15 smithi194 ceph-mon[33017]: pgmap v11398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:17.875+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:17.875+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:17.875+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:33:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:17 smithi033 ceph-mon[27845]: pgmap v11399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:17 smithi096 ceph-mon[32753]: pgmap v11399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:17 smithi194 ceph-mon[33017]: pgmap v11399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20537.008833 secs 2024-04-26T17:33:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20537.008833 secs 2024-04-26T17:33:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20537.008833 secs 2024-04-26T17:33:20.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20539.861213 secs 2024-04-26T17:33:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:19 smithi033 ceph-mon[27845]: pgmap v11400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20539.861213 secs 2024-04-26T17:33:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:19 smithi096 ceph-mon[32753]: pgmap v11400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20539.861213 secs 2024-04-26T17:33:20.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:19 smithi194 ceph-mon[33017]: pgmap v11400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:21 smithi033 ceph-mon[27845]: pgmap v11401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:21 smithi096 ceph-mon[32753]: pgmap v11401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:21 smithi194 ceph-mon[33017]: pgmap v11401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:23 smithi033 ceph-mon[27845]: pgmap v11402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20542.008902 secs 2024-04-26T17:33:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:23 smithi096 ceph-mon[32753]: pgmap v11402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20542.008902 secs 2024-04-26T17:33:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:23 smithi194 ceph-mon[33017]: pgmap v11402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20542.008902 secs 2024-04-26T17:33:25.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20544.861280 secs 2024-04-26T17:33:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20544.861280 secs 2024-04-26T17:33:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20544.861280 secs 2024-04-26T17:33:26.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:25 smithi033 ceph-mon[27845]: pgmap v11403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:25 smithi096 ceph-mon[32753]: pgmap v11403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:25 smithi194 ceph-mon[33017]: pgmap v11403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:28.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:27 smithi033 ceph-mon[27845]: pgmap v11404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:27 smithi096 ceph-mon[32753]: pgmap v11404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:27 smithi194 ceph-mon[33017]: pgmap v11404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20547.008962 secs 2024-04-26T17:33:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20547.008962 secs 2024-04-26T17:33:29.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20547.008962 secs 2024-04-26T17:33:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20549.861402 secs 2024-04-26T17:33:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:29 smithi096 ceph-mon[32753]: pgmap v11405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20549.861402 secs 2024-04-26T17:33:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:29 smithi194 ceph-mon[33017]: pgmap v11405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20549.861402 secs 2024-04-26T17:33:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:29 smithi033 ceph-mon[27845]: pgmap v11405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:31 smithi096 ceph-mon[32753]: pgmap v11406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:32.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:31 smithi033 ceph-mon[27845]: pgmap v11406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:31 smithi194 ceph-mon[33017]: pgmap v11406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:32.877+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:32.877+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:32.877+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:33:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:33 smithi194 ceph-mon[33017]: pgmap v11407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20552.009045 secs 2024-04-26T17:33:34.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:33 smithi033 ceph-mon[27845]: pgmap v11407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20552.009045 secs 2024-04-26T17:33:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:33 smithi096 ceph-mon[32753]: pgmap v11407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20552.009045 secs 2024-04-26T17:33:35.253 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20554.861519 secs 2024-04-26T17:33:35.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20554.861519 secs 2024-04-26T17:33:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20554.861519 secs 2024-04-26T17:33:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:35 smithi194 ceph-mon[33017]: pgmap v11408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:35 smithi033 ceph-mon[27845]: pgmap v11408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:35 smithi096 ceph-mon[32753]: pgmap v11408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:37 smithi194 ceph-mon[33017]: pgmap v11409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:37 smithi033 ceph-mon[27845]: pgmap v11409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:37 smithi096 ceph-mon[32753]: pgmap v11409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:39.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20557.009135 secs 2024-04-26T17:33:39.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20557.009135 secs 2024-04-26T17:33:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20557.009135 secs 2024-04-26T17:33:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20559.861636 secs 2024-04-26T17:33:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:39 smithi194 ceph-mon[33017]: pgmap v11410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20559.861636 secs 2024-04-26T17:33:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:39 smithi033 ceph-mon[27845]: pgmap v11410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20559.861636 secs 2024-04-26T17:33:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:39 smithi096 ceph-mon[32753]: pgmap v11410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:41 smithi194 ceph-mon[33017]: pgmap v11411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:42.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:41 smithi033 ceph-mon[27845]: pgmap v11411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:41 smithi096 ceph-mon[32753]: pgmap v11411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:43 smithi194 ceph-mon[33017]: pgmap v11412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20562.009224 secs 2024-04-26T17:33:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:43 smithi033 ceph-mon[27845]: pgmap v11412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:44.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20562.009224 secs 2024-04-26T17:33:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:43 smithi096 ceph-mon[32753]: pgmap v11412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20562.009224 secs 2024-04-26T17:33:45.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20564.861746 secs 2024-04-26T17:33:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20564.861746 secs 2024-04-26T17:33:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20564.861746 secs 2024-04-26T17:33:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:45 smithi194 ceph-mon[33017]: pgmap v11413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:45 smithi033 ceph-mon[27845]: pgmap v11413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:45 smithi096 ceph-mon[32753]: pgmap v11413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:47.891+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:47.891+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:33:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:33:47.891+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:33:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:47 smithi194 ceph-mon[33017]: pgmap v11414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:48.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:47 smithi033 ceph-mon[27845]: pgmap v11414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:47 smithi096 ceph-mon[32753]: pgmap v11414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20567.009306 secs 2024-04-26T17:33:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20567.009306 secs 2024-04-26T17:33:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20567.009306 secs 2024-04-26T17:33:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20569.861861 secs 2024-04-26T17:33:50.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:50 smithi194 ceph-mon[33017]: pgmap v11415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:50.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20569.861861 secs 2024-04-26T17:33:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:50 smithi033 ceph-mon[27845]: pgmap v11415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20569.861861 secs 2024-04-26T17:33:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:50 smithi096 ceph-mon[32753]: pgmap v11415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:52 smithi194 ceph-mon[33017]: pgmap v11416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:52.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:52 smithi033 ceph-mon[27845]: pgmap v11416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:52 smithi096 ceph-mon[32753]: pgmap v11416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:33:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:33:53.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:33:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:33:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:54 smithi194 ceph-mon[33017]: pgmap v11417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20572.009389 secs 2024-04-26T17:33:54.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:54 smithi033 ceph-mon[27845]: pgmap v11417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20572.009389 secs 2024-04-26T17:33:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:54 smithi096 ceph-mon[32753]: pgmap v11417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:33:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20572.009389 secs 2024-04-26T17:33:55.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20574.861946 secs 2024-04-26T17:33:55.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:33:55.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:33:55.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:33:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20574.861946 secs 2024-04-26T17:33:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:33:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:33:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:33:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20574.861946 secs 2024-04-26T17:33:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:33:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:33:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:33:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:56 smithi194 ceph-mon[33017]: pgmap v11418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:56 smithi194 ceph-mon[33017]: pgmap v11419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:33:56.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:56 smithi033 ceph-mon[27845]: pgmap v11418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:56 smithi033 ceph-mon[27845]: pgmap v11419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:33:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:56 smithi096 ceph-mon[32753]: pgmap v11418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:56 smithi096 ceph-mon[32753]: pgmap v11419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:33:58.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:58 smithi194 ceph-mon[33017]: pgmap v11420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:58.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:58 smithi033 ceph-mon[27845]: pgmap v11420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:58 smithi096 ceph-mon[32753]: pgmap v11420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:33:59.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:33:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20577.009479 secs 2024-04-26T17:33:59.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:33:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20577.009479 secs 2024-04-26T17:33:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:33:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20577.009479 secs 2024-04-26T17:34:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:00 smithi194 ceph-mon[33017]: pgmap v11421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20579.862067 secs 2024-04-26T17:34:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:00 smithi033 ceph-mon[27845]: pgmap v11421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20579.862067 secs 2024-04-26T17:34:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:00 smithi096 ceph-mon[32753]: pgmap v11421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20579.862067 secs 2024-04-26T17:34:02.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:02 smithi194 ceph-mon[33017]: pgmap v11422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:02.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:02 smithi033 ceph-mon[27845]: pgmap v11422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:02 smithi096 ceph-mon[32753]: pgmap v11422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:02.894+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:02.894+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:02.894+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:34:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:04 smithi194 ceph-mon[33017]: pgmap v11423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20582.009560 secs 2024-04-26T17:34:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:04 smithi033 ceph-mon[27845]: pgmap v11423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20582.009560 secs 2024-04-26T17:34:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:04 smithi096 ceph-mon[32753]: pgmap v11423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20582.009560 secs 2024-04-26T17:34:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20584.862206 secs 2024-04-26T17:34:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20584.862206 secs 2024-04-26T17:34:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20584.862206 secs 2024-04-26T17:34:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:06 smithi194 ceph-mon[33017]: pgmap v11424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:34:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:06 smithi033 ceph-mon[27845]: pgmap v11424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:34:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:06 smithi096 ceph-mon[32753]: pgmap v11424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:34:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:08 smithi194 ceph-mon[33017]: pgmap v11425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:08 smithi033 ceph-mon[27845]: pgmap v11425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:08 smithi096 ceph-mon[32753]: pgmap v11425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20587.009643 secs 2024-04-26T17:34:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20587.009643 secs 2024-04-26T17:34:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20587.009643 secs 2024-04-26T17:34:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20589.862315 secs 2024-04-26T17:34:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:10 smithi194 ceph-mon[33017]: pgmap v11426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20589.862315 secs 2024-04-26T17:34:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:10 smithi033 ceph-mon[27845]: pgmap v11426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20589.862315 secs 2024-04-26T17:34:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:10 smithi096 ceph-mon[32753]: pgmap v11426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:12 smithi033 ceph-mon[27845]: pgmap v11427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:12 smithi096 ceph-mon[32753]: pgmap v11427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:12 smithi194 ceph-mon[33017]: pgmap v11427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:14 smithi033 ceph-mon[27845]: pgmap v11428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20592.009724 secs 2024-04-26T17:34:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:14 smithi096 ceph-mon[32753]: pgmap v11428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20592.009724 secs 2024-04-26T17:34:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:14 smithi194 ceph-mon[33017]: pgmap v11428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20592.009724 secs 2024-04-26T17:34:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20594.862429 secs 2024-04-26T17:34:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20594.862429 secs 2024-04-26T17:34:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20594.862429 secs 2024-04-26T17:34:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:16 smithi033 ceph-mon[27845]: pgmap v11429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:16 smithi096 ceph-mon[32753]: pgmap v11429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:16 smithi194 ceph-mon[33017]: pgmap v11429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:17.899+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:17.899+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:17.899+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:34:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:18 smithi033 ceph-mon[27845]: pgmap v11430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:18 smithi096 ceph-mon[32753]: pgmap v11430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:18 smithi194 ceph-mon[33017]: pgmap v11430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20597.009804 secs 2024-04-26T17:34:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20597.009804 secs 2024-04-26T17:34:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20597.009804 secs 2024-04-26T17:34:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20599.862545 secs 2024-04-26T17:34:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:20 smithi033 ceph-mon[27845]: pgmap v11431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20599.862545 secs 2024-04-26T17:34:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:20 smithi096 ceph-mon[32753]: pgmap v11431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20599.862545 secs 2024-04-26T17:34:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:20 smithi194 ceph-mon[33017]: pgmap v11431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:22.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:22 smithi033 ceph-mon[27845]: pgmap v11432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:22 smithi096 ceph-mon[32753]: pgmap v11432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:22 smithi194 ceph-mon[33017]: pgmap v11432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:24 smithi033 ceph-mon[27845]: pgmap v11433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20602.009880 secs 2024-04-26T17:34:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:24 smithi096 ceph-mon[32753]: pgmap v11433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20602.009880 secs 2024-04-26T17:34:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:24 smithi194 ceph-mon[33017]: pgmap v11433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20602.009880 secs 2024-04-26T17:34:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20604.862609 secs 2024-04-26T17:34:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20604.862609 secs 2024-04-26T17:34:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20604.862609 secs 2024-04-26T17:34:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:26 smithi033 ceph-mon[27845]: pgmap v11434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:26 smithi096 ceph-mon[32753]: pgmap v11434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:26 smithi194 ceph-mon[33017]: pgmap v11434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:34:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:28 smithi033 ceph-mon[27845]: pgmap v11435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:28 smithi096 ceph-mon[32753]: pgmap v11435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:28 smithi194 ceph-mon[33017]: pgmap v11435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20607.009989 secs 2024-04-26T17:34:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20607.009989 secs 2024-04-26T17:34:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20607.009989 secs 2024-04-26T17:34:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20609.862727 secs 2024-04-26T17:34:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:30 smithi194 ceph-mon[33017]: pgmap v11436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20609.862727 secs 2024-04-26T17:34:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:30 smithi033 ceph-mon[27845]: pgmap v11436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20609.862727 secs 2024-04-26T17:34:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:30 smithi096 ceph-mon[32753]: pgmap v11436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:32 smithi194 ceph-mon[33017]: pgmap v11437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:32 smithi033 ceph-mon[27845]: pgmap v11437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:32 smithi096 ceph-mon[32753]: pgmap v11437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:32.908+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:32.908+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:32.909+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:34:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:34 smithi194 ceph-mon[33017]: pgmap v11438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20612.010049 secs 2024-04-26T17:34:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20614.862846 secs 2024-04-26T17:34:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:34 smithi033 ceph-mon[27845]: pgmap v11438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20612.010049 secs 2024-04-26T17:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20614.862846 secs 2024-04-26T17:34:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:34 smithi096 ceph-mon[32753]: pgmap v11438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20612.010049 secs 2024-04-26T17:34:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20614.862846 secs 2024-04-26T17:34:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:36 smithi194 ceph-mon[33017]: pgmap v11439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:36 smithi033 ceph-mon[27845]: pgmap v11439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:36 smithi096 ceph-mon[32753]: pgmap v11439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:38 smithi194 ceph-mon[33017]: pgmap v11440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:38 smithi033 ceph-mon[27845]: pgmap v11440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:38 smithi096 ceph-mon[32753]: pgmap v11440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:34:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20617.010136 secs 2024-04-26T17:34:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20619.862938 secs 2024-04-26T17:34:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20617.010136 secs 2024-04-26T17:34:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20619.862938 secs 2024-04-26T17:34:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20617.010136 secs 2024-04-26T17:34:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20619.862938 secs 2024-04-26T17:34:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:40 smithi194 ceph-mon[33017]: pgmap v11441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:40 smithi033 ceph-mon[27845]: pgmap v11441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:40 smithi096 ceph-mon[32753]: pgmap v11441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:42 smithi194 ceph-mon[33017]: pgmap v11442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:42 smithi033 ceph-mon[27845]: pgmap v11442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:42 smithi096 ceph-mon[32753]: pgmap v11442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:44 smithi194 ceph-mon[33017]: pgmap v11443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20622.010219 secs 2024-04-26T17:34:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20624.863058 secs 2024-04-26T17:34:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:44 smithi033 ceph-mon[27845]: pgmap v11443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20622.010219 secs 2024-04-26T17:34:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20624.863058 secs 2024-04-26T17:34:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:44 smithi096 ceph-mon[32753]: pgmap v11443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20622.010219 secs 2024-04-26T17:34:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20624.863058 secs 2024-04-26T17:34:46.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:46 smithi096 ceph-mon[32753]: pgmap v11444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:46 smithi194 ceph-mon[33017]: pgmap v11444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:46 smithi033 ceph-mon[27845]: pgmap v11444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:34:47.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:47.919+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:47.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:47.919+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:34:47.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:34:47.919+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:34:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:48 smithi194 ceph-mon[33017]: pgmap v11445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:48 smithi033 ceph-mon[27845]: pgmap v11445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:48 smithi096 ceph-mon[32753]: pgmap v11445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20627.010312 secs 2024-04-26T17:34:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20629.863178 secs 2024-04-26T17:34:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20627.010312 secs 2024-04-26T17:34:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20629.863178 secs 2024-04-26T17:34:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20627.010312 secs 2024-04-26T17:34:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20629.863178 secs 2024-04-26T17:34:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:50 smithi194 ceph-mon[33017]: pgmap v11446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:50 smithi033 ceph-mon[27845]: pgmap v11446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:50 smithi096 ceph-mon[32753]: pgmap v11446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:34:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:52 smithi194 ceph-mon[33017]: pgmap v11447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:34:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:52 smithi033 ceph-mon[27845]: pgmap v11447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:34:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:52 smithi096 ceph-mon[32753]: pgmap v11447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:34:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:54 smithi194 ceph-mon[33017]: pgmap v11448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20632.010394 secs 2024-04-26T17:34:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20634.863265 secs 2024-04-26T17:34:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:54 smithi033 ceph-mon[27845]: pgmap v11448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20632.010394 secs 2024-04-26T17:34:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20634.863265 secs 2024-04-26T17:34:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:54 smithi096 ceph-mon[32753]: pgmap v11448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20632.010394 secs 2024-04-26T17:34:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20634.863265 secs 2024-04-26T17:34:55.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:34:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:34:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:34:56.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:56 smithi033 ceph-mon[27845]: pgmap v11449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:34:56.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:56 smithi096 ceph-mon[32753]: pgmap v11449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:34:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:56 smithi194 ceph-mon[33017]: pgmap v11449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:34:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: pgmap v11450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: pgmap v11450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: pgmap v11450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:34:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20637.010487 secs 2024-04-26T17:34:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:34:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:34:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:34:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:34:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20639.863327 secs 2024-04-26T17:34:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:34:59 smithi194 ceph-mon[33017]: pgmap v11451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:59.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20637.010487 secs 2024-04-26T17:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20639.863327 secs 2024-04-26T17:34:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:34:59 smithi033 ceph-mon[27845]: pgmap v11451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:34:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20637.010487 secs 2024-04-26T17:34:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:34:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:34:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:34:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:34:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20639.863327 secs 2024-04-26T17:34:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:34:59 smithi096 ceph-mon[32753]: pgmap v11451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:35:00.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: pgmap v11452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:35:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: pgmap v11453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: pgmap v11454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: pgmap v11452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: pgmap v11453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: pgmap v11454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: pgmap v11452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:35:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: pgmap v11453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: pgmap v11454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:35:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:02.921+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:02.921+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:02.921+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:35:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20642.010568 secs 2024-04-26T17:35:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:04 smithi194 ceph-mon[33017]: pgmap v11455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:35:04.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20642.010568 secs 2024-04-26T17:35:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:04 smithi033 ceph-mon[27845]: pgmap v11455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:35:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20642.010568 secs 2024-04-26T17:35:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:04 smithi096 ceph-mon[32753]: pgmap v11455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:35:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20644.863450 secs 2024-04-26T17:35:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20644.863450 secs 2024-04-26T17:35:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20644.863450 secs 2024-04-26T17:35:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:06 smithi194 ceph-mon[33017]: pgmap v11456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:06.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:06 smithi033 ceph-mon[27845]: pgmap v11456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:06 smithi096 ceph-mon[32753]: pgmap v11456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:08 smithi194 ceph-mon[33017]: pgmap v11457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:08.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:08 smithi033 ceph-mon[27845]: pgmap v11457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:08 smithi096 ceph-mon[32753]: pgmap v11457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:35:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20647.010669 secs 2024-04-26T17:35:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20647.010669 secs 2024-04-26T17:35:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20647.010669 secs 2024-04-26T17:35:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20649.863540 secs 2024-04-26T17:35:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:10 smithi194 ceph-mon[33017]: pgmap v11458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:35:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20649.863540 secs 2024-04-26T17:35:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:10 smithi033 ceph-mon[27845]: pgmap v11458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:35:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20649.863540 secs 2024-04-26T17:35:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:10 smithi096 ceph-mon[32753]: pgmap v11458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:35:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:12 smithi194 ceph-mon[33017]: pgmap v11459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:35:12.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:12 smithi033 ceph-mon[27845]: pgmap v11459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:35:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:12 smithi096 ceph-mon[32753]: pgmap v11459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:35:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20652.010750 secs 2024-04-26T17:35:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:14 smithi194 ceph-mon[33017]: pgmap v11460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20652.010750 secs 2024-04-26T17:35:14.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:14 smithi033 ceph-mon[27845]: pgmap v11460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20652.010750 secs 2024-04-26T17:35:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:14 smithi096 ceph-mon[32753]: pgmap v11460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20654.863680 secs 2024-04-26T17:35:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20654.863680 secs 2024-04-26T17:35:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20654.863680 secs 2024-04-26T17:35:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:16 smithi194 ceph-mon[33017]: pgmap v11461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:16 smithi033 ceph-mon[27845]: pgmap v11461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:16 smithi096 ceph-mon[32753]: pgmap v11461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:17.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:17.935+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:17.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:17.935+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:17.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:17.935+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:35:18.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:18 smithi033 ceph-mon[27845]: pgmap v11462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:18 smithi096 ceph-mon[32753]: pgmap v11462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:18 smithi194 ceph-mon[33017]: pgmap v11462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20657.010835 secs 2024-04-26T17:35:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20657.010835 secs 2024-04-26T17:35:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20657.010835 secs 2024-04-26T17:35:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20659.863756 secs 2024-04-26T17:35:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:20 smithi033 ceph-mon[27845]: pgmap v11463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20659.863756 secs 2024-04-26T17:35:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:20 smithi096 ceph-mon[32753]: pgmap v11463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20659.863756 secs 2024-04-26T17:35:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:20 smithi194 ceph-mon[33017]: pgmap v11463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:22.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:22 smithi033 ceph-mon[27845]: pgmap v11464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:22 smithi096 ceph-mon[32753]: pgmap v11464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:22 smithi194 ceph-mon[33017]: pgmap v11464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20662.010916 secs 2024-04-26T17:35:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:24 smithi033 ceph-mon[27845]: pgmap v11465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20662.010916 secs 2024-04-26T17:35:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:24 smithi096 ceph-mon[32753]: pgmap v11465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20662.010916 secs 2024-04-26T17:35:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:24 smithi194 ceph-mon[33017]: pgmap v11465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:25.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:25.309+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:35:25.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:25.309+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:35:25.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:25.309+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 165 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:35:25.309 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:25.309+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 165 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:35:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20664.863849 secs 2024-04-26T17:35:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20664.863849 secs 2024-04-26T17:35:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20664.863849 secs 2024-04-26T17:35:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:26 smithi033 ceph-mon[27845]: pgmap v11466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:26 smithi096 ceph-mon[32753]: pgmap v11466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:26 smithi194 ceph-mon[33017]: pgmap v11466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:28 smithi033 ceph-mon[27845]: pgmap v11467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:28 smithi096 ceph-mon[32753]: pgmap v11467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:28 smithi194 ceph-mon[33017]: pgmap v11467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:29.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20667.011009 secs 2024-04-26T17:35:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20667.011009 secs 2024-04-26T17:35:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20667.011009 secs 2024-04-26T17:35:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20669.863967 secs 2024-04-26T17:35:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:30 smithi033 ceph-mon[27845]: pgmap v11468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20669.863967 secs 2024-04-26T17:35:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:30 smithi096 ceph-mon[32753]: pgmap v11468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20669.863967 secs 2024-04-26T17:35:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:30 smithi194 ceph-mon[33017]: pgmap v11468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:32 smithi096 ceph-mon[32753]: pgmap v11469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:32 smithi194 ceph-mon[33017]: pgmap v11469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:32 smithi033 ceph-mon[27845]: pgmap v11469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:32.947+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:32.947+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:35:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:32.947+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20672.011096 secs 2024-04-26T17:35:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:34 smithi194 ceph-mon[33017]: pgmap v11470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20672.011096 secs 2024-04-26T17:35:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:34 smithi033 ceph-mon[27845]: pgmap v11470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20672.011096 secs 2024-04-26T17:35:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:34 smithi096 ceph-mon[32753]: pgmap v11470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20674.864089 secs 2024-04-26T17:35:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20674.864089 secs 2024-04-26T17:35:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20674.864089 secs 2024-04-26T17:35:36.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:36 smithi194 ceph-mon[33017]: pgmap v11471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:36 smithi033 ceph-mon[27845]: pgmap v11471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:36 smithi096 ceph-mon[32753]: pgmap v11471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:38 smithi194 ceph-mon[33017]: pgmap v11472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:38 smithi033 ceph-mon[27845]: pgmap v11472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:38 smithi096 ceph-mon[32753]: pgmap v11472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20677.011156 secs 2024-04-26T17:35:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20679.864213 secs 2024-04-26T17:35:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20677.011156 secs 2024-04-26T17:35:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20679.864213 secs 2024-04-26T17:35:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20677.011156 secs 2024-04-26T17:35:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20679.864213 secs 2024-04-26T17:35:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:40 smithi194 ceph-mon[33017]: pgmap v11473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:40 smithi033 ceph-mon[27845]: pgmap v11473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:40 smithi096 ceph-mon[32753]: pgmap v11473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:42 smithi194 ceph-mon[33017]: pgmap v11474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:42 smithi033 ceph-mon[27845]: pgmap v11474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:42 smithi096 ceph-mon[32753]: pgmap v11474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20682.011242 secs 2024-04-26T17:35:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:44 smithi194 ceph-mon[33017]: pgmap v11475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20684.864331 secs 2024-04-26T17:35:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20682.011242 secs 2024-04-26T17:35:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:44 smithi033 ceph-mon[27845]: pgmap v11475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20684.864331 secs 2024-04-26T17:35:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20682.011242 secs 2024-04-26T17:35:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:44 smithi096 ceph-mon[32753]: pgmap v11475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20684.864331 secs 2024-04-26T17:35:46.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:46 smithi096 ceph-mon[32753]: pgmap v11476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:46 smithi194 ceph-mon[33017]: pgmap v11476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:46 smithi033 ceph-mon[27845]: pgmap v11476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:47.952+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:47.952+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:35:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:35:47.952+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:35:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:48 smithi194 ceph-mon[33017]: pgmap v11477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:48 smithi033 ceph-mon[27845]: pgmap v11477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:48 smithi096 ceph-mon[32753]: pgmap v11477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20687.011323 secs 2024-04-26T17:35:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20689.864448 secs 2024-04-26T17:35:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20687.011323 secs 2024-04-26T17:35:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20689.864448 secs 2024-04-26T17:35:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20687.011323 secs 2024-04-26T17:35:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20689.864448 secs 2024-04-26T17:35:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:50 smithi194 ceph-mon[33017]: pgmap v11478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:50 smithi033 ceph-mon[27845]: pgmap v11478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:50 smithi096 ceph-mon[32753]: pgmap v11478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:52 smithi194 ceph-mon[33017]: pgmap v11479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:52.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:52 smithi033 ceph-mon[27845]: pgmap v11479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:52 smithi096 ceph-mon[32753]: pgmap v11479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20692.011408 secs 2024-04-26T17:35:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:54 smithi194 ceph-mon[33017]: pgmap v11480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20694.864569 secs 2024-04-26T17:35:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20692.011408 secs 2024-04-26T17:35:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:54 smithi033 ceph-mon[27845]: pgmap v11480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20694.864569 secs 2024-04-26T17:35:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20692.011408 secs 2024-04-26T17:35:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:54 smithi096 ceph-mon[32753]: pgmap v11480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20694.864569 secs 2024-04-26T17:35:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:56 smithi194 ceph-mon[33017]: pgmap v11481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:56.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:56 smithi033 ceph-mon[27845]: pgmap v11481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:56 smithi096 ceph-mon[32753]: pgmap v11481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:35:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:58 smithi194 ceph-mon[33017]: pgmap v11482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:58.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:58 smithi033 ceph-mon[27845]: pgmap v11482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:58 smithi096 ceph-mon[32753]: pgmap v11482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:35:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20697.011494 secs 2024-04-26T17:35:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20699.864687 secs 2024-04-26T17:35:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20697.011494 secs 2024-04-26T17:35:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20699.864687 secs 2024-04-26T17:35:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20697.011494 secs 2024-04-26T17:35:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20699.864687 secs 2024-04-26T17:36:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:00 smithi194 ceph-mon[33017]: pgmap v11483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:00 smithi033 ceph-mon[27845]: pgmap v11483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:00 smithi096 ceph-mon[32753]: pgmap v11483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:02 smithi194 ceph-mon[33017]: pgmap v11484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:36:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:02 smithi033 ceph-mon[27845]: pgmap v11484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:36:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:02 smithi096 ceph-mon[32753]: pgmap v11484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:36:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:02.967+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:02.967+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:02.967+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:36:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:36:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:36:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:36:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:36:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:36:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:36:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:36:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:36:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:36:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:04 smithi194 ceph-mon[33017]: pgmap v11485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:04 smithi194 ceph-mon[33017]: pgmap v11486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20702.011552 secs 2024-04-26T17:36:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20704.864769 secs 2024-04-26T17:36:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:04 smithi033 ceph-mon[27845]: pgmap v11485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:04 smithi033 ceph-mon[27845]: pgmap v11486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20702.011552 secs 2024-04-26T17:36:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20704.864769 secs 2024-04-26T17:36:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:04 smithi096 ceph-mon[32753]: pgmap v11485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:04 smithi096 ceph-mon[32753]: pgmap v11486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20702.011552 secs 2024-04-26T17:36:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20704.864769 secs 2024-04-26T17:36:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:06 smithi194 ceph-mon[33017]: pgmap v11487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:06.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:06 smithi033 ceph-mon[27845]: pgmap v11487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:06 smithi096 ceph-mon[32753]: pgmap v11487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:08 smithi194 ceph-mon[33017]: pgmap v11488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:08 smithi033 ceph-mon[27845]: pgmap v11488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:08 smithi096 ceph-mon[32753]: pgmap v11488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20707.011640 secs 2024-04-26T17:36:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20709.864908 secs 2024-04-26T17:36:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20707.011640 secs 2024-04-26T17:36:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20709.864908 secs 2024-04-26T17:36:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20707.011640 secs 2024-04-26T17:36:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20709.864908 secs 2024-04-26T17:36:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:10.317+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:36:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:10.317+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:36:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:10.317+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 166 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:36:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:10.318+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 166 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:36:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:10 smithi194 ceph-mon[33017]: pgmap v11489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:10 smithi033 ceph-mon[27845]: pgmap v11489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:10 smithi096 ceph-mon[32753]: pgmap v11489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:12 smithi194 ceph-mon[33017]: pgmap v11490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:12 smithi033 ceph-mon[27845]: pgmap v11490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:12 smithi096 ceph-mon[32753]: pgmap v11490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:36:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:14 smithi194 ceph-mon[33017]: pgmap v11491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:36:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20712.011722 secs 2024-04-26T17:36:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20714.865046 secs 2024-04-26T17:36:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:14 smithi033 ceph-mon[27845]: pgmap v11491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:36:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20712.011722 secs 2024-04-26T17:36:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20714.865046 secs 2024-04-26T17:36:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:14 smithi096 ceph-mon[32753]: pgmap v11491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:36:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20712.011722 secs 2024-04-26T17:36:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20714.865046 secs 2024-04-26T17:36:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:16 smithi194 ceph-mon[33017]: pgmap v11492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:16.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:16 smithi033 ceph-mon[27845]: pgmap v11492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:16 smithi096 ceph-mon[32753]: pgmap v11492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:17.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:17.983+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:17.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:17.984+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:17.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:17.984+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:36:18.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:18 smithi194 ceph-mon[33017]: pgmap v11493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:18 smithi033 ceph-mon[27845]: pgmap v11493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:18 smithi096 ceph-mon[32753]: pgmap v11493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20717.011805 secs 2024-04-26T17:36:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20719.865164 secs 2024-04-26T17:36:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20717.011805 secs 2024-04-26T17:36:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20719.865164 secs 2024-04-26T17:36:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20717.011805 secs 2024-04-26T17:36:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20719.865164 secs 2024-04-26T17:36:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:20 smithi033 ceph-mon[27845]: pgmap v11494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:20 smithi096 ceph-mon[32753]: pgmap v11494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:20 smithi194 ceph-mon[33017]: pgmap v11494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:22 smithi033 ceph-mon[27845]: pgmap v11495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:22 smithi096 ceph-mon[32753]: pgmap v11495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:22 smithi194 ceph-mon[33017]: pgmap v11495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:24 smithi033 ceph-mon[27845]: pgmap v11496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20722.011889 secs 2024-04-26T17:36:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20724.865280 secs 2024-04-26T17:36:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:24 smithi096 ceph-mon[32753]: pgmap v11496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20722.011889 secs 2024-04-26T17:36:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20724.865280 secs 2024-04-26T17:36:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:24 smithi194 ceph-mon[33017]: pgmap v11496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20722.011889 secs 2024-04-26T17:36:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20724.865280 secs 2024-04-26T17:36:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:26 smithi033 ceph-mon[27845]: pgmap v11497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:26 smithi096 ceph-mon[32753]: pgmap v11497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:26 smithi194 ceph-mon[33017]: pgmap v11497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:28 smithi033 ceph-mon[27845]: pgmap v11498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:28 smithi096 ceph-mon[32753]: pgmap v11498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:28 smithi194 ceph-mon[33017]: pgmap v11498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20727.011993 secs 2024-04-26T17:36:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20729.865398 secs 2024-04-26T17:36:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20727.011993 secs 2024-04-26T17:36:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20729.865398 secs 2024-04-26T17:36:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20727.011993 secs 2024-04-26T17:36:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20729.865398 secs 2024-04-26T17:36:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:30 smithi033 ceph-mon[27845]: pgmap v11499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:30 smithi096 ceph-mon[32753]: pgmap v11499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:30 smithi194 ceph-mon[33017]: pgmap v11499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:32 smithi033 ceph-mon[27845]: pgmap v11500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:32 smithi096 ceph-mon[32753]: pgmap v11500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:32 smithi194 ceph-mon[33017]: pgmap v11500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:32.995+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:32.995+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:32.995+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:36:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:34 smithi033 ceph-mon[27845]: pgmap v11501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20732.012052 secs 2024-04-26T17:36:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20734.865514 secs 2024-04-26T17:36:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:34 smithi096 ceph-mon[32753]: pgmap v11501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20732.012052 secs 2024-04-26T17:36:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20734.865514 secs 2024-04-26T17:36:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:34 smithi194 ceph-mon[33017]: pgmap v11501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20732.012052 secs 2024-04-26T17:36:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20734.865514 secs 2024-04-26T17:36:36.654 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:36 smithi194 ceph-mon[33017]: pgmap v11502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:36 smithi033 ceph-mon[27845]: pgmap v11502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:36 smithi096 ceph-mon[32753]: pgmap v11502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:38 smithi033 ceph-mon[27845]: pgmap v11503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:38 smithi096 ceph-mon[32753]: pgmap v11503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:38 smithi194 ceph-mon[33017]: pgmap v11503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:39.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20737.012139 secs 2024-04-26T17:36:39.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20739.865658 secs 2024-04-26T17:36:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20737.012139 secs 2024-04-26T17:36:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20739.865658 secs 2024-04-26T17:36:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20737.012139 secs 2024-04-26T17:36:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20739.865658 secs 2024-04-26T17:36:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:40 smithi033 ceph-mon[27845]: pgmap v11504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:40 smithi096 ceph-mon[32753]: pgmap v11504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:40 smithi194 ceph-mon[33017]: pgmap v11504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:42 smithi096 ceph-mon[32753]: pgmap v11505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:42 smithi194 ceph-mon[33017]: pgmap v11505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:42 smithi033 ceph-mon[27845]: pgmap v11505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:44 smithi194 ceph-mon[33017]: pgmap v11506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20742.012224 secs 2024-04-26T17:36:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20744.865722 secs 2024-04-26T17:36:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:44 smithi033 ceph-mon[27845]: pgmap v11506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20742.012224 secs 2024-04-26T17:36:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20744.865722 secs 2024-04-26T17:36:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:44 smithi096 ceph-mon[32753]: pgmap v11506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20742.012224 secs 2024-04-26T17:36:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20744.865722 secs 2024-04-26T17:36:46.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:46 smithi096 ceph-mon[32753]: pgmap v11507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:46 smithi194 ceph-mon[33017]: pgmap v11507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:46 smithi033 ceph-mon[27845]: pgmap v11507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:47.999+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:47.999+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:36:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:36:47.999+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:36:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:48 smithi194 ceph-mon[33017]: pgmap v11508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:48 smithi033 ceph-mon[27845]: pgmap v11508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:48 smithi096 ceph-mon[32753]: pgmap v11508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20747.012307 secs 2024-04-26T17:36:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20749.865845 secs 2024-04-26T17:36:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20747.012307 secs 2024-04-26T17:36:49.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20749.865845 secs 2024-04-26T17:36:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20747.012307 secs 2024-04-26T17:36:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20749.865845 secs 2024-04-26T17:36:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:50 smithi194 ceph-mon[33017]: pgmap v11509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:50 smithi033 ceph-mon[27845]: pgmap v11509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:50 smithi096 ceph-mon[32753]: pgmap v11509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:51 smithi194 ceph-mon[33017]: pgmap v11510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:51 smithi033 ceph-mon[27845]: pgmap v11510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:51 smithi096 ceph-mon[32753]: pgmap v11510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:53 smithi194 ceph-mon[33017]: pgmap v11511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20752.012391 secs 2024-04-26T17:36:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:53 smithi033 ceph-mon[27845]: pgmap v11511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:54.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20752.012391 secs 2024-04-26T17:36:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:53 smithi096 ceph-mon[32753]: pgmap v11511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20752.012391 secs 2024-04-26T17:36:55.087 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20754.865977 secs 2024-04-26T17:36:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20754.865977 secs 2024-04-26T17:36:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20754.865977 secs 2024-04-26T17:36:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:55 smithi194 ceph-mon[33017]: pgmap v11512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:55 smithi033 ceph-mon[27845]: pgmap v11512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:55 smithi096 ceph-mon[32753]: pgmap v11512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:36:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:57 smithi194 ceph-mon[33017]: pgmap v11513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:57 smithi033 ceph-mon[27845]: pgmap v11513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:57 smithi096 ceph-mon[32753]: pgmap v11513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:36:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20757.012453 secs 2024-04-26T17:36:59.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20757.012453 secs 2024-04-26T17:36:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20757.012453 secs 2024-04-26T17:37:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20759.866113 secs 2024-04-26T17:37:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:36:59 smithi194 ceph-mon[33017]: pgmap v11514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20759.866113 secs 2024-04-26T17:37:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:36:59 smithi033 ceph-mon[27845]: pgmap v11514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20759.866113 secs 2024-04-26T17:37:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:36:59 smithi096 ceph-mon[32753]: pgmap v11514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:01 smithi194 ceph-mon[33017]: pgmap v11515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:02.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:01 smithi033 ceph-mon[27845]: pgmap v11515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:01 smithi096 ceph-mon[32753]: pgmap v11515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:03.014+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:03.014+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:03.014+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:37:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:03 smithi194 ceph-mon[33017]: pgmap v11516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20762.012534 secs 2024-04-26T17:37:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:37:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:03 smithi033 ceph-mon[27845]: pgmap v11516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20762.012534 secs 2024-04-26T17:37:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:37:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:03 smithi096 ceph-mon[32753]: pgmap v11516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20762.012534 secs 2024-04-26T17:37:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:37:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20764.866173 secs 2024-04-26T17:37:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:37:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:37:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:04 smithi194 ceph-mon[33017]: pgmap v11517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:04 smithi194 ceph-mon[33017]: pgmap v11518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:37:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20764.866173 secs 2024-04-26T17:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:04 smithi033 ceph-mon[27845]: pgmap v11517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:04 smithi033 ceph-mon[27845]: pgmap v11518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:37:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20764.866173 secs 2024-04-26T17:37:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:37:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:37:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:04 smithi096 ceph-mon[32753]: pgmap v11517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:04 smithi096 ceph-mon[32753]: pgmap v11518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:37:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:37:07.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:07 smithi033 ceph-mon[27845]: pgmap v11519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:07 smithi096 ceph-mon[32753]: pgmap v11519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:07.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:07 smithi194 ceph-mon[33017]: pgmap v11519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20767.012617 secs 2024-04-26T17:37:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:09 smithi033 ceph-mon[27845]: pgmap v11520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20769.866266 secs 2024-04-26T17:37:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20767.012617 secs 2024-04-26T17:37:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:09 smithi096 ceph-mon[32753]: pgmap v11520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20769.866266 secs 2024-04-26T17:37:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20767.012617 secs 2024-04-26T17:37:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:09 smithi194 ceph-mon[33017]: pgmap v11520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20769.866266 secs 2024-04-26T17:37:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:11 smithi033 ceph-mon[27845]: pgmap v11521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:11 smithi096 ceph-mon[32753]: pgmap v11521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:11 smithi194 ceph-mon[33017]: pgmap v11521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:13 smithi096 ceph-mon[32753]: pgmap v11522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:13 smithi194 ceph-mon[33017]: pgmap v11522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:13.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:13 smithi033 ceph-mon[27845]: pgmap v11522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20772.012701 secs 2024-04-26T17:37:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20774.866333 secs 2024-04-26T17:37:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20772.012701 secs 2024-04-26T17:37:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20774.866333 secs 2024-04-26T17:37:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20772.012701 secs 2024-04-26T17:37:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20774.866333 secs 2024-04-26T17:37:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:15 smithi194 ceph-mon[33017]: pgmap v11523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:37:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:15 smithi033 ceph-mon[27845]: pgmap v11523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:37:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:15 smithi096 ceph-mon[32753]: pgmap v11523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:37:17.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:17 smithi096 ceph-mon[32753]: pgmap v11524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:17 smithi194 ceph-mon[33017]: pgmap v11524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:17 smithi033 ceph-mon[27845]: pgmap v11524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:18.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:18.031+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:18.031+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:37:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20777.012788 secs 2024-04-26T17:37:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:19 smithi194 ceph-mon[33017]: pgmap v11525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20779.866453 secs 2024-04-26T17:37:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20777.012788 secs 2024-04-26T17:37:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:19 smithi033 ceph-mon[27845]: pgmap v11525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20779.866453 secs 2024-04-26T17:37:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20777.012788 secs 2024-04-26T17:37:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:19 smithi096 ceph-mon[32753]: pgmap v11525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20779.866453 secs 2024-04-26T17:37:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:20 smithi194 ceph-mon[33017]: pgmap v11526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:21.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:20 smithi033 ceph-mon[27845]: pgmap v11526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:20 smithi096 ceph-mon[32753]: pgmap v11526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:23 smithi033 ceph-mon[27845]: pgmap v11527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:23 smithi096 ceph-mon[32753]: pgmap v11527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:23 smithi194 ceph-mon[33017]: pgmap v11527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20782.012872 secs 2024-04-26T17:37:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20784.866601 secs 2024-04-26T17:37:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20782.012872 secs 2024-04-26T17:37:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20784.866601 secs 2024-04-26T17:37:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20782.012872 secs 2024-04-26T17:37:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20784.866601 secs 2024-04-26T17:37:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:25 smithi194 ceph-mon[33017]: pgmap v11528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:25 smithi033 ceph-mon[27845]: pgmap v11528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:25 smithi096 ceph-mon[32753]: pgmap v11528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:27 smithi194 ceph-mon[33017]: pgmap v11529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:27 smithi033 ceph-mon[27845]: pgmap v11529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:27 smithi096 ceph-mon[32753]: pgmap v11529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20787.012960 secs 2024-04-26T17:37:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:29 smithi194 ceph-mon[33017]: pgmap v11530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20789.866738 secs 2024-04-26T17:37:29.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20787.012960 secs 2024-04-26T17:37:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:29 smithi033 ceph-mon[27845]: pgmap v11530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20789.866738 secs 2024-04-26T17:37:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20787.012960 secs 2024-04-26T17:37:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:29 smithi096 ceph-mon[32753]: pgmap v11530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20789.866738 secs 2024-04-26T17:37:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:31 smithi194 ceph-mon[33017]: pgmap v11531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:31 smithi033 ceph-mon[27845]: pgmap v11531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:31 smithi096 ceph-mon[32753]: pgmap v11531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:33.038+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:33.038+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:33.038+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:37:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:33 smithi194 ceph-mon[33017]: pgmap v11532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:33 smithi033 ceph-mon[27845]: pgmap v11532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:33 smithi096 ceph-mon[32753]: pgmap v11532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20792.013047 secs 2024-04-26T17:37:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20794.866874 secs 2024-04-26T17:37:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20792.013047 secs 2024-04-26T17:37:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20794.866874 secs 2024-04-26T17:37:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20792.013047 secs 2024-04-26T17:37:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20794.866874 secs 2024-04-26T17:37:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:35 smithi194 ceph-mon[33017]: pgmap v11533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:35 smithi033 ceph-mon[27845]: pgmap v11533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:35 smithi096 ceph-mon[32753]: pgmap v11533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:36 smithi194 ceph-mon[33017]: pgmap v11534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:36 smithi033 ceph-mon[27845]: pgmap v11534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:36 smithi096 ceph-mon[32753]: pgmap v11534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20797.013131 secs 2024-04-26T17:37:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:39 smithi096 ceph-mon[32753]: pgmap v11535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20799.867026 secs 2024-04-26T17:37:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20797.013131 secs 2024-04-26T17:37:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:39 smithi194 ceph-mon[33017]: pgmap v11535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20799.867026 secs 2024-04-26T17:37:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20797.013131 secs 2024-04-26T17:37:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:39 smithi033 ceph-mon[27845]: pgmap v11535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20799.867026 secs 2024-04-26T17:37:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:41 smithi194 ceph-mon[33017]: pgmap v11536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:41 smithi033 ceph-mon[27845]: pgmap v11536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:41 smithi096 ceph-mon[32753]: pgmap v11536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:43 smithi194 ceph-mon[33017]: pgmap v11537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:43 smithi033 ceph-mon[27845]: pgmap v11537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:43 smithi096 ceph-mon[32753]: pgmap v11537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20802.013218 secs 2024-04-26T17:37:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20804.867142 secs 2024-04-26T17:37:44.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20802.013218 secs 2024-04-26T17:37:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20804.867142 secs 2024-04-26T17:37:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20802.013218 secs 2024-04-26T17:37:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20804.867142 secs 2024-04-26T17:37:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:45 smithi194 ceph-mon[33017]: pgmap v11538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:45 smithi033 ceph-mon[27845]: pgmap v11538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:45 smithi096 ceph-mon[32753]: pgmap v11538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:47.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:47 smithi096 ceph-mon[32753]: pgmap v11539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:47 smithi194 ceph-mon[33017]: pgmap v11539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:47 smithi033 ceph-mon[27845]: pgmap v11539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:48.041+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:48.041+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:37:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:37:48.041+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:37:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20807.013306 secs 2024-04-26T17:37:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:49 smithi194 ceph-mon[33017]: pgmap v11540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20809.867265 secs 2024-04-26T17:37:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20807.013306 secs 2024-04-26T17:37:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:49 smithi033 ceph-mon[27845]: pgmap v11540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20809.867265 secs 2024-04-26T17:37:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20807.013306 secs 2024-04-26T17:37:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:49 smithi096 ceph-mon[32753]: pgmap v11540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20809.867265 secs 2024-04-26T17:37:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:50 smithi194 ceph-mon[33017]: pgmap v11541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:51.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:50 smithi033 ceph-mon[27845]: pgmap v11541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:50 smithi096 ceph-mon[32753]: pgmap v11541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:37:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:53 smithi194 ceph-mon[33017]: pgmap v11542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:53 smithi033 ceph-mon[27845]: pgmap v11542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:53 smithi096 ceph-mon[32753]: pgmap v11542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20812.013391 secs 2024-04-26T17:37:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20814.867336 secs 2024-04-26T17:37:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20812.013391 secs 2024-04-26T17:37:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20814.867336 secs 2024-04-26T17:37:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20812.013391 secs 2024-04-26T17:37:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20814.867336 secs 2024-04-26T17:37:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:55 smithi194 ceph-mon[33017]: pgmap v11543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:55 smithi033 ceph-mon[27845]: pgmap v11543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:55 smithi096 ceph-mon[32753]: pgmap v11543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:57.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:57 smithi033 ceph-mon[27845]: pgmap v11544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:57 smithi194 ceph-mon[33017]: pgmap v11544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:57 smithi096 ceph-mon[32753]: pgmap v11544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:37:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20817.013474 secs 2024-04-26T17:37:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:59 smithi194 ceph-mon[33017]: pgmap v11545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20819.867399 secs 2024-04-26T17:37:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20817.013474 secs 2024-04-26T17:37:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:59 smithi033 ceph-mon[27845]: pgmap v11545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20819.867399 secs 2024-04-26T17:37:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20817.013474 secs 2024-04-26T17:37:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:59 smithi096 ceph-mon[32753]: pgmap v11545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:37:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20819.867399 secs 2024-04-26T17:38:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:01 smithi194 ceph-mon[33017]: pgmap v11546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:01.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:01 smithi033 ceph-mon[27845]: pgmap v11546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:01 smithi096 ceph-mon[32753]: pgmap v11546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:03.056+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:03.056+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:03.056+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:38:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:03 smithi194 ceph-mon[33017]: pgmap v11547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:03 smithi033 ceph-mon[27845]: pgmap v11547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:03 smithi096 ceph-mon[32753]: pgmap v11547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20822.013557 secs 2024-04-26T17:38:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20824.867518 secs 2024-04-26T17:38:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:04 smithi194 ceph-mon[33017]: pgmap v11548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20822.013557 secs 2024-04-26T17:38:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20824.867518 secs 2024-04-26T17:38:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:04 smithi033 ceph-mon[27845]: pgmap v11548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20822.013557 secs 2024-04-26T17:38:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20824.867518 secs 2024-04-26T17:38:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:04 smithi096 ceph-mon[32753]: pgmap v11548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:38:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:38:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:38:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:38:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:38:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:06 smithi194 ceph-mon[33017]: pgmap v11549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:06 smithi194 ceph-mon[33017]: pgmap v11550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:38:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:38:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:38:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:06 smithi033 ceph-mon[27845]: pgmap v11549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:06 smithi033 ceph-mon[27845]: pgmap v11550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:38:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:38:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:38:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:06 smithi096 ceph-mon[32753]: pgmap v11549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:06 smithi096 ceph-mon[32753]: pgmap v11550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:38:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20827.013638 secs 2024-04-26T17:38:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:09 smithi194 ceph-mon[33017]: pgmap v11551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20827.013638 secs 2024-04-26T17:38:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:09 smithi033 ceph-mon[27845]: pgmap v11551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20827.013638 secs 2024-04-26T17:38:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:09 smithi096 ceph-mon[32753]: pgmap v11551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20829.867636 secs 2024-04-26T17:38:10.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20829.867636 secs 2024-04-26T17:38:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20829.867636 secs 2024-04-26T17:38:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:11 smithi194 ceph-mon[33017]: pgmap v11552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:11.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:11 smithi033 ceph-mon[27845]: pgmap v11552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:11 smithi096 ceph-mon[32753]: pgmap v11552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:13 smithi194 ceph-mon[33017]: pgmap v11553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:13.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:13 smithi033 ceph-mon[27845]: pgmap v11553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:13 smithi096 ceph-mon[32753]: pgmap v11553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:38:14.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20832.013718 secs 2024-04-26T17:38:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20832.013718 secs 2024-04-26T17:38:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20832.013718 secs 2024-04-26T17:38:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20834.867788 secs 2024-04-26T17:38:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:15 smithi033 ceph-mon[27845]: pgmap v11554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20834.867788 secs 2024-04-26T17:38:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:15 smithi096 ceph-mon[32753]: pgmap v11554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20834.867788 secs 2024-04-26T17:38:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:15 smithi194 ceph-mon[33017]: pgmap v11554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:17 smithi033 ceph-mon[27845]: pgmap v11555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:38:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:17 smithi096 ceph-mon[32753]: pgmap v11555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:38:17.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:17 smithi194 ceph-mon[33017]: pgmap v11555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:18.072+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:18.072+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:18.072+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:19.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20837.013804 secs 2024-04-26T17:38:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:19 smithi033 ceph-mon[27845]: pgmap v11556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20837.013804 secs 2024-04-26T17:38:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:19 smithi096 ceph-mon[32753]: pgmap v11556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20837.013804 secs 2024-04-26T17:38:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:19 smithi194 ceph-mon[33017]: pgmap v11556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20839.867913 secs 2024-04-26T17:38:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20839.867913 secs 2024-04-26T17:38:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20839.867913 secs 2024-04-26T17:38:21.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:21 smithi033 ceph-mon[27845]: pgmap v11557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:21 smithi096 ceph-mon[32753]: pgmap v11557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:21 smithi194 ceph-mon[33017]: pgmap v11557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:23.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:23 smithi033 ceph-mon[27845]: pgmap v11558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:23 smithi096 ceph-mon[32753]: pgmap v11558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:23 smithi194 ceph-mon[33017]: pgmap v11558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20842.013885 secs 2024-04-26T17:38:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20842.013885 secs 2024-04-26T17:38:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20842.013885 secs 2024-04-26T17:38:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:25.341+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006630 con 0x564385821900 2024-04-26T17:38:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:25.341+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:38:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:25.342+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 167 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:38:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:25.342+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 167 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006630 con 0x7fef00077780 2024-04-26T17:38:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20844.868037 secs 2024-04-26T17:38:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:25 smithi033 ceph-mon[27845]: pgmap v11559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20844.868037 secs 2024-04-26T17:38:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:25 smithi096 ceph-mon[32753]: pgmap v11559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20844.868037 secs 2024-04-26T17:38:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:25 smithi194 ceph-mon[33017]: pgmap v11559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:27 smithi033 ceph-mon[27845]: pgmap v11560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:27 smithi096 ceph-mon[32753]: pgmap v11560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:27 smithi194 ceph-mon[33017]: pgmap v11560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20847.013995 secs 2024-04-26T17:38:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:29 smithi033 ceph-mon[27845]: pgmap v11561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20847.013995 secs 2024-04-26T17:38:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:29 smithi096 ceph-mon[32753]: pgmap v11561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20847.013995 secs 2024-04-26T17:38:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:29 smithi194 ceph-mon[33017]: pgmap v11561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20849.868138 secs 2024-04-26T17:38:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20849.868138 secs 2024-04-26T17:38:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20849.868138 secs 2024-04-26T17:38:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:31 smithi194 ceph-mon[33017]: pgmap v11562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:31 smithi033 ceph-mon[27845]: pgmap v11562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:31 smithi096 ceph-mon[32753]: pgmap v11562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:33.082+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:33.082+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:33.082+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:38:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:33 smithi194 ceph-mon[33017]: pgmap v11563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:33 smithi033 ceph-mon[27845]: pgmap v11563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:33 smithi096 ceph-mon[32753]: pgmap v11563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20852.014078 secs 2024-04-26T17:38:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20852.014078 secs 2024-04-26T17:38:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20852.014078 secs 2024-04-26T17:38:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20854.868258 secs 2024-04-26T17:38:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:35 smithi194 ceph-mon[33017]: pgmap v11564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:35.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20854.868258 secs 2024-04-26T17:38:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:35 smithi033 ceph-mon[27845]: pgmap v11564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20854.868258 secs 2024-04-26T17:38:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:35 smithi096 ceph-mon[32753]: pgmap v11564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:37 smithi194 ceph-mon[33017]: pgmap v11565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:37.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:37 smithi033 ceph-mon[27845]: pgmap v11565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:37 smithi096 ceph-mon[32753]: pgmap v11565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20857.014166 secs 2024-04-26T17:38:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:39 smithi194 ceph-mon[33017]: pgmap v11566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20857.014166 secs 2024-04-26T17:38:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:39 smithi033 ceph-mon[27845]: pgmap v11566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20857.014166 secs 2024-04-26T17:38:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:39 smithi096 ceph-mon[32753]: pgmap v11566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20859.868412 secs 2024-04-26T17:38:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20859.868412 secs 2024-04-26T17:38:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20859.868412 secs 2024-04-26T17:38:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:41 smithi194 ceph-mon[33017]: pgmap v11567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:41 smithi033 ceph-mon[27845]: pgmap v11567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:41 smithi096 ceph-mon[32753]: pgmap v11567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:43 smithi194 ceph-mon[33017]: pgmap v11568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:43.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:43 smithi033 ceph-mon[27845]: pgmap v11568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:43 smithi096 ceph-mon[32753]: pgmap v11568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20862.014248 secs 2024-04-26T17:38:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20864.868476 secs 2024-04-26T17:38:44.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20862.014248 secs 2024-04-26T17:38:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20864.868476 secs 2024-04-26T17:38:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20862.014248 secs 2024-04-26T17:38:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20864.868476 secs 2024-04-26T17:38:45.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:45 smithi033 ceph-mon[27845]: pgmap v11569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:45 smithi194 ceph-mon[33017]: pgmap v11569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:45 smithi096 ceph-mon[32753]: pgmap v11569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:38:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:47 smithi194 ceph-mon[33017]: pgmap v11570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:47.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:47 smithi033 ceph-mon[27845]: pgmap v11570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:47 smithi096 ceph-mon[32753]: pgmap v11570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:38:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:48.089+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:48.089+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:38:48.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:38:48.089+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:38:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20867.014331 secs 2024-04-26T17:38:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:49 smithi194 ceph-mon[33017]: pgmap v11571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20869.868593 secs 2024-04-26T17:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20867.014331 secs 2024-04-26T17:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:49 smithi033 ceph-mon[27845]: pgmap v11571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20869.868593 secs 2024-04-26T17:38:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20867.014331 secs 2024-04-26T17:38:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:49 smithi096 ceph-mon[32753]: pgmap v11571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20869.868593 secs 2024-04-26T17:38:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:51 smithi194 ceph-mon[33017]: pgmap v11572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:51 smithi033 ceph-mon[27845]: pgmap v11572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:51 smithi096 ceph-mon[32753]: pgmap v11572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:53 smithi194 ceph-mon[33017]: pgmap v11573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:53 smithi033 ceph-mon[27845]: pgmap v11573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:53 smithi096 ceph-mon[32753]: pgmap v11573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20872.014414 secs 2024-04-26T17:38:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20874.868686 secs 2024-04-26T17:38:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20872.014414 secs 2024-04-26T17:38:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20874.868686 secs 2024-04-26T17:38:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20872.014414 secs 2024-04-26T17:38:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20874.868686 secs 2024-04-26T17:38:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:55 smithi194 ceph-mon[33017]: pgmap v11574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:55.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:55 smithi033 ceph-mon[27845]: pgmap v11574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:55 smithi096 ceph-mon[32753]: pgmap v11574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:57 smithi194 ceph-mon[33017]: pgmap v11575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:57 smithi033 ceph-mon[27845]: pgmap v11575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:57 smithi096 ceph-mon[32753]: pgmap v11575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:38:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20877.014493 secs 2024-04-26T17:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:59 smithi194 ceph-mon[33017]: pgmap v11576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20879.868814 secs 2024-04-26T17:38:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20877.014493 secs 2024-04-26T17:38:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:59 smithi033 ceph-mon[27845]: pgmap v11576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20879.868814 secs 2024-04-26T17:38:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20877.014493 secs 2024-04-26T17:38:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:59 smithi096 ceph-mon[32753]: pgmap v11576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:38:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20879.868814 secs 2024-04-26T17:39:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:01 smithi194 ceph-mon[33017]: pgmap v11577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:01 smithi033 ceph-mon[27845]: pgmap v11577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:01 smithi096 ceph-mon[32753]: pgmap v11577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:03.103+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:03.103+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:39:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:03.103+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:03 smithi194 ceph-mon[33017]: pgmap v11578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:03 smithi033 ceph-mon[27845]: pgmap v11578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:03 smithi096 ceph-mon[32753]: pgmap v11578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20882.014574 secs 2024-04-26T17:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20884.868933 secs 2024-04-26T17:39:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20882.014574 secs 2024-04-26T17:39:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20884.868933 secs 2024-04-26T17:39:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20882.014574 secs 2024-04-26T17:39:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20884.868933 secs 2024-04-26T17:39:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:05 smithi194 ceph-mon[33017]: pgmap v11579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:05 smithi033 ceph-mon[27845]: pgmap v11579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:05 smithi096 ceph-mon[32753]: pgmap v11579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:07 smithi194 ceph-mon[33017]: pgmap v11580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:39:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:07 smithi033 ceph-mon[27845]: pgmap v11580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:39:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:07 smithi096 ceph-mon[32753]: pgmap v11580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:39:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:39:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:39:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:39:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:39:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:39:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:39:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:39:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:39:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:39:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:09 smithi194 ceph-mon[33017]: pgmap v11581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:09 smithi194 ceph-mon[33017]: pgmap v11582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20887.014659 secs 2024-04-26T17:39:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20889.869054 secs 2024-04-26T17:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:09 smithi033 ceph-mon[27845]: pgmap v11581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:09 smithi033 ceph-mon[27845]: pgmap v11582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20887.014659 secs 2024-04-26T17:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20889.869054 secs 2024-04-26T17:39:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:09 smithi096 ceph-mon[32753]: pgmap v11581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:09 smithi096 ceph-mon[32753]: pgmap v11582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20887.014659 secs 2024-04-26T17:39:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20889.869054 secs 2024-04-26T17:39:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:11 smithi194 ceph-mon[33017]: pgmap v11583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:11 smithi033 ceph-mon[27845]: pgmap v11583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:11 smithi096 ceph-mon[32753]: pgmap v11583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:13 smithi033 ceph-mon[27845]: pgmap v11584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:13 smithi096 ceph-mon[32753]: pgmap v11584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:13 smithi194 ceph-mon[33017]: pgmap v11584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20892.014745 secs 2024-04-26T17:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20894.869208 secs 2024-04-26T17:39:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20892.014745 secs 2024-04-26T17:39:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20894.869208 secs 2024-04-26T17:39:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20892.014745 secs 2024-04-26T17:39:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20894.869208 secs 2024-04-26T17:39:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:15 smithi033 ceph-mon[27845]: pgmap v11585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:15 smithi096 ceph-mon[32753]: pgmap v11585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:15 smithi194 ceph-mon[33017]: pgmap v11585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:17 smithi033 ceph-mon[27845]: pgmap v11586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:17 smithi096 ceph-mon[32753]: pgmap v11586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:17 smithi194 ceph-mon[33017]: pgmap v11586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:39:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:18.112+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:18.112+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:18.112+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:19 smithi033 ceph-mon[27845]: pgmap v11587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20897.014832 secs 2024-04-26T17:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20899.869318 secs 2024-04-26T17:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:19 smithi096 ceph-mon[32753]: pgmap v11587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20897.014832 secs 2024-04-26T17:39:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20899.869318 secs 2024-04-26T17:39:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:19 smithi194 ceph-mon[33017]: pgmap v11587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:39:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20897.014832 secs 2024-04-26T17:39:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20899.869318 secs 2024-04-26T17:39:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:21 smithi033 ceph-mon[27845]: pgmap v11588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:21 smithi096 ceph-mon[32753]: pgmap v11588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:21 smithi194 ceph-mon[33017]: pgmap v11588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:23 smithi033 ceph-mon[27845]: pgmap v11589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:23 smithi096 ceph-mon[32753]: pgmap v11589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:23 smithi194 ceph-mon[33017]: pgmap v11589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20902.014913 secs 2024-04-26T17:39:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20904.869402 secs 2024-04-26T17:39:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20902.014913 secs 2024-04-26T17:39:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20904.869402 secs 2024-04-26T17:39:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20902.014913 secs 2024-04-26T17:39:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20904.869402 secs 2024-04-26T17:39:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:25 smithi033 ceph-mon[27845]: pgmap v11590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:25 smithi096 ceph-mon[32753]: pgmap v11590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:25 smithi194 ceph-mon[33017]: pgmap v11590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:27 smithi033 ceph-mon[27845]: pgmap v11591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:27 smithi096 ceph-mon[32753]: pgmap v11591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:27 smithi194 ceph-mon[33017]: pgmap v11591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:29 smithi033 ceph-mon[27845]: pgmap v11592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20907.015026 secs 2024-04-26T17:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20909.869518 secs 2024-04-26T17:39:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:29 smithi096 ceph-mon[32753]: pgmap v11592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20907.015026 secs 2024-04-26T17:39:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20909.869518 secs 2024-04-26T17:39:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:29 smithi194 ceph-mon[33017]: pgmap v11592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20907.015026 secs 2024-04-26T17:39:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20909.869518 secs 2024-04-26T17:39:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:31 smithi033 ceph-mon[27845]: pgmap v11593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:31 smithi096 ceph-mon[32753]: pgmap v11593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:31 smithi194 ceph-mon[33017]: pgmap v11593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:33.127+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:33.128+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:39:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:33.128+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:33.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:33 smithi033 ceph-mon[27845]: pgmap v11594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:33 smithi096 ceph-mon[32753]: pgmap v11594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:33 smithi194 ceph-mon[33017]: pgmap v11594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20912.015086 secs 2024-04-26T17:39:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20914.869656 secs 2024-04-26T17:39:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20912.015086 secs 2024-04-26T17:39:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20914.869656 secs 2024-04-26T17:39:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20912.015086 secs 2024-04-26T17:39:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20914.869656 secs 2024-04-26T17:39:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:35 smithi194 ceph-mon[33017]: pgmap v11595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:35 smithi033 ceph-mon[27845]: pgmap v11595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:35 smithi096 ceph-mon[32753]: pgmap v11595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:37 smithi194 ceph-mon[33017]: pgmap v11596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:37 smithi033 ceph-mon[27845]: pgmap v11596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:37 smithi096 ceph-mon[32753]: pgmap v11596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:39 smithi194 ceph-mon[33017]: pgmap v11597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20917.015175 secs 2024-04-26T17:39:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20919.869824 secs 2024-04-26T17:39:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:39 smithi033 ceph-mon[27845]: pgmap v11597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20917.015175 secs 2024-04-26T17:39:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20919.869824 secs 2024-04-26T17:39:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:39 smithi096 ceph-mon[32753]: pgmap v11597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20917.015175 secs 2024-04-26T17:39:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20919.869824 secs 2024-04-26T17:39:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:41 smithi194 ceph-mon[33017]: pgmap v11598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:41 smithi033 ceph-mon[27845]: pgmap v11598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:41 smithi096 ceph-mon[32753]: pgmap v11598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:43 smithi194 ceph-mon[33017]: pgmap v11599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:43 smithi033 ceph-mon[27845]: pgmap v11599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:43 smithi096 ceph-mon[32753]: pgmap v11599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20922.015258 secs 2024-04-26T17:39:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20924.869940 secs 2024-04-26T17:39:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20922.015258 secs 2024-04-26T17:39:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20924.869940 secs 2024-04-26T17:39:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20922.015258 secs 2024-04-26T17:39:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20924.869940 secs 2024-04-26T17:39:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:45 smithi194 ceph-mon[33017]: pgmap v11600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:45 smithi033 ceph-mon[27845]: pgmap v11600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:45 smithi096 ceph-mon[32753]: pgmap v11600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:47.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:47 smithi096 ceph-mon[32753]: pgmap v11601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:47 smithi194 ceph-mon[33017]: pgmap v11601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:47 smithi033 ceph-mon[27845]: pgmap v11601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:48.143+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:48.143+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:39:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:39:48.143+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:39:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:49 smithi194 ceph-mon[33017]: pgmap v11602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20927.015345 secs 2024-04-26T17:39:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20929.870056 secs 2024-04-26T17:39:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:49 smithi033 ceph-mon[27845]: pgmap v11602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20927.015345 secs 2024-04-26T17:39:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20929.870056 secs 2024-04-26T17:39:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:49 smithi096 ceph-mon[32753]: pgmap v11602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20927.015345 secs 2024-04-26T17:39:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20929.870056 secs 2024-04-26T17:39:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:51 smithi194 ceph-mon[33017]: pgmap v11603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:51 smithi033 ceph-mon[27845]: pgmap v11603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:51 smithi096 ceph-mon[32753]: pgmap v11603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:53 smithi194 ceph-mon[33017]: pgmap v11604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:53 smithi033 ceph-mon[27845]: pgmap v11604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:53 smithi096 ceph-mon[32753]: pgmap v11604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20932.015425 secs 2024-04-26T17:39:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20934.870174 secs 2024-04-26T17:39:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:54 smithi194 ceph-mon[33017]: pgmap v11605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20932.015425 secs 2024-04-26T17:39:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20934.870174 secs 2024-04-26T17:39:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:54 smithi033 ceph-mon[27845]: pgmap v11605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20932.015425 secs 2024-04-26T17:39:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20934.870174 secs 2024-04-26T17:39:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:54 smithi096 ceph-mon[32753]: pgmap v11605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:56 smithi194 ceph-mon[33017]: pgmap v11606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:56 smithi033 ceph-mon[27845]: pgmap v11606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:56 smithi096 ceph-mon[32753]: pgmap v11606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:39:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:58 smithi194 ceph-mon[33017]: pgmap v11607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20937.015509 secs 2024-04-26T17:39:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:58 smithi033 ceph-mon[27845]: pgmap v11607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20937.015509 secs 2024-04-26T17:39:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:58 smithi096 ceph-mon[32753]: pgmap v11607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:39:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20937.015509 secs 2024-04-26T17:40:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20939.870291 secs 2024-04-26T17:40:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20939.870291 secs 2024-04-26T17:40:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20939.870291 secs 2024-04-26T17:40:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:00 smithi194 ceph-mon[33017]: pgmap v11608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:40:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:00 smithi033 ceph-mon[27845]: pgmap v11608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:40:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:00 smithi096 ceph-mon[32753]: pgmap v11608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:40:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:02 smithi194 ceph-mon[33017]: pgmap v11609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:03.151+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:03.151+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:03.151+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:40:03.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:02 smithi033 ceph-mon[27845]: pgmap v11609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:02 smithi096 ceph-mon[32753]: pgmap v11609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20942.015591 secs 2024-04-26T17:40:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20942.015591 secs 2024-04-26T17:40:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20942.015591 secs 2024-04-26T17:40:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20944.870357 secs 2024-04-26T17:40:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:04 smithi194 ceph-mon[33017]: pgmap v11610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:05.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20944.870357 secs 2024-04-26T17:40:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:04 smithi033 ceph-mon[27845]: pgmap v11610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20944.870357 secs 2024-04-26T17:40:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:04 smithi096 ceph-mon[32753]: pgmap v11610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:06 smithi194 ceph-mon[33017]: pgmap v11611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:06 smithi033 ceph-mon[27845]: pgmap v11611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:06 smithi096 ceph-mon[32753]: pgmap v11611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:08 smithi194 ceph-mon[33017]: pgmap v11612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20947.015677 secs 2024-04-26T17:40:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:40:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:08 smithi033 ceph-mon[27845]: pgmap v11612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20947.015677 secs 2024-04-26T17:40:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:40:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:08 smithi096 ceph-mon[32753]: pgmap v11612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20947.015677 secs 2024-04-26T17:40:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:40:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20949.870424 secs 2024-04-26T17:40:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:40:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:40:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:09 smithi194 ceph-mon[33017]: pgmap v11613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:09 smithi194 ceph-mon[33017]: pgmap v11614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:40:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20949.870424 secs 2024-04-26T17:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:09 smithi033 ceph-mon[27845]: pgmap v11613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:09 smithi033 ceph-mon[27845]: pgmap v11614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:40:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20949.870424 secs 2024-04-26T17:40:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:40:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:40:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:09 smithi096 ceph-mon[32753]: pgmap v11613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:09 smithi096 ceph-mon[32753]: pgmap v11614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T17:40:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:40:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:12 smithi194 ceph-mon[33017]: pgmap v11615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:12 smithi033 ceph-mon[27845]: pgmap v11615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:12 smithi096 ceph-mon[32753]: pgmap v11615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20952.015762 secs 2024-04-26T17:40:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:14 smithi194 ceph-mon[33017]: pgmap v11616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20954.870506 secs 2024-04-26T17:40:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20952.015762 secs 2024-04-26T17:40:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:14 smithi033 ceph-mon[27845]: pgmap v11616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20954.870506 secs 2024-04-26T17:40:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20952.015762 secs 2024-04-26T17:40:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:14 smithi096 ceph-mon[32753]: pgmap v11616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20954.870506 secs 2024-04-26T17:40:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:15 smithi194 ceph-mon[33017]: pgmap v11617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:15 smithi033 ceph-mon[27845]: pgmap v11617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:15 smithi096 ceph-mon[32753]: pgmap v11617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:18.167+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:18.167+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:18.167+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:40:18.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:18 smithi033 ceph-mon[27845]: pgmap v11618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:18 smithi194 ceph-mon[33017]: pgmap v11618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:18 smithi096 ceph-mon[32753]: pgmap v11618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20957.015846 secs 2024-04-26T17:40:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20959.870630 secs 2024-04-26T17:40:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20957.015846 secs 2024-04-26T17:40:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20959.870630 secs 2024-04-26T17:40:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20957.015846 secs 2024-04-26T17:40:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20959.870630 secs 2024-04-26T17:40:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:20 smithi194 ceph-mon[33017]: pgmap v11619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:40:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:20 smithi033 ceph-mon[27845]: pgmap v11619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:40:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:20 smithi096 ceph-mon[32753]: pgmap v11619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:40:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:21 smithi194 ceph-mon[33017]: pgmap v11620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:21.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:21 smithi033 ceph-mon[27845]: pgmap v11620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:21 smithi096 ceph-mon[32753]: pgmap v11620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20962.015929 secs 2024-04-26T17:40:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:24 smithi194 ceph-mon[33017]: pgmap v11621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20964.870754 secs 2024-04-26T17:40:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20962.015929 secs 2024-04-26T17:40:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:24 smithi033 ceph-mon[27845]: pgmap v11621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20964.870754 secs 2024-04-26T17:40:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20962.015929 secs 2024-04-26T17:40:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:24 smithi096 ceph-mon[32753]: pgmap v11621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20964.870754 secs 2024-04-26T17:40:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:26 smithi194 ceph-mon[33017]: pgmap v11622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:26 smithi033 ceph-mon[27845]: pgmap v11622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:26 smithi096 ceph-mon[32753]: pgmap v11622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:27.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:27 smithi033 ceph-mon[27845]: pgmap v11623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:27 smithi194 ceph-mon[33017]: pgmap v11623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:27.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:27 smithi096 ceph-mon[32753]: pgmap v11623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20967.016015 secs 2024-04-26T17:40:29.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20967.016015 secs 2024-04-26T17:40:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20967.016015 secs 2024-04-26T17:40:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20969.870878 secs 2024-04-26T17:40:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:29 smithi194 ceph-mon[33017]: pgmap v11624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20969.870878 secs 2024-04-26T17:40:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:29 smithi033 ceph-mon[27845]: pgmap v11624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20969.870878 secs 2024-04-26T17:40:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:29 smithi096 ceph-mon[32753]: pgmap v11624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:32 smithi194 ceph-mon[33017]: pgmap v11625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:32 smithi033 ceph-mon[27845]: pgmap v11625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:32 smithi096 ceph-mon[32753]: pgmap v11625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:33.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:33.182+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:33.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:33.183+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:33.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:33.183+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:40:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:33 smithi194 ceph-mon[33017]: pgmap v11626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:33 smithi033 ceph-mon[27845]: pgmap v11626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:33 smithi096 ceph-mon[32753]: pgmap v11626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20972.016097 secs 2024-04-26T17:40:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20974.871001 secs 2024-04-26T17:40:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20972.016097 secs 2024-04-26T17:40:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20974.871001 secs 2024-04-26T17:40:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20972.016097 secs 2024-04-26T17:40:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20974.871001 secs 2024-04-26T17:40:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:35 smithi194 ceph-mon[33017]: pgmap v11627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:35 smithi033 ceph-mon[27845]: pgmap v11627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:35 smithi096 ceph-mon[32753]: pgmap v11627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:38 smithi194 ceph-mon[33017]: pgmap v11628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:38 smithi033 ceph-mon[27845]: pgmap v11628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:38 smithi096 ceph-mon[32753]: pgmap v11628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20977.016157 secs 2024-04-26T17:40:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20979.871121 secs 2024-04-26T17:40:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:39 smithi194 ceph-mon[33017]: pgmap v11629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20977.016157 secs 2024-04-26T17:40:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20979.871121 secs 2024-04-26T17:40:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:39 smithi033 ceph-mon[27845]: pgmap v11629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20977.016157 secs 2024-04-26T17:40:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20979.871121 secs 2024-04-26T17:40:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:39 smithi096 ceph-mon[32753]: pgmap v11629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:42 smithi194 ceph-mon[33017]: pgmap v11630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:42 smithi033 ceph-mon[27845]: pgmap v11630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:42 smithi096 ceph-mon[32753]: pgmap v11630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:43 smithi194 ceph-mon[33017]: pgmap v11631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:43 smithi033 ceph-mon[27845]: pgmap v11631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:43 smithi096 ceph-mon[32753]: pgmap v11631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20982.016216 secs 2024-04-26T17:40:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20984.871230 secs 2024-04-26T17:40:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20982.016216 secs 2024-04-26T17:40:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20984.871230 secs 2024-04-26T17:40:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20982.016216 secs 2024-04-26T17:40:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20984.871230 secs 2024-04-26T17:40:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:45 smithi194 ceph-mon[33017]: pgmap v11632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:45 smithi033 ceph-mon[27845]: pgmap v11632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:45 smithi096 ceph-mon[32753]: pgmap v11632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:47 smithi194 ceph-mon[33017]: pgmap v11633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:47 smithi033 ceph-mon[27845]: pgmap v11633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:47 smithi096 ceph-mon[32753]: pgmap v11633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:48.198+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:48.198+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:40:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:40:48.198+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:40:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20987.016299 secs 2024-04-26T17:40:49.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20987.016299 secs 2024-04-26T17:40:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20987.016299 secs 2024-04-26T17:40:50.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20989.871378 secs 2024-04-26T17:40:50.096 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:49 smithi033 ceph-mon[27845]: pgmap v11634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20989.871378 secs 2024-04-26T17:40:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:49 smithi194 ceph-mon[33017]: pgmap v11634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20989.871378 secs 2024-04-26T17:40:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:49 smithi096 ceph-mon[32753]: pgmap v11634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:51 smithi194 ceph-mon[33017]: pgmap v11635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:51 smithi033 ceph-mon[27845]: pgmap v11635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:51 smithi096 ceph-mon[32753]: pgmap v11635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:40:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:53 smithi194 ceph-mon[33017]: pgmap v11636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:53 smithi033 ceph-mon[27845]: pgmap v11636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:53 smithi096 ceph-mon[32753]: pgmap v11636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20992.016359 secs 2024-04-26T17:40:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20994.871497 secs 2024-04-26T17:40:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20992.016359 secs 2024-04-26T17:40:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20994.871497 secs 2024-04-26T17:40:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20992.016359 secs 2024-04-26T17:40:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20994.871497 secs 2024-04-26T17:40:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:55 smithi194 ceph-mon[33017]: pgmap v11637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:55 smithi033 ceph-mon[27845]: pgmap v11637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:55 smithi096 ceph-mon[32753]: pgmap v11637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:40:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:57 smithi194 ceph-mon[33017]: pgmap v11638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:57.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:57 smithi033 ceph-mon[27845]: pgmap v11638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:57 smithi096 ceph-mon[32753]: pgmap v11638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:40:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 20997.016422 secs 2024-04-26T17:40:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 20997.016422 secs 2024-04-26T17:40:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 20997.016422 secs 2024-04-26T17:41:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 20999.871615 secs 2024-04-26T17:41:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:40:59 smithi194 ceph-mon[33017]: pgmap v11639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 20999.871615 secs 2024-04-26T17:41:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:40:59 smithi033 ceph-mon[27845]: pgmap v11639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 20999.871615 secs 2024-04-26T17:41:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:40:59 smithi096 ceph-mon[32753]: pgmap v11639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:01 smithi194 ceph-mon[33017]: pgmap v11640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:01 smithi033 ceph-mon[27845]: pgmap v11640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:01 smithi096 ceph-mon[32753]: pgmap v11640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:03.211+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:03.211+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:03.212+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:41:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:03 smithi194 ceph-mon[33017]: pgmap v11641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:03.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:03 smithi033 ceph-mon[27845]: pgmap v11641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:03 smithi096 ceph-mon[32753]: pgmap v11641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21002.016505 secs 2024-04-26T17:41:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21004.871754 secs 2024-04-26T17:41:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21002.016505 secs 2024-04-26T17:41:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21004.871754 secs 2024-04-26T17:41:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21002.016505 secs 2024-04-26T17:41:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21004.871754 secs 2024-04-26T17:41:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:05 smithi194 ceph-mon[33017]: pgmap v11642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:05 smithi033 ceph-mon[27845]: pgmap v11642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:05 smithi096 ceph-mon[32753]: pgmap v11642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:07.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:07 smithi194 ceph-mon[33017]: pgmap v11643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:07 smithi033 ceph-mon[27845]: pgmap v11643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:07 smithi096 ceph-mon[32753]: pgmap v11643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21007.016589 secs 2024-04-26T17:41:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21007.016589 secs 2024-04-26T17:41:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21007.016589 secs 2024-04-26T17:41:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21009.871872 secs 2024-04-26T17:41:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:09 smithi194 ceph-mon[33017]: pgmap v11644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:41:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21009.871872 secs 2024-04-26T17:41:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:09 smithi033 ceph-mon[27845]: pgmap v11644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:41:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21009.871872 secs 2024-04-26T17:41:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:09 smithi096 ceph-mon[32753]: pgmap v11644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:41:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:41:11.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:41:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:41:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:41:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:41:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:41:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:41:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:41:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:41:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:12 smithi033 ceph-mon[27845]: pgmap v11645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:41:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:12 smithi033 ceph-mon[27845]: pgmap v11646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:12 smithi096 ceph-mon[32753]: pgmap v11645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:41:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:12 smithi096 ceph-mon[32753]: pgmap v11646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:12 smithi194 ceph-mon[33017]: pgmap v11645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T17:41:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:12 smithi194 ceph-mon[33017]: pgmap v11646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21012.016673 secs 2024-04-26T17:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:14 smithi033 ceph-mon[27845]: pgmap v11647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21012.016673 secs 2024-04-26T17:41:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:14 smithi096 ceph-mon[32753]: pgmap v11647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21012.016673 secs 2024-04-26T17:41:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:14 smithi194 ceph-mon[33017]: pgmap v11647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:15.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21014.872005 secs 2024-04-26T17:41:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21014.872005 secs 2024-04-26T17:41:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21014.872005 secs 2024-04-26T17:41:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:16 smithi033 ceph-mon[27845]: pgmap v11648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:16 smithi096 ceph-mon[32753]: pgmap v11648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:16 smithi194 ceph-mon[33017]: pgmap v11648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:18.215+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:18.215+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:41:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:18.215+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:18 smithi033 ceph-mon[27845]: pgmap v11649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:18 smithi096 ceph-mon[32753]: pgmap v11649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:18 smithi194 ceph-mon[33017]: pgmap v11649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:41:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21017.016756 secs 2024-04-26T17:41:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21017.016756 secs 2024-04-26T17:41:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21017.016756 secs 2024-04-26T17:41:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21019.872145 secs 2024-04-26T17:41:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:20 smithi033 ceph-mon[27845]: pgmap v11650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:41:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21019.872145 secs 2024-04-26T17:41:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:20 smithi096 ceph-mon[32753]: pgmap v11650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:41:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21019.872145 secs 2024-04-26T17:41:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:20 smithi194 ceph-mon[33017]: pgmap v11650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:41:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T17:41:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T17:41:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0074000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T17:41:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0074000 comp rx=0 tx=0).stop 2024-04-26T17:41:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T17:41:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T17:41:21.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3710 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:41:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.771+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 0x7feee40b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:41:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.772+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 0x7feee40b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:41:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:21.774+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 0x7feee40b31b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0069000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:41:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:22 smithi033 ceph-mon[27845]: pgmap v11651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:41:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:22 smithi096 ceph-mon[32753]: pgmap v11651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:41:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:22 smithi194 ceph-mon[33017]: pgmap v11651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:41:24.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21022.016841 secs 2024-04-26T17:41:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:24 smithi033 ceph-mon[27845]: pgmap v11652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21022.016841 secs 2024-04-26T17:41:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:24 smithi096 ceph-mon[32753]: pgmap v11652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21022.016841 secs 2024-04-26T17:41:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:24 smithi194 ceph-mon[33017]: pgmap v11652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:25.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:25.371+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:41:25.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:25.371+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:41:25.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:25.372+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 168 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:41:25.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:25.372+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 168 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:41:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21024.872260 secs 2024-04-26T17:41:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21024.872260 secs 2024-04-26T17:41:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21024.872260 secs 2024-04-26T17:41:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:26 smithi096 ceph-mon[32753]: pgmap v11653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:26 smithi194 ceph-mon[33017]: pgmap v11653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:26 smithi033 ceph-mon[27845]: pgmap v11653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:28.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:28 smithi033 ceph-mon[27845]: pgmap v11654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:28 smithi194 ceph-mon[33017]: pgmap v11654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:28 smithi096 ceph-mon[32753]: pgmap v11654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21027.016926 secs 2024-04-26T17:41:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21027.016926 secs 2024-04-26T17:41:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21027.016926 secs 2024-04-26T17:41:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21029.872378 secs 2024-04-26T17:41:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:30 smithi194 ceph-mon[33017]: pgmap v11655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21029.872378 secs 2024-04-26T17:41:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:30 smithi033 ceph-mon[27845]: pgmap v11655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21029.872378 secs 2024-04-26T17:41:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:30 smithi096 ceph-mon[32753]: pgmap v11655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:32 smithi194 ceph-mon[33017]: pgmap v11656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:32.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:32 smithi033 ceph-mon[27845]: pgmap v11656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:32 smithi096 ceph-mon[32753]: pgmap v11656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:33.218+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:33.218+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:41:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:33.218+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21032.017009 secs 2024-04-26T17:41:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:34 smithi194 ceph-mon[33017]: pgmap v11657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21032.017009 secs 2024-04-26T17:41:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:34 smithi033 ceph-mon[27845]: pgmap v11657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21032.017009 secs 2024-04-26T17:41:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:34 smithi096 ceph-mon[32753]: pgmap v11657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21034.872495 secs 2024-04-26T17:41:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21034.872495 secs 2024-04-26T17:41:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21034.872495 secs 2024-04-26T17:41:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:36 smithi194 ceph-mon[33017]: pgmap v11658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:36 smithi033 ceph-mon[27845]: pgmap v11658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:36 smithi096 ceph-mon[32753]: pgmap v11658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:38 smithi194 ceph-mon[33017]: pgmap v11659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:38 smithi033 ceph-mon[27845]: pgmap v11659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:38 smithi096 ceph-mon[32753]: pgmap v11659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:39.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21037.017071 secs 2024-04-26T17:41:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21037.017071 secs 2024-04-26T17:41:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21037.017071 secs 2024-04-26T17:41:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21039.872591 secs 2024-04-26T17:41:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:40 smithi194 ceph-mon[33017]: pgmap v11660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21039.872591 secs 2024-04-26T17:41:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:40 smithi033 ceph-mon[27845]: pgmap v11660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21039.872591 secs 2024-04-26T17:41:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:40 smithi096 ceph-mon[32753]: pgmap v11660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:42 smithi194 ceph-mon[33017]: pgmap v11661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:42 smithi033 ceph-mon[27845]: pgmap v11661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:42 smithi096 ceph-mon[32753]: pgmap v11661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21042.017134 secs 2024-04-26T17:41:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:44 smithi194 ceph-mon[33017]: pgmap v11662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21044.872721 secs 2024-04-26T17:41:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21042.017134 secs 2024-04-26T17:41:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:44 smithi033 ceph-mon[27845]: pgmap v11662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21044.872721 secs 2024-04-26T17:41:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21042.017134 secs 2024-04-26T17:41:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:44 smithi096 ceph-mon[32753]: pgmap v11662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21044.872721 secs 2024-04-26T17:41:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:46 smithi194 ceph-mon[33017]: pgmap v11663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:46 smithi033 ceph-mon[27845]: pgmap v11663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:46 smithi096 ceph-mon[32753]: pgmap v11663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:48.233+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:48.233+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:41:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:41:48.233+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:41:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:48 smithi194 ceph-mon[33017]: pgmap v11664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:48 smithi033 ceph-mon[27845]: pgmap v11664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:48 smithi096 ceph-mon[32753]: pgmap v11664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21047.017195 secs 2024-04-26T17:41:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21049.872855 secs 2024-04-26T17:41:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21047.017195 secs 2024-04-26T17:41:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21049.872855 secs 2024-04-26T17:41:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21047.017195 secs 2024-04-26T17:41:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21049.872855 secs 2024-04-26T17:41:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:50 smithi194 ceph-mon[33017]: pgmap v11665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:50 smithi033 ceph-mon[27845]: pgmap v11665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:50 smithi096 ceph-mon[32753]: pgmap v11665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:52 smithi194 ceph-mon[33017]: pgmap v11666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:52 smithi033 ceph-mon[27845]: pgmap v11666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:52 smithi096 ceph-mon[32753]: pgmap v11666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21052.017281 secs 2024-04-26T17:41:54.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:54 smithi194 ceph-mon[33017]: pgmap v11667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21054.872954 secs 2024-04-26T17:41:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21052.017281 secs 2024-04-26T17:41:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:54 smithi033 ceph-mon[27845]: pgmap v11667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21054.872954 secs 2024-04-26T17:41:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21052.017281 secs 2024-04-26T17:41:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:54 smithi096 ceph-mon[32753]: pgmap v11667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21054.872954 secs 2024-04-26T17:41:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:56 smithi194 ceph-mon[33017]: pgmap v11668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:56 smithi033 ceph-mon[27845]: pgmap v11668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:56 smithi096 ceph-mon[32753]: pgmap v11668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:41:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:58 smithi194 ceph-mon[33017]: pgmap v11669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:58 smithi033 ceph-mon[27845]: pgmap v11669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:58 smithi096 ceph-mon[32753]: pgmap v11669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:41:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21057.017344 secs 2024-04-26T17:41:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:41:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21059.873068 secs 2024-04-26T17:41:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21057.017344 secs 2024-04-26T17:41:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:41:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21059.873068 secs 2024-04-26T17:41:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21057.017344 secs 2024-04-26T17:41:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:41:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21059.873068 secs 2024-04-26T17:42:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:00 smithi194 ceph-mon[33017]: pgmap v11670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:00 smithi033 ceph-mon[27845]: pgmap v11670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:00 smithi096 ceph-mon[32753]: pgmap v11670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:02 smithi194 ceph-mon[33017]: pgmap v11671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:02.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:02 smithi033 ceph-mon[27845]: pgmap v11671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:02 smithi096 ceph-mon[32753]: pgmap v11671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:03.241+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:03.241+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:42:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:03.241+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21062.017429 secs 2024-04-26T17:42:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:04 smithi194 ceph-mon[33017]: pgmap v11672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21064.873184 secs 2024-04-26T17:42:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21062.017429 secs 2024-04-26T17:42:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:04 smithi033 ceph-mon[27845]: pgmap v11672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21064.873184 secs 2024-04-26T17:42:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21062.017429 secs 2024-04-26T17:42:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:04 smithi096 ceph-mon[32753]: pgmap v11672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21064.873184 secs 2024-04-26T17:42:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:06 smithi194 ceph-mon[33017]: pgmap v11673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:06 smithi033 ceph-mon[27845]: pgmap v11673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:06 smithi096 ceph-mon[32753]: pgmap v11673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:08 smithi194 ceph-mon[33017]: pgmap v11674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:08 smithi033 ceph-mon[27845]: pgmap v11674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:08 smithi096 ceph-mon[32753]: pgmap v11674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21067.017513 secs 2024-04-26T17:42:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21069.873263 secs 2024-04-26T17:42:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21067.017513 secs 2024-04-26T17:42:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21069.873263 secs 2024-04-26T17:42:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21067.017513 secs 2024-04-26T17:42:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21069.873263 secs 2024-04-26T17:42:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:10 smithi194 ceph-mon[33017]: pgmap v11675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:10 smithi033 ceph-mon[27845]: pgmap v11675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:10 smithi096 ceph-mon[32753]: pgmap v11675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:12 smithi194 ceph-mon[33017]: pgmap v11676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:42:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:12 smithi033 ceph-mon[27845]: pgmap v11676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:42:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:12 smithi096 ceph-mon[32753]: pgmap v11676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:42:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:42:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:42:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:42:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:42:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:42:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:42:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:42:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:42:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:14 smithi033 ceph-mon[27845]: pgmap v11677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:14 smithi033 ceph-mon[27845]: pgmap v11678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21072.017594 secs 2024-04-26T17:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21074.873386 secs 2024-04-26T17:42:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:14 smithi096 ceph-mon[32753]: pgmap v11677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:14 smithi096 ceph-mon[32753]: pgmap v11678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21072.017594 secs 2024-04-26T17:42:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21074.873386 secs 2024-04-26T17:42:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:14 smithi194 ceph-mon[33017]: pgmap v11677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:14 smithi194 ceph-mon[33017]: pgmap v11678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21072.017594 secs 2024-04-26T17:42:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21074.873386 secs 2024-04-26T17:42:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:16 smithi033 ceph-mon[27845]: pgmap v11679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:16 smithi096 ceph-mon[32753]: pgmap v11679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:16 smithi194 ceph-mon[33017]: pgmap v11679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:18.247+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:18.247+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:42:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:18.247+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:18 smithi033 ceph-mon[27845]: pgmap v11680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:18 smithi096 ceph-mon[32753]: pgmap v11680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:18 smithi194 ceph-mon[33017]: pgmap v11680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:19.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21077.017678 secs 2024-04-26T17:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21079.873532 secs 2024-04-26T17:42:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21077.017678 secs 2024-04-26T17:42:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21079.873532 secs 2024-04-26T17:42:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21077.017678 secs 2024-04-26T17:42:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21079.873532 secs 2024-04-26T17:42:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:20 smithi033 ceph-mon[27845]: pgmap v11681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:20 smithi096 ceph-mon[32753]: pgmap v11681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:20 smithi194 ceph-mon[33017]: pgmap v11681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:22 smithi033 ceph-mon[27845]: pgmap v11682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:22 smithi096 ceph-mon[32753]: pgmap v11682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:22 smithi194 ceph-mon[33017]: pgmap v11682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:42:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:24 smithi033 ceph-mon[27845]: pgmap v11683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:42:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21082.017759 secs 2024-04-26T17:42:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21084.873670 secs 2024-04-26T17:42:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:24 smithi096 ceph-mon[32753]: pgmap v11683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:42:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21082.017759 secs 2024-04-26T17:42:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21084.873670 secs 2024-04-26T17:42:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:24 smithi194 ceph-mon[33017]: pgmap v11683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:42:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21082.017759 secs 2024-04-26T17:42:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21084.873670 secs 2024-04-26T17:42:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:26 smithi033 ceph-mon[27845]: pgmap v11684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:26 smithi096 ceph-mon[32753]: pgmap v11684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:26 smithi194 ceph-mon[33017]: pgmap v11684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:28 smithi033 ceph-mon[27845]: pgmap v11685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:28 smithi096 ceph-mon[32753]: pgmap v11685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:28 smithi194 ceph-mon[33017]: pgmap v11685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21087.017842 secs 2024-04-26T17:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21089.873812 secs 2024-04-26T17:42:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21087.017842 secs 2024-04-26T17:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21089.873812 secs 2024-04-26T17:42:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21087.017842 secs 2024-04-26T17:42:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21089.873812 secs 2024-04-26T17:42:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:30 smithi033 ceph-mon[27845]: pgmap v11686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:30 smithi096 ceph-mon[32753]: pgmap v11686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:30 smithi194 ceph-mon[33017]: pgmap v11686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:32 smithi096 ceph-mon[32753]: pgmap v11687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:32 smithi194 ceph-mon[33017]: pgmap v11687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:32 smithi033 ceph-mon[27845]: pgmap v11687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:33.258+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:33.258+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:33.258+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:42:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:34 smithi194 ceph-mon[33017]: pgmap v11688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21092.017931 secs 2024-04-26T17:42:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21094.873922 secs 2024-04-26T17:42:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:34 smithi033 ceph-mon[27845]: pgmap v11688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21092.017931 secs 2024-04-26T17:42:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21094.873922 secs 2024-04-26T17:42:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:34 smithi096 ceph-mon[32753]: pgmap v11688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21092.017931 secs 2024-04-26T17:42:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21094.873922 secs 2024-04-26T17:42:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:36 smithi194 ceph-mon[33017]: pgmap v11689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:36 smithi033 ceph-mon[27845]: pgmap v11689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:36 smithi096 ceph-mon[32753]: pgmap v11689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:38 smithi194 ceph-mon[33017]: pgmap v11690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:38 smithi033 ceph-mon[27845]: pgmap v11690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:38 smithi096 ceph-mon[32753]: pgmap v11690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21097.018009 secs 2024-04-26T17:42:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21099.874000 secs 2024-04-26T17:42:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21097.018009 secs 2024-04-26T17:42:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21099.874000 secs 2024-04-26T17:42:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21097.018009 secs 2024-04-26T17:42:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21099.874000 secs 2024-04-26T17:42:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:40 smithi194 ceph-mon[33017]: pgmap v11691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:40 smithi033 ceph-mon[27845]: pgmap v11691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:40 smithi096 ceph-mon[32753]: pgmap v11691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:42 smithi194 ceph-mon[33017]: pgmap v11692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:42 smithi033 ceph-mon[27845]: pgmap v11692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:42 smithi096 ceph-mon[32753]: pgmap v11692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21102.018072 secs 2024-04-26T17:42:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:44 smithi194 ceph-mon[33017]: pgmap v11693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21104.874143 secs 2024-04-26T17:42:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21102.018072 secs 2024-04-26T17:42:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:44 smithi033 ceph-mon[27845]: pgmap v11693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21104.874143 secs 2024-04-26T17:42:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21102.018072 secs 2024-04-26T17:42:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:44 smithi096 ceph-mon[32753]: pgmap v11693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21104.874143 secs 2024-04-26T17:42:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:46 smithi194 ceph-mon[33017]: pgmap v11694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:46 smithi033 ceph-mon[27845]: pgmap v11694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:46 smithi096 ceph-mon[32753]: pgmap v11694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:48.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:48.266+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:48.266 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:48.266+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:42:48.267 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:42:48.266+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:42:48.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:48 smithi096 ceph-mon[32753]: pgmap v11695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:48 smithi194 ceph-mon[33017]: pgmap v11695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:48 smithi033 ceph-mon[27845]: pgmap v11695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21107.018133 secs 2024-04-26T17:42:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21109.874261 secs 2024-04-26T17:42:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21107.018133 secs 2024-04-26T17:42:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21109.874261 secs 2024-04-26T17:42:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21107.018133 secs 2024-04-26T17:42:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21109.874261 secs 2024-04-26T17:42:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:50 smithi194 ceph-mon[33017]: pgmap v11696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:50 smithi033 ceph-mon[27845]: pgmap v11696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:50 smithi096 ceph-mon[32753]: pgmap v11696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:52 smithi194 ceph-mon[33017]: pgmap v11697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:52 smithi033 ceph-mon[27845]: pgmap v11697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:52 smithi096 ceph-mon[32753]: pgmap v11697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21112.018191 secs 2024-04-26T17:42:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:54 smithi194 ceph-mon[33017]: pgmap v11698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21114.874324 secs 2024-04-26T17:42:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21112.018191 secs 2024-04-26T17:42:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:54 smithi033 ceph-mon[27845]: pgmap v11698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21114.874324 secs 2024-04-26T17:42:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21112.018191 secs 2024-04-26T17:42:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:54 smithi096 ceph-mon[32753]: pgmap v11698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21114.874324 secs 2024-04-26T17:42:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:56 smithi194 ceph-mon[33017]: pgmap v11699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:56 smithi033 ceph-mon[27845]: pgmap v11699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:56 smithi096 ceph-mon[32753]: pgmap v11699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:42:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:58 smithi194 ceph-mon[33017]: pgmap v11700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:58 smithi033 ceph-mon[27845]: pgmap v11700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:58 smithi096 ceph-mon[32753]: pgmap v11700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:42:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21117.018272 secs 2024-04-26T17:42:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21119.874444 secs 2024-04-26T17:42:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21117.018272 secs 2024-04-26T17:42:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21119.874444 secs 2024-04-26T17:42:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21117.018272 secs 2024-04-26T17:42:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21119.874444 secs 2024-04-26T17:43:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:00 smithi194 ceph-mon[33017]: pgmap v11701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:00 smithi033 ceph-mon[27845]: pgmap v11701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:00 smithi096 ceph-mon[32753]: pgmap v11701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:02 smithi194 ceph-mon[33017]: pgmap v11702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:02 smithi033 ceph-mon[27845]: pgmap v11702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:02 smithi096 ceph-mon[32753]: pgmap v11702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:03.267+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:03.268+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:03.268+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:43:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:03 smithi194 ceph-mon[33017]: pgmap v11703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:03 smithi033 ceph-mon[27845]: pgmap v11703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:03 smithi096 ceph-mon[32753]: pgmap v11703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21122.018353 secs 2024-04-26T17:43:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21124.874564 secs 2024-04-26T17:43:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21122.018353 secs 2024-04-26T17:43:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21124.874564 secs 2024-04-26T17:43:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21122.018353 secs 2024-04-26T17:43:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21124.874564 secs 2024-04-26T17:43:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:05 smithi194 ceph-mon[33017]: pgmap v11704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:05 smithi033 ceph-mon[27845]: pgmap v11704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:05 smithi096 ceph-mon[32753]: pgmap v11704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:07 smithi194 ceph-mon[33017]: pgmap v11705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:07 smithi033 ceph-mon[27845]: pgmap v11705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:07 smithi096 ceph-mon[32753]: pgmap v11705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21127.018435 secs 2024-04-26T17:43:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21127.018435 secs 2024-04-26T17:43:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21127.018435 secs 2024-04-26T17:43:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21129.874674 secs 2024-04-26T17:43:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:09 smithi033 ceph-mon[27845]: pgmap v11706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21129.874674 secs 2024-04-26T17:43:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:09 smithi096 ceph-mon[32753]: pgmap v11706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21129.874674 secs 2024-04-26T17:43:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:09 smithi194 ceph-mon[33017]: pgmap v11706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:12.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:11 smithi194 ceph-mon[33017]: pgmap v11707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:11 smithi033 ceph-mon[27845]: pgmap v11707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:11 smithi096 ceph-mon[32753]: pgmap v11707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21132.018521 secs 2024-04-26T17:43:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:13 smithi033 ceph-mon[27845]: pgmap v11708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:43:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21132.018521 secs 2024-04-26T17:43:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:13 smithi096 ceph-mon[32753]: pgmap v11708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:43:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21132.018521 secs 2024-04-26T17:43:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:13 smithi194 ceph-mon[33017]: pgmap v11708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:43:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21134.874764 secs 2024-04-26T17:43:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21134.874764 secs 2024-04-26T17:43:15.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21134.874764 secs 2024-04-26T17:43:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:16 smithi194 ceph-mon[33017]: pgmap v11709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:43:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:43:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:16 smithi194 ceph-mon[33017]: pgmap v11710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:43:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:16 smithi194 ceph-mon[33017]: pgmap v11711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:16 smithi033 ceph-mon[27845]: pgmap v11709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:43:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:43:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:16 smithi033 ceph-mon[27845]: pgmap v11710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:43:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:16 smithi033 ceph-mon[27845]: pgmap v11711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:16 smithi096 ceph-mon[32753]: pgmap v11709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:43:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:43:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:16 smithi096 ceph-mon[32753]: pgmap v11710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:43:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:16 smithi096 ceph-mon[32753]: pgmap v11711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:43:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:18.284+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:18.284+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:43:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:18.284+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:18.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:18 smithi033 ceph-mon[27845]: pgmap v11712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:18 smithi096 ceph-mon[32753]: pgmap v11712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:18 smithi194 ceph-mon[33017]: pgmap v11712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21137.018586 secs 2024-04-26T17:43:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21139.874902 secs 2024-04-26T17:43:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21137.018586 secs 2024-04-26T17:43:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21139.874902 secs 2024-04-26T17:43:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21137.018586 secs 2024-04-26T17:43:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21139.874902 secs 2024-04-26T17:43:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:20 smithi033 ceph-mon[27845]: pgmap v11713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:43:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:20 smithi096 ceph-mon[32753]: pgmap v11713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:43:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:20 smithi194 ceph-mon[33017]: pgmap v11713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:43:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:22 smithi033 ceph-mon[27845]: pgmap v11714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:22 smithi096 ceph-mon[32753]: pgmap v11714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:22 smithi194 ceph-mon[33017]: pgmap v11714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:43:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21142.018670 secs 2024-04-26T17:43:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:24 smithi033 ceph-mon[27845]: pgmap v11715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:43:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21144.875027 secs 2024-04-26T17:43:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21142.018670 secs 2024-04-26T17:43:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:24 smithi096 ceph-mon[32753]: pgmap v11715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:43:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21144.875027 secs 2024-04-26T17:43:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21142.018670 secs 2024-04-26T17:43:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:24 smithi194 ceph-mon[33017]: pgmap v11715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:43:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21144.875027 secs 2024-04-26T17:43:26.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:26 smithi033 ceph-mon[27845]: pgmap v11716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:43:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:26 smithi096 ceph-mon[32753]: pgmap v11716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:43:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:26 smithi194 ceph-mon[33017]: pgmap v11716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:43:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:28 smithi096 ceph-mon[32753]: pgmap v11717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:28 smithi194 ceph-mon[33017]: pgmap v11717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:28 smithi033 ceph-mon[27845]: pgmap v11717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21147.018754 secs 2024-04-26T17:43:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21149.875096 secs 2024-04-26T17:43:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21147.018754 secs 2024-04-26T17:43:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21149.875096 secs 2024-04-26T17:43:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21147.018754 secs 2024-04-26T17:43:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21149.875096 secs 2024-04-26T17:43:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:30 smithi194 ceph-mon[33017]: pgmap v11718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:30 smithi033 ceph-mon[27845]: pgmap v11718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:30 smithi096 ceph-mon[32753]: pgmap v11718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:32 smithi194 ceph-mon[33017]: pgmap v11719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:32 smithi033 ceph-mon[27845]: pgmap v11719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:32 smithi096 ceph-mon[32753]: pgmap v11719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:33.300+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:33.300+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:33.300+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:43:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21152.018835 secs 2024-04-26T17:43:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:34 smithi194 ceph-mon[33017]: pgmap v11720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21154.875191 secs 2024-04-26T17:43:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21152.018835 secs 2024-04-26T17:43:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:34 smithi033 ceph-mon[27845]: pgmap v11720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21154.875191 secs 2024-04-26T17:43:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21152.018835 secs 2024-04-26T17:43:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:34 smithi096 ceph-mon[32753]: pgmap v11720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21154.875191 secs 2024-04-26T17:43:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:36 smithi194 ceph-mon[33017]: pgmap v11721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:36 smithi033 ceph-mon[27845]: pgmap v11721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:36 smithi096 ceph-mon[32753]: pgmap v11721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:43:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:38 smithi194 ceph-mon[33017]: pgmap v11722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:38.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:38 smithi033 ceph-mon[27845]: pgmap v11722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:38 smithi096 ceph-mon[32753]: pgmap v11722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21157.018917 secs 2024-04-26T17:43:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21159.875309 secs 2024-04-26T17:43:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21157.018917 secs 2024-04-26T17:43:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21159.875309 secs 2024-04-26T17:43:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21157.018917 secs 2024-04-26T17:43:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21159.875309 secs 2024-04-26T17:43:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:40 smithi194 ceph-mon[33017]: pgmap v11723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:40 smithi033 ceph-mon[27845]: pgmap v11723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:40 smithi096 ceph-mon[32753]: pgmap v11723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:42 smithi194 ceph-mon[33017]: pgmap v11724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:42.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:42 smithi033 ceph-mon[27845]: pgmap v11724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:42 smithi096 ceph-mon[32753]: pgmap v11724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:43:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21162.019000 secs 2024-04-26T17:43:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:44 smithi194 ceph-mon[33017]: pgmap v11725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21164.875425 secs 2024-04-26T17:43:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21162.019000 secs 2024-04-26T17:43:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:44 smithi033 ceph-mon[27845]: pgmap v11725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21164.875425 secs 2024-04-26T17:43:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21162.019000 secs 2024-04-26T17:43:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:44 smithi096 ceph-mon[32753]: pgmap v11725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21164.875425 secs 2024-04-26T17:43:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:46 smithi194 ceph-mon[33017]: pgmap v11726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:43:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:46 smithi033 ceph-mon[27845]: pgmap v11726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:43:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:46 smithi096 ceph-mon[32753]: pgmap v11726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:43:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:48.315+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:48.315+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:43:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:43:48.315+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:43:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:48 smithi194 ceph-mon[33017]: pgmap v11727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:43:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:48 smithi033 ceph-mon[27845]: pgmap v11727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:43:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:48 smithi096 ceph-mon[32753]: pgmap v11727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:43:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21167.019061 secs 2024-04-26T17:43:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21169.875517 secs 2024-04-26T17:43:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21167.019061 secs 2024-04-26T17:43:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21169.875517 secs 2024-04-26T17:43:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21167.019061 secs 2024-04-26T17:43:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21169.875517 secs 2024-04-26T17:43:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:50 smithi194 ceph-mon[33017]: pgmap v11728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:43:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:50 smithi033 ceph-mon[27845]: pgmap v11728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:43:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:50 smithi096 ceph-mon[32753]: pgmap v11728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:43:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:52 smithi194 ceph-mon[33017]: pgmap v11729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:43:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:52 smithi033 ceph-mon[27845]: pgmap v11729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:43:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:52 smithi096 ceph-mon[32753]: pgmap v11729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:43:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21172.019122 secs 2024-04-26T17:43:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:54 smithi194 ceph-mon[33017]: pgmap v11730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:43:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21174.875628 secs 2024-04-26T17:43:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21172.019122 secs 2024-04-26T17:43:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:54 smithi033 ceph-mon[27845]: pgmap v11730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:43:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21174.875628 secs 2024-04-26T17:43:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21172.019122 secs 2024-04-26T17:43:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:54 smithi096 ceph-mon[32753]: pgmap v11730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:43:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21174.875628 secs 2024-04-26T17:43:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:56 smithi194 ceph-mon[33017]: pgmap v11731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:43:56.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:56 smithi033 ceph-mon[27845]: pgmap v11731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:43:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:56 smithi096 ceph-mon[32753]: pgmap v11731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:43:58.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:58 smithi033 ceph-mon[27845]: pgmap v11732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:58 smithi194 ceph-mon[33017]: pgmap v11732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:58 smithi096 ceph-mon[32753]: pgmap v11732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:43:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21177.019206 secs 2024-04-26T17:43:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21179.875707 secs 2024-04-26T17:43:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21177.019206 secs 2024-04-26T17:43:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21179.875707 secs 2024-04-26T17:43:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21177.019206 secs 2024-04-26T17:43:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21179.875707 secs 2024-04-26T17:44:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:00 smithi194 ceph-mon[33017]: pgmap v11733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:00 smithi033 ceph-mon[27845]: pgmap v11733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:00 smithi096 ceph-mon[32753]: pgmap v11733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:02 smithi194 ceph-mon[33017]: pgmap v11734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:02 smithi033 ceph-mon[27845]: pgmap v11734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:02 smithi096 ceph-mon[32753]: pgmap v11734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:03.328+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:03.328+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:03.328+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:44:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:03 smithi194 ceph-mon[33017]: pgmap v11735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:03 smithi033 ceph-mon[27845]: pgmap v11735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:03 smithi096 ceph-mon[32753]: pgmap v11735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21182.019268 secs 2024-04-26T17:44:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21184.875823 secs 2024-04-26T17:44:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21182.019268 secs 2024-04-26T17:44:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21184.875823 secs 2024-04-26T17:44:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21182.019268 secs 2024-04-26T17:44:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21184.875823 secs 2024-04-26T17:44:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:05 smithi194 ceph-mon[33017]: pgmap v11736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:05 smithi033 ceph-mon[27845]: pgmap v11736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:05 smithi096 ceph-mon[32753]: pgmap v11736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:08 smithi033 ceph-mon[27845]: pgmap v11737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:08 smithi096 ceph-mon[32753]: pgmap v11737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:08 smithi194 ceph-mon[33017]: pgmap v11737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21187.019350 secs 2024-04-26T17:44:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21189.875934 secs 2024-04-26T17:44:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21187.019350 secs 2024-04-26T17:44:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21189.875934 secs 2024-04-26T17:44:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21187.019350 secs 2024-04-26T17:44:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21189.875934 secs 2024-04-26T17:44:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:10 smithi096 ceph-mon[32753]: pgmap v11738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:10 smithi194 ceph-mon[33017]: pgmap v11738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:10 smithi033 ceph-mon[27845]: pgmap v11738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:12 smithi033 ceph-mon[27845]: pgmap v11739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:12 smithi194 ceph-mon[33017]: pgmap v11739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:12 smithi096 ceph-mon[32753]: pgmap v11739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21192.019434 secs 2024-04-26T17:44:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:14 smithi194 ceph-mon[33017]: pgmap v11740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21194.876002 secs 2024-04-26T17:44:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21192.019434 secs 2024-04-26T17:44:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:14 smithi033 ceph-mon[27845]: pgmap v11740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21194.876002 secs 2024-04-26T17:44:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21192.019434 secs 2024-04-26T17:44:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:14 smithi096 ceph-mon[32753]: pgmap v11740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21194.876002 secs 2024-04-26T17:44:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:16 smithi194 ceph-mon[33017]: pgmap v11741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:44:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:16 smithi033 ceph-mon[27845]: pgmap v11741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:44:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:16 smithi096 ceph-mon[32753]: pgmap v11741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:44:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:44:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:44:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:44:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:44:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:44:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:44:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:44:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:44:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:44:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:18.334+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:18.334+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:18.334+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:44:18.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:18 smithi096 ceph-mon[32753]: pgmap v11742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:18.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:18 smithi096 ceph-mon[32753]: pgmap v11743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:18 smithi194 ceph-mon[33017]: pgmap v11742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:18 smithi194 ceph-mon[33017]: pgmap v11743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:18 smithi033 ceph-mon[27845]: pgmap v11742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:18 smithi033 ceph-mon[27845]: pgmap v11743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21197.019519 secs 2024-04-26T17:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21199.876065 secs 2024-04-26T17:44:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21197.019519 secs 2024-04-26T17:44:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21199.876065 secs 2024-04-26T17:44:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21197.019519 secs 2024-04-26T17:44:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21199.876065 secs 2024-04-26T17:44:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:20 smithi194 ceph-mon[33017]: pgmap v11744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:44:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:20 smithi033 ceph-mon[27845]: pgmap v11744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:44:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:20 smithi096 ceph-mon[32753]: pgmap v11744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:44:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:22 smithi194 ceph-mon[33017]: pgmap v11745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:22 smithi033 ceph-mon[27845]: pgmap v11745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:22 smithi096 ceph-mon[32753]: pgmap v11745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21202.019604 secs 2024-04-26T17:44:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:24 smithi194 ceph-mon[33017]: pgmap v11746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21204.876184 secs 2024-04-26T17:44:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21202.019604 secs 2024-04-26T17:44:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:24 smithi033 ceph-mon[27845]: pgmap v11746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21204.876184 secs 2024-04-26T17:44:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21202.019604 secs 2024-04-26T17:44:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:24 smithi096 ceph-mon[32753]: pgmap v11746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21204.876184 secs 2024-04-26T17:44:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.402+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:44:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.402+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T17:44:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.402+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 169 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T17:44:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.402+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T17:44:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.403+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 169 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc004910 con 0x7fef00077780 2024-04-26T17:44:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.403+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 170 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:44:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.403+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:44:25.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:25.403+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 170 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:44:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:26 smithi194 ceph-mon[33017]: pgmap v11747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:26 smithi033 ceph-mon[27845]: pgmap v11747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:26 smithi096 ceph-mon[32753]: pgmap v11747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:44:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:28 smithi194 ceph-mon[33017]: pgmap v11748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:44:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:28 smithi033 ceph-mon[27845]: pgmap v11748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:44:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:28 smithi096 ceph-mon[32753]: pgmap v11748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:44:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21207.019688 secs 2024-04-26T17:44:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21209.876294 secs 2024-04-26T17:44:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21207.019688 secs 2024-04-26T17:44:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21209.876294 secs 2024-04-26T17:44:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21207.019688 secs 2024-04-26T17:44:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21209.876294 secs 2024-04-26T17:44:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:30 smithi194 ceph-mon[33017]: pgmap v11749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:30 smithi033 ceph-mon[27845]: pgmap v11749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:30 smithi096 ceph-mon[32753]: pgmap v11749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:32 smithi194 ceph-mon[33017]: pgmap v11750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:32.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:32 smithi033 ceph-mon[27845]: pgmap v11750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:32 smithi096 ceph-mon[32753]: pgmap v11750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:33.339+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:33.339+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:44:33.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:33.339+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21212.019773 secs 2024-04-26T17:44:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:34 smithi194 ceph-mon[33017]: pgmap v11751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21214.876411 secs 2024-04-26T17:44:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21212.019773 secs 2024-04-26T17:44:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:34 smithi033 ceph-mon[27845]: pgmap v11751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21214.876411 secs 2024-04-26T17:44:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21212.019773 secs 2024-04-26T17:44:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:34 smithi096 ceph-mon[32753]: pgmap v11751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21214.876411 secs 2024-04-26T17:44:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:36 smithi194 ceph-mon[33017]: pgmap v11752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:36 smithi033 ceph-mon[27845]: pgmap v11752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:36 smithi096 ceph-mon[32753]: pgmap v11752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:38 smithi194 ceph-mon[33017]: pgmap v11753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:38 smithi033 ceph-mon[27845]: pgmap v11753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:38 smithi096 ceph-mon[32753]: pgmap v11753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21217.019856 secs 2024-04-26T17:44:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21219.876535 secs 2024-04-26T17:44:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21217.019856 secs 2024-04-26T17:44:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21219.876535 secs 2024-04-26T17:44:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21217.019856 secs 2024-04-26T17:44:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21219.876535 secs 2024-04-26T17:44:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:40 smithi194 ceph-mon[33017]: pgmap v11754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:40 smithi033 ceph-mon[27845]: pgmap v11754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:40 smithi096 ceph-mon[32753]: pgmap v11754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:42 smithi194 ceph-mon[33017]: pgmap v11755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:42 smithi033 ceph-mon[27845]: pgmap v11755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:42 smithi096 ceph-mon[32753]: pgmap v11755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21222.019939 secs 2024-04-26T17:44:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:44 smithi194 ceph-mon[33017]: pgmap v11756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21224.876622 secs 2024-04-26T17:44:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21222.019939 secs 2024-04-26T17:44:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:44 smithi033 ceph-mon[27845]: pgmap v11756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21224.876622 secs 2024-04-26T17:44:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21222.019939 secs 2024-04-26T17:44:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:44 smithi096 ceph-mon[32753]: pgmap v11756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21224.876622 secs 2024-04-26T17:44:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:46 smithi194 ceph-mon[33017]: pgmap v11757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:46 smithi033 ceph-mon[27845]: pgmap v11757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:46 smithi096 ceph-mon[32753]: pgmap v11757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:47 smithi194 ceph-mon[33017]: pgmap v11758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:47 smithi033 ceph-mon[27845]: pgmap v11758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:47 smithi096 ceph-mon[32753]: pgmap v11758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:48.354+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:48.354+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:44:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:44:48.354+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:44:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21227.020019 secs 2024-04-26T17:44:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21227.020019 secs 2024-04-26T17:44:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21227.020019 secs 2024-04-26T17:44:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21229.876754 secs 2024-04-26T17:44:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:49 smithi194 ceph-mon[33017]: pgmap v11759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21229.876754 secs 2024-04-26T17:44:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:49 smithi033 ceph-mon[27845]: pgmap v11759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21229.876754 secs 2024-04-26T17:44:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:49 smithi096 ceph-mon[32753]: pgmap v11759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:52 smithi194 ceph-mon[33017]: pgmap v11760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:52 smithi033 ceph-mon[27845]: pgmap v11760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:52 smithi096 ceph-mon[32753]: pgmap v11760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21232.020079 secs 2024-04-26T17:44:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:54 smithi194 ceph-mon[33017]: pgmap v11761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21232.020079 secs 2024-04-26T17:44:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:54 smithi033 ceph-mon[27845]: pgmap v11761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21232.020079 secs 2024-04-26T17:44:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:54 smithi096 ceph-mon[32753]: pgmap v11761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21234.876861 secs 2024-04-26T17:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21234.876861 secs 2024-04-26T17:44:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21234.876861 secs 2024-04-26T17:44:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:56 smithi194 ceph-mon[33017]: pgmap v11762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:56 smithi033 ceph-mon[27845]: pgmap v11762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:56 smithi096 ceph-mon[32753]: pgmap v11762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:44:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:58 smithi033 ceph-mon[27845]: pgmap v11763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:58 smithi096 ceph-mon[32753]: pgmap v11763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:58 smithi194 ceph-mon[33017]: pgmap v11763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:44:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:44:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21237.020135 secs 2024-04-26T17:44:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:44:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21237.020135 secs 2024-04-26T17:44:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:44:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21237.020135 secs 2024-04-26T17:45:00.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21239.876980 secs 2024-04-26T17:45:00.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:00 smithi033 ceph-mon[27845]: pgmap v11764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21239.876980 secs 2024-04-26T17:45:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:00 smithi096 ceph-mon[32753]: pgmap v11764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21239.876980 secs 2024-04-26T17:45:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:00 smithi194 ceph-mon[33017]: pgmap v11764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:02 smithi033 ceph-mon[27845]: pgmap v11765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:02 smithi096 ceph-mon[32753]: pgmap v11765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:02 smithi194 ceph-mon[33017]: pgmap v11765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:03.369+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:03.369+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:45:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:03.369+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21242.020216 secs 2024-04-26T17:45:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:04 smithi033 ceph-mon[27845]: pgmap v11766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21242.020216 secs 2024-04-26T17:45:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:04 smithi096 ceph-mon[32753]: pgmap v11766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21242.020216 secs 2024-04-26T17:45:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:04 smithi194 ceph-mon[33017]: pgmap v11766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21244.877099 secs 2024-04-26T17:45:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21244.877099 secs 2024-04-26T17:45:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21244.877099 secs 2024-04-26T17:45:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:06 smithi033 ceph-mon[27845]: pgmap v11767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:06 smithi096 ceph-mon[32753]: pgmap v11767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:06 smithi194 ceph-mon[33017]: pgmap v11767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:08 smithi033 ceph-mon[27845]: pgmap v11768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:08 smithi096 ceph-mon[32753]: pgmap v11768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:08 smithi194 ceph-mon[33017]: pgmap v11768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:09.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21247.020300 secs 2024-04-26T17:45:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21247.020300 secs 2024-04-26T17:45:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21247.020300 secs 2024-04-26T17:45:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21249.877220 secs 2024-04-26T17:45:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:10 smithi033 ceph-mon[27845]: pgmap v11769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21249.877220 secs 2024-04-26T17:45:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:10 smithi096 ceph-mon[32753]: pgmap v11769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21249.877220 secs 2024-04-26T17:45:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:10 smithi194 ceph-mon[33017]: pgmap v11769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:12 smithi033 ceph-mon[27845]: pgmap v11770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:12 smithi096 ceph-mon[32753]: pgmap v11770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:12 smithi194 ceph-mon[33017]: pgmap v11770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21252.020360 secs 2024-04-26T17:45:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:14 smithi033 ceph-mon[27845]: pgmap v11771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21252.020360 secs 2024-04-26T17:45:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:14 smithi096 ceph-mon[32753]: pgmap v11771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21252.020360 secs 2024-04-26T17:45:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:14 smithi194 ceph-mon[33017]: pgmap v11771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21254.877376 secs 2024-04-26T17:45:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21254.877376 secs 2024-04-26T17:45:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21254.877376 secs 2024-04-26T17:45:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:16 smithi194 ceph-mon[33017]: pgmap v11772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:16 smithi033 ceph-mon[27845]: pgmap v11772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:16 smithi096 ceph-mon[32753]: pgmap v11772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:18.380+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:18.380+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:45:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:18.380+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:18.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:18 smithi096 ceph-mon[32753]: pgmap v11773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:18.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:18.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:18 smithi194 ceph-mon[33017]: pgmap v11773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:18 smithi033 ceph-mon[27845]: pgmap v11773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21257.020426 secs 2024-04-26T17:45:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21257.020426 secs 2024-04-26T17:45:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21257.020426 secs 2024-04-26T17:45:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21259.877442 secs 2024-04-26T17:45:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:20 smithi194 ceph-mon[33017]: pgmap v11774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21259.877442 secs 2024-04-26T17:45:20.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:20 smithi033 ceph-mon[27845]: pgmap v11774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21259.877442 secs 2024-04-26T17:45:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:20 smithi096 ceph-mon[32753]: pgmap v11774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:21.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:22 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:45:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:22 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:45:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:22 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:45:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:22 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:45:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:22 smithi194 ceph-mon[33017]: pgmap v11775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:22 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:45:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:22 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:45:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:22 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:45:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:22 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:45:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:22 smithi033 ceph-mon[27845]: pgmap v11775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:22 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:45:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:22 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:45:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:22 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:45:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:22 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:45:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:22 smithi096 ceph-mon[32753]: pgmap v11775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21262.020507 secs 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: pgmap v11776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: pgmap v11777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: pgmap v11778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:45:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21262.020507 secs 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: pgmap v11776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:45:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:45:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: pgmap v11777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:45:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: pgmap v11778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:45:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21262.020507 secs 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: pgmap v11776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:45:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: pgmap v11777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:45:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: pgmap v11778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:45:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:45:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21264.877517 secs 2024-04-26T17:45:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21264.877517 secs 2024-04-26T17:45:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21264.877517 secs 2024-04-26T17:45:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:26 smithi194 ceph-mon[33017]: pgmap v11779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:26 smithi033 ceph-mon[27845]: pgmap v11779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:26 smithi096 ceph-mon[32753]: pgmap v11779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:28 smithi194 ceph-mon[33017]: pgmap v11780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:28 smithi033 ceph-mon[27845]: pgmap v11780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:28 smithi096 ceph-mon[32753]: pgmap v11780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:29.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21267.020590 secs 2024-04-26T17:45:29.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21269.877616 secs 2024-04-26T17:45:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21267.020590 secs 2024-04-26T17:45:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21269.877616 secs 2024-04-26T17:45:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21267.020590 secs 2024-04-26T17:45:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21269.877616 secs 2024-04-26T17:45:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:30 smithi194 ceph-mon[33017]: pgmap v11781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:30 smithi033 ceph-mon[27845]: pgmap v11781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:30 smithi096 ceph-mon[32753]: pgmap v11781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:31 smithi194 ceph-mon[33017]: pgmap v11782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:31 smithi033 ceph-mon[27845]: pgmap v11782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:31 smithi096 ceph-mon[32753]: pgmap v11782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:45:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:33.395+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:33.395+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:33.395+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:45:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21272.020671 secs 2024-04-26T17:45:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21274.877730 secs 2024-04-26T17:45:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:34 smithi194 ceph-mon[33017]: pgmap v11783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:45:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21272.020671 secs 2024-04-26T17:45:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21274.877730 secs 2024-04-26T17:45:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:34 smithi033 ceph-mon[27845]: pgmap v11783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:45:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21272.020671 secs 2024-04-26T17:45:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21274.877730 secs 2024-04-26T17:45:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:34 smithi096 ceph-mon[32753]: pgmap v11783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:45:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:36 smithi194 ceph-mon[33017]: pgmap v11784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:36 smithi033 ceph-mon[27845]: pgmap v11784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:36 smithi096 ceph-mon[32753]: pgmap v11784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:37 smithi194 ceph-mon[33017]: pgmap v11785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:37 smithi033 ceph-mon[27845]: pgmap v11785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:37 smithi096 ceph-mon[32753]: pgmap v11785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21277.020755 secs 2024-04-26T17:45:39.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21277.020755 secs 2024-04-26T17:45:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21277.020755 secs 2024-04-26T17:45:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21279.877880 secs 2024-04-26T17:45:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:39 smithi194 ceph-mon[33017]: pgmap v11786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21279.877880 secs 2024-04-26T17:45:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:39 smithi033 ceph-mon[27845]: pgmap v11786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21279.877880 secs 2024-04-26T17:45:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:39 smithi096 ceph-mon[32753]: pgmap v11786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:42 smithi194 ceph-mon[33017]: pgmap v11787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:42 smithi033 ceph-mon[27845]: pgmap v11787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:42 smithi096 ceph-mon[32753]: pgmap v11787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:43 smithi033 ceph-mon[27845]: pgmap v11788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:43 smithi096 ceph-mon[32753]: pgmap v11788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:43 smithi194 ceph-mon[33017]: pgmap v11788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21282.020833 secs 2024-04-26T17:45:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21284.877998 secs 2024-04-26T17:45:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21282.020833 secs 2024-04-26T17:45:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21284.877998 secs 2024-04-26T17:45:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21282.020833 secs 2024-04-26T17:45:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21284.877998 secs 2024-04-26T17:45:45.651 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:45 smithi033 ceph-mon[27845]: pgmap v11789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:45 smithi096 ceph-mon[32753]: pgmap v11789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:45 smithi194 ceph-mon[33017]: pgmap v11789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:48.411+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:48.411+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:45:48.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:45:48.411+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:45:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:48 smithi194 ceph-mon[33017]: pgmap v11790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:48 smithi033 ceph-mon[27845]: pgmap v11790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:48 smithi096 ceph-mon[32753]: pgmap v11790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21287.020918 secs 2024-04-26T17:45:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21289.878135 secs 2024-04-26T17:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:49 smithi194 ceph-mon[33017]: pgmap v11791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21287.020918 secs 2024-04-26T17:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21289.878135 secs 2024-04-26T17:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:49 smithi033 ceph-mon[27845]: pgmap v11791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21287.020918 secs 2024-04-26T17:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21289.878135 secs 2024-04-26T17:45:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:49 smithi096 ceph-mon[32753]: pgmap v11791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:52 smithi194 ceph-mon[33017]: pgmap v11792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:52 smithi033 ceph-mon[27845]: pgmap v11792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:52 smithi096 ceph-mon[32753]: pgmap v11792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:53 smithi194 ceph-mon[33017]: pgmap v11793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:53 smithi033 ceph-mon[27845]: pgmap v11793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:53 smithi096 ceph-mon[32753]: pgmap v11793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21292.020998 secs 2024-04-26T17:45:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21294.878205 secs 2024-04-26T17:45:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21292.020998 secs 2024-04-26T17:45:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21294.878205 secs 2024-04-26T17:45:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21292.020998 secs 2024-04-26T17:45:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21294.878205 secs 2024-04-26T17:45:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:55 smithi194 ceph-mon[33017]: pgmap v11794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:55 smithi033 ceph-mon[27845]: pgmap v11794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:55 smithi096 ceph-mon[32753]: pgmap v11794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:45:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:57 smithi033 ceph-mon[27845]: pgmap v11795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:57 smithi096 ceph-mon[32753]: pgmap v11795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:57 smithi194 ceph-mon[33017]: pgmap v11795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:45:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21297.021083 secs 2024-04-26T17:45:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21297.021083 secs 2024-04-26T17:45:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21297.021083 secs 2024-04-26T17:46:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21299.878321 secs 2024-04-26T17:46:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:45:59 smithi194 ceph-mon[33017]: pgmap v11796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:00.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21299.878321 secs 2024-04-26T17:46:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:45:59 smithi033 ceph-mon[27845]: pgmap v11796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21299.878321 secs 2024-04-26T17:46:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:45:59 smithi096 ceph-mon[32753]: pgmap v11796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:01.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:01 smithi033 ceph-mon[27845]: pgmap v11797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:01 smithi096 ceph-mon[32753]: pgmap v11797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:01 smithi194 ceph-mon[33017]: pgmap v11797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:03.422+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:03.422+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:03.422+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:46:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:03 smithi033 ceph-mon[27845]: pgmap v11798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:03 smithi096 ceph-mon[32753]: pgmap v11798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:03 smithi194 ceph-mon[33017]: pgmap v11798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21302.021142 secs 2024-04-26T17:46:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21304.878436 secs 2024-04-26T17:46:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21302.021142 secs 2024-04-26T17:46:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21304.878436 secs 2024-04-26T17:46:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21302.021142 secs 2024-04-26T17:46:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21304.878436 secs 2024-04-26T17:46:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:05 smithi033 ceph-mon[27845]: pgmap v11799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:05 smithi096 ceph-mon[32753]: pgmap v11799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:05 smithi194 ceph-mon[33017]: pgmap v11799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:07 smithi033 ceph-mon[27845]: pgmap v11800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:07 smithi096 ceph-mon[32753]: pgmap v11800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:07 smithi194 ceph-mon[33017]: pgmap v11800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21307.021222 secs 2024-04-26T17:46:09.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21307.021222 secs 2024-04-26T17:46:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21307.021222 secs 2024-04-26T17:46:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21309.878559 secs 2024-04-26T17:46:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:09 smithi194 ceph-mon[33017]: pgmap v11801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21309.878559 secs 2024-04-26T17:46:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:09 smithi033 ceph-mon[27845]: pgmap v11801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21309.878559 secs 2024-04-26T17:46:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:09 smithi096 ceph-mon[32753]: pgmap v11801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:11 smithi033 ceph-mon[27845]: pgmap v11802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:11 smithi096 ceph-mon[32753]: pgmap v11802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:11 smithi194 ceph-mon[33017]: pgmap v11802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:13.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:13 smithi194 ceph-mon[33017]: pgmap v11803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:13 smithi033 ceph-mon[27845]: pgmap v11803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:13 smithi096 ceph-mon[32753]: pgmap v11803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21312.021300 secs 2024-04-26T17:46:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21314.878636 secs 2024-04-26T17:46:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21312.021300 secs 2024-04-26T17:46:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21314.878636 secs 2024-04-26T17:46:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21312.021300 secs 2024-04-26T17:46:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21314.878636 secs 2024-04-26T17:46:15.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:15 smithi033 ceph-mon[27845]: pgmap v11804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:15 smithi096 ceph-mon[32753]: pgmap v11804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:15 smithi194 ceph-mon[33017]: pgmap v11804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:17 smithi033 ceph-mon[27845]: pgmap v11805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:17 smithi096 ceph-mon[32753]: pgmap v11805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:17 smithi194 ceph-mon[33017]: pgmap v11805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:18.437+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:18.437+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:18.437+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:46:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21317.021387 secs 2024-04-26T17:46:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21317.021387 secs 2024-04-26T17:46:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21317.021387 secs 2024-04-26T17:46:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21319.878732 secs 2024-04-26T17:46:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:19 smithi194 ceph-mon[33017]: pgmap v11806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21319.878732 secs 2024-04-26T17:46:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:19 smithi033 ceph-mon[27845]: pgmap v11806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21319.878732 secs 2024-04-26T17:46:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:19 smithi096 ceph-mon[32753]: pgmap v11806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:21 smithi033 ceph-mon[27845]: pgmap v11807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:21 smithi096 ceph-mon[32753]: pgmap v11807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:21 smithi194 ceph-mon[33017]: pgmap v11807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:23 smithi033 ceph-mon[27845]: pgmap v11808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:23 smithi096 ceph-mon[32753]: pgmap v11808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:23 smithi194 ceph-mon[33017]: pgmap v11808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21322.021468 secs 2024-04-26T17:46:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21324.878850 secs 2024-04-26T17:46:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21322.021468 secs 2024-04-26T17:46:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21324.878850 secs 2024-04-26T17:46:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21322.021468 secs 2024-04-26T17:46:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21324.878850 secs 2024-04-26T17:46:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:25 smithi033 ceph-mon[27845]: pgmap v11809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:46:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:25 smithi033 ceph-mon[27845]: pgmap v11810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:46:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:25 smithi096 ceph-mon[32753]: pgmap v11809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:46:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:25 smithi096 ceph-mon[32753]: pgmap v11810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:46:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:25 smithi194 ceph-mon[33017]: pgmap v11809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:46:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:25 smithi194 ceph-mon[33017]: pgmap v11810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:46:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:28 smithi194 ceph-mon[33017]: pgmap v11811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:28.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:28 smithi033 ceph-mon[27845]: pgmap v11811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:28 smithi096 ceph-mon[32753]: pgmap v11811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21327.021551 secs 2024-04-26T17:46:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21329.878969 secs 2024-04-26T17:46:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21327.021551 secs 2024-04-26T17:46:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21329.878969 secs 2024-04-26T17:46:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21327.021551 secs 2024-04-26T17:46:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21329.878969 secs 2024-04-26T17:46:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:30 smithi194 ceph-mon[33017]: pgmap v11812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T17:46:30.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:30 smithi033 ceph-mon[27845]: pgmap v11812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T17:46:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:30 smithi096 ceph-mon[32753]: pgmap v11812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T17:46:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:32 smithi194 ceph-mon[33017]: pgmap v11813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:32 smithi033 ceph-mon[27845]: pgmap v11813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:32 smithi096 ceph-mon[32753]: pgmap v11813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:33.453+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:33.454+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:33.454+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:46:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21332.021633 secs 2024-04-26T17:46:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:34 smithi194 ceph-mon[33017]: pgmap v11814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21334.879090 secs 2024-04-26T17:46:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21332.021633 secs 2024-04-26T17:46:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:34 smithi033 ceph-mon[27845]: pgmap v11814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21334.879090 secs 2024-04-26T17:46:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21332.021633 secs 2024-04-26T17:46:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:34 smithi096 ceph-mon[32753]: pgmap v11814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21334.879090 secs 2024-04-26T17:46:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:36 smithi194 ceph-mon[33017]: pgmap v11815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:46:36.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:36 smithi033 ceph-mon[27845]: pgmap v11815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:46:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:36 smithi096 ceph-mon[32753]: pgmap v11815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:46:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:38 smithi194 ceph-mon[33017]: pgmap v11816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:38 smithi033 ceph-mon[27845]: pgmap v11816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:38 smithi096 ceph-mon[32753]: pgmap v11816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21337.021717 secs 2024-04-26T17:46:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21339.879211 secs 2024-04-26T17:46:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21337.021717 secs 2024-04-26T17:46:39.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21339.879211 secs 2024-04-26T17:46:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21337.021717 secs 2024-04-26T17:46:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21339.879211 secs 2024-04-26T17:46:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:40 smithi194 ceph-mon[33017]: pgmap v11817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:46:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:40 smithi033 ceph-mon[27845]: pgmap v11817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:46:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:40 smithi096 ceph-mon[32753]: pgmap v11817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:46:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:42 smithi194 ceph-mon[33017]: pgmap v11818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:42.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:42 smithi033 ceph-mon[27845]: pgmap v11818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:42 smithi096 ceph-mon[32753]: pgmap v11818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21342.021821 secs 2024-04-26T17:46:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:44 smithi194 ceph-mon[33017]: pgmap v11819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21344.879323 secs 2024-04-26T17:46:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21342.021821 secs 2024-04-26T17:46:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:44 smithi033 ceph-mon[27845]: pgmap v11819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21344.879323 secs 2024-04-26T17:46:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21342.021821 secs 2024-04-26T17:46:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:44 smithi096 ceph-mon[32753]: pgmap v11819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21344.879323 secs 2024-04-26T17:46:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:46 smithi033 ceph-mon[27845]: pgmap v11820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:46 smithi096 ceph-mon[32753]: pgmap v11820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:46 smithi194 ceph-mon[33017]: pgmap v11820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:48.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:48.469+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:48.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:48.469+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:46:48.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:46:48.469+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:46:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:48 smithi033 ceph-mon[27845]: pgmap v11821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:48 smithi096 ceph-mon[32753]: pgmap v11821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:48 smithi194 ceph-mon[33017]: pgmap v11821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21347.021900 secs 2024-04-26T17:46:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21349.879405 secs 2024-04-26T17:46:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21347.021900 secs 2024-04-26T17:46:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21349.879405 secs 2024-04-26T17:46:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21347.021900 secs 2024-04-26T17:46:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21349.879405 secs 2024-04-26T17:46:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:50 smithi033 ceph-mon[27845]: pgmap v11822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:46:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:50 smithi096 ceph-mon[32753]: pgmap v11822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:46:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:50 smithi194 ceph-mon[33017]: pgmap v11822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:46:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:52 smithi033 ceph-mon[27845]: pgmap v11823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:52 smithi096 ceph-mon[32753]: pgmap v11823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:52 smithi194 ceph-mon[33017]: pgmap v11823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:46:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21352.021988 secs 2024-04-26T17:46:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:54 smithi033 ceph-mon[27845]: pgmap v11824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21354.879511 secs 2024-04-26T17:46:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21352.021988 secs 2024-04-26T17:46:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:54 smithi096 ceph-mon[32753]: pgmap v11824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21354.879511 secs 2024-04-26T17:46:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21352.021988 secs 2024-04-26T17:46:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:54 smithi194 ceph-mon[33017]: pgmap v11824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21354.879511 secs 2024-04-26T17:46:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:56 smithi033 ceph-mon[27845]: pgmap v11825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:56 smithi096 ceph-mon[32753]: pgmap v11825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:56 smithi194 ceph-mon[33017]: pgmap v11825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:46:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:58 smithi033 ceph-mon[27845]: pgmap v11826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:58 smithi096 ceph-mon[32753]: pgmap v11826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:58 smithi194 ceph-mon[33017]: pgmap v11826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:46:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21357.022049 secs 2024-04-26T17:46:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:46:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21359.879624 secs 2024-04-26T17:46:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21357.022049 secs 2024-04-26T17:46:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:46:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21359.879624 secs 2024-04-26T17:46:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21357.022049 secs 2024-04-26T17:46:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:46:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21359.879624 secs 2024-04-26T17:47:00.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:00 smithi033 ceph-mon[27845]: pgmap v11827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:00 smithi096 ceph-mon[32753]: pgmap v11827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:00 smithi194 ceph-mon[33017]: pgmap v11827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:02 smithi033 ceph-mon[27845]: pgmap v11828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:02 smithi096 ceph-mon[32753]: pgmap v11828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:02 smithi194 ceph-mon[33017]: pgmap v11828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:03.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:03.470+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:03.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:03.470+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:03.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:03.470+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:47:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21362.022129 secs 2024-04-26T17:47:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:04 smithi033 ceph-mon[27845]: pgmap v11829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21364.879745 secs 2024-04-26T17:47:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21362.022129 secs 2024-04-26T17:47:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:04 smithi096 ceph-mon[32753]: pgmap v11829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21364.879745 secs 2024-04-26T17:47:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21362.022129 secs 2024-04-26T17:47:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:04 smithi194 ceph-mon[33017]: pgmap v11829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21364.879745 secs 2024-04-26T17:47:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:06 smithi033 ceph-mon[27845]: pgmap v11830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:06 smithi096 ceph-mon[32753]: pgmap v11830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:06.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:06 smithi194 ceph-mon[33017]: pgmap v11830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:08 smithi033 ceph-mon[27845]: pgmap v11831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:08 smithi096 ceph-mon[32753]: pgmap v11831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:08 smithi194 ceph-mon[33017]: pgmap v11831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21367.022189 secs 2024-04-26T17:47:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21369.879882 secs 2024-04-26T17:47:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21367.022189 secs 2024-04-26T17:47:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21369.879882 secs 2024-04-26T17:47:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21367.022189 secs 2024-04-26T17:47:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21369.879882 secs 2024-04-26T17:47:10.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:10 smithi194 ceph-mon[33017]: pgmap v11832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:10 smithi033 ceph-mon[27845]: pgmap v11832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:10 smithi096 ceph-mon[32753]: pgmap v11832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:12 smithi194 ceph-mon[33017]: pgmap v11833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:12.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:12 smithi033 ceph-mon[27845]: pgmap v11833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:12 smithi096 ceph-mon[32753]: pgmap v11833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21372.022251 secs 2024-04-26T17:47:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:14 smithi194 ceph-mon[33017]: pgmap v11834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21374.879975 secs 2024-04-26T17:47:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21372.022251 secs 2024-04-26T17:47:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:14 smithi033 ceph-mon[27845]: pgmap v11834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21374.879975 secs 2024-04-26T17:47:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21372.022251 secs 2024-04-26T17:47:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:14 smithi096 ceph-mon[32753]: pgmap v11834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21374.879975 secs 2024-04-26T17:47:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:16 smithi194 ceph-mon[33017]: pgmap v11835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:16 smithi033 ceph-mon[27845]: pgmap v11835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:16 smithi096 ceph-mon[32753]: pgmap v11835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:18.480+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:18.480+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:18.480+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:47:18.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:18 smithi096 ceph-mon[32753]: pgmap v11836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:18 smithi194 ceph-mon[33017]: pgmap v11836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:18.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:18 smithi033 ceph-mon[27845]: pgmap v11836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21377.022311 secs 2024-04-26T17:47:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21379.880088 secs 2024-04-26T17:47:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21377.022311 secs 2024-04-26T17:47:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21379.880088 secs 2024-04-26T17:47:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21377.022311 secs 2024-04-26T17:47:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21379.880088 secs 2024-04-26T17:47:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:20 smithi194 ceph-mon[33017]: pgmap v11837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:20.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:20 smithi033 ceph-mon[27845]: pgmap v11837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:20 smithi096 ceph-mon[32753]: pgmap v11837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T17:47:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:22 smithi194 ceph-mon[33017]: pgmap v11838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:22.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:22 smithi033 ceph-mon[27845]: pgmap v11838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:22 smithi096 ceph-mon[32753]: pgmap v11838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21382.022370 secs 2024-04-26T17:47:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:24 smithi194 ceph-mon[33017]: pgmap v11839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21384.880208 secs 2024-04-26T17:47:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21382.022370 secs 2024-04-26T17:47:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:24 smithi033 ceph-mon[27845]: pgmap v11839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21384.880208 secs 2024-04-26T17:47:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21382.022370 secs 2024-04-26T17:47:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:24 smithi096 ceph-mon[32753]: pgmap v11839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21384.880208 secs 2024-04-26T17:47:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:25.432+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:47:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:25.432+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T17:47:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:25.432+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 171 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:47:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:25.433+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 171 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x7fef00077780 2024-04-26T17:47:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:26 smithi194 ceph-mon[33017]: pgmap v11840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:26 smithi033 ceph-mon[27845]: pgmap v11840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:26 smithi096 ceph-mon[32753]: pgmap v11840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:47:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:47:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:47:28.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:28 smithi033 ceph-mon[27845]: pgmap v11841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:28.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:28 smithi033 ceph-mon[27845]: pgmap v11842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:28 smithi194 ceph-mon[33017]: pgmap v11841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:28 smithi194 ceph-mon[33017]: pgmap v11842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:28 smithi096 ceph-mon[32753]: pgmap v11841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:28 smithi096 ceph-mon[32753]: pgmap v11842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21387.022433 secs 2024-04-26T17:47:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21389.880320 secs 2024-04-26T17:47:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21387.022433 secs 2024-04-26T17:47:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21389.880320 secs 2024-04-26T17:47:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21387.022433 secs 2024-04-26T17:47:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21389.880320 secs 2024-04-26T17:47:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:30 smithi194 ceph-mon[33017]: pgmap v11843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:47:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:30 smithi033 ceph-mon[27845]: pgmap v11843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:47:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:30 smithi096 ceph-mon[32753]: pgmap v11843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:47:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:32 smithi194 ceph-mon[33017]: pgmap v11844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:32 smithi033 ceph-mon[27845]: pgmap v11844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:32 smithi096 ceph-mon[32753]: pgmap v11844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:33.496+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:33.496+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:33.496+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:47:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21392.022517 secs 2024-04-26T17:47:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:34 smithi194 ceph-mon[33017]: pgmap v11845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21394.880437 secs 2024-04-26T17:47:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21392.022517 secs 2024-04-26T17:47:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:34 smithi033 ceph-mon[27845]: pgmap v11845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21394.880437 secs 2024-04-26T17:47:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21392.022517 secs 2024-04-26T17:47:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:34 smithi096 ceph-mon[32753]: pgmap v11845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21394.880437 secs 2024-04-26T17:47:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:36 smithi194 ceph-mon[33017]: pgmap v11846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:36 smithi033 ceph-mon[27845]: pgmap v11846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:36 smithi096 ceph-mon[32753]: pgmap v11846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:47:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:38 smithi194 ceph-mon[33017]: pgmap v11847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:47:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:38 smithi033 ceph-mon[27845]: pgmap v11847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:47:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:38 smithi096 ceph-mon[32753]: pgmap v11847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:47:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21397.022599 secs 2024-04-26T17:47:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21399.880567 secs 2024-04-26T17:47:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21397.022599 secs 2024-04-26T17:47:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21399.880567 secs 2024-04-26T17:47:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21397.022599 secs 2024-04-26T17:47:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21399.880567 secs 2024-04-26T17:47:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:40 smithi194 ceph-mon[33017]: pgmap v11848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:40 smithi033 ceph-mon[27845]: pgmap v11848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:40 smithi096 ceph-mon[32753]: pgmap v11848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:42 smithi194 ceph-mon[33017]: pgmap v11849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:42 smithi033 ceph-mon[27845]: pgmap v11849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:42 smithi096 ceph-mon[32753]: pgmap v11849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21402.022681 secs 2024-04-26T17:47:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:44 smithi194 ceph-mon[33017]: pgmap v11850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21404.880691 secs 2024-04-26T17:47:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21402.022681 secs 2024-04-26T17:47:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:44 smithi033 ceph-mon[27845]: pgmap v11850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21404.880691 secs 2024-04-26T17:47:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21402.022681 secs 2024-04-26T17:47:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:44 smithi096 ceph-mon[32753]: pgmap v11850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21404.880691 secs 2024-04-26T17:47:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:46 smithi194 ceph-mon[33017]: pgmap v11851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:46 smithi033 ceph-mon[27845]: pgmap v11851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:46 smithi096 ceph-mon[32753]: pgmap v11851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:48.512+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:48.512 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:48.512+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:47:48.512 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:47:48.512+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:47:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:48 smithi194 ceph-mon[33017]: pgmap v11852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:48 smithi033 ceph-mon[27845]: pgmap v11852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:48 smithi096 ceph-mon[32753]: pgmap v11852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21407.022764 secs 2024-04-26T17:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21409.880829 secs 2024-04-26T17:47:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21407.022764 secs 2024-04-26T17:47:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21409.880829 secs 2024-04-26T17:47:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21407.022764 secs 2024-04-26T17:47:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21409.880829 secs 2024-04-26T17:47:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:50 smithi194 ceph-mon[33017]: pgmap v11853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:50 smithi033 ceph-mon[27845]: pgmap v11853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:50 smithi096 ceph-mon[32753]: pgmap v11853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:52 smithi194 ceph-mon[33017]: pgmap v11854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:52 smithi033 ceph-mon[27845]: pgmap v11854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:52 smithi096 ceph-mon[32753]: pgmap v11854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21412.022852 secs 2024-04-26T17:47:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:54 smithi194 ceph-mon[33017]: pgmap v11855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21414.880969 secs 2024-04-26T17:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21412.022852 secs 2024-04-26T17:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:54 smithi033 ceph-mon[27845]: pgmap v11855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21414.880969 secs 2024-04-26T17:47:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21412.022852 secs 2024-04-26T17:47:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:54 smithi096 ceph-mon[32753]: pgmap v11855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21414.880969 secs 2024-04-26T17:47:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:56 smithi194 ceph-mon[33017]: pgmap v11856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:56 smithi033 ceph-mon[27845]: pgmap v11856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:56 smithi096 ceph-mon[32753]: pgmap v11856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:47:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:57 smithi033 ceph-mon[27845]: pgmap v11857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:57 smithi096 ceph-mon[32753]: pgmap v11857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:57 smithi194 ceph-mon[33017]: pgmap v11857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:47:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21417.022936 secs 2024-04-26T17:47:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21417.022936 secs 2024-04-26T17:47:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21417.022936 secs 2024-04-26T17:48:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21419.881124 secs 2024-04-26T17:48:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:47:59 smithi194 ceph-mon[33017]: pgmap v11858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21419.881124 secs 2024-04-26T17:48:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:47:59 smithi033 ceph-mon[27845]: pgmap v11858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21419.881124 secs 2024-04-26T17:48:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:47:59 smithi096 ceph-mon[32753]: pgmap v11858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:01 smithi194 ceph-mon[33017]: pgmap v11859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:01 smithi033 ceph-mon[27845]: pgmap v11859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:01 smithi096 ceph-mon[32753]: pgmap v11859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:03.527+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:03.527+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:48:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:03.527+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21422.023016 secs 2024-04-26T17:48:04.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:04 smithi194 ceph-mon[33017]: pgmap v11860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21422.023016 secs 2024-04-26T17:48:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:04 smithi033 ceph-mon[27845]: pgmap v11860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21422.023016 secs 2024-04-26T17:48:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:04 smithi096 ceph-mon[32753]: pgmap v11860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21424.881237 secs 2024-04-26T17:48:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21424.881237 secs 2024-04-26T17:48:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21424.881237 secs 2024-04-26T17:48:06.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:06 smithi033 ceph-mon[27845]: pgmap v11861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:06 smithi194 ceph-mon[33017]: pgmap v11861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:06 smithi096 ceph-mon[32753]: pgmap v11861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:08 smithi194 ceph-mon[33017]: pgmap v11862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:08 smithi033 ceph-mon[27845]: pgmap v11862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:08 smithi096 ceph-mon[32753]: pgmap v11862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:09.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21427.023075 secs 2024-04-26T17:48:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21427.023075 secs 2024-04-26T17:48:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21427.023075 secs 2024-04-26T17:48:10.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21429.881360 secs 2024-04-26T17:48:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:10 smithi194 ceph-mon[33017]: pgmap v11863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21429.881360 secs 2024-04-26T17:48:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:10 smithi033 ceph-mon[27845]: pgmap v11863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21429.881360 secs 2024-04-26T17:48:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:10 smithi096 ceph-mon[32753]: pgmap v11863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:48:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:12 smithi194 ceph-mon[33017]: pgmap v11864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:12 smithi033 ceph-mon[27845]: pgmap v11864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:12 smithi096 ceph-mon[32753]: pgmap v11864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:48:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21432.023134 secs 2024-04-26T17:48:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:14 smithi194 ceph-mon[33017]: pgmap v11865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21432.023134 secs 2024-04-26T17:48:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:14 smithi033 ceph-mon[27845]: pgmap v11865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21432.023134 secs 2024-04-26T17:48:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:14 smithi096 ceph-mon[32753]: pgmap v11865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21434.881480 secs 2024-04-26T17:48:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21434.881480 secs 2024-04-26T17:48:15.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21434.881480 secs 2024-04-26T17:48:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:16 smithi194 ceph-mon[33017]: pgmap v11866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:16.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:16 smithi033 ceph-mon[27845]: pgmap v11866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:16 smithi096 ceph-mon[32753]: pgmap v11866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:18 smithi194 ceph-mon[33017]: pgmap v11867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:18 smithi033 ceph-mon[27845]: pgmap v11867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:18 smithi096 ceph-mon[32753]: pgmap v11867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:18.531+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:18.531+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:18.531+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:48:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21437.023217 secs 2024-04-26T17:48:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21437.023217 secs 2024-04-26T17:48:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21437.023217 secs 2024-04-26T17:48:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21439.881590 secs 2024-04-26T17:48:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:20 smithi194 ceph-mon[33017]: pgmap v11868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21439.881590 secs 2024-04-26T17:48:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:20 smithi033 ceph-mon[27845]: pgmap v11868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21439.881590 secs 2024-04-26T17:48:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:20 smithi096 ceph-mon[32753]: pgmap v11868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:22 smithi194 ceph-mon[33017]: pgmap v11869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:22 smithi033 ceph-mon[27845]: pgmap v11869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:22 smithi096 ceph-mon[32753]: pgmap v11869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21442.023299 secs 2024-04-26T17:48:24.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:24 smithi194 ceph-mon[33017]: pgmap v11870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21442.023299 secs 2024-04-26T17:48:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:24 smithi033 ceph-mon[27845]: pgmap v11870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21442.023299 secs 2024-04-26T17:48:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:24 smithi096 ceph-mon[32753]: pgmap v11870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:25.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21444.881711 secs 2024-04-26T17:48:25.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21444.881711 secs 2024-04-26T17:48:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21444.881711 secs 2024-04-26T17:48:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:26 smithi194 ceph-mon[33017]: pgmap v11871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:26.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:26 smithi033 ceph-mon[27845]: pgmap v11871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:26 smithi096 ceph-mon[32753]: pgmap v11871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:28 smithi033 ceph-mon[27845]: pgmap v11872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:28 smithi096 ceph-mon[32753]: pgmap v11872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:28 smithi194 ceph-mon[33017]: pgmap v11872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21447.023362 secs 2024-04-26T17:48:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:48:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21447.023362 secs 2024-04-26T17:48:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:48:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21447.023362 secs 2024-04-26T17:48:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:48:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21449.881826 secs 2024-04-26T17:48:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:30 smithi033 ceph-mon[27845]: pgmap v11873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:30 smithi033 ceph-mon[27845]: pgmap v11874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21449.881826 secs 2024-04-26T17:48:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:30 smithi096 ceph-mon[32753]: pgmap v11873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:30 smithi096 ceph-mon[32753]: pgmap v11874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21449.881826 secs 2024-04-26T17:48:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:30 smithi194 ceph-mon[33017]: pgmap v11873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:30 smithi194 ceph-mon[33017]: pgmap v11874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:32 smithi033 ceph-mon[27845]: pgmap v11875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:32 smithi096 ceph-mon[32753]: pgmap v11875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:32 smithi194 ceph-mon[33017]: pgmap v11875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:33.539 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:33.539+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:33.539 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:33.539+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:33.540 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:33.539+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:48:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:34 smithi033 ceph-mon[27845]: pgmap v11876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21452.023419 secs 2024-04-26T17:48:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:34 smithi096 ceph-mon[32753]: pgmap v11876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21452.023419 secs 2024-04-26T17:48:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:34 smithi194 ceph-mon[33017]: pgmap v11876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21452.023419 secs 2024-04-26T17:48:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21454.881943 secs 2024-04-26T17:48:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21454.881943 secs 2024-04-26T17:48:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21454.881943 secs 2024-04-26T17:48:36.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:36 smithi033 ceph-mon[27845]: pgmap v11877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:36 smithi096 ceph-mon[32753]: pgmap v11877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:36 smithi194 ceph-mon[33017]: pgmap v11877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:38 smithi033 ceph-mon[27845]: pgmap v11878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:38 smithi096 ceph-mon[32753]: pgmap v11878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:38 smithi194 ceph-mon[33017]: pgmap v11878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:48:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21457.023482 secs 2024-04-26T17:48:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21457.023482 secs 2024-04-26T17:48:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21457.023482 secs 2024-04-26T17:48:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21459.882014 secs 2024-04-26T17:48:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:40 smithi033 ceph-mon[27845]: pgmap v11879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:48:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21459.882014 secs 2024-04-26T17:48:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:40 smithi096 ceph-mon[32753]: pgmap v11879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:48:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21459.882014 secs 2024-04-26T17:48:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:40 smithi194 ceph-mon[33017]: pgmap v11879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:48:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:42 smithi033 ceph-mon[27845]: pgmap v11880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:42 smithi096 ceph-mon[32753]: pgmap v11880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:42.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:42 smithi194 ceph-mon[33017]: pgmap v11880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:44 smithi033 ceph-mon[27845]: pgmap v11881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21462.023562 secs 2024-04-26T17:48:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:44 smithi096 ceph-mon[32753]: pgmap v11881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21462.023562 secs 2024-04-26T17:48:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:44 smithi194 ceph-mon[33017]: pgmap v11881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21462.023562 secs 2024-04-26T17:48:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21464.882143 secs 2024-04-26T17:48:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21464.882143 secs 2024-04-26T17:48:45.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21464.882143 secs 2024-04-26T17:48:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:46 smithi194 ceph-mon[33017]: pgmap v11882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:46.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:46 smithi033 ceph-mon[27845]: pgmap v11882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:46 smithi096 ceph-mon[32753]: pgmap v11882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:48.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:48 smithi096 ceph-mon[32753]: pgmap v11883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:48.542+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:48.542+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:48:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:48:48.542+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:48:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:48 smithi194 ceph-mon[33017]: pgmap v11883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:48 smithi033 ceph-mon[27845]: pgmap v11883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21467.023646 secs 2024-04-26T17:48:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21467.023646 secs 2024-04-26T17:48:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21467.023646 secs 2024-04-26T17:48:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21469.882269 secs 2024-04-26T17:48:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:50 smithi194 ceph-mon[33017]: pgmap v11884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21469.882269 secs 2024-04-26T17:48:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:50 smithi033 ceph-mon[27845]: pgmap v11884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21469.882269 secs 2024-04-26T17:48:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:50 smithi096 ceph-mon[32753]: pgmap v11884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:52.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:52 smithi194 ceph-mon[33017]: pgmap v11885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:52 smithi033 ceph-mon[27845]: pgmap v11885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:52 smithi096 ceph-mon[32753]: pgmap v11885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:54 smithi194 ceph-mon[33017]: pgmap v11886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21472.023726 secs 2024-04-26T17:48:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:54 smithi033 ceph-mon[27845]: pgmap v11886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21472.023726 secs 2024-04-26T17:48:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:54 smithi096 ceph-mon[32753]: pgmap v11886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21472.023726 secs 2024-04-26T17:48:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21474.882388 secs 2024-04-26T17:48:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21474.882388 secs 2024-04-26T17:48:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21474.882388 secs 2024-04-26T17:48:56.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:56 smithi194 ceph-mon[33017]: pgmap v11887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:56 smithi033 ceph-mon[27845]: pgmap v11887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:56 smithi096 ceph-mon[32753]: pgmap v11887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:48:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:58 smithi194 ceph-mon[33017]: pgmap v11888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:58 smithi033 ceph-mon[27845]: pgmap v11888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:58 smithi096 ceph-mon[32753]: pgmap v11888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:48:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:48:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21477.023810 secs 2024-04-26T17:48:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:48:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21477.023810 secs 2024-04-26T17:48:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:48:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21477.023810 secs 2024-04-26T17:49:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21479.882462 secs 2024-04-26T17:49:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:00 smithi194 ceph-mon[33017]: pgmap v11889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21479.882462 secs 2024-04-26T17:49:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:00 smithi033 ceph-mon[27845]: pgmap v11889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21479.882462 secs 2024-04-26T17:49:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:00 smithi096 ceph-mon[32753]: pgmap v11889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:02.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:02 smithi194 ceph-mon[33017]: pgmap v11890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:02 smithi033 ceph-mon[27845]: pgmap v11890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:02 smithi096 ceph-mon[32753]: pgmap v11890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:03.543+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:03.543+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:03.543+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:49:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:04 smithi194 ceph-mon[33017]: pgmap v11891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:04.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21482.023892 secs 2024-04-26T17:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:04 smithi033 ceph-mon[27845]: pgmap v11891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21482.023892 secs 2024-04-26T17:49:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:04 smithi096 ceph-mon[32753]: pgmap v11891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21482.023892 secs 2024-04-26T17:49:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21484.882579 secs 2024-04-26T17:49:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21484.882579 secs 2024-04-26T17:49:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21484.882579 secs 2024-04-26T17:49:06.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:06 smithi194 ceph-mon[33017]: pgmap v11892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:06 smithi033 ceph-mon[27845]: pgmap v11892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:06 smithi096 ceph-mon[32753]: pgmap v11892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:08 smithi194 ceph-mon[33017]: pgmap v11893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:08 smithi033 ceph-mon[27845]: pgmap v11893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:08 smithi096 ceph-mon[32753]: pgmap v11893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21487.023998 secs 2024-04-26T17:49:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21487.023998 secs 2024-04-26T17:49:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21487.023998 secs 2024-04-26T17:49:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21489.882714 secs 2024-04-26T17:49:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:10 smithi194 ceph-mon[33017]: pgmap v11894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21489.882714 secs 2024-04-26T17:49:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:10 smithi033 ceph-mon[27845]: pgmap v11894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21489.882714 secs 2024-04-26T17:49:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:10 smithi096 ceph-mon[32753]: pgmap v11894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:12 smithi194 ceph-mon[33017]: pgmap v11895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:12.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:12 smithi033 ceph-mon[27845]: pgmap v11895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:12 smithi096 ceph-mon[32753]: pgmap v11895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:14 smithi194 ceph-mon[33017]: pgmap v11896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21492.024057 secs 2024-04-26T17:49:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:14 smithi033 ceph-mon[27845]: pgmap v11896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21492.024057 secs 2024-04-26T17:49:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:14 smithi096 ceph-mon[32753]: pgmap v11896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21492.024057 secs 2024-04-26T17:49:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21494.882849 secs 2024-04-26T17:49:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21494.882849 secs 2024-04-26T17:49:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21494.882849 secs 2024-04-26T17:49:16.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:16 smithi194 ceph-mon[33017]: pgmap v11897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:16 smithi033 ceph-mon[27845]: pgmap v11897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:16 smithi096 ceph-mon[32753]: pgmap v11897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:18.552+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:18.552+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:18.552+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:49:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:18 smithi194 ceph-mon[33017]: pgmap v11898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:18 smithi033 ceph-mon[27845]: pgmap v11898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:18 smithi096 ceph-mon[32753]: pgmap v11898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21497.024119 secs 2024-04-26T17:49:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21499.882910 secs 2024-04-26T17:49:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21497.024119 secs 2024-04-26T17:49:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21499.882910 secs 2024-04-26T17:49:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21497.024119 secs 2024-04-26T17:49:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21499.882910 secs 2024-04-26T17:49:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:20 smithi194 ceph-mon[33017]: pgmap v11899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:20 smithi033 ceph-mon[27845]: pgmap v11899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:20 smithi096 ceph-mon[32753]: pgmap v11899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:22 smithi194 ceph-mon[33017]: pgmap v11900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:22.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:22 smithi033 ceph-mon[27845]: pgmap v11900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:22 smithi096 ceph-mon[32753]: pgmap v11900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:24 smithi033 ceph-mon[27845]: pgmap v11901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21502.024195 secs 2024-04-26T17:49:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21504.883032 secs 2024-04-26T17:49:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:24 smithi194 ceph-mon[33017]: pgmap v11901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21502.024195 secs 2024-04-26T17:49:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21504.883032 secs 2024-04-26T17:49:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:24 smithi096 ceph-mon[32753]: pgmap v11901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21502.024195 secs 2024-04-26T17:49:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21504.883032 secs 2024-04-26T17:49:26.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:26 smithi194 ceph-mon[33017]: pgmap v11902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:26 smithi033 ceph-mon[27845]: pgmap v11902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:26 smithi096 ceph-mon[32753]: pgmap v11902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:28 smithi033 ceph-mon[27845]: pgmap v11903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:28 smithi096 ceph-mon[32753]: pgmap v11903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:28 smithi194 ceph-mon[33017]: pgmap v11903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21507.024256 secs 2024-04-26T17:49:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21509.883150 secs 2024-04-26T17:49:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21507.024256 secs 2024-04-26T17:49:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21509.883150 secs 2024-04-26T17:49:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21507.024256 secs 2024-04-26T17:49:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21509.883150 secs 2024-04-26T17:49:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:30 smithi033 ceph-mon[27845]: pgmap v11904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:30 smithi096 ceph-mon[32753]: pgmap v11904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:30 smithi194 ceph-mon[33017]: pgmap v11904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:31 smithi033 ceph-mon[27845]: pgmap v11905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:49:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:31 smithi033 ceph-mon[27845]: pgmap v11906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:49:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:31 smithi096 ceph-mon[32753]: pgmap v11905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:49:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:31 smithi096 ceph-mon[32753]: pgmap v11906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:49:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:31 smithi194 ceph-mon[33017]: pgmap v11905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:49:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:31 smithi194 ceph-mon[33017]: pgmap v11906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:49:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:33.557+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:33.557+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:33.557+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:49:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:33 smithi194 ceph-mon[33017]: pgmap v11907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:33 smithi033 ceph-mon[27845]: pgmap v11907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:33 smithi096 ceph-mon[32753]: pgmap v11907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21512.024317 secs 2024-04-26T17:49:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21514.883226 secs 2024-04-26T17:49:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21512.024317 secs 2024-04-26T17:49:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21514.883226 secs 2024-04-26T17:49:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21512.024317 secs 2024-04-26T17:49:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21514.883226 secs 2024-04-26T17:49:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:35 smithi194 ceph-mon[33017]: pgmap v11908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:35.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:35 smithi033 ceph-mon[27845]: pgmap v11908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:35 smithi096 ceph-mon[32753]: pgmap v11908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:37 smithi194 ceph-mon[33017]: pgmap v11909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:37 smithi033 ceph-mon[27845]: pgmap v11909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:37 smithi096 ceph-mon[32753]: pgmap v11909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21517.024398 secs 2024-04-26T17:49:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:39 smithi194 ceph-mon[33017]: pgmap v11910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21519.883289 secs 2024-04-26T17:49:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21517.024398 secs 2024-04-26T17:49:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:39 smithi033 ceph-mon[27845]: pgmap v11910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21519.883289 secs 2024-04-26T17:49:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21517.024398 secs 2024-04-26T17:49:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:39 smithi096 ceph-mon[32753]: pgmap v11910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:49:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21519.883289 secs 2024-04-26T17:49:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:41 smithi194 ceph-mon[33017]: pgmap v11911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:49:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:41 smithi033 ceph-mon[27845]: pgmap v11911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:49:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:41 smithi096 ceph-mon[32753]: pgmap v11911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:49:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:43 smithi194 ceph-mon[33017]: pgmap v11912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:43 smithi033 ceph-mon[27845]: pgmap v11912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:43 smithi096 ceph-mon[32753]: pgmap v11912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21522.024481 secs 2024-04-26T17:49:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21524.883369 secs 2024-04-26T17:49:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21522.024481 secs 2024-04-26T17:49:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21524.883369 secs 2024-04-26T17:49:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21522.024481 secs 2024-04-26T17:49:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21524.883369 secs 2024-04-26T17:49:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:45 smithi194 ceph-mon[33017]: pgmap v11913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:45 smithi033 ceph-mon[27845]: pgmap v11913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:45 smithi096 ceph-mon[32753]: pgmap v11913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:47 smithi194 ceph-mon[33017]: pgmap v11914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:47 smithi033 ceph-mon[27845]: pgmap v11914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:47 smithi096 ceph-mon[32753]: pgmap v11914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:48.571+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:48.571+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:49:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:49:48.571+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21527.024563 secs 2024-04-26T17:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:49 smithi194 ceph-mon[33017]: pgmap v11915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21529.883459 secs 2024-04-26T17:49:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21527.024563 secs 2024-04-26T17:49:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:49 smithi033 ceph-mon[27845]: pgmap v11915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21529.883459 secs 2024-04-26T17:49:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21527.024563 secs 2024-04-26T17:49:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:49 smithi096 ceph-mon[32753]: pgmap v11915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21529.883459 secs 2024-04-26T17:49:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:51 smithi194 ceph-mon[33017]: pgmap v11916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:51 smithi033 ceph-mon[27845]: pgmap v11916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:51 smithi096 ceph-mon[32753]: pgmap v11916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:49:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:53 smithi194 ceph-mon[33017]: pgmap v11917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:53 smithi033 ceph-mon[27845]: pgmap v11917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:53 smithi096 ceph-mon[32753]: pgmap v11917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:49:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21532.024646 secs 2024-04-26T17:49:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21534.883522 secs 2024-04-26T17:49:54.863 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21532.024646 secs 2024-04-26T17:49:54.863 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21534.883522 secs 2024-04-26T17:49:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21532.024646 secs 2024-04-26T17:49:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21534.883522 secs 2024-04-26T17:49:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:55 smithi194 ceph-mon[33017]: pgmap v11918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:49:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:55 smithi033 ceph-mon[27845]: pgmap v11918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:49:55.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:55 smithi096 ceph-mon[32753]: pgmap v11918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:49:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:57 smithi194 ceph-mon[33017]: pgmap v11919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:49:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:57 smithi033 ceph-mon[27845]: pgmap v11919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:49:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:57 smithi096 ceph-mon[32753]: pgmap v11919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:49:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21537.024727 secs 2024-04-26T17:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:59 smithi194 ceph-mon[33017]: pgmap v11920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21539.883602 secs 2024-04-26T17:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21537.024727 secs 2024-04-26T17:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:59 smithi033 ceph-mon[27845]: pgmap v11920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21539.883602 secs 2024-04-26T17:49:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21537.024727 secs 2024-04-26T17:49:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:59 smithi096 ceph-mon[32753]: pgmap v11920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:49:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21539.883602 secs 2024-04-26T17:50:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:50:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:50:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T17:50:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:01 smithi194 ceph-mon[33017]: pgmap v11921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:50:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:01 smithi033 ceph-mon[27845]: pgmap v11921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:50:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:01 smithi096 ceph-mon[32753]: pgmap v11921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:50:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:03.578+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:03.578+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:50:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:03.578+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:03 smithi194 ceph-mon[33017]: pgmap v11922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:50:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:03 smithi033 ceph-mon[27845]: pgmap v11922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:50:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:03 smithi096 ceph-mon[32753]: pgmap v11922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:50:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21542.024810 secs 2024-04-26T17:50:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21544.883738 secs 2024-04-26T17:50:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21542.024810 secs 2024-04-26T17:50:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21544.883738 secs 2024-04-26T17:50:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21542.024810 secs 2024-04-26T17:50:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21544.883738 secs 2024-04-26T17:50:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:05 smithi194 ceph-mon[33017]: pgmap v11923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:50:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:05 smithi033 ceph-mon[27845]: pgmap v11923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:50:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:05 smithi096 ceph-mon[32753]: pgmap v11923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:50:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:07 smithi194 ceph-mon[33017]: pgmap v11924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:07 smithi033 ceph-mon[27845]: pgmap v11924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:07 smithi096 ceph-mon[32753]: pgmap v11924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21547.024874 secs 2024-04-26T17:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:09 smithi194 ceph-mon[33017]: pgmap v11925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21549.883859 secs 2024-04-26T17:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21547.024874 secs 2024-04-26T17:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:09 smithi033 ceph-mon[27845]: pgmap v11925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21549.883859 secs 2024-04-26T17:50:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21547.024874 secs 2024-04-26T17:50:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:09 smithi096 ceph-mon[32753]: pgmap v11925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21549.883859 secs 2024-04-26T17:50:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:11 smithi194 ceph-mon[33017]: pgmap v11926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:11 smithi033 ceph-mon[27845]: pgmap v11926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:11 smithi096 ceph-mon[32753]: pgmap v11926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:13 smithi194 ceph-mon[33017]: pgmap v11927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:13 smithi033 ceph-mon[27845]: pgmap v11927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:13 smithi096 ceph-mon[32753]: pgmap v11927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21552.024955 secs 2024-04-26T17:50:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21554.883977 secs 2024-04-26T17:50:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21552.024955 secs 2024-04-26T17:50:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21554.883977 secs 2024-04-26T17:50:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21552.024955 secs 2024-04-26T17:50:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21554.883977 secs 2024-04-26T17:50:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:15 smithi194 ceph-mon[33017]: pgmap v11928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:15 smithi033 ceph-mon[27845]: pgmap v11928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:15 smithi096 ceph-mon[32753]: pgmap v11928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:17 smithi194 ceph-mon[33017]: pgmap v11929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:17 smithi033 ceph-mon[27845]: pgmap v11929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:17 smithi096 ceph-mon[32753]: pgmap v11929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:18.591+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:18.591+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:18.591+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:50:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21557.025005 secs 2024-04-26T17:50:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:19 smithi033 ceph-mon[27845]: pgmap v11930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21559.884099 secs 2024-04-26T17:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21557.025005 secs 2024-04-26T17:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:19 smithi096 ceph-mon[32753]: pgmap v11930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21559.884099 secs 2024-04-26T17:50:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21557.025005 secs 2024-04-26T17:50:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:19 smithi194 ceph-mon[33017]: pgmap v11930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21559.884099 secs 2024-04-26T17:50:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:21 smithi033 ceph-mon[27845]: pgmap v11931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:21 smithi096 ceph-mon[32753]: pgmap v11931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:21 smithi194 ceph-mon[33017]: pgmap v11931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:23 smithi033 ceph-mon[27845]: pgmap v11932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:23 smithi096 ceph-mon[32753]: pgmap v11932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:23 smithi194 ceph-mon[33017]: pgmap v11932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21562.025088 secs 2024-04-26T17:50:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21564.884222 secs 2024-04-26T17:50:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21562.025088 secs 2024-04-26T17:50:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21564.884222 secs 2024-04-26T17:50:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21562.025088 secs 2024-04-26T17:50:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21564.884222 secs 2024-04-26T17:50:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:25.463+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:50:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:25.463+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T17:50:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:25.463+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 172 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:50:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:25.463+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 172 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:50:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:25 smithi033 ceph-mon[27845]: pgmap v11933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:25 smithi096 ceph-mon[32753]: pgmap v11933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:25 smithi194 ceph-mon[33017]: pgmap v11933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:26 smithi033 ceph-mon[27845]: pgmap v11934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:26 smithi096 ceph-mon[32753]: pgmap v11934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:26 smithi194 ceph-mon[33017]: pgmap v11934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21567.025150 secs 2024-04-26T17:50:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:29 smithi194 ceph-mon[33017]: pgmap v11935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21569.884304 secs 2024-04-26T17:50:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21567.025150 secs 2024-04-26T17:50:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:29 smithi033 ceph-mon[27845]: pgmap v11935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21569.884304 secs 2024-04-26T17:50:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21567.025150 secs 2024-04-26T17:50:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:29 smithi096 ceph-mon[32753]: pgmap v11935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21569.884304 secs 2024-04-26T17:50:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:31 smithi194 ceph-mon[33017]: pgmap v11936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:31.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:31 smithi033 ceph-mon[27845]: pgmap v11936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:31 smithi096 ceph-mon[32753]: pgmap v11936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:50:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:50:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:50:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:33.607+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:33.607+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:50:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:33.607+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:33 smithi194 ceph-mon[33017]: pgmap v11937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:33 smithi194 ceph-mon[33017]: pgmap v11938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:33 smithi033 ceph-mon[27845]: pgmap v11937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:33 smithi033 ceph-mon[27845]: pgmap v11938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:33 smithi096 ceph-mon[32753]: pgmap v11937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:33 smithi096 ceph-mon[32753]: pgmap v11938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21572.025213 secs 2024-04-26T17:50:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21574.884441 secs 2024-04-26T17:50:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21572.025213 secs 2024-04-26T17:50:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21574.884441 secs 2024-04-26T17:50:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21572.025213 secs 2024-04-26T17:50:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21574.884441 secs 2024-04-26T17:50:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:35 smithi194 ceph-mon[33017]: pgmap v11939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:50:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:35 smithi033 ceph-mon[27845]: pgmap v11939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:50:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:35 smithi096 ceph-mon[32753]: pgmap v11939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T17:50:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:37 smithi194 ceph-mon[33017]: pgmap v11940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:37.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:37 smithi033 ceph-mon[27845]: pgmap v11940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:37 smithi096 ceph-mon[32753]: pgmap v11940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21577.025280 secs 2024-04-26T17:50:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:39 smithi194 ceph-mon[33017]: pgmap v11941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21579.884558 secs 2024-04-26T17:50:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21577.025280 secs 2024-04-26T17:50:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:39 smithi033 ceph-mon[27845]: pgmap v11941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21579.884558 secs 2024-04-26T17:50:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21577.025280 secs 2024-04-26T17:50:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:39 smithi096 ceph-mon[32753]: pgmap v11941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21579.884558 secs 2024-04-26T17:50:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:41 smithi194 ceph-mon[33017]: pgmap v11942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:41 smithi033 ceph-mon[27845]: pgmap v11942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:41 smithi096 ceph-mon[32753]: pgmap v11942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:50:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:43 smithi194 ceph-mon[33017]: pgmap v11943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:50:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:43 smithi033 ceph-mon[27845]: pgmap v11943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:50:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:43 smithi096 ceph-mon[32753]: pgmap v11943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:50:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21582.025340 secs 2024-04-26T17:50:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21584.884681 secs 2024-04-26T17:50:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21582.025340 secs 2024-04-26T17:50:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21584.884681 secs 2024-04-26T17:50:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21582.025340 secs 2024-04-26T17:50:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21584.884681 secs 2024-04-26T17:50:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:45 smithi194 ceph-mon[33017]: pgmap v11944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:45 smithi033 ceph-mon[27845]: pgmap v11944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:45 smithi096 ceph-mon[32753]: pgmap v11944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:47 smithi194 ceph-mon[33017]: pgmap v11945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:47 smithi033 ceph-mon[27845]: pgmap v11945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:47 smithi096 ceph-mon[32753]: pgmap v11945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:48.611+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:48.612+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:50:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:50:48.612+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:50:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21587.025421 secs 2024-04-26T17:50:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:49 smithi194 ceph-mon[33017]: pgmap v11946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21589.884796 secs 2024-04-26T17:50:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21587.025421 secs 2024-04-26T17:50:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:49 smithi033 ceph-mon[27845]: pgmap v11946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21589.884796 secs 2024-04-26T17:50:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21587.025421 secs 2024-04-26T17:50:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:49 smithi096 ceph-mon[32753]: pgmap v11946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21589.884796 secs 2024-04-26T17:50:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:51 smithi194 ceph-mon[33017]: pgmap v11947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:51 smithi033 ceph-mon[27845]: pgmap v11947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:51 smithi096 ceph-mon[32753]: pgmap v11947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:53 smithi194 ceph-mon[33017]: pgmap v11948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:53 smithi033 ceph-mon[27845]: pgmap v11948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:53 smithi096 ceph-mon[32753]: pgmap v11948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21592.025482 secs 2024-04-26T17:50:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21594.884961 secs 2024-04-26T17:50:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21592.025482 secs 2024-04-26T17:50:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21594.884961 secs 2024-04-26T17:50:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21592.025482 secs 2024-04-26T17:50:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21594.884961 secs 2024-04-26T17:50:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:55 smithi194 ceph-mon[33017]: pgmap v11949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:55 smithi033 ceph-mon[27845]: pgmap v11949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:55 smithi096 ceph-mon[32753]: pgmap v11949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:57 smithi033 ceph-mon[27845]: pgmap v11950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:57 smithi194 ceph-mon[33017]: pgmap v11950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:57 smithi096 ceph-mon[32753]: pgmap v11950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:50:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21597.025565 secs 2024-04-26T17:50:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:59 smithi194 ceph-mon[33017]: pgmap v11951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21599.885099 secs 2024-04-26T17:50:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21597.025565 secs 2024-04-26T17:50:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:59 smithi033 ceph-mon[27845]: pgmap v11951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21599.885099 secs 2024-04-26T17:50:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21597.025565 secs 2024-04-26T17:50:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:59 smithi096 ceph-mon[32753]: pgmap v11951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:50:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21599.885099 secs 2024-04-26T17:51:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:01 smithi194 ceph-mon[33017]: pgmap v11952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:01 smithi033 ceph-mon[27845]: pgmap v11952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:01 smithi096 ceph-mon[32753]: pgmap v11952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:03.626+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:03.626+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:51:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:03.626+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:03 smithi194 ceph-mon[33017]: pgmap v11953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:03 smithi033 ceph-mon[27845]: pgmap v11953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:03 smithi096 ceph-mon[32753]: pgmap v11953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21602.025648 secs 2024-04-26T17:51:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21604.885215 secs 2024-04-26T17:51:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21602.025648 secs 2024-04-26T17:51:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21604.885215 secs 2024-04-26T17:51:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21602.025648 secs 2024-04-26T17:51:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21604.885215 secs 2024-04-26T17:51:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:05 smithi194 ceph-mon[33017]: pgmap v11954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:05 smithi033 ceph-mon[27845]: pgmap v11954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:05 smithi096 ceph-mon[32753]: pgmap v11954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:07 smithi033 ceph-mon[27845]: pgmap v11955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:07 smithi096 ceph-mon[32753]: pgmap v11955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:07 smithi194 ceph-mon[33017]: pgmap v11955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21607.025731 secs 2024-04-26T17:51:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:09 smithi033 ceph-mon[27845]: pgmap v11956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21609.885324 secs 2024-04-26T17:51:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21607.025731 secs 2024-04-26T17:51:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:09 smithi096 ceph-mon[32753]: pgmap v11956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21609.885324 secs 2024-04-26T17:51:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21607.025731 secs 2024-04-26T17:51:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:09 smithi194 ceph-mon[33017]: pgmap v11956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21609.885324 secs 2024-04-26T17:51:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:11 smithi033 ceph-mon[27845]: pgmap v11957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:11 smithi096 ceph-mon[32753]: pgmap v11957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:11 smithi194 ceph-mon[33017]: pgmap v11957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:13 smithi033 ceph-mon[27845]: pgmap v11958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:13 smithi096 ceph-mon[32753]: pgmap v11958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:13 smithi194 ceph-mon[33017]: pgmap v11958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:14.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21612.025815 secs 2024-04-26T17:51:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21614.885444 secs 2024-04-26T17:51:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21612.025815 secs 2024-04-26T17:51:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21614.885444 secs 2024-04-26T17:51:15.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21612.025815 secs 2024-04-26T17:51:15.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21614.885444 secs 2024-04-26T17:51:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:15 smithi033 ceph-mon[27845]: pgmap v11959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:15 smithi096 ceph-mon[32753]: pgmap v11959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:15 smithi194 ceph-mon[33017]: pgmap v11959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:17 smithi033 ceph-mon[27845]: pgmap v11960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:17 smithi096 ceph-mon[32753]: pgmap v11960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:17 smithi194 ceph-mon[33017]: pgmap v11960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:18.628+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:18.628+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:18.628+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:51:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:18 smithi194 ceph-mon[33017]: pgmap v11961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:18 smithi033 ceph-mon[27845]: pgmap v11961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:18 smithi096 ceph-mon[32753]: pgmap v11961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:20.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21617.025898 secs 2024-04-26T17:51:20.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21619.885522 secs 2024-04-26T17:51:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21617.025898 secs 2024-04-26T17:51:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21619.885522 secs 2024-04-26T17:51:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21617.025898 secs 2024-04-26T17:51:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21619.885522 secs 2024-04-26T17:51:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:20 smithi194 ceph-mon[33017]: pgmap v11962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:20 smithi033 ceph-mon[27845]: pgmap v11962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:20 smithi096 ceph-mon[32753]: pgmap v11962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:23 smithi033 ceph-mon[27845]: pgmap v11963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21622.025980 secs 2024-04-26T17:51:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:23 smithi096 ceph-mon[32753]: pgmap v11963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21622.025980 secs 2024-04-26T17:51:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:23 smithi194 ceph-mon[33017]: pgmap v11963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21622.025980 secs 2024-04-26T17:51:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21624.885639 secs 2024-04-26T17:51:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:25 smithi033 ceph-mon[27845]: pgmap v11964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21624.885639 secs 2024-04-26T17:51:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:25 smithi096 ceph-mon[32753]: pgmap v11964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21624.885639 secs 2024-04-26T17:51:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:25 smithi194 ceph-mon[33017]: pgmap v11964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:27.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:27 smithi033 ceph-mon[27845]: pgmap v11965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:27 smithi096 ceph-mon[32753]: pgmap v11965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:27 smithi194 ceph-mon[33017]: pgmap v11965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:28.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21627.026064 secs 2024-04-26T17:51:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21627.026064 secs 2024-04-26T17:51:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21627.026064 secs 2024-04-26T17:51:29.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:29 smithi033 ceph-mon[27845]: pgmap v11966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:29 smithi096 ceph-mon[32753]: pgmap v11966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:29 smithi194 ceph-mon[33017]: pgmap v11966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:30.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21629.885702 secs 2024-04-26T17:51:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21629.885702 secs 2024-04-26T17:51:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21629.885702 secs 2024-04-26T17:51:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:31 smithi033 ceph-mon[27845]: pgmap v11967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:31 smithi096 ceph-mon[32753]: pgmap v11967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:31 smithi194 ceph-mon[33017]: pgmap v11967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:33 smithi033 ceph-mon[27845]: pgmap v11968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21632.026136 secs 2024-04-26T17:51:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:33 smithi096 ceph-mon[32753]: pgmap v11968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21632.026136 secs 2024-04-26T17:51:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:33 smithi194 ceph-mon[33017]: pgmap v11968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21632.026136 secs 2024-04-26T17:51:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:33.645+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:33.645+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:33.645+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:51:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:51:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:51:34.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:51:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21634.885816 secs 2024-04-26T17:51:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:35 smithi096 ceph-mon[32753]: pgmap v11969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:35 smithi096 ceph-mon[32753]: pgmap v11970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21634.885816 secs 2024-04-26T17:51:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:35 smithi194 ceph-mon[33017]: pgmap v11969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:35 smithi194 ceph-mon[33017]: pgmap v11970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21634.885816 secs 2024-04-26T17:51:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:35 smithi033 ceph-mon[27845]: pgmap v11969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:35 smithi033 ceph-mon[27845]: pgmap v11970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:37 smithi194 ceph-mon[33017]: pgmap v11971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:37 smithi033 ceph-mon[27845]: pgmap v11971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:37 smithi096 ceph-mon[32753]: pgmap v11971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21637.026198 secs 2024-04-26T17:51:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21637.026198 secs 2024-04-26T17:51:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21637.026198 secs 2024-04-26T17:51:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:39 smithi194 ceph-mon[33017]: pgmap v11972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:39 smithi033 ceph-mon[27845]: pgmap v11972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:39 smithi096 ceph-mon[32753]: pgmap v11972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21639.885952 secs 2024-04-26T17:51:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21639.885952 secs 2024-04-26T17:51:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21639.885952 secs 2024-04-26T17:51:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:41 smithi194 ceph-mon[33017]: pgmap v11973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:41.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:41 smithi033 ceph-mon[27845]: pgmap v11973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:41 smithi096 ceph-mon[32753]: pgmap v11973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:43 smithi194 ceph-mon[33017]: pgmap v11974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21642.026269 secs 2024-04-26T17:51:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:43 smithi033 ceph-mon[27845]: pgmap v11974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21642.026269 secs 2024-04-26T17:51:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:43 smithi096 ceph-mon[32753]: pgmap v11974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:51:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21642.026269 secs 2024-04-26T17:51:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21644.886071 secs 2024-04-26T17:51:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:45 smithi194 ceph-mon[33017]: pgmap v11975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:51:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21644.886071 secs 2024-04-26T17:51:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:45 smithi033 ceph-mon[27845]: pgmap v11975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:51:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21644.886071 secs 2024-04-26T17:51:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:45 smithi096 ceph-mon[32753]: pgmap v11975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:51:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:47 smithi194 ceph-mon[33017]: pgmap v11976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:47 smithi033 ceph-mon[27845]: pgmap v11976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:47 smithi096 ceph-mon[32753]: pgmap v11976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21647.026353 secs 2024-04-26T17:51:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:48.652+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:48.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:48.652+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:51:48.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:51:48.652+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:51:48.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21647.026353 secs 2024-04-26T17:51:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21647.026353 secs 2024-04-26T17:51:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:49 smithi194 ceph-mon[33017]: pgmap v11977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:49 smithi033 ceph-mon[27845]: pgmap v11977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:49 smithi096 ceph-mon[32753]: pgmap v11977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:50.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21649.886193 secs 2024-04-26T17:51:50.609 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21649.886193 secs 2024-04-26T17:51:50.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21649.886193 secs 2024-04-26T17:51:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:51 smithi194 ceph-mon[33017]: pgmap v11978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:51 smithi033 ceph-mon[27845]: pgmap v11978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:51 smithi096 ceph-mon[32753]: pgmap v11978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:53 smithi194 ceph-mon[33017]: pgmap v11979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21652.026412 secs 2024-04-26T17:51:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:53 smithi033 ceph-mon[27845]: pgmap v11979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21652.026412 secs 2024-04-26T17:51:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:53 smithi096 ceph-mon[32753]: pgmap v11979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21652.026412 secs 2024-04-26T17:51:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21654.886333 secs 2024-04-26T17:51:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:55 smithi194 ceph-mon[33017]: pgmap v11980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21654.886333 secs 2024-04-26T17:51:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:55 smithi033 ceph-mon[27845]: pgmap v11980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21654.886333 secs 2024-04-26T17:51:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:55 smithi096 ceph-mon[32753]: pgmap v11980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:57 smithi194 ceph-mon[33017]: pgmap v11981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:57 smithi033 ceph-mon[27845]: pgmap v11981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:57 smithi096 ceph-mon[32753]: pgmap v11981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:51:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21657.026496 secs 2024-04-26T17:51:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21657.026496 secs 2024-04-26T17:51:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21657.026496 secs 2024-04-26T17:51:59.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:51:59 smithi033 ceph-mon[27845]: pgmap v11982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:51:59 smithi194 ceph-mon[33017]: pgmap v11982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:51:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:51:59 smithi096 ceph-mon[32753]: pgmap v11982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21659.886421 secs 2024-04-26T17:52:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21659.886421 secs 2024-04-26T17:52:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21659.886421 secs 2024-04-26T17:52:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:01 smithi194 ceph-mon[33017]: pgmap v11983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:01 smithi033 ceph-mon[27845]: pgmap v11983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:01 smithi096 ceph-mon[32753]: pgmap v11983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:03 smithi194 ceph-mon[33017]: pgmap v11984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21662.026580 secs 2024-04-26T17:52:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:03.665+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:03.665+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:52:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:03.665+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:03 smithi033 ceph-mon[27845]: pgmap v11984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21662.026580 secs 2024-04-26T17:52:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:03 smithi096 ceph-mon[32753]: pgmap v11984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21662.026580 secs 2024-04-26T17:52:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21664.886565 secs 2024-04-26T17:52:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:05 smithi194 ceph-mon[33017]: pgmap v11985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21664.886565 secs 2024-04-26T17:52:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:05 smithi033 ceph-mon[27845]: pgmap v11985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21664.886565 secs 2024-04-26T17:52:05.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:05 smithi096 ceph-mon[32753]: pgmap v11985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:07 smithi194 ceph-mon[33017]: pgmap v11986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:07 smithi033 ceph-mon[27845]: pgmap v11986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:07 smithi096 ceph-mon[32753]: pgmap v11986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21667.026665 secs 2024-04-26T17:52:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21667.026665 secs 2024-04-26T17:52:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21667.026665 secs 2024-04-26T17:52:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:09 smithi194 ceph-mon[33017]: pgmap v11987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:09.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:09 smithi033 ceph-mon[27845]: pgmap v11987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:09 smithi096 ceph-mon[32753]: pgmap v11987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21669.886683 secs 2024-04-26T17:52:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21669.886683 secs 2024-04-26T17:52:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21669.886683 secs 2024-04-26T17:52:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:11 smithi194 ceph-mon[33017]: pgmap v11988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:11 smithi033 ceph-mon[27845]: pgmap v11988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:11 smithi096 ceph-mon[32753]: pgmap v11988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:13 smithi194 ceph-mon[33017]: pgmap v11989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21672.026746 secs 2024-04-26T17:52:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:13 smithi033 ceph-mon[27845]: pgmap v11989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21672.026746 secs 2024-04-26T17:52:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:13 smithi096 ceph-mon[32753]: pgmap v11989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21672.026746 secs 2024-04-26T17:52:14.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21674.886807 secs 2024-04-26T17:52:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21674.886807 secs 2024-04-26T17:52:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21674.886807 secs 2024-04-26T17:52:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:15 smithi033 ceph-mon[27845]: pgmap v11990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:15 smithi096 ceph-mon[32753]: pgmap v11990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:15 smithi194 ceph-mon[33017]: pgmap v11990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:17 smithi033 ceph-mon[27845]: pgmap v11991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:17 smithi096 ceph-mon[32753]: pgmap v11991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:17 smithi194 ceph-mon[33017]: pgmap v11991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21677.026829 secs 2024-04-26T17:52:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:18.674+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:18.674+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:18.674+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:52:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21677.026829 secs 2024-04-26T17:52:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21677.026829 secs 2024-04-26T17:52:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:19 smithi033 ceph-mon[27845]: pgmap v11992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21679.886926 secs 2024-04-26T17:52:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:19 smithi096 ceph-mon[32753]: pgmap v11992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21679.886926 secs 2024-04-26T17:52:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:19 smithi194 ceph-mon[33017]: pgmap v11992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21679.886926 secs 2024-04-26T17:52:21.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:21 smithi033 ceph-mon[27845]: pgmap v11993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:21 smithi096 ceph-mon[32753]: pgmap v11993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:21 smithi194 ceph-mon[33017]: pgmap v11993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:23.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:23 smithi033 ceph-mon[27845]: pgmap v11994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21682.026910 secs 2024-04-26T17:52:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:23 smithi096 ceph-mon[32753]: pgmap v11994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21682.026910 secs 2024-04-26T17:52:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:23 smithi194 ceph-mon[33017]: pgmap v11994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21682.026910 secs 2024-04-26T17:52:24.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21684.887044 secs 2024-04-26T17:52:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21684.887044 secs 2024-04-26T17:52:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21684.887044 secs 2024-04-26T17:52:25.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:25 smithi033 ceph-mon[27845]: pgmap v11995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:25 smithi096 ceph-mon[32753]: pgmap v11995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:25 smithi194 ceph-mon[33017]: pgmap v11995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:27 smithi033 ceph-mon[27845]: pgmap v11996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:27 smithi096 ceph-mon[32753]: pgmap v11996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:27 smithi194 ceph-mon[33017]: pgmap v11996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:28.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21687.026993 secs 2024-04-26T17:52:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21687.026993 secs 2024-04-26T17:52:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21687.026993 secs 2024-04-26T17:52:29.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:29 smithi033 ceph-mon[27845]: pgmap v11997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21689.887143 secs 2024-04-26T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:29 smithi096 ceph-mon[32753]: pgmap v11997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21689.887143 secs 2024-04-26T17:52:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:29 smithi194 ceph-mon[33017]: pgmap v11997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21689.887143 secs 2024-04-26T17:52:31.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:31 smithi033 ceph-mon[27845]: pgmap v11998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:31 smithi096 ceph-mon[32753]: pgmap v11998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:31 smithi194 ceph-mon[33017]: pgmap v11998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:33 smithi096 ceph-mon[32753]: pgmap v11999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21692.027053 secs 2024-04-26T17:52:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:33.691+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:33.691+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:52:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:33.691+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:33 smithi194 ceph-mon[33017]: pgmap v11999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21692.027053 secs 2024-04-26T17:52:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:33 smithi033 ceph-mon[27845]: pgmap v11999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21692.027053 secs 2024-04-26T17:52:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21694.887259 secs 2024-04-26T17:52:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21694.887259 secs 2024-04-26T17:52:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21694.887259 secs 2024-04-26T17:52:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:35 smithi194 ceph-mon[33017]: pgmap v12000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:35 smithi033 ceph-mon[27845]: pgmap v12000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:35 smithi096 ceph-mon[32753]: pgmap v12000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:36 smithi194 ceph-mon[33017]: pgmap v12001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:36 smithi194 ceph-mon[33017]: pgmap v12002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:52:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:36 smithi033 ceph-mon[27845]: pgmap v12001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:52:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:36 smithi033 ceph-mon[27845]: pgmap v12002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:52:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:36 smithi096 ceph-mon[32753]: pgmap v12001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:52:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:36 smithi096 ceph-mon[32753]: pgmap v12002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:52:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:38 smithi194 ceph-mon[33017]: pgmap v12003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21697.027138 secs 2024-04-26T17:52:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:38 smithi033 ceph-mon[27845]: pgmap v12003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21697.027138 secs 2024-04-26T17:52:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:38 smithi096 ceph-mon[32753]: pgmap v12003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21697.027138 secs 2024-04-26T17:52:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21699.887377 secs 2024-04-26T17:52:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21699.887377 secs 2024-04-26T17:52:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21699.887377 secs 2024-04-26T17:52:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:40.486+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:52:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:40.486+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T17:52:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:40.487+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 173 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:52:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:40.487+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 173 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:52:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:40 smithi194 ceph-mon[33017]: pgmap v12004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:40 smithi033 ceph-mon[27845]: pgmap v12004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:40 smithi096 ceph-mon[32753]: pgmap v12004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:42 smithi194 ceph-mon[33017]: pgmap v12005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:42 smithi033 ceph-mon[27845]: pgmap v12005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:42 smithi096 ceph-mon[32753]: pgmap v12005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21702.027195 secs 2024-04-26T17:52:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21702.027195 secs 2024-04-26T17:52:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21702.027195 secs 2024-04-26T17:52:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21704.887499 secs 2024-04-26T17:52:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:44 smithi194 ceph-mon[33017]: pgmap v12006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21704.887499 secs 2024-04-26T17:52:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:44 smithi033 ceph-mon[27845]: pgmap v12006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21704.887499 secs 2024-04-26T17:52:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:44 smithi096 ceph-mon[32753]: pgmap v12006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:52:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:46 smithi194 ceph-mon[33017]: pgmap v12007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:52:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:46 smithi033 ceph-mon[27845]: pgmap v12007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:52:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:46 smithi096 ceph-mon[32753]: pgmap v12007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:52:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:48.706+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:48.706+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:52:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:52:48.706+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:52:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:48 smithi194 ceph-mon[33017]: pgmap v12008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21707.027254 secs 2024-04-26T17:52:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:48 smithi033 ceph-mon[27845]: pgmap v12008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21707.027254 secs 2024-04-26T17:52:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:48 smithi096 ceph-mon[32753]: pgmap v12008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21707.027254 secs 2024-04-26T17:52:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21709.887614 secs 2024-04-26T17:52:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21709.887614 secs 2024-04-26T17:52:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21709.887614 secs 2024-04-26T17:52:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:50 smithi194 ceph-mon[33017]: pgmap v12009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:50 smithi033 ceph-mon[27845]: pgmap v12009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:50 smithi096 ceph-mon[32753]: pgmap v12009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:52 smithi194 ceph-mon[33017]: pgmap v12010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:52 smithi033 ceph-mon[27845]: pgmap v12010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:52 smithi096 ceph-mon[32753]: pgmap v12010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21712.027338 secs 2024-04-26T17:52:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21712.027338 secs 2024-04-26T17:52:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21712.027338 secs 2024-04-26T17:52:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21714.887738 secs 2024-04-26T17:52:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:54 smithi194 ceph-mon[33017]: pgmap v12011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21714.887738 secs 2024-04-26T17:52:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:54 smithi033 ceph-mon[27845]: pgmap v12011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21714.887738 secs 2024-04-26T17:52:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:54 smithi096 ceph-mon[32753]: pgmap v12011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:56 smithi194 ceph-mon[33017]: pgmap v12012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:56.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:56 smithi033 ceph-mon[27845]: pgmap v12012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:56 smithi096 ceph-mon[32753]: pgmap v12012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:52:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:58 smithi033 ceph-mon[27845]: pgmap v12013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21717.027418 secs 2024-04-26T17:52:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:58 smithi096 ceph-mon[32753]: pgmap v12013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21717.027418 secs 2024-04-26T17:52:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:58 smithi194 ceph-mon[33017]: pgmap v12013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:52:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21717.027418 secs 2024-04-26T17:52:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:52:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21719.887874 secs 2024-04-26T17:52:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:52:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21719.887874 secs 2024-04-26T17:53:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:52:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21719.887874 secs 2024-04-26T17:53:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:00 smithi033 ceph-mon[27845]: pgmap v12014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:00 smithi096 ceph-mon[32753]: pgmap v12014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:00 smithi194 ceph-mon[33017]: pgmap v12014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:02.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:02 smithi033 ceph-mon[27845]: pgmap v12015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:02 smithi096 ceph-mon[32753]: pgmap v12015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:02 smithi194 ceph-mon[33017]: pgmap v12015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:03.722+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:03.722+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:53:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:03.722+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21722.027478 secs 2024-04-26T17:53:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21722.027478 secs 2024-04-26T17:53:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21722.027478 secs 2024-04-26T17:53:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21724.888010 secs 2024-04-26T17:53:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:04 smithi033 ceph-mon[27845]: pgmap v12016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21724.888010 secs 2024-04-26T17:53:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:04 smithi096 ceph-mon[32753]: pgmap v12016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21724.888010 secs 2024-04-26T17:53:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:04 smithi194 ceph-mon[33017]: pgmap v12016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:06 smithi033 ceph-mon[27845]: pgmap v12017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:06 smithi096 ceph-mon[32753]: pgmap v12017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:06 smithi194 ceph-mon[33017]: pgmap v12017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:08 smithi033 ceph-mon[27845]: pgmap v12018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21727.027561 secs 2024-04-26T17:53:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:08 smithi096 ceph-mon[32753]: pgmap v12018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21727.027561 secs 2024-04-26T17:53:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:08 smithi194 ceph-mon[33017]: pgmap v12018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21727.027561 secs 2024-04-26T17:53:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21729.888127 secs 2024-04-26T17:53:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21729.888127 secs 2024-04-26T17:53:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21729.888127 secs 2024-04-26T17:53:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:10 smithi033 ceph-mon[27845]: pgmap v12019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:10 smithi096 ceph-mon[32753]: pgmap v12019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:10 smithi194 ceph-mon[33017]: pgmap v12019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:11 smithi194 ceph-mon[33017]: pgmap v12020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:11 smithi033 ceph-mon[27845]: pgmap v12020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:11 smithi096 ceph-mon[32753]: pgmap v12020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21732.027644 secs 2024-04-26T17:53:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21732.027644 secs 2024-04-26T17:53:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21732.027644 secs 2024-04-26T17:53:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:13 smithi033 ceph-mon[27845]: pgmap v12021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:13 smithi096 ceph-mon[32753]: pgmap v12021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:13 smithi194 ceph-mon[33017]: pgmap v12021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21734.888237 secs 2024-04-26T17:53:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21734.888237 secs 2024-04-26T17:53:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21734.888237 secs 2024-04-26T17:53:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:15 smithi033 ceph-mon[27845]: pgmap v12022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:15 smithi096 ceph-mon[32753]: pgmap v12022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:16.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:15 smithi194 ceph-mon[33017]: pgmap v12022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:18.737+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:18.737+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:18.737+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:53:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:18 smithi194 ceph-mon[33017]: pgmap v12023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21737.027729 secs 2024-04-26T17:53:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:18 smithi033 ceph-mon[27845]: pgmap v12023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21737.027729 secs 2024-04-26T17:53:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:18 smithi096 ceph-mon[32753]: pgmap v12023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21737.027729 secs 2024-04-26T17:53:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21739.888331 secs 2024-04-26T17:53:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21739.888331 secs 2024-04-26T17:53:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21739.888331 secs 2024-04-26T17:53:20.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:20 smithi194 ceph-mon[33017]: pgmap v12024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:20 smithi033 ceph-mon[27845]: pgmap v12024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:20 smithi096 ceph-mon[32753]: pgmap v12024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:22 smithi194 ceph-mon[33017]: pgmap v12025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:22 smithi033 ceph-mon[27845]: pgmap v12025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:22 smithi096 ceph-mon[32753]: pgmap v12025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21742.027790 secs 2024-04-26T17:53:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21742.027790 secs 2024-04-26T17:53:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21742.027790 secs 2024-04-26T17:53:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21744.888463 secs 2024-04-26T17:53:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:24 smithi194 ceph-mon[33017]: pgmap v12026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21744.888463 secs 2024-04-26T17:53:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:24 smithi033 ceph-mon[27845]: pgmap v12026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21744.888463 secs 2024-04-26T17:53:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:24 smithi096 ceph-mon[32753]: pgmap v12026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:25.494 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:25.494+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:53:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:25.494+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T17:53:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:25.495+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 174 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T17:53:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:25.495+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 174 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:53:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:26 smithi194 ceph-mon[33017]: pgmap v12027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:26 smithi033 ceph-mon[27845]: pgmap v12027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:26 smithi096 ceph-mon[32753]: pgmap v12027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:28 smithi194 ceph-mon[33017]: pgmap v12028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21747.027874 secs 2024-04-26T17:53:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:28 smithi033 ceph-mon[27845]: pgmap v12028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21747.027874 secs 2024-04-26T17:53:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:28 smithi096 ceph-mon[32753]: pgmap v12028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21747.027874 secs 2024-04-26T17:53:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21749.888589 secs 2024-04-26T17:53:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21749.888589 secs 2024-04-26T17:53:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21749.888589 secs 2024-04-26T17:53:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:30 smithi194 ceph-mon[33017]: pgmap v12029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:30 smithi033 ceph-mon[27845]: pgmap v12029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:30 smithi096 ceph-mon[32753]: pgmap v12029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:32 smithi194 ceph-mon[33017]: pgmap v12030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:32 smithi033 ceph-mon[27845]: pgmap v12030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:32 smithi096 ceph-mon[32753]: pgmap v12030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:33.753+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:33.753+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:53:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:33.753+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21752.027958 secs 2024-04-26T17:53:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21752.027958 secs 2024-04-26T17:53:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21752.027958 secs 2024-04-26T17:53:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21754.888703 secs 2024-04-26T17:53:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:34 smithi033 ceph-mon[27845]: pgmap v12031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21754.888703 secs 2024-04-26T17:53:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:34 smithi096 ceph-mon[32753]: pgmap v12031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21754.888703 secs 2024-04-26T17:53:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:34 smithi194 ceph-mon[33017]: pgmap v12031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:36 smithi033 ceph-mon[27845]: pgmap v12032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:53:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:36 smithi096 ceph-mon[32753]: pgmap v12032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:53:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:36 smithi194 ceph-mon[33017]: pgmap v12032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:53:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:37 smithi194 ceph-mon[33017]: pgmap v12033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:38.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:37 smithi033 ceph-mon[27845]: pgmap v12033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:37 smithi096 ceph-mon[32753]: pgmap v12033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21757.028020 secs 2024-04-26T17:53:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:53:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:53:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:38 smithi194 ceph-mon[33017]: pgmap v12034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:53:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:38 smithi194 ceph-mon[33017]: pgmap v12035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:53:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21757.028020 secs 2024-04-26T17:53:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:53:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:53:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:38 smithi033 ceph-mon[27845]: pgmap v12034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:53:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:38 smithi033 ceph-mon[27845]: pgmap v12035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:53:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21757.028020 secs 2024-04-26T17:53:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:53:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:53:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:38 smithi096 ceph-mon[32753]: pgmap v12034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:53:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:38 smithi096 ceph-mon[32753]: pgmap v12035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:53:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:53:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21759.888836 secs 2024-04-26T17:53:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21759.888836 secs 2024-04-26T17:53:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21759.888836 secs 2024-04-26T17:53:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:40 smithi194 ceph-mon[33017]: pgmap v12036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:40 smithi033 ceph-mon[27845]: pgmap v12036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:40 smithi096 ceph-mon[32753]: pgmap v12036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:43 smithi194 ceph-mon[33017]: pgmap v12037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21762.028080 secs 2024-04-26T17:53:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:43 smithi033 ceph-mon[27845]: pgmap v12037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21762.028080 secs 2024-04-26T17:53:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:43 smithi096 ceph-mon[32753]: pgmap v12037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21762.028080 secs 2024-04-26T17:53:45.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21764.888958 secs 2024-04-26T17:53:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:45 smithi194 ceph-mon[33017]: pgmap v12038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:53:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21764.888958 secs 2024-04-26T17:53:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:45 smithi033 ceph-mon[27845]: pgmap v12038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:53:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21764.888958 secs 2024-04-26T17:53:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:45 smithi096 ceph-mon[32753]: pgmap v12038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:53:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:47 smithi033 ceph-mon[27845]: pgmap v12039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:47 smithi096 ceph-mon[32753]: pgmap v12039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:47 smithi194 ceph-mon[33017]: pgmap v12039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T17:53:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21767.028140 secs 2024-04-26T17:53:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21767.028140 secs 2024-04-26T17:53:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21767.028140 secs 2024-04-26T17:53:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:48.769+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:48.769+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:53:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:53:48.769+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:53:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:49 smithi033 ceph-mon[27845]: pgmap v12040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:53:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:49 smithi096 ceph-mon[32753]: pgmap v12040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:53:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:49 smithi194 ceph-mon[33017]: pgmap v12040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:53:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21769.889081 secs 2024-04-26T17:53:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21769.889081 secs 2024-04-26T17:53:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21769.889081 secs 2024-04-26T17:53:51.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:51 smithi033 ceph-mon[27845]: pgmap v12041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:51 smithi096 ceph-mon[32753]: pgmap v12041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:51 smithi194 ceph-mon[33017]: pgmap v12041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:53 smithi033 ceph-mon[27845]: pgmap v12042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21772.028227 secs 2024-04-26T17:53:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:53 smithi096 ceph-mon[32753]: pgmap v12042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21772.028227 secs 2024-04-26T17:53:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:53 smithi194 ceph-mon[33017]: pgmap v12042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21772.028227 secs 2024-04-26T17:53:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21774.889186 secs 2024-04-26T17:53:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:55 smithi033 ceph-mon[27845]: pgmap v12043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21774.889186 secs 2024-04-26T17:53:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:55 smithi096 ceph-mon[32753]: pgmap v12043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21774.889186 secs 2024-04-26T17:53:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:55 smithi194 ceph-mon[33017]: pgmap v12043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:57 smithi033 ceph-mon[27845]: pgmap v12044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:57 smithi096 ceph-mon[32753]: pgmap v12044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:57 smithi194 ceph-mon[33017]: pgmap v12044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:53:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21777.028289 secs 2024-04-26T17:53:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21777.028289 secs 2024-04-26T17:53:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21777.028289 secs 2024-04-26T17:53:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:53:59 smithi033 ceph-mon[27845]: pgmap v12045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:53:59 smithi096 ceph-mon[32753]: pgmap v12045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:53:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:53:59 smithi194 ceph-mon[33017]: pgmap v12045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:00.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21779.889305 secs 2024-04-26T17:54:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21779.889305 secs 2024-04-26T17:54:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21779.889305 secs 2024-04-26T17:54:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:01 smithi033 ceph-mon[27845]: pgmap v12046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:01 smithi096 ceph-mon[32753]: pgmap v12046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:01 smithi194 ceph-mon[33017]: pgmap v12046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:03 smithi096 ceph-mon[32753]: pgmap v12047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21782.028371 secs 2024-04-26T17:54:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:03 smithi194 ceph-mon[33017]: pgmap v12047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21782.028371 secs 2024-04-26T17:54:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:03 smithi033 ceph-mon[27845]: pgmap v12047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21782.028371 secs 2024-04-26T17:54:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:03.773+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:03.773+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:03.773+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:54:05.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21784.889423 secs 2024-04-26T17:54:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:05 smithi194 ceph-mon[33017]: pgmap v12048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21784.889423 secs 2024-04-26T17:54:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:05 smithi033 ceph-mon[27845]: pgmap v12048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21784.889423 secs 2024-04-26T17:54:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:05 smithi096 ceph-mon[32753]: pgmap v12048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:07.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:07 smithi194 ceph-mon[33017]: pgmap v12049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:07 smithi033 ceph-mon[27845]: pgmap v12049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:07 smithi096 ceph-mon[32753]: pgmap v12049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:08.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21787.028455 secs 2024-04-26T17:54:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21787.028455 secs 2024-04-26T17:54:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21787.028455 secs 2024-04-26T17:54:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:09 smithi194 ceph-mon[33017]: pgmap v12050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:09 smithi033 ceph-mon[27845]: pgmap v12050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:09 smithi096 ceph-mon[32753]: pgmap v12050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21789.889545 secs 2024-04-26T17:54:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21789.889545 secs 2024-04-26T17:54:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21789.889545 secs 2024-04-26T17:54:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:11 smithi194 ceph-mon[33017]: pgmap v12051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:11 smithi033 ceph-mon[27845]: pgmap v12051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:11 smithi096 ceph-mon[32753]: pgmap v12051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:13 smithi194 ceph-mon[33017]: pgmap v12052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21792.028533 secs 2024-04-26T17:54:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:13 smithi033 ceph-mon[27845]: pgmap v12052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21792.028533 secs 2024-04-26T17:54:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:13 smithi096 ceph-mon[32753]: pgmap v12052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21792.028533 secs 2024-04-26T17:54:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21794.889663 secs 2024-04-26T17:54:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:15 smithi194 ceph-mon[33017]: pgmap v12053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:15.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21794.889663 secs 2024-04-26T17:54:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:15 smithi033 ceph-mon[27845]: pgmap v12053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21794.889663 secs 2024-04-26T17:54:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:15 smithi096 ceph-mon[32753]: pgmap v12053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:17 smithi194 ceph-mon[33017]: pgmap v12054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:17 smithi033 ceph-mon[27845]: pgmap v12054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:17 smithi096 ceph-mon[32753]: pgmap v12054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21797.028616 secs 2024-04-26T17:54:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21797.028616 secs 2024-04-26T17:54:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21797.028616 secs 2024-04-26T17:54:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:18.778+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:18.778+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:54:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:18.778+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:19.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:19 smithi194 ceph-mon[33017]: pgmap v12055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:19 smithi033 ceph-mon[27845]: pgmap v12055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:19 smithi096 ceph-mon[32753]: pgmap v12055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21799.889783 secs 2024-04-26T17:54:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21799.889783 secs 2024-04-26T17:54:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21799.889783 secs 2024-04-26T17:54:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:21 smithi194 ceph-mon[33017]: pgmap v12056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:21 smithi033 ceph-mon[27845]: pgmap v12056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:21 smithi096 ceph-mon[32753]: pgmap v12056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:23 smithi194 ceph-mon[33017]: pgmap v12057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21802.028696 secs 2024-04-26T17:54:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:23 smithi033 ceph-mon[27845]: pgmap v12057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21802.028696 secs 2024-04-26T17:54:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:23 smithi096 ceph-mon[32753]: pgmap v12057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21802.028696 secs 2024-04-26T17:54:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21804.889900 secs 2024-04-26T17:54:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:25 smithi194 ceph-mon[33017]: pgmap v12058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21804.889900 secs 2024-04-26T17:54:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:25 smithi033 ceph-mon[27845]: pgmap v12058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21804.889900 secs 2024-04-26T17:54:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:25 smithi096 ceph-mon[32753]: pgmap v12058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:27.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:27 smithi194 ceph-mon[33017]: pgmap v12059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:27 smithi033 ceph-mon[27845]: pgmap v12059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:27 smithi096 ceph-mon[32753]: pgmap v12059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21807.028758 secs 2024-04-26T17:54:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21807.028758 secs 2024-04-26T17:54:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21807.028758 secs 2024-04-26T17:54:29.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:29 smithi033 ceph-mon[27845]: pgmap v12060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:29 smithi194 ceph-mon[33017]: pgmap v12060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:29 smithi096 ceph-mon[32753]: pgmap v12060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21809.890003 secs 2024-04-26T17:54:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21809.890003 secs 2024-04-26T17:54:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21809.890003 secs 2024-04-26T17:54:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:31 smithi194 ceph-mon[33017]: pgmap v12061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:31 smithi033 ceph-mon[27845]: pgmap v12061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:31 smithi096 ceph-mon[32753]: pgmap v12061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:33 smithi194 ceph-mon[33017]: pgmap v12062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21812.028845 secs 2024-04-26T17:54:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:33 smithi033 ceph-mon[27845]: pgmap v12062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21812.028845 secs 2024-04-26T17:54:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:33 smithi096 ceph-mon[32753]: pgmap v12062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21812.028845 secs 2024-04-26T17:54:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:33.784+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:33.784+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:54:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:33.784+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21814.890143 secs 2024-04-26T17:54:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:35 smithi194 ceph-mon[33017]: pgmap v12063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21814.890143 secs 2024-04-26T17:54:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:35 smithi033 ceph-mon[27845]: pgmap v12063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21814.890143 secs 2024-04-26T17:54:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:35 smithi096 ceph-mon[32753]: pgmap v12063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:37 smithi194 ceph-mon[33017]: pgmap v12064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:37 smithi033 ceph-mon[27845]: pgmap v12064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:37 smithi096 ceph-mon[32753]: pgmap v12064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21817.028930 secs 2024-04-26T17:54:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21817.028930 secs 2024-04-26T17:54:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21817.028930 secs 2024-04-26T17:54:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:39 smithi194 ceph-mon[33017]: pgmap v12065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:54:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:39 smithi033 ceph-mon[27845]: pgmap v12065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:54:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:39 smithi096 ceph-mon[32753]: pgmap v12065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:54:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21819.890217 secs 2024-04-26T17:54:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:54:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:54:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:54:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21819.890217 secs 2024-04-26T17:54:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:54:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:54:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:54:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21819.890217 secs 2024-04-26T17:54:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:54:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:54:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:54:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:41 smithi194 ceph-mon[33017]: pgmap v12066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:54:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:41 smithi194 ceph-mon[33017]: pgmap v12067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:41 smithi033 ceph-mon[27845]: pgmap v12066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:54:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:41 smithi033 ceph-mon[27845]: pgmap v12067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:41 smithi096 ceph-mon[32753]: pgmap v12066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:54:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:41 smithi096 ceph-mon[32753]: pgmap v12067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:43 smithi194 ceph-mon[33017]: pgmap v12068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21822.029012 secs 2024-04-26T17:54:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:43 smithi033 ceph-mon[27845]: pgmap v12068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21822.029012 secs 2024-04-26T17:54:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:43 smithi096 ceph-mon[32753]: pgmap v12068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21822.029012 secs 2024-04-26T17:54:45.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21824.890328 secs 2024-04-26T17:54:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:45 smithi194 ceph-mon[33017]: pgmap v12069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21824.890328 secs 2024-04-26T17:54:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:45 smithi033 ceph-mon[27845]: pgmap v12069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21824.890328 secs 2024-04-26T17:54:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:45 smithi096 ceph-mon[32753]: pgmap v12069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:47 smithi033 ceph-mon[27845]: pgmap v12070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:47 smithi096 ceph-mon[32753]: pgmap v12070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:47 smithi194 ceph-mon[33017]: pgmap v12070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21827.029074 secs 2024-04-26T17:54:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21827.029074 secs 2024-04-26T17:54:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:48.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:48.789+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:54:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:54:48.789+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:54:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21827.029074 secs 2024-04-26T17:54:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:49 smithi033 ceph-mon[27845]: pgmap v12071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21829.890406 secs 2024-04-26T17:54:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:49 smithi096 ceph-mon[32753]: pgmap v12071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21829.890406 secs 2024-04-26T17:54:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:49 smithi194 ceph-mon[33017]: pgmap v12071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:54:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21829.890406 secs 2024-04-26T17:54:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:51 smithi033 ceph-mon[27845]: pgmap v12072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:54:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:51 smithi096 ceph-mon[32753]: pgmap v12072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:54:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:51 smithi194 ceph-mon[33017]: pgmap v12072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:54:53.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:53 smithi033 ceph-mon[27845]: pgmap v12073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21832.029135 secs 2024-04-26T17:54:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:53 smithi096 ceph-mon[32753]: pgmap v12073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21832.029135 secs 2024-04-26T17:54:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:53 smithi194 ceph-mon[33017]: pgmap v12073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21832.029135 secs 2024-04-26T17:54:54.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21834.890523 secs 2024-04-26T17:54:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21834.890523 secs 2024-04-26T17:54:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21834.890523 secs 2024-04-26T17:54:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:55 smithi033 ceph-mon[27845]: pgmap v12074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:55 smithi096 ceph-mon[32753]: pgmap v12074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:55 smithi194 ceph-mon[33017]: pgmap v12074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:57 smithi033 ceph-mon[27845]: pgmap v12075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:57 smithi096 ceph-mon[32753]: pgmap v12075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:57 smithi194 ceph-mon[33017]: pgmap v12075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21837.029214 secs 2024-04-26T17:54:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21837.029214 secs 2024-04-26T17:54:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21837.029214 secs 2024-04-26T17:54:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:59 smithi033 ceph-mon[27845]: pgmap v12076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21839.890642 secs 2024-04-26T17:54:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:59 smithi096 ceph-mon[32753]: pgmap v12076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21839.890642 secs 2024-04-26T17:54:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:59 smithi194 ceph-mon[33017]: pgmap v12076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:54:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21839.890642 secs 2024-04-26T17:55:01.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:01 smithi033 ceph-mon[27845]: pgmap v12077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:01 smithi096 ceph-mon[32753]: pgmap v12077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:01 smithi194 ceph-mon[33017]: pgmap v12077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:03 smithi033 ceph-mon[27845]: pgmap v12078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21842.029312 secs 2024-04-26T17:55:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:03 smithi096 ceph-mon[32753]: pgmap v12078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21842.029312 secs 2024-04-26T17:55:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:03.790+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:03.790+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:55:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:03.790+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:03 smithi194 ceph-mon[33017]: pgmap v12078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21842.029312 secs 2024-04-26T17:55:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21844.890714 secs 2024-04-26T17:55:04.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21844.890714 secs 2024-04-26T17:55:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21844.890714 secs 2024-04-26T17:55:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:05 smithi194 ceph-mon[33017]: pgmap v12079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:05.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:05 smithi033 ceph-mon[27845]: pgmap v12079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:05 smithi096 ceph-mon[32753]: pgmap v12079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:07.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:07 smithi194 ceph-mon[33017]: pgmap v12080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:07 smithi033 ceph-mon[27845]: pgmap v12080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:07 smithi096 ceph-mon[32753]: pgmap v12080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21847.029394 secs 2024-04-26T17:55:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21847.029394 secs 2024-04-26T17:55:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21847.029394 secs 2024-04-26T17:55:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:09 smithi194 ceph-mon[33017]: pgmap v12081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21849.890777 secs 2024-04-26T17:55:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:09 smithi033 ceph-mon[27845]: pgmap v12081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21849.890777 secs 2024-04-26T17:55:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:09 smithi096 ceph-mon[32753]: pgmap v12081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21849.890777 secs 2024-04-26T17:55:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:11 smithi194 ceph-mon[33017]: pgmap v12082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:11 smithi033 ceph-mon[27845]: pgmap v12082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:11 smithi096 ceph-mon[32753]: pgmap v12082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:13 smithi194 ceph-mon[33017]: pgmap v12083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21852.029478 secs 2024-04-26T17:55:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:13 smithi033 ceph-mon[27845]: pgmap v12083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21852.029478 secs 2024-04-26T17:55:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:13 smithi096 ceph-mon[32753]: pgmap v12083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21852.029478 secs 2024-04-26T17:55:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21854.890913 secs 2024-04-26T17:55:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21854.890913 secs 2024-04-26T17:55:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21854.890913 secs 2024-04-26T17:55:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:15 smithi194 ceph-mon[33017]: pgmap v12084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:15 smithi033 ceph-mon[27845]: pgmap v12084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:15 smithi096 ceph-mon[32753]: pgmap v12084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:17 smithi194 ceph-mon[33017]: pgmap v12085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:17 smithi033 ceph-mon[27845]: pgmap v12085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:17 smithi096 ceph-mon[32753]: pgmap v12085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:18.805+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:18.805+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:18.805+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:55:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21857.029561 secs 2024-04-26T17:55:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21857.029561 secs 2024-04-26T17:55:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21857.029561 secs 2024-04-26T17:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:19 smithi194 ceph-mon[33017]: pgmap v12086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21859.891058 secs 2024-04-26T17:55:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:19 smithi033 ceph-mon[27845]: pgmap v12086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21859.891058 secs 2024-04-26T17:55:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:19 smithi096 ceph-mon[32753]: pgmap v12086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21859.891058 secs 2024-04-26T17:55:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:21 smithi194 ceph-mon[33017]: pgmap v12087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:21 smithi033 ceph-mon[27845]: pgmap v12087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:21 smithi096 ceph-mon[32753]: pgmap v12087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:23 smithi194 ceph-mon[33017]: pgmap v12088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21862.029643 secs 2024-04-26T17:55:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:23 smithi033 ceph-mon[27845]: pgmap v12088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21862.029643 secs 2024-04-26T17:55:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:23 smithi096 ceph-mon[32753]: pgmap v12088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21862.029643 secs 2024-04-26T17:55:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21864.891178 secs 2024-04-26T17:55:24.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21864.891178 secs 2024-04-26T17:55:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21864.891178 secs 2024-04-26T17:55:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:25 smithi194 ceph-mon[33017]: pgmap v12089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:25.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:25 smithi033 ceph-mon[27845]: pgmap v12089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:25 smithi096 ceph-mon[32753]: pgmap v12089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:27 smithi194 ceph-mon[33017]: pgmap v12090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:27 smithi033 ceph-mon[27845]: pgmap v12090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:27 smithi096 ceph-mon[32753]: pgmap v12090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21867.029721 secs 2024-04-26T17:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21867.029721 secs 2024-04-26T17:55:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21867.029721 secs 2024-04-26T17:55:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:29 smithi033 ceph-mon[27845]: pgmap v12091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21869.891275 secs 2024-04-26T17:55:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:29 smithi194 ceph-mon[33017]: pgmap v12091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21869.891275 secs 2024-04-26T17:55:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:29 smithi096 ceph-mon[32753]: pgmap v12091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21869.891275 secs 2024-04-26T17:55:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:31 smithi194 ceph-mon[33017]: pgmap v12092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:31 smithi033 ceph-mon[27845]: pgmap v12092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:31 smithi096 ceph-mon[32753]: pgmap v12092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:33.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:33.821+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:33.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:33.821+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:55:33.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:33.821+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:33 smithi194 ceph-mon[33017]: pgmap v12093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21872.029799 secs 2024-04-26T17:55:33.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:33 smithi033 ceph-mon[27845]: pgmap v12093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21872.029799 secs 2024-04-26T17:55:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:33 smithi096 ceph-mon[32753]: pgmap v12093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21872.029799 secs 2024-04-26T17:55:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21874.891392 secs 2024-04-26T17:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21874.891392 secs 2024-04-26T17:55:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21874.891392 secs 2024-04-26T17:55:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:35 smithi194 ceph-mon[33017]: pgmap v12094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:35 smithi033 ceph-mon[27845]: pgmap v12094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:35 smithi096 ceph-mon[32753]: pgmap v12094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:37 smithi194 ceph-mon[33017]: pgmap v12095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:37 smithi033 ceph-mon[27845]: pgmap v12095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:37 smithi096 ceph-mon[32753]: pgmap v12095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21877.029883 secs 2024-04-26T17:55:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21877.029883 secs 2024-04-26T17:55:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21877.029883 secs 2024-04-26T17:55:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:39 smithi194 ceph-mon[33017]: pgmap v12096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21879.891503 secs 2024-04-26T17:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:39 smithi033 ceph-mon[27845]: pgmap v12096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21879.891503 secs 2024-04-26T17:55:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:39 smithi096 ceph-mon[32753]: pgmap v12096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21879.891503 secs 2024-04-26T17:55:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:55:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:55:40.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:41 smithi096 ceph-mon[32753]: pgmap v12097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:41 smithi194 ceph-mon[33017]: pgmap v12097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:41.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:41 smithi033 ceph-mon[27845]: pgmap v12097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: pgmap v12098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21882.029983 secs 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: pgmap v12098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21882.029983 secs 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:43.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: pgmap v12098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21882.029983 secs 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21884.891624 secs 2024-04-26T17:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21884.891624 secs 2024-04-26T17:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T17:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T17:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T17:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T17:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21884.891624 secs 2024-04-26T17:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:45 smithi033 ceph-mon[27845]: pgmap v12099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:45 smithi096 ceph-mon[32753]: pgmap v12099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:45 smithi194 ceph-mon[33017]: pgmap v12099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:55:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:55:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:55:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:55:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:47.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:55:47.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:55:47.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:55:47.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:55:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:47 smithi033 ceph-mon[27845]: pgmap v12100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:47 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:55:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:47 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:55:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:47 smithi033 ceph-mon[27845]: pgmap v12101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:55:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:47 smithi033 ceph-mon[27845]: pgmap v12102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:47 smithi096 ceph-mon[32753]: pgmap v12100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:47 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:55:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:47 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:55:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:47 smithi096 ceph-mon[32753]: pgmap v12101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:55:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:47 smithi096 ceph-mon[32753]: pgmap v12102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:47 smithi194 ceph-mon[33017]: pgmap v12100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:47 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T17:55:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:47 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T17:55:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:47 smithi194 ceph-mon[33017]: pgmap v12101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:55:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:47 smithi194 ceph-mon[33017]: pgmap v12102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:48.837+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:48.837+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:55:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:55:48.837+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:55:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21887.030002 secs 2024-04-26T17:55:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21887.030002 secs 2024-04-26T17:55:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21887.030002 secs 2024-04-26T17:55:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:49 smithi033 ceph-mon[27845]: pgmap v12103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:55:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21889.891743 secs 2024-04-26T17:55:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:49 smithi096 ceph-mon[32753]: pgmap v12103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:55:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21889.891743 secs 2024-04-26T17:55:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:49 smithi194 ceph-mon[33017]: pgmap v12103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:55:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21889.891743 secs 2024-04-26T17:55:51.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:51 smithi033 ceph-mon[27845]: pgmap v12104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:51 smithi096 ceph-mon[32753]: pgmap v12104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:51 smithi194 ceph-mon[33017]: pgmap v12104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:53 smithi033 ceph-mon[27845]: pgmap v12105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21892.030063 secs 2024-04-26T17:55:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:53 smithi096 ceph-mon[32753]: pgmap v12105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21892.030063 secs 2024-04-26T17:55:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:53 smithi194 ceph-mon[33017]: pgmap v12105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:55:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21892.030063 secs 2024-04-26T17:55:54.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21894.891860 secs 2024-04-26T17:55:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21894.891860 secs 2024-04-26T17:55:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21894.891860 secs 2024-04-26T17:55:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:55 smithi096 ceph-mon[32753]: pgmap v12106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:55:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:55 smithi194 ceph-mon[33017]: pgmap v12106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:55:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:55 smithi033 ceph-mon[27845]: pgmap v12106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T17:55:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:57 smithi194 ceph-mon[33017]: pgmap v12107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:55:58.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:57 smithi033 ceph-mon[27845]: pgmap v12107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:55:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:57 smithi096 ceph-mon[32753]: pgmap v12107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:55:59.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21897.030124 secs 2024-04-26T17:55:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21897.030124 secs 2024-04-26T17:55:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21897.030124 secs 2024-04-26T17:56:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:59 smithi194 ceph-mon[33017]: pgmap v12108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:55:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21899.891966 secs 2024-04-26T17:56:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:59 smithi033 ceph-mon[27845]: pgmap v12108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:55:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21899.891966 secs 2024-04-26T17:56:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:59 smithi096 ceph-mon[32753]: pgmap v12108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:55:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21899.891966 secs 2024-04-26T17:56:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:00 smithi194 ceph-mon[33017]: pgmap v12109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:00 smithi033 ceph-mon[27845]: pgmap v12109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:00 smithi096 ceph-mon[32753]: pgmap v12109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:03 smithi194 ceph-mon[33017]: pgmap v12110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21902.030202 secs 2024-04-26T17:56:03.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:03 smithi033 ceph-mon[27845]: pgmap v12110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21902.030202 secs 2024-04-26T17:56:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:03 smithi096 ceph-mon[32753]: pgmap v12110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21902.030202 secs 2024-04-26T17:56:03.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:03.852+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:03.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:03.852+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:56:03.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:03.852+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21904.892084 secs 2024-04-26T17:56:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:05 smithi194 ceph-mon[33017]: pgmap v12111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21904.892084 secs 2024-04-26T17:56:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:05 smithi033 ceph-mon[27845]: pgmap v12111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21904.892084 secs 2024-04-26T17:56:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:05 smithi096 ceph-mon[32753]: pgmap v12111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:07.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:07 smithi194 ceph-mon[33017]: pgmap v12112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:07 smithi033 ceph-mon[27845]: pgmap v12112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:07 smithi096 ceph-mon[32753]: pgmap v12112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21907.030259 secs 2024-04-26T17:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21907.030259 secs 2024-04-26T17:56:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21907.030259 secs 2024-04-26T17:56:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:09 smithi033 ceph-mon[27845]: pgmap v12113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:09 smithi096 ceph-mon[32753]: pgmap v12113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:09.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:09 smithi194 ceph-mon[33017]: pgmap v12113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21909.892206 secs 2024-04-26T17:56:10.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21909.892206 secs 2024-04-26T17:56:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21909.892206 secs 2024-04-26T17:56:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:11 smithi033 ceph-mon[27845]: pgmap v12114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:11 smithi096 ceph-mon[32753]: pgmap v12114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:11.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:11 smithi194 ceph-mon[33017]: pgmap v12114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:13 smithi033 ceph-mon[27845]: pgmap v12115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21912.030335 secs 2024-04-26T17:56:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:13 smithi096 ceph-mon[32753]: pgmap v12115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21912.030335 secs 2024-04-26T17:56:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:13 smithi194 ceph-mon[33017]: pgmap v12115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21912.030335 secs 2024-04-26T17:56:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21914.892278 secs 2024-04-26T17:56:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:15 smithi033 ceph-mon[27845]: pgmap v12116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21914.892278 secs 2024-04-26T17:56:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:15 smithi096 ceph-mon[32753]: pgmap v12116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:15.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21914.892278 secs 2024-04-26T17:56:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:15 smithi194 ceph-mon[33017]: pgmap v12116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:17.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:17 smithi194 ceph-mon[33017]: pgmap v12117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:17.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:17 smithi033 ceph-mon[27845]: pgmap v12117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:17 smithi096 ceph-mon[32753]: pgmap v12117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21917.030419 secs 2024-04-26T17:56:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21917.030419 secs 2024-04-26T17:56:18.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21917.030419 secs 2024-04-26T17:56:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:18.867+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:18.867+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:56:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:18.867+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:19 smithi033 ceph-mon[27845]: pgmap v12118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:19 smithi096 ceph-mon[32753]: pgmap v12118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:19 smithi194 ceph-mon[33017]: pgmap v12118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:20.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21919.892338 secs 2024-04-26T17:56:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21919.892338 secs 2024-04-26T17:56:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21919.892338 secs 2024-04-26T17:56:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:21 smithi033 ceph-mon[27845]: pgmap v12119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:21 smithi096 ceph-mon[32753]: pgmap v12119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:21 smithi194 ceph-mon[33017]: pgmap v12119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.773+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 msgr2=0x7feee40b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T17:56:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.773+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 msgr2=0x7feee40b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T17:56:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.773+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 0x7feee40b31b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0069000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T17:56:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.773+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 0x7feee40b31b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0069000 comp rx=0 tx=0).stop 2024-04-26T17:56:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.773+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 msgr2=0x7feee40b31b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T17:56:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.773+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c50 0x7feee40b31b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.773+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.774+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:21.776+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0022040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:23 smithi096 ceph-mon[32753]: pgmap v12120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21922.030515 secs 2024-04-26T17:56:23.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:23 smithi194 ceph-mon[33017]: pgmap v12120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21922.030515 secs 2024-04-26T17:56:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:23 smithi033 ceph-mon[27845]: pgmap v12120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21922.030515 secs 2024-04-26T17:56:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21924.892461 secs 2024-04-26T17:56:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:25 smithi096 ceph-mon[32753]: pgmap v12121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:25.525+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:56:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:25.525+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T17:56:25.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:25.526+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 175 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:56:25.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:25.526+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 175 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:56:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21924.892461 secs 2024-04-26T17:56:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:25 smithi194 ceph-mon[33017]: pgmap v12121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21924.892461 secs 2024-04-26T17:56:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:25 smithi033 ceph-mon[27845]: pgmap v12121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:27 smithi194 ceph-mon[33017]: pgmap v12122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:27.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:27 smithi033 ceph-mon[27845]: pgmap v12122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:27 smithi096 ceph-mon[32753]: pgmap v12122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:28.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21927.030594 secs 2024-04-26T17:56:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21927.030594 secs 2024-04-26T17:56:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21927.030594 secs 2024-04-26T17:56:29.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:29 smithi033 ceph-mon[27845]: pgmap v12123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:29 smithi194 ceph-mon[33017]: pgmap v12123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:29 smithi096 ceph-mon[32753]: pgmap v12123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21929.892580 secs 2024-04-26T17:56:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21929.892580 secs 2024-04-26T17:56:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21929.892580 secs 2024-04-26T17:56:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:31 smithi194 ceph-mon[33017]: pgmap v12124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:31 smithi033 ceph-mon[27845]: pgmap v12124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:31 smithi096 ceph-mon[32753]: pgmap v12124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:33 smithi194 ceph-mon[33017]: pgmap v12125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21932.030674 secs 2024-04-26T17:56:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:33 smithi033 ceph-mon[27845]: pgmap v12125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21932.030674 secs 2024-04-26T17:56:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:33 smithi096 ceph-mon[32753]: pgmap v12125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21932.030674 secs 2024-04-26T17:56:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:33.879+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:33.879+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:33.879+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:56:35.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21934.892692 secs 2024-04-26T17:56:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:35 smithi194 ceph-mon[33017]: pgmap v12126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21934.892692 secs 2024-04-26T17:56:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:35 smithi033 ceph-mon[27845]: pgmap v12126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21934.892692 secs 2024-04-26T17:56:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:35 smithi096 ceph-mon[32753]: pgmap v12126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:37 smithi194 ceph-mon[33017]: pgmap v12127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:37 smithi033 ceph-mon[27845]: pgmap v12127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:37 smithi096 ceph-mon[32753]: pgmap v12127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21937.030757 secs 2024-04-26T17:56:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21937.030757 secs 2024-04-26T17:56:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21937.030757 secs 2024-04-26T17:56:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:39 smithi194 ceph-mon[33017]: pgmap v12128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:39 smithi033 ceph-mon[27845]: pgmap v12128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:39 smithi096 ceph-mon[32753]: pgmap v12128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:40.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21939.892826 secs 2024-04-26T17:56:40.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21939.892826 secs 2024-04-26T17:56:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21939.892826 secs 2024-04-26T17:56:41.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:41 smithi194 ceph-mon[33017]: pgmap v12129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:41 smithi033 ceph-mon[27845]: pgmap v12129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:41 smithi096 ceph-mon[32753]: pgmap v12129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:43.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:43 smithi194 ceph-mon[33017]: pgmap v12130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21942.030839 secs 2024-04-26T17:56:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:43 smithi033 ceph-mon[27845]: pgmap v12130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21942.030839 secs 2024-04-26T17:56:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:43 smithi096 ceph-mon[32753]: pgmap v12130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21942.030839 secs 2024-04-26T17:56:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21944.892920 secs 2024-04-26T17:56:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:45 smithi194 ceph-mon[33017]: pgmap v12131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21944.892920 secs 2024-04-26T17:56:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:45 smithi033 ceph-mon[27845]: pgmap v12131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21944.892920 secs 2024-04-26T17:56:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:45 smithi096 ceph-mon[32753]: pgmap v12131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:47.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:47 smithi194 ceph-mon[33017]: pgmap v12132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:47 smithi033 ceph-mon[27845]: pgmap v12132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:47 smithi096 ceph-mon[32753]: pgmap v12132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:48.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:56:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21947.030922 secs 2024-04-26T17:56:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:56:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21947.030922 secs 2024-04-26T17:56:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:56:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21947.030922 secs 2024-04-26T17:56:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:48.883+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:48.883+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:56:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:56:48.883+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:56:49.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:49 smithi194 ceph-mon[33017]: pgmap v12133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:49 smithi194 ceph-mon[33017]: pgmap v12134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:49 smithi033 ceph-mon[27845]: pgmap v12133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:49 smithi033 ceph-mon[27845]: pgmap v12134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:49 smithi096 ceph-mon[32753]: pgmap v12133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:56:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:49 smithi096 ceph-mon[32753]: pgmap v12134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:50.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21949.893011 secs 2024-04-26T17:56:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21949.893011 secs 2024-04-26T17:56:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21949.893011 secs 2024-04-26T17:56:51.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:51 smithi194 ceph-mon[33017]: pgmap v12135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:51 smithi033 ceph-mon[27845]: pgmap v12135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:51 smithi096 ceph-mon[32753]: pgmap v12135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:53.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:53 smithi194 ceph-mon[33017]: pgmap v12136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21952.031003 secs 2024-04-26T17:56:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:53 smithi033 ceph-mon[27845]: pgmap v12136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21952.031003 secs 2024-04-26T17:56:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:53 smithi096 ceph-mon[32753]: pgmap v12136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21952.031003 secs 2024-04-26T17:56:55.477 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21954.893132 secs 2024-04-26T17:56:55.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:55 smithi033 ceph-mon[27845]: pgmap v12137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21954.893132 secs 2024-04-26T17:56:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:55 smithi194 ceph-mon[33017]: pgmap v12137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21954.893132 secs 2024-04-26T17:56:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:55 smithi096 ceph-mon[32753]: pgmap v12137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:57.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:57 smithi194 ceph-mon[33017]: pgmap v12138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:57 smithi033 ceph-mon[27845]: pgmap v12138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:57 smithi096 ceph-mon[32753]: pgmap v12138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:56:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21957.031062 secs 2024-04-26T17:56:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21957.031062 secs 2024-04-26T17:56:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21957.031062 secs 2024-04-26T17:56:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:56:59 smithi194 ceph-mon[33017]: pgmap v12139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:56:59.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:56:59 smithi033 ceph-mon[27845]: pgmap v12139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:56:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:56:59 smithi096 ceph-mon[32753]: pgmap v12139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:57:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21959.893252 secs 2024-04-26T17:57:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21959.893252 secs 2024-04-26T17:57:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21959.893252 secs 2024-04-26T17:57:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:01 smithi194 ceph-mon[33017]: pgmap v12140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:01 smithi033 ceph-mon[27845]: pgmap v12140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:01 smithi096 ceph-mon[32753]: pgmap v12140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:03 smithi194 ceph-mon[33017]: pgmap v12141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21962.031121 secs 2024-04-26T17:57:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:03 smithi033 ceph-mon[27845]: pgmap v12141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:03.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21962.031121 secs 2024-04-26T17:57:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:03 smithi096 ceph-mon[32753]: pgmap v12141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21962.031121 secs 2024-04-26T17:57:03.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:03.898+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:03.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:03.898+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:57:03.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:03.898+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21964.893346 secs 2024-04-26T17:57:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:05 smithi033 ceph-mon[27845]: pgmap v12142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21964.893346 secs 2024-04-26T17:57:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:05 smithi096 ceph-mon[32753]: pgmap v12142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21964.893346 secs 2024-04-26T17:57:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:05 smithi194 ceph-mon[33017]: pgmap v12142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:07 smithi033 ceph-mon[27845]: pgmap v12143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:07 smithi096 ceph-mon[32753]: pgmap v12143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:07 smithi194 ceph-mon[33017]: pgmap v12143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:08.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21967.031182 secs 2024-04-26T17:57:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21967.031182 secs 2024-04-26T17:57:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21967.031182 secs 2024-04-26T17:57:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:09 smithi033 ceph-mon[27845]: pgmap v12144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:09 smithi096 ceph-mon[32753]: pgmap v12144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:09 smithi194 ceph-mon[33017]: pgmap v12144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21969.893426 secs 2024-04-26T17:57:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21969.893426 secs 2024-04-26T17:57:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21969.893426 secs 2024-04-26T17:57:11.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:11 smithi033 ceph-mon[27845]: pgmap v12145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:11 smithi096 ceph-mon[32753]: pgmap v12145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:11 smithi194 ceph-mon[33017]: pgmap v12145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:13 smithi033 ceph-mon[27845]: pgmap v12146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21972.031240 secs 2024-04-26T17:57:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:13 smithi096 ceph-mon[32753]: pgmap v12146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21972.031240 secs 2024-04-26T17:57:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:13 smithi194 ceph-mon[33017]: pgmap v12146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21972.031240 secs 2024-04-26T17:57:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21974.893547 secs 2024-04-26T17:57:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21974.893547 secs 2024-04-26T17:57:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21974.893547 secs 2024-04-26T17:57:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:15 smithi033 ceph-mon[27845]: pgmap v12147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:15 smithi096 ceph-mon[32753]: pgmap v12147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:15 smithi194 ceph-mon[33017]: pgmap v12147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:17 smithi033 ceph-mon[27845]: pgmap v12148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:17 smithi096 ceph-mon[32753]: pgmap v12148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:17 smithi194 ceph-mon[33017]: pgmap v12148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21977.031320 secs 2024-04-26T17:57:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21977.031320 secs 2024-04-26T17:57:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21977.031320 secs 2024-04-26T17:57:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:18.913+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:18.913+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:57:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:18.913+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:19 smithi033 ceph-mon[27845]: pgmap v12149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21979.893671 secs 2024-04-26T17:57:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:19 smithi096 ceph-mon[32753]: pgmap v12149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21979.893671 secs 2024-04-26T17:57:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:19 smithi194 ceph-mon[33017]: pgmap v12149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21979.893671 secs 2024-04-26T17:57:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:21 smithi096 ceph-mon[32753]: pgmap v12150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:21.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:21 smithi194 ceph-mon[33017]: pgmap v12150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:21 smithi033 ceph-mon[27845]: pgmap v12150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:23 smithi194 ceph-mon[33017]: pgmap v12151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21982.031389 secs 2024-04-26T17:57:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:23 smithi033 ceph-mon[27845]: pgmap v12151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21982.031389 secs 2024-04-26T17:57:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:23 smithi096 ceph-mon[32753]: pgmap v12151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21982.031389 secs 2024-04-26T17:57:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21984.893808 secs 2024-04-26T17:57:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21984.893808 secs 2024-04-26T17:57:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21984.893808 secs 2024-04-26T17:57:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:25 smithi194 ceph-mon[33017]: pgmap v12152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:25 smithi033 ceph-mon[27845]: pgmap v12152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:25 smithi096 ceph-mon[32753]: pgmap v12152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:27 smithi194 ceph-mon[33017]: pgmap v12153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:27 smithi033 ceph-mon[27845]: pgmap v12153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:27 smithi096 ceph-mon[32753]: pgmap v12153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21987.031469 secs 2024-04-26T17:57:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21987.031469 secs 2024-04-26T17:57:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21987.031469 secs 2024-04-26T17:57:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:29 smithi033 ceph-mon[27845]: pgmap v12154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21989.893878 secs 2024-04-26T17:57:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:29 smithi194 ceph-mon[33017]: pgmap v12154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21989.893878 secs 2024-04-26T17:57:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:29 smithi096 ceph-mon[32753]: pgmap v12154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21989.893878 secs 2024-04-26T17:57:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:31 smithi194 ceph-mon[33017]: pgmap v12155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:31 smithi033 ceph-mon[27845]: pgmap v12155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:31 smithi096 ceph-mon[32753]: pgmap v12155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:33 smithi194 ceph-mon[33017]: pgmap v12156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21992.031550 secs 2024-04-26T17:57:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:33 smithi033 ceph-mon[27845]: pgmap v12156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21992.031550 secs 2024-04-26T17:57:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:33.929+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:33.929+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:57:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:33.929+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:33 smithi096 ceph-mon[32753]: pgmap v12156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21992.031550 secs 2024-04-26T17:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21994.894003 secs 2024-04-26T17:57:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21994.894003 secs 2024-04-26T17:57:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21994.894003 secs 2024-04-26T17:57:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:35 smithi194 ceph-mon[33017]: pgmap v12157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:35 smithi033 ceph-mon[27845]: pgmap v12157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:35 smithi096 ceph-mon[32753]: pgmap v12157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:37 smithi194 ceph-mon[33017]: pgmap v12158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:37 smithi033 ceph-mon[27845]: pgmap v12158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:37 smithi096 ceph-mon[32753]: pgmap v12158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 21997.031628 secs 2024-04-26T17:57:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 21997.031628 secs 2024-04-26T17:57:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 21997.031628 secs 2024-04-26T17:57:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:39 smithi194 ceph-mon[33017]: pgmap v12159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 21999.894141 secs 2024-04-26T17:57:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:39 smithi033 ceph-mon[27845]: pgmap v12159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 21999.894141 secs 2024-04-26T17:57:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:39 smithi096 ceph-mon[32753]: pgmap v12159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 21999.894141 secs 2024-04-26T17:57:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:41 smithi194 ceph-mon[33017]: pgmap v12160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:41 smithi033 ceph-mon[27845]: pgmap v12160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:41 smithi096 ceph-mon[32753]: pgmap v12160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:43 smithi194 ceph-mon[33017]: pgmap v12161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22002.031724 secs 2024-04-26T17:57:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:43 smithi033 ceph-mon[27845]: pgmap v12161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22002.031724 secs 2024-04-26T17:57:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:43 smithi096 ceph-mon[32753]: pgmap v12161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22002.031724 secs 2024-04-26T17:57:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22004.894239 secs 2024-04-26T17:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22004.894239 secs 2024-04-26T17:57:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22004.894239 secs 2024-04-26T17:57:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:45 smithi194 ceph-mon[33017]: pgmap v12162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:45 smithi033 ceph-mon[27845]: pgmap v12162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:45 smithi096 ceph-mon[32753]: pgmap v12162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:47 smithi194 ceph-mon[33017]: pgmap v12163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:47 smithi033 ceph-mon[27845]: pgmap v12163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:47 smithi096 ceph-mon[32753]: pgmap v12163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22007.031806 secs 2024-04-26T17:57:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22007.031806 secs 2024-04-26T17:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:57:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22007.031806 secs 2024-04-26T17:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:57:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:48.945+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:48.945+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:57:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:57:48.945+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:57:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:49 smithi096 ceph-mon[32753]: pgmap v12164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22009.894331 secs 2024-04-26T17:57:49.780 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:49 smithi194 ceph-mon[33017]: pgmap v12164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:49.780 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22009.894331 secs 2024-04-26T17:57:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:49 smithi033 ceph-mon[27845]: pgmap v12164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22009.894331 secs 2024-04-26T17:57:51.633 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:51 smithi194 ceph-mon[33017]: pgmap v12165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:51 smithi033 ceph-mon[27845]: pgmap v12165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:51.929 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:51 smithi096 ceph-mon[32753]: pgmap v12165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:57:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:53 smithi033 ceph-mon[27845]: pgmap v12166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22012.031894 secs 2024-04-26T17:57:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:53 smithi096 ceph-mon[32753]: pgmap v12166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22012.031894 secs 2024-04-26T17:57:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:53 smithi194 ceph-mon[33017]: pgmap v12166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22012.031894 secs 2024-04-26T17:57:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22014.894452 secs 2024-04-26T17:57:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22014.894452 secs 2024-04-26T17:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22014.894452 secs 2024-04-26T17:57:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:55 smithi033 ceph-mon[27845]: pgmap v12167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:57:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:57:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:55 smithi096 ceph-mon[32753]: pgmap v12167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:57:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:57:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:55 smithi194 ceph-mon[33017]: pgmap v12167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:57:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:57:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:57:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:57:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:56 smithi033 ceph-mon[27845]: pgmap v12168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:57:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:56 smithi033 ceph-mon[27845]: pgmap v12169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:57:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:56 smithi096 ceph-mon[32753]: pgmap v12168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:57:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:56 smithi096 ceph-mon[32753]: pgmap v12169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:57:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:56 smithi194 ceph-mon[33017]: pgmap v12168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T17:57:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:56 smithi194 ceph-mon[33017]: pgmap v12169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:57:58.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:58 smithi033 ceph-mon[27845]: pgmap v12170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:57:58.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22017.031976 secs 2024-04-26T17:57:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:58 smithi096 ceph-mon[32753]: pgmap v12170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:57:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22017.031976 secs 2024-04-26T17:57:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:58 smithi194 ceph-mon[33017]: pgmap v12170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:57:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22017.031976 secs 2024-04-26T17:57:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22019.894578 secs 2024-04-26T17:57:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22019.894578 secs 2024-04-26T17:57:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22019.894578 secs 2024-04-26T17:58:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:00 smithi033 ceph-mon[27845]: pgmap v12171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T17:58:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:00 smithi096 ceph-mon[32753]: pgmap v12171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T17:58:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:00 smithi194 ceph-mon[33017]: pgmap v12171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T17:58:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:02 smithi033 ceph-mon[27845]: pgmap v12172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:02 smithi096 ceph-mon[32753]: pgmap v12172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:02 smithi194 ceph-mon[33017]: pgmap v12172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22022.032004 secs 2024-04-26T17:58:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22022.032004 secs 2024-04-26T17:58:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22022.032004 secs 2024-04-26T17:58:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:03.961+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:03.961+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:03.961+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:58:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:04 smithi194 ceph-mon[33017]: pgmap v12173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22024.894694 secs 2024-04-26T17:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:04 smithi033 ceph-mon[27845]: pgmap v12173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22024.894694 secs 2024-04-26T17:58:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:04 smithi096 ceph-mon[32753]: pgmap v12173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22024.894694 secs 2024-04-26T17:58:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:06 smithi194 ceph-mon[33017]: pgmap v12174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:58:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:06 smithi033 ceph-mon[27845]: pgmap v12174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:58:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:06 smithi096 ceph-mon[32753]: pgmap v12174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T17:58:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:08 smithi194 ceph-mon[33017]: pgmap v12175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22027.032064 secs 2024-04-26T17:58:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:08 smithi033 ceph-mon[27845]: pgmap v12175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22027.032064 secs 2024-04-26T17:58:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:08 smithi096 ceph-mon[32753]: pgmap v12175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22027.032064 secs 2024-04-26T17:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22029.894806 secs 2024-04-26T17:58:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22029.894806 secs 2024-04-26T17:58:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22029.894806 secs 2024-04-26T17:58:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:10 smithi194 ceph-mon[33017]: pgmap v12176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:10 smithi033 ceph-mon[27845]: pgmap v12176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:10 smithi096 ceph-mon[32753]: pgmap v12176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:12.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:12 smithi194 ceph-mon[33017]: pgmap v12177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:12 smithi033 ceph-mon[27845]: pgmap v12177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:12 smithi096 ceph-mon[32753]: pgmap v12177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22032.032127 secs 2024-04-26T17:58:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22032.032127 secs 2024-04-26T17:58:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22032.032127 secs 2024-04-26T17:58:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:14 smithi194 ceph-mon[33017]: pgmap v12178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22034.894948 secs 2024-04-26T17:58:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:14 smithi033 ceph-mon[27845]: pgmap v12178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22034.894948 secs 2024-04-26T17:58:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:14 smithi096 ceph-mon[32753]: pgmap v12178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22034.894948 secs 2024-04-26T17:58:16.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:16 smithi194 ceph-mon[33017]: pgmap v12179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:16 smithi033 ceph-mon[27845]: pgmap v12179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:16 smithi096 ceph-mon[32753]: pgmap v12179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:18 smithi194 ceph-mon[33017]: pgmap v12180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22037.032188 secs 2024-04-26T17:58:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:18 smithi033 ceph-mon[27845]: pgmap v12180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22037.032188 secs 2024-04-26T17:58:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:18 smithi096 ceph-mon[32753]: pgmap v12180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22037.032188 secs 2024-04-26T17:58:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:18.977+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:18.977+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:18.977+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:58:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22039.895068 secs 2024-04-26T17:58:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22039.895068 secs 2024-04-26T17:58:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22039.895068 secs 2024-04-26T17:58:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:20 smithi194 ceph-mon[33017]: pgmap v12181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:20 smithi033 ceph-mon[27845]: pgmap v12181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:20 smithi096 ceph-mon[32753]: pgmap v12181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:22.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:22 smithi194 ceph-mon[33017]: pgmap v12182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:22 smithi033 ceph-mon[27845]: pgmap v12182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:22 smithi096 ceph-mon[32753]: pgmap v12182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:23.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22042.032265 secs 2024-04-26T17:58:23.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22042.032265 secs 2024-04-26T17:58:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22042.032265 secs 2024-04-26T17:58:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:24 smithi194 ceph-mon[33017]: pgmap v12183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22044.895212 secs 2024-04-26T17:58:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:24 smithi033 ceph-mon[27845]: pgmap v12183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22044.895212 secs 2024-04-26T17:58:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:24 smithi096 ceph-mon[32753]: pgmap v12183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22044.895212 secs 2024-04-26T17:58:26.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:26 smithi194 ceph-mon[33017]: pgmap v12184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:26 smithi033 ceph-mon[27845]: pgmap v12184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:26 smithi096 ceph-mon[32753]: pgmap v12184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:28 smithi194 ceph-mon[33017]: pgmap v12185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22047.032325 secs 2024-04-26T17:58:28.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:28 smithi033 ceph-mon[27845]: pgmap v12185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:28.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22047.032325 secs 2024-04-26T17:58:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:28 smithi096 ceph-mon[32753]: pgmap v12185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22047.032325 secs 2024-04-26T17:58:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22049.895304 secs 2024-04-26T17:58:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22049.895304 secs 2024-04-26T17:58:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22049.895304 secs 2024-04-26T17:58:30.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:30 smithi033 ceph-mon[27845]: pgmap v12186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:30 smithi194 ceph-mon[33017]: pgmap v12186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:30 smithi096 ceph-mon[32753]: pgmap v12186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:32 smithi194 ceph-mon[33017]: pgmap v12187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:32 smithi033 ceph-mon[27845]: pgmap v12187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:32 smithi096 ceph-mon[32753]: pgmap v12187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22052.032384 secs 2024-04-26T17:58:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22052.032384 secs 2024-04-26T17:58:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22052.032384 secs 2024-04-26T17:58:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:33.992+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:33.992+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:58:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:33.992+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:34 smithi194 ceph-mon[33017]: pgmap v12188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22054.895414 secs 2024-04-26T17:58:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:34 smithi033 ceph-mon[27845]: pgmap v12188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22054.895414 secs 2024-04-26T17:58:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:34 smithi096 ceph-mon[32753]: pgmap v12188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22054.895414 secs 2024-04-26T17:58:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:36 smithi194 ceph-mon[33017]: pgmap v12189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:36 smithi033 ceph-mon[27845]: pgmap v12189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:36 smithi096 ceph-mon[32753]: pgmap v12189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:38 smithi194 ceph-mon[33017]: pgmap v12190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22057.032467 secs 2024-04-26T17:58:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:38 smithi033 ceph-mon[27845]: pgmap v12190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22057.032467 secs 2024-04-26T17:58:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:38 smithi096 ceph-mon[32753]: pgmap v12190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22057.032467 secs 2024-04-26T17:58:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22059.895535 secs 2024-04-26T17:58:39.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22059.895535 secs 2024-04-26T17:58:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22059.895535 secs 2024-04-26T17:58:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:40 smithi194 ceph-mon[33017]: pgmap v12191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:40 smithi033 ceph-mon[27845]: pgmap v12191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:40 smithi096 ceph-mon[32753]: pgmap v12191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:42 smithi194 ceph-mon[33017]: pgmap v12192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:42 smithi033 ceph-mon[27845]: pgmap v12192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:42 smithi096 ceph-mon[32753]: pgmap v12192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:58:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22062.032555 secs 2024-04-26T17:58:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22062.032555 secs 2024-04-26T17:58:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22062.032555 secs 2024-04-26T17:58:44.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:44 smithi194 ceph-mon[33017]: pgmap v12193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22064.895658 secs 2024-04-26T17:58:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:44 smithi033 ceph-mon[27845]: pgmap v12193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22064.895658 secs 2024-04-26T17:58:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:44 smithi096 ceph-mon[32753]: pgmap v12193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22064.895658 secs 2024-04-26T17:58:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:46 smithi033 ceph-mon[27845]: pgmap v12194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:46 smithi096 ceph-mon[32753]: pgmap v12194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:46 smithi194 ceph-mon[33017]: pgmap v12194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:48.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:48 smithi194 ceph-mon[33017]: pgmap v12195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:48.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22067.032640 secs 2024-04-26T17:58:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:48 smithi033 ceph-mon[27845]: pgmap v12195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22067.032640 secs 2024-04-26T17:58:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:48 smithi096 ceph-mon[32753]: pgmap v12195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22067.032640 secs 2024-04-26T17:58:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:49.008+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:49.008+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:58:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:58:49.008+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:58:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22069.895792 secs 2024-04-26T17:58:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22069.895792 secs 2024-04-26T17:58:50.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22069.895792 secs 2024-04-26T17:58:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:50 smithi033 ceph-mon[27845]: pgmap v12196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:50 smithi096 ceph-mon[32753]: pgmap v12196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:50 smithi194 ceph-mon[33017]: pgmap v12196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:52 smithi033 ceph-mon[27845]: pgmap v12197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:52 smithi096 ceph-mon[32753]: pgmap v12197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:52 smithi194 ceph-mon[33017]: pgmap v12197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22072.032723 secs 2024-04-26T17:58:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22072.032723 secs 2024-04-26T17:58:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22072.032723 secs 2024-04-26T17:58:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:54 smithi033 ceph-mon[27845]: pgmap v12198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22074.895938 secs 2024-04-26T17:58:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:54 smithi096 ceph-mon[32753]: pgmap v12198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22074.895938 secs 2024-04-26T17:58:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:54 smithi194 ceph-mon[33017]: pgmap v12198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:58:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22074.895938 secs 2024-04-26T17:58:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:58:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:58:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:58:56.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:56 smithi033 ceph-mon[27845]: pgmap v12199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:56 smithi096 ceph-mon[32753]: pgmap v12199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:56 smithi194 ceph-mon[33017]: pgmap v12199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:58:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:58:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:58:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:58:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:58:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:58:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:58:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:58:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:58:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:58:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:58 smithi033 ceph-mon[27845]: pgmap v12200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:58 smithi033 ceph-mon[27845]: pgmap v12201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22077.032811 secs 2024-04-26T17:58:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:58 smithi096 ceph-mon[32753]: pgmap v12200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:58 smithi096 ceph-mon[32753]: pgmap v12201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22077.032811 secs 2024-04-26T17:58:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:58 smithi194 ceph-mon[33017]: pgmap v12200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:58 smithi194 ceph-mon[33017]: pgmap v12201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22077.032811 secs 2024-04-26T17:58:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22079.896062 secs 2024-04-26T17:59:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22079.896062 secs 2024-04-26T17:59:00.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22079.896062 secs 2024-04-26T17:59:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:00 smithi096 ceph-mon[32753]: pgmap v12202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:59:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:00 smithi194 ceph-mon[33017]: pgmap v12202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:59:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:00 smithi033 ceph-mon[27845]: pgmap v12202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:59:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:02 smithi194 ceph-mon[33017]: pgmap v12203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:02 smithi033 ceph-mon[27845]: pgmap v12203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:02 smithi096 ceph-mon[32753]: pgmap v12203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:04.015+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:04.015+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:59:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:04.015+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22082.032872 secs 2024-04-26T17:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22082.032872 secs 2024-04-26T17:59:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22082.032872 secs 2024-04-26T17:59:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:04 smithi194 ceph-mon[33017]: pgmap v12204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22084.896181 secs 2024-04-26T17:59:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:04 smithi033 ceph-mon[27845]: pgmap v12204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22084.896181 secs 2024-04-26T17:59:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:04 smithi096 ceph-mon[32753]: pgmap v12204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22084.896181 secs 2024-04-26T17:59:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:06 smithi194 ceph-mon[33017]: pgmap v12205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:06 smithi033 ceph-mon[27845]: pgmap v12205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:06 smithi096 ceph-mon[32753]: pgmap v12205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T17:59:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:08 smithi194 ceph-mon[33017]: pgmap v12206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:59:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22087.032956 secs 2024-04-26T17:59:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:08 smithi033 ceph-mon[27845]: pgmap v12206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:59:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22087.032956 secs 2024-04-26T17:59:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:08 smithi096 ceph-mon[32753]: pgmap v12206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T17:59:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22087.032956 secs 2024-04-26T17:59:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22089.896296 secs 2024-04-26T17:59:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22089.896296 secs 2024-04-26T17:59:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22089.896296 secs 2024-04-26T17:59:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:10 smithi194 ceph-mon[33017]: pgmap v12207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:59:11.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:10 smithi033 ceph-mon[27845]: pgmap v12207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:59:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:10 smithi096 ceph-mon[32753]: pgmap v12207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T17:59:13.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:12 smithi194 ceph-mon[33017]: pgmap v12208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:59:13.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:12 smithi033 ceph-mon[27845]: pgmap v12208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:59:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:12 smithi096 ceph-mon[32753]: pgmap v12208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T17:59:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22092.033011 secs 2024-04-26T17:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22092.033011 secs 2024-04-26T17:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22092.033011 secs 2024-04-26T17:59:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:14 smithi194 ceph-mon[33017]: pgmap v12209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22094.896390 secs 2024-04-26T17:59:15.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:14 smithi033 ceph-mon[27845]: pgmap v12209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22094.896390 secs 2024-04-26T17:59:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:14 smithi096 ceph-mon[32753]: pgmap v12209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22094.896390 secs 2024-04-26T17:59:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:15 smithi194 ceph-mon[33017]: pgmap v12210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:15 smithi033 ceph-mon[27845]: pgmap v12210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:15 smithi096 ceph-mon[32753]: pgmap v12210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:17 smithi096 ceph-mon[32753]: pgmap v12211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:18.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:17 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22097.033072 secs 2024-04-26T17:59:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:17 smithi194 ceph-mon[33017]: pgmap v12211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:17 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22097.033072 secs 2024-04-26T17:59:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:17 smithi033 ceph-mon[27845]: pgmap v12211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:17 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22097.033072 secs 2024-04-26T17:59:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:19.020+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:19.020+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:59:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:19.020+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22099.896481 secs 2024-04-26T17:59:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:19 smithi096 ceph-mon[32753]: pgmap v12212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22099.896481 secs 2024-04-26T17:59:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:19 smithi194 ceph-mon[33017]: pgmap v12212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22099.896481 secs 2024-04-26T17:59:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:19 smithi033 ceph-mon[27845]: pgmap v12212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:22.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:21 smithi194 ceph-mon[33017]: pgmap v12213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:21 smithi033 ceph-mon[27845]: pgmap v12213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:21 smithi096 ceph-mon[32753]: pgmap v12213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:23.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:22 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22102.033152 secs 2024-04-26T17:59:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:22 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22102.033152 secs 2024-04-26T17:59:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:22 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22102.033152 secs 2024-04-26T17:59:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:23 smithi194 ceph-mon[33017]: pgmap v12214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:23 smithi033 ceph-mon[27845]: pgmap v12214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:23 smithi096 ceph-mon[32753]: pgmap v12214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22104.896584 secs 2024-04-26T17:59:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22104.896584 secs 2024-04-26T17:59:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22104.896584 secs 2024-04-26T17:59:25.556 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:25.556+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T17:59:25.556 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:25.556+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T17:59:25.557 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:25.556+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 176 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T17:59:25.557 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:25.556+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 176 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T17:59:26.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:25 smithi194 ceph-mon[33017]: pgmap v12215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:25 smithi033 ceph-mon[27845]: pgmap v12215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:25 smithi096 ceph-mon[32753]: pgmap v12215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:27 smithi194 ceph-mon[33017]: pgmap v12216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22107.033213 secs 2024-04-26T17:59:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:27 smithi033 ceph-mon[27845]: pgmap v12216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22107.033213 secs 2024-04-26T17:59:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:27 smithi096 ceph-mon[32753]: pgmap v12216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22107.033213 secs 2024-04-26T17:59:30.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22109.896723 secs 2024-04-26T17:59:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:29 smithi194 ceph-mon[33017]: pgmap v12217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22109.896723 secs 2024-04-26T17:59:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:29 smithi033 ceph-mon[27845]: pgmap v12217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22109.896723 secs 2024-04-26T17:59:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:29 smithi096 ceph-mon[32753]: pgmap v12217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:31 smithi194 ceph-mon[33017]: pgmap v12218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:31 smithi033 ceph-mon[27845]: pgmap v12218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:31 smithi096 ceph-mon[32753]: pgmap v12218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22112.033294 secs 2024-04-26T17:59:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22112.033294 secs 2024-04-26T17:59:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22112.033294 secs 2024-04-26T17:59:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:34.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:34.031+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:59:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:34.031+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:34.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:34 smithi194 ceph-mon[33017]: pgmap v12219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:34 smithi033 ceph-mon[27845]: pgmap v12219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:34 smithi096 ceph-mon[32753]: pgmap v12219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:35.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22114.896858 secs 2024-04-26T17:59:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22114.896858 secs 2024-04-26T17:59:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22114.896858 secs 2024-04-26T17:59:36.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:36 smithi194 ceph-mon[33017]: pgmap v12220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:36 smithi033 ceph-mon[27845]: pgmap v12220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:36 smithi096 ceph-mon[32753]: pgmap v12220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:38 smithi194 ceph-mon[33017]: pgmap v12221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22117.033357 secs 2024-04-26T17:59:38.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:38 smithi033 ceph-mon[27845]: pgmap v12221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22117.033357 secs 2024-04-26T17:59:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:38 smithi096 ceph-mon[32753]: pgmap v12221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22117.033357 secs 2024-04-26T17:59:40.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22119.896954 secs 2024-04-26T17:59:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:40 smithi194 ceph-mon[33017]: pgmap v12222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:40.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22119.896954 secs 2024-04-26T17:59:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:40 smithi033 ceph-mon[27845]: pgmap v12222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22119.896954 secs 2024-04-26T17:59:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:40 smithi096 ceph-mon[32753]: pgmap v12222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:42 smithi194 ceph-mon[33017]: pgmap v12223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:42 smithi033 ceph-mon[27845]: pgmap v12223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:42 smithi096 ceph-mon[32753]: pgmap v12223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:43.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22122.033418 secs 2024-04-26T17:59:43.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22122.033418 secs 2024-04-26T17:59:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22122.033418 secs 2024-04-26T17:59:44.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:44 smithi194 ceph-mon[33017]: pgmap v12224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:44 smithi033 ceph-mon[27845]: pgmap v12224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:44 smithi096 ceph-mon[32753]: pgmap v12224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22124.897066 secs 2024-04-26T17:59:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22124.897066 secs 2024-04-26T17:59:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22124.897066 secs 2024-04-26T17:59:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:46 smithi194 ceph-mon[33017]: pgmap v12225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:46.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:46 smithi033 ceph-mon[27845]: pgmap v12225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:46 smithi096 ceph-mon[32753]: pgmap v12225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:48.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:48 smithi194 ceph-mon[33017]: pgmap v12226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22127.033501 secs 2024-04-26T17:59:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:48 smithi033 ceph-mon[27845]: pgmap v12226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22127.033501 secs 2024-04-26T17:59:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:48 smithi096 ceph-mon[32753]: pgmap v12226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22127.033501 secs 2024-04-26T17:59:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:49.033+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:49.033+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T17:59:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T17:59:49.033+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T17:59:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22129.897131 secs 2024-04-26T17:59:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:50 smithi194 ceph-mon[33017]: pgmap v12227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22129.897131 secs 2024-04-26T17:59:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:50 smithi033 ceph-mon[27845]: pgmap v12227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22129.897131 secs 2024-04-26T17:59:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:50 smithi096 ceph-mon[32753]: pgmap v12227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:52.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:52 smithi194 ceph-mon[33017]: pgmap v12228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:52 smithi033 ceph-mon[27845]: pgmap v12228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:52 smithi096 ceph-mon[32753]: pgmap v12228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:53.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22132.033583 secs 2024-04-26T17:59:53.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22132.033583 secs 2024-04-26T17:59:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22132.033583 secs 2024-04-26T17:59:54.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:54 smithi194 ceph-mon[33017]: pgmap v12229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:54 smithi033 ceph-mon[27845]: pgmap v12229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:54 smithi096 ceph-mon[32753]: pgmap v12229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:55.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22134.897240 secs 2024-04-26T17:59:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22134.897240 secs 2024-04-26T17:59:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22134.897240 secs 2024-04-26T17:59:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:56 smithi194 ceph-mon[33017]: pgmap v12230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:56 smithi033 ceph-mon[27845]: pgmap v12230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:56 smithi096 ceph-mon[32753]: pgmap v12230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:59:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:58 smithi194 ceph-mon[33017]: pgmap v12231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:59:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22137.033668 secs 2024-04-26T17:59:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:58 smithi033 ceph-mon[27845]: pgmap v12231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:59:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22137.033668 secs 2024-04-26T17:59:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:58 smithi096 ceph-mon[32753]: pgmap v12231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:59:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22137.033668 secs 2024-04-26T17:59:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:59:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:59:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:59 smithi033 ceph-mon[27845]: pgmap v12232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:59 smithi033 ceph-mon[27845]: pgmap v12233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:59:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 17:59:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:59:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:59 smithi096 ceph-mon[32753]: pgmap v12232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:59 smithi096 ceph-mon[32753]: pgmap v12233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 17:59:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T17:59:59.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:59 smithi194 ceph-mon[33017]: pgmap v12232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T17:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:59 smithi194 ceph-mon[33017]: pgmap v12233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T17:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 17:59:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:00:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22139.897335 secs 2024-04-26T18:00:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:00:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22139.897335 secs 2024-04-26T18:00:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:00:00.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22139.897335 secs 2024-04-26T18:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:00:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:01 smithi033 ceph-mon[27845]: pgmap v12234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:01 smithi096 ceph-mon[32753]: pgmap v12234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:01 smithi194 ceph-mon[33017]: pgmap v12234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:03 smithi194 ceph-mon[33017]: pgmap v12235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22142.033754 secs 2024-04-26T18:00:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:03 smithi033 ceph-mon[27845]: pgmap v12235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22142.033754 secs 2024-04-26T18:00:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:03 smithi096 ceph-mon[32753]: pgmap v12235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22142.033754 secs 2024-04-26T18:00:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:04.044+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:04.044+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:04.044+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:00:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22144.897459 secs 2024-04-26T18:00:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22144.897459 secs 2024-04-26T18:00:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22144.897459 secs 2024-04-26T18:00:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:05 smithi033 ceph-mon[27845]: pgmap v12236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:05 smithi096 ceph-mon[32753]: pgmap v12236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:05 smithi194 ceph-mon[33017]: pgmap v12236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:07 smithi033 ceph-mon[27845]: pgmap v12237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:07 smithi096 ceph-mon[32753]: pgmap v12237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:07 smithi194 ceph-mon[33017]: pgmap v12237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:00:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22147.033841 secs 2024-04-26T18:00:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22147.033841 secs 2024-04-26T18:00:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22147.033841 secs 2024-04-26T18:00:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22149.897576 secs 2024-04-26T18:00:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:09 smithi033 ceph-mon[27845]: pgmap v12238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:00:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22149.897576 secs 2024-04-26T18:00:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:09 smithi096 ceph-mon[32753]: pgmap v12238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:00:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22149.897576 secs 2024-04-26T18:00:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:09 smithi194 ceph-mon[33017]: pgmap v12238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:00:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:11 smithi033 ceph-mon[27845]: pgmap v12239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:11 smithi096 ceph-mon[32753]: pgmap v12239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:11 smithi194 ceph-mon[33017]: pgmap v12239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:13 smithi033 ceph-mon[27845]: pgmap v12240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22152.033922 secs 2024-04-26T18:00:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:13 smithi096 ceph-mon[32753]: pgmap v12240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22152.033922 secs 2024-04-26T18:00:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:13 smithi194 ceph-mon[33017]: pgmap v12240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22152.033922 secs 2024-04-26T18:00:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22154.897693 secs 2024-04-26T18:00:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22154.897693 secs 2024-04-26T18:00:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22154.897693 secs 2024-04-26T18:00:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:15 smithi033 ceph-mon[27845]: pgmap v12241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:15 smithi096 ceph-mon[32753]: pgmap v12241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:15 smithi194 ceph-mon[33017]: pgmap v12241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:17 smithi033 ceph-mon[27845]: pgmap v12242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:17 smithi096 ceph-mon[32753]: pgmap v12242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:17 smithi194 ceph-mon[33017]: pgmap v12242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22157.034028 secs 2024-04-26T18:00:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22157.034028 secs 2024-04-26T18:00:19.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22157.034028 secs 2024-04-26T18:00:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:19.049+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:19.049+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:00:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:19.049+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22159.897790 secs 2024-04-26T18:00:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:19 smithi033 ceph-mon[27845]: pgmap v12243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22159.897790 secs 2024-04-26T18:00:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:19 smithi096 ceph-mon[32753]: pgmap v12243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22159.897790 secs 2024-04-26T18:00:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:19 smithi194 ceph-mon[33017]: pgmap v12243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:21 smithi033 ceph-mon[27845]: pgmap v12244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:21 smithi096 ceph-mon[32753]: pgmap v12244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:21 smithi194 ceph-mon[33017]: pgmap v12244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:23 smithi096 ceph-mon[32753]: pgmap v12245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22162.034088 secs 2024-04-26T18:00:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:23 smithi194 ceph-mon[33017]: pgmap v12245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22162.034088 secs 2024-04-26T18:00:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:23 smithi033 ceph-mon[27845]: pgmap v12245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22162.034088 secs 2024-04-26T18:00:25.005 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22164.897906 secs 2024-04-26T18:00:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22164.897906 secs 2024-04-26T18:00:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22164.897906 secs 2024-04-26T18:00:25.956 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:25 smithi096 ceph-mon[32753]: pgmap v12246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:26.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:25 smithi194 ceph-mon[33017]: pgmap v12246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:25 smithi033 ceph-mon[27845]: pgmap v12246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:27 smithi194 ceph-mon[33017]: pgmap v12247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:27 smithi033 ceph-mon[27845]: pgmap v12247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:27 smithi096 ceph-mon[32753]: pgmap v12247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22167.034171 secs 2024-04-26T18:00:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22167.034171 secs 2024-04-26T18:00:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22167.034171 secs 2024-04-26T18:00:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22169.898024 secs 2024-04-26T18:00:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:29 smithi033 ceph-mon[27845]: pgmap v12248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22169.898024 secs 2024-04-26T18:00:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:29 smithi194 ceph-mon[33017]: pgmap v12248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22169.898024 secs 2024-04-26T18:00:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:29 smithi096 ceph-mon[32753]: pgmap v12248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:31 smithi194 ceph-mon[33017]: pgmap v12249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:31 smithi033 ceph-mon[27845]: pgmap v12249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:31 smithi096 ceph-mon[32753]: pgmap v12249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:34.063+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:34.064+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:00:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:34.064+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:33 smithi194 ceph-mon[33017]: pgmap v12250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22172.034231 secs 2024-04-26T18:00:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:33 smithi033 ceph-mon[27845]: pgmap v12250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22172.034231 secs 2024-04-26T18:00:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:33 smithi096 ceph-mon[32753]: pgmap v12250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22172.034231 secs 2024-04-26T18:00:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22174.898141 secs 2024-04-26T18:00:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22174.898141 secs 2024-04-26T18:00:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22174.898141 secs 2024-04-26T18:00:36.015 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:35 smithi096 ceph-mon[32753]: pgmap v12251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:35 smithi194 ceph-mon[33017]: pgmap v12251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:35 smithi033 ceph-mon[27845]: pgmap v12251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:37 smithi194 ceph-mon[33017]: pgmap v12252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:37 smithi033 ceph-mon[27845]: pgmap v12252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:37 smithi096 ceph-mon[32753]: pgmap v12252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22177.034316 secs 2024-04-26T18:00:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22177.034316 secs 2024-04-26T18:00:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22177.034316 secs 2024-04-26T18:00:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22179.898272 secs 2024-04-26T18:00:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:39 smithi194 ceph-mon[33017]: pgmap v12253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22179.898272 secs 2024-04-26T18:00:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:39 smithi033 ceph-mon[27845]: pgmap v12253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22179.898272 secs 2024-04-26T18:00:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:39 smithi096 ceph-mon[32753]: pgmap v12253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:41 smithi194 ceph-mon[33017]: pgmap v12254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:41 smithi033 ceph-mon[27845]: pgmap v12254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:41 smithi096 ceph-mon[32753]: pgmap v12254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:42 smithi194 ceph-mon[33017]: pgmap v12255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:42 smithi033 ceph-mon[27845]: pgmap v12255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:42 smithi096 ceph-mon[32753]: pgmap v12255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22182.034375 secs 2024-04-26T18:00:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22182.034375 secs 2024-04-26T18:00:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22182.034375 secs 2024-04-26T18:00:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22184.898387 secs 2024-04-26T18:00:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:44 smithi194 ceph-mon[33017]: pgmap v12256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22184.898387 secs 2024-04-26T18:00:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:44 smithi033 ceph-mon[27845]: pgmap v12256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22184.898387 secs 2024-04-26T18:00:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:44 smithi096 ceph-mon[32753]: pgmap v12256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:47 smithi033 ceph-mon[27845]: pgmap v12257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:47 smithi096 ceph-mon[32753]: pgmap v12257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:47 smithi194 ceph-mon[33017]: pgmap v12257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22187.034460 secs 2024-04-26T18:00:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22187.034460 secs 2024-04-26T18:00:49.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22187.034460 secs 2024-04-26T18:00:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:49.077+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:49.077+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:00:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:49.077+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:00:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22189.898508 secs 2024-04-26T18:00:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:49 smithi033 ceph-mon[27845]: pgmap v12258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22189.898508 secs 2024-04-26T18:00:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:49 smithi096 ceph-mon[32753]: pgmap v12258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22189.898508 secs 2024-04-26T18:00:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:49 smithi194 ceph-mon[33017]: pgmap v12258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:51 smithi033 ceph-mon[27845]: pgmap v12259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:51 smithi096 ceph-mon[32753]: pgmap v12259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:51 smithi194 ceph-mon[33017]: pgmap v12259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:53 smithi194 ceph-mon[33017]: pgmap v12260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22192.034543 secs 2024-04-26T18:00:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:53 smithi033 ceph-mon[27845]: pgmap v12260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22192.034543 secs 2024-04-26T18:00:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:53 smithi096 ceph-mon[32753]: pgmap v12260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22192.034543 secs 2024-04-26T18:00:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22194.898585 secs 2024-04-26T18:00:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22194.898585 secs 2024-04-26T18:00:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22194.898585 secs 2024-04-26T18:00:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:55.571+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:00:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:55.571+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:00:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:55.572+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 177 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T18:00:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:00:55.572+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 177 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:00:55.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:55 smithi033 ceph-mon[27845]: pgmap v12261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:55 smithi096 ceph-mon[32753]: pgmap v12261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:55 smithi194 ceph-mon[33017]: pgmap v12261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:00:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:57 smithi033 ceph-mon[27845]: pgmap v12262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:57 smithi096 ceph-mon[32753]: pgmap v12262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:57 smithi194 ceph-mon[33017]: pgmap v12262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22197.034602 secs 2024-04-26T18:00:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22197.034602 secs 2024-04-26T18:00:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22197.034602 secs 2024-04-26T18:00:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22199.898707 secs 2024-04-26T18:00:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:59 smithi033 ceph-mon[27845]: pgmap v12263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:00:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:00:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22199.898707 secs 2024-04-26T18:00:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:59 smithi096 ceph-mon[32753]: pgmap v12263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:00:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:00:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:01:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22199.898707 secs 2024-04-26T18:01:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:59 smithi194 ceph-mon[33017]: pgmap v12263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:00:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:01:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:01:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:01:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:01:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:01:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:01:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:01:01.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:01:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:01:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:01:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:01 smithi194 ceph-mon[33017]: pgmap v12264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:01:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:01 smithi194 ceph-mon[33017]: pgmap v12265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:01 smithi033 ceph-mon[27845]: pgmap v12264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:01:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:01 smithi033 ceph-mon[27845]: pgmap v12265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:01 smithi096 ceph-mon[32753]: pgmap v12264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:01:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:01 smithi096 ceph-mon[32753]: pgmap v12265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:04.090+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:04.090+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:04.090+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:01:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:03 smithi194 ceph-mon[33017]: pgmap v12266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22202.034683 secs 2024-04-26T18:01:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:03 smithi033 ceph-mon[27845]: pgmap v12266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22202.034683 secs 2024-04-26T18:01:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:03 smithi096 ceph-mon[32753]: pgmap v12266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22202.034683 secs 2024-04-26T18:01:05.071 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22204.898835 secs 2024-04-26T18:01:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22204.898835 secs 2024-04-26T18:01:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22204.898835 secs 2024-04-26T18:01:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:05 smithi194 ceph-mon[33017]: pgmap v12267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:01:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:05 smithi033 ceph-mon[27845]: pgmap v12267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:01:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:05 smithi096 ceph-mon[32753]: pgmap v12267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:01:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:07 smithi194 ceph-mon[33017]: pgmap v12268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:07 smithi033 ceph-mon[27845]: pgmap v12268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:07 smithi096 ceph-mon[32753]: pgmap v12268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22207.034767 secs 2024-04-26T18:01:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22207.034767 secs 2024-04-26T18:01:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22207.034767 secs 2024-04-26T18:01:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:09 smithi194 ceph-mon[33017]: pgmap v12269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22209.898959 secs 2024-04-26T18:01:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:09 smithi033 ceph-mon[27845]: pgmap v12269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22209.898959 secs 2024-04-26T18:01:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:09 smithi096 ceph-mon[32753]: pgmap v12269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:01:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22209.898959 secs 2024-04-26T18:01:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:11 smithi194 ceph-mon[33017]: pgmap v12270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:01:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:11 smithi033 ceph-mon[27845]: pgmap v12270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:01:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:11 smithi096 ceph-mon[32753]: pgmap v12270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:13 smithi194 ceph-mon[33017]: pgmap v12271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22212.034850 secs 2024-04-26T18:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:13 smithi033 ceph-mon[27845]: pgmap v12271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22212.034850 secs 2024-04-26T18:01:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:13 smithi096 ceph-mon[32753]: pgmap v12271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22212.034850 secs 2024-04-26T18:01:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22214.899056 secs 2024-04-26T18:01:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22214.899056 secs 2024-04-26T18:01:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22214.899056 secs 2024-04-26T18:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:15 smithi194 ceph-mon[33017]: pgmap v12272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:15 smithi033 ceph-mon[27845]: pgmap v12272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:15 smithi096 ceph-mon[32753]: pgmap v12272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:17 smithi194 ceph-mon[33017]: pgmap v12273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:17 smithi033 ceph-mon[27845]: pgmap v12273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:17 smithi096 ceph-mon[32753]: pgmap v12273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:19.103+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:19.103+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:19.103+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:01:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22217.034934 secs 2024-04-26T18:01:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22217.034934 secs 2024-04-26T18:01:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22217.034934 secs 2024-04-26T18:01:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:19 smithi194 ceph-mon[33017]: pgmap v12274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22219.899185 secs 2024-04-26T18:01:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:19 smithi033 ceph-mon[27845]: pgmap v12274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22219.899185 secs 2024-04-26T18:01:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:19 smithi096 ceph-mon[32753]: pgmap v12274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22219.899185 secs 2024-04-26T18:01:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:21 smithi194 ceph-mon[33017]: pgmap v12275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:21 smithi033 ceph-mon[27845]: pgmap v12275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:21 smithi096 ceph-mon[32753]: pgmap v12275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:23.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:22 smithi096 ceph-mon[32753]: pgmap v12276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:22 smithi194 ceph-mon[33017]: pgmap v12276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:22 smithi033 ceph-mon[27845]: pgmap v12276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22222.035000 secs 2024-04-26T18:01:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22222.035000 secs 2024-04-26T18:01:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22222.035000 secs 2024-04-26T18:01:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22224.899330 secs 2024-04-26T18:01:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:24 smithi194 ceph-mon[33017]: pgmap v12277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22224.899330 secs 2024-04-26T18:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:24 smithi033 ceph-mon[27845]: pgmap v12277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22224.899330 secs 2024-04-26T18:01:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:24 smithi096 ceph-mon[32753]: pgmap v12277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:27 smithi194 ceph-mon[33017]: pgmap v12278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:27 smithi033 ceph-mon[27845]: pgmap v12278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:27 smithi096 ceph-mon[32753]: pgmap v12278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22227.035084 secs 2024-04-26T18:01:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22227.035084 secs 2024-04-26T18:01:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22227.035084 secs 2024-04-26T18:01:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:29 smithi194 ceph-mon[33017]: pgmap v12279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:29 smithi033 ceph-mon[27845]: pgmap v12279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:29 smithi096 ceph-mon[32753]: pgmap v12279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:30.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22229.899450 secs 2024-04-26T18:01:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22229.899450 secs 2024-04-26T18:01:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22229.899450 secs 2024-04-26T18:01:31.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:31 smithi194 ceph-mon[33017]: pgmap v12280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:31 smithi033 ceph-mon[27845]: pgmap v12280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:31 smithi096 ceph-mon[32753]: pgmap v12280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:33.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:33 smithi194 ceph-mon[33017]: pgmap v12281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22232.035143 secs 2024-04-26T18:01:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:33 smithi033 ceph-mon[27845]: pgmap v12281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22232.035143 secs 2024-04-26T18:01:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:33 smithi096 ceph-mon[32753]: pgmap v12281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22232.035143 secs 2024-04-26T18:01:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:34.110+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:34.110+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:01:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:34.110+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22234.899598 secs 2024-04-26T18:01:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:35 smithi194 ceph-mon[33017]: pgmap v12282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22234.899598 secs 2024-04-26T18:01:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:35 smithi033 ceph-mon[27845]: pgmap v12282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22234.899598 secs 2024-04-26T18:01:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:35 smithi096 ceph-mon[32753]: pgmap v12282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:37 smithi194 ceph-mon[33017]: pgmap v12283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:37 smithi033 ceph-mon[27845]: pgmap v12283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:37 smithi096 ceph-mon[32753]: pgmap v12283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:38.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22237.035204 secs 2024-04-26T18:01:38.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22237.035204 secs 2024-04-26T18:01:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22237.035204 secs 2024-04-26T18:01:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:39 smithi194 ceph-mon[33017]: pgmap v12284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:39 smithi033 ceph-mon[27845]: pgmap v12284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:39 smithi096 ceph-mon[32753]: pgmap v12284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22239.899723 secs 2024-04-26T18:01:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22239.899723 secs 2024-04-26T18:01:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22239.899723 secs 2024-04-26T18:01:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:41 smithi033 ceph-mon[27845]: pgmap v12285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:41 smithi096 ceph-mon[32753]: pgmap v12285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:41 smithi194 ceph-mon[33017]: pgmap v12285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:43 smithi033 ceph-mon[27845]: pgmap v12286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22242.035286 secs 2024-04-26T18:01:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:43 smithi096 ceph-mon[32753]: pgmap v12286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22242.035286 secs 2024-04-26T18:01:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:43 smithi194 ceph-mon[33017]: pgmap v12286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22242.035286 secs 2024-04-26T18:01:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22244.899790 secs 2024-04-26T18:01:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:45 smithi033 ceph-mon[27845]: pgmap v12287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22244.899790 secs 2024-04-26T18:01:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:45 smithi096 ceph-mon[32753]: pgmap v12287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22244.899790 secs 2024-04-26T18:01:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:45 smithi194 ceph-mon[33017]: pgmap v12287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:47 smithi033 ceph-mon[27845]: pgmap v12288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:47 smithi096 ceph-mon[32753]: pgmap v12288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:47 smithi194 ceph-mon[33017]: pgmap v12288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22247.035348 secs 2024-04-26T18:01:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22247.035348 secs 2024-04-26T18:01:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22247.035348 secs 2024-04-26T18:01:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:49.119+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:49.119+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:01:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:01:49.119+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:01:49.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:49 smithi033 ceph-mon[27845]: pgmap v12289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:49 smithi096 ceph-mon[32753]: pgmap v12289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:49.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:49 smithi194 ceph-mon[33017]: pgmap v12289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22249.899906 secs 2024-04-26T18:01:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22249.899906 secs 2024-04-26T18:01:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22249.899906 secs 2024-04-26T18:01:51.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:51 smithi033 ceph-mon[27845]: pgmap v12290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:51 smithi096 ceph-mon[32753]: pgmap v12290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:51.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:51 smithi194 ceph-mon[33017]: pgmap v12290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:53 smithi033 ceph-mon[27845]: pgmap v12291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22252.035404 secs 2024-04-26T18:01:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:53 smithi096 ceph-mon[32753]: pgmap v12291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22252.035404 secs 2024-04-26T18:01:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:53 smithi194 ceph-mon[33017]: pgmap v12291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22252.035404 secs 2024-04-26T18:01:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22254.900021 secs 2024-04-26T18:01:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:55 smithi033 ceph-mon[27845]: pgmap v12292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22254.900021 secs 2024-04-26T18:01:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:55 smithi096 ceph-mon[32753]: pgmap v12292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22254.900021 secs 2024-04-26T18:01:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:55 smithi194 ceph-mon[33017]: pgmap v12292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:57.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:57 smithi033 ceph-mon[27845]: pgmap v12293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:57 smithi096 ceph-mon[32753]: pgmap v12293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:57 smithi194 ceph-mon[33017]: pgmap v12293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:01:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22257.035461 secs 2024-04-26T18:01:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22257.035461 secs 2024-04-26T18:01:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22257.035461 secs 2024-04-26T18:01:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:59 smithi096 ceph-mon[32753]: pgmap v12294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22259.900168 secs 2024-04-26T18:01:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:59 smithi194 ceph-mon[33017]: pgmap v12294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22259.900168 secs 2024-04-26T18:01:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:59 smithi033 ceph-mon[27845]: pgmap v12294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:01:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22259.900168 secs 2024-04-26T18:02:01.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:01 smithi033 ceph-mon[27845]: pgmap v12295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:01.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:02:01.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:01 smithi194 ceph-mon[33017]: pgmap v12295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:02:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:01 smithi096 ceph-mon[32753]: pgmap v12295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:02:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:02:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:02:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:02:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:02:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:02:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:02:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:02:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:02:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:03 smithi194 ceph-mon[33017]: pgmap v12296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:03 smithi194 ceph-mon[33017]: pgmap v12297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22262.035551 secs 2024-04-26T18:02:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:03 smithi033 ceph-mon[27845]: pgmap v12296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:03 smithi033 ceph-mon[27845]: pgmap v12297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22262.035551 secs 2024-04-26T18:02:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:03 smithi096 ceph-mon[32753]: pgmap v12296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:03 smithi096 ceph-mon[32753]: pgmap v12297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22262.035551 secs 2024-04-26T18:02:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:04.129+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:04.129+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:04.129+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:02:04.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22264.900286 secs 2024-04-26T18:02:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22264.900286 secs 2024-04-26T18:02:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22264.900286 secs 2024-04-26T18:02:05.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:05 smithi194 ceph-mon[33017]: pgmap v12298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:02:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:05 smithi033 ceph-mon[27845]: pgmap v12298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:02:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:05 smithi096 ceph-mon[32753]: pgmap v12298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:02:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:07 smithi194 ceph-mon[33017]: pgmap v12299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:07 smithi033 ceph-mon[27845]: pgmap v12299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:07 smithi096 ceph-mon[32753]: pgmap v12299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:08.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22267.035635 secs 2024-04-26T18:02:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22267.035635 secs 2024-04-26T18:02:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22267.035635 secs 2024-04-26T18:02:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:09 smithi194 ceph-mon[33017]: pgmap v12300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22269.900349 secs 2024-04-26T18:02:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:09 smithi033 ceph-mon[27845]: pgmap v12300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22269.900349 secs 2024-04-26T18:02:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:09 smithi096 ceph-mon[32753]: pgmap v12300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22269.900349 secs 2024-04-26T18:02:11.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:11 smithi194 ceph-mon[33017]: pgmap v12301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:11 smithi033 ceph-mon[27845]: pgmap v12301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:11 smithi096 ceph-mon[32753]: pgmap v12301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:02:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:13 smithi194 ceph-mon[33017]: pgmap v12302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:02:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22272.035718 secs 2024-04-26T18:02:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:13 smithi033 ceph-mon[27845]: pgmap v12302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:02:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22272.035718 secs 2024-04-26T18:02:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:13 smithi096 ceph-mon[32753]: pgmap v12302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:02:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22272.035718 secs 2024-04-26T18:02:14.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22274.900460 secs 2024-04-26T18:02:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22274.900460 secs 2024-04-26T18:02:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22274.900460 secs 2024-04-26T18:02:15.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:15 smithi194 ceph-mon[33017]: pgmap v12303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:15 smithi033 ceph-mon[27845]: pgmap v12303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:15 smithi096 ceph-mon[32753]: pgmap v12303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:17 smithi194 ceph-mon[33017]: pgmap v12304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:17 smithi033 ceph-mon[27845]: pgmap v12304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:17 smithi096 ceph-mon[32753]: pgmap v12304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:18.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22277.035802 secs 2024-04-26T18:02:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22277.035802 secs 2024-04-26T18:02:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22277.035802 secs 2024-04-26T18:02:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:19.145+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:19.145+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:02:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:19.145+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:19.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:19 smithi194 ceph-mon[33017]: pgmap v12305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22279.900557 secs 2024-04-26T18:02:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:19 smithi033 ceph-mon[27845]: pgmap v12305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22279.900557 secs 2024-04-26T18:02:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:19 smithi096 ceph-mon[32753]: pgmap v12305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22279.900557 secs 2024-04-26T18:02:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:21 smithi194 ceph-mon[33017]: pgmap v12306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:21 smithi033 ceph-mon[27845]: pgmap v12306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:21 smithi096 ceph-mon[32753]: pgmap v12306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:23 smithi194 ceph-mon[33017]: pgmap v12307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22282.035862 secs 2024-04-26T18:02:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:23 smithi033 ceph-mon[27845]: pgmap v12307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22282.035862 secs 2024-04-26T18:02:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:23 smithi096 ceph-mon[32753]: pgmap v12307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22282.035862 secs 2024-04-26T18:02:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22284.900674 secs 2024-04-26T18:02:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22284.900674 secs 2024-04-26T18:02:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22284.900674 secs 2024-04-26T18:02:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:25.586+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:02:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:25.586+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:02:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:25.586+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 178 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:02:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:25.586+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 178 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:02:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:25 smithi194 ceph-mon[33017]: pgmap v12308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:25 smithi033 ceph-mon[27845]: pgmap v12308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:25 smithi096 ceph-mon[32753]: pgmap v12308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:27.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:27 smithi194 ceph-mon[33017]: pgmap v12309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:27 smithi033 ceph-mon[27845]: pgmap v12309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:27 smithi096 ceph-mon[32753]: pgmap v12309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22287.035946 secs 2024-04-26T18:02:28.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22287.035946 secs 2024-04-26T18:02:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22287.035946 secs 2024-04-26T18:02:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:29 smithi194 ceph-mon[33017]: pgmap v12310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22289.900799 secs 2024-04-26T18:02:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:29 smithi033 ceph-mon[27845]: pgmap v12310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22289.900799 secs 2024-04-26T18:02:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:29 smithi096 ceph-mon[32753]: pgmap v12310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22289.900799 secs 2024-04-26T18:02:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:31 smithi194 ceph-mon[33017]: pgmap v12311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:31 smithi033 ceph-mon[27845]: pgmap v12311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:31 smithi096 ceph-mon[32753]: pgmap v12311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:33 smithi194 ceph-mon[33017]: pgmap v12312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22292.036034 secs 2024-04-26T18:02:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:33 smithi033 ceph-mon[27845]: pgmap v12312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22292.036034 secs 2024-04-26T18:02:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:33 smithi096 ceph-mon[32753]: pgmap v12312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22292.036034 secs 2024-04-26T18:02:34.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:34.147+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:34.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:34.147+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:34.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:34.147+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:02:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22294.900934 secs 2024-04-26T18:02:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22294.900934 secs 2024-04-26T18:02:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22294.900934 secs 2024-04-26T18:02:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:35 smithi194 ceph-mon[33017]: pgmap v12313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:35 smithi033 ceph-mon[27845]: pgmap v12313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:35 smithi096 ceph-mon[32753]: pgmap v12313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:37 smithi194 ceph-mon[33017]: pgmap v12314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:37 smithi033 ceph-mon[27845]: pgmap v12314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:37 smithi096 ceph-mon[32753]: pgmap v12314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22297.036119 secs 2024-04-26T18:02:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22297.036119 secs 2024-04-26T18:02:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22297.036119 secs 2024-04-26T18:02:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:39 smithi033 ceph-mon[27845]: pgmap v12315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22299.901050 secs 2024-04-26T18:02:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:39 smithi096 ceph-mon[32753]: pgmap v12315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22299.901050 secs 2024-04-26T18:02:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:39 smithi194 ceph-mon[33017]: pgmap v12315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22299.901050 secs 2024-04-26T18:02:41.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:41 smithi033 ceph-mon[27845]: pgmap v12316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:41.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:41 smithi096 ceph-mon[32753]: pgmap v12316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:41 smithi194 ceph-mon[33017]: pgmap v12316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:43 smithi033 ceph-mon[27845]: pgmap v12317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22302.036179 secs 2024-04-26T18:02:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:43 smithi096 ceph-mon[32753]: pgmap v12317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22302.036179 secs 2024-04-26T18:02:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:43 smithi194 ceph-mon[33017]: pgmap v12317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22302.036179 secs 2024-04-26T18:02:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22304.901167 secs 2024-04-26T18:02:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22304.901167 secs 2024-04-26T18:02:45.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22304.901167 secs 2024-04-26T18:02:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:45 smithi033 ceph-mon[27845]: pgmap v12318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:45 smithi096 ceph-mon[32753]: pgmap v12318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:45 smithi194 ceph-mon[33017]: pgmap v12318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:47.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:47 smithi033 ceph-mon[27845]: pgmap v12319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:47 smithi096 ceph-mon[32753]: pgmap v12319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:47 smithi194 ceph-mon[33017]: pgmap v12319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22307.036242 secs 2024-04-26T18:02:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22307.036242 secs 2024-04-26T18:02:49.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22307.036242 secs 2024-04-26T18:02:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:49.154+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:49.154+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:02:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:02:49.154+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:02:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:49 smithi033 ceph-mon[27845]: pgmap v12320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22309.901287 secs 2024-04-26T18:02:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:49 smithi096 ceph-mon[32753]: pgmap v12320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22309.901287 secs 2024-04-26T18:02:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:49 smithi194 ceph-mon[33017]: pgmap v12320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22309.901287 secs 2024-04-26T18:02:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:51 smithi033 ceph-mon[27845]: pgmap v12321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:51 smithi096 ceph-mon[32753]: pgmap v12321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:51 smithi194 ceph-mon[33017]: pgmap v12321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:53 smithi033 ceph-mon[27845]: pgmap v12322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22312.036299 secs 2024-04-26T18:02:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:53 smithi096 ceph-mon[32753]: pgmap v12322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22312.036299 secs 2024-04-26T18:02:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:53 smithi194 ceph-mon[33017]: pgmap v12322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22312.036299 secs 2024-04-26T18:02:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22314.901409 secs 2024-04-26T18:02:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22314.901409 secs 2024-04-26T18:02:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22314.901409 secs 2024-04-26T18:02:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:55 smithi194 ceph-mon[33017]: pgmap v12323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:56.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:55 smithi033 ceph-mon[27845]: pgmap v12323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:55 smithi096 ceph-mon[32753]: pgmap v12323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:02:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:57 smithi194 ceph-mon[33017]: pgmap v12324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:57 smithi033 ceph-mon[27845]: pgmap v12324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:57 smithi096 ceph-mon[32753]: pgmap v12324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:02:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22317.036381 secs 2024-04-26T18:02:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22317.036381 secs 2024-04-26T18:02:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22317.036381 secs 2024-04-26T18:03:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:59 smithi194 ceph-mon[33017]: pgmap v12325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22319.901523 secs 2024-04-26T18:03:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:59 smithi033 ceph-mon[27845]: pgmap v12325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22319.901523 secs 2024-04-26T18:03:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:59 smithi096 ceph-mon[32753]: pgmap v12325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22319.901523 secs 2024-04-26T18:03:01.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:01 smithi033 ceph-mon[27845]: pgmap v12326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:01 smithi194 ceph-mon[33017]: pgmap v12326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:01 smithi096 ceph-mon[32753]: pgmap v12326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:03:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:03:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:03:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:03 smithi194 ceph-mon[33017]: pgmap v12327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22322.036446 secs 2024-04-26T18:03:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:04.169+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:04.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:04.169+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:03:04.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:04.169+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:03 smithi033 ceph-mon[27845]: pgmap v12327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22322.036446 secs 2024-04-26T18:03:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:03 smithi096 ceph-mon[32753]: pgmap v12327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22322.036446 secs 2024-04-26T18:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22324.901606 secs 2024-04-26T18:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22324.901606 secs 2024-04-26T18:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22324.901606 secs 2024-04-26T18:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:03:06.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:05 smithi194 ceph-mon[33017]: pgmap v12328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:05 smithi194 ceph-mon[33017]: pgmap v12329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:03:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:05 smithi194 ceph-mon[33017]: pgmap v12330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:05 smithi033 ceph-mon[27845]: pgmap v12328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:05 smithi033 ceph-mon[27845]: pgmap v12329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:03:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:05 smithi033 ceph-mon[27845]: pgmap v12330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:05 smithi096 ceph-mon[32753]: pgmap v12328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:05 smithi096 ceph-mon[32753]: pgmap v12329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:03:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:05 smithi096 ceph-mon[32753]: pgmap v12330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:07 smithi194 ceph-mon[33017]: pgmap v12331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:07 smithi033 ceph-mon[27845]: pgmap v12331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:07 smithi096 ceph-mon[32753]: pgmap v12331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22327.036525 secs 2024-04-26T18:03:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22327.036525 secs 2024-04-26T18:03:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22327.036525 secs 2024-04-26T18:03:10.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22329.901731 secs 2024-04-26T18:03:10.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:09 smithi033 ceph-mon[27845]: pgmap v12332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:03:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22329.901731 secs 2024-04-26T18:03:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:09 smithi194 ceph-mon[33017]: pgmap v12332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:03:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22329.901731 secs 2024-04-26T18:03:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:09 smithi096 ceph-mon[32753]: pgmap v12332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:03:12.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:11 smithi194 ceph-mon[33017]: pgmap v12333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:11 smithi033 ceph-mon[27845]: pgmap v12333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:11 smithi096 ceph-mon[32753]: pgmap v12333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:13 smithi194 ceph-mon[33017]: pgmap v12334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22332.036605 secs 2024-04-26T18:03:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:13 smithi033 ceph-mon[27845]: pgmap v12334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22332.036605 secs 2024-04-26T18:03:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:13 smithi096 ceph-mon[32753]: pgmap v12334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:03:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22332.036605 secs 2024-04-26T18:03:15.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22334.901808 secs 2024-04-26T18:03:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22334.901808 secs 2024-04-26T18:03:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22334.901808 secs 2024-04-26T18:03:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:15 smithi194 ceph-mon[33017]: pgmap v12335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:03:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:15 smithi033 ceph-mon[27845]: pgmap v12335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:03:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:15 smithi096 ceph-mon[32753]: pgmap v12335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:03:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:17 smithi194 ceph-mon[33017]: pgmap v12336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:18.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:17 smithi033 ceph-mon[27845]: pgmap v12336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:17 smithi096 ceph-mon[32753]: pgmap v12336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22337.036688 secs 2024-04-26T18:03:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22337.036688 secs 2024-04-26T18:03:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22337.036688 secs 2024-04-26T18:03:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:19.185+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:19.185+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:19.185+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:03:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22339.901928 secs 2024-04-26T18:03:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:19 smithi194 ceph-mon[33017]: pgmap v12337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22339.901928 secs 2024-04-26T18:03:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:19 smithi033 ceph-mon[27845]: pgmap v12337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22339.901928 secs 2024-04-26T18:03:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:19 smithi096 ceph-mon[32753]: pgmap v12337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:20 smithi194 ceph-mon[33017]: pgmap v12338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:20 smithi033 ceph-mon[27845]: pgmap v12338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:20 smithi096 ceph-mon[32753]: pgmap v12338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:23 smithi194 ceph-mon[33017]: pgmap v12339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22342.036772 secs 2024-04-26T18:03:23.885 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:23 smithi096 ceph-mon[32753]: pgmap v12339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:23.885 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22342.036772 secs 2024-04-26T18:03:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:23 smithi033 ceph-mon[27845]: pgmap v12339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22342.036772 secs 2024-04-26T18:03:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22344.902080 secs 2024-04-26T18:03:24.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22344.902080 secs 2024-04-26T18:03:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22344.902080 secs 2024-04-26T18:03:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:25 smithi194 ceph-mon[33017]: pgmap v12340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:25 smithi033 ceph-mon[27845]: pgmap v12340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:25 smithi096 ceph-mon[32753]: pgmap v12340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:27 smithi033 ceph-mon[27845]: pgmap v12341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:27 smithi096 ceph-mon[32753]: pgmap v12341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:28.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:27 smithi194 ceph-mon[33017]: pgmap v12341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22347.036856 secs 2024-04-26T18:03:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22347.036856 secs 2024-04-26T18:03:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22347.036856 secs 2024-04-26T18:03:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22349.902198 secs 2024-04-26T18:03:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:29 smithi033 ceph-mon[27845]: pgmap v12342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22349.902198 secs 2024-04-26T18:03:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:29 smithi096 ceph-mon[32753]: pgmap v12342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22349.902198 secs 2024-04-26T18:03:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:29 smithi194 ceph-mon[33017]: pgmap v12342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:31 smithi033 ceph-mon[27845]: pgmap v12343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:31 smithi096 ceph-mon[32753]: pgmap v12343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:32.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:31 smithi194 ceph-mon[33017]: pgmap v12343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:33 smithi033 ceph-mon[27845]: pgmap v12344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22352.036938 secs 2024-04-26T18:03:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:33 smithi096 ceph-mon[32753]: pgmap v12344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22352.036938 secs 2024-04-26T18:03:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:33 smithi194 ceph-mon[33017]: pgmap v12344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22352.036938 secs 2024-04-26T18:03:34.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:34.199+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:34.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:34.199+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:03:34.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:34.199+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22354.902321 secs 2024-04-26T18:03:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22354.902321 secs 2024-04-26T18:03:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22354.902321 secs 2024-04-26T18:03:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:35 smithi033 ceph-mon[27845]: pgmap v12345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:35 smithi096 ceph-mon[32753]: pgmap v12345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:35 smithi194 ceph-mon[33017]: pgmap v12345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:37 smithi033 ceph-mon[27845]: pgmap v12346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:37 smithi096 ceph-mon[32753]: pgmap v12346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:37 smithi194 ceph-mon[33017]: pgmap v12346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22357.037004 secs 2024-04-26T18:03:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22357.037004 secs 2024-04-26T18:03:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22357.037004 secs 2024-04-26T18:03:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22359.902438 secs 2024-04-26T18:03:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:39 smithi033 ceph-mon[27845]: pgmap v12347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22359.902438 secs 2024-04-26T18:03:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:39 smithi096 ceph-mon[32753]: pgmap v12347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22359.902438 secs 2024-04-26T18:03:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:39 smithi194 ceph-mon[33017]: pgmap v12347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:41 smithi033 ceph-mon[27845]: pgmap v12348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:41 smithi096 ceph-mon[32753]: pgmap v12348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:41 smithi194 ceph-mon[33017]: pgmap v12348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:43 smithi033 ceph-mon[27845]: pgmap v12349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22362.037065 secs 2024-04-26T18:03:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:43 smithi096 ceph-mon[32753]: pgmap v12349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22362.037065 secs 2024-04-26T18:03:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:43 smithi194 ceph-mon[33017]: pgmap v12349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22362.037065 secs 2024-04-26T18:03:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22364.902555 secs 2024-04-26T18:03:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22364.902555 secs 2024-04-26T18:03:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22364.902555 secs 2024-04-26T18:03:46.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:45 smithi194 ceph-mon[33017]: pgmap v12350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:45 smithi033 ceph-mon[27845]: pgmap v12350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:45 smithi096 ceph-mon[32753]: pgmap v12350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:47 smithi194 ceph-mon[33017]: pgmap v12351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:47 smithi033 ceph-mon[27845]: pgmap v12351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:47 smithi096 ceph-mon[32753]: pgmap v12351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:49.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22367.037150 secs 2024-04-26T18:03:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22367.037150 secs 2024-04-26T18:03:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22367.037150 secs 2024-04-26T18:03:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:49.216+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:49.216+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:03:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:03:49.216+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:03:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22369.902682 secs 2024-04-26T18:03:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:49 smithi194 ceph-mon[33017]: pgmap v12352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22369.902682 secs 2024-04-26T18:03:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:49 smithi033 ceph-mon[27845]: pgmap v12352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22369.902682 secs 2024-04-26T18:03:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:49 smithi096 ceph-mon[32753]: pgmap v12352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:52.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:51 smithi194 ceph-mon[33017]: pgmap v12353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:51 smithi033 ceph-mon[27845]: pgmap v12353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:51 smithi096 ceph-mon[32753]: pgmap v12353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:53.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:53 smithi096 ceph-mon[32753]: pgmap v12354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:53.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22372.037210 secs 2024-04-26T18:03:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:53 smithi194 ceph-mon[33017]: pgmap v12354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22372.037210 secs 2024-04-26T18:03:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:53 smithi033 ceph-mon[27845]: pgmap v12354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22372.037210 secs 2024-04-26T18:03:55.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22374.902797 secs 2024-04-26T18:03:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22374.902797 secs 2024-04-26T18:03:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22374.902797 secs 2024-04-26T18:03:56.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:55 smithi033 ceph-mon[27845]: pgmap v12355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:56.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:55 smithi194 ceph-mon[33017]: pgmap v12355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:55 smithi096 ceph-mon[32753]: pgmap v12355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:03:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:57 smithi194 ceph-mon[33017]: pgmap v12356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:57 smithi033 ceph-mon[27845]: pgmap v12356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:57 smithi096 ceph-mon[32753]: pgmap v12356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:03:59.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22377.037266 secs 2024-04-26T18:03:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22377.037266 secs 2024-04-26T18:03:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22377.037266 secs 2024-04-26T18:04:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22379.902914 secs 2024-04-26T18:04:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:03:59 smithi194 ceph-mon[33017]: pgmap v12357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22379.902914 secs 2024-04-26T18:04:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:03:59 smithi033 ceph-mon[27845]: pgmap v12357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22379.902914 secs 2024-04-26T18:04:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:03:59 smithi096 ceph-mon[32753]: pgmap v12357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:01 smithi194 ceph-mon[33017]: pgmap v12358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:01 smithi033 ceph-mon[27845]: pgmap v12358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:01 smithi096 ceph-mon[32753]: pgmap v12358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:03 smithi194 ceph-mon[33017]: pgmap v12359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22382.037350 secs 2024-04-26T18:04:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:03 smithi033 ceph-mon[27845]: pgmap v12359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22382.037350 secs 2024-04-26T18:04:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:03 smithi096 ceph-mon[32753]: pgmap v12359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22382.037350 secs 2024-04-26T18:04:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:04.231+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:04.231+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:04:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:04.231+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22384.903052 secs 2024-04-26T18:04:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22384.903052 secs 2024-04-26T18:04:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22384.903052 secs 2024-04-26T18:04:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:05 smithi194 ceph-mon[33017]: pgmap v12360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:04:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:05 smithi033 ceph-mon[27845]: pgmap v12360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:04:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:05 smithi096 ceph-mon[32753]: pgmap v12360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:04:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:07 smithi194 ceph-mon[33017]: pgmap v12361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:04:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:04:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:07 smithi033 ceph-mon[27845]: pgmap v12361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:04:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:04:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:07 smithi096 ceph-mon[32753]: pgmap v12361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:04:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:04:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:04:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:08 smithi194 ceph-mon[33017]: pgmap v12362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:04:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:08 smithi194 ceph-mon[33017]: pgmap v12363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22387.037412 secs 2024-04-26T18:04:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:08 smithi033 ceph-mon[27845]: pgmap v12362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:04:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:08 smithi033 ceph-mon[27845]: pgmap v12363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22387.037412 secs 2024-04-26T18:04:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:08 smithi096 ceph-mon[32753]: pgmap v12362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:04:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:08 smithi096 ceph-mon[32753]: pgmap v12363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22387.037412 secs 2024-04-26T18:04:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22389.903154 secs 2024-04-26T18:04:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22389.903154 secs 2024-04-26T18:04:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22389.903154 secs 2024-04-26T18:04:11.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:10 smithi033 ceph-mon[27845]: pgmap v12364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:04:11.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:10 smithi194 ceph-mon[33017]: pgmap v12364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:04:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:10 smithi096 ceph-mon[32753]: pgmap v12364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:04:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:11 smithi033 ceph-mon[27845]: pgmap v12365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:11 smithi096 ceph-mon[32753]: pgmap v12365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:11 smithi194 ceph-mon[33017]: pgmap v12365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:12 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22392.037471 secs 2024-04-26T18:04:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:12 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22392.037471 secs 2024-04-26T18:04:13.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:12 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22392.037471 secs 2024-04-26T18:04:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:13 smithi096 ceph-mon[32753]: pgmap v12366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:04:14.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:13 smithi194 ceph-mon[33017]: pgmap v12366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:04:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:13 smithi033 ceph-mon[27845]: pgmap v12366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:04:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22394.903269 secs 2024-04-26T18:04:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22394.903269 secs 2024-04-26T18:04:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22394.903269 secs 2024-04-26T18:04:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:15 smithi194 ceph-mon[33017]: pgmap v12367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:15 smithi033 ceph-mon[27845]: pgmap v12367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:15 smithi096 ceph-mon[32753]: pgmap v12367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:04:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:18 smithi194 ceph-mon[33017]: pgmap v12368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:04:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22397.037531 secs 2024-04-26T18:04:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:18 smithi033 ceph-mon[27845]: pgmap v12368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:04:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22397.037531 secs 2024-04-26T18:04:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:18 smithi096 ceph-mon[32753]: pgmap v12368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:04:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22397.037531 secs 2024-04-26T18:04:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:19.247+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:19.247+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:04:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:19.247+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22399.903389 secs 2024-04-26T18:04:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:20 smithi194 ceph-mon[33017]: pgmap v12369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:20.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22399.903389 secs 2024-04-26T18:04:20.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:20 smithi033 ceph-mon[27845]: pgmap v12369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22399.903389 secs 2024-04-26T18:04:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:20 smithi096 ceph-mon[32753]: pgmap v12369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:22 smithi033 ceph-mon[27845]: pgmap v12370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:22 smithi096 ceph-mon[32753]: pgmap v12370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:22 smithi194 ceph-mon[33017]: pgmap v12370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22402.037627 secs 2024-04-26T18:04:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22402.037627 secs 2024-04-26T18:04:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22402.037627 secs 2024-04-26T18:04:24.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:24 smithi194 ceph-mon[33017]: pgmap v12371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:24 smithi033 ceph-mon[27845]: pgmap v12371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:24 smithi096 ceph-mon[32753]: pgmap v12371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:25.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22404.903502 secs 2024-04-26T18:04:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22404.903502 secs 2024-04-26T18:04:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22404.903502 secs 2024-04-26T18:04:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:26 smithi033 ceph-mon[27845]: pgmap v12372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:26 smithi194 ceph-mon[33017]: pgmap v12372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:26 smithi096 ceph-mon[32753]: pgmap v12372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:28 smithi194 ceph-mon[33017]: pgmap v12373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22407.037709 secs 2024-04-26T18:04:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:28 smithi033 ceph-mon[27845]: pgmap v12373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22407.037709 secs 2024-04-26T18:04:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:28 smithi096 ceph-mon[32753]: pgmap v12373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22407.037709 secs 2024-04-26T18:04:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22409.903637 secs 2024-04-26T18:04:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22409.903637 secs 2024-04-26T18:04:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22409.903637 secs 2024-04-26T18:04:30.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:30 smithi194 ceph-mon[33017]: pgmap v12374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:30 smithi033 ceph-mon[27845]: pgmap v12374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:30 smithi096 ceph-mon[32753]: pgmap v12374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:32.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:32 smithi033 ceph-mon[27845]: pgmap v12375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:32 smithi194 ceph-mon[33017]: pgmap v12375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:32 smithi096 ceph-mon[32753]: pgmap v12375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22412.037792 secs 2024-04-26T18:04:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22412.037792 secs 2024-04-26T18:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22412.037792 secs 2024-04-26T18:04:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:34.255+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:34.255+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:34.255+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:04:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:34 smithi194 ceph-mon[33017]: pgmap v12376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22414.903766 secs 2024-04-26T18:04:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:34 smithi033 ceph-mon[27845]: pgmap v12376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:35.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22414.903766 secs 2024-04-26T18:04:35.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:34 smithi096 ceph-mon[32753]: pgmap v12376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:35.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22414.903766 secs 2024-04-26T18:04:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:36 smithi194 ceph-mon[33017]: pgmap v12377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:36 smithi033 ceph-mon[27845]: pgmap v12377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:36 smithi096 ceph-mon[32753]: pgmap v12377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:38 smithi194 ceph-mon[33017]: pgmap v12378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22417.037879 secs 2024-04-26T18:04:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:38 smithi033 ceph-mon[27845]: pgmap v12378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22417.037879 secs 2024-04-26T18:04:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:38 smithi096 ceph-mon[32753]: pgmap v12378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:38.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22417.037879 secs 2024-04-26T18:04:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22419.903909 secs 2024-04-26T18:04:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22419.903909 secs 2024-04-26T18:04:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22419.903909 secs 2024-04-26T18:04:40.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:40 smithi194 ceph-mon[33017]: pgmap v12379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:40 smithi033 ceph-mon[27845]: pgmap v12379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:40 smithi096 ceph-mon[32753]: pgmap v12379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:42 smithi194 ceph-mon[33017]: pgmap v12380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:42 smithi033 ceph-mon[27845]: pgmap v12380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:42 smithi096 ceph-mon[32753]: pgmap v12380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22422.037960 secs 2024-04-26T18:04:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22422.037960 secs 2024-04-26T18:04:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22422.037960 secs 2024-04-26T18:04:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:44 smithi194 ceph-mon[33017]: pgmap v12381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22424.904041 secs 2024-04-26T18:04:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:44 smithi033 ceph-mon[27845]: pgmap v12381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22424.904041 secs 2024-04-26T18:04:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:44 smithi096 ceph-mon[32753]: pgmap v12381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22424.904041 secs 2024-04-26T18:04:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:46 smithi194 ceph-mon[33017]: pgmap v12382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:46 smithi033 ceph-mon[27845]: pgmap v12382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:46 smithi096 ceph-mon[32753]: pgmap v12382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:48 smithi194 ceph-mon[33017]: pgmap v12383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22427.038001 secs 2024-04-26T18:04:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:48 smithi033 ceph-mon[27845]: pgmap v12383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22427.038001 secs 2024-04-26T18:04:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:48 smithi096 ceph-mon[32753]: pgmap v12383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22427.038001 secs 2024-04-26T18:04:49.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:49.270+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:49.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:49.270+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:04:49.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:04:49.270+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:04:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22429.904188 secs 2024-04-26T18:04:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22429.904188 secs 2024-04-26T18:04:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22429.904188 secs 2024-04-26T18:04:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:50 smithi194 ceph-mon[33017]: pgmap v12384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:50 smithi033 ceph-mon[27845]: pgmap v12384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:50 smithi096 ceph-mon[32753]: pgmap v12384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:52 smithi194 ceph-mon[33017]: pgmap v12385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:52 smithi033 ceph-mon[27845]: pgmap v12385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:52 smithi096 ceph-mon[32753]: pgmap v12385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:53.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22432.038081 secs 2024-04-26T18:04:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22432.038081 secs 2024-04-26T18:04:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22432.038081 secs 2024-04-26T18:04:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:54 smithi194 ceph-mon[33017]: pgmap v12386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22434.904308 secs 2024-04-26T18:04:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:54 smithi033 ceph-mon[27845]: pgmap v12386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22434.904308 secs 2024-04-26T18:04:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:54 smithi096 ceph-mon[32753]: pgmap v12386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22434.904308 secs 2024-04-26T18:04:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:56 smithi194 ceph-mon[33017]: pgmap v12387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:56 smithi033 ceph-mon[27845]: pgmap v12387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:56 smithi096 ceph-mon[32753]: pgmap v12387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:04:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:58 smithi194 ceph-mon[33017]: pgmap v12388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22437.038144 secs 2024-04-26T18:04:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:58 smithi033 ceph-mon[27845]: pgmap v12388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22437.038144 secs 2024-04-26T18:04:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:58 smithi096 ceph-mon[32753]: pgmap v12388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:04:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22437.038144 secs 2024-04-26T18:04:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:04:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22439.904385 secs 2024-04-26T18:04:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:04:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22439.904385 secs 2024-04-26T18:04:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:04:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22439.904385 secs 2024-04-26T18:05:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:00 smithi194 ceph-mon[33017]: pgmap v12389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:00 smithi033 ceph-mon[27845]: pgmap v12389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:00 smithi096 ceph-mon[32753]: pgmap v12389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:02 smithi194 ceph-mon[33017]: pgmap v12390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:02 smithi033 ceph-mon[27845]: pgmap v12390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:02 smithi096 ceph-mon[32753]: pgmap v12390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22442.038228 secs 2024-04-26T18:05:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22442.038228 secs 2024-04-26T18:05:04.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22442.038228 secs 2024-04-26T18:05:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:04.282+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:04.282+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:04.282+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:04 smithi033 ceph-mon[27845]: pgmap v12391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22444.904509 secs 2024-04-26T18:05:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:04 smithi096 ceph-mon[32753]: pgmap v12391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22444.904509 secs 2024-04-26T18:05:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:04 smithi194 ceph-mon[33017]: pgmap v12391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22444.904509 secs 2024-04-26T18:05:06.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:06 smithi033 ceph-mon[27845]: pgmap v12392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:06 smithi096 ceph-mon[32753]: pgmap v12392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:06 smithi194 ceph-mon[33017]: pgmap v12392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:07.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:05:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:05:08.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:05:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:08 smithi033 ceph-mon[27845]: pgmap v12393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22447.038291 secs 2024-04-26T18:05:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:08 smithi096 ceph-mon[32753]: pgmap v12393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22447.038291 secs 2024-04-26T18:05:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:08 smithi194 ceph-mon[33017]: pgmap v12393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22447.038291 secs 2024-04-26T18:05:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22449.904615 secs 2024-04-26T18:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22449.904615 secs 2024-04-26T18:05:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:05:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:05:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22449.904615 secs 2024-04-26T18:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:05:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:10 smithi033 ceph-mon[27845]: pgmap v12394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:10 smithi033 ceph-mon[27845]: pgmap v12395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:10 smithi096 ceph-mon[32753]: pgmap v12394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:10 smithi096 ceph-mon[32753]: pgmap v12395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:10 smithi194 ceph-mon[33017]: pgmap v12394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:10 smithi194 ceph-mon[33017]: pgmap v12395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:12 smithi033 ceph-mon[27845]: pgmap v12396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:12 smithi096 ceph-mon[32753]: pgmap v12396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:12 smithi194 ceph-mon[33017]: pgmap v12396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22452.038373 secs 2024-04-26T18:05:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22452.038373 secs 2024-04-26T18:05:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22452.038373 secs 2024-04-26T18:05:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:14 smithi033 ceph-mon[27845]: pgmap v12397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22454.904747 secs 2024-04-26T18:05:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:14 smithi096 ceph-mon[32753]: pgmap v12397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22454.904747 secs 2024-04-26T18:05:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:14 smithi194 ceph-mon[33017]: pgmap v12397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22454.904747 secs 2024-04-26T18:05:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:16 smithi033 ceph-mon[27845]: pgmap v12398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:16 smithi096 ceph-mon[32753]: pgmap v12398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:17.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:16 smithi194 ceph-mon[33017]: pgmap v12398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:18 smithi033 ceph-mon[27845]: pgmap v12399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22457.038455 secs 2024-04-26T18:05:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:18 smithi096 ceph-mon[32753]: pgmap v12399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22457.038455 secs 2024-04-26T18:05:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:18 smithi194 ceph-mon[33017]: pgmap v12399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:05:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22457.038455 secs 2024-04-26T18:05:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:19.294+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:19.294+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:19.294+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22459.904876 secs 2024-04-26T18:05:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22459.904876 secs 2024-04-26T18:05:20.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22459.904876 secs 2024-04-26T18:05:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:20 smithi096 ceph-mon[32753]: pgmap v12400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:05:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:20 smithi194 ceph-mon[33017]: pgmap v12400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:05:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:20 smithi033 ceph-mon[27845]: pgmap v12400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:05:23.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:22 smithi194 ceph-mon[33017]: pgmap v12401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:22 smithi033 ceph-mon[27845]: pgmap v12401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:22 smithi096 ceph-mon[32753]: pgmap v12401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:24.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22462.038537 secs 2024-04-26T18:05:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22462.038537 secs 2024-04-26T18:05:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22462.038537 secs 2024-04-26T18:05:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:24 smithi194 ceph-mon[33017]: pgmap v12402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22464.904977 secs 2024-04-26T18:05:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:24 smithi033 ceph-mon[27845]: pgmap v12402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22464.904977 secs 2024-04-26T18:05:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:24 smithi096 ceph-mon[32753]: pgmap v12402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22464.904977 secs 2024-04-26T18:05:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:25.618+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:05:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:25.618+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:05:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:25.618+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 179 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T18:05:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:25.618+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 179 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:05:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:26 smithi194 ceph-mon[33017]: pgmap v12403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:26 smithi033 ceph-mon[27845]: pgmap v12403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:26 smithi096 ceph-mon[32753]: pgmap v12403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:28 smithi194 ceph-mon[33017]: pgmap v12404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22467.038622 secs 2024-04-26T18:05:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:28 smithi033 ceph-mon[27845]: pgmap v12404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22467.038622 secs 2024-04-26T18:05:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:28 smithi096 ceph-mon[32753]: pgmap v12404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22467.038622 secs 2024-04-26T18:05:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22469.905094 secs 2024-04-26T18:05:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22469.905094 secs 2024-04-26T18:05:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22469.905094 secs 2024-04-26T18:05:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:30 smithi194 ceph-mon[33017]: pgmap v12405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:30 smithi033 ceph-mon[27845]: pgmap v12405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:30 smithi096 ceph-mon[32753]: pgmap v12405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:32.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:32 smithi033 ceph-mon[27845]: pgmap v12406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:32 smithi194 ceph-mon[33017]: pgmap v12406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:32 smithi096 ceph-mon[32753]: pgmap v12406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22472.038703 secs 2024-04-26T18:05:34.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22472.038703 secs 2024-04-26T18:05:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22472.038703 secs 2024-04-26T18:05:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:34.297+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:34.297+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:34.297+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:34 smithi194 ceph-mon[33017]: pgmap v12407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22474.905215 secs 2024-04-26T18:05:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:34 smithi033 ceph-mon[27845]: pgmap v12407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22474.905215 secs 2024-04-26T18:05:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:34 smithi096 ceph-mon[32753]: pgmap v12407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22474.905215 secs 2024-04-26T18:05:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:36 smithi194 ceph-mon[33017]: pgmap v12408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:36 smithi033 ceph-mon[27845]: pgmap v12408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:36 smithi096 ceph-mon[32753]: pgmap v12408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:38 smithi194 ceph-mon[33017]: pgmap v12409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22477.038792 secs 2024-04-26T18:05:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:38 smithi033 ceph-mon[27845]: pgmap v12409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22477.038792 secs 2024-04-26T18:05:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:38 smithi096 ceph-mon[32753]: pgmap v12409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22477.038792 secs 2024-04-26T18:05:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22479.905331 secs 2024-04-26T18:05:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22479.905331 secs 2024-04-26T18:05:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22479.905331 secs 2024-04-26T18:05:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:40 smithi194 ceph-mon[33017]: pgmap v12410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:41.502 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:40 smithi096 ceph-mon[32753]: pgmap v12410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:41.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:40 smithi033 ceph-mon[27845]: pgmap v12410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:42 smithi194 ceph-mon[33017]: pgmap v12411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:42 smithi033 ceph-mon[27845]: pgmap v12411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:42 smithi096 ceph-mon[32753]: pgmap v12411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22482.038875 secs 2024-04-26T18:05:44.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22482.038875 secs 2024-04-26T18:05:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22482.038875 secs 2024-04-26T18:05:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:44 smithi194 ceph-mon[33017]: pgmap v12412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22484.905447 secs 2024-04-26T18:05:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:44 smithi033 ceph-mon[27845]: pgmap v12412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22484.905447 secs 2024-04-26T18:05:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:44 smithi096 ceph-mon[32753]: pgmap v12412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22484.905447 secs 2024-04-26T18:05:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:46 smithi194 ceph-mon[33017]: pgmap v12413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:46 smithi033 ceph-mon[27845]: pgmap v12413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:46 smithi096 ceph-mon[32753]: pgmap v12413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:48 smithi194 ceph-mon[33017]: pgmap v12414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22487.038958 secs 2024-04-26T18:05:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:48 smithi033 ceph-mon[27845]: pgmap v12414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22487.038958 secs 2024-04-26T18:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:48 smithi096 ceph-mon[32753]: pgmap v12414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22487.038958 secs 2024-04-26T18:05:49.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:49.311+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:49.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:49.311+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:05:49.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:05:49.311+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22489.905570 secs 2024-04-26T18:05:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22489.905570 secs 2024-04-26T18:05:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22489.905570 secs 2024-04-26T18:05:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:50 smithi194 ceph-mon[33017]: pgmap v12415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:50 smithi033 ceph-mon[27845]: pgmap v12415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:50 smithi096 ceph-mon[32753]: pgmap v12415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:53.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:52 smithi194 ceph-mon[33017]: pgmap v12416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:52 smithi033 ceph-mon[27845]: pgmap v12416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:52 smithi096 ceph-mon[32753]: pgmap v12416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22492.039016 secs 2024-04-26T18:05:54.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22492.039016 secs 2024-04-26T18:05:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22492.039016 secs 2024-04-26T18:05:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:54 smithi194 ceph-mon[33017]: pgmap v12417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22494.905685 secs 2024-04-26T18:05:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:54 smithi033 ceph-mon[27845]: pgmap v12417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22494.905685 secs 2024-04-26T18:05:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:54 smithi096 ceph-mon[32753]: pgmap v12417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22494.905685 secs 2024-04-26T18:05:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:55 smithi033 ceph-mon[27845]: pgmap v12418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:55 smithi096 ceph-mon[32753]: pgmap v12418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:55 smithi194 ceph-mon[33017]: pgmap v12418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:05:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:57 smithi033 ceph-mon[27845]: pgmap v12419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:57 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22497.039078 secs 2024-04-26T18:05:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:57 smithi096 ceph-mon[32753]: pgmap v12419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:57 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22497.039078 secs 2024-04-26T18:05:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:57 smithi194 ceph-mon[33017]: pgmap v12419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:05:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:57 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22497.039078 secs 2024-04-26T18:06:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22499.905788 secs 2024-04-26T18:06:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:05:59 smithi033 ceph-mon[27845]: pgmap v12420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22499.905788 secs 2024-04-26T18:06:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:05:59 smithi096 ceph-mon[32753]: pgmap v12420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:00.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22499.905788 secs 2024-04-26T18:06:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:05:59 smithi194 ceph-mon[33017]: pgmap v12420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:01 smithi033 ceph-mon[27845]: pgmap v12421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:01 smithi096 ceph-mon[32753]: pgmap v12421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:01 smithi194 ceph-mon[33017]: pgmap v12421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:02 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22502.039163 secs 2024-04-26T18:06:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:02 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22502.039163 secs 2024-04-26T18:06:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:02 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22502.039163 secs 2024-04-26T18:06:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:03 smithi033 ceph-mon[27845]: pgmap v12422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:03 smithi096 ceph-mon[32753]: pgmap v12422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:04.324+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:04.325+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:04.325+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:03 smithi194 ceph-mon[33017]: pgmap v12422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22504.905898 secs 2024-04-26T18:06:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22504.905898 secs 2024-04-26T18:06:05.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22504.905898 secs 2024-04-26T18:06:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:05 smithi096 ceph-mon[32753]: pgmap v12423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:05 smithi194 ceph-mon[33017]: pgmap v12423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:05 smithi033 ceph-mon[27845]: pgmap v12423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:07 smithi194 ceph-mon[33017]: pgmap v12424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22507.039241 secs 2024-04-26T18:06:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:07 smithi033 ceph-mon[27845]: pgmap v12424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22507.039241 secs 2024-04-26T18:06:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:07 smithi096 ceph-mon[32753]: pgmap v12424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22507.039241 secs 2024-04-26T18:06:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22509.906014 secs 2024-04-26T18:06:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:09 smithi194 ceph-mon[33017]: pgmap v12425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:06:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22509.906014 secs 2024-04-26T18:06:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:09 smithi033 ceph-mon[27845]: pgmap v12425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:06:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22509.906014 secs 2024-04-26T18:06:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:09 smithi096 ceph-mon[32753]: pgmap v12425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:06:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: pgmap v12426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: pgmap v12426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: pgmap v12426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:13.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:13 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:06:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:13 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:06:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:13 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:06:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:13 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:06:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22512.039302 secs 2024-04-26T18:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:13 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:13 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:13 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:13 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:06:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22512.039302 secs 2024-04-26T18:06:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:13 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:06:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:13 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:06:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:13 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:06:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:13 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:06:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22512.039302 secs 2024-04-26T18:06:15.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:14 smithi033 ceph-mon[27845]: pgmap v12427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:15.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22514.906110 secs 2024-04-26T18:06:15.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:15.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:14 smithi194 ceph-mon[33017]: pgmap v12427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22514.906110 secs 2024-04-26T18:06:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:14 smithi096 ceph-mon[32753]: pgmap v12427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22514.906110 secs 2024-04-26T18:06:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:06:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:06:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:06:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:06:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:06:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:16 smithi194 ceph-mon[33017]: pgmap v12428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:16 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:06:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:16 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:06:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:16 smithi194 ceph-mon[33017]: pgmap v12429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:06:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:16 smithi194 ceph-mon[33017]: pgmap v12430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:16 smithi033 ceph-mon[27845]: pgmap v12428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:16 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:06:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:16 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:06:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:16 smithi033 ceph-mon[27845]: pgmap v12429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:06:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:16 smithi033 ceph-mon[27845]: pgmap v12430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:16 smithi096 ceph-mon[32753]: pgmap v12428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:16 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:06:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:16 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:06:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:16 smithi096 ceph-mon[32753]: pgmap v12429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:06:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:16 smithi096 ceph-mon[32753]: pgmap v12430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:17 smithi194 ceph-mon[33017]: pgmap v12431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:17 smithi033 ceph-mon[27845]: pgmap v12431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:17 smithi096 ceph-mon[32753]: pgmap v12431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22517.039386 secs 2024-04-26T18:06:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22517.039386 secs 2024-04-26T18:06:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22517.039386 secs 2024-04-26T18:06:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:19.340+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:19.340+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:19.340+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22519.906223 secs 2024-04-26T18:06:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:19 smithi033 ceph-mon[27845]: pgmap v12432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:06:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22519.906223 secs 2024-04-26T18:06:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:19 smithi096 ceph-mon[32753]: pgmap v12432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:06:20.278 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22519.906223 secs 2024-04-26T18:06:20.278 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:19 smithi194 ceph-mon[33017]: pgmap v12432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:06:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:22 smithi194 ceph-mon[33017]: pgmap v12433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:22 smithi033 ceph-mon[27845]: pgmap v12433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:22 smithi096 ceph-mon[32753]: pgmap v12433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22522.039447 secs 2024-04-26T18:06:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22522.039447 secs 2024-04-26T18:06:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22522.039447 secs 2024-04-26T18:06:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:24 smithi096 ceph-mon[32753]: pgmap v12434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:06:24.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:24 smithi194 ceph-mon[33017]: pgmap v12434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:06:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:24 smithi033 ceph-mon[27845]: pgmap v12434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:06:25.599 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22524.906341 secs 2024-04-26T18:06:25.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22524.906341 secs 2024-04-26T18:06:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22524.906341 secs 2024-04-26T18:06:26.794 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:26 smithi096 ceph-mon[32753]: pgmap v12435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:06:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:26 smithi194 ceph-mon[33017]: pgmap v12435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:06:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:26 smithi033 ceph-mon[27845]: pgmap v12435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:06:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:28 smithi194 ceph-mon[33017]: pgmap v12436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22527.039537 secs 2024-04-26T18:06:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:28 smithi033 ceph-mon[27845]: pgmap v12436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22527.039537 secs 2024-04-26T18:06:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:28 smithi096 ceph-mon[32753]: pgmap v12436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22527.039537 secs 2024-04-26T18:06:29.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22529.906450 secs 2024-04-26T18:06:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22529.906450 secs 2024-04-26T18:06:31.131 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:30 smithi194 ceph-mon[33017]: pgmap v12437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:31.132 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:30 smithi033 ceph-mon[27845]: pgmap v12437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:31.132 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22529.906450 secs 2024-04-26T18:06:31.132 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:30 smithi096 ceph-mon[32753]: pgmap v12437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:32 smithi194 ceph-mon[33017]: pgmap v12438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:32 smithi033 ceph-mon[27845]: pgmap v12438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:32 smithi096 ceph-mon[32753]: pgmap v12438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:33.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22532.039621 secs 2024-04-26T18:06:33.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22532.039621 secs 2024-04-26T18:06:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22532.039621 secs 2024-04-26T18:06:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:34.342+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:34.342+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:34.342+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:34 smithi194 ceph-mon[33017]: pgmap v12439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22534.906558 secs 2024-04-26T18:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:34 smithi033 ceph-mon[27845]: pgmap v12439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22534.906558 secs 2024-04-26T18:06:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:34 smithi096 ceph-mon[32753]: pgmap v12439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22534.906558 secs 2024-04-26T18:06:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:36 smithi096 ceph-mon[32753]: pgmap v12440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:36.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:36 smithi194 ceph-mon[33017]: pgmap v12440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:36.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:36 smithi033 ceph-mon[27845]: pgmap v12440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:38 smithi194 ceph-mon[33017]: pgmap v12441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22537.039704 secs 2024-04-26T18:06:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:38 smithi033 ceph-mon[27845]: pgmap v12441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22537.039704 secs 2024-04-26T18:06:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:38 smithi096 ceph-mon[32753]: pgmap v12441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22537.039704 secs 2024-04-26T18:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22539.906693 secs 2024-04-26T18:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22539.906693 secs 2024-04-26T18:06:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22539.906693 secs 2024-04-26T18:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:40 smithi194 ceph-mon[33017]: pgmap v12442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:40 smithi033 ceph-mon[27845]: pgmap v12442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:40 smithi096 ceph-mon[32753]: pgmap v12442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:42 smithi194 ceph-mon[33017]: pgmap v12443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:42 smithi033 ceph-mon[27845]: pgmap v12443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:42 smithi096 ceph-mon[32753]: pgmap v12443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:06:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22542.039786 secs 2024-04-26T18:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22542.039786 secs 2024-04-26T18:06:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22542.039786 secs 2024-04-26T18:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:44 smithi194 ceph-mon[33017]: pgmap v12444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22544.906815 secs 2024-04-26T18:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:44 smithi033 ceph-mon[27845]: pgmap v12444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22544.906815 secs 2024-04-26T18:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:44 smithi096 ceph-mon[32753]: pgmap v12444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22544.906815 secs 2024-04-26T18:06:46.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:46 smithi194 ceph-mon[33017]: pgmap v12445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:46 smithi033 ceph-mon[27845]: pgmap v12445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:46 smithi096 ceph-mon[32753]: pgmap v12445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:48.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:48 smithi194 ceph-mon[33017]: pgmap v12446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22547.039872 secs 2024-04-26T18:06:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:48 smithi033 ceph-mon[27845]: pgmap v12446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22547.039872 secs 2024-04-26T18:06:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:48 smithi096 ceph-mon[32753]: pgmap v12446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22547.039872 secs 2024-04-26T18:06:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:49.355+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:49.355+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:06:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:06:49.355+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22549.906930 secs 2024-04-26T18:06:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22549.906930 secs 2024-04-26T18:06:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22549.906930 secs 2024-04-26T18:06:50.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:50 smithi194 ceph-mon[33017]: pgmap v12447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:50 smithi033 ceph-mon[27845]: pgmap v12447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:50 smithi096 ceph-mon[32753]: pgmap v12447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:52.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:52 smithi194 ceph-mon[33017]: pgmap v12448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:52 smithi033 ceph-mon[27845]: pgmap v12448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:52 smithi096 ceph-mon[32753]: pgmap v12448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22552.039954 secs 2024-04-26T18:06:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22552.039954 secs 2024-04-26T18:06:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22552.039954 secs 2024-04-26T18:06:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:54 smithi194 ceph-mon[33017]: pgmap v12449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22554.907065 secs 2024-04-26T18:06:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:54 smithi033 ceph-mon[27845]: pgmap v12449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22554.907065 secs 2024-04-26T18:06:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:54 smithi096 ceph-mon[32753]: pgmap v12449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:06:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22554.907065 secs 2024-04-26T18:06:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:56 smithi194 ceph-mon[33017]: pgmap v12450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:56 smithi033 ceph-mon[27845]: pgmap v12450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:56 smithi096 ceph-mon[32753]: pgmap v12450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:06:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:58 smithi194 ceph-mon[33017]: pgmap v12451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22557.040001 secs 2024-04-26T18:06:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:58 smithi033 ceph-mon[27845]: pgmap v12451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22557.040001 secs 2024-04-26T18:06:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:58 smithi096 ceph-mon[32753]: pgmap v12451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:06:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22557.040001 secs 2024-04-26T18:06:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22559.907182 secs 2024-04-26T18:06:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22559.907182 secs 2024-04-26T18:07:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22559.907182 secs 2024-04-26T18:07:00.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:00 smithi033 ceph-mon[27845]: pgmap v12452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:00 smithi096 ceph-mon[32753]: pgmap v12452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:00 smithi194 ceph-mon[33017]: pgmap v12452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:02 smithi033 ceph-mon[27845]: pgmap v12453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:02 smithi096 ceph-mon[32753]: pgmap v12453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:03.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:02 smithi194 ceph-mon[33017]: pgmap v12453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22562.040087 secs 2024-04-26T18:07:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22562.040087 secs 2024-04-26T18:07:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22562.040087 secs 2024-04-26T18:07:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:04.367+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:04.367+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:04.367+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:04 smithi033 ceph-mon[27845]: pgmap v12454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22564.907258 secs 2024-04-26T18:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:04 smithi096 ceph-mon[32753]: pgmap v12454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22564.907258 secs 2024-04-26T18:07:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:04 smithi194 ceph-mon[33017]: pgmap v12454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22564.907258 secs 2024-04-26T18:07:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:06 smithi033 ceph-mon[27845]: pgmap v12455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:06 smithi096 ceph-mon[32753]: pgmap v12455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:07.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:06 smithi194 ceph-mon[33017]: pgmap v12455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:08 smithi033 ceph-mon[27845]: pgmap v12456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22567.040170 secs 2024-04-26T18:07:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:08 smithi096 ceph-mon[32753]: pgmap v12456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22567.040170 secs 2024-04-26T18:07:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:08 smithi194 ceph-mon[33017]: pgmap v12456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22567.040170 secs 2024-04-26T18:07:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22569.907337 secs 2024-04-26T18:07:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22569.907337 secs 2024-04-26T18:07:10.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22569.907337 secs 2024-04-26T18:07:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:10 smithi033 ceph-mon[27845]: pgmap v12457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:10 smithi096 ceph-mon[32753]: pgmap v12457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:10 smithi194 ceph-mon[33017]: pgmap v12457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:12 smithi033 ceph-mon[27845]: pgmap v12458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:12 smithi096 ceph-mon[32753]: pgmap v12458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:12 smithi194 ceph-mon[33017]: pgmap v12458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22572.040255 secs 2024-04-26T18:07:14.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22572.040255 secs 2024-04-26T18:07:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22572.040255 secs 2024-04-26T18:07:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:14 smithi096 ceph-mon[32753]: pgmap v12459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22574.907456 secs 2024-04-26T18:07:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:14 smithi194 ceph-mon[33017]: pgmap v12459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22574.907456 secs 2024-04-26T18:07:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:14 smithi033 ceph-mon[27845]: pgmap v12459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22574.907456 secs 2024-04-26T18:07:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:07:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:07:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:07:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:16 smithi194 ceph-mon[33017]: pgmap v12460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:17.183 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:16 smithi033 ceph-mon[27845]: pgmap v12460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:17.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:16 smithi096 ceph-mon[32753]: pgmap v12460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:18.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:07:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:07:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:07:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:07:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:07:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:07:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:07:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:07:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:18 smithi194 ceph-mon[33017]: pgmap v12461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:18 smithi194 ceph-mon[33017]: pgmap v12462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22577.040337 secs 2024-04-26T18:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:18 smithi033 ceph-mon[27845]: pgmap v12461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:18 smithi033 ceph-mon[27845]: pgmap v12462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22577.040337 secs 2024-04-26T18:07:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:18 smithi096 ceph-mon[32753]: pgmap v12461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:18 smithi096 ceph-mon[32753]: pgmap v12462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22577.040337 secs 2024-04-26T18:07:19.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:19.379+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:19.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:19.380+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:19.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:19.380+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:20.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22579.907531 secs 2024-04-26T18:07:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22579.907531 secs 2024-04-26T18:07:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22579.907531 secs 2024-04-26T18:07:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:20 smithi194 ceph-mon[33017]: pgmap v12463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:07:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:20 smithi033 ceph-mon[27845]: pgmap v12463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:07:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:20 smithi096 ceph-mon[32753]: pgmap v12463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:07:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:22 smithi194 ceph-mon[33017]: pgmap v12464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:22 smithi033 ceph-mon[27845]: pgmap v12464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:22 smithi096 ceph-mon[32753]: pgmap v12464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:24.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22582.040400 secs 2024-04-26T18:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22582.040400 secs 2024-04-26T18:07:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22582.040400 secs 2024-04-26T18:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:24 smithi194 ceph-mon[33017]: pgmap v12465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22584.907665 secs 2024-04-26T18:07:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:24 smithi033 ceph-mon[27845]: pgmap v12465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22584.907665 secs 2024-04-26T18:07:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:24 smithi096 ceph-mon[32753]: pgmap v12465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22584.907665 secs 2024-04-26T18:07:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:26 smithi194 ceph-mon[33017]: pgmap v12466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:26 smithi033 ceph-mon[27845]: pgmap v12466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:26 smithi096 ceph-mon[32753]: pgmap v12466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:07:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:28 smithi194 ceph-mon[33017]: pgmap v12467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22587.040459 secs 2024-04-26T18:07:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:28 smithi033 ceph-mon[27845]: pgmap v12467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:07:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22587.040459 secs 2024-04-26T18:07:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:28 smithi096 ceph-mon[32753]: pgmap v12467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:07:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22587.040459 secs 2024-04-26T18:07:30.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22589.907782 secs 2024-04-26T18:07:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22589.907782 secs 2024-04-26T18:07:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22589.907782 secs 2024-04-26T18:07:31.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:30 smithi194 ceph-mon[33017]: pgmap v12468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:30 smithi033 ceph-mon[27845]: pgmap v12468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:30 smithi096 ceph-mon[32753]: pgmap v12468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:33.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:32 smithi194 ceph-mon[33017]: pgmap v12469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:32 smithi033 ceph-mon[27845]: pgmap v12469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:32 smithi096 ceph-mon[32753]: pgmap v12469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22592.040544 secs 2024-04-26T18:07:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22592.040544 secs 2024-04-26T18:07:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22592.040544 secs 2024-04-26T18:07:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:34.391+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:34.391+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:34.391+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:35.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:34 smithi194 ceph-mon[33017]: pgmap v12470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22594.907902 secs 2024-04-26T18:07:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:34 smithi033 ceph-mon[27845]: pgmap v12470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22594.907902 secs 2024-04-26T18:07:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:34 smithi096 ceph-mon[32753]: pgmap v12470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22594.907902 secs 2024-04-26T18:07:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:36 smithi194 ceph-mon[33017]: pgmap v12471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:36 smithi033 ceph-mon[27845]: pgmap v12471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:36 smithi096 ceph-mon[32753]: pgmap v12471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:38 smithi194 ceph-mon[33017]: pgmap v12472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22597.040627 secs 2024-04-26T18:07:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:38 smithi033 ceph-mon[27845]: pgmap v12472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22597.040627 secs 2024-04-26T18:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:38 smithi096 ceph-mon[32753]: pgmap v12472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22597.040627 secs 2024-04-26T18:07:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22599.908039 secs 2024-04-26T18:07:40.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22599.908039 secs 2024-04-26T18:07:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22599.908039 secs 2024-04-26T18:07:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:40 smithi194 ceph-mon[33017]: pgmap v12473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:41.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:40 smithi033 ceph-mon[27845]: pgmap v12473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:40 smithi096 ceph-mon[32753]: pgmap v12473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:42 smithi194 ceph-mon[33017]: pgmap v12474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:42 smithi033 ceph-mon[27845]: pgmap v12474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:42 smithi096 ceph-mon[32753]: pgmap v12474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22602.040712 secs 2024-04-26T18:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22602.040712 secs 2024-04-26T18:07:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22602.040712 secs 2024-04-26T18:07:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:44 smithi194 ceph-mon[33017]: pgmap v12475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22604.908135 secs 2024-04-26T18:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:44 smithi033 ceph-mon[27845]: pgmap v12475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22604.908135 secs 2024-04-26T18:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:44 smithi096 ceph-mon[32753]: pgmap v12475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22604.908135 secs 2024-04-26T18:07:47.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:46 smithi194 ceph-mon[33017]: pgmap v12476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:46 smithi033 ceph-mon[27845]: pgmap v12476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:46 smithi096 ceph-mon[32753]: pgmap v12476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:48 smithi194 ceph-mon[33017]: pgmap v12477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22607.040794 secs 2024-04-26T18:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:48 smithi033 ceph-mon[27845]: pgmap v12477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22607.040794 secs 2024-04-26T18:07:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:48 smithi096 ceph-mon[32753]: pgmap v12477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22607.040794 secs 2024-04-26T18:07:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:49.397+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:49.397+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:07:49.397+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:07:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22609.908268 secs 2024-04-26T18:07:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22609.908268 secs 2024-04-26T18:07:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22609.908268 secs 2024-04-26T18:07:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:50 smithi194 ceph-mon[33017]: pgmap v12478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:50 smithi033 ceph-mon[27845]: pgmap v12478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:50 smithi096 ceph-mon[32753]: pgmap v12478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:52 smithi194 ceph-mon[33017]: pgmap v12479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:52 smithi033 ceph-mon[27845]: pgmap v12479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:52 smithi096 ceph-mon[32753]: pgmap v12479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22612.040877 secs 2024-04-26T18:07:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22612.040877 secs 2024-04-26T18:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22612.040877 secs 2024-04-26T18:07:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:54 smithi033 ceph-mon[27845]: pgmap v12480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22614.908379 secs 2024-04-26T18:07:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:54 smithi096 ceph-mon[32753]: pgmap v12480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22614.908379 secs 2024-04-26T18:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:54 smithi194 ceph-mon[33017]: pgmap v12480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22614.908379 secs 2024-04-26T18:07:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:55 smithi033 ceph-mon[27845]: pgmap v12481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:55 smithi096 ceph-mon[32753]: pgmap v12481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:56.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:55 smithi194 ceph-mon[33017]: pgmap v12481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:07:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:58 smithi194 ceph-mon[33017]: pgmap v12482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:07:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22617.040960 secs 2024-04-26T18:07:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:58 smithi033 ceph-mon[27845]: pgmap v12482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:07:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22617.040960 secs 2024-04-26T18:07:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:58 smithi096 ceph-mon[32753]: pgmap v12482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:07:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:07:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22617.040960 secs 2024-04-26T18:08:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22619.908488 secs 2024-04-26T18:08:00.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:00 smithi194 ceph-mon[33017]: pgmap v12483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22619.908488 secs 2024-04-26T18:08:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:00 smithi033 ceph-mon[27845]: pgmap v12483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22619.908488 secs 2024-04-26T18:08:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:00 smithi096 ceph-mon[32753]: pgmap v12483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:02 smithi194 ceph-mon[33017]: pgmap v12484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:02 smithi033 ceph-mon[27845]: pgmap v12484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:02 smithi096 ceph-mon[32753]: pgmap v12484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22622.041048 secs 2024-04-26T18:08:03.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22622.041048 secs 2024-04-26T18:08:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22622.041048 secs 2024-04-26T18:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:04 smithi194 ceph-mon[33017]: pgmap v12485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:04.414+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:04.414+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:04.414+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:04 smithi033 ceph-mon[27845]: pgmap v12485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:04 smithi096 ceph-mon[32753]: pgmap v12485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:05.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22624.908550 secs 2024-04-26T18:08:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22624.908550 secs 2024-04-26T18:08:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22624.908550 secs 2024-04-26T18:08:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:06 smithi194 ceph-mon[33017]: pgmap v12486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:06 smithi033 ceph-mon[27845]: pgmap v12486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:06 smithi096 ceph-mon[32753]: pgmap v12486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:08.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:08 smithi194 ceph-mon[33017]: pgmap v12487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22627.041136 secs 2024-04-26T18:08:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:08 smithi033 ceph-mon[27845]: pgmap v12487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22627.041136 secs 2024-04-26T18:08:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:08 smithi096 ceph-mon[32753]: pgmap v12487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22627.041136 secs 2024-04-26T18:08:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22629.908668 secs 2024-04-26T18:08:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:10 smithi194 ceph-mon[33017]: pgmap v12488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22629.908668 secs 2024-04-26T18:08:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:10 smithi033 ceph-mon[27845]: pgmap v12488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22629.908668 secs 2024-04-26T18:08:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:10 smithi096 ceph-mon[32753]: pgmap v12488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:12.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:12 smithi194 ceph-mon[33017]: pgmap v12489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:12 smithi033 ceph-mon[27845]: pgmap v12489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:12 smithi096 ceph-mon[32753]: pgmap v12489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22632.041223 secs 2024-04-26T18:08:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22632.041223 secs 2024-04-26T18:08:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22632.041223 secs 2024-04-26T18:08:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:14 smithi194 ceph-mon[33017]: pgmap v12490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:14 smithi033 ceph-mon[27845]: pgmap v12490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:14 smithi096 ceph-mon[32753]: pgmap v12490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:15.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22634.908788 secs 2024-04-26T18:08:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22634.908788 secs 2024-04-26T18:08:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22634.908788 secs 2024-04-26T18:08:16.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:16 smithi194 ceph-mon[33017]: pgmap v12491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:16 smithi033 ceph-mon[27845]: pgmap v12491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:16 smithi096 ceph-mon[32753]: pgmap v12491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:18.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:18 smithi194 ceph-mon[33017]: pgmap v12492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:08:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22637.041292 secs 2024-04-26T18:08:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:18 smithi033 ceph-mon[27845]: pgmap v12492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:08:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22637.041292 secs 2024-04-26T18:08:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:18 smithi096 ceph-mon[32753]: pgmap v12492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:08:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22637.041292 secs 2024-04-26T18:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:19 smithi194 ceph-mon[33017]: pgmap v12493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:19 smithi194 ceph-mon[33017]: pgmap v12494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:19 smithi033 ceph-mon[27845]: pgmap v12493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:19 smithi033 ceph-mon[27845]: pgmap v12494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:08:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:19.429+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:19.429+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:19.429+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:19 smithi096 ceph-mon[32753]: pgmap v12493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:19 smithi096 ceph-mon[32753]: pgmap v12494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:08:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22639.908845 secs 2024-04-26T18:08:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22639.908845 secs 2024-04-26T18:08:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22639.908845 secs 2024-04-26T18:08:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:21 smithi033 ceph-mon[27845]: pgmap v12495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:21 smithi096 ceph-mon[32753]: pgmap v12495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:21.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:21 smithi194 ceph-mon[33017]: pgmap v12495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:23 smithi194 ceph-mon[33017]: pgmap v12496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22642.041377 secs 2024-04-26T18:08:24.135 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:23 smithi096 ceph-mon[32753]: pgmap v12496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:24.135 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22642.041377 secs 2024-04-26T18:08:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:23 smithi033 ceph-mon[27845]: pgmap v12496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22642.041377 secs 2024-04-26T18:08:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22644.908906 secs 2024-04-26T18:08:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22644.908906 secs 2024-04-26T18:08:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22644.908906 secs 2024-04-26T18:08:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:25.649+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:08:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:25.649+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:08:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:25.649+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 180 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T18:08:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:25.649+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 180 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:08:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:25 smithi194 ceph-mon[33017]: pgmap v12497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:25 smithi033 ceph-mon[27845]: pgmap v12497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:25 smithi096 ceph-mon[32753]: pgmap v12497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:27 smithi194 ceph-mon[33017]: pgmap v12498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:27 smithi033 ceph-mon[27845]: pgmap v12498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:27 smithi096 ceph-mon[32753]: pgmap v12498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:08:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22647.041461 secs 2024-04-26T18:08:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22649.909023 secs 2024-04-26T18:08:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22647.041461 secs 2024-04-26T18:08:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22649.909023 secs 2024-04-26T18:08:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22647.041461 secs 2024-04-26T18:08:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22649.909023 secs 2024-04-26T18:08:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:29 smithi194 ceph-mon[33017]: pgmap v12499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:08:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:29 smithi033 ceph-mon[27845]: pgmap v12499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:08:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:29 smithi096 ceph-mon[32753]: pgmap v12499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:08:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:31 smithi194 ceph-mon[33017]: pgmap v12500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:31 smithi033 ceph-mon[27845]: pgmap v12500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:31 smithi096 ceph-mon[32753]: pgmap v12500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:34.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:33 smithi033 ceph-mon[27845]: pgmap v12501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:34.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22652.041543 secs 2024-04-26T18:08:34.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22654.909153 secs 2024-04-26T18:08:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:33 smithi194 ceph-mon[33017]: pgmap v12501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:34.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22652.041543 secs 2024-04-26T18:08:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22654.909153 secs 2024-04-26T18:08:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:33 smithi096 ceph-mon[32753]: pgmap v12501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22652.041543 secs 2024-04-26T18:08:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22654.909153 secs 2024-04-26T18:08:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:34.434+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:34.434+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:34.434+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:36.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:35 smithi194 ceph-mon[33017]: pgmap v12502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:35 smithi033 ceph-mon[27845]: pgmap v12502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:35 smithi096 ceph-mon[32753]: pgmap v12502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:37 smithi194 ceph-mon[33017]: pgmap v12503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:37 smithi033 ceph-mon[27845]: pgmap v12503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:38.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:37 smithi096 ceph-mon[32753]: pgmap v12503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:39.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22657.041627 secs 2024-04-26T18:08:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22659.909222 secs 2024-04-26T18:08:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22657.041627 secs 2024-04-26T18:08:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22659.909222 secs 2024-04-26T18:08:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22657.041627 secs 2024-04-26T18:08:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22659.909222 secs 2024-04-26T18:08:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:39 smithi194 ceph-mon[33017]: pgmap v12504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:39 smithi033 ceph-mon[27845]: pgmap v12504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:39 smithi096 ceph-mon[32753]: pgmap v12504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:41 smithi194 ceph-mon[33017]: pgmap v12505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:41 smithi033 ceph-mon[27845]: pgmap v12505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:41 smithi096 ceph-mon[32753]: pgmap v12505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:43 smithi194 ceph-mon[33017]: pgmap v12506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22662.041687 secs 2024-04-26T18:08:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22664.909340 secs 2024-04-26T18:08:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:43 smithi033 ceph-mon[27845]: pgmap v12506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22662.041687 secs 2024-04-26T18:08:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22664.909340 secs 2024-04-26T18:08:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:43 smithi096 ceph-mon[32753]: pgmap v12506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22662.041687 secs 2024-04-26T18:08:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22664.909340 secs 2024-04-26T18:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:45 smithi194 ceph-mon[33017]: pgmap v12507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:45 smithi033 ceph-mon[27845]: pgmap v12507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:45 smithi096 ceph-mon[32753]: pgmap v12507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:47 smithi194 ceph-mon[33017]: pgmap v12508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:47 smithi033 ceph-mon[27845]: pgmap v12508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:47 smithi096 ceph-mon[32753]: pgmap v12508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22667.041774 secs 2024-04-26T18:08:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22669.909452 secs 2024-04-26T18:08:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22667.041774 secs 2024-04-26T18:08:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22669.909452 secs 2024-04-26T18:08:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22667.041774 secs 2024-04-26T18:08:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22669.909452 secs 2024-04-26T18:08:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:49.450+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:49.450+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:08:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:08:49.450+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:50.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:49 smithi033 ceph-mon[27845]: pgmap v12509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:49 smithi194 ceph-mon[33017]: pgmap v12509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:49 smithi096 ceph-mon[32753]: pgmap v12509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:51 smithi194 ceph-mon[33017]: pgmap v12510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:51 smithi033 ceph-mon[27845]: pgmap v12510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:51 smithi096 ceph-mon[32753]: pgmap v12510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:53 smithi194 ceph-mon[33017]: pgmap v12511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22672.041856 secs 2024-04-26T18:08:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22674.909566 secs 2024-04-26T18:08:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:53 smithi033 ceph-mon[27845]: pgmap v12511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22672.041856 secs 2024-04-26T18:08:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22674.909566 secs 2024-04-26T18:08:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:53 smithi096 ceph-mon[32753]: pgmap v12511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22672.041856 secs 2024-04-26T18:08:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22674.909566 secs 2024-04-26T18:08:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:55 smithi194 ceph-mon[33017]: pgmap v12512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:55 smithi033 ceph-mon[27845]: pgmap v12512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:55 smithi096 ceph-mon[32753]: pgmap v12512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:08:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:57 smithi194 ceph-mon[33017]: pgmap v12513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:57 smithi033 ceph-mon[27845]: pgmap v12513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:57 smithi096 ceph-mon[32753]: pgmap v12513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:08:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22677.041938 secs 2024-04-26T18:08:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22679.909687 secs 2024-04-26T18:08:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22677.041938 secs 2024-04-26T18:08:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22679.909687 secs 2024-04-26T18:08:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22677.041938 secs 2024-04-26T18:08:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22679.909687 secs 2024-04-26T18:09:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:08:59 smithi194 ceph-mon[33017]: pgmap v12514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:08:59 smithi033 ceph-mon[27845]: pgmap v12514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:08:59 smithi096 ceph-mon[32753]: pgmap v12514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:01 smithi194 ceph-mon[33017]: pgmap v12515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:01 smithi033 ceph-mon[27845]: pgmap v12515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:01 smithi096 ceph-mon[32753]: pgmap v12515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:03 smithi033 ceph-mon[27845]: pgmap v12516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22682.042021 secs 2024-04-26T18:09:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22684.909804 secs 2024-04-26T18:09:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:03 smithi096 ceph-mon[32753]: pgmap v12516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22682.042021 secs 2024-04-26T18:09:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22684.909804 secs 2024-04-26T18:09:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:03 smithi194 ceph-mon[33017]: pgmap v12516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22682.042021 secs 2024-04-26T18:09:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22684.909804 secs 2024-04-26T18:09:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:04.462+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:04.463+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:04.463+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:05 smithi033 ceph-mon[27845]: pgmap v12517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:05 smithi096 ceph-mon[32753]: pgmap v12517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:06.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:05 smithi194 ceph-mon[33017]: pgmap v12517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:07 smithi033 ceph-mon[27845]: pgmap v12518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:07 smithi096 ceph-mon[32753]: pgmap v12518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:07 smithi194 ceph-mon[33017]: pgmap v12518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22687.042106 secs 2024-04-26T18:09:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22689.909920 secs 2024-04-26T18:09:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:08 smithi033 ceph-mon[27845]: pgmap v12519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22687.042106 secs 2024-04-26T18:09:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22689.909920 secs 2024-04-26T18:09:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:08 smithi096 ceph-mon[32753]: pgmap v12519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22687.042106 secs 2024-04-26T18:09:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22689.909920 secs 2024-04-26T18:09:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:08 smithi194 ceph-mon[33017]: pgmap v12519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:10.656+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:09:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:10.656+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:09:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:10.657+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 181 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T18:09:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:10.657+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 181 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:09:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:11 smithi194 ceph-mon[33017]: pgmap v12520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:09:12.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:11 smithi033 ceph-mon[27845]: pgmap v12520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:09:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:11 smithi096 ceph-mon[32753]: pgmap v12520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:09:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:13 smithi194 ceph-mon[33017]: pgmap v12521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22692.042159 secs 2024-04-26T18:09:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22694.909994 secs 2024-04-26T18:09:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:13 smithi033 ceph-mon[27845]: pgmap v12521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22692.042159 secs 2024-04-26T18:09:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22694.909994 secs 2024-04-26T18:09:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:13 smithi096 ceph-mon[32753]: pgmap v12521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22692.042159 secs 2024-04-26T18:09:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22694.909994 secs 2024-04-26T18:09:16.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:15 smithi194 ceph-mon[33017]: pgmap v12522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:09:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:15 smithi033 ceph-mon[27845]: pgmap v12522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:09:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:15 smithi096 ceph-mon[32753]: pgmap v12522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:09:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:17 smithi194 ceph-mon[33017]: pgmap v12523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:17 smithi033 ceph-mon[27845]: pgmap v12523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:17 smithi096 ceph-mon[32753]: pgmap v12523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22697.042224 secs 2024-04-26T18:09:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22699.910141 secs 2024-04-26T18:09:19.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22697.042224 secs 2024-04-26T18:09:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22699.910141 secs 2024-04-26T18:09:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22697.042224 secs 2024-04-26T18:09:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22699.910141 secs 2024-04-26T18:09:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:19.473+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:19.474+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:19.474+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:19 smithi194 ceph-mon[33017]: pgmap v12524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:09:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:19 smithi033 ceph-mon[27845]: pgmap v12524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:09:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:19 smithi096 ceph-mon[32753]: pgmap v12524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:09:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:09:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:09:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:09:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:09:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:09:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:09:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:09:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:09:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:09:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:21 smithi194 ceph-mon[33017]: pgmap v12525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T18:09:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:21 smithi194 ceph-mon[33017]: pgmap v12526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:21 smithi033 ceph-mon[27845]: pgmap v12525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T18:09:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:21 smithi033 ceph-mon[27845]: pgmap v12526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:21 smithi096 ceph-mon[32753]: pgmap v12525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T18:09:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:21 smithi096 ceph-mon[32753]: pgmap v12526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:24.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:23 smithi194 ceph-mon[33017]: pgmap v12527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22702.042289 secs 2024-04-26T18:09:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22704.910292 secs 2024-04-26T18:09:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:23 smithi033 ceph-mon[27845]: pgmap v12527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22702.042289 secs 2024-04-26T18:09:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22704.910292 secs 2024-04-26T18:09:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:23 smithi096 ceph-mon[32753]: pgmap v12527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22702.042289 secs 2024-04-26T18:09:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22704.910292 secs 2024-04-26T18:09:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:25 smithi194 ceph-mon[33017]: pgmap v12528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:25 smithi033 ceph-mon[27845]: pgmap v12528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:25 smithi096 ceph-mon[32753]: pgmap v12528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:09:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:27 smithi194 ceph-mon[33017]: pgmap v12529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:27 smithi033 ceph-mon[27845]: pgmap v12529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:27 smithi096 ceph-mon[32753]: pgmap v12529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22707.042351 secs 2024-04-26T18:09:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22709.910366 secs 2024-04-26T18:09:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22707.042351 secs 2024-04-26T18:09:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22709.910366 secs 2024-04-26T18:09:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22707.042351 secs 2024-04-26T18:09:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22709.910366 secs 2024-04-26T18:09:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:29 smithi194 ceph-mon[33017]: pgmap v12530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:29 smithi033 ceph-mon[27845]: pgmap v12530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:29 smithi096 ceph-mon[32753]: pgmap v12530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:31 smithi194 ceph-mon[33017]: pgmap v12531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:09:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:31 smithi033 ceph-mon[27845]: pgmap v12531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:09:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:31 smithi096 ceph-mon[32753]: pgmap v12531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:09:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:33 smithi194 ceph-mon[33017]: pgmap v12532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22712.042434 secs 2024-04-26T18:09:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22714.910482 secs 2024-04-26T18:09:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:33 smithi033 ceph-mon[27845]: pgmap v12532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22712.042434 secs 2024-04-26T18:09:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22714.910482 secs 2024-04-26T18:09:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:33 smithi096 ceph-mon[32753]: pgmap v12532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22712.042434 secs 2024-04-26T18:09:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22714.910482 secs 2024-04-26T18:09:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:34.483+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:34.483+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:34.483+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:35 smithi194 ceph-mon[33017]: pgmap v12533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:35 smithi033 ceph-mon[27845]: pgmap v12533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:35 smithi096 ceph-mon[32753]: pgmap v12533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:38.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:37 smithi194 ceph-mon[33017]: pgmap v12534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:37 smithi033 ceph-mon[27845]: pgmap v12534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:37 smithi096 ceph-mon[32753]: pgmap v12534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:39.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22717.042518 secs 2024-04-26T18:09:39.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22719.910550 secs 2024-04-26T18:09:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22717.042518 secs 2024-04-26T18:09:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22719.910550 secs 2024-04-26T18:09:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22717.042518 secs 2024-04-26T18:09:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22719.910550 secs 2024-04-26T18:09:40.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:39 smithi194 ceph-mon[33017]: pgmap v12535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:39 smithi033 ceph-mon[27845]: pgmap v12535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:39 smithi096 ceph-mon[32753]: pgmap v12535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:42.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:41 smithi194 ceph-mon[33017]: pgmap v12536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:41 smithi033 ceph-mon[27845]: pgmap v12536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:41 smithi096 ceph-mon[32753]: pgmap v12536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:43 smithi033 ceph-mon[27845]: pgmap v12537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22722.042598 secs 2024-04-26T18:09:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22724.910703 secs 2024-04-26T18:09:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:43 smithi096 ceph-mon[32753]: pgmap v12537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22722.042598 secs 2024-04-26T18:09:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22724.910703 secs 2024-04-26T18:09:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:43 smithi194 ceph-mon[33017]: pgmap v12537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22722.042598 secs 2024-04-26T18:09:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22724.910703 secs 2024-04-26T18:09:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:45 smithi033 ceph-mon[27845]: pgmap v12538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:45 smithi096 ceph-mon[32753]: pgmap v12538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:45 smithi194 ceph-mon[33017]: pgmap v12538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:47 smithi033 ceph-mon[27845]: pgmap v12539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:47 smithi096 ceph-mon[32753]: pgmap v12539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:47 smithi194 ceph-mon[33017]: pgmap v12539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22727.042684 secs 2024-04-26T18:09:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22729.910820 secs 2024-04-26T18:09:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22727.042684 secs 2024-04-26T18:09:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22729.910820 secs 2024-04-26T18:09:49.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22727.042684 secs 2024-04-26T18:09:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22729.910820 secs 2024-04-26T18:09:49.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:49.497+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:49.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:49.497+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:49.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:09:49.497+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:09:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:49 smithi033 ceph-mon[27845]: pgmap v12540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:49 smithi096 ceph-mon[32753]: pgmap v12540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:50.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:49 smithi194 ceph-mon[33017]: pgmap v12540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:50 smithi033 ceph-mon[27845]: pgmap v12541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:50 smithi096 ceph-mon[32753]: pgmap v12541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:51.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:50 smithi194 ceph-mon[33017]: pgmap v12541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:09:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:53 smithi033 ceph-mon[27845]: pgmap v12542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22732.042770 secs 2024-04-26T18:09:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:53 smithi096 ceph-mon[32753]: pgmap v12542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22732.042770 secs 2024-04-26T18:09:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:53 smithi194 ceph-mon[33017]: pgmap v12542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22732.042770 secs 2024-04-26T18:09:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22734.910946 secs 2024-04-26T18:09:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22734.910946 secs 2024-04-26T18:09:54.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22734.910946 secs 2024-04-26T18:09:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:55 smithi033 ceph-mon[27845]: pgmap v12543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:55 smithi096 ceph-mon[32753]: pgmap v12543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:55 smithi194 ceph-mon[33017]: pgmap v12543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:57 smithi033 ceph-mon[27845]: pgmap v12544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:57 smithi096 ceph-mon[32753]: pgmap v12544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:57 smithi194 ceph-mon[33017]: pgmap v12544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:09:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22737.042853 secs 2024-04-26T18:09:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22737.042853 secs 2024-04-26T18:09:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22737.042853 secs 2024-04-26T18:09:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:59 smithi096 ceph-mon[32753]: pgmap v12545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22739.911059 secs 2024-04-26T18:09:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:59 smithi194 ceph-mon[33017]: pgmap v12545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22739.911059 secs 2024-04-26T18:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:59 smithi033 ceph-mon[27845]: pgmap v12545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22739.911059 secs 2024-04-26T18:10:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:10:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:10:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:10:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:01 smithi194 ceph-mon[33017]: pgmap v12546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:01 smithi033 ceph-mon[27845]: pgmap v12546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:01 smithi096 ceph-mon[32753]: pgmap v12546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:03.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:03 smithi033 ceph-mon[27845]: pgmap v12547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:03.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22742.042932 secs 2024-04-26T18:10:03.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:03 smithi194 ceph-mon[33017]: pgmap v12547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22742.042932 secs 2024-04-26T18:10:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:03 smithi096 ceph-mon[32753]: pgmap v12547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22742.042932 secs 2024-04-26T18:10:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:04.510+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:04.510+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:04.510+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22744.911197 secs 2024-04-26T18:10:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22744.911197 secs 2024-04-26T18:10:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22744.911197 secs 2024-04-26T18:10:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:05 smithi194 ceph-mon[33017]: pgmap v12548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:05 smithi033 ceph-mon[27845]: pgmap v12548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:05 smithi096 ceph-mon[32753]: pgmap v12548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:07 smithi194 ceph-mon[33017]: pgmap v12549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:07 smithi033 ceph-mon[27845]: pgmap v12549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:07 smithi096 ceph-mon[32753]: pgmap v12549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22747.043014 secs 2024-04-26T18:10:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22747.043014 secs 2024-04-26T18:10:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22747.043014 secs 2024-04-26T18:10:09.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:09 smithi194 ceph-mon[33017]: pgmap v12550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22749.911261 secs 2024-04-26T18:10:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:09 smithi033 ceph-mon[27845]: pgmap v12550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22749.911261 secs 2024-04-26T18:10:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:09 smithi096 ceph-mon[32753]: pgmap v12550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22749.911261 secs 2024-04-26T18:10:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:11 smithi194 ceph-mon[33017]: pgmap v12551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:11 smithi033 ceph-mon[27845]: pgmap v12551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:11 smithi096 ceph-mon[32753]: pgmap v12551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:13 smithi194 ceph-mon[33017]: pgmap v12552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22752.043075 secs 2024-04-26T18:10:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:13 smithi033 ceph-mon[27845]: pgmap v12552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22752.043075 secs 2024-04-26T18:10:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:13 smithi096 ceph-mon[32753]: pgmap v12552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22752.043075 secs 2024-04-26T18:10:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22754.911341 secs 2024-04-26T18:10:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22754.911341 secs 2024-04-26T18:10:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22754.911341 secs 2024-04-26T18:10:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:15 smithi194 ceph-mon[33017]: pgmap v12553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:15 smithi033 ceph-mon[27845]: pgmap v12553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:15 smithi096 ceph-mon[32753]: pgmap v12553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:17 smithi194 ceph-mon[33017]: pgmap v12554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:17 smithi033 ceph-mon[27845]: pgmap v12554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:17 smithi096 ceph-mon[32753]: pgmap v12554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22757.043163 secs 2024-04-26T18:10:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22759.911466 secs 2024-04-26T18:10:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22757.043163 secs 2024-04-26T18:10:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22759.911466 secs 2024-04-26T18:10:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22757.043163 secs 2024-04-26T18:10:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22759.911466 secs 2024-04-26T18:10:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:19.526+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:19.526+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:19.526+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:19 smithi194 ceph-mon[33017]: pgmap v12555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:19 smithi033 ceph-mon[27845]: pgmap v12555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:19 smithi096 ceph-mon[32753]: pgmap v12555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:21 smithi194 ceph-mon[33017]: pgmap v12556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:10:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:21 smithi033 ceph-mon[27845]: pgmap v12556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:10:21.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:21 smithi096 ceph-mon[32753]: pgmap v12556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:21.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:10:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:10:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:10:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:10:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:10:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:10:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:10:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:10:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:10:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:10:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:23 smithi194 ceph-mon[33017]: pgmap v12557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:23 smithi194 ceph-mon[33017]: pgmap v12558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22762.043266 secs 2024-04-26T18:10:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22764.911588 secs 2024-04-26T18:10:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:23 smithi033 ceph-mon[27845]: pgmap v12557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:23 smithi033 ceph-mon[27845]: pgmap v12558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22762.043266 secs 2024-04-26T18:10:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22764.911588 secs 2024-04-26T18:10:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:23 smithi096 ceph-mon[32753]: pgmap v12557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:23 smithi096 ceph-mon[32753]: pgmap v12558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22762.043266 secs 2024-04-26T18:10:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22764.911588 secs 2024-04-26T18:10:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:25 smithi194 ceph-mon[33017]: pgmap v12559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:10:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:25 smithi033 ceph-mon[27845]: pgmap v12559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:10:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:25 smithi096 ceph-mon[32753]: pgmap v12559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:10:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:27 smithi194 ceph-mon[33017]: pgmap v12560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:27 smithi033 ceph-mon[27845]: pgmap v12560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:27 smithi096 ceph-mon[32753]: pgmap v12560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22767.043353 secs 2024-04-26T18:10:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22769.911678 secs 2024-04-26T18:10:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22767.043353 secs 2024-04-26T18:10:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22769.911678 secs 2024-04-26T18:10:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22767.043353 secs 2024-04-26T18:10:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22769.911678 secs 2024-04-26T18:10:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:29 smithi194 ceph-mon[33017]: pgmap v12561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:29 smithi033 ceph-mon[27845]: pgmap v12561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:29 smithi096 ceph-mon[32753]: pgmap v12561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:31 smithi194 ceph-mon[33017]: pgmap v12562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:31 smithi033 ceph-mon[27845]: pgmap v12562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:31 smithi096 ceph-mon[32753]: pgmap v12562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:10:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:33 smithi194 ceph-mon[33017]: pgmap v12563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:10:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22772.043436 secs 2024-04-26T18:10:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22774.911797 secs 2024-04-26T18:10:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:33 smithi033 ceph-mon[27845]: pgmap v12563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:10:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22772.043436 secs 2024-04-26T18:10:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22774.911797 secs 2024-04-26T18:10:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:33 smithi096 ceph-mon[32753]: pgmap v12563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:10:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22772.043436 secs 2024-04-26T18:10:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22774.911797 secs 2024-04-26T18:10:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:34.526+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:34.526+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:34.526+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:35.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:35 smithi194 ceph-mon[33017]: pgmap v12564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:35 smithi033 ceph-mon[27845]: pgmap v12564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:35 smithi096 ceph-mon[32753]: pgmap v12564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:37.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:37 smithi194 ceph-mon[33017]: pgmap v12565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:37 smithi033 ceph-mon[27845]: pgmap v12565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:37 smithi096 ceph-mon[32753]: pgmap v12565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:38.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22777.043521 secs 2024-04-26T18:10:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22779.911931 secs 2024-04-26T18:10:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22777.043521 secs 2024-04-26T18:10:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22779.911931 secs 2024-04-26T18:10:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22777.043521 secs 2024-04-26T18:10:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22779.911931 secs 2024-04-26T18:10:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:39 smithi194 ceph-mon[33017]: pgmap v12566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:39 smithi033 ceph-mon[27845]: pgmap v12566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:39 smithi096 ceph-mon[32753]: pgmap v12566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:41.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:41 smithi194 ceph-mon[33017]: pgmap v12567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:41 smithi033 ceph-mon[27845]: pgmap v12567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:41 smithi096 ceph-mon[32753]: pgmap v12567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:43 smithi033 ceph-mon[27845]: pgmap v12568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22782.043607 secs 2024-04-26T18:10:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22784.912047 secs 2024-04-26T18:10:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:43 smithi096 ceph-mon[32753]: pgmap v12568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22782.043607 secs 2024-04-26T18:10:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22784.912047 secs 2024-04-26T18:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:43 smithi194 ceph-mon[33017]: pgmap v12568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22782.043607 secs 2024-04-26T18:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22784.912047 secs 2024-04-26T18:10:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:45 smithi033 ceph-mon[27845]: pgmap v12569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:45 smithi096 ceph-mon[32753]: pgmap v12569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:45 smithi194 ceph-mon[33017]: pgmap v12569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:47 smithi033 ceph-mon[27845]: pgmap v12570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:47 smithi096 ceph-mon[32753]: pgmap v12570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:48.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:47 smithi194 ceph-mon[33017]: pgmap v12570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22787.043692 secs 2024-04-26T18:10:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22789.912164 secs 2024-04-26T18:10:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22787.043692 secs 2024-04-26T18:10:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22789.912164 secs 2024-04-26T18:10:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22787.043692 secs 2024-04-26T18:10:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22789.912164 secs 2024-04-26T18:10:49.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:49.541+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:49.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:49.541+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:10:49.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:10:49.541+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:49 smithi033 ceph-mon[27845]: pgmap v12571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:49 smithi096 ceph-mon[32753]: pgmap v12571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:49 smithi194 ceph-mon[33017]: pgmap v12571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:51.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:51 smithi194 ceph-mon[33017]: pgmap v12572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:51 smithi033 ceph-mon[27845]: pgmap v12572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:51 smithi096 ceph-mon[32753]: pgmap v12572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:53 smithi033 ceph-mon[27845]: pgmap v12573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22792.043779 secs 2024-04-26T18:10:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22794.912225 secs 2024-04-26T18:10:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:53 smithi096 ceph-mon[32753]: pgmap v12573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22792.043779 secs 2024-04-26T18:10:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22794.912225 secs 2024-04-26T18:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:53 smithi194 ceph-mon[33017]: pgmap v12573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22792.043779 secs 2024-04-26T18:10:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22794.912225 secs 2024-04-26T18:10:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:55 smithi033 ceph-mon[27845]: pgmap v12574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:55 smithi096 ceph-mon[32753]: pgmap v12574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:55 smithi194 ceph-mon[33017]: pgmap v12574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:57 smithi033 ceph-mon[27845]: pgmap v12575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:57 smithi096 ceph-mon[32753]: pgmap v12575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:58.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:57 smithi194 ceph-mon[33017]: pgmap v12575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:10:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22797.043860 secs 2024-04-26T18:10:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22799.912344 secs 2024-04-26T18:10:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22797.043860 secs 2024-04-26T18:10:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22799.912344 secs 2024-04-26T18:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22797.043860 secs 2024-04-26T18:10:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22799.912344 secs 2024-04-26T18:10:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:10:59 smithi033 ceph-mon[27845]: pgmap v12576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:10:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:10:59 smithi096 ceph-mon[32753]: pgmap v12576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:00.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:10:59 smithi194 ceph-mon[33017]: pgmap v12576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:01 smithi033 ceph-mon[27845]: pgmap v12577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:01 smithi096 ceph-mon[32753]: pgmap v12577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:02.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:01 smithi194 ceph-mon[33017]: pgmap v12577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:03 smithi033 ceph-mon[27845]: pgmap v12578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22802.043946 secs 2024-04-26T18:11:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22804.912459 secs 2024-04-26T18:11:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:03 smithi096 ceph-mon[32753]: pgmap v12578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22802.043946 secs 2024-04-26T18:11:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22804.912459 secs 2024-04-26T18:11:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:03 smithi194 ceph-mon[33017]: pgmap v12578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22802.043946 secs 2024-04-26T18:11:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22804.912459 secs 2024-04-26T18:11:04.557 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:04.558+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:04.558+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:04.558+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:05 smithi096 ceph-mon[32753]: pgmap v12579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:05 smithi194 ceph-mon[33017]: pgmap v12579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:05 smithi033 ceph-mon[27845]: pgmap v12579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:07 smithi096 ceph-mon[32753]: pgmap v12580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:07 smithi194 ceph-mon[33017]: pgmap v12580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:07 smithi033 ceph-mon[27845]: pgmap v12580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:09.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22807.044025 secs 2024-04-26T18:11:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22809.912602 secs 2024-04-26T18:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22807.044025 secs 2024-04-26T18:11:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22809.912602 secs 2024-04-26T18:11:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22807.044025 secs 2024-04-26T18:11:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22809.912602 secs 2024-04-26T18:11:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:09 smithi194 ceph-mon[33017]: pgmap v12581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:09 smithi033 ceph-mon[27845]: pgmap v12581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:09 smithi096 ceph-mon[32753]: pgmap v12581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:11 smithi194 ceph-mon[33017]: pgmap v12582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:11 smithi033 ceph-mon[27845]: pgmap v12582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:11 smithi096 ceph-mon[32753]: pgmap v12582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:13 smithi194 ceph-mon[33017]: pgmap v12583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22812.044110 secs 2024-04-26T18:11:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22814.912698 secs 2024-04-26T18:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:13 smithi033 ceph-mon[27845]: pgmap v12583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22812.044110 secs 2024-04-26T18:11:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22814.912698 secs 2024-04-26T18:11:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:13 smithi096 ceph-mon[32753]: pgmap v12583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22812.044110 secs 2024-04-26T18:11:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22814.912698 secs 2024-04-26T18:11:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:15 smithi194 ceph-mon[33017]: pgmap v12584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:15 smithi033 ceph-mon[27845]: pgmap v12584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:15 smithi096 ceph-mon[32753]: pgmap v12584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:17 smithi194 ceph-mon[33017]: pgmap v12585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:17 smithi033 ceph-mon[27845]: pgmap v12585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:17 smithi096 ceph-mon[32753]: pgmap v12585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:19.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22817.044197 secs 2024-04-26T18:11:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22819.912817 secs 2024-04-26T18:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22817.044197 secs 2024-04-26T18:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22819.912817 secs 2024-04-26T18:11:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22817.044197 secs 2024-04-26T18:11:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22819.912817 secs 2024-04-26T18:11:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:19.572+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:19.572+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:19.572+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:19 smithi194 ceph-mon[33017]: pgmap v12586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:19 smithi033 ceph-mon[27845]: pgmap v12586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:19 smithi096 ceph-mon[32753]: pgmap v12586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T18:11:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T18:11:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0022040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:11:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef0022040 comp rx=0 tx=0).stop 2024-04-26T18:11:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T18:11:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T18:11:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:11:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.777+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:11:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.778+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:11:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:21.780+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0022040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:11:22.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:21 smithi194 ceph-mon[33017]: pgmap v12587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:21 smithi033 ceph-mon[27845]: pgmap v12587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:21 smithi096 ceph-mon[32753]: pgmap v12587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:23.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:11:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:11:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:11:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:23 smithi194 ceph-mon[33017]: pgmap v12588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22822.044264 secs 2024-04-26T18:11:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22824.912960 secs 2024-04-26T18:11:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:11:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:11:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:23 smithi033 ceph-mon[27845]: pgmap v12588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22822.044264 secs 2024-04-26T18:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22824.912960 secs 2024-04-26T18:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:11:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:23 smithi096 ceph-mon[32753]: pgmap v12588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22822.044264 secs 2024-04-26T18:11:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22824.912960 secs 2024-04-26T18:11:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:11:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:11:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:11:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:24 smithi096 ceph-mon[32753]: pgmap v12589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:24 smithi096 ceph-mon[32753]: pgmap v12590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:25.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:24 smithi194 ceph-mon[33017]: pgmap v12589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:24 smithi194 ceph-mon[33017]: pgmap v12590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:24 smithi033 ceph-mon[27845]: pgmap v12589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:24 smithi033 ceph-mon[27845]: pgmap v12590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:25.680+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:11:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:25.680+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:11:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:25.680+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 182 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T18:11:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:25.681+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 182 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:11:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:26 smithi194 ceph-mon[33017]: pgmap v12591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:26 smithi033 ceph-mon[27845]: pgmap v12591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:26 smithi096 ceph-mon[32753]: pgmap v12591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:28 smithi194 ceph-mon[33017]: pgmap v12592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22827.044349 secs 2024-04-26T18:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22829.913080 secs 2024-04-26T18:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:28 smithi033 ceph-mon[27845]: pgmap v12592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22827.044349 secs 2024-04-26T18:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22829.913080 secs 2024-04-26T18:11:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:28 smithi096 ceph-mon[32753]: pgmap v12592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22827.044349 secs 2024-04-26T18:11:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22829.913080 secs 2024-04-26T18:11:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:30 smithi194 ceph-mon[33017]: pgmap v12593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:30 smithi033 ceph-mon[27845]: pgmap v12593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:30 smithi096 ceph-mon[32753]: pgmap v12593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:32 smithi194 ceph-mon[33017]: pgmap v12594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:32 smithi033 ceph-mon[27845]: pgmap v12594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:32 smithi096 ceph-mon[32753]: pgmap v12594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:11:34.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22832.044413 secs 2024-04-26T18:11:34.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22834.913200 secs 2024-04-26T18:11:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22832.044413 secs 2024-04-26T18:11:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22834.913200 secs 2024-04-26T18:11:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22832.044413 secs 2024-04-26T18:11:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22834.913200 secs 2024-04-26T18:11:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:34.583+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:34.583+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:34.583+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:34 smithi194 ceph-mon[33017]: pgmap v12595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:11:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:34 smithi033 ceph-mon[27845]: pgmap v12595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:11:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:34 smithi096 ceph-mon[32753]: pgmap v12595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:11:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:36 smithi194 ceph-mon[33017]: pgmap v12596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:36 smithi033 ceph-mon[27845]: pgmap v12596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:36 smithi096 ceph-mon[32753]: pgmap v12596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:38 smithi194 ceph-mon[33017]: pgmap v12597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22837.044499 secs 2024-04-26T18:11:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22839.913321 secs 2024-04-26T18:11:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:38 smithi033 ceph-mon[27845]: pgmap v12597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22837.044499 secs 2024-04-26T18:11:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22839.913321 secs 2024-04-26T18:11:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:38 smithi096 ceph-mon[32753]: pgmap v12597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22837.044499 secs 2024-04-26T18:11:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22839.913321 secs 2024-04-26T18:11:41.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:40 smithi194 ceph-mon[33017]: pgmap v12598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:40 smithi033 ceph-mon[27845]: pgmap v12598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:40 smithi096 ceph-mon[32753]: pgmap v12598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:43.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:42 smithi194 ceph-mon[33017]: pgmap v12599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:42 smithi033 ceph-mon[27845]: pgmap v12599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:42 smithi096 ceph-mon[32753]: pgmap v12599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22842.044583 secs 2024-04-26T18:11:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22844.913445 secs 2024-04-26T18:11:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22842.044583 secs 2024-04-26T18:11:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22844.913445 secs 2024-04-26T18:11:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22842.044583 secs 2024-04-26T18:11:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22844.913445 secs 2024-04-26T18:11:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:44 smithi194 ceph-mon[33017]: pgmap v12600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:44 smithi033 ceph-mon[27845]: pgmap v12600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:44 smithi096 ceph-mon[32753]: pgmap v12600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:46 smithi194 ceph-mon[33017]: pgmap v12601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:46 smithi033 ceph-mon[27845]: pgmap v12601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:46 smithi096 ceph-mon[32753]: pgmap v12601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:48 smithi194 ceph-mon[33017]: pgmap v12602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22847.044671 secs 2024-04-26T18:11:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22849.913542 secs 2024-04-26T18:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:48 smithi033 ceph-mon[27845]: pgmap v12602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22847.044671 secs 2024-04-26T18:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22849.913542 secs 2024-04-26T18:11:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:48 smithi096 ceph-mon[32753]: pgmap v12602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22847.044671 secs 2024-04-26T18:11:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22849.913542 secs 2024-04-26T18:11:49.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:49.598+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:49.598+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:11:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:11:49.598+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:50 smithi194 ceph-mon[33017]: pgmap v12603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:50 smithi033 ceph-mon[27845]: pgmap v12603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:50 smithi096 ceph-mon[32753]: pgmap v12603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:52 smithi194 ceph-mon[33017]: pgmap v12604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:52 smithi033 ceph-mon[27845]: pgmap v12604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:52 smithi096 ceph-mon[32753]: pgmap v12604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22852.044756 secs 2024-04-26T18:11:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22854.913647 secs 2024-04-26T18:11:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22852.044756 secs 2024-04-26T18:11:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22854.913647 secs 2024-04-26T18:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22852.044756 secs 2024-04-26T18:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22854.913647 secs 2024-04-26T18:11:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:54 smithi033 ceph-mon[27845]: pgmap v12605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:54 smithi096 ceph-mon[32753]: pgmap v12605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:54 smithi194 ceph-mon[33017]: pgmap v12605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:56 smithi033 ceph-mon[27845]: pgmap v12606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:56 smithi096 ceph-mon[32753]: pgmap v12606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:56 smithi194 ceph-mon[33017]: pgmap v12606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:11:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:58 smithi033 ceph-mon[27845]: pgmap v12607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22857.044839 secs 2024-04-26T18:11:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:11:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22859.913765 secs 2024-04-26T18:11:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:58 smithi096 ceph-mon[32753]: pgmap v12607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22857.044839 secs 2024-04-26T18:11:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:11:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22859.913765 secs 2024-04-26T18:11:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:58 smithi194 ceph-mon[33017]: pgmap v12607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:11:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22857.044839 secs 2024-04-26T18:11:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:11:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22859.913765 secs 2024-04-26T18:12:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:00 smithi033 ceph-mon[27845]: pgmap v12608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:00 smithi096 ceph-mon[32753]: pgmap v12608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:00 smithi194 ceph-mon[33017]: pgmap v12608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:02 smithi033 ceph-mon[27845]: pgmap v12609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:02 smithi096 ceph-mon[32753]: pgmap v12609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:02 smithi194 ceph-mon[33017]: pgmap v12609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22862.044924 secs 2024-04-26T18:12:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22864.913889 secs 2024-04-26T18:12:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22862.044924 secs 2024-04-26T18:12:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22864.913889 secs 2024-04-26T18:12:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22862.044924 secs 2024-04-26T18:12:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22864.913889 secs 2024-04-26T18:12:04.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:04.610+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:04.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:04.610+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:04.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:04.610+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:12:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:04 smithi033 ceph-mon[27845]: pgmap v12610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:04 smithi096 ceph-mon[32753]: pgmap v12610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:04 smithi194 ceph-mon[33017]: pgmap v12610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:06 smithi033 ceph-mon[27845]: pgmap v12611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:06 smithi096 ceph-mon[32753]: pgmap v12611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:06 smithi194 ceph-mon[33017]: pgmap v12611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:08 smithi033 ceph-mon[27845]: pgmap v12612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22867.044983 secs 2024-04-26T18:12:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22869.914010 secs 2024-04-26T18:12:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:08 smithi096 ceph-mon[32753]: pgmap v12612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22867.044983 secs 2024-04-26T18:12:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22869.914010 secs 2024-04-26T18:12:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:08 smithi194 ceph-mon[33017]: pgmap v12612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22867.044983 secs 2024-04-26T18:12:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22869.914010 secs 2024-04-26T18:12:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:09 smithi194 ceph-mon[33017]: pgmap v12613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:09 smithi033 ceph-mon[27845]: pgmap v12613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:09 smithi096 ceph-mon[32753]: pgmap v12613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:12 smithi194 ceph-mon[33017]: pgmap v12614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:12 smithi033 ceph-mon[27845]: pgmap v12614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:12 smithi096 ceph-mon[32753]: pgmap v12614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22872.045074 secs 2024-04-26T18:12:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22874.914120 secs 2024-04-26T18:12:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22872.045074 secs 2024-04-26T18:12:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22874.914120 secs 2024-04-26T18:12:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22872.045074 secs 2024-04-26T18:12:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22874.914120 secs 2024-04-26T18:12:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:14 smithi194 ceph-mon[33017]: pgmap v12615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:14 smithi033 ceph-mon[27845]: pgmap v12615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:14 smithi096 ceph-mon[32753]: pgmap v12615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:16 smithi194 ceph-mon[33017]: pgmap v12616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:16 smithi033 ceph-mon[27845]: pgmap v12616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:16 smithi096 ceph-mon[32753]: pgmap v12616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:18 smithi194 ceph-mon[33017]: pgmap v12617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22877.045163 secs 2024-04-26T18:12:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22879.914249 secs 2024-04-26T18:12:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:18 smithi033 ceph-mon[27845]: pgmap v12617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22877.045163 secs 2024-04-26T18:12:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22879.914249 secs 2024-04-26T18:12:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:18 smithi096 ceph-mon[32753]: pgmap v12617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22877.045163 secs 2024-04-26T18:12:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22879.914249 secs 2024-04-26T18:12:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:19.618+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:19.618+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:19.618+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:12:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:20 smithi194 ceph-mon[33017]: pgmap v12618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:20 smithi033 ceph-mon[27845]: pgmap v12618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:20 smithi096 ceph-mon[32753]: pgmap v12618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:22 smithi194 ceph-mon[33017]: pgmap v12619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:22 smithi033 ceph-mon[27845]: pgmap v12619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:22 smithi096 ceph-mon[32753]: pgmap v12619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22882.045243 secs 2024-04-26T18:12:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22884.914358 secs 2024-04-26T18:12:24.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22882.045243 secs 2024-04-26T18:12:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22884.914358 secs 2024-04-26T18:12:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22882.045243 secs 2024-04-26T18:12:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22884.914358 secs 2024-04-26T18:12:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:24 smithi194 ceph-mon[33017]: pgmap v12620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:12:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:24 smithi033 ceph-mon[27845]: pgmap v12620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:12:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:24 smithi096 ceph-mon[32753]: pgmap v12620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:12:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:12:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:12:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:12:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:12:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:12:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:12:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:12:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:12:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:12:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:26 smithi033 ceph-mon[27845]: pgmap v12621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:12:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:26 smithi033 ceph-mon[27845]: pgmap v12622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:26 smithi096 ceph-mon[32753]: pgmap v12621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:12:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:26 smithi096 ceph-mon[32753]: pgmap v12622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:26 smithi194 ceph-mon[33017]: pgmap v12621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:12:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:26 smithi194 ceph-mon[33017]: pgmap v12622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:28 smithi033 ceph-mon[27845]: pgmap v12623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22887.045330 secs 2024-04-26T18:12:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22889.914449 secs 2024-04-26T18:12:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:28 smithi096 ceph-mon[32753]: pgmap v12623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22887.045330 secs 2024-04-26T18:12:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22889.914449 secs 2024-04-26T18:12:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:28 smithi194 ceph-mon[33017]: pgmap v12623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22887.045330 secs 2024-04-26T18:12:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22889.914449 secs 2024-04-26T18:12:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:30 smithi033 ceph-mon[27845]: pgmap v12624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:12:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:30 smithi096 ceph-mon[32753]: pgmap v12624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:12:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:30 smithi194 ceph-mon[33017]: pgmap v12624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:12:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:32 smithi033 ceph-mon[27845]: pgmap v12625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:32 smithi096 ceph-mon[32753]: pgmap v12625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:32 smithi194 ceph-mon[33017]: pgmap v12625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22892.045408 secs 2024-04-26T18:12:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22894.914580 secs 2024-04-26T18:12:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22892.045408 secs 2024-04-26T18:12:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22894.914580 secs 2024-04-26T18:12:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22892.045408 secs 2024-04-26T18:12:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22894.914580 secs 2024-04-26T18:12:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:34.633+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:34.633+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:12:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:34.633+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:34 smithi033 ceph-mon[27845]: pgmap v12626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:34 smithi096 ceph-mon[32753]: pgmap v12626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:34 smithi194 ceph-mon[33017]: pgmap v12626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:12:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:36 smithi033 ceph-mon[27845]: pgmap v12627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:12:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:36 smithi096 ceph-mon[32753]: pgmap v12627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:12:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:36 smithi194 ceph-mon[33017]: pgmap v12627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:12:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:38 smithi033 ceph-mon[27845]: pgmap v12628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22897.045489 secs 2024-04-26T18:12:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22899.914731 secs 2024-04-26T18:12:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:38 smithi096 ceph-mon[32753]: pgmap v12628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22897.045489 secs 2024-04-26T18:12:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22899.914731 secs 2024-04-26T18:12:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:38 smithi194 ceph-mon[33017]: pgmap v12628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22897.045489 secs 2024-04-26T18:12:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22899.914731 secs 2024-04-26T18:12:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:40 smithi033 ceph-mon[27845]: pgmap v12629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:40 smithi096 ceph-mon[32753]: pgmap v12629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:40 smithi194 ceph-mon[33017]: pgmap v12629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:42 smithi033 ceph-mon[27845]: pgmap v12630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:42 smithi096 ceph-mon[32753]: pgmap v12630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:42 smithi194 ceph-mon[33017]: pgmap v12630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22902.045573 secs 2024-04-26T18:12:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22904.914851 secs 2024-04-26T18:12:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22902.045573 secs 2024-04-26T18:12:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22904.914851 secs 2024-04-26T18:12:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22902.045573 secs 2024-04-26T18:12:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22904.914851 secs 2024-04-26T18:12:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:44 smithi033 ceph-mon[27845]: pgmap v12631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:44 smithi096 ceph-mon[32753]: pgmap v12631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:44 smithi194 ceph-mon[33017]: pgmap v12631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:45 smithi033 ceph-mon[27845]: pgmap v12632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:45 smithi096 ceph-mon[32753]: pgmap v12632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:45 smithi194 ceph-mon[33017]: pgmap v12632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:48 smithi194 ceph-mon[33017]: pgmap v12633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22907.045656 secs 2024-04-26T18:12:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:48 smithi033 ceph-mon[27845]: pgmap v12633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22907.045656 secs 2024-04-26T18:12:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:48 smithi096 ceph-mon[32753]: pgmap v12633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22907.045656 secs 2024-04-26T18:12:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:49.648+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:49.648+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:12:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:12:49.648+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:12:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22909.914973 secs 2024-04-26T18:12:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22909.914973 secs 2024-04-26T18:12:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22909.914973 secs 2024-04-26T18:12:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:50 smithi194 ceph-mon[33017]: pgmap v12634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:50 smithi033 ceph-mon[27845]: pgmap v12634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:50 smithi096 ceph-mon[32753]: pgmap v12634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:52 smithi194 ceph-mon[33017]: pgmap v12635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:52.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:52 smithi033 ceph-mon[27845]: pgmap v12635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:52 smithi096 ceph-mon[32753]: pgmap v12635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22912.045718 secs 2024-04-26T18:12:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22912.045718 secs 2024-04-26T18:12:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22912.045718 secs 2024-04-26T18:12:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:54 smithi194 ceph-mon[33017]: pgmap v12636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22914.915069 secs 2024-04-26T18:12:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:54 smithi033 ceph-mon[27845]: pgmap v12636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22914.915069 secs 2024-04-26T18:12:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:54 smithi096 ceph-mon[32753]: pgmap v12636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22914.915069 secs 2024-04-26T18:12:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:56 smithi194 ceph-mon[33017]: pgmap v12637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:56 smithi033 ceph-mon[27845]: pgmap v12637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:56 smithi096 ceph-mon[32753]: pgmap v12637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:12:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:58 smithi194 ceph-mon[33017]: pgmap v12638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22917.045802 secs 2024-04-26T18:12:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:58 smithi033 ceph-mon[27845]: pgmap v12638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22917.045802 secs 2024-04-26T18:12:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:58 smithi096 ceph-mon[32753]: pgmap v12638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:12:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22917.045802 secs 2024-04-26T18:12:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:12:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22919.915187 secs 2024-04-26T18:12:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:12:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22919.915187 secs 2024-04-26T18:12:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:12:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22919.915187 secs 2024-04-26T18:13:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:00 smithi194 ceph-mon[33017]: pgmap v12639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:00 smithi033 ceph-mon[27845]: pgmap v12639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:00 smithi096 ceph-mon[32753]: pgmap v12639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:02.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:02 smithi194 ceph-mon[33017]: pgmap v12640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:02 smithi033 ceph-mon[27845]: pgmap v12640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:02 smithi096 ceph-mon[32753]: pgmap v12640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22922.045888 secs 2024-04-26T18:13:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22924.915310 secs 2024-04-26T18:13:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22922.045888 secs 2024-04-26T18:13:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22924.915310 secs 2024-04-26T18:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22922.045888 secs 2024-04-26T18:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22924.915310 secs 2024-04-26T18:13:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:04.659+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:04.659+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:04.659+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:13:04.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:04 smithi033 ceph-mon[27845]: pgmap v12641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:04 smithi194 ceph-mon[33017]: pgmap v12641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:04 smithi096 ceph-mon[32753]: pgmap v12641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:06 smithi194 ceph-mon[33017]: pgmap v12642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:06 smithi033 ceph-mon[27845]: pgmap v12642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:06 smithi096 ceph-mon[32753]: pgmap v12642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:08 smithi194 ceph-mon[33017]: pgmap v12643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22927.045974 secs 2024-04-26T18:13:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22929.915427 secs 2024-04-26T18:13:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:08 smithi033 ceph-mon[27845]: pgmap v12643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22927.045974 secs 2024-04-26T18:13:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22929.915427 secs 2024-04-26T18:13:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:08 smithi096 ceph-mon[32753]: pgmap v12643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22927.045974 secs 2024-04-26T18:13:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22929.915427 secs 2024-04-26T18:13:10.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:10 smithi194 ceph-mon[33017]: pgmap v12644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:10 smithi033 ceph-mon[27845]: pgmap v12644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:10 smithi096 ceph-mon[32753]: pgmap v12644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:12 smithi194 ceph-mon[33017]: pgmap v12645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:12 smithi033 ceph-mon[27845]: pgmap v12645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:12 smithi096 ceph-mon[32753]: pgmap v12645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22932.046015 secs 2024-04-26T18:13:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22934.915509 secs 2024-04-26T18:13:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22932.046015 secs 2024-04-26T18:13:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22934.915509 secs 2024-04-26T18:13:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22932.046015 secs 2024-04-26T18:13:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22934.915509 secs 2024-04-26T18:13:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:14 smithi194 ceph-mon[33017]: pgmap v12646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:14 smithi033 ceph-mon[27845]: pgmap v12646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:14 smithi096 ceph-mon[32753]: pgmap v12646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:16 smithi194 ceph-mon[33017]: pgmap v12647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:16 smithi033 ceph-mon[27845]: pgmap v12647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:16 smithi096 ceph-mon[32753]: pgmap v12647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:18 smithi194 ceph-mon[33017]: pgmap v12648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22937.046105 secs 2024-04-26T18:13:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22939.915631 secs 2024-04-26T18:13:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:18 smithi033 ceph-mon[27845]: pgmap v12648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22937.046105 secs 2024-04-26T18:13:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22939.915631 secs 2024-04-26T18:13:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:18 smithi096 ceph-mon[32753]: pgmap v12648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22937.046105 secs 2024-04-26T18:13:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22939.915631 secs 2024-04-26T18:13:19.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:19.664+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:19.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:19.664+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:13:19.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:19.664+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:20 smithi194 ceph-mon[33017]: pgmap v12649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:20 smithi033 ceph-mon[27845]: pgmap v12649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:20 smithi096 ceph-mon[32753]: pgmap v12649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:22 smithi033 ceph-mon[27845]: pgmap v12650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:22 smithi096 ceph-mon[32753]: pgmap v12650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:22 smithi194 ceph-mon[33017]: pgmap v12650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22942.046196 secs 2024-04-26T18:13:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22944.915748 secs 2024-04-26T18:13:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22942.046196 secs 2024-04-26T18:13:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22944.915748 secs 2024-04-26T18:13:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22942.046196 secs 2024-04-26T18:13:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22944.915748 secs 2024-04-26T18:13:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:24 smithi033 ceph-mon[27845]: pgmap v12651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:24 smithi096 ceph-mon[32753]: pgmap v12651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:24 smithi194 ceph-mon[33017]: pgmap v12651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:26 smithi033 ceph-mon[27845]: pgmap v12652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:13:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:26 smithi096 ceph-mon[32753]: pgmap v12652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:13:27.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:26 smithi194 ceph-mon[33017]: pgmap v12652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:13:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:13:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:13:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:13:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:13:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:13:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:13:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:13:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:13:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:13:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:28 smithi033 ceph-mon[27845]: pgmap v12653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:28 smithi033 ceph-mon[27845]: pgmap v12654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22947.046287 secs 2024-04-26T18:13:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22949.915848 secs 2024-04-26T18:13:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:28 smithi096 ceph-mon[32753]: pgmap v12653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:28 smithi096 ceph-mon[32753]: pgmap v12654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22947.046287 secs 2024-04-26T18:13:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22949.915848 secs 2024-04-26T18:13:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:28 smithi194 ceph-mon[33017]: pgmap v12653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:28 smithi194 ceph-mon[33017]: pgmap v12654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22947.046287 secs 2024-04-26T18:13:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22949.915848 secs 2024-04-26T18:13:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:30 smithi033 ceph-mon[27845]: pgmap v12655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:13:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:30 smithi096 ceph-mon[32753]: pgmap v12655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:13:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:30 smithi194 ceph-mon[33017]: pgmap v12655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:13:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:32 smithi033 ceph-mon[27845]: pgmap v12656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:32 smithi096 ceph-mon[32753]: pgmap v12656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:32 smithi194 ceph-mon[33017]: pgmap v12656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22952.046372 secs 2024-04-26T18:13:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22954.915968 secs 2024-04-26T18:13:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22952.046372 secs 2024-04-26T18:13:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22954.915968 secs 2024-04-26T18:13:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22952.046372 secs 2024-04-26T18:13:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22954.915968 secs 2024-04-26T18:13:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:34.680+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:34.680+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:34.680+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:13:35.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:34 smithi033 ceph-mon[27845]: pgmap v12657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:34 smithi194 ceph-mon[33017]: pgmap v12657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:34 smithi096 ceph-mon[32753]: pgmap v12657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:37.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:36 smithi194 ceph-mon[33017]: pgmap v12658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:36 smithi033 ceph-mon[27845]: pgmap v12658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:36 smithi096 ceph-mon[32753]: pgmap v12658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:13:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:38 smithi194 ceph-mon[33017]: pgmap v12659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:13:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22957.046458 secs 2024-04-26T18:13:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22959.916091 secs 2024-04-26T18:13:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:38 smithi033 ceph-mon[27845]: pgmap v12659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:13:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22957.046458 secs 2024-04-26T18:13:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22959.916091 secs 2024-04-26T18:13:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:38 smithi096 ceph-mon[32753]: pgmap v12659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:13:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22957.046458 secs 2024-04-26T18:13:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22959.916091 secs 2024-04-26T18:13:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:40 smithi194 ceph-mon[33017]: pgmap v12660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:40 smithi033 ceph-mon[27845]: pgmap v12660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:40 smithi096 ceph-mon[32753]: pgmap v12660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:42 smithi194 ceph-mon[33017]: pgmap v12661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:42 smithi033 ceph-mon[27845]: pgmap v12661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:42 smithi096 ceph-mon[32753]: pgmap v12661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:44.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22962.046543 secs 2024-04-26T18:13:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22964.916212 secs 2024-04-26T18:13:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22962.046543 secs 2024-04-26T18:13:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22964.916212 secs 2024-04-26T18:13:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22962.046543 secs 2024-04-26T18:13:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22964.916212 secs 2024-04-26T18:13:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:44 smithi194 ceph-mon[33017]: pgmap v12662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:44 smithi033 ceph-mon[27845]: pgmap v12662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:44 smithi096 ceph-mon[32753]: pgmap v12662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:46 smithi194 ceph-mon[33017]: pgmap v12663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:46 smithi033 ceph-mon[27845]: pgmap v12663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:46 smithi096 ceph-mon[32753]: pgmap v12663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:48 smithi194 ceph-mon[33017]: pgmap v12664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22967.046628 secs 2024-04-26T18:13:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22969.916331 secs 2024-04-26T18:13:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:48 smithi033 ceph-mon[27845]: pgmap v12664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22967.046628 secs 2024-04-26T18:13:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22969.916331 secs 2024-04-26T18:13:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:48 smithi096 ceph-mon[32753]: pgmap v12664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22967.046628 secs 2024-04-26T18:13:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22969.916331 secs 2024-04-26T18:13:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:49.681+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:49.681+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:13:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:13:49.681+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:13:51.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:50 smithi194 ceph-mon[33017]: pgmap v12665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:50 smithi033 ceph-mon[27845]: pgmap v12665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:50 smithi096 ceph-mon[32753]: pgmap v12665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:52 smithi194 ceph-mon[33017]: pgmap v12666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:52 smithi033 ceph-mon[27845]: pgmap v12666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:52 smithi096 ceph-mon[32753]: pgmap v12666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22972.046713 secs 2024-04-26T18:13:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22974.916449 secs 2024-04-26T18:13:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22972.046713 secs 2024-04-26T18:13:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22974.916449 secs 2024-04-26T18:13:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22972.046713 secs 2024-04-26T18:13:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22974.916449 secs 2024-04-26T18:13:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:54 smithi194 ceph-mon[33017]: pgmap v12667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:54 smithi033 ceph-mon[27845]: pgmap v12667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:54 smithi096 ceph-mon[32753]: pgmap v12667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:56 smithi194 ceph-mon[33017]: pgmap v12668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:56 smithi033 ceph-mon[27845]: pgmap v12668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:56 smithi096 ceph-mon[32753]: pgmap v12668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:13:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:58 smithi194 ceph-mon[33017]: pgmap v12669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22977.046798 secs 2024-04-26T18:13:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:13:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22979.916591 secs 2024-04-26T18:13:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:58 smithi033 ceph-mon[27845]: pgmap v12669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22977.046798 secs 2024-04-26T18:13:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:13:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22979.916591 secs 2024-04-26T18:13:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:58 smithi096 ceph-mon[32753]: pgmap v12669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:13:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22977.046798 secs 2024-04-26T18:13:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:13:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22979.916591 secs 2024-04-26T18:14:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:00 smithi194 ceph-mon[33017]: pgmap v12670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:00 smithi033 ceph-mon[27845]: pgmap v12670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:00 smithi096 ceph-mon[32753]: pgmap v12670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:02 smithi194 ceph-mon[33017]: pgmap v12671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:02 smithi033 ceph-mon[27845]: pgmap v12671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:02 smithi096 ceph-mon[32753]: pgmap v12671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22982.046882 secs 2024-04-26T18:14:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22984.916708 secs 2024-04-26T18:14:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22982.046882 secs 2024-04-26T18:14:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22984.916708 secs 2024-04-26T18:14:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22982.046882 secs 2024-04-26T18:14:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22984.916708 secs 2024-04-26T18:14:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:04.696+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:04.696+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:04.696+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:14:05.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:04 smithi033 ceph-mon[27845]: pgmap v12672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:04 smithi194 ceph-mon[33017]: pgmap v12672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:04 smithi096 ceph-mon[32753]: pgmap v12672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:06 smithi194 ceph-mon[33017]: pgmap v12673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:06 smithi033 ceph-mon[27845]: pgmap v12673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:06 smithi096 ceph-mon[32753]: pgmap v12673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:08 smithi194 ceph-mon[33017]: pgmap v12674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22987.046980 secs 2024-04-26T18:14:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22989.916850 secs 2024-04-26T18:14:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:08 smithi033 ceph-mon[27845]: pgmap v12674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22987.046980 secs 2024-04-26T18:14:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22989.916850 secs 2024-04-26T18:14:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:08 smithi096 ceph-mon[32753]: pgmap v12674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22987.046980 secs 2024-04-26T18:14:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22989.916850 secs 2024-04-26T18:14:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:10 smithi194 ceph-mon[33017]: pgmap v12675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:10 smithi033 ceph-mon[27845]: pgmap v12675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:10 smithi096 ceph-mon[32753]: pgmap v12675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:12 smithi194 ceph-mon[33017]: pgmap v12676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:12 smithi033 ceph-mon[27845]: pgmap v12676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:12 smithi096 ceph-mon[32753]: pgmap v12676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22992.047009 secs 2024-04-26T18:14:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22994.916967 secs 2024-04-26T18:14:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22992.047009 secs 2024-04-26T18:14:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22994.916967 secs 2024-04-26T18:14:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22992.047009 secs 2024-04-26T18:14:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22994.916967 secs 2024-04-26T18:14:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:14 smithi194 ceph-mon[33017]: pgmap v12677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:14 smithi033 ceph-mon[27845]: pgmap v12677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:14 smithi096 ceph-mon[32753]: pgmap v12677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:16 smithi194 ceph-mon[33017]: pgmap v12678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:17.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:16 smithi033 ceph-mon[27845]: pgmap v12678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:16 smithi096 ceph-mon[32753]: pgmap v12678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:18 smithi194 ceph-mon[33017]: pgmap v12679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 22997.047098 secs 2024-04-26T18:14:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 22999.917094 secs 2024-04-26T18:14:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:18 smithi033 ceph-mon[27845]: pgmap v12679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 22997.047098 secs 2024-04-26T18:14:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 22999.917094 secs 2024-04-26T18:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:18 smithi096 ceph-mon[32753]: pgmap v12679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 22997.047098 secs 2024-04-26T18:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 22999.917094 secs 2024-04-26T18:14:19.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:19.711+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:19.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:19.711+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:19.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:19.711+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:14:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:20 smithi194 ceph-mon[33017]: pgmap v12680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:20 smithi033 ceph-mon[27845]: pgmap v12680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:20 smithi096 ceph-mon[32753]: pgmap v12680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:22 smithi194 ceph-mon[33017]: pgmap v12681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:22 smithi033 ceph-mon[27845]: pgmap v12681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:22 smithi096 ceph-mon[32753]: pgmap v12681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23002.047158 secs 2024-04-26T18:14:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23004.917212 secs 2024-04-26T18:14:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23002.047158 secs 2024-04-26T18:14:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23004.917212 secs 2024-04-26T18:14:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23002.047158 secs 2024-04-26T18:14:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23004.917212 secs 2024-04-26T18:14:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:24 smithi033 ceph-mon[27845]: pgmap v12682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:24 smithi096 ceph-mon[32753]: pgmap v12682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:24 smithi194 ceph-mon[33017]: pgmap v12682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:25.712+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:14:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:25.712+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:14:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:25.712+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 183 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:14:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:25.712+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 183 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T18:14:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:26 smithi033 ceph-mon[27845]: pgmap v12683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:26 smithi096 ceph-mon[32753]: pgmap v12683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:26 smithi194 ceph-mon[33017]: pgmap v12683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:14:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:14:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:14:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:28 smithi033 ceph-mon[27845]: pgmap v12684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23007.047225 secs 2024-04-26T18:14:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23009.917303 secs 2024-04-26T18:14:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:28 smithi096 ceph-mon[32753]: pgmap v12684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23007.047225 secs 2024-04-26T18:14:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23009.917303 secs 2024-04-26T18:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:28 smithi194 ceph-mon[33017]: pgmap v12684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23007.047225 secs 2024-04-26T18:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23009.917303 secs 2024-04-26T18:14:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:14:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:14:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:14:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:14:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:14:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:14:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:14:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:30 smithi033 ceph-mon[27845]: pgmap v12685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:30 smithi033 ceph-mon[27845]: pgmap v12686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:14:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:30 smithi033 ceph-mon[27845]: pgmap v12687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:30 smithi096 ceph-mon[32753]: pgmap v12685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:30 smithi096 ceph-mon[32753]: pgmap v12686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:14:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:30 smithi096 ceph-mon[32753]: pgmap v12687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:30 smithi194 ceph-mon[33017]: pgmap v12685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:30 smithi194 ceph-mon[33017]: pgmap v12686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:14:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:30 smithi194 ceph-mon[33017]: pgmap v12687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:32 smithi033 ceph-mon[27845]: pgmap v12688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:32 smithi096 ceph-mon[32753]: pgmap v12688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:32 smithi194 ceph-mon[33017]: pgmap v12688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23012.047324 secs 2024-04-26T18:14:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23014.917422 secs 2024-04-26T18:14:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23012.047324 secs 2024-04-26T18:14:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23014.917422 secs 2024-04-26T18:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23012.047324 secs 2024-04-26T18:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23014.917422 secs 2024-04-26T18:14:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:34.723+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:34.723+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:34.723+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:14:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:34 smithi033 ceph-mon[27845]: pgmap v12689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:14:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:34 smithi096 ceph-mon[32753]: pgmap v12689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:34 smithi194 ceph-mon[33017]: pgmap v12689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:14:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:36 smithi033 ceph-mon[27845]: pgmap v12690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:36 smithi096 ceph-mon[32753]: pgmap v12690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:36 smithi194 ceph-mon[33017]: pgmap v12690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23017.047422 secs 2024-04-26T18:14:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:38 smithi096 ceph-mon[32753]: pgmap v12691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23019.917487 secs 2024-04-26T18:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23017.047422 secs 2024-04-26T18:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:38 smithi194 ceph-mon[33017]: pgmap v12691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23019.917487 secs 2024-04-26T18:14:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23017.047422 secs 2024-04-26T18:14:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:38 smithi033 ceph-mon[27845]: pgmap v12691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:14:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23019.917487 secs 2024-04-26T18:14:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:40 smithi096 ceph-mon[32753]: pgmap v12692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:14:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:40 smithi194 ceph-mon[33017]: pgmap v12692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:14:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:40 smithi033 ceph-mon[27845]: pgmap v12692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:14:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:42 smithi194 ceph-mon[33017]: pgmap v12693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:42 smithi033 ceph-mon[27845]: pgmap v12693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:42 smithi096 ceph-mon[32753]: pgmap v12693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23022.047504 secs 2024-04-26T18:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23024.917608 secs 2024-04-26T18:14:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23022.047504 secs 2024-04-26T18:14:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23024.917608 secs 2024-04-26T18:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23022.047504 secs 2024-04-26T18:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23024.917608 secs 2024-04-26T18:14:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:44 smithi194 ceph-mon[33017]: pgmap v12694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:44 smithi033 ceph-mon[27845]: pgmap v12694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:44 smithi096 ceph-mon[32753]: pgmap v12694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:46 smithi033 ceph-mon[27845]: pgmap v12695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:46 smithi096 ceph-mon[32753]: pgmap v12695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:46 smithi194 ceph-mon[33017]: pgmap v12695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23027.047589 secs 2024-04-26T18:14:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:48 smithi033 ceph-mon[27845]: pgmap v12696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23029.917723 secs 2024-04-26T18:14:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23027.047589 secs 2024-04-26T18:14:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:48 smithi096 ceph-mon[32753]: pgmap v12696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23029.917723 secs 2024-04-26T18:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23027.047589 secs 2024-04-26T18:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:48 smithi194 ceph-mon[33017]: pgmap v12696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23029.917723 secs 2024-04-26T18:14:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:49.726+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:49.726+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:14:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:14:49.726+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:14:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:50 smithi033 ceph-mon[27845]: pgmap v12697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:50 smithi096 ceph-mon[32753]: pgmap v12697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:50 smithi194 ceph-mon[33017]: pgmap v12697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:53.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:52 smithi194 ceph-mon[33017]: pgmap v12698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:52 smithi033 ceph-mon[27845]: pgmap v12698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:52 smithi096 ceph-mon[32753]: pgmap v12698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23032.047672 secs 2024-04-26T18:14:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23034.917859 secs 2024-04-26T18:14:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23032.047672 secs 2024-04-26T18:14:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23034.917859 secs 2024-04-26T18:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23032.047672 secs 2024-04-26T18:14:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23034.917859 secs 2024-04-26T18:14:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:54 smithi033 ceph-mon[27845]: pgmap v12699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:54 smithi096 ceph-mon[32753]: pgmap v12699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:54 smithi194 ceph-mon[33017]: pgmap v12699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:56 smithi033 ceph-mon[27845]: pgmap v12700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:56 smithi096 ceph-mon[32753]: pgmap v12700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:56 smithi194 ceph-mon[33017]: pgmap v12700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:14:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23037.047771 secs 2024-04-26T18:14:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:58 smithi033 ceph-mon[27845]: pgmap v12701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:14:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23039.917981 secs 2024-04-26T18:14:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23037.047771 secs 2024-04-26T18:14:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:58 smithi096 ceph-mon[32753]: pgmap v12701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:14:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23039.917981 secs 2024-04-26T18:14:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23037.047771 secs 2024-04-26T18:14:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:58 smithi194 ceph-mon[33017]: pgmap v12701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:14:59.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:14:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23039.917981 secs 2024-04-26T18:15:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:00 smithi096 ceph-mon[32753]: pgmap v12702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:00 smithi194 ceph-mon[33017]: pgmap v12702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:00 smithi033 ceph-mon[27845]: pgmap v12702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:02 smithi096 ceph-mon[32753]: pgmap v12703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:03.215 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:02 smithi033 ceph-mon[27845]: pgmap v12703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:02 smithi194 ceph-mon[33017]: pgmap v12703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 1 included below; oldest blocked for > 23042.047854 secs 2024-04-26T18:15:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:03 smithi194 ceph-mon[33017]: slow request 23042.047853 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T18:15:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:03 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 23044.918097 secs 2024-04-26T18:15:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:03 smithi194 ceph-mon[33017]: slow request 23044.918096 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T18:15:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 1 included below; oldest blocked for > 23042.047854 secs 2024-04-26T18:15:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:03 smithi033 ceph-mon[27845]: slow request 23042.047853 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T18:15:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:03 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 23044.918097 secs 2024-04-26T18:15:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:03 smithi033 ceph-mon[27845]: slow request 23044.918096 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T18:15:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 1 included below; oldest blocked for > 23042.047854 secs 2024-04-26T18:15:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:03 smithi096 ceph-mon[32753]: slow request 23042.047853 seconds old, received at 2024-04-26T11:51:00.758802+0000: client_request(client.25216:250 mksnap #0x1//scheduled-2024-04-26-11_51_00_UTC 2024-04-26T11:51:00.758374+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-26T18:15:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:03 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 23044.918097 secs 2024-04-26T18:15:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:03 smithi096 ceph-mon[32753]: slow request 23044.918096 seconds old, received at 2024-04-26T11:50:58.567864+0000: peer_request:mds.1:1 currently dispatched 2024-04-26T18:15:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:04.742+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:04.742+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:04.742+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:15:05.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:04 smithi033 ceph-mon[27845]: pgmap v12704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:04 smithi194 ceph-mon[33017]: pgmap v12704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:04 smithi096 ceph-mon[32753]: pgmap v12704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:06 smithi194 ceph-mon[33017]: pgmap v12705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:06 smithi033 ceph-mon[27845]: pgmap v12705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:06 smithi096 ceph-mon[32753]: pgmap v12705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23047.047937 secs 2024-04-26T18:15:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:08 smithi033 ceph-mon[27845]: pgmap v12706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23049.918193 secs 2024-04-26T18:15:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23047.047937 secs 2024-04-26T18:15:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:08 smithi096 ceph-mon[32753]: pgmap v12706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23049.918193 secs 2024-04-26T18:15:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23047.047937 secs 2024-04-26T18:15:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:08 smithi194 ceph-mon[33017]: pgmap v12706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23049.918193 secs 2024-04-26T18:15:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:10 smithi033 ceph-mon[27845]: pgmap v12707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:10 smithi096 ceph-mon[32753]: pgmap v12707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:10 smithi194 ceph-mon[33017]: pgmap v12707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:12 smithi033 ceph-mon[27845]: pgmap v12708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:12 smithi096 ceph-mon[32753]: pgmap v12708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:12 smithi194 ceph-mon[33017]: pgmap v12708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23052.048024 secs 2024-04-26T18:15:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23054.918330 secs 2024-04-26T18:15:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23052.048024 secs 2024-04-26T18:15:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23054.918330 secs 2024-04-26T18:15:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23052.048024 secs 2024-04-26T18:15:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23054.918330 secs 2024-04-26T18:15:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:14 smithi033 ceph-mon[27845]: pgmap v12709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:14 smithi096 ceph-mon[32753]: pgmap v12709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:14 smithi194 ceph-mon[33017]: pgmap v12709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:17.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:16 smithi033 ceph-mon[27845]: pgmap v12710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:16 smithi096 ceph-mon[32753]: pgmap v12710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:16 smithi194 ceph-mon[33017]: pgmap v12710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23057.048113 secs 2024-04-26T18:15:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:18 smithi096 ceph-mon[32753]: pgmap v12711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23059.918448 secs 2024-04-26T18:15:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23057.048113 secs 2024-04-26T18:15:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:18 smithi194 ceph-mon[33017]: pgmap v12711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23059.918448 secs 2024-04-26T18:15:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23057.048113 secs 2024-04-26T18:15:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:18 smithi033 ceph-mon[27845]: pgmap v12711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23059.918448 secs 2024-04-26T18:15:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:19.757+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:19.757+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:19.757+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:15:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:20 smithi194 ceph-mon[33017]: pgmap v12712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:20 smithi033 ceph-mon[27845]: pgmap v12712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:20 smithi096 ceph-mon[32753]: pgmap v12712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:22 smithi194 ceph-mon[33017]: pgmap v12713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:22 smithi033 ceph-mon[27845]: pgmap v12713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:22 smithi096 ceph-mon[32753]: pgmap v12713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23062.048198 secs 2024-04-26T18:15:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23064.918566 secs 2024-04-26T18:15:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23062.048198 secs 2024-04-26T18:15:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23064.918566 secs 2024-04-26T18:15:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23062.048198 secs 2024-04-26T18:15:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23064.918566 secs 2024-04-26T18:15:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:24 smithi194 ceph-mon[33017]: pgmap v12714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:24 smithi033 ceph-mon[27845]: pgmap v12714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:24 smithi096 ceph-mon[32753]: pgmap v12714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:26.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:25 smithi096 ceph-mon[32753]: pgmap v12715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:25 smithi194 ceph-mon[33017]: pgmap v12715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:25 smithi033 ceph-mon[27845]: pgmap v12715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23067.048282 secs 2024-04-26T18:15:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:28 smithi033 ceph-mon[27845]: pgmap v12716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23069.918708 secs 2024-04-26T18:15:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23067.048282 secs 2024-04-26T18:15:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:28 smithi096 ceph-mon[32753]: pgmap v12716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23069.918708 secs 2024-04-26T18:15:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23067.048282 secs 2024-04-26T18:15:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:28 smithi194 ceph-mon[33017]: pgmap v12716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23069.918708 secs 2024-04-26T18:15:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:30 smithi033 ceph-mon[27845]: pgmap v12717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:15:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:30 smithi096 ceph-mon[32753]: pgmap v12717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:15:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:30 smithi194 ceph-mon[33017]: pgmap v12717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:15:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:15:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:15:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:15:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:15:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:15:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:15:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:15:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:15:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:15:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:15:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:32 smithi033 ceph-mon[27845]: pgmap v12718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:32 smithi033 ceph-mon[27845]: pgmap v12719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:32 smithi096 ceph-mon[32753]: pgmap v12718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:32 smithi096 ceph-mon[32753]: pgmap v12719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:32 smithi194 ceph-mon[33017]: pgmap v12718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:32 smithi194 ceph-mon[33017]: pgmap v12719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23072.048364 secs 2024-04-26T18:15:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23074.918825 secs 2024-04-26T18:15:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23072.048364 secs 2024-04-26T18:15:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23074.918825 secs 2024-04-26T18:15:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23072.048364 secs 2024-04-26T18:15:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23074.918825 secs 2024-04-26T18:15:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:34.773+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:34.773+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:34.773+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:15:35.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:34 smithi033 ceph-mon[27845]: pgmap v12720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:34 smithi194 ceph-mon[33017]: pgmap v12720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:34 smithi096 ceph-mon[32753]: pgmap v12720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:36 smithi194 ceph-mon[33017]: pgmap v12721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:36 smithi033 ceph-mon[27845]: pgmap v12721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:36 smithi096 ceph-mon[32753]: pgmap v12721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:38 smithi194 ceph-mon[33017]: pgmap v12722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23077.048426 secs 2024-04-26T18:15:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23079.918924 secs 2024-04-26T18:15:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:38 smithi033 ceph-mon[27845]: pgmap v12722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23077.048426 secs 2024-04-26T18:15:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23079.918924 secs 2024-04-26T18:15:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:38 smithi096 ceph-mon[32753]: pgmap v12722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23077.048426 secs 2024-04-26T18:15:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23079.918924 secs 2024-04-26T18:15:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:40 smithi194 ceph-mon[33017]: pgmap v12723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:40 smithi033 ceph-mon[27845]: pgmap v12723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:40 smithi096 ceph-mon[32753]: pgmap v12723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:15:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:42 smithi194 ceph-mon[33017]: pgmap v12724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:15:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:42 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23082.048509 secs 2024-04-26T18:15:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:42 smithi033 ceph-mon[27845]: pgmap v12724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:15:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:42 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23082.048509 secs 2024-04-26T18:15:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:42 smithi096 ceph-mon[32753]: pgmap v12724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:15:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:42 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23082.048509 secs 2024-04-26T18:15:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23084.919058 secs 2024-04-26T18:15:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:43 smithi194 ceph-mon[33017]: pgmap v12725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23084.919058 secs 2024-04-26T18:15:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:43 smithi033 ceph-mon[27845]: pgmap v12725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23084.919058 secs 2024-04-26T18:15:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:43 smithi096 ceph-mon[32753]: pgmap v12725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:46 smithi033 ceph-mon[27845]: pgmap v12726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:46 smithi096 ceph-mon[32753]: pgmap v12726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:46 smithi194 ceph-mon[33017]: pgmap v12726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:48 smithi033 ceph-mon[27845]: pgmap v12727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23087.048594 secs 2024-04-26T18:15:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23089.919173 secs 2024-04-26T18:15:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:48 smithi096 ceph-mon[32753]: pgmap v12727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23087.048594 secs 2024-04-26T18:15:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23089.919173 secs 2024-04-26T18:15:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:48 smithi194 ceph-mon[33017]: pgmap v12727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23087.048594 secs 2024-04-26T18:15:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23089.919173 secs 2024-04-26T18:15:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:49.787+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:49.787+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:15:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:15:49.787+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:15:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:50 smithi033 ceph-mon[27845]: pgmap v12728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:50 smithi096 ceph-mon[32753]: pgmap v12728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:50 smithi194 ceph-mon[33017]: pgmap v12728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:52 smithi033 ceph-mon[27845]: pgmap v12729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:52 smithi096 ceph-mon[32753]: pgmap v12729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:52 smithi194 ceph-mon[33017]: pgmap v12729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23092.048677 secs 2024-04-26T18:15:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23094.919291 secs 2024-04-26T18:15:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23092.048677 secs 2024-04-26T18:15:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23094.919291 secs 2024-04-26T18:15:54.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23092.048677 secs 2024-04-26T18:15:54.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23094.919291 secs 2024-04-26T18:15:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:54 smithi033 ceph-mon[27845]: pgmap v12730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:54 smithi096 ceph-mon[32753]: pgmap v12730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:54 smithi194 ceph-mon[33017]: pgmap v12730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:15:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:56 smithi033 ceph-mon[27845]: pgmap v12731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:56 smithi096 ceph-mon[32753]: pgmap v12731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:56 smithi194 ceph-mon[33017]: pgmap v12731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:15:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:58 smithi033 ceph-mon[27845]: pgmap v12732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23097.048766 secs 2024-04-26T18:15:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:58 smithi033 ceph-mon[27845]: 2 slow requests, 1 included below; oldest blocked for > 23099.919409 secs 2024-04-26T18:15:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:15:58 smithi033 ceph-mon[27845]: slow request 23041.647198 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T18:15:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:58 smithi096 ceph-mon[32753]: pgmap v12732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23097.048766 secs 2024-04-26T18:15:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:58 smithi096 ceph-mon[32753]: 2 slow requests, 1 included below; oldest blocked for > 23099.919409 secs 2024-04-26T18:15:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:15:58 smithi096 ceph-mon[32753]: slow request 23041.647198 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T18:15:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:58 smithi194 ceph-mon[33017]: pgmap v12732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:15:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23097.048766 secs 2024-04-26T18:15:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:58 smithi194 ceph-mon[33017]: 2 slow requests, 1 included below; oldest blocked for > 23099.919409 secs 2024-04-26T18:15:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:15:58 smithi194 ceph-mon[33017]: slow request 23041.647198 seconds old, received at 2024-04-26T11:51:56.840074+0000: client_request(client.14952:4138 open #0x10000000006/postmaster.pid 2024-04-26T11:51:56.838653+0000 caller_uid=26, caller_gid=26{26,}) currently failed to acquire snap lock 2024-04-26T18:16:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:00 smithi033 ceph-mon[27845]: pgmap v12733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:00 smithi096 ceph-mon[32753]: pgmap v12733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:00 smithi194 ceph-mon[33017]: pgmap v12733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:02 smithi033 ceph-mon[27845]: pgmap v12734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:02 smithi096 ceph-mon[32753]: pgmap v12734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:02 smithi194 ceph-mon[33017]: pgmap v12734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23102.048850 secs 2024-04-26T18:16:03.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23104.919526 secs 2024-04-26T18:16:03.945 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23102.048850 secs 2024-04-26T18:16:03.945 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23104.919526 secs 2024-04-26T18:16:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23102.048850 secs 2024-04-26T18:16:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23104.919526 secs 2024-04-26T18:16:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:04.804+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:04.804+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:04.804+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:16:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:04 smithi096 ceph-mon[32753]: pgmap v12735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:04 smithi194 ceph-mon[33017]: pgmap v12735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:04 smithi033 ceph-mon[27845]: pgmap v12735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:06 smithi096 ceph-mon[32753]: pgmap v12736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:07.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:06 smithi194 ceph-mon[33017]: pgmap v12736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:06 smithi033 ceph-mon[27845]: pgmap v12736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:08 smithi194 ceph-mon[33017]: pgmap v12737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23107.048935 secs 2024-04-26T18:16:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23109.919676 secs 2024-04-26T18:16:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:08 smithi033 ceph-mon[27845]: pgmap v12737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23107.048935 secs 2024-04-26T18:16:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23109.919676 secs 2024-04-26T18:16:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:08 smithi096 ceph-mon[32753]: pgmap v12737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23107.048935 secs 2024-04-26T18:16:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23109.919676 secs 2024-04-26T18:16:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:10 smithi194 ceph-mon[33017]: pgmap v12738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:10 smithi033 ceph-mon[27845]: pgmap v12738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:10 smithi096 ceph-mon[32753]: pgmap v12738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:12 smithi194 ceph-mon[33017]: pgmap v12739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:12 smithi033 ceph-mon[27845]: pgmap v12739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:12 smithi096 ceph-mon[32753]: pgmap v12739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23112.049017 secs 2024-04-26T18:16:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23114.919792 secs 2024-04-26T18:16:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23112.049017 secs 2024-04-26T18:16:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23114.919792 secs 2024-04-26T18:16:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23112.049017 secs 2024-04-26T18:16:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23114.919792 secs 2024-04-26T18:16:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:14 smithi194 ceph-mon[33017]: pgmap v12740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:14 smithi033 ceph-mon[27845]: pgmap v12740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:14 smithi096 ceph-mon[32753]: pgmap v12740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:16 smithi194 ceph-mon[33017]: pgmap v12741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:16 smithi033 ceph-mon[27845]: pgmap v12741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:16 smithi096 ceph-mon[32753]: pgmap v12741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:18 smithi194 ceph-mon[33017]: pgmap v12742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23117.049105 secs 2024-04-26T18:16:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23119.919910 secs 2024-04-26T18:16:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:18 smithi033 ceph-mon[27845]: pgmap v12742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23117.049105 secs 2024-04-26T18:16:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23119.919910 secs 2024-04-26T18:16:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:18 smithi096 ceph-mon[32753]: pgmap v12742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23117.049105 secs 2024-04-26T18:16:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23119.919910 secs 2024-04-26T18:16:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:19.816+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:19.816+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:19.816+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:16:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:20 smithi194 ceph-mon[33017]: pgmap v12743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:20 smithi033 ceph-mon[27845]: pgmap v12743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:20 smithi096 ceph-mon[32753]: pgmap v12743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:22 smithi194 ceph-mon[33017]: pgmap v12744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:22 smithi033 ceph-mon[27845]: pgmap v12744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:22 smithi096 ceph-mon[32753]: pgmap v12744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:24.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23122.049190 secs 2024-04-26T18:16:24.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23124.919973 secs 2024-04-26T18:16:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23122.049190 secs 2024-04-26T18:16:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23124.919973 secs 2024-04-26T18:16:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23122.049190 secs 2024-04-26T18:16:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23124.919973 secs 2024-04-26T18:16:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:24 smithi194 ceph-mon[33017]: pgmap v12745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:24 smithi033 ceph-mon[27845]: pgmap v12745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:24 smithi096 ceph-mon[32753]: pgmap v12745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:26 smithi194 ceph-mon[33017]: pgmap v12746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:26 smithi033 ceph-mon[27845]: pgmap v12746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:26 smithi096 ceph-mon[32753]: pgmap v12746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:28 smithi194 ceph-mon[33017]: pgmap v12747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23127.049276 secs 2024-04-26T18:16:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23129.920130 secs 2024-04-26T18:16:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:28 smithi033 ceph-mon[27845]: pgmap v12747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23127.049276 secs 2024-04-26T18:16:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23129.920130 secs 2024-04-26T18:16:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:28 smithi096 ceph-mon[32753]: pgmap v12747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23127.049276 secs 2024-04-26T18:16:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23129.920130 secs 2024-04-26T18:16:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:30 smithi194 ceph-mon[33017]: pgmap v12748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:30 smithi033 ceph-mon[27845]: pgmap v12748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:30 smithi096 ceph-mon[32753]: pgmap v12748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:32 smithi194 ceph-mon[33017]: pgmap v12749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:16:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:32 smithi033 ceph-mon[27845]: pgmap v12749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:16:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:32 smithi096 ceph-mon[32753]: pgmap v12749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:16:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23132.049368 secs 2024-04-26T18:16:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23134.920253 secs 2024-04-26T18:16:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23132.049368 secs 2024-04-26T18:16:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23134.920253 secs 2024-04-26T18:16:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23132.049368 secs 2024-04-26T18:16:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23134.920253 secs 2024-04-26T18:16:34.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:34.822+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:34.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:34.822+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:34.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:34.822+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:16:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:34 smithi194 ceph-mon[33017]: pgmap v12750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:34 smithi033 ceph-mon[27845]: pgmap v12750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:34 smithi096 ceph-mon[32753]: pgmap v12750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:36.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:37.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:36 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:16:37.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:36 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:16:37.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:36 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:16:37.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:36 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:16:37.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:36 smithi033 ceph-mon[27845]: pgmap v12751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:37.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:37.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:36 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:16:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:36 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:16:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:36 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:16:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:36 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:16:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:36 smithi194 ceph-mon[33017]: pgmap v12751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:36 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:36 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:36 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:36 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:36 smithi096 ceph-mon[32753]: pgmap v12751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: pgmap v12752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23137.049452 secs 2024-04-26T18:16:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23139.920375 secs 2024-04-26T18:16:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: pgmap v12752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23137.049452 secs 2024-04-26T18:16:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:39.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23139.920375 secs 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: pgmap v12752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23137.049452 secs 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:16:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:16:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:16:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:16:39.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23139.920375 secs 2024-04-26T18:16:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:39 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:16:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:39 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:16:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:39 smithi194 ceph-mon[33017]: pgmap v12753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:16:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:39 smithi194 ceph-mon[33017]: pgmap v12754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:16:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:39 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:16:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:39 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:16:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:39 smithi033 ceph-mon[27845]: pgmap v12753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:16:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:39 smithi033 ceph-mon[27845]: pgmap v12754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:16:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:39 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:16:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:39 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:16:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:39 smithi096 ceph-mon[32753]: pgmap v12753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:16:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:39 smithi096 ceph-mon[32753]: pgmap v12754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:16:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:41 smithi194 ceph-mon[33017]: pgmap v12755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:41 smithi033 ceph-mon[27845]: pgmap v12755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:41 smithi096 ceph-mon[32753]: pgmap v12755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:43 smithi194 ceph-mon[33017]: pgmap v12756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23142.049531 secs 2024-04-26T18:16:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23144.920490 secs 2024-04-26T18:16:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:43 smithi033 ceph-mon[27845]: pgmap v12756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23142.049531 secs 2024-04-26T18:16:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23144.920490 secs 2024-04-26T18:16:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:43 smithi096 ceph-mon[32753]: pgmap v12756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23142.049531 secs 2024-04-26T18:16:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23144.920490 secs 2024-04-26T18:16:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:45 smithi194 ceph-mon[33017]: pgmap v12757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:16:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:45 smithi033 ceph-mon[27845]: pgmap v12757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:16:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:45 smithi096 ceph-mon[32753]: pgmap v12757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:16:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:47 smithi194 ceph-mon[33017]: pgmap v12758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:47 smithi033 ceph-mon[27845]: pgmap v12758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:47 smithi096 ceph-mon[32753]: pgmap v12758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:16:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23147.049614 secs 2024-04-26T18:16:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23149.920613 secs 2024-04-26T18:16:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23147.049614 secs 2024-04-26T18:16:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23149.920613 secs 2024-04-26T18:16:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23147.049614 secs 2024-04-26T18:16:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23149.920613 secs 2024-04-26T18:16:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:49.837+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:49.837+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:16:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:16:49.837+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:16:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:49 smithi194 ceph-mon[33017]: pgmap v12759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:16:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:49 smithi033 ceph-mon[27845]: pgmap v12759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:16:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:49 smithi096 ceph-mon[32753]: pgmap v12759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:16:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:51 smithi194 ceph-mon[33017]: pgmap v12760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:51 smithi033 ceph-mon[27845]: pgmap v12760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:51 smithi096 ceph-mon[32753]: pgmap v12760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:53 smithi033 ceph-mon[27845]: pgmap v12761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23152.049673 secs 2024-04-26T18:16:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23154.920752 secs 2024-04-26T18:16:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:53 smithi096 ceph-mon[32753]: pgmap v12761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23152.049673 secs 2024-04-26T18:16:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23154.920752 secs 2024-04-26T18:16:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:53 smithi194 ceph-mon[33017]: pgmap v12761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23152.049673 secs 2024-04-26T18:16:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23154.920752 secs 2024-04-26T18:16:56.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:55 smithi194 ceph-mon[33017]: pgmap v12762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:55 smithi033 ceph-mon[27845]: pgmap v12762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:55 smithi096 ceph-mon[32753]: pgmap v12762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:16:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:57 smithi033 ceph-mon[27845]: pgmap v12763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:57 smithi096 ceph-mon[32753]: pgmap v12763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:57 smithi194 ceph-mon[33017]: pgmap v12763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:16:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23157.049734 secs 2024-04-26T18:16:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23159.920873 secs 2024-04-26T18:16:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23157.049734 secs 2024-04-26T18:16:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23159.920873 secs 2024-04-26T18:16:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23157.049734 secs 2024-04-26T18:16:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23159.920873 secs 2024-04-26T18:17:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:16:59 smithi033 ceph-mon[27845]: pgmap v12764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:16:59 smithi096 ceph-mon[32753]: pgmap v12764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:16:59 smithi194 ceph-mon[33017]: pgmap v12764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:01 smithi033 ceph-mon[27845]: pgmap v12765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:01 smithi096 ceph-mon[32753]: pgmap v12765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:01 smithi194 ceph-mon[33017]: pgmap v12765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:03 smithi033 ceph-mon[27845]: pgmap v12766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23162.049817 secs 2024-04-26T18:17:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23164.921021 secs 2024-04-26T18:17:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:03 smithi096 ceph-mon[32753]: pgmap v12766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23162.049817 secs 2024-04-26T18:17:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23164.921021 secs 2024-04-26T18:17:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:03 smithi194 ceph-mon[33017]: pgmap v12766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23162.049817 secs 2024-04-26T18:17:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23164.921021 secs 2024-04-26T18:17:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:04.853+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:04.853+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:04.853+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:17:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:05 smithi033 ceph-mon[27845]: pgmap v12767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:05 smithi096 ceph-mon[32753]: pgmap v12767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:05 smithi194 ceph-mon[33017]: pgmap v12767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:07 smithi033 ceph-mon[27845]: pgmap v12768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:07 smithi096 ceph-mon[32753]: pgmap v12768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:07 smithi194 ceph-mon[33017]: pgmap v12768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23167.049900 secs 2024-04-26T18:17:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23169.921168 secs 2024-04-26T18:17:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23167.049900 secs 2024-04-26T18:17:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23169.921168 secs 2024-04-26T18:17:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23167.049900 secs 2024-04-26T18:17:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23169.921168 secs 2024-04-26T18:17:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:09 smithi096 ceph-mon[32753]: pgmap v12769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:09 smithi033 ceph-mon[27845]: pgmap v12769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:09 smithi194 ceph-mon[33017]: pgmap v12769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:11 smithi033 ceph-mon[27845]: pgmap v12770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:11 smithi096 ceph-mon[32753]: pgmap v12770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:11 smithi194 ceph-mon[33017]: pgmap v12770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:13 smithi033 ceph-mon[27845]: pgmap v12771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23172.049980 secs 2024-04-26T18:17:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23174.921304 secs 2024-04-26T18:17:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:13 smithi096 ceph-mon[32753]: pgmap v12771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23172.049980 secs 2024-04-26T18:17:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23174.921304 secs 2024-04-26T18:17:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:13 smithi194 ceph-mon[33017]: pgmap v12771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23172.049980 secs 2024-04-26T18:17:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23174.921304 secs 2024-04-26T18:17:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:15 smithi096 ceph-mon[32753]: pgmap v12772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:15 smithi194 ceph-mon[33017]: pgmap v12772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:15 smithi033 ceph-mon[27845]: pgmap v12772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:17 smithi194 ceph-mon[33017]: pgmap v12773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:17 smithi033 ceph-mon[27845]: pgmap v12773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:17 smithi096 ceph-mon[32753]: pgmap v12773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23177.050037 secs 2024-04-26T18:17:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23179.921423 secs 2024-04-26T18:17:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23177.050037 secs 2024-04-26T18:17:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23179.921423 secs 2024-04-26T18:17:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23177.050037 secs 2024-04-26T18:17:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23179.921423 secs 2024-04-26T18:17:19.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:19.861+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:19.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:19.861+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:19.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:19.861+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:17:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:19 smithi194 ceph-mon[33017]: pgmap v12774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:19 smithi033 ceph-mon[27845]: pgmap v12774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:19 smithi096 ceph-mon[32753]: pgmap v12774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:21 smithi194 ceph-mon[33017]: pgmap v12775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:21 smithi033 ceph-mon[27845]: pgmap v12775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:21 smithi096 ceph-mon[32753]: pgmap v12775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:23 smithi194 ceph-mon[33017]: pgmap v12776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23182.050126 secs 2024-04-26T18:17:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23184.921541 secs 2024-04-26T18:17:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:23 smithi033 ceph-mon[27845]: pgmap v12776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23182.050126 secs 2024-04-26T18:17:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23184.921541 secs 2024-04-26T18:17:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:23 smithi096 ceph-mon[32753]: pgmap v12776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23182.050126 secs 2024-04-26T18:17:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23184.921541 secs 2024-04-26T18:17:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.742+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:17:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.742+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:17:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.742+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 184 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T18:17:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.743+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x564385821900 2024-04-26T18:17:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.743+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 184 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x7fef00077780 2024-04-26T18:17:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.743+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x7fef00077780 2024-04-26T18:17:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.743+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 185 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T18:17:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:25.743+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 185 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:17:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:25 smithi194 ceph-mon[33017]: pgmap v12777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:25 smithi033 ceph-mon[27845]: pgmap v12777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:25 smithi096 ceph-mon[32753]: pgmap v12777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:28.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:27 smithi096 ceph-mon[32753]: pgmap v12778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:28.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23187.050211 secs 2024-04-26T18:17:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:27 smithi194 ceph-mon[33017]: pgmap v12778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23187.050211 secs 2024-04-26T18:17:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:27 smithi033 ceph-mon[27845]: pgmap v12778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23187.050211 secs 2024-04-26T18:17:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23189.921661 secs 2024-04-26T18:17:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23189.921661 secs 2024-04-26T18:17:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23189.921661 secs 2024-04-26T18:17:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:29 smithi194 ceph-mon[33017]: pgmap v12779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:29 smithi033 ceph-mon[27845]: pgmap v12779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:29 smithi096 ceph-mon[32753]: pgmap v12779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:30 smithi194 ceph-mon[33017]: pgmap v12780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:30 smithi033 ceph-mon[27845]: pgmap v12780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:30 smithi096 ceph-mon[32753]: pgmap v12780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:33 smithi033 ceph-mon[27845]: pgmap v12781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23192.050270 secs 2024-04-26T18:17:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:33 smithi096 ceph-mon[32753]: pgmap v12781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23192.050270 secs 2024-04-26T18:17:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:33 smithi194 ceph-mon[33017]: pgmap v12781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23192.050270 secs 2024-04-26T18:17:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23194.921734 secs 2024-04-26T18:17:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23194.921734 secs 2024-04-26T18:17:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23194.921734 secs 2024-04-26T18:17:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:34.865+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:34.865+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:34.865+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:17:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:35 smithi033 ceph-mon[27845]: pgmap v12782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:35 smithi096 ceph-mon[32753]: pgmap v12782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:35 smithi194 ceph-mon[33017]: pgmap v12782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:37 smithi033 ceph-mon[27845]: pgmap v12783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:37 smithi096 ceph-mon[32753]: pgmap v12783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:37 smithi194 ceph-mon[33017]: pgmap v12783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23197.050355 secs 2024-04-26T18:17:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23197.050355 secs 2024-04-26T18:17:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23197.050355 secs 2024-04-26T18:17:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:39 smithi033 ceph-mon[27845]: pgmap v12784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:17:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23199.921813 secs 2024-04-26T18:17:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:39 smithi096 ceph-mon[32753]: pgmap v12784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:17:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23199.921813 secs 2024-04-26T18:17:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:39 smithi194 ceph-mon[33017]: pgmap v12784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:17:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23199.921813 secs 2024-04-26T18:17:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:17:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:17:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:40 smithi033 ceph-mon[27845]: pgmap v12785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:17:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:40 smithi033 ceph-mon[27845]: pgmap v12786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:17:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:17:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:17:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:40 smithi096 ceph-mon[32753]: pgmap v12785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:17:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:40 smithi096 ceph-mon[32753]: pgmap v12786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:17:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:17:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:17:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:40 smithi194 ceph-mon[33017]: pgmap v12785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:17:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:40 smithi194 ceph-mon[33017]: pgmap v12786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:17:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:42 smithi194 ceph-mon[33017]: pgmap v12787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:42 smithi033 ceph-mon[27845]: pgmap v12787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:42 smithi096 ceph-mon[32753]: pgmap v12787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23202.050437 secs 2024-04-26T18:17:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23204.921928 secs 2024-04-26T18:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23202.050437 secs 2024-04-26T18:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23204.921928 secs 2024-04-26T18:17:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23202.050437 secs 2024-04-26T18:17:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23204.921928 secs 2024-04-26T18:17:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:44 smithi194 ceph-mon[33017]: pgmap v12788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:44 smithi033 ceph-mon[27845]: pgmap v12788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:44 smithi096 ceph-mon[32753]: pgmap v12788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:46 smithi194 ceph-mon[33017]: pgmap v12789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:46 smithi033 ceph-mon[27845]: pgmap v12789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:46 smithi096 ceph-mon[32753]: pgmap v12789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:48 smithi194 ceph-mon[33017]: pgmap v12790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23207.050519 secs 2024-04-26T18:17:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23209.921993 secs 2024-04-26T18:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:48 smithi033 ceph-mon[27845]: pgmap v12790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23207.050519 secs 2024-04-26T18:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23209.921993 secs 2024-04-26T18:17:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:48 smithi096 ceph-mon[32753]: pgmap v12790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:17:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23207.050519 secs 2024-04-26T18:17:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23209.921993 secs 2024-04-26T18:17:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:49.879+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:49.879+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:17:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:17:49.879+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:17:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:50 smithi033 ceph-mon[27845]: pgmap v12791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:17:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:50 smithi096 ceph-mon[32753]: pgmap v12791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:17:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:50 smithi194 ceph-mon[33017]: pgmap v12791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:17:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:52 smithi033 ceph-mon[27845]: pgmap v12792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:52 smithi096 ceph-mon[32753]: pgmap v12792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:52 smithi194 ceph-mon[33017]: pgmap v12792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:53.872 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23212.050602 secs 2024-04-26T18:17:53.872 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23214.922133 secs 2024-04-26T18:17:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23212.050602 secs 2024-04-26T18:17:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23214.922133 secs 2024-04-26T18:17:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23212.050602 secs 2024-04-26T18:17:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23214.922133 secs 2024-04-26T18:17:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:54 smithi033 ceph-mon[27845]: pgmap v12793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:54 smithi096 ceph-mon[32753]: pgmap v12793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:54 smithi194 ceph-mon[33017]: pgmap v12793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:57.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:56 smithi194 ceph-mon[33017]: pgmap v12794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:56 smithi033 ceph-mon[27845]: pgmap v12794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:56 smithi096 ceph-mon[32753]: pgmap v12794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:58 smithi033 ceph-mon[27845]: pgmap v12795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23217.050687 secs 2024-04-26T18:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:17:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23219.922171 secs 2024-04-26T18:17:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:58 smithi096 ceph-mon[32753]: pgmap v12795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23217.050687 secs 2024-04-26T18:17:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:17:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23219.922171 secs 2024-04-26T18:17:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:58 smithi194 ceph-mon[33017]: pgmap v12795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:17:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23217.050687 secs 2024-04-26T18:17:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:17:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23219.922171 secs 2024-04-26T18:18:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:00 smithi033 ceph-mon[27845]: pgmap v12796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:00 smithi096 ceph-mon[32753]: pgmap v12796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:00 smithi194 ceph-mon[33017]: pgmap v12796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:02 smithi033 ceph-mon[27845]: pgmap v12797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:02 smithi096 ceph-mon[32753]: pgmap v12797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:02 smithi194 ceph-mon[33017]: pgmap v12797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23222.050768 secs 2024-04-26T18:18:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23224.922243 secs 2024-04-26T18:18:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23222.050768 secs 2024-04-26T18:18:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23224.922243 secs 2024-04-26T18:18:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23222.050768 secs 2024-04-26T18:18:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23224.922243 secs 2024-04-26T18:18:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:04.884+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:04.884+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:04.884+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:18:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:04 smithi033 ceph-mon[27845]: pgmap v12798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:04 smithi096 ceph-mon[32753]: pgmap v12798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:04 smithi194 ceph-mon[33017]: pgmap v12798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:06 smithi033 ceph-mon[27845]: pgmap v12799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:06 smithi096 ceph-mon[32753]: pgmap v12799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:06 smithi194 ceph-mon[33017]: pgmap v12799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:08 smithi033 ceph-mon[27845]: pgmap v12800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23227.050857 secs 2024-04-26T18:18:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23229.922365 secs 2024-04-26T18:18:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:08 smithi096 ceph-mon[32753]: pgmap v12800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23227.050857 secs 2024-04-26T18:18:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23229.922365 secs 2024-04-26T18:18:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:08 smithi194 ceph-mon[33017]: pgmap v12800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23227.050857 secs 2024-04-26T18:18:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23229.922365 secs 2024-04-26T18:18:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:10 smithi033 ceph-mon[27845]: pgmap v12801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:10 smithi096 ceph-mon[32753]: pgmap v12801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:10 smithi194 ceph-mon[33017]: pgmap v12801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:12 smithi096 ceph-mon[32753]: pgmap v12802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:12 smithi194 ceph-mon[33017]: pgmap v12802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:12 smithi033 ceph-mon[27845]: pgmap v12802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23232.050936 secs 2024-04-26T18:18:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23234.922459 secs 2024-04-26T18:18:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23232.050936 secs 2024-04-26T18:18:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23234.922459 secs 2024-04-26T18:18:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23232.050936 secs 2024-04-26T18:18:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23234.922459 secs 2024-04-26T18:18:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:14 smithi194 ceph-mon[33017]: pgmap v12803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:14 smithi033 ceph-mon[27845]: pgmap v12803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:14 smithi096 ceph-mon[32753]: pgmap v12803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:16 smithi194 ceph-mon[33017]: pgmap v12804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:16 smithi033 ceph-mon[27845]: pgmap v12804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:16 smithi096 ceph-mon[32753]: pgmap v12804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:18 smithi194 ceph-mon[33017]: pgmap v12805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23237.051025 secs 2024-04-26T18:18:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23239.922538 secs 2024-04-26T18:18:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:18 smithi033 ceph-mon[27845]: pgmap v12805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23237.051025 secs 2024-04-26T18:18:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23239.922538 secs 2024-04-26T18:18:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:18 smithi096 ceph-mon[32753]: pgmap v12805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23237.051025 secs 2024-04-26T18:18:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23239.922538 secs 2024-04-26T18:18:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:19.896+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:19.896+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:19.896+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:18:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:20 smithi194 ceph-mon[33017]: pgmap v12806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:20 smithi033 ceph-mon[27845]: pgmap v12806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:20 smithi096 ceph-mon[32753]: pgmap v12806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:22 smithi194 ceph-mon[33017]: pgmap v12807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:22 smithi033 ceph-mon[27845]: pgmap v12807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:22 smithi096 ceph-mon[32753]: pgmap v12807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23242.051109 secs 2024-04-26T18:18:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23244.922658 secs 2024-04-26T18:18:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23242.051109 secs 2024-04-26T18:18:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23244.922658 secs 2024-04-26T18:18:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23242.051109 secs 2024-04-26T18:18:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23244.922658 secs 2024-04-26T18:18:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:24 smithi194 ceph-mon[33017]: pgmap v12808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:24 smithi033 ceph-mon[27845]: pgmap v12808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:24 smithi096 ceph-mon[32753]: pgmap v12808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:26 smithi194 ceph-mon[33017]: pgmap v12809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:26 smithi033 ceph-mon[27845]: pgmap v12809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:26 smithi096 ceph-mon[32753]: pgmap v12809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:28.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:27 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23247.051196 secs 2024-04-26T18:18:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:27 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23247.051196 secs 2024-04-26T18:18:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:27 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23247.051196 secs 2024-04-26T18:18:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:28 smithi194 ceph-mon[33017]: pgmap v12810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23249.922773 secs 2024-04-26T18:18:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:28 smithi033 ceph-mon[27845]: pgmap v12810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23249.922773 secs 2024-04-26T18:18:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:28 smithi096 ceph-mon[32753]: pgmap v12810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23249.922773 secs 2024-04-26T18:18:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:30 smithi194 ceph-mon[33017]: pgmap v12811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:30 smithi033 ceph-mon[27845]: pgmap v12811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:30 smithi096 ceph-mon[32753]: pgmap v12811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:18:33.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:32 smithi194 ceph-mon[33017]: pgmap v12812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:32 smithi033 ceph-mon[27845]: pgmap v12812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:32 smithi096 ceph-mon[32753]: pgmap v12812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23252.051278 secs 2024-04-26T18:18:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23254.922871 secs 2024-04-26T18:18:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23252.051278 secs 2024-04-26T18:18:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23254.922871 secs 2024-04-26T18:18:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23252.051278 secs 2024-04-26T18:18:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23254.922871 secs 2024-04-26T18:18:34.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:34.900+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:34.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:34.900+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:34.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:34.900+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:18:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:34 smithi033 ceph-mon[27845]: pgmap v12813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:34 smithi096 ceph-mon[32753]: pgmap v12813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:34 smithi194 ceph-mon[33017]: pgmap v12813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:36 smithi033 ceph-mon[27845]: pgmap v12814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:36 smithi096 ceph-mon[32753]: pgmap v12814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:36 smithi194 ceph-mon[33017]: pgmap v12814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:38 smithi033 ceph-mon[27845]: pgmap v12815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23257.051365 secs 2024-04-26T18:18:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23259.922981 secs 2024-04-26T18:18:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:38 smithi096 ceph-mon[32753]: pgmap v12815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23257.051365 secs 2024-04-26T18:18:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23259.922981 secs 2024-04-26T18:18:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:38 smithi194 ceph-mon[33017]: pgmap v12815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23257.051365 secs 2024-04-26T18:18:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23259.922981 secs 2024-04-26T18:18:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:40 smithi033 ceph-mon[27845]: pgmap v12816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:18:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:40 smithi096 ceph-mon[32753]: pgmap v12816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:18:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:40 smithi194 ceph-mon[33017]: pgmap v12816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:18:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:18:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:18:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:18:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:18:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:18:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:18:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:18:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:18:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:18:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:42 smithi033 ceph-mon[27845]: pgmap v12817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:42 smithi033 ceph-mon[27845]: pgmap v12818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:42 smithi096 ceph-mon[32753]: pgmap v12817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:42 smithi096 ceph-mon[32753]: pgmap v12818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:42 smithi194 ceph-mon[33017]: pgmap v12817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:42 smithi194 ceph-mon[33017]: pgmap v12818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23262.051446 secs 2024-04-26T18:18:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23264.923097 secs 2024-04-26T18:18:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23262.051446 secs 2024-04-26T18:18:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23264.923097 secs 2024-04-26T18:18:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23262.051446 secs 2024-04-26T18:18:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23264.923097 secs 2024-04-26T18:18:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:44 smithi033 ceph-mon[27845]: pgmap v12819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:44 smithi096 ceph-mon[32753]: pgmap v12819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:44 smithi194 ceph-mon[33017]: pgmap v12819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:46 smithi033 ceph-mon[27845]: pgmap v12820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:46 smithi096 ceph-mon[32753]: pgmap v12820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:46 smithi194 ceph-mon[33017]: pgmap v12820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:48 smithi033 ceph-mon[27845]: pgmap v12821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23267.051531 secs 2024-04-26T18:18:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23269.923233 secs 2024-04-26T18:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:48 smithi096 ceph-mon[32753]: pgmap v12821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23267.051531 secs 2024-04-26T18:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23269.923233 secs 2024-04-26T18:18:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:48 smithi194 ceph-mon[33017]: pgmap v12821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:18:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23267.051531 secs 2024-04-26T18:18:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23269.923233 secs 2024-04-26T18:18:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:49.913+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:49.913+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:18:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:18:49.913+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:18:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:50 smithi033 ceph-mon[27845]: pgmap v12822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:50 smithi096 ceph-mon[32753]: pgmap v12822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:50 smithi194 ceph-mon[33017]: pgmap v12822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:18:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:52 smithi096 ceph-mon[32753]: pgmap v12823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:18:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:52 smithi194 ceph-mon[33017]: pgmap v12823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:18:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:52 smithi033 ceph-mon[27845]: pgmap v12823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:18:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23272.051614 secs 2024-04-26T18:18:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23274.923344 secs 2024-04-26T18:18:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23272.051614 secs 2024-04-26T18:18:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23274.923344 secs 2024-04-26T18:18:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23272.051614 secs 2024-04-26T18:18:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23274.923344 secs 2024-04-26T18:18:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:54 smithi194 ceph-mon[33017]: pgmap v12824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:54 smithi033 ceph-mon[27845]: pgmap v12824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:54 smithi096 ceph-mon[32753]: pgmap v12824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:57.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:56 smithi033 ceph-mon[27845]: pgmap v12825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:56 smithi194 ceph-mon[33017]: pgmap v12825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:56 smithi096 ceph-mon[32753]: pgmap v12825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:18:59.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:58 smithi096 ceph-mon[32753]: pgmap v12826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:59.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23277.051698 secs 2024-04-26T18:18:59.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:18:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23279.923440 secs 2024-04-26T18:18:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:58 smithi194 ceph-mon[33017]: pgmap v12826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:59.520 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23277.051698 secs 2024-04-26T18:18:59.520 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:18:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23279.923440 secs 2024-04-26T18:18:59.520 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:58 smithi033 ceph-mon[27845]: pgmap v12826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:18:59.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23277.051698 secs 2024-04-26T18:18:59.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:18:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23279.923440 secs 2024-04-26T18:19:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:00 smithi194 ceph-mon[33017]: pgmap v12827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:00 smithi033 ceph-mon[27845]: pgmap v12827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:00 smithi096 ceph-mon[32753]: pgmap v12827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:02 smithi194 ceph-mon[33017]: pgmap v12828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:02 smithi033 ceph-mon[27845]: pgmap v12828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:02 smithi096 ceph-mon[32753]: pgmap v12828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23282.051802 secs 2024-04-26T18:19:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23284.923581 secs 2024-04-26T18:19:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23282.051802 secs 2024-04-26T18:19:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23284.923581 secs 2024-04-26T18:19:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23282.051802 secs 2024-04-26T18:19:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23284.923581 secs 2024-04-26T18:19:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:04.927+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:04.927+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:04.927+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:19:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:04 smithi194 ceph-mon[33017]: pgmap v12829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:04 smithi033 ceph-mon[27845]: pgmap v12829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:04 smithi096 ceph-mon[32753]: pgmap v12829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:06 smithi194 ceph-mon[33017]: pgmap v12830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:06 smithi033 ceph-mon[27845]: pgmap v12830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:06 smithi096 ceph-mon[32753]: pgmap v12830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:08.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:07 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23287.051887 secs 2024-04-26T18:19:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:07 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23287.051887 secs 2024-04-26T18:19:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:07 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23287.051887 secs 2024-04-26T18:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:08 smithi194 ceph-mon[33017]: pgmap v12831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23289.923683 secs 2024-04-26T18:19:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:08 smithi033 ceph-mon[27845]: pgmap v12831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23289.923683 secs 2024-04-26T18:19:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:08 smithi096 ceph-mon[32753]: pgmap v12831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23289.923683 secs 2024-04-26T18:19:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:10 smithi194 ceph-mon[33017]: pgmap v12832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:10 smithi033 ceph-mon[27845]: pgmap v12832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:10 smithi096 ceph-mon[32753]: pgmap v12832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:12 smithi194 ceph-mon[33017]: pgmap v12833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:12 smithi033 ceph-mon[27845]: pgmap v12833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:12 smithi096 ceph-mon[32753]: pgmap v12833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:13.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23292.051992 secs 2024-04-26T18:19:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23292.051992 secs 2024-04-26T18:19:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23292.051992 secs 2024-04-26T18:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:14 smithi194 ceph-mon[33017]: pgmap v12834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23294.923818 secs 2024-04-26T18:19:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:14 smithi033 ceph-mon[27845]: pgmap v12834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23294.923818 secs 2024-04-26T18:19:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:14 smithi096 ceph-mon[32753]: pgmap v12834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23294.923818 secs 2024-04-26T18:19:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:16 smithi194 ceph-mon[33017]: pgmap v12835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:16 smithi033 ceph-mon[27845]: pgmap v12835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:16 smithi096 ceph-mon[32753]: pgmap v12835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:18 smithi194 ceph-mon[33017]: pgmap v12836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23297.052055 secs 2024-04-26T18:19:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:18 smithi033 ceph-mon[27845]: pgmap v12836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23297.052055 secs 2024-04-26T18:19:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:18 smithi096 ceph-mon[32753]: pgmap v12836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23297.052055 secs 2024-04-26T18:19:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23299.923947 secs 2024-04-26T18:19:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23299.923947 secs 2024-04-26T18:19:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23299.923947 secs 2024-04-26T18:19:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:19.937+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:19.937+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:19.937+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:19:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:20 smithi194 ceph-mon[33017]: pgmap v12837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:20 smithi033 ceph-mon[27845]: pgmap v12837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:20 smithi096 ceph-mon[32753]: pgmap v12837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:22 smithi194 ceph-mon[33017]: pgmap v12838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:22 smithi033 ceph-mon[27845]: pgmap v12838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:22 smithi096 ceph-mon[32753]: pgmap v12838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23302.052140 secs 2024-04-26T18:19:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23302.052140 secs 2024-04-26T18:19:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23302.052140 secs 2024-04-26T18:19:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:24 smithi194 ceph-mon[33017]: pgmap v12839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23304.924044 secs 2024-04-26T18:19:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:24 smithi033 ceph-mon[27845]: pgmap v12839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23304.924044 secs 2024-04-26T18:19:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:24 smithi096 ceph-mon[32753]: pgmap v12839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23304.924044 secs 2024-04-26T18:19:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:26 smithi194 ceph-mon[33017]: pgmap v12840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:26 smithi033 ceph-mon[27845]: pgmap v12840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:26 smithi096 ceph-mon[32753]: pgmap v12840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:28 smithi194 ceph-mon[33017]: pgmap v12841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23307.052229 secs 2024-04-26T18:19:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:28 smithi033 ceph-mon[27845]: pgmap v12841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23307.052229 secs 2024-04-26T18:19:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:28 smithi096 ceph-mon[32753]: pgmap v12841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23307.052229 secs 2024-04-26T18:19:29.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23309.924176 secs 2024-04-26T18:19:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23309.924176 secs 2024-04-26T18:19:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23309.924176 secs 2024-04-26T18:19:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:30 smithi194 ceph-mon[33017]: pgmap v12842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:30 smithi033 ceph-mon[27845]: pgmap v12842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:30 smithi096 ceph-mon[32753]: pgmap v12842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:32 smithi194 ceph-mon[33017]: pgmap v12843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:32 smithi033 ceph-mon[27845]: pgmap v12843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:32 smithi096 ceph-mon[32753]: pgmap v12843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23312.052329 secs 2024-04-26T18:19:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23312.052329 secs 2024-04-26T18:19:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23312.052329 secs 2024-04-26T18:19:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:34 smithi194 ceph-mon[33017]: pgmap v12844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23314.924237 secs 2024-04-26T18:19:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:34 smithi033 ceph-mon[27845]: pgmap v12844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23314.924237 secs 2024-04-26T18:19:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:34 smithi096 ceph-mon[32753]: pgmap v12844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23314.924237 secs 2024-04-26T18:19:34.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:34.951+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:34.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:34.951+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:34.951+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:19:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:36 smithi194 ceph-mon[33017]: pgmap v12845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:36.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:36 smithi033 ceph-mon[27845]: pgmap v12845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:36 smithi096 ceph-mon[32753]: pgmap v12845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:38 smithi194 ceph-mon[33017]: pgmap v12846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23317.052428 secs 2024-04-26T18:19:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:38 smithi033 ceph-mon[27845]: pgmap v12846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23317.052428 secs 2024-04-26T18:19:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:38 smithi096 ceph-mon[32753]: pgmap v12846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23317.052428 secs 2024-04-26T18:19:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23319.924315 secs 2024-04-26T18:19:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23319.924315 secs 2024-04-26T18:19:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23319.924315 secs 2024-04-26T18:19:40.867 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:40 smithi194 ceph-mon[33017]: pgmap v12847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:41.131 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:40 smithi096 ceph-mon[32753]: pgmap v12847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:41.131 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:40 smithi033 ceph-mon[27845]: pgmap v12847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:42 smithi194 ceph-mon[33017]: pgmap v12848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:42.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:19:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:42 smithi033 ceph-mon[27845]: pgmap v12848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:19:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:42 smithi096 ceph-mon[32753]: pgmap v12848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:19:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:19:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23322.052529 secs 2024-04-26T18:19:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:19:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:19:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:19:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23324.924385 secs 2024-04-26T18:19:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23322.052529 secs 2024-04-26T18:19:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:19:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:19:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:19:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23324.924385 secs 2024-04-26T18:19:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23322.052529 secs 2024-04-26T18:19:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:19:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:19:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:19:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23324.924385 secs 2024-04-26T18:19:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:44 smithi194 ceph-mon[33017]: pgmap v12849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:44 smithi194 ceph-mon[33017]: pgmap v12850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:44 smithi033 ceph-mon[27845]: pgmap v12849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:44 smithi033 ceph-mon[27845]: pgmap v12850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:44 smithi096 ceph-mon[32753]: pgmap v12849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:44 smithi096 ceph-mon[32753]: pgmap v12850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:46 smithi194 ceph-mon[33017]: pgmap v12851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:46 smithi033 ceph-mon[27845]: pgmap v12851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:46 smithi096 ceph-mon[32753]: pgmap v12851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:48 smithi033 ceph-mon[27845]: pgmap v12852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23327.052630 secs 2024-04-26T18:19:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23329.924499 secs 2024-04-26T18:19:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:48 smithi096 ceph-mon[32753]: pgmap v12852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23327.052630 secs 2024-04-26T18:19:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23329.924499 secs 2024-04-26T18:19:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:48 smithi194 ceph-mon[33017]: pgmap v12852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23327.052630 secs 2024-04-26T18:19:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23329.924499 secs 2024-04-26T18:19:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:49.960+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:49.960+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:19:49.961 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:19:49.960+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:19:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:50 smithi033 ceph-mon[27845]: pgmap v12853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:19:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:50 smithi096 ceph-mon[32753]: pgmap v12853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:19:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:50 smithi194 ceph-mon[33017]: pgmap v12853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:19:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:52 smithi033 ceph-mon[27845]: pgmap v12854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:52 smithi096 ceph-mon[32753]: pgmap v12854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:52 smithi194 ceph-mon[33017]: pgmap v12854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:19:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23332.052728 secs 2024-04-26T18:19:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23334.924614 secs 2024-04-26T18:19:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23332.052728 secs 2024-04-26T18:19:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23334.924614 secs 2024-04-26T18:19:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23332.052728 secs 2024-04-26T18:19:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23334.924614 secs 2024-04-26T18:19:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:54 smithi033 ceph-mon[27845]: pgmap v12855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:19:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:54 smithi096 ceph-mon[32753]: pgmap v12855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:19:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:54 smithi194 ceph-mon[33017]: pgmap v12855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:19:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:56 smithi033 ceph-mon[27845]: pgmap v12856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:56 smithi096 ceph-mon[32753]: pgmap v12856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:56 smithi194 ceph-mon[33017]: pgmap v12856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:19:58.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:58 smithi194 ceph-mon[33017]: pgmap v12857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:58.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23337.052832 secs 2024-04-26T18:19:58.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:19:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23339.924747 secs 2024-04-26T18:19:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:58 smithi033 ceph-mon[27845]: pgmap v12857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23337.052832 secs 2024-04-26T18:19:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:19:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23339.924747 secs 2024-04-26T18:19:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:58 smithi096 ceph-mon[32753]: pgmap v12857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:19:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23337.052832 secs 2024-04-26T18:19:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:19:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23339.924747 secs 2024-04-26T18:20:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:00 smithi033 ceph-mon[27845]: pgmap v12858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:20:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:00 smithi096 ceph-mon[32753]: pgmap v12858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:20:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:00 smithi194 ceph-mon[33017]: pgmap v12858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:20:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:02 smithi033 ceph-mon[27845]: pgmap v12859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:02 smithi096 ceph-mon[32753]: pgmap v12859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:02 smithi194 ceph-mon[33017]: pgmap v12859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23342.052930 secs 2024-04-26T18:20:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23344.924829 secs 2024-04-26T18:20:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23342.052930 secs 2024-04-26T18:20:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23344.924829 secs 2024-04-26T18:20:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23342.052930 secs 2024-04-26T18:20:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23344.924829 secs 2024-04-26T18:20:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:04 smithi096 ceph-mon[32753]: pgmap v12860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:04.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:04.976+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:04.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:04.976+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:04.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:04.976+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:20:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:04 smithi194 ceph-mon[33017]: pgmap v12860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:04 smithi033 ceph-mon[27845]: pgmap v12860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:06 smithi096 ceph-mon[32753]: pgmap v12861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:06 smithi194 ceph-mon[33017]: pgmap v12861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:07.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:06 smithi033 ceph-mon[27845]: pgmap v12861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:08.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:08 smithi033 ceph-mon[27845]: pgmap v12862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:08.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23347.053032 secs 2024-04-26T18:20:08.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23349.924943 secs 2024-04-26T18:20:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:08 smithi194 ceph-mon[33017]: pgmap v12862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23347.053032 secs 2024-04-26T18:20:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23349.924943 secs 2024-04-26T18:20:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:08 smithi096 ceph-mon[32753]: pgmap v12862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23347.053032 secs 2024-04-26T18:20:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23349.924943 secs 2024-04-26T18:20:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:10 smithi194 ceph-mon[33017]: pgmap v12863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:10 smithi033 ceph-mon[27845]: pgmap v12863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:10 smithi096 ceph-mon[32753]: pgmap v12863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:13.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:12 smithi194 ceph-mon[33017]: pgmap v12864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:12 smithi033 ceph-mon[27845]: pgmap v12864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:12 smithi096 ceph-mon[32753]: pgmap v12864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23352.053133 secs 2024-04-26T18:20:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23354.925063 secs 2024-04-26T18:20:14.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23352.053133 secs 2024-04-26T18:20:14.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23354.925063 secs 2024-04-26T18:20:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23352.053133 secs 2024-04-26T18:20:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23354.925063 secs 2024-04-26T18:20:15.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:14 smithi033 ceph-mon[27845]: pgmap v12865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:14 smithi194 ceph-mon[33017]: pgmap v12865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:14 smithi096 ceph-mon[32753]: pgmap v12865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:16 smithi194 ceph-mon[33017]: pgmap v12866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:16 smithi033 ceph-mon[27845]: pgmap v12866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:16 smithi096 ceph-mon[32753]: pgmap v12866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:18 smithi194 ceph-mon[33017]: pgmap v12867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23357.053218 secs 2024-04-26T18:20:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23359.925162 secs 2024-04-26T18:20:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:18 smithi033 ceph-mon[27845]: pgmap v12867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23357.053218 secs 2024-04-26T18:20:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23359.925162 secs 2024-04-26T18:20:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:18 smithi096 ceph-mon[32753]: pgmap v12867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23357.053218 secs 2024-04-26T18:20:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23359.925162 secs 2024-04-26T18:20:19.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:19.992+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:19.992 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:19.992+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:19.992+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:20:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:20 smithi194 ceph-mon[33017]: pgmap v12868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:20 smithi033 ceph-mon[27845]: pgmap v12868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:20 smithi096 ceph-mon[32753]: pgmap v12868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:22 smithi194 ceph-mon[33017]: pgmap v12869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:22 smithi033 ceph-mon[27845]: pgmap v12869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:22 smithi096 ceph-mon[32753]: pgmap v12869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23362.053309 secs 2024-04-26T18:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23364.925303 secs 2024-04-26T18:20:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23362.053309 secs 2024-04-26T18:20:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23364.925303 secs 2024-04-26T18:20:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23362.053309 secs 2024-04-26T18:20:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23364.925303 secs 2024-04-26T18:20:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:24 smithi194 ceph-mon[33017]: pgmap v12870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:24 smithi033 ceph-mon[27845]: pgmap v12870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:24 smithi096 ceph-mon[32753]: pgmap v12870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:25.772+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T18:20:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:25.772+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:20:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:25.772+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 186 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:20:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:25.772+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 186 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T18:20:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:26 smithi194 ceph-mon[33017]: pgmap v12871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:26 smithi033 ceph-mon[27845]: pgmap v12871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:26 smithi096 ceph-mon[32753]: pgmap v12871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:28 smithi194 ceph-mon[33017]: pgmap v12872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23367.053411 secs 2024-04-26T18:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23369.925417 secs 2024-04-26T18:20:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:28 smithi033 ceph-mon[27845]: pgmap v12872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23367.053411 secs 2024-04-26T18:20:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23369.925417 secs 2024-04-26T18:20:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:28 smithi096 ceph-mon[32753]: pgmap v12872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23367.053411 secs 2024-04-26T18:20:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23369.925417 secs 2024-04-26T18:20:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:30 smithi194 ceph-mon[33017]: pgmap v12873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:30 smithi033 ceph-mon[27845]: pgmap v12873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:30 smithi096 ceph-mon[32753]: pgmap v12873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:32 smithi194 ceph-mon[33017]: pgmap v12874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:32 smithi033 ceph-mon[27845]: pgmap v12874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:32 smithi096 ceph-mon[32753]: pgmap v12874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23372.053513 secs 2024-04-26T18:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23374.925536 secs 2024-04-26T18:20:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23372.053513 secs 2024-04-26T18:20:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23374.925536 secs 2024-04-26T18:20:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23372.053513 secs 2024-04-26T18:20:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23374.925536 secs 2024-04-26T18:20:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:35.008+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:35.008+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:35.008+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:20:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:34 smithi194 ceph-mon[33017]: pgmap v12875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:34 smithi033 ceph-mon[27845]: pgmap v12875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:34 smithi096 ceph-mon[32753]: pgmap v12875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:36 smithi194 ceph-mon[33017]: pgmap v12876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:36 smithi033 ceph-mon[27845]: pgmap v12876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:36 smithi096 ceph-mon[32753]: pgmap v12876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:38 smithi194 ceph-mon[33017]: pgmap v12877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23377.053610 secs 2024-04-26T18:20:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23379.925652 secs 2024-04-26T18:20:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:38 smithi033 ceph-mon[27845]: pgmap v12877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23377.053610 secs 2024-04-26T18:20:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23379.925652 secs 2024-04-26T18:20:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:38 smithi096 ceph-mon[32753]: pgmap v12877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23377.053610 secs 2024-04-26T18:20:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23379.925652 secs 2024-04-26T18:20:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:40 smithi194 ceph-mon[33017]: pgmap v12878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:40 smithi033 ceph-mon[27845]: pgmap v12878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:40 smithi096 ceph-mon[32753]: pgmap v12878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:42 smithi194 ceph-mon[33017]: pgmap v12879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:42 smithi033 ceph-mon[27845]: pgmap v12879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:42 smithi096 ceph-mon[32753]: pgmap v12879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:20:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23382.053711 secs 2024-04-26T18:20:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:20:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23384.925745 secs 2024-04-26T18:20:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23382.053711 secs 2024-04-26T18:20:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:20:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23384.925745 secs 2024-04-26T18:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23382.053711 secs 2024-04-26T18:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:20:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23384.925745 secs 2024-04-26T18:20:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:44 smithi194 ceph-mon[33017]: pgmap v12880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:20:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:20:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:20:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:44 smithi033 ceph-mon[27845]: pgmap v12880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:20:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:20:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:20:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:44 smithi096 ceph-mon[32753]: pgmap v12880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:20:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:20:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:20:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:45 smithi194 ceph-mon[33017]: pgmap v12881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:20:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:45 smithi194 ceph-mon[33017]: pgmap v12882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:45 smithi033 ceph-mon[27845]: pgmap v12881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:20:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:45 smithi033 ceph-mon[27845]: pgmap v12882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:45 smithi096 ceph-mon[32753]: pgmap v12881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:20:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:45 smithi096 ceph-mon[32753]: pgmap v12882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:47 smithi194 ceph-mon[33017]: pgmap v12883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:47 smithi033 ceph-mon[27845]: pgmap v12883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:47 smithi096 ceph-mon[32753]: pgmap v12883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23387.053771 secs 2024-04-26T18:20:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23389.925864 secs 2024-04-26T18:20:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23387.053771 secs 2024-04-26T18:20:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23389.925864 secs 2024-04-26T18:20:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23387.053771 secs 2024-04-26T18:20:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23389.925864 secs 2024-04-26T18:20:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:50.020+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:50.020+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:20:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:20:50.020+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:20:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:49 smithi194 ceph-mon[33017]: pgmap v12884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:49 smithi033 ceph-mon[27845]: pgmap v12884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:49 smithi096 ceph-mon[32753]: pgmap v12884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:51 smithi194 ceph-mon[33017]: pgmap v12885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:51 smithi033 ceph-mon[27845]: pgmap v12885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:51 smithi096 ceph-mon[32753]: pgmap v12885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:53 smithi194 ceph-mon[33017]: pgmap v12886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23392.053853 secs 2024-04-26T18:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23394.925989 secs 2024-04-26T18:20:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:53 smithi033 ceph-mon[27845]: pgmap v12886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23392.053853 secs 2024-04-26T18:20:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23394.925989 secs 2024-04-26T18:20:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:53 smithi096 ceph-mon[32753]: pgmap v12886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:20:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23392.053853 secs 2024-04-26T18:20:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23394.925989 secs 2024-04-26T18:20:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:55 smithi194 ceph-mon[33017]: pgmap v12887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:20:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:55 smithi033 ceph-mon[27845]: pgmap v12887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:20:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:55 smithi096 ceph-mon[32753]: pgmap v12887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:20:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:57 smithi033 ceph-mon[27845]: pgmap v12888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:57 smithi096 ceph-mon[32753]: pgmap v12888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:57 smithi194 ceph-mon[33017]: pgmap v12888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:20:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23397.053938 secs 2024-04-26T18:20:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23399.926135 secs 2024-04-26T18:20:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23397.053938 secs 2024-04-26T18:20:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23399.926135 secs 2024-04-26T18:20:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23397.053938 secs 2024-04-26T18:20:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23399.926135 secs 2024-04-26T18:21:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:20:59 smithi033 ceph-mon[27845]: pgmap v12889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:20:59 smithi096 ceph-mon[32753]: pgmap v12889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:20:59 smithi194 ceph-mon[33017]: pgmap v12889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:01 smithi033 ceph-mon[27845]: pgmap v12890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:01 smithi096 ceph-mon[32753]: pgmap v12890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:01 smithi194 ceph-mon[33017]: pgmap v12890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:03 smithi033 ceph-mon[27845]: pgmap v12891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23402.054005 secs 2024-04-26T18:21:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23404.926254 secs 2024-04-26T18:21:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:03 smithi096 ceph-mon[32753]: pgmap v12891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23402.054005 secs 2024-04-26T18:21:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23404.926254 secs 2024-04-26T18:21:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:03 smithi194 ceph-mon[33017]: pgmap v12891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23402.054005 secs 2024-04-26T18:21:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23404.926254 secs 2024-04-26T18:21:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:05.036+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:05.036+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:05.036+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:21:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:05 smithi033 ceph-mon[27845]: pgmap v12892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:05 smithi096 ceph-mon[32753]: pgmap v12892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:05 smithi194 ceph-mon[33017]: pgmap v12892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:07 smithi033 ceph-mon[27845]: pgmap v12893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:07 smithi096 ceph-mon[32753]: pgmap v12893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:07 smithi194 ceph-mon[33017]: pgmap v12893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23407.054065 secs 2024-04-26T18:21:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23409.926370 secs 2024-04-26T18:21:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23407.054065 secs 2024-04-26T18:21:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23409.926370 secs 2024-04-26T18:21:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23407.054065 secs 2024-04-26T18:21:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23409.926370 secs 2024-04-26T18:21:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:09 smithi033 ceph-mon[27845]: pgmap v12894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:09 smithi096 ceph-mon[32753]: pgmap v12894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:09 smithi194 ceph-mon[33017]: pgmap v12894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:11 smithi033 ceph-mon[27845]: pgmap v12895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:11 smithi096 ceph-mon[32753]: pgmap v12895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:11 smithi194 ceph-mon[33017]: pgmap v12895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:13 smithi033 ceph-mon[27845]: pgmap v12896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23412.054152 secs 2024-04-26T18:21:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23414.926492 secs 2024-04-26T18:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:13 smithi096 ceph-mon[32753]: pgmap v12896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23412.054152 secs 2024-04-26T18:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23414.926492 secs 2024-04-26T18:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:13 smithi194 ceph-mon[33017]: pgmap v12896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23412.054152 secs 2024-04-26T18:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23414.926492 secs 2024-04-26T18:21:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:15 smithi033 ceph-mon[27845]: pgmap v12897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:15 smithi096 ceph-mon[32753]: pgmap v12897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:15 smithi194 ceph-mon[33017]: pgmap v12897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:17 smithi096 ceph-mon[32753]: pgmap v12898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:17 smithi194 ceph-mon[33017]: pgmap v12898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:17 smithi033 ceph-mon[27845]: pgmap v12898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23417.054258 secs 2024-04-26T18:21:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23419.926612 secs 2024-04-26T18:21:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23417.054258 secs 2024-04-26T18:21:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23419.926612 secs 2024-04-26T18:21:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23417.054258 secs 2024-04-26T18:21:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23419.926612 secs 2024-04-26T18:21:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:20.040+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:20.040+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:20.040+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:21:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:19 smithi194 ceph-mon[33017]: pgmap v12899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:19 smithi033 ceph-mon[27845]: pgmap v12899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:19 smithi096 ceph-mon[32753]: pgmap v12899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:21 smithi194 ceph-mon[33017]: pgmap v12900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:21 smithi033 ceph-mon[27845]: pgmap v12900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:21 smithi096 ceph-mon[32753]: pgmap v12900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:23 smithi194 ceph-mon[33017]: pgmap v12901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23422.054339 secs 2024-04-26T18:21:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23424.926734 secs 2024-04-26T18:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:23 smithi033 ceph-mon[27845]: pgmap v12901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23422.054339 secs 2024-04-26T18:21:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23424.926734 secs 2024-04-26T18:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:23 smithi096 ceph-mon[32753]: pgmap v12901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23422.054339 secs 2024-04-26T18:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23424.926734 secs 2024-04-26T18:21:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:25 smithi194 ceph-mon[33017]: pgmap v12902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:25 smithi033 ceph-mon[27845]: pgmap v12902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:25 smithi096 ceph-mon[32753]: pgmap v12902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:27 smithi194 ceph-mon[33017]: pgmap v12903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:27 smithi033 ceph-mon[27845]: pgmap v12903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:27 smithi096 ceph-mon[32753]: pgmap v12903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23427.054423 secs 2024-04-26T18:21:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23429.926874 secs 2024-04-26T18:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23427.054423 secs 2024-04-26T18:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23429.926874 secs 2024-04-26T18:21:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23427.054423 secs 2024-04-26T18:21:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23429.926874 secs 2024-04-26T18:21:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:29 smithi096 ceph-mon[32753]: pgmap v12904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:29 smithi194 ceph-mon[33017]: pgmap v12904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:29 smithi033 ceph-mon[27845]: pgmap v12904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:32.335 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:31 smithi033 ceph-mon[27845]: pgmap v12905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:31 smithi194 ceph-mon[33017]: pgmap v12905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:31 smithi096 ceph-mon[32753]: pgmap v12905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:32 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23432.054503 secs 2024-04-26T18:21:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:32 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23432.054503 secs 2024-04-26T18:21:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:32 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23432.054503 secs 2024-04-26T18:21:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:33 smithi194 ceph-mon[33017]: pgmap v12906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23434.926976 secs 2024-04-26T18:21:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:33 smithi033 ceph-mon[27845]: pgmap v12906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23434.926976 secs 2024-04-26T18:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:33 smithi096 ceph-mon[32753]: pgmap v12906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23434.926976 secs 2024-04-26T18:21:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:35.055+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:35.055+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:35.055+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:21:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:36 smithi194 ceph-mon[33017]: pgmap v12907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:36 smithi033 ceph-mon[27845]: pgmap v12907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:36 smithi096 ceph-mon[32753]: pgmap v12907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:38 smithi194 ceph-mon[33017]: pgmap v12908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23437.054580 secs 2024-04-26T18:21:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:38 smithi033 ceph-mon[27845]: pgmap v12908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23437.054580 secs 2024-04-26T18:21:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:38 smithi096 ceph-mon[32753]: pgmap v12908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23437.054580 secs 2024-04-26T18:21:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23439.927096 secs 2024-04-26T18:21:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23439.927096 secs 2024-04-26T18:21:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23439.927096 secs 2024-04-26T18:21:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:40 smithi194 ceph-mon[33017]: pgmap v12909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:40 smithi033 ceph-mon[27845]: pgmap v12909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:40 smithi096 ceph-mon[32753]: pgmap v12909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:42 smithi194 ceph-mon[33017]: pgmap v12910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:42 smithi033 ceph-mon[27845]: pgmap v12910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:42 smithi096 ceph-mon[32753]: pgmap v12910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23442.054665 secs 2024-04-26T18:21:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23442.054665 secs 2024-04-26T18:21:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23442.054665 secs 2024-04-26T18:21:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:44 smithi194 ceph-mon[33017]: pgmap v12911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23444.927207 secs 2024-04-26T18:21:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:44 smithi033 ceph-mon[27845]: pgmap v12911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23444.927207 secs 2024-04-26T18:21:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:44 smithi096 ceph-mon[32753]: pgmap v12911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23444.927207 secs 2024-04-26T18:21:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:45 smithi194 ceph-mon[33017]: pgmap v12912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:21:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:45 smithi033 ceph-mon[27845]: pgmap v12912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:21:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:45 smithi096 ceph-mon[32753]: pgmap v12912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:21:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:21:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:21:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:21:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:21:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:21:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:21:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:21:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:21:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:21:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:21:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:47 smithi194 ceph-mon[33017]: pgmap v12913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:47 smithi194 ceph-mon[33017]: pgmap v12914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:47 smithi033 ceph-mon[27845]: pgmap v12913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:47 smithi033 ceph-mon[27845]: pgmap v12914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:47 smithi096 ceph-mon[32753]: pgmap v12913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:21:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:47 smithi096 ceph-mon[32753]: pgmap v12914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23447.054758 secs 2024-04-26T18:21:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23449.927328 secs 2024-04-26T18:21:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23447.054758 secs 2024-04-26T18:21:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23449.927328 secs 2024-04-26T18:21:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23447.054758 secs 2024-04-26T18:21:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23449.927328 secs 2024-04-26T18:21:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:50.057+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:50.057+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:21:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:21:50.057+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:21:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:49 smithi033 ceph-mon[27845]: pgmap v12915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:49 smithi096 ceph-mon[32753]: pgmap v12915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:49 smithi194 ceph-mon[33017]: pgmap v12915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:51 smithi033 ceph-mon[27845]: pgmap v12916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:51 smithi096 ceph-mon[32753]: pgmap v12916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:51 smithi194 ceph-mon[33017]: pgmap v12916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:53 smithi033 ceph-mon[27845]: pgmap v12917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23452.054864 secs 2024-04-26T18:21:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23454.927395 secs 2024-04-26T18:21:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:53 smithi096 ceph-mon[32753]: pgmap v12917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23452.054864 secs 2024-04-26T18:21:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23454.927395 secs 2024-04-26T18:21:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:53 smithi194 ceph-mon[33017]: pgmap v12917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:21:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23452.054864 secs 2024-04-26T18:21:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23454.927395 secs 2024-04-26T18:21:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:55 smithi033 ceph-mon[27845]: pgmap v12918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:21:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:55 smithi096 ceph-mon[32753]: pgmap v12918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:21:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:55 smithi194 ceph-mon[33017]: pgmap v12918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:21:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:57 smithi033 ceph-mon[27845]: pgmap v12919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:21:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:57 smithi096 ceph-mon[32753]: pgmap v12919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:21:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:57 smithi194 ceph-mon[33017]: pgmap v12919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:21:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23457.054977 secs 2024-04-26T18:21:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23459.927545 secs 2024-04-26T18:21:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23457.054977 secs 2024-04-26T18:21:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23459.927545 secs 2024-04-26T18:21:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23457.054977 secs 2024-04-26T18:21:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23459.927545 secs 2024-04-26T18:22:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:21:59 smithi033 ceph-mon[27845]: pgmap v12920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:21:59 smithi096 ceph-mon[32753]: pgmap v12920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:21:59 smithi194 ceph-mon[33017]: pgmap v12920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:01 smithi033 ceph-mon[27845]: pgmap v12921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:01 smithi096 ceph-mon[32753]: pgmap v12921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:01 smithi194 ceph-mon[33017]: pgmap v12921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:03 smithi033 ceph-mon[27845]: pgmap v12922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23462.055067 secs 2024-04-26T18:22:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23464.927678 secs 2024-04-26T18:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:03 smithi096 ceph-mon[32753]: pgmap v12922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23462.055067 secs 2024-04-26T18:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23464.927678 secs 2024-04-26T18:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:03 smithi194 ceph-mon[33017]: pgmap v12922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23462.055067 secs 2024-04-26T18:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23464.927678 secs 2024-04-26T18:22:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:05.063+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:05.063+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:05.063+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:22:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:05 smithi033 ceph-mon[27845]: pgmap v12923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:05 smithi096 ceph-mon[32753]: pgmap v12923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:05 smithi194 ceph-mon[33017]: pgmap v12923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:07 smithi033 ceph-mon[27845]: pgmap v12924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:07 smithi096 ceph-mon[32753]: pgmap v12924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:07 smithi194 ceph-mon[33017]: pgmap v12924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23467.055156 secs 2024-04-26T18:22:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23469.927793 secs 2024-04-26T18:22:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23467.055156 secs 2024-04-26T18:22:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23469.927793 secs 2024-04-26T18:22:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23467.055156 secs 2024-04-26T18:22:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23469.927793 secs 2024-04-26T18:22:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:09 smithi096 ceph-mon[32753]: pgmap v12925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:09 smithi194 ceph-mon[33017]: pgmap v12925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:09 smithi033 ceph-mon[27845]: pgmap v12925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:11 smithi194 ceph-mon[33017]: pgmap v12926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:11 smithi033 ceph-mon[27845]: pgmap v12926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:11 smithi096 ceph-mon[32753]: pgmap v12926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:13 smithi194 ceph-mon[33017]: pgmap v12927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23472.055244 secs 2024-04-26T18:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23474.927914 secs 2024-04-26T18:22:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:13 smithi033 ceph-mon[27845]: pgmap v12927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23472.055244 secs 2024-04-26T18:22:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23474.927914 secs 2024-04-26T18:22:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:13 smithi096 ceph-mon[32753]: pgmap v12927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23472.055244 secs 2024-04-26T18:22:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23474.927914 secs 2024-04-26T18:22:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:15 smithi194 ceph-mon[33017]: pgmap v12928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:15 smithi033 ceph-mon[27845]: pgmap v12928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:15 smithi096 ceph-mon[32753]: pgmap v12928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:17 smithi194 ceph-mon[33017]: pgmap v12929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:17 smithi033 ceph-mon[27845]: pgmap v12929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:17 smithi096 ceph-mon[32753]: pgmap v12929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23477.055329 secs 2024-04-26T18:22:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23479.928030 secs 2024-04-26T18:22:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23477.055329 secs 2024-04-26T18:22:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23479.928030 secs 2024-04-26T18:22:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23477.055329 secs 2024-04-26T18:22:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23479.928030 secs 2024-04-26T18:22:20.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:20.075+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:20.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:20.075+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:20.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:20.075+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:22:20.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:19 smithi033 ceph-mon[27845]: pgmap v12930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:19 smithi194 ceph-mon[33017]: pgmap v12930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:19 smithi096 ceph-mon[32753]: pgmap v12930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:21 smithi194 ceph-mon[33017]: pgmap v12931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:21 smithi033 ceph-mon[27845]: pgmap v12931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:21 smithi096 ceph-mon[32753]: pgmap v12931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:23 smithi194 ceph-mon[33017]: pgmap v12932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23482.055413 secs 2024-04-26T18:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23484.928147 secs 2024-04-26T18:22:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:23 smithi033 ceph-mon[27845]: pgmap v12932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23482.055413 secs 2024-04-26T18:22:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23484.928147 secs 2024-04-26T18:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:23 smithi096 ceph-mon[32753]: pgmap v12932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23482.055413 secs 2024-04-26T18:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23484.928147 secs 2024-04-26T18:22:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:25 smithi194 ceph-mon[33017]: pgmap v12933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:26.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:25 smithi033 ceph-mon[27845]: pgmap v12933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:26.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:25 smithi096 ceph-mon[32753]: pgmap v12933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:27 smithi194 ceph-mon[33017]: pgmap v12934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:27 smithi033 ceph-mon[27845]: pgmap v12934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:27 smithi096 ceph-mon[32753]: pgmap v12934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23487.055511 secs 2024-04-26T18:22:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23489.928263 secs 2024-04-26T18:22:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23487.055511 secs 2024-04-26T18:22:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23489.928263 secs 2024-04-26T18:22:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23487.055511 secs 2024-04-26T18:22:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23489.928263 secs 2024-04-26T18:22:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:29 smithi096 ceph-mon[32753]: pgmap v12935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:29 smithi194 ceph-mon[33017]: pgmap v12935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:29 smithi033 ceph-mon[27845]: pgmap v12935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:31 smithi194 ceph-mon[33017]: pgmap v12936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:31 smithi033 ceph-mon[27845]: pgmap v12936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:31 smithi096 ceph-mon[32753]: pgmap v12936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23492.055592 secs 2024-04-26T18:22:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23492.055592 secs 2024-04-26T18:22:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23492.055592 secs 2024-04-26T18:22:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:34 smithi194 ceph-mon[33017]: pgmap v12937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23494.928371 secs 2024-04-26T18:22:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:34 smithi033 ceph-mon[27845]: pgmap v12937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23494.928371 secs 2024-04-26T18:22:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:34 smithi096 ceph-mon[32753]: pgmap v12937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23494.928371 secs 2024-04-26T18:22:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:35.085+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:35.085+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:35.085+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:22:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:36 smithi194 ceph-mon[33017]: pgmap v12938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:36 smithi033 ceph-mon[27845]: pgmap v12938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:36 smithi096 ceph-mon[32753]: pgmap v12938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:38 smithi194 ceph-mon[33017]: pgmap v12939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23497.055674 secs 2024-04-26T18:22:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:38 smithi033 ceph-mon[27845]: pgmap v12939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23497.055674 secs 2024-04-26T18:22:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:38 smithi096 ceph-mon[32753]: pgmap v12939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23497.055674 secs 2024-04-26T18:22:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23499.928447 secs 2024-04-26T18:22:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23499.928447 secs 2024-04-26T18:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23499.928447 secs 2024-04-26T18:22:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:40 smithi194 ceph-mon[33017]: pgmap v12940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:40 smithi033 ceph-mon[27845]: pgmap v12940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:40 smithi096 ceph-mon[32753]: pgmap v12940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:42 smithi194 ceph-mon[33017]: pgmap v12941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:42 smithi033 ceph-mon[27845]: pgmap v12941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:42 smithi096 ceph-mon[32753]: pgmap v12941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:22:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23502.055754 secs 2024-04-26T18:22:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23502.055754 secs 2024-04-26T18:22:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23502.055754 secs 2024-04-26T18:22:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:44 smithi194 ceph-mon[33017]: pgmap v12942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23504.928563 secs 2024-04-26T18:22:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:44 smithi033 ceph-mon[27845]: pgmap v12942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23504.928563 secs 2024-04-26T18:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:44 smithi096 ceph-mon[32753]: pgmap v12942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23504.928563 secs 2024-04-26T18:22:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:45 smithi194 ceph-mon[33017]: pgmap v12943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:45 smithi033 ceph-mon[27845]: pgmap v12943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:45 smithi096 ceph-mon[32753]: pgmap v12943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:47 smithi194 ceph-mon[33017]: pgmap v12944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:22:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:47 smithi033 ceph-mon[27845]: pgmap v12944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:22:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:47 smithi096 ceph-mon[32753]: pgmap v12944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:22:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:22:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23507.055839 secs 2024-04-26T18:22:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:22:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:22:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:22:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23507.055839 secs 2024-04-26T18:22:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:22:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:22:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:22:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23507.055839 secs 2024-04-26T18:22:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:22:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:22:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:22:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:49 smithi194 ceph-mon[33017]: pgmap v12945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:49 smithi194 ceph-mon[33017]: pgmap v12946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23509.928634 secs 2024-04-26T18:22:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:49 smithi033 ceph-mon[27845]: pgmap v12945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:49 smithi033 ceph-mon[27845]: pgmap v12946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23509.928634 secs 2024-04-26T18:22:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:49 smithi096 ceph-mon[32753]: pgmap v12945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:22:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:49 smithi096 ceph-mon[32753]: pgmap v12946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23509.928634 secs 2024-04-26T18:22:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:50.100+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:50.100+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:22:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:22:50.100+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:22:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:51 smithi194 ceph-mon[33017]: pgmap v12947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:51 smithi033 ceph-mon[27845]: pgmap v12947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:51 smithi096 ceph-mon[32753]: pgmap v12947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:53 smithi194 ceph-mon[33017]: pgmap v12948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23512.055918 secs 2024-04-26T18:22:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:53 smithi033 ceph-mon[27845]: pgmap v12948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23512.055918 secs 2024-04-26T18:22:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:53 smithi096 ceph-mon[32753]: pgmap v12948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23512.055918 secs 2024-04-26T18:22:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23514.928751 secs 2024-04-26T18:22:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23514.928751 secs 2024-04-26T18:22:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23514.928751 secs 2024-04-26T18:22:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:55 smithi194 ceph-mon[33017]: pgmap v12949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:22:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:55 smithi033 ceph-mon[27845]: pgmap v12949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:22:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:55 smithi096 ceph-mon[32753]: pgmap v12949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:22:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:57 smithi194 ceph-mon[33017]: pgmap v12950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:57 smithi033 ceph-mon[27845]: pgmap v12950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:57 smithi096 ceph-mon[32753]: pgmap v12950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:22:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23517.055999 secs 2024-04-26T18:22:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23517.055999 secs 2024-04-26T18:22:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23517.055999 secs 2024-04-26T18:22:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:59 smithi194 ceph-mon[33017]: pgmap v12951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:22:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23519.928866 secs 2024-04-26T18:22:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:59 smithi033 ceph-mon[27845]: pgmap v12951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:22:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23519.928866 secs 2024-04-26T18:22:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:59 smithi096 ceph-mon[32753]: pgmap v12951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:22:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23519.928866 secs 2024-04-26T18:23:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:01 smithi033 ceph-mon[27845]: pgmap v12952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:01 smithi096 ceph-mon[32753]: pgmap v12952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:01 smithi194 ceph-mon[33017]: pgmap v12952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:03 smithi033 ceph-mon[27845]: pgmap v12953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23522.056086 secs 2024-04-26T18:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23524.928989 secs 2024-04-26T18:23:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:03 smithi096 ceph-mon[32753]: pgmap v12953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23522.056086 secs 2024-04-26T18:23:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23524.928989 secs 2024-04-26T18:23:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:03 smithi194 ceph-mon[33017]: pgmap v12953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23522.056086 secs 2024-04-26T18:23:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23524.928989 secs 2024-04-26T18:23:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:05.115+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:05.115+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:05.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:05.115+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:23:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:05 smithi033 ceph-mon[27845]: pgmap v12954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:05 smithi096 ceph-mon[32753]: pgmap v12954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:05 smithi194 ceph-mon[33017]: pgmap v12954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:07 smithi033 ceph-mon[27845]: pgmap v12955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:07 smithi096 ceph-mon[32753]: pgmap v12955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:07 smithi194 ceph-mon[33017]: pgmap v12955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23527.056173 secs 2024-04-26T18:23:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23529.929139 secs 2024-04-26T18:23:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23527.056173 secs 2024-04-26T18:23:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23529.929139 secs 2024-04-26T18:23:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23527.056173 secs 2024-04-26T18:23:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23529.929139 secs 2024-04-26T18:23:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:09 smithi033 ceph-mon[27845]: pgmap v12956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:09 smithi096 ceph-mon[32753]: pgmap v12956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:09 smithi194 ceph-mon[33017]: pgmap v12956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:11 smithi033 ceph-mon[27845]: pgmap v12957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:11 smithi096 ceph-mon[32753]: pgmap v12957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:11 smithi194 ceph-mon[33017]: pgmap v12957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:13 smithi033 ceph-mon[27845]: pgmap v12958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23532.056272 secs 2024-04-26T18:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23534.929255 secs 2024-04-26T18:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:13 smithi096 ceph-mon[32753]: pgmap v12958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23532.056272 secs 2024-04-26T18:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23534.929255 secs 2024-04-26T18:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:13 smithi194 ceph-mon[33017]: pgmap v12958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23532.056272 secs 2024-04-26T18:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23534.929255 secs 2024-04-26T18:23:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:15 smithi033 ceph-mon[27845]: pgmap v12959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:15 smithi096 ceph-mon[32753]: pgmap v12959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:15 smithi194 ceph-mon[33017]: pgmap v12959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:17 smithi033 ceph-mon[27845]: pgmap v12960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:17 smithi096 ceph-mon[32753]: pgmap v12960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:17 smithi194 ceph-mon[33017]: pgmap v12960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23537.056355 secs 2024-04-26T18:23:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23539.929352 secs 2024-04-26T18:23:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23537.056355 secs 2024-04-26T18:23:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23539.929352 secs 2024-04-26T18:23:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23537.056355 secs 2024-04-26T18:23:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23539.929352 secs 2024-04-26T18:23:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:20.117+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:20.117+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:20.117+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:23:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:19 smithi194 ceph-mon[33017]: pgmap v12961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:19 smithi033 ceph-mon[27845]: pgmap v12961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:19 smithi096 ceph-mon[32753]: pgmap v12961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:21 smithi033 ceph-mon[27845]: pgmap v12962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:21 smithi194 ceph-mon[33017]: pgmap v12962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:21 smithi096 ceph-mon[32753]: pgmap v12962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:23 smithi194 ceph-mon[33017]: pgmap v12963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23542.056438 secs 2024-04-26T18:23:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23544.929473 secs 2024-04-26T18:23:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:23 smithi033 ceph-mon[27845]: pgmap v12963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23542.056438 secs 2024-04-26T18:23:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23544.929473 secs 2024-04-26T18:23:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:23 smithi096 ceph-mon[32753]: pgmap v12963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23542.056438 secs 2024-04-26T18:23:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23544.929473 secs 2024-04-26T18:23:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:25.802+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T18:23:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:25.802+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:23:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:25.803+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 187 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:23:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:25.803+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 187 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T18:23:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:25 smithi194 ceph-mon[33017]: pgmap v12964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:25 smithi033 ceph-mon[27845]: pgmap v12964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:25 smithi096 ceph-mon[32753]: pgmap v12964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:27 smithi194 ceph-mon[33017]: pgmap v12965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:27 smithi033 ceph-mon[27845]: pgmap v12965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:27 smithi096 ceph-mon[32753]: pgmap v12965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23547.056497 secs 2024-04-26T18:23:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23549.929584 secs 2024-04-26T18:23:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23547.056497 secs 2024-04-26T18:23:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23549.929584 secs 2024-04-26T18:23:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23547.056497 secs 2024-04-26T18:23:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23549.929584 secs 2024-04-26T18:23:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:29 smithi194 ceph-mon[33017]: pgmap v12966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:29 smithi033 ceph-mon[27845]: pgmap v12966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:29 smithi096 ceph-mon[32753]: pgmap v12966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:31 smithi194 ceph-mon[33017]: pgmap v12967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:31 smithi033 ceph-mon[27845]: pgmap v12967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:31 smithi096 ceph-mon[32753]: pgmap v12967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:33 smithi194 ceph-mon[33017]: pgmap v12968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23552.056580 secs 2024-04-26T18:23:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23554.929723 secs 2024-04-26T18:23:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:33 smithi033 ceph-mon[27845]: pgmap v12968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23552.056580 secs 2024-04-26T18:23:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23554.929723 secs 2024-04-26T18:23:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:33 smithi096 ceph-mon[32753]: pgmap v12968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23552.056580 secs 2024-04-26T18:23:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23554.929723 secs 2024-04-26T18:23:35.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:35.121+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:35.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:35.121+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:35.121 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:35.121+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:23:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:35 smithi194 ceph-mon[33017]: pgmap v12969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:35 smithi033 ceph-mon[27845]: pgmap v12969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:35 smithi096 ceph-mon[32753]: pgmap v12969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:38 smithi194 ceph-mon[33017]: pgmap v12970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23557.056639 secs 2024-04-26T18:23:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:38 smithi033 ceph-mon[27845]: pgmap v12970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23557.056639 secs 2024-04-26T18:23:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:38 smithi096 ceph-mon[32753]: pgmap v12970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23557.056639 secs 2024-04-26T18:23:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23559.929862 secs 2024-04-26T18:23:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23559.929862 secs 2024-04-26T18:23:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23559.929862 secs 2024-04-26T18:23:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:40 smithi194 ceph-mon[33017]: pgmap v12971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:40 smithi033 ceph-mon[27845]: pgmap v12971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:40 smithi096 ceph-mon[32753]: pgmap v12971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:42 smithi194 ceph-mon[33017]: pgmap v12972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:42 smithi033 ceph-mon[27845]: pgmap v12972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:42 smithi096 ceph-mon[32753]: pgmap v12972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23562.056722 secs 2024-04-26T18:23:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23562.056722 secs 2024-04-26T18:23:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23562.056722 secs 2024-04-26T18:23:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:44 smithi194 ceph-mon[33017]: pgmap v12973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23564.929981 secs 2024-04-26T18:23:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:44 smithi033 ceph-mon[27845]: pgmap v12973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23564.929981 secs 2024-04-26T18:23:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:44 smithi096 ceph-mon[32753]: pgmap v12973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23564.929981 secs 2024-04-26T18:23:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:46 smithi194 ceph-mon[33017]: pgmap v12974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:46 smithi033 ceph-mon[27845]: pgmap v12974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:46 smithi096 ceph-mon[32753]: pgmap v12974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:48 smithi194 ceph-mon[33017]: pgmap v12975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23567.056807 secs 2024-04-26T18:23:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:48 smithi033 ceph-mon[27845]: pgmap v12975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23567.056807 secs 2024-04-26T18:23:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:48 smithi096 ceph-mon[32753]: pgmap v12975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:23:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23567.056807 secs 2024-04-26T18:23:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:23:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23569.930048 secs 2024-04-26T18:23:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:23:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23569.930048 secs 2024-04-26T18:23:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:23:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23569.930048 secs 2024-04-26T18:23:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:50.132+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:50.132+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:23:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:23:50.132+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:23:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:50 smithi194 ceph-mon[33017]: pgmap v12976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:23:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:23:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:50 smithi194 ceph-mon[33017]: pgmap v12977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:23:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:50 smithi194 ceph-mon[33017]: pgmap v12978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:23:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:50 smithi033 ceph-mon[27845]: pgmap v12976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:23:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:23:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:50 smithi033 ceph-mon[27845]: pgmap v12977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:23:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:50 smithi033 ceph-mon[27845]: pgmap v12978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:23:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:50 smithi096 ceph-mon[32753]: pgmap v12976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:23:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:23:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:23:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:50 smithi096 ceph-mon[32753]: pgmap v12977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:23:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:50 smithi096 ceph-mon[32753]: pgmap v12978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:23:53.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:52 smithi033 ceph-mon[27845]: pgmap v12979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:52 smithi096 ceph-mon[32753]: pgmap v12979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:52 smithi194 ceph-mon[33017]: pgmap v12979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23572.056889 secs 2024-04-26T18:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23574.930169 secs 2024-04-26T18:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23572.056889 secs 2024-04-26T18:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23574.930169 secs 2024-04-26T18:23:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23572.056889 secs 2024-04-26T18:23:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23574.930169 secs 2024-04-26T18:23:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:54 smithi033 ceph-mon[27845]: pgmap v12980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:54 smithi096 ceph-mon[32753]: pgmap v12980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:54 smithi194 ceph-mon[33017]: pgmap v12980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:56 smithi033 ceph-mon[27845]: pgmap v12981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:56 smithi096 ceph-mon[32753]: pgmap v12981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:56 smithi194 ceph-mon[33017]: pgmap v12981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23577.056998 secs 2024-04-26T18:23:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:58 smithi033 ceph-mon[27845]: pgmap v12982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:23:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23579.930294 secs 2024-04-26T18:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23577.056998 secs 2024-04-26T18:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:58 smithi096 ceph-mon[32753]: pgmap v12982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:23:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23579.930294 secs 2024-04-26T18:23:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23577.056998 secs 2024-04-26T18:23:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:58 smithi194 ceph-mon[33017]: pgmap v12982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:23:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:23:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23579.930294 secs 2024-04-26T18:24:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:00 smithi033 ceph-mon[27845]: pgmap v12983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:24:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:00 smithi096 ceph-mon[32753]: pgmap v12983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:24:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:00 smithi194 ceph-mon[33017]: pgmap v12983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:24:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:02 smithi033 ceph-mon[27845]: pgmap v12984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:02 smithi096 ceph-mon[32753]: pgmap v12984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:02 smithi194 ceph-mon[33017]: pgmap v12984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23582.057083 secs 2024-04-26T18:24:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23584.930404 secs 2024-04-26T18:24:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23582.057083 secs 2024-04-26T18:24:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23584.930404 secs 2024-04-26T18:24:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23582.057083 secs 2024-04-26T18:24:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23584.930404 secs 2024-04-26T18:24:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:05.136+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:05.136+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:05.136+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:24:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:04 smithi096 ceph-mon[32753]: pgmap v12985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:04 smithi194 ceph-mon[33017]: pgmap v12985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:04 smithi033 ceph-mon[27845]: pgmap v12985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:06 smithi194 ceph-mon[33017]: pgmap v12986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:06 smithi033 ceph-mon[27845]: pgmap v12986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:06 smithi096 ceph-mon[32753]: pgmap v12986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:09.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23587.057171 secs 2024-04-26T18:24:09.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:08 smithi033 ceph-mon[27845]: pgmap v12987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:09.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23589.930523 secs 2024-04-26T18:24:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23587.057171 secs 2024-04-26T18:24:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:08 smithi194 ceph-mon[33017]: pgmap v12987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23589.930523 secs 2024-04-26T18:24:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23587.057171 secs 2024-04-26T18:24:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:08 smithi096 ceph-mon[32753]: pgmap v12987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23589.930523 secs 2024-04-26T18:24:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:10 smithi194 ceph-mon[33017]: pgmap v12988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:10 smithi033 ceph-mon[27845]: pgmap v12988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:10 smithi096 ceph-mon[32753]: pgmap v12988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:12 smithi194 ceph-mon[33017]: pgmap v12989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:12 smithi033 ceph-mon[27845]: pgmap v12989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:12 smithi096 ceph-mon[32753]: pgmap v12989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23592.057237 secs 2024-04-26T18:24:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23594.930650 secs 2024-04-26T18:24:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23592.057237 secs 2024-04-26T18:24:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23594.930650 secs 2024-04-26T18:24:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23592.057237 secs 2024-04-26T18:24:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23594.930650 secs 2024-04-26T18:24:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:14 smithi194 ceph-mon[33017]: pgmap v12990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:14 smithi033 ceph-mon[27845]: pgmap v12990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:14 smithi096 ceph-mon[32753]: pgmap v12990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:16 smithi194 ceph-mon[33017]: pgmap v12991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:16 smithi033 ceph-mon[27845]: pgmap v12991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:16 smithi096 ceph-mon[32753]: pgmap v12991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23597.057295 secs 2024-04-26T18:24:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:18 smithi194 ceph-mon[33017]: pgmap v12992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23599.930787 secs 2024-04-26T18:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23597.057295 secs 2024-04-26T18:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:18 smithi033 ceph-mon[27845]: pgmap v12992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23599.930787 secs 2024-04-26T18:24:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23597.057295 secs 2024-04-26T18:24:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:18 smithi096 ceph-mon[32753]: pgmap v12992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23599.930787 secs 2024-04-26T18:24:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:20.149+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:20.149+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:20.149+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:24:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:20 smithi194 ceph-mon[33017]: pgmap v12993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:20 smithi033 ceph-mon[27845]: pgmap v12993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:20 smithi096 ceph-mon[32753]: pgmap v12993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:22 smithi194 ceph-mon[33017]: pgmap v12994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:22 smithi033 ceph-mon[27845]: pgmap v12994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:22 smithi096 ceph-mon[32753]: pgmap v12994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23602.057357 secs 2024-04-26T18:24:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23604.930920 secs 2024-04-26T18:24:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23602.057357 secs 2024-04-26T18:24:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23604.930920 secs 2024-04-26T18:24:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23602.057357 secs 2024-04-26T18:24:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23604.930920 secs 2024-04-26T18:24:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:25 smithi194 ceph-mon[33017]: pgmap v12995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:25 smithi033 ceph-mon[27845]: pgmap v12995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:25 smithi096 ceph-mon[32753]: pgmap v12995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:27 smithi194 ceph-mon[33017]: pgmap v12996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:27 smithi033 ceph-mon[27845]: pgmap v12996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:27 smithi096 ceph-mon[32753]: pgmap v12996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23607.057439 secs 2024-04-26T18:24:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23607.057439 secs 2024-04-26T18:24:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23607.057439 secs 2024-04-26T18:24:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:29 smithi194 ceph-mon[33017]: pgmap v12997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23609.930982 secs 2024-04-26T18:24:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:29 smithi033 ceph-mon[27845]: pgmap v12997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23609.930982 secs 2024-04-26T18:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:29 smithi096 ceph-mon[32753]: pgmap v12997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23609.930982 secs 2024-04-26T18:24:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:31 smithi194 ceph-mon[33017]: pgmap v12998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:31 smithi033 ceph-mon[27845]: pgmap v12998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:31 smithi096 ceph-mon[32753]: pgmap v12998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:33 smithi194 ceph-mon[33017]: pgmap v12999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23612.057499 secs 2024-04-26T18:24:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:33 smithi033 ceph-mon[27845]: pgmap v12999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23612.057499 secs 2024-04-26T18:24:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:33 smithi096 ceph-mon[32753]: pgmap v12999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23612.057499 secs 2024-04-26T18:24:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23614.931156 secs 2024-04-26T18:24:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23614.931156 secs 2024-04-26T18:24:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23614.931156 secs 2024-04-26T18:24:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:35.150+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:35.150+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:35.150+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:24:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:35 smithi194 ceph-mon[33017]: pgmap v13000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:35 smithi033 ceph-mon[27845]: pgmap v13000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:35 smithi096 ceph-mon[32753]: pgmap v13000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:37 smithi194 ceph-mon[33017]: pgmap v13001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:37 smithi033 ceph-mon[27845]: pgmap v13001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:37 smithi096 ceph-mon[32753]: pgmap v13001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23617.057561 secs 2024-04-26T18:24:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23617.057561 secs 2024-04-26T18:24:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23617.057561 secs 2024-04-26T18:24:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:39 smithi033 ceph-mon[27845]: pgmap v13002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23619.931237 secs 2024-04-26T18:24:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:39 smithi194 ceph-mon[33017]: pgmap v13002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23619.931237 secs 2024-04-26T18:24:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:39 smithi096 ceph-mon[32753]: pgmap v13002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23619.931237 secs 2024-04-26T18:24:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:40 smithi033 ceph-mon[27845]: pgmap v13003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:40 smithi096 ceph-mon[32753]: pgmap v13003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:40 smithi194 ceph-mon[33017]: pgmap v13003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:42 smithi033 ceph-mon[27845]: pgmap v13004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:42 smithi096 ceph-mon[32753]: pgmap v13004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:42 smithi194 ceph-mon[33017]: pgmap v13004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23622.057617 secs 2024-04-26T18:24:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23624.931588 secs 2024-04-26T18:24:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23622.057617 secs 2024-04-26T18:24:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23624.931588 secs 2024-04-26T18:24:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23622.057617 secs 2024-04-26T18:24:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23624.931588 secs 2024-04-26T18:24:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:44 smithi033 ceph-mon[27845]: pgmap v13005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:44 smithi096 ceph-mon[32753]: pgmap v13005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:44 smithi194 ceph-mon[33017]: pgmap v13005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:24:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:46 smithi033 ceph-mon[27845]: pgmap v13006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:46 smithi096 ceph-mon[32753]: pgmap v13006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:46 smithi194 ceph-mon[33017]: pgmap v13006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:24:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23627.057673 secs 2024-04-26T18:24:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:48 smithi096 ceph-mon[32753]: pgmap v13007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23629.931669 secs 2024-04-26T18:24:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23627.057673 secs 2024-04-26T18:24:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:48 smithi194 ceph-mon[33017]: pgmap v13007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23629.931669 secs 2024-04-26T18:24:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23627.057673 secs 2024-04-26T18:24:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:48 smithi033 ceph-mon[27845]: pgmap v13007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23629.931669 secs 2024-04-26T18:24:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:50.165+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:50.165+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:24:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:24:50.165+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:24:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:50 smithi096 ceph-mon[32753]: pgmap v13008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:24:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:50 smithi194 ceph-mon[33017]: pgmap v13008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:24:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:50 smithi033 ceph-mon[27845]: pgmap v13008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:24:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:24:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:53 smithi194 ceph-mon[33017]: pgmap v13009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:24:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:24:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23632.057732 secs 2024-04-26T18:24:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:53 smithi033 ceph-mon[27845]: pgmap v13009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:24:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:24:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23632.057732 secs 2024-04-26T18:24:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:53 smithi096 ceph-mon[32753]: pgmap v13009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:24:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:24:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:24:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:24:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23632.057732 secs 2024-04-26T18:24:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:54 smithi194 ceph-mon[33017]: pgmap v13010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:24:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:54 smithi194 ceph-mon[33017]: pgmap v13011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:24:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23634.931764 secs 2024-04-26T18:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:54 smithi033 ceph-mon[27845]: pgmap v13010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:54 smithi033 ceph-mon[27845]: pgmap v13011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23634.931764 secs 2024-04-26T18:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:54 smithi096 ceph-mon[32753]: pgmap v13010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:54 smithi096 ceph-mon[32753]: pgmap v13011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23634.931764 secs 2024-04-26T18:24:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:56 smithi194 ceph-mon[33017]: pgmap v13012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T18:24:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:56 smithi033 ceph-mon[27845]: pgmap v13012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T18:24:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:56 smithi096 ceph-mon[32753]: pgmap v13012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T18:24:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:57 smithi194 ceph-mon[33017]: pgmap v13013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:24:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:57 smithi033 ceph-mon[27845]: pgmap v13013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:24:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:57 smithi096 ceph-mon[32753]: pgmap v13013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:24:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23637.057792 secs 2024-04-26T18:24:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23637.057792 secs 2024-04-26T18:24:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23637.057792 secs 2024-04-26T18:24:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:59 smithi194 ceph-mon[33017]: pgmap v13014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:24:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23639.931861 secs 2024-04-26T18:24:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:59 smithi033 ceph-mon[27845]: pgmap v13014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:24:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23639.931861 secs 2024-04-26T18:24:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:59 smithi096 ceph-mon[32753]: pgmap v13014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:24:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23639.931861 secs 2024-04-26T18:25:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:01 smithi194 ceph-mon[33017]: pgmap v13015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:25:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:01 smithi033 ceph-mon[27845]: pgmap v13015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:25:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:01 smithi096 ceph-mon[32753]: pgmap v13015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:25:03.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:03 smithi096 ceph-mon[32753]: pgmap v13016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:25:03.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23642.057853 secs 2024-04-26T18:25:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:03 smithi194 ceph-mon[33017]: pgmap v13016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:25:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23642.057853 secs 2024-04-26T18:25:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:03 smithi033 ceph-mon[27845]: pgmap v13016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:25:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23642.057853 secs 2024-04-26T18:25:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23644.931975 secs 2024-04-26T18:25:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23644.931975 secs 2024-04-26T18:25:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23644.931975 secs 2024-04-26T18:25:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:05.172+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:05.172+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:25:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:05.172+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:05 smithi194 ceph-mon[33017]: pgmap v13017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:05 smithi033 ceph-mon[27845]: pgmap v13017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:05 smithi096 ceph-mon[32753]: pgmap v13017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:07 smithi194 ceph-mon[33017]: pgmap v13018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:07 smithi033 ceph-mon[27845]: pgmap v13018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:07 smithi096 ceph-mon[32753]: pgmap v13018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23647.057914 secs 2024-04-26T18:25:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23647.057914 secs 2024-04-26T18:25:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23647.057914 secs 2024-04-26T18:25:09.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:09 smithi033 ceph-mon[27845]: pgmap v13019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:09.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23649.932050 secs 2024-04-26T18:25:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:09 smithi194 ceph-mon[33017]: pgmap v13019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23649.932050 secs 2024-04-26T18:25:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:09 smithi096 ceph-mon[32753]: pgmap v13019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23649.932050 secs 2024-04-26T18:25:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:11 smithi194 ceph-mon[33017]: pgmap v13020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:11 smithi033 ceph-mon[27845]: pgmap v13020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:11 smithi096 ceph-mon[32753]: pgmap v13020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:13 smithi194 ceph-mon[33017]: pgmap v13021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23652.057975 secs 2024-04-26T18:25:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:13 smithi033 ceph-mon[27845]: pgmap v13021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23652.057975 secs 2024-04-26T18:25:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:13 smithi096 ceph-mon[32753]: pgmap v13021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23652.057975 secs 2024-04-26T18:25:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23654.932143 secs 2024-04-26T18:25:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23654.932143 secs 2024-04-26T18:25:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23654.932143 secs 2024-04-26T18:25:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:15 smithi194 ceph-mon[33017]: pgmap v13022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:15 smithi033 ceph-mon[27845]: pgmap v13022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:15 smithi096 ceph-mon[32753]: pgmap v13022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:17 smithi194 ceph-mon[33017]: pgmap v13023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:17 smithi033 ceph-mon[27845]: pgmap v13023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:17 smithi096 ceph-mon[32753]: pgmap v13023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23657.058035 secs 2024-04-26T18:25:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23657.058035 secs 2024-04-26T18:25:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23657.058035 secs 2024-04-26T18:25:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:19 smithi194 ceph-mon[33017]: pgmap v13024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23659.932237 secs 2024-04-26T18:25:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:19 smithi033 ceph-mon[27845]: pgmap v13024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23659.932237 secs 2024-04-26T18:25:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:19 smithi096 ceph-mon[32753]: pgmap v13024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23659.932237 secs 2024-04-26T18:25:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:20.188+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:20.188+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:25:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:20.188+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:21 smithi033 ceph-mon[27845]: pgmap v13025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:21 smithi096 ceph-mon[32753]: pgmap v13025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:21 smithi194 ceph-mon[33017]: pgmap v13025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:23 smithi096 ceph-mon[32753]: pgmap v13026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23662.058095 secs 2024-04-26T18:25:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23664.932333 secs 2024-04-26T18:25:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:23 smithi194 ceph-mon[33017]: pgmap v13026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23662.058095 secs 2024-04-26T18:25:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23664.932333 secs 2024-04-26T18:25:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:23 smithi033 ceph-mon[27845]: pgmap v13026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23662.058095 secs 2024-04-26T18:25:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23664.932333 secs 2024-04-26T18:25:26.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:25 smithi033 ceph-mon[27845]: pgmap v13027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:25 smithi194 ceph-mon[33017]: pgmap v13027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:25 smithi096 ceph-mon[32753]: pgmap v13027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:27 smithi194 ceph-mon[33017]: pgmap v13028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:27 smithi033 ceph-mon[27845]: pgmap v13028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:27 smithi096 ceph-mon[32753]: pgmap v13028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23667.058155 secs 2024-04-26T18:25:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23669.932444 secs 2024-04-26T18:25:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23667.058155 secs 2024-04-26T18:25:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23669.932444 secs 2024-04-26T18:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23667.058155 secs 2024-04-26T18:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23669.932444 secs 2024-04-26T18:25:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:29 smithi194 ceph-mon[33017]: pgmap v13029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:29 smithi033 ceph-mon[27845]: pgmap v13029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:29 smithi096 ceph-mon[32753]: pgmap v13029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:31 smithi194 ceph-mon[33017]: pgmap v13030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:31 smithi033 ceph-mon[27845]: pgmap v13030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:31 smithi096 ceph-mon[32753]: pgmap v13030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:34.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:33 smithi096 ceph-mon[32753]: pgmap v13031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:34.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23672.058215 secs 2024-04-26T18:25:34.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23674.932557 secs 2024-04-26T18:25:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:33 smithi194 ceph-mon[33017]: pgmap v13031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23672.058215 secs 2024-04-26T18:25:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23674.932557 secs 2024-04-26T18:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:33 smithi033 ceph-mon[27845]: pgmap v13031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23672.058215 secs 2024-04-26T18:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23674.932557 secs 2024-04-26T18:25:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:35.200+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:35.200+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:35.200+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:25:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:35 smithi194 ceph-mon[33017]: pgmap v13032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:35 smithi033 ceph-mon[27845]: pgmap v13032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:35 smithi096 ceph-mon[32753]: pgmap v13032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:37 smithi194 ceph-mon[33017]: pgmap v13033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:37 smithi033 ceph-mon[27845]: pgmap v13033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:37 smithi096 ceph-mon[32753]: pgmap v13033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23677.058295 secs 2024-04-26T18:25:39.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23679.932673 secs 2024-04-26T18:25:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23677.058295 secs 2024-04-26T18:25:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23679.932673 secs 2024-04-26T18:25:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23677.058295 secs 2024-04-26T18:25:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23679.932673 secs 2024-04-26T18:25:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:39 smithi194 ceph-mon[33017]: pgmap v13034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:39 smithi033 ceph-mon[27845]: pgmap v13034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:39 smithi096 ceph-mon[32753]: pgmap v13034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:40.826+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T18:25:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:40.826+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:25:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:40.826+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 188 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T18:25:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:40.826+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 188 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:25:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:41 smithi194 ceph-mon[33017]: pgmap v13035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:41 smithi033 ceph-mon[27845]: pgmap v13035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:41 smithi096 ceph-mon[32753]: pgmap v13035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:43 smithi194 ceph-mon[33017]: pgmap v13036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23682.058357 secs 2024-04-26T18:25:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23684.932802 secs 2024-04-26T18:25:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:43 smithi033 ceph-mon[27845]: pgmap v13036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23682.058357 secs 2024-04-26T18:25:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23684.932802 secs 2024-04-26T18:25:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:43 smithi096 ceph-mon[32753]: pgmap v13036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23682.058357 secs 2024-04-26T18:25:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23684.932802 secs 2024-04-26T18:25:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:46 smithi194 ceph-mon[33017]: pgmap v13037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:46 smithi033 ceph-mon[27845]: pgmap v13037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:46 smithi096 ceph-mon[32753]: pgmap v13037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:48 smithi194 ceph-mon[33017]: pgmap v13038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:48 smithi033 ceph-mon[27845]: pgmap v13038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:48 smithi096 ceph-mon[32753]: pgmap v13038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:25:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23687.058441 secs 2024-04-26T18:25:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23689.932917 secs 2024-04-26T18:25:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23687.058441 secs 2024-04-26T18:25:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23689.932917 secs 2024-04-26T18:25:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23687.058441 secs 2024-04-26T18:25:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23689.932917 secs 2024-04-26T18:25:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:50.208+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:50.209+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:25:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:25:50.209+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:25:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:50 smithi194 ceph-mon[33017]: pgmap v13039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:50 smithi033 ceph-mon[27845]: pgmap v13039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:50 smithi096 ceph-mon[32753]: pgmap v13039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:52 smithi194 ceph-mon[33017]: pgmap v13040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:52 smithi033 ceph-mon[27845]: pgmap v13040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:52 smithi096 ceph-mon[32753]: pgmap v13040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:25:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:25:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23692.058510 secs 2024-04-26T18:25:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:25:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23692.058510 secs 2024-04-26T18:25:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:25:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23692.058510 secs 2024-04-26T18:25:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:54 smithi194 ceph-mon[33017]: pgmap v13041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23694.933012 secs 2024-04-26T18:25:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:54 smithi033 ceph-mon[27845]: pgmap v13041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23694.933012 secs 2024-04-26T18:25:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:54 smithi096 ceph-mon[32753]: pgmap v13041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23694.933012 secs 2024-04-26T18:25:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:25:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:25:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:55 smithi194 ceph-mon[33017]: pgmap v13042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:55 smithi194 ceph-mon[33017]: pgmap v13043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:25:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:25:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:25:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:25:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:55 smithi033 ceph-mon[27845]: pgmap v13042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:55 smithi033 ceph-mon[27845]: pgmap v13043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:25:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:25:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:25:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:25:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:55 smithi096 ceph-mon[32753]: pgmap v13042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:25:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:55 smithi096 ceph-mon[32753]: pgmap v13043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:25:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:25:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:56 smithi033 ceph-mon[27845]: pgmap v13044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:56 smithi096 ceph-mon[32753]: pgmap v13044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:56 smithi194 ceph-mon[33017]: pgmap v13044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23697.058591 secs 2024-04-26T18:25:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23697.058591 secs 2024-04-26T18:25:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23697.058591 secs 2024-04-26T18:25:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:59 smithi194 ceph-mon[33017]: pgmap v13045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:25:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23699.933133 secs 2024-04-26T18:25:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:59 smithi033 ceph-mon[27845]: pgmap v13045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:25:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23699.933133 secs 2024-04-26T18:25:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:59 smithi096 ceph-mon[32753]: pgmap v13045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:25:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:25:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23699.933133 secs 2024-04-26T18:26:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:01 smithi194 ceph-mon[33017]: pgmap v13046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:26:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:01 smithi033 ceph-mon[27845]: pgmap v13046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:26:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:01 smithi096 ceph-mon[32753]: pgmap v13046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:26:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:03 smithi194 ceph-mon[33017]: pgmap v13047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:26:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23702.058674 secs 2024-04-26T18:26:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:03 smithi033 ceph-mon[27845]: pgmap v13047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:26:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23702.058674 secs 2024-04-26T18:26:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:03 smithi096 ceph-mon[32753]: pgmap v13047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:26:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23702.058674 secs 2024-04-26T18:26:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23704.933252 secs 2024-04-26T18:26:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23704.933252 secs 2024-04-26T18:26:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23704.933252 secs 2024-04-26T18:26:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:05.223+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:05.223+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:05.223+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:26:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:05 smithi194 ceph-mon[33017]: pgmap v13048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:26:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:05 smithi033 ceph-mon[27845]: pgmap v13048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:26:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:05 smithi096 ceph-mon[32753]: pgmap v13048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:26:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:06 smithi033 ceph-mon[27845]: pgmap v13049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:06 smithi096 ceph-mon[32753]: pgmap v13049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:06 smithi194 ceph-mon[33017]: pgmap v13049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23707.058757 secs 2024-04-26T18:26:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23707.058757 secs 2024-04-26T18:26:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23707.058757 secs 2024-04-26T18:26:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:09 smithi033 ceph-mon[27845]: pgmap v13050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:09.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23709.933375 secs 2024-04-26T18:26:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:09 smithi194 ceph-mon[33017]: pgmap v13050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23709.933375 secs 2024-04-26T18:26:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:09 smithi096 ceph-mon[32753]: pgmap v13050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23709.933375 secs 2024-04-26T18:26:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:11 smithi194 ceph-mon[33017]: pgmap v13051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:11 smithi033 ceph-mon[27845]: pgmap v13051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:11 smithi096 ceph-mon[32753]: pgmap v13051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:13 smithi033 ceph-mon[27845]: pgmap v13052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23712.058842 secs 2024-04-26T18:26:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:13 smithi194 ceph-mon[33017]: pgmap v13052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23712.058842 secs 2024-04-26T18:26:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:13 smithi096 ceph-mon[32753]: pgmap v13052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23712.058842 secs 2024-04-26T18:26:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23714.933487 secs 2024-04-26T18:26:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23714.933487 secs 2024-04-26T18:26:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23714.933487 secs 2024-04-26T18:26:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:15 smithi194 ceph-mon[33017]: pgmap v13053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:15 smithi033 ceph-mon[27845]: pgmap v13053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:15 smithi096 ceph-mon[32753]: pgmap v13053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:16 smithi033 ceph-mon[27845]: pgmap v13054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:16 smithi096 ceph-mon[32753]: pgmap v13054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:16 smithi194 ceph-mon[33017]: pgmap v13054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23717.058928 secs 2024-04-26T18:26:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23717.058928 secs 2024-04-26T18:26:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23717.058928 secs 2024-04-26T18:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:19 smithi194 ceph-mon[33017]: pgmap v13055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23719.933600 secs 2024-04-26T18:26:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:19 smithi033 ceph-mon[27845]: pgmap v13055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23719.933600 secs 2024-04-26T18:26:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:19 smithi096 ceph-mon[32753]: pgmap v13055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23719.933600 secs 2024-04-26T18:26:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:20.240+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:20.240+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:20.240+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:26:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:20 smithi033 ceph-mon[27845]: pgmap v13056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:20 smithi096 ceph-mon[32753]: pgmap v13056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:20 smithi194 ceph-mon[33017]: pgmap v13056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:21.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T18:26:21.780 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T18:26:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0022040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:26:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0022040 comp rx=0 tx=0).stop 2024-04-26T18:26:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b31b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T18:26:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b31b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T18:26:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:26:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.780+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:26:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.781+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:26:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:21.783+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef007e000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:26:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:22 smithi033 ceph-mon[27845]: pgmap v13057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:22 smithi096 ceph-mon[32753]: pgmap v13057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:22 smithi194 ceph-mon[33017]: pgmap v13057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23722.059011 secs 2024-04-26T18:26:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23722.059011 secs 2024-04-26T18:26:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23722.059011 secs 2024-04-26T18:26:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23724.933727 secs 2024-04-26T18:26:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:24 smithi033 ceph-mon[27845]: pgmap v13058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23724.933727 secs 2024-04-26T18:26:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:24 smithi096 ceph-mon[32753]: pgmap v13058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23724.933727 secs 2024-04-26T18:26:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:24 smithi194 ceph-mon[33017]: pgmap v13058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:25.833+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:26:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:25.833+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:26:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:25.834+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 189 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:26:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:25.834+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 189 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:26:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:26 smithi033 ceph-mon[27845]: pgmap v13059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:26 smithi096 ceph-mon[32753]: pgmap v13059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:26 smithi194 ceph-mon[33017]: pgmap v13059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23727.059097 secs 2024-04-26T18:26:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23727.059097 secs 2024-04-26T18:26:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23727.059097 secs 2024-04-26T18:26:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:29 smithi194 ceph-mon[33017]: pgmap v13060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23729.933830 secs 2024-04-26T18:26:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:29 smithi033 ceph-mon[27845]: pgmap v13060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23729.933830 secs 2024-04-26T18:26:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:29 smithi096 ceph-mon[32753]: pgmap v13060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23729.933830 secs 2024-04-26T18:26:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:30 smithi033 ceph-mon[27845]: pgmap v13061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:30 smithi096 ceph-mon[32753]: pgmap v13061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:30 smithi194 ceph-mon[33017]: pgmap v13061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:32 smithi033 ceph-mon[27845]: pgmap v13062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:32 smithi096 ceph-mon[32753]: pgmap v13062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:32 smithi194 ceph-mon[33017]: pgmap v13062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23732.059177 secs 2024-04-26T18:26:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23732.059177 secs 2024-04-26T18:26:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23732.059177 secs 2024-04-26T18:26:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23734.933963 secs 2024-04-26T18:26:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:34 smithi033 ceph-mon[27845]: pgmap v13063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23734.933963 secs 2024-04-26T18:26:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:34 smithi096 ceph-mon[32753]: pgmap v13063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23734.933963 secs 2024-04-26T18:26:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:34 smithi194 ceph-mon[33017]: pgmap v13063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:35.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:35.248+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:35.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:35.248+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:35.248 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:35.248+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:26:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:36 smithi033 ceph-mon[27845]: pgmap v13064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:36 smithi096 ceph-mon[32753]: pgmap v13064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:36 smithi194 ceph-mon[33017]: pgmap v13064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23737.059235 secs 2024-04-26T18:26:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23737.059235 secs 2024-04-26T18:26:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23737.059235 secs 2024-04-26T18:26:39.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:39 smithi033 ceph-mon[27845]: pgmap v13065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:39.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23739.934089 secs 2024-04-26T18:26:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:39 smithi194 ceph-mon[33017]: pgmap v13065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23739.934089 secs 2024-04-26T18:26:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:39 smithi096 ceph-mon[32753]: pgmap v13065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23739.934089 secs 2024-04-26T18:26:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:40 smithi033 ceph-mon[27845]: pgmap v13066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:40 smithi096 ceph-mon[32753]: pgmap v13066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:40 smithi194 ceph-mon[33017]: pgmap v13066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:42 smithi033 ceph-mon[27845]: pgmap v13067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:42 smithi096 ceph-mon[32753]: pgmap v13067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:42 smithi194 ceph-mon[33017]: pgmap v13067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23742.059320 secs 2024-04-26T18:26:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23742.059320 secs 2024-04-26T18:26:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23742.059320 secs 2024-04-26T18:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23744.934225 secs 2024-04-26T18:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:44 smithi033 ceph-mon[27845]: pgmap v13068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23744.934225 secs 2024-04-26T18:26:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:44 smithi096 ceph-mon[32753]: pgmap v13068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23744.934225 secs 2024-04-26T18:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:44 smithi194 ceph-mon[33017]: pgmap v13068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:46 smithi033 ceph-mon[27845]: pgmap v13069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:46 smithi096 ceph-mon[32753]: pgmap v13069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:46 smithi194 ceph-mon[33017]: pgmap v13069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23747.059397 secs 2024-04-26T18:26:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23747.059397 secs 2024-04-26T18:26:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23747.059397 secs 2024-04-26T18:26:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:49 smithi194 ceph-mon[33017]: pgmap v13070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23749.934336 secs 2024-04-26T18:26:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:49 smithi033 ceph-mon[27845]: pgmap v13070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23749.934336 secs 2024-04-26T18:26:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:49 smithi096 ceph-mon[32753]: pgmap v13070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23749.934336 secs 2024-04-26T18:26:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:50.252+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:50.252+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:26:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:26:50.252+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:26:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:50 smithi033 ceph-mon[27845]: pgmap v13071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:50 smithi096 ceph-mon[32753]: pgmap v13071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:50 smithi194 ceph-mon[33017]: pgmap v13071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:52 smithi033 ceph-mon[27845]: pgmap v13072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:52 smithi096 ceph-mon[32753]: pgmap v13072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:52 smithi194 ceph-mon[33017]: pgmap v13072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23752.059468 secs 2024-04-26T18:26:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23752.059468 secs 2024-04-26T18:26:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23752.059468 secs 2024-04-26T18:26:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23754.934448 secs 2024-04-26T18:26:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:54 smithi096 ceph-mon[32753]: pgmap v13073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23754.934448 secs 2024-04-26T18:26:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:54 smithi194 ceph-mon[33017]: pgmap v13073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23754.934448 secs 2024-04-26T18:26:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:54 smithi033 ceph-mon[27845]: pgmap v13073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:26:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:26:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:26:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:56 smithi194 ceph-mon[33017]: pgmap v13074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:56 smithi033 ceph-mon[27845]: pgmap v13074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:56 smithi096 ceph-mon[32753]: pgmap v13074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:26:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23757.059554 secs 2024-04-26T18:26:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23757.059554 secs 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:26:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23757.059554 secs 2024-04-26T18:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:59 smithi194 ceph-mon[33017]: pgmap v13075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:26:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23759.934556 secs 2024-04-26T18:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:26:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:59 smithi033 ceph-mon[27845]: pgmap v13075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:26:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23759.934556 secs 2024-04-26T18:26:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:26:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:26:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:26:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:26:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:59 smithi096 ceph-mon[32753]: pgmap v13075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:26:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:26:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23759.934556 secs 2024-04-26T18:27:00.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:00.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: pgmap v13076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:27:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:27:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: pgmap v13076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:27:01.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: pgmap v13076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:27:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:27:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:27:01.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:27:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:02 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:27:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:02 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:27:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:02 smithi033 ceph-mon[27845]: pgmap v13077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:27:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:02 smithi033 ceph-mon[27845]: pgmap v13078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:02 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:27:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:02 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:27:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:02 smithi096 ceph-mon[32753]: pgmap v13077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:27:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:02 smithi096 ceph-mon[32753]: pgmap v13078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:02 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:27:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:02 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:27:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:02 smithi194 ceph-mon[33017]: pgmap v13077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:27:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:02 smithi194 ceph-mon[33017]: pgmap v13078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:03 smithi033 ceph-mon[27845]: pgmap v13079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23762.059635 secs 2024-04-26T18:27:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:03 smithi096 ceph-mon[32753]: pgmap v13079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23762.059635 secs 2024-04-26T18:27:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:03 smithi194 ceph-mon[33017]: pgmap v13079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23762.059635 secs 2024-04-26T18:27:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23764.934684 secs 2024-04-26T18:27:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23764.934684 secs 2024-04-26T18:27:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23764.934684 secs 2024-04-26T18:27:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:05.256+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:05.256+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:27:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:05.256+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:05 smithi033 ceph-mon[27845]: pgmap v13080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T18:27:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:05 smithi096 ceph-mon[32753]: pgmap v13080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T18:27:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:05 smithi194 ceph-mon[33017]: pgmap v13080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T18:27:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:07 smithi194 ceph-mon[33017]: pgmap v13081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:07 smithi033 ceph-mon[27845]: pgmap v13081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:07 smithi096 ceph-mon[32753]: pgmap v13081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:27:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23767.059724 secs 2024-04-26T18:27:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23767.059724 secs 2024-04-26T18:27:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23767.059724 secs 2024-04-26T18:27:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23769.934801 secs 2024-04-26T18:27:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:09 smithi194 ceph-mon[33017]: pgmap v13082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:27:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23769.934801 secs 2024-04-26T18:27:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:09 smithi033 ceph-mon[27845]: pgmap v13082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:27:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23769.934801 secs 2024-04-26T18:27:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:09 smithi096 ceph-mon[32753]: pgmap v13082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:27:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:11 smithi194 ceph-mon[33017]: pgmap v13083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:27:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:11 smithi033 ceph-mon[27845]: pgmap v13083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:27:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:11 smithi096 ceph-mon[32753]: pgmap v13083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:27:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:13 smithi194 ceph-mon[33017]: pgmap v13084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23772.059785 secs 2024-04-26T18:27:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:13 smithi033 ceph-mon[27845]: pgmap v13084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23772.059785 secs 2024-04-26T18:27:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:13 smithi096 ceph-mon[32753]: pgmap v13084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23772.059785 secs 2024-04-26T18:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23774.934919 secs 2024-04-26T18:27:14.863 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23774.934919 secs 2024-04-26T18:27:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23774.934919 secs 2024-04-26T18:27:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:15 smithi033 ceph-mon[27845]: pgmap v13085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:27:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:15 smithi096 ceph-mon[32753]: pgmap v13085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:27:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:15 smithi194 ceph-mon[33017]: pgmap v13085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:27:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:17 smithi033 ceph-mon[27845]: pgmap v13086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:27:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:17 smithi096 ceph-mon[32753]: pgmap v13086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:27:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:17 smithi194 ceph-mon[33017]: pgmap v13086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:27:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23777.059872 secs 2024-04-26T18:27:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23777.059872 secs 2024-04-26T18:27:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23777.059872 secs 2024-04-26T18:27:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23779.935060 secs 2024-04-26T18:27:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:19 smithi033 ceph-mon[27845]: pgmap v13087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23779.935060 secs 2024-04-26T18:27:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:19 smithi096 ceph-mon[32753]: pgmap v13087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23779.935060 secs 2024-04-26T18:27:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:19 smithi194 ceph-mon[33017]: pgmap v13087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:20.262+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:20.262+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:20.262+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:27:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:21 smithi033 ceph-mon[27845]: pgmap v13088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:21 smithi096 ceph-mon[32753]: pgmap v13088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:21 smithi194 ceph-mon[33017]: pgmap v13088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:23 smithi033 ceph-mon[27845]: pgmap v13089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23782.059956 secs 2024-04-26T18:27:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:23 smithi096 ceph-mon[32753]: pgmap v13089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23782.059956 secs 2024-04-26T18:27:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:23 smithi194 ceph-mon[33017]: pgmap v13089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23782.059956 secs 2024-04-26T18:27:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23784.935182 secs 2024-04-26T18:27:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23784.935182 secs 2024-04-26T18:27:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23784.935182 secs 2024-04-26T18:27:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:25 smithi033 ceph-mon[27845]: pgmap v13090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:25 smithi096 ceph-mon[32753]: pgmap v13090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:25 smithi194 ceph-mon[33017]: pgmap v13090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:27 smithi033 ceph-mon[27845]: pgmap v13091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:27:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:27 smithi096 ceph-mon[32753]: pgmap v13091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:27:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:27 smithi194 ceph-mon[33017]: pgmap v13091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:27:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23787.060015 secs 2024-04-26T18:27:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23789.935259 secs 2024-04-26T18:27:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23787.060015 secs 2024-04-26T18:27:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23789.935259 secs 2024-04-26T18:27:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23787.060015 secs 2024-04-26T18:27:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23789.935259 secs 2024-04-26T18:27:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:29 smithi096 ceph-mon[32753]: pgmap v13092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:29 smithi194 ceph-mon[33017]: pgmap v13092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:29 smithi033 ceph-mon[27845]: pgmap v13092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:31 smithi096 ceph-mon[32753]: pgmap v13093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:31 smithi194 ceph-mon[33017]: pgmap v13093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:32.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:31 smithi033 ceph-mon[27845]: pgmap v13093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:33 smithi194 ceph-mon[33017]: pgmap v13094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23792.060103 secs 2024-04-26T18:27:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23794.935338 secs 2024-04-26T18:27:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:33 smithi033 ceph-mon[27845]: pgmap v13094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23792.060103 secs 2024-04-26T18:27:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23794.935338 secs 2024-04-26T18:27:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:33 smithi096 ceph-mon[32753]: pgmap v13094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23792.060103 secs 2024-04-26T18:27:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23794.935338 secs 2024-04-26T18:27:35.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:35.275+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:35.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:35.275+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:27:35.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:35.275+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:35.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:35 smithi096 ceph-mon[32753]: pgmap v13095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:35 smithi194 ceph-mon[33017]: pgmap v13095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:35 smithi033 ceph-mon[27845]: pgmap v13095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:37 smithi194 ceph-mon[33017]: pgmap v13096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:37 smithi033 ceph-mon[27845]: pgmap v13096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:37 smithi096 ceph-mon[32753]: pgmap v13096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:39.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23797.060192 secs 2024-04-26T18:27:39.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23799.935422 secs 2024-04-26T18:27:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23797.060192 secs 2024-04-26T18:27:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23799.935422 secs 2024-04-26T18:27:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23797.060192 secs 2024-04-26T18:27:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23799.935422 secs 2024-04-26T18:27:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:39 smithi194 ceph-mon[33017]: pgmap v13097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:39 smithi033 ceph-mon[27845]: pgmap v13097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:39 smithi096 ceph-mon[32753]: pgmap v13097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:41 smithi194 ceph-mon[33017]: pgmap v13098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:41 smithi033 ceph-mon[27845]: pgmap v13098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:41 smithi096 ceph-mon[32753]: pgmap v13098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:43 smithi194 ceph-mon[33017]: pgmap v13099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23802.060282 secs 2024-04-26T18:27:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23804.935541 secs 2024-04-26T18:27:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:43 smithi033 ceph-mon[27845]: pgmap v13099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23802.060282 secs 2024-04-26T18:27:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23804.935541 secs 2024-04-26T18:27:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:43 smithi096 ceph-mon[32753]: pgmap v13099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23802.060282 secs 2024-04-26T18:27:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23804.935541 secs 2024-04-26T18:27:46.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:45 smithi033 ceph-mon[27845]: pgmap v13100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:45 smithi194 ceph-mon[33017]: pgmap v13100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:45 smithi096 ceph-mon[32753]: pgmap v13100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:47 smithi194 ceph-mon[33017]: pgmap v13101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:47 smithi033 ceph-mon[27845]: pgmap v13101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:47 smithi096 ceph-mon[32753]: pgmap v13101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23807.060365 secs 2024-04-26T18:27:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23809.935614 secs 2024-04-26T18:27:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23807.060365 secs 2024-04-26T18:27:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23809.935614 secs 2024-04-26T18:27:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23807.060365 secs 2024-04-26T18:27:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23809.935614 secs 2024-04-26T18:27:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:49 smithi194 ceph-mon[33017]: pgmap v13102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:49 smithi033 ceph-mon[27845]: pgmap v13102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:49 smithi096 ceph-mon[32753]: pgmap v13102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:50.287+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:50.287+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:27:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:27:50.287+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:27:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:51 smithi194 ceph-mon[33017]: pgmap v13103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:51 smithi033 ceph-mon[27845]: pgmap v13103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:51 smithi096 ceph-mon[32753]: pgmap v13103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:53 smithi194 ceph-mon[33017]: pgmap v13104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23812.060453 secs 2024-04-26T18:27:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23814.935750 secs 2024-04-26T18:27:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:53 smithi033 ceph-mon[27845]: pgmap v13104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23812.060453 secs 2024-04-26T18:27:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23814.935750 secs 2024-04-26T18:27:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:53 smithi096 ceph-mon[32753]: pgmap v13104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23812.060453 secs 2024-04-26T18:27:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23814.935750 secs 2024-04-26T18:27:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:55 smithi194 ceph-mon[33017]: pgmap v13105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:55 smithi033 ceph-mon[27845]: pgmap v13105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:55 smithi096 ceph-mon[32753]: pgmap v13105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:27:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:57 smithi194 ceph-mon[33017]: pgmap v13106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:57 smithi033 ceph-mon[27845]: pgmap v13106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:57 smithi096 ceph-mon[32753]: pgmap v13106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:27:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23817.060538 secs 2024-04-26T18:27:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23819.935868 secs 2024-04-26T18:27:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23817.060538 secs 2024-04-26T18:27:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23819.935868 secs 2024-04-26T18:27:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23817.060538 secs 2024-04-26T18:27:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23819.935868 secs 2024-04-26T18:28:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:27:59 smithi194 ceph-mon[33017]: pgmap v13107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:27:59 smithi033 ceph-mon[27845]: pgmap v13107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:27:59 smithi096 ceph-mon[32753]: pgmap v13107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:01 smithi194 ceph-mon[33017]: pgmap v13108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:28:02.160 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:01 smithi096 ceph-mon[32753]: pgmap v13108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:02.160 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:28:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:01 smithi033 ceph-mon[27845]: pgmap v13108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:28:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:03 smithi194 ceph-mon[33017]: pgmap v13109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23822.060628 secs 2024-04-26T18:28:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23824.935945 secs 2024-04-26T18:28:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:03 smithi033 ceph-mon[27845]: pgmap v13109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23822.060628 secs 2024-04-26T18:28:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23824.935945 secs 2024-04-26T18:28:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:03 smithi096 ceph-mon[32753]: pgmap v13109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23822.060628 secs 2024-04-26T18:28:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23824.935945 secs 2024-04-26T18:28:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:05.288+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:05.288+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:05.288+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:28:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:06 smithi033 ceph-mon[27845]: pgmap v13110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:06 smithi096 ceph-mon[32753]: pgmap v13110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:06 smithi194 ceph-mon[33017]: pgmap v13110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:07 smithi033 ceph-mon[27845]: pgmap v13111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:07 smithi096 ceph-mon[32753]: pgmap v13111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:07 smithi194 ceph-mon[33017]: pgmap v13111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23827.060709 secs 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: pgmap v13112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: pgmap v13113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23829.936068 secs 2024-04-26T18:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23827.060709 secs 2024-04-26T18:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: pgmap v13112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T18:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: pgmap v13113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23829.936068 secs 2024-04-26T18:28:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23827.060709 secs 2024-04-26T18:28:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:28:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:28:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: pgmap v13112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T18:28:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: pgmap v13113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:28:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23829.936068 secs 2024-04-26T18:28:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:11 smithi033 ceph-mon[27845]: pgmap v13114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:11 smithi194 ceph-mon[33017]: pgmap v13114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:11 smithi096 ceph-mon[32753]: pgmap v13114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:13 smithi194 ceph-mon[33017]: pgmap v13115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:13 smithi033 ceph-mon[27845]: pgmap v13115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:13 smithi096 ceph-mon[32753]: pgmap v13115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23832.060793 secs 2024-04-26T18:28:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23834.936205 secs 2024-04-26T18:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23832.060793 secs 2024-04-26T18:28:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23834.936205 secs 2024-04-26T18:28:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23832.060793 secs 2024-04-26T18:28:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23834.936205 secs 2024-04-26T18:28:15.315 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:15 smithi096 ceph-mon[32753]: pgmap v13116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T18:28:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:15 smithi194 ceph-mon[33017]: pgmap v13116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T18:28:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:15 smithi033 ceph-mon[27845]: pgmap v13116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T18:28:17.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:17 smithi033 ceph-mon[27845]: pgmap v13117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:17 smithi194 ceph-mon[33017]: pgmap v13117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:17 smithi096 ceph-mon[32753]: pgmap v13117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:28:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23837.060874 secs 2024-04-26T18:28:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23837.060874 secs 2024-04-26T18:28:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23837.060874 secs 2024-04-26T18:28:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:19 smithi194 ceph-mon[33017]: pgmap v13118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:28:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23839.936288 secs 2024-04-26T18:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:19 smithi033 ceph-mon[27845]: pgmap v13118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23839.936288 secs 2024-04-26T18:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:19 smithi096 ceph-mon[32753]: pgmap v13118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23839.936288 secs 2024-04-26T18:28:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:20.304+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:20.304+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:28:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:20.304+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:21 smithi194 ceph-mon[33017]: pgmap v13119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:21 smithi033 ceph-mon[27845]: pgmap v13119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:21 smithi096 ceph-mon[32753]: pgmap v13119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:23 smithi194 ceph-mon[33017]: pgmap v13120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23842.060951 secs 2024-04-26T18:28:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:23 smithi033 ceph-mon[27845]: pgmap v13120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23842.060951 secs 2024-04-26T18:28:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:23 smithi096 ceph-mon[32753]: pgmap v13120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23842.060951 secs 2024-04-26T18:28:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23844.936353 secs 2024-04-26T18:28:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23844.936353 secs 2024-04-26T18:28:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23844.936353 secs 2024-04-26T18:28:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:25 smithi194 ceph-mon[33017]: pgmap v13121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:25 smithi033 ceph-mon[27845]: pgmap v13121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:25 smithi096 ceph-mon[32753]: pgmap v13121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:27 smithi194 ceph-mon[33017]: pgmap v13122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:27 smithi033 ceph-mon[27845]: pgmap v13122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:27 smithi096 ceph-mon[32753]: pgmap v13122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23847.061040 secs 2024-04-26T18:28:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23847.061040 secs 2024-04-26T18:28:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23847.061040 secs 2024-04-26T18:28:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:29 smithi194 ceph-mon[33017]: pgmap v13123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23849.936463 secs 2024-04-26T18:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:29 smithi033 ceph-mon[27845]: pgmap v13123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23849.936463 secs 2024-04-26T18:28:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:29 smithi096 ceph-mon[32753]: pgmap v13123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23849.936463 secs 2024-04-26T18:28:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:31 smithi194 ceph-mon[33017]: pgmap v13124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:31 smithi033 ceph-mon[27845]: pgmap v13124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:31 smithi096 ceph-mon[32753]: pgmap v13124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:33 smithi033 ceph-mon[27845]: pgmap v13125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23852.061127 secs 2024-04-26T18:28:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:33 smithi096 ceph-mon[32753]: pgmap v13125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23852.061127 secs 2024-04-26T18:28:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:33 smithi194 ceph-mon[33017]: pgmap v13125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23852.061127 secs 2024-04-26T18:28:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23854.936586 secs 2024-04-26T18:28:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23854.936586 secs 2024-04-26T18:28:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23854.936586 secs 2024-04-26T18:28:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:35.317+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:35.317+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:35.317+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:28:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:35 smithi033 ceph-mon[27845]: pgmap v13126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:35 smithi096 ceph-mon[32753]: pgmap v13126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:35 smithi194 ceph-mon[33017]: pgmap v13126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:36 smithi033 ceph-mon[27845]: pgmap v13127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:36 smithi096 ceph-mon[32753]: pgmap v13127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:36 smithi194 ceph-mon[33017]: pgmap v13127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23857.061187 secs 2024-04-26T18:28:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:39 smithi194 ceph-mon[33017]: pgmap v13128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23859.936707 secs 2024-04-26T18:28:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23857.061187 secs 2024-04-26T18:28:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:39 smithi033 ceph-mon[27845]: pgmap v13128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23859.936707 secs 2024-04-26T18:28:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23857.061187 secs 2024-04-26T18:28:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:39 smithi096 ceph-mon[32753]: pgmap v13128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23859.936707 secs 2024-04-26T18:28:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:41 smithi194 ceph-mon[33017]: pgmap v13129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:41 smithi033 ceph-mon[27845]: pgmap v13129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:41 smithi096 ceph-mon[32753]: pgmap v13129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:43 smithi194 ceph-mon[33017]: pgmap v13130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23862.061268 secs 2024-04-26T18:28:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:43 smithi033 ceph-mon[27845]: pgmap v13130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23862.061268 secs 2024-04-26T18:28:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:43 smithi096 ceph-mon[32753]: pgmap v13130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23862.061268 secs 2024-04-26T18:28:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23864.936830 secs 2024-04-26T18:28:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23864.936830 secs 2024-04-26T18:28:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23864.936830 secs 2024-04-26T18:28:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:45 smithi194 ceph-mon[33017]: pgmap v13131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:45 smithi033 ceph-mon[27845]: pgmap v13131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:45 smithi096 ceph-mon[32753]: pgmap v13131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:47 smithi194 ceph-mon[33017]: pgmap v13132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:47 smithi033 ceph-mon[27845]: pgmap v13132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:47 smithi096 ceph-mon[32753]: pgmap v13132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23867.061354 secs 2024-04-26T18:28:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23867.061354 secs 2024-04-26T18:28:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23867.061354 secs 2024-04-26T18:28:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:49 smithi194 ceph-mon[33017]: pgmap v13133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23869.936947 secs 2024-04-26T18:28:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:49 smithi033 ceph-mon[27845]: pgmap v13133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23869.936947 secs 2024-04-26T18:28:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:49 smithi096 ceph-mon[32753]: pgmap v13133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23869.936947 secs 2024-04-26T18:28:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:50.333+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:50.333+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:28:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:28:50.333+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:28:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:51 smithi194 ceph-mon[33017]: pgmap v13134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:51 smithi033 ceph-mon[27845]: pgmap v13134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:51 smithi096 ceph-mon[32753]: pgmap v13134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:53 smithi194 ceph-mon[33017]: pgmap v13135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23872.061413 secs 2024-04-26T18:28:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:53 smithi033 ceph-mon[27845]: pgmap v13135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23872.061413 secs 2024-04-26T18:28:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:53 smithi096 ceph-mon[32753]: pgmap v13135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23872.061413 secs 2024-04-26T18:28:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23874.937070 secs 2024-04-26T18:28:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23874.937070 secs 2024-04-26T18:28:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23874.937070 secs 2024-04-26T18:28:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:55 smithi194 ceph-mon[33017]: pgmap v13136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:55 smithi033 ceph-mon[27845]: pgmap v13136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:55 smithi096 ceph-mon[32753]: pgmap v13136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:57 smithi194 ceph-mon[33017]: pgmap v13137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:57 smithi033 ceph-mon[27845]: pgmap v13137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:57 smithi096 ceph-mon[32753]: pgmap v13137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:28:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23877.061475 secs 2024-04-26T18:28:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23877.061475 secs 2024-04-26T18:28:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23877.061475 secs 2024-04-26T18:28:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:59 smithi033 ceph-mon[27845]: pgmap v13138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:28:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23879.937180 secs 2024-04-26T18:28:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:59 smithi096 ceph-mon[32753]: pgmap v13138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:28:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23879.937180 secs 2024-04-26T18:28:59.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:59 smithi194 ceph-mon[33017]: pgmap v13138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:28:59.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:28:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23879.937180 secs 2024-04-26T18:29:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:01 smithi033 ceph-mon[27845]: pgmap v13139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:01 smithi096 ceph-mon[32753]: pgmap v13139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:01 smithi194 ceph-mon[33017]: pgmap v13139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:03 smithi033 ceph-mon[27845]: pgmap v13140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23882.061561 secs 2024-04-26T18:29:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:03 smithi096 ceph-mon[32753]: pgmap v13140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23882.061561 secs 2024-04-26T18:29:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:03 smithi194 ceph-mon[33017]: pgmap v13140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23882.061561 secs 2024-04-26T18:29:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23884.937308 secs 2024-04-26T18:29:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23884.937308 secs 2024-04-26T18:29:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23884.937308 secs 2024-04-26T18:29:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:05 smithi033 ceph-mon[27845]: pgmap v13141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:05.347+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:05.347+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:05.347+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:29:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:05 smithi096 ceph-mon[32753]: pgmap v13141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:05 smithi194 ceph-mon[33017]: pgmap v13141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:06 smithi033 ceph-mon[27845]: pgmap v13142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:06 smithi096 ceph-mon[32753]: pgmap v13142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:06 smithi194 ceph-mon[33017]: pgmap v13142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23887.061665 secs 2024-04-26T18:29:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23887.061665 secs 2024-04-26T18:29:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23887.061665 secs 2024-04-26T18:29:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:09 smithi194 ceph-mon[33017]: pgmap v13143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:29:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23889.937375 secs 2024-04-26T18:29:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:09 smithi033 ceph-mon[27845]: pgmap v13143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:29:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23889.937375 secs 2024-04-26T18:29:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:09 smithi096 ceph-mon[32753]: pgmap v13143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:29:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23889.937375 secs 2024-04-26T18:29:10.346 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:29:10.346 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:29:10.346 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:29:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:29:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:29:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:29:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:29:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:29:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:29:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:11 smithi194 ceph-mon[33017]: pgmap v13144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T18:29:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:11 smithi194 ceph-mon[33017]: pgmap v13145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:11.368 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:11 smithi033 ceph-mon[27845]: pgmap v13144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T18:29:11.369 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:11 smithi033 ceph-mon[27845]: pgmap v13145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:11 smithi096 ceph-mon[32753]: pgmap v13144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T18:29:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:11 smithi096 ceph-mon[32753]: pgmap v13145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:13 smithi033 ceph-mon[27845]: pgmap v13146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23892.061763 secs 2024-04-26T18:29:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:13 smithi096 ceph-mon[32753]: pgmap v13146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23892.061763 secs 2024-04-26T18:29:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:13 smithi194 ceph-mon[33017]: pgmap v13146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23892.061763 secs 2024-04-26T18:29:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23894.937526 secs 2024-04-26T18:29:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23894.937526 secs 2024-04-26T18:29:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23894.937526 secs 2024-04-26T18:29:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:15 smithi033 ceph-mon[27845]: pgmap v13147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:15 smithi096 ceph-mon[32753]: pgmap v13147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:15 smithi194 ceph-mon[33017]: pgmap v13147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:17 smithi033 ceph-mon[27845]: pgmap v13148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:17 smithi096 ceph-mon[32753]: pgmap v13148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:17 smithi194 ceph-mon[33017]: pgmap v13148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23897.061865 secs 2024-04-26T18:29:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23897.061865 secs 2024-04-26T18:29:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23897.061865 secs 2024-04-26T18:29:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:19 smithi033 ceph-mon[27845]: pgmap v13149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23899.937594 secs 2024-04-26T18:29:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:19 smithi096 ceph-mon[32753]: pgmap v13149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23899.937594 secs 2024-04-26T18:29:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:19 smithi194 ceph-mon[33017]: pgmap v13149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:29:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23899.937594 secs 2024-04-26T18:29:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:20.363+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:20.363+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:20.363+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:29:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:20 smithi033 ceph-mon[27845]: pgmap v13150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:29:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:20 smithi096 ceph-mon[32753]: pgmap v13150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:29:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:20 smithi194 ceph-mon[33017]: pgmap v13150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:29:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:23 smithi194 ceph-mon[33017]: pgmap v13151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:23 smithi033 ceph-mon[27845]: pgmap v13151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:23 smithi096 ceph-mon[32753]: pgmap v13151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23902.061962 secs 2024-04-26T18:29:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23904.937732 secs 2024-04-26T18:29:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23902.061962 secs 2024-04-26T18:29:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23904.937732 secs 2024-04-26T18:29:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23902.061962 secs 2024-04-26T18:29:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23904.937732 secs 2024-04-26T18:29:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:25 smithi194 ceph-mon[33017]: pgmap v13152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:25 smithi033 ceph-mon[27845]: pgmap v13152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:25 smithi096 ceph-mon[32753]: pgmap v13152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:25.863+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:29:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:25.863+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:29:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:25.863+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 190 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:29:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:25.864+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 190 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T18:29:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:27 smithi194 ceph-mon[33017]: pgmap v13153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:27 smithi033 ceph-mon[27845]: pgmap v13153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:27 smithi096 ceph-mon[32753]: pgmap v13153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23907.062060 secs 2024-04-26T18:29:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23907.062060 secs 2024-04-26T18:29:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23907.062060 secs 2024-04-26T18:29:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:29 smithi194 ceph-mon[33017]: pgmap v13154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23909.937848 secs 2024-04-26T18:29:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:29 smithi033 ceph-mon[27845]: pgmap v13154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23909.937848 secs 2024-04-26T18:29:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:29 smithi096 ceph-mon[32753]: pgmap v13154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23909.937848 secs 2024-04-26T18:29:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:31 smithi194 ceph-mon[33017]: pgmap v13155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:31 smithi033 ceph-mon[27845]: pgmap v13155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:31 smithi096 ceph-mon[32753]: pgmap v13155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:33 smithi194 ceph-mon[33017]: pgmap v13156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23912.062143 secs 2024-04-26T18:29:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:33 smithi033 ceph-mon[27845]: pgmap v13156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23912.062143 secs 2024-04-26T18:29:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:33 smithi096 ceph-mon[32753]: pgmap v13156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23912.062143 secs 2024-04-26T18:29:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23914.937959 secs 2024-04-26T18:29:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23914.937959 secs 2024-04-26T18:29:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23914.937959 secs 2024-04-26T18:29:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:35.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:35.371+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:35.371+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:29:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:35 smithi033 ceph-mon[27845]: pgmap v13157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:35 smithi096 ceph-mon[32753]: pgmap v13157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:35 smithi194 ceph-mon[33017]: pgmap v13157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:37 smithi033 ceph-mon[27845]: pgmap v13158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:37 smithi096 ceph-mon[32753]: pgmap v13158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:37 smithi194 ceph-mon[33017]: pgmap v13158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23917.062246 secs 2024-04-26T18:29:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23917.062246 secs 2024-04-26T18:29:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23917.062246 secs 2024-04-26T18:29:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:39 smithi033 ceph-mon[27845]: pgmap v13159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23919.938084 secs 2024-04-26T18:29:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:39 smithi096 ceph-mon[32753]: pgmap v13159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23919.938084 secs 2024-04-26T18:29:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:39 smithi194 ceph-mon[33017]: pgmap v13159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23919.938084 secs 2024-04-26T18:29:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:40 smithi194 ceph-mon[33017]: pgmap v13160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:40 smithi033 ceph-mon[27845]: pgmap v13160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:40 smithi096 ceph-mon[32753]: pgmap v13160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:42 smithi194 ceph-mon[33017]: pgmap v13161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:42 smithi033 ceph-mon[27845]: pgmap v13161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:42 smithi096 ceph-mon[32753]: pgmap v13161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23922.062351 secs 2024-04-26T18:29:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23924.938204 secs 2024-04-26T18:29:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23922.062351 secs 2024-04-26T18:29:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23924.938204 secs 2024-04-26T18:29:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23922.062351 secs 2024-04-26T18:29:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23924.938204 secs 2024-04-26T18:29:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:44 smithi194 ceph-mon[33017]: pgmap v13162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:44 smithi033 ceph-mon[27845]: pgmap v13162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:44 smithi096 ceph-mon[32753]: pgmap v13162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:47.034 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:46 smithi096 ceph-mon[32753]: pgmap v13163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:46 smithi194 ceph-mon[33017]: pgmap v13163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:46 smithi033 ceph-mon[27845]: pgmap v13163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:48 smithi194 ceph-mon[33017]: pgmap v13164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23927.062412 secs 2024-04-26T18:29:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23929.938321 secs 2024-04-26T18:29:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:48 smithi033 ceph-mon[27845]: pgmap v13164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23927.062412 secs 2024-04-26T18:29:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23929.938321 secs 2024-04-26T18:29:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:48 smithi096 ceph-mon[32753]: pgmap v13164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23927.062412 secs 2024-04-26T18:29:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23929.938321 secs 2024-04-26T18:29:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:50.387+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:50.387+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:29:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:29:50.387+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:29:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:50 smithi194 ceph-mon[33017]: pgmap v13165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:50 smithi033 ceph-mon[27845]: pgmap v13165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:50 smithi096 ceph-mon[32753]: pgmap v13165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:52 smithi194 ceph-mon[33017]: pgmap v13166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:52 smithi033 ceph-mon[27845]: pgmap v13166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:52 smithi096 ceph-mon[32753]: pgmap v13166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23932.062469 secs 2024-04-26T18:29:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23934.938418 secs 2024-04-26T18:29:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23932.062469 secs 2024-04-26T18:29:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23934.938418 secs 2024-04-26T18:29:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23932.062469 secs 2024-04-26T18:29:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23934.938418 secs 2024-04-26T18:29:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:54 smithi194 ceph-mon[33017]: pgmap v13167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:54 smithi033 ceph-mon[27845]: pgmap v13167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:54 smithi096 ceph-mon[32753]: pgmap v13167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:56 smithi194 ceph-mon[33017]: pgmap v13168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:56 smithi033 ceph-mon[27845]: pgmap v13168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:56 smithi096 ceph-mon[32753]: pgmap v13168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:29:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:58 smithi194 ceph-mon[33017]: pgmap v13169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23937.062530 secs 2024-04-26T18:29:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:29:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23939.938539 secs 2024-04-26T18:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:58 smithi033 ceph-mon[27845]: pgmap v13169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23937.062530 secs 2024-04-26T18:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:29:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23939.938539 secs 2024-04-26T18:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:58 smithi096 ceph-mon[32753]: pgmap v13169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23937.062530 secs 2024-04-26T18:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:29:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23939.938539 secs 2024-04-26T18:30:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:00 smithi194 ceph-mon[33017]: pgmap v13170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:00 smithi033 ceph-mon[27845]: pgmap v13170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:30:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:00 smithi096 ceph-mon[32753]: pgmap v13170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:30:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:02 smithi194 ceph-mon[33017]: pgmap v13171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:02 smithi033 ceph-mon[27845]: pgmap v13171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:02 smithi096 ceph-mon[32753]: pgmap v13171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23942.062614 secs 2024-04-26T18:30:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23944.938658 secs 2024-04-26T18:30:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23942.062614 secs 2024-04-26T18:30:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23944.938658 secs 2024-04-26T18:30:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23942.062614 secs 2024-04-26T18:30:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23944.938658 secs 2024-04-26T18:30:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:04 smithi096 ceph-mon[32753]: pgmap v13172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:04 smithi194 ceph-mon[33017]: pgmap v13172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:04 smithi033 ceph-mon[27845]: pgmap v13172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:05.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:05.401+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:05.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:05.401+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:05.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:05.401+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:30:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:06 smithi194 ceph-mon[33017]: pgmap v13173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:06 smithi033 ceph-mon[27845]: pgmap v13173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:06 smithi096 ceph-mon[32753]: pgmap v13173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:08 smithi194 ceph-mon[33017]: pgmap v13174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23947.062698 secs 2024-04-26T18:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23949.938793 secs 2024-04-26T18:30:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:08 smithi033 ceph-mon[27845]: pgmap v13174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23947.062698 secs 2024-04-26T18:30:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23949.938793 secs 2024-04-26T18:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:08 smithi096 ceph-mon[32753]: pgmap v13174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23947.062698 secs 2024-04-26T18:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23949.938793 secs 2024-04-26T18:30:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:10 smithi194 ceph-mon[33017]: pgmap v13175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:30:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:10 smithi033 ceph-mon[27845]: pgmap v13175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:30:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:10 smithi096 ceph-mon[32753]: pgmap v13175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:30:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:30:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:30:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:30:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:30:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:30:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:30:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:30:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:30:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:30:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:12 smithi194 ceph-mon[33017]: pgmap v13176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:12 smithi194 ceph-mon[33017]: pgmap v13177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:12 smithi033 ceph-mon[27845]: pgmap v13176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:12 smithi033 ceph-mon[27845]: pgmap v13177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:12 smithi096 ceph-mon[32753]: pgmap v13176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:12 smithi096 ceph-mon[32753]: pgmap v13177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23952.062755 secs 2024-04-26T18:30:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23954.938929 secs 2024-04-26T18:30:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23952.062755 secs 2024-04-26T18:30:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23954.938929 secs 2024-04-26T18:30:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23952.062755 secs 2024-04-26T18:30:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23954.938929 secs 2024-04-26T18:30:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:14 smithi194 ceph-mon[33017]: pgmap v13178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:14 smithi033 ceph-mon[27845]: pgmap v13178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:14 smithi096 ceph-mon[32753]: pgmap v13178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:16 smithi194 ceph-mon[33017]: pgmap v13179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:16 smithi033 ceph-mon[27845]: pgmap v13179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:16 smithi096 ceph-mon[32753]: pgmap v13179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:18 smithi194 ceph-mon[33017]: pgmap v13180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23957.062838 secs 2024-04-26T18:30:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23959.939079 secs 2024-04-26T18:30:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:18 smithi033 ceph-mon[27845]: pgmap v13180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23957.062838 secs 2024-04-26T18:30:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23959.939079 secs 2024-04-26T18:30:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:18 smithi096 ceph-mon[32753]: pgmap v13180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:30:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23957.062838 secs 2024-04-26T18:30:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23959.939079 secs 2024-04-26T18:30:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:20.416+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:20.416+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:20.416+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:30:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:20 smithi194 ceph-mon[33017]: pgmap v13181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:30:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:20 smithi033 ceph-mon[27845]: pgmap v13181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:30:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:20 smithi096 ceph-mon[32753]: pgmap v13181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:30:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:22 smithi033 ceph-mon[27845]: pgmap v13182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:30:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:22 smithi096 ceph-mon[32753]: pgmap v13182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:30:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:22 smithi194 ceph-mon[33017]: pgmap v13182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:30:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23962.062921 secs 2024-04-26T18:30:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23964.939175 secs 2024-04-26T18:30:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23962.062921 secs 2024-04-26T18:30:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23964.939175 secs 2024-04-26T18:30:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23962.062921 secs 2024-04-26T18:30:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23964.939175 secs 2024-04-26T18:30:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:24 smithi033 ceph-mon[27845]: pgmap v13183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:24 smithi096 ceph-mon[32753]: pgmap v13183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:24 smithi194 ceph-mon[33017]: pgmap v13183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:26 smithi033 ceph-mon[27845]: pgmap v13184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:26 smithi096 ceph-mon[32753]: pgmap v13184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:26 smithi194 ceph-mon[33017]: pgmap v13184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:28 smithi033 ceph-mon[27845]: pgmap v13185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23967.063025 secs 2024-04-26T18:30:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23969.939291 secs 2024-04-26T18:30:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:28 smithi096 ceph-mon[32753]: pgmap v13185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23967.063025 secs 2024-04-26T18:30:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23969.939291 secs 2024-04-26T18:30:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:28 smithi194 ceph-mon[33017]: pgmap v13185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23967.063025 secs 2024-04-26T18:30:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23969.939291 secs 2024-04-26T18:30:31.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:30 smithi194 ceph-mon[33017]: pgmap v13186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:30 smithi033 ceph-mon[27845]: pgmap v13186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:30 smithi096 ceph-mon[32753]: pgmap v13186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:32 smithi033 ceph-mon[27845]: pgmap v13187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:32 smithi096 ceph-mon[32753]: pgmap v13187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:32 smithi194 ceph-mon[33017]: pgmap v13187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23972.063108 secs 2024-04-26T18:30:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23974.939409 secs 2024-04-26T18:30:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23972.063108 secs 2024-04-26T18:30:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23974.939409 secs 2024-04-26T18:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23972.063108 secs 2024-04-26T18:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23974.939409 secs 2024-04-26T18:30:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:34 smithi033 ceph-mon[27845]: pgmap v13188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:34 smithi096 ceph-mon[32753]: pgmap v13188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:34 smithi194 ceph-mon[33017]: pgmap v13188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:35.430+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:35.430+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:35.430+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:30:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:36 smithi033 ceph-mon[27845]: pgmap v13189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:36 smithi096 ceph-mon[32753]: pgmap v13189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:36 smithi194 ceph-mon[33017]: pgmap v13189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:38 smithi033 ceph-mon[27845]: pgmap v13190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23977.063170 secs 2024-04-26T18:30:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23979.939529 secs 2024-04-26T18:30:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:38 smithi096 ceph-mon[32753]: pgmap v13190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23977.063170 secs 2024-04-26T18:30:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23979.939529 secs 2024-04-26T18:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:38 smithi194 ceph-mon[33017]: pgmap v13190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23977.063170 secs 2024-04-26T18:30:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23979.939529 secs 2024-04-26T18:30:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:40 smithi033 ceph-mon[27845]: pgmap v13191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:40 smithi096 ceph-mon[32753]: pgmap v13191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:40 smithi194 ceph-mon[33017]: pgmap v13191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:42 smithi033 ceph-mon[27845]: pgmap v13192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:42 smithi096 ceph-mon[32753]: pgmap v13192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:42 smithi194 ceph-mon[33017]: pgmap v13192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23982.063277 secs 2024-04-26T18:30:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23984.939648 secs 2024-04-26T18:30:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23982.063277 secs 2024-04-26T18:30:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23984.939648 secs 2024-04-26T18:30:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23982.063277 secs 2024-04-26T18:30:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23984.939648 secs 2024-04-26T18:30:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:44 smithi194 ceph-mon[33017]: pgmap v13193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:44 smithi033 ceph-mon[27845]: pgmap v13193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:44 smithi096 ceph-mon[32753]: pgmap v13193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:46 smithi194 ceph-mon[33017]: pgmap v13194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:46 smithi033 ceph-mon[27845]: pgmap v13194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:46 smithi096 ceph-mon[32753]: pgmap v13194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:48 smithi194 ceph-mon[33017]: pgmap v13195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23987.063365 secs 2024-04-26T18:30:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23989.939764 secs 2024-04-26T18:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:48 smithi033 ceph-mon[27845]: pgmap v13195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23987.063365 secs 2024-04-26T18:30:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23989.939764 secs 2024-04-26T18:30:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:48 smithi096 ceph-mon[32753]: pgmap v13195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23987.063365 secs 2024-04-26T18:30:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23989.939764 secs 2024-04-26T18:30:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:50.438+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:50.438+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:30:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:30:50.438+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:30:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:50 smithi194 ceph-mon[33017]: pgmap v13196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:50 smithi033 ceph-mon[27845]: pgmap v13196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:50 smithi096 ceph-mon[32753]: pgmap v13196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:30:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:52 smithi194 ceph-mon[33017]: pgmap v13197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:52 smithi033 ceph-mon[27845]: pgmap v13197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:52 smithi096 ceph-mon[32753]: pgmap v13197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23992.063434 secs 2024-04-26T18:30:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23994.939915 secs 2024-04-26T18:30:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23992.063434 secs 2024-04-26T18:30:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23994.939915 secs 2024-04-26T18:30:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23992.063434 secs 2024-04-26T18:30:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23994.939915 secs 2024-04-26T18:30:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:54 smithi194 ceph-mon[33017]: pgmap v13198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:54 smithi033 ceph-mon[27845]: pgmap v13198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:54 smithi096 ceph-mon[32753]: pgmap v13198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:30:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:56 smithi194 ceph-mon[33017]: pgmap v13199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:30:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:56 smithi033 ceph-mon[27845]: pgmap v13199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:30:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:56 smithi096 ceph-mon[32753]: pgmap v13199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:30:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:58 smithi194 ceph-mon[33017]: pgmap v13200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:30:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 23997.063515 secs 2024-04-26T18:30:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:30:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 23999.940026 secs 2024-04-26T18:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:58 smithi033 ceph-mon[27845]: pgmap v13200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 23997.063515 secs 2024-04-26T18:30:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:30:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 23999.940026 secs 2024-04-26T18:30:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:58 smithi096 ceph-mon[32753]: pgmap v13200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:30:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 23997.063515 secs 2024-04-26T18:30:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:30:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 23999.940026 secs 2024-04-26T18:31:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:00 smithi194 ceph-mon[33017]: pgmap v13201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:31:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:00 smithi033 ceph-mon[27845]: pgmap v13201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:31:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:00 smithi096 ceph-mon[32753]: pgmap v13201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:31:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:02 smithi194 ceph-mon[33017]: pgmap v13202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:31:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:02 smithi033 ceph-mon[27845]: pgmap v13202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:31:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:02 smithi096 ceph-mon[32753]: pgmap v13202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:31:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24002.063587 secs 2024-04-26T18:31:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24004.940143 secs 2024-04-26T18:31:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24002.063587 secs 2024-04-26T18:31:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24004.940143 secs 2024-04-26T18:31:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24002.063587 secs 2024-04-26T18:31:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24004.940143 secs 2024-04-26T18:31:05.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:05 smithi096 ceph-mon[32753]: pgmap v13203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:31:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:05 smithi194 ceph-mon[33017]: pgmap v13203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:31:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:05 smithi033 ceph-mon[27845]: pgmap v13203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:31:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:05.453+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:05.453+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:05.453+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:31:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:07 smithi194 ceph-mon[33017]: pgmap v13204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:31:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:07 smithi033 ceph-mon[27845]: pgmap v13204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:31:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:07 smithi096 ceph-mon[32753]: pgmap v13204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:31:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:09 smithi033 ceph-mon[27845]: pgmap v13205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24007.063688 secs 2024-04-26T18:31:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24009.940240 secs 2024-04-26T18:31:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:09 smithi194 ceph-mon[33017]: pgmap v13205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24007.063688 secs 2024-04-26T18:31:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24009.940240 secs 2024-04-26T18:31:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:09 smithi096 ceph-mon[32753]: pgmap v13205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24007.063688 secs 2024-04-26T18:31:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24009.940240 secs 2024-04-26T18:31:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:11 smithi194 ceph-mon[33017]: pgmap v13206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:11 smithi033 ceph-mon[27845]: pgmap v13206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:11 smithi096 ceph-mon[32753]: pgmap v13206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:31:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:31:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:31:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:13 smithi194 ceph-mon[33017]: pgmap v13207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:31:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:31:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:31:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:13 smithi033 ceph-mon[27845]: pgmap v13207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:31:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:31:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:31:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:13 smithi096 ceph-mon[32753]: pgmap v13207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:31:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:31:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:31:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24012.063786 secs 2024-04-26T18:31:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:14 smithi194 ceph-mon[33017]: pgmap v13208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:14 smithi194 ceph-mon[33017]: pgmap v13209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24014.940333 secs 2024-04-26T18:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24012.063786 secs 2024-04-26T18:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:14 smithi033 ceph-mon[27845]: pgmap v13208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:14 smithi033 ceph-mon[27845]: pgmap v13209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24014.940333 secs 2024-04-26T18:31:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24012.063786 secs 2024-04-26T18:31:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:14 smithi096 ceph-mon[32753]: pgmap v13208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:14 smithi096 ceph-mon[32753]: pgmap v13209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24014.940333 secs 2024-04-26T18:31:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:16 smithi194 ceph-mon[33017]: pgmap v13210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:16 smithi033 ceph-mon[27845]: pgmap v13210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:16 smithi096 ceph-mon[32753]: pgmap v13210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:18 smithi194 ceph-mon[33017]: pgmap v13211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24017.063889 secs 2024-04-26T18:31:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:18 smithi033 ceph-mon[27845]: pgmap v13211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24017.063889 secs 2024-04-26T18:31:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:18 smithi096 ceph-mon[32753]: pgmap v13211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24017.063889 secs 2024-04-26T18:31:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24019.940451 secs 2024-04-26T18:31:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24019.940451 secs 2024-04-26T18:31:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24019.940451 secs 2024-04-26T18:31:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:20 smithi194 ceph-mon[33017]: pgmap v13212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:31:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:20 smithi033 ceph-mon[27845]: pgmap v13212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:31:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:20 smithi096 ceph-mon[32753]: pgmap v13212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:31:20.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:20.469+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:20.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:20.469+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:20.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:20.469+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:31:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:22 smithi194 ceph-mon[33017]: pgmap v13213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:22 smithi033 ceph-mon[27845]: pgmap v13213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:22 smithi096 ceph-mon[32753]: pgmap v13213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:31:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24022.064012 secs 2024-04-26T18:31:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24022.064012 secs 2024-04-26T18:31:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24022.064012 secs 2024-04-26T18:31:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:24 smithi194 ceph-mon[33017]: pgmap v13214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:31:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24024.940562 secs 2024-04-26T18:31:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:24 smithi033 ceph-mon[27845]: pgmap v13214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:31:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24024.940562 secs 2024-04-26T18:31:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:24 smithi096 ceph-mon[32753]: pgmap v13214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:31:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24024.940562 secs 2024-04-26T18:31:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:26 smithi194 ceph-mon[33017]: pgmap v13215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:26 smithi033 ceph-mon[27845]: pgmap v13215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:26 smithi096 ceph-mon[32753]: pgmap v13215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:28 smithi194 ceph-mon[33017]: pgmap v13216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24027.064101 secs 2024-04-26T18:31:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:28 smithi033 ceph-mon[27845]: pgmap v13216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24027.064101 secs 2024-04-26T18:31:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:28 smithi096 ceph-mon[32753]: pgmap v13216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24027.064101 secs 2024-04-26T18:31:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24029.940681 secs 2024-04-26T18:31:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24029.940681 secs 2024-04-26T18:31:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24029.940681 secs 2024-04-26T18:31:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:30 smithi033 ceph-mon[27845]: pgmap v13217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:30 smithi096 ceph-mon[32753]: pgmap v13217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:30 smithi194 ceph-mon[33017]: pgmap v13217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:32.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:32 smithi194 ceph-mon[33017]: pgmap v13218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:32 smithi033 ceph-mon[27845]: pgmap v13218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:32 smithi096 ceph-mon[32753]: pgmap v13218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24032.064184 secs 2024-04-26T18:31:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24032.064184 secs 2024-04-26T18:31:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24032.064184 secs 2024-04-26T18:31:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:34 smithi033 ceph-mon[27845]: pgmap v13219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24034.940801 secs 2024-04-26T18:31:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:34 smithi096 ceph-mon[32753]: pgmap v13219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24034.940801 secs 2024-04-26T18:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:34 smithi194 ceph-mon[33017]: pgmap v13219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24034.940801 secs 2024-04-26T18:31:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:35.485+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:35.485+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:35.485+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:31:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:36 smithi033 ceph-mon[27845]: pgmap v13220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:36 smithi096 ceph-mon[32753]: pgmap v13220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:36 smithi194 ceph-mon[33017]: pgmap v13220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:38 smithi033 ceph-mon[27845]: pgmap v13221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24037.064288 secs 2024-04-26T18:31:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:38 smithi096 ceph-mon[32753]: pgmap v13221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:38.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24037.064288 secs 2024-04-26T18:31:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:38 smithi194 ceph-mon[33017]: pgmap v13221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24037.064288 secs 2024-04-26T18:31:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24039.940862 secs 2024-04-26T18:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24039.940862 secs 2024-04-26T18:31:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24039.940862 secs 2024-04-26T18:31:40.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:40 smithi033 ceph-mon[27845]: pgmap v13222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:40 smithi096 ceph-mon[32753]: pgmap v13222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:40 smithi194 ceph-mon[33017]: pgmap v13222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:42 smithi033 ceph-mon[27845]: pgmap v13223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:42 smithi096 ceph-mon[32753]: pgmap v13223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:42 smithi194 ceph-mon[33017]: pgmap v13223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24042.064372 secs 2024-04-26T18:31:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24042.064372 secs 2024-04-26T18:31:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24042.064372 secs 2024-04-26T18:31:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:44 smithi033 ceph-mon[27845]: pgmap v13224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24044.940996 secs 2024-04-26T18:31:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:44 smithi096 ceph-mon[32753]: pgmap v13224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24044.940996 secs 2024-04-26T18:31:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:44 smithi194 ceph-mon[33017]: pgmap v13224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24044.940996 secs 2024-04-26T18:31:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:45 smithi194 ceph-mon[33017]: pgmap v13225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:45 smithi033 ceph-mon[27845]: pgmap v13225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:45 smithi096 ceph-mon[32753]: pgmap v13225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:48 smithi194 ceph-mon[33017]: pgmap v13226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:48 smithi033 ceph-mon[27845]: pgmap v13226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:48 smithi096 ceph-mon[32753]: pgmap v13226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24047.064456 secs 2024-04-26T18:31:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24049.941140 secs 2024-04-26T18:31:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24047.064456 secs 2024-04-26T18:31:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24049.941140 secs 2024-04-26T18:31:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24047.064456 secs 2024-04-26T18:31:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24049.941140 secs 2024-04-26T18:31:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:50 smithi194 ceph-mon[33017]: pgmap v13227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:50 smithi033 ceph-mon[27845]: pgmap v13227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:50 smithi096 ceph-mon[32753]: pgmap v13227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:50.500+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:50.501+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:31:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:31:50.501+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:31:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:52 smithi194 ceph-mon[33017]: pgmap v13228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:52 smithi033 ceph-mon[27845]: pgmap v13228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:52 smithi096 ceph-mon[32753]: pgmap v13228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24052.064542 secs 2024-04-26T18:31:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24052.064542 secs 2024-04-26T18:31:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24052.064542 secs 2024-04-26T18:31:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:54 smithi194 ceph-mon[33017]: pgmap v13229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24054.941272 secs 2024-04-26T18:31:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:54 smithi033 ceph-mon[27845]: pgmap v13229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24054.941272 secs 2024-04-26T18:31:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:54 smithi096 ceph-mon[32753]: pgmap v13229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24054.941272 secs 2024-04-26T18:31:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:56 smithi194 ceph-mon[33017]: pgmap v13230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:56 smithi033 ceph-mon[27845]: pgmap v13230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:56 smithi096 ceph-mon[32753]: pgmap v13230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:31:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:58 smithi194 ceph-mon[33017]: pgmap v13231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24057.064601 secs 2024-04-26T18:31:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:58 smithi033 ceph-mon[27845]: pgmap v13231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24057.064601 secs 2024-04-26T18:31:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:58 smithi096 ceph-mon[32753]: pgmap v13231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:31:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24057.064601 secs 2024-04-26T18:31:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24059.941401 secs 2024-04-26T18:31:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24059.941401 secs 2024-04-26T18:31:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24059.941401 secs 2024-04-26T18:32:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:00 smithi194 ceph-mon[33017]: pgmap v13232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:00 smithi033 ceph-mon[27845]: pgmap v13232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:00 smithi096 ceph-mon[32753]: pgmap v13232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:02 smithi033 ceph-mon[27845]: pgmap v13233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:02 smithi096 ceph-mon[32753]: pgmap v13233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:02 smithi194 ceph-mon[33017]: pgmap v13233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24062.064689 secs 2024-04-26T18:32:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24062.064689 secs 2024-04-26T18:32:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24062.064689 secs 2024-04-26T18:32:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:04 smithi033 ceph-mon[27845]: pgmap v13234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24064.941525 secs 2024-04-26T18:32:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:04 smithi096 ceph-mon[32753]: pgmap v13234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24064.941525 secs 2024-04-26T18:32:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:04 smithi194 ceph-mon[33017]: pgmap v13234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24064.941525 secs 2024-04-26T18:32:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:05.514+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:05.514+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:05.514+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:32:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:06 smithi033 ceph-mon[27845]: pgmap v13235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:06 smithi096 ceph-mon[32753]: pgmap v13235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:06 smithi194 ceph-mon[33017]: pgmap v13235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:08 smithi033 ceph-mon[27845]: pgmap v13236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24067.064769 secs 2024-04-26T18:32:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:08 smithi096 ceph-mon[32753]: pgmap v13236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24067.064769 secs 2024-04-26T18:32:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:08 smithi194 ceph-mon[33017]: pgmap v13236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24067.064769 secs 2024-04-26T18:32:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24069.941644 secs 2024-04-26T18:32:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24069.941644 secs 2024-04-26T18:32:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24069.941644 secs 2024-04-26T18:32:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:10 smithi033 ceph-mon[27845]: pgmap v13237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:10 smithi096 ceph-mon[32753]: pgmap v13237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:10 smithi194 ceph-mon[33017]: pgmap v13237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:12 smithi033 ceph-mon[27845]: pgmap v13238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:12 smithi096 ceph-mon[32753]: pgmap v13238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:12 smithi194 ceph-mon[33017]: pgmap v13238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24072.064850 secs 2024-04-26T18:32:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24072.064850 secs 2024-04-26T18:32:13.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24072.064850 secs 2024-04-26T18:32:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:14 smithi033 ceph-mon[27845]: pgmap v13239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:32:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24074.941732 secs 2024-04-26T18:32:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:14 smithi096 ceph-mon[32753]: pgmap v13239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:32:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24074.941732 secs 2024-04-26T18:32:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:14 smithi194 ceph-mon[33017]: pgmap v13239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:32:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24074.941732 secs 2024-04-26T18:32:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:32:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:32:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:32:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:32:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:32:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:32:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:32:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:32:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:32:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:16 smithi033 ceph-mon[27845]: pgmap v13240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:32:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:16 smithi033 ceph-mon[27845]: pgmap v13241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:16 smithi096 ceph-mon[32753]: pgmap v13240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:32:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:16 smithi096 ceph-mon[32753]: pgmap v13241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:16 smithi194 ceph-mon[33017]: pgmap v13240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:32:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:16 smithi194 ceph-mon[33017]: pgmap v13241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:17 smithi096 ceph-mon[32753]: pgmap v13242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:17 smithi194 ceph-mon[33017]: pgmap v13242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:17 smithi033 ceph-mon[27845]: pgmap v13242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24077.064932 secs 2024-04-26T18:32:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24077.064932 secs 2024-04-26T18:32:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24077.064932 secs 2024-04-26T18:32:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24079.941860 secs 2024-04-26T18:32:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:19 smithi096 ceph-mon[32753]: pgmap v13243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24079.941860 secs 2024-04-26T18:32:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:19 smithi194 ceph-mon[33017]: pgmap v13243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24079.941860 secs 2024-04-26T18:32:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:19 smithi033 ceph-mon[27845]: pgmap v13243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:20.519+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:20.519+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:20.519+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:32:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:21 smithi194 ceph-mon[33017]: pgmap v13244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:21 smithi033 ceph-mon[27845]: pgmap v13244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:21 smithi096 ceph-mon[32753]: pgmap v13244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:23 smithi194 ceph-mon[33017]: pgmap v13245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24082.065026 secs 2024-04-26T18:32:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24084.942002 secs 2024-04-26T18:32:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:23 smithi033 ceph-mon[27845]: pgmap v13245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24082.065026 secs 2024-04-26T18:32:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24084.942002 secs 2024-04-26T18:32:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:23 smithi096 ceph-mon[32753]: pgmap v13245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:32:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24082.065026 secs 2024-04-26T18:32:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24084.942002 secs 2024-04-26T18:32:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:25.893+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:32:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:25.893+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:32:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:25.893+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 191 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:32:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:25.893+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 191 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:32:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:25 smithi194 ceph-mon[33017]: pgmap v13246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:32:26.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:25 smithi033 ceph-mon[27845]: pgmap v13246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:32:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:25 smithi096 ceph-mon[32753]: pgmap v13246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:32:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:27 smithi194 ceph-mon[33017]: pgmap v13247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:28.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:27 smithi033 ceph-mon[27845]: pgmap v13247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:27 smithi096 ceph-mon[32753]: pgmap v13247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24087.065086 secs 2024-04-26T18:32:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24089.942137 secs 2024-04-26T18:32:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24087.065086 secs 2024-04-26T18:32:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24089.942137 secs 2024-04-26T18:32:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24087.065086 secs 2024-04-26T18:32:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24089.942137 secs 2024-04-26T18:32:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:29 smithi194 ceph-mon[33017]: pgmap v13248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:29 smithi033 ceph-mon[27845]: pgmap v13248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:29 smithi096 ceph-mon[32753]: pgmap v13248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:31 smithi194 ceph-mon[33017]: pgmap v13249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:31 smithi033 ceph-mon[27845]: pgmap v13249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:31 smithi096 ceph-mon[32753]: pgmap v13249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:33 smithi194 ceph-mon[33017]: pgmap v13250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24092.065173 secs 2024-04-26T18:32:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24094.942260 secs 2024-04-26T18:32:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:33 smithi033 ceph-mon[27845]: pgmap v13250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24092.065173 secs 2024-04-26T18:32:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24094.942260 secs 2024-04-26T18:32:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:33 smithi096 ceph-mon[32753]: pgmap v13250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24092.065173 secs 2024-04-26T18:32:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24094.942260 secs 2024-04-26T18:32:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:35.532+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:35.532+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:35.532+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:32:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:35 smithi194 ceph-mon[33017]: pgmap v13251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:35 smithi033 ceph-mon[27845]: pgmap v13251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:35 smithi096 ceph-mon[32753]: pgmap v13251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:37 smithi194 ceph-mon[33017]: pgmap v13252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:37 smithi033 ceph-mon[27845]: pgmap v13252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:37 smithi096 ceph-mon[32753]: pgmap v13252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24097.065278 secs 2024-04-26T18:32:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24099.942389 secs 2024-04-26T18:32:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24097.065278 secs 2024-04-26T18:32:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24099.942389 secs 2024-04-26T18:32:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24097.065278 secs 2024-04-26T18:32:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24099.942389 secs 2024-04-26T18:32:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:39 smithi194 ceph-mon[33017]: pgmap v13253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:39 smithi033 ceph-mon[27845]: pgmap v13253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:39 smithi096 ceph-mon[32753]: pgmap v13253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:42.038 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:41 smithi033 ceph-mon[27845]: pgmap v13254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:41 smithi194 ceph-mon[33017]: pgmap v13254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:41 smithi096 ceph-mon[32753]: pgmap v13254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:43 smithi194 ceph-mon[33017]: pgmap v13255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24102.065366 secs 2024-04-26T18:32:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24104.942514 secs 2024-04-26T18:32:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:43 smithi033 ceph-mon[27845]: pgmap v13255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24102.065366 secs 2024-04-26T18:32:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24104.942514 secs 2024-04-26T18:32:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:43 smithi096 ceph-mon[32753]: pgmap v13255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24102.065366 secs 2024-04-26T18:32:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24104.942514 secs 2024-04-26T18:32:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:45 smithi194 ceph-mon[33017]: pgmap v13256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:45 smithi033 ceph-mon[27845]: pgmap v13256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:45 smithi096 ceph-mon[32753]: pgmap v13256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:47 smithi194 ceph-mon[33017]: pgmap v13257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:47 smithi033 ceph-mon[27845]: pgmap v13257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:47 smithi096 ceph-mon[32753]: pgmap v13257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24107.065451 secs 2024-04-26T18:32:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24109.942639 secs 2024-04-26T18:32:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24107.065451 secs 2024-04-26T18:32:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24109.942639 secs 2024-04-26T18:32:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24107.065451 secs 2024-04-26T18:32:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24109.942639 secs 2024-04-26T18:32:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:49 smithi194 ceph-mon[33017]: pgmap v13258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:49 smithi033 ceph-mon[27845]: pgmap v13258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:49 smithi096 ceph-mon[32753]: pgmap v13258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:50.540 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:50.540+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:50.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:50.540+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:32:50.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:32:50.540+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:32:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:51 smithi194 ceph-mon[33017]: pgmap v13259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:51 smithi033 ceph-mon[27845]: pgmap v13259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:51 smithi096 ceph-mon[32753]: pgmap v13259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:53 smithi194 ceph-mon[33017]: pgmap v13260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24112.065539 secs 2024-04-26T18:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24114.942734 secs 2024-04-26T18:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:53 smithi033 ceph-mon[27845]: pgmap v13260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24112.065539 secs 2024-04-26T18:32:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24114.942734 secs 2024-04-26T18:32:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:53 smithi096 ceph-mon[32753]: pgmap v13260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24112.065539 secs 2024-04-26T18:32:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24114.942734 secs 2024-04-26T18:32:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:55 smithi194 ceph-mon[33017]: pgmap v13261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:55 smithi033 ceph-mon[27845]: pgmap v13261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:55 smithi096 ceph-mon[32753]: pgmap v13261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:32:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:57 smithi194 ceph-mon[33017]: pgmap v13262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:57 smithi033 ceph-mon[27845]: pgmap v13262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:57 smithi096 ceph-mon[32753]: pgmap v13262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:32:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24117.065619 secs 2024-04-26T18:32:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24119.942883 secs 2024-04-26T18:32:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24117.065619 secs 2024-04-26T18:32:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24119.942883 secs 2024-04-26T18:32:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24117.065619 secs 2024-04-26T18:32:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24119.942883 secs 2024-04-26T18:33:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:32:59 smithi194 ceph-mon[33017]: pgmap v13263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:32:59 smithi033 ceph-mon[27845]: pgmap v13263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:32:59 smithi096 ceph-mon[32753]: pgmap v13263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:01 smithi194 ceph-mon[33017]: pgmap v13264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:01 smithi033 ceph-mon[27845]: pgmap v13264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:01 smithi096 ceph-mon[32753]: pgmap v13264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:03 smithi033 ceph-mon[27845]: pgmap v13265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24122.065679 secs 2024-04-26T18:33:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24124.943028 secs 2024-04-26T18:33:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:03 smithi096 ceph-mon[32753]: pgmap v13265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24122.065679 secs 2024-04-26T18:33:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24124.943028 secs 2024-04-26T18:33:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:03 smithi194 ceph-mon[33017]: pgmap v13265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24122.065679 secs 2024-04-26T18:33:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24124.943028 secs 2024-04-26T18:33:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:05.554+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:05.554+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:05.554+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:33:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:05 smithi033 ceph-mon[27845]: pgmap v13266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:05 smithi096 ceph-mon[32753]: pgmap v13266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:05 smithi194 ceph-mon[33017]: pgmap v13266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:07 smithi033 ceph-mon[27845]: pgmap v13267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:07 smithi096 ceph-mon[32753]: pgmap v13267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:07 smithi194 ceph-mon[33017]: pgmap v13267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24127.065738 secs 2024-04-26T18:33:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24129.943152 secs 2024-04-26T18:33:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24127.065738 secs 2024-04-26T18:33:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24129.943152 secs 2024-04-26T18:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24127.065738 secs 2024-04-26T18:33:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24129.943152 secs 2024-04-26T18:33:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:09 smithi033 ceph-mon[27845]: pgmap v13268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:09 smithi096 ceph-mon[32753]: pgmap v13268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:09 smithi194 ceph-mon[33017]: pgmap v13268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:11 smithi033 ceph-mon[27845]: pgmap v13269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:11 smithi096 ceph-mon[32753]: pgmap v13269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:11 smithi194 ceph-mon[33017]: pgmap v13269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:13 smithi033 ceph-mon[27845]: pgmap v13270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24132.065797 secs 2024-04-26T18:33:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24134.943277 secs 2024-04-26T18:33:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:13 smithi096 ceph-mon[32753]: pgmap v13270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24132.065797 secs 2024-04-26T18:33:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24134.943277 secs 2024-04-26T18:33:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:13 smithi194 ceph-mon[33017]: pgmap v13270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24132.065797 secs 2024-04-26T18:33:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24134.943277 secs 2024-04-26T18:33:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:15 smithi033 ceph-mon[27845]: pgmap v13271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:33:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:15 smithi096 ceph-mon[32753]: pgmap v13271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:33:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:15 smithi194 ceph-mon[33017]: pgmap v13271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:33:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: pgmap v13272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:33:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:33:17.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: pgmap v13272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:33:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:33:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: pgmap v13272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:33:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:33:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:33:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:18 smithi194 ceph-mon[33017]: pgmap v13273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:33:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:18 smithi194 ceph-mon[33017]: pgmap v13274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24137.065859 secs 2024-04-26T18:33:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:18 smithi033 ceph-mon[27845]: pgmap v13273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:33:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:18 smithi033 ceph-mon[27845]: pgmap v13274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24137.065859 secs 2024-04-26T18:33:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:18 smithi096 ceph-mon[32753]: pgmap v13273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:33:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:18 smithi096 ceph-mon[32753]: pgmap v13274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24137.065859 secs 2024-04-26T18:33:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24139.943399 secs 2024-04-26T18:33:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24139.943399 secs 2024-04-26T18:33:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24139.943399 secs 2024-04-26T18:33:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:20.569+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:20.569+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:33:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:20.569+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:20 smithi194 ceph-mon[33017]: pgmap v13275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:33:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:20 smithi033 ceph-mon[27845]: pgmap v13275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:33:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:20 smithi096 ceph-mon[32753]: pgmap v13275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:33:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:23 smithi033 ceph-mon[27845]: pgmap v13276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24142.065918 secs 2024-04-26T18:33:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:23 smithi096 ceph-mon[32753]: pgmap v13276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24142.065918 secs 2024-04-26T18:33:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:23 smithi194 ceph-mon[33017]: pgmap v13276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24142.065918 secs 2024-04-26T18:33:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24144.943524 secs 2024-04-26T18:33:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24144.943524 secs 2024-04-26T18:33:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24144.943524 secs 2024-04-26T18:33:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:25 smithi033 ceph-mon[27845]: pgmap v13277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:33:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:25 smithi096 ceph-mon[32753]: pgmap v13277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:33:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:25 smithi194 ceph-mon[33017]: pgmap v13277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:33:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:27 smithi033 ceph-mon[27845]: pgmap v13278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:27 smithi096 ceph-mon[32753]: pgmap v13278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:27 smithi194 ceph-mon[33017]: pgmap v13278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:33:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24147.065979 secs 2024-04-26T18:33:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24147.065979 secs 2024-04-26T18:33:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24147.065979 secs 2024-04-26T18:33:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:29 smithi033 ceph-mon[27845]: pgmap v13279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:33:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24149.943649 secs 2024-04-26T18:33:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:29 smithi096 ceph-mon[32753]: pgmap v13279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:33:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24149.943649 secs 2024-04-26T18:33:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:29 smithi194 ceph-mon[33017]: pgmap v13279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:33:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24149.943649 secs 2024-04-26T18:33:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:31 smithi033 ceph-mon[27845]: pgmap v13280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:31 smithi096 ceph-mon[32753]: pgmap v13280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:31 smithi194 ceph-mon[33017]: pgmap v13280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:33.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:33 smithi194 ceph-mon[33017]: pgmap v13281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:33.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24152.066035 secs 2024-04-26T18:33:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:33 smithi033 ceph-mon[27845]: pgmap v13281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24152.066035 secs 2024-04-26T18:33:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:33 smithi096 ceph-mon[32753]: pgmap v13281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24152.066035 secs 2024-04-26T18:33:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24154.943804 secs 2024-04-26T18:33:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24154.943804 secs 2024-04-26T18:33:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24154.943804 secs 2024-04-26T18:33:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:35 smithi033 ceph-mon[27845]: pgmap v13282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:35 smithi096 ceph-mon[32753]: pgmap v13282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:35.573+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:35.573+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:33:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:35.573+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:35 smithi194 ceph-mon[33017]: pgmap v13282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:37 smithi033 ceph-mon[27845]: pgmap v13283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:37 smithi096 ceph-mon[32753]: pgmap v13283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:37 smithi194 ceph-mon[33017]: pgmap v13283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24157.066094 secs 2024-04-26T18:33:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24157.066094 secs 2024-04-26T18:33:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24157.066094 secs 2024-04-26T18:33:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:39 smithi194 ceph-mon[33017]: pgmap v13284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24159.943932 secs 2024-04-26T18:33:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:39 smithi033 ceph-mon[27845]: pgmap v13284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24159.943932 secs 2024-04-26T18:33:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:39 smithi096 ceph-mon[32753]: pgmap v13284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24159.943932 secs 2024-04-26T18:33:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:40 smithi194 ceph-mon[33017]: pgmap v13285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:40 smithi033 ceph-mon[27845]: pgmap v13285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:40 smithi096 ceph-mon[32753]: pgmap v13285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:42 smithi194 ceph-mon[33017]: pgmap v13286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:42 smithi033 ceph-mon[27845]: pgmap v13286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:42 smithi096 ceph-mon[32753]: pgmap v13286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:43.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24162.066178 secs 2024-04-26T18:33:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24162.066178 secs 2024-04-26T18:33:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24162.066178 secs 2024-04-26T18:33:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:44 smithi033 ceph-mon[27845]: pgmap v13287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24164.944072 secs 2024-04-26T18:33:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:44 smithi194 ceph-mon[33017]: pgmap v13287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24164.944072 secs 2024-04-26T18:33:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:44 smithi096 ceph-mon[32753]: pgmap v13287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24164.944072 secs 2024-04-26T18:33:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:46 smithi194 ceph-mon[33017]: pgmap v13288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:46 smithi033 ceph-mon[27845]: pgmap v13288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:46 smithi096 ceph-mon[32753]: pgmap v13288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:48 smithi194 ceph-mon[33017]: pgmap v13289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24167.066279 secs 2024-04-26T18:33:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:48 smithi033 ceph-mon[27845]: pgmap v13289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24167.066279 secs 2024-04-26T18:33:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:48 smithi096 ceph-mon[32753]: pgmap v13289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24167.066279 secs 2024-04-26T18:33:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24169.944196 secs 2024-04-26T18:33:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24169.944196 secs 2024-04-26T18:33:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24169.944196 secs 2024-04-26T18:33:50.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:50.586+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:50.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:50.586+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:33:50.586 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:50.586+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:33:50.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:50 smithi194 ceph-mon[33017]: pgmap v13290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:50 smithi033 ceph-mon[27845]: pgmap v13290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:50 smithi096 ceph-mon[32753]: pgmap v13290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:52 smithi194 ceph-mon[33017]: pgmap v13291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:52 smithi033 ceph-mon[27845]: pgmap v13291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:52 smithi096 ceph-mon[32753]: pgmap v13291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24172.066365 secs 2024-04-26T18:33:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24172.066365 secs 2024-04-26T18:33:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24172.066365 secs 2024-04-26T18:33:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:54 smithi194 ceph-mon[33017]: pgmap v13292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24174.944342 secs 2024-04-26T18:33:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:54 smithi033 ceph-mon[27845]: pgmap v13292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24174.944342 secs 2024-04-26T18:33:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:54 smithi096 ceph-mon[32753]: pgmap v13292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24174.944342 secs 2024-04-26T18:33:55.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:55.908+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:33:55.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:55.908+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:33:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:55.909+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 192 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:33:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:33:55.909+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 192 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T18:33:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:56 smithi194 ceph-mon[33017]: pgmap v13293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:56 smithi033 ceph-mon[27845]: pgmap v13293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:56 smithi096 ceph-mon[32753]: pgmap v13293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:33:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:58 smithi194 ceph-mon[33017]: pgmap v13294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24177.066451 secs 2024-04-26T18:33:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:58 smithi033 ceph-mon[27845]: pgmap v13294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24177.066451 secs 2024-04-26T18:33:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:58 smithi096 ceph-mon[32753]: pgmap v13294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:33:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24177.066451 secs 2024-04-26T18:33:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:33:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24179.944433 secs 2024-04-26T18:33:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:33:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24179.944433 secs 2024-04-26T18:33:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:33:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24179.944433 secs 2024-04-26T18:34:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:00 smithi194 ceph-mon[33017]: pgmap v13295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:00 smithi033 ceph-mon[27845]: pgmap v13295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:00 smithi096 ceph-mon[32753]: pgmap v13295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:02 smithi194 ceph-mon[33017]: pgmap v13296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:02 smithi033 ceph-mon[27845]: pgmap v13296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:02 smithi096 ceph-mon[32753]: pgmap v13296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24182.066532 secs 2024-04-26T18:34:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24182.066532 secs 2024-04-26T18:34:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24182.066532 secs 2024-04-26T18:34:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:04 smithi194 ceph-mon[33017]: pgmap v13297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24184.944513 secs 2024-04-26T18:34:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:04 smithi033 ceph-mon[27845]: pgmap v13297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24184.944513 secs 2024-04-26T18:34:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:04 smithi096 ceph-mon[32753]: pgmap v13297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24184.944513 secs 2024-04-26T18:34:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:05.601+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:05.601+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:05.601+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:34:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:06 smithi194 ceph-mon[33017]: pgmap v13298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:06 smithi033 ceph-mon[27845]: pgmap v13298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:06 smithi096 ceph-mon[32753]: pgmap v13298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:08 smithi194 ceph-mon[33017]: pgmap v13299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24187.066592 secs 2024-04-26T18:34:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:08 smithi033 ceph-mon[27845]: pgmap v13299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24187.066592 secs 2024-04-26T18:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:08 smithi096 ceph-mon[32753]: pgmap v13299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24187.066592 secs 2024-04-26T18:34:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24189.944638 secs 2024-04-26T18:34:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24189.944638 secs 2024-04-26T18:34:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24189.944638 secs 2024-04-26T18:34:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:10 smithi194 ceph-mon[33017]: pgmap v13300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:10 smithi033 ceph-mon[27845]: pgmap v13300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:10 smithi096 ceph-mon[32753]: pgmap v13300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:12 smithi194 ceph-mon[33017]: pgmap v13301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:12 smithi033 ceph-mon[27845]: pgmap v13301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:12 smithi096 ceph-mon[32753]: pgmap v13301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24192.066674 secs 2024-04-26T18:34:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24192.066674 secs 2024-04-26T18:34:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24192.066674 secs 2024-04-26T18:34:14.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:14 smithi033 ceph-mon[27845]: pgmap v13302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:14.581 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24194.944765 secs 2024-04-26T18:34:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:14 smithi194 ceph-mon[33017]: pgmap v13302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24194.944765 secs 2024-04-26T18:34:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:14 smithi096 ceph-mon[32753]: pgmap v13302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24194.944765 secs 2024-04-26T18:34:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:16 smithi194 ceph-mon[33017]: pgmap v13303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:16 smithi033 ceph-mon[27845]: pgmap v13303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:16 smithi096 ceph-mon[32753]: pgmap v13303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:18.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:18 smithi194 ceph-mon[33017]: pgmap v13304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:18.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:34:18.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24197.066766 secs 2024-04-26T18:34:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:18 smithi033 ceph-mon[27845]: pgmap v13304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:34:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24197.066766 secs 2024-04-26T18:34:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:18 smithi096 ceph-mon[32753]: pgmap v13304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:34:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24197.066766 secs 2024-04-26T18:34:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24199.944892 secs 2024-04-26T18:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:19 smithi033 ceph-mon[27845]: pgmap v13305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:19 smithi033 ceph-mon[27845]: pgmap v13306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24199.944892 secs 2024-04-26T18:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:19 smithi096 ceph-mon[32753]: pgmap v13305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:19 smithi096 ceph-mon[32753]: pgmap v13306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:34:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24199.944892 secs 2024-04-26T18:34:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:34:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:34:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:19 smithi194 ceph-mon[33017]: pgmap v13305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:19 smithi194 ceph-mon[33017]: pgmap v13306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:34:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:20.616+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:20.616+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:34:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:20.616+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:22 smithi033 ceph-mon[27845]: pgmap v13307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:22 smithi096 ceph-mon[32753]: pgmap v13307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:22.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:22 smithi194 ceph-mon[33017]: pgmap v13307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24202.066848 secs 2024-04-26T18:34:23.457 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24202.066848 secs 2024-04-26T18:34:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24202.066848 secs 2024-04-26T18:34:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:24 smithi033 ceph-mon[27845]: pgmap v13308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24204.945052 secs 2024-04-26T18:34:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:24 smithi096 ceph-mon[32753]: pgmap v13308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24204.945052 secs 2024-04-26T18:34:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:24 smithi194 ceph-mon[33017]: pgmap v13308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24204.945052 secs 2024-04-26T18:34:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:26 smithi033 ceph-mon[27845]: pgmap v13309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:26 smithi096 ceph-mon[32753]: pgmap v13309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:26 smithi194 ceph-mon[33017]: pgmap v13309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:28 smithi033 ceph-mon[27845]: pgmap v13310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24207.066903 secs 2024-04-26T18:34:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:28 smithi096 ceph-mon[32753]: pgmap v13310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24207.066903 secs 2024-04-26T18:34:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:28 smithi194 ceph-mon[33017]: pgmap v13310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:34:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24207.066903 secs 2024-04-26T18:34:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24209.945156 secs 2024-04-26T18:34:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24209.945156 secs 2024-04-26T18:34:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24209.945156 secs 2024-04-26T18:34:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:30 smithi033 ceph-mon[27845]: pgmap v13311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:34:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:30 smithi096 ceph-mon[32753]: pgmap v13311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:34:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:30 smithi194 ceph-mon[33017]: pgmap v13311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:34:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:32 smithi096 ceph-mon[32753]: pgmap v13312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:32 smithi194 ceph-mon[33017]: pgmap v13312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:32 smithi033 ceph-mon[27845]: pgmap v13312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:33.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24212.066984 secs 2024-04-26T18:34:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24212.066984 secs 2024-04-26T18:34:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24212.066984 secs 2024-04-26T18:34:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:34 smithi194 ceph-mon[33017]: pgmap v13313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24214.945213 secs 2024-04-26T18:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:34 smithi033 ceph-mon[27845]: pgmap v13313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24214.945213 secs 2024-04-26T18:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:34 smithi096 ceph-mon[32753]: pgmap v13313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24214.945213 secs 2024-04-26T18:34:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:35.627+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:35.627+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:35.627+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:34:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:36 smithi194 ceph-mon[33017]: pgmap v13314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:36 smithi033 ceph-mon[27845]: pgmap v13314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:36 smithi096 ceph-mon[32753]: pgmap v13314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:37 smithi194 ceph-mon[33017]: pgmap v13315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:37 smithi033 ceph-mon[27845]: pgmap v13315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:37 smithi096 ceph-mon[32753]: pgmap v13315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:38.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24217.067073 secs 2024-04-26T18:34:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24217.067073 secs 2024-04-26T18:34:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24217.067073 secs 2024-04-26T18:34:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24219.945334 secs 2024-04-26T18:34:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:39 smithi194 ceph-mon[33017]: pgmap v13316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24219.945334 secs 2024-04-26T18:34:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:39 smithi033 ceph-mon[27845]: pgmap v13316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24219.945334 secs 2024-04-26T18:34:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:39 smithi096 ceph-mon[32753]: pgmap v13316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:41 smithi194 ceph-mon[33017]: pgmap v13317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:41 smithi033 ceph-mon[27845]: pgmap v13317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:41 smithi096 ceph-mon[32753]: pgmap v13317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24222.067153 secs 2024-04-26T18:34:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:43 smithi194 ceph-mon[33017]: pgmap v13318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24224.945453 secs 2024-04-26T18:34:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24222.067153 secs 2024-04-26T18:34:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:43 smithi033 ceph-mon[27845]: pgmap v13318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24224.945453 secs 2024-04-26T18:34:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24222.067153 secs 2024-04-26T18:34:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:43 smithi096 ceph-mon[32753]: pgmap v13318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24224.945453 secs 2024-04-26T18:34:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:45 smithi194 ceph-mon[33017]: pgmap v13319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:45 smithi033 ceph-mon[27845]: pgmap v13319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:45 smithi096 ceph-mon[32753]: pgmap v13319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:47 smithi194 ceph-mon[33017]: pgmap v13320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:47 smithi033 ceph-mon[27845]: pgmap v13320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:47 smithi096 ceph-mon[32753]: pgmap v13320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24227.067215 secs 2024-04-26T18:34:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24229.945569 secs 2024-04-26T18:34:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24227.067215 secs 2024-04-26T18:34:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24229.945569 secs 2024-04-26T18:34:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24227.067215 secs 2024-04-26T18:34:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24229.945569 secs 2024-04-26T18:34:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:50 smithi194 ceph-mon[33017]: pgmap v13321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:50 smithi033 ceph-mon[27845]: pgmap v13321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:50 smithi096 ceph-mon[32753]: pgmap v13321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:50.629+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:50.629+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:34:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:34:50.629+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:34:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:52 smithi194 ceph-mon[33017]: pgmap v13322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:52 smithi033 ceph-mon[27845]: pgmap v13322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:52 smithi096 ceph-mon[32753]: pgmap v13322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24232.067298 secs 2024-04-26T18:34:54.387 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:54 smithi194 ceph-mon[33017]: pgmap v13323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:54.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24234.945667 secs 2024-04-26T18:34:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24232.067298 secs 2024-04-26T18:34:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:54 smithi033 ceph-mon[27845]: pgmap v13323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24234.945667 secs 2024-04-26T18:34:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24232.067298 secs 2024-04-26T18:34:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:54 smithi096 ceph-mon[32753]: pgmap v13323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24234.945667 secs 2024-04-26T18:34:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:56 smithi194 ceph-mon[33017]: pgmap v13324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:56 smithi033 ceph-mon[27845]: pgmap v13324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:56 smithi096 ceph-mon[32753]: pgmap v13324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:34:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:58 smithi194 ceph-mon[33017]: pgmap v13325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:58 smithi033 ceph-mon[27845]: pgmap v13325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:58 smithi096 ceph-mon[32753]: pgmap v13325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:34:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24237.067382 secs 2024-04-26T18:34:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24239.945809 secs 2024-04-26T18:34:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24237.067382 secs 2024-04-26T18:34:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24239.945809 secs 2024-04-26T18:34:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24237.067382 secs 2024-04-26T18:34:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24239.945809 secs 2024-04-26T18:35:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:00 smithi194 ceph-mon[33017]: pgmap v13326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:00 smithi033 ceph-mon[27845]: pgmap v13326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:00 smithi096 ceph-mon[32753]: pgmap v13326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:02 smithi194 ceph-mon[33017]: pgmap v13327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:02 smithi033 ceph-mon[27845]: pgmap v13327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:02 smithi096 ceph-mon[32753]: pgmap v13327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24242.067439 secs 2024-04-26T18:35:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:04 smithi194 ceph-mon[33017]: pgmap v13328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24244.945870 secs 2024-04-26T18:35:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24242.067439 secs 2024-04-26T18:35:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:04 smithi033 ceph-mon[27845]: pgmap v13328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24244.945870 secs 2024-04-26T18:35:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24242.067439 secs 2024-04-26T18:35:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:04 smithi096 ceph-mon[32753]: pgmap v13328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24244.945870 secs 2024-04-26T18:35:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:05.645+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:05.645+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:05.645+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:35:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:06 smithi194 ceph-mon[33017]: pgmap v13329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:06 smithi033 ceph-mon[27845]: pgmap v13329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:06 smithi096 ceph-mon[32753]: pgmap v13329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:08 smithi194 ceph-mon[33017]: pgmap v13330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:08 smithi033 ceph-mon[27845]: pgmap v13330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:08 smithi096 ceph-mon[32753]: pgmap v13330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24247.067522 secs 2024-04-26T18:35:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24249.945996 secs 2024-04-26T18:35:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24247.067522 secs 2024-04-26T18:35:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24249.945996 secs 2024-04-26T18:35:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24247.067522 secs 2024-04-26T18:35:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24249.945996 secs 2024-04-26T18:35:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:10 smithi194 ceph-mon[33017]: pgmap v13331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:10 smithi033 ceph-mon[27845]: pgmap v13331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:10 smithi096 ceph-mon[32753]: pgmap v13331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:12 smithi194 ceph-mon[33017]: pgmap v13332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:12 smithi033 ceph-mon[27845]: pgmap v13332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:12 smithi096 ceph-mon[32753]: pgmap v13332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24252.067605 secs 2024-04-26T18:35:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24252.067605 secs 2024-04-26T18:35:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24252.067605 secs 2024-04-26T18:35:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:14 smithi194 ceph-mon[33017]: pgmap v13333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24254.946140 secs 2024-04-26T18:35:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:14 smithi033 ceph-mon[27845]: pgmap v13333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24254.946140 secs 2024-04-26T18:35:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:14 smithi096 ceph-mon[32753]: pgmap v13333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24254.946140 secs 2024-04-26T18:35:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:16 smithi194 ceph-mon[33017]: pgmap v13334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:16 smithi033 ceph-mon[27845]: pgmap v13334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:16 smithi096 ceph-mon[32753]: pgmap v13334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:18 smithi033 ceph-mon[27845]: pgmap v13335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24257.067685 secs 2024-04-26T18:35:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:18 smithi096 ceph-mon[32753]: pgmap v13335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24257.067685 secs 2024-04-26T18:35:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:18 smithi194 ceph-mon[33017]: pgmap v13335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24257.067685 secs 2024-04-26T18:35:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24259.946209 secs 2024-04-26T18:35:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24259.946209 secs 2024-04-26T18:35:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24259.946209 secs 2024-04-26T18:35:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:20 smithi033 ceph-mon[27845]: pgmap v13336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:35:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:20 smithi096 ceph-mon[32753]: pgmap v13336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:35:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:20 smithi194 ceph-mon[33017]: pgmap v13336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:35:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:20.655+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:20.655+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:20.655+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:35:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:22 smithi033 ceph-mon[27845]: pgmap v13337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:35:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:35:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:22 smithi096 ceph-mon[32753]: pgmap v13337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:35:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:35:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:22 smithi194 ceph-mon[33017]: pgmap v13337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:35:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:35:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:35:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:23 smithi033 ceph-mon[27845]: pgmap v13338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:35:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:23 smithi033 ceph-mon[27845]: pgmap v13339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24262.067766 secs 2024-04-26T18:35:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:23 smithi096 ceph-mon[32753]: pgmap v13338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:35:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:23 smithi096 ceph-mon[32753]: pgmap v13339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24262.067766 secs 2024-04-26T18:35:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:23 smithi194 ceph-mon[33017]: pgmap v13338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:35:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:23 smithi194 ceph-mon[33017]: pgmap v13339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24262.067766 secs 2024-04-26T18:35:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24264.946335 secs 2024-04-26T18:35:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24264.946335 secs 2024-04-26T18:35:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24264.946335 secs 2024-04-26T18:35:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:25 smithi096 ceph-mon[32753]: pgmap v13340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:35:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:25 smithi194 ceph-mon[33017]: pgmap v13340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:35:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:25 smithi033 ceph-mon[27845]: pgmap v13340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:35:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:25.922+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:35:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:25.922+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:35:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:25.923+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 193 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:35:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:25.923+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 193 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:35:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:27 smithi096 ceph-mon[32753]: pgmap v13341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:27 smithi194 ceph-mon[33017]: pgmap v13341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:27 smithi033 ceph-mon[27845]: pgmap v13341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24267.067857 secs 2024-04-26T18:35:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24267.067857 secs 2024-04-26T18:35:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24267.067857 secs 2024-04-26T18:35:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:29 smithi194 ceph-mon[33017]: pgmap v13342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24269.946476 secs 2024-04-26T18:35:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:29 smithi033 ceph-mon[27845]: pgmap v13342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24269.946476 secs 2024-04-26T18:35:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:29 smithi096 ceph-mon[32753]: pgmap v13342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24269.946476 secs 2024-04-26T18:35:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:31 smithi194 ceph-mon[33017]: pgmap v13343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:31 smithi033 ceph-mon[27845]: pgmap v13343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:31 smithi096 ceph-mon[32753]: pgmap v13343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:35:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:33 smithi194 ceph-mon[33017]: pgmap v13344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:35:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24272.067939 secs 2024-04-26T18:35:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:33 smithi033 ceph-mon[27845]: pgmap v13344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:35:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24272.067939 secs 2024-04-26T18:35:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:33 smithi096 ceph-mon[32753]: pgmap v13344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:35:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24272.067939 secs 2024-04-26T18:35:34.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24274.946610 secs 2024-04-26T18:35:34.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:34 smithi194 ceph-mon[33017]: pgmap v13345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24274.946610 secs 2024-04-26T18:35:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:34 smithi033 ceph-mon[27845]: pgmap v13345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24274.946610 secs 2024-04-26T18:35:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:34 smithi096 ceph-mon[32753]: pgmap v13345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:35.660+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:35.660+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:35:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:35.660+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:36 smithi194 ceph-mon[33017]: pgmap v13346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:36 smithi033 ceph-mon[27845]: pgmap v13346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:36 smithi096 ceph-mon[32753]: pgmap v13346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:38 smithi033 ceph-mon[27845]: pgmap v13347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24277.068025 secs 2024-04-26T18:35:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:38 smithi096 ceph-mon[32753]: pgmap v13347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24277.068025 secs 2024-04-26T18:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:38 smithi194 ceph-mon[33017]: pgmap v13347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24277.068025 secs 2024-04-26T18:35:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24279.946759 secs 2024-04-26T18:35:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24279.946759 secs 2024-04-26T18:35:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24279.946759 secs 2024-04-26T18:35:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:40 smithi033 ceph-mon[27845]: pgmap v13348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:40 smithi096 ceph-mon[32753]: pgmap v13348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:40 smithi194 ceph-mon[33017]: pgmap v13348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:42 smithi033 ceph-mon[27845]: pgmap v13349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:42 smithi096 ceph-mon[32753]: pgmap v13349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:42 smithi194 ceph-mon[33017]: pgmap v13349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24282.068115 secs 2024-04-26T18:35:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24282.068115 secs 2024-04-26T18:35:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24282.068115 secs 2024-04-26T18:35:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24284.946890 secs 2024-04-26T18:35:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:44 smithi033 ceph-mon[27845]: pgmap v13350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24284.946890 secs 2024-04-26T18:35:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:44 smithi096 ceph-mon[32753]: pgmap v13350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24284.946890 secs 2024-04-26T18:35:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:44 smithi194 ceph-mon[33017]: pgmap v13350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:46 smithi033 ceph-mon[27845]: pgmap v13351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:46 smithi096 ceph-mon[32753]: pgmap v13351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:46 smithi194 ceph-mon[33017]: pgmap v13351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:48 smithi033 ceph-mon[27845]: pgmap v13352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24287.068180 secs 2024-04-26T18:35:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:48 smithi096 ceph-mon[32753]: pgmap v13352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24287.068180 secs 2024-04-26T18:35:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:48 smithi194 ceph-mon[33017]: pgmap v13352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24287.068180 secs 2024-04-26T18:35:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24289.946990 secs 2024-04-26T18:35:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24289.946990 secs 2024-04-26T18:35:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24289.946990 secs 2024-04-26T18:35:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:50.675+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:50.675+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:35:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:35:50.675+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:35:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:50 smithi033 ceph-mon[27845]: pgmap v13353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:50 smithi096 ceph-mon[32753]: pgmap v13353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:50 smithi194 ceph-mon[33017]: pgmap v13353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:52 smithi033 ceph-mon[27845]: pgmap v13354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:52 smithi096 ceph-mon[32753]: pgmap v13354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:52 smithi194 ceph-mon[33017]: pgmap v13354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24292.068267 secs 2024-04-26T18:35:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24292.068267 secs 2024-04-26T18:35:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24292.068267 secs 2024-04-26T18:35:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24294.947128 secs 2024-04-26T18:35:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:54 smithi033 ceph-mon[27845]: pgmap v13355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24294.947128 secs 2024-04-26T18:35:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:54 smithi096 ceph-mon[32753]: pgmap v13355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24294.947128 secs 2024-04-26T18:35:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:54 smithi194 ceph-mon[33017]: pgmap v13355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:56 smithi096 ceph-mon[32753]: pgmap v13356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:56 smithi194 ceph-mon[33017]: pgmap v13356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:56 smithi033 ceph-mon[27845]: pgmap v13356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:35:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:58 smithi194 ceph-mon[33017]: pgmap v13357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24297.068325 secs 2024-04-26T18:35:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:58 smithi033 ceph-mon[27845]: pgmap v13357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24297.068325 secs 2024-04-26T18:35:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:58 smithi096 ceph-mon[32753]: pgmap v13357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:35:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24297.068325 secs 2024-04-26T18:36:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24299.947274 secs 2024-04-26T18:36:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24299.947274 secs 2024-04-26T18:36:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24299.947274 secs 2024-04-26T18:36:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:00 smithi194 ceph-mon[33017]: pgmap v13358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:00 smithi033 ceph-mon[27845]: pgmap v13358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:00 smithi096 ceph-mon[32753]: pgmap v13358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:02 smithi194 ceph-mon[33017]: pgmap v13359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:02 smithi033 ceph-mon[27845]: pgmap v13359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:02 smithi096 ceph-mon[32753]: pgmap v13359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:04.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24302.068413 secs 2024-04-26T18:36:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24302.068413 secs 2024-04-26T18:36:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24302.068413 secs 2024-04-26T18:36:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24304.947421 secs 2024-04-26T18:36:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:04 smithi194 ceph-mon[33017]: pgmap v13360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24304.947421 secs 2024-04-26T18:36:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:04 smithi033 ceph-mon[27845]: pgmap v13360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24304.947421 secs 2024-04-26T18:36:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:04 smithi096 ceph-mon[32753]: pgmap v13360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:05.684+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:05.684+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:05.684+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:36:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:06 smithi194 ceph-mon[33017]: pgmap v13361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:06 smithi033 ceph-mon[27845]: pgmap v13361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:06 smithi096 ceph-mon[32753]: pgmap v13361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:08.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:08 smithi096 ceph-mon[32753]: pgmap v13362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:08.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24307.068496 secs 2024-04-26T18:36:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:08 smithi194 ceph-mon[33017]: pgmap v13362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24307.068496 secs 2024-04-26T18:36:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:08 smithi033 ceph-mon[27845]: pgmap v13362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24307.068496 secs 2024-04-26T18:36:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24309.947566 secs 2024-04-26T18:36:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24309.947566 secs 2024-04-26T18:36:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24309.947566 secs 2024-04-26T18:36:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:10 smithi194 ceph-mon[33017]: pgmap v13363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:10 smithi033 ceph-mon[27845]: pgmap v13363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:10 smithi096 ceph-mon[32753]: pgmap v13363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:12 smithi194 ceph-mon[33017]: pgmap v13364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:12 smithi033 ceph-mon[27845]: pgmap v13364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:12 smithi096 ceph-mon[32753]: pgmap v13364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:14.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24312.068581 secs 2024-04-26T18:36:14.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24314.947690 secs 2024-04-26T18:36:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24312.068581 secs 2024-04-26T18:36:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24314.947690 secs 2024-04-26T18:36:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24312.068581 secs 2024-04-26T18:36:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24314.947690 secs 2024-04-26T18:36:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:14 smithi194 ceph-mon[33017]: pgmap v13365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:14 smithi033 ceph-mon[27845]: pgmap v13365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:14 smithi096 ceph-mon[32753]: pgmap v13365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:16 smithi194 ceph-mon[33017]: pgmap v13366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:16 smithi033 ceph-mon[27845]: pgmap v13366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:16 smithi096 ceph-mon[32753]: pgmap v13366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:18 smithi194 ceph-mon[33017]: pgmap v13367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24317.068665 secs 2024-04-26T18:36:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24319.947848 secs 2024-04-26T18:36:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:18 smithi033 ceph-mon[27845]: pgmap v13367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24317.068665 secs 2024-04-26T18:36:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24319.947848 secs 2024-04-26T18:36:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:18 smithi096 ceph-mon[32753]: pgmap v13367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24317.068665 secs 2024-04-26T18:36:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24319.947848 secs 2024-04-26T18:36:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:20.690+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:20.690+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:20.690+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:36:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:20 smithi194 ceph-mon[33017]: pgmap v13368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:20 smithi033 ceph-mon[27845]: pgmap v13368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:20 smithi096 ceph-mon[32753]: pgmap v13368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:22 smithi194 ceph-mon[33017]: pgmap v13369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:36:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:22 smithi033 ceph-mon[27845]: pgmap v13369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:36:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:22 smithi096 ceph-mon[32753]: pgmap v13369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:36:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24322.068750 secs 2024-04-26T18:36:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:36:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:36:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:23 smithi194 ceph-mon[33017]: pgmap v13370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:23 smithi194 ceph-mon[33017]: pgmap v13371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:36:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24324.947976 secs 2024-04-26T18:36:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24322.068750 secs 2024-04-26T18:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:23 smithi033 ceph-mon[27845]: pgmap v13370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:23 smithi033 ceph-mon[27845]: pgmap v13371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24324.947976 secs 2024-04-26T18:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24322.068750 secs 2024-04-26T18:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:23 smithi096 ceph-mon[32753]: pgmap v13370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:23 smithi096 ceph-mon[32753]: pgmap v13371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24324.947976 secs 2024-04-26T18:36:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:25 smithi194 ceph-mon[33017]: pgmap v13372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:25 smithi033 ceph-mon[27845]: pgmap v13372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:25 smithi096 ceph-mon[32753]: pgmap v13372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:27 smithi194 ceph-mon[33017]: pgmap v13373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:27 smithi033 ceph-mon[27845]: pgmap v13373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:27 smithi096 ceph-mon[32753]: pgmap v13373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24327.068833 secs 2024-04-26T18:36:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24327.068833 secs 2024-04-26T18:36:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24327.068833 secs 2024-04-26T18:36:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24329.948048 secs 2024-04-26T18:36:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:29 smithi194 ceph-mon[33017]: pgmap v13374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:36:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24329.948048 secs 2024-04-26T18:36:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:29 smithi033 ceph-mon[27845]: pgmap v13374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:36:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24329.948048 secs 2024-04-26T18:36:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:29 smithi096 ceph-mon[32753]: pgmap v13374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:36:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:31 smithi194 ceph-mon[33017]: pgmap v13375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:31 smithi033 ceph-mon[27845]: pgmap v13375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:31 smithi096 ceph-mon[32753]: pgmap v13375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:36:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24332.068921 secs 2024-04-26T18:36:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24332.068921 secs 2024-04-26T18:36:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24332.068921 secs 2024-04-26T18:36:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:34 smithi033 ceph-mon[27845]: pgmap v13376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:36:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24334.948164 secs 2024-04-26T18:36:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:34 smithi096 ceph-mon[32753]: pgmap v13376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:36:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24334.948164 secs 2024-04-26T18:36:34.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:34 smithi194 ceph-mon[33017]: pgmap v13376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:36:34.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24334.948164 secs 2024-04-26T18:36:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:35 smithi194 ceph-mon[33017]: pgmap v13377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:35 smithi033 ceph-mon[27845]: pgmap v13377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:35 smithi096 ceph-mon[32753]: pgmap v13377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:35.706+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:35.706+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:35.706+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:36:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:38 smithi194 ceph-mon[33017]: pgmap v13378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24337.068999 secs 2024-04-26T18:36:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:38 smithi033 ceph-mon[27845]: pgmap v13378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24337.068999 secs 2024-04-26T18:36:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:38 smithi096 ceph-mon[32753]: pgmap v13378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24337.068999 secs 2024-04-26T18:36:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24339.948304 secs 2024-04-26T18:36:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24339.948304 secs 2024-04-26T18:36:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24339.948304 secs 2024-04-26T18:36:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:40 smithi194 ceph-mon[33017]: pgmap v13379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:40 smithi033 ceph-mon[27845]: pgmap v13379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:40 smithi096 ceph-mon[32753]: pgmap v13379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:42 smithi194 ceph-mon[33017]: pgmap v13380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:42 smithi033 ceph-mon[27845]: pgmap v13380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:42 smithi096 ceph-mon[32753]: pgmap v13380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24342.069081 secs 2024-04-26T18:36:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24342.069081 secs 2024-04-26T18:36:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24342.069081 secs 2024-04-26T18:36:44.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:44 smithi033 ceph-mon[27845]: pgmap v13381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:44.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24344.948401 secs 2024-04-26T18:36:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:44 smithi194 ceph-mon[33017]: pgmap v13381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24344.948401 secs 2024-04-26T18:36:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:44 smithi096 ceph-mon[32753]: pgmap v13381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24344.948401 secs 2024-04-26T18:36:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:46 smithi194 ceph-mon[33017]: pgmap v13382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:46 smithi033 ceph-mon[27845]: pgmap v13382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:46 smithi096 ceph-mon[32753]: pgmap v13382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:48 smithi194 ceph-mon[33017]: pgmap v13383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24347.069170 secs 2024-04-26T18:36:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:48 smithi033 ceph-mon[27845]: pgmap v13383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24347.069170 secs 2024-04-26T18:36:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:48 smithi096 ceph-mon[32753]: pgmap v13383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24347.069170 secs 2024-04-26T18:36:49.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24349.948523 secs 2024-04-26T18:36:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24349.948523 secs 2024-04-26T18:36:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24349.948523 secs 2024-04-26T18:36:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:50 smithi194 ceph-mon[33017]: pgmap v13384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:50 smithi033 ceph-mon[27845]: pgmap v13384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:50 smithi096 ceph-mon[32753]: pgmap v13384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:50.722+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:50.722+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:36:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:36:50.722+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:36:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:52 smithi194 ceph-mon[33017]: pgmap v13385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:52 smithi033 ceph-mon[27845]: pgmap v13385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:52 smithi096 ceph-mon[32753]: pgmap v13385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24352.069253 secs 2024-04-26T18:36:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24352.069253 secs 2024-04-26T18:36:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24352.069253 secs 2024-04-26T18:36:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:54 smithi033 ceph-mon[27845]: pgmap v13386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24354.948647 secs 2024-04-26T18:36:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:54 smithi096 ceph-mon[32753]: pgmap v13386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24354.948647 secs 2024-04-26T18:36:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:54 smithi194 ceph-mon[33017]: pgmap v13386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24354.948647 secs 2024-04-26T18:36:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:56 smithi033 ceph-mon[27845]: pgmap v13387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:56 smithi096 ceph-mon[32753]: pgmap v13387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:56 smithi194 ceph-mon[33017]: pgmap v13387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:36:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:58 smithi033 ceph-mon[27845]: pgmap v13388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24357.069336 secs 2024-04-26T18:36:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:58 smithi096 ceph-mon[32753]: pgmap v13388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24357.069336 secs 2024-04-26T18:36:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:58 smithi194 ceph-mon[33017]: pgmap v13388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:36:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24357.069336 secs 2024-04-26T18:36:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24359.948756 secs 2024-04-26T18:36:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24359.948756 secs 2024-04-26T18:36:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24359.948756 secs 2024-04-26T18:37:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:00 smithi033 ceph-mon[27845]: pgmap v13389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:00 smithi096 ceph-mon[32753]: pgmap v13389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:00 smithi194 ceph-mon[33017]: pgmap v13389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:02 smithi033 ceph-mon[27845]: pgmap v13390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:02 smithi096 ceph-mon[32753]: pgmap v13390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:02 smithi194 ceph-mon[33017]: pgmap v13390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24362.069400 secs 2024-04-26T18:37:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24362.069400 secs 2024-04-26T18:37:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24362.069400 secs 2024-04-26T18:37:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:04 smithi033 ceph-mon[27845]: pgmap v13391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24364.948899 secs 2024-04-26T18:37:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:04 smithi096 ceph-mon[32753]: pgmap v13391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24364.948899 secs 2024-04-26T18:37:04.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:04 smithi194 ceph-mon[33017]: pgmap v13391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:04.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24364.948899 secs 2024-04-26T18:37:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:05.733+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:05.733+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:37:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:05.733+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:06 smithi033 ceph-mon[27845]: pgmap v13392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:06 smithi096 ceph-mon[32753]: pgmap v13392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:06 smithi194 ceph-mon[33017]: pgmap v13392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:08 smithi033 ceph-mon[27845]: pgmap v13393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24367.069485 secs 2024-04-26T18:37:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:08 smithi096 ceph-mon[32753]: pgmap v13393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24367.069485 secs 2024-04-26T18:37:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:08 smithi194 ceph-mon[33017]: pgmap v13393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24367.069485 secs 2024-04-26T18:37:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24369.949015 secs 2024-04-26T18:37:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24369.949015 secs 2024-04-26T18:37:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24369.949015 secs 2024-04-26T18:37:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:10 smithi194 ceph-mon[33017]: pgmap v13394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:10 smithi033 ceph-mon[27845]: pgmap v13394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:10 smithi096 ceph-mon[32753]: pgmap v13394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:12 smithi194 ceph-mon[33017]: pgmap v13395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:12 smithi033 ceph-mon[27845]: pgmap v13395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:12 smithi096 ceph-mon[32753]: pgmap v13395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24372.069570 secs 2024-04-26T18:37:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24372.069570 secs 2024-04-26T18:37:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24372.069570 secs 2024-04-26T18:37:14.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:14 smithi033 ceph-mon[27845]: pgmap v13396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:14.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24374.949159 secs 2024-04-26T18:37:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:14 smithi194 ceph-mon[33017]: pgmap v13396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24374.949159 secs 2024-04-26T18:37:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:14 smithi096 ceph-mon[32753]: pgmap v13396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24374.949159 secs 2024-04-26T18:37:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:16 smithi194 ceph-mon[33017]: pgmap v13397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:16 smithi033 ceph-mon[27845]: pgmap v13397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:16 smithi096 ceph-mon[32753]: pgmap v13397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:18 smithi194 ceph-mon[33017]: pgmap v13398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24377.069632 secs 2024-04-26T18:37:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:18 smithi033 ceph-mon[27845]: pgmap v13398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24377.069632 secs 2024-04-26T18:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:18 smithi096 ceph-mon[32753]: pgmap v13398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24377.069632 secs 2024-04-26T18:37:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24379.949290 secs 2024-04-26T18:37:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24379.949290 secs 2024-04-26T18:37:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24379.949290 secs 2024-04-26T18:37:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:20.741+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:20.741+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:37:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:20.741+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:20 smithi194 ceph-mon[33017]: pgmap v13399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:20 smithi033 ceph-mon[27845]: pgmap v13399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:20 smithi096 ceph-mon[32753]: pgmap v13399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:23 smithi194 ceph-mon[33017]: pgmap v13400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24382.069689 secs 2024-04-26T18:37:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:23 smithi033 ceph-mon[27845]: pgmap v13400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24382.069689 secs 2024-04-26T18:37:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:23 smithi096 ceph-mon[32753]: pgmap v13400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24382.069689 secs 2024-04-26T18:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24384.949369 secs 2024-04-26T18:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24384.949369 secs 2024-04-26T18:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24384.949369 secs 2024-04-26T18:37:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:37:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:25 smithi096 ceph-mon[32753]: pgmap v13401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:25 smithi194 ceph-mon[33017]: pgmap v13401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:25 smithi033 ceph-mon[27845]: pgmap v13401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: pgmap v13402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: pgmap v13402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: pgmap v13402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:27.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:28 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:37:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:28 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:37:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:28 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:37:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:28 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:37:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24387.069748 secs 2024-04-26T18:37:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:28 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:37:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:28 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:37:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:28 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:37:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:28 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:37:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24387.069748 secs 2024-04-26T18:37:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:28 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:37:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:28 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:37:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:28 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:37:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:28 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:37:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24387.069748 secs 2024-04-26T18:37:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:29 smithi033 ceph-mon[27845]: pgmap v13403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24389.949485 secs 2024-04-26T18:37:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:29 smithi096 ceph-mon[32753]: pgmap v13403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24389.949485 secs 2024-04-26T18:37:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:29 smithi194 ceph-mon[33017]: pgmap v13403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24389.949485 secs 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:37:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:37:30.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:37:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:37:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:37:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:37:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:37:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:37:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:31 smithi033 ceph-mon[27845]: pgmap v13404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:31 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:37:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:31 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:37:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:31 smithi033 ceph-mon[27845]: pgmap v13405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:37:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:31 smithi033 ceph-mon[27845]: pgmap v13406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:31 smithi096 ceph-mon[32753]: pgmap v13404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:31 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:37:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:31 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:37:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:31 smithi096 ceph-mon[32753]: pgmap v13405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:37:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:31 smithi096 ceph-mon[32753]: pgmap v13406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:31 smithi194 ceph-mon[33017]: pgmap v13404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:31 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:37:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:31 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:37:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:31 smithi194 ceph-mon[33017]: pgmap v13405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T18:37:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:31 smithi194 ceph-mon[33017]: pgmap v13406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:33 smithi033 ceph-mon[27845]: pgmap v13407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24392.069810 secs 2024-04-26T18:37:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:33 smithi096 ceph-mon[32753]: pgmap v13407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24392.069810 secs 2024-04-26T18:37:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:33 smithi194 ceph-mon[33017]: pgmap v13407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24392.069810 secs 2024-04-26T18:37:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24394.949607 secs 2024-04-26T18:37:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24394.949607 secs 2024-04-26T18:37:34.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24394.949607 secs 2024-04-26T18:37:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:35 smithi033 ceph-mon[27845]: pgmap v13408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:37:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:35 smithi096 ceph-mon[32753]: pgmap v13408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:37:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:35 smithi194 ceph-mon[33017]: pgmap v13408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:37:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:35.757+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:35.758+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:35.758+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:37:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:37 smithi096 ceph-mon[32753]: pgmap v13409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:37 smithi194 ceph-mon[33017]: pgmap v13409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:37 smithi033 ceph-mon[27845]: pgmap v13409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24397.069872 secs 2024-04-26T18:37:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24397.069872 secs 2024-04-26T18:37:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24397.069872 secs 2024-04-26T18:37:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:39 smithi194 ceph-mon[33017]: pgmap v13410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24399.949721 secs 2024-04-26T18:37:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:39 smithi033 ceph-mon[27845]: pgmap v13410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24399.949721 secs 2024-04-26T18:37:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:39 smithi096 ceph-mon[32753]: pgmap v13410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:37:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24399.949721 secs 2024-04-26T18:37:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:41 smithi194 ceph-mon[33017]: pgmap v13411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:37:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:41 smithi033 ceph-mon[27845]: pgmap v13411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:37:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:41 smithi096 ceph-mon[32753]: pgmap v13411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:37:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:43 smithi194 ceph-mon[33017]: pgmap v13412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24402.069932 secs 2024-04-26T18:37:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:43 smithi033 ceph-mon[27845]: pgmap v13412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24402.069932 secs 2024-04-26T18:37:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:43 smithi096 ceph-mon[32753]: pgmap v13412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24402.069932 secs 2024-04-26T18:37:44.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24404.949840 secs 2024-04-26T18:37:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24404.949840 secs 2024-04-26T18:37:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24404.949840 secs 2024-04-26T18:37:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:45 smithi194 ceph-mon[33017]: pgmap v13413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:45 smithi033 ceph-mon[27845]: pgmap v13413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:45 smithi096 ceph-mon[32753]: pgmap v13413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:47 smithi194 ceph-mon[33017]: pgmap v13414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:47 smithi033 ceph-mon[27845]: pgmap v13414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:47 smithi096 ceph-mon[32753]: pgmap v13414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24407.069994 secs 2024-04-26T18:37:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24407.069994 secs 2024-04-26T18:37:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24407.069994 secs 2024-04-26T18:37:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:49 smithi194 ceph-mon[33017]: pgmap v13415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24409.949981 secs 2024-04-26T18:37:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:49 smithi033 ceph-mon[27845]: pgmap v13415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24409.949981 secs 2024-04-26T18:37:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:49 smithi096 ceph-mon[32753]: pgmap v13415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24409.949981 secs 2024-04-26T18:37:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:50.773+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:50.773+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:37:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:37:50.773+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:37:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:51 smithi194 ceph-mon[33017]: pgmap v13416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:51 smithi033 ceph-mon[27845]: pgmap v13416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:51 smithi096 ceph-mon[32753]: pgmap v13416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:53 smithi194 ceph-mon[33017]: pgmap v13417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24412.070054 secs 2024-04-26T18:37:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:53 smithi033 ceph-mon[27845]: pgmap v13417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24412.070054 secs 2024-04-26T18:37:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:53 smithi096 ceph-mon[32753]: pgmap v13417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24412.070054 secs 2024-04-26T18:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24414.950103 secs 2024-04-26T18:37:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:54 smithi194 ceph-mon[33017]: pgmap v13418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24414.950103 secs 2024-04-26T18:37:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:54 smithi033 ceph-mon[27845]: pgmap v13418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24414.950103 secs 2024-04-26T18:37:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:54 smithi096 ceph-mon[32753]: pgmap v13418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:56 smithi033 ceph-mon[27845]: pgmap v13419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:56 smithi096 ceph-mon[32753]: pgmap v13419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:56 smithi194 ceph-mon[33017]: pgmap v13419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:37:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24417.070116 secs 2024-04-26T18:37:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:58 smithi033 ceph-mon[27845]: pgmap v13420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:37:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24419.950184 secs 2024-04-26T18:37:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24417.070116 secs 2024-04-26T18:37:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:58 smithi096 ceph-mon[32753]: pgmap v13420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:37:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24419.950184 secs 2024-04-26T18:37:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24417.070116 secs 2024-04-26T18:37:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:58 smithi194 ceph-mon[33017]: pgmap v13420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:37:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:37:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24419.950184 secs 2024-04-26T18:38:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:00 smithi033 ceph-mon[27845]: pgmap v13421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:00 smithi096 ceph-mon[32753]: pgmap v13421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:00 smithi194 ceph-mon[33017]: pgmap v13421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:02 smithi033 ceph-mon[27845]: pgmap v13422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:02 smithi096 ceph-mon[32753]: pgmap v13422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:02 smithi194 ceph-mon[33017]: pgmap v13422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24422.070177 secs 2024-04-26T18:38:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24424.950266 secs 2024-04-26T18:38:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24422.070177 secs 2024-04-26T18:38:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24424.950266 secs 2024-04-26T18:38:04.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24422.070177 secs 2024-04-26T18:38:04.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24424.950266 secs 2024-04-26T18:38:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:04 smithi033 ceph-mon[27845]: pgmap v13423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:04 smithi096 ceph-mon[32753]: pgmap v13423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:04 smithi194 ceph-mon[33017]: pgmap v13423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:05.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:05.789+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:38:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:05.789+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:06 smithi033 ceph-mon[27845]: pgmap v13424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:06 smithi096 ceph-mon[32753]: pgmap v13424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:06 smithi194 ceph-mon[33017]: pgmap v13424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24427.070237 secs 2024-04-26T18:38:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:08 smithi096 ceph-mon[32753]: pgmap v13425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24429.950330 secs 2024-04-26T18:38:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24427.070237 secs 2024-04-26T18:38:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:08 smithi194 ceph-mon[33017]: pgmap v13425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24429.950330 secs 2024-04-26T18:38:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24427.070237 secs 2024-04-26T18:38:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:08 smithi033 ceph-mon[27845]: pgmap v13425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24429.950330 secs 2024-04-26T18:38:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:10 smithi096 ceph-mon[32753]: pgmap v13426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:10 smithi194 ceph-mon[33017]: pgmap v13426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:10 smithi033 ceph-mon[27845]: pgmap v13426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:12 smithi194 ceph-mon[33017]: pgmap v13427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:12 smithi033 ceph-mon[27845]: pgmap v13427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:12 smithi096 ceph-mon[32753]: pgmap v13427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24432.070295 secs 2024-04-26T18:38:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24434.950469 secs 2024-04-26T18:38:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24432.070295 secs 2024-04-26T18:38:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24434.950469 secs 2024-04-26T18:38:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24432.070295 secs 2024-04-26T18:38:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24434.950469 secs 2024-04-26T18:38:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:14 smithi194 ceph-mon[33017]: pgmap v13428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:14 smithi033 ceph-mon[27845]: pgmap v13428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:14 smithi096 ceph-mon[32753]: pgmap v13428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:16 smithi194 ceph-mon[33017]: pgmap v13429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:16 smithi033 ceph-mon[27845]: pgmap v13429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:16 smithi096 ceph-mon[32753]: pgmap v13429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24437.070357 secs 2024-04-26T18:38:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:18 smithi194 ceph-mon[33017]: pgmap v13430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24439.950558 secs 2024-04-26T18:38:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24437.070357 secs 2024-04-26T18:38:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:18 smithi033 ceph-mon[27845]: pgmap v13430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24439.950558 secs 2024-04-26T18:38:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24437.070357 secs 2024-04-26T18:38:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:18 smithi096 ceph-mon[32753]: pgmap v13430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24439.950558 secs 2024-04-26T18:38:20.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:20.792+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:20.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:20.792+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:20.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:20.792+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:38:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:20 smithi194 ceph-mon[33017]: pgmap v13431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:20 smithi033 ceph-mon[27845]: pgmap v13431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:20 smithi096 ceph-mon[32753]: pgmap v13431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:23.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:22 smithi033 ceph-mon[27845]: pgmap v13432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:22 smithi194 ceph-mon[33017]: pgmap v13432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:22 smithi096 ceph-mon[32753]: pgmap v13432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24442.070416 secs 2024-04-26T18:38:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24444.950698 secs 2024-04-26T18:38:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24442.070416 secs 2024-04-26T18:38:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24444.950698 secs 2024-04-26T18:38:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24442.070416 secs 2024-04-26T18:38:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24444.950698 secs 2024-04-26T18:38:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:24 smithi194 ceph-mon[33017]: pgmap v13433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:24 smithi033 ceph-mon[27845]: pgmap v13433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:24 smithi096 ceph-mon[32753]: pgmap v13433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:25.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:25.953+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:38:25.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:25.953+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:38:25.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:25.953+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 194 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:38:25.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:25.953+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 194 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T18:38:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:26 smithi194 ceph-mon[33017]: pgmap v13434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:26 smithi033 ceph-mon[27845]: pgmap v13434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:26 smithi096 ceph-mon[32753]: pgmap v13434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24447.070475 secs 2024-04-26T18:38:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:28 smithi194 ceph-mon[33017]: pgmap v13435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24449.950829 secs 2024-04-26T18:38:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24447.070475 secs 2024-04-26T18:38:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:28 smithi033 ceph-mon[27845]: pgmap v13435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24449.950829 secs 2024-04-26T18:38:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24447.070475 secs 2024-04-26T18:38:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:28 smithi096 ceph-mon[32753]: pgmap v13435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24449.950829 secs 2024-04-26T18:38:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:30 smithi194 ceph-mon[33017]: pgmap v13436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:38:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:30 smithi033 ceph-mon[27845]: pgmap v13436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:38:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:30 smithi096 ceph-mon[32753]: pgmap v13436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:38:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:38:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:38:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:38:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:38:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:38:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:38:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:38:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:38:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:38:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:33 smithi194 ceph-mon[33017]: pgmap v13437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:33 smithi194 ceph-mon[33017]: pgmap v13438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:33 smithi033 ceph-mon[27845]: pgmap v13437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:33 smithi033 ceph-mon[27845]: pgmap v13438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:33 smithi096 ceph-mon[32753]: pgmap v13437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:33 smithi096 ceph-mon[32753]: pgmap v13438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24452.070533 secs 2024-04-26T18:38:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24454.950977 secs 2024-04-26T18:38:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24452.070533 secs 2024-04-26T18:38:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24454.950977 secs 2024-04-26T18:38:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24452.070533 secs 2024-04-26T18:38:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24454.950977 secs 2024-04-26T18:38:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:35 smithi194 ceph-mon[33017]: pgmap v13439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:35 smithi033 ceph-mon[27845]: pgmap v13439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:35 smithi096 ceph-mon[32753]: pgmap v13439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:35.807+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:35.807+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:35.807+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:38:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:37 smithi194 ceph-mon[33017]: pgmap v13440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:37 smithi033 ceph-mon[27845]: pgmap v13440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:37 smithi096 ceph-mon[32753]: pgmap v13440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:39 smithi194 ceph-mon[33017]: pgmap v13441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24457.070614 secs 2024-04-26T18:38:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24459.951078 secs 2024-04-26T18:38:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:39 smithi033 ceph-mon[27845]: pgmap v13441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24457.070614 secs 2024-04-26T18:38:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24459.951078 secs 2024-04-26T18:38:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:39 smithi096 ceph-mon[32753]: pgmap v13441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24457.070614 secs 2024-04-26T18:38:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24459.951078 secs 2024-04-26T18:38:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:41 smithi194 ceph-mon[33017]: pgmap v13442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:41 smithi033 ceph-mon[27845]: pgmap v13442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:41 smithi096 ceph-mon[32753]: pgmap v13442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:38:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:43 smithi194 ceph-mon[33017]: pgmap v13443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:38:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:43 smithi033 ceph-mon[27845]: pgmap v13443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:38:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:43 smithi096 ceph-mon[32753]: pgmap v13443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:38:44.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24462.070671 secs 2024-04-26T18:38:44.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24464.951195 secs 2024-04-26T18:38:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24462.070671 secs 2024-04-26T18:38:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24464.951195 secs 2024-04-26T18:38:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24462.070671 secs 2024-04-26T18:38:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24464.951195 secs 2024-04-26T18:38:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:45 smithi194 ceph-mon[33017]: pgmap v13444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:45 smithi033 ceph-mon[27845]: pgmap v13444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:45 smithi096 ceph-mon[32753]: pgmap v13444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:47 smithi194 ceph-mon[33017]: pgmap v13445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:47 smithi033 ceph-mon[27845]: pgmap v13445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:47 smithi096 ceph-mon[32753]: pgmap v13445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:49 smithi194 ceph-mon[33017]: pgmap v13446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24467.070753 secs 2024-04-26T18:38:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24469.951271 secs 2024-04-26T18:38:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:49 smithi033 ceph-mon[27845]: pgmap v13446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24467.070753 secs 2024-04-26T18:38:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24469.951271 secs 2024-04-26T18:38:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:49 smithi096 ceph-mon[32753]: pgmap v13446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24467.070753 secs 2024-04-26T18:38:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24469.951271 secs 2024-04-26T18:38:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:50.810+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:50.810+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:38:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:38:50.810+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:38:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:51 smithi194 ceph-mon[33017]: pgmap v13447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:51 smithi033 ceph-mon[27845]: pgmap v13447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:51 smithi096 ceph-mon[32753]: pgmap v13447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:53 smithi194 ceph-mon[33017]: pgmap v13448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:53 smithi033 ceph-mon[27845]: pgmap v13448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:53 smithi096 ceph-mon[32753]: pgmap v13448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24472.070832 secs 2024-04-26T18:38:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24474.951389 secs 2024-04-26T18:38:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24472.070832 secs 2024-04-26T18:38:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24474.951389 secs 2024-04-26T18:38:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24472.070832 secs 2024-04-26T18:38:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24474.951389 secs 2024-04-26T18:38:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:55 smithi194 ceph-mon[33017]: pgmap v13449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:55 smithi033 ceph-mon[27845]: pgmap v13449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:55 smithi096 ceph-mon[32753]: pgmap v13449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:57 smithi194 ceph-mon[33017]: pgmap v13450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:57 smithi033 ceph-mon[27845]: pgmap v13450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:57 smithi096 ceph-mon[32753]: pgmap v13450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:38:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24477.070914 secs 2024-04-26T18:38:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24477.070914 secs 2024-04-26T18:38:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24477.070914 secs 2024-04-26T18:38:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:59 smithi033 ceph-mon[27845]: pgmap v13451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24479.951502 secs 2024-04-26T18:38:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:59 smithi096 ceph-mon[32753]: pgmap v13451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24479.951502 secs 2024-04-26T18:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:59 smithi194 ceph-mon[33017]: pgmap v13451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24479.951502 secs 2024-04-26T18:39:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:01 smithi033 ceph-mon[27845]: pgmap v13452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:01 smithi096 ceph-mon[32753]: pgmap v13452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:01 smithi194 ceph-mon[33017]: pgmap v13452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:03 smithi033 ceph-mon[27845]: pgmap v13453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24482.070995 secs 2024-04-26T18:39:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:03 smithi096 ceph-mon[32753]: pgmap v13453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24482.070995 secs 2024-04-26T18:39:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:03 smithi194 ceph-mon[33017]: pgmap v13453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24482.070995 secs 2024-04-26T18:39:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24484.951581 secs 2024-04-26T18:39:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24484.951581 secs 2024-04-26T18:39:04.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24484.951581 secs 2024-04-26T18:39:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:05 smithi033 ceph-mon[27845]: pgmap v13454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:05 smithi096 ceph-mon[32753]: pgmap v13454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:05 smithi194 ceph-mon[33017]: pgmap v13454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:05.822+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:05.822+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:05.822+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:39:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:07 smithi033 ceph-mon[27845]: pgmap v13455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:07 smithi096 ceph-mon[32753]: pgmap v13455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:07 smithi194 ceph-mon[33017]: pgmap v13455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24487.071084 secs 2024-04-26T18:39:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24487.071084 secs 2024-04-26T18:39:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24487.071084 secs 2024-04-26T18:39:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:09 smithi033 ceph-mon[27845]: pgmap v13456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24489.951654 secs 2024-04-26T18:39:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:09 smithi096 ceph-mon[32753]: pgmap v13456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24489.951654 secs 2024-04-26T18:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:09 smithi194 ceph-mon[33017]: pgmap v13456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24489.951654 secs 2024-04-26T18:39:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:11 smithi033 ceph-mon[27845]: pgmap v13457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:11 smithi096 ceph-mon[32753]: pgmap v13457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:11 smithi194 ceph-mon[33017]: pgmap v13457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:13 smithi033 ceph-mon[27845]: pgmap v13458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24492.071174 secs 2024-04-26T18:39:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:13 smithi096 ceph-mon[32753]: pgmap v13458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24492.071174 secs 2024-04-26T18:39:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:13 smithi194 ceph-mon[33017]: pgmap v13458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24492.071174 secs 2024-04-26T18:39:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24494.951793 secs 2024-04-26T18:39:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24494.951793 secs 2024-04-26T18:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24494.951793 secs 2024-04-26T18:39:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:15 smithi033 ceph-mon[27845]: pgmap v13459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:15 smithi096 ceph-mon[32753]: pgmap v13459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:15 smithi194 ceph-mon[33017]: pgmap v13459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:17 smithi096 ceph-mon[32753]: pgmap v13460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:17 smithi194 ceph-mon[33017]: pgmap v13460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:17 smithi033 ceph-mon[27845]: pgmap v13460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24497.071256 secs 2024-04-26T18:39:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24497.071256 secs 2024-04-26T18:39:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24497.071256 secs 2024-04-26T18:39:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:19 smithi194 ceph-mon[33017]: pgmap v13461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24499.951925 secs 2024-04-26T18:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:19 smithi033 ceph-mon[27845]: pgmap v13461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24499.951925 secs 2024-04-26T18:39:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:19 smithi096 ceph-mon[32753]: pgmap v13461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:39:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24499.951925 secs 2024-04-26T18:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:20.822+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:20.822+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:20.822+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:21 smithi194 ceph-mon[33017]: pgmap v13462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:21 smithi033 ceph-mon[27845]: pgmap v13462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:21 smithi096 ceph-mon[32753]: pgmap v13462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:23 smithi194 ceph-mon[33017]: pgmap v13463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24502.071311 secs 2024-04-26T18:39:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:23 smithi033 ceph-mon[27845]: pgmap v13463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24502.071311 secs 2024-04-26T18:39:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:23 smithi096 ceph-mon[32753]: pgmap v13463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24502.071311 secs 2024-04-26T18:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24504.952056 secs 2024-04-26T18:39:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24504.952056 secs 2024-04-26T18:39:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24504.952056 secs 2024-04-26T18:39:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:25 smithi033 ceph-mon[27845]: pgmap v13464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:25 smithi194 ceph-mon[33017]: pgmap v13464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:25 smithi096 ceph-mon[32753]: pgmap v13464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:27 smithi194 ceph-mon[33017]: pgmap v13465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:27 smithi033 ceph-mon[27845]: pgmap v13465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:27 smithi096 ceph-mon[32753]: pgmap v13465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24507.071374 secs 2024-04-26T18:39:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24507.071374 secs 2024-04-26T18:39:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24507.071374 secs 2024-04-26T18:39:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:29 smithi194 ceph-mon[33017]: pgmap v13466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24509.952172 secs 2024-04-26T18:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:29 smithi033 ceph-mon[27845]: pgmap v13466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24509.952172 secs 2024-04-26T18:39:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:29 smithi096 ceph-mon[32753]: pgmap v13466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24509.952172 secs 2024-04-26T18:39:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:31 smithi194 ceph-mon[33017]: pgmap v13467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:31 smithi033 ceph-mon[27845]: pgmap v13467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:31 smithi096 ceph-mon[32753]: pgmap v13467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:39:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:39:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:39:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:33 smithi194 ceph-mon[33017]: pgmap v13468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24512.071429 secs 2024-04-26T18:39:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:33 smithi033 ceph-mon[27845]: pgmap v13468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24512.071429 secs 2024-04-26T18:39:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:33 smithi096 ceph-mon[32753]: pgmap v13468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24512.071429 secs 2024-04-26T18:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24514.952290 secs 2024-04-26T18:39:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:39:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24514.952290 secs 2024-04-26T18:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24514.952290 secs 2024-04-26T18:39:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:35 smithi194 ceph-mon[33017]: pgmap v13469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:35 smithi194 ceph-mon[33017]: pgmap v13470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:35 smithi033 ceph-mon[27845]: pgmap v13469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:35 smithi033 ceph-mon[27845]: pgmap v13470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:35 smithi096 ceph-mon[32753]: pgmap v13469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:35 smithi096 ceph-mon[32753]: pgmap v13470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:35.830+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:35.830+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:35.830+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:39:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:36 smithi194 ceph-mon[33017]: pgmap v13471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:36 smithi033 ceph-mon[27845]: pgmap v13471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:36 smithi096 ceph-mon[32753]: pgmap v13471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:38 smithi194 ceph-mon[33017]: pgmap v13472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24517.071486 secs 2024-04-26T18:39:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:38 smithi033 ceph-mon[27845]: pgmap v13472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24517.071486 secs 2024-04-26T18:39:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:38 smithi096 ceph-mon[32753]: pgmap v13472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24517.071486 secs 2024-04-26T18:39:39.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24519.952412 secs 2024-04-26T18:39:39.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24519.952412 secs 2024-04-26T18:39:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24519.952412 secs 2024-04-26T18:39:40.594 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:40 smithi096 ceph-mon[32753]: pgmap v13473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-26T18:39:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:40 smithi194 ceph-mon[33017]: pgmap v13473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-26T18:39:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:40 smithi033 ceph-mon[27845]: pgmap v13473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-26T18:39:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:43 smithi194 ceph-mon[33017]: pgmap v13474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24522.071567 secs 2024-04-26T18:39:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:43 smithi033 ceph-mon[27845]: pgmap v13474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24522.071567 secs 2024-04-26T18:39:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:43 smithi096 ceph-mon[32753]: pgmap v13474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:39:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24522.071567 secs 2024-04-26T18:39:44.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24524.952529 secs 2024-04-26T18:39:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24524.952529 secs 2024-04-26T18:39:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24524.952529 secs 2024-04-26T18:39:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:45 smithi194 ceph-mon[33017]: pgmap v13475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:39:45.607 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:45 smithi033 ceph-mon[27845]: pgmap v13475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:39:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:45 smithi096 ceph-mon[32753]: pgmap v13475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:39:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:46 smithi194 ceph-mon[33017]: pgmap v13476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:46 smithi033 ceph-mon[27845]: pgmap v13476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:46 smithi096 ceph-mon[32753]: pgmap v13476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:39:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:48 smithi194 ceph-mon[33017]: pgmap v13477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24527.071650 secs 2024-04-26T18:39:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:48 smithi033 ceph-mon[27845]: pgmap v13477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24527.071650 secs 2024-04-26T18:39:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:48 smithi096 ceph-mon[32753]: pgmap v13477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24527.071650 secs 2024-04-26T18:39:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24529.952604 secs 2024-04-26T18:39:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24529.952604 secs 2024-04-26T18:39:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24529.952604 secs 2024-04-26T18:39:50.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:50 smithi194 ceph-mon[33017]: pgmap v13478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:50 smithi033 ceph-mon[27845]: pgmap v13478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:50 smithi096 ceph-mon[32753]: pgmap v13478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:50.841+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:50.842+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:39:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:39:50.842+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:39:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:52 smithi194 ceph-mon[33017]: pgmap v13479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:52 smithi033 ceph-mon[27845]: pgmap v13479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:52 smithi096 ceph-mon[32753]: pgmap v13479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:39:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24532.071731 secs 2024-04-26T18:39:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24532.071731 secs 2024-04-26T18:39:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24532.071731 secs 2024-04-26T18:39:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:54 smithi194 ceph-mon[33017]: pgmap v13480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24534.952681 secs 2024-04-26T18:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:54 smithi033 ceph-mon[27845]: pgmap v13480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24534.952681 secs 2024-04-26T18:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:54 smithi096 ceph-mon[32753]: pgmap v13480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24534.952681 secs 2024-04-26T18:39:56.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:56 smithi033 ceph-mon[27845]: pgmap v13481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:56 smithi194 ceph-mon[33017]: pgmap v13481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:56 smithi096 ceph-mon[32753]: pgmap v13481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:39:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:58 smithi194 ceph-mon[33017]: pgmap v13482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24537.071817 secs 2024-04-26T18:39:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:58 smithi033 ceph-mon[27845]: pgmap v13482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24537.071817 secs 2024-04-26T18:39:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:58 smithi096 ceph-mon[32753]: pgmap v13482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:39:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24537.071817 secs 2024-04-26T18:39:59.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24539.952800 secs 2024-04-26T18:39:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24539.952800 secs 2024-04-26T18:39:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24539.952800 secs 2024-04-26T18:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:00 smithi194 ceph-mon[33017]: pgmap v13483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:40:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:00 smithi033 ceph-mon[27845]: pgmap v13483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:00 smithi096 ceph-mon[32753]: pgmap v13483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:40:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:02 smithi194 ceph-mon[33017]: pgmap v13484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:02 smithi033 ceph-mon[27845]: pgmap v13484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:02 smithi096 ceph-mon[32753]: pgmap v13484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24542.071898 secs 2024-04-26T18:40:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24542.071898 secs 2024-04-26T18:40:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24542.071898 secs 2024-04-26T18:40:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:04 smithi033 ceph-mon[27845]: pgmap v13485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24544.952918 secs 2024-04-26T18:40:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:04 smithi096 ceph-mon[32753]: pgmap v13485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24544.952918 secs 2024-04-26T18:40:04.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:04 smithi194 ceph-mon[33017]: pgmap v13485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:04.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24544.952918 secs 2024-04-26T18:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:05.857+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:05.857+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:05.857+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:06 smithi033 ceph-mon[27845]: pgmap v13486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:06 smithi096 ceph-mon[32753]: pgmap v13486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:06 smithi194 ceph-mon[33017]: pgmap v13486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:08 smithi033 ceph-mon[27845]: pgmap v13487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24547.071995 secs 2024-04-26T18:40:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:08 smithi096 ceph-mon[32753]: pgmap v13487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24547.071995 secs 2024-04-26T18:40:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:08 smithi194 ceph-mon[33017]: pgmap v13487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24547.071995 secs 2024-04-26T18:40:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24549.953050 secs 2024-04-26T18:40:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24549.953050 secs 2024-04-26T18:40:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24549.953050 secs 2024-04-26T18:40:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:10 smithi033 ceph-mon[27845]: pgmap v13488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:10 smithi096 ceph-mon[32753]: pgmap v13488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:10 smithi194 ceph-mon[33017]: pgmap v13488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:12 smithi033 ceph-mon[27845]: pgmap v13489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:12 smithi096 ceph-mon[32753]: pgmap v13489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:12 smithi194 ceph-mon[33017]: pgmap v13489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24552.072079 secs 2024-04-26T18:40:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24552.072079 secs 2024-04-26T18:40:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24552.072079 secs 2024-04-26T18:40:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:14 smithi033 ceph-mon[27845]: pgmap v13490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24554.953171 secs 2024-04-26T18:40:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:14 smithi096 ceph-mon[32753]: pgmap v13490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24554.953171 secs 2024-04-26T18:40:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:14 smithi194 ceph-mon[33017]: pgmap v13490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24554.953171 secs 2024-04-26T18:40:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:16 smithi033 ceph-mon[27845]: pgmap v13491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:16 smithi096 ceph-mon[32753]: pgmap v13491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:16 smithi194 ceph-mon[33017]: pgmap v13491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:18 smithi033 ceph-mon[27845]: pgmap v13492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24557.072164 secs 2024-04-26T18:40:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:18 smithi096 ceph-mon[32753]: pgmap v13492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24557.072164 secs 2024-04-26T18:40:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:18 smithi194 ceph-mon[33017]: pgmap v13492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24557.072164 secs 2024-04-26T18:40:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24559.953306 secs 2024-04-26T18:40:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24559.953306 secs 2024-04-26T18:40:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24559.953306 secs 2024-04-26T18:40:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:20 smithi194 ceph-mon[33017]: pgmap v13493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:20.858+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:20.858+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:20.858+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:40:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:20 smithi033 ceph-mon[27845]: pgmap v13493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:20 smithi096 ceph-mon[32753]: pgmap v13493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:22 smithi194 ceph-mon[33017]: pgmap v13494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:22 smithi033 ceph-mon[27845]: pgmap v13494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:22 smithi096 ceph-mon[32753]: pgmap v13494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24562.072251 secs 2024-04-26T18:40:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24562.072251 secs 2024-04-26T18:40:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24562.072251 secs 2024-04-26T18:40:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:24 smithi194 ceph-mon[33017]: pgmap v13495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24564.953435 secs 2024-04-26T18:40:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:24 smithi033 ceph-mon[27845]: pgmap v13495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24564.953435 secs 2024-04-26T18:40:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:24 smithi096 ceph-mon[32753]: pgmap v13495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24564.953435 secs 2024-04-26T18:40:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:26 smithi194 ceph-mon[33017]: pgmap v13496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:26 smithi033 ceph-mon[27845]: pgmap v13496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:26 smithi096 ceph-mon[32753]: pgmap v13496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:28 smithi194 ceph-mon[33017]: pgmap v13497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24567.072333 secs 2024-04-26T18:40:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:28 smithi033 ceph-mon[27845]: pgmap v13497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24567.072333 secs 2024-04-26T18:40:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:28 smithi096 ceph-mon[32753]: pgmap v13497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24567.072333 secs 2024-04-26T18:40:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24569.953562 secs 2024-04-26T18:40:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24569.953562 secs 2024-04-26T18:40:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24569.953562 secs 2024-04-26T18:40:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:30 smithi194 ceph-mon[33017]: pgmap v13498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:30 smithi033 ceph-mon[27845]: pgmap v13498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:30 smithi096 ceph-mon[32753]: pgmap v13498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:32 smithi194 ceph-mon[33017]: pgmap v13499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:32 smithi033 ceph-mon[27845]: pgmap v13499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:32 smithi096 ceph-mon[32753]: pgmap v13499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24572.072393 secs 2024-04-26T18:40:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24572.072393 secs 2024-04-26T18:40:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24572.072393 secs 2024-04-26T18:40:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:34 smithi194 ceph-mon[33017]: pgmap v13500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24574.953659 secs 2024-04-26T18:40:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:40:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:34 smithi033 ceph-mon[27845]: pgmap v13500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24574.953659 secs 2024-04-26T18:40:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:40:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:34 smithi096 ceph-mon[32753]: pgmap v13500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24574.953659 secs 2024-04-26T18:40:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:40:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:40:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:40:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:35 smithi194 ceph-mon[33017]: pgmap v13501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:40:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:35 smithi194 ceph-mon[33017]: pgmap v13502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:40:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:40:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:40:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:35 smithi033 ceph-mon[27845]: pgmap v13501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:40:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:35 smithi033 ceph-mon[27845]: pgmap v13502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:40:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:40:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:40:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:35 smithi096 ceph-mon[32753]: pgmap v13501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:40:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:35 smithi096 ceph-mon[32753]: pgmap v13502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:40:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:35.874+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:35.874+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:35.874+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:40:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:38 smithi194 ceph-mon[33017]: pgmap v13503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:38 smithi033 ceph-mon[27845]: pgmap v13503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:38 smithi096 ceph-mon[32753]: pgmap v13503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:39.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24577.072453 secs 2024-04-26T18:40:39.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24579.953803 secs 2024-04-26T18:40:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24577.072453 secs 2024-04-26T18:40:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24579.953803 secs 2024-04-26T18:40:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24577.072453 secs 2024-04-26T18:40:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24579.953803 secs 2024-04-26T18:40:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:40 smithi194 ceph-mon[33017]: pgmap v13504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T18:40:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:40 smithi033 ceph-mon[27845]: pgmap v13504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T18:40:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:40 smithi096 ceph-mon[32753]: pgmap v13504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T18:40:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:42 smithi194 ceph-mon[33017]: pgmap v13505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:42 smithi033 ceph-mon[27845]: pgmap v13505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:42 smithi096 ceph-mon[32753]: pgmap v13505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24582.072534 secs 2024-04-26T18:40:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:44 smithi194 ceph-mon[33017]: pgmap v13506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24584.953920 secs 2024-04-26T18:40:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24582.072534 secs 2024-04-26T18:40:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:44 smithi033 ceph-mon[27845]: pgmap v13506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24584.953920 secs 2024-04-26T18:40:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24582.072534 secs 2024-04-26T18:40:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:44 smithi096 ceph-mon[32753]: pgmap v13506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:40:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24584.953920 secs 2024-04-26T18:40:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:46 smithi194 ceph-mon[33017]: pgmap v13507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:40:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:46 smithi033 ceph-mon[27845]: pgmap v13507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:40:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:46 smithi096 ceph-mon[32753]: pgmap v13507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:40:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:48 smithi194 ceph-mon[33017]: pgmap v13508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:48 smithi033 ceph-mon[27845]: pgmap v13508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:48 smithi096 ceph-mon[32753]: pgmap v13508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24587.072635 secs 2024-04-26T18:40:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24589.954042 secs 2024-04-26T18:40:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24587.072635 secs 2024-04-26T18:40:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24589.954042 secs 2024-04-26T18:40:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24587.072635 secs 2024-04-26T18:40:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24589.954042 secs 2024-04-26T18:40:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:50 smithi033 ceph-mon[27845]: pgmap v13509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:50 smithi096 ceph-mon[32753]: pgmap v13509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:50 smithi194 ceph-mon[33017]: pgmap v13509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:50.883+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:50.883+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:40:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:40:50.883+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:40:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:52 smithi033 ceph-mon[27845]: pgmap v13510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:52 smithi096 ceph-mon[32753]: pgmap v13510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:52 smithi194 ceph-mon[33017]: pgmap v13510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24592.072715 secs 2024-04-26T18:40:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24592.072715 secs 2024-04-26T18:40:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24592.072715 secs 2024-04-26T18:40:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:54 smithi033 ceph-mon[27845]: pgmap v13511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24594.954161 secs 2024-04-26T18:40:54.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:54 smithi096 ceph-mon[32753]: pgmap v13511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:54.435 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24594.954161 secs 2024-04-26T18:40:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:54 smithi194 ceph-mon[33017]: pgmap v13511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24594.954161 secs 2024-04-26T18:40:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:56 smithi033 ceph-mon[27845]: pgmap v13512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:56 smithi096 ceph-mon[32753]: pgmap v13512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:56 smithi194 ceph-mon[33017]: pgmap v13512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:40:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:58 smithi033 ceph-mon[27845]: pgmap v13513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24597.072799 secs 2024-04-26T18:40:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:58 smithi096 ceph-mon[32753]: pgmap v13513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24597.072799 secs 2024-04-26T18:40:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:58 smithi194 ceph-mon[33017]: pgmap v13513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:40:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24597.072799 secs 2024-04-26T18:40:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:40:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24599.954282 secs 2024-04-26T18:40:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:40:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24599.954282 secs 2024-04-26T18:40:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:40:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24599.954282 secs 2024-04-26T18:41:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:00 smithi033 ceph-mon[27845]: pgmap v13514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:00 smithi096 ceph-mon[32753]: pgmap v13514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:00 smithi194 ceph-mon[33017]: pgmap v13514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:02 smithi033 ceph-mon[27845]: pgmap v13515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:02 smithi096 ceph-mon[32753]: pgmap v13515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:02 smithi194 ceph-mon[33017]: pgmap v13515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24602.072882 secs 2024-04-26T18:41:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24602.072882 secs 2024-04-26T18:41:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24602.072882 secs 2024-04-26T18:41:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:04 smithi033 ceph-mon[27845]: pgmap v13516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24604.954391 secs 2024-04-26T18:41:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:04 smithi096 ceph-mon[32753]: pgmap v13516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24604.954391 secs 2024-04-26T18:41:04.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:04 smithi194 ceph-mon[33017]: pgmap v13516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:04.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24604.954391 secs 2024-04-26T18:41:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:05.896+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:05.896+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:05.896+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:41:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:06 smithi096 ceph-mon[32753]: pgmap v13517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:06 smithi194 ceph-mon[33017]: pgmap v13517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:06 smithi033 ceph-mon[27845]: pgmap v13517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:08 smithi194 ceph-mon[33017]: pgmap v13518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24607.072989 secs 2024-04-26T18:41:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:08 smithi033 ceph-mon[27845]: pgmap v13518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24607.072989 secs 2024-04-26T18:41:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:08 smithi096 ceph-mon[32753]: pgmap v13518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24607.072989 secs 2024-04-26T18:41:09.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24609.954511 secs 2024-04-26T18:41:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24609.954511 secs 2024-04-26T18:41:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24609.954511 secs 2024-04-26T18:41:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:10 smithi194 ceph-mon[33017]: pgmap v13519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:10 smithi033 ceph-mon[27845]: pgmap v13519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:10 smithi096 ceph-mon[32753]: pgmap v13519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:12 smithi194 ceph-mon[33017]: pgmap v13520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:12 smithi033 ceph-mon[27845]: pgmap v13520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:12 smithi096 ceph-mon[32753]: pgmap v13520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24612.073055 secs 2024-04-26T18:41:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24612.073055 secs 2024-04-26T18:41:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24612.073055 secs 2024-04-26T18:41:14.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:14 smithi033 ceph-mon[27845]: pgmap v13521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:14.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24614.954623 secs 2024-04-26T18:41:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:14 smithi194 ceph-mon[33017]: pgmap v13521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24614.954623 secs 2024-04-26T18:41:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:14 smithi096 ceph-mon[32753]: pgmap v13521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24614.954623 secs 2024-04-26T18:41:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:16 smithi194 ceph-mon[33017]: pgmap v13522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:16 smithi033 ceph-mon[27845]: pgmap v13522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:16 smithi096 ceph-mon[32753]: pgmap v13522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:18 smithi194 ceph-mon[33017]: pgmap v13523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24617.073141 secs 2024-04-26T18:41:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:18 smithi033 ceph-mon[27845]: pgmap v13523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24617.073141 secs 2024-04-26T18:41:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:18 smithi096 ceph-mon[32753]: pgmap v13523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24617.073141 secs 2024-04-26T18:41:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24619.954745 secs 2024-04-26T18:41:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24619.954745 secs 2024-04-26T18:41:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24619.954745 secs 2024-04-26T18:41:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:20 smithi194 ceph-mon[33017]: pgmap v13524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:20 smithi033 ceph-mon[27845]: pgmap v13524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:20 smithi096 ceph-mon[32753]: pgmap v13524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:20.908+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:20.908+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:41:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:20.908+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef007e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef007e000 comp rx=0 tx=0).stop 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:41:21.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.783+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:41:21.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:21.785+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0043050 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:41:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:22 smithi194 ceph-mon[33017]: pgmap v13525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:22 smithi033 ceph-mon[27845]: pgmap v13525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:22 smithi096 ceph-mon[32753]: pgmap v13525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24622.073230 secs 2024-04-26T18:41:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24622.073230 secs 2024-04-26T18:41:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24622.073230 secs 2024-04-26T18:41:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:24 smithi194 ceph-mon[33017]: pgmap v13526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24624.954823 secs 2024-04-26T18:41:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:24 smithi033 ceph-mon[27845]: pgmap v13526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24624.954823 secs 2024-04-26T18:41:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:24 smithi096 ceph-mon[32753]: pgmap v13526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24624.954823 secs 2024-04-26T18:41:25.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:25.985+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:41:25.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:25.985+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:41:25.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:25.985+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 195 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:41:25.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:25.985+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 195 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T18:41:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:26 smithi194 ceph-mon[33017]: pgmap v13527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:26 smithi033 ceph-mon[27845]: pgmap v13527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:26 smithi096 ceph-mon[32753]: pgmap v13527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:28 smithi194 ceph-mon[33017]: pgmap v13528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24627.073292 secs 2024-04-26T18:41:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:28 smithi033 ceph-mon[27845]: pgmap v13528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24627.073292 secs 2024-04-26T18:41:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:28 smithi096 ceph-mon[32753]: pgmap v13528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24627.073292 secs 2024-04-26T18:41:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24629.954948 secs 2024-04-26T18:41:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24629.954948 secs 2024-04-26T18:41:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24629.954948 secs 2024-04-26T18:41:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:30 smithi194 ceph-mon[33017]: pgmap v13529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:30 smithi033 ceph-mon[27845]: pgmap v13529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:30 smithi096 ceph-mon[32753]: pgmap v13529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:32 smithi194 ceph-mon[33017]: pgmap v13530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:32 smithi033 ceph-mon[27845]: pgmap v13530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:32 smithi096 ceph-mon[32753]: pgmap v13530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24632.073377 secs 2024-04-26T18:41:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24632.073377 secs 2024-04-26T18:41:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24632.073377 secs 2024-04-26T18:41:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:34 smithi194 ceph-mon[33017]: pgmap v13531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24634.955083 secs 2024-04-26T18:41:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:34 smithi033 ceph-mon[27845]: pgmap v13531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24634.955083 secs 2024-04-26T18:41:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:34 smithi096 ceph-mon[32753]: pgmap v13531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24634.955083 secs 2024-04-26T18:41:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:35 smithi194 ceph-mon[33017]: pgmap v13532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:35 smithi033 ceph-mon[27845]: pgmap v13532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:35 smithi096 ceph-mon[32753]: pgmap v13532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:35.922+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:35.922+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:35.922+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:41:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:41:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:41:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:41:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:41:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:41:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:37 smithi194 ceph-mon[33017]: pgmap v13533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:37 smithi194 ceph-mon[33017]: pgmap v13534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:41:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:41:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:41:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:37 smithi033 ceph-mon[27845]: pgmap v13533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:37 smithi033 ceph-mon[27845]: pgmap v13534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:41:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:41:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:41:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:37 smithi096 ceph-mon[32753]: pgmap v13533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:37 smithi096 ceph-mon[32753]: pgmap v13534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:41:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24637.073451 secs 2024-04-26T18:41:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24637.073451 secs 2024-04-26T18:41:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24637.073451 secs 2024-04-26T18:41:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24639.955210 secs 2024-04-26T18:41:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:39 smithi194 ceph-mon[33017]: pgmap v13535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24639.955210 secs 2024-04-26T18:41:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:39 smithi033 ceph-mon[27845]: pgmap v13535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24639.955210 secs 2024-04-26T18:41:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:39 smithi096 ceph-mon[32753]: pgmap v13535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:41 smithi194 ceph-mon[33017]: pgmap v13536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:41 smithi033 ceph-mon[27845]: pgmap v13536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:41 smithi096 ceph-mon[32753]: pgmap v13536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:43 smithi194 ceph-mon[33017]: pgmap v13537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24642.073535 secs 2024-04-26T18:41:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:43 smithi033 ceph-mon[27845]: pgmap v13537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24642.073535 secs 2024-04-26T18:41:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:43 smithi096 ceph-mon[32753]: pgmap v13537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24642.073535 secs 2024-04-26T18:41:45.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24644.955287 secs 2024-04-26T18:41:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24644.955287 secs 2024-04-26T18:41:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24644.955287 secs 2024-04-26T18:41:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:45 smithi194 ceph-mon[33017]: pgmap v13538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:45 smithi033 ceph-mon[27845]: pgmap v13538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:45 smithi096 ceph-mon[32753]: pgmap v13538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:41:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:47 smithi194 ceph-mon[33017]: pgmap v13539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:41:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:47 smithi033 ceph-mon[27845]: pgmap v13539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:41:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:47 smithi096 ceph-mon[32753]: pgmap v13539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:41:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24647.073596 secs 2024-04-26T18:41:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24647.073596 secs 2024-04-26T18:41:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24647.073596 secs 2024-04-26T18:41:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24649.955421 secs 2024-04-26T18:41:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:49 smithi194 ceph-mon[33017]: pgmap v13540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24649.955421 secs 2024-04-26T18:41:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:49 smithi033 ceph-mon[27845]: pgmap v13540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24649.955421 secs 2024-04-26T18:41:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:49 smithi096 ceph-mon[32753]: pgmap v13540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:50.928+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:50.928+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:41:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:41:50.928+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:41:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:51 smithi194 ceph-mon[33017]: pgmap v13541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:52.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:51 smithi033 ceph-mon[27845]: pgmap v13541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:51 smithi096 ceph-mon[32753]: pgmap v13541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:53 smithi194 ceph-mon[33017]: pgmap v13542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24652.073682 secs 2024-04-26T18:41:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24654.955542 secs 2024-04-26T18:41:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:53 smithi033 ceph-mon[27845]: pgmap v13542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24652.073682 secs 2024-04-26T18:41:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24654.955542 secs 2024-04-26T18:41:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:53 smithi096 ceph-mon[32753]: pgmap v13542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24652.073682 secs 2024-04-26T18:41:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24654.955542 secs 2024-04-26T18:41:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:55 smithi194 ceph-mon[33017]: pgmap v13543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:55 smithi033 ceph-mon[27845]: pgmap v13543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:55 smithi096 ceph-mon[32753]: pgmap v13543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:41:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:57 smithi194 ceph-mon[33017]: pgmap v13544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:57 smithi033 ceph-mon[27845]: pgmap v13544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:57 smithi096 ceph-mon[32753]: pgmap v13544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:41:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24657.073747 secs 2024-04-26T18:41:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24659.955661 secs 2024-04-26T18:41:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24657.073747 secs 2024-04-26T18:41:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24659.955661 secs 2024-04-26T18:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24657.073747 secs 2024-04-26T18:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24659.955661 secs 2024-04-26T18:42:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:41:59 smithi194 ceph-mon[33017]: pgmap v13545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:41:59 smithi033 ceph-mon[27845]: pgmap v13545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:41:59 smithi096 ceph-mon[32753]: pgmap v13545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:01 smithi194 ceph-mon[33017]: pgmap v13546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:01 smithi033 ceph-mon[27845]: pgmap v13546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:01 smithi096 ceph-mon[32753]: pgmap v13546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:03 smithi194 ceph-mon[33017]: pgmap v13547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24662.073831 secs 2024-04-26T18:42:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24664.955783 secs 2024-04-26T18:42:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:03 smithi033 ceph-mon[27845]: pgmap v13547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24662.073831 secs 2024-04-26T18:42:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24664.955783 secs 2024-04-26T18:42:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:03 smithi096 ceph-mon[32753]: pgmap v13547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24662.073831 secs 2024-04-26T18:42:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24664.955783 secs 2024-04-26T18:42:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:05.931+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:05.931+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:42:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:05.931+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:05 smithi194 ceph-mon[33017]: pgmap v13548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:05 smithi033 ceph-mon[27845]: pgmap v13548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:05 smithi096 ceph-mon[32753]: pgmap v13548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:07 smithi194 ceph-mon[33017]: pgmap v13549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:07 smithi033 ceph-mon[27845]: pgmap v13549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:07 smithi096 ceph-mon[32753]: pgmap v13549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24667.073892 secs 2024-04-26T18:42:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24669.955856 secs 2024-04-26T18:42:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24667.073892 secs 2024-04-26T18:42:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24669.955856 secs 2024-04-26T18:42:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24667.073892 secs 2024-04-26T18:42:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24669.955856 secs 2024-04-26T18:42:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:09 smithi194 ceph-mon[33017]: pgmap v13550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:09 smithi033 ceph-mon[27845]: pgmap v13550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:09 smithi096 ceph-mon[32753]: pgmap v13550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:10.993+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:42:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:10.993+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:42:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:10.994+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 196 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:42:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:10.994+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 196 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T18:42:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:11 smithi194 ceph-mon[33017]: pgmap v13551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:11 smithi033 ceph-mon[27845]: pgmap v13551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:11 smithi096 ceph-mon[32753]: pgmap v13551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:13 smithi033 ceph-mon[27845]: pgmap v13552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24672.073975 secs 2024-04-26T18:42:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24674.955989 secs 2024-04-26T18:42:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:13 smithi096 ceph-mon[32753]: pgmap v13552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24672.073975 secs 2024-04-26T18:42:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24674.955989 secs 2024-04-26T18:42:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:13 smithi194 ceph-mon[33017]: pgmap v13552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24672.073975 secs 2024-04-26T18:42:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24674.955989 secs 2024-04-26T18:42:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:15 smithi033 ceph-mon[27845]: pgmap v13553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:15 smithi096 ceph-mon[32753]: pgmap v13553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:15 smithi194 ceph-mon[33017]: pgmap v13553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:17 smithi033 ceph-mon[27845]: pgmap v13554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:17 smithi096 ceph-mon[32753]: pgmap v13554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:17 smithi194 ceph-mon[33017]: pgmap v13554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24677.074064 secs 2024-04-26T18:42:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24679.956133 secs 2024-04-26T18:42:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24677.074064 secs 2024-04-26T18:42:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24679.956133 secs 2024-04-26T18:42:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24677.074064 secs 2024-04-26T18:42:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24679.956133 secs 2024-04-26T18:42:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:19 smithi033 ceph-mon[27845]: pgmap v13555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:19 smithi096 ceph-mon[32753]: pgmap v13555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:19 smithi194 ceph-mon[33017]: pgmap v13555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:20.938+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:20.939+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:20.939+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:42:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:21 smithi033 ceph-mon[27845]: pgmap v13556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:21 smithi096 ceph-mon[32753]: pgmap v13556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:21 smithi194 ceph-mon[33017]: pgmap v13556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:23 smithi033 ceph-mon[27845]: pgmap v13557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24682.074149 secs 2024-04-26T18:42:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24684.956251 secs 2024-04-26T18:42:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:23 smithi096 ceph-mon[32753]: pgmap v13557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24682.074149 secs 2024-04-26T18:42:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24684.956251 secs 2024-04-26T18:42:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:23 smithi194 ceph-mon[33017]: pgmap v13557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24682.074149 secs 2024-04-26T18:42:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24684.956251 secs 2024-04-26T18:42:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:25 smithi033 ceph-mon[27845]: pgmap v13558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:25 smithi096 ceph-mon[32753]: pgmap v13558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:25 smithi194 ceph-mon[33017]: pgmap v13558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:27 smithi033 ceph-mon[27845]: pgmap v13559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:27 smithi096 ceph-mon[32753]: pgmap v13559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:28.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:27 smithi194 ceph-mon[33017]: pgmap v13559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24687.074210 secs 2024-04-26T18:42:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24689.956378 secs 2024-04-26T18:42:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24687.074210 secs 2024-04-26T18:42:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24689.956378 secs 2024-04-26T18:42:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24687.074210 secs 2024-04-26T18:42:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24689.956378 secs 2024-04-26T18:42:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:29 smithi033 ceph-mon[27845]: pgmap v13560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:29 smithi096 ceph-mon[32753]: pgmap v13560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:29 smithi194 ceph-mon[33017]: pgmap v13560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:31 smithi033 ceph-mon[27845]: pgmap v13561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:31 smithi096 ceph-mon[32753]: pgmap v13561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:31 smithi194 ceph-mon[33017]: pgmap v13561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:33 smithi096 ceph-mon[32753]: pgmap v13562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24692.074294 secs 2024-04-26T18:42:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24694.956503 secs 2024-04-26T18:42:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:33 smithi194 ceph-mon[33017]: pgmap v13562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24692.074294 secs 2024-04-26T18:42:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24694.956503 secs 2024-04-26T18:42:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:33 smithi033 ceph-mon[27845]: pgmap v13562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24692.074294 secs 2024-04-26T18:42:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24694.956503 secs 2024-04-26T18:42:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:35.955+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:35.955+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:35.955+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:42:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:35 smithi096 ceph-mon[32753]: pgmap v13563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:35 smithi194 ceph-mon[33017]: pgmap v13563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:35 smithi033 ceph-mon[27845]: pgmap v13563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:37 smithi194 ceph-mon[33017]: pgmap v13564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:42:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:37 smithi033 ceph-mon[27845]: pgmap v13564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:42:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:37 smithi096 ceph-mon[32753]: pgmap v13564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:42:39.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24697.074378 secs 2024-04-26T18:42:39.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:42:39.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:42:39.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:42:39.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24699.956628 secs 2024-04-26T18:42:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24697.074378 secs 2024-04-26T18:42:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:42:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:42:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:42:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24699.956628 secs 2024-04-26T18:42:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24697.074378 secs 2024-04-26T18:42:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:42:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:42:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:42:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24699.956628 secs 2024-04-26T18:42:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:39 smithi194 ceph-mon[33017]: pgmap v13565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:39 smithi194 ceph-mon[33017]: pgmap v13566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:39 smithi033 ceph-mon[27845]: pgmap v13565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:39 smithi033 ceph-mon[27845]: pgmap v13566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:39 smithi096 ceph-mon[32753]: pgmap v13565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:39 smithi096 ceph-mon[32753]: pgmap v13566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:41 smithi194 ceph-mon[33017]: pgmap v13567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:41 smithi033 ceph-mon[27845]: pgmap v13567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:41 smithi096 ceph-mon[32753]: pgmap v13567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:43 smithi194 ceph-mon[33017]: pgmap v13568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24702.074461 secs 2024-04-26T18:42:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24704.956750 secs 2024-04-26T18:42:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:43 smithi033 ceph-mon[27845]: pgmap v13568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24702.074461 secs 2024-04-26T18:42:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24704.956750 secs 2024-04-26T18:42:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:43 smithi096 ceph-mon[32753]: pgmap v13568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24702.074461 secs 2024-04-26T18:42:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24704.956750 secs 2024-04-26T18:42:46.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:45 smithi033 ceph-mon[27845]: pgmap v13569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:42:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:45 smithi194 ceph-mon[33017]: pgmap v13569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:42:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:45 smithi096 ceph-mon[32753]: pgmap v13569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:42:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:47 smithi194 ceph-mon[33017]: pgmap v13570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:47 smithi033 ceph-mon[27845]: pgmap v13570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:47 smithi096 ceph-mon[32753]: pgmap v13570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:42:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24707.074546 secs 2024-04-26T18:42:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24709.956871 secs 2024-04-26T18:42:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24707.074546 secs 2024-04-26T18:42:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24709.956871 secs 2024-04-26T18:42:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24707.074546 secs 2024-04-26T18:42:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24709.956871 secs 2024-04-26T18:42:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:49 smithi194 ceph-mon[33017]: pgmap v13571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:42:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:49 smithi033 ceph-mon[27845]: pgmap v13571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:42:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:49 smithi096 ceph-mon[32753]: pgmap v13571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:42:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:50.966+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:50.967+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:42:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:42:50.967+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:42:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:51 smithi194 ceph-mon[33017]: pgmap v13572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:51 smithi033 ceph-mon[27845]: pgmap v13572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:51 smithi096 ceph-mon[32753]: pgmap v13572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:53 smithi194 ceph-mon[33017]: pgmap v13573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24712.074629 secs 2024-04-26T18:42:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24714.956991 secs 2024-04-26T18:42:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:53 smithi033 ceph-mon[27845]: pgmap v13573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24712.074629 secs 2024-04-26T18:42:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24714.956991 secs 2024-04-26T18:42:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:53 smithi096 ceph-mon[32753]: pgmap v13573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24712.074629 secs 2024-04-26T18:42:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24714.956991 secs 2024-04-26T18:42:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:55 smithi194 ceph-mon[33017]: pgmap v13574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:55 smithi033 ceph-mon[27845]: pgmap v13574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:55 smithi096 ceph-mon[32753]: pgmap v13574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:42:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:58 smithi194 ceph-mon[33017]: pgmap v13575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:58 smithi033 ceph-mon[27845]: pgmap v13575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:58 smithi096 ceph-mon[32753]: pgmap v13575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:42:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24717.074714 secs 2024-04-26T18:42:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24719.957136 secs 2024-04-26T18:42:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24717.074714 secs 2024-04-26T18:42:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24719.957136 secs 2024-04-26T18:42:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24717.074714 secs 2024-04-26T18:42:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24719.957136 secs 2024-04-26T18:43:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:00 smithi194 ceph-mon[33017]: pgmap v13576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:00 smithi033 ceph-mon[27845]: pgmap v13576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:00 smithi096 ceph-mon[32753]: pgmap v13576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:02 smithi194 ceph-mon[33017]: pgmap v13577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:02 smithi033 ceph-mon[27845]: pgmap v13577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:02 smithi096 ceph-mon[32753]: pgmap v13577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:04 smithi194 ceph-mon[33017]: pgmap v13578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24722.074798 secs 2024-04-26T18:43:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24724.957202 secs 2024-04-26T18:43:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:04 smithi033 ceph-mon[27845]: pgmap v13578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24722.074798 secs 2024-04-26T18:43:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24724.957202 secs 2024-04-26T18:43:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:04 smithi096 ceph-mon[32753]: pgmap v13578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24722.074798 secs 2024-04-26T18:43:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24724.957202 secs 2024-04-26T18:43:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:05.982+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:05.982+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:05.982+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:43:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:06 smithi194 ceph-mon[33017]: pgmap v13579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:06 smithi033 ceph-mon[27845]: pgmap v13579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:06 smithi096 ceph-mon[32753]: pgmap v13579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:08 smithi194 ceph-mon[33017]: pgmap v13580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:08 smithi033 ceph-mon[27845]: pgmap v13580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:08 smithi096 ceph-mon[32753]: pgmap v13580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:09.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24727.074884 secs 2024-04-26T18:43:09.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24729.957303 secs 2024-04-26T18:43:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24727.074884 secs 2024-04-26T18:43:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24729.957303 secs 2024-04-26T18:43:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24727.074884 secs 2024-04-26T18:43:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24729.957303 secs 2024-04-26T18:43:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:10 smithi194 ceph-mon[33017]: pgmap v13581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:10 smithi033 ceph-mon[27845]: pgmap v13581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:10 smithi096 ceph-mon[32753]: pgmap v13581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:12 smithi194 ceph-mon[33017]: pgmap v13582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:12 smithi033 ceph-mon[27845]: pgmap v13582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:12 smithi096 ceph-mon[32753]: pgmap v13582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:14.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:14 smithi194 ceph-mon[33017]: pgmap v13583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:14.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24732.074991 secs 2024-04-26T18:43:14.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24734.957380 secs 2024-04-26T18:43:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:14 smithi033 ceph-mon[27845]: pgmap v13583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24732.074991 secs 2024-04-26T18:43:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24734.957380 secs 2024-04-26T18:43:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:14 smithi096 ceph-mon[32753]: pgmap v13583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24732.074991 secs 2024-04-26T18:43:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24734.957380 secs 2024-04-26T18:43:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:16 smithi194 ceph-mon[33017]: pgmap v13584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:16 smithi033 ceph-mon[27845]: pgmap v13584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:16 smithi096 ceph-mon[32753]: pgmap v13584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:18 smithi194 ceph-mon[33017]: pgmap v13585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:18 smithi033 ceph-mon[27845]: pgmap v13585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:18 smithi096 ceph-mon[32753]: pgmap v13585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24737.075050 secs 2024-04-26T18:43:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24739.957502 secs 2024-04-26T18:43:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24737.075050 secs 2024-04-26T18:43:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24739.957502 secs 2024-04-26T18:43:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24737.075050 secs 2024-04-26T18:43:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24739.957502 secs 2024-04-26T18:43:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:20 smithi194 ceph-mon[33017]: pgmap v13586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:20 smithi033 ceph-mon[27845]: pgmap v13586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:20 smithi096 ceph-mon[32753]: pgmap v13586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:20.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:20.997+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:20.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:20.997+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:20.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:20.997+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:43:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:22 smithi194 ceph-mon[33017]: pgmap v13587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:22 smithi033 ceph-mon[27845]: pgmap v13587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:22 smithi096 ceph-mon[32753]: pgmap v13587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:24 smithi194 ceph-mon[33017]: pgmap v13588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24742.075109 secs 2024-04-26T18:43:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24744.957604 secs 2024-04-26T18:43:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:24 smithi033 ceph-mon[27845]: pgmap v13588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24742.075109 secs 2024-04-26T18:43:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24744.957604 secs 2024-04-26T18:43:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:24 smithi096 ceph-mon[32753]: pgmap v13588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24742.075109 secs 2024-04-26T18:43:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24744.957604 secs 2024-04-26T18:43:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:26 smithi194 ceph-mon[33017]: pgmap v13589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:26 smithi033 ceph-mon[27845]: pgmap v13589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:26 smithi096 ceph-mon[32753]: pgmap v13589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:28 smithi194 ceph-mon[33017]: pgmap v13590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:28 smithi033 ceph-mon[27845]: pgmap v13590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:28 smithi096 ceph-mon[32753]: pgmap v13590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24747.075197 secs 2024-04-26T18:43:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24749.957725 secs 2024-04-26T18:43:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24747.075197 secs 2024-04-26T18:43:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24749.957725 secs 2024-04-26T18:43:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24747.075197 secs 2024-04-26T18:43:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24749.957725 secs 2024-04-26T18:43:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:30 smithi033 ceph-mon[27845]: pgmap v13591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:30 smithi096 ceph-mon[32753]: pgmap v13591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:30 smithi194 ceph-mon[33017]: pgmap v13591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:32 smithi033 ceph-mon[27845]: pgmap v13592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:32 smithi096 ceph-mon[32753]: pgmap v13592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:32 smithi194 ceph-mon[33017]: pgmap v13592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:34 smithi033 ceph-mon[27845]: pgmap v13593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24752.075279 secs 2024-04-26T18:43:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24754.957834 secs 2024-04-26T18:43:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:34 smithi096 ceph-mon[32753]: pgmap v13593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24752.075279 secs 2024-04-26T18:43:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24754.957834 secs 2024-04-26T18:43:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:34 smithi194 ceph-mon[33017]: pgmap v13593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24752.075279 secs 2024-04-26T18:43:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24754.957834 secs 2024-04-26T18:43:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:36.013+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:36.013+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:36.013+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:43:36.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:36 smithi194 ceph-mon[33017]: pgmap v13594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:36 smithi033 ceph-mon[27845]: pgmap v13594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:36 smithi096 ceph-mon[32753]: pgmap v13594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:38 smithi033 ceph-mon[27845]: pgmap v13595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24757.075359 secs 2024-04-26T18:43:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:38 smithi096 ceph-mon[32753]: pgmap v13595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24757.075359 secs 2024-04-26T18:43:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:38 smithi194 ceph-mon[33017]: pgmap v13595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24757.075359 secs 2024-04-26T18:43:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24759.957953 secs 2024-04-26T18:43:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:43:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24759.957953 secs 2024-04-26T18:43:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:43:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24759.957953 secs 2024-04-26T18:43:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:43:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:40 smithi033 ceph-mon[27845]: pgmap v13596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:43:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:43:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:43:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:40 smithi096 ceph-mon[32753]: pgmap v13596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:43:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:43:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:43:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:40 smithi194 ceph-mon[33017]: pgmap v13596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:43:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:43:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:43:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:41 smithi033 ceph-mon[27845]: pgmap v13597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:43:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:41 smithi033 ceph-mon[27845]: pgmap v13598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:41 smithi096 ceph-mon[32753]: pgmap v13597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:43:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:41 smithi096 ceph-mon[32753]: pgmap v13598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:41 smithi194 ceph-mon[33017]: pgmap v13597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:43:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:41 smithi194 ceph-mon[33017]: pgmap v13598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:43 smithi033 ceph-mon[27845]: pgmap v13599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24762.075419 secs 2024-04-26T18:43:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:43 smithi096 ceph-mon[32753]: pgmap v13599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24762.075419 secs 2024-04-26T18:43:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:43 smithi194 ceph-mon[33017]: pgmap v13599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24762.075419 secs 2024-04-26T18:43:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24764.958075 secs 2024-04-26T18:43:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24764.958075 secs 2024-04-26T18:43:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24764.958075 secs 2024-04-26T18:43:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:45 smithi096 ceph-mon[32753]: pgmap v13600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:43:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:45 smithi194 ceph-mon[33017]: pgmap v13600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:43:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:45 smithi033 ceph-mon[27845]: pgmap v13600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:43:47.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:47 smithi033 ceph-mon[27845]: pgmap v13601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:47 smithi194 ceph-mon[33017]: pgmap v13601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:47 smithi096 ceph-mon[32753]: pgmap v13601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24767.075506 secs 2024-04-26T18:43:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24767.075506 secs 2024-04-26T18:43:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24767.075506 secs 2024-04-26T18:43:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:49 smithi194 ceph-mon[33017]: pgmap v13602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24769.958195 secs 2024-04-26T18:43:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:49 smithi033 ceph-mon[27845]: pgmap v13602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24769.958195 secs 2024-04-26T18:43:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:49 smithi096 ceph-mon[32753]: pgmap v13602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:43:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24769.958195 secs 2024-04-26T18:43:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:51.027+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:51.027+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:43:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:43:51.027+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:43:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:51 smithi194 ceph-mon[33017]: pgmap v13603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:43:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:51 smithi033 ceph-mon[27845]: pgmap v13603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:43:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:51 smithi096 ceph-mon[32753]: pgmap v13603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:43:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:53 smithi194 ceph-mon[33017]: pgmap v13604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24772.075589 secs 2024-04-26T18:43:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:53 smithi033 ceph-mon[27845]: pgmap v13604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24772.075589 secs 2024-04-26T18:43:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:53 smithi096 ceph-mon[32753]: pgmap v13604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24772.075589 secs 2024-04-26T18:43:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24774.958315 secs 2024-04-26T18:43:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24774.958315 secs 2024-04-26T18:43:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24774.958315 secs 2024-04-26T18:43:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:55 smithi194 ceph-mon[33017]: pgmap v13605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:55 smithi033 ceph-mon[27845]: pgmap v13605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:55 smithi096 ceph-mon[32753]: pgmap v13605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:57 smithi194 ceph-mon[33017]: pgmap v13606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:57 smithi033 ceph-mon[27845]: pgmap v13606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:57 smithi096 ceph-mon[32753]: pgmap v13606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:43:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24777.075668 secs 2024-04-26T18:43:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24777.075668 secs 2024-04-26T18:43:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24777.075668 secs 2024-04-26T18:43:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:59 smithi194 ceph-mon[33017]: pgmap v13607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24779.958402 secs 2024-04-26T18:43:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:59 smithi033 ceph-mon[27845]: pgmap v13607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24779.958402 secs 2024-04-26T18:43:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:59 smithi096 ceph-mon[32753]: pgmap v13607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:43:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24779.958402 secs 2024-04-26T18:44:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:01 smithi194 ceph-mon[33017]: pgmap v13608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:01 smithi033 ceph-mon[27845]: pgmap v13608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:01 smithi096 ceph-mon[32753]: pgmap v13608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:03 smithi194 ceph-mon[33017]: pgmap v13609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24782.075751 secs 2024-04-26T18:44:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:03 smithi033 ceph-mon[27845]: pgmap v13609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24782.075751 secs 2024-04-26T18:44:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:03 smithi096 ceph-mon[32753]: pgmap v13609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24782.075751 secs 2024-04-26T18:44:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24784.958544 secs 2024-04-26T18:44:04.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24784.958544 secs 2024-04-26T18:44:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24784.958544 secs 2024-04-26T18:44:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:05 smithi194 ceph-mon[33017]: pgmap v13610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:05 smithi033 ceph-mon[27845]: pgmap v13610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:05 smithi096 ceph-mon[32753]: pgmap v13610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:06.030+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:06.030+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:06.030+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:44:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:07 smithi194 ceph-mon[33017]: pgmap v13611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:07 smithi033 ceph-mon[27845]: pgmap v13611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:07 smithi096 ceph-mon[32753]: pgmap v13611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24787.075835 secs 2024-04-26T18:44:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24787.075835 secs 2024-04-26T18:44:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24787.075835 secs 2024-04-26T18:44:09.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:09 smithi096 ceph-mon[32753]: pgmap v13612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:09.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24789.958664 secs 2024-04-26T18:44:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:09 smithi194 ceph-mon[33017]: pgmap v13612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24789.958664 secs 2024-04-26T18:44:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:09 smithi033 ceph-mon[27845]: pgmap v13612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24789.958664 secs 2024-04-26T18:44:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:11 smithi194 ceph-mon[33017]: pgmap v13613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:11 smithi033 ceph-mon[27845]: pgmap v13613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:11 smithi096 ceph-mon[32753]: pgmap v13613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:13 smithi194 ceph-mon[33017]: pgmap v13614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24792.075918 secs 2024-04-26T18:44:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:13 smithi033 ceph-mon[27845]: pgmap v13614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24792.075918 secs 2024-04-26T18:44:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:13 smithi096 ceph-mon[32753]: pgmap v13614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24792.075918 secs 2024-04-26T18:44:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24794.958808 secs 2024-04-26T18:44:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24794.958808 secs 2024-04-26T18:44:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24794.958808 secs 2024-04-26T18:44:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:15 smithi194 ceph-mon[33017]: pgmap v13615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:15 smithi033 ceph-mon[27845]: pgmap v13615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:15 smithi096 ceph-mon[32753]: pgmap v13615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:16 smithi194 ceph-mon[33017]: pgmap v13616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:16 smithi033 ceph-mon[27845]: pgmap v13616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:16 smithi096 ceph-mon[32753]: pgmap v13616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24797.075998 secs 2024-04-26T18:44:18.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24797.075998 secs 2024-04-26T18:44:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24797.075998 secs 2024-04-26T18:44:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:19 smithi194 ceph-mon[33017]: pgmap v13617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24799.958911 secs 2024-04-26T18:44:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:19 smithi033 ceph-mon[27845]: pgmap v13617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24799.958911 secs 2024-04-26T18:44:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:19 smithi096 ceph-mon[32753]: pgmap v13617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24799.958911 secs 2024-04-26T18:44:21.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:21.040+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:21.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:21.040+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:21.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:21.040+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:44:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:21 smithi194 ceph-mon[33017]: pgmap v13618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:21 smithi033 ceph-mon[27845]: pgmap v13618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:21 smithi096 ceph-mon[32753]: pgmap v13618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:23 smithi194 ceph-mon[33017]: pgmap v13619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24802.076085 secs 2024-04-26T18:44:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:23 smithi033 ceph-mon[27845]: pgmap v13619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24802.076085 secs 2024-04-26T18:44:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:23 smithi096 ceph-mon[32753]: pgmap v13619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24802.076085 secs 2024-04-26T18:44:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24804.959036 secs 2024-04-26T18:44:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24804.959036 secs 2024-04-26T18:44:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24804.959036 secs 2024-04-26T18:44:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:25 smithi194 ceph-mon[33017]: pgmap v13620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:25 smithi033 ceph-mon[27845]: pgmap v13620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:25 smithi096 ceph-mon[32753]: pgmap v13620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:26.015+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:44:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:26.015+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:44:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:26.016+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 197 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:44:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:26.016+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 197 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:44:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:27 smithi194 ceph-mon[33017]: pgmap v13621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:27 smithi033 ceph-mon[27845]: pgmap v13621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:27 smithi096 ceph-mon[32753]: pgmap v13621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24807.076172 secs 2024-04-26T18:44:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24807.076172 secs 2024-04-26T18:44:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24807.076172 secs 2024-04-26T18:44:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:29 smithi194 ceph-mon[33017]: pgmap v13622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24809.959169 secs 2024-04-26T18:44:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:29 smithi033 ceph-mon[27845]: pgmap v13622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24809.959169 secs 2024-04-26T18:44:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:29 smithi096 ceph-mon[32753]: pgmap v13622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24809.959169 secs 2024-04-26T18:44:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:31 smithi194 ceph-mon[33017]: pgmap v13623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:31 smithi033 ceph-mon[27845]: pgmap v13623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:31 smithi096 ceph-mon[32753]: pgmap v13623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:33 smithi194 ceph-mon[33017]: pgmap v13624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24812.076277 secs 2024-04-26T18:44:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:33 smithi033 ceph-mon[27845]: pgmap v13624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24812.076277 secs 2024-04-26T18:44:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:33 smithi096 ceph-mon[32753]: pgmap v13624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24812.076277 secs 2024-04-26T18:44:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24814.959266 secs 2024-04-26T18:44:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24814.959266 secs 2024-04-26T18:44:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24814.959266 secs 2024-04-26T18:44:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:35 smithi194 ceph-mon[33017]: pgmap v13625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:35 smithi033 ceph-mon[27845]: pgmap v13625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:35 smithi096 ceph-mon[32753]: pgmap v13625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:36.053+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:36.054+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:36.054+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:44:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:37 smithi194 ceph-mon[33017]: pgmap v13626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:37 smithi033 ceph-mon[27845]: pgmap v13626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:37 smithi096 ceph-mon[32753]: pgmap v13626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24817.076361 secs 2024-04-26T18:44:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24817.076361 secs 2024-04-26T18:44:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24817.076361 secs 2024-04-26T18:44:39.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:39 smithi096 ceph-mon[32753]: pgmap v13627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:39.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24819.959364 secs 2024-04-26T18:44:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:39 smithi194 ceph-mon[33017]: pgmap v13627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24819.959364 secs 2024-04-26T18:44:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:39 smithi033 ceph-mon[27845]: pgmap v13627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24819.959364 secs 2024-04-26T18:44:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:41 smithi194 ceph-mon[33017]: pgmap v13628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:44:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:41 smithi033 ceph-mon[27845]: pgmap v13628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:44:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:41 smithi096 ceph-mon[32753]: pgmap v13628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:44:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:44:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:44:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:44:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:44:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:44:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:44:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:44:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:44:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:44:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:43 smithi194 ceph-mon[33017]: pgmap v13629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:43 smithi194 ceph-mon[33017]: pgmap v13630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24822.076444 secs 2024-04-26T18:44:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:43 smithi033 ceph-mon[27845]: pgmap v13629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:43 smithi033 ceph-mon[27845]: pgmap v13630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24822.076444 secs 2024-04-26T18:44:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:43 smithi096 ceph-mon[32753]: pgmap v13629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:43 smithi096 ceph-mon[32753]: pgmap v13630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24822.076444 secs 2024-04-26T18:44:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24824.959460 secs 2024-04-26T18:44:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24824.959460 secs 2024-04-26T18:44:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24824.959460 secs 2024-04-26T18:44:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:45 smithi194 ceph-mon[33017]: pgmap v13631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:45 smithi033 ceph-mon[27845]: pgmap v13631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:45 smithi096 ceph-mon[32753]: pgmap v13631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:47 smithi194 ceph-mon[33017]: pgmap v13632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:47 smithi033 ceph-mon[27845]: pgmap v13632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:47 smithi096 ceph-mon[32753]: pgmap v13632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24827.076528 secs 2024-04-26T18:44:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24827.076528 secs 2024-04-26T18:44:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24827.076528 secs 2024-04-26T18:44:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:49 smithi194 ceph-mon[33017]: pgmap v13633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24829.959598 secs 2024-04-26T18:44:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:49 smithi033 ceph-mon[27845]: pgmap v13633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24829.959598 secs 2024-04-26T18:44:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:49 smithi096 ceph-mon[32753]: pgmap v13633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24829.959598 secs 2024-04-26T18:44:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:50 smithi194 ceph-mon[33017]: pgmap v13634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:50 smithi033 ceph-mon[27845]: pgmap v13634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:50 smithi096 ceph-mon[32753]: pgmap v13634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:51.064+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:51.064+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:44:51.064+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:44:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:52 smithi194 ceph-mon[33017]: pgmap v13635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:44:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:52 smithi033 ceph-mon[27845]: pgmap v13635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:44:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:52 smithi096 ceph-mon[32753]: pgmap v13635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:44:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24832.076612 secs 2024-04-26T18:44:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24834.959673 secs 2024-04-26T18:44:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24832.076612 secs 2024-04-26T18:44:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24834.959673 secs 2024-04-26T18:44:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24832.076612 secs 2024-04-26T18:44:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24834.959673 secs 2024-04-26T18:44:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:54 smithi194 ceph-mon[33017]: pgmap v13636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:54 smithi033 ceph-mon[27845]: pgmap v13636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:54 smithi096 ceph-mon[32753]: pgmap v13636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:56 smithi194 ceph-mon[33017]: pgmap v13637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:56 smithi033 ceph-mon[27845]: pgmap v13637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:56 smithi096 ceph-mon[32753]: pgmap v13637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:44:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:58 smithi194 ceph-mon[33017]: pgmap v13638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24837.076692 secs 2024-04-26T18:44:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:44:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24839.959789 secs 2024-04-26T18:44:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:58 smithi033 ceph-mon[27845]: pgmap v13638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24837.076692 secs 2024-04-26T18:44:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:44:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24839.959789 secs 2024-04-26T18:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:58 smithi096 ceph-mon[32753]: pgmap v13638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24837.076692 secs 2024-04-26T18:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:44:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24839.959789 secs 2024-04-26T18:45:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:00 smithi194 ceph-mon[33017]: pgmap v13639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:00 smithi033 ceph-mon[27845]: pgmap v13639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:00 smithi096 ceph-mon[32753]: pgmap v13639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:02 smithi194 ceph-mon[33017]: pgmap v13640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:02 smithi033 ceph-mon[27845]: pgmap v13640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:02 smithi096 ceph-mon[32753]: pgmap v13640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24842.076774 secs 2024-04-26T18:45:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24844.959926 secs 2024-04-26T18:45:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24842.076774 secs 2024-04-26T18:45:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24844.959926 secs 2024-04-26T18:45:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24842.076774 secs 2024-04-26T18:45:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24844.959926 secs 2024-04-26T18:45:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:04 smithi033 ceph-mon[27845]: pgmap v13641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:04 smithi096 ceph-mon[32753]: pgmap v13641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:04 smithi194 ceph-mon[33017]: pgmap v13641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:06.069+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:06.069+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:06.069+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:45:07.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:06 smithi194 ceph-mon[33017]: pgmap v13642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:06 smithi033 ceph-mon[27845]: pgmap v13642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:06 smithi096 ceph-mon[32753]: pgmap v13642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:08 smithi033 ceph-mon[27845]: pgmap v13643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24847.076858 secs 2024-04-26T18:45:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24849.960048 secs 2024-04-26T18:45:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:08 smithi096 ceph-mon[32753]: pgmap v13643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24847.076858 secs 2024-04-26T18:45:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24849.960048 secs 2024-04-26T18:45:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:08 smithi194 ceph-mon[33017]: pgmap v13643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24847.076858 secs 2024-04-26T18:45:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24849.960048 secs 2024-04-26T18:45:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:10 smithi033 ceph-mon[27845]: pgmap v13644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:10 smithi096 ceph-mon[32753]: pgmap v13644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:10 smithi194 ceph-mon[33017]: pgmap v13644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:12 smithi033 ceph-mon[27845]: pgmap v13645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:12 smithi096 ceph-mon[32753]: pgmap v13645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:12 smithi194 ceph-mon[33017]: pgmap v13645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24852.076940 secs 2024-04-26T18:45:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24854.960166 secs 2024-04-26T18:45:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24852.076940 secs 2024-04-26T18:45:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24854.960166 secs 2024-04-26T18:45:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24852.076940 secs 2024-04-26T18:45:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24854.960166 secs 2024-04-26T18:45:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:14 smithi033 ceph-mon[27845]: pgmap v13646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:14 smithi096 ceph-mon[32753]: pgmap v13646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:14 smithi194 ceph-mon[33017]: pgmap v13646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:16 smithi033 ceph-mon[27845]: pgmap v13647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:16 smithi096 ceph-mon[32753]: pgmap v13647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:16 smithi194 ceph-mon[33017]: pgmap v13647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:18 smithi033 ceph-mon[27845]: pgmap v13648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24857.076995 secs 2024-04-26T18:45:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24859.960263 secs 2024-04-26T18:45:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:18 smithi096 ceph-mon[32753]: pgmap v13648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24857.076995 secs 2024-04-26T18:45:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24859.960263 secs 2024-04-26T18:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:18 smithi194 ceph-mon[33017]: pgmap v13648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24857.076995 secs 2024-04-26T18:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24859.960263 secs 2024-04-26T18:45:21.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:21.081+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:21.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:21.081+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:45:21.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:21.081+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:20 smithi033 ceph-mon[27845]: pgmap v13649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:20 smithi096 ceph-mon[32753]: pgmap v13649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:20 smithi194 ceph-mon[33017]: pgmap v13649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:22 smithi033 ceph-mon[27845]: pgmap v13650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:22 smithi096 ceph-mon[32753]: pgmap v13650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:22 smithi194 ceph-mon[33017]: pgmap v13650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24862.077057 secs 2024-04-26T18:45:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24862.077057 secs 2024-04-26T18:45:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24862.077057 secs 2024-04-26T18:45:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24864.960333 secs 2024-04-26T18:45:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:24 smithi033 ceph-mon[27845]: pgmap v13651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24864.960333 secs 2024-04-26T18:45:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:24 smithi096 ceph-mon[32753]: pgmap v13651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24864.960333 secs 2024-04-26T18:45:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:24 smithi194 ceph-mon[33017]: pgmap v13651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:26 smithi194 ceph-mon[33017]: pgmap v13652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:26 smithi033 ceph-mon[27845]: pgmap v13652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:26 smithi096 ceph-mon[32753]: pgmap v13652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:28 smithi194 ceph-mon[33017]: pgmap v13653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24867.077121 secs 2024-04-26T18:45:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24869.960441 secs 2024-04-26T18:45:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:28 smithi033 ceph-mon[27845]: pgmap v13653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24867.077121 secs 2024-04-26T18:45:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24869.960441 secs 2024-04-26T18:45:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:28 smithi096 ceph-mon[32753]: pgmap v13653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24867.077121 secs 2024-04-26T18:45:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24869.960441 secs 2024-04-26T18:45:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:30 smithi194 ceph-mon[33017]: pgmap v13654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:30 smithi033 ceph-mon[27845]: pgmap v13654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:30 smithi096 ceph-mon[32753]: pgmap v13654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:32 smithi194 ceph-mon[33017]: pgmap v13655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:32 smithi033 ceph-mon[27845]: pgmap v13655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:32 smithi096 ceph-mon[32753]: pgmap v13655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24872.077203 secs 2024-04-26T18:45:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24874.960588 secs 2024-04-26T18:45:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24872.077203 secs 2024-04-26T18:45:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24874.960588 secs 2024-04-26T18:45:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24872.077203 secs 2024-04-26T18:45:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24874.960588 secs 2024-04-26T18:45:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:34 smithi033 ceph-mon[27845]: pgmap v13656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:34 smithi096 ceph-mon[32753]: pgmap v13656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:34 smithi194 ceph-mon[33017]: pgmap v13656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:36.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:36.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:36.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:45:37.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:36 smithi194 ceph-mon[33017]: pgmap v13657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:36 smithi033 ceph-mon[27845]: pgmap v13657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:36 smithi096 ceph-mon[32753]: pgmap v13657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:38 smithi033 ceph-mon[27845]: pgmap v13658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24877.077287 secs 2024-04-26T18:45:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24879.960707 secs 2024-04-26T18:45:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:38 smithi096 ceph-mon[32753]: pgmap v13658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24877.077287 secs 2024-04-26T18:45:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24879.960707 secs 2024-04-26T18:45:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:38 smithi194 ceph-mon[33017]: pgmap v13658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24877.077287 secs 2024-04-26T18:45:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24879.960707 secs 2024-04-26T18:45:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:40 smithi033 ceph-mon[27845]: pgmap v13659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:40 smithi096 ceph-mon[32753]: pgmap v13659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:40 smithi194 ceph-mon[33017]: pgmap v13659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:45:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:42 smithi033 ceph-mon[27845]: pgmap v13660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:45:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:42 smithi096 ceph-mon[32753]: pgmap v13660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:45:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:42 smithi194 ceph-mon[33017]: pgmap v13660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:45:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24882.077348 secs 2024-04-26T18:45:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24884.960824 secs 2024-04-26T18:45:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24882.077348 secs 2024-04-26T18:45:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24884.960824 secs 2024-04-26T18:45:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24882.077348 secs 2024-04-26T18:45:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24884.960824 secs 2024-04-26T18:45:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:45 smithi194 ceph-mon[33017]: pgmap v13661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:45:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:45:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:45 smithi033 ceph-mon[27845]: pgmap v13661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:45:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:45:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:45 smithi096 ceph-mon[32753]: pgmap v13661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:45:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:45:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:45:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:45:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:46 smithi194 ceph-mon[33017]: pgmap v13662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T18:45:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:46 smithi194 ceph-mon[33017]: pgmap v13663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T18:45:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:46 smithi033 ceph-mon[27845]: pgmap v13662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T18:45:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:46 smithi033 ceph-mon[27845]: pgmap v13663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T18:45:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:46 smithi096 ceph-mon[32753]: pgmap v13662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T18:45:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:46 smithi096 ceph-mon[32753]: pgmap v13663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-26T18:45:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:48 smithi194 ceph-mon[33017]: pgmap v13664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:48 smithi033 ceph-mon[27845]: pgmap v13664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:48 smithi096 ceph-mon[32753]: pgmap v13664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24887.077434 secs 2024-04-26T18:45:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24889.960941 secs 2024-04-26T18:45:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24887.077434 secs 2024-04-26T18:45:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24889.960941 secs 2024-04-26T18:45:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24887.077434 secs 2024-04-26T18:45:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24889.960941 secs 2024-04-26T18:45:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:50 smithi194 ceph-mon[33017]: pgmap v13665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:45:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:50 smithi033 ceph-mon[27845]: pgmap v13665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:45:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:50 smithi096 ceph-mon[32753]: pgmap v13665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:45:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:51.102+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:51.102+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:45:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:45:51.102+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:45:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:52 smithi194 ceph-mon[33017]: pgmap v13666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:52 smithi033 ceph-mon[27845]: pgmap v13666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:52 smithi096 ceph-mon[32753]: pgmap v13666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:54 smithi194 ceph-mon[33017]: pgmap v13667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24892.077516 secs 2024-04-26T18:45:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24894.961056 secs 2024-04-26T18:45:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:54 smithi033 ceph-mon[27845]: pgmap v13667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24892.077516 secs 2024-04-26T18:45:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24894.961056 secs 2024-04-26T18:45:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:54 smithi096 ceph-mon[32753]: pgmap v13667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:45:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24892.077516 secs 2024-04-26T18:45:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24894.961056 secs 2024-04-26T18:45:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:56 smithi194 ceph-mon[33017]: pgmap v13668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:45:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:56 smithi033 ceph-mon[27845]: pgmap v13668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:45:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:56 smithi096 ceph-mon[32753]: pgmap v13668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:45:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:58 smithi194 ceph-mon[33017]: pgmap v13669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:45:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:58 smithi033 ceph-mon[27845]: pgmap v13669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:45:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:58 smithi096 ceph-mon[32753]: pgmap v13669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:45:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24897.077600 secs 2024-04-26T18:45:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24899.961209 secs 2024-04-26T18:45:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24897.077600 secs 2024-04-26T18:45:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24899.961209 secs 2024-04-26T18:45:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24897.077600 secs 2024-04-26T18:45:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24899.961209 secs 2024-04-26T18:46:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:00 smithi194 ceph-mon[33017]: pgmap v13670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:00 smithi033 ceph-mon[27845]: pgmap v13670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:00 smithi096 ceph-mon[32753]: pgmap v13670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:02 smithi194 ceph-mon[33017]: pgmap v13671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:02 smithi033 ceph-mon[27845]: pgmap v13671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:02 smithi096 ceph-mon[32753]: pgmap v13671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:04 smithi033 ceph-mon[27845]: pgmap v13672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24902.077683 secs 2024-04-26T18:46:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24904.961329 secs 2024-04-26T18:46:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:04 smithi096 ceph-mon[32753]: pgmap v13672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24902.077683 secs 2024-04-26T18:46:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24904.961329 secs 2024-04-26T18:46:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:04 smithi194 ceph-mon[33017]: pgmap v13672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24902.077683 secs 2024-04-26T18:46:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24904.961329 secs 2024-04-26T18:46:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:06.117+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:06.117+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:06.117+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:46:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:06 smithi033 ceph-mon[27845]: pgmap v13673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:06 smithi096 ceph-mon[32753]: pgmap v13673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:06 smithi194 ceph-mon[33017]: pgmap v13673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:08.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:08 smithi194 ceph-mon[33017]: pgmap v13674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:08 smithi033 ceph-mon[27845]: pgmap v13674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:08 smithi096 ceph-mon[32753]: pgmap v13674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24907.077768 secs 2024-04-26T18:46:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24909.961442 secs 2024-04-26T18:46:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24907.077768 secs 2024-04-26T18:46:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24909.961442 secs 2024-04-26T18:46:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24907.077768 secs 2024-04-26T18:46:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24909.961442 secs 2024-04-26T18:46:10.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:10 smithi033 ceph-mon[27845]: pgmap v13675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:10 smithi096 ceph-mon[32753]: pgmap v13675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:10 smithi194 ceph-mon[33017]: pgmap v13675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:12 smithi033 ceph-mon[27845]: pgmap v13676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:12 smithi096 ceph-mon[32753]: pgmap v13676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:12 smithi194 ceph-mon[33017]: pgmap v13676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:14 smithi033 ceph-mon[27845]: pgmap v13677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24912.077855 secs 2024-04-26T18:46:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24914.961580 secs 2024-04-26T18:46:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:14 smithi096 ceph-mon[32753]: pgmap v13677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24912.077855 secs 2024-04-26T18:46:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24914.961580 secs 2024-04-26T18:46:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:14 smithi194 ceph-mon[33017]: pgmap v13677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24912.077855 secs 2024-04-26T18:46:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24914.961580 secs 2024-04-26T18:46:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:16 smithi033 ceph-mon[27845]: pgmap v13678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:16 smithi096 ceph-mon[32753]: pgmap v13678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:16 smithi194 ceph-mon[33017]: pgmap v13678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:18 smithi033 ceph-mon[27845]: pgmap v13679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24917.077940 secs 2024-04-26T18:46:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:18 smithi096 ceph-mon[32753]: pgmap v13679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24917.077940 secs 2024-04-26T18:46:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:18 smithi194 ceph-mon[33017]: pgmap v13679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24917.077940 secs 2024-04-26T18:46:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24919.961704 secs 2024-04-26T18:46:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24919.961704 secs 2024-04-26T18:46:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24919.961704 secs 2024-04-26T18:46:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:20 smithi033 ceph-mon[27845]: pgmap v13680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:20 smithi096 ceph-mon[32753]: pgmap v13680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:20 smithi194 ceph-mon[33017]: pgmap v13680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:21.133 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:21.132+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:21.133 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:21.133+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:21.133 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:21.133+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:46:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:22 smithi033 ceph-mon[27845]: pgmap v13681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:22 smithi096 ceph-mon[32753]: pgmap v13681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:22 smithi194 ceph-mon[33017]: pgmap v13681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24922.078002 secs 2024-04-26T18:46:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24922.078002 secs 2024-04-26T18:46:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24922.078002 secs 2024-04-26T18:46:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:24 smithi096 ceph-mon[32753]: pgmap v13682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24924.961822 secs 2024-04-26T18:46:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:24 smithi194 ceph-mon[33017]: pgmap v13682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24924.961822 secs 2024-04-26T18:46:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:24 smithi033 ceph-mon[27845]: pgmap v13682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24924.961822 secs 2024-04-26T18:46:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:26 smithi194 ceph-mon[33017]: pgmap v13683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:26 smithi033 ceph-mon[27845]: pgmap v13683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:26 smithi096 ceph-mon[32753]: pgmap v13683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:28 smithi194 ceph-mon[33017]: pgmap v13684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24927.078061 secs 2024-04-26T18:46:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:28 smithi033 ceph-mon[27845]: pgmap v13684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24927.078061 secs 2024-04-26T18:46:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:28 smithi096 ceph-mon[32753]: pgmap v13684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24927.078061 secs 2024-04-26T18:46:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24929.961941 secs 2024-04-26T18:46:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24929.961941 secs 2024-04-26T18:46:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24929.961941 secs 2024-04-26T18:46:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:30 smithi194 ceph-mon[33017]: pgmap v13685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:30 smithi033 ceph-mon[27845]: pgmap v13685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:30 smithi096 ceph-mon[32753]: pgmap v13685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:32 smithi194 ceph-mon[33017]: pgmap v13686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:32 smithi033 ceph-mon[27845]: pgmap v13686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:32 smithi096 ceph-mon[32753]: pgmap v13686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24932.078122 secs 2024-04-26T18:46:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24932.078122 secs 2024-04-26T18:46:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24932.078122 secs 2024-04-26T18:46:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:34 smithi194 ceph-mon[33017]: pgmap v13687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24934.962075 secs 2024-04-26T18:46:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:34 smithi033 ceph-mon[27845]: pgmap v13687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24934.962075 secs 2024-04-26T18:46:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:34 smithi096 ceph-mon[32753]: pgmap v13687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24934.962075 secs 2024-04-26T18:46:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:36.146+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:36.146+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:36.146+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:46:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:36 smithi194 ceph-mon[33017]: pgmap v13688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:36 smithi033 ceph-mon[27845]: pgmap v13688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:36 smithi096 ceph-mon[32753]: pgmap v13688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:38 smithi194 ceph-mon[33017]: pgmap v13689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24937.078206 secs 2024-04-26T18:46:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:38 smithi033 ceph-mon[27845]: pgmap v13689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:38.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24937.078206 secs 2024-04-26T18:46:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:38 smithi096 ceph-mon[32753]: pgmap v13689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24937.078206 secs 2024-04-26T18:46:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24939.962207 secs 2024-04-26T18:46:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24939.962207 secs 2024-04-26T18:46:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24939.962207 secs 2024-04-26T18:46:40.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:40 smithi096 ceph-mon[32753]: pgmap v13690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:40 smithi194 ceph-mon[33017]: pgmap v13690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:40 smithi033 ceph-mon[27845]: pgmap v13690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:42 smithi194 ceph-mon[33017]: pgmap v13691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:42 smithi033 ceph-mon[27845]: pgmap v13691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:42 smithi096 ceph-mon[32753]: pgmap v13691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:46:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24942.078287 secs 2024-04-26T18:46:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24942.078287 secs 2024-04-26T18:46:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24942.078287 secs 2024-04-26T18:46:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:44 smithi194 ceph-mon[33017]: pgmap v13692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24944.962326 secs 2024-04-26T18:46:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:44 smithi033 ceph-mon[27845]: pgmap v13692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24944.962326 secs 2024-04-26T18:46:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:44 smithi096 ceph-mon[32753]: pgmap v13692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24944.962326 secs 2024-04-26T18:46:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:46:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:46:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:46:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:46 smithi194 ceph-mon[33017]: pgmap v13693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:46:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:46:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:46:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:46 smithi033 ceph-mon[27845]: pgmap v13693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:46:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:46:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:46:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:46 smithi096 ceph-mon[32753]: pgmap v13693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:46:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:46:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:46:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:47 smithi194 ceph-mon[33017]: pgmap v13694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:47 smithi194 ceph-mon[33017]: pgmap v13695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:47 smithi033 ceph-mon[27845]: pgmap v13694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:47 smithi033 ceph-mon[27845]: pgmap v13695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:47 smithi096 ceph-mon[32753]: pgmap v13694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:46:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:47 smithi096 ceph-mon[32753]: pgmap v13695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:48.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24947.078348 secs 2024-04-26T18:46:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24947.078348 secs 2024-04-26T18:46:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24947.078348 secs 2024-04-26T18:46:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:49 smithi194 ceph-mon[33017]: pgmap v13696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24949.962410 secs 2024-04-26T18:46:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:49 smithi033 ceph-mon[27845]: pgmap v13696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24949.962410 secs 2024-04-26T18:46:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:49 smithi096 ceph-mon[32753]: pgmap v13696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24949.962410 secs 2024-04-26T18:46:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:51.151+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:51.151+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:46:51.151+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:46:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:51 smithi194 ceph-mon[33017]: pgmap v13697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:51 smithi033 ceph-mon[27845]: pgmap v13697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:51 smithi096 ceph-mon[32753]: pgmap v13697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:52 smithi096 ceph-mon[32753]: pgmap v13698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:52 smithi194 ceph-mon[33017]: pgmap v13698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:52 smithi033 ceph-mon[27845]: pgmap v13698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:46:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24952.078431 secs 2024-04-26T18:46:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24952.078431 secs 2024-04-26T18:46:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24952.078431 secs 2024-04-26T18:46:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24954.962535 secs 2024-04-26T18:46:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:54 smithi194 ceph-mon[33017]: pgmap v13699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T18:46:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24954.962535 secs 2024-04-26T18:46:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:54 smithi033 ceph-mon[27845]: pgmap v13699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T18:46:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24954.962535 secs 2024-04-26T18:46:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:54 smithi096 ceph-mon[32753]: pgmap v13699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T18:46:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:57 smithi033 ceph-mon[27845]: pgmap v13700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:46:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:57 smithi096 ceph-mon[32753]: pgmap v13700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:46:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:57 smithi194 ceph-mon[33017]: pgmap v13700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:46:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24957.078492 secs 2024-04-26T18:46:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24957.078492 secs 2024-04-26T18:46:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24957.078492 secs 2024-04-26T18:46:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:59 smithi033 ceph-mon[27845]: pgmap v13701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:46:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24959.962656 secs 2024-04-26T18:46:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:59 smithi096 ceph-mon[32753]: pgmap v13701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:46:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24959.962656 secs 2024-04-26T18:46:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:59 smithi194 ceph-mon[33017]: pgmap v13701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:46:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:46:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24959.962656 secs 2024-04-26T18:47:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:01 smithi096 ceph-mon[32753]: pgmap v13702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:01 smithi194 ceph-mon[33017]: pgmap v13702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:01 smithi033 ceph-mon[27845]: pgmap v13702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:03 smithi194 ceph-mon[33017]: pgmap v13703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24962.078575 secs 2024-04-26T18:47:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:03 smithi033 ceph-mon[27845]: pgmap v13703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24962.078575 secs 2024-04-26T18:47:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:03 smithi096 ceph-mon[32753]: pgmap v13703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24962.078575 secs 2024-04-26T18:47:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24964.962767 secs 2024-04-26T18:47:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24964.962767 secs 2024-04-26T18:47:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24964.962767 secs 2024-04-26T18:47:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:05 smithi194 ceph-mon[33017]: pgmap v13704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:05 smithi033 ceph-mon[27845]: pgmap v13704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:05 smithi096 ceph-mon[32753]: pgmap v13704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:06.166+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:06.166+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:06.166+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:47:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:07 smithi194 ceph-mon[33017]: pgmap v13705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:07 smithi033 ceph-mon[27845]: pgmap v13705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:07 smithi096 ceph-mon[32753]: pgmap v13705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:08.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24967.078659 secs 2024-04-26T18:47:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24967.078659 secs 2024-04-26T18:47:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24967.078659 secs 2024-04-26T18:47:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:09 smithi194 ceph-mon[33017]: pgmap v13706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24969.962886 secs 2024-04-26T18:47:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:09 smithi033 ceph-mon[27845]: pgmap v13706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24969.962886 secs 2024-04-26T18:47:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:09 smithi096 ceph-mon[32753]: pgmap v13706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24969.962886 secs 2024-04-26T18:47:11.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:11 smithi096 ceph-mon[32753]: pgmap v13707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:11 smithi194 ceph-mon[33017]: pgmap v13707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:11 smithi033 ceph-mon[27845]: pgmap v13707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:13 smithi194 ceph-mon[33017]: pgmap v13708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24972.078739 secs 2024-04-26T18:47:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:13 smithi033 ceph-mon[27845]: pgmap v13708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24972.078739 secs 2024-04-26T18:47:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:13 smithi096 ceph-mon[32753]: pgmap v13708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24972.078739 secs 2024-04-26T18:47:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24974.963005 secs 2024-04-26T18:47:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24974.963005 secs 2024-04-26T18:47:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24974.963005 secs 2024-04-26T18:47:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:15 smithi194 ceph-mon[33017]: pgmap v13709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:15 smithi033 ceph-mon[27845]: pgmap v13709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:15 smithi096 ceph-mon[32753]: pgmap v13709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:17 smithi194 ceph-mon[33017]: pgmap v13710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:17 smithi033 ceph-mon[27845]: pgmap v13710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:17 smithi096 ceph-mon[32753]: pgmap v13710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:18.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24977.078802 secs 2024-04-26T18:47:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24977.078802 secs 2024-04-26T18:47:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24977.078802 secs 2024-04-26T18:47:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:19 smithi194 ceph-mon[33017]: pgmap v13711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24979.963146 secs 2024-04-26T18:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:19 smithi033 ceph-mon[27845]: pgmap v13711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24979.963146 secs 2024-04-26T18:47:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:19 smithi096 ceph-mon[32753]: pgmap v13711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24979.963146 secs 2024-04-26T18:47:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:21.174+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:21.174+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:21.174+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:47:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:21 smithi194 ceph-mon[33017]: pgmap v13712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:21 smithi033 ceph-mon[27845]: pgmap v13712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:21 smithi096 ceph-mon[32753]: pgmap v13712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:23 smithi194 ceph-mon[33017]: pgmap v13713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24982.078883 secs 2024-04-26T18:47:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:23 smithi033 ceph-mon[27845]: pgmap v13713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24982.078883 secs 2024-04-26T18:47:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:23 smithi096 ceph-mon[32753]: pgmap v13713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24982.078883 secs 2024-04-26T18:47:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24984.963265 secs 2024-04-26T18:47:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24984.963265 secs 2024-04-26T18:47:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24984.963265 secs 2024-04-26T18:47:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:25 smithi194 ceph-mon[33017]: pgmap v13714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:25 smithi033 ceph-mon[27845]: pgmap v13714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:25 smithi096 ceph-mon[32753]: pgmap v13714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:26.046+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:47:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:26.046+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:47:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:26.046+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 198 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T18:47:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:26.046+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 198 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T18:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:27 smithi194 ceph-mon[33017]: pgmap v13715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:27 smithi033 ceph-mon[27845]: pgmap v13715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:27 smithi096 ceph-mon[32753]: pgmap v13715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24987.078990 secs 2024-04-26T18:47:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24987.078990 secs 2024-04-26T18:47:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24987.078990 secs 2024-04-26T18:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:29 smithi194 ceph-mon[33017]: pgmap v13716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24989.963364 secs 2024-04-26T18:47:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:29 smithi033 ceph-mon[27845]: pgmap v13716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24989.963364 secs 2024-04-26T18:47:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:29 smithi096 ceph-mon[32753]: pgmap v13716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24989.963364 secs 2024-04-26T18:47:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:31 smithi194 ceph-mon[33017]: pgmap v13717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:31 smithi033 ceph-mon[27845]: pgmap v13717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:31 smithi096 ceph-mon[32753]: pgmap v13717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:33 smithi194 ceph-mon[33017]: pgmap v13718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24992.079062 secs 2024-04-26T18:47:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:33 smithi033 ceph-mon[27845]: pgmap v13718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24992.079062 secs 2024-04-26T18:47:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:33 smithi096 ceph-mon[32753]: pgmap v13718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24992.079062 secs 2024-04-26T18:47:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24994.963482 secs 2024-04-26T18:47:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24994.963482 secs 2024-04-26T18:47:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24994.963482 secs 2024-04-26T18:47:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:35 smithi194 ceph-mon[33017]: pgmap v13719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:35 smithi033 ceph-mon[27845]: pgmap v13719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:35 smithi096 ceph-mon[32753]: pgmap v13719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:36.184+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:36.185+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:36.185+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:47:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:37 smithi194 ceph-mon[33017]: pgmap v13720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:37 smithi033 ceph-mon[27845]: pgmap v13720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:37 smithi096 ceph-mon[32753]: pgmap v13720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 24997.079143 secs 2024-04-26T18:47:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 24997.079143 secs 2024-04-26T18:47:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 24997.079143 secs 2024-04-26T18:47:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:39 smithi194 ceph-mon[33017]: pgmap v13721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 24999.963560 secs 2024-04-26T18:47:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:39 smithi033 ceph-mon[27845]: pgmap v13721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 24999.963560 secs 2024-04-26T18:47:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:39 smithi096 ceph-mon[32753]: pgmap v13721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 24999.963560 secs 2024-04-26T18:47:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:41 smithi033 ceph-mon[27845]: pgmap v13722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:41 smithi096 ceph-mon[32753]: pgmap v13722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:41 smithi194 ceph-mon[33017]: pgmap v13722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:43 smithi033 ceph-mon[27845]: pgmap v13723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25002.079223 secs 2024-04-26T18:47:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:43 smithi096 ceph-mon[32753]: pgmap v13723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25002.079223 secs 2024-04-26T18:47:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:43 smithi194 ceph-mon[33017]: pgmap v13723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25002.079223 secs 2024-04-26T18:47:44.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25004.963698 secs 2024-04-26T18:47:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25004.963698 secs 2024-04-26T18:47:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25004.963698 secs 2024-04-26T18:47:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:45 smithi033 ceph-mon[27845]: pgmap v13724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:45 smithi096 ceph-mon[32753]: pgmap v13724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:45 smithi194 ceph-mon[33017]: pgmap v13724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:46 smithi033 ceph-mon[27845]: pgmap v13725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:46 smithi096 ceph-mon[32753]: pgmap v13725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:46 smithi194 ceph-mon[33017]: pgmap v13725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:47:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:47:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:47:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25007.079286 secs 2024-04-26T18:47:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:48 smithi033 ceph-mon[27845]: pgmap v13726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25007.079286 secs 2024-04-26T18:47:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:48 smithi096 ceph-mon[32753]: pgmap v13726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25007.079286 secs 2024-04-26T18:47:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:48 smithi194 ceph-mon[33017]: pgmap v13726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25009.963798 secs 2024-04-26T18:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25009.963798 secs 2024-04-26T18:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25009.963798 secs 2024-04-26T18:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:47:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:47:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:50 smithi033 ceph-mon[27845]: pgmap v13727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:47:50.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:50 smithi096 ceph-mon[32753]: pgmap v13727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:47:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:47:50.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:50 smithi194 ceph-mon[33017]: pgmap v13727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:47:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:51.201+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:51.201+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:47:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:47:51.201+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:47:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: pgmap v13728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: pgmap v13729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: pgmap v13730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:47:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: pgmap v13728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:47:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:47:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:47:53.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:47:53.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: pgmap v13729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:47:53.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: pgmap v13730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:47:53.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: pgmap v13728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:47:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:47:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:47:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:47:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: pgmap v13729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:47:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: pgmap v13730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:47:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:47:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25012.079366 secs 2024-04-26T18:47:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25014.963934 secs 2024-04-26T18:47:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25012.079366 secs 2024-04-26T18:47:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25014.963934 secs 2024-04-26T18:47:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25012.079366 secs 2024-04-26T18:47:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25014.963934 secs 2024-04-26T18:47:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:54 smithi194 ceph-mon[33017]: pgmap v13731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T18:47:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:54 smithi033 ceph-mon[27845]: pgmap v13731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T18:47:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:54 smithi096 ceph-mon[32753]: pgmap v13731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T18:47:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:57 smithi033 ceph-mon[27845]: pgmap v13732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:47:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:57 smithi096 ceph-mon[32753]: pgmap v13732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:47:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:57 smithi194 ceph-mon[33017]: pgmap v13732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:47:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25017.079452 secs 2024-04-26T18:47:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25017.079452 secs 2024-04-26T18:47:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25017.079452 secs 2024-04-26T18:47:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:59 smithi033 ceph-mon[27845]: pgmap v13733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:47:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:47:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25019.964008 secs 2024-04-26T18:47:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:59 smithi096 ceph-mon[32753]: pgmap v13733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:47:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:47:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25019.964008 secs 2024-04-26T18:47:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:59 smithi194 ceph-mon[33017]: pgmap v13733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:47:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:47:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25019.964008 secs 2024-04-26T18:48:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:00 smithi033 ceph-mon[27845]: pgmap v13734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:48:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:00 smithi096 ceph-mon[32753]: pgmap v13734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:48:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:00 smithi194 ceph-mon[33017]: pgmap v13734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:48:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:03 smithi033 ceph-mon[27845]: pgmap v13735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25022.079535 secs 2024-04-26T18:48:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:03 smithi096 ceph-mon[32753]: pgmap v13735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25022.079535 secs 2024-04-26T18:48:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:03 smithi194 ceph-mon[33017]: pgmap v13735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25022.079535 secs 2024-04-26T18:48:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25024.964147 secs 2024-04-26T18:48:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25024.964147 secs 2024-04-26T18:48:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25024.964147 secs 2024-04-26T18:48:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:05 smithi033 ceph-mon[27845]: pgmap v13736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:05 smithi096 ceph-mon[32753]: pgmap v13736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:05 smithi194 ceph-mon[33017]: pgmap v13736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:06.207+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:06.207+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:06.207+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:48:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:06 smithi033 ceph-mon[27845]: pgmap v13737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:06 smithi096 ceph-mon[32753]: pgmap v13737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:06 smithi194 ceph-mon[33017]: pgmap v13737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25027.079619 secs 2024-04-26T18:48:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25027.079619 secs 2024-04-26T18:48:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25027.079619 secs 2024-04-26T18:48:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:09 smithi194 ceph-mon[33017]: pgmap v13738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25029.964266 secs 2024-04-26T18:48:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:09 smithi033 ceph-mon[27845]: pgmap v13738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25029.964266 secs 2024-04-26T18:48:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:09 smithi096 ceph-mon[32753]: pgmap v13738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25029.964266 secs 2024-04-26T18:48:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:11 smithi033 ceph-mon[27845]: pgmap v13739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:11 smithi096 ceph-mon[32753]: pgmap v13739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:11 smithi194 ceph-mon[33017]: pgmap v13739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:13 smithi033 ceph-mon[27845]: pgmap v13740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25032.079701 secs 2024-04-26T18:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:13 smithi096 ceph-mon[32753]: pgmap v13740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25032.079701 secs 2024-04-26T18:48:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:13 smithi194 ceph-mon[33017]: pgmap v13740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25032.079701 secs 2024-04-26T18:48:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25034.964385 secs 2024-04-26T18:48:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:14 smithi033 ceph-mon[27845]: pgmap v13741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25034.964385 secs 2024-04-26T18:48:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:14 smithi096 ceph-mon[32753]: pgmap v13741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25034.964385 secs 2024-04-26T18:48:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:14 smithi194 ceph-mon[33017]: pgmap v13741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:17 smithi033 ceph-mon[27845]: pgmap v13742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:17 smithi096 ceph-mon[32753]: pgmap v13742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:17 smithi194 ceph-mon[33017]: pgmap v13742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25037.079786 secs 2024-04-26T18:48:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25037.079786 secs 2024-04-26T18:48:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25037.079786 secs 2024-04-26T18:48:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:19 smithi033 ceph-mon[27845]: pgmap v13743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25039.964506 secs 2024-04-26T18:48:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:19 smithi096 ceph-mon[32753]: pgmap v13743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25039.964506 secs 2024-04-26T18:48:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:19 smithi194 ceph-mon[33017]: pgmap v13743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25039.964506 secs 2024-04-26T18:48:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:20 smithi033 ceph-mon[27845]: pgmap v13744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:20 smithi096 ceph-mon[32753]: pgmap v13744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:20 smithi194 ceph-mon[33017]: pgmap v13744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:21.208+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:21.208+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:48:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:21.208+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:22 smithi194 ceph-mon[33017]: pgmap v13745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:22 smithi033 ceph-mon[27845]: pgmap v13745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:22 smithi096 ceph-mon[32753]: pgmap v13745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25042.079870 secs 2024-04-26T18:48:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25042.079870 secs 2024-04-26T18:48:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25042.079870 secs 2024-04-26T18:48:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25044.964627 secs 2024-04-26T18:48:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:24 smithi194 ceph-mon[33017]: pgmap v13746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25044.964627 secs 2024-04-26T18:48:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:24 smithi033 ceph-mon[27845]: pgmap v13746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25044.964627 secs 2024-04-26T18:48:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:24 smithi096 ceph-mon[32753]: pgmap v13746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:27 smithi033 ceph-mon[27845]: pgmap v13747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:48:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:27 smithi096 ceph-mon[32753]: pgmap v13747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:48:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:27 smithi194 ceph-mon[33017]: pgmap v13747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:48:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25047.079953 secs 2024-04-26T18:48:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:28 smithi033 ceph-mon[27845]: pgmap v13748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25047.079953 secs 2024-04-26T18:48:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:28 smithi096 ceph-mon[32753]: pgmap v13748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25047.079953 secs 2024-04-26T18:48:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:28 smithi194 ceph-mon[33017]: pgmap v13748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25049.964771 secs 2024-04-26T18:48:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25049.964771 secs 2024-04-26T18:48:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25049.964771 secs 2024-04-26T18:48:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:30 smithi033 ceph-mon[27845]: pgmap v13749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:30 smithi096 ceph-mon[32753]: pgmap v13749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:30 smithi194 ceph-mon[33017]: pgmap v13749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:33 smithi033 ceph-mon[27845]: pgmap v13750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25052.080028 secs 2024-04-26T18:48:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:33 smithi096 ceph-mon[32753]: pgmap v13750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25052.080028 secs 2024-04-26T18:48:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:33 smithi194 ceph-mon[33017]: pgmap v13750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25052.080028 secs 2024-04-26T18:48:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25054.964910 secs 2024-04-26T18:48:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:34 smithi033 ceph-mon[27845]: pgmap v13751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25054.964910 secs 2024-04-26T18:48:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:34 smithi096 ceph-mon[32753]: pgmap v13751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25054.964910 secs 2024-04-26T18:48:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:34 smithi194 ceph-mon[33017]: pgmap v13751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:36.223+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:36.223+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:36.223+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:48:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:36 smithi096 ceph-mon[32753]: pgmap v13752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:36 smithi194 ceph-mon[33017]: pgmap v13752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:36 smithi033 ceph-mon[27845]: pgmap v13752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25057.080114 secs 2024-04-26T18:48:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25057.080114 secs 2024-04-26T18:48:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25057.080114 secs 2024-04-26T18:48:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:39 smithi194 ceph-mon[33017]: pgmap v13753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:48:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25059.964983 secs 2024-04-26T18:48:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:39 smithi033 ceph-mon[27845]: pgmap v13753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:48:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25059.964983 secs 2024-04-26T18:48:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:39 smithi096 ceph-mon[32753]: pgmap v13753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:48:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25059.964983 secs 2024-04-26T18:48:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:40 smithi096 ceph-mon[32753]: pgmap v13754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:48:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:40 smithi194 ceph-mon[33017]: pgmap v13754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:48:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:40 smithi033 ceph-mon[27845]: pgmap v13754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:48:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:42 smithi096 ceph-mon[32753]: pgmap v13755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:48:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:42 smithi194 ceph-mon[33017]: pgmap v13755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:48:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:42 smithi033 ceph-mon[27845]: pgmap v13755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:48:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25062.080202 secs 2024-04-26T18:48:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25062.080202 secs 2024-04-26T18:48:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25062.080202 secs 2024-04-26T18:48:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25064.965102 secs 2024-04-26T18:48:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:44 smithi194 ceph-mon[33017]: pgmap v13756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:48:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25064.965102 secs 2024-04-26T18:48:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:44 smithi033 ceph-mon[27845]: pgmap v13756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:48:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25064.965102 secs 2024-04-26T18:48:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:44 smithi096 ceph-mon[32753]: pgmap v13756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:48:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:46 smithi096 ceph-mon[32753]: pgmap v13757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:46 smithi194 ceph-mon[33017]: pgmap v13757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:46 smithi033 ceph-mon[27845]: pgmap v13757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:48:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25067.080285 secs 2024-04-26T18:48:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25067.080285 secs 2024-04-26T18:48:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25067.080285 secs 2024-04-26T18:48:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:49 smithi194 ceph-mon[33017]: pgmap v13758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25069.965145 secs 2024-04-26T18:48:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:49 smithi033 ceph-mon[27845]: pgmap v13758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25069.965145 secs 2024-04-26T18:48:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:49 smithi096 ceph-mon[32753]: pgmap v13758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25069.965145 secs 2024-04-26T18:48:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:50 smithi194 ceph-mon[33017]: pgmap v13759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:50 smithi033 ceph-mon[27845]: pgmap v13759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:50 smithi096 ceph-mon[32753]: pgmap v13759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:48:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:51.234+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:51.234+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:48:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:48:51.234+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:48:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:52 smithi194 ceph-mon[33017]: pgmap v13760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:52 smithi033 ceph-mon[27845]: pgmap v13760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:52 smithi096 ceph-mon[32753]: pgmap v13760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:48:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25072.080378 secs 2024-04-26T18:48:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:48:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25072.080378 secs 2024-04-26T18:48:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:48:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25072.080378 secs 2024-04-26T18:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25074.965257 secs 2024-04-26T18:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:54 smithi194 ceph-mon[33017]: pgmap v13761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:54 smithi194 ceph-mon[33017]: pgmap v13762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:48:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25074.965257 secs 2024-04-26T18:48:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:48:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:48:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:54 smithi033 ceph-mon[27845]: pgmap v13761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:54 smithi033 ceph-mon[27845]: pgmap v13762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:48:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:48:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25074.965257 secs 2024-04-26T18:48:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:48:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:48:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:54 smithi096 ceph-mon[32753]: pgmap v13761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:48:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:54 smithi096 ceph-mon[32753]: pgmap v13762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:48:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:48:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:57 smithi194 ceph-mon[33017]: pgmap v13763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:57 smithi033 ceph-mon[27845]: pgmap v13763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:57 smithi096 ceph-mon[32753]: pgmap v13763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25077.080460 secs 2024-04-26T18:48:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:59 smithi194 ceph-mon[33017]: pgmap v13764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25079.965337 secs 2024-04-26T18:48:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25077.080460 secs 2024-04-26T18:48:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:59 smithi033 ceph-mon[27845]: pgmap v13764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25079.965337 secs 2024-04-26T18:48:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25077.080460 secs 2024-04-26T18:48:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:59 smithi096 ceph-mon[32753]: pgmap v13764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:48:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25079.965337 secs 2024-04-26T18:49:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:01 smithi194 ceph-mon[33017]: pgmap v13765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:01 smithi033 ceph-mon[27845]: pgmap v13765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:01 smithi096 ceph-mon[32753]: pgmap v13765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:03 smithi194 ceph-mon[33017]: pgmap v13766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:03 smithi033 ceph-mon[27845]: pgmap v13766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:03 smithi096 ceph-mon[32753]: pgmap v13766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25082.080543 secs 2024-04-26T18:49:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25084.965458 secs 2024-04-26T18:49:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25082.080543 secs 2024-04-26T18:49:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25084.965458 secs 2024-04-26T18:49:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25082.080543 secs 2024-04-26T18:49:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25084.965458 secs 2024-04-26T18:49:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:05 smithi194 ceph-mon[33017]: pgmap v13767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:49:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:05 smithi033 ceph-mon[27845]: pgmap v13767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:49:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:05 smithi096 ceph-mon[32753]: pgmap v13767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:49:06.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:06.250+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:06.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:06.250+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:06.250+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:49:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:07 smithi194 ceph-mon[33017]: pgmap v13768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:07 smithi033 ceph-mon[27845]: pgmap v13768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:07 smithi096 ceph-mon[32753]: pgmap v13768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25087.080621 secs 2024-04-26T18:49:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:09 smithi194 ceph-mon[33017]: pgmap v13769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25089.965583 secs 2024-04-26T18:49:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25087.080621 secs 2024-04-26T18:49:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:09 smithi033 ceph-mon[27845]: pgmap v13769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25089.965583 secs 2024-04-26T18:49:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25087.080621 secs 2024-04-26T18:49:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:09 smithi096 ceph-mon[32753]: pgmap v13769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25089.965583 secs 2024-04-26T18:49:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:11 smithi194 ceph-mon[33017]: pgmap v13770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:11 smithi033 ceph-mon[27845]: pgmap v13770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:11 smithi096 ceph-mon[32753]: pgmap v13770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:13 smithi194 ceph-mon[33017]: pgmap v13771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:13 smithi033 ceph-mon[27845]: pgmap v13771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:13 smithi096 ceph-mon[32753]: pgmap v13771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25092.080704 secs 2024-04-26T18:49:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25094.965699 secs 2024-04-26T18:49:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25092.080704 secs 2024-04-26T18:49:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25094.965699 secs 2024-04-26T18:49:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25092.080704 secs 2024-04-26T18:49:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25094.965699 secs 2024-04-26T18:49:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:15 smithi194 ceph-mon[33017]: pgmap v13772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:15 smithi033 ceph-mon[27845]: pgmap v13772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:15 smithi096 ceph-mon[32753]: pgmap v13772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:17 smithi033 ceph-mon[27845]: pgmap v13773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:17 smithi194 ceph-mon[33017]: pgmap v13773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:17 smithi096 ceph-mon[32753]: pgmap v13773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25097.080787 secs 2024-04-26T18:49:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:19 smithi194 ceph-mon[33017]: pgmap v13774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25099.965847 secs 2024-04-26T18:49:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25097.080787 secs 2024-04-26T18:49:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:19 smithi033 ceph-mon[27845]: pgmap v13774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25099.965847 secs 2024-04-26T18:49:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25097.080787 secs 2024-04-26T18:49:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:19 smithi096 ceph-mon[32753]: pgmap v13774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25099.965847 secs 2024-04-26T18:49:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:21.251+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:21.251+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:49:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:21.251+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:21.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:21 smithi194 ceph-mon[33017]: pgmap v13775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:21 smithi033 ceph-mon[27845]: pgmap v13775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:21 smithi096 ceph-mon[32753]: pgmap v13775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:22 smithi194 ceph-mon[33017]: pgmap v13776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:22 smithi033 ceph-mon[27845]: pgmap v13776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:22 smithi096 ceph-mon[32753]: pgmap v13776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25102.080869 secs 2024-04-26T18:49:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25102.080869 secs 2024-04-26T18:49:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25102.080869 secs 2024-04-26T18:49:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25104.965966 secs 2024-04-26T18:49:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:24 smithi194 ceph-mon[33017]: pgmap v13777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25104.965966 secs 2024-04-26T18:49:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:24 smithi033 ceph-mon[27845]: pgmap v13777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25104.965966 secs 2024-04-26T18:49:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:24 smithi096 ceph-mon[32753]: pgmap v13777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:27 smithi194 ceph-mon[33017]: pgmap v13778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:27 smithi033 ceph-mon[27845]: pgmap v13778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:27 smithi096 ceph-mon[32753]: pgmap v13778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25107.080962 secs 2024-04-26T18:49:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:29 smithi194 ceph-mon[33017]: pgmap v13779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25109.966156 secs 2024-04-26T18:49:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25107.080962 secs 2024-04-26T18:49:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:29 smithi033 ceph-mon[27845]: pgmap v13779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25109.966156 secs 2024-04-26T18:49:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25107.080962 secs 2024-04-26T18:49:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:29 smithi096 ceph-mon[32753]: pgmap v13779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25109.966156 secs 2024-04-26T18:49:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:31 smithi194 ceph-mon[33017]: pgmap v13780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:31 smithi033 ceph-mon[27845]: pgmap v13780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:31 smithi096 ceph-mon[32753]: pgmap v13780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:33 smithi194 ceph-mon[33017]: pgmap v13781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:33 smithi033 ceph-mon[27845]: pgmap v13781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:33 smithi096 ceph-mon[32753]: pgmap v13781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25112.081023 secs 2024-04-26T18:49:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25114.966274 secs 2024-04-26T18:49:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25112.081023 secs 2024-04-26T18:49:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25114.966274 secs 2024-04-26T18:49:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25112.081023 secs 2024-04-26T18:49:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25114.966274 secs 2024-04-26T18:49:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:35 smithi194 ceph-mon[33017]: pgmap v13782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:35 smithi033 ceph-mon[27845]: pgmap v13782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:35 smithi096 ceph-mon[32753]: pgmap v13782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:36.262+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:36.262+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:36.262+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:49:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:37 smithi194 ceph-mon[33017]: pgmap v13783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:37 smithi033 ceph-mon[27845]: pgmap v13783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:37 smithi096 ceph-mon[32753]: pgmap v13783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:49:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25117.081108 secs 2024-04-26T18:49:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:39 smithi194 ceph-mon[33017]: pgmap v13784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25119.966397 secs 2024-04-26T18:49:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25117.081108 secs 2024-04-26T18:49:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:39 smithi033 ceph-mon[27845]: pgmap v13784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25119.966397 secs 2024-04-26T18:49:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25117.081108 secs 2024-04-26T18:49:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:39 smithi096 ceph-mon[32753]: pgmap v13784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25119.966397 secs 2024-04-26T18:49:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:41 smithi194 ceph-mon[33017]: pgmap v13785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:41 smithi033 ceph-mon[27845]: pgmap v13785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:41 smithi096 ceph-mon[32753]: pgmap v13785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:43 smithi194 ceph-mon[33017]: pgmap v13786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:43 smithi033 ceph-mon[27845]: pgmap v13786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:43 smithi096 ceph-mon[32753]: pgmap v13786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25122.081170 secs 2024-04-26T18:49:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25124.966513 secs 2024-04-26T18:49:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25122.081170 secs 2024-04-26T18:49:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25124.966513 secs 2024-04-26T18:49:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25122.081170 secs 2024-04-26T18:49:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25124.966513 secs 2024-04-26T18:49:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:45 smithi194 ceph-mon[33017]: pgmap v13787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:45 smithi033 ceph-mon[27845]: pgmap v13787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:45 smithi096 ceph-mon[32753]: pgmap v13787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:47 smithi033 ceph-mon[27845]: pgmap v13788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:47 smithi096 ceph-mon[32753]: pgmap v13788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:47 smithi194 ceph-mon[33017]: pgmap v13788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25127.081234 secs 2024-04-26T18:49:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:49 smithi033 ceph-mon[27845]: pgmap v13789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25129.966633 secs 2024-04-26T18:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25127.081234 secs 2024-04-26T18:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:49 smithi096 ceph-mon[32753]: pgmap v13789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25129.966633 secs 2024-04-26T18:49:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25127.081234 secs 2024-04-26T18:49:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:49 smithi194 ceph-mon[33017]: pgmap v13789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:49:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25129.966633 secs 2024-04-26T18:49:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:51.277+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:51.278+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:49:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:49:51.278+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:49:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:51 smithi033 ceph-mon[27845]: pgmap v13790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:51 smithi096 ceph-mon[32753]: pgmap v13790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:51 smithi194 ceph-mon[33017]: pgmap v13790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:49:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:53 smithi033 ceph-mon[27845]: pgmap v13791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:53 smithi096 ceph-mon[32753]: pgmap v13791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:53 smithi194 ceph-mon[33017]: pgmap v13791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25132.081317 secs 2024-04-26T18:49:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25134.966753 secs 2024-04-26T18:49:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25132.081317 secs 2024-04-26T18:49:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25134.966753 secs 2024-04-26T18:49:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25132.081317 secs 2024-04-26T18:49:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25134.966753 secs 2024-04-26T18:49:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:55 smithi096 ceph-mon[32753]: pgmap v13792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:49:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:55 smithi033 ceph-mon[27845]: pgmap v13792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:49:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:55 smithi194 ceph-mon[33017]: pgmap v13792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:49:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:49:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:49:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:49:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:49:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:49:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:49:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:49:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:49:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:49:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:49:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:57 smithi033 ceph-mon[27845]: pgmap v13793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:49:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:57 smithi033 ceph-mon[27845]: pgmap v13794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:57 smithi096 ceph-mon[32753]: pgmap v13793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:49:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:57 smithi096 ceph-mon[32753]: pgmap v13794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:57 smithi194 ceph-mon[33017]: pgmap v13793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:49:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:57 smithi194 ceph-mon[33017]: pgmap v13794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:59 smithi033 ceph-mon[27845]: pgmap v13795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25137.081400 secs 2024-04-26T18:49:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25139.966887 secs 2024-04-26T18:49:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:59 smithi096 ceph-mon[32753]: pgmap v13795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25137.081400 secs 2024-04-26T18:49:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25139.966887 secs 2024-04-26T18:49:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:59 smithi194 ceph-mon[33017]: pgmap v13795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:49:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25137.081400 secs 2024-04-26T18:49:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25139.966887 secs 2024-04-26T18:50:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:50:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:50:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T18:50:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:01 smithi033 ceph-mon[27845]: pgmap v13796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:01 smithi096 ceph-mon[32753]: pgmap v13796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:01 smithi194 ceph-mon[33017]: pgmap v13796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:03 smithi033 ceph-mon[27845]: pgmap v13797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25142.081461 secs 2024-04-26T18:50:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:03 smithi096 ceph-mon[32753]: pgmap v13797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25142.081461 secs 2024-04-26T18:50:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:03 smithi194 ceph-mon[33017]: pgmap v13797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25142.081461 secs 2024-04-26T18:50:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25144.967007 secs 2024-04-26T18:50:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25144.967007 secs 2024-04-26T18:50:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25144.967007 secs 2024-04-26T18:50:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:05 smithi194 ceph-mon[33017]: pgmap v13798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:05 smithi033 ceph-mon[27845]: pgmap v13798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:05 smithi096 ceph-mon[32753]: pgmap v13798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:06.292+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:06.292+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:06.292+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:50:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:07 smithi194 ceph-mon[33017]: pgmap v13799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:50:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:07 smithi033 ceph-mon[27845]: pgmap v13799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:50:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:07 smithi096 ceph-mon[32753]: pgmap v13799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:50:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25147.081545 secs 2024-04-26T18:50:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25147.081545 secs 2024-04-26T18:50:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25147.081545 secs 2024-04-26T18:50:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:09 smithi194 ceph-mon[33017]: pgmap v13800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25149.967143 secs 2024-04-26T18:50:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:09 smithi033 ceph-mon[27845]: pgmap v13800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25149.967143 secs 2024-04-26T18:50:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:09 smithi096 ceph-mon[32753]: pgmap v13800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25149.967143 secs 2024-04-26T18:50:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:11 smithi194 ceph-mon[33017]: pgmap v13801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:11 smithi033 ceph-mon[27845]: pgmap v13801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:11 smithi096 ceph-mon[32753]: pgmap v13801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:13.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:13 smithi096 ceph-mon[32753]: pgmap v13802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:13.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25152.081627 secs 2024-04-26T18:50:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:13 smithi194 ceph-mon[33017]: pgmap v13802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25152.081627 secs 2024-04-26T18:50:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:13 smithi033 ceph-mon[27845]: pgmap v13802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25152.081627 secs 2024-04-26T18:50:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25154.967260 secs 2024-04-26T18:50:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25154.967260 secs 2024-04-26T18:50:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25154.967260 secs 2024-04-26T18:50:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:15 smithi194 ceph-mon[33017]: pgmap v13803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:15 smithi033 ceph-mon[27845]: pgmap v13803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:15 smithi096 ceph-mon[32753]: pgmap v13803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:17 smithi194 ceph-mon[33017]: pgmap v13804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:17 smithi033 ceph-mon[27845]: pgmap v13804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:17 smithi096 ceph-mon[32753]: pgmap v13804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:18.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25157.081712 secs 2024-04-26T18:50:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25157.081712 secs 2024-04-26T18:50:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25157.081712 secs 2024-04-26T18:50:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:19 smithi194 ceph-mon[33017]: pgmap v13805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25159.967338 secs 2024-04-26T18:50:19.606 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:19 smithi033 ceph-mon[27845]: pgmap v13805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:19.606 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25159.967338 secs 2024-04-26T18:50:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:19 smithi096 ceph-mon[32753]: pgmap v13805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25159.967338 secs 2024-04-26T18:50:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:21.308+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:21.308+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:50:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:21.308+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:21 smithi194 ceph-mon[33017]: pgmap v13806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:21 smithi033 ceph-mon[27845]: pgmap v13806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:21 smithi096 ceph-mon[32753]: pgmap v13806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:23 smithi194 ceph-mon[33017]: pgmap v13807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25162.081798 secs 2024-04-26T18:50:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:23 smithi033 ceph-mon[27845]: pgmap v13807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25162.081798 secs 2024-04-26T18:50:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:23 smithi096 ceph-mon[32753]: pgmap v13807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25162.081798 secs 2024-04-26T18:50:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25164.967461 secs 2024-04-26T18:50:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25164.967461 secs 2024-04-26T18:50:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25164.967461 secs 2024-04-26T18:50:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:25 smithi194 ceph-mon[33017]: pgmap v13808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:25 smithi033 ceph-mon[27845]: pgmap v13808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:25 smithi096 ceph-mon[32753]: pgmap v13808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.077+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:50:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.077+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:50:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.077+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 199 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T18:50:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.077+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x564385821900 2024-04-26T18:50:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.077+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 199 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T18:50:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.077+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0050 con 0x7fef00077780 2024-04-26T18:50:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.077+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 200 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x564385821900 2024-04-26T18:50:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:26.078+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 200 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:50:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:27 smithi194 ceph-mon[33017]: pgmap v13809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:27 smithi033 ceph-mon[27845]: pgmap v13809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:27 smithi096 ceph-mon[32753]: pgmap v13809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25167.081882 secs 2024-04-26T18:50:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25167.081882 secs 2024-04-26T18:50:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25167.081882 secs 2024-04-26T18:50:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:29 smithi194 ceph-mon[33017]: pgmap v13810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25169.967589 secs 2024-04-26T18:50:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:29 smithi033 ceph-mon[27845]: pgmap v13810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25169.967589 secs 2024-04-26T18:50:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:29 smithi096 ceph-mon[32753]: pgmap v13810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25169.967589 secs 2024-04-26T18:50:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:31 smithi194 ceph-mon[33017]: pgmap v13811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:31 smithi033 ceph-mon[27845]: pgmap v13811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:31 smithi096 ceph-mon[32753]: pgmap v13811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:33 smithi194 ceph-mon[33017]: pgmap v13812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25172.081989 secs 2024-04-26T18:50:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:33 smithi033 ceph-mon[27845]: pgmap v13812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25172.081989 secs 2024-04-26T18:50:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:33 smithi096 ceph-mon[32753]: pgmap v13812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25172.081989 secs 2024-04-26T18:50:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25174.967707 secs 2024-04-26T18:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25174.967707 secs 2024-04-26T18:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25174.967707 secs 2024-04-26T18:50:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:35 smithi194 ceph-mon[33017]: pgmap v13813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:35 smithi033 ceph-mon[27845]: pgmap v13813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:35 smithi096 ceph-mon[32753]: pgmap v13813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:36.323+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:36.323+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:36.323+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:50:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:37 smithi194 ceph-mon[33017]: pgmap v13814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:37 smithi033 ceph-mon[27845]: pgmap v13814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:37 smithi096 ceph-mon[32753]: pgmap v13814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25177.082074 secs 2024-04-26T18:50:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25177.082074 secs 2024-04-26T18:50:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25177.082074 secs 2024-04-26T18:50:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:39 smithi194 ceph-mon[33017]: pgmap v13815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25179.967847 secs 2024-04-26T18:50:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:39 smithi033 ceph-mon[27845]: pgmap v13815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25179.967847 secs 2024-04-26T18:50:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:39 smithi096 ceph-mon[32753]: pgmap v13815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25179.967847 secs 2024-04-26T18:50:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:41 smithi194 ceph-mon[33017]: pgmap v13816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:41 smithi033 ceph-mon[27845]: pgmap v13816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:41 smithi096 ceph-mon[32753]: pgmap v13816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:43 smithi194 ceph-mon[33017]: pgmap v13817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25182.082155 secs 2024-04-26T18:50:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:43 smithi033 ceph-mon[27845]: pgmap v13817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25182.082155 secs 2024-04-26T18:50:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:43 smithi096 ceph-mon[32753]: pgmap v13817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25182.082155 secs 2024-04-26T18:50:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25184.967968 secs 2024-04-26T18:50:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25184.967968 secs 2024-04-26T18:50:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25184.967968 secs 2024-04-26T18:50:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:45 smithi033 ceph-mon[27845]: pgmap v13818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:45 smithi096 ceph-mon[32753]: pgmap v13818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:45 smithi194 ceph-mon[33017]: pgmap v13818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:47 smithi033 ceph-mon[27845]: pgmap v13819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:47 smithi096 ceph-mon[32753]: pgmap v13819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:47 smithi194 ceph-mon[33017]: pgmap v13819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25187.082236 secs 2024-04-26T18:50:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25187.082236 secs 2024-04-26T18:50:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25187.082236 secs 2024-04-26T18:50:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:49 smithi033 ceph-mon[27845]: pgmap v13820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25189.968122 secs 2024-04-26T18:50:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:49 smithi096 ceph-mon[32753]: pgmap v13820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25189.968122 secs 2024-04-26T18:50:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:49 smithi194 ceph-mon[33017]: pgmap v13820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25189.968122 secs 2024-04-26T18:50:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:51.335+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:51.335+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:50:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:50:51.335+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:50:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:51 smithi033 ceph-mon[27845]: pgmap v13821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:51 smithi096 ceph-mon[32753]: pgmap v13821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:51 smithi194 ceph-mon[33017]: pgmap v13821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:53 smithi033 ceph-mon[27845]: pgmap v13822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25192.082316 secs 2024-04-26T18:50:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:53 smithi096 ceph-mon[32753]: pgmap v13822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25192.082316 secs 2024-04-26T18:50:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:53 smithi194 ceph-mon[33017]: pgmap v13822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25192.082316 secs 2024-04-26T18:50:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25194.968258 secs 2024-04-26T18:50:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25194.968258 secs 2024-04-26T18:50:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25194.968258 secs 2024-04-26T18:50:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:55 smithi033 ceph-mon[27845]: pgmap v13823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:55 smithi096 ceph-mon[32753]: pgmap v13823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:55 smithi194 ceph-mon[33017]: pgmap v13823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:50:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:50:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:50:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:57 smithi033 ceph-mon[27845]: pgmap v13824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:50:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:50:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:57 smithi096 ceph-mon[32753]: pgmap v13824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:50:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:50:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:57 smithi194 ceph-mon[33017]: pgmap v13824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:50:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:50:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:50:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:58 smithi096 ceph-mon[32753]: pgmap v13825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:58 smithi096 ceph-mon[32753]: pgmap v13826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:50:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25197.082400 secs 2024-04-26T18:50:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:58 smithi194 ceph-mon[33017]: pgmap v13825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:58 smithi194 ceph-mon[33017]: pgmap v13826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:50:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25197.082400 secs 2024-04-26T18:50:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:58 smithi033 ceph-mon[27845]: pgmap v13825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:50:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:58 smithi033 ceph-mon[27845]: pgmap v13826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:50:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:50:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25197.082400 secs 2024-04-26T18:50:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25199.968351 secs 2024-04-26T18:50:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25199.968351 secs 2024-04-26T18:50:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25199.968351 secs 2024-04-26T18:51:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:00 smithi194 ceph-mon[33017]: pgmap v13827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T18:51:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:00 smithi033 ceph-mon[27845]: pgmap v13827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T18:51:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:00 smithi096 ceph-mon[32753]: pgmap v13827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T18:51:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:02 smithi194 ceph-mon[33017]: pgmap v13828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:02 smithi033 ceph-mon[27845]: pgmap v13828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:02 smithi096 ceph-mon[32753]: pgmap v13828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25202.082484 secs 2024-04-26T18:51:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25202.082484 secs 2024-04-26T18:51:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25202.082484 secs 2024-04-26T18:51:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:04 smithi194 ceph-mon[33017]: pgmap v13829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25204.968465 secs 2024-04-26T18:51:04.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:04 smithi033 ceph-mon[27845]: pgmap v13829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:04.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25204.968465 secs 2024-04-26T18:51:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:04 smithi096 ceph-mon[32753]: pgmap v13829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25204.968465 secs 2024-04-26T18:51:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:06.349+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:06.349+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:06.349+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:51:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:06 smithi194 ceph-mon[33017]: pgmap v13830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:06 smithi033 ceph-mon[27845]: pgmap v13830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:06 smithi096 ceph-mon[32753]: pgmap v13830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:08 smithi194 ceph-mon[33017]: pgmap v13831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:51:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25207.082568 secs 2024-04-26T18:51:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:08 smithi033 ceph-mon[27845]: pgmap v13831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:51:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25207.082568 secs 2024-04-26T18:51:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:08 smithi096 ceph-mon[32753]: pgmap v13831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:51:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25207.082568 secs 2024-04-26T18:51:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25209.968583 secs 2024-04-26T18:51:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25209.968583 secs 2024-04-26T18:51:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25209.968583 secs 2024-04-26T18:51:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:10 smithi194 ceph-mon[33017]: pgmap v13832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:10 smithi033 ceph-mon[27845]: pgmap v13832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:10 smithi096 ceph-mon[32753]: pgmap v13832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:12 smithi194 ceph-mon[33017]: pgmap v13833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:12 smithi033 ceph-mon[27845]: pgmap v13833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:12 smithi096 ceph-mon[32753]: pgmap v13833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:13.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25212.082651 secs 2024-04-26T18:51:13.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25212.082651 secs 2024-04-26T18:51:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25212.082651 secs 2024-04-26T18:51:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:14 smithi194 ceph-mon[33017]: pgmap v13834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25214.968709 secs 2024-04-26T18:51:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:14 smithi033 ceph-mon[27845]: pgmap v13834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25214.968709 secs 2024-04-26T18:51:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:14 smithi096 ceph-mon[32753]: pgmap v13834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25214.968709 secs 2024-04-26T18:51:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:16 smithi194 ceph-mon[33017]: pgmap v13835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:16 smithi033 ceph-mon[27845]: pgmap v13835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:16 smithi096 ceph-mon[32753]: pgmap v13835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:18 smithi194 ceph-mon[33017]: pgmap v13836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25217.082709 secs 2024-04-26T18:51:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:18 smithi033 ceph-mon[27845]: pgmap v13836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25217.082709 secs 2024-04-26T18:51:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:18 smithi096 ceph-mon[32753]: pgmap v13836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25217.082709 secs 2024-04-26T18:51:19.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25219.968827 secs 2024-04-26T18:51:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25219.968827 secs 2024-04-26T18:51:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25219.968827 secs 2024-04-26T18:51:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:20 smithi194 ceph-mon[33017]: pgmap v13837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:20 smithi033 ceph-mon[27845]: pgmap v13837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:20 smithi096 ceph-mon[32753]: pgmap v13837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:21.353+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:21.353+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:21.353+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:51:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:22 smithi194 ceph-mon[33017]: pgmap v13838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:22 smithi033 ceph-mon[27845]: pgmap v13838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:22 smithi096 ceph-mon[32753]: pgmap v13838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25222.082765 secs 2024-04-26T18:51:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25222.082765 secs 2024-04-26T18:51:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25222.082765 secs 2024-04-26T18:51:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:24 smithi194 ceph-mon[33017]: pgmap v13839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25224.968947 secs 2024-04-26T18:51:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:24 smithi033 ceph-mon[27845]: pgmap v13839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25224.968947 secs 2024-04-26T18:51:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:24 smithi096 ceph-mon[32753]: pgmap v13839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25224.968947 secs 2024-04-26T18:51:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:26 smithi194 ceph-mon[33017]: pgmap v13840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:26 smithi033 ceph-mon[27845]: pgmap v13840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:26 smithi096 ceph-mon[32753]: pgmap v13840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:28 smithi194 ceph-mon[33017]: pgmap v13841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25227.082820 secs 2024-04-26T18:51:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:28 smithi033 ceph-mon[27845]: pgmap v13841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25227.082820 secs 2024-04-26T18:51:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:28 smithi096 ceph-mon[32753]: pgmap v13841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25227.082820 secs 2024-04-26T18:51:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25229.969078 secs 2024-04-26T18:51:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25229.969078 secs 2024-04-26T18:51:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25229.969078 secs 2024-04-26T18:51:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:30 smithi194 ceph-mon[33017]: pgmap v13842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:30 smithi033 ceph-mon[27845]: pgmap v13842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:30 smithi096 ceph-mon[32753]: pgmap v13842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:32 smithi194 ceph-mon[33017]: pgmap v13843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:32 smithi033 ceph-mon[27845]: pgmap v13843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:32 smithi096 ceph-mon[32753]: pgmap v13843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25232.082929 secs 2024-04-26T18:51:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25232.082929 secs 2024-04-26T18:51:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25232.082929 secs 2024-04-26T18:51:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:34 smithi194 ceph-mon[33017]: pgmap v13844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25234.969176 secs 2024-04-26T18:51:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:34 smithi033 ceph-mon[27845]: pgmap v13844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25234.969176 secs 2024-04-26T18:51:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:34 smithi096 ceph-mon[32753]: pgmap v13844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25234.969176 secs 2024-04-26T18:51:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:36.354+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:36.354+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:36.354+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:51:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:36 smithi194 ceph-mon[33017]: pgmap v13845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:36 smithi033 ceph-mon[27845]: pgmap v13845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:36 smithi096 ceph-mon[32753]: pgmap v13845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:38 smithi194 ceph-mon[33017]: pgmap v13846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25237.083025 secs 2024-04-26T18:51:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:38 smithi033 ceph-mon[27845]: pgmap v13846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25237.083025 secs 2024-04-26T18:51:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:38 smithi096 ceph-mon[32753]: pgmap v13846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25237.083025 secs 2024-04-26T18:51:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25239.969312 secs 2024-04-26T18:51:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25239.969312 secs 2024-04-26T18:51:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25239.969312 secs 2024-04-26T18:51:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:40 smithi033 ceph-mon[27845]: pgmap v13847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:40 smithi096 ceph-mon[32753]: pgmap v13847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:40 smithi194 ceph-mon[33017]: pgmap v13847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:42 smithi033 ceph-mon[27845]: pgmap v13848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:42 smithi096 ceph-mon[32753]: pgmap v13848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:42 smithi194 ceph-mon[33017]: pgmap v13848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25242.083113 secs 2024-04-26T18:51:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25242.083113 secs 2024-04-26T18:51:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25242.083113 secs 2024-04-26T18:51:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:44 smithi033 ceph-mon[27845]: pgmap v13849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25244.969428 secs 2024-04-26T18:51:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:44 smithi096 ceph-mon[32753]: pgmap v13849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25244.969428 secs 2024-04-26T18:51:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:44 smithi194 ceph-mon[33017]: pgmap v13849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25244.969428 secs 2024-04-26T18:51:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:46 smithi033 ceph-mon[27845]: pgmap v13850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:46 smithi096 ceph-mon[32753]: pgmap v13850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:46 smithi194 ceph-mon[33017]: pgmap v13850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:48 smithi033 ceph-mon[27845]: pgmap v13851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:48.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25247.083214 secs 2024-04-26T18:51:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:48 smithi096 ceph-mon[32753]: pgmap v13851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25247.083214 secs 2024-04-26T18:51:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:48 smithi194 ceph-mon[33017]: pgmap v13851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25247.083214 secs 2024-04-26T18:51:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25249.969536 secs 2024-04-26T18:51:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25249.969536 secs 2024-04-26T18:51:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25249.969536 secs 2024-04-26T18:51:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:50 smithi033 ceph-mon[27845]: pgmap v13852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:50 smithi096 ceph-mon[32753]: pgmap v13852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:50 smithi194 ceph-mon[33017]: pgmap v13852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:51:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:51.362+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:51.362+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:51:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:51:51.362+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:51:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:52 smithi033 ceph-mon[27845]: pgmap v13853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:52 smithi096 ceph-mon[32753]: pgmap v13853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:52 smithi194 ceph-mon[33017]: pgmap v13853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:51:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25252.083273 secs 2024-04-26T18:51:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25252.083273 secs 2024-04-26T18:51:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25252.083273 secs 2024-04-26T18:51:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:54 smithi033 ceph-mon[27845]: pgmap v13854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25254.969659 secs 2024-04-26T18:51:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:54 smithi096 ceph-mon[32753]: pgmap v13854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25254.969659 secs 2024-04-26T18:51:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:54 smithi194 ceph-mon[33017]: pgmap v13854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25254.969659 secs 2024-04-26T18:51:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:56 smithi096 ceph-mon[32753]: pgmap v13855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:56 smithi194 ceph-mon[33017]: pgmap v13855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:56 smithi033 ceph-mon[27845]: pgmap v13855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:51:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:58 smithi194 ceph-mon[33017]: pgmap v13856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:51:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25257.083341 secs 2024-04-26T18:51:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:58 smithi033 ceph-mon[27845]: pgmap v13856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:51:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25257.083341 secs 2024-04-26T18:51:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:58 smithi096 ceph-mon[32753]: pgmap v13856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:51:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:51:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25257.083341 secs 2024-04-26T18:52:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25259.969731 secs 2024-04-26T18:52:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:52:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:52:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:59 smithi194 ceph-mon[33017]: pgmap v13857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:59 smithi194 ceph-mon[33017]: pgmap v13858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:52:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:51:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:52:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25259.969731 secs 2024-04-26T18:52:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:52:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:52:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:59 smithi033 ceph-mon[27845]: pgmap v13857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:59 smithi033 ceph-mon[27845]: pgmap v13858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:52:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:51:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25259.969731 secs 2024-04-26T18:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:59 smithi096 ceph-mon[32753]: pgmap v13857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:59 smithi096 ceph-mon[32753]: pgmap v13858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T18:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:51:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:52:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:02 smithi033 ceph-mon[27845]: pgmap v13859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:02 smithi096 ceph-mon[32753]: pgmap v13859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:02 smithi194 ceph-mon[33017]: pgmap v13859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25262.083397 secs 2024-04-26T18:52:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:04 smithi033 ceph-mon[27845]: pgmap v13860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25264.969797 secs 2024-04-26T18:52:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25262.083397 secs 2024-04-26T18:52:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:04 smithi096 ceph-mon[32753]: pgmap v13860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25264.969797 secs 2024-04-26T18:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25262.083397 secs 2024-04-26T18:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:04 smithi194 ceph-mon[33017]: pgmap v13860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25264.969797 secs 2024-04-26T18:52:06.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:06.375+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:06.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:06.375+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:06.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:06.375+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:52:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:06 smithi033 ceph-mon[27845]: pgmap v13861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:06 smithi096 ceph-mon[32753]: pgmap v13861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:06 smithi194 ceph-mon[33017]: pgmap v13861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:08 smithi033 ceph-mon[27845]: pgmap v13862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:08 smithi096 ceph-mon[32753]: pgmap v13862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:08 smithi194 ceph-mon[33017]: pgmap v13862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:52:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25267.083480 secs 2024-04-26T18:52:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25269.969913 secs 2024-04-26T18:52:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25267.083480 secs 2024-04-26T18:52:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25269.969913 secs 2024-04-26T18:52:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25267.083480 secs 2024-04-26T18:52:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25269.969913 secs 2024-04-26T18:52:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:10 smithi096 ceph-mon[32753]: pgmap v13863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:52:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:10 smithi194 ceph-mon[33017]: pgmap v13863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:52:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:10 smithi033 ceph-mon[27845]: pgmap v13863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:52:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:12 smithi194 ceph-mon[33017]: pgmap v13864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:12 smithi033 ceph-mon[27845]: pgmap v13864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:12 smithi096 ceph-mon[32753]: pgmap v13864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:13.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25272.083560 secs 2024-04-26T18:52:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25272.083560 secs 2024-04-26T18:52:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25272.083560 secs 2024-04-26T18:52:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:14 smithi194 ceph-mon[33017]: pgmap v13865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25274.970031 secs 2024-04-26T18:52:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:14 smithi033 ceph-mon[27845]: pgmap v13865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25274.970031 secs 2024-04-26T18:52:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:14 smithi096 ceph-mon[32753]: pgmap v13865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25274.970031 secs 2024-04-26T18:52:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:16 smithi194 ceph-mon[33017]: pgmap v13866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:16 smithi033 ceph-mon[27845]: pgmap v13866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:16 smithi096 ceph-mon[32753]: pgmap v13866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:18 smithi194 ceph-mon[33017]: pgmap v13867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25277.083644 secs 2024-04-26T18:52:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:18 smithi033 ceph-mon[27845]: pgmap v13867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25277.083644 secs 2024-04-26T18:52:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:18 smithi096 ceph-mon[32753]: pgmap v13867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25277.083644 secs 2024-04-26T18:52:19.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25279.970176 secs 2024-04-26T18:52:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25279.970176 secs 2024-04-26T18:52:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25279.970176 secs 2024-04-26T18:52:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:20 smithi194 ceph-mon[33017]: pgmap v13868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:20 smithi033 ceph-mon[27845]: pgmap v13868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:20 smithi096 ceph-mon[32753]: pgmap v13868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:21.376+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:21.376+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:21.376+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:22.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:22 smithi033 ceph-mon[27845]: pgmap v13869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:22 smithi194 ceph-mon[33017]: pgmap v13869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:22 smithi096 ceph-mon[32753]: pgmap v13869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25282.083728 secs 2024-04-26T18:52:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25282.083728 secs 2024-04-26T18:52:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25282.083728 secs 2024-04-26T18:52:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:24 smithi194 ceph-mon[33017]: pgmap v13870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25284.970272 secs 2024-04-26T18:52:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:24 smithi033 ceph-mon[27845]: pgmap v13870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25284.970272 secs 2024-04-26T18:52:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:24 smithi096 ceph-mon[32753]: pgmap v13870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25284.970272 secs 2024-04-26T18:52:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:26 smithi194 ceph-mon[33017]: pgmap v13871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:26 smithi033 ceph-mon[27845]: pgmap v13871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:26 smithi096 ceph-mon[32753]: pgmap v13871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:28 smithi194 ceph-mon[33017]: pgmap v13872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25287.083788 secs 2024-04-26T18:52:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:28 smithi033 ceph-mon[27845]: pgmap v13872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25287.083788 secs 2024-04-26T18:52:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:28 smithi096 ceph-mon[32753]: pgmap v13872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25287.083788 secs 2024-04-26T18:52:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25289.970398 secs 2024-04-26T18:52:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25289.970398 secs 2024-04-26T18:52:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25289.970398 secs 2024-04-26T18:52:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:30 smithi194 ceph-mon[33017]: pgmap v13873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:30 smithi033 ceph-mon[27845]: pgmap v13873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:30 smithi096 ceph-mon[32753]: pgmap v13873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:32 smithi194 ceph-mon[33017]: pgmap v13874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:32 smithi033 ceph-mon[27845]: pgmap v13874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:32 smithi096 ceph-mon[32753]: pgmap v13874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25292.083872 secs 2024-04-26T18:52:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25292.083872 secs 2024-04-26T18:52:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25292.083872 secs 2024-04-26T18:52:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:34 smithi194 ceph-mon[33017]: pgmap v13875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25294.970514 secs 2024-04-26T18:52:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:34 smithi033 ceph-mon[27845]: pgmap v13875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25294.970514 secs 2024-04-26T18:52:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:34 smithi096 ceph-mon[32753]: pgmap v13875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25294.970514 secs 2024-04-26T18:52:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:36.378+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:36.378+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:36.378+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:52:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:36 smithi194 ceph-mon[33017]: pgmap v13876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:36 smithi033 ceph-mon[27845]: pgmap v13876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:36 smithi096 ceph-mon[32753]: pgmap v13876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:38 smithi033 ceph-mon[27845]: pgmap v13877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25297.083955 secs 2024-04-26T18:52:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:38 smithi194 ceph-mon[33017]: pgmap v13877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25297.083955 secs 2024-04-26T18:52:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:38 smithi096 ceph-mon[32753]: pgmap v13877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25297.083955 secs 2024-04-26T18:52:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25299.970643 secs 2024-04-26T18:52:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25299.970643 secs 2024-04-26T18:52:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25299.970643 secs 2024-04-26T18:52:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:40 smithi194 ceph-mon[33017]: pgmap v13878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:40 smithi033 ceph-mon[27845]: pgmap v13878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:40 smithi096 ceph-mon[32753]: pgmap v13878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:42 smithi194 ceph-mon[33017]: pgmap v13879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:42 smithi033 ceph-mon[27845]: pgmap v13879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:42 smithi096 ceph-mon[32753]: pgmap v13879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:43.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25302.084038 secs 2024-04-26T18:52:43.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25302.084038 secs 2024-04-26T18:52:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25302.084038 secs 2024-04-26T18:52:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:44 smithi194 ceph-mon[33017]: pgmap v13880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25304.970781 secs 2024-04-26T18:52:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:44 smithi033 ceph-mon[27845]: pgmap v13880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25304.970781 secs 2024-04-26T18:52:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:44 smithi096 ceph-mon[32753]: pgmap v13880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25304.970781 secs 2024-04-26T18:52:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:46 smithi194 ceph-mon[33017]: pgmap v13881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:46 smithi033 ceph-mon[27845]: pgmap v13881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:46 smithi096 ceph-mon[32753]: pgmap v13881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:48 smithi194 ceph-mon[33017]: pgmap v13882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25307.084096 secs 2024-04-26T18:52:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:48 smithi033 ceph-mon[27845]: pgmap v13882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25307.084096 secs 2024-04-26T18:52:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:48 smithi096 ceph-mon[32753]: pgmap v13882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25307.084096 secs 2024-04-26T18:52:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25309.970905 secs 2024-04-26T18:52:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25309.970905 secs 2024-04-26T18:52:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25309.970905 secs 2024-04-26T18:52:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:50 smithi194 ceph-mon[33017]: pgmap v13883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:50 smithi033 ceph-mon[27845]: pgmap v13883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:50 smithi096 ceph-mon[32753]: pgmap v13883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:51.392+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:51.392+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:52:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:52:51.392+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:52:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:52 smithi194 ceph-mon[33017]: pgmap v13884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:52 smithi033 ceph-mon[27845]: pgmap v13884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:52 smithi096 ceph-mon[32753]: pgmap v13884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25312.084179 secs 2024-04-26T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25312.084179 secs 2024-04-26T18:52:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25312.084179 secs 2024-04-26T18:52:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:54 smithi033 ceph-mon[27845]: pgmap v13885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25314.971021 secs 2024-04-26T18:52:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:54 smithi096 ceph-mon[32753]: pgmap v13885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25314.971021 secs 2024-04-26T18:52:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:54 smithi194 ceph-mon[33017]: pgmap v13885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25314.971021 secs 2024-04-26T18:52:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:56 smithi033 ceph-mon[27845]: pgmap v13886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:56 smithi096 ceph-mon[32753]: pgmap v13886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:56 smithi194 ceph-mon[33017]: pgmap v13886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:52:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:57 smithi194 ceph-mon[33017]: pgmap v13887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:57 smithi033 ceph-mon[27845]: pgmap v13887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:57 smithi096 ceph-mon[32753]: pgmap v13887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25317.084264 secs 2024-04-26T18:52:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25317.084264 secs 2024-04-26T18:52:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25317.084264 secs 2024-04-26T18:52:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25319.971141 secs 2024-04-26T18:52:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:59 smithi194 ceph-mon[33017]: pgmap v13888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:52:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:52:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25319.971141 secs 2024-04-26T18:52:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:59 smithi033 ceph-mon[27845]: pgmap v13888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:52:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:52:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25319.971141 secs 2024-04-26T18:52:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:59 smithi096 ceph-mon[32753]: pgmap v13888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:52:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:52:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:53:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:53:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:53:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:53:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:53:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:53:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:53:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:53:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:53:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:53:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:01 smithi033 ceph-mon[27845]: pgmap v13889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:53:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:01 smithi033 ceph-mon[27845]: pgmap v13890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:01 smithi096 ceph-mon[32753]: pgmap v13889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:53:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:01 smithi096 ceph-mon[32753]: pgmap v13890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:01 smithi194 ceph-mon[33017]: pgmap v13889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T18:53:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:01 smithi194 ceph-mon[33017]: pgmap v13890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:03 smithi033 ceph-mon[27845]: pgmap v13891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25322.084350 secs 2024-04-26T18:53:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25324.971259 secs 2024-04-26T18:53:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:03 smithi096 ceph-mon[32753]: pgmap v13891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25322.084350 secs 2024-04-26T18:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25324.971259 secs 2024-04-26T18:53:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:03 smithi194 ceph-mon[33017]: pgmap v13891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25322.084350 secs 2024-04-26T18:53:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25324.971259 secs 2024-04-26T18:53:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:05 smithi033 ceph-mon[27845]: pgmap v13892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:05 smithi096 ceph-mon[32753]: pgmap v13892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:05 smithi194 ceph-mon[33017]: pgmap v13892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:06.397+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:06.397+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:53:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:06.397+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:07 smithi033 ceph-mon[27845]: pgmap v13893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:08.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:07 smithi096 ceph-mon[32753]: pgmap v13893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:07 smithi194 ceph-mon[33017]: pgmap v13893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25327.084412 secs 2024-04-26T18:53:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25329.971333 secs 2024-04-26T18:53:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25327.084412 secs 2024-04-26T18:53:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25329.971333 secs 2024-04-26T18:53:09.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25327.084412 secs 2024-04-26T18:53:09.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25329.971333 secs 2024-04-26T18:53:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:09 smithi033 ceph-mon[27845]: pgmap v13894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:09 smithi096 ceph-mon[32753]: pgmap v13894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:09 smithi194 ceph-mon[33017]: pgmap v13894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:53:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:11 smithi033 ceph-mon[27845]: pgmap v13895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:53:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:11 smithi096 ceph-mon[32753]: pgmap v13895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:53:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:11 smithi194 ceph-mon[33017]: pgmap v13895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:53:14.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:13 smithi033 ceph-mon[27845]: pgmap v13896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:14.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25332.084500 secs 2024-04-26T18:53:14.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25334.971455 secs 2024-04-26T18:53:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:13 smithi096 ceph-mon[32753]: pgmap v13896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25332.084500 secs 2024-04-26T18:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25334.971455 secs 2024-04-26T18:53:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:13 smithi194 ceph-mon[33017]: pgmap v13896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25332.084500 secs 2024-04-26T18:53:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25334.971455 secs 2024-04-26T18:53:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:15 smithi033 ceph-mon[27845]: pgmap v13897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:15 smithi096 ceph-mon[32753]: pgmap v13897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:15 smithi194 ceph-mon[33017]: pgmap v13897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:17 smithi033 ceph-mon[27845]: pgmap v13898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:17 smithi096 ceph-mon[32753]: pgmap v13898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:17 smithi194 ceph-mon[33017]: pgmap v13898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25337.084559 secs 2024-04-26T18:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25339.971577 secs 2024-04-26T18:53:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25337.084559 secs 2024-04-26T18:53:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25339.971577 secs 2024-04-26T18:53:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25337.084559 secs 2024-04-26T18:53:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25339.971577 secs 2024-04-26T18:53:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:19 smithi194 ceph-mon[33017]: pgmap v13899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:19 smithi033 ceph-mon[27845]: pgmap v13899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:19 smithi096 ceph-mon[32753]: pgmap v13899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:21.399 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:21.399+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:21.399 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:21.399+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:21.399 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:21.399+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:53:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:21 smithi194 ceph-mon[33017]: pgmap v13900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:21 smithi033 ceph-mon[27845]: pgmap v13900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:21 smithi096 ceph-mon[32753]: pgmap v13900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:24.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:23 smithi033 ceph-mon[27845]: pgmap v13901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:24.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25342.084643 secs 2024-04-26T18:53:24.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25344.971717 secs 2024-04-26T18:53:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:23 smithi194 ceph-mon[33017]: pgmap v13901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25342.084643 secs 2024-04-26T18:53:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25344.971717 secs 2024-04-26T18:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:23 smithi096 ceph-mon[32753]: pgmap v13901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25342.084643 secs 2024-04-26T18:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25344.971717 secs 2024-04-26T18:53:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:26.107+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:53:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:26.107+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T18:53:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:26.107+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 201 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T18:53:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:26.108+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 201 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:53:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:25 smithi194 ceph-mon[33017]: pgmap v13902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:25 smithi033 ceph-mon[27845]: pgmap v13902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:25 smithi096 ceph-mon[32753]: pgmap v13902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:27 smithi194 ceph-mon[33017]: pgmap v13903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:27 smithi033 ceph-mon[27845]: pgmap v13903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:27 smithi096 ceph-mon[32753]: pgmap v13903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25347.084729 secs 2024-04-26T18:53:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25349.971814 secs 2024-04-26T18:53:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25347.084729 secs 2024-04-26T18:53:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25349.971814 secs 2024-04-26T18:53:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25347.084729 secs 2024-04-26T18:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25349.971814 secs 2024-04-26T18:53:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:29 smithi194 ceph-mon[33017]: pgmap v13904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:29 smithi033 ceph-mon[27845]: pgmap v13904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:29 smithi096 ceph-mon[32753]: pgmap v13904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:31 smithi194 ceph-mon[33017]: pgmap v13905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:31 smithi033 ceph-mon[27845]: pgmap v13905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:31 smithi096 ceph-mon[32753]: pgmap v13905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:33 smithi194 ceph-mon[33017]: pgmap v13906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25352.084814 secs 2024-04-26T18:53:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25354.971962 secs 2024-04-26T18:53:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:33 smithi033 ceph-mon[27845]: pgmap v13906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25352.084814 secs 2024-04-26T18:53:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25354.971962 secs 2024-04-26T18:53:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:33 smithi096 ceph-mon[32753]: pgmap v13906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25352.084814 secs 2024-04-26T18:53:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25354.971962 secs 2024-04-26T18:53:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:35 smithi194 ceph-mon[33017]: pgmap v13907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:36.409+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:36.409+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:36.409+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:53:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:35 smithi033 ceph-mon[27845]: pgmap v13907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:35 smithi096 ceph-mon[32753]: pgmap v13907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:37 smithi194 ceph-mon[33017]: pgmap v13908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:37 smithi033 ceph-mon[27845]: pgmap v13908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:37 smithi096 ceph-mon[32753]: pgmap v13908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:39.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25357.084899 secs 2024-04-26T18:53:39.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25359.972081 secs 2024-04-26T18:53:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25357.084899 secs 2024-04-26T18:53:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25359.972081 secs 2024-04-26T18:53:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25357.084899 secs 2024-04-26T18:53:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25359.972081 secs 2024-04-26T18:53:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:40 smithi194 ceph-mon[33017]: pgmap v13909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:40 smithi033 ceph-mon[27845]: pgmap v13909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:40 smithi096 ceph-mon[32753]: pgmap v13909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:42 smithi194 ceph-mon[33017]: pgmap v13910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:42 smithi033 ceph-mon[27845]: pgmap v13910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:42 smithi096 ceph-mon[32753]: pgmap v13910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:44 smithi194 ceph-mon[33017]: pgmap v13911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25362.084961 secs 2024-04-26T18:53:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25364.972212 secs 2024-04-26T18:53:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:44 smithi033 ceph-mon[27845]: pgmap v13911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25362.084961 secs 2024-04-26T18:53:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25364.972212 secs 2024-04-26T18:53:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:44 smithi096 ceph-mon[32753]: pgmap v13911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25362.084961 secs 2024-04-26T18:53:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25364.972212 secs 2024-04-26T18:53:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:46 smithi194 ceph-mon[33017]: pgmap v13912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:46 smithi033 ceph-mon[27845]: pgmap v13912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:46 smithi096 ceph-mon[32753]: pgmap v13912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:48 smithi194 ceph-mon[33017]: pgmap v13913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:48 smithi033 ceph-mon[27845]: pgmap v13913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:48 smithi096 ceph-mon[32753]: pgmap v13913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:49.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25367.085022 secs 2024-04-26T18:53:49.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25369.972287 secs 2024-04-26T18:53:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25367.085022 secs 2024-04-26T18:53:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25369.972287 secs 2024-04-26T18:53:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25367.085022 secs 2024-04-26T18:53:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25369.972287 secs 2024-04-26T18:53:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:50 smithi194 ceph-mon[33017]: pgmap v13914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:50 smithi033 ceph-mon[27845]: pgmap v13914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:50.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:50 smithi096 ceph-mon[32753]: pgmap v13914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:51.420+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:51.420+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:53:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:53:51.420+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:53:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:52 smithi194 ceph-mon[33017]: pgmap v13915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:52 smithi033 ceph-mon[27845]: pgmap v13915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:52 smithi096 ceph-mon[32753]: pgmap v13915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:54 smithi194 ceph-mon[33017]: pgmap v13916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25372.085084 secs 2024-04-26T18:53:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25374.972403 secs 2024-04-26T18:53:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:54 smithi033 ceph-mon[27845]: pgmap v13916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25372.085084 secs 2024-04-26T18:53:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25374.972403 secs 2024-04-26T18:53:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:54 smithi096 ceph-mon[32753]: pgmap v13916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25372.085084 secs 2024-04-26T18:53:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25374.972403 secs 2024-04-26T18:53:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:56 smithi194 ceph-mon[33017]: pgmap v13917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:56 smithi033 ceph-mon[27845]: pgmap v13917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:56 smithi096 ceph-mon[32753]: pgmap v13917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:53:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:58 smithi194 ceph-mon[33017]: pgmap v13918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:58 smithi033 ceph-mon[27845]: pgmap v13918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:58 smithi096 ceph-mon[32753]: pgmap v13918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:53:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25377.085145 secs 2024-04-26T18:53:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25379.972484 secs 2024-04-26T18:53:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25377.085145 secs 2024-04-26T18:53:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25379.972484 secs 2024-04-26T18:53:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25377.085145 secs 2024-04-26T18:53:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25379.972484 secs 2024-04-26T18:54:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:00 smithi194 ceph-mon[33017]: pgmap v13919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:00 smithi033 ceph-mon[27845]: pgmap v13919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:00 smithi096 ceph-mon[32753]: pgmap v13919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:02 smithi194 ceph-mon[33017]: pgmap v13920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:54:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:02 smithi033 ceph-mon[27845]: pgmap v13920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:54:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:02 smithi096 ceph-mon[32753]: pgmap v13920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:54:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:54:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:54:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:54:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:54:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:54:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:54:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:54:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:54:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:54:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:04 smithi194 ceph-mon[33017]: pgmap v13921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:04 smithi194 ceph-mon[33017]: pgmap v13922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25382.085206 secs 2024-04-26T18:54:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25384.972600 secs 2024-04-26T18:54:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:04 smithi033 ceph-mon[27845]: pgmap v13921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:04 smithi033 ceph-mon[27845]: pgmap v13922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25382.085206 secs 2024-04-26T18:54:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25384.972600 secs 2024-04-26T18:54:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:04 smithi096 ceph-mon[32753]: pgmap v13921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:04 smithi096 ceph-mon[32753]: pgmap v13922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25382.085206 secs 2024-04-26T18:54:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25384.972600 secs 2024-04-26T18:54:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:06 smithi033 ceph-mon[27845]: pgmap v13923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:54:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:06 smithi096 ceph-mon[32753]: pgmap v13923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:54:06.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:06.436+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:06.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:06.437+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:06.438 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:06.437+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:54:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:06 smithi194 ceph-mon[33017]: pgmap v13923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:54:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:08 smithi033 ceph-mon[27845]: pgmap v13924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:08 smithi096 ceph-mon[32753]: pgmap v13924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:08 smithi194 ceph-mon[33017]: pgmap v13924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25387.085267 secs 2024-04-26T18:54:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25389.972716 secs 2024-04-26T18:54:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25387.085267 secs 2024-04-26T18:54:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25389.972716 secs 2024-04-26T18:54:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25387.085267 secs 2024-04-26T18:54:09.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25389.972716 secs 2024-04-26T18:54:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:10 smithi033 ceph-mon[27845]: pgmap v13925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:10 smithi096 ceph-mon[32753]: pgmap v13925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:10 smithi194 ceph-mon[33017]: pgmap v13925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:12 smithi033 ceph-mon[27845]: pgmap v13926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:12 smithi096 ceph-mon[32753]: pgmap v13926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:12 smithi194 ceph-mon[33017]: pgmap v13926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:54:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:14 smithi033 ceph-mon[27845]: pgmap v13927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:54:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25392.085325 secs 2024-04-26T18:54:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25394.972836 secs 2024-04-26T18:54:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:14 smithi096 ceph-mon[32753]: pgmap v13927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:54:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25392.085325 secs 2024-04-26T18:54:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25394.972836 secs 2024-04-26T18:54:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:14 smithi194 ceph-mon[33017]: pgmap v13927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:54:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25392.085325 secs 2024-04-26T18:54:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25394.972836 secs 2024-04-26T18:54:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:16 smithi033 ceph-mon[27845]: pgmap v13928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:16 smithi096 ceph-mon[32753]: pgmap v13928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:16 smithi194 ceph-mon[33017]: pgmap v13928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:18 smithi033 ceph-mon[27845]: pgmap v13929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:54:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:18 smithi096 ceph-mon[32753]: pgmap v13929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:54:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:18 smithi194 ceph-mon[33017]: pgmap v13929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:54:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25397.085627 secs 2024-04-26T18:54:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25399.972951 secs 2024-04-26T18:54:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25397.085627 secs 2024-04-26T18:54:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25399.972951 secs 2024-04-26T18:54:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25397.085627 secs 2024-04-26T18:54:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25399.972951 secs 2024-04-26T18:54:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:20 smithi033 ceph-mon[27845]: pgmap v13930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:20 smithi096 ceph-mon[32753]: pgmap v13930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:20 smithi194 ceph-mon[33017]: pgmap v13930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:21.452+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:21.452+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:54:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:21.452+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:22 smithi033 ceph-mon[27845]: pgmap v13931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:22 smithi096 ceph-mon[32753]: pgmap v13931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:22 smithi194 ceph-mon[33017]: pgmap v13931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:24 smithi033 ceph-mon[27845]: pgmap v13932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25402.085715 secs 2024-04-26T18:54:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25404.973086 secs 2024-04-26T18:54:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:24 smithi096 ceph-mon[32753]: pgmap v13932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25402.085715 secs 2024-04-26T18:54:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25404.973086 secs 2024-04-26T18:54:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:24 smithi194 ceph-mon[33017]: pgmap v13932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25402.085715 secs 2024-04-26T18:54:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25404.973086 secs 2024-04-26T18:54:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:26 smithi033 ceph-mon[27845]: pgmap v13933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:26 smithi096 ceph-mon[32753]: pgmap v13933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:26 smithi194 ceph-mon[33017]: pgmap v13933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:28 smithi096 ceph-mon[32753]: pgmap v13934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:28 smithi194 ceph-mon[33017]: pgmap v13934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:28 smithi033 ceph-mon[27845]: pgmap v13934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25407.085817 secs 2024-04-26T18:54:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25409.973196 secs 2024-04-26T18:54:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25407.085817 secs 2024-04-26T18:54:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25409.973196 secs 2024-04-26T18:54:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25407.085817 secs 2024-04-26T18:54:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25409.973196 secs 2024-04-26T18:54:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:30 smithi194 ceph-mon[33017]: pgmap v13935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:30 smithi033 ceph-mon[27845]: pgmap v13935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:30 smithi096 ceph-mon[32753]: pgmap v13935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:32 smithi194 ceph-mon[33017]: pgmap v13936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:32 smithi033 ceph-mon[27845]: pgmap v13936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:32 smithi096 ceph-mon[32753]: pgmap v13936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25412.085904 secs 2024-04-26T18:54:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25412.085904 secs 2024-04-26T18:54:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25412.085904 secs 2024-04-26T18:54:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:34 smithi194 ceph-mon[33017]: pgmap v13937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25414.973269 secs 2024-04-26T18:54:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:34 smithi033 ceph-mon[27845]: pgmap v13937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25414.973269 secs 2024-04-26T18:54:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:34 smithi096 ceph-mon[32753]: pgmap v13937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25414.973269 secs 2024-04-26T18:54:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:36.467+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:36.467+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:36.467+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:54:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:36 smithi194 ceph-mon[33017]: pgmap v13938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:36 smithi033 ceph-mon[27845]: pgmap v13938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:36 smithi096 ceph-mon[32753]: pgmap v13938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:38 smithi194 ceph-mon[33017]: pgmap v13939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25417.086008 secs 2024-04-26T18:54:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:38 smithi033 ceph-mon[27845]: pgmap v13939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25417.086008 secs 2024-04-26T18:54:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:38 smithi096 ceph-mon[32753]: pgmap v13939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25417.086008 secs 2024-04-26T18:54:39.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25419.973380 secs 2024-04-26T18:54:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25419.973380 secs 2024-04-26T18:54:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25419.973380 secs 2024-04-26T18:54:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:40 smithi194 ceph-mon[33017]: pgmap v13940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:40 smithi033 ceph-mon[27845]: pgmap v13940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:40 smithi096 ceph-mon[32753]: pgmap v13940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:42 smithi194 ceph-mon[33017]: pgmap v13941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:42 smithi033 ceph-mon[27845]: pgmap v13941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:42 smithi096 ceph-mon[32753]: pgmap v13941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25422.086071 secs 2024-04-26T18:54:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25422.086071 secs 2024-04-26T18:54:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25422.086071 secs 2024-04-26T18:54:44.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:44 smithi096 ceph-mon[32753]: pgmap v13942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:44.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25424.973491 secs 2024-04-26T18:54:44.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:44 smithi194 ceph-mon[33017]: pgmap v13942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25424.973491 secs 2024-04-26T18:54:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:44 smithi033 ceph-mon[27845]: pgmap v13942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25424.973491 secs 2024-04-26T18:54:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:46 smithi194 ceph-mon[33017]: pgmap v13943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:46 smithi033 ceph-mon[27845]: pgmap v13943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:46 smithi096 ceph-mon[32753]: pgmap v13943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:48 smithi194 ceph-mon[33017]: pgmap v13944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25427.086159 secs 2024-04-26T18:54:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:48 smithi033 ceph-mon[27845]: pgmap v13944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25427.086159 secs 2024-04-26T18:54:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:48 smithi096 ceph-mon[32753]: pgmap v13944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25427.086159 secs 2024-04-26T18:54:49.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25429.973627 secs 2024-04-26T18:54:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25429.973627 secs 2024-04-26T18:54:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25429.973627 secs 2024-04-26T18:54:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:50 smithi194 ceph-mon[33017]: pgmap v13945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:50 smithi033 ceph-mon[27845]: pgmap v13945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:50 smithi096 ceph-mon[32753]: pgmap v13945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:51.468+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:51.468+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:54:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:54:51.468+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:54:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:52 smithi194 ceph-mon[33017]: pgmap v13946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:52 smithi033 ceph-mon[27845]: pgmap v13946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:52 smithi096 ceph-mon[32753]: pgmap v13946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25432.086243 secs 2024-04-26T18:54:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25432.086243 secs 2024-04-26T18:54:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25432.086243 secs 2024-04-26T18:54:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:54 smithi194 ceph-mon[33017]: pgmap v13947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25434.973750 secs 2024-04-26T18:54:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:54 smithi033 ceph-mon[27845]: pgmap v13947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25434.973750 secs 2024-04-26T18:54:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:54 smithi096 ceph-mon[32753]: pgmap v13947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25434.973750 secs 2024-04-26T18:54:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:56 smithi194 ceph-mon[33017]: pgmap v13948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:56 smithi033 ceph-mon[27845]: pgmap v13948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:56 smithi096 ceph-mon[32753]: pgmap v13948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:54:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:58 smithi194 ceph-mon[33017]: pgmap v13949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25437.086306 secs 2024-04-26T18:54:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:58 smithi033 ceph-mon[27845]: pgmap v13949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25437.086306 secs 2024-04-26T18:54:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:58 smithi096 ceph-mon[32753]: pgmap v13949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:54:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25437.086306 secs 2024-04-26T18:54:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25439.973868 secs 2024-04-26T18:54:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25439.973868 secs 2024-04-26T18:54:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25439.973868 secs 2024-04-26T18:55:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:00 smithi194 ceph-mon[33017]: pgmap v13950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:00 smithi033 ceph-mon[27845]: pgmap v13950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:00 smithi096 ceph-mon[32753]: pgmap v13950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:02 smithi194 ceph-mon[33017]: pgmap v13951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:02 smithi033 ceph-mon[27845]: pgmap v13951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:02 smithi096 ceph-mon[32753]: pgmap v13951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:55:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25442.086370 secs 2024-04-26T18:55:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:55:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25442.086370 secs 2024-04-26T18:55:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:55:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25442.086370 secs 2024-04-26T18:55:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:04 smithi194 ceph-mon[33017]: pgmap v13952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25444.973958 secs 2024-04-26T18:55:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:55:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:55:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:55:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:04 smithi033 ceph-mon[27845]: pgmap v13952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25444.973958 secs 2024-04-26T18:55:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:55:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:55:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:04 smithi096 ceph-mon[32753]: pgmap v13952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25444.973958 secs 2024-04-26T18:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:55:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:55:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:05 smithi194 ceph-mon[33017]: pgmap v13953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:05 smithi194 ceph-mon[33017]: pgmap v13954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:55:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:05 smithi033 ceph-mon[27845]: pgmap v13953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:05 smithi033 ceph-mon[27845]: pgmap v13954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:55:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:05 smithi096 ceph-mon[32753]: pgmap v13953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:05 smithi096 ceph-mon[32753]: pgmap v13954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:55:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:06.484+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:06.484+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:06.484+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:55:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:07 smithi033 ceph-mon[27845]: pgmap v13955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:07 smithi096 ceph-mon[32753]: pgmap v13955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:07 smithi194 ceph-mon[33017]: pgmap v13955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25447.086457 secs 2024-04-26T18:55:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25447.086457 secs 2024-04-26T18:55:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25447.086457 secs 2024-04-26T18:55:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:09 smithi033 ceph-mon[27845]: pgmap v13956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25449.974028 secs 2024-04-26T18:55:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:09 smithi096 ceph-mon[32753]: pgmap v13956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25449.974028 secs 2024-04-26T18:55:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:09 smithi194 ceph-mon[33017]: pgmap v13956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25449.974028 secs 2024-04-26T18:55:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:11 smithi033 ceph-mon[27845]: pgmap v13957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:11 smithi096 ceph-mon[32753]: pgmap v13957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:11 smithi194 ceph-mon[33017]: pgmap v13957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:13 smithi033 ceph-mon[27845]: pgmap v13958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25452.086540 secs 2024-04-26T18:55:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:13 smithi096 ceph-mon[32753]: pgmap v13958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25452.086540 secs 2024-04-26T18:55:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:13 smithi194 ceph-mon[33017]: pgmap v13958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:55:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25452.086540 secs 2024-04-26T18:55:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25454.974175 secs 2024-04-26T18:55:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25454.974175 secs 2024-04-26T18:55:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25454.974175 secs 2024-04-26T18:55:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:15 smithi033 ceph-mon[27845]: pgmap v13959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:55:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:15 smithi096 ceph-mon[32753]: pgmap v13959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:55:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:15 smithi194 ceph-mon[33017]: pgmap v13959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:55:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:17 smithi033 ceph-mon[27845]: pgmap v13960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:17 smithi096 ceph-mon[32753]: pgmap v13960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:17 smithi194 ceph-mon[33017]: pgmap v13960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25457.086600 secs 2024-04-26T18:55:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25457.086600 secs 2024-04-26T18:55:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25457.086600 secs 2024-04-26T18:55:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:19 smithi033 ceph-mon[27845]: pgmap v13961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25459.974304 secs 2024-04-26T18:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:19 smithi096 ceph-mon[32753]: pgmap v13961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25459.974304 secs 2024-04-26T18:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:19 smithi194 ceph-mon[33017]: pgmap v13961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25459.974304 secs 2024-04-26T18:55:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:21.498+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:21.498+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:21.498+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:55:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:21 smithi033 ceph-mon[27845]: pgmap v13962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:21 smithi096 ceph-mon[32753]: pgmap v13962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:21 smithi194 ceph-mon[33017]: pgmap v13962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:23 smithi033 ceph-mon[27845]: pgmap v13963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25462.086682 secs 2024-04-26T18:55:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:23 smithi096 ceph-mon[32753]: pgmap v13963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25462.086682 secs 2024-04-26T18:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:23 smithi194 ceph-mon[33017]: pgmap v13963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25462.086682 secs 2024-04-26T18:55:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25464.974422 secs 2024-04-26T18:55:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25464.974422 secs 2024-04-26T18:55:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25464.974422 secs 2024-04-26T18:55:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:25 smithi194 ceph-mon[33017]: pgmap v13964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:25 smithi033 ceph-mon[27845]: pgmap v13964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:25 smithi096 ceph-mon[32753]: pgmap v13964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:27 smithi194 ceph-mon[33017]: pgmap v13965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:27 smithi033 ceph-mon[27845]: pgmap v13965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:27 smithi096 ceph-mon[32753]: pgmap v13965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25467.086768 secs 2024-04-26T18:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25467.086768 secs 2024-04-26T18:55:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25467.086768 secs 2024-04-26T18:55:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:29 smithi194 ceph-mon[33017]: pgmap v13966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25469.974541 secs 2024-04-26T18:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:29 smithi033 ceph-mon[27845]: pgmap v13966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25469.974541 secs 2024-04-26T18:55:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:29 smithi096 ceph-mon[32753]: pgmap v13966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25469.974541 secs 2024-04-26T18:55:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:31 smithi194 ceph-mon[33017]: pgmap v13967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:31 smithi033 ceph-mon[27845]: pgmap v13967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:31 smithi096 ceph-mon[32753]: pgmap v13967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:33 smithi194 ceph-mon[33017]: pgmap v13968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25472.086852 secs 2024-04-26T18:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:33 smithi033 ceph-mon[27845]: pgmap v13968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25472.086852 secs 2024-04-26T18:55:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:33 smithi096 ceph-mon[32753]: pgmap v13968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25472.086852 secs 2024-04-26T18:55:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25474.974653 secs 2024-04-26T18:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25474.974653 secs 2024-04-26T18:55:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25474.974653 secs 2024-04-26T18:55:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:35 smithi194 ceph-mon[33017]: pgmap v13969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:35 smithi033 ceph-mon[27845]: pgmap v13969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:35 smithi096 ceph-mon[32753]: pgmap v13969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:36.509+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:36.509+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:55:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:36.509+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:36 smithi194 ceph-mon[33017]: pgmap v13970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:36 smithi033 ceph-mon[27845]: pgmap v13970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:36 smithi096 ceph-mon[32753]: pgmap v13970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25477.086939 secs 2024-04-26T18:55:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25477.086939 secs 2024-04-26T18:55:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25477.086939 secs 2024-04-26T18:55:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:39 smithi194 ceph-mon[33017]: pgmap v13971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25479.974772 secs 2024-04-26T18:55:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:39 smithi033 ceph-mon[27845]: pgmap v13971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25479.974772 secs 2024-04-26T18:55:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:39 smithi096 ceph-mon[32753]: pgmap v13971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25479.974772 secs 2024-04-26T18:55:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:41 smithi194 ceph-mon[33017]: pgmap v13972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:41 smithi033 ceph-mon[27845]: pgmap v13972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:41 smithi096 ceph-mon[32753]: pgmap v13972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:43 smithi194 ceph-mon[33017]: pgmap v13973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25482.087033 secs 2024-04-26T18:55:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:43 smithi033 ceph-mon[27845]: pgmap v13973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25482.087033 secs 2024-04-26T18:55:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:43 smithi096 ceph-mon[32753]: pgmap v13973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25482.087033 secs 2024-04-26T18:55:44.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25484.974905 secs 2024-04-26T18:55:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25484.974905 secs 2024-04-26T18:55:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25484.974905 secs 2024-04-26T18:55:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:45 smithi194 ceph-mon[33017]: pgmap v13974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:45 smithi033 ceph-mon[27845]: pgmap v13974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:45 smithi096 ceph-mon[32753]: pgmap v13974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:47 smithi194 ceph-mon[33017]: pgmap v13975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:47 smithi033 ceph-mon[27845]: pgmap v13975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:47 smithi096 ceph-mon[32753]: pgmap v13975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25487.087118 secs 2024-04-26T18:55:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25487.087118 secs 2024-04-26T18:55:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25487.087118 secs 2024-04-26T18:55:49.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:49 smithi033 ceph-mon[27845]: pgmap v13976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:49.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25489.975025 secs 2024-04-26T18:55:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:49 smithi194 ceph-mon[33017]: pgmap v13976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25489.975025 secs 2024-04-26T18:55:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:49 smithi096 ceph-mon[32753]: pgmap v13976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25489.975025 secs 2024-04-26T18:55:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:51.514+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:51.514+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:55:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:55:51.514+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:55:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:51 smithi194 ceph-mon[33017]: pgmap v13977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:51 smithi033 ceph-mon[27845]: pgmap v13977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:51 smithi096 ceph-mon[32753]: pgmap v13977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:53 smithi194 ceph-mon[33017]: pgmap v13978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25492.087180 secs 2024-04-26T18:55:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:53 smithi033 ceph-mon[27845]: pgmap v13978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25492.087180 secs 2024-04-26T18:55:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:53 smithi096 ceph-mon[32753]: pgmap v13978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25492.087180 secs 2024-04-26T18:55:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25494.975193 secs 2024-04-26T18:55:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25494.975193 secs 2024-04-26T18:55:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25494.975193 secs 2024-04-26T18:55:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:55 smithi194 ceph-mon[33017]: pgmap v13979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:55 smithi033 ceph-mon[27845]: pgmap v13979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:55 smithi096 ceph-mon[32753]: pgmap v13979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:57 smithi033 ceph-mon[27845]: pgmap v13980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:57 smithi096 ceph-mon[32753]: pgmap v13980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:57 smithi194 ceph-mon[33017]: pgmap v13980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:55:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25497.087270 secs 2024-04-26T18:55:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25497.087270 secs 2024-04-26T18:55:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25497.087270 secs 2024-04-26T18:55:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:59 smithi033 ceph-mon[27845]: pgmap v13981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:55:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25499.975306 secs 2024-04-26T18:55:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:59 smithi096 ceph-mon[32753]: pgmap v13981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:55:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25499.975306 secs 2024-04-26T18:55:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:59 smithi194 ceph-mon[33017]: pgmap v13981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:55:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:55:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25499.975306 secs 2024-04-26T18:56:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:01 smithi033 ceph-mon[27845]: pgmap v13982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:01 smithi096 ceph-mon[32753]: pgmap v13982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:01 smithi194 ceph-mon[33017]: pgmap v13982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:03 smithi033 ceph-mon[27845]: pgmap v13983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25502.087352 secs 2024-04-26T18:56:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:03 smithi096 ceph-mon[32753]: pgmap v13983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25502.087352 secs 2024-04-26T18:56:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:03 smithi194 ceph-mon[33017]: pgmap v13983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25502.087352 secs 2024-04-26T18:56:04.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25504.975385 secs 2024-04-26T18:56:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25504.975385 secs 2024-04-26T18:56:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25504.975385 secs 2024-04-26T18:56:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:05 smithi033 ceph-mon[27845]: pgmap v13984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:56:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:05 smithi096 ceph-mon[32753]: pgmap v13984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:56:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:05 smithi194 ceph-mon[33017]: pgmap v13984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:56:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:06.530+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:06.530+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:06.530+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:56:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:07 smithi194 ceph-mon[33017]: pgmap v13985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:56:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:56:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:07 smithi033 ceph-mon[27845]: pgmap v13985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:56:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:56:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:07 smithi096 ceph-mon[32753]: pgmap v13985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:56:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:56:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:56:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:08 smithi194 ceph-mon[33017]: pgmap v13986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:56:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:08 smithi194 ceph-mon[33017]: pgmap v13987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25507.087432 secs 2024-04-26T18:56:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:08 smithi033 ceph-mon[27845]: pgmap v13986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:56:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:08 smithi033 ceph-mon[27845]: pgmap v13987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25507.087432 secs 2024-04-26T18:56:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:08 smithi096 ceph-mon[32753]: pgmap v13986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:56:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:08 smithi096 ceph-mon[32753]: pgmap v13987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25507.087432 secs 2024-04-26T18:56:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25509.975497 secs 2024-04-26T18:56:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25509.975497 secs 2024-04-26T18:56:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25509.975497 secs 2024-04-26T18:56:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:10 smithi194 ceph-mon[33017]: pgmap v13988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:56:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:10 smithi033 ceph-mon[27845]: pgmap v13988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:56:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:10 smithi096 ceph-mon[32753]: pgmap v13988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:56:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:12 smithi194 ceph-mon[33017]: pgmap v13989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:12 smithi033 ceph-mon[27845]: pgmap v13989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:12 smithi096 ceph-mon[32753]: pgmap v13989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:13 smithi194 ceph-mon[33017]: pgmap v13990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25512.087492 secs 2024-04-26T18:56:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:13 smithi033 ceph-mon[27845]: pgmap v13990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25512.087492 secs 2024-04-26T18:56:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:13 smithi096 ceph-mon[32753]: pgmap v13990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25512.087492 secs 2024-04-26T18:56:14.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25514.975571 secs 2024-04-26T18:56:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25514.975571 secs 2024-04-26T18:56:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25514.975571 secs 2024-04-26T18:56:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:15 smithi194 ceph-mon[33017]: pgmap v13991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:15 smithi033 ceph-mon[27845]: pgmap v13991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:15 smithi096 ceph-mon[32753]: pgmap v13991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T18:56:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:17 smithi033 ceph-mon[27845]: pgmap v13992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:56:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:17 smithi096 ceph-mon[32753]: pgmap v13992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:56:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:17 smithi194 ceph-mon[33017]: pgmap v13992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:56:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25517.087578 secs 2024-04-26T18:56:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25517.087578 secs 2024-04-26T18:56:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25517.087578 secs 2024-04-26T18:56:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25519.975694 secs 2024-04-26T18:56:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:19 smithi033 ceph-mon[27845]: pgmap v13993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25519.975694 secs 2024-04-26T18:56:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:19 smithi096 ceph-mon[32753]: pgmap v13993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25519.975694 secs 2024-04-26T18:56:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:19 smithi194 ceph-mon[33017]: pgmap v13993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:21.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.532+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:21.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.532+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:21.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.532+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:56:21.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.786+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000343) for more than 900000000 us, fault. 2024-04-26T18:56:21.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.786+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0043050 comp rx=0 tx=0).stop 2024-04-26T18:56:21.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.786+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b31b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T18:56:21.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.786+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b31b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:56:21.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.786+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:56:21.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.787+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:56:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:21.788+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feef000a010 tx=0x7feef0022040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:56:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:21 smithi033 ceph-mon[27845]: pgmap v13994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:21 smithi096 ceph-mon[32753]: pgmap v13994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:21 smithi194 ceph-mon[33017]: pgmap v13994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25522.087656 secs 2024-04-26T18:56:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:23 smithi033 ceph-mon[27845]: pgmap v13995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25524.975831 secs 2024-04-26T18:56:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25522.087656 secs 2024-04-26T18:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:23 smithi096 ceph-mon[32753]: pgmap v13995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25524.975831 secs 2024-04-26T18:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25522.087656 secs 2024-04-26T18:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:23 smithi194 ceph-mon[33017]: pgmap v13995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25524.975831 secs 2024-04-26T18:56:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:26.137+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:56:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:26.137+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T18:56:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:26.138+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 202 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T18:56:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:26.138+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 202 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:56:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:25 smithi033 ceph-mon[27845]: pgmap v13996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:25 smithi096 ceph-mon[32753]: pgmap v13996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:25 smithi194 ceph-mon[33017]: pgmap v13996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:27 smithi033 ceph-mon[27845]: pgmap v13997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:27 smithi096 ceph-mon[32753]: pgmap v13997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:27 smithi194 ceph-mon[33017]: pgmap v13997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25527.087740 secs 2024-04-26T18:56:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25529.975969 secs 2024-04-26T18:56:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25527.087740 secs 2024-04-26T18:56:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25529.975969 secs 2024-04-26T18:56:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25527.087740 secs 2024-04-26T18:56:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25529.975969 secs 2024-04-26T18:56:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:29 smithi194 ceph-mon[33017]: pgmap v13998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:29 smithi033 ceph-mon[27845]: pgmap v13998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:29 smithi096 ceph-mon[32753]: pgmap v13998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:32.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:31 smithi033 ceph-mon[27845]: pgmap v13999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:31 smithi194 ceph-mon[33017]: pgmap v13999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:31 smithi096 ceph-mon[32753]: pgmap v13999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25532.087822 secs 2024-04-26T18:56:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:33 smithi194 ceph-mon[33017]: pgmap v14000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25534.976150 secs 2024-04-26T18:56:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25532.087822 secs 2024-04-26T18:56:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:33 smithi033 ceph-mon[27845]: pgmap v14000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25534.976150 secs 2024-04-26T18:56:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25532.087822 secs 2024-04-26T18:56:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:33 smithi096 ceph-mon[32753]: pgmap v14000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25534.976150 secs 2024-04-26T18:56:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:35 smithi194 ceph-mon[33017]: pgmap v14001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:35 smithi033 ceph-mon[27845]: pgmap v14001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:35 smithi096 ceph-mon[32753]: pgmap v14001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:36.547+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:36.547+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:36.547+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:56:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:37 smithi194 ceph-mon[33017]: pgmap v14002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:37 smithi033 ceph-mon[27845]: pgmap v14002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:37 smithi096 ceph-mon[32753]: pgmap v14002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:39.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25537.087907 secs 2024-04-26T18:56:39.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25539.976228 secs 2024-04-26T18:56:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25537.087907 secs 2024-04-26T18:56:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25539.976228 secs 2024-04-26T18:56:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25537.087907 secs 2024-04-26T18:56:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25539.976228 secs 2024-04-26T18:56:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:39 smithi194 ceph-mon[33017]: pgmap v14003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:39 smithi033 ceph-mon[27845]: pgmap v14003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:39 smithi096 ceph-mon[32753]: pgmap v14003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:41 smithi194 ceph-mon[33017]: pgmap v14004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:41 smithi033 ceph-mon[27845]: pgmap v14004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:41 smithi096 ceph-mon[32753]: pgmap v14004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:44.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25542.087991 secs 2024-04-26T18:56:44.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:43 smithi096 ceph-mon[32753]: pgmap v14005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:44.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25544.976348 secs 2024-04-26T18:56:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25542.087991 secs 2024-04-26T18:56:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:43 smithi194 ceph-mon[33017]: pgmap v14005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25544.976348 secs 2024-04-26T18:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25542.087991 secs 2024-04-26T18:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:43 smithi033 ceph-mon[27845]: pgmap v14005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25544.976348 secs 2024-04-26T18:56:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:45 smithi194 ceph-mon[33017]: pgmap v14006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:45 smithi033 ceph-mon[27845]: pgmap v14006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:45 smithi096 ceph-mon[32753]: pgmap v14006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:47 smithi194 ceph-mon[33017]: pgmap v14007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:47 smithi033 ceph-mon[27845]: pgmap v14007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:47 smithi096 ceph-mon[32753]: pgmap v14007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:49.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25547.088074 secs 2024-04-26T18:56:49.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25549.976459 secs 2024-04-26T18:56:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25547.088074 secs 2024-04-26T18:56:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25549.976459 secs 2024-04-26T18:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25547.088074 secs 2024-04-26T18:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25549.976459 secs 2024-04-26T18:56:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:50 smithi194 ceph-mon[33017]: pgmap v14008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:50 smithi033 ceph-mon[27845]: pgmap v14008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:50 smithi096 ceph-mon[32753]: pgmap v14008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:51.548+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:51.548+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:56:51.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:56:51.548+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:56:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:52 smithi194 ceph-mon[33017]: pgmap v14009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:52 smithi033 ceph-mon[27845]: pgmap v14009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:52 smithi096 ceph-mon[32753]: pgmap v14009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25552.088155 secs 2024-04-26T18:56:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:54 smithi194 ceph-mon[33017]: pgmap v14010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25554.976582 secs 2024-04-26T18:56:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25552.088155 secs 2024-04-26T18:56:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:54 smithi033 ceph-mon[27845]: pgmap v14010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25554.976582 secs 2024-04-26T18:56:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25552.088155 secs 2024-04-26T18:56:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:54 smithi096 ceph-mon[32753]: pgmap v14010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25554.976582 secs 2024-04-26T18:56:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:56 smithi194 ceph-mon[33017]: pgmap v14011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:56 smithi033 ceph-mon[27845]: pgmap v14011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:56 smithi096 ceph-mon[32753]: pgmap v14011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:56:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:58 smithi194 ceph-mon[33017]: pgmap v14012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:58 smithi033 ceph-mon[27845]: pgmap v14012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:58 smithi096 ceph-mon[32753]: pgmap v14012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:56:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25557.088242 secs 2024-04-26T18:56:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:56:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25559.976700 secs 2024-04-26T18:56:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25557.088242 secs 2024-04-26T18:56:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:56:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25559.976700 secs 2024-04-26T18:56:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25557.088242 secs 2024-04-26T18:56:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:56:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25559.976700 secs 2024-04-26T18:57:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:00 smithi194 ceph-mon[33017]: pgmap v14013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:00 smithi033 ceph-mon[27845]: pgmap v14013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:00 smithi096 ceph-mon[32753]: pgmap v14013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:02 smithi194 ceph-mon[33017]: pgmap v14014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:02 smithi033 ceph-mon[27845]: pgmap v14014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:02 smithi096 ceph-mon[32753]: pgmap v14014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25562.088327 secs 2024-04-26T18:57:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:04 smithi194 ceph-mon[33017]: pgmap v14015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25564.976848 secs 2024-04-26T18:57:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25562.088327 secs 2024-04-26T18:57:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:04 smithi033 ceph-mon[27845]: pgmap v14015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25564.976848 secs 2024-04-26T18:57:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25562.088327 secs 2024-04-26T18:57:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:04 smithi096 ceph-mon[32753]: pgmap v14015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25564.976848 secs 2024-04-26T18:57:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:06 smithi194 ceph-mon[33017]: pgmap v14016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:06 smithi033 ceph-mon[27845]: pgmap v14016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:06 smithi096 ceph-mon[32753]: pgmap v14016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:06.564+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:06.564+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:57:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:06.564+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:08 smithi194 ceph-mon[33017]: pgmap v14017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:57:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:08 smithi033 ceph-mon[27845]: pgmap v14017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:57:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:08 smithi096 ceph-mon[32753]: pgmap v14017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:57:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25567.088416 secs 2024-04-26T18:57:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:57:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:57:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:09 smithi194 ceph-mon[33017]: pgmap v14018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:09 smithi194 ceph-mon[33017]: pgmap v14019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:57:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25569.976910 secs 2024-04-26T18:57:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25567.088416 secs 2024-04-26T18:57:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:57:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:57:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:09 smithi033 ceph-mon[27845]: pgmap v14018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:09 smithi033 ceph-mon[27845]: pgmap v14019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:57:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25569.976910 secs 2024-04-26T18:57:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25567.088416 secs 2024-04-26T18:57:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:57:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:57:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:09 smithi096 ceph-mon[32753]: pgmap v14018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:09 smithi096 ceph-mon[32753]: pgmap v14019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:57:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25569.976910 secs 2024-04-26T18:57:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:10 smithi194 ceph-mon[33017]: pgmap v14020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:10 smithi033 ceph-mon[27845]: pgmap v14020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:10 smithi096 ceph-mon[32753]: pgmap v14020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:12 smithi194 ceph-mon[33017]: pgmap v14021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:12 smithi033 ceph-mon[27845]: pgmap v14021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:12 smithi096 ceph-mon[32753]: pgmap v14021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25572.088496 secs 2024-04-26T18:57:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25572.088496 secs 2024-04-26T18:57:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25572.088496 secs 2024-04-26T18:57:14.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25574.977018 secs 2024-04-26T18:57:14.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:14 smithi096 ceph-mon[32753]: pgmap v14022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25574.977018 secs 2024-04-26T18:57:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:14 smithi194 ceph-mon[33017]: pgmap v14022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25574.977018 secs 2024-04-26T18:57:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:14 smithi033 ceph-mon[27845]: pgmap v14022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:16 smithi194 ceph-mon[33017]: pgmap v14023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:16 smithi033 ceph-mon[27845]: pgmap v14023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:16 smithi096 ceph-mon[32753]: pgmap v14023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25577.088580 secs 2024-04-26T18:57:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25577.088580 secs 2024-04-26T18:57:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25577.088580 secs 2024-04-26T18:57:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:19 smithi194 ceph-mon[33017]: pgmap v14024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25579.977164 secs 2024-04-26T18:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:19 smithi033 ceph-mon[27845]: pgmap v14024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25579.977164 secs 2024-04-26T18:57:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:19 smithi096 ceph-mon[32753]: pgmap v14024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25579.977164 secs 2024-04-26T18:57:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:20 smithi194 ceph-mon[33017]: pgmap v14025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:20 smithi033 ceph-mon[27845]: pgmap v14025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:20 smithi096 ceph-mon[32753]: pgmap v14025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:21.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:21.569+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:21.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:21.569+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:21.569+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:57:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:22 smithi033 ceph-mon[27845]: pgmap v14026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:22 smithi096 ceph-mon[32753]: pgmap v14026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:22 smithi194 ceph-mon[33017]: pgmap v14026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25582.088665 secs 2024-04-26T18:57:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25582.088665 secs 2024-04-26T18:57:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25582.088665 secs 2024-04-26T18:57:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25584.977288 secs 2024-04-26T18:57:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:24 smithi194 ceph-mon[33017]: pgmap v14027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25584.977288 secs 2024-04-26T18:57:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:24 smithi033 ceph-mon[27845]: pgmap v14027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25584.977288 secs 2024-04-26T18:57:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:24 smithi096 ceph-mon[32753]: pgmap v14027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:26 smithi194 ceph-mon[33017]: pgmap v14028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:57:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:26 smithi033 ceph-mon[27845]: pgmap v14028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:57:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:26 smithi096 ceph-mon[32753]: pgmap v14028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:57:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25587.088752 secs 2024-04-26T18:57:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25587.088752 secs 2024-04-26T18:57:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25587.088752 secs 2024-04-26T18:57:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:29 smithi194 ceph-mon[33017]: pgmap v14029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25589.977436 secs 2024-04-26T18:57:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:29 smithi033 ceph-mon[27845]: pgmap v14029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25589.977436 secs 2024-04-26T18:57:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:29 smithi096 ceph-mon[32753]: pgmap v14029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25589.977436 secs 2024-04-26T18:57:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:30 smithi194 ceph-mon[33017]: pgmap v14030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:30 smithi033 ceph-mon[27845]: pgmap v14030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:30 smithi096 ceph-mon[32753]: pgmap v14030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:32 smithi194 ceph-mon[33017]: pgmap v14031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:32 smithi033 ceph-mon[27845]: pgmap v14031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:32 smithi096 ceph-mon[32753]: pgmap v14031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25592.088835 secs 2024-04-26T18:57:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25592.088835 secs 2024-04-26T18:57:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25592.088835 secs 2024-04-26T18:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25594.977564 secs 2024-04-26T18:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:34 smithi194 ceph-mon[33017]: pgmap v14032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25594.977564 secs 2024-04-26T18:57:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:34 smithi033 ceph-mon[27845]: pgmap v14032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25594.977564 secs 2024-04-26T18:57:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:34 smithi096 ceph-mon[32753]: pgmap v14032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:36.580+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:36.580+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:36.580+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:57:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:36 smithi194 ceph-mon[33017]: pgmap v14033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:36 smithi033 ceph-mon[27845]: pgmap v14033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:36 smithi096 ceph-mon[32753]: pgmap v14033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25597.088919 secs 2024-04-26T18:57:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25597.088919 secs 2024-04-26T18:57:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25597.088919 secs 2024-04-26T18:57:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:39 smithi194 ceph-mon[33017]: pgmap v14034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25599.977677 secs 2024-04-26T18:57:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:39 smithi033 ceph-mon[27845]: pgmap v14034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25599.977677 secs 2024-04-26T18:57:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:39 smithi096 ceph-mon[32753]: pgmap v14034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25599.977677 secs 2024-04-26T18:57:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:40 smithi194 ceph-mon[33017]: pgmap v14035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:57:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:40 smithi033 ceph-mon[27845]: pgmap v14035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:57:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:40 smithi096 ceph-mon[32753]: pgmap v14035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T18:57:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:42 smithi194 ceph-mon[33017]: pgmap v14036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:42 smithi033 ceph-mon[27845]: pgmap v14036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:42 smithi096 ceph-mon[32753]: pgmap v14036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T18:57:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25602.088999 secs 2024-04-26T18:57:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25602.088999 secs 2024-04-26T18:57:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25602.088999 secs 2024-04-26T18:57:44.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25604.977799 secs 2024-04-26T18:57:44.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:44 smithi096 ceph-mon[32753]: pgmap v14037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:57:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25604.977799 secs 2024-04-26T18:57:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:44 smithi194 ceph-mon[33017]: pgmap v14037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25604.977799 secs 2024-04-26T18:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:44 smithi033 ceph-mon[27845]: pgmap v14037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T18:57:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:46 smithi194 ceph-mon[33017]: pgmap v14038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:46 smithi033 ceph-mon[27845]: pgmap v14038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:46 smithi096 ceph-mon[32753]: pgmap v14038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:57:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25607.089061 secs 2024-04-26T18:57:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25607.089061 secs 2024-04-26T18:57:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25607.089061 secs 2024-04-26T18:57:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:49 smithi194 ceph-mon[33017]: pgmap v14039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25609.977959 secs 2024-04-26T18:57:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:49 smithi033 ceph-mon[27845]: pgmap v14039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25609.977959 secs 2024-04-26T18:57:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:49 smithi096 ceph-mon[32753]: pgmap v14039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25609.977959 secs 2024-04-26T18:57:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:50 smithi194 ceph-mon[33017]: pgmap v14040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:50 smithi033 ceph-mon[27845]: pgmap v14040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:50 smithi096 ceph-mon[32753]: pgmap v14040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:57:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:51.588+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:51.588+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:57:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:57:51.588+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:57:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:52 smithi194 ceph-mon[33017]: pgmap v14041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:52 smithi033 ceph-mon[27845]: pgmap v14041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:52 smithi096 ceph-mon[32753]: pgmap v14041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25612.089150 secs 2024-04-26T18:57:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25612.089150 secs 2024-04-26T18:57:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25612.089150 secs 2024-04-26T18:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25614.978102 secs 2024-04-26T18:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:54 smithi194 ceph-mon[33017]: pgmap v14042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25614.978102 secs 2024-04-26T18:57:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:54 smithi033 ceph-mon[27845]: pgmap v14042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25614.978102 secs 2024-04-26T18:57:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:54 smithi096 ceph-mon[32753]: pgmap v14042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:57:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:56 smithi194 ceph-mon[33017]: pgmap v14043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:57:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:56 smithi033 ceph-mon[27845]: pgmap v14043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:57:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:56 smithi096 ceph-mon[32753]: pgmap v14043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:57:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25617.089214 secs 2024-04-26T18:57:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25617.089214 secs 2024-04-26T18:57:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25617.089214 secs 2024-04-26T18:57:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:59 smithi194 ceph-mon[33017]: pgmap v14044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25619.978235 secs 2024-04-26T18:57:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:59 smithi033 ceph-mon[27845]: pgmap v14044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25619.978235 secs 2024-04-26T18:57:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:59 smithi096 ceph-mon[32753]: pgmap v14044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:57:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25619.978235 secs 2024-04-26T18:58:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:01 smithi194 ceph-mon[33017]: pgmap v14045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:01 smithi033 ceph-mon[27845]: pgmap v14045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:01 smithi096 ceph-mon[32753]: pgmap v14045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:03 smithi194 ceph-mon[33017]: pgmap v14046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25622.089274 secs 2024-04-26T18:58:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:03 smithi033 ceph-mon[27845]: pgmap v14046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25622.089274 secs 2024-04-26T18:58:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:03 smithi096 ceph-mon[32753]: pgmap v14046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25622.089274 secs 2024-04-26T18:58:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25624.978363 secs 2024-04-26T18:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25624.978363 secs 2024-04-26T18:58:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25624.978363 secs 2024-04-26T18:58:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:05 smithi194 ceph-mon[33017]: pgmap v14047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:58:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:05 smithi033 ceph-mon[27845]: pgmap v14047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:58:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:05 smithi096 ceph-mon[32753]: pgmap v14047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:58:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:06.600+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:06.600+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:06.600+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:58:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:07 smithi194 ceph-mon[33017]: pgmap v14048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:58:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:07 smithi033 ceph-mon[27845]: pgmap v14048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:58:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:07 smithi096 ceph-mon[32753]: pgmap v14048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:58:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25627.089335 secs 2024-04-26T18:58:08.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25627.089335 secs 2024-04-26T18:58:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25627.089335 secs 2024-04-26T18:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:09 smithi194 ceph-mon[33017]: pgmap v14049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25629.978499 secs 2024-04-26T18:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:09 smithi033 ceph-mon[27845]: pgmap v14049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25629.978499 secs 2024-04-26T18:58:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:58:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:09 smithi096 ceph-mon[32753]: pgmap v14049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25629.978499 secs 2024-04-26T18:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:12 smithi194 ceph-mon[33017]: pgmap v14050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:12 smithi033 ceph-mon[27845]: pgmap v14050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:12 smithi096 ceph-mon[32753]: pgmap v14050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25632.089398 secs 2024-04-26T18:58:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25632.089398 secs 2024-04-26T18:58:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25632.089398 secs 2024-04-26T18:58:14.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:14 smithi194 ceph-mon[33017]: pgmap v14051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:14.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:14.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:14.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25634.978631 secs 2024-04-26T18:58:14.533 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:14 smithi096 ceph-mon[32753]: pgmap v14051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:14.533 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:14.533 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:14.533 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25634.978631 secs 2024-04-26T18:58:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:14 smithi033 ceph-mon[27845]: pgmap v14051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25634.978631 secs 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: pgmap v14052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: pgmap v14052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: pgmap v14052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:58:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:16.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:17 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:58:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:17 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:58:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:17 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:58:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:17 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:58:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:17 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:58:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:17 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:58:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:17 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:58:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:17 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:58:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:17 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T18:58:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:17 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T18:58:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:17 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T18:58:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:17 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T18:58:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:18 smithi033 ceph-mon[27845]: pgmap v14053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:58:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25637.089480 secs 2024-04-26T18:58:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:18 smithi096 ceph-mon[32753]: pgmap v14053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:58:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25637.089480 secs 2024-04-26T18:58:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:18 smithi194 ceph-mon[33017]: pgmap v14053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T18:58:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25637.089480 secs 2024-04-26T18:58:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25639.978768 secs 2024-04-26T18:58:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25639.978768 secs 2024-04-26T18:58:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25639.978768 secs 2024-04-26T18:58:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:20 smithi033 ceph-mon[27845]: pgmap v14054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:20 smithi194 ceph-mon[33017]: pgmap v14054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:20 smithi096 ceph-mon[32753]: pgmap v14054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:21.601+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:21.601+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:58:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:21.601+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:58:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:58:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:58:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:58:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:58:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:58:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:58:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:22 smithi194 ceph-mon[33017]: pgmap v14055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:22 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:58:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:22 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:58:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:22 smithi194 ceph-mon[33017]: pgmap v14056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:58:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:22 smithi194 ceph-mon[33017]: pgmap v14057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:22 smithi033 ceph-mon[27845]: pgmap v14055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:22 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:58:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:22 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:58:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:22 smithi033 ceph-mon[27845]: pgmap v14056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:58:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:22 smithi033 ceph-mon[27845]: pgmap v14057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:22 smithi096 ceph-mon[32753]: pgmap v14055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:22 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T18:58:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:22 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T18:58:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:22 smithi096 ceph-mon[32753]: pgmap v14056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:58:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:22 smithi096 ceph-mon[32753]: pgmap v14057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25642.089564 secs 2024-04-26T18:58:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25642.089564 secs 2024-04-26T18:58:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25642.089564 secs 2024-04-26T18:58:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:24 smithi194 ceph-mon[33017]: pgmap v14058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:58:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25644.978909 secs 2024-04-26T18:58:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:24 smithi033 ceph-mon[27845]: pgmap v14058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:58:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25644.978909 secs 2024-04-26T18:58:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:24 smithi096 ceph-mon[32753]: pgmap v14058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T18:58:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25644.978909 secs 2024-04-26T18:58:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:25 smithi194 ceph-mon[33017]: pgmap v14059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:25.874 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:25 smithi096 ceph-mon[32753]: pgmap v14059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:25 smithi033 ceph-mon[27845]: pgmap v14059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:27 smithi194 ceph-mon[33017]: pgmap v14060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:27 smithi033 ceph-mon[27845]: pgmap v14060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:27 smithi096 ceph-mon[32753]: pgmap v14060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25647.089647 secs 2024-04-26T18:58:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25647.089647 secs 2024-04-26T18:58:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25647.089647 secs 2024-04-26T18:58:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:29 smithi033 ceph-mon[27845]: pgmap v14061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:58:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25649.979051 secs 2024-04-26T18:58:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:29 smithi096 ceph-mon[32753]: pgmap v14061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:58:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25649.979051 secs 2024-04-26T18:58:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:29 smithi194 ceph-mon[33017]: pgmap v14061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T18:58:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25649.979051 secs 2024-04-26T18:58:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:31 smithi033 ceph-mon[27845]: pgmap v14062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:58:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:31 smithi096 ceph-mon[32753]: pgmap v14062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:58:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:31 smithi194 ceph-mon[33017]: pgmap v14062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:58:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:33 smithi033 ceph-mon[27845]: pgmap v14063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25652.089730 secs 2024-04-26T18:58:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:33 smithi096 ceph-mon[32753]: pgmap v14063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25652.089730 secs 2024-04-26T18:58:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:33 smithi194 ceph-mon[33017]: pgmap v14063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25652.089730 secs 2024-04-26T18:58:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25654.979190 secs 2024-04-26T18:58:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25654.979190 secs 2024-04-26T18:58:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25654.979190 secs 2024-04-26T18:58:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:35 smithi033 ceph-mon[27845]: pgmap v14064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:35 smithi096 ceph-mon[32753]: pgmap v14064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:35 smithi194 ceph-mon[33017]: pgmap v14064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:36.609+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:36.609+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:36.609+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:58:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:37 smithi033 ceph-mon[27845]: pgmap v14065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:37 smithi096 ceph-mon[32753]: pgmap v14065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:37 smithi194 ceph-mon[33017]: pgmap v14065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25657.089814 secs 2024-04-26T18:58:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25657.089814 secs 2024-04-26T18:58:39.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25657.089814 secs 2024-04-26T18:58:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:39 smithi033 ceph-mon[27845]: pgmap v14066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25659.979338 secs 2024-04-26T18:58:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:39 smithi096 ceph-mon[32753]: pgmap v14066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25659.979338 secs 2024-04-26T18:58:40.021 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:39 smithi194 ceph-mon[33017]: pgmap v14066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:40.021 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25659.979338 secs 2024-04-26T18:58:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:41.160+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T18:58:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:41.160+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:58:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:41.160+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 203 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T18:58:41.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:41.160+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 203 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:58:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:41 smithi033 ceph-mon[27845]: pgmap v14067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:41 smithi096 ceph-mon[32753]: pgmap v14067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:41 smithi194 ceph-mon[33017]: pgmap v14067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:43 smithi033 ceph-mon[27845]: pgmap v14068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25662.089896 secs 2024-04-26T18:58:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:43 smithi096 ceph-mon[32753]: pgmap v14068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25662.089896 secs 2024-04-26T18:58:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:43 smithi194 ceph-mon[33017]: pgmap v14068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25662.089896 secs 2024-04-26T18:58:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25664.979449 secs 2024-04-26T18:58:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25664.979449 secs 2024-04-26T18:58:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25664.979449 secs 2024-04-26T18:58:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:45 smithi033 ceph-mon[27845]: pgmap v14069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:45 smithi096 ceph-mon[32753]: pgmap v14069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:45 smithi194 ceph-mon[33017]: pgmap v14069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:47 smithi033 ceph-mon[27845]: pgmap v14070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:47 smithi096 ceph-mon[32753]: pgmap v14070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:47 smithi194 ceph-mon[33017]: pgmap v14070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25667.089983 secs 2024-04-26T18:58:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25667.089983 secs 2024-04-26T18:58:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25667.089983 secs 2024-04-26T18:58:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25669.979525 secs 2024-04-26T18:58:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:49 smithi194 ceph-mon[33017]: pgmap v14071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25669.979525 secs 2024-04-26T18:58:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:49 smithi033 ceph-mon[27845]: pgmap v14071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25669.979525 secs 2024-04-26T18:58:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:49 smithi096 ceph-mon[32753]: pgmap v14071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:51.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:51.623+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:51.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:51.623+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:58:51.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:58:51.623+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:58:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:51 smithi194 ceph-mon[33017]: pgmap v14072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:51 smithi033 ceph-mon[27845]: pgmap v14072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:51 smithi096 ceph-mon[32753]: pgmap v14072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:53 smithi194 ceph-mon[33017]: pgmap v14073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25672.090062 secs 2024-04-26T18:58:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:53 smithi033 ceph-mon[27845]: pgmap v14073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25672.090062 secs 2024-04-26T18:58:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:53 smithi096 ceph-mon[32753]: pgmap v14073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25672.090062 secs 2024-04-26T18:58:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25674.979602 secs 2024-04-26T18:58:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25674.979602 secs 2024-04-26T18:58:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25674.979602 secs 2024-04-26T18:58:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:55 smithi096 ceph-mon[32753]: pgmap v14074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:55 smithi194 ceph-mon[33017]: pgmap v14074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:55 smithi033 ceph-mon[27845]: pgmap v14074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:58:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:57 smithi194 ceph-mon[33017]: pgmap v14075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:57 smithi033 ceph-mon[27845]: pgmap v14075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:57 smithi096 ceph-mon[32753]: pgmap v14075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25677.090152 secs 2024-04-26T18:58:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25677.090152 secs 2024-04-26T18:58:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25677.090152 secs 2024-04-26T18:59:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25679.979719 secs 2024-04-26T18:59:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:58:59 smithi194 ceph-mon[33017]: pgmap v14076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25679.979719 secs 2024-04-26T18:59:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:58:59 smithi033 ceph-mon[27845]: pgmap v14076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25679.979719 secs 2024-04-26T18:59:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:58:59 smithi096 ceph-mon[32753]: pgmap v14076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:01 smithi194 ceph-mon[33017]: pgmap v14077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:01 smithi033 ceph-mon[27845]: pgmap v14077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:01 smithi096 ceph-mon[32753]: pgmap v14077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:03 smithi194 ceph-mon[33017]: pgmap v14078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25682.090238 secs 2024-04-26T18:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:03 smithi033 ceph-mon[27845]: pgmap v14078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25682.090238 secs 2024-04-26T18:59:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:03 smithi096 ceph-mon[32753]: pgmap v14078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25682.090238 secs 2024-04-26T18:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25684.979864 secs 2024-04-26T18:59:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25684.979864 secs 2024-04-26T18:59:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25684.979864 secs 2024-04-26T18:59:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:05 smithi194 ceph-mon[33017]: pgmap v14079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:05 smithi033 ceph-mon[27845]: pgmap v14079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:06.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:05 smithi096 ceph-mon[32753]: pgmap v14079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:06.640+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:06.640+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:06.640+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:59:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:07 smithi194 ceph-mon[33017]: pgmap v14080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:07 smithi033 ceph-mon[27845]: pgmap v14080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:07 smithi096 ceph-mon[32753]: pgmap v14080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25687.090324 secs 2024-04-26T18:59:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25687.090324 secs 2024-04-26T18:59:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25687.090324 secs 2024-04-26T18:59:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25689.979987 secs 2024-04-26T18:59:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:09 smithi194 ceph-mon[33017]: pgmap v14081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25689.979987 secs 2024-04-26T18:59:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:09 smithi033 ceph-mon[27845]: pgmap v14081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25689.979987 secs 2024-04-26T18:59:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:09 smithi096 ceph-mon[32753]: pgmap v14081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:11 smithi194 ceph-mon[33017]: pgmap v14082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:11 smithi033 ceph-mon[27845]: pgmap v14082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:12.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:11 smithi096 ceph-mon[32753]: pgmap v14082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:13 smithi194 ceph-mon[33017]: pgmap v14083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25692.090411 secs 2024-04-26T18:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:13 smithi033 ceph-mon[27845]: pgmap v14083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25692.090411 secs 2024-04-26T18:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:13 smithi096 ceph-mon[32753]: pgmap v14083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25692.090411 secs 2024-04-26T18:59:15.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25694.980051 secs 2024-04-26T18:59:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25694.980051 secs 2024-04-26T18:59:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25694.980051 secs 2024-04-26T18:59:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:15 smithi194 ceph-mon[33017]: pgmap v14084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:15 smithi033 ceph-mon[27845]: pgmap v14084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:15 smithi096 ceph-mon[32753]: pgmap v14084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:17 smithi194 ceph-mon[33017]: pgmap v14085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:17 smithi033 ceph-mon[27845]: pgmap v14085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:17 smithi096 ceph-mon[32753]: pgmap v14085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:19.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25697.090498 secs 2024-04-26T18:59:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25697.090498 secs 2024-04-26T18:59:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25697.090498 secs 2024-04-26T18:59:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25699.980173 secs 2024-04-26T18:59:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:19 smithi194 ceph-mon[33017]: pgmap v14086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25699.980173 secs 2024-04-26T18:59:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:19 smithi033 ceph-mon[27845]: pgmap v14086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25699.980173 secs 2024-04-26T18:59:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:19 smithi096 ceph-mon[32753]: pgmap v14086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:59:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:59:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:59:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:21.642+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:21.642+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:59:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:21.642+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:21 smithi194 ceph-mon[33017]: pgmap v14087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:21 smithi033 ceph-mon[27845]: pgmap v14087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:21 smithi096 ceph-mon[32753]: pgmap v14087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:59:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:59:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:22 smithi194 ceph-mon[33017]: pgmap v14088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:22 smithi194 ceph-mon[33017]: pgmap v14089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:59:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:59:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:59:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:22 smithi033 ceph-mon[27845]: pgmap v14088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:22 smithi033 ceph-mon[27845]: pgmap v14089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:59:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:59:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:59:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:22 smithi096 ceph-mon[32753]: pgmap v14088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:22 smithi096 ceph-mon[32753]: pgmap v14089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T18:59:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25702.090559 secs 2024-04-26T18:59:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25702.090559 secs 2024-04-26T18:59:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25702.090559 secs 2024-04-26T18:59:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25704.980269 secs 2024-04-26T18:59:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:24 smithi033 ceph-mon[27845]: pgmap v14090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:59:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25704.980269 secs 2024-04-26T18:59:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:24 smithi096 ceph-mon[32753]: pgmap v14090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25704.980269 secs 2024-04-26T18:59:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:24 smithi194 ceph-mon[33017]: pgmap v14090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T18:59:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:26.168+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T18:59:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:26.168+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T18:59:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:26.168+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 204 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T18:59:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:26.169+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 204 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T18:59:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:27 smithi194 ceph-mon[33017]: pgmap v14091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:27 smithi033 ceph-mon[27845]: pgmap v14091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:27 smithi096 ceph-mon[32753]: pgmap v14091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25707.090643 secs 2024-04-26T18:59:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25707.090643 secs 2024-04-26T18:59:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25707.090643 secs 2024-04-26T18:59:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:29 smithi194 ceph-mon[33017]: pgmap v14092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25709.980336 secs 2024-04-26T18:59:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:29 smithi033 ceph-mon[27845]: pgmap v14092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25709.980336 secs 2024-04-26T18:59:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:29 smithi096 ceph-mon[32753]: pgmap v14092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25709.980336 secs 2024-04-26T18:59:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:31 smithi194 ceph-mon[33017]: pgmap v14093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:31 smithi033 ceph-mon[27845]: pgmap v14093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:31 smithi096 ceph-mon[32753]: pgmap v14093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T18:59:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:33 smithi194 ceph-mon[33017]: pgmap v14094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:59:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25712.090700 secs 2024-04-26T18:59:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:33 smithi033 ceph-mon[27845]: pgmap v14094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:59:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25712.090700 secs 2024-04-26T18:59:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:33 smithi096 ceph-mon[32753]: pgmap v14094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T18:59:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25712.090700 secs 2024-04-26T18:59:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25714.980447 secs 2024-04-26T18:59:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25714.980447 secs 2024-04-26T18:59:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25714.980447 secs 2024-04-26T18:59:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:35 smithi194 ceph-mon[33017]: pgmap v14095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:35 smithi033 ceph-mon[27845]: pgmap v14095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:35 smithi096 ceph-mon[32753]: pgmap v14095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:36.653+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:36.653+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:36.653+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:59:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:37 smithi194 ceph-mon[33017]: pgmap v14096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:37 smithi033 ceph-mon[27845]: pgmap v14096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:37 smithi096 ceph-mon[32753]: pgmap v14096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25717.090787 secs 2024-04-26T18:59:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25717.090787 secs 2024-04-26T18:59:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25717.090787 secs 2024-04-26T18:59:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:39 smithi194 ceph-mon[33017]: pgmap v14097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25719.980520 secs 2024-04-26T18:59:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:39 smithi033 ceph-mon[27845]: pgmap v14097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25719.980520 secs 2024-04-26T18:59:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:39 smithi096 ceph-mon[32753]: pgmap v14097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25719.980520 secs 2024-04-26T18:59:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:41 smithi194 ceph-mon[33017]: pgmap v14098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:41 smithi033 ceph-mon[27845]: pgmap v14098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:41 smithi096 ceph-mon[32753]: pgmap v14098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:43 smithi194 ceph-mon[33017]: pgmap v14099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25722.090870 secs 2024-04-26T18:59:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:43 smithi033 ceph-mon[27845]: pgmap v14099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25722.090870 secs 2024-04-26T18:59:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:43 smithi096 ceph-mon[32753]: pgmap v14099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25722.090870 secs 2024-04-26T18:59:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25724.980653 secs 2024-04-26T18:59:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25724.980653 secs 2024-04-26T18:59:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25724.980653 secs 2024-04-26T18:59:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:45 smithi194 ceph-mon[33017]: pgmap v14100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:45 smithi033 ceph-mon[27845]: pgmap v14100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:45 smithi096 ceph-mon[32753]: pgmap v14100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:47 smithi194 ceph-mon[33017]: pgmap v14101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:47 smithi033 ceph-mon[27845]: pgmap v14101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:47 smithi096 ceph-mon[32753]: pgmap v14101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25727.090929 secs 2024-04-26T18:59:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25727.090929 secs 2024-04-26T18:59:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25727.090929 secs 2024-04-26T18:59:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:49 smithi033 ceph-mon[27845]: pgmap v14102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25729.980790 secs 2024-04-26T18:59:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:49 smithi096 ceph-mon[32753]: pgmap v14102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25729.980790 secs 2024-04-26T18:59:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:49 smithi194 ceph-mon[33017]: pgmap v14102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25729.980790 secs 2024-04-26T18:59:51.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:51.657+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:51.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:51.657+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T18:59:51.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T18:59:51.657+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:59:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:51 smithi033 ceph-mon[27845]: pgmap v14103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:51 smithi096 ceph-mon[32753]: pgmap v14103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:51 smithi194 ceph-mon[33017]: pgmap v14103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:53 smithi033 ceph-mon[27845]: pgmap v14104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25732.091024 secs 2024-04-26T18:59:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:53 smithi096 ceph-mon[32753]: pgmap v14104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25732.091024 secs 2024-04-26T18:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:53 smithi194 ceph-mon[33017]: pgmap v14104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25732.091024 secs 2024-04-26T18:59:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25734.980905 secs 2024-04-26T18:59:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25734.980905 secs 2024-04-26T18:59:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25734.980905 secs 2024-04-26T18:59:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:55 smithi033 ceph-mon[27845]: pgmap v14105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:55 smithi096 ceph-mon[32753]: pgmap v14105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:55 smithi194 ceph-mon[33017]: pgmap v14105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:57.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:57 smithi033 ceph-mon[27845]: pgmap v14106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:57 smithi096 ceph-mon[32753]: pgmap v14106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:57 smithi194 ceph-mon[33017]: pgmap v14106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T18:59:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25737.091103 secs 2024-04-26T18:59:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25737.091103 secs 2024-04-26T18:59:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25737.091103 secs 2024-04-26T18:59:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:59 smithi033 ceph-mon[27845]: pgmap v14107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 18:59:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25739.981025 secs 2024-04-26T18:59:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:59 smithi096 ceph-mon[32753]: pgmap v14107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 18:59:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25739.981025 secs 2024-04-26T18:59:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:59 smithi194 ceph-mon[33017]: pgmap v14107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T18:59:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 18:59:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25739.981025 secs 2024-04-26T19:00:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:00:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:00:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:01 smithi033 ceph-mon[27845]: pgmap v14108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:01 smithi096 ceph-mon[32753]: pgmap v14108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:01 smithi194 ceph-mon[33017]: pgmap v14108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:03 smithi096 ceph-mon[32753]: pgmap v14109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25742.091183 secs 2024-04-26T19:00:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:03 smithi194 ceph-mon[33017]: pgmap v14109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25742.091183 secs 2024-04-26T19:00:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:03 smithi033 ceph-mon[27845]: pgmap v14109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25742.091183 secs 2024-04-26T19:00:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25744.981098 secs 2024-04-26T19:00:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25744.981098 secs 2024-04-26T19:00:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25744.981098 secs 2024-04-26T19:00:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:05 smithi194 ceph-mon[33017]: pgmap v14110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:05 smithi033 ceph-mon[27845]: pgmap v14110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:05 smithi096 ceph-mon[32753]: pgmap v14110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:06.664+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:06.664+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:06.664+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:00:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:07 smithi194 ceph-mon[33017]: pgmap v14111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:07 smithi033 ceph-mon[27845]: pgmap v14111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:07 smithi096 ceph-mon[32753]: pgmap v14111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25747.091268 secs 2024-04-26T19:00:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25747.091268 secs 2024-04-26T19:00:08.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25747.091268 secs 2024-04-26T19:00:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:09 smithi194 ceph-mon[33017]: pgmap v14112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:09.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25749.981226 secs 2024-04-26T19:00:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:09 smithi033 ceph-mon[27845]: pgmap v14112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25749.981226 secs 2024-04-26T19:00:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:09 smithi096 ceph-mon[32753]: pgmap v14112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25749.981226 secs 2024-04-26T19:00:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:11 smithi194 ceph-mon[33017]: pgmap v14113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:11 smithi033 ceph-mon[27845]: pgmap v14113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:11 smithi096 ceph-mon[32753]: pgmap v14113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:13 smithi194 ceph-mon[33017]: pgmap v14114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25752.091351 secs 2024-04-26T19:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:13 smithi033 ceph-mon[27845]: pgmap v14114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25752.091351 secs 2024-04-26T19:00:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:13 smithi096 ceph-mon[32753]: pgmap v14114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25752.091351 secs 2024-04-26T19:00:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25754.981332 secs 2024-04-26T19:00:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25754.981332 secs 2024-04-26T19:00:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25754.981332 secs 2024-04-26T19:00:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:15 smithi194 ceph-mon[33017]: pgmap v14115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:15 smithi033 ceph-mon[27845]: pgmap v14115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:15 smithi096 ceph-mon[32753]: pgmap v14115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:17 smithi194 ceph-mon[33017]: pgmap v14116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:17 smithi033 ceph-mon[27845]: pgmap v14116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:17 smithi096 ceph-mon[32753]: pgmap v14116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25757.091413 secs 2024-04-26T19:00:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25757.091413 secs 2024-04-26T19:00:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25757.091413 secs 2024-04-26T19:00:19.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:19 smithi033 ceph-mon[27845]: pgmap v14117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:19.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25759.981455 secs 2024-04-26T19:00:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:19 smithi194 ceph-mon[33017]: pgmap v14117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25759.981455 secs 2024-04-26T19:00:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:19 smithi096 ceph-mon[32753]: pgmap v14117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25759.981455 secs 2024-04-26T19:00:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:21.673+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:21.673+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:00:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:21.673+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:21 smithi194 ceph-mon[33017]: pgmap v14118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:00:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:21 smithi033 ceph-mon[27845]: pgmap v14118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:00:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:21 smithi096 ceph-mon[32753]: pgmap v14118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:00:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:23 smithi194 ceph-mon[33017]: pgmap v14119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:00:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:00:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25762.091479 secs 2024-04-26T19:00:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:23 smithi033 ceph-mon[27845]: pgmap v14119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:00:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:00:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25762.091479 secs 2024-04-26T19:00:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:23 smithi096 ceph-mon[32753]: pgmap v14119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:00:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:00:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25762.091479 secs 2024-04-26T19:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25764.981562 secs 2024-04-26T19:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:00:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25764.981562 secs 2024-04-26T19:00:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:00:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:00:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:00:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25764.981562 secs 2024-04-26T19:00:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:00:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:00:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:00:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:25 smithi194 ceph-mon[33017]: pgmap v14120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:00:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:25 smithi194 ceph-mon[33017]: pgmap v14121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:25 smithi033 ceph-mon[27845]: pgmap v14120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:00:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:25 smithi033 ceph-mon[27845]: pgmap v14121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:25 smithi096 ceph-mon[32753]: pgmap v14120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:00:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:25 smithi096 ceph-mon[32753]: pgmap v14121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:27 smithi194 ceph-mon[33017]: pgmap v14122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:27 smithi033 ceph-mon[27845]: pgmap v14122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:27 smithi096 ceph-mon[32753]: pgmap v14122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25767.091563 secs 2024-04-26T19:00:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25767.091563 secs 2024-04-26T19:00:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25767.091563 secs 2024-04-26T19:00:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:29 smithi194 ceph-mon[33017]: pgmap v14123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25769.981698 secs 2024-04-26T19:00:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:29 smithi033 ceph-mon[27845]: pgmap v14123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25769.981698 secs 2024-04-26T19:00:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:29 smithi096 ceph-mon[32753]: pgmap v14123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:00:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25769.981698 secs 2024-04-26T19:00:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:30 smithi194 ceph-mon[33017]: pgmap v14124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:00:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:30 smithi033 ceph-mon[27845]: pgmap v14124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:00:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:30 smithi096 ceph-mon[32753]: pgmap v14124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:00:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:33 smithi194 ceph-mon[33017]: pgmap v14125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:00:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:33 smithi033 ceph-mon[27845]: pgmap v14125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:00:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:33 smithi096 ceph-mon[32753]: pgmap v14125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:00:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25772.091644 secs 2024-04-26T19:00:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25774.981834 secs 2024-04-26T19:00:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25772.091644 secs 2024-04-26T19:00:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25774.981834 secs 2024-04-26T19:00:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25772.091644 secs 2024-04-26T19:00:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25774.981834 secs 2024-04-26T19:00:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:35 smithi194 ceph-mon[33017]: pgmap v14126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:00:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:35 smithi033 ceph-mon[27845]: pgmap v14126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:00:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:35 smithi096 ceph-mon[32753]: pgmap v14126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:00:36.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:36.688+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:36.689+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:36.689+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:00:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:37 smithi194 ceph-mon[33017]: pgmap v14127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:37 smithi033 ceph-mon[27845]: pgmap v14127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:37 smithi096 ceph-mon[32753]: pgmap v14127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25777.091728 secs 2024-04-26T19:00:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:39 smithi194 ceph-mon[33017]: pgmap v14128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25779.981950 secs 2024-04-26T19:00:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25777.091728 secs 2024-04-26T19:00:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:39 smithi033 ceph-mon[27845]: pgmap v14128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25779.981950 secs 2024-04-26T19:00:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25777.091728 secs 2024-04-26T19:00:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:39 smithi096 ceph-mon[32753]: pgmap v14128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25779.981950 secs 2024-04-26T19:00:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:41 smithi194 ceph-mon[33017]: pgmap v14129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:41 smithi033 ceph-mon[27845]: pgmap v14129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:41 smithi096 ceph-mon[32753]: pgmap v14129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:43 smithi194 ceph-mon[33017]: pgmap v14130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:43 smithi033 ceph-mon[27845]: pgmap v14130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:43 smithi096 ceph-mon[32753]: pgmap v14130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25782.091809 secs 2024-04-26T19:00:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25784.982051 secs 2024-04-26T19:00:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25782.091809 secs 2024-04-26T19:00:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25784.982051 secs 2024-04-26T19:00:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25782.091809 secs 2024-04-26T19:00:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25784.982051 secs 2024-04-26T19:00:45.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:45 smithi096 ceph-mon[32753]: pgmap v14131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:45 smithi194 ceph-mon[33017]: pgmap v14131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:45 smithi033 ceph-mon[27845]: pgmap v14131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:47 smithi194 ceph-mon[33017]: pgmap v14132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:47 smithi033 ceph-mon[27845]: pgmap v14132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:47 smithi096 ceph-mon[32753]: pgmap v14132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25787.091896 secs 2024-04-26T19:00:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:49 smithi194 ceph-mon[33017]: pgmap v14133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25789.982174 secs 2024-04-26T19:00:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25787.091896 secs 2024-04-26T19:00:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:49 smithi033 ceph-mon[27845]: pgmap v14133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25789.982174 secs 2024-04-26T19:00:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25787.091896 secs 2024-04-26T19:00:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:49 smithi096 ceph-mon[32753]: pgmap v14133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25789.982174 secs 2024-04-26T19:00:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:51 smithi194 ceph-mon[33017]: pgmap v14134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:51 smithi033 ceph-mon[27845]: pgmap v14134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:51 smithi096 ceph-mon[32753]: pgmap v14134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:51.701+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:51.701+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:00:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:00:51.701+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:00:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:53 smithi194 ceph-mon[33017]: pgmap v14135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:53 smithi033 ceph-mon[27845]: pgmap v14135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:53 smithi096 ceph-mon[32753]: pgmap v14135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25792.091996 secs 2024-04-26T19:00:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25794.982299 secs 2024-04-26T19:00:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25792.091996 secs 2024-04-26T19:00:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25794.982299 secs 2024-04-26T19:00:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25792.091996 secs 2024-04-26T19:00:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25794.982299 secs 2024-04-26T19:00:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:55 smithi194 ceph-mon[33017]: pgmap v14136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:55 smithi033 ceph-mon[27845]: pgmap v14136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:55 smithi096 ceph-mon[32753]: pgmap v14136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:57 smithi194 ceph-mon[33017]: pgmap v14137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:57 smithi033 ceph-mon[27845]: pgmap v14137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:57 smithi096 ceph-mon[32753]: pgmap v14137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:00:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25797.092058 secs 2024-04-26T19:00:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:59 smithi033 ceph-mon[27845]: pgmap v14138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25799.982437 secs 2024-04-26T19:00:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25797.092058 secs 2024-04-26T19:00:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:59 smithi096 ceph-mon[32753]: pgmap v14138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25799.982437 secs 2024-04-26T19:00:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25797.092058 secs 2024-04-26T19:00:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:59 smithi194 ceph-mon[33017]: pgmap v14138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:00:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25799.982437 secs 2024-04-26T19:01:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:01 smithi033 ceph-mon[27845]: pgmap v14139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:01.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:01 smithi096 ceph-mon[32753]: pgmap v14139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:01 smithi194 ceph-mon[33017]: pgmap v14139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:03 smithi033 ceph-mon[27845]: pgmap v14140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:03 smithi096 ceph-mon[32753]: pgmap v14140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:03 smithi194 ceph-mon[33017]: pgmap v14140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25802.092120 secs 2024-04-26T19:01:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25804.982559 secs 2024-04-26T19:01:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25802.092120 secs 2024-04-26T19:01:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25804.982559 secs 2024-04-26T19:01:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25802.092120 secs 2024-04-26T19:01:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25804.982559 secs 2024-04-26T19:01:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:05 smithi033 ceph-mon[27845]: pgmap v14141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:05 smithi096 ceph-mon[32753]: pgmap v14141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:05 smithi194 ceph-mon[33017]: pgmap v14141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:06.716+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:06.716+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:06.716+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:01:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:07 smithi033 ceph-mon[27845]: pgmap v14142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:07 smithi096 ceph-mon[32753]: pgmap v14142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:07 smithi194 ceph-mon[33017]: pgmap v14142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:09.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25807.092195 secs 2024-04-26T19:01:09.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:09 smithi194 ceph-mon[33017]: pgmap v14143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:09.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25809.982696 secs 2024-04-26T19:01:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25807.092195 secs 2024-04-26T19:01:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:09 smithi033 ceph-mon[27845]: pgmap v14143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25809.982696 secs 2024-04-26T19:01:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25807.092195 secs 2024-04-26T19:01:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:09 smithi096 ceph-mon[32753]: pgmap v14143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25809.982696 secs 2024-04-26T19:01:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:11 smithi033 ceph-mon[27845]: pgmap v14144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:11 smithi096 ceph-mon[32753]: pgmap v14144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:11 smithi194 ceph-mon[33017]: pgmap v14144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:13 smithi033 ceph-mon[27845]: pgmap v14145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:13 smithi096 ceph-mon[32753]: pgmap v14145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:13 smithi194 ceph-mon[33017]: pgmap v14145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25812.092278 secs 2024-04-26T19:01:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25814.982814 secs 2024-04-26T19:01:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25812.092278 secs 2024-04-26T19:01:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25814.982814 secs 2024-04-26T19:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25812.092278 secs 2024-04-26T19:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25814.982814 secs 2024-04-26T19:01:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:15 smithi033 ceph-mon[27845]: pgmap v14146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:15 smithi096 ceph-mon[32753]: pgmap v14146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:15 smithi194 ceph-mon[33017]: pgmap v14146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:17 smithi033 ceph-mon[27845]: pgmap v14147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:17 smithi096 ceph-mon[32753]: pgmap v14147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:17 smithi194 ceph-mon[33017]: pgmap v14147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25817.092339 secs 2024-04-26T19:01:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:19 smithi033 ceph-mon[27845]: pgmap v14148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25819.982944 secs 2024-04-26T19:01:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25817.092339 secs 2024-04-26T19:01:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:19 smithi096 ceph-mon[32753]: pgmap v14148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25819.982944 secs 2024-04-26T19:01:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25817.092339 secs 2024-04-26T19:01:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:19 smithi194 ceph-mon[33017]: pgmap v14148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25819.982944 secs 2024-04-26T19:01:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:21 smithi096 ceph-mon[32753]: pgmap v14149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:21 smithi194 ceph-mon[33017]: pgmap v14149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:21 smithi033 ceph-mon[27845]: pgmap v14149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:21.720+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:21.720+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:21.720+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:01:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:22 smithi194 ceph-mon[33017]: pgmap v14150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:22 smithi033 ceph-mon[27845]: pgmap v14150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:22 smithi096 ceph-mon[32753]: pgmap v14150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25822.092423 secs 2024-04-26T19:01:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25822.092423 secs 2024-04-26T19:01:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25822.092423 secs 2024-04-26T19:01:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25824.983090 secs 2024-04-26T19:01:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:24 smithi194 ceph-mon[33017]: pgmap v14151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25824.983090 secs 2024-04-26T19:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:24 smithi033 ceph-mon[27845]: pgmap v14151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:01:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25824.983090 secs 2024-04-26T19:01:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:24 smithi096 ceph-mon[32753]: pgmap v14151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:01:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:01:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:01:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:01:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:01:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:01:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:01:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:01:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:01:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:01:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:26 smithi194 ceph-mon[33017]: pgmap v14152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:01:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:26 smithi194 ceph-mon[33017]: pgmap v14153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:26 smithi033 ceph-mon[27845]: pgmap v14152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:01:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:26 smithi033 ceph-mon[27845]: pgmap v14153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:26 smithi096 ceph-mon[32753]: pgmap v14152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:01:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:26 smithi096 ceph-mon[32753]: pgmap v14153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:28 smithi194 ceph-mon[33017]: pgmap v14154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25827.092516 secs 2024-04-26T19:01:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:28 smithi033 ceph-mon[27845]: pgmap v14154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25827.092516 secs 2024-04-26T19:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:28 smithi096 ceph-mon[32753]: pgmap v14154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25827.092516 secs 2024-04-26T19:01:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25829.983198 secs 2024-04-26T19:01:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25829.983198 secs 2024-04-26T19:01:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25829.983198 secs 2024-04-26T19:01:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:30 smithi194 ceph-mon[33017]: pgmap v14155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:30 smithi033 ceph-mon[27845]: pgmap v14155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:30 smithi096 ceph-mon[32753]: pgmap v14155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:32 smithi194 ceph-mon[33017]: pgmap v14156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:32 smithi033 ceph-mon[27845]: pgmap v14156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:32 smithi096 ceph-mon[32753]: pgmap v14156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25832.092595 secs 2024-04-26T19:01:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25832.092595 secs 2024-04-26T19:01:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25832.092595 secs 2024-04-26T19:01:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25834.983320 secs 2024-04-26T19:01:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:34 smithi194 ceph-mon[33017]: pgmap v14157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25834.983320 secs 2024-04-26T19:01:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:34 smithi033 ceph-mon[27845]: pgmap v14157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25834.983320 secs 2024-04-26T19:01:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:34 smithi096 ceph-mon[32753]: pgmap v14157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:01:36.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:36.733+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:36.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:36.733+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:36.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:36.733+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:01:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:36 smithi194 ceph-mon[33017]: pgmap v14158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:01:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:36 smithi033 ceph-mon[27845]: pgmap v14158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:01:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:36 smithi096 ceph-mon[32753]: pgmap v14158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:01:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:38 smithi194 ceph-mon[33017]: pgmap v14159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25837.092680 secs 2024-04-26T19:01:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:38 smithi033 ceph-mon[27845]: pgmap v14159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25837.092680 secs 2024-04-26T19:01:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:38 smithi096 ceph-mon[32753]: pgmap v14159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25837.092680 secs 2024-04-26T19:01:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25839.983400 secs 2024-04-26T19:01:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25839.983400 secs 2024-04-26T19:01:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25839.983400 secs 2024-04-26T19:01:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:40 smithi194 ceph-mon[33017]: pgmap v14160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:40 smithi033 ceph-mon[27845]: pgmap v14160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:40 smithi096 ceph-mon[32753]: pgmap v14160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:42 smithi033 ceph-mon[27845]: pgmap v14161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:42 smithi096 ceph-mon[32753]: pgmap v14161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:42 smithi194 ceph-mon[33017]: pgmap v14161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25842.092743 secs 2024-04-26T19:01:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25842.092743 secs 2024-04-26T19:01:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25842.092743 secs 2024-04-26T19:01:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25844.983526 secs 2024-04-26T19:01:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:44 smithi033 ceph-mon[27845]: pgmap v14162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25844.983526 secs 2024-04-26T19:01:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:44 smithi096 ceph-mon[32753]: pgmap v14162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25844.983526 secs 2024-04-26T19:01:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:44 smithi194 ceph-mon[33017]: pgmap v14162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:46 smithi033 ceph-mon[27845]: pgmap v14163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:46 smithi096 ceph-mon[32753]: pgmap v14163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:46 smithi194 ceph-mon[33017]: pgmap v14163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:48 smithi033 ceph-mon[27845]: pgmap v14164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25847.092829 secs 2024-04-26T19:01:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:48 smithi096 ceph-mon[32753]: pgmap v14164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25847.092829 secs 2024-04-26T19:01:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:48 smithi194 ceph-mon[33017]: pgmap v14164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25847.092829 secs 2024-04-26T19:01:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25849.983650 secs 2024-04-26T19:01:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25849.983650 secs 2024-04-26T19:01:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25849.983650 secs 2024-04-26T19:01:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:50 smithi033 ceph-mon[27845]: pgmap v14165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:50 smithi096 ceph-mon[32753]: pgmap v14165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:50 smithi194 ceph-mon[33017]: pgmap v14165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:51.745+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:51.745+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:01:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:01:51.745+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:01:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:52 smithi033 ceph-mon[27845]: pgmap v14166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:52 smithi096 ceph-mon[32753]: pgmap v14166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:52 smithi194 ceph-mon[33017]: pgmap v14166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25852.092912 secs 2024-04-26T19:01:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25852.092912 secs 2024-04-26T19:01:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25852.092912 secs 2024-04-26T19:01:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25854.983735 secs 2024-04-26T19:01:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:54 smithi033 ceph-mon[27845]: pgmap v14167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25854.983735 secs 2024-04-26T19:01:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:54 smithi096 ceph-mon[32753]: pgmap v14167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25854.983735 secs 2024-04-26T19:01:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:54 smithi194 ceph-mon[33017]: pgmap v14167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:56 smithi033 ceph-mon[27845]: pgmap v14168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:56 smithi096 ceph-mon[32753]: pgmap v14168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:56 smithi194 ceph-mon[33017]: pgmap v14168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:01:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:58 smithi033 ceph-mon[27845]: pgmap v14169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25857.092999 secs 2024-04-26T19:01:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:58 smithi096 ceph-mon[32753]: pgmap v14169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25857.092999 secs 2024-04-26T19:01:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:58 smithi194 ceph-mon[33017]: pgmap v14169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:01:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25857.092999 secs 2024-04-26T19:02:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25859.983812 secs 2024-04-26T19:02:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25859.983812 secs 2024-04-26T19:02:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25859.983812 secs 2024-04-26T19:02:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:00 smithi096 ceph-mon[32753]: pgmap v14170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:00 smithi194 ceph-mon[33017]: pgmap v14170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:00 smithi033 ceph-mon[27845]: pgmap v14170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:02 smithi194 ceph-mon[33017]: pgmap v14171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:02 smithi033 ceph-mon[27845]: pgmap v14171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:02 smithi096 ceph-mon[32753]: pgmap v14171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25862.093087 secs 2024-04-26T19:02:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25862.093087 secs 2024-04-26T19:02:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25862.093087 secs 2024-04-26T19:02:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25864.983926 secs 2024-04-26T19:02:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:04 smithi194 ceph-mon[33017]: pgmap v14172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25864.983926 secs 2024-04-26T19:02:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:04 smithi033 ceph-mon[27845]: pgmap v14172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25864.983926 secs 2024-04-26T19:02:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:04 smithi096 ceph-mon[32753]: pgmap v14172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:06.761+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:06.761+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:06.761+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:02:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:06 smithi194 ceph-mon[33017]: pgmap v14173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:06 smithi033 ceph-mon[27845]: pgmap v14173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:06 smithi096 ceph-mon[32753]: pgmap v14173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:08 smithi194 ceph-mon[33017]: pgmap v14174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25867.093171 secs 2024-04-26T19:02:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25869.984079 secs 2024-04-26T19:02:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:08 smithi033 ceph-mon[27845]: pgmap v14174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25867.093171 secs 2024-04-26T19:02:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25869.984079 secs 2024-04-26T19:02:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:08 smithi096 ceph-mon[32753]: pgmap v14174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25867.093171 secs 2024-04-26T19:02:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25869.984079 secs 2024-04-26T19:02:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:10 smithi194 ceph-mon[33017]: pgmap v14175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:10 smithi033 ceph-mon[27845]: pgmap v14175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:11.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:10 smithi096 ceph-mon[32753]: pgmap v14175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:12 smithi194 ceph-mon[33017]: pgmap v14176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:12 smithi033 ceph-mon[27845]: pgmap v14176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:12 smithi096 ceph-mon[32753]: pgmap v14176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25872.093252 secs 2024-04-26T19:02:14.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25874.984222 secs 2024-04-26T19:02:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25872.093252 secs 2024-04-26T19:02:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25874.984222 secs 2024-04-26T19:02:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25872.093252 secs 2024-04-26T19:02:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25874.984222 secs 2024-04-26T19:02:15.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:14 smithi096 ceph-mon[32753]: pgmap v14177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:14 smithi194 ceph-mon[33017]: pgmap v14177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:14 smithi033 ceph-mon[27845]: pgmap v14177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:16 smithi194 ceph-mon[33017]: pgmap v14178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:16 smithi033 ceph-mon[27845]: pgmap v14178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:16 smithi096 ceph-mon[32753]: pgmap v14178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:18 smithi194 ceph-mon[33017]: pgmap v14179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25877.093335 secs 2024-04-26T19:02:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25879.984290 secs 2024-04-26T19:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:18 smithi033 ceph-mon[27845]: pgmap v14179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25877.093335 secs 2024-04-26T19:02:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25879.984290 secs 2024-04-26T19:02:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:18 smithi096 ceph-mon[32753]: pgmap v14179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25877.093335 secs 2024-04-26T19:02:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25879.984290 secs 2024-04-26T19:02:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:20 smithi194 ceph-mon[33017]: pgmap v14180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:20 smithi033 ceph-mon[27845]: pgmap v14180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:20 smithi096 ceph-mon[32753]: pgmap v14180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:21.777+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:21.777+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:21.777+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:02:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:23 smithi194 ceph-mon[33017]: pgmap v14181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:23 smithi033 ceph-mon[27845]: pgmap v14181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:23 smithi096 ceph-mon[32753]: pgmap v14181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25882.093414 secs 2024-04-26T19:02:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25884.984376 secs 2024-04-26T19:02:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25882.093414 secs 2024-04-26T19:02:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25884.984376 secs 2024-04-26T19:02:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25882.093414 secs 2024-04-26T19:02:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25884.984376 secs 2024-04-26T19:02:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:25 smithi194 ceph-mon[33017]: pgmap v14182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:25 smithi033 ceph-mon[27845]: pgmap v14182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:25 smithi096 ceph-mon[32753]: pgmap v14182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:26.198+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:02:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:26.198+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:02:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:26.198+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 205 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:02:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:26.198+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 205 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:02:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:02:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:02:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:02:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:27 smithi194 ceph-mon[33017]: pgmap v14183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:27 smithi033 ceph-mon[27845]: pgmap v14183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:27 smithi096 ceph-mon[32753]: pgmap v14183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:02:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:02:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:28 smithi194 ceph-mon[33017]: pgmap v14184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:28 smithi194 ceph-mon[33017]: pgmap v14185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:02:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:02:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:02:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:28 smithi033 ceph-mon[27845]: pgmap v14184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:28 smithi033 ceph-mon[27845]: pgmap v14185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:02:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:02:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:02:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:28 smithi096 ceph-mon[32753]: pgmap v14184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:28 smithi096 ceph-mon[32753]: pgmap v14185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:02:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25887.093476 secs 2024-04-26T19:02:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25889.984491 secs 2024-04-26T19:02:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25887.093476 secs 2024-04-26T19:02:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25889.984491 secs 2024-04-26T19:02:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25887.093476 secs 2024-04-26T19:02:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25889.984491 secs 2024-04-26T19:02:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:30 smithi194 ceph-mon[33017]: pgmap v14186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:02:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:30 smithi033 ceph-mon[27845]: pgmap v14186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:02:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:30 smithi096 ceph-mon[32753]: pgmap v14186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:02:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:32 smithi033 ceph-mon[27845]: pgmap v14187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:32 smithi096 ceph-mon[32753]: pgmap v14187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:32 smithi194 ceph-mon[33017]: pgmap v14187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25892.093555 secs 2024-04-26T19:02:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25892.093555 secs 2024-04-26T19:02:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25892.093555 secs 2024-04-26T19:02:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:34 smithi033 ceph-mon[27845]: pgmap v14188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25894.984597 secs 2024-04-26T19:02:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:34 smithi096 ceph-mon[32753]: pgmap v14188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25894.984597 secs 2024-04-26T19:02:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:34 smithi194 ceph-mon[33017]: pgmap v14188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25894.984597 secs 2024-04-26T19:02:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:36 smithi033 ceph-mon[27845]: pgmap v14189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:36 smithi096 ceph-mon[32753]: pgmap v14189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:36.783+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:36.783+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:02:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:36.783+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:36 smithi194 ceph-mon[33017]: pgmap v14189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:02:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:38 smithi033 ceph-mon[27845]: pgmap v14190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:02:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25897.093640 secs 2024-04-26T19:02:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:38 smithi096 ceph-mon[32753]: pgmap v14190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:02:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25897.093640 secs 2024-04-26T19:02:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:38 smithi194 ceph-mon[33017]: pgmap v14190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:02:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25897.093640 secs 2024-04-26T19:02:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25899.984666 secs 2024-04-26T19:02:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25899.984666 secs 2024-04-26T19:02:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25899.984666 secs 2024-04-26T19:02:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:40 smithi194 ceph-mon[33017]: pgmap v14191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:40 smithi033 ceph-mon[27845]: pgmap v14191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:40 smithi096 ceph-mon[32753]: pgmap v14191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:42 smithi194 ceph-mon[33017]: pgmap v14192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:42 smithi033 ceph-mon[27845]: pgmap v14192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:42 smithi096 ceph-mon[32753]: pgmap v14192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25902.093722 secs 2024-04-26T19:02:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25902.093722 secs 2024-04-26T19:02:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25902.093722 secs 2024-04-26T19:02:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:44 smithi033 ceph-mon[27845]: pgmap v14193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25904.984778 secs 2024-04-26T19:02:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:44 smithi194 ceph-mon[33017]: pgmap v14193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25904.984778 secs 2024-04-26T19:02:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:44 smithi096 ceph-mon[32753]: pgmap v14193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25904.984778 secs 2024-04-26T19:02:46.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:46 smithi096 ceph-mon[32753]: pgmap v14194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:46 smithi194 ceph-mon[33017]: pgmap v14194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:46 smithi033 ceph-mon[27845]: pgmap v14194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:48 smithi194 ceph-mon[33017]: pgmap v14195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25907.093808 secs 2024-04-26T19:02:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:48 smithi033 ceph-mon[27845]: pgmap v14195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25907.093808 secs 2024-04-26T19:02:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:48 smithi096 ceph-mon[32753]: pgmap v14195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25907.093808 secs 2024-04-26T19:02:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25909.984908 secs 2024-04-26T19:02:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25909.984908 secs 2024-04-26T19:02:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25909.984908 secs 2024-04-26T19:02:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:50 smithi194 ceph-mon[33017]: pgmap v14196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:50 smithi033 ceph-mon[27845]: pgmap v14196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:50 smithi096 ceph-mon[32753]: pgmap v14196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:51.797+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:51.797+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:02:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:02:51.797+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:02:52.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:52 smithi033 ceph-mon[27845]: pgmap v14197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:52 smithi194 ceph-mon[33017]: pgmap v14197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:52 smithi096 ceph-mon[32753]: pgmap v14197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25912.093864 secs 2024-04-26T19:02:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25912.093864 secs 2024-04-26T19:02:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25912.093864 secs 2024-04-26T19:02:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:54 smithi194 ceph-mon[33017]: pgmap v14198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25914.984986 secs 2024-04-26T19:02:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:54 smithi033 ceph-mon[27845]: pgmap v14198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25914.984986 secs 2024-04-26T19:02:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:54 smithi096 ceph-mon[32753]: pgmap v14198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25914.984986 secs 2024-04-26T19:02:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:56 smithi194 ceph-mon[33017]: pgmap v14199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:56 smithi033 ceph-mon[27845]: pgmap v14199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:56 smithi096 ceph-mon[32753]: pgmap v14199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:02:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:58 smithi194 ceph-mon[33017]: pgmap v14200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25917.093948 secs 2024-04-26T19:02:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:58 smithi033 ceph-mon[27845]: pgmap v14200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25917.093948 secs 2024-04-26T19:02:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:58 smithi096 ceph-mon[32753]: pgmap v14200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:02:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25917.093948 secs 2024-04-26T19:02:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25919.985165 secs 2024-04-26T19:02:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25919.985165 secs 2024-04-26T19:02:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25919.985165 secs 2024-04-26T19:03:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:00 smithi194 ceph-mon[33017]: pgmap v14201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:00 smithi033 ceph-mon[27845]: pgmap v14201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:00 smithi096 ceph-mon[32753]: pgmap v14201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:02 smithi194 ceph-mon[33017]: pgmap v14202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:02 smithi033 ceph-mon[27845]: pgmap v14202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:02 smithi096 ceph-mon[32753]: pgmap v14202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25922.093982 secs 2024-04-26T19:03:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25922.093982 secs 2024-04-26T19:03:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25922.093982 secs 2024-04-26T19:03:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:04 smithi194 ceph-mon[33017]: pgmap v14203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25924.985279 secs 2024-04-26T19:03:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:04 smithi033 ceph-mon[27845]: pgmap v14203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25924.985279 secs 2024-04-26T19:03:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:04 smithi096 ceph-mon[32753]: pgmap v14203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25924.985279 secs 2024-04-26T19:03:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:06.812+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:06.812+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:03:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:06.812+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:06 smithi194 ceph-mon[33017]: pgmap v14204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:06 smithi033 ceph-mon[27845]: pgmap v14204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:06 smithi096 ceph-mon[32753]: pgmap v14204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:08 smithi194 ceph-mon[33017]: pgmap v14205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25927.094071 secs 2024-04-26T19:03:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:08 smithi033 ceph-mon[27845]: pgmap v14205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25927.094071 secs 2024-04-26T19:03:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:08 smithi096 ceph-mon[32753]: pgmap v14205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25927.094071 secs 2024-04-26T19:03:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25929.985406 secs 2024-04-26T19:03:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25929.985406 secs 2024-04-26T19:03:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25929.985406 secs 2024-04-26T19:03:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:10 smithi194 ceph-mon[33017]: pgmap v14206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:10 smithi033 ceph-mon[27845]: pgmap v14206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:10 smithi096 ceph-mon[32753]: pgmap v14206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:12 smithi194 ceph-mon[33017]: pgmap v14207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:12 smithi033 ceph-mon[27845]: pgmap v14207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:12 smithi096 ceph-mon[32753]: pgmap v14207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25932.094156 secs 2024-04-26T19:03:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25932.094156 secs 2024-04-26T19:03:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25932.094156 secs 2024-04-26T19:03:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:14 smithi194 ceph-mon[33017]: pgmap v14208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25934.985529 secs 2024-04-26T19:03:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:14 smithi033 ceph-mon[27845]: pgmap v14208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25934.985529 secs 2024-04-26T19:03:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:14 smithi096 ceph-mon[32753]: pgmap v14208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25934.985529 secs 2024-04-26T19:03:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:15 smithi194 ceph-mon[33017]: pgmap v14209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:15 smithi033 ceph-mon[27845]: pgmap v14209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:15 smithi096 ceph-mon[32753]: pgmap v14209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:18 smithi033 ceph-mon[27845]: pgmap v14210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25937.094239 secs 2024-04-26T19:03:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:18 smithi096 ceph-mon[32753]: pgmap v14210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25937.094239 secs 2024-04-26T19:03:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:18 smithi194 ceph-mon[33017]: pgmap v14210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25937.094239 secs 2024-04-26T19:03:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25939.985624 secs 2024-04-26T19:03:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25939.985624 secs 2024-04-26T19:03:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25939.985624 secs 2024-04-26T19:03:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:20 smithi194 ceph-mon[33017]: pgmap v14211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:20 smithi033 ceph-mon[27845]: pgmap v14211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:20 smithi096 ceph-mon[32753]: pgmap v14211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:21.829+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:21.829+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:21.829+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:03:22.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:22 smithi033 ceph-mon[27845]: pgmap v14212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:22 smithi194 ceph-mon[33017]: pgmap v14212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:22 smithi096 ceph-mon[32753]: pgmap v14212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25942.094330 secs 2024-04-26T19:03:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25942.094330 secs 2024-04-26T19:03:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25942.094330 secs 2024-04-26T19:03:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:24 smithi194 ceph-mon[33017]: pgmap v14213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25944.985747 secs 2024-04-26T19:03:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:24 smithi033 ceph-mon[27845]: pgmap v14213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25944.985747 secs 2024-04-26T19:03:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:24 smithi096 ceph-mon[32753]: pgmap v14213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25944.985747 secs 2024-04-26T19:03:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:26 smithi194 ceph-mon[33017]: pgmap v14214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:26 smithi033 ceph-mon[27845]: pgmap v14214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:26 smithi096 ceph-mon[32753]: pgmap v14214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:28 smithi194 ceph-mon[33017]: pgmap v14215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:03:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25947.094402 secs 2024-04-26T19:03:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:28 smithi033 ceph-mon[27845]: pgmap v14215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:03:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25947.094402 secs 2024-04-26T19:03:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:28 smithi096 ceph-mon[32753]: pgmap v14215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:03:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25947.094402 secs 2024-04-26T19:03:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25949.985838 secs 2024-04-26T19:03:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:29 smithi033 ceph-mon[27845]: pgmap v14216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25949.985838 secs 2024-04-26T19:03:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:29 smithi096 ceph-mon[32753]: pgmap v14216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25949.985838 secs 2024-04-26T19:03:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:29 smithi194 ceph-mon[33017]: pgmap v14216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:03:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:03:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:30 smithi033 ceph-mon[27845]: pgmap v14217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T19:03:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:30 smithi033 ceph-mon[27845]: pgmap v14218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:03:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:03:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:30 smithi096 ceph-mon[32753]: pgmap v14217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T19:03:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:30 smithi096 ceph-mon[32753]: pgmap v14218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:03:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:03:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:30 smithi194 ceph-mon[33017]: pgmap v14217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T19:03:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:30 smithi194 ceph-mon[33017]: pgmap v14218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:03:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:32 smithi194 ceph-mon[33017]: pgmap v14219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:32 smithi033 ceph-mon[27845]: pgmap v14219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:32 smithi096 ceph-mon[32753]: pgmap v14219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25952.094484 secs 2024-04-26T19:03:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25954.985971 secs 2024-04-26T19:03:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25952.094484 secs 2024-04-26T19:03:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25954.985971 secs 2024-04-26T19:03:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25952.094484 secs 2024-04-26T19:03:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25954.985971 secs 2024-04-26T19:03:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:34 smithi194 ceph-mon[33017]: pgmap v14220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:03:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:34 smithi033 ceph-mon[27845]: pgmap v14220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:03:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:34 smithi096 ceph-mon[32753]: pgmap v14220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:03:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:36.844+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:36.844 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:36.844+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:36.844+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:03:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:36 smithi194 ceph-mon[33017]: pgmap v14221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:36 smithi033 ceph-mon[27845]: pgmap v14221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:36 smithi096 ceph-mon[32753]: pgmap v14221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:03:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25957.094568 secs 2024-04-26T19:03:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:39 smithi194 ceph-mon[33017]: pgmap v14222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:03:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25959.986099 secs 2024-04-26T19:03:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25957.094568 secs 2024-04-26T19:03:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:39 smithi033 ceph-mon[27845]: pgmap v14222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:03:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25959.986099 secs 2024-04-26T19:03:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25957.094568 secs 2024-04-26T19:03:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:39 smithi096 ceph-mon[32753]: pgmap v14222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:03:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25959.986099 secs 2024-04-26T19:03:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:41 smithi194 ceph-mon[33017]: pgmap v14223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:03:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:41 smithi033 ceph-mon[27845]: pgmap v14223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:03:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:41 smithi096 ceph-mon[32753]: pgmap v14223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:03:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:43 smithi194 ceph-mon[33017]: pgmap v14224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:43 smithi033 ceph-mon[27845]: pgmap v14224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:43 smithi096 ceph-mon[32753]: pgmap v14224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25962.094650 secs 2024-04-26T19:03:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25964.986222 secs 2024-04-26T19:03:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25962.094650 secs 2024-04-26T19:03:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25964.986222 secs 2024-04-26T19:03:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25962.094650 secs 2024-04-26T19:03:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25964.986222 secs 2024-04-26T19:03:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:45 smithi194 ceph-mon[33017]: pgmap v14225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:45 smithi033 ceph-mon[27845]: pgmap v14225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:45 smithi096 ceph-mon[32753]: pgmap v14225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:47 smithi194 ceph-mon[33017]: pgmap v14226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:47 smithi033 ceph-mon[27845]: pgmap v14226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:47 smithi096 ceph-mon[32753]: pgmap v14226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:03:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25967.094735 secs 2024-04-26T19:03:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:49 smithi194 ceph-mon[33017]: pgmap v14227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25969.986352 secs 2024-04-26T19:03:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25967.094735 secs 2024-04-26T19:03:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:49 smithi033 ceph-mon[27845]: pgmap v14227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25969.986352 secs 2024-04-26T19:03:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25967.094735 secs 2024-04-26T19:03:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:49 smithi096 ceph-mon[32753]: pgmap v14227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:03:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25969.986352 secs 2024-04-26T19:03:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:51 smithi194 ceph-mon[33017]: pgmap v14228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:03:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:51 smithi033 ceph-mon[27845]: pgmap v14228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:03:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:51 smithi096 ceph-mon[32753]: pgmap v14228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:03:51.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:51.853+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:51.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:51.853+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:03:51.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:03:51.853+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:03:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:53 smithi194 ceph-mon[33017]: pgmap v14229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:53 smithi033 ceph-mon[27845]: pgmap v14229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:53 smithi096 ceph-mon[32753]: pgmap v14229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25972.094795 secs 2024-04-26T19:03:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25974.986438 secs 2024-04-26T19:03:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25972.094795 secs 2024-04-26T19:03:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25974.986438 secs 2024-04-26T19:03:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25972.094795 secs 2024-04-26T19:03:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25974.986438 secs 2024-04-26T19:03:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:55 smithi194 ceph-mon[33017]: pgmap v14230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:55 smithi033 ceph-mon[27845]: pgmap v14230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:55 smithi096 ceph-mon[32753]: pgmap v14230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:57 smithi194 ceph-mon[33017]: pgmap v14231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:03:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:57 smithi033 ceph-mon[27845]: pgmap v14231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:03:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:57 smithi096 ceph-mon[32753]: pgmap v14231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:03:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25977.094884 secs 2024-04-26T19:03:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:59 smithi194 ceph-mon[33017]: pgmap v14232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25979.986523 secs 2024-04-26T19:03:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25977.094884 secs 2024-04-26T19:03:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:59 smithi033 ceph-mon[27845]: pgmap v14232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25979.986523 secs 2024-04-26T19:03:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25977.094884 secs 2024-04-26T19:03:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:59 smithi096 ceph-mon[32753]: pgmap v14232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:03:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25979.986523 secs 2024-04-26T19:04:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:01 smithi194 ceph-mon[33017]: pgmap v14233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:04:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:01 smithi033 ceph-mon[27845]: pgmap v14233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:04:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:01 smithi096 ceph-mon[32753]: pgmap v14233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:04:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:03 smithi194 ceph-mon[33017]: pgmap v14234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:03 smithi033 ceph-mon[27845]: pgmap v14234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:03 smithi096 ceph-mon[32753]: pgmap v14234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25982.094962 secs 2024-04-26T19:04:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25984.986606 secs 2024-04-26T19:04:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25982.094962 secs 2024-04-26T19:04:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25984.986606 secs 2024-04-26T19:04:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25982.094962 secs 2024-04-26T19:04:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25984.986606 secs 2024-04-26T19:04:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:05 smithi194 ceph-mon[33017]: pgmap v14235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:05 smithi033 ceph-mon[27845]: pgmap v14235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:05 smithi096 ceph-mon[32753]: pgmap v14235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:06.865+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:06.865+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:06.865+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:04:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:07 smithi194 ceph-mon[33017]: pgmap v14236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:07 smithi033 ceph-mon[27845]: pgmap v14236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:07 smithi096 ceph-mon[32753]: pgmap v14236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25987.095041 secs 2024-04-26T19:04:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:09 smithi033 ceph-mon[27845]: pgmap v14237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25989.986689 secs 2024-04-26T19:04:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25987.095041 secs 2024-04-26T19:04:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:09 smithi096 ceph-mon[32753]: pgmap v14237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25989.986689 secs 2024-04-26T19:04:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25987.095041 secs 2024-04-26T19:04:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:09 smithi194 ceph-mon[33017]: pgmap v14237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25989.986689 secs 2024-04-26T19:04:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:11 smithi033 ceph-mon[27845]: pgmap v14238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:11 smithi096 ceph-mon[32753]: pgmap v14238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:11 smithi194 ceph-mon[33017]: pgmap v14238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:13.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:13 smithi194 ceph-mon[33017]: pgmap v14239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:13 smithi033 ceph-mon[27845]: pgmap v14239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:13 smithi096 ceph-mon[32753]: pgmap v14239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25992.095102 secs 2024-04-26T19:04:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25994.986785 secs 2024-04-26T19:04:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25992.095102 secs 2024-04-26T19:04:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25994.986785 secs 2024-04-26T19:04:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25992.095102 secs 2024-04-26T19:04:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25994.986785 secs 2024-04-26T19:04:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:15 smithi033 ceph-mon[27845]: pgmap v14240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:15 smithi096 ceph-mon[32753]: pgmap v14240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:15 smithi194 ceph-mon[33017]: pgmap v14240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:17 smithi033 ceph-mon[27845]: pgmap v14241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:17.501 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:17 smithi096 ceph-mon[32753]: pgmap v14241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:17 smithi194 ceph-mon[33017]: pgmap v14241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 25997.095191 secs 2024-04-26T19:04:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:19 smithi033 ceph-mon[27845]: pgmap v14242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 25999.986946 secs 2024-04-26T19:04:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 25997.095191 secs 2024-04-26T19:04:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:19 smithi096 ceph-mon[32753]: pgmap v14242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 25999.986946 secs 2024-04-26T19:04:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 25997.095191 secs 2024-04-26T19:04:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:19 smithi194 ceph-mon[33017]: pgmap v14242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 25999.986946 secs 2024-04-26T19:04:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:21 smithi033 ceph-mon[27845]: pgmap v14243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:21 smithi096 ceph-mon[32753]: pgmap v14243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:21 smithi194 ceph-mon[33017]: pgmap v14243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:21.868+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:21.868+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:21.868+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:04:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:23 smithi033 ceph-mon[27845]: pgmap v14244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:23 smithi096 ceph-mon[32753]: pgmap v14244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:23 smithi194 ceph-mon[33017]: pgmap v14244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26002.095248 secs 2024-04-26T19:04:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26004.987074 secs 2024-04-26T19:04:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26002.095248 secs 2024-04-26T19:04:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26004.987074 secs 2024-04-26T19:04:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26002.095248 secs 2024-04-26T19:04:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26004.987074 secs 2024-04-26T19:04:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:25 smithi033 ceph-mon[27845]: pgmap v14245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:25 smithi096 ceph-mon[32753]: pgmap v14245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:25 smithi194 ceph-mon[33017]: pgmap v14245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:27 smithi033 ceph-mon[27845]: pgmap v14246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:27 smithi096 ceph-mon[32753]: pgmap v14246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:27 smithi194 ceph-mon[33017]: pgmap v14246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26007.095330 secs 2024-04-26T19:04:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:29 smithi033 ceph-mon[27845]: pgmap v14247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26009.987175 secs 2024-04-26T19:04:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26007.095330 secs 2024-04-26T19:04:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:29 smithi096 ceph-mon[32753]: pgmap v14247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26009.987175 secs 2024-04-26T19:04:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26007.095330 secs 2024-04-26T19:04:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:29 smithi194 ceph-mon[33017]: pgmap v14247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26009.987175 secs 2024-04-26T19:04:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:31 smithi096 ceph-mon[32753]: pgmap v14248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:04:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:31 smithi194 ceph-mon[33017]: pgmap v14248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:04:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:31 smithi033 ceph-mon[27845]: pgmap v14248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:04:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:04:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:04:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:04:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:04:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:04:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:04:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:04:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:04:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:04:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:33 smithi194 ceph-mon[33017]: pgmap v14249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:33 smithi194 ceph-mon[33017]: pgmap v14250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:33 smithi033 ceph-mon[27845]: pgmap v14249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:33 smithi033 ceph-mon[27845]: pgmap v14250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:33 smithi096 ceph-mon[32753]: pgmap v14249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:33 smithi096 ceph-mon[32753]: pgmap v14250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26012.095414 secs 2024-04-26T19:04:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26014.987330 secs 2024-04-26T19:04:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26012.095414 secs 2024-04-26T19:04:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26014.987330 secs 2024-04-26T19:04:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26012.095414 secs 2024-04-26T19:04:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26014.987330 secs 2024-04-26T19:04:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:35 smithi194 ceph-mon[33017]: pgmap v14251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:35 smithi033 ceph-mon[27845]: pgmap v14251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:35 smithi096 ceph-mon[32753]: pgmap v14251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:36.881+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:36.882+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:36.882+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:04:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:37 smithi194 ceph-mon[33017]: pgmap v14252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:37 smithi033 ceph-mon[27845]: pgmap v14252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:37 smithi096 ceph-mon[32753]: pgmap v14252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:39 smithi194 ceph-mon[33017]: pgmap v14253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26017.095497 secs 2024-04-26T19:04:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26019.987453 secs 2024-04-26T19:04:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:39 smithi033 ceph-mon[27845]: pgmap v14253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26017.095497 secs 2024-04-26T19:04:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26019.987453 secs 2024-04-26T19:04:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:39 smithi096 ceph-mon[32753]: pgmap v14253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26017.095497 secs 2024-04-26T19:04:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26019.987453 secs 2024-04-26T19:04:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:41 smithi194 ceph-mon[33017]: pgmap v14254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:41 smithi033 ceph-mon[27845]: pgmap v14254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:41 smithi096 ceph-mon[32753]: pgmap v14254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:04:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:43 smithi194 ceph-mon[33017]: pgmap v14255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:04:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:43 smithi033 ceph-mon[27845]: pgmap v14255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:04:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:43 smithi096 ceph-mon[32753]: pgmap v14255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:04:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26022.095580 secs 2024-04-26T19:04:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26024.987579 secs 2024-04-26T19:04:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26022.095580 secs 2024-04-26T19:04:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26024.987579 secs 2024-04-26T19:04:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26022.095580 secs 2024-04-26T19:04:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26024.987579 secs 2024-04-26T19:04:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:45 smithi194 ceph-mon[33017]: pgmap v14256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:45 smithi033 ceph-mon[27845]: pgmap v14256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:45 smithi096 ceph-mon[32753]: pgmap v14256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:47 smithi194 ceph-mon[33017]: pgmap v14257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:47 smithi033 ceph-mon[27845]: pgmap v14257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:47 smithi096 ceph-mon[32753]: pgmap v14257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:49.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:49 smithi033 ceph-mon[27845]: pgmap v14258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:49.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26027.095639 secs 2024-04-26T19:04:49.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26029.987723 secs 2024-04-26T19:04:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:49 smithi194 ceph-mon[33017]: pgmap v14258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26027.095639 secs 2024-04-26T19:04:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26029.987723 secs 2024-04-26T19:04:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:49 smithi096 ceph-mon[32753]: pgmap v14258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26027.095639 secs 2024-04-26T19:04:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26029.987723 secs 2024-04-26T19:04:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:51 smithi194 ceph-mon[33017]: pgmap v14259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:51 smithi033 ceph-mon[27845]: pgmap v14259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:51 smithi096 ceph-mon[32753]: pgmap v14259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:51.886+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:51.886+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:04:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:04:51.886+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:04:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:53 smithi194 ceph-mon[33017]: pgmap v14260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:53 smithi033 ceph-mon[27845]: pgmap v14260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:53 smithi096 ceph-mon[32753]: pgmap v14260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26032.095721 secs 2024-04-26T19:04:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26034.987865 secs 2024-04-26T19:04:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26032.095721 secs 2024-04-26T19:04:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26034.987865 secs 2024-04-26T19:04:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26032.095721 secs 2024-04-26T19:04:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26034.987865 secs 2024-04-26T19:04:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:55 smithi194 ceph-mon[33017]: pgmap v14261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:55 smithi033 ceph-mon[27845]: pgmap v14261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:55 smithi096 ceph-mon[32753]: pgmap v14261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:57 smithi194 ceph-mon[33017]: pgmap v14262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:57 smithi033 ceph-mon[27845]: pgmap v14262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:57 smithi096 ceph-mon[32753]: pgmap v14262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:04:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26037.095806 secs 2024-04-26T19:04:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26037.095806 secs 2024-04-26T19:04:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26037.095806 secs 2024-04-26T19:04:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:59 smithi194 ceph-mon[33017]: pgmap v14263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:04:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26039.987986 secs 2024-04-26T19:04:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:59 smithi033 ceph-mon[27845]: pgmap v14263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:04:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26039.987986 secs 2024-04-26T19:04:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:59 smithi096 ceph-mon[32753]: pgmap v14263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:04:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:04:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26039.987986 secs 2024-04-26T19:05:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:01 smithi194 ceph-mon[33017]: pgmap v14264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:01 smithi033 ceph-mon[27845]: pgmap v14264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:01 smithi096 ceph-mon[32753]: pgmap v14264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:03 smithi194 ceph-mon[33017]: pgmap v14265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26042.095887 secs 2024-04-26T19:05:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:03 smithi033 ceph-mon[27845]: pgmap v14265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26042.095887 secs 2024-04-26T19:05:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:03 smithi096 ceph-mon[32753]: pgmap v14265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26042.095887 secs 2024-04-26T19:05:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26044.988144 secs 2024-04-26T19:05:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26044.988144 secs 2024-04-26T19:05:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26044.988144 secs 2024-04-26T19:05:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:05 smithi033 ceph-mon[27845]: pgmap v14266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:05 smithi194 ceph-mon[33017]: pgmap v14266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:05 smithi096 ceph-mon[32753]: pgmap v14266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:06.898+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:06.898+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:06.898+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:05:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:07 smithi194 ceph-mon[33017]: pgmap v14267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:07 smithi033 ceph-mon[27845]: pgmap v14267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:07 smithi096 ceph-mon[32753]: pgmap v14267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26047.095993 secs 2024-04-26T19:05:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26047.095993 secs 2024-04-26T19:05:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26047.095993 secs 2024-04-26T19:05:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:09 smithi194 ceph-mon[33017]: pgmap v14268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26049.988274 secs 2024-04-26T19:05:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:09 smithi033 ceph-mon[27845]: pgmap v14268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26049.988274 secs 2024-04-26T19:05:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:09 smithi096 ceph-mon[32753]: pgmap v14268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26049.988274 secs 2024-04-26T19:05:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:11 smithi194 ceph-mon[33017]: pgmap v14269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:11 smithi033 ceph-mon[27845]: pgmap v14269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:11 smithi096 ceph-mon[32753]: pgmap v14269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:13 smithi194 ceph-mon[33017]: pgmap v14270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26052.096053 secs 2024-04-26T19:05:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:13 smithi033 ceph-mon[27845]: pgmap v14270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26052.096053 secs 2024-04-26T19:05:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:13 smithi096 ceph-mon[32753]: pgmap v14270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26052.096053 secs 2024-04-26T19:05:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26054.988386 secs 2024-04-26T19:05:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26054.988386 secs 2024-04-26T19:05:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26054.988386 secs 2024-04-26T19:05:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:15 smithi033 ceph-mon[27845]: pgmap v14271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:15 smithi096 ceph-mon[32753]: pgmap v14271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:15 smithi194 ceph-mon[33017]: pgmap v14271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:17 smithi033 ceph-mon[27845]: pgmap v14272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:17 smithi096 ceph-mon[32753]: pgmap v14272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:17 smithi194 ceph-mon[33017]: pgmap v14272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26057.096115 secs 2024-04-26T19:05:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26057.096115 secs 2024-04-26T19:05:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26057.096115 secs 2024-04-26T19:05:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:19 smithi033 ceph-mon[27845]: pgmap v14273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26059.988507 secs 2024-04-26T19:05:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:19 smithi096 ceph-mon[32753]: pgmap v14273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26059.988507 secs 2024-04-26T19:05:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:19 smithi194 ceph-mon[33017]: pgmap v14273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26059.988507 secs 2024-04-26T19:05:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:21 smithi033 ceph-mon[27845]: pgmap v14274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:21 smithi096 ceph-mon[32753]: pgmap v14274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:21 smithi194 ceph-mon[33017]: pgmap v14274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:21.915+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:21.915+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:21.915+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:05:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:23 smithi033 ceph-mon[27845]: pgmap v14275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26062.096196 secs 2024-04-26T19:05:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:23 smithi096 ceph-mon[32753]: pgmap v14275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26062.096196 secs 2024-04-26T19:05:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:23 smithi194 ceph-mon[33017]: pgmap v14275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26062.096196 secs 2024-04-26T19:05:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26064.988626 secs 2024-04-26T19:05:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26064.988626 secs 2024-04-26T19:05:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26064.988626 secs 2024-04-26T19:05:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:25 smithi033 ceph-mon[27845]: pgmap v14276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:25 smithi096 ceph-mon[32753]: pgmap v14276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:25 smithi194 ceph-mon[33017]: pgmap v14276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:26.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:26.228+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:05:26.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:26.228+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:05:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:26.229+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 206 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0100 con 0x564385821900 2024-04-26T19:05:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:26.229+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 206 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:05:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:27 smithi033 ceph-mon[27845]: pgmap v14277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:27 smithi096 ceph-mon[32753]: pgmap v14277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:27 smithi194 ceph-mon[33017]: pgmap v14277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26067.096266 secs 2024-04-26T19:05:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26067.096266 secs 2024-04-26T19:05:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26067.096266 secs 2024-04-26T19:05:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:29 smithi033 ceph-mon[27845]: pgmap v14278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26069.988774 secs 2024-04-26T19:05:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:29 smithi096 ceph-mon[32753]: pgmap v14278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26069.988774 secs 2024-04-26T19:05:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:29 smithi194 ceph-mon[33017]: pgmap v14278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26069.988774 secs 2024-04-26T19:05:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:31 smithi033 ceph-mon[27845]: pgmap v14279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:31 smithi096 ceph-mon[32753]: pgmap v14279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:31 smithi194 ceph-mon[33017]: pgmap v14279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:05:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:05:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:05:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:33 smithi194 ceph-mon[33017]: pgmap v14280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26072.096350 secs 2024-04-26T19:05:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:05:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:05:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:05:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:33 smithi033 ceph-mon[27845]: pgmap v14280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26072.096350 secs 2024-04-26T19:05:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:05:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:05:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:05:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:33 smithi096 ceph-mon[32753]: pgmap v14280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26072.096350 secs 2024-04-26T19:05:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:05:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:05:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:05:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:34 smithi194 ceph-mon[33017]: pgmap v14281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:34 smithi194 ceph-mon[33017]: pgmap v14282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26074.988898 secs 2024-04-26T19:05:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:34 smithi033 ceph-mon[27845]: pgmap v14281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:34 smithi033 ceph-mon[27845]: pgmap v14282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26074.988898 secs 2024-04-26T19:05:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:34 smithi096 ceph-mon[32753]: pgmap v14281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:34 smithi096 ceph-mon[32753]: pgmap v14282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26074.988898 secs 2024-04-26T19:05:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:36 smithi194 ceph-mon[33017]: pgmap v14283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:36.925 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:36.925+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:36.925 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:36.925+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:36 smithi033 ceph-mon[27845]: pgmap v14283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:36.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:36.925+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:05:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:36 smithi096 ceph-mon[32753]: pgmap v14283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:38 smithi194 ceph-mon[33017]: pgmap v14284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26077.096437 secs 2024-04-26T19:05:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:38 smithi033 ceph-mon[27845]: pgmap v14284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26077.096437 secs 2024-04-26T19:05:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:38 smithi096 ceph-mon[32753]: pgmap v14284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26077.096437 secs 2024-04-26T19:05:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26079.989034 secs 2024-04-26T19:05:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26079.989034 secs 2024-04-26T19:05:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26079.989034 secs 2024-04-26T19:05:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:40 smithi194 ceph-mon[33017]: pgmap v14285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:05:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:40 smithi033 ceph-mon[27845]: pgmap v14285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:05:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:40 smithi096 ceph-mon[32753]: pgmap v14285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:05:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:42 smithi194 ceph-mon[33017]: pgmap v14286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:42 smithi033 ceph-mon[27845]: pgmap v14286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:42 smithi096 ceph-mon[32753]: pgmap v14286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:05:43.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26082.096520 secs 2024-04-26T19:05:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26082.096520 secs 2024-04-26T19:05:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26082.096520 secs 2024-04-26T19:05:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:44 smithi194 ceph-mon[33017]: pgmap v14287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:05:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26084.989174 secs 2024-04-26T19:05:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:44 smithi033 ceph-mon[27845]: pgmap v14287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:05:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26084.989174 secs 2024-04-26T19:05:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:44 smithi096 ceph-mon[32753]: pgmap v14287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:05:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26084.989174 secs 2024-04-26T19:05:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:46 smithi194 ceph-mon[33017]: pgmap v14288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:46 smithi033 ceph-mon[27845]: pgmap v14288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:46 smithi096 ceph-mon[32753]: pgmap v14288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:48 smithi194 ceph-mon[33017]: pgmap v14289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26087.096582 secs 2024-04-26T19:05:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:48 smithi033 ceph-mon[27845]: pgmap v14289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26087.096582 secs 2024-04-26T19:05:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:48 smithi096 ceph-mon[32753]: pgmap v14289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26087.096582 secs 2024-04-26T19:05:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26089.989308 secs 2024-04-26T19:05:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26089.989308 secs 2024-04-26T19:05:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26089.989308 secs 2024-04-26T19:05:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:50 smithi194 ceph-mon[33017]: pgmap v14290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:50 smithi033 ceph-mon[27845]: pgmap v14290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:50 smithi096 ceph-mon[32753]: pgmap v14290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:51.940+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:51.940+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:05:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:05:51.940+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:05:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:52 smithi194 ceph-mon[33017]: pgmap v14291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:52 smithi033 ceph-mon[27845]: pgmap v14291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:52 smithi096 ceph-mon[32753]: pgmap v14291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:53.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26092.096664 secs 2024-04-26T19:05:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26092.096664 secs 2024-04-26T19:05:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26092.096664 secs 2024-04-26T19:05:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:54 smithi194 ceph-mon[33017]: pgmap v14292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26094.989399 secs 2024-04-26T19:05:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:54 smithi033 ceph-mon[27845]: pgmap v14292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26094.989399 secs 2024-04-26T19:05:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:54 smithi096 ceph-mon[32753]: pgmap v14292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26094.989399 secs 2024-04-26T19:05:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:56 smithi194 ceph-mon[33017]: pgmap v14293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:56 smithi033 ceph-mon[27845]: pgmap v14293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:56 smithi096 ceph-mon[32753]: pgmap v14293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:05:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:58 smithi194 ceph-mon[33017]: pgmap v14294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26097.096750 secs 2024-04-26T19:05:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:58 smithi033 ceph-mon[27845]: pgmap v14294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26097.096750 secs 2024-04-26T19:05:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:58 smithi096 ceph-mon[32753]: pgmap v14294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:05:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26097.096750 secs 2024-04-26T19:05:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26099.989527 secs 2024-04-26T19:05:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26099.989527 secs 2024-04-26T19:05:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26099.989527 secs 2024-04-26T19:06:00.612 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:00 smithi033 ceph-mon[27845]: pgmap v14295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:00 smithi194 ceph-mon[33017]: pgmap v14295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:00 smithi096 ceph-mon[32753]: pgmap v14295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:02 smithi194 ceph-mon[33017]: pgmap v14296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:02 smithi033 ceph-mon[27845]: pgmap v14296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:02 smithi096 ceph-mon[32753]: pgmap v14296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26102.096831 secs 2024-04-26T19:06:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26102.096831 secs 2024-04-26T19:06:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26102.096831 secs 2024-04-26T19:06:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:04 smithi194 ceph-mon[33017]: pgmap v14297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26104.989606 secs 2024-04-26T19:06:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:04 smithi033 ceph-mon[27845]: pgmap v14297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26104.989606 secs 2024-04-26T19:06:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:04 smithi096 ceph-mon[32753]: pgmap v14297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26104.989606 secs 2024-04-26T19:06:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:06 smithi194 ceph-mon[33017]: pgmap v14298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:06 smithi033 ceph-mon[27845]: pgmap v14298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:06 smithi096 ceph-mon[32753]: pgmap v14298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:06.944+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:06.944+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:06.944+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:06:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:08 smithi194 ceph-mon[33017]: pgmap v14299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26107.096917 secs 2024-04-26T19:06:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:08 smithi033 ceph-mon[27845]: pgmap v14299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26107.096917 secs 2024-04-26T19:06:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:08 smithi096 ceph-mon[32753]: pgmap v14299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26107.096917 secs 2024-04-26T19:06:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26109.989758 secs 2024-04-26T19:06:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26109.989758 secs 2024-04-26T19:06:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26109.989758 secs 2024-04-26T19:06:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:10 smithi194 ceph-mon[33017]: pgmap v14300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:10 smithi033 ceph-mon[27845]: pgmap v14300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:10 smithi096 ceph-mon[32753]: pgmap v14300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:12 smithi194 ceph-mon[33017]: pgmap v14301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:12 smithi033 ceph-mon[27845]: pgmap v14301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:12 smithi096 ceph-mon[32753]: pgmap v14301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26112.097025 secs 2024-04-26T19:06:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26112.097025 secs 2024-04-26T19:06:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26112.097025 secs 2024-04-26T19:06:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:14 smithi194 ceph-mon[33017]: pgmap v14302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26114.989912 secs 2024-04-26T19:06:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:14 smithi033 ceph-mon[27845]: pgmap v14302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26114.989912 secs 2024-04-26T19:06:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:14 smithi096 ceph-mon[32753]: pgmap v14302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26114.989912 secs 2024-04-26T19:06:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:15 smithi033 ceph-mon[27845]: pgmap v14303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:15 smithi096 ceph-mon[32753]: pgmap v14303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:15 smithi194 ceph-mon[33017]: pgmap v14303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:18 smithi033 ceph-mon[27845]: pgmap v14304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26117.097114 secs 2024-04-26T19:06:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:18 smithi096 ceph-mon[32753]: pgmap v14304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26117.097114 secs 2024-04-26T19:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:18 smithi194 ceph-mon[33017]: pgmap v14304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26117.097114 secs 2024-04-26T19:06:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26119.990050 secs 2024-04-26T19:06:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26119.990050 secs 2024-04-26T19:06:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26119.990050 secs 2024-04-26T19:06:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:20 smithi033 ceph-mon[27845]: pgmap v14305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:20 smithi096 ceph-mon[32753]: pgmap v14305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:20 smithi194 ceph-mon[33017]: pgmap v14305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:21.960+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:21.960+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:21.960+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:06:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:22 smithi033 ceph-mon[27845]: pgmap v14306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:22 smithi096 ceph-mon[32753]: pgmap v14306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:22 smithi194 ceph-mon[33017]: pgmap v14306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26122.097199 secs 2024-04-26T19:06:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26122.097199 secs 2024-04-26T19:06:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26122.097199 secs 2024-04-26T19:06:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:24 smithi096 ceph-mon[32753]: pgmap v14307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26124.990176 secs 2024-04-26T19:06:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:24 smithi194 ceph-mon[33017]: pgmap v14307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26124.990176 secs 2024-04-26T19:06:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:24 smithi033 ceph-mon[27845]: pgmap v14307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26124.990176 secs 2024-04-26T19:06:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:26 smithi194 ceph-mon[33017]: pgmap v14308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:26 smithi033 ceph-mon[27845]: pgmap v14308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:26 smithi096 ceph-mon[32753]: pgmap v14308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:28 smithi194 ceph-mon[33017]: pgmap v14309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26127.097283 secs 2024-04-26T19:06:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:28 smithi033 ceph-mon[27845]: pgmap v14309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26127.097283 secs 2024-04-26T19:06:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:28 smithi096 ceph-mon[32753]: pgmap v14309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26127.097283 secs 2024-04-26T19:06:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26129.990298 secs 2024-04-26T19:06:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26129.990298 secs 2024-04-26T19:06:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26129.990298 secs 2024-04-26T19:06:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:30 smithi194 ceph-mon[33017]: pgmap v14310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:30 smithi033 ceph-mon[27845]: pgmap v14310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:30 smithi096 ceph-mon[32753]: pgmap v14310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:32 smithi194 ceph-mon[33017]: pgmap v14311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:32 smithi033 ceph-mon[27845]: pgmap v14311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:32 smithi096 ceph-mon[32753]: pgmap v14311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26132.097342 secs 2024-04-26T19:06:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26132.097342 secs 2024-04-26T19:06:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26132.097342 secs 2024-04-26T19:06:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:34 smithi194 ceph-mon[33017]: pgmap v14312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26134.990431 secs 2024-04-26T19:06:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:34 smithi033 ceph-mon[27845]: pgmap v14312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26134.990431 secs 2024-04-26T19:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:34 smithi096 ceph-mon[32753]: pgmap v14312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26134.990431 secs 2024-04-26T19:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:06:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:36 smithi194 ceph-mon[33017]: pgmap v14313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:06:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:06:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:36 smithi033 ceph-mon[27845]: pgmap v14313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:06:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:06:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:36 smithi096 ceph-mon[32753]: pgmap v14313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:06:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:06:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:06:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:36.964+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:36.964+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:36.965+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:37 smithi194 ceph-mon[33017]: pgmap v14314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:06:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:37 smithi194 ceph-mon[33017]: pgmap v14315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:37 smithi033 ceph-mon[27845]: pgmap v14314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:06:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:37 smithi033 ceph-mon[27845]: pgmap v14315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:37 smithi096 ceph-mon[32753]: pgmap v14314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:06:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:37 smithi096 ceph-mon[32753]: pgmap v14315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26137.097404 secs 2024-04-26T19:06:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26137.097404 secs 2024-04-26T19:06:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26137.097404 secs 2024-04-26T19:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:39 smithi194 ceph-mon[33017]: pgmap v14316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26139.990569 secs 2024-04-26T19:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:39 smithi033 ceph-mon[27845]: pgmap v14316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26139.990569 secs 2024-04-26T19:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:39 smithi096 ceph-mon[32753]: pgmap v14316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26139.990569 secs 2024-04-26T19:06:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:41 smithi194 ceph-mon[33017]: pgmap v14317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:41 smithi033 ceph-mon[27845]: pgmap v14317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:41 smithi096 ceph-mon[32753]: pgmap v14317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:43 smithi194 ceph-mon[33017]: pgmap v14318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26142.097489 secs 2024-04-26T19:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:43 smithi033 ceph-mon[27845]: pgmap v14318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26142.097489 secs 2024-04-26T19:06:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:43 smithi096 ceph-mon[32753]: pgmap v14318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:06:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26142.097489 secs 2024-04-26T19:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26144.990666 secs 2024-04-26T19:06:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26144.990666 secs 2024-04-26T19:06:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26144.990666 secs 2024-04-26T19:06:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:45 smithi033 ceph-mon[27845]: pgmap v14319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:06:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:45 smithi096 ceph-mon[32753]: pgmap v14319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:06:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:45 smithi194 ceph-mon[33017]: pgmap v14319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:06:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:46 smithi033 ceph-mon[27845]: pgmap v14320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:06:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:46 smithi096 ceph-mon[32753]: pgmap v14320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:06:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:46 smithi194 ceph-mon[33017]: pgmap v14320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:06:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26147.097571 secs 2024-04-26T19:06:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:49 smithi194 ceph-mon[33017]: pgmap v14321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26149.990790 secs 2024-04-26T19:06:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26147.097571 secs 2024-04-26T19:06:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:49 smithi033 ceph-mon[27845]: pgmap v14321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26149.990790 secs 2024-04-26T19:06:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26147.097571 secs 2024-04-26T19:06:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:49 smithi096 ceph-mon[32753]: pgmap v14321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26149.990790 secs 2024-04-26T19:06:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:51 smithi194 ceph-mon[33017]: pgmap v14322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:51 smithi033 ceph-mon[27845]: pgmap v14322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:51 smithi096 ceph-mon[32753]: pgmap v14322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:51.979+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:51.979+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:06:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:51.979+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:06:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:53 smithi194 ceph-mon[33017]: pgmap v14323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:53 smithi033 ceph-mon[27845]: pgmap v14323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:53 smithi096 ceph-mon[32753]: pgmap v14323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26152.097654 secs 2024-04-26T19:06:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26154.990912 secs 2024-04-26T19:06:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26152.097654 secs 2024-04-26T19:06:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26154.990912 secs 2024-04-26T19:06:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26152.097654 secs 2024-04-26T19:06:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26154.990912 secs 2024-04-26T19:06:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:55 smithi194 ceph-mon[33017]: pgmap v14324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:55 smithi033 ceph-mon[27845]: pgmap v14324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:55 smithi096 ceph-mon[32753]: pgmap v14324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:56.243+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:06:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:56.243+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:06:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:56.243+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 207 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:06:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:06:56.243+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 207 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T19:06:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:57 smithi194 ceph-mon[33017]: pgmap v14325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:57 smithi033 ceph-mon[27845]: pgmap v14325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:57 smithi096 ceph-mon[32753]: pgmap v14325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:06:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26157.097738 secs 2024-04-26T19:06:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:59 smithi194 ceph-mon[33017]: pgmap v14326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26159.991062 secs 2024-04-26T19:06:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26157.097738 secs 2024-04-26T19:06:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:59 smithi033 ceph-mon[27845]: pgmap v14326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26159.991062 secs 2024-04-26T19:06:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26157.097738 secs 2024-04-26T19:06:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:59 smithi096 ceph-mon[32753]: pgmap v14326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:06:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26159.991062 secs 2024-04-26T19:07:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:01 smithi194 ceph-mon[33017]: pgmap v14327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:01 smithi033 ceph-mon[27845]: pgmap v14327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:01 smithi096 ceph-mon[32753]: pgmap v14327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:03 smithi194 ceph-mon[33017]: pgmap v14328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:03 smithi033 ceph-mon[27845]: pgmap v14328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:03 smithi096 ceph-mon[32753]: pgmap v14328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26162.097820 secs 2024-04-26T19:07:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26164.991166 secs 2024-04-26T19:07:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26162.097820 secs 2024-04-26T19:07:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26164.991166 secs 2024-04-26T19:07:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26162.097820 secs 2024-04-26T19:07:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26164.991166 secs 2024-04-26T19:07:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:05 smithi194 ceph-mon[33017]: pgmap v14329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:05 smithi033 ceph-mon[27845]: pgmap v14329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:05 smithi096 ceph-mon[32753]: pgmap v14329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:06.982+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:06.982+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:06.982+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:07:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:07 smithi194 ceph-mon[33017]: pgmap v14330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:07 smithi033 ceph-mon[27845]: pgmap v14330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:07 smithi096 ceph-mon[32753]: pgmap v14330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26167.097904 secs 2024-04-26T19:07:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:09 smithi194 ceph-mon[33017]: pgmap v14331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26169.991316 secs 2024-04-26T19:07:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26167.097904 secs 2024-04-26T19:07:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:09 smithi033 ceph-mon[27845]: pgmap v14331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26169.991316 secs 2024-04-26T19:07:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26167.097904 secs 2024-04-26T19:07:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:09 smithi096 ceph-mon[32753]: pgmap v14331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26169.991316 secs 2024-04-26T19:07:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:11 smithi194 ceph-mon[33017]: pgmap v14332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:11 smithi033 ceph-mon[27845]: pgmap v14332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:11 smithi096 ceph-mon[32753]: pgmap v14332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:13 smithi194 ceph-mon[33017]: pgmap v14333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:13 smithi033 ceph-mon[27845]: pgmap v14333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:13 smithi096 ceph-mon[32753]: pgmap v14333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26172.097995 secs 2024-04-26T19:07:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26174.991383 secs 2024-04-26T19:07:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26172.097995 secs 2024-04-26T19:07:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26174.991383 secs 2024-04-26T19:07:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26172.097995 secs 2024-04-26T19:07:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26174.991383 secs 2024-04-26T19:07:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:15 smithi194 ceph-mon[33017]: pgmap v14334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:15 smithi033 ceph-mon[27845]: pgmap v14334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:15 smithi096 ceph-mon[32753]: pgmap v14334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:17 smithi194 ceph-mon[33017]: pgmap v14335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:17 smithi033 ceph-mon[27845]: pgmap v14335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:17 smithi096 ceph-mon[32753]: pgmap v14335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26177.098057 secs 2024-04-26T19:07:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:19 smithi033 ceph-mon[27845]: pgmap v14336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26179.991517 secs 2024-04-26T19:07:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26177.098057 secs 2024-04-26T19:07:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:19 smithi096 ceph-mon[32753]: pgmap v14336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26179.991517 secs 2024-04-26T19:07:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26177.098057 secs 2024-04-26T19:07:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:19 smithi194 ceph-mon[33017]: pgmap v14336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26179.991517 secs 2024-04-26T19:07:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:21 smithi033 ceph-mon[27845]: pgmap v14337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:21 smithi096 ceph-mon[32753]: pgmap v14337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:21 smithi194 ceph-mon[33017]: pgmap v14337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:21.998+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:21.999+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:21.999+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:07:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:23 smithi033 ceph-mon[27845]: pgmap v14338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:23 smithi096 ceph-mon[32753]: pgmap v14338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:23 smithi194 ceph-mon[33017]: pgmap v14338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26182.098118 secs 2024-04-26T19:07:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26184.991646 secs 2024-04-26T19:07:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26182.098118 secs 2024-04-26T19:07:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26184.991646 secs 2024-04-26T19:07:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26182.098118 secs 2024-04-26T19:07:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26184.991646 secs 2024-04-26T19:07:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:25 smithi033 ceph-mon[27845]: pgmap v14339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:25 smithi096 ceph-mon[32753]: pgmap v14339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:25 smithi194 ceph-mon[33017]: pgmap v14339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:27 smithi033 ceph-mon[27845]: pgmap v14340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:27 smithi096 ceph-mon[32753]: pgmap v14340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:27.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:27 smithi194 ceph-mon[33017]: pgmap v14340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26187.098207 secs 2024-04-26T19:07:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:29 smithi033 ceph-mon[27845]: pgmap v14341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26189.991730 secs 2024-04-26T19:07:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26187.098207 secs 2024-04-26T19:07:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:29 smithi096 ceph-mon[32753]: pgmap v14341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26189.991730 secs 2024-04-26T19:07:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26187.098207 secs 2024-04-26T19:07:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:29 smithi194 ceph-mon[33017]: pgmap v14341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26189.991730 secs 2024-04-26T19:07:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:31 smithi033 ceph-mon[27845]: pgmap v14342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:31 smithi096 ceph-mon[32753]: pgmap v14342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:31 smithi194 ceph-mon[33017]: pgmap v14342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:33 smithi033 ceph-mon[27845]: pgmap v14343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:33 smithi096 ceph-mon[32753]: pgmap v14343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:33 smithi194 ceph-mon[33017]: pgmap v14343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26192.098283 secs 2024-04-26T19:07:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26194.991873 secs 2024-04-26T19:07:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26192.098283 secs 2024-04-26T19:07:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26194.991873 secs 2024-04-26T19:07:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26192.098283 secs 2024-04-26T19:07:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26194.991873 secs 2024-04-26T19:07:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:35 smithi033 ceph-mon[27845]: pgmap v14344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:35 smithi096 ceph-mon[32753]: pgmap v14344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:35 smithi194 ceph-mon[33017]: pgmap v14344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:37.014+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:37.014+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:37.014+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:07:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:37 smithi096 ceph-mon[32753]: pgmap v14345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:37.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:07:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:37 smithi194 ceph-mon[33017]: pgmap v14345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:07:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:37 smithi033 ceph-mon[27845]: pgmap v14345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:07:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:07:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:07:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:07:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:07:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:07:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:07:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:07:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:07:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:07:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:39 smithi194 ceph-mon[33017]: pgmap v14346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:39 smithi194 ceph-mon[33017]: pgmap v14347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26197.098350 secs 2024-04-26T19:07:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26199.992024 secs 2024-04-26T19:07:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:39 smithi033 ceph-mon[27845]: pgmap v14346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:39 smithi033 ceph-mon[27845]: pgmap v14347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26197.098350 secs 2024-04-26T19:07:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26199.992024 secs 2024-04-26T19:07:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:39 smithi096 ceph-mon[32753]: pgmap v14346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:39 smithi096 ceph-mon[32753]: pgmap v14347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26197.098350 secs 2024-04-26T19:07:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26199.992024 secs 2024-04-26T19:07:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:41 smithi194 ceph-mon[33017]: pgmap v14348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:41 smithi033 ceph-mon[27845]: pgmap v14348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:41 smithi096 ceph-mon[32753]: pgmap v14348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:43 smithi194 ceph-mon[33017]: pgmap v14349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:43 smithi033 ceph-mon[27845]: pgmap v14349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:43 smithi096 ceph-mon[32753]: pgmap v14349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26202.098429 secs 2024-04-26T19:07:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26204.992177 secs 2024-04-26T19:07:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26202.098429 secs 2024-04-26T19:07:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26204.992177 secs 2024-04-26T19:07:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26202.098429 secs 2024-04-26T19:07:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26204.992177 secs 2024-04-26T19:07:45.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:45 smithi194 ceph-mon[33017]: pgmap v14350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:45 smithi033 ceph-mon[27845]: pgmap v14350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:45 smithi096 ceph-mon[32753]: pgmap v14350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:47 smithi194 ceph-mon[33017]: pgmap v14351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:47 smithi033 ceph-mon[27845]: pgmap v14351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:47 smithi096 ceph-mon[32753]: pgmap v14351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:07:49.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:49 smithi096 ceph-mon[32753]: pgmap v14352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:07:49.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26207.098510 secs 2024-04-26T19:07:49.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26209.992239 secs 2024-04-26T19:07:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:49 smithi194 ceph-mon[33017]: pgmap v14352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:07:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26207.098510 secs 2024-04-26T19:07:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26209.992239 secs 2024-04-26T19:07:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:49 smithi033 ceph-mon[27845]: pgmap v14352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:07:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26207.098510 secs 2024-04-26T19:07:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26209.992239 secs 2024-04-26T19:07:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:51 smithi194 ceph-mon[33017]: pgmap v14353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:51 smithi033 ceph-mon[27845]: pgmap v14353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:51 smithi096 ceph-mon[32753]: pgmap v14353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:52.029 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:52.030+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:52.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:52.030+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:07:52.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:07:52.030+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:07:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:53 smithi194 ceph-mon[33017]: pgmap v14354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:53 smithi033 ceph-mon[27845]: pgmap v14354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:53 smithi096 ceph-mon[32753]: pgmap v14354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26212.098593 secs 2024-04-26T19:07:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26214.992344 secs 2024-04-26T19:07:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26212.098593 secs 2024-04-26T19:07:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26214.992344 secs 2024-04-26T19:07:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26212.098593 secs 2024-04-26T19:07:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26214.992344 secs 2024-04-26T19:07:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:55 smithi194 ceph-mon[33017]: pgmap v14355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:55 smithi033 ceph-mon[27845]: pgmap v14355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:55 smithi096 ceph-mon[32753]: pgmap v14355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:57 smithi194 ceph-mon[33017]: pgmap v14356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:57 smithi033 ceph-mon[27845]: pgmap v14356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:57 smithi096 ceph-mon[32753]: pgmap v14356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:07:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:59 smithi194 ceph-mon[33017]: pgmap v14357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26217.098677 secs 2024-04-26T19:07:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:07:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26219.992474 secs 2024-04-26T19:07:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:59 smithi033 ceph-mon[27845]: pgmap v14357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26217.098677 secs 2024-04-26T19:07:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:07:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26219.992474 secs 2024-04-26T19:07:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:59 smithi096 ceph-mon[32753]: pgmap v14357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:07:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26217.098677 secs 2024-04-26T19:07:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:07:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26219.992474 secs 2024-04-26T19:08:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:01 smithi194 ceph-mon[33017]: pgmap v14358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:01 smithi033 ceph-mon[27845]: pgmap v14358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:01 smithi096 ceph-mon[32753]: pgmap v14358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:03 smithi194 ceph-mon[33017]: pgmap v14359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:03 smithi033 ceph-mon[27845]: pgmap v14359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:03 smithi096 ceph-mon[32753]: pgmap v14359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26222.098781 secs 2024-04-26T19:08:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26224.992555 secs 2024-04-26T19:08:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26222.098781 secs 2024-04-26T19:08:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26224.992555 secs 2024-04-26T19:08:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26222.098781 secs 2024-04-26T19:08:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26224.992555 secs 2024-04-26T19:08:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:05 smithi194 ceph-mon[33017]: pgmap v14360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:05 smithi033 ceph-mon[27845]: pgmap v14360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:05 smithi096 ceph-mon[32753]: pgmap v14360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:07.038+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:07.038+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:07.038+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:08:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:07 smithi194 ceph-mon[33017]: pgmap v14361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:07 smithi033 ceph-mon[27845]: pgmap v14361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:07 smithi096 ceph-mon[32753]: pgmap v14361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26227.098865 secs 2024-04-26T19:08:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26227.098865 secs 2024-04-26T19:08:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26227.098865 secs 2024-04-26T19:08:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:09 smithi194 ceph-mon[33017]: pgmap v14362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26229.992676 secs 2024-04-26T19:08:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:09 smithi033 ceph-mon[27845]: pgmap v14362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26229.992676 secs 2024-04-26T19:08:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:09 smithi096 ceph-mon[32753]: pgmap v14362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26229.992676 secs 2024-04-26T19:08:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:11 smithi194 ceph-mon[33017]: pgmap v14363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:11 smithi033 ceph-mon[27845]: pgmap v14363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:11 smithi096 ceph-mon[32753]: pgmap v14363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:13 smithi194 ceph-mon[33017]: pgmap v14364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26232.098950 secs 2024-04-26T19:08:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:13 smithi033 ceph-mon[27845]: pgmap v14364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26232.098950 secs 2024-04-26T19:08:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:13 smithi096 ceph-mon[32753]: pgmap v14364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26232.098950 secs 2024-04-26T19:08:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26234.992800 secs 2024-04-26T19:08:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26234.992800 secs 2024-04-26T19:08:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26234.992800 secs 2024-04-26T19:08:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:15 smithi194 ceph-mon[33017]: pgmap v14365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:15 smithi033 ceph-mon[27845]: pgmap v14365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:15 smithi096 ceph-mon[32753]: pgmap v14365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:17 smithi194 ceph-mon[33017]: pgmap v14366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:17 smithi033 ceph-mon[27845]: pgmap v14366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:17 smithi096 ceph-mon[32753]: pgmap v14366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26237.099006 secs 2024-04-26T19:08:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26237.099006 secs 2024-04-26T19:08:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26237.099006 secs 2024-04-26T19:08:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:19 smithi194 ceph-mon[33017]: pgmap v14367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26239.992923 secs 2024-04-26T19:08:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:19 smithi033 ceph-mon[27845]: pgmap v14367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26239.992923 secs 2024-04-26T19:08:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:19 smithi096 ceph-mon[32753]: pgmap v14367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26239.992923 secs 2024-04-26T19:08:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:21 smithi194 ceph-mon[33017]: pgmap v14368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:21 smithi033 ceph-mon[27845]: pgmap v14368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:21 smithi096 ceph-mon[32753]: pgmap v14368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:22.053+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:22.053+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:22.053+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:08:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:23 smithi033 ceph-mon[27845]: pgmap v14369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26242.099067 secs 2024-04-26T19:08:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:23 smithi096 ceph-mon[32753]: pgmap v14369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26242.099067 secs 2024-04-26T19:08:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:23 smithi194 ceph-mon[33017]: pgmap v14369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26242.099067 secs 2024-04-26T19:08:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26244.993063 secs 2024-04-26T19:08:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26244.993063 secs 2024-04-26T19:08:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26244.993063 secs 2024-04-26T19:08:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:25 smithi033 ceph-mon[27845]: pgmap v14370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:25 smithi096 ceph-mon[32753]: pgmap v14370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:25 smithi194 ceph-mon[33017]: pgmap v14370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:26.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:26.259+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:08:26.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:26.259+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:08:26.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:26.260+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 208 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:08:26.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:26.260+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 208 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:08:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:27 smithi033 ceph-mon[27845]: pgmap v14371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:27 smithi096 ceph-mon[32753]: pgmap v14371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:27 smithi194 ceph-mon[33017]: pgmap v14371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26247.099154 secs 2024-04-26T19:08:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26247.099154 secs 2024-04-26T19:08:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26247.099154 secs 2024-04-26T19:08:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:29 smithi033 ceph-mon[27845]: pgmap v14372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26249.993168 secs 2024-04-26T19:08:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:29 smithi096 ceph-mon[32753]: pgmap v14372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26249.993168 secs 2024-04-26T19:08:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:29 smithi194 ceph-mon[33017]: pgmap v14372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26249.993168 secs 2024-04-26T19:08:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:31 smithi033 ceph-mon[27845]: pgmap v14373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:31 smithi096 ceph-mon[32753]: pgmap v14373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:31 smithi194 ceph-mon[33017]: pgmap v14373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:33 smithi033 ceph-mon[27845]: pgmap v14374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26252.099240 secs 2024-04-26T19:08:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:33 smithi096 ceph-mon[32753]: pgmap v14374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26252.099240 secs 2024-04-26T19:08:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:33 smithi194 ceph-mon[33017]: pgmap v14374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26252.099240 secs 2024-04-26T19:08:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26254.993282 secs 2024-04-26T19:08:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26254.993282 secs 2024-04-26T19:08:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26254.993282 secs 2024-04-26T19:08:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:35 smithi033 ceph-mon[27845]: pgmap v14375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:35 smithi096 ceph-mon[32753]: pgmap v14375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:35 smithi194 ceph-mon[33017]: pgmap v14375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:37.068+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:37.068+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:37.068+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:08:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:37 smithi033 ceph-mon[27845]: pgmap v14376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:37 smithi096 ceph-mon[32753]: pgmap v14376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:37 smithi194 ceph-mon[33017]: pgmap v14376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26257.099303 secs 2024-04-26T19:08:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:08:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26257.099303 secs 2024-04-26T19:08:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:08:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26257.099303 secs 2024-04-26T19:08:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:08:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:39 smithi033 ceph-mon[27845]: pgmap v14377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26259.993345 secs 2024-04-26T19:08:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:39 smithi194 ceph-mon[33017]: pgmap v14377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26259.993345 secs 2024-04-26T19:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:39 smithi096 ceph-mon[32753]: pgmap v14377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26259.993345 secs 2024-04-26T19:08:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: pgmap v14378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: pgmap v14378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: pgmap v14378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:41.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:08:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:08:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:08:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:42 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:08:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:08:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:08:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:42 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:08:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:42 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:08:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:43 smithi194 ceph-mon[33017]: pgmap v14379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26262.099392 secs 2024-04-26T19:08:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:43 smithi033 ceph-mon[27845]: pgmap v14379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26262.099392 secs 2024-04-26T19:08:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:43 smithi096 ceph-mon[32753]: pgmap v14379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26262.099392 secs 2024-04-26T19:08:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26264.993454 secs 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:08:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26264.993454 secs 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:08:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:08:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26264.993454 secs 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:08:44.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:08:44.966 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:08:44.966 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:08:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:45 smithi033 ceph-mon[27845]: pgmap v14380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:45 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:08:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:45 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:08:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:45 smithi033 ceph-mon[27845]: pgmap v14381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:08:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:45 smithi033 ceph-mon[27845]: pgmap v14382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:08:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:45 smithi096 ceph-mon[32753]: pgmap v14380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:45 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:08:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:45 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:08:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:45 smithi096 ceph-mon[32753]: pgmap v14381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:08:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:45 smithi096 ceph-mon[32753]: pgmap v14382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:45 smithi194 ceph-mon[33017]: pgmap v14380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:45 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:45 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:45 smithi194 ceph-mon[33017]: pgmap v14381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:08:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:45 smithi194 ceph-mon[33017]: pgmap v14382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:08:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:47 smithi033 ceph-mon[27845]: pgmap v14383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:47 smithi096 ceph-mon[32753]: pgmap v14383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:47 smithi194 ceph-mon[33017]: pgmap v14383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26267.099479 secs 2024-04-26T19:08:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26267.099479 secs 2024-04-26T19:08:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26267.099479 secs 2024-04-26T19:08:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:49 smithi033 ceph-mon[27845]: pgmap v14384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:08:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26269.993580 secs 2024-04-26T19:08:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:49 smithi096 ceph-mon[32753]: pgmap v14384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:08:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26269.993580 secs 2024-04-26T19:08:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:49 smithi194 ceph-mon[33017]: pgmap v14384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:08:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26269.993580 secs 2024-04-26T19:08:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:51 smithi033 ceph-mon[27845]: pgmap v14385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:51 smithi096 ceph-mon[32753]: pgmap v14385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:52.080+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:52.080+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:08:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:08:52.080+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:08:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:51 smithi194 ceph-mon[33017]: pgmap v14385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:52 smithi033 ceph-mon[27845]: pgmap v14386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:52 smithi096 ceph-mon[32753]: pgmap v14386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:52 smithi194 ceph-mon[33017]: pgmap v14386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:08:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26272.099585 secs 2024-04-26T19:08:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26272.099585 secs 2024-04-26T19:08:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26272.099585 secs 2024-04-26T19:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26274.993730 secs 2024-04-26T19:08:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:54 smithi194 ceph-mon[33017]: pgmap v14387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26274.993730 secs 2024-04-26T19:08:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:54 smithi033 ceph-mon[27845]: pgmap v14387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:08:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26274.993730 secs 2024-04-26T19:08:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:54 smithi096 ceph-mon[32753]: pgmap v14387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:08:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:57 smithi194 ceph-mon[33017]: pgmap v14388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:57 smithi033 ceph-mon[27845]: pgmap v14388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:57 smithi096 ceph-mon[32753]: pgmap v14388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:08:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26277.099688 secs 2024-04-26T19:08:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:59 smithi194 ceph-mon[33017]: pgmap v14389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26279.993868 secs 2024-04-26T19:08:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26277.099688 secs 2024-04-26T19:08:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:59 smithi033 ceph-mon[27845]: pgmap v14389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26279.993868 secs 2024-04-26T19:08:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26277.099688 secs 2024-04-26T19:08:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:59 smithi096 ceph-mon[32753]: pgmap v14389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:08:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26279.993868 secs 2024-04-26T19:09:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:01 smithi194 ceph-mon[33017]: pgmap v14390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:01 smithi033 ceph-mon[27845]: pgmap v14390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:01 smithi096 ceph-mon[32753]: pgmap v14390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:03 smithi194 ceph-mon[33017]: pgmap v14391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:03 smithi033 ceph-mon[27845]: pgmap v14391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:03 smithi096 ceph-mon[32753]: pgmap v14391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26282.099790 secs 2024-04-26T19:09:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26284.993992 secs 2024-04-26T19:09:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26282.099790 secs 2024-04-26T19:09:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26284.993992 secs 2024-04-26T19:09:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26282.099790 secs 2024-04-26T19:09:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26284.993992 secs 2024-04-26T19:09:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:05 smithi194 ceph-mon[33017]: pgmap v14392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:05 smithi033 ceph-mon[27845]: pgmap v14392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:05 smithi096 ceph-mon[32753]: pgmap v14392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:07.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:07.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:07.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:09:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:07 smithi194 ceph-mon[33017]: pgmap v14393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:07 smithi033 ceph-mon[27845]: pgmap v14393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:07 smithi096 ceph-mon[32753]: pgmap v14393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26287.099891 secs 2024-04-26T19:09:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:09 smithi194 ceph-mon[33017]: pgmap v14394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26289.994162 secs 2024-04-26T19:09:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26287.099891 secs 2024-04-26T19:09:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:09 smithi033 ceph-mon[27845]: pgmap v14394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26289.994162 secs 2024-04-26T19:09:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26287.099891 secs 2024-04-26T19:09:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:09 smithi096 ceph-mon[32753]: pgmap v14394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26289.994162 secs 2024-04-26T19:09:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:11 smithi194 ceph-mon[33017]: pgmap v14395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:11 smithi033 ceph-mon[27845]: pgmap v14395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:11 smithi096 ceph-mon[32753]: pgmap v14395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:13 smithi194 ceph-mon[33017]: pgmap v14396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:13 smithi033 ceph-mon[27845]: pgmap v14396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:13 smithi096 ceph-mon[32753]: pgmap v14396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26292.099993 secs 2024-04-26T19:09:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26294.994289 secs 2024-04-26T19:09:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26292.099993 secs 2024-04-26T19:09:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26294.994289 secs 2024-04-26T19:09:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26292.099993 secs 2024-04-26T19:09:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26294.994289 secs 2024-04-26T19:09:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:15 smithi194 ceph-mon[33017]: pgmap v14397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:15 smithi033 ceph-mon[27845]: pgmap v14397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:15 smithi096 ceph-mon[32753]: pgmap v14397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:17 smithi033 ceph-mon[27845]: pgmap v14398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:17 smithi096 ceph-mon[32753]: pgmap v14398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:17 smithi194 ceph-mon[33017]: pgmap v14398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26297.100058 secs 2024-04-26T19:09:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:19 smithi033 ceph-mon[27845]: pgmap v14399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26299.994392 secs 2024-04-26T19:09:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26297.100058 secs 2024-04-26T19:09:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:19 smithi096 ceph-mon[32753]: pgmap v14399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26299.994392 secs 2024-04-26T19:09:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26297.100058 secs 2024-04-26T19:09:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:19 smithi194 ceph-mon[33017]: pgmap v14399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26299.994392 secs 2024-04-26T19:09:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:21 smithi033 ceph-mon[27845]: pgmap v14400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:21 smithi096 ceph-mon[32753]: pgmap v14400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:21 smithi194 ceph-mon[33017]: pgmap v14400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:22.105+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:22.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:22.106+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:22.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:22.106+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:09:23.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:22 smithi033 ceph-mon[27845]: pgmap v14401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:22 smithi194 ceph-mon[33017]: pgmap v14401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:22 smithi096 ceph-mon[32753]: pgmap v14401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26302.100145 secs 2024-04-26T19:09:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26302.100145 secs 2024-04-26T19:09:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26302.100145 secs 2024-04-26T19:09:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26304.994492 secs 2024-04-26T19:09:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:24 smithi194 ceph-mon[33017]: pgmap v14402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26304.994492 secs 2024-04-26T19:09:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:24 smithi033 ceph-mon[27845]: pgmap v14402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26304.994492 secs 2024-04-26T19:09:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:24 smithi096 ceph-mon[32753]: pgmap v14402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:27 smithi194 ceph-mon[33017]: pgmap v14403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:27 smithi033 ceph-mon[27845]: pgmap v14403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:27 smithi096 ceph-mon[32753]: pgmap v14403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26307.100219 secs 2024-04-26T19:09:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:29 smithi194 ceph-mon[33017]: pgmap v14404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26309.994583 secs 2024-04-26T19:09:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26307.100219 secs 2024-04-26T19:09:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:29 smithi033 ceph-mon[27845]: pgmap v14404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26309.994583 secs 2024-04-26T19:09:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26307.100219 secs 2024-04-26T19:09:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:29 smithi096 ceph-mon[32753]: pgmap v14404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26309.994583 secs 2024-04-26T19:09:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:31 smithi194 ceph-mon[33017]: pgmap v14405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:31 smithi033 ceph-mon[27845]: pgmap v14405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:31 smithi096 ceph-mon[32753]: pgmap v14405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:09:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:33 smithi194 ceph-mon[33017]: pgmap v14406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:33 smithi033 ceph-mon[27845]: pgmap v14406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:33 smithi096 ceph-mon[32753]: pgmap v14406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26312.100308 secs 2024-04-26T19:09:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26314.994713 secs 2024-04-26T19:09:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26312.100308 secs 2024-04-26T19:09:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26314.994713 secs 2024-04-26T19:09:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26312.100308 secs 2024-04-26T19:09:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26314.994713 secs 2024-04-26T19:09:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:35 smithi194 ceph-mon[33017]: pgmap v14407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:35 smithi033 ceph-mon[27845]: pgmap v14407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:35 smithi096 ceph-mon[32753]: pgmap v14407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:37.108+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:37.108+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:37.108+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:09:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:37 smithi194 ceph-mon[33017]: pgmap v14408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:09:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:37 smithi033 ceph-mon[27845]: pgmap v14408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:09:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:37 smithi096 ceph-mon[32753]: pgmap v14408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:09:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26317.100400 secs 2024-04-26T19:09:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:39 smithi194 ceph-mon[33017]: pgmap v14409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26319.994846 secs 2024-04-26T19:09:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26317.100400 secs 2024-04-26T19:09:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:39 smithi033 ceph-mon[27845]: pgmap v14409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26319.994846 secs 2024-04-26T19:09:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26317.100400 secs 2024-04-26T19:09:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:39 smithi096 ceph-mon[32753]: pgmap v14409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26319.994846 secs 2024-04-26T19:09:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:41 smithi194 ceph-mon[33017]: pgmap v14410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:09:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:41 smithi033 ceph-mon[27845]: pgmap v14410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:09:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:41 smithi096 ceph-mon[32753]: pgmap v14410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:09:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:43 smithi194 ceph-mon[33017]: pgmap v14411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:43 smithi033 ceph-mon[27845]: pgmap v14411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:43 smithi096 ceph-mon[32753]: pgmap v14411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26322.100463 secs 2024-04-26T19:09:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26324.994985 secs 2024-04-26T19:09:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26322.100463 secs 2024-04-26T19:09:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26324.994985 secs 2024-04-26T19:09:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26322.100463 secs 2024-04-26T19:09:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26324.994985 secs 2024-04-26T19:09:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:45 smithi033 ceph-mon[27845]: pgmap v14412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:09:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:45 smithi096 ceph-mon[32753]: pgmap v14412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:09:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:45 smithi194 ceph-mon[33017]: pgmap v14412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:09:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:09:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:09:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:09:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:09:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:09:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:09:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:09:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:09:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:09:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:09:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:47 smithi033 ceph-mon[27845]: pgmap v14413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T19:09:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:47 smithi033 ceph-mon[27845]: pgmap v14414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:47 smithi096 ceph-mon[32753]: pgmap v14413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T19:09:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:47 smithi096 ceph-mon[32753]: pgmap v14414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:47 smithi194 ceph-mon[33017]: pgmap v14413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T19:09:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:47 smithi194 ceph-mon[33017]: pgmap v14414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:49 smithi033 ceph-mon[27845]: pgmap v14415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26327.100523 secs 2024-04-26T19:09:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26329.995143 secs 2024-04-26T19:09:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:49 smithi096 ceph-mon[32753]: pgmap v14415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26327.100523 secs 2024-04-26T19:09:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26329.995143 secs 2024-04-26T19:09:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:49 smithi194 ceph-mon[33017]: pgmap v14415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26327.100523 secs 2024-04-26T19:09:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26329.995143 secs 2024-04-26T19:09:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:51 smithi033 ceph-mon[27845]: pgmap v14416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:51 smithi096 ceph-mon[32753]: pgmap v14416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:51 smithi194 ceph-mon[33017]: pgmap v14416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:52.123+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:52.123+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:09:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:09:52.123+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:09:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:53 smithi033 ceph-mon[27845]: pgmap v14417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:53 smithi096 ceph-mon[32753]: pgmap v14417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:53 smithi194 ceph-mon[33017]: pgmap v14417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26332.100606 secs 2024-04-26T19:09:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26334.995261 secs 2024-04-26T19:09:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26332.100606 secs 2024-04-26T19:09:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26334.995261 secs 2024-04-26T19:09:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26332.100606 secs 2024-04-26T19:09:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26334.995261 secs 2024-04-26T19:09:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:55 smithi033 ceph-mon[27845]: pgmap v14418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:55 smithi096 ceph-mon[32753]: pgmap v14418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:55 smithi194 ceph-mon[33017]: pgmap v14418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:09:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:57 smithi033 ceph-mon[27845]: pgmap v14419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:09:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:57 smithi096 ceph-mon[32753]: pgmap v14419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:09:57.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:57 smithi194 ceph-mon[33017]: pgmap v14419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:09:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:59 smithi033 ceph-mon[27845]: pgmap v14420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26337.100691 secs 2024-04-26T19:09:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26339.995346 secs 2024-04-26T19:09:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:59 smithi096 ceph-mon[32753]: pgmap v14420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26337.100691 secs 2024-04-26T19:09:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26339.995346 secs 2024-04-26T19:09:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:59 smithi194 ceph-mon[33017]: pgmap v14420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:09:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26337.100691 secs 2024-04-26T19:09:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26339.995346 secs 2024-04-26T19:10:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:10:00.448 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:10:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:10:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:01 smithi033 ceph-mon[27845]: pgmap v14421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:01 smithi096 ceph-mon[32753]: pgmap v14421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:01 smithi194 ceph-mon[33017]: pgmap v14421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:03 smithi194 ceph-mon[33017]: pgmap v14422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:03 smithi033 ceph-mon[27845]: pgmap v14422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:03 smithi096 ceph-mon[32753]: pgmap v14422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26342.100749 secs 2024-04-26T19:10:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26344.995429 secs 2024-04-26T19:10:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26342.100749 secs 2024-04-26T19:10:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26344.995429 secs 2024-04-26T19:10:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26342.100749 secs 2024-04-26T19:10:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26344.995429 secs 2024-04-26T19:10:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:05 smithi194 ceph-mon[33017]: pgmap v14423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:05 smithi033 ceph-mon[27845]: pgmap v14423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:05 smithi096 ceph-mon[32753]: pgmap v14423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:07.138+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:07.138+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:10:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:07.138+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:07 smithi194 ceph-mon[33017]: pgmap v14424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:07 smithi033 ceph-mon[27845]: pgmap v14424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:07 smithi096 ceph-mon[32753]: pgmap v14424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:09 smithi194 ceph-mon[33017]: pgmap v14425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26347.100834 secs 2024-04-26T19:10:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26349.995511 secs 2024-04-26T19:10:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:09 smithi033 ceph-mon[27845]: pgmap v14425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26347.100834 secs 2024-04-26T19:10:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26349.995511 secs 2024-04-26T19:10:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:09 smithi096 ceph-mon[32753]: pgmap v14425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26347.100834 secs 2024-04-26T19:10:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26349.995511 secs 2024-04-26T19:10:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:11 smithi194 ceph-mon[33017]: pgmap v14426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:11 smithi033 ceph-mon[27845]: pgmap v14426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:11 smithi096 ceph-mon[32753]: pgmap v14426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:13 smithi194 ceph-mon[33017]: pgmap v14427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:13 smithi033 ceph-mon[27845]: pgmap v14427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:13 smithi096 ceph-mon[32753]: pgmap v14427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26352.100890 secs 2024-04-26T19:10:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26354.995629 secs 2024-04-26T19:10:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26352.100890 secs 2024-04-26T19:10:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26354.995629 secs 2024-04-26T19:10:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26352.100890 secs 2024-04-26T19:10:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26354.995629 secs 2024-04-26T19:10:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:15 smithi194 ceph-mon[33017]: pgmap v14428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:15 smithi033 ceph-mon[27845]: pgmap v14428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:15 smithi096 ceph-mon[32753]: pgmap v14428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:17 smithi194 ceph-mon[33017]: pgmap v14429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:17 smithi033 ceph-mon[27845]: pgmap v14429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:17 smithi096 ceph-mon[32753]: pgmap v14429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:19 smithi194 ceph-mon[33017]: pgmap v14430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26357.100974 secs 2024-04-26T19:10:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26359.995787 secs 2024-04-26T19:10:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:19 smithi033 ceph-mon[27845]: pgmap v14430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26357.100974 secs 2024-04-26T19:10:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26359.995787 secs 2024-04-26T19:10:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:19 smithi096 ceph-mon[32753]: pgmap v14430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26357.100974 secs 2024-04-26T19:10:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26359.995787 secs 2024-04-26T19:10:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:21 smithi194 ceph-mon[33017]: pgmap v14431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:21 smithi033 ceph-mon[27845]: pgmap v14431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:21 smithi096 ceph-mon[32753]: pgmap v14431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:22.150+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:22.150+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:10:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:22.150+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:23 smithi194 ceph-mon[33017]: pgmap v14432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:23 smithi033 ceph-mon[27845]: pgmap v14432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:23 smithi096 ceph-mon[32753]: pgmap v14432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26362.101060 secs 2024-04-26T19:10:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26364.995907 secs 2024-04-26T19:10:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26362.101060 secs 2024-04-26T19:10:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26364.995907 secs 2024-04-26T19:10:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26362.101060 secs 2024-04-26T19:10:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26364.995907 secs 2024-04-26T19:10:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:25 smithi194 ceph-mon[33017]: pgmap v14433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:25 smithi033 ceph-mon[27845]: pgmap v14433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:25 smithi096 ceph-mon[32753]: pgmap v14433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:27 smithi194 ceph-mon[33017]: pgmap v14434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:27 smithi033 ceph-mon[27845]: pgmap v14434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:27 smithi096 ceph-mon[32753]: pgmap v14434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:29 smithi194 ceph-mon[33017]: pgmap v14435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26367.101144 secs 2024-04-26T19:10:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26369.996010 secs 2024-04-26T19:10:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:29 smithi033 ceph-mon[27845]: pgmap v14435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26367.101144 secs 2024-04-26T19:10:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26369.996010 secs 2024-04-26T19:10:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:29 smithi096 ceph-mon[32753]: pgmap v14435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26367.101144 secs 2024-04-26T19:10:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26369.996010 secs 2024-04-26T19:10:31.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:31 smithi033 ceph-mon[27845]: pgmap v14436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:31 smithi194 ceph-mon[33017]: pgmap v14436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:31 smithi096 ceph-mon[32753]: pgmap v14436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:33 smithi194 ceph-mon[33017]: pgmap v14437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:33 smithi033 ceph-mon[27845]: pgmap v14437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:33 smithi096 ceph-mon[32753]: pgmap v14437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26372.101232 secs 2024-04-26T19:10:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26374.996167 secs 2024-04-26T19:10:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26372.101232 secs 2024-04-26T19:10:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26374.996167 secs 2024-04-26T19:10:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26372.101232 secs 2024-04-26T19:10:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26374.996167 secs 2024-04-26T19:10:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:35 smithi194 ceph-mon[33017]: pgmap v14438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:35 smithi033 ceph-mon[27845]: pgmap v14438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:35 smithi096 ceph-mon[32753]: pgmap v14438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:37.158+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:37.158+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:10:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:37.158+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:37 smithi194 ceph-mon[33017]: pgmap v14439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:37 smithi033 ceph-mon[27845]: pgmap v14439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:37 smithi096 ceph-mon[32753]: pgmap v14439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26377.101319 secs 2024-04-26T19:10:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26377.101319 secs 2024-04-26T19:10:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26377.101319 secs 2024-04-26T19:10:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:39 smithi194 ceph-mon[33017]: pgmap v14440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26379.996299 secs 2024-04-26T19:10:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:39 smithi033 ceph-mon[27845]: pgmap v14440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26379.996299 secs 2024-04-26T19:10:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:39 smithi096 ceph-mon[32753]: pgmap v14440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26379.996299 secs 2024-04-26T19:10:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:41 smithi194 ceph-mon[33017]: pgmap v14441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:41 smithi033 ceph-mon[27845]: pgmap v14441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:41 smithi096 ceph-mon[32753]: pgmap v14441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:43 smithi194 ceph-mon[33017]: pgmap v14442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26382.101407 secs 2024-04-26T19:10:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:43 smithi033 ceph-mon[27845]: pgmap v14442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26382.101407 secs 2024-04-26T19:10:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:43 smithi096 ceph-mon[32753]: pgmap v14442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26382.101407 secs 2024-04-26T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26384.996430 secs 2024-04-26T19:10:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26384.996430 secs 2024-04-26T19:10:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26384.996430 secs 2024-04-26T19:10:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:45 smithi194 ceph-mon[33017]: pgmap v14443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:45 smithi033 ceph-mon[27845]: pgmap v14443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:45 smithi096 ceph-mon[32753]: pgmap v14443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:10:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:10:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:10:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:47 smithi033 ceph-mon[27845]: pgmap v14444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:10:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:10:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:10:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:47 smithi096 ceph-mon[32753]: pgmap v14444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:10:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:10:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:10:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:47 smithi194 ceph-mon[33017]: pgmap v14444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:10:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:10:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:10:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:10:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:48 smithi033 ceph-mon[27845]: pgmap v14445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:48 smithi033 ceph-mon[27845]: pgmap v14446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26387.101487 secs 2024-04-26T19:10:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:48 smithi096 ceph-mon[32753]: pgmap v14445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:48 smithi096 ceph-mon[32753]: pgmap v14446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26387.101487 secs 2024-04-26T19:10:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:48 smithi194 ceph-mon[33017]: pgmap v14445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:10:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:48 smithi194 ceph-mon[33017]: pgmap v14446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26387.101487 secs 2024-04-26T19:10:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26389.996562 secs 2024-04-26T19:10:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26389.996562 secs 2024-04-26T19:10:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26389.996562 secs 2024-04-26T19:10:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:50 smithi033 ceph-mon[27845]: pgmap v14447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:10:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:50 smithi096 ceph-mon[32753]: pgmap v14447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:10:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:50 smithi194 ceph-mon[33017]: pgmap v14447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:10:52.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:52.162+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:52.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:52.162+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:10:52.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:10:52.162+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:10:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:52 smithi033 ceph-mon[27845]: pgmap v14448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:52 smithi096 ceph-mon[32753]: pgmap v14448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:52 smithi194 ceph-mon[33017]: pgmap v14448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26392.101586 secs 2024-04-26T19:10:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26392.101586 secs 2024-04-26T19:10:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26392.101586 secs 2024-04-26T19:10:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:54 smithi033 ceph-mon[27845]: pgmap v14449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26394.996658 secs 2024-04-26T19:10:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:54 smithi096 ceph-mon[32753]: pgmap v14449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26394.996658 secs 2024-04-26T19:10:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:54 smithi194 ceph-mon[33017]: pgmap v14449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26394.996658 secs 2024-04-26T19:10:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:56 smithi033 ceph-mon[27845]: pgmap v14450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:56 smithi096 ceph-mon[32753]: pgmap v14450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:56 smithi194 ceph-mon[33017]: pgmap v14450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:10:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:58 smithi096 ceph-mon[32753]: pgmap v14451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:10:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26397.101687 secs 2024-04-26T19:10:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:58 smithi194 ceph-mon[33017]: pgmap v14451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:10:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26397.101687 secs 2024-04-26T19:10:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:58 smithi033 ceph-mon[27845]: pgmap v14451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:10:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26397.101687 secs 2024-04-26T19:10:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:10:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26399.996781 secs 2024-04-26T19:10:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:10:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26399.996781 secs 2024-04-26T19:10:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:10:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26399.996781 secs 2024-04-26T19:11:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:00 smithi194 ceph-mon[33017]: pgmap v14452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:00 smithi033 ceph-mon[27845]: pgmap v14452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:00 smithi096 ceph-mon[32753]: pgmap v14452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:02 smithi194 ceph-mon[33017]: pgmap v14453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:02 smithi033 ceph-mon[27845]: pgmap v14453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:02 smithi096 ceph-mon[32753]: pgmap v14453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26402.101786 secs 2024-04-26T19:11:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26402.101786 secs 2024-04-26T19:11:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26402.101786 secs 2024-04-26T19:11:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:04 smithi194 ceph-mon[33017]: pgmap v14454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26404.996899 secs 2024-04-26T19:11:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:04 smithi033 ceph-mon[27845]: pgmap v14454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26404.996899 secs 2024-04-26T19:11:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:04 smithi096 ceph-mon[32753]: pgmap v14454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26404.996899 secs 2024-04-26T19:11:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:06 smithi194 ceph-mon[33017]: pgmap v14455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:06 smithi033 ceph-mon[27845]: pgmap v14455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:06 smithi096 ceph-mon[32753]: pgmap v14455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:07.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:07.174+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:07.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:07.174+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:07.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:07.174+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:11:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:08 smithi194 ceph-mon[33017]: pgmap v14456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26407.101888 secs 2024-04-26T19:11:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:08 smithi033 ceph-mon[27845]: pgmap v14456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26407.101888 secs 2024-04-26T19:11:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:08 smithi096 ceph-mon[32753]: pgmap v14456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26407.101888 secs 2024-04-26T19:11:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26409.997051 secs 2024-04-26T19:11:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26409.997051 secs 2024-04-26T19:11:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26409.997051 secs 2024-04-26T19:11:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:10 smithi194 ceph-mon[33017]: pgmap v14457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:10 smithi033 ceph-mon[27845]: pgmap v14457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:10 smithi096 ceph-mon[32753]: pgmap v14457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:12 smithi194 ceph-mon[33017]: pgmap v14458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:12 smithi033 ceph-mon[27845]: pgmap v14458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:12 smithi096 ceph-mon[32753]: pgmap v14458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:13.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26412.101992 secs 2024-04-26T19:11:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26412.101992 secs 2024-04-26T19:11:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26412.101992 secs 2024-04-26T19:11:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:14 smithi194 ceph-mon[33017]: pgmap v14459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26414.997145 secs 2024-04-26T19:11:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:14 smithi033 ceph-mon[27845]: pgmap v14459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26414.997145 secs 2024-04-26T19:11:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:14 smithi096 ceph-mon[32753]: pgmap v14459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26414.997145 secs 2024-04-26T19:11:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:16 smithi194 ceph-mon[33017]: pgmap v14460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:16 smithi033 ceph-mon[27845]: pgmap v14460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:16 smithi096 ceph-mon[32753]: pgmap v14460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:18 smithi033 ceph-mon[27845]: pgmap v14461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26417.102054 secs 2024-04-26T19:11:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:18 smithi194 ceph-mon[33017]: pgmap v14461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26417.102054 secs 2024-04-26T19:11:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:18 smithi096 ceph-mon[32753]: pgmap v14461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26417.102054 secs 2024-04-26T19:11:19.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26419.997267 secs 2024-04-26T19:11:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26419.997267 secs 2024-04-26T19:11:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26419.997267 secs 2024-04-26T19:11:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:20 smithi194 ceph-mon[33017]: pgmap v14462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:20 smithi033 ceph-mon[27845]: pgmap v14462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:20 smithi096 ceph-mon[32753]: pgmap v14462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T19:11:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T19:11:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feef000a010 tx=0x7feef0022040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T19:11:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feef000a010 tx=0x7feef0022040 comp rx=0 tx=0).stop 2024-04-26T19:11:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T19:11:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b36f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T19:11:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b36f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:11:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.788+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:11:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.789+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:11:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:21.791+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b3120 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef004f040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:11:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:22.175+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:22.175+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:11:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:22.175+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:22 smithi194 ceph-mon[33017]: pgmap v14463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:22 smithi033 ceph-mon[27845]: pgmap v14463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:22 smithi096 ceph-mon[32753]: pgmap v14463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:23.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26422.102156 secs 2024-04-26T19:11:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26422.102156 secs 2024-04-26T19:11:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26422.102156 secs 2024-04-26T19:11:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:24 smithi194 ceph-mon[33017]: pgmap v14464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26424.997380 secs 2024-04-26T19:11:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:24 smithi033 ceph-mon[27845]: pgmap v14464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26424.997380 secs 2024-04-26T19:11:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:24 smithi096 ceph-mon[32753]: pgmap v14464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26424.997380 secs 2024-04-26T19:11:26.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:26.289+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:11:26.289 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:26.289+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:11:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:26.290+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 209 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T19:11:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:26.290+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 209 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:11:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:26 smithi194 ceph-mon[33017]: pgmap v14465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:26 smithi033 ceph-mon[27845]: pgmap v14465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:26 smithi096 ceph-mon[32753]: pgmap v14465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:28 smithi194 ceph-mon[33017]: pgmap v14466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26427.102244 secs 2024-04-26T19:11:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:28 smithi033 ceph-mon[27845]: pgmap v14466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26427.102244 secs 2024-04-26T19:11:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:28 smithi096 ceph-mon[32753]: pgmap v14466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26427.102244 secs 2024-04-26T19:11:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26429.997474 secs 2024-04-26T19:11:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26429.997474 secs 2024-04-26T19:11:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26429.997474 secs 2024-04-26T19:11:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:30 smithi194 ceph-mon[33017]: pgmap v14467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:30 smithi033 ceph-mon[27845]: pgmap v14467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:30 smithi096 ceph-mon[32753]: pgmap v14467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:32 smithi194 ceph-mon[33017]: pgmap v14468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:32 smithi033 ceph-mon[27845]: pgmap v14468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:32 smithi096 ceph-mon[32753]: pgmap v14468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26432.102345 secs 2024-04-26T19:11:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26432.102345 secs 2024-04-26T19:11:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26432.102345 secs 2024-04-26T19:11:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:34 smithi194 ceph-mon[33017]: pgmap v14469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26434.997587 secs 2024-04-26T19:11:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:34 smithi033 ceph-mon[27845]: pgmap v14469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26434.997587 secs 2024-04-26T19:11:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:34 smithi096 ceph-mon[32753]: pgmap v14469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26434.997587 secs 2024-04-26T19:11:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:36 smithi194 ceph-mon[33017]: pgmap v14470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:36 smithi033 ceph-mon[27845]: pgmap v14470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:36 smithi096 ceph-mon[32753]: pgmap v14470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:37.182+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:37.182+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:37.182+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:11:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:38 smithi194 ceph-mon[33017]: pgmap v14471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:38 smithi033 ceph-mon[27845]: pgmap v14471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:38 smithi096 ceph-mon[32753]: pgmap v14471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26437.102434 secs 2024-04-26T19:11:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26437.102434 secs 2024-04-26T19:11:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26437.102434 secs 2024-04-26T19:11:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26439.997710 secs 2024-04-26T19:11:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:40 smithi194 ceph-mon[33017]: pgmap v14472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26439.997710 secs 2024-04-26T19:11:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:40 smithi033 ceph-mon[27845]: pgmap v14472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26439.997710 secs 2024-04-26T19:11:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:40 smithi096 ceph-mon[32753]: pgmap v14472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:42 smithi033 ceph-mon[27845]: pgmap v14473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:43.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:42 smithi194 ceph-mon[33017]: pgmap v14473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:43.447 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:42 smithi096 ceph-mon[32753]: pgmap v14473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:43.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26442.102522 secs 2024-04-26T19:11:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26442.102522 secs 2024-04-26T19:11:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26442.102522 secs 2024-04-26T19:11:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:44 smithi194 ceph-mon[33017]: pgmap v14474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26444.997855 secs 2024-04-26T19:11:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:44 smithi033 ceph-mon[27845]: pgmap v14474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26444.997855 secs 2024-04-26T19:11:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:44 smithi096 ceph-mon[32753]: pgmap v14474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26444.997855 secs 2024-04-26T19:11:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:46 smithi194 ceph-mon[33017]: pgmap v14475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:46 smithi033 ceph-mon[27845]: pgmap v14475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:46 smithi096 ceph-mon[32753]: pgmap v14475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:11:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:48 smithi194 ceph-mon[33017]: pgmap v14476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:11:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26447.102616 secs 2024-04-26T19:11:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:48 smithi033 ceph-mon[27845]: pgmap v14476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:11:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26447.102616 secs 2024-04-26T19:11:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:48 smithi096 ceph-mon[32753]: pgmap v14476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:11:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26447.102616 secs 2024-04-26T19:11:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26449.997979 secs 2024-04-26T19:11:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:11:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:11:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:11:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26449.997979 secs 2024-04-26T19:11:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:11:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:11:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:11:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26449.997979 secs 2024-04-26T19:11:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:11:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:11:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:11:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:50 smithi194 ceph-mon[33017]: pgmap v14477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:50 smithi194 ceph-mon[33017]: pgmap v14478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:11:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:50 smithi033 ceph-mon[27845]: pgmap v14477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:50 smithi033 ceph-mon[27845]: pgmap v14478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:11:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:50 smithi096 ceph-mon[32753]: pgmap v14477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:11:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:50 smithi096 ceph-mon[32753]: pgmap v14478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:11:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:52.191+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:52.191+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:11:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:11:52.191+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:11:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:52 smithi194 ceph-mon[33017]: pgmap v14479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:52 smithi033 ceph-mon[27845]: pgmap v14479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:53.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:52 smithi096 ceph-mon[32753]: pgmap v14479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:53.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26452.102716 secs 2024-04-26T19:11:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26452.102716 secs 2024-04-26T19:11:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26452.102716 secs 2024-04-26T19:11:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:54 smithi194 ceph-mon[33017]: pgmap v14480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26454.998138 secs 2024-04-26T19:11:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:54 smithi033 ceph-mon[27845]: pgmap v14480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26454.998138 secs 2024-04-26T19:11:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:54 smithi096 ceph-mon[32753]: pgmap v14480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26454.998138 secs 2024-04-26T19:11:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:56 smithi194 ceph-mon[33017]: pgmap v14481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:56 smithi033 ceph-mon[27845]: pgmap v14481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:56 smithi096 ceph-mon[32753]: pgmap v14481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:58 smithi194 ceph-mon[33017]: pgmap v14482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26457.102818 secs 2024-04-26T19:11:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:58 smithi033 ceph-mon[27845]: pgmap v14482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26457.102818 secs 2024-04-26T19:11:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:58 smithi096 ceph-mon[32753]: pgmap v14482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:11:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26457.102818 secs 2024-04-26T19:11:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26459.998267 secs 2024-04-26T19:11:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26459.998267 secs 2024-04-26T19:11:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26459.998267 secs 2024-04-26T19:12:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:00 smithi194 ceph-mon[33017]: pgmap v14483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:12:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:00 smithi033 ceph-mon[27845]: pgmap v14483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:12:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:00 smithi096 ceph-mon[32753]: pgmap v14483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:12:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:02 smithi194 ceph-mon[33017]: pgmap v14484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:02 smithi033 ceph-mon[27845]: pgmap v14484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:02 smithi096 ceph-mon[32753]: pgmap v14484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26462.102878 secs 2024-04-26T19:12:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26462.102878 secs 2024-04-26T19:12:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26462.102878 secs 2024-04-26T19:12:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:04 smithi194 ceph-mon[33017]: pgmap v14485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26464.998353 secs 2024-04-26T19:12:04.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:04 smithi033 ceph-mon[27845]: pgmap v14485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:04.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26464.998353 secs 2024-04-26T19:12:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:04 smithi096 ceph-mon[32753]: pgmap v14485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26464.998353 secs 2024-04-26T19:12:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:06 smithi194 ceph-mon[33017]: pgmap v14486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:06 smithi033 ceph-mon[27845]: pgmap v14486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:06 smithi096 ceph-mon[32753]: pgmap v14486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:07.195+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:07.195+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:07.195+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:12:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:08 smithi194 ceph-mon[33017]: pgmap v14487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26467.102969 secs 2024-04-26T19:12:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:08 smithi033 ceph-mon[27845]: pgmap v14487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26467.102969 secs 2024-04-26T19:12:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:08 smithi096 ceph-mon[32753]: pgmap v14487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26467.102969 secs 2024-04-26T19:12:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26469.998491 secs 2024-04-26T19:12:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26469.998491 secs 2024-04-26T19:12:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26469.998491 secs 2024-04-26T19:12:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:10 smithi194 ceph-mon[33017]: pgmap v14488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:10 smithi033 ceph-mon[27845]: pgmap v14488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:10 smithi096 ceph-mon[32753]: pgmap v14488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:12 smithi194 ceph-mon[33017]: pgmap v14489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:12 smithi033 ceph-mon[27845]: pgmap v14489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:12 smithi096 ceph-mon[32753]: pgmap v14489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26472.103049 secs 2024-04-26T19:12:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26472.103049 secs 2024-04-26T19:12:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26472.103049 secs 2024-04-26T19:12:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:14 smithi194 ceph-mon[33017]: pgmap v14490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26474.998560 secs 2024-04-26T19:12:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:14 smithi033 ceph-mon[27845]: pgmap v14490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26474.998560 secs 2024-04-26T19:12:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:14 smithi096 ceph-mon[32753]: pgmap v14490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26474.998560 secs 2024-04-26T19:12:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:16 smithi194 ceph-mon[33017]: pgmap v14491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:16 smithi033 ceph-mon[27845]: pgmap v14491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:16 smithi096 ceph-mon[32753]: pgmap v14491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:18 smithi194 ceph-mon[33017]: pgmap v14492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26477.103153 secs 2024-04-26T19:12:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:18 smithi033 ceph-mon[27845]: pgmap v14492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26477.103153 secs 2024-04-26T19:12:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:18 smithi096 ceph-mon[32753]: pgmap v14492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26477.103153 secs 2024-04-26T19:12:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26479.998688 secs 2024-04-26T19:12:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26479.998688 secs 2024-04-26T19:12:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26479.998688 secs 2024-04-26T19:12:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:20 smithi194 ceph-mon[33017]: pgmap v14493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:20 smithi033 ceph-mon[27845]: pgmap v14493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:20 smithi096 ceph-mon[32753]: pgmap v14493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:22.205+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:22.205+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:22.205+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:12:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:22 smithi194 ceph-mon[33017]: pgmap v14494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:22 smithi033 ceph-mon[27845]: pgmap v14494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:22 smithi096 ceph-mon[32753]: pgmap v14494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26482.103195 secs 2024-04-26T19:12:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26482.103195 secs 2024-04-26T19:12:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26482.103195 secs 2024-04-26T19:12:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:24 smithi033 ceph-mon[27845]: pgmap v14495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26484.998784 secs 2024-04-26T19:12:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:24 smithi096 ceph-mon[32753]: pgmap v14495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26484.998784 secs 2024-04-26T19:12:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:24 smithi194 ceph-mon[33017]: pgmap v14495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26484.998784 secs 2024-04-26T19:12:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:26 smithi033 ceph-mon[27845]: pgmap v14496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:26 smithi096 ceph-mon[32753]: pgmap v14496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:26 smithi194 ceph-mon[33017]: pgmap v14496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:28 smithi033 ceph-mon[27845]: pgmap v14497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26487.103284 secs 2024-04-26T19:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:28 smithi096 ceph-mon[32753]: pgmap v14497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26487.103284 secs 2024-04-26T19:12:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:28 smithi194 ceph-mon[33017]: pgmap v14497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26487.103284 secs 2024-04-26T19:12:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26489.998929 secs 2024-04-26T19:12:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26489.998929 secs 2024-04-26T19:12:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26489.998929 secs 2024-04-26T19:12:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:30 smithi033 ceph-mon[27845]: pgmap v14498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:30 smithi096 ceph-mon[32753]: pgmap v14498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:30 smithi194 ceph-mon[33017]: pgmap v14498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:32 smithi033 ceph-mon[27845]: pgmap v14499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:32 smithi096 ceph-mon[32753]: pgmap v14499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:32 smithi194 ceph-mon[33017]: pgmap v14499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26492.103367 secs 2024-04-26T19:12:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26492.103367 secs 2024-04-26T19:12:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26492.103367 secs 2024-04-26T19:12:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:34 smithi033 ceph-mon[27845]: pgmap v14500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26494.999074 secs 2024-04-26T19:12:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:34 smithi096 ceph-mon[32753]: pgmap v14500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26494.999074 secs 2024-04-26T19:12:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:34 smithi194 ceph-mon[33017]: pgmap v14500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26494.999074 secs 2024-04-26T19:12:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:36 smithi033 ceph-mon[27845]: pgmap v14501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:36 smithi096 ceph-mon[32753]: pgmap v14501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:36 smithi194 ceph-mon[33017]: pgmap v14501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:12:37.218 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:37.218+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:37.218 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:37.218+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:37.218+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:38 smithi033 ceph-mon[27845]: pgmap v14502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26497.103452 secs 2024-04-26T19:12:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:38 smithi096 ceph-mon[32753]: pgmap v14502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26497.103452 secs 2024-04-26T19:12:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:38 smithi194 ceph-mon[33017]: pgmap v14502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26497.103452 secs 2024-04-26T19:12:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26499.999199 secs 2024-04-26T19:12:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26499.999199 secs 2024-04-26T19:12:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26499.999199 secs 2024-04-26T19:12:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:40 smithi033 ceph-mon[27845]: pgmap v14503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:40 smithi096 ceph-mon[32753]: pgmap v14503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:40 smithi194 ceph-mon[33017]: pgmap v14503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:12:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:42 smithi096 ceph-mon[32753]: pgmap v14504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:12:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:42 smithi194 ceph-mon[33017]: pgmap v14504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:12:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:42 smithi033 ceph-mon[27845]: pgmap v14504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:12:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26502.103534 secs 2024-04-26T19:12:44.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26502.103534 secs 2024-04-26T19:12:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26502.103534 secs 2024-04-26T19:12:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:44 smithi096 ceph-mon[32753]: pgmap v14505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26504.999314 secs 2024-04-26T19:12:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:44 smithi194 ceph-mon[33017]: pgmap v14505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26504.999314 secs 2024-04-26T19:12:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:44 smithi033 ceph-mon[27845]: pgmap v14505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26504.999314 secs 2024-04-26T19:12:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:46 smithi194 ceph-mon[33017]: pgmap v14506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:12:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:46 smithi033 ceph-mon[27845]: pgmap v14506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:12:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:46 smithi096 ceph-mon[32753]: pgmap v14506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:12:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:47 smithi194 ceph-mon[33017]: pgmap v14507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:47 smithi033 ceph-mon[27845]: pgmap v14507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:47 smithi096 ceph-mon[32753]: pgmap v14507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:49.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26507.103623 secs 2024-04-26T19:12:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26507.103623 secs 2024-04-26T19:12:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26507.103623 secs 2024-04-26T19:12:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26509.999462 secs 2024-04-26T19:12:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:49 smithi194 ceph-mon[33017]: pgmap v14508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:12:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26509.999462 secs 2024-04-26T19:12:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:49 smithi033 ceph-mon[27845]: pgmap v14508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:12:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26509.999462 secs 2024-04-26T19:12:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:49 smithi096 ceph-mon[32753]: pgmap v14508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:12:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:12:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:12:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:12:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:12:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:12:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:12:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:12:52.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:51 smithi033 ceph-mon[27845]: pgmap v14509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T19:12:52.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:51 smithi033 ceph-mon[27845]: pgmap v14510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:52.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:12:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:51 smithi194 ceph-mon[33017]: pgmap v14509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T19:12:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:51 smithi194 ceph-mon[33017]: pgmap v14510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:12:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:51 smithi096 ceph-mon[32753]: pgmap v14509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T19:12:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:51 smithi096 ceph-mon[32753]: pgmap v14510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:12:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:52.227+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:52.227+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:12:52.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:12:52.227+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:12:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:53 smithi194 ceph-mon[33017]: pgmap v14511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26512.103707 secs 2024-04-26T19:12:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:53 smithi033 ceph-mon[27845]: pgmap v14511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26512.103707 secs 2024-04-26T19:12:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:53 smithi096 ceph-mon[32753]: pgmap v14511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26512.103707 secs 2024-04-26T19:12:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26514.999538 secs 2024-04-26T19:12:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26514.999538 secs 2024-04-26T19:12:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26514.999538 secs 2024-04-26T19:12:56.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:55 smithi194 ceph-mon[33017]: pgmap v14512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:55 smithi033 ceph-mon[27845]: pgmap v14512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:56.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:55 smithi096 ceph-mon[32753]: pgmap v14512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:58 smithi194 ceph-mon[33017]: pgmap v14513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:58 smithi033 ceph-mon[27845]: pgmap v14513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:58 smithi096 ceph-mon[32753]: pgmap v14513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:12:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:12:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26517.103788 secs 2024-04-26T19:12:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:12:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26517.103788 secs 2024-04-26T19:12:59.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:12:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26517.103788 secs 2024-04-26T19:13:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26519.999671 secs 2024-04-26T19:13:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:00 smithi194 ceph-mon[33017]: pgmap v14514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26519.999671 secs 2024-04-26T19:13:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:00 smithi033 ceph-mon[27845]: pgmap v14514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26519.999671 secs 2024-04-26T19:13:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:00 smithi096 ceph-mon[32753]: pgmap v14514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:02 smithi194 ceph-mon[33017]: pgmap v14515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:13:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:02 smithi033 ceph-mon[27845]: pgmap v14515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:13:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:02 smithi096 ceph-mon[32753]: pgmap v14515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:13:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:04 smithi194 ceph-mon[33017]: pgmap v14516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26522.103867 secs 2024-04-26T19:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:04 smithi033 ceph-mon[27845]: pgmap v14516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26522.103867 secs 2024-04-26T19:13:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:04 smithi096 ceph-mon[32753]: pgmap v14516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26522.103867 secs 2024-04-26T19:13:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26524.999790 secs 2024-04-26T19:13:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26524.999790 secs 2024-04-26T19:13:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26524.999790 secs 2024-04-26T19:13:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:06 smithi194 ceph-mon[33017]: pgmap v14517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:06 smithi033 ceph-mon[27845]: pgmap v14517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:06 smithi096 ceph-mon[32753]: pgmap v14517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:07.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:07.239+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:07.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:07.239+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:07.240 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:07.239+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:13:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:08 smithi194 ceph-mon[33017]: pgmap v14518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:08 smithi033 ceph-mon[27845]: pgmap v14518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:08 smithi096 ceph-mon[32753]: pgmap v14518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26527.103951 secs 2024-04-26T19:13:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26529.999941 secs 2024-04-26T19:13:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26527.103951 secs 2024-04-26T19:13:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26529.999941 secs 2024-04-26T19:13:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26527.103951 secs 2024-04-26T19:13:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26529.999941 secs 2024-04-26T19:13:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:10 smithi194 ceph-mon[33017]: pgmap v14519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:10 smithi033 ceph-mon[27845]: pgmap v14519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:10 smithi096 ceph-mon[32753]: pgmap v14519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:12 smithi194 ceph-mon[33017]: pgmap v14520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:12 smithi033 ceph-mon[27845]: pgmap v14520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:12 smithi096 ceph-mon[32753]: pgmap v14520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:14 smithi194 ceph-mon[33017]: pgmap v14521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26532.104037 secs 2024-04-26T19:13:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26535.000077 secs 2024-04-26T19:13:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:14 smithi033 ceph-mon[27845]: pgmap v14521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26532.104037 secs 2024-04-26T19:13:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26535.000077 secs 2024-04-26T19:13:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:14 smithi096 ceph-mon[32753]: pgmap v14521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26532.104037 secs 2024-04-26T19:13:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26535.000077 secs 2024-04-26T19:13:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:16 smithi194 ceph-mon[33017]: pgmap v14522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:16 smithi033 ceph-mon[27845]: pgmap v14522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:16 smithi096 ceph-mon[32753]: pgmap v14522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:18 smithi033 ceph-mon[27845]: pgmap v14523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:18 smithi096 ceph-mon[32753]: pgmap v14523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:18 smithi194 ceph-mon[33017]: pgmap v14523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26537.104100 secs 2024-04-26T19:13:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26540.000206 secs 2024-04-26T19:13:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26537.104100 secs 2024-04-26T19:13:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26540.000206 secs 2024-04-26T19:13:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26537.104100 secs 2024-04-26T19:13:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26540.000206 secs 2024-04-26T19:13:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:20 smithi033 ceph-mon[27845]: pgmap v14524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:20 smithi096 ceph-mon[32753]: pgmap v14524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:20 smithi194 ceph-mon[33017]: pgmap v14524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:22.256+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:22.256+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:22.256+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:13:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:22 smithi033 ceph-mon[27845]: pgmap v14525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:22 smithi096 ceph-mon[32753]: pgmap v14525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:22 smithi194 ceph-mon[33017]: pgmap v14525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:24 smithi033 ceph-mon[27845]: pgmap v14526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26542.104186 secs 2024-04-26T19:13:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26545.000324 secs 2024-04-26T19:13:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:24 smithi096 ceph-mon[32753]: pgmap v14526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26542.104186 secs 2024-04-26T19:13:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26545.000324 secs 2024-04-26T19:13:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:24 smithi194 ceph-mon[33017]: pgmap v14526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26542.104186 secs 2024-04-26T19:13:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26545.000324 secs 2024-04-26T19:13:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:26 smithi033 ceph-mon[27845]: pgmap v14527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:26 smithi096 ceph-mon[32753]: pgmap v14527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:26.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:26 smithi194 ceph-mon[33017]: pgmap v14527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:28 smithi033 ceph-mon[27845]: pgmap v14528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:28 smithi096 ceph-mon[32753]: pgmap v14528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:28.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:28 smithi194 ceph-mon[33017]: pgmap v14528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26547.104290 secs 2024-04-26T19:13:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26550.000461 secs 2024-04-26T19:13:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26547.104290 secs 2024-04-26T19:13:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26550.000461 secs 2024-04-26T19:13:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26547.104290 secs 2024-04-26T19:13:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26550.000461 secs 2024-04-26T19:13:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:30 smithi033 ceph-mon[27845]: pgmap v14529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:30 smithi096 ceph-mon[32753]: pgmap v14529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:30 smithi194 ceph-mon[33017]: pgmap v14529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:32 smithi033 ceph-mon[27845]: pgmap v14530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:32.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:32 smithi096 ceph-mon[32753]: pgmap v14530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:32 smithi194 ceph-mon[33017]: pgmap v14530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:34 smithi033 ceph-mon[27845]: pgmap v14531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26552.104372 secs 2024-04-26T19:13:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26555.000581 secs 2024-04-26T19:13:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:34 smithi096 ceph-mon[32753]: pgmap v14531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26552.104372 secs 2024-04-26T19:13:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26555.000581 secs 2024-04-26T19:13:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:34 smithi194 ceph-mon[33017]: pgmap v14531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26552.104372 secs 2024-04-26T19:13:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26555.000581 secs 2024-04-26T19:13:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:36 smithi033 ceph-mon[27845]: pgmap v14532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:36 smithi096 ceph-mon[32753]: pgmap v14532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:36 smithi194 ceph-mon[33017]: pgmap v14532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:37.271+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:37.271+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:37.271+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:13:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:38 smithi033 ceph-mon[27845]: pgmap v14533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:38 smithi096 ceph-mon[32753]: pgmap v14533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:38 smithi194 ceph-mon[33017]: pgmap v14533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26557.104433 secs 2024-04-26T19:13:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26560.000713 secs 2024-04-26T19:13:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26557.104433 secs 2024-04-26T19:13:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26560.000713 secs 2024-04-26T19:13:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26557.104433 secs 2024-04-26T19:13:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26560.000713 secs 2024-04-26T19:13:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:40 smithi194 ceph-mon[33017]: pgmap v14534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:40 smithi033 ceph-mon[27845]: pgmap v14534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:40 smithi096 ceph-mon[32753]: pgmap v14534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:42 smithi194 ceph-mon[33017]: pgmap v14535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:42 smithi033 ceph-mon[27845]: pgmap v14535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:42 smithi096 ceph-mon[32753]: pgmap v14535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:44 smithi194 ceph-mon[33017]: pgmap v14536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26562.104514 secs 2024-04-26T19:13:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26565.000778 secs 2024-04-26T19:13:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:44 smithi033 ceph-mon[27845]: pgmap v14536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26562.104514 secs 2024-04-26T19:13:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26565.000778 secs 2024-04-26T19:13:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:44 smithi096 ceph-mon[32753]: pgmap v14536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26562.104514 secs 2024-04-26T19:13:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26565.000778 secs 2024-04-26T19:13:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:46 smithi194 ceph-mon[33017]: pgmap v14537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:46 smithi033 ceph-mon[27845]: pgmap v14537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:46 smithi096 ceph-mon[32753]: pgmap v14537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:48 smithi194 ceph-mon[33017]: pgmap v14538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:48 smithi033 ceph-mon[27845]: pgmap v14538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:48 smithi096 ceph-mon[32753]: pgmap v14538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26567.104599 secs 2024-04-26T19:13:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26570.000898 secs 2024-04-26T19:13:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26567.104599 secs 2024-04-26T19:13:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26570.000898 secs 2024-04-26T19:13:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26567.104599 secs 2024-04-26T19:13:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26570.000898 secs 2024-04-26T19:13:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:50 smithi194 ceph-mon[33017]: pgmap v14539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:50 smithi033 ceph-mon[27845]: pgmap v14539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:50.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:50 smithi096 ceph-mon[32753]: pgmap v14539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:13:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:13:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:13:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:52.282+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:52.282+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:13:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:13:52.282+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:13:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:52 smithi194 ceph-mon[33017]: pgmap v14540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:52.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:52 smithi033 ceph-mon[27845]: pgmap v14540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:52.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:52 smithi096 ceph-mon[32753]: pgmap v14540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:13:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:13:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:13:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:53 smithi194 ceph-mon[33017]: pgmap v14541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:53 smithi194 ceph-mon[33017]: pgmap v14542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:13:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:13:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:13:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:53 smithi033 ceph-mon[27845]: pgmap v14541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:53 smithi033 ceph-mon[27845]: pgmap v14542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:13:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:13:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:13:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:53 smithi096 ceph-mon[32753]: pgmap v14541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:13:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:53 smithi096 ceph-mon[32753]: pgmap v14542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:13:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26572.104688 secs 2024-04-26T19:13:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26575.001013 secs 2024-04-26T19:13:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26572.104688 secs 2024-04-26T19:13:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26575.001013 secs 2024-04-26T19:13:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26572.104688 secs 2024-04-26T19:13:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26575.001013 secs 2024-04-26T19:13:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:55 smithi194 ceph-mon[33017]: pgmap v14543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:13:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:55 smithi033 ceph-mon[27845]: pgmap v14543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:13:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:55 smithi096 ceph-mon[32753]: pgmap v14543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:13:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:57 smithi194 ceph-mon[33017]: pgmap v14544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:57.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:57 smithi033 ceph-mon[27845]: pgmap v14544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:57 smithi096 ceph-mon[32753]: pgmap v14544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26577.104773 secs 2024-04-26T19:13:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26577.104773 secs 2024-04-26T19:13:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26577.104773 secs 2024-04-26T19:13:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:59 smithi194 ceph-mon[33017]: pgmap v14545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:13:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26580.001149 secs 2024-04-26T19:13:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:59 smithi033 ceph-mon[27845]: pgmap v14545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:13:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26580.001149 secs 2024-04-26T19:13:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:59 smithi096 ceph-mon[32753]: pgmap v14545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:13:59.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:13:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26580.001149 secs 2024-04-26T19:14:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:01 smithi194 ceph-mon[33017]: pgmap v14546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:01 smithi033 ceph-mon[27845]: pgmap v14546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:01 smithi096 ceph-mon[32753]: pgmap v14546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:03 smithi194 ceph-mon[33017]: pgmap v14547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:14:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26582.104852 secs 2024-04-26T19:14:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:03 smithi033 ceph-mon[27845]: pgmap v14547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:14:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26582.104852 secs 2024-04-26T19:14:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:03 smithi096 ceph-mon[32753]: pgmap v14547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:14:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26582.104852 secs 2024-04-26T19:14:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26585.001234 secs 2024-04-26T19:14:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26585.001234 secs 2024-04-26T19:14:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26585.001234 secs 2024-04-26T19:14:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:05 smithi194 ceph-mon[33017]: pgmap v14548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:05 smithi033 ceph-mon[27845]: pgmap v14548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:05 smithi096 ceph-mon[32753]: pgmap v14548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:07.297+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:07.297+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:14:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:07.297+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:07 smithi194 ceph-mon[33017]: pgmap v14549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:07 smithi033 ceph-mon[27845]: pgmap v14549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:07 smithi096 ceph-mon[32753]: pgmap v14549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26587.104936 secs 2024-04-26T19:14:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26587.104936 secs 2024-04-26T19:14:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26587.104936 secs 2024-04-26T19:14:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:09 smithi033 ceph-mon[27845]: pgmap v14550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26590.001313 secs 2024-04-26T19:14:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:09 smithi194 ceph-mon[33017]: pgmap v14550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26590.001313 secs 2024-04-26T19:14:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:09 smithi096 ceph-mon[32753]: pgmap v14550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26590.001313 secs 2024-04-26T19:14:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:11 smithi194 ceph-mon[33017]: pgmap v14551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:11 smithi033 ceph-mon[27845]: pgmap v14551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:11 smithi096 ceph-mon[32753]: pgmap v14551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:13 smithi194 ceph-mon[33017]: pgmap v14552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26592.104999 secs 2024-04-26T19:14:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:13 smithi033 ceph-mon[27845]: pgmap v14552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26592.104999 secs 2024-04-26T19:14:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:13 smithi096 ceph-mon[32753]: pgmap v14552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26592.104999 secs 2024-04-26T19:14:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26595.001435 secs 2024-04-26T19:14:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26595.001435 secs 2024-04-26T19:14:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26595.001435 secs 2024-04-26T19:14:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:15 smithi194 ceph-mon[33017]: pgmap v14553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:15 smithi033 ceph-mon[27845]: pgmap v14553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:15 smithi096 ceph-mon[32753]: pgmap v14553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:17 smithi194 ceph-mon[33017]: pgmap v14554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:17 smithi033 ceph-mon[27845]: pgmap v14554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:17 smithi096 ceph-mon[32753]: pgmap v14554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26597.105061 secs 2024-04-26T19:14:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26597.105061 secs 2024-04-26T19:14:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26597.105061 secs 2024-04-26T19:14:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:19 smithi194 ceph-mon[33017]: pgmap v14555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26600.001558 secs 2024-04-26T19:14:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:19 smithi033 ceph-mon[27845]: pgmap v14555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26600.001558 secs 2024-04-26T19:14:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:19 smithi096 ceph-mon[32753]: pgmap v14555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26600.001558 secs 2024-04-26T19:14:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:21 smithi194 ceph-mon[33017]: pgmap v14556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:21 smithi033 ceph-mon[27845]: pgmap v14556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:21 smithi096 ceph-mon[32753]: pgmap v14556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:22.313+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:22.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:22.313+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:14:22.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:22.313+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:23 smithi194 ceph-mon[33017]: pgmap v14557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26602.105119 secs 2024-04-26T19:14:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:23 smithi033 ceph-mon[27845]: pgmap v14557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26602.105119 secs 2024-04-26T19:14:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:23 smithi096 ceph-mon[32753]: pgmap v14557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26602.105119 secs 2024-04-26T19:14:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26605.001674 secs 2024-04-26T19:14:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26605.001674 secs 2024-04-26T19:14:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26605.001674 secs 2024-04-26T19:14:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:25 smithi033 ceph-mon[27845]: pgmap v14558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:25 smithi096 ceph-mon[32753]: pgmap v14558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:25 smithi194 ceph-mon[33017]: pgmap v14558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:26.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:26.321+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:14:26.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:26.321+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:14:26.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:26.321+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 210 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T19:14:26.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:26.321+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 210 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:14:27.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:27 smithi033 ceph-mon[27845]: pgmap v14559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:27 smithi096 ceph-mon[32753]: pgmap v14559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:27 smithi194 ceph-mon[33017]: pgmap v14559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26607.105180 secs 2024-04-26T19:14:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26607.105180 secs 2024-04-26T19:14:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26607.105180 secs 2024-04-26T19:14:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:29 smithi033 ceph-mon[27845]: pgmap v14560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26610.001789 secs 2024-04-26T19:14:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:29 smithi096 ceph-mon[32753]: pgmap v14560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26610.001789 secs 2024-04-26T19:14:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:29 smithi194 ceph-mon[33017]: pgmap v14560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26610.001789 secs 2024-04-26T19:14:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:31 smithi033 ceph-mon[27845]: pgmap v14561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:31 smithi096 ceph-mon[32753]: pgmap v14561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:31 smithi194 ceph-mon[33017]: pgmap v14561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:33 smithi033 ceph-mon[27845]: pgmap v14562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26612.105240 secs 2024-04-26T19:14:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:33 smithi096 ceph-mon[32753]: pgmap v14562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26612.105240 secs 2024-04-26T19:14:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:33 smithi194 ceph-mon[33017]: pgmap v14562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26612.105240 secs 2024-04-26T19:14:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26615.001911 secs 2024-04-26T19:14:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26615.001911 secs 2024-04-26T19:14:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26615.001911 secs 2024-04-26T19:14:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:35 smithi033 ceph-mon[27845]: pgmap v14563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:35 smithi096 ceph-mon[32753]: pgmap v14563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:35 smithi194 ceph-mon[33017]: pgmap v14563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:37.317+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:37.317+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:37.317+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:14:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:37 smithi033 ceph-mon[27845]: pgmap v14564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:37 smithi096 ceph-mon[32753]: pgmap v14564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:37 smithi194 ceph-mon[33017]: pgmap v14564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26617.105301 secs 2024-04-26T19:14:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26617.105301 secs 2024-04-26T19:14:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26617.105301 secs 2024-04-26T19:14:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:39 smithi033 ceph-mon[27845]: pgmap v14565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26620.002049 secs 2024-04-26T19:14:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:39 smithi096 ceph-mon[32753]: pgmap v14565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26620.002049 secs 2024-04-26T19:14:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:39 smithi194 ceph-mon[33017]: pgmap v14565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26620.002049 secs 2024-04-26T19:14:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:41 smithi033 ceph-mon[27845]: pgmap v14566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:41 smithi096 ceph-mon[32753]: pgmap v14566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:41 smithi194 ceph-mon[33017]: pgmap v14566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:43 smithi096 ceph-mon[32753]: pgmap v14567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26622.105361 secs 2024-04-26T19:14:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:43 smithi194 ceph-mon[33017]: pgmap v14567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26622.105361 secs 2024-04-26T19:14:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:43 smithi033 ceph-mon[27845]: pgmap v14567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26622.105361 secs 2024-04-26T19:14:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26625.002166 secs 2024-04-26T19:14:45.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26625.002166 secs 2024-04-26T19:14:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26625.002166 secs 2024-04-26T19:14:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:45 smithi194 ceph-mon[33017]: pgmap v14568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:45 smithi033 ceph-mon[27845]: pgmap v14568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:45 smithi096 ceph-mon[32753]: pgmap v14568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:47 smithi194 ceph-mon[33017]: pgmap v14569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:47 smithi033 ceph-mon[27845]: pgmap v14569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:47 smithi096 ceph-mon[32753]: pgmap v14569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:49.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26627.105423 secs 2024-04-26T19:14:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26627.105423 secs 2024-04-26T19:14:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26627.105423 secs 2024-04-26T19:14:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:49 smithi194 ceph-mon[33017]: pgmap v14570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26630.002243 secs 2024-04-26T19:14:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:49 smithi033 ceph-mon[27845]: pgmap v14570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26630.002243 secs 2024-04-26T19:14:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:49 smithi096 ceph-mon[32753]: pgmap v14570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26630.002243 secs 2024-04-26T19:14:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:50 smithi194 ceph-mon[33017]: pgmap v14571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:50 smithi033 ceph-mon[27845]: pgmap v14571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:50 smithi096 ceph-mon[32753]: pgmap v14571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:14:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:52.330+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:52.330+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:14:52.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:14:52.330+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:14:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:53 smithi194 ceph-mon[33017]: pgmap v14572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:14:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26632.105488 secs 2024-04-26T19:14:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:53 smithi033 ceph-mon[27845]: pgmap v14572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:14:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26632.105488 secs 2024-04-26T19:14:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:53 smithi096 ceph-mon[32753]: pgmap v14572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:14:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26632.105488 secs 2024-04-26T19:14:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26635.002337 secs 2024-04-26T19:14:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:14:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:14:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:14:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26635.002337 secs 2024-04-26T19:14:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:14:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:14:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:14:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26635.002337 secs 2024-04-26T19:14:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:14:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:14:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:14:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:55 smithi194 ceph-mon[33017]: pgmap v14573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:55 smithi194 ceph-mon[33017]: pgmap v14574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:14:55.919 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:55 smithi033 ceph-mon[27845]: pgmap v14573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:55.919 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:55 smithi033 ceph-mon[27845]: pgmap v14574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:14:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:55 smithi096 ceph-mon[32753]: pgmap v14573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:14:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:55 smithi096 ceph-mon[32753]: pgmap v14574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:14:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:57 smithi194 ceph-mon[33017]: pgmap v14575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:57 smithi033 ceph-mon[27845]: pgmap v14575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:57 smithi096 ceph-mon[32753]: pgmap v14575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26637.105547 secs 2024-04-26T19:14:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26637.105547 secs 2024-04-26T19:14:58.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26637.105547 secs 2024-04-26T19:14:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:59 smithi194 ceph-mon[33017]: pgmap v14576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26640.002457 secs 2024-04-26T19:14:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:59 smithi033 ceph-mon[27845]: pgmap v14576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26640.002457 secs 2024-04-26T19:14:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:59 smithi096 ceph-mon[32753]: pgmap v14576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:14:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26640.002457 secs 2024-04-26T19:15:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:01 smithi194 ceph-mon[33017]: pgmap v14577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:01 smithi033 ceph-mon[27845]: pgmap v14577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:01 smithi096 ceph-mon[32753]: pgmap v14577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:03 smithi194 ceph-mon[33017]: pgmap v14578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26642.105627 secs 2024-04-26T19:15:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:03 smithi033 ceph-mon[27845]: pgmap v14578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26642.105627 secs 2024-04-26T19:15:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:03 smithi096 ceph-mon[32753]: pgmap v14578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26642.105627 secs 2024-04-26T19:15:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26645.002583 secs 2024-04-26T19:15:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26645.002583 secs 2024-04-26T19:15:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26645.002583 secs 2024-04-26T19:15:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:05 smithi194 ceph-mon[33017]: pgmap v14579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:15:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:05 smithi033 ceph-mon[27845]: pgmap v14579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:15:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:05 smithi096 ceph-mon[32753]: pgmap v14579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:15:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:07.345+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:07.345+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:15:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:07.345+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:07 smithi194 ceph-mon[33017]: pgmap v14580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:07.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:07 smithi033 ceph-mon[27845]: pgmap v14580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:07 smithi096 ceph-mon[32753]: pgmap v14580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26647.105729 secs 2024-04-26T19:15:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26647.105729 secs 2024-04-26T19:15:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26647.105729 secs 2024-04-26T19:15:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:09 smithi194 ceph-mon[33017]: pgmap v14581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26650.002703 secs 2024-04-26T19:15:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:09 smithi033 ceph-mon[27845]: pgmap v14581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26650.002703 secs 2024-04-26T19:15:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:09 smithi096 ceph-mon[32753]: pgmap v14581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26650.002703 secs 2024-04-26T19:15:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:11.327+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:15:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:11.327+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:15:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:11.328+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 211 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:15:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:11.328+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 211 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:15:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:11 smithi194 ceph-mon[33017]: pgmap v14582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:11 smithi033 ceph-mon[27845]: pgmap v14582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:11 smithi096 ceph-mon[32753]: pgmap v14582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:13 smithi194 ceph-mon[33017]: pgmap v14583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26652.105770 secs 2024-04-26T19:15:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:13 smithi033 ceph-mon[27845]: pgmap v14583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26652.105770 secs 2024-04-26T19:15:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:13 smithi096 ceph-mon[32753]: pgmap v14583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26652.105770 secs 2024-04-26T19:15:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26655.002836 secs 2024-04-26T19:15:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26655.002836 secs 2024-04-26T19:15:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26655.002836 secs 2024-04-26T19:15:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:15 smithi033 ceph-mon[27845]: pgmap v14584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:15 smithi096 ceph-mon[32753]: pgmap v14584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:16.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:15 smithi194 ceph-mon[33017]: pgmap v14584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:17 smithi033 ceph-mon[27845]: pgmap v14585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:17 smithi096 ceph-mon[32753]: pgmap v14585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:17 smithi194 ceph-mon[33017]: pgmap v14585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26657.105854 secs 2024-04-26T19:15:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26657.105854 secs 2024-04-26T19:15:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26657.105854 secs 2024-04-26T19:15:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:19 smithi033 ceph-mon[27845]: pgmap v14586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26660.002971 secs 2024-04-26T19:15:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:19 smithi096 ceph-mon[32753]: pgmap v14586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26660.002971 secs 2024-04-26T19:15:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:19 smithi194 ceph-mon[33017]: pgmap v14586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26660.002971 secs 2024-04-26T19:15:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:21 smithi033 ceph-mon[27845]: pgmap v14587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:21 smithi096 ceph-mon[32753]: pgmap v14587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:21 smithi194 ceph-mon[33017]: pgmap v14587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:22.355+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:22.355+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:22.355+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:15:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:23 smithi033 ceph-mon[27845]: pgmap v14588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26662.105940 secs 2024-04-26T19:15:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:23 smithi096 ceph-mon[32753]: pgmap v14588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26662.105940 secs 2024-04-26T19:15:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:23 smithi194 ceph-mon[33017]: pgmap v14588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26662.105940 secs 2024-04-26T19:15:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26665.003094 secs 2024-04-26T19:15:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26665.003094 secs 2024-04-26T19:15:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26665.003094 secs 2024-04-26T19:15:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:25 smithi033 ceph-mon[27845]: pgmap v14589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:25 smithi096 ceph-mon[32753]: pgmap v14589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:25 smithi194 ceph-mon[33017]: pgmap v14589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:27 smithi033 ceph-mon[27845]: pgmap v14590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:27 smithi096 ceph-mon[32753]: pgmap v14590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:27 smithi194 ceph-mon[33017]: pgmap v14590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26667.106029 secs 2024-04-26T19:15:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26667.106029 secs 2024-04-26T19:15:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26667.106029 secs 2024-04-26T19:15:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:29 smithi033 ceph-mon[27845]: pgmap v14591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26670.003209 secs 2024-04-26T19:15:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:29 smithi096 ceph-mon[32753]: pgmap v14591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26670.003209 secs 2024-04-26T19:15:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:29 smithi194 ceph-mon[33017]: pgmap v14591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26670.003209 secs 2024-04-26T19:15:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:31 smithi033 ceph-mon[27845]: pgmap v14592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:31 smithi096 ceph-mon[32753]: pgmap v14592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:31 smithi194 ceph-mon[33017]: pgmap v14592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:33 smithi096 ceph-mon[32753]: pgmap v14593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26672.106112 secs 2024-04-26T19:15:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:33 smithi194 ceph-mon[33017]: pgmap v14593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26672.106112 secs 2024-04-26T19:15:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:33 smithi033 ceph-mon[27845]: pgmap v14593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26672.106112 secs 2024-04-26T19:15:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26675.003310 secs 2024-04-26T19:15:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26675.003310 secs 2024-04-26T19:15:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26675.003310 secs 2024-04-26T19:15:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:35 smithi194 ceph-mon[33017]: pgmap v14594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:35 smithi033 ceph-mon[27845]: pgmap v14594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:35 smithi096 ceph-mon[32753]: pgmap v14594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:37.364+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:37.365+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:37.365+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:15:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:37 smithi194 ceph-mon[33017]: pgmap v14595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:37 smithi033 ceph-mon[27845]: pgmap v14595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:37 smithi096 ceph-mon[32753]: pgmap v14595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26677.106200 secs 2024-04-26T19:15:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26677.106200 secs 2024-04-26T19:15:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26677.106200 secs 2024-04-26T19:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:39 smithi194 ceph-mon[33017]: pgmap v14596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26680.003431 secs 2024-04-26T19:15:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:39 smithi033 ceph-mon[27845]: pgmap v14596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26680.003431 secs 2024-04-26T19:15:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:39 smithi096 ceph-mon[32753]: pgmap v14596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26680.003431 secs 2024-04-26T19:15:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:41 smithi194 ceph-mon[33017]: pgmap v14597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:41 smithi033 ceph-mon[27845]: pgmap v14597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:41 smithi096 ceph-mon[32753]: pgmap v14597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:43 smithi194 ceph-mon[33017]: pgmap v14598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26682.106262 secs 2024-04-26T19:15:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:43 smithi033 ceph-mon[27845]: pgmap v14598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26682.106262 secs 2024-04-26T19:15:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:43 smithi096 ceph-mon[32753]: pgmap v14598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26682.106262 secs 2024-04-26T19:15:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26685.003528 secs 2024-04-26T19:15:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26685.003528 secs 2024-04-26T19:15:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26685.003528 secs 2024-04-26T19:15:46.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:45 smithi194 ceph-mon[33017]: pgmap v14599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:45 smithi033 ceph-mon[27845]: pgmap v14599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:45 smithi096 ceph-mon[32753]: pgmap v14599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:46 smithi194 ceph-mon[33017]: pgmap v14600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:46 smithi033 ceph-mon[27845]: pgmap v14600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:46 smithi096 ceph-mon[32753]: pgmap v14600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26687.106346 secs 2024-04-26T19:15:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:49 smithi194 ceph-mon[33017]: pgmap v14601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26690.003649 secs 2024-04-26T19:15:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26687.106346 secs 2024-04-26T19:15:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:49 smithi033 ceph-mon[27845]: pgmap v14601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26690.003649 secs 2024-04-26T19:15:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26687.106346 secs 2024-04-26T19:15:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:49 smithi096 ceph-mon[32753]: pgmap v14601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26690.003649 secs 2024-04-26T19:15:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:51 smithi194 ceph-mon[33017]: pgmap v14602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:51 smithi033 ceph-mon[27845]: pgmap v14602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:51 smithi096 ceph-mon[32753]: pgmap v14602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:15:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:52.375+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:52.375+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:15:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:15:52.375+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:15:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:53 smithi194 ceph-mon[33017]: pgmap v14603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:53 smithi033 ceph-mon[27845]: pgmap v14603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:53 smithi096 ceph-mon[32753]: pgmap v14603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:54.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26692.106431 secs 2024-04-26T19:15:54.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26695.003768 secs 2024-04-26T19:15:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26692.106431 secs 2024-04-26T19:15:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26695.003768 secs 2024-04-26T19:15:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26692.106431 secs 2024-04-26T19:15:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26695.003768 secs 2024-04-26T19:15:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:55 smithi194 ceph-mon[33017]: pgmap v14604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:15:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:55 smithi033 ceph-mon[27845]: pgmap v14604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:15:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:55 smithi096 ceph-mon[32753]: pgmap v14604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:15:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:15:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:15:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:15:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:15:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:15:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:15:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:15:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:15:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:15:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:15:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:57 smithi194 ceph-mon[33017]: pgmap v14605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:15:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:57 smithi194 ceph-mon[33017]: pgmap v14606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:57 smithi033 ceph-mon[27845]: pgmap v14605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:15:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:57 smithi033 ceph-mon[27845]: pgmap v14606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:57 smithi096 ceph-mon[32753]: pgmap v14605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:15:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:57 smithi096 ceph-mon[32753]: pgmap v14606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:59 smithi194 ceph-mon[33017]: pgmap v14607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26697.106517 secs 2024-04-26T19:15:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:15:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26700.003908 secs 2024-04-26T19:15:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:59 smithi033 ceph-mon[27845]: pgmap v14607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26697.106517 secs 2024-04-26T19:15:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:15:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26700.003908 secs 2024-04-26T19:15:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:59 smithi096 ceph-mon[32753]: pgmap v14607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:15:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26697.106517 secs 2024-04-26T19:15:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:15:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26700.003908 secs 2024-04-26T19:16:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:01 smithi194 ceph-mon[33017]: pgmap v14608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:01 smithi033 ceph-mon[27845]: pgmap v14608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:01 smithi096 ceph-mon[32753]: pgmap v14608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:03 smithi194 ceph-mon[33017]: pgmap v14609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:03 smithi033 ceph-mon[27845]: pgmap v14609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:03 smithi096 ceph-mon[32753]: pgmap v14609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26702.106594 secs 2024-04-26T19:16:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26705.004034 secs 2024-04-26T19:16:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26702.106594 secs 2024-04-26T19:16:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26705.004034 secs 2024-04-26T19:16:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26702.106594 secs 2024-04-26T19:16:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26705.004034 secs 2024-04-26T19:16:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:05 smithi194 ceph-mon[33017]: pgmap v14610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:05 smithi033 ceph-mon[27845]: pgmap v14610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:05 smithi096 ceph-mon[32753]: pgmap v14610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:16:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:07.375+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:07.375+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:07.375+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:16:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:07 smithi194 ceph-mon[33017]: pgmap v14611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:16:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:07 smithi033 ceph-mon[27845]: pgmap v14611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:16:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:07 smithi096 ceph-mon[32753]: pgmap v14611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:16:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:09 smithi194 ceph-mon[33017]: pgmap v14612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26707.106675 secs 2024-04-26T19:16:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26710.004100 secs 2024-04-26T19:16:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:09 smithi033 ceph-mon[27845]: pgmap v14612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26707.106675 secs 2024-04-26T19:16:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26710.004100 secs 2024-04-26T19:16:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:09 smithi096 ceph-mon[32753]: pgmap v14612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26707.106675 secs 2024-04-26T19:16:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26710.004100 secs 2024-04-26T19:16:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:11 smithi194 ceph-mon[33017]: pgmap v14613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:11.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:11 smithi033 ceph-mon[27845]: pgmap v14613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:11 smithi096 ceph-mon[32753]: pgmap v14613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:13 smithi194 ceph-mon[33017]: pgmap v14614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:13 smithi033 ceph-mon[27845]: pgmap v14614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:13 smithi096 ceph-mon[32753]: pgmap v14614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26712.106782 secs 2024-04-26T19:16:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26715.004215 secs 2024-04-26T19:16:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26712.106782 secs 2024-04-26T19:16:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26715.004215 secs 2024-04-26T19:16:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26712.106782 secs 2024-04-26T19:16:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26715.004215 secs 2024-04-26T19:16:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:15 smithi194 ceph-mon[33017]: pgmap v14615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:15 smithi033 ceph-mon[27845]: pgmap v14615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:15 smithi096 ceph-mon[32753]: pgmap v14615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:17 smithi194 ceph-mon[33017]: pgmap v14616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:17 smithi033 ceph-mon[27845]: pgmap v14616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:17.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:17 smithi096 ceph-mon[32753]: pgmap v14616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26717.106866 secs 2024-04-26T19:16:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26717.106866 secs 2024-04-26T19:16:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26717.106866 secs 2024-04-26T19:16:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:19 smithi194 ceph-mon[33017]: pgmap v14617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26720.004316 secs 2024-04-26T19:16:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:19 smithi033 ceph-mon[27845]: pgmap v14617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26720.004316 secs 2024-04-26T19:16:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:19 smithi096 ceph-mon[32753]: pgmap v14617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26720.004316 secs 2024-04-26T19:16:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:21 smithi033 ceph-mon[27845]: pgmap v14618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:21 smithi096 ceph-mon[32753]: pgmap v14618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:21 smithi194 ceph-mon[33017]: pgmap v14618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:22.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:22.390+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:22.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:22.390+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:22.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:22.390+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:16:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:23 smithi033 ceph-mon[27845]: pgmap v14619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26722.106948 secs 2024-04-26T19:16:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:23 smithi096 ceph-mon[32753]: pgmap v14619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26722.106948 secs 2024-04-26T19:16:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:23 smithi194 ceph-mon[33017]: pgmap v14619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26722.106948 secs 2024-04-26T19:16:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26725.004455 secs 2024-04-26T19:16:24.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26725.004455 secs 2024-04-26T19:16:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26725.004455 secs 2024-04-26T19:16:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:25 smithi033 ceph-mon[27845]: pgmap v14620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:25 smithi096 ceph-mon[32753]: pgmap v14620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:25 smithi194 ceph-mon[33017]: pgmap v14620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:27 smithi033 ceph-mon[27845]: pgmap v14621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:27 smithi096 ceph-mon[32753]: pgmap v14621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:27 smithi194 ceph-mon[33017]: pgmap v14621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26727.107032 secs 2024-04-26T19:16:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26727.107032 secs 2024-04-26T19:16:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26727.107032 secs 2024-04-26T19:16:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:29 smithi033 ceph-mon[27845]: pgmap v14622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26730.004522 secs 2024-04-26T19:16:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:29 smithi096 ceph-mon[32753]: pgmap v14622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26730.004522 secs 2024-04-26T19:16:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:29 smithi194 ceph-mon[33017]: pgmap v14622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26730.004522 secs 2024-04-26T19:16:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:31 smithi033 ceph-mon[27845]: pgmap v14623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:31 smithi096 ceph-mon[32753]: pgmap v14623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:31 smithi194 ceph-mon[33017]: pgmap v14623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:33 smithi033 ceph-mon[27845]: pgmap v14624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26732.107110 secs 2024-04-26T19:16:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:33 smithi096 ceph-mon[32753]: pgmap v14624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26732.107110 secs 2024-04-26T19:16:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:33 smithi194 ceph-mon[33017]: pgmap v14624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26732.107110 secs 2024-04-26T19:16:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26735.004643 secs 2024-04-26T19:16:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26735.004643 secs 2024-04-26T19:16:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26735.004643 secs 2024-04-26T19:16:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:35 smithi033 ceph-mon[27845]: pgmap v14625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:35 smithi096 ceph-mon[32753]: pgmap v14625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:35 smithi194 ceph-mon[33017]: pgmap v14625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:37.392+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:37.392+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:37.392+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:16:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:37 smithi033 ceph-mon[27845]: pgmap v14626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:37 smithi096 ceph-mon[32753]: pgmap v14626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:37 smithi194 ceph-mon[33017]: pgmap v14626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26737.107168 secs 2024-04-26T19:16:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26737.107168 secs 2024-04-26T19:16:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26737.107168 secs 2024-04-26T19:16:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:39 smithi194 ceph-mon[33017]: pgmap v14627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26740.004764 secs 2024-04-26T19:16:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:39 smithi033 ceph-mon[27845]: pgmap v14627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26740.004764 secs 2024-04-26T19:16:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:39 smithi096 ceph-mon[32753]: pgmap v14627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26740.004764 secs 2024-04-26T19:16:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:41 smithi194 ceph-mon[33017]: pgmap v14628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:41 smithi033 ceph-mon[27845]: pgmap v14628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:41 smithi096 ceph-mon[32753]: pgmap v14628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:43 smithi194 ceph-mon[33017]: pgmap v14629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26742.107244 secs 2024-04-26T19:16:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:43 smithi033 ceph-mon[27845]: pgmap v14629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26742.107244 secs 2024-04-26T19:16:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:43 smithi096 ceph-mon[32753]: pgmap v14629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26742.107244 secs 2024-04-26T19:16:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26745.004875 secs 2024-04-26T19:16:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26745.004875 secs 2024-04-26T19:16:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26745.004875 secs 2024-04-26T19:16:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:45 smithi194 ceph-mon[33017]: pgmap v14630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:45 smithi033 ceph-mon[27845]: pgmap v14630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:45 smithi096 ceph-mon[32753]: pgmap v14630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:47 smithi194 ceph-mon[33017]: pgmap v14631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:47 smithi033 ceph-mon[27845]: pgmap v14631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:47 smithi096 ceph-mon[32753]: pgmap v14631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26747.107305 secs 2024-04-26T19:16:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26747.107305 secs 2024-04-26T19:16:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26747.107305 secs 2024-04-26T19:16:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:49 smithi096 ceph-mon[32753]: pgmap v14632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26750.004941 secs 2024-04-26T19:16:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:49 smithi194 ceph-mon[33017]: pgmap v14632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26750.004941 secs 2024-04-26T19:16:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:49 smithi033 ceph-mon[27845]: pgmap v14632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26750.004941 secs 2024-04-26T19:16:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:51 smithi194 ceph-mon[33017]: pgmap v14633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:51 smithi033 ceph-mon[27845]: pgmap v14633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:51 smithi096 ceph-mon[32753]: pgmap v14633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:52.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:52.395+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:52.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:52.395+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:16:52.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:16:52.395+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:16:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:53 smithi194 ceph-mon[33017]: pgmap v14634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26752.107364 secs 2024-04-26T19:16:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:53 smithi033 ceph-mon[27845]: pgmap v14634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26752.107364 secs 2024-04-26T19:16:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:53 smithi096 ceph-mon[32753]: pgmap v14634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26752.107364 secs 2024-04-26T19:16:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26755.005063 secs 2024-04-26T19:16:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26755.005063 secs 2024-04-26T19:16:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26755.005063 secs 2024-04-26T19:16:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:55 smithi194 ceph-mon[33017]: pgmap v14635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:55 smithi033 ceph-mon[27845]: pgmap v14635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:55 smithi096 ceph-mon[32753]: pgmap v14635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:16:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:16:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:16:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:57 smithi194 ceph-mon[33017]: pgmap v14636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:57 smithi033 ceph-mon[27845]: pgmap v14636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:57 smithi096 ceph-mon[32753]: pgmap v14636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:16:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26757.107424 secs 2024-04-26T19:16:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:16:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:16:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26757.107424 secs 2024-04-26T19:16:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:16:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:16:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26757.107424 secs 2024-04-26T19:16:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:16:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:16:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:16:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:59 smithi194 ceph-mon[33017]: pgmap v14637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:59 smithi194 ceph-mon[33017]: pgmap v14638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:16:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:59 smithi194 ceph-mon[33017]: pgmap v14639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:16:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26760.005185 secs 2024-04-26T19:16:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:59 smithi033 ceph-mon[27845]: pgmap v14637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:59 smithi033 ceph-mon[27845]: pgmap v14638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:16:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:59 smithi033 ceph-mon[27845]: pgmap v14639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:16:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26760.005185 secs 2024-04-26T19:16:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:59 smithi096 ceph-mon[32753]: pgmap v14637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:16:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:59 smithi096 ceph-mon[32753]: pgmap v14638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:16:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:59 smithi096 ceph-mon[32753]: pgmap v14639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:16:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26760.005185 secs 2024-04-26T19:17:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:01 smithi194 ceph-mon[33017]: pgmap v14640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:01 smithi033 ceph-mon[27845]: pgmap v14640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:01 smithi096 ceph-mon[32753]: pgmap v14640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:03 smithi194 ceph-mon[33017]: pgmap v14641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26762.107506 secs 2024-04-26T19:17:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:03 smithi033 ceph-mon[27845]: pgmap v14641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26762.107506 secs 2024-04-26T19:17:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:03 smithi096 ceph-mon[32753]: pgmap v14641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26762.107506 secs 2024-04-26T19:17:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26765.005307 secs 2024-04-26T19:17:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26765.005307 secs 2024-04-26T19:17:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26765.005307 secs 2024-04-26T19:17:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:05 smithi194 ceph-mon[33017]: pgmap v14642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:05 smithi033 ceph-mon[27845]: pgmap v14642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:05 smithi096 ceph-mon[32753]: pgmap v14642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:07.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:07.403+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:07.403+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:07.403+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:17:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:07 smithi194 ceph-mon[33017]: pgmap v14643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:07 smithi033 ceph-mon[27845]: pgmap v14643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:07 smithi096 ceph-mon[32753]: pgmap v14643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:17:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26767.107590 secs 2024-04-26T19:17:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26767.107590 secs 2024-04-26T19:17:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26767.107590 secs 2024-04-26T19:17:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:09 smithi194 ceph-mon[33017]: pgmap v14644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:17:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26770.005392 secs 2024-04-26T19:17:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:09 smithi033 ceph-mon[27845]: pgmap v14644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:17:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26770.005392 secs 2024-04-26T19:17:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:09 smithi096 ceph-mon[32753]: pgmap v14644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:17:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26770.005392 secs 2024-04-26T19:17:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:11 smithi194 ceph-mon[33017]: pgmap v14645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:11 smithi033 ceph-mon[27845]: pgmap v14645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:11 smithi096 ceph-mon[32753]: pgmap v14645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:13 smithi194 ceph-mon[33017]: pgmap v14646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26772.107671 secs 2024-04-26T19:17:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:13 smithi033 ceph-mon[27845]: pgmap v14646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26772.107671 secs 2024-04-26T19:17:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:13 smithi096 ceph-mon[32753]: pgmap v14646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26772.107671 secs 2024-04-26T19:17:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26775.005457 secs 2024-04-26T19:17:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26775.005457 secs 2024-04-26T19:17:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26775.005457 secs 2024-04-26T19:17:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:15 smithi194 ceph-mon[33017]: pgmap v14647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:15 smithi033 ceph-mon[27845]: pgmap v14647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:15 smithi096 ceph-mon[32753]: pgmap v14647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:17 smithi194 ceph-mon[33017]: pgmap v14648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:17 smithi033 ceph-mon[27845]: pgmap v14648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:17 smithi096 ceph-mon[32753]: pgmap v14648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26777.107757 secs 2024-04-26T19:17:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26777.107757 secs 2024-04-26T19:17:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26777.107757 secs 2024-04-26T19:17:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:19 smithi033 ceph-mon[27845]: pgmap v14649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26780.005573 secs 2024-04-26T19:17:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:19 smithi096 ceph-mon[32753]: pgmap v14649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26780.005573 secs 2024-04-26T19:17:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:19 smithi194 ceph-mon[33017]: pgmap v14649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26780.005573 secs 2024-04-26T19:17:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:21 smithi033 ceph-mon[27845]: pgmap v14650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:21 smithi096 ceph-mon[32753]: pgmap v14650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:21 smithi194 ceph-mon[33017]: pgmap v14650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:22.417+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:22.417+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:17:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:22.417+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:23 smithi033 ceph-mon[27845]: pgmap v14651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:23 smithi096 ceph-mon[32753]: pgmap v14651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:23 smithi194 ceph-mon[33017]: pgmap v14651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26782.107837 secs 2024-04-26T19:17:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26785.005673 secs 2024-04-26T19:17:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26782.107837 secs 2024-04-26T19:17:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26785.005673 secs 2024-04-26T19:17:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26782.107837 secs 2024-04-26T19:17:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26785.005673 secs 2024-04-26T19:17:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:25 smithi033 ceph-mon[27845]: pgmap v14652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:25 smithi096 ceph-mon[32753]: pgmap v14652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:25 smithi194 ceph-mon[33017]: pgmap v14652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:26.351+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:17:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:26.351+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:17:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:26.351+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 212 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:17:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:26.351+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 212 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:17:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:27 smithi194 ceph-mon[33017]: pgmap v14653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:27 smithi033 ceph-mon[27845]: pgmap v14653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:27 smithi096 ceph-mon[32753]: pgmap v14653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26787.107925 secs 2024-04-26T19:17:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26787.107925 secs 2024-04-26T19:17:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26787.107925 secs 2024-04-26T19:17:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:29 smithi194 ceph-mon[33017]: pgmap v14654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26790.005791 secs 2024-04-26T19:17:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:29 smithi033 ceph-mon[27845]: pgmap v14654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26790.005791 secs 2024-04-26T19:17:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:29 smithi096 ceph-mon[32753]: pgmap v14654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26790.005791 secs 2024-04-26T19:17:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:31 smithi194 ceph-mon[33017]: pgmap v14655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:31 smithi033 ceph-mon[27845]: pgmap v14655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:31 smithi096 ceph-mon[32753]: pgmap v14655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:33 smithi194 ceph-mon[33017]: pgmap v14656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26792.107998 secs 2024-04-26T19:17:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:33 smithi033 ceph-mon[27845]: pgmap v14656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26792.107998 secs 2024-04-26T19:17:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:33 smithi096 ceph-mon[32753]: pgmap v14656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26792.107998 secs 2024-04-26T19:17:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26795.005902 secs 2024-04-26T19:17:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26795.005902 secs 2024-04-26T19:17:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26795.005902 secs 2024-04-26T19:17:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:35 smithi194 ceph-mon[33017]: pgmap v14657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:35 smithi033 ceph-mon[27845]: pgmap v14657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:35 smithi096 ceph-mon[32753]: pgmap v14657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:37.423+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:37.423+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:37.423+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:17:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:37 smithi194 ceph-mon[33017]: pgmap v14658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:37 smithi033 ceph-mon[27845]: pgmap v14658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:37 smithi096 ceph-mon[32753]: pgmap v14658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26797.108061 secs 2024-04-26T19:17:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26797.108061 secs 2024-04-26T19:17:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26797.108061 secs 2024-04-26T19:17:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:39 smithi194 ceph-mon[33017]: pgmap v14659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26800.005965 secs 2024-04-26T19:17:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:39 smithi033 ceph-mon[27845]: pgmap v14659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26800.005965 secs 2024-04-26T19:17:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:39 smithi096 ceph-mon[32753]: pgmap v14659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26800.005965 secs 2024-04-26T19:17:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:41 smithi194 ceph-mon[33017]: pgmap v14660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:41 smithi033 ceph-mon[27845]: pgmap v14660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:41 smithi096 ceph-mon[32753]: pgmap v14660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:43 smithi033 ceph-mon[27845]: pgmap v14661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:43 smithi096 ceph-mon[32753]: pgmap v14661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:43 smithi194 ceph-mon[33017]: pgmap v14661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26802.108148 secs 2024-04-26T19:17:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26805.006091 secs 2024-04-26T19:17:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26802.108148 secs 2024-04-26T19:17:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26805.006091 secs 2024-04-26T19:17:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26802.108148 secs 2024-04-26T19:17:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26805.006091 secs 2024-04-26T19:17:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:45 smithi033 ceph-mon[27845]: pgmap v14662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:45 smithi096 ceph-mon[32753]: pgmap v14662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:45 smithi194 ceph-mon[33017]: pgmap v14662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:47 smithi194 ceph-mon[33017]: pgmap v14663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:47 smithi033 ceph-mon[27845]: pgmap v14663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:47 smithi096 ceph-mon[32753]: pgmap v14663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26807.108210 secs 2024-04-26T19:17:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26807.108210 secs 2024-04-26T19:17:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26807.108210 secs 2024-04-26T19:17:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:49 smithi096 ceph-mon[32753]: pgmap v14664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:49.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26810.006213 secs 2024-04-26T19:17:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:49 smithi194 ceph-mon[33017]: pgmap v14664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26810.006213 secs 2024-04-26T19:17:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:49 smithi033 ceph-mon[27845]: pgmap v14664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26810.006213 secs 2024-04-26T19:17:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:51 smithi194 ceph-mon[33017]: pgmap v14665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:51 smithi033 ceph-mon[27845]: pgmap v14665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:51 smithi096 ceph-mon[32753]: pgmap v14665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:52.431+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:52.432+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:17:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:17:52.432+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:17:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:53 smithi194 ceph-mon[33017]: pgmap v14666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26812.108285 secs 2024-04-26T19:17:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:53 smithi033 ceph-mon[27845]: pgmap v14666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26812.108285 secs 2024-04-26T19:17:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:53 smithi096 ceph-mon[32753]: pgmap v14666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26812.108285 secs 2024-04-26T19:17:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26815.006333 secs 2024-04-26T19:17:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26815.006333 secs 2024-04-26T19:17:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26815.006333 secs 2024-04-26T19:17:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:55 smithi194 ceph-mon[33017]: pgmap v14667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:55 smithi033 ceph-mon[27845]: pgmap v14667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:55 smithi096 ceph-mon[32753]: pgmap v14667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:17:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:57 smithi194 ceph-mon[33017]: pgmap v14668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:57 smithi033 ceph-mon[27845]: pgmap v14668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:57 smithi096 ceph-mon[32753]: pgmap v14668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26817.108349 secs 2024-04-26T19:17:58.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26817.108349 secs 2024-04-26T19:17:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26817.108349 secs 2024-04-26T19:17:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:59 smithi194 ceph-mon[33017]: pgmap v14669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26820.006401 secs 2024-04-26T19:17:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:17:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:17:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:59 smithi033 ceph-mon[27845]: pgmap v14669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26820.006401 secs 2024-04-26T19:17:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:17:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:17:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:59 smithi096 ceph-mon[32753]: pgmap v14669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:17:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26820.006401 secs 2024-04-26T19:17:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:17:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:18:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:18:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:18:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:18:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:18:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:18:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:18:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:18:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:18:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:18:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:01 smithi033 ceph-mon[27845]: pgmap v14670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:18:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:01 smithi033 ceph-mon[27845]: pgmap v14671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:01 smithi096 ceph-mon[32753]: pgmap v14670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:18:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:01 smithi096 ceph-mon[32753]: pgmap v14671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:01 smithi194 ceph-mon[33017]: pgmap v14670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:18:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:01 smithi194 ceph-mon[33017]: pgmap v14671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:03 smithi033 ceph-mon[27845]: pgmap v14672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26822.108436 secs 2024-04-26T19:18:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:03 smithi096 ceph-mon[32753]: pgmap v14672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26822.108436 secs 2024-04-26T19:18:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:03 smithi194 ceph-mon[33017]: pgmap v14672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26822.108436 secs 2024-04-26T19:18:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26825.006498 secs 2024-04-26T19:18:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26825.006498 secs 2024-04-26T19:18:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26825.006498 secs 2024-04-26T19:18:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:05 smithi033 ceph-mon[27845]: pgmap v14673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:18:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:05 smithi096 ceph-mon[32753]: pgmap v14673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:18:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:05 smithi194 ceph-mon[33017]: pgmap v14673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:18:07.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:07.443+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:07.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:07.443+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:07.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:07.443+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:18:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:07 smithi033 ceph-mon[27845]: pgmap v14674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:07 smithi096 ceph-mon[32753]: pgmap v14674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:08.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:07 smithi194 ceph-mon[33017]: pgmap v14674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26827.108516 secs 2024-04-26T19:18:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26827.108516 secs 2024-04-26T19:18:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26827.108516 secs 2024-04-26T19:18:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:09 smithi033 ceph-mon[27845]: pgmap v14675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26830.006560 secs 2024-04-26T19:18:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:09 smithi096 ceph-mon[32753]: pgmap v14675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26830.006560 secs 2024-04-26T19:18:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:09 smithi194 ceph-mon[33017]: pgmap v14675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:18:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26830.006560 secs 2024-04-26T19:18:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:11 smithi033 ceph-mon[27845]: pgmap v14676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:18:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:11 smithi096 ceph-mon[32753]: pgmap v14676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:18:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:11 smithi194 ceph-mon[33017]: pgmap v14676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:18:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:13 smithi033 ceph-mon[27845]: pgmap v14677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26832.108602 secs 2024-04-26T19:18:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:13 smithi096 ceph-mon[32753]: pgmap v14677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26832.108602 secs 2024-04-26T19:18:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:13 smithi194 ceph-mon[33017]: pgmap v14677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26832.108602 secs 2024-04-26T19:18:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26835.006683 secs 2024-04-26T19:18:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26835.006683 secs 2024-04-26T19:18:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26835.006683 secs 2024-04-26T19:18:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:15 smithi033 ceph-mon[27845]: pgmap v14678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:15 smithi096 ceph-mon[32753]: pgmap v14678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:15 smithi194 ceph-mon[33017]: pgmap v14678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:17 smithi033 ceph-mon[27845]: pgmap v14679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:18.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:17 smithi096 ceph-mon[32753]: pgmap v14679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:17 smithi194 ceph-mon[33017]: pgmap v14679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26837.108687 secs 2024-04-26T19:18:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26837.108687 secs 2024-04-26T19:18:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26837.108687 secs 2024-04-26T19:18:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:19 smithi096 ceph-mon[32753]: pgmap v14680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26840.006799 secs 2024-04-26T19:18:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:19 smithi194 ceph-mon[33017]: pgmap v14680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26840.006799 secs 2024-04-26T19:18:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:19 smithi033 ceph-mon[27845]: pgmap v14680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26840.006799 secs 2024-04-26T19:18:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:21 smithi194 ceph-mon[33017]: pgmap v14681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:21 smithi033 ceph-mon[27845]: pgmap v14681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:21 smithi096 ceph-mon[32753]: pgmap v14681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:22.458+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:22.458+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:22.458+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:18:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:23 smithi194 ceph-mon[33017]: pgmap v14682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26842.108772 secs 2024-04-26T19:18:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:23 smithi033 ceph-mon[27845]: pgmap v14682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26842.108772 secs 2024-04-26T19:18:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:23 smithi096 ceph-mon[32753]: pgmap v14682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26842.108772 secs 2024-04-26T19:18:25.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26845.006915 secs 2024-04-26T19:18:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26845.006915 secs 2024-04-26T19:18:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26845.006915 secs 2024-04-26T19:18:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:25 smithi194 ceph-mon[33017]: pgmap v14683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:25 smithi033 ceph-mon[27845]: pgmap v14683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:25 smithi096 ceph-mon[32753]: pgmap v14683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:27 smithi194 ceph-mon[33017]: pgmap v14684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:27 smithi033 ceph-mon[27845]: pgmap v14684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:27 smithi096 ceph-mon[32753]: pgmap v14684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26847.108838 secs 2024-04-26T19:18:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26847.108838 secs 2024-04-26T19:18:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26847.108838 secs 2024-04-26T19:18:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:29 smithi194 ceph-mon[33017]: pgmap v14685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26850.007035 secs 2024-04-26T19:18:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:29 smithi033 ceph-mon[27845]: pgmap v14685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26850.007035 secs 2024-04-26T19:18:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:29 smithi096 ceph-mon[32753]: pgmap v14685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26850.007035 secs 2024-04-26T19:18:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:31 smithi194 ceph-mon[33017]: pgmap v14686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:31 smithi033 ceph-mon[27845]: pgmap v14686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:31 smithi096 ceph-mon[32753]: pgmap v14686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:33 smithi194 ceph-mon[33017]: pgmap v14687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26852.108924 secs 2024-04-26T19:18:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:33 smithi033 ceph-mon[27845]: pgmap v14687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26852.108924 secs 2024-04-26T19:18:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:33 smithi096 ceph-mon[32753]: pgmap v14687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26852.108924 secs 2024-04-26T19:18:35.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26855.007150 secs 2024-04-26T19:18:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26855.007150 secs 2024-04-26T19:18:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26855.007150 secs 2024-04-26T19:18:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:35 smithi194 ceph-mon[33017]: pgmap v14688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:35 smithi033 ceph-mon[27845]: pgmap v14688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:35 smithi096 ceph-mon[32753]: pgmap v14688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:37.472+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:37.472+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:37.472+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:18:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:37 smithi194 ceph-mon[33017]: pgmap v14689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:37 smithi033 ceph-mon[27845]: pgmap v14689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:37 smithi096 ceph-mon[32753]: pgmap v14689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26857.109009 secs 2024-04-26T19:18:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26857.109009 secs 2024-04-26T19:18:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26857.109009 secs 2024-04-26T19:18:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:39 smithi194 ceph-mon[33017]: pgmap v14690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26860.007277 secs 2024-04-26T19:18:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:39 smithi033 ceph-mon[27845]: pgmap v14690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26860.007277 secs 2024-04-26T19:18:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:39 smithi096 ceph-mon[32753]: pgmap v14690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26860.007277 secs 2024-04-26T19:18:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:40 smithi194 ceph-mon[33017]: pgmap v14691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:40 smithi033 ceph-mon[27845]: pgmap v14691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:40 smithi096 ceph-mon[32753]: pgmap v14691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:43 smithi194 ceph-mon[33017]: pgmap v14692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:43 smithi033 ceph-mon[27845]: pgmap v14692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:43 smithi096 ceph-mon[32753]: pgmap v14692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26862.109090 secs 2024-04-26T19:18:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26865.007390 secs 2024-04-26T19:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26862.109090 secs 2024-04-26T19:18:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26865.007390 secs 2024-04-26T19:18:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26862.109090 secs 2024-04-26T19:18:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26865.007390 secs 2024-04-26T19:18:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:45 smithi194 ceph-mon[33017]: pgmap v14693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:45 smithi033 ceph-mon[27845]: pgmap v14693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:45 smithi096 ceph-mon[32753]: pgmap v14693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:47 smithi194 ceph-mon[33017]: pgmap v14694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:47 smithi033 ceph-mon[27845]: pgmap v14694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:47 smithi096 ceph-mon[32753]: pgmap v14694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26867.109172 secs 2024-04-26T19:18:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:49 smithi194 ceph-mon[33017]: pgmap v14695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26870.007503 secs 2024-04-26T19:18:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26867.109172 secs 2024-04-26T19:18:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:49 smithi033 ceph-mon[27845]: pgmap v14695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26870.007503 secs 2024-04-26T19:18:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26867.109172 secs 2024-04-26T19:18:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:49 smithi096 ceph-mon[32753]: pgmap v14695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26870.007503 secs 2024-04-26T19:18:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:51 smithi194 ceph-mon[33017]: pgmap v14696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:51 smithi033 ceph-mon[27845]: pgmap v14696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:51 smithi096 ceph-mon[32753]: pgmap v14696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:52.479+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:52.479+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:18:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:18:52.479+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:18:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:53 smithi194 ceph-mon[33017]: pgmap v14697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:53 smithi033 ceph-mon[27845]: pgmap v14697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:53 smithi096 ceph-mon[32753]: pgmap v14697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26872.109253 secs 2024-04-26T19:18:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26875.007601 secs 2024-04-26T19:18:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26872.109253 secs 2024-04-26T19:18:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26875.007601 secs 2024-04-26T19:18:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26872.109253 secs 2024-04-26T19:18:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26875.007601 secs 2024-04-26T19:18:55.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:55 smithi033 ceph-mon[27845]: pgmap v14698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:55 smithi194 ceph-mon[33017]: pgmap v14698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:55 smithi096 ceph-mon[32753]: pgmap v14698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:57 smithi194 ceph-mon[33017]: pgmap v14699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:57 smithi033 ceph-mon[27845]: pgmap v14699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:57 smithi096 ceph-mon[32753]: pgmap v14699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:18:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26877.109333 secs 2024-04-26T19:18:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:59 smithi194 ceph-mon[33017]: pgmap v14700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:18:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26880.007697 secs 2024-04-26T19:18:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26877.109333 secs 2024-04-26T19:18:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:59 smithi033 ceph-mon[27845]: pgmap v14700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:18:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26880.007697 secs 2024-04-26T19:18:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26877.109333 secs 2024-04-26T19:18:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:59 smithi096 ceph-mon[32753]: pgmap v14700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:18:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:18:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26880.007697 secs 2024-04-26T19:19:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:01 smithi194 ceph-mon[33017]: pgmap v14701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:19:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:01 smithi033 ceph-mon[27845]: pgmap v14701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:19:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:01 smithi096 ceph-mon[32753]: pgmap v14701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:19:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:03 smithi194 ceph-mon[33017]: pgmap v14702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:03 smithi033 ceph-mon[27845]: pgmap v14702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:03 smithi096 ceph-mon[32753]: pgmap v14702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26882.109404 secs 2024-04-26T19:19:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26885.007808 secs 2024-04-26T19:19:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26882.109404 secs 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26885.007808 secs 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26882.109404 secs 2024-04-26T19:19:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26885.007808 secs 2024-04-26T19:19:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:05 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:19:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:05 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:19:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:05 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:19:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:05 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:19:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:05 smithi194 ceph-mon[33017]: pgmap v14703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:05 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:19:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:05 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:19:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:05 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:19:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:05 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:19:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:05 smithi033 ceph-mon[27845]: pgmap v14703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:05 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:19:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:05 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:19:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:05 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:19:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:05 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:19:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:05 smithi096 ceph-mon[32753]: pgmap v14703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:07.492+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:07.492+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:07.493 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:07.492+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:19:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: pgmap v14704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: pgmap v14705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: pgmap v14706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: pgmap v14704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: pgmap v14705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: pgmap v14706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: pgmap v14704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: pgmap v14705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: pgmap v14706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:07.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:19:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26887.109485 secs 2024-04-26T19:19:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26887.109485 secs 2024-04-26T19:19:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26887.109485 secs 2024-04-26T19:19:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:09 smithi033 ceph-mon[27845]: pgmap v14707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:19:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26890.007928 secs 2024-04-26T19:19:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:09 smithi096 ceph-mon[32753]: pgmap v14707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:19:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26890.007928 secs 2024-04-26T19:19:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:09 smithi194 ceph-mon[33017]: pgmap v14707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:19:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26890.007928 secs 2024-04-26T19:19:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:11 smithi033 ceph-mon[27845]: pgmap v14708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:11 smithi096 ceph-mon[32753]: pgmap v14708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:11 smithi194 ceph-mon[33017]: pgmap v14708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:13 smithi033 ceph-mon[27845]: pgmap v14709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26892.109565 secs 2024-04-26T19:19:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:13 smithi096 ceph-mon[32753]: pgmap v14709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26892.109565 secs 2024-04-26T19:19:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:13 smithi194 ceph-mon[33017]: pgmap v14709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26892.109565 secs 2024-04-26T19:19:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26895.008043 secs 2024-04-26T19:19:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26895.008043 secs 2024-04-26T19:19:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26895.008043 secs 2024-04-26T19:19:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:15 smithi033 ceph-mon[27845]: pgmap v14710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:15 smithi096 ceph-mon[32753]: pgmap v14710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:15 smithi194 ceph-mon[33017]: pgmap v14710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:19:17.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:17 smithi194 ceph-mon[33017]: pgmap v14711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:19:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:17 smithi033 ceph-mon[27845]: pgmap v14711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:19:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:17 smithi096 ceph-mon[32753]: pgmap v14711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:19:18.921 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26897.109654 secs 2024-04-26T19:19:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26897.109654 secs 2024-04-26T19:19:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26897.109654 secs 2024-04-26T19:19:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:19 smithi033 ceph-mon[27845]: pgmap v14712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26900.008195 secs 2024-04-26T19:19:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:19 smithi096 ceph-mon[32753]: pgmap v14712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26900.008195 secs 2024-04-26T19:19:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:19 smithi194 ceph-mon[33017]: pgmap v14712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26900.008195 secs 2024-04-26T19:19:21.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:21 smithi033 ceph-mon[27845]: pgmap v14713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:21 smithi096 ceph-mon[32753]: pgmap v14713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:21 smithi194 ceph-mon[33017]: pgmap v14713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:22.505+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:22.505+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:19:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:22.505+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:23 smithi033 ceph-mon[27845]: pgmap v14714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26902.109740 secs 2024-04-26T19:19:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:23 smithi096 ceph-mon[32753]: pgmap v14714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26902.109740 secs 2024-04-26T19:19:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:23 smithi194 ceph-mon[33017]: pgmap v14714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26902.109740 secs 2024-04-26T19:19:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26905.008327 secs 2024-04-26T19:19:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26905.008327 secs 2024-04-26T19:19:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26905.008327 secs 2024-04-26T19:19:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:25 smithi033 ceph-mon[27845]: pgmap v14715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:25 smithi096 ceph-mon[32753]: pgmap v14715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:25 smithi194 ceph-mon[33017]: pgmap v14715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:27 smithi033 ceph-mon[27845]: pgmap v14716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:27 smithi096 ceph-mon[32753]: pgmap v14716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:27 smithi194 ceph-mon[33017]: pgmap v14716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26907.109825 secs 2024-04-26T19:19:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26907.109825 secs 2024-04-26T19:19:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26907.109825 secs 2024-04-26T19:19:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:29 smithi194 ceph-mon[33017]: pgmap v14717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26910.008449 secs 2024-04-26T19:19:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:29 smithi033 ceph-mon[27845]: pgmap v14717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26910.008449 secs 2024-04-26T19:19:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:29 smithi096 ceph-mon[32753]: pgmap v14717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26910.008449 secs 2024-04-26T19:19:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:31 smithi194 ceph-mon[33017]: pgmap v14718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:31 smithi033 ceph-mon[27845]: pgmap v14718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:31 smithi096 ceph-mon[32753]: pgmap v14718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:33 smithi194 ceph-mon[33017]: pgmap v14719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26912.109903 secs 2024-04-26T19:19:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:33 smithi033 ceph-mon[27845]: pgmap v14719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26912.109903 secs 2024-04-26T19:19:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:33 smithi096 ceph-mon[32753]: pgmap v14719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26912.109903 secs 2024-04-26T19:19:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26915.008591 secs 2024-04-26T19:19:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26915.008591 secs 2024-04-26T19:19:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26915.008591 secs 2024-04-26T19:19:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:35 smithi194 ceph-mon[33017]: pgmap v14720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:35 smithi033 ceph-mon[27845]: pgmap v14720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:35 smithi096 ceph-mon[32753]: pgmap v14720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:37.518+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:37.518+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:37.518+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:19:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:37 smithi194 ceph-mon[33017]: pgmap v14721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:37 smithi033 ceph-mon[27845]: pgmap v14721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:37 smithi096 ceph-mon[32753]: pgmap v14721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26917.109989 secs 2024-04-26T19:19:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26917.109989 secs 2024-04-26T19:19:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26917.109989 secs 2024-04-26T19:19:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:39 smithi194 ceph-mon[33017]: pgmap v14722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26920.008736 secs 2024-04-26T19:19:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:39 smithi033 ceph-mon[27845]: pgmap v14722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26920.008736 secs 2024-04-26T19:19:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:39 smithi096 ceph-mon[32753]: pgmap v14722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26920.008736 secs 2024-04-26T19:19:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:41 smithi194 ceph-mon[33017]: pgmap v14723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:41 smithi033 ceph-mon[27845]: pgmap v14723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:41 smithi096 ceph-mon[32753]: pgmap v14723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:43 smithi194 ceph-mon[33017]: pgmap v14724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26922.110075 secs 2024-04-26T19:19:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:43 smithi033 ceph-mon[27845]: pgmap v14724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26922.110075 secs 2024-04-26T19:19:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:43 smithi096 ceph-mon[32753]: pgmap v14724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26922.110075 secs 2024-04-26T19:19:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26925.008868 secs 2024-04-26T19:19:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26925.008868 secs 2024-04-26T19:19:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26925.008868 secs 2024-04-26T19:19:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:45 smithi194 ceph-mon[33017]: pgmap v14725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:45 smithi033 ceph-mon[27845]: pgmap v14725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:45 smithi096 ceph-mon[32753]: pgmap v14725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:47 smithi194 ceph-mon[33017]: pgmap v14726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:47 smithi033 ceph-mon[27845]: pgmap v14726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:47 smithi096 ceph-mon[32753]: pgmap v14726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:49.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26927.110137 secs 2024-04-26T19:19:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26927.110137 secs 2024-04-26T19:19:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26927.110137 secs 2024-04-26T19:19:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:49 smithi194 ceph-mon[33017]: pgmap v14727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:50.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26930.008943 secs 2024-04-26T19:19:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:49 smithi033 ceph-mon[27845]: pgmap v14727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26930.008943 secs 2024-04-26T19:19:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:49 smithi096 ceph-mon[32753]: pgmap v14727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26930.008943 secs 2024-04-26T19:19:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:51 smithi194 ceph-mon[33017]: pgmap v14728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:51 smithi033 ceph-mon[27845]: pgmap v14728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:51 smithi096 ceph-mon[32753]: pgmap v14728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:52.530+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:52.530+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:19:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:19:52.530+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:19:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:53 smithi194 ceph-mon[33017]: pgmap v14729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26932.110224 secs 2024-04-26T19:19:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:53 smithi033 ceph-mon[27845]: pgmap v14729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26932.110224 secs 2024-04-26T19:19:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:53 smithi096 ceph-mon[32753]: pgmap v14729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26932.110224 secs 2024-04-26T19:19:55.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26935.009088 secs 2024-04-26T19:19:55.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:54 smithi033 ceph-mon[27845]: pgmap v14730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26935.009088 secs 2024-04-26T19:19:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:54 smithi194 ceph-mon[33017]: pgmap v14730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26935.009088 secs 2024-04-26T19:19:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:54 smithi096 ceph-mon[32753]: pgmap v14730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:19:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:57 smithi033 ceph-mon[27845]: pgmap v14731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:57 smithi096 ceph-mon[32753]: pgmap v14731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:57 smithi194 ceph-mon[33017]: pgmap v14731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26937.110310 secs 2024-04-26T19:19:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26937.110310 secs 2024-04-26T19:19:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26937.110310 secs 2024-04-26T19:19:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:59 smithi033 ceph-mon[27845]: pgmap v14732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:19:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26940.009205 secs 2024-04-26T19:19:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:59 smithi096 ceph-mon[32753]: pgmap v14732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:19:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:19:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26940.009205 secs 2024-04-26T19:20:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:59 smithi194 ceph-mon[33017]: pgmap v14732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:19:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26940.009205 secs 2024-04-26T19:20:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:20:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:20:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:20:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:01 smithi033 ceph-mon[27845]: pgmap v14733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:01 smithi096 ceph-mon[32753]: pgmap v14733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:01 smithi194 ceph-mon[33017]: pgmap v14733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:03 smithi033 ceph-mon[27845]: pgmap v14734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26942.110394 secs 2024-04-26T19:20:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:03 smithi096 ceph-mon[32753]: pgmap v14734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26942.110394 secs 2024-04-26T19:20:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:03 smithi194 ceph-mon[33017]: pgmap v14734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26942.110394 secs 2024-04-26T19:20:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26945.009327 secs 2024-04-26T19:20:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26945.009327 secs 2024-04-26T19:20:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26945.009327 secs 2024-04-26T19:20:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:05 smithi096 ceph-mon[32753]: pgmap v14735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:06.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:05 smithi194 ceph-mon[33017]: pgmap v14735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:05 smithi033 ceph-mon[27845]: pgmap v14735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:07.545+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:07.545+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:07.545+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:20:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:07 smithi096 ceph-mon[32753]: pgmap v14736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:20:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:07 smithi194 ceph-mon[33017]: pgmap v14736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:20:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:07 smithi033 ceph-mon[27845]: pgmap v14736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:20:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26947.110479 secs 2024-04-26T19:20:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:20:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:20:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:20:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26947.110479 secs 2024-04-26T19:20:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:20:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:20:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:20:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26947.110479 secs 2024-04-26T19:20:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:20:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:20:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:20:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:09 smithi194 ceph-mon[33017]: pgmap v14737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:09 smithi194 ceph-mon[33017]: pgmap v14738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26950.009411 secs 2024-04-26T19:20:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:09 smithi033 ceph-mon[27845]: pgmap v14737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:09 smithi033 ceph-mon[27845]: pgmap v14738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26950.009411 secs 2024-04-26T19:20:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:09 smithi096 ceph-mon[32753]: pgmap v14737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:09 smithi096 ceph-mon[32753]: pgmap v14738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26950.009411 secs 2024-04-26T19:20:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:11 smithi194 ceph-mon[33017]: pgmap v14739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:11 smithi033 ceph-mon[27845]: pgmap v14739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:11 smithi096 ceph-mon[32753]: pgmap v14739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:13 smithi194 ceph-mon[33017]: pgmap v14740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26952.110562 secs 2024-04-26T19:20:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:13 smithi033 ceph-mon[27845]: pgmap v14740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26952.110562 secs 2024-04-26T19:20:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:13 smithi096 ceph-mon[32753]: pgmap v14740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26952.110562 secs 2024-04-26T19:20:15.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26955.009488 secs 2024-04-26T19:20:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26955.009488 secs 2024-04-26T19:20:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26955.009488 secs 2024-04-26T19:20:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:15 smithi194 ceph-mon[33017]: pgmap v14741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:20:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:15 smithi033 ceph-mon[27845]: pgmap v14741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:20:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:15 smithi096 ceph-mon[32753]: pgmap v14741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:20:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:17 smithi194 ceph-mon[33017]: pgmap v14742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:17 smithi033 ceph-mon[27845]: pgmap v14742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:17 smithi096 ceph-mon[32753]: pgmap v14742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:20:19.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26957.110649 secs 2024-04-26T19:20:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26957.110649 secs 2024-04-26T19:20:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26957.110649 secs 2024-04-26T19:20:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:19 smithi194 ceph-mon[33017]: pgmap v14743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:20:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26960.009604 secs 2024-04-26T19:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:19 smithi033 ceph-mon[27845]: pgmap v14743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:20:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26960.009604 secs 2024-04-26T19:20:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:19 smithi096 ceph-mon[32753]: pgmap v14743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:20:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26960.009604 secs 2024-04-26T19:20:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:21 smithi194 ceph-mon[33017]: pgmap v14744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:21 smithi033 ceph-mon[27845]: pgmap v14744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:21 smithi096 ceph-mon[32753]: pgmap v14744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:22.560+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:22.560+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:22.560+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:23 smithi194 ceph-mon[33017]: pgmap v14745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26962.110732 secs 2024-04-26T19:20:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:23 smithi033 ceph-mon[27845]: pgmap v14745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26962.110732 secs 2024-04-26T19:20:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:23 smithi096 ceph-mon[32753]: pgmap v14745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26962.110732 secs 2024-04-26T19:20:25.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26965.009716 secs 2024-04-26T19:20:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26965.009716 secs 2024-04-26T19:20:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26965.009716 secs 2024-04-26T19:20:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:25 smithi194 ceph-mon[33017]: pgmap v14746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:25 smithi033 ceph-mon[27845]: pgmap v14746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:25 smithi096 ceph-mon[32753]: pgmap v14746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:26.383+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:20:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:26.383+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:20:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:26.383+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 213 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00066b0 con 0x564385821900 2024-04-26T19:20:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:26.383+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 213 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:20:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:27 smithi194 ceph-mon[33017]: pgmap v14747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:27 smithi033 ceph-mon[27845]: pgmap v14747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:27 smithi096 ceph-mon[32753]: pgmap v14747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26967.110818 secs 2024-04-26T19:20:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26967.110818 secs 2024-04-26T19:20:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26967.110818 secs 2024-04-26T19:20:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:29 smithi194 ceph-mon[33017]: pgmap v14748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26970.009833 secs 2024-04-26T19:20:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:29 smithi033 ceph-mon[27845]: pgmap v14748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26970.009833 secs 2024-04-26T19:20:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:29 smithi096 ceph-mon[32753]: pgmap v14748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26970.009833 secs 2024-04-26T19:20:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:31 smithi194 ceph-mon[33017]: pgmap v14749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:31 smithi033 ceph-mon[27845]: pgmap v14749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:31 smithi096 ceph-mon[32753]: pgmap v14749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:33 smithi194 ceph-mon[33017]: pgmap v14750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26972.110906 secs 2024-04-26T19:20:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:33 smithi033 ceph-mon[27845]: pgmap v14750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26972.110906 secs 2024-04-26T19:20:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:33 smithi096 ceph-mon[32753]: pgmap v14750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26972.110906 secs 2024-04-26T19:20:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26975.009972 secs 2024-04-26T19:20:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26975.009972 secs 2024-04-26T19:20:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26975.009972 secs 2024-04-26T19:20:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:35 smithi194 ceph-mon[33017]: pgmap v14751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:35 smithi033 ceph-mon[27845]: pgmap v14751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:35 smithi096 ceph-mon[32753]: pgmap v14751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:20:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:36 smithi194 ceph-mon[33017]: pgmap v14752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:36 smithi033 ceph-mon[27845]: pgmap v14752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:36 smithi096 ceph-mon[32753]: pgmap v14752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:20:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:37.565+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:37.566+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:37.566+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:20:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26977.110992 secs 2024-04-26T19:20:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:39 smithi194 ceph-mon[33017]: pgmap v14753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26980.010103 secs 2024-04-26T19:20:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26977.110992 secs 2024-04-26T19:20:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:39 smithi033 ceph-mon[27845]: pgmap v14753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26980.010103 secs 2024-04-26T19:20:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26977.110992 secs 2024-04-26T19:20:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:39 smithi096 ceph-mon[32753]: pgmap v14753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26980.010103 secs 2024-04-26T19:20:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:41 smithi194 ceph-mon[33017]: pgmap v14754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:41 smithi033 ceph-mon[27845]: pgmap v14754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:41 smithi096 ceph-mon[32753]: pgmap v14754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:43 smithi033 ceph-mon[27845]: pgmap v14755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:43 smithi096 ceph-mon[32753]: pgmap v14755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:43 smithi194 ceph-mon[33017]: pgmap v14755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26982.111072 secs 2024-04-26T19:20:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26985.010197 secs 2024-04-26T19:20:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26982.111072 secs 2024-04-26T19:20:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26985.010197 secs 2024-04-26T19:20:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26982.111072 secs 2024-04-26T19:20:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26985.010197 secs 2024-04-26T19:20:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:45 smithi033 ceph-mon[27845]: pgmap v14756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:45 smithi096 ceph-mon[32753]: pgmap v14756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:45 smithi194 ceph-mon[33017]: pgmap v14756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:47.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:47 smithi194 ceph-mon[33017]: pgmap v14757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:47 smithi033 ceph-mon[27845]: pgmap v14757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:47 smithi096 ceph-mon[32753]: pgmap v14757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26987.111157 secs 2024-04-26T19:20:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26987.111157 secs 2024-04-26T19:20:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26987.111157 secs 2024-04-26T19:20:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:49 smithi033 ceph-mon[27845]: pgmap v14758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26990.010313 secs 2024-04-26T19:20:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:49 smithi096 ceph-mon[32753]: pgmap v14758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26990.010313 secs 2024-04-26T19:20:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:49 smithi194 ceph-mon[33017]: pgmap v14758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26990.010313 secs 2024-04-26T19:20:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:51 smithi033 ceph-mon[27845]: pgmap v14759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:51 smithi096 ceph-mon[32753]: pgmap v14759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:51 smithi194 ceph-mon[33017]: pgmap v14759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:52.581+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:52.581+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:20:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:20:52.581+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:20:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:53 smithi033 ceph-mon[27845]: pgmap v14760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26992.111240 secs 2024-04-26T19:20:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:53 smithi096 ceph-mon[32753]: pgmap v14760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26992.111240 secs 2024-04-26T19:20:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:53 smithi194 ceph-mon[33017]: pgmap v14760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26992.111240 secs 2024-04-26T19:20:54.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 26995.010424 secs 2024-04-26T19:20:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 26995.010424 secs 2024-04-26T19:20:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 26995.010424 secs 2024-04-26T19:20:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:55 smithi033 ceph-mon[27845]: pgmap v14761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:55 smithi096 ceph-mon[32753]: pgmap v14761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:55 smithi194 ceph-mon[33017]: pgmap v14761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:57 smithi033 ceph-mon[27845]: pgmap v14762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:57 smithi096 ceph-mon[32753]: pgmap v14762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:57 smithi194 ceph-mon[33017]: pgmap v14762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:20:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 26997.111304 secs 2024-04-26T19:20:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 26997.111304 secs 2024-04-26T19:20:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 26997.111304 secs 2024-04-26T19:20:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:59 smithi194 ceph-mon[33017]: pgmap v14763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:20:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27000.010548 secs 2024-04-26T19:20:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:59 smithi033 ceph-mon[27845]: pgmap v14763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:20:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27000.010548 secs 2024-04-26T19:20:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:59 smithi096 ceph-mon[32753]: pgmap v14763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:20:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:20:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27000.010548 secs 2024-04-26T19:21:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:01 smithi194 ceph-mon[33017]: pgmap v14764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:01 smithi033 ceph-mon[27845]: pgmap v14764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:01.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:01 smithi096 ceph-mon[32753]: pgmap v14764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:03 smithi194 ceph-mon[33017]: pgmap v14765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27002.111387 secs 2024-04-26T19:21:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:03 smithi033 ceph-mon[27845]: pgmap v14765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27002.111387 secs 2024-04-26T19:21:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:03 smithi096 ceph-mon[32753]: pgmap v14765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27002.111387 secs 2024-04-26T19:21:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27005.010671 secs 2024-04-26T19:21:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27005.010671 secs 2024-04-26T19:21:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27005.010671 secs 2024-04-26T19:21:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:05 smithi194 ceph-mon[33017]: pgmap v14766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:05 smithi033 ceph-mon[27845]: pgmap v14766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:05 smithi096 ceph-mon[32753]: pgmap v14766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:07.596+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:07.596+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:07.596+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:21:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:07 smithi194 ceph-mon[33017]: pgmap v14767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:07 smithi033 ceph-mon[27845]: pgmap v14767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:07 smithi096 ceph-mon[32753]: pgmap v14767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27007.111472 secs 2024-04-26T19:21:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27007.111472 secs 2024-04-26T19:21:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27007.111472 secs 2024-04-26T19:21:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:09 smithi194 ceph-mon[33017]: pgmap v14768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27010.010806 secs 2024-04-26T19:21:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:21:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:09 smithi033 ceph-mon[27845]: pgmap v14768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27010.010806 secs 2024-04-26T19:21:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:21:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:09 smithi096 ceph-mon[32753]: pgmap v14768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27010.010806 secs 2024-04-26T19:21:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:21:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:21:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:21:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:21:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:21:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:21:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:21:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:21:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:21:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:21:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:11 smithi194 ceph-mon[33017]: pgmap v14769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:21:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:11 smithi194 ceph-mon[33017]: pgmap v14770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:11 smithi033 ceph-mon[27845]: pgmap v14769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:21:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:11 smithi033 ceph-mon[27845]: pgmap v14770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:11 smithi096 ceph-mon[32753]: pgmap v14769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:21:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:11 smithi096 ceph-mon[32753]: pgmap v14770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:13 smithi194 ceph-mon[33017]: pgmap v14771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27012.111550 secs 2024-04-26T19:21:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:13 smithi033 ceph-mon[27845]: pgmap v14771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27012.111550 secs 2024-04-26T19:21:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:13 smithi096 ceph-mon[32753]: pgmap v14771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27012.111550 secs 2024-04-26T19:21:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27015.010904 secs 2024-04-26T19:21:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27015.010904 secs 2024-04-26T19:21:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27015.010904 secs 2024-04-26T19:21:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:15 smithi194 ceph-mon[33017]: pgmap v14772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:15 smithi033 ceph-mon[27845]: pgmap v14772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:15 smithi096 ceph-mon[32753]: pgmap v14772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:17 smithi194 ceph-mon[33017]: pgmap v14773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:17 smithi033 ceph-mon[27845]: pgmap v14773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:17 smithi096 ceph-mon[32753]: pgmap v14773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27017.111614 secs 2024-04-26T19:21:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27017.111614 secs 2024-04-26T19:21:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27017.111614 secs 2024-04-26T19:21:19.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:19 smithi096 ceph-mon[32753]: pgmap v14774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:19.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27020.011019 secs 2024-04-26T19:21:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:19 smithi194 ceph-mon[33017]: pgmap v14774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27020.011019 secs 2024-04-26T19:21:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:19 smithi033 ceph-mon[27845]: pgmap v14774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:21:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27020.011019 secs 2024-04-26T19:21:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:21 smithi194 ceph-mon[33017]: pgmap v14775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:21:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:21 smithi033 ceph-mon[27845]: pgmap v14775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:21:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:21 smithi096 ceph-mon[32753]: pgmap v14775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:21:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:22.608+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:22.608+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:22.608+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:21:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:23 smithi194 ceph-mon[33017]: pgmap v14776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27022.111699 secs 2024-04-26T19:21:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:23 smithi033 ceph-mon[27845]: pgmap v14776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27022.111699 secs 2024-04-26T19:21:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:23 smithi096 ceph-mon[32753]: pgmap v14776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27022.111699 secs 2024-04-26T19:21:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27025.011139 secs 2024-04-26T19:21:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27025.011139 secs 2024-04-26T19:21:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27025.011139 secs 2024-04-26T19:21:25.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:25 smithi033 ceph-mon[27845]: pgmap v14777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:25 smithi194 ceph-mon[33017]: pgmap v14777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:25 smithi096 ceph-mon[32753]: pgmap v14777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:27 smithi194 ceph-mon[33017]: pgmap v14778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:27 smithi033 ceph-mon[27845]: pgmap v14778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:27 smithi096 ceph-mon[32753]: pgmap v14778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27027.111785 secs 2024-04-26T19:21:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27027.111785 secs 2024-04-26T19:21:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27027.111785 secs 2024-04-26T19:21:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:29 smithi194 ceph-mon[33017]: pgmap v14779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27030.011222 secs 2024-04-26T19:21:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:29 smithi033 ceph-mon[27845]: pgmap v14779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27030.011222 secs 2024-04-26T19:21:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:29 smithi096 ceph-mon[32753]: pgmap v14779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27030.011222 secs 2024-04-26T19:21:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:31 smithi194 ceph-mon[33017]: pgmap v14780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:31 smithi033 ceph-mon[27845]: pgmap v14780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:31 smithi096 ceph-mon[32753]: pgmap v14780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:33 smithi194 ceph-mon[33017]: pgmap v14781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27032.111869 secs 2024-04-26T19:21:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:33 smithi033 ceph-mon[27845]: pgmap v14781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27032.111869 secs 2024-04-26T19:21:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:33 smithi096 ceph-mon[32753]: pgmap v14781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27032.111869 secs 2024-04-26T19:21:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27035.011291 secs 2024-04-26T19:21:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27035.011291 secs 2024-04-26T19:21:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27035.011291 secs 2024-04-26T19:21:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:35 smithi194 ceph-mon[33017]: pgmap v14782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:35 smithi033 ceph-mon[27845]: pgmap v14782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:35 smithi096 ceph-mon[32753]: pgmap v14782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:37.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:37.624+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:37.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:37.624+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:37.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:37.624+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:21:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:37 smithi194 ceph-mon[33017]: pgmap v14783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:37 smithi033 ceph-mon[27845]: pgmap v14783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:37 smithi096 ceph-mon[32753]: pgmap v14783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27037.111955 secs 2024-04-26T19:21:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27037.111955 secs 2024-04-26T19:21:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27037.111955 secs 2024-04-26T19:21:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:39 smithi033 ceph-mon[27845]: pgmap v14784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27040.011416 secs 2024-04-26T19:21:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:39 smithi096 ceph-mon[32753]: pgmap v14784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27040.011416 secs 2024-04-26T19:21:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:39 smithi194 ceph-mon[33017]: pgmap v14784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27040.011416 secs 2024-04-26T19:21:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:41 smithi033 ceph-mon[27845]: pgmap v14785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:41 smithi096 ceph-mon[32753]: pgmap v14785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:41 smithi194 ceph-mon[33017]: pgmap v14785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:43 smithi033 ceph-mon[27845]: pgmap v14786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27042.112041 secs 2024-04-26T19:21:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:43 smithi096 ceph-mon[32753]: pgmap v14786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27042.112041 secs 2024-04-26T19:21:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:43 smithi194 ceph-mon[33017]: pgmap v14786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27042.112041 secs 2024-04-26T19:21:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27045.011501 secs 2024-04-26T19:21:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27045.011501 secs 2024-04-26T19:21:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27045.011501 secs 2024-04-26T19:21:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:45 smithi033 ceph-mon[27845]: pgmap v14787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:45 smithi096 ceph-mon[32753]: pgmap v14787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:45 smithi194 ceph-mon[33017]: pgmap v14787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:21:47.905 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:47 smithi194 ceph-mon[33017]: pgmap v14788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:47 smithi033 ceph-mon[27845]: pgmap v14788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:47 smithi096 ceph-mon[32753]: pgmap v14788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:21:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27047.112104 secs 2024-04-26T19:21:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27047.112104 secs 2024-04-26T19:21:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27047.112104 secs 2024-04-26T19:21:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:49 smithi033 ceph-mon[27845]: pgmap v14789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27050.011637 secs 2024-04-26T19:21:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:49 smithi096 ceph-mon[32753]: pgmap v14789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27050.011637 secs 2024-04-26T19:21:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:49 smithi194 ceph-mon[33017]: pgmap v14789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27050.011637 secs 2024-04-26T19:21:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:51 smithi033 ceph-mon[27845]: pgmap v14790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:51 smithi096 ceph-mon[32753]: pgmap v14790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:51 smithi194 ceph-mon[33017]: pgmap v14790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:52.634+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:52.634+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:21:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:21:52.634+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:21:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:53 smithi033 ceph-mon[27845]: pgmap v14791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27052.112168 secs 2024-04-26T19:21:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:53 smithi096 ceph-mon[32753]: pgmap v14791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27052.112168 secs 2024-04-26T19:21:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:53 smithi194 ceph-mon[33017]: pgmap v14791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27052.112168 secs 2024-04-26T19:21:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27055.011708 secs 2024-04-26T19:21:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27055.011708 secs 2024-04-26T19:21:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27055.011708 secs 2024-04-26T19:21:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:55 smithi033 ceph-mon[27845]: pgmap v14792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:55 smithi096 ceph-mon[32753]: pgmap v14792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:55 smithi194 ceph-mon[33017]: pgmap v14792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:21:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:57 smithi096 ceph-mon[32753]: pgmap v14793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:57 smithi194 ceph-mon[33017]: pgmap v14793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:57 smithi033 ceph-mon[27845]: pgmap v14793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:21:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27057.112230 secs 2024-04-26T19:21:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27057.112230 secs 2024-04-26T19:21:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27057.112230 secs 2024-04-26T19:22:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:59 smithi194 ceph-mon[33017]: pgmap v14794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:21:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27060.011825 secs 2024-04-26T19:22:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:59 smithi033 ceph-mon[27845]: pgmap v14794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:21:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27060.011825 secs 2024-04-26T19:22:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:59 smithi096 ceph-mon[32753]: pgmap v14794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:21:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27060.011825 secs 2024-04-26T19:22:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:01 smithi194 ceph-mon[33017]: pgmap v14795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:01 smithi033 ceph-mon[27845]: pgmap v14795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:01 smithi096 ceph-mon[32753]: pgmap v14795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:03 smithi194 ceph-mon[33017]: pgmap v14796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27062.112316 secs 2024-04-26T19:22:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:03 smithi033 ceph-mon[27845]: pgmap v14796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27062.112316 secs 2024-04-26T19:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:03 smithi096 ceph-mon[32753]: pgmap v14796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27062.112316 secs 2024-04-26T19:22:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27065.011946 secs 2024-04-26T19:22:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27065.011946 secs 2024-04-26T19:22:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27065.011946 secs 2024-04-26T19:22:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:05 smithi194 ceph-mon[33017]: pgmap v14797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:05 smithi033 ceph-mon[27845]: pgmap v14797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:05 smithi096 ceph-mon[32753]: pgmap v14797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:07.645+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:07.645+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:07.645+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:22:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:07 smithi194 ceph-mon[33017]: pgmap v14798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:07 smithi033 ceph-mon[27845]: pgmap v14798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:07 smithi096 ceph-mon[32753]: pgmap v14798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27067.112403 secs 2024-04-26T19:22:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27067.112403 secs 2024-04-26T19:22:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27067.112403 secs 2024-04-26T19:22:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:09 smithi194 ceph-mon[33017]: pgmap v14799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27070.012079 secs 2024-04-26T19:22:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:09 smithi033 ceph-mon[27845]: pgmap v14799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27070.012079 secs 2024-04-26T19:22:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:09 smithi096 ceph-mon[32753]: pgmap v14799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27070.012079 secs 2024-04-26T19:22:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:22:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:22:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:22:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:11 smithi194 ceph-mon[33017]: pgmap v14800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:22:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:22:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:22:12.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:11 smithi033 ceph-mon[27845]: pgmap v14800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:12.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:22:12.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:22:12.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:22:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:11 smithi096 ceph-mon[32753]: pgmap v14800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:22:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:22:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:22:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:12 smithi194 ceph-mon[33017]: pgmap v14801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:12 smithi194 ceph-mon[33017]: pgmap v14802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:12 smithi033 ceph-mon[27845]: pgmap v14801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:12 smithi033 ceph-mon[27845]: pgmap v14802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:12 smithi096 ceph-mon[32753]: pgmap v14801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:12 smithi096 ceph-mon[32753]: pgmap v14802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27072.112485 secs 2024-04-26T19:22:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27072.112485 secs 2024-04-26T19:22:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27072.112485 secs 2024-04-26T19:22:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27075.012179 secs 2024-04-26T19:22:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:14 smithi194 ceph-mon[33017]: pgmap v14803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27075.012179 secs 2024-04-26T19:22:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:14 smithi033 ceph-mon[27845]: pgmap v14803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27075.012179 secs 2024-04-26T19:22:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:14 smithi096 ceph-mon[32753]: pgmap v14803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:16 smithi194 ceph-mon[33017]: pgmap v14804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:16 smithi033 ceph-mon[27845]: pgmap v14804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:16 smithi096 ceph-mon[32753]: pgmap v14804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:18 smithi194 ceph-mon[33017]: pgmap v14805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27077.112569 secs 2024-04-26T19:22:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:18 smithi033 ceph-mon[27845]: pgmap v14805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27077.112569 secs 2024-04-26T19:22:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:18 smithi096 ceph-mon[32753]: pgmap v14805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27077.112569 secs 2024-04-26T19:22:20.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27080.012298 secs 2024-04-26T19:22:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27080.012298 secs 2024-04-26T19:22:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27080.012298 secs 2024-04-26T19:22:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:20 smithi194 ceph-mon[33017]: pgmap v14806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:20 smithi033 ceph-mon[27845]: pgmap v14806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:20 smithi096 ceph-mon[32753]: pgmap v14806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:22:22.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:22.650+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:22.651 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:22.650+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:22.652 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:22.650+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:22:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:22 smithi194 ceph-mon[33017]: pgmap v14807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:22:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:22 smithi033 ceph-mon[27845]: pgmap v14807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:22:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:22 smithi096 ceph-mon[32753]: pgmap v14807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:22:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27082.112651 secs 2024-04-26T19:22:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27082.112651 secs 2024-04-26T19:22:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27082.112651 secs 2024-04-26T19:22:25.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27085.012366 secs 2024-04-26T19:22:25.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:24 smithi033 ceph-mon[27845]: pgmap v14808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27085.012366 secs 2024-04-26T19:22:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:24 smithi194 ceph-mon[33017]: pgmap v14808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27085.012366 secs 2024-04-26T19:22:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:24 smithi096 ceph-mon[32753]: pgmap v14808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:25 smithi194 ceph-mon[33017]: pgmap v14809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:25 smithi033 ceph-mon[27845]: pgmap v14809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:25 smithi096 ceph-mon[32753]: pgmap v14809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:28 smithi194 ceph-mon[33017]: pgmap v14810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27087.112729 secs 2024-04-26T19:22:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:28 smithi033 ceph-mon[27845]: pgmap v14810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27087.112729 secs 2024-04-26T19:22:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:28 smithi096 ceph-mon[32753]: pgmap v14810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27087.112729 secs 2024-04-26T19:22:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27090.012429 secs 2024-04-26T19:22:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27090.012429 secs 2024-04-26T19:22:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27090.012429 secs 2024-04-26T19:22:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:30 smithi194 ceph-mon[33017]: pgmap v14811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:30 smithi033 ceph-mon[27845]: pgmap v14811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:30 smithi096 ceph-mon[32753]: pgmap v14811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:32 smithi194 ceph-mon[33017]: pgmap v14812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:32 smithi033 ceph-mon[27845]: pgmap v14812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:32 smithi096 ceph-mon[32753]: pgmap v14812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27092.112810 secs 2024-04-26T19:22:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27092.112810 secs 2024-04-26T19:22:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27092.112810 secs 2024-04-26T19:22:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27095.012549 secs 2024-04-26T19:22:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:34 smithi194 ceph-mon[33017]: pgmap v14813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27095.012549 secs 2024-04-26T19:22:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:34 smithi033 ceph-mon[27845]: pgmap v14813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27095.012549 secs 2024-04-26T19:22:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:34 smithi096 ceph-mon[32753]: pgmap v14813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:36 smithi194 ceph-mon[33017]: pgmap v14814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:36 smithi033 ceph-mon[27845]: pgmap v14814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:36 smithi096 ceph-mon[32753]: pgmap v14814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:37.667+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:37.667+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:37.667+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:22:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:38 smithi194 ceph-mon[33017]: pgmap v14815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:39.180 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27097.112897 secs 2024-04-26T19:22:39.181 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:38 smithi033 ceph-mon[27845]: pgmap v14815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:39.181 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27097.112897 secs 2024-04-26T19:22:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:38 smithi096 ceph-mon[32753]: pgmap v14815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27097.112897 secs 2024-04-26T19:22:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27100.012664 secs 2024-04-26T19:22:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27100.012664 secs 2024-04-26T19:22:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27100.012664 secs 2024-04-26T19:22:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:40 smithi194 ceph-mon[33017]: pgmap v14816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:40 smithi033 ceph-mon[27845]: pgmap v14816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:40 smithi096 ceph-mon[32753]: pgmap v14816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:42 smithi194 ceph-mon[33017]: pgmap v14817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:42 smithi033 ceph-mon[27845]: pgmap v14817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:42 smithi096 ceph-mon[32753]: pgmap v14817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27102.112999 secs 2024-04-26T19:22:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27102.112999 secs 2024-04-26T19:22:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27102.112999 secs 2024-04-26T19:22:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27105.012780 secs 2024-04-26T19:22:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:44 smithi194 ceph-mon[33017]: pgmap v14818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27105.012780 secs 2024-04-26T19:22:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:44 smithi033 ceph-mon[27845]: pgmap v14818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27105.012780 secs 2024-04-26T19:22:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:44 smithi096 ceph-mon[32753]: pgmap v14818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:46 smithi194 ceph-mon[33017]: pgmap v14819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:46 smithi033 ceph-mon[27845]: pgmap v14819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:46 smithi096 ceph-mon[32753]: pgmap v14819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:47 smithi194 ceph-mon[33017]: pgmap v14820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:47 smithi033 ceph-mon[27845]: pgmap v14820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:47 smithi096 ceph-mon[32753]: pgmap v14820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27107.113085 secs 2024-04-26T19:22:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27107.113085 secs 2024-04-26T19:22:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27107.113085 secs 2024-04-26T19:22:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27110.012894 secs 2024-04-26T19:22:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:49 smithi194 ceph-mon[33017]: pgmap v14821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27110.012894 secs 2024-04-26T19:22:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:49 smithi033 ceph-mon[27845]: pgmap v14821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27110.012894 secs 2024-04-26T19:22:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:49 smithi096 ceph-mon[32753]: pgmap v14821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:52.682+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:52.682+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:22:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:22:52.682+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:22:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:52 smithi194 ceph-mon[33017]: pgmap v14822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:52 smithi033 ceph-mon[27845]: pgmap v14822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:52 smithi096 ceph-mon[32753]: pgmap v14822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27112.113173 secs 2024-04-26T19:22:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27112.113173 secs 2024-04-26T19:22:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27112.113173 secs 2024-04-26T19:22:55.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27115.013049 secs 2024-04-26T19:22:55.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:54 smithi033 ceph-mon[27845]: pgmap v14823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27115.013049 secs 2024-04-26T19:22:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:54 smithi194 ceph-mon[33017]: pgmap v14823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27115.013049 secs 2024-04-26T19:22:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:54 smithi096 ceph-mon[32753]: pgmap v14823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:56 smithi194 ceph-mon[33017]: pgmap v14824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:56 smithi033 ceph-mon[27845]: pgmap v14824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:56 smithi096 ceph-mon[32753]: pgmap v14824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:22:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:58 smithi194 ceph-mon[33017]: pgmap v14825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27117.113254 secs 2024-04-26T19:22:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:58 smithi033 ceph-mon[27845]: pgmap v14825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27117.113254 secs 2024-04-26T19:22:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:58 smithi096 ceph-mon[32753]: pgmap v14825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:22:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27117.113254 secs 2024-04-26T19:23:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27120.013191 secs 2024-04-26T19:23:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27120.013191 secs 2024-04-26T19:23:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27120.013191 secs 2024-04-26T19:23:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:00 smithi194 ceph-mon[33017]: pgmap v14826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:00 smithi033 ceph-mon[27845]: pgmap v14826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:00 smithi096 ceph-mon[32753]: pgmap v14826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:02 smithi194 ceph-mon[33017]: pgmap v14827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:02 smithi033 ceph-mon[27845]: pgmap v14827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:02 smithi096 ceph-mon[32753]: pgmap v14827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27122.113337 secs 2024-04-26T19:23:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27122.113337 secs 2024-04-26T19:23:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27122.113337 secs 2024-04-26T19:23:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27125.013303 secs 2024-04-26T19:23:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:04 smithi194 ceph-mon[33017]: pgmap v14828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27125.013303 secs 2024-04-26T19:23:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:04 smithi033 ceph-mon[27845]: pgmap v14828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27125.013303 secs 2024-04-26T19:23:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:04 smithi096 ceph-mon[32753]: pgmap v14828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:06 smithi194 ceph-mon[33017]: pgmap v14829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:06 smithi033 ceph-mon[27845]: pgmap v14829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:06 smithi096 ceph-mon[32753]: pgmap v14829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:07.690+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:07.690+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:07.690+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:23:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:08 smithi194 ceph-mon[33017]: pgmap v14830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27127.113443 secs 2024-04-26T19:23:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:08 smithi033 ceph-mon[27845]: pgmap v14830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27127.113443 secs 2024-04-26T19:23:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:08 smithi096 ceph-mon[32753]: pgmap v14830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27127.113443 secs 2024-04-26T19:23:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27130.013418 secs 2024-04-26T19:23:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27130.013418 secs 2024-04-26T19:23:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27130.013418 secs 2024-04-26T19:23:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:10 smithi194 ceph-mon[33017]: pgmap v14831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:10 smithi033 ceph-mon[27845]: pgmap v14831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:10 smithi096 ceph-mon[32753]: pgmap v14831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:11 smithi194 ceph-mon[33017]: pgmap v14832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:11 smithi033 ceph-mon[27845]: pgmap v14832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:11 smithi096 ceph-mon[32753]: pgmap v14832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:23:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:23:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27132.113500 secs 2024-04-26T19:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:23:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:23:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:13 smithi033 ceph-mon[27845]: pgmap v14833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:13 smithi033 ceph-mon[27845]: pgmap v14834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27132.113500 secs 2024-04-26T19:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:13 smithi096 ceph-mon[32753]: pgmap v14833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:13 smithi096 ceph-mon[32753]: pgmap v14834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27132.113500 secs 2024-04-26T19:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:13 smithi194 ceph-mon[33017]: pgmap v14833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:13 smithi194 ceph-mon[33017]: pgmap v14834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:23:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27135.013549 secs 2024-04-26T19:23:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27135.013549 secs 2024-04-26T19:23:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27135.013549 secs 2024-04-26T19:23:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:15 smithi033 ceph-mon[27845]: pgmap v14835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:15 smithi096 ceph-mon[32753]: pgmap v14835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:15 smithi194 ceph-mon[33017]: pgmap v14835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:18 smithi033 ceph-mon[27845]: pgmap v14836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27137.113562 secs 2024-04-26T19:23:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:18 smithi096 ceph-mon[32753]: pgmap v14836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27137.113562 secs 2024-04-26T19:23:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:18 smithi194 ceph-mon[33017]: pgmap v14836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27137.113562 secs 2024-04-26T19:23:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27140.013666 secs 2024-04-26T19:23:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27140.013666 secs 2024-04-26T19:23:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27140.013666 secs 2024-04-26T19:23:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:20 smithi033 ceph-mon[27845]: pgmap v14837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:23:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:20 smithi096 ceph-mon[32753]: pgmap v14837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:23:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:20 smithi194 ceph-mon[33017]: pgmap v14837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:23:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:22 smithi033 ceph-mon[27845]: pgmap v14838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:22 smithi096 ceph-mon[32753]: pgmap v14838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:22.705+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:22.705+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:22.705+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:23:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:22 smithi194 ceph-mon[33017]: pgmap v14838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27142.113642 secs 2024-04-26T19:23:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27142.113642 secs 2024-04-26T19:23:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27142.113642 secs 2024-04-26T19:23:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:24 smithi033 ceph-mon[27845]: pgmap v14839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:23:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27145.013786 secs 2024-04-26T19:23:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:24 smithi096 ceph-mon[32753]: pgmap v14839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:23:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27145.013786 secs 2024-04-26T19:23:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:24 smithi194 ceph-mon[33017]: pgmap v14839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:23:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27145.013786 secs 2024-04-26T19:23:26.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.412+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:23:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.412+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00066b0 con 0x7fef00077780 2024-04-26T19:23:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.413+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 214 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feee80d00e0 con 0x564385821900 2024-04-26T19:23:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.413+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T19:23:26.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.414+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 214 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T19:23:26.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.414+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:23:26.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.414+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 215 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:23:26.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:26.414+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 215 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:23:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:26 smithi096 ceph-mon[32753]: pgmap v14840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:26 smithi194 ceph-mon[33017]: pgmap v14840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:26 smithi033 ceph-mon[27845]: pgmap v14840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:28 smithi194 ceph-mon[33017]: pgmap v14841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27147.113742 secs 2024-04-26T19:23:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:28 smithi033 ceph-mon[27845]: pgmap v14841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27147.113742 secs 2024-04-26T19:23:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:28 smithi096 ceph-mon[32753]: pgmap v14841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27147.113742 secs 2024-04-26T19:23:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27150.013887 secs 2024-04-26T19:23:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27150.013887 secs 2024-04-26T19:23:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27150.013887 secs 2024-04-26T19:23:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:30 smithi194 ceph-mon[33017]: pgmap v14842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:30 smithi033 ceph-mon[27845]: pgmap v14842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:30 smithi096 ceph-mon[32753]: pgmap v14842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:32 smithi194 ceph-mon[33017]: pgmap v14843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:32 smithi033 ceph-mon[27845]: pgmap v14843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:32 smithi096 ceph-mon[32753]: pgmap v14843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27152.113826 secs 2024-04-26T19:23:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27152.113826 secs 2024-04-26T19:23:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27152.113826 secs 2024-04-26T19:23:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:34 smithi194 ceph-mon[33017]: pgmap v14844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27155.014006 secs 2024-04-26T19:23:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:34 smithi033 ceph-mon[27845]: pgmap v14844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27155.014006 secs 2024-04-26T19:23:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:34 smithi096 ceph-mon[32753]: pgmap v14844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27155.014006 secs 2024-04-26T19:23:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:36 smithi194 ceph-mon[33017]: pgmap v14845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:36 smithi033 ceph-mon[27845]: pgmap v14845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:36 smithi096 ceph-mon[32753]: pgmap v14845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:37.714+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:37.714+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:37.715+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:23:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:38 smithi194 ceph-mon[33017]: pgmap v14846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:38 smithi033 ceph-mon[27845]: pgmap v14846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:38 smithi096 ceph-mon[32753]: pgmap v14846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27157.113906 secs 2024-04-26T19:23:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27160.014146 secs 2024-04-26T19:23:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27157.113906 secs 2024-04-26T19:23:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27160.014146 secs 2024-04-26T19:23:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27157.113906 secs 2024-04-26T19:23:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27160.014146 secs 2024-04-26T19:23:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:40 smithi194 ceph-mon[33017]: pgmap v14847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:40 smithi033 ceph-mon[27845]: pgmap v14847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:40 smithi096 ceph-mon[32753]: pgmap v14847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:42 smithi033 ceph-mon[27845]: pgmap v14848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:42 smithi096 ceph-mon[32753]: pgmap v14848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:42 smithi194 ceph-mon[33017]: pgmap v14848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27162.114006 secs 2024-04-26T19:23:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27162.114006 secs 2024-04-26T19:23:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27162.114006 secs 2024-04-26T19:23:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:44 smithi033 ceph-mon[27845]: pgmap v14849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27165.014273 secs 2024-04-26T19:23:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:44 smithi096 ceph-mon[32753]: pgmap v14849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27165.014273 secs 2024-04-26T19:23:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:44 smithi194 ceph-mon[33017]: pgmap v14849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27165.014273 secs 2024-04-26T19:23:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:46 smithi033 ceph-mon[27845]: pgmap v14850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:46 smithi096 ceph-mon[32753]: pgmap v14850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:46 smithi194 ceph-mon[33017]: pgmap v14850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:48 smithi096 ceph-mon[32753]: pgmap v14851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27167.114095 secs 2024-04-26T19:23:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:48 smithi194 ceph-mon[33017]: pgmap v14851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27167.114095 secs 2024-04-26T19:23:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:48 smithi033 ceph-mon[27845]: pgmap v14851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27167.114095 secs 2024-04-26T19:23:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27170.014337 secs 2024-04-26T19:23:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27170.014337 secs 2024-04-26T19:23:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27170.014337 secs 2024-04-26T19:23:50.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:50 smithi096 ceph-mon[32753]: pgmap v14852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:50 smithi194 ceph-mon[33017]: pgmap v14852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:50 smithi033 ceph-mon[27845]: pgmap v14852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:23:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:52.729+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:52.729+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:23:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:23:52.729+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:23:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:52 smithi194 ceph-mon[33017]: pgmap v14853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:52 smithi033 ceph-mon[27845]: pgmap v14853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:52.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:52 smithi096 ceph-mon[32753]: pgmap v14853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:23:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27172.114179 secs 2024-04-26T19:23:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27172.114179 secs 2024-04-26T19:23:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27172.114179 secs 2024-04-26T19:23:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:54 smithi194 ceph-mon[33017]: pgmap v14854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27175.014460 secs 2024-04-26T19:23:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:54 smithi033 ceph-mon[27845]: pgmap v14854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27175.014460 secs 2024-04-26T19:23:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:54 smithi096 ceph-mon[32753]: pgmap v14854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27175.014460 secs 2024-04-26T19:23:56.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:56 smithi033 ceph-mon[27845]: pgmap v14855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:56 smithi194 ceph-mon[33017]: pgmap v14855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:56 smithi096 ceph-mon[32753]: pgmap v14855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:23:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:58 smithi194 ceph-mon[33017]: pgmap v14856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27177.114285 secs 2024-04-26T19:23:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:58 smithi033 ceph-mon[27845]: pgmap v14856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27177.114285 secs 2024-04-26T19:23:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:58 smithi096 ceph-mon[32753]: pgmap v14856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:23:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27177.114285 secs 2024-04-26T19:23:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:23:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27180.014558 secs 2024-04-26T19:23:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:23:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27180.014558 secs 2024-04-26T19:23:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:23:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27180.014558 secs 2024-04-26T19:24:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:00 smithi194 ceph-mon[33017]: pgmap v14857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:00 smithi033 ceph-mon[27845]: pgmap v14857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:00 smithi096 ceph-mon[32753]: pgmap v14857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:02 smithi194 ceph-mon[33017]: pgmap v14858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:24:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:02 smithi033 ceph-mon[27845]: pgmap v14858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:24:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:02 smithi096 ceph-mon[32753]: pgmap v14858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:24:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27182.114345 secs 2024-04-26T19:24:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27182.114345 secs 2024-04-26T19:24:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27182.114345 secs 2024-04-26T19:24:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:04 smithi194 ceph-mon[33017]: pgmap v14859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27185.014626 secs 2024-04-26T19:24:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:04 smithi033 ceph-mon[27845]: pgmap v14859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27185.014626 secs 2024-04-26T19:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:04 smithi096 ceph-mon[32753]: pgmap v14859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27185.014626 secs 2024-04-26T19:24:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:06 smithi194 ceph-mon[33017]: pgmap v14860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:06 smithi033 ceph-mon[27845]: pgmap v14860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:06 smithi096 ceph-mon[32753]: pgmap v14860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:07.742+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:07.742+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:07.742+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:24:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:08 smithi194 ceph-mon[33017]: pgmap v14861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27187.114430 secs 2024-04-26T19:24:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:08 smithi033 ceph-mon[27845]: pgmap v14861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27187.114430 secs 2024-04-26T19:24:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:08 smithi096 ceph-mon[32753]: pgmap v14861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27187.114430 secs 2024-04-26T19:24:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27190.014762 secs 2024-04-26T19:24:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27190.014762 secs 2024-04-26T19:24:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27190.014762 secs 2024-04-26T19:24:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:10 smithi194 ceph-mon[33017]: pgmap v14862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:10 smithi033 ceph-mon[27845]: pgmap v14862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:10 smithi096 ceph-mon[32753]: pgmap v14862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:12 smithi194 ceph-mon[33017]: pgmap v14863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:24:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:12 smithi033 ceph-mon[27845]: pgmap v14863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:24:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:12 smithi096 ceph-mon[32753]: pgmap v14863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:24:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27192.114516 secs 2024-04-26T19:24:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27192.114516 secs 2024-04-26T19:24:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27192.114516 secs 2024-04-26T19:24:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:14 smithi194 ceph-mon[33017]: pgmap v14864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27195.014882 secs 2024-04-26T19:24:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:24:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:14 smithi033 ceph-mon[27845]: pgmap v14864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27195.014882 secs 2024-04-26T19:24:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:24:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:14 smithi096 ceph-mon[32753]: pgmap v14864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27195.014882 secs 2024-04-26T19:24:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:24:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:24:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:24:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:24:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:24:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:24:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:24:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:24:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:24:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:24:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:16 smithi194 ceph-mon[33017]: pgmap v14865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:24:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:16 smithi194 ceph-mon[33017]: pgmap v14866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:16 smithi033 ceph-mon[27845]: pgmap v14865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:24:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:16 smithi033 ceph-mon[27845]: pgmap v14866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:16 smithi096 ceph-mon[32753]: pgmap v14865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:24:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:16 smithi096 ceph-mon[32753]: pgmap v14866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:18 smithi194 ceph-mon[33017]: pgmap v14867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27197.114605 secs 2024-04-26T19:24:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:18 smithi033 ceph-mon[27845]: pgmap v14867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27197.114605 secs 2024-04-26T19:24:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:18 smithi096 ceph-mon[32753]: pgmap v14867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27197.114605 secs 2024-04-26T19:24:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27200.014978 secs 2024-04-26T19:24:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27200.014978 secs 2024-04-26T19:24:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27200.014978 secs 2024-04-26T19:24:20.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:20 smithi096 ceph-mon[32753]: pgmap v14868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:24:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:20 smithi194 ceph-mon[33017]: pgmap v14868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:24:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:20 smithi033 ceph-mon[27845]: pgmap v14868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:24:22.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:22.749+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:22.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:22.749+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:22.750 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:22.749+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:24:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:22 smithi194 ceph-mon[33017]: pgmap v14869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:22 smithi033 ceph-mon[27845]: pgmap v14869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:22 smithi096 ceph-mon[32753]: pgmap v14869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27202.114692 secs 2024-04-26T19:24:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27202.114692 secs 2024-04-26T19:24:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27202.114692 secs 2024-04-26T19:24:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:24 smithi194 ceph-mon[33017]: pgmap v14870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27205.015096 secs 2024-04-26T19:24:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:24 smithi033 ceph-mon[27845]: pgmap v14870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27205.015096 secs 2024-04-26T19:24:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:24 smithi096 ceph-mon[32753]: pgmap v14870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:24:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27205.015096 secs 2024-04-26T19:24:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:26 smithi033 ceph-mon[27845]: pgmap v14871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:24:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:26 smithi096 ceph-mon[32753]: pgmap v14871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:24:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:26 smithi194 ceph-mon[33017]: pgmap v14871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:24:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:28 smithi033 ceph-mon[27845]: pgmap v14872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27207.114792 secs 2024-04-26T19:24:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:28 smithi096 ceph-mon[32753]: pgmap v14872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27207.114792 secs 2024-04-26T19:24:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:28 smithi194 ceph-mon[33017]: pgmap v14872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27207.114792 secs 2024-04-26T19:24:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27210.015210 secs 2024-04-26T19:24:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27210.015210 secs 2024-04-26T19:24:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27210.015210 secs 2024-04-26T19:24:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:30 smithi033 ceph-mon[27845]: pgmap v14873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:30 smithi096 ceph-mon[32753]: pgmap v14873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:30 smithi194 ceph-mon[33017]: pgmap v14873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:32 smithi033 ceph-mon[27845]: pgmap v14874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:32 smithi096 ceph-mon[32753]: pgmap v14874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:32 smithi194 ceph-mon[33017]: pgmap v14874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27212.114893 secs 2024-04-26T19:24:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27212.114893 secs 2024-04-26T19:24:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27212.114893 secs 2024-04-26T19:24:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:34 smithi033 ceph-mon[27845]: pgmap v14875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27215.015332 secs 2024-04-26T19:24:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:34 smithi096 ceph-mon[32753]: pgmap v14875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27215.015332 secs 2024-04-26T19:24:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:34 smithi194 ceph-mon[33017]: pgmap v14875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27215.015332 secs 2024-04-26T19:24:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:36 smithi033 ceph-mon[27845]: pgmap v14876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:36 smithi096 ceph-mon[32753]: pgmap v14876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:36 smithi194 ceph-mon[33017]: pgmap v14876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:37.764+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:37.764+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:37.765+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:24:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:38 smithi033 ceph-mon[27845]: pgmap v14877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27217.115001 secs 2024-04-26T19:24:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:38 smithi096 ceph-mon[32753]: pgmap v14877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27217.115001 secs 2024-04-26T19:24:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:38 smithi194 ceph-mon[33017]: pgmap v14877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27217.115001 secs 2024-04-26T19:24:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27220.015411 secs 2024-04-26T19:24:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27220.015411 secs 2024-04-26T19:24:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27220.015411 secs 2024-04-26T19:24:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:40 smithi033 ceph-mon[27845]: pgmap v14878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:40 smithi096 ceph-mon[32753]: pgmap v14878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:40 smithi194 ceph-mon[33017]: pgmap v14878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:42 smithi033 ceph-mon[27845]: pgmap v14879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:42 smithi096 ceph-mon[32753]: pgmap v14879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:42 smithi194 ceph-mon[33017]: pgmap v14879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27222.115105 secs 2024-04-26T19:24:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27222.115105 secs 2024-04-26T19:24:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27222.115105 secs 2024-04-26T19:24:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:44 smithi194 ceph-mon[33017]: pgmap v14880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27225.015540 secs 2024-04-26T19:24:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:44 smithi033 ceph-mon[27845]: pgmap v14880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27225.015540 secs 2024-04-26T19:24:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:44 smithi096 ceph-mon[32753]: pgmap v14880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27225.015540 secs 2024-04-26T19:24:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:46 smithi194 ceph-mon[33017]: pgmap v14881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:46 smithi033 ceph-mon[27845]: pgmap v14881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:46 smithi096 ceph-mon[32753]: pgmap v14881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:48 smithi194 ceph-mon[33017]: pgmap v14882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27227.115167 secs 2024-04-26T19:24:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:48 smithi033 ceph-mon[27845]: pgmap v14882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27227.115167 secs 2024-04-26T19:24:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:48 smithi096 ceph-mon[32753]: pgmap v14882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27227.115167 secs 2024-04-26T19:24:50.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27230.015663 secs 2024-04-26T19:24:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27230.015663 secs 2024-04-26T19:24:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27230.015663 secs 2024-04-26T19:24:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:50 smithi194 ceph-mon[33017]: pgmap v14883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:50 smithi033 ceph-mon[27845]: pgmap v14883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:50 smithi096 ceph-mon[32753]: pgmap v14883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:52.772+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:52.772+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:24:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:24:52.772+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:24:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:52 smithi194 ceph-mon[33017]: pgmap v14884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:52 smithi033 ceph-mon[27845]: pgmap v14884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:52 smithi096 ceph-mon[32753]: pgmap v14884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27232.115247 secs 2024-04-26T19:24:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27232.115247 secs 2024-04-26T19:24:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27232.115247 secs 2024-04-26T19:24:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:54 smithi194 ceph-mon[33017]: pgmap v14885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27235.015755 secs 2024-04-26T19:24:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:54 smithi033 ceph-mon[27845]: pgmap v14885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27235.015755 secs 2024-04-26T19:24:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:54 smithi096 ceph-mon[32753]: pgmap v14885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27235.015755 secs 2024-04-26T19:24:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:56 smithi194 ceph-mon[33017]: pgmap v14886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:56 smithi033 ceph-mon[27845]: pgmap v14886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:56 smithi096 ceph-mon[32753]: pgmap v14886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:24:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:58 smithi194 ceph-mon[33017]: pgmap v14887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27237.115333 secs 2024-04-26T19:24:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:58 smithi033 ceph-mon[27845]: pgmap v14887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27237.115333 secs 2024-04-26T19:24:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:58 smithi096 ceph-mon[32753]: pgmap v14887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:24:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27237.115333 secs 2024-04-26T19:25:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27240.015888 secs 2024-04-26T19:25:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27240.015888 secs 2024-04-26T19:25:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27240.015888 secs 2024-04-26T19:25:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:00 smithi194 ceph-mon[33017]: pgmap v14888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:00 smithi033 ceph-mon[27845]: pgmap v14888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:00 smithi096 ceph-mon[32753]: pgmap v14888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:02 smithi194 ceph-mon[33017]: pgmap v14889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:03.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:02 smithi033 ceph-mon[27845]: pgmap v14889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:02 smithi096 ceph-mon[32753]: pgmap v14889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27242.115419 secs 2024-04-26T19:25:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27242.115419 secs 2024-04-26T19:25:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27242.115419 secs 2024-04-26T19:25:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:04 smithi194 ceph-mon[33017]: pgmap v14890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27245.016014 secs 2024-04-26T19:25:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:04 smithi033 ceph-mon[27845]: pgmap v14890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27245.016014 secs 2024-04-26T19:25:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:04 smithi096 ceph-mon[32753]: pgmap v14890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27245.016014 secs 2024-04-26T19:25:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:06 smithi194 ceph-mon[33017]: pgmap v14891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:06 smithi033 ceph-mon[27845]: pgmap v14891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:06 smithi096 ceph-mon[32753]: pgmap v14891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:07.774+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:07.774+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:07.774+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:25:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:08 smithi194 ceph-mon[33017]: pgmap v14892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27247.115482 secs 2024-04-26T19:25:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:08 smithi033 ceph-mon[27845]: pgmap v14892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27247.115482 secs 2024-04-26T19:25:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:08 smithi096 ceph-mon[32753]: pgmap v14892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27247.115482 secs 2024-04-26T19:25:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27250.016153 secs 2024-04-26T19:25:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27250.016153 secs 2024-04-26T19:25:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27250.016153 secs 2024-04-26T19:25:11.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:10 smithi194 ceph-mon[33017]: pgmap v14893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:10 smithi033 ceph-mon[27845]: pgmap v14893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:10 smithi096 ceph-mon[32753]: pgmap v14893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:12 smithi194 ceph-mon[33017]: pgmap v14894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:12 smithi033 ceph-mon[27845]: pgmap v14894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:12 smithi096 ceph-mon[32753]: pgmap v14894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27252.115568 secs 2024-04-26T19:25:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27252.115568 secs 2024-04-26T19:25:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27252.115568 secs 2024-04-26T19:25:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:14 smithi194 ceph-mon[33017]: pgmap v14895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27255.016275 secs 2024-04-26T19:25:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:14 smithi033 ceph-mon[27845]: pgmap v14895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27255.016275 secs 2024-04-26T19:25:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:14 smithi096 ceph-mon[32753]: pgmap v14895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27255.016275 secs 2024-04-26T19:25:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:25:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:25:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:25:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:16 smithi194 ceph-mon[33017]: pgmap v14896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:25:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:25:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:25:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:16 smithi033 ceph-mon[27845]: pgmap v14896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:25:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:25:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:25:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:16 smithi096 ceph-mon[32753]: pgmap v14896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:25:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:25:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:25:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:17 smithi194 ceph-mon[33017]: pgmap v14897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:17 smithi194 ceph-mon[33017]: pgmap v14898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:17 smithi033 ceph-mon[27845]: pgmap v14897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:17 smithi033 ceph-mon[27845]: pgmap v14898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:17 smithi096 ceph-mon[32753]: pgmap v14897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:17 smithi096 ceph-mon[32753]: pgmap v14898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27257.115656 secs 2024-04-26T19:25:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27257.115656 secs 2024-04-26T19:25:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27257.115656 secs 2024-04-26T19:25:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27260.016399 secs 2024-04-26T19:25:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:19 smithi194 ceph-mon[33017]: pgmap v14899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27260.016399 secs 2024-04-26T19:25:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:19 smithi033 ceph-mon[27845]: pgmap v14899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27260.016399 secs 2024-04-26T19:25:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:19 smithi096 ceph-mon[32753]: pgmap v14899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:20 smithi033 ceph-mon[27845]: pgmap v14900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:20 smithi096 ceph-mon[32753]: pgmap v14900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:20 smithi194 ceph-mon[33017]: pgmap v14900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:22.787+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:22.787+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:22.787+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:25:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:23 smithi194 ceph-mon[33017]: pgmap v14901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:23 smithi033 ceph-mon[27845]: pgmap v14901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:23 smithi096 ceph-mon[32753]: pgmap v14901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27262.115737 secs 2024-04-26T19:25:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27265.016516 secs 2024-04-26T19:25:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27262.115737 secs 2024-04-26T19:25:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27265.016516 secs 2024-04-26T19:25:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27262.115737 secs 2024-04-26T19:25:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27265.016516 secs 2024-04-26T19:25:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:25 smithi194 ceph-mon[33017]: pgmap v14902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:25 smithi033 ceph-mon[27845]: pgmap v14902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:25 smithi096 ceph-mon[32753]: pgmap v14902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:25:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:26 smithi033 ceph-mon[27845]: pgmap v14903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:25:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:26 smithi096 ceph-mon[32753]: pgmap v14903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:25:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:26 smithi194 ceph-mon[33017]: pgmap v14903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:25:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27267.115821 secs 2024-04-26T19:25:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27267.115821 secs 2024-04-26T19:25:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27267.115821 secs 2024-04-26T19:25:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27270.016636 secs 2024-04-26T19:25:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:29 smithi194 ceph-mon[33017]: pgmap v14904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27270.016636 secs 2024-04-26T19:25:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:29 smithi033 ceph-mon[27845]: pgmap v14904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27270.016636 secs 2024-04-26T19:25:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:29 smithi096 ceph-mon[32753]: pgmap v14904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:30 smithi033 ceph-mon[27845]: pgmap v14905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:30 smithi096 ceph-mon[32753]: pgmap v14905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:30 smithi194 ceph-mon[33017]: pgmap v14905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:32 smithi033 ceph-mon[27845]: pgmap v14906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:32 smithi096 ceph-mon[32753]: pgmap v14906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:32 smithi194 ceph-mon[33017]: pgmap v14906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27272.115904 secs 2024-04-26T19:25:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27272.115904 secs 2024-04-26T19:25:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27272.115904 secs 2024-04-26T19:25:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27275.016751 secs 2024-04-26T19:25:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:34 smithi033 ceph-mon[27845]: pgmap v14907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27275.016751 secs 2024-04-26T19:25:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:34 smithi096 ceph-mon[32753]: pgmap v14907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27275.016751 secs 2024-04-26T19:25:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:34 smithi194 ceph-mon[33017]: pgmap v14907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:36 smithi033 ceph-mon[27845]: pgmap v14908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:36 smithi096 ceph-mon[32753]: pgmap v14908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:36 smithi194 ceph-mon[33017]: pgmap v14908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:37.801+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:37.801+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:37.801+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:25:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27277.115994 secs 2024-04-26T19:25:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27277.115994 secs 2024-04-26T19:25:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27277.115994 secs 2024-04-26T19:25:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27280.016879 secs 2024-04-26T19:25:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:39 smithi194 ceph-mon[33017]: pgmap v14909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27280.016879 secs 2024-04-26T19:25:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:39 smithi033 ceph-mon[27845]: pgmap v14909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27280.016879 secs 2024-04-26T19:25:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:39 smithi096 ceph-mon[32753]: pgmap v14909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:40 smithi033 ceph-mon[27845]: pgmap v14910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:40 smithi096 ceph-mon[32753]: pgmap v14910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:40 smithi194 ceph-mon[33017]: pgmap v14910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:42 smithi033 ceph-mon[27845]: pgmap v14911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:42 smithi096 ceph-mon[32753]: pgmap v14911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:42 smithi194 ceph-mon[33017]: pgmap v14911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27282.116078 secs 2024-04-26T19:25:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27282.116078 secs 2024-04-26T19:25:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27282.116078 secs 2024-04-26T19:25:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27285.017002 secs 2024-04-26T19:25:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:44 smithi033 ceph-mon[27845]: pgmap v14912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27285.017002 secs 2024-04-26T19:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:44 smithi096 ceph-mon[32753]: pgmap v14912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27285.017002 secs 2024-04-26T19:25:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:44 smithi194 ceph-mon[33017]: pgmap v14912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:46 smithi033 ceph-mon[27845]: pgmap v14913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:46 smithi096 ceph-mon[32753]: pgmap v14913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:46 smithi194 ceph-mon[33017]: pgmap v14913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27287.116162 secs 2024-04-26T19:25:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27287.116162 secs 2024-04-26T19:25:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27287.116162 secs 2024-04-26T19:25:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27290.017142 secs 2024-04-26T19:25:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:49 smithi033 ceph-mon[27845]: pgmap v14914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27290.017142 secs 2024-04-26T19:25:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:49 smithi194 ceph-mon[33017]: pgmap v14914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27290.017142 secs 2024-04-26T19:25:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:49 smithi096 ceph-mon[32753]: pgmap v14914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:50 smithi033 ceph-mon[27845]: pgmap v14915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:50 smithi096 ceph-mon[32753]: pgmap v14915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:50 smithi194 ceph-mon[33017]: pgmap v14915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:52.815+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:52.815+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:25:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:25:52.815+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:25:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:52 smithi033 ceph-mon[27845]: pgmap v14916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:52 smithi096 ceph-mon[32753]: pgmap v14916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:52 smithi194 ceph-mon[33017]: pgmap v14916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27292.116246 secs 2024-04-26T19:25:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27292.116246 secs 2024-04-26T19:25:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27292.116246 secs 2024-04-26T19:25:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27295.017234 secs 2024-04-26T19:25:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:54 smithi033 ceph-mon[27845]: pgmap v14917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27295.017234 secs 2024-04-26T19:25:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:54 smithi096 ceph-mon[32753]: pgmap v14917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27295.017234 secs 2024-04-26T19:25:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:54 smithi194 ceph-mon[33017]: pgmap v14917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:25:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:56 smithi033 ceph-mon[27845]: pgmap v14918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:56 smithi096 ceph-mon[32753]: pgmap v14918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:56 smithi194 ceph-mon[33017]: pgmap v14918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27297.116330 secs 2024-04-26T19:25:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27297.116330 secs 2024-04-26T19:25:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27297.116330 secs 2024-04-26T19:25:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27300.017379 secs 2024-04-26T19:25:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:25:59 smithi194 ceph-mon[33017]: pgmap v14919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27300.017379 secs 2024-04-26T19:25:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:25:59 smithi033 ceph-mon[27845]: pgmap v14919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:25:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27300.017379 secs 2024-04-26T19:25:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:25:59 smithi096 ceph-mon[32753]: pgmap v14919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:00 smithi033 ceph-mon[27845]: pgmap v14920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:00 smithi096 ceph-mon[32753]: pgmap v14920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:00 smithi194 ceph-mon[33017]: pgmap v14920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:02 smithi033 ceph-mon[27845]: pgmap v14921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:02 smithi096 ceph-mon[32753]: pgmap v14921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:02 smithi194 ceph-mon[33017]: pgmap v14921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27302.116391 secs 2024-04-26T19:26:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27302.116391 secs 2024-04-26T19:26:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27302.116391 secs 2024-04-26T19:26:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27305.017497 secs 2024-04-26T19:26:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:04 smithi033 ceph-mon[27845]: pgmap v14922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27305.017497 secs 2024-04-26T19:26:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:04 smithi096 ceph-mon[32753]: pgmap v14922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27305.017497 secs 2024-04-26T19:26:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:04 smithi194 ceph-mon[33017]: pgmap v14922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:06 smithi033 ceph-mon[27845]: pgmap v14923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:06 smithi096 ceph-mon[32753]: pgmap v14923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:06 smithi194 ceph-mon[33017]: pgmap v14923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:07.817+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:07.817+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:07.817+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:26:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27307.116478 secs 2024-04-26T19:26:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27307.116478 secs 2024-04-26T19:26:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27307.116478 secs 2024-04-26T19:26:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27310.017620 secs 2024-04-26T19:26:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:09 smithi194 ceph-mon[33017]: pgmap v14924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27310.017620 secs 2024-04-26T19:26:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:09 smithi033 ceph-mon[27845]: pgmap v14924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27310.017620 secs 2024-04-26T19:26:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:09 smithi096 ceph-mon[32753]: pgmap v14924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:10 smithi033 ceph-mon[27845]: pgmap v14925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:10 smithi096 ceph-mon[32753]: pgmap v14925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:10 smithi194 ceph-mon[33017]: pgmap v14925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:12 smithi033 ceph-mon[27845]: pgmap v14926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:12 smithi096 ceph-mon[32753]: pgmap v14926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:12 smithi194 ceph-mon[33017]: pgmap v14926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27312.116562 secs 2024-04-26T19:26:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27312.116562 secs 2024-04-26T19:26:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27312.116562 secs 2024-04-26T19:26:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27315.017753 secs 2024-04-26T19:26:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:14 smithi033 ceph-mon[27845]: pgmap v14927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27315.017753 secs 2024-04-26T19:26:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:14 smithi096 ceph-mon[32753]: pgmap v14927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27315.017753 secs 2024-04-26T19:26:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:14 smithi194 ceph-mon[33017]: pgmap v14927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:16 smithi033 ceph-mon[27845]: pgmap v14928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:16 smithi096 ceph-mon[32753]: pgmap v14928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:16 smithi194 ceph-mon[33017]: pgmap v14928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:26:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:26:18.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:26:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27317.116667 secs 2024-04-26T19:26:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:26:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:26:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:18 smithi096 ceph-mon[32753]: pgmap v14929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:18 smithi096 ceph-mon[32753]: pgmap v14930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27317.116667 secs 2024-04-26T19:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:18 smithi194 ceph-mon[33017]: pgmap v14929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:18 smithi194 ceph-mon[33017]: pgmap v14930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:26:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27317.116667 secs 2024-04-26T19:26:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:26:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:26:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:18 smithi033 ceph-mon[27845]: pgmap v14929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:18 smithi033 ceph-mon[27845]: pgmap v14930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:26:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27320.017871 secs 2024-04-26T19:26:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27320.017871 secs 2024-04-26T19:26:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27320.017871 secs 2024-04-26T19:26:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:20 smithi194 ceph-mon[33017]: pgmap v14931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:20 smithi033 ceph-mon[27845]: pgmap v14931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:20 smithi096 ceph-mon[32753]: pgmap v14931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T19:26:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T19:26:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b3120 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef004f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T19:26:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b3120 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef004f040 comp rx=0 tx=0).stop 2024-04-26T19:26:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T19:26:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T19:26:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3c30 0x7feee40b3120 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:26:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.791+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:26:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.792+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:26:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:21.794+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0075070 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:26:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:22.818+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:22.818+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:22.818+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:26:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:23 smithi194 ceph-mon[33017]: pgmap v14932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27322.116740 secs 2024-04-26T19:26:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:23 smithi033 ceph-mon[27845]: pgmap v14932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27322.116740 secs 2024-04-26T19:26:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:23 smithi096 ceph-mon[32753]: pgmap v14932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27322.116740 secs 2024-04-26T19:26:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27325.018021 secs 2024-04-26T19:26:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27325.018021 secs 2024-04-26T19:26:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27325.018021 secs 2024-04-26T19:26:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:25 smithi194 ceph-mon[33017]: pgmap v14933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:26:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:25 smithi033 ceph-mon[27845]: pgmap v14933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:26:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:25 smithi096 ceph-mon[32753]: pgmap v14933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:26:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:26.443+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:26:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:26.443+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:26:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:26.444+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 216 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:26:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:26.444+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 216 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:26:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:27 smithi194 ceph-mon[33017]: pgmap v14934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:27 smithi033 ceph-mon[27845]: pgmap v14934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:27 smithi096 ceph-mon[32753]: pgmap v14934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:28.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27327.116823 secs 2024-04-26T19:26:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27327.116823 secs 2024-04-26T19:26:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27327.116823 secs 2024-04-26T19:26:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:29 smithi194 ceph-mon[33017]: pgmap v14935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:26:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27330.018140 secs 2024-04-26T19:26:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:29 smithi033 ceph-mon[27845]: pgmap v14935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:26:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27330.018140 secs 2024-04-26T19:26:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:29 smithi096 ceph-mon[32753]: pgmap v14935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:26:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27330.018140 secs 2024-04-26T19:26:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:31 smithi194 ceph-mon[33017]: pgmap v14936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:31 smithi033 ceph-mon[27845]: pgmap v14936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:31 smithi096 ceph-mon[32753]: pgmap v14936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:33 smithi194 ceph-mon[33017]: pgmap v14937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27332.116903 secs 2024-04-26T19:26:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:33 smithi033 ceph-mon[27845]: pgmap v14937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27332.116903 secs 2024-04-26T19:26:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:33 smithi096 ceph-mon[32753]: pgmap v14937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27332.116903 secs 2024-04-26T19:26:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27335.018212 secs 2024-04-26T19:26:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27335.018212 secs 2024-04-26T19:26:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27335.018212 secs 2024-04-26T19:26:35.831 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:35 smithi033 ceph-mon[27845]: pgmap v14938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:35 smithi194 ceph-mon[33017]: pgmap v14938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:35 smithi096 ceph-mon[32753]: pgmap v14938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:37.825+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:37.825+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:37.825+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:26:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:37 smithi194 ceph-mon[33017]: pgmap v14939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:37 smithi033 ceph-mon[27845]: pgmap v14939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:37 smithi096 ceph-mon[32753]: pgmap v14939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27337.117008 secs 2024-04-26T19:26:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27337.117008 secs 2024-04-26T19:26:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27337.117008 secs 2024-04-26T19:26:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:39 smithi194 ceph-mon[33017]: pgmap v14940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27340.018276 secs 2024-04-26T19:26:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:39 smithi033 ceph-mon[27845]: pgmap v14940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27340.018276 secs 2024-04-26T19:26:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:39 smithi096 ceph-mon[32753]: pgmap v14940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27340.018276 secs 2024-04-26T19:26:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:41 smithi194 ceph-mon[33017]: pgmap v14941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:41 smithi033 ceph-mon[27845]: pgmap v14941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:41 smithi096 ceph-mon[32753]: pgmap v14941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:43 smithi033 ceph-mon[27845]: pgmap v14942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27342.117065 secs 2024-04-26T19:26:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:43 smithi096 ceph-mon[32753]: pgmap v14942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27342.117065 secs 2024-04-26T19:26:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:43 smithi194 ceph-mon[33017]: pgmap v14942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27342.117065 secs 2024-04-26T19:26:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27345.018396 secs 2024-04-26T19:26:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27345.018396 secs 2024-04-26T19:26:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27345.018396 secs 2024-04-26T19:26:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:45 smithi033 ceph-mon[27845]: pgmap v14943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:45 smithi096 ceph-mon[32753]: pgmap v14943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:45 smithi194 ceph-mon[33017]: pgmap v14943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:47 smithi033 ceph-mon[27845]: pgmap v14944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:47 smithi096 ceph-mon[32753]: pgmap v14944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:47 smithi194 ceph-mon[33017]: pgmap v14944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27347.117154 secs 2024-04-26T19:26:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27347.117154 secs 2024-04-26T19:26:49.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27347.117154 secs 2024-04-26T19:26:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:49 smithi033 ceph-mon[27845]: pgmap v14945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27350.018506 secs 2024-04-26T19:26:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:49 smithi096 ceph-mon[32753]: pgmap v14945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27350.018506 secs 2024-04-26T19:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:49 smithi194 ceph-mon[33017]: pgmap v14945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27350.018506 secs 2024-04-26T19:26:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:51 smithi033 ceph-mon[27845]: pgmap v14946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:51 smithi096 ceph-mon[32753]: pgmap v14946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:51 smithi194 ceph-mon[33017]: pgmap v14946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:26:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:52.840+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:52.840+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:26:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:26:52.840+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:26:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:53 smithi033 ceph-mon[27845]: pgmap v14947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27352.117216 secs 2024-04-26T19:26:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:53 smithi096 ceph-mon[32753]: pgmap v14947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27352.117216 secs 2024-04-26T19:26:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:53 smithi194 ceph-mon[33017]: pgmap v14947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:26:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27352.117216 secs 2024-04-26T19:26:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27355.018627 secs 2024-04-26T19:26:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27355.018627 secs 2024-04-26T19:26:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27355.018627 secs 2024-04-26T19:26:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:55 smithi033 ceph-mon[27845]: pgmap v14948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:55 smithi096 ceph-mon[32753]: pgmap v14948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:55 smithi194 ceph-mon[33017]: pgmap v14948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:26:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:57 smithi033 ceph-mon[27845]: pgmap v14949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:57 smithi096 ceph-mon[32753]: pgmap v14949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:57 smithi194 ceph-mon[33017]: pgmap v14949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:26:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27357.117299 secs 2024-04-26T19:26:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27357.117299 secs 2024-04-26T19:26:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27357.117299 secs 2024-04-26T19:27:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:59 smithi194 ceph-mon[33017]: pgmap v14950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:26:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27360.018745 secs 2024-04-26T19:27:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:59 smithi033 ceph-mon[27845]: pgmap v14950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:26:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27360.018745 secs 2024-04-26T19:27:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:59 smithi096 ceph-mon[32753]: pgmap v14950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:26:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27360.018745 secs 2024-04-26T19:27:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:01 smithi194 ceph-mon[33017]: pgmap v14951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:01 smithi033 ceph-mon[27845]: pgmap v14951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:01 smithi096 ceph-mon[32753]: pgmap v14951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:03 smithi194 ceph-mon[33017]: pgmap v14952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27362.117387 secs 2024-04-26T19:27:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:03 smithi033 ceph-mon[27845]: pgmap v14952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27362.117387 secs 2024-04-26T19:27:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:03 smithi096 ceph-mon[32753]: pgmap v14952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27362.117387 secs 2024-04-26T19:27:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27365.018863 secs 2024-04-26T19:27:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27365.018863 secs 2024-04-26T19:27:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27365.018863 secs 2024-04-26T19:27:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:05 smithi194 ceph-mon[33017]: pgmap v14953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:27:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:05 smithi033 ceph-mon[27845]: pgmap v14953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:27:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:05 smithi096 ceph-mon[32753]: pgmap v14953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:27:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:07.853+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:07.853+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:07.853+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:27:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:07 smithi194 ceph-mon[33017]: pgmap v14954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:27:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:07 smithi033 ceph-mon[27845]: pgmap v14954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:27:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:07 smithi096 ceph-mon[32753]: pgmap v14954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:27:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27367.117449 secs 2024-04-26T19:27:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27367.117449 secs 2024-04-26T19:27:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27367.117449 secs 2024-04-26T19:27:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:09 smithi194 ceph-mon[33017]: pgmap v14955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27370.018999 secs 2024-04-26T19:27:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:09 smithi033 ceph-mon[27845]: pgmap v14955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27370.018999 secs 2024-04-26T19:27:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:09 smithi096 ceph-mon[32753]: pgmap v14955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27370.018999 secs 2024-04-26T19:27:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:11 smithi194 ceph-mon[33017]: pgmap v14956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:11 smithi033 ceph-mon[27845]: pgmap v14956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:11 smithi096 ceph-mon[32753]: pgmap v14956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:13 smithi194 ceph-mon[33017]: pgmap v14957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27372.117527 secs 2024-04-26T19:27:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:13 smithi033 ceph-mon[27845]: pgmap v14957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27372.117527 secs 2024-04-26T19:27:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:13 smithi096 ceph-mon[32753]: pgmap v14957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27372.117527 secs 2024-04-26T19:27:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27375.019121 secs 2024-04-26T19:27:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27375.019121 secs 2024-04-26T19:27:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27375.019121 secs 2024-04-26T19:27:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:15 smithi194 ceph-mon[33017]: pgmap v14958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:27:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:15 smithi033 ceph-mon[27845]: pgmap v14958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:27:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:15 smithi096 ceph-mon[32753]: pgmap v14958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:27:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:17 smithi194 ceph-mon[33017]: pgmap v14959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:27:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:17 smithi033 ceph-mon[27845]: pgmap v14959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:27:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:17 smithi096 ceph-mon[32753]: pgmap v14959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:27:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27377.117621 secs 2024-04-26T19:27:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27377.117621 secs 2024-04-26T19:27:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27377.117621 secs 2024-04-26T19:27:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:19 smithi194 ceph-mon[33017]: pgmap v14960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27380.019245 secs 2024-04-26T19:27:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:27:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:19 smithi033 ceph-mon[27845]: pgmap v14960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27380.019245 secs 2024-04-26T19:27:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:27:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:19 smithi096 ceph-mon[32753]: pgmap v14960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27380.019245 secs 2024-04-26T19:27:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:27:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:21 smithi194 ceph-mon[33017]: pgmap v14961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:27:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:27:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:21 smithi033 ceph-mon[27845]: pgmap v14961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:27:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:27:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:21 smithi096 ceph-mon[32753]: pgmap v14961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:27:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:27:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:27:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:22.863+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:22.863+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:22.863+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:27:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:22 smithi194 ceph-mon[33017]: pgmap v14962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:27:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:22 smithi194 ceph-mon[33017]: pgmap v14963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:22 smithi033 ceph-mon[27845]: pgmap v14962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:27:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:22 smithi033 ceph-mon[27845]: pgmap v14963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:22 smithi096 ceph-mon[32753]: pgmap v14962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:27:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:22 smithi096 ceph-mon[32753]: pgmap v14963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27382.117705 secs 2024-04-26T19:27:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27382.117705 secs 2024-04-26T19:27:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27382.117705 secs 2024-04-26T19:27:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:24 smithi194 ceph-mon[33017]: pgmap v14964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T19:27:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27385.019362 secs 2024-04-26T19:27:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:24 smithi033 ceph-mon[27845]: pgmap v14964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T19:27:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27385.019362 secs 2024-04-26T19:27:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:24 smithi096 ceph-mon[32753]: pgmap v14964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T19:27:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27385.019362 secs 2024-04-26T19:27:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:26 smithi194 ceph-mon[33017]: pgmap v14965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:26 smithi033 ceph-mon[27845]: pgmap v14965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:26 smithi096 ceph-mon[32753]: pgmap v14965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:28 smithi194 ceph-mon[33017]: pgmap v14966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27387.117789 secs 2024-04-26T19:27:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:28 smithi033 ceph-mon[27845]: pgmap v14966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27387.117789 secs 2024-04-26T19:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:28 smithi096 ceph-mon[32753]: pgmap v14966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:27:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27387.117789 secs 2024-04-26T19:27:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27390.019478 secs 2024-04-26T19:27:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27390.019478 secs 2024-04-26T19:27:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27390.019478 secs 2024-04-26T19:27:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:30 smithi194 ceph-mon[33017]: pgmap v14967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:27:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:30 smithi033 ceph-mon[27845]: pgmap v14967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:27:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:30 smithi096 ceph-mon[32753]: pgmap v14967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:27:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:32 smithi194 ceph-mon[33017]: pgmap v14968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:27:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:32 smithi033 ceph-mon[27845]: pgmap v14968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:27:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:32 smithi096 ceph-mon[32753]: pgmap v14968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:27:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27392.117874 secs 2024-04-26T19:27:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27392.117874 secs 2024-04-26T19:27:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27392.117874 secs 2024-04-26T19:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:34 smithi194 ceph-mon[33017]: pgmap v14969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27395.019579 secs 2024-04-26T19:27:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:34 smithi033 ceph-mon[27845]: pgmap v14969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27395.019579 secs 2024-04-26T19:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:34 smithi096 ceph-mon[32753]: pgmap v14969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27395.019579 secs 2024-04-26T19:27:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:36 smithi194 ceph-mon[33017]: pgmap v14970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:36 smithi033 ceph-mon[27845]: pgmap v14970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:36 smithi096 ceph-mon[32753]: pgmap v14970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:37.873+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:37.873+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:37.873+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:27:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:38 smithi033 ceph-mon[27845]: pgmap v14971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27397.117981 secs 2024-04-26T19:27:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:38 smithi096 ceph-mon[32753]: pgmap v14971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27397.117981 secs 2024-04-26T19:27:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:38 smithi194 ceph-mon[33017]: pgmap v14971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27397.117981 secs 2024-04-26T19:27:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27400.019655 secs 2024-04-26T19:27:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27400.019655 secs 2024-04-26T19:27:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27400.019655 secs 2024-04-26T19:27:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:40 smithi033 ceph-mon[27845]: pgmap v14972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:40 smithi096 ceph-mon[32753]: pgmap v14972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:40 smithi194 ceph-mon[33017]: pgmap v14972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:42 smithi033 ceph-mon[27845]: pgmap v14973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:42 smithi096 ceph-mon[32753]: pgmap v14973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:42 smithi194 ceph-mon[33017]: pgmap v14973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27402.118066 secs 2024-04-26T19:27:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:43 smithi033 ceph-mon[27845]: pgmap v14974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27402.118066 secs 2024-04-26T19:27:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:43 smithi096 ceph-mon[32753]: pgmap v14974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27402.118066 secs 2024-04-26T19:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:43 smithi194 ceph-mon[33017]: pgmap v14974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27405.019781 secs 2024-04-26T19:27:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27405.019781 secs 2024-04-26T19:27:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27405.019781 secs 2024-04-26T19:27:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:45 smithi033 ceph-mon[27845]: pgmap v14975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:45 smithi096 ceph-mon[32753]: pgmap v14975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:45 smithi194 ceph-mon[33017]: pgmap v14975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:48 smithi033 ceph-mon[27845]: pgmap v14976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:48 smithi096 ceph-mon[32753]: pgmap v14976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:48 smithi194 ceph-mon[33017]: pgmap v14976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27407.118154 secs 2024-04-26T19:27:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27407.118154 secs 2024-04-26T19:27:49.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27407.118154 secs 2024-04-26T19:27:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27410.019916 secs 2024-04-26T19:27:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:50 smithi033 ceph-mon[27845]: pgmap v14977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27410.019916 secs 2024-04-26T19:27:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:50 smithi096 ceph-mon[32753]: pgmap v14977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27410.019916 secs 2024-04-26T19:27:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:50 smithi194 ceph-mon[33017]: pgmap v14977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:52 smithi033 ceph-mon[27845]: pgmap v14978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:52 smithi096 ceph-mon[32753]: pgmap v14978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:52 smithi194 ceph-mon[33017]: pgmap v14978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:52.886+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:52.886+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:27:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:27:52.886+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:27:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27412.118214 secs 2024-04-26T19:27:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:54 smithi033 ceph-mon[27845]: pgmap v14979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27412.118214 secs 2024-04-26T19:27:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:54 smithi096 ceph-mon[32753]: pgmap v14979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27412.118214 secs 2024-04-26T19:27:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:54 smithi194 ceph-mon[33017]: pgmap v14979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27415.020037 secs 2024-04-26T19:27:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27415.020037 secs 2024-04-26T19:27:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27415.020037 secs 2024-04-26T19:27:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:56 smithi194 ceph-mon[33017]: pgmap v14980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:56 smithi033 ceph-mon[27845]: pgmap v14980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:56 smithi096 ceph-mon[32753]: pgmap v14980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:27:58.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:58 smithi033 ceph-mon[27845]: pgmap v14981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:58 smithi194 ceph-mon[33017]: pgmap v14981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:58 smithi096 ceph-mon[32753]: pgmap v14981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:27:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27417.118300 secs 2024-04-26T19:27:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27420.020119 secs 2024-04-26T19:27:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27417.118300 secs 2024-04-26T19:27:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27420.020119 secs 2024-04-26T19:27:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27417.118300 secs 2024-04-26T19:27:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27420.020119 secs 2024-04-26T19:28:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:00 smithi194 ceph-mon[33017]: pgmap v14982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:00 smithi033 ceph-mon[27845]: pgmap v14982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:00 smithi096 ceph-mon[32753]: pgmap v14982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:02 smithi194 ceph-mon[33017]: pgmap v14983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:02 smithi033 ceph-mon[27845]: pgmap v14983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:02 smithi096 ceph-mon[32753]: pgmap v14983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27422.118386 secs 2024-04-26T19:28:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:04 smithi194 ceph-mon[33017]: pgmap v14984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27425.020224 secs 2024-04-26T19:28:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27422.118386 secs 2024-04-26T19:28:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:04 smithi033 ceph-mon[27845]: pgmap v14984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27425.020224 secs 2024-04-26T19:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27422.118386 secs 2024-04-26T19:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:04 smithi096 ceph-mon[32753]: pgmap v14984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27425.020224 secs 2024-04-26T19:28:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:06 smithi194 ceph-mon[33017]: pgmap v14985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:06 smithi033 ceph-mon[27845]: pgmap v14985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:06 smithi096 ceph-mon[32753]: pgmap v14985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:07.902+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:07.902+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:07.902+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:28:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:08 smithi194 ceph-mon[33017]: pgmap v14986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:08 smithi033 ceph-mon[27845]: pgmap v14986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:08 smithi096 ceph-mon[32753]: pgmap v14986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27427.118489 secs 2024-04-26T19:28:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27430.020340 secs 2024-04-26T19:28:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27427.118489 secs 2024-04-26T19:28:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27430.020340 secs 2024-04-26T19:28:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27427.118489 secs 2024-04-26T19:28:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27430.020340 secs 2024-04-26T19:28:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:10 smithi194 ceph-mon[33017]: pgmap v14987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:10 smithi033 ceph-mon[27845]: pgmap v14987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:10 smithi096 ceph-mon[32753]: pgmap v14987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:12 smithi194 ceph-mon[33017]: pgmap v14988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:12 smithi033 ceph-mon[27845]: pgmap v14988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:12 smithi096 ceph-mon[32753]: pgmap v14988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27432.118564 secs 2024-04-26T19:28:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:14 smithi194 ceph-mon[33017]: pgmap v14989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27435.020457 secs 2024-04-26T19:28:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27432.118564 secs 2024-04-26T19:28:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:14 smithi033 ceph-mon[27845]: pgmap v14989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27435.020457 secs 2024-04-26T19:28:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27432.118564 secs 2024-04-26T19:28:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:14 smithi096 ceph-mon[32753]: pgmap v14989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27435.020457 secs 2024-04-26T19:28:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:16 smithi194 ceph-mon[33017]: pgmap v14990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:16 smithi033 ceph-mon[27845]: pgmap v14990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:16 smithi096 ceph-mon[32753]: pgmap v14990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:18 smithi194 ceph-mon[33017]: pgmap v14991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:18 smithi033 ceph-mon[27845]: pgmap v14991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:18 smithi096 ceph-mon[32753]: pgmap v14991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27437.118649 secs 2024-04-26T19:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27440.020563 secs 2024-04-26T19:28:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27437.118649 secs 2024-04-26T19:28:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27440.020563 secs 2024-04-26T19:28:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27437.118649 secs 2024-04-26T19:28:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27440.020563 secs 2024-04-26T19:28:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:20 smithi194 ceph-mon[33017]: pgmap v14992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:20 smithi033 ceph-mon[27845]: pgmap v14992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:20 smithi096 ceph-mon[32753]: pgmap v14992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:22 smithi194 ceph-mon[33017]: pgmap v14993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:28:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:22 smithi033 ceph-mon[27845]: pgmap v14993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:28:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:22 smithi096 ceph-mon[32753]: pgmap v14993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:28:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:22.912+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:22.912+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:22.912+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:28:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27442.118736 secs 2024-04-26T19:28:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:24 smithi194 ceph-mon[33017]: pgmap v14994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27445.020640 secs 2024-04-26T19:28:24.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27442.118736 secs 2024-04-26T19:28:24.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:24 smithi096 ceph-mon[32753]: pgmap v14994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:24.400 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27445.020640 secs 2024-04-26T19:28:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27442.118736 secs 2024-04-26T19:28:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:24 smithi033 ceph-mon[27845]: pgmap v14994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27445.020640 secs 2024-04-26T19:28:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:26 smithi194 ceph-mon[33017]: pgmap v14995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:26 smithi033 ceph-mon[27845]: pgmap v14995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:26 smithi096 ceph-mon[32753]: pgmap v14995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:28 smithi194 ceph-mon[33017]: pgmap v14996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27447.118822 secs 2024-04-26T19:28:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:28:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:28:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:28 smithi033 ceph-mon[27845]: pgmap v14996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27447.118822 secs 2024-04-26T19:28:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:28:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:28:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:28 smithi096 ceph-mon[32753]: pgmap v14996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27447.118822 secs 2024-04-26T19:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:28:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:29 smithi194 ceph-mon[33017]: pgmap v14997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T19:28:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:29 smithi194 ceph-mon[33017]: pgmap v14998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27450.020774 secs 2024-04-26T19:28:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:29 smithi033 ceph-mon[27845]: pgmap v14997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T19:28:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:29 smithi033 ceph-mon[27845]: pgmap v14998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27450.020774 secs 2024-04-26T19:28:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:29 smithi096 ceph-mon[32753]: pgmap v14997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T19:28:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:29 smithi096 ceph-mon[32753]: pgmap v14998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27450.020774 secs 2024-04-26T19:28:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:31 smithi194 ceph-mon[33017]: pgmap v14999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:31 smithi033 ceph-mon[27845]: pgmap v14999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:31 smithi096 ceph-mon[32753]: pgmap v14999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:33 smithi194 ceph-mon[33017]: pgmap v15000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27452.118894 secs 2024-04-26T19:28:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:33 smithi033 ceph-mon[27845]: pgmap v15000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27452.118894 secs 2024-04-26T19:28:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:33 smithi096 ceph-mon[32753]: pgmap v15000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27452.118894 secs 2024-04-26T19:28:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27455.020911 secs 2024-04-26T19:28:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27455.020911 secs 2024-04-26T19:28:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27455.020911 secs 2024-04-26T19:28:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:35 smithi033 ceph-mon[27845]: pgmap v15001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:28:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:35 smithi096 ceph-mon[32753]: pgmap v15001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:28:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:35 smithi194 ceph-mon[33017]: pgmap v15001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:28:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:37.921+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:37.921+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:37.921+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:28:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:37 smithi033 ceph-mon[27845]: pgmap v15002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:37 smithi096 ceph-mon[32753]: pgmap v15002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:37 smithi194 ceph-mon[33017]: pgmap v15002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27457.118984 secs 2024-04-26T19:28:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27457.118984 secs 2024-04-26T19:28:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27457.118984 secs 2024-04-26T19:28:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:39 smithi033 ceph-mon[27845]: pgmap v15003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:28:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27460.021051 secs 2024-04-26T19:28:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:39 smithi096 ceph-mon[32753]: pgmap v15003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:28:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27460.021051 secs 2024-04-26T19:28:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:39 smithi194 ceph-mon[33017]: pgmap v15003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:28:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27460.021051 secs 2024-04-26T19:28:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:41 smithi033 ceph-mon[27845]: pgmap v15004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:41.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:41 smithi096 ceph-mon[32753]: pgmap v15004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:41 smithi194 ceph-mon[33017]: pgmap v15004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:43 smithi033 ceph-mon[27845]: pgmap v15005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27462.119041 secs 2024-04-26T19:28:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:43 smithi096 ceph-mon[32753]: pgmap v15005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27462.119041 secs 2024-04-26T19:28:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:43 smithi194 ceph-mon[33017]: pgmap v15005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27462.119041 secs 2024-04-26T19:28:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27465.021169 secs 2024-04-26T19:28:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27465.021169 secs 2024-04-26T19:28:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27465.021169 secs 2024-04-26T19:28:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:45 smithi033 ceph-mon[27845]: pgmap v15006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:45 smithi096 ceph-mon[32753]: pgmap v15006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:45 smithi194 ceph-mon[33017]: pgmap v15006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:47 smithi033 ceph-mon[27845]: pgmap v15007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:28:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:47 smithi096 ceph-mon[32753]: pgmap v15007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:28:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:47 smithi194 ceph-mon[33017]: pgmap v15007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:28:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27467.119129 secs 2024-04-26T19:28:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27467.119129 secs 2024-04-26T19:28:49.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27467.119129 secs 2024-04-26T19:28:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:49 smithi033 ceph-mon[27845]: pgmap v15008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27470.021248 secs 2024-04-26T19:28:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:49 smithi096 ceph-mon[32753]: pgmap v15008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27470.021248 secs 2024-04-26T19:28:50.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:49 smithi194 ceph-mon[33017]: pgmap v15008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:50.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27470.021248 secs 2024-04-26T19:28:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:51 smithi033 ceph-mon[27845]: pgmap v15009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:51 smithi096 ceph-mon[32753]: pgmap v15009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:51 smithi194 ceph-mon[33017]: pgmap v15009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:28:52.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:52.934+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:52.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:52.934+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:28:52.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:28:52.934+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:28:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:53 smithi096 ceph-mon[32753]: pgmap v15010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27472.119189 secs 2024-04-26T19:28:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:53 smithi194 ceph-mon[33017]: pgmap v15010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27472.119189 secs 2024-04-26T19:28:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:53 smithi033 ceph-mon[27845]: pgmap v15010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:28:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27472.119189 secs 2024-04-26T19:28:54.955 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27475.021345 secs 2024-04-26T19:28:55.087 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27475.021345 secs 2024-04-26T19:28:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27475.021345 secs 2024-04-26T19:28:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:55 smithi194 ceph-mon[33017]: pgmap v15011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:55 smithi033 ceph-mon[27845]: pgmap v15011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:55 smithi096 ceph-mon[32753]: pgmap v15011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:28:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:57 smithi194 ceph-mon[33017]: pgmap v15012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:28:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:57 smithi033 ceph-mon[27845]: pgmap v15012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:28:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:57 smithi096 ceph-mon[32753]: pgmap v15012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:28:59.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27477.119252 secs 2024-04-26T19:28:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27477.119252 secs 2024-04-26T19:28:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27477.119252 secs 2024-04-26T19:29:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:59 smithi194 ceph-mon[33017]: pgmap v15013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:28:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27480.021454 secs 2024-04-26T19:29:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:59 smithi033 ceph-mon[27845]: pgmap v15013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:28:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27480.021454 secs 2024-04-26T19:29:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:59 smithi096 ceph-mon[32753]: pgmap v15013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:28:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27480.021454 secs 2024-04-26T19:29:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:01 smithi194 ceph-mon[33017]: pgmap v15014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:01 smithi033 ceph-mon[27845]: pgmap v15014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:01 smithi096 ceph-mon[32753]: pgmap v15014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:03 smithi194 ceph-mon[33017]: pgmap v15015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27482.119272 secs 2024-04-26T19:29:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:03 smithi033 ceph-mon[27845]: pgmap v15015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27482.119272 secs 2024-04-26T19:29:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:03 smithi096 ceph-mon[32753]: pgmap v15015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27482.119272 secs 2024-04-26T19:29:05.016 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27485.021591 secs 2024-04-26T19:29:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27485.021591 secs 2024-04-26T19:29:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27485.021591 secs 2024-04-26T19:29:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:05 smithi194 ceph-mon[33017]: pgmap v15016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:05 smithi033 ceph-mon[27845]: pgmap v15016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:05 smithi096 ceph-mon[32753]: pgmap v15016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:07.950+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:07.950+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:07.950+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:29:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:07 smithi194 ceph-mon[33017]: pgmap v15017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:07 smithi033 ceph-mon[27845]: pgmap v15017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:08.241 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:07 smithi096 ceph-mon[32753]: pgmap v15017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27487.119352 secs 2024-04-26T19:29:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27487.119352 secs 2024-04-26T19:29:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27487.119352 secs 2024-04-26T19:29:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:09 smithi194 ceph-mon[33017]: pgmap v15018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27490.021700 secs 2024-04-26T19:29:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:09 smithi033 ceph-mon[27845]: pgmap v15018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27490.021700 secs 2024-04-26T19:29:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:09 smithi096 ceph-mon[32753]: pgmap v15018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27490.021700 secs 2024-04-26T19:29:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:11 smithi194 ceph-mon[33017]: pgmap v15019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:11 smithi033 ceph-mon[27845]: pgmap v15019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:11 smithi096 ceph-mon[32753]: pgmap v15019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:13 smithi194 ceph-mon[33017]: pgmap v15020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27492.119437 secs 2024-04-26T19:29:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:13 smithi033 ceph-mon[27845]: pgmap v15020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27492.119437 secs 2024-04-26T19:29:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:13 smithi096 ceph-mon[32753]: pgmap v15020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27492.119437 secs 2024-04-26T19:29:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27495.021823 secs 2024-04-26T19:29:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27495.021823 secs 2024-04-26T19:29:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27495.021823 secs 2024-04-26T19:29:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:15 smithi194 ceph-mon[33017]: pgmap v15021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:15 smithi033 ceph-mon[27845]: pgmap v15021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:15 smithi096 ceph-mon[32753]: pgmap v15021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:17 smithi194 ceph-mon[33017]: pgmap v15022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:17 smithi033 ceph-mon[27845]: pgmap v15022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:17 smithi096 ceph-mon[32753]: pgmap v15022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27497.119524 secs 2024-04-26T19:29:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27497.119524 secs 2024-04-26T19:29:19.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27497.119524 secs 2024-04-26T19:29:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:19 smithi194 ceph-mon[33017]: pgmap v15023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27500.021938 secs 2024-04-26T19:29:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:19 smithi033 ceph-mon[27845]: pgmap v15023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27500.021938 secs 2024-04-26T19:29:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:19 smithi096 ceph-mon[32753]: pgmap v15023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27500.021938 secs 2024-04-26T19:29:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:21 smithi194 ceph-mon[33017]: pgmap v15024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:21 smithi033 ceph-mon[27845]: pgmap v15024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:21 smithi096 ceph-mon[32753]: pgmap v15024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:22.953+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:22.953+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:22.953+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:29:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:23 smithi033 ceph-mon[27845]: pgmap v15025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:23 smithi096 ceph-mon[32753]: pgmap v15025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:23 smithi194 ceph-mon[33017]: pgmap v15025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27502.119607 secs 2024-04-26T19:29:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27505.022055 secs 2024-04-26T19:29:24.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27502.119607 secs 2024-04-26T19:29:24.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27505.022055 secs 2024-04-26T19:29:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27502.119607 secs 2024-04-26T19:29:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27505.022055 secs 2024-04-26T19:29:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:25 smithi096 ceph-mon[32753]: pgmap v15026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:25 smithi194 ceph-mon[33017]: pgmap v15026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:25 smithi033 ceph-mon[27845]: pgmap v15026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:26.473+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:29:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:26.473+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:29:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:26.474+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 217 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:29:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:26.474+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 217 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:29:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:27 smithi194 ceph-mon[33017]: pgmap v15027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:27 smithi033 ceph-mon[27845]: pgmap v15027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:27 smithi096 ceph-mon[32753]: pgmap v15027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:29:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27507.119693 secs 2024-04-26T19:29:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27507.119693 secs 2024-04-26T19:29:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27507.119693 secs 2024-04-26T19:29:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:29 smithi033 ceph-mon[27845]: pgmap v15028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27510.022160 secs 2024-04-26T19:29:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:29:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:29 smithi194 ceph-mon[33017]: pgmap v15028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27510.022160 secs 2024-04-26T19:29:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:29:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:29 smithi096 ceph-mon[32753]: pgmap v15028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27510.022160 secs 2024-04-26T19:29:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:29:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:31 smithi194 ceph-mon[33017]: pgmap v15029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:31 smithi033 ceph-mon[27845]: pgmap v15029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:31 smithi096 ceph-mon[32753]: pgmap v15029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:32.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: pgmap v15030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27512.119750 secs 2024-04-26T19:29:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:29:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:29:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: pgmap v15030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27512.119750 secs 2024-04-26T19:29:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: pgmap v15030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27512.119750 secs 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27515.022263 secs 2024-04-26T19:29:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27515.022263 secs 2024-04-26T19:29:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27515.022263 secs 2024-04-26T19:29:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: pgmap v15031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:29:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: pgmap v15031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:29:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: pgmap v15031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:29:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:29:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:29:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:29:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:36 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:29:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:36 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:29:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:36 smithi194 ceph-mon[33017]: pgmap v15032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:36 smithi194 ceph-mon[33017]: pgmap v15033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:36 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:29:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:36 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:29:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:36 smithi033 ceph-mon[27845]: pgmap v15032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:36 smithi033 ceph-mon[27845]: pgmap v15033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:36 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:29:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:36 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:29:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:36 smithi096 ceph-mon[32753]: pgmap v15032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:36 smithi096 ceph-mon[32753]: pgmap v15033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:37.969+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:37.969+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:37.969+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:29:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:38 smithi194 ceph-mon[33017]: pgmap v15034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27517.119837 secs 2024-04-26T19:29:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:38 smithi033 ceph-mon[27845]: pgmap v15034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27517.119837 secs 2024-04-26T19:29:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:38 smithi096 ceph-mon[32753]: pgmap v15034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27517.119837 secs 2024-04-26T19:29:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27520.022379 secs 2024-04-26T19:29:40.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27520.022379 secs 2024-04-26T19:29:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27520.022379 secs 2024-04-26T19:29:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:40 smithi194 ceph-mon[33017]: pgmap v15035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:29:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:40 smithi033 ceph-mon[27845]: pgmap v15035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:29:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:40 smithi096 ceph-mon[32753]: pgmap v15035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:29:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:42 smithi194 ceph-mon[33017]: pgmap v15036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:42 smithi033 ceph-mon[27845]: pgmap v15036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:42 smithi096 ceph-mon[32753]: pgmap v15036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:29:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27522.119922 secs 2024-04-26T19:29:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27522.119922 secs 2024-04-26T19:29:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27522.119922 secs 2024-04-26T19:29:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:44 smithi194 ceph-mon[33017]: pgmap v15037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T19:29:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27525.022495 secs 2024-04-26T19:29:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:44 smithi033 ceph-mon[27845]: pgmap v15037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T19:29:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27525.022495 secs 2024-04-26T19:29:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:44 smithi096 ceph-mon[32753]: pgmap v15037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T19:29:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27525.022495 secs 2024-04-26T19:29:47.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:46 smithi194 ceph-mon[33017]: pgmap v15038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:29:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:46 smithi033 ceph-mon[27845]: pgmap v15038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:29:47.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:46 smithi096 ceph-mon[32753]: pgmap v15038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:29:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:48 smithi194 ceph-mon[33017]: pgmap v15039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27527.120000 secs 2024-04-26T19:29:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:48 smithi033 ceph-mon[27845]: pgmap v15039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27527.120000 secs 2024-04-26T19:29:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:48 smithi096 ceph-mon[32753]: pgmap v15039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27527.120000 secs 2024-04-26T19:29:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27530.022610 secs 2024-04-26T19:29:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27530.022610 secs 2024-04-26T19:29:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27530.022610 secs 2024-04-26T19:29:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:50 smithi194 ceph-mon[33017]: pgmap v15040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:50 smithi033 ceph-mon[27845]: pgmap v15040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:50 smithi096 ceph-mon[32753]: pgmap v15040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:52.985+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:52.985+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:29:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:29:52.985+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:29:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:52 smithi194 ceph-mon[33017]: pgmap v15041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:52 smithi033 ceph-mon[27845]: pgmap v15041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:52 smithi096 ceph-mon[32753]: pgmap v15041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27532.120086 secs 2024-04-26T19:29:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27532.120086 secs 2024-04-26T19:29:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27532.120086 secs 2024-04-26T19:29:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:54 smithi194 ceph-mon[33017]: pgmap v15042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27535.022762 secs 2024-04-26T19:29:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:54 smithi033 ceph-mon[27845]: pgmap v15042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27535.022762 secs 2024-04-26T19:29:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:54 smithi096 ceph-mon[32753]: pgmap v15042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27535.022762 secs 2024-04-26T19:29:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:56 smithi194 ceph-mon[33017]: pgmap v15043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:56 smithi033 ceph-mon[27845]: pgmap v15043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:56 smithi096 ceph-mon[32753]: pgmap v15043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:29:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:58 smithi194 ceph-mon[33017]: pgmap v15044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27537.120173 secs 2024-04-26T19:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:58 smithi033 ceph-mon[27845]: pgmap v15044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27537.120173 secs 2024-04-26T19:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:58 smithi096 ceph-mon[32753]: pgmap v15044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27537.120173 secs 2024-04-26T19:30:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:29:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27540.022877 secs 2024-04-26T19:30:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:29:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27540.022877 secs 2024-04-26T19:30:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:29:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27540.022877 secs 2024-04-26T19:30:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:00 smithi194 ceph-mon[33017]: pgmap v15045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:00 smithi033 ceph-mon[27845]: pgmap v15045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:30:01.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:00 smithi096 ceph-mon[32753]: pgmap v15045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:01.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:30:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:02 smithi194 ceph-mon[33017]: pgmap v15046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:02 smithi033 ceph-mon[27845]: pgmap v15046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:02 smithi096 ceph-mon[32753]: pgmap v15046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27542.120275 secs 2024-04-26T19:30:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27542.120275 secs 2024-04-26T19:30:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27542.120275 secs 2024-04-26T19:30:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:04 smithi194 ceph-mon[33017]: pgmap v15047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27545.023015 secs 2024-04-26T19:30:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:04 smithi033 ceph-mon[27845]: pgmap v15047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27545.023015 secs 2024-04-26T19:30:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:04 smithi096 ceph-mon[32753]: pgmap v15047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27545.023015 secs 2024-04-26T19:30:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:06 smithi194 ceph-mon[33017]: pgmap v15048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:07.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:06 smithi033 ceph-mon[27845]: pgmap v15048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:06 smithi096 ceph-mon[32753]: pgmap v15048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:07.999+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:07.999+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:07.999+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:08 smithi194 ceph-mon[33017]: pgmap v15049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27547.120357 secs 2024-04-26T19:30:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:08 smithi033 ceph-mon[27845]: pgmap v15049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27547.120357 secs 2024-04-26T19:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:08 smithi096 ceph-mon[32753]: pgmap v15049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27547.120357 secs 2024-04-26T19:30:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27550.023118 secs 2024-04-26T19:30:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27550.023118 secs 2024-04-26T19:30:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27550.023118 secs 2024-04-26T19:30:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:10 smithi033 ceph-mon[27845]: pgmap v15050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:11.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:10 smithi096 ceph-mon[32753]: pgmap v15050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:10 smithi194 ceph-mon[33017]: pgmap v15050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:12 smithi033 ceph-mon[27845]: pgmap v15051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:12 smithi096 ceph-mon[32753]: pgmap v15051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:12 smithi194 ceph-mon[33017]: pgmap v15051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27552.120416 secs 2024-04-26T19:30:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27552.120416 secs 2024-04-26T19:30:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27552.120416 secs 2024-04-26T19:30:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:14 smithi033 ceph-mon[27845]: pgmap v15052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27555.023183 secs 2024-04-26T19:30:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:14 smithi096 ceph-mon[32753]: pgmap v15052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27555.023183 secs 2024-04-26T19:30:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:14 smithi194 ceph-mon[33017]: pgmap v15052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27555.023183 secs 2024-04-26T19:30:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:16 smithi033 ceph-mon[27845]: pgmap v15053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:16 smithi096 ceph-mon[32753]: pgmap v15053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:17.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:16 smithi194 ceph-mon[33017]: pgmap v15053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:18 smithi033 ceph-mon[27845]: pgmap v15054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27557.120496 secs 2024-04-26T19:30:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:18 smithi096 ceph-mon[32753]: pgmap v15054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27557.120496 secs 2024-04-26T19:30:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:18 smithi194 ceph-mon[33017]: pgmap v15054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27557.120496 secs 2024-04-26T19:30:20.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27560.023255 secs 2024-04-26T19:30:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27560.023255 secs 2024-04-26T19:30:20.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27560.023255 secs 2024-04-26T19:30:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:20 smithi033 ceph-mon[27845]: pgmap v15055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:20 smithi096 ceph-mon[32753]: pgmap v15055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:20 smithi194 ceph-mon[33017]: pgmap v15055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:23.006+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:23.006+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:23.006+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:30:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:22 smithi033 ceph-mon[27845]: pgmap v15056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:22 smithi096 ceph-mon[32753]: pgmap v15056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:22 smithi194 ceph-mon[33017]: pgmap v15056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27562.120578 secs 2024-04-26T19:30:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27562.120578 secs 2024-04-26T19:30:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27562.120578 secs 2024-04-26T19:30:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:24 smithi033 ceph-mon[27845]: pgmap v15057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27565.023384 secs 2024-04-26T19:30:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:24 smithi096 ceph-mon[32753]: pgmap v15057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27565.023384 secs 2024-04-26T19:30:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:24 smithi194 ceph-mon[33017]: pgmap v15057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27565.023384 secs 2024-04-26T19:30:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:25 smithi033 ceph-mon[27845]: pgmap v15058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:25 smithi096 ceph-mon[32753]: pgmap v15058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:25 smithi194 ceph-mon[33017]: pgmap v15058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:28 smithi033 ceph-mon[27845]: pgmap v15059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:28 smithi194 ceph-mon[33017]: pgmap v15059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:28 smithi096 ceph-mon[32753]: pgmap v15059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27567.120665 secs 2024-04-26T19:30:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27567.120665 secs 2024-04-26T19:30:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27567.120665 secs 2024-04-26T19:30:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27570.023503 secs 2024-04-26T19:30:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:30 smithi194 ceph-mon[33017]: pgmap v15060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27570.023503 secs 2024-04-26T19:30:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:30 smithi033 ceph-mon[27845]: pgmap v15060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27570.023503 secs 2024-04-26T19:30:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:30 smithi096 ceph-mon[32753]: pgmap v15060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:32.357 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:32 smithi194 ceph-mon[33017]: pgmap v15061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:32 smithi033 ceph-mon[27845]: pgmap v15061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:32 smithi096 ceph-mon[32753]: pgmap v15061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27572.120742 secs 2024-04-26T19:30:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:34 smithi194 ceph-mon[33017]: pgmap v15062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27572.120742 secs 2024-04-26T19:30:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:34 smithi033 ceph-mon[27845]: pgmap v15062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27572.120742 secs 2024-04-26T19:30:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:34 smithi096 ceph-mon[32753]: pgmap v15062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:30:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27575.023616 secs 2024-04-26T19:30:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27575.023616 secs 2024-04-26T19:30:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27575.023616 secs 2024-04-26T19:30:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:36 smithi194 ceph-mon[33017]: pgmap v15063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:30:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:36 smithi033 ceph-mon[27845]: pgmap v15063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:30:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:36 smithi096 ceph-mon[32753]: pgmap v15063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:30:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:37 smithi194 ceph-mon[33017]: pgmap v15064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:37 smithi194 ceph-mon[33017]: pgmap v15065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:30:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:30:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:30:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:37 smithi033 ceph-mon[27845]: pgmap v15064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:37 smithi033 ceph-mon[27845]: pgmap v15065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:30:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:30:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:30:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:37 smithi096 ceph-mon[32753]: pgmap v15064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:37 smithi096 ceph-mon[32753]: pgmap v15065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:30:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:38.017+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:38.017+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:30:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:38.017+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27577.120829 secs 2024-04-26T19:30:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27577.120829 secs 2024-04-26T19:30:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27577.120829 secs 2024-04-26T19:30:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27580.023716 secs 2024-04-26T19:30:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:39 smithi194 ceph-mon[33017]: pgmap v15066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27580.023716 secs 2024-04-26T19:30:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:39 smithi033 ceph-mon[27845]: pgmap v15066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27580.023716 secs 2024-04-26T19:30:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:39 smithi096 ceph-mon[32753]: pgmap v15066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:41 smithi096 ceph-mon[32753]: pgmap v15067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:41 smithi194 ceph-mon[33017]: pgmap v15067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:41 smithi033 ceph-mon[27845]: pgmap v15067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:43 smithi096 ceph-mon[32753]: pgmap v15068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27582.120917 secs 2024-04-26T19:30:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:43 smithi194 ceph-mon[33017]: pgmap v15068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27582.120917 secs 2024-04-26T19:30:44.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:43 smithi033 ceph-mon[27845]: pgmap v15068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:44.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27582.120917 secs 2024-04-26T19:30:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27585.023848 secs 2024-04-26T19:30:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27585.023848 secs 2024-04-26T19:30:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27585.023848 secs 2024-04-26T19:30:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:45 smithi194 ceph-mon[33017]: pgmap v15069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:45 smithi033 ceph-mon[27845]: pgmap v15069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:45 smithi096 ceph-mon[32753]: pgmap v15069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:30:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:47 smithi194 ceph-mon[33017]: pgmap v15070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:30:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:47 smithi033 ceph-mon[27845]: pgmap v15070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:30:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:47 smithi096 ceph-mon[32753]: pgmap v15070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:30:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27587.121015 secs 2024-04-26T19:30:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27587.121015 secs 2024-04-26T19:30:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27587.121015 secs 2024-04-26T19:30:50.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27590.023982 secs 2024-04-26T19:30:50.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:49 smithi194 ceph-mon[33017]: pgmap v15071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27590.023982 secs 2024-04-26T19:30:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:49 smithi033 ceph-mon[27845]: pgmap v15071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27590.023982 secs 2024-04-26T19:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:49 smithi096 ceph-mon[32753]: pgmap v15071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:51 smithi194 ceph-mon[33017]: pgmap v15072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:51 smithi033 ceph-mon[27845]: pgmap v15072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:51 smithi096 ceph-mon[32753]: pgmap v15072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:53.033+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:53.033+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:30:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:30:53.033+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:30:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:53 smithi194 ceph-mon[33017]: pgmap v15073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27592.121095 secs 2024-04-26T19:30:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:53 smithi033 ceph-mon[27845]: pgmap v15073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27592.121095 secs 2024-04-26T19:30:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:53 smithi096 ceph-mon[32753]: pgmap v15073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27592.121095 secs 2024-04-26T19:30:55.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27595.024063 secs 2024-04-26T19:30:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27595.024063 secs 2024-04-26T19:30:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27595.024063 secs 2024-04-26T19:30:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:55 smithi194 ceph-mon[33017]: pgmap v15074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:55 smithi033 ceph-mon[27845]: pgmap v15074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:55 smithi096 ceph-mon[32753]: pgmap v15074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:30:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:57 smithi194 ceph-mon[33017]: pgmap v15075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:57 smithi033 ceph-mon[27845]: pgmap v15075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:57 smithi096 ceph-mon[32753]: pgmap v15075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:30:59.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27597.121181 secs 2024-04-26T19:30:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27597.121181 secs 2024-04-26T19:30:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27597.121181 secs 2024-04-26T19:31:00.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27600.024183 secs 2024-04-26T19:31:00.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:30:59 smithi033 ceph-mon[27845]: pgmap v15076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27600.024183 secs 2024-04-26T19:31:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:30:59 smithi194 ceph-mon[33017]: pgmap v15076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27600.024183 secs 2024-04-26T19:31:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:30:59 smithi096 ceph-mon[32753]: pgmap v15076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:01 smithi194 ceph-mon[33017]: pgmap v15077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:01 smithi033 ceph-mon[27845]: pgmap v15077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:01 smithi096 ceph-mon[32753]: pgmap v15077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:04.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:03 smithi194 ceph-mon[33017]: pgmap v15078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:04.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27602.121242 secs 2024-04-26T19:31:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:03 smithi033 ceph-mon[27845]: pgmap v15078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27602.121242 secs 2024-04-26T19:31:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:03 smithi096 ceph-mon[32753]: pgmap v15078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:04.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27602.121242 secs 2024-04-26T19:31:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27605.024334 secs 2024-04-26T19:31:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27605.024334 secs 2024-04-26T19:31:05.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27605.024334 secs 2024-04-26T19:31:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:05 smithi194 ceph-mon[33017]: pgmap v15079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:05 smithi033 ceph-mon[27845]: pgmap v15079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:05 smithi096 ceph-mon[32753]: pgmap v15079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:08.041+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:08.041+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:31:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:08.041+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:07 smithi194 ceph-mon[33017]: pgmap v15080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:07 smithi033 ceph-mon[27845]: pgmap v15080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:08.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:07 smithi096 ceph-mon[32753]: pgmap v15080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27607.121329 secs 2024-04-26T19:31:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27607.121329 secs 2024-04-26T19:31:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27607.121329 secs 2024-04-26T19:31:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27610.024453 secs 2024-04-26T19:31:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:09 smithi194 ceph-mon[33017]: pgmap v15081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27610.024453 secs 2024-04-26T19:31:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:09 smithi033 ceph-mon[27845]: pgmap v15081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27610.024453 secs 2024-04-26T19:31:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:09 smithi096 ceph-mon[32753]: pgmap v15081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:11 smithi194 ceph-mon[33017]: pgmap v15082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:11 smithi033 ceph-mon[27845]: pgmap v15082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:12.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:11 smithi096 ceph-mon[32753]: pgmap v15082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:14.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:13 smithi194 ceph-mon[33017]: pgmap v15083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:14.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27612.121410 secs 2024-04-26T19:31:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:13 smithi033 ceph-mon[27845]: pgmap v15083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27612.121410 secs 2024-04-26T19:31:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:13 smithi096 ceph-mon[32753]: pgmap v15083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27612.121410 secs 2024-04-26T19:31:15.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27615.024569 secs 2024-04-26T19:31:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27615.024569 secs 2024-04-26T19:31:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27615.024569 secs 2024-04-26T19:31:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:15 smithi194 ceph-mon[33017]: pgmap v15084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:15 smithi033 ceph-mon[27845]: pgmap v15084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:16.304 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:15 smithi096 ceph-mon[32753]: pgmap v15084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:17 smithi194 ceph-mon[33017]: pgmap v15085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:17 smithi033 ceph-mon[27845]: pgmap v15085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:17 smithi096 ceph-mon[32753]: pgmap v15085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27617.121497 secs 2024-04-26T19:31:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27617.121497 secs 2024-04-26T19:31:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27617.121497 secs 2024-04-26T19:31:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27620.024691 secs 2024-04-26T19:31:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:19 smithi033 ceph-mon[27845]: pgmap v15086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27620.024691 secs 2024-04-26T19:31:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:19 smithi096 ceph-mon[32753]: pgmap v15086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:20.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27620.024691 secs 2024-04-26T19:31:20.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:19 smithi194 ceph-mon[33017]: pgmap v15086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:22.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:21 smithi033 ceph-mon[27845]: pgmap v15087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:21 smithi096 ceph-mon[32753]: pgmap v15087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:21 smithi194 ceph-mon[33017]: pgmap v15087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:23.055+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:23.055+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:23.055+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:31:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:23 smithi033 ceph-mon[27845]: pgmap v15088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27622.121579 secs 2024-04-26T19:31:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:23 smithi096 ceph-mon[32753]: pgmap v15088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27622.121579 secs 2024-04-26T19:31:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:23 smithi194 ceph-mon[33017]: pgmap v15088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27622.121579 secs 2024-04-26T19:31:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27625.024765 secs 2024-04-26T19:31:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27625.024765 secs 2024-04-26T19:31:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27625.024765 secs 2024-04-26T19:31:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:25 smithi033 ceph-mon[27845]: pgmap v15089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:25 smithi096 ceph-mon[32753]: pgmap v15089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:25 smithi194 ceph-mon[33017]: pgmap v15089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:27 smithi033 ceph-mon[27845]: pgmap v15090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:28.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:27 smithi096 ceph-mon[32753]: pgmap v15090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:27 smithi194 ceph-mon[33017]: pgmap v15090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27627.121661 secs 2024-04-26T19:31:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27627.121661 secs 2024-04-26T19:31:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27627.121661 secs 2024-04-26T19:31:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27630.024879 secs 2024-04-26T19:31:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:29 smithi033 ceph-mon[27845]: pgmap v15091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27630.024879 secs 2024-04-26T19:31:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:29 smithi096 ceph-mon[32753]: pgmap v15091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:30.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27630.024879 secs 2024-04-26T19:31:30.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:29 smithi194 ceph-mon[33017]: pgmap v15091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:31 smithi033 ceph-mon[27845]: pgmap v15092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:31 smithi096 ceph-mon[32753]: pgmap v15092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:31 smithi194 ceph-mon[33017]: pgmap v15092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:33 smithi033 ceph-mon[27845]: pgmap v15093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27632.121739 secs 2024-04-26T19:31:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:33 smithi096 ceph-mon[32753]: pgmap v15093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27632.121739 secs 2024-04-26T19:31:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:33 smithi194 ceph-mon[33017]: pgmap v15093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27632.121739 secs 2024-04-26T19:31:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27635.025003 secs 2024-04-26T19:31:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27635.025003 secs 2024-04-26T19:31:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27635.025003 secs 2024-04-26T19:31:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:35 smithi033 ceph-mon[27845]: pgmap v15094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:36.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:35 smithi096 ceph-mon[32753]: pgmap v15094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:35 smithi194 ceph-mon[33017]: pgmap v15094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:38.059+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:38.059+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:31:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:38.059+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:38.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:37 smithi096 ceph-mon[32753]: pgmap v15095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:38.187 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:31:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:37 smithi194 ceph-mon[33017]: pgmap v15095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:31:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:37 smithi033 ceph-mon[27845]: pgmap v15095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:31:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27637.121816 secs 2024-04-26T19:31:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:31:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:31:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:38 smithi194 ceph-mon[33017]: pgmap v15096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:38 smithi194 ceph-mon[33017]: pgmap v15097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:31:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27637.121816 secs 2024-04-26T19:31:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:31:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:31:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:38 smithi033 ceph-mon[27845]: pgmap v15096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:38 smithi033 ceph-mon[27845]: pgmap v15097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27637.121816 secs 2024-04-26T19:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:38 smithi096 ceph-mon[32753]: pgmap v15096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:38 smithi096 ceph-mon[32753]: pgmap v15097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:31:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27640.025144 secs 2024-04-26T19:31:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27640.025144 secs 2024-04-26T19:31:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27640.025144 secs 2024-04-26T19:31:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:40 smithi194 ceph-mon[33017]: pgmap v15098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:40 smithi033 ceph-mon[27845]: pgmap v15098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:40 smithi096 ceph-mon[32753]: pgmap v15098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:41.497+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:31:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:41.497+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:31:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:41.497+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 218 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T19:31:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:41.497+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 218 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T19:31:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:43 smithi194 ceph-mon[33017]: pgmap v15099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:43.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:43 smithi033 ceph-mon[27845]: pgmap v15099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:43.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:43 smithi096 ceph-mon[32753]: pgmap v15099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27642.121898 secs 2024-04-26T19:31:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27645.025282 secs 2024-04-26T19:31:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27642.121898 secs 2024-04-26T19:31:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27645.025282 secs 2024-04-26T19:31:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27642.121898 secs 2024-04-26T19:31:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27645.025282 secs 2024-04-26T19:31:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:45 smithi033 ceph-mon[27845]: pgmap v15100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:31:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:45 smithi096 ceph-mon[32753]: pgmap v15100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:31:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:45 smithi194 ceph-mon[33017]: pgmap v15100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:31:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:47 smithi033 ceph-mon[27845]: pgmap v15101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:47 smithi096 ceph-mon[32753]: pgmap v15101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:47 smithi194 ceph-mon[33017]: pgmap v15101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27647.122001 secs 2024-04-26T19:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:49 smithi033 ceph-mon[27845]: pgmap v15102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27650.025404 secs 2024-04-26T19:31:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27647.122001 secs 2024-04-26T19:31:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:49 smithi096 ceph-mon[32753]: pgmap v15102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:31:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27650.025404 secs 2024-04-26T19:31:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27647.122001 secs 2024-04-26T19:31:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:49 smithi194 ceph-mon[33017]: pgmap v15102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:31:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27650.025404 secs 2024-04-26T19:31:51.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:51 smithi194 ceph-mon[33017]: pgmap v15103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:51 smithi033 ceph-mon[27845]: pgmap v15103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:51.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:51 smithi096 ceph-mon[32753]: pgmap v15103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:53.074+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:53.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:53.075+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:31:53.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:31:53.075+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:31:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:53 smithi033 ceph-mon[27845]: pgmap v15104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:53 smithi096 ceph-mon[32753]: pgmap v15104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:53 smithi194 ceph-mon[33017]: pgmap v15104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27652.122061 secs 2024-04-26T19:31:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27655.025484 secs 2024-04-26T19:31:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27652.122061 secs 2024-04-26T19:31:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27655.025484 secs 2024-04-26T19:31:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27652.122061 secs 2024-04-26T19:31:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27655.025484 secs 2024-04-26T19:31:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:55 smithi033 ceph-mon[27845]: pgmap v15105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:55 smithi096 ceph-mon[32753]: pgmap v15105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:55 smithi194 ceph-mon[33017]: pgmap v15105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:31:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:57 smithi033 ceph-mon[27845]: pgmap v15106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:57 smithi096 ceph-mon[32753]: pgmap v15106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:57 smithi194 ceph-mon[33017]: pgmap v15106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27657.122148 secs 2024-04-26T19:31:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:59 smithi033 ceph-mon[27845]: pgmap v15107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27660.025580 secs 2024-04-26T19:31:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27657.122148 secs 2024-04-26T19:31:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:59 smithi096 ceph-mon[32753]: pgmap v15107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27660.025580 secs 2024-04-26T19:31:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27657.122148 secs 2024-04-26T19:31:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:59 smithi194 ceph-mon[33017]: pgmap v15107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:31:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27660.025580 secs 2024-04-26T19:32:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:01 smithi033 ceph-mon[27845]: pgmap v15108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:01 smithi096 ceph-mon[32753]: pgmap v15108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:01 smithi194 ceph-mon[33017]: pgmap v15108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:03 smithi033 ceph-mon[27845]: pgmap v15109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:03 smithi096 ceph-mon[32753]: pgmap v15109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:03 smithi194 ceph-mon[33017]: pgmap v15109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27662.122226 secs 2024-04-26T19:32:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27665.025696 secs 2024-04-26T19:32:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27662.122226 secs 2024-04-26T19:32:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27665.025696 secs 2024-04-26T19:32:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27662.122226 secs 2024-04-26T19:32:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27665.025696 secs 2024-04-26T19:32:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:05 smithi096 ceph-mon[32753]: pgmap v15110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:05 smithi194 ceph-mon[33017]: pgmap v15110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:05 smithi033 ceph-mon[27845]: pgmap v15110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:07 smithi096 ceph-mon[32753]: pgmap v15111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:07 smithi194 ceph-mon[33017]: pgmap v15111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:07 smithi033 ceph-mon[27845]: pgmap v15111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:08.076+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:08.076+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:32:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:08.076+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27667.122305 secs 2024-04-26T19:32:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:09 smithi194 ceph-mon[33017]: pgmap v15112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27670.025818 secs 2024-04-26T19:32:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27667.122305 secs 2024-04-26T19:32:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:09 smithi033 ceph-mon[27845]: pgmap v15112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27670.025818 secs 2024-04-26T19:32:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27667.122305 secs 2024-04-26T19:32:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:09 smithi096 ceph-mon[32753]: pgmap v15112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27670.025818 secs 2024-04-26T19:32:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:11 smithi194 ceph-mon[33017]: pgmap v15113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:11 smithi033 ceph-mon[27845]: pgmap v15113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:11 smithi096 ceph-mon[32753]: pgmap v15113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:13 smithi194 ceph-mon[33017]: pgmap v15114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27672.122384 secs 2024-04-26T19:32:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:13 smithi033 ceph-mon[27845]: pgmap v15114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27672.122384 secs 2024-04-26T19:32:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:13 smithi096 ceph-mon[32753]: pgmap v15114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27672.122384 secs 2024-04-26T19:32:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27675.025955 secs 2024-04-26T19:32:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27675.025955 secs 2024-04-26T19:32:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27675.025955 secs 2024-04-26T19:32:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:15 smithi194 ceph-mon[33017]: pgmap v15115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:15 smithi033 ceph-mon[27845]: pgmap v15115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:15 smithi096 ceph-mon[32753]: pgmap v15115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:17 smithi033 ceph-mon[27845]: pgmap v15116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:17 smithi194 ceph-mon[33017]: pgmap v15116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:17 smithi096 ceph-mon[32753]: pgmap v15116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27677.122471 secs 2024-04-26T19:32:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27677.122471 secs 2024-04-26T19:32:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27677.122471 secs 2024-04-26T19:32:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:19 smithi194 ceph-mon[33017]: pgmap v15117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27680.026094 secs 2024-04-26T19:32:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:19 smithi033 ceph-mon[27845]: pgmap v15117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27680.026094 secs 2024-04-26T19:32:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:19 smithi096 ceph-mon[32753]: pgmap v15117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27680.026094 secs 2024-04-26T19:32:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:21 smithi194 ceph-mon[33017]: pgmap v15118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:21 smithi033 ceph-mon[27845]: pgmap v15118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:21.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:21 smithi096 ceph-mon[32753]: pgmap v15118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:23.092+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:23.092+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:23.092+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:32:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:23 smithi194 ceph-mon[33017]: pgmap v15119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27682.122528 secs 2024-04-26T19:32:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:23 smithi033 ceph-mon[27845]: pgmap v15119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27682.122528 secs 2024-04-26T19:32:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:23 smithi096 ceph-mon[32753]: pgmap v15119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27682.122528 secs 2024-04-26T19:32:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27685.026187 secs 2024-04-26T19:32:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27685.026187 secs 2024-04-26T19:32:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27685.026187 secs 2024-04-26T19:32:25.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:25 smithi096 ceph-mon[32753]: pgmap v15120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:25 smithi194 ceph-mon[33017]: pgmap v15120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:25 smithi033 ceph-mon[27845]: pgmap v15120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:26.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:26.504+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:32:26.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:26.504+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:32:26.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:26.504+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 219 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:32:26.505 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:26.504+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 219 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:32:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:27 smithi194 ceph-mon[33017]: pgmap v15121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:27 smithi033 ceph-mon[27845]: pgmap v15121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:27 smithi096 ceph-mon[32753]: pgmap v15121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27687.122609 secs 2024-04-26T19:32:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27687.122609 secs 2024-04-26T19:32:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27687.122609 secs 2024-04-26T19:32:29.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:29 smithi033 ceph-mon[27845]: pgmap v15122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:29.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27690.026304 secs 2024-04-26T19:32:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:29 smithi194 ceph-mon[33017]: pgmap v15122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27690.026304 secs 2024-04-26T19:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:29 smithi096 ceph-mon[32753]: pgmap v15122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27690.026304 secs 2024-04-26T19:32:31.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:31 smithi194 ceph-mon[33017]: pgmap v15123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:31 smithi033 ceph-mon[27845]: pgmap v15123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:31.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:31 smithi096 ceph-mon[32753]: pgmap v15123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:33 smithi194 ceph-mon[33017]: pgmap v15124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27692.122690 secs 2024-04-26T19:32:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:33 smithi033 ceph-mon[27845]: pgmap v15124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27692.122690 secs 2024-04-26T19:32:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:33 smithi096 ceph-mon[32753]: pgmap v15124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27692.122690 secs 2024-04-26T19:32:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27695.026424 secs 2024-04-26T19:32:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27695.026424 secs 2024-04-26T19:32:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27695.026424 secs 2024-04-26T19:32:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:35 smithi194 ceph-mon[33017]: pgmap v15125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:35 smithi033 ceph-mon[27845]: pgmap v15125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:35 smithi096 ceph-mon[32753]: pgmap v15125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:37 smithi194 ceph-mon[33017]: pgmap v15126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:37 smithi033 ceph-mon[27845]: pgmap v15126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:37 smithi096 ceph-mon[32753]: pgmap v15126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:38.099+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:38.099+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:38.099+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:32:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27697.122776 secs 2024-04-26T19:32:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27697.122776 secs 2024-04-26T19:32:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27697.122776 secs 2024-04-26T19:32:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:39 smithi194 ceph-mon[33017]: pgmap v15127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27700.026542 secs 2024-04-26T19:32:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:32:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:39 smithi033 ceph-mon[27845]: pgmap v15127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27700.026542 secs 2024-04-26T19:32:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:32:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:39 smithi096 ceph-mon[32753]: pgmap v15127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27700.026542 secs 2024-04-26T19:32:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:32:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:32:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:32:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:32:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:32:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:32:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:32:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:32:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:32:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:32:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:41 smithi194 ceph-mon[33017]: pgmap v15128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:32:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:41 smithi194 ceph-mon[33017]: pgmap v15129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:41 smithi033 ceph-mon[27845]: pgmap v15128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:32:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:41 smithi033 ceph-mon[27845]: pgmap v15129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:41 smithi096 ceph-mon[32753]: pgmap v15128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:32:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:41 smithi096 ceph-mon[32753]: pgmap v15129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:43 smithi194 ceph-mon[33017]: pgmap v15130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27702.122856 secs 2024-04-26T19:32:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:43 smithi033 ceph-mon[27845]: pgmap v15130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27702.122856 secs 2024-04-26T19:32:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:43 smithi096 ceph-mon[32753]: pgmap v15130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27702.122856 secs 2024-04-26T19:32:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27705.026610 secs 2024-04-26T19:32:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27705.026610 secs 2024-04-26T19:32:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27705.026610 secs 2024-04-26T19:32:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:45 smithi194 ceph-mon[33017]: pgmap v15131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:32:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:45 smithi033 ceph-mon[27845]: pgmap v15131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:32:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:45 smithi096 ceph-mon[32753]: pgmap v15131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:32:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:47 smithi033 ceph-mon[27845]: pgmap v15132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:47 smithi096 ceph-mon[32753]: pgmap v15132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:47 smithi194 ceph-mon[33017]: pgmap v15132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27707.122939 secs 2024-04-26T19:32:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27707.122939 secs 2024-04-26T19:32:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27707.122939 secs 2024-04-26T19:32:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:49 smithi033 ceph-mon[27845]: pgmap v15133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27710.026689 secs 2024-04-26T19:32:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:49 smithi096 ceph-mon[32753]: pgmap v15133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27710.026689 secs 2024-04-26T19:32:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:49 smithi194 ceph-mon[33017]: pgmap v15133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:32:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27710.026689 secs 2024-04-26T19:32:51.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:51 smithi194 ceph-mon[33017]: pgmap v15134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:32:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:51 smithi033 ceph-mon[27845]: pgmap v15134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:32:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:51 smithi096 ceph-mon[32753]: pgmap v15134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:32:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:53.109+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:53.109+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:32:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:32:53.109+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:32:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:53 smithi033 ceph-mon[27845]: pgmap v15135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27712.122999 secs 2024-04-26T19:32:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:53 smithi096 ceph-mon[32753]: pgmap v15135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27712.122999 secs 2024-04-26T19:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:53 smithi194 ceph-mon[33017]: pgmap v15135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27712.122999 secs 2024-04-26T19:32:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27715.026804 secs 2024-04-26T19:32:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27715.026804 secs 2024-04-26T19:32:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27715.026804 secs 2024-04-26T19:32:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:55 smithi033 ceph-mon[27845]: pgmap v15136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:55 smithi096 ceph-mon[32753]: pgmap v15136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:55 smithi194 ceph-mon[33017]: pgmap v15136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:57 smithi033 ceph-mon[27845]: pgmap v15137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:57 smithi096 ceph-mon[32753]: pgmap v15137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:57 smithi194 ceph-mon[33017]: pgmap v15137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:32:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27717.123081 secs 2024-04-26T19:32:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27717.123081 secs 2024-04-26T19:32:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27717.123081 secs 2024-04-26T19:32:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:59 smithi033 ceph-mon[27845]: pgmap v15138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27720.026923 secs 2024-04-26T19:32:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:59 smithi096 ceph-mon[32753]: pgmap v15138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:32:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27720.026923 secs 2024-04-26T19:33:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:59 smithi194 ceph-mon[33017]: pgmap v15138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27720.026923 secs 2024-04-26T19:33:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:01 smithi033 ceph-mon[27845]: pgmap v15139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:01 smithi096 ceph-mon[32753]: pgmap v15139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:01 smithi194 ceph-mon[33017]: pgmap v15139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:03 smithi096 ceph-mon[32753]: pgmap v15140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27722.123165 secs 2024-04-26T19:33:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:03 smithi194 ceph-mon[33017]: pgmap v15140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27722.123165 secs 2024-04-26T19:33:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:03 smithi033 ceph-mon[27845]: pgmap v15140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27722.123165 secs 2024-04-26T19:33:04.943 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27725.027044 secs 2024-04-26T19:33:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27725.027044 secs 2024-04-26T19:33:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27725.027044 secs 2024-04-26T19:33:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:05 smithi194 ceph-mon[33017]: pgmap v15141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:05 smithi033 ceph-mon[27845]: pgmap v15141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:05 smithi096 ceph-mon[32753]: pgmap v15141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:07 smithi194 ceph-mon[33017]: pgmap v15142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:08.119+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:08.119+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:08.119+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:33:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:07 smithi033 ceph-mon[27845]: pgmap v15142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:07 smithi096 ceph-mon[32753]: pgmap v15142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27727.123227 secs 2024-04-26T19:33:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27727.123227 secs 2024-04-26T19:33:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27727.123227 secs 2024-04-26T19:33:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:09 smithi194 ceph-mon[33017]: pgmap v15143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27730.027165 secs 2024-04-26T19:33:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:09 smithi033 ceph-mon[27845]: pgmap v15143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27730.027165 secs 2024-04-26T19:33:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:09 smithi096 ceph-mon[32753]: pgmap v15143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27730.027165 secs 2024-04-26T19:33:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:11 smithi194 ceph-mon[33017]: pgmap v15144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:11 smithi033 ceph-mon[27845]: pgmap v15144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:11 smithi096 ceph-mon[32753]: pgmap v15144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:13 smithi194 ceph-mon[33017]: pgmap v15145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27732.123284 secs 2024-04-26T19:33:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:13 smithi033 ceph-mon[27845]: pgmap v15145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27732.123284 secs 2024-04-26T19:33:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:13 smithi096 ceph-mon[32753]: pgmap v15145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27732.123284 secs 2024-04-26T19:33:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27735.027248 secs 2024-04-26T19:33:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27735.027248 secs 2024-04-26T19:33:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27735.027248 secs 2024-04-26T19:33:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:15 smithi194 ceph-mon[33017]: pgmap v15146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:15 smithi033 ceph-mon[27845]: pgmap v15146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:15 smithi096 ceph-mon[32753]: pgmap v15146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:17 smithi194 ceph-mon[33017]: pgmap v15147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:17 smithi033 ceph-mon[27845]: pgmap v15147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:17 smithi096 ceph-mon[32753]: pgmap v15147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27737.123347 secs 2024-04-26T19:33:19.260 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27737.123347 secs 2024-04-26T19:33:19.260 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27737.123347 secs 2024-04-26T19:33:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:19 smithi194 ceph-mon[33017]: pgmap v15148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27740.027372 secs 2024-04-26T19:33:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:19 smithi033 ceph-mon[27845]: pgmap v15148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27740.027372 secs 2024-04-26T19:33:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:19 smithi096 ceph-mon[32753]: pgmap v15148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27740.027372 secs 2024-04-26T19:33:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:21 smithi194 ceph-mon[33017]: pgmap v15149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:21 smithi033 ceph-mon[27845]: pgmap v15149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:21 smithi096 ceph-mon[32753]: pgmap v15149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:23.132+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:23.132+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:23.132+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:33:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:23 smithi194 ceph-mon[33017]: pgmap v15150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27742.123424 secs 2024-04-26T19:33:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:23 smithi033 ceph-mon[27845]: pgmap v15150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27742.123424 secs 2024-04-26T19:33:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:23 smithi096 ceph-mon[32753]: pgmap v15150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27742.123424 secs 2024-04-26T19:33:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27745.027431 secs 2024-04-26T19:33:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27745.027431 secs 2024-04-26T19:33:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27745.027431 secs 2024-04-26T19:33:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:25 smithi194 ceph-mon[33017]: pgmap v15151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:25 smithi033 ceph-mon[27845]: pgmap v15151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:25 smithi096 ceph-mon[32753]: pgmap v15151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:27 smithi194 ceph-mon[33017]: pgmap v15152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:27 smithi033 ceph-mon[27845]: pgmap v15152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:27 smithi096 ceph-mon[32753]: pgmap v15152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:29.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27747.123509 secs 2024-04-26T19:33:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27747.123509 secs 2024-04-26T19:33:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27747.123509 secs 2024-04-26T19:33:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:29 smithi194 ceph-mon[33017]: pgmap v15153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27750.027538 secs 2024-04-26T19:33:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:29 smithi033 ceph-mon[27845]: pgmap v15153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27750.027538 secs 2024-04-26T19:33:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:29 smithi096 ceph-mon[32753]: pgmap v15153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27750.027538 secs 2024-04-26T19:33:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:31 smithi194 ceph-mon[33017]: pgmap v15154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:31 smithi033 ceph-mon[27845]: pgmap v15154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:31 smithi096 ceph-mon[32753]: pgmap v15154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:33 smithi194 ceph-mon[33017]: pgmap v15155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27752.123592 secs 2024-04-26T19:33:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:33 smithi033 ceph-mon[27845]: pgmap v15155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27752.123592 secs 2024-04-26T19:33:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:33 smithi096 ceph-mon[32753]: pgmap v15155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27752.123592 secs 2024-04-26T19:33:35.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27755.027660 secs 2024-04-26T19:33:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27755.027660 secs 2024-04-26T19:33:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27755.027660 secs 2024-04-26T19:33:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:35 smithi194 ceph-mon[33017]: pgmap v15156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:35 smithi033 ceph-mon[27845]: pgmap v15156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:35 smithi096 ceph-mon[32753]: pgmap v15156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:37 smithi194 ceph-mon[33017]: pgmap v15157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:38.136+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:38.136+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:38.136+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:33:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:37 smithi033 ceph-mon[27845]: pgmap v15157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:37 smithi096 ceph-mon[32753]: pgmap v15157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27757.123677 secs 2024-04-26T19:33:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27757.123677 secs 2024-04-26T19:33:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27757.123677 secs 2024-04-26T19:33:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:39 smithi194 ceph-mon[33017]: pgmap v15158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27760.027777 secs 2024-04-26T19:33:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:39 smithi033 ceph-mon[27845]: pgmap v15158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27760.027777 secs 2024-04-26T19:33:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:39 smithi096 ceph-mon[32753]: pgmap v15158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27760.027777 secs 2024-04-26T19:33:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:33:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:33:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:33:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:41 smithi194 ceph-mon[33017]: pgmap v15159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:41 smithi033 ceph-mon[27845]: pgmap v15159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:41 smithi096 ceph-mon[32753]: pgmap v15159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: pgmap v15160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:33:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:33:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: pgmap v15161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:33:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: pgmap v15162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27762.123763 secs 2024-04-26T19:33:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: pgmap v15160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: pgmap v15161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: pgmap v15162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27762.123763 secs 2024-04-26T19:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: pgmap v15160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: pgmap v15161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:33:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: pgmap v15162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:33:43.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27762.123763 secs 2024-04-26T19:33:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27765.027900 secs 2024-04-26T19:33:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27765.027900 secs 2024-04-26T19:33:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27765.027900 secs 2024-04-26T19:33:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:45 smithi194 ceph-mon[33017]: pgmap v15163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:45 smithi033 ceph-mon[27845]: pgmap v15163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:45 smithi096 ceph-mon[32753]: pgmap v15163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:47 smithi033 ceph-mon[27845]: pgmap v15164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:47 smithi096 ceph-mon[32753]: pgmap v15164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:47 smithi194 ceph-mon[33017]: pgmap v15164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27767.123843 secs 2024-04-26T19:33:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27767.123843 secs 2024-04-26T19:33:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27767.123843 secs 2024-04-26T19:33:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:49 smithi033 ceph-mon[27845]: pgmap v15165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:33:50.055 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27770.028019 secs 2024-04-26T19:33:50.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:49 smithi096 ceph-mon[32753]: pgmap v15165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:33:50.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27770.028019 secs 2024-04-26T19:33:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:49 smithi194 ceph-mon[33017]: pgmap v15165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:33:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27770.028019 secs 2024-04-26T19:33:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:51 smithi033 ceph-mon[27845]: pgmap v15166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:51 smithi096 ceph-mon[32753]: pgmap v15166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:51 smithi194 ceph-mon[33017]: pgmap v15166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:33:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:53.148+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:53.148+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:33:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:33:53.148+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:33:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:53 smithi033 ceph-mon[27845]: pgmap v15167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:33:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27772.123927 secs 2024-04-26T19:33:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:53 smithi096 ceph-mon[32753]: pgmap v15167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:33:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27772.123927 secs 2024-04-26T19:33:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:53 smithi194 ceph-mon[33017]: pgmap v15167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:33:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27772.123927 secs 2024-04-26T19:33:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27775.028137 secs 2024-04-26T19:33:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27775.028137 secs 2024-04-26T19:33:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27775.028137 secs 2024-04-26T19:33:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:55 smithi194 ceph-mon[33017]: pgmap v15168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:55 smithi033 ceph-mon[27845]: pgmap v15168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:55 smithi096 ceph-mon[32753]: pgmap v15168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:33:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:57 smithi194 ceph-mon[33017]: pgmap v15169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:57 smithi033 ceph-mon[27845]: pgmap v15169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:57 smithi096 ceph-mon[32753]: pgmap v15169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:33:59.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27777.124009 secs 2024-04-26T19:33:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27777.124009 secs 2024-04-26T19:33:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27777.124009 secs 2024-04-26T19:34:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:59 smithi194 ceph-mon[33017]: pgmap v15170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:33:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27780.028179 secs 2024-04-26T19:34:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:59 smithi033 ceph-mon[27845]: pgmap v15170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:33:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27780.028179 secs 2024-04-26T19:34:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:59 smithi096 ceph-mon[32753]: pgmap v15170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:33:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27780.028179 secs 2024-04-26T19:34:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:01 smithi194 ceph-mon[33017]: pgmap v15171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:01 smithi033 ceph-mon[27845]: pgmap v15171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:01 smithi096 ceph-mon[32753]: pgmap v15171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:03 smithi194 ceph-mon[33017]: pgmap v15172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27782.124099 secs 2024-04-26T19:34:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:03 smithi033 ceph-mon[27845]: pgmap v15172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27782.124099 secs 2024-04-26T19:34:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:03 smithi096 ceph-mon[32753]: pgmap v15172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27782.124099 secs 2024-04-26T19:34:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27785.028300 secs 2024-04-26T19:34:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27785.028300 secs 2024-04-26T19:34:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27785.028300 secs 2024-04-26T19:34:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:05 smithi194 ceph-mon[33017]: pgmap v15173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:05 smithi033 ceph-mon[27845]: pgmap v15173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:05 smithi096 ceph-mon[32753]: pgmap v15173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:07 smithi194 ceph-mon[33017]: pgmap v15174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:08.149+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:08.149+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:34:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:08.149+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:07 smithi033 ceph-mon[27845]: pgmap v15174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:07 smithi096 ceph-mon[32753]: pgmap v15174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27787.124163 secs 2024-04-26T19:34:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27787.124163 secs 2024-04-26T19:34:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27787.124163 secs 2024-04-26T19:34:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27790.028421 secs 2024-04-26T19:34:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:09 smithi194 ceph-mon[33017]: pgmap v15175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27790.028421 secs 2024-04-26T19:34:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:09 smithi033 ceph-mon[27845]: pgmap v15175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27790.028421 secs 2024-04-26T19:34:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:09 smithi096 ceph-mon[32753]: pgmap v15175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:11 smithi194 ceph-mon[33017]: pgmap v15176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:11 smithi033 ceph-mon[27845]: pgmap v15176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:11 smithi096 ceph-mon[32753]: pgmap v15176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:13 smithi194 ceph-mon[33017]: pgmap v15177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27792.124247 secs 2024-04-26T19:34:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:13 smithi033 ceph-mon[27845]: pgmap v15177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27792.124247 secs 2024-04-26T19:34:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:13 smithi096 ceph-mon[32753]: pgmap v15177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27792.124247 secs 2024-04-26T19:34:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27795.028532 secs 2024-04-26T19:34:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27795.028532 secs 2024-04-26T19:34:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27795.028532 secs 2024-04-26T19:34:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:15 smithi194 ceph-mon[33017]: pgmap v15178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:15 smithi033 ceph-mon[27845]: pgmap v15178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:15 smithi096 ceph-mon[32753]: pgmap v15178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:17 smithi194 ceph-mon[33017]: pgmap v15179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:17 smithi033 ceph-mon[27845]: pgmap v15179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:17 smithi096 ceph-mon[32753]: pgmap v15179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27797.124336 secs 2024-04-26T19:34:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27797.124336 secs 2024-04-26T19:34:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27797.124336 secs 2024-04-26T19:34:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27800.028650 secs 2024-04-26T19:34:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:19 smithi194 ceph-mon[33017]: pgmap v15180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27800.028650 secs 2024-04-26T19:34:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:19 smithi033 ceph-mon[27845]: pgmap v15180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27800.028650 secs 2024-04-26T19:34:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:19 smithi096 ceph-mon[32753]: pgmap v15180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:21 smithi194 ceph-mon[33017]: pgmap v15181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:21 smithi033 ceph-mon[27845]: pgmap v15181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:21 smithi096 ceph-mon[32753]: pgmap v15181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:23.161+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:23.161+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:23.161+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:34:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:23 smithi194 ceph-mon[33017]: pgmap v15182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27802.124439 secs 2024-04-26T19:34:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:23 smithi033 ceph-mon[27845]: pgmap v15182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27802.124439 secs 2024-04-26T19:34:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:23 smithi096 ceph-mon[32753]: pgmap v15182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27802.124439 secs 2024-04-26T19:34:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27805.028783 secs 2024-04-26T19:34:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27805.028783 secs 2024-04-26T19:34:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27805.028783 secs 2024-04-26T19:34:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:25 smithi194 ceph-mon[33017]: pgmap v15183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:25 smithi033 ceph-mon[27845]: pgmap v15183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:25 smithi096 ceph-mon[32753]: pgmap v15183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:27 smithi194 ceph-mon[33017]: pgmap v15184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:27 smithi033 ceph-mon[27845]: pgmap v15184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:27 smithi096 ceph-mon[32753]: pgmap v15184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27807.124519 secs 2024-04-26T19:34:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27807.124519 secs 2024-04-26T19:34:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27807.124519 secs 2024-04-26T19:34:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27810.028926 secs 2024-04-26T19:34:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:29 smithi194 ceph-mon[33017]: pgmap v15185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27810.028926 secs 2024-04-26T19:34:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:29 smithi033 ceph-mon[27845]: pgmap v15185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27810.028926 secs 2024-04-26T19:34:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:29 smithi096 ceph-mon[32753]: pgmap v15185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:31 smithi194 ceph-mon[33017]: pgmap v15186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:31 smithi033 ceph-mon[27845]: pgmap v15186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:31 smithi096 ceph-mon[32753]: pgmap v15186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:33 smithi033 ceph-mon[27845]: pgmap v15187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27812.124603 secs 2024-04-26T19:34:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:33 smithi096 ceph-mon[32753]: pgmap v15187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27812.124603 secs 2024-04-26T19:34:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:33 smithi194 ceph-mon[33017]: pgmap v15187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27812.124603 secs 2024-04-26T19:34:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27815.028994 secs 2024-04-26T19:34:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27815.028994 secs 2024-04-26T19:34:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27815.028994 secs 2024-04-26T19:34:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:35 smithi033 ceph-mon[27845]: pgmap v15188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:35 smithi096 ceph-mon[32753]: pgmap v15188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:35 smithi194 ceph-mon[33017]: pgmap v15188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:38.127 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:37 smithi033 ceph-mon[27845]: pgmap v15189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:38.161+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:38.161+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:34:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:38.161+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:37 smithi096 ceph-mon[32753]: pgmap v15189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:37 smithi194 ceph-mon[33017]: pgmap v15189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27817.124666 secs 2024-04-26T19:34:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27817.124666 secs 2024-04-26T19:34:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27817.124666 secs 2024-04-26T19:34:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27820.029149 secs 2024-04-26T19:34:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:39 smithi033 ceph-mon[27845]: pgmap v15190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27820.029149 secs 2024-04-26T19:34:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:39 smithi096 ceph-mon[32753]: pgmap v15190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27820.029149 secs 2024-04-26T19:34:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:39 smithi194 ceph-mon[33017]: pgmap v15190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:41 smithi033 ceph-mon[27845]: pgmap v15191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:41 smithi096 ceph-mon[32753]: pgmap v15191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:41 smithi194 ceph-mon[33017]: pgmap v15191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:34:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:34:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:34:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:43 smithi033 ceph-mon[27845]: pgmap v15192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27822.124749 secs 2024-04-26T19:34:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:43 smithi096 ceph-mon[32753]: pgmap v15192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27822.124749 secs 2024-04-26T19:34:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:43 smithi194 ceph-mon[33017]: pgmap v15192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27822.124749 secs 2024-04-26T19:34:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27825.029208 secs 2024-04-26T19:34:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:34:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:34:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:34:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27825.029208 secs 2024-04-26T19:34:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:34:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:34:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:34:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27825.029208 secs 2024-04-26T19:34:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:34:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:34:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:34:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:45 smithi033 ceph-mon[27845]: pgmap v15193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:45 smithi033 ceph-mon[27845]: pgmap v15194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:34:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:45 smithi096 ceph-mon[32753]: pgmap v15193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:45 smithi096 ceph-mon[32753]: pgmap v15194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:34:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:45 smithi194 ceph-mon[33017]: pgmap v15193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:34:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:45 smithi194 ceph-mon[33017]: pgmap v15194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:34:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:47 smithi033 ceph-mon[27845]: pgmap v15195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:47 smithi096 ceph-mon[32753]: pgmap v15195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:47 smithi194 ceph-mon[33017]: pgmap v15195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27827.124831 secs 2024-04-26T19:34:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27827.124831 secs 2024-04-26T19:34:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27827.124831 secs 2024-04-26T19:34:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:49 smithi194 ceph-mon[33017]: pgmap v15196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27830.029634 secs 2024-04-26T19:34:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:49 smithi033 ceph-mon[27845]: pgmap v15196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27830.029634 secs 2024-04-26T19:34:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:49 smithi096 ceph-mon[32753]: pgmap v15196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27830.029634 secs 2024-04-26T19:34:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:51 smithi194 ceph-mon[33017]: pgmap v15197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:51 smithi033 ceph-mon[27845]: pgmap v15197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:51 smithi096 ceph-mon[32753]: pgmap v15197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:53.162 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:53.162+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:53.162+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:34:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:34:53.162+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:34:54.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:53 smithi033 ceph-mon[27845]: pgmap v15198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:54.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27832.124917 secs 2024-04-26T19:34:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:53 smithi194 ceph-mon[33017]: pgmap v15198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27832.124917 secs 2024-04-26T19:34:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:53 smithi096 ceph-mon[32753]: pgmap v15198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:34:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27832.124917 secs 2024-04-26T19:34:55.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27835.029749 secs 2024-04-26T19:34:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27835.029749 secs 2024-04-26T19:34:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27835.029749 secs 2024-04-26T19:34:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:55 smithi194 ceph-mon[33017]: pgmap v15199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:34:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:55 smithi033 ceph-mon[27845]: pgmap v15199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:34:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:55 smithi096 ceph-mon[32753]: pgmap v15199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:34:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:57 smithi194 ceph-mon[33017]: pgmap v15200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:57 smithi033 ceph-mon[27845]: pgmap v15200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:57 smithi096 ceph-mon[32753]: pgmap v15200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:34:59.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27837.125006 secs 2024-04-26T19:34:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27837.125006 secs 2024-04-26T19:34:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27837.125006 secs 2024-04-26T19:35:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:59 smithi194 ceph-mon[33017]: pgmap v15201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27840.029860 secs 2024-04-26T19:35:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:59 smithi033 ceph-mon[27845]: pgmap v15201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27840.029860 secs 2024-04-26T19:35:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:59 smithi096 ceph-mon[32753]: pgmap v15201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27840.029860 secs 2024-04-26T19:35:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:00 smithi194 ceph-mon[33017]: pgmap v15202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:00 smithi033 ceph-mon[27845]: pgmap v15202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:01.536 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:00 smithi096 ceph-mon[32753]: pgmap v15202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:03 smithi194 ceph-mon[33017]: pgmap v15203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:03 smithi033 ceph-mon[27845]: pgmap v15203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:03 smithi096 ceph-mon[32753]: pgmap v15203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27842.125065 secs 2024-04-26T19:35:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27845.029974 secs 2024-04-26T19:35:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27842.125065 secs 2024-04-26T19:35:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27845.029974 secs 2024-04-26T19:35:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27842.125065 secs 2024-04-26T19:35:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27845.029974 secs 2024-04-26T19:35:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:05 smithi194 ceph-mon[33017]: pgmap v15204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:05 smithi033 ceph-mon[27845]: pgmap v15204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:05 smithi096 ceph-mon[32753]: pgmap v15204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:07 smithi194 ceph-mon[33017]: pgmap v15205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:07 smithi033 ceph-mon[27845]: pgmap v15205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:07 smithi096 ceph-mon[32753]: pgmap v15205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:08.172+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:08.172+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:35:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:08.172+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27847.125156 secs 2024-04-26T19:35:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:09 smithi194 ceph-mon[33017]: pgmap v15206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27850.030088 secs 2024-04-26T19:35:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27847.125156 secs 2024-04-26T19:35:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:09 smithi033 ceph-mon[27845]: pgmap v15206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27850.030088 secs 2024-04-26T19:35:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27847.125156 secs 2024-04-26T19:35:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:09 smithi096 ceph-mon[32753]: pgmap v15206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27850.030088 secs 2024-04-26T19:35:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:11 smithi194 ceph-mon[33017]: pgmap v15207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:11 smithi033 ceph-mon[27845]: pgmap v15207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:11 smithi096 ceph-mon[32753]: pgmap v15207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:13 smithi194 ceph-mon[33017]: pgmap v15208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:13 smithi033 ceph-mon[27845]: pgmap v15208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:13 smithi096 ceph-mon[32753]: pgmap v15208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27852.125236 secs 2024-04-26T19:35:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27855.030181 secs 2024-04-26T19:35:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27852.125236 secs 2024-04-26T19:35:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27855.030181 secs 2024-04-26T19:35:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27852.125236 secs 2024-04-26T19:35:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27855.030181 secs 2024-04-26T19:35:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:15 smithi194 ceph-mon[33017]: pgmap v15209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:15 smithi033 ceph-mon[27845]: pgmap v15209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:15 smithi096 ceph-mon[32753]: pgmap v15209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:17 smithi194 ceph-mon[33017]: pgmap v15210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:17 smithi033 ceph-mon[27845]: pgmap v15210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:17 smithi096 ceph-mon[32753]: pgmap v15210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27857.125297 secs 2024-04-26T19:35:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:19 smithi194 ceph-mon[33017]: pgmap v15211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27860.030294 secs 2024-04-26T19:35:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27857.125297 secs 2024-04-26T19:35:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:19 smithi033 ceph-mon[27845]: pgmap v15211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27860.030294 secs 2024-04-26T19:35:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27857.125297 secs 2024-04-26T19:35:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:19 smithi096 ceph-mon[32753]: pgmap v15211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27860.030294 secs 2024-04-26T19:35:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:21 smithi194 ceph-mon[33017]: pgmap v15212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:21 smithi033 ceph-mon[27845]: pgmap v15212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:21 smithi096 ceph-mon[32753]: pgmap v15212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:23.184+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:23.184+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:23.184+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:35:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:23 smithi194 ceph-mon[33017]: pgmap v15213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:23 smithi033 ceph-mon[27845]: pgmap v15213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:23 smithi096 ceph-mon[32753]: pgmap v15213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:24.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27862.125382 secs 2024-04-26T19:35:24.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27865.030433 secs 2024-04-26T19:35:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27862.125382 secs 2024-04-26T19:35:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27865.030433 secs 2024-04-26T19:35:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27862.125382 secs 2024-04-26T19:35:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27865.030433 secs 2024-04-26T19:35:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:25 smithi033 ceph-mon[27845]: pgmap v15214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:25 smithi096 ceph-mon[32753]: pgmap v15214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:25 smithi194 ceph-mon[33017]: pgmap v15214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:26.536+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:35:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:26.536+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:35:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:26.537+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 220 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:35:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:26.537+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 220 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:35:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:27 smithi033 ceph-mon[27845]: pgmap v15215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:27 smithi096 ceph-mon[32753]: pgmap v15215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:27 smithi194 ceph-mon[33017]: pgmap v15215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27867.125469 secs 2024-04-26T19:35:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:29 smithi033 ceph-mon[27845]: pgmap v15216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27870.030565 secs 2024-04-26T19:35:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27867.125469 secs 2024-04-26T19:35:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:29 smithi096 ceph-mon[32753]: pgmap v15216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27870.030565 secs 2024-04-26T19:35:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27867.125469 secs 2024-04-26T19:35:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:29 smithi194 ceph-mon[33017]: pgmap v15216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27870.030565 secs 2024-04-26T19:35:31.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:31 smithi033 ceph-mon[27845]: pgmap v15217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:31 smithi096 ceph-mon[32753]: pgmap v15217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:31 smithi194 ceph-mon[33017]: pgmap v15217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:33 smithi033 ceph-mon[27845]: pgmap v15218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:33 smithi096 ceph-mon[32753]: pgmap v15218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:33 smithi194 ceph-mon[33017]: pgmap v15218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27872.125553 secs 2024-04-26T19:35:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27875.030718 secs 2024-04-26T19:35:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27872.125553 secs 2024-04-26T19:35:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27875.030718 secs 2024-04-26T19:35:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27872.125553 secs 2024-04-26T19:35:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27875.030718 secs 2024-04-26T19:35:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:35 smithi033 ceph-mon[27845]: pgmap v15219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:35:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:35 smithi096 ceph-mon[32753]: pgmap v15219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:35:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:35 smithi194 ceph-mon[33017]: pgmap v15219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:35:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:37 smithi033 ceph-mon[27845]: pgmap v15220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:37 smithi096 ceph-mon[32753]: pgmap v15220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:37 smithi194 ceph-mon[33017]: pgmap v15220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:38.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:38.196+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:38.196+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:38.196+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:35:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27877.125640 secs 2024-04-26T19:35:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27877.125640 secs 2024-04-26T19:35:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27877.125640 secs 2024-04-26T19:35:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:39 smithi194 ceph-mon[33017]: pgmap v15221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27880.030873 secs 2024-04-26T19:35:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:39 smithi033 ceph-mon[27845]: pgmap v15221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27880.030873 secs 2024-04-26T19:35:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:39 smithi096 ceph-mon[32753]: pgmap v15221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27880.030873 secs 2024-04-26T19:35:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:41 smithi194 ceph-mon[33017]: pgmap v15222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:41 smithi033 ceph-mon[27845]: pgmap v15222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:41 smithi096 ceph-mon[32753]: pgmap v15222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:35:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:43 smithi194 ceph-mon[33017]: pgmap v15223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27882.125699 secs 2024-04-26T19:35:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:43 smithi033 ceph-mon[27845]: pgmap v15223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27882.125699 secs 2024-04-26T19:35:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:43 smithi096 ceph-mon[32753]: pgmap v15223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27882.125699 secs 2024-04-26T19:35:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27885.031001 secs 2024-04-26T19:35:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:35:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27885.031001 secs 2024-04-26T19:35:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:35:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27885.031001 secs 2024-04-26T19:35:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:35:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:45 smithi194 ceph-mon[33017]: pgmap v15224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:35:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:45 smithi033 ceph-mon[27845]: pgmap v15224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:35:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:45 smithi096 ceph-mon[32753]: pgmap v15224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:35:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:35:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:35:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:46 smithi033 ceph-mon[27845]: pgmap v15225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:46 smithi033 ceph-mon[27845]: pgmap v15226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:35:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:35:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:35:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:46 smithi096 ceph-mon[32753]: pgmap v15225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:46 smithi096 ceph-mon[32753]: pgmap v15226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:35:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:35:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:35:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:46 smithi194 ceph-mon[33017]: pgmap v15225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:46 smithi194 ceph-mon[33017]: pgmap v15226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:35:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27887.125786 secs 2024-04-26T19:35:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:48 smithi194 ceph-mon[33017]: pgmap v15227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27887.125786 secs 2024-04-26T19:35:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:48 smithi033 ceph-mon[27845]: pgmap v15227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27887.125786 secs 2024-04-26T19:35:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:48 smithi096 ceph-mon[32753]: pgmap v15227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27890.031138 secs 2024-04-26T19:35:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27890.031138 secs 2024-04-26T19:35:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27890.031138 secs 2024-04-26T19:35:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:50 smithi194 ceph-mon[33017]: pgmap v15228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:50 smithi033 ceph-mon[27845]: pgmap v15228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:50 smithi096 ceph-mon[32753]: pgmap v15228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:52 smithi194 ceph-mon[33017]: pgmap v15229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:52 smithi033 ceph-mon[27845]: pgmap v15229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:52 smithi096 ceph-mon[32753]: pgmap v15229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:35:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:53.212+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:53.212+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:35:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:35:53.212+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:35:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27892.125874 secs 2024-04-26T19:35:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27892.125874 secs 2024-04-26T19:35:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27892.125874 secs 2024-04-26T19:35:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27895.031264 secs 2024-04-26T19:35:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:54 smithi194 ceph-mon[33017]: pgmap v15230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:35:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27895.031264 secs 2024-04-26T19:35:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:54 smithi033 ceph-mon[27845]: pgmap v15230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:35:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27895.031264 secs 2024-04-26T19:35:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:54 smithi096 ceph-mon[32753]: pgmap v15230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:35:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:56 smithi194 ceph-mon[33017]: pgmap v15231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:35:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:56 smithi033 ceph-mon[27845]: pgmap v15231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:35:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:56 smithi096 ceph-mon[32753]: pgmap v15231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27897.125972 secs 2024-04-26T19:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:58 smithi194 ceph-mon[33017]: pgmap v15232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27897.125972 secs 2024-04-26T19:35:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:58 smithi033 ceph-mon[27845]: pgmap v15232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27897.125972 secs 2024-04-26T19:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:58 smithi096 ceph-mon[32753]: pgmap v15232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27900.031397 secs 2024-04-26T19:36:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27900.031397 secs 2024-04-26T19:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27900.031397 secs 2024-04-26T19:36:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:00 smithi194 ceph-mon[33017]: pgmap v15233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:00 smithi033 ceph-mon[27845]: pgmap v15233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:00 smithi096 ceph-mon[32753]: pgmap v15233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:01 smithi194 ceph-mon[33017]: pgmap v15234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:01 smithi033 ceph-mon[27845]: pgmap v15234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:01 smithi096 ceph-mon[32753]: pgmap v15234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27902.126050 secs 2024-04-26T19:36:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27902.126050 secs 2024-04-26T19:36:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27902.126050 secs 2024-04-26T19:36:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27905.031525 secs 2024-04-26T19:36:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:04 smithi194 ceph-mon[33017]: pgmap v15235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27905.031525 secs 2024-04-26T19:36:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:04 smithi033 ceph-mon[27845]: pgmap v15235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27905.031525 secs 2024-04-26T19:36:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:04 smithi096 ceph-mon[32753]: pgmap v15235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:06 smithi194 ceph-mon[33017]: pgmap v15236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:06 smithi033 ceph-mon[27845]: pgmap v15236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:06 smithi096 ceph-mon[32753]: pgmap v15236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:08.225+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:08.225+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:08.225+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:36:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27907.126138 secs 2024-04-26T19:36:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:08 smithi194 ceph-mon[33017]: pgmap v15237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27907.126138 secs 2024-04-26T19:36:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:08 smithi033 ceph-mon[27845]: pgmap v15237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27907.126138 secs 2024-04-26T19:36:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:08 smithi096 ceph-mon[32753]: pgmap v15237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27910.031604 secs 2024-04-26T19:36:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27910.031604 secs 2024-04-26T19:36:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27910.031604 secs 2024-04-26T19:36:11.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:11 smithi033 ceph-mon[27845]: pgmap v15238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:11 smithi194 ceph-mon[33017]: pgmap v15238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:11 smithi096 ceph-mon[32753]: pgmap v15238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:12 smithi194 ceph-mon[33017]: pgmap v15239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:12 smithi033 ceph-mon[27845]: pgmap v15239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:12 smithi096 ceph-mon[32753]: pgmap v15239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27912.126224 secs 2024-04-26T19:36:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27912.126224 secs 2024-04-26T19:36:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27912.126224 secs 2024-04-26T19:36:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27915.031680 secs 2024-04-26T19:36:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:14 smithi194 ceph-mon[33017]: pgmap v15240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27915.031680 secs 2024-04-26T19:36:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:14 smithi033 ceph-mon[27845]: pgmap v15240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27915.031680 secs 2024-04-26T19:36:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:14 smithi096 ceph-mon[32753]: pgmap v15240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:16 smithi194 ceph-mon[33017]: pgmap v15241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:16 smithi033 ceph-mon[27845]: pgmap v15241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:16 smithi096 ceph-mon[32753]: pgmap v15241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27917.126328 secs 2024-04-26T19:36:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:18 smithi194 ceph-mon[33017]: pgmap v15242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27917.126328 secs 2024-04-26T19:36:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:18 smithi033 ceph-mon[27845]: pgmap v15242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27917.126328 secs 2024-04-26T19:36:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:18 smithi096 ceph-mon[32753]: pgmap v15242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27920.031812 secs 2024-04-26T19:36:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27920.031812 secs 2024-04-26T19:36:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27920.031812 secs 2024-04-26T19:36:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:21 smithi194 ceph-mon[33017]: pgmap v15243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:21 smithi033 ceph-mon[27845]: pgmap v15243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:21 smithi096 ceph-mon[32753]: pgmap v15243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:22 smithi194 ceph-mon[33017]: pgmap v15244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:22 smithi033 ceph-mon[27845]: pgmap v15244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:22 smithi096 ceph-mon[32753]: pgmap v15244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:23.240+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:23.240+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:23.240+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:36:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27922.126385 secs 2024-04-26T19:36:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27922.126385 secs 2024-04-26T19:36:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27922.126385 secs 2024-04-26T19:36:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27925.031947 secs 2024-04-26T19:36:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:24 smithi194 ceph-mon[33017]: pgmap v15245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27925.031947 secs 2024-04-26T19:36:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:24 smithi033 ceph-mon[27845]: pgmap v15245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27925.031947 secs 2024-04-26T19:36:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:24 smithi096 ceph-mon[32753]: pgmap v15245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:26 smithi194 ceph-mon[33017]: pgmap v15246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:26 smithi033 ceph-mon[27845]: pgmap v15246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:26 smithi096 ceph-mon[32753]: pgmap v15246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27927.126468 secs 2024-04-26T19:36:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:29 smithi194 ceph-mon[33017]: pgmap v15247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27927.126468 secs 2024-04-26T19:36:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:29 smithi033 ceph-mon[27845]: pgmap v15247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27927.126468 secs 2024-04-26T19:36:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:29 smithi096 ceph-mon[32753]: pgmap v15247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27930.032062 secs 2024-04-26T19:36:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:30 smithi194 ceph-mon[33017]: pgmap v15248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27930.032062 secs 2024-04-26T19:36:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:30 smithi033 ceph-mon[27845]: pgmap v15248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27930.032062 secs 2024-04-26T19:36:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:30 smithi096 ceph-mon[32753]: pgmap v15248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:32 smithi194 ceph-mon[33017]: pgmap v15249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:32 smithi033 ceph-mon[27845]: pgmap v15249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:32 smithi096 ceph-mon[32753]: pgmap v15249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27932.126552 secs 2024-04-26T19:36:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27932.126552 secs 2024-04-26T19:36:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27932.126552 secs 2024-04-26T19:36:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27935.032176 secs 2024-04-26T19:36:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:35 smithi194 ceph-mon[33017]: pgmap v15250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27935.032176 secs 2024-04-26T19:36:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:35 smithi033 ceph-mon[27845]: pgmap v15250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27935.032176 secs 2024-04-26T19:36:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:35 smithi096 ceph-mon[32753]: pgmap v15250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:36 smithi194 ceph-mon[33017]: pgmap v15251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:36 smithi033 ceph-mon[27845]: pgmap v15251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:36 smithi096 ceph-mon[32753]: pgmap v15251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:38.254+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:38.254+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:38.254+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:36:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27937.126638 secs 2024-04-26T19:36:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:39 smithi194 ceph-mon[33017]: pgmap v15252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27937.126638 secs 2024-04-26T19:36:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:39 smithi033 ceph-mon[27845]: pgmap v15252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27937.126638 secs 2024-04-26T19:36:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:39 smithi096 ceph-mon[32753]: pgmap v15252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27940.032293 secs 2024-04-26T19:36:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:40 smithi194 ceph-mon[33017]: pgmap v15253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27940.032293 secs 2024-04-26T19:36:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:40 smithi033 ceph-mon[27845]: pgmap v15253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27940.032293 secs 2024-04-26T19:36:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:40 smithi096 ceph-mon[32753]: pgmap v15253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:42 smithi194 ceph-mon[33017]: pgmap v15254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:42 smithi033 ceph-mon[27845]: pgmap v15254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:42 smithi096 ceph-mon[32753]: pgmap v15254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27942.126724 secs 2024-04-26T19:36:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27942.126724 secs 2024-04-26T19:36:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27942.126724 secs 2024-04-26T19:36:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27945.032386 secs 2024-04-26T19:36:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:44 smithi194 ceph-mon[33017]: pgmap v15255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27945.032386 secs 2024-04-26T19:36:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:44 smithi033 ceph-mon[27845]: pgmap v15255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27945.032386 secs 2024-04-26T19:36:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:44 smithi096 ceph-mon[32753]: pgmap v15255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:46 smithi194 ceph-mon[33017]: pgmap v15256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:46 smithi033 ceph-mon[27845]: pgmap v15256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:46 smithi096 ceph-mon[32753]: pgmap v15256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:36:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:36:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:36:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:36:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:36:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:36:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:48 smithi194 ceph-mon[33017]: pgmap v15257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:48 smithi194 ceph-mon[33017]: pgmap v15258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:36:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:36:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:36:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:48 smithi033 ceph-mon[27845]: pgmap v15257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:48 smithi033 ceph-mon[27845]: pgmap v15258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:36:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:36:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:36:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:48 smithi096 ceph-mon[32753]: pgmap v15257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:36:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:48 smithi096 ceph-mon[32753]: pgmap v15258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:36:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27947.126813 secs 2024-04-26T19:36:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27947.126813 secs 2024-04-26T19:36:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27947.126813 secs 2024-04-26T19:36:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27950.032540 secs 2024-04-26T19:36:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:50 smithi194 ceph-mon[33017]: pgmap v15259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27950.032540 secs 2024-04-26T19:36:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:50 smithi033 ceph-mon[27845]: pgmap v15259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27950.032540 secs 2024-04-26T19:36:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:50 smithi096 ceph-mon[32753]: pgmap v15259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:52 smithi033 ceph-mon[27845]: pgmap v15260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:52 smithi096 ceph-mon[32753]: pgmap v15260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:53.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:52 smithi194 ceph-mon[33017]: pgmap v15260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:53.256+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:53.256+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:36:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:36:53.256+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:36:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27952.126898 secs 2024-04-26T19:36:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27952.126898 secs 2024-04-26T19:36:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27952.126898 secs 2024-04-26T19:36:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27955.032688 secs 2024-04-26T19:36:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:54 smithi033 ceph-mon[27845]: pgmap v15261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27955.032688 secs 2024-04-26T19:36:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:54 smithi096 ceph-mon[32753]: pgmap v15261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27955.032688 secs 2024-04-26T19:36:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:54 smithi194 ceph-mon[33017]: pgmap v15261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:56 smithi096 ceph-mon[32753]: pgmap v15262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:56 smithi194 ceph-mon[33017]: pgmap v15262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:56 smithi033 ceph-mon[27845]: pgmap v15262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:36:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:58 smithi096 ceph-mon[32753]: pgmap v15263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:36:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27957.127005 secs 2024-04-26T19:36:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:58 smithi033 ceph-mon[27845]: pgmap v15263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:36:59.081 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27957.127005 secs 2024-04-26T19:36:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:58 smithi194 ceph-mon[33017]: pgmap v15263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:36:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27957.127005 secs 2024-04-26T19:37:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27960.032805 secs 2024-04-26T19:37:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27960.032805 secs 2024-04-26T19:37:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27960.032805 secs 2024-04-26T19:37:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:00 smithi194 ceph-mon[33017]: pgmap v15264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:00 smithi033 ceph-mon[27845]: pgmap v15264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:00 smithi096 ceph-mon[32753]: pgmap v15264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:02 smithi194 ceph-mon[33017]: pgmap v15265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:02 smithi033 ceph-mon[27845]: pgmap v15265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:02 smithi096 ceph-mon[32753]: pgmap v15265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27962.127068 secs 2024-04-26T19:37:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27962.127068 secs 2024-04-26T19:37:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27962.127068 secs 2024-04-26T19:37:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27965.032924 secs 2024-04-26T19:37:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:04 smithi194 ceph-mon[33017]: pgmap v15266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27965.032924 secs 2024-04-26T19:37:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:04 smithi033 ceph-mon[27845]: pgmap v15266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27965.032924 secs 2024-04-26T19:37:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:04 smithi096 ceph-mon[32753]: pgmap v15266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:06 smithi194 ceph-mon[33017]: pgmap v15267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:06 smithi033 ceph-mon[27845]: pgmap v15267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:06 smithi096 ceph-mon[32753]: pgmap v15267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:08.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:08.272+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:08.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:08.272+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:37:08.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:08.272+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:08 smithi194 ceph-mon[33017]: pgmap v15268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27967.127127 secs 2024-04-26T19:37:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:08 smithi033 ceph-mon[27845]: pgmap v15268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27967.127127 secs 2024-04-26T19:37:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:08 smithi096 ceph-mon[32753]: pgmap v15268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:37:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27967.127127 secs 2024-04-26T19:37:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27970.033065 secs 2024-04-26T19:37:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27970.033065 secs 2024-04-26T19:37:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27970.033065 secs 2024-04-26T19:37:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:10 smithi194 ceph-mon[33017]: pgmap v15269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:10 smithi033 ceph-mon[27845]: pgmap v15269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:10 smithi096 ceph-mon[32753]: pgmap v15269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:37:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:12 smithi194 ceph-mon[33017]: pgmap v15270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:12 smithi033 ceph-mon[27845]: pgmap v15270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:12 smithi096 ceph-mon[32753]: pgmap v15270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27972.127214 secs 2024-04-26T19:37:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27972.127214 secs 2024-04-26T19:37:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27972.127214 secs 2024-04-26T19:37:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27975.033189 secs 2024-04-26T19:37:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:14 smithi194 ceph-mon[33017]: pgmap v15271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27975.033189 secs 2024-04-26T19:37:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:14 smithi033 ceph-mon[27845]: pgmap v15271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27975.033189 secs 2024-04-26T19:37:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:14 smithi096 ceph-mon[32753]: pgmap v15271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:16 smithi194 ceph-mon[33017]: pgmap v15272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:16 smithi033 ceph-mon[27845]: pgmap v15272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:16 smithi096 ceph-mon[32753]: pgmap v15272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:18 smithi194 ceph-mon[33017]: pgmap v15273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27977.127292 secs 2024-04-26T19:37:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:18 smithi033 ceph-mon[27845]: pgmap v15273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27977.127292 secs 2024-04-26T19:37:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:18 smithi096 ceph-mon[32753]: pgmap v15273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27977.127292 secs 2024-04-26T19:37:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27980.033282 secs 2024-04-26T19:37:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27980.033282 secs 2024-04-26T19:37:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27980.033282 secs 2024-04-26T19:37:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:20 smithi194 ceph-mon[33017]: pgmap v15274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:20 smithi033 ceph-mon[27845]: pgmap v15274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:20 smithi096 ceph-mon[32753]: pgmap v15274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:22 smithi194 ceph-mon[33017]: pgmap v15275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:22 smithi033 ceph-mon[27845]: pgmap v15275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:22 smithi096 ceph-mon[32753]: pgmap v15275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:23.288+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:23.288+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:37:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:23.288+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27982.127393 secs 2024-04-26T19:37:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27982.127393 secs 2024-04-26T19:37:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27982.127393 secs 2024-04-26T19:37:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27985.033375 secs 2024-04-26T19:37:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:24 smithi194 ceph-mon[33017]: pgmap v15276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27985.033375 secs 2024-04-26T19:37:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:24 smithi033 ceph-mon[27845]: pgmap v15276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27985.033375 secs 2024-04-26T19:37:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:24 smithi096 ceph-mon[32753]: pgmap v15276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:26 smithi194 ceph-mon[33017]: pgmap v15277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:26 smithi033 ceph-mon[27845]: pgmap v15277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:26 smithi096 ceph-mon[32753]: pgmap v15277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:28 smithi194 ceph-mon[33017]: pgmap v15278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27987.127494 secs 2024-04-26T19:37:29.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:28 smithi033 ceph-mon[27845]: pgmap v15278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:29.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27987.127494 secs 2024-04-26T19:37:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:28 smithi096 ceph-mon[32753]: pgmap v15278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27987.127494 secs 2024-04-26T19:37:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27990.033498 secs 2024-04-26T19:37:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27990.033498 secs 2024-04-26T19:37:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27990.033498 secs 2024-04-26T19:37:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:30 smithi194 ceph-mon[33017]: pgmap v15279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:30 smithi033 ceph-mon[27845]: pgmap v15279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:30 smithi096 ceph-mon[32753]: pgmap v15279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:32 smithi194 ceph-mon[33017]: pgmap v15280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:32 smithi033 ceph-mon[27845]: pgmap v15280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:32 smithi096 ceph-mon[32753]: pgmap v15280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27992.127584 secs 2024-04-26T19:37:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27992.127584 secs 2024-04-26T19:37:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27992.127584 secs 2024-04-26T19:37:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 27995.033588 secs 2024-04-26T19:37:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:34 smithi194 ceph-mon[33017]: pgmap v15281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 27995.033588 secs 2024-04-26T19:37:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:34 smithi033 ceph-mon[27845]: pgmap v15281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 27995.033588 secs 2024-04-26T19:37:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:34 smithi096 ceph-mon[32753]: pgmap v15281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:36 smithi194 ceph-mon[33017]: pgmap v15282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:36 smithi033 ceph-mon[27845]: pgmap v15282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:36 smithi096 ceph-mon[32753]: pgmap v15282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:38.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:38.295+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:38.295+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:38.295+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:37:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:38 smithi194 ceph-mon[33017]: pgmap v15283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 27997.127669 secs 2024-04-26T19:37:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:38 smithi033 ceph-mon[27845]: pgmap v15283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 27997.127669 secs 2024-04-26T19:37:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:38 smithi096 ceph-mon[32753]: pgmap v15283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 27997.127669 secs 2024-04-26T19:37:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28000.033723 secs 2024-04-26T19:37:40.346 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28000.033723 secs 2024-04-26T19:37:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28000.033723 secs 2024-04-26T19:37:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:40 smithi033 ceph-mon[27845]: pgmap v15284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:40 smithi096 ceph-mon[32753]: pgmap v15284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:40 smithi194 ceph-mon[33017]: pgmap v15284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:42 smithi033 ceph-mon[27845]: pgmap v15285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:42 smithi096 ceph-mon[32753]: pgmap v15285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:42 smithi194 ceph-mon[33017]: pgmap v15285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28002.127732 secs 2024-04-26T19:37:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28002.127732 secs 2024-04-26T19:37:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28002.127732 secs 2024-04-26T19:37:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28005.033819 secs 2024-04-26T19:37:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:44 smithi033 ceph-mon[27845]: pgmap v15286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28005.033819 secs 2024-04-26T19:37:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:44 smithi096 ceph-mon[32753]: pgmap v15286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28005.033819 secs 2024-04-26T19:37:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:44 smithi194 ceph-mon[33017]: pgmap v15286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:46 smithi033 ceph-mon[27845]: pgmap v15287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:46 smithi096 ceph-mon[32753]: pgmap v15287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:46 smithi194 ceph-mon[33017]: pgmap v15287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:48 smithi033 ceph-mon[27845]: pgmap v15288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28007.127835 secs 2024-04-26T19:37:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:37:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:48 smithi096 ceph-mon[32753]: pgmap v15288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28007.127835 secs 2024-04-26T19:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:48 smithi194 ceph-mon[33017]: pgmap v15288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:37:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28007.127835 secs 2024-04-26T19:37:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:37:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28010.033914 secs 2024-04-26T19:37:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28010.033914 secs 2024-04-26T19:37:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28010.033914 secs 2024-04-26T19:37:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:50 smithi033 ceph-mon[27845]: pgmap v15289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:37:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:37:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:50 smithi096 ceph-mon[32753]: pgmap v15289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:37:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:37:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:50 smithi194 ceph-mon[33017]: pgmap v15289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:37:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:37:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:37:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:37:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:51 smithi033 ceph-mon[27845]: pgmap v15290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:37:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:51 smithi033 ceph-mon[27845]: pgmap v15291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:51 smithi096 ceph-mon[32753]: pgmap v15290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:37:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:51 smithi096 ceph-mon[32753]: pgmap v15291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:51 smithi194 ceph-mon[33017]: pgmap v15290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:37:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:51 smithi194 ceph-mon[33017]: pgmap v15291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:53.304+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:53.304+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:37:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:37:53.304+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:37:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:53 smithi096 ceph-mon[32753]: pgmap v15292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28012.127938 secs 2024-04-26T19:37:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:53 smithi194 ceph-mon[33017]: pgmap v15292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28012.127938 secs 2024-04-26T19:37:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:53 smithi033 ceph-mon[27845]: pgmap v15292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28012.127938 secs 2024-04-26T19:37:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28015.034025 secs 2024-04-26T19:37:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28015.034025 secs 2024-04-26T19:37:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28015.034025 secs 2024-04-26T19:37:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:55 smithi194 ceph-mon[33017]: pgmap v15293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:37:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:55 smithi033 ceph-mon[27845]: pgmap v15293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:37:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:55 smithi096 ceph-mon[32753]: pgmap v15293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:37:58.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:57 smithi096 ceph-mon[32753]: pgmap v15294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:57 smithi194 ceph-mon[33017]: pgmap v15294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:57 smithi033 ceph-mon[27845]: pgmap v15294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:37:59.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28017.128021 secs 2024-04-26T19:37:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28017.128021 secs 2024-04-26T19:37:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28017.128021 secs 2024-04-26T19:38:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:59 smithi033 ceph-mon[27845]: pgmap v15295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:38:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28020.034167 secs 2024-04-26T19:38:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:59 smithi194 ceph-mon[33017]: pgmap v15295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:38:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28020.034167 secs 2024-04-26T19:38:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:59 smithi096 ceph-mon[32753]: pgmap v15295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:38:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28020.034167 secs 2024-04-26T19:38:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:01 smithi194 ceph-mon[33017]: pgmap v15296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:38:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:01 smithi033 ceph-mon[27845]: pgmap v15296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:38:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:01 smithi096 ceph-mon[32753]: pgmap v15296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:38:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:03 smithi194 ceph-mon[33017]: pgmap v15297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28022.128067 secs 2024-04-26T19:38:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:03 smithi033 ceph-mon[27845]: pgmap v15297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28022.128067 secs 2024-04-26T19:38:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:03 smithi096 ceph-mon[32753]: pgmap v15297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28022.128067 secs 2024-04-26T19:38:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28025.034233 secs 2024-04-26T19:38:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28025.034233 secs 2024-04-26T19:38:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28025.034233 secs 2024-04-26T19:38:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:05 smithi194 ceph-mon[33017]: pgmap v15298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:05 smithi033 ceph-mon[27845]: pgmap v15298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:05 smithi096 ceph-mon[32753]: pgmap v15298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:08.320+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:08.320+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:08.320+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:38:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:07 smithi194 ceph-mon[33017]: pgmap v15299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:07 smithi033 ceph-mon[27845]: pgmap v15299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:07 smithi096 ceph-mon[32753]: pgmap v15299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28027.128133 secs 2024-04-26T19:38:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28027.128133 secs 2024-04-26T19:38:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28027.128133 secs 2024-04-26T19:38:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:10 smithi194 ceph-mon[33017]: pgmap v15300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28030.034354 secs 2024-04-26T19:38:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:09 smithi033 ceph-mon[27845]: pgmap v15300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28030.034354 secs 2024-04-26T19:38:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:09 smithi096 ceph-mon[32753]: pgmap v15300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28030.034354 secs 2024-04-26T19:38:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:12 smithi194 ceph-mon[33017]: pgmap v15301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:12 smithi033 ceph-mon[27845]: pgmap v15301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:12 smithi096 ceph-mon[32753]: pgmap v15301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:14 smithi194 ceph-mon[33017]: pgmap v15302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28032.128195 secs 2024-04-26T19:38:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:14 smithi033 ceph-mon[27845]: pgmap v15302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28032.128195 secs 2024-04-26T19:38:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:14 smithi096 ceph-mon[32753]: pgmap v15302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28032.128195 secs 2024-04-26T19:38:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28035.034470 secs 2024-04-26T19:38:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28035.034470 secs 2024-04-26T19:38:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28035.034470 secs 2024-04-26T19:38:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:16 smithi194 ceph-mon[33017]: pgmap v15303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:16 smithi033 ceph-mon[27845]: pgmap v15303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:16 smithi096 ceph-mon[32753]: pgmap v15303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:17 smithi194 ceph-mon[33017]: pgmap v15304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:17 smithi033 ceph-mon[27845]: pgmap v15304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:17 smithi096 ceph-mon[32753]: pgmap v15304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28037.128282 secs 2024-04-26T19:38:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:19 smithi194 ceph-mon[33017]: pgmap v15305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28037.128282 secs 2024-04-26T19:38:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:19 smithi033 ceph-mon[27845]: pgmap v15305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28037.128282 secs 2024-04-26T19:38:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:19 smithi096 ceph-mon[32753]: pgmap v15305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28040.034588 secs 2024-04-26T19:38:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28040.034588 secs 2024-04-26T19:38:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28040.034588 secs 2024-04-26T19:38:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:21 smithi194 ceph-mon[33017]: pgmap v15306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:21 smithi033 ceph-mon[27845]: pgmap v15306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:21 smithi096 ceph-mon[32753]: pgmap v15306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:23.323+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:23.323+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:38:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:23.323+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:23 smithi194 ceph-mon[33017]: pgmap v15307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:23 smithi033 ceph-mon[27845]: pgmap v15307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:23 smithi096 ceph-mon[32753]: pgmap v15307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28042.128368 secs 2024-04-26T19:38:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28042.128368 secs 2024-04-26T19:38:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28042.128368 secs 2024-04-26T19:38:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28045.034665 secs 2024-04-26T19:38:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:25 smithi194 ceph-mon[33017]: pgmap v15308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28045.034665 secs 2024-04-26T19:38:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:25 smithi033 ceph-mon[27845]: pgmap v15308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28045.034665 secs 2024-04-26T19:38:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:25 smithi096 ceph-mon[32753]: pgmap v15308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:26.569+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:38:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:26.569+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:38:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:26.569+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 221 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:38:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:26.570+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 221 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:38:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:27 smithi194 ceph-mon[33017]: pgmap v15309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:27 smithi033 ceph-mon[27845]: pgmap v15309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:27 smithi096 ceph-mon[32753]: pgmap v15309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28047.128440 secs 2024-04-26T19:38:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:29 smithi194 ceph-mon[33017]: pgmap v15310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28050.034819 secs 2024-04-26T19:38:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28047.128440 secs 2024-04-26T19:38:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:29 smithi033 ceph-mon[27845]: pgmap v15310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28050.034819 secs 2024-04-26T19:38:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28047.128440 secs 2024-04-26T19:38:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:29 smithi096 ceph-mon[32753]: pgmap v15310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28050.034819 secs 2024-04-26T19:38:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:31 smithi033 ceph-mon[27845]: pgmap v15311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:31 smithi194 ceph-mon[33017]: pgmap v15311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:31 smithi096 ceph-mon[32753]: pgmap v15311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:33 smithi194 ceph-mon[33017]: pgmap v15312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:33 smithi033 ceph-mon[27845]: pgmap v15312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:33 smithi096 ceph-mon[32753]: pgmap v15312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28052.128522 secs 2024-04-26T19:38:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28055.034960 secs 2024-04-26T19:38:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28052.128522 secs 2024-04-26T19:38:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28055.034960 secs 2024-04-26T19:38:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28052.128522 secs 2024-04-26T19:38:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28055.034960 secs 2024-04-26T19:38:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:35 smithi194 ceph-mon[33017]: pgmap v15313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:35 smithi033 ceph-mon[27845]: pgmap v15313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:35 smithi096 ceph-mon[32753]: pgmap v15313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:37 smithi194 ceph-mon[33017]: pgmap v15314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:37 smithi033 ceph-mon[27845]: pgmap v15314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:37 smithi096 ceph-mon[32753]: pgmap v15314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:38.338+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:38.338+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:38.338+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:38:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28057.128606 secs 2024-04-26T19:38:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:39 smithi194 ceph-mon[33017]: pgmap v15315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28060.035152 secs 2024-04-26T19:38:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28057.128606 secs 2024-04-26T19:38:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:39 smithi033 ceph-mon[27845]: pgmap v15315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28060.035152 secs 2024-04-26T19:38:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28057.128606 secs 2024-04-26T19:38:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:39 smithi096 ceph-mon[32753]: pgmap v15315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28060.035152 secs 2024-04-26T19:38:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:41 smithi194 ceph-mon[33017]: pgmap v15316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:41 smithi033 ceph-mon[27845]: pgmap v15316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:41 smithi096 ceph-mon[32753]: pgmap v15316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:43 smithi194 ceph-mon[33017]: pgmap v15317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:43 smithi033 ceph-mon[27845]: pgmap v15317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:43 smithi096 ceph-mon[32753]: pgmap v15317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28062.128691 secs 2024-04-26T19:38:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28065.035220 secs 2024-04-26T19:38:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28062.128691 secs 2024-04-26T19:38:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28065.035220 secs 2024-04-26T19:38:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28062.128691 secs 2024-04-26T19:38:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28065.035220 secs 2024-04-26T19:38:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:45 smithi194 ceph-mon[33017]: pgmap v15318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:45 smithi033 ceph-mon[27845]: pgmap v15318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:45 smithi096 ceph-mon[32753]: pgmap v15318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:47 smithi194 ceph-mon[33017]: pgmap v15319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:47 smithi033 ceph-mon[27845]: pgmap v15319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:47 smithi096 ceph-mon[32753]: pgmap v15319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28067.128755 secs 2024-04-26T19:38:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:49 smithi194 ceph-mon[33017]: pgmap v15320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28070.035297 secs 2024-04-26T19:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28067.128755 secs 2024-04-26T19:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:49 smithi033 ceph-mon[27845]: pgmap v15320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28070.035297 secs 2024-04-26T19:38:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28067.128755 secs 2024-04-26T19:38:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:49 smithi096 ceph-mon[32753]: pgmap v15320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28070.035297 secs 2024-04-26T19:38:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:51 smithi194 ceph-mon[33017]: pgmap v15321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:38:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:51 smithi033 ceph-mon[27845]: pgmap v15321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:38:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:51 smithi096 ceph-mon[32753]: pgmap v15321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:38:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:38:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:38:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:38:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:52 smithi033 ceph-mon[27845]: pgmap v15322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:52 smithi033 ceph-mon[27845]: pgmap v15323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:38:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:38:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:38:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:52 smithi096 ceph-mon[32753]: pgmap v15322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:52 smithi096 ceph-mon[32753]: pgmap v15323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:38:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:38:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:38:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:52 smithi194 ceph-mon[33017]: pgmap v15322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:38:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:52 smithi194 ceph-mon[33017]: pgmap v15323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:38:53.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:53.340+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:53.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:53.340+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:38:53.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:38:53.340+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:38:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28072.128840 secs 2024-04-26T19:38:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28072.128840 secs 2024-04-26T19:38:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28072.128840 secs 2024-04-26T19:38:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28075.035437 secs 2024-04-26T19:38:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:54 smithi194 ceph-mon[33017]: pgmap v15324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28075.035437 secs 2024-04-26T19:38:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:54 smithi033 ceph-mon[27845]: pgmap v15324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28075.035437 secs 2024-04-26T19:38:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:54 smithi096 ceph-mon[32753]: pgmap v15324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:56 smithi194 ceph-mon[33017]: pgmap v15325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:56 smithi033 ceph-mon[27845]: pgmap v15325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:56 smithi096 ceph-mon[32753]: pgmap v15325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:59.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28077.128921 secs 2024-04-26T19:38:59.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:58 smithi033 ceph-mon[27845]: pgmap v15326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:59.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28077.128921 secs 2024-04-26T19:38:59.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:58 smithi096 ceph-mon[32753]: pgmap v15326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:38:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28077.128921 secs 2024-04-26T19:38:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:58 smithi194 ceph-mon[33017]: pgmap v15326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:39:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28080.035563 secs 2024-04-26T19:39:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28080.035563 secs 2024-04-26T19:39:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28080.035563 secs 2024-04-26T19:39:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:00 smithi194 ceph-mon[33017]: pgmap v15327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:39:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:00 smithi033 ceph-mon[27845]: pgmap v15327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:39:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:00 smithi096 ceph-mon[32753]: pgmap v15327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:39:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:02 smithi194 ceph-mon[33017]: pgmap v15328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:39:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:02 smithi033 ceph-mon[27845]: pgmap v15328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:39:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:02 smithi096 ceph-mon[32753]: pgmap v15328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:39:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28082.129024 secs 2024-04-26T19:39:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28082.129024 secs 2024-04-26T19:39:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28082.129024 secs 2024-04-26T19:39:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28085.035664 secs 2024-04-26T19:39:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:04 smithi194 ceph-mon[33017]: pgmap v15329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28085.035664 secs 2024-04-26T19:39:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:04 smithi033 ceph-mon[27845]: pgmap v15329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28085.035664 secs 2024-04-26T19:39:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:04 smithi096 ceph-mon[32753]: pgmap v15329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:06 smithi194 ceph-mon[33017]: pgmap v15330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:06 smithi033 ceph-mon[27845]: pgmap v15330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:06 smithi096 ceph-mon[32753]: pgmap v15330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:08.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:08.357+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:08.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:08.357+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:08.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:08.357+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:39:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28087.129114 secs 2024-04-26T19:39:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:08 smithi194 ceph-mon[33017]: pgmap v15331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28087.129114 secs 2024-04-26T19:39:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:08 smithi033 ceph-mon[27845]: pgmap v15331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28087.129114 secs 2024-04-26T19:39:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:08 smithi096 ceph-mon[32753]: pgmap v15331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28090.035730 secs 2024-04-26T19:39:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28090.035730 secs 2024-04-26T19:39:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28090.035730 secs 2024-04-26T19:39:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:11 smithi194 ceph-mon[33017]: pgmap v15332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:11 smithi033 ceph-mon[27845]: pgmap v15332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:11 smithi096 ceph-mon[32753]: pgmap v15332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:13 smithi194 ceph-mon[33017]: pgmap v15333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:13 smithi033 ceph-mon[27845]: pgmap v15333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:13 smithi096 ceph-mon[32753]: pgmap v15333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28092.129191 secs 2024-04-26T19:39:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28092.129191 secs 2024-04-26T19:39:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28092.129191 secs 2024-04-26T19:39:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28095.035867 secs 2024-04-26T19:39:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:15 smithi194 ceph-mon[33017]: pgmap v15334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28095.035867 secs 2024-04-26T19:39:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:15 smithi033 ceph-mon[27845]: pgmap v15334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28095.035867 secs 2024-04-26T19:39:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:15 smithi096 ceph-mon[32753]: pgmap v15334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:17 smithi194 ceph-mon[33017]: pgmap v15335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:17 smithi033 ceph-mon[27845]: pgmap v15335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:17 smithi096 ceph-mon[32753]: pgmap v15335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28097.129281 secs 2024-04-26T19:39:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:19 smithi194 ceph-mon[33017]: pgmap v15336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28097.129281 secs 2024-04-26T19:39:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:19 smithi033 ceph-mon[27845]: pgmap v15336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28097.129281 secs 2024-04-26T19:39:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:19 smithi096 ceph-mon[32753]: pgmap v15336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28100.036006 secs 2024-04-26T19:39:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28100.036006 secs 2024-04-26T19:39:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28100.036006 secs 2024-04-26T19:39:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:21 smithi194 ceph-mon[33017]: pgmap v15337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:21 smithi033 ceph-mon[27845]: pgmap v15337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:21 smithi096 ceph-mon[32753]: pgmap v15337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:22 smithi194 ceph-mon[33017]: pgmap v15338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:22 smithi033 ceph-mon[27845]: pgmap v15338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:22 smithi096 ceph-mon[32753]: pgmap v15338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:23.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:23.372+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:39:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:23.372+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28102.129361 secs 2024-04-26T19:39:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28102.129361 secs 2024-04-26T19:39:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28102.129361 secs 2024-04-26T19:39:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28105.036174 secs 2024-04-26T19:39:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:24 smithi194 ceph-mon[33017]: pgmap v15339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28105.036174 secs 2024-04-26T19:39:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:24 smithi033 ceph-mon[27845]: pgmap v15339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28105.036174 secs 2024-04-26T19:39:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:24 smithi096 ceph-mon[32753]: pgmap v15339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:26 smithi194 ceph-mon[33017]: pgmap v15340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:26 smithi033 ceph-mon[27845]: pgmap v15340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:26 smithi096 ceph-mon[32753]: pgmap v15340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:29.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28107.129462 secs 2024-04-26T19:39:29.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:28 smithi033 ceph-mon[27845]: pgmap v15341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:29.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28107.129462 secs 2024-04-26T19:39:29.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:28 smithi096 ceph-mon[32753]: pgmap v15341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28107.129462 secs 2024-04-26T19:39:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:28 smithi194 ceph-mon[33017]: pgmap v15341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28110.036290 secs 2024-04-26T19:39:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28110.036290 secs 2024-04-26T19:39:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28110.036290 secs 2024-04-26T19:39:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:30 smithi194 ceph-mon[33017]: pgmap v15342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:30 smithi033 ceph-mon[27845]: pgmap v15342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:30 smithi096 ceph-mon[32753]: pgmap v15342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:32 smithi194 ceph-mon[33017]: pgmap v15343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:32 smithi033 ceph-mon[27845]: pgmap v15343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:32 smithi096 ceph-mon[32753]: pgmap v15343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:34.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28112.129559 secs 2024-04-26T19:39:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28112.129559 secs 2024-04-26T19:39:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28112.129559 secs 2024-04-26T19:39:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28115.036404 secs 2024-04-26T19:39:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:34 smithi194 ceph-mon[33017]: pgmap v15344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28115.036404 secs 2024-04-26T19:39:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:34 smithi033 ceph-mon[27845]: pgmap v15344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28115.036404 secs 2024-04-26T19:39:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:34 smithi096 ceph-mon[32753]: pgmap v15344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:37 smithi194 ceph-mon[33017]: pgmap v15345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:37 smithi033 ceph-mon[27845]: pgmap v15345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:37 smithi096 ceph-mon[32753]: pgmap v15345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:38.382+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:38.382+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:38.382+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:39:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28117.129660 secs 2024-04-26T19:39:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:39 smithi194 ceph-mon[33017]: pgmap v15346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28117.129660 secs 2024-04-26T19:39:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:39 smithi033 ceph-mon[27845]: pgmap v15346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28117.129660 secs 2024-04-26T19:39:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:39 smithi096 ceph-mon[32753]: pgmap v15346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28120.036524 secs 2024-04-26T19:39:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28120.036524 secs 2024-04-26T19:39:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28120.036524 secs 2024-04-26T19:39:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:41 smithi194 ceph-mon[33017]: pgmap v15347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:41 smithi033 ceph-mon[27845]: pgmap v15347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:41 smithi096 ceph-mon[32753]: pgmap v15347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:43 smithi194 ceph-mon[33017]: pgmap v15348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:43 smithi033 ceph-mon[27845]: pgmap v15348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:43 smithi096 ceph-mon[32753]: pgmap v15348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28122.129759 secs 2024-04-26T19:39:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28122.129759 secs 2024-04-26T19:39:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28122.129759 secs 2024-04-26T19:39:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28125.036604 secs 2024-04-26T19:39:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:45 smithi194 ceph-mon[33017]: pgmap v15349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28125.036604 secs 2024-04-26T19:39:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:45 smithi033 ceph-mon[27845]: pgmap v15349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28125.036604 secs 2024-04-26T19:39:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:45 smithi096 ceph-mon[32753]: pgmap v15349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:46 smithi194 ceph-mon[33017]: pgmap v15350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:46 smithi033 ceph-mon[27845]: pgmap v15350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:46 smithi096 ceph-mon[32753]: pgmap v15350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28127.129856 secs 2024-04-26T19:39:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:48 smithi033 ceph-mon[27845]: pgmap v15351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28127.129856 secs 2024-04-26T19:39:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:48 smithi194 ceph-mon[33017]: pgmap v15351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28127.129856 secs 2024-04-26T19:39:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:48 smithi096 ceph-mon[32753]: pgmap v15351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28130.036730 secs 2024-04-26T19:39:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28130.036730 secs 2024-04-26T19:39:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28130.036730 secs 2024-04-26T19:39:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:50 smithi194 ceph-mon[33017]: pgmap v15352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:50 smithi033 ceph-mon[27845]: pgmap v15352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:50 smithi096 ceph-mon[32753]: pgmap v15352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:52 smithi194 ceph-mon[33017]: pgmap v15353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:39:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:53.395+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:53.395+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:39:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:53.395+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:39:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:52 smithi033 ceph-mon[27845]: pgmap v15353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:39:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:52 smithi096 ceph-mon[32753]: pgmap v15353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:39:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28132.129959 secs 2024-04-26T19:39:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28132.129959 secs 2024-04-26T19:39:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28132.129959 secs 2024-04-26T19:39:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28135.036856 secs 2024-04-26T19:39:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:55 smithi194 ceph-mon[33017]: pgmap v15354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28135.036856 secs 2024-04-26T19:39:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:55 smithi033 ceph-mon[27845]: pgmap v15354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28135.036856 secs 2024-04-26T19:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:55 smithi096 ceph-mon[32753]: pgmap v15354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:56.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:56.584+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:39:56.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:56.584+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:39:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:56.584+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 222 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T19:39:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:39:56.585+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 222 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T19:39:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:57 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:39:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:57 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:39:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:57 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:39:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:57 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:39:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:57 smithi194 ceph-mon[33017]: pgmap v15355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:57 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:39:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:57 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:39:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:57 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:39:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:57 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:39:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:57 smithi033 ceph-mon[27845]: pgmap v15355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:57 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:39:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:57 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:39:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:57 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:39:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:57 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:39:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:57 smithi096 ceph-mon[32753]: pgmap v15355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:39:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28137.130022 secs 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: pgmap v15356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:39:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:39:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28137.130022 secs 2024-04-26T19:39:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: pgmap v15356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:39:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28137.130022 secs 2024-04-26T19:39:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: pgmap v15356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:39:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:39:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:39:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:39:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:39:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:39:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:39:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:59 smithi033 ceph-mon[27845]: pgmap v15357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:39:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:59 smithi033 ceph-mon[27845]: pgmap v15358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:39:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28140.036996 secs 2024-04-26T19:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:59 smithi096 ceph-mon[32753]: pgmap v15357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:59 smithi096 ceph-mon[32753]: pgmap v15358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28140.036996 secs 2024-04-26T19:40:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:40:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:40:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:59 smithi194 ceph-mon[33017]: pgmap v15357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:40:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:59 smithi194 ceph-mon[33017]: pgmap v15358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:40:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28140.036996 secs 2024-04-26T19:40:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:40:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:40:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:40:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:01 smithi194 ceph-mon[33017]: pgmap v15359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:01 smithi033 ceph-mon[27845]: pgmap v15359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:01 smithi096 ceph-mon[32753]: pgmap v15359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:03 smithi194 ceph-mon[33017]: pgmap v15360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28142.130105 secs 2024-04-26T19:40:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:03 smithi033 ceph-mon[27845]: pgmap v15360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28142.130105 secs 2024-04-26T19:40:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:03 smithi096 ceph-mon[32753]: pgmap v15360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28142.130105 secs 2024-04-26T19:40:05.060 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28145.037133 secs 2024-04-26T19:40:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28145.037133 secs 2024-04-26T19:40:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28145.037133 secs 2024-04-26T19:40:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:05 smithi194 ceph-mon[33017]: pgmap v15361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:40:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:05 smithi033 ceph-mon[27845]: pgmap v15361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:40:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:05 smithi096 ceph-mon[32753]: pgmap v15361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:40:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:07 smithi194 ceph-mon[33017]: pgmap v15362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:07 smithi033 ceph-mon[27845]: pgmap v15362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:07 smithi096 ceph-mon[32753]: pgmap v15362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:08.411+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:08.411+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:08.411+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:40:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28147.130196 secs 2024-04-26T19:40:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28147.130196 secs 2024-04-26T19:40:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28147.130196 secs 2024-04-26T19:40:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:09 smithi194 ceph-mon[33017]: pgmap v15363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:40:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28150.037224 secs 2024-04-26T19:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:09 smithi033 ceph-mon[27845]: pgmap v15363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:40:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28150.037224 secs 2024-04-26T19:40:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:09 smithi096 ceph-mon[32753]: pgmap v15363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:40:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28150.037224 secs 2024-04-26T19:40:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:11 smithi194 ceph-mon[33017]: pgmap v15364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:11 smithi033 ceph-mon[27845]: pgmap v15364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:11 smithi096 ceph-mon[32753]: pgmap v15364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:13 smithi194 ceph-mon[33017]: pgmap v15365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28152.130258 secs 2024-04-26T19:40:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:13 smithi033 ceph-mon[27845]: pgmap v15365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28152.130258 secs 2024-04-26T19:40:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:13 smithi096 ceph-mon[32753]: pgmap v15365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28152.130258 secs 2024-04-26T19:40:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28155.037334 secs 2024-04-26T19:40:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28155.037334 secs 2024-04-26T19:40:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28155.037334 secs 2024-04-26T19:40:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:15 smithi194 ceph-mon[33017]: pgmap v15366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:15 smithi033 ceph-mon[27845]: pgmap v15366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:15 smithi096 ceph-mon[32753]: pgmap v15366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:17 smithi194 ceph-mon[33017]: pgmap v15367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:17 smithi033 ceph-mon[27845]: pgmap v15367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:17 smithi096 ceph-mon[32753]: pgmap v15367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28157.130348 secs 2024-04-26T19:40:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28157.130348 secs 2024-04-26T19:40:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28157.130348 secs 2024-04-26T19:40:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:19 smithi194 ceph-mon[33017]: pgmap v15368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28160.037448 secs 2024-04-26T19:40:20.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:19 smithi033 ceph-mon[27845]: pgmap v15368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:20.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28160.037448 secs 2024-04-26T19:40:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:19 smithi096 ceph-mon[32753]: pgmap v15368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:40:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28160.037448 secs 2024-04-26T19:40:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:21 smithi194 ceph-mon[33017]: pgmap v15369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:21 smithi033 ceph-mon[27845]: pgmap v15369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:21 smithi096 ceph-mon[32753]: pgmap v15369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:40:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:23.427+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:23.427+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:23.427+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:23 smithi194 ceph-mon[33017]: pgmap v15370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28162.130410 secs 2024-04-26T19:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:23 smithi033 ceph-mon[27845]: pgmap v15370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28162.130410 secs 2024-04-26T19:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:23 smithi096 ceph-mon[32753]: pgmap v15370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28162.130410 secs 2024-04-26T19:40:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28165.037602 secs 2024-04-26T19:40:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28165.037602 secs 2024-04-26T19:40:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28165.037602 secs 2024-04-26T19:40:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:25 smithi194 ceph-mon[33017]: pgmap v15371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:25 smithi033 ceph-mon[27845]: pgmap v15371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:25 smithi096 ceph-mon[32753]: pgmap v15371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:27 smithi194 ceph-mon[33017]: pgmap v15372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:27 smithi033 ceph-mon[27845]: pgmap v15372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:27 smithi096 ceph-mon[32753]: pgmap v15372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:29.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28167.130496 secs 2024-04-26T19:40:29.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28167.130496 secs 2024-04-26T19:40:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28167.130496 secs 2024-04-26T19:40:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:29 smithi194 ceph-mon[33017]: pgmap v15373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28170.037744 secs 2024-04-26T19:40:30.104 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:29 smithi096 ceph-mon[32753]: pgmap v15373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:30.104 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28170.037744 secs 2024-04-26T19:40:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:29 smithi033 ceph-mon[27845]: pgmap v15373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28170.037744 secs 2024-04-26T19:40:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:31 smithi194 ceph-mon[33017]: pgmap v15374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:31 smithi033 ceph-mon[27845]: pgmap v15374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:31 smithi096 ceph-mon[32753]: pgmap v15374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:33 smithi194 ceph-mon[33017]: pgmap v15375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28172.130579 secs 2024-04-26T19:40:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:33 smithi033 ceph-mon[27845]: pgmap v15375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28172.130579 secs 2024-04-26T19:40:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:33 smithi096 ceph-mon[32753]: pgmap v15375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28172.130579 secs 2024-04-26T19:40:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28175.037870 secs 2024-04-26T19:40:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28175.037870 secs 2024-04-26T19:40:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28175.037870 secs 2024-04-26T19:40:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:35 smithi194 ceph-mon[33017]: pgmap v15376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:35 smithi033 ceph-mon[27845]: pgmap v15376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:35 smithi096 ceph-mon[32753]: pgmap v15376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:37 smithi194 ceph-mon[33017]: pgmap v15377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:37 smithi033 ceph-mon[27845]: pgmap v15377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:37 smithi096 ceph-mon[32753]: pgmap v15377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:38.440+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:38.440+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:38.440+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:40:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28177.130662 secs 2024-04-26T19:40:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28177.130662 secs 2024-04-26T19:40:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28177.130662 secs 2024-04-26T19:40:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:39 smithi194 ceph-mon[33017]: pgmap v15378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28180.037990 secs 2024-04-26T19:40:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:39 smithi033 ceph-mon[27845]: pgmap v15378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28180.037990 secs 2024-04-26T19:40:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:39 smithi096 ceph-mon[32753]: pgmap v15378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28180.037990 secs 2024-04-26T19:40:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:41 smithi033 ceph-mon[27845]: pgmap v15379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:41 smithi096 ceph-mon[32753]: pgmap v15379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:41 smithi194 ceph-mon[33017]: pgmap v15379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:43 smithi033 ceph-mon[27845]: pgmap v15380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28182.130748 secs 2024-04-26T19:40:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:43 smithi096 ceph-mon[32753]: pgmap v15380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28182.130748 secs 2024-04-26T19:40:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:43 smithi194 ceph-mon[33017]: pgmap v15380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28182.130748 secs 2024-04-26T19:40:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28185.038134 secs 2024-04-26T19:40:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28185.038134 secs 2024-04-26T19:40:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28185.038134 secs 2024-04-26T19:40:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:45 smithi033 ceph-mon[27845]: pgmap v15381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:45 smithi096 ceph-mon[32753]: pgmap v15381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:45 smithi194 ceph-mon[33017]: pgmap v15381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:47 smithi033 ceph-mon[27845]: pgmap v15382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:47 smithi096 ceph-mon[32753]: pgmap v15382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:47 smithi194 ceph-mon[33017]: pgmap v15382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28187.130835 secs 2024-04-26T19:40:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28187.130835 secs 2024-04-26T19:40:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28187.130835 secs 2024-04-26T19:40:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:49 smithi033 ceph-mon[27845]: pgmap v15383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28190.038258 secs 2024-04-26T19:40:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:49 smithi096 ceph-mon[32753]: pgmap v15383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28190.038258 secs 2024-04-26T19:40:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:49 smithi194 ceph-mon[33017]: pgmap v15383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28190.038258 secs 2024-04-26T19:40:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:51 smithi033 ceph-mon[27845]: pgmap v15384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:51 smithi096 ceph-mon[32753]: pgmap v15384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:51 smithi194 ceph-mon[33017]: pgmap v15384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:53.441+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:53.441+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:40:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:40:53.441+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:40:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:53 smithi033 ceph-mon[27845]: pgmap v15385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28192.130916 secs 2024-04-26T19:40:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:53 smithi096 ceph-mon[32753]: pgmap v15385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28192.130916 secs 2024-04-26T19:40:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:53 smithi194 ceph-mon[33017]: pgmap v15385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28192.130916 secs 2024-04-26T19:40:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28195.038369 secs 2024-04-26T19:40:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28195.038369 secs 2024-04-26T19:40:55.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28195.038369 secs 2024-04-26T19:40:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:55 smithi033 ceph-mon[27845]: pgmap v15386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:55 smithi096 ceph-mon[32753]: pgmap v15386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:55 smithi194 ceph-mon[33017]: pgmap v15386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:40:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:57 smithi033 ceph-mon[27845]: pgmap v15387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:57 smithi096 ceph-mon[32753]: pgmap v15387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:57 smithi194 ceph-mon[33017]: pgmap v15387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:40:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28197.130995 secs 2024-04-26T19:40:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:40:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28197.130995 secs 2024-04-26T19:40:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:40:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28197.130995 secs 2024-04-26T19:40:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:41:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:59 smithi194 ceph-mon[33017]: pgmap v15388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28200.038462 secs 2024-04-26T19:41:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:41:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:41:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:40:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:41:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:59 smithi033 ceph-mon[27845]: pgmap v15388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28200.038462 secs 2024-04-26T19:41:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:41:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:41:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:40:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:41:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:59 smithi096 ceph-mon[32753]: pgmap v15388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28200.038462 secs 2024-04-26T19:41:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:41:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:41:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:40:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:41:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:00 smithi194 ceph-mon[33017]: pgmap v15389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:41:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:00 smithi194 ceph-mon[33017]: pgmap v15390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:00 smithi033 ceph-mon[27845]: pgmap v15389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:41:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:00 smithi033 ceph-mon[27845]: pgmap v15390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:00 smithi096 ceph-mon[32753]: pgmap v15389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:41:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:00 smithi096 ceph-mon[32753]: pgmap v15390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:02 smithi194 ceph-mon[33017]: pgmap v15391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:02 smithi033 ceph-mon[27845]: pgmap v15391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:02 smithi096 ceph-mon[32753]: pgmap v15391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28202.131054 secs 2024-04-26T19:41:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28202.131054 secs 2024-04-26T19:41:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28202.131054 secs 2024-04-26T19:41:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28205.038526 secs 2024-04-26T19:41:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:04 smithi194 ceph-mon[33017]: pgmap v15392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28205.038526 secs 2024-04-26T19:41:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:04 smithi033 ceph-mon[27845]: pgmap v15392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28205.038526 secs 2024-04-26T19:41:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:04 smithi096 ceph-mon[32753]: pgmap v15392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:06 smithi194 ceph-mon[33017]: pgmap v15393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:06 smithi033 ceph-mon[27845]: pgmap v15393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:06 smithi096 ceph-mon[32753]: pgmap v15393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:08.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:08.447+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:08.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:08.447+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:08.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:08.447+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:41:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28207.131115 secs 2024-04-26T19:41:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:08 smithi194 ceph-mon[33017]: pgmap v15394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28207.131115 secs 2024-04-26T19:41:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:08 smithi033 ceph-mon[27845]: pgmap v15394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28207.131115 secs 2024-04-26T19:41:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:08 smithi096 ceph-mon[32753]: pgmap v15394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:41:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28210.038637 secs 2024-04-26T19:41:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28210.038637 secs 2024-04-26T19:41:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28210.038637 secs 2024-04-26T19:41:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:10 smithi194 ceph-mon[33017]: pgmap v15395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:41:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:10 smithi033 ceph-mon[27845]: pgmap v15395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:41:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:10 smithi096 ceph-mon[32753]: pgmap v15395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:41:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:12 smithi194 ceph-mon[33017]: pgmap v15396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:12 smithi033 ceph-mon[27845]: pgmap v15396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:12 smithi096 ceph-mon[32753]: pgmap v15396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28212.131196 secs 2024-04-26T19:41:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28212.131196 secs 2024-04-26T19:41:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28212.131196 secs 2024-04-26T19:41:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28215.038787 secs 2024-04-26T19:41:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:15 smithi194 ceph-mon[33017]: pgmap v15397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28215.038787 secs 2024-04-26T19:41:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:15 smithi033 ceph-mon[27845]: pgmap v15397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28215.038787 secs 2024-04-26T19:41:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:15 smithi096 ceph-mon[32753]: pgmap v15397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:17 smithi194 ceph-mon[33017]: pgmap v15398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:17 smithi033 ceph-mon[27845]: pgmap v15398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:17 smithi096 ceph-mon[32753]: pgmap v15398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28217.131281 secs 2024-04-26T19:41:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:19 smithi194 ceph-mon[33017]: pgmap v15399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28217.131281 secs 2024-04-26T19:41:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:19 smithi033 ceph-mon[27845]: pgmap v15399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28217.131281 secs 2024-04-26T19:41:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:19 smithi096 ceph-mon[32753]: pgmap v15399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28220.038886 secs 2024-04-26T19:41:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28220.038886 secs 2024-04-26T19:41:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28220.038886 secs 2024-04-26T19:41:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:21 smithi194 ceph-mon[33017]: pgmap v15400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:21 smithi033 ceph-mon[27845]: pgmap v15400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:21 smithi096 ceph-mon[32753]: pgmap v15400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T19:41:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T19:41:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0075070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T19:41:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0075070 comp rx=0 tx=0).stop 2024-04-26T19:41:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T19:41:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T19:41:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:41:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.795+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:41:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.796+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:21.797+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef007f040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:41:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:23 smithi194 ceph-mon[33017]: pgmap v15401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:23 smithi033 ceph-mon[27845]: pgmap v15401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:23 smithi096 ceph-mon[32753]: pgmap v15401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:23.449+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:23.450+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:23.450+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:41:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28222.131363 secs 2024-04-26T19:41:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28222.131363 secs 2024-04-26T19:41:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28222.131363 secs 2024-04-26T19:41:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28225.039002 secs 2024-04-26T19:41:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:25 smithi194 ceph-mon[33017]: pgmap v15402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28225.039002 secs 2024-04-26T19:41:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:25 smithi033 ceph-mon[27845]: pgmap v15402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28225.039002 secs 2024-04-26T19:41:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:25 smithi096 ceph-mon[32753]: pgmap v15402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:26.600+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:41:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:26.600+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:41:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:26.600+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 223 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T19:41:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:26.600+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 223 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T19:41:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:27 smithi194 ceph-mon[33017]: pgmap v15403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:27 smithi033 ceph-mon[27845]: pgmap v15403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:27 smithi096 ceph-mon[32753]: pgmap v15403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28227.131449 secs 2024-04-26T19:41:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:29 smithi194 ceph-mon[33017]: pgmap v15404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28227.131449 secs 2024-04-26T19:41:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:29 smithi033 ceph-mon[27845]: pgmap v15404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28227.131449 secs 2024-04-26T19:41:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:29 smithi096 ceph-mon[32753]: pgmap v15404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28230.039134 secs 2024-04-26T19:41:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28230.039134 secs 2024-04-26T19:41:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28230.039134 secs 2024-04-26T19:41:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:31 smithi194 ceph-mon[33017]: pgmap v15405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:31 smithi033 ceph-mon[27845]: pgmap v15405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:31 smithi096 ceph-mon[32753]: pgmap v15405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:33 smithi194 ceph-mon[33017]: pgmap v15406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:33 smithi033 ceph-mon[27845]: pgmap v15406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:33 smithi096 ceph-mon[32753]: pgmap v15406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28232.131535 secs 2024-04-26T19:41:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:34 smithi194 ceph-mon[33017]: pgmap v15407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28232.131535 secs 2024-04-26T19:41:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:34 smithi033 ceph-mon[27845]: pgmap v15407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28232.131535 secs 2024-04-26T19:41:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:34 smithi096 ceph-mon[32753]: pgmap v15407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28235.039228 secs 2024-04-26T19:41:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28235.039228 secs 2024-04-26T19:41:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28235.039228 secs 2024-04-26T19:41:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:36 smithi194 ceph-mon[33017]: pgmap v15408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:36 smithi033 ceph-mon[27845]: pgmap v15408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:36 smithi096 ceph-mon[32753]: pgmap v15408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:38.463+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:38.463+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:38.463+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:41:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28237.131620 secs 2024-04-26T19:41:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:38 smithi194 ceph-mon[33017]: pgmap v15409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28237.131620 secs 2024-04-26T19:41:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:38 smithi033 ceph-mon[27845]: pgmap v15409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28237.131620 secs 2024-04-26T19:41:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:38 smithi096 ceph-mon[32753]: pgmap v15409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28240.039311 secs 2024-04-26T19:41:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28240.039311 secs 2024-04-26T19:41:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28240.039311 secs 2024-04-26T19:41:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:40 smithi194 ceph-mon[33017]: pgmap v15410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:40 smithi033 ceph-mon[27845]: pgmap v15410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:40 smithi096 ceph-mon[32753]: pgmap v15410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:42 smithi194 ceph-mon[33017]: pgmap v15411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:42 smithi033 ceph-mon[27845]: pgmap v15411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:42 smithi096 ceph-mon[32753]: pgmap v15411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28242.131677 secs 2024-04-26T19:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28242.131677 secs 2024-04-26T19:41:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28242.131677 secs 2024-04-26T19:41:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28245.039420 secs 2024-04-26T19:41:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:45 smithi194 ceph-mon[33017]: pgmap v15412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28245.039420 secs 2024-04-26T19:41:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:45 smithi033 ceph-mon[27845]: pgmap v15412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28245.039420 secs 2024-04-26T19:41:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:45 smithi096 ceph-mon[32753]: pgmap v15412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:47 smithi194 ceph-mon[33017]: pgmap v15413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:47 smithi033 ceph-mon[27845]: pgmap v15413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:47 smithi096 ceph-mon[32753]: pgmap v15413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28247.131762 secs 2024-04-26T19:41:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:49 smithi194 ceph-mon[33017]: pgmap v15414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28247.131762 secs 2024-04-26T19:41:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:49 smithi033 ceph-mon[27845]: pgmap v15414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28247.131762 secs 2024-04-26T19:41:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:49 smithi096 ceph-mon[32753]: pgmap v15414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28250.039536 secs 2024-04-26T19:41:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28250.039536 secs 2024-04-26T19:41:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28250.039536 secs 2024-04-26T19:41:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:51 smithi194 ceph-mon[33017]: pgmap v15415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:51 smithi033 ceph-mon[27845]: pgmap v15415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:51 smithi096 ceph-mon[32753]: pgmap v15415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:53.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:53 smithi033 ceph-mon[27845]: pgmap v15416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:53 smithi194 ceph-mon[33017]: pgmap v15416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:53 smithi096 ceph-mon[32753]: pgmap v15416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:53.469+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:53.469+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:41:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:41:53.469+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28252.131865 secs 2024-04-26T19:41:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28252.131865 secs 2024-04-26T19:41:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28252.131865 secs 2024-04-26T19:41:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28255.039634 secs 2024-04-26T19:41:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:55 smithi194 ceph-mon[33017]: pgmap v15417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28255.039634 secs 2024-04-26T19:41:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:55 smithi033 ceph-mon[27845]: pgmap v15417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28255.039634 secs 2024-04-26T19:41:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:55 smithi096 ceph-mon[32753]: pgmap v15417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:57 smithi194 ceph-mon[33017]: pgmap v15418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:57 smithi033 ceph-mon[27845]: pgmap v15418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:57 smithi096 ceph-mon[32753]: pgmap v15418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:41:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28257.131948 secs 2024-04-26T19:41:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:41:59 smithi194 ceph-mon[33017]: pgmap v15419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28257.131948 secs 2024-04-26T19:41:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:41:59 smithi033 ceph-mon[27845]: pgmap v15419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:41:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28257.131948 secs 2024-04-26T19:41:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:41:59 smithi096 ceph-mon[32753]: pgmap v15419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28260.039783 secs 2024-04-26T19:42:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28260.039783 secs 2024-04-26T19:42:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28260.039783 secs 2024-04-26T19:42:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:01 smithi194 ceph-mon[33017]: pgmap v15420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:42:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:01 smithi033 ceph-mon[27845]: pgmap v15420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:42:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:01 smithi096 ceph-mon[32753]: pgmap v15420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:42:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:42:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:42:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:02 smithi194 ceph-mon[33017]: pgmap v15421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:02 smithi194 ceph-mon[33017]: pgmap v15422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:42:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:42:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:42:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:02 smithi033 ceph-mon[27845]: pgmap v15421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:02 smithi033 ceph-mon[27845]: pgmap v15422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:02 smithi096 ceph-mon[32753]: pgmap v15421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:02 smithi096 ceph-mon[32753]: pgmap v15422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:42:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28262.132028 secs 2024-04-26T19:42:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28262.132028 secs 2024-04-26T19:42:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28262.132028 secs 2024-04-26T19:42:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:04 smithi194 ceph-mon[33017]: pgmap v15423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28265.039919 secs 2024-04-26T19:42:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:04 smithi033 ceph-mon[27845]: pgmap v15423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28265.039919 secs 2024-04-26T19:42:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:04 smithi096 ceph-mon[32753]: pgmap v15423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28265.039919 secs 2024-04-26T19:42:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:06 smithi033 ceph-mon[27845]: pgmap v15424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:06 smithi096 ceph-mon[32753]: pgmap v15424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:06 smithi194 ceph-mon[33017]: pgmap v15424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:08.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:08.486+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:08.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:08.486+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:08.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:08.486+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:42:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:08 smithi033 ceph-mon[27845]: pgmap v15425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28267.132118 secs 2024-04-26T19:42:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:08 smithi096 ceph-mon[32753]: pgmap v15425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28267.132118 secs 2024-04-26T19:42:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:08 smithi194 ceph-mon[33017]: pgmap v15425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28267.132118 secs 2024-04-26T19:42:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28270.040028 secs 2024-04-26T19:42:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28270.040028 secs 2024-04-26T19:42:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28270.040028 secs 2024-04-26T19:42:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:10 smithi033 ceph-mon[27845]: pgmap v15426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:10 smithi096 ceph-mon[32753]: pgmap v15426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:10 smithi194 ceph-mon[33017]: pgmap v15426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:42:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:12 smithi033 ceph-mon[27845]: pgmap v15427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:42:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:12 smithi096 ceph-mon[32753]: pgmap v15427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:42:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:12 smithi194 ceph-mon[33017]: pgmap v15427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:42:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28272.132205 secs 2024-04-26T19:42:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28272.132205 secs 2024-04-26T19:42:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28272.132205 secs 2024-04-26T19:42:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:14 smithi194 ceph-mon[33017]: pgmap v15428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28275.040180 secs 2024-04-26T19:42:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:14 smithi033 ceph-mon[27845]: pgmap v15428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28275.040180 secs 2024-04-26T19:42:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:14 smithi096 ceph-mon[32753]: pgmap v15428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28275.040180 secs 2024-04-26T19:42:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:16 smithi194 ceph-mon[33017]: pgmap v15429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:16 smithi033 ceph-mon[27845]: pgmap v15429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:16 smithi096 ceph-mon[32753]: pgmap v15429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:18 smithi194 ceph-mon[33017]: pgmap v15430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28277.132266 secs 2024-04-26T19:42:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:18 smithi033 ceph-mon[27845]: pgmap v15430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28277.132266 secs 2024-04-26T19:42:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:18 smithi096 ceph-mon[32753]: pgmap v15430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28277.132266 secs 2024-04-26T19:42:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28280.040304 secs 2024-04-26T19:42:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28280.040304 secs 2024-04-26T19:42:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28280.040304 secs 2024-04-26T19:42:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:20 smithi194 ceph-mon[33017]: pgmap v15431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:20 smithi033 ceph-mon[27845]: pgmap v15431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:20 smithi096 ceph-mon[32753]: pgmap v15431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:22 smithi194 ceph-mon[33017]: pgmap v15432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:22 smithi033 ceph-mon[27845]: pgmap v15432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:22 smithi096 ceph-mon[32753]: pgmap v15432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:23.498+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:23.498+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:23.498+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:42:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28282.132326 secs 2024-04-26T19:42:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28282.132326 secs 2024-04-26T19:42:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28282.132326 secs 2024-04-26T19:42:25.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:24 smithi194 ceph-mon[33017]: pgmap v15433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:25.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28285.040423 secs 2024-04-26T19:42:25.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:24 smithi033 ceph-mon[27845]: pgmap v15433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:25.072 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28285.040423 secs 2024-04-26T19:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:24 smithi096 ceph-mon[32753]: pgmap v15433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28285.040423 secs 2024-04-26T19:42:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:26 smithi194 ceph-mon[33017]: pgmap v15434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:26 smithi033 ceph-mon[27845]: pgmap v15434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:26 smithi096 ceph-mon[32753]: pgmap v15434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:28 smithi194 ceph-mon[33017]: pgmap v15435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28287.132407 secs 2024-04-26T19:42:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:28 smithi033 ceph-mon[27845]: pgmap v15435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28287.132407 secs 2024-04-26T19:42:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:28 smithi096 ceph-mon[32753]: pgmap v15435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28287.132407 secs 2024-04-26T19:42:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28290.040498 secs 2024-04-26T19:42:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28290.040498 secs 2024-04-26T19:42:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28290.040498 secs 2024-04-26T19:42:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:30 smithi194 ceph-mon[33017]: pgmap v15436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:30 smithi033 ceph-mon[27845]: pgmap v15436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:30 smithi096 ceph-mon[32753]: pgmap v15436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:32 smithi194 ceph-mon[33017]: pgmap v15437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:32 smithi033 ceph-mon[27845]: pgmap v15437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:32 smithi096 ceph-mon[32753]: pgmap v15437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28292.132510 secs 2024-04-26T19:42:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28292.132510 secs 2024-04-26T19:42:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28292.132510 secs 2024-04-26T19:42:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:34 smithi194 ceph-mon[33017]: pgmap v15438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28295.040616 secs 2024-04-26T19:42:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:34 smithi033 ceph-mon[27845]: pgmap v15438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28295.040616 secs 2024-04-26T19:42:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:34 smithi096 ceph-mon[32753]: pgmap v15438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28295.040616 secs 2024-04-26T19:42:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:36 smithi194 ceph-mon[33017]: pgmap v15439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:36 smithi033 ceph-mon[27845]: pgmap v15439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:36 smithi096 ceph-mon[32753]: pgmap v15439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:38.513+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:38.513+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:38.513+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:42:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:38 smithi194 ceph-mon[33017]: pgmap v15440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28297.132592 secs 2024-04-26T19:42:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:38 smithi033 ceph-mon[27845]: pgmap v15440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28297.132592 secs 2024-04-26T19:42:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:38 smithi096 ceph-mon[32753]: pgmap v15440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28297.132592 secs 2024-04-26T19:42:40.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28300.040739 secs 2024-04-26T19:42:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28300.040739 secs 2024-04-26T19:42:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28300.040739 secs 2024-04-26T19:42:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:40 smithi194 ceph-mon[33017]: pgmap v15441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:40 smithi033 ceph-mon[27845]: pgmap v15441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:40 smithi096 ceph-mon[32753]: pgmap v15441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:42 smithi194 ceph-mon[33017]: pgmap v15442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:42 smithi033 ceph-mon[27845]: pgmap v15442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:42 smithi096 ceph-mon[32753]: pgmap v15442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28302.132674 secs 2024-04-26T19:42:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28302.132674 secs 2024-04-26T19:42:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28302.132674 secs 2024-04-26T19:42:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:44 smithi194 ceph-mon[33017]: pgmap v15443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28305.040819 secs 2024-04-26T19:42:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:44 smithi033 ceph-mon[27845]: pgmap v15443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28305.040819 secs 2024-04-26T19:42:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:44 smithi096 ceph-mon[32753]: pgmap v15443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28305.040819 secs 2024-04-26T19:42:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:46 smithi194 ceph-mon[33017]: pgmap v15444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:46 smithi033 ceph-mon[27845]: pgmap v15444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:46 smithi096 ceph-mon[32753]: pgmap v15444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:48 smithi194 ceph-mon[33017]: pgmap v15445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28307.132757 secs 2024-04-26T19:42:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:48 smithi033 ceph-mon[27845]: pgmap v15445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28307.132757 secs 2024-04-26T19:42:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:48 smithi096 ceph-mon[32753]: pgmap v15445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28307.132757 secs 2024-04-26T19:42:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28310.040939 secs 2024-04-26T19:42:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28310.040939 secs 2024-04-26T19:42:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28310.040939 secs 2024-04-26T19:42:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:50 smithi194 ceph-mon[33017]: pgmap v15446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:50 smithi033 ceph-mon[27845]: pgmap v15446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:50 smithi096 ceph-mon[32753]: pgmap v15446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:52 smithi194 ceph-mon[33017]: pgmap v15447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:52 smithi033 ceph-mon[27845]: pgmap v15447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:52 smithi096 ceph-mon[32753]: pgmap v15447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:53.527+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:53.527+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:42:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:42:53.527+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:42:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28312.132842 secs 2024-04-26T19:42:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28312.132842 secs 2024-04-26T19:42:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28312.132842 secs 2024-04-26T19:42:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28315.041056 secs 2024-04-26T19:42:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:54 smithi033 ceph-mon[27845]: pgmap v15448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28315.041056 secs 2024-04-26T19:42:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:54 smithi096 ceph-mon[32753]: pgmap v15448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28315.041056 secs 2024-04-26T19:42:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:54 smithi194 ceph-mon[33017]: pgmap v15448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:56 smithi033 ceph-mon[27845]: pgmap v15449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:56 smithi096 ceph-mon[32753]: pgmap v15449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:56 smithi194 ceph-mon[33017]: pgmap v15449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:42:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:58 smithi033 ceph-mon[27845]: pgmap v15450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28317.132926 secs 2024-04-26T19:42:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:58 smithi096 ceph-mon[32753]: pgmap v15450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28317.132926 secs 2024-04-26T19:42:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:58 smithi194 ceph-mon[33017]: pgmap v15450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:42:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28317.132926 secs 2024-04-26T19:43:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28320.041184 secs 2024-04-26T19:43:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28320.041184 secs 2024-04-26T19:43:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28320.041184 secs 2024-04-26T19:43:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:00 smithi033 ceph-mon[27845]: pgmap v15451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:00 smithi096 ceph-mon[32753]: pgmap v15451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:00 smithi194 ceph-mon[33017]: pgmap v15451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:02 smithi033 ceph-mon[27845]: pgmap v15452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:43:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:02 smithi096 ceph-mon[32753]: pgmap v15452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:43:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:02 smithi194 ceph-mon[33017]: pgmap v15452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:43:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28322.132977 secs 2024-04-26T19:43:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:43:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:43:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:43:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28322.132977 secs 2024-04-26T19:43:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:43:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:43:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:43:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28322.132977 secs 2024-04-26T19:43:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:43:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:43:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:43:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:04 smithi033 ceph-mon[27845]: pgmap v15453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:04 smithi033 ceph-mon[27845]: pgmap v15454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28325.041310 secs 2024-04-26T19:43:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:04 smithi096 ceph-mon[32753]: pgmap v15453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:04 smithi096 ceph-mon[32753]: pgmap v15454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28325.041310 secs 2024-04-26T19:43:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:04 smithi194 ceph-mon[33017]: pgmap v15453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:04 smithi194 ceph-mon[33017]: pgmap v15454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28325.041310 secs 2024-04-26T19:43:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:06 smithi033 ceph-mon[27845]: pgmap v15455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:06 smithi096 ceph-mon[32753]: pgmap v15455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:06 smithi194 ceph-mon[33017]: pgmap v15455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:08.529+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:08.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:08.529+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:43:08.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:08.529+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:08 smithi033 ceph-mon[27845]: pgmap v15456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28327.133037 secs 2024-04-26T19:43:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:08 smithi096 ceph-mon[32753]: pgmap v15456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28327.133037 secs 2024-04-26T19:43:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:08 smithi194 ceph-mon[33017]: pgmap v15456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28327.133037 secs 2024-04-26T19:43:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28330.041424 secs 2024-04-26T19:43:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28330.041424 secs 2024-04-26T19:43:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28330.041424 secs 2024-04-26T19:43:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:10 smithi033 ceph-mon[27845]: pgmap v15457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:43:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:10 smithi096 ceph-mon[32753]: pgmap v15457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:43:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:10 smithi194 ceph-mon[33017]: pgmap v15457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:43:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:12 smithi096 ceph-mon[32753]: pgmap v15458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:12 smithi194 ceph-mon[33017]: pgmap v15458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:12 smithi033 ceph-mon[27845]: pgmap v15458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:43:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28332.133090 secs 2024-04-26T19:43:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28332.133090 secs 2024-04-26T19:43:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28332.133090 secs 2024-04-26T19:43:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:14 smithi194 ceph-mon[33017]: pgmap v15459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:43:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28335.041541 secs 2024-04-26T19:43:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:14 smithi033 ceph-mon[27845]: pgmap v15459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:43:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28335.041541 secs 2024-04-26T19:43:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:14 smithi096 ceph-mon[32753]: pgmap v15459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:43:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28335.041541 secs 2024-04-26T19:43:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:16 smithi194 ceph-mon[33017]: pgmap v15460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:16 smithi033 ceph-mon[27845]: pgmap v15460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:16 smithi096 ceph-mon[32753]: pgmap v15460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:18 smithi194 ceph-mon[33017]: pgmap v15461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28337.133147 secs 2024-04-26T19:43:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:18 smithi033 ceph-mon[27845]: pgmap v15461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28337.133147 secs 2024-04-26T19:43:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:18 smithi096 ceph-mon[32753]: pgmap v15461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28337.133147 secs 2024-04-26T19:43:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28340.041637 secs 2024-04-26T19:43:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28340.041637 secs 2024-04-26T19:43:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28340.041637 secs 2024-04-26T19:43:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:20 smithi194 ceph-mon[33017]: pgmap v15462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:20 smithi033 ceph-mon[27845]: pgmap v15462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:20 smithi096 ceph-mon[32753]: pgmap v15462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:22 smithi194 ceph-mon[33017]: pgmap v15463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:22 smithi033 ceph-mon[27845]: pgmap v15463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:22 smithi096 ceph-mon[32753]: pgmap v15463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:23.531+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:23.531+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:23.531+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:43:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28342.133205 secs 2024-04-26T19:43:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28342.133205 secs 2024-04-26T19:43:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28342.133205 secs 2024-04-26T19:43:25.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:24 smithi194 ceph-mon[33017]: pgmap v15464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:25.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28345.041781 secs 2024-04-26T19:43:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:24 smithi033 ceph-mon[27845]: pgmap v15464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28345.041781 secs 2024-04-26T19:43:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:24 smithi096 ceph-mon[32753]: pgmap v15464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28345.041781 secs 2024-04-26T19:43:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:26 smithi194 ceph-mon[33017]: pgmap v15465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:26 smithi033 ceph-mon[27845]: pgmap v15465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:26 smithi096 ceph-mon[32753]: pgmap v15465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:28 smithi194 ceph-mon[33017]: pgmap v15466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28347.133266 secs 2024-04-26T19:43:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:28 smithi033 ceph-mon[27845]: pgmap v15466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28347.133266 secs 2024-04-26T19:43:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:28 smithi096 ceph-mon[32753]: pgmap v15466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28347.133266 secs 2024-04-26T19:43:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28350.041915 secs 2024-04-26T19:43:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28350.041915 secs 2024-04-26T19:43:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28350.041915 secs 2024-04-26T19:43:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:31 smithi194 ceph-mon[33017]: pgmap v15467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:31 smithi033 ceph-mon[27845]: pgmap v15467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:31 smithi096 ceph-mon[32753]: pgmap v15467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:33 smithi194 ceph-mon[33017]: pgmap v15468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:33 smithi033 ceph-mon[27845]: pgmap v15468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:33 smithi096 ceph-mon[32753]: pgmap v15468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28352.133324 secs 2024-04-26T19:43:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28352.133324 secs 2024-04-26T19:43:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28352.133324 secs 2024-04-26T19:43:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:35 smithi194 ceph-mon[33017]: pgmap v15469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28355.042037 secs 2024-04-26T19:43:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:35 smithi033 ceph-mon[27845]: pgmap v15469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28355.042037 secs 2024-04-26T19:43:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:35 smithi096 ceph-mon[32753]: pgmap v15469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28355.042037 secs 2024-04-26T19:43:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:37 smithi194 ceph-mon[33017]: pgmap v15470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:37 smithi033 ceph-mon[27845]: pgmap v15470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:37 smithi096 ceph-mon[32753]: pgmap v15470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:38.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:38.546+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:38.546+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:38.546+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:43:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:39 smithi194 ceph-mon[33017]: pgmap v15471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28357.133384 secs 2024-04-26T19:43:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:39 smithi033 ceph-mon[27845]: pgmap v15471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28357.133384 secs 2024-04-26T19:43:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:39 smithi096 ceph-mon[32753]: pgmap v15471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28357.133384 secs 2024-04-26T19:43:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28360.042159 secs 2024-04-26T19:43:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28360.042159 secs 2024-04-26T19:43:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28360.042159 secs 2024-04-26T19:43:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:41 smithi194 ceph-mon[33017]: pgmap v15472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:41 smithi033 ceph-mon[27845]: pgmap v15472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:41 smithi096 ceph-mon[32753]: pgmap v15472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:43 smithi194 ceph-mon[33017]: pgmap v15473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:43 smithi033 ceph-mon[27845]: pgmap v15473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:43 smithi096 ceph-mon[32753]: pgmap v15473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28362.133442 secs 2024-04-26T19:43:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28362.133442 secs 2024-04-26T19:43:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28362.133442 secs 2024-04-26T19:43:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:45 smithi194 ceph-mon[33017]: pgmap v15474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28365.042277 secs 2024-04-26T19:43:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:45 smithi033 ceph-mon[27845]: pgmap v15474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28365.042277 secs 2024-04-26T19:43:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:45 smithi096 ceph-mon[32753]: pgmap v15474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28365.042277 secs 2024-04-26T19:43:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:47 smithi194 ceph-mon[33017]: pgmap v15475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:47 smithi033 ceph-mon[27845]: pgmap v15475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:47 smithi096 ceph-mon[32753]: pgmap v15475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:49 smithi194 ceph-mon[33017]: pgmap v15476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28367.133500 secs 2024-04-26T19:43:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:49 smithi033 ceph-mon[27845]: pgmap v15476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28367.133500 secs 2024-04-26T19:43:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:49 smithi096 ceph-mon[32753]: pgmap v15476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28367.133500 secs 2024-04-26T19:43:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28370.042416 secs 2024-04-26T19:43:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28370.042416 secs 2024-04-26T19:43:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28370.042416 secs 2024-04-26T19:43:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:51 smithi194 ceph-mon[33017]: pgmap v15477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:51 smithi033 ceph-mon[27845]: pgmap v15477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:51 smithi096 ceph-mon[32753]: pgmap v15477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:52 smithi194 ceph-mon[33017]: pgmap v15478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:52 smithi033 ceph-mon[27845]: pgmap v15478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:52 smithi096 ceph-mon[32753]: pgmap v15478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:53.559+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:53.559+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:43:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:43:53.559+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:43:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28372.133557 secs 2024-04-26T19:43:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:54 smithi194 ceph-mon[33017]: pgmap v15479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28375.042534 secs 2024-04-26T19:43:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28372.133557 secs 2024-04-26T19:43:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:54 smithi033 ceph-mon[27845]: pgmap v15479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:54.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28375.042534 secs 2024-04-26T19:43:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28372.133557 secs 2024-04-26T19:43:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:54 smithi096 ceph-mon[32753]: pgmap v15479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28375.042534 secs 2024-04-26T19:43:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:56 smithi194 ceph-mon[33017]: pgmap v15480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:56 smithi033 ceph-mon[27845]: pgmap v15480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:56 smithi096 ceph-mon[32753]: pgmap v15480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:43:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:58 smithi194 ceph-mon[33017]: pgmap v15481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:58 smithi033 ceph-mon[27845]: pgmap v15481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:58 smithi096 ceph-mon[32753]: pgmap v15481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:43:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28377.133640 secs 2024-04-26T19:43:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28380.042612 secs 2024-04-26T19:43:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28377.133640 secs 2024-04-26T19:43:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28380.042612 secs 2024-04-26T19:43:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28377.133640 secs 2024-04-26T19:43:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28380.042612 secs 2024-04-26T19:44:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:00 smithi033 ceph-mon[27845]: pgmap v15482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:00 smithi096 ceph-mon[32753]: pgmap v15482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:00 smithi194 ceph-mon[33017]: pgmap v15482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:02 smithi033 ceph-mon[27845]: pgmap v15483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:02 smithi096 ceph-mon[32753]: pgmap v15483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:02 smithi194 ceph-mon[33017]: pgmap v15483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28382.133723 secs 2024-04-26T19:44:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:04 smithi033 ceph-mon[27845]: pgmap v15484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:44:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28385.042685 secs 2024-04-26T19:44:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28382.133723 secs 2024-04-26T19:44:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:04 smithi096 ceph-mon[32753]: pgmap v15484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:44:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28385.042685 secs 2024-04-26T19:44:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28382.133723 secs 2024-04-26T19:44:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:04 smithi194 ceph-mon[33017]: pgmap v15484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:44:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28385.042685 secs 2024-04-26T19:44:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:44:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:44:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:05 smithi033 ceph-mon[27845]: pgmap v15485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:44:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:05 smithi033 ceph-mon[27845]: pgmap v15486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:44:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:44:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:44:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:05 smithi096 ceph-mon[32753]: pgmap v15485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:44:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:05 smithi096 ceph-mon[32753]: pgmap v15486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:44:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:44:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:44:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:05 smithi194 ceph-mon[33017]: pgmap v15485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:44:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:05 smithi194 ceph-mon[33017]: pgmap v15486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:44:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:07 smithi194 ceph-mon[33017]: pgmap v15487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:07 smithi033 ceph-mon[27845]: pgmap v15487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:07 smithi096 ceph-mon[32753]: pgmap v15487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:08.574+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:08.574+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:44:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:08.574+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28387.133807 secs 2024-04-26T19:44:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28387.133807 secs 2024-04-26T19:44:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28387.133807 secs 2024-04-26T19:44:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28390.042800 secs 2024-04-26T19:44:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:10 smithi194 ceph-mon[33017]: pgmap v15488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28390.042800 secs 2024-04-26T19:44:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:10 smithi033 ceph-mon[27845]: pgmap v15488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28390.042800 secs 2024-04-26T19:44:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:10 smithi096 ceph-mon[32753]: pgmap v15488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:12 smithi194 ceph-mon[33017]: pgmap v15489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:12 smithi033 ceph-mon[27845]: pgmap v15489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:12 smithi096 ceph-mon[32753]: pgmap v15489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28392.133894 secs 2024-04-26T19:44:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:14 smithi194 ceph-mon[33017]: pgmap v15490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28392.133894 secs 2024-04-26T19:44:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:14 smithi033 ceph-mon[27845]: pgmap v15490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28392.133894 secs 2024-04-26T19:44:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:14 smithi096 ceph-mon[32753]: pgmap v15490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:44:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28395.042896 secs 2024-04-26T19:44:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28395.042896 secs 2024-04-26T19:44:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28395.042896 secs 2024-04-26T19:44:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:16 smithi194 ceph-mon[33017]: pgmap v15491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:44:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:16 smithi033 ceph-mon[27845]: pgmap v15491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:44:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:16 smithi096 ceph-mon[32753]: pgmap v15491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:44:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:18 smithi194 ceph-mon[33017]: pgmap v15492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:18 smithi033 ceph-mon[27845]: pgmap v15492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:18 smithi096 ceph-mon[32753]: pgmap v15492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28397.133994 secs 2024-04-26T19:44:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28397.133994 secs 2024-04-26T19:44:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28397.133994 secs 2024-04-26T19:44:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28400.043015 secs 2024-04-26T19:44:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:20 smithi194 ceph-mon[33017]: pgmap v15493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28400.043015 secs 2024-04-26T19:44:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:20 smithi033 ceph-mon[27845]: pgmap v15493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28400.043015 secs 2024-04-26T19:44:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:20 smithi096 ceph-mon[32753]: pgmap v15493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:22 smithi194 ceph-mon[33017]: pgmap v15494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:22 smithi033 ceph-mon[27845]: pgmap v15494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:22 smithi096 ceph-mon[32753]: pgmap v15494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:23.589+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:23.589+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:44:23.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:23.589+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28402.134075 secs 2024-04-26T19:44:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:24 smithi194 ceph-mon[33017]: pgmap v15495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28402.134075 secs 2024-04-26T19:44:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:24 smithi033 ceph-mon[27845]: pgmap v15495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28402.134075 secs 2024-04-26T19:44:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:24 smithi096 ceph-mon[32753]: pgmap v15495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28405.043132 secs 2024-04-26T19:44:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28405.043132 secs 2024-04-26T19:44:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28405.043132 secs 2024-04-26T19:44:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:26 smithi194 ceph-mon[33017]: pgmap v15496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:26 smithi033 ceph-mon[27845]: pgmap v15496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:26 smithi096 ceph-mon[32753]: pgmap v15496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:26.631+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:44:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:26.631+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:44:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:26.631+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 224 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:44:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:26.632+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 224 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:44:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:28 smithi194 ceph-mon[33017]: pgmap v15497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:28 smithi033 ceph-mon[27845]: pgmap v15497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:28 smithi096 ceph-mon[32753]: pgmap v15497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28407.134137 secs 2024-04-26T19:44:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28407.134137 secs 2024-04-26T19:44:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28407.134137 secs 2024-04-26T19:44:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28410.043248 secs 2024-04-26T19:44:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:30 smithi194 ceph-mon[33017]: pgmap v15498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28410.043248 secs 2024-04-26T19:44:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:30 smithi033 ceph-mon[27845]: pgmap v15498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28410.043248 secs 2024-04-26T19:44:30.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:30 smithi096 ceph-mon[32753]: pgmap v15498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:31 smithi033 ceph-mon[27845]: pgmap v15499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:31 smithi096 ceph-mon[32753]: pgmap v15499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:31 smithi194 ceph-mon[33017]: pgmap v15499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28412.134222 secs 2024-04-26T19:44:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:34 smithi194 ceph-mon[33017]: pgmap v15500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28412.134222 secs 2024-04-26T19:44:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:34 smithi033 ceph-mon[27845]: pgmap v15500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28412.134222 secs 2024-04-26T19:44:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:34 smithi096 ceph-mon[32753]: pgmap v15500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28415.043358 secs 2024-04-26T19:44:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28415.043358 secs 2024-04-26T19:44:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28415.043358 secs 2024-04-26T19:44:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:36 smithi194 ceph-mon[33017]: pgmap v15501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:36 smithi033 ceph-mon[27845]: pgmap v15501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:36 smithi096 ceph-mon[32753]: pgmap v15501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:38 smithi194 ceph-mon[33017]: pgmap v15502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:38 smithi033 ceph-mon[27845]: pgmap v15502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:38 smithi096 ceph-mon[32753]: pgmap v15502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:38.594+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:38.594+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:38.594+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:44:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28417.134308 secs 2024-04-26T19:44:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28417.134308 secs 2024-04-26T19:44:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28417.134308 secs 2024-04-26T19:44:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28420.043495 secs 2024-04-26T19:44:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:40 smithi194 ceph-mon[33017]: pgmap v15503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28420.043495 secs 2024-04-26T19:44:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:40 smithi033 ceph-mon[27845]: pgmap v15503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28420.043495 secs 2024-04-26T19:44:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:40 smithi096 ceph-mon[32753]: pgmap v15503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:42 smithi194 ceph-mon[33017]: pgmap v15504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:42 smithi033 ceph-mon[27845]: pgmap v15504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:42 smithi096 ceph-mon[32753]: pgmap v15504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:44.336 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28422.134390 secs 2024-04-26T19:44:44.336 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:44 smithi033 ceph-mon[27845]: pgmap v15505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:44.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28422.134390 secs 2024-04-26T19:44:44.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:44 smithi194 ceph-mon[33017]: pgmap v15505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28422.134390 secs 2024-04-26T19:44:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:44 smithi096 ceph-mon[32753]: pgmap v15505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28425.043643 secs 2024-04-26T19:44:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28425.043643 secs 2024-04-26T19:44:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28425.043643 secs 2024-04-26T19:44:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:46 smithi194 ceph-mon[33017]: pgmap v15506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:46 smithi033 ceph-mon[27845]: pgmap v15506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:46.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:46 smithi096 ceph-mon[32753]: pgmap v15506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:48 smithi194 ceph-mon[33017]: pgmap v15507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:48 smithi033 ceph-mon[27845]: pgmap v15507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:48 smithi096 ceph-mon[32753]: pgmap v15507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28427.134474 secs 2024-04-26T19:44:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28427.134474 secs 2024-04-26T19:44:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28427.134474 secs 2024-04-26T19:44:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28430.043776 secs 2024-04-26T19:44:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:50 smithi194 ceph-mon[33017]: pgmap v15508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28430.043776 secs 2024-04-26T19:44:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:50 smithi033 ceph-mon[27845]: pgmap v15508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28430.043776 secs 2024-04-26T19:44:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:50 smithi096 ceph-mon[32753]: pgmap v15508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:52 smithi194 ceph-mon[33017]: pgmap v15509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:52 smithi033 ceph-mon[27845]: pgmap v15509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:52 smithi096 ceph-mon[32753]: pgmap v15509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:53.605+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:53.605+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:44:53.605+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:44:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28432.134561 secs 2024-04-26T19:44:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:54 smithi194 ceph-mon[33017]: pgmap v15510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28432.134561 secs 2024-04-26T19:44:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:54 smithi033 ceph-mon[27845]: pgmap v15510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28432.134561 secs 2024-04-26T19:44:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:54 smithi096 ceph-mon[32753]: pgmap v15510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28435.043871 secs 2024-04-26T19:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28435.043871 secs 2024-04-26T19:44:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28435.043871 secs 2024-04-26T19:44:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:56 smithi194 ceph-mon[33017]: pgmap v15511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:56 smithi033 ceph-mon[27845]: pgmap v15511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:56 smithi096 ceph-mon[32753]: pgmap v15511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:44:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:58 smithi194 ceph-mon[33017]: pgmap v15512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:58 smithi033 ceph-mon[27845]: pgmap v15512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:58 smithi096 ceph-mon[32753]: pgmap v15512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:44:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:44:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28437.134619 secs 2024-04-26T19:44:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:44:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28437.134619 secs 2024-04-26T19:44:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:44:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28437.134619 secs 2024-04-26T19:45:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28440.044020 secs 2024-04-26T19:45:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:00 smithi033 ceph-mon[27845]: pgmap v15513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28440.044020 secs 2024-04-26T19:45:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:00 smithi096 ceph-mon[32753]: pgmap v15513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28440.044020 secs 2024-04-26T19:45:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:00 smithi194 ceph-mon[33017]: pgmap v15513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:01 smithi033 ceph-mon[27845]: pgmap v15514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:01 smithi096 ceph-mon[32753]: pgmap v15514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:01 smithi194 ceph-mon[33017]: pgmap v15514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28442.134699 secs 2024-04-26T19:45:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:04 smithi194 ceph-mon[33017]: pgmap v15515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28442.134699 secs 2024-04-26T19:45:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:04 smithi033 ceph-mon[27845]: pgmap v15515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28442.134699 secs 2024-04-26T19:45:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:04 smithi096 ceph-mon[32753]: pgmap v15515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28445.044143 secs 2024-04-26T19:45:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28445.044143 secs 2024-04-26T19:45:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28445.044143 secs 2024-04-26T19:45:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:06 smithi194 ceph-mon[33017]: pgmap v15516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:45:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:06 smithi033 ceph-mon[27845]: pgmap v15516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:45:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:06 smithi096 ceph-mon[32753]: pgmap v15516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:45:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:45:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:45:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:45:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:45:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:45:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:45:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:45:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:45:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:45:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:08 smithi194 ceph-mon[33017]: pgmap v15517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:08 smithi194 ceph-mon[33017]: pgmap v15518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:08 smithi033 ceph-mon[27845]: pgmap v15517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:08 smithi033 ceph-mon[27845]: pgmap v15518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:08 smithi096 ceph-mon[32753]: pgmap v15517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:08 smithi096 ceph-mon[32753]: pgmap v15518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:08.621+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:08.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:08.621+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:45:08.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:08.621+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28447.134782 secs 2024-04-26T19:45:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28447.134782 secs 2024-04-26T19:45:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28447.134782 secs 2024-04-26T19:45:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28450.044265 secs 2024-04-26T19:45:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:10 smithi194 ceph-mon[33017]: pgmap v15519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28450.044265 secs 2024-04-26T19:45:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:10 smithi033 ceph-mon[27845]: pgmap v15519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28450.044265 secs 2024-04-26T19:45:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:10 smithi096 ceph-mon[32753]: pgmap v15519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:12 smithi194 ceph-mon[33017]: pgmap v15520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:12 smithi033 ceph-mon[27845]: pgmap v15520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:12 smithi096 ceph-mon[32753]: pgmap v15520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:14 smithi194 ceph-mon[33017]: pgmap v15521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28452.134839 secs 2024-04-26T19:45:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:14 smithi033 ceph-mon[27845]: pgmap v15521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28452.134839 secs 2024-04-26T19:45:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:14 smithi096 ceph-mon[32753]: pgmap v15521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28452.134839 secs 2024-04-26T19:45:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28455.044353 secs 2024-04-26T19:45:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28455.044353 secs 2024-04-26T19:45:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28455.044353 secs 2024-04-26T19:45:16.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:16 smithi033 ceph-mon[27845]: pgmap v15522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:16 smithi194 ceph-mon[33017]: pgmap v15522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:16 smithi096 ceph-mon[32753]: pgmap v15522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:45:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:18 smithi194 ceph-mon[33017]: pgmap v15523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:45:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:18 smithi033 ceph-mon[27845]: pgmap v15523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:45:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:18 smithi096 ceph-mon[32753]: pgmap v15523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28457.134902 secs 2024-04-26T19:45:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28457.134902 secs 2024-04-26T19:45:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28457.134902 secs 2024-04-26T19:45:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28460.044464 secs 2024-04-26T19:45:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:20 smithi194 ceph-mon[33017]: pgmap v15524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28460.044464 secs 2024-04-26T19:45:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:20 smithi033 ceph-mon[27845]: pgmap v15524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28460.044464 secs 2024-04-26T19:45:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:20 smithi096 ceph-mon[32753]: pgmap v15524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:22 smithi033 ceph-mon[27845]: pgmap v15525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:22 smithi096 ceph-mon[32753]: pgmap v15525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:22 smithi194 ceph-mon[33017]: pgmap v15525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:23.627+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:23.627+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:23.627+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:45:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:24 smithi033 ceph-mon[27845]: pgmap v15526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28462.134962 secs 2024-04-26T19:45:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:24 smithi096 ceph-mon[32753]: pgmap v15526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28462.134962 secs 2024-04-26T19:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:24 smithi194 ceph-mon[33017]: pgmap v15526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28462.134962 secs 2024-04-26T19:45:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28465.044589 secs 2024-04-26T19:45:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:25 smithi033 ceph-mon[27845]: pgmap v15527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28465.044589 secs 2024-04-26T19:45:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:25 smithi096 ceph-mon[32753]: pgmap v15527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28465.044589 secs 2024-04-26T19:45:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:25 smithi194 ceph-mon[33017]: pgmap v15527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:27 smithi194 ceph-mon[33017]: pgmap v15528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:27 smithi033 ceph-mon[27845]: pgmap v15528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:27 smithi096 ceph-mon[32753]: pgmap v15528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28467.135050 secs 2024-04-26T19:45:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28467.135050 secs 2024-04-26T19:45:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28467.135050 secs 2024-04-26T19:45:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28470.044667 secs 2024-04-26T19:45:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:29 smithi033 ceph-mon[27845]: pgmap v15529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28470.044667 secs 2024-04-26T19:45:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:29 smithi096 ceph-mon[32753]: pgmap v15529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28470.044667 secs 2024-04-26T19:45:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:29 smithi194 ceph-mon[33017]: pgmap v15529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:31 smithi194 ceph-mon[33017]: pgmap v15530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:31 smithi033 ceph-mon[27845]: pgmap v15530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:31 smithi096 ceph-mon[32753]: pgmap v15530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:33 smithi194 ceph-mon[33017]: pgmap v15531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28472.135134 secs 2024-04-26T19:45:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:33 smithi033 ceph-mon[27845]: pgmap v15531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28472.135134 secs 2024-04-26T19:45:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:33 smithi096 ceph-mon[32753]: pgmap v15531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28472.135134 secs 2024-04-26T19:45:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28475.044782 secs 2024-04-26T19:45:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28475.044782 secs 2024-04-26T19:45:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28475.044782 secs 2024-04-26T19:45:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:35 smithi194 ceph-mon[33017]: pgmap v15532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:35 smithi033 ceph-mon[27845]: pgmap v15532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:35 smithi096 ceph-mon[32753]: pgmap v15532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:37 smithi194 ceph-mon[33017]: pgmap v15533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:37 smithi033 ceph-mon[27845]: pgmap v15533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:37 smithi096 ceph-mon[32753]: pgmap v15533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:38.632+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:38.632+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:38.632+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:45:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28477.135218 secs 2024-04-26T19:45:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28477.135218 secs 2024-04-26T19:45:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28477.135218 secs 2024-04-26T19:45:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28480.044898 secs 2024-04-26T19:45:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:39 smithi194 ceph-mon[33017]: pgmap v15534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28480.044898 secs 2024-04-26T19:45:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:39 smithi033 ceph-mon[27845]: pgmap v15534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28480.044898 secs 2024-04-26T19:45:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:39 smithi096 ceph-mon[32753]: pgmap v15534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:41 smithi194 ceph-mon[33017]: pgmap v15535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:41 smithi033 ceph-mon[27845]: pgmap v15535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:41 smithi096 ceph-mon[32753]: pgmap v15535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:43 smithi194 ceph-mon[33017]: pgmap v15536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28482.135280 secs 2024-04-26T19:45:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:43 smithi033 ceph-mon[27845]: pgmap v15536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28482.135280 secs 2024-04-26T19:45:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:43 smithi096 ceph-mon[32753]: pgmap v15536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28482.135280 secs 2024-04-26T19:45:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28485.045020 secs 2024-04-26T19:45:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28485.045020 secs 2024-04-26T19:45:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28485.045020 secs 2024-04-26T19:45:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:45 smithi194 ceph-mon[33017]: pgmap v15537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:45 smithi033 ceph-mon[27845]: pgmap v15537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:45 smithi096 ceph-mon[32753]: pgmap v15537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:47 smithi194 ceph-mon[33017]: pgmap v15538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:47 smithi033 ceph-mon[27845]: pgmap v15538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:47 smithi096 ceph-mon[32753]: pgmap v15538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28487.135370 secs 2024-04-26T19:45:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28487.135370 secs 2024-04-26T19:45:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28487.135370 secs 2024-04-26T19:45:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28490.045134 secs 2024-04-26T19:45:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:49 smithi194 ceph-mon[33017]: pgmap v15539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28490.045134 secs 2024-04-26T19:45:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:49 smithi033 ceph-mon[27845]: pgmap v15539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28490.045134 secs 2024-04-26T19:45:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:49 smithi096 ceph-mon[32753]: pgmap v15539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:51 smithi194 ceph-mon[33017]: pgmap v15540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:51 smithi033 ceph-mon[27845]: pgmap v15540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:51 smithi096 ceph-mon[32753]: pgmap v15540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:45:53.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:53.646+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:53.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:53.646+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:45:53.647 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:45:53.646+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:45:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:53 smithi194 ceph-mon[33017]: pgmap v15541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28492.135454 secs 2024-04-26T19:45:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:53 smithi033 ceph-mon[27845]: pgmap v15541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28492.135454 secs 2024-04-26T19:45:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:53 smithi096 ceph-mon[32753]: pgmap v15541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:45:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28492.135454 secs 2024-04-26T19:45:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28495.045250 secs 2024-04-26T19:45:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28495.045250 secs 2024-04-26T19:45:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28495.045250 secs 2024-04-26T19:45:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:55 smithi194 ceph-mon[33017]: pgmap v15542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:45:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:55 smithi033 ceph-mon[27845]: pgmap v15542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:45:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:55 smithi096 ceph-mon[32753]: pgmap v15542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:45:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:57 smithi194 ceph-mon[33017]: pgmap v15543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:45:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:57 smithi033 ceph-mon[27845]: pgmap v15543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:45:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:57 smithi096 ceph-mon[32753]: pgmap v15543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:45:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28497.135538 secs 2024-04-26T19:45:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28497.135538 secs 2024-04-26T19:45:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28497.135538 secs 2024-04-26T19:46:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28500.045372 secs 2024-04-26T19:46:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:45:59 smithi033 ceph-mon[27845]: pgmap v15544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:46:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28500.045372 secs 2024-04-26T19:46:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:45:59 smithi096 ceph-mon[32753]: pgmap v15544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:46:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28500.045372 secs 2024-04-26T19:46:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:45:59 smithi194 ceph-mon[33017]: pgmap v15544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:46:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:01 smithi033 ceph-mon[27845]: pgmap v15545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:46:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:01 smithi096 ceph-mon[32753]: pgmap v15545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:46:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:01 smithi194 ceph-mon[33017]: pgmap v15545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:46:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:03 smithi033 ceph-mon[27845]: pgmap v15546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:46:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28502.135623 secs 2024-04-26T19:46:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:03 smithi096 ceph-mon[32753]: pgmap v15546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:46:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28502.135623 secs 2024-04-26T19:46:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:03 smithi194 ceph-mon[33017]: pgmap v15546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:46:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28502.135623 secs 2024-04-26T19:46:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28505.045448 secs 2024-04-26T19:46:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28505.045448 secs 2024-04-26T19:46:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28505.045448 secs 2024-04-26T19:46:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:05 smithi033 ceph-mon[27845]: pgmap v15547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:46:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:05 smithi096 ceph-mon[32753]: pgmap v15547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:46:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:05 smithi194 ceph-mon[33017]: pgmap v15547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:46:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:07 smithi033 ceph-mon[27845]: pgmap v15548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:46:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:07 smithi096 ceph-mon[32753]: pgmap v15548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:46:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:07 smithi194 ceph-mon[33017]: pgmap v15548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:46:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:08.659+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:08.659+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:08.659+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:46:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28507.135707 secs 2024-04-26T19:46:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:46:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:46:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:46:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28507.135707 secs 2024-04-26T19:46:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:46:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:46:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:46:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28507.135707 secs 2024-04-26T19:46:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:46:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:46:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:46:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:09 smithi033 ceph-mon[27845]: pgmap v15549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:09 smithi033 ceph-mon[27845]: pgmap v15550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28510.045565 secs 2024-04-26T19:46:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:09 smithi096 ceph-mon[32753]: pgmap v15549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:09 smithi096 ceph-mon[32753]: pgmap v15550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28510.045565 secs 2024-04-26T19:46:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:09 smithi194 ceph-mon[33017]: pgmap v15549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:09 smithi194 ceph-mon[33017]: pgmap v15550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28510.045565 secs 2024-04-26T19:46:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:11 smithi033 ceph-mon[27845]: pgmap v15551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:11 smithi096 ceph-mon[32753]: pgmap v15551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:11 smithi194 ceph-mon[33017]: pgmap v15551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:13 smithi033 ceph-mon[27845]: pgmap v15552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28512.135790 secs 2024-04-26T19:46:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:13 smithi096 ceph-mon[32753]: pgmap v15552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28512.135790 secs 2024-04-26T19:46:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:13 smithi194 ceph-mon[33017]: pgmap v15552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28512.135790 secs 2024-04-26T19:46:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28515.045688 secs 2024-04-26T19:46:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28515.045688 secs 2024-04-26T19:46:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28515.045688 secs 2024-04-26T19:46:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:15 smithi033 ceph-mon[27845]: pgmap v15553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:46:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:15 smithi096 ceph-mon[32753]: pgmap v15553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:46:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:15 smithi194 ceph-mon[33017]: pgmap v15553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:46:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:17 smithi096 ceph-mon[32753]: pgmap v15554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:17 smithi194 ceph-mon[33017]: pgmap v15554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:17 smithi033 ceph-mon[27845]: pgmap v15554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:19.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28517.135878 secs 2024-04-26T19:46:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28517.135878 secs 2024-04-26T19:46:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28517.135878 secs 2024-04-26T19:46:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:19 smithi194 ceph-mon[33017]: pgmap v15555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:46:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28520.045805 secs 2024-04-26T19:46:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:19 smithi033 ceph-mon[27845]: pgmap v15555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:46:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28520.045805 secs 2024-04-26T19:46:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:19 smithi096 ceph-mon[32753]: pgmap v15555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:46:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28520.045805 secs 2024-04-26T19:46:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:21 smithi194 ceph-mon[33017]: pgmap v15556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:21 smithi033 ceph-mon[27845]: pgmap v15556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:21 smithi096 ceph-mon[32753]: pgmap v15556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:23.671+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:23.671+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:23.671+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:46:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:23 smithi194 ceph-mon[33017]: pgmap v15557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28522.135934 secs 2024-04-26T19:46:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:23 smithi033 ceph-mon[27845]: pgmap v15557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28522.135934 secs 2024-04-26T19:46:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:23 smithi096 ceph-mon[32753]: pgmap v15557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28522.135934 secs 2024-04-26T19:46:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28525.045921 secs 2024-04-26T19:46:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28525.045921 secs 2024-04-26T19:46:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28525.045921 secs 2024-04-26T19:46:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:25 smithi194 ceph-mon[33017]: pgmap v15558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:25 smithi033 ceph-mon[27845]: pgmap v15558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:25 smithi096 ceph-mon[32753]: pgmap v15558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:27 smithi194 ceph-mon[33017]: pgmap v15559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:27 smithi033 ceph-mon[27845]: pgmap v15559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:27 smithi096 ceph-mon[32753]: pgmap v15559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:29.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28527.136018 secs 2024-04-26T19:46:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28527.136018 secs 2024-04-26T19:46:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28527.136018 secs 2024-04-26T19:46:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:29 smithi033 ceph-mon[27845]: pgmap v15560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28530.046049 secs 2024-04-26T19:46:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:29 smithi096 ceph-mon[32753]: pgmap v15560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28530.046049 secs 2024-04-26T19:46:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:29 smithi194 ceph-mon[33017]: pgmap v15560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28530.046049 secs 2024-04-26T19:46:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:31 smithi194 ceph-mon[33017]: pgmap v15561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:31 smithi033 ceph-mon[27845]: pgmap v15561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:31 smithi096 ceph-mon[32753]: pgmap v15561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:33 smithi033 ceph-mon[27845]: pgmap v15562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28532.136100 secs 2024-04-26T19:46:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:33 smithi194 ceph-mon[33017]: pgmap v15562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28532.136100 secs 2024-04-26T19:46:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:33 smithi096 ceph-mon[32753]: pgmap v15562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28532.136100 secs 2024-04-26T19:46:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28535.046169 secs 2024-04-26T19:46:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28535.046169 secs 2024-04-26T19:46:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28535.046169 secs 2024-04-26T19:46:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:36 smithi194 ceph-mon[33017]: pgmap v15563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:36 smithi033 ceph-mon[27845]: pgmap v15563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:36 smithi096 ceph-mon[32753]: pgmap v15563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:38 smithi194 ceph-mon[33017]: pgmap v15564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:38 smithi033 ceph-mon[27845]: pgmap v15564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:38 smithi096 ceph-mon[32753]: pgmap v15564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:38.684+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:38.684+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:38.685+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:46:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28537.136185 secs 2024-04-26T19:46:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28537.136185 secs 2024-04-26T19:46:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28537.136185 secs 2024-04-26T19:46:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:40 smithi194 ceph-mon[33017]: pgmap v15565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28540.046286 secs 2024-04-26T19:46:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:40 smithi033 ceph-mon[27845]: pgmap v15565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28540.046286 secs 2024-04-26T19:46:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:40 smithi096 ceph-mon[32753]: pgmap v15565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28540.046286 secs 2024-04-26T19:46:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:42 smithi194 ceph-mon[33017]: pgmap v15566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:42 smithi033 ceph-mon[27845]: pgmap v15566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:42 smithi096 ceph-mon[32753]: pgmap v15566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:44 smithi194 ceph-mon[33017]: pgmap v15567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28542.136244 secs 2024-04-26T19:46:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:44 smithi033 ceph-mon[27845]: pgmap v15567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28542.136244 secs 2024-04-26T19:46:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:44 smithi096 ceph-mon[32753]: pgmap v15567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28542.136244 secs 2024-04-26T19:46:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28545.046408 secs 2024-04-26T19:46:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28545.046408 secs 2024-04-26T19:46:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28545.046408 secs 2024-04-26T19:46:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:46 smithi194 ceph-mon[33017]: pgmap v15568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:46 smithi033 ceph-mon[27845]: pgmap v15568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:46 smithi096 ceph-mon[32753]: pgmap v15568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:48 smithi194 ceph-mon[33017]: pgmap v15569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:48 smithi033 ceph-mon[27845]: pgmap v15569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:48 smithi096 ceph-mon[32753]: pgmap v15569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:49.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28547.136307 secs 2024-04-26T19:46:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28547.136307 secs 2024-04-26T19:46:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28547.136307 secs 2024-04-26T19:46:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:50 smithi194 ceph-mon[33017]: pgmap v15570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28550.046525 secs 2024-04-26T19:46:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:50 smithi033 ceph-mon[27845]: pgmap v15570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28550.046525 secs 2024-04-26T19:46:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:50 smithi096 ceph-mon[32753]: pgmap v15570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28550.046525 secs 2024-04-26T19:46:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:52 smithi194 ceph-mon[33017]: pgmap v15571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:52 smithi033 ceph-mon[27845]: pgmap v15571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:52 smithi096 ceph-mon[32753]: pgmap v15571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:46:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:53.695+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:53.695+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:46:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:46:53.695+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:46:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:54 smithi194 ceph-mon[33017]: pgmap v15572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28552.136363 secs 2024-04-26T19:46:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:54 smithi033 ceph-mon[27845]: pgmap v15572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28552.136363 secs 2024-04-26T19:46:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:54 smithi096 ceph-mon[32753]: pgmap v15572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:46:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28552.136363 secs 2024-04-26T19:46:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28555.046646 secs 2024-04-26T19:46:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28555.046646 secs 2024-04-26T19:46:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28555.046646 secs 2024-04-26T19:46:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:56 smithi194 ceph-mon[33017]: pgmap v15573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:56 smithi033 ceph-mon[27845]: pgmap v15573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:56 smithi096 ceph-mon[32753]: pgmap v15573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:46:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:58 smithi194 ceph-mon[33017]: pgmap v15574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:58 smithi033 ceph-mon[27845]: pgmap v15574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:58 smithi096 ceph-mon[32753]: pgmap v15574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:46:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:46:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28557.136421 secs 2024-04-26T19:46:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:46:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28557.136421 secs 2024-04-26T19:46:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:46:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28557.136421 secs 2024-04-26T19:47:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:00 smithi033 ceph-mon[27845]: pgmap v15575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28560.046766 secs 2024-04-26T19:47:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:00 smithi096 ceph-mon[32753]: pgmap v15575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28560.046766 secs 2024-04-26T19:47:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:00 smithi194 ceph-mon[33017]: pgmap v15575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28560.046766 secs 2024-04-26T19:47:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:02 smithi033 ceph-mon[27845]: pgmap v15576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:02 smithi096 ceph-mon[32753]: pgmap v15576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:02 smithi194 ceph-mon[33017]: pgmap v15576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:04 smithi033 ceph-mon[27845]: pgmap v15577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28562.136478 secs 2024-04-26T19:47:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:04 smithi096 ceph-mon[32753]: pgmap v15577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28562.136478 secs 2024-04-26T19:47:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:04 smithi194 ceph-mon[33017]: pgmap v15577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28562.136478 secs 2024-04-26T19:47:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28565.046883 secs 2024-04-26T19:47:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28565.046883 secs 2024-04-26T19:47:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28565.046883 secs 2024-04-26T19:47:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:06 smithi033 ceph-mon[27845]: pgmap v15578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:06 smithi096 ceph-mon[32753]: pgmap v15578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:06 smithi194 ceph-mon[33017]: pgmap v15578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:07 smithi033 ceph-mon[27845]: pgmap v15579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:47:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:07 smithi096 ceph-mon[32753]: pgmap v15579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:47:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:07 smithi194 ceph-mon[33017]: pgmap v15579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:47:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:08.712+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:08.712+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:08.712+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:47:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28567.136539 secs 2024-04-26T19:47:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:09 smithi194 ceph-mon[33017]: pgmap v15580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28570.047005 secs 2024-04-26T19:47:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:47:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28567.136539 secs 2024-04-26T19:47:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:09 smithi033 ceph-mon[27845]: pgmap v15580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28570.047005 secs 2024-04-26T19:47:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:47:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28567.136539 secs 2024-04-26T19:47:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:09 smithi096 ceph-mon[32753]: pgmap v15580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28570.047005 secs 2024-04-26T19:47:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:47:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:47:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:47:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:10 smithi194 ceph-mon[33017]: pgmap v15581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:47:10.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:10 smithi194 ceph-mon[33017]: pgmap v15582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:10.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:47:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:47:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:47:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:10 smithi033 ceph-mon[27845]: pgmap v15581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:47:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:10 smithi033 ceph-mon[27845]: pgmap v15582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:47:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:47:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:47:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:10 smithi096 ceph-mon[32753]: pgmap v15581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T19:47:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:10 smithi096 ceph-mon[32753]: pgmap v15582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:47:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:12 smithi194 ceph-mon[33017]: pgmap v15583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:12 smithi033 ceph-mon[27845]: pgmap v15583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:12 smithi096 ceph-mon[32753]: pgmap v15583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28572.136621 secs 2024-04-26T19:47:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28572.136621 secs 2024-04-26T19:47:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28572.136621 secs 2024-04-26T19:47:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28575.047151 secs 2024-04-26T19:47:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:14 smithi194 ceph-mon[33017]: pgmap v15584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:47:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28575.047151 secs 2024-04-26T19:47:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:14 smithi033 ceph-mon[27845]: pgmap v15584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:47:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28575.047151 secs 2024-04-26T19:47:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:14 smithi096 ceph-mon[32753]: pgmap v15584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:47:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:16 smithi194 ceph-mon[33017]: pgmap v15585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:16 smithi033 ceph-mon[27845]: pgmap v15585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:16 smithi096 ceph-mon[32753]: pgmap v15585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:18 smithi194 ceph-mon[33017]: pgmap v15586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:18 smithi033 ceph-mon[27845]: pgmap v15586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:18 smithi096 ceph-mon[32753]: pgmap v15586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:47:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28577.136704 secs 2024-04-26T19:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28577.136704 secs 2024-04-26T19:47:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28577.136704 secs 2024-04-26T19:47:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28580.047191 secs 2024-04-26T19:47:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:20 smithi194 ceph-mon[33017]: pgmap v15587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:47:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28580.047191 secs 2024-04-26T19:47:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:20 smithi033 ceph-mon[27845]: pgmap v15587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:47:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28580.047191 secs 2024-04-26T19:47:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:20 smithi096 ceph-mon[32753]: pgmap v15587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:47:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:22 smithi194 ceph-mon[33017]: pgmap v15588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:22 smithi033 ceph-mon[27845]: pgmap v15588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:22 smithi096 ceph-mon[32753]: pgmap v15588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:23.727+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:23.727+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:23.727+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:47:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28582.136786 secs 2024-04-26T19:47:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28582.136786 secs 2024-04-26T19:47:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28582.136786 secs 2024-04-26T19:47:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28585.047273 secs 2024-04-26T19:47:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:24 smithi194 ceph-mon[33017]: pgmap v15589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28585.047273 secs 2024-04-26T19:47:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:24 smithi033 ceph-mon[27845]: pgmap v15589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28585.047273 secs 2024-04-26T19:47:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:24 smithi096 ceph-mon[32753]: pgmap v15589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:26 smithi194 ceph-mon[33017]: pgmap v15590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:26.662+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:47:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:26.662+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:47:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:26.662+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 225 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:47:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:26.663+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 225 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:47:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:26 smithi033 ceph-mon[27845]: pgmap v15590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:26 smithi096 ceph-mon[32753]: pgmap v15590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:28 smithi194 ceph-mon[33017]: pgmap v15591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:28 smithi033 ceph-mon[27845]: pgmap v15591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:28 smithi096 ceph-mon[32753]: pgmap v15591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:29.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28587.136870 secs 2024-04-26T19:47:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28587.136870 secs 2024-04-26T19:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28587.136870 secs 2024-04-26T19:47:30.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28590.047334 secs 2024-04-26T19:47:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:30 smithi033 ceph-mon[27845]: pgmap v15592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28590.047334 secs 2024-04-26T19:47:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:30 smithi096 ceph-mon[32753]: pgmap v15592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28590.047334 secs 2024-04-26T19:47:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:30 smithi194 ceph-mon[33017]: pgmap v15592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:32 smithi194 ceph-mon[33017]: pgmap v15593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:32 smithi033 ceph-mon[27845]: pgmap v15593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:32 smithi096 ceph-mon[32753]: pgmap v15593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28592.136951 secs 2024-04-26T19:47:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28592.136951 secs 2024-04-26T19:47:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28592.136951 secs 2024-04-26T19:47:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28595.047455 secs 2024-04-26T19:47:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:34 smithi033 ceph-mon[27845]: pgmap v15594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28595.047455 secs 2024-04-26T19:47:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:34 smithi096 ceph-mon[32753]: pgmap v15594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28595.047455 secs 2024-04-26T19:47:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:34 smithi194 ceph-mon[33017]: pgmap v15594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:36 smithi194 ceph-mon[33017]: pgmap v15595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:36 smithi033 ceph-mon[27845]: pgmap v15595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:36 smithi096 ceph-mon[32753]: pgmap v15595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:38.733+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:38.733+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:38.733+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:47:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28597.137008 secs 2024-04-26T19:47:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:38 smithi033 ceph-mon[27845]: pgmap v15596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28597.137008 secs 2024-04-26T19:47:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:38 smithi096 ceph-mon[32753]: pgmap v15596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28597.137008 secs 2024-04-26T19:47:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:38 smithi194 ceph-mon[33017]: pgmap v15596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28600.047575 secs 2024-04-26T19:47:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28600.047575 secs 2024-04-26T19:47:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28600.047575 secs 2024-04-26T19:47:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:40 smithi033 ceph-mon[27845]: pgmap v15597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:40 smithi096 ceph-mon[32753]: pgmap v15597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:40 smithi194 ceph-mon[33017]: pgmap v15597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:42 smithi194 ceph-mon[33017]: pgmap v15598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:42 smithi033 ceph-mon[27845]: pgmap v15598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:42 smithi096 ceph-mon[32753]: pgmap v15598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28602.137093 secs 2024-04-26T19:47:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28602.137093 secs 2024-04-26T19:47:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28602.137093 secs 2024-04-26T19:47:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28605.047696 secs 2024-04-26T19:47:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:44 smithi033 ceph-mon[27845]: pgmap v15599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28605.047696 secs 2024-04-26T19:47:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:44 smithi096 ceph-mon[32753]: pgmap v15599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28605.047696 secs 2024-04-26T19:47:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:44 smithi194 ceph-mon[33017]: pgmap v15599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:46 smithi194 ceph-mon[33017]: pgmap v15600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:46 smithi033 ceph-mon[27845]: pgmap v15600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:46 smithi096 ceph-mon[32753]: pgmap v15600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:48 smithi194 ceph-mon[33017]: pgmap v15601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:48 smithi033 ceph-mon[27845]: pgmap v15601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:48 smithi096 ceph-mon[32753]: pgmap v15601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28607.137176 secs 2024-04-26T19:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28607.137176 secs 2024-04-26T19:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28607.137176 secs 2024-04-26T19:47:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28610.047829 secs 2024-04-26T19:47:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:50 smithi194 ceph-mon[33017]: pgmap v15602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28610.047829 secs 2024-04-26T19:47:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:50 smithi033 ceph-mon[27845]: pgmap v15602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28610.047829 secs 2024-04-26T19:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:50 smithi096 ceph-mon[32753]: pgmap v15602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:52 smithi194 ceph-mon[33017]: pgmap v15603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:52 smithi033 ceph-mon[27845]: pgmap v15603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:52 smithi096 ceph-mon[32753]: pgmap v15603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:53.748+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:53.748+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:47:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:47:53.748+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:47:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28612.137279 secs 2024-04-26T19:47:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28612.137279 secs 2024-04-26T19:47:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28612.137279 secs 2024-04-26T19:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28615.047967 secs 2024-04-26T19:47:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:54 smithi033 ceph-mon[27845]: pgmap v15604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28615.047967 secs 2024-04-26T19:47:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:54 smithi096 ceph-mon[32753]: pgmap v15604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28615.047967 secs 2024-04-26T19:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:54 smithi194 ceph-mon[33017]: pgmap v15604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:56 smithi194 ceph-mon[33017]: pgmap v15605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:56 smithi033 ceph-mon[27845]: pgmap v15605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:56 smithi096 ceph-mon[32753]: pgmap v15605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:47:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:58 smithi194 ceph-mon[33017]: pgmap v15606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:58 smithi033 ceph-mon[27845]: pgmap v15606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:58 smithi096 ceph-mon[32753]: pgmap v15606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:47:59.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:47:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28617.137363 secs 2024-04-26T19:47:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:47:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28617.137363 secs 2024-04-26T19:47:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:47:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28617.137363 secs 2024-04-26T19:48:00.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28620.048097 secs 2024-04-26T19:48:00.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:00 smithi033 ceph-mon[27845]: pgmap v15607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28620.048097 secs 2024-04-26T19:48:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:00 smithi096 ceph-mon[32753]: pgmap v15607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28620.048097 secs 2024-04-26T19:48:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:00 smithi194 ceph-mon[33017]: pgmap v15607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:02 smithi194 ceph-mon[33017]: pgmap v15608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:02 smithi033 ceph-mon[27845]: pgmap v15608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:02 smithi096 ceph-mon[32753]: pgmap v15608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28622.137446 secs 2024-04-26T19:48:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28622.137446 secs 2024-04-26T19:48:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28622.137446 secs 2024-04-26T19:48:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28625.048148 secs 2024-04-26T19:48:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:04 smithi194 ceph-mon[33017]: pgmap v15609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28625.048148 secs 2024-04-26T19:48:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:04 smithi033 ceph-mon[27845]: pgmap v15609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28625.048148 secs 2024-04-26T19:48:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:04 smithi096 ceph-mon[32753]: pgmap v15609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:06 smithi194 ceph-mon[33017]: pgmap v15610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:06 smithi033 ceph-mon[27845]: pgmap v15610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:06 smithi096 ceph-mon[32753]: pgmap v15610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:08 smithi194 ceph-mon[33017]: pgmap v15611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:08 smithi033 ceph-mon[27845]: pgmap v15611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:08 smithi096 ceph-mon[32753]: pgmap v15611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:08.764+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:08.764+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:08.764+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:48:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28627.137530 secs 2024-04-26T19:48:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28627.137530 secs 2024-04-26T19:48:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28627.137530 secs 2024-04-26T19:48:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28630.048213 secs 2024-04-26T19:48:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:10 smithi194 ceph-mon[33017]: pgmap v15612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28630.048213 secs 2024-04-26T19:48:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:10 smithi033 ceph-mon[27845]: pgmap v15612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28630.048213 secs 2024-04-26T19:48:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:10 smithi096 ceph-mon[32753]: pgmap v15612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:48:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:11.669+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:48:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:11.669+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:48:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:11.670+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 226 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:48:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:11.670+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 226 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:48:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:48:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:48:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:12 smithi194 ceph-mon[33017]: pgmap v15613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:12 smithi033 ceph-mon[27845]: pgmap v15613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:12 smithi096 ceph-mon[32753]: pgmap v15613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:48:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:48:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:13 smithi194 ceph-mon[33017]: pgmap v15614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:48:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:13 smithi194 ceph-mon[33017]: pgmap v15615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:48:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:48:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:13 smithi033 ceph-mon[27845]: pgmap v15614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:48:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:13 smithi033 ceph-mon[27845]: pgmap v15615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:13 smithi096 ceph-mon[32753]: pgmap v15614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:13 smithi096 ceph-mon[32753]: pgmap v15615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:48:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28632.137612 secs 2024-04-26T19:48:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28632.137612 secs 2024-04-26T19:48:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28632.137612 secs 2024-04-26T19:48:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28635.048294 secs 2024-04-26T19:48:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:15 smithi033 ceph-mon[27845]: pgmap v15616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28635.048294 secs 2024-04-26T19:48:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:15 smithi096 ceph-mon[32753]: pgmap v15616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28635.048294 secs 2024-04-26T19:48:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:15 smithi194 ceph-mon[33017]: pgmap v15616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:17 smithi194 ceph-mon[33017]: pgmap v15617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:17 smithi033 ceph-mon[27845]: pgmap v15617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:17 smithi096 ceph-mon[32753]: pgmap v15617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28637.137700 secs 2024-04-26T19:48:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28637.137700 secs 2024-04-26T19:48:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28637.137700 secs 2024-04-26T19:48:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28640.048365 secs 2024-04-26T19:48:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:19 smithi194 ceph-mon[33017]: pgmap v15618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:48:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28640.048365 secs 2024-04-26T19:48:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:19 smithi033 ceph-mon[27845]: pgmap v15618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:48:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28640.048365 secs 2024-04-26T19:48:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:19 smithi096 ceph-mon[32753]: pgmap v15618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:48:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:21 smithi194 ceph-mon[33017]: pgmap v15619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:21 smithi033 ceph-mon[27845]: pgmap v15619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:21 smithi096 ceph-mon[32753]: pgmap v15619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:48:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:23.773+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:23.774+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:23.774+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:48:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:23 smithi194 ceph-mon[33017]: pgmap v15620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:48:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28642.137775 secs 2024-04-26T19:48:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:23 smithi033 ceph-mon[27845]: pgmap v15620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:48:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28642.137775 secs 2024-04-26T19:48:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:23 smithi096 ceph-mon[32753]: pgmap v15620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:48:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28642.137775 secs 2024-04-26T19:48:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28645.048489 secs 2024-04-26T19:48:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28645.048489 secs 2024-04-26T19:48:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28645.048489 secs 2024-04-26T19:48:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:25 smithi194 ceph-mon[33017]: pgmap v15621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:25 smithi033 ceph-mon[27845]: pgmap v15621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:25 smithi096 ceph-mon[32753]: pgmap v15621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:27 smithi194 ceph-mon[33017]: pgmap v15622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:27 smithi033 ceph-mon[27845]: pgmap v15622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:27 smithi096 ceph-mon[32753]: pgmap v15622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28647.137859 secs 2024-04-26T19:48:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28647.137859 secs 2024-04-26T19:48:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28647.137859 secs 2024-04-26T19:48:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28650.048605 secs 2024-04-26T19:48:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:29 smithi096 ceph-mon[32753]: pgmap v15623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28650.048605 secs 2024-04-26T19:48:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:29 smithi194 ceph-mon[33017]: pgmap v15623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28650.048605 secs 2024-04-26T19:48:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:29 smithi033 ceph-mon[27845]: pgmap v15623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:31 smithi194 ceph-mon[33017]: pgmap v15624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:31 smithi033 ceph-mon[27845]: pgmap v15624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:31 smithi096 ceph-mon[32753]: pgmap v15624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:33 smithi194 ceph-mon[33017]: pgmap v15625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28652.137942 secs 2024-04-26T19:48:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:33 smithi033 ceph-mon[27845]: pgmap v15625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28652.137942 secs 2024-04-26T19:48:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:33 smithi096 ceph-mon[32753]: pgmap v15625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28652.137942 secs 2024-04-26T19:48:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28655.048741 secs 2024-04-26T19:48:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28655.048741 secs 2024-04-26T19:48:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28655.048741 secs 2024-04-26T19:48:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:35 smithi033 ceph-mon[27845]: pgmap v15626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:35 smithi096 ceph-mon[32753]: pgmap v15626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:35 smithi194 ceph-mon[33017]: pgmap v15626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:37 smithi033 ceph-mon[27845]: pgmap v15627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:37 smithi096 ceph-mon[32753]: pgmap v15627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:37 smithi194 ceph-mon[33017]: pgmap v15627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:38.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:38.789+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:38.789+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:48:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28657.138034 secs 2024-04-26T19:48:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28657.138034 secs 2024-04-26T19:48:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28657.138034 secs 2024-04-26T19:48:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28660.048813 secs 2024-04-26T19:48:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:39 smithi033 ceph-mon[27845]: pgmap v15628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28660.048813 secs 2024-04-26T19:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:39 smithi096 ceph-mon[32753]: pgmap v15628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28660.048813 secs 2024-04-26T19:48:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:39 smithi194 ceph-mon[33017]: pgmap v15628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:41 smithi033 ceph-mon[27845]: pgmap v15629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:41 smithi096 ceph-mon[32753]: pgmap v15629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:41 smithi194 ceph-mon[33017]: pgmap v15629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:43 smithi033 ceph-mon[27845]: pgmap v15630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28662.138115 secs 2024-04-26T19:48:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:43 smithi096 ceph-mon[32753]: pgmap v15630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28662.138115 secs 2024-04-26T19:48:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:43 smithi194 ceph-mon[33017]: pgmap v15630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28662.138115 secs 2024-04-26T19:48:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28665.048880 secs 2024-04-26T19:48:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28665.048880 secs 2024-04-26T19:48:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28665.048880 secs 2024-04-26T19:48:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:45 smithi033 ceph-mon[27845]: pgmap v15631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:45 smithi096 ceph-mon[32753]: pgmap v15631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:45 smithi194 ceph-mon[33017]: pgmap v15631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:47 smithi033 ceph-mon[27845]: pgmap v15632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:47 smithi096 ceph-mon[32753]: pgmap v15632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:47 smithi194 ceph-mon[33017]: pgmap v15632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28667.138197 secs 2024-04-26T19:48:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28667.138197 secs 2024-04-26T19:48:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28667.138197 secs 2024-04-26T19:48:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28670.049015 secs 2024-04-26T19:48:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:49 smithi033 ceph-mon[27845]: pgmap v15633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28670.049015 secs 2024-04-26T19:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:49 smithi096 ceph-mon[32753]: pgmap v15633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28670.049015 secs 2024-04-26T19:48:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:49 smithi194 ceph-mon[33017]: pgmap v15633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:51 smithi033 ceph-mon[27845]: pgmap v15634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:51 smithi096 ceph-mon[32753]: pgmap v15634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:51 smithi194 ceph-mon[33017]: pgmap v15634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:53.793+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:53.793+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:48:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:48:53.793+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:48:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:53 smithi096 ceph-mon[32753]: pgmap v15635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28672.138276 secs 2024-04-26T19:48:54.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:53 smithi033 ceph-mon[27845]: pgmap v15635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:54.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28672.138276 secs 2024-04-26T19:48:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:53 smithi194 ceph-mon[33017]: pgmap v15635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28672.138276 secs 2024-04-26T19:48:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28675.049165 secs 2024-04-26T19:48:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28675.049165 secs 2024-04-26T19:48:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28675.049165 secs 2024-04-26T19:48:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:55 smithi194 ceph-mon[33017]: pgmap v15636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:55 smithi033 ceph-mon[27845]: pgmap v15636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:55 smithi096 ceph-mon[32753]: pgmap v15636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:48:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:57 smithi194 ceph-mon[33017]: pgmap v15637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:57 smithi033 ceph-mon[27845]: pgmap v15637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:57 smithi096 ceph-mon[32753]: pgmap v15637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:48:59.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28677.138336 secs 2024-04-26T19:48:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28677.138336 secs 2024-04-26T19:48:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28677.138336 secs 2024-04-26T19:49:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28680.049256 secs 2024-04-26T19:49:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:48:59 smithi033 ceph-mon[27845]: pgmap v15638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28680.049256 secs 2024-04-26T19:49:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:48:59 smithi096 ceph-mon[32753]: pgmap v15638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28680.049256 secs 2024-04-26T19:49:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:48:59 smithi194 ceph-mon[33017]: pgmap v15638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:01 smithi194 ceph-mon[33017]: pgmap v15639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:01 smithi033 ceph-mon[27845]: pgmap v15639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:01 smithi096 ceph-mon[32753]: pgmap v15639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:03 smithi194 ceph-mon[33017]: pgmap v15640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28682.138394 secs 2024-04-26T19:49:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:03 smithi033 ceph-mon[27845]: pgmap v15640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28682.138394 secs 2024-04-26T19:49:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:03 smithi096 ceph-mon[32753]: pgmap v15640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28682.138394 secs 2024-04-26T19:49:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28685.049331 secs 2024-04-26T19:49:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28685.049331 secs 2024-04-26T19:49:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28685.049331 secs 2024-04-26T19:49:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:05 smithi194 ceph-mon[33017]: pgmap v15641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:05 smithi033 ceph-mon[27845]: pgmap v15641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:05 smithi096 ceph-mon[32753]: pgmap v15641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:07 smithi194 ceph-mon[33017]: pgmap v15642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:07 smithi033 ceph-mon[27845]: pgmap v15642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:07 smithi096 ceph-mon[32753]: pgmap v15642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:08.809+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:08.809+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:08.809+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:49:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28687.138477 secs 2024-04-26T19:49:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28687.138477 secs 2024-04-26T19:49:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28687.138477 secs 2024-04-26T19:49:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28690.049449 secs 2024-04-26T19:49:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:10 smithi194 ceph-mon[33017]: pgmap v15643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28690.049449 secs 2024-04-26T19:49:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:10 smithi033 ceph-mon[27845]: pgmap v15643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28690.049449 secs 2024-04-26T19:49:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:10 smithi096 ceph-mon[32753]: pgmap v15643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:12 smithi194 ceph-mon[33017]: pgmap v15644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:12 smithi033 ceph-mon[27845]: pgmap v15644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:12 smithi096 ceph-mon[32753]: pgmap v15644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:14 smithi194 ceph-mon[33017]: pgmap v15645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:14.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28692.138553 secs 2024-04-26T19:49:14.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:49:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:14 smithi033 ceph-mon[27845]: pgmap v15645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28692.138553 secs 2024-04-26T19:49:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:49:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:14 smithi096 ceph-mon[32753]: pgmap v15645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28692.138553 secs 2024-04-26T19:49:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:49:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28695.049518 secs 2024-04-26T19:49:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:49:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:49:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:49:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28695.049518 secs 2024-04-26T19:49:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:49:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:49:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:49:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28695.049518 secs 2024-04-26T19:49:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:49:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:49:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:49:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:16 smithi194 ceph-mon[33017]: pgmap v15646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:16 smithi194 ceph-mon[33017]: pgmap v15647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:49:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:16 smithi033 ceph-mon[27845]: pgmap v15646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:16 smithi033 ceph-mon[27845]: pgmap v15647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:49:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:16 smithi096 ceph-mon[32753]: pgmap v15646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:16 smithi096 ceph-mon[32753]: pgmap v15647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:49:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:18 smithi194 ceph-mon[33017]: pgmap v15648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:18 smithi033 ceph-mon[27845]: pgmap v15648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:18 smithi096 ceph-mon[32753]: pgmap v15648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28697.138635 secs 2024-04-26T19:49:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28697.138635 secs 2024-04-26T19:49:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28697.138635 secs 2024-04-26T19:49:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:20 smithi194 ceph-mon[33017]: pgmap v15649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28700.049640 secs 2024-04-26T19:49:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:20 smithi033 ceph-mon[27845]: pgmap v15649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28700.049640 secs 2024-04-26T19:49:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:20 smithi096 ceph-mon[32753]: pgmap v15649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28700.049640 secs 2024-04-26T19:49:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:22 smithi194 ceph-mon[33017]: pgmap v15650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:22 smithi033 ceph-mon[27845]: pgmap v15650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:22 smithi096 ceph-mon[32753]: pgmap v15650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:23.816+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:23.816+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:23.816+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:49:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:24 smithi194 ceph-mon[33017]: pgmap v15651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28702.138717 secs 2024-04-26T19:49:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:24 smithi033 ceph-mon[27845]: pgmap v15651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28702.138717 secs 2024-04-26T19:49:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:24 smithi096 ceph-mon[32753]: pgmap v15651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:49:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28702.138717 secs 2024-04-26T19:49:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28705.049773 secs 2024-04-26T19:49:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28705.049773 secs 2024-04-26T19:49:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28705.049773 secs 2024-04-26T19:49:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:26 smithi194 ceph-mon[33017]: pgmap v15652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:49:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:26 smithi033 ceph-mon[27845]: pgmap v15652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:49:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:26 smithi096 ceph-mon[32753]: pgmap v15652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:49:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:28 smithi194 ceph-mon[33017]: pgmap v15653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:49:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:28 smithi033 ceph-mon[27845]: pgmap v15653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:49:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:28 smithi096 ceph-mon[32753]: pgmap v15653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:49:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28707.138798 secs 2024-04-26T19:49:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28707.138798 secs 2024-04-26T19:49:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28707.138798 secs 2024-04-26T19:49:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:30 smithi194 ceph-mon[33017]: pgmap v15654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28710.049887 secs 2024-04-26T19:49:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:30 smithi033 ceph-mon[27845]: pgmap v15654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28710.049887 secs 2024-04-26T19:49:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:30 smithi096 ceph-mon[32753]: pgmap v15654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28710.049887 secs 2024-04-26T19:49:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:32 smithi194 ceph-mon[33017]: pgmap v15655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:32 smithi033 ceph-mon[27845]: pgmap v15655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:32 smithi096 ceph-mon[32753]: pgmap v15655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:34 smithi194 ceph-mon[33017]: pgmap v15656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28712.138881 secs 2024-04-26T19:49:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:34 smithi033 ceph-mon[27845]: pgmap v15656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28712.138881 secs 2024-04-26T19:49:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:34 smithi096 ceph-mon[32753]: pgmap v15656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28712.138881 secs 2024-04-26T19:49:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28715.050022 secs 2024-04-26T19:49:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28715.050022 secs 2024-04-26T19:49:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28715.050022 secs 2024-04-26T19:49:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:36 smithi033 ceph-mon[27845]: pgmap v15657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:36 smithi096 ceph-mon[32753]: pgmap v15657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:36 smithi194 ceph-mon[33017]: pgmap v15657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:38 smithi033 ceph-mon[27845]: pgmap v15658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:49:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:38 smithi096 ceph-mon[32753]: pgmap v15658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:49:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:38 smithi194 ceph-mon[33017]: pgmap v15658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:49:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:38.827+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:38.827+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:38.827+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:49:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28717.138970 secs 2024-04-26T19:49:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28717.138970 secs 2024-04-26T19:49:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28717.138970 secs 2024-04-26T19:49:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:40 smithi033 ceph-mon[27845]: pgmap v15659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28720.050168 secs 2024-04-26T19:49:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:40 smithi096 ceph-mon[32753]: pgmap v15659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28720.050168 secs 2024-04-26T19:49:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:40 smithi194 ceph-mon[33017]: pgmap v15659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28720.050168 secs 2024-04-26T19:49:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:42 smithi033 ceph-mon[27845]: pgmap v15660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:42 smithi096 ceph-mon[32753]: pgmap v15660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:42 smithi194 ceph-mon[33017]: pgmap v15660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:44 smithi033 ceph-mon[27845]: pgmap v15661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28722.139051 secs 2024-04-26T19:49:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:44 smithi096 ceph-mon[32753]: pgmap v15661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28722.139051 secs 2024-04-26T19:49:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:44 smithi194 ceph-mon[33017]: pgmap v15661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28722.139051 secs 2024-04-26T19:49:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28725.050321 secs 2024-04-26T19:49:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28725.050321 secs 2024-04-26T19:49:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28725.050321 secs 2024-04-26T19:49:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:46 smithi033 ceph-mon[27845]: pgmap v15662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:46 smithi096 ceph-mon[32753]: pgmap v15662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:46 smithi194 ceph-mon[33017]: pgmap v15662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:48 smithi033 ceph-mon[27845]: pgmap v15663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:48 smithi096 ceph-mon[32753]: pgmap v15663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:48 smithi194 ceph-mon[33017]: pgmap v15663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28727.139130 secs 2024-04-26T19:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28727.139130 secs 2024-04-26T19:49:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28727.139130 secs 2024-04-26T19:49:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:50 smithi033 ceph-mon[27845]: pgmap v15664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28730.050400 secs 2024-04-26T19:49:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:50 smithi096 ceph-mon[32753]: pgmap v15664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28730.050400 secs 2024-04-26T19:49:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:50 smithi194 ceph-mon[33017]: pgmap v15664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28730.050400 secs 2024-04-26T19:49:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:51 smithi194 ceph-mon[33017]: pgmap v15665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:51 smithi033 ceph-mon[27845]: pgmap v15665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:51 smithi096 ceph-mon[32753]: pgmap v15665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:53.834+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:53.834+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:49:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:49:53.834+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:49:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:53 smithi194 ceph-mon[33017]: pgmap v15666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:53 smithi033 ceph-mon[27845]: pgmap v15666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:53 smithi096 ceph-mon[32753]: pgmap v15666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28732.139188 secs 2024-04-26T19:49:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28735.050516 secs 2024-04-26T19:49:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28732.139188 secs 2024-04-26T19:49:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28735.050516 secs 2024-04-26T19:49:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28732.139188 secs 2024-04-26T19:49:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28735.050516 secs 2024-04-26T19:49:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:55 smithi194 ceph-mon[33017]: pgmap v15667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:55 smithi033 ceph-mon[27845]: pgmap v15667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:55 smithi096 ceph-mon[32753]: pgmap v15667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:49:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:57 smithi194 ceph-mon[33017]: pgmap v15668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:57 smithi033 ceph-mon[27845]: pgmap v15668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:57 smithi096 ceph-mon[32753]: pgmap v15668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28737.139250 secs 2024-04-26T19:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:59 smithi194 ceph-mon[33017]: pgmap v15669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28740.050651 secs 2024-04-26T19:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28737.139250 secs 2024-04-26T19:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:59 smithi033 ceph-mon[27845]: pgmap v15669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28740.050651 secs 2024-04-26T19:49:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28737.139250 secs 2024-04-26T19:49:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:59 smithi096 ceph-mon[32753]: pgmap v15669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:49:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28740.050651 secs 2024-04-26T19:50:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:50:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:50:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T19:50:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:01 smithi194 ceph-mon[33017]: pgmap v15670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:01 smithi033 ceph-mon[27845]: pgmap v15670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:01 smithi096 ceph-mon[32753]: pgmap v15670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:03 smithi194 ceph-mon[33017]: pgmap v15671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28742.139330 secs 2024-04-26T19:50:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:03 smithi033 ceph-mon[27845]: pgmap v15671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28742.139330 secs 2024-04-26T19:50:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:03 smithi096 ceph-mon[32753]: pgmap v15671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28742.139330 secs 2024-04-26T19:50:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28745.050732 secs 2024-04-26T19:50:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28745.050732 secs 2024-04-26T19:50:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28745.050732 secs 2024-04-26T19:50:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:05 smithi194 ceph-mon[33017]: pgmap v15672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:05 smithi033 ceph-mon[27845]: pgmap v15672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:05 smithi096 ceph-mon[32753]: pgmap v15672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:07 smithi194 ceph-mon[33017]: pgmap v15673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:07 smithi033 ceph-mon[27845]: pgmap v15673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:07.957 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:07 smithi096 ceph-mon[32753]: pgmap v15673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:08.851+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:08.851+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:08.851+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:50:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28747.139413 secs 2024-04-26T19:50:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28747.139413 secs 2024-04-26T19:50:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28747.139413 secs 2024-04-26T19:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:09 smithi194 ceph-mon[33017]: pgmap v15674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28750.050830 secs 2024-04-26T19:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:09 smithi033 ceph-mon[27845]: pgmap v15674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28750.050830 secs 2024-04-26T19:50:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:09 smithi096 ceph-mon[32753]: pgmap v15674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28750.050830 secs 2024-04-26T19:50:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:11 smithi033 ceph-mon[27845]: pgmap v15675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:11 smithi096 ceph-mon[32753]: pgmap v15675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:11 smithi194 ceph-mon[33017]: pgmap v15675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:13 smithi033 ceph-mon[27845]: pgmap v15676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28752.139494 secs 2024-04-26T19:50:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:13 smithi096 ceph-mon[32753]: pgmap v15676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28752.139494 secs 2024-04-26T19:50:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:13 smithi194 ceph-mon[33017]: pgmap v15676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28752.139494 secs 2024-04-26T19:50:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28755.050948 secs 2024-04-26T19:50:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28755.050948 secs 2024-04-26T19:50:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28755.050948 secs 2024-04-26T19:50:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:15 smithi033 ceph-mon[27845]: pgmap v15677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:50:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:15 smithi096 ceph-mon[32753]: pgmap v15677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:50:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:15 smithi194 ceph-mon[33017]: pgmap v15677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:50:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:17 smithi033 ceph-mon[27845]: pgmap v15678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:17 smithi096 ceph-mon[32753]: pgmap v15678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:17.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:17 smithi194 ceph-mon[33017]: pgmap v15678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28757.139586 secs 2024-04-26T19:50:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:50:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:50:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:50:18.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:18 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28757.139586 secs 2024-04-26T19:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:50:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:18 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:50:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28757.139586 secs 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T19:50:19.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T19:50:19.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:18 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T19:50:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:19 smithi033 ceph-mon[27845]: pgmap v15679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28760.051097 secs 2024-04-26T19:50:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:19 smithi096 ceph-mon[32753]: pgmap v15679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28760.051097 secs 2024-04-26T19:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:19 smithi194 ceph-mon[33017]: pgmap v15679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28760.051097 secs 2024-04-26T19:50:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: pgmap v15680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: pgmap v15681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: pgmap v15682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: pgmap v15680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: pgmap v15681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: pgmap v15682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: pgmap v15680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:50:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: pgmap v15681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:50:22.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: pgmap v15682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:22.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:50:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:23.867+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:23.867+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:23.867+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:50:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28762.139669 secs 2024-04-26T19:50:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:24 smithi194 ceph-mon[33017]: pgmap v15683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:50:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28762.139669 secs 2024-04-26T19:50:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:24 smithi033 ceph-mon[27845]: pgmap v15683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:50:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28762.139669 secs 2024-04-26T19:50:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:24 smithi096 ceph-mon[32753]: pgmap v15683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:50:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28765.051218 secs 2024-04-26T19:50:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28765.051218 secs 2024-04-26T19:50:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28765.051218 secs 2024-04-26T19:50:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:26 smithi194 ceph-mon[33017]: pgmap v15684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:26 smithi033 ceph-mon[27845]: pgmap v15684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:26 smithi096 ceph-mon[32753]: pgmap v15684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:26.691+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:50:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:26.691+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:50:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:26.692+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 227 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T19:50:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:26.692+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 227 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:50:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:28 smithi194 ceph-mon[33017]: pgmap v15685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:28 smithi033 ceph-mon[27845]: pgmap v15685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:28 smithi096 ceph-mon[32753]: pgmap v15685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T19:50:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28767.139754 secs 2024-04-26T19:50:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28767.139754 secs 2024-04-26T19:50:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28767.139754 secs 2024-04-26T19:50:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28770.051333 secs 2024-04-26T19:50:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:30 smithi194 ceph-mon[33017]: pgmap v15686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:50:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28770.051333 secs 2024-04-26T19:50:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:30 smithi033 ceph-mon[27845]: pgmap v15686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:50:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28770.051333 secs 2024-04-26T19:50:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:30 smithi096 ceph-mon[32753]: pgmap v15686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:50:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:32 smithi194 ceph-mon[33017]: pgmap v15687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:50:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:32 smithi033 ceph-mon[27845]: pgmap v15687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:50:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:32 smithi096 ceph-mon[32753]: pgmap v15687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:50:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28772.139836 secs 2024-04-26T19:50:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:34 smithi033 ceph-mon[27845]: pgmap v15688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28772.139836 secs 2024-04-26T19:50:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:34 smithi096 ceph-mon[32753]: pgmap v15688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28772.139836 secs 2024-04-26T19:50:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:34 smithi194 ceph-mon[33017]: pgmap v15688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28775.051412 secs 2024-04-26T19:50:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28775.051412 secs 2024-04-26T19:50:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28775.051412 secs 2024-04-26T19:50:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:36 smithi033 ceph-mon[27845]: pgmap v15689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:36 smithi096 ceph-mon[32753]: pgmap v15689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:36 smithi194 ceph-mon[33017]: pgmap v15689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:38 smithi033 ceph-mon[27845]: pgmap v15690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:38 smithi096 ceph-mon[32753]: pgmap v15690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:38 smithi194 ceph-mon[33017]: pgmap v15690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:38.871+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:38.871+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:38.871+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:50:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28777.139920 secs 2024-04-26T19:50:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28777.139920 secs 2024-04-26T19:50:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28777.139920 secs 2024-04-26T19:50:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28780.051510 secs 2024-04-26T19:50:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:40 smithi033 ceph-mon[27845]: pgmap v15691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28780.051510 secs 2024-04-26T19:50:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:40 smithi096 ceph-mon[32753]: pgmap v15691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28780.051510 secs 2024-04-26T19:50:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:40 smithi194 ceph-mon[33017]: pgmap v15691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:42 smithi033 ceph-mon[27845]: pgmap v15692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:42 smithi096 ceph-mon[32753]: pgmap v15692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:42 smithi194 ceph-mon[33017]: pgmap v15692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28782.140002 secs 2024-04-26T19:50:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:44 smithi033 ceph-mon[27845]: pgmap v15693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28782.140002 secs 2024-04-26T19:50:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:44 smithi096 ceph-mon[32753]: pgmap v15693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28782.140002 secs 2024-04-26T19:50:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:44 smithi194 ceph-mon[33017]: pgmap v15693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28785.051574 secs 2024-04-26T19:50:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28785.051574 secs 2024-04-26T19:50:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28785.051574 secs 2024-04-26T19:50:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:46 smithi033 ceph-mon[27845]: pgmap v15694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:46 smithi096 ceph-mon[32753]: pgmap v15694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:46 smithi194 ceph-mon[33017]: pgmap v15694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:50:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:48 smithi033 ceph-mon[27845]: pgmap v15695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:48 smithi096 ceph-mon[32753]: pgmap v15695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:48 smithi194 ceph-mon[33017]: pgmap v15695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28787.140085 secs 2024-04-26T19:50:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:49 smithi194 ceph-mon[33017]: pgmap v15696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28787.140085 secs 2024-04-26T19:50:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:49 smithi033 ceph-mon[27845]: pgmap v15696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28787.140085 secs 2024-04-26T19:50:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:49 smithi096 ceph-mon[32753]: pgmap v15696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:50:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28790.051673 secs 2024-04-26T19:50:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28790.051673 secs 2024-04-26T19:50:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28790.051673 secs 2024-04-26T19:50:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:51 smithi194 ceph-mon[33017]: pgmap v15697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:50:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:51 smithi033 ceph-mon[27845]: pgmap v15697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:50:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:51 smithi096 ceph-mon[32753]: pgmap v15697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:50:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:53.885+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:53.885+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:50:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:50:53.885+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:50:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28792.140146 secs 2024-04-26T19:50:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:54 smithi194 ceph-mon[33017]: pgmap v15698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:50:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28792.140146 secs 2024-04-26T19:50:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:54 smithi033 ceph-mon[27845]: pgmap v15698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:50:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28792.140146 secs 2024-04-26T19:50:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:54 smithi096 ceph-mon[32753]: pgmap v15698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:50:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28795.051786 secs 2024-04-26T19:50:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28795.051786 secs 2024-04-26T19:50:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28795.051786 secs 2024-04-26T19:50:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:56 smithi194 ceph-mon[33017]: pgmap v15699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:50:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:56 smithi033 ceph-mon[27845]: pgmap v15699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:50:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:56 smithi096 ceph-mon[32753]: pgmap v15699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:50:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:58 smithi194 ceph-mon[33017]: pgmap v15700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:50:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:58 smithi033 ceph-mon[27845]: pgmap v15700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:50:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:58 smithi096 ceph-mon[32753]: pgmap v15700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:50:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:50:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28797.140210 secs 2024-04-26T19:50:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:50:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28797.140210 secs 2024-04-26T19:50:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:50:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28797.140210 secs 2024-04-26T19:51:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28800.051921 secs 2024-04-26T19:51:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:00 smithi033 ceph-mon[27845]: pgmap v15701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:51:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28800.051921 secs 2024-04-26T19:51:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:00 smithi096 ceph-mon[32753]: pgmap v15701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:51:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28800.051921 secs 2024-04-26T19:51:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:00 smithi194 ceph-mon[33017]: pgmap v15701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:51:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:02 smithi033 ceph-mon[27845]: pgmap v15702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:51:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:02 smithi096 ceph-mon[32753]: pgmap v15702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:51:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:02 smithi194 ceph-mon[33017]: pgmap v15702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:51:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28802.140267 secs 2024-04-26T19:51:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:04 smithi033 ceph-mon[27845]: pgmap v15703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28802.140267 secs 2024-04-26T19:51:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:04 smithi096 ceph-mon[32753]: pgmap v15703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:04.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28802.140267 secs 2024-04-26T19:51:04.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:04 smithi194 ceph-mon[33017]: pgmap v15703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28805.052071 secs 2024-04-26T19:51:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28805.052071 secs 2024-04-26T19:51:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28805.052071 secs 2024-04-26T19:51:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:06 smithi033 ceph-mon[27845]: pgmap v15704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:06 smithi096 ceph-mon[32753]: pgmap v15704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:06 smithi194 ceph-mon[33017]: pgmap v15704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:08 smithi033 ceph-mon[27845]: pgmap v15705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:08 smithi096 ceph-mon[32753]: pgmap v15705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:08 smithi194 ceph-mon[33017]: pgmap v15705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:08.902+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:08.902+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:08.902+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:51:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28807.140350 secs 2024-04-26T19:51:09.439 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28807.140350 secs 2024-04-26T19:51:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28807.140350 secs 2024-04-26T19:51:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28810.052194 secs 2024-04-26T19:51:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:10 smithi033 ceph-mon[27845]: pgmap v15706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28810.052194 secs 2024-04-26T19:51:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:10 smithi096 ceph-mon[32753]: pgmap v15706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28810.052194 secs 2024-04-26T19:51:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:10 smithi194 ceph-mon[33017]: pgmap v15706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:12 smithi033 ceph-mon[27845]: pgmap v15707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:12 smithi096 ceph-mon[32753]: pgmap v15707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:12 smithi194 ceph-mon[33017]: pgmap v15707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28812.140433 secs 2024-04-26T19:51:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:14 smithi033 ceph-mon[27845]: pgmap v15708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28812.140433 secs 2024-04-26T19:51:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:14 smithi096 ceph-mon[32753]: pgmap v15708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28812.140433 secs 2024-04-26T19:51:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:14 smithi194 ceph-mon[33017]: pgmap v15708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28815.052272 secs 2024-04-26T19:51:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28815.052272 secs 2024-04-26T19:51:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28815.052272 secs 2024-04-26T19:51:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:16 smithi096 ceph-mon[32753]: pgmap v15709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:16 smithi194 ceph-mon[33017]: pgmap v15709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:16 smithi033 ceph-mon[27845]: pgmap v15709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:18 smithi194 ceph-mon[33017]: pgmap v15710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:18 smithi033 ceph-mon[27845]: pgmap v15710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:18 smithi096 ceph-mon[32753]: pgmap v15710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28817.140494 secs 2024-04-26T19:51:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28817.140494 secs 2024-04-26T19:51:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28817.140494 secs 2024-04-26T19:51:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28820.052333 secs 2024-04-26T19:51:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:20 smithi194 ceph-mon[33017]: pgmap v15711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28820.052333 secs 2024-04-26T19:51:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:20 smithi033 ceph-mon[27845]: pgmap v15711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28820.052333 secs 2024-04-26T19:51:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:20 smithi096 ceph-mon[32753]: pgmap v15711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:21 smithi194 ceph-mon[33017]: pgmap v15712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:21 smithi033 ceph-mon[27845]: pgmap v15712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:21 smithi096 ceph-mon[32753]: pgmap v15712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:51:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:51:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:51:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:51:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:51:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:23 smithi194 ceph-mon[33017]: pgmap v15713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:23 smithi194 ceph-mon[33017]: pgmap v15714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:51:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:51:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:51:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:23 smithi033 ceph-mon[27845]: pgmap v15713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:23 smithi033 ceph-mon[27845]: pgmap v15714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:51:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:51:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:51:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:23 smithi096 ceph-mon[32753]: pgmap v15713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:23 smithi096 ceph-mon[32753]: pgmap v15714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:51:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:23.917+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:23.917+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:51:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:23.917+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28822.140572 secs 2024-04-26T19:51:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28822.140572 secs 2024-04-26T19:51:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28822.140572 secs 2024-04-26T19:51:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28825.052415 secs 2024-04-26T19:51:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:25 smithi194 ceph-mon[33017]: pgmap v15715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28825.052415 secs 2024-04-26T19:51:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:25 smithi033 ceph-mon[27845]: pgmap v15715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28825.052415 secs 2024-04-26T19:51:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:25 smithi096 ceph-mon[32753]: pgmap v15715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:27 smithi194 ceph-mon[33017]: pgmap v15716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:27 smithi033 ceph-mon[27845]: pgmap v15716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:27 smithi096 ceph-mon[32753]: pgmap v15716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:29.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28827.140652 secs 2024-04-26T19:51:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28827.140652 secs 2024-04-26T19:51:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28827.140652 secs 2024-04-26T19:51:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28830.052490 secs 2024-04-26T19:51:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:29 smithi096 ceph-mon[32753]: pgmap v15717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28830.052490 secs 2024-04-26T19:51:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:29 smithi194 ceph-mon[33017]: pgmap v15717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28830.052490 secs 2024-04-26T19:51:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:29 smithi033 ceph-mon[27845]: pgmap v15717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:31 smithi194 ceph-mon[33017]: pgmap v15718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:31 smithi033 ceph-mon[27845]: pgmap v15718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:31 smithi096 ceph-mon[32753]: pgmap v15718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:51:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:33 smithi194 ceph-mon[33017]: pgmap v15719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:51:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28832.140711 secs 2024-04-26T19:51:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:33 smithi033 ceph-mon[27845]: pgmap v15719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:51:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28832.140711 secs 2024-04-26T19:51:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:33 smithi096 ceph-mon[32753]: pgmap v15719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:51:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28832.140711 secs 2024-04-26T19:51:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28835.052623 secs 2024-04-26T19:51:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28835.052623 secs 2024-04-26T19:51:35.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28835.052623 secs 2024-04-26T19:51:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:35 smithi194 ceph-mon[33017]: pgmap v15720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:35 smithi033 ceph-mon[27845]: pgmap v15720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:35 smithi096 ceph-mon[32753]: pgmap v15720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:37 smithi194 ceph-mon[33017]: pgmap v15721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:37 smithi033 ceph-mon[27845]: pgmap v15721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:37 smithi096 ceph-mon[32753]: pgmap v15721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:38.928+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:38.928+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:38.928+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:51:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28837.140798 secs 2024-04-26T19:51:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28837.140798 secs 2024-04-26T19:51:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28837.140798 secs 2024-04-26T19:51:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28840.052745 secs 2024-04-26T19:51:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:39 smithi194 ceph-mon[33017]: pgmap v15722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28840.052745 secs 2024-04-26T19:51:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:39 smithi033 ceph-mon[27845]: pgmap v15722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28840.052745 secs 2024-04-26T19:51:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:39 smithi096 ceph-mon[32753]: pgmap v15722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:41 smithi194 ceph-mon[33017]: pgmap v15723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:42.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:41 smithi033 ceph-mon[27845]: pgmap v15723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:41 smithi096 ceph-mon[32753]: pgmap v15723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:43 smithi194 ceph-mon[33017]: pgmap v15724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28842.140884 secs 2024-04-26T19:51:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:43 smithi033 ceph-mon[27845]: pgmap v15724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28842.140884 secs 2024-04-26T19:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:43 smithi096 ceph-mon[32753]: pgmap v15724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28842.140884 secs 2024-04-26T19:51:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28845.052863 secs 2024-04-26T19:51:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28845.052863 secs 2024-04-26T19:51:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28845.052863 secs 2024-04-26T19:51:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:45 smithi194 ceph-mon[33017]: pgmap v15725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:45 smithi033 ceph-mon[27845]: pgmap v15725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:45 smithi096 ceph-mon[32753]: pgmap v15725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:47 smithi194 ceph-mon[33017]: pgmap v15726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:47 smithi033 ceph-mon[27845]: pgmap v15726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:47 smithi096 ceph-mon[32753]: pgmap v15726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28847.140977 secs 2024-04-26T19:51:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28847.140977 secs 2024-04-26T19:51:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28847.140977 secs 2024-04-26T19:51:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28850.052977 secs 2024-04-26T19:51:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:49 smithi194 ceph-mon[33017]: pgmap v15727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28850.052977 secs 2024-04-26T19:51:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:49 smithi033 ceph-mon[27845]: pgmap v15727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28850.052977 secs 2024-04-26T19:51:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:49 smithi096 ceph-mon[32753]: pgmap v15727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:51 smithi194 ceph-mon[33017]: pgmap v15728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:51 smithi033 ceph-mon[27845]: pgmap v15728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:51 smithi096 ceph-mon[32753]: pgmap v15728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:53.933+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:53.933+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:51:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:51:53.933+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:51:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:53 smithi194 ceph-mon[33017]: pgmap v15729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28852.141062 secs 2024-04-26T19:51:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:53 smithi033 ceph-mon[27845]: pgmap v15729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28852.141062 secs 2024-04-26T19:51:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:53 smithi096 ceph-mon[32753]: pgmap v15729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28852.141062 secs 2024-04-26T19:51:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28855.053094 secs 2024-04-26T19:51:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28855.053094 secs 2024-04-26T19:51:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28855.053094 secs 2024-04-26T19:51:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:55 smithi194 ceph-mon[33017]: pgmap v15730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:55 smithi033 ceph-mon[27845]: pgmap v15730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:55 smithi096 ceph-mon[32753]: pgmap v15730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:51:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:57 smithi194 ceph-mon[33017]: pgmap v15731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:57 smithi033 ceph-mon[27845]: pgmap v15731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:57 smithi096 ceph-mon[32753]: pgmap v15731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:51:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28857.141125 secs 2024-04-26T19:51:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28857.141125 secs 2024-04-26T19:51:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28857.141125 secs 2024-04-26T19:52:00.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28860.053209 secs 2024-04-26T19:52:00.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:51:59 smithi194 ceph-mon[33017]: pgmap v15732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28860.053209 secs 2024-04-26T19:52:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:51:59 smithi033 ceph-mon[27845]: pgmap v15732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28860.053209 secs 2024-04-26T19:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:51:59 smithi096 ceph-mon[32753]: pgmap v15732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:01 smithi194 ceph-mon[33017]: pgmap v15733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:01 smithi033 ceph-mon[27845]: pgmap v15733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:01 smithi096 ceph-mon[32753]: pgmap v15733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:04.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:03 smithi194 ceph-mon[33017]: pgmap v15734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:04.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28862.141215 secs 2024-04-26T19:52:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:03 smithi033 ceph-mon[27845]: pgmap v15734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28862.141215 secs 2024-04-26T19:52:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:03 smithi096 ceph-mon[32753]: pgmap v15734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28862.141215 secs 2024-04-26T19:52:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28865.053325 secs 2024-04-26T19:52:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28865.053325 secs 2024-04-26T19:52:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28865.053325 secs 2024-04-26T19:52:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:05 smithi033 ceph-mon[27845]: pgmap v15735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:05 smithi096 ceph-mon[32753]: pgmap v15735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:05 smithi194 ceph-mon[33017]: pgmap v15735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:07 smithi033 ceph-mon[27845]: pgmap v15736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:07 smithi096 ceph-mon[32753]: pgmap v15736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:07 smithi194 ceph-mon[33017]: pgmap v15736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:08.949+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:08.949+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:08.949+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:52:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28867.141299 secs 2024-04-26T19:52:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28867.141299 secs 2024-04-26T19:52:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28867.141299 secs 2024-04-26T19:52:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28870.053433 secs 2024-04-26T19:52:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:09 smithi033 ceph-mon[27845]: pgmap v15737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28870.053433 secs 2024-04-26T19:52:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:09 smithi096 ceph-mon[32753]: pgmap v15737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28870.053433 secs 2024-04-26T19:52:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:09 smithi194 ceph-mon[33017]: pgmap v15737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:11 smithi033 ceph-mon[27845]: pgmap v15738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:11 smithi096 ceph-mon[32753]: pgmap v15738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:11 smithi194 ceph-mon[33017]: pgmap v15738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:13 smithi033 ceph-mon[27845]: pgmap v15739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28872.141387 secs 2024-04-26T19:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:13 smithi096 ceph-mon[32753]: pgmap v15739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28872.141387 secs 2024-04-26T19:52:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:13 smithi194 ceph-mon[33017]: pgmap v15739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28872.141387 secs 2024-04-26T19:52:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28875.053552 secs 2024-04-26T19:52:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28875.053552 secs 2024-04-26T19:52:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28875.053552 secs 2024-04-26T19:52:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:15 smithi033 ceph-mon[27845]: pgmap v15740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:15 smithi096 ceph-mon[32753]: pgmap v15740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:15 smithi194 ceph-mon[33017]: pgmap v15740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:17 smithi033 ceph-mon[27845]: pgmap v15741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:17 smithi096 ceph-mon[32753]: pgmap v15741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:17 smithi194 ceph-mon[33017]: pgmap v15741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28877.141470 secs 2024-04-26T19:52:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28877.141470 secs 2024-04-26T19:52:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28877.141470 secs 2024-04-26T19:52:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28880.053668 secs 2024-04-26T19:52:20.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:19 smithi096 ceph-mon[32753]: pgmap v15742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28880.053668 secs 2024-04-26T19:52:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:19 smithi194 ceph-mon[33017]: pgmap v15742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28880.053668 secs 2024-04-26T19:52:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:19 smithi033 ceph-mon[27845]: pgmap v15742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:21 smithi096 ceph-mon[32753]: pgmap v15743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:21 smithi194 ceph-mon[33017]: pgmap v15743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:21 smithi033 ceph-mon[27845]: pgmap v15743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:23.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:23.952+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:23.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:23.952+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:23.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:23.952+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:52:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:23 smithi194 ceph-mon[33017]: pgmap v15744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28882.141553 secs 2024-04-26T19:52:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:52:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:23 smithi033 ceph-mon[27845]: pgmap v15744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28882.141553 secs 2024-04-26T19:52:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:52:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:23 smithi096 ceph-mon[32753]: pgmap v15744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28882.141553 secs 2024-04-26T19:52:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:52:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28885.053764 secs 2024-04-26T19:52:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:52:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:52:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:52:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28885.053764 secs 2024-04-26T19:52:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:52:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:52:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:52:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28885.053764 secs 2024-04-26T19:52:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:52:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:52:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:52:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:25 smithi194 ceph-mon[33017]: pgmap v15745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:25 smithi194 ceph-mon[33017]: pgmap v15746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:52:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:25 smithi033 ceph-mon[27845]: pgmap v15745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:25 smithi033 ceph-mon[27845]: pgmap v15746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:52:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:25 smithi096 ceph-mon[32753]: pgmap v15745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:25 smithi096 ceph-mon[32753]: pgmap v15746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T19:52:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:27 smithi194 ceph-mon[33017]: pgmap v15747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:27 smithi033 ceph-mon[27845]: pgmap v15747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:27 smithi096 ceph-mon[32753]: pgmap v15747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:29.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28887.141638 secs 2024-04-26T19:52:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28887.141638 secs 2024-04-26T19:52:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28887.141638 secs 2024-04-26T19:52:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:29 smithi033 ceph-mon[27845]: pgmap v15748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28890.053885 secs 2024-04-26T19:52:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:29 smithi096 ceph-mon[32753]: pgmap v15748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28890.053885 secs 2024-04-26T19:52:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:29 smithi194 ceph-mon[33017]: pgmap v15748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28890.053885 secs 2024-04-26T19:52:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:31 smithi194 ceph-mon[33017]: pgmap v15749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:31 smithi033 ceph-mon[27845]: pgmap v15749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:31 smithi096 ceph-mon[32753]: pgmap v15749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:33 smithi194 ceph-mon[33017]: pgmap v15750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28892.141719 secs 2024-04-26T19:52:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:33 smithi033 ceph-mon[27845]: pgmap v15750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28892.141719 secs 2024-04-26T19:52:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:33 smithi096 ceph-mon[32753]: pgmap v15750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:52:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28892.141719 secs 2024-04-26T19:52:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28895.054007 secs 2024-04-26T19:52:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28895.054007 secs 2024-04-26T19:52:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28895.054007 secs 2024-04-26T19:52:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:35 smithi194 ceph-mon[33017]: pgmap v15751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:52:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:35 smithi033 ceph-mon[27845]: pgmap v15751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:52:36.625 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:35 smithi096 ceph-mon[32753]: pgmap v15751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:52:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:38 smithi194 ceph-mon[33017]: pgmap v15752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:37 smithi033 ceph-mon[27845]: pgmap v15752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:37 smithi096 ceph-mon[32753]: pgmap v15752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:38.964+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:38.964+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:52:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:38.964+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28897.141802 secs 2024-04-26T19:52:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28897.141802 secs 2024-04-26T19:52:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28897.141802 secs 2024-04-26T19:52:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:40 smithi194 ceph-mon[33017]: pgmap v15753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28900.054138 secs 2024-04-26T19:52:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:40 smithi033 ceph-mon[27845]: pgmap v15753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28900.054138 secs 2024-04-26T19:52:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:40 smithi096 ceph-mon[32753]: pgmap v15753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28900.054138 secs 2024-04-26T19:52:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:42 smithi194 ceph-mon[33017]: pgmap v15754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:42 smithi033 ceph-mon[27845]: pgmap v15754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:42 smithi096 ceph-mon[32753]: pgmap v15754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:44 smithi194 ceph-mon[33017]: pgmap v15755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28902.141859 secs 2024-04-26T19:52:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:44 smithi033 ceph-mon[27845]: pgmap v15755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28902.141859 secs 2024-04-26T19:52:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:44 smithi096 ceph-mon[32753]: pgmap v15755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28902.141859 secs 2024-04-26T19:52:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28905.054233 secs 2024-04-26T19:52:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28905.054233 secs 2024-04-26T19:52:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28905.054233 secs 2024-04-26T19:52:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:46 smithi194 ceph-mon[33017]: pgmap v15756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:46 smithi033 ceph-mon[27845]: pgmap v15756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:46 smithi096 ceph-mon[32753]: pgmap v15756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:48 smithi194 ceph-mon[33017]: pgmap v15757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:48 smithi033 ceph-mon[27845]: pgmap v15757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:48 smithi096 ceph-mon[32753]: pgmap v15757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28907.141941 secs 2024-04-26T19:52:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28907.141941 secs 2024-04-26T19:52:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28907.141941 secs 2024-04-26T19:52:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:50 smithi194 ceph-mon[33017]: pgmap v15758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28910.054350 secs 2024-04-26T19:52:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:50 smithi033 ceph-mon[27845]: pgmap v15758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28910.054350 secs 2024-04-26T19:52:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:50 smithi096 ceph-mon[32753]: pgmap v15758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28910.054350 secs 2024-04-26T19:52:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:52 smithi194 ceph-mon[33017]: pgmap v15759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:52 smithi033 ceph-mon[27845]: pgmap v15759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:52 smithi096 ceph-mon[32753]: pgmap v15759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:53.978+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:53.978+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:52:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:52:53.978+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:52:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:54 smithi194 ceph-mon[33017]: pgmap v15760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28912.142003 secs 2024-04-26T19:52:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:54 smithi033 ceph-mon[27845]: pgmap v15760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28912.142003 secs 2024-04-26T19:52:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:54 smithi096 ceph-mon[32753]: pgmap v15760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:52:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28912.142003 secs 2024-04-26T19:52:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28915.054464 secs 2024-04-26T19:52:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28915.054464 secs 2024-04-26T19:52:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28915.054464 secs 2024-04-26T19:52:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:56 smithi194 ceph-mon[33017]: pgmap v15761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:52:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:56 smithi033 ceph-mon[27845]: pgmap v15761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:52:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:56 smithi096 ceph-mon[32753]: pgmap v15761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:52:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:58 smithi194 ceph-mon[33017]: pgmap v15762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:58 smithi033 ceph-mon[27845]: pgmap v15762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:58 smithi096 ceph-mon[32753]: pgmap v15762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:52:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:52:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28917.142063 secs 2024-04-26T19:52:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:52:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28917.142063 secs 2024-04-26T19:52:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:52:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28917.142063 secs 2024-04-26T19:53:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:00 smithi194 ceph-mon[33017]: pgmap v15763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28920.054584 secs 2024-04-26T19:53:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:00 smithi033 ceph-mon[27845]: pgmap v15763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28920.054584 secs 2024-04-26T19:53:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:00 smithi096 ceph-mon[32753]: pgmap v15763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28920.054584 secs 2024-04-26T19:53:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:02 smithi033 ceph-mon[27845]: pgmap v15764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:02 smithi096 ceph-mon[32753]: pgmap v15764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:02 smithi194 ceph-mon[33017]: pgmap v15764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:04 smithi033 ceph-mon[27845]: pgmap v15765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28922.142119 secs 2024-04-26T19:53:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:04 smithi096 ceph-mon[32753]: pgmap v15765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28922.142119 secs 2024-04-26T19:53:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:04 smithi194 ceph-mon[33017]: pgmap v15765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28922.142119 secs 2024-04-26T19:53:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28925.054702 secs 2024-04-26T19:53:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28925.054702 secs 2024-04-26T19:53:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28925.054702 secs 2024-04-26T19:53:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:06 smithi033 ceph-mon[27845]: pgmap v15766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:53:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:06 smithi096 ceph-mon[32753]: pgmap v15766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:53:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:06 smithi194 ceph-mon[33017]: pgmap v15766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:53:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:08 smithi033 ceph-mon[27845]: pgmap v15767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:08 smithi096 ceph-mon[32753]: pgmap v15767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:08 smithi194 ceph-mon[33017]: pgmap v15767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:08.980+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:08.980+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:53:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:08.980+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28927.142180 secs 2024-04-26T19:53:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28927.142180 secs 2024-04-26T19:53:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28927.142180 secs 2024-04-26T19:53:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:10 smithi033 ceph-mon[27845]: pgmap v15768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28930.054845 secs 2024-04-26T19:53:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:10 smithi096 ceph-mon[32753]: pgmap v15768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28930.054845 secs 2024-04-26T19:53:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:10 smithi194 ceph-mon[33017]: pgmap v15768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28930.054845 secs 2024-04-26T19:53:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:12 smithi033 ceph-mon[27845]: pgmap v15769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:12 smithi096 ceph-mon[32753]: pgmap v15769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:12 smithi194 ceph-mon[33017]: pgmap v15769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:14 smithi033 ceph-mon[27845]: pgmap v15770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28932.142282 secs 2024-04-26T19:53:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:14 smithi096 ceph-mon[32753]: pgmap v15770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28932.142282 secs 2024-04-26T19:53:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:14 smithi194 ceph-mon[33017]: pgmap v15770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28932.142282 secs 2024-04-26T19:53:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28935.054906 secs 2024-04-26T19:53:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28935.054906 secs 2024-04-26T19:53:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28935.054906 secs 2024-04-26T19:53:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:16 smithi033 ceph-mon[27845]: pgmap v15771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:16 smithi096 ceph-mon[32753]: pgmap v15771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:16 smithi194 ceph-mon[33017]: pgmap v15771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:18 smithi033 ceph-mon[27845]: pgmap v15772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:18 smithi096 ceph-mon[32753]: pgmap v15772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:18 smithi194 ceph-mon[33017]: pgmap v15772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28937.142368 secs 2024-04-26T19:53:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28937.142368 secs 2024-04-26T19:53:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28937.142368 secs 2024-04-26T19:53:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:20 smithi194 ceph-mon[33017]: pgmap v15773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28940.055042 secs 2024-04-26T19:53:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:20 smithi033 ceph-mon[27845]: pgmap v15773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28940.055042 secs 2024-04-26T19:53:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:20 smithi096 ceph-mon[32753]: pgmap v15773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28940.055042 secs 2024-04-26T19:53:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:22 smithi194 ceph-mon[33017]: pgmap v15774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:22 smithi033 ceph-mon[27845]: pgmap v15774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:22 smithi096 ceph-mon[32753]: pgmap v15774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:23.982+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:23.982+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:23.982+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:53:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:24 smithi194 ceph-mon[33017]: pgmap v15775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28942.142424 secs 2024-04-26T19:53:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:24 smithi033 ceph-mon[27845]: pgmap v15775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28942.142424 secs 2024-04-26T19:53:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:24 smithi096 ceph-mon[32753]: pgmap v15775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28942.142424 secs 2024-04-26T19:53:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28945.055164 secs 2024-04-26T19:53:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:53:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28945.055164 secs 2024-04-26T19:53:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:53:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28945.055164 secs 2024-04-26T19:53:25.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:53:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:26 smithi194 ceph-mon[33017]: pgmap v15776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:53:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:53:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:53:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:26 smithi033 ceph-mon[27845]: pgmap v15776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:53:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:53:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:53:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:26 smithi096 ceph-mon[32753]: pgmap v15776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:53:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:53:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:53:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:26.722+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:53:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:26.722+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:53:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:26.722+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 228 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T19:53:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:26.722+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 228 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T19:53:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:27 smithi194 ceph-mon[33017]: pgmap v15777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T19:53:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:27 smithi194 ceph-mon[33017]: pgmap v15778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:27 smithi033 ceph-mon[27845]: pgmap v15777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T19:53:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:27 smithi033 ceph-mon[27845]: pgmap v15778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:27 smithi096 ceph-mon[32753]: pgmap v15777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T19:53:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:27 smithi096 ceph-mon[32753]: pgmap v15778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:28 smithi194 ceph-mon[33017]: pgmap v15779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:28 smithi033 ceph-mon[27845]: pgmap v15779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:28 smithi096 ceph-mon[32753]: pgmap v15779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28947.142486 secs 2024-04-26T19:53:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28947.142486 secs 2024-04-26T19:53:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28947.142486 secs 2024-04-26T19:53:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28950.055227 secs 2024-04-26T19:53:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:30 smithi033 ceph-mon[27845]: pgmap v15780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28950.055227 secs 2024-04-26T19:53:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:30 smithi096 ceph-mon[32753]: pgmap v15780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28950.055227 secs 2024-04-26T19:53:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:30 smithi194 ceph-mon[33017]: pgmap v15780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:33.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:32 smithi033 ceph-mon[27845]: pgmap v15781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:32 smithi194 ceph-mon[33017]: pgmap v15781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:32 smithi096 ceph-mon[32753]: pgmap v15781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:53:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28952.142544 secs 2024-04-26T19:53:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28952.142544 secs 2024-04-26T19:53:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28952.142544 secs 2024-04-26T19:53:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28955.055338 secs 2024-04-26T19:53:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:35 smithi194 ceph-mon[33017]: pgmap v15782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T19:53:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28955.055338 secs 2024-04-26T19:53:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:35 smithi033 ceph-mon[27845]: pgmap v15782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T19:53:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28955.055338 secs 2024-04-26T19:53:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:35 smithi096 ceph-mon[32753]: pgmap v15782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T19:53:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:37 smithi194 ceph-mon[33017]: pgmap v15783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:53:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:37 smithi033 ceph-mon[27845]: pgmap v15783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:53:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:37 smithi096 ceph-mon[32753]: pgmap v15783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:53:38.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:38.997+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:38.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:38.997+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:53:38.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:38.997+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:39 smithi194 ceph-mon[33017]: pgmap v15784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28957.142631 secs 2024-04-26T19:53:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:39 smithi033 ceph-mon[27845]: pgmap v15784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28957.142631 secs 2024-04-26T19:53:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:39 smithi096 ceph-mon[32753]: pgmap v15784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28957.142631 secs 2024-04-26T19:53:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28960.055467 secs 2024-04-26T19:53:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28960.055467 secs 2024-04-26T19:53:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28960.055467 secs 2024-04-26T19:53:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:41 smithi194 ceph-mon[33017]: pgmap v15785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:41 smithi033 ceph-mon[27845]: pgmap v15785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:41 smithi096 ceph-mon[32753]: pgmap v15785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:43 smithi194 ceph-mon[33017]: pgmap v15786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:43 smithi033 ceph-mon[27845]: pgmap v15786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:43 smithi096 ceph-mon[32753]: pgmap v15786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28962.142710 secs 2024-04-26T19:53:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28962.142710 secs 2024-04-26T19:53:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28962.142710 secs 2024-04-26T19:53:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28965.055586 secs 2024-04-26T19:53:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:45 smithi194 ceph-mon[33017]: pgmap v15787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28965.055586 secs 2024-04-26T19:53:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:45 smithi033 ceph-mon[27845]: pgmap v15787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28965.055586 secs 2024-04-26T19:53:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:45 smithi096 ceph-mon[32753]: pgmap v15787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:47 smithi194 ceph-mon[33017]: pgmap v15788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:47 smithi033 ceph-mon[27845]: pgmap v15788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:47 smithi096 ceph-mon[32753]: pgmap v15788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:49 smithi194 ceph-mon[33017]: pgmap v15789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28967.142793 secs 2024-04-26T19:53:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:49 smithi033 ceph-mon[27845]: pgmap v15789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28967.142793 secs 2024-04-26T19:53:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:49 smithi096 ceph-mon[32753]: pgmap v15789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28967.142793 secs 2024-04-26T19:53:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28970.055770 secs 2024-04-26T19:53:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28970.055770 secs 2024-04-26T19:53:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28970.055770 secs 2024-04-26T19:53:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:51 smithi194 ceph-mon[33017]: pgmap v15790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:51 smithi033 ceph-mon[27845]: pgmap v15790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:51 smithi096 ceph-mon[32753]: pgmap v15790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:53 smithi194 ceph-mon[33017]: pgmap v15791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:53 smithi033 ceph-mon[27845]: pgmap v15791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:53 smithi096 ceph-mon[32753]: pgmap v15791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:54.010+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:54.010+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:53:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:53:54.010+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:53:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28972.142876 secs 2024-04-26T19:53:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28972.142876 secs 2024-04-26T19:53:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28972.142876 secs 2024-04-26T19:53:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28975.055887 secs 2024-04-26T19:53:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:55 smithi194 ceph-mon[33017]: pgmap v15792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28975.055887 secs 2024-04-26T19:53:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:55 smithi033 ceph-mon[27845]: pgmap v15792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28975.055887 secs 2024-04-26T19:53:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:55 smithi096 ceph-mon[32753]: pgmap v15792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:57 smithi194 ceph-mon[33017]: pgmap v15793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:57 smithi033 ceph-mon[27845]: pgmap v15793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:57 smithi096 ceph-mon[32753]: pgmap v15793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:53:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:59 smithi194 ceph-mon[33017]: pgmap v15794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:53:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28977.142959 secs 2024-04-26T19:53:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:59 smithi033 ceph-mon[27845]: pgmap v15794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:53:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28977.142959 secs 2024-04-26T19:53:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:59 smithi096 ceph-mon[32753]: pgmap v15794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:53:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:53:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28977.142959 secs 2024-04-26T19:54:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28980.056004 secs 2024-04-26T19:54:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28980.056004 secs 2024-04-26T19:54:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28980.056004 secs 2024-04-26T19:54:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:01 smithi033 ceph-mon[27845]: pgmap v15795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:01 smithi096 ceph-mon[32753]: pgmap v15795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:01 smithi194 ceph-mon[33017]: pgmap v15795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:03 smithi033 ceph-mon[27845]: pgmap v15796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:03 smithi096 ceph-mon[32753]: pgmap v15796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:03.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:03 smithi194 ceph-mon[33017]: pgmap v15796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28982.143016 secs 2024-04-26T19:54:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28982.143016 secs 2024-04-26T19:54:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28982.143016 secs 2024-04-26T19:54:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28985.056130 secs 2024-04-26T19:54:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:05 smithi033 ceph-mon[27845]: pgmap v15797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28985.056130 secs 2024-04-26T19:54:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:05 smithi096 ceph-mon[32753]: pgmap v15797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28985.056130 secs 2024-04-26T19:54:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:05 smithi194 ceph-mon[33017]: pgmap v15797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:07 smithi033 ceph-mon[27845]: pgmap v15798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:07 smithi096 ceph-mon[32753]: pgmap v15798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:07 smithi194 ceph-mon[33017]: pgmap v15798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:09.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:09.026+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:09.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:09.026+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:09.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:09.026+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:54:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28987.143100 secs 2024-04-26T19:54:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:09 smithi033 ceph-mon[27845]: pgmap v15799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28987.143100 secs 2024-04-26T19:54:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:09 smithi096 ceph-mon[32753]: pgmap v15799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28987.143100 secs 2024-04-26T19:54:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:09 smithi194 ceph-mon[33017]: pgmap v15799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28990.056176 secs 2024-04-26T19:54:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28990.056176 secs 2024-04-26T19:54:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28990.056176 secs 2024-04-26T19:54:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:11 smithi033 ceph-mon[27845]: pgmap v15800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:11 smithi096 ceph-mon[32753]: pgmap v15800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:11 smithi194 ceph-mon[33017]: pgmap v15800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:13 smithi033 ceph-mon[27845]: pgmap v15801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:13 smithi096 ceph-mon[32753]: pgmap v15801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:13 smithi194 ceph-mon[33017]: pgmap v15801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28992.143159 secs 2024-04-26T19:54:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28992.143159 secs 2024-04-26T19:54:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28992.143159 secs 2024-04-26T19:54:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 28995.056215 secs 2024-04-26T19:54:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:15 smithi096 ceph-mon[32753]: pgmap v15802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 28995.056215 secs 2024-04-26T19:54:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:15 smithi194 ceph-mon[33017]: pgmap v15802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 28995.056215 secs 2024-04-26T19:54:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:15 smithi033 ceph-mon[27845]: pgmap v15802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:17 smithi194 ceph-mon[33017]: pgmap v15803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:17 smithi033 ceph-mon[27845]: pgmap v15803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:17 smithi096 ceph-mon[32753]: pgmap v15803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 28997.143218 secs 2024-04-26T19:54:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:19 smithi194 ceph-mon[33017]: pgmap v15804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 28997.143218 secs 2024-04-26T19:54:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:19 smithi033 ceph-mon[27845]: pgmap v15804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 28997.143218 secs 2024-04-26T19:54:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:19 smithi096 ceph-mon[32753]: pgmap v15804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29000.056353 secs 2024-04-26T19:54:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29000.056353 secs 2024-04-26T19:54:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29000.056353 secs 2024-04-26T19:54:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:21 smithi194 ceph-mon[33017]: pgmap v15805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:21 smithi033 ceph-mon[27845]: pgmap v15805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:21 smithi096 ceph-mon[32753]: pgmap v15805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:23 smithi194 ceph-mon[33017]: pgmap v15806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:23 smithi033 ceph-mon[27845]: pgmap v15806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:23 smithi096 ceph-mon[32753]: pgmap v15806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:24.042+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:24.042+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:24.042+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:54:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29002.143279 secs 2024-04-26T19:54:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29002.143279 secs 2024-04-26T19:54:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29002.143279 secs 2024-04-26T19:54:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29005.056468 secs 2024-04-26T19:54:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:25 smithi194 ceph-mon[33017]: pgmap v15807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29005.056468 secs 2024-04-26T19:54:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:25 smithi033 ceph-mon[27845]: pgmap v15807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29005.056468 secs 2024-04-26T19:54:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:25 smithi096 ceph-mon[32753]: pgmap v15807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:54:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:54:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:54:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:27 smithi194 ceph-mon[33017]: pgmap v15808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:27 smithi033 ceph-mon[27845]: pgmap v15808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:27 smithi096 ceph-mon[32753]: pgmap v15808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:54:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:54:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:28 smithi194 ceph-mon[33017]: pgmap v15809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:28 smithi194 ceph-mon[33017]: pgmap v15810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:54:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:54:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:54:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:28 smithi033 ceph-mon[27845]: pgmap v15809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:28 smithi033 ceph-mon[27845]: pgmap v15810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:54:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:54:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:54:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:28 smithi096 ceph-mon[32753]: pgmap v15809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:28 smithi096 ceph-mon[32753]: pgmap v15810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:54:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29007.143367 secs 2024-04-26T19:54:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29007.143367 secs 2024-04-26T19:54:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29007.143367 secs 2024-04-26T19:54:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29010.056516 secs 2024-04-26T19:54:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:30 smithi033 ceph-mon[27845]: pgmap v15811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29010.056516 secs 2024-04-26T19:54:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:30 smithi096 ceph-mon[32753]: pgmap v15811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29010.056516 secs 2024-04-26T19:54:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:30 smithi194 ceph-mon[33017]: pgmap v15811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:32 smithi033 ceph-mon[27845]: pgmap v15812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:32 smithi096 ceph-mon[32753]: pgmap v15812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:32 smithi194 ceph-mon[33017]: pgmap v15812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29012.143438 secs 2024-04-26T19:54:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29012.143438 secs 2024-04-26T19:54:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29012.143438 secs 2024-04-26T19:54:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:35 smithi194 ceph-mon[33017]: pgmap v15813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29015.056613 secs 2024-04-26T19:54:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:35 smithi033 ceph-mon[27845]: pgmap v15813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29015.056613 secs 2024-04-26T19:54:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:35 smithi096 ceph-mon[32753]: pgmap v15813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29015.056613 secs 2024-04-26T19:54:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:37 smithi194 ceph-mon[33017]: pgmap v15814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:37 smithi033 ceph-mon[27845]: pgmap v15814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:37 smithi096 ceph-mon[32753]: pgmap v15814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:54:39.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:39.056+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:39.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:39.056+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:39.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:39.056+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:54:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:39 smithi194 ceph-mon[33017]: pgmap v15815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:54:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29017.143522 secs 2024-04-26T19:54:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:39 smithi033 ceph-mon[27845]: pgmap v15815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:54:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29017.143522 secs 2024-04-26T19:54:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:39 smithi096 ceph-mon[32753]: pgmap v15815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:54:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29017.143522 secs 2024-04-26T19:54:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29020.056730 secs 2024-04-26T19:54:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29020.056730 secs 2024-04-26T19:54:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29020.056730 secs 2024-04-26T19:54:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:41 smithi194 ceph-mon[33017]: pgmap v15816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:41 smithi033 ceph-mon[27845]: pgmap v15816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:41 smithi096 ceph-mon[32753]: pgmap v15816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:54:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:43 smithi194 ceph-mon[33017]: pgmap v15817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:43 smithi033 ceph-mon[27845]: pgmap v15817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:43 smithi096 ceph-mon[32753]: pgmap v15817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29022.143597 secs 2024-04-26T19:54:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29022.143597 secs 2024-04-26T19:54:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29022.143597 secs 2024-04-26T19:54:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:45 smithi194 ceph-mon[33017]: pgmap v15818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29025.056851 secs 2024-04-26T19:54:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:45 smithi033 ceph-mon[27845]: pgmap v15818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29025.056851 secs 2024-04-26T19:54:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:45 smithi096 ceph-mon[32753]: pgmap v15818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29025.056851 secs 2024-04-26T19:54:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:47 smithi194 ceph-mon[33017]: pgmap v15819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:47 smithi033 ceph-mon[27845]: pgmap v15819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:47 smithi096 ceph-mon[32753]: pgmap v15819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:49 smithi194 ceph-mon[33017]: pgmap v15820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29027.143684 secs 2024-04-26T19:54:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:49 smithi033 ceph-mon[27845]: pgmap v15820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29027.143684 secs 2024-04-26T19:54:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:49 smithi096 ceph-mon[32753]: pgmap v15820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29027.143684 secs 2024-04-26T19:54:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29030.056966 secs 2024-04-26T19:54:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29030.056966 secs 2024-04-26T19:54:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29030.056966 secs 2024-04-26T19:54:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:51 smithi194 ceph-mon[33017]: pgmap v15821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:51 smithi033 ceph-mon[27845]: pgmap v15821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:51 smithi096 ceph-mon[32753]: pgmap v15821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:53 smithi194 ceph-mon[33017]: pgmap v15822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:53 smithi033 ceph-mon[27845]: pgmap v15822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:53 smithi096 ceph-mon[32753]: pgmap v15822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:54.067 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:54.066+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:54.066+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:54:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:54:54.066+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:54:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29032.143763 secs 2024-04-26T19:54:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29032.143763 secs 2024-04-26T19:54:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29032.143763 secs 2024-04-26T19:54:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29035.057084 secs 2024-04-26T19:54:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:55 smithi033 ceph-mon[27845]: pgmap v15823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29035.057084 secs 2024-04-26T19:54:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:55 smithi096 ceph-mon[32753]: pgmap v15823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29035.057084 secs 2024-04-26T19:54:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:55 smithi194 ceph-mon[33017]: pgmap v15823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T19:54:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:57 smithi033 ceph-mon[27845]: pgmap v15824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:57 smithi096 ceph-mon[32753]: pgmap v15824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:57 smithi194 ceph-mon[33017]: pgmap v15824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:54:59.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:59 smithi194 ceph-mon[33017]: pgmap v15825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:59.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:54:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29037.143847 secs 2024-04-26T19:54:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:59 smithi033 ceph-mon[27845]: pgmap v15825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:54:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29037.143847 secs 2024-04-26T19:54:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:59 smithi096 ceph-mon[32753]: pgmap v15825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:54:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:54:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29037.143847 secs 2024-04-26T19:55:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29040.057191 secs 2024-04-26T19:55:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29040.057191 secs 2024-04-26T19:55:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29040.057191 secs 2024-04-26T19:55:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:01 smithi033 ceph-mon[27845]: pgmap v15826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:01 smithi096 ceph-mon[32753]: pgmap v15826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:01 smithi194 ceph-mon[33017]: pgmap v15826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:03 smithi033 ceph-mon[27845]: pgmap v15827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:03 smithi096 ceph-mon[32753]: pgmap v15827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:03 smithi194 ceph-mon[33017]: pgmap v15827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29042.143931 secs 2024-04-26T19:55:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29042.143931 secs 2024-04-26T19:55:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29042.143931 secs 2024-04-26T19:55:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29045.057310 secs 2024-04-26T19:55:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:05 smithi033 ceph-mon[27845]: pgmap v15828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29045.057310 secs 2024-04-26T19:55:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:05 smithi096 ceph-mon[32753]: pgmap v15828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29045.057310 secs 2024-04-26T19:55:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:05 smithi194 ceph-mon[33017]: pgmap v15828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:07 smithi033 ceph-mon[27845]: pgmap v15829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:07 smithi096 ceph-mon[32753]: pgmap v15829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:07 smithi194 ceph-mon[33017]: pgmap v15829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:09.069+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:09.069+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:09.069+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:09 smithi033 ceph-mon[27845]: pgmap v15830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29047.144022 secs 2024-04-26T19:55:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:09 smithi096 ceph-mon[32753]: pgmap v15830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29047.144022 secs 2024-04-26T19:55:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:09 smithi194 ceph-mon[33017]: pgmap v15830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29047.144022 secs 2024-04-26T19:55:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29050.057374 secs 2024-04-26T19:55:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29050.057374 secs 2024-04-26T19:55:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29050.057374 secs 2024-04-26T19:55:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:11 smithi096 ceph-mon[32753]: pgmap v15831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:11 smithi194 ceph-mon[33017]: pgmap v15831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:11 smithi033 ceph-mon[27845]: pgmap v15831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:13 smithi096 ceph-mon[32753]: pgmap v15832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:13 smithi194 ceph-mon[33017]: pgmap v15832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:13 smithi033 ceph-mon[27845]: pgmap v15832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29052.144107 secs 2024-04-26T19:55:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29052.144107 secs 2024-04-26T19:55:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29052.144107 secs 2024-04-26T19:55:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29055.057495 secs 2024-04-26T19:55:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:15 smithi194 ceph-mon[33017]: pgmap v15833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29055.057495 secs 2024-04-26T19:55:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:15 smithi033 ceph-mon[27845]: pgmap v15833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29055.057495 secs 2024-04-26T19:55:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:15 smithi096 ceph-mon[32753]: pgmap v15833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:17 smithi194 ceph-mon[33017]: pgmap v15834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:17 smithi033 ceph-mon[27845]: pgmap v15834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:17 smithi096 ceph-mon[32753]: pgmap v15834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:19 smithi194 ceph-mon[33017]: pgmap v15835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29057.144170 secs 2024-04-26T19:55:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:19 smithi033 ceph-mon[27845]: pgmap v15835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29057.144170 secs 2024-04-26T19:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:19 smithi096 ceph-mon[32753]: pgmap v15835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29057.144170 secs 2024-04-26T19:55:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29060.057618 secs 2024-04-26T19:55:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29060.057618 secs 2024-04-26T19:55:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29060.057618 secs 2024-04-26T19:55:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:21 smithi194 ceph-mon[33017]: pgmap v15836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:21 smithi033 ceph-mon[27845]: pgmap v15836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:21 smithi096 ceph-mon[32753]: pgmap v15836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:23 smithi194 ceph-mon[33017]: pgmap v15837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:23 smithi033 ceph-mon[27845]: pgmap v15837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:23 smithi096 ceph-mon[32753]: pgmap v15837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:24.072+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:24.072+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:24.072+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:55:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29062.144256 secs 2024-04-26T19:55:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29062.144256 secs 2024-04-26T19:55:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29062.144256 secs 2024-04-26T19:55:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29065.057738 secs 2024-04-26T19:55:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:25 smithi194 ceph-mon[33017]: pgmap v15838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29065.057738 secs 2024-04-26T19:55:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:25 smithi033 ceph-mon[27845]: pgmap v15838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29065.057738 secs 2024-04-26T19:55:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:25 smithi096 ceph-mon[32753]: pgmap v15838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:27 smithi194 ceph-mon[33017]: pgmap v15839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:27 smithi033 ceph-mon[27845]: pgmap v15839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:27 smithi096 ceph-mon[32753]: pgmap v15839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:28 smithi194 ceph-mon[33017]: pgmap v15840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:55:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:28 smithi033 ceph-mon[27845]: pgmap v15840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:55:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:28 smithi096 ceph-mon[32753]: pgmap v15840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:55:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29067.144317 secs 2024-04-26T19:55:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29067.144317 secs 2024-04-26T19:55:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29067.144317 secs 2024-04-26T19:55:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29070.057832 secs 2024-04-26T19:55:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:55:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:55:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:30 smithi033 ceph-mon[27845]: pgmap v15841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:30.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:30 smithi033 ceph-mon[27845]: pgmap v15842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:55:30.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:55:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29070.057832 secs 2024-04-26T19:55:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:55:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:55:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:30 smithi096 ceph-mon[32753]: pgmap v15841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:30 smithi096 ceph-mon[32753]: pgmap v15842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:55:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:55:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29070.057832 secs 2024-04-26T19:55:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:55:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:55:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:30 smithi194 ceph-mon[33017]: pgmap v15841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:30 smithi194 ceph-mon[33017]: pgmap v15842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T19:55:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:55:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:32 smithi033 ceph-mon[27845]: pgmap v15843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:32 smithi096 ceph-mon[32753]: pgmap v15843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:32 smithi194 ceph-mon[33017]: pgmap v15843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29072.144396 secs 2024-04-26T19:55:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:34 smithi033 ceph-mon[27845]: pgmap v15844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29075.057928 secs 2024-04-26T19:55:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29072.144396 secs 2024-04-26T19:55:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:34 smithi096 ceph-mon[32753]: pgmap v15844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29075.057928 secs 2024-04-26T19:55:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29072.144396 secs 2024-04-26T19:55:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:34 smithi194 ceph-mon[33017]: pgmap v15844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29075.057928 secs 2024-04-26T19:55:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:36 smithi096 ceph-mon[32753]: pgmap v15845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:36 smithi194 ceph-mon[33017]: pgmap v15845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:36 smithi033 ceph-mon[27845]: pgmap v15845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:38 smithi194 ceph-mon[33017]: pgmap v15846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:38 smithi033 ceph-mon[27845]: pgmap v15846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:38 smithi096 ceph-mon[32753]: pgmap v15846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:39.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:39.080+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:39.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:39.080+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:39.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:39.080+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29077.144478 secs 2024-04-26T19:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29080.058024 secs 2024-04-26T19:55:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29077.144478 secs 2024-04-26T19:55:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29080.058024 secs 2024-04-26T19:55:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29077.144478 secs 2024-04-26T19:55:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29080.058024 secs 2024-04-26T19:55:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:40 smithi194 ceph-mon[33017]: pgmap v15847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:55:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:40 smithi033 ceph-mon[27845]: pgmap v15847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:40 smithi096 ceph-mon[32753]: pgmap v15847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T19:55:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:42 smithi194 ceph-mon[33017]: pgmap v15848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:42 smithi033 ceph-mon[27845]: pgmap v15848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:42 smithi096 ceph-mon[32753]: pgmap v15848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29082.144558 secs 2024-04-26T19:55:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:44 smithi194 ceph-mon[33017]: pgmap v15849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29085.058169 secs 2024-04-26T19:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29082.144558 secs 2024-04-26T19:55:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:44 smithi033 ceph-mon[27845]: pgmap v15849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29085.058169 secs 2024-04-26T19:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29082.144558 secs 2024-04-26T19:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:44 smithi096 ceph-mon[32753]: pgmap v15849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29085.058169 secs 2024-04-26T19:55:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:46 smithi194 ceph-mon[33017]: pgmap v15850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:46 smithi033 ceph-mon[27845]: pgmap v15850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:46 smithi096 ceph-mon[32753]: pgmap v15850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:48 smithi194 ceph-mon[33017]: pgmap v15851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:48 smithi033 ceph-mon[27845]: pgmap v15851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:48 smithi096 ceph-mon[32753]: pgmap v15851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29087.144643 secs 2024-04-26T19:55:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29090.058243 secs 2024-04-26T19:55:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29087.144643 secs 2024-04-26T19:55:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29090.058243 secs 2024-04-26T19:55:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29087.144643 secs 2024-04-26T19:55:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29090.058243 secs 2024-04-26T19:55:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:50 smithi194 ceph-mon[33017]: pgmap v15852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:50 smithi033 ceph-mon[27845]: pgmap v15852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:50 smithi096 ceph-mon[32753]: pgmap v15852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:55:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:52 smithi194 ceph-mon[33017]: pgmap v15853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:55:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:52 smithi033 ceph-mon[27845]: pgmap v15853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:55:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:52 smithi096 ceph-mon[32753]: pgmap v15853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:55:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:54.083+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:54.083+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:55:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:55:54.083+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:55:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29092.144731 secs 2024-04-26T19:55:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:54 smithi194 ceph-mon[33017]: pgmap v15854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29095.058356 secs 2024-04-26T19:55:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29092.144731 secs 2024-04-26T19:55:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:54 smithi033 ceph-mon[27845]: pgmap v15854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29095.058356 secs 2024-04-26T19:55:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29092.144731 secs 2024-04-26T19:55:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:54 smithi096 ceph-mon[32753]: pgmap v15854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29095.058356 secs 2024-04-26T19:55:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:56 smithi194 ceph-mon[33017]: pgmap v15855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:56 smithi033 ceph-mon[27845]: pgmap v15855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:56 smithi096 ceph-mon[32753]: pgmap v15855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:55:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:58 smithi194 ceph-mon[33017]: pgmap v15856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:58 smithi033 ceph-mon[27845]: pgmap v15856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:58 smithi096 ceph-mon[32753]: pgmap v15856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:55:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29097.144793 secs 2024-04-26T19:55:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:55:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29100.058445 secs 2024-04-26T19:55:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29097.144793 secs 2024-04-26T19:55:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:55:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29100.058445 secs 2024-04-26T19:55:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29097.144793 secs 2024-04-26T19:55:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:55:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29100.058445 secs 2024-04-26T19:56:00.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:00 smithi033 ceph-mon[27845]: pgmap v15857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:00 smithi096 ceph-mon[32753]: pgmap v15857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:00 smithi194 ceph-mon[33017]: pgmap v15857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:02 smithi194 ceph-mon[33017]: pgmap v15858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:02 smithi033 ceph-mon[27845]: pgmap v15858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:02 smithi096 ceph-mon[32753]: pgmap v15858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29102.144875 secs 2024-04-26T19:56:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:04 smithi194 ceph-mon[33017]: pgmap v15859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29105.058559 secs 2024-04-26T19:56:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29102.144875 secs 2024-04-26T19:56:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:04 smithi033 ceph-mon[27845]: pgmap v15859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29105.058559 secs 2024-04-26T19:56:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29102.144875 secs 2024-04-26T19:56:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:04 smithi096 ceph-mon[32753]: pgmap v15859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29105.058559 secs 2024-04-26T19:56:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:06 smithi194 ceph-mon[33017]: pgmap v15860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:06 smithi033 ceph-mon[27845]: pgmap v15860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:06 smithi096 ceph-mon[32753]: pgmap v15860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:08 smithi194 ceph-mon[33017]: pgmap v15861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:08 smithi033 ceph-mon[27845]: pgmap v15861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:08 smithi096 ceph-mon[32753]: pgmap v15861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:09.093+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:09.093+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:56:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:09.093+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29107.144959 secs 2024-04-26T19:56:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29110.058723 secs 2024-04-26T19:56:09.904 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29107.144959 secs 2024-04-26T19:56:09.904 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29110.058723 secs 2024-04-26T19:56:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29107.144959 secs 2024-04-26T19:56:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29110.058723 secs 2024-04-26T19:56:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:10 smithi194 ceph-mon[33017]: pgmap v15862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:10 smithi033 ceph-mon[27845]: pgmap v15862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:10 smithi096 ceph-mon[32753]: pgmap v15862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:12 smithi194 ceph-mon[33017]: pgmap v15863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:12 smithi033 ceph-mon[27845]: pgmap v15863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:12 smithi096 ceph-mon[32753]: pgmap v15863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29112.145041 secs 2024-04-26T19:56:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:14 smithi194 ceph-mon[33017]: pgmap v15864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29115.058844 secs 2024-04-26T19:56:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29112.145041 secs 2024-04-26T19:56:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:14 smithi033 ceph-mon[27845]: pgmap v15864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29115.058844 secs 2024-04-26T19:56:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29112.145041 secs 2024-04-26T19:56:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:14 smithi096 ceph-mon[32753]: pgmap v15864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29115.058844 secs 2024-04-26T19:56:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:16 smithi194 ceph-mon[33017]: pgmap v15865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:16 smithi033 ceph-mon[27845]: pgmap v15865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:16 smithi096 ceph-mon[32753]: pgmap v15865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:18 smithi194 ceph-mon[33017]: pgmap v15866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:18 smithi033 ceph-mon[27845]: pgmap v15866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:18 smithi096 ceph-mon[32753]: pgmap v15866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29117.145132 secs 2024-04-26T19:56:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29120.058971 secs 2024-04-26T19:56:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29117.145132 secs 2024-04-26T19:56:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29120.058971 secs 2024-04-26T19:56:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29117.145132 secs 2024-04-26T19:56:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29120.058971 secs 2024-04-26T19:56:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:20 smithi194 ceph-mon[33017]: pgmap v15867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:20 smithi033 ceph-mon[27845]: pgmap v15867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:20 smithi096 ceph-mon[32753]: pgmap v15867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T19:56:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T19:56:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef007f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T19:56:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef007f040 comp rx=0 tx=0).stop 2024-04-26T19:56:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T19:56:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T19:56:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:56:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.797+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:56:21.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.798+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:56:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:21.799+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0062070 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:56:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:22 smithi194 ceph-mon[33017]: pgmap v15868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:22 smithi033 ceph-mon[27845]: pgmap v15868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:22 smithi096 ceph-mon[32753]: pgmap v15868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:24.095+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:24.095+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:56:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:24.095+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29122.145212 secs 2024-04-26T19:56:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:24 smithi194 ceph-mon[33017]: pgmap v15869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29125.059066 secs 2024-04-26T19:56:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29122.145212 secs 2024-04-26T19:56:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:24 smithi033 ceph-mon[27845]: pgmap v15869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29125.059066 secs 2024-04-26T19:56:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29122.145212 secs 2024-04-26T19:56:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:24 smithi096 ceph-mon[32753]: pgmap v15869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29125.059066 secs 2024-04-26T19:56:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.752+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T19:56:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.752+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T19:56:26.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.752+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 229 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T19:56:26.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.752+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 229 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T19:56:26.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.752+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x7fef00077780 2024-04-26T19:56:26.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.752+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d00e0 con 0x564385821900 2024-04-26T19:56:26.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.753+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 230 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d00e0 con 0x7fef00077780 2024-04-26T19:56:26.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:26.753+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 230 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T19:56:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:26 smithi194 ceph-mon[33017]: pgmap v15870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:26 smithi033 ceph-mon[27845]: pgmap v15870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:26 smithi096 ceph-mon[32753]: pgmap v15870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:28 smithi194 ceph-mon[33017]: pgmap v15871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:28 smithi033 ceph-mon[27845]: pgmap v15871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:28 smithi096 ceph-mon[32753]: pgmap v15871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29127.145297 secs 2024-04-26T19:56:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29130.059195 secs 2024-04-26T19:56:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29127.145297 secs 2024-04-26T19:56:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29130.059195 secs 2024-04-26T19:56:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29127.145297 secs 2024-04-26T19:56:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29130.059195 secs 2024-04-26T19:56:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:30 smithi033 ceph-mon[27845]: pgmap v15872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:56:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:30 smithi096 ceph-mon[32753]: pgmap v15872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:56:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:30 smithi194 ceph-mon[33017]: pgmap v15872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:56:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:56:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:56:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:31 smithi033 ceph-mon[27845]: pgmap v15873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:31 smithi033 ceph-mon[27845]: pgmap v15874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:56:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:56:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:56:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:31 smithi096 ceph-mon[32753]: pgmap v15873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:31 smithi096 ceph-mon[32753]: pgmap v15874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:56:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:56:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:56:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:31 smithi194 ceph-mon[33017]: pgmap v15873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T19:56:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:31 smithi194 ceph-mon[33017]: pgmap v15874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:56:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29132.145378 secs 2024-04-26T19:56:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:34 smithi033 ceph-mon[27845]: pgmap v15875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29132.145378 secs 2024-04-26T19:56:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:34 smithi096 ceph-mon[32753]: pgmap v15875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29132.145378 secs 2024-04-26T19:56:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:34 smithi194 ceph-mon[33017]: pgmap v15875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29135.059314 secs 2024-04-26T19:56:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29135.059314 secs 2024-04-26T19:56:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29135.059314 secs 2024-04-26T19:56:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:36 smithi033 ceph-mon[27845]: pgmap v15876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:36 smithi096 ceph-mon[32753]: pgmap v15876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:36 smithi194 ceph-mon[33017]: pgmap v15876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:38 smithi033 ceph-mon[27845]: pgmap v15877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:38 smithi096 ceph-mon[32753]: pgmap v15877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:38 smithi194 ceph-mon[33017]: pgmap v15877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:56:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:39.107+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:39.107+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:39.107+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:56:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29137.145442 secs 2024-04-26T19:56:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29137.145442 secs 2024-04-26T19:56:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29137.145442 secs 2024-04-26T19:56:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29140.059434 secs 2024-04-26T19:56:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:40 smithi033 ceph-mon[27845]: pgmap v15878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:56:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29140.059434 secs 2024-04-26T19:56:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:40 smithi096 ceph-mon[32753]: pgmap v15878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:56:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29140.059434 secs 2024-04-26T19:56:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:40 smithi194 ceph-mon[33017]: pgmap v15878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T19:56:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:42 smithi096 ceph-mon[32753]: pgmap v15879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:56:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:42 smithi194 ceph-mon[33017]: pgmap v15879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:56:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:42 smithi033 ceph-mon[27845]: pgmap v15879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:56:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29142.145499 secs 2024-04-26T19:56:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:44 smithi194 ceph-mon[33017]: pgmap v15880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29142.145499 secs 2024-04-26T19:56:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:44 smithi033 ceph-mon[27845]: pgmap v15880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29142.145499 secs 2024-04-26T19:56:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:44 smithi096 ceph-mon[32753]: pgmap v15880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29145.059531 secs 2024-04-26T19:56:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29145.059531 secs 2024-04-26T19:56:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29145.059531 secs 2024-04-26T19:56:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:46 smithi194 ceph-mon[33017]: pgmap v15881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:46 smithi033 ceph-mon[27845]: pgmap v15881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:46 smithi096 ceph-mon[32753]: pgmap v15881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:48 smithi194 ceph-mon[33017]: pgmap v15882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:48 smithi033 ceph-mon[27845]: pgmap v15882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:48 smithi096 ceph-mon[32753]: pgmap v15882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29147.145583 secs 2024-04-26T19:56:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29147.145583 secs 2024-04-26T19:56:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29147.145583 secs 2024-04-26T19:56:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29150.059668 secs 2024-04-26T19:56:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:50 smithi194 ceph-mon[33017]: pgmap v15883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29150.059668 secs 2024-04-26T19:56:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:50 smithi033 ceph-mon[27845]: pgmap v15883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29150.059668 secs 2024-04-26T19:56:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:50 smithi096 ceph-mon[32753]: pgmap v15883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:52 smithi194 ceph-mon[33017]: pgmap v15884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:52 smithi033 ceph-mon[27845]: pgmap v15884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:52 smithi096 ceph-mon[32753]: pgmap v15884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:54.123+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:54.123+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:56:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:56:54.123+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:56:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29152.145665 secs 2024-04-26T19:56:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:54 smithi194 ceph-mon[33017]: pgmap v15885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29152.145665 secs 2024-04-26T19:56:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:54 smithi033 ceph-mon[27845]: pgmap v15885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29152.145665 secs 2024-04-26T19:56:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:54 smithi096 ceph-mon[32753]: pgmap v15885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29155.059791 secs 2024-04-26T19:56:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29155.059791 secs 2024-04-26T19:56:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29155.059791 secs 2024-04-26T19:56:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:56 smithi194 ceph-mon[33017]: pgmap v15886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:56 smithi033 ceph-mon[27845]: pgmap v15886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:56 smithi096 ceph-mon[32753]: pgmap v15886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:56:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:58 smithi194 ceph-mon[33017]: pgmap v15887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:58 smithi033 ceph-mon[27845]: pgmap v15887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:58 smithi096 ceph-mon[32753]: pgmap v15887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:56:59.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:56:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29157.145727 secs 2024-04-26T19:56:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:56:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29157.145727 secs 2024-04-26T19:56:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:56:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29157.145727 secs 2024-04-26T19:57:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29160.059912 secs 2024-04-26T19:57:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:00 smithi033 ceph-mon[27845]: pgmap v15888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29160.059912 secs 2024-04-26T19:57:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:00 smithi096 ceph-mon[32753]: pgmap v15888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29160.059912 secs 2024-04-26T19:57:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:00 smithi194 ceph-mon[33017]: pgmap v15888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:02 smithi194 ceph-mon[33017]: pgmap v15889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:02 smithi033 ceph-mon[27845]: pgmap v15889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:02 smithi096 ceph-mon[32753]: pgmap v15889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:03 smithi194 ceph-mon[33017]: pgmap v15890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:03 smithi033 ceph-mon[27845]: pgmap v15890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:03 smithi096 ceph-mon[32753]: pgmap v15890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29162.145810 secs 2024-04-26T19:57:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29162.145810 secs 2024-04-26T19:57:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29162.145810 secs 2024-04-26T19:57:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29165.060029 secs 2024-04-26T19:57:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:05 smithi194 ceph-mon[33017]: pgmap v15891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29165.060029 secs 2024-04-26T19:57:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:05 smithi033 ceph-mon[27845]: pgmap v15891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29165.060029 secs 2024-04-26T19:57:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:05 smithi096 ceph-mon[32753]: pgmap v15891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:08 smithi033 ceph-mon[27845]: pgmap v15892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:08 smithi096 ceph-mon[32753]: pgmap v15892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:08 smithi194 ceph-mon[33017]: pgmap v15892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:09.133+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:09.134+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:09.134+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:57:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29167.145894 secs 2024-04-26T19:57:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29167.145894 secs 2024-04-26T19:57:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29167.145894 secs 2024-04-26T19:57:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29170.060173 secs 2024-04-26T19:57:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:10 smithi033 ceph-mon[27845]: pgmap v15893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29170.060173 secs 2024-04-26T19:57:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:10 smithi096 ceph-mon[32753]: pgmap v15893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29170.060173 secs 2024-04-26T19:57:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:10 smithi194 ceph-mon[33017]: pgmap v15893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:12 smithi096 ceph-mon[32753]: pgmap v15894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:12 smithi194 ceph-mon[33017]: pgmap v15894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:12 smithi033 ceph-mon[27845]: pgmap v15894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29172.145979 secs 2024-04-26T19:57:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:14 smithi096 ceph-mon[32753]: pgmap v15895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29172.145979 secs 2024-04-26T19:57:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:14 smithi194 ceph-mon[33017]: pgmap v15895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29172.145979 secs 2024-04-26T19:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:14 smithi033 ceph-mon[27845]: pgmap v15895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29175.060270 secs 2024-04-26T19:57:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29175.060270 secs 2024-04-26T19:57:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29175.060270 secs 2024-04-26T19:57:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:16 smithi194 ceph-mon[33017]: pgmap v15896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:16 smithi033 ceph-mon[27845]: pgmap v15896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:16 smithi096 ceph-mon[32753]: pgmap v15896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:18 smithi194 ceph-mon[33017]: pgmap v15897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:18 smithi033 ceph-mon[27845]: pgmap v15897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:18 smithi096 ceph-mon[32753]: pgmap v15897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29177.146071 secs 2024-04-26T19:57:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29177.146071 secs 2024-04-26T19:57:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29177.146071 secs 2024-04-26T19:57:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29180.060392 secs 2024-04-26T19:57:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:20 smithi194 ceph-mon[33017]: pgmap v15898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29180.060392 secs 2024-04-26T19:57:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:20 smithi033 ceph-mon[27845]: pgmap v15898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29180.060392 secs 2024-04-26T19:57:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:20 smithi096 ceph-mon[32753]: pgmap v15898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:22 smithi194 ceph-mon[33017]: pgmap v15899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:22 smithi033 ceph-mon[27845]: pgmap v15899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:22 smithi096 ceph-mon[32753]: pgmap v15899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:24.149+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:24.149+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:24.149+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:57:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29182.146130 secs 2024-04-26T19:57:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:24 smithi194 ceph-mon[33017]: pgmap v15900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29182.146130 secs 2024-04-26T19:57:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:24 smithi033 ceph-mon[27845]: pgmap v15900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29182.146130 secs 2024-04-26T19:57:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:24 smithi096 ceph-mon[32753]: pgmap v15900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29185.060527 secs 2024-04-26T19:57:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29185.060527 secs 2024-04-26T19:57:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29185.060527 secs 2024-04-26T19:57:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:26 smithi194 ceph-mon[33017]: pgmap v15901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:26 smithi033 ceph-mon[27845]: pgmap v15901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:26 smithi096 ceph-mon[32753]: pgmap v15901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:28 smithi033 ceph-mon[27845]: pgmap v15902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:28 smithi096 ceph-mon[32753]: pgmap v15902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:28 smithi194 ceph-mon[33017]: pgmap v15902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29187.146215 secs 2024-04-26T19:57:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29187.146215 secs 2024-04-26T19:57:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29187.146215 secs 2024-04-26T19:57:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29190.060619 secs 2024-04-26T19:57:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:30 smithi096 ceph-mon[32753]: pgmap v15903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29190.060619 secs 2024-04-26T19:57:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:30 smithi033 ceph-mon[27845]: pgmap v15903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29190.060619 secs 2024-04-26T19:57:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:30 smithi194 ceph-mon[33017]: pgmap v15903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:32 smithi194 ceph-mon[33017]: pgmap v15904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:57:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:32 smithi033 ceph-mon[27845]: pgmap v15904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:57:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:32 smithi096 ceph-mon[32753]: pgmap v15904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:57:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:57:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:57:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:57:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:57:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:57:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:57:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:57:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:57:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:57:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:34 smithi194 ceph-mon[33017]: pgmap v15905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:34 smithi194 ceph-mon[33017]: pgmap v15906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29192.146299 secs 2024-04-26T19:57:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:34 smithi033 ceph-mon[27845]: pgmap v15905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:34 smithi033 ceph-mon[27845]: pgmap v15906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29192.146299 secs 2024-04-26T19:57:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:34 smithi096 ceph-mon[32753]: pgmap v15905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:34 smithi096 ceph-mon[32753]: pgmap v15906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29192.146299 secs 2024-04-26T19:57:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29195.060737 secs 2024-04-26T19:57:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29195.060737 secs 2024-04-26T19:57:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29195.060737 secs 2024-04-26T19:57:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:36 smithi194 ceph-mon[33017]: pgmap v15907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:36 smithi033 ceph-mon[27845]: pgmap v15907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:36 smithi096 ceph-mon[32753]: pgmap v15907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:38 smithi194 ceph-mon[33017]: pgmap v15908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:38 smithi033 ceph-mon[27845]: pgmap v15908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:38 smithi096 ceph-mon[32753]: pgmap v15908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:39.149+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:39.149+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:39.149+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:57:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29197.146363 secs 2024-04-26T19:57:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29197.146363 secs 2024-04-26T19:57:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29197.146363 secs 2024-04-26T19:57:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29200.060854 secs 2024-04-26T19:57:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:40 smithi194 ceph-mon[33017]: pgmap v15909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29200.060854 secs 2024-04-26T19:57:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:40 smithi033 ceph-mon[27845]: pgmap v15909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29200.060854 secs 2024-04-26T19:57:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:40 smithi096 ceph-mon[32753]: pgmap v15909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:42 smithi194 ceph-mon[33017]: pgmap v15910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:42 smithi033 ceph-mon[27845]: pgmap v15910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:42 smithi096 ceph-mon[32753]: pgmap v15910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:57:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:44 smithi194 ceph-mon[33017]: pgmap v15911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:57:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29202.146443 secs 2024-04-26T19:57:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:44 smithi033 ceph-mon[27845]: pgmap v15911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:57:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29202.146443 secs 2024-04-26T19:57:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:44 smithi096 ceph-mon[32753]: pgmap v15911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:57:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29202.146443 secs 2024-04-26T19:57:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29205.060971 secs 2024-04-26T19:57:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29205.060971 secs 2024-04-26T19:57:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29205.060971 secs 2024-04-26T19:57:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:46 smithi194 ceph-mon[33017]: pgmap v15912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:46 smithi033 ceph-mon[27845]: pgmap v15912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:46 smithi096 ceph-mon[32753]: pgmap v15912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:48 smithi194 ceph-mon[33017]: pgmap v15913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:48 smithi033 ceph-mon[27845]: pgmap v15913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:48 smithi096 ceph-mon[32753]: pgmap v15913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29207.146530 secs 2024-04-26T19:57:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:49 smithi194 ceph-mon[33017]: pgmap v15914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29207.146530 secs 2024-04-26T19:57:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:49 smithi033 ceph-mon[27845]: pgmap v15914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29207.146530 secs 2024-04-26T19:57:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:49 smithi096 ceph-mon[32753]: pgmap v15914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29210.061124 secs 2024-04-26T19:57:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29210.061124 secs 2024-04-26T19:57:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29210.061124 secs 2024-04-26T19:57:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:51 smithi194 ceph-mon[33017]: pgmap v15915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:51 smithi033 ceph-mon[27845]: pgmap v15915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:51 smithi096 ceph-mon[32753]: pgmap v15915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:53 smithi194 ceph-mon[33017]: pgmap v15916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29212.146616 secs 2024-04-26T19:57:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:54.164+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:54.164+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:57:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:57:54.164+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:57:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:53 smithi033 ceph-mon[27845]: pgmap v15916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29212.146616 secs 2024-04-26T19:57:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:53 smithi096 ceph-mon[32753]: pgmap v15916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29212.146616 secs 2024-04-26T19:57:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29215.061223 secs 2024-04-26T19:57:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29215.061223 secs 2024-04-26T19:57:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29215.061223 secs 2024-04-26T19:57:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:55 smithi194 ceph-mon[33017]: pgmap v15917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:55 smithi033 ceph-mon[27845]: pgmap v15917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:55 smithi096 ceph-mon[32753]: pgmap v15917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:57:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:57 smithi194 ceph-mon[33017]: pgmap v15918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:57 smithi033 ceph-mon[27845]: pgmap v15918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:57 smithi096 ceph-mon[32753]: pgmap v15918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29217.146702 secs 2024-04-26T19:57:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29217.146702 secs 2024-04-26T19:57:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29217.146702 secs 2024-04-26T19:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29220.061344 secs 2024-04-26T19:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:57:59 smithi033 ceph-mon[27845]: pgmap v15919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29220.061344 secs 2024-04-26T19:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:57:59 smithi096 ceph-mon[32753]: pgmap v15919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29220.061344 secs 2024-04-26T19:58:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:57:59 smithi194 ceph-mon[33017]: pgmap v15919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:01 smithi033 ceph-mon[27845]: pgmap v15920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:01 smithi096 ceph-mon[32753]: pgmap v15920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:01 smithi194 ceph-mon[33017]: pgmap v15920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:03 smithi033 ceph-mon[27845]: pgmap v15921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29222.146786 secs 2024-04-26T19:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:03 smithi096 ceph-mon[32753]: pgmap v15921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29222.146786 secs 2024-04-26T19:58:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:03 smithi194 ceph-mon[33017]: pgmap v15921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29222.146786 secs 2024-04-26T19:58:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29225.061465 secs 2024-04-26T19:58:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29225.061465 secs 2024-04-26T19:58:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29225.061465 secs 2024-04-26T19:58:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:05 smithi033 ceph-mon[27845]: pgmap v15922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:05 smithi096 ceph-mon[32753]: pgmap v15922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:05 smithi194 ceph-mon[33017]: pgmap v15922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:07 smithi033 ceph-mon[27845]: pgmap v15923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:07 smithi096 ceph-mon[32753]: pgmap v15923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:07 smithi194 ceph-mon[33017]: pgmap v15923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29227.146847 secs 2024-04-26T19:58:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:09.180+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:09.180+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:58:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:09.180+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29227.146847 secs 2024-04-26T19:58:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29227.146847 secs 2024-04-26T19:58:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29230.061583 secs 2024-04-26T19:58:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:09 smithi033 ceph-mon[27845]: pgmap v15924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29230.061583 secs 2024-04-26T19:58:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:09 smithi096 ceph-mon[32753]: pgmap v15924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29230.061583 secs 2024-04-26T19:58:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:09 smithi194 ceph-mon[33017]: pgmap v15924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:11 smithi033 ceph-mon[27845]: pgmap v15925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:11 smithi096 ceph-mon[32753]: pgmap v15925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:11 smithi194 ceph-mon[33017]: pgmap v15925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:13 smithi033 ceph-mon[27845]: pgmap v15926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29232.146931 secs 2024-04-26T19:58:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:13 smithi096 ceph-mon[32753]: pgmap v15926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29232.146931 secs 2024-04-26T19:58:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:13 smithi194 ceph-mon[33017]: pgmap v15926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29232.146931 secs 2024-04-26T19:58:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29235.061716 secs 2024-04-26T19:58:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29235.061716 secs 2024-04-26T19:58:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29235.061716 secs 2024-04-26T19:58:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:15 smithi096 ceph-mon[32753]: pgmap v15927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:15 smithi194 ceph-mon[33017]: pgmap v15927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:15 smithi033 ceph-mon[27845]: pgmap v15927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:17 smithi096 ceph-mon[32753]: pgmap v15928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:17 smithi194 ceph-mon[33017]: pgmap v15928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:17 smithi033 ceph-mon[27845]: pgmap v15928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29237.147021 secs 2024-04-26T19:58:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29237.147021 secs 2024-04-26T19:58:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29237.147021 secs 2024-04-26T19:58:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29240.061831 secs 2024-04-26T19:58:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:19 smithi194 ceph-mon[33017]: pgmap v15929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29240.061831 secs 2024-04-26T19:58:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:19 smithi033 ceph-mon[27845]: pgmap v15929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29240.061831 secs 2024-04-26T19:58:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:19 smithi096 ceph-mon[32753]: pgmap v15929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:21 smithi194 ceph-mon[33017]: pgmap v15930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:21 smithi033 ceph-mon[27845]: pgmap v15930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:21 smithi096 ceph-mon[32753]: pgmap v15930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:24.196+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:24.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:24.196+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:58:24.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:24.196+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:23 smithi194 ceph-mon[33017]: pgmap v15931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29242.147080 secs 2024-04-26T19:58:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:23 smithi033 ceph-mon[27845]: pgmap v15931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29242.147080 secs 2024-04-26T19:58:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:23 smithi096 ceph-mon[32753]: pgmap v15931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29242.147080 secs 2024-04-26T19:58:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29245.061927 secs 2024-04-26T19:58:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29245.061927 secs 2024-04-26T19:58:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29245.061927 secs 2024-04-26T19:58:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:25 smithi194 ceph-mon[33017]: pgmap v15932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:25 smithi033 ceph-mon[27845]: pgmap v15932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:25 smithi096 ceph-mon[32753]: pgmap v15932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:27 smithi194 ceph-mon[33017]: pgmap v15933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:27 smithi033 ceph-mon[27845]: pgmap v15933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:27 smithi096 ceph-mon[32753]: pgmap v15933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:29.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29247.147166 secs 2024-04-26T19:58:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29247.147166 secs 2024-04-26T19:58:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29247.147166 secs 2024-04-26T19:58:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29250.062050 secs 2024-04-26T19:58:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:29 smithi033 ceph-mon[27845]: pgmap v15934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29250.062050 secs 2024-04-26T19:58:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:29 smithi096 ceph-mon[32753]: pgmap v15934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29250.062050 secs 2024-04-26T19:58:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:29 smithi194 ceph-mon[33017]: pgmap v15934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:31 smithi194 ceph-mon[33017]: pgmap v15935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:31 smithi033 ceph-mon[27845]: pgmap v15935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:31 smithi096 ceph-mon[32753]: pgmap v15935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:33 smithi194 ceph-mon[33017]: pgmap v15936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29252.147247 secs 2024-04-26T19:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:33 smithi033 ceph-mon[27845]: pgmap v15936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29252.147247 secs 2024-04-26T19:58:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:58:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:33 smithi096 ceph-mon[32753]: pgmap v15936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29252.147247 secs 2024-04-26T19:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29255.062143 secs 2024-04-26T19:58:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29255.062143 secs 2024-04-26T19:58:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29255.062143 secs 2024-04-26T19:58:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:36 smithi194 ceph-mon[33017]: pgmap v15937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:36.109 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:36 smithi096 ceph-mon[32753]: pgmap v15937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:36.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:36 smithi033 ceph-mon[27845]: pgmap v15937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:38.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:38 smithi033 ceph-mon[27845]: pgmap v15938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:38.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.274 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.274 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:38 smithi194 ceph-mon[33017]: pgmap v15938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:38 smithi096 ceph-mon[32753]: pgmap v15938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:39.211+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:39.211+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:39.211+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:58:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29257.147340 secs 2024-04-26T19:58:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29257.147340 secs 2024-04-26T19:58:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29257.147340 secs 2024-04-26T19:58:40.314 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29260.062216 secs 2024-04-26T19:58:40.314 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:40 smithi033 ceph-mon[27845]: pgmap v15939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29260.062216 secs 2024-04-26T19:58:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:40 smithi194 ceph-mon[33017]: pgmap v15939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29260.062216 secs 2024-04-26T19:58:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:40 smithi096 ceph-mon[32753]: pgmap v15939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: pgmap v15940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:58:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:58:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: pgmap v15941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:58:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: pgmap v15942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: pgmap v15940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: pgmap v15941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: pgmap v15942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: pgmap v15940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:58:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:58:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: pgmap v15941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T19:58:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: pgmap v15942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:58:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29262.147422 secs 2024-04-26T19:58:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:44 smithi194 ceph-mon[33017]: pgmap v15943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:58:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29262.147422 secs 2024-04-26T19:58:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:44 smithi033 ceph-mon[27845]: pgmap v15943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:58:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29262.147422 secs 2024-04-26T19:58:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:44 smithi096 ceph-mon[32753]: pgmap v15943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:58:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29265.062332 secs 2024-04-26T19:58:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29265.062332 secs 2024-04-26T19:58:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29265.062332 secs 2024-04-26T19:58:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:46 smithi033 ceph-mon[27845]: pgmap v15944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:46 smithi194 ceph-mon[33017]: pgmap v15944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:46 smithi096 ceph-mon[32753]: pgmap v15944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:48 smithi033 ceph-mon[27845]: pgmap v15945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:48 smithi096 ceph-mon[32753]: pgmap v15945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:48 smithi194 ceph-mon[33017]: pgmap v15945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T19:58:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29267.147511 secs 2024-04-26T19:58:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29267.147511 secs 2024-04-26T19:58:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29267.147511 secs 2024-04-26T19:58:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29270.062453 secs 2024-04-26T19:58:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:50 smithi194 ceph-mon[33017]: pgmap v15946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:58:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29270.062453 secs 2024-04-26T19:58:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:50 smithi033 ceph-mon[27845]: pgmap v15946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:58:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29270.062453 secs 2024-04-26T19:58:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:50 smithi096 ceph-mon[32753]: pgmap v15946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T19:58:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:51 smithi194 ceph-mon[33017]: pgmap v15947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:58:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:51 smithi033 ceph-mon[27845]: pgmap v15947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:58:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:51 smithi096 ceph-mon[32753]: pgmap v15947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:58:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:54.226+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:54.226+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:58:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:58:54.226+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:58:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29272.147566 secs 2024-04-26T19:58:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:54 smithi194 ceph-mon[33017]: pgmap v15948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29272.147566 secs 2024-04-26T19:58:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:54 smithi033 ceph-mon[27845]: pgmap v15948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29272.147566 secs 2024-04-26T19:58:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:54 smithi096 ceph-mon[32753]: pgmap v15948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29275.062574 secs 2024-04-26T19:58:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29275.062574 secs 2024-04-26T19:58:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29275.062574 secs 2024-04-26T19:58:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:56 smithi194 ceph-mon[33017]: pgmap v15949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:56 smithi033 ceph-mon[27845]: pgmap v15949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:56 smithi096 ceph-mon[32753]: pgmap v15949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:58:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:58 smithi194 ceph-mon[33017]: pgmap v15950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:58 smithi033 ceph-mon[27845]: pgmap v15950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:58 smithi096 ceph-mon[32753]: pgmap v15950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:58:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:58:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29277.147651 secs 2024-04-26T19:58:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:58:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29277.147651 secs 2024-04-26T19:58:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:58:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29277.147651 secs 2024-04-26T19:59:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29280.062733 secs 2024-04-26T19:59:00.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:00 smithi033 ceph-mon[27845]: pgmap v15951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:00.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29280.062733 secs 2024-04-26T19:59:00.657 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:00 smithi194 ceph-mon[33017]: pgmap v15951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:00.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29280.062733 secs 2024-04-26T19:59:00.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:00 smithi096 ceph-mon[32753]: pgmap v15951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:01 smithi194 ceph-mon[33017]: pgmap v15952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:01 smithi033 ceph-mon[27845]: pgmap v15952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:01 smithi096 ceph-mon[32753]: pgmap v15952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29282.147732 secs 2024-04-26T19:59:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:04 smithi194 ceph-mon[33017]: pgmap v15953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29282.147732 secs 2024-04-26T19:59:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:04 smithi033 ceph-mon[27845]: pgmap v15953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29282.147732 secs 2024-04-26T19:59:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:04 smithi096 ceph-mon[32753]: pgmap v15953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29285.062855 secs 2024-04-26T19:59:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29285.062855 secs 2024-04-26T19:59:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29285.062855 secs 2024-04-26T19:59:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:06 smithi194 ceph-mon[33017]: pgmap v15954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:06 smithi033 ceph-mon[27845]: pgmap v15954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:06 smithi096 ceph-mon[32753]: pgmap v15954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:08 smithi194 ceph-mon[33017]: pgmap v15955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:08 smithi033 ceph-mon[27845]: pgmap v15955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:08 smithi096 ceph-mon[32753]: pgmap v15955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:09.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:09.243+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:09.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:09.243+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:09.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:09.243+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:59:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29287.147818 secs 2024-04-26T19:59:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29287.147818 secs 2024-04-26T19:59:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29287.147818 secs 2024-04-26T19:59:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29290.062989 secs 2024-04-26T19:59:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:10 smithi194 ceph-mon[33017]: pgmap v15956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29290.062989 secs 2024-04-26T19:59:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:10 smithi033 ceph-mon[27845]: pgmap v15956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29290.062989 secs 2024-04-26T19:59:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:10 smithi096 ceph-mon[32753]: pgmap v15956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:11 smithi194 ceph-mon[33017]: pgmap v15957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:11 smithi033 ceph-mon[27845]: pgmap v15957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:11 smithi096 ceph-mon[32753]: pgmap v15957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29292.147903 secs 2024-04-26T19:59:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:14 smithi194 ceph-mon[33017]: pgmap v15958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29292.147903 secs 2024-04-26T19:59:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:14 smithi033 ceph-mon[27845]: pgmap v15958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29292.147903 secs 2024-04-26T19:59:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:14 smithi096 ceph-mon[32753]: pgmap v15958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29295.063062 secs 2024-04-26T19:59:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29295.063062 secs 2024-04-26T19:59:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29295.063062 secs 2024-04-26T19:59:16.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:16 smithi033 ceph-mon[27845]: pgmap v15959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:16 smithi194 ceph-mon[33017]: pgmap v15959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:16 smithi096 ceph-mon[32753]: pgmap v15959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:18 smithi194 ceph-mon[33017]: pgmap v15960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:18 smithi033 ceph-mon[27845]: pgmap v15960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:18 smithi096 ceph-mon[32753]: pgmap v15960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29297.148006 secs 2024-04-26T19:59:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:19 smithi194 ceph-mon[33017]: pgmap v15961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29297.148006 secs 2024-04-26T19:59:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:19 smithi033 ceph-mon[27845]: pgmap v15961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29297.148006 secs 2024-04-26T19:59:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:19 smithi096 ceph-mon[32753]: pgmap v15961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29300.063180 secs 2024-04-26T19:59:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29300.063180 secs 2024-04-26T19:59:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29300.063180 secs 2024-04-26T19:59:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:21 smithi194 ceph-mon[33017]: pgmap v15962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:21 smithi033 ceph-mon[27845]: pgmap v15962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:21 smithi096 ceph-mon[32753]: pgmap v15962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:24.245+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:24.245+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:59:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:24.245+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29302.148094 secs 2024-04-26T19:59:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:24 smithi194 ceph-mon[33017]: pgmap v15963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29302.148094 secs 2024-04-26T19:59:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:24 smithi033 ceph-mon[27845]: pgmap v15963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29302.148094 secs 2024-04-26T19:59:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:24 smithi096 ceph-mon[32753]: pgmap v15963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29305.063275 secs 2024-04-26T19:59:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29305.063275 secs 2024-04-26T19:59:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29305.063275 secs 2024-04-26T19:59:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:26 smithi194 ceph-mon[33017]: pgmap v15964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:26 smithi033 ceph-mon[27845]: pgmap v15964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:26 smithi096 ceph-mon[32753]: pgmap v15964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:26.781+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T19:59:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:26.781+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T19:59:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:26.782+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 231 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T19:59:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:26.782+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 231 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T19:59:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:28 smithi194 ceph-mon[33017]: pgmap v15965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:28 smithi033 ceph-mon[27845]: pgmap v15965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:28 smithi096 ceph-mon[32753]: pgmap v15965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29307.148158 secs 2024-04-26T19:59:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:29 smithi194 ceph-mon[33017]: pgmap v15966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29307.148158 secs 2024-04-26T19:59:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:29 smithi033 ceph-mon[27845]: pgmap v15966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29307.148158 secs 2024-04-26T19:59:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:29 smithi096 ceph-mon[32753]: pgmap v15966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29310.063354 secs 2024-04-26T19:59:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29310.063354 secs 2024-04-26T19:59:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29310.063354 secs 2024-04-26T19:59:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:31 smithi194 ceph-mon[33017]: pgmap v15967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:31 smithi033 ceph-mon[27845]: pgmap v15967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:31 smithi096 ceph-mon[32753]: pgmap v15967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29312.148264 secs 2024-04-26T19:59:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:34 smithi194 ceph-mon[33017]: pgmap v15968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29312.148264 secs 2024-04-26T19:59:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:34 smithi033 ceph-mon[27845]: pgmap v15968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29312.148264 secs 2024-04-26T19:59:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:34 smithi096 ceph-mon[32753]: pgmap v15968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29315.063478 secs 2024-04-26T19:59:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29315.063478 secs 2024-04-26T19:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29315.063478 secs 2024-04-26T19:59:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:36 smithi194 ceph-mon[33017]: pgmap v15969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:36 smithi033 ceph-mon[27845]: pgmap v15969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:36 smithi096 ceph-mon[32753]: pgmap v15969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:37 smithi194 ceph-mon[33017]: pgmap v15970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:37 smithi033 ceph-mon[27845]: pgmap v15970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:37 smithi096 ceph-mon[32753]: pgmap v15970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:39.261+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:39.261+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:39.261+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:59:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29317.148324 secs 2024-04-26T19:59:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29317.148324 secs 2024-04-26T19:59:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29317.148324 secs 2024-04-26T19:59:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29320.063554 secs 2024-04-26T19:59:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:40 smithi033 ceph-mon[27845]: pgmap v15971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29320.063554 secs 2024-04-26T19:59:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:40 smithi096 ceph-mon[32753]: pgmap v15971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29320.063554 secs 2024-04-26T19:59:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:40 smithi194 ceph-mon[33017]: pgmap v15971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:42 smithi194 ceph-mon[33017]: pgmap v15972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:42 smithi033 ceph-mon[27845]: pgmap v15972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:42 smithi096 ceph-mon[32753]: pgmap v15972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:59:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:59:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:59:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:59:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:59:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:59:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:59:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T19:59:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T19:59:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29322.148383 secs 2024-04-26T19:59:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:44 smithi194 ceph-mon[33017]: pgmap v15973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:44 smithi194 ceph-mon[33017]: pgmap v15974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29322.148383 secs 2024-04-26T19:59:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:44 smithi033 ceph-mon[27845]: pgmap v15973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:44 smithi033 ceph-mon[27845]: pgmap v15974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29322.148383 secs 2024-04-26T19:59:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:44 smithi096 ceph-mon[32753]: pgmap v15973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:44 smithi096 ceph-mon[32753]: pgmap v15974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29325.063687 secs 2024-04-26T19:59:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:45 smithi194 ceph-mon[33017]: pgmap v15975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29325.063687 secs 2024-04-26T19:59:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:45 smithi033 ceph-mon[27845]: pgmap v15975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29325.063687 secs 2024-04-26T19:59:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:45 smithi096 ceph-mon[32753]: pgmap v15975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:47 smithi194 ceph-mon[33017]: pgmap v15976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:48.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:47 smithi033 ceph-mon[27845]: pgmap v15976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:47 smithi096 ceph-mon[32753]: pgmap v15976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29327.148468 secs 2024-04-26T19:59:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29327.148468 secs 2024-04-26T19:59:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29327.148468 secs 2024-04-26T19:59:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29330.063807 secs 2024-04-26T19:59:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:49 smithi194 ceph-mon[33017]: pgmap v15977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29330.063807 secs 2024-04-26T19:59:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:49 smithi033 ceph-mon[27845]: pgmap v15977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29330.063807 secs 2024-04-26T19:59:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:49 smithi096 ceph-mon[32753]: pgmap v15977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:51 smithi194 ceph-mon[33017]: pgmap v15978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:51 smithi033 ceph-mon[27845]: pgmap v15978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:51 smithi096 ceph-mon[32753]: pgmap v15978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T19:59:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:54.268+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:54.268+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T19:59:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T19:59:54.268+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T19:59:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29332.148552 secs 2024-04-26T19:59:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:54 smithi194 ceph-mon[33017]: pgmap v15979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:59:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29332.148552 secs 2024-04-26T19:59:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:54 smithi033 ceph-mon[27845]: pgmap v15979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:59:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29332.148552 secs 2024-04-26T19:59:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:54 smithi096 ceph-mon[32753]: pgmap v15979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T19:59:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29335.063929 secs 2024-04-26T19:59:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29335.063929 secs 2024-04-26T19:59:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29335.063929 secs 2024-04-26T19:59:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:56 smithi194 ceph-mon[33017]: pgmap v15980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:56 smithi033 ceph-mon[27845]: pgmap v15980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:56 smithi096 ceph-mon[32753]: pgmap v15980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T19:59:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:58 smithi194 ceph-mon[33017]: pgmap v15981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:58 smithi033 ceph-mon[27845]: pgmap v15981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:58 smithi096 ceph-mon[32753]: pgmap v15981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T19:59:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 19:59:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29337.148638 secs 2024-04-26T19:59:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 19:59:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29337.148638 secs 2024-04-26T19:59:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 19:59:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29337.148638 secs 2024-04-26T20:00:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29340.064064 secs 2024-04-26T20:00:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:00 smithi033 ceph-mon[27845]: pgmap v15982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:00:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29340.064064 secs 2024-04-26T20:00:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:00 smithi096 ceph-mon[32753]: pgmap v15982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:00:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29340.064064 secs 2024-04-26T20:00:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:00 smithi194 ceph-mon[33017]: pgmap v15982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:00:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:02 smithi194 ceph-mon[33017]: pgmap v15983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:02 smithi033 ceph-mon[27845]: pgmap v15983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:02 smithi096 ceph-mon[32753]: pgmap v15983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29342.148729 secs 2024-04-26T20:00:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:04 smithi194 ceph-mon[33017]: pgmap v15984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29342.148729 secs 2024-04-26T20:00:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:04 smithi033 ceph-mon[27845]: pgmap v15984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29342.148729 secs 2024-04-26T20:00:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:04 smithi096 ceph-mon[32753]: pgmap v15984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29345.064180 secs 2024-04-26T20:00:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29345.064180 secs 2024-04-26T20:00:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29345.064180 secs 2024-04-26T20:00:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:06 smithi194 ceph-mon[33017]: pgmap v15985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:06 smithi033 ceph-mon[27845]: pgmap v15985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:06 smithi096 ceph-mon[32753]: pgmap v15985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:08 smithi194 ceph-mon[33017]: pgmap v15986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:08 smithi033 ceph-mon[27845]: pgmap v15986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:08 smithi096 ceph-mon[32753]: pgmap v15986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:09.279+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:09.279+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:09.279+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:00:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29347.148792 secs 2024-04-26T20:00:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29347.148792 secs 2024-04-26T20:00:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29347.148792 secs 2024-04-26T20:00:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29350.064257 secs 2024-04-26T20:00:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:10 smithi194 ceph-mon[33017]: pgmap v15987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29350.064257 secs 2024-04-26T20:00:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:10 smithi033 ceph-mon[27845]: pgmap v15987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29350.064257 secs 2024-04-26T20:00:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:10 smithi096 ceph-mon[32753]: pgmap v15987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:12 smithi194 ceph-mon[33017]: pgmap v15988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:12 smithi033 ceph-mon[27845]: pgmap v15988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:12 smithi096 ceph-mon[32753]: pgmap v15988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29352.148848 secs 2024-04-26T20:00:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:14 smithi194 ceph-mon[33017]: pgmap v15989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29352.148848 secs 2024-04-26T20:00:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:14 smithi033 ceph-mon[27845]: pgmap v15989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29352.148848 secs 2024-04-26T20:00:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:14 smithi096 ceph-mon[32753]: pgmap v15989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29355.064345 secs 2024-04-26T20:00:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29355.064345 secs 2024-04-26T20:00:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29355.064345 secs 2024-04-26T20:00:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:16 smithi194 ceph-mon[33017]: pgmap v15990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:16 smithi033 ceph-mon[27845]: pgmap v15990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:16 smithi096 ceph-mon[32753]: pgmap v15990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:18 smithi033 ceph-mon[27845]: pgmap v15991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:18 smithi096 ceph-mon[32753]: pgmap v15991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:18 smithi194 ceph-mon[33017]: pgmap v15991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29357.148934 secs 2024-04-26T20:00:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29357.148934 secs 2024-04-26T20:00:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29357.148934 secs 2024-04-26T20:00:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29360.064462 secs 2024-04-26T20:00:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:20 smithi033 ceph-mon[27845]: pgmap v15992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29360.064462 secs 2024-04-26T20:00:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:20 smithi096 ceph-mon[32753]: pgmap v15992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29360.064462 secs 2024-04-26T20:00:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:20 smithi194 ceph-mon[33017]: pgmap v15992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:22 smithi033 ceph-mon[27845]: pgmap v15993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:22 smithi096 ceph-mon[32753]: pgmap v15993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:22 smithi194 ceph-mon[33017]: pgmap v15993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:24.294+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:24.294+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:24.294+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:00:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29362.149023 secs 2024-04-26T20:00:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:24 smithi033 ceph-mon[27845]: pgmap v15994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29362.149023 secs 2024-04-26T20:00:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:24 smithi096 ceph-mon[32753]: pgmap v15994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29362.149023 secs 2024-04-26T20:00:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:24 smithi194 ceph-mon[33017]: pgmap v15994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29365.064579 secs 2024-04-26T20:00:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29365.064579 secs 2024-04-26T20:00:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29365.064579 secs 2024-04-26T20:00:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:26 smithi033 ceph-mon[27845]: pgmap v15995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:26 smithi096 ceph-mon[32753]: pgmap v15995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:26 smithi194 ceph-mon[33017]: pgmap v15995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:28 smithi033 ceph-mon[27845]: pgmap v15996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:28 smithi096 ceph-mon[32753]: pgmap v15996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:28 smithi194 ceph-mon[33017]: pgmap v15996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29367.149083 secs 2024-04-26T20:00:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29367.149083 secs 2024-04-26T20:00:29.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29367.149083 secs 2024-04-26T20:00:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29370.064644 secs 2024-04-26T20:00:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:30 smithi033 ceph-mon[27845]: pgmap v15997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29370.064644 secs 2024-04-26T20:00:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:30 smithi096 ceph-mon[32753]: pgmap v15997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29370.064644 secs 2024-04-26T20:00:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:30 smithi194 ceph-mon[33017]: pgmap v15997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:32 smithi033 ceph-mon[27845]: pgmap v15998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:32 smithi096 ceph-mon[32753]: pgmap v15998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:32 smithi194 ceph-mon[33017]: pgmap v15998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29372.149141 secs 2024-04-26T20:00:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:34 smithi033 ceph-mon[27845]: pgmap v15999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29372.149141 secs 2024-04-26T20:00:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:34 smithi096 ceph-mon[32753]: pgmap v15999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29372.149141 secs 2024-04-26T20:00:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:34 smithi194 ceph-mon[33017]: pgmap v15999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:00:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29375.064777 secs 2024-04-26T20:00:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29375.064777 secs 2024-04-26T20:00:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29375.064777 secs 2024-04-26T20:00:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:36 smithi194 ceph-mon[33017]: pgmap v16000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:36 smithi033 ceph-mon[27845]: pgmap v16000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:36 smithi096 ceph-mon[32753]: pgmap v16000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:38 smithi194 ceph-mon[33017]: pgmap v16001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:38 smithi033 ceph-mon[27845]: pgmap v16001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:38 smithi096 ceph-mon[32753]: pgmap v16001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:39.297+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:39.298+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:39.298+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:00:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29377.149203 secs 2024-04-26T20:00:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29377.149203 secs 2024-04-26T20:00:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29377.149203 secs 2024-04-26T20:00:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29380.064919 secs 2024-04-26T20:00:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:40 smithi194 ceph-mon[33017]: pgmap v16002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29380.064919 secs 2024-04-26T20:00:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:40 smithi033 ceph-mon[27845]: pgmap v16002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29380.064919 secs 2024-04-26T20:00:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:40 smithi096 ceph-mon[32753]: pgmap v16002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:42 smithi194 ceph-mon[33017]: pgmap v16003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:42 smithi033 ceph-mon[27845]: pgmap v16003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:42 smithi096 ceph-mon[32753]: pgmap v16003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29382.149281 secs 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:44 smithi194 ceph-mon[33017]: pgmap v16004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29382.149281 secs 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:44 smithi033 ceph-mon[27845]: pgmap v16004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.604 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.605 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.605 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29382.149281 secs 2024-04-26T20:00:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:44 smithi096 ceph-mon[32753]: pgmap v16004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29385.065009 secs 2024-04-26T20:00:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29385.065009 secs 2024-04-26T20:00:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29385.065009 secs 2024-04-26T20:00:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:46 smithi194 ceph-mon[33017]: pgmap v16005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:46 smithi033 ceph-mon[27845]: pgmap v16005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:46 smithi096 ceph-mon[32753]: pgmap v16005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:00:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:47.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:48 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:00:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:48 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:00:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:48 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:00:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:48 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:00:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:48 smithi194 ceph-mon[33017]: pgmap v16006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:48 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:00:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:48 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:00:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:48 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:00:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:48 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:00:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:48 smithi033 ceph-mon[27845]: pgmap v16006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:48 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:00:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:48 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:00:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:48 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:00:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:48 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:00:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:48 smithi096 ceph-mon[32753]: pgmap v16006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29387.149370 secs 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:00:49.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:00:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29387.149370 secs 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:00:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:00:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29387.149370 secs 2024-04-26T20:00:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:00:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:00:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29390.065136 secs 2024-04-26T20:00:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:50 smithi194 ceph-mon[33017]: pgmap v16007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:50 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:00:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:50 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:00:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:50 smithi194 ceph-mon[33017]: pgmap v16008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:50.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:50 smithi194 ceph-mon[33017]: pgmap v16009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T20:00:50.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29390.065136 secs 2024-04-26T20:00:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:50 smithi033 ceph-mon[27845]: pgmap v16007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:50 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:00:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:50 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:00:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:50 smithi033 ceph-mon[27845]: pgmap v16008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:50 smithi033 ceph-mon[27845]: pgmap v16009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T20:00:50.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29390.065136 secs 2024-04-26T20:00:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:50 smithi096 ceph-mon[32753]: pgmap v16007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:00:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:50 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:00:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:50 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:00:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:50 smithi096 ceph-mon[32753]: pgmap v16008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:50 smithi096 ceph-mon[32753]: pgmap v16009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T20:00:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:00:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:52 smithi194 ceph-mon[33017]: pgmap v16010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:52 smithi033 ceph-mon[27845]: pgmap v16010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:52 smithi096 ceph-mon[32753]: pgmap v16010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:00:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:54.312+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:54.312+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:00:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:00:54.312+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:00:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29392.149455 secs 2024-04-26T20:00:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:54 smithi194 ceph-mon[33017]: pgmap v16011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:00:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29392.149455 secs 2024-04-26T20:00:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:54 smithi033 ceph-mon[27845]: pgmap v16011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:00:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29392.149455 secs 2024-04-26T20:00:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:54 smithi096 ceph-mon[32753]: pgmap v16011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:00:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29395.065252 secs 2024-04-26T20:00:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29395.065252 secs 2024-04-26T20:00:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29395.065252 secs 2024-04-26T20:00:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:56 smithi194 ceph-mon[33017]: pgmap v16012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:00:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:56 smithi033 ceph-mon[27845]: pgmap v16012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:00:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:56 smithi096 ceph-mon[32753]: pgmap v16012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:00:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:58 smithi194 ceph-mon[33017]: pgmap v16013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:00:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:58 smithi033 ceph-mon[27845]: pgmap v16013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:00:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:58 smithi096 ceph-mon[32753]: pgmap v16013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:00:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:00:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29397.149515 secs 2024-04-26T20:00:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:00:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29397.149515 secs 2024-04-26T20:00:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:00:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29397.149515 secs 2024-04-26T20:01:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29400.065331 secs 2024-04-26T20:01:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:00 smithi033 ceph-mon[27845]: pgmap v16014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:01:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29400.065331 secs 2024-04-26T20:01:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:00 smithi096 ceph-mon[32753]: pgmap v16014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:01:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29400.065331 secs 2024-04-26T20:01:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:00 smithi194 ceph-mon[33017]: pgmap v16014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:01:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:02 smithi194 ceph-mon[33017]: pgmap v16015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:01:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:02 smithi033 ceph-mon[27845]: pgmap v16015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:01:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:02 smithi096 ceph-mon[32753]: pgmap v16015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:01:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29402.149596 secs 2024-04-26T20:01:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:04 smithi194 ceph-mon[33017]: pgmap v16016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:01:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29402.149596 secs 2024-04-26T20:01:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:04 smithi033 ceph-mon[27845]: pgmap v16016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:01:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29402.149596 secs 2024-04-26T20:01:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:04 smithi096 ceph-mon[32753]: pgmap v16016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:01:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29405.065448 secs 2024-04-26T20:01:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:05 smithi194 ceph-mon[33017]: pgmap v16017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:01:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29405.065448 secs 2024-04-26T20:01:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:05 smithi033 ceph-mon[27845]: pgmap v16017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:01:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29405.065448 secs 2024-04-26T20:01:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:05 smithi096 ceph-mon[32753]: pgmap v16017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:01:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:08 smithi194 ceph-mon[33017]: pgmap v16018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:08 smithi033 ceph-mon[27845]: pgmap v16018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:08 smithi096 ceph-mon[32753]: pgmap v16018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:09.328+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:09.328+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:09.328+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:01:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29407.149676 secs 2024-04-26T20:01:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29407.149676 secs 2024-04-26T20:01:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29407.149676 secs 2024-04-26T20:01:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29410.065568 secs 2024-04-26T20:01:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:10 smithi194 ceph-mon[33017]: pgmap v16019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29410.065568 secs 2024-04-26T20:01:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:10 smithi033 ceph-mon[27845]: pgmap v16019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29410.065568 secs 2024-04-26T20:01:10.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:10 smithi096 ceph-mon[32753]: pgmap v16019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:11 smithi194 ceph-mon[33017]: pgmap v16020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:11 smithi033 ceph-mon[27845]: pgmap v16020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:11 smithi096 ceph-mon[32753]: pgmap v16020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29412.149756 secs 2024-04-26T20:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:14 smithi194 ceph-mon[33017]: pgmap v16021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29412.149756 secs 2024-04-26T20:01:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:14 smithi033 ceph-mon[27845]: pgmap v16021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29412.149756 secs 2024-04-26T20:01:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:14 smithi096 ceph-mon[32753]: pgmap v16021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29415.065706 secs 2024-04-26T20:01:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29415.065706 secs 2024-04-26T20:01:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29415.065706 secs 2024-04-26T20:01:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:16 smithi194 ceph-mon[33017]: pgmap v16022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:16 smithi033 ceph-mon[27845]: pgmap v16022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:16 smithi096 ceph-mon[32753]: pgmap v16022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:17 smithi194 ceph-mon[33017]: pgmap v16023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:17 smithi033 ceph-mon[27845]: pgmap v16023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:17 smithi096 ceph-mon[32753]: pgmap v16023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29417.149815 secs 2024-04-26T20:01:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29417.149815 secs 2024-04-26T20:01:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29417.149815 secs 2024-04-26T20:01:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29420.065841 secs 2024-04-26T20:01:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:19 smithi194 ceph-mon[33017]: pgmap v16024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29420.065841 secs 2024-04-26T20:01:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:19 smithi033 ceph-mon[27845]: pgmap v16024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29420.065841 secs 2024-04-26T20:01:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:19 smithi096 ceph-mon[32753]: pgmap v16024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:22 smithi194 ceph-mon[33017]: pgmap v16025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:22 smithi033 ceph-mon[27845]: pgmap v16025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:22 smithi096 ceph-mon[32753]: pgmap v16025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:24.344+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:24.344+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:24.344+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:01:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29422.149899 secs 2024-04-26T20:01:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:24 smithi194 ceph-mon[33017]: pgmap v16026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29422.149899 secs 2024-04-26T20:01:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:24 smithi033 ceph-mon[27845]: pgmap v16026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29422.149899 secs 2024-04-26T20:01:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:24 smithi096 ceph-mon[32753]: pgmap v16026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29425.065951 secs 2024-04-26T20:01:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29425.065951 secs 2024-04-26T20:01:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29425.065951 secs 2024-04-26T20:01:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:26 smithi194 ceph-mon[33017]: pgmap v16027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:26 smithi033 ceph-mon[27845]: pgmap v16027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:26 smithi096 ceph-mon[32753]: pgmap v16027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:27 smithi033 ceph-mon[27845]: pgmap v16028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:27 smithi096 ceph-mon[32753]: pgmap v16028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:27 smithi194 ceph-mon[33017]: pgmap v16028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29427.150002 secs 2024-04-26T20:01:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29427.150002 secs 2024-04-26T20:01:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29427.150002 secs 2024-04-26T20:01:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29430.066069 secs 2024-04-26T20:01:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:29 smithi194 ceph-mon[33017]: pgmap v16029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:30.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29430.066069 secs 2024-04-26T20:01:30.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:29 smithi033 ceph-mon[27845]: pgmap v16029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29430.066069 secs 2024-04-26T20:01:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:29 smithi096 ceph-mon[32753]: pgmap v16029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:31 smithi033 ceph-mon[27845]: pgmap v16030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:31 smithi096 ceph-mon[32753]: pgmap v16030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:31 smithi194 ceph-mon[33017]: pgmap v16030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:33 smithi033 ceph-mon[27845]: pgmap v16031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:33 smithi096 ceph-mon[32753]: pgmap v16031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:33 smithi194 ceph-mon[33017]: pgmap v16031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29432.150085 secs 2024-04-26T20:01:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29432.150085 secs 2024-04-26T20:01:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29432.150085 secs 2024-04-26T20:01:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29435.066144 secs 2024-04-26T20:01:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:35 smithi033 ceph-mon[27845]: pgmap v16032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29435.066144 secs 2024-04-26T20:01:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:35 smithi096 ceph-mon[32753]: pgmap v16032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29435.066144 secs 2024-04-26T20:01:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:35 smithi194 ceph-mon[33017]: pgmap v16032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:37 smithi033 ceph-mon[27845]: pgmap v16033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:37 smithi096 ceph-mon[32753]: pgmap v16033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:37 smithi194 ceph-mon[33017]: pgmap v16033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29437.150176 secs 2024-04-26T20:01:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29437.150176 secs 2024-04-26T20:01:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29437.150176 secs 2024-04-26T20:01:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:39.356+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:39.356+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:39.356+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:01:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29440.066269 secs 2024-04-26T20:01:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:39 smithi194 ceph-mon[33017]: pgmap v16034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29440.066269 secs 2024-04-26T20:01:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:39 smithi033 ceph-mon[27845]: pgmap v16034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29440.066269 secs 2024-04-26T20:01:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:39 smithi096 ceph-mon[32753]: pgmap v16034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:41 smithi033 ceph-mon[27845]: pgmap v16035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:41 smithi096 ceph-mon[32753]: pgmap v16035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:41 smithi194 ceph-mon[33017]: pgmap v16035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:43 smithi033 ceph-mon[27845]: pgmap v16036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:43 smithi096 ceph-mon[32753]: pgmap v16036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:43 smithi194 ceph-mon[33017]: pgmap v16036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29442.150259 secs 2024-04-26T20:01:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29442.150259 secs 2024-04-26T20:01:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29442.150259 secs 2024-04-26T20:01:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29445.066385 secs 2024-04-26T20:01:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:45 smithi033 ceph-mon[27845]: pgmap v16037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29445.066385 secs 2024-04-26T20:01:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:45 smithi096 ceph-mon[32753]: pgmap v16037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29445.066385 secs 2024-04-26T20:01:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:45 smithi194 ceph-mon[33017]: pgmap v16037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:01:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:47 smithi033 ceph-mon[27845]: pgmap v16038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:47 smithi096 ceph-mon[32753]: pgmap v16038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:47 smithi194 ceph-mon[33017]: pgmap v16038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:01:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29447.150323 secs 2024-04-26T20:01:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29447.150323 secs 2024-04-26T20:01:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29447.150323 secs 2024-04-26T20:01:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29450.066506 secs 2024-04-26T20:01:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:49 smithi194 ceph-mon[33017]: pgmap v16039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29450.066506 secs 2024-04-26T20:01:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:49 smithi033 ceph-mon[27845]: pgmap v16039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29450.066506 secs 2024-04-26T20:01:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:49 smithi096 ceph-mon[32753]: pgmap v16039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:01:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:01:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:01:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:51 smithi194 ceph-mon[33017]: pgmap v16040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:51 smithi194 ceph-mon[33017]: pgmap v16041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:51 smithi033 ceph-mon[27845]: pgmap v16040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:51 smithi033 ceph-mon[27845]: pgmap v16041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:52.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:51 smithi096 ceph-mon[32753]: pgmap v16040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:52.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:51 smithi096 ceph-mon[32753]: pgmap v16041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29452.150400 secs 2024-04-26T20:01:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:53 smithi194 ceph-mon[33017]: pgmap v16042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:54.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:54.371+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:01:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:01:54.371+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:01:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29452.150400 secs 2024-04-26T20:01:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:53 smithi033 ceph-mon[27845]: pgmap v16042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29452.150400 secs 2024-04-26T20:01:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:53 smithi096 ceph-mon[32753]: pgmap v16042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29455.066624 secs 2024-04-26T20:01:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29455.066624 secs 2024-04-26T20:01:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29455.066624 secs 2024-04-26T20:01:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:55 smithi194 ceph-mon[33017]: pgmap v16043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:55 smithi033 ceph-mon[27845]: pgmap v16043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:55 smithi096 ceph-mon[32753]: pgmap v16043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:58 smithi194 ceph-mon[33017]: pgmap v16044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:58 smithi033 ceph-mon[27845]: pgmap v16044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:58 smithi096 ceph-mon[32753]: pgmap v16044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:01:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:01:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29457.150486 secs 2024-04-26T20:01:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:01:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29457.150486 secs 2024-04-26T20:01:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:01:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29457.150486 secs 2024-04-26T20:02:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29460.066764 secs 2024-04-26T20:02:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:00 smithi033 ceph-mon[27845]: pgmap v16045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:02:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29460.066764 secs 2024-04-26T20:02:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:00 smithi096 ceph-mon[32753]: pgmap v16045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:02:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29460.066764 secs 2024-04-26T20:02:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:00 smithi194 ceph-mon[33017]: pgmap v16045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:02:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:02 smithi194 ceph-mon[33017]: pgmap v16046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:02:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:02 smithi033 ceph-mon[27845]: pgmap v16046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:02:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:02 smithi096 ceph-mon[32753]: pgmap v16046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:02:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29462.150570 secs 2024-04-26T20:02:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:04 smithi194 ceph-mon[33017]: pgmap v16047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29462.150570 secs 2024-04-26T20:02:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:04 smithi033 ceph-mon[27845]: pgmap v16047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29462.150570 secs 2024-04-26T20:02:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:04 smithi096 ceph-mon[32753]: pgmap v16047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29465.066883 secs 2024-04-26T20:02:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29465.066883 secs 2024-04-26T20:02:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29465.066883 secs 2024-04-26T20:02:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:06 smithi194 ceph-mon[33017]: pgmap v16048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:06 smithi033 ceph-mon[27845]: pgmap v16048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:06 smithi096 ceph-mon[32753]: pgmap v16048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:08.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:08 smithi033 ceph-mon[27845]: pgmap v16049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:08 smithi194 ceph-mon[33017]: pgmap v16049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:08 smithi096 ceph-mon[32753]: pgmap v16049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29467.150648 secs 2024-04-26T20:02:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:09.387+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:09.387+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:02:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:09.387+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29467.150648 secs 2024-04-26T20:02:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29467.150648 secs 2024-04-26T20:02:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29470.067018 secs 2024-04-26T20:02:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:10 smithi194 ceph-mon[33017]: pgmap v16050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29470.067018 secs 2024-04-26T20:02:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:10 smithi033 ceph-mon[27845]: pgmap v16050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29470.067018 secs 2024-04-26T20:02:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:10 smithi096 ceph-mon[32753]: pgmap v16050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:12 smithi194 ceph-mon[33017]: pgmap v16051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:12.476 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:12 smithi096 ceph-mon[32753]: pgmap v16051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:12.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:12 smithi033 ceph-mon[27845]: pgmap v16051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29472.150746 secs 2024-04-26T20:02:14.583 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:14 smithi194 ceph-mon[33017]: pgmap v16052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:14.817 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29472.150746 secs 2024-04-26T20:02:14.858 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:14 smithi096 ceph-mon[32753]: pgmap v16052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:14.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29472.150746 secs 2024-04-26T20:02:15.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:14 smithi033 ceph-mon[27845]: pgmap v16052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29475.067143 secs 2024-04-26T20:02:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29475.067143 secs 2024-04-26T20:02:15.618 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29475.067143 secs 2024-04-26T20:02:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:16 smithi194 ceph-mon[33017]: pgmap v16053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:16.482 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:16 smithi096 ceph-mon[32753]: pgmap v16053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:16.955 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:16 smithi033 ceph-mon[27845]: pgmap v16053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:18 smithi194 ceph-mon[33017]: pgmap v16054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:18 smithi033 ceph-mon[27845]: pgmap v16054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:18.460 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:18 smithi096 ceph-mon[32753]: pgmap v16054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29477.150846 secs 2024-04-26T20:02:19.627 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29477.150846 secs 2024-04-26T20:02:19.659 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29477.150846 secs 2024-04-26T20:02:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29480.067255 secs 2024-04-26T20:02:20.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:20 smithi194 ceph-mon[33017]: pgmap v16055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:20.460 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29480.067255 secs 2024-04-26T20:02:20.709 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:20 smithi096 ceph-mon[32753]: pgmap v16055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:20.743 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29480.067255 secs 2024-04-26T20:02:21.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:20 smithi033 ceph-mon[27845]: pgmap v16055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:22 smithi033 ceph-mon[27845]: pgmap v16056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:22.585 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:22 smithi096 ceph-mon[32753]: pgmap v16056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:22.618 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:22 smithi194 ceph-mon[33017]: pgmap v16056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:24.403+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:24.403+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:24.403+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:02:24.880 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29482.150924 secs 2024-04-26T20:02:25.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:24 smithi194 ceph-mon[33017]: pgmap v16057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:25.100 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29482.150924 secs 2024-04-26T20:02:25.359 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:24 smithi096 ceph-mon[32753]: pgmap v16057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:25.409 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29482.150924 secs 2024-04-26T20:02:25.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:24 smithi033 ceph-mon[27845]: pgmap v16057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:25.626 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29485.067334 secs 2024-04-26T20:02:25.659 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29485.067334 secs 2024-04-26T20:02:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29485.067334 secs 2024-04-26T20:02:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:26 smithi033 ceph-mon[27845]: pgmap v16058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:26.643 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:26 smithi194 ceph-mon[33017]: pgmap v16058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:26.817 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:26 smithi096 ceph-mon[32753]: pgmap v16058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:26.811+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:02:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:26.811+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T20:02:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:26.812+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 232 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T20:02:27.379 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:26.812+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 232 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:02:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:28 smithi033 ceph-mon[27845]: pgmap v16059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:28.494 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:28 smithi096 ceph-mon[32753]: pgmap v16059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:28.689 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:28 smithi194 ceph-mon[33017]: pgmap v16059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29487.151010 secs 2024-04-26T20:02:29.559 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29487.151010 secs 2024-04-26T20:02:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29487.151010 secs 2024-04-26T20:02:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29490.067465 secs 2024-04-26T20:02:30.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:30 smithi033 ceph-mon[27845]: pgmap v16060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:30.926 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29490.067465 secs 2024-04-26T20:02:30.959 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:30 smithi194 ceph-mon[33017]: pgmap v16060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:30.992 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29490.067465 secs 2024-04-26T20:02:31.276 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:30 smithi096 ceph-mon[32753]: pgmap v16060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:32 smithi033 ceph-mon[27845]: pgmap v16061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:32.568 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:32 smithi096 ceph-mon[32753]: pgmap v16061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:32 smithi194 ceph-mon[33017]: pgmap v16061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29492.151092 secs 2024-04-26T20:02:34.752 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:34 smithi033 ceph-mon[27845]: pgmap v16062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:34.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29492.151092 secs 2024-04-26T20:02:35.000 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:34 smithi194 ceph-mon[33017]: pgmap v16062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:35.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29492.151092 secs 2024-04-26T20:02:35.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:34 smithi096 ceph-mon[32753]: pgmap v16062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29495.067585 secs 2024-04-26T20:02:35.659 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29495.067585 secs 2024-04-26T20:02:35.912 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29495.067585 secs 2024-04-26T20:02:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:36 smithi033 ceph-mon[27845]: pgmap v16063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:36.531 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:36 smithi096 ceph-mon[32753]: pgmap v16063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:37.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:36 smithi194 ceph-mon[33017]: pgmap v16063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:38 smithi033 ceph-mon[27845]: pgmap v16064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:38.614 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:38 smithi194 ceph-mon[33017]: pgmap v16064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:38.662 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:38 smithi096 ceph-mon[32753]: pgmap v16064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:39.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:39.419+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:39.419+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:39.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:39.419+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:02:39.746 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29497.151154 secs 2024-04-26T20:02:39.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29497.151154 secs 2024-04-26T20:02:40.395 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29497.151154 secs 2024-04-26T20:02:40.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29500.067664 secs 2024-04-26T20:02:40.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:40 smithi194 ceph-mon[33017]: pgmap v16065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:41.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29500.067664 secs 2024-04-26T20:02:41.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:40 smithi033 ceph-mon[27845]: pgmap v16065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:41.354 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29500.067664 secs 2024-04-26T20:02:41.590 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:40 smithi096 ceph-mon[32753]: pgmap v16065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:42 smithi194 ceph-mon[33017]: pgmap v16066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:42.696 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:42 smithi096 ceph-mon[32753]: pgmap v16066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:42.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:42 smithi033 ceph-mon[27845]: pgmap v16066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29502.151205 secs 2024-04-26T20:02:44.662 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:44 smithi194 ceph-mon[33017]: pgmap v16067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:44.696 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29502.151205 secs 2024-04-26T20:02:44.903 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:44 smithi096 ceph-mon[32753]: pgmap v16067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:44.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29502.151205 secs 2024-04-26T20:02:45.189 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:44 smithi033 ceph-mon[27845]: pgmap v16067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29505.067798 secs 2024-04-26T20:02:45.704 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29505.067798 secs 2024-04-26T20:02:45.987 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29505.067798 secs 2024-04-26T20:02:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:46 smithi194 ceph-mon[33017]: pgmap v16068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:46.746 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:46 smithi096 ceph-mon[32753]: pgmap v16068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:46.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:46 smithi033 ceph-mon[27845]: pgmap v16068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:48 smithi194 ceph-mon[33017]: pgmap v16069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:48 smithi033 ceph-mon[27845]: pgmap v16069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:48.887 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:48 smithi096 ceph-mon[32753]: pgmap v16069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29507.151288 secs 2024-04-26T20:02:49.737 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29507.151288 secs 2024-04-26T20:02:49.770 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29507.151288 secs 2024-04-26T20:02:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29510.067914 secs 2024-04-26T20:02:50.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:50 smithi194 ceph-mon[33017]: pgmap v16070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:50.870 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29510.067914 secs 2024-04-26T20:02:51.086 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:50 smithi096 ceph-mon[32753]: pgmap v16070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:51.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29510.067914 secs 2024-04-26T20:02:51.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:50 smithi033 ceph-mon[27845]: pgmap v16070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:51.820 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:52.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:52 smithi194 ceph-mon[33017]: pgmap v16071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:52.845 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:52 smithi096 ceph-mon[32753]: pgmap v16071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:53.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:52 smithi033 ceph-mon[27845]: pgmap v16071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:02:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:53.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:53.728 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:02:53.979 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:54.011 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:54.566 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:02:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:54.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:54.845 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:02:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:54 smithi194 ceph-mon[33017]: pgmap v16072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:55.395 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:54 smithi194 ceph-mon[33017]: pgmap v16073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:55.428 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29512.151369 secs 2024-04-26T20:02:55.703 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:54 smithi096 ceph-mon[32753]: pgmap v16072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:54 smithi096 ceph-mon[32753]: pgmap v16073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:55.972 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29512.151369 secs 2024-04-26T20:02:56.153 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:54.433+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:54.433+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:02:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:02:54.433+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:02:56.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:54 smithi033 ceph-mon[27845]: pgmap v16072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:02:56.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:54 smithi033 ceph-mon[27845]: pgmap v16073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:56.802 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29512.151369 secs 2024-04-26T20:02:56.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29515.068053 secs 2024-04-26T20:02:56.954 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:56 smithi033 ceph-mon[27845]: pgmap v16074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:57.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29515.068053 secs 2024-04-26T20:02:57.245 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:56 smithi194 ceph-mon[33017]: pgmap v16074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:57.303 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29515.068053 secs 2024-04-26T20:02:57.453 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:56 smithi096 ceph-mon[32753]: pgmap v16074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:58 smithi194 ceph-mon[33017]: pgmap v16075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:58 smithi033 ceph-mon[27845]: pgmap v16075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:58.904 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:58 smithi096 ceph-mon[32753]: pgmap v16075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:02:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:02:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29517.151454 secs 2024-04-26T20:02:59.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:02:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29517.151454 secs 2024-04-26T20:02:59.812 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:02:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29517.151454 secs 2024-04-26T20:03:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:00 smithi033 ceph-mon[27845]: pgmap v16076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:03:00.637 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29520.068174 secs 2024-04-26T20:03:00.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:00 smithi194 ceph-mon[33017]: pgmap v16076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:03:00.704 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29520.068174 secs 2024-04-26T20:03:00.970 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:00 smithi096 ceph-mon[32753]: pgmap v16076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:03:01.003 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29520.068174 secs 2024-04-26T20:03:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:02 smithi194 ceph-mon[33017]: pgmap v16077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:03:02.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:02 smithi096 ceph-mon[32753]: pgmap v16077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:03:02.820 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:02 smithi033 ceph-mon[27845]: pgmap v16077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:03:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:04 smithi194 ceph-mon[33017]: pgmap v16078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:03:04.778 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29522.151534 secs 2024-04-26T20:03:04.812 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:04 smithi096 ceph-mon[32753]: pgmap v16078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:03:05.036 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29522.151534 secs 2024-04-26T20:03:05.070 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:04 smithi033 ceph-mon[27845]: pgmap v16078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:03:05.103 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29522.151534 secs 2024-04-26T20:03:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29525.068291 secs 2024-04-26T20:03:06.042 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29525.068291 secs 2024-04-26T20:03:06.086 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29525.068291 secs 2024-04-26T20:03:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:06 smithi194 ceph-mon[33017]: pgmap v16079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:06 smithi033 ceph-mon[27845]: pgmap v16079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:06.703 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:06 smithi096 ceph-mon[32753]: pgmap v16079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:08 smithi194 ceph-mon[33017]: pgmap v16080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:08 smithi033 ceph-mon[27845]: pgmap v16080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:08.845 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:08 smithi096 ceph-mon[32753]: pgmap v16080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:09.436+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:09.436+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:09.436+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:03:09.912 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29527.151622 secs 2024-04-26T20:03:10.244 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29527.151622 secs 2024-04-26T20:03:10.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29527.151622 secs 2024-04-26T20:03:10.728 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:10 smithi096 ceph-mon[32753]: pgmap v16081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:10.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29530.068365 secs 2024-04-26T20:03:10.994 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:10 smithi194 ceph-mon[33017]: pgmap v16081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:11.028 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29530.068365 secs 2024-04-26T20:03:11.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:10 smithi033 ceph-mon[27845]: pgmap v16081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:11.261 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29530.068365 secs 2024-04-26T20:03:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:12 smithi194 ceph-mon[33017]: pgmap v16082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:12 smithi033 ceph-mon[27845]: pgmap v16082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:12.828 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:12 smithi096 ceph-mon[32753]: pgmap v16082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:13 smithi194 ceph-mon[33017]: pgmap v16083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:13.997 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:13 smithi096 ceph-mon[32753]: pgmap v16083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:14.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:13 smithi033 ceph-mon[27845]: pgmap v16083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29532.151705 secs 2024-04-26T20:03:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29532.151705 secs 2024-04-26T20:03:14.845 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29532.151705 secs 2024-04-26T20:03:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29535.068466 secs 2024-04-26T20:03:15.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:15 smithi033 ceph-mon[27845]: pgmap v16084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:16.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29535.068466 secs 2024-04-26T20:03:16.111 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:15 smithi194 ceph-mon[33017]: pgmap v16084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:16.240 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29535.068466 secs 2024-04-26T20:03:16.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:15 smithi096 ceph-mon[32753]: pgmap v16084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:17 smithi033 ceph-mon[27845]: pgmap v16085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:18.010 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:17 smithi096 ceph-mon[32753]: pgmap v16085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:18.229 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:17 smithi194 ceph-mon[33017]: pgmap v16085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29537.151794 secs 2024-04-26T20:03:19.037 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29537.151794 secs 2024-04-26T20:03:19.131 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29537.151794 secs 2024-04-26T20:03:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:19 smithi033 ceph-mon[27845]: pgmap v16086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:20.102 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29540.068530 secs 2024-04-26T20:03:20.136 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:19 smithi194 ceph-mon[33017]: pgmap v16086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:20.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29540.068530 secs 2024-04-26T20:03:20.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:19 smithi096 ceph-mon[32753]: pgmap v16086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:20.569 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29540.068530 secs 2024-04-26T20:03:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:21 smithi033 ceph-mon[27845]: pgmap v16087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:22.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:21 smithi096 ceph-mon[32753]: pgmap v16087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:22.244 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:21 smithi194 ceph-mon[33017]: pgmap v16087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:23 smithi096 ceph-mon[32753]: pgmap v16088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:24.127 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29542.151876 secs 2024-04-26T20:03:24.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:23 smithi033 ceph-mon[27845]: pgmap v16088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:24.685 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29542.151876 secs 2024-04-26T20:03:24.728 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:23 smithi194 ceph-mon[33017]: pgmap v16088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:25.498 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29542.151876 secs 2024-04-26T20:03:25.698 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29545.068655 secs 2024-04-26T20:03:25.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:24.452+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:24.452+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:24.452+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:03:26.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29545.068655 secs 2024-04-26T20:03:26.569 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29545.068655 secs 2024-04-26T20:03:26.602 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:25 smithi033 ceph-mon[27845]: pgmap v16089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:26.902 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:25 smithi096 ceph-mon[32753]: pgmap v16089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:27.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:25 smithi194 ceph-mon[33017]: pgmap v16089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:27 smithi194 ceph-mon[33017]: pgmap v16090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:28.535 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:27 smithi096 ceph-mon[32753]: pgmap v16090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:28.585 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:27 smithi033 ceph-mon[27845]: pgmap v16090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29547.151989 secs 2024-04-26T20:03:29.352 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29547.151989 secs 2024-04-26T20:03:29.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29547.151989 secs 2024-04-26T20:03:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:29 smithi194 ceph-mon[33017]: pgmap v16091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:30.351 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29550.068751 secs 2024-04-26T20:03:30.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:29 smithi096 ceph-mon[32753]: pgmap v16091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:30.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29550.068751 secs 2024-04-26T20:03:30.957 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:29 smithi033 ceph-mon[27845]: pgmap v16091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:31.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29550.068751 secs 2024-04-26T20:03:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:31 smithi194 ceph-mon[33017]: pgmap v16092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:31 smithi033 ceph-mon[27845]: pgmap v16092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:32.406 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:31 smithi096 ceph-mon[32753]: pgmap v16092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:33 smithi194 ceph-mon[33017]: pgmap v16093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:34.152 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29552.152041 secs 2024-04-26T20:03:34.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:33 smithi096 ceph-mon[32753]: pgmap v16093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:34.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29552.152041 secs 2024-04-26T20:03:34.374 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:33 smithi033 ceph-mon[27845]: pgmap v16093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:34.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29552.152041 secs 2024-04-26T20:03:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29555.068879 secs 2024-04-26T20:03:35.277 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29555.068879 secs 2024-04-26T20:03:35.310 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29555.068879 secs 2024-04-26T20:03:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:35 smithi194 ceph-mon[33017]: pgmap v16094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:35 smithi033 ceph-mon[27845]: pgmap v16094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:36.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:35 smithi096 ceph-mon[32753]: pgmap v16094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:37 smithi194 ceph-mon[33017]: pgmap v16095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:38.219 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:37 smithi096 ceph-mon[32753]: pgmap v16095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:38.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:37 smithi033 ceph-mon[27845]: pgmap v16095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29557.152136 secs 2024-04-26T20:03:39.378 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29557.152136 secs 2024-04-26T20:03:39.626 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29557.152136 secs 2024-04-26T20:03:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:39.467+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:39.843 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:39.467+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:39.467+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:03:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:39 smithi194 ceph-mon[33017]: pgmap v16096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:40.135 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29560.068976 secs 2024-04-26T20:03:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:39 smithi033 ceph-mon[27845]: pgmap v16096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:40.435 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29560.068976 secs 2024-04-26T20:03:40.469 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:39 smithi096 ceph-mon[32753]: pgmap v16096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:40.501 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29560.068976 secs 2024-04-26T20:03:42.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:41 smithi033 ceph-mon[27845]: pgmap v16097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:42.194 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:41 smithi096 ceph-mon[32753]: pgmap v16097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:42.474 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:41 smithi194 ceph-mon[33017]: pgmap v16097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:43 smithi194 ceph-mon[33017]: pgmap v16098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:44.384 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29562.152221 secs 2024-04-26T20:03:44.577 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:43 smithi096 ceph-mon[32753]: pgmap v16098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:44.609 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29562.152221 secs 2024-04-26T20:03:44.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:43 smithi033 ceph-mon[27845]: pgmap v16098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:44.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29562.152221 secs 2024-04-26T20:03:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29565.069133 secs 2024-04-26T20:03:45.193 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29565.069133 secs 2024-04-26T20:03:45.226 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29565.069133 secs 2024-04-26T20:03:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:45 smithi194 ceph-mon[33017]: pgmap v16099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:45 smithi033 ceph-mon[27845]: pgmap v16099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:46.376 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:45 smithi096 ceph-mon[32753]: pgmap v16099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:47 smithi194 ceph-mon[33017]: pgmap v16100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:47 smithi033 ceph-mon[27845]: pgmap v16100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:48.201 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:47 smithi096 ceph-mon[32753]: pgmap v16100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29567.152311 secs 2024-04-26T20:03:49.277 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29567.152311 secs 2024-04-26T20:03:49.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29567.152311 secs 2024-04-26T20:03:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:49 smithi194 ceph-mon[33017]: pgmap v16101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:50.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29570.069272 secs 2024-04-26T20:03:50.543 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:49 smithi096 ceph-mon[32753]: pgmap v16101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:50.576 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29570.069272 secs 2024-04-26T20:03:50.809 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:49 smithi033 ceph-mon[27845]: pgmap v16101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:50.859 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29570.069272 secs 2024-04-26T20:03:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:51 smithi194 ceph-mon[33017]: pgmap v16102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:52.193 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:51 smithi096 ceph-mon[32753]: pgmap v16102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:52.442 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:51 smithi033 ceph-mon[27845]: pgmap v16102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:53 smithi194 ceph-mon[33017]: pgmap v16103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:54.217 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:54.250 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29572.152401 secs 2024-04-26T20:03:54.289 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:53 smithi096 ceph-mon[32753]: pgmap v16103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:54.516 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:54.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29572.152401 secs 2024-04-26T20:03:54.584 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:53 smithi033 ceph-mon[27845]: pgmap v16103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:03:54.776 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:54.832 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29572.152401 secs 2024-04-26T20:03:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:54.481+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:55.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:54.481+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:03:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:03:54.481+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:03:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29575.069333 secs 2024-04-26T20:03:55.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:55.692 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:55.726 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:55.984 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:56.043 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29575.069333 secs 2024-04-26T20:03:56.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:56.389 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:56.642 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:56.909 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:57.176 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29575.069333 secs 2024-04-26T20:03:57.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:57.425 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:57.632 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:57.667 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:57.701 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: pgmap v16104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:58.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:58.201 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:58.234 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:58.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:58.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: pgmap v16105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:03:58.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: pgmap v16106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:03:58.834 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:58.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: pgmap v16104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:03:59.101 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:59.359 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:59.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:59.651 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:59.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: pgmap v16105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:03:59.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: pgmap v16106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:03:59.909 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:03:59.942 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: pgmap v16104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:00.001 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:04:00.226 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:04:00.259 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:00.292 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:00.470 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: pgmap v16105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:04:00.517 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: pgmap v16106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:04:00.751 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:04:00.786 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:58 smithi033 ceph-mon[27845]: pgmap v16107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:04:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:03:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29577.152489 secs 2024-04-26T20:04:01.464 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29580.069455 secs 2024-04-26T20:04:01.668 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:00 smithi033 ceph-mon[27845]: pgmap v16108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:04:01.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:58 smithi194 ceph-mon[33017]: pgmap v16107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:04:01.734 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:03:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29577.152489 secs 2024-04-26T20:04:01.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29580.069455 secs 2024-04-26T20:04:01.976 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:00 smithi194 ceph-mon[33017]: pgmap v16108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:04:02.009 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:58 smithi096 ceph-mon[32753]: pgmap v16107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:04:02.243 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:03:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29577.152489 secs 2024-04-26T20:04:02.512 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29580.069455 secs 2024-04-26T20:04:02.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:00 smithi096 ceph-mon[32753]: pgmap v16108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:04:02.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:02 smithi033 ceph-mon[27845]: pgmap v16109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:04:02.793 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:02 smithi194 ceph-mon[33017]: pgmap v16109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:04:02.843 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:02 smithi096 ceph-mon[32753]: pgmap v16109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:04:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29582.152570 secs 2024-04-26T20:04:04.651 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:04 smithi194 ceph-mon[33017]: pgmap v16110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:04:04.806 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29582.152570 secs 2024-04-26T20:04:04.867 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:04 smithi096 ceph-mon[32753]: pgmap v16110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:04:04.901 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29582.152570 secs 2024-04-26T20:04:04.934 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:04 smithi033 ceph-mon[27845]: pgmap v16110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:04:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29585.069574 secs 2024-04-26T20:04:05.743 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29585.069574 secs 2024-04-26T20:04:05.776 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29585.069574 secs 2024-04-26T20:04:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:06 smithi194 ceph-mon[33017]: pgmap v16111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:04:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:06 smithi033 ceph-mon[27845]: pgmap v16111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:04:06.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:06 smithi096 ceph-mon[32753]: pgmap v16111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:04:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:08 smithi194 ceph-mon[33017]: pgmap v16112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:08.702 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:08 smithi096 ceph-mon[32753]: pgmap v16112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:08.734 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:08 smithi033 ceph-mon[27845]: pgmap v16112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:09.488+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:09.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:09.488+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:04:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:09.488+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:09.793 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29587.152656 secs 2024-04-26T20:04:09.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29587.152656 secs 2024-04-26T20:04:10.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29587.152656 secs 2024-04-26T20:04:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29590.069691 secs 2024-04-26T20:04:10.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:10 smithi194 ceph-mon[33017]: pgmap v16113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29590.069691 secs 2024-04-26T20:04:10.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:10 smithi033 ceph-mon[27845]: pgmap v16113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:11.226 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29590.069691 secs 2024-04-26T20:04:11.259 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:10 smithi096 ceph-mon[32753]: pgmap v16113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:12 smithi194 ceph-mon[33017]: pgmap v16114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:12 smithi033 ceph-mon[27845]: pgmap v16114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:12.893 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:12 smithi096 ceph-mon[32753]: pgmap v16114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29592.152743 secs 2024-04-26T20:04:14.642 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:14 smithi194 ceph-mon[33017]: pgmap v16115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29592.152743 secs 2024-04-26T20:04:14.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:14 smithi033 ceph-mon[27845]: pgmap v16115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:14.926 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29592.152743 secs 2024-04-26T20:04:15.125 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:14 smithi096 ceph-mon[32753]: pgmap v16115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29595.069816 secs 2024-04-26T20:04:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29595.069816 secs 2024-04-26T20:04:15.901 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29595.069816 secs 2024-04-26T20:04:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:16 smithi194 ceph-mon[33017]: pgmap v16116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:16.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:16 smithi096 ceph-mon[32753]: pgmap v16116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:16.934 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:16 smithi033 ceph-mon[27845]: pgmap v16116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:17 smithi033 ceph-mon[27845]: pgmap v16117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:17.725 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:17 smithi096 ceph-mon[32753]: pgmap v16117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:18.000 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:17 smithi194 ceph-mon[33017]: pgmap v16117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29597.152834 secs 2024-04-26T20:04:19.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29597.152834 secs 2024-04-26T20:04:19.308 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29597.152834 secs 2024-04-26T20:04:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29600.069958 secs 2024-04-26T20:04:20.242 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:19 smithi194 ceph-mon[33017]: pgmap v16118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:20.276 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29600.069958 secs 2024-04-26T20:04:20.475 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:19 smithi096 ceph-mon[32753]: pgmap v16118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:20.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29600.069958 secs 2024-04-26T20:04:20.564 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:19 smithi033 ceph-mon[27845]: pgmap v16118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:22 smithi194 ceph-mon[33017]: pgmap v16119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:22.817 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:22 smithi096 ceph-mon[32753]: pgmap v16119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:23.000 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:22 smithi033 ceph-mon[27845]: pgmap v16119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:24.503+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:24.504+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:04:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:24.504+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:24.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29602.152916 secs 2024-04-26T20:04:24.950 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:24 smithi096 ceph-mon[32753]: pgmap v16120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:24.983 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29602.152916 secs 2024-04-26T20:04:25.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:24 smithi194 ceph-mon[33017]: pgmap v16120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:25.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29602.152916 secs 2024-04-26T20:04:25.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:24 smithi033 ceph-mon[27845]: pgmap v16120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:25.817 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29605.070088 secs 2024-04-26T20:04:26.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29605.070088 secs 2024-04-26T20:04:26.083 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29605.070088 secs 2024-04-26T20:04:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:26 smithi194 ceph-mon[33017]: pgmap v16121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:26 smithi033 ceph-mon[27845]: pgmap v16121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:26.792 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:26 smithi096 ceph-mon[32753]: pgmap v16121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:28 smithi033 ceph-mon[27845]: pgmap v16122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:28.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:28 smithi096 ceph-mon[32753]: pgmap v16122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:28.915 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:28 smithi194 ceph-mon[33017]: pgmap v16122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29607.153002 secs 2024-04-26T20:04:29.767 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29607.153002 secs 2024-04-26T20:04:29.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29607.153002 secs 2024-04-26T20:04:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29610.070233 secs 2024-04-26T20:04:30.641 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:30 smithi033 ceph-mon[27845]: pgmap v16123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:30.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29610.070233 secs 2024-04-26T20:04:30.866 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:30 smithi194 ceph-mon[33017]: pgmap v16123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:30.899 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29610.070233 secs 2024-04-26T20:04:31.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:30 smithi096 ceph-mon[32753]: pgmap v16123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:32 smithi194 ceph-mon[33017]: pgmap v16124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:32.751 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:32 smithi096 ceph-mon[32753]: pgmap v16124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:32.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:32 smithi033 ceph-mon[27845]: pgmap v16124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29612.153088 secs 2024-04-26T20:04:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:34 smithi194 ceph-mon[33017]: pgmap v16125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:34.869 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29612.153088 secs 2024-04-26T20:04:34.907 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:34 smithi096 ceph-mon[32753]: pgmap v16125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:34.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29612.153088 secs 2024-04-26T20:04:34.974 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:34 smithi033 ceph-mon[27845]: pgmap v16125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29615.070356 secs 2024-04-26T20:04:35.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29615.070356 secs 2024-04-26T20:04:35.824 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29615.070356 secs 2024-04-26T20:04:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:36 smithi194 ceph-mon[33017]: pgmap v16126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:36.802 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:36 smithi096 ceph-mon[32753]: pgmap v16126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:36.832 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:36 smithi033 ceph-mon[27845]: pgmap v16126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:38 smithi194 ceph-mon[33017]: pgmap v16127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:38 smithi033 ceph-mon[27845]: pgmap v16127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:38.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:38 smithi096 ceph-mon[32753]: pgmap v16127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:39.512+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:39.512+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:39.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:39.512+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:04:40.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29617.153178 secs 2024-04-26T20:04:40.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29617.153178 secs 2024-04-26T20:04:40.124 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29617.153178 secs 2024-04-26T20:04:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29620.070481 secs 2024-04-26T20:04:40.716 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:40 smithi194 ceph-mon[33017]: pgmap v16128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:40.900 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29620.070481 secs 2024-04-26T20:04:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:40 smithi096 ceph-mon[32753]: pgmap v16128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:41.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29620.070481 secs 2024-04-26T20:04:41.216 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:40 smithi033 ceph-mon[27845]: pgmap v16128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:41.835+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:04:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:41.835+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T20:04:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:41.835+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 233 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:04:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:41.835+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 233 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:04:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:42 smithi033 ceph-mon[27845]: pgmap v16129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:42.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:42 smithi096 ceph-mon[32753]: pgmap v16129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:42 smithi194 ceph-mon[33017]: pgmap v16129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:43 smithi033 ceph-mon[27845]: pgmap v16130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:43.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:43 smithi096 ceph-mon[32753]: pgmap v16130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:43 smithi194 ceph-mon[33017]: pgmap v16130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29622.153234 secs 2024-04-26T20:04:44.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29622.153234 secs 2024-04-26T20:04:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29622.153234 secs 2024-04-26T20:04:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29625.070610 secs 2024-04-26T20:04:45.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:45 smithi033 ceph-mon[27845]: pgmap v16131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:45.958 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29625.070610 secs 2024-04-26T20:04:46.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:45 smithi194 ceph-mon[33017]: pgmap v16131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:46.407 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29625.070610 secs 2024-04-26T20:04:46.441 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:45 smithi096 ceph-mon[32753]: pgmap v16131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:48 smithi194 ceph-mon[33017]: pgmap v16132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:48.862 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:48 smithi096 ceph-mon[32753]: pgmap v16132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:48.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:48 smithi033 ceph-mon[27845]: pgmap v16132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29627.153321 secs 2024-04-26T20:04:49.707 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29627.153321 secs 2024-04-26T20:04:49.740 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29627.153321 secs 2024-04-26T20:04:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29630.070765 secs 2024-04-26T20:04:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:50 smithi194 ceph-mon[33017]: pgmap v16133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:51.041 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29630.070765 secs 2024-04-26T20:04:51.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:50 smithi096 ceph-mon[32753]: pgmap v16133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:51.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29630.070765 secs 2024-04-26T20:04:51.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:50 smithi033 ceph-mon[27845]: pgmap v16133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:52 smithi194 ceph-mon[33017]: pgmap v16134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:52 smithi033 ceph-mon[27845]: pgmap v16134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:52.857 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:52 smithi096 ceph-mon[32753]: pgmap v16134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:54.517+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:54.517+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:04:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:04:54.517+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:04:54.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29632.153379 secs 2024-04-26T20:04:55.031 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:54 smithi096 ceph-mon[32753]: pgmap v16135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:55.266 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29632.153379 secs 2024-04-26T20:04:55.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:54 smithi194 ceph-mon[33017]: pgmap v16135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:55.340 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29632.153379 secs 2024-04-26T20:04:55.640 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:54 smithi033 ceph-mon[27845]: pgmap v16135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:55.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29635.070889 secs 2024-04-26T20:04:55.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29635.070889 secs 2024-04-26T20:04:56.361 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29635.070889 secs 2024-04-26T20:04:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:56 smithi194 ceph-mon[33017]: pgmap v16136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:56.698 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:56.732 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:56 smithi096 ceph-mon[32753]: pgmap v16136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:56.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:57.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:56 smithi033 ceph-mon[27845]: pgmap v16136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:04:57.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:57.840 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:57.878 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:04:58.129 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:58.370 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:58.613 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:04:58.653 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:58.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:58.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:04:58.971 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:58 smithi194 ceph-mon[33017]: pgmap v16137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:59.287 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:58 smithi194 ceph-mon[33017]: pgmap v16138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:04:59.470 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:58 smithi096 ceph-mon[32753]: pgmap v16137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:59.526 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:58 smithi096 ceph-mon[32753]: pgmap v16138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:04:59.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:58 smithi033 ceph-mon[27845]: pgmap v16137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:04:59.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:58 smithi033 ceph-mon[27845]: pgmap v16138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:04:59.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:04:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29637.153464 secs 2024-04-26T20:04:59.837 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:04:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29637.153464 secs 2024-04-26T20:05:00.123 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:04:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29637.153464 secs 2024-04-26T20:05:00.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29640.071001 secs 2024-04-26T20:05:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29640.071001 secs 2024-04-26T20:05:00.678 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:00 smithi194 ceph-mon[33017]: pgmap v16139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:00.726 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29640.071001 secs 2024-04-26T20:05:00.860 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:00 smithi096 ceph-mon[32753]: pgmap v16139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:00.949 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:00 smithi033 ceph-mon[27845]: pgmap v16139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:02 smithi033 ceph-mon[27845]: pgmap v16140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:02.721 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:02 smithi096 ceph-mon[32753]: pgmap v16140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:02 smithi194 ceph-mon[33017]: pgmap v16140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29642.153546 secs 2024-04-26T20:05:04.803 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:04 smithi033 ceph-mon[27845]: pgmap v16141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:04.837 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29642.153546 secs 2024-04-26T20:05:05.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:04 smithi096 ceph-mon[32753]: pgmap v16141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:05.087 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29642.153546 secs 2024-04-26T20:05:05.120 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:04 smithi194 ceph-mon[33017]: pgmap v16141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29645.071149 secs 2024-04-26T20:05:05.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:05 smithi033 ceph-mon[27845]: pgmap v16142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:06.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29645.071149 secs 2024-04-26T20:05:06.087 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:05 smithi194 ceph-mon[33017]: pgmap v16142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:06.120 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29645.071149 secs 2024-04-26T20:05:06.303 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:05 smithi096 ceph-mon[32753]: pgmap v16142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:05:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:07 smithi194 ceph-mon[33017]: pgmap v16143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:05:08.544 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:07 smithi096 ceph-mon[32753]: pgmap v16143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:05:08.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:07 smithi033 ceph-mon[27845]: pgmap v16143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:05:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29647.153631 secs 2024-04-26T20:05:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29647.153631 secs 2024-04-26T20:05:09.453 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29647.153631 secs 2024-04-26T20:05:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:09.532+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:09.532+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:09.532+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:05:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29650.071271 secs 2024-04-26T20:05:10.378 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:09 smithi194 ceph-mon[33017]: pgmap v16144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:10.562 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29650.071271 secs 2024-04-26T20:05:10.637 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:09 smithi096 ceph-mon[32753]: pgmap v16144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:10.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29650.071271 secs 2024-04-26T20:05:10.703 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:09 smithi033 ceph-mon[27845]: pgmap v16144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:12 smithi194 ceph-mon[33017]: pgmap v16145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:12 smithi033 ceph-mon[27845]: pgmap v16145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:12.871 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:12 smithi096 ceph-mon[32753]: pgmap v16145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29652.153716 secs 2024-04-26T20:05:14.403 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:14 smithi194 ceph-mon[33017]: pgmap v16146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:14.620 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29652.153716 secs 2024-04-26T20:05:14.673 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:14 smithi096 ceph-mon[32753]: pgmap v16146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:14.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29652.153716 secs 2024-04-26T20:05:14.895 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:14 smithi033 ceph-mon[27845]: pgmap v16146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29655.071387 secs 2024-04-26T20:05:15.454 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29655.071387 secs 2024-04-26T20:05:15.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29655.071387 secs 2024-04-26T20:05:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:16 smithi194 ceph-mon[33017]: pgmap v16147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:16.487 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:16 smithi096 ceph-mon[32753]: pgmap v16147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:16.520 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:16 smithi033 ceph-mon[27845]: pgmap v16147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:18 smithi194 ceph-mon[33017]: pgmap v16148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:18.470 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:18 smithi096 ceph-mon[32753]: pgmap v16148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:18.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:18 smithi033 ceph-mon[27845]: pgmap v16148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29657.153805 secs 2024-04-26T20:05:19.587 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29657.153805 secs 2024-04-26T20:05:19.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29657.153805 secs 2024-04-26T20:05:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29660.071510 secs 2024-04-26T20:05:20.461 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:20 smithi194 ceph-mon[33017]: pgmap v16149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:20.662 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29660.071510 secs 2024-04-26T20:05:20.903 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:20 smithi096 ceph-mon[32753]: pgmap v16149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:20.936 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29660.071510 secs 2024-04-26T20:05:20.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:20 smithi033 ceph-mon[27845]: pgmap v16149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:22 smithi194 ceph-mon[33017]: pgmap v16150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:22.482 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:22 smithi096 ceph-mon[32753]: pgmap v16150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:22.686 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:22 smithi033 ceph-mon[27845]: pgmap v16150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:24.538+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:24.538+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:24.538+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:05:24.920 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29662.153867 secs 2024-04-26T20:05:24.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:24 smithi096 ceph-mon[32753]: pgmap v16151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:24.986 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29662.153867 secs 2024-04-26T20:05:25.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:24 smithi194 ceph-mon[33017]: pgmap v16151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:25.211 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29662.153867 secs 2024-04-26T20:05:25.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:24 smithi033 ceph-mon[27845]: pgmap v16151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29665.071631 secs 2024-04-26T20:05:25.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29665.071631 secs 2024-04-26T20:05:25.836 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29665.071631 secs 2024-04-26T20:05:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:26 smithi194 ceph-mon[33017]: pgmap v16152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:26 smithi033 ceph-mon[27845]: pgmap v16152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:26.870 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:26 smithi096 ceph-mon[32753]: pgmap v16152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:26.842+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T20:05:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:26.842+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:05:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:26.842+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 234 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T20:05:27.557 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:26.843+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 234 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:05:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:28 smithi194 ceph-mon[33017]: pgmap v16153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:28.812 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:28 smithi096 ceph-mon[32753]: pgmap v16153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:28.852 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:28 smithi033 ceph-mon[27845]: pgmap v16153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29667.153941 secs 2024-04-26T20:05:29.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29667.153941 secs 2024-04-26T20:05:29.719 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29667.153941 secs 2024-04-26T20:05:30.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29670.071771 secs 2024-04-26T20:05:30.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:30 smithi033 ceph-mon[27845]: pgmap v16154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:30.919 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29670.071771 secs 2024-04-26T20:05:31.144 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:30 smithi194 ceph-mon[33017]: pgmap v16154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:31.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29670.071771 secs 2024-04-26T20:05:31.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:30 smithi096 ceph-mon[32753]: pgmap v16154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:32 smithi194 ceph-mon[33017]: pgmap v16155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:32.844 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:32 smithi096 ceph-mon[32753]: pgmap v16155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:33.119 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:32 smithi033 ceph-mon[27845]: pgmap v16155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29672.154023 secs 2024-04-26T20:05:34.710 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:34 smithi194 ceph-mon[33017]: pgmap v16156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29672.154023 secs 2024-04-26T20:05:34.977 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:34 smithi096 ceph-mon[32753]: pgmap v16156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:35.185 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29672.154023 secs 2024-04-26T20:05:35.269 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:34 smithi033 ceph-mon[27845]: pgmap v16156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29675.071911 secs 2024-04-26T20:05:35.702 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29675.071911 secs 2024-04-26T20:05:35.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29675.071911 secs 2024-04-26T20:05:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:36 smithi194 ceph-mon[33017]: pgmap v16157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:36.902 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:36 smithi096 ceph-mon[32753]: pgmap v16157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:37.102 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:36 smithi033 ceph-mon[27845]: pgmap v16157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:38 smithi194 ceph-mon[33017]: pgmap v16158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:38.819 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:38 smithi096 ceph-mon[32753]: pgmap v16158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:38.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:38 smithi033 ceph-mon[27845]: pgmap v16158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:39.546+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:39.547+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:39.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:39.547+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:05:39.844 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29677.154086 secs 2024-04-26T20:05:39.877 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29677.154086 secs 2024-04-26T20:05:39.910 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29677.154086 secs 2024-04-26T20:05:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29680.072051 secs 2024-04-26T20:05:40.984 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:40 smithi194 ceph-mon[33017]: pgmap v16159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:41.019 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29680.072051 secs 2024-04-26T20:05:41.295 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:40 smithi096 ceph-mon[32753]: pgmap v16159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:41.477 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29680.072051 secs 2024-04-26T20:05:41.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:40 smithi033 ceph-mon[27845]: pgmap v16159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:42 smithi194 ceph-mon[33017]: pgmap v16160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:42.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:42 smithi096 ceph-mon[32753]: pgmap v16160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:42.943 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:42 smithi033 ceph-mon[27845]: pgmap v16160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29682.154143 secs 2024-04-26T20:05:44.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:44 smithi033 ceph-mon[27845]: pgmap v16161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:45.168 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29682.154143 secs 2024-04-26T20:05:45.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:44 smithi194 ceph-mon[33017]: pgmap v16161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:45.438 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29682.154143 secs 2024-04-26T20:05:45.476 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:44 smithi096 ceph-mon[32753]: pgmap v16161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:45.711 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29685.072173 secs 2024-04-26T20:05:45.743 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29685.072173 secs 2024-04-26T20:05:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29685.072173 secs 2024-04-26T20:05:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:46 smithi033 ceph-mon[27845]: pgmap v16162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:47.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:46 smithi194 ceph-mon[33017]: pgmap v16162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:47.301 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:46 smithi096 ceph-mon[32753]: pgmap v16162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:48 smithi033 ceph-mon[27845]: pgmap v16163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:49.113 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:48 smithi194 ceph-mon[33017]: pgmap v16163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:49.358 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:48 smithi096 ceph-mon[32753]: pgmap v16163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29687.154202 secs 2024-04-26T20:05:49.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:49 smithi033 ceph-mon[27845]: pgmap v16164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:50.244 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29687.154202 secs 2024-04-26T20:05:50.276 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:49 smithi194 ceph-mon[33017]: pgmap v16164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:50.472 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29687.154202 secs 2024-04-26T20:05:50.509 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:49 smithi096 ceph-mon[32753]: pgmap v16164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:50.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29690.072297 secs 2024-04-26T20:05:50.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29690.072297 secs 2024-04-26T20:05:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29690.072297 secs 2024-04-26T20:05:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:51 smithi033 ceph-mon[27845]: pgmap v16165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:51.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:51 smithi096 ceph-mon[32753]: pgmap v16165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:51 smithi194 ceph-mon[33017]: pgmap v16165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29692.154261 secs 2024-04-26T20:05:54.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:54 smithi194 ceph-mon[33017]: pgmap v16166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:54.560 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29692.154261 secs 2024-04-26T20:05:54.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:54 smithi096 ceph-mon[32753]: pgmap v16166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:54.834 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29692.154261 secs 2024-04-26T20:05:54.867 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:54 smithi033 ceph-mon[27845]: pgmap v16166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:54.561+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:55.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:54.561+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:05:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:05:54.561+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:05:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29695.072440 secs 2024-04-26T20:05:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29695.072440 secs 2024-04-26T20:05:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29695.072440 secs 2024-04-26T20:05:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:56 smithi194 ceph-mon[33017]: pgmap v16167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:56 smithi033 ceph-mon[27845]: pgmap v16167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:56 smithi096 ceph-mon[32753]: pgmap v16167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:05:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:58 smithi194 ceph-mon[33017]: pgmap v16168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:58 smithi033 ceph-mon[27845]: pgmap v16168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:58 smithi096 ceph-mon[32753]: pgmap v16168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:05:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29697.154324 secs 2024-04-26T20:05:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:05:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:05:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29697.154324 secs 2024-04-26T20:05:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:05:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:05:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29697.154324 secs 2024-04-26T20:05:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:05:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:06:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:00 smithi033 ceph-mon[27845]: pgmap v16169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:00 smithi033 ceph-mon[27845]: pgmap v16170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29700.072527 secs 2024-04-26T20:06:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:00 smithi096 ceph-mon[32753]: pgmap v16169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:00.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:00 smithi096 ceph-mon[32753]: pgmap v16170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:00.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29700.072527 secs 2024-04-26T20:06:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:00 smithi194 ceph-mon[33017]: pgmap v16169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:00 smithi194 ceph-mon[33017]: pgmap v16170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29700.072527 secs 2024-04-26T20:06:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:02 smithi194 ceph-mon[33017]: pgmap v16171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:02 smithi033 ceph-mon[27845]: pgmap v16171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:02 smithi096 ceph-mon[32753]: pgmap v16171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:04 smithi194 ceph-mon[33017]: pgmap v16172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29702.154382 secs 2024-04-26T20:06:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:04 smithi033 ceph-mon[27845]: pgmap v16172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29702.154382 secs 2024-04-26T20:06:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:04 smithi096 ceph-mon[32753]: pgmap v16172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29702.154382 secs 2024-04-26T20:06:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29705.072649 secs 2024-04-26T20:06:05.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29705.072649 secs 2024-04-26T20:06:05.849 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29705.072649 secs 2024-04-26T20:06:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:06 smithi194 ceph-mon[33017]: pgmap v16173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:06 smithi033 ceph-mon[27845]: pgmap v16173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:06.624 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:06 smithi096 ceph-mon[32753]: pgmap v16173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:08 smithi194 ceph-mon[33017]: pgmap v16174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:08.508 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:08 smithi096 ceph-mon[32753]: pgmap v16174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:08.691 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:08 smithi033 ceph-mon[27845]: pgmap v16174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:06:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29707.154467 secs 2024-04-26T20:06:09.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29707.154467 secs 2024-04-26T20:06:09.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29707.154467 secs 2024-04-26T20:06:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:09.577+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:10.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:09.577+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:06:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:09.577+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:10.391 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:10 smithi194 ceph-mon[33017]: pgmap v16175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:06:10.923 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29710.072787 secs 2024-04-26T20:06:10.974 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:10 smithi096 ceph-mon[32753]: pgmap v16175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:06:11.207 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29710.072787 secs 2024-04-26T20:06:11.382 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:10 smithi033 ceph-mon[27845]: pgmap v16175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:06:11.415 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29710.072787 secs 2024-04-26T20:06:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:12 smithi194 ceph-mon[33017]: pgmap v16176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:12.524 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:12 smithi096 ceph-mon[32753]: pgmap v16176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:12.557 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:12 smithi033 ceph-mon[27845]: pgmap v16176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:14 smithi194 ceph-mon[33017]: pgmap v16177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:14.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29712.154548 secs 2024-04-26T20:06:14.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:14 smithi096 ceph-mon[32753]: pgmap v16177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:14.582 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29712.154548 secs 2024-04-26T20:06:14.790 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:14 smithi033 ceph-mon[27845]: pgmap v16177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:14.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29712.154548 secs 2024-04-26T20:06:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29715.072914 secs 2024-04-26T20:06:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29715.072914 secs 2024-04-26T20:06:15.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29715.072914 secs 2024-04-26T20:06:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:16 smithi194 ceph-mon[33017]: pgmap v16178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:16.625 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:16 smithi096 ceph-mon[32753]: pgmap v16178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:16.998 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:16 smithi033 ceph-mon[27845]: pgmap v16178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:18 smithi033 ceph-mon[27845]: pgmap v16179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:18.507 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:18 smithi096 ceph-mon[32753]: pgmap v16179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:18 smithi194 ceph-mon[33017]: pgmap v16179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29717.154630 secs 2024-04-26T20:06:19.657 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29717.154630 secs 2024-04-26T20:06:19.856 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29717.154630 secs 2024-04-26T20:06:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:20 smithi033 ceph-mon[27845]: pgmap v16180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:20.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29720.073035 secs 2024-04-26T20:06:21.049 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:20 smithi194 ceph-mon[33017]: pgmap v16180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:21.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29720.073035 secs 2024-04-26T20:06:21.563 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:20 smithi096 ceph-mon[32753]: pgmap v16180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:21.606 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29720.073035 secs 2024-04-26T20:06:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:22 smithi033 ceph-mon[27845]: pgmap v16181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:22.498 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:22 smithi096 ceph-mon[32753]: pgmap v16181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:22.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:22 smithi194 ceph-mon[33017]: pgmap v16181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:24 smithi033 ceph-mon[27845]: pgmap v16182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:24.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29722.154709 secs 2024-04-26T20:06:24.740 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:24 smithi194 ceph-mon[33017]: pgmap v16182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:25.014 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29722.154709 secs 2024-04-26T20:06:25.239 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:24 smithi096 ceph-mon[32753]: pgmap v16182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:25.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29722.154709 secs 2024-04-26T20:06:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:24.586+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:25.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:24.586+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:06:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:24.586+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:25.598 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29725.073162 secs 2024-04-26T20:06:25.939 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29725.073162 secs 2024-04-26T20:06:26.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29725.073162 secs 2024-04-26T20:06:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:26 smithi033 ceph-mon[27845]: pgmap v16183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:26.456 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:26 smithi096 ceph-mon[32753]: pgmap v16183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:26.773 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:26 smithi194 ceph-mon[33017]: pgmap v16183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:28 smithi033 ceph-mon[27845]: pgmap v16184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:28.588 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:28 smithi096 ceph-mon[32753]: pgmap v16184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:28.623 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:28 smithi194 ceph-mon[33017]: pgmap v16184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29727.154790 secs 2024-04-26T20:06:29.481 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29727.154790 secs 2024-04-26T20:06:29.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29727.154790 secs 2024-04-26T20:06:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:30 smithi033 ceph-mon[27845]: pgmap v16185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:30.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29730.073222 secs 2024-04-26T20:06:30.536 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:30 smithi194 ceph-mon[33017]: pgmap v16185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:30.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29730.073222 secs 2024-04-26T20:06:30.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:30 smithi096 ceph-mon[32753]: pgmap v16185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:30.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29730.073222 secs 2024-04-26T20:06:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:32 smithi033 ceph-mon[27845]: pgmap v16186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:32.472 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:32 smithi096 ceph-mon[32753]: pgmap v16186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:32.763 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:32 smithi194 ceph-mon[33017]: pgmap v16186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:34 smithi033 ceph-mon[27845]: pgmap v16187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:34.841 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29732.154868 secs 2024-04-26T20:06:34.923 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:34 smithi194 ceph-mon[33017]: pgmap v16187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:35.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29732.154868 secs 2024-04-26T20:06:35.172 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:34 smithi096 ceph-mon[32753]: pgmap v16187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:35.372 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29732.154868 secs 2024-04-26T20:06:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29735.073351 secs 2024-04-26T20:06:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29735.073351 secs 2024-04-26T20:06:35.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29735.073351 secs 2024-04-26T20:06:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:36 smithi194 ceph-mon[33017]: pgmap v16188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:36.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:36 smithi096 ceph-mon[32753]: pgmap v16188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:37.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:36 smithi033 ceph-mon[27845]: pgmap v16188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:38 smithi194 ceph-mon[33017]: pgmap v16189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:38.850 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:38 smithi096 ceph-mon[32753]: pgmap v16189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:38.880 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:38 smithi033 ceph-mon[27845]: pgmap v16189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:39.588+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:39.588+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:06:39.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:39.588+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:39.914 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29737.154955 secs 2024-04-26T20:06:39.947 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29737.154955 secs 2024-04-26T20:06:40.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29737.154955 secs 2024-04-26T20:06:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:40 smithi194 ceph-mon[33017]: pgmap v16190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:41.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29740.073414 secs 2024-04-26T20:06:41.114 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:40 smithi096 ceph-mon[32753]: pgmap v16190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:42.748 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29740.073414 secs 2024-04-26T20:06:42.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:40 smithi033 ceph-mon[27845]: pgmap v16190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:42.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29740.073414 secs 2024-04-26T20:06:43.031 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:42 smithi194 ceph-mon[33017]: pgmap v16191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:43.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:42 smithi096 ceph-mon[32753]: pgmap v16191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:43.322 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:42 smithi033 ceph-mon[27845]: pgmap v16191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:44 smithi194 ceph-mon[33017]: pgmap v16192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:44.722 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29742.155042 secs 2024-04-26T20:06:44.756 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:44 smithi096 ceph-mon[32753]: pgmap v16192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:45.002 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29742.155042 secs 2024-04-26T20:06:45.039 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:44 smithi033 ceph-mon[27845]: pgmap v16192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:45.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29742.155042 secs 2024-04-26T20:06:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29745.073566 secs 2024-04-26T20:06:45.831 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29745.073566 secs 2024-04-26T20:06:45.863 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29745.073566 secs 2024-04-26T20:06:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:46 smithi194 ceph-mon[33017]: pgmap v16193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:46 smithi033 ceph-mon[27845]: pgmap v16193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:46.720 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:46 smithi096 ceph-mon[32753]: pgmap v16193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:48 smithi194 ceph-mon[33017]: pgmap v16194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:48 smithi033 ceph-mon[27845]: pgmap v16194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:48.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:48 smithi096 ceph-mon[32753]: pgmap v16194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29747.155106 secs 2024-04-26T20:06:49.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29747.155106 secs 2024-04-26T20:06:49.714 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29747.155106 secs 2024-04-26T20:06:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:50 smithi194 ceph-mon[33017]: pgmap v16195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:51.612 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29750.073688 secs 2024-04-26T20:06:51.814 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:50 smithi096 ceph-mon[32753]: pgmap v16195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:51.846 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29750.073688 secs 2024-04-26T20:06:52.063 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:50 smithi033 ceph-mon[27845]: pgmap v16195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:52.096 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29750.073688 secs 2024-04-26T20:06:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:52 smithi194 ceph-mon[33017]: pgmap v16196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:52 smithi033 ceph-mon[27845]: pgmap v16196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:52.946 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:52 smithi096 ceph-mon[32753]: pgmap v16196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:54.601+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:54.601+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:06:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:06:54.601+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:06:54.914 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:54 smithi096 ceph-mon[32753]: pgmap v16197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:54.946 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29752.155163 secs 2024-04-26T20:06:54.980 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:54 smithi194 ceph-mon[33017]: pgmap v16197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:55.205 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29752.155163 secs 2024-04-26T20:06:55.238 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:54 smithi033 ceph-mon[27845]: pgmap v16197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:55.271 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29752.155163 secs 2024-04-26T20:06:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29755.073817 secs 2024-04-26T20:06:55.864 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29755.073817 secs 2024-04-26T20:06:56.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29755.073817 secs 2024-04-26T20:06:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:56 smithi194 ceph-mon[33017]: pgmap v16198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:56.872 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:56 smithi096 ceph-mon[32753]: pgmap v16198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:57.121 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:56 smithi033 ceph-mon[27845]: pgmap v16198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:06:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:58 smithi194 ceph-mon[33017]: pgmap v16199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:58.869 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:58 smithi096 ceph-mon[32753]: pgmap v16199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:58.896 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:58 smithi033 ceph-mon[27845]: pgmap v16199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:06:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29757.155225 secs 2024-04-26T20:06:59.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:06:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:59.805 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29757.155225 secs 2024-04-26T20:07:00.046 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:06:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:00.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29757.155225 secs 2024-04-26T20:07:00.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:06:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:00.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:00 smithi033 ceph-mon[27845]: pgmap v16200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:00.812 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29760.073961 secs 2024-04-26T20:07:00.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:00 smithi194 ceph-mon[33017]: pgmap v16200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:00.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29760.073961 secs 2024-04-26T20:07:01.096 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:00 smithi096 ceph-mon[32753]: pgmap v16200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:01.129 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29760.073961 secs 2024-04-26T20:07:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:01.938 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:01 smithi194 ceph-mon[33017]: pgmap v16201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:07:02.129 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:01 smithi194 ceph-mon[33017]: pgmap v16202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:02.396 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:07:02.527 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:02.615 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:02.646 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:01 smithi096 ceph-mon[32753]: pgmap v16201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:07:02.735 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:01 smithi096 ceph-mon[32753]: pgmap v16202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:02.930 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:07:02.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:03.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:03.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:01 smithi033 ceph-mon[27845]: pgmap v16201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:07:03.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:01 smithi033 ceph-mon[27845]: pgmap v16202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:03.813 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:07:04.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:03 smithi194 ceph-mon[33017]: pgmap v16203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:04.080 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:03 smithi096 ceph-mon[32753]: pgmap v16203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:04.503 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:03 smithi033 ceph-mon[27845]: pgmap v16203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29762.155285 secs 2024-04-26T20:07:04.839 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29762.155285 secs 2024-04-26T20:07:04.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29762.155285 secs 2024-04-26T20:07:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29765.074087 secs 2024-04-26T20:07:05.983 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:05 smithi033 ceph-mon[27845]: pgmap v16204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:07:06.246 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29765.074087 secs 2024-04-26T20:07:06.280 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:05 smithi194 ceph-mon[33017]: pgmap v16204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:07:06.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29765.074087 secs 2024-04-26T20:07:06.740 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:05 smithi096 ceph-mon[32753]: pgmap v16204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:07:06.772 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:06 smithi033 ceph-mon[27845]: pgmap v16205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:06.805 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:06 smithi096 ceph-mon[32753]: pgmap v16205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:06 smithi194 ceph-mon[33017]: pgmap v16205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:09.606+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:09.606+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:09.606+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:07:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29767.155369 secs 2024-04-26T20:07:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:09 smithi033 ceph-mon[27845]: pgmap v16206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29767.155369 secs 2024-04-26T20:07:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:09 smithi096 ceph-mon[32753]: pgmap v16206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29767.155369 secs 2024-04-26T20:07:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:09 smithi194 ceph-mon[33017]: pgmap v16206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29770.074167 secs 2024-04-26T20:07:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29770.074167 secs 2024-04-26T20:07:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29770.074167 secs 2024-04-26T20:07:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:11 smithi033 ceph-mon[27845]: pgmap v16207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:07:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:11 smithi096 ceph-mon[32753]: pgmap v16207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:07:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:11 smithi194 ceph-mon[33017]: pgmap v16207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:07:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:12 smithi033 ceph-mon[27845]: pgmap v16208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:12 smithi096 ceph-mon[32753]: pgmap v16208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:12 smithi194 ceph-mon[33017]: pgmap v16208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29772.155427 secs 2024-04-26T20:07:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29772.155427 secs 2024-04-26T20:07:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29772.155427 secs 2024-04-26T20:07:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29775.074282 secs 2024-04-26T20:07:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:14 smithi194 ceph-mon[33017]: pgmap v16209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29775.074282 secs 2024-04-26T20:07:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:14 smithi033 ceph-mon[27845]: pgmap v16209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29775.074282 secs 2024-04-26T20:07:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:14 smithi096 ceph-mon[32753]: pgmap v16209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:17 smithi033 ceph-mon[27845]: pgmap v16210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:17 smithi096 ceph-mon[32753]: pgmap v16210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:17 smithi194 ceph-mon[33017]: pgmap v16210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29777.155530 secs 2024-04-26T20:07:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:19 smithi033 ceph-mon[27845]: pgmap v16211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29777.155530 secs 2024-04-26T20:07:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:19 smithi096 ceph-mon[32753]: pgmap v16211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29777.155530 secs 2024-04-26T20:07:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:19 smithi194 ceph-mon[33017]: pgmap v16211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29780.074399 secs 2024-04-26T20:07:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29780.074399 secs 2024-04-26T20:07:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29780.074399 secs 2024-04-26T20:07:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:21 smithi033 ceph-mon[27845]: pgmap v16212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:21 smithi096 ceph-mon[32753]: pgmap v16212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:21 smithi194 ceph-mon[33017]: pgmap v16212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:22 smithi033 ceph-mon[27845]: pgmap v16213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:22 smithi096 ceph-mon[32753]: pgmap v16213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:22 smithi194 ceph-mon[33017]: pgmap v16213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29782.155603 secs 2024-04-26T20:07:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29782.155603 secs 2024-04-26T20:07:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29782.155603 secs 2024-04-26T20:07:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:24.622+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:24.622+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:24.622+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29785.074527 secs 2024-04-26T20:07:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:24 smithi194 ceph-mon[33017]: pgmap v16214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29785.074527 secs 2024-04-26T20:07:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:24 smithi033 ceph-mon[27845]: pgmap v16214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29785.074527 secs 2024-04-26T20:07:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:24 smithi096 ceph-mon[32753]: pgmap v16214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:27 smithi033 ceph-mon[27845]: pgmap v16215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:27 smithi096 ceph-mon[32753]: pgmap v16215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:27 smithi194 ceph-mon[33017]: pgmap v16215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29787.155688 secs 2024-04-26T20:07:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:29 smithi033 ceph-mon[27845]: pgmap v16216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29787.155688 secs 2024-04-26T20:07:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:29 smithi096 ceph-mon[32753]: pgmap v16216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29787.155688 secs 2024-04-26T20:07:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:29 smithi194 ceph-mon[33017]: pgmap v16216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29790.074645 secs 2024-04-26T20:07:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29790.074645 secs 2024-04-26T20:07:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29790.074645 secs 2024-04-26T20:07:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:31 smithi033 ceph-mon[27845]: pgmap v16217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:31 smithi096 ceph-mon[32753]: pgmap v16217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:31 smithi194 ceph-mon[33017]: pgmap v16217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:32 smithi033 ceph-mon[27845]: pgmap v16218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:32 smithi096 ceph-mon[32753]: pgmap v16218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:32 smithi194 ceph-mon[33017]: pgmap v16218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29792.155769 secs 2024-04-26T20:07:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29792.155769 secs 2024-04-26T20:07:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29792.155769 secs 2024-04-26T20:07:35.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29795.074762 secs 2024-04-26T20:07:35.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:34 smithi033 ceph-mon[27845]: pgmap v16219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29795.074762 secs 2024-04-26T20:07:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:34 smithi194 ceph-mon[33017]: pgmap v16219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29795.074762 secs 2024-04-26T20:07:35.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:34 smithi096 ceph-mon[32753]: pgmap v16219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:37 smithi033 ceph-mon[27845]: pgmap v16220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:37 smithi096 ceph-mon[32753]: pgmap v16220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:37 smithi194 ceph-mon[33017]: pgmap v16220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:39.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:39.633+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:39.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:39.633+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:07:39.634 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:39.633+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29797.155855 secs 2024-04-26T20:07:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:39 smithi033 ceph-mon[27845]: pgmap v16221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29797.155855 secs 2024-04-26T20:07:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:39 smithi096 ceph-mon[32753]: pgmap v16221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29797.155855 secs 2024-04-26T20:07:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:39 smithi194 ceph-mon[33017]: pgmap v16221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29800.074898 secs 2024-04-26T20:07:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:40 smithi033 ceph-mon[27845]: pgmap v16222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29800.074898 secs 2024-04-26T20:07:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:40 smithi096 ceph-mon[32753]: pgmap v16222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29800.074898 secs 2024-04-26T20:07:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:40 smithi194 ceph-mon[33017]: pgmap v16222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:43 smithi033 ceph-mon[27845]: pgmap v16223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:43 smithi096 ceph-mon[32753]: pgmap v16223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:43 smithi194 ceph-mon[33017]: pgmap v16223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29802.155939 secs 2024-04-26T20:07:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29802.155939 secs 2024-04-26T20:07:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29802.155939 secs 2024-04-26T20:07:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29805.075016 secs 2024-04-26T20:07:45.874 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:45 smithi033 ceph-mon[27845]: pgmap v16224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:46.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29805.075016 secs 2024-04-26T20:07:46.115 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:45 smithi194 ceph-mon[33017]: pgmap v16224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:46.332 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29805.075016 secs 2024-04-26T20:07:46.365 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:45 smithi096 ceph-mon[32753]: pgmap v16224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:46 smithi033 ceph-mon[27845]: pgmap v16225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:46.908 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:46 smithi194 ceph-mon[33017]: pgmap v16225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:47.124 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:46 smithi096 ceph-mon[32753]: pgmap v16225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29807.156023 secs 2024-04-26T20:07:49.723 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:49 smithi033 ceph-mon[27845]: pgmap v16226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:49.982 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29807.156023 secs 2024-04-26T20:07:50.579 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:49 smithi194 ceph-mon[33017]: pgmap v16226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:50.848 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29807.156023 secs 2024-04-26T20:07:50.882 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:49 smithi096 ceph-mon[32753]: pgmap v16226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:51.085 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29810.075167 secs 2024-04-26T20:07:51.115 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29810.075167 secs 2024-04-26T20:07:51.149 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29810.075167 secs 2024-04-26T20:07:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:51 smithi033 ceph-mon[27845]: pgmap v16227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:52.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:51 smithi194 ceph-mon[33017]: pgmap v16227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:52.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:52 smithi033 ceph-mon[27845]: pgmap v16228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:53.160 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:51 smithi096 ceph-mon[32753]: pgmap v16227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:53.398 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:52 smithi096 ceph-mon[32753]: pgmap v16228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:53.432 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:52 smithi194 ceph-mon[33017]: pgmap v16228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29812.156102 secs 2024-04-26T20:07:54.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29812.156102 secs 2024-04-26T20:07:54.623 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29812.156102 secs 2024-04-26T20:07:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:54.633+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:54.633+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:07:54.915 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:07:54.633+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:07:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29815.075279 secs 2024-04-26T20:07:55.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:54 smithi194 ceph-mon[33017]: pgmap v16229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:55.566 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29815.075279 secs 2024-04-26T20:07:55.898 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:54 smithi096 ceph-mon[32753]: pgmap v16229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:56.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29815.075279 secs 2024-04-26T20:07:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:54 smithi033 ceph-mon[27845]: pgmap v16229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:07:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:57 smithi033 ceph-mon[27845]: pgmap v16230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:57.826 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:57 smithi096 ceph-mon[32753]: pgmap v16230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:57.857 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:57 smithi194 ceph-mon[33017]: pgmap v16230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:07:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:58 smithi033 ceph-mon[27845]: pgmap v16231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:58.907 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:58 smithi194 ceph-mon[33017]: pgmap v16231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:58.940 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:58 smithi096 ceph-mon[32753]: pgmap v16231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:07:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:07:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29817.156166 secs 2024-04-26T20:07:59.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:07:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29817.156166 secs 2024-04-26T20:07:59.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:07:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29817.156166 secs 2024-04-26T20:08:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29820.075377 secs 2024-04-26T20:08:00.738 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:00 smithi033 ceph-mon[27845]: pgmap v16232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:00.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29820.075377 secs 2024-04-26T20:08:00.973 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:00 smithi194 ceph-mon[33017]: pgmap v16232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:01.006 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29820.075377 secs 2024-04-26T20:08:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:00 smithi096 ceph-mon[32753]: pgmap v16232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:02.040 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:03.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:03.531 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:02 smithi194 ceph-mon[33017]: pgmap v16233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:03.622 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:02 smithi194 ceph-mon[33017]: pgmap v16234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:03.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:08:03.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:04.023 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:04.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:02 smithi096 ceph-mon[32753]: pgmap v16233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:04.377 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:02 smithi096 ceph-mon[32753]: pgmap v16234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:04.615 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:08:04.648 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:04.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:05.139 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:02 smithi033 ceph-mon[27845]: pgmap v16233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:05.190 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:02 smithi033 ceph-mon[27845]: pgmap v16234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:05.686 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:08:05.724 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29822.156249 secs 2024-04-26T20:08:06.014 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29825.075494 secs 2024-04-26T20:08:06.240 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:04 smithi194 ceph-mon[33017]: pgmap v16235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:08:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29822.156249 secs 2024-04-26T20:08:07.574 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29825.075494 secs 2024-04-26T20:08:07.778 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:04 smithi096 ceph-mon[32753]: pgmap v16235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:08:07.806 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29822.156249 secs 2024-04-26T20:08:07.840 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29825.075494 secs 2024-04-26T20:08:07.873 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:04 smithi033 ceph-mon[27845]: pgmap v16235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:08:08.099 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:07 smithi194 ceph-mon[33017]: pgmap v16236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:08.600 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:07 smithi096 ceph-mon[32753]: pgmap v16236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:08.631 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:07 smithi033 ceph-mon[27845]: pgmap v16236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29827.156333 secs 2024-04-26T20:08:09.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:09 smithi033 ceph-mon[27845]: pgmap v16237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:09.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29827.156333 secs 2024-04-26T20:08:09.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:09 smithi194 ceph-mon[33017]: pgmap v16237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:10.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29827.156333 secs 2024-04-26T20:08:10.109 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:09 smithi096 ceph-mon[32753]: pgmap v16237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:10.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:09.645+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:10.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:09.645+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:09.645+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:08:10.858 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29830.075611 secs 2024-04-26T20:08:10.890 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29830.075611 secs 2024-04-26T20:08:10.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29830.075611 secs 2024-04-26T20:08:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:11 smithi033 ceph-mon[27845]: pgmap v16238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:11.499 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:11 smithi096 ceph-mon[32753]: pgmap v16238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:11.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:11 smithi194 ceph-mon[33017]: pgmap v16238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:08:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:13 smithi033 ceph-mon[27845]: pgmap v16239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:08:13.545 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:13 smithi096 ceph-mon[32753]: pgmap v16239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:08:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:13 smithi194 ceph-mon[33017]: pgmap v16239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:08:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29832.156392 secs 2024-04-26T20:08:14.672 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29832.156392 secs 2024-04-26T20:08:14.706 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29832.156392 secs 2024-04-26T20:08:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29835.075747 secs 2024-04-26T20:08:15.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:15 smithi033 ceph-mon[27845]: pgmap v16240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:15.666 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29835.075747 secs 2024-04-26T20:08:15.840 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:15 smithi194 ceph-mon[33017]: pgmap v16240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:16.140 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29835.075747 secs 2024-04-26T20:08:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:15 smithi096 ceph-mon[32753]: pgmap v16240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:17 smithi033 ceph-mon[27845]: pgmap v16241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:17.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:17 smithi194 ceph-mon[33017]: pgmap v16241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:17.706 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:17 smithi096 ceph-mon[32753]: pgmap v16241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29837.156478 secs 2024-04-26T20:08:19.523 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:19 smithi033 ceph-mon[27845]: pgmap v16242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:19.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29837.156478 secs 2024-04-26T20:08:19.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:19 smithi194 ceph-mon[33017]: pgmap v16242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:19.700 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29837.156478 secs 2024-04-26T20:08:19.700 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:19 smithi096 ceph-mon[32753]: pgmap v16242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29840.075811 secs 2024-04-26T20:08:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29840.075811 secs 2024-04-26T20:08:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29840.075811 secs 2024-04-26T20:08:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:21 smithi033 ceph-mon[27845]: pgmap v16243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:21 smithi096 ceph-mon[32753]: pgmap v16243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:21 smithi194 ceph-mon[33017]: pgmap v16243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:23 smithi033 ceph-mon[27845]: pgmap v16244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:23 smithi096 ceph-mon[32753]: pgmap v16244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:23.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:23 smithi194 ceph-mon[33017]: pgmap v16244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29842.156560 secs 2024-04-26T20:08:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29842.156560 secs 2024-04-26T20:08:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:24.654+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:24.654+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:24.654+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:08:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29842.156560 secs 2024-04-26T20:08:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29845.075949 secs 2024-04-26T20:08:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:25 smithi194 ceph-mon[33017]: pgmap v16245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29845.075949 secs 2024-04-26T20:08:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:25 smithi033 ceph-mon[27845]: pgmap v16245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29845.075949 secs 2024-04-26T20:08:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:25 smithi096 ceph-mon[32753]: pgmap v16245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:26.872+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T20:08:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:26.872+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:08:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:26.873+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 235 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T20:08:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:26.873+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 235 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:08:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:27 smithi194 ceph-mon[33017]: pgmap v16246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:27 smithi033 ceph-mon[27845]: pgmap v16246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:27 smithi096 ceph-mon[32753]: pgmap v16246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29847.156621 secs 2024-04-26T20:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:29 smithi194 ceph-mon[33017]: pgmap v16247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29847.156621 secs 2024-04-26T20:08:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:29 smithi033 ceph-mon[27845]: pgmap v16247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29847.156621 secs 2024-04-26T20:08:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:29 smithi096 ceph-mon[32753]: pgmap v16247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29850.076072 secs 2024-04-26T20:08:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29850.076072 secs 2024-04-26T20:08:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29850.076072 secs 2024-04-26T20:08:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:31 smithi194 ceph-mon[33017]: pgmap v16248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:31 smithi033 ceph-mon[27845]: pgmap v16248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:31 smithi096 ceph-mon[32753]: pgmap v16248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:33 smithi194 ceph-mon[33017]: pgmap v16249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:33 smithi033 ceph-mon[27845]: pgmap v16249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:33 smithi096 ceph-mon[32753]: pgmap v16249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29852.156703 secs 2024-04-26T20:08:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29852.156703 secs 2024-04-26T20:08:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29852.156703 secs 2024-04-26T20:08:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29855.076194 secs 2024-04-26T20:08:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:35 smithi194 ceph-mon[33017]: pgmap v16250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29855.076194 secs 2024-04-26T20:08:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:35 smithi033 ceph-mon[27845]: pgmap v16250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29855.076194 secs 2024-04-26T20:08:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:35 smithi096 ceph-mon[32753]: pgmap v16250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:37 smithi194 ceph-mon[33017]: pgmap v16251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:37 smithi033 ceph-mon[27845]: pgmap v16251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:37 smithi096 ceph-mon[32753]: pgmap v16251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29857.156787 secs 2024-04-26T20:08:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:39 smithi194 ceph-mon[33017]: pgmap v16252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:39.662+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:39.662+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:08:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:39.662+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29857.156787 secs 2024-04-26T20:08:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:39 smithi033 ceph-mon[27845]: pgmap v16252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29857.156787 secs 2024-04-26T20:08:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:39 smithi096 ceph-mon[32753]: pgmap v16252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29860.076270 secs 2024-04-26T20:08:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29860.076270 secs 2024-04-26T20:08:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29860.076270 secs 2024-04-26T20:08:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:41 smithi194 ceph-mon[33017]: pgmap v16253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:41 smithi033 ceph-mon[27845]: pgmap v16253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:41 smithi096 ceph-mon[32753]: pgmap v16253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:43 smithi194 ceph-mon[33017]: pgmap v16254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:43 smithi033 ceph-mon[27845]: pgmap v16254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:43 smithi096 ceph-mon[32753]: pgmap v16254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29862.156866 secs 2024-04-26T20:08:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29862.156866 secs 2024-04-26T20:08:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29862.156866 secs 2024-04-26T20:08:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29865.076426 secs 2024-04-26T20:08:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:45 smithi194 ceph-mon[33017]: pgmap v16255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29865.076426 secs 2024-04-26T20:08:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:45 smithi033 ceph-mon[27845]: pgmap v16255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29865.076426 secs 2024-04-26T20:08:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:45 smithi096 ceph-mon[32753]: pgmap v16255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:47 smithi194 ceph-mon[33017]: pgmap v16256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:47 smithi033 ceph-mon[27845]: pgmap v16256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:47 smithi096 ceph-mon[32753]: pgmap v16256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29867.156951 secs 2024-04-26T20:08:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:49 smithi194 ceph-mon[33017]: pgmap v16257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29867.156951 secs 2024-04-26T20:08:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:49 smithi033 ceph-mon[27845]: pgmap v16257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29867.156951 secs 2024-04-26T20:08:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:49 smithi096 ceph-mon[32753]: pgmap v16257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29870.076559 secs 2024-04-26T20:08:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29870.076559 secs 2024-04-26T20:08:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29870.076559 secs 2024-04-26T20:08:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:51 smithi194 ceph-mon[33017]: pgmap v16258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:51 smithi033 ceph-mon[27845]: pgmap v16258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:51 smithi096 ceph-mon[32753]: pgmap v16258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:53 smithi194 ceph-mon[33017]: pgmap v16259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:53 smithi033 ceph-mon[27845]: pgmap v16259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:53 smithi096 ceph-mon[32753]: pgmap v16259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29872.157034 secs 2024-04-26T20:08:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29872.157034 secs 2024-04-26T20:08:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:54.674+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:54.674+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:08:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:08:54.674+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:08:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29872.157034 secs 2024-04-26T20:08:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29875.076678 secs 2024-04-26T20:08:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:55 smithi194 ceph-mon[33017]: pgmap v16260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29875.076678 secs 2024-04-26T20:08:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:55 smithi033 ceph-mon[27845]: pgmap v16260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29875.076678 secs 2024-04-26T20:08:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:55 smithi096 ceph-mon[32753]: pgmap v16260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:57 smithi194 ceph-mon[33017]: pgmap v16261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:57 smithi033 ceph-mon[27845]: pgmap v16261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:57 smithi096 ceph-mon[32753]: pgmap v16261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:08:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29877.157124 secs 2024-04-26T20:08:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:08:59 smithi194 ceph-mon[33017]: pgmap v16262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29877.157124 secs 2024-04-26T20:08:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:08:59 smithi033 ceph-mon[27845]: pgmap v16262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:08:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29877.157124 secs 2024-04-26T20:08:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:08:59 smithi096 ceph-mon[32753]: pgmap v16262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:00.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29880.076777 secs 2024-04-26T20:09:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29880.076777 secs 2024-04-26T20:09:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29880.076777 secs 2024-04-26T20:09:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:01 smithi194 ceph-mon[33017]: pgmap v16263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:01 smithi033 ceph-mon[27845]: pgmap v16263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:01 smithi096 ceph-mon[32753]: pgmap v16263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:03 smithi194 ceph-mon[33017]: pgmap v16264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:09:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:03 smithi033 ceph-mon[27845]: pgmap v16264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:09:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:03 smithi096 ceph-mon[32753]: pgmap v16264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:09:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29882.157193 secs 2024-04-26T20:09:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29882.157193 secs 2024-04-26T20:09:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29882.157193 secs 2024-04-26T20:09:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29885.076921 secs 2024-04-26T20:09:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:05 smithi033 ceph-mon[27845]: pgmap v16265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:09:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:09:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29885.076921 secs 2024-04-26T20:09:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:05 smithi096 ceph-mon[32753]: pgmap v16265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:09:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:09:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29885.076921 secs 2024-04-26T20:09:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:05 smithi194 ceph-mon[33017]: pgmap v16265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:09:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:09:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:09:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:06 smithi033 ceph-mon[27845]: pgmap v16266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:09:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:06 smithi033 ceph-mon[27845]: pgmap v16267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:06 smithi096 ceph-mon[32753]: pgmap v16266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:09:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:06 smithi096 ceph-mon[32753]: pgmap v16267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:06 smithi194 ceph-mon[33017]: pgmap v16266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:09:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:06 smithi194 ceph-mon[33017]: pgmap v16267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:08 smithi033 ceph-mon[27845]: pgmap v16268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:08 smithi096 ceph-mon[32753]: pgmap v16268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:08 smithi194 ceph-mon[33017]: pgmap v16268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29887.157255 secs 2024-04-26T20:09:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29887.157255 secs 2024-04-26T20:09:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:09.686+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:09.686+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:09.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:09.687+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:09:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29887.157255 secs 2024-04-26T20:09:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29890.077038 secs 2024-04-26T20:09:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:10 smithi033 ceph-mon[27845]: pgmap v16269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:09:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29890.077038 secs 2024-04-26T20:09:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:10 smithi096 ceph-mon[32753]: pgmap v16269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29890.077038 secs 2024-04-26T20:09:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:10 smithi194 ceph-mon[33017]: pgmap v16269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:09:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:12 smithi033 ceph-mon[27845]: pgmap v16270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:12 smithi096 ceph-mon[32753]: pgmap v16270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:12 smithi194 ceph-mon[33017]: pgmap v16270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:14 smithi033 ceph-mon[27845]: pgmap v16271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:14.815 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29892.157331 secs 2024-04-26T20:09:14.848 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:14 smithi096 ceph-mon[32753]: pgmap v16271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:14.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29892.157331 secs 2024-04-26T20:09:15.101 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:14 smithi194 ceph-mon[33017]: pgmap v16271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:09:15.131 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29892.157331 secs 2024-04-26T20:09:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29895.077166 secs 2024-04-26T20:09:15.732 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29895.077166 secs 2024-04-26T20:09:15.924 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29895.077166 secs 2024-04-26T20:09:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:16 smithi033 ceph-mon[27845]: pgmap v16272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:09:16.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:16 smithi096 ceph-mon[32753]: pgmap v16272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:09:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:16 smithi194 ceph-mon[33017]: pgmap v16272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:09:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:17 smithi194 ceph-mon[33017]: pgmap v16273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:17 smithi033 ceph-mon[27845]: pgmap v16273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:18.024 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:17 smithi096 ceph-mon[32753]: pgmap v16273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29897.157417 secs 2024-04-26T20:09:19.290 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29897.157417 secs 2024-04-26T20:09:19.535 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29897.157417 secs 2024-04-26T20:09:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29900.077284 secs 2024-04-26T20:09:20.139 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:19 smithi194 ceph-mon[33017]: pgmap v16274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:20.586 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29900.077284 secs 2024-04-26T20:09:20.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:19 smithi096 ceph-mon[32753]: pgmap v16274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:20.761 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29900.077284 secs 2024-04-26T20:09:20.964 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:19 smithi033 ceph-mon[27845]: pgmap v16274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:21 smithi194 ceph-mon[33017]: pgmap v16275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:22.207 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:21 smithi096 ceph-mon[32753]: pgmap v16275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:22.464 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:21 smithi033 ceph-mon[27845]: pgmap v16275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:24 smithi194 ceph-mon[33017]: pgmap v16276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:24.680 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29902.157500 secs 2024-04-26T20:09:24.973 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:24 smithi096 ceph-mon[32753]: pgmap v16276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:25.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29902.157500 secs 2024-04-26T20:09:25.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:24 smithi033 ceph-mon[27845]: pgmap v16276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:25.239 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29902.157500 secs 2024-04-26T20:09:25.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:24.702+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:24.702+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:09:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:24.702+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:26.422 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29905.077381 secs 2024-04-26T20:09:26.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29905.077381 secs 2024-04-26T20:09:26.689 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29905.077381 secs 2024-04-26T20:09:26.723 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:26 smithi033 ceph-mon[27845]: pgmap v16277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:26.756 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:26 smithi096 ceph-mon[32753]: pgmap v16277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:26.998 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:26 smithi194 ceph-mon[33017]: pgmap v16277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:28 smithi033 ceph-mon[27845]: pgmap v16278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:28.745 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:28 smithi096 ceph-mon[32753]: pgmap v16278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:29.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:28 smithi194 ceph-mon[33017]: pgmap v16278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29907.157589 secs 2024-04-26T20:09:29.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29907.157589 secs 2024-04-26T20:09:29.914 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29907.157589 secs 2024-04-26T20:09:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29910.077452 secs 2024-04-26T20:09:30.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:30 smithi033 ceph-mon[27845]: pgmap v16279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:30.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29910.077452 secs 2024-04-26T20:09:30.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:30 smithi194 ceph-mon[33017]: pgmap v16279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:30.889 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29910.077452 secs 2024-04-26T20:09:31.114 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:30 smithi096 ceph-mon[32753]: pgmap v16279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:32 smithi033 ceph-mon[27845]: pgmap v16280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:32.739 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:32 smithi096 ceph-mon[32753]: pgmap v16280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:32 smithi194 ceph-mon[33017]: pgmap v16280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:34 smithi033 ceph-mon[27845]: pgmap v16281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:34.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29912.157671 secs 2024-04-26T20:09:34.931 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:34 smithi194 ceph-mon[33017]: pgmap v16281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:34.964 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29912.157671 secs 2024-04-26T20:09:34.997 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:34 smithi096 ceph-mon[32753]: pgmap v16281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:35.255 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29912.157671 secs 2024-04-26T20:09:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29915.077528 secs 2024-04-26T20:09:35.754 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29915.077528 secs 2024-04-26T20:09:35.980 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29915.077528 secs 2024-04-26T20:09:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:36 smithi033 ceph-mon[27845]: pgmap v16282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:36.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:36 smithi096 ceph-mon[32753]: pgmap v16282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:36 smithi194 ceph-mon[33017]: pgmap v16282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:38 smithi033 ceph-mon[27845]: pgmap v16283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:38.964 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:38 smithi194 ceph-mon[33017]: pgmap v16283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:39.188 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:38 smithi096 ceph-mon[32753]: pgmap v16283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:39.721 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29917.157754 secs 2024-04-26T20:09:39.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29917.157754 secs 2024-04-26T20:09:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:39.716+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:39.717+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:09:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:39.717+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:40.539 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29917.157754 secs 2024-04-26T20:09:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29920.077602 secs 2024-04-26T20:09:40.897 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:40 smithi096 ceph-mon[32753]: pgmap v16284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:40.939 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29920.077602 secs 2024-04-26T20:09:41.202 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:40 smithi033 ceph-mon[27845]: pgmap v16284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:41.741 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29920.077602 secs 2024-04-26T20:09:41.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:40 smithi194 ceph-mon[33017]: pgmap v16284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:42.031 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:41 smithi096 ceph-mon[32753]: pgmap v16285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:42.064 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:41 smithi033 ceph-mon[27845]: pgmap v16285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:42.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:41 smithi194 ceph-mon[33017]: pgmap v16285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:43 smithi194 ceph-mon[33017]: pgmap v16286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:44.238 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29922.157822 secs 2024-04-26T20:09:44.272 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:43 smithi096 ceph-mon[32753]: pgmap v16286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:44.480 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29922.157822 secs 2024-04-26T20:09:44.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:43 smithi033 ceph-mon[27845]: pgmap v16286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:44.547 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29922.157822 secs 2024-04-26T20:09:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29925.077714 secs 2024-04-26T20:09:45.297 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29925.077714 secs 2024-04-26T20:09:45.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29925.077714 secs 2024-04-26T20:09:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:45 smithi194 ceph-mon[33017]: pgmap v16287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:45 smithi033 ceph-mon[27845]: pgmap v16287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:46.439 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:45 smithi096 ceph-mon[32753]: pgmap v16287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:47 smithi194 ceph-mon[33017]: pgmap v16288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:48.306 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:47 smithi096 ceph-mon[32753]: pgmap v16288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:48.338 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:47 smithi033 ceph-mon[27845]: pgmap v16288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29927.157883 secs 2024-04-26T20:09:49.195 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29927.157883 secs 2024-04-26T20:09:49.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29927.157883 secs 2024-04-26T20:09:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29930.077809 secs 2024-04-26T20:09:50.523 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:49 smithi194 ceph-mon[33017]: pgmap v16289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:50.556 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29930.077809 secs 2024-04-26T20:09:50.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:49 smithi096 ceph-mon[32753]: pgmap v16289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:51.421 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29930.077809 secs 2024-04-26T20:09:51.463 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:49 smithi033 ceph-mon[27845]: pgmap v16289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:51 smithi194 ceph-mon[33017]: pgmap v16290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:52.231 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:51 smithi096 ceph-mon[32753]: pgmap v16290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:52.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:51 smithi033 ceph-mon[27845]: pgmap v16290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:53 smithi194 ceph-mon[33017]: pgmap v16291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:54.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29932.157972 secs 2024-04-26T20:09:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:53 smithi033 ceph-mon[27845]: pgmap v16291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:54.396 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29932.157972 secs 2024-04-26T20:09:54.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:53 smithi096 ceph-mon[32753]: pgmap v16291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:54.463 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29932.157972 secs 2024-04-26T20:09:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:54.731+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:54.731+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:09:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:09:54.731+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:09:55.205 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29935.078012 secs 2024-04-26T20:09:55.410 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29935.078012 secs 2024-04-26T20:09:55.463 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29935.078012 secs 2024-04-26T20:09:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:55 smithi194 ceph-mon[33017]: pgmap v16292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:56.472 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:55 smithi096 ceph-mon[32753]: pgmap v16292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:56.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:55 smithi033 ceph-mon[27845]: pgmap v16292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:09:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:57 smithi194 ceph-mon[33017]: pgmap v16293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:57 smithi033 ceph-mon[27845]: pgmap v16293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:58.205 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:57 smithi096 ceph-mon[32753]: pgmap v16293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:09:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29937.158056 secs 2024-04-26T20:09:59.288 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29937.158056 secs 2024-04-26T20:09:59.321 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29937.158056 secs 2024-04-26T20:10:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29940.078149 secs 2024-04-26T20:10:00.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:09:59 smithi033 ceph-mon[27845]: pgmap v16294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:00.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29940.078149 secs 2024-04-26T20:10:00.338 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:09:59 smithi194 ceph-mon[33017]: pgmap v16294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:00.589 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29940.078149 secs 2024-04-26T20:10:00.813 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:09:59 smithi096 ceph-mon[32753]: pgmap v16294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:10:01.197 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:10:01.496 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:10:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:01 smithi194 ceph-mon[33017]: pgmap v16295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:02.322 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:01 smithi096 ceph-mon[32753]: pgmap v16295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:02.554 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:01 smithi033 ceph-mon[27845]: pgmap v16295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:03 smithi194 ceph-mon[33017]: pgmap v16296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:04.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29942.158136 secs 2024-04-26T20:10:04.455 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:03 smithi096 ceph-mon[32753]: pgmap v16296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:04.693 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29942.158136 secs 2024-04-26T20:10:04.721 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:03 smithi033 ceph-mon[27845]: pgmap v16296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:04.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29942.158136 secs 2024-04-26T20:10:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29945.078267 secs 2024-04-26T20:10:05.305 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29945.078267 secs 2024-04-26T20:10:05.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29945.078267 secs 2024-04-26T20:10:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:05 smithi033 ceph-mon[27845]: pgmap v16297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:06.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:10:06.330 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:05 smithi096 ceph-mon[32753]: pgmap v16297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:06.363 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:10:06.615 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:05 smithi194 ceph-mon[33017]: pgmap v16297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:06.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:10:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:10:07.568 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:10:07.596 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:06 smithi033 ceph-mon[27845]: pgmap v16298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:07.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:06 smithi033 ceph-mon[27845]: pgmap v16299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:10:07.846 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:10:07.897 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:10:08.171 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:10:08.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:06 smithi194 ceph-mon[33017]: pgmap v16298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:08.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:06 smithi194 ceph-mon[33017]: pgmap v16299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:10:08.463 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:10:08.867 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:10:08.921 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:10:08.955 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:06 smithi096 ceph-mon[32753]: pgmap v16298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:08.988 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:06 smithi096 ceph-mon[32753]: pgmap v16299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:10:09.542 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:10:09.964 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:08 smithi033 ceph-mon[27845]: pgmap v16300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:10:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:09.741+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:10.071 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:09.742+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:09.742+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:10:10.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:08 smithi194 ceph-mon[33017]: pgmap v16300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:10:10.338 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29947.158197 secs 2024-04-26T20:10:10.563 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:08 smithi096 ceph-mon[32753]: pgmap v16300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:10:10.596 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29947.158197 secs 2024-04-26T20:10:10.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29947.158197 secs 2024-04-26T20:10:10.897 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29950.078391 secs 2024-04-26T20:10:11.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:10 smithi194 ceph-mon[33017]: pgmap v16301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:10:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29950.078391 secs 2024-04-26T20:10:11.430 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:10 smithi096 ceph-mon[32753]: pgmap v16301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:10:11.463 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29950.078391 secs 2024-04-26T20:10:11.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:10 smithi033 ceph-mon[27845]: pgmap v16301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:10:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:12 smithi194 ceph-mon[33017]: pgmap v16302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:10:13.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:12 smithi096 ceph-mon[32753]: pgmap v16302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:10:13.096 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:12 smithi033 ceph-mon[27845]: pgmap v16302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:10:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29952.158280 secs 2024-04-26T20:10:14.205 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29952.158280 secs 2024-04-26T20:10:14.238 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29952.158280 secs 2024-04-26T20:10:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29955.078541 secs 2024-04-26T20:10:15.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:14 smithi194 ceph-mon[33017]: pgmap v16303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:10:15.230 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29955.078541 secs 2024-04-26T20:10:15.301 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:14 smithi096 ceph-mon[32753]: pgmap v16303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:10:15.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29955.078541 secs 2024-04-26T20:10:15.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:14 smithi033 ceph-mon[27845]: pgmap v16303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:10:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:16 smithi194 ceph-mon[33017]: pgmap v16304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:10:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:16 smithi033 ceph-mon[27845]: pgmap v16304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:10:17.167 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:16 smithi096 ceph-mon[32753]: pgmap v16304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:10:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:18 smithi194 ceph-mon[33017]: pgmap v16305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:19.098 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:18 smithi096 ceph-mon[32753]: pgmap v16305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:19.129 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:18 smithi033 ceph-mon[27845]: pgmap v16305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29957.158359 secs 2024-04-26T20:10:19.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29957.158359 secs 2024-04-26T20:10:19.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29957.158359 secs 2024-04-26T20:10:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29960.078620 secs 2024-04-26T20:10:20.939 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:20 smithi194 ceph-mon[33017]: pgmap v16306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:21.180 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29960.078620 secs 2024-04-26T20:10:21.423 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:20 smithi096 ceph-mon[32753]: pgmap v16306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:21.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29960.078620 secs 2024-04-26T20:10:21.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:20 smithi033 ceph-mon[27845]: pgmap v16306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:22 smithi194 ceph-mon[33017]: pgmap v16307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:22 smithi033 ceph-mon[27845]: pgmap v16307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:23.070 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:22 smithi096 ceph-mon[32753]: pgmap v16307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29962.158437 secs 2024-04-26T20:10:24.296 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29962.158437 secs 2024-04-26T20:10:24.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29962.158437 secs 2024-04-26T20:10:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:24.758+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:24.758+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:24.758+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:10:25.130 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29965.078710 secs 2024-04-26T20:10:25.329 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:24 smithi194 ceph-mon[33017]: pgmap v16308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:25.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29965.078710 secs 2024-04-26T20:10:25.596 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:24 smithi096 ceph-mon[32753]: pgmap v16308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:25.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29965.078710 secs 2024-04-26T20:10:25.663 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:24 smithi033 ceph-mon[27845]: pgmap v16308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:26 smithi194 ceph-mon[33017]: pgmap v16309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:26.996 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:26 smithi096 ceph-mon[32753]: pgmap v16309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:27.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:26 smithi033 ceph-mon[27845]: pgmap v16309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:28 smithi194 ceph-mon[33017]: pgmap v16310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:29.121 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:28 smithi096 ceph-mon[32753]: pgmap v16310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:29.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:28 smithi033 ceph-mon[27845]: pgmap v16310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29967.158507 secs 2024-04-26T20:10:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29967.158507 secs 2024-04-26T20:10:29.946 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29967.158507 secs 2024-04-26T20:10:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29970.078844 secs 2024-04-26T20:10:30.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:30 smithi194 ceph-mon[33017]: pgmap v16311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:31.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29970.078844 secs 2024-04-26T20:10:31.287 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:30 smithi096 ceph-mon[32753]: pgmap v16311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:31.320 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29970.078844 secs 2024-04-26T20:10:31.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:30 smithi033 ceph-mon[27845]: pgmap v16311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:32 smithi194 ceph-mon[33017]: pgmap v16312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:32 smithi033 ceph-mon[27845]: pgmap v16312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:33.162 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:32 smithi096 ceph-mon[32753]: pgmap v16312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29972.158612 secs 2024-04-26T20:10:34.254 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29972.158612 secs 2024-04-26T20:10:34.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29972.158612 secs 2024-04-26T20:10:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29975.078960 secs 2024-04-26T20:10:35.270 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:34 smithi194 ceph-mon[33017]: pgmap v16313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:35.304 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29975.078960 secs 2024-04-26T20:10:35.337 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:34 smithi096 ceph-mon[32753]: pgmap v16313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:35.545 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29975.078960 secs 2024-04-26T20:10:35.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:34 smithi033 ceph-mon[27845]: pgmap v16313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:36 smithi194 ceph-mon[33017]: pgmap v16314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:37.006 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:36 smithi096 ceph-mon[32753]: pgmap v16314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:37.037 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:36 smithi033 ceph-mon[27845]: pgmap v16314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:38 smithi194 ceph-mon[33017]: pgmap v16315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:38.987 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:38 smithi096 ceph-mon[32753]: pgmap v16315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:39.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:38 smithi033 ceph-mon[27845]: pgmap v16315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:39.768+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:39.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:39.768+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:10:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:39.768+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:40.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29977.158696 secs 2024-04-26T20:10:40.337 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29977.158696 secs 2024-04-26T20:10:40.370 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29977.158696 secs 2024-04-26T20:10:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29980.079130 secs 2024-04-26T20:10:40.936 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:40 smithi194 ceph-mon[33017]: pgmap v16316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:40.971 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29980.079130 secs 2024-04-26T20:10:41.011 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:40 smithi096 ceph-mon[32753]: pgmap v16316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:41.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29980.079130 secs 2024-04-26T20:10:41.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:40 smithi033 ceph-mon[27845]: pgmap v16316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:42 smithi194 ceph-mon[33017]: pgmap v16317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:43.091 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:42 smithi096 ceph-mon[32753]: pgmap v16317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:43.161 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:42 smithi033 ceph-mon[27845]: pgmap v16317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29982.158776 secs 2024-04-26T20:10:44.195 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29982.158776 secs 2024-04-26T20:10:44.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29982.158776 secs 2024-04-26T20:10:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29985.079189 secs 2024-04-26T20:10:45.211 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:44 smithi194 ceph-mon[33017]: pgmap v16318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:45.245 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29985.079189 secs 2024-04-26T20:10:45.520 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:44 smithi096 ceph-mon[32753]: pgmap v16318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:45.554 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29985.079189 secs 2024-04-26T20:10:45.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:44 smithi033 ceph-mon[27845]: pgmap v16318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:46 smithi194 ceph-mon[33017]: pgmap v16319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:46.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:46 smithi033 ceph-mon[27845]: pgmap v16319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:46.961 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:46 smithi096 ceph-mon[32753]: pgmap v16319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:48 smithi194 ceph-mon[33017]: pgmap v16320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:49.040 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:48 smithi096 ceph-mon[32753]: pgmap v16320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:49.069 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:48 smithi033 ceph-mon[27845]: pgmap v16320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29987.158859 secs 2024-04-26T20:10:49.961 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29987.158859 secs 2024-04-26T20:10:50.094 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29987.158859 secs 2024-04-26T20:10:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29990.079255 secs 2024-04-26T20:10:50.897 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:50 smithi194 ceph-mon[33017]: pgmap v16321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:50.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29990.079255 secs 2024-04-26T20:10:50.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:50 smithi033 ceph-mon[27845]: pgmap v16321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:51.178 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29990.079255 secs 2024-04-26T20:10:51.211 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:50 smithi096 ceph-mon[32753]: pgmap v16321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:10:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:52 smithi194 ceph-mon[33017]: pgmap v16322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:52 smithi033 ceph-mon[27845]: pgmap v16322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:53.169 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:52 smithi096 ceph-mon[32753]: pgmap v16322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:10:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29992.158939 secs 2024-04-26T20:10:54.270 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29992.158939 secs 2024-04-26T20:10:54.519 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29992.158939 secs 2024-04-26T20:10:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:54.782+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:54.782+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:10:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:10:54.782+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:10:55.270 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 29995.079340 secs 2024-04-26T20:10:55.302 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:54 smithi096 ceph-mon[32753]: pgmap v16323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:55.360 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 29995.079340 secs 2024-04-26T20:10:55.664 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:54 smithi194 ceph-mon[33017]: pgmap v16323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:55.742 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 29995.079340 secs 2024-04-26T20:10:55.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:54 smithi033 ceph-mon[27845]: pgmap v16323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:10:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:56 smithi194 ceph-mon[33017]: pgmap v16324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:56.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:56 smithi096 ceph-mon[32753]: pgmap v16324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:57.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:56 smithi033 ceph-mon[27845]: pgmap v16324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:58 smithi194 ceph-mon[33017]: pgmap v16325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:58.994 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:58 smithi096 ceph-mon[32753]: pgmap v16325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:59.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:58 smithi033 ceph-mon[27845]: pgmap v16325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:10:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:10:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 29997.159025 secs 2024-04-26T20:10:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:10:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 29997.159025 secs 2024-04-26T20:11:00.111 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:10:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 29997.159025 secs 2024-04-26T20:11:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30000.079465 secs 2024-04-26T20:11:00.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:00 smithi194 ceph-mon[33017]: pgmap v16326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:00.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30000.079465 secs 2024-04-26T20:11:01.127 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:00 smithi096 ceph-mon[32753]: pgmap v16326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:01.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30000.079465 secs 2024-04-26T20:11:01.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:00 smithi033 ceph-mon[27845]: pgmap v16326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:02 smithi194 ceph-mon[33017]: pgmap v16327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:11:02.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:02 smithi033 ceph-mon[27845]: pgmap v16327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:11:02.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:02 smithi096 ceph-mon[32753]: pgmap v16327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:11:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30002.159083 secs 2024-04-26T20:11:04.326 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30002.159083 secs 2024-04-26T20:11:04.626 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30002.159083 secs 2024-04-26T20:11:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30005.079528 secs 2024-04-26T20:11:05.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:04 smithi194 ceph-mon[33017]: pgmap v16328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:05.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30005.079528 secs 2024-04-26T20:11:05.210 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:04 smithi096 ceph-mon[32753]: pgmap v16328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:05.235 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30005.079528 secs 2024-04-26T20:11:05.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:04 smithi033 ceph-mon[27845]: pgmap v16328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:06 smithi194 ceph-mon[33017]: pgmap v16329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:07.035 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:06 smithi096 ceph-mon[32753]: pgmap v16329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:07.107 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:06 smithi033 ceph-mon[27845]: pgmap v16329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:11:07.893 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.143 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.364 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.394 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:11:08.418 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.443 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.501 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.701 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.726 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:11:08.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.935 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:08.985 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30007.159151 secs 2024-04-26T20:11:09.826 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:09 smithi033 ceph-mon[27845]: pgmap v16330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:09.876 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:10.035 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:10.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30007.159151 secs 2024-04-26T20:11:10.131 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:09 smithi194 ceph-mon[33017]: pgmap v16330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:10.343 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:10.368 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:10.610 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30007.159151 secs 2024-04-26T20:11:10.843 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:09 smithi096 ceph-mon[32753]: pgmap v16330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:10.868 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:11.135 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:09.797+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:09.797+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:11:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:09.797+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:11.419 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30010.079596 secs 2024-04-26T20:11:11.452 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:11.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:11.694 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:11.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:11.977 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:12.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:12.294 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:12.527 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:12.561 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:12.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:12.794 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:11:12.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:11:13.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.111 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.336 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.369 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.436 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:13.645 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:11:13.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:11:13.928 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:10 smithi194 ceph-mon[33017]: pgmap v16331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:14.374 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30010.079596 secs 2024-04-26T20:11:14.421 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:14.461 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:14.678 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:14.712 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:14.737 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:14.970 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:15.003 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:15.170 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:15.203 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:15.283 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:15.479 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:11:15.531 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:11:15.720 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:15.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:15.987 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:16.020 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:16.262 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:16.537 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:16.712 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:16.745 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:16.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:11:17.020 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:11:17.054 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:10 smithi096 ceph-mon[32753]: pgmap v16331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:17.108 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30010.079596 secs 2024-04-26T20:11:17.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:17.387 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:17.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:17.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:17.913 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:17.954 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:18.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:18.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:18.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:18.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:18.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:11:18.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:11:19.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.212 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.471 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.554 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:19.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:11:19.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:11:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:10 smithi033 ceph-mon[27845]: pgmap v16331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:20.232 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:20.263 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:20.296 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:12 smithi194 ceph-mon[33017]: pgmap v16332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:20.521 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:20.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:21.012 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:21.279 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:21.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:21.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:21.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:21.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:22.021 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:11:22.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:11:22.215 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:11:22.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:11:22.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: pgmap v16333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:11:22.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: pgmap v16334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:11:22.743 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:23.458 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30012.159232 secs 2024-04-26T20:11:23.588 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30015.079728 secs 2024-04-26T20:11:23.621 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:15 smithi194 ceph-mon[33017]: pgmap v16335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:23.682 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:17 smithi194 ceph-mon[33017]: pgmap v16336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:24.154 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30017.159315 secs 2024-04-26T20:11:24.196 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:24.266 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:24.454 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:12 smithi096 ceph-mon[32753]: pgmap v16332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:24.631 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:24.704 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:24.738 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:24.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:24.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:24.996 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:25.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:25.296 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:25.329 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:11:25.563 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:11:25.596 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:11:25.846 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:11:25.879 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: pgmap v16333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:11:25.923 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: pgmap v16334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:11:26.080 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:26.113 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30012.159232 secs 2024-04-26T20:11:26.163 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30015.079728 secs 2024-04-26T20:11:26.196 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:15 smithi096 ceph-mon[32753]: pgmap v16335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:26.413 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:17 smithi096 ceph-mon[32753]: pgmap v16336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:26.446 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30017.159315 secs 2024-04-26T20:11:26.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:26.721 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:26.738 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:12 smithi033 ceph-mon[27845]: pgmap v16332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:26.787 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:26.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:26.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:27.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:27.038 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:27.172 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:27.255 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:27.271 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:11:27.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:11:27.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:11:27.538 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:11:27.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:11:27.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: pgmap v16333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:11:27.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: pgmap v16334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:11:28.063 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:11:28.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30012.159232 secs 2024-04-26T20:11:28.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30015.079728 secs 2024-04-26T20:11:28.373 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:15 smithi033 ceph-mon[27845]: pgmap v16335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:28.555 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:17 smithi033 ceph-mon[27845]: pgmap v16336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:28.813 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30017.159315 secs 2024-04-26T20:11:29.006 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30020.079844 secs 2024-04-26T20:11:29.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:19 smithi033 ceph-mon[27845]: pgmap v16337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:11:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T20:11:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:11:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0062070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:11:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0062070 comp rx=0 tx=0).stop 2024-04-26T20:11:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T20:11:29.422 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T20:11:29.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.799+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.800+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:30.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:21.802+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:30.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30020.079844 secs 2024-04-26T20:11:30.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:19 smithi194 ceph-mon[33017]: pgmap v16337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:11:30.750 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:21 smithi194 ceph-mon[33017]: pgmap v16338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:30.868 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:23 smithi194 ceph-mon[33017]: pgmap v16339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:11:30.913 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30022.159398 secs 2024-04-26T20:11:31.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30025.079977 secs 2024-04-26T20:11:31.264 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:25 smithi194 ceph-mon[33017]: pgmap v16340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:31.297 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:28 smithi194 ceph-mon[33017]: pgmap v16341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:31.405 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30020.079844 secs 2024-04-26T20:11:31.530 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:19 smithi096 ceph-mon[32753]: pgmap v16337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:11:31.580 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:21 smithi096 ceph-mon[32753]: pgmap v16338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:31.614 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:23 smithi096 ceph-mon[32753]: pgmap v16339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:11:31.680 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30022.159398 secs 2024-04-26T20:11:31.830 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30025.079977 secs 2024-04-26T20:11:31.864 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:25 smithi096 ceph-mon[32753]: pgmap v16340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:32.097 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:28 smithi096 ceph-mon[32753]: pgmap v16341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:32.132 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:21 smithi033 ceph-mon[27845]: pgmap v16338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:11:32.439 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:23 smithi033 ceph-mon[27845]: pgmap v16339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:11:32.630 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30022.159398 secs 2024-04-26T20:11:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:24.801+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:24.801+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:24.801+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:11:32.965 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30025.079977 secs 2024-04-26T20:11:32.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:25 smithi033 ceph-mon[27845]: pgmap v16340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:33.257 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30027.159482 secs 2024-04-26T20:11:33.448 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30030.080129 secs 2024-04-26T20:11:33.556 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:30 smithi194 ceph-mon[33017]: pgmap v16342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:33.797 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:32 smithi194 ceph-mon[33017]: pgmap v16343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:33.831 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30027.159482 secs 2024-04-26T20:11:34.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30030.080129 secs 2024-04-26T20:11:34.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:30 smithi096 ceph-mon[32753]: pgmap v16342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:34.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:32 smithi096 ceph-mon[32753]: pgmap v16343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:26.902+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:11:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:26.902+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T20:11:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:26.903+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 236 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:11:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:26.903+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 236 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:11:35.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:28 smithi033 ceph-mon[27845]: pgmap v16341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:35.074 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30027.159482 secs 2024-04-26T20:11:35.322 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30030.080129 secs 2024-04-26T20:11:35.356 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:30 smithi033 ceph-mon[27845]: pgmap v16342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:35.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:32 smithi033 ceph-mon[27845]: pgmap v16343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:35.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:34 smithi033 ceph-mon[27845]: pgmap v16344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:35.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30032.159564 secs 2024-04-26T20:11:36.148 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30035.080247 secs 2024-04-26T20:11:36.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:34 smithi194 ceph-mon[33017]: pgmap v16344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:36.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30032.159564 secs 2024-04-26T20:11:36.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30035.080247 secs 2024-04-26T20:11:36.714 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:34 smithi096 ceph-mon[32753]: pgmap v16344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:36.747 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30032.159564 secs 2024-04-26T20:11:36.972 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30035.080247 secs 2024-04-26T20:11:37.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:36 smithi033 ceph-mon[27845]: pgmap v16345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:37.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:36 smithi194 ceph-mon[33017]: pgmap v16345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:37.256 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:36 smithi096 ceph-mon[32753]: pgmap v16345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:38 smithi194 ceph-mon[33017]: pgmap v16346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:38.957 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:38 smithi096 ceph-mon[32753]: pgmap v16346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:39.006 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:38 smithi033 ceph-mon[27845]: pgmap v16346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:39.815+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:39.815+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:11:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:39.815+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:40.115 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30037.159649 secs 2024-04-26T20:11:40.148 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:39 smithi096 ceph-mon[32753]: pgmap v16347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:40.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30037.159649 secs 2024-04-26T20:11:40.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:39 smithi194 ceph-mon[33017]: pgmap v16347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:40.398 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30037.159649 secs 2024-04-26T20:11:40.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:39 smithi033 ceph-mon[27845]: pgmap v16347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30040.080342 secs 2024-04-26T20:11:40.957 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30040.080342 secs 2024-04-26T20:11:41.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30040.080342 secs 2024-04-26T20:11:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:41 smithi096 ceph-mon[32753]: pgmap v16348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:41.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:41 smithi194 ceph-mon[33017]: pgmap v16348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:42.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:41 smithi033 ceph-mon[27845]: pgmap v16348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:44.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:43 smithi033 ceph-mon[27845]: pgmap v16349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:44.339 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30042.159731 secs 2024-04-26T20:11:44.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:43 smithi194 ceph-mon[33017]: pgmap v16349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:44.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30042.159731 secs 2024-04-26T20:11:44.665 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:43 smithi096 ceph-mon[32753]: pgmap v16349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:44.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30042.159731 secs 2024-04-26T20:11:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30045.080443 secs 2024-04-26T20:11:45.440 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30045.080443 secs 2024-04-26T20:11:45.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30045.080443 secs 2024-04-26T20:11:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:45 smithi194 ceph-mon[33017]: pgmap v16350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:46.456 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:45 smithi096 ceph-mon[32753]: pgmap v16350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:45 smithi033 ceph-mon[27845]: pgmap v16350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:47 smithi194 ceph-mon[33017]: pgmap v16351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:48.523 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:47 smithi096 ceph-mon[32753]: pgmap v16351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:48.555 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:47 smithi033 ceph-mon[27845]: pgmap v16351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30047.159818 secs 2024-04-26T20:11:49.508 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30047.159818 secs 2024-04-26T20:11:49.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30047.159818 secs 2024-04-26T20:11:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30050.080557 secs 2024-04-26T20:11:50.389 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:50 smithi194 ceph-mon[33017]: pgmap v16352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:50.589 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30050.080557 secs 2024-04-26T20:11:50.643 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:50 smithi096 ceph-mon[32753]: pgmap v16352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:50.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30050.080557 secs 2024-04-26T20:11:50.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:50 smithi033 ceph-mon[27845]: pgmap v16352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:52 smithi194 ceph-mon[33017]: pgmap v16353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:52.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:52 smithi096 ceph-mon[32753]: pgmap v16353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:52.739 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:52 smithi033 ceph-mon[27845]: pgmap v16353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:54 smithi194 ceph-mon[33017]: pgmap v16354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:54.413 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30052.159898 secs 2024-04-26T20:11:54.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:54 smithi096 ceph-mon[32753]: pgmap v16354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:54.688 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30052.159898 secs 2024-04-26T20:11:54.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:54 smithi033 ceph-mon[27845]: pgmap v16354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:54.955 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30052.159898 secs 2024-04-26T20:11:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:54.821+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:54.822+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:11:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:11:54.822+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:11:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30055.080674 secs 2024-04-26T20:11:55.431 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30055.080674 secs 2024-04-26T20:11:55.464 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30055.080674 secs 2024-04-26T20:11:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:56 smithi194 ceph-mon[33017]: pgmap v16355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:56 smithi033 ceph-mon[27845]: pgmap v16355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:56.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:56 smithi096 ceph-mon[32753]: pgmap v16355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:58 smithi194 ceph-mon[33017]: pgmap v16356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:58.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:58 smithi096 ceph-mon[32753]: pgmap v16356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:58.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:58 smithi033 ceph-mon[27845]: pgmap v16356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:11:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:11:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30057.160000 secs 2024-04-26T20:11:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:11:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30057.160000 secs 2024-04-26T20:11:59.649 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:11:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30057.160000 secs 2024-04-26T20:12:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30060.080807 secs 2024-04-26T20:12:00.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:00 smithi033 ceph-mon[27845]: pgmap v16357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:00.539 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30060.080807 secs 2024-04-26T20:12:00.763 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:00 smithi194 ceph-mon[33017]: pgmap v16357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:00.796 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30060.080807 secs 2024-04-26T20:12:00.997 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:00 smithi096 ceph-mon[32753]: pgmap v16357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:02 smithi194 ceph-mon[33017]: pgmap v16358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:02.530 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:02 smithi096 ceph-mon[32753]: pgmap v16358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:02.563 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:02 smithi033 ceph-mon[27845]: pgmap v16358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30062.160058 secs 2024-04-26T20:12:04.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:04 smithi194 ceph-mon[33017]: pgmap v16359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:04.713 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30062.160058 secs 2024-04-26T20:12:04.778 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:04 smithi096 ceph-mon[32753]: pgmap v16359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:04.985 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30062.160058 secs 2024-04-26T20:12:05.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:04 smithi033 ceph-mon[27845]: pgmap v16359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:05.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30065.080872 secs 2024-04-26T20:12:05.471 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30065.080872 secs 2024-04-26T20:12:05.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30065.080872 secs 2024-04-26T20:12:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:06 smithi194 ceph-mon[33017]: pgmap v16360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:06.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:06 smithi096 ceph-mon[32753]: pgmap v16360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:06.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:06 smithi033 ceph-mon[27845]: pgmap v16360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:08 smithi194 ceph-mon[33017]: pgmap v16361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:08.596 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:08 smithi096 ceph-mon[32753]: pgmap v16361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:08.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:08 smithi033 ceph-mon[27845]: pgmap v16361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30067.160137 secs 2024-04-26T20:12:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30067.160137 secs 2024-04-26T20:12:09.688 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30067.160137 secs 2024-04-26T20:12:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:09.823+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:09.823+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:09.823+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:12:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30070.080996 secs 2024-04-26T20:12:10.454 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:10 smithi194 ceph-mon[33017]: pgmap v16362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:10.488 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30070.080996 secs 2024-04-26T20:12:10.528 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:10 smithi096 ceph-mon[32753]: pgmap v16362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:10.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30070.080996 secs 2024-04-26T20:12:10.804 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:10 smithi033 ceph-mon[27845]: pgmap v16362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:12 smithi194 ceph-mon[33017]: pgmap v16363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:12 smithi033 ceph-mon[27845]: pgmap v16363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:12.613 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:12 smithi096 ceph-mon[32753]: pgmap v16363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30072.160219 secs 2024-04-26T20:12:14.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:14 smithi194 ceph-mon[33017]: pgmap v16364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:14.724 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:12:14.725 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30072.160219 secs 2024-04-26T20:12:14.725 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:14 smithi096 ceph-mon[32753]: pgmap v16364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:14.725 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:12:14.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30072.160219 secs 2024-04-26T20:12:14.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:14 smithi033 ceph-mon[27845]: pgmap v16364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:14.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:12:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30075.081082 secs 2024-04-26T20:12:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:12:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:12:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:12:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30075.081082 secs 2024-04-26T20:12:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:12:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:12:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:12:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30075.081082 secs 2024-04-26T20:12:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:12:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:12:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:12:16.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:16 smithi033 ceph-mon[27845]: pgmap v16365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:12:16.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:16 smithi033 ceph-mon[27845]: pgmap v16366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:16 smithi096 ceph-mon[32753]: pgmap v16365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:12:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:16 smithi096 ceph-mon[32753]: pgmap v16366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:16 smithi194 ceph-mon[33017]: pgmap v16365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:12:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:16 smithi194 ceph-mon[33017]: pgmap v16366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:18 smithi033 ceph-mon[27845]: pgmap v16367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:18 smithi096 ceph-mon[32753]: pgmap v16367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:18 smithi194 ceph-mon[33017]: pgmap v16367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30077.160303 secs 2024-04-26T20:12:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30077.160303 secs 2024-04-26T20:12:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30077.160303 secs 2024-04-26T20:12:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:20 smithi033 ceph-mon[27845]: pgmap v16368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30080.081173 secs 2024-04-26T20:12:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:20 smithi096 ceph-mon[32753]: pgmap v16368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30080.081173 secs 2024-04-26T20:12:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:20 smithi194 ceph-mon[33017]: pgmap v16368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30080.081173 secs 2024-04-26T20:12:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:22 smithi033 ceph-mon[27845]: pgmap v16369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:22 smithi096 ceph-mon[32753]: pgmap v16369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:22 smithi194 ceph-mon[33017]: pgmap v16369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:24 smithi033 ceph-mon[27845]: pgmap v16370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30082.160361 secs 2024-04-26T20:12:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:24 smithi096 ceph-mon[32753]: pgmap v16370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30082.160361 secs 2024-04-26T20:12:24.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:24 smithi194 ceph-mon[33017]: pgmap v16370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:12:24.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30082.160361 secs 2024-04-26T20:12:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:24.834+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:24.834+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:24.834+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:12:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30085.081235 secs 2024-04-26T20:12:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30085.081235 secs 2024-04-26T20:12:25.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30085.081235 secs 2024-04-26T20:12:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:26 smithi033 ceph-mon[27845]: pgmap v16371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:12:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:26 smithi096 ceph-mon[32753]: pgmap v16371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:12:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:26 smithi194 ceph-mon[33017]: pgmap v16371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:12:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:28 smithi033 ceph-mon[27845]: pgmap v16372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:28 smithi096 ceph-mon[32753]: pgmap v16372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:28 smithi194 ceph-mon[33017]: pgmap v16372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30087.160447 secs 2024-04-26T20:12:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30087.160447 secs 2024-04-26T20:12:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30087.160447 secs 2024-04-26T20:12:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:30 smithi033 ceph-mon[27845]: pgmap v16373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30090.081336 secs 2024-04-26T20:12:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:30 smithi096 ceph-mon[32753]: pgmap v16373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30090.081336 secs 2024-04-26T20:12:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:30 smithi194 ceph-mon[33017]: pgmap v16373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30090.081336 secs 2024-04-26T20:12:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:32 smithi096 ceph-mon[32753]: pgmap v16374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:32 smithi194 ceph-mon[33017]: pgmap v16374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:32 smithi033 ceph-mon[27845]: pgmap v16374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:34 smithi096 ceph-mon[32753]: pgmap v16375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30092.160530 secs 2024-04-26T20:12:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:34 smithi194 ceph-mon[33017]: pgmap v16375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30092.160530 secs 2024-04-26T20:12:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:34 smithi033 ceph-mon[27845]: pgmap v16375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30092.160530 secs 2024-04-26T20:12:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30095.081460 secs 2024-04-26T20:12:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30095.081460 secs 2024-04-26T20:12:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30095.081460 secs 2024-04-26T20:12:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:36 smithi194 ceph-mon[33017]: pgmap v16376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:36 smithi033 ceph-mon[27845]: pgmap v16376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:36 smithi096 ceph-mon[32753]: pgmap v16376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:38 smithi194 ceph-mon[33017]: pgmap v16377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:38 smithi033 ceph-mon[27845]: pgmap v16377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:38 smithi096 ceph-mon[32753]: pgmap v16377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30097.160616 secs 2024-04-26T20:12:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30097.160616 secs 2024-04-26T20:12:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30097.160616 secs 2024-04-26T20:12:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:39.849+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:39.849+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:39.849+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:12:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:40 smithi194 ceph-mon[33017]: pgmap v16378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30100.081577 secs 2024-04-26T20:12:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:40 smithi033 ceph-mon[27845]: pgmap v16378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30100.081577 secs 2024-04-26T20:12:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:40 smithi096 ceph-mon[32753]: pgmap v16378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30100.081577 secs 2024-04-26T20:12:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:42 smithi194 ceph-mon[33017]: pgmap v16379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:42 smithi033 ceph-mon[27845]: pgmap v16379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:42 smithi096 ceph-mon[32753]: pgmap v16379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:44 smithi194 ceph-mon[33017]: pgmap v16380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30102.160697 secs 2024-04-26T20:12:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:44 smithi033 ceph-mon[27845]: pgmap v16380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30102.160697 secs 2024-04-26T20:12:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:44 smithi096 ceph-mon[32753]: pgmap v16380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30102.160697 secs 2024-04-26T20:12:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30105.081655 secs 2024-04-26T20:12:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30105.081655 secs 2024-04-26T20:12:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30105.081655 secs 2024-04-26T20:12:46.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:46 smithi033 ceph-mon[27845]: pgmap v16381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:46 smithi194 ceph-mon[33017]: pgmap v16381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:46 smithi096 ceph-mon[32753]: pgmap v16381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:48 smithi194 ceph-mon[33017]: pgmap v16382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:48 smithi033 ceph-mon[27845]: pgmap v16382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:48 smithi096 ceph-mon[32753]: pgmap v16382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30107.160783 secs 2024-04-26T20:12:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30107.160783 secs 2024-04-26T20:12:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30107.160783 secs 2024-04-26T20:12:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:50 smithi194 ceph-mon[33017]: pgmap v16383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30110.081771 secs 2024-04-26T20:12:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:50 smithi033 ceph-mon[27845]: pgmap v16383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30110.081771 secs 2024-04-26T20:12:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:50 smithi096 ceph-mon[32753]: pgmap v16383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30110.081771 secs 2024-04-26T20:12:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:52 smithi194 ceph-mon[33017]: pgmap v16384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:52 smithi033 ceph-mon[27845]: pgmap v16384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:52 smithi096 ceph-mon[32753]: pgmap v16384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:54 smithi194 ceph-mon[33017]: pgmap v16385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30112.160865 secs 2024-04-26T20:12:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:54 smithi033 ceph-mon[27845]: pgmap v16385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30112.160865 secs 2024-04-26T20:12:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:54 smithi096 ceph-mon[32753]: pgmap v16385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30112.160865 secs 2024-04-26T20:12:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:54.863+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:54.863+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:12:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:54.863+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:12:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30115.081867 secs 2024-04-26T20:12:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30115.081867 secs 2024-04-26T20:12:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30115.081867 secs 2024-04-26T20:12:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:56 smithi194 ceph-mon[33017]: pgmap v16386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:56 smithi033 ceph-mon[27845]: pgmap v16386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:56 smithi096 ceph-mon[32753]: pgmap v16386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:56.916+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T20:12:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:56.916+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:12:56.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:56.917+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 237 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T20:12:56.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:12:56.917+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 237 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:12:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:58 smithi194 ceph-mon[33017]: pgmap v16387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:58 smithi033 ceph-mon[27845]: pgmap v16387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:58 smithi096 ceph-mon[32753]: pgmap v16387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:12:59.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:12:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30117.160948 secs 2024-04-26T20:12:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:12:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30117.160948 secs 2024-04-26T20:12:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:12:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30117.160948 secs 2024-04-26T20:13:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:00 smithi033 ceph-mon[27845]: pgmap v16388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30120.081989 secs 2024-04-26T20:13:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:00 smithi096 ceph-mon[32753]: pgmap v16388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30120.081989 secs 2024-04-26T20:13:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:00 smithi194 ceph-mon[33017]: pgmap v16388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30120.081989 secs 2024-04-26T20:13:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:02 smithi033 ceph-mon[27845]: pgmap v16389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:02 smithi194 ceph-mon[33017]: pgmap v16389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:02 smithi096 ceph-mon[32753]: pgmap v16389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:04 smithi194 ceph-mon[33017]: pgmap v16390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30122.161030 secs 2024-04-26T20:13:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:04 smithi033 ceph-mon[27845]: pgmap v16390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30122.161030 secs 2024-04-26T20:13:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:04 smithi096 ceph-mon[32753]: pgmap v16390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30122.161030 secs 2024-04-26T20:13:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30125.082141 secs 2024-04-26T20:13:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30125.082141 secs 2024-04-26T20:13:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30125.082141 secs 2024-04-26T20:13:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:06 smithi194 ceph-mon[33017]: pgmap v16391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:06 smithi033 ceph-mon[27845]: pgmap v16391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:06 smithi096 ceph-mon[32753]: pgmap v16391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:08 smithi194 ceph-mon[33017]: pgmap v16392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:08 smithi033 ceph-mon[27845]: pgmap v16392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:08 smithi096 ceph-mon[32753]: pgmap v16392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30127.161117 secs 2024-04-26T20:13:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30127.161117 secs 2024-04-26T20:13:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30127.161117 secs 2024-04-26T20:13:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:09.863+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:09.863+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:09.863+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:13:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:10 smithi194 ceph-mon[33017]: pgmap v16393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30130.082196 secs 2024-04-26T20:13:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:10 smithi033 ceph-mon[27845]: pgmap v16393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30130.082196 secs 2024-04-26T20:13:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:10 smithi096 ceph-mon[32753]: pgmap v16393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30130.082196 secs 2024-04-26T20:13:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:12 smithi194 ceph-mon[33017]: pgmap v16394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:12 smithi033 ceph-mon[27845]: pgmap v16394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:12 smithi096 ceph-mon[32753]: pgmap v16394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:14 smithi194 ceph-mon[33017]: pgmap v16395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30132.161173 secs 2024-04-26T20:13:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:14 smithi033 ceph-mon[27845]: pgmap v16395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30132.161173 secs 2024-04-26T20:13:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:14 smithi096 ceph-mon[32753]: pgmap v16395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30132.161173 secs 2024-04-26T20:13:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30135.082333 secs 2024-04-26T20:13:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:13:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30135.082333 secs 2024-04-26T20:13:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:13:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30135.082333 secs 2024-04-26T20:13:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:13:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:16 smithi033 ceph-mon[27845]: pgmap v16396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:13:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:13:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:13:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:16 smithi096 ceph-mon[32753]: pgmap v16396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:13:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:13:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:13:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:16 smithi194 ceph-mon[33017]: pgmap v16396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:13:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:13:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:13:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:17 smithi033 ceph-mon[27845]: pgmap v16397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:17 smithi033 ceph-mon[27845]: pgmap v16398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:17 smithi096 ceph-mon[32753]: pgmap v16397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:17 smithi096 ceph-mon[32753]: pgmap v16398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:17 smithi194 ceph-mon[33017]: pgmap v16397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:17 smithi194 ceph-mon[33017]: pgmap v16398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30137.161238 secs 2024-04-26T20:13:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:19 smithi033 ceph-mon[27845]: pgmap v16399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30137.161238 secs 2024-04-26T20:13:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:19 smithi096 ceph-mon[32753]: pgmap v16399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30137.161238 secs 2024-04-26T20:13:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:19 smithi194 ceph-mon[33017]: pgmap v16399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30140.082451 secs 2024-04-26T20:13:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30140.082451 secs 2024-04-26T20:13:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30140.082451 secs 2024-04-26T20:13:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:21 smithi033 ceph-mon[27845]: pgmap v16400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:21 smithi096 ceph-mon[32753]: pgmap v16400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:21 smithi194 ceph-mon[33017]: pgmap v16400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:23 smithi033 ceph-mon[27845]: pgmap v16401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:23 smithi096 ceph-mon[32753]: pgmap v16401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:23 smithi194 ceph-mon[33017]: pgmap v16401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:13:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30142.161294 secs 2024-04-26T20:13:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:24.873+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:24.873+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:24.873+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:13:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30142.161294 secs 2024-04-26T20:13:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30142.161294 secs 2024-04-26T20:13:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30145.082568 secs 2024-04-26T20:13:25.924 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:25 smithi194 ceph-mon[33017]: pgmap v16402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:13:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30145.082568 secs 2024-04-26T20:13:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:25 smithi033 ceph-mon[27845]: pgmap v16402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:13:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30145.082568 secs 2024-04-26T20:13:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:25 smithi096 ceph-mon[32753]: pgmap v16402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:13:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:27 smithi194 ceph-mon[33017]: pgmap v16403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:13:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:27 smithi033 ceph-mon[27845]: pgmap v16403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:13:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:27 smithi096 ceph-mon[32753]: pgmap v16403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:13:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30147.161356 secs 2024-04-26T20:13:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:29 smithi194 ceph-mon[33017]: pgmap v16404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30147.161356 secs 2024-04-26T20:13:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:29 smithi033 ceph-mon[27845]: pgmap v16404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30147.161356 secs 2024-04-26T20:13:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:29 smithi096 ceph-mon[32753]: pgmap v16404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30150.082643 secs 2024-04-26T20:13:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30150.082643 secs 2024-04-26T20:13:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30150.082643 secs 2024-04-26T20:13:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:31 smithi194 ceph-mon[33017]: pgmap v16405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:31 smithi033 ceph-mon[27845]: pgmap v16405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:31 smithi096 ceph-mon[32753]: pgmap v16405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:32.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:32 smithi033 ceph-mon[27845]: pgmap v16406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:32 smithi194 ceph-mon[33017]: pgmap v16406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:32 smithi096 ceph-mon[32753]: pgmap v16406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30152.161409 secs 2024-04-26T20:13:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30152.161409 secs 2024-04-26T20:13:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30152.161409 secs 2024-04-26T20:13:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30155.082760 secs 2024-04-26T20:13:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:34 smithi194 ceph-mon[33017]: pgmap v16407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30155.082760 secs 2024-04-26T20:13:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:34 smithi033 ceph-mon[27845]: pgmap v16407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30155.082760 secs 2024-04-26T20:13:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:34 smithi096 ceph-mon[32753]: pgmap v16407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:37 smithi194 ceph-mon[33017]: pgmap v16408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:37 smithi033 ceph-mon[27845]: pgmap v16408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:37 smithi096 ceph-mon[32753]: pgmap v16408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30157.161470 secs 2024-04-26T20:13:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:39 smithi194 ceph-mon[33017]: pgmap v16409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30157.161470 secs 2024-04-26T20:13:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:39 smithi033 ceph-mon[27845]: pgmap v16409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30157.161470 secs 2024-04-26T20:13:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:39 smithi096 ceph-mon[32753]: pgmap v16409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:39.883+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:39.883+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:13:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:39.883+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30160.082876 secs 2024-04-26T20:13:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30160.082876 secs 2024-04-26T20:13:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30160.082876 secs 2024-04-26T20:13:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:41 smithi194 ceph-mon[33017]: pgmap v16410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:41 smithi033 ceph-mon[27845]: pgmap v16410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:41 smithi096 ceph-mon[32753]: pgmap v16410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:43 smithi194 ceph-mon[33017]: pgmap v16411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:43.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:43 smithi033 ceph-mon[27845]: pgmap v16411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:43 smithi096 ceph-mon[32753]: pgmap v16411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30162.161524 secs 2024-04-26T20:13:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30162.161524 secs 2024-04-26T20:13:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30162.161524 secs 2024-04-26T20:13:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30165.082940 secs 2024-04-26T20:13:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:45 smithi194 ceph-mon[33017]: pgmap v16412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30165.082940 secs 2024-04-26T20:13:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:45 smithi033 ceph-mon[27845]: pgmap v16412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30165.082940 secs 2024-04-26T20:13:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:45 smithi096 ceph-mon[32753]: pgmap v16412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:47 smithi194 ceph-mon[33017]: pgmap v16413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:47 smithi033 ceph-mon[27845]: pgmap v16413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:47 smithi096 ceph-mon[32753]: pgmap v16413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30167.161583 secs 2024-04-26T20:13:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:49 smithi033 ceph-mon[27845]: pgmap v16414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30167.161583 secs 2024-04-26T20:13:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:49 smithi096 ceph-mon[32753]: pgmap v16414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30167.161583 secs 2024-04-26T20:13:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:49 smithi194 ceph-mon[33017]: pgmap v16414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30170.083076 secs 2024-04-26T20:13:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30170.083076 secs 2024-04-26T20:13:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30170.083076 secs 2024-04-26T20:13:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:51 smithi033 ceph-mon[27845]: pgmap v16415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:51 smithi096 ceph-mon[32753]: pgmap v16415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:51 smithi194 ceph-mon[33017]: pgmap v16415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:53 smithi033 ceph-mon[27845]: pgmap v16416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:53 smithi096 ceph-mon[32753]: pgmap v16416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:53 smithi194 ceph-mon[33017]: pgmap v16416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30172.161637 secs 2024-04-26T20:13:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30172.161637 secs 2024-04-26T20:13:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30172.161637 secs 2024-04-26T20:13:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:54.898+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:54.898+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:13:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:13:54.898+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:13:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30175.083198 secs 2024-04-26T20:13:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:55 smithi033 ceph-mon[27845]: pgmap v16417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30175.083198 secs 2024-04-26T20:13:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:55 smithi096 ceph-mon[32753]: pgmap v16417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30175.083198 secs 2024-04-26T20:13:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:55 smithi194 ceph-mon[33017]: pgmap v16417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:57 smithi033 ceph-mon[27845]: pgmap v16418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:57 smithi096 ceph-mon[32753]: pgmap v16418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:57 smithi194 ceph-mon[33017]: pgmap v16418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30177.161698 secs 2024-04-26T20:13:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:13:59 smithi033 ceph-mon[27845]: pgmap v16419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30177.161698 secs 2024-04-26T20:13:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:13:59 smithi096 ceph-mon[32753]: pgmap v16419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:13:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30177.161698 secs 2024-04-26T20:13:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:13:59 smithi194 ceph-mon[33017]: pgmap v16419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30180.083334 secs 2024-04-26T20:14:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30180.083334 secs 2024-04-26T20:14:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30180.083334 secs 2024-04-26T20:14:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:01 smithi033 ceph-mon[27845]: pgmap v16420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:01 smithi096 ceph-mon[32753]: pgmap v16420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:01 smithi194 ceph-mon[33017]: pgmap v16420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:03 smithi033 ceph-mon[27845]: pgmap v16421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:03 smithi096 ceph-mon[32753]: pgmap v16421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:03 smithi194 ceph-mon[33017]: pgmap v16421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30182.161755 secs 2024-04-26T20:14:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30182.161755 secs 2024-04-26T20:14:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30182.161755 secs 2024-04-26T20:14:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30185.083455 secs 2024-04-26T20:14:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:05 smithi194 ceph-mon[33017]: pgmap v16422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30185.083455 secs 2024-04-26T20:14:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:05 smithi033 ceph-mon[27845]: pgmap v16422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30185.083455 secs 2024-04-26T20:14:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:05 smithi096 ceph-mon[32753]: pgmap v16422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:07 smithi194 ceph-mon[33017]: pgmap v16423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:07 smithi033 ceph-mon[27845]: pgmap v16423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:07 smithi096 ceph-mon[32753]: pgmap v16423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30187.161817 secs 2024-04-26T20:14:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:09 smithi194 ceph-mon[33017]: pgmap v16424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:09.908+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:09.908+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:14:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:09.908+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30187.161817 secs 2024-04-26T20:14:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:09 smithi033 ceph-mon[27845]: pgmap v16424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30187.161817 secs 2024-04-26T20:14:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:09 smithi096 ceph-mon[32753]: pgmap v16424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30190.083552 secs 2024-04-26T20:14:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30190.083552 secs 2024-04-26T20:14:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30190.083552 secs 2024-04-26T20:14:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:11 smithi194 ceph-mon[33017]: pgmap v16425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:11 smithi033 ceph-mon[27845]: pgmap v16425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:11 smithi096 ceph-mon[32753]: pgmap v16425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:13 smithi194 ceph-mon[33017]: pgmap v16426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:13 smithi033 ceph-mon[27845]: pgmap v16426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:13 smithi096 ceph-mon[32753]: pgmap v16426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30192.161874 secs 2024-04-26T20:14:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30192.161874 secs 2024-04-26T20:14:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30192.161874 secs 2024-04-26T20:14:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30195.083673 secs 2024-04-26T20:14:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:15 smithi194 ceph-mon[33017]: pgmap v16427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30195.083673 secs 2024-04-26T20:14:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:15 smithi033 ceph-mon[27845]: pgmap v16427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30195.083673 secs 2024-04-26T20:14:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:15 smithi096 ceph-mon[32753]: pgmap v16427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:17 smithi194 ceph-mon[33017]: pgmap v16428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:14:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:17 smithi033 ceph-mon[27845]: pgmap v16428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:14:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:17 smithi096 ceph-mon[32753]: pgmap v16428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:14:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:14:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:14:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:18 smithi194 ceph-mon[33017]: pgmap v16429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:18 smithi194 ceph-mon[33017]: pgmap v16430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:14:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:14:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:14:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:18 smithi033 ceph-mon[27845]: pgmap v16429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:18 smithi033 ceph-mon[27845]: pgmap v16430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:14:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:14:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:14:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:18 smithi096 ceph-mon[32753]: pgmap v16429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:18 smithi096 ceph-mon[32753]: pgmap v16430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:14:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30197.161936 secs 2024-04-26T20:14:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30197.161936 secs 2024-04-26T20:14:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30197.161936 secs 2024-04-26T20:14:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30200.083787 secs 2024-04-26T20:14:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:20 smithi194 ceph-mon[33017]: pgmap v16431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30200.083787 secs 2024-04-26T20:14:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:20 smithi033 ceph-mon[27845]: pgmap v16431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30200.083787 secs 2024-04-26T20:14:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:20 smithi096 ceph-mon[32753]: pgmap v16431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:22 smithi194 ceph-mon[33017]: pgmap v16432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:22 smithi033 ceph-mon[27845]: pgmap v16432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:22 smithi096 ceph-mon[32753]: pgmap v16432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30202.162022 secs 2024-04-26T20:14:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30202.162022 secs 2024-04-26T20:14:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30202.162022 secs 2024-04-26T20:14:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:24.919+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:24.920+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:24.920+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:14:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30205.083909 secs 2024-04-26T20:14:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:24 smithi194 ceph-mon[33017]: pgmap v16433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30205.083909 secs 2024-04-26T20:14:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:24 smithi033 ceph-mon[27845]: pgmap v16433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30205.083909 secs 2024-04-26T20:14:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:24 smithi096 ceph-mon[32753]: pgmap v16433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:26.933+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T20:14:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:26.933+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:14:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:26.933+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 238 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T20:14:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:26.933+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 238 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:14:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:26 smithi194 ceph-mon[33017]: pgmap v16434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:26 smithi033 ceph-mon[27845]: pgmap v16434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:26 smithi096 ceph-mon[32753]: pgmap v16434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:28 smithi194 ceph-mon[33017]: pgmap v16435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:14:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30207.162109 secs 2024-04-26T20:14:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:28 smithi033 ceph-mon[27845]: pgmap v16435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:14:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30207.162109 secs 2024-04-26T20:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:28 smithi096 ceph-mon[32753]: pgmap v16435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30207.162109 secs 2024-04-26T20:14:30.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30210.084027 secs 2024-04-26T20:14:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30210.084027 secs 2024-04-26T20:14:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30210.084027 secs 2024-04-26T20:14:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:31 smithi194 ceph-mon[33017]: pgmap v16436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:31 smithi033 ceph-mon[27845]: pgmap v16436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:31 smithi096 ceph-mon[32753]: pgmap v16436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:33 smithi194 ceph-mon[33017]: pgmap v16437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:33 smithi033 ceph-mon[27845]: pgmap v16437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:33 smithi096 ceph-mon[32753]: pgmap v16437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30212.162198 secs 2024-04-26T20:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30212.162198 secs 2024-04-26T20:14:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30212.162198 secs 2024-04-26T20:14:35.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30215.084158 secs 2024-04-26T20:14:35.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:35 smithi033 ceph-mon[27845]: pgmap v16438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30215.084158 secs 2024-04-26T20:14:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:35 smithi194 ceph-mon[33017]: pgmap v16438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30215.084158 secs 2024-04-26T20:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:35 smithi096 ceph-mon[32753]: pgmap v16438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:37 smithi194 ceph-mon[33017]: pgmap v16439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:37 smithi033 ceph-mon[27845]: pgmap v16439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:37 smithi096 ceph-mon[32753]: pgmap v16439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:39 smithi194 ceph-mon[33017]: pgmap v16440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30217.162262 secs 2024-04-26T20:14:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:39 smithi033 ceph-mon[27845]: pgmap v16440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30217.162262 secs 2024-04-26T20:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:39 smithi096 ceph-mon[32753]: pgmap v16440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30217.162262 secs 2024-04-26T20:14:39.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:39.935+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:39.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:39.935+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:39.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:39.935+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:14:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30220.084274 secs 2024-04-26T20:14:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30220.084274 secs 2024-04-26T20:14:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30220.084274 secs 2024-04-26T20:14:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:41 smithi194 ceph-mon[33017]: pgmap v16441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:41 smithi033 ceph-mon[27845]: pgmap v16441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:41 smithi096 ceph-mon[32753]: pgmap v16441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:43 smithi194 ceph-mon[33017]: pgmap v16442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:43 smithi033 ceph-mon[27845]: pgmap v16442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:43 smithi096 ceph-mon[32753]: pgmap v16442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30222.162320 secs 2024-04-26T20:14:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30222.162320 secs 2024-04-26T20:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30222.162320 secs 2024-04-26T20:14:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30225.084391 secs 2024-04-26T20:14:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:45 smithi194 ceph-mon[33017]: pgmap v16443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30225.084391 secs 2024-04-26T20:14:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:45 smithi033 ceph-mon[27845]: pgmap v16443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30225.084391 secs 2024-04-26T20:14:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:45 smithi096 ceph-mon[32753]: pgmap v16443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:47 smithi194 ceph-mon[33017]: pgmap v16444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:47 smithi033 ceph-mon[27845]: pgmap v16444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:47 smithi096 ceph-mon[32753]: pgmap v16444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30227.162381 secs 2024-04-26T20:14:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:49 smithi194 ceph-mon[33017]: pgmap v16445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30227.162381 secs 2024-04-26T20:14:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:49 smithi033 ceph-mon[27845]: pgmap v16445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30227.162381 secs 2024-04-26T20:14:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:49 smithi096 ceph-mon[32753]: pgmap v16445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30230.084507 secs 2024-04-26T20:14:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30230.084507 secs 2024-04-26T20:14:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30230.084507 secs 2024-04-26T20:14:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:51 smithi194 ceph-mon[33017]: pgmap v16446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:51 smithi033 ceph-mon[27845]: pgmap v16446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:51 smithi096 ceph-mon[32753]: pgmap v16446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:53 smithi194 ceph-mon[33017]: pgmap v16447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:53 smithi033 ceph-mon[27845]: pgmap v16447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:53 smithi096 ceph-mon[32753]: pgmap v16447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30232.162438 secs 2024-04-26T20:14:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30232.162438 secs 2024-04-26T20:14:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30232.162438 secs 2024-04-26T20:14:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:54.951+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:54.951+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:14:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:14:54.951+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:14:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30235.084631 secs 2024-04-26T20:14:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:55 smithi033 ceph-mon[27845]: pgmap v16448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30235.084631 secs 2024-04-26T20:14:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:55 smithi096 ceph-mon[32753]: pgmap v16448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30235.084631 secs 2024-04-26T20:14:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:55 smithi194 ceph-mon[33017]: pgmap v16448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:57 smithi033 ceph-mon[27845]: pgmap v16449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:57 smithi096 ceph-mon[32753]: pgmap v16449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:57 smithi194 ceph-mon[33017]: pgmap v16449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30237.162500 secs 2024-04-26T20:14:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:14:59 smithi033 ceph-mon[27845]: pgmap v16450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30237.162500 secs 2024-04-26T20:14:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:14:59 smithi096 ceph-mon[32753]: pgmap v16450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:14:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30237.162500 secs 2024-04-26T20:14:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:14:59 smithi194 ceph-mon[33017]: pgmap v16450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30240.084745 secs 2024-04-26T20:15:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30240.084745 secs 2024-04-26T20:15:00.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30240.084745 secs 2024-04-26T20:15:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:01 smithi033 ceph-mon[27845]: pgmap v16451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:01 smithi096 ceph-mon[32753]: pgmap v16451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:01 smithi194 ceph-mon[33017]: pgmap v16451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:03 smithi033 ceph-mon[27845]: pgmap v16452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:03 smithi096 ceph-mon[32753]: pgmap v16452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:03 smithi194 ceph-mon[33017]: pgmap v16452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30242.162558 secs 2024-04-26T20:15:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30242.162558 secs 2024-04-26T20:15:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30242.162558 secs 2024-04-26T20:15:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30245.084868 secs 2024-04-26T20:15:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:05 smithi033 ceph-mon[27845]: pgmap v16453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30245.084868 secs 2024-04-26T20:15:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:05 smithi096 ceph-mon[32753]: pgmap v16453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30245.084868 secs 2024-04-26T20:15:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:05 smithi194 ceph-mon[33017]: pgmap v16453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:07 smithi033 ceph-mon[27845]: pgmap v16454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:07 smithi096 ceph-mon[32753]: pgmap v16454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:07 smithi194 ceph-mon[33017]: pgmap v16454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30247.162617 secs 2024-04-26T20:15:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:09 smithi033 ceph-mon[27845]: pgmap v16455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30247.162617 secs 2024-04-26T20:15:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:09 smithi096 ceph-mon[32753]: pgmap v16455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30247.162617 secs 2024-04-26T20:15:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:09 smithi194 ceph-mon[33017]: pgmap v16455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:09.966+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:09.966+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:15:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:09.966+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30250.084943 secs 2024-04-26T20:15:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30250.084943 secs 2024-04-26T20:15:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30250.084943 secs 2024-04-26T20:15:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:11 smithi033 ceph-mon[27845]: pgmap v16456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:11 smithi096 ceph-mon[32753]: pgmap v16456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:11 smithi194 ceph-mon[33017]: pgmap v16456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:13 smithi033 ceph-mon[27845]: pgmap v16457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:13 smithi096 ceph-mon[32753]: pgmap v16457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:13 smithi194 ceph-mon[33017]: pgmap v16457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30252.162673 secs 2024-04-26T20:15:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30252.162673 secs 2024-04-26T20:15:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30252.162673 secs 2024-04-26T20:15:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30255.085040 secs 2024-04-26T20:15:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:15 smithi194 ceph-mon[33017]: pgmap v16458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30255.085040 secs 2024-04-26T20:15:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:15 smithi033 ceph-mon[27845]: pgmap v16458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30255.085040 secs 2024-04-26T20:15:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:15 smithi096 ceph-mon[32753]: pgmap v16458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:17.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:17 smithi194 ceph-mon[33017]: pgmap v16459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:17 smithi033 ceph-mon[27845]: pgmap v16459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:17 smithi096 ceph-mon[32753]: pgmap v16459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30257.162734 secs 2024-04-26T20:15:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:19 smithi194 ceph-mon[33017]: pgmap v16460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:15:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30257.162734 secs 2024-04-26T20:15:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:19 smithi033 ceph-mon[27845]: pgmap v16460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:15:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30257.162734 secs 2024-04-26T20:15:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:19 smithi096 ceph-mon[32753]: pgmap v16460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:15:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30260.085129 secs 2024-04-26T20:15:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:15:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:15:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:15:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30260.085129 secs 2024-04-26T20:15:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:15:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:15:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:15:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30260.085129 secs 2024-04-26T20:15:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:15:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:15:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:15:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:21 smithi194 ceph-mon[33017]: pgmap v16461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:15:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:21 smithi194 ceph-mon[33017]: pgmap v16462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:21 smithi033 ceph-mon[27845]: pgmap v16461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:15:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:21 smithi033 ceph-mon[27845]: pgmap v16462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:21 smithi096 ceph-mon[32753]: pgmap v16461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:15:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:21 smithi096 ceph-mon[32753]: pgmap v16462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:23 smithi194 ceph-mon[33017]: pgmap v16463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:23 smithi033 ceph-mon[27845]: pgmap v16463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:23 smithi096 ceph-mon[32753]: pgmap v16463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30262.162811 secs 2024-04-26T20:15:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30262.162811 secs 2024-04-26T20:15:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30262.162811 secs 2024-04-26T20:15:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:24.968+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:24.968+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:15:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:24.968+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:25 smithi194 ceph-mon[33017]: pgmap v16464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30265.085251 secs 2024-04-26T20:15:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:25 smithi033 ceph-mon[27845]: pgmap v16464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30265.085251 secs 2024-04-26T20:15:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:25 smithi096 ceph-mon[32753]: pgmap v16464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30265.085251 secs 2024-04-26T20:15:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:27 smithi194 ceph-mon[33017]: pgmap v16465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:27 smithi033 ceph-mon[27845]: pgmap v16465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:27 smithi096 ceph-mon[32753]: pgmap v16465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:29 smithi194 ceph-mon[33017]: pgmap v16466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30267.162897 secs 2024-04-26T20:15:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:29 smithi033 ceph-mon[27845]: pgmap v16466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30267.162897 secs 2024-04-26T20:15:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:29 smithi096 ceph-mon[32753]: pgmap v16466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:15:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30267.162897 secs 2024-04-26T20:15:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30270.085367 secs 2024-04-26T20:15:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30270.085367 secs 2024-04-26T20:15:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30270.085367 secs 2024-04-26T20:15:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:31 smithi194 ceph-mon[33017]: pgmap v16467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:15:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:31 smithi033 ceph-mon[27845]: pgmap v16467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:15:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:31 smithi096 ceph-mon[32753]: pgmap v16467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:15:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:33 smithi194 ceph-mon[33017]: pgmap v16468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:33 smithi033 ceph-mon[27845]: pgmap v16468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:33 smithi096 ceph-mon[32753]: pgmap v16468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:34.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30272.162992 secs 2024-04-26T20:15:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30272.162992 secs 2024-04-26T20:15:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30272.162992 secs 2024-04-26T20:15:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:35 smithi194 ceph-mon[33017]: pgmap v16469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30275.085494 secs 2024-04-26T20:15:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:35 smithi033 ceph-mon[27845]: pgmap v16469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30275.085494 secs 2024-04-26T20:15:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:35 smithi096 ceph-mon[32753]: pgmap v16469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30275.085494 secs 2024-04-26T20:15:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:37 smithi194 ceph-mon[33017]: pgmap v16470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:37 smithi033 ceph-mon[27845]: pgmap v16470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:37 smithi096 ceph-mon[32753]: pgmap v16470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:39 smithi194 ceph-mon[33017]: pgmap v16471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30277.163073 secs 2024-04-26T20:15:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:39 smithi033 ceph-mon[27845]: pgmap v16471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30277.163073 secs 2024-04-26T20:15:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:39 smithi096 ceph-mon[32753]: pgmap v16471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30277.163073 secs 2024-04-26T20:15:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:39.975+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:39.975+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:39.975+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:15:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30280.085606 secs 2024-04-26T20:15:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30280.085606 secs 2024-04-26T20:15:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30280.085606 secs 2024-04-26T20:15:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:41 smithi194 ceph-mon[33017]: pgmap v16472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:41 smithi033 ceph-mon[27845]: pgmap v16472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:41 smithi096 ceph-mon[32753]: pgmap v16472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:43 smithi194 ceph-mon[33017]: pgmap v16473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:43 smithi033 ceph-mon[27845]: pgmap v16473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:43 smithi096 ceph-mon[32753]: pgmap v16473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30282.163156 secs 2024-04-26T20:15:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30282.163156 secs 2024-04-26T20:15:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30282.163156 secs 2024-04-26T20:15:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:45 smithi194 ceph-mon[33017]: pgmap v16474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30285.085723 secs 2024-04-26T20:15:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:45 smithi033 ceph-mon[27845]: pgmap v16474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30285.085723 secs 2024-04-26T20:15:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:45 smithi096 ceph-mon[32753]: pgmap v16474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30285.085723 secs 2024-04-26T20:15:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:47 smithi194 ceph-mon[33017]: pgmap v16475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:47 smithi033 ceph-mon[27845]: pgmap v16475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:47 smithi096 ceph-mon[32753]: pgmap v16475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:49 smithi194 ceph-mon[33017]: pgmap v16476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30287.163240 secs 2024-04-26T20:15:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:49 smithi033 ceph-mon[27845]: pgmap v16476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30287.163240 secs 2024-04-26T20:15:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:49 smithi096 ceph-mon[32753]: pgmap v16476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30287.163240 secs 2024-04-26T20:15:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30290.085849 secs 2024-04-26T20:15:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30290.085849 secs 2024-04-26T20:15:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30290.085849 secs 2024-04-26T20:15:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:51 smithi194 ceph-mon[33017]: pgmap v16477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:51 smithi033 ceph-mon[27845]: pgmap v16477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:51 smithi096 ceph-mon[32753]: pgmap v16477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:53 smithi033 ceph-mon[27845]: pgmap v16478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:53 smithi194 ceph-mon[33017]: pgmap v16478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:53 smithi096 ceph-mon[32753]: pgmap v16478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30292.163325 secs 2024-04-26T20:15:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30292.163325 secs 2024-04-26T20:15:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30292.163325 secs 2024-04-26T20:15:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:54.984+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:54.984+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:15:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:15:54.984+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:15:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:55 smithi033 ceph-mon[27845]: pgmap v16479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30295.085966 secs 2024-04-26T20:15:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:55 smithi096 ceph-mon[32753]: pgmap v16479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30295.085966 secs 2024-04-26T20:15:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:55 smithi194 ceph-mon[33017]: pgmap v16479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30295.085966 secs 2024-04-26T20:15:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:57 smithi033 ceph-mon[27845]: pgmap v16480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:57 smithi096 ceph-mon[32753]: pgmap v16480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:57 smithi194 ceph-mon[33017]: pgmap v16480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:59 smithi033 ceph-mon[27845]: pgmap v16481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:15:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30297.163409 secs 2024-04-26T20:15:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:59 smithi096 ceph-mon[32753]: pgmap v16481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:15:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30297.163409 secs 2024-04-26T20:15:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:59 smithi194 ceph-mon[33017]: pgmap v16481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:15:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:15:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30297.163409 secs 2024-04-26T20:16:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30300.086063 secs 2024-04-26T20:16:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30300.086063 secs 2024-04-26T20:16:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30300.086063 secs 2024-04-26T20:16:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:01 smithi033 ceph-mon[27845]: pgmap v16482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:01 smithi096 ceph-mon[32753]: pgmap v16482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:01 smithi194 ceph-mon[33017]: pgmap v16482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:03 smithi033 ceph-mon[27845]: pgmap v16483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:03 smithi096 ceph-mon[32753]: pgmap v16483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:03 smithi194 ceph-mon[33017]: pgmap v16483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30302.163491 secs 2024-04-26T20:16:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30302.163491 secs 2024-04-26T20:16:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30302.163491 secs 2024-04-26T20:16:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:05 smithi033 ceph-mon[27845]: pgmap v16484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30305.086180 secs 2024-04-26T20:16:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:05 smithi096 ceph-mon[32753]: pgmap v16484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30305.086180 secs 2024-04-26T20:16:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:05 smithi194 ceph-mon[33017]: pgmap v16484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30305.086180 secs 2024-04-26T20:16:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:07 smithi033 ceph-mon[27845]: pgmap v16485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:07 smithi096 ceph-mon[32753]: pgmap v16485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:07 smithi194 ceph-mon[33017]: pgmap v16485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:09 smithi033 ceph-mon[27845]: pgmap v16486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30307.163574 secs 2024-04-26T20:16:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:09 smithi096 ceph-mon[32753]: pgmap v16486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30307.163574 secs 2024-04-26T20:16:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:09 smithi194 ceph-mon[33017]: pgmap v16486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30307.163574 secs 2024-04-26T20:16:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:09.996+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:09.996+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:09.996+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:16:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30310.086296 secs 2024-04-26T20:16:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30310.086296 secs 2024-04-26T20:16:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30310.086296 secs 2024-04-26T20:16:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:11 smithi096 ceph-mon[32753]: pgmap v16487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:11 smithi194 ceph-mon[33017]: pgmap v16487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:11 smithi033 ceph-mon[27845]: pgmap v16487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:13 smithi096 ceph-mon[32753]: pgmap v16488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:13 smithi194 ceph-mon[33017]: pgmap v16488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:13 smithi033 ceph-mon[27845]: pgmap v16488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30312.163658 secs 2024-04-26T20:16:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30312.163658 secs 2024-04-26T20:16:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30312.163658 secs 2024-04-26T20:16:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:15 smithi194 ceph-mon[33017]: pgmap v16489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30315.086367 secs 2024-04-26T20:16:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:15 smithi033 ceph-mon[27845]: pgmap v16489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30315.086367 secs 2024-04-26T20:16:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:15 smithi096 ceph-mon[32753]: pgmap v16489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30315.086367 secs 2024-04-26T20:16:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:17 smithi194 ceph-mon[33017]: pgmap v16490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:17 smithi033 ceph-mon[27845]: pgmap v16490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:17 smithi096 ceph-mon[32753]: pgmap v16490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:19 smithi194 ceph-mon[33017]: pgmap v16491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30317.163741 secs 2024-04-26T20:16:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:19 smithi033 ceph-mon[27845]: pgmap v16491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30317.163741 secs 2024-04-26T20:16:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:19 smithi096 ceph-mon[32753]: pgmap v16491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30317.163741 secs 2024-04-26T20:16:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30320.086506 secs 2024-04-26T20:16:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:16:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30320.086506 secs 2024-04-26T20:16:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:16:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30320.086506 secs 2024-04-26T20:16:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:16:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:21 smithi194 ceph-mon[33017]: pgmap v16492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:16:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:16:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:16:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:21 smithi033 ceph-mon[27845]: pgmap v16492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:16:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:16:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:16:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:21 smithi096 ceph-mon[32753]: pgmap v16492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:16:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:16:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:16:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:22 smithi194 ceph-mon[33017]: pgmap v16493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:22 smithi194 ceph-mon[33017]: pgmap v16494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:22 smithi033 ceph-mon[27845]: pgmap v16493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:22 smithi033 ceph-mon[27845]: pgmap v16494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:22 smithi096 ceph-mon[32753]: pgmap v16493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:22 smithi096 ceph-mon[32753]: pgmap v16494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30322.163821 secs 2024-04-26T20:16:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:24 smithi194 ceph-mon[33017]: pgmap v16495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30322.163821 secs 2024-04-26T20:16:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:24 smithi033 ceph-mon[27845]: pgmap v16495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30322.163821 secs 2024-04-26T20:16:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:24 smithi096 ceph-mon[32753]: pgmap v16495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:24.999+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:24.999+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:24.999+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:16:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30325.086657 secs 2024-04-26T20:16:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30325.086657 secs 2024-04-26T20:16:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30325.086657 secs 2024-04-26T20:16:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:26 smithi194 ceph-mon[33017]: pgmap v16496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:26 smithi033 ceph-mon[27845]: pgmap v16496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:26 smithi096 ceph-mon[32753]: pgmap v16496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:28 smithi194 ceph-mon[33017]: pgmap v16497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:28 smithi033 ceph-mon[27845]: pgmap v16497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:28 smithi096 ceph-mon[32753]: pgmap v16497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30327.163905 secs 2024-04-26T20:16:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30327.163905 secs 2024-04-26T20:16:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30327.163905 secs 2024-04-26T20:16:30.520 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30330.086773 secs 2024-04-26T20:16:30.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30330.086773 secs 2024-04-26T20:16:30.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:30 smithi033 ceph-mon[27845]: pgmap v16498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:16:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30330.086773 secs 2024-04-26T20:16:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:30 smithi194 ceph-mon[33017]: pgmap v16498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:16:30.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:30 smithi096 ceph-mon[32753]: pgmap v16498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:16:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:31 smithi194 ceph-mon[33017]: pgmap v16499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:16:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:31 smithi033 ceph-mon[27845]: pgmap v16499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:16:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:31 smithi096 ceph-mon[32753]: pgmap v16499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:16:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30332.163991 secs 2024-04-26T20:16:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:34 smithi033 ceph-mon[27845]: pgmap v16500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30332.163991 secs 2024-04-26T20:16:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:34 smithi096 ceph-mon[32753]: pgmap v16500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30332.163991 secs 2024-04-26T20:16:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:34 smithi194 ceph-mon[33017]: pgmap v16500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30335.086889 secs 2024-04-26T20:16:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30335.086889 secs 2024-04-26T20:16:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30335.086889 secs 2024-04-26T20:16:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:36 smithi033 ceph-mon[27845]: pgmap v16501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:36 smithi096 ceph-mon[32753]: pgmap v16501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:36 smithi194 ceph-mon[33017]: pgmap v16501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:38 smithi033 ceph-mon[27845]: pgmap v16502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:38 smithi096 ceph-mon[32753]: pgmap v16502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:38 smithi194 ceph-mon[33017]: pgmap v16502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30337.164081 secs 2024-04-26T20:16:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30337.164081 secs 2024-04-26T20:16:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30337.164081 secs 2024-04-26T20:16:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:40.010+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:40.010+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:40.010+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:16:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30340.087041 secs 2024-04-26T20:16:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:40 smithi033 ceph-mon[27845]: pgmap v16503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30340.087041 secs 2024-04-26T20:16:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:40 smithi096 ceph-mon[32753]: pgmap v16503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30340.087041 secs 2024-04-26T20:16:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:40 smithi194 ceph-mon[33017]: pgmap v16503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:42 smithi033 ceph-mon[27845]: pgmap v16504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:16:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:42 smithi096 ceph-mon[32753]: pgmap v16504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:16:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:42 smithi194 ceph-mon[33017]: pgmap v16504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:16:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30342.164165 secs 2024-04-26T20:16:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:44 smithi096 ceph-mon[32753]: pgmap v16505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30342.164165 secs 2024-04-26T20:16:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:44 smithi194 ceph-mon[33017]: pgmap v16505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30342.164165 secs 2024-04-26T20:16:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:44 smithi033 ceph-mon[27845]: pgmap v16505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30345.087155 secs 2024-04-26T20:16:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30345.087155 secs 2024-04-26T20:16:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30345.087155 secs 2024-04-26T20:16:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:46 smithi194 ceph-mon[33017]: pgmap v16506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:46 smithi033 ceph-mon[27845]: pgmap v16506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:46 smithi096 ceph-mon[32753]: pgmap v16506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:48 smithi194 ceph-mon[33017]: pgmap v16507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:48 smithi033 ceph-mon[27845]: pgmap v16507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:48 smithi096 ceph-mon[32753]: pgmap v16507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30347.164270 secs 2024-04-26T20:16:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30347.164270 secs 2024-04-26T20:16:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30347.164270 secs 2024-04-26T20:16:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30350.087221 secs 2024-04-26T20:16:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:50 smithi194 ceph-mon[33017]: pgmap v16508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30350.087221 secs 2024-04-26T20:16:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:50 smithi033 ceph-mon[27845]: pgmap v16508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30350.087221 secs 2024-04-26T20:16:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:50 smithi096 ceph-mon[32753]: pgmap v16508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:16:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:52 smithi194 ceph-mon[33017]: pgmap v16509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:16:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:52 smithi033 ceph-mon[27845]: pgmap v16509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:16:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:52 smithi096 ceph-mon[32753]: pgmap v16509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:16:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30352.164350 secs 2024-04-26T20:16:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:54 smithi194 ceph-mon[33017]: pgmap v16510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30352.164350 secs 2024-04-26T20:16:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:54 smithi033 ceph-mon[27845]: pgmap v16510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30352.164350 secs 2024-04-26T20:16:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:54 smithi096 ceph-mon[32753]: pgmap v16510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:55.016+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:55.016+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:16:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:16:55.016+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:16:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30355.087342 secs 2024-04-26T20:16:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30355.087342 secs 2024-04-26T20:16:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30355.087342 secs 2024-04-26T20:16:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:56 smithi194 ceph-mon[33017]: pgmap v16511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:56 smithi033 ceph-mon[27845]: pgmap v16511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:56 smithi096 ceph-mon[32753]: pgmap v16511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:58 smithi194 ceph-mon[33017]: pgmap v16512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:58 smithi033 ceph-mon[27845]: pgmap v16512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:58 smithi096 ceph-mon[32753]: pgmap v16512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:16:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:16:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30357.164435 secs 2024-04-26T20:16:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:16:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30357.164435 secs 2024-04-26T20:16:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:16:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30357.164435 secs 2024-04-26T20:17:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30360.087451 secs 2024-04-26T20:17:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:00 smithi194 ceph-mon[33017]: pgmap v16513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:00.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30360.087451 secs 2024-04-26T20:17:00.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:00 smithi033 ceph-mon[27845]: pgmap v16513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30360.087451 secs 2024-04-26T20:17:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:00 smithi096 ceph-mon[32753]: pgmap v16513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:02 smithi194 ceph-mon[33017]: pgmap v16514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:02 smithi033 ceph-mon[27845]: pgmap v16514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:02 smithi096 ceph-mon[32753]: pgmap v16514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30362.164512 secs 2024-04-26T20:17:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:04 smithi194 ceph-mon[33017]: pgmap v16515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30362.164512 secs 2024-04-26T20:17:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:04 smithi033 ceph-mon[27845]: pgmap v16515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30362.164512 secs 2024-04-26T20:17:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:04 smithi096 ceph-mon[32753]: pgmap v16515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30365.087582 secs 2024-04-26T20:17:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:05 smithi194 ceph-mon[33017]: pgmap v16516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30365.087582 secs 2024-04-26T20:17:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:05 smithi033 ceph-mon[27845]: pgmap v16516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30365.087582 secs 2024-04-26T20:17:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:05 smithi096 ceph-mon[32753]: pgmap v16516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:08 smithi033 ceph-mon[27845]: pgmap v16517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:08 smithi096 ceph-mon[32753]: pgmap v16517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:08 smithi194 ceph-mon[33017]: pgmap v16517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30367.164597 secs 2024-04-26T20:17:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30367.164597 secs 2024-04-26T20:17:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30367.164597 secs 2024-04-26T20:17:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:10.026+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:10.026+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:17:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:10.026+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30370.087711 secs 2024-04-26T20:17:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:10 smithi033 ceph-mon[27845]: pgmap v16518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30370.087711 secs 2024-04-26T20:17:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:10 smithi096 ceph-mon[32753]: pgmap v16518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30370.087711 secs 2024-04-26T20:17:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:10 smithi194 ceph-mon[33017]: pgmap v16518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:12 smithi033 ceph-mon[27845]: pgmap v16519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:12 smithi096 ceph-mon[32753]: pgmap v16519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:12 smithi194 ceph-mon[33017]: pgmap v16519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30372.164678 secs 2024-04-26T20:17:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:14 smithi096 ceph-mon[32753]: pgmap v16520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30372.164678 secs 2024-04-26T20:17:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:14 smithi194 ceph-mon[33017]: pgmap v16520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30372.164678 secs 2024-04-26T20:17:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:14 smithi033 ceph-mon[27845]: pgmap v16520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30375.087829 secs 2024-04-26T20:17:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30375.087829 secs 2024-04-26T20:17:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30375.087829 secs 2024-04-26T20:17:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:16 smithi194 ceph-mon[33017]: pgmap v16521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:16 smithi033 ceph-mon[27845]: pgmap v16521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:16 smithi096 ceph-mon[32753]: pgmap v16521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:18 smithi194 ceph-mon[33017]: pgmap v16522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:18 smithi033 ceph-mon[27845]: pgmap v16522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:18 smithi096 ceph-mon[32753]: pgmap v16522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30377.164739 secs 2024-04-26T20:17:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30377.164739 secs 2024-04-26T20:17:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30377.164739 secs 2024-04-26T20:17:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30380.087920 secs 2024-04-26T20:17:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:20 smithi194 ceph-mon[33017]: pgmap v16523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30380.087920 secs 2024-04-26T20:17:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:20 smithi033 ceph-mon[27845]: pgmap v16523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30380.087920 secs 2024-04-26T20:17:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:20 smithi096 ceph-mon[32753]: pgmap v16523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:22 smithi194 ceph-mon[33017]: pgmap v16524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:17:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:22 smithi033 ceph-mon[27845]: pgmap v16524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:17:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:22 smithi096 ceph-mon[32753]: pgmap v16524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:17:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:17:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:17:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:17:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:17:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:17:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:17:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:17:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:17:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:17:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:17:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30382.164820 secs 2024-04-26T20:17:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:24 smithi194 ceph-mon[33017]: pgmap v16525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:24 smithi194 ceph-mon[33017]: pgmap v16526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30382.164820 secs 2024-04-26T20:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:24 smithi033 ceph-mon[27845]: pgmap v16525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:24 smithi033 ceph-mon[27845]: pgmap v16526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30382.164820 secs 2024-04-26T20:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:24 smithi096 ceph-mon[32753]: pgmap v16525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:24 smithi096 ceph-mon[32753]: pgmap v16526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:25.042+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:25.042+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:25.042+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:17:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30385.088041 secs 2024-04-26T20:17:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30385.088041 secs 2024-04-26T20:17:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30385.088041 secs 2024-04-26T20:17:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:26 smithi033 ceph-mon[27845]: pgmap v16527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:26 smithi194 ceph-mon[33017]: pgmap v16527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:26 smithi096 ceph-mon[32753]: pgmap v16527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:26.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:26.965+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T20:17:26.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:26.965+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:17:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:26.965+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 239 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T20:17:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:26.966+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 239 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x7fef00077780 2024-04-26T20:17:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:28 smithi194 ceph-mon[33017]: pgmap v16528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:28 smithi033 ceph-mon[27845]: pgmap v16528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:28 smithi096 ceph-mon[32753]: pgmap v16528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30387.164900 secs 2024-04-26T20:17:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:29 smithi194 ceph-mon[33017]: pgmap v16529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T20:17:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30387.164900 secs 2024-04-26T20:17:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:29 smithi033 ceph-mon[27845]: pgmap v16529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T20:17:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30387.164900 secs 2024-04-26T20:17:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:29 smithi096 ceph-mon[32753]: pgmap v16529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T20:17:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30390.088163 secs 2024-04-26T20:17:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30390.088163 secs 2024-04-26T20:17:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30390.088163 secs 2024-04-26T20:17:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:31 smithi194 ceph-mon[33017]: pgmap v16530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:31 smithi033 ceph-mon[27845]: pgmap v16530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:31 smithi096 ceph-mon[32753]: pgmap v16530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:17:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30392.164998 secs 2024-04-26T20:17:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:34 smithi194 ceph-mon[33017]: pgmap v16531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:17:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30392.164998 secs 2024-04-26T20:17:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:34 smithi033 ceph-mon[27845]: pgmap v16531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:17:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30392.164998 secs 2024-04-26T20:17:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:34 smithi096 ceph-mon[32753]: pgmap v16531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:17:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30395.088243 secs 2024-04-26T20:17:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30395.088243 secs 2024-04-26T20:17:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30395.088243 secs 2024-04-26T20:17:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:36 smithi194 ceph-mon[33017]: pgmap v16532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:36 smithi033 ceph-mon[27845]: pgmap v16532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:36 smithi096 ceph-mon[32753]: pgmap v16532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:38 smithi194 ceph-mon[33017]: pgmap v16533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:38 smithi033 ceph-mon[27845]: pgmap v16533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:38 smithi096 ceph-mon[32753]: pgmap v16533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30397.165059 secs 2024-04-26T20:17:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30397.165059 secs 2024-04-26T20:17:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30397.165059 secs 2024-04-26T20:17:40.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:40.054+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:40.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:40.054+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:40.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:40.054+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:17:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30400.088368 secs 2024-04-26T20:17:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:40 smithi194 ceph-mon[33017]: pgmap v16534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30400.088368 secs 2024-04-26T20:17:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:40 smithi033 ceph-mon[27845]: pgmap v16534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30400.088368 secs 2024-04-26T20:17:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:40 smithi096 ceph-mon[32753]: pgmap v16534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:42 smithi194 ceph-mon[33017]: pgmap v16535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:42 smithi033 ceph-mon[27845]: pgmap v16535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:42 smithi096 ceph-mon[32753]: pgmap v16535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30402.165140 secs 2024-04-26T20:17:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:44 smithi033 ceph-mon[27845]: pgmap v16536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30402.165140 secs 2024-04-26T20:17:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:44 smithi096 ceph-mon[32753]: pgmap v16536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30402.165140 secs 2024-04-26T20:17:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:44 smithi194 ceph-mon[33017]: pgmap v16536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30405.088498 secs 2024-04-26T20:17:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30405.088498 secs 2024-04-26T20:17:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30405.088498 secs 2024-04-26T20:17:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:46 smithi033 ceph-mon[27845]: pgmap v16537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:46 smithi096 ceph-mon[32753]: pgmap v16537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:46 smithi194 ceph-mon[33017]: pgmap v16537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:48 smithi033 ceph-mon[27845]: pgmap v16538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:48 smithi096 ceph-mon[32753]: pgmap v16538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:48 smithi194 ceph-mon[33017]: pgmap v16538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30407.165224 secs 2024-04-26T20:17:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30407.165224 secs 2024-04-26T20:17:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30407.165224 secs 2024-04-26T20:17:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30410.088627 secs 2024-04-26T20:17:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:50 smithi033 ceph-mon[27845]: pgmap v16539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30410.088627 secs 2024-04-26T20:17:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:50 smithi096 ceph-mon[32753]: pgmap v16539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30410.088627 secs 2024-04-26T20:17:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:50 smithi194 ceph-mon[33017]: pgmap v16539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:52 smithi033 ceph-mon[27845]: pgmap v16540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:52 smithi096 ceph-mon[32753]: pgmap v16540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:52 smithi194 ceph-mon[33017]: pgmap v16540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30412.165303 secs 2024-04-26T20:17:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:54 smithi033 ceph-mon[27845]: pgmap v16541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30412.165303 secs 2024-04-26T20:17:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:54 smithi096 ceph-mon[32753]: pgmap v16541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30412.165303 secs 2024-04-26T20:17:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:54 smithi194 ceph-mon[33017]: pgmap v16541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:55.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:55.060+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:55.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:55.060+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:17:55.061 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:17:55.060+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:17:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30415.088757 secs 2024-04-26T20:17:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30415.088757 secs 2024-04-26T20:17:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30415.088757 secs 2024-04-26T20:17:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:56 smithi033 ceph-mon[27845]: pgmap v16542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:56 smithi096 ceph-mon[32753]: pgmap v16542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:56 smithi194 ceph-mon[33017]: pgmap v16542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:58 smithi033 ceph-mon[27845]: pgmap v16543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:58 smithi096 ceph-mon[32753]: pgmap v16543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:58 smithi194 ceph-mon[33017]: pgmap v16543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:17:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:17:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30417.165390 secs 2024-04-26T20:17:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:17:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30417.165390 secs 2024-04-26T20:17:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:17:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30417.165390 secs 2024-04-26T20:18:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30420.088892 secs 2024-04-26T20:18:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:00 smithi096 ceph-mon[32753]: pgmap v16544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:00.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30420.088892 secs 2024-04-26T20:18:00.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:00 smithi194 ceph-mon[33017]: pgmap v16544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30420.088892 secs 2024-04-26T20:18:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:00 smithi033 ceph-mon[27845]: pgmap v16544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:02 smithi194 ceph-mon[33017]: pgmap v16545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:02 smithi033 ceph-mon[27845]: pgmap v16545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:02 smithi096 ceph-mon[32753]: pgmap v16545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30422.165469 secs 2024-04-26T20:18:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:04 smithi194 ceph-mon[33017]: pgmap v16546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30422.165469 secs 2024-04-26T20:18:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:04 smithi033 ceph-mon[27845]: pgmap v16546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30422.165469 secs 2024-04-26T20:18:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:04 smithi096 ceph-mon[32753]: pgmap v16546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30425.089005 secs 2024-04-26T20:18:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30425.089005 secs 2024-04-26T20:18:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30425.089005 secs 2024-04-26T20:18:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:06 smithi194 ceph-mon[33017]: pgmap v16547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:06 smithi033 ceph-mon[27845]: pgmap v16547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:06 smithi096 ceph-mon[32753]: pgmap v16547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:08 smithi194 ceph-mon[33017]: pgmap v16548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:08 smithi033 ceph-mon[27845]: pgmap v16548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:08 smithi096 ceph-mon[32753]: pgmap v16548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30427.165555 secs 2024-04-26T20:18:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30427.165555 secs 2024-04-26T20:18:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30427.165555 secs 2024-04-26T20:18:10.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:10.074+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:10.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:10.074+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:10.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:10.074+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:18:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30430.089122 secs 2024-04-26T20:18:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:10 smithi194 ceph-mon[33017]: pgmap v16549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30430.089122 secs 2024-04-26T20:18:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:10 smithi033 ceph-mon[27845]: pgmap v16549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30430.089122 secs 2024-04-26T20:18:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:10 smithi096 ceph-mon[32753]: pgmap v16549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:12 smithi194 ceph-mon[33017]: pgmap v16550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:12 smithi033 ceph-mon[27845]: pgmap v16550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:12 smithi096 ceph-mon[32753]: pgmap v16550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30432.165637 secs 2024-04-26T20:18:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:14 smithi194 ceph-mon[33017]: pgmap v16551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30432.165637 secs 2024-04-26T20:18:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:14 smithi033 ceph-mon[27845]: pgmap v16551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30432.165637 secs 2024-04-26T20:18:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:14 smithi096 ceph-mon[32753]: pgmap v16551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30435.089241 secs 2024-04-26T20:18:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30435.089241 secs 2024-04-26T20:18:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30435.089241 secs 2024-04-26T20:18:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:16 smithi194 ceph-mon[33017]: pgmap v16552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:16 smithi033 ceph-mon[27845]: pgmap v16552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:16 smithi096 ceph-mon[32753]: pgmap v16552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:18 smithi194 ceph-mon[33017]: pgmap v16553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:18 smithi033 ceph-mon[27845]: pgmap v16553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:18 smithi096 ceph-mon[32753]: pgmap v16553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30437.165719 secs 2024-04-26T20:18:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30437.165719 secs 2024-04-26T20:18:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30437.165719 secs 2024-04-26T20:18:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30440.089358 secs 2024-04-26T20:18:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:20 smithi194 ceph-mon[33017]: pgmap v16554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30440.089358 secs 2024-04-26T20:18:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:20 smithi033 ceph-mon[27845]: pgmap v16554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30440.089358 secs 2024-04-26T20:18:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:20 smithi096 ceph-mon[32753]: pgmap v16554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:22 smithi194 ceph-mon[33017]: pgmap v16555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:22 smithi033 ceph-mon[27845]: pgmap v16555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:22 smithi096 ceph-mon[32753]: pgmap v16555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30442.165805 secs 2024-04-26T20:18:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:24 smithi194 ceph-mon[33017]: pgmap v16556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:18:24.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30442.165805 secs 2024-04-26T20:18:24.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:24 smithi033 ceph-mon[27845]: pgmap v16556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:24.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:18:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30442.165805 secs 2024-04-26T20:18:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:24 smithi096 ceph-mon[32753]: pgmap v16556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:18:25.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:25.080+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:25.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:25.080+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:18:25.081 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:25.080+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30445.089450 secs 2024-04-26T20:18:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:18:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:18:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:18:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30445.089450 secs 2024-04-26T20:18:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:18:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:18:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:18:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30445.089450 secs 2024-04-26T20:18:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:18:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:18:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:18:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:26 smithi194 ceph-mon[33017]: pgmap v16557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:18:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:26 smithi194 ceph-mon[33017]: pgmap v16558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:26 smithi033 ceph-mon[27845]: pgmap v16557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:18:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:26 smithi033 ceph-mon[27845]: pgmap v16558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:26 smithi096 ceph-mon[32753]: pgmap v16557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:18:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:26 smithi096 ceph-mon[32753]: pgmap v16558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:28 smithi194 ceph-mon[33017]: pgmap v16559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:28 smithi033 ceph-mon[27845]: pgmap v16559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:28 smithi096 ceph-mon[32753]: pgmap v16559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30447.165890 secs 2024-04-26T20:18:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30447.165890 secs 2024-04-26T20:18:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30447.165890 secs 2024-04-26T20:18:30.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:30 smithi033 ceph-mon[27845]: pgmap v16560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30450.089593 secs 2024-04-26T20:18:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:30 smithi096 ceph-mon[32753]: pgmap v16560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30450.089593 secs 2024-04-26T20:18:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:30 smithi194 ceph-mon[33017]: pgmap v16560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30450.089593 secs 2024-04-26T20:18:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:32 smithi194 ceph-mon[33017]: pgmap v16561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:32 smithi033 ceph-mon[27845]: pgmap v16561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:32 smithi096 ceph-mon[32753]: pgmap v16561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:34 smithi194 ceph-mon[33017]: pgmap v16562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30452.165961 secs 2024-04-26T20:18:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:34 smithi033 ceph-mon[27845]: pgmap v16562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30452.165961 secs 2024-04-26T20:18:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:34 smithi096 ceph-mon[32753]: pgmap v16562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:18:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30452.165961 secs 2024-04-26T20:18:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30455.089691 secs 2024-04-26T20:18:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30455.089691 secs 2024-04-26T20:18:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30455.089691 secs 2024-04-26T20:18:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:36 smithi194 ceph-mon[33017]: pgmap v16563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:18:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:36 smithi033 ceph-mon[27845]: pgmap v16563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:18:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:36 smithi096 ceph-mon[32753]: pgmap v16563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:18:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:38 smithi194 ceph-mon[33017]: pgmap v16564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:38 smithi033 ceph-mon[27845]: pgmap v16564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:38 smithi096 ceph-mon[32753]: pgmap v16564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30457.166050 secs 2024-04-26T20:18:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30457.166050 secs 2024-04-26T20:18:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30457.166050 secs 2024-04-26T20:18:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:40.090+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:40.090+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:40.090+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:18:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:40 smithi194 ceph-mon[33017]: pgmap v16565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30460.089827 secs 2024-04-26T20:18:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:40 smithi033 ceph-mon[27845]: pgmap v16565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30460.089827 secs 2024-04-26T20:18:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:40 smithi096 ceph-mon[32753]: pgmap v16565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30460.089827 secs 2024-04-26T20:18:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:42 smithi033 ceph-mon[27845]: pgmap v16566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:42 smithi096 ceph-mon[32753]: pgmap v16566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:42 smithi194 ceph-mon[33017]: pgmap v16566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:44 smithi033 ceph-mon[27845]: pgmap v16567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30462.166126 secs 2024-04-26T20:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:44 smithi096 ceph-mon[32753]: pgmap v16567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30462.166126 secs 2024-04-26T20:18:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:44 smithi194 ceph-mon[33017]: pgmap v16567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30462.166126 secs 2024-04-26T20:18:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30465.089899 secs 2024-04-26T20:18:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30465.089899 secs 2024-04-26T20:18:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30465.089899 secs 2024-04-26T20:18:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:46 smithi033 ceph-mon[27845]: pgmap v16568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:46 smithi096 ceph-mon[32753]: pgmap v16568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:46 smithi194 ceph-mon[33017]: pgmap v16568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:48 smithi033 ceph-mon[27845]: pgmap v16569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:48 smithi096 ceph-mon[32753]: pgmap v16569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:48 smithi194 ceph-mon[33017]: pgmap v16569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30467.166216 secs 2024-04-26T20:18:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30467.166216 secs 2024-04-26T20:18:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30467.166216 secs 2024-04-26T20:18:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:50 smithi033 ceph-mon[27845]: pgmap v16570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30470.090048 secs 2024-04-26T20:18:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:50 smithi096 ceph-mon[32753]: pgmap v16570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30470.090048 secs 2024-04-26T20:18:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:50 smithi194 ceph-mon[33017]: pgmap v16570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30470.090048 secs 2024-04-26T20:18:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:52 smithi033 ceph-mon[27845]: pgmap v16571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:52 smithi096 ceph-mon[32753]: pgmap v16571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:52 smithi194 ceph-mon[33017]: pgmap v16571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:54 smithi033 ceph-mon[27845]: pgmap v16572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30472.166298 secs 2024-04-26T20:18:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:54 smithi096 ceph-mon[32753]: pgmap v16572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30472.166298 secs 2024-04-26T20:18:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:54 smithi194 ceph-mon[33017]: pgmap v16572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30472.166298 secs 2024-04-26T20:18:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:55.101+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:55.101+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:18:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:18:55.101+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:18:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30475.090165 secs 2024-04-26T20:18:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30475.090165 secs 2024-04-26T20:18:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30475.090165 secs 2024-04-26T20:18:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:56 smithi033 ceph-mon[27845]: pgmap v16573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:56 smithi096 ceph-mon[32753]: pgmap v16573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:56 smithi194 ceph-mon[33017]: pgmap v16573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:58 smithi033 ceph-mon[27845]: pgmap v16574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:58 smithi096 ceph-mon[32753]: pgmap v16574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:58 smithi194 ceph-mon[33017]: pgmap v16574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:18:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:18:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30477.166381 secs 2024-04-26T20:18:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:18:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30477.166381 secs 2024-04-26T20:18:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:18:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30477.166381 secs 2024-04-26T20:19:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:00 smithi096 ceph-mon[32753]: pgmap v16575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30480.090284 secs 2024-04-26T20:19:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:00 smithi194 ceph-mon[33017]: pgmap v16575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30480.090284 secs 2024-04-26T20:19:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:00 smithi033 ceph-mon[27845]: pgmap v16575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30480.090284 secs 2024-04-26T20:19:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:02 smithi194 ceph-mon[33017]: pgmap v16576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:02 smithi033 ceph-mon[27845]: pgmap v16576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:02 smithi096 ceph-mon[32753]: pgmap v16576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:04 smithi194 ceph-mon[33017]: pgmap v16577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30482.166463 secs 2024-04-26T20:19:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:04 smithi033 ceph-mon[27845]: pgmap v16577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30482.166463 secs 2024-04-26T20:19:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:04 smithi096 ceph-mon[32753]: pgmap v16577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30482.166463 secs 2024-04-26T20:19:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30485.090401 secs 2024-04-26T20:19:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30485.090401 secs 2024-04-26T20:19:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30485.090401 secs 2024-04-26T20:19:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:06 smithi194 ceph-mon[33017]: pgmap v16578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:06 smithi033 ceph-mon[27845]: pgmap v16578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:06 smithi096 ceph-mon[32753]: pgmap v16578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:08 smithi194 ceph-mon[33017]: pgmap v16579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:08 smithi033 ceph-mon[27845]: pgmap v16579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:08.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:08 smithi096 ceph-mon[32753]: pgmap v16579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30487.166548 secs 2024-04-26T20:19:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30487.166548 secs 2024-04-26T20:19:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30487.166548 secs 2024-04-26T20:19:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:10.114+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:10.114+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:19:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:10.114+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:10 smithi194 ceph-mon[33017]: pgmap v16580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30490.090480 secs 2024-04-26T20:19:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:10 smithi033 ceph-mon[27845]: pgmap v16580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30490.090480 secs 2024-04-26T20:19:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:10 smithi096 ceph-mon[32753]: pgmap v16580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:19:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30490.090480 secs 2024-04-26T20:19:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:12 smithi194 ceph-mon[33017]: pgmap v16581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:12 smithi033 ceph-mon[27845]: pgmap v16581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:12 smithi096 ceph-mon[32753]: pgmap v16581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:14 smithi194 ceph-mon[33017]: pgmap v16582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30492.166623 secs 2024-04-26T20:19:14.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:14 smithi033 ceph-mon[27845]: pgmap v16582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:14.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30492.166623 secs 2024-04-26T20:19:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:14 smithi096 ceph-mon[32753]: pgmap v16582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30492.166623 secs 2024-04-26T20:19:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30495.090596 secs 2024-04-26T20:19:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30495.090596 secs 2024-04-26T20:19:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30495.090596 secs 2024-04-26T20:19:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:16 smithi194 ceph-mon[33017]: pgmap v16583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:16 smithi033 ceph-mon[27845]: pgmap v16583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:16 smithi096 ceph-mon[32753]: pgmap v16583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:18 smithi194 ceph-mon[33017]: pgmap v16584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:18 smithi033 ceph-mon[27845]: pgmap v16584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:18 smithi096 ceph-mon[32753]: pgmap v16584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30497.166685 secs 2024-04-26T20:19:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30497.166685 secs 2024-04-26T20:19:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30497.166685 secs 2024-04-26T20:19:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:20 smithi194 ceph-mon[33017]: pgmap v16585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30500.090704 secs 2024-04-26T20:19:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:20 smithi033 ceph-mon[27845]: pgmap v16585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30500.090704 secs 2024-04-26T20:19:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:20 smithi096 ceph-mon[32753]: pgmap v16585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30500.090704 secs 2024-04-26T20:19:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:22 smithi194 ceph-mon[33017]: pgmap v16586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:22 smithi033 ceph-mon[27845]: pgmap v16586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:22 smithi096 ceph-mon[32753]: pgmap v16586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:24 smithi194 ceph-mon[33017]: pgmap v16587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30502.166768 secs 2024-04-26T20:19:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:24 smithi033 ceph-mon[27845]: pgmap v16587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30502.166768 secs 2024-04-26T20:19:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:24 smithi096 ceph-mon[32753]: pgmap v16587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30502.166768 secs 2024-04-26T20:19:25.130 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:25.129+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:25.129+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:25.129+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:19:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30505.090856 secs 2024-04-26T20:19:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:19:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30505.090856 secs 2024-04-26T20:19:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:19:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30505.090856 secs 2024-04-26T20:19:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:19:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:26 smithi194 ceph-mon[33017]: pgmap v16588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:26 smithi033 ceph-mon[27845]: pgmap v16588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:26 smithi096 ceph-mon[32753]: pgmap v16588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:19:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:19:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:19:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:19:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:19:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:19:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:19:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:28 smithi194 ceph-mon[33017]: pgmap v16589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:28 smithi194 ceph-mon[33017]: pgmap v16590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:19:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:28 smithi194 ceph-mon[33017]: pgmap v16591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:28 smithi033 ceph-mon[27845]: pgmap v16589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:28 smithi033 ceph-mon[27845]: pgmap v16590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:19:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:28 smithi033 ceph-mon[27845]: pgmap v16591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:28 smithi096 ceph-mon[32753]: pgmap v16589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:28 smithi096 ceph-mon[32753]: pgmap v16590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:19:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:28 smithi096 ceph-mon[32753]: pgmap v16591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30507.166846 secs 2024-04-26T20:19:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:29 smithi194 ceph-mon[33017]: pgmap v16592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30507.166846 secs 2024-04-26T20:19:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:29 smithi033 ceph-mon[27845]: pgmap v16592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:19:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30507.166846 secs 2024-04-26T20:19:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:29 smithi096 ceph-mon[32753]: pgmap v16592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:19:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30510.090998 secs 2024-04-26T20:19:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30510.090998 secs 2024-04-26T20:19:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30510.090998 secs 2024-04-26T20:19:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:31 smithi194 ceph-mon[33017]: pgmap v16593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:31 smithi033 ceph-mon[27845]: pgmap v16593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:31 smithi096 ceph-mon[32753]: pgmap v16593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30512.166929 secs 2024-04-26T20:19:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:34 smithi194 ceph-mon[33017]: pgmap v16594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:19:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30512.166929 secs 2024-04-26T20:19:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:34 smithi033 ceph-mon[27845]: pgmap v16594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:19:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30512.166929 secs 2024-04-26T20:19:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:34 smithi096 ceph-mon[32753]: pgmap v16594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:19:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30515.091128 secs 2024-04-26T20:19:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30515.091128 secs 2024-04-26T20:19:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30515.091128 secs 2024-04-26T20:19:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:36 smithi194 ceph-mon[33017]: pgmap v16595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:36 smithi033 ceph-mon[27845]: pgmap v16595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:36 smithi096 ceph-mon[32753]: pgmap v16595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:19:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:38 smithi194 ceph-mon[33017]: pgmap v16596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:19:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:38 smithi033 ceph-mon[27845]: pgmap v16596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:19:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:38 smithi096 ceph-mon[32753]: pgmap v16596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:19:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30517.167025 secs 2024-04-26T20:19:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30517.167025 secs 2024-04-26T20:19:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30517.167025 secs 2024-04-26T20:19:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:40.142+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:40.142+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:40.142+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:19:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30520.091249 secs 2024-04-26T20:19:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:40 smithi194 ceph-mon[33017]: pgmap v16597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30520.091249 secs 2024-04-26T20:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:40 smithi033 ceph-mon[27845]: pgmap v16597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30520.091249 secs 2024-04-26T20:19:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:40 smithi096 ceph-mon[32753]: pgmap v16597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:42 smithi194 ceph-mon[33017]: pgmap v16598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:42 smithi033 ceph-mon[27845]: pgmap v16598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:42 smithi096 ceph-mon[32753]: pgmap v16598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30522.167104 secs 2024-04-26T20:19:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:44 smithi194 ceph-mon[33017]: pgmap v16599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30522.167104 secs 2024-04-26T20:19:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:44 smithi033 ceph-mon[27845]: pgmap v16599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30522.167104 secs 2024-04-26T20:19:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:44 smithi096 ceph-mon[32753]: pgmap v16599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30525.091327 secs 2024-04-26T20:19:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30525.091327 secs 2024-04-26T20:19:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30525.091327 secs 2024-04-26T20:19:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:46 smithi033 ceph-mon[27845]: pgmap v16600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:46 smithi194 ceph-mon[33017]: pgmap v16600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:46 smithi096 ceph-mon[32753]: pgmap v16600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:48 smithi194 ceph-mon[33017]: pgmap v16601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:48 smithi033 ceph-mon[27845]: pgmap v16601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:48 smithi096 ceph-mon[32753]: pgmap v16601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30527.167190 secs 2024-04-26T20:19:49.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30527.167190 secs 2024-04-26T20:19:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30527.167190 secs 2024-04-26T20:19:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30530.091462 secs 2024-04-26T20:19:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:50 smithi194 ceph-mon[33017]: pgmap v16602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30530.091462 secs 2024-04-26T20:19:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:50 smithi033 ceph-mon[27845]: pgmap v16602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30530.091462 secs 2024-04-26T20:19:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:50 smithi096 ceph-mon[32753]: pgmap v16602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:52 smithi194 ceph-mon[33017]: pgmap v16603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:52 smithi033 ceph-mon[27845]: pgmap v16603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:52 smithi096 ceph-mon[32753]: pgmap v16603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30532.167270 secs 2024-04-26T20:19:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:54 smithi194 ceph-mon[33017]: pgmap v16604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30532.167270 secs 2024-04-26T20:19:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:54 smithi033 ceph-mon[27845]: pgmap v16604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30532.167270 secs 2024-04-26T20:19:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:54 smithi096 ceph-mon[32753]: pgmap v16604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:55.149+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:55.149+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:19:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:19:55.149+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:19:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30535.091585 secs 2024-04-26T20:19:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30535.091585 secs 2024-04-26T20:19:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30535.091585 secs 2024-04-26T20:19:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:56 smithi194 ceph-mon[33017]: pgmap v16605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:56 smithi033 ceph-mon[27845]: pgmap v16605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:56 smithi096 ceph-mon[32753]: pgmap v16605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:58 smithi194 ceph-mon[33017]: pgmap v16606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:58.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:58 smithi033 ceph-mon[27845]: pgmap v16606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:58 smithi096 ceph-mon[32753]: pgmap v16606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:19:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:19:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30537.167354 secs 2024-04-26T20:19:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:19:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30537.167354 secs 2024-04-26T20:19:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:19:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30537.167354 secs 2024-04-26T20:20:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30540.091720 secs 2024-04-26T20:20:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:00 smithi033 ceph-mon[27845]: pgmap v16607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:20:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30540.091720 secs 2024-04-26T20:20:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:00 smithi096 ceph-mon[32753]: pgmap v16607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:20:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30540.091720 secs 2024-04-26T20:20:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:00 smithi194 ceph-mon[33017]: pgmap v16607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:20:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:02 smithi194 ceph-mon[33017]: pgmap v16608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:02 smithi033 ceph-mon[27845]: pgmap v16608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:02 smithi096 ceph-mon[32753]: pgmap v16608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:04.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30542.167438 secs 2024-04-26T20:20:04.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:04 smithi194 ceph-mon[33017]: pgmap v16609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30542.167438 secs 2024-04-26T20:20:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:04 smithi033 ceph-mon[27845]: pgmap v16609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30542.167438 secs 2024-04-26T20:20:04.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:04 smithi096 ceph-mon[32753]: pgmap v16609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30545.091856 secs 2024-04-26T20:20:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30545.091856 secs 2024-04-26T20:20:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30545.091856 secs 2024-04-26T20:20:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:06 smithi194 ceph-mon[33017]: pgmap v16610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:06 smithi033 ceph-mon[27845]: pgmap v16610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:06 smithi096 ceph-mon[32753]: pgmap v16610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:08 smithi194 ceph-mon[33017]: pgmap v16611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:08 smithi033 ceph-mon[27845]: pgmap v16611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:08 smithi096 ceph-mon[32753]: pgmap v16611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30547.167524 secs 2024-04-26T20:20:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30547.167524 secs 2024-04-26T20:20:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30547.167524 secs 2024-04-26T20:20:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:10.159+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:10.159+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:10.159+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:20:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30550.091990 secs 2024-04-26T20:20:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:10 smithi033 ceph-mon[27845]: pgmap v16612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30550.091990 secs 2024-04-26T20:20:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:10 smithi096 ceph-mon[32753]: pgmap v16612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30550.091990 secs 2024-04-26T20:20:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:10 smithi194 ceph-mon[33017]: pgmap v16612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:12 smithi033 ceph-mon[27845]: pgmap v16613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:12 smithi096 ceph-mon[32753]: pgmap v16613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:12 smithi194 ceph-mon[33017]: pgmap v16613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30552.167580 secs 2024-04-26T20:20:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:14 smithi033 ceph-mon[27845]: pgmap v16614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30552.167580 secs 2024-04-26T20:20:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:14 smithi096 ceph-mon[32753]: pgmap v16614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30552.167580 secs 2024-04-26T20:20:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:14 smithi194 ceph-mon[33017]: pgmap v16614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30555.092124 secs 2024-04-26T20:20:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30555.092124 secs 2024-04-26T20:20:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30555.092124 secs 2024-04-26T20:20:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:16 smithi033 ceph-mon[27845]: pgmap v16615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:16 smithi096 ceph-mon[32753]: pgmap v16615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:16 smithi194 ceph-mon[33017]: pgmap v16615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:18 smithi033 ceph-mon[27845]: pgmap v16616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:18 smithi096 ceph-mon[32753]: pgmap v16616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:18 smithi194 ceph-mon[33017]: pgmap v16616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30557.167664 secs 2024-04-26T20:20:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30557.167664 secs 2024-04-26T20:20:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30557.167664 secs 2024-04-26T20:20:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30560.092224 secs 2024-04-26T20:20:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:20 smithi033 ceph-mon[27845]: pgmap v16617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30560.092224 secs 2024-04-26T20:20:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:20 smithi096 ceph-mon[32753]: pgmap v16617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30560.092224 secs 2024-04-26T20:20:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:20 smithi194 ceph-mon[33017]: pgmap v16617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:22 smithi033 ceph-mon[27845]: pgmap v16618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:22 smithi096 ceph-mon[32753]: pgmap v16618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:22 smithi194 ceph-mon[33017]: pgmap v16618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30562.167746 secs 2024-04-26T20:20:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:24 smithi033 ceph-mon[27845]: pgmap v16619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30562.167746 secs 2024-04-26T20:20:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:24 smithi096 ceph-mon[32753]: pgmap v16619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30562.167746 secs 2024-04-26T20:20:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:24 smithi194 ceph-mon[33017]: pgmap v16619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:25.175+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:25.175+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:25.175+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:20:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30565.092370 secs 2024-04-26T20:20:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30565.092370 secs 2024-04-26T20:20:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30565.092370 secs 2024-04-26T20:20:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:26 smithi033 ceph-mon[27845]: pgmap v16620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:26 smithi096 ceph-mon[32753]: pgmap v16620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:26 smithi194 ceph-mon[33017]: pgmap v16620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:26.995+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:20:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:26.995+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T20:20:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:26.995+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 240 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:20:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:26.995+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 240 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:20:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:28 smithi096 ceph-mon[32753]: pgmap v16621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:20:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:28 smithi194 ceph-mon[33017]: pgmap v16621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:20:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:28 smithi033 ceph-mon[27845]: pgmap v16621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:20:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30567.167816 secs 2024-04-26T20:20:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:20:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:20:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:20:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30567.167816 secs 2024-04-26T20:20:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:20:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:20:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:20:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30567.167816 secs 2024-04-26T20:20:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:20:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:20:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:20:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30570.092487 secs 2024-04-26T20:20:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:30 smithi194 ceph-mon[33017]: pgmap v16622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:30 smithi194 ceph-mon[33017]: pgmap v16623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30570.092487 secs 2024-04-26T20:20:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:30 smithi033 ceph-mon[27845]: pgmap v16622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:30 smithi033 ceph-mon[27845]: pgmap v16623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:30.850 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30570.092487 secs 2024-04-26T20:20:30.850 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:30 smithi096 ceph-mon[32753]: pgmap v16622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:30.850 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:30 smithi096 ceph-mon[32753]: pgmap v16623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:32 smithi194 ceph-mon[33017]: pgmap v16624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:32 smithi033 ceph-mon[27845]: pgmap v16624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:32 smithi096 ceph-mon[32753]: pgmap v16624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:34 smithi194 ceph-mon[33017]: pgmap v16625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30572.167868 secs 2024-04-26T20:20:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:34 smithi033 ceph-mon[27845]: pgmap v16625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30572.167868 secs 2024-04-26T20:20:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:34 smithi096 ceph-mon[32753]: pgmap v16625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30572.167868 secs 2024-04-26T20:20:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30575.092564 secs 2024-04-26T20:20:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30575.092564 secs 2024-04-26T20:20:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30575.092564 secs 2024-04-26T20:20:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:36 smithi194 ceph-mon[33017]: pgmap v16626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:36 smithi033 ceph-mon[27845]: pgmap v16626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:36 smithi096 ceph-mon[32753]: pgmap v16626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:38 smithi194 ceph-mon[33017]: pgmap v16627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:38 smithi033 ceph-mon[27845]: pgmap v16627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:38 smithi096 ceph-mon[32753]: pgmap v16627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:20:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30577.167929 secs 2024-04-26T20:20:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30577.167929 secs 2024-04-26T20:20:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30577.167929 secs 2024-04-26T20:20:40.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:40.187+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:40.187+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:40.187+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:20:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30580.092625 secs 2024-04-26T20:20:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:40 smithi194 ceph-mon[33017]: pgmap v16628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:20:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30580.092625 secs 2024-04-26T20:20:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:40 smithi033 ceph-mon[27845]: pgmap v16628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:20:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30580.092625 secs 2024-04-26T20:20:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:40 smithi096 ceph-mon[32753]: pgmap v16628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:20:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:42 smithi194 ceph-mon[33017]: pgmap v16629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:42 smithi033 ceph-mon[27845]: pgmap v16629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:42 smithi096 ceph-mon[32753]: pgmap v16629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:44 smithi194 ceph-mon[33017]: pgmap v16630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30582.168010 secs 2024-04-26T20:20:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:44 smithi033 ceph-mon[27845]: pgmap v16630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30582.168010 secs 2024-04-26T20:20:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:44 smithi096 ceph-mon[32753]: pgmap v16630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30582.168010 secs 2024-04-26T20:20:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30585.092762 secs 2024-04-26T20:20:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30585.092762 secs 2024-04-26T20:20:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30585.092762 secs 2024-04-26T20:20:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:46 smithi194 ceph-mon[33017]: pgmap v16631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:46 smithi033 ceph-mon[27845]: pgmap v16631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:46 smithi096 ceph-mon[32753]: pgmap v16631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:48 smithi194 ceph-mon[33017]: pgmap v16632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:48 smithi033 ceph-mon[27845]: pgmap v16632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:48 smithi096 ceph-mon[32753]: pgmap v16632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30587.168095 secs 2024-04-26T20:20:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30587.168095 secs 2024-04-26T20:20:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30587.168095 secs 2024-04-26T20:20:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30590.092880 secs 2024-04-26T20:20:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:50 smithi194 ceph-mon[33017]: pgmap v16633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30590.092880 secs 2024-04-26T20:20:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:50 smithi033 ceph-mon[27845]: pgmap v16633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30590.092880 secs 2024-04-26T20:20:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:50 smithi096 ceph-mon[32753]: pgmap v16633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:52 smithi194 ceph-mon[33017]: pgmap v16634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:52 smithi033 ceph-mon[27845]: pgmap v16634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:52 smithi096 ceph-mon[32753]: pgmap v16634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:54 smithi194 ceph-mon[33017]: pgmap v16635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30592.168158 secs 2024-04-26T20:20:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:54 smithi033 ceph-mon[27845]: pgmap v16635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30592.168158 secs 2024-04-26T20:20:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:54 smithi096 ceph-mon[32753]: pgmap v16635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30592.168158 secs 2024-04-26T20:20:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:55.189+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:55.189+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:20:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:20:55.189+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:20:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30595.092995 secs 2024-04-26T20:20:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30595.092995 secs 2024-04-26T20:20:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30595.092995 secs 2024-04-26T20:20:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:56 smithi194 ceph-mon[33017]: pgmap v16636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:56 smithi033 ceph-mon[27845]: pgmap v16636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:56 smithi096 ceph-mon[32753]: pgmap v16636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:58 smithi194 ceph-mon[33017]: pgmap v16637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:58 smithi033 ceph-mon[27845]: pgmap v16637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:58 smithi096 ceph-mon[32753]: pgmap v16637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:20:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:20:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30597.168240 secs 2024-04-26T20:20:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:20:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30597.168240 secs 2024-04-26T20:20:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:20:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30597.168240 secs 2024-04-26T20:21:00.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30600.093128 secs 2024-04-26T20:21:00.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:00 smithi033 ceph-mon[27845]: pgmap v16638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30600.093128 secs 2024-04-26T20:21:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:00 smithi096 ceph-mon[32753]: pgmap v16638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30600.093128 secs 2024-04-26T20:21:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:00 smithi194 ceph-mon[33017]: pgmap v16638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:01 smithi194 ceph-mon[33017]: pgmap v16639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:01 smithi033 ceph-mon[27845]: pgmap v16639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:01 smithi096 ceph-mon[32753]: pgmap v16639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:03 smithi033 ceph-mon[27845]: pgmap v16640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30602.168323 secs 2024-04-26T20:21:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:03 smithi096 ceph-mon[32753]: pgmap v16640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30602.168323 secs 2024-04-26T20:21:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:03 smithi194 ceph-mon[33017]: pgmap v16640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30602.168323 secs 2024-04-26T20:21:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30605.093248 secs 2024-04-26T20:21:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30605.093248 secs 2024-04-26T20:21:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30605.093248 secs 2024-04-26T20:21:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:05 smithi194 ceph-mon[33017]: pgmap v16641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:05 smithi033 ceph-mon[27845]: pgmap v16641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:06.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:05 smithi096 ceph-mon[32753]: pgmap v16641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:07 smithi194 ceph-mon[33017]: pgmap v16642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:07 smithi033 ceph-mon[27845]: pgmap v16642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:07 smithi096 ceph-mon[32753]: pgmap v16642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30607.168385 secs 2024-04-26T20:21:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30607.168385 secs 2024-04-26T20:21:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30607.168385 secs 2024-04-26T20:21:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:10.201+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:10.201+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:21:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:10.201+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30610.093324 secs 2024-04-26T20:21:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:09 smithi194 ceph-mon[33017]: pgmap v16643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30610.093324 secs 2024-04-26T20:21:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:09 smithi033 ceph-mon[27845]: pgmap v16643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30610.093324 secs 2024-04-26T20:21:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:09 smithi096 ceph-mon[32753]: pgmap v16643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:12.003 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:12.002+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T20:21:12.004 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:12.002+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:21:12.004 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:12.002+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 241 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T20:21:12.004 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:12.003+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 241 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:21:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:11 smithi194 ceph-mon[33017]: pgmap v16644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:11 smithi033 ceph-mon[27845]: pgmap v16644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:11 smithi096 ceph-mon[32753]: pgmap v16644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:13 smithi194 ceph-mon[33017]: pgmap v16645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30612.168469 secs 2024-04-26T20:21:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:13 smithi033 ceph-mon[27845]: pgmap v16645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30612.168469 secs 2024-04-26T20:21:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:13 smithi096 ceph-mon[32753]: pgmap v16645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30612.168469 secs 2024-04-26T20:21:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30615.093435 secs 2024-04-26T20:21:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30615.093435 secs 2024-04-26T20:21:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30615.093435 secs 2024-04-26T20:21:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:15 smithi194 ceph-mon[33017]: pgmap v16646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:15 smithi033 ceph-mon[27845]: pgmap v16646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:15 smithi096 ceph-mon[32753]: pgmap v16646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:17 smithi194 ceph-mon[33017]: pgmap v16647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:17 smithi033 ceph-mon[27845]: pgmap v16647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:17 smithi096 ceph-mon[32753]: pgmap v16647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:19.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30617.168531 secs 2024-04-26T20:21:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30617.168531 secs 2024-04-26T20:21:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30617.168531 secs 2024-04-26T20:21:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30620.093525 secs 2024-04-26T20:21:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:20 smithi194 ceph-mon[33017]: pgmap v16648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30620.093525 secs 2024-04-26T20:21:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:20 smithi033 ceph-mon[27845]: pgmap v16648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30620.093525 secs 2024-04-26T20:21:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:20 smithi096 ceph-mon[32753]: pgmap v16648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:22 smithi194 ceph-mon[33017]: pgmap v16649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:22 smithi033 ceph-mon[27845]: pgmap v16649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:22 smithi096 ceph-mon[32753]: pgmap v16649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:24 smithi194 ceph-mon[33017]: pgmap v16650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30622.168618 secs 2024-04-26T20:21:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:24 smithi033 ceph-mon[27845]: pgmap v16650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30622.168618 secs 2024-04-26T20:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:24 smithi096 ceph-mon[32753]: pgmap v16650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30622.168618 secs 2024-04-26T20:21:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:25.202+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:25.202+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:21:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:25.202+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30625.093652 secs 2024-04-26T20:21:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30625.093652 secs 2024-04-26T20:21:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30625.093652 secs 2024-04-26T20:21:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:26 smithi194 ceph-mon[33017]: pgmap v16651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:26 smithi033 ceph-mon[27845]: pgmap v16651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:26 smithi096 ceph-mon[32753]: pgmap v16651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:28 smithi194 ceph-mon[33017]: pgmap v16652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:28 smithi033 ceph-mon[27845]: pgmap v16652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:28 smithi096 ceph-mon[32753]: pgmap v16652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30627.168703 secs 2024-04-26T20:21:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30627.168703 secs 2024-04-26T20:21:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30627.168703 secs 2024-04-26T20:21:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30630.093771 secs 2024-04-26T20:21:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:30 smithi033 ceph-mon[27845]: pgmap v16653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:21:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30630.093771 secs 2024-04-26T20:21:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:30 smithi096 ceph-mon[32753]: pgmap v16653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:21:30.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30630.093771 secs 2024-04-26T20:21:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:30 smithi194 ceph-mon[33017]: pgmap v16653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:21:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:32 smithi194 ceph-mon[33017]: pgmap v16654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:32 smithi033 ceph-mon[27845]: pgmap v16654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:32 smithi096 ceph-mon[32753]: pgmap v16654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:21:33.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:33 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:21:33.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:21:33.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:33 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:21:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:33.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:21:33.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:33 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:21:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:34 smithi194 ceph-mon[33017]: pgmap v16655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30632.168784 secs 2024-04-26T20:21:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:34 smithi033 ceph-mon[27845]: pgmap v16655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30632.168784 secs 2024-04-26T20:21:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:34 smithi096 ceph-mon[32753]: pgmap v16655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30632.168784 secs 2024-04-26T20:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30635.093886 secs 2024-04-26T20:21:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:21:35.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:21:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30635.093886 secs 2024-04-26T20:21:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:21:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:21:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30635.093886 secs 2024-04-26T20:21:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:21:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:21:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:36 smithi194 ceph-mon[33017]: pgmap v16656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:36 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:36 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:36 smithi194 ceph-mon[33017]: pgmap v16657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:36 smithi194 ceph-mon[33017]: pgmap v16658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:36 smithi033 ceph-mon[27845]: pgmap v16656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:21:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:36 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:21:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:36 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:21:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:36 smithi033 ceph-mon[27845]: pgmap v16657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:21:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:36 smithi033 ceph-mon[27845]: pgmap v16658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:36.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:36 smithi096 ceph-mon[32753]: pgmap v16656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:21:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:36 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:21:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:36 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:21:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:36 smithi096 ceph-mon[32753]: pgmap v16657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:21:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:36 smithi096 ceph-mon[32753]: pgmap v16658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:36.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:21:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:38 smithi194 ceph-mon[33017]: pgmap v16659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:38 smithi033 ceph-mon[27845]: pgmap v16659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:38 smithi096 ceph-mon[32753]: pgmap v16659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30637.168868 secs 2024-04-26T20:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30637.168868 secs 2024-04-26T20:21:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30637.168868 secs 2024-04-26T20:21:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:40.211+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:40.211+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:40.211+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:21:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30640.094008 secs 2024-04-26T20:21:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:40 smithi194 ceph-mon[33017]: pgmap v16660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:21:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30640.094008 secs 2024-04-26T20:21:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:40 smithi033 ceph-mon[27845]: pgmap v16660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:21:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30640.094008 secs 2024-04-26T20:21:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:40 smithi096 ceph-mon[32753]: pgmap v16660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:21:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:42 smithi033 ceph-mon[27845]: pgmap v16661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:42.593 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:42 smithi096 ceph-mon[32753]: pgmap v16661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:42 smithi194 ceph-mon[33017]: pgmap v16661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:21:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30642.168946 secs 2024-04-26T20:21:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:44 smithi033 ceph-mon[27845]: pgmap v16662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:21:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30642.168946 secs 2024-04-26T20:21:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:44 smithi096 ceph-mon[32753]: pgmap v16662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:21:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30642.168946 secs 2024-04-26T20:21:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:44 smithi194 ceph-mon[33017]: pgmap v16662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:21:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30645.094144 secs 2024-04-26T20:21:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30645.094144 secs 2024-04-26T20:21:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30645.094144 secs 2024-04-26T20:21:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:46 smithi033 ceph-mon[27845]: pgmap v16663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:21:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:46 smithi096 ceph-mon[32753]: pgmap v16663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:21:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:46 smithi194 ceph-mon[33017]: pgmap v16663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:21:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:48 smithi033 ceph-mon[27845]: pgmap v16664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:48 smithi096 ceph-mon[32753]: pgmap v16664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:48 smithi194 ceph-mon[33017]: pgmap v16664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30647.169031 secs 2024-04-26T20:21:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30647.169031 secs 2024-04-26T20:21:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30647.169031 secs 2024-04-26T20:21:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30650.094268 secs 2024-04-26T20:21:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:50 smithi033 ceph-mon[27845]: pgmap v16665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30650.094268 secs 2024-04-26T20:21:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:50 smithi096 ceph-mon[32753]: pgmap v16665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30650.094268 secs 2024-04-26T20:21:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:50 smithi194 ceph-mon[33017]: pgmap v16665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:52 smithi033 ceph-mon[27845]: pgmap v16666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:52 smithi096 ceph-mon[32753]: pgmap v16666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:52 smithi194 ceph-mon[33017]: pgmap v16666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30652.169117 secs 2024-04-26T20:21:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:54 smithi033 ceph-mon[27845]: pgmap v16667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30652.169117 secs 2024-04-26T20:21:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:54 smithi096 ceph-mon[32753]: pgmap v16667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30652.169117 secs 2024-04-26T20:21:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:54 smithi194 ceph-mon[33017]: pgmap v16667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:55.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:55.224+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:55.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:55.224+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:21:55.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:21:55.224+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:21:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30655.094349 secs 2024-04-26T20:21:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30655.094349 secs 2024-04-26T20:21:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30655.094349 secs 2024-04-26T20:21:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:56 smithi033 ceph-mon[27845]: pgmap v16668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:56 smithi096 ceph-mon[32753]: pgmap v16668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:56 smithi194 ceph-mon[33017]: pgmap v16668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:58 smithi033 ceph-mon[27845]: pgmap v16669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:58 smithi096 ceph-mon[32753]: pgmap v16669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:58 smithi194 ceph-mon[33017]: pgmap v16669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:21:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:21:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30657.169180 secs 2024-04-26T20:21:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:21:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30657.169180 secs 2024-04-26T20:21:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:21:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30657.169180 secs 2024-04-26T20:22:00.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30660.094423 secs 2024-04-26T20:22:00.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:00 smithi194 ceph-mon[33017]: pgmap v16670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30660.094423 secs 2024-04-26T20:22:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:00 smithi033 ceph-mon[27845]: pgmap v16670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30660.094423 secs 2024-04-26T20:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:00 smithi096 ceph-mon[32753]: pgmap v16670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:02 smithi194 ceph-mon[33017]: pgmap v16671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:02 smithi033 ceph-mon[27845]: pgmap v16671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:02 smithi096 ceph-mon[32753]: pgmap v16671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30662.169267 secs 2024-04-26T20:22:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:04 smithi194 ceph-mon[33017]: pgmap v16672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30662.169267 secs 2024-04-26T20:22:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:04 smithi033 ceph-mon[27845]: pgmap v16672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30662.169267 secs 2024-04-26T20:22:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:04 smithi096 ceph-mon[32753]: pgmap v16672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30665.094540 secs 2024-04-26T20:22:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30665.094540 secs 2024-04-26T20:22:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30665.094540 secs 2024-04-26T20:22:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:06 smithi194 ceph-mon[33017]: pgmap v16673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:06 smithi033 ceph-mon[27845]: pgmap v16673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:06 smithi096 ceph-mon[32753]: pgmap v16673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:08 smithi194 ceph-mon[33017]: pgmap v16674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:08 smithi033 ceph-mon[27845]: pgmap v16674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:08 smithi096 ceph-mon[32753]: pgmap v16674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30667.169328 secs 2024-04-26T20:22:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30667.169328 secs 2024-04-26T20:22:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30667.169328 secs 2024-04-26T20:22:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:10.236+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:10.236+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:22:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:10.236+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30670.094657 secs 2024-04-26T20:22:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:10 smithi194 ceph-mon[33017]: pgmap v16675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30670.094657 secs 2024-04-26T20:22:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:10 smithi033 ceph-mon[27845]: pgmap v16675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30670.094657 secs 2024-04-26T20:22:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:10 smithi096 ceph-mon[32753]: pgmap v16675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:12 smithi194 ceph-mon[33017]: pgmap v16676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:12 smithi033 ceph-mon[27845]: pgmap v16676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:12 smithi096 ceph-mon[32753]: pgmap v16676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30672.169400 secs 2024-04-26T20:22:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:14 smithi194 ceph-mon[33017]: pgmap v16677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30672.169400 secs 2024-04-26T20:22:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:14 smithi033 ceph-mon[27845]: pgmap v16677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30672.169400 secs 2024-04-26T20:22:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:14 smithi096 ceph-mon[32753]: pgmap v16677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30675.094798 secs 2024-04-26T20:22:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30675.094798 secs 2024-04-26T20:22:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30675.094798 secs 2024-04-26T20:22:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:16 smithi194 ceph-mon[33017]: pgmap v16678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:16 smithi033 ceph-mon[27845]: pgmap v16678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:16 smithi096 ceph-mon[32753]: pgmap v16678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:18 smithi194 ceph-mon[33017]: pgmap v16679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:18 smithi033 ceph-mon[27845]: pgmap v16679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:18 smithi096 ceph-mon[32753]: pgmap v16679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30677.169460 secs 2024-04-26T20:22:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30677.169460 secs 2024-04-26T20:22:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30677.169460 secs 2024-04-26T20:22:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30680.094925 secs 2024-04-26T20:22:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:20 smithi194 ceph-mon[33017]: pgmap v16680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30680.094925 secs 2024-04-26T20:22:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:20 smithi033 ceph-mon[27845]: pgmap v16680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30680.094925 secs 2024-04-26T20:22:20.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:20 smithi096 ceph-mon[32753]: pgmap v16680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:22 smithi194 ceph-mon[33017]: pgmap v16681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:22 smithi033 ceph-mon[27845]: pgmap v16681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:22 smithi096 ceph-mon[32753]: pgmap v16681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30682.169542 secs 2024-04-26T20:22:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:24 smithi194 ceph-mon[33017]: pgmap v16682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30682.169542 secs 2024-04-26T20:22:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:24 smithi033 ceph-mon[27845]: pgmap v16682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30682.169542 secs 2024-04-26T20:22:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:24 smithi096 ceph-mon[32753]: pgmap v16682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:25.242+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:25.242+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:22:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:25.242+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30685.095059 secs 2024-04-26T20:22:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30685.095059 secs 2024-04-26T20:22:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30685.095059 secs 2024-04-26T20:22:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:26 smithi194 ceph-mon[33017]: pgmap v16683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:26 smithi033 ceph-mon[27845]: pgmap v16683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:26 smithi096 ceph-mon[32753]: pgmap v16683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:28 smithi194 ceph-mon[33017]: pgmap v16684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:28 smithi033 ceph-mon[27845]: pgmap v16684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:28 smithi096 ceph-mon[32753]: pgmap v16684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30687.169626 secs 2024-04-26T20:22:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30687.169626 secs 2024-04-26T20:22:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30687.169626 secs 2024-04-26T20:22:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30690.095186 secs 2024-04-26T20:22:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:30 smithi033 ceph-mon[27845]: pgmap v16685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30690.095186 secs 2024-04-26T20:22:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:30 smithi096 ceph-mon[32753]: pgmap v16685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30690.095186 secs 2024-04-26T20:22:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:30 smithi194 ceph-mon[33017]: pgmap v16685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:32 smithi194 ceph-mon[33017]: pgmap v16686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:32 smithi033 ceph-mon[27845]: pgmap v16686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:32 smithi096 ceph-mon[32753]: pgmap v16686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30692.169704 secs 2024-04-26T20:22:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:34 smithi194 ceph-mon[33017]: pgmap v16687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30692.169704 secs 2024-04-26T20:22:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:34 smithi033 ceph-mon[27845]: pgmap v16687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30692.169704 secs 2024-04-26T20:22:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:34 smithi096 ceph-mon[32753]: pgmap v16687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30695.095330 secs 2024-04-26T20:22:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30695.095330 secs 2024-04-26T20:22:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30695.095330 secs 2024-04-26T20:22:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:36 smithi194 ceph-mon[33017]: pgmap v16688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:22:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:36 smithi033 ceph-mon[27845]: pgmap v16688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:22:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:36 smithi096 ceph-mon[32753]: pgmap v16688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:22:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:22:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:22:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:22:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:22:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:22:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:22:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:22:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:22:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:22:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:38 smithi194 ceph-mon[33017]: pgmap v16689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:38 smithi194 ceph-mon[33017]: pgmap v16690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:38 smithi033 ceph-mon[27845]: pgmap v16689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:38 smithi033 ceph-mon[27845]: pgmap v16690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:38 smithi096 ceph-mon[32753]: pgmap v16689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:38 smithi096 ceph-mon[32753]: pgmap v16690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30697.169788 secs 2024-04-26T20:22:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30697.169788 secs 2024-04-26T20:22:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30697.169788 secs 2024-04-26T20:22:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:40.254+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:40.254+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:40.254+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:22:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30700.095470 secs 2024-04-26T20:22:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:40 smithi033 ceph-mon[27845]: pgmap v16691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30700.095470 secs 2024-04-26T20:22:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:40 smithi096 ceph-mon[32753]: pgmap v16691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30700.095470 secs 2024-04-26T20:22:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:40 smithi194 ceph-mon[33017]: pgmap v16691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:42 smithi033 ceph-mon[27845]: pgmap v16692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:42 smithi096 ceph-mon[32753]: pgmap v16692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:42 smithi194 ceph-mon[33017]: pgmap v16692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:44 smithi033 ceph-mon[27845]: pgmap v16693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30702.169869 secs 2024-04-26T20:22:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:44 smithi096 ceph-mon[32753]: pgmap v16693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30702.169869 secs 2024-04-26T20:22:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:44 smithi194 ceph-mon[33017]: pgmap v16693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30702.169869 secs 2024-04-26T20:22:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30705.095554 secs 2024-04-26T20:22:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30705.095554 secs 2024-04-26T20:22:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30705.095554 secs 2024-04-26T20:22:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:46 smithi033 ceph-mon[27845]: pgmap v16694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:46 smithi096 ceph-mon[32753]: pgmap v16694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:46 smithi194 ceph-mon[33017]: pgmap v16694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:22:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:48 smithi033 ceph-mon[27845]: pgmap v16695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:22:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:48 smithi096 ceph-mon[32753]: pgmap v16695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:22:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:48 smithi194 ceph-mon[33017]: pgmap v16695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:22:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30707.169949 secs 2024-04-26T20:22:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30707.169949 secs 2024-04-26T20:22:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30707.169949 secs 2024-04-26T20:22:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30710.095714 secs 2024-04-26T20:22:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:50 smithi033 ceph-mon[27845]: pgmap v16696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30710.095714 secs 2024-04-26T20:22:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:50 smithi096 ceph-mon[32753]: pgmap v16696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30710.095714 secs 2024-04-26T20:22:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:50 smithi194 ceph-mon[33017]: pgmap v16696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:52 smithi033 ceph-mon[27845]: pgmap v16697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:52 smithi096 ceph-mon[32753]: pgmap v16697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:52 smithi194 ceph-mon[33017]: pgmap v16697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:54 smithi033 ceph-mon[27845]: pgmap v16698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30712.170037 secs 2024-04-26T20:22:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:54 smithi096 ceph-mon[32753]: pgmap v16698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30712.170037 secs 2024-04-26T20:22:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:54 smithi194 ceph-mon[33017]: pgmap v16698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30712.170037 secs 2024-04-26T20:22:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:55.268+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:55.268+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:22:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:22:55.268+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:22:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30715.095842 secs 2024-04-26T20:22:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30715.095842 secs 2024-04-26T20:22:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30715.095842 secs 2024-04-26T20:22:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:56 smithi033 ceph-mon[27845]: pgmap v16699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:56 smithi096 ceph-mon[32753]: pgmap v16699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:56 smithi194 ceph-mon[33017]: pgmap v16699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:58 smithi096 ceph-mon[32753]: pgmap v16700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:58 smithi194 ceph-mon[33017]: pgmap v16700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:58 smithi033 ceph-mon[27845]: pgmap v16700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:22:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:22:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30717.170129 secs 2024-04-26T20:22:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:22:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30717.170129 secs 2024-04-26T20:22:59.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:22:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30717.170129 secs 2024-04-26T20:23:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30720.095975 secs 2024-04-26T20:23:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:00 smithi194 ceph-mon[33017]: pgmap v16701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:00.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30720.095975 secs 2024-04-26T20:23:00.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:00 smithi033 ceph-mon[27845]: pgmap v16701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30720.095975 secs 2024-04-26T20:23:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:00 smithi096 ceph-mon[32753]: pgmap v16701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:02 smithi194 ceph-mon[33017]: pgmap v16702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:02 smithi033 ceph-mon[27845]: pgmap v16702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:02 smithi096 ceph-mon[32753]: pgmap v16702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:04 smithi194 ceph-mon[33017]: pgmap v16703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30722.170180 secs 2024-04-26T20:23:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:04 smithi033 ceph-mon[27845]: pgmap v16703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30722.170180 secs 2024-04-26T20:23:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:04 smithi096 ceph-mon[32753]: pgmap v16703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30722.170180 secs 2024-04-26T20:23:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30725.096132 secs 2024-04-26T20:23:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30725.096132 secs 2024-04-26T20:23:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30725.096132 secs 2024-04-26T20:23:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:06 smithi194 ceph-mon[33017]: pgmap v16704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:06 smithi033 ceph-mon[27845]: pgmap v16704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:06 smithi096 ceph-mon[32753]: pgmap v16704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:08 smithi033 ceph-mon[27845]: pgmap v16705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:08 smithi194 ceph-mon[33017]: pgmap v16705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:08 smithi096 ceph-mon[32753]: pgmap v16705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30727.170281 secs 2024-04-26T20:23:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30727.170281 secs 2024-04-26T20:23:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30727.170281 secs 2024-04-26T20:23:10.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:10.279+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:10.279+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:10.279+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:23:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30730.096176 secs 2024-04-26T20:23:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:10 smithi194 ceph-mon[33017]: pgmap v16706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30730.096176 secs 2024-04-26T20:23:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:10 smithi033 ceph-mon[27845]: pgmap v16706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30730.096176 secs 2024-04-26T20:23:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:10 smithi096 ceph-mon[32753]: pgmap v16706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:12 smithi194 ceph-mon[33017]: pgmap v16707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:12 smithi033 ceph-mon[27845]: pgmap v16707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:12 smithi096 ceph-mon[32753]: pgmap v16707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:14 smithi194 ceph-mon[33017]: pgmap v16708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30732.170361 secs 2024-04-26T20:23:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:14 smithi033 ceph-mon[27845]: pgmap v16708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30732.170361 secs 2024-04-26T20:23:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:14 smithi096 ceph-mon[32753]: pgmap v16708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30732.170361 secs 2024-04-26T20:23:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30735.096292 secs 2024-04-26T20:23:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30735.096292 secs 2024-04-26T20:23:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30735.096292 secs 2024-04-26T20:23:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:16 smithi194 ceph-mon[33017]: pgmap v16709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:16 smithi033 ceph-mon[27845]: pgmap v16709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:16 smithi096 ceph-mon[32753]: pgmap v16709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:18 smithi194 ceph-mon[33017]: pgmap v16710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:18 smithi033 ceph-mon[27845]: pgmap v16710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:18 smithi096 ceph-mon[32753]: pgmap v16710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30737.170421 secs 2024-04-26T20:23:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30737.170421 secs 2024-04-26T20:23:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30737.170421 secs 2024-04-26T20:23:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30740.096384 secs 2024-04-26T20:23:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:20 smithi194 ceph-mon[33017]: pgmap v16711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30740.096384 secs 2024-04-26T20:23:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:20 smithi033 ceph-mon[27845]: pgmap v16711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30740.096384 secs 2024-04-26T20:23:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:20 smithi096 ceph-mon[32753]: pgmap v16711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:22 smithi194 ceph-mon[33017]: pgmap v16712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:22 smithi033 ceph-mon[27845]: pgmap v16712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:22 smithi096 ceph-mon[32753]: pgmap v16712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:24 smithi194 ceph-mon[33017]: pgmap v16713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30742.170500 secs 2024-04-26T20:23:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:24 smithi033 ceph-mon[27845]: pgmap v16713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30742.170500 secs 2024-04-26T20:23:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:24 smithi096 ceph-mon[32753]: pgmap v16713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30742.170500 secs 2024-04-26T20:23:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:25.290+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:25.290+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:25.290+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:23:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30745.096501 secs 2024-04-26T20:23:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30745.096501 secs 2024-04-26T20:23:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30745.096501 secs 2024-04-26T20:23:26.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:26 smithi194 ceph-mon[33017]: pgmap v16714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:26 smithi033 ceph-mon[27845]: pgmap v16714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:26 smithi096 ceph-mon[32753]: pgmap v16714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:27.025+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T20:23:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:27.025+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc0065f0 con 0x564385821900 2024-04-26T20:23:27.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:27.025+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 242 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T20:23:27.027 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:27.025+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 242 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:23:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:28 smithi194 ceph-mon[33017]: pgmap v16715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:28 smithi033 ceph-mon[27845]: pgmap v16715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:28 smithi096 ceph-mon[32753]: pgmap v16715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30747.170586 secs 2024-04-26T20:23:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30747.170586 secs 2024-04-26T20:23:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30747.170586 secs 2024-04-26T20:23:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30750.096617 secs 2024-04-26T20:23:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:30 smithi033 ceph-mon[27845]: pgmap v16716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30750.096617 secs 2024-04-26T20:23:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:30 smithi096 ceph-mon[32753]: pgmap v16716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30750.096617 secs 2024-04-26T20:23:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:30 smithi194 ceph-mon[33017]: pgmap v16716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:32 smithi194 ceph-mon[33017]: pgmap v16717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:32 smithi033 ceph-mon[27845]: pgmap v16717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:32 smithi096 ceph-mon[32753]: pgmap v16717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:34 smithi194 ceph-mon[33017]: pgmap v16718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30752.170666 secs 2024-04-26T20:23:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:34 smithi033 ceph-mon[27845]: pgmap v16718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30752.170666 secs 2024-04-26T20:23:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:34 smithi096 ceph-mon[32753]: pgmap v16718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30752.170666 secs 2024-04-26T20:23:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30755.096733 secs 2024-04-26T20:23:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30755.096733 secs 2024-04-26T20:23:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30755.096733 secs 2024-04-26T20:23:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:36 smithi194 ceph-mon[33017]: pgmap v16719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:36 smithi033 ceph-mon[27845]: pgmap v16719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:36 smithi096 ceph-mon[32753]: pgmap v16719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:38 smithi033 ceph-mon[27845]: pgmap v16720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:23:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:23:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:38 smithi033 ceph-mon[27845]: pgmap v16721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:38 smithi033 ceph-mon[27845]: pgmap v16722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:38 smithi096 ceph-mon[32753]: pgmap v16720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:38 smithi096 ceph-mon[32753]: pgmap v16721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:38 smithi096 ceph-mon[32753]: pgmap v16722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:23:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:38 smithi194 ceph-mon[33017]: pgmap v16720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:38 smithi194 ceph-mon[33017]: pgmap v16721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:38 smithi194 ceph-mon[33017]: pgmap v16722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:23:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30757.170752 secs 2024-04-26T20:23:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30760.096868 secs 2024-04-26T20:23:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30757.170752 secs 2024-04-26T20:23:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30760.096868 secs 2024-04-26T20:23:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30757.170752 secs 2024-04-26T20:23:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30760.096868 secs 2024-04-26T20:23:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:40.307+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:40.307+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:23:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:40.307+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:40 smithi033 ceph-mon[27845]: pgmap v16723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:40 smithi096 ceph-mon[32753]: pgmap v16723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:40 smithi194 ceph-mon[33017]: pgmap v16723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:43 smithi194 ceph-mon[33017]: pgmap v16724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:43 smithi033 ceph-mon[27845]: pgmap v16724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:43 smithi096 ceph-mon[32753]: pgmap v16724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30762.170799 secs 2024-04-26T20:23:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30762.170799 secs 2024-04-26T20:23:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30762.170799 secs 2024-04-26T20:23:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30765.096991 secs 2024-04-26T20:23:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:45 smithi194 ceph-mon[33017]: pgmap v16725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:23:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30765.096991 secs 2024-04-26T20:23:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:45 smithi033 ceph-mon[27845]: pgmap v16725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:23:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30765.096991 secs 2024-04-26T20:23:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:45 smithi096 ceph-mon[32753]: pgmap v16725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:23:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:47 smithi194 ceph-mon[33017]: pgmap v16726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:47 smithi033 ceph-mon[27845]: pgmap v16726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:47 smithi096 ceph-mon[32753]: pgmap v16726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30767.170857 secs 2024-04-26T20:23:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:49 smithi194 ceph-mon[33017]: pgmap v16727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:23:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30767.170857 secs 2024-04-26T20:23:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:49 smithi033 ceph-mon[27845]: pgmap v16727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:23:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30767.170857 secs 2024-04-26T20:23:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:49 smithi096 ceph-mon[32753]: pgmap v16727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:23:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30770.097124 secs 2024-04-26T20:23:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30770.097124 secs 2024-04-26T20:23:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30770.097124 secs 2024-04-26T20:23:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:51 smithi194 ceph-mon[33017]: pgmap v16728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:51 smithi033 ceph-mon[27845]: pgmap v16728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:51 smithi096 ceph-mon[32753]: pgmap v16728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:53 smithi194 ceph-mon[33017]: pgmap v16729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:53 smithi033 ceph-mon[27845]: pgmap v16729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:53 smithi096 ceph-mon[32753]: pgmap v16729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30772.170913 secs 2024-04-26T20:23:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30772.170913 secs 2024-04-26T20:23:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30772.170913 secs 2024-04-26T20:23:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:55.317+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:55.317+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:23:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:23:55.317+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:23:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30775.097209 secs 2024-04-26T20:23:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:55 smithi194 ceph-mon[33017]: pgmap v16730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30775.097209 secs 2024-04-26T20:23:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:55 smithi033 ceph-mon[27845]: pgmap v16730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30775.097209 secs 2024-04-26T20:23:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:55 smithi096 ceph-mon[32753]: pgmap v16730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:23:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:57 smithi194 ceph-mon[33017]: pgmap v16731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:23:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:57 smithi033 ceph-mon[27845]: pgmap v16731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:23:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:57 smithi096 ceph-mon[32753]: pgmap v16731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:23:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30777.170976 secs 2024-04-26T20:23:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:23:59 smithi194 ceph-mon[33017]: pgmap v16732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30777.170976 secs 2024-04-26T20:23:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:23:59 smithi033 ceph-mon[27845]: pgmap v16732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:23:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30777.170976 secs 2024-04-26T20:23:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:23:59 smithi096 ceph-mon[32753]: pgmap v16732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:00.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30780.097337 secs 2024-04-26T20:24:00.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30780.097337 secs 2024-04-26T20:24:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30780.097337 secs 2024-04-26T20:24:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:01 smithi194 ceph-mon[33017]: pgmap v16733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:01 smithi033 ceph-mon[27845]: pgmap v16733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:01 smithi096 ceph-mon[32753]: pgmap v16733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:03 smithi194 ceph-mon[33017]: pgmap v16734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:03 smithi033 ceph-mon[27845]: pgmap v16734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:03 smithi096 ceph-mon[32753]: pgmap v16734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30782.171032 secs 2024-04-26T20:24:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30782.171032 secs 2024-04-26T20:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30782.171032 secs 2024-04-26T20:24:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30785.097431 secs 2024-04-26T20:24:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:05 smithi194 ceph-mon[33017]: pgmap v16735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30785.097431 secs 2024-04-26T20:24:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:05 smithi033 ceph-mon[27845]: pgmap v16735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30785.097431 secs 2024-04-26T20:24:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:05 smithi096 ceph-mon[32753]: pgmap v16735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:07 smithi194 ceph-mon[33017]: pgmap v16736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:07 smithi033 ceph-mon[27845]: pgmap v16736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:07 smithi096 ceph-mon[32753]: pgmap v16736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30787.171123 secs 2024-04-26T20:24:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:09 smithi194 ceph-mon[33017]: pgmap v16737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30787.171123 secs 2024-04-26T20:24:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:09 smithi033 ceph-mon[27845]: pgmap v16737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30787.171123 secs 2024-04-26T20:24:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:09 smithi096 ceph-mon[32753]: pgmap v16737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:10.333+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:10.333+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:10.333+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:24:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30790.097549 secs 2024-04-26T20:24:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30790.097549 secs 2024-04-26T20:24:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30790.097549 secs 2024-04-26T20:24:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:11 smithi033 ceph-mon[27845]: pgmap v16738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:11 smithi096 ceph-mon[32753]: pgmap v16738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:11 smithi194 ceph-mon[33017]: pgmap v16738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:12 smithi033 ceph-mon[27845]: pgmap v16739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:12 smithi096 ceph-mon[32753]: pgmap v16739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:12 smithi194 ceph-mon[33017]: pgmap v16739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30792.171205 secs 2024-04-26T20:24:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30792.171205 secs 2024-04-26T20:24:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30792.171205 secs 2024-04-26T20:24:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30795.097718 secs 2024-04-26T20:24:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:14 smithi194 ceph-mon[33017]: pgmap v16740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30795.097718 secs 2024-04-26T20:24:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:14 smithi033 ceph-mon[27845]: pgmap v16740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30795.097718 secs 2024-04-26T20:24:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:14 smithi096 ceph-mon[32753]: pgmap v16740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:17 smithi194 ceph-mon[33017]: pgmap v16741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:17 smithi033 ceph-mon[27845]: pgmap v16741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:17 smithi096 ceph-mon[32753]: pgmap v16741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30797.171287 secs 2024-04-26T20:24:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:19 smithi194 ceph-mon[33017]: pgmap v16742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30797.171287 secs 2024-04-26T20:24:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:19 smithi033 ceph-mon[27845]: pgmap v16742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30797.171287 secs 2024-04-26T20:24:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:19 smithi096 ceph-mon[32753]: pgmap v16742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30800.097832 secs 2024-04-26T20:24:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30800.097832 secs 2024-04-26T20:24:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30800.097832 secs 2024-04-26T20:24:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:21 smithi194 ceph-mon[33017]: pgmap v16743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:21 smithi033 ceph-mon[27845]: pgmap v16743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:21 smithi096 ceph-mon[32753]: pgmap v16743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:23 smithi194 ceph-mon[33017]: pgmap v16744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:23 smithi033 ceph-mon[27845]: pgmap v16744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:23 smithi096 ceph-mon[32753]: pgmap v16744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30802.171370 secs 2024-04-26T20:24:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30802.171370 secs 2024-04-26T20:24:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30802.171370 secs 2024-04-26T20:24:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:25.348+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:25.348+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:25.348+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:24:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30805.097956 secs 2024-04-26T20:24:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:25 smithi194 ceph-mon[33017]: pgmap v16745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30805.097956 secs 2024-04-26T20:24:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:25 smithi033 ceph-mon[27845]: pgmap v16745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30805.097956 secs 2024-04-26T20:24:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:25 smithi096 ceph-mon[32753]: pgmap v16745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:27 smithi194 ceph-mon[33017]: pgmap v16746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:27 smithi033 ceph-mon[27845]: pgmap v16746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:27 smithi096 ceph-mon[32753]: pgmap v16746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30807.171436 secs 2024-04-26T20:24:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:29 smithi194 ceph-mon[33017]: pgmap v16747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30807.171436 secs 2024-04-26T20:24:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:29 smithi033 ceph-mon[27845]: pgmap v16747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30807.171436 secs 2024-04-26T20:24:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:29 smithi096 ceph-mon[32753]: pgmap v16747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30810.098075 secs 2024-04-26T20:24:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30810.098075 secs 2024-04-26T20:24:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30810.098075 secs 2024-04-26T20:24:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:31 smithi194 ceph-mon[33017]: pgmap v16748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:31 smithi033 ceph-mon[27845]: pgmap v16748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:31 smithi096 ceph-mon[32753]: pgmap v16748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:33 smithi194 ceph-mon[33017]: pgmap v16749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:33 smithi033 ceph-mon[27845]: pgmap v16749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:33 smithi096 ceph-mon[32753]: pgmap v16749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30812.171521 secs 2024-04-26T20:24:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30812.171521 secs 2024-04-26T20:24:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30812.171521 secs 2024-04-26T20:24:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30815.098193 secs 2024-04-26T20:24:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:35 smithi194 ceph-mon[33017]: pgmap v16750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30815.098193 secs 2024-04-26T20:24:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:35 smithi033 ceph-mon[27845]: pgmap v16750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30815.098193 secs 2024-04-26T20:24:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:35 smithi096 ceph-mon[32753]: pgmap v16750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:37 smithi194 ceph-mon[33017]: pgmap v16751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:37 smithi033 ceph-mon[27845]: pgmap v16751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:37 smithi096 ceph-mon[32753]: pgmap v16751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:24:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30817.171582 secs 2024-04-26T20:24:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:39 smithi194 ceph-mon[33017]: pgmap v16752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30817.171582 secs 2024-04-26T20:24:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:39 smithi033 ceph-mon[27845]: pgmap v16752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30817.171582 secs 2024-04-26T20:24:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:39 smithi096 ceph-mon[32753]: pgmap v16752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:40.364+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:40.364+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:40.364+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:24:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30820.098311 secs 2024-04-26T20:24:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:24:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:24:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:24:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30820.098311 secs 2024-04-26T20:24:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:24:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:24:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:24:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30820.098311 secs 2024-04-26T20:24:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:24:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:24:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:24:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:41 smithi194 ceph-mon[33017]: pgmap v16753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:24:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:41 smithi194 ceph-mon[33017]: pgmap v16754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:41 smithi033 ceph-mon[27845]: pgmap v16753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:24:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:41 smithi033 ceph-mon[27845]: pgmap v16754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:41 smithi096 ceph-mon[32753]: pgmap v16753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:24:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:41 smithi096 ceph-mon[32753]: pgmap v16754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:43 smithi033 ceph-mon[27845]: pgmap v16755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:43 smithi096 ceph-mon[32753]: pgmap v16755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:43 smithi194 ceph-mon[33017]: pgmap v16755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30822.171664 secs 2024-04-26T20:24:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30822.171664 secs 2024-04-26T20:24:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30822.171664 secs 2024-04-26T20:24:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30825.098431 secs 2024-04-26T20:24:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:45 smithi033 ceph-mon[27845]: pgmap v16756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T20:24:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30825.098431 secs 2024-04-26T20:24:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:45 smithi096 ceph-mon[32753]: pgmap v16756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T20:24:46.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30825.098431 secs 2024-04-26T20:24:46.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:45 smithi194 ceph-mon[33017]: pgmap v16756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T20:24:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:46 smithi033 ceph-mon[27845]: pgmap v16757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:46 smithi096 ceph-mon[32753]: pgmap v16757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:46 smithi194 ceph-mon[33017]: pgmap v16757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30827.171751 secs 2024-04-26T20:24:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:49 smithi194 ceph-mon[33017]: pgmap v16758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30827.171751 secs 2024-04-26T20:24:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:49 smithi033 ceph-mon[27845]: pgmap v16758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30827.171751 secs 2024-04-26T20:24:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:49 smithi096 ceph-mon[32753]: pgmap v16758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:24:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30830.098513 secs 2024-04-26T20:24:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30830.098513 secs 2024-04-26T20:24:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30830.098513 secs 2024-04-26T20:24:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:51 smithi194 ceph-mon[33017]: pgmap v16759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:24:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:51 smithi033 ceph-mon[27845]: pgmap v16759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:24:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:51 smithi096 ceph-mon[32753]: pgmap v16759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:24:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:53 smithi194 ceph-mon[33017]: pgmap v16760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:53 smithi033 ceph-mon[27845]: pgmap v16760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:53 smithi096 ceph-mon[32753]: pgmap v16760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30832.171832 secs 2024-04-26T20:24:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30832.171832 secs 2024-04-26T20:24:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30832.171832 secs 2024-04-26T20:24:55.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:55.370+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:55.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:55.370+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:24:55.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:24:55.370+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:24:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30835.098666 secs 2024-04-26T20:24:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:55 smithi194 ceph-mon[33017]: pgmap v16761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30835.098666 secs 2024-04-26T20:24:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:55 smithi033 ceph-mon[27845]: pgmap v16761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30835.098666 secs 2024-04-26T20:24:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:55 smithi096 ceph-mon[32753]: pgmap v16761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:57 smithi194 ceph-mon[33017]: pgmap v16762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:57 smithi033 ceph-mon[27845]: pgmap v16762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:57 smithi096 ceph-mon[32753]: pgmap v16762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30837.171917 secs 2024-04-26T20:24:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:24:59 smithi194 ceph-mon[33017]: pgmap v16763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30837.171917 secs 2024-04-26T20:24:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:24:59 smithi033 ceph-mon[27845]: pgmap v16763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:24:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30837.171917 secs 2024-04-26T20:24:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:24:59 smithi096 ceph-mon[32753]: pgmap v16763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:00.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30840.098800 secs 2024-04-26T20:25:00.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30840.098800 secs 2024-04-26T20:25:00.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30840.098800 secs 2024-04-26T20:25:01.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:01 smithi194 ceph-mon[33017]: pgmap v16764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:01 smithi033 ceph-mon[27845]: pgmap v16764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:01 smithi096 ceph-mon[32753]: pgmap v16764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:03 smithi194 ceph-mon[33017]: pgmap v16765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:03 smithi033 ceph-mon[27845]: pgmap v16765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:03 smithi096 ceph-mon[32753]: pgmap v16765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30842.171993 secs 2024-04-26T20:25:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30842.171993 secs 2024-04-26T20:25:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30842.171993 secs 2024-04-26T20:25:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30845.098947 secs 2024-04-26T20:25:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:05 smithi194 ceph-mon[33017]: pgmap v16766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30845.098947 secs 2024-04-26T20:25:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:05 smithi033 ceph-mon[27845]: pgmap v16766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30845.098947 secs 2024-04-26T20:25:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:05 smithi096 ceph-mon[32753]: pgmap v16766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:07 smithi194 ceph-mon[33017]: pgmap v16767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:07 smithi033 ceph-mon[27845]: pgmap v16767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:07 smithi096 ceph-mon[32753]: pgmap v16767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30847.172056 secs 2024-04-26T20:25:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:09 smithi194 ceph-mon[33017]: pgmap v16768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30847.172056 secs 2024-04-26T20:25:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:09 smithi033 ceph-mon[27845]: pgmap v16768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30847.172056 secs 2024-04-26T20:25:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:09 smithi096 ceph-mon[32753]: pgmap v16768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:10.383+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:10.383+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:25:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:10.383+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30850.099081 secs 2024-04-26T20:25:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30850.099081 secs 2024-04-26T20:25:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30850.099081 secs 2024-04-26T20:25:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:11 smithi194 ceph-mon[33017]: pgmap v16769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:11 smithi033 ceph-mon[27845]: pgmap v16769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:11 smithi096 ceph-mon[32753]: pgmap v16769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:13 smithi194 ceph-mon[33017]: pgmap v16770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:13 smithi033 ceph-mon[27845]: pgmap v16770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:13 smithi096 ceph-mon[32753]: pgmap v16770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30852.172140 secs 2024-04-26T20:25:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30852.172140 secs 2024-04-26T20:25:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30852.172140 secs 2024-04-26T20:25:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30855.099214 secs 2024-04-26T20:25:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:15 smithi194 ceph-mon[33017]: pgmap v16771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30855.099214 secs 2024-04-26T20:25:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:15 smithi033 ceph-mon[27845]: pgmap v16771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30855.099214 secs 2024-04-26T20:25:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:15 smithi096 ceph-mon[32753]: pgmap v16771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:17 smithi194 ceph-mon[33017]: pgmap v16772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:17 smithi033 ceph-mon[27845]: pgmap v16772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:17 smithi096 ceph-mon[32753]: pgmap v16772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30857.172203 secs 2024-04-26T20:25:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:19 smithi194 ceph-mon[33017]: pgmap v16773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30857.172203 secs 2024-04-26T20:25:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:19 smithi033 ceph-mon[27845]: pgmap v16773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30857.172203 secs 2024-04-26T20:25:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:19 smithi096 ceph-mon[32753]: pgmap v16773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30860.099326 secs 2024-04-26T20:25:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30860.099326 secs 2024-04-26T20:25:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30860.099326 secs 2024-04-26T20:25:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:21 smithi194 ceph-mon[33017]: pgmap v16774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:21 smithi033 ceph-mon[27845]: pgmap v16774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:21 smithi096 ceph-mon[32753]: pgmap v16774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:23 smithi194 ceph-mon[33017]: pgmap v16775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:23 smithi033 ceph-mon[27845]: pgmap v16775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:23 smithi096 ceph-mon[32753]: pgmap v16775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30862.172265 secs 2024-04-26T20:25:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30865.099449 secs 2024-04-26T20:25:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:24 smithi194 ceph-mon[33017]: pgmap v16776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30862.172265 secs 2024-04-26T20:25:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30865.099449 secs 2024-04-26T20:25:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:24 smithi033 ceph-mon[27845]: pgmap v16776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30862.172265 secs 2024-04-26T20:25:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30865.099449 secs 2024-04-26T20:25:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:24 smithi096 ceph-mon[32753]: pgmap v16776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:25.388+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:25.388+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:25.388+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:25:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:27 smithi194 ceph-mon[33017]: pgmap v16777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:27 smithi033 ceph-mon[27845]: pgmap v16777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:27 smithi096 ceph-mon[32753]: pgmap v16777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30867.172325 secs 2024-04-26T20:25:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:29 smithi194 ceph-mon[33017]: pgmap v16778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30867.172325 secs 2024-04-26T20:25:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:29 smithi033 ceph-mon[27845]: pgmap v16778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30867.172325 secs 2024-04-26T20:25:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:29 smithi096 ceph-mon[32753]: pgmap v16778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30870.099563 secs 2024-04-26T20:25:30.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30870.099563 secs 2024-04-26T20:25:30.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30870.099563 secs 2024-04-26T20:25:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:31 smithi194 ceph-mon[33017]: pgmap v16779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:31 smithi033 ceph-mon[27845]: pgmap v16779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:31 smithi096 ceph-mon[32753]: pgmap v16779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:33 smithi194 ceph-mon[33017]: pgmap v16780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:33 smithi033 ceph-mon[27845]: pgmap v16780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:33 smithi096 ceph-mon[32753]: pgmap v16780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30872.172383 secs 2024-04-26T20:25:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30872.172383 secs 2024-04-26T20:25:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30872.172383 secs 2024-04-26T20:25:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30875.099702 secs 2024-04-26T20:25:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:35 smithi194 ceph-mon[33017]: pgmap v16781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30875.099702 secs 2024-04-26T20:25:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:35 smithi033 ceph-mon[27845]: pgmap v16781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30875.099702 secs 2024-04-26T20:25:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:35 smithi096 ceph-mon[32753]: pgmap v16781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:37 smithi194 ceph-mon[33017]: pgmap v16782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:37 smithi033 ceph-mon[27845]: pgmap v16782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:37 smithi096 ceph-mon[32753]: pgmap v16782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30877.172443 secs 2024-04-26T20:25:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:39 smithi194 ceph-mon[33017]: pgmap v16783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30877.172443 secs 2024-04-26T20:25:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:39 smithi033 ceph-mon[27845]: pgmap v16783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30877.172443 secs 2024-04-26T20:25:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:39 smithi096 ceph-mon[32753]: pgmap v16783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:40.404+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:40.404+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:40.404+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:25:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30880.099841 secs 2024-04-26T20:25:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30880.099841 secs 2024-04-26T20:25:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30880.099841 secs 2024-04-26T20:25:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:41 smithi194 ceph-mon[33017]: pgmap v16784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:41 smithi033 ceph-mon[27845]: pgmap v16784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:41 smithi096 ceph-mon[32753]: pgmap v16784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:25:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:25:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:25:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:43 smithi194 ceph-mon[33017]: pgmap v16785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:43 smithi194 ceph-mon[33017]: pgmap v16786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:43 smithi033 ceph-mon[27845]: pgmap v16785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:43 smithi033 ceph-mon[27845]: pgmap v16786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:43 smithi096 ceph-mon[32753]: pgmap v16785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:43 smithi096 ceph-mon[32753]: pgmap v16786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30882.172499 secs 2024-04-26T20:25:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30882.172499 secs 2024-04-26T20:25:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30882.172499 secs 2024-04-26T20:25:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30885.099946 secs 2024-04-26T20:25:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:45 smithi194 ceph-mon[33017]: pgmap v16787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30885.099946 secs 2024-04-26T20:25:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:45 smithi033 ceph-mon[27845]: pgmap v16787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30885.099946 secs 2024-04-26T20:25:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:45 smithi096 ceph-mon[32753]: pgmap v16787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:47 smithi194 ceph-mon[33017]: pgmap v16788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:47.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:47 smithi033 ceph-mon[27845]: pgmap v16788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:47 smithi096 ceph-mon[32753]: pgmap v16788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:49 smithi194 ceph-mon[33017]: pgmap v16789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30887.172558 secs 2024-04-26T20:25:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:49 smithi033 ceph-mon[27845]: pgmap v16789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30887.172558 secs 2024-04-26T20:25:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:49 smithi096 ceph-mon[32753]: pgmap v16789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30887.172558 secs 2024-04-26T20:25:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30890.100099 secs 2024-04-26T20:25:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30890.100099 secs 2024-04-26T20:25:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30890.100099 secs 2024-04-26T20:25:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:51 smithi194 ceph-mon[33017]: pgmap v16790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:51 smithi033 ceph-mon[27845]: pgmap v16790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:51 smithi096 ceph-mon[32753]: pgmap v16790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:25:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:53 smithi194 ceph-mon[33017]: pgmap v16791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:25:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:53 smithi033 ceph-mon[27845]: pgmap v16791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:25:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:53 smithi096 ceph-mon[32753]: pgmap v16791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:25:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30892.172614 secs 2024-04-26T20:25:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30892.172614 secs 2024-04-26T20:25:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30892.172614 secs 2024-04-26T20:25:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:55.419+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:55.420+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:25:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:25:55.420+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:25:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30895.100220 secs 2024-04-26T20:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:55 smithi194 ceph-mon[33017]: pgmap v16792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30895.100220 secs 2024-04-26T20:25:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:55 smithi033 ceph-mon[27845]: pgmap v16792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30895.100220 secs 2024-04-26T20:25:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:55 smithi096 ceph-mon[32753]: pgmap v16792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:57 smithi194 ceph-mon[33017]: pgmap v16793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:57 smithi033 ceph-mon[27845]: pgmap v16793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:57 smithi096 ceph-mon[32753]: pgmap v16793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:59 smithi033 ceph-mon[27845]: pgmap v16794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:25:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30897.172675 secs 2024-04-26T20:25:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:59 smithi096 ceph-mon[32753]: pgmap v16794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:25:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30897.172675 secs 2024-04-26T20:25:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:59 smithi194 ceph-mon[33017]: pgmap v16794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:25:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:25:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30897.172675 secs 2024-04-26T20:26:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30900.100334 secs 2024-04-26T20:26:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30900.100334 secs 2024-04-26T20:26:00.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30900.100334 secs 2024-04-26T20:26:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:01 smithi033 ceph-mon[27845]: pgmap v16795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:01 smithi096 ceph-mon[32753]: pgmap v16795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:01 smithi194 ceph-mon[33017]: pgmap v16795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:03 smithi033 ceph-mon[27845]: pgmap v16796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:03 smithi096 ceph-mon[32753]: pgmap v16796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:03 smithi194 ceph-mon[33017]: pgmap v16796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30902.172758 secs 2024-04-26T20:26:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30902.172758 secs 2024-04-26T20:26:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30902.172758 secs 2024-04-26T20:26:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30905.100451 secs 2024-04-26T20:26:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:05 smithi033 ceph-mon[27845]: pgmap v16797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30905.100451 secs 2024-04-26T20:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:05 smithi096 ceph-mon[32753]: pgmap v16797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30905.100451 secs 2024-04-26T20:26:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:05 smithi194 ceph-mon[33017]: pgmap v16797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:07 smithi033 ceph-mon[27845]: pgmap v16798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:07 smithi096 ceph-mon[32753]: pgmap v16798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:07 smithi194 ceph-mon[33017]: pgmap v16798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:09 smithi033 ceph-mon[27845]: pgmap v16799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30907.172842 secs 2024-04-26T20:26:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:09 smithi096 ceph-mon[32753]: pgmap v16799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30907.172842 secs 2024-04-26T20:26:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:09 smithi194 ceph-mon[33017]: pgmap v16799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30907.172842 secs 2024-04-26T20:26:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:10.426+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:10.426+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:26:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:10.426+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30910.100574 secs 2024-04-26T20:26:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30910.100574 secs 2024-04-26T20:26:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30910.100574 secs 2024-04-26T20:26:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:11 smithi033 ceph-mon[27845]: pgmap v16800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:11 smithi096 ceph-mon[32753]: pgmap v16800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:11 smithi194 ceph-mon[33017]: pgmap v16800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:13 smithi033 ceph-mon[27845]: pgmap v16801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:13 smithi096 ceph-mon[32753]: pgmap v16801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:13 smithi194 ceph-mon[33017]: pgmap v16801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30912.172924 secs 2024-04-26T20:26:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30912.172924 secs 2024-04-26T20:26:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30912.172924 secs 2024-04-26T20:26:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30915.100695 secs 2024-04-26T20:26:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:15 smithi033 ceph-mon[27845]: pgmap v16802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30915.100695 secs 2024-04-26T20:26:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:15 smithi096 ceph-mon[32753]: pgmap v16802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30915.100695 secs 2024-04-26T20:26:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:15 smithi194 ceph-mon[33017]: pgmap v16802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:17 smithi194 ceph-mon[33017]: pgmap v16803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:17 smithi033 ceph-mon[27845]: pgmap v16803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:17 smithi096 ceph-mon[32753]: pgmap v16803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:19 smithi194 ceph-mon[33017]: pgmap v16804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30917.173021 secs 2024-04-26T20:26:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:19 smithi033 ceph-mon[27845]: pgmap v16804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30917.173021 secs 2024-04-26T20:26:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:19 smithi096 ceph-mon[32753]: pgmap v16804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30917.173021 secs 2024-04-26T20:26:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30920.100756 secs 2024-04-26T20:26:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30920.100756 secs 2024-04-26T20:26:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30920.100756 secs 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0062040 comp rx=0 tx=0).stop 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.802+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.803+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:26:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:21.804+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef007f040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:26:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:21 smithi194 ceph-mon[33017]: pgmap v16805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:21 smithi033 ceph-mon[27845]: pgmap v16805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:21 smithi096 ceph-mon[32753]: pgmap v16805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:23 smithi194 ceph-mon[33017]: pgmap v16806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:23 smithi033 ceph-mon[27845]: pgmap v16806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:23 smithi096 ceph-mon[32753]: pgmap v16806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30922.173108 secs 2024-04-26T20:26:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30922.173108 secs 2024-04-26T20:26:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30922.173108 secs 2024-04-26T20:26:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:25.430+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:25.430+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:25.430+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:26:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30925.100909 secs 2024-04-26T20:26:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:25 smithi194 ceph-mon[33017]: pgmap v16807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30925.100909 secs 2024-04-26T20:26:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:25 smithi033 ceph-mon[27845]: pgmap v16807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30925.100909 secs 2024-04-26T20:26:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:25 smithi096 ceph-mon[32753]: pgmap v16807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:27.056+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:26:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:27.056+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T20:26:27.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:27.057+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 243 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:26:27.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:27.057+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 243 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:26:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:27 smithi194 ceph-mon[33017]: pgmap v16808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:27 smithi033 ceph-mon[27845]: pgmap v16808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:27 smithi096 ceph-mon[32753]: pgmap v16808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:29 smithi194 ceph-mon[33017]: pgmap v16809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30927.173193 secs 2024-04-26T20:26:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:29 smithi033 ceph-mon[27845]: pgmap v16809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30927.173193 secs 2024-04-26T20:26:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:29 smithi096 ceph-mon[32753]: pgmap v16809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30927.173193 secs 2024-04-26T20:26:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30930.101019 secs 2024-04-26T20:26:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30930.101019 secs 2024-04-26T20:26:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30930.101019 secs 2024-04-26T20:26:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:31 smithi194 ceph-mon[33017]: pgmap v16810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:31 smithi033 ceph-mon[27845]: pgmap v16810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:31 smithi096 ceph-mon[32753]: pgmap v16810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:33 smithi194 ceph-mon[33017]: pgmap v16811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:33 smithi033 ceph-mon[27845]: pgmap v16811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:33 smithi096 ceph-mon[32753]: pgmap v16811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30932.173276 secs 2024-04-26T20:26:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30932.173276 secs 2024-04-26T20:26:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30932.173276 secs 2024-04-26T20:26:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30935.101169 secs 2024-04-26T20:26:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:35 smithi194 ceph-mon[33017]: pgmap v16812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30935.101169 secs 2024-04-26T20:26:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:35 smithi033 ceph-mon[27845]: pgmap v16812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30935.101169 secs 2024-04-26T20:26:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:35 smithi096 ceph-mon[32753]: pgmap v16812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:37 smithi194 ceph-mon[33017]: pgmap v16813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:37 smithi033 ceph-mon[27845]: pgmap v16813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:37 smithi096 ceph-mon[32753]: pgmap v16813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:39 smithi194 ceph-mon[33017]: pgmap v16814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30937.173340 secs 2024-04-26T20:26:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:39 smithi033 ceph-mon[27845]: pgmap v16814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30937.173340 secs 2024-04-26T20:26:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:39 smithi096 ceph-mon[32753]: pgmap v16814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30937.173340 secs 2024-04-26T20:26:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:40.446+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:40.446+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:26:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:40.446+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30940.101264 secs 2024-04-26T20:26:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30940.101264 secs 2024-04-26T20:26:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30940.101264 secs 2024-04-26T20:26:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:41 smithi194 ceph-mon[33017]: pgmap v16815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:41 smithi033 ceph-mon[27845]: pgmap v16815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:41 smithi096 ceph-mon[32753]: pgmap v16815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:43 smithi194 ceph-mon[33017]: pgmap v16816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:43 smithi033 ceph-mon[27845]: pgmap v16816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:43 smithi096 ceph-mon[32753]: pgmap v16816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30942.173420 secs 2024-04-26T20:26:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:44 smithi194 ceph-mon[33017]: pgmap v16817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:44 smithi194 ceph-mon[33017]: pgmap v16818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30942.173420 secs 2024-04-26T20:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:44 smithi033 ceph-mon[27845]: pgmap v16817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:44 smithi033 ceph-mon[27845]: pgmap v16818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:26:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30942.173420 secs 2024-04-26T20:26:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:44 smithi096 ceph-mon[32753]: pgmap v16817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:44 smithi096 ceph-mon[32753]: pgmap v16818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:26:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30945.101390 secs 2024-04-26T20:26:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30945.101390 secs 2024-04-26T20:26:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30945.101390 secs 2024-04-26T20:26:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:46 smithi194 ceph-mon[33017]: pgmap v16819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:46 smithi033 ceph-mon[27845]: pgmap v16819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:46 smithi096 ceph-mon[32753]: pgmap v16819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:48 smithi194 ceph-mon[33017]: pgmap v16820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:48 smithi033 ceph-mon[27845]: pgmap v16820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:48 smithi096 ceph-mon[32753]: pgmap v16820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30947.173506 secs 2024-04-26T20:26:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30947.173506 secs 2024-04-26T20:26:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30947.173506 secs 2024-04-26T20:26:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30950.101486 secs 2024-04-26T20:26:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:50 smithi194 ceph-mon[33017]: pgmap v16821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30950.101486 secs 2024-04-26T20:26:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:50 smithi033 ceph-mon[27845]: pgmap v16821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30950.101486 secs 2024-04-26T20:26:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:50 smithi096 ceph-mon[32753]: pgmap v16821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:52 smithi194 ceph-mon[33017]: pgmap v16822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:52 smithi033 ceph-mon[27845]: pgmap v16822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:52 smithi096 ceph-mon[32753]: pgmap v16822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:26:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30952.173589 secs 2024-04-26T20:26:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:54 smithi033 ceph-mon[27845]: pgmap v16823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:26:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30952.173589 secs 2024-04-26T20:26:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:54 smithi096 ceph-mon[32753]: pgmap v16823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:26:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30952.173589 secs 2024-04-26T20:26:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:54 smithi194 ceph-mon[33017]: pgmap v16823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:26:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:55.449+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:55.449+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:26:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:26:55.449+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:26:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30955.101604 secs 2024-04-26T20:26:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30955.101604 secs 2024-04-26T20:26:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30955.101604 secs 2024-04-26T20:26:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:56 smithi033 ceph-mon[27845]: pgmap v16824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:56 smithi096 ceph-mon[32753]: pgmap v16824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:56 smithi194 ceph-mon[33017]: pgmap v16824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:58 smithi033 ceph-mon[27845]: pgmap v16825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:58 smithi096 ceph-mon[32753]: pgmap v16825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:58 smithi194 ceph-mon[33017]: pgmap v16825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:26:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:26:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30957.173672 secs 2024-04-26T20:26:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:26:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30957.173672 secs 2024-04-26T20:27:00.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:26:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30957.173672 secs 2024-04-26T20:27:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30960.101700 secs 2024-04-26T20:27:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:00 smithi033 ceph-mon[27845]: pgmap v16826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30960.101700 secs 2024-04-26T20:27:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:00 smithi096 ceph-mon[32753]: pgmap v16826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30960.101700 secs 2024-04-26T20:27:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:00 smithi194 ceph-mon[33017]: pgmap v16826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:01 smithi194 ceph-mon[33017]: pgmap v16827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:01 smithi033 ceph-mon[27845]: pgmap v16827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:01 smithi096 ceph-mon[32753]: pgmap v16827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30962.173752 secs 2024-04-26T20:27:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:04 smithi033 ceph-mon[27845]: pgmap v16828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30962.173752 secs 2024-04-26T20:27:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:04 smithi096 ceph-mon[32753]: pgmap v16828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30962.173752 secs 2024-04-26T20:27:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:04 smithi194 ceph-mon[33017]: pgmap v16828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30965.101815 secs 2024-04-26T20:27:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30965.101815 secs 2024-04-26T20:27:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30965.101815 secs 2024-04-26T20:27:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:06 smithi033 ceph-mon[27845]: pgmap v16829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:06 smithi096 ceph-mon[32753]: pgmap v16829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:06 smithi194 ceph-mon[33017]: pgmap v16829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:08 smithi033 ceph-mon[27845]: pgmap v16830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:08 smithi096 ceph-mon[32753]: pgmap v16830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:08 smithi194 ceph-mon[33017]: pgmap v16830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30967.173839 secs 2024-04-26T20:27:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30970.101927 secs 2024-04-26T20:27:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30967.173839 secs 2024-04-26T20:27:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30970.101927 secs 2024-04-26T20:27:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30967.173839 secs 2024-04-26T20:27:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30970.101927 secs 2024-04-26T20:27:10.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:10.462+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:10.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:10.463+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:10.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:10.463+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:27:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:10 smithi033 ceph-mon[27845]: pgmap v16831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:10 smithi096 ceph-mon[32753]: pgmap v16831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:10 smithi194 ceph-mon[33017]: pgmap v16831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:11 smithi033 ceph-mon[27845]: pgmap v16832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:11 smithi096 ceph-mon[32753]: pgmap v16832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:11 smithi194 ceph-mon[33017]: pgmap v16832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30972.173921 secs 2024-04-26T20:27:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:14 smithi033 ceph-mon[27845]: pgmap v16833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30972.173921 secs 2024-04-26T20:27:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:14 smithi096 ceph-mon[32753]: pgmap v16833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30972.173921 secs 2024-04-26T20:27:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:14 smithi194 ceph-mon[33017]: pgmap v16833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30975.102051 secs 2024-04-26T20:27:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30975.102051 secs 2024-04-26T20:27:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30975.102051 secs 2024-04-26T20:27:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:16 smithi033 ceph-mon[27845]: pgmap v16834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:16 smithi096 ceph-mon[32753]: pgmap v16834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:17.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:16 smithi194 ceph-mon[33017]: pgmap v16834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:18 smithi033 ceph-mon[27845]: pgmap v16835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:18 smithi096 ceph-mon[32753]: pgmap v16835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:18 smithi194 ceph-mon[33017]: pgmap v16835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30977.173981 secs 2024-04-26T20:27:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30980.102178 secs 2024-04-26T20:27:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30977.173981 secs 2024-04-26T20:27:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30980.102178 secs 2024-04-26T20:27:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30977.173981 secs 2024-04-26T20:27:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30980.102178 secs 2024-04-26T20:27:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:20 smithi033 ceph-mon[27845]: pgmap v16836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:20 smithi096 ceph-mon[32753]: pgmap v16836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:20 smithi194 ceph-mon[33017]: pgmap v16836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:21 smithi033 ceph-mon[27845]: pgmap v16837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:21 smithi096 ceph-mon[32753]: pgmap v16837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:21 smithi194 ceph-mon[33017]: pgmap v16837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30982.174062 secs 2024-04-26T20:27:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:24 smithi033 ceph-mon[27845]: pgmap v16838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30982.174062 secs 2024-04-26T20:27:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:24 smithi096 ceph-mon[32753]: pgmap v16838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30982.174062 secs 2024-04-26T20:27:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:24 smithi194 ceph-mon[33017]: pgmap v16838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:25.469+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:25.469+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:25.469+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:27:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30985.102289 secs 2024-04-26T20:27:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30985.102289 secs 2024-04-26T20:27:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30985.102289 secs 2024-04-26T20:27:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:26 smithi033 ceph-mon[27845]: pgmap v16839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:26 smithi096 ceph-mon[32753]: pgmap v16839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:26 smithi194 ceph-mon[33017]: pgmap v16839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:28 smithi033 ceph-mon[27845]: pgmap v16840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:28 smithi096 ceph-mon[32753]: pgmap v16840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:28 smithi194 ceph-mon[33017]: pgmap v16840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30987.174152 secs 2024-04-26T20:27:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30990.102370 secs 2024-04-26T20:27:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30987.174152 secs 2024-04-26T20:27:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30990.102370 secs 2024-04-26T20:27:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30987.174152 secs 2024-04-26T20:27:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30990.102370 secs 2024-04-26T20:27:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:30 smithi033 ceph-mon[27845]: pgmap v16841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:30 smithi096 ceph-mon[32753]: pgmap v16841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:30 smithi194 ceph-mon[33017]: pgmap v16841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:31 smithi033 ceph-mon[27845]: pgmap v16842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:31 smithi096 ceph-mon[32753]: pgmap v16842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:31 smithi194 ceph-mon[33017]: pgmap v16842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30992.174210 secs 2024-04-26T20:27:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:34 smithi033 ceph-mon[27845]: pgmap v16843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30992.174210 secs 2024-04-26T20:27:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:34 smithi096 ceph-mon[32753]: pgmap v16843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30992.174210 secs 2024-04-26T20:27:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:34 smithi194 ceph-mon[33017]: pgmap v16843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 30995.102501 secs 2024-04-26T20:27:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 30995.102501 secs 2024-04-26T20:27:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 30995.102501 secs 2024-04-26T20:27:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:36 smithi033 ceph-mon[27845]: pgmap v16844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:36 smithi096 ceph-mon[32753]: pgmap v16844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:36 smithi194 ceph-mon[33017]: pgmap v16844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:38 smithi033 ceph-mon[27845]: pgmap v16845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:38 smithi096 ceph-mon[32753]: pgmap v16845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:38 smithi194 ceph-mon[33017]: pgmap v16845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 30997.174295 secs 2024-04-26T20:27:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31000.102648 secs 2024-04-26T20:27:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:39 smithi033 ceph-mon[27845]: pgmap v16846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 30997.174295 secs 2024-04-26T20:27:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31000.102648 secs 2024-04-26T20:27:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:39 smithi096 ceph-mon[32753]: pgmap v16846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 30997.174295 secs 2024-04-26T20:27:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31000.102648 secs 2024-04-26T20:27:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:39 smithi194 ceph-mon[33017]: pgmap v16846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:40.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:40.473+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:40.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:40.473+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:40.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:40.473+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:27:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:42 smithi033 ceph-mon[27845]: pgmap v16847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:42 smithi096 ceph-mon[32753]: pgmap v16847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:42 smithi194 ceph-mon[33017]: pgmap v16847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31002.174353 secs 2024-04-26T20:27:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:44 smithi033 ceph-mon[27845]: pgmap v16848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31005.102790 secs 2024-04-26T20:27:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31002.174353 secs 2024-04-26T20:27:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:44 smithi096 ceph-mon[32753]: pgmap v16848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31005.102790 secs 2024-04-26T20:27:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31002.174353 secs 2024-04-26T20:27:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:44 smithi194 ceph-mon[33017]: pgmap v16848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:27:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31005.102790 secs 2024-04-26T20:27:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:27:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:27:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:27:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:46 smithi033 ceph-mon[27845]: pgmap v16849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:27:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:46 smithi033 ceph-mon[27845]: pgmap v16850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:46 smithi096 ceph-mon[32753]: pgmap v16849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:27:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:46 smithi096 ceph-mon[32753]: pgmap v16850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:46 smithi194 ceph-mon[33017]: pgmap v16849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:27:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:46 smithi194 ceph-mon[33017]: pgmap v16850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:47 smithi033 ceph-mon[27845]: pgmap v16851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:47 smithi096 ceph-mon[32753]: pgmap v16851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:47 smithi194 ceph-mon[33017]: pgmap v16851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31007.174413 secs 2024-04-26T20:27:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31007.174413 secs 2024-04-26T20:27:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31007.174413 secs 2024-04-26T20:27:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31010.102879 secs 2024-04-26T20:27:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:49 smithi033 ceph-mon[27845]: pgmap v16852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:27:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31010.102879 secs 2024-04-26T20:27:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:49 smithi096 ceph-mon[32753]: pgmap v16852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:27:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31010.102879 secs 2024-04-26T20:27:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:49 smithi194 ceph-mon[33017]: pgmap v16852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:27:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:52 smithi194 ceph-mon[33017]: pgmap v16853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:52 smithi033 ceph-mon[27845]: pgmap v16853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:52 smithi096 ceph-mon[32753]: pgmap v16853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31012.174481 secs 2024-04-26T20:27:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:54 smithi194 ceph-mon[33017]: pgmap v16854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31012.174481 secs 2024-04-26T20:27:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:54 smithi033 ceph-mon[27845]: pgmap v16854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31012.174481 secs 2024-04-26T20:27:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:54 smithi096 ceph-mon[32753]: pgmap v16854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:27:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:55.481+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:55.481+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:27:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:27:55.481+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:27:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31015.103030 secs 2024-04-26T20:27:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31015.103030 secs 2024-04-26T20:27:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31015.103030 secs 2024-04-26T20:27:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:56 smithi033 ceph-mon[27845]: pgmap v16855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:27:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:56 smithi096 ceph-mon[32753]: pgmap v16855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:27:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:56 smithi194 ceph-mon[33017]: pgmap v16855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:27:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:58 smithi033 ceph-mon[27845]: pgmap v16856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:27:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:58 smithi096 ceph-mon[32753]: pgmap v16856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:27:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:58 smithi194 ceph-mon[33017]: pgmap v16856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:27:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:27:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31017.174548 secs 2024-04-26T20:27:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:27:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31017.174548 secs 2024-04-26T20:27:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:27:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31017.174548 secs 2024-04-26T20:28:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31020.103165 secs 2024-04-26T20:28:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:00 smithi033 ceph-mon[27845]: pgmap v16857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:28:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31020.103165 secs 2024-04-26T20:28:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:00 smithi096 ceph-mon[32753]: pgmap v16857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:28:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31020.103165 secs 2024-04-26T20:28:00.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:00 smithi194 ceph-mon[33017]: pgmap v16857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:28:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:02 smithi033 ceph-mon[27845]: pgmap v16858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:28:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:02 smithi096 ceph-mon[32753]: pgmap v16858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:28:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:02 smithi194 ceph-mon[33017]: pgmap v16858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:28:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31022.174631 secs 2024-04-26T20:28:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:04 smithi033 ceph-mon[27845]: pgmap v16859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31022.174631 secs 2024-04-26T20:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:04 smithi096 ceph-mon[32753]: pgmap v16859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:28:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31022.174631 secs 2024-04-26T20:28:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:04 smithi194 ceph-mon[33017]: pgmap v16859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:28:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31025.103290 secs 2024-04-26T20:28:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31025.103290 secs 2024-04-26T20:28:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31025.103290 secs 2024-04-26T20:28:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:06 smithi033 ceph-mon[27845]: pgmap v16860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:28:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:06 smithi096 ceph-mon[32753]: pgmap v16860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:28:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:06 smithi194 ceph-mon[33017]: pgmap v16860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:28:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:08 smithi033 ceph-mon[27845]: pgmap v16861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:08 smithi096 ceph-mon[32753]: pgmap v16861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:08 smithi194 ceph-mon[33017]: pgmap v16861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31027.174712 secs 2024-04-26T20:28:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31027.174712 secs 2024-04-26T20:28:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31027.174712 secs 2024-04-26T20:28:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:10.485+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:10.485+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:28:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:10.485+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31030.103374 secs 2024-04-26T20:28:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:10 smithi033 ceph-mon[27845]: pgmap v16862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31030.103374 secs 2024-04-26T20:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:10 smithi096 ceph-mon[32753]: pgmap v16862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31030.103374 secs 2024-04-26T20:28:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:10 smithi194 ceph-mon[33017]: pgmap v16862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:12 smithi096 ceph-mon[32753]: pgmap v16863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:12 smithi194 ceph-mon[33017]: pgmap v16863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:12 smithi033 ceph-mon[27845]: pgmap v16863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31032.174769 secs 2024-04-26T20:28:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:14 smithi194 ceph-mon[33017]: pgmap v16864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31032.174769 secs 2024-04-26T20:28:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:14 smithi033 ceph-mon[27845]: pgmap v16864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31032.174769 secs 2024-04-26T20:28:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:14 smithi096 ceph-mon[32753]: pgmap v16864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31035.103490 secs 2024-04-26T20:28:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31035.103490 secs 2024-04-26T20:28:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31035.103490 secs 2024-04-26T20:28:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:16 smithi194 ceph-mon[33017]: pgmap v16865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:16 smithi033 ceph-mon[27845]: pgmap v16865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:16 smithi096 ceph-mon[32753]: pgmap v16865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:18 smithi194 ceph-mon[33017]: pgmap v16866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:18 smithi033 ceph-mon[27845]: pgmap v16866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:18 smithi096 ceph-mon[32753]: pgmap v16866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31037.174854 secs 2024-04-26T20:28:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31037.174854 secs 2024-04-26T20:28:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31037.174854 secs 2024-04-26T20:28:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31040.103588 secs 2024-04-26T20:28:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:20 smithi194 ceph-mon[33017]: pgmap v16867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31040.103588 secs 2024-04-26T20:28:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:20 smithi033 ceph-mon[27845]: pgmap v16867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31040.103588 secs 2024-04-26T20:28:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:20 smithi096 ceph-mon[32753]: pgmap v16867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:22 smithi194 ceph-mon[33017]: pgmap v16868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:22 smithi033 ceph-mon[27845]: pgmap v16868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:22 smithi096 ceph-mon[32753]: pgmap v16868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31042.174938 secs 2024-04-26T20:28:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:24 smithi194 ceph-mon[33017]: pgmap v16869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31042.174938 secs 2024-04-26T20:28:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:24 smithi033 ceph-mon[27845]: pgmap v16869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31042.174938 secs 2024-04-26T20:28:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:24 smithi096 ceph-mon[32753]: pgmap v16869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:25.499+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:25.499+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:28:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:25.499+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31045.103669 secs 2024-04-26T20:28:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31045.103669 secs 2024-04-26T20:28:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31045.103669 secs 2024-04-26T20:28:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:26 smithi194 ceph-mon[33017]: pgmap v16870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:26 smithi033 ceph-mon[27845]: pgmap v16870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:26 smithi096 ceph-mon[32753]: pgmap v16870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:28 smithi194 ceph-mon[33017]: pgmap v16871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:28 smithi033 ceph-mon[27845]: pgmap v16871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:28 smithi096 ceph-mon[32753]: pgmap v16871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31047.175007 secs 2024-04-26T20:28:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31047.175007 secs 2024-04-26T20:28:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31047.175007 secs 2024-04-26T20:28:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31050.103782 secs 2024-04-26T20:28:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:30 smithi194 ceph-mon[33017]: pgmap v16872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31050.103782 secs 2024-04-26T20:28:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:30 smithi033 ceph-mon[27845]: pgmap v16872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31050.103782 secs 2024-04-26T20:28:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:30 smithi096 ceph-mon[32753]: pgmap v16872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:32 smithi194 ceph-mon[33017]: pgmap v16873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:32 smithi033 ceph-mon[27845]: pgmap v16873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:32 smithi096 ceph-mon[32753]: pgmap v16873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31052.175066 secs 2024-04-26T20:28:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:34 smithi194 ceph-mon[33017]: pgmap v16874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31052.175066 secs 2024-04-26T20:28:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:34 smithi033 ceph-mon[27845]: pgmap v16874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31052.175066 secs 2024-04-26T20:28:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:34 smithi096 ceph-mon[32753]: pgmap v16874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31055.103901 secs 2024-04-26T20:28:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31055.103901 secs 2024-04-26T20:28:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31055.103901 secs 2024-04-26T20:28:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:36 smithi194 ceph-mon[33017]: pgmap v16875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:36 smithi033 ceph-mon[27845]: pgmap v16875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:36 smithi096 ceph-mon[32753]: pgmap v16875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:38 smithi194 ceph-mon[33017]: pgmap v16876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:38 smithi033 ceph-mon[27845]: pgmap v16876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:38 smithi096 ceph-mon[32753]: pgmap v16876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31057.175137 secs 2024-04-26T20:28:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31057.175137 secs 2024-04-26T20:28:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31057.175137 secs 2024-04-26T20:28:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:40.507+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:40.508+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:40.508+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:28:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31060.104039 secs 2024-04-26T20:28:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:40 smithi194 ceph-mon[33017]: pgmap v16877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31060.104039 secs 2024-04-26T20:28:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:40 smithi033 ceph-mon[27845]: pgmap v16877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31060.104039 secs 2024-04-26T20:28:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:40 smithi096 ceph-mon[32753]: pgmap v16877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:42 smithi194 ceph-mon[33017]: pgmap v16878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:42 smithi033 ceph-mon[27845]: pgmap v16878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:42 smithi096 ceph-mon[32753]: pgmap v16878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31062.175196 secs 2024-04-26T20:28:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:44 smithi194 ceph-mon[33017]: pgmap v16879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31062.175196 secs 2024-04-26T20:28:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:44 smithi033 ceph-mon[27845]: pgmap v16879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31062.175196 secs 2024-04-26T20:28:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:44 smithi096 ceph-mon[32753]: pgmap v16879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31065.104138 secs 2024-04-26T20:28:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31065.104138 secs 2024-04-26T20:28:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31065.104138 secs 2024-04-26T20:28:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:46 smithi194 ceph-mon[33017]: pgmap v16880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:46 smithi033 ceph-mon[27845]: pgmap v16880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:46 smithi096 ceph-mon[32753]: pgmap v16880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:48 smithi194 ceph-mon[33017]: pgmap v16881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:48.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:48 smithi033 ceph-mon[27845]: pgmap v16881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:48.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:48 smithi096 ceph-mon[32753]: pgmap v16881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31067.175259 secs 2024-04-26T20:28:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31067.175259 secs 2024-04-26T20:28:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31067.175259 secs 2024-04-26T20:28:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31070.104206 secs 2024-04-26T20:28:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:50 smithi194 ceph-mon[33017]: pgmap v16882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31070.104206 secs 2024-04-26T20:28:50.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:50 smithi033 ceph-mon[27845]: pgmap v16882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:50.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31070.104206 secs 2024-04-26T20:28:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:50 smithi096 ceph-mon[32753]: pgmap v16882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:52.814 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:52 smithi033 ceph-mon[27845]: pgmap v16883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:52 smithi194 ceph-mon[33017]: pgmap v16883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:52 smithi096 ceph-mon[32753]: pgmap v16883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:28:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: pgmap v16884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: pgmap v16885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31072.175320 secs 2024-04-26T20:28:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: pgmap v16884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: pgmap v16885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31072.175320 secs 2024-04-26T20:28:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: pgmap v16884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: pgmap v16885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31072.175320 secs 2024-04-26T20:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:28:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31075.104339 secs 2024-04-26T20:28:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31075.104339 secs 2024-04-26T20:28:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31075.104339 secs 2024-04-26T20:28:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:55.525+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:55.525+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:28:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:28:55.525+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:28:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:55 smithi194 ceph-mon[33017]: pgmap v16886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:55 smithi033 ceph-mon[27845]: pgmap v16886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:55 smithi096 ceph-mon[32753]: pgmap v16886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:57 smithi194 ceph-mon[33017]: pgmap v16887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:57 smithi033 ceph-mon[27845]: pgmap v16887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:57 smithi096 ceph-mon[32753]: pgmap v16887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:28:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:28:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31077.175404 secs 2024-04-26T20:28:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:28:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31077.175404 secs 2024-04-26T20:28:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:28:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31077.175404 secs 2024-04-26T20:29:00.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31080.104431 secs 2024-04-26T20:29:00.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:00 smithi033 ceph-mon[27845]: pgmap v16888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:29:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31080.104431 secs 2024-04-26T20:29:00.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:00 smithi096 ceph-mon[32753]: pgmap v16888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:29:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31080.104431 secs 2024-04-26T20:29:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:00 smithi194 ceph-mon[33017]: pgmap v16888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:29:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:02 smithi096 ceph-mon[32753]: pgmap v16889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:29:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:02 smithi194 ceph-mon[33017]: pgmap v16889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:29:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:02 smithi033 ceph-mon[27845]: pgmap v16889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:29:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31082.175489 secs 2024-04-26T20:29:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:04 smithi194 ceph-mon[33017]: pgmap v16890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:29:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31082.175489 secs 2024-04-26T20:29:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:04 smithi033 ceph-mon[27845]: pgmap v16890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:29:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31082.175489 secs 2024-04-26T20:29:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:04 smithi096 ceph-mon[32753]: pgmap v16890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31085.104502 secs 2024-04-26T20:29:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31085.104502 secs 2024-04-26T20:29:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31085.104502 secs 2024-04-26T20:29:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:06 smithi194 ceph-mon[33017]: pgmap v16891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:06 smithi033 ceph-mon[27845]: pgmap v16891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:06 smithi096 ceph-mon[32753]: pgmap v16891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:08 smithi194 ceph-mon[33017]: pgmap v16892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:08 smithi033 ceph-mon[27845]: pgmap v16892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:08 smithi096 ceph-mon[32753]: pgmap v16892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31087.175572 secs 2024-04-26T20:29:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31087.175572 secs 2024-04-26T20:29:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31087.175572 secs 2024-04-26T20:29:10.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31090.104597 secs 2024-04-26T20:29:10.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:10 smithi194 ceph-mon[33017]: pgmap v16893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31090.104597 secs 2024-04-26T20:29:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:10 smithi033 ceph-mon[27845]: pgmap v16893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31090.104597 secs 2024-04-26T20:29:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:10 smithi096 ceph-mon[32753]: pgmap v16893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:10.527+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:10.528+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:10.528+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:29:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:12 smithi096 ceph-mon[32753]: pgmap v16894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:12 smithi194 ceph-mon[33017]: pgmap v16894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:12 smithi033 ceph-mon[27845]: pgmap v16894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31092.175654 secs 2024-04-26T20:29:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:14 smithi194 ceph-mon[33017]: pgmap v16895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31092.175654 secs 2024-04-26T20:29:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:14 smithi033 ceph-mon[27845]: pgmap v16895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31092.175654 secs 2024-04-26T20:29:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:14 smithi096 ceph-mon[32753]: pgmap v16895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31095.104725 secs 2024-04-26T20:29:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31095.104725 secs 2024-04-26T20:29:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31095.104725 secs 2024-04-26T20:29:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:16 smithi194 ceph-mon[33017]: pgmap v16896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:16 smithi033 ceph-mon[27845]: pgmap v16896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:16 smithi096 ceph-mon[32753]: pgmap v16896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:18 smithi194 ceph-mon[33017]: pgmap v16897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:18 smithi033 ceph-mon[27845]: pgmap v16897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:18 smithi096 ceph-mon[32753]: pgmap v16897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31097.175735 secs 2024-04-26T20:29:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31097.175735 secs 2024-04-26T20:29:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31097.175735 secs 2024-04-26T20:29:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31100.104869 secs 2024-04-26T20:29:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:20 smithi194 ceph-mon[33017]: pgmap v16898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31100.104869 secs 2024-04-26T20:29:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:20 smithi033 ceph-mon[27845]: pgmap v16898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31100.104869 secs 2024-04-26T20:29:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:20 smithi096 ceph-mon[32753]: pgmap v16898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:22 smithi194 ceph-mon[33017]: pgmap v16899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:22 smithi033 ceph-mon[27845]: pgmap v16899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:22 smithi096 ceph-mon[32753]: pgmap v16899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31102.175792 secs 2024-04-26T20:29:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:24 smithi194 ceph-mon[33017]: pgmap v16900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31102.175792 secs 2024-04-26T20:29:24.447 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:24 smithi033 ceph-mon[27845]: pgmap v16900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:24.448 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31102.175792 secs 2024-04-26T20:29:24.448 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:24 smithi096 ceph-mon[32753]: pgmap v16900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31105.104991 secs 2024-04-26T20:29:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31105.104991 secs 2024-04-26T20:29:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:25.535+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:25.535+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:29:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:25.535+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31105.104991 secs 2024-04-26T20:29:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:26 smithi033 ceph-mon[27845]: pgmap v16901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:26 smithi096 ceph-mon[32753]: pgmap v16901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:26 smithi194 ceph-mon[33017]: pgmap v16901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:27.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.087+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T20:29:27.088 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.087+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:29:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.087+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 244 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T20:29:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.088+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 244 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T20:29:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.088+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x564385821900 2024-04-26T20:29:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.088+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x7fef00077780 2024-04-26T20:29:27.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.088+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 245 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T20:29:27.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:27.088+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 245 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:29:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:28 smithi033 ceph-mon[27845]: pgmap v16902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:28 smithi096 ceph-mon[32753]: pgmap v16902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:28 smithi194 ceph-mon[33017]: pgmap v16902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31107.175877 secs 2024-04-26T20:29:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31107.175877 secs 2024-04-26T20:29:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31107.175877 secs 2024-04-26T20:29:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31110.105137 secs 2024-04-26T20:29:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:30 smithi033 ceph-mon[27845]: pgmap v16903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31110.105137 secs 2024-04-26T20:29:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:30 smithi096 ceph-mon[32753]: pgmap v16903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31110.105137 secs 2024-04-26T20:29:30.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:30 smithi194 ceph-mon[33017]: pgmap v16903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:32 smithi033 ceph-mon[27845]: pgmap v16904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:32 smithi096 ceph-mon[32753]: pgmap v16904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:32 smithi194 ceph-mon[33017]: pgmap v16904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31112.175955 secs 2024-04-26T20:29:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:34 smithi033 ceph-mon[27845]: pgmap v16905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31112.175955 secs 2024-04-26T20:29:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:34 smithi096 ceph-mon[32753]: pgmap v16905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31112.175955 secs 2024-04-26T20:29:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:34 smithi194 ceph-mon[33017]: pgmap v16905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31115.105207 secs 2024-04-26T20:29:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31115.105207 secs 2024-04-26T20:29:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31115.105207 secs 2024-04-26T20:29:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:36 smithi033 ceph-mon[27845]: pgmap v16906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:36 smithi096 ceph-mon[32753]: pgmap v16906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:36 smithi194 ceph-mon[33017]: pgmap v16906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:38 smithi033 ceph-mon[27845]: pgmap v16907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:38 smithi096 ceph-mon[32753]: pgmap v16907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:38 smithi194 ceph-mon[33017]: pgmap v16907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31117.176037 secs 2024-04-26T20:29:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31117.176037 secs 2024-04-26T20:29:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31117.176037 secs 2024-04-26T20:29:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31120.105325 secs 2024-04-26T20:29:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:40 smithi033 ceph-mon[27845]: pgmap v16908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31120.105325 secs 2024-04-26T20:29:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:40 smithi096 ceph-mon[32753]: pgmap v16908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:40.550+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:40.550+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:40.550+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:29:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31120.105325 secs 2024-04-26T20:29:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:40 smithi194 ceph-mon[33017]: pgmap v16908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:42 smithi033 ceph-mon[27845]: pgmap v16909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:42 smithi096 ceph-mon[32753]: pgmap v16909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:42 smithi194 ceph-mon[33017]: pgmap v16909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31122.176097 secs 2024-04-26T20:29:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:44 smithi096 ceph-mon[32753]: pgmap v16910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31122.176097 secs 2024-04-26T20:29:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:44 smithi194 ceph-mon[33017]: pgmap v16910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31122.176097 secs 2024-04-26T20:29:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:44 smithi033 ceph-mon[27845]: pgmap v16910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31125.105445 secs 2024-04-26T20:29:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31125.105445 secs 2024-04-26T20:29:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31125.105445 secs 2024-04-26T20:29:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:46 smithi194 ceph-mon[33017]: pgmap v16911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:46 smithi033 ceph-mon[27845]: pgmap v16911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:46 smithi096 ceph-mon[32753]: pgmap v16911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:48 smithi194 ceph-mon[33017]: pgmap v16912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:48 smithi033 ceph-mon[27845]: pgmap v16912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:48 smithi096 ceph-mon[32753]: pgmap v16912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31127.176178 secs 2024-04-26T20:29:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31127.176178 secs 2024-04-26T20:29:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31127.176178 secs 2024-04-26T20:29:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31130.105562 secs 2024-04-26T20:29:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:50 smithi194 ceph-mon[33017]: pgmap v16913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31130.105562 secs 2024-04-26T20:29:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:50 smithi033 ceph-mon[27845]: pgmap v16913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31130.105562 secs 2024-04-26T20:29:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:50 smithi096 ceph-mon[32753]: pgmap v16913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:52 smithi194 ceph-mon[33017]: pgmap v16914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:52 smithi033 ceph-mon[27845]: pgmap v16914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:52 smithi096 ceph-mon[32753]: pgmap v16914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31132.176283 secs 2024-04-26T20:29:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:54 smithi194 ceph-mon[33017]: pgmap v16915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31132.176283 secs 2024-04-26T20:29:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:54 smithi033 ceph-mon[27845]: pgmap v16915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31132.176283 secs 2024-04-26T20:29:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:54 smithi096 ceph-mon[32753]: pgmap v16915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:29:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:55.565+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:55.565+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:29:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:29:55.565+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:29:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31135.105648 secs 2024-04-26T20:29:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31135.105648 secs 2024-04-26T20:29:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31135.105648 secs 2024-04-26T20:29:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:56 smithi194 ceph-mon[33017]: pgmap v16916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:56 smithi033 ceph-mon[27845]: pgmap v16916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:56 smithi096 ceph-mon[32753]: pgmap v16916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:29:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:29:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:57 smithi194 ceph-mon[33017]: pgmap v16917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:29:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:57 smithi194 ceph-mon[33017]: pgmap v16918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:29:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:57 smithi033 ceph-mon[27845]: pgmap v16917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:29:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:57 smithi033 ceph-mon[27845]: pgmap v16918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:29:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:57 smithi096 ceph-mon[32753]: pgmap v16917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:29:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:57 smithi096 ceph-mon[32753]: pgmap v16918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:29:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:59 smithi194 ceph-mon[33017]: pgmap v16919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:29:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:29:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31137.176369 secs 2024-04-26T20:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:59 smithi033 ceph-mon[27845]: pgmap v16919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:29:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31137.176369 secs 2024-04-26T20:29:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:59 smithi096 ceph-mon[32753]: pgmap v16919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:29:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:29:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31137.176369 secs 2024-04-26T20:30:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31140.105784 secs 2024-04-26T20:30:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:30:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31140.105784 secs 2024-04-26T20:30:00.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31140.105784 secs 2024-04-26T20:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:30:01.611 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:01 smithi033 ceph-mon[27845]: pgmap v16920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:30:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:01 smithi096 ceph-mon[32753]: pgmap v16920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:30:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:01 smithi194 ceph-mon[33017]: pgmap v16920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:30:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:03 smithi033 ceph-mon[27845]: pgmap v16921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:30:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:03 smithi096 ceph-mon[32753]: pgmap v16921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:30:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:03 smithi194 ceph-mon[33017]: pgmap v16921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:30:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31142.176464 secs 2024-04-26T20:30:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31142.176464 secs 2024-04-26T20:30:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31142.176464 secs 2024-04-26T20:30:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31145.105902 secs 2024-04-26T20:30:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:05 smithi033 ceph-mon[27845]: pgmap v16922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:30:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31145.105902 secs 2024-04-26T20:30:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:05 smithi096 ceph-mon[32753]: pgmap v16922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:30:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31145.105902 secs 2024-04-26T20:30:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:05 smithi194 ceph-mon[33017]: pgmap v16922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:30:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:07 smithi033 ceph-mon[27845]: pgmap v16923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:30:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:07 smithi096 ceph-mon[32753]: pgmap v16923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:30:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:07 smithi194 ceph-mon[33017]: pgmap v16923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:30:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:09 smithi033 ceph-mon[27845]: pgmap v16924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31147.176544 secs 2024-04-26T20:30:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:09 smithi096 ceph-mon[32753]: pgmap v16924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31147.176544 secs 2024-04-26T20:30:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:09 smithi194 ceph-mon[33017]: pgmap v16924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31147.176544 secs 2024-04-26T20:30:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:10.576+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:10.577+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:10.577+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:30:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31150.106041 secs 2024-04-26T20:30:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31150.106041 secs 2024-04-26T20:30:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31150.106041 secs 2024-04-26T20:30:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:11 smithi033 ceph-mon[27845]: pgmap v16925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:11 smithi096 ceph-mon[32753]: pgmap v16925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:11 smithi194 ceph-mon[33017]: pgmap v16925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:13 smithi033 ceph-mon[27845]: pgmap v16926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:13 smithi096 ceph-mon[32753]: pgmap v16926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:13 smithi194 ceph-mon[33017]: pgmap v16926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31152.176624 secs 2024-04-26T20:30:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31152.176624 secs 2024-04-26T20:30:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31152.176624 secs 2024-04-26T20:30:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31155.106159 secs 2024-04-26T20:30:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:15 smithi033 ceph-mon[27845]: pgmap v16927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31155.106159 secs 2024-04-26T20:30:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:15 smithi096 ceph-mon[32753]: pgmap v16927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31155.106159 secs 2024-04-26T20:30:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:15 smithi194 ceph-mon[33017]: pgmap v16927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:17 smithi033 ceph-mon[27845]: pgmap v16928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:17 smithi096 ceph-mon[32753]: pgmap v16928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:17 smithi194 ceph-mon[33017]: pgmap v16928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:19 smithi096 ceph-mon[32753]: pgmap v16929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31157.176713 secs 2024-04-26T20:30:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:19 smithi194 ceph-mon[33017]: pgmap v16929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31157.176713 secs 2024-04-26T20:30:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:19 smithi033 ceph-mon[27845]: pgmap v16929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31157.176713 secs 2024-04-26T20:30:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31160.106283 secs 2024-04-26T20:30:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31160.106283 secs 2024-04-26T20:30:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31160.106283 secs 2024-04-26T20:30:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:21 smithi194 ceph-mon[33017]: pgmap v16930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:21 smithi033 ceph-mon[27845]: pgmap v16930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:21 smithi096 ceph-mon[32753]: pgmap v16930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:23 smithi194 ceph-mon[33017]: pgmap v16931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:23 smithi033 ceph-mon[27845]: pgmap v16931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:23 smithi096 ceph-mon[32753]: pgmap v16931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31162.176793 secs 2024-04-26T20:30:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31162.176793 secs 2024-04-26T20:30:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31162.176793 secs 2024-04-26T20:30:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:25.589+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:25.589+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:25.589+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:30:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31165.106391 secs 2024-04-26T20:30:26.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:25 smithi194 ceph-mon[33017]: pgmap v16932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:26.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31165.106391 secs 2024-04-26T20:30:26.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:25 smithi096 ceph-mon[32753]: pgmap v16932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:26.047 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31165.106391 secs 2024-04-26T20:30:26.048 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:25 smithi033 ceph-mon[27845]: pgmap v16932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:27 smithi194 ceph-mon[33017]: pgmap v16933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:27 smithi033 ceph-mon[27845]: pgmap v16933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:27 smithi096 ceph-mon[32753]: pgmap v16933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:29 smithi194 ceph-mon[33017]: pgmap v16934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31167.176854 secs 2024-04-26T20:30:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:29 smithi033 ceph-mon[27845]: pgmap v16934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31167.176854 secs 2024-04-26T20:30:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:29 smithi096 ceph-mon[32753]: pgmap v16934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31167.176854 secs 2024-04-26T20:30:30.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31170.106508 secs 2024-04-26T20:30:30.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31170.106508 secs 2024-04-26T20:30:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31170.106508 secs 2024-04-26T20:30:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:31 smithi194 ceph-mon[33017]: pgmap v16935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:31 smithi033 ceph-mon[27845]: pgmap v16935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:31 smithi096 ceph-mon[32753]: pgmap v16935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:33 smithi194 ceph-mon[33017]: pgmap v16936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:33 smithi033 ceph-mon[27845]: pgmap v16936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:33 smithi096 ceph-mon[32753]: pgmap v16936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31172.176935 secs 2024-04-26T20:30:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31172.176935 secs 2024-04-26T20:30:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31172.176935 secs 2024-04-26T20:30:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31175.106600 secs 2024-04-26T20:30:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:35 smithi194 ceph-mon[33017]: pgmap v16937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31175.106600 secs 2024-04-26T20:30:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:35 smithi033 ceph-mon[27845]: pgmap v16937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31175.106600 secs 2024-04-26T20:30:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:35 smithi096 ceph-mon[32753]: pgmap v16937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:37 smithi194 ceph-mon[33017]: pgmap v16938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:37 smithi033 ceph-mon[27845]: pgmap v16938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:37 smithi096 ceph-mon[32753]: pgmap v16938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:39 smithi194 ceph-mon[33017]: pgmap v16939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31177.177005 secs 2024-04-26T20:30:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:39 smithi033 ceph-mon[27845]: pgmap v16939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31177.177005 secs 2024-04-26T20:30:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:39 smithi096 ceph-mon[32753]: pgmap v16939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31177.177005 secs 2024-04-26T20:30:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:40.590+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:40.590+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:30:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:40.590+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31180.106718 secs 2024-04-26T20:30:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31180.106718 secs 2024-04-26T20:30:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31180.106718 secs 2024-04-26T20:30:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:41 smithi194 ceph-mon[33017]: pgmap v16940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:41 smithi033 ceph-mon[27845]: pgmap v16940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:41 smithi096 ceph-mon[32753]: pgmap v16940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:43 smithi194 ceph-mon[33017]: pgmap v16941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:43 smithi033 ceph-mon[27845]: pgmap v16941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:43 smithi096 ceph-mon[32753]: pgmap v16941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31182.177065 secs 2024-04-26T20:30:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31182.177065 secs 2024-04-26T20:30:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31182.177065 secs 2024-04-26T20:30:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31185.106828 secs 2024-04-26T20:30:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:45 smithi194 ceph-mon[33017]: pgmap v16942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31185.106828 secs 2024-04-26T20:30:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:45 smithi033 ceph-mon[27845]: pgmap v16942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31185.106828 secs 2024-04-26T20:30:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:45 smithi096 ceph-mon[32753]: pgmap v16942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:30:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:47 smithi194 ceph-mon[33017]: pgmap v16943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:47 smithi033 ceph-mon[27845]: pgmap v16943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:47 smithi096 ceph-mon[32753]: pgmap v16943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:30:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:49 smithi194 ceph-mon[33017]: pgmap v16944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31187.177127 secs 2024-04-26T20:30:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:49 smithi033 ceph-mon[27845]: pgmap v16944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31187.177127 secs 2024-04-26T20:30:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:49 smithi096 ceph-mon[32753]: pgmap v16944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31187.177127 secs 2024-04-26T20:30:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31190.106947 secs 2024-04-26T20:30:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31190.106947 secs 2024-04-26T20:30:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31190.106947 secs 2024-04-26T20:30:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:51 smithi194 ceph-mon[33017]: pgmap v16945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:51 smithi033 ceph-mon[27845]: pgmap v16945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:51 smithi096 ceph-mon[32753]: pgmap v16945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:53 smithi194 ceph-mon[33017]: pgmap v16946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:53 smithi033 ceph-mon[27845]: pgmap v16946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:53 smithi096 ceph-mon[32753]: pgmap v16946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31192.177213 secs 2024-04-26T20:30:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31192.177213 secs 2024-04-26T20:30:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31192.177213 secs 2024-04-26T20:30:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:55.606+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:55.606+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:30:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:30:55.606+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:30:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31195.107064 secs 2024-04-26T20:30:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:55 smithi194 ceph-mon[33017]: pgmap v16947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31195.107064 secs 2024-04-26T20:30:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:55 smithi033 ceph-mon[27845]: pgmap v16947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31195.107064 secs 2024-04-26T20:30:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:55 smithi096 ceph-mon[32753]: pgmap v16947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:57 smithi033 ceph-mon[27845]: pgmap v16948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:30:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:57 smithi096 ceph-mon[32753]: pgmap v16948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:30:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:57 smithi194 ceph-mon[33017]: pgmap v16948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:30:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:30:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:58 smithi033 ceph-mon[27845]: pgmap v16949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:58 smithi033 ceph-mon[27845]: pgmap v16950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:30:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:58 smithi096 ceph-mon[32753]: pgmap v16949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:58 smithi096 ceph-mon[32753]: pgmap v16950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:30:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:58 smithi194 ceph-mon[33017]: pgmap v16949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:30:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:58 smithi194 ceph-mon[33017]: pgmap v16950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:30:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:30:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31197.177277 secs 2024-04-26T20:30:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:30:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31197.177277 secs 2024-04-26T20:31:00.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:30:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31197.177277 secs 2024-04-26T20:31:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31200.107170 secs 2024-04-26T20:31:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:00 smithi033 ceph-mon[27845]: pgmap v16951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31200.107170 secs 2024-04-26T20:31:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:00 smithi096 ceph-mon[32753]: pgmap v16951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31200.107170 secs 2024-04-26T20:31:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:00 smithi194 ceph-mon[33017]: pgmap v16951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:02 smithi033 ceph-mon[27845]: pgmap v16952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:02 smithi096 ceph-mon[32753]: pgmap v16952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:02 smithi194 ceph-mon[33017]: pgmap v16952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31202.177336 secs 2024-04-26T20:31:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:04 smithi033 ceph-mon[27845]: pgmap v16953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31202.177336 secs 2024-04-26T20:31:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:04 smithi096 ceph-mon[32753]: pgmap v16953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31202.177336 secs 2024-04-26T20:31:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:04 smithi194 ceph-mon[33017]: pgmap v16953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31205.107251 secs 2024-04-26T20:31:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31205.107251 secs 2024-04-26T20:31:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31205.107251 secs 2024-04-26T20:31:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:06 smithi033 ceph-mon[27845]: pgmap v16954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:06 smithi096 ceph-mon[32753]: pgmap v16954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:06 smithi194 ceph-mon[33017]: pgmap v16954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:31:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:08 smithi033 ceph-mon[27845]: pgmap v16955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:31:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:08 smithi096 ceph-mon[32753]: pgmap v16955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:31:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:08 smithi194 ceph-mon[33017]: pgmap v16955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:31:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31207.177420 secs 2024-04-26T20:31:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31210.107317 secs 2024-04-26T20:31:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31207.177420 secs 2024-04-26T20:31:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31210.107317 secs 2024-04-26T20:31:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31207.177420 secs 2024-04-26T20:31:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31210.107317 secs 2024-04-26T20:31:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:10.615+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:10.615+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:10.615+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:31:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:10 smithi096 ceph-mon[32753]: pgmap v16956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:10 smithi194 ceph-mon[33017]: pgmap v16956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:10 smithi033 ceph-mon[27845]: pgmap v16956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:12 smithi194 ceph-mon[33017]: pgmap v16957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:12 smithi033 ceph-mon[27845]: pgmap v16957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:12 smithi096 ceph-mon[32753]: pgmap v16957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31212.177500 secs 2024-04-26T20:31:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:14 smithi194 ceph-mon[33017]: pgmap v16958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31215.107419 secs 2024-04-26T20:31:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31212.177500 secs 2024-04-26T20:31:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:14 smithi033 ceph-mon[27845]: pgmap v16958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31215.107419 secs 2024-04-26T20:31:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31212.177500 secs 2024-04-26T20:31:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:14 smithi096 ceph-mon[32753]: pgmap v16958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31215.107419 secs 2024-04-26T20:31:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:16 smithi194 ceph-mon[33017]: pgmap v16959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:16 smithi033 ceph-mon[27845]: pgmap v16959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:16 smithi096 ceph-mon[32753]: pgmap v16959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:17 smithi194 ceph-mon[33017]: pgmap v16960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:17 smithi033 ceph-mon[27845]: pgmap v16960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:17 smithi096 ceph-mon[32753]: pgmap v16960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31217.177586 secs 2024-04-26T20:31:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31217.177586 secs 2024-04-26T20:31:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31217.177586 secs 2024-04-26T20:31:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31220.107534 secs 2024-04-26T20:31:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:19 smithi033 ceph-mon[27845]: pgmap v16961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31220.107534 secs 2024-04-26T20:31:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:19 smithi096 ceph-mon[32753]: pgmap v16961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31220.107534 secs 2024-04-26T20:31:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:19 smithi194 ceph-mon[33017]: pgmap v16961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:22 smithi194 ceph-mon[33017]: pgmap v16962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:22 smithi033 ceph-mon[27845]: pgmap v16962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:22 smithi096 ceph-mon[32753]: pgmap v16962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31222.177669 secs 2024-04-26T20:31:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:24 smithi194 ceph-mon[33017]: pgmap v16963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31222.177669 secs 2024-04-26T20:31:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:24 smithi033 ceph-mon[27845]: pgmap v16963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31222.177669 secs 2024-04-26T20:31:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:24 smithi096 ceph-mon[32753]: pgmap v16963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:25.629+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:25.629+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:31:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:25.629+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31225.107651 secs 2024-04-26T20:31:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31225.107651 secs 2024-04-26T20:31:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31225.107651 secs 2024-04-26T20:31:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:26 smithi194 ceph-mon[33017]: pgmap v16964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:26 smithi033 ceph-mon[27845]: pgmap v16964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:26 smithi096 ceph-mon[32753]: pgmap v16964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:28 smithi194 ceph-mon[33017]: pgmap v16965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:28 smithi033 ceph-mon[27845]: pgmap v16965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:28 smithi096 ceph-mon[32753]: pgmap v16965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31227.177752 secs 2024-04-26T20:31:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31227.177752 secs 2024-04-26T20:31:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31227.177752 secs 2024-04-26T20:31:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31230.107770 secs 2024-04-26T20:31:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:30 smithi194 ceph-mon[33017]: pgmap v16966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31230.107770 secs 2024-04-26T20:31:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:30 smithi033 ceph-mon[27845]: pgmap v16966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31230.107770 secs 2024-04-26T20:31:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:30 smithi096 ceph-mon[32753]: pgmap v16966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:32 smithi033 ceph-mon[27845]: pgmap v16967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:32 smithi096 ceph-mon[32753]: pgmap v16967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:32 smithi194 ceph-mon[33017]: pgmap v16967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31232.177835 secs 2024-04-26T20:31:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:34 smithi033 ceph-mon[27845]: pgmap v16968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31232.177835 secs 2024-04-26T20:31:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:34 smithi096 ceph-mon[32753]: pgmap v16968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31232.177835 secs 2024-04-26T20:31:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:34 smithi194 ceph-mon[33017]: pgmap v16968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31235.107894 secs 2024-04-26T20:31:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31235.107894 secs 2024-04-26T20:31:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31235.107894 secs 2024-04-26T20:31:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:36 smithi033 ceph-mon[27845]: pgmap v16969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:36 smithi096 ceph-mon[32753]: pgmap v16969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:36 smithi194 ceph-mon[33017]: pgmap v16969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:38 smithi033 ceph-mon[27845]: pgmap v16970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:38 smithi096 ceph-mon[32753]: pgmap v16970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:38 smithi194 ceph-mon[33017]: pgmap v16970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31237.177921 secs 2024-04-26T20:31:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31237.177921 secs 2024-04-26T20:31:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31237.177921 secs 2024-04-26T20:31:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:40.644+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:40.644+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:40.644+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:31:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31240.108009 secs 2024-04-26T20:31:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:40 smithi033 ceph-mon[27845]: pgmap v16971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31240.108009 secs 2024-04-26T20:31:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:40 smithi096 ceph-mon[32753]: pgmap v16971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31240.108009 secs 2024-04-26T20:31:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:40 smithi194 ceph-mon[33017]: pgmap v16971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:42.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:42 smithi033 ceph-mon[27845]: pgmap v16972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:42 smithi096 ceph-mon[32753]: pgmap v16972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:42 smithi194 ceph-mon[33017]: pgmap v16972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31242.178000 secs 2024-04-26T20:31:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:44 smithi033 ceph-mon[27845]: pgmap v16973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31242.178000 secs 2024-04-26T20:31:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:44 smithi096 ceph-mon[32753]: pgmap v16973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:45.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31242.178000 secs 2024-04-26T20:31:45.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:44 smithi194 ceph-mon[33017]: pgmap v16973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31245.108149 secs 2024-04-26T20:31:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31245.108149 secs 2024-04-26T20:31:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31245.108149 secs 2024-04-26T20:31:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:46 smithi033 ceph-mon[27845]: pgmap v16974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:46 smithi096 ceph-mon[32753]: pgmap v16974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:46 smithi194 ceph-mon[33017]: pgmap v16974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:48 smithi033 ceph-mon[27845]: pgmap v16975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:48 smithi096 ceph-mon[32753]: pgmap v16975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:48 smithi194 ceph-mon[33017]: pgmap v16975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31247.178077 secs 2024-04-26T20:31:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31247.178077 secs 2024-04-26T20:31:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31247.178077 secs 2024-04-26T20:31:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31250.108213 secs 2024-04-26T20:31:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:50 smithi194 ceph-mon[33017]: pgmap v16976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31250.108213 secs 2024-04-26T20:31:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:50 smithi033 ceph-mon[27845]: pgmap v16976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31250.108213 secs 2024-04-26T20:31:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:50 smithi096 ceph-mon[32753]: pgmap v16976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:52 smithi194 ceph-mon[33017]: pgmap v16977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:52 smithi033 ceph-mon[27845]: pgmap v16977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:52 smithi096 ceph-mon[32753]: pgmap v16977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:55.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31252.178153 secs 2024-04-26T20:31:55.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:54 smithi194 ceph-mon[33017]: pgmap v16978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:55.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31255.108336 secs 2024-04-26T20:31:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31252.178153 secs 2024-04-26T20:31:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:54 smithi033 ceph-mon[27845]: pgmap v16978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31255.108336 secs 2024-04-26T20:31:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31252.178153 secs 2024-04-26T20:31:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:54 smithi096 ceph-mon[32753]: pgmap v16978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31255.108336 secs 2024-04-26T20:31:55.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:55.655+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:55.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:55.656+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:31:55.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:31:55.656+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:31:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:56 smithi194 ceph-mon[33017]: pgmap v16979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:56 smithi033 ceph-mon[27845]: pgmap v16979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:56 smithi096 ceph-mon[32753]: pgmap v16979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:31:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:58 smithi194 ceph-mon[33017]: pgmap v16980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:58 smithi033 ceph-mon[27845]: pgmap v16980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:58 smithi096 ceph-mon[32753]: pgmap v16980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:31:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:31:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31257.178219 secs 2024-04-26T20:31:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31260.108400 secs 2024-04-26T20:32:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31257.178219 secs 2024-04-26T20:32:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31260.108400 secs 2024-04-26T20:32:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31257.178219 secs 2024-04-26T20:32:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31260.108400 secs 2024-04-26T20:32:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:00 smithi194 ceph-mon[33017]: pgmap v16981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:00 smithi033 ceph-mon[27845]: pgmap v16981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:00 smithi096 ceph-mon[32753]: pgmap v16981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:01.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:32:02.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:32:02.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:01 smithi194 ceph-mon[33017]: pgmap v16982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:32:02.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:32:02.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:01 smithi033 ceph-mon[27845]: pgmap v16982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:02.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:32:02.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:32:02.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:01 smithi096 ceph-mon[32753]: pgmap v16982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31262.178299 secs 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: pgmap v16983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:04.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:04.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31262.178299 secs 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: pgmap v16983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31262.178299 secs 2024-04-26T20:32:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: pgmap v16983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:32:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31265.108524 secs 2024-04-26T20:32:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:05 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:32:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:05 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:32:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:05 smithi194 ceph-mon[33017]: pgmap v16984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:32:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:05 smithi194 ceph-mon[33017]: pgmap v16985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:32:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31265.108524 secs 2024-04-26T20:32:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:05 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:32:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:05 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:32:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:05 smithi033 ceph-mon[27845]: pgmap v16984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:32:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:05 smithi033 ceph-mon[27845]: pgmap v16985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:32:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31265.108524 secs 2024-04-26T20:32:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:05 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:32:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:05 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:32:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:05 smithi096 ceph-mon[32753]: pgmap v16984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:32:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:05 smithi096 ceph-mon[32753]: pgmap v16985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:32:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:07 smithi194 ceph-mon[33017]: pgmap v16986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:07 smithi033 ceph-mon[27845]: pgmap v16986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:07 smithi096 ceph-mon[32753]: pgmap v16986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31267.178360 secs 2024-04-26T20:32:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:09 smithi033 ceph-mon[27845]: pgmap v16987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:32:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31267.178360 secs 2024-04-26T20:32:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:09 smithi096 ceph-mon[32753]: pgmap v16987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:32:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31267.178360 secs 2024-04-26T20:32:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:09 smithi194 ceph-mon[33017]: pgmap v16987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:32:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:10.671+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:10.671+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:10.671+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:32:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31270.108608 secs 2024-04-26T20:32:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31270.108608 secs 2024-04-26T20:32:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31270.108608 secs 2024-04-26T20:32:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:11 smithi033 ceph-mon[27845]: pgmap v16988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:11 smithi096 ceph-mon[32753]: pgmap v16988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:11 smithi194 ceph-mon[33017]: pgmap v16988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:13 smithi033 ceph-mon[27845]: pgmap v16989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:13 smithi096 ceph-mon[32753]: pgmap v16989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:13 smithi194 ceph-mon[33017]: pgmap v16989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:32:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31272.178435 secs 2024-04-26T20:32:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31272.178435 secs 2024-04-26T20:32:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31272.178435 secs 2024-04-26T20:32:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31275.108684 secs 2024-04-26T20:32:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:15 smithi033 ceph-mon[27845]: pgmap v16990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:32:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31275.108684 secs 2024-04-26T20:32:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:15 smithi096 ceph-mon[32753]: pgmap v16990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:32:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31275.108684 secs 2024-04-26T20:32:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:15 smithi194 ceph-mon[33017]: pgmap v16990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:32:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:17 smithi033 ceph-mon[27845]: pgmap v16991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:17 smithi096 ceph-mon[32753]: pgmap v16991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:17 smithi194 ceph-mon[33017]: pgmap v16991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31277.178499 secs 2024-04-26T20:32:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:19 smithi033 ceph-mon[27845]: pgmap v16992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31277.178499 secs 2024-04-26T20:32:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:19 smithi096 ceph-mon[32753]: pgmap v16992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31277.178499 secs 2024-04-26T20:32:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:19 smithi194 ceph-mon[33017]: pgmap v16992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31280.108746 secs 2024-04-26T20:32:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31280.108746 secs 2024-04-26T20:32:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31280.108746 secs 2024-04-26T20:32:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:21 smithi033 ceph-mon[27845]: pgmap v16993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:21 smithi096 ceph-mon[32753]: pgmap v16993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:21 smithi194 ceph-mon[33017]: pgmap v16993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:23 smithi033 ceph-mon[27845]: pgmap v16994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:23 smithi096 ceph-mon[32753]: pgmap v16994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:23 smithi194 ceph-mon[33017]: pgmap v16994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31282.178577 secs 2024-04-26T20:32:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31285.108823 secs 2024-04-26T20:32:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31282.178577 secs 2024-04-26T20:32:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31285.108823 secs 2024-04-26T20:32:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31282.178577 secs 2024-04-26T20:32:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31285.108823 secs 2024-04-26T20:32:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:25.685+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:25.685+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:25.685+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:32:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:25 smithi096 ceph-mon[32753]: pgmap v16995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:25 smithi194 ceph-mon[33017]: pgmap v16995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:25 smithi033 ceph-mon[27845]: pgmap v16995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:27.117+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:32:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:27.117+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:32:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:27.118+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 246 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:32:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:27.118+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 246 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:32:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:27 smithi096 ceph-mon[32753]: pgmap v16996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:27 smithi194 ceph-mon[33017]: pgmap v16996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:27 smithi033 ceph-mon[27845]: pgmap v16996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31290.108940 secs 2024-04-26T20:32:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31290.108940 secs 2024-04-26T20:32:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31290.108940 secs 2024-04-26T20:32:30.000 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31287.178663 secs 2024-04-26T20:32:30.000 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:29 smithi033 ceph-mon[27845]: pgmap v16997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31287.178663 secs 2024-04-26T20:32:30.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:29 smithi194 ceph-mon[33017]: pgmap v16997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:30.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31287.178663 secs 2024-04-26T20:32:30.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:29 smithi096 ceph-mon[32753]: pgmap v16997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:31 smithi194 ceph-mon[33017]: pgmap v16998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:31 smithi033 ceph-mon[27845]: pgmap v16998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:31 smithi096 ceph-mon[32753]: pgmap v16998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:33 smithi194 ceph-mon[33017]: pgmap v16999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31295.109039 secs 2024-04-26T20:32:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:33 smithi033 ceph-mon[27845]: pgmap v16999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31295.109039 secs 2024-04-26T20:32:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:33 smithi096 ceph-mon[32753]: pgmap v16999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31295.109039 secs 2024-04-26T20:32:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31292.178744 secs 2024-04-26T20:32:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31292.178744 secs 2024-04-26T20:32:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31292.178744 secs 2024-04-26T20:32:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:35 smithi194 ceph-mon[33017]: pgmap v17000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:35 smithi033 ceph-mon[27845]: pgmap v17000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:35 smithi096 ceph-mon[32753]: pgmap v17000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:36 smithi194 ceph-mon[33017]: pgmap v17001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:36 smithi033 ceph-mon[27845]: pgmap v17001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:36 smithi096 ceph-mon[32753]: pgmap v17001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31297.178831 secs 2024-04-26T20:32:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:39 smithi194 ceph-mon[33017]: pgmap v17002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31300.109164 secs 2024-04-26T20:32:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31297.178831 secs 2024-04-26T20:32:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:39 smithi033 ceph-mon[27845]: pgmap v17002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31300.109164 secs 2024-04-26T20:32:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31297.178831 secs 2024-04-26T20:32:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:39 smithi096 ceph-mon[32753]: pgmap v17002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31300.109164 secs 2024-04-26T20:32:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:40.691+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:40.691+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:40.691+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:32:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:41 smithi194 ceph-mon[33017]: pgmap v17003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:41 smithi033 ceph-mon[27845]: pgmap v17003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:41 smithi096 ceph-mon[32753]: pgmap v17003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:43 smithi194 ceph-mon[33017]: pgmap v17004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:43 smithi033 ceph-mon[27845]: pgmap v17004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:43 smithi096 ceph-mon[32753]: pgmap v17004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31302.178911 secs 2024-04-26T20:32:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31305.109288 secs 2024-04-26T20:32:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31302.178911 secs 2024-04-26T20:32:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31305.109288 secs 2024-04-26T20:32:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31302.178911 secs 2024-04-26T20:32:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31305.109288 secs 2024-04-26T20:32:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:45 smithi033 ceph-mon[27845]: pgmap v17005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:45 smithi194 ceph-mon[33017]: pgmap v17005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:45 smithi096 ceph-mon[32753]: pgmap v17005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:47 smithi194 ceph-mon[33017]: pgmap v17006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:47 smithi033 ceph-mon[27845]: pgmap v17006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:47 smithi096 ceph-mon[32753]: pgmap v17006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31307.179013 secs 2024-04-26T20:32:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:49 smithi194 ceph-mon[33017]: pgmap v17007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31310.109439 secs 2024-04-26T20:32:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31307.179013 secs 2024-04-26T20:32:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:49 smithi033 ceph-mon[27845]: pgmap v17007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31310.109439 secs 2024-04-26T20:32:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31307.179013 secs 2024-04-26T20:32:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:49 smithi096 ceph-mon[32753]: pgmap v17007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31310.109439 secs 2024-04-26T20:32:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:51 smithi194 ceph-mon[33017]: pgmap v17008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:51 smithi033 ceph-mon[27845]: pgmap v17008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:51 smithi096 ceph-mon[32753]: pgmap v17008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:53 smithi194 ceph-mon[33017]: pgmap v17009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:53 smithi033 ceph-mon[27845]: pgmap v17009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:53 smithi096 ceph-mon[32753]: pgmap v17009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31312.179094 secs 2024-04-26T20:32:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31315.109579 secs 2024-04-26T20:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31312.179094 secs 2024-04-26T20:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31315.109579 secs 2024-04-26T20:32:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31312.179094 secs 2024-04-26T20:32:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31315.109579 secs 2024-04-26T20:32:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:55 smithi033 ceph-mon[27845]: pgmap v17010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:55.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:55 smithi096 ceph-mon[32753]: pgmap v17010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:55 smithi194 ceph-mon[33017]: pgmap v17010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:55.696+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:55.696+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:32:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:32:55.696+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:32:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:57 smithi033 ceph-mon[27845]: pgmap v17011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:57 smithi096 ceph-mon[32753]: pgmap v17011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:57 smithi194 ceph-mon[33017]: pgmap v17011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:32:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31317.179183 secs 2024-04-26T20:32:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:59 smithi033 ceph-mon[27845]: pgmap v17012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31320.109676 secs 2024-04-26T20:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31317.179183 secs 2024-04-26T20:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:59 smithi096 ceph-mon[32753]: pgmap v17012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31320.109676 secs 2024-04-26T20:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31317.179183 secs 2024-04-26T20:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:59 smithi194 ceph-mon[33017]: pgmap v17012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:32:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31320.109676 secs 2024-04-26T20:33:01.409 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:01 smithi194 ceph-mon[33017]: pgmap v17013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:01 smithi033 ceph-mon[27845]: pgmap v17013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:01 smithi096 ceph-mon[32753]: pgmap v17013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:03 smithi033 ceph-mon[27845]: pgmap v17014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:03 smithi096 ceph-mon[32753]: pgmap v17014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:03 smithi194 ceph-mon[33017]: pgmap v17014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31322.179264 secs 2024-04-26T20:33:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31325.109827 secs 2024-04-26T20:33:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31322.179264 secs 2024-04-26T20:33:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31325.109827 secs 2024-04-26T20:33:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31322.179264 secs 2024-04-26T20:33:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31325.109827 secs 2024-04-26T20:33:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:05 smithi033 ceph-mon[27845]: pgmap v17015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:33:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:05 smithi096 ceph-mon[32753]: pgmap v17015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:33:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:05 smithi194 ceph-mon[33017]: pgmap v17015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:05.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:33:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:33:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:33:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:06 smithi033 ceph-mon[27845]: pgmap v17016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:33:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:06 smithi033 ceph-mon[27845]: pgmap v17017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:33:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:33:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:33:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:06 smithi096 ceph-mon[32753]: pgmap v17016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:33:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:06 smithi096 ceph-mon[32753]: pgmap v17017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:33:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:33:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:33:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:06 smithi194 ceph-mon[33017]: pgmap v17016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:33:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:06 smithi194 ceph-mon[33017]: pgmap v17017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:33:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:08 smithi194 ceph-mon[33017]: pgmap v17018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31330.109946 secs 2024-04-26T20:33:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:08 smithi033 ceph-mon[27845]: pgmap v17018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31330.109946 secs 2024-04-26T20:33:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:08 smithi096 ceph-mon[32753]: pgmap v17018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31330.109946 secs 2024-04-26T20:33:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31327.179346 secs 2024-04-26T20:33:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:09 smithi194 ceph-mon[33017]: pgmap v17019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31327.179346 secs 2024-04-26T20:33:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:09 smithi033 ceph-mon[27845]: pgmap v17019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31327.179346 secs 2024-04-26T20:33:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:09 smithi096 ceph-mon[32753]: pgmap v17019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:10.710+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:10.710+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:10.710+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:33:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:12 smithi194 ceph-mon[33017]: pgmap v17020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:12 smithi033 ceph-mon[27845]: pgmap v17020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:12 smithi096 ceph-mon[32753]: pgmap v17020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:13 smithi194 ceph-mon[33017]: pgmap v17021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31335.110083 secs 2024-04-26T20:33:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:13 smithi033 ceph-mon[27845]: pgmap v17021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31335.110083 secs 2024-04-26T20:33:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:13 smithi096 ceph-mon[32753]: pgmap v17021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:33:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31335.110083 secs 2024-04-26T20:33:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31332.179427 secs 2024-04-26T20:33:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31332.179427 secs 2024-04-26T20:33:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31332.179427 secs 2024-04-26T20:33:16.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:15 smithi033 ceph-mon[27845]: pgmap v17022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:33:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:15 smithi194 ceph-mon[33017]: pgmap v17022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:33:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:15 smithi096 ceph-mon[32753]: pgmap v17022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:33:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:17 smithi194 ceph-mon[33017]: pgmap v17023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:17 smithi033 ceph-mon[27845]: pgmap v17023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:17 smithi096 ceph-mon[32753]: pgmap v17023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31340.110148 secs 2024-04-26T20:33:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31340.110148 secs 2024-04-26T20:33:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31340.110148 secs 2024-04-26T20:33:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31337.179529 secs 2024-04-26T20:33:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:19 smithi194 ceph-mon[33017]: pgmap v17024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31337.179529 secs 2024-04-26T20:33:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:19 smithi033 ceph-mon[27845]: pgmap v17024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31337.179529 secs 2024-04-26T20:33:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:19 smithi096 ceph-mon[32753]: pgmap v17024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:22.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:21 smithi194 ceph-mon[33017]: pgmap v17025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:21 smithi033 ceph-mon[27845]: pgmap v17025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:21 smithi096 ceph-mon[32753]: pgmap v17025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:23 smithi194 ceph-mon[33017]: pgmap v17026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31345.110243 secs 2024-04-26T20:33:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:23 smithi033 ceph-mon[27845]: pgmap v17026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31345.110243 secs 2024-04-26T20:33:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:23 smithi096 ceph-mon[32753]: pgmap v17026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31345.110243 secs 2024-04-26T20:33:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31342.179609 secs 2024-04-26T20:33:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31342.179609 secs 2024-04-26T20:33:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31342.179609 secs 2024-04-26T20:33:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:25.723+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:25.723+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:33:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:25.723+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:25 smithi194 ceph-mon[33017]: pgmap v17027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:25 smithi033 ceph-mon[27845]: pgmap v17027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:25 smithi096 ceph-mon[32753]: pgmap v17027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:27 smithi194 ceph-mon[33017]: pgmap v17028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:27 smithi033 ceph-mon[27845]: pgmap v17028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:27 smithi096 ceph-mon[32753]: pgmap v17028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31350.110359 secs 2024-04-26T20:33:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31350.110359 secs 2024-04-26T20:33:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31350.110359 secs 2024-04-26T20:33:30.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31347.179700 secs 2024-04-26T20:33:30.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:29 smithi033 ceph-mon[27845]: pgmap v17029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31347.179700 secs 2024-04-26T20:33:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:29 smithi194 ceph-mon[33017]: pgmap v17029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31347.179700 secs 2024-04-26T20:33:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:29 smithi096 ceph-mon[32753]: pgmap v17029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:31 smithi194 ceph-mon[33017]: pgmap v17030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:31 smithi033 ceph-mon[27845]: pgmap v17030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:31 smithi096 ceph-mon[32753]: pgmap v17030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:33 smithi194 ceph-mon[33017]: pgmap v17031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31355.110434 secs 2024-04-26T20:33:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:33 smithi033 ceph-mon[27845]: pgmap v17031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31355.110434 secs 2024-04-26T20:33:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:33 smithi096 ceph-mon[32753]: pgmap v17031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31355.110434 secs 2024-04-26T20:33:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31352.179786 secs 2024-04-26T20:33:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31352.179786 secs 2024-04-26T20:33:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31352.179786 secs 2024-04-26T20:33:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:35 smithi194 ceph-mon[33017]: pgmap v17032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:35 smithi033 ceph-mon[27845]: pgmap v17032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:35 smithi096 ceph-mon[32753]: pgmap v17032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:37 smithi194 ceph-mon[33017]: pgmap v17033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:37 smithi033 ceph-mon[27845]: pgmap v17033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:37 smithi096 ceph-mon[32753]: pgmap v17033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:39.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31360.110551 secs 2024-04-26T20:33:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31360.110551 secs 2024-04-26T20:33:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31360.110551 secs 2024-04-26T20:33:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31357.179871 secs 2024-04-26T20:33:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:39 smithi194 ceph-mon[33017]: pgmap v17034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31357.179871 secs 2024-04-26T20:33:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:39 smithi033 ceph-mon[27845]: pgmap v17034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31357.179871 secs 2024-04-26T20:33:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:39 smithi096 ceph-mon[32753]: pgmap v17034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:40.735+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:40.735+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:40.735+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:33:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:41 smithi194 ceph-mon[33017]: pgmap v17035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:41 smithi033 ceph-mon[27845]: pgmap v17035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:41 smithi096 ceph-mon[32753]: pgmap v17035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31365.110646 secs 2024-04-26T20:33:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:43 smithi194 ceph-mon[33017]: pgmap v17036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31365.110646 secs 2024-04-26T20:33:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:43 smithi033 ceph-mon[27845]: pgmap v17036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31365.110646 secs 2024-04-26T20:33:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:43 smithi096 ceph-mon[32753]: pgmap v17036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31362.179953 secs 2024-04-26T20:33:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31362.179953 secs 2024-04-26T20:33:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31362.179953 secs 2024-04-26T20:33:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:45 smithi194 ceph-mon[33017]: pgmap v17037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:45 smithi033 ceph-mon[27845]: pgmap v17037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:45 smithi096 ceph-mon[32753]: pgmap v17037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:47 smithi194 ceph-mon[33017]: pgmap v17038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:47 smithi033 ceph-mon[27845]: pgmap v17038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:47 smithi096 ceph-mon[32753]: pgmap v17038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31370.110764 secs 2024-04-26T20:33:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31370.110764 secs 2024-04-26T20:33:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31370.110764 secs 2024-04-26T20:33:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31367.180038 secs 2024-04-26T20:33:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:49 smithi194 ceph-mon[33017]: pgmap v17039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31367.180038 secs 2024-04-26T20:33:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:49 smithi033 ceph-mon[27845]: pgmap v17039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31367.180038 secs 2024-04-26T20:33:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:49 smithi096 ceph-mon[32753]: pgmap v17039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:51 smithi194 ceph-mon[33017]: pgmap v17040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:51 smithi033 ceph-mon[27845]: pgmap v17040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:51 smithi096 ceph-mon[32753]: pgmap v17040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31375.110882 secs 2024-04-26T20:33:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:53 smithi194 ceph-mon[33017]: pgmap v17041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31375.110882 secs 2024-04-26T20:33:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:53 smithi033 ceph-mon[27845]: pgmap v17041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31375.110882 secs 2024-04-26T20:33:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:53 smithi096 ceph-mon[32753]: pgmap v17041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31372.180117 secs 2024-04-26T20:33:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31372.180117 secs 2024-04-26T20:33:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31372.180117 secs 2024-04-26T20:33:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:55.751+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:55.751+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:33:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:33:55.751+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:33:56.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:55 smithi194 ceph-mon[33017]: pgmap v17042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:55 smithi033 ceph-mon[27845]: pgmap v17042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:55 smithi096 ceph-mon[32753]: pgmap v17042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:33:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:57 smithi194 ceph-mon[33017]: pgmap v17043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:57 smithi033 ceph-mon[27845]: pgmap v17043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:57 smithi096 ceph-mon[32753]: pgmap v17043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:33:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31380.111004 secs 2024-04-26T20:33:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31380.111004 secs 2024-04-26T20:33:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31380.111004 secs 2024-04-26T20:34:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31377.180203 secs 2024-04-26T20:34:00.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:33:59 smithi033 ceph-mon[27845]: pgmap v17044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31377.180203 secs 2024-04-26T20:34:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:33:59 smithi194 ceph-mon[33017]: pgmap v17044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31377.180203 secs 2024-04-26T20:34:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:33:59 smithi096 ceph-mon[32753]: pgmap v17044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:01 smithi194 ceph-mon[33017]: pgmap v17045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:01 smithi033 ceph-mon[27845]: pgmap v17045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:01 smithi096 ceph-mon[32753]: pgmap v17045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31385.111121 secs 2024-04-26T20:34:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:03 smithi194 ceph-mon[33017]: pgmap v17046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31385.111121 secs 2024-04-26T20:34:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:03 smithi033 ceph-mon[27845]: pgmap v17046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31385.111121 secs 2024-04-26T20:34:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:03 smithi096 ceph-mon[32753]: pgmap v17046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31382.180285 secs 2024-04-26T20:34:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31382.180285 secs 2024-04-26T20:34:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31382.180285 secs 2024-04-26T20:34:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:05 smithi194 ceph-mon[33017]: pgmap v17047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:05 smithi033 ceph-mon[27845]: pgmap v17047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:05 smithi096 ceph-mon[32753]: pgmap v17047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:34:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:34:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:34:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:08 smithi194 ceph-mon[33017]: pgmap v17048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:08 smithi033 ceph-mon[27845]: pgmap v17048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:08 smithi096 ceph-mon[32753]: pgmap v17048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31387.180369 secs 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: pgmap v17049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: pgmap v17050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31390.111251 secs 2024-04-26T20:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31387.180369 secs 2024-04-26T20:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: pgmap v17049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:34:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: pgmap v17050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:34:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31390.111251 secs 2024-04-26T20:34:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31387.180369 secs 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: pgmap v17049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: pgmap v17050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:34:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31390.111251 secs 2024-04-26T20:34:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:10.766+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:10.766+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:34:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:10.766+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:11 smithi194 ceph-mon[33017]: pgmap v17051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:11 smithi033 ceph-mon[27845]: pgmap v17051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:11 smithi096 ceph-mon[32753]: pgmap v17051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:13 smithi194 ceph-mon[33017]: pgmap v17052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:13 smithi033 ceph-mon[27845]: pgmap v17052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:13 smithi096 ceph-mon[32753]: pgmap v17052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31392.180429 secs 2024-04-26T20:34:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31395.111315 secs 2024-04-26T20:34:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31392.180429 secs 2024-04-26T20:34:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31395.111315 secs 2024-04-26T20:34:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31392.180429 secs 2024-04-26T20:34:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31395.111315 secs 2024-04-26T20:34:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:15 smithi033 ceph-mon[27845]: pgmap v17053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-26T20:34:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:15 smithi096 ceph-mon[32753]: pgmap v17053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-26T20:34:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:15 smithi194 ceph-mon[33017]: pgmap v17053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-26T20:34:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:17 smithi033 ceph-mon[27845]: pgmap v17054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:17 smithi096 ceph-mon[32753]: pgmap v17054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:17 smithi194 ceph-mon[33017]: pgmap v17054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T20:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31397.180516 secs 2024-04-26T20:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:19 smithi033 ceph-mon[27845]: pgmap v17055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31400.111476 secs 2024-04-26T20:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31397.180516 secs 2024-04-26T20:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:19 smithi096 ceph-mon[32753]: pgmap v17055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31400.111476 secs 2024-04-26T20:34:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31397.180516 secs 2024-04-26T20:34:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:19 smithi194 ceph-mon[33017]: pgmap v17055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:34:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31400.111476 secs 2024-04-26T20:34:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:21 smithi033 ceph-mon[27845]: pgmap v17056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:34:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:21 smithi096 ceph-mon[32753]: pgmap v17056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:34:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:21 smithi194 ceph-mon[33017]: pgmap v17056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:34:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:23 smithi033 ceph-mon[27845]: pgmap v17057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:23 smithi096 ceph-mon[32753]: pgmap v17057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:23 smithi194 ceph-mon[33017]: pgmap v17057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31402.180585 secs 2024-04-26T20:34:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31405.111554 secs 2024-04-26T20:34:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31402.180585 secs 2024-04-26T20:34:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31405.111554 secs 2024-04-26T20:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31402.180585 secs 2024-04-26T20:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31405.111554 secs 2024-04-26T20:34:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:25 smithi033 ceph-mon[27845]: pgmap v17058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:25 smithi096 ceph-mon[32753]: pgmap v17058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:25.770+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:25.770+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:34:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:25.770+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:25 smithi194 ceph-mon[33017]: pgmap v17058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:27 smithi033 ceph-mon[27845]: pgmap v17059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:27 smithi096 ceph-mon[32753]: pgmap v17059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:27 smithi194 ceph-mon[33017]: pgmap v17059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31407.180667 secs 2024-04-26T20:34:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:29 smithi033 ceph-mon[27845]: pgmap v17060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31410.111671 secs 2024-04-26T20:34:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31407.180667 secs 2024-04-26T20:34:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:29 smithi096 ceph-mon[32753]: pgmap v17060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31410.111671 secs 2024-04-26T20:34:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31407.180667 secs 2024-04-26T20:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:29 smithi194 ceph-mon[33017]: pgmap v17060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31410.111671 secs 2024-04-26T20:34:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:31 smithi033 ceph-mon[27845]: pgmap v17061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:31 smithi096 ceph-mon[32753]: pgmap v17061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:31 smithi194 ceph-mon[33017]: pgmap v17061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:33 smithi033 ceph-mon[27845]: pgmap v17062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:33 smithi096 ceph-mon[32753]: pgmap v17062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:33 smithi194 ceph-mon[33017]: pgmap v17062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31412.180752 secs 2024-04-26T20:34:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31415.111809 secs 2024-04-26T20:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31412.180752 secs 2024-04-26T20:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31415.111809 secs 2024-04-26T20:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31412.180752 secs 2024-04-26T20:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31415.111809 secs 2024-04-26T20:34:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:35 smithi096 ceph-mon[32753]: pgmap v17063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:35 smithi194 ceph-mon[33017]: pgmap v17063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:35 smithi033 ceph-mon[27845]: pgmap v17063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:37 smithi096 ceph-mon[32753]: pgmap v17064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:37 smithi194 ceph-mon[33017]: pgmap v17064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:37 smithi033 ceph-mon[27845]: pgmap v17064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31417.180837 secs 2024-04-26T20:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:39 smithi194 ceph-mon[33017]: pgmap v17065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31420.111926 secs 2024-04-26T20:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31417.180837 secs 2024-04-26T20:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:39 smithi033 ceph-mon[27845]: pgmap v17065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31420.111926 secs 2024-04-26T20:34:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31417.180837 secs 2024-04-26T20:34:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:39 smithi096 ceph-mon[32753]: pgmap v17065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31420.111926 secs 2024-04-26T20:34:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:40.782+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:40.782+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:40.782+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:34:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:41 smithi194 ceph-mon[33017]: pgmap v17066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:41 smithi033 ceph-mon[27845]: pgmap v17066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:41 smithi096 ceph-mon[32753]: pgmap v17066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:43 smithi194 ceph-mon[33017]: pgmap v17067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:43 smithi033 ceph-mon[27845]: pgmap v17067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:43 smithi096 ceph-mon[32753]: pgmap v17067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31422.180920 secs 2024-04-26T20:34:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31425.112039 secs 2024-04-26T20:34:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31422.180920 secs 2024-04-26T20:34:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31425.112039 secs 2024-04-26T20:34:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31422.180920 secs 2024-04-26T20:34:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31425.112039 secs 2024-04-26T20:34:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:45 smithi194 ceph-mon[33017]: pgmap v17068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:45 smithi033 ceph-mon[27845]: pgmap v17068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:45 smithi096 ceph-mon[32753]: pgmap v17068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:47 smithi194 ceph-mon[33017]: pgmap v17069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:47 smithi033 ceph-mon[27845]: pgmap v17069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:47 smithi096 ceph-mon[32753]: pgmap v17069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31427.181009 secs 2024-04-26T20:34:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:49 smithi033 ceph-mon[27845]: pgmap v17070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31430.112191 secs 2024-04-26T20:34:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31427.181009 secs 2024-04-26T20:34:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:49 smithi194 ceph-mon[33017]: pgmap v17070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31430.112191 secs 2024-04-26T20:34:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31427.181009 secs 2024-04-26T20:34:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:49 smithi096 ceph-mon[32753]: pgmap v17070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31430.112191 secs 2024-04-26T20:34:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:51 smithi194 ceph-mon[33017]: pgmap v17071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:51 smithi033 ceph-mon[27845]: pgmap v17071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:51 smithi096 ceph-mon[32753]: pgmap v17071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:53 smithi194 ceph-mon[33017]: pgmap v17072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:53 smithi033 ceph-mon[27845]: pgmap v17072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:53 smithi096 ceph-mon[32753]: pgmap v17072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31432.181066 secs 2024-04-26T20:34:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31435.112312 secs 2024-04-26T20:34:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31432.181066 secs 2024-04-26T20:34:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31435.112312 secs 2024-04-26T20:34:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31432.181066 secs 2024-04-26T20:34:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31435.112312 secs 2024-04-26T20:34:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:55.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:55.789+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:34:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:34:55.789+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:34:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:55 smithi194 ceph-mon[33017]: pgmap v17073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:55 smithi033 ceph-mon[27845]: pgmap v17073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:55 smithi096 ceph-mon[32753]: pgmap v17073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:34:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:57 smithi194 ceph-mon[33017]: pgmap v17074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:57 smithi033 ceph-mon[27845]: pgmap v17074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:57 smithi096 ceph-mon[32753]: pgmap v17074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31437.181127 secs 2024-04-26T20:34:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:59 smithi194 ceph-mon[33017]: pgmap v17075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31440.112388 secs 2024-04-26T20:34:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31437.181127 secs 2024-04-26T20:34:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:59 smithi033 ceph-mon[27845]: pgmap v17075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31440.112388 secs 2024-04-26T20:34:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31437.181127 secs 2024-04-26T20:34:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:59 smithi096 ceph-mon[32753]: pgmap v17075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:34:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31440.112388 secs 2024-04-26T20:35:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:01 smithi194 ceph-mon[33017]: pgmap v17076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:01 smithi033 ceph-mon[27845]: pgmap v17076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:01 smithi096 ceph-mon[32753]: pgmap v17076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:03 smithi194 ceph-mon[33017]: pgmap v17077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:03 smithi033 ceph-mon[27845]: pgmap v17077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:03 smithi096 ceph-mon[32753]: pgmap v17077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31442.181224 secs 2024-04-26T20:35:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31445.112497 secs 2024-04-26T20:35:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31442.181224 secs 2024-04-26T20:35:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31445.112497 secs 2024-04-26T20:35:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31442.181224 secs 2024-04-26T20:35:04.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31445.112497 secs 2024-04-26T20:35:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:05 smithi194 ceph-mon[33017]: pgmap v17078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:05 smithi033 ceph-mon[27845]: pgmap v17078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:05 smithi096 ceph-mon[32753]: pgmap v17078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:07 smithi194 ceph-mon[33017]: pgmap v17079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:07 smithi033 ceph-mon[27845]: pgmap v17079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:07 smithi096 ceph-mon[32753]: pgmap v17079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31447.181287 secs 2024-04-26T20:35:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:09 smithi194 ceph-mon[33017]: pgmap v17080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:35:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31450.112571 secs 2024-04-26T20:35:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31447.181287 secs 2024-04-26T20:35:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:09 smithi033 ceph-mon[27845]: pgmap v17080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:35:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31450.112571 secs 2024-04-26T20:35:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31447.181287 secs 2024-04-26T20:35:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:09 smithi096 ceph-mon[32753]: pgmap v17080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:35:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31450.112571 secs 2024-04-26T20:35:10.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:10.797+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:10.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:10.797+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:35:10.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:10.797+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:35:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:35:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:35:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:35:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:35:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:35:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:35:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:35:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:35:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:11 smithi194 ceph-mon[33017]: pgmap v17081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:35:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:11 smithi194 ceph-mon[33017]: pgmap v17082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:11 smithi033 ceph-mon[27845]: pgmap v17081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:35:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:11 smithi033 ceph-mon[27845]: pgmap v17082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:11 smithi096 ceph-mon[32753]: pgmap v17081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:35:11.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:11 smithi096 ceph-mon[32753]: pgmap v17082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:13 smithi194 ceph-mon[33017]: pgmap v17083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:13 smithi033 ceph-mon[27845]: pgmap v17083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:13 smithi096 ceph-mon[32753]: pgmap v17083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31452.181368 secs 2024-04-26T20:35:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31455.112713 secs 2024-04-26T20:35:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31452.181368 secs 2024-04-26T20:35:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31455.112713 secs 2024-04-26T20:35:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31452.181368 secs 2024-04-26T20:35:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31455.112713 secs 2024-04-26T20:35:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:15 smithi194 ceph-mon[33017]: pgmap v17084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:35:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:15 smithi033 ceph-mon[27845]: pgmap v17084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:35:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:15 smithi096 ceph-mon[32753]: pgmap v17084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:35:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:17 smithi194 ceph-mon[33017]: pgmap v17085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:17 smithi033 ceph-mon[27845]: pgmap v17085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:17 smithi096 ceph-mon[32753]: pgmap v17085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31457.181430 secs 2024-04-26T20:35:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:19 smithi194 ceph-mon[33017]: pgmap v17086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31460.112850 secs 2024-04-26T20:35:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31457.181430 secs 2024-04-26T20:35:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:19 smithi033 ceph-mon[27845]: pgmap v17086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31460.112850 secs 2024-04-26T20:35:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31457.181430 secs 2024-04-26T20:35:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:19 smithi096 ceph-mon[32753]: pgmap v17086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:35:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31460.112850 secs 2024-04-26T20:35:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:21 smithi194 ceph-mon[33017]: pgmap v17087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:35:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:21 smithi033 ceph-mon[27845]: pgmap v17087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:35:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:21 smithi096 ceph-mon[32753]: pgmap v17087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:35:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:23 smithi194 ceph-mon[33017]: pgmap v17088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:23 smithi033 ceph-mon[27845]: pgmap v17088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:23 smithi096 ceph-mon[32753]: pgmap v17088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31462.181527 secs 2024-04-26T20:35:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31465.112975 secs 2024-04-26T20:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31462.181527 secs 2024-04-26T20:35:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31465.112975 secs 2024-04-26T20:35:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31462.181527 secs 2024-04-26T20:35:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31465.112975 secs 2024-04-26T20:35:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:25.807+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:25.807+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:25.807+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:35:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:25 smithi194 ceph-mon[33017]: pgmap v17089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:25 smithi033 ceph-mon[27845]: pgmap v17089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:25 smithi096 ceph-mon[32753]: pgmap v17089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:27.149+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:35:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:27.149+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:35:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:27.150+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 247 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T20:35:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:27.150+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 247 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T20:35:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:27 smithi033 ceph-mon[27845]: pgmap v17090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:27 smithi096 ceph-mon[32753]: pgmap v17090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:27 smithi194 ceph-mon[33017]: pgmap v17090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31467.181631 secs 2024-04-26T20:35:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:29 smithi033 ceph-mon[27845]: pgmap v17091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31470.113086 secs 2024-04-26T20:35:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31467.181631 secs 2024-04-26T20:35:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:29 smithi096 ceph-mon[32753]: pgmap v17091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31470.113086 secs 2024-04-26T20:35:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31467.181631 secs 2024-04-26T20:35:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:29 smithi194 ceph-mon[33017]: pgmap v17091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:30.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31470.113086 secs 2024-04-26T20:35:31.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:31 smithi194 ceph-mon[33017]: pgmap v17092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:31 smithi033 ceph-mon[27845]: pgmap v17092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:31 smithi096 ceph-mon[32753]: pgmap v17092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:33 smithi033 ceph-mon[27845]: pgmap v17093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:33 smithi096 ceph-mon[32753]: pgmap v17093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:33 smithi194 ceph-mon[33017]: pgmap v17093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31472.181730 secs 2024-04-26T20:35:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31475.113208 secs 2024-04-26T20:35:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31472.181730 secs 2024-04-26T20:35:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31475.113208 secs 2024-04-26T20:35:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31472.181730 secs 2024-04-26T20:35:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31475.113208 secs 2024-04-26T20:35:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:35 smithi033 ceph-mon[27845]: pgmap v17094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:35 smithi096 ceph-mon[32753]: pgmap v17094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:35 smithi194 ceph-mon[33017]: pgmap v17094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:37 smithi033 ceph-mon[27845]: pgmap v17095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:37 smithi096 ceph-mon[32753]: pgmap v17095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:37 smithi194 ceph-mon[33017]: pgmap v17095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31477.181833 secs 2024-04-26T20:35:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:39 smithi033 ceph-mon[27845]: pgmap v17096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31480.113274 secs 2024-04-26T20:35:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31477.181833 secs 2024-04-26T20:35:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:39 smithi096 ceph-mon[32753]: pgmap v17096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31480.113274 secs 2024-04-26T20:35:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31477.181833 secs 2024-04-26T20:35:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:39 smithi194 ceph-mon[33017]: pgmap v17096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31480.113274 secs 2024-04-26T20:35:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:40.821+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:40.822+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:40.822+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:35:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:41 smithi033 ceph-mon[27845]: pgmap v17097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:41 smithi096 ceph-mon[32753]: pgmap v17097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:41 smithi194 ceph-mon[33017]: pgmap v17097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:43 smithi033 ceph-mon[27845]: pgmap v17098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:43 smithi096 ceph-mon[32753]: pgmap v17098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:43 smithi194 ceph-mon[33017]: pgmap v17098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31482.181932 secs 2024-04-26T20:35:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31485.113402 secs 2024-04-26T20:35:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31482.181932 secs 2024-04-26T20:35:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31485.113402 secs 2024-04-26T20:35:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31482.181932 secs 2024-04-26T20:35:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31485.113402 secs 2024-04-26T20:35:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:45 smithi033 ceph-mon[27845]: pgmap v17099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:45 smithi096 ceph-mon[32753]: pgmap v17099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:45 smithi194 ceph-mon[33017]: pgmap v17099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:47 smithi033 ceph-mon[27845]: pgmap v17100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:47 smithi096 ceph-mon[32753]: pgmap v17100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:47 smithi194 ceph-mon[33017]: pgmap v17100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31487.182005 secs 2024-04-26T20:35:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:49 smithi096 ceph-mon[32753]: pgmap v17101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31490.113489 secs 2024-04-26T20:35:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31487.182005 secs 2024-04-26T20:35:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:49 smithi194 ceph-mon[33017]: pgmap v17101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31490.113489 secs 2024-04-26T20:35:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31487.182005 secs 2024-04-26T20:35:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:49 smithi033 ceph-mon[27845]: pgmap v17101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31490.113489 secs 2024-04-26T20:35:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:51 smithi096 ceph-mon[32753]: pgmap v17102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:51 smithi194 ceph-mon[33017]: pgmap v17102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:51 smithi033 ceph-mon[27845]: pgmap v17102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:53 smithi194 ceph-mon[33017]: pgmap v17103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:53 smithi033 ceph-mon[27845]: pgmap v17103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:53 smithi096 ceph-mon[32753]: pgmap v17103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31492.182087 secs 2024-04-26T20:35:55.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31495.113579 secs 2024-04-26T20:35:55.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31492.182087 secs 2024-04-26T20:35:55.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31495.113579 secs 2024-04-26T20:35:55.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31492.182087 secs 2024-04-26T20:35:55.273 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31495.113579 secs 2024-04-26T20:35:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:55.826+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:55.826+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:35:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:35:55.826+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:35:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:55 smithi194 ceph-mon[33017]: pgmap v17104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:55 smithi033 ceph-mon[27845]: pgmap v17104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:55 smithi096 ceph-mon[32753]: pgmap v17104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:35:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:57 smithi194 ceph-mon[33017]: pgmap v17105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:57 smithi033 ceph-mon[27845]: pgmap v17105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:35:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:57 smithi096 ceph-mon[32753]: pgmap v17105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31497.182171 secs 2024-04-26T20:36:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:59 smithi194 ceph-mon[33017]: pgmap v17106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31500.113730 secs 2024-04-26T20:36:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31497.182171 secs 2024-04-26T20:36:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:59 smithi033 ceph-mon[27845]: pgmap v17106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31500.113730 secs 2024-04-26T20:36:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31497.182171 secs 2024-04-26T20:36:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:59 smithi096 ceph-mon[32753]: pgmap v17106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31500.113730 secs 2024-04-26T20:36:01.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:01 smithi033 ceph-mon[27845]: pgmap v17107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:01.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:01 smithi096 ceph-mon[32753]: pgmap v17107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:01 smithi194 ceph-mon[33017]: pgmap v17107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:03 smithi194 ceph-mon[33017]: pgmap v17108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31505.113876 secs 2024-04-26T20:36:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:03 smithi033 ceph-mon[27845]: pgmap v17108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31505.113876 secs 2024-04-26T20:36:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:03 smithi096 ceph-mon[32753]: pgmap v17108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31505.113876 secs 2024-04-26T20:36:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31502.182257 secs 2024-04-26T20:36:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31502.182257 secs 2024-04-26T20:36:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31502.182257 secs 2024-04-26T20:36:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:05 smithi194 ceph-mon[33017]: pgmap v17109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:05 smithi033 ceph-mon[27845]: pgmap v17109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:05 smithi096 ceph-mon[32753]: pgmap v17109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:07 smithi194 ceph-mon[33017]: pgmap v17110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:07 smithi033 ceph-mon[27845]: pgmap v17110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:07 smithi096 ceph-mon[32753]: pgmap v17110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31510.114004 secs 2024-04-26T20:36:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31510.114004 secs 2024-04-26T20:36:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31510.114004 secs 2024-04-26T20:36:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31507.182357 secs 2024-04-26T20:36:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:09 smithi194 ceph-mon[33017]: pgmap v17111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31507.182357 secs 2024-04-26T20:36:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:09 smithi033 ceph-mon[27845]: pgmap v17111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31507.182357 secs 2024-04-26T20:36:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:09 smithi096 ceph-mon[32753]: pgmap v17111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:10.841+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:10.841+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:10.841+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:36:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:36:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:36:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:36:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:11 smithi194 ceph-mon[33017]: pgmap v17112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:11 smithi033 ceph-mon[27845]: pgmap v17112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:11 smithi096 ceph-mon[32753]: pgmap v17112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:36:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:36:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:12 smithi194 ceph-mon[33017]: pgmap v17113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:12 smithi194 ceph-mon[33017]: pgmap v17114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:36:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:36:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:36:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:12 smithi033 ceph-mon[27845]: pgmap v17113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:12 smithi033 ceph-mon[27845]: pgmap v17114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:13.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:36:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:36:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:36:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:12 smithi096 ceph-mon[32753]: pgmap v17113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:12 smithi096 ceph-mon[32753]: pgmap v17114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:13.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:36:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31515.114115 secs 2024-04-26T20:36:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31515.114115 secs 2024-04-26T20:36:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31515.114115 secs 2024-04-26T20:36:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31512.182456 secs 2024-04-26T20:36:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:14 smithi194 ceph-mon[33017]: pgmap v17115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31512.182456 secs 2024-04-26T20:36:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:14 smithi033 ceph-mon[27845]: pgmap v17115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31512.182456 secs 2024-04-26T20:36:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:14 smithi096 ceph-mon[32753]: pgmap v17115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:16 smithi033 ceph-mon[27845]: pgmap v17116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:16 smithi096 ceph-mon[32753]: pgmap v17116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:16 smithi194 ceph-mon[33017]: pgmap v17116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:18 smithi033 ceph-mon[27845]: pgmap v17117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31520.114223 secs 2024-04-26T20:36:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:18 smithi096 ceph-mon[32753]: pgmap v17117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31520.114223 secs 2024-04-26T20:36:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:18 smithi194 ceph-mon[33017]: pgmap v17117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31520.114223 secs 2024-04-26T20:36:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31517.182558 secs 2024-04-26T20:36:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31517.182558 secs 2024-04-26T20:36:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31517.182558 secs 2024-04-26T20:36:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:20 smithi033 ceph-mon[27845]: pgmap v17118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:20 smithi096 ceph-mon[32753]: pgmap v17118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:20 smithi194 ceph-mon[33017]: pgmap v17118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:36:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:22 smithi033 ceph-mon[27845]: pgmap v17119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:36:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:22 smithi096 ceph-mon[32753]: pgmap v17119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:36:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:22 smithi194 ceph-mon[33017]: pgmap v17119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31522.182656 secs 2024-04-26T20:36:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31525.114288 secs 2024-04-26T20:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31522.182656 secs 2024-04-26T20:36:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31525.114288 secs 2024-04-26T20:36:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31522.182656 secs 2024-04-26T20:36:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31525.114288 secs 2024-04-26T20:36:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:24 smithi033 ceph-mon[27845]: pgmap v17120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:24 smithi096 ceph-mon[32753]: pgmap v17120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:25.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:24 smithi194 ceph-mon[33017]: pgmap v17120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:25.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:25.845+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:25.845+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:25.845+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:36:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:26 smithi033 ceph-mon[27845]: pgmap v17121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:26 smithi096 ceph-mon[32753]: pgmap v17121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:26 smithi194 ceph-mon[33017]: pgmap v17121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:28 smithi033 ceph-mon[27845]: pgmap v17122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31527.182740 secs 2024-04-26T20:36:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31530.114412 secs 2024-04-26T20:36:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:28 smithi096 ceph-mon[32753]: pgmap v17122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31527.182740 secs 2024-04-26T20:36:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31530.114412 secs 2024-04-26T20:36:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:28 smithi194 ceph-mon[33017]: pgmap v17122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31527.182740 secs 2024-04-26T20:36:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31530.114412 secs 2024-04-26T20:36:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:30 smithi033 ceph-mon[27845]: pgmap v17123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:30 smithi096 ceph-mon[32753]: pgmap v17123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:30 smithi194 ceph-mon[33017]: pgmap v17123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:32 smithi033 ceph-mon[27845]: pgmap v17124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:32 smithi096 ceph-mon[32753]: pgmap v17124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:32 smithi194 ceph-mon[33017]: pgmap v17124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31532.182848 secs 2024-04-26T20:36:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31535.114532 secs 2024-04-26T20:36:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31532.182848 secs 2024-04-26T20:36:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31535.114532 secs 2024-04-26T20:36:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31532.182848 secs 2024-04-26T20:36:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31535.114532 secs 2024-04-26T20:36:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:34 smithi096 ceph-mon[32753]: pgmap v17125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:34 smithi194 ceph-mon[33017]: pgmap v17125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:34 smithi033 ceph-mon[27845]: pgmap v17125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:36 smithi096 ceph-mon[32753]: pgmap v17126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:36 smithi194 ceph-mon[33017]: pgmap v17126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:36 smithi033 ceph-mon[27845]: pgmap v17126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:38 smithi194 ceph-mon[33017]: pgmap v17127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31537.182950 secs 2024-04-26T20:36:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31540.114652 secs 2024-04-26T20:36:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:38 smithi033 ceph-mon[27845]: pgmap v17127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31537.182950 secs 2024-04-26T20:36:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31540.114652 secs 2024-04-26T20:36:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:38 smithi096 ceph-mon[32753]: pgmap v17127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31537.182950 secs 2024-04-26T20:36:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31540.114652 secs 2024-04-26T20:36:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:40.851+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:40.851+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:36:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:40.851+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:40 smithi194 ceph-mon[33017]: pgmap v17128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:40 smithi033 ceph-mon[27845]: pgmap v17128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:40 smithi096 ceph-mon[32753]: pgmap v17128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:42 smithi194 ceph-mon[33017]: pgmap v17129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:42 smithi033 ceph-mon[27845]: pgmap v17129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:42 smithi096 ceph-mon[32753]: pgmap v17129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31542.183048 secs 2024-04-26T20:36:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31545.114767 secs 2024-04-26T20:36:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31542.183048 secs 2024-04-26T20:36:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31545.114767 secs 2024-04-26T20:36:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31542.183048 secs 2024-04-26T20:36:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31545.114767 secs 2024-04-26T20:36:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:44 smithi194 ceph-mon[33017]: pgmap v17130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:44 smithi033 ceph-mon[27845]: pgmap v17130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:44 smithi096 ceph-mon[32753]: pgmap v17130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:46 smithi194 ceph-mon[33017]: pgmap v17131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:46 smithi033 ceph-mon[27845]: pgmap v17131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:46 smithi096 ceph-mon[32753]: pgmap v17131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:48 smithi194 ceph-mon[33017]: pgmap v17132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31547.183135 secs 2024-04-26T20:36:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31550.114880 secs 2024-04-26T20:36:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:48 smithi033 ceph-mon[27845]: pgmap v17132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31547.183135 secs 2024-04-26T20:36:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31550.114880 secs 2024-04-26T20:36:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:48 smithi096 ceph-mon[32753]: pgmap v17132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31547.183135 secs 2024-04-26T20:36:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31550.114880 secs 2024-04-26T20:36:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:50 smithi194 ceph-mon[33017]: pgmap v17133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:50 smithi033 ceph-mon[27845]: pgmap v17133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:50 smithi096 ceph-mon[32753]: pgmap v17133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:52 smithi194 ceph-mon[33017]: pgmap v17134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:52 smithi033 ceph-mon[27845]: pgmap v17134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:52 smithi096 ceph-mon[32753]: pgmap v17134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31552.183221 secs 2024-04-26T20:36:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31555.114993 secs 2024-04-26T20:36:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31552.183221 secs 2024-04-26T20:36:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31555.114993 secs 2024-04-26T20:36:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31552.183221 secs 2024-04-26T20:36:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31555.114993 secs 2024-04-26T20:36:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:54 smithi194 ceph-mon[33017]: pgmap v17135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:54 smithi033 ceph-mon[27845]: pgmap v17135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:54 smithi096 ceph-mon[32753]: pgmap v17135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:55.860+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:55.860+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:36:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:36:55.860+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:36:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:57 smithi194 ceph-mon[33017]: pgmap v17136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:57 smithi033 ceph-mon[27845]: pgmap v17136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:57 smithi096 ceph-mon[32753]: pgmap v17136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:36:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:59 smithi194 ceph-mon[33017]: pgmap v17137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31557.183304 secs 2024-04-26T20:36:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:36:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31560.115055 secs 2024-04-26T20:36:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:59 smithi033 ceph-mon[27845]: pgmap v17137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31557.183304 secs 2024-04-26T20:36:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:36:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31560.115055 secs 2024-04-26T20:36:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:59 smithi096 ceph-mon[32753]: pgmap v17137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:36:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31557.183304 secs 2024-04-26T20:36:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:36:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31560.115055 secs 2024-04-26T20:37:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:01 smithi194 ceph-mon[33017]: pgmap v17138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:01 smithi033 ceph-mon[27845]: pgmap v17138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:01 smithi096 ceph-mon[32753]: pgmap v17138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:03 smithi194 ceph-mon[33017]: pgmap v17139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:03 smithi033 ceph-mon[27845]: pgmap v17139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:03 smithi096 ceph-mon[32753]: pgmap v17139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31562.183401 secs 2024-04-26T20:37:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31565.115182 secs 2024-04-26T20:37:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31562.183401 secs 2024-04-26T20:37:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31565.115182 secs 2024-04-26T20:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31562.183401 secs 2024-04-26T20:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31565.115182 secs 2024-04-26T20:37:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:05 smithi194 ceph-mon[33017]: pgmap v17140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:05 smithi033 ceph-mon[27845]: pgmap v17140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:05 smithi096 ceph-mon[32753]: pgmap v17140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:07 smithi194 ceph-mon[33017]: pgmap v17141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:07 smithi033 ceph-mon[27845]: pgmap v17141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:07 smithi096 ceph-mon[32753]: pgmap v17141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:09 smithi194 ceph-mon[33017]: pgmap v17142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31567.183501 secs 2024-04-26T20:37:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31570.115272 secs 2024-04-26T20:37:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:09 smithi033 ceph-mon[27845]: pgmap v17142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31567.183501 secs 2024-04-26T20:37:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31570.115272 secs 2024-04-26T20:37:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:09 smithi096 ceph-mon[32753]: pgmap v17142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31567.183501 secs 2024-04-26T20:37:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31570.115272 secs 2024-04-26T20:37:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:10.875+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:10.875+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:37:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:10.876+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:11 smithi194 ceph-mon[33017]: pgmap v17143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:11 smithi033 ceph-mon[27845]: pgmap v17143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:11 smithi096 ceph-mon[32753]: pgmap v17143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:13 smithi194 ceph-mon[33017]: pgmap v17144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:37:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:13 smithi033 ceph-mon[27845]: pgmap v17144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:37:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:13 smithi096 ceph-mon[32753]: pgmap v17144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:37:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31572.183602 secs 2024-04-26T20:37:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:37:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:37:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:14 smithi194 ceph-mon[33017]: pgmap v17145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:14 smithi194 ceph-mon[33017]: pgmap v17146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:37:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31575.115401 secs 2024-04-26T20:37:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31572.183602 secs 2024-04-26T20:37:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:37:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:37:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:14 smithi033 ceph-mon[27845]: pgmap v17145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:14 smithi033 ceph-mon[27845]: pgmap v17146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:37:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31575.115401 secs 2024-04-26T20:37:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31572.183602 secs 2024-04-26T20:37:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:37:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:37:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:14 smithi096 ceph-mon[32753]: pgmap v17145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:14 smithi096 ceph-mon[32753]: pgmap v17146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:37:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31575.115401 secs 2024-04-26T20:37:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:16 smithi194 ceph-mon[33017]: pgmap v17147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:16 smithi033 ceph-mon[27845]: pgmap v17147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:16 smithi096 ceph-mon[32753]: pgmap v17147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:18 smithi194 ceph-mon[33017]: pgmap v17148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:18 smithi033 ceph-mon[27845]: pgmap v17148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:18 smithi096 ceph-mon[32753]: pgmap v17148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31577.183704 secs 2024-04-26T20:37:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31580.115522 secs 2024-04-26T20:37:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31577.183704 secs 2024-04-26T20:37:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31580.115522 secs 2024-04-26T20:37:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31577.183704 secs 2024-04-26T20:37:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31580.115522 secs 2024-04-26T20:37:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:20 smithi194 ceph-mon[33017]: pgmap v17149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:20 smithi033 ceph-mon[27845]: pgmap v17149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:20 smithi096 ceph-mon[32753]: pgmap v17149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:22 smithi194 ceph-mon[33017]: pgmap v17150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:22 smithi033 ceph-mon[27845]: pgmap v17150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:22 smithi096 ceph-mon[32753]: pgmap v17150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31582.183803 secs 2024-04-26T20:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:24 smithi194 ceph-mon[33017]: pgmap v17151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:37:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31585.115603 secs 2024-04-26T20:37:24.912 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31582.183803 secs 2024-04-26T20:37:24.912 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:24 smithi033 ceph-mon[27845]: pgmap v17151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:37:24.912 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31585.115603 secs 2024-04-26T20:37:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31582.183803 secs 2024-04-26T20:37:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:24 smithi096 ceph-mon[32753]: pgmap v17151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:37:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31585.115603 secs 2024-04-26T20:37:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:25.890+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:25.890+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:37:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:25.890+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:26 smithi033 ceph-mon[27845]: pgmap v17152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:26 smithi096 ceph-mon[32753]: pgmap v17152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:26 smithi194 ceph-mon[33017]: pgmap v17152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:28 smithi033 ceph-mon[27845]: pgmap v17153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:28 smithi096 ceph-mon[32753]: pgmap v17153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:28 smithi194 ceph-mon[33017]: pgmap v17153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31587.183904 secs 2024-04-26T20:37:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31590.115755 secs 2024-04-26T20:37:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31587.183904 secs 2024-04-26T20:37:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31590.115755 secs 2024-04-26T20:37:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31587.183904 secs 2024-04-26T20:37:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31590.115755 secs 2024-04-26T20:37:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:30 smithi033 ceph-mon[27845]: pgmap v17154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:30 smithi096 ceph-mon[32753]: pgmap v17154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:31.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:30 smithi194 ceph-mon[33017]: pgmap v17154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:32 smithi033 ceph-mon[27845]: pgmap v17155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:32 smithi096 ceph-mon[32753]: pgmap v17155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:32 smithi194 ceph-mon[33017]: pgmap v17155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31592.184004 secs 2024-04-26T20:37:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:34 smithi033 ceph-mon[27845]: pgmap v17156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31595.115854 secs 2024-04-26T20:37:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31592.184004 secs 2024-04-26T20:37:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:34 smithi096 ceph-mon[32753]: pgmap v17156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31595.115854 secs 2024-04-26T20:37:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31592.184004 secs 2024-04-26T20:37:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:34 smithi194 ceph-mon[33017]: pgmap v17156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31595.115854 secs 2024-04-26T20:37:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:36 smithi033 ceph-mon[27845]: pgmap v17157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:36 smithi096 ceph-mon[32753]: pgmap v17157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:36 smithi194 ceph-mon[33017]: pgmap v17157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:38 smithi033 ceph-mon[27845]: pgmap v17158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:38 smithi096 ceph-mon[32753]: pgmap v17158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:38 smithi194 ceph-mon[33017]: pgmap v17158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31597.184088 secs 2024-04-26T20:37:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31600.115975 secs 2024-04-26T20:37:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31597.184088 secs 2024-04-26T20:37:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31600.115975 secs 2024-04-26T20:37:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31597.184088 secs 2024-04-26T20:37:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31600.115975 secs 2024-04-26T20:37:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:40.896+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:40.896+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:40.896+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:37:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:40 smithi033 ceph-mon[27845]: pgmap v17159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:40 smithi096 ceph-mon[32753]: pgmap v17159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:40 smithi194 ceph-mon[33017]: pgmap v17159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:42.173+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:37:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:42.173+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:37:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:42.174+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 248 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:37:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:42.174+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 248 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:37:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:42 smithi033 ceph-mon[27845]: pgmap v17160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:42 smithi096 ceph-mon[32753]: pgmap v17160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:43.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:42 smithi194 ceph-mon[33017]: pgmap v17160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31602.184148 secs 2024-04-26T20:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:44 smithi096 ceph-mon[32753]: pgmap v17161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:44.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31605.116135 secs 2024-04-26T20:37:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31602.184148 secs 2024-04-26T20:37:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:44 smithi194 ceph-mon[33017]: pgmap v17161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31605.116135 secs 2024-04-26T20:37:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31602.184148 secs 2024-04-26T20:37:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:44 smithi033 ceph-mon[27845]: pgmap v17161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31605.116135 secs 2024-04-26T20:37:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:46 smithi096 ceph-mon[32753]: pgmap v17162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:46 smithi194 ceph-mon[33017]: pgmap v17162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:46 smithi033 ceph-mon[27845]: pgmap v17162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:48 smithi194 ceph-mon[33017]: pgmap v17163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:48 smithi033 ceph-mon[27845]: pgmap v17163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:48 smithi096 ceph-mon[32753]: pgmap v17163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31607.184210 secs 2024-04-26T20:37:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31610.116215 secs 2024-04-26T20:37:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31607.184210 secs 2024-04-26T20:37:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31610.116215 secs 2024-04-26T20:37:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31607.184210 secs 2024-04-26T20:37:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31610.116215 secs 2024-04-26T20:37:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:50 smithi194 ceph-mon[33017]: pgmap v17164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:50 smithi033 ceph-mon[27845]: pgmap v17164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:50 smithi096 ceph-mon[32753]: pgmap v17164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:52 smithi194 ceph-mon[33017]: pgmap v17165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:52 smithi033 ceph-mon[27845]: pgmap v17165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:52 smithi096 ceph-mon[32753]: pgmap v17165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31612.184294 secs 2024-04-26T20:37:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:54 smithi194 ceph-mon[33017]: pgmap v17166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31615.116330 secs 2024-04-26T20:37:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31612.184294 secs 2024-04-26T20:37:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:54 smithi033 ceph-mon[27845]: pgmap v17166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31615.116330 secs 2024-04-26T20:37:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31612.184294 secs 2024-04-26T20:37:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:54 smithi096 ceph-mon[32753]: pgmap v17166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31615.116330 secs 2024-04-26T20:37:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:55.903+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:55.903+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:37:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:37:55.903+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:37:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:56 smithi194 ceph-mon[33017]: pgmap v17167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:56 smithi033 ceph-mon[27845]: pgmap v17167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:56 smithi096 ceph-mon[32753]: pgmap v17167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:37:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:58 smithi194 ceph-mon[33017]: pgmap v17168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:58 smithi033 ceph-mon[27845]: pgmap v17168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:37:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:58 smithi096 ceph-mon[32753]: pgmap v17168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31617.184395 secs 2024-04-26T20:38:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31620.116445 secs 2024-04-26T20:38:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31617.184395 secs 2024-04-26T20:38:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31620.116445 secs 2024-04-26T20:38:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31617.184395 secs 2024-04-26T20:38:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31620.116445 secs 2024-04-26T20:38:01.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:00 smithi194 ceph-mon[33017]: pgmap v17169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:00 smithi033 ceph-mon[27845]: pgmap v17169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:00 smithi096 ceph-mon[32753]: pgmap v17169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:02 smithi194 ceph-mon[33017]: pgmap v17170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:03.276 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:02 smithi096 ceph-mon[32753]: pgmap v17170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:03.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:02 smithi033 ceph-mon[27845]: pgmap v17170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:04.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31625.116511 secs 2024-04-26T20:38:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31625.116511 secs 2024-04-26T20:38:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31625.116511 secs 2024-04-26T20:38:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31622.184480 secs 2024-04-26T20:38:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:04 smithi194 ceph-mon[33017]: pgmap v17171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31622.184480 secs 2024-04-26T20:38:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:04 smithi033 ceph-mon[27845]: pgmap v17171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31622.184480 secs 2024-04-26T20:38:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:04 smithi096 ceph-mon[32753]: pgmap v17171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:06 smithi194 ceph-mon[33017]: pgmap v17172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:06 smithi033 ceph-mon[27845]: pgmap v17172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:06 smithi096 ceph-mon[32753]: pgmap v17172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:08 smithi194 ceph-mon[33017]: pgmap v17173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31630.116630 secs 2024-04-26T20:38:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:08 smithi033 ceph-mon[27845]: pgmap v17173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31630.116630 secs 2024-04-26T20:38:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:08 smithi096 ceph-mon[32753]: pgmap v17173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31630.116630 secs 2024-04-26T20:38:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31627.184581 secs 2024-04-26T20:38:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31627.184581 secs 2024-04-26T20:38:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31627.184581 secs 2024-04-26T20:38:10.910 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:10.910+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:10.910+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:10.910+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:38:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:10 smithi194 ceph-mon[33017]: pgmap v17174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:10 smithi033 ceph-mon[27845]: pgmap v17174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:10 smithi096 ceph-mon[32753]: pgmap v17174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:12 smithi033 ceph-mon[27845]: pgmap v17175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:12 smithi194 ceph-mon[33017]: pgmap v17175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:12 smithi096 ceph-mon[32753]: pgmap v17175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31635.116728 secs 2024-04-26T20:38:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31635.116728 secs 2024-04-26T20:38:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31635.116728 secs 2024-04-26T20:38:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31632.184667 secs 2024-04-26T20:38:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:14 smithi194 ceph-mon[33017]: pgmap v17176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:38:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31632.184667 secs 2024-04-26T20:38:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:14 smithi033 ceph-mon[27845]: pgmap v17176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:38:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31632.184667 secs 2024-04-26T20:38:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:14 smithi096 ceph-mon[32753]: pgmap v17176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:38:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:38:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:38:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:15 smithi194 ceph-mon[33017]: pgmap v17177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:38:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:15 smithi194 ceph-mon[33017]: pgmap v17178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:38:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:38:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:38:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:15 smithi033 ceph-mon[27845]: pgmap v17177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:38:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:15 smithi033 ceph-mon[27845]: pgmap v17178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:38:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:38:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:38:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:15 smithi096 ceph-mon[32753]: pgmap v17177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:38:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:15 smithi096 ceph-mon[32753]: pgmap v17178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:16.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:38:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:18 smithi194 ceph-mon[33017]: pgmap v17179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:18 smithi033 ceph-mon[27845]: pgmap v17179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:18 smithi096 ceph-mon[32753]: pgmap v17179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31637.184768 secs 2024-04-26T20:38:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31640.116868 secs 2024-04-26T20:38:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31637.184768 secs 2024-04-26T20:38:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31640.116868 secs 2024-04-26T20:38:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31637.184768 secs 2024-04-26T20:38:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31640.116868 secs 2024-04-26T20:38:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:20 smithi194 ceph-mon[33017]: pgmap v17180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:38:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:20 smithi033 ceph-mon[27845]: pgmap v17180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:38:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:20 smithi096 ceph-mon[32753]: pgmap v17180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:38:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:22 smithi194 ceph-mon[33017]: pgmap v17181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:22 smithi033 ceph-mon[27845]: pgmap v17181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:22 smithi096 ceph-mon[32753]: pgmap v17181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31642.184864 secs 2024-04-26T20:38:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:24 smithi033 ceph-mon[27845]: pgmap v17182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31645.117005 secs 2024-04-26T20:38:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31642.184864 secs 2024-04-26T20:38:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:24 smithi194 ceph-mon[33017]: pgmap v17182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31645.117005 secs 2024-04-26T20:38:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31642.184864 secs 2024-04-26T20:38:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:24 smithi096 ceph-mon[32753]: pgmap v17182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:38:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31645.117005 secs 2024-04-26T20:38:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:25.925+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:25.925+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:25.925+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:38:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:26 smithi194 ceph-mon[33017]: pgmap v17183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:38:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:26 smithi033 ceph-mon[27845]: pgmap v17183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:38:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:26 smithi096 ceph-mon[32753]: pgmap v17183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:38:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:27.181+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:38:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:27.181+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:38:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:27.182+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 249 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T20:38:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:27.182+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 249 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T20:38:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:28 smithi194 ceph-mon[33017]: pgmap v17184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:28 smithi033 ceph-mon[27845]: pgmap v17184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:28 smithi096 ceph-mon[32753]: pgmap v17184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31647.184974 secs 2024-04-26T20:38:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31650.117134 secs 2024-04-26T20:38:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31647.184974 secs 2024-04-26T20:38:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31650.117134 secs 2024-04-26T20:38:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31647.184974 secs 2024-04-26T20:38:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31650.117134 secs 2024-04-26T20:38:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:30 smithi194 ceph-mon[33017]: pgmap v17185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:30 smithi033 ceph-mon[27845]: pgmap v17185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:30 smithi096 ceph-mon[32753]: pgmap v17185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:32 smithi194 ceph-mon[33017]: pgmap v17186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:32 smithi033 ceph-mon[27845]: pgmap v17186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:32 smithi096 ceph-mon[32753]: pgmap v17186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31652.185058 secs 2024-04-26T20:38:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:34 smithi194 ceph-mon[33017]: pgmap v17187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31655.117246 secs 2024-04-26T20:38:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31652.185058 secs 2024-04-26T20:38:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:34 smithi033 ceph-mon[27845]: pgmap v17187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31655.117246 secs 2024-04-26T20:38:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31652.185058 secs 2024-04-26T20:38:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:34 smithi096 ceph-mon[32753]: pgmap v17187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31655.117246 secs 2024-04-26T20:38:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:36 smithi194 ceph-mon[33017]: pgmap v17188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:36 smithi033 ceph-mon[27845]: pgmap v17188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:36 smithi096 ceph-mon[32753]: pgmap v17188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:38 smithi194 ceph-mon[33017]: pgmap v17189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:38 smithi033 ceph-mon[27845]: pgmap v17189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:38 smithi096 ceph-mon[32753]: pgmap v17189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31657.185139 secs 2024-04-26T20:38:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31660.117322 secs 2024-04-26T20:38:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31657.185139 secs 2024-04-26T20:38:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31660.117322 secs 2024-04-26T20:38:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31657.185139 secs 2024-04-26T20:38:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31660.117322 secs 2024-04-26T20:38:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:40 smithi194 ceph-mon[33017]: pgmap v17190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:40 smithi033 ceph-mon[27845]: pgmap v17190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:40 smithi096 ceph-mon[32753]: pgmap v17190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:40.938+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:40.938+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:40.938+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:38:42.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:42 smithi033 ceph-mon[27845]: pgmap v17191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:42 smithi194 ceph-mon[33017]: pgmap v17191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:42 smithi096 ceph-mon[32753]: pgmap v17191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31662.185219 secs 2024-04-26T20:38:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:44 smithi194 ceph-mon[33017]: pgmap v17192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31665.117388 secs 2024-04-26T20:38:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31662.185219 secs 2024-04-26T20:38:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:44 smithi033 ceph-mon[27845]: pgmap v17192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31665.117388 secs 2024-04-26T20:38:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31662.185219 secs 2024-04-26T20:38:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:44 smithi096 ceph-mon[32753]: pgmap v17192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31665.117388 secs 2024-04-26T20:38:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:46 smithi194 ceph-mon[33017]: pgmap v17193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:46 smithi033 ceph-mon[27845]: pgmap v17193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:46 smithi096 ceph-mon[32753]: pgmap v17193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:48 smithi194 ceph-mon[33017]: pgmap v17194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:48 smithi033 ceph-mon[27845]: pgmap v17194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:48 smithi096 ceph-mon[32753]: pgmap v17194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31667.185302 secs 2024-04-26T20:38:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31670.117525 secs 2024-04-26T20:38:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31667.185302 secs 2024-04-26T20:38:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31670.117525 secs 2024-04-26T20:38:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31667.185302 secs 2024-04-26T20:38:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31670.117525 secs 2024-04-26T20:38:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:50 smithi033 ceph-mon[27845]: pgmap v17195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:50 smithi096 ceph-mon[32753]: pgmap v17195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:50 smithi194 ceph-mon[33017]: pgmap v17195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:52 smithi033 ceph-mon[27845]: pgmap v17196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:52 smithi096 ceph-mon[32753]: pgmap v17196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:52 smithi194 ceph-mon[33017]: pgmap v17196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31672.185382 secs 2024-04-26T20:38:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:54 smithi033 ceph-mon[27845]: pgmap v17197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31675.117602 secs 2024-04-26T20:38:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31672.185382 secs 2024-04-26T20:38:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:54 smithi096 ceph-mon[32753]: pgmap v17197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31675.117602 secs 2024-04-26T20:38:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31672.185382 secs 2024-04-26T20:38:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:54 smithi194 ceph-mon[33017]: pgmap v17197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31675.117602 secs 2024-04-26T20:38:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:55.951+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:55.951+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:38:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:38:55.951+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:38:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:56 smithi033 ceph-mon[27845]: pgmap v17198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:56 smithi096 ceph-mon[32753]: pgmap v17198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:56 smithi194 ceph-mon[33017]: pgmap v17198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:38:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:58 smithi033 ceph-mon[27845]: pgmap v17199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:58 smithi096 ceph-mon[32753]: pgmap v17199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:58 smithi194 ceph-mon[33017]: pgmap v17199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:38:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31677.185466 secs 2024-04-26T20:38:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31680.117711 secs 2024-04-26T20:38:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31677.185466 secs 2024-04-26T20:38:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31680.117711 secs 2024-04-26T20:38:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31677.185466 secs 2024-04-26T20:38:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31680.117711 secs 2024-04-26T20:39:00.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:00 smithi033 ceph-mon[27845]: pgmap v17200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:00 smithi096 ceph-mon[32753]: pgmap v17200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:00 smithi194 ceph-mon[33017]: pgmap v17200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:02.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:02 smithi194 ceph-mon[33017]: pgmap v17201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:02 smithi033 ceph-mon[27845]: pgmap v17201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:02 smithi096 ceph-mon[32753]: pgmap v17201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31682.185548 secs 2024-04-26T20:39:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:04 smithi033 ceph-mon[27845]: pgmap v17202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31685.117848 secs 2024-04-26T20:39:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31682.185548 secs 2024-04-26T20:39:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:04 smithi096 ceph-mon[32753]: pgmap v17202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31685.117848 secs 2024-04-26T20:39:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31682.185548 secs 2024-04-26T20:39:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:04 smithi194 ceph-mon[33017]: pgmap v17202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31685.117848 secs 2024-04-26T20:39:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:06 smithi033 ceph-mon[27845]: pgmap v17203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:06 smithi096 ceph-mon[32753]: pgmap v17203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:06 smithi194 ceph-mon[33017]: pgmap v17203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:08 smithi033 ceph-mon[27845]: pgmap v17204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:08 smithi096 ceph-mon[32753]: pgmap v17204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:08 smithi194 ceph-mon[33017]: pgmap v17204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:09.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31687.185633 secs 2024-04-26T20:39:09.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31690.117963 secs 2024-04-26T20:39:09.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31687.185633 secs 2024-04-26T20:39:09.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31690.117963 secs 2024-04-26T20:39:09.892 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31687.185633 secs 2024-04-26T20:39:09.892 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31690.117963 secs 2024-04-26T20:39:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:10 smithi096 ceph-mon[32753]: pgmap v17205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:10 smithi194 ceph-mon[33017]: pgmap v17205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:10 smithi033 ceph-mon[27845]: pgmap v17205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:10.963+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:10.963+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:39:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:10.963+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:12 smithi096 ceph-mon[32753]: pgmap v17206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:12 smithi194 ceph-mon[33017]: pgmap v17206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:12 smithi033 ceph-mon[27845]: pgmap v17206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31692.185733 secs 2024-04-26T20:39:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:14 smithi194 ceph-mon[33017]: pgmap v17207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31695.118087 secs 2024-04-26T20:39:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31692.185733 secs 2024-04-26T20:39:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:14 smithi033 ceph-mon[27845]: pgmap v17207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31695.118087 secs 2024-04-26T20:39:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31692.185733 secs 2024-04-26T20:39:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:14 smithi096 ceph-mon[32753]: pgmap v17207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31695.118087 secs 2024-04-26T20:39:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:16 smithi194 ceph-mon[33017]: pgmap v17208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:39:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:16 smithi033 ceph-mon[27845]: pgmap v17208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:39:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:16 smithi096 ceph-mon[32753]: pgmap v17208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:39:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:39:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:39:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:39:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:39:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:39:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:39:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:39:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:39:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:39:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:18 smithi194 ceph-mon[33017]: pgmap v17209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:18 smithi194 ceph-mon[33017]: pgmap v17210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:18 smithi033 ceph-mon[27845]: pgmap v17209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:18 smithi033 ceph-mon[27845]: pgmap v17210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:18 smithi096 ceph-mon[32753]: pgmap v17209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:18 smithi096 ceph-mon[32753]: pgmap v17210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31697.185818 secs 2024-04-26T20:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31700.118209 secs 2024-04-26T20:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31697.185818 secs 2024-04-26T20:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31700.118209 secs 2024-04-26T20:39:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31697.185818 secs 2024-04-26T20:39:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31700.118209 secs 2024-04-26T20:39:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:20 smithi194 ceph-mon[33017]: pgmap v17211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:20 smithi033 ceph-mon[27845]: pgmap v17211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:20 smithi096 ceph-mon[32753]: pgmap v17211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:22 smithi194 ceph-mon[33017]: pgmap v17212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:22 smithi033 ceph-mon[27845]: pgmap v17212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:22 smithi096 ceph-mon[32753]: pgmap v17212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:24 smithi194 ceph-mon[33017]: pgmap v17213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31702.185902 secs 2024-04-26T20:39:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31705.118334 secs 2024-04-26T20:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:24 smithi033 ceph-mon[27845]: pgmap v17213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31702.185902 secs 2024-04-26T20:39:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31705.118334 secs 2024-04-26T20:39:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:24 smithi096 ceph-mon[32753]: pgmap v17213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31702.185902 secs 2024-04-26T20:39:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31705.118334 secs 2024-04-26T20:39:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:25.970+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:25.970+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:25.970+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:39:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:26 smithi194 ceph-mon[33017]: pgmap v17214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:26 smithi033 ceph-mon[27845]: pgmap v17214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:26 smithi096 ceph-mon[32753]: pgmap v17214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:39:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:28 smithi194 ceph-mon[33017]: pgmap v17215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:39:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:28 smithi033 ceph-mon[27845]: pgmap v17215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:39:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:28 smithi096 ceph-mon[32753]: pgmap v17215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:39:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31707.186006 secs 2024-04-26T20:39:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31710.118397 secs 2024-04-26T20:39:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31707.186006 secs 2024-04-26T20:39:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31710.118397 secs 2024-04-26T20:39:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31707.186006 secs 2024-04-26T20:39:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31710.118397 secs 2024-04-26T20:39:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:30 smithi194 ceph-mon[33017]: pgmap v17216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:30 smithi033 ceph-mon[27845]: pgmap v17216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:30 smithi096 ceph-mon[32753]: pgmap v17216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:32 smithi194 ceph-mon[33017]: pgmap v17217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:32 smithi033 ceph-mon[27845]: pgmap v17217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:32 smithi096 ceph-mon[32753]: pgmap v17217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:34 smithi194 ceph-mon[33017]: pgmap v17218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31712.186066 secs 2024-04-26T20:39:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31715.118518 secs 2024-04-26T20:39:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:34 smithi033 ceph-mon[27845]: pgmap v17218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31712.186066 secs 2024-04-26T20:39:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31715.118518 secs 2024-04-26T20:39:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:34 smithi096 ceph-mon[32753]: pgmap v17218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31712.186066 secs 2024-04-26T20:39:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31715.118518 secs 2024-04-26T20:39:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:36 smithi194 ceph-mon[33017]: pgmap v17219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:36 smithi033 ceph-mon[27845]: pgmap v17219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:36 smithi096 ceph-mon[32753]: pgmap v17219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:38 smithi194 ceph-mon[33017]: pgmap v17220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:38 smithi033 ceph-mon[27845]: pgmap v17220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:38 smithi096 ceph-mon[32753]: pgmap v17220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31717.186129 secs 2024-04-26T20:39:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31720.118597 secs 2024-04-26T20:39:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31717.186129 secs 2024-04-26T20:39:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31720.118597 secs 2024-04-26T20:39:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31717.186129 secs 2024-04-26T20:39:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31720.118597 secs 2024-04-26T20:39:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:40 smithi194 ceph-mon[33017]: pgmap v17221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:40 smithi033 ceph-mon[27845]: pgmap v17221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:40 smithi096 ceph-mon[32753]: pgmap v17221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:40.972+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:40.973+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:40.973+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:39:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:42 smithi194 ceph-mon[33017]: pgmap v17222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:42 smithi033 ceph-mon[27845]: pgmap v17222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:42 smithi096 ceph-mon[32753]: pgmap v17222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:44 smithi194 ceph-mon[33017]: pgmap v17223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31722.186189 secs 2024-04-26T20:39:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31725.118716 secs 2024-04-26T20:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:44 smithi033 ceph-mon[27845]: pgmap v17223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31722.186189 secs 2024-04-26T20:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31725.118716 secs 2024-04-26T20:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:44 smithi096 ceph-mon[32753]: pgmap v17223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31722.186189 secs 2024-04-26T20:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31725.118716 secs 2024-04-26T20:39:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:46 smithi194 ceph-mon[33017]: pgmap v17224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:46 smithi033 ceph-mon[27845]: pgmap v17224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:46 smithi096 ceph-mon[32753]: pgmap v17224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:48 smithi194 ceph-mon[33017]: pgmap v17225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:48 smithi033 ceph-mon[27845]: pgmap v17225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:48 smithi096 ceph-mon[32753]: pgmap v17225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31727.186273 secs 2024-04-26T20:39:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31730.118850 secs 2024-04-26T20:39:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31727.186273 secs 2024-04-26T20:39:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31730.118850 secs 2024-04-26T20:39:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31727.186273 secs 2024-04-26T20:39:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31730.118850 secs 2024-04-26T20:39:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:50 smithi194 ceph-mon[33017]: pgmap v17226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:50 smithi033 ceph-mon[27845]: pgmap v17226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:50 smithi096 ceph-mon[32753]: pgmap v17226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:52 smithi194 ceph-mon[33017]: pgmap v17227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:52 smithi033 ceph-mon[27845]: pgmap v17227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:52 smithi096 ceph-mon[32753]: pgmap v17227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:54 smithi194 ceph-mon[33017]: pgmap v17228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31732.186333 secs 2024-04-26T20:39:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31735.118972 secs 2024-04-26T20:39:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:54 smithi033 ceph-mon[27845]: pgmap v17228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31732.186333 secs 2024-04-26T20:39:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31735.118972 secs 2024-04-26T20:39:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:54 smithi096 ceph-mon[32753]: pgmap v17228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31732.186333 secs 2024-04-26T20:39:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31735.118972 secs 2024-04-26T20:39:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:55.989+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:55.989+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:39:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:39:55.989+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:39:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:56 smithi194 ceph-mon[33017]: pgmap v17229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:56 smithi033 ceph-mon[27845]: pgmap v17229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:56 smithi096 ceph-mon[32753]: pgmap v17229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:39:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:58 smithi194 ceph-mon[33017]: pgmap v17230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:58 smithi033 ceph-mon[27845]: pgmap v17230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:58 smithi096 ceph-mon[32753]: pgmap v17230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:39:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31737.186416 secs 2024-04-26T20:39:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:39:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31740.119091 secs 2024-04-26T20:39:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31737.186416 secs 2024-04-26T20:39:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:39:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31740.119091 secs 2024-04-26T20:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31737.186416 secs 2024-04-26T20:39:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:39:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31740.119091 secs 2024-04-26T20:40:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:00 smithi194 ceph-mon[33017]: pgmap v17231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:40:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:00 smithi033 ceph-mon[27845]: pgmap v17231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:40:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:00 smithi096 ceph-mon[32753]: pgmap v17231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:40:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:02 smithi033 ceph-mon[27845]: pgmap v17232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:02 smithi096 ceph-mon[32753]: pgmap v17232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:02 smithi194 ceph-mon[33017]: pgmap v17232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:04 smithi033 ceph-mon[27845]: pgmap v17233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31742.186498 secs 2024-04-26T20:40:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31745.119185 secs 2024-04-26T20:40:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:04 smithi096 ceph-mon[32753]: pgmap v17233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31742.186498 secs 2024-04-26T20:40:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31745.119185 secs 2024-04-26T20:40:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:04 smithi194 ceph-mon[33017]: pgmap v17233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31742.186498 secs 2024-04-26T20:40:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31745.119185 secs 2024-04-26T20:40:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:06 smithi033 ceph-mon[27845]: pgmap v17234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:06 smithi096 ceph-mon[32753]: pgmap v17234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:06 smithi194 ceph-mon[33017]: pgmap v17234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:08 smithi033 ceph-mon[27845]: pgmap v17235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:08 smithi096 ceph-mon[32753]: pgmap v17235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:08 smithi194 ceph-mon[33017]: pgmap v17235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31747.186582 secs 2024-04-26T20:40:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31750.119312 secs 2024-04-26T20:40:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31747.186582 secs 2024-04-26T20:40:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31750.119312 secs 2024-04-26T20:40:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31747.186582 secs 2024-04-26T20:40:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31750.119312 secs 2024-04-26T20:40:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:10 smithi033 ceph-mon[27845]: pgmap v17236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:10 smithi096 ceph-mon[32753]: pgmap v17236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:11.001+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:11.001+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:11.001+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:40:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:10 smithi194 ceph-mon[33017]: pgmap v17236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:12 smithi033 ceph-mon[27845]: pgmap v17237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:12 smithi096 ceph-mon[32753]: pgmap v17237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:12 smithi194 ceph-mon[33017]: pgmap v17237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:14 smithi033 ceph-mon[27845]: pgmap v17238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31752.186663 secs 2024-04-26T20:40:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31755.119439 secs 2024-04-26T20:40:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:14 smithi096 ceph-mon[32753]: pgmap v17238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31752.186663 secs 2024-04-26T20:40:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31755.119439 secs 2024-04-26T20:40:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:14 smithi194 ceph-mon[33017]: pgmap v17238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31752.186663 secs 2024-04-26T20:40:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31755.119439 secs 2024-04-26T20:40:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:16 smithi033 ceph-mon[27845]: pgmap v17239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:16 smithi096 ceph-mon[32753]: pgmap v17239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:16 smithi194 ceph-mon[33017]: pgmap v17239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:40:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:40:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:40:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:18 smithi033 ceph-mon[27845]: pgmap v17240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:18 smithi096 ceph-mon[32753]: pgmap v17240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:18 smithi194 ceph-mon[33017]: pgmap v17240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31757.186750 secs 2024-04-26T20:40:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31760.119583 secs 2024-04-26T20:40:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:40:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:40:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31757.186750 secs 2024-04-26T20:40:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31760.119583 secs 2024-04-26T20:40:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:40:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:40:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31757.186750 secs 2024-04-26T20:40:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31760.119583 secs 2024-04-26T20:40:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:40:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:40:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:40:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:20 smithi096 ceph-mon[32753]: pgmap v17241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:20 smithi096 ceph-mon[32753]: pgmap v17242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:40:20.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:20 smithi096 ceph-mon[32753]: pgmap v17243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:20 smithi194 ceph-mon[33017]: pgmap v17241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:20 smithi194 ceph-mon[33017]: pgmap v17242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:40:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:20 smithi194 ceph-mon[33017]: pgmap v17243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:20 smithi033 ceph-mon[27845]: pgmap v17241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:20 smithi033 ceph-mon[27845]: pgmap v17242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:40:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:20 smithi033 ceph-mon[27845]: pgmap v17243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:22 smithi194 ceph-mon[33017]: pgmap v17244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:22 smithi033 ceph-mon[27845]: pgmap v17244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:22 smithi096 ceph-mon[32753]: pgmap v17244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31762.186830 secs 2024-04-26T20:40:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:24 smithi194 ceph-mon[33017]: pgmap v17245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:40:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31765.119654 secs 2024-04-26T20:40:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31762.186830 secs 2024-04-26T20:40:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:24 smithi033 ceph-mon[27845]: pgmap v17245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:40:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31765.119654 secs 2024-04-26T20:40:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31762.186830 secs 2024-04-26T20:40:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:24 smithi096 ceph-mon[32753]: pgmap v17245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:40:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31765.119654 secs 2024-04-26T20:40:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:26.015+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:26.015+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:40:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:26.015+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:26 smithi194 ceph-mon[33017]: pgmap v17246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:26 smithi033 ceph-mon[27845]: pgmap v17246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:26 smithi096 ceph-mon[32753]: pgmap v17246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:28 smithi194 ceph-mon[33017]: pgmap v17247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:28 smithi033 ceph-mon[27845]: pgmap v17247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:28 smithi096 ceph-mon[32753]: pgmap v17247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:40:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31767.186914 secs 2024-04-26T20:40:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31770.119781 secs 2024-04-26T20:40:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31767.186914 secs 2024-04-26T20:40:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31770.119781 secs 2024-04-26T20:40:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31767.186914 secs 2024-04-26T20:40:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31770.119781 secs 2024-04-26T20:40:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:30 smithi194 ceph-mon[33017]: pgmap v17248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:40:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:30 smithi033 ceph-mon[27845]: pgmap v17248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:40:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:30 smithi096 ceph-mon[32753]: pgmap v17248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:40:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:32 smithi194 ceph-mon[33017]: pgmap v17249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:32 smithi033 ceph-mon[27845]: pgmap v17249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:32 smithi096 ceph-mon[32753]: pgmap v17249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:34.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31772.187011 secs 2024-04-26T20:40:34.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:34 smithi033 ceph-mon[27845]: pgmap v17250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:34.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31775.119882 secs 2024-04-26T20:40:34.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31772.187011 secs 2024-04-26T20:40:34.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:34 smithi096 ceph-mon[32753]: pgmap v17250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:34.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31775.119882 secs 2024-04-26T20:40:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31772.187011 secs 2024-04-26T20:40:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:34 smithi194 ceph-mon[33017]: pgmap v17250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31775.119882 secs 2024-04-26T20:40:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:36 smithi194 ceph-mon[33017]: pgmap v17251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:36 smithi033 ceph-mon[27845]: pgmap v17251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:36 smithi096 ceph-mon[32753]: pgmap v17251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:38 smithi194 ceph-mon[33017]: pgmap v17252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:38 smithi033 ceph-mon[27845]: pgmap v17252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:38 smithi096 ceph-mon[32753]: pgmap v17252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31777.187100 secs 2024-04-26T20:40:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31780.120018 secs 2024-04-26T20:40:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31777.187100 secs 2024-04-26T20:40:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31780.120018 secs 2024-04-26T20:40:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31777.187100 secs 2024-04-26T20:40:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31780.120018 secs 2024-04-26T20:40:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:41.030+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:41.030+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:40:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:41.030+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:40 smithi194 ceph-mon[33017]: pgmap v17253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:40 smithi033 ceph-mon[27845]: pgmap v17253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:40 smithi096 ceph-mon[32753]: pgmap v17253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:42 smithi194 ceph-mon[33017]: pgmap v17254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:42 smithi033 ceph-mon[27845]: pgmap v17254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:42 smithi096 ceph-mon[32753]: pgmap v17254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31785.120176 secs 2024-04-26T20:40:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31785.120176 secs 2024-04-26T20:40:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31785.120176 secs 2024-04-26T20:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31782.187211 secs 2024-04-26T20:40:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:44 smithi194 ceph-mon[33017]: pgmap v17255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31782.187211 secs 2024-04-26T20:40:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:44 smithi033 ceph-mon[27845]: pgmap v17255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31782.187211 secs 2024-04-26T20:40:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:44 smithi096 ceph-mon[32753]: pgmap v17255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:46 smithi194 ceph-mon[33017]: pgmap v17256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:46 smithi033 ceph-mon[27845]: pgmap v17256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:46 smithi096 ceph-mon[32753]: pgmap v17256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:48 smithi194 ceph-mon[33017]: pgmap v17257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31790.120239 secs 2024-04-26T20:40:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:48 smithi033 ceph-mon[27845]: pgmap v17257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31790.120239 secs 2024-04-26T20:40:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:48 smithi096 ceph-mon[32753]: pgmap v17257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31790.120239 secs 2024-04-26T20:40:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31787.187307 secs 2024-04-26T20:40:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31787.187307 secs 2024-04-26T20:40:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31787.187307 secs 2024-04-26T20:40:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:50 smithi194 ceph-mon[33017]: pgmap v17258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:50 smithi033 ceph-mon[27845]: pgmap v17258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:50 smithi096 ceph-mon[32753]: pgmap v17258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:51 smithi194 ceph-mon[33017]: pgmap v17259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:51 smithi033 ceph-mon[27845]: pgmap v17259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:51 smithi096 ceph-mon[32753]: pgmap v17259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31792.187393 secs 2024-04-26T20:40:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:54 smithi194 ceph-mon[33017]: pgmap v17260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31795.120318 secs 2024-04-26T20:40:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31792.187393 secs 2024-04-26T20:40:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:54 smithi033 ceph-mon[27845]: pgmap v17260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31795.120318 secs 2024-04-26T20:40:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31792.187393 secs 2024-04-26T20:40:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:54 smithi096 ceph-mon[32753]: pgmap v17260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31795.120318 secs 2024-04-26T20:40:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:56.046+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:56.046+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:40:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:40:56.046+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:40:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:56 smithi194 ceph-mon[33017]: pgmap v17261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:56 smithi033 ceph-mon[27845]: pgmap v17261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:56 smithi096 ceph-mon[32753]: pgmap v17261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:40:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:58 smithi194 ceph-mon[33017]: pgmap v17262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:58 smithi033 ceph-mon[27845]: pgmap v17262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:58 smithi096 ceph-mon[32753]: pgmap v17262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:40:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31797.187501 secs 2024-04-26T20:40:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:40:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31800.120435 secs 2024-04-26T20:40:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31797.187501 secs 2024-04-26T20:40:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:40:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31800.120435 secs 2024-04-26T20:40:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31797.187501 secs 2024-04-26T20:40:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:40:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31800.120435 secs 2024-04-26T20:41:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:00 smithi194 ceph-mon[33017]: pgmap v17263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:00 smithi033 ceph-mon[27845]: pgmap v17263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:00 smithi096 ceph-mon[32753]: pgmap v17263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:02 smithi194 ceph-mon[33017]: pgmap v17264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:02 smithi033 ceph-mon[27845]: pgmap v17264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:02 smithi096 ceph-mon[32753]: pgmap v17264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31802.187607 secs 2024-04-26T20:41:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:04 smithi194 ceph-mon[33017]: pgmap v17265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31805.120532 secs 2024-04-26T20:41:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31802.187607 secs 2024-04-26T20:41:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:04 smithi033 ceph-mon[27845]: pgmap v17265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31805.120532 secs 2024-04-26T20:41:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31802.187607 secs 2024-04-26T20:41:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:04 smithi096 ceph-mon[32753]: pgmap v17265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31805.120532 secs 2024-04-26T20:41:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:06 smithi194 ceph-mon[33017]: pgmap v17266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:06 smithi033 ceph-mon[27845]: pgmap v17266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:06 smithi096 ceph-mon[32753]: pgmap v17266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:08 smithi194 ceph-mon[33017]: pgmap v17267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:08 smithi033 ceph-mon[27845]: pgmap v17267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:08 smithi096 ceph-mon[32753]: pgmap v17267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31807.187715 secs 2024-04-26T20:41:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31810.120653 secs 2024-04-26T20:41:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31807.187715 secs 2024-04-26T20:41:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31810.120653 secs 2024-04-26T20:41:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31807.187715 secs 2024-04-26T20:41:09.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31810.120653 secs 2024-04-26T20:41:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:10 smithi194 ceph-mon[33017]: pgmap v17268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:10 smithi033 ceph-mon[27845]: pgmap v17268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:10 smithi096 ceph-mon[32753]: pgmap v17268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:11.062+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:11.062+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:41:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:11.062+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:12 smithi194 ceph-mon[33017]: pgmap v17269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:12 smithi033 ceph-mon[27845]: pgmap v17269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:12 smithi096 ceph-mon[32753]: pgmap v17269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31812.187823 secs 2024-04-26T20:41:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:14 smithi194 ceph-mon[33017]: pgmap v17270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31815.120770 secs 2024-04-26T20:41:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31812.187823 secs 2024-04-26T20:41:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:14 smithi033 ceph-mon[27845]: pgmap v17270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31815.120770 secs 2024-04-26T20:41:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31812.187823 secs 2024-04-26T20:41:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:14 smithi096 ceph-mon[32753]: pgmap v17270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31815.120770 secs 2024-04-26T20:41:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:16 smithi194 ceph-mon[33017]: pgmap v17271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:16 smithi033 ceph-mon[27845]: pgmap v17271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:16 smithi096 ceph-mon[32753]: pgmap v17271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:18 smithi194 ceph-mon[33017]: pgmap v17272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:18.896 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:18 smithi033 ceph-mon[27845]: pgmap v17272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:18 smithi096 ceph-mon[32753]: pgmap v17272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31817.187929 secs 2024-04-26T20:41:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31820.120887 secs 2024-04-26T20:41:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31817.187929 secs 2024-04-26T20:41:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31820.120887 secs 2024-04-26T20:41:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31817.187929 secs 2024-04-26T20:41:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31820.120887 secs 2024-04-26T20:41:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:20 smithi033 ceph-mon[27845]: pgmap v17273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:41:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:41:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:20 smithi096 ceph-mon[32753]: pgmap v17273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:41:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:41:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:20 smithi194 ceph-mon[33017]: pgmap v17273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:41:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef007f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef007f040 comp rx=0 tx=0).stop 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:41:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.805+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:41:21.807 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.806+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:41:21.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:21.807+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0058040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:41:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:41:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:41:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:41:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:41:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:41:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:41:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:41:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:41:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:41:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:22 smithi033 ceph-mon[27845]: pgmap v17274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:41:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:22 smithi033 ceph-mon[27845]: pgmap v17275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:22 smithi096 ceph-mon[32753]: pgmap v17274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:41:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:22 smithi096 ceph-mon[32753]: pgmap v17275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:22 smithi194 ceph-mon[33017]: pgmap v17274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:41:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:22 smithi194 ceph-mon[33017]: pgmap v17275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31822.188002 secs 2024-04-26T20:41:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:24 smithi033 ceph-mon[27845]: pgmap v17276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31825.121014 secs 2024-04-26T20:41:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31822.188002 secs 2024-04-26T20:41:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:24 smithi096 ceph-mon[32753]: pgmap v17276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31825.121014 secs 2024-04-26T20:41:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31822.188002 secs 2024-04-26T20:41:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:24 smithi194 ceph-mon[33017]: pgmap v17276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31825.121014 secs 2024-04-26T20:41:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:26.067+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:26.067+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:41:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:26.067+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:26 smithi033 ceph-mon[27845]: pgmap v17277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:26 smithi096 ceph-mon[32753]: pgmap v17277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:26 smithi194 ceph-mon[33017]: pgmap v17277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:27.212+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:41:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:27.212+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:41:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:27.213+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 250 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T20:41:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:27.213+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 250 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T20:41:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:28 smithi033 ceph-mon[27845]: pgmap v17278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:28 smithi096 ceph-mon[32753]: pgmap v17278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:28 smithi194 ceph-mon[33017]: pgmap v17278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T20:41:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31827.188089 secs 2024-04-26T20:41:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31830.121152 secs 2024-04-26T20:41:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31827.188089 secs 2024-04-26T20:41:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31830.121152 secs 2024-04-26T20:41:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31827.188089 secs 2024-04-26T20:41:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31830.121152 secs 2024-04-26T20:41:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:30 smithi033 ceph-mon[27845]: pgmap v17279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:41:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:30 smithi096 ceph-mon[32753]: pgmap v17279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:41:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:30 smithi194 ceph-mon[33017]: pgmap v17279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:41:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:32 smithi033 ceph-mon[27845]: pgmap v17280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:41:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:32 smithi096 ceph-mon[32753]: pgmap v17280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:41:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:32 smithi194 ceph-mon[33017]: pgmap v17280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:41:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31832.188148 secs 2024-04-26T20:41:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:34 smithi033 ceph-mon[27845]: pgmap v17281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31835.121200 secs 2024-04-26T20:41:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31832.188148 secs 2024-04-26T20:41:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:34 smithi096 ceph-mon[32753]: pgmap v17281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31835.121200 secs 2024-04-26T20:41:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31832.188148 secs 2024-04-26T20:41:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:34 smithi194 ceph-mon[33017]: pgmap v17281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31835.121200 secs 2024-04-26T20:41:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:36 smithi033 ceph-mon[27845]: pgmap v17282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:36 smithi096 ceph-mon[32753]: pgmap v17282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:36 smithi194 ceph-mon[33017]: pgmap v17282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:38 smithi033 ceph-mon[27845]: pgmap v17283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:38 smithi096 ceph-mon[32753]: pgmap v17283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:38 smithi194 ceph-mon[33017]: pgmap v17283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31837.188237 secs 2024-04-26T20:41:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31840.121324 secs 2024-04-26T20:41:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31837.188237 secs 2024-04-26T20:41:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31840.121324 secs 2024-04-26T20:41:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31837.188237 secs 2024-04-26T20:41:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31840.121324 secs 2024-04-26T20:41:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:41.074+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:41.074+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:41.074+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:41:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:40 smithi194 ceph-mon[33017]: pgmap v17284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:40 smithi033 ceph-mon[27845]: pgmap v17284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:40 smithi096 ceph-mon[32753]: pgmap v17284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:42 smithi194 ceph-mon[33017]: pgmap v17285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:42 smithi033 ceph-mon[27845]: pgmap v17285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:42 smithi096 ceph-mon[32753]: pgmap v17285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31842.188316 secs 2024-04-26T20:41:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:44 smithi194 ceph-mon[33017]: pgmap v17286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31845.121441 secs 2024-04-26T20:41:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31842.188316 secs 2024-04-26T20:41:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:44 smithi033 ceph-mon[27845]: pgmap v17286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31845.121441 secs 2024-04-26T20:41:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31842.188316 secs 2024-04-26T20:41:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:44 smithi096 ceph-mon[32753]: pgmap v17286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31845.121441 secs 2024-04-26T20:41:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:46 smithi194 ceph-mon[33017]: pgmap v17287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:46 smithi033 ceph-mon[27845]: pgmap v17287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:46 smithi096 ceph-mon[32753]: pgmap v17287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:48 smithi194 ceph-mon[33017]: pgmap v17288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:48 smithi033 ceph-mon[27845]: pgmap v17288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:48 smithi096 ceph-mon[32753]: pgmap v17288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31847.188406 secs 2024-04-26T20:41:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31850.121553 secs 2024-04-26T20:41:50.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31847.188406 secs 2024-04-26T20:41:50.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31850.121553 secs 2024-04-26T20:41:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31847.188406 secs 2024-04-26T20:41:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31850.121553 secs 2024-04-26T20:41:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:50 smithi194 ceph-mon[33017]: pgmap v17289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:50 smithi033 ceph-mon[27845]: pgmap v17289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:50 smithi096 ceph-mon[32753]: pgmap v17289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:52 smithi194 ceph-mon[33017]: pgmap v17290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:52 smithi033 ceph-mon[27845]: pgmap v17290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:52 smithi096 ceph-mon[32753]: pgmap v17290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31852.188487 secs 2024-04-26T20:41:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:54 smithi194 ceph-mon[33017]: pgmap v17291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31855.121670 secs 2024-04-26T20:41:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31852.188487 secs 2024-04-26T20:41:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:54 smithi033 ceph-mon[27845]: pgmap v17291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31855.121670 secs 2024-04-26T20:41:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31852.188487 secs 2024-04-26T20:41:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:54 smithi096 ceph-mon[32753]: pgmap v17291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31855.121670 secs 2024-04-26T20:41:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:56.077+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:56.077+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:41:56.077+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:41:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:56 smithi194 ceph-mon[33017]: pgmap v17292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:56 smithi033 ceph-mon[27845]: pgmap v17292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:56 smithi096 ceph-mon[32753]: pgmap v17292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:41:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:58 smithi194 ceph-mon[33017]: pgmap v17293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:58 smithi033 ceph-mon[27845]: pgmap v17293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:41:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:58 smithi096 ceph-mon[32753]: pgmap v17293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31857.188570 secs 2024-04-26T20:42:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:41:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31860.121785 secs 2024-04-26T20:42:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31857.188570 secs 2024-04-26T20:42:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:41:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31860.121785 secs 2024-04-26T20:42:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31857.188570 secs 2024-04-26T20:42:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:41:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31860.121785 secs 2024-04-26T20:42:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:00 smithi194 ceph-mon[33017]: pgmap v17294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:00 smithi033 ceph-mon[27845]: pgmap v17294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:00 smithi096 ceph-mon[32753]: pgmap v17294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:02 smithi194 ceph-mon[33017]: pgmap v17295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:02 smithi033 ceph-mon[27845]: pgmap v17295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:02 smithi096 ceph-mon[32753]: pgmap v17295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31862.188651 secs 2024-04-26T20:42:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:04 smithi194 ceph-mon[33017]: pgmap v17296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31865.121903 secs 2024-04-26T20:42:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31862.188651 secs 2024-04-26T20:42:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:04 smithi033 ceph-mon[27845]: pgmap v17296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31865.121903 secs 2024-04-26T20:42:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31862.188651 secs 2024-04-26T20:42:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:04 smithi096 ceph-mon[32753]: pgmap v17296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31865.121903 secs 2024-04-26T20:42:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:06 smithi194 ceph-mon[33017]: pgmap v17297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:06 smithi033 ceph-mon[27845]: pgmap v17297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:06 smithi096 ceph-mon[32753]: pgmap v17297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:08 smithi194 ceph-mon[33017]: pgmap v17298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:09.195 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31870.122024 secs 2024-04-26T20:42:09.196 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:08 smithi096 ceph-mon[32753]: pgmap v17298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:09.196 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31870.122024 secs 2024-04-26T20:42:09.196 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:08 smithi033 ceph-mon[27845]: pgmap v17298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:09.196 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31870.122024 secs 2024-04-26T20:42:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31867.188734 secs 2024-04-26T20:42:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31867.188734 secs 2024-04-26T20:42:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31867.188734 secs 2024-04-26T20:42:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:11.088+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:11.088+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:42:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:11.088+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:10 smithi194 ceph-mon[33017]: pgmap v17299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:10 smithi033 ceph-mon[27845]: pgmap v17299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:10 smithi096 ceph-mon[32753]: pgmap v17299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:12 smithi194 ceph-mon[33017]: pgmap v17300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:12 smithi033 ceph-mon[27845]: pgmap v17300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:12 smithi096 ceph-mon[32753]: pgmap v17300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31875.122176 secs 2024-04-26T20:42:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31875.122176 secs 2024-04-26T20:42:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31875.122176 secs 2024-04-26T20:42:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31872.188823 secs 2024-04-26T20:42:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:14 smithi194 ceph-mon[33017]: pgmap v17301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31872.188823 secs 2024-04-26T20:42:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:14 smithi033 ceph-mon[27845]: pgmap v17301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31872.188823 secs 2024-04-26T20:42:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:14 smithi096 ceph-mon[32753]: pgmap v17301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:16 smithi194 ceph-mon[33017]: pgmap v17302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:16 smithi033 ceph-mon[27845]: pgmap v17302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:16 smithi096 ceph-mon[32753]: pgmap v17302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:18 smithi194 ceph-mon[33017]: pgmap v17303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31880.122254 secs 2024-04-26T20:42:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:18 smithi033 ceph-mon[27845]: pgmap v17303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31880.122254 secs 2024-04-26T20:42:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:18 smithi096 ceph-mon[32753]: pgmap v17303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31880.122254 secs 2024-04-26T20:42:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31877.188883 secs 2024-04-26T20:42:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31877.188883 secs 2024-04-26T20:42:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31877.188883 secs 2024-04-26T20:42:21.080 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:20 smithi033 ceph-mon[27845]: pgmap v17304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:20 smithi194 ceph-mon[33017]: pgmap v17304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:20 smithi096 ceph-mon[32753]: pgmap v17304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:42:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:42:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:42:23.086 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:22 smithi096 ceph-mon[32753]: pgmap v17305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:23.086 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.086 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.087 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.087 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:22 smithi194 ceph-mon[33017]: pgmap v17305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.312 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:22 smithi033 ceph-mon[27845]: pgmap v17305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:23.313 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.313 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.313 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:23.313 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31885.122345 secs 2024-04-26T20:42:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31885.122345 secs 2024-04-26T20:42:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31885.122345 secs 2024-04-26T20:42:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31882.189008 secs 2024-04-26T20:42:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:24 smithi194 ceph-mon[33017]: pgmap v17306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31882.189008 secs 2024-04-26T20:42:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:24 smithi033 ceph-mon[27845]: pgmap v17306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31882.189008 secs 2024-04-26T20:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:24 smithi096 ceph-mon[32753]: pgmap v17306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:26.104+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:26.104+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:26.104+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:26.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:26 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:42:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:26 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:42:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:26 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:42:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:26 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:42:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:26 smithi194 ceph-mon[33017]: pgmap v17307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:26 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:42:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:26 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:42:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:26 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:42:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:26 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:42:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:26 smithi033 ceph-mon[27845]: pgmap v17307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:27.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:26 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:42:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:26 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:42:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:26 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:42:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:26 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:42:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:26 smithi096 ceph-mon[32753]: pgmap v17307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:27 smithi033 ceph-mon[27845]: pgmap v17308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:27 smithi096 ceph-mon[32753]: pgmap v17308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:27 smithi194 ceph-mon[33017]: pgmap v17308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: pgmap v17309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: pgmap v17310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:29.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31890.122468 secs 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:42:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: pgmap v17309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:42:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: pgmap v17310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31890.122468 secs 2024-04-26T20:42:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: pgmap v17309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: pgmap v17310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:42:29.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31890.122468 secs 2024-04-26T20:42:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31887.189096 secs 2024-04-26T20:42:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31887.189096 secs 2024-04-26T20:42:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31887.189096 secs 2024-04-26T20:42:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:30 smithi033 ceph-mon[27845]: pgmap v17311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:30 smithi096 ceph-mon[32753]: pgmap v17311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:30 smithi194 ceph-mon[33017]: pgmap v17311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:33 smithi194 ceph-mon[33017]: pgmap v17312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:33 smithi033 ceph-mon[27845]: pgmap v17312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:33 smithi096 ceph-mon[32753]: pgmap v17312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:34.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31892.189155 secs 2024-04-26T20:42:34.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31895.122580 secs 2024-04-26T20:42:34.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31892.189155 secs 2024-04-26T20:42:34.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31895.122580 secs 2024-04-26T20:42:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31892.189155 secs 2024-04-26T20:42:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31895.122580 secs 2024-04-26T20:42:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:35 smithi194 ceph-mon[33017]: pgmap v17313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:42:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:35 smithi033 ceph-mon[27845]: pgmap v17313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:42:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:35 smithi096 ceph-mon[32753]: pgmap v17313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:42:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:37 smithi194 ceph-mon[33017]: pgmap v17314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:37 smithi033 ceph-mon[27845]: pgmap v17314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:37 smithi096 ceph-mon[32753]: pgmap v17314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:42:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31897.189259 secs 2024-04-26T20:42:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:39 smithi194 ceph-mon[33017]: pgmap v17315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:42:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31900.122699 secs 2024-04-26T20:42:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31897.189259 secs 2024-04-26T20:42:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:39 smithi033 ceph-mon[27845]: pgmap v17315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:42:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31900.122699 secs 2024-04-26T20:42:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31897.189259 secs 2024-04-26T20:42:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:39 smithi096 ceph-mon[32753]: pgmap v17315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:42:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31900.122699 secs 2024-04-26T20:42:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:41.118+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:41.118+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:42:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:41.118+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:41 smithi194 ceph-mon[33017]: pgmap v17316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:41 smithi033 ceph-mon[27845]: pgmap v17316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:41 smithi096 ceph-mon[32753]: pgmap v17316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:43 smithi194 ceph-mon[33017]: pgmap v17317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:43 smithi033 ceph-mon[27845]: pgmap v17317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:43 smithi096 ceph-mon[32753]: pgmap v17317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31902.189339 secs 2024-04-26T20:42:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31905.122763 secs 2024-04-26T20:42:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31902.189339 secs 2024-04-26T20:42:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31905.122763 secs 2024-04-26T20:42:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31902.189339 secs 2024-04-26T20:42:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31905.122763 secs 2024-04-26T20:42:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:45 smithi194 ceph-mon[33017]: pgmap v17318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:45 smithi033 ceph-mon[27845]: pgmap v17318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:45 smithi096 ceph-mon[32753]: pgmap v17318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:47 smithi194 ceph-mon[33017]: pgmap v17319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:47 smithi033 ceph-mon[27845]: pgmap v17319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:47 smithi096 ceph-mon[32753]: pgmap v17319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31907.189450 secs 2024-04-26T20:42:49.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:49 smithi033 ceph-mon[27845]: pgmap v17320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:49.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31910.122885 secs 2024-04-26T20:42:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31907.189450 secs 2024-04-26T20:42:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:49 smithi096 ceph-mon[32753]: pgmap v17320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31910.122885 secs 2024-04-26T20:42:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31907.189450 secs 2024-04-26T20:42:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:49 smithi194 ceph-mon[33017]: pgmap v17320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31910.122885 secs 2024-04-26T20:42:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:51 smithi033 ceph-mon[27845]: pgmap v17321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:51 smithi096 ceph-mon[32753]: pgmap v17321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:51 smithi194 ceph-mon[33017]: pgmap v17321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:53 smithi033 ceph-mon[27845]: pgmap v17322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:53 smithi096 ceph-mon[32753]: pgmap v17322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:53 smithi194 ceph-mon[33017]: pgmap v17322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31912.189556 secs 2024-04-26T20:42:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31915.123020 secs 2024-04-26T20:42:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31912.189556 secs 2024-04-26T20:42:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31915.123020 secs 2024-04-26T20:42:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31912.189556 secs 2024-04-26T20:42:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31915.123020 secs 2024-04-26T20:42:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:55 smithi033 ceph-mon[27845]: pgmap v17323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:55 smithi096 ceph-mon[32753]: pgmap v17323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:55 smithi194 ceph-mon[33017]: pgmap v17323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:56.133+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:56.133+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:42:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:42:56.133+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:42:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:57 smithi033 ceph-mon[27845]: pgmap v17324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:57 smithi096 ceph-mon[32753]: pgmap v17324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:57 smithi194 ceph-mon[33017]: pgmap v17324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:42:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31917.189648 secs 2024-04-26T20:42:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:59 smithi033 ceph-mon[27845]: pgmap v17325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31920.123160 secs 2024-04-26T20:42:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31917.189648 secs 2024-04-26T20:42:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:59 smithi096 ceph-mon[32753]: pgmap v17325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31920.123160 secs 2024-04-26T20:42:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31917.189648 secs 2024-04-26T20:42:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:59 smithi194 ceph-mon[33017]: pgmap v17325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:42:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31920.123160 secs 2024-04-26T20:43:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:01 smithi033 ceph-mon[27845]: pgmap v17326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:01 smithi096 ceph-mon[32753]: pgmap v17326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:01 smithi194 ceph-mon[33017]: pgmap v17326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:03.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:03 smithi194 ceph-mon[33017]: pgmap v17327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:03 smithi033 ceph-mon[27845]: pgmap v17327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:03 smithi096 ceph-mon[32753]: pgmap v17327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31922.189727 secs 2024-04-26T20:43:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31925.123225 secs 2024-04-26T20:43:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31922.189727 secs 2024-04-26T20:43:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31925.123225 secs 2024-04-26T20:43:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31922.189727 secs 2024-04-26T20:43:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31925.123225 secs 2024-04-26T20:43:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:05 smithi033 ceph-mon[27845]: pgmap v17328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:05 smithi096 ceph-mon[32753]: pgmap v17328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:05 smithi194 ceph-mon[33017]: pgmap v17328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:07 smithi033 ceph-mon[27845]: pgmap v17329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:07.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:07 smithi096 ceph-mon[32753]: pgmap v17329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:07 smithi194 ceph-mon[33017]: pgmap v17329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31927.189812 secs 2024-04-26T20:43:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:09 smithi096 ceph-mon[32753]: pgmap v17330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31930.123343 secs 2024-04-26T20:43:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31927.189812 secs 2024-04-26T20:43:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:09 smithi194 ceph-mon[33017]: pgmap v17330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31930.123343 secs 2024-04-26T20:43:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31927.189812 secs 2024-04-26T20:43:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:09 smithi033 ceph-mon[27845]: pgmap v17330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:43:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31930.123343 secs 2024-04-26T20:43:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:11.144+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:11.144+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:43:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:11.144+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:11 smithi194 ceph-mon[33017]: pgmap v17331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:11 smithi033 ceph-mon[27845]: pgmap v17331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:11 smithi096 ceph-mon[32753]: pgmap v17331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:43:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:13 smithi194 ceph-mon[33017]: pgmap v17332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:13.906 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:13 smithi096 ceph-mon[32753]: pgmap v17332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:13.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:13 smithi033 ceph-mon[27845]: pgmap v17332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31932.189891 secs 2024-04-26T20:43:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31935.123466 secs 2024-04-26T20:43:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31932.189891 secs 2024-04-26T20:43:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31935.123466 secs 2024-04-26T20:43:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31932.189891 secs 2024-04-26T20:43:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31935.123466 secs 2024-04-26T20:43:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:15 smithi194 ceph-mon[33017]: pgmap v17333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:15 smithi033 ceph-mon[27845]: pgmap v17333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:15 smithi096 ceph-mon[32753]: pgmap v17333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:17 smithi194 ceph-mon[33017]: pgmap v17334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:17 smithi033 ceph-mon[27845]: pgmap v17334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:17 smithi096 ceph-mon[32753]: pgmap v17334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31937.189995 secs 2024-04-26T20:43:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:19 smithi194 ceph-mon[33017]: pgmap v17335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31940.123585 secs 2024-04-26T20:43:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31937.189995 secs 2024-04-26T20:43:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:19 smithi033 ceph-mon[27845]: pgmap v17335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31940.123585 secs 2024-04-26T20:43:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31937.189995 secs 2024-04-26T20:43:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:19 smithi096 ceph-mon[32753]: pgmap v17335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31940.123585 secs 2024-04-26T20:43:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:21 smithi194 ceph-mon[33017]: pgmap v17336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:21 smithi033 ceph-mon[27845]: pgmap v17336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:21 smithi096 ceph-mon[32753]: pgmap v17336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:23.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:23 smithi033 ceph-mon[27845]: pgmap v17337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:23 smithi194 ceph-mon[33017]: pgmap v17337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:23 smithi096 ceph-mon[32753]: pgmap v17337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31942.190077 secs 2024-04-26T20:43:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31945.123662 secs 2024-04-26T20:43:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31942.190077 secs 2024-04-26T20:43:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31945.123662 secs 2024-04-26T20:43:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31942.190077 secs 2024-04-26T20:43:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31945.123662 secs 2024-04-26T20:43:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:25 smithi194 ceph-mon[33017]: pgmap v17338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:25 smithi033 ceph-mon[27845]: pgmap v17338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:25 smithi096 ceph-mon[32753]: pgmap v17338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:26.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:26.156+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:26.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:26.157+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:43:26.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:26.157+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:27 smithi194 ceph-mon[33017]: pgmap v17339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:27 smithi033 ceph-mon[27845]: pgmap v17339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:27 smithi096 ceph-mon[32753]: pgmap v17339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31947.190137 secs 2024-04-26T20:43:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:29 smithi194 ceph-mon[33017]: pgmap v17340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:43:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31950.123749 secs 2024-04-26T20:43:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31947.190137 secs 2024-04-26T20:43:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:29 smithi033 ceph-mon[27845]: pgmap v17340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:43:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31950.123749 secs 2024-04-26T20:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31947.190137 secs 2024-04-26T20:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:29 smithi096 ceph-mon[32753]: pgmap v17340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31950.123749 secs 2024-04-26T20:43:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:43:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:43:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:30 smithi194 ceph-mon[33017]: pgmap v17341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:43:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:30 smithi194 ceph-mon[33017]: pgmap v17342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:43:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:43:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:43:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:30 smithi033 ceph-mon[27845]: pgmap v17341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:43:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:30 smithi033 ceph-mon[27845]: pgmap v17342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:43:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:43:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:43:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:30 smithi096 ceph-mon[32753]: pgmap v17341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:43:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:30 smithi096 ceph-mon[32753]: pgmap v17342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:30.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:43:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:32 smithi194 ceph-mon[33017]: pgmap v17343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:32 smithi033 ceph-mon[27845]: pgmap v17343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:32 smithi096 ceph-mon[32753]: pgmap v17343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:34.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31952.190224 secs 2024-04-26T20:43:34.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:34 smithi033 ceph-mon[27845]: pgmap v17344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:34.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31955.123868 secs 2024-04-26T20:43:34.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31952.190224 secs 2024-04-26T20:43:34.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:34 smithi096 ceph-mon[32753]: pgmap v17344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:34.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31955.123868 secs 2024-04-26T20:43:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31952.190224 secs 2024-04-26T20:43:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:34 smithi194 ceph-mon[33017]: pgmap v17344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31955.123868 secs 2024-04-26T20:43:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:36 smithi194 ceph-mon[33017]: pgmap v17345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:36 smithi033 ceph-mon[27845]: pgmap v17345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:36 smithi096 ceph-mon[32753]: pgmap v17345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:38 smithi194 ceph-mon[33017]: pgmap v17346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:38 smithi033 ceph-mon[27845]: pgmap v17346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:38 smithi096 ceph-mon[32753]: pgmap v17346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:43:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31957.190314 secs 2024-04-26T20:43:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31960.123973 secs 2024-04-26T20:43:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31957.190314 secs 2024-04-26T20:43:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31960.123973 secs 2024-04-26T20:43:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31957.190314 secs 2024-04-26T20:43:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31960.123973 secs 2024-04-26T20:43:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:40 smithi194 ceph-mon[33017]: pgmap v17347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:43:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:41.172+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:41.172+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:43:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:41.172+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:40 smithi033 ceph-mon[27845]: pgmap v17347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:43:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:40 smithi096 ceph-mon[32753]: pgmap v17347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:43:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:42 smithi194 ceph-mon[33017]: pgmap v17348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:42 smithi033 ceph-mon[27845]: pgmap v17348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:42 smithi096 ceph-mon[32753]: pgmap v17348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31962.190372 secs 2024-04-26T20:43:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:44 smithi194 ceph-mon[33017]: pgmap v17349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31965.124092 secs 2024-04-26T20:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31962.190372 secs 2024-04-26T20:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:44 smithi033 ceph-mon[27845]: pgmap v17349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31965.124092 secs 2024-04-26T20:43:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31962.190372 secs 2024-04-26T20:43:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:44 smithi096 ceph-mon[32753]: pgmap v17349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31965.124092 secs 2024-04-26T20:43:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:46 smithi194 ceph-mon[33017]: pgmap v17350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:46 smithi033 ceph-mon[27845]: pgmap v17350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:46 smithi096 ceph-mon[32753]: pgmap v17350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:48 smithi194 ceph-mon[33017]: pgmap v17351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:48 smithi033 ceph-mon[27845]: pgmap v17351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:48 smithi096 ceph-mon[32753]: pgmap v17351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31967.190460 secs 2024-04-26T20:43:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31970.124200 secs 2024-04-26T20:43:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31967.190460 secs 2024-04-26T20:43:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31970.124200 secs 2024-04-26T20:43:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31967.190460 secs 2024-04-26T20:43:50.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31970.124200 secs 2024-04-26T20:43:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:50 smithi194 ceph-mon[33017]: pgmap v17352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:50 smithi033 ceph-mon[27845]: pgmap v17352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:50 smithi096 ceph-mon[32753]: pgmap v17352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:52 smithi194 ceph-mon[33017]: pgmap v17353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:52 smithi033 ceph-mon[27845]: pgmap v17353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:52 smithi096 ceph-mon[32753]: pgmap v17353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31975.124335 secs 2024-04-26T20:43:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31975.124335 secs 2024-04-26T20:43:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31975.124335 secs 2024-04-26T20:43:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31972.190547 secs 2024-04-26T20:43:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:54 smithi194 ceph-mon[33017]: pgmap v17354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31972.190547 secs 2024-04-26T20:43:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:54 smithi033 ceph-mon[27845]: pgmap v17354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31972.190547 secs 2024-04-26T20:43:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:54 smithi096 ceph-mon[32753]: pgmap v17354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:56.176+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:56.176+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:43:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:43:56.176+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:43:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:56 smithi194 ceph-mon[33017]: pgmap v17355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:56 smithi033 ceph-mon[27845]: pgmap v17355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:56 smithi096 ceph-mon[32753]: pgmap v17355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:58 smithi194 ceph-mon[33017]: pgmap v17356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31980.124429 secs 2024-04-26T20:43:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:58 smithi033 ceph-mon[27845]: pgmap v17356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31980.124429 secs 2024-04-26T20:43:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:58 smithi096 ceph-mon[32753]: pgmap v17356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:43:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31980.124429 secs 2024-04-26T20:44:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:43:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31977.190636 secs 2024-04-26T20:44:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:43:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31977.190636 secs 2024-04-26T20:44:00.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:43:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31977.190636 secs 2024-04-26T20:44:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:00 smithi033 ceph-mon[27845]: pgmap v17357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:00 smithi096 ceph-mon[32753]: pgmap v17357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:00 smithi194 ceph-mon[33017]: pgmap v17357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:03.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:02 smithi194 ceph-mon[33017]: pgmap v17358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:02 smithi033 ceph-mon[27845]: pgmap v17358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:02 smithi096 ceph-mon[32753]: pgmap v17358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:03 smithi033 ceph-mon[27845]: pgmap v17359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31985.124530 secs 2024-04-26T20:44:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:03 smithi096 ceph-mon[32753]: pgmap v17359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31985.124530 secs 2024-04-26T20:44:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:03 smithi194 ceph-mon[33017]: pgmap v17359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31985.124530 secs 2024-04-26T20:44:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31982.190725 secs 2024-04-26T20:44:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31982.190725 secs 2024-04-26T20:44:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31982.190725 secs 2024-04-26T20:44:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:05 smithi033 ceph-mon[27845]: pgmap v17360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:05 smithi096 ceph-mon[32753]: pgmap v17360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:05 smithi194 ceph-mon[33017]: pgmap v17360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:08 smithi194 ceph-mon[33017]: pgmap v17361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:08 smithi033 ceph-mon[27845]: pgmap v17361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:08 smithi096 ceph-mon[32753]: pgmap v17361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31987.190814 secs 2024-04-26T20:44:10.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31990.124647 secs 2024-04-26T20:44:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31987.190814 secs 2024-04-26T20:44:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31990.124647 secs 2024-04-26T20:44:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31987.190814 secs 2024-04-26T20:44:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31990.124647 secs 2024-04-26T20:44:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:10 smithi194 ceph-mon[33017]: pgmap v17362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:10 smithi033 ceph-mon[27845]: pgmap v17362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:10 smithi096 ceph-mon[32753]: pgmap v17362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:11.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:11.193+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:11.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:11.193+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:11.194 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:11.193+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:44:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:12 smithi194 ceph-mon[33017]: pgmap v17363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:12 smithi033 ceph-mon[27845]: pgmap v17363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:12 smithi096 ceph-mon[32753]: pgmap v17363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31992.190900 secs 2024-04-26T20:44:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:14 smithi194 ceph-mon[33017]: pgmap v17364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 31995.124746 secs 2024-04-26T20:44:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31992.190900 secs 2024-04-26T20:44:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:14 smithi033 ceph-mon[27845]: pgmap v17364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 31995.124746 secs 2024-04-26T20:44:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31992.190900 secs 2024-04-26T20:44:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:14 smithi096 ceph-mon[32753]: pgmap v17364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 31995.124746 secs 2024-04-26T20:44:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:16 smithi194 ceph-mon[33017]: pgmap v17365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:16 smithi033 ceph-mon[27845]: pgmap v17365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:16 smithi096 ceph-mon[32753]: pgmap v17365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:18 smithi194 ceph-mon[33017]: pgmap v17366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:18 smithi033 ceph-mon[27845]: pgmap v17366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:18 smithi096 ceph-mon[32753]: pgmap v17366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 31997.190989 secs 2024-04-26T20:44:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32000.124858 secs 2024-04-26T20:44:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 31997.190989 secs 2024-04-26T20:44:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32000.124858 secs 2024-04-26T20:44:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 31997.190989 secs 2024-04-26T20:44:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32000.124858 secs 2024-04-26T20:44:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:20 smithi194 ceph-mon[33017]: pgmap v17367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:20 smithi033 ceph-mon[27845]: pgmap v17367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:20 smithi096 ceph-mon[32753]: pgmap v17367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:22 smithi194 ceph-mon[33017]: pgmap v17368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:22 smithi033 ceph-mon[27845]: pgmap v17368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:22 smithi096 ceph-mon[32753]: pgmap v17368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32002.191085 secs 2024-04-26T20:44:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:24 smithi194 ceph-mon[33017]: pgmap v17369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32005.124975 secs 2024-04-26T20:44:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32002.191085 secs 2024-04-26T20:44:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:24 smithi033 ceph-mon[27845]: pgmap v17369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32005.124975 secs 2024-04-26T20:44:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32002.191085 secs 2024-04-26T20:44:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:24 smithi096 ceph-mon[32753]: pgmap v17369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32005.124975 secs 2024-04-26T20:44:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:26.208+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:26.209+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:26.209+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:44:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:26 smithi194 ceph-mon[33017]: pgmap v17370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:26 smithi033 ceph-mon[27845]: pgmap v17370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:26 smithi096 ceph-mon[32753]: pgmap v17370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:27.242+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:44:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:27.242+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:44:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:27.242+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 251 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:44:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:27.242+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 251 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:44:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:28 smithi194 ceph-mon[33017]: pgmap v17371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32010.125089 secs 2024-04-26T20:44:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:28 smithi033 ceph-mon[27845]: pgmap v17371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32010.125089 secs 2024-04-26T20:44:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:28 smithi096 ceph-mon[32753]: pgmap v17371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32010.125089 secs 2024-04-26T20:44:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32007.191147 secs 2024-04-26T20:44:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32007.191147 secs 2024-04-26T20:44:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32007.191147 secs 2024-04-26T20:44:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:30 smithi194 ceph-mon[33017]: pgmap v17372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:44:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:30 smithi033 ceph-mon[27845]: pgmap v17372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:44:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:30 smithi096 ceph-mon[32753]: pgmap v17372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:44:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:44:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:44:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:44:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:44:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:44:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:44:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:44:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:44:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:44:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:32 smithi194 ceph-mon[33017]: pgmap v17373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:32 smithi194 ceph-mon[33017]: pgmap v17374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:32 smithi033 ceph-mon[27845]: pgmap v17373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:32 smithi033 ceph-mon[27845]: pgmap v17374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:32 smithi096 ceph-mon[32753]: pgmap v17373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:32 smithi096 ceph-mon[32753]: pgmap v17374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32015.125207 secs 2024-04-26T20:44:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32015.125207 secs 2024-04-26T20:44:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32015.125207 secs 2024-04-26T20:44:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32012.191204 secs 2024-04-26T20:44:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:34 smithi194 ceph-mon[33017]: pgmap v17375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32012.191204 secs 2024-04-26T20:44:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:34 smithi033 ceph-mon[27845]: pgmap v17375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32012.191204 secs 2024-04-26T20:44:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:34 smithi096 ceph-mon[32753]: pgmap v17375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:36 smithi194 ceph-mon[33017]: pgmap v17376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:36 smithi033 ceph-mon[27845]: pgmap v17376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:36 smithi096 ceph-mon[32753]: pgmap v17376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:38 smithi194 ceph-mon[33017]: pgmap v17377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32020.125324 secs 2024-04-26T20:44:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:38 smithi033 ceph-mon[27845]: pgmap v17377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32020.125324 secs 2024-04-26T20:44:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:38 smithi096 ceph-mon[32753]: pgmap v17377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32020.125324 secs 2024-04-26T20:44:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32017.191274 secs 2024-04-26T20:44:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32017.191274 secs 2024-04-26T20:44:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32017.191274 secs 2024-04-26T20:44:41.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:40 smithi033 ceph-mon[27845]: pgmap v17378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:44:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:40 smithi096 ceph-mon[32753]: pgmap v17378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:44:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:41.220+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:41.220+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:41.220+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:44:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:40 smithi194 ceph-mon[33017]: pgmap v17378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:44:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:42 smithi033 ceph-mon[27845]: pgmap v17379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:44:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:42 smithi096 ceph-mon[32753]: pgmap v17379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:44:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:42 smithi194 ceph-mon[33017]: pgmap v17379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:44:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32025.125435 secs 2024-04-26T20:44:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32025.125435 secs 2024-04-26T20:44:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32025.125435 secs 2024-04-26T20:44:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32022.191353 secs 2024-04-26T20:44:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:44 smithi033 ceph-mon[27845]: pgmap v17380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32022.191353 secs 2024-04-26T20:44:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:44 smithi096 ceph-mon[32753]: pgmap v17380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32022.191353 secs 2024-04-26T20:44:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:44 smithi194 ceph-mon[33017]: pgmap v17380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:45 smithi033 ceph-mon[27845]: pgmap v17381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:45 smithi096 ceph-mon[32753]: pgmap v17381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:45 smithi194 ceph-mon[33017]: pgmap v17381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:48 smithi033 ceph-mon[27845]: pgmap v17382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:48 smithi096 ceph-mon[32753]: pgmap v17382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:48 smithi194 ceph-mon[33017]: pgmap v17382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32027.191434 secs 2024-04-26T20:44:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32030.125535 secs 2024-04-26T20:44:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32027.191434 secs 2024-04-26T20:44:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32030.125535 secs 2024-04-26T20:44:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32027.191434 secs 2024-04-26T20:44:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32030.125535 secs 2024-04-26T20:44:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:50 smithi033 ceph-mon[27845]: pgmap v17383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:50 smithi096 ceph-mon[32753]: pgmap v17383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:50 smithi194 ceph-mon[33017]: pgmap v17383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:52 smithi033 ceph-mon[27845]: pgmap v17384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:52 smithi096 ceph-mon[32753]: pgmap v17384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:52 smithi194 ceph-mon[33017]: pgmap v17384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:44:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32032.191514 secs 2024-04-26T20:44:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:54 smithi033 ceph-mon[27845]: pgmap v17385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32035.125608 secs 2024-04-26T20:44:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32032.191514 secs 2024-04-26T20:44:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:54 smithi096 ceph-mon[32753]: pgmap v17385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32035.125608 secs 2024-04-26T20:44:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32032.191514 secs 2024-04-26T20:44:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:54 smithi194 ceph-mon[33017]: pgmap v17385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32035.125608 secs 2024-04-26T20:44:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:56.225+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:56.225+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:44:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:44:56.225+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:44:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:56 smithi033 ceph-mon[27845]: pgmap v17386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:56 smithi096 ceph-mon[32753]: pgmap v17386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:56 smithi194 ceph-mon[33017]: pgmap v17386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:44:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:58 smithi033 ceph-mon[27845]: pgmap v17387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:58 smithi096 ceph-mon[32753]: pgmap v17387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:58 smithi194 ceph-mon[33017]: pgmap v17387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:44:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32037.191600 secs 2024-04-26T20:44:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:44:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32040.125731 secs 2024-04-26T20:44:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32037.191600 secs 2024-04-26T20:44:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:44:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32040.125731 secs 2024-04-26T20:44:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32037.191600 secs 2024-04-26T20:44:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:44:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32040.125731 secs 2024-04-26T20:45:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:00 smithi096 ceph-mon[32753]: pgmap v17388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:00 smithi194 ceph-mon[33017]: pgmap v17388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:00 smithi033 ceph-mon[27845]: pgmap v17388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:02 smithi096 ceph-mon[32753]: pgmap v17389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:02 smithi194 ceph-mon[33017]: pgmap v17389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:02 smithi033 ceph-mon[27845]: pgmap v17389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:04.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32042.191684 secs 2024-04-26T20:45:04.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:04 smithi033 ceph-mon[27845]: pgmap v17390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:04.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32045.125855 secs 2024-04-26T20:45:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32042.191684 secs 2024-04-26T20:45:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:04 smithi194 ceph-mon[33017]: pgmap v17390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32045.125855 secs 2024-04-26T20:45:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32042.191684 secs 2024-04-26T20:45:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:04 smithi096 ceph-mon[32753]: pgmap v17390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32045.125855 secs 2024-04-26T20:45:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:06 smithi194 ceph-mon[33017]: pgmap v17391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:06 smithi033 ceph-mon[27845]: pgmap v17391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:06 smithi096 ceph-mon[32753]: pgmap v17391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:08 smithi194 ceph-mon[33017]: pgmap v17392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:08 smithi033 ceph-mon[27845]: pgmap v17392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:08 smithi096 ceph-mon[32753]: pgmap v17392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32047.191771 secs 2024-04-26T20:45:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32050.125991 secs 2024-04-26T20:45:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32047.191771 secs 2024-04-26T20:45:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32050.125991 secs 2024-04-26T20:45:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32047.191771 secs 2024-04-26T20:45:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32050.125991 secs 2024-04-26T20:45:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:10 smithi194 ceph-mon[33017]: pgmap v17393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:10 smithi033 ceph-mon[27845]: pgmap v17393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:10 smithi096 ceph-mon[32753]: pgmap v17393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:11.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:11.240+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:11.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:11.240+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:45:11.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:11.240+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:12 smithi194 ceph-mon[33017]: pgmap v17394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:12 smithi033 ceph-mon[27845]: pgmap v17394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:12 smithi096 ceph-mon[32753]: pgmap v17394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32052.191855 secs 2024-04-26T20:45:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:14 smithi194 ceph-mon[33017]: pgmap v17395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32055.126129 secs 2024-04-26T20:45:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32052.191855 secs 2024-04-26T20:45:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:14 smithi033 ceph-mon[27845]: pgmap v17395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32055.126129 secs 2024-04-26T20:45:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32052.191855 secs 2024-04-26T20:45:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:14 smithi096 ceph-mon[32753]: pgmap v17395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32055.126129 secs 2024-04-26T20:45:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:16 smithi194 ceph-mon[33017]: pgmap v17396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:16 smithi033 ceph-mon[27845]: pgmap v17396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:16 smithi096 ceph-mon[32753]: pgmap v17396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:18 smithi194 ceph-mon[33017]: pgmap v17397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:18 smithi033 ceph-mon[27845]: pgmap v17397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:18 smithi096 ceph-mon[32753]: pgmap v17397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32057.191935 secs 2024-04-26T20:45:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32060.126225 secs 2024-04-26T20:45:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32057.191935 secs 2024-04-26T20:45:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32060.126225 secs 2024-04-26T20:45:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32057.191935 secs 2024-04-26T20:45:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32060.126225 secs 2024-04-26T20:45:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:20 smithi194 ceph-mon[33017]: pgmap v17398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:20 smithi033 ceph-mon[27845]: pgmap v17398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:20 smithi096 ceph-mon[32753]: pgmap v17398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:22 smithi194 ceph-mon[33017]: pgmap v17399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:22 smithi033 ceph-mon[27845]: pgmap v17399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:22 smithi096 ceph-mon[32753]: pgmap v17399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32062.192003 secs 2024-04-26T20:45:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:24 smithi194 ceph-mon[33017]: pgmap v17400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32065.126302 secs 2024-04-26T20:45:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32062.192003 secs 2024-04-26T20:45:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:24 smithi033 ceph-mon[27845]: pgmap v17400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32065.126302 secs 2024-04-26T20:45:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32062.192003 secs 2024-04-26T20:45:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:24 smithi096 ceph-mon[32753]: pgmap v17400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32065.126302 secs 2024-04-26T20:45:26.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:26.251+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:26.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:26.251+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:45:26.252 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:26.251+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:26 smithi194 ceph-mon[33017]: pgmap v17401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:26 smithi033 ceph-mon[27845]: pgmap v17401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:26 smithi096 ceph-mon[32753]: pgmap v17401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:28 smithi194 ceph-mon[33017]: pgmap v17402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:28 smithi033 ceph-mon[27845]: pgmap v17402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:28 smithi096 ceph-mon[32753]: pgmap v17402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32067.192087 secs 2024-04-26T20:45:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32070.126415 secs 2024-04-26T20:45:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32067.192087 secs 2024-04-26T20:45:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32070.126415 secs 2024-04-26T20:45:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32067.192087 secs 2024-04-26T20:45:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32070.126415 secs 2024-04-26T20:45:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:30 smithi194 ceph-mon[33017]: pgmap v17403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:30 smithi033 ceph-mon[27845]: pgmap v17403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:30 smithi096 ceph-mon[32753]: pgmap v17403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:32 smithi194 ceph-mon[33017]: pgmap v17404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:45:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:32 smithi033 ceph-mon[27845]: pgmap v17404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:45:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:32 smithi096 ceph-mon[32753]: pgmap v17404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:45:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:45:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:45:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:45:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:45:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:45:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:45:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:45:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:45:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:45:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32072.192174 secs 2024-04-26T20:45:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:34 smithi033 ceph-mon[27845]: pgmap v17405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:34 smithi033 ceph-mon[27845]: pgmap v17406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:34.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32075.126536 secs 2024-04-26T20:45:34.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32072.192174 secs 2024-04-26T20:45:34.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:34 smithi096 ceph-mon[32753]: pgmap v17405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:34.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:34 smithi096 ceph-mon[32753]: pgmap v17406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:34.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32075.126536 secs 2024-04-26T20:45:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32072.192174 secs 2024-04-26T20:45:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:34 smithi194 ceph-mon[33017]: pgmap v17405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:34 smithi194 ceph-mon[33017]: pgmap v17406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32075.126536 secs 2024-04-26T20:45:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:36 smithi194 ceph-mon[33017]: pgmap v17407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:36 smithi033 ceph-mon[27845]: pgmap v17407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:36 smithi096 ceph-mon[32753]: pgmap v17407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:38 smithi194 ceph-mon[33017]: pgmap v17408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:38 smithi033 ceph-mon[27845]: pgmap v17408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:38 smithi096 ceph-mon[32753]: pgmap v17408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32077.192235 secs 2024-04-26T20:45:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32080.126653 secs 2024-04-26T20:45:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32077.192235 secs 2024-04-26T20:45:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32080.126653 secs 2024-04-26T20:45:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32077.192235 secs 2024-04-26T20:45:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32080.126653 secs 2024-04-26T20:45:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:40 smithi194 ceph-mon[33017]: pgmap v17409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T20:45:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:40 smithi033 ceph-mon[27845]: pgmap v17409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T20:45:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:40 smithi096 ceph-mon[32753]: pgmap v17409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T20:45:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:41.255+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:41.255+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:45:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:41.255+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:42 smithi194 ceph-mon[33017]: pgmap v17410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:42 smithi033 ceph-mon[27845]: pgmap v17410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:42 smithi096 ceph-mon[32753]: pgmap v17410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:45:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32082.192317 secs 2024-04-26T20:45:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:44 smithi194 ceph-mon[33017]: pgmap v17411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:45:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32085.126770 secs 2024-04-26T20:45:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32082.192317 secs 2024-04-26T20:45:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:44 smithi033 ceph-mon[27845]: pgmap v17411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:45:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32085.126770 secs 2024-04-26T20:45:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32082.192317 secs 2024-04-26T20:45:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:44 smithi096 ceph-mon[32753]: pgmap v17411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:45:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32085.126770 secs 2024-04-26T20:45:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:46 smithi194 ceph-mon[33017]: pgmap v17412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:46 smithi033 ceph-mon[27845]: pgmap v17412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:46 smithi096 ceph-mon[32753]: pgmap v17412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:48 smithi194 ceph-mon[33017]: pgmap v17413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:48 smithi033 ceph-mon[27845]: pgmap v17413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:48 smithi096 ceph-mon[32753]: pgmap v17413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32087.192399 secs 2024-04-26T20:45:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32090.126887 secs 2024-04-26T20:45:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32087.192399 secs 2024-04-26T20:45:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32090.126887 secs 2024-04-26T20:45:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32087.192399 secs 2024-04-26T20:45:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32090.126887 secs 2024-04-26T20:45:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:50 smithi033 ceph-mon[27845]: pgmap v17414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:50 smithi096 ceph-mon[32753]: pgmap v17414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:50 smithi194 ceph-mon[33017]: pgmap v17414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:52 smithi033 ceph-mon[27845]: pgmap v17415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:52 smithi096 ceph-mon[32753]: pgmap v17415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:52 smithi194 ceph-mon[33017]: pgmap v17415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32092.192481 secs 2024-04-26T20:45:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:54 smithi033 ceph-mon[27845]: pgmap v17416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32095.127008 secs 2024-04-26T20:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32092.192481 secs 2024-04-26T20:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:54 smithi096 ceph-mon[32753]: pgmap v17416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32095.127008 secs 2024-04-26T20:45:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32092.192481 secs 2024-04-26T20:45:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:54 smithi194 ceph-mon[33017]: pgmap v17416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32095.127008 secs 2024-04-26T20:45:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:56.261+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:56.261+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:45:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:56.261+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:45:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:56 smithi033 ceph-mon[27845]: pgmap v17417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:56 smithi096 ceph-mon[32753]: pgmap v17417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:56 smithi194 ceph-mon[33017]: pgmap v17417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:45:57.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:57.256+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:45:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:57.256+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:45:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:57.256+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 252 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T20:45:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:45:57.257+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 252 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T20:45:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:58 smithi033 ceph-mon[27845]: pgmap v17418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:58 smithi096 ceph-mon[32753]: pgmap v17418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:58 smithi194 ceph-mon[33017]: pgmap v17418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:45:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32097.192566 secs 2024-04-26T20:45:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32100.127080 secs 2024-04-26T20:45:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32097.192566 secs 2024-04-26T20:45:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32100.127080 secs 2024-04-26T20:46:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32097.192566 secs 2024-04-26T20:46:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32100.127080 secs 2024-04-26T20:46:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:00 smithi033 ceph-mon[27845]: pgmap v17419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:00 smithi096 ceph-mon[32753]: pgmap v17419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:00 smithi194 ceph-mon[33017]: pgmap v17419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:02 smithi033 ceph-mon[27845]: pgmap v17420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:02 smithi096 ceph-mon[32753]: pgmap v17420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:02 smithi194 ceph-mon[33017]: pgmap v17420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:04.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32102.192650 secs 2024-04-26T20:46:04.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:04 smithi194 ceph-mon[33017]: pgmap v17421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:04.906 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32105.127200 secs 2024-04-26T20:46:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32102.192650 secs 2024-04-26T20:46:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:04 smithi033 ceph-mon[27845]: pgmap v17421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32105.127200 secs 2024-04-26T20:46:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32102.192650 secs 2024-04-26T20:46:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:04 smithi096 ceph-mon[32753]: pgmap v17421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32105.127200 secs 2024-04-26T20:46:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:06 smithi033 ceph-mon[27845]: pgmap v17422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:06 smithi096 ceph-mon[32753]: pgmap v17422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:06 smithi194 ceph-mon[33017]: pgmap v17422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:08 smithi033 ceph-mon[27845]: pgmap v17423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:08 smithi096 ceph-mon[32753]: pgmap v17423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:08 smithi194 ceph-mon[33017]: pgmap v17423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32107.192731 secs 2024-04-26T20:46:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32110.127310 secs 2024-04-26T20:46:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32107.192731 secs 2024-04-26T20:46:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32110.127310 secs 2024-04-26T20:46:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32107.192731 secs 2024-04-26T20:46:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32110.127310 secs 2024-04-26T20:46:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:10 smithi096 ceph-mon[32753]: pgmap v17424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:10 smithi194 ceph-mon[33017]: pgmap v17424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:10 smithi033 ceph-mon[27845]: pgmap v17424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:11.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:11.267+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:11.268+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:11.268+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:46:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:12 smithi096 ceph-mon[32753]: pgmap v17425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:12 smithi194 ceph-mon[33017]: pgmap v17425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:12 smithi033 ceph-mon[27845]: pgmap v17425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32112.192812 secs 2024-04-26T20:46:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:14 smithi194 ceph-mon[33017]: pgmap v17426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32115.127435 secs 2024-04-26T20:46:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32112.192812 secs 2024-04-26T20:46:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:14 smithi033 ceph-mon[27845]: pgmap v17426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32115.127435 secs 2024-04-26T20:46:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32112.192812 secs 2024-04-26T20:46:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:14 smithi096 ceph-mon[32753]: pgmap v17426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32115.127435 secs 2024-04-26T20:46:17.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:16 smithi194 ceph-mon[33017]: pgmap v17427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:16 smithi033 ceph-mon[27845]: pgmap v17427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:16 smithi096 ceph-mon[32753]: pgmap v17427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:18 smithi194 ceph-mon[33017]: pgmap v17428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:18 smithi033 ceph-mon[27845]: pgmap v17428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:18 smithi096 ceph-mon[32753]: pgmap v17428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32117.192872 secs 2024-04-26T20:46:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32120.127504 secs 2024-04-26T20:46:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32117.192872 secs 2024-04-26T20:46:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32120.127504 secs 2024-04-26T20:46:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32117.192872 secs 2024-04-26T20:46:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32120.127504 secs 2024-04-26T20:46:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:20 smithi194 ceph-mon[33017]: pgmap v17429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:20 smithi033 ceph-mon[27845]: pgmap v17429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:20 smithi096 ceph-mon[32753]: pgmap v17429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:22 smithi194 ceph-mon[33017]: pgmap v17430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:22 smithi033 ceph-mon[27845]: pgmap v17430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:22 smithi096 ceph-mon[32753]: pgmap v17430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32122.192954 secs 2024-04-26T20:46:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:24 smithi194 ceph-mon[33017]: pgmap v17431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32125.127621 secs 2024-04-26T20:46:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32122.192954 secs 2024-04-26T20:46:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:24 smithi033 ceph-mon[27845]: pgmap v17431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32125.127621 secs 2024-04-26T20:46:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32122.192954 secs 2024-04-26T20:46:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:24 smithi096 ceph-mon[32753]: pgmap v17431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32125.127621 secs 2024-04-26T20:46:26.281 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:26.281+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:26.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:26.281+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:46:26.282 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:26.281+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:26 smithi194 ceph-mon[33017]: pgmap v17432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:26 smithi033 ceph-mon[27845]: pgmap v17432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:26 smithi096 ceph-mon[32753]: pgmap v17432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:28 smithi194 ceph-mon[33017]: pgmap v17433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:28 smithi033 ceph-mon[27845]: pgmap v17433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:28 smithi096 ceph-mon[32753]: pgmap v17433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32127.192977 secs 2024-04-26T20:46:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32130.127735 secs 2024-04-26T20:46:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32127.192977 secs 2024-04-26T20:46:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32130.127735 secs 2024-04-26T20:46:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32127.192977 secs 2024-04-26T20:46:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32130.127735 secs 2024-04-26T20:46:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:30 smithi194 ceph-mon[33017]: pgmap v17434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:30 smithi033 ceph-mon[27845]: pgmap v17434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:30 smithi096 ceph-mon[32753]: pgmap v17434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:32 smithi194 ceph-mon[33017]: pgmap v17435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:32 smithi033 ceph-mon[27845]: pgmap v17435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:32 smithi096 ceph-mon[32753]: pgmap v17435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:34.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:46:34.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:46:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:46:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32132.193062 secs 2024-04-26T20:46:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:34 smithi194 ceph-mon[33017]: pgmap v17436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32135.127825 secs 2024-04-26T20:46:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:46:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32132.193062 secs 2024-04-26T20:46:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:34 smithi033 ceph-mon[27845]: pgmap v17436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32135.127825 secs 2024-04-26T20:46:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:46:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32132.193062 secs 2024-04-26T20:46:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:34 smithi096 ceph-mon[32753]: pgmap v17436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32135.127825 secs 2024-04-26T20:46:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:35 smithi194 ceph-mon[33017]: pgmap v17437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:35 smithi194 ceph-mon[33017]: pgmap v17438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:46:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:46:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:35 smithi033 ceph-mon[27845]: pgmap v17437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:46:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:35 smithi033 ceph-mon[27845]: pgmap v17438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:36.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:46:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:46:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:35 smithi096 ceph-mon[32753]: pgmap v17437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:46:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:35 smithi096 ceph-mon[32753]: pgmap v17438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:46:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:37 smithi194 ceph-mon[33017]: pgmap v17439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:37 smithi033 ceph-mon[27845]: pgmap v17439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:37 smithi096 ceph-mon[32753]: pgmap v17439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32140.127947 secs 2024-04-26T20:46:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32140.127947 secs 2024-04-26T20:46:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32140.127947 secs 2024-04-26T20:46:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32137.193146 secs 2024-04-26T20:46:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:39 smithi194 ceph-mon[33017]: pgmap v17440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32137.193146 secs 2024-04-26T20:46:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:39 smithi033 ceph-mon[27845]: pgmap v17440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32137.193146 secs 2024-04-26T20:46:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:39 smithi096 ceph-mon[32753]: pgmap v17440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:41.283+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:41.283+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:46:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:41.283+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:41 smithi033 ceph-mon[27845]: pgmap v17441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:41 smithi096 ceph-mon[32753]: pgmap v17441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:41 smithi194 ceph-mon[33017]: pgmap v17441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:43 smithi033 ceph-mon[27845]: pgmap v17442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32145.128071 secs 2024-04-26T20:46:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:43 smithi096 ceph-mon[32753]: pgmap v17442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32145.128071 secs 2024-04-26T20:46:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:43 smithi194 ceph-mon[33017]: pgmap v17442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:46:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32145.128071 secs 2024-04-26T20:46:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32142.193230 secs 2024-04-26T20:46:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32142.193230 secs 2024-04-26T20:46:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32142.193230 secs 2024-04-26T20:46:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:45 smithi033 ceph-mon[27845]: pgmap v17443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:46:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:45 smithi096 ceph-mon[32753]: pgmap v17443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:46:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:45 smithi194 ceph-mon[33017]: pgmap v17443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:46:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:47 smithi033 ceph-mon[27845]: pgmap v17444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:47 smithi096 ceph-mon[32753]: pgmap v17444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:47 smithi194 ceph-mon[33017]: pgmap v17444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32150.128186 secs 2024-04-26T20:46:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32150.128186 secs 2024-04-26T20:46:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32150.128186 secs 2024-04-26T20:46:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32147.193293 secs 2024-04-26T20:46:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:49 smithi033 ceph-mon[27845]: pgmap v17445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32147.193293 secs 2024-04-26T20:46:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:49 smithi096 ceph-mon[32753]: pgmap v17445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32147.193293 secs 2024-04-26T20:46:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:49 smithi194 ceph-mon[33017]: pgmap v17445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:51 smithi033 ceph-mon[27845]: pgmap v17446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:51 smithi096 ceph-mon[32753]: pgmap v17446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:51 smithi194 ceph-mon[33017]: pgmap v17446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:53 smithi033 ceph-mon[27845]: pgmap v17447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32155.128302 secs 2024-04-26T20:46:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:53 smithi096 ceph-mon[32753]: pgmap v17447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32155.128302 secs 2024-04-26T20:46:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:53 smithi194 ceph-mon[33017]: pgmap v17447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32155.128302 secs 2024-04-26T20:46:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32152.193353 secs 2024-04-26T20:46:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:54 smithi033 ceph-mon[27845]: pgmap v17448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32152.193353 secs 2024-04-26T20:46:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:54 smithi096 ceph-mon[32753]: pgmap v17448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32152.193353 secs 2024-04-26T20:46:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:54 smithi194 ceph-mon[33017]: pgmap v17448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:46:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:56.295+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:56.295+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:46:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:46:56.295+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:46:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:57 smithi194 ceph-mon[33017]: pgmap v17449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:57 smithi033 ceph-mon[27845]: pgmap v17449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:57 smithi096 ceph-mon[32753]: pgmap v17449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:46:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32160.128383 secs 2024-04-26T20:46:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32160.128383 secs 2024-04-26T20:46:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32160.128383 secs 2024-04-26T20:47:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32157.193414 secs 2024-04-26T20:47:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:46:59 smithi033 ceph-mon[27845]: pgmap v17450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32157.193414 secs 2024-04-26T20:47:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:46:59 smithi096 ceph-mon[32753]: pgmap v17450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32157.193414 secs 2024-04-26T20:47:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:46:59 smithi194 ceph-mon[33017]: pgmap v17450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:01 smithi033 ceph-mon[27845]: pgmap v17451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:01 smithi096 ceph-mon[32753]: pgmap v17451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:01 smithi194 ceph-mon[33017]: pgmap v17451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:04.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:03 smithi194 ceph-mon[33017]: pgmap v17452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:04.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32165.128494 secs 2024-04-26T20:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:03 smithi033 ceph-mon[27845]: pgmap v17452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32165.128494 secs 2024-04-26T20:47:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:03 smithi096 ceph-mon[32753]: pgmap v17452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32165.128494 secs 2024-04-26T20:47:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32162.193495 secs 2024-04-26T20:47:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32162.193495 secs 2024-04-26T20:47:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32162.193495 secs 2024-04-26T20:47:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:05 smithi033 ceph-mon[27845]: pgmap v17453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:05 smithi096 ceph-mon[32753]: pgmap v17453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:05 smithi194 ceph-mon[33017]: pgmap v17453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:07 smithi033 ceph-mon[27845]: pgmap v17454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:07 smithi096 ceph-mon[32753]: pgmap v17454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:07 smithi194 ceph-mon[33017]: pgmap v17454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32170.128601 secs 2024-04-26T20:47:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32170.128601 secs 2024-04-26T20:47:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32170.128601 secs 2024-04-26T20:47:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32167.193581 secs 2024-04-26T20:47:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:09 smithi033 ceph-mon[27845]: pgmap v17455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32167.193581 secs 2024-04-26T20:47:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:09 smithi096 ceph-mon[32753]: pgmap v17455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32167.193581 secs 2024-04-26T20:47:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:09 smithi194 ceph-mon[33017]: pgmap v17455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:10 smithi033 ceph-mon[27845]: pgmap v17456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:10 smithi096 ceph-mon[32753]: pgmap v17456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:11.311+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:11.312+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:11.312+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:47:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:10 smithi194 ceph-mon[33017]: pgmap v17456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:13 smithi033 ceph-mon[27845]: pgmap v17457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32175.128717 secs 2024-04-26T20:47:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:13 smithi096 ceph-mon[32753]: pgmap v17457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32175.128717 secs 2024-04-26T20:47:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:13 smithi194 ceph-mon[33017]: pgmap v17457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32175.128717 secs 2024-04-26T20:47:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32172.193659 secs 2024-04-26T20:47:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32172.193659 secs 2024-04-26T20:47:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32172.193659 secs 2024-04-26T20:47:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:15 smithi033 ceph-mon[27845]: pgmap v17458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:15 smithi096 ceph-mon[32753]: pgmap v17458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:15 smithi194 ceph-mon[33017]: pgmap v17458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:17 smithi033 ceph-mon[27845]: pgmap v17459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:17 smithi096 ceph-mon[32753]: pgmap v17459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:17 smithi194 ceph-mon[33017]: pgmap v17459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32180.128833 secs 2024-04-26T20:47:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32180.128833 secs 2024-04-26T20:47:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32180.128833 secs 2024-04-26T20:47:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32177.193743 secs 2024-04-26T20:47:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:19 smithi033 ceph-mon[27845]: pgmap v17460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32177.193743 secs 2024-04-26T20:47:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:19 smithi096 ceph-mon[32753]: pgmap v17460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32177.193743 secs 2024-04-26T20:47:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:19 smithi194 ceph-mon[33017]: pgmap v17460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:21 smithi033 ceph-mon[27845]: pgmap v17461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:21 smithi096 ceph-mon[32753]: pgmap v17461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:21 smithi194 ceph-mon[33017]: pgmap v17461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:23 smithi033 ceph-mon[27845]: pgmap v17462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32185.128946 secs 2024-04-26T20:47:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:23 smithi096 ceph-mon[32753]: pgmap v17462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32185.128946 secs 2024-04-26T20:47:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:23 smithi194 ceph-mon[33017]: pgmap v17462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32185.128946 secs 2024-04-26T20:47:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32182.193826 secs 2024-04-26T20:47:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32182.193826 secs 2024-04-26T20:47:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32182.193826 secs 2024-04-26T20:47:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:25 smithi033 ceph-mon[27845]: pgmap v17463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:25 smithi096 ceph-mon[32753]: pgmap v17463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:26.325+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:26.326+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:26.326+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:47:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:25 smithi194 ceph-mon[33017]: pgmap v17463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:26 smithi033 ceph-mon[27845]: pgmap v17464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:26 smithi096 ceph-mon[32753]: pgmap v17464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:27.271+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:47:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:27.271+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:47:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:27.272+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 253 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:47:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:27.272+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 253 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:47:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:26 smithi194 ceph-mon[33017]: pgmap v17464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32190.129060 secs 2024-04-26T20:47:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32190.129060 secs 2024-04-26T20:47:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32190.129060 secs 2024-04-26T20:47:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32187.193911 secs 2024-04-26T20:47:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:29 smithi033 ceph-mon[27845]: pgmap v17465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32187.193911 secs 2024-04-26T20:47:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:29 smithi096 ceph-mon[32753]: pgmap v17465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32187.193911 secs 2024-04-26T20:47:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:29 smithi194 ceph-mon[33017]: pgmap v17465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:32.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:31 smithi033 ceph-mon[27845]: pgmap v17466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:31 smithi096 ceph-mon[32753]: pgmap v17466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:31 smithi194 ceph-mon[33017]: pgmap v17466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:34.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:33 smithi194 ceph-mon[33017]: pgmap v17467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:34.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32195.129178 secs 2024-04-26T20:47:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:33 smithi033 ceph-mon[27845]: pgmap v17467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32195.129178 secs 2024-04-26T20:47:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:33 smithi096 ceph-mon[32753]: pgmap v17467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32195.129178 secs 2024-04-26T20:47:35.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32192.193996 secs 2024-04-26T20:47:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32192.193996 secs 2024-04-26T20:47:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32192.193996 secs 2024-04-26T20:47:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:35 smithi033 ceph-mon[27845]: pgmap v17468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:47:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:35 smithi096 ceph-mon[32753]: pgmap v17468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:47:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:35 smithi194 ceph-mon[33017]: pgmap v17468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:47:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:47:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:47:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:47:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:47:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:47:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:47:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:47:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:47:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:47:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:37 smithi033 ceph-mon[27845]: pgmap v17469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:37 smithi033 ceph-mon[27845]: pgmap v17470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:37 smithi096 ceph-mon[32753]: pgmap v17469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:37 smithi096 ceph-mon[32753]: pgmap v17470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:37 smithi194 ceph-mon[33017]: pgmap v17469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:37 smithi194 ceph-mon[33017]: pgmap v17470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:38 smithi033 ceph-mon[27845]: pgmap v17471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32200.129289 secs 2024-04-26T20:47:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:38 smithi096 ceph-mon[32753]: pgmap v17471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32200.129289 secs 2024-04-26T20:47:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:38 smithi194 ceph-mon[33017]: pgmap v17471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32200.129289 secs 2024-04-26T20:47:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32197.194036 secs 2024-04-26T20:47:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32197.194036 secs 2024-04-26T20:47:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32197.194036 secs 2024-04-26T20:47:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:41.336+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:41.336+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:41.336+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:47:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:40 smithi194 ceph-mon[33017]: pgmap v17472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:40 smithi033 ceph-mon[27845]: pgmap v17472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:40 smithi096 ceph-mon[32753]: pgmap v17472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:43 smithi194 ceph-mon[33017]: pgmap v17473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:43 smithi033 ceph-mon[27845]: pgmap v17473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:43 smithi096 ceph-mon[32753]: pgmap v17473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32202.194118 secs 2024-04-26T20:47:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32205.129414 secs 2024-04-26T20:47:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32202.194118 secs 2024-04-26T20:47:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32205.129414 secs 2024-04-26T20:47:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32202.194118 secs 2024-04-26T20:47:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32205.129414 secs 2024-04-26T20:47:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:45 smithi194 ceph-mon[33017]: pgmap v17474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:45 smithi033 ceph-mon[27845]: pgmap v17474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:45 smithi096 ceph-mon[32753]: pgmap v17474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:47:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:47 smithi194 ceph-mon[33017]: pgmap v17475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:47:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:47 smithi033 ceph-mon[27845]: pgmap v17475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:47:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:47 smithi096 ceph-mon[32753]: pgmap v17475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:47:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32207.194177 secs 2024-04-26T20:47:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:49 smithi194 ceph-mon[33017]: pgmap v17476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32210.129508 secs 2024-04-26T20:47:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32207.194177 secs 2024-04-26T20:47:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:49 smithi033 ceph-mon[27845]: pgmap v17476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32210.129508 secs 2024-04-26T20:47:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32207.194177 secs 2024-04-26T20:47:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:49 smithi096 ceph-mon[32753]: pgmap v17476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32210.129508 secs 2024-04-26T20:47:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:51 smithi194 ceph-mon[33017]: pgmap v17477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:51 smithi033 ceph-mon[27845]: pgmap v17477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:51 smithi096 ceph-mon[32753]: pgmap v17477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:53 smithi194 ceph-mon[33017]: pgmap v17478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:53 smithi033 ceph-mon[27845]: pgmap v17478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:53 smithi096 ceph-mon[32753]: pgmap v17478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32212.194282 secs 2024-04-26T20:47:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32215.129629 secs 2024-04-26T20:47:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32212.194282 secs 2024-04-26T20:47:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32215.129629 secs 2024-04-26T20:47:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32212.194282 secs 2024-04-26T20:47:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32215.129629 secs 2024-04-26T20:47:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:55 smithi194 ceph-mon[33017]: pgmap v17479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:55.949 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:55 smithi096 ceph-mon[32753]: pgmap v17479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:55.949 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:55 smithi033 ceph-mon[27845]: pgmap v17479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:47:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:56.344+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:56.344+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:47:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:47:56.344+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:47:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:57 smithi194 ceph-mon[33017]: pgmap v17480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:57 smithi033 ceph-mon[27845]: pgmap v17480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:57 smithi096 ceph-mon[32753]: pgmap v17480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32217.194343 secs 2024-04-26T20:47:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:59 smithi194 ceph-mon[33017]: pgmap v17481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:47:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32220.129770 secs 2024-04-26T20:47:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32217.194343 secs 2024-04-26T20:47:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:59 smithi033 ceph-mon[27845]: pgmap v17481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:47:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32220.129770 secs 2024-04-26T20:47:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32217.194343 secs 2024-04-26T20:47:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:59 smithi096 ceph-mon[32753]: pgmap v17481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:47:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:47:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32220.129770 secs 2024-04-26T20:48:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:01 smithi194 ceph-mon[33017]: pgmap v17482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:01 smithi033 ceph-mon[27845]: pgmap v17482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:01 smithi096 ceph-mon[32753]: pgmap v17482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:03 smithi033 ceph-mon[27845]: pgmap v17483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:03 smithi096 ceph-mon[32753]: pgmap v17483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:03 smithi194 ceph-mon[33017]: pgmap v17483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32222.194426 secs 2024-04-26T20:48:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32225.129927 secs 2024-04-26T20:48:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32222.194426 secs 2024-04-26T20:48:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32225.129927 secs 2024-04-26T20:48:05.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32222.194426 secs 2024-04-26T20:48:05.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32225.129927 secs 2024-04-26T20:48:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:05 smithi033 ceph-mon[27845]: pgmap v17484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:05 smithi096 ceph-mon[32753]: pgmap v17484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:05 smithi194 ceph-mon[33017]: pgmap v17484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:07 smithi033 ceph-mon[27845]: pgmap v17485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:07 smithi096 ceph-mon[32753]: pgmap v17485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:07 smithi194 ceph-mon[33017]: pgmap v17485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32227.194510 secs 2024-04-26T20:48:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:09 smithi033 ceph-mon[27845]: pgmap v17486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32230.130050 secs 2024-04-26T20:48:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32227.194510 secs 2024-04-26T20:48:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:09 smithi096 ceph-mon[32753]: pgmap v17486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32230.130050 secs 2024-04-26T20:48:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32227.194510 secs 2024-04-26T20:48:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:09 smithi194 ceph-mon[33017]: pgmap v17486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32230.130050 secs 2024-04-26T20:48:11.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:11.358+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:11.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:11.359+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:11.359+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:48:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:11 smithi033 ceph-mon[27845]: pgmap v17487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:11 smithi096 ceph-mon[32753]: pgmap v17487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:11 smithi194 ceph-mon[33017]: pgmap v17487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:13 smithi033 ceph-mon[27845]: pgmap v17488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:13 smithi096 ceph-mon[32753]: pgmap v17488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:13 smithi194 ceph-mon[33017]: pgmap v17488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32232.194568 secs 2024-04-26T20:48:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32235.130176 secs 2024-04-26T20:48:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32232.194568 secs 2024-04-26T20:48:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32235.130176 secs 2024-04-26T20:48:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32232.194568 secs 2024-04-26T20:48:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32235.130176 secs 2024-04-26T20:48:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:15 smithi033 ceph-mon[27845]: pgmap v17489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:15 smithi096 ceph-mon[32753]: pgmap v17489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:15 smithi194 ceph-mon[33017]: pgmap v17489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:17 smithi033 ceph-mon[27845]: pgmap v17490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:17 smithi096 ceph-mon[32753]: pgmap v17490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:17 smithi194 ceph-mon[33017]: pgmap v17490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32237.194653 secs 2024-04-26T20:48:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:19 smithi033 ceph-mon[27845]: pgmap v17491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32240.130300 secs 2024-04-26T20:48:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32237.194653 secs 2024-04-26T20:48:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:19 smithi096 ceph-mon[32753]: pgmap v17491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32240.130300 secs 2024-04-26T20:48:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32237.194653 secs 2024-04-26T20:48:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:19 smithi194 ceph-mon[33017]: pgmap v17491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32240.130300 secs 2024-04-26T20:48:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:21 smithi033 ceph-mon[27845]: pgmap v17492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:21 smithi096 ceph-mon[32753]: pgmap v17492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:21 smithi194 ceph-mon[33017]: pgmap v17492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:23 smithi033 ceph-mon[27845]: pgmap v17493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:23 smithi096 ceph-mon[32753]: pgmap v17493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:23 smithi194 ceph-mon[33017]: pgmap v17493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32242.194737 secs 2024-04-26T20:48:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32245.130442 secs 2024-04-26T20:48:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32242.194737 secs 2024-04-26T20:48:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32245.130442 secs 2024-04-26T20:48:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32242.194737 secs 2024-04-26T20:48:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32245.130442 secs 2024-04-26T20:48:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:25 smithi194 ceph-mon[33017]: pgmap v17494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:25 smithi033 ceph-mon[27845]: pgmap v17494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:25 smithi096 ceph-mon[32753]: pgmap v17494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:26.374+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:26.374+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:26.374+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:48:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:27 smithi194 ceph-mon[33017]: pgmap v17495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:27 smithi033 ceph-mon[27845]: pgmap v17495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:27 smithi096 ceph-mon[32753]: pgmap v17495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32247.194820 secs 2024-04-26T20:48:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:29 smithi194 ceph-mon[33017]: pgmap v17496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32250.130501 secs 2024-04-26T20:48:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32247.194820 secs 2024-04-26T20:48:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:29 smithi033 ceph-mon[27845]: pgmap v17496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32250.130501 secs 2024-04-26T20:48:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32247.194820 secs 2024-04-26T20:48:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:29 smithi096 ceph-mon[32753]: pgmap v17496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:30.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32250.130501 secs 2024-04-26T20:48:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:31 smithi194 ceph-mon[33017]: pgmap v17497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:31 smithi033 ceph-mon[27845]: pgmap v17497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:31 smithi096 ceph-mon[32753]: pgmap v17497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:34.096 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:33 smithi033 ceph-mon[27845]: pgmap v17498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:33 smithi194 ceph-mon[33017]: pgmap v17498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:33 smithi096 ceph-mon[32753]: pgmap v17498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:35.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32252.194902 secs 2024-04-26T20:48:35.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32255.130632 secs 2024-04-26T20:48:35.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32252.194902 secs 2024-04-26T20:48:35.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32255.130632 secs 2024-04-26T20:48:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32252.194902 secs 2024-04-26T20:48:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32255.130632 secs 2024-04-26T20:48:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:35 smithi194 ceph-mon[33017]: pgmap v17499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:35 smithi033 ceph-mon[27845]: pgmap v17499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:35 smithi096 ceph-mon[32753]: pgmap v17499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:37 smithi194 ceph-mon[33017]: pgmap v17500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:48:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:37 smithi033 ceph-mon[27845]: pgmap v17500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:48:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:37 smithi096 ceph-mon[32753]: pgmap v17500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:48:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:48:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:48:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:48:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:48:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:48:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:48:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:48:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32257.195004 secs 2024-04-26T20:48:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:39 smithi194 ceph-mon[33017]: pgmap v17501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:39 smithi194 ceph-mon[33017]: pgmap v17502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32260.130716 secs 2024-04-26T20:48:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32257.195004 secs 2024-04-26T20:48:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:39 smithi033 ceph-mon[27845]: pgmap v17501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:39 smithi033 ceph-mon[27845]: pgmap v17502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32260.130716 secs 2024-04-26T20:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32257.195004 secs 2024-04-26T20:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:39 smithi096 ceph-mon[32753]: pgmap v17501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:39 smithi096 ceph-mon[32753]: pgmap v17502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32260.130716 secs 2024-04-26T20:48:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:41.390+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:41.390+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:41.390+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:48:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:41 smithi194 ceph-mon[33017]: pgmap v17503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:41 smithi033 ceph-mon[27845]: pgmap v17503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:41 smithi096 ceph-mon[32753]: pgmap v17503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:43 smithi194 ceph-mon[33017]: pgmap v17504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:43 smithi033 ceph-mon[27845]: pgmap v17504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:43 smithi096 ceph-mon[32753]: pgmap v17504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32262.195086 secs 2024-04-26T20:48:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32265.130839 secs 2024-04-26T20:48:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32262.195086 secs 2024-04-26T20:48:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32265.130839 secs 2024-04-26T20:48:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32262.195086 secs 2024-04-26T20:48:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32265.130839 secs 2024-04-26T20:48:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:45 smithi194 ceph-mon[33017]: pgmap v17505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:48:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:45 smithi033 ceph-mon[27845]: pgmap v17505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:48:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:45 smithi096 ceph-mon[32753]: pgmap v17505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:48:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:47 smithi194 ceph-mon[33017]: pgmap v17506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:47 smithi033 ceph-mon[27845]: pgmap v17506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:47 smithi096 ceph-mon[32753]: pgmap v17506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:48:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32267.195170 secs 2024-04-26T20:48:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:49 smithi194 ceph-mon[33017]: pgmap v17507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:48:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32270.130989 secs 2024-04-26T20:48:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32267.195170 secs 2024-04-26T20:48:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:49 smithi033 ceph-mon[27845]: pgmap v17507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:48:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32270.130989 secs 2024-04-26T20:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32267.195170 secs 2024-04-26T20:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:49 smithi096 ceph-mon[32753]: pgmap v17507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32270.130989 secs 2024-04-26T20:48:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:51 smithi194 ceph-mon[33017]: pgmap v17508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:51 smithi033 ceph-mon[27845]: pgmap v17508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:51 smithi096 ceph-mon[32753]: pgmap v17508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:53 smithi194 ceph-mon[33017]: pgmap v17509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:53 smithi033 ceph-mon[27845]: pgmap v17509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:53 smithi096 ceph-mon[32753]: pgmap v17509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32272.195229 secs 2024-04-26T20:48:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32275.131141 secs 2024-04-26T20:48:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32272.195229 secs 2024-04-26T20:48:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32275.131141 secs 2024-04-26T20:48:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32272.195229 secs 2024-04-26T20:48:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32275.131141 secs 2024-04-26T20:48:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:55 smithi194 ceph-mon[33017]: pgmap v17510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:55 smithi033 ceph-mon[27845]: pgmap v17510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:55 smithi096 ceph-mon[32753]: pgmap v17510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:48:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:56.396+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:56.396+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:48:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:48:56.396+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:48:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:57 smithi194 ceph-mon[33017]: pgmap v17511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:57 smithi033 ceph-mon[27845]: pgmap v17511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:48:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:57 smithi096 ceph-mon[32753]: pgmap v17511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32277.195313 secs 2024-04-26T20:49:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:59 smithi194 ceph-mon[33017]: pgmap v17512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32280.131263 secs 2024-04-26T20:49:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32277.195313 secs 2024-04-26T20:49:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:59 smithi033 ceph-mon[27845]: pgmap v17512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32280.131263 secs 2024-04-26T20:49:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32277.195313 secs 2024-04-26T20:49:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:59 smithi096 ceph-mon[32753]: pgmap v17512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32280.131263 secs 2024-04-26T20:49:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:01 smithi194 ceph-mon[33017]: pgmap v17513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:01 smithi033 ceph-mon[27845]: pgmap v17513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:01 smithi096 ceph-mon[32753]: pgmap v17513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:03 smithi194 ceph-mon[33017]: pgmap v17514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:03 smithi033 ceph-mon[27845]: pgmap v17514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:03 smithi096 ceph-mon[32753]: pgmap v17514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:05.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32282.195398 secs 2024-04-26T20:49:05.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32285.131374 secs 2024-04-26T20:49:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32282.195398 secs 2024-04-26T20:49:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32285.131374 secs 2024-04-26T20:49:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32282.195398 secs 2024-04-26T20:49:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32285.131374 secs 2024-04-26T20:49:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:05 smithi194 ceph-mon[33017]: pgmap v17515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:05 smithi033 ceph-mon[27845]: pgmap v17515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:05 smithi096 ceph-mon[32753]: pgmap v17515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:07 smithi194 ceph-mon[33017]: pgmap v17516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:07 smithi033 ceph-mon[27845]: pgmap v17516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:07 smithi096 ceph-mon[32753]: pgmap v17516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32290.131521 secs 2024-04-26T20:49:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32290.131521 secs 2024-04-26T20:49:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32290.131521 secs 2024-04-26T20:49:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32287.195481 secs 2024-04-26T20:49:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:09 smithi194 ceph-mon[33017]: pgmap v17517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32287.195481 secs 2024-04-26T20:49:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:09 smithi033 ceph-mon[27845]: pgmap v17517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32287.195481 secs 2024-04-26T20:49:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:09 smithi096 ceph-mon[32753]: pgmap v17517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:11.403+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:11.403+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:11.403+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:49:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:11 smithi194 ceph-mon[33017]: pgmap v17518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:11 smithi033 ceph-mon[27845]: pgmap v17518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:11 smithi096 ceph-mon[32753]: pgmap v17518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:13 smithi033 ceph-mon[27845]: pgmap v17519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32295.131582 secs 2024-04-26T20:49:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:13 smithi096 ceph-mon[32753]: pgmap v17519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32295.131582 secs 2024-04-26T20:49:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:13 smithi194 ceph-mon[33017]: pgmap v17519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32295.131582 secs 2024-04-26T20:49:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32292.195563 secs 2024-04-26T20:49:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32292.195563 secs 2024-04-26T20:49:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32292.195563 secs 2024-04-26T20:49:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:15 smithi033 ceph-mon[27845]: pgmap v17520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:15 smithi096 ceph-mon[32753]: pgmap v17520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:15 smithi194 ceph-mon[33017]: pgmap v17520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:17 smithi033 ceph-mon[27845]: pgmap v17521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:17 smithi096 ceph-mon[32753]: pgmap v17521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:17 smithi194 ceph-mon[33017]: pgmap v17521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32300.131711 secs 2024-04-26T20:49:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32300.131711 secs 2024-04-26T20:49:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32300.131711 secs 2024-04-26T20:49:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32297.195647 secs 2024-04-26T20:49:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:19 smithi033 ceph-mon[27845]: pgmap v17522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32297.195647 secs 2024-04-26T20:49:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:19 smithi096 ceph-mon[32753]: pgmap v17522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32297.195647 secs 2024-04-26T20:49:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:19 smithi194 ceph-mon[33017]: pgmap v17522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:21 smithi033 ceph-mon[27845]: pgmap v17523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:21 smithi096 ceph-mon[32753]: pgmap v17523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:21 smithi194 ceph-mon[33017]: pgmap v17523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:23 smithi033 ceph-mon[27845]: pgmap v17524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32305.131840 secs 2024-04-26T20:49:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:23 smithi096 ceph-mon[32753]: pgmap v17524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32305.131840 secs 2024-04-26T20:49:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:23 smithi194 ceph-mon[33017]: pgmap v17524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32305.131840 secs 2024-04-26T20:49:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32302.195728 secs 2024-04-26T20:49:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32302.195728 secs 2024-04-26T20:49:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32302.195728 secs 2024-04-26T20:49:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:25 smithi033 ceph-mon[27845]: pgmap v17525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:25 smithi096 ceph-mon[32753]: pgmap v17525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:25 smithi194 ceph-mon[33017]: pgmap v17525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:26.415+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:26.415+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:26.415+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:49:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:27 smithi033 ceph-mon[27845]: pgmap v17526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:27 smithi096 ceph-mon[32753]: pgmap v17526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:27 smithi194 ceph-mon[33017]: pgmap v17526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32310.131965 secs 2024-04-26T20:49:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32310.131965 secs 2024-04-26T20:49:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32310.131965 secs 2024-04-26T20:49:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32307.195811 secs 2024-04-26T20:49:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:29 smithi096 ceph-mon[32753]: pgmap v17527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32307.195811 secs 2024-04-26T20:49:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:29 smithi194 ceph-mon[33017]: pgmap v17527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32307.195811 secs 2024-04-26T20:49:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:29 smithi033 ceph-mon[27845]: pgmap v17527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:30 smithi194 ceph-mon[33017]: pgmap v17528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:30 smithi033 ceph-mon[27845]: pgmap v17528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:30 smithi096 ceph-mon[32753]: pgmap v17528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:33 smithi194 ceph-mon[33017]: pgmap v17529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:33 smithi033 ceph-mon[27845]: pgmap v17529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:33 smithi096 ceph-mon[32753]: pgmap v17529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32312.195893 secs 2024-04-26T20:49:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32315.132044 secs 2024-04-26T20:49:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32312.195893 secs 2024-04-26T20:49:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32315.132044 secs 2024-04-26T20:49:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32312.195893 secs 2024-04-26T20:49:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32315.132044 secs 2024-04-26T20:49:35.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:35 smithi194 ceph-mon[33017]: pgmap v17530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:35.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:35 smithi033 ceph-mon[27845]: pgmap v17530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:35.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:35 smithi096 ceph-mon[32753]: pgmap v17530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:37 smithi033 ceph-mon[27845]: pgmap v17531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:37 smithi194 ceph-mon[33017]: pgmap v17531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:37 smithi096 ceph-mon[32753]: pgmap v17531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32317.195981 secs 2024-04-26T20:49:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:39 smithi194 ceph-mon[33017]: pgmap v17532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:49:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32320.132134 secs 2024-04-26T20:49:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32317.195981 secs 2024-04-26T20:49:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:39 smithi033 ceph-mon[27845]: pgmap v17532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:49:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32320.132134 secs 2024-04-26T20:49:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32317.195981 secs 2024-04-26T20:49:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:39 smithi096 ceph-mon[32753]: pgmap v17532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:49:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32320.132134 secs 2024-04-26T20:49:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:49:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:49:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:40 smithi194 ceph-mon[33017]: pgmap v17533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:49:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:40 smithi194 ceph-mon[33017]: pgmap v17534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:49:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:49:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:49:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:40 smithi033 ceph-mon[27845]: pgmap v17533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:49:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:40 smithi033 ceph-mon[27845]: pgmap v17534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:49:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:49:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:49:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:40 smithi096 ceph-mon[32753]: pgmap v17533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:49:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:40 smithi096 ceph-mon[32753]: pgmap v17534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:49:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:41.430+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:41.431+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:41.431+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:49:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:42 smithi096 ceph-mon[32753]: pgmap v17535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:42 smithi194 ceph-mon[33017]: pgmap v17535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:42 smithi033 ceph-mon[27845]: pgmap v17535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32325.132269 secs 2024-04-26T20:49:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32325.132269 secs 2024-04-26T20:49:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32325.132269 secs 2024-04-26T20:49:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32322.196084 secs 2024-04-26T20:49:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:44 smithi194 ceph-mon[33017]: pgmap v17536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32322.196084 secs 2024-04-26T20:49:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:44 smithi033 ceph-mon[27845]: pgmap v17536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32322.196084 secs 2024-04-26T20:49:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:44 smithi096 ceph-mon[32753]: pgmap v17536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:45 smithi194 ceph-mon[33017]: pgmap v17537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:45 smithi033 ceph-mon[27845]: pgmap v17537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:45 smithi096 ceph-mon[32753]: pgmap v17537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:48 smithi194 ceph-mon[33017]: pgmap v17538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32330.132346 secs 2024-04-26T20:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:48 smithi033 ceph-mon[27845]: pgmap v17538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32330.132346 secs 2024-04-26T20:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:48 smithi096 ceph-mon[32753]: pgmap v17538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:49:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32330.132346 secs 2024-04-26T20:49:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32327.196170 secs 2024-04-26T20:49:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32327.196170 secs 2024-04-26T20:49:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32327.196170 secs 2024-04-26T20:49:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:50 smithi194 ceph-mon[33017]: pgmap v17539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:49:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:50 smithi033 ceph-mon[27845]: pgmap v17539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:49:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:50 smithi096 ceph-mon[32753]: pgmap v17539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:49:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:51 smithi194 ceph-mon[33017]: pgmap v17540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:51 smithi033 ceph-mon[27845]: pgmap v17540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:51 smithi096 ceph-mon[32753]: pgmap v17540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32335.132490 secs 2024-04-26T20:49:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32335.132490 secs 2024-04-26T20:49:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32335.132490 secs 2024-04-26T20:49:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32332.196252 secs 2024-04-26T20:49:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:54 smithi194 ceph-mon[33017]: pgmap v17541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32332.196252 secs 2024-04-26T20:49:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:54 smithi033 ceph-mon[27845]: pgmap v17541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32332.196252 secs 2024-04-26T20:49:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:54 smithi096 ceph-mon[32753]: pgmap v17541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:55 smithi194 ceph-mon[33017]: pgmap v17542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:55 smithi033 ceph-mon[27845]: pgmap v17542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:55 smithi096 ceph-mon[32753]: pgmap v17542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:49:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:56.445+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:56.445+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:49:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:49:56.445+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:49:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:58 smithi194 ceph-mon[33017]: pgmap v17543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32340.132609 secs 2024-04-26T20:49:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:58 smithi033 ceph-mon[27845]: pgmap v17543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32340.132609 secs 2024-04-26T20:49:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:58 smithi096 ceph-mon[32753]: pgmap v17543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:49:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32340.132609 secs 2024-04-26T20:50:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32337.196315 secs 2024-04-26T20:50:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:49:59 smithi194 ceph-mon[33017]: pgmap v17544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32337.196315 secs 2024-04-26T20:50:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:49:59 smithi033 ceph-mon[27845]: pgmap v17544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32337.196315 secs 2024-04-26T20:50:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:49:59 smithi096 ceph-mon[32753]: pgmap v17544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:50:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:50:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T20:50:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:01 smithi194 ceph-mon[33017]: pgmap v17545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:01 smithi033 ceph-mon[27845]: pgmap v17545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:01 smithi096 ceph-mon[32753]: pgmap v17545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32345.132737 secs 2024-04-26T20:50:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32345.132737 secs 2024-04-26T20:50:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32345.132737 secs 2024-04-26T20:50:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32342.196402 secs 2024-04-26T20:50:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:04 smithi033 ceph-mon[27845]: pgmap v17546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32342.196402 secs 2024-04-26T20:50:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:04 smithi096 ceph-mon[32753]: pgmap v17546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:05.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32342.196402 secs 2024-04-26T20:50:05.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:04 smithi194 ceph-mon[33017]: pgmap v17546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:06 smithi033 ceph-mon[27845]: pgmap v17547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:06 smithi096 ceph-mon[32753]: pgmap v17547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:06 smithi194 ceph-mon[33017]: pgmap v17547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:08 smithi033 ceph-mon[27845]: pgmap v17548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32350.132833 secs 2024-04-26T20:50:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:08 smithi096 ceph-mon[32753]: pgmap v17548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32350.132833 secs 2024-04-26T20:50:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:08 smithi194 ceph-mon[33017]: pgmap v17548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32350.132833 secs 2024-04-26T20:50:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32347.196487 secs 2024-04-26T20:50:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32347.196487 secs 2024-04-26T20:50:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32347.196487 secs 2024-04-26T20:50:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:10 smithi033 ceph-mon[27845]: pgmap v17549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:10 smithi096 ceph-mon[32753]: pgmap v17549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:10 smithi194 ceph-mon[33017]: pgmap v17549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:11.452+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:11.452+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:11.452+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:50:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:12 smithi033 ceph-mon[27845]: pgmap v17550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:12 smithi096 ceph-mon[32753]: pgmap v17550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:12 smithi194 ceph-mon[33017]: pgmap v17550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32355.132960 secs 2024-04-26T20:50:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32355.132960 secs 2024-04-26T20:50:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32355.132960 secs 2024-04-26T20:50:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32352.196575 secs 2024-04-26T20:50:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:14 smithi096 ceph-mon[32753]: pgmap v17551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32352.196575 secs 2024-04-26T20:50:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:14 smithi194 ceph-mon[33017]: pgmap v17551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32352.196575 secs 2024-04-26T20:50:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:14 smithi033 ceph-mon[27845]: pgmap v17551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:16 smithi194 ceph-mon[33017]: pgmap v17552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:16 smithi033 ceph-mon[27845]: pgmap v17552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:16 smithi096 ceph-mon[32753]: pgmap v17552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:18 smithi194 ceph-mon[33017]: pgmap v17553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:19.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32360.133075 secs 2024-04-26T20:50:19.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:18 smithi096 ceph-mon[32753]: pgmap v17553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:19.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32360.133075 secs 2024-04-26T20:50:19.801 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:18 smithi033 ceph-mon[27845]: pgmap v17553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:19.801 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32360.133075 secs 2024-04-26T20:50:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32357.196657 secs 2024-04-26T20:50:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:19 smithi194 ceph-mon[33017]: pgmap v17554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32357.196657 secs 2024-04-26T20:50:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:19 smithi033 ceph-mon[27845]: pgmap v17554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32357.196657 secs 2024-04-26T20:50:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:19 smithi096 ceph-mon[32753]: pgmap v17554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:21 smithi194 ceph-mon[33017]: pgmap v17555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:21 smithi033 ceph-mon[27845]: pgmap v17555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:21 smithi096 ceph-mon[32753]: pgmap v17555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32365.133203 secs 2024-04-26T20:50:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32365.133203 secs 2024-04-26T20:50:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32365.133203 secs 2024-04-26T20:50:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32362.196712 secs 2024-04-26T20:50:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:24 smithi033 ceph-mon[27845]: pgmap v17556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32362.196712 secs 2024-04-26T20:50:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:24 smithi096 ceph-mon[32753]: pgmap v17556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32362.196712 secs 2024-04-26T20:50:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:24 smithi194 ceph-mon[33017]: pgmap v17556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:26.464+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:26.464+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:26.464+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:50:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:26 smithi033 ceph-mon[27845]: pgmap v17557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:26 smithi096 ceph-mon[32753]: pgmap v17557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:27.302+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:50:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:27.302+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:50:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:27.303+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 254 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:50:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:27.303+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 254 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:50:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:26 smithi194 ceph-mon[33017]: pgmap v17557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:28 smithi033 ceph-mon[27845]: pgmap v17558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32370.133264 secs 2024-04-26T20:50:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:28 smithi096 ceph-mon[32753]: pgmap v17558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32370.133264 secs 2024-04-26T20:50:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:28 smithi194 ceph-mon[33017]: pgmap v17558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32370.133264 secs 2024-04-26T20:50:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32367.196768 secs 2024-04-26T20:50:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32367.196768 secs 2024-04-26T20:50:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32367.196768 secs 2024-04-26T20:50:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:30 smithi096 ceph-mon[32753]: pgmap v17559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:30 smithi194 ceph-mon[33017]: pgmap v17559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:30 smithi033 ceph-mon[27845]: pgmap v17559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:32 smithi096 ceph-mon[32753]: pgmap v17560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:32 smithi194 ceph-mon[33017]: pgmap v17560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:32 smithi033 ceph-mon[27845]: pgmap v17560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32372.196869 secs 2024-04-26T20:50:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32375.133393 secs 2024-04-26T20:50:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32372.196869 secs 2024-04-26T20:50:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32375.133393 secs 2024-04-26T20:50:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32372.196869 secs 2024-04-26T20:50:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32375.133393 secs 2024-04-26T20:50:35.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:34 smithi194 ceph-mon[33017]: pgmap v17561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:35.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:34 smithi033 ceph-mon[27845]: pgmap v17561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:35.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:34 smithi096 ceph-mon[32753]: pgmap v17561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:35 smithi194 ceph-mon[33017]: pgmap v17562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:35 smithi033 ceph-mon[27845]: pgmap v17562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:35 smithi096 ceph-mon[32753]: pgmap v17562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:37 smithi194 ceph-mon[33017]: pgmap v17563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32377.196975 secs 2024-04-26T20:50:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:37 smithi033 ceph-mon[27845]: pgmap v17563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32377.196975 secs 2024-04-26T20:50:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:37 smithi096 ceph-mon[32753]: pgmap v17563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32377.196975 secs 2024-04-26T20:50:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32380.133507 secs 2024-04-26T20:50:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32380.133507 secs 2024-04-26T20:50:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32380.133507 secs 2024-04-26T20:50:40.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:40 smithi033 ceph-mon[27845]: pgmap v17564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:40 smithi194 ceph-mon[33017]: pgmap v17564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:40 smithi096 ceph-mon[32753]: pgmap v17564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:50:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:50:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:50:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:41.479+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:41.479+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:41.479+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:50:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:42 smithi194 ceph-mon[33017]: pgmap v17565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:42 smithi033 ceph-mon[27845]: pgmap v17565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:42 smithi096 ceph-mon[32753]: pgmap v17565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:50:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:50:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:43 smithi033 ceph-mon[27845]: pgmap v17566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:50:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:43 smithi033 ceph-mon[27845]: pgmap v17567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32382.197050 secs 2024-04-26T20:50:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:50:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:50:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:43 smithi096 ceph-mon[32753]: pgmap v17566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:50:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:43 smithi096 ceph-mon[32753]: pgmap v17567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32382.197050 secs 2024-04-26T20:50:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:50:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:50:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:43 smithi194 ceph-mon[33017]: pgmap v17566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:50:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:43 smithi194 ceph-mon[33017]: pgmap v17567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:50:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32382.197050 secs 2024-04-26T20:50:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32385.133629 secs 2024-04-26T20:50:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32385.133629 secs 2024-04-26T20:50:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32385.133629 secs 2024-04-26T20:50:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:45 smithi033 ceph-mon[27845]: pgmap v17568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:45 smithi096 ceph-mon[32753]: pgmap v17568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:45 smithi194 ceph-mon[33017]: pgmap v17568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:47 smithi194 ceph-mon[33017]: pgmap v17569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:47 smithi033 ceph-mon[27845]: pgmap v17569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:47 smithi096 ceph-mon[32753]: pgmap v17569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32387.197138 secs 2024-04-26T20:50:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32387.197138 secs 2024-04-26T20:50:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32387.197138 secs 2024-04-26T20:50:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:49 smithi194 ceph-mon[33017]: pgmap v17570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:50:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32390.133751 secs 2024-04-26T20:50:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:49 smithi033 ceph-mon[27845]: pgmap v17570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:50:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32390.133751 secs 2024-04-26T20:50:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:49 smithi096 ceph-mon[32753]: pgmap v17570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T20:50:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32390.133751 secs 2024-04-26T20:50:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:51 smithi194 ceph-mon[33017]: pgmap v17571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:51 smithi033 ceph-mon[27845]: pgmap v17571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:51 smithi096 ceph-mon[32753]: pgmap v17571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:50:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:53 smithi194 ceph-mon[33017]: pgmap v17572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:50:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32392.197222 secs 2024-04-26T20:50:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:53 smithi033 ceph-mon[27845]: pgmap v17572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:50:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32392.197222 secs 2024-04-26T20:50:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:53 smithi096 ceph-mon[32753]: pgmap v17572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:50:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32392.197222 secs 2024-04-26T20:50:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32395.133871 secs 2024-04-26T20:50:54.896 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32395.133871 secs 2024-04-26T20:50:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32395.133871 secs 2024-04-26T20:50:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:55 smithi194 ceph-mon[33017]: pgmap v17573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:55 smithi033 ceph-mon[27845]: pgmap v17573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:55 smithi096 ceph-mon[32753]: pgmap v17573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:50:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:56.493+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:56.493+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:50:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:50:56.493+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:50:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:57 smithi194 ceph-mon[33017]: pgmap v17574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:57 smithi033 ceph-mon[27845]: pgmap v17574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:57 smithi096 ceph-mon[32753]: pgmap v17574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32397.197307 secs 2024-04-26T20:50:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32397.197307 secs 2024-04-26T20:50:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32397.197307 secs 2024-04-26T20:50:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:59 smithi033 ceph-mon[27845]: pgmap v17575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32400.133935 secs 2024-04-26T20:50:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:59 smithi096 ceph-mon[32753]: pgmap v17575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:50:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32400.133935 secs 2024-04-26T20:51:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:59 smithi194 ceph-mon[33017]: pgmap v17575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32400.133935 secs 2024-04-26T20:51:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:01 smithi033 ceph-mon[27845]: pgmap v17576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:01 smithi096 ceph-mon[32753]: pgmap v17576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:01 smithi194 ceph-mon[33017]: pgmap v17576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:03 smithi033 ceph-mon[27845]: pgmap v17577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32402.197387 secs 2024-04-26T20:51:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:03 smithi096 ceph-mon[32753]: pgmap v17577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32402.197387 secs 2024-04-26T20:51:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:03 smithi194 ceph-mon[33017]: pgmap v17577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32402.197387 secs 2024-04-26T20:51:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32405.134077 secs 2024-04-26T20:51:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32405.134077 secs 2024-04-26T20:51:05.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32405.134077 secs 2024-04-26T20:51:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:05 smithi033 ceph-mon[27845]: pgmap v17578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:05 smithi096 ceph-mon[32753]: pgmap v17578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:06.021 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:05 smithi194 ceph-mon[33017]: pgmap v17578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:07 smithi033 ceph-mon[27845]: pgmap v17579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:07 smithi096 ceph-mon[32753]: pgmap v17579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:07 smithi194 ceph-mon[33017]: pgmap v17579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32407.197492 secs 2024-04-26T20:51:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32407.197492 secs 2024-04-26T20:51:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32407.197492 secs 2024-04-26T20:51:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:09 smithi033 ceph-mon[27845]: pgmap v17580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32410.134191 secs 2024-04-26T20:51:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:09 smithi096 ceph-mon[32753]: pgmap v17580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32410.134191 secs 2024-04-26T20:51:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:09 smithi194 ceph-mon[33017]: pgmap v17580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32410.134191 secs 2024-04-26T20:51:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:11.509+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:11.509+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:51:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:11.509+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:11 smithi033 ceph-mon[27845]: pgmap v17581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:11 smithi096 ceph-mon[32753]: pgmap v17581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:11 smithi194 ceph-mon[33017]: pgmap v17581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:13 smithi033 ceph-mon[27845]: pgmap v17582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32412.197574 secs 2024-04-26T20:51:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:13 smithi096 ceph-mon[32753]: pgmap v17582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32412.197574 secs 2024-04-26T20:51:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:13 smithi194 ceph-mon[33017]: pgmap v17582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32412.197574 secs 2024-04-26T20:51:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32415.134323 secs 2024-04-26T20:51:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32415.134323 secs 2024-04-26T20:51:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32415.134323 secs 2024-04-26T20:51:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:15 smithi033 ceph-mon[27845]: pgmap v17583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:15 smithi096 ceph-mon[32753]: pgmap v17583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:15 smithi194 ceph-mon[33017]: pgmap v17583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:17 smithi096 ceph-mon[32753]: pgmap v17584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:17 smithi194 ceph-mon[33017]: pgmap v17584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:17 smithi033 ceph-mon[27845]: pgmap v17584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32417.197659 secs 2024-04-26T20:51:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32417.197659 secs 2024-04-26T20:51:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32417.197659 secs 2024-04-26T20:51:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:19 smithi194 ceph-mon[33017]: pgmap v17585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32420.134434 secs 2024-04-26T20:51:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:19 smithi033 ceph-mon[27845]: pgmap v17585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32420.134434 secs 2024-04-26T20:51:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:19 smithi096 ceph-mon[32753]: pgmap v17585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32420.134434 secs 2024-04-26T20:51:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:21 smithi194 ceph-mon[33017]: pgmap v17586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:21 smithi033 ceph-mon[27845]: pgmap v17586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:21 smithi096 ceph-mon[32753]: pgmap v17586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:23 smithi194 ceph-mon[33017]: pgmap v17587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32422.197742 secs 2024-04-26T20:51:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:23 smithi033 ceph-mon[27845]: pgmap v17587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32422.197742 secs 2024-04-26T20:51:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:23 smithi096 ceph-mon[32753]: pgmap v17587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32422.197742 secs 2024-04-26T20:51:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32425.134535 secs 2024-04-26T20:51:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32425.134535 secs 2024-04-26T20:51:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32425.134535 secs 2024-04-26T20:51:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:25 smithi194 ceph-mon[33017]: pgmap v17588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:26.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:25 smithi033 ceph-mon[27845]: pgmap v17588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:25 smithi096 ceph-mon[32753]: pgmap v17588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:26.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:26.517+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:26.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:26.518+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:26.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:26.518+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:51:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:27 smithi194 ceph-mon[33017]: pgmap v17589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:27 smithi033 ceph-mon[27845]: pgmap v17589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:27 smithi096 ceph-mon[32753]: pgmap v17589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32427.197824 secs 2024-04-26T20:51:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32427.197824 secs 2024-04-26T20:51:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32427.197824 secs 2024-04-26T20:51:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:29 smithi194 ceph-mon[33017]: pgmap v17590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32430.134666 secs 2024-04-26T20:51:30.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:29 smithi033 ceph-mon[27845]: pgmap v17590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:30.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32430.134666 secs 2024-04-26T20:51:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:29 smithi096 ceph-mon[32753]: pgmap v17590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32430.134666 secs 2024-04-26T20:51:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:31 smithi194 ceph-mon[33017]: pgmap v17591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:31 smithi033 ceph-mon[27845]: pgmap v17591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:31 smithi096 ceph-mon[32753]: pgmap v17591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:33 smithi194 ceph-mon[33017]: pgmap v17592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32432.197904 secs 2024-04-26T20:51:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:33 smithi033 ceph-mon[27845]: pgmap v17592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32432.197904 secs 2024-04-26T20:51:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:33 smithi096 ceph-mon[32753]: pgmap v17592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32432.197904 secs 2024-04-26T20:51:35.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32435.134733 secs 2024-04-26T20:51:35.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32435.134733 secs 2024-04-26T20:51:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32435.134733 secs 2024-04-26T20:51:36.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:35 smithi194 ceph-mon[33017]: pgmap v17593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:35 smithi033 ceph-mon[27845]: pgmap v17593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:35 smithi096 ceph-mon[32753]: pgmap v17593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:37 smithi194 ceph-mon[33017]: pgmap v17594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:38.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:37 smithi033 ceph-mon[27845]: pgmap v17594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:37 smithi096 ceph-mon[32753]: pgmap v17594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32437.198010 secs 2024-04-26T20:51:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32437.198010 secs 2024-04-26T20:51:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32437.198010 secs 2024-04-26T20:51:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:39 smithi194 ceph-mon[33017]: pgmap v17595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32440.134875 secs 2024-04-26T20:51:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:39 smithi033 ceph-mon[27845]: pgmap v17595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32440.134875 secs 2024-04-26T20:51:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:39 smithi096 ceph-mon[32753]: pgmap v17595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32440.134875 secs 2024-04-26T20:51:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:41.527+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:41.527+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:41.527+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:51:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:41 smithi194 ceph-mon[33017]: pgmap v17596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:41 smithi033 ceph-mon[27845]: pgmap v17596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:41 smithi096 ceph-mon[32753]: pgmap v17596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:51:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:43 smithi194 ceph-mon[33017]: pgmap v17597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:51:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32442.198078 secs 2024-04-26T20:51:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:43 smithi033 ceph-mon[27845]: pgmap v17597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:51:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32442.198078 secs 2024-04-26T20:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:43 smithi096 ceph-mon[32753]: pgmap v17597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:51:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32442.198078 secs 2024-04-26T20:51:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32445.134985 secs 2024-04-26T20:51:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:51:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:51:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:51:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32445.134985 secs 2024-04-26T20:51:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:51:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:51:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:51:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32445.134985 secs 2024-04-26T20:51:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:51:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:51:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:51:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:45 smithi194 ceph-mon[33017]: pgmap v17598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:45 smithi194 ceph-mon[33017]: pgmap v17599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:51:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:45 smithi033 ceph-mon[27845]: pgmap v17598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:45 smithi033 ceph-mon[27845]: pgmap v17599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:51:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:45 smithi096 ceph-mon[32753]: pgmap v17598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:51:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:45 smithi096 ceph-mon[32753]: pgmap v17599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:51:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:47 smithi194 ceph-mon[33017]: pgmap v17600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:47 smithi033 ceph-mon[27845]: pgmap v17600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:47 smithi096 ceph-mon[32753]: pgmap v17600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32447.198166 secs 2024-04-26T20:51:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32447.198166 secs 2024-04-26T20:51:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32447.198166 secs 2024-04-26T20:51:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:49 smithi194 ceph-mon[33017]: pgmap v17601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32450.135141 secs 2024-04-26T20:51:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:49 smithi033 ceph-mon[27845]: pgmap v17601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32450.135141 secs 2024-04-26T20:51:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:49 smithi096 ceph-mon[32753]: pgmap v17601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32450.135141 secs 2024-04-26T20:51:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:51 smithi194 ceph-mon[33017]: pgmap v17602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:51 smithi033 ceph-mon[27845]: pgmap v17602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:51 smithi096 ceph-mon[32753]: pgmap v17602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:53 smithi194 ceph-mon[33017]: pgmap v17603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32452.198224 secs 2024-04-26T20:51:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:53 smithi033 ceph-mon[27845]: pgmap v17603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32452.198224 secs 2024-04-26T20:51:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:53 smithi096 ceph-mon[32753]: pgmap v17603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:51:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32452.198224 secs 2024-04-26T20:51:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32455.135267 secs 2024-04-26T20:51:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32455.135267 secs 2024-04-26T20:51:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32455.135267 secs 2024-04-26T20:51:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:55 smithi194 ceph-mon[33017]: pgmap v17604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:51:56.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:55 smithi033 ceph-mon[27845]: pgmap v17604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:51:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:55 smithi096 ceph-mon[32753]: pgmap v17604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:51:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:56.529+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:56.529+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:51:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:51:56.529+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:51:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:57 smithi033 ceph-mon[27845]: pgmap v17605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:51:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:57 smithi096 ceph-mon[32753]: pgmap v17605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:51:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:57 smithi194 ceph-mon[33017]: pgmap v17605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:51:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32457.198305 secs 2024-04-26T20:51:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32460.135331 secs 2024-04-26T20:51:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32457.198305 secs 2024-04-26T20:51:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32460.135331 secs 2024-04-26T20:51:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32457.198305 secs 2024-04-26T20:51:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32460.135331 secs 2024-04-26T20:52:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:51:59 smithi033 ceph-mon[27845]: pgmap v17606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:52:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:51:59 smithi096 ceph-mon[32753]: pgmap v17606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:52:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:51:59 smithi194 ceph-mon[33017]: pgmap v17606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:52:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:01 smithi033 ceph-mon[27845]: pgmap v17607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:01 smithi096 ceph-mon[32753]: pgmap v17607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:01 smithi194 ceph-mon[33017]: pgmap v17607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:03 smithi033 ceph-mon[27845]: pgmap v17608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:52:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32462.198387 secs 2024-04-26T20:52:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32465.135418 secs 2024-04-26T20:52:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:03 smithi096 ceph-mon[32753]: pgmap v17608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:52:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32462.198387 secs 2024-04-26T20:52:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32465.135418 secs 2024-04-26T20:52:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:03 smithi194 ceph-mon[33017]: pgmap v17608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:52:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32462.198387 secs 2024-04-26T20:52:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32465.135418 secs 2024-04-26T20:52:06.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:05 smithi194 ceph-mon[33017]: pgmap v17609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:52:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:05 smithi033 ceph-mon[27845]: pgmap v17609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:52:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:05 smithi096 ceph-mon[32753]: pgmap v17609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T20:52:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:07 smithi033 ceph-mon[27845]: pgmap v17610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:07 smithi096 ceph-mon[32753]: pgmap v17610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:07 smithi194 ceph-mon[33017]: pgmap v17610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32467.198472 secs 2024-04-26T20:52:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32470.135489 secs 2024-04-26T20:52:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32467.198472 secs 2024-04-26T20:52:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32470.135489 secs 2024-04-26T20:52:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32467.198472 secs 2024-04-26T20:52:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32470.135489 secs 2024-04-26T20:52:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:09 smithi033 ceph-mon[27845]: pgmap v17611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:52:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:09 smithi096 ceph-mon[32753]: pgmap v17611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:52:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:09 smithi194 ceph-mon[33017]: pgmap v17611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:52:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:11.544+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:11.545+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:11.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:11.545+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:52:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:11 smithi033 ceph-mon[27845]: pgmap v17612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:52:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:11 smithi096 ceph-mon[32753]: pgmap v17612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:52:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:11 smithi194 ceph-mon[33017]: pgmap v17612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:52:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:13 smithi033 ceph-mon[27845]: pgmap v17613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32472.198555 secs 2024-04-26T20:52:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32475.135556 secs 2024-04-26T20:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:13 smithi096 ceph-mon[32753]: pgmap v17613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32472.198555 secs 2024-04-26T20:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32475.135556 secs 2024-04-26T20:52:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:13 smithi194 ceph-mon[33017]: pgmap v17613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32472.198555 secs 2024-04-26T20:52:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32475.135556 secs 2024-04-26T20:52:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:15 smithi033 ceph-mon[27845]: pgmap v17614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:15 smithi096 ceph-mon[32753]: pgmap v17614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:15 smithi194 ceph-mon[33017]: pgmap v17614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:17 smithi096 ceph-mon[32753]: pgmap v17615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:17 smithi194 ceph-mon[33017]: pgmap v17615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:17 smithi033 ceph-mon[27845]: pgmap v17615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32477.198642 secs 2024-04-26T20:52:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32480.135696 secs 2024-04-26T20:52:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32477.198642 secs 2024-04-26T20:52:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32480.135696 secs 2024-04-26T20:52:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32477.198642 secs 2024-04-26T20:52:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32480.135696 secs 2024-04-26T20:52:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:19 smithi194 ceph-mon[33017]: pgmap v17616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:19 smithi033 ceph-mon[27845]: pgmap v17616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:19 smithi096 ceph-mon[32753]: pgmap v17616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:21 smithi194 ceph-mon[33017]: pgmap v17617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:21 smithi033 ceph-mon[27845]: pgmap v17617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:21 smithi096 ceph-mon[32753]: pgmap v17617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:23 smithi194 ceph-mon[33017]: pgmap v17618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32482.198726 secs 2024-04-26T20:52:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32485.135811 secs 2024-04-26T20:52:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:23 smithi033 ceph-mon[27845]: pgmap v17618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32482.198726 secs 2024-04-26T20:52:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32485.135811 secs 2024-04-26T20:52:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:23 smithi096 ceph-mon[32753]: pgmap v17618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32482.198726 secs 2024-04-26T20:52:24.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32485.135811 secs 2024-04-26T20:52:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:25 smithi194 ceph-mon[33017]: pgmap v17619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:25 smithi033 ceph-mon[27845]: pgmap v17619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:25 smithi096 ceph-mon[32753]: pgmap v17619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:26.561+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:26.561+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:26.561+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:52:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:27 smithi194 ceph-mon[33017]: pgmap v17620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:27 smithi033 ceph-mon[27845]: pgmap v17620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:27 smithi096 ceph-mon[32753]: pgmap v17620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32487.198812 secs 2024-04-26T20:52:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32490.135925 secs 2024-04-26T20:52:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32487.198812 secs 2024-04-26T20:52:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32490.135925 secs 2024-04-26T20:52:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32487.198812 secs 2024-04-26T20:52:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32490.135925 secs 2024-04-26T20:52:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:29 smithi194 ceph-mon[33017]: pgmap v17621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:29 smithi033 ceph-mon[27845]: pgmap v17621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:29 smithi096 ceph-mon[32753]: pgmap v17621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:30 smithi194 ceph-mon[33017]: pgmap v17622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:30 smithi033 ceph-mon[27845]: pgmap v17622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:30 smithi096 ceph-mon[32753]: pgmap v17622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:33 smithi194 ceph-mon[33017]: pgmap v17623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32492.198893 secs 2024-04-26T20:52:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:33 smithi033 ceph-mon[27845]: pgmap v17623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32492.198893 secs 2024-04-26T20:52:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:33 smithi096 ceph-mon[32753]: pgmap v17623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32492.198893 secs 2024-04-26T20:52:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32495.136063 secs 2024-04-26T20:52:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32495.136063 secs 2024-04-26T20:52:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32495.136063 secs 2024-04-26T20:52:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:35 smithi033 ceph-mon[27845]: pgmap v17624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:35 smithi096 ceph-mon[32753]: pgmap v17624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:35 smithi194 ceph-mon[33017]: pgmap v17624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:37.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:37 smithi194 ceph-mon[33017]: pgmap v17625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:37 smithi033 ceph-mon[27845]: pgmap v17625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:37 smithi096 ceph-mon[32753]: pgmap v17625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32497.198976 secs 2024-04-26T20:52:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32497.198976 secs 2024-04-26T20:52:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32497.198976 secs 2024-04-26T20:52:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:39 smithi033 ceph-mon[27845]: pgmap v17626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32500.136147 secs 2024-04-26T20:52:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:39 smithi096 ceph-mon[32753]: pgmap v17626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32500.136147 secs 2024-04-26T20:52:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:39 smithi194 ceph-mon[33017]: pgmap v17626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32500.136147 secs 2024-04-26T20:52:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:41.576+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:41.576+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:41.576+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:52:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:41 smithi033 ceph-mon[27845]: pgmap v17627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:41 smithi096 ceph-mon[32753]: pgmap v17627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:41 smithi194 ceph-mon[33017]: pgmap v17627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:43 smithi033 ceph-mon[27845]: pgmap v17628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32502.199035 secs 2024-04-26T20:52:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:43 smithi096 ceph-mon[32753]: pgmap v17628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32502.199035 secs 2024-04-26T20:52:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:43 smithi194 ceph-mon[33017]: pgmap v17628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32502.199035 secs 2024-04-26T20:52:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32505.136295 secs 2024-04-26T20:52:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32505.136295 secs 2024-04-26T20:52:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32505.136295 secs 2024-04-26T20:52:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:45 smithi033 ceph-mon[27845]: pgmap v17629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:52:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:45 smithi096 ceph-mon[32753]: pgmap v17629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:52:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:45 smithi194 ceph-mon[33017]: pgmap v17629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:45.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:52:45.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:45.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:47 smithi033 ceph-mon[27845]: pgmap v17630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:47 smithi096 ceph-mon[32753]: pgmap v17630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:47 smithi194 ceph-mon[33017]: pgmap v17630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:52:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:52:48.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32507.199098 secs 2024-04-26T20:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:52:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32507.199098 secs 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T20:52:48.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T20:52:48.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32507.199098 secs 2024-04-26T20:52:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:49 smithi096 ceph-mon[32753]: pgmap v17631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32510.136448 secs 2024-04-26T20:52:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:49 smithi194 ceph-mon[33017]: pgmap v17631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32510.136448 secs 2024-04-26T20:52:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:49 smithi033 ceph-mon[27845]: pgmap v17631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:52:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32510.136448 secs 2024-04-26T20:52:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: pgmap v17632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:52:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:52:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:52:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:52:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: pgmap v17633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:52:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: pgmap v17634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:51.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: pgmap v17632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: pgmap v17633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:52:51.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: pgmap v17634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:51.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: pgmap v17632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:52:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: pgmap v17633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: pgmap v17634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:51.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:53 smithi096 ceph-mon[32753]: pgmap v17635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32512.199157 secs 2024-04-26T20:52:54.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32515.136526 secs 2024-04-26T20:52:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:53 smithi194 ceph-mon[33017]: pgmap v17635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32512.199157 secs 2024-04-26T20:52:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32515.136526 secs 2024-04-26T20:52:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:53 smithi033 ceph-mon[27845]: pgmap v17635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32512.199157 secs 2024-04-26T20:52:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32515.136526 secs 2024-04-26T20:52:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:55 smithi194 ceph-mon[33017]: pgmap v17636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:55 smithi033 ceph-mon[27845]: pgmap v17636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:55 smithi096 ceph-mon[32753]: pgmap v17636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:56.577+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:56.578+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:52:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:52:56.578+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:52:58.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:57 smithi033 ceph-mon[27845]: pgmap v17637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:57 smithi194 ceph-mon[33017]: pgmap v17637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:57 smithi096 ceph-mon[32753]: pgmap v17637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:52:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32517.199218 secs 2024-04-26T20:52:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32520.136654 secs 2024-04-26T20:52:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32517.199218 secs 2024-04-26T20:52:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32520.136654 secs 2024-04-26T20:52:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32517.199218 secs 2024-04-26T20:52:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32520.136654 secs 2024-04-26T20:53:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:52:59 smithi194 ceph-mon[33017]: pgmap v17638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:53:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:52:59 smithi033 ceph-mon[27845]: pgmap v17638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:53:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:52:59 smithi096 ceph-mon[32753]: pgmap v17638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T20:53:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:01 smithi194 ceph-mon[33017]: pgmap v17639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:53:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:01 smithi033 ceph-mon[27845]: pgmap v17639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:53:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:01 smithi096 ceph-mon[32753]: pgmap v17639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:53:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:03 smithi194 ceph-mon[33017]: pgmap v17640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32522.199277 secs 2024-04-26T20:53:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32525.136797 secs 2024-04-26T20:53:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:03 smithi033 ceph-mon[27845]: pgmap v17640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32522.199277 secs 2024-04-26T20:53:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32525.136797 secs 2024-04-26T20:53:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:03 smithi096 ceph-mon[32753]: pgmap v17640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32522.199277 secs 2024-04-26T20:53:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32525.136797 secs 2024-04-26T20:53:06.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:05 smithi033 ceph-mon[27845]: pgmap v17641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:06.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:05 smithi096 ceph-mon[32753]: pgmap v17641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:05 smithi194 ceph-mon[33017]: pgmap v17641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:07 smithi194 ceph-mon[33017]: pgmap v17642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:07 smithi033 ceph-mon[27845]: pgmap v17642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:07 smithi096 ceph-mon[32753]: pgmap v17642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32527.199338 secs 2024-04-26T20:53:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32530.136921 secs 2024-04-26T20:53:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32527.199338 secs 2024-04-26T20:53:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32530.136921 secs 2024-04-26T20:53:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32527.199338 secs 2024-04-26T20:53:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32530.136921 secs 2024-04-26T20:53:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:09 smithi194 ceph-mon[33017]: pgmap v17643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:09 smithi033 ceph-mon[27845]: pgmap v17643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:09 smithi096 ceph-mon[32753]: pgmap v17643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:10 smithi194 ceph-mon[33017]: pgmap v17644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:10 smithi033 ceph-mon[27845]: pgmap v17644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:10 smithi096 ceph-mon[32753]: pgmap v17644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:11.593+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:11.593+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:11.593+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:13 smithi194 ceph-mon[33017]: pgmap v17645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32532.199398 secs 2024-04-26T20:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32535.137040 secs 2024-04-26T20:53:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:13 smithi033 ceph-mon[27845]: pgmap v17645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32532.199398 secs 2024-04-26T20:53:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32535.137040 secs 2024-04-26T20:53:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:13 smithi096 ceph-mon[32753]: pgmap v17645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32532.199398 secs 2024-04-26T20:53:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32535.137040 secs 2024-04-26T20:53:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:15 smithi194 ceph-mon[33017]: pgmap v17646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:15 smithi033 ceph-mon[27845]: pgmap v17646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:15 smithi096 ceph-mon[32753]: pgmap v17646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:17 smithi194 ceph-mon[33017]: pgmap v17647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:17 smithi033 ceph-mon[27845]: pgmap v17647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:17 smithi096 ceph-mon[32753]: pgmap v17647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32537.199463 secs 2024-04-26T20:53:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32540.137155 secs 2024-04-26T20:53:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32537.199463 secs 2024-04-26T20:53:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32540.137155 secs 2024-04-26T20:53:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32537.199463 secs 2024-04-26T20:53:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32540.137155 secs 2024-04-26T20:53:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:19 smithi194 ceph-mon[33017]: pgmap v17648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:19 smithi033 ceph-mon[27845]: pgmap v17648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:19 smithi096 ceph-mon[32753]: pgmap v17648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:21 smithi194 ceph-mon[33017]: pgmap v17649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:21 smithi033 ceph-mon[27845]: pgmap v17649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:21 smithi096 ceph-mon[32753]: pgmap v17649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:23 smithi194 ceph-mon[33017]: pgmap v17650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32542.199547 secs 2024-04-26T20:53:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32545.137277 secs 2024-04-26T20:53:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:23 smithi033 ceph-mon[27845]: pgmap v17650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32542.199547 secs 2024-04-26T20:53:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32545.137277 secs 2024-04-26T20:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:23 smithi096 ceph-mon[32753]: pgmap v17650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32542.199547 secs 2024-04-26T20:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32545.137277 secs 2024-04-26T20:53:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:25 smithi033 ceph-mon[27845]: pgmap v17651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:25 smithi096 ceph-mon[32753]: pgmap v17651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:25 smithi194 ceph-mon[33017]: pgmap v17651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:26.610+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:26.610+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:26.610+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:27.333+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:27.333+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:53:27.334 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:27.333+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 255 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T20:53:27.334 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:27.334+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 255 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T20:53:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:27 smithi194 ceph-mon[33017]: pgmap v17652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:27 smithi033 ceph-mon[27845]: pgmap v17652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:27 smithi096 ceph-mon[32753]: pgmap v17652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32547.199652 secs 2024-04-26T20:53:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32550.137396 secs 2024-04-26T20:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32547.199652 secs 2024-04-26T20:53:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32550.137396 secs 2024-04-26T20:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32547.199652 secs 2024-04-26T20:53:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32550.137396 secs 2024-04-26T20:53:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:29 smithi194 ceph-mon[33017]: pgmap v17653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:29 smithi033 ceph-mon[27845]: pgmap v17653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:29 smithi096 ceph-mon[32753]: pgmap v17653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:31 smithi194 ceph-mon[33017]: pgmap v17654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:31 smithi033 ceph-mon[27845]: pgmap v17654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:31 smithi096 ceph-mon[32753]: pgmap v17654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:33 smithi194 ceph-mon[33017]: pgmap v17655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32552.199709 secs 2024-04-26T20:53:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32555.137521 secs 2024-04-26T20:53:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:33 smithi033 ceph-mon[27845]: pgmap v17655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32552.199709 secs 2024-04-26T20:53:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32555.137521 secs 2024-04-26T20:53:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:33 smithi096 ceph-mon[32753]: pgmap v17655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32552.199709 secs 2024-04-26T20:53:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32555.137521 secs 2024-04-26T20:53:36.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:35 smithi033 ceph-mon[27845]: pgmap v17656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:36.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:35 smithi096 ceph-mon[32753]: pgmap v17656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:35 smithi194 ceph-mon[33017]: pgmap v17656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:37 smithi194 ceph-mon[33017]: pgmap v17657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:37 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32557.199768 secs 2024-04-26T20:53:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:37 smithi033 ceph-mon[27845]: pgmap v17657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:37 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32557.199768 secs 2024-04-26T20:53:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:37 smithi096 ceph-mon[32753]: pgmap v17657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:37 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32557.199768 secs 2024-04-26T20:53:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32560.137632 secs 2024-04-26T20:53:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32560.137632 secs 2024-04-26T20:53:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32560.137632 secs 2024-04-26T20:53:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:39 smithi194 ceph-mon[33017]: pgmap v17658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:39 smithi033 ceph-mon[27845]: pgmap v17658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:39 smithi096 ceph-mon[32753]: pgmap v17658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:41 smithi194 ceph-mon[33017]: pgmap v17659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:41 smithi033 ceph-mon[27845]: pgmap v17659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:41 smithi096 ceph-mon[32753]: pgmap v17659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:41.621+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:41.621+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:41.621+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:53:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:43 smithi194 ceph-mon[33017]: pgmap v17660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32562.199851 secs 2024-04-26T20:53:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32565.137757 secs 2024-04-26T20:53:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:43 smithi033 ceph-mon[27845]: pgmap v17660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32562.199851 secs 2024-04-26T20:53:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32565.137757 secs 2024-04-26T20:53:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:43 smithi096 ceph-mon[32753]: pgmap v17660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32562.199851 secs 2024-04-26T20:53:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32565.137757 secs 2024-04-26T20:53:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:45 smithi194 ceph-mon[33017]: pgmap v17661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:45 smithi033 ceph-mon[27845]: pgmap v17661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:45 smithi096 ceph-mon[32753]: pgmap v17661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:47 smithi194 ceph-mon[33017]: pgmap v17662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:47 smithi033 ceph-mon[27845]: pgmap v17662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:47 smithi096 ceph-mon[32753]: pgmap v17662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32567.199933 secs 2024-04-26T20:53:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32570.137883 secs 2024-04-26T20:53:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32567.199933 secs 2024-04-26T20:53:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32570.137883 secs 2024-04-26T20:53:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32567.199933 secs 2024-04-26T20:53:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32570.137883 secs 2024-04-26T20:53:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:49 smithi194 ceph-mon[33017]: pgmap v17663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:49 smithi033 ceph-mon[27845]: pgmap v17663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:49 smithi096 ceph-mon[32753]: pgmap v17663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:51 smithi194 ceph-mon[33017]: pgmap v17664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:53:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:51 smithi033 ceph-mon[27845]: pgmap v17664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:53:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:51 smithi096 ceph-mon[32753]: pgmap v17664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:53:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:53:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:53:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:53:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:53:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:52 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32572.200011 secs 2024-04-26T20:53:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:53:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:53:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:53:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32572.200011 secs 2024-04-26T20:53:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:53:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:53:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:53:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:52 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32572.200011 secs 2024-04-26T20:53:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:54 smithi194 ceph-mon[33017]: pgmap v17665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:54 smithi194 ceph-mon[33017]: pgmap v17666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32575.138005 secs 2024-04-26T20:53:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:54 smithi033 ceph-mon[27845]: pgmap v17665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:54 smithi033 ceph-mon[27845]: pgmap v17666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32575.138005 secs 2024-04-26T20:53:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:54 smithi096 ceph-mon[32753]: pgmap v17665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:53:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:54 smithi096 ceph-mon[32753]: pgmap v17666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32575.138005 secs 2024-04-26T20:53:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:55 smithi194 ceph-mon[33017]: pgmap v17667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:53:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:55 smithi033 ceph-mon[27845]: pgmap v17667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:53:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:55 smithi096 ceph-mon[32753]: pgmap v17667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T20:53:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:56.623+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:56.623+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:53:56.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:53:56.623+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:53:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:57 smithi194 ceph-mon[33017]: pgmap v17668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:57 smithi033 ceph-mon[27845]: pgmap v17668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:57 smithi096 ceph-mon[32753]: pgmap v17668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32577.200096 secs 2024-04-26T20:53:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32577.200096 secs 2024-04-26T20:53:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32577.200096 secs 2024-04-26T20:53:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:59 smithi194 ceph-mon[33017]: pgmap v17669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32580.138155 secs 2024-04-26T20:53:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:59 smithi033 ceph-mon[27845]: pgmap v17669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32580.138155 secs 2024-04-26T20:53:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:59 smithi096 ceph-mon[32753]: pgmap v17669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:53:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32580.138155 secs 2024-04-26T20:54:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:01 smithi194 ceph-mon[33017]: pgmap v17670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:01 smithi033 ceph-mon[27845]: pgmap v17670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:01 smithi096 ceph-mon[32753]: pgmap v17670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:03 smithi194 ceph-mon[33017]: pgmap v17671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:54:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32582.200178 secs 2024-04-26T20:54:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:03 smithi033 ceph-mon[27845]: pgmap v17671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:54:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32582.200178 secs 2024-04-26T20:54:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:03 smithi096 ceph-mon[32753]: pgmap v17671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:54:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32582.200178 secs 2024-04-26T20:54:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32585.138260 secs 2024-04-26T20:54:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32585.138260 secs 2024-04-26T20:54:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32585.138260 secs 2024-04-26T20:54:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:05 smithi194 ceph-mon[33017]: pgmap v17672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:05 smithi033 ceph-mon[27845]: pgmap v17672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:05 smithi096 ceph-mon[32753]: pgmap v17672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:07 smithi194 ceph-mon[33017]: pgmap v17673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:07 smithi033 ceph-mon[27845]: pgmap v17673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:07 smithi096 ceph-mon[32753]: pgmap v17673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32587.200240 secs 2024-04-26T20:54:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32587.200240 secs 2024-04-26T20:54:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32587.200240 secs 2024-04-26T20:54:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:09 smithi033 ceph-mon[27845]: pgmap v17674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32590.138370 secs 2024-04-26T20:54:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:09 smithi096 ceph-mon[32753]: pgmap v17674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32590.138370 secs 2024-04-26T20:54:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:09 smithi194 ceph-mon[33017]: pgmap v17674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32590.138370 secs 2024-04-26T20:54:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:11.638+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:11.638+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:54:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:11.638+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:11 smithi033 ceph-mon[27845]: pgmap v17675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:11 smithi096 ceph-mon[32753]: pgmap v17675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:11 smithi194 ceph-mon[33017]: pgmap v17675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:12.339+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:54:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:12.339+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:54:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:12.340+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 256 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:54:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:12.340+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 256 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:54:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:13 smithi033 ceph-mon[27845]: pgmap v17676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32592.200321 secs 2024-04-26T20:54:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:13 smithi096 ceph-mon[32753]: pgmap v17676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32592.200321 secs 2024-04-26T20:54:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:13 smithi194 ceph-mon[33017]: pgmap v17676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32592.200321 secs 2024-04-26T20:54:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32595.138507 secs 2024-04-26T20:54:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32595.138507 secs 2024-04-26T20:54:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32595.138507 secs 2024-04-26T20:54:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:15 smithi033 ceph-mon[27845]: pgmap v17677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:15 smithi096 ceph-mon[32753]: pgmap v17677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:16.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:15 smithi194 ceph-mon[33017]: pgmap v17677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:17 smithi033 ceph-mon[27845]: pgmap v17678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:17 smithi096 ceph-mon[32753]: pgmap v17678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:18.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:17 smithi194 ceph-mon[33017]: pgmap v17678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32597.200404 secs 2024-04-26T20:54:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32597.200404 secs 2024-04-26T20:54:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32597.200404 secs 2024-04-26T20:54:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:19 smithi033 ceph-mon[27845]: pgmap v17679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32600.138656 secs 2024-04-26T20:54:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:19 smithi096 ceph-mon[32753]: pgmap v17679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32600.138656 secs 2024-04-26T20:54:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:19 smithi194 ceph-mon[33017]: pgmap v17679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32600.138656 secs 2024-04-26T20:54:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:21 smithi033 ceph-mon[27845]: pgmap v17680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:21 smithi096 ceph-mon[32753]: pgmap v17680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:21 smithi194 ceph-mon[33017]: pgmap v17680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:23 smithi033 ceph-mon[27845]: pgmap v17681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32602.200484 secs 2024-04-26T20:54:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:23 smithi096 ceph-mon[32753]: pgmap v17681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32602.200484 secs 2024-04-26T20:54:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:23 smithi194 ceph-mon[33017]: pgmap v17681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32602.200484 secs 2024-04-26T20:54:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32605.138796 secs 2024-04-26T20:54:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32605.138796 secs 2024-04-26T20:54:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32605.138796 secs 2024-04-26T20:54:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:25 smithi096 ceph-mon[32753]: pgmap v17682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:25 smithi194 ceph-mon[33017]: pgmap v17682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:25 smithi033 ceph-mon[27845]: pgmap v17682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:26.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:26.654+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:26.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:26.654+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:54:26.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:26.654+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:27 smithi194 ceph-mon[33017]: pgmap v17683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:27 smithi033 ceph-mon[27845]: pgmap v17683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:27 smithi096 ceph-mon[32753]: pgmap v17683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T20:54:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32607.200569 secs 2024-04-26T20:54:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32607.200569 secs 2024-04-26T20:54:29.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32607.200569 secs 2024-04-26T20:54:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:29 smithi194 ceph-mon[33017]: pgmap v17684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32610.138933 secs 2024-04-26T20:54:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:29 smithi033 ceph-mon[27845]: pgmap v17684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32610.138933 secs 2024-04-26T20:54:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:29 smithi096 ceph-mon[32753]: pgmap v17684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32610.138933 secs 2024-04-26T20:54:31.984 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:31 smithi033 ceph-mon[27845]: pgmap v17685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:31 smithi194 ceph-mon[33017]: pgmap v17685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:31 smithi096 ceph-mon[32753]: pgmap v17685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:33 smithi194 ceph-mon[33017]: pgmap v17686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32612.200646 secs 2024-04-26T20:54:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:33 smithi033 ceph-mon[27845]: pgmap v17686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32612.200646 secs 2024-04-26T20:54:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:33 smithi096 ceph-mon[32753]: pgmap v17686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32612.200646 secs 2024-04-26T20:54:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32615.139049 secs 2024-04-26T20:54:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32615.139049 secs 2024-04-26T20:54:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32615.139049 secs 2024-04-26T20:54:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:35 smithi194 ceph-mon[33017]: pgmap v17687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:36.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:35 smithi033 ceph-mon[27845]: pgmap v17687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:35 smithi096 ceph-mon[32753]: pgmap v17687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:37 smithi033 ceph-mon[27845]: pgmap v17688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:37.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:37 smithi096 ceph-mon[32753]: pgmap v17688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:38.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:37 smithi194 ceph-mon[33017]: pgmap v17688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:39.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32617.200727 secs 2024-04-26T20:54:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32617.200727 secs 2024-04-26T20:54:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32617.200727 secs 2024-04-26T20:54:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:39 smithi194 ceph-mon[33017]: pgmap v17689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32620.139163 secs 2024-04-26T20:54:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:39 smithi033 ceph-mon[27845]: pgmap v17689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32620.139163 secs 2024-04-26T20:54:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:39 smithi096 ceph-mon[32753]: pgmap v17689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32620.139163 secs 2024-04-26T20:54:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:41.666+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:41.666+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:41.666+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:54:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:41 smithi194 ceph-mon[33017]: pgmap v17690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:41 smithi033 ceph-mon[27845]: pgmap v17690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:41 smithi096 ceph-mon[32753]: pgmap v17690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:43 smithi194 ceph-mon[33017]: pgmap v17691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32622.200809 secs 2024-04-26T20:54:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:43 smithi033 ceph-mon[27845]: pgmap v17691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32622.200809 secs 2024-04-26T20:54:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:43 smithi096 ceph-mon[32753]: pgmap v17691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32622.200809 secs 2024-04-26T20:54:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32625.139281 secs 2024-04-26T20:54:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32625.139281 secs 2024-04-26T20:54:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32625.139281 secs 2024-04-26T20:54:46.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:45 smithi194 ceph-mon[33017]: pgmap v17692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:45 smithi033 ceph-mon[27845]: pgmap v17692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:45 smithi096 ceph-mon[32753]: pgmap v17692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:47 smithi194 ceph-mon[33017]: pgmap v17693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:47 smithi033 ceph-mon[27845]: pgmap v17693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:47 smithi096 ceph-mon[32753]: pgmap v17693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32627.200894 secs 2024-04-26T20:54:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32627.200894 secs 2024-04-26T20:54:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32627.200894 secs 2024-04-26T20:54:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:49 smithi194 ceph-mon[33017]: pgmap v17694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32630.139419 secs 2024-04-26T20:54:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:49 smithi033 ceph-mon[27845]: pgmap v17694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32630.139419 secs 2024-04-26T20:54:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:49 smithi096 ceph-mon[32753]: pgmap v17694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32630.139419 secs 2024-04-26T20:54:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:51 smithi194 ceph-mon[33017]: pgmap v17695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:51 smithi033 ceph-mon[27845]: pgmap v17695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:51 smithi096 ceph-mon[32753]: pgmap v17695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:54:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:53 smithi194 ceph-mon[33017]: pgmap v17696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:54:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32632.200989 secs 2024-04-26T20:54:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:53 smithi033 ceph-mon[27845]: pgmap v17696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:54:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32632.200989 secs 2024-04-26T20:54:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:53 smithi096 ceph-mon[32753]: pgmap v17696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:54:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32632.200989 secs 2024-04-26T20:54:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32635.139536 secs 2024-04-26T20:54:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:54:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:54:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:54:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32635.139536 secs 2024-04-26T20:54:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:54:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:54:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:54:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32635.139536 secs 2024-04-26T20:54:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:54:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:54:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:54:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:55 smithi194 ceph-mon[33017]: pgmap v17697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:55 smithi194 ceph-mon[33017]: pgmap v17698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:54:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:55 smithi033 ceph-mon[27845]: pgmap v17697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:55 smithi033 ceph-mon[27845]: pgmap v17698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:54:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:55 smithi096 ceph-mon[32753]: pgmap v17697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:54:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:55 smithi096 ceph-mon[32753]: pgmap v17698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:54:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:56.679+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:56.679+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:54:56.680 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:54:56.679+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:54:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:57 smithi194 ceph-mon[33017]: pgmap v17699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:57 smithi033 ceph-mon[27845]: pgmap v17699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:57 smithi096 ceph-mon[32753]: pgmap v17699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:54:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32637.201076 secs 2024-04-26T20:54:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32637.201076 secs 2024-04-26T20:54:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32637.201076 secs 2024-04-26T20:55:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:59 smithi194 ceph-mon[33017]: pgmap v17700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32640.139670 secs 2024-04-26T20:55:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:59 smithi033 ceph-mon[27845]: pgmap v17700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32640.139670 secs 2024-04-26T20:55:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:59 smithi096 ceph-mon[32753]: pgmap v17700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32640.139670 secs 2024-04-26T20:55:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:01 smithi194 ceph-mon[33017]: pgmap v17701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:01 smithi033 ceph-mon[27845]: pgmap v17701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:01 smithi096 ceph-mon[32753]: pgmap v17701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:03 smithi194 ceph-mon[33017]: pgmap v17702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32642.201137 secs 2024-04-26T20:55:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:03 smithi033 ceph-mon[27845]: pgmap v17702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32642.201137 secs 2024-04-26T20:55:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:03 smithi096 ceph-mon[32753]: pgmap v17702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32642.201137 secs 2024-04-26T20:55:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32645.139807 secs 2024-04-26T20:55:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32645.139807 secs 2024-04-26T20:55:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32645.139807 secs 2024-04-26T20:55:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:05 smithi033 ceph-mon[27845]: pgmap v17703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:55:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:05 smithi096 ceph-mon[32753]: pgmap v17703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:55:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:05 smithi194 ceph-mon[33017]: pgmap v17703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:55:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:07 smithi033 ceph-mon[27845]: pgmap v17704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:07 smithi096 ceph-mon[32753]: pgmap v17704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:07 smithi194 ceph-mon[33017]: pgmap v17704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32647.201213 secs 2024-04-26T20:55:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32647.201213 secs 2024-04-26T20:55:09.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32647.201213 secs 2024-04-26T20:55:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:09 smithi033 ceph-mon[27845]: pgmap v17705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32650.139923 secs 2024-04-26T20:55:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:09 smithi096 ceph-mon[32753]: pgmap v17705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32650.139923 secs 2024-04-26T20:55:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:09 smithi194 ceph-mon[33017]: pgmap v17705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32650.139923 secs 2024-04-26T20:55:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:11.689+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:11.689+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:11.689+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:55:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:11 smithi033 ceph-mon[27845]: pgmap v17706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:11 smithi096 ceph-mon[32753]: pgmap v17706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:11 smithi194 ceph-mon[33017]: pgmap v17706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:13 smithi033 ceph-mon[27845]: pgmap v17707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32652.201295 secs 2024-04-26T20:55:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32655.140043 secs 2024-04-26T20:55:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:13 smithi096 ceph-mon[32753]: pgmap v17707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32652.201295 secs 2024-04-26T20:55:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32655.140043 secs 2024-04-26T20:55:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:13 smithi194 ceph-mon[33017]: pgmap v17707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32652.201295 secs 2024-04-26T20:55:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32655.140043 secs 2024-04-26T20:55:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:15 smithi033 ceph-mon[27845]: pgmap v17708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:15 smithi096 ceph-mon[32753]: pgmap v17708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:15 smithi194 ceph-mon[33017]: pgmap v17708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:17 smithi033 ceph-mon[27845]: pgmap v17709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:17 smithi096 ceph-mon[32753]: pgmap v17709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:17 smithi194 ceph-mon[33017]: pgmap v17709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32657.201350 secs 2024-04-26T20:55:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32660.140180 secs 2024-04-26T20:55:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32657.201350 secs 2024-04-26T20:55:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32660.140180 secs 2024-04-26T20:55:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32657.201350 secs 2024-04-26T20:55:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32660.140180 secs 2024-04-26T20:55:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:19 smithi033 ceph-mon[27845]: pgmap v17710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:19 smithi096 ceph-mon[32753]: pgmap v17710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:19 smithi194 ceph-mon[33017]: pgmap v17710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:21 smithi033 ceph-mon[27845]: pgmap v17711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:21 smithi096 ceph-mon[32753]: pgmap v17711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:21 smithi194 ceph-mon[33017]: pgmap v17711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:23 smithi096 ceph-mon[32753]: pgmap v17712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32662.201433 secs 2024-04-26T20:55:24.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32665.140303 secs 2024-04-26T20:55:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:23 smithi194 ceph-mon[33017]: pgmap v17712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32662.201433 secs 2024-04-26T20:55:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32665.140303 secs 2024-04-26T20:55:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:23 smithi033 ceph-mon[27845]: pgmap v17712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32662.201433 secs 2024-04-26T20:55:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32665.140303 secs 2024-04-26T20:55:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:25 smithi096 ceph-mon[32753]: pgmap v17713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:25 smithi194 ceph-mon[33017]: pgmap v17713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:25 smithi033 ceph-mon[27845]: pgmap v17713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:26.706+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:26.706+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:26.706+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:55:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:27 smithi194 ceph-mon[33017]: pgmap v17714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:27 smithi033 ceph-mon[27845]: pgmap v17714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:27 smithi096 ceph-mon[32753]: pgmap v17714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32667.201519 secs 2024-04-26T20:55:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32670.140364 secs 2024-04-26T20:55:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32667.201519 secs 2024-04-26T20:55:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32670.140364 secs 2024-04-26T20:55:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32667.201519 secs 2024-04-26T20:55:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32670.140364 secs 2024-04-26T20:55:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:29 smithi194 ceph-mon[33017]: pgmap v17715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:29 smithi033 ceph-mon[27845]: pgmap v17715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:29 smithi096 ceph-mon[32753]: pgmap v17715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:31 smithi194 ceph-mon[33017]: pgmap v17716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:31 smithi033 ceph-mon[27845]: pgmap v17716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:31 smithi096 ceph-mon[32753]: pgmap v17716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:33 smithi194 ceph-mon[33017]: pgmap v17717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32672.201577 secs 2024-04-26T20:55:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32675.140459 secs 2024-04-26T20:55:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:33 smithi033 ceph-mon[27845]: pgmap v17717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32672.201577 secs 2024-04-26T20:55:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32675.140459 secs 2024-04-26T20:55:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:33 smithi096 ceph-mon[32753]: pgmap v17717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32672.201577 secs 2024-04-26T20:55:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32675.140459 secs 2024-04-26T20:55:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:35 smithi194 ceph-mon[33017]: pgmap v17718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:35 smithi033 ceph-mon[27845]: pgmap v17718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:35 smithi096 ceph-mon[32753]: pgmap v17718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:38.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:37 smithi033 ceph-mon[27845]: pgmap v17719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:38.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:37 smithi096 ceph-mon[32753]: pgmap v17719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:37 smithi194 ceph-mon[33017]: pgmap v17719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:39.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32677.201661 secs 2024-04-26T20:55:39.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32680.140567 secs 2024-04-26T20:55:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32677.201661 secs 2024-04-26T20:55:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32680.140567 secs 2024-04-26T20:55:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32677.201661 secs 2024-04-26T20:55:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32680.140567 secs 2024-04-26T20:55:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:39 smithi194 ceph-mon[33017]: pgmap v17720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:39 smithi033 ceph-mon[27845]: pgmap v17720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:39 smithi096 ceph-mon[32753]: pgmap v17720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:41.714+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:41.715+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:41.716 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:41.715+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:55:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:41 smithi194 ceph-mon[33017]: pgmap v17721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:41 smithi033 ceph-mon[27845]: pgmap v17721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:41 smithi096 ceph-mon[32753]: pgmap v17721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32682.201742 secs 2024-04-26T20:55:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32682.201742 secs 2024-04-26T20:55:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32682.201742 secs 2024-04-26T20:55:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:44 smithi194 ceph-mon[33017]: pgmap v17722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32685.140628 secs 2024-04-26T20:55:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:44 smithi033 ceph-mon[27845]: pgmap v17722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32685.140628 secs 2024-04-26T20:55:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:44 smithi096 ceph-mon[32753]: pgmap v17722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32685.140628 secs 2024-04-26T20:55:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:46 smithi194 ceph-mon[33017]: pgmap v17723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:46 smithi033 ceph-mon[27845]: pgmap v17723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:46 smithi096 ceph-mon[32753]: pgmap v17723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:48.272 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:48 smithi033 ceph-mon[27845]: pgmap v17724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:48.272 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32687.201828 secs 2024-04-26T20:55:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:48 smithi194 ceph-mon[33017]: pgmap v17724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32687.201828 secs 2024-04-26T20:55:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:48 smithi096 ceph-mon[32753]: pgmap v17724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32687.201828 secs 2024-04-26T20:55:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:49 smithi194 ceph-mon[33017]: pgmap v17725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32690.140697 secs 2024-04-26T20:55:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:49 smithi033 ceph-mon[27845]: pgmap v17725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32690.140697 secs 2024-04-26T20:55:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:49 smithi096 ceph-mon[32753]: pgmap v17725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32690.140697 secs 2024-04-26T20:55:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:51 smithi194 ceph-mon[33017]: pgmap v17726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:51 smithi033 ceph-mon[27845]: pgmap v17726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:51 smithi096 ceph-mon[32753]: pgmap v17726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:55:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:53 smithi194 ceph-mon[33017]: pgmap v17727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32692.201911 secs 2024-04-26T20:55:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:53 smithi033 ceph-mon[27845]: pgmap v17727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32692.201911 secs 2024-04-26T20:55:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:53 smithi096 ceph-mon[32753]: pgmap v17727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32692.201911 secs 2024-04-26T20:55:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32695.140814 secs 2024-04-26T20:55:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32695.140814 secs 2024-04-26T20:55:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32695.140814 secs 2024-04-26T20:55:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:55 smithi194 ceph-mon[33017]: pgmap v17728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:55:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:55 smithi033 ceph-mon[27845]: pgmap v17728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:55:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:55 smithi096 ceph-mon[32753]: pgmap v17728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:55:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:55:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:55:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:55:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:56 smithi194 ceph-mon[33017]: pgmap v17729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:55:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:56 smithi194 ceph-mon[33017]: pgmap v17730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:55:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:55:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:55:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:56 smithi033 ceph-mon[27845]: pgmap v17729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:55:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:56 smithi033 ceph-mon[27845]: pgmap v17730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:55:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:55:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:55:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:56 smithi096 ceph-mon[32753]: pgmap v17729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T20:55:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:56 smithi096 ceph-mon[32753]: pgmap v17730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:55:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:56.730+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:56.730+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:55:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:55:56.730+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:55:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:58 smithi194 ceph-mon[33017]: pgmap v17731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32697.202012 secs 2024-04-26T20:55:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:55:58 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32700.140931 secs 2024-04-26T20:55:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:58 smithi033 ceph-mon[27845]: pgmap v17731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32697.202012 secs 2024-04-26T20:55:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:55:58 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32700.140931 secs 2024-04-26T20:55:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:58 smithi096 ceph-mon[32753]: pgmap v17731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:55:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32697.202012 secs 2024-04-26T20:55:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:55:58 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32700.140931 secs 2024-04-26T20:56:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:00 smithi194 ceph-mon[33017]: pgmap v17732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:00 smithi033 ceph-mon[27845]: pgmap v17732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:00 smithi096 ceph-mon[32753]: pgmap v17732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:02 smithi194 ceph-mon[33017]: pgmap v17733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:02 smithi033 ceph-mon[27845]: pgmap v17733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:02 smithi096 ceph-mon[32753]: pgmap v17733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32702.202071 secs 2024-04-26T20:56:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32705.141035 secs 2024-04-26T20:56:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32702.202071 secs 2024-04-26T20:56:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32705.141035 secs 2024-04-26T20:56:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32702.202071 secs 2024-04-26T20:56:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32705.141035 secs 2024-04-26T20:56:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:04 smithi194 ceph-mon[33017]: pgmap v17734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:04 smithi033 ceph-mon[27845]: pgmap v17734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:04 smithi096 ceph-mon[32753]: pgmap v17734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:06 smithi194 ceph-mon[33017]: pgmap v17735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:56:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:06 smithi033 ceph-mon[27845]: pgmap v17735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:56:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:06 smithi096 ceph-mon[32753]: pgmap v17735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:56:09.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:08 smithi096 ceph-mon[32753]: pgmap v17736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:09.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32707.202159 secs 2024-04-26T20:56:09.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32710.141151 secs 2024-04-26T20:56:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:08 smithi194 ceph-mon[33017]: pgmap v17736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32707.202159 secs 2024-04-26T20:56:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32710.141151 secs 2024-04-26T20:56:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:08 smithi033 ceph-mon[27845]: pgmap v17736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32707.202159 secs 2024-04-26T20:56:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32710.141151 secs 2024-04-26T20:56:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:10 smithi194 ceph-mon[33017]: pgmap v17737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:10 smithi033 ceph-mon[27845]: pgmap v17737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:10 smithi096 ceph-mon[32753]: pgmap v17737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:11.744+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:11.744+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:11.744+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:56:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:12 smithi194 ceph-mon[33017]: pgmap v17738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:12 smithi033 ceph-mon[27845]: pgmap v17738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:12 smithi096 ceph-mon[32753]: pgmap v17738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32712.202242 secs 2024-04-26T20:56:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32715.141269 secs 2024-04-26T20:56:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32712.202242 secs 2024-04-26T20:56:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32715.141269 secs 2024-04-26T20:56:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32712.202242 secs 2024-04-26T20:56:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32715.141269 secs 2024-04-26T20:56:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:15 smithi194 ceph-mon[33017]: pgmap v17739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:15 smithi033 ceph-mon[27845]: pgmap v17739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:15 smithi096 ceph-mon[32753]: pgmap v17739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:17 smithi194 ceph-mon[33017]: pgmap v17740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:17 smithi033 ceph-mon[27845]: pgmap v17740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:17 smithi096 ceph-mon[32753]: pgmap v17740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32717.202298 secs 2024-04-26T20:56:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32717.202298 secs 2024-04-26T20:56:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32717.202298 secs 2024-04-26T20:56:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:19 smithi194 ceph-mon[33017]: pgmap v17741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32720.141385 secs 2024-04-26T20:56:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:19 smithi033 ceph-mon[27845]: pgmap v17741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32720.141385 secs 2024-04-26T20:56:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:19 smithi096 ceph-mon[32753]: pgmap v17741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32720.141385 secs 2024-04-26T20:56:20.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:20 smithi033 ceph-mon[27845]: pgmap v17742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:20 smithi194 ceph-mon[33017]: pgmap v17742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:20 smithi096 ceph-mon[32753]: pgmap v17742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:21.808 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0058040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7feef00120c0 tx=0x7feef0058040 comp rx=0 tx=0).stop 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:56:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.807+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:56:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:21.809+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef006a040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:56:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:22 smithi194 ceph-mon[33017]: pgmap v17743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:22 smithi033 ceph-mon[27845]: pgmap v17743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:22 smithi096 ceph-mon[32753]: pgmap v17743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32722.202357 secs 2024-04-26T20:56:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32725.141506 secs 2024-04-26T20:56:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32722.202357 secs 2024-04-26T20:56:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32725.141506 secs 2024-04-26T20:56:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32722.202357 secs 2024-04-26T20:56:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32725.141506 secs 2024-04-26T20:56:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:24 smithi194 ceph-mon[33017]: pgmap v17744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:24 smithi033 ceph-mon[27845]: pgmap v17744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:24 smithi096 ceph-mon[32753]: pgmap v17744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:26.748+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:26.748+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:26.749+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:56:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:26 smithi194 ceph-mon[33017]: pgmap v17745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:27.364+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:56:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:27.364+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:56:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:27.364+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 257 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T20:56:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:27.364+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 257 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:56:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:26 smithi033 ceph-mon[27845]: pgmap v17745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:26 smithi096 ceph-mon[32753]: pgmap v17745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:28 smithi194 ceph-mon[33017]: pgmap v17746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32727.202433 secs 2024-04-26T20:56:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:28 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32730.141659 secs 2024-04-26T20:56:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:28 smithi033 ceph-mon[27845]: pgmap v17746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32727.202433 secs 2024-04-26T20:56:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:28 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32730.141659 secs 2024-04-26T20:56:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:28 smithi096 ceph-mon[32753]: pgmap v17746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32727.202433 secs 2024-04-26T20:56:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:28 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32730.141659 secs 2024-04-26T20:56:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:30 smithi194 ceph-mon[33017]: pgmap v17747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:30 smithi033 ceph-mon[27845]: pgmap v17747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:30 smithi096 ceph-mon[32753]: pgmap v17747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:32 smithi194 ceph-mon[33017]: pgmap v17748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:32 smithi033 ceph-mon[27845]: pgmap v17748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:32 smithi096 ceph-mon[32753]: pgmap v17748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32732.202516 secs 2024-04-26T20:56:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32735.141778 secs 2024-04-26T20:56:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32732.202516 secs 2024-04-26T20:56:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32735.141778 secs 2024-04-26T20:56:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32732.202516 secs 2024-04-26T20:56:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32735.141778 secs 2024-04-26T20:56:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:35 smithi033 ceph-mon[27845]: pgmap v17749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:35 smithi194 ceph-mon[33017]: pgmap v17749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:35 smithi096 ceph-mon[32753]: pgmap v17749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:37 smithi194 ceph-mon[33017]: pgmap v17750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:37 smithi033 ceph-mon[27845]: pgmap v17750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:37 smithi096 ceph-mon[32753]: pgmap v17750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32737.202601 secs 2024-04-26T20:56:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32737.202601 secs 2024-04-26T20:56:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32737.202601 secs 2024-04-26T20:56:39.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:39 smithi033 ceph-mon[27845]: pgmap v17751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:39.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32740.141873 secs 2024-04-26T20:56:39.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:39 smithi096 ceph-mon[32753]: pgmap v17751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:39.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32740.141873 secs 2024-04-26T20:56:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:39 smithi194 ceph-mon[33017]: pgmap v17751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32740.141873 secs 2024-04-26T20:56:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:41 smithi194 ceph-mon[33017]: pgmap v17752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:41 smithi033 ceph-mon[27845]: pgmap v17752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:41 smithi096 ceph-mon[32753]: pgmap v17752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:41.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:41.762+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:41.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:41.762+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:56:41.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:41.762+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:42 smithi194 ceph-mon[33017]: pgmap v17753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:42 smithi033 ceph-mon[27845]: pgmap v17753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:42 smithi096 ceph-mon[32753]: pgmap v17753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32742.202683 secs 2024-04-26T20:56:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32742.202683 secs 2024-04-26T20:56:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32742.202683 secs 2024-04-26T20:56:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32745.141991 secs 2024-04-26T20:56:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:44 smithi194 ceph-mon[33017]: pgmap v17754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32745.141991 secs 2024-04-26T20:56:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:44 smithi033 ceph-mon[27845]: pgmap v17754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32745.141991 secs 2024-04-26T20:56:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:44 smithi096 ceph-mon[32753]: pgmap v17754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:46 smithi194 ceph-mon[33017]: pgmap v17755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:46 smithi033 ceph-mon[27845]: pgmap v17755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:46 smithi096 ceph-mon[32753]: pgmap v17755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:48 smithi194 ceph-mon[33017]: pgmap v17756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32747.202764 secs 2024-04-26T20:56:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32750.142052 secs 2024-04-26T20:56:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:48 smithi033 ceph-mon[27845]: pgmap v17756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32747.202764 secs 2024-04-26T20:56:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32750.142052 secs 2024-04-26T20:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:48 smithi096 ceph-mon[32753]: pgmap v17756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32747.202764 secs 2024-04-26T20:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32750.142052 secs 2024-04-26T20:56:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:50 smithi194 ceph-mon[33017]: pgmap v17757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:50 smithi033 ceph-mon[27845]: pgmap v17757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:50 smithi096 ceph-mon[32753]: pgmap v17757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:52 smithi194 ceph-mon[33017]: pgmap v17758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:52 smithi033 ceph-mon[27845]: pgmap v17758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:52 smithi096 ceph-mon[32753]: pgmap v17758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32752.202836 secs 2024-04-26T20:56:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32755.142151 secs 2024-04-26T20:56:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32752.202836 secs 2024-04-26T20:56:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32755.142151 secs 2024-04-26T20:56:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32752.202836 secs 2024-04-26T20:56:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32755.142151 secs 2024-04-26T20:56:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:55 smithi194 ceph-mon[33017]: pgmap v17759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:55 smithi033 ceph-mon[27845]: pgmap v17759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:55 smithi096 ceph-mon[32753]: pgmap v17759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:56.777+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:56.777+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:56:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:56:56.777+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:56:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:57 smithi194 ceph-mon[33017]: pgmap v17760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:56:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:57 smithi033 ceph-mon[27845]: pgmap v17760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:56:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:57 smithi096 ceph-mon[32753]: pgmap v17760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:56:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:56:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:56:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:56:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:56:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32757.202922 secs 2024-04-26T20:56:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:56:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:56:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:56:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32757.202922 secs 2024-04-26T20:56:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:56:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:56:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:56:58.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32757.202922 secs 2024-04-26T20:56:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:59 smithi194 ceph-mon[33017]: pgmap v17761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:59 smithi194 ceph-mon[33017]: pgmap v17762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:56:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32760.142269 secs 2024-04-26T20:56:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:59 smithi033 ceph-mon[27845]: pgmap v17761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:59 smithi033 ceph-mon[27845]: pgmap v17762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:56:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32760.142269 secs 2024-04-26T20:56:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:59 smithi096 ceph-mon[32753]: pgmap v17761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:56:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:59 smithi096 ceph-mon[32753]: pgmap v17762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:56:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:56:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32760.142269 secs 2024-04-26T20:57:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:01 smithi194 ceph-mon[33017]: pgmap v17763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:01 smithi033 ceph-mon[27845]: pgmap v17763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:01 smithi096 ceph-mon[32753]: pgmap v17763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:03 smithi194 ceph-mon[33017]: pgmap v17764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32762.202993 secs 2024-04-26T20:57:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:03 smithi033 ceph-mon[27845]: pgmap v17764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32762.202993 secs 2024-04-26T20:57:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:03 smithi096 ceph-mon[32753]: pgmap v17764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32762.202993 secs 2024-04-26T20:57:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:04 smithi194 ceph-mon[33017]: pgmap v17765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32765.142343 secs 2024-04-26T20:57:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:04 smithi033 ceph-mon[27845]: pgmap v17765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32765.142343 secs 2024-04-26T20:57:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:04 smithi096 ceph-mon[32753]: pgmap v17765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32765.142343 secs 2024-04-26T20:57:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:06 smithi194 ceph-mon[33017]: pgmap v17766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:06 smithi033 ceph-mon[27845]: pgmap v17766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:06 smithi096 ceph-mon[32753]: pgmap v17766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:57:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:08 smithi194 ceph-mon[33017]: pgmap v17767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:57:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32767.203083 secs 2024-04-26T20:57:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:08 smithi033 ceph-mon[27845]: pgmap v17767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:57:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32767.203083 secs 2024-04-26T20:57:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:08 smithi096 ceph-mon[32753]: pgmap v17767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:57:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32767.203083 secs 2024-04-26T20:57:10.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32770.142469 secs 2024-04-26T20:57:10.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32770.142469 secs 2024-04-26T20:57:10.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32770.142469 secs 2024-04-26T20:57:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:10 smithi194 ceph-mon[33017]: pgmap v17768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:10 smithi033 ceph-mon[27845]: pgmap v17768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:10 smithi096 ceph-mon[32753]: pgmap v17768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:11.792+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:11.792+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:11.792+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:57:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:12 smithi194 ceph-mon[33017]: pgmap v17769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:12 smithi033 ceph-mon[27845]: pgmap v17769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:12 smithi096 ceph-mon[32753]: pgmap v17769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32772.203140 secs 2024-04-26T20:57:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32772.203140 secs 2024-04-26T20:57:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32772.203140 secs 2024-04-26T20:57:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:14 smithi194 ceph-mon[33017]: pgmap v17770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32775.142582 secs 2024-04-26T20:57:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:14 smithi033 ceph-mon[27845]: pgmap v17770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32775.142582 secs 2024-04-26T20:57:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:14 smithi096 ceph-mon[32753]: pgmap v17770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32775.142582 secs 2024-04-26T20:57:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:16 smithi194 ceph-mon[33017]: pgmap v17771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:16 smithi033 ceph-mon[27845]: pgmap v17771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:16 smithi096 ceph-mon[32753]: pgmap v17771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:18 smithi194 ceph-mon[33017]: pgmap v17772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32777.203228 secs 2024-04-26T20:57:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:18 smithi033 ceph-mon[27845]: pgmap v17772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32777.203228 secs 2024-04-26T20:57:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:18 smithi096 ceph-mon[32753]: pgmap v17772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32777.203228 secs 2024-04-26T20:57:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32780.142651 secs 2024-04-26T20:57:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32780.142651 secs 2024-04-26T20:57:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32780.142651 secs 2024-04-26T20:57:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:20 smithi194 ceph-mon[33017]: pgmap v17773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:20 smithi033 ceph-mon[27845]: pgmap v17773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:20 smithi096 ceph-mon[32753]: pgmap v17773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:22 smithi194 ceph-mon[33017]: pgmap v17774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:22 smithi033 ceph-mon[27845]: pgmap v17774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:22 smithi096 ceph-mon[32753]: pgmap v17774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32782.203311 secs 2024-04-26T20:57:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32782.203311 secs 2024-04-26T20:57:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32782.203311 secs 2024-04-26T20:57:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:24 smithi194 ceph-mon[33017]: pgmap v17775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32785.142727 secs 2024-04-26T20:57:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:24 smithi033 ceph-mon[27845]: pgmap v17775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32785.142727 secs 2024-04-26T20:57:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:24 smithi096 ceph-mon[32753]: pgmap v17775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32785.142727 secs 2024-04-26T20:57:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:26.808+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:26.808+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:26.808+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:57:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:26 smithi194 ceph-mon[33017]: pgmap v17776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:26 smithi033 ceph-mon[27845]: pgmap v17776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:26 smithi096 ceph-mon[32753]: pgmap v17776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:28 smithi194 ceph-mon[33017]: pgmap v17777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32787.203370 secs 2024-04-26T20:57:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:28 smithi033 ceph-mon[27845]: pgmap v17777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32787.203370 secs 2024-04-26T20:57:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:28 smithi096 ceph-mon[32753]: pgmap v17777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32787.203370 secs 2024-04-26T20:57:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32790.142860 secs 2024-04-26T20:57:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32790.142860 secs 2024-04-26T20:57:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32790.142860 secs 2024-04-26T20:57:31.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:30 smithi194 ceph-mon[33017]: pgmap v17778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:30 smithi033 ceph-mon[27845]: pgmap v17778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:30 smithi096 ceph-mon[32753]: pgmap v17778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:32 smithi194 ceph-mon[33017]: pgmap v17779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:32 smithi033 ceph-mon[27845]: pgmap v17779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:32 smithi096 ceph-mon[32753]: pgmap v17779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32792.203449 secs 2024-04-26T20:57:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32792.203449 secs 2024-04-26T20:57:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32792.203449 secs 2024-04-26T20:57:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:34 smithi194 ceph-mon[33017]: pgmap v17780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32795.142983 secs 2024-04-26T20:57:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:34 smithi033 ceph-mon[27845]: pgmap v17780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:35.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32795.142983 secs 2024-04-26T20:57:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:34 smithi096 ceph-mon[32753]: pgmap v17780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32795.142983 secs 2024-04-26T20:57:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:36 smithi194 ceph-mon[33017]: pgmap v17781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:36 smithi033 ceph-mon[27845]: pgmap v17781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:36 smithi096 ceph-mon[32753]: pgmap v17781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:38 smithi194 ceph-mon[33017]: pgmap v17782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32797.203534 secs 2024-04-26T20:57:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:38 smithi033 ceph-mon[27845]: pgmap v17782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32797.203534 secs 2024-04-26T20:57:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:38 smithi096 ceph-mon[32753]: pgmap v17782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32797.203534 secs 2024-04-26T20:57:40.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32800.143138 secs 2024-04-26T20:57:40.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32800.143138 secs 2024-04-26T20:57:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32800.143138 secs 2024-04-26T20:57:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:40 smithi194 ceph-mon[33017]: pgmap v17783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:40 smithi033 ceph-mon[27845]: pgmap v17783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:40 smithi096 ceph-mon[32753]: pgmap v17783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:41.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:41.820+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:41.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:41.820+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:41.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:41.820+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:57:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:42 smithi194 ceph-mon[33017]: pgmap v17784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:43.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:42 smithi033 ceph-mon[27845]: pgmap v17784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:42 smithi096 ceph-mon[32753]: pgmap v17784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32802.203614 secs 2024-04-26T20:57:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32802.203614 secs 2024-04-26T20:57:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32802.203614 secs 2024-04-26T20:57:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:44 smithi194 ceph-mon[33017]: pgmap v17785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32805.143236 secs 2024-04-26T20:57:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:44 smithi033 ceph-mon[27845]: pgmap v17785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32805.143236 secs 2024-04-26T20:57:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:44 smithi096 ceph-mon[32753]: pgmap v17785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32805.143236 secs 2024-04-26T20:57:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:46 smithi194 ceph-mon[33017]: pgmap v17786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:46 smithi033 ceph-mon[27845]: pgmap v17786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:46 smithi096 ceph-mon[32753]: pgmap v17786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:48 smithi194 ceph-mon[33017]: pgmap v17787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32807.203692 secs 2024-04-26T20:57:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:48 smithi033 ceph-mon[27845]: pgmap v17787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32807.203692 secs 2024-04-26T20:57:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:48 smithi096 ceph-mon[32753]: pgmap v17787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32807.203692 secs 2024-04-26T20:57:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32810.143344 secs 2024-04-26T20:57:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32810.143344 secs 2024-04-26T20:57:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32810.143344 secs 2024-04-26T20:57:51.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:50 smithi033 ceph-mon[27845]: pgmap v17788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:50 smithi096 ceph-mon[32753]: pgmap v17788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:50 smithi194 ceph-mon[33017]: pgmap v17788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:52 smithi033 ceph-mon[27845]: pgmap v17789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:52 smithi096 ceph-mon[32753]: pgmap v17789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:52 smithi194 ceph-mon[33017]: pgmap v17789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32812.203780 secs 2024-04-26T20:57:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32812.203780 secs 2024-04-26T20:57:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32812.203780 secs 2024-04-26T20:57:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:54 smithi033 ceph-mon[27845]: pgmap v17790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32815.143412 secs 2024-04-26T20:57:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:54 smithi096 ceph-mon[32753]: pgmap v17790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32815.143412 secs 2024-04-26T20:57:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:54 smithi194 ceph-mon[33017]: pgmap v17790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32815.143412 secs 2024-04-26T20:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:56.830+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:56.830+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:57:56.830+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:57:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:56 smithi033 ceph-mon[27845]: pgmap v17791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:56 smithi096 ceph-mon[32753]: pgmap v17791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:56 smithi194 ceph-mon[33017]: pgmap v17791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:58 smithi033 ceph-mon[27845]: pgmap v17792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:57:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32817.203866 secs 2024-04-26T20:57:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:58 smithi096 ceph-mon[32753]: pgmap v17792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:57:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32817.203866 secs 2024-04-26T20:57:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:58 smithi194 ceph-mon[33017]: pgmap v17792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:57:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:57:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32817.203866 secs 2024-04-26T20:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32820.143550 secs 2024-04-26T20:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:58:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:57:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32820.143550 secs 2024-04-26T20:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:58:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:57:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:58:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32820.143550 secs 2024-04-26T20:58:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:58:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:58:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:57:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:58:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:00 smithi033 ceph-mon[27845]: pgmap v17793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:00 smithi033 ceph-mon[27845]: pgmap v17794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:58:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:00 smithi096 ceph-mon[32753]: pgmap v17793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:00 smithi096 ceph-mon[32753]: pgmap v17794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:58:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:00 smithi194 ceph-mon[33017]: pgmap v17793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:00 smithi194 ceph-mon[33017]: pgmap v17794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T20:58:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:02 smithi033 ceph-mon[27845]: pgmap v17795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:02 smithi096 ceph-mon[32753]: pgmap v17795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:02 smithi194 ceph-mon[33017]: pgmap v17795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32822.203947 secs 2024-04-26T20:58:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:03 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32825.143669 secs 2024-04-26T20:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32822.203947 secs 2024-04-26T20:58:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:03 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32825.143669 secs 2024-04-26T20:58:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32822.203947 secs 2024-04-26T20:58:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:03 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32825.143669 secs 2024-04-26T20:58:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:04 smithi096 ceph-mon[32753]: pgmap v17796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:04 smithi194 ceph-mon[33017]: pgmap v17796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:04 smithi033 ceph-mon[27845]: pgmap v17796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:06 smithi096 ceph-mon[32753]: pgmap v17797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:06 smithi194 ceph-mon[33017]: pgmap v17797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:06 smithi033 ceph-mon[27845]: pgmap v17797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:08 smithi194 ceph-mon[33017]: pgmap v17798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32827.204035 secs 2024-04-26T20:58:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:08 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32830.143790 secs 2024-04-26T20:58:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:08 smithi033 ceph-mon[27845]: pgmap v17798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32827.204035 secs 2024-04-26T20:58:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:08 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32830.143790 secs 2024-04-26T20:58:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:08 smithi096 ceph-mon[32753]: pgmap v17798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:58:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32827.204035 secs 2024-04-26T20:58:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:08 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32830.143790 secs 2024-04-26T20:58:11.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:10 smithi033 ceph-mon[27845]: pgmap v17799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:58:11.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:10 smithi096 ceph-mon[32753]: pgmap v17799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:58:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:10 smithi194 ceph-mon[33017]: pgmap v17799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:58:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:11.845+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:11.845+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:11.845+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:58:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:12 smithi194 ceph-mon[33017]: pgmap v17800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:12 smithi033 ceph-mon[27845]: pgmap v17800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:12 smithi096 ceph-mon[32753]: pgmap v17800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32832.204121 secs 2024-04-26T20:58:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:13 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32835.143926 secs 2024-04-26T20:58:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32832.204121 secs 2024-04-26T20:58:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:13 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32835.143926 secs 2024-04-26T20:58:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32832.204121 secs 2024-04-26T20:58:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:13 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32835.143926 secs 2024-04-26T20:58:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:14 smithi194 ceph-mon[33017]: pgmap v17801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:14 smithi033 ceph-mon[27845]: pgmap v17801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:14 smithi096 ceph-mon[32753]: pgmap v17801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:16 smithi194 ceph-mon[33017]: pgmap v17802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:16 smithi033 ceph-mon[27845]: pgmap v17802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:16 smithi096 ceph-mon[32753]: pgmap v17802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:18 smithi194 ceph-mon[33017]: pgmap v17803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32837.204183 secs 2024-04-26T20:58:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:18 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32840.144063 secs 2024-04-26T20:58:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:18 smithi033 ceph-mon[27845]: pgmap v17803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32837.204183 secs 2024-04-26T20:58:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:18 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32840.144063 secs 2024-04-26T20:58:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:18 smithi096 ceph-mon[32753]: pgmap v17803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32837.204183 secs 2024-04-26T20:58:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:18 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32840.144063 secs 2024-04-26T20:58:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:20 smithi194 ceph-mon[33017]: pgmap v17804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:20 smithi033 ceph-mon[27845]: pgmap v17804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:20 smithi096 ceph-mon[32753]: pgmap v17804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:22 smithi194 ceph-mon[33017]: pgmap v17805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:22 smithi033 ceph-mon[27845]: pgmap v17805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:22 smithi096 ceph-mon[32753]: pgmap v17805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32842.204263 secs 2024-04-26T20:58:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32845.144191 secs 2024-04-26T20:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32842.204263 secs 2024-04-26T20:58:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32845.144191 secs 2024-04-26T20:58:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32842.204263 secs 2024-04-26T20:58:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32845.144191 secs 2024-04-26T20:58:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:25 smithi194 ceph-mon[33017]: pgmap v17806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:25 smithi033 ceph-mon[27845]: pgmap v17806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:25 smithi096 ceph-mon[32753]: pgmap v17806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:26.862+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:26.862+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:26.862+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:58:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:27 smithi194 ceph-mon[33017]: pgmap v17807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:27 smithi033 ceph-mon[27845]: pgmap v17807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:27 smithi096 ceph-mon[32753]: pgmap v17807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32847.204349 secs 2024-04-26T20:58:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32847.204349 secs 2024-04-26T20:58:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32847.204349 secs 2024-04-26T20:58:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:29 smithi194 ceph-mon[33017]: pgmap v17808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32850.144299 secs 2024-04-26T20:58:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:29 smithi033 ceph-mon[27845]: pgmap v17808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32850.144299 secs 2024-04-26T20:58:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:29 smithi096 ceph-mon[32753]: pgmap v17808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32850.144299 secs 2024-04-26T20:58:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:31 smithi194 ceph-mon[33017]: pgmap v17809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:31 smithi033 ceph-mon[27845]: pgmap v17809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:31 smithi096 ceph-mon[32753]: pgmap v17809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:33 smithi194 ceph-mon[33017]: pgmap v17810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32852.204424 secs 2024-04-26T20:58:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:33 smithi033 ceph-mon[27845]: pgmap v17810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32852.204424 secs 2024-04-26T20:58:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:33 smithi096 ceph-mon[32753]: pgmap v17810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32852.204424 secs 2024-04-26T20:58:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32855.144430 secs 2024-04-26T20:58:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32855.144430 secs 2024-04-26T20:58:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32855.144430 secs 2024-04-26T20:58:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:35 smithi194 ceph-mon[33017]: pgmap v17811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:35 smithi033 ceph-mon[27845]: pgmap v17811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:35 smithi096 ceph-mon[32753]: pgmap v17811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:36 smithi194 ceph-mon[33017]: pgmap v17812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:36 smithi033 ceph-mon[27845]: pgmap v17812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:36 smithi096 ceph-mon[32753]: pgmap v17812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:38 smithi033 ceph-mon[27845]: pgmap v17813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32857.204509 secs 2024-04-26T20:58:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:38 smithi096 ceph-mon[32753]: pgmap v17813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32857.204509 secs 2024-04-26T20:58:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:38 smithi194 ceph-mon[33017]: pgmap v17813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32857.204509 secs 2024-04-26T20:58:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32860.144554 secs 2024-04-26T20:58:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32860.144554 secs 2024-04-26T20:58:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32860.144554 secs 2024-04-26T20:58:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:40 smithi033 ceph-mon[27845]: pgmap v17814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:40 smithi096 ceph-mon[32753]: pgmap v17814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:40 smithi194 ceph-mon[33017]: pgmap v17814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:41.877+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:41.877+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:58:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:41.877+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:42.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:42 smithi194 ceph-mon[33017]: pgmap v17815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:42 smithi033 ceph-mon[27845]: pgmap v17815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:42 smithi096 ceph-mon[32753]: pgmap v17815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32862.204592 secs 2024-04-26T20:58:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32862.204592 secs 2024-04-26T20:58:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32862.204592 secs 2024-04-26T20:58:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:44 smithi033 ceph-mon[27845]: pgmap v17816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32865.144693 secs 2024-04-26T20:58:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:44 smithi096 ceph-mon[32753]: pgmap v17816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32865.144693 secs 2024-04-26T20:58:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:44 smithi194 ceph-mon[33017]: pgmap v17816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32865.144693 secs 2024-04-26T20:58:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:46 smithi033 ceph-mon[27845]: pgmap v17817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:46 smithi096 ceph-mon[32753]: pgmap v17817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:46 smithi194 ceph-mon[33017]: pgmap v17817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:48 smithi096 ceph-mon[32753]: pgmap v17818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32867.204675 secs 2024-04-26T20:58:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:48 smithi194 ceph-mon[33017]: pgmap v17818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32867.204675 secs 2024-04-26T20:58:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:48 smithi033 ceph-mon[27845]: pgmap v17818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32867.204675 secs 2024-04-26T20:58:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32870.144814 secs 2024-04-26T20:58:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32870.144814 secs 2024-04-26T20:58:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32870.144814 secs 2024-04-26T20:58:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:50 smithi194 ceph-mon[33017]: pgmap v17819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:50 smithi033 ceph-mon[27845]: pgmap v17819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:50 smithi096 ceph-mon[32753]: pgmap v17819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:52 smithi194 ceph-mon[33017]: pgmap v17820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:52 smithi033 ceph-mon[27845]: pgmap v17820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:52 smithi096 ceph-mon[32753]: pgmap v17820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32872.204756 secs 2024-04-26T20:58:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32872.204756 secs 2024-04-26T20:58:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32872.204756 secs 2024-04-26T20:58:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:54 smithi194 ceph-mon[33017]: pgmap v17821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32875.144937 secs 2024-04-26T20:58:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:54 smithi033 ceph-mon[27845]: pgmap v17821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32875.144937 secs 2024-04-26T20:58:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:54 smithi096 ceph-mon[32753]: pgmap v17821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32875.144937 secs 2024-04-26T20:58:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:56 smithi194 ceph-mon[33017]: pgmap v17822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:56.888+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:56.888+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:58:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:58:56.888+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:58:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:56 smithi033 ceph-mon[27845]: pgmap v17822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:56 smithi096 ceph-mon[32753]: pgmap v17822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:58:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:58 smithi194 ceph-mon[33017]: pgmap v17823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32877.204841 secs 2024-04-26T20:58:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:58 smithi033 ceph-mon[27845]: pgmap v17823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32877.204841 secs 2024-04-26T20:58:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:58 smithi096 ceph-mon[32753]: pgmap v17823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:58:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32877.204841 secs 2024-04-26T20:58:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32880.145057 secs 2024-04-26T20:58:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32880.145057 secs 2024-04-26T20:58:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32880.145057 secs 2024-04-26T20:59:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:00 smithi194 ceph-mon[33017]: pgmap v17824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:59:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:00 smithi033 ceph-mon[27845]: pgmap v17824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:59:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:00 smithi096 ceph-mon[32753]: pgmap v17824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:59:02.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:02 smithi194 ceph-mon[33017]: pgmap v17825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:02.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:02 smithi033 ceph-mon[27845]: pgmap v17825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:02 smithi096 ceph-mon[32753]: pgmap v17825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32882.204948 secs 2024-04-26T20:59:03.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32882.204948 secs 2024-04-26T20:59:03.877 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32882.204948 secs 2024-04-26T20:59:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:04 smithi194 ceph-mon[33017]: pgmap v17826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32885.145157 secs 2024-04-26T20:59:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:04 smithi033 ceph-mon[27845]: pgmap v17826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32885.145157 secs 2024-04-26T20:59:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:04 smithi096 ceph-mon[32753]: pgmap v17826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32885.145157 secs 2024-04-26T20:59:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:04.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:06 smithi194 ceph-mon[33017]: pgmap v17827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:06.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:06 smithi033 ceph-mon[27845]: pgmap v17827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:06 smithi096 ceph-mon[32753]: pgmap v17827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: pgmap v17828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: pgmap v17829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32887.205032 secs 2024-04-26T20:59:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:59:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:59:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: pgmap v17828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: pgmap v17829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32887.205032 secs 2024-04-26T20:59:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:59:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:59:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: pgmap v17828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: pgmap v17829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T20:59:08.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32887.205032 secs 2024-04-26T20:59:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32890.145273 secs 2024-04-26T20:59:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32890.145273 secs 2024-04-26T20:59:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32890.145273 secs 2024-04-26T20:59:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:10 smithi194 ceph-mon[33017]: pgmap v17830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:10 smithi033 ceph-mon[27845]: pgmap v17830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:10 smithi096 ceph-mon[32753]: pgmap v17830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:11 smithi033 ceph-mon[27845]: pgmap v17831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:11 smithi096 ceph-mon[32753]: pgmap v17831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:11 smithi194 ceph-mon[33017]: pgmap v17831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:11.897+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:11.897+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:59:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:11.897+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32892.205102 secs 2024-04-26T20:59:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:13 smithi194 ceph-mon[33017]: pgmap v17832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32892.205102 secs 2024-04-26T20:59:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:13 smithi033 ceph-mon[27845]: pgmap v17832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32892.205102 secs 2024-04-26T20:59:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:13 smithi096 ceph-mon[32753]: pgmap v17832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32895.145395 secs 2024-04-26T20:59:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32895.145395 secs 2024-04-26T20:59:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32895.145395 secs 2024-04-26T20:59:15.421 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:15 smithi033 ceph-mon[27845]: pgmap v17833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:15 smithi096 ceph-mon[32753]: pgmap v17833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:15 smithi194 ceph-mon[33017]: pgmap v17833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T20:59:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:17 smithi033 ceph-mon[27845]: pgmap v17834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:59:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:17 smithi096 ceph-mon[32753]: pgmap v17834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:59:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:17 smithi194 ceph-mon[33017]: pgmap v17834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T20:59:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32897.205186 secs 2024-04-26T20:59:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32897.205186 secs 2024-04-26T20:59:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32897.205186 secs 2024-04-26T20:59:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32900.145512 secs 2024-04-26T20:59:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:19 smithi033 ceph-mon[27845]: pgmap v17835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32900.145512 secs 2024-04-26T20:59:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:19 smithi096 ceph-mon[32753]: pgmap v17835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32900.145512 secs 2024-04-26T20:59:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:19 smithi194 ceph-mon[33017]: pgmap v17835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:21 smithi033 ceph-mon[27845]: pgmap v17836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:21 smithi096 ceph-mon[32753]: pgmap v17836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:21 smithi194 ceph-mon[33017]: pgmap v17836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32902.205288 secs 2024-04-26T20:59:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:23 smithi194 ceph-mon[33017]: pgmap v17837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32902.205288 secs 2024-04-26T20:59:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:23 smithi033 ceph-mon[27845]: pgmap v17837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32902.205288 secs 2024-04-26T20:59:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:23 smithi096 ceph-mon[32753]: pgmap v17837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32905.145672 secs 2024-04-26T20:59:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32905.145672 secs 2024-04-26T20:59:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32905.145672 secs 2024-04-26T20:59:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:25 smithi033 ceph-mon[27845]: pgmap v17838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:25 smithi096 ceph-mon[32753]: pgmap v17838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:25 smithi194 ceph-mon[33017]: pgmap v17838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:26.912+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:26.912+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:26.912+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:59:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:27.394+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T20:59:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:27.394+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T20:59:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:27.394+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 258 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T20:59:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:27.394+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 258 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T20:59:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:27 smithi033 ceph-mon[27845]: pgmap v17839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:27 smithi096 ceph-mon[32753]: pgmap v17839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:27 smithi194 ceph-mon[33017]: pgmap v17839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32907.205374 secs 2024-04-26T20:59:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32907.205374 secs 2024-04-26T20:59:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32907.205374 secs 2024-04-26T20:59:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32910.145761 secs 2024-04-26T20:59:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:29 smithi033 ceph-mon[27845]: pgmap v17840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32910.145761 secs 2024-04-26T20:59:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:29 smithi096 ceph-mon[32753]: pgmap v17840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32910.145761 secs 2024-04-26T20:59:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:29 smithi194 ceph-mon[33017]: pgmap v17840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:31 smithi033 ceph-mon[27845]: pgmap v17841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:31 smithi096 ceph-mon[32753]: pgmap v17841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:31 smithi194 ceph-mon[33017]: pgmap v17841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32912.205453 secs 2024-04-26T20:59:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:33 smithi194 ceph-mon[33017]: pgmap v17842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32912.205453 secs 2024-04-26T20:59:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:33 smithi033 ceph-mon[27845]: pgmap v17842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32912.205453 secs 2024-04-26T20:59:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:33 smithi096 ceph-mon[32753]: pgmap v17842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32915.145872 secs 2024-04-26T20:59:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32915.145872 secs 2024-04-26T20:59:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32915.145872 secs 2024-04-26T20:59:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:36 smithi033 ceph-mon[27845]: pgmap v17843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:36 smithi096 ceph-mon[32753]: pgmap v17843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:36 smithi194 ceph-mon[33017]: pgmap v17843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:38 smithi033 ceph-mon[27845]: pgmap v17844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32917.205540 secs 2024-04-26T20:59:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:38 smithi096 ceph-mon[32753]: pgmap v17844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32917.205540 secs 2024-04-26T20:59:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:38 smithi194 ceph-mon[33017]: pgmap v17844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32917.205540 secs 2024-04-26T20:59:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32920.146007 secs 2024-04-26T20:59:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32920.146007 secs 2024-04-26T20:59:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32920.146007 secs 2024-04-26T20:59:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:40 smithi033 ceph-mon[27845]: pgmap v17845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:40 smithi096 ceph-mon[32753]: pgmap v17845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:40 smithi194 ceph-mon[33017]: pgmap v17845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:41.926+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:41.927+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:41.927+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:59:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:42 smithi033 ceph-mon[27845]: pgmap v17846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:42 smithi096 ceph-mon[32753]: pgmap v17846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:42 smithi194 ceph-mon[33017]: pgmap v17846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32922.205616 secs 2024-04-26T20:59:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32922.205616 secs 2024-04-26T20:59:43.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32922.205616 secs 2024-04-26T20:59:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:44 smithi033 ceph-mon[27845]: pgmap v17847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32925.146183 secs 2024-04-26T20:59:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:44 smithi096 ceph-mon[32753]: pgmap v17847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32925.146183 secs 2024-04-26T20:59:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:44 smithi194 ceph-mon[33017]: pgmap v17847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32925.146183 secs 2024-04-26T20:59:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:46 smithi033 ceph-mon[27845]: pgmap v17848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:46 smithi096 ceph-mon[32753]: pgmap v17848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:46 smithi194 ceph-mon[33017]: pgmap v17848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:48 smithi033 ceph-mon[27845]: pgmap v17849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32927.205699 secs 2024-04-26T20:59:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:48 smithi096 ceph-mon[32753]: pgmap v17849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32927.205699 secs 2024-04-26T20:59:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:48 smithi194 ceph-mon[33017]: pgmap v17849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32927.205699 secs 2024-04-26T20:59:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32930.146304 secs 2024-04-26T20:59:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32930.146304 secs 2024-04-26T20:59:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32930.146304 secs 2024-04-26T20:59:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:50 smithi096 ceph-mon[32753]: pgmap v17850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:50 smithi194 ceph-mon[33017]: pgmap v17850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:50 smithi033 ceph-mon[27845]: pgmap v17850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:52 smithi194 ceph-mon[33017]: pgmap v17851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:52 smithi033 ceph-mon[27845]: pgmap v17851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:52 smithi096 ceph-mon[32753]: pgmap v17851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32932.205780 secs 2024-04-26T20:59:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32932.205780 secs 2024-04-26T20:59:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32932.205780 secs 2024-04-26T20:59:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:54 smithi194 ceph-mon[33017]: pgmap v17852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32935.146398 secs 2024-04-26T20:59:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:54 smithi033 ceph-mon[27845]: pgmap v17852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32935.146398 secs 2024-04-26T20:59:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:54 smithi096 ceph-mon[32753]: pgmap v17852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32935.146398 secs 2024-04-26T20:59:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:56 smithi194 ceph-mon[33017]: pgmap v17853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:56 smithi033 ceph-mon[27845]: pgmap v17853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:56 smithi096 ceph-mon[32753]: pgmap v17853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:59:56.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:56.931+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:56.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:56.931+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T20:59:56.932 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T20:59:56.931+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T20:59:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:58 smithi194 ceph-mon[33017]: pgmap v17854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32937.205865 secs 2024-04-26T20:59:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:58 smithi033 ceph-mon[27845]: pgmap v17854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32937.205865 secs 2024-04-26T20:59:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:58 smithi096 ceph-mon[32753]: pgmap v17854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T20:59:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32937.205865 secs 2024-04-26T20:59:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 20:59:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32940.146468 secs 2024-04-26T20:59:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 20:59:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32940.146468 secs 2024-04-26T20:59:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 20:59:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32940.146468 secs 2024-04-26T21:00:00.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:00 smithi033 ceph-mon[27845]: pgmap v17855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:00.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:00:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:00 smithi194 ceph-mon[33017]: pgmap v17855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:00 smithi096 ceph-mon[32753]: pgmap v17855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:00:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:02 smithi194 ceph-mon[33017]: pgmap v17856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:02 smithi033 ceph-mon[27845]: pgmap v17856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:02 smithi096 ceph-mon[32753]: pgmap v17856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32942.205918 secs 2024-04-26T21:00:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32942.205918 secs 2024-04-26T21:00:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32942.205918 secs 2024-04-26T21:00:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:04 smithi194 ceph-mon[33017]: pgmap v17857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32945.146534 secs 2024-04-26T21:00:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:04 smithi033 ceph-mon[27845]: pgmap v17857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32945.146534 secs 2024-04-26T21:00:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:04 smithi096 ceph-mon[32753]: pgmap v17857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32945.146534 secs 2024-04-26T21:00:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:06 smithi194 ceph-mon[33017]: pgmap v17858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:06 smithi033 ceph-mon[27845]: pgmap v17858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:06 smithi096 ceph-mon[32753]: pgmap v17858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:08 smithi194 ceph-mon[33017]: pgmap v17859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:00:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32947.206015 secs 2024-04-26T21:00:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:08 smithi033 ceph-mon[27845]: pgmap v17859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:00:08.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32947.206015 secs 2024-04-26T21:00:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:08 smithi096 ceph-mon[32753]: pgmap v17859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:00:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32947.206015 secs 2024-04-26T21:00:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32950.146668 secs 2024-04-26T21:00:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:00:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:00:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:09 smithi194 ceph-mon[33017]: pgmap v17860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:09 smithi194 ceph-mon[33017]: pgmap v17861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:09.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:00:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32950.146668 secs 2024-04-26T21:00:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:00:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:00:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:09 smithi033 ceph-mon[27845]: pgmap v17860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:09 smithi033 ceph-mon[27845]: pgmap v17861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:09.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:00:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32950.146668 secs 2024-04-26T21:00:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:00:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:00:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:09 smithi096 ceph-mon[32753]: pgmap v17860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:09 smithi096 ceph-mon[32753]: pgmap v17861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:00:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:11.945+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:11.945+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:11.945+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:00:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:11 smithi194 ceph-mon[33017]: pgmap v17862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:11 smithi033 ceph-mon[27845]: pgmap v17862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:11 smithi096 ceph-mon[32753]: pgmap v17862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:13 smithi194 ceph-mon[33017]: pgmap v17863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32952.206095 secs 2024-04-26T21:00:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:13 smithi033 ceph-mon[27845]: pgmap v17863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32952.206095 secs 2024-04-26T21:00:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:13 smithi096 ceph-mon[32753]: pgmap v17863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32952.206095 secs 2024-04-26T21:00:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32955.146766 secs 2024-04-26T21:00:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32955.146766 secs 2024-04-26T21:00:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32955.146766 secs 2024-04-26T21:00:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:15 smithi033 ceph-mon[27845]: pgmap v17864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:15 smithi096 ceph-mon[32753]: pgmap v17864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:15 smithi194 ceph-mon[33017]: pgmap v17864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:17 smithi033 ceph-mon[27845]: pgmap v17865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:17 smithi096 ceph-mon[32753]: pgmap v17865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:17 smithi194 ceph-mon[33017]: pgmap v17865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:00:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32957.206183 secs 2024-04-26T21:00:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32957.206183 secs 2024-04-26T21:00:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32957.206183 secs 2024-04-26T21:00:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32960.146881 secs 2024-04-26T21:00:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:19 smithi033 ceph-mon[27845]: pgmap v17866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:00:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32960.146881 secs 2024-04-26T21:00:20.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:19 smithi096 ceph-mon[32753]: pgmap v17866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:00:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32960.146881 secs 2024-04-26T21:00:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:19 smithi194 ceph-mon[33017]: pgmap v17866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:00:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:21 smithi033 ceph-mon[27845]: pgmap v17867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:21 smithi096 ceph-mon[32753]: pgmap v17867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:21 smithi194 ceph-mon[33017]: pgmap v17867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:23 smithi033 ceph-mon[27845]: pgmap v17868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32962.206287 secs 2024-04-26T21:00:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:23 smithi096 ceph-mon[32753]: pgmap v17868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32962.206287 secs 2024-04-26T21:00:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:23 smithi194 ceph-mon[33017]: pgmap v17868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32962.206287 secs 2024-04-26T21:00:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32965.146946 secs 2024-04-26T21:00:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32965.146946 secs 2024-04-26T21:00:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32965.146946 secs 2024-04-26T21:00:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:25 smithi033 ceph-mon[27845]: pgmap v17869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:25 smithi096 ceph-mon[32753]: pgmap v17869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:25 smithi194 ceph-mon[33017]: pgmap v17869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:26.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:26.961+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:26.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:26.961+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:00:26.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:26.961+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:27 smithi033 ceph-mon[27845]: pgmap v17870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:27 smithi096 ceph-mon[32753]: pgmap v17870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:27 smithi194 ceph-mon[33017]: pgmap v17870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32967.206371 secs 2024-04-26T21:00:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32967.206371 secs 2024-04-26T21:00:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32967.206371 secs 2024-04-26T21:00:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32970.147088 secs 2024-04-26T21:00:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:29 smithi033 ceph-mon[27845]: pgmap v17871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32970.147088 secs 2024-04-26T21:00:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:29 smithi096 ceph-mon[32753]: pgmap v17871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32970.147088 secs 2024-04-26T21:00:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:29 smithi194 ceph-mon[33017]: pgmap v17871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:31 smithi033 ceph-mon[27845]: pgmap v17872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:31 smithi096 ceph-mon[32753]: pgmap v17872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:31 smithi194 ceph-mon[33017]: pgmap v17872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:33 smithi033 ceph-mon[27845]: pgmap v17873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32972.206452 secs 2024-04-26T21:00:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:33 smithi096 ceph-mon[32753]: pgmap v17873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:34.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32972.206452 secs 2024-04-26T21:00:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:33 smithi194 ceph-mon[33017]: pgmap v17873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:00:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32972.206452 secs 2024-04-26T21:00:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32975.147221 secs 2024-04-26T21:00:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32975.147221 secs 2024-04-26T21:00:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32975.147221 secs 2024-04-26T21:00:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:35 smithi096 ceph-mon[32753]: pgmap v17874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:35 smithi194 ceph-mon[33017]: pgmap v17874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:35 smithi033 ceph-mon[27845]: pgmap v17874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:00:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:37 smithi194 ceph-mon[33017]: pgmap v17875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:37 smithi033 ceph-mon[27845]: pgmap v17875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:37 smithi096 ceph-mon[32753]: pgmap v17875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32977.206537 secs 2024-04-26T21:00:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:38 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32980.147333 secs 2024-04-26T21:00:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32977.206537 secs 2024-04-26T21:00:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:38 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32980.147333 secs 2024-04-26T21:00:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32977.206537 secs 2024-04-26T21:00:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:38 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32980.147333 secs 2024-04-26T21:00:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:39 smithi194 ceph-mon[33017]: pgmap v17876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:39 smithi033 ceph-mon[27845]: pgmap v17876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:39 smithi096 ceph-mon[32753]: pgmap v17876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:41.977+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:41.977+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:00:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:41.977+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:41 smithi194 ceph-mon[33017]: pgmap v17877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:41 smithi033 ceph-mon[27845]: pgmap v17877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:41 smithi096 ceph-mon[32753]: pgmap v17877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:44.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:43 smithi096 ceph-mon[32753]: pgmap v17878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:44.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32982.206619 secs 2024-04-26T21:00:44.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32985.147453 secs 2024-04-26T21:00:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:43 smithi194 ceph-mon[33017]: pgmap v17878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32982.206619 secs 2024-04-26T21:00:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32985.147453 secs 2024-04-26T21:00:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:43 smithi033 ceph-mon[27845]: pgmap v17878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32982.206619 secs 2024-04-26T21:00:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32985.147453 secs 2024-04-26T21:00:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:45 smithi194 ceph-mon[33017]: pgmap v17879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:45 smithi033 ceph-mon[27845]: pgmap v17879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:45 smithi096 ceph-mon[32753]: pgmap v17879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:47 smithi194 ceph-mon[33017]: pgmap v17880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:47 smithi033 ceph-mon[27845]: pgmap v17880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:47 smithi096 ceph-mon[32753]: pgmap v17880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32987.206704 secs 2024-04-26T21:00:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32990.147514 secs 2024-04-26T21:00:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32987.206704 secs 2024-04-26T21:00:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32990.147514 secs 2024-04-26T21:00:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32987.206704 secs 2024-04-26T21:00:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32990.147514 secs 2024-04-26T21:00:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:49 smithi194 ceph-mon[33017]: pgmap v17881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:49 smithi033 ceph-mon[27845]: pgmap v17881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:49 smithi096 ceph-mon[32753]: pgmap v17881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:51 smithi194 ceph-mon[33017]: pgmap v17882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:51 smithi033 ceph-mon[27845]: pgmap v17882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:51 smithi096 ceph-mon[32753]: pgmap v17882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:53 smithi194 ceph-mon[33017]: pgmap v17883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32992.206792 secs 2024-04-26T21:00:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 32995.147660 secs 2024-04-26T21:00:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:53 smithi033 ceph-mon[27845]: pgmap v17883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32992.206792 secs 2024-04-26T21:00:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 32995.147660 secs 2024-04-26T21:00:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:53 smithi096 ceph-mon[32753]: pgmap v17883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32992.206792 secs 2024-04-26T21:00:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 32995.147660 secs 2024-04-26T21:00:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:55 smithi194 ceph-mon[33017]: pgmap v17884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:55 smithi033 ceph-mon[27845]: pgmap v17884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:55 smithi096 ceph-mon[32753]: pgmap v17884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:00:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:56.987+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:56.987+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:00:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:00:56.987+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:00:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:57 smithi033 ceph-mon[27845]: pgmap v17885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:57 smithi096 ceph-mon[32753]: pgmap v17885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:57 smithi194 ceph-mon[33017]: pgmap v17885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:00:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 32997.206900 secs 2024-04-26T21:00:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 32997.206900 secs 2024-04-26T21:00:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 32997.206900 secs 2024-04-26T21:01:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33000.147753 secs 2024-04-26T21:01:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:00:59 smithi033 ceph-mon[27845]: pgmap v17886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33000.147753 secs 2024-04-26T21:01:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:00:59 smithi096 ceph-mon[32753]: pgmap v17886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33000.147753 secs 2024-04-26T21:01:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:00:59 smithi194 ceph-mon[33017]: pgmap v17886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:01 smithi194 ceph-mon[33017]: pgmap v17887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:01 smithi033 ceph-mon[27845]: pgmap v17887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:01 smithi096 ceph-mon[32753]: pgmap v17887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:03 smithi033 ceph-mon[27845]: pgmap v17888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33002.206995 secs 2024-04-26T21:01:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:03 smithi096 ceph-mon[32753]: pgmap v17888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33002.206995 secs 2024-04-26T21:01:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:03 smithi194 ceph-mon[33017]: pgmap v17888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33002.206995 secs 2024-04-26T21:01:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33005.147817 secs 2024-04-26T21:01:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33005.147817 secs 2024-04-26T21:01:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33005.147817 secs 2024-04-26T21:01:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:05 smithi033 ceph-mon[27845]: pgmap v17889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:01:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:05 smithi096 ceph-mon[32753]: pgmap v17889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:01:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:05 smithi194 ceph-mon[33017]: pgmap v17889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:01:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:07 smithi033 ceph-mon[27845]: pgmap v17890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:01:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:07 smithi096 ceph-mon[32753]: pgmap v17890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:01:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:07 smithi194 ceph-mon[33017]: pgmap v17890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:01:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33007.207057 secs 2024-04-26T21:01:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33007.207057 secs 2024-04-26T21:01:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33007.207057 secs 2024-04-26T21:01:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33010.147934 secs 2024-04-26T21:01:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:09 smithi033 ceph-mon[27845]: pgmap v17891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:01:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:01:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33010.147934 secs 2024-04-26T21:01:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:09 smithi096 ceph-mon[32753]: pgmap v17891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:01:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:01:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33010.147934 secs 2024-04-26T21:01:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:09 smithi194 ceph-mon[33017]: pgmap v17891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:01:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:01:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:12.003 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:12.001+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:12.004 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:12.002+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:12.004 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:12.002+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:01:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:12 smithi194 ceph-mon[33017]: pgmap v17892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:01:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:01:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:01:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:12 smithi033 ceph-mon[27845]: pgmap v17892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:01:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:01:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:01:12.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:12 smithi096 ceph-mon[32753]: pgmap v17892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:01:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:01:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:01:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:01:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:13 smithi194 ceph-mon[33017]: pgmap v17893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T21:01:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:13 smithi194 ceph-mon[33017]: pgmap v17894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:01:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33012.207135 secs 2024-04-26T21:01:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:13 smithi033 ceph-mon[27845]: pgmap v17893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T21:01:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:13 smithi033 ceph-mon[27845]: pgmap v17894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:01:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33012.207135 secs 2024-04-26T21:01:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:13 smithi096 ceph-mon[32753]: pgmap v17893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T21:01:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:13 smithi096 ceph-mon[32753]: pgmap v17894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:01:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33012.207135 secs 2024-04-26T21:01:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33015.148058 secs 2024-04-26T21:01:14.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33015.148058 secs 2024-04-26T21:01:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33015.148058 secs 2024-04-26T21:01:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:15 smithi194 ceph-mon[33017]: pgmap v17895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:01:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:15 smithi033 ceph-mon[27845]: pgmap v17895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:01:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:15 smithi096 ceph-mon[32753]: pgmap v17895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:01:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:17 smithi194 ceph-mon[33017]: pgmap v17896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:17 smithi033 ceph-mon[27845]: pgmap v17896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:17 smithi096 ceph-mon[32753]: pgmap v17896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33017.207219 secs 2024-04-26T21:01:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33017.207219 secs 2024-04-26T21:01:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33017.207219 secs 2024-04-26T21:01:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:19 smithi194 ceph-mon[33017]: pgmap v17897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33020.148182 secs 2024-04-26T21:01:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:19 smithi033 ceph-mon[27845]: pgmap v17897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33020.148182 secs 2024-04-26T21:01:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:19 smithi096 ceph-mon[32753]: pgmap v17897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33020.148182 secs 2024-04-26T21:01:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:21 smithi194 ceph-mon[33017]: pgmap v17898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:21 smithi033 ceph-mon[27845]: pgmap v17898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:21 smithi096 ceph-mon[32753]: pgmap v17898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:01:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:23 smithi194 ceph-mon[33017]: pgmap v17899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:01:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33022.207304 secs 2024-04-26T21:01:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:23 smithi033 ceph-mon[27845]: pgmap v17899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:01:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33022.207304 secs 2024-04-26T21:01:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:23 smithi096 ceph-mon[32753]: pgmap v17899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:01:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33022.207304 secs 2024-04-26T21:01:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33025.148276 secs 2024-04-26T21:01:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33025.148276 secs 2024-04-26T21:01:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33025.148276 secs 2024-04-26T21:01:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:25 smithi194 ceph-mon[33017]: pgmap v17900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:25 smithi033 ceph-mon[27845]: pgmap v17900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:25 smithi096 ceph-mon[32753]: pgmap v17900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:26 smithi194 ceph-mon[33017]: pgmap v17901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:26 smithi033 ceph-mon[27845]: pgmap v17901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:26 smithi096 ceph-mon[32753]: pgmap v17901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:27.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:27.015+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:27.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:27.015+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:27.017 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:27.015+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:01:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:28 smithi194 ceph-mon[33017]: pgmap v17902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33027.207389 secs 2024-04-26T21:01:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:28 smithi033 ceph-mon[27845]: pgmap v17902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33027.207389 secs 2024-04-26T21:01:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:28 smithi096 ceph-mon[32753]: pgmap v17902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33027.207389 secs 2024-04-26T21:01:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33030.148371 secs 2024-04-26T21:01:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33030.148371 secs 2024-04-26T21:01:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33030.148371 secs 2024-04-26T21:01:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:30 smithi194 ceph-mon[33017]: pgmap v17903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:30 smithi033 ceph-mon[27845]: pgmap v17903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:30.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:30 smithi096 ceph-mon[32753]: pgmap v17903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:32 smithi194 ceph-mon[33017]: pgmap v17904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:32 smithi033 ceph-mon[27845]: pgmap v17904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:32 smithi096 ceph-mon[32753]: pgmap v17904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33032.207471 secs 2024-04-26T21:01:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33032.207471 secs 2024-04-26T21:01:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33032.207471 secs 2024-04-26T21:01:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:34 smithi194 ceph-mon[33017]: pgmap v17905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33035.148450 secs 2024-04-26T21:01:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:34 smithi033 ceph-mon[27845]: pgmap v17905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33035.148450 secs 2024-04-26T21:01:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:34 smithi096 ceph-mon[32753]: pgmap v17905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:34.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33035.148450 secs 2024-04-26T21:01:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:36 smithi194 ceph-mon[33017]: pgmap v17906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:36 smithi033 ceph-mon[27845]: pgmap v17906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:36 smithi096 ceph-mon[32753]: pgmap v17906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:38 smithi194 ceph-mon[33017]: pgmap v17907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33037.207555 secs 2024-04-26T21:01:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:38 smithi033 ceph-mon[27845]: pgmap v17907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33037.207555 secs 2024-04-26T21:01:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:38 smithi096 ceph-mon[32753]: pgmap v17907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33037.207555 secs 2024-04-26T21:01:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33040.148571 secs 2024-04-26T21:01:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33040.148571 secs 2024-04-26T21:01:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33040.148571 secs 2024-04-26T21:01:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:40 smithi194 ceph-mon[33017]: pgmap v17908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:40 smithi033 ceph-mon[27845]: pgmap v17908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:40 smithi096 ceph-mon[32753]: pgmap v17908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:42.027+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:42.027+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:42.027+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:01:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:42 smithi194 ceph-mon[33017]: pgmap v17909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:42 smithi033 ceph-mon[27845]: pgmap v17909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:42 smithi096 ceph-mon[32753]: pgmap v17909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33042.207639 secs 2024-04-26T21:01:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33042.207639 secs 2024-04-26T21:01:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33042.207639 secs 2024-04-26T21:01:44.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:44 smithi033 ceph-mon[27845]: pgmap v17910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:44.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33045.148725 secs 2024-04-26T21:01:44.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:44 smithi096 ceph-mon[32753]: pgmap v17910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:44.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33045.148725 secs 2024-04-26T21:01:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:44 smithi194 ceph-mon[33017]: pgmap v17910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33045.148725 secs 2024-04-26T21:01:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:46 smithi194 ceph-mon[33017]: pgmap v17911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:46 smithi033 ceph-mon[27845]: pgmap v17911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:46 smithi096 ceph-mon[32753]: pgmap v17911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:48 smithi194 ceph-mon[33017]: pgmap v17912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33047.207724 secs 2024-04-26T21:01:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:48 smithi033 ceph-mon[27845]: pgmap v17912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33047.207724 secs 2024-04-26T21:01:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:48 smithi096 ceph-mon[32753]: pgmap v17912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33047.207724 secs 2024-04-26T21:01:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33050.148869 secs 2024-04-26T21:01:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33050.148869 secs 2024-04-26T21:01:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33050.148869 secs 2024-04-26T21:01:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:50 smithi194 ceph-mon[33017]: pgmap v17913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:50 smithi033 ceph-mon[27845]: pgmap v17913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:50 smithi096 ceph-mon[32753]: pgmap v17913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:52 smithi194 ceph-mon[33017]: pgmap v17914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:52 smithi033 ceph-mon[27845]: pgmap v17914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:52 smithi096 ceph-mon[32753]: pgmap v17914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33052.207820 secs 2024-04-26T21:01:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33052.207820 secs 2024-04-26T21:01:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33052.207820 secs 2024-04-26T21:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:54 smithi194 ceph-mon[33017]: pgmap v17915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33055.149007 secs 2024-04-26T21:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:54 smithi033 ceph-mon[27845]: pgmap v17915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33055.149007 secs 2024-04-26T21:01:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:54 smithi096 ceph-mon[32753]: pgmap v17915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33055.149007 secs 2024-04-26T21:01:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:56 smithi033 ceph-mon[27845]: pgmap v17916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:56 smithi096 ceph-mon[32753]: pgmap v17916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:57.036+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:57.036+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:01:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:01:57.036+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:01:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:56 smithi194 ceph-mon[33017]: pgmap v17916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:01:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:58 smithi033 ceph-mon[27845]: pgmap v17917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33057.207897 secs 2024-04-26T21:01:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:58 smithi096 ceph-mon[32753]: pgmap v17917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33057.207897 secs 2024-04-26T21:01:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:58 smithi194 ceph-mon[33017]: pgmap v17917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:01:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33057.207897 secs 2024-04-26T21:01:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33060.149149 secs 2024-04-26T21:01:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33060.149149 secs 2024-04-26T21:02:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33060.149149 secs 2024-04-26T21:02:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:00 smithi033 ceph-mon[27845]: pgmap v17918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:00 smithi096 ceph-mon[32753]: pgmap v17918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:00 smithi194 ceph-mon[33017]: pgmap v17918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:02 smithi033 ceph-mon[27845]: pgmap v17919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:02 smithi096 ceph-mon[32753]: pgmap v17919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:02 smithi194 ceph-mon[33017]: pgmap v17919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33062.207993 secs 2024-04-26T21:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33062.207993 secs 2024-04-26T21:02:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33062.207993 secs 2024-04-26T21:02:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:04 smithi033 ceph-mon[27845]: pgmap v17920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33065.149271 secs 2024-04-26T21:02:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:04 smithi096 ceph-mon[32753]: pgmap v17920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33065.149271 secs 2024-04-26T21:02:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:04 smithi194 ceph-mon[33017]: pgmap v17920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33065.149271 secs 2024-04-26T21:02:06.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:06 smithi033 ceph-mon[27845]: pgmap v17921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:06 smithi096 ceph-mon[32753]: pgmap v17921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:06 smithi194 ceph-mon[33017]: pgmap v17921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:08 smithi033 ceph-mon[27845]: pgmap v17922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33067.208079 secs 2024-04-26T21:02:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:08 smithi096 ceph-mon[32753]: pgmap v17922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33067.208079 secs 2024-04-26T21:02:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:08 smithi194 ceph-mon[33017]: pgmap v17922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33067.208079 secs 2024-04-26T21:02:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33070.149390 secs 2024-04-26T21:02:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33070.149390 secs 2024-04-26T21:02:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33070.149390 secs 2024-04-26T21:02:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:10 smithi033 ceph-mon[27845]: pgmap v17923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:10 smithi096 ceph-mon[32753]: pgmap v17923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:11.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:10 smithi194 ceph-mon[33017]: pgmap v17923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:12.048+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:12.049+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:12.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:12.049+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:02:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:12 smithi033 ceph-mon[27845]: pgmap v17924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:02:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:12 smithi096 ceph-mon[32753]: pgmap v17924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:02:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:12 smithi194 ceph-mon[33017]: pgmap v17924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:02:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33072.208187 secs 2024-04-26T21:02:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:02:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:02:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:02:14.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33072.208187 secs 2024-04-26T21:02:14.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:02:14.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:02:14.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:02:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33072.208187 secs 2024-04-26T21:02:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:02:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:02:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:02:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:14 smithi096 ceph-mon[32753]: pgmap v17925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:14 smithi096 ceph-mon[32753]: pgmap v17926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:14.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33075.149535 secs 2024-04-26T21:02:15.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:14 smithi033 ceph-mon[27845]: pgmap v17925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:15.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:14 smithi033 ceph-mon[27845]: pgmap v17926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:15.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33075.149535 secs 2024-04-26T21:02:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:14 smithi194 ceph-mon[33017]: pgmap v17925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:14 smithi194 ceph-mon[33017]: pgmap v17926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33075.149535 secs 2024-04-26T21:02:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:16 smithi194 ceph-mon[33017]: pgmap v17927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:16 smithi033 ceph-mon[27845]: pgmap v17927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:16 smithi096 ceph-mon[32753]: pgmap v17927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:18 smithi194 ceph-mon[33017]: pgmap v17928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33077.208272 secs 2024-04-26T21:02:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:18 smithi033 ceph-mon[27845]: pgmap v17928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33077.208272 secs 2024-04-26T21:02:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:18 smithi096 ceph-mon[32753]: pgmap v17928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33077.208272 secs 2024-04-26T21:02:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33080.149655 secs 2024-04-26T21:02:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33080.149655 secs 2024-04-26T21:02:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33080.149655 secs 2024-04-26T21:02:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:20 smithi194 ceph-mon[33017]: pgmap v17929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:02:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:20 smithi033 ceph-mon[27845]: pgmap v17929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:02:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:20 smithi096 ceph-mon[32753]: pgmap v17929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:02:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:22 smithi194 ceph-mon[33017]: pgmap v17930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:22 smithi033 ceph-mon[27845]: pgmap v17930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:22 smithi096 ceph-mon[32753]: pgmap v17930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:02:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33082.208332 secs 2024-04-26T21:02:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33082.208332 secs 2024-04-26T21:02:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33082.208332 secs 2024-04-26T21:02:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:24 smithi194 ceph-mon[33017]: pgmap v17931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:02:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33085.149724 secs 2024-04-26T21:02:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:24 smithi033 ceph-mon[27845]: pgmap v17931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:02:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33085.149724 secs 2024-04-26T21:02:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:24 smithi096 ceph-mon[32753]: pgmap v17931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:02:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33085.149724 secs 2024-04-26T21:02:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.064+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.064+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.064+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:02:27.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:26 smithi194 ceph-mon[33017]: pgmap v17932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:26 smithi033 ceph-mon[27845]: pgmap v17932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:26 smithi096 ceph-mon[32753]: pgmap v17932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.424+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:02:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.424+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T21:02:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.425+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 259 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T21:02:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.425+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x564385821900 2024-04-26T21:02:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.425+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 259 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:02:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.425+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:02:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.425+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 260 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T21:02:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:27.425+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 260 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0004740 con 0x7fef00077780 2024-04-26T21:02:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:28 smithi194 ceph-mon[33017]: pgmap v17933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33087.208419 secs 2024-04-26T21:02:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:28 smithi033 ceph-mon[27845]: pgmap v17933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33087.208419 secs 2024-04-26T21:02:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:28 smithi096 ceph-mon[32753]: pgmap v17933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33087.208419 secs 2024-04-26T21:02:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33090.149873 secs 2024-04-26T21:02:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33090.149873 secs 2024-04-26T21:02:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33090.149873 secs 2024-04-26T21:02:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:30 smithi194 ceph-mon[33017]: pgmap v17934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:30 smithi033 ceph-mon[27845]: pgmap v17934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:30 smithi096 ceph-mon[32753]: pgmap v17934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:32 smithi194 ceph-mon[33017]: pgmap v17935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:32 smithi033 ceph-mon[27845]: pgmap v17935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:32 smithi096 ceph-mon[32753]: pgmap v17935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33092.208516 secs 2024-04-26T21:02:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33092.208516 secs 2024-04-26T21:02:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33092.208516 secs 2024-04-26T21:02:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:34 smithi194 ceph-mon[33017]: pgmap v17936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33095.150013 secs 2024-04-26T21:02:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:34 smithi033 ceph-mon[27845]: pgmap v17936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33095.150013 secs 2024-04-26T21:02:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:34 smithi096 ceph-mon[32753]: pgmap v17936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33095.150013 secs 2024-04-26T21:02:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:36 smithi194 ceph-mon[33017]: pgmap v17937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:36 smithi033 ceph-mon[27845]: pgmap v17937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:36 smithi096 ceph-mon[32753]: pgmap v17937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:38 smithi194 ceph-mon[33017]: pgmap v17938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33097.208612 secs 2024-04-26T21:02:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:38 smithi033 ceph-mon[27845]: pgmap v17938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33097.208612 secs 2024-04-26T21:02:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:38 smithi096 ceph-mon[32753]: pgmap v17938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33097.208612 secs 2024-04-26T21:02:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33100.150132 secs 2024-04-26T21:02:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33100.150132 secs 2024-04-26T21:02:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33100.150132 secs 2024-04-26T21:02:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:40 smithi194 ceph-mon[33017]: pgmap v17939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:40 smithi033 ceph-mon[27845]: pgmap v17939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:40 smithi096 ceph-mon[32753]: pgmap v17939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:42.078+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:42.079+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:42.080 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:42.079+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:02:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:42 smithi194 ceph-mon[33017]: pgmap v17940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:42 smithi033 ceph-mon[27845]: pgmap v17940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:42 smithi096 ceph-mon[32753]: pgmap v17940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:44.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33102.208711 secs 2024-04-26T21:02:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33102.208711 secs 2024-04-26T21:02:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33102.208711 secs 2024-04-26T21:02:45.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:44 smithi033 ceph-mon[27845]: pgmap v17941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:45.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33105.150261 secs 2024-04-26T21:02:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:44 smithi194 ceph-mon[33017]: pgmap v17941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33105.150261 secs 2024-04-26T21:02:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:44 smithi096 ceph-mon[32753]: pgmap v17941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:45.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33105.150261 secs 2024-04-26T21:02:47.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:46 smithi033 ceph-mon[27845]: pgmap v17942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:46 smithi194 ceph-mon[33017]: pgmap v17942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:46 smithi096 ceph-mon[32753]: pgmap v17942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:48 smithi194 ceph-mon[33017]: pgmap v17943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33107.208813 secs 2024-04-26T21:02:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:48 smithi033 ceph-mon[27845]: pgmap v17943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33107.208813 secs 2024-04-26T21:02:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:48 smithi096 ceph-mon[32753]: pgmap v17943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33107.208813 secs 2024-04-26T21:02:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33110.150321 secs 2024-04-26T21:02:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33110.150321 secs 2024-04-26T21:02:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33110.150321 secs 2024-04-26T21:02:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:50 smithi194 ceph-mon[33017]: pgmap v17944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:50 smithi033 ceph-mon[27845]: pgmap v17944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:50 smithi096 ceph-mon[32753]: pgmap v17944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:52 smithi194 ceph-mon[33017]: pgmap v17945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:52 smithi033 ceph-mon[27845]: pgmap v17945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:52 smithi096 ceph-mon[32753]: pgmap v17945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:54.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33112.208911 secs 2024-04-26T21:02:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33112.208911 secs 2024-04-26T21:02:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33112.208911 secs 2024-04-26T21:02:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:54 smithi194 ceph-mon[33017]: pgmap v17946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33115.150449 secs 2024-04-26T21:02:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:54 smithi033 ceph-mon[27845]: pgmap v17946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33115.150449 secs 2024-04-26T21:02:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:54 smithi096 ceph-mon[32753]: pgmap v17946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33115.150449 secs 2024-04-26T21:02:57.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:57.089+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:57.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:57.089+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:02:57.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:02:57.089+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:02:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:56 smithi194 ceph-mon[33017]: pgmap v17947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:57.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:56 smithi033 ceph-mon[27845]: pgmap v17947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:56 smithi096 ceph-mon[32753]: pgmap v17947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:02:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:58 smithi033 ceph-mon[27845]: pgmap v17948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33117.209016 secs 2024-04-26T21:02:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:58 smithi096 ceph-mon[32753]: pgmap v17948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33117.209016 secs 2024-04-26T21:02:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:58 smithi194 ceph-mon[33017]: pgmap v17948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:02:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33117.209016 secs 2024-04-26T21:03:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33120.150566 secs 2024-04-26T21:03:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33120.150566 secs 2024-04-26T21:03:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33120.150566 secs 2024-04-26T21:03:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:00 smithi033 ceph-mon[27845]: pgmap v17949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:00 smithi096 ceph-mon[32753]: pgmap v17949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:00 smithi194 ceph-mon[33017]: pgmap v17949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:02 smithi033 ceph-mon[27845]: pgmap v17950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:02 smithi096 ceph-mon[32753]: pgmap v17950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:02 smithi194 ceph-mon[33017]: pgmap v17950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33122.209075 secs 2024-04-26T21:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33122.209075 secs 2024-04-26T21:03:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33122.209075 secs 2024-04-26T21:03:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:04 smithi033 ceph-mon[27845]: pgmap v17951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33125.150711 secs 2024-04-26T21:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:04 smithi096 ceph-mon[32753]: pgmap v17951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33125.150711 secs 2024-04-26T21:03:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:04 smithi194 ceph-mon[33017]: pgmap v17951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33125.150711 secs 2024-04-26T21:03:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:06 smithi033 ceph-mon[27845]: pgmap v17952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:06 smithi096 ceph-mon[32753]: pgmap v17952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:06 smithi194 ceph-mon[33017]: pgmap v17952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:08 smithi033 ceph-mon[27845]: pgmap v17953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33127.209158 secs 2024-04-26T21:03:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:08 smithi096 ceph-mon[32753]: pgmap v17953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33127.209158 secs 2024-04-26T21:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:08 smithi194 ceph-mon[33017]: pgmap v17953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33127.209158 secs 2024-04-26T21:03:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33130.150839 secs 2024-04-26T21:03:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33130.150839 secs 2024-04-26T21:03:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33130.150839 secs 2024-04-26T21:03:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:10 smithi033 ceph-mon[27845]: pgmap v17954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:10 smithi096 ceph-mon[32753]: pgmap v17954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:10 smithi194 ceph-mon[33017]: pgmap v17954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:12.093+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:12.093+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:12.093+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:03:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:12 smithi033 ceph-mon[27845]: pgmap v17955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:12 smithi096 ceph-mon[32753]: pgmap v17955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:12 smithi194 ceph-mon[33017]: pgmap v17955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:14.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33132.209258 secs 2024-04-26T21:03:14.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:03:14.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33132.209258 secs 2024-04-26T21:03:14.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:03:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33132.209258 secs 2024-04-26T21:03:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:03:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:14 smithi033 ceph-mon[27845]: pgmap v17956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33135.150931 secs 2024-04-26T21:03:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:14 smithi096 ceph-mon[32753]: pgmap v17956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33135.150931 secs 2024-04-26T21:03:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:14 smithi194 ceph-mon[33017]: pgmap v17956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33135.150931 secs 2024-04-26T21:03:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: pgmap v17957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:03:17.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:17 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:03:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: pgmap v17957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:03:17.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:03:17.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:17 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: pgmap v17957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:17.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:03:17.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:17 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:03:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33137.209362 secs 2024-04-26T21:03:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33137.209362 secs 2024-04-26T21:03:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33137.209362 secs 2024-04-26T21:03:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:19 smithi033 ceph-mon[27845]: pgmap v17958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33140.151073 secs 2024-04-26T21:03:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:19 smithi096 ceph-mon[32753]: pgmap v17958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33140.151073 secs 2024-04-26T21:03:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:19 smithi194 ceph-mon[33017]: pgmap v17958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33140.151073 secs 2024-04-26T21:03:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: pgmap v17959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:03:20.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: pgmap v17960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T21:03:20.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: pgmap v17961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:20.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: pgmap v17959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: pgmap v17960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: pgmap v17961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:20.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: pgmap v17959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:03:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:03:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:03:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:03:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:03:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: pgmap v17960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T21:03:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: pgmap v17961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:20.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:03:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:22 smithi033 ceph-mon[27845]: pgmap v17962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:22 smithi096 ceph-mon[32753]: pgmap v17962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:22 smithi194 ceph-mon[33017]: pgmap v17962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33142.209460 secs 2024-04-26T21:03:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33142.209460 secs 2024-04-26T21:03:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33142.209460 secs 2024-04-26T21:03:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:24 smithi033 ceph-mon[27845]: pgmap v17963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:03:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33145.151213 secs 2024-04-26T21:03:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:24 smithi096 ceph-mon[32753]: pgmap v17963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:03:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33145.151213 secs 2024-04-26T21:03:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:24 smithi194 ceph-mon[33017]: pgmap v17963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:03:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33145.151213 secs 2024-04-26T21:03:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:26 smithi033 ceph-mon[27845]: pgmap v17964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:26 smithi096 ceph-mon[32753]: pgmap v17964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:27.098+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:27.098+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:03:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:27.098+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:26 smithi194 ceph-mon[33017]: pgmap v17964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:28 smithi033 ceph-mon[27845]: pgmap v17965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33147.209563 secs 2024-04-26T21:03:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:28 smithi096 ceph-mon[32753]: pgmap v17965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33147.209563 secs 2024-04-26T21:03:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:28 smithi194 ceph-mon[33017]: pgmap v17965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:03:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33147.209563 secs 2024-04-26T21:03:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33150.151338 secs 2024-04-26T21:03:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33150.151338 secs 2024-04-26T21:03:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33150.151338 secs 2024-04-26T21:03:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:30 smithi033 ceph-mon[27845]: pgmap v17966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:03:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:30 smithi096 ceph-mon[32753]: pgmap v17966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:03:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:30 smithi194 ceph-mon[33017]: pgmap v17966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:03:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:32 smithi033 ceph-mon[27845]: pgmap v17967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:32 smithi096 ceph-mon[32753]: pgmap v17967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:32 smithi194 ceph-mon[33017]: pgmap v17967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33152.209660 secs 2024-04-26T21:03:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33152.209660 secs 2024-04-26T21:03:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33152.209660 secs 2024-04-26T21:03:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:34 smithi033 ceph-mon[27845]: pgmap v17968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33155.151453 secs 2024-04-26T21:03:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:34 smithi096 ceph-mon[32753]: pgmap v17968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33155.151453 secs 2024-04-26T21:03:35.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:34 smithi194 ceph-mon[33017]: pgmap v17968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:35.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33155.151453 secs 2024-04-26T21:03:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:36 smithi033 ceph-mon[27845]: pgmap v17969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:36 smithi096 ceph-mon[32753]: pgmap v17969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:36 smithi194 ceph-mon[33017]: pgmap v17969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:38 smithi033 ceph-mon[27845]: pgmap v17970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33157.209765 secs 2024-04-26T21:03:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:38 smithi096 ceph-mon[32753]: pgmap v17970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33157.209765 secs 2024-04-26T21:03:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:38 smithi194 ceph-mon[33017]: pgmap v17970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33157.209765 secs 2024-04-26T21:03:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33160.151583 secs 2024-04-26T21:03:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33160.151583 secs 2024-04-26T21:03:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33160.151583 secs 2024-04-26T21:03:41.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:40 smithi194 ceph-mon[33017]: pgmap v17971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:40 smithi033 ceph-mon[27845]: pgmap v17971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:40 smithi096 ceph-mon[32753]: pgmap v17971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:42.114+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:42.114+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:03:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:42.114+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:42 smithi194 ceph-mon[33017]: pgmap v17972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:42 smithi033 ceph-mon[27845]: pgmap v17972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:42 smithi096 ceph-mon[32753]: pgmap v17972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:44.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33162.209862 secs 2024-04-26T21:03:44.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33162.209862 secs 2024-04-26T21:03:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33162.209862 secs 2024-04-26T21:03:45.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:44 smithi033 ceph-mon[27845]: pgmap v17973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:45.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33165.151661 secs 2024-04-26T21:03:45.099 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:44 smithi096 ceph-mon[32753]: pgmap v17973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:45.100 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33165.151661 secs 2024-04-26T21:03:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:44 smithi194 ceph-mon[33017]: pgmap v17973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33165.151661 secs 2024-04-26T21:03:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:46 smithi194 ceph-mon[33017]: pgmap v17974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:46 smithi033 ceph-mon[27845]: pgmap v17974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:47.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:46 smithi096 ceph-mon[32753]: pgmap v17974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:48 smithi194 ceph-mon[33017]: pgmap v17975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33167.209973 secs 2024-04-26T21:03:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:48 smithi033 ceph-mon[27845]: pgmap v17975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33167.209973 secs 2024-04-26T21:03:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:48 smithi096 ceph-mon[32753]: pgmap v17975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33167.209973 secs 2024-04-26T21:03:50.065 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33170.151754 secs 2024-04-26T21:03:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33170.151754 secs 2024-04-26T21:03:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33170.151754 secs 2024-04-26T21:03:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:50 smithi194 ceph-mon[33017]: pgmap v17976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:50 smithi033 ceph-mon[27845]: pgmap v17976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:50 smithi096 ceph-mon[32753]: pgmap v17976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:52 smithi194 ceph-mon[33017]: pgmap v17977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:52 smithi033 ceph-mon[27845]: pgmap v17977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:52 smithi096 ceph-mon[32753]: pgmap v17977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33172.210062 secs 2024-04-26T21:03:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33172.210062 secs 2024-04-26T21:03:54.202 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33172.210062 secs 2024-04-26T21:03:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:54 smithi194 ceph-mon[33017]: pgmap v17978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33175.151898 secs 2024-04-26T21:03:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:54 smithi033 ceph-mon[27845]: pgmap v17978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33175.151898 secs 2024-04-26T21:03:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:54 smithi096 ceph-mon[32753]: pgmap v17978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33175.151898 secs 2024-04-26T21:03:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:56 smithi194 ceph-mon[33017]: pgmap v17979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:57.125+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:57.125+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:03:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:03:57.125+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:03:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:56 smithi033 ceph-mon[27845]: pgmap v17979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:56 smithi096 ceph-mon[32753]: pgmap v17979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:03:59.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:58 smithi194 ceph-mon[33017]: pgmap v17980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:59.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33177.210148 secs 2024-04-26T21:03:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:58 smithi033 ceph-mon[27845]: pgmap v17980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33177.210148 secs 2024-04-26T21:03:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:58 smithi096 ceph-mon[32753]: pgmap v17980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:03:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33177.210148 secs 2024-04-26T21:04:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33180.152015 secs 2024-04-26T21:04:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33180.152015 secs 2024-04-26T21:04:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33180.152015 secs 2024-04-26T21:04:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:00 smithi194 ceph-mon[33017]: pgmap v17981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:00 smithi033 ceph-mon[27845]: pgmap v17981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:00 smithi096 ceph-mon[32753]: pgmap v17981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:02 smithi194 ceph-mon[33017]: pgmap v17982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:02 smithi033 ceph-mon[27845]: pgmap v17982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:02 smithi096 ceph-mon[32753]: pgmap v17982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33182.210234 secs 2024-04-26T21:04:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33182.210234 secs 2024-04-26T21:04:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33182.210234 secs 2024-04-26T21:04:05.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:04 smithi033 ceph-mon[27845]: pgmap v17983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:05.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33185.152144 secs 2024-04-26T21:04:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:04 smithi194 ceph-mon[33017]: pgmap v17983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33185.152144 secs 2024-04-26T21:04:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:04 smithi096 ceph-mon[32753]: pgmap v17983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33185.152144 secs 2024-04-26T21:04:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:06 smithi194 ceph-mon[33017]: pgmap v17984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:06 smithi033 ceph-mon[27845]: pgmap v17984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:06 smithi096 ceph-mon[32753]: pgmap v17984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:08 smithi194 ceph-mon[33017]: pgmap v17985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33187.210324 secs 2024-04-26T21:04:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:08 smithi033 ceph-mon[27845]: pgmap v17985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33187.210324 secs 2024-04-26T21:04:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:08 smithi096 ceph-mon[32753]: pgmap v17985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33187.210324 secs 2024-04-26T21:04:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33190.152225 secs 2024-04-26T21:04:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33190.152225 secs 2024-04-26T21:04:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33190.152225 secs 2024-04-26T21:04:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:10 smithi194 ceph-mon[33017]: pgmap v17986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:10 smithi033 ceph-mon[27845]: pgmap v17986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:10 smithi096 ceph-mon[32753]: pgmap v17986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:12.140+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:12.140+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:12.140+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:04:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:12 smithi194 ceph-mon[33017]: pgmap v17987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:12 smithi033 ceph-mon[27845]: pgmap v17987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:12 smithi096 ceph-mon[32753]: pgmap v17987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:14.084 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33192.210411 secs 2024-04-26T21:04:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33192.210411 secs 2024-04-26T21:04:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33192.210411 secs 2024-04-26T21:04:15.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:14 smithi033 ceph-mon[27845]: pgmap v17988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:15.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33195.152301 secs 2024-04-26T21:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:14 smithi194 ceph-mon[33017]: pgmap v17988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33195.152301 secs 2024-04-26T21:04:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:14 smithi096 ceph-mon[32753]: pgmap v17988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33195.152301 secs 2024-04-26T21:04:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:16 smithi194 ceph-mon[33017]: pgmap v17989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:16 smithi033 ceph-mon[27845]: pgmap v17989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:16 smithi096 ceph-mon[32753]: pgmap v17989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:18 smithi194 ceph-mon[33017]: pgmap v17990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33197.210489 secs 2024-04-26T21:04:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:18 smithi033 ceph-mon[27845]: pgmap v17990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33197.210489 secs 2024-04-26T21:04:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:18 smithi096 ceph-mon[32753]: pgmap v17990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33197.210489 secs 2024-04-26T21:04:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33200.152414 secs 2024-04-26T21:04:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33200.152414 secs 2024-04-26T21:04:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33200.152414 secs 2024-04-26T21:04:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:20 smithi194 ceph-mon[33017]: pgmap v17991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:21.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:04:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:20 smithi033 ceph-mon[27845]: pgmap v17991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:04:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:20 smithi096 ceph-mon[32753]: pgmap v17991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:04:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:22 smithi033 ceph-mon[27845]: pgmap v17992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:22 smithi096 ceph-mon[32753]: pgmap v17992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:22 smithi194 ceph-mon[33017]: pgmap v17992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:04:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:04:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: pgmap v17993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:04:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: pgmap v17994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33202.210576 secs 2024-04-26T21:04:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: pgmap v17993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: pgmap v17994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33202.210576 secs 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: pgmap v17993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: pgmap v17994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:04:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33202.210576 secs 2024-04-26T21:04:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33205.152549 secs 2024-04-26T21:04:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33205.152549 secs 2024-04-26T21:04:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33205.152549 secs 2024-04-26T21:04:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:25 smithi033 ceph-mon[27845]: pgmap v17995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:04:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:25 smithi096 ceph-mon[32753]: pgmap v17995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:04:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:25 smithi194 ceph-mon[33017]: pgmap v17995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:04:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:27.154+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:27.154+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:27.154+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:04:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:27 smithi194 ceph-mon[33017]: pgmap v17996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:27 smithi033 ceph-mon[27845]: pgmap v17996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:27 smithi096 ceph-mon[32753]: pgmap v17996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33207.210635 secs 2024-04-26T21:04:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33207.210635 secs 2024-04-26T21:04:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33207.210635 secs 2024-04-26T21:04:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:29 smithi194 ceph-mon[33017]: pgmap v17997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:04:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33210.152670 secs 2024-04-26T21:04:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:29 smithi033 ceph-mon[27845]: pgmap v17997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:04:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33210.152670 secs 2024-04-26T21:04:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:29 smithi096 ceph-mon[32753]: pgmap v17997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:04:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33210.152670 secs 2024-04-26T21:04:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:31 smithi194 ceph-mon[33017]: pgmap v17998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:31 smithi033 ceph-mon[27845]: pgmap v17998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:31.685 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:31 smithi096 ceph-mon[32753]: pgmap v17998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:04:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:33 smithi194 ceph-mon[33017]: pgmap v17999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:04:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33212.210721 secs 2024-04-26T21:04:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:33 smithi033 ceph-mon[27845]: pgmap v17999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:04:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33212.210721 secs 2024-04-26T21:04:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:33 smithi096 ceph-mon[32753]: pgmap v17999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:04:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33212.210721 secs 2024-04-26T21:04:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33215.152754 secs 2024-04-26T21:04:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33215.152754 secs 2024-04-26T21:04:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33215.152754 secs 2024-04-26T21:04:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:35 smithi194 ceph-mon[33017]: pgmap v18000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:35 smithi033 ceph-mon[27845]: pgmap v18000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:35 smithi096 ceph-mon[32753]: pgmap v18000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:37 smithi194 ceph-mon[33017]: pgmap v18001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:37 smithi033 ceph-mon[27845]: pgmap v18001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:37 smithi096 ceph-mon[32753]: pgmap v18001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33217.210778 secs 2024-04-26T21:04:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33217.210778 secs 2024-04-26T21:04:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33217.210778 secs 2024-04-26T21:04:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:39 smithi194 ceph-mon[33017]: pgmap v18002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33220.152850 secs 2024-04-26T21:04:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:39 smithi033 ceph-mon[27845]: pgmap v18002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33220.152850 secs 2024-04-26T21:04:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:39 smithi096 ceph-mon[32753]: pgmap v18002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33220.152850 secs 2024-04-26T21:04:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:41 smithi194 ceph-mon[33017]: pgmap v18003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:41 smithi033 ceph-mon[27845]: pgmap v18003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:41 smithi096 ceph-mon[32753]: pgmap v18003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:42.160+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:42.160+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:42.160+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:04:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:43 smithi194 ceph-mon[33017]: pgmap v18004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33222.210862 secs 2024-04-26T21:04:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:43 smithi033 ceph-mon[27845]: pgmap v18004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33222.210862 secs 2024-04-26T21:04:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:43 smithi096 ceph-mon[32753]: pgmap v18004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33222.210862 secs 2024-04-26T21:04:44.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33225.152972 secs 2024-04-26T21:04:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33225.152972 secs 2024-04-26T21:04:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33225.152972 secs 2024-04-26T21:04:45.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:45 smithi033 ceph-mon[27845]: pgmap v18005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:45 smithi194 ceph-mon[33017]: pgmap v18005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:45 smithi096 ceph-mon[32753]: pgmap v18005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:47 smithi194 ceph-mon[33017]: pgmap v18006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:47 smithi033 ceph-mon[27845]: pgmap v18006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:47 smithi096 ceph-mon[32753]: pgmap v18006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33227.210921 secs 2024-04-26T21:04:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33227.210921 secs 2024-04-26T21:04:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33227.210921 secs 2024-04-26T21:04:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:49 smithi194 ceph-mon[33017]: pgmap v18007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33230.153095 secs 2024-04-26T21:04:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:49 smithi033 ceph-mon[27845]: pgmap v18007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33230.153095 secs 2024-04-26T21:04:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:49 smithi096 ceph-mon[32753]: pgmap v18007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33230.153095 secs 2024-04-26T21:04:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:51 smithi033 ceph-mon[27845]: pgmap v18008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:51 smithi096 ceph-mon[32753]: pgmap v18008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:51 smithi194 ceph-mon[33017]: pgmap v18008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:53 smithi033 ceph-mon[27845]: pgmap v18009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33232.211019 secs 2024-04-26T21:04:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:53 smithi096 ceph-mon[32753]: pgmap v18009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33232.211019 secs 2024-04-26T21:04:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:53 smithi194 ceph-mon[33017]: pgmap v18009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33232.211019 secs 2024-04-26T21:04:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33235.153218 secs 2024-04-26T21:04:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33235.153218 secs 2024-04-26T21:04:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33235.153218 secs 2024-04-26T21:04:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:55 smithi033 ceph-mon[27845]: pgmap v18010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:55 smithi096 ceph-mon[32753]: pgmap v18010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:55 smithi194 ceph-mon[33017]: pgmap v18010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:57.176+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:57.176+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:04:57.176+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:04:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:57 smithi033 ceph-mon[27845]: pgmap v18011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:57 smithi096 ceph-mon[32753]: pgmap v18011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:57 smithi194 ceph-mon[33017]: pgmap v18011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:04:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33237.211123 secs 2024-04-26T21:04:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33237.211123 secs 2024-04-26T21:04:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33237.211123 secs 2024-04-26T21:04:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:59 smithi033 ceph-mon[27845]: pgmap v18012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:04:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33240.153340 secs 2024-04-26T21:04:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:59 smithi096 ceph-mon[32753]: pgmap v18012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:04:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33240.153340 secs 2024-04-26T21:04:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:59 smithi194 ceph-mon[33017]: pgmap v18012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:04:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:04:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33240.153340 secs 2024-04-26T21:05:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:01 smithi033 ceph-mon[27845]: pgmap v18013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:01 smithi096 ceph-mon[32753]: pgmap v18013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:01 smithi194 ceph-mon[33017]: pgmap v18013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:03 smithi033 ceph-mon[27845]: pgmap v18014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33242.211208 secs 2024-04-26T21:05:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:03 smithi096 ceph-mon[32753]: pgmap v18014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33242.211208 secs 2024-04-26T21:05:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:03 smithi194 ceph-mon[33017]: pgmap v18014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33242.211208 secs 2024-04-26T21:05:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33245.153462 secs 2024-04-26T21:05:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33245.153462 secs 2024-04-26T21:05:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33245.153462 secs 2024-04-26T21:05:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:05 smithi033 ceph-mon[27845]: pgmap v18015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:05 smithi096 ceph-mon[32753]: pgmap v18015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:05 smithi194 ceph-mon[33017]: pgmap v18015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:07 smithi033 ceph-mon[27845]: pgmap v18016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:07 smithi096 ceph-mon[32753]: pgmap v18016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:07 smithi194 ceph-mon[33017]: pgmap v18016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33247.211270 secs 2024-04-26T21:05:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33247.211270 secs 2024-04-26T21:05:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33247.211270 secs 2024-04-26T21:05:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:09 smithi194 ceph-mon[33017]: pgmap v18017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33250.153601 secs 2024-04-26T21:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:09 smithi033 ceph-mon[27845]: pgmap v18017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33250.153601 secs 2024-04-26T21:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:09 smithi096 ceph-mon[32753]: pgmap v18017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33250.153601 secs 2024-04-26T21:05:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:11 smithi194 ceph-mon[33017]: pgmap v18018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:11 smithi033 ceph-mon[27845]: pgmap v18018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:11 smithi096 ceph-mon[32753]: pgmap v18018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:12.190+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:12.190+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:12.190+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:05:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:13 smithi194 ceph-mon[33017]: pgmap v18019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33252.211353 secs 2024-04-26T21:05:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:13 smithi033 ceph-mon[27845]: pgmap v18019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33252.211353 secs 2024-04-26T21:05:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:13 smithi096 ceph-mon[32753]: pgmap v18019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33252.211353 secs 2024-04-26T21:05:14.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33255.153701 secs 2024-04-26T21:05:14.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33255.153701 secs 2024-04-26T21:05:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33255.153701 secs 2024-04-26T21:05:15.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:15 smithi033 ceph-mon[27845]: pgmap v18020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:15 smithi194 ceph-mon[33017]: pgmap v18020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:15 smithi096 ceph-mon[32753]: pgmap v18020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:17 smithi194 ceph-mon[33017]: pgmap v18021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:17 smithi033 ceph-mon[27845]: pgmap v18021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:17 smithi096 ceph-mon[32753]: pgmap v18021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33257.211437 secs 2024-04-26T21:05:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33257.211437 secs 2024-04-26T21:05:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33257.211437 secs 2024-04-26T21:05:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:19 smithi194 ceph-mon[33017]: pgmap v18022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33260.153843 secs 2024-04-26T21:05:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:19 smithi033 ceph-mon[27845]: pgmap v18022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33260.153843 secs 2024-04-26T21:05:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:19 smithi096 ceph-mon[32753]: pgmap v18022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:19.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33260.153843 secs 2024-04-26T21:05:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:21 smithi194 ceph-mon[33017]: pgmap v18023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:21 smithi033 ceph-mon[27845]: pgmap v18023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:21 smithi096 ceph-mon[32753]: pgmap v18023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:23 smithi194 ceph-mon[33017]: pgmap v18024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:05:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33262.211505 secs 2024-04-26T21:05:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:23 smithi033 ceph-mon[27845]: pgmap v18024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:05:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33262.211505 secs 2024-04-26T21:05:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:23 smithi096 ceph-mon[32753]: pgmap v18024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:05:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33262.211505 secs 2024-04-26T21:05:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33265.153995 secs 2024-04-26T21:05:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:05:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:05:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:24 smithi194 ceph-mon[33017]: pgmap v18025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:24 smithi194 ceph-mon[33017]: pgmap v18026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:05:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33265.153995 secs 2024-04-26T21:05:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:05:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:05:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:24 smithi033 ceph-mon[27845]: pgmap v18025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:24.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:24 smithi033 ceph-mon[27845]: pgmap v18026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:24.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:05:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33265.153995 secs 2024-04-26T21:05:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:05:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:05:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:24 smithi096 ceph-mon[32753]: pgmap v18025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:24 smithi096 ceph-mon[32753]: pgmap v18026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:05:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:26 smithi033 ceph-mon[27845]: pgmap v18027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:26 smithi096 ceph-mon[32753]: pgmap v18027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:27.198+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:27.198+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:27.198+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:05:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:26 smithi194 ceph-mon[33017]: pgmap v18027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:27.455+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:27.455+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T21:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:27.455+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 261 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T21:05:27.457 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:27.455+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 261 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:05:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:28 smithi033 ceph-mon[27845]: pgmap v18028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33267.211588 secs 2024-04-26T21:05:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:28 smithi096 ceph-mon[32753]: pgmap v18028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33267.211588 secs 2024-04-26T21:05:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:28 smithi194 ceph-mon[33017]: pgmap v18028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33267.211588 secs 2024-04-26T21:05:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33270.154127 secs 2024-04-26T21:05:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33270.154127 secs 2024-04-26T21:05:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33270.154127 secs 2024-04-26T21:05:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:30 smithi096 ceph-mon[32753]: pgmap v18029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:30 smithi194 ceph-mon[33017]: pgmap v18029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:30 smithi033 ceph-mon[27845]: pgmap v18029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:32 smithi194 ceph-mon[33017]: pgmap v18030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:32 smithi033 ceph-mon[27845]: pgmap v18030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:32 smithi096 ceph-mon[32753]: pgmap v18030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:05:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33272.211671 secs 2024-04-26T21:05:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33272.211671 secs 2024-04-26T21:05:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33272.211671 secs 2024-04-26T21:05:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33275.154227 secs 2024-04-26T21:05:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:34 smithi194 ceph-mon[33017]: pgmap v18031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:05:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33275.154227 secs 2024-04-26T21:05:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:34 smithi033 ceph-mon[27845]: pgmap v18031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:05:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33275.154227 secs 2024-04-26T21:05:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:34 smithi096 ceph-mon[32753]: pgmap v18031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:05:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:36 smithi194 ceph-mon[33017]: pgmap v18032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:36 smithi033 ceph-mon[27845]: pgmap v18032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:36 smithi096 ceph-mon[32753]: pgmap v18032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:38 smithi194 ceph-mon[33017]: pgmap v18033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33277.211757 secs 2024-04-26T21:05:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:38 smithi033 ceph-mon[27845]: pgmap v18033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33277.211757 secs 2024-04-26T21:05:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:38 smithi096 ceph-mon[32753]: pgmap v18033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33277.211757 secs 2024-04-26T21:05:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33280.154348 secs 2024-04-26T21:05:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33280.154348 secs 2024-04-26T21:05:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33280.154348 secs 2024-04-26T21:05:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:40 smithi194 ceph-mon[33017]: pgmap v18034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:40 smithi033 ceph-mon[27845]: pgmap v18034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:40 smithi096 ceph-mon[32753]: pgmap v18034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:42.200+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:42.200+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:42.200+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:05:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:42 smithi194 ceph-mon[33017]: pgmap v18035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:42 smithi033 ceph-mon[27845]: pgmap v18035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:42 smithi096 ceph-mon[32753]: pgmap v18035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:44.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33282.211834 secs 2024-04-26T21:05:44.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:43 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33285.154429 secs 2024-04-26T21:05:44.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33282.211834 secs 2024-04-26T21:05:44.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:43 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33285.154429 secs 2024-04-26T21:05:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33282.211834 secs 2024-04-26T21:05:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:43 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33285.154429 secs 2024-04-26T21:05:45.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:44 smithi033 ceph-mon[27845]: pgmap v18036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:44 smithi194 ceph-mon[33017]: pgmap v18036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:44 smithi096 ceph-mon[32753]: pgmap v18036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:46 smithi194 ceph-mon[33017]: pgmap v18037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:46 smithi033 ceph-mon[27845]: pgmap v18037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:46 smithi096 ceph-mon[32753]: pgmap v18037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:48 smithi194 ceph-mon[33017]: pgmap v18038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33287.211923 secs 2024-04-26T21:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:48 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33290.154548 secs 2024-04-26T21:05:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:48 smithi033 ceph-mon[27845]: pgmap v18038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33287.211923 secs 2024-04-26T21:05:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:48 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33290.154548 secs 2024-04-26T21:05:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:48 smithi096 ceph-mon[32753]: pgmap v18038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33287.211923 secs 2024-04-26T21:05:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:48 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33290.154548 secs 2024-04-26T21:05:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:51 smithi194 ceph-mon[33017]: pgmap v18039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:51 smithi033 ceph-mon[27845]: pgmap v18039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:51 smithi096 ceph-mon[32753]: pgmap v18039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:53 smithi194 ceph-mon[33017]: pgmap v18040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:53 smithi033 ceph-mon[27845]: pgmap v18040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:53 smithi096 ceph-mon[32753]: pgmap v18040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33292.212022 secs 2024-04-26T21:05:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33295.154675 secs 2024-04-26T21:05:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33292.212022 secs 2024-04-26T21:05:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33295.154675 secs 2024-04-26T21:05:54.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33292.212022 secs 2024-04-26T21:05:54.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33295.154675 secs 2024-04-26T21:05:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:55 smithi194 ceph-mon[33017]: pgmap v18041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:55 smithi033 ceph-mon[27845]: pgmap v18041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:55 smithi096 ceph-mon[32753]: pgmap v18041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:57.204+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:57.204+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:05:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:05:57.204+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:05:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:57 smithi194 ceph-mon[33017]: pgmap v18042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:57 smithi033 ceph-mon[27845]: pgmap v18042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:57 smithi096 ceph-mon[32753]: pgmap v18042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:05:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:59 smithi194 ceph-mon[33017]: pgmap v18043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33297.212099 secs 2024-04-26T21:05:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33300.154794 secs 2024-04-26T21:05:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:59 smithi033 ceph-mon[27845]: pgmap v18043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33297.212099 secs 2024-04-26T21:05:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33300.154794 secs 2024-04-26T21:05:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:59 smithi096 ceph-mon[32753]: pgmap v18043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:05:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33297.212099 secs 2024-04-26T21:05:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33300.154794 secs 2024-04-26T21:06:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:01 smithi194 ceph-mon[33017]: pgmap v18044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:01 smithi033 ceph-mon[27845]: pgmap v18044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:01 smithi096 ceph-mon[32753]: pgmap v18044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:03 smithi194 ceph-mon[33017]: pgmap v18045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:03 smithi033 ceph-mon[27845]: pgmap v18045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:03 smithi096 ceph-mon[32753]: pgmap v18045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33302.212183 secs 2024-04-26T21:06:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33305.154913 secs 2024-04-26T21:06:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33302.212183 secs 2024-04-26T21:06:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33305.154913 secs 2024-04-26T21:06:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33302.212183 secs 2024-04-26T21:06:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33305.154913 secs 2024-04-26T21:06:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:05 smithi194 ceph-mon[33017]: pgmap v18046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:05 smithi033 ceph-mon[27845]: pgmap v18046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:05 smithi096 ceph-mon[32753]: pgmap v18046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:07 smithi194 ceph-mon[33017]: pgmap v18047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:07 smithi033 ceph-mon[27845]: pgmap v18047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:07 smithi096 ceph-mon[32753]: pgmap v18047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33307.212268 secs 2024-04-26T21:06:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33307.212268 secs 2024-04-26T21:06:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33307.212268 secs 2024-04-26T21:06:09.336 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:09 smithi033 ceph-mon[27845]: pgmap v18048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:09.336 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33310.155036 secs 2024-04-26T21:06:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:09 smithi194 ceph-mon[33017]: pgmap v18048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33310.155036 secs 2024-04-26T21:06:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:09 smithi096 ceph-mon[32753]: pgmap v18048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33310.155036 secs 2024-04-26T21:06:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:11 smithi194 ceph-mon[33017]: pgmap v18049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:11 smithi033 ceph-mon[27845]: pgmap v18049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:11 smithi096 ceph-mon[32753]: pgmap v18049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:12.219 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:12.219+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:12.219+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:12.219+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:06:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:13 smithi194 ceph-mon[33017]: pgmap v18050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33312.212326 secs 2024-04-26T21:06:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:13 smithi033 ceph-mon[27845]: pgmap v18050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33312.212326 secs 2024-04-26T21:06:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:13 smithi096 ceph-mon[32753]: pgmap v18050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33312.212326 secs 2024-04-26T21:06:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33315.155159 secs 2024-04-26T21:06:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33315.155159 secs 2024-04-26T21:06:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33315.155159 secs 2024-04-26T21:06:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:15 smithi194 ceph-mon[33017]: pgmap v18051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:15 smithi033 ceph-mon[27845]: pgmap v18051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:15 smithi096 ceph-mon[32753]: pgmap v18051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:17 smithi033 ceph-mon[27845]: pgmap v18052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:17 smithi096 ceph-mon[32753]: pgmap v18052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:17 smithi194 ceph-mon[33017]: pgmap v18052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33317.212410 secs 2024-04-26T21:06:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33317.212410 secs 2024-04-26T21:06:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33317.212410 secs 2024-04-26T21:06:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:19 smithi033 ceph-mon[27845]: pgmap v18053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33320.155227 secs 2024-04-26T21:06:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:19 smithi096 ceph-mon[32753]: pgmap v18053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33320.155227 secs 2024-04-26T21:06:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:19 smithi194 ceph-mon[33017]: pgmap v18053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33320.155227 secs 2024-04-26T21:06:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:21 smithi033 ceph-mon[27845]: pgmap v18054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:21 smithi096 ceph-mon[32753]: pgmap v18054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:21 smithi194 ceph-mon[33017]: pgmap v18054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:23 smithi033 ceph-mon[27845]: pgmap v18055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33322.212490 secs 2024-04-26T21:06:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:23 smithi096 ceph-mon[32753]: pgmap v18055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33322.212490 secs 2024-04-26T21:06:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:23 smithi194 ceph-mon[33017]: pgmap v18055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33322.212490 secs 2024-04-26T21:06:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33325.155348 secs 2024-04-26T21:06:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33325.155348 secs 2024-04-26T21:06:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33325.155348 secs 2024-04-26T21:06:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:25 smithi033 ceph-mon[27845]: pgmap v18056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:06:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:25 smithi096 ceph-mon[32753]: pgmap v18056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:06:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:25 smithi194 ceph-mon[33017]: pgmap v18056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:06:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:06:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:06:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:06:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:06:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:06:26.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:06:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:06:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:06:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:27.233+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:27.233+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:27.233+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:06:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:27 smithi033 ceph-mon[27845]: pgmap v18057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:06:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:27 smithi033 ceph-mon[27845]: pgmap v18058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:27 smithi096 ceph-mon[32753]: pgmap v18057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:06:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:27 smithi096 ceph-mon[32753]: pgmap v18058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:27 smithi194 ceph-mon[33017]: pgmap v18057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:06:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:27 smithi194 ceph-mon[33017]: pgmap v18058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:28 smithi033 ceph-mon[27845]: pgmap v18059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33327.212575 secs 2024-04-26T21:06:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:28 smithi096 ceph-mon[32753]: pgmap v18059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33327.212575 secs 2024-04-26T21:06:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:28 smithi194 ceph-mon[33017]: pgmap v18059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33327.212575 secs 2024-04-26T21:06:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33330.155460 secs 2024-04-26T21:06:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33330.155460 secs 2024-04-26T21:06:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33330.155460 secs 2024-04-26T21:06:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:30 smithi194 ceph-mon[33017]: pgmap v18060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:30 smithi033 ceph-mon[27845]: pgmap v18060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:30 smithi096 ceph-mon[32753]: pgmap v18060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:32 smithi033 ceph-mon[27845]: pgmap v18061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:32 smithi096 ceph-mon[32753]: pgmap v18061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:33.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:32 smithi194 ceph-mon[33017]: pgmap v18061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33332.212655 secs 2024-04-26T21:06:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33332.212655 secs 2024-04-26T21:06:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33332.212655 secs 2024-04-26T21:06:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:34 smithi033 ceph-mon[27845]: pgmap v18062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33335.155583 secs 2024-04-26T21:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:34 smithi096 ceph-mon[32753]: pgmap v18062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33335.155583 secs 2024-04-26T21:06:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:34 smithi194 ceph-mon[33017]: pgmap v18062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:06:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33335.155583 secs 2024-04-26T21:06:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:36 smithi033 ceph-mon[27845]: pgmap v18063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:06:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:36 smithi096 ceph-mon[32753]: pgmap v18063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:06:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:36 smithi194 ceph-mon[33017]: pgmap v18063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:06:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:38 smithi033 ceph-mon[27845]: pgmap v18064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33337.212736 secs 2024-04-26T21:06:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:38 smithi096 ceph-mon[32753]: pgmap v18064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33337.212736 secs 2024-04-26T21:06:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:38 smithi194 ceph-mon[33017]: pgmap v18064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33337.212736 secs 2024-04-26T21:06:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33340.155701 secs 2024-04-26T21:06:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33340.155701 secs 2024-04-26T21:06:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33340.155701 secs 2024-04-26T21:06:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:40 smithi033 ceph-mon[27845]: pgmap v18065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:40 smithi096 ceph-mon[32753]: pgmap v18065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:40 smithi194 ceph-mon[33017]: pgmap v18065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:42.249+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:42.249+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:42.249+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:06:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:42 smithi096 ceph-mon[32753]: pgmap v18066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:42 smithi194 ceph-mon[33017]: pgmap v18066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:42 smithi033 ceph-mon[27845]: pgmap v18066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:44.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33342.212820 secs 2024-04-26T21:06:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33342.212820 secs 2024-04-26T21:06:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33342.212820 secs 2024-04-26T21:06:45.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:44 smithi033 ceph-mon[27845]: pgmap v18067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:45.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33345.155847 secs 2024-04-26T21:06:45.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:44 smithi096 ceph-mon[32753]: pgmap v18067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:45.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33345.155847 secs 2024-04-26T21:06:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:44 smithi194 ceph-mon[33017]: pgmap v18067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33345.155847 secs 2024-04-26T21:06:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:46 smithi194 ceph-mon[33017]: pgmap v18068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:46 smithi033 ceph-mon[27845]: pgmap v18068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:46 smithi096 ceph-mon[32753]: pgmap v18068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:48 smithi194 ceph-mon[33017]: pgmap v18069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33347.212922 secs 2024-04-26T21:06:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:48 smithi033 ceph-mon[27845]: pgmap v18069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33347.212922 secs 2024-04-26T21:06:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:48 smithi096 ceph-mon[32753]: pgmap v18069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33347.212922 secs 2024-04-26T21:06:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33350.155919 secs 2024-04-26T21:06:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33350.155919 secs 2024-04-26T21:06:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33350.155919 secs 2024-04-26T21:06:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:50 smithi194 ceph-mon[33017]: pgmap v18070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:50 smithi033 ceph-mon[27845]: pgmap v18070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:50 smithi096 ceph-mon[32753]: pgmap v18070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:52 smithi194 ceph-mon[33017]: pgmap v18071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:52 smithi033 ceph-mon[27845]: pgmap v18071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:52 smithi096 ceph-mon[32753]: pgmap v18071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33352.213007 secs 2024-04-26T21:06:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33352.213007 secs 2024-04-26T21:06:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33352.213007 secs 2024-04-26T21:06:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:54 smithi194 ceph-mon[33017]: pgmap v18072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33355.156054 secs 2024-04-26T21:06:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:54 smithi033 ceph-mon[27845]: pgmap v18072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33355.156054 secs 2024-04-26T21:06:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:54 smithi096 ceph-mon[32753]: pgmap v18072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33355.156054 secs 2024-04-26T21:06:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:56 smithi194 ceph-mon[33017]: pgmap v18073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:56 smithi033 ceph-mon[27845]: pgmap v18073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:56 smithi096 ceph-mon[32753]: pgmap v18073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:06:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:57.264+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:57.264+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:06:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:06:57.264+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:06:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:58 smithi194 ceph-mon[33017]: pgmap v18074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33357.213071 secs 2024-04-26T21:06:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:58 smithi033 ceph-mon[27845]: pgmap v18074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33357.213071 secs 2024-04-26T21:06:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:58 smithi096 ceph-mon[32753]: pgmap v18074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:06:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33357.213071 secs 2024-04-26T21:07:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33360.156170 secs 2024-04-26T21:07:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33360.156170 secs 2024-04-26T21:07:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33360.156170 secs 2024-04-26T21:07:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:00 smithi194 ceph-mon[33017]: pgmap v18075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:00 smithi033 ceph-mon[27845]: pgmap v18075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:00 smithi096 ceph-mon[32753]: pgmap v18075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:02 smithi194 ceph-mon[33017]: pgmap v18076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:02 smithi033 ceph-mon[27845]: pgmap v18076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:02 smithi096 ceph-mon[32753]: pgmap v18076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33362.213156 secs 2024-04-26T21:07:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33362.213156 secs 2024-04-26T21:07:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33362.213156 secs 2024-04-26T21:07:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:04 smithi194 ceph-mon[33017]: pgmap v18077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33365.156288 secs 2024-04-26T21:07:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:04 smithi033 ceph-mon[27845]: pgmap v18077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33365.156288 secs 2024-04-26T21:07:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:04 smithi096 ceph-mon[32753]: pgmap v18077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33365.156288 secs 2024-04-26T21:07:07.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:06 smithi194 ceph-mon[33017]: pgmap v18078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:06 smithi033 ceph-mon[27845]: pgmap v18078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:06 smithi096 ceph-mon[32753]: pgmap v18078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:08 smithi194 ceph-mon[33017]: pgmap v18079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33367.213257 secs 2024-04-26T21:07:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:08 smithi033 ceph-mon[27845]: pgmap v18079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:09.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33367.213257 secs 2024-04-26T21:07:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:08 smithi096 ceph-mon[32753]: pgmap v18079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33367.213257 secs 2024-04-26T21:07:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33370.156376 secs 2024-04-26T21:07:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33370.156376 secs 2024-04-26T21:07:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33370.156376 secs 2024-04-26T21:07:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:10 smithi194 ceph-mon[33017]: pgmap v18080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:10 smithi033 ceph-mon[27845]: pgmap v18080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:10 smithi096 ceph-mon[32753]: pgmap v18080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:12.269+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:12.269+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:07:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:12.269+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:12 smithi194 ceph-mon[33017]: pgmap v18081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:12 smithi033 ceph-mon[27845]: pgmap v18081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:12 smithi096 ceph-mon[32753]: pgmap v18081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33372.213340 secs 2024-04-26T21:07:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33372.213340 secs 2024-04-26T21:07:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33372.213340 secs 2024-04-26T21:07:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:14 smithi194 ceph-mon[33017]: pgmap v18082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33375.156520 secs 2024-04-26T21:07:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:14 smithi033 ceph-mon[27845]: pgmap v18082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33375.156520 secs 2024-04-26T21:07:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:14 smithi096 ceph-mon[32753]: pgmap v18082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33375.156520 secs 2024-04-26T21:07:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:16 smithi194 ceph-mon[33017]: pgmap v18083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:16 smithi033 ceph-mon[27845]: pgmap v18083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:16 smithi096 ceph-mon[32753]: pgmap v18083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:18 smithi194 ceph-mon[33017]: pgmap v18084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33377.213427 secs 2024-04-26T21:07:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:18 smithi033 ceph-mon[27845]: pgmap v18084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33377.213427 secs 2024-04-26T21:07:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:18 smithi096 ceph-mon[32753]: pgmap v18084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:07:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33377.213427 secs 2024-04-26T21:07:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33380.156670 secs 2024-04-26T21:07:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33380.156670 secs 2024-04-26T21:07:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33380.156670 secs 2024-04-26T21:07:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:20 smithi033 ceph-mon[27845]: pgmap v18085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:20 smithi096 ceph-mon[32753]: pgmap v18085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:20 smithi194 ceph-mon[33017]: pgmap v18085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:07:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:22 smithi033 ceph-mon[27845]: pgmap v18086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:22 smithi096 ceph-mon[32753]: pgmap v18086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:22 smithi194 ceph-mon[33017]: pgmap v18086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33382.213506 secs 2024-04-26T21:07:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33382.213506 secs 2024-04-26T21:07:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33382.213506 secs 2024-04-26T21:07:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:24 smithi033 ceph-mon[27845]: pgmap v18087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33385.156794 secs 2024-04-26T21:07:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:24 smithi096 ceph-mon[32753]: pgmap v18087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33385.156794 secs 2024-04-26T21:07:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:24 smithi194 ceph-mon[33017]: pgmap v18087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33385.156794 secs 2024-04-26T21:07:27.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:26 smithi033 ceph-mon[27845]: pgmap v18088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:27.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:07:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:26 smithi096 ceph-mon[32753]: pgmap v18088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:27.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:07:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:27.284+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:27.284+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:27.284+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:07:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:26 smithi194 ceph-mon[33017]: pgmap v18088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:07:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:07:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:07:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:07:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:07:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:07:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:07:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:07:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:07:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:07:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:28 smithi033 ceph-mon[27845]: pgmap v18089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:28 smithi033 ceph-mon[27845]: pgmap v18090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33387.213591 secs 2024-04-26T21:07:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:28 smithi096 ceph-mon[32753]: pgmap v18089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:28 smithi096 ceph-mon[32753]: pgmap v18090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33387.213591 secs 2024-04-26T21:07:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:28 smithi194 ceph-mon[33017]: pgmap v18089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:28 smithi194 ceph-mon[33017]: pgmap v18090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33387.213591 secs 2024-04-26T21:07:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33390.156861 secs 2024-04-26T21:07:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33390.156861 secs 2024-04-26T21:07:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33390.156861 secs 2024-04-26T21:07:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:30 smithi033 ceph-mon[27845]: pgmap v18091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:07:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:30 smithi096 ceph-mon[32753]: pgmap v18091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:07:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:30 smithi194 ceph-mon[33017]: pgmap v18091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:07:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:32 smithi033 ceph-mon[27845]: pgmap v18092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:32 smithi096 ceph-mon[32753]: pgmap v18092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:32 smithi194 ceph-mon[33017]: pgmap v18092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33392.213643 secs 2024-04-26T21:07:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33392.213643 secs 2024-04-26T21:07:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33392.213643 secs 2024-04-26T21:07:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:34 smithi033 ceph-mon[27845]: pgmap v18093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33395.156997 secs 2024-04-26T21:07:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:34 smithi096 ceph-mon[32753]: pgmap v18093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33395.156997 secs 2024-04-26T21:07:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:34 smithi194 ceph-mon[33017]: pgmap v18093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33395.156997 secs 2024-04-26T21:07:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:36 smithi096 ceph-mon[32753]: pgmap v18094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:36 smithi194 ceph-mon[33017]: pgmap v18094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:36 smithi033 ceph-mon[27845]: pgmap v18094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:38 smithi096 ceph-mon[32753]: pgmap v18095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33397.213705 secs 2024-04-26T21:07:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:38 smithi194 ceph-mon[33017]: pgmap v18095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:07:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33397.213705 secs 2024-04-26T21:07:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:38 smithi033 ceph-mon[27845]: pgmap v18095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:07:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33397.213705 secs 2024-04-26T21:07:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33400.157124 secs 2024-04-26T21:07:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33400.157124 secs 2024-04-26T21:07:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33400.157124 secs 2024-04-26T21:07:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:40 smithi194 ceph-mon[33017]: pgmap v18096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:40 smithi033 ceph-mon[27845]: pgmap v18096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:40 smithi096 ceph-mon[32753]: pgmap v18096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:42.297+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:42.297+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:42.297+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:07:43.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:42 smithi033 ceph-mon[27845]: pgmap v18097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:42 smithi194 ceph-mon[33017]: pgmap v18097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:42 smithi096 ceph-mon[32753]: pgmap v18097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:44.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33402.213762 secs 2024-04-26T21:07:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33402.213762 secs 2024-04-26T21:07:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33402.213762 secs 2024-04-26T21:07:45.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:44 smithi033 ceph-mon[27845]: pgmap v18098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:45.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33405.157247 secs 2024-04-26T21:07:45.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:44 smithi096 ceph-mon[32753]: pgmap v18098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:45.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33405.157247 secs 2024-04-26T21:07:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:44 smithi194 ceph-mon[33017]: pgmap v18098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33405.157247 secs 2024-04-26T21:07:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:46 smithi194 ceph-mon[33017]: pgmap v18099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:46 smithi033 ceph-mon[27845]: pgmap v18099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:46 smithi096 ceph-mon[32753]: pgmap v18099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:48 smithi194 ceph-mon[33017]: pgmap v18100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33407.213821 secs 2024-04-26T21:07:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:48 smithi033 ceph-mon[27845]: pgmap v18100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33407.213821 secs 2024-04-26T21:07:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:48 smithi096 ceph-mon[32753]: pgmap v18100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33407.213821 secs 2024-04-26T21:07:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33410.157364 secs 2024-04-26T21:07:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33410.157364 secs 2024-04-26T21:07:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33410.157364 secs 2024-04-26T21:07:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:50 smithi194 ceph-mon[33017]: pgmap v18101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:50 smithi033 ceph-mon[27845]: pgmap v18101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:50 smithi096 ceph-mon[32753]: pgmap v18101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:52 smithi194 ceph-mon[33017]: pgmap v18102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:52 smithi033 ceph-mon[27845]: pgmap v18102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:52 smithi096 ceph-mon[32753]: pgmap v18102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:07:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33412.213877 secs 2024-04-26T21:07:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33412.213877 secs 2024-04-26T21:07:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33412.213877 secs 2024-04-26T21:07:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:55 smithi194 ceph-mon[33017]: pgmap v18103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33415.157485 secs 2024-04-26T21:07:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:55 smithi033 ceph-mon[27845]: pgmap v18103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33415.157485 secs 2024-04-26T21:07:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:55 smithi096 ceph-mon[32753]: pgmap v18103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33415.157485 secs 2024-04-26T21:07:57.302 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:57.301+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:57.302+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:07:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:07:57.302+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:07:57.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:57 smithi194 ceph-mon[33017]: pgmap v18104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:57 smithi033 ceph-mon[27845]: pgmap v18104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:57 smithi096 ceph-mon[32753]: pgmap v18104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:59 smithi194 ceph-mon[33017]: pgmap v18105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33417.213933 secs 2024-04-26T21:07:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:07:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33420.157605 secs 2024-04-26T21:07:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:59 smithi033 ceph-mon[27845]: pgmap v18105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33417.213933 secs 2024-04-26T21:07:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:07:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33420.157605 secs 2024-04-26T21:07:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:59 smithi096 ceph-mon[32753]: pgmap v18105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:07:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33417.213933 secs 2024-04-26T21:07:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:07:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33420.157605 secs 2024-04-26T21:08:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:01 smithi194 ceph-mon[33017]: pgmap v18106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:01 smithi033 ceph-mon[27845]: pgmap v18106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:01 smithi096 ceph-mon[32753]: pgmap v18106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:03.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:03 smithi194 ceph-mon[33017]: pgmap v18107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:03 smithi033 ceph-mon[27845]: pgmap v18107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:03 smithi096 ceph-mon[32753]: pgmap v18107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33422.213987 secs 2024-04-26T21:08:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33425.157742 secs 2024-04-26T21:08:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33422.213987 secs 2024-04-26T21:08:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33425.157742 secs 2024-04-26T21:08:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33422.213987 secs 2024-04-26T21:08:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33425.157742 secs 2024-04-26T21:08:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:05 smithi194 ceph-mon[33017]: pgmap v18108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:05 smithi033 ceph-mon[27845]: pgmap v18108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:05 smithi096 ceph-mon[32753]: pgmap v18108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:07 smithi194 ceph-mon[33017]: pgmap v18109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:07 smithi033 ceph-mon[27845]: pgmap v18109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:07 smithi096 ceph-mon[32753]: pgmap v18109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:09 smithi194 ceph-mon[33017]: pgmap v18110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33427.214046 secs 2024-04-26T21:08:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33430.157877 secs 2024-04-26T21:08:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:09 smithi033 ceph-mon[27845]: pgmap v18110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33427.214046 secs 2024-04-26T21:08:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33430.157877 secs 2024-04-26T21:08:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:09 smithi096 ceph-mon[32753]: pgmap v18110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33427.214046 secs 2024-04-26T21:08:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33430.157877 secs 2024-04-26T21:08:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:11 smithi194 ceph-mon[33017]: pgmap v18111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:11 smithi033 ceph-mon[27845]: pgmap v18111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:11 smithi096 ceph-mon[32753]: pgmap v18111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:12.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:12.311+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:12.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:12.311+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:12.312 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:12.311+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:08:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:13 smithi194 ceph-mon[33017]: pgmap v18112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:13 smithi033 ceph-mon[27845]: pgmap v18112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:13 smithi096 ceph-mon[32753]: pgmap v18112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33432.214102 secs 2024-04-26T21:08:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33435.158013 secs 2024-04-26T21:08:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33432.214102 secs 2024-04-26T21:08:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33435.158013 secs 2024-04-26T21:08:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33432.214102 secs 2024-04-26T21:08:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33435.158013 secs 2024-04-26T21:08:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:15 smithi194 ceph-mon[33017]: pgmap v18113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:15 smithi033 ceph-mon[27845]: pgmap v18113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:15 smithi096 ceph-mon[32753]: pgmap v18113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:17 smithi194 ceph-mon[33017]: pgmap v18114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:17 smithi033 ceph-mon[27845]: pgmap v18114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:17 smithi096 ceph-mon[32753]: pgmap v18114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33437.214164 secs 2024-04-26T21:08:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33437.214164 secs 2024-04-26T21:08:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33437.214164 secs 2024-04-26T21:08:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:19 smithi194 ceph-mon[33017]: pgmap v18115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33440.158150 secs 2024-04-26T21:08:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:19 smithi033 ceph-mon[27845]: pgmap v18115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33440.158150 secs 2024-04-26T21:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:19 smithi096 ceph-mon[32753]: pgmap v18115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33440.158150 secs 2024-04-26T21:08:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:21 smithi033 ceph-mon[27845]: pgmap v18116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:21 smithi096 ceph-mon[32753]: pgmap v18116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:21 smithi194 ceph-mon[33017]: pgmap v18116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:23 smithi033 ceph-mon[27845]: pgmap v18117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33442.214221 secs 2024-04-26T21:08:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:23 smithi096 ceph-mon[32753]: pgmap v18117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33442.214221 secs 2024-04-26T21:08:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:23 smithi194 ceph-mon[33017]: pgmap v18117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33442.214221 secs 2024-04-26T21:08:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33445.158268 secs 2024-04-26T21:08:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33445.158268 secs 2024-04-26T21:08:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33445.158268 secs 2024-04-26T21:08:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:25 smithi033 ceph-mon[27845]: pgmap v18118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:25 smithi096 ceph-mon[32753]: pgmap v18118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:25 smithi194 ceph-mon[33017]: pgmap v18118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:27.326+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:27.326+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:27.326+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:08:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:27 smithi033 ceph-mon[27845]: pgmap v18119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:27 smithi096 ceph-mon[32753]: pgmap v18119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:27.485+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:08:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:27.485+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:08:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:27.486+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 262 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T21:08:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:27.486+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 262 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:08:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:27 smithi194 ceph-mon[33017]: pgmap v18119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:08:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33447.214280 secs 2024-04-26T21:08:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:08:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33447.214280 secs 2024-04-26T21:08:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:08:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33447.214280 secs 2024-04-26T21:08:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:29 smithi033 ceph-mon[27845]: pgmap v18120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33450.158356 secs 2024-04-26T21:08:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:08:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:08:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:08:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:29 smithi096 ceph-mon[32753]: pgmap v18120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33450.158356 secs 2024-04-26T21:08:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:08:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:08:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:29 smithi194 ceph-mon[33017]: pgmap v18120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33450.158356 secs 2024-04-26T21:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:08:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:08:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:30 smithi033 ceph-mon[27845]: pgmap v18121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:30 smithi033 ceph-mon[27845]: pgmap v18122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:08:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:30 smithi096 ceph-mon[32753]: pgmap v18121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:30 smithi096 ceph-mon[32753]: pgmap v18122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:08:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:30 smithi194 ceph-mon[33017]: pgmap v18121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:30 smithi194 ceph-mon[33017]: pgmap v18122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:08:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:32 smithi033 ceph-mon[27845]: pgmap v18123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:32 smithi096 ceph-mon[32753]: pgmap v18123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:32 smithi194 ceph-mon[33017]: pgmap v18123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33452.214334 secs 2024-04-26T21:08:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33452.214334 secs 2024-04-26T21:08:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33452.214334 secs 2024-04-26T21:08:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:34 smithi033 ceph-mon[27845]: pgmap v18124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33455.158438 secs 2024-04-26T21:08:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:34 smithi096 ceph-mon[32753]: pgmap v18124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33455.158438 secs 2024-04-26T21:08:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:34 smithi194 ceph-mon[33017]: pgmap v18124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33455.158438 secs 2024-04-26T21:08:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:36 smithi096 ceph-mon[32753]: pgmap v18125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:36 smithi194 ceph-mon[33017]: pgmap v18125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:36 smithi033 ceph-mon[27845]: pgmap v18125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:38 smithi096 ceph-mon[32753]: pgmap v18126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33457.214395 secs 2024-04-26T21:08:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:38 smithi194 ceph-mon[33017]: pgmap v18126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33457.214395 secs 2024-04-26T21:08:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:38 smithi033 ceph-mon[27845]: pgmap v18126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:08:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33457.214395 secs 2024-04-26T21:08:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33460.158557 secs 2024-04-26T21:08:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:39 smithi194 ceph-mon[33017]: pgmap v18127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:08:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33460.158557 secs 2024-04-26T21:08:39.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:39 smithi033 ceph-mon[27845]: pgmap v18127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:08:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33460.158557 secs 2024-04-26T21:08:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:39 smithi096 ceph-mon[32753]: pgmap v18127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:08:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:42.341+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:42.341+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:42.341+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:08:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:42 smithi194 ceph-mon[33017]: pgmap v18128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:42 smithi033 ceph-mon[27845]: pgmap v18128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:42 smithi096 ceph-mon[32753]: pgmap v18128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33462.214446 secs 2024-04-26T21:08:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:44 smithi194 ceph-mon[33017]: pgmap v18129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33465.158674 secs 2024-04-26T21:08:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33462.214446 secs 2024-04-26T21:08:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:44 smithi033 ceph-mon[27845]: pgmap v18129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33465.158674 secs 2024-04-26T21:08:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33462.214446 secs 2024-04-26T21:08:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:44 smithi096 ceph-mon[32753]: pgmap v18129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33465.158674 secs 2024-04-26T21:08:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:46 smithi194 ceph-mon[33017]: pgmap v18130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:46 smithi033 ceph-mon[27845]: pgmap v18130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:46 smithi096 ceph-mon[32753]: pgmap v18130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:48 smithi194 ceph-mon[33017]: pgmap v18131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:48 smithi033 ceph-mon[27845]: pgmap v18131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:48 smithi096 ceph-mon[32753]: pgmap v18131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33467.214505 secs 2024-04-26T21:08:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33470.158789 secs 2024-04-26T21:08:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33467.214505 secs 2024-04-26T21:08:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33470.158789 secs 2024-04-26T21:08:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33467.214505 secs 2024-04-26T21:08:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33470.158789 secs 2024-04-26T21:08:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:50 smithi194 ceph-mon[33017]: pgmap v18132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:50 smithi033 ceph-mon[27845]: pgmap v18132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:50 smithi096 ceph-mon[32753]: pgmap v18132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:52 smithi194 ceph-mon[33017]: pgmap v18133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:52 smithi033 ceph-mon[27845]: pgmap v18133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:52 smithi096 ceph-mon[32753]: pgmap v18133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33472.214561 secs 2024-04-26T21:08:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33472.214561 secs 2024-04-26T21:08:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33472.214561 secs 2024-04-26T21:08:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:54 smithi194 ceph-mon[33017]: pgmap v18134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33475.158904 secs 2024-04-26T21:08:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:54 smithi033 ceph-mon[27845]: pgmap v18134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33475.158904 secs 2024-04-26T21:08:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:54 smithi096 ceph-mon[32753]: pgmap v18134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33475.158904 secs 2024-04-26T21:08:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:56 smithi194 ceph-mon[33017]: pgmap v18135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:56 smithi033 ceph-mon[27845]: pgmap v18135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:56 smithi096 ceph-mon[32753]: pgmap v18135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:08:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:57.343+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:57.343+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:08:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:08:57.343+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:08:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:58 smithi033 ceph-mon[27845]: pgmap v18136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33477.214622 secs 2024-04-26T21:08:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:58 smithi096 ceph-mon[32753]: pgmap v18136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33477.214622 secs 2024-04-26T21:08:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:58 smithi194 ceph-mon[33017]: pgmap v18136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:08:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33477.214622 secs 2024-04-26T21:08:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33480.159040 secs 2024-04-26T21:08:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33480.159040 secs 2024-04-26T21:08:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33480.159040 secs 2024-04-26T21:09:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:00 smithi033 ceph-mon[27845]: pgmap v18137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:00 smithi096 ceph-mon[32753]: pgmap v18137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:00 smithi194 ceph-mon[33017]: pgmap v18137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:02.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:02 smithi033 ceph-mon[27845]: pgmap v18138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:02 smithi096 ceph-mon[32753]: pgmap v18138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:02 smithi194 ceph-mon[33017]: pgmap v18138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33482.214670 secs 2024-04-26T21:09:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33482.214670 secs 2024-04-26T21:09:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33482.214670 secs 2024-04-26T21:09:04.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:04 smithi033 ceph-mon[27845]: pgmap v18139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:04.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33485.159158 secs 2024-04-26T21:09:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:04 smithi096 ceph-mon[32753]: pgmap v18139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33485.159158 secs 2024-04-26T21:09:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:04 smithi194 ceph-mon[33017]: pgmap v18139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33485.159158 secs 2024-04-26T21:09:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:06 smithi033 ceph-mon[27845]: pgmap v18140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:06 smithi096 ceph-mon[32753]: pgmap v18140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:06 smithi194 ceph-mon[33017]: pgmap v18140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:08 smithi033 ceph-mon[27845]: pgmap v18141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33487.214731 secs 2024-04-26T21:09:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:08 smithi096 ceph-mon[32753]: pgmap v18141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33487.214731 secs 2024-04-26T21:09:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:08 smithi194 ceph-mon[33017]: pgmap v18141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33487.214731 secs 2024-04-26T21:09:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33490.159273 secs 2024-04-26T21:09:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33490.159273 secs 2024-04-26T21:09:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33490.159273 secs 2024-04-26T21:09:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:10 smithi033 ceph-mon[27845]: pgmap v18142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:10 smithi096 ceph-mon[32753]: pgmap v18142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:10 smithi194 ceph-mon[33017]: pgmap v18142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:12.348+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:12.348+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:12.348+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:09:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:12 smithi033 ceph-mon[27845]: pgmap v18143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:12 smithi096 ceph-mon[32753]: pgmap v18143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:12 smithi194 ceph-mon[33017]: pgmap v18143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33492.214779 secs 2024-04-26T21:09:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33492.214779 secs 2024-04-26T21:09:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33492.214779 secs 2024-04-26T21:09:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:14 smithi096 ceph-mon[32753]: pgmap v18144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33495.159389 secs 2024-04-26T21:09:14.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:14 smithi194 ceph-mon[33017]: pgmap v18144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:14.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33495.159389 secs 2024-04-26T21:09:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:14 smithi033 ceph-mon[27845]: pgmap v18144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33495.159389 secs 2024-04-26T21:09:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:15 smithi194 ceph-mon[33017]: pgmap v18145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:15 smithi033 ceph-mon[27845]: pgmap v18145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:15 smithi096 ceph-mon[32753]: pgmap v18145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:18 smithi194 ceph-mon[33017]: pgmap v18146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:18 smithi033 ceph-mon[27845]: pgmap v18146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:18 smithi096 ceph-mon[32753]: pgmap v18146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33497.214833 secs 2024-04-26T21:09:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33500.159511 secs 2024-04-26T21:09:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33497.214833 secs 2024-04-26T21:09:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33500.159511 secs 2024-04-26T21:09:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33497.214833 secs 2024-04-26T21:09:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33500.159511 secs 2024-04-26T21:09:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:20 smithi194 ceph-mon[33017]: pgmap v18147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:20 smithi033 ceph-mon[27845]: pgmap v18147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:20 smithi096 ceph-mon[32753]: pgmap v18147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:22 smithi194 ceph-mon[33017]: pgmap v18148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:22 smithi033 ceph-mon[27845]: pgmap v18148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:22 smithi096 ceph-mon[32753]: pgmap v18148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33502.214914 secs 2024-04-26T21:09:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33502.214914 secs 2024-04-26T21:09:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33502.214914 secs 2024-04-26T21:09:24.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:24 smithi194 ceph-mon[33017]: pgmap v18149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:24.356 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33505.159590 secs 2024-04-26T21:09:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:24 smithi033 ceph-mon[27845]: pgmap v18149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33505.159590 secs 2024-04-26T21:09:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:24 smithi096 ceph-mon[32753]: pgmap v18149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33505.159590 secs 2024-04-26T21:09:26.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:25 smithi194 ceph-mon[33017]: pgmap v18150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:25 smithi033 ceph-mon[27845]: pgmap v18150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:25 smithi096 ceph-mon[32753]: pgmap v18150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:27.363+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:27.363+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:27.363+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:09:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:28 smithi194 ceph-mon[33017]: pgmap v18151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:28 smithi033 ceph-mon[27845]: pgmap v18151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:28 smithi096 ceph-mon[32753]: pgmap v18151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33507.214999 secs 2024-04-26T21:09:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33510.159709 secs 2024-04-26T21:09:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33507.214999 secs 2024-04-26T21:09:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33510.159709 secs 2024-04-26T21:09:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33507.214999 secs 2024-04-26T21:09:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33510.159709 secs 2024-04-26T21:09:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:30 smithi194 ceph-mon[33017]: pgmap v18152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:09:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:30 smithi033 ceph-mon[27845]: pgmap v18152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:09:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:30 smithi096 ceph-mon[32753]: pgmap v18152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:09:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:09:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:09:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:09:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:09:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:09:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:09:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:09:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:09:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:09:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:32 smithi194 ceph-mon[33017]: pgmap v18153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:09:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:32 smithi194 ceph-mon[33017]: pgmap v18154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:32 smithi033 ceph-mon[27845]: pgmap v18153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:09:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:32 smithi033 ceph-mon[27845]: pgmap v18154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:32 smithi096 ceph-mon[32753]: pgmap v18153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:09:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:32 smithi096 ceph-mon[32753]: pgmap v18154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33512.215059 secs 2024-04-26T21:09:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33512.215059 secs 2024-04-26T21:09:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33512.215059 secs 2024-04-26T21:09:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:34 smithi033 ceph-mon[27845]: pgmap v18155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33515.159825 secs 2024-04-26T21:09:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:34 smithi096 ceph-mon[32753]: pgmap v18155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33515.159825 secs 2024-04-26T21:09:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:34 smithi194 ceph-mon[33017]: pgmap v18155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33515.159825 secs 2024-04-26T21:09:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:36 smithi033 ceph-mon[27845]: pgmap v18156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:36 smithi096 ceph-mon[32753]: pgmap v18156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:36 smithi194 ceph-mon[33017]: pgmap v18156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:38 smithi033 ceph-mon[27845]: pgmap v18157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33517.215121 secs 2024-04-26T21:09:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:38 smithi096 ceph-mon[32753]: pgmap v18157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33517.215121 secs 2024-04-26T21:09:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:38 smithi194 ceph-mon[33017]: pgmap v18157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33517.215121 secs 2024-04-26T21:09:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33520.159959 secs 2024-04-26T21:09:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33520.159959 secs 2024-04-26T21:09:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33520.159959 secs 2024-04-26T21:09:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:40 smithi033 ceph-mon[27845]: pgmap v18158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:40 smithi096 ceph-mon[32753]: pgmap v18158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:40 smithi194 ceph-mon[33017]: pgmap v18158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:09:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:42.379+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:42.379+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:42.379+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:09:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:42 smithi033 ceph-mon[27845]: pgmap v18159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:09:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:42 smithi096 ceph-mon[32753]: pgmap v18159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:09:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:42 smithi194 ceph-mon[33017]: pgmap v18159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:09:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33522.215180 secs 2024-04-26T21:09:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33522.215180 secs 2024-04-26T21:09:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33522.215180 secs 2024-04-26T21:09:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:44 smithi033 ceph-mon[27845]: pgmap v18160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33525.160079 secs 2024-04-26T21:09:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:44 smithi096 ceph-mon[32753]: pgmap v18160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33525.160079 secs 2024-04-26T21:09:44.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:44 smithi194 ceph-mon[33017]: pgmap v18160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:44.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33525.160079 secs 2024-04-26T21:09:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:46 smithi033 ceph-mon[27845]: pgmap v18161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:46 smithi096 ceph-mon[32753]: pgmap v18161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:46 smithi194 ceph-mon[33017]: pgmap v18161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:48.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:48 smithi033 ceph-mon[27845]: pgmap v18162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:48.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33527.215241 secs 2024-04-26T21:09:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:48 smithi096 ceph-mon[32753]: pgmap v18162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33527.215241 secs 2024-04-26T21:09:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:48 smithi194 ceph-mon[33017]: pgmap v18162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33527.215241 secs 2024-04-26T21:09:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33530.160200 secs 2024-04-26T21:09:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33530.160200 secs 2024-04-26T21:09:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33530.160200 secs 2024-04-26T21:09:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:50 smithi096 ceph-mon[32753]: pgmap v18163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:50 smithi194 ceph-mon[33017]: pgmap v18163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:50 smithi033 ceph-mon[27845]: pgmap v18163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:52 smithi096 ceph-mon[32753]: pgmap v18164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:52 smithi194 ceph-mon[33017]: pgmap v18164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:52 smithi033 ceph-mon[27845]: pgmap v18164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33532.215300 secs 2024-04-26T21:09:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33532.215300 secs 2024-04-26T21:09:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33532.215300 secs 2024-04-26T21:09:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:54 smithi194 ceph-mon[33017]: pgmap v18165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33535.160347 secs 2024-04-26T21:09:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:54 smithi033 ceph-mon[27845]: pgmap v18165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33535.160347 secs 2024-04-26T21:09:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:54 smithi096 ceph-mon[32753]: pgmap v18165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33535.160347 secs 2024-04-26T21:09:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:55 smithi194 ceph-mon[33017]: pgmap v18166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:55 smithi033 ceph-mon[27845]: pgmap v18166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:55 smithi096 ceph-mon[32753]: pgmap v18166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:09:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:57.385+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:57.385+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:09:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:09:57.385+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:09:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:57 smithi194 ceph-mon[33017]: pgmap v18167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:57 smithi033 ceph-mon[27845]: pgmap v18167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:57 smithi096 ceph-mon[32753]: pgmap v18167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:09:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33537.215362 secs 2024-04-26T21:09:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33537.215362 secs 2024-04-26T21:09:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33537.215362 secs 2024-04-26T21:10:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:59 smithi194 ceph-mon[33017]: pgmap v18168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33540.160429 secs 2024-04-26T21:10:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:59 smithi033 ceph-mon[27845]: pgmap v18168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33540.160429 secs 2024-04-26T21:10:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:59 smithi096 ceph-mon[32753]: pgmap v18168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33540.160429 secs 2024-04-26T21:10:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:10:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:10:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:10:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:01 smithi194 ceph-mon[33017]: pgmap v18169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:01 smithi033 ceph-mon[27845]: pgmap v18169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:01 smithi096 ceph-mon[32753]: pgmap v18169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:03 smithi194 ceph-mon[33017]: pgmap v18170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33542.215442 secs 2024-04-26T21:10:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:03 smithi033 ceph-mon[27845]: pgmap v18170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33542.215442 secs 2024-04-26T21:10:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:03 smithi096 ceph-mon[32753]: pgmap v18170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33542.215442 secs 2024-04-26T21:10:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33545.160548 secs 2024-04-26T21:10:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33545.160548 secs 2024-04-26T21:10:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33545.160548 secs 2024-04-26T21:10:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:05 smithi194 ceph-mon[33017]: pgmap v18171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:05 smithi033 ceph-mon[27845]: pgmap v18171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:05 smithi096 ceph-mon[32753]: pgmap v18171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:07 smithi194 ceph-mon[33017]: pgmap v18172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:07 smithi033 ceph-mon[27845]: pgmap v18172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:07 smithi096 ceph-mon[32753]: pgmap v18172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33547.215526 secs 2024-04-26T21:10:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33547.215526 secs 2024-04-26T21:10:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33547.215526 secs 2024-04-26T21:10:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:09 smithi194 ceph-mon[33017]: pgmap v18173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33550.160697 secs 2024-04-26T21:10:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:09 smithi033 ceph-mon[27845]: pgmap v18173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33550.160697 secs 2024-04-26T21:10:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:09 smithi096 ceph-mon[32753]: pgmap v18173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33550.160697 secs 2024-04-26T21:10:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:11 smithi194 ceph-mon[33017]: pgmap v18174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:11 smithi033 ceph-mon[27845]: pgmap v18174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:11 smithi096 ceph-mon[32753]: pgmap v18174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:12.401+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:12.401+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:12.401+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:13 smithi194 ceph-mon[33017]: pgmap v18175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33552.215584 secs 2024-04-26T21:10:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:13 smithi033 ceph-mon[27845]: pgmap v18175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33552.215584 secs 2024-04-26T21:10:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:13 smithi096 ceph-mon[32753]: pgmap v18175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33552.215584 secs 2024-04-26T21:10:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33555.160818 secs 2024-04-26T21:10:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33555.160818 secs 2024-04-26T21:10:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33555.160818 secs 2024-04-26T21:10:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:15 smithi194 ceph-mon[33017]: pgmap v18176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:15 smithi033 ceph-mon[27845]: pgmap v18176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:15 smithi096 ceph-mon[32753]: pgmap v18176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:17 smithi194 ceph-mon[33017]: pgmap v18177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:17 smithi033 ceph-mon[27845]: pgmap v18177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:17 smithi096 ceph-mon[32753]: pgmap v18177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33557.215645 secs 2024-04-26T21:10:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33557.215645 secs 2024-04-26T21:10:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33557.215645 secs 2024-04-26T21:10:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:19 smithi033 ceph-mon[27845]: pgmap v18178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33560.160940 secs 2024-04-26T21:10:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:19 smithi096 ceph-mon[32753]: pgmap v18178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33560.160940 secs 2024-04-26T21:10:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:19 smithi194 ceph-mon[33017]: pgmap v18178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33560.160940 secs 2024-04-26T21:10:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:21 smithi033 ceph-mon[27845]: pgmap v18179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:21 smithi096 ceph-mon[32753]: pgmap v18179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:21 smithi194 ceph-mon[33017]: pgmap v18179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:23 smithi033 ceph-mon[27845]: pgmap v18180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33562.215726 secs 2024-04-26T21:10:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:23 smithi096 ceph-mon[32753]: pgmap v18180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33562.215726 secs 2024-04-26T21:10:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:23 smithi194 ceph-mon[33017]: pgmap v18180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33562.215726 secs 2024-04-26T21:10:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33565.161060 secs 2024-04-26T21:10:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33565.161060 secs 2024-04-26T21:10:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33565.161060 secs 2024-04-26T21:10:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:25 smithi033 ceph-mon[27845]: pgmap v18181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:25 smithi096 ceph-mon[32753]: pgmap v18181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:25 smithi194 ceph-mon[33017]: pgmap v18181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:27.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:27.411+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:27.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:27.411+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:27.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:27.411+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:10:28.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:27 smithi033 ceph-mon[27845]: pgmap v18182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:27 smithi096 ceph-mon[32753]: pgmap v18182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:28.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:27 smithi194 ceph-mon[33017]: pgmap v18182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33567.215810 secs 2024-04-26T21:10:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33567.215810 secs 2024-04-26T21:10:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33567.215810 secs 2024-04-26T21:10:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:29 smithi033 ceph-mon[27845]: pgmap v18183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33570.161189 secs 2024-04-26T21:10:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:29 smithi096 ceph-mon[32753]: pgmap v18183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33570.161189 secs 2024-04-26T21:10:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:29 smithi194 ceph-mon[33017]: pgmap v18183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33570.161189 secs 2024-04-26T21:10:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:31 smithi033 ceph-mon[27845]: pgmap v18184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:10:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:31 smithi096 ceph-mon[32753]: pgmap v18184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:10:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:31 smithi194 ceph-mon[33017]: pgmap v18184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:10:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:10:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:10:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:10:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:10:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:10:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:10:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:10:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:10:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:10:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:33 smithi033 ceph-mon[27845]: pgmap v18185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:33 smithi033 ceph-mon[27845]: pgmap v18186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33572.215887 secs 2024-04-26T21:10:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:33 smithi096 ceph-mon[32753]: pgmap v18185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:33 smithi096 ceph-mon[32753]: pgmap v18186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33572.215887 secs 2024-04-26T21:10:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:33 smithi194 ceph-mon[33017]: pgmap v18185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:33 smithi194 ceph-mon[33017]: pgmap v18186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33572.215887 secs 2024-04-26T21:10:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33575.161274 secs 2024-04-26T21:10:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33575.161274 secs 2024-04-26T21:10:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33575.161274 secs 2024-04-26T21:10:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:35 smithi096 ceph-mon[32753]: pgmap v18187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:10:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:35 smithi194 ceph-mon[33017]: pgmap v18187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:10:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:35 smithi033 ceph-mon[27845]: pgmap v18187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:10:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:37 smithi194 ceph-mon[33017]: pgmap v18188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:37 smithi033 ceph-mon[27845]: pgmap v18188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:37 smithi096 ceph-mon[32753]: pgmap v18188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33577.215975 secs 2024-04-26T21:10:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33577.215975 secs 2024-04-26T21:10:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33577.215975 secs 2024-04-26T21:10:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:39 smithi194 ceph-mon[33017]: pgmap v18189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33580.161393 secs 2024-04-26T21:10:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:39 smithi033 ceph-mon[27845]: pgmap v18189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33580.161393 secs 2024-04-26T21:10:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:39 smithi096 ceph-mon[32753]: pgmap v18189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33580.161393 secs 2024-04-26T21:10:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:41 smithi194 ceph-mon[33017]: pgmap v18190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:41 smithi033 ceph-mon[27845]: pgmap v18190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:42.425+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:42.425+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:42.425+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:10:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:41 smithi096 ceph-mon[32753]: pgmap v18190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:42.510 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:42.509+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:10:42.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:42.509+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T21:10:42.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:42.509+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 263 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T21:10:42.511 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:42.509+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 263 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:10:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:43 smithi194 ceph-mon[33017]: pgmap v18191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:10:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33582.215997 secs 2024-04-26T21:10:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:43 smithi033 ceph-mon[27845]: pgmap v18191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:10:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33582.215997 secs 2024-04-26T21:10:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:43 smithi096 ceph-mon[32753]: pgmap v18191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:10:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33582.215997 secs 2024-04-26T21:10:45.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33585.161458 secs 2024-04-26T21:10:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33585.161458 secs 2024-04-26T21:10:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33585.161458 secs 2024-04-26T21:10:46.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:45 smithi033 ceph-mon[27845]: pgmap v18192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:46.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:45 smithi096 ceph-mon[32753]: pgmap v18192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:45 smithi194 ceph-mon[33017]: pgmap v18192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:47 smithi194 ceph-mon[33017]: pgmap v18193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:47 smithi033 ceph-mon[27845]: pgmap v18193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:47 smithi096 ceph-mon[32753]: pgmap v18193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33587.216057 secs 2024-04-26T21:10:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33587.216057 secs 2024-04-26T21:10:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33587.216057 secs 2024-04-26T21:10:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:49 smithi033 ceph-mon[27845]: pgmap v18194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33590.161575 secs 2024-04-26T21:10:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:49 smithi194 ceph-mon[33017]: pgmap v18194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33590.161575 secs 2024-04-26T21:10:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:49 smithi096 ceph-mon[32753]: pgmap v18194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33590.161575 secs 2024-04-26T21:10:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:51 smithi194 ceph-mon[33017]: pgmap v18195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:51 smithi033 ceph-mon[27845]: pgmap v18195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:51 smithi096 ceph-mon[32753]: pgmap v18195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:10:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:54 smithi194 ceph-mon[33017]: pgmap v18196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33592.216113 secs 2024-04-26T21:10:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:54 smithi033 ceph-mon[27845]: pgmap v18196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33592.216113 secs 2024-04-26T21:10:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:54 smithi096 ceph-mon[32753]: pgmap v18196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:10:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33592.216113 secs 2024-04-26T21:10:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33595.161712 secs 2024-04-26T21:10:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33595.161712 secs 2024-04-26T21:10:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33595.161712 secs 2024-04-26T21:10:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:56 smithi194 ceph-mon[33017]: pgmap v18197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:56 smithi033 ceph-mon[27845]: pgmap v18197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:56 smithi096 ceph-mon[32753]: pgmap v18197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:57.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:57.432+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:57.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:57.432+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:10:57.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:10:57.432+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:10:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:58 smithi194 ceph-mon[33017]: pgmap v18198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:58 smithi033 ceph-mon[27845]: pgmap v18198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:58 smithi096 ceph-mon[32753]: pgmap v18198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:10:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33597.216175 secs 2024-04-26T21:10:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:10:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33600.161832 secs 2024-04-26T21:10:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33597.216175 secs 2024-04-26T21:10:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:10:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33600.161832 secs 2024-04-26T21:10:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33597.216175 secs 2024-04-26T21:10:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:10:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33600.161832 secs 2024-04-26T21:11:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:00 smithi194 ceph-mon[33017]: pgmap v18199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:00 smithi033 ceph-mon[27845]: pgmap v18199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:00 smithi096 ceph-mon[32753]: pgmap v18199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:02 smithi194 ceph-mon[33017]: pgmap v18200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:02 smithi033 ceph-mon[27845]: pgmap v18200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:02 smithi096 ceph-mon[32753]: pgmap v18200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:04 smithi194 ceph-mon[33017]: pgmap v18201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33602.216234 secs 2024-04-26T21:11:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33605.161965 secs 2024-04-26T21:11:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:04 smithi033 ceph-mon[27845]: pgmap v18201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33602.216234 secs 2024-04-26T21:11:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33605.161965 secs 2024-04-26T21:11:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:04 smithi096 ceph-mon[32753]: pgmap v18201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33602.216234 secs 2024-04-26T21:11:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33605.161965 secs 2024-04-26T21:11:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:06 smithi194 ceph-mon[33017]: pgmap v18202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:06 smithi033 ceph-mon[27845]: pgmap v18202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:06 smithi096 ceph-mon[32753]: pgmap v18202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:08 smithi194 ceph-mon[33017]: pgmap v18203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:08 smithi033 ceph-mon[27845]: pgmap v18203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:08 smithi096 ceph-mon[32753]: pgmap v18203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33607.216296 secs 2024-04-26T21:11:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33610.162081 secs 2024-04-26T21:11:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33607.216296 secs 2024-04-26T21:11:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33610.162081 secs 2024-04-26T21:11:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33607.216296 secs 2024-04-26T21:11:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33610.162081 secs 2024-04-26T21:11:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:10 smithi194 ceph-mon[33017]: pgmap v18204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:10 smithi033 ceph-mon[27845]: pgmap v18204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:10 smithi096 ceph-mon[32753]: pgmap v18204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:12.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:12 smithi194 ceph-mon[33017]: pgmap v18205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:12 smithi033 ceph-mon[27845]: pgmap v18205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:12 smithi096 ceph-mon[32753]: pgmap v18205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:12.448+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:12.448+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:12.448+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:11:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:14 smithi194 ceph-mon[33017]: pgmap v18206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33612.216355 secs 2024-04-26T21:11:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33615.162176 secs 2024-04-26T21:11:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:14 smithi033 ceph-mon[27845]: pgmap v18206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33612.216355 secs 2024-04-26T21:11:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33615.162176 secs 2024-04-26T21:11:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:14 smithi096 ceph-mon[32753]: pgmap v18206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33612.216355 secs 2024-04-26T21:11:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33615.162176 secs 2024-04-26T21:11:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:16 smithi194 ceph-mon[33017]: pgmap v18207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:16 smithi033 ceph-mon[27845]: pgmap v18207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:16 smithi096 ceph-mon[32753]: pgmap v18207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:18 smithi194 ceph-mon[33017]: pgmap v18208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:18 smithi033 ceph-mon[27845]: pgmap v18208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:18 smithi096 ceph-mon[32753]: pgmap v18208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:11:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33617.216438 secs 2024-04-26T21:11:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33620.162270 secs 2024-04-26T21:11:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33617.216438 secs 2024-04-26T21:11:19.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33620.162270 secs 2024-04-26T21:11:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33617.216438 secs 2024-04-26T21:11:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33620.162270 secs 2024-04-26T21:11:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:20 smithi033 ceph-mon[27845]: pgmap v18209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:20 smithi096 ceph-mon[32753]: pgmap v18209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:20 smithi194 ceph-mon[33017]: pgmap v18209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef006a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef006a040 comp rx=0 tx=0).stop 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:11:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.810+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:11:21.812 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.811+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:11:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:21.812+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef004f000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:11:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:22 smithi033 ceph-mon[27845]: pgmap v18210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:22 smithi096 ceph-mon[32753]: pgmap v18210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:22 smithi194 ceph-mon[33017]: pgmap v18210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33622.216518 secs 2024-04-26T21:11:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33622.216518 secs 2024-04-26T21:11:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33622.216518 secs 2024-04-26T21:11:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:24 smithi033 ceph-mon[27845]: pgmap v18211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33625.162400 secs 2024-04-26T21:11:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:24 smithi096 ceph-mon[32753]: pgmap v18211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33625.162400 secs 2024-04-26T21:11:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:24 smithi194 ceph-mon[33017]: pgmap v18211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33625.162400 secs 2024-04-26T21:11:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:26 smithi033 ceph-mon[27845]: pgmap v18212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:26 smithi096 ceph-mon[32753]: pgmap v18212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:26 smithi194 ceph-mon[33017]: pgmap v18212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:27.453+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:27.453+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:11:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:27.453+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:27.517+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:11:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:27.517+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:11:27.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:27.517+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 264 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T21:11:27.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:27.517+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 264 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:11:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:28 smithi033 ceph-mon[27845]: pgmap v18213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33627.216597 secs 2024-04-26T21:11:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:28 smithi096 ceph-mon[32753]: pgmap v18213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33627.216597 secs 2024-04-26T21:11:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:28 smithi194 ceph-mon[33017]: pgmap v18213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33627.216597 secs 2024-04-26T21:11:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33630.162465 secs 2024-04-26T21:11:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33630.162465 secs 2024-04-26T21:11:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33630.162465 secs 2024-04-26T21:11:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:30 smithi033 ceph-mon[27845]: pgmap v18214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:30 smithi096 ceph-mon[32753]: pgmap v18214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:30 smithi194 ceph-mon[33017]: pgmap v18214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:32 smithi033 ceph-mon[27845]: pgmap v18215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:32 smithi096 ceph-mon[32753]: pgmap v18215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:32 smithi194 ceph-mon[33017]: pgmap v18215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:11:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33632.216687 secs 2024-04-26T21:11:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:11:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33632.216687 secs 2024-04-26T21:11:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:11:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33632.216687 secs 2024-04-26T21:11:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:34 smithi033 ceph-mon[27845]: pgmap v18216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33635.162569 secs 2024-04-26T21:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:34 smithi096 ceph-mon[32753]: pgmap v18216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33635.162569 secs 2024-04-26T21:11:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:34 smithi194 ceph-mon[33017]: pgmap v18216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33635.162569 secs 2024-04-26T21:11:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:11:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:11:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:11:35.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:11:35.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:11:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:11:35.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:11:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:36 smithi194 ceph-mon[33017]: pgmap v18217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:36 smithi194 ceph-mon[33017]: pgmap v18218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:11:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:36 smithi194 ceph-mon[33017]: pgmap v18219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:11:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:36 smithi033 ceph-mon[27845]: pgmap v18217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:36 smithi033 ceph-mon[27845]: pgmap v18218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:11:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:36 smithi033 ceph-mon[27845]: pgmap v18219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:11:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:36 smithi096 ceph-mon[32753]: pgmap v18217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:36 smithi096 ceph-mon[32753]: pgmap v18218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:11:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:36 smithi096 ceph-mon[32753]: pgmap v18219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:11:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:38 smithi194 ceph-mon[33017]: pgmap v18220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:11:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33637.216771 secs 2024-04-26T21:11:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:38 smithi033 ceph-mon[27845]: pgmap v18220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:11:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33637.216771 secs 2024-04-26T21:11:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:38 smithi096 ceph-mon[32753]: pgmap v18220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:11:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33637.216771 secs 2024-04-26T21:11:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33640.162669 secs 2024-04-26T21:11:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33640.162669 secs 2024-04-26T21:11:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33640.162669 secs 2024-04-26T21:11:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:40 smithi194 ceph-mon[33017]: pgmap v18221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:11:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:40 smithi033 ceph-mon[27845]: pgmap v18221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:11:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:40 smithi096 ceph-mon[32753]: pgmap v18221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:11:42.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:42.456+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:42.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:42.456+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:11:42.458 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:42.456+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:42 smithi194 ceph-mon[33017]: pgmap v18222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:11:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:42 smithi033 ceph-mon[27845]: pgmap v18222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:11:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:42 smithi096 ceph-mon[32753]: pgmap v18222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:11:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33642.216851 secs 2024-04-26T21:11:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33642.216851 secs 2024-04-26T21:11:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33642.216851 secs 2024-04-26T21:11:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:44 smithi194 ceph-mon[33017]: pgmap v18223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:11:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33645.162795 secs 2024-04-26T21:11:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:44 smithi033 ceph-mon[27845]: pgmap v18223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:11:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33645.162795 secs 2024-04-26T21:11:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:44 smithi096 ceph-mon[32753]: pgmap v18223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:11:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33645.162795 secs 2024-04-26T21:11:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:45 smithi194 ceph-mon[33017]: pgmap v18224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:11:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:45 smithi033 ceph-mon[27845]: pgmap v18224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:11:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:45 smithi096 ceph-mon[32753]: pgmap v18224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:11:48.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:48 smithi194 ceph-mon[33017]: pgmap v18225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:11:48.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:48 smithi033 ceph-mon[27845]: pgmap v18225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:11:48.385 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:48 smithi096 ceph-mon[32753]: pgmap v18225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:11:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33647.216911 secs 2024-04-26T21:11:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33650.162925 secs 2024-04-26T21:11:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33647.216911 secs 2024-04-26T21:11:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33650.162925 secs 2024-04-26T21:11:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33647.216911 secs 2024-04-26T21:11:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33650.162925 secs 2024-04-26T21:11:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:50 smithi194 ceph-mon[33017]: pgmap v18226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:11:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:50 smithi033 ceph-mon[27845]: pgmap v18226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:11:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:50 smithi096 ceph-mon[32753]: pgmap v18226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:11:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:52 smithi033 ceph-mon[27845]: pgmap v18227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:11:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:52 smithi194 ceph-mon[33017]: pgmap v18227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:11:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:52 smithi096 ceph-mon[32753]: pgmap v18227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:11:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33652.216996 secs 2024-04-26T21:11:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:54 smithi194 ceph-mon[33017]: pgmap v18228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33655.163044 secs 2024-04-26T21:11:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33652.216996 secs 2024-04-26T21:11:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:54 smithi033 ceph-mon[27845]: pgmap v18228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33655.163044 secs 2024-04-26T21:11:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33652.216996 secs 2024-04-26T21:11:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:54 smithi096 ceph-mon[32753]: pgmap v18228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33655.163044 secs 2024-04-26T21:11:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:56 smithi194 ceph-mon[33017]: pgmap v18229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:56 smithi033 ceph-mon[27845]: pgmap v18229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:56 smithi096 ceph-mon[32753]: pgmap v18229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:11:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:57.461+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:57.461+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:11:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:11:57.461+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:11:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:58 smithi194 ceph-mon[33017]: pgmap v18230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:58 smithi033 ceph-mon[27845]: pgmap v18230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:58 smithi096 ceph-mon[32753]: pgmap v18230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:11:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33657.217078 secs 2024-04-26T21:11:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33660.163188 secs 2024-04-26T21:11:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33657.217078 secs 2024-04-26T21:11:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33660.163188 secs 2024-04-26T21:11:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33657.217078 secs 2024-04-26T21:11:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33660.163188 secs 2024-04-26T21:12:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:00 smithi194 ceph-mon[33017]: pgmap v18231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:00 smithi033 ceph-mon[27845]: pgmap v18231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:00 smithi096 ceph-mon[32753]: pgmap v18231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:02 smithi194 ceph-mon[33017]: pgmap v18232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:02 smithi033 ceph-mon[27845]: pgmap v18232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:02 smithi096 ceph-mon[32753]: pgmap v18232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33662.217136 secs 2024-04-26T21:12:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33662.217136 secs 2024-04-26T21:12:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33662.217136 secs 2024-04-26T21:12:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:04 smithi033 ceph-mon[27845]: pgmap v18233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33665.163289 secs 2024-04-26T21:12:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:04 smithi096 ceph-mon[32753]: pgmap v18233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33665.163289 secs 2024-04-26T21:12:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:04 smithi194 ceph-mon[33017]: pgmap v18233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33665.163289 secs 2024-04-26T21:12:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:06 smithi033 ceph-mon[27845]: pgmap v18234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:06 smithi096 ceph-mon[32753]: pgmap v18234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:06 smithi194 ceph-mon[33017]: pgmap v18234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:08 smithi033 ceph-mon[27845]: pgmap v18235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33667.217198 secs 2024-04-26T21:12:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:08 smithi096 ceph-mon[32753]: pgmap v18235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33667.217198 secs 2024-04-26T21:12:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:08 smithi194 ceph-mon[33017]: pgmap v18235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33667.217198 secs 2024-04-26T21:12:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33670.163418 secs 2024-04-26T21:12:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33670.163418 secs 2024-04-26T21:12:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33670.163418 secs 2024-04-26T21:12:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:10 smithi033 ceph-mon[27845]: pgmap v18236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:10 smithi096 ceph-mon[32753]: pgmap v18236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:10 smithi194 ceph-mon[33017]: pgmap v18236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:12.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:12 smithi033 ceph-mon[27845]: pgmap v18237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:12 smithi096 ceph-mon[32753]: pgmap v18237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:12.476+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:12.476+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:12:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:12.476+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:12 smithi194 ceph-mon[33017]: pgmap v18237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33672.217274 secs 2024-04-26T21:12:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33672.217274 secs 2024-04-26T21:12:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33672.217274 secs 2024-04-26T21:12:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:14 smithi033 ceph-mon[27845]: pgmap v18238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33675.163565 secs 2024-04-26T21:12:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:14 smithi096 ceph-mon[32753]: pgmap v18238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33675.163565 secs 2024-04-26T21:12:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:14 smithi194 ceph-mon[33017]: pgmap v18238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33675.163565 secs 2024-04-26T21:12:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:16 smithi033 ceph-mon[27845]: pgmap v18239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:16 smithi096 ceph-mon[32753]: pgmap v18239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:16 smithi194 ceph-mon[33017]: pgmap v18239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:18 smithi033 ceph-mon[27845]: pgmap v18240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33677.217335 secs 2024-04-26T21:12:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:18 smithi096 ceph-mon[32753]: pgmap v18240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33677.217335 secs 2024-04-26T21:12:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:18 smithi194 ceph-mon[33017]: pgmap v18240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33677.217335 secs 2024-04-26T21:12:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33680.163706 secs 2024-04-26T21:12:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33680.163706 secs 2024-04-26T21:12:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33680.163706 secs 2024-04-26T21:12:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:20 smithi096 ceph-mon[32753]: pgmap v18241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:20 smithi194 ceph-mon[33017]: pgmap v18241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:20 smithi033 ceph-mon[27845]: pgmap v18241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:22 smithi194 ceph-mon[33017]: pgmap v18242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:22 smithi033 ceph-mon[27845]: pgmap v18242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:22 smithi096 ceph-mon[32753]: pgmap v18242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33682.217411 secs 2024-04-26T21:12:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33682.217411 secs 2024-04-26T21:12:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33682.217411 secs 2024-04-26T21:12:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:24 smithi194 ceph-mon[33017]: pgmap v18243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33685.163835 secs 2024-04-26T21:12:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:24 smithi033 ceph-mon[27845]: pgmap v18243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33685.163835 secs 2024-04-26T21:12:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:24 smithi096 ceph-mon[32753]: pgmap v18243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33685.163835 secs 2024-04-26T21:12:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:26 smithi194 ceph-mon[33017]: pgmap v18244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:26 smithi033 ceph-mon[27845]: pgmap v18244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:26 smithi096 ceph-mon[32753]: pgmap v18244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:27.480+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:27.480+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:27.480+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:12:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:28 smithi194 ceph-mon[33017]: pgmap v18245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33687.217455 secs 2024-04-26T21:12:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:28 smithi033 ceph-mon[27845]: pgmap v18245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33687.217455 secs 2024-04-26T21:12:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:28 smithi096 ceph-mon[32753]: pgmap v18245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33687.217455 secs 2024-04-26T21:12:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33690.163980 secs 2024-04-26T21:12:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33690.163980 secs 2024-04-26T21:12:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33690.163980 secs 2024-04-26T21:12:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:30 smithi194 ceph-mon[33017]: pgmap v18246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:30 smithi033 ceph-mon[27845]: pgmap v18246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:30 smithi096 ceph-mon[32753]: pgmap v18246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:32 smithi194 ceph-mon[33017]: pgmap v18247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:32 smithi033 ceph-mon[27845]: pgmap v18247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:32 smithi096 ceph-mon[32753]: pgmap v18247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33692.217536 secs 2024-04-26T21:12:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33692.217536 secs 2024-04-26T21:12:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33692.217536 secs 2024-04-26T21:12:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:34 smithi194 ceph-mon[33017]: pgmap v18248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33695.164147 secs 2024-04-26T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:34 smithi033 ceph-mon[27845]: pgmap v18248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33695.164147 secs 2024-04-26T21:12:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:34 smithi096 ceph-mon[32753]: pgmap v18248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33695.164147 secs 2024-04-26T21:12:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:35 smithi194 ceph-mon[33017]: pgmap v18249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:35 smithi033 ceph-mon[27845]: pgmap v18249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:35 smithi096 ceph-mon[32753]: pgmap v18249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:12:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:12:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:12:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:12:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:12:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:37 smithi194 ceph-mon[33017]: pgmap v18250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:37 smithi194 ceph-mon[33017]: pgmap v18251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:12:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:12:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:12:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:37 smithi033 ceph-mon[27845]: pgmap v18250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:37 smithi033 ceph-mon[27845]: pgmap v18251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:12:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:12:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:12:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:37 smithi096 ceph-mon[32753]: pgmap v18250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:37 smithi096 ceph-mon[32753]: pgmap v18251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:37.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:12:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33697.217618 secs 2024-04-26T21:12:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33697.217618 secs 2024-04-26T21:12:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33697.217618 secs 2024-04-26T21:12:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:39 smithi194 ceph-mon[33017]: pgmap v18252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33700.164266 secs 2024-04-26T21:12:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:39 smithi033 ceph-mon[27845]: pgmap v18252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:39.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33700.164266 secs 2024-04-26T21:12:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:39 smithi096 ceph-mon[32753]: pgmap v18252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33700.164266 secs 2024-04-26T21:12:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:41 smithi194 ceph-mon[33017]: pgmap v18253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:41 smithi033 ceph-mon[27845]: pgmap v18253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:41 smithi096 ceph-mon[32753]: pgmap v18253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:42.485+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:42.485+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:12:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:42.485+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:43 smithi194 ceph-mon[33017]: pgmap v18254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33702.217700 secs 2024-04-26T21:12:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:43 smithi033 ceph-mon[27845]: pgmap v18254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33702.217700 secs 2024-04-26T21:12:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:43 smithi096 ceph-mon[32753]: pgmap v18254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33702.217700 secs 2024-04-26T21:12:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33705.164398 secs 2024-04-26T21:12:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33705.164398 secs 2024-04-26T21:12:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33705.164398 secs 2024-04-26T21:12:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:45 smithi194 ceph-mon[33017]: pgmap v18255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:45 smithi033 ceph-mon[27845]: pgmap v18255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:45 smithi096 ceph-mon[32753]: pgmap v18255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:12:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:47 smithi194 ceph-mon[33017]: pgmap v18256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:12:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:47 smithi033 ceph-mon[27845]: pgmap v18256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:12:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:47 smithi096 ceph-mon[32753]: pgmap v18256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:12:49.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33707.217783 secs 2024-04-26T21:12:49.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33707.217783 secs 2024-04-26T21:12:49.086 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33707.217783 secs 2024-04-26T21:12:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:49 smithi194 ceph-mon[33017]: pgmap v18257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33710.164486 secs 2024-04-26T21:12:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:49 smithi033 ceph-mon[27845]: pgmap v18257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33710.164486 secs 2024-04-26T21:12:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:49 smithi096 ceph-mon[32753]: pgmap v18257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33710.164486 secs 2024-04-26T21:12:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:51 smithi194 ceph-mon[33017]: pgmap v18258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:51 smithi033 ceph-mon[27845]: pgmap v18258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:51 smithi096 ceph-mon[32753]: pgmap v18258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:53 smithi194 ceph-mon[33017]: pgmap v18259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33712.217863 secs 2024-04-26T21:12:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:53 smithi033 ceph-mon[27845]: pgmap v18259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33712.217863 secs 2024-04-26T21:12:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:53 smithi096 ceph-mon[32753]: pgmap v18259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33712.217863 secs 2024-04-26T21:12:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33715.164610 secs 2024-04-26T21:12:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33715.164610 secs 2024-04-26T21:12:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33715.164610 secs 2024-04-26T21:12:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:55 smithi194 ceph-mon[33017]: pgmap v18260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:55 smithi033 ceph-mon[27845]: pgmap v18260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:55 smithi096 ceph-mon[32753]: pgmap v18260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:12:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:57.501+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:57.501+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:12:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:12:57.501+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:12:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:57 smithi194 ceph-mon[33017]: pgmap v18261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:57 smithi033 ceph-mon[27845]: pgmap v18261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:57 smithi096 ceph-mon[32753]: pgmap v18261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:12:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33717.217948 secs 2024-04-26T21:12:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33717.217948 secs 2024-04-26T21:12:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33717.217948 secs 2024-04-26T21:13:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:59 smithi194 ceph-mon[33017]: pgmap v18262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:12:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33720.164682 secs 2024-04-26T21:13:00.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:59 smithi033 ceph-mon[27845]: pgmap v18262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:00.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:12:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33720.164682 secs 2024-04-26T21:13:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:59 smithi096 ceph-mon[32753]: pgmap v18262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:00.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:12:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33720.164682 secs 2024-04-26T21:13:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:01 smithi194 ceph-mon[33017]: pgmap v18263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:01 smithi033 ceph-mon[27845]: pgmap v18263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:01 smithi096 ceph-mon[32753]: pgmap v18263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:03 smithi194 ceph-mon[33017]: pgmap v18264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33722.218000 secs 2024-04-26T21:13:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:03 smithi033 ceph-mon[27845]: pgmap v18264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33722.218000 secs 2024-04-26T21:13:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:03 smithi096 ceph-mon[32753]: pgmap v18264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33722.218000 secs 2024-04-26T21:13:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33725.164817 secs 2024-04-26T21:13:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33725.164817 secs 2024-04-26T21:13:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33725.164817 secs 2024-04-26T21:13:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:05 smithi194 ceph-mon[33017]: pgmap v18265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:05 smithi033 ceph-mon[27845]: pgmap v18265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:05 smithi096 ceph-mon[32753]: pgmap v18265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:07 smithi194 ceph-mon[33017]: pgmap v18266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:07 smithi033 ceph-mon[27845]: pgmap v18266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:07 smithi096 ceph-mon[32753]: pgmap v18266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33727.218063 secs 2024-04-26T21:13:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33727.218063 secs 2024-04-26T21:13:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33727.218063 secs 2024-04-26T21:13:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:09 smithi194 ceph-mon[33017]: pgmap v18267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33730.164878 secs 2024-04-26T21:13:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:09 smithi033 ceph-mon[27845]: pgmap v18267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33730.164878 secs 2024-04-26T21:13:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:09 smithi096 ceph-mon[32753]: pgmap v18267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33730.164878 secs 2024-04-26T21:13:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:11 smithi033 ceph-mon[27845]: pgmap v18268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:11 smithi096 ceph-mon[32753]: pgmap v18268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:11 smithi194 ceph-mon[33017]: pgmap v18268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:12.517+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:12.517+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:12.517+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:13:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:13 smithi033 ceph-mon[27845]: pgmap v18269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33732.218121 secs 2024-04-26T21:13:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:13 smithi096 ceph-mon[32753]: pgmap v18269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33732.218121 secs 2024-04-26T21:13:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:13 smithi194 ceph-mon[33017]: pgmap v18269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33732.218121 secs 2024-04-26T21:13:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33735.164993 secs 2024-04-26T21:13:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33735.164993 secs 2024-04-26T21:13:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33735.164993 secs 2024-04-26T21:13:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:15 smithi033 ceph-mon[27845]: pgmap v18270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:15 smithi096 ceph-mon[32753]: pgmap v18270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:15 smithi194 ceph-mon[33017]: pgmap v18270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:17 smithi033 ceph-mon[27845]: pgmap v18271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:17 smithi096 ceph-mon[32753]: pgmap v18271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:17 smithi194 ceph-mon[33017]: pgmap v18271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33737.218182 secs 2024-04-26T21:13:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33737.218182 secs 2024-04-26T21:13:19.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33737.218182 secs 2024-04-26T21:13:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:19 smithi033 ceph-mon[27845]: pgmap v18272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33740.165128 secs 2024-04-26T21:13:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:19 smithi096 ceph-mon[32753]: pgmap v18272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33740.165128 secs 2024-04-26T21:13:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:19 smithi194 ceph-mon[33017]: pgmap v18272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33740.165128 secs 2024-04-26T21:13:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:21 smithi033 ceph-mon[27845]: pgmap v18273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:21 smithi096 ceph-mon[32753]: pgmap v18273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:21 smithi194 ceph-mon[33017]: pgmap v18273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:23 smithi033 ceph-mon[27845]: pgmap v18274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33742.218241 secs 2024-04-26T21:13:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:23 smithi096 ceph-mon[32753]: pgmap v18274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33742.218241 secs 2024-04-26T21:13:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:23 smithi194 ceph-mon[33017]: pgmap v18274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33742.218241 secs 2024-04-26T21:13:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33745.165202 secs 2024-04-26T21:13:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33745.165202 secs 2024-04-26T21:13:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33745.165202 secs 2024-04-26T21:13:26.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:25 smithi033 ceph-mon[27845]: pgmap v18275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:25 smithi096 ceph-mon[32753]: pgmap v18275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:25 smithi194 ceph-mon[33017]: pgmap v18275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:27.531+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:27.531+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:27.532+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:13:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:27 smithi096 ceph-mon[32753]: pgmap v18276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:27 smithi194 ceph-mon[33017]: pgmap v18276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:27 smithi033 ceph-mon[27845]: pgmap v18276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33747.218302 secs 2024-04-26T21:13:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33747.218302 secs 2024-04-26T21:13:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33747.218302 secs 2024-04-26T21:13:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:29 smithi194 ceph-mon[33017]: pgmap v18277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33750.165265 secs 2024-04-26T21:13:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:29 smithi033 ceph-mon[27845]: pgmap v18277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33750.165265 secs 2024-04-26T21:13:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:29 smithi096 ceph-mon[32753]: pgmap v18277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33750.165265 secs 2024-04-26T21:13:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:31 smithi194 ceph-mon[33017]: pgmap v18278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:31 smithi033 ceph-mon[27845]: pgmap v18278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:31 smithi096 ceph-mon[32753]: pgmap v18278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:33 smithi194 ceph-mon[33017]: pgmap v18279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33752.218363 secs 2024-04-26T21:13:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:33 smithi033 ceph-mon[27845]: pgmap v18279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33752.218363 secs 2024-04-26T21:13:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:33 smithi096 ceph-mon[32753]: pgmap v18279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33752.218363 secs 2024-04-26T21:13:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33755.165381 secs 2024-04-26T21:13:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33755.165381 secs 2024-04-26T21:13:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33755.165381 secs 2024-04-26T21:13:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:35 smithi194 ceph-mon[33017]: pgmap v18280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:35 smithi033 ceph-mon[27845]: pgmap v18280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:35 smithi096 ceph-mon[32753]: pgmap v18280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:38.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:37 smithi033 ceph-mon[27845]: pgmap v18281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:38.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:13:38.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.331 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:37 smithi194 ceph-mon[33017]: pgmap v18281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:13:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:37 smithi096 ceph-mon[32753]: pgmap v18281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:13:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:38.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33757.218568 secs 2024-04-26T21:13:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33757.218568 secs 2024-04-26T21:13:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33757.218568 secs 2024-04-26T21:13:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:39 smithi194 ceph-mon[33017]: pgmap v18282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33760.165462 secs 2024-04-26T21:13:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:39 smithi033 ceph-mon[27845]: pgmap v18282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33760.165462 secs 2024-04-26T21:13:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:39 smithi096 ceph-mon[32753]: pgmap v18282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33760.165462 secs 2024-04-26T21:13:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:40.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:41.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:41 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:13:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:41 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:13:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:41 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:13:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:41 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:13:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:41 smithi194 ceph-mon[33017]: pgmap v18283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:41 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:13:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:41 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:13:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:41 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:13:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:41 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:13:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:41 smithi033 ceph-mon[27845]: pgmap v18283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:42.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:41 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:13:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:41 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:13:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:41 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:13:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:41 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:13:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:41 smithi096 ceph-mon[32753]: pgmap v18283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:42.547+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:42.547+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:13:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:42.547+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:43.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: pgmap v18284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33762.218666 secs 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: pgmap v18285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:13:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: pgmap v18286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:13:44.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: pgmap v18284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33762.218666 secs 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: pgmap v18285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: pgmap v18286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:13:44.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: pgmap v18284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33762.218666 secs 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: pgmap v18285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: pgmap v18286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:13:44.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:13:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33765.165606 secs 2024-04-26T21:13:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33765.165606 secs 2024-04-26T21:13:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33765.165606 secs 2024-04-26T21:13:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:46 smithi194 ceph-mon[33017]: pgmap v18287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:46 smithi033 ceph-mon[27845]: pgmap v18287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:46 smithi096 ceph-mon[32753]: pgmap v18287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:48 smithi194 ceph-mon[33017]: pgmap v18288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:48 smithi033 ceph-mon[27845]: pgmap v18288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:48 smithi096 ceph-mon[32753]: pgmap v18288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:49.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33767.218754 secs 2024-04-26T21:13:49.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33767.218754 secs 2024-04-26T21:13:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33767.218754 secs 2024-04-26T21:13:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33770.165753 secs 2024-04-26T21:13:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:50 smithi194 ceph-mon[33017]: pgmap v18289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:13:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33770.165753 secs 2024-04-26T21:13:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:50 smithi033 ceph-mon[27845]: pgmap v18289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:13:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33770.165753 secs 2024-04-26T21:13:50.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:50 smithi096 ceph-mon[32753]: pgmap v18289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:13:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:52 smithi194 ceph-mon[33017]: pgmap v18290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:52 smithi033 ceph-mon[27845]: pgmap v18290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:52 smithi096 ceph-mon[32753]: pgmap v18290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:13:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33772.218860 secs 2024-04-26T21:13:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:54 smithi194 ceph-mon[33017]: pgmap v18291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:13:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33775.165869 secs 2024-04-26T21:13:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33772.218860 secs 2024-04-26T21:13:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:54 smithi033 ceph-mon[27845]: pgmap v18291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:13:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33775.165869 secs 2024-04-26T21:13:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33772.218860 secs 2024-04-26T21:13:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:54 smithi096 ceph-mon[32753]: pgmap v18291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:13:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33775.165869 secs 2024-04-26T21:13:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:56 smithi194 ceph-mon[33017]: pgmap v18292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:56 smithi033 ceph-mon[27845]: pgmap v18292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:56 smithi096 ceph-mon[32753]: pgmap v18292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:13:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:57.563+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:57.563+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:13:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:13:57.563+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:13:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:58 smithi194 ceph-mon[33017]: pgmap v18293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:58 smithi033 ceph-mon[27845]: pgmap v18293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:58 smithi096 ceph-mon[32753]: pgmap v18293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:13:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33777.218962 secs 2024-04-26T21:13:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:13:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33780.165978 secs 2024-04-26T21:13:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33777.218962 secs 2024-04-26T21:13:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:13:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33780.165978 secs 2024-04-26T21:13:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33777.218962 secs 2024-04-26T21:13:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:13:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33780.165978 secs 2024-04-26T21:14:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:00 smithi194 ceph-mon[33017]: pgmap v18294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:00 smithi033 ceph-mon[27845]: pgmap v18294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:00 smithi096 ceph-mon[32753]: pgmap v18294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:02 smithi194 ceph-mon[33017]: pgmap v18295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:02 smithi033 ceph-mon[27845]: pgmap v18295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:02 smithi096 ceph-mon[32753]: pgmap v18295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33782.219060 secs 2024-04-26T21:14:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:04 smithi194 ceph-mon[33017]: pgmap v18296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33785.166094 secs 2024-04-26T21:14:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33782.219060 secs 2024-04-26T21:14:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:04 smithi033 ceph-mon[27845]: pgmap v18296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33785.166094 secs 2024-04-26T21:14:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33782.219060 secs 2024-04-26T21:14:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:04 smithi096 ceph-mon[32753]: pgmap v18296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33785.166094 secs 2024-04-26T21:14:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:06 smithi194 ceph-mon[33017]: pgmap v18297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:06 smithi033 ceph-mon[27845]: pgmap v18297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:06 smithi096 ceph-mon[32753]: pgmap v18297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:08 smithi194 ceph-mon[33017]: pgmap v18298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:08 smithi033 ceph-mon[27845]: pgmap v18298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:08 smithi096 ceph-mon[32753]: pgmap v18298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33787.219161 secs 2024-04-26T21:14:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33790.166205 secs 2024-04-26T21:14:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33787.219161 secs 2024-04-26T21:14:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33790.166205 secs 2024-04-26T21:14:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33787.219161 secs 2024-04-26T21:14:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33790.166205 secs 2024-04-26T21:14:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:10 smithi033 ceph-mon[27845]: pgmap v18299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:10 smithi096 ceph-mon[32753]: pgmap v18299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:10 smithi194 ceph-mon[33017]: pgmap v18299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:12 smithi033 ceph-mon[27845]: pgmap v18300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:12 smithi096 ceph-mon[32753]: pgmap v18300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:12.565+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:12.565+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:14:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:12.565+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:12 smithi194 ceph-mon[33017]: pgmap v18300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33792.219262 secs 2024-04-26T21:14:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33792.219262 secs 2024-04-26T21:14:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33792.219262 secs 2024-04-26T21:14:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:14 smithi033 ceph-mon[27845]: pgmap v18301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33795.166268 secs 2024-04-26T21:14:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:14 smithi096 ceph-mon[32753]: pgmap v18301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33795.166268 secs 2024-04-26T21:14:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:14 smithi194 ceph-mon[33017]: pgmap v18301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33795.166268 secs 2024-04-26T21:14:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:16 smithi033 ceph-mon[27845]: pgmap v18302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:16 smithi096 ceph-mon[32753]: pgmap v18302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:16 smithi194 ceph-mon[33017]: pgmap v18302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:18 smithi033 ceph-mon[27845]: pgmap v18303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33797.219320 secs 2024-04-26T21:14:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:18 smithi096 ceph-mon[32753]: pgmap v18303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33797.219320 secs 2024-04-26T21:14:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:18 smithi194 ceph-mon[33017]: pgmap v18303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33797.219320 secs 2024-04-26T21:14:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33800.166345 secs 2024-04-26T21:14:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33800.166345 secs 2024-04-26T21:14:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33800.166345 secs 2024-04-26T21:14:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:20 smithi033 ceph-mon[27845]: pgmap v18304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:20 smithi096 ceph-mon[32753]: pgmap v18304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:20 smithi194 ceph-mon[33017]: pgmap v18304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:22 smithi033 ceph-mon[27845]: pgmap v18305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:22 smithi096 ceph-mon[32753]: pgmap v18305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:22 smithi194 ceph-mon[33017]: pgmap v18305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33802.219403 secs 2024-04-26T21:14:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33802.219403 secs 2024-04-26T21:14:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33802.219403 secs 2024-04-26T21:14:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:24 smithi033 ceph-mon[27845]: pgmap v18306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33805.166462 secs 2024-04-26T21:14:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:24 smithi096 ceph-mon[32753]: pgmap v18306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33805.166462 secs 2024-04-26T21:14:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:24 smithi194 ceph-mon[33017]: pgmap v18306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33805.166462 secs 2024-04-26T21:14:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:26 smithi096 ceph-mon[32753]: pgmap v18307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:26 smithi194 ceph-mon[33017]: pgmap v18307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:26 smithi033 ceph-mon[27845]: pgmap v18307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:27.548+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T21:14:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:27.548+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:14:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:27.549+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 265 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T21:14:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:27.549+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 265 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:14:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:27.578+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:27.578+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:27.578+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:14:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:28 smithi033 ceph-mon[27845]: pgmap v18308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33807.219489 secs 2024-04-26T21:14:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:28 smithi194 ceph-mon[33017]: pgmap v18308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33807.219489 secs 2024-04-26T21:14:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:28 smithi096 ceph-mon[32753]: pgmap v18308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33807.219489 secs 2024-04-26T21:14:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33810.166578 secs 2024-04-26T21:14:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33810.166578 secs 2024-04-26T21:14:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33810.166578 secs 2024-04-26T21:14:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:30 smithi194 ceph-mon[33017]: pgmap v18309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:30 smithi033 ceph-mon[27845]: pgmap v18309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:30 smithi096 ceph-mon[32753]: pgmap v18309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:32 smithi194 ceph-mon[33017]: pgmap v18310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:32 smithi033 ceph-mon[27845]: pgmap v18310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:32 smithi096 ceph-mon[32753]: pgmap v18310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33812.219571 secs 2024-04-26T21:14:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33812.219571 secs 2024-04-26T21:14:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33812.219571 secs 2024-04-26T21:14:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:34 smithi194 ceph-mon[33017]: pgmap v18311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33815.166712 secs 2024-04-26T21:14:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:34 smithi033 ceph-mon[27845]: pgmap v18311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33815.166712 secs 2024-04-26T21:14:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:34 smithi096 ceph-mon[32753]: pgmap v18311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33815.166712 secs 2024-04-26T21:14:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:36 smithi194 ceph-mon[33017]: pgmap v18312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:36 smithi033 ceph-mon[27845]: pgmap v18312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:36 smithi096 ceph-mon[32753]: pgmap v18312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:38 smithi194 ceph-mon[33017]: pgmap v18313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33817.219650 secs 2024-04-26T21:14:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:38 smithi033 ceph-mon[27845]: pgmap v18313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33817.219650 secs 2024-04-26T21:14:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:38 smithi096 ceph-mon[32753]: pgmap v18313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33817.219650 secs 2024-04-26T21:14:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33820.166846 secs 2024-04-26T21:14:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33820.166846 secs 2024-04-26T21:14:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33820.166846 secs 2024-04-26T21:14:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:40 smithi194 ceph-mon[33017]: pgmap v18314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:40 smithi033 ceph-mon[27845]: pgmap v18314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:40 smithi096 ceph-mon[32753]: pgmap v18314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:42.592+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:42.592+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:42.592+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:14:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:42 smithi194 ceph-mon[33017]: pgmap v18315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:42.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:42 smithi033 ceph-mon[27845]: pgmap v18315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:42 smithi096 ceph-mon[32753]: pgmap v18315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:14:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33822.219732 secs 2024-04-26T21:14:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33822.219732 secs 2024-04-26T21:14:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33822.219732 secs 2024-04-26T21:14:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:44 smithi194 ceph-mon[33017]: pgmap v18316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:14:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33825.166937 secs 2024-04-26T21:14:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:44 smithi033 ceph-mon[27845]: pgmap v18316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:14:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33825.166937 secs 2024-04-26T21:14:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:44 smithi096 ceph-mon[32753]: pgmap v18316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:14:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33825.166937 secs 2024-04-26T21:14:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:14:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:14:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:45 smithi194 ceph-mon[33017]: pgmap v18317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:14:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:45 smithi194 ceph-mon[33017]: pgmap v18318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:14:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:14:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:14:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:45 smithi033 ceph-mon[27845]: pgmap v18317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:14:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:45 smithi033 ceph-mon[27845]: pgmap v18318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:14:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:14:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:14:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:45 smithi096 ceph-mon[32753]: pgmap v18317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:14:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:45 smithi096 ceph-mon[32753]: pgmap v18318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:45.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:14:48.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:47 smithi194 ceph-mon[33017]: pgmap v18319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:47 smithi033 ceph-mon[27845]: pgmap v18319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:47 smithi096 ceph-mon[32753]: pgmap v18319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:49.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33827.219794 secs 2024-04-26T21:14:49.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33827.219794 secs 2024-04-26T21:14:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33827.219794 secs 2024-04-26T21:14:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:49 smithi194 ceph-mon[33017]: pgmap v18320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33830.167052 secs 2024-04-26T21:14:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:49 smithi033 ceph-mon[27845]: pgmap v18320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33830.167052 secs 2024-04-26T21:14:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:49 smithi096 ceph-mon[32753]: pgmap v18320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33830.167052 secs 2024-04-26T21:14:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:51 smithi194 ceph-mon[33017]: pgmap v18321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:51 smithi033 ceph-mon[27845]: pgmap v18321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:51 smithi096 ceph-mon[32753]: pgmap v18321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:53 smithi194 ceph-mon[33017]: pgmap v18322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33832.219875 secs 2024-04-26T21:14:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:53 smithi033 ceph-mon[27845]: pgmap v18322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33832.219875 secs 2024-04-26T21:14:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:53 smithi096 ceph-mon[32753]: pgmap v18322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:14:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33832.219875 secs 2024-04-26T21:14:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33835.167174 secs 2024-04-26T21:14:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33835.167174 secs 2024-04-26T21:14:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33835.167174 secs 2024-04-26T21:14:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:55 smithi194 ceph-mon[33017]: pgmap v18323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:14:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:55 smithi033 ceph-mon[27845]: pgmap v18323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:14:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:55 smithi096 ceph-mon[32753]: pgmap v18323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:14:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:57.609+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:57.609+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:14:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:14:57.609+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:14:58.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:57 smithi194 ceph-mon[33017]: pgmap v18324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:58.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:57 smithi033 ceph-mon[27845]: pgmap v18324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:57 smithi096 ceph-mon[32753]: pgmap v18324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:14:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33837.219936 secs 2024-04-26T21:14:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33837.219936 secs 2024-04-26T21:14:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33837.219936 secs 2024-04-26T21:15:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:59 smithi194 ceph-mon[33017]: pgmap v18325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33840.167283 secs 2024-04-26T21:15:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:59 smithi033 ceph-mon[27845]: pgmap v18325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33840.167283 secs 2024-04-26T21:15:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:59 smithi096 ceph-mon[32753]: pgmap v18325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33840.167283 secs 2024-04-26T21:15:02.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:01 smithi194 ceph-mon[33017]: pgmap v18326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:02.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:01 smithi033 ceph-mon[27845]: pgmap v18326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:01 smithi096 ceph-mon[32753]: pgmap v18326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:03 smithi194 ceph-mon[33017]: pgmap v18327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33842.219994 secs 2024-04-26T21:15:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:03 smithi033 ceph-mon[27845]: pgmap v18327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33842.219994 secs 2024-04-26T21:15:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:03 smithi096 ceph-mon[32753]: pgmap v18327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33842.219994 secs 2024-04-26T21:15:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33845.167379 secs 2024-04-26T21:15:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33845.167379 secs 2024-04-26T21:15:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33845.167379 secs 2024-04-26T21:15:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:05 smithi033 ceph-mon[27845]: pgmap v18328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:05 smithi096 ceph-mon[32753]: pgmap v18328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:05 smithi194 ceph-mon[33017]: pgmap v18328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:08.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:07 smithi033 ceph-mon[27845]: pgmap v18329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:07 smithi096 ceph-mon[32753]: pgmap v18329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:07 smithi194 ceph-mon[33017]: pgmap v18329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33847.220057 secs 2024-04-26T21:15:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33847.220057 secs 2024-04-26T21:15:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33847.220057 secs 2024-04-26T21:15:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:09 smithi033 ceph-mon[27845]: pgmap v18330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33850.167488 secs 2024-04-26T21:15:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:09 smithi096 ceph-mon[32753]: pgmap v18330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33850.167488 secs 2024-04-26T21:15:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:09 smithi194 ceph-mon[33017]: pgmap v18330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33850.167488 secs 2024-04-26T21:15:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:11 smithi033 ceph-mon[27845]: pgmap v18331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:11 smithi096 ceph-mon[32753]: pgmap v18331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:11 smithi194 ceph-mon[33017]: pgmap v18331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:12.617+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:12.617+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:12.617+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:15:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:13 smithi033 ceph-mon[27845]: pgmap v18332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33852.220115 secs 2024-04-26T21:15:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:13 smithi096 ceph-mon[32753]: pgmap v18332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33852.220115 secs 2024-04-26T21:15:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:13 smithi194 ceph-mon[33017]: pgmap v18332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33852.220115 secs 2024-04-26T21:15:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33855.167608 secs 2024-04-26T21:15:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33855.167608 secs 2024-04-26T21:15:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33855.167608 secs 2024-04-26T21:15:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:15 smithi033 ceph-mon[27845]: pgmap v18333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:15 smithi096 ceph-mon[32753]: pgmap v18333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:15 smithi194 ceph-mon[33017]: pgmap v18333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:18.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:17 smithi033 ceph-mon[27845]: pgmap v18334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:17 smithi096 ceph-mon[32753]: pgmap v18334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:17 smithi194 ceph-mon[33017]: pgmap v18334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33857.220203 secs 2024-04-26T21:15:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33857.220203 secs 2024-04-26T21:15:19.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33857.220203 secs 2024-04-26T21:15:20.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:19 smithi033 ceph-mon[27845]: pgmap v18335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:20.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33860.167725 secs 2024-04-26T21:15:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:19 smithi096 ceph-mon[32753]: pgmap v18335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33860.167725 secs 2024-04-26T21:15:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:19 smithi194 ceph-mon[33017]: pgmap v18335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33860.167725 secs 2024-04-26T21:15:22.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:21 smithi033 ceph-mon[27845]: pgmap v18336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:21 smithi096 ceph-mon[32753]: pgmap v18336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:21 smithi194 ceph-mon[33017]: pgmap v18336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:23 smithi033 ceph-mon[27845]: pgmap v18337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33862.220283 secs 2024-04-26T21:15:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:23 smithi096 ceph-mon[32753]: pgmap v18337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33862.220283 secs 2024-04-26T21:15:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:23 smithi194 ceph-mon[33017]: pgmap v18337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33862.220283 secs 2024-04-26T21:15:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33865.167859 secs 2024-04-26T21:15:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33865.167859 secs 2024-04-26T21:15:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33865.167859 secs 2024-04-26T21:15:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:25 smithi194 ceph-mon[33017]: pgmap v18338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:25 smithi033 ceph-mon[27845]: pgmap v18338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:25 smithi096 ceph-mon[32753]: pgmap v18338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:27.626+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:27.626+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:27.626+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:15:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:27 smithi194 ceph-mon[33017]: pgmap v18339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:27 smithi033 ceph-mon[27845]: pgmap v18339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:28.624 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:27 smithi096 ceph-mon[32753]: pgmap v18339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33867.220346 secs 2024-04-26T21:15:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33867.220346 secs 2024-04-26T21:15:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33867.220346 secs 2024-04-26T21:15:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:29 smithi033 ceph-mon[27845]: pgmap v18340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:30.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33870.167992 secs 2024-04-26T21:15:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:29 smithi194 ceph-mon[33017]: pgmap v18340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33870.167992 secs 2024-04-26T21:15:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:29 smithi096 ceph-mon[32753]: pgmap v18340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33870.167992 secs 2024-04-26T21:15:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:31 smithi194 ceph-mon[33017]: pgmap v18341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:31 smithi033 ceph-mon[27845]: pgmap v18341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:31 smithi096 ceph-mon[32753]: pgmap v18341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:33 smithi194 ceph-mon[33017]: pgmap v18342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33872.220429 secs 2024-04-26T21:15:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:33 smithi033 ceph-mon[27845]: pgmap v18342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33872.220429 secs 2024-04-26T21:15:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:33 smithi096 ceph-mon[32753]: pgmap v18342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33872.220429 secs 2024-04-26T21:15:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33875.168131 secs 2024-04-26T21:15:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33875.168131 secs 2024-04-26T21:15:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33875.168131 secs 2024-04-26T21:15:36.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:35 smithi194 ceph-mon[33017]: pgmap v18343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:35 smithi033 ceph-mon[27845]: pgmap v18343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:35 smithi096 ceph-mon[32753]: pgmap v18343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:37 smithi194 ceph-mon[33017]: pgmap v18344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:37 smithi033 ceph-mon[27845]: pgmap v18344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:37 smithi096 ceph-mon[32753]: pgmap v18344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33877.220512 secs 2024-04-26T21:15:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33877.220512 secs 2024-04-26T21:15:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33877.220512 secs 2024-04-26T21:15:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:40 smithi194 ceph-mon[33017]: pgmap v18345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33880.168251 secs 2024-04-26T21:15:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:40 smithi033 ceph-mon[27845]: pgmap v18345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33880.168251 secs 2024-04-26T21:15:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:40 smithi096 ceph-mon[32753]: pgmap v18345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33880.168251 secs 2024-04-26T21:15:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:42 smithi194 ceph-mon[33017]: pgmap v18346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:42 smithi033 ceph-mon[27845]: pgmap v18346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:42 smithi096 ceph-mon[32753]: pgmap v18346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:42.640+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:42.640+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:42.640+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:15:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:44 smithi194 ceph-mon[33017]: pgmap v18347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33882.220593 secs 2024-04-26T21:15:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:44 smithi033 ceph-mon[27845]: pgmap v18347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33882.220593 secs 2024-04-26T21:15:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:44 smithi096 ceph-mon[32753]: pgmap v18347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33882.220593 secs 2024-04-26T21:15:45.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33885.168406 secs 2024-04-26T21:15:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33885.168406 secs 2024-04-26T21:15:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33885.168406 secs 2024-04-26T21:15:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:46 smithi194 ceph-mon[33017]: pgmap v18348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:46.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:15:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:46 smithi033 ceph-mon[27845]: pgmap v18348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:15:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:46 smithi096 ceph-mon[32753]: pgmap v18348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:15:46.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:15:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:15:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:15:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:47 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:15:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:15:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:15:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:47 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:15:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:15:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:15:47.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:47 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:15:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:48 smithi194 ceph-mon[33017]: pgmap v18349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:48 smithi194 ceph-mon[33017]: pgmap v18350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:48 smithi033 ceph-mon[27845]: pgmap v18349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:48 smithi033 ceph-mon[27845]: pgmap v18350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:48 smithi096 ceph-mon[32753]: pgmap v18349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:15:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:48 smithi096 ceph-mon[32753]: pgmap v18350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:49.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33887.220672 secs 2024-04-26T21:15:49.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33887.220672 secs 2024-04-26T21:15:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33887.220672 secs 2024-04-26T21:15:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:50 smithi194 ceph-mon[33017]: pgmap v18351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33890.168524 secs 2024-04-26T21:15:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:50 smithi033 ceph-mon[27845]: pgmap v18351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33890.168524 secs 2024-04-26T21:15:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:50 smithi096 ceph-mon[32753]: pgmap v18351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33890.168524 secs 2024-04-26T21:15:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:52 smithi194 ceph-mon[33017]: pgmap v18352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:52 smithi033 ceph-mon[27845]: pgmap v18352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:52.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:52 smithi096 ceph-mon[32753]: pgmap v18352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:54 smithi194 ceph-mon[33017]: pgmap v18353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33892.220753 secs 2024-04-26T21:15:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:54 smithi033 ceph-mon[27845]: pgmap v18353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33892.220753 secs 2024-04-26T21:15:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:54 smithi096 ceph-mon[32753]: pgmap v18353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:54.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33892.220753 secs 2024-04-26T21:15:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33895.168640 secs 2024-04-26T21:15:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33895.168640 secs 2024-04-26T21:15:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33895.168640 secs 2024-04-26T21:15:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:56 smithi194 ceph-mon[33017]: pgmap v18354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:56 smithi033 ceph-mon[27845]: pgmap v18354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:56 smithi096 ceph-mon[32753]: pgmap v18354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:15:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:57.644+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:57.644+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:15:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:15:57.644+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:15:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:58 smithi194 ceph-mon[33017]: pgmap v18355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:15:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:58 smithi033 ceph-mon[27845]: pgmap v18355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:15:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:58 smithi096 ceph-mon[32753]: pgmap v18355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:15:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33897.220837 secs 2024-04-26T21:15:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:15:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33900.168755 secs 2024-04-26T21:15:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33897.220837 secs 2024-04-26T21:15:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:15:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33900.168755 secs 2024-04-26T21:15:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33897.220837 secs 2024-04-26T21:15:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:15:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33900.168755 secs 2024-04-26T21:16:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:00 smithi194 ceph-mon[33017]: pgmap v18356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:00 smithi033 ceph-mon[27845]: pgmap v18356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:00 smithi096 ceph-mon[32753]: pgmap v18356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:02 smithi194 ceph-mon[33017]: pgmap v18357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:02 smithi033 ceph-mon[27845]: pgmap v18357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:02 smithi096 ceph-mon[32753]: pgmap v18357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:04 smithi033 ceph-mon[27845]: pgmap v18358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33902.220918 secs 2024-04-26T21:16:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33905.168870 secs 2024-04-26T21:16:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:04 smithi096 ceph-mon[32753]: pgmap v18358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33902.220918 secs 2024-04-26T21:16:04.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33905.168870 secs 2024-04-26T21:16:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:04 smithi194 ceph-mon[33017]: pgmap v18358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33902.220918 secs 2024-04-26T21:16:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33905.168870 secs 2024-04-26T21:16:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:06 smithi033 ceph-mon[27845]: pgmap v18359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:06 smithi096 ceph-mon[32753]: pgmap v18359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:06 smithi194 ceph-mon[33017]: pgmap v18359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:08 smithi033 ceph-mon[27845]: pgmap v18360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:08 smithi096 ceph-mon[32753]: pgmap v18360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:08 smithi194 ceph-mon[33017]: pgmap v18360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33907.220977 secs 2024-04-26T21:16:09.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33910.169002 secs 2024-04-26T21:16:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33907.220977 secs 2024-04-26T21:16:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33910.169002 secs 2024-04-26T21:16:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33907.220977 secs 2024-04-26T21:16:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33910.169002 secs 2024-04-26T21:16:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:10 smithi033 ceph-mon[27845]: pgmap v18361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:10 smithi096 ceph-mon[32753]: pgmap v18361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:10 smithi194 ceph-mon[33017]: pgmap v18361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:12 smithi033 ceph-mon[27845]: pgmap v18362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:12 smithi096 ceph-mon[32753]: pgmap v18362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:12 smithi194 ceph-mon[33017]: pgmap v18362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:12.657+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:12.657+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:12.657+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33912.221033 secs 2024-04-26T21:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33912.221033 secs 2024-04-26T21:16:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33912.221033 secs 2024-04-26T21:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:14 smithi033 ceph-mon[27845]: pgmap v18363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33915.169153 secs 2024-04-26T21:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:14 smithi096 ceph-mon[32753]: pgmap v18363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33915.169153 secs 2024-04-26T21:16:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:14 smithi194 ceph-mon[33017]: pgmap v18363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:16:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33915.169153 secs 2024-04-26T21:16:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:16 smithi033 ceph-mon[27845]: pgmap v18364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:16 smithi096 ceph-mon[32753]: pgmap v18364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:16 smithi194 ceph-mon[33017]: pgmap v18364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:16:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:18 smithi033 ceph-mon[27845]: pgmap v18365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33917.221093 secs 2024-04-26T21:16:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:18 smithi096 ceph-mon[32753]: pgmap v18365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33917.221093 secs 2024-04-26T21:16:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:18 smithi194 ceph-mon[33017]: pgmap v18365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33917.221093 secs 2024-04-26T21:16:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33920.169231 secs 2024-04-26T21:16:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33920.169231 secs 2024-04-26T21:16:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33920.169231 secs 2024-04-26T21:16:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:20 smithi033 ceph-mon[27845]: pgmap v18366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:20 smithi096 ceph-mon[32753]: pgmap v18366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:20 smithi194 ceph-mon[33017]: pgmap v18366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:22 smithi096 ceph-mon[32753]: pgmap v18367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:22 smithi194 ceph-mon[33017]: pgmap v18367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:22 smithi033 ceph-mon[27845]: pgmap v18367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33922.221179 secs 2024-04-26T21:16:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33922.221179 secs 2024-04-26T21:16:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33922.221179 secs 2024-04-26T21:16:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:24 smithi194 ceph-mon[33017]: pgmap v18368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33925.169324 secs 2024-04-26T21:16:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:24 smithi033 ceph-mon[27845]: pgmap v18368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33925.169324 secs 2024-04-26T21:16:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:24 smithi096 ceph-mon[32753]: pgmap v18368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:24.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33925.169324 secs 2024-04-26T21:16:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:26 smithi194 ceph-mon[33017]: pgmap v18369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:26 smithi033 ceph-mon[27845]: pgmap v18369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:26 smithi096 ceph-mon[32753]: pgmap v18369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:27.673+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:27.673+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:27.673+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:16:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:28 smithi194 ceph-mon[33017]: pgmap v18370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33927.221286 secs 2024-04-26T21:16:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:28 smithi033 ceph-mon[27845]: pgmap v18370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33927.221286 secs 2024-04-26T21:16:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:28 smithi096 ceph-mon[32753]: pgmap v18370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33927.221286 secs 2024-04-26T21:16:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33930.169440 secs 2024-04-26T21:16:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33930.169440 secs 2024-04-26T21:16:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33930.169440 secs 2024-04-26T21:16:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:30 smithi194 ceph-mon[33017]: pgmap v18371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:30 smithi033 ceph-mon[27845]: pgmap v18371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:30 smithi096 ceph-mon[32753]: pgmap v18371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:32 smithi194 ceph-mon[33017]: pgmap v18372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:32 smithi033 ceph-mon[27845]: pgmap v18372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:32 smithi096 ceph-mon[32753]: pgmap v18372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33932.221347 secs 2024-04-26T21:16:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33932.221347 secs 2024-04-26T21:16:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33932.221347 secs 2024-04-26T21:16:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:34 smithi194 ceph-mon[33017]: pgmap v18373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33935.169558 secs 2024-04-26T21:16:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:34 smithi033 ceph-mon[27845]: pgmap v18373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33935.169558 secs 2024-04-26T21:16:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:34 smithi096 ceph-mon[32753]: pgmap v18373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33935.169558 secs 2024-04-26T21:16:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:36 smithi194 ceph-mon[33017]: pgmap v18374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:36 smithi033 ceph-mon[27845]: pgmap v18374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:36 smithi096 ceph-mon[32753]: pgmap v18374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:38 smithi194 ceph-mon[33017]: pgmap v18375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33937.221431 secs 2024-04-26T21:16:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:38 smithi033 ceph-mon[27845]: pgmap v18375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33937.221431 secs 2024-04-26T21:16:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:38 smithi096 ceph-mon[32753]: pgmap v18375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33937.221431 secs 2024-04-26T21:16:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33940.169693 secs 2024-04-26T21:16:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33940.169693 secs 2024-04-26T21:16:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33940.169693 secs 2024-04-26T21:16:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:40 smithi194 ceph-mon[33017]: pgmap v18376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:40.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:40 smithi033 ceph-mon[27845]: pgmap v18376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:40 smithi096 ceph-mon[32753]: pgmap v18376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:42 smithi194 ceph-mon[33017]: pgmap v18377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:42 smithi033 ceph-mon[27845]: pgmap v18377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:42 smithi096 ceph-mon[32753]: pgmap v18377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:42.687+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:42.687+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:42.689 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:42.687+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:16:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33942.221489 secs 2024-04-26T21:16:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33942.221489 secs 2024-04-26T21:16:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33942.221489 secs 2024-04-26T21:16:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:44 smithi194 ceph-mon[33017]: pgmap v18378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33945.169810 secs 2024-04-26T21:16:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:44 smithi033 ceph-mon[27845]: pgmap v18378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33945.169810 secs 2024-04-26T21:16:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:44 smithi096 ceph-mon[32753]: pgmap v18378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33945.169810 secs 2024-04-26T21:16:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:46 smithi194 ceph-mon[33017]: pgmap v18379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:46 smithi033 ceph-mon[27845]: pgmap v18379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:46 smithi096 ceph-mon[32753]: pgmap v18379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:16:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:47 smithi194 ceph-mon[33017]: pgmap v18380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:16:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:47 smithi033 ceph-mon[27845]: pgmap v18380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:16:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:47 smithi096 ceph-mon[32753]: pgmap v18380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:16:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33947.221574 secs 2024-04-26T21:16:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:16:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:16:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:48 smithi194 ceph-mon[33017]: pgmap v18381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:48 smithi194 ceph-mon[33017]: pgmap v18382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:16:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33947.221574 secs 2024-04-26T21:16:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:16:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:16:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:48 smithi033 ceph-mon[27845]: pgmap v18381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:48 smithi033 ceph-mon[27845]: pgmap v18382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:16:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33947.221574 secs 2024-04-26T21:16:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:16:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:16:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:48 smithi096 ceph-mon[32753]: pgmap v18381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:16:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:48 smithi096 ceph-mon[32753]: pgmap v18382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:16:49.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33950.169918 secs 2024-04-26T21:16:49.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33950.169918 secs 2024-04-26T21:16:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33950.169918 secs 2024-04-26T21:16:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:50 smithi194 ceph-mon[33017]: pgmap v18383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:50 smithi033 ceph-mon[27845]: pgmap v18383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:50 smithi096 ceph-mon[32753]: pgmap v18383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:53 smithi033 ceph-mon[27845]: pgmap v18384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33952.221655 secs 2024-04-26T21:16:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:53 smithi096 ceph-mon[32753]: pgmap v18384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33952.221655 secs 2024-04-26T21:16:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:53 smithi194 ceph-mon[33017]: pgmap v18384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33952.221655 secs 2024-04-26T21:16:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33955.170055 secs 2024-04-26T21:16:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33955.170055 secs 2024-04-26T21:16:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33955.170055 secs 2024-04-26T21:16:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:55 smithi033 ceph-mon[27845]: pgmap v18385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:16:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:55 smithi096 ceph-mon[32753]: pgmap v18385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:16:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:55 smithi194 ceph-mon[33017]: pgmap v18385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:16:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:57 smithi033 ceph-mon[27845]: pgmap v18386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:57 smithi096 ceph-mon[32753]: pgmap v18386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:57 smithi194 ceph-mon[33017]: pgmap v18386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:16:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:57.693+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:57.693+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:16:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:16:57.693+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:16:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33957.221716 secs 2024-04-26T21:16:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33957.221716 secs 2024-04-26T21:16:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33957.221716 secs 2024-04-26T21:16:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:59 smithi194 ceph-mon[33017]: pgmap v18387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:16:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33960.170193 secs 2024-04-26T21:16:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:59 smithi033 ceph-mon[27845]: pgmap v18387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:16:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33960.170193 secs 2024-04-26T21:16:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:59 smithi096 ceph-mon[32753]: pgmap v18387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:16:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33960.170193 secs 2024-04-26T21:17:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:01 smithi194 ceph-mon[33017]: pgmap v18388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:01 smithi033 ceph-mon[27845]: pgmap v18388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:01 smithi096 ceph-mon[32753]: pgmap v18388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:03 smithi194 ceph-mon[33017]: pgmap v18389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33962.221772 secs 2024-04-26T21:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:03 smithi033 ceph-mon[27845]: pgmap v18389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33962.221772 secs 2024-04-26T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:03 smithi096 ceph-mon[32753]: pgmap v18389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33962.221772 secs 2024-04-26T21:17:04.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33965.170304 secs 2024-04-26T21:17:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33965.170304 secs 2024-04-26T21:17:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33965.170304 secs 2024-04-26T21:17:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:05 smithi194 ceph-mon[33017]: pgmap v18390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:05 smithi033 ceph-mon[27845]: pgmap v18390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:05 smithi096 ceph-mon[32753]: pgmap v18390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:07 smithi194 ceph-mon[33017]: pgmap v18391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:07 smithi033 ceph-mon[27845]: pgmap v18391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:07 smithi096 ceph-mon[32753]: pgmap v18391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33967.221832 secs 2024-04-26T21:17:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33967.221832 secs 2024-04-26T21:17:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33967.221832 secs 2024-04-26T21:17:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:09 smithi194 ceph-mon[33017]: pgmap v18392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33970.170453 secs 2024-04-26T21:17:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:09 smithi033 ceph-mon[27845]: pgmap v18392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33970.170453 secs 2024-04-26T21:17:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:09 smithi096 ceph-mon[32753]: pgmap v18392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33970.170453 secs 2024-04-26T21:17:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:11 smithi194 ceph-mon[33017]: pgmap v18393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:11 smithi033 ceph-mon[27845]: pgmap v18393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:11 smithi096 ceph-mon[32753]: pgmap v18393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:12.702+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:12.702+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:12.702+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:17:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:13 smithi194 ceph-mon[33017]: pgmap v18394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33972.221915 secs 2024-04-26T21:17:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:13 smithi033 ceph-mon[27845]: pgmap v18394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33972.221915 secs 2024-04-26T21:17:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:13 smithi096 ceph-mon[32753]: pgmap v18394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33972.221915 secs 2024-04-26T21:17:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33975.170570 secs 2024-04-26T21:17:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33975.170570 secs 2024-04-26T21:17:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33975.170570 secs 2024-04-26T21:17:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:15 smithi194 ceph-mon[33017]: pgmap v18395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:15 smithi033 ceph-mon[27845]: pgmap v18395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:15 smithi096 ceph-mon[32753]: pgmap v18395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:17 smithi194 ceph-mon[33017]: pgmap v18396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:17 smithi033 ceph-mon[27845]: pgmap v18396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:17 smithi096 ceph-mon[32753]: pgmap v18396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33977.221999 secs 2024-04-26T21:17:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33977.221999 secs 2024-04-26T21:17:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33977.221999 secs 2024-04-26T21:17:19.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:19 smithi033 ceph-mon[27845]: pgmap v18397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:19.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33980.170721 secs 2024-04-26T21:17:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:19 smithi096 ceph-mon[32753]: pgmap v18397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33980.170721 secs 2024-04-26T21:17:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:19 smithi194 ceph-mon[33017]: pgmap v18397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33980.170721 secs 2024-04-26T21:17:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:21 smithi194 ceph-mon[33017]: pgmap v18398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:21 smithi033 ceph-mon[27845]: pgmap v18398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:21 smithi096 ceph-mon[32753]: pgmap v18398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:23 smithi194 ceph-mon[33017]: pgmap v18399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33982.222059 secs 2024-04-26T21:17:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:23 smithi033 ceph-mon[27845]: pgmap v18399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33982.222059 secs 2024-04-26T21:17:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:23 smithi096 ceph-mon[32753]: pgmap v18399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33982.222059 secs 2024-04-26T21:17:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33985.170832 secs 2024-04-26T21:17:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33985.170832 secs 2024-04-26T21:17:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33985.170832 secs 2024-04-26T21:17:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:25 smithi194 ceph-mon[33017]: pgmap v18400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:25 smithi033 ceph-mon[27845]: pgmap v18400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:25 smithi096 ceph-mon[32753]: pgmap v18400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:26 smithi194 ceph-mon[33017]: pgmap v18401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:26 smithi033 ceph-mon[27845]: pgmap v18401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:26 smithi096 ceph-mon[32753]: pgmap v18401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:27.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:27.581+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T21:17:27.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:27.581+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T21:17:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:27.581+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 266 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T21:17:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:27.582+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 266 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:17:27.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:27.718+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:27.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:27.718+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:27.719 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:27.718+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:17:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33987.222146 secs 2024-04-26T21:17:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33987.222146 secs 2024-04-26T21:17:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33987.222146 secs 2024-04-26T21:17:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:29 smithi194 ceph-mon[33017]: pgmap v18402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33990.170945 secs 2024-04-26T21:17:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:29 smithi033 ceph-mon[27845]: pgmap v18402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33990.170945 secs 2024-04-26T21:17:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:29 smithi096 ceph-mon[32753]: pgmap v18402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33990.170945 secs 2024-04-26T21:17:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:31 smithi194 ceph-mon[33017]: pgmap v18403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:31 smithi033 ceph-mon[27845]: pgmap v18403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:31 smithi096 ceph-mon[32753]: pgmap v18403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:33 smithi194 ceph-mon[33017]: pgmap v18404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33992.222246 secs 2024-04-26T21:17:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:33 smithi033 ceph-mon[27845]: pgmap v18404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33992.222246 secs 2024-04-26T21:17:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:33 smithi096 ceph-mon[32753]: pgmap v18404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33992.222246 secs 2024-04-26T21:17:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 33995.171063 secs 2024-04-26T21:17:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 33995.171063 secs 2024-04-26T21:17:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 33995.171063 secs 2024-04-26T21:17:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:35 smithi194 ceph-mon[33017]: pgmap v18405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:35 smithi033 ceph-mon[27845]: pgmap v18405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:35 smithi096 ceph-mon[32753]: pgmap v18405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:37 smithi194 ceph-mon[33017]: pgmap v18406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:37 smithi033 ceph-mon[27845]: pgmap v18406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:37 smithi096 ceph-mon[32753]: pgmap v18406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 33997.222308 secs 2024-04-26T21:17:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 33997.222308 secs 2024-04-26T21:17:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 33997.222308 secs 2024-04-26T21:17:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:39 smithi194 ceph-mon[33017]: pgmap v18407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34000.171179 secs 2024-04-26T21:17:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:39 smithi033 ceph-mon[27845]: pgmap v18407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34000.171179 secs 2024-04-26T21:17:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:39 smithi096 ceph-mon[32753]: pgmap v18407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34000.171179 secs 2024-04-26T21:17:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:41 smithi194 ceph-mon[33017]: pgmap v18408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:41 smithi033 ceph-mon[27845]: pgmap v18408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:41 smithi096 ceph-mon[32753]: pgmap v18408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:42.725+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:42.725+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:42.725+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:17:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:43 smithi194 ceph-mon[33017]: pgmap v18409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34002.222390 secs 2024-04-26T21:17:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:43 smithi033 ceph-mon[27845]: pgmap v18409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34002.222390 secs 2024-04-26T21:17:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:43 smithi096 ceph-mon[32753]: pgmap v18409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34002.222390 secs 2024-04-26T21:17:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34005.171298 secs 2024-04-26T21:17:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34005.171298 secs 2024-04-26T21:17:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34005.171298 secs 2024-04-26T21:17:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:45 smithi194 ceph-mon[33017]: pgmap v18410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:45 smithi033 ceph-mon[27845]: pgmap v18410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:45 smithi096 ceph-mon[32753]: pgmap v18410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:47 smithi194 ceph-mon[33017]: pgmap v18411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:47 smithi033 ceph-mon[27845]: pgmap v18411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:47 smithi096 ceph-mon[32753]: pgmap v18411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:17:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34007.222474 secs 2024-04-26T21:17:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34007.222474 secs 2024-04-26T21:17:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34007.222474 secs 2024-04-26T21:17:49.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:49 smithi033 ceph-mon[27845]: pgmap v18412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:49.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:17:49.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34010.171363 secs 2024-04-26T21:17:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:49 smithi096 ceph-mon[32753]: pgmap v18412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:17:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34010.171363 secs 2024-04-26T21:17:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:49 smithi194 ceph-mon[33017]: pgmap v18412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:17:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:17:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34010.171363 secs 2024-04-26T21:17:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:17:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:17:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:17:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:17:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:17:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:17:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:17:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:17:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:17:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:51 smithi194 ceph-mon[33017]: pgmap v18413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:17:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:51 smithi194 ceph-mon[33017]: pgmap v18414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:51 smithi033 ceph-mon[27845]: pgmap v18413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:17:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:51 smithi033 ceph-mon[27845]: pgmap v18414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:51 smithi096 ceph-mon[32753]: pgmap v18413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:17:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:51 smithi096 ceph-mon[32753]: pgmap v18414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:53 smithi194 ceph-mon[33017]: pgmap v18415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34012.222551 secs 2024-04-26T21:17:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:53 smithi033 ceph-mon[27845]: pgmap v18415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34012.222551 secs 2024-04-26T21:17:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:53 smithi096 ceph-mon[32753]: pgmap v18415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34012.222551 secs 2024-04-26T21:17:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34015.171444 secs 2024-04-26T21:17:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34015.171444 secs 2024-04-26T21:17:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34015.171444 secs 2024-04-26T21:17:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:55 smithi194 ceph-mon[33017]: pgmap v18416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:55 smithi033 ceph-mon[27845]: pgmap v18416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:55 smithi096 ceph-mon[32753]: pgmap v18416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:56 smithi194 ceph-mon[33017]: pgmap v18417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:56 smithi033 ceph-mon[27845]: pgmap v18417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:56 smithi096 ceph-mon[32753]: pgmap v18417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:57.739+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:57.739+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:17:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:17:57.739+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:17:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:58 smithi033 ceph-mon[27845]: pgmap v18418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34017.222636 secs 2024-04-26T21:17:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:58 smithi096 ceph-mon[32753]: pgmap v18418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34017.222636 secs 2024-04-26T21:17:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:58 smithi194 ceph-mon[33017]: pgmap v18418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:17:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34017.222636 secs 2024-04-26T21:18:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34020.171507 secs 2024-04-26T21:18:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34020.171507 secs 2024-04-26T21:18:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34020.171507 secs 2024-04-26T21:18:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:00 smithi033 ceph-mon[27845]: pgmap v18419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:18:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:00 smithi096 ceph-mon[32753]: pgmap v18419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:18:01.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:00 smithi194 ceph-mon[33017]: pgmap v18419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:18:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:02 smithi033 ceph-mon[27845]: pgmap v18420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:02 smithi096 ceph-mon[32753]: pgmap v18420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:02 smithi194 ceph-mon[33017]: pgmap v18420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34022.222718 secs 2024-04-26T21:18:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34022.222718 secs 2024-04-26T21:18:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34022.222718 secs 2024-04-26T21:18:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34025.171624 secs 2024-04-26T21:18:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:04 smithi033 ceph-mon[27845]: pgmap v18421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34025.171624 secs 2024-04-26T21:18:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:04 smithi096 ceph-mon[32753]: pgmap v18421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34025.171624 secs 2024-04-26T21:18:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:04 smithi194 ceph-mon[33017]: pgmap v18421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:06 smithi033 ceph-mon[27845]: pgmap v18422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:06 smithi096 ceph-mon[32753]: pgmap v18422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:06 smithi194 ceph-mon[33017]: pgmap v18422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:08 smithi033 ceph-mon[27845]: pgmap v18423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34027.222802 secs 2024-04-26T21:18:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:08 smithi096 ceph-mon[32753]: pgmap v18423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34027.222802 secs 2024-04-26T21:18:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:08 smithi194 ceph-mon[33017]: pgmap v18423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34027.222802 secs 2024-04-26T21:18:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34030.171740 secs 2024-04-26T21:18:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34030.171740 secs 2024-04-26T21:18:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34030.171740 secs 2024-04-26T21:18:11.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:10 smithi194 ceph-mon[33017]: pgmap v18424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:10 smithi033 ceph-mon[27845]: pgmap v18424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:10 smithi096 ceph-mon[32753]: pgmap v18424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:12.754+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:12.754+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:12.754+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:18:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:12 smithi194 ceph-mon[33017]: pgmap v18425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:12 smithi033 ceph-mon[27845]: pgmap v18425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:12 smithi096 ceph-mon[32753]: pgmap v18425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34032.222885 secs 2024-04-26T21:18:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34032.222885 secs 2024-04-26T21:18:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34032.222885 secs 2024-04-26T21:18:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34035.171879 secs 2024-04-26T21:18:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:14 smithi194 ceph-mon[33017]: pgmap v18426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34035.171879 secs 2024-04-26T21:18:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:14 smithi033 ceph-mon[27845]: pgmap v18426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34035.171879 secs 2024-04-26T21:18:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:14 smithi096 ceph-mon[32753]: pgmap v18426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:16 smithi194 ceph-mon[33017]: pgmap v18427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:16 smithi033 ceph-mon[27845]: pgmap v18427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:16 smithi096 ceph-mon[32753]: pgmap v18427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:19.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:18 smithi033 ceph-mon[27845]: pgmap v18428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:19.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34037.222946 secs 2024-04-26T21:18:19.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:18 smithi096 ceph-mon[32753]: pgmap v18428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:19.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34037.222946 secs 2024-04-26T21:18:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:18 smithi194 ceph-mon[33017]: pgmap v18428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34037.222946 secs 2024-04-26T21:18:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34040.171959 secs 2024-04-26T21:18:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34040.171959 secs 2024-04-26T21:18:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34040.171959 secs 2024-04-26T21:18:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:20 smithi194 ceph-mon[33017]: pgmap v18429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:20 smithi033 ceph-mon[27845]: pgmap v18429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:20 smithi096 ceph-mon[32753]: pgmap v18429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:22 smithi194 ceph-mon[33017]: pgmap v18430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:22 smithi033 ceph-mon[27845]: pgmap v18430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:22 smithi096 ceph-mon[32753]: pgmap v18430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34042.223003 secs 2024-04-26T21:18:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34042.223003 secs 2024-04-26T21:18:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34042.223003 secs 2024-04-26T21:18:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34045.172096 secs 2024-04-26T21:18:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:25 smithi194 ceph-mon[33017]: pgmap v18431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34045.172096 secs 2024-04-26T21:18:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:25 smithi033 ceph-mon[27845]: pgmap v18431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34045.172096 secs 2024-04-26T21:18:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:25 smithi096 ceph-mon[32753]: pgmap v18431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:27 smithi194 ceph-mon[33017]: pgmap v18432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:27 smithi033 ceph-mon[27845]: pgmap v18432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:27 smithi096 ceph-mon[32753]: pgmap v18432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:27.770+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:27.770+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:27.770+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:18:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:29 smithi194 ceph-mon[33017]: pgmap v18433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34047.223065 secs 2024-04-26T21:18:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:29 smithi033 ceph-mon[27845]: pgmap v18433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34047.223065 secs 2024-04-26T21:18:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:29 smithi096 ceph-mon[32753]: pgmap v18433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:29.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34047.223065 secs 2024-04-26T21:18:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34050.172140 secs 2024-04-26T21:18:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34050.172140 secs 2024-04-26T21:18:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34050.172140 secs 2024-04-26T21:18:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:31 smithi194 ceph-mon[33017]: pgmap v18434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:31 smithi033 ceph-mon[27845]: pgmap v18434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:31 smithi096 ceph-mon[32753]: pgmap v18434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:33 smithi194 ceph-mon[33017]: pgmap v18435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:33 smithi033 ceph-mon[27845]: pgmap v18435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:33.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:33 smithi096 ceph-mon[32753]: pgmap v18435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34052.223124 secs 2024-04-26T21:18:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34052.223124 secs 2024-04-26T21:18:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34052.223124 secs 2024-04-26T21:18:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34055.172202 secs 2024-04-26T21:18:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:35 smithi194 ceph-mon[33017]: pgmap v18436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34055.172202 secs 2024-04-26T21:18:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:35 smithi033 ceph-mon[27845]: pgmap v18436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34055.172202 secs 2024-04-26T21:18:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:35 smithi096 ceph-mon[32753]: pgmap v18436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:37 smithi033 ceph-mon[27845]: pgmap v18437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:37 smithi194 ceph-mon[33017]: pgmap v18437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:37 smithi096 ceph-mon[32753]: pgmap v18437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:39 smithi194 ceph-mon[33017]: pgmap v18438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34057.223183 secs 2024-04-26T21:18:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:39 smithi033 ceph-mon[27845]: pgmap v18438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34057.223183 secs 2024-04-26T21:18:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:39 smithi096 ceph-mon[32753]: pgmap v18438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34057.223183 secs 2024-04-26T21:18:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34060.172318 secs 2024-04-26T21:18:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34060.172318 secs 2024-04-26T21:18:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34060.172318 secs 2024-04-26T21:18:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:41 smithi194 ceph-mon[33017]: pgmap v18439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:41 smithi033 ceph-mon[27845]: pgmap v18439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:41 smithi096 ceph-mon[32753]: pgmap v18439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:42.784+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:42.784+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:42.784+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:18:43.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:43 smithi194 ceph-mon[33017]: pgmap v18440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:43 smithi033 ceph-mon[27845]: pgmap v18440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:43 smithi096 ceph-mon[32753]: pgmap v18440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34062.223241 secs 2024-04-26T21:18:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34062.223241 secs 2024-04-26T21:18:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34062.223241 secs 2024-04-26T21:18:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34065.172436 secs 2024-04-26T21:18:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:45 smithi194 ceph-mon[33017]: pgmap v18441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34065.172436 secs 2024-04-26T21:18:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:45 smithi033 ceph-mon[27845]: pgmap v18441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34065.172436 secs 2024-04-26T21:18:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:45 smithi096 ceph-mon[32753]: pgmap v18441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:47.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:47 smithi194 ceph-mon[33017]: pgmap v18442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:47 smithi033 ceph-mon[27845]: pgmap v18442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:47 smithi096 ceph-mon[32753]: pgmap v18442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:49 smithi194 ceph-mon[33017]: pgmap v18443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34067.223305 secs 2024-04-26T21:18:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34070.172540 secs 2024-04-26T21:18:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:49 smithi033 ceph-mon[27845]: pgmap v18443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34067.223305 secs 2024-04-26T21:18:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34070.172540 secs 2024-04-26T21:18:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:49 smithi096 ceph-mon[32753]: pgmap v18443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34067.223305 secs 2024-04-26T21:18:49.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34070.172540 secs 2024-04-26T21:18:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:51 smithi033 ceph-mon[27845]: pgmap v18444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:51 smithi096 ceph-mon[32753]: pgmap v18444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:51 smithi194 ceph-mon[33017]: pgmap v18444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:18:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:18:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:18:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:18:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:18:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:18:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:18:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:18:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:18:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:18:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:53 smithi033 ceph-mon[27845]: pgmap v18445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:53 smithi033 ceph-mon[27845]: pgmap v18446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:53 smithi096 ceph-mon[32753]: pgmap v18445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:53 smithi096 ceph-mon[32753]: pgmap v18446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:53 smithi194 ceph-mon[33017]: pgmap v18445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:18:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:53 smithi194 ceph-mon[33017]: pgmap v18446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34072.223363 secs 2024-04-26T21:18:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34075.172681 secs 2024-04-26T21:18:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34072.223363 secs 2024-04-26T21:18:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34075.172681 secs 2024-04-26T21:18:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34072.223363 secs 2024-04-26T21:18:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34075.172681 secs 2024-04-26T21:18:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:55 smithi033 ceph-mon[27845]: pgmap v18447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:55 smithi096 ceph-mon[32753]: pgmap v18447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:55 smithi194 ceph-mon[33017]: pgmap v18447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:57 smithi033 ceph-mon[27845]: pgmap v18448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:57 smithi096 ceph-mon[32753]: pgmap v18448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:57.596+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:18:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:57.596+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:18:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:57.597+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 267 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T21:18:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:57.597+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 267 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:18:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:57 smithi194 ceph-mon[33017]: pgmap v18448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:57.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:57.800+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:57.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:57.800+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:18:57.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:18:57.800+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:18:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34077.223446 secs 2024-04-26T21:18:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34077.223446 secs 2024-04-26T21:18:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34077.223446 secs 2024-04-26T21:18:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:59 smithi033 ceph-mon[27845]: pgmap v18449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:18:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34080.172817 secs 2024-04-26T21:18:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:59 smithi096 ceph-mon[32753]: pgmap v18449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:18:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34080.172817 secs 2024-04-26T21:18:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:59 smithi194 ceph-mon[33017]: pgmap v18449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:18:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:18:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34080.172817 secs 2024-04-26T21:19:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:01 smithi033 ceph-mon[27845]: pgmap v18450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:01 smithi096 ceph-mon[32753]: pgmap v18450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:01 smithi194 ceph-mon[33017]: pgmap v18450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:03 smithi033 ceph-mon[27845]: pgmap v18451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:19:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34082.223503 secs 2024-04-26T21:19:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:03 smithi096 ceph-mon[32753]: pgmap v18451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:19:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34082.223503 secs 2024-04-26T21:19:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:03 smithi194 ceph-mon[33017]: pgmap v18451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:19:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34082.223503 secs 2024-04-26T21:19:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34085.172930 secs 2024-04-26T21:19:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34085.172930 secs 2024-04-26T21:19:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34085.172930 secs 2024-04-26T21:19:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:05 smithi096 ceph-mon[32753]: pgmap v18452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:05 smithi194 ceph-mon[33017]: pgmap v18452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:05 smithi033 ceph-mon[27845]: pgmap v18452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:07 smithi194 ceph-mon[33017]: pgmap v18453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:07 smithi033 ceph-mon[27845]: pgmap v18453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:07 smithi096 ceph-mon[32753]: pgmap v18453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34087.223586 secs 2024-04-26T21:19:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34087.223586 secs 2024-04-26T21:19:08.691 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34087.223586 secs 2024-04-26T21:19:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:09 smithi194 ceph-mon[33017]: pgmap v18454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34090.173047 secs 2024-04-26T21:19:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:09 smithi033 ceph-mon[27845]: pgmap v18454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34090.173047 secs 2024-04-26T21:19:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:09 smithi096 ceph-mon[32753]: pgmap v18454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34090.173047 secs 2024-04-26T21:19:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:11 smithi194 ceph-mon[33017]: pgmap v18455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:11 smithi033 ceph-mon[27845]: pgmap v18455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:11 smithi096 ceph-mon[32753]: pgmap v18455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:12.808+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:12.808+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:12.808+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:19:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:13 smithi194 ceph-mon[33017]: pgmap v18456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34092.223644 secs 2024-04-26T21:19:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:13 smithi033 ceph-mon[27845]: pgmap v18456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34092.223644 secs 2024-04-26T21:19:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:13 smithi096 ceph-mon[32753]: pgmap v18456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34092.223644 secs 2024-04-26T21:19:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34095.173170 secs 2024-04-26T21:19:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34095.173170 secs 2024-04-26T21:19:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34095.173170 secs 2024-04-26T21:19:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:15 smithi194 ceph-mon[33017]: pgmap v18457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:15 smithi033 ceph-mon[27845]: pgmap v18457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:15 smithi096 ceph-mon[32753]: pgmap v18457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:17 smithi194 ceph-mon[33017]: pgmap v18458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:17 smithi033 ceph-mon[27845]: pgmap v18458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:17 smithi096 ceph-mon[32753]: pgmap v18458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34097.223727 secs 2024-04-26T21:19:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34097.223727 secs 2024-04-26T21:19:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34097.223727 secs 2024-04-26T21:19:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:19 smithi194 ceph-mon[33017]: pgmap v18459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:19.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34100.173285 secs 2024-04-26T21:19:19.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:19 smithi033 ceph-mon[27845]: pgmap v18459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:19.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34100.173285 secs 2024-04-26T21:19:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:19 smithi096 ceph-mon[32753]: pgmap v18459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34100.173285 secs 2024-04-26T21:19:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:21 smithi194 ceph-mon[33017]: pgmap v18460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:21 smithi033 ceph-mon[27845]: pgmap v18460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:21 smithi096 ceph-mon[32753]: pgmap v18460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:23 smithi194 ceph-mon[33017]: pgmap v18461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34102.223808 secs 2024-04-26T21:19:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:23 smithi033 ceph-mon[27845]: pgmap v18461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34102.223808 secs 2024-04-26T21:19:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:23 smithi096 ceph-mon[32753]: pgmap v18461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34102.223808 secs 2024-04-26T21:19:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34105.173424 secs 2024-04-26T21:19:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34105.173424 secs 2024-04-26T21:19:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34105.173424 secs 2024-04-26T21:19:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:25 smithi194 ceph-mon[33017]: pgmap v18462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:25 smithi033 ceph-mon[27845]: pgmap v18462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:25 smithi096 ceph-mon[32753]: pgmap v18462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:27 smithi194 ceph-mon[33017]: pgmap v18463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:27 smithi033 ceph-mon[27845]: pgmap v18463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:27 smithi096 ceph-mon[32753]: pgmap v18463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:27.814+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:27.814+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:19:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:27.814+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34107.223889 secs 2024-04-26T21:19:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34107.223889 secs 2024-04-26T21:19:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34107.223889 secs 2024-04-26T21:19:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:29 smithi194 ceph-mon[33017]: pgmap v18464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34110.173493 secs 2024-04-26T21:19:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:29 smithi033 ceph-mon[27845]: pgmap v18464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34110.173493 secs 2024-04-26T21:19:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:29 smithi096 ceph-mon[32753]: pgmap v18464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34110.173493 secs 2024-04-26T21:19:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:31 smithi194 ceph-mon[33017]: pgmap v18465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:31 smithi033 ceph-mon[27845]: pgmap v18465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:31 smithi096 ceph-mon[32753]: pgmap v18465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:33 smithi194 ceph-mon[33017]: pgmap v18466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34112.223990 secs 2024-04-26T21:19:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:33 smithi033 ceph-mon[27845]: pgmap v18466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34112.223990 secs 2024-04-26T21:19:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:33 smithi096 ceph-mon[32753]: pgmap v18466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34112.223990 secs 2024-04-26T21:19:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34115.173608 secs 2024-04-26T21:19:34.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34115.173608 secs 2024-04-26T21:19:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34115.173608 secs 2024-04-26T21:19:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:35 smithi194 ceph-mon[33017]: pgmap v18467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:35 smithi033 ceph-mon[27845]: pgmap v18467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:35 smithi096 ceph-mon[32753]: pgmap v18467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:37 smithi194 ceph-mon[33017]: pgmap v18468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:37 smithi033 ceph-mon[27845]: pgmap v18468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:37 smithi096 ceph-mon[32753]: pgmap v18468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:38.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34117.224051 secs 2024-04-26T21:19:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34117.224051 secs 2024-04-26T21:19:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34117.224051 secs 2024-04-26T21:19:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:39 smithi194 ceph-mon[33017]: pgmap v18469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34120.173728 secs 2024-04-26T21:19:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:39 smithi033 ceph-mon[27845]: pgmap v18469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34120.173728 secs 2024-04-26T21:19:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:39 smithi096 ceph-mon[32753]: pgmap v18469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34120.173728 secs 2024-04-26T21:19:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:40 smithi194 ceph-mon[33017]: pgmap v18470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:40 smithi033 ceph-mon[27845]: pgmap v18470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:40 smithi096 ceph-mon[32753]: pgmap v18470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:42.826+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:42.826+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:19:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:42.826+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:42 smithi194 ceph-mon[33017]: pgmap v18471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:42 smithi033 ceph-mon[27845]: pgmap v18471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:42 smithi096 ceph-mon[32753]: pgmap v18471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34122.224111 secs 2024-04-26T21:19:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34122.224111 secs 2024-04-26T21:19:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34122.224111 secs 2024-04-26T21:19:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:44 smithi194 ceph-mon[33017]: pgmap v18472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34125.173883 secs 2024-04-26T21:19:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:44 smithi033 ceph-mon[27845]: pgmap v18472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34125.173883 secs 2024-04-26T21:19:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:44 smithi096 ceph-mon[32753]: pgmap v18472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34125.173883 secs 2024-04-26T21:19:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:46 smithi194 ceph-mon[33017]: pgmap v18473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:46 smithi033 ceph-mon[27845]: pgmap v18473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:46 smithi096 ceph-mon[32753]: pgmap v18473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:48 smithi194 ceph-mon[33017]: pgmap v18474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34127.224199 secs 2024-04-26T21:19:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:48 smithi033 ceph-mon[27845]: pgmap v18474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34127.224199 secs 2024-04-26T21:19:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:48 smithi096 ceph-mon[32753]: pgmap v18474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34127.224199 secs 2024-04-26T21:19:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34130.173998 secs 2024-04-26T21:19:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34130.173998 secs 2024-04-26T21:19:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34130.173998 secs 2024-04-26T21:19:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:50 smithi033 ceph-mon[27845]: pgmap v18475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:50 smithi096 ceph-mon[32753]: pgmap v18475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:50 smithi194 ceph-mon[33017]: pgmap v18475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:19:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:52 smithi033 ceph-mon[27845]: pgmap v18476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:52 smithi096 ceph-mon[32753]: pgmap v18476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:52 smithi194 ceph-mon[33017]: pgmap v18476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34132.224257 secs 2024-04-26T21:19:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:19:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:53 smithi033 ceph-mon[27845]: pgmap v18477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:53 smithi033 ceph-mon[27845]: pgmap v18478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:53.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:19:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34132.224257 secs 2024-04-26T21:19:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:19:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:53 smithi096 ceph-mon[32753]: pgmap v18477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:53 smithi096 ceph-mon[32753]: pgmap v18478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:53.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:19:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34132.224257 secs 2024-04-26T21:19:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:19:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:53 smithi194 ceph-mon[33017]: pgmap v18477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:19:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:53 smithi194 ceph-mon[33017]: pgmap v18478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:19:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34135.174135 secs 2024-04-26T21:19:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34135.174135 secs 2024-04-26T21:19:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34135.174135 secs 2024-04-26T21:19:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:55 smithi033 ceph-mon[27845]: pgmap v18479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:55 smithi096 ceph-mon[32753]: pgmap v18479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:55 smithi194 ceph-mon[33017]: pgmap v18479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:57.837+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:57.837+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:19:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:19:57.837+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:19:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:58 smithi194 ceph-mon[33017]: pgmap v18480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34137.224340 secs 2024-04-26T21:19:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:58 smithi033 ceph-mon[27845]: pgmap v18480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34137.224340 secs 2024-04-26T21:19:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:58 smithi096 ceph-mon[32753]: pgmap v18480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:19:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34137.224340 secs 2024-04-26T21:19:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:19:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34140.174251 secs 2024-04-26T21:19:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:19:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34140.174251 secs 2024-04-26T21:19:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:19:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34140.174251 secs 2024-04-26T21:20:00.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:00 smithi033 ceph-mon[27845]: pgmap v18481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:20:00.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:20:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:00 smithi194 ceph-mon[33017]: pgmap v18481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:20:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:20:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:00 smithi096 ceph-mon[32753]: pgmap v18481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:20:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:20:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:02 smithi194 ceph-mon[33017]: pgmap v18482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:02 smithi033 ceph-mon[27845]: pgmap v18482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:02 smithi096 ceph-mon[32753]: pgmap v18482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34142.224426 secs 2024-04-26T21:20:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34142.224426 secs 2024-04-26T21:20:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34142.224426 secs 2024-04-26T21:20:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:04 smithi194 ceph-mon[33017]: pgmap v18483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:20:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34145.174325 secs 2024-04-26T21:20:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:04 smithi033 ceph-mon[27845]: pgmap v18483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:20:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34145.174325 secs 2024-04-26T21:20:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:04 smithi096 ceph-mon[32753]: pgmap v18483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:20:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34145.174325 secs 2024-04-26T21:20:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:06 smithi194 ceph-mon[33017]: pgmap v18484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:06 smithi033 ceph-mon[27845]: pgmap v18484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:06 smithi096 ceph-mon[32753]: pgmap v18484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:08 smithi194 ceph-mon[33017]: pgmap v18485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34147.224512 secs 2024-04-26T21:20:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:08 smithi033 ceph-mon[27845]: pgmap v18485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34147.224512 secs 2024-04-26T21:20:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:08 smithi096 ceph-mon[32753]: pgmap v18485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34147.224512 secs 2024-04-26T21:20:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34150.174440 secs 2024-04-26T21:20:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34150.174440 secs 2024-04-26T21:20:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34150.174440 secs 2024-04-26T21:20:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:10 smithi194 ceph-mon[33017]: pgmap v18486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:10 smithi033 ceph-mon[27845]: pgmap v18486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:10 smithi096 ceph-mon[32753]: pgmap v18486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:11 smithi194 ceph-mon[33017]: pgmap v18487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:11 smithi033 ceph-mon[27845]: pgmap v18487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:11 smithi096 ceph-mon[32753]: pgmap v18487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:12.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:12.851+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:12.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:12.852+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:12.853 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:12.852+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:20:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34152.224594 secs 2024-04-26T21:20:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34152.224594 secs 2024-04-26T21:20:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34152.224594 secs 2024-04-26T21:20:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:14 smithi194 ceph-mon[33017]: pgmap v18488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34155.174553 secs 2024-04-26T21:20:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:14 smithi033 ceph-mon[27845]: pgmap v18488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34155.174553 secs 2024-04-26T21:20:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:14 smithi096 ceph-mon[32753]: pgmap v18488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34155.174553 secs 2024-04-26T21:20:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:16 smithi194 ceph-mon[33017]: pgmap v18489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:16 smithi033 ceph-mon[27845]: pgmap v18489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:16 smithi096 ceph-mon[32753]: pgmap v18489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:18 smithi194 ceph-mon[33017]: pgmap v18490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34157.224671 secs 2024-04-26T21:20:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:18 smithi033 ceph-mon[27845]: pgmap v18490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34157.224671 secs 2024-04-26T21:20:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:18 smithi096 ceph-mon[32753]: pgmap v18490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34157.224671 secs 2024-04-26T21:20:19.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34160.174638 secs 2024-04-26T21:20:19.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34160.174638 secs 2024-04-26T21:20:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34160.174638 secs 2024-04-26T21:20:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:20 smithi194 ceph-mon[33017]: pgmap v18491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:20 smithi033 ceph-mon[27845]: pgmap v18491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:20 smithi096 ceph-mon[32753]: pgmap v18491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:22 smithi194 ceph-mon[33017]: pgmap v18492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:22 smithi033 ceph-mon[27845]: pgmap v18492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:22 smithi096 ceph-mon[32753]: pgmap v18492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34162.224753 secs 2024-04-26T21:20:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34162.224753 secs 2024-04-26T21:20:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34162.224753 secs 2024-04-26T21:20:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:24 smithi194 ceph-mon[33017]: pgmap v18493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34165.174760 secs 2024-04-26T21:20:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:24 smithi033 ceph-mon[27845]: pgmap v18493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34165.174760 secs 2024-04-26T21:20:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:24 smithi096 ceph-mon[32753]: pgmap v18493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34165.174760 secs 2024-04-26T21:20:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:26 smithi194 ceph-mon[33017]: pgmap v18494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:26 smithi033 ceph-mon[27845]: pgmap v18494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:26 smithi096 ceph-mon[32753]: pgmap v18494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:27.612+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:20:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:27.612+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T21:20:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:27.612+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 268 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:20:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:27.612+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 268 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T21:20:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:27.857+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:27.857+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:20:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:27.857+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:28 smithi194 ceph-mon[33017]: pgmap v18495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34167.224813 secs 2024-04-26T21:20:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:28 smithi033 ceph-mon[27845]: pgmap v18495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34167.224813 secs 2024-04-26T21:20:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:28 smithi096 ceph-mon[32753]: pgmap v18495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34167.224813 secs 2024-04-26T21:20:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34170.174875 secs 2024-04-26T21:20:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34170.174875 secs 2024-04-26T21:20:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34170.174875 secs 2024-04-26T21:20:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:30 smithi194 ceph-mon[33017]: pgmap v18496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:30 smithi033 ceph-mon[27845]: pgmap v18496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:30 smithi096 ceph-mon[32753]: pgmap v18496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:31 smithi194 ceph-mon[33017]: pgmap v18497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:31 smithi033 ceph-mon[27845]: pgmap v18497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:31 smithi096 ceph-mon[32753]: pgmap v18497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34172.224893 secs 2024-04-26T21:20:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34172.224893 secs 2024-04-26T21:20:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34172.224893 secs 2024-04-26T21:20:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:34 smithi194 ceph-mon[33017]: pgmap v18498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34175.175011 secs 2024-04-26T21:20:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:34 smithi033 ceph-mon[27845]: pgmap v18498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34175.175011 secs 2024-04-26T21:20:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:34 smithi096 ceph-mon[32753]: pgmap v18498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34175.175011 secs 2024-04-26T21:20:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:36 smithi194 ceph-mon[33017]: pgmap v18499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:36 smithi033 ceph-mon[27845]: pgmap v18499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:36 smithi096 ceph-mon[32753]: pgmap v18499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:38 smithi194 ceph-mon[33017]: pgmap v18500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34177.224978 secs 2024-04-26T21:20:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:38 smithi033 ceph-mon[27845]: pgmap v18500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34177.224978 secs 2024-04-26T21:20:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:38 smithi096 ceph-mon[32753]: pgmap v18500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34177.224978 secs 2024-04-26T21:20:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34180.175163 secs 2024-04-26T21:20:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34180.175163 secs 2024-04-26T21:20:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34180.175163 secs 2024-04-26T21:20:40.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:40 smithi033 ceph-mon[27845]: pgmap v18501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:40 smithi194 ceph-mon[33017]: pgmap v18501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:40 smithi096 ceph-mon[32753]: pgmap v18501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:42 smithi194 ceph-mon[33017]: pgmap v18502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:42 smithi033 ceph-mon[27845]: pgmap v18502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:42 smithi096 ceph-mon[32753]: pgmap v18502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:42.860+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:42.860+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:20:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:42.860+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34182.225062 secs 2024-04-26T21:20:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34182.225062 secs 2024-04-26T21:20:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34182.225062 secs 2024-04-26T21:20:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:44 smithi194 ceph-mon[33017]: pgmap v18503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34185.175301 secs 2024-04-26T21:20:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:44 smithi033 ceph-mon[27845]: pgmap v18503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34185.175301 secs 2024-04-26T21:20:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:44 smithi096 ceph-mon[32753]: pgmap v18503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34185.175301 secs 2024-04-26T21:20:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:46 smithi194 ceph-mon[33017]: pgmap v18504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:46 smithi033 ceph-mon[27845]: pgmap v18504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:46 smithi096 ceph-mon[32753]: pgmap v18504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:47 smithi033 ceph-mon[27845]: pgmap v18505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:47 smithi096 ceph-mon[32753]: pgmap v18505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:47 smithi194 ceph-mon[33017]: pgmap v18505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34187.225125 secs 2024-04-26T21:20:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34187.225125 secs 2024-04-26T21:20:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34187.225125 secs 2024-04-26T21:20:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34190.175419 secs 2024-04-26T21:20:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:49 smithi033 ceph-mon[27845]: pgmap v18506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34190.175419 secs 2024-04-26T21:20:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:49 smithi096 ceph-mon[32753]: pgmap v18506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34190.175419 secs 2024-04-26T21:20:49.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:49 smithi194 ceph-mon[33017]: pgmap v18506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:52 smithi194 ceph-mon[33017]: pgmap v18507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:52 smithi033 ceph-mon[27845]: pgmap v18507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:52 smithi096 ceph-mon[32753]: pgmap v18507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:20:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34192.225183 secs 2024-04-26T21:20:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34192.225183 secs 2024-04-26T21:20:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34192.225183 secs 2024-04-26T21:20:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:54 smithi194 ceph-mon[33017]: pgmap v18508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34195.175504 secs 2024-04-26T21:20:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:54 smithi033 ceph-mon[27845]: pgmap v18508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34195.175504 secs 2024-04-26T21:20:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:54 smithi096 ceph-mon[32753]: pgmap v18508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:20:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34195.175504 secs 2024-04-26T21:20:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:20:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:20:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:20:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:56 smithi033 ceph-mon[27845]: pgmap v18509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:20:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:56 smithi033 ceph-mon[27845]: pgmap v18510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:56 smithi194 ceph-mon[33017]: pgmap v18509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:20:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:56 smithi194 ceph-mon[33017]: pgmap v18510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:56 smithi096 ceph-mon[32753]: pgmap v18509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:20:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:56 smithi096 ceph-mon[32753]: pgmap v18510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:57.876+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:57.876+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:20:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:20:57.876+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:20:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:58 smithi194 ceph-mon[33017]: pgmap v18511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34197.225244 secs 2024-04-26T21:20:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:58 smithi033 ceph-mon[27845]: pgmap v18511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34197.225244 secs 2024-04-26T21:20:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:58 smithi096 ceph-mon[32753]: pgmap v18511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:20:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34197.225244 secs 2024-04-26T21:20:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:20:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34200.175639 secs 2024-04-26T21:20:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:20:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34200.175639 secs 2024-04-26T21:20:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:20:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34200.175639 secs 2024-04-26T21:21:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:00 smithi194 ceph-mon[33017]: pgmap v18512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:00 smithi033 ceph-mon[27845]: pgmap v18512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:00 smithi096 ceph-mon[32753]: pgmap v18512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:01 smithi194 ceph-mon[33017]: pgmap v18513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:01 smithi033 ceph-mon[27845]: pgmap v18513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:01 smithi096 ceph-mon[32753]: pgmap v18513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:04 smithi194 ceph-mon[33017]: pgmap v18514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34202.225327 secs 2024-04-26T21:21:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:04 smithi033 ceph-mon[27845]: pgmap v18514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34202.225327 secs 2024-04-26T21:21:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:04 smithi096 ceph-mon[32753]: pgmap v18514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:21:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34202.225327 secs 2024-04-26T21:21:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34205.175771 secs 2024-04-26T21:21:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34205.175771 secs 2024-04-26T21:21:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34205.175771 secs 2024-04-26T21:21:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:06 smithi194 ceph-mon[33017]: pgmap v18515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:21:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:06 smithi033 ceph-mon[27845]: pgmap v18515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:21:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:06 smithi096 ceph-mon[32753]: pgmap v18515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:21:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:08 smithi194 ceph-mon[33017]: pgmap v18516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:08 smithi033 ceph-mon[27845]: pgmap v18516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:08 smithi096 ceph-mon[32753]: pgmap v18516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34207.225412 secs 2024-04-26T21:21:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34207.225412 secs 2024-04-26T21:21:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34207.225412 secs 2024-04-26T21:21:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34210.175868 secs 2024-04-26T21:21:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:10 smithi194 ceph-mon[33017]: pgmap v18517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34210.175868 secs 2024-04-26T21:21:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:10 smithi033 ceph-mon[27845]: pgmap v18517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34210.175868 secs 2024-04-26T21:21:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:10 smithi096 ceph-mon[32753]: pgmap v18517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:12 smithi194 ceph-mon[33017]: pgmap v18518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:12.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:12 smithi033 ceph-mon[27845]: pgmap v18518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:12 smithi096 ceph-mon[32753]: pgmap v18518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:12.882+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:12.882+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:21:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:12.882+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:14 smithi194 ceph-mon[33017]: pgmap v18519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34212.225465 secs 2024-04-26T21:21:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:14 smithi033 ceph-mon[27845]: pgmap v18519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34212.225465 secs 2024-04-26T21:21:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:14 smithi096 ceph-mon[32753]: pgmap v18519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34212.225465 secs 2024-04-26T21:21:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34215.176004 secs 2024-04-26T21:21:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34215.176004 secs 2024-04-26T21:21:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34215.176004 secs 2024-04-26T21:21:16.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:16 smithi194 ceph-mon[33017]: pgmap v18520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:16 smithi033 ceph-mon[27845]: pgmap v18520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:16 smithi096 ceph-mon[32753]: pgmap v18520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:18.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:18 smithi194 ceph-mon[33017]: pgmap v18521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:18 smithi033 ceph-mon[27845]: pgmap v18521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:18 smithi096 ceph-mon[32753]: pgmap v18521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:19.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34217.225550 secs 2024-04-26T21:21:19.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34217.225550 secs 2024-04-26T21:21:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34217.225550 secs 2024-04-26T21:21:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34220.176065 secs 2024-04-26T21:21:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:20 smithi194 ceph-mon[33017]: pgmap v18522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34220.176065 secs 2024-04-26T21:21:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:20 smithi033 ceph-mon[27845]: pgmap v18522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34220.176065 secs 2024-04-26T21:21:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:20 smithi096 ceph-mon[32753]: pgmap v18522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:22 smithi194 ceph-mon[33017]: pgmap v18523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:22.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:22 smithi033 ceph-mon[27845]: pgmap v18523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:22 smithi096 ceph-mon[32753]: pgmap v18523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:24 smithi194 ceph-mon[33017]: pgmap v18524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34222.225633 secs 2024-04-26T21:21:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:24 smithi033 ceph-mon[27845]: pgmap v18524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34222.225633 secs 2024-04-26T21:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:24 smithi096 ceph-mon[32753]: pgmap v18524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34222.225633 secs 2024-04-26T21:21:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34225.176188 secs 2024-04-26T21:21:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34225.176188 secs 2024-04-26T21:21:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34225.176188 secs 2024-04-26T21:21:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:26 smithi033 ceph-mon[27845]: pgmap v18525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:26 smithi096 ceph-mon[32753]: pgmap v18525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:26 smithi194 ceph-mon[33017]: pgmap v18525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:27.893+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:27.893+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:27.893+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:21:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:28 smithi033 ceph-mon[27845]: pgmap v18526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:28 smithi096 ceph-mon[32753]: pgmap v18526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:28 smithi194 ceph-mon[33017]: pgmap v18526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34227.225716 secs 2024-04-26T21:21:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34227.225716 secs 2024-04-26T21:21:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34227.225716 secs 2024-04-26T21:21:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34230.176252 secs 2024-04-26T21:21:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:30 smithi033 ceph-mon[27845]: pgmap v18527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34230.176252 secs 2024-04-26T21:21:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:30 smithi096 ceph-mon[32753]: pgmap v18527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34230.176252 secs 2024-04-26T21:21:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:30 smithi194 ceph-mon[33017]: pgmap v18527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:32 smithi033 ceph-mon[27845]: pgmap v18528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:32 smithi096 ceph-mon[32753]: pgmap v18528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:32 smithi194 ceph-mon[33017]: pgmap v18528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:34 smithi033 ceph-mon[27845]: pgmap v18529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34232.225796 secs 2024-04-26T21:21:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34235.176372 secs 2024-04-26T21:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:34 smithi096 ceph-mon[32753]: pgmap v18529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34232.225796 secs 2024-04-26T21:21:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34235.176372 secs 2024-04-26T21:21:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:34 smithi194 ceph-mon[33017]: pgmap v18529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34232.225796 secs 2024-04-26T21:21:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34235.176372 secs 2024-04-26T21:21:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:36 smithi033 ceph-mon[27845]: pgmap v18530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:36 smithi096 ceph-mon[32753]: pgmap v18530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:36 smithi194 ceph-mon[33017]: pgmap v18530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:38 smithi033 ceph-mon[27845]: pgmap v18531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:38 smithi096 ceph-mon[32753]: pgmap v18531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:38 smithi194 ceph-mon[33017]: pgmap v18531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34237.225880 secs 2024-04-26T21:21:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34240.176492 secs 2024-04-26T21:21:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34237.225880 secs 2024-04-26T21:21:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34240.176492 secs 2024-04-26T21:21:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34237.225880 secs 2024-04-26T21:21:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34240.176492 secs 2024-04-26T21:21:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:40 smithi033 ceph-mon[27845]: pgmap v18532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:40 smithi096 ceph-mon[32753]: pgmap v18532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:40 smithi194 ceph-mon[33017]: pgmap v18532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:42 smithi096 ceph-mon[32753]: pgmap v18533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:42 smithi194 ceph-mon[33017]: pgmap v18533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:42 smithi033 ceph-mon[27845]: pgmap v18533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:42.906+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:42.906+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:42.906+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:21:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34242.225960 secs 2024-04-26T21:21:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34242.225960 secs 2024-04-26T21:21:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34242.225960 secs 2024-04-26T21:21:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:44 smithi194 ceph-mon[33017]: pgmap v18534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34245.176619 secs 2024-04-26T21:21:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:44 smithi033 ceph-mon[27845]: pgmap v18534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34245.176619 secs 2024-04-26T21:21:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:44 smithi096 ceph-mon[32753]: pgmap v18534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34245.176619 secs 2024-04-26T21:21:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:46 smithi194 ceph-mon[33017]: pgmap v18535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:46 smithi033 ceph-mon[27845]: pgmap v18535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:46 smithi096 ceph-mon[32753]: pgmap v18535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:48 smithi194 ceph-mon[33017]: pgmap v18536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34247.226044 secs 2024-04-26T21:21:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:48 smithi033 ceph-mon[27845]: pgmap v18536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:48.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34247.226044 secs 2024-04-26T21:21:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:48 smithi096 ceph-mon[32753]: pgmap v18536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34247.226044 secs 2024-04-26T21:21:49.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34250.176751 secs 2024-04-26T21:21:49.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34250.176751 secs 2024-04-26T21:21:49.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34250.176751 secs 2024-04-26T21:21:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:50 smithi194 ceph-mon[33017]: pgmap v18537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:50 smithi033 ceph-mon[27845]: pgmap v18537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:50 smithi096 ceph-mon[32753]: pgmap v18537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:52 smithi194 ceph-mon[33017]: pgmap v18538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:52 smithi033 ceph-mon[27845]: pgmap v18538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:52 smithi096 ceph-mon[32753]: pgmap v18538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34252.226125 secs 2024-04-26T21:21:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34252.226125 secs 2024-04-26T21:21:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34252.226125 secs 2024-04-26T21:21:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:54 smithi194 ceph-mon[33017]: pgmap v18539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34255.176871 secs 2024-04-26T21:21:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:54 smithi033 ceph-mon[27845]: pgmap v18539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34255.176871 secs 2024-04-26T21:21:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:54 smithi096 ceph-mon[32753]: pgmap v18539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34255.176871 secs 2024-04-26T21:21:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:56 smithi194 ceph-mon[33017]: pgmap v18540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:56 smithi033 ceph-mon[27845]: pgmap v18540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:56 smithi096 ceph-mon[32753]: pgmap v18540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:21:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:57.922+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:57.922+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:21:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:21:57.922+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:21:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:58 smithi194 ceph-mon[33017]: pgmap v18541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34257.226192 secs 2024-04-26T21:21:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:58 smithi033 ceph-mon[27845]: pgmap v18541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34257.226192 secs 2024-04-26T21:21:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:58 smithi096 ceph-mon[32753]: pgmap v18541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:21:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:21:58.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34257.226192 secs 2024-04-26T21:21:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:59 smithi033 ceph-mon[27845]: pgmap v18542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:21:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:59 smithi033 ceph-mon[27845]: pgmap v18543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:21:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:21:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34260.176989 secs 2024-04-26T21:21:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:59 smithi096 ceph-mon[32753]: pgmap v18542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:21:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:59 smithi096 ceph-mon[32753]: pgmap v18543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:21:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:21:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34260.176989 secs 2024-04-26T21:21:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:59 smithi194 ceph-mon[33017]: pgmap v18542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:21:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:59 smithi194 ceph-mon[33017]: pgmap v18543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:21:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:21:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34260.176989 secs 2024-04-26T21:22:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:00 smithi033 ceph-mon[27845]: pgmap v18544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:00 smithi096 ceph-mon[32753]: pgmap v18544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:00 smithi194 ceph-mon[33017]: pgmap v18544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:02 smithi194 ceph-mon[33017]: pgmap v18545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:02 smithi033 ceph-mon[27845]: pgmap v18545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:02 smithi096 ceph-mon[32753]: pgmap v18545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34262.226272 secs 2024-04-26T21:22:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34262.226272 secs 2024-04-26T21:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34262.226272 secs 2024-04-26T21:22:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34265.177136 secs 2024-04-26T21:22:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:04 smithi194 ceph-mon[33017]: pgmap v18546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T21:22:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34265.177136 secs 2024-04-26T21:22:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:04 smithi033 ceph-mon[27845]: pgmap v18546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T21:22:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34265.177136 secs 2024-04-26T21:22:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:04 smithi096 ceph-mon[32753]: pgmap v18546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-26T21:22:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:06 smithi194 ceph-mon[33017]: pgmap v18547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:06 smithi033 ceph-mon[27845]: pgmap v18547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:06 smithi096 ceph-mon[32753]: pgmap v18547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:22:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:08 smithi194 ceph-mon[33017]: pgmap v18548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:22:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34267.226356 secs 2024-04-26T21:22:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:08 smithi033 ceph-mon[27845]: pgmap v18548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:22:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34267.226356 secs 2024-04-26T21:22:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:08 smithi096 ceph-mon[32753]: pgmap v18548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:22:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34267.226356 secs 2024-04-26T21:22:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34270.177264 secs 2024-04-26T21:22:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34270.177264 secs 2024-04-26T21:22:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34270.177264 secs 2024-04-26T21:22:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:10 smithi194 ceph-mon[33017]: pgmap v18549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:10 smithi033 ceph-mon[27845]: pgmap v18549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:10 smithi096 ceph-mon[32753]: pgmap v18549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:12.935+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:12.935+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:22:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:12.935+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:13.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:12 smithi194 ceph-mon[33017]: pgmap v18550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:12 smithi033 ceph-mon[27845]: pgmap v18550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:12 smithi096 ceph-mon[32753]: pgmap v18550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34272.226414 secs 2024-04-26T21:22:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34272.226414 secs 2024-04-26T21:22:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34272.226414 secs 2024-04-26T21:22:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34275.177385 secs 2024-04-26T21:22:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:14 smithi033 ceph-mon[27845]: pgmap v18551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34275.177385 secs 2024-04-26T21:22:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:14 smithi096 ceph-mon[32753]: pgmap v18551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34275.177385 secs 2024-04-26T21:22:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:14 smithi194 ceph-mon[33017]: pgmap v18551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:16 smithi033 ceph-mon[27845]: pgmap v18552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:16 smithi096 ceph-mon[32753]: pgmap v18552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:16 smithi194 ceph-mon[33017]: pgmap v18552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:19.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:18 smithi194 ceph-mon[33017]: pgmap v18553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:19.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34277.226474 secs 2024-04-26T21:22:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:18 smithi033 ceph-mon[27845]: pgmap v18553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34277.226474 secs 2024-04-26T21:22:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:18 smithi096 ceph-mon[32753]: pgmap v18553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34277.226474 secs 2024-04-26T21:22:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34280.177507 secs 2024-04-26T21:22:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34280.177507 secs 2024-04-26T21:22:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34280.177507 secs 2024-04-26T21:22:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:20 smithi033 ceph-mon[27845]: pgmap v18554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:20 smithi096 ceph-mon[32753]: pgmap v18554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:20 smithi194 ceph-mon[33017]: pgmap v18554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:22 smithi033 ceph-mon[27845]: pgmap v18555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:22 smithi096 ceph-mon[32753]: pgmap v18555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:22 smithi194 ceph-mon[33017]: pgmap v18555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34282.226554 secs 2024-04-26T21:22:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34282.226554 secs 2024-04-26T21:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34282.226554 secs 2024-04-26T21:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34285.177653 secs 2024-04-26T21:22:25.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:24 smithi033 ceph-mon[27845]: pgmap v18556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34285.177653 secs 2024-04-26T21:22:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:24 smithi096 ceph-mon[32753]: pgmap v18556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34285.177653 secs 2024-04-26T21:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:24 smithi194 ceph-mon[33017]: pgmap v18556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:27 smithi194 ceph-mon[33017]: pgmap v18557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:27 smithi033 ceph-mon[27845]: pgmap v18557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:27 smithi096 ceph-mon[32753]: pgmap v18557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:27.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:27.949+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:27.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:27.950+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:27.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:27.950+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:22:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:29 smithi194 ceph-mon[33017]: pgmap v18558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34287.226639 secs 2024-04-26T21:22:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:29 smithi033 ceph-mon[27845]: pgmap v18558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34287.226639 secs 2024-04-26T21:22:29.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:29 smithi096 ceph-mon[32753]: pgmap v18558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:29.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34287.226639 secs 2024-04-26T21:22:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34290.177788 secs 2024-04-26T21:22:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34290.177788 secs 2024-04-26T21:22:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34290.177788 secs 2024-04-26T21:22:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:31 smithi033 ceph-mon[27845]: pgmap v18559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:31 smithi096 ceph-mon[32753]: pgmap v18559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:31 smithi194 ceph-mon[33017]: pgmap v18559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:33 smithi033 ceph-mon[27845]: pgmap v18560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:33.461 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:33 smithi096 ceph-mon[32753]: pgmap v18560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:33 smithi194 ceph-mon[33017]: pgmap v18560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34292.226715 secs 2024-04-26T21:22:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34292.226715 secs 2024-04-26T21:22:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34292.226715 secs 2024-04-26T21:22:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34295.177938 secs 2024-04-26T21:22:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:35 smithi033 ceph-mon[27845]: pgmap v18561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34295.177938 secs 2024-04-26T21:22:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:35 smithi096 ceph-mon[32753]: pgmap v18561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34295.177938 secs 2024-04-26T21:22:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:35 smithi194 ceph-mon[33017]: pgmap v18561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:37 smithi033 ceph-mon[27845]: pgmap v18562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:37 smithi096 ceph-mon[32753]: pgmap v18562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:37 smithi194 ceph-mon[33017]: pgmap v18562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:39 smithi033 ceph-mon[27845]: pgmap v18563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34297.226799 secs 2024-04-26T21:22:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34300.178002 secs 2024-04-26T21:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:39 smithi096 ceph-mon[32753]: pgmap v18563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34297.226799 secs 2024-04-26T21:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34300.178002 secs 2024-04-26T21:22:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:39 smithi194 ceph-mon[33017]: pgmap v18563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:22:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34297.226799 secs 2024-04-26T21:22:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34300.178002 secs 2024-04-26T21:22:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:41 smithi033 ceph-mon[27845]: pgmap v18564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:41 smithi096 ceph-mon[32753]: pgmap v18564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:41 smithi194 ceph-mon[33017]: pgmap v18564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:22:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:42.956+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:42.956+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:22:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:42.956+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:43 smithi033 ceph-mon[27845]: pgmap v18565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:43 smithi096 ceph-mon[32753]: pgmap v18565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:43 smithi194 ceph-mon[33017]: pgmap v18565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34302.226877 secs 2024-04-26T21:22:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34305.178122 secs 2024-04-26T21:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34302.226877 secs 2024-04-26T21:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34305.178122 secs 2024-04-26T21:22:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34302.226877 secs 2024-04-26T21:22:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34305.178122 secs 2024-04-26T21:22:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:45 smithi033 ceph-mon[27845]: pgmap v18566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:45 smithi096 ceph-mon[32753]: pgmap v18566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:45 smithi194 ceph-mon[33017]: pgmap v18566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:47 smithi033 ceph-mon[27845]: pgmap v18567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:47 smithi096 ceph-mon[32753]: pgmap v18567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:47 smithi194 ceph-mon[33017]: pgmap v18567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:49 smithi033 ceph-mon[27845]: pgmap v18568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34307.226961 secs 2024-04-26T21:22:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34310.178232 secs 2024-04-26T21:22:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:49 smithi096 ceph-mon[32753]: pgmap v18568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34307.226961 secs 2024-04-26T21:22:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34310.178232 secs 2024-04-26T21:22:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:49 smithi194 ceph-mon[33017]: pgmap v18568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34307.226961 secs 2024-04-26T21:22:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34310.178232 secs 2024-04-26T21:22:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:51 smithi033 ceph-mon[27845]: pgmap v18569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:51 smithi096 ceph-mon[32753]: pgmap v18569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:51 smithi194 ceph-mon[33017]: pgmap v18569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:53 smithi096 ceph-mon[32753]: pgmap v18570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:53.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34312.226990 secs 2024-04-26T21:22:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:53 smithi194 ceph-mon[33017]: pgmap v18570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34312.226990 secs 2024-04-26T21:22:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:53 smithi033 ceph-mon[27845]: pgmap v18570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34312.226990 secs 2024-04-26T21:22:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34315.178384 secs 2024-04-26T21:22:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34315.178384 secs 2024-04-26T21:22:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34315.178384 secs 2024-04-26T21:22:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:55 smithi194 ceph-mon[33017]: pgmap v18571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:55 smithi033 ceph-mon[27845]: pgmap v18571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:55 smithi096 ceph-mon[32753]: pgmap v18571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:57 smithi194 ceph-mon[33017]: pgmap v18572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:57 smithi033 ceph-mon[27845]: pgmap v18572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:57 smithi096 ceph-mon[32753]: pgmap v18572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:22:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:57.957+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:57.957+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:22:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:22:57.957+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:22:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34317.227052 secs 2024-04-26T21:22:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34317.227052 secs 2024-04-26T21:22:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34317.227052 secs 2024-04-26T21:22:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:59 smithi194 ceph-mon[33017]: pgmap v18573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34320.178461 secs 2024-04-26T21:22:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:59 smithi033 ceph-mon[27845]: pgmap v18573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34320.178461 secs 2024-04-26T21:22:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:59 smithi096 ceph-mon[32753]: pgmap v18573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:22:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34320.178461 secs 2024-04-26T21:23:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:23:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:23:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:23:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:01 smithi033 ceph-mon[27845]: pgmap v18574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:01.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:01 smithi033 ceph-mon[27845]: pgmap v18575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:23:01.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:01 smithi194 ceph-mon[33017]: pgmap v18574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:01.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:01 smithi194 ceph-mon[33017]: pgmap v18575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:23:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:01 smithi096 ceph-mon[32753]: pgmap v18574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:01.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:01 smithi096 ceph-mon[32753]: pgmap v18575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:23:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:03 smithi194 ceph-mon[33017]: pgmap v18576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34322.227110 secs 2024-04-26T21:23:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:03 smithi033 ceph-mon[27845]: pgmap v18576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34322.227110 secs 2024-04-26T21:23:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:03 smithi096 ceph-mon[32753]: pgmap v18576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34322.227110 secs 2024-04-26T21:23:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34325.178580 secs 2024-04-26T21:23:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34325.178580 secs 2024-04-26T21:23:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34325.178580 secs 2024-04-26T21:23:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:05 smithi194 ceph-mon[33017]: pgmap v18577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:05 smithi033 ceph-mon[27845]: pgmap v18577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:05 smithi096 ceph-mon[32753]: pgmap v18577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:07 smithi194 ceph-mon[33017]: pgmap v18578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:07 smithi033 ceph-mon[27845]: pgmap v18578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:07 smithi096 ceph-mon[32753]: pgmap v18578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34327.227172 secs 2024-04-26T21:23:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34327.227172 secs 2024-04-26T21:23:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34327.227172 secs 2024-04-26T21:23:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:09 smithi194 ceph-mon[33017]: pgmap v18579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34330.178728 secs 2024-04-26T21:23:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:09 smithi033 ceph-mon[27845]: pgmap v18579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34330.178728 secs 2024-04-26T21:23:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:09 smithi096 ceph-mon[32753]: pgmap v18579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34330.178728 secs 2024-04-26T21:23:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:11 smithi194 ceph-mon[33017]: pgmap v18580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:23:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:11 smithi033 ceph-mon[27845]: pgmap v18580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:23:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:11 smithi096 ceph-mon[32753]: pgmap v18580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:23:12.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:12.973+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:12.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:12.973+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:12.974 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:12.973+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:23:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:13 smithi194 ceph-mon[33017]: pgmap v18581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34332.227252 secs 2024-04-26T21:23:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:13 smithi033 ceph-mon[27845]: pgmap v18581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34332.227252 secs 2024-04-26T21:23:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:13 smithi096 ceph-mon[32753]: pgmap v18581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:13.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34332.227252 secs 2024-04-26T21:23:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34335.178865 secs 2024-04-26T21:23:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34335.178865 secs 2024-04-26T21:23:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34335.178865 secs 2024-04-26T21:23:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:15 smithi194 ceph-mon[33017]: pgmap v18582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:15 smithi033 ceph-mon[27845]: pgmap v18582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:15 smithi096 ceph-mon[32753]: pgmap v18582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:17.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:17 smithi194 ceph-mon[33017]: pgmap v18583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:17 smithi033 ceph-mon[27845]: pgmap v18583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:17 smithi096 ceph-mon[32753]: pgmap v18583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34337.227314 secs 2024-04-26T21:23:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34337.227314 secs 2024-04-26T21:23:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34337.227314 secs 2024-04-26T21:23:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:19 smithi194 ceph-mon[33017]: pgmap v18584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34340.178989 secs 2024-04-26T21:23:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:19 smithi033 ceph-mon[27845]: pgmap v18584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34340.178989 secs 2024-04-26T21:23:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:19 smithi096 ceph-mon[32753]: pgmap v18584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34340.178989 secs 2024-04-26T21:23:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:21 smithi194 ceph-mon[33017]: pgmap v18585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:21 smithi033 ceph-mon[27845]: pgmap v18585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:21 smithi096 ceph-mon[32753]: pgmap v18585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:23 smithi194 ceph-mon[33017]: pgmap v18586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34342.227399 secs 2024-04-26T21:23:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:23 smithi033 ceph-mon[27845]: pgmap v18586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34342.227399 secs 2024-04-26T21:23:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:23 smithi096 ceph-mon[32753]: pgmap v18586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34342.227399 secs 2024-04-26T21:23:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34345.179126 secs 2024-04-26T21:23:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34345.179126 secs 2024-04-26T21:23:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34345.179126 secs 2024-04-26T21:23:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:25 smithi194 ceph-mon[33017]: pgmap v18587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:25 smithi033 ceph-mon[27845]: pgmap v18587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:25 smithi096 ceph-mon[32753]: pgmap v18587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:27 smithi194 ceph-mon[33017]: pgmap v18588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:27.643+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:23:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:27.643+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:23:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:27.643+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 269 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T21:23:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:27.644+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 269 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:23:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:27 smithi033 ceph-mon[27845]: pgmap v18588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:27 smithi096 ceph-mon[32753]: pgmap v18588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:27.985+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:27.985+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:27.985+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:23:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34347.227480 secs 2024-04-26T21:23:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34347.227480 secs 2024-04-26T21:23:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34347.227480 secs 2024-04-26T21:23:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:29 smithi194 ceph-mon[33017]: pgmap v18589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34350.179224 secs 2024-04-26T21:23:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:29 smithi033 ceph-mon[27845]: pgmap v18589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34350.179224 secs 2024-04-26T21:23:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:29 smithi096 ceph-mon[32753]: pgmap v18589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34350.179224 secs 2024-04-26T21:23:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:31 smithi194 ceph-mon[33017]: pgmap v18590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:31 smithi033 ceph-mon[27845]: pgmap v18590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:31 smithi096 ceph-mon[32753]: pgmap v18590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:33 smithi194 ceph-mon[33017]: pgmap v18591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34352.227562 secs 2024-04-26T21:23:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:33 smithi033 ceph-mon[27845]: pgmap v18591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34352.227562 secs 2024-04-26T21:23:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:33 smithi096 ceph-mon[32753]: pgmap v18591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34352.227562 secs 2024-04-26T21:23:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34355.179378 secs 2024-04-26T21:23:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34355.179378 secs 2024-04-26T21:23:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34355.179378 secs 2024-04-26T21:23:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:35 smithi033 ceph-mon[27845]: pgmap v18592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:35 smithi096 ceph-mon[32753]: pgmap v18592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:35 smithi194 ceph-mon[33017]: pgmap v18592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:37 smithi033 ceph-mon[27845]: pgmap v18593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:37 smithi096 ceph-mon[32753]: pgmap v18593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:37.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:37 smithi194 ceph-mon[33017]: pgmap v18593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34357.227647 secs 2024-04-26T21:23:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34357.227647 secs 2024-04-26T21:23:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34357.227647 secs 2024-04-26T21:23:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:39 smithi033 ceph-mon[27845]: pgmap v18594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34360.179477 secs 2024-04-26T21:23:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:39 smithi096 ceph-mon[32753]: pgmap v18594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34360.179477 secs 2024-04-26T21:23:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:39 smithi194 ceph-mon[33017]: pgmap v18594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34360.179477 secs 2024-04-26T21:23:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:40 smithi033 ceph-mon[27845]: pgmap v18595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:40 smithi096 ceph-mon[32753]: pgmap v18595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:40 smithi194 ceph-mon[33017]: pgmap v18595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:42 smithi194 ceph-mon[33017]: pgmap v18596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:42 smithi033 ceph-mon[27845]: pgmap v18596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:42 smithi096 ceph-mon[32753]: pgmap v18596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:42.999+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:42.999+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:42.999+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:23:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34362.227730 secs 2024-04-26T21:23:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34362.227730 secs 2024-04-26T21:23:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34362.227730 secs 2024-04-26T21:23:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:44 smithi194 ceph-mon[33017]: pgmap v18597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34365.179632 secs 2024-04-26T21:23:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:44 smithi033 ceph-mon[27845]: pgmap v18597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34365.179632 secs 2024-04-26T21:23:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:44 smithi096 ceph-mon[32753]: pgmap v18597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34365.179632 secs 2024-04-26T21:23:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:46 smithi194 ceph-mon[33017]: pgmap v18598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:46 smithi033 ceph-mon[27845]: pgmap v18598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:46 smithi096 ceph-mon[32753]: pgmap v18598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:48 smithi194 ceph-mon[33017]: pgmap v18599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34367.227807 secs 2024-04-26T21:23:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:48 smithi033 ceph-mon[27845]: pgmap v18599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34367.227807 secs 2024-04-26T21:23:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:48 smithi096 ceph-mon[32753]: pgmap v18599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34367.227807 secs 2024-04-26T21:23:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34370.179711 secs 2024-04-26T21:23:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34370.179711 secs 2024-04-26T21:23:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34370.179711 secs 2024-04-26T21:23:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:50 smithi194 ceph-mon[33017]: pgmap v18600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:50 smithi033 ceph-mon[27845]: pgmap v18600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:50 smithi096 ceph-mon[32753]: pgmap v18600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:23:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:52 smithi194 ceph-mon[33017]: pgmap v18601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:52 smithi033 ceph-mon[27845]: pgmap v18601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:52 smithi096 ceph-mon[32753]: pgmap v18601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:23:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34372.227889 secs 2024-04-26T21:23:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34372.227889 secs 2024-04-26T21:23:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34372.227889 secs 2024-04-26T21:23:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:54 smithi194 ceph-mon[33017]: pgmap v18602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34375.179830 secs 2024-04-26T21:23:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:54 smithi033 ceph-mon[27845]: pgmap v18602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34375.179830 secs 2024-04-26T21:23:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:54 smithi096 ceph-mon[32753]: pgmap v18602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34375.179830 secs 2024-04-26T21:23:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:56 smithi194 ceph-mon[33017]: pgmap v18603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:56 smithi033 ceph-mon[27845]: pgmap v18603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:56.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:56 smithi096 ceph-mon[32753]: pgmap v18603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:23:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:58.013+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:58.013+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:23:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:23:58.013+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:23:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:58 smithi194 ceph-mon[33017]: pgmap v18604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34377.227996 secs 2024-04-26T21:23:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:58 smithi033 ceph-mon[27845]: pgmap v18604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34377.227996 secs 2024-04-26T21:23:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:58 smithi096 ceph-mon[32753]: pgmap v18604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:23:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34377.227996 secs 2024-04-26T21:23:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:23:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34380.179949 secs 2024-04-26T21:23:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:23:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34380.179949 secs 2024-04-26T21:23:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:23:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34380.179949 secs 2024-04-26T21:24:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:00 smithi194 ceph-mon[33017]: pgmap v18605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:24:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:00 smithi033 ceph-mon[27845]: pgmap v18605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:24:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:00 smithi096 ceph-mon[32753]: pgmap v18605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:24:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:00.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:02 smithi194 ceph-mon[33017]: pgmap v18606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:24:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:02 smithi033 ceph-mon[27845]: pgmap v18606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:24:02.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:02 smithi096 ceph-mon[32753]: pgmap v18606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:24:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:02.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34382.228067 secs 2024-04-26T21:24:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:24:03.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:03 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34382.228067 secs 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:24:03.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:03 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:24:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34382.228067 secs 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:24:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:03 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:24:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:04 smithi194 ceph-mon[33017]: pgmap v18607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34385.180070 secs 2024-04-26T21:24:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:04 smithi033 ceph-mon[27845]: pgmap v18607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34385.180070 secs 2024-04-26T21:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:04 smithi096 ceph-mon[32753]: pgmap v18607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34385.180070 secs 2024-04-26T21:24:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:06.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: pgmap v18608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:24:06.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:24:06.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:06.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:06.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: pgmap v18609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:24:06.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: pgmap v18610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:06.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: pgmap v18608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:06.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:24:06.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:24:06.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:06.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:06.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: pgmap v18609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:24:06.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: pgmap v18610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:06.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: pgmap v18608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:07.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:24:07.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:24:07.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:07.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:07.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: pgmap v18609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:24:07.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: pgmap v18610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:07.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:24:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34387.228126 secs 2024-04-26T21:24:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:09 smithi033 ceph-mon[27845]: pgmap v18611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T21:24:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34387.228126 secs 2024-04-26T21:24:09.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:09 smithi096 ceph-mon[32753]: pgmap v18611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T21:24:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34387.228126 secs 2024-04-26T21:24:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:09 smithi194 ceph-mon[33017]: pgmap v18611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-26T21:24:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34390.180185 secs 2024-04-26T21:24:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34390.180185 secs 2024-04-26T21:24:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34390.180185 secs 2024-04-26T21:24:11.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:11 smithi033 ceph-mon[27845]: pgmap v18612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:11 smithi096 ceph-mon[32753]: pgmap v18612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:11 smithi194 ceph-mon[33017]: pgmap v18612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:13.025+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:13.025+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:13.025+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:24:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:13 smithi033 ceph-mon[27845]: pgmap v18613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:13 smithi096 ceph-mon[32753]: pgmap v18613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:13 smithi194 ceph-mon[33017]: pgmap v18613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:24:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34392.228184 secs 2024-04-26T21:24:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34395.180303 secs 2024-04-26T21:24:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34392.228184 secs 2024-04-26T21:24:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34395.180303 secs 2024-04-26T21:24:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34392.228184 secs 2024-04-26T21:24:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34395.180303 secs 2024-04-26T21:24:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:15 smithi033 ceph-mon[27845]: pgmap v18614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:24:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:15 smithi096 ceph-mon[32753]: pgmap v18614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:24:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:15 smithi194 ceph-mon[33017]: pgmap v18614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:24:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:17 smithi033 ceph-mon[27845]: pgmap v18615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:24:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:17 smithi096 ceph-mon[32753]: pgmap v18615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:24:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:17 smithi194 ceph-mon[33017]: pgmap v18615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:24:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34397.228245 secs 2024-04-26T21:24:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:19 smithi033 ceph-mon[27845]: pgmap v18616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34400.180419 secs 2024-04-26T21:24:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34397.228245 secs 2024-04-26T21:24:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:19 smithi096 ceph-mon[32753]: pgmap v18616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34400.180419 secs 2024-04-26T21:24:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34397.228245 secs 2024-04-26T21:24:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:19 smithi194 ceph-mon[33017]: pgmap v18616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34400.180419 secs 2024-04-26T21:24:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:21 smithi033 ceph-mon[27845]: pgmap v18617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:21 smithi096 ceph-mon[32753]: pgmap v18617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:21.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:21 smithi194 ceph-mon[33017]: pgmap v18617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:23 smithi096 ceph-mon[32753]: pgmap v18618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:23.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34402.228327 secs 2024-04-26T21:24:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:23 smithi194 ceph-mon[33017]: pgmap v18618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34402.228327 secs 2024-04-26T21:24:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:23 smithi033 ceph-mon[27845]: pgmap v18618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34402.228327 secs 2024-04-26T21:24:24.452 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34405.180533 secs 2024-04-26T21:24:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34405.180533 secs 2024-04-26T21:24:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34405.180533 secs 2024-04-26T21:24:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:25 smithi194 ceph-mon[33017]: pgmap v18619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:25 smithi033 ceph-mon[27845]: pgmap v18619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:25 smithi096 ceph-mon[32753]: pgmap v18619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:27 smithi194 ceph-mon[33017]: pgmap v18620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:27 smithi033 ceph-mon[27845]: pgmap v18620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:27 smithi096 ceph-mon[32753]: pgmap v18620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:28.040+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:28.040+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:28.040+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:24:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34407.228411 secs 2024-04-26T21:24:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34407.228411 secs 2024-04-26T21:24:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34407.228411 secs 2024-04-26T21:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:29 smithi194 ceph-mon[33017]: pgmap v18621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34410.180648 secs 2024-04-26T21:24:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:29 smithi033 ceph-mon[27845]: pgmap v18621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34410.180648 secs 2024-04-26T21:24:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:29 smithi096 ceph-mon[32753]: pgmap v18621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34410.180648 secs 2024-04-26T21:24:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:31 smithi194 ceph-mon[33017]: pgmap v18622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:31 smithi033 ceph-mon[27845]: pgmap v18622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:31 smithi096 ceph-mon[32753]: pgmap v18622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:33 smithi194 ceph-mon[33017]: pgmap v18623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34412.228469 secs 2024-04-26T21:24:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:33 smithi033 ceph-mon[27845]: pgmap v18623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34412.228469 secs 2024-04-26T21:24:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:33 smithi096 ceph-mon[32753]: pgmap v18623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34412.228469 secs 2024-04-26T21:24:34.513 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34415.180765 secs 2024-04-26T21:24:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34415.180765 secs 2024-04-26T21:24:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34415.180765 secs 2024-04-26T21:24:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:35 smithi194 ceph-mon[33017]: pgmap v18624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:35 smithi033 ceph-mon[27845]: pgmap v18624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:35 smithi096 ceph-mon[32753]: pgmap v18624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:37 smithi194 ceph-mon[33017]: pgmap v18625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:37 smithi033 ceph-mon[27845]: pgmap v18625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:37 smithi096 ceph-mon[32753]: pgmap v18625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34417.228553 secs 2024-04-26T21:24:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34417.228553 secs 2024-04-26T21:24:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34417.228553 secs 2024-04-26T21:24:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:39 smithi194 ceph-mon[33017]: pgmap v18626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34420.180881 secs 2024-04-26T21:24:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:39 smithi033 ceph-mon[27845]: pgmap v18626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34420.180881 secs 2024-04-26T21:24:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:39 smithi096 ceph-mon[32753]: pgmap v18626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:39.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34420.180881 secs 2024-04-26T21:24:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:41 smithi194 ceph-mon[33017]: pgmap v18627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:41 smithi033 ceph-mon[27845]: pgmap v18627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:41 smithi096 ceph-mon[32753]: pgmap v18627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:43.047+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:43.047+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:24:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:43.047+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:43 smithi194 ceph-mon[33017]: pgmap v18628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34422.228634 secs 2024-04-26T21:24:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:43 smithi033 ceph-mon[27845]: pgmap v18628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34422.228634 secs 2024-04-26T21:24:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:43 smithi096 ceph-mon[32753]: pgmap v18628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34422.228634 secs 2024-04-26T21:24:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34425.181007 secs 2024-04-26T21:24:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34425.181007 secs 2024-04-26T21:24:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34425.181007 secs 2024-04-26T21:24:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:45 smithi194 ceph-mon[33017]: pgmap v18629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:45 smithi033 ceph-mon[27845]: pgmap v18629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:45 smithi096 ceph-mon[32753]: pgmap v18629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:47 smithi194 ceph-mon[33017]: pgmap v18630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:47 smithi033 ceph-mon[27845]: pgmap v18630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:47 smithi096 ceph-mon[32753]: pgmap v18630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34427.228720 secs 2024-04-26T21:24:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34427.228720 secs 2024-04-26T21:24:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34427.228720 secs 2024-04-26T21:24:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:49 smithi033 ceph-mon[27845]: pgmap v18631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34430.181132 secs 2024-04-26T21:24:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:49 smithi194 ceph-mon[33017]: pgmap v18631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34430.181132 secs 2024-04-26T21:24:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:49 smithi096 ceph-mon[32753]: pgmap v18631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:49.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34430.181132 secs 2024-04-26T21:24:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:51 smithi194 ceph-mon[33017]: pgmap v18632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:51 smithi033 ceph-mon[27845]: pgmap v18632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:51 smithi096 ceph-mon[32753]: pgmap v18632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:53 smithi194 ceph-mon[33017]: pgmap v18633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:53.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34432.228799 secs 2024-04-26T21:24:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:53 smithi033 ceph-mon[27845]: pgmap v18633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34432.228799 secs 2024-04-26T21:24:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:53 smithi096 ceph-mon[32753]: pgmap v18633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34432.228799 secs 2024-04-26T21:24:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34435.181186 secs 2024-04-26T21:24:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34435.181186 secs 2024-04-26T21:24:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34435.181186 secs 2024-04-26T21:24:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:55 smithi194 ceph-mon[33017]: pgmap v18634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:55 smithi033 ceph-mon[27845]: pgmap v18634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:55 smithi096 ceph-mon[32753]: pgmap v18634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:57 smithi194 ceph-mon[33017]: pgmap v18635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:57 smithi033 ceph-mon[27845]: pgmap v18635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:57 smithi096 ceph-mon[32753]: pgmap v18635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:24:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:58.058+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:58.058+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:24:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:24:58.058+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:24:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34437.228884 secs 2024-04-26T21:24:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34437.228884 secs 2024-04-26T21:24:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34437.228884 secs 2024-04-26T21:24:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:59 smithi194 ceph-mon[33017]: pgmap v18636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:24:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34440.181302 secs 2024-04-26T21:24:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:59 smithi033 ceph-mon[27845]: pgmap v18636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:24:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34440.181302 secs 2024-04-26T21:24:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:59 smithi096 ceph-mon[32753]: pgmap v18636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:24:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:24:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34440.181302 secs 2024-04-26T21:25:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:01 smithi194 ceph-mon[33017]: pgmap v18637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:25:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:01 smithi033 ceph-mon[27845]: pgmap v18637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:25:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:01 smithi096 ceph-mon[32753]: pgmap v18637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:25:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:03 smithi194 ceph-mon[33017]: pgmap v18638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34442.228971 secs 2024-04-26T21:25:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:03 smithi033 ceph-mon[27845]: pgmap v18638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34442.228971 secs 2024-04-26T21:25:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:03 smithi096 ceph-mon[32753]: pgmap v18638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34442.228971 secs 2024-04-26T21:25:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34445.181421 secs 2024-04-26T21:25:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34445.181421 secs 2024-04-26T21:25:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34445.181421 secs 2024-04-26T21:25:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:05 smithi033 ceph-mon[27845]: pgmap v18639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:05 smithi096 ceph-mon[32753]: pgmap v18639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:05 smithi194 ceph-mon[33017]: pgmap v18639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:07 smithi033 ceph-mon[27845]: pgmap v18640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:25:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:07 smithi096 ceph-mon[32753]: pgmap v18640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:25:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:07 smithi194 ceph-mon[33017]: pgmap v18640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:25:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:25:08.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34447.229034 secs 2024-04-26T21:25:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:25:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34447.229034 secs 2024-04-26T21:25:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:25:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34447.229034 secs 2024-04-26T21:25:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:09 smithi033 ceph-mon[27845]: pgmap v18641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:09 smithi033 ceph-mon[27845]: pgmap v18642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:25:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34450.181496 secs 2024-04-26T21:25:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:09 smithi096 ceph-mon[32753]: pgmap v18641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:09 smithi096 ceph-mon[32753]: pgmap v18642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:25:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34450.181496 secs 2024-04-26T21:25:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:09 smithi194 ceph-mon[33017]: pgmap v18641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:25:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:09 smithi194 ceph-mon[33017]: pgmap v18642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:25:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34450.181496 secs 2024-04-26T21:25:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:11 smithi033 ceph-mon[27845]: pgmap v18643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:11 smithi096 ceph-mon[32753]: pgmap v18643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:11 smithi194 ceph-mon[33017]: pgmap v18643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:13.066+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:13.067+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:13.067+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:13 smithi033 ceph-mon[27845]: pgmap v18644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34452.229111 secs 2024-04-26T21:25:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:13 smithi096 ceph-mon[32753]: pgmap v18644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34452.229111 secs 2024-04-26T21:25:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:13 smithi194 ceph-mon[33017]: pgmap v18644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34452.229111 secs 2024-04-26T21:25:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34455.181630 secs 2024-04-26T21:25:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34455.181630 secs 2024-04-26T21:25:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34455.181630 secs 2024-04-26T21:25:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:15 smithi033 ceph-mon[27845]: pgmap v18645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:15 smithi096 ceph-mon[32753]: pgmap v18645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:15 smithi194 ceph-mon[33017]: pgmap v18645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:16 smithi033 ceph-mon[27845]: pgmap v18646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:16 smithi096 ceph-mon[32753]: pgmap v18646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:16 smithi194 ceph-mon[33017]: pgmap v18646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:25:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:18 smithi033 ceph-mon[27845]: pgmap v18647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:25:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34457.229197 secs 2024-04-26T21:25:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:18 smithi096 ceph-mon[32753]: pgmap v18647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:25:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34457.229197 secs 2024-04-26T21:25:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:18 smithi194 ceph-mon[33017]: pgmap v18647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:25:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34457.229197 secs 2024-04-26T21:25:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34460.181720 secs 2024-04-26T21:25:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34460.181720 secs 2024-04-26T21:25:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34460.181720 secs 2024-04-26T21:25:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:20 smithi033 ceph-mon[27845]: pgmap v18648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:20 smithi096 ceph-mon[32753]: pgmap v18648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:20 smithi194 ceph-mon[33017]: pgmap v18648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:22 smithi033 ceph-mon[27845]: pgmap v18649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:22 smithi096 ceph-mon[32753]: pgmap v18649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:22 smithi194 ceph-mon[33017]: pgmap v18649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34462.229280 secs 2024-04-26T21:25:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34462.229280 secs 2024-04-26T21:25:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34462.229280 secs 2024-04-26T21:25:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34465.181852 secs 2024-04-26T21:25:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:24 smithi033 ceph-mon[27845]: pgmap v18650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34465.181852 secs 2024-04-26T21:25:25.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:24 smithi096 ceph-mon[32753]: pgmap v18650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34465.181852 secs 2024-04-26T21:25:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:24 smithi194 ceph-mon[33017]: pgmap v18650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:27 smithi033 ceph-mon[27845]: pgmap v18651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:27 smithi096 ceph-mon[32753]: pgmap v18651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:27 smithi194 ceph-mon[33017]: pgmap v18651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:28.082+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:28.082+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:28.082+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:25:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:29 smithi033 ceph-mon[27845]: pgmap v18652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34467.229341 secs 2024-04-26T21:25:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34470.181966 secs 2024-04-26T21:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:29 smithi096 ceph-mon[32753]: pgmap v18652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34467.229341 secs 2024-04-26T21:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34470.181966 secs 2024-04-26T21:25:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:29 smithi194 ceph-mon[33017]: pgmap v18652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34467.229341 secs 2024-04-26T21:25:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34470.181966 secs 2024-04-26T21:25:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:31 smithi096 ceph-mon[32753]: pgmap v18653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:31 smithi194 ceph-mon[33017]: pgmap v18653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:31 smithi033 ceph-mon[27845]: pgmap v18653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:33 smithi194 ceph-mon[33017]: pgmap v18654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:33 smithi033 ceph-mon[27845]: pgmap v18654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:33 smithi096 ceph-mon[32753]: pgmap v18654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34472.229396 secs 2024-04-26T21:25:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34475.182078 secs 2024-04-26T21:25:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34472.229396 secs 2024-04-26T21:25:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34475.182078 secs 2024-04-26T21:25:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34472.229396 secs 2024-04-26T21:25:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34475.182078 secs 2024-04-26T21:25:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:35 smithi033 ceph-mon[27845]: pgmap v18655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:35 smithi194 ceph-mon[33017]: pgmap v18655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:35 smithi096 ceph-mon[32753]: pgmap v18655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:37 smithi194 ceph-mon[33017]: pgmap v18656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:37 smithi033 ceph-mon[27845]: pgmap v18656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:37 smithi096 ceph-mon[32753]: pgmap v18656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34477.229458 secs 2024-04-26T21:25:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34477.229458 secs 2024-04-26T21:25:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34477.229458 secs 2024-04-26T21:25:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:39 smithi194 ceph-mon[33017]: pgmap v18657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34480.182175 secs 2024-04-26T21:25:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:39 smithi033 ceph-mon[27845]: pgmap v18657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34480.182175 secs 2024-04-26T21:25:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:39 smithi096 ceph-mon[32753]: pgmap v18657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34480.182175 secs 2024-04-26T21:25:41.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:41 smithi194 ceph-mon[33017]: pgmap v18658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:41 smithi033 ceph-mon[27845]: pgmap v18658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:41 smithi096 ceph-mon[32753]: pgmap v18658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:43.098+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:43.098+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:43.098+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:25:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:43 smithi194 ceph-mon[33017]: pgmap v18659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34482.229541 secs 2024-04-26T21:25:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:43 smithi033 ceph-mon[27845]: pgmap v18659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34482.229541 secs 2024-04-26T21:25:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:43 smithi096 ceph-mon[32753]: pgmap v18659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34482.229541 secs 2024-04-26T21:25:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34485.182317 secs 2024-04-26T21:25:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34485.182317 secs 2024-04-26T21:25:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34485.182317 secs 2024-04-26T21:25:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:45 smithi194 ceph-mon[33017]: pgmap v18660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:45 smithi033 ceph-mon[27845]: pgmap v18660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:45 smithi096 ceph-mon[32753]: pgmap v18660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:47.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:47 smithi194 ceph-mon[33017]: pgmap v18661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:47 smithi033 ceph-mon[27845]: pgmap v18661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:47 smithi096 ceph-mon[32753]: pgmap v18661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34487.229625 secs 2024-04-26T21:25:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34487.229625 secs 2024-04-26T21:25:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34487.229625 secs 2024-04-26T21:25:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:49 smithi194 ceph-mon[33017]: pgmap v18662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34490.182384 secs 2024-04-26T21:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:49 smithi033 ceph-mon[27845]: pgmap v18662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34490.182384 secs 2024-04-26T21:25:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:49 smithi096 ceph-mon[32753]: pgmap v18662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34490.182384 secs 2024-04-26T21:25:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:51 smithi033 ceph-mon[27845]: pgmap v18663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:51 smithi194 ceph-mon[33017]: pgmap v18663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:51 smithi096 ceph-mon[32753]: pgmap v18663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:53 smithi194 ceph-mon[33017]: pgmap v18664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34492.229709 secs 2024-04-26T21:25:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:53 smithi033 ceph-mon[27845]: pgmap v18664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34492.229709 secs 2024-04-26T21:25:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:53 smithi096 ceph-mon[32753]: pgmap v18664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34492.229709 secs 2024-04-26T21:25:54.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34495.182524 secs 2024-04-26T21:25:54.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34495.182524 secs 2024-04-26T21:25:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34495.182524 secs 2024-04-26T21:25:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:55 smithi194 ceph-mon[33017]: pgmap v18665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:55 smithi033 ceph-mon[27845]: pgmap v18665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:55 smithi096 ceph-mon[32753]: pgmap v18665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:57 smithi194 ceph-mon[33017]: pgmap v18666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:57 smithi033 ceph-mon[27845]: pgmap v18666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:57 smithi096 ceph-mon[32753]: pgmap v18666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:25:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:58.110+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:58.110+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:25:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:25:58.110+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:25:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34497.229794 secs 2024-04-26T21:25:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34497.229794 secs 2024-04-26T21:25:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34497.229794 secs 2024-04-26T21:25:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:59 smithi194 ceph-mon[33017]: pgmap v18667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:25:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34500.182638 secs 2024-04-26T21:25:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:59 smithi033 ceph-mon[27845]: pgmap v18667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:25:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34500.182638 secs 2024-04-26T21:25:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:59 smithi096 ceph-mon[32753]: pgmap v18667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:25:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:25:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34500.182638 secs 2024-04-26T21:26:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:01 smithi194 ceph-mon[33017]: pgmap v18668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:01 smithi033 ceph-mon[27845]: pgmap v18668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:01 smithi096 ceph-mon[32753]: pgmap v18668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:03 smithi194 ceph-mon[33017]: pgmap v18669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34502.229876 secs 2024-04-26T21:26:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:03 smithi033 ceph-mon[27845]: pgmap v18669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34502.229876 secs 2024-04-26T21:26:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:03 smithi096 ceph-mon[32753]: pgmap v18669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34502.229876 secs 2024-04-26T21:26:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34505.182775 secs 2024-04-26T21:26:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34505.182775 secs 2024-04-26T21:26:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34505.182775 secs 2024-04-26T21:26:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:05 smithi194 ceph-mon[33017]: pgmap v18670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:05 smithi033 ceph-mon[27845]: pgmap v18670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:05 smithi096 ceph-mon[32753]: pgmap v18670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:07 smithi194 ceph-mon[33017]: pgmap v18671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:07 smithi033 ceph-mon[27845]: pgmap v18671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:07 smithi096 ceph-mon[32753]: pgmap v18671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34507.229959 secs 2024-04-26T21:26:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34507.229959 secs 2024-04-26T21:26:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34507.229959 secs 2024-04-26T21:26:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:09 smithi194 ceph-mon[33017]: pgmap v18672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34510.182925 secs 2024-04-26T21:26:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:09 smithi033 ceph-mon[27845]: pgmap v18672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34510.182925 secs 2024-04-26T21:26:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:09 smithi096 ceph-mon[32753]: pgmap v18672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34510.182925 secs 2024-04-26T21:26:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:26:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:26:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:26:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:11 smithi194 ceph-mon[33017]: pgmap v18673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:11 smithi194 ceph-mon[33017]: pgmap v18674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:26:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:11 smithi033 ceph-mon[27845]: pgmap v18673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:11 smithi033 ceph-mon[27845]: pgmap v18674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:26:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:11 smithi096 ceph-mon[32753]: pgmap v18673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:11 smithi096 ceph-mon[32753]: pgmap v18674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:26:13.127 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:13.126+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:13.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:13.127+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:13.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:13.127+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:26:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:13 smithi033 ceph-mon[27845]: pgmap v18675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34512.230005 secs 2024-04-26T21:26:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:13 smithi096 ceph-mon[32753]: pgmap v18675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34512.230005 secs 2024-04-26T21:26:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:13 smithi194 ceph-mon[33017]: pgmap v18675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34512.230005 secs 2024-04-26T21:26:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34515.183042 secs 2024-04-26T21:26:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34515.183042 secs 2024-04-26T21:26:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34515.183042 secs 2024-04-26T21:26:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:15 smithi033 ceph-mon[27845]: pgmap v18676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:15 smithi096 ceph-mon[32753]: pgmap v18676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:15 smithi194 ceph-mon[33017]: pgmap v18676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:17.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:17 smithi033 ceph-mon[27845]: pgmap v18677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:17 smithi096 ceph-mon[32753]: pgmap v18677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:17 smithi194 ceph-mon[33017]: pgmap v18677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34517.230066 secs 2024-04-26T21:26:18.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34517.230066 secs 2024-04-26T21:26:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34517.230066 secs 2024-04-26T21:26:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:19 smithi033 ceph-mon[27845]: pgmap v18678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34520.183178 secs 2024-04-26T21:26:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:19 smithi096 ceph-mon[32753]: pgmap v18678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34520.183178 secs 2024-04-26T21:26:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:19 smithi194 ceph-mon[33017]: pgmap v18678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:19.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34520.183178 secs 2024-04-26T21:26:21.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:21 smithi033 ceph-mon[27845]: pgmap v18679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:26:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:21 smithi096 ceph-mon[32753]: pgmap v18679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:26:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T21:26:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:26:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef004f000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:26:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef004f000 comp rx=0 tx=0).stop 2024-04-26T21:26:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T21:26:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T21:26:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.812+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.813+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:21.815+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0012040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:21 smithi194 ceph-mon[33017]: pgmap v18679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:26:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:23 smithi033 ceph-mon[27845]: pgmap v18680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34522.230150 secs 2024-04-26T21:26:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:23 smithi096 ceph-mon[32753]: pgmap v18680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34522.230150 secs 2024-04-26T21:26:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:23 smithi194 ceph-mon[33017]: pgmap v18680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34522.230150 secs 2024-04-26T21:26:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34525.183301 secs 2024-04-26T21:26:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34525.183301 secs 2024-04-26T21:26:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34525.183301 secs 2024-04-26T21:26:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:25 smithi033 ceph-mon[27845]: pgmap v18681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:25 smithi096 ceph-mon[32753]: pgmap v18681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:25 smithi194 ceph-mon[33017]: pgmap v18681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:26 smithi194 ceph-mon[33017]: pgmap v18682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:26 smithi033 ceph-mon[27845]: pgmap v18682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:26 smithi096 ceph-mon[32753]: pgmap v18682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:27.674+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:26:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:27.674+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:26:27.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:27.675+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 270 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T21:26:27.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:27.675+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 270 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:26:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:28.134+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:28.134+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:28.134+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:26:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:28 smithi194 ceph-mon[33017]: pgmap v18683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34527.230233 secs 2024-04-26T21:26:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:28 smithi033 ceph-mon[27845]: pgmap v18683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34527.230233 secs 2024-04-26T21:26:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:28 smithi096 ceph-mon[32753]: pgmap v18683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34527.230233 secs 2024-04-26T21:26:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34530.183365 secs 2024-04-26T21:26:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34530.183365 secs 2024-04-26T21:26:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34530.183365 secs 2024-04-26T21:26:30.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:30 smithi194 ceph-mon[33017]: pgmap v18684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:30 smithi033 ceph-mon[27845]: pgmap v18684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:30 smithi096 ceph-mon[32753]: pgmap v18684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:32 smithi194 ceph-mon[33017]: pgmap v18685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:32 smithi033 ceph-mon[27845]: pgmap v18685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:32 smithi096 ceph-mon[32753]: pgmap v18685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34532.230318 secs 2024-04-26T21:26:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34532.230318 secs 2024-04-26T21:26:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34532.230318 secs 2024-04-26T21:26:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:34 smithi194 ceph-mon[33017]: pgmap v18686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34535.183446 secs 2024-04-26T21:26:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:34 smithi033 ceph-mon[27845]: pgmap v18686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34535.183446 secs 2024-04-26T21:26:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:34 smithi096 ceph-mon[32753]: pgmap v18686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34535.183446 secs 2024-04-26T21:26:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:36 smithi194 ceph-mon[33017]: pgmap v18687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:36 smithi033 ceph-mon[27845]: pgmap v18687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:36 smithi096 ceph-mon[32753]: pgmap v18687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:38 smithi194 ceph-mon[33017]: pgmap v18688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34537.230378 secs 2024-04-26T21:26:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:38 smithi033 ceph-mon[27845]: pgmap v18688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34537.230378 secs 2024-04-26T21:26:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:38 smithi096 ceph-mon[32753]: pgmap v18688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34537.230378 secs 2024-04-26T21:26:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34540.183598 secs 2024-04-26T21:26:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34540.183598 secs 2024-04-26T21:26:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34540.183598 secs 2024-04-26T21:26:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:40 smithi194 ceph-mon[33017]: pgmap v18689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:40 smithi033 ceph-mon[27845]: pgmap v18689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:40 smithi096 ceph-mon[32753]: pgmap v18689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:42 smithi194 ceph-mon[33017]: pgmap v18690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:42.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:42 smithi033 ceph-mon[27845]: pgmap v18690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:42 smithi096 ceph-mon[32753]: pgmap v18690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:43.146+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:43.146+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:43.146+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:26:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34542.230459 secs 2024-04-26T21:26:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34542.230459 secs 2024-04-26T21:26:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34542.230459 secs 2024-04-26T21:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:44 smithi033 ceph-mon[27845]: pgmap v18691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34545.183681 secs 2024-04-26T21:26:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:44 smithi096 ceph-mon[32753]: pgmap v18691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34545.183681 secs 2024-04-26T21:26:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:44 smithi194 ceph-mon[33017]: pgmap v18691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34545.183681 secs 2024-04-26T21:26:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:46 smithi033 ceph-mon[27845]: pgmap v18692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:46 smithi096 ceph-mon[32753]: pgmap v18692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:46 smithi194 ceph-mon[33017]: pgmap v18692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:48 smithi033 ceph-mon[27845]: pgmap v18693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34547.230543 secs 2024-04-26T21:26:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:48 smithi096 ceph-mon[32753]: pgmap v18693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34547.230543 secs 2024-04-26T21:26:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:48 smithi194 ceph-mon[33017]: pgmap v18693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34547.230543 secs 2024-04-26T21:26:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34550.183772 secs 2024-04-26T21:26:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34550.183772 secs 2024-04-26T21:26:50.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34550.183772 secs 2024-04-26T21:26:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:50 smithi033 ceph-mon[27845]: pgmap v18694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:50 smithi096 ceph-mon[32753]: pgmap v18694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:50 smithi194 ceph-mon[33017]: pgmap v18694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:26:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:52 smithi033 ceph-mon[27845]: pgmap v18695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:52 smithi096 ceph-mon[32753]: pgmap v18695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:53.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:52 smithi194 ceph-mon[33017]: pgmap v18695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:26:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34552.230624 secs 2024-04-26T21:26:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34552.230624 secs 2024-04-26T21:26:54.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34552.230624 secs 2024-04-26T21:26:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:54 smithi033 ceph-mon[27845]: pgmap v18696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34555.183913 secs 2024-04-26T21:26:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:54 smithi096 ceph-mon[32753]: pgmap v18696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34555.183913 secs 2024-04-26T21:26:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:54 smithi194 ceph-mon[33017]: pgmap v18696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34555.183913 secs 2024-04-26T21:26:56.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:56 smithi033 ceph-mon[27845]: pgmap v18697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:56 smithi096 ceph-mon[32753]: pgmap v18697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:56 smithi194 ceph-mon[33017]: pgmap v18697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:26:58.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:58.152+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:58.154 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:58.152+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:26:58.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:26:58.152+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:26:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:58 smithi033 ceph-mon[27845]: pgmap v18698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34557.230707 secs 2024-04-26T21:26:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:58 smithi096 ceph-mon[32753]: pgmap v18698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34557.230707 secs 2024-04-26T21:26:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:58 smithi194 ceph-mon[33017]: pgmap v18698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:26:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34557.230707 secs 2024-04-26T21:26:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:26:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34560.184013 secs 2024-04-26T21:27:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:26:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34560.184013 secs 2024-04-26T21:27:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:26:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34560.184013 secs 2024-04-26T21:27:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:00 smithi194 ceph-mon[33017]: pgmap v18699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:00 smithi033 ceph-mon[27845]: pgmap v18699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:00 smithi096 ceph-mon[32753]: pgmap v18699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:02 smithi194 ceph-mon[33017]: pgmap v18700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:27:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:02 smithi033 ceph-mon[27845]: pgmap v18700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:27:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:02 smithi096 ceph-mon[32753]: pgmap v18700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:27:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34562.230786 secs 2024-04-26T21:27:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34562.230786 secs 2024-04-26T21:27:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34562.230786 secs 2024-04-26T21:27:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:04 smithi194 ceph-mon[33017]: pgmap v18701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34565.184143 secs 2024-04-26T21:27:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:04 smithi033 ceph-mon[27845]: pgmap v18701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34565.184143 secs 2024-04-26T21:27:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:04 smithi096 ceph-mon[32753]: pgmap v18701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34565.184143 secs 2024-04-26T21:27:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:06 smithi194 ceph-mon[33017]: pgmap v18702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:06 smithi033 ceph-mon[27845]: pgmap v18702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:06 smithi096 ceph-mon[32753]: pgmap v18702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:08 smithi194 ceph-mon[33017]: pgmap v18703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34567.230871 secs 2024-04-26T21:27:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:08 smithi033 ceph-mon[27845]: pgmap v18703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34567.230871 secs 2024-04-26T21:27:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:08 smithi096 ceph-mon[32753]: pgmap v18703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34567.230871 secs 2024-04-26T21:27:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34570.184239 secs 2024-04-26T21:27:10.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34570.184239 secs 2024-04-26T21:27:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34570.184239 secs 2024-04-26T21:27:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:10 smithi194 ceph-mon[33017]: pgmap v18704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:11.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:10 smithi033 ceph-mon[27845]: pgmap v18704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:10 smithi096 ceph-mon[32753]: pgmap v18704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:11 smithi194 ceph-mon[33017]: pgmap v18705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:27:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:11 smithi194 ceph-mon[33017]: pgmap v18706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:12.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:27:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:11 smithi033 ceph-mon[27845]: pgmap v18705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:27:12.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:11 smithi033 ceph-mon[27845]: pgmap v18706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:12.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:27:12.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:12.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:12.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:11 smithi096 ceph-mon[32753]: pgmap v18705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:27:12.185 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:11 smithi096 ceph-mon[32753]: pgmap v18706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:12.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:27:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:12.682+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T21:27:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:12.682+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:27:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:12.682+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 271 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T21:27:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:12.683+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 271 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:27:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:13.168+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:13.169+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:13.169+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:27:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34572.230952 secs 2024-04-26T21:27:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34572.230952 secs 2024-04-26T21:27:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34572.230952 secs 2024-04-26T21:27:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:14 smithi194 ceph-mon[33017]: pgmap v18707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34575.184308 secs 2024-04-26T21:27:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:14 smithi033 ceph-mon[27845]: pgmap v18707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34575.184308 secs 2024-04-26T21:27:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:14 smithi096 ceph-mon[32753]: pgmap v18707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34575.184308 secs 2024-04-26T21:27:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:16 smithi194 ceph-mon[33017]: pgmap v18708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:16 smithi033 ceph-mon[27845]: pgmap v18708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:16 smithi096 ceph-mon[32753]: pgmap v18708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:18 smithi194 ceph-mon[33017]: pgmap v18709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34577.231003 secs 2024-04-26T21:27:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:18 smithi033 ceph-mon[27845]: pgmap v18709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34577.231003 secs 2024-04-26T21:27:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:18 smithi096 ceph-mon[32753]: pgmap v18709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:27:18.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34577.231003 secs 2024-04-26T21:27:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34580.184403 secs 2024-04-26T21:27:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34580.184403 secs 2024-04-26T21:27:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34580.184403 secs 2024-04-26T21:27:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:20 smithi194 ceph-mon[33017]: pgmap v18710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:27:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:20 smithi033 ceph-mon[27845]: pgmap v18710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:27:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:20 smithi096 ceph-mon[32753]: pgmap v18710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:27:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:22 smithi194 ceph-mon[33017]: pgmap v18711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:27:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:22 smithi033 ceph-mon[27845]: pgmap v18711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:27:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:22 smithi096 ceph-mon[32753]: pgmap v18711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:27:23.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34582.231081 secs 2024-04-26T21:27:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34582.231081 secs 2024-04-26T21:27:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34582.231081 secs 2024-04-26T21:27:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:24 smithi096 ceph-mon[32753]: pgmap v18712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34585.184511 secs 2024-04-26T21:27:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:24 smithi194 ceph-mon[33017]: pgmap v18712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34585.184511 secs 2024-04-26T21:27:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:24 smithi033 ceph-mon[27845]: pgmap v18712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34585.184511 secs 2024-04-26T21:27:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:26 smithi194 ceph-mon[33017]: pgmap v18713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:26 smithi033 ceph-mon[27845]: pgmap v18713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:26 smithi096 ceph-mon[32753]: pgmap v18713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:28.184+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:28.184+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:27:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:28.184+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:28 smithi194 ceph-mon[33017]: pgmap v18714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34587.231178 secs 2024-04-26T21:27:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:28 smithi033 ceph-mon[27845]: pgmap v18714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34587.231178 secs 2024-04-26T21:27:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:28 smithi096 ceph-mon[32753]: pgmap v18714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34587.231178 secs 2024-04-26T21:27:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34590.184629 secs 2024-04-26T21:27:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34590.184629 secs 2024-04-26T21:27:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34590.184629 secs 2024-04-26T21:27:30.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:30 smithi194 ceph-mon[33017]: pgmap v18715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:30 smithi033 ceph-mon[27845]: pgmap v18715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:30 smithi096 ceph-mon[32753]: pgmap v18715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:32 smithi194 ceph-mon[33017]: pgmap v18716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:32 smithi033 ceph-mon[27845]: pgmap v18716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:32 smithi096 ceph-mon[32753]: pgmap v18716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34592.231260 secs 2024-04-26T21:27:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34592.231260 secs 2024-04-26T21:27:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34592.231260 secs 2024-04-26T21:27:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:34 smithi194 ceph-mon[33017]: pgmap v18717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34595.184766 secs 2024-04-26T21:27:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:34 smithi033 ceph-mon[27845]: pgmap v18717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34595.184766 secs 2024-04-26T21:27:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:34 smithi096 ceph-mon[32753]: pgmap v18717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34595.184766 secs 2024-04-26T21:27:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:36 smithi194 ceph-mon[33017]: pgmap v18718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:36 smithi033 ceph-mon[27845]: pgmap v18718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:36 smithi096 ceph-mon[32753]: pgmap v18718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:38 smithi194 ceph-mon[33017]: pgmap v18719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34597.231321 secs 2024-04-26T21:27:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:38 smithi033 ceph-mon[27845]: pgmap v18719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34597.231321 secs 2024-04-26T21:27:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:38 smithi096 ceph-mon[32753]: pgmap v18719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34597.231321 secs 2024-04-26T21:27:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34600.184866 secs 2024-04-26T21:27:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34600.184866 secs 2024-04-26T21:27:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34600.184866 secs 2024-04-26T21:27:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:40 smithi033 ceph-mon[27845]: pgmap v18720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:40 smithi096 ceph-mon[32753]: pgmap v18720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:40 smithi194 ceph-mon[33017]: pgmap v18720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:42 smithi033 ceph-mon[27845]: pgmap v18721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:42 smithi096 ceph-mon[32753]: pgmap v18721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:42 smithi194 ceph-mon[33017]: pgmap v18721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:43.200+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:43.200+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:43.200+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:27:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34602.231381 secs 2024-04-26T21:27:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34602.231381 secs 2024-04-26T21:27:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34602.231381 secs 2024-04-26T21:27:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:44 smithi033 ceph-mon[27845]: pgmap v18722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34605.184968 secs 2024-04-26T21:27:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:44 smithi096 ceph-mon[32753]: pgmap v18722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:44.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34605.184968 secs 2024-04-26T21:27:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:44 smithi194 ceph-mon[33017]: pgmap v18722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34605.184968 secs 2024-04-26T21:27:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:46 smithi033 ceph-mon[27845]: pgmap v18723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:46 smithi096 ceph-mon[32753]: pgmap v18723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:46 smithi194 ceph-mon[33017]: pgmap v18723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:48 smithi033 ceph-mon[27845]: pgmap v18724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34607.231444 secs 2024-04-26T21:27:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:48 smithi096 ceph-mon[32753]: pgmap v18724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34607.231444 secs 2024-04-26T21:27:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:48 smithi194 ceph-mon[33017]: pgmap v18724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34607.231444 secs 2024-04-26T21:27:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34610.185080 secs 2024-04-26T21:27:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34610.185080 secs 2024-04-26T21:27:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34610.185080 secs 2024-04-26T21:27:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:50 smithi033 ceph-mon[27845]: pgmap v18725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:50 smithi096 ceph-mon[32753]: pgmap v18725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:50.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:50 smithi194 ceph-mon[33017]: pgmap v18725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:52 smithi033 ceph-mon[27845]: pgmap v18726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:52 smithi096 ceph-mon[32753]: pgmap v18726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:52 smithi194 ceph-mon[33017]: pgmap v18726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34612.231528 secs 2024-04-26T21:27:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34612.231528 secs 2024-04-26T21:27:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34612.231528 secs 2024-04-26T21:27:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:54 smithi033 ceph-mon[27845]: pgmap v18727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34615.185198 secs 2024-04-26T21:27:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:54 smithi096 ceph-mon[32753]: pgmap v18727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:54.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34615.185198 secs 2024-04-26T21:27:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:54 smithi194 ceph-mon[33017]: pgmap v18727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34615.185198 secs 2024-04-26T21:27:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:56 smithi096 ceph-mon[32753]: pgmap v18728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:56 smithi194 ceph-mon[33017]: pgmap v18728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:56.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:56 smithi033 ceph-mon[27845]: pgmap v18728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:27:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:57 smithi194 ceph-mon[33017]: pgmap v18729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:57 smithi033 ceph-mon[27845]: pgmap v18729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:57 smithi096 ceph-mon[32753]: pgmap v18729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:58.211+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:58.211+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:27:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:27:58.211+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:27:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34617.231613 secs 2024-04-26T21:27:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34617.231613 secs 2024-04-26T21:27:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34617.231613 secs 2024-04-26T21:27:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34620.185264 secs 2024-04-26T21:27:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:27:59 smithi194 ceph-mon[33017]: pgmap v18730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34620.185264 secs 2024-04-26T21:27:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:27:59 smithi033 ceph-mon[27845]: pgmap v18730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:27:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34620.185264 secs 2024-04-26T21:27:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:27:59 smithi096 ceph-mon[32753]: pgmap v18730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:02 smithi194 ceph-mon[33017]: pgmap v18731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:02 smithi033 ceph-mon[27845]: pgmap v18731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:02 smithi096 ceph-mon[32753]: pgmap v18731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34622.231694 secs 2024-04-26T21:28:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34622.231694 secs 2024-04-26T21:28:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34622.231694 secs 2024-04-26T21:28:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:04 smithi194 ceph-mon[33017]: pgmap v18732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34625.185348 secs 2024-04-26T21:28:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:04 smithi033 ceph-mon[27845]: pgmap v18732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34625.185348 secs 2024-04-26T21:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:04 smithi096 ceph-mon[32753]: pgmap v18732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34625.185348 secs 2024-04-26T21:28:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:06 smithi194 ceph-mon[33017]: pgmap v18733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:06 smithi033 ceph-mon[27845]: pgmap v18733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:06 smithi096 ceph-mon[32753]: pgmap v18733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:08 smithi194 ceph-mon[33017]: pgmap v18734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34627.231779 secs 2024-04-26T21:28:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:08 smithi033 ceph-mon[27845]: pgmap v18734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34627.231779 secs 2024-04-26T21:28:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:08 smithi096 ceph-mon[32753]: pgmap v18734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34627.231779 secs 2024-04-26T21:28:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34630.185473 secs 2024-04-26T21:28:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34630.185473 secs 2024-04-26T21:28:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34630.185473 secs 2024-04-26T21:28:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:10 smithi194 ceph-mon[33017]: pgmap v18735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:10 smithi033 ceph-mon[27845]: pgmap v18735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:10 smithi096 ceph-mon[32753]: pgmap v18735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:12 smithi194 ceph-mon[33017]: pgmap v18736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:12 smithi033 ceph-mon[27845]: pgmap v18736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:12 smithi096 ceph-mon[32753]: pgmap v18736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:13.227+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:13.227+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:13.227+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:28:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:28:13.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34632.231859 secs 2024-04-26T21:28:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:28:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34632.231859 secs 2024-04-26T21:28:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:28:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34632.231859 secs 2024-04-26T21:28:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:14 smithi194 ceph-mon[33017]: pgmap v18737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:14 smithi194 ceph-mon[33017]: pgmap v18738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34635.185613 secs 2024-04-26T21:28:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:14 smithi033 ceph-mon[27845]: pgmap v18737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:14 smithi033 ceph-mon[27845]: pgmap v18738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34635.185613 secs 2024-04-26T21:28:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:14 smithi096 ceph-mon[32753]: pgmap v18737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:14 smithi096 ceph-mon[32753]: pgmap v18738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34635.185613 secs 2024-04-26T21:28:16.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:16 smithi194 ceph-mon[33017]: pgmap v18739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:16 smithi033 ceph-mon[27845]: pgmap v18739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:16 smithi096 ceph-mon[32753]: pgmap v18739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:18 smithi194 ceph-mon[33017]: pgmap v18740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34637.231947 secs 2024-04-26T21:28:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:18 smithi033 ceph-mon[27845]: pgmap v18740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34637.231947 secs 2024-04-26T21:28:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:18 smithi096 ceph-mon[32753]: pgmap v18740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34637.231947 secs 2024-04-26T21:28:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34640.185681 secs 2024-04-26T21:28:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34640.185681 secs 2024-04-26T21:28:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34640.185681 secs 2024-04-26T21:28:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:20 smithi194 ceph-mon[33017]: pgmap v18741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:20 smithi033 ceph-mon[27845]: pgmap v18741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:20 smithi096 ceph-mon[32753]: pgmap v18741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:22 smithi194 ceph-mon[33017]: pgmap v18742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:22 smithi033 ceph-mon[27845]: pgmap v18742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:22 smithi096 ceph-mon[32753]: pgmap v18742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:28:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34642.232025 secs 2024-04-26T21:28:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34642.232025 secs 2024-04-26T21:28:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34642.232025 secs 2024-04-26T21:28:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:24 smithi194 ceph-mon[33017]: pgmap v18743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:28:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34645.185790 secs 2024-04-26T21:28:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:24 smithi033 ceph-mon[27845]: pgmap v18743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:28:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34645.185790 secs 2024-04-26T21:28:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:24 smithi096 ceph-mon[32753]: pgmap v18743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:28:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34645.185790 secs 2024-04-26T21:28:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:26 smithi194 ceph-mon[33017]: pgmap v18744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:26 smithi033 ceph-mon[27845]: pgmap v18744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:26 smithi096 ceph-mon[32753]: pgmap v18744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:28.242+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:28.242+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:28.242+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:28:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:28 smithi194 ceph-mon[33017]: pgmap v18745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34647.232088 secs 2024-04-26T21:28:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:28 smithi033 ceph-mon[27845]: pgmap v18745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34647.232088 secs 2024-04-26T21:28:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:28 smithi096 ceph-mon[32753]: pgmap v18745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34647.232088 secs 2024-04-26T21:28:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34650.185906 secs 2024-04-26T21:28:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34650.185906 secs 2024-04-26T21:28:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34650.185906 secs 2024-04-26T21:28:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:30 smithi033 ceph-mon[27845]: pgmap v18746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:30 smithi096 ceph-mon[32753]: pgmap v18746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:30 smithi194 ceph-mon[33017]: pgmap v18746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:32 smithi033 ceph-mon[27845]: pgmap v18747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:32 smithi096 ceph-mon[32753]: pgmap v18747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:32 smithi194 ceph-mon[33017]: pgmap v18747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34652.232174 secs 2024-04-26T21:28:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34652.232174 secs 2024-04-26T21:28:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34652.232174 secs 2024-04-26T21:28:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:34 smithi033 ceph-mon[27845]: pgmap v18748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34655.186051 secs 2024-04-26T21:28:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:34 smithi096 ceph-mon[32753]: pgmap v18748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34655.186051 secs 2024-04-26T21:28:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:34 smithi194 ceph-mon[33017]: pgmap v18748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34655.186051 secs 2024-04-26T21:28:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:36 smithi033 ceph-mon[27845]: pgmap v18749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:36 smithi096 ceph-mon[32753]: pgmap v18749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:36 smithi194 ceph-mon[33017]: pgmap v18749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:38 smithi033 ceph-mon[27845]: pgmap v18750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34657.232236 secs 2024-04-26T21:28:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:38 smithi096 ceph-mon[32753]: pgmap v18750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34657.232236 secs 2024-04-26T21:28:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:38 smithi194 ceph-mon[33017]: pgmap v18750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34657.232236 secs 2024-04-26T21:28:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34660.186147 secs 2024-04-26T21:28:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34660.186147 secs 2024-04-26T21:28:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34660.186147 secs 2024-04-26T21:28:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:40 smithi033 ceph-mon[27845]: pgmap v18751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:40 smithi096 ceph-mon[32753]: pgmap v18751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:40 smithi194 ceph-mon[33017]: pgmap v18751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:42 smithi033 ceph-mon[27845]: pgmap v18752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:42 smithi096 ceph-mon[32753]: pgmap v18752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:42 smithi194 ceph-mon[33017]: pgmap v18752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:43.253+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:43.253+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:43.253+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:28:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34662.232319 secs 2024-04-26T21:28:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34662.232319 secs 2024-04-26T21:28:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34662.232319 secs 2024-04-26T21:28:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:44 smithi033 ceph-mon[27845]: pgmap v18753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34665.186272 secs 2024-04-26T21:28:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:44 smithi096 ceph-mon[32753]: pgmap v18753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34665.186272 secs 2024-04-26T21:28:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:44 smithi194 ceph-mon[33017]: pgmap v18753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34665.186272 secs 2024-04-26T21:28:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:46 smithi033 ceph-mon[27845]: pgmap v18754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:46 smithi096 ceph-mon[32753]: pgmap v18754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:46.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:46 smithi194 ceph-mon[33017]: pgmap v18754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:48 smithi096 ceph-mon[32753]: pgmap v18755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:48.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34667.232380 secs 2024-04-26T21:28:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:48 smithi194 ceph-mon[33017]: pgmap v18755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34667.232380 secs 2024-04-26T21:28:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:48 smithi033 ceph-mon[27845]: pgmap v18755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34667.232380 secs 2024-04-26T21:28:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34670.186348 secs 2024-04-26T21:28:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34670.186348 secs 2024-04-26T21:28:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34670.186348 secs 2024-04-26T21:28:50.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:50 smithi194 ceph-mon[33017]: pgmap v18756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:50 smithi033 ceph-mon[27845]: pgmap v18756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:50 smithi096 ceph-mon[32753]: pgmap v18756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:51 smithi194 ceph-mon[33017]: pgmap v18757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:51 smithi033 ceph-mon[27845]: pgmap v18757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:51 smithi096 ceph-mon[32753]: pgmap v18757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:54 smithi194 ceph-mon[33017]: pgmap v18758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34672.232440 secs 2024-04-26T21:28:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:54 smithi033 ceph-mon[27845]: pgmap v18758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34672.232440 secs 2024-04-26T21:28:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:54 smithi096 ceph-mon[32753]: pgmap v18758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34672.232440 secs 2024-04-26T21:28:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34675.186484 secs 2024-04-26T21:28:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34675.186484 secs 2024-04-26T21:28:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34675.186484 secs 2024-04-26T21:28:56.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:56 smithi033 ceph-mon[27845]: pgmap v18759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:56 smithi194 ceph-mon[33017]: pgmap v18759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:56 smithi096 ceph-mon[32753]: pgmap v18759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:58.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:58.267+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:58.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:58.267+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:28:58.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:28:58.267+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:28:58.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:58 smithi194 ceph-mon[33017]: pgmap v18760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:58 smithi033 ceph-mon[27845]: pgmap v18760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:58 smithi096 ceph-mon[32753]: pgmap v18760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:28:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:28:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34677.232524 secs 2024-04-26T21:28:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:28:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34677.232524 secs 2024-04-26T21:28:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:28:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34677.232524 secs 2024-04-26T21:29:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34680.186605 secs 2024-04-26T21:29:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:00 smithi194 ceph-mon[33017]: pgmap v18761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34680.186605 secs 2024-04-26T21:29:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:00 smithi033 ceph-mon[27845]: pgmap v18761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34680.186605 secs 2024-04-26T21:29:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:00 smithi096 ceph-mon[32753]: pgmap v18761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:02 smithi194 ceph-mon[33017]: pgmap v18762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:02 smithi033 ceph-mon[27845]: pgmap v18762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:02 smithi096 ceph-mon[32753]: pgmap v18762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:04 smithi194 ceph-mon[33017]: pgmap v18763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34682.232605 secs 2024-04-26T21:29:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:04 smithi033 ceph-mon[27845]: pgmap v18763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34682.232605 secs 2024-04-26T21:29:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:04 smithi096 ceph-mon[32753]: pgmap v18763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34682.232605 secs 2024-04-26T21:29:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34685.186742 secs 2024-04-26T21:29:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34685.186742 secs 2024-04-26T21:29:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34685.186742 secs 2024-04-26T21:29:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:06 smithi194 ceph-mon[33017]: pgmap v18764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:06 smithi033 ceph-mon[27845]: pgmap v18764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:06 smithi096 ceph-mon[32753]: pgmap v18764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:08 smithi194 ceph-mon[33017]: pgmap v18765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:08 smithi033 ceph-mon[27845]: pgmap v18765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:08 smithi096 ceph-mon[32753]: pgmap v18765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34687.232668 secs 2024-04-26T21:29:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34687.232668 secs 2024-04-26T21:29:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34687.232668 secs 2024-04-26T21:29:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34690.186864 secs 2024-04-26T21:29:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:10 smithi194 ceph-mon[33017]: pgmap v18766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34690.186864 secs 2024-04-26T21:29:10.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:10 smithi033 ceph-mon[27845]: pgmap v18766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34690.186864 secs 2024-04-26T21:29:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:10 smithi096 ceph-mon[32753]: pgmap v18766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:12 smithi194 ceph-mon[33017]: pgmap v18767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:12 smithi033 ceph-mon[27845]: pgmap v18767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:12 smithi096 ceph-mon[32753]: pgmap v18767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:13.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:13.282+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:13.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:13.282+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:13.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:13.282+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:29:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:14 smithi194 ceph-mon[33017]: pgmap v18768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34692.232750 secs 2024-04-26T21:29:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:14 smithi033 ceph-mon[27845]: pgmap v18768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34692.232750 secs 2024-04-26T21:29:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:14 smithi096 ceph-mon[32753]: pgmap v18768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34692.232750 secs 2024-04-26T21:29:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34695.186929 secs 2024-04-26T21:29:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34695.186929 secs 2024-04-26T21:29:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34695.186929 secs 2024-04-26T21:29:16.230 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:16 smithi194 ceph-mon[33017]: pgmap v18769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:16.307 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:16 smithi096 ceph-mon[32753]: pgmap v18769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:16 smithi033 ceph-mon[27845]: pgmap v18769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:18 smithi033 ceph-mon[27845]: pgmap v18770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:18 smithi096 ceph-mon[32753]: pgmap v18770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:18 smithi194 ceph-mon[33017]: pgmap v18770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:18.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34697.232841 secs 2024-04-26T21:29:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34697.232841 secs 2024-04-26T21:29:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34697.232841 secs 2024-04-26T21:29:20.390 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34700.187052 secs 2024-04-26T21:29:20.391 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:20 smithi033 ceph-mon[27845]: pgmap v18771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34700.187052 secs 2024-04-26T21:29:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:20 smithi096 ceph-mon[32753]: pgmap v18771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:20.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34700.187052 secs 2024-04-26T21:29:20.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:20 smithi194 ceph-mon[33017]: pgmap v18771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:21 smithi194 ceph-mon[33017]: pgmap v18772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:29:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:21 smithi194 ceph-mon[33017]: pgmap v18773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:21.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:21 smithi033 ceph-mon[27845]: pgmap v18772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:29:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:21 smithi033 ceph-mon[27845]: pgmap v18773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:21 smithi096 ceph-mon[32753]: pgmap v18772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:29:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:21 smithi096 ceph-mon[32753]: pgmap v18773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:29:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:23 smithi194 ceph-mon[33017]: pgmap v18774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34702.232923 secs 2024-04-26T21:29:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:23 smithi033 ceph-mon[27845]: pgmap v18774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34702.232923 secs 2024-04-26T21:29:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:23 smithi096 ceph-mon[32753]: pgmap v18774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:23.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34702.232923 secs 2024-04-26T21:29:24.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34705.187201 secs 2024-04-26T21:29:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34705.187201 secs 2024-04-26T21:29:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34705.187201 secs 2024-04-26T21:29:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:26 smithi194 ceph-mon[33017]: pgmap v18775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:29:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:26 smithi033 ceph-mon[27845]: pgmap v18775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:29:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:26 smithi096 ceph-mon[32753]: pgmap v18775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:29:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:27.705+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T21:29:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:27.705+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:29:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:27.706+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 272 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T21:29:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:27.706+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 272 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:29:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:28.297+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:28.297+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:28.297+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:29:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:28 smithi194 ceph-mon[33017]: pgmap v18776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34707.233027 secs 2024-04-26T21:29:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:28 smithi033 ceph-mon[27845]: pgmap v18776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34707.233027 secs 2024-04-26T21:29:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:28 smithi096 ceph-mon[32753]: pgmap v18776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:28.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34707.233027 secs 2024-04-26T21:29:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34710.187318 secs 2024-04-26T21:29:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34710.187318 secs 2024-04-26T21:29:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34710.187318 secs 2024-04-26T21:29:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:30 smithi194 ceph-mon[33017]: pgmap v18777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:30 smithi033 ceph-mon[27845]: pgmap v18777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:30 smithi096 ceph-mon[32753]: pgmap v18777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:32.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:32 smithi194 ceph-mon[33017]: pgmap v18778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:29:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:32 smithi033 ceph-mon[27845]: pgmap v18778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:29:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:32 smithi096 ceph-mon[32753]: pgmap v18778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:29:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34712.233115 secs 2024-04-26T21:29:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34712.233115 secs 2024-04-26T21:29:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34712.233115 secs 2024-04-26T21:29:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:34 smithi194 ceph-mon[33017]: pgmap v18779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34715.187389 secs 2024-04-26T21:29:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:34 smithi033 ceph-mon[27845]: pgmap v18779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34715.187389 secs 2024-04-26T21:29:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:34 smithi096 ceph-mon[32753]: pgmap v18779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34715.187389 secs 2024-04-26T21:29:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:36 smithi194 ceph-mon[33017]: pgmap v18780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:36 smithi033 ceph-mon[27845]: pgmap v18780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:36 smithi096 ceph-mon[32753]: pgmap v18780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:38 smithi194 ceph-mon[33017]: pgmap v18781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34717.233204 secs 2024-04-26T21:29:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:38 smithi033 ceph-mon[27845]: pgmap v18781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34717.233204 secs 2024-04-26T21:29:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:38 smithi096 ceph-mon[32753]: pgmap v18781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34717.233204 secs 2024-04-26T21:29:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34720.187506 secs 2024-04-26T21:29:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34720.187506 secs 2024-04-26T21:29:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34720.187506 secs 2024-04-26T21:29:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:40 smithi194 ceph-mon[33017]: pgmap v18782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:40 smithi033 ceph-mon[27845]: pgmap v18782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:40 smithi096 ceph-mon[32753]: pgmap v18782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:42 smithi194 ceph-mon[33017]: pgmap v18783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:42 smithi033 ceph-mon[27845]: pgmap v18783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:42 smithi096 ceph-mon[32753]: pgmap v18783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:43.313+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:43.313+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:29:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:43.313+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34722.233304 secs 2024-04-26T21:29:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34722.233304 secs 2024-04-26T21:29:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34722.233304 secs 2024-04-26T21:29:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:44 smithi194 ceph-mon[33017]: pgmap v18784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34725.187628 secs 2024-04-26T21:29:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:44 smithi033 ceph-mon[27845]: pgmap v18784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34725.187628 secs 2024-04-26T21:29:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:44 smithi096 ceph-mon[32753]: pgmap v18784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34725.187628 secs 2024-04-26T21:29:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:46 smithi194 ceph-mon[33017]: pgmap v18785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:46 smithi033 ceph-mon[27845]: pgmap v18785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:46 smithi096 ceph-mon[32753]: pgmap v18785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:48 smithi194 ceph-mon[33017]: pgmap v18786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34727.233407 secs 2024-04-26T21:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:48 smithi033 ceph-mon[27845]: pgmap v18786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34727.233407 secs 2024-04-26T21:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:48 smithi096 ceph-mon[32753]: pgmap v18786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34727.233407 secs 2024-04-26T21:29:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34730.187761 secs 2024-04-26T21:29:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34730.187761 secs 2024-04-26T21:29:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34730.187761 secs 2024-04-26T21:29:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:50 smithi033 ceph-mon[27845]: pgmap v18787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:50 smithi096 ceph-mon[32753]: pgmap v18787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:50.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:50 smithi194 ceph-mon[33017]: pgmap v18787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:52 smithi033 ceph-mon[27845]: pgmap v18788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:52 smithi096 ceph-mon[32753]: pgmap v18788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:52 smithi194 ceph-mon[33017]: pgmap v18788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34732.233485 secs 2024-04-26T21:29:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34732.233485 secs 2024-04-26T21:29:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34732.233485 secs 2024-04-26T21:29:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:54 smithi033 ceph-mon[27845]: pgmap v18789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34735.187875 secs 2024-04-26T21:29:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:54 smithi096 ceph-mon[32753]: pgmap v18789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34735.187875 secs 2024-04-26T21:29:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:54 smithi194 ceph-mon[33017]: pgmap v18789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:29:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34735.187875 secs 2024-04-26T21:29:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:56 smithi033 ceph-mon[27845]: pgmap v18790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:56 smithi096 ceph-mon[32753]: pgmap v18790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:56 smithi194 ceph-mon[33017]: pgmap v18790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:29:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:58.325+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:58.326+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:29:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:29:58.326+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:29:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:58 smithi033 ceph-mon[27845]: pgmap v18791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34737.233586 secs 2024-04-26T21:29:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:58 smithi096 ceph-mon[32753]: pgmap v18791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34737.233586 secs 2024-04-26T21:29:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:58 smithi194 ceph-mon[33017]: pgmap v18791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:29:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34737.233586 secs 2024-04-26T21:29:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:29:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34740.187975 secs 2024-04-26T21:29:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:29:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34740.187975 secs 2024-04-26T21:29:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:29:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34740.187975 secs 2024-04-26T21:30:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:00 smithi033 ceph-mon[27845]: pgmap v18792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:00 smithi096 ceph-mon[32753]: pgmap v18792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:00 smithi194 ceph-mon[33017]: pgmap v18792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:30:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:02 smithi033 ceph-mon[27845]: pgmap v18793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:02 smithi096 ceph-mon[32753]: pgmap v18793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:02.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:02 smithi194 ceph-mon[33017]: pgmap v18793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34742.233685 secs 2024-04-26T21:30:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34742.233685 secs 2024-04-26T21:30:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34742.233685 secs 2024-04-26T21:30:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:04 smithi033 ceph-mon[27845]: pgmap v18794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34745.188149 secs 2024-04-26T21:30:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:04 smithi096 ceph-mon[32753]: pgmap v18794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34745.188149 secs 2024-04-26T21:30:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:04 smithi194 ceph-mon[33017]: pgmap v18794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34745.188149 secs 2024-04-26T21:30:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:06 smithi033 ceph-mon[27845]: pgmap v18795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:06 smithi096 ceph-mon[32753]: pgmap v18795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:06.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:06 smithi194 ceph-mon[33017]: pgmap v18795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:08 smithi096 ceph-mon[32753]: pgmap v18796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:30:08.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34747.233786 secs 2024-04-26T21:30:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:08 smithi194 ceph-mon[33017]: pgmap v18796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:30:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34747.233786 secs 2024-04-26T21:30:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:08 smithi033 ceph-mon[27845]: pgmap v18796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:30:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34747.233786 secs 2024-04-26T21:30:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34750.188279 secs 2024-04-26T21:30:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34750.188279 secs 2024-04-26T21:30:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34750.188279 secs 2024-04-26T21:30:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:10 smithi194 ceph-mon[33017]: pgmap v18797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:10 smithi033 ceph-mon[27845]: pgmap v18797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:10 smithi096 ceph-mon[32753]: pgmap v18797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:12 smithi194 ceph-mon[33017]: pgmap v18798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:12 smithi033 ceph-mon[27845]: pgmap v18798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:12.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:12 smithi096 ceph-mon[32753]: pgmap v18798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:13.340+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:13.340+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:30:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:13.340+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34752.233882 secs 2024-04-26T21:30:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34752.233882 secs 2024-04-26T21:30:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34752.233882 secs 2024-04-26T21:30:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:14 smithi194 ceph-mon[33017]: pgmap v18799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34755.188378 secs 2024-04-26T21:30:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:14 smithi033 ceph-mon[27845]: pgmap v18799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34755.188378 secs 2024-04-26T21:30:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:14 smithi096 ceph-mon[32753]: pgmap v18799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34755.188378 secs 2024-04-26T21:30:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:15 smithi194 ceph-mon[33017]: pgmap v18800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:15 smithi033 ceph-mon[27845]: pgmap v18800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:15 smithi096 ceph-mon[32753]: pgmap v18800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:17 smithi033 ceph-mon[27845]: pgmap v18801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:30:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:17 smithi096 ceph-mon[32753]: pgmap v18801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:30:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:17 smithi194 ceph-mon[33017]: pgmap v18801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:30:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34757.233993 secs 2024-04-26T21:30:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34757.233993 secs 2024-04-26T21:30:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34757.233993 secs 2024-04-26T21:30:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:19 smithi033 ceph-mon[27845]: pgmap v18802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34760.188469 secs 2024-04-26T21:30:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:19 smithi096 ceph-mon[32753]: pgmap v18802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34760.188469 secs 2024-04-26T21:30:20.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:19 smithi194 ceph-mon[33017]: pgmap v18802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:20.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34760.188469 secs 2024-04-26T21:30:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:21 smithi033 ceph-mon[27845]: pgmap v18803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:21.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:21 smithi096 ceph-mon[32753]: pgmap v18803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:21.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:21 smithi194 ceph-mon[33017]: pgmap v18803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:22 smithi033 ceph-mon[27845]: pgmap v18804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:22 smithi033 ceph-mon[27845]: pgmap v18805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:30:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:22 smithi096 ceph-mon[32753]: pgmap v18804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:22 smithi096 ceph-mon[32753]: pgmap v18805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:22.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:30:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:22 smithi194 ceph-mon[33017]: pgmap v18804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:22 smithi194 ceph-mon[33017]: pgmap v18805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:30:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34762.234087 secs 2024-04-26T21:30:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34762.234087 secs 2024-04-26T21:30:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34762.234087 secs 2024-04-26T21:30:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34765.188548 secs 2024-04-26T21:30:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:24 smithi033 ceph-mon[27845]: pgmap v18806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:30:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34765.188548 secs 2024-04-26T21:30:24.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:24 smithi096 ceph-mon[32753]: pgmap v18806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:30:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34765.188548 secs 2024-04-26T21:30:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:24 smithi194 ceph-mon[33017]: pgmap v18806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:30:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:27 smithi194 ceph-mon[33017]: pgmap v18807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:27 smithi033 ceph-mon[27845]: pgmap v18807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:27 smithi096 ceph-mon[32753]: pgmap v18807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:28.356+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:28.357+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:28.358 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:28.357+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:30:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34767.234148 secs 2024-04-26T21:30:28.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34767.234148 secs 2024-04-26T21:30:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34767.234148 secs 2024-04-26T21:30:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:29 smithi194 ceph-mon[33017]: pgmap v18808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34770.188672 secs 2024-04-26T21:30:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:29 smithi033 ceph-mon[27845]: pgmap v18808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34770.188672 secs 2024-04-26T21:30:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:29 smithi096 ceph-mon[32753]: pgmap v18808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34770.188672 secs 2024-04-26T21:30:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:31 smithi194 ceph-mon[33017]: pgmap v18809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:31 smithi033 ceph-mon[27845]: pgmap v18809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:31 smithi096 ceph-mon[32753]: pgmap v18809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:30:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:33 smithi194 ceph-mon[33017]: pgmap v18810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:30:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34772.234200 secs 2024-04-26T21:30:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:33 smithi033 ceph-mon[27845]: pgmap v18810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:30:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34772.234200 secs 2024-04-26T21:30:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:33 smithi096 ceph-mon[32753]: pgmap v18810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:30:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34772.234200 secs 2024-04-26T21:30:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34775.188820 secs 2024-04-26T21:30:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34775.188820 secs 2024-04-26T21:30:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34775.188820 secs 2024-04-26T21:30:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:35 smithi194 ceph-mon[33017]: pgmap v18811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:35 smithi033 ceph-mon[27845]: pgmap v18811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:35 smithi096 ceph-mon[32753]: pgmap v18811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:37 smithi194 ceph-mon[33017]: pgmap v18812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:37 smithi033 ceph-mon[27845]: pgmap v18812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:37 smithi096 ceph-mon[32753]: pgmap v18812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34777.234263 secs 2024-04-26T21:30:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34777.234263 secs 2024-04-26T21:30:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34777.234263 secs 2024-04-26T21:30:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:39 smithi194 ceph-mon[33017]: pgmap v18813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34780.188941 secs 2024-04-26T21:30:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:39 smithi033 ceph-mon[27845]: pgmap v18813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34780.188941 secs 2024-04-26T21:30:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:39 smithi096 ceph-mon[32753]: pgmap v18813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34780.188941 secs 2024-04-26T21:30:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:41 smithi194 ceph-mon[33017]: pgmap v18814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:41 smithi033 ceph-mon[27845]: pgmap v18814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:41 smithi096 ceph-mon[32753]: pgmap v18814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:43.368+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:43.368+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:43.368+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:30:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:43 smithi194 ceph-mon[33017]: pgmap v18815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34782.234315 secs 2024-04-26T21:30:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:43 smithi033 ceph-mon[27845]: pgmap v18815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34782.234315 secs 2024-04-26T21:30:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:43 smithi096 ceph-mon[32753]: pgmap v18815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34782.234315 secs 2024-04-26T21:30:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34785.189071 secs 2024-04-26T21:30:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34785.189071 secs 2024-04-26T21:30:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34785.189071 secs 2024-04-26T21:30:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:45 smithi194 ceph-mon[33017]: pgmap v18816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:45 smithi033 ceph-mon[27845]: pgmap v18816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:45 smithi096 ceph-mon[32753]: pgmap v18816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:47 smithi194 ceph-mon[33017]: pgmap v18817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:47 smithi033 ceph-mon[27845]: pgmap v18817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:47 smithi096 ceph-mon[32753]: pgmap v18817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34787.234369 secs 2024-04-26T21:30:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34787.234369 secs 2024-04-26T21:30:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34787.234369 secs 2024-04-26T21:30:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:49 smithi194 ceph-mon[33017]: pgmap v18818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34790.189203 secs 2024-04-26T21:30:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:49 smithi033 ceph-mon[27845]: pgmap v18818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34790.189203 secs 2024-04-26T21:30:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:49 smithi096 ceph-mon[32753]: pgmap v18818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34790.189203 secs 2024-04-26T21:30:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:51 smithi033 ceph-mon[27845]: pgmap v18819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:51 smithi096 ceph-mon[32753]: pgmap v18819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:51 smithi194 ceph-mon[33017]: pgmap v18819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:53.614 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:53 smithi194 ceph-mon[33017]: pgmap v18820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:53.615 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34792.234423 secs 2024-04-26T21:30:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:53 smithi033 ceph-mon[27845]: pgmap v18820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34792.234423 secs 2024-04-26T21:30:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:53 smithi096 ceph-mon[32753]: pgmap v18820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34792.234423 secs 2024-04-26T21:30:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34795.189329 secs 2024-04-26T21:30:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34795.189329 secs 2024-04-26T21:30:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34795.189329 secs 2024-04-26T21:30:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:55 smithi033 ceph-mon[27845]: pgmap v18821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:55 smithi096 ceph-mon[32753]: pgmap v18821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:55 smithi194 ceph-mon[33017]: pgmap v18821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:57 smithi033 ceph-mon[27845]: pgmap v18822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:57 smithi096 ceph-mon[32753]: pgmap v18822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:57 smithi194 ceph-mon[33017]: pgmap v18822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:58.374+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:58.374+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:30:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:30:58.374+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:30:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34797.234486 secs 2024-04-26T21:30:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34797.234486 secs 2024-04-26T21:30:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34797.234486 secs 2024-04-26T21:30:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:59 smithi033 ceph-mon[27845]: pgmap v18823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:30:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34800.189447 secs 2024-04-26T21:30:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:59 smithi096 ceph-mon[32753]: pgmap v18823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:30:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34800.189447 secs 2024-04-26T21:30:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:59 smithi194 ceph-mon[33017]: pgmap v18823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:30:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:30:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34800.189447 secs 2024-04-26T21:31:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:01 smithi033 ceph-mon[27845]: pgmap v18824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:01 smithi096 ceph-mon[32753]: pgmap v18824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:01 smithi194 ceph-mon[33017]: pgmap v18824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:03 smithi033 ceph-mon[27845]: pgmap v18825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:03.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34802.234540 secs 2024-04-26T21:31:03.682 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:03 smithi194 ceph-mon[33017]: pgmap v18825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:03.682 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34802.234540 secs 2024-04-26T21:31:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:03 smithi096 ceph-mon[32753]: pgmap v18825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34802.234540 secs 2024-04-26T21:31:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34805.189532 secs 2024-04-26T21:31:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34805.189532 secs 2024-04-26T21:31:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34805.189532 secs 2024-04-26T21:31:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:05 smithi033 ceph-mon[27845]: pgmap v18826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:05 smithi096 ceph-mon[32753]: pgmap v18826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:05 smithi194 ceph-mon[33017]: pgmap v18826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:07 smithi033 ceph-mon[27845]: pgmap v18827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:07 smithi096 ceph-mon[32753]: pgmap v18827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:07 smithi194 ceph-mon[33017]: pgmap v18827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34807.234599 secs 2024-04-26T21:31:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34807.234599 secs 2024-04-26T21:31:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34807.234599 secs 2024-04-26T21:31:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:09 smithi194 ceph-mon[33017]: pgmap v18828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34810.189609 secs 2024-04-26T21:31:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:09 smithi033 ceph-mon[27845]: pgmap v18828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34810.189609 secs 2024-04-26T21:31:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:09 smithi096 ceph-mon[32753]: pgmap v18828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34810.189609 secs 2024-04-26T21:31:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:11 smithi194 ceph-mon[33017]: pgmap v18829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:11 smithi033 ceph-mon[27845]: pgmap v18829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:11 smithi096 ceph-mon[32753]: pgmap v18829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:13.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:13.380+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:13.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:13.380+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:13.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:13.380+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:31:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:13 smithi194 ceph-mon[33017]: pgmap v18830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34812.234652 secs 2024-04-26T21:31:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:13 smithi033 ceph-mon[27845]: pgmap v18830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34812.234652 secs 2024-04-26T21:31:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:13 smithi096 ceph-mon[32753]: pgmap v18830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34812.234652 secs 2024-04-26T21:31:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34815.189751 secs 2024-04-26T21:31:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34815.189751 secs 2024-04-26T21:31:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34815.189751 secs 2024-04-26T21:31:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:15 smithi194 ceph-mon[33017]: pgmap v18831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:15 smithi033 ceph-mon[27845]: pgmap v18831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:15 smithi096 ceph-mon[32753]: pgmap v18831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:17.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:17 smithi194 ceph-mon[33017]: pgmap v18832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:17 smithi033 ceph-mon[27845]: pgmap v18832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:17 smithi096 ceph-mon[32753]: pgmap v18832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34817.234714 secs 2024-04-26T21:31:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34817.234714 secs 2024-04-26T21:31:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34817.234714 secs 2024-04-26T21:31:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:19 smithi194 ceph-mon[33017]: pgmap v18833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34820.189891 secs 2024-04-26T21:31:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:19 smithi033 ceph-mon[27845]: pgmap v18833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34820.189891 secs 2024-04-26T21:31:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:19 smithi096 ceph-mon[32753]: pgmap v18833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34820.189891 secs 2024-04-26T21:31:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:20 smithi194 ceph-mon[33017]: pgmap v18834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:20 smithi033 ceph-mon[27845]: pgmap v18834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:20 smithi096 ceph-mon[32753]: pgmap v18834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:23.556 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:23 smithi194 ceph-mon[33017]: pgmap v18835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:23.556 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:23.556 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34822.234780 secs 2024-04-26T21:31:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:23 smithi033 ceph-mon[27845]: pgmap v18835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:23.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34822.234780 secs 2024-04-26T21:31:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:23 smithi096 ceph-mon[32753]: pgmap v18835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34822.234780 secs 2024-04-26T21:31:24.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34825.189998 secs 2024-04-26T21:31:24.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:24.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:24.579 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:31:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34825.189998 secs 2024-04-26T21:31:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:24.584 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:31:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34825.189998 secs 2024-04-26T21:31:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:31:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:25 smithi194 ceph-mon[33017]: pgmap v18836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:25 smithi194 ceph-mon[33017]: pgmap v18837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:31:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:25 smithi033 ceph-mon[27845]: pgmap v18836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:25 smithi033 ceph-mon[27845]: pgmap v18837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:31:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:25 smithi096 ceph-mon[32753]: pgmap v18836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:25 smithi096 ceph-mon[32753]: pgmap v18837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:31:27.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:27 smithi194 ceph-mon[33017]: pgmap v18838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:27 smithi033 ceph-mon[27845]: pgmap v18838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:27 smithi096 ceph-mon[32753]: pgmap v18838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:28.387+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:28.387+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:31:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:28.387+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34827.234839 secs 2024-04-26T21:31:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34827.234839 secs 2024-04-26T21:31:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34827.234839 secs 2024-04-26T21:31:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:29 smithi194 ceph-mon[33017]: pgmap v18839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34830.190140 secs 2024-04-26T21:31:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:29 smithi033 ceph-mon[27845]: pgmap v18839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34830.190140 secs 2024-04-26T21:31:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:29 smithi096 ceph-mon[32753]: pgmap v18839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34830.190140 secs 2024-04-26T21:31:31.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:31 smithi194 ceph-mon[33017]: pgmap v18840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:31.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:31 smithi033 ceph-mon[27845]: pgmap v18840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:31 smithi096 ceph-mon[32753]: pgmap v18840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:33 smithi194 ceph-mon[33017]: pgmap v18841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34832.234893 secs 2024-04-26T21:31:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:33 smithi033 ceph-mon[27845]: pgmap v18841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34832.234893 secs 2024-04-26T21:31:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:33 smithi096 ceph-mon[32753]: pgmap v18841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:31:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34832.234893 secs 2024-04-26T21:31:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34835.190264 secs 2024-04-26T21:31:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34835.190264 secs 2024-04-26T21:31:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34835.190264 secs 2024-04-26T21:31:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:35 smithi194 ceph-mon[33017]: pgmap v18842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:31:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:35 smithi033 ceph-mon[27845]: pgmap v18842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:31:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:35 smithi096 ceph-mon[32753]: pgmap v18842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:31:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:37 smithi194 ceph-mon[33017]: pgmap v18843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:37 smithi033 ceph-mon[27845]: pgmap v18843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:37 smithi096 ceph-mon[32753]: pgmap v18843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34837.234954 secs 2024-04-26T21:31:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34837.234954 secs 2024-04-26T21:31:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34837.234954 secs 2024-04-26T21:31:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:39 smithi033 ceph-mon[27845]: pgmap v18844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34840.190389 secs 2024-04-26T21:31:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:39 smithi096 ceph-mon[32753]: pgmap v18844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34840.190389 secs 2024-04-26T21:31:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:39 smithi194 ceph-mon[33017]: pgmap v18844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34840.190389 secs 2024-04-26T21:31:41.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:41 smithi033 ceph-mon[27845]: pgmap v18845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:41 smithi096 ceph-mon[32753]: pgmap v18845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:41 smithi194 ceph-mon[33017]: pgmap v18845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:43.403 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:43.403+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:43.403+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:31:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:43.403+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:43 smithi033 ceph-mon[27845]: pgmap v18846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34842.235011 secs 2024-04-26T21:31:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:43 smithi096 ceph-mon[32753]: pgmap v18846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34842.235011 secs 2024-04-26T21:31:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:43 smithi194 ceph-mon[33017]: pgmap v18846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34842.235011 secs 2024-04-26T21:31:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34845.190517 secs 2024-04-26T21:31:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34845.190517 secs 2024-04-26T21:31:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34845.190517 secs 2024-04-26T21:31:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:45 smithi033 ceph-mon[27845]: pgmap v18847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:45 smithi096 ceph-mon[32753]: pgmap v18847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:45 smithi194 ceph-mon[33017]: pgmap v18847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:31:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:47 smithi033 ceph-mon[27845]: pgmap v18848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:47 smithi096 ceph-mon[32753]: pgmap v18848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:47 smithi194 ceph-mon[33017]: pgmap v18848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:31:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34847.235071 secs 2024-04-26T21:31:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34847.235071 secs 2024-04-26T21:31:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34847.235071 secs 2024-04-26T21:31:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:49 smithi033 ceph-mon[27845]: pgmap v18849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34850.190594 secs 2024-04-26T21:31:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:49 smithi096 ceph-mon[32753]: pgmap v18849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34850.190594 secs 2024-04-26T21:31:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:49 smithi194 ceph-mon[33017]: pgmap v18849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34850.190594 secs 2024-04-26T21:31:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:51 smithi033 ceph-mon[27845]: pgmap v18850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:51 smithi096 ceph-mon[32753]: pgmap v18850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:51 smithi194 ceph-mon[33017]: pgmap v18850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:53 smithi033 ceph-mon[27845]: pgmap v18851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34852.235124 secs 2024-04-26T21:31:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:53 smithi096 ceph-mon[32753]: pgmap v18851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34852.235124 secs 2024-04-26T21:31:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:53 smithi194 ceph-mon[33017]: pgmap v18851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34852.235124 secs 2024-04-26T21:31:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34855.190737 secs 2024-04-26T21:31:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34855.190737 secs 2024-04-26T21:31:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34855.190737 secs 2024-04-26T21:31:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:55 smithi096 ceph-mon[32753]: pgmap v18852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:55 smithi194 ceph-mon[33017]: pgmap v18852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:55 smithi033 ceph-mon[27845]: pgmap v18852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:57 smithi194 ceph-mon[33017]: pgmap v18853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:57 smithi033 ceph-mon[27845]: pgmap v18853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:57 smithi096 ceph-mon[32753]: pgmap v18853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:58.406+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:58.406+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:31:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:31:58.406+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:31:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34857.235184 secs 2024-04-26T21:31:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34857.235184 secs 2024-04-26T21:31:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34857.235184 secs 2024-04-26T21:31:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:59 smithi194 ceph-mon[33017]: pgmap v18854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:31:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34860.190876 secs 2024-04-26T21:31:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:59 smithi033 ceph-mon[27845]: pgmap v18854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:31:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34860.190876 secs 2024-04-26T21:31:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:59 smithi096 ceph-mon[32753]: pgmap v18854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:31:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:31:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34860.190876 secs 2024-04-26T21:32:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:01 smithi194 ceph-mon[33017]: pgmap v18855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:01 smithi033 ceph-mon[27845]: pgmap v18855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:01 smithi096 ceph-mon[32753]: pgmap v18855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:03 smithi194 ceph-mon[33017]: pgmap v18856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34862.235236 secs 2024-04-26T21:32:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:03 smithi033 ceph-mon[27845]: pgmap v18856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34862.235236 secs 2024-04-26T21:32:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:03 smithi096 ceph-mon[32753]: pgmap v18856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34862.235236 secs 2024-04-26T21:32:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34865.191022 secs 2024-04-26T21:32:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34865.191022 secs 2024-04-26T21:32:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34865.191022 secs 2024-04-26T21:32:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:05 smithi194 ceph-mon[33017]: pgmap v18857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:05.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:05 smithi033 ceph-mon[27845]: pgmap v18857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:05 smithi096 ceph-mon[32753]: pgmap v18857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:07 smithi194 ceph-mon[33017]: pgmap v18858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:07 smithi033 ceph-mon[27845]: pgmap v18858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:07 smithi096 ceph-mon[32753]: pgmap v18858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34867.235297 secs 2024-04-26T21:32:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34867.235297 secs 2024-04-26T21:32:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34867.235297 secs 2024-04-26T21:32:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:09 smithi194 ceph-mon[33017]: pgmap v18859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34870.191171 secs 2024-04-26T21:32:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:09 smithi033 ceph-mon[27845]: pgmap v18859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34870.191171 secs 2024-04-26T21:32:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:09 smithi096 ceph-mon[32753]: pgmap v18859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34870.191171 secs 2024-04-26T21:32:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:10 smithi194 ceph-mon[33017]: pgmap v18860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:10.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:10 smithi033 ceph-mon[27845]: pgmap v18860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:10 smithi096 ceph-mon[32753]: pgmap v18860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:12 smithi194 ceph-mon[33017]: pgmap v18861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:13.418 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:13.416+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:13.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:13.416+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:13.419 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:13.416+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:32:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:12 smithi033 ceph-mon[27845]: pgmap v18861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:12 smithi096 ceph-mon[32753]: pgmap v18861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34872.235353 secs 2024-04-26T21:32:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34872.235353 secs 2024-04-26T21:32:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34872.235353 secs 2024-04-26T21:32:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34875.191236 secs 2024-04-26T21:32:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:14 smithi194 ceph-mon[33017]: pgmap v18862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34875.191236 secs 2024-04-26T21:32:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:14 smithi033 ceph-mon[27845]: pgmap v18862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34875.191236 secs 2024-04-26T21:32:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:14 smithi096 ceph-mon[32753]: pgmap v18862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:16 smithi194 ceph-mon[33017]: pgmap v18863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:16 smithi033 ceph-mon[27845]: pgmap v18863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:16 smithi096 ceph-mon[32753]: pgmap v18863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:18 smithi194 ceph-mon[33017]: pgmap v18864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34877.235415 secs 2024-04-26T21:32:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:18 smithi033 ceph-mon[27845]: pgmap v18864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34877.235415 secs 2024-04-26T21:32:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:18 smithi096 ceph-mon[32753]: pgmap v18864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34877.235415 secs 2024-04-26T21:32:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34880.191314 secs 2024-04-26T21:32:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34880.191314 secs 2024-04-26T21:32:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34880.191314 secs 2024-04-26T21:32:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:20 smithi194 ceph-mon[33017]: pgmap v18865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:20 smithi033 ceph-mon[27845]: pgmap v18865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:21.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:20 smithi096 ceph-mon[32753]: pgmap v18865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:22 smithi194 ceph-mon[33017]: pgmap v18866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:23.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:22 smithi033 ceph-mon[27845]: pgmap v18866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:22 smithi096 ceph-mon[32753]: pgmap v18866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:24.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34882.235474 secs 2024-04-26T21:32:24.334 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34882.235474 secs 2024-04-26T21:32:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34882.235474 secs 2024-04-26T21:32:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34885.191444 secs 2024-04-26T21:32:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:24 smithi194 ceph-mon[33017]: pgmap v18867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34885.191444 secs 2024-04-26T21:32:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:24 smithi033 ceph-mon[27845]: pgmap v18867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34885.191444 secs 2024-04-26T21:32:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:24 smithi096 ceph-mon[32753]: pgmap v18867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:27 smithi033 ceph-mon[27845]: pgmap v18868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:27.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:27 smithi096 ceph-mon[32753]: pgmap v18868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:27 smithi194 ceph-mon[33017]: pgmap v18868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:27.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:32:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:27.736+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0004740 con 0x7fef00077780 2024-04-26T21:32:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:27.736+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T21:32:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:27.737+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 273 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0004740 con 0x564385821900 2024-04-26T21:32:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:27.737+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 273 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:32:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:28 smithi033 ceph-mon[27845]: pgmap v18869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:32:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:28 smithi033 ceph-mon[27845]: pgmap v18870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:28 smithi096 ceph-mon[32753]: pgmap v18869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:32:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:28 smithi096 ceph-mon[32753]: pgmap v18870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:28.433+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:28.433+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:28.433+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:32:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:28 smithi194 ceph-mon[33017]: pgmap v18869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:32:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:28 smithi194 ceph-mon[33017]: pgmap v18870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34887.235531 secs 2024-04-26T21:32:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34887.235531 secs 2024-04-26T21:32:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34887.235531 secs 2024-04-26T21:32:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:30 smithi096 ceph-mon[32753]: pgmap v18871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:32:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34890.191523 secs 2024-04-26T21:32:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:30 smithi194 ceph-mon[33017]: pgmap v18871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:32:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34890.191523 secs 2024-04-26T21:32:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:30 smithi033 ceph-mon[27845]: pgmap v18871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:32:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34890.191523 secs 2024-04-26T21:32:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:32 smithi194 ceph-mon[33017]: pgmap v18872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:32.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:32 smithi033 ceph-mon[27845]: pgmap v18872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:32 smithi096 ceph-mon[32753]: pgmap v18872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:34 smithi033 ceph-mon[27845]: pgmap v18873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34892.235585 secs 2024-04-26T21:32:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:34 smithi194 ceph-mon[33017]: pgmap v18873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34892.235585 secs 2024-04-26T21:32:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:34 smithi096 ceph-mon[32753]: pgmap v18873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:34.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34892.235585 secs 2024-04-26T21:32:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34895.191647 secs 2024-04-26T21:32:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34895.191647 secs 2024-04-26T21:32:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34895.191647 secs 2024-04-26T21:32:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:36 smithi194 ceph-mon[33017]: pgmap v18874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:36 smithi033 ceph-mon[27845]: pgmap v18874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:36 smithi096 ceph-mon[32753]: pgmap v18874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:32:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:38 smithi194 ceph-mon[33017]: pgmap v18875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:32:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:38 smithi033 ceph-mon[27845]: pgmap v18875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:32:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:38 smithi096 ceph-mon[32753]: pgmap v18875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:32:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34897.235645 secs 2024-04-26T21:32:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34897.235645 secs 2024-04-26T21:32:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34897.235645 secs 2024-04-26T21:32:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:40 smithi194 ceph-mon[33017]: pgmap v18876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34900.191788 secs 2024-04-26T21:32:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:40 smithi033 ceph-mon[27845]: pgmap v18876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34900.191788 secs 2024-04-26T21:32:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:40 smithi096 ceph-mon[32753]: pgmap v18876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34900.191788 secs 2024-04-26T21:32:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:42 smithi194 ceph-mon[33017]: pgmap v18877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:42 smithi033 ceph-mon[27845]: pgmap v18877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:42 smithi096 ceph-mon[32753]: pgmap v18877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:43.445+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:43.445+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:32:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:43.445+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:43.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34902.235700 secs 2024-04-26T21:32:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34902.235700 secs 2024-04-26T21:32:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34902.235700 secs 2024-04-26T21:32:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:44 smithi194 ceph-mon[33017]: pgmap v18878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34905.191914 secs 2024-04-26T21:32:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:44 smithi033 ceph-mon[27845]: pgmap v18878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34905.191914 secs 2024-04-26T21:32:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:44 smithi096 ceph-mon[32753]: pgmap v18878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34905.191914 secs 2024-04-26T21:32:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:46 smithi194 ceph-mon[33017]: pgmap v18879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:46 smithi033 ceph-mon[27845]: pgmap v18879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:46 smithi096 ceph-mon[32753]: pgmap v18879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:48 smithi194 ceph-mon[33017]: pgmap v18880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34907.235760 secs 2024-04-26T21:32:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:48 smithi033 ceph-mon[27845]: pgmap v18880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34907.235760 secs 2024-04-26T21:32:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:48 smithi096 ceph-mon[32753]: pgmap v18880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34907.235760 secs 2024-04-26T21:32:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34910.192049 secs 2024-04-26T21:32:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34910.192049 secs 2024-04-26T21:32:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34910.192049 secs 2024-04-26T21:32:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:50 smithi194 ceph-mon[33017]: pgmap v18881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:50 smithi033 ceph-mon[27845]: pgmap v18881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:50 smithi096 ceph-mon[32753]: pgmap v18881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:52 smithi194 ceph-mon[33017]: pgmap v18882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:52 smithi033 ceph-mon[27845]: pgmap v18882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:52 smithi096 ceph-mon[32753]: pgmap v18882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34912.235812 secs 2024-04-26T21:32:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34912.235812 secs 2024-04-26T21:32:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34912.235812 secs 2024-04-26T21:32:54.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:54 smithi033 ceph-mon[27845]: pgmap v18883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:54.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34915.192194 secs 2024-04-26T21:32:54.587 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:54 smithi096 ceph-mon[32753]: pgmap v18883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:54.587 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34915.192194 secs 2024-04-26T21:32:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:54 smithi194 ceph-mon[33017]: pgmap v18883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34915.192194 secs 2024-04-26T21:32:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:56 smithi194 ceph-mon[33017]: pgmap v18884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:56 smithi033 ceph-mon[27845]: pgmap v18884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:56 smithi096 ceph-mon[32753]: pgmap v18884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:32:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:58.460+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:58.460+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:32:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:32:58.460+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:32:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:58 smithi194 ceph-mon[33017]: pgmap v18885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34917.235873 secs 2024-04-26T21:32:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:58 smithi033 ceph-mon[27845]: pgmap v18885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34917.235873 secs 2024-04-26T21:32:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:58 smithi096 ceph-mon[32753]: pgmap v18885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:32:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34917.235873 secs 2024-04-26T21:32:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:32:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34920.192259 secs 2024-04-26T21:32:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:32:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34920.192259 secs 2024-04-26T21:32:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:32:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34920.192259 secs 2024-04-26T21:33:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:00 smithi194 ceph-mon[33017]: pgmap v18886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:00 smithi033 ceph-mon[27845]: pgmap v18886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:00 smithi096 ceph-mon[32753]: pgmap v18886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:02 smithi194 ceph-mon[33017]: pgmap v18887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:02 smithi033 ceph-mon[27845]: pgmap v18887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:02 smithi096 ceph-mon[32753]: pgmap v18887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34922.235927 secs 2024-04-26T21:33:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34922.235927 secs 2024-04-26T21:33:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34922.235927 secs 2024-04-26T21:33:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:04 smithi194 ceph-mon[33017]: pgmap v18888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34925.192382 secs 2024-04-26T21:33:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:04 smithi033 ceph-mon[27845]: pgmap v18888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34925.192382 secs 2024-04-26T21:33:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:04 smithi096 ceph-mon[32753]: pgmap v18888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34925.192382 secs 2024-04-26T21:33:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:06 smithi194 ceph-mon[33017]: pgmap v18889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:06 smithi033 ceph-mon[27845]: pgmap v18889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:06 smithi096 ceph-mon[32753]: pgmap v18889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:08 smithi194 ceph-mon[33017]: pgmap v18890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34927.235987 secs 2024-04-26T21:33:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:08 smithi033 ceph-mon[27845]: pgmap v18890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34927.235987 secs 2024-04-26T21:33:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:08 smithi096 ceph-mon[32753]: pgmap v18890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34927.235987 secs 2024-04-26T21:33:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34930.192476 secs 2024-04-26T21:33:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34930.192476 secs 2024-04-26T21:33:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34930.192476 secs 2024-04-26T21:33:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:10 smithi194 ceph-mon[33017]: pgmap v18891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:10 smithi033 ceph-mon[27845]: pgmap v18891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:10 smithi096 ceph-mon[32753]: pgmap v18891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:12 smithi033 ceph-mon[27845]: pgmap v18892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:12 smithi096 ceph-mon[32753]: pgmap v18892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:12 smithi194 ceph-mon[33017]: pgmap v18892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:13.471+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:13.471+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:33:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:13.471+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34932.236044 secs 2024-04-26T21:33:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34932.236044 secs 2024-04-26T21:33:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34932.236044 secs 2024-04-26T21:33:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:14 smithi033 ceph-mon[27845]: pgmap v18893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34935.192597 secs 2024-04-26T21:33:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:14 smithi096 ceph-mon[32753]: pgmap v18893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34935.192597 secs 2024-04-26T21:33:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:14 smithi194 ceph-mon[33017]: pgmap v18893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34935.192597 secs 2024-04-26T21:33:16.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:16 smithi033 ceph-mon[27845]: pgmap v18894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:16 smithi096 ceph-mon[32753]: pgmap v18894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:16 smithi194 ceph-mon[33017]: pgmap v18894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:18 smithi033 ceph-mon[27845]: pgmap v18895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34937.236133 secs 2024-04-26T21:33:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:18 smithi096 ceph-mon[32753]: pgmap v18895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34937.236133 secs 2024-04-26T21:33:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:18 smithi194 ceph-mon[33017]: pgmap v18895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34937.236133 secs 2024-04-26T21:33:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34940.192731 secs 2024-04-26T21:33:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34940.192731 secs 2024-04-26T21:33:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34940.192731 secs 2024-04-26T21:33:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:20 smithi033 ceph-mon[27845]: pgmap v18896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:20 smithi096 ceph-mon[32753]: pgmap v18896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:20 smithi194 ceph-mon[33017]: pgmap v18896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:22 smithi033 ceph-mon[27845]: pgmap v18897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:22 smithi096 ceph-mon[32753]: pgmap v18897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:22.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:22 smithi194 ceph-mon[33017]: pgmap v18897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34942.236220 secs 2024-04-26T21:33:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34942.236220 secs 2024-04-26T21:33:23.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34942.236220 secs 2024-04-26T21:33:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:24 smithi033 ceph-mon[27845]: pgmap v18898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34945.192873 secs 2024-04-26T21:33:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:24 smithi096 ceph-mon[32753]: pgmap v18898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34945.192873 secs 2024-04-26T21:33:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:24 smithi194 ceph-mon[33017]: pgmap v18898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34945.192873 secs 2024-04-26T21:33:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:26 smithi033 ceph-mon[27845]: pgmap v18899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:26 smithi096 ceph-mon[32753]: pgmap v18899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:26.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:26 smithi194 ceph-mon[33017]: pgmap v18899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:28.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:28.488+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:28.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:28.488+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:28.489 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:28.488+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:33:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:28 smithi194 ceph-mon[33017]: pgmap v18900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34947.236303 secs 2024-04-26T21:33:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:33:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:33:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:33:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:28 smithi033 ceph-mon[27845]: pgmap v18900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34947.236303 secs 2024-04-26T21:33:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:33:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:33:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:33:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:28 smithi096 ceph-mon[32753]: pgmap v18900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34947.236303 secs 2024-04-26T21:33:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:33:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:33:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:33:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:29 smithi194 ceph-mon[33017]: pgmap v18901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:29 smithi194 ceph-mon[33017]: pgmap v18902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34950.192976 secs 2024-04-26T21:33:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:29 smithi033 ceph-mon[27845]: pgmap v18901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:29 smithi033 ceph-mon[27845]: pgmap v18902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34950.192976 secs 2024-04-26T21:33:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:29 smithi096 ceph-mon[32753]: pgmap v18901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:29 smithi096 ceph-mon[32753]: pgmap v18902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34950.192976 secs 2024-04-26T21:33:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:31 smithi194 ceph-mon[33017]: pgmap v18903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:31 smithi033 ceph-mon[27845]: pgmap v18903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:31 smithi096 ceph-mon[32753]: pgmap v18903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:33 smithi194 ceph-mon[33017]: pgmap v18904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34952.236386 secs 2024-04-26T21:33:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:33 smithi033 ceph-mon[27845]: pgmap v18904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34952.236386 secs 2024-04-26T21:33:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:33 smithi096 ceph-mon[32753]: pgmap v18904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34952.236386 secs 2024-04-26T21:33:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34955.193139 secs 2024-04-26T21:33:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34955.193139 secs 2024-04-26T21:33:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34955.193139 secs 2024-04-26T21:33:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:35 smithi194 ceph-mon[33017]: pgmap v18905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:33:35.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:35 smithi033 ceph-mon[27845]: pgmap v18905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:33:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:35 smithi096 ceph-mon[32753]: pgmap v18905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:33:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:37 smithi194 ceph-mon[33017]: pgmap v18906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:37 smithi033 ceph-mon[27845]: pgmap v18906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:37 smithi096 ceph-mon[32753]: pgmap v18906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34957.236471 secs 2024-04-26T21:33:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34957.236471 secs 2024-04-26T21:33:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34957.236471 secs 2024-04-26T21:33:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:39 smithi194 ceph-mon[33017]: pgmap v18907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:33:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34960.193180 secs 2024-04-26T21:33:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:39 smithi033 ceph-mon[27845]: pgmap v18907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:33:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34960.193180 secs 2024-04-26T21:33:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:39 smithi096 ceph-mon[32753]: pgmap v18907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:33:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34960.193180 secs 2024-04-26T21:33:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:41 smithi194 ceph-mon[33017]: pgmap v18908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:41 smithi033 ceph-mon[27845]: pgmap v18908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:41 smithi096 ceph-mon[32753]: pgmap v18908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:43.503+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:43.503+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:43.503+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:33:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:43 smithi194 ceph-mon[33017]: pgmap v18909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34962.236557 secs 2024-04-26T21:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:43 smithi033 ceph-mon[27845]: pgmap v18909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34962.236557 secs 2024-04-26T21:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:43 smithi096 ceph-mon[32753]: pgmap v18909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34962.236557 secs 2024-04-26T21:33:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34965.193279 secs 2024-04-26T21:33:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34965.193279 secs 2024-04-26T21:33:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34965.193279 secs 2024-04-26T21:33:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:45 smithi194 ceph-mon[33017]: pgmap v18910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:45 smithi033 ceph-mon[27845]: pgmap v18910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:45 smithi096 ceph-mon[32753]: pgmap v18910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:47 smithi194 ceph-mon[33017]: pgmap v18911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:47 smithi033 ceph-mon[27845]: pgmap v18911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:47 smithi096 ceph-mon[32753]: pgmap v18911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34967.236641 secs 2024-04-26T21:33:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:48 smithi194 ceph-mon[33017]: pgmap v18912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34967.236641 secs 2024-04-26T21:33:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:48 smithi033 ceph-mon[27845]: pgmap v18912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34967.236641 secs 2024-04-26T21:33:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:48 smithi096 ceph-mon[32753]: pgmap v18912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34970.193419 secs 2024-04-26T21:33:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34970.193419 secs 2024-04-26T21:33:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34970.193419 secs 2024-04-26T21:33:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:50 smithi194 ceph-mon[33017]: pgmap v18913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:50 smithi033 ceph-mon[27845]: pgmap v18913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:50.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:50 smithi096 ceph-mon[32753]: pgmap v18913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:53 smithi194 ceph-mon[33017]: pgmap v18914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34972.236695 secs 2024-04-26T21:33:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:53 smithi033 ceph-mon[27845]: pgmap v18914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:53.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34972.236695 secs 2024-04-26T21:33:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:53 smithi096 ceph-mon[32753]: pgmap v18914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34972.236695 secs 2024-04-26T21:33:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34975.193505 secs 2024-04-26T21:33:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34975.193505 secs 2024-04-26T21:33:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34975.193505 secs 2024-04-26T21:33:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:55 smithi033 ceph-mon[27845]: pgmap v18915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:55 smithi096 ceph-mon[32753]: pgmap v18915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:55 smithi194 ceph-mon[33017]: pgmap v18915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:57.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:57 smithi033 ceph-mon[27845]: pgmap v18916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:57 smithi096 ceph-mon[32753]: pgmap v18916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:57.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:57 smithi194 ceph-mon[33017]: pgmap v18916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:58.519+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:58.519+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:33:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:33:58.519+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:33:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34977.236755 secs 2024-04-26T21:33:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34977.236755 secs 2024-04-26T21:33:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34977.236755 secs 2024-04-26T21:33:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:59 smithi033 ceph-mon[27845]: pgmap v18917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:33:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34980.193585 secs 2024-04-26T21:33:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:59 smithi096 ceph-mon[32753]: pgmap v18917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:33:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34980.193585 secs 2024-04-26T21:33:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:59 smithi194 ceph-mon[33017]: pgmap v18917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:33:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:33:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34980.193585 secs 2024-04-26T21:34:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:01 smithi033 ceph-mon[27845]: pgmap v18918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:01 smithi096 ceph-mon[32753]: pgmap v18918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:01.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:01 smithi194 ceph-mon[33017]: pgmap v18918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:03 smithi033 ceph-mon[27845]: pgmap v18919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34982.236811 secs 2024-04-26T21:34:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:03 smithi096 ceph-mon[32753]: pgmap v18919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34982.236811 secs 2024-04-26T21:34:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:03 smithi194 ceph-mon[33017]: pgmap v18919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34982.236811 secs 2024-04-26T21:34:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34985.193708 secs 2024-04-26T21:34:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34985.193708 secs 2024-04-26T21:34:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34985.193708 secs 2024-04-26T21:34:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:05 smithi033 ceph-mon[27845]: pgmap v18920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:05 smithi096 ceph-mon[32753]: pgmap v18920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:05 smithi194 ceph-mon[33017]: pgmap v18920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:07 smithi033 ceph-mon[27845]: pgmap v18921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:07 smithi096 ceph-mon[32753]: pgmap v18921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:07 smithi194 ceph-mon[33017]: pgmap v18921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:08.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34987.236868 secs 2024-04-26T21:34:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34987.236868 secs 2024-04-26T21:34:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34987.236868 secs 2024-04-26T21:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:09 smithi033 ceph-mon[27845]: pgmap v18922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34990.193828 secs 2024-04-26T21:34:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:09 smithi096 ceph-mon[32753]: pgmap v18922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34990.193828 secs 2024-04-26T21:34:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:09 smithi194 ceph-mon[33017]: pgmap v18922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34990.193828 secs 2024-04-26T21:34:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:11 smithi033 ceph-mon[27845]: pgmap v18923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:11 smithi096 ceph-mon[32753]: pgmap v18923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:11 smithi194 ceph-mon[33017]: pgmap v18923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:13.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:13.533+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:13.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:13.533+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:13.535 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:13.533+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:34:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:13 smithi033 ceph-mon[27845]: pgmap v18924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34992.236922 secs 2024-04-26T21:34:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:13 smithi096 ceph-mon[32753]: pgmap v18924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34992.236922 secs 2024-04-26T21:34:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:13 smithi194 ceph-mon[33017]: pgmap v18924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:13.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34992.236922 secs 2024-04-26T21:34:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 34995.193950 secs 2024-04-26T21:34:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 34995.193950 secs 2024-04-26T21:34:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 34995.193950 secs 2024-04-26T21:34:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:15 smithi194 ceph-mon[33017]: pgmap v18925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:15.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:15 smithi033 ceph-mon[27845]: pgmap v18925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:15 smithi096 ceph-mon[32753]: pgmap v18925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:17 smithi194 ceph-mon[33017]: pgmap v18926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:17 smithi033 ceph-mon[27845]: pgmap v18926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:17 smithi096 ceph-mon[32753]: pgmap v18926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 34997.236983 secs 2024-04-26T21:34:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 34997.236983 secs 2024-04-26T21:34:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 34997.236983 secs 2024-04-26T21:34:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:19 smithi194 ceph-mon[33017]: pgmap v18927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35000.194089 secs 2024-04-26T21:34:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:19 smithi033 ceph-mon[27845]: pgmap v18927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35000.194089 secs 2024-04-26T21:34:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:19 smithi096 ceph-mon[32753]: pgmap v18927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35000.194089 secs 2024-04-26T21:34:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:21 smithi194 ceph-mon[33017]: pgmap v18928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:21.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:21 smithi033 ceph-mon[27845]: pgmap v18928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:21 smithi096 ceph-mon[32753]: pgmap v18928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:23 smithi194 ceph-mon[33017]: pgmap v18929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35002.237038 secs 2024-04-26T21:34:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:23 smithi033 ceph-mon[27845]: pgmap v18929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35002.237038 secs 2024-04-26T21:34:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:23 smithi096 ceph-mon[32753]: pgmap v18929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35002.237038 secs 2024-04-26T21:34:24.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35005.194212 secs 2024-04-26T21:34:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35005.194212 secs 2024-04-26T21:34:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35005.194212 secs 2024-04-26T21:34:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:25 smithi194 ceph-mon[33017]: pgmap v18930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:25 smithi033 ceph-mon[27845]: pgmap v18930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:25 smithi096 ceph-mon[32753]: pgmap v18930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:27.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:27 smithi194 ceph-mon[33017]: pgmap v18931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:27.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:27 smithi033 ceph-mon[27845]: pgmap v18931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:27 smithi096 ceph-mon[32753]: pgmap v18931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:28.541 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:28.540+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:28.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:28.540+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:28.542 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:28.540+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:34:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35007.237098 secs 2024-04-26T21:34:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35007.237098 secs 2024-04-26T21:34:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35007.237098 secs 2024-04-26T21:34:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:29 smithi194 ceph-mon[33017]: pgmap v18932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35010.194307 secs 2024-04-26T21:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:29 smithi033 ceph-mon[27845]: pgmap v18932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:34:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35010.194307 secs 2024-04-26T21:34:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:29 smithi096 ceph-mon[32753]: pgmap v18932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:34:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35010.194307 secs 2024-04-26T21:34:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:29.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:31 smithi194 ceph-mon[33017]: pgmap v18933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:31 smithi033 ceph-mon[27845]: pgmap v18933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:31 smithi096 ceph-mon[32753]: pgmap v18933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: pgmap v18934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35012.237151 secs 2024-04-26T21:34:33.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: pgmap v18934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35012.237151 secs 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: pgmap v18934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:33.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35012.237151 secs 2024-04-26T21:34:33.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:33.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:34 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:34:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:34 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:34:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35015.194389 secs 2024-04-26T21:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:34 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:34 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:34:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35015.194389 secs 2024-04-26T21:34:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:34 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:34 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:34:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35015.194389 secs 2024-04-26T21:34:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:35 smithi033 ceph-mon[27845]: pgmap v18935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:35 smithi096 ceph-mon[32753]: pgmap v18935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:35 smithi194 ceph-mon[33017]: pgmap v18935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:36.602 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.602 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:34:36.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:34:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: pgmap v18936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: pgmap v18937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:36.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:36.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: pgmap v18936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:34:36.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: pgmap v18937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:36.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: pgmap v18936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: pgmap v18937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:36.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:34:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:38 smithi033 ceph-mon[27845]: pgmap v18938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35017.237212 secs 2024-04-26T21:34:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:38 smithi096 ceph-mon[32753]: pgmap v18938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35017.237212 secs 2024-04-26T21:34:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:38 smithi194 ceph-mon[33017]: pgmap v18938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35017.237212 secs 2024-04-26T21:34:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35020.194465 secs 2024-04-26T21:34:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35020.194465 secs 2024-04-26T21:34:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35020.194465 secs 2024-04-26T21:34:41.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:40 smithi194 ceph-mon[33017]: pgmap v18939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:40 smithi033 ceph-mon[27845]: pgmap v18939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:40 smithi096 ceph-mon[32753]: pgmap v18939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:42 smithi194 ceph-mon[33017]: pgmap v18940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:43.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:42 smithi033 ceph-mon[27845]: pgmap v18940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:42 smithi096 ceph-mon[32753]: pgmap v18940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:34:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:43.552+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:43.552+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:43.552+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:34:44.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35022.237266 secs 2024-04-26T21:34:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35022.237266 secs 2024-04-26T21:34:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35022.237266 secs 2024-04-26T21:34:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:44 smithi194 ceph-mon[33017]: pgmap v18941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:34:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35025.194550 secs 2024-04-26T21:34:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:44 smithi033 ceph-mon[27845]: pgmap v18941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:34:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35025.194550 secs 2024-04-26T21:34:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:44 smithi096 ceph-mon[32753]: pgmap v18941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:34:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35025.194550 secs 2024-04-26T21:34:47.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:46 smithi194 ceph-mon[33017]: pgmap v18942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:34:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:46 smithi033 ceph-mon[27845]: pgmap v18942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:34:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:46 smithi096 ceph-mon[32753]: pgmap v18942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:34:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:48 smithi194 ceph-mon[33017]: pgmap v18943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35027.237327 secs 2024-04-26T21:34:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:48 smithi033 ceph-mon[27845]: pgmap v18943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35027.237327 secs 2024-04-26T21:34:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:48 smithi096 ceph-mon[32753]: pgmap v18943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35027.237327 secs 2024-04-26T21:34:50.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35030.194637 secs 2024-04-26T21:34:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35030.194637 secs 2024-04-26T21:34:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35030.194637 secs 2024-04-26T21:34:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:50 smithi194 ceph-mon[33017]: pgmap v18944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:50 smithi033 ceph-mon[27845]: pgmap v18944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:50 smithi096 ceph-mon[32753]: pgmap v18944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:52 smithi194 ceph-mon[33017]: pgmap v18945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:52 smithi033 ceph-mon[27845]: pgmap v18945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:52 smithi096 ceph-mon[32753]: pgmap v18945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:54.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35032.237381 secs 2024-04-26T21:34:54.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35032.237381 secs 2024-04-26T21:34:54.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35032.237381 secs 2024-04-26T21:34:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:54 smithi194 ceph-mon[33017]: pgmap v18946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35035.194765 secs 2024-04-26T21:34:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:54 smithi033 ceph-mon[27845]: pgmap v18946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35035.194765 secs 2024-04-26T21:34:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:54 smithi096 ceph-mon[32753]: pgmap v18946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35035.194765 secs 2024-04-26T21:34:57.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:56 smithi194 ceph-mon[33017]: pgmap v18947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:57.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:56 smithi033 ceph-mon[27845]: pgmap v18947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:56 smithi096 ceph-mon[32753]: pgmap v18947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:58.568+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:58.568+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:34:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:34:58.568+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:58 smithi194 ceph-mon[33017]: pgmap v18948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35037.237442 secs 2024-04-26T21:34:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:58 smithi033 ceph-mon[27845]: pgmap v18948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35037.237442 secs 2024-04-26T21:34:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:58 smithi096 ceph-mon[32753]: pgmap v18948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:34:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35037.237442 secs 2024-04-26T21:35:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:34:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35040.194842 secs 2024-04-26T21:35:00.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:34:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35040.194842 secs 2024-04-26T21:35:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:34:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35040.194842 secs 2024-04-26T21:35:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:00 smithi194 ceph-mon[33017]: pgmap v18949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:00 smithi033 ceph-mon[27845]: pgmap v18949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:00 smithi096 ceph-mon[32753]: pgmap v18949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:02 smithi194 ceph-mon[33017]: pgmap v18950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:02 smithi033 ceph-mon[27845]: pgmap v18950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:02 smithi096 ceph-mon[32753]: pgmap v18950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35042.237497 secs 2024-04-26T21:35:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35042.237497 secs 2024-04-26T21:35:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35042.237497 secs 2024-04-26T21:35:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:04 smithi194 ceph-mon[33017]: pgmap v18951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35045.194962 secs 2024-04-26T21:35:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:04 smithi033 ceph-mon[27845]: pgmap v18951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35045.194962 secs 2024-04-26T21:35:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:04 smithi096 ceph-mon[32753]: pgmap v18951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35045.194962 secs 2024-04-26T21:35:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:06 smithi194 ceph-mon[33017]: pgmap v18952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:07.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:06 smithi033 ceph-mon[27845]: pgmap v18952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:06 smithi096 ceph-mon[32753]: pgmap v18952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:08 smithi194 ceph-mon[33017]: pgmap v18953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35047.237557 secs 2024-04-26T21:35:09.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:08 smithi033 ceph-mon[27845]: pgmap v18953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:09.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35047.237557 secs 2024-04-26T21:35:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:08 smithi096 ceph-mon[32753]: pgmap v18953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35047.237557 secs 2024-04-26T21:35:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35050.195070 secs 2024-04-26T21:35:10.128 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35050.195070 secs 2024-04-26T21:35:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35050.195070 secs 2024-04-26T21:35:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:10 smithi194 ceph-mon[33017]: pgmap v18954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:10 smithi033 ceph-mon[27845]: pgmap v18954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:10 smithi096 ceph-mon[32753]: pgmap v18954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:12 smithi194 ceph-mon[33017]: pgmap v18955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:13.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:12 smithi033 ceph-mon[27845]: pgmap v18955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:12 smithi096 ceph-mon[32753]: pgmap v18955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:13.581+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:13.581+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:13.581+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:35:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35052.237612 secs 2024-04-26T21:35:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35052.237612 secs 2024-04-26T21:35:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35052.237612 secs 2024-04-26T21:35:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:14 smithi194 ceph-mon[33017]: pgmap v18956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35055.195212 secs 2024-04-26T21:35:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:14 smithi033 ceph-mon[27845]: pgmap v18956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35055.195212 secs 2024-04-26T21:35:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:14 smithi096 ceph-mon[32753]: pgmap v18956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35055.195212 secs 2024-04-26T21:35:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:16 smithi194 ceph-mon[33017]: pgmap v18957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:16 smithi033 ceph-mon[27845]: pgmap v18957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:16 smithi096 ceph-mon[32753]: pgmap v18957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:19.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:18 smithi194 ceph-mon[33017]: pgmap v18958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:19.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35057.237671 secs 2024-04-26T21:35:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:18 smithi033 ceph-mon[27845]: pgmap v18958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35057.237671 secs 2024-04-26T21:35:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:18 smithi096 ceph-mon[32753]: pgmap v18958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35057.237671 secs 2024-04-26T21:35:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35060.195338 secs 2024-04-26T21:35:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35060.195338 secs 2024-04-26T21:35:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35060.195338 secs 2024-04-26T21:35:21.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:20 smithi033 ceph-mon[27845]: pgmap v18959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:20 smithi096 ceph-mon[32753]: pgmap v18959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:20 smithi194 ceph-mon[33017]: pgmap v18959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:22 smithi033 ceph-mon[27845]: pgmap v18960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:22 smithi096 ceph-mon[32753]: pgmap v18960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:22 smithi194 ceph-mon[33017]: pgmap v18960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35062.237725 secs 2024-04-26T21:35:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35062.237725 secs 2024-04-26T21:35:24.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35062.237725 secs 2024-04-26T21:35:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:24 smithi033 ceph-mon[27845]: pgmap v18961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35065.195403 secs 2024-04-26T21:35:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:24 smithi096 ceph-mon[32753]: pgmap v18961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35065.195403 secs 2024-04-26T21:35:25.268 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:24 smithi194 ceph-mon[33017]: pgmap v18961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:25.268 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35065.195403 secs 2024-04-26T21:35:27.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:26 smithi033 ceph-mon[27845]: pgmap v18962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:26 smithi096 ceph-mon[32753]: pgmap v18962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:27.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:26 smithi194 ceph-mon[33017]: pgmap v18962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.766+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T21:35:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.766+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T21:35:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.767+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 274 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T21:35:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.767+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T21:35:27.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.767+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 274 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T21:35:27.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.767+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:35:27.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.767+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 275 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T21:35:27.769 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:27.768+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 275 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:35:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:28.583+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:28.583+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:28.583+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:35:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:28 smithi033 ceph-mon[27845]: pgmap v18963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35067.237767 secs 2024-04-26T21:35:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:28 smithi096 ceph-mon[32753]: pgmap v18963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:29.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35067.237767 secs 2024-04-26T21:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:28 smithi194 ceph-mon[33017]: pgmap v18963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35067.237767 secs 2024-04-26T21:35:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35070.195482 secs 2024-04-26T21:35:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35070.195482 secs 2024-04-26T21:35:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35070.195482 secs 2024-04-26T21:35:31.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:30 smithi033 ceph-mon[27845]: pgmap v18964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:30 smithi096 ceph-mon[32753]: pgmap v18964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:30 smithi194 ceph-mon[33017]: pgmap v18964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:33.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:32 smithi033 ceph-mon[27845]: pgmap v18965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:32 smithi096 ceph-mon[32753]: pgmap v18965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:32 smithi194 ceph-mon[33017]: pgmap v18965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35072.237824 secs 2024-04-26T21:35:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35072.237824 secs 2024-04-26T21:35:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35072.237824 secs 2024-04-26T21:35:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:34 smithi033 ceph-mon[27845]: pgmap v18966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35075.195579 secs 2024-04-26T21:35:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:34 smithi096 ceph-mon[32753]: pgmap v18966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35075.195579 secs 2024-04-26T21:35:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:34 smithi194 ceph-mon[33017]: pgmap v18966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35075.195579 secs 2024-04-26T21:35:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:36 smithi096 ceph-mon[32753]: pgmap v18967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:36 smithi194 ceph-mon[33017]: pgmap v18967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:36 smithi033 ceph-mon[27845]: pgmap v18967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:37 smithi194 ceph-mon[33017]: pgmap v18968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:37 smithi194 ceph-mon[33017]: pgmap v18969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:38.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:35:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:37 smithi033 ceph-mon[27845]: pgmap v18968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:37 smithi033 ceph-mon[27845]: pgmap v18969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:35:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:37 smithi096 ceph-mon[32753]: pgmap v18968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:37 smithi096 ceph-mon[32753]: pgmap v18969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:35:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35077.237886 secs 2024-04-26T21:35:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35077.237886 secs 2024-04-26T21:35:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35077.237886 secs 2024-04-26T21:35:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35080.195710 secs 2024-04-26T21:35:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:39 smithi194 ceph-mon[33017]: pgmap v18970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:35:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35080.195710 secs 2024-04-26T21:35:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:39 smithi033 ceph-mon[27845]: pgmap v18970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:35:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35080.195710 secs 2024-04-26T21:35:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:39 smithi096 ceph-mon[32753]: pgmap v18970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:35:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:42 smithi194 ceph-mon[33017]: pgmap v18971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:42 smithi033 ceph-mon[27845]: pgmap v18971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:42 smithi096 ceph-mon[32753]: pgmap v18971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:43.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:43.587+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:43.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:43.587+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:43.587 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:43.587+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:35:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35082.237986 secs 2024-04-26T21:35:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35082.237986 secs 2024-04-26T21:35:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35082.237986 secs 2024-04-26T21:35:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:44 smithi194 ceph-mon[33017]: pgmap v18972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35085.195832 secs 2024-04-26T21:35:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:44 smithi033 ceph-mon[27845]: pgmap v18972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35085.195832 secs 2024-04-26T21:35:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:44 smithi096 ceph-mon[32753]: pgmap v18972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35085.195832 secs 2024-04-26T21:35:46.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:46 smithi194 ceph-mon[33017]: pgmap v18973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:46 smithi033 ceph-mon[27845]: pgmap v18973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:46 smithi096 ceph-mon[32753]: pgmap v18973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:35:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:48 smithi194 ceph-mon[33017]: pgmap v18974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:35:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35087.238024 secs 2024-04-26T21:35:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:48 smithi033 ceph-mon[27845]: pgmap v18974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:35:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35087.238024 secs 2024-04-26T21:35:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:48 smithi096 ceph-mon[32753]: pgmap v18974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:35:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35087.238024 secs 2024-04-26T21:35:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35090.195956 secs 2024-04-26T21:35:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35090.195956 secs 2024-04-26T21:35:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35090.195956 secs 2024-04-26T21:35:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:50 smithi194 ceph-mon[33017]: pgmap v18975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:50 smithi033 ceph-mon[27845]: pgmap v18975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:50.686 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:50 smithi096 ceph-mon[32753]: pgmap v18975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:52 smithi194 ceph-mon[33017]: pgmap v18976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:52 smithi033 ceph-mon[27845]: pgmap v18976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:52 smithi096 ceph-mon[32753]: pgmap v18976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35092.238082 secs 2024-04-26T21:35:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35092.238082 secs 2024-04-26T21:35:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35092.238082 secs 2024-04-26T21:35:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:54 smithi033 ceph-mon[27845]: pgmap v18977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35095.196040 secs 2024-04-26T21:35:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:54 smithi096 ceph-mon[32753]: pgmap v18977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35095.196040 secs 2024-04-26T21:35:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:54 smithi194 ceph-mon[33017]: pgmap v18977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35095.196040 secs 2024-04-26T21:35:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:56 smithi033 ceph-mon[27845]: pgmap v18978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:56 smithi096 ceph-mon[32753]: pgmap v18978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:56 smithi194 ceph-mon[33017]: pgmap v18978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:58.594+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:58.594+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:35:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:35:58.595+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:35:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:58 smithi033 ceph-mon[27845]: pgmap v18979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35097.238143 secs 2024-04-26T21:35:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:58 smithi096 ceph-mon[32753]: pgmap v18979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35097.238143 secs 2024-04-26T21:35:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:58 smithi194 ceph-mon[33017]: pgmap v18979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:35:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35097.238143 secs 2024-04-26T21:35:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35100.196174 secs 2024-04-26T21:35:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35100.196174 secs 2024-04-26T21:35:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35100.196174 secs 2024-04-26T21:36:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:00 smithi033 ceph-mon[27845]: pgmap v18980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:00 smithi096 ceph-mon[32753]: pgmap v18980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:00 smithi194 ceph-mon[33017]: pgmap v18980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:02.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:02 smithi033 ceph-mon[27845]: pgmap v18981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:02 smithi096 ceph-mon[32753]: pgmap v18981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:02 smithi194 ceph-mon[33017]: pgmap v18981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35102.238229 secs 2024-04-26T21:36:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35102.238229 secs 2024-04-26T21:36:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35102.238229 secs 2024-04-26T21:36:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:04 smithi033 ceph-mon[27845]: pgmap v18982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35105.196309 secs 2024-04-26T21:36:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:04 smithi096 ceph-mon[32753]: pgmap v18982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35105.196309 secs 2024-04-26T21:36:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:04 smithi194 ceph-mon[33017]: pgmap v18982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:04.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35105.196309 secs 2024-04-26T21:36:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:06 smithi033 ceph-mon[27845]: pgmap v18983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:06 smithi096 ceph-mon[32753]: pgmap v18983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:06 smithi194 ceph-mon[33017]: pgmap v18983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:08 smithi033 ceph-mon[27845]: pgmap v18984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:08.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35107.238312 secs 2024-04-26T21:36:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:08 smithi096 ceph-mon[32753]: pgmap v18984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35107.238312 secs 2024-04-26T21:36:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:08 smithi194 ceph-mon[33017]: pgmap v18984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35107.238312 secs 2024-04-26T21:36:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35110.196370 secs 2024-04-26T21:36:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35110.196370 secs 2024-04-26T21:36:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35110.196370 secs 2024-04-26T21:36:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:10 smithi033 ceph-mon[27845]: pgmap v18985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:10 smithi096 ceph-mon[32753]: pgmap v18985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:10 smithi194 ceph-mon[33017]: pgmap v18985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:12 smithi096 ceph-mon[32753]: pgmap v18986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:12 smithi194 ceph-mon[33017]: pgmap v18986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:12 smithi033 ceph-mon[27845]: pgmap v18986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:13.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:13.596+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:13.596+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:13.596+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:36:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35112.238392 secs 2024-04-26T21:36:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35112.238392 secs 2024-04-26T21:36:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35112.238392 secs 2024-04-26T21:36:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:14 smithi194 ceph-mon[33017]: pgmap v18987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35115.196511 secs 2024-04-26T21:36:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:14 smithi033 ceph-mon[27845]: pgmap v18987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35115.196511 secs 2024-04-26T21:36:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:14 smithi096 ceph-mon[32753]: pgmap v18987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35115.196511 secs 2024-04-26T21:36:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:16 smithi194 ceph-mon[33017]: pgmap v18988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:16 smithi033 ceph-mon[27845]: pgmap v18988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:16 smithi096 ceph-mon[32753]: pgmap v18988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:18 smithi194 ceph-mon[33017]: pgmap v18989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35117.238479 secs 2024-04-26T21:36:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:18 smithi033 ceph-mon[27845]: pgmap v18989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35117.238479 secs 2024-04-26T21:36:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:18 smithi096 ceph-mon[32753]: pgmap v18989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35117.238479 secs 2024-04-26T21:36:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35120.196558 secs 2024-04-26T21:36:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35120.196558 secs 2024-04-26T21:36:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35120.196558 secs 2024-04-26T21:36:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:20 smithi194 ceph-mon[33017]: pgmap v18990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:36:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:20 smithi033 ceph-mon[27845]: pgmap v18990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:36:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:20 smithi096 ceph-mon[32753]: pgmap v18990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:36:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:22 smithi194 ceph-mon[33017]: pgmap v18991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:22 smithi033 ceph-mon[27845]: pgmap v18991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:22 smithi096 ceph-mon[32753]: pgmap v18991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:23.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35122.238561 secs 2024-04-26T21:36:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35122.238561 secs 2024-04-26T21:36:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35122.238561 secs 2024-04-26T21:36:24.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:24 smithi194 ceph-mon[33017]: pgmap v18992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:24.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35125.196704 secs 2024-04-26T21:36:24.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:24 smithi033 ceph-mon[27845]: pgmap v18992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:24.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35125.196704 secs 2024-04-26T21:36:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:24 smithi096 ceph-mon[32753]: pgmap v18992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:24.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35125.196704 secs 2024-04-26T21:36:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:26 smithi194 ceph-mon[33017]: pgmap v18993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:26 smithi033 ceph-mon[27845]: pgmap v18993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:26 smithi096 ceph-mon[32753]: pgmap v18993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:28.607+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:28.607+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:28.607+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:36:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:28 smithi194 ceph-mon[33017]: pgmap v18994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35127.238642 secs 2024-04-26T21:36:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:28 smithi033 ceph-mon[27845]: pgmap v18994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35127.238642 secs 2024-04-26T21:36:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:28 smithi096 ceph-mon[32753]: pgmap v18994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35127.238642 secs 2024-04-26T21:36:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35130.196824 secs 2024-04-26T21:36:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35130.196824 secs 2024-04-26T21:36:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35130.196824 secs 2024-04-26T21:36:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:30 smithi194 ceph-mon[33017]: pgmap v18995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:30 smithi033 ceph-mon[27845]: pgmap v18995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:30 smithi096 ceph-mon[32753]: pgmap v18995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:32.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:32 smithi194 ceph-mon[33017]: pgmap v18996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:32.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:32 smithi033 ceph-mon[27845]: pgmap v18996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:32 smithi096 ceph-mon[32753]: pgmap v18996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35132.238723 secs 2024-04-26T21:36:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35132.238723 secs 2024-04-26T21:36:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35132.238723 secs 2024-04-26T21:36:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:34 smithi194 ceph-mon[33017]: pgmap v18997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35135.196955 secs 2024-04-26T21:36:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:34 smithi033 ceph-mon[27845]: pgmap v18997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35135.196955 secs 2024-04-26T21:36:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:34 smithi096 ceph-mon[32753]: pgmap v18997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35135.196955 secs 2024-04-26T21:36:36.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:36 smithi194 ceph-mon[33017]: pgmap v18998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:36 smithi033 ceph-mon[27845]: pgmap v18998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:36 smithi096 ceph-mon[32753]: pgmap v18998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:38 smithi194 ceph-mon[33017]: pgmap v18999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35137.238817 secs 2024-04-26T21:36:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:38 smithi033 ceph-mon[27845]: pgmap v18999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35137.238817 secs 2024-04-26T21:36:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:38 smithi096 ceph-mon[32753]: pgmap v18999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35137.238817 secs 2024-04-26T21:36:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35140.197096 secs 2024-04-26T21:36:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:36:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35140.197096 secs 2024-04-26T21:36:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:36:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35140.197096 secs 2024-04-26T21:36:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:36:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:40 smithi194 ceph-mon[33017]: pgmap v19000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:40 smithi194 ceph-mon[33017]: pgmap v19001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:36:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:40 smithi033 ceph-mon[27845]: pgmap v19000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:40 smithi033 ceph-mon[27845]: pgmap v19001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:36:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:40 smithi096 ceph-mon[32753]: pgmap v19000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:40 smithi096 ceph-mon[32753]: pgmap v19001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:36:41.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:41 smithi194 ceph-mon[33017]: pgmap v19002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:41.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:41 smithi033 ceph-mon[27845]: pgmap v19002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:41 smithi096 ceph-mon[32753]: pgmap v19002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:43.623+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:43.623+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:43.623+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:36:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:44 smithi194 ceph-mon[33017]: pgmap v19003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35142.238898 secs 2024-04-26T21:36:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:44 smithi033 ceph-mon[27845]: pgmap v19003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35142.238898 secs 2024-04-26T21:36:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:44 smithi096 ceph-mon[32753]: pgmap v19003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35142.238898 secs 2024-04-26T21:36:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35145.197198 secs 2024-04-26T21:36:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35145.197198 secs 2024-04-26T21:36:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35145.197198 secs 2024-04-26T21:36:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:46 smithi194 ceph-mon[33017]: pgmap v19004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:46 smithi033 ceph-mon[27845]: pgmap v19004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:46 smithi096 ceph-mon[32753]: pgmap v19004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:48 smithi194 ceph-mon[33017]: pgmap v19005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:48 smithi033 ceph-mon[27845]: pgmap v19005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:48 smithi096 ceph-mon[32753]: pgmap v19005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:36:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35147.239004 secs 2024-04-26T21:36:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35147.239004 secs 2024-04-26T21:36:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35147.239004 secs 2024-04-26T21:36:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35150.197321 secs 2024-04-26T21:36:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:50 smithi194 ceph-mon[33017]: pgmap v19006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:36:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35150.197321 secs 2024-04-26T21:36:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:50 smithi033 ceph-mon[27845]: pgmap v19006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:36:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35150.197321 secs 2024-04-26T21:36:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:50 smithi096 ceph-mon[32753]: pgmap v19006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:36:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:52 smithi194 ceph-mon[33017]: pgmap v19007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:52 smithi033 ceph-mon[27845]: pgmap v19007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:52 smithi096 ceph-mon[32753]: pgmap v19007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:54 smithi194 ceph-mon[33017]: pgmap v19008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35152.239091 secs 2024-04-26T21:36:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:54 smithi033 ceph-mon[27845]: pgmap v19008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35152.239091 secs 2024-04-26T21:36:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:54 smithi096 ceph-mon[32753]: pgmap v19008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35152.239091 secs 2024-04-26T21:36:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35155.197384 secs 2024-04-26T21:36:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35155.197384 secs 2024-04-26T21:36:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35155.197384 secs 2024-04-26T21:36:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:56 smithi194 ceph-mon[33017]: pgmap v19009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:56 smithi033 ceph-mon[27845]: pgmap v19009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:56 smithi096 ceph-mon[32753]: pgmap v19009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:58 smithi194 ceph-mon[33017]: pgmap v19010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:58 smithi033 ceph-mon[27845]: pgmap v19010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:58 smithi096 ceph-mon[32753]: pgmap v19010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:36:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:58.637+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:58.637+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:36:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:36:58.637+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:36:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:36:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35157.239153 secs 2024-04-26T21:36:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:36:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35157.239153 secs 2024-04-26T21:36:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:36:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35157.239153 secs 2024-04-26T21:37:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35160.197467 secs 2024-04-26T21:37:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:00 smithi194 ceph-mon[33017]: pgmap v19011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35160.197467 secs 2024-04-26T21:37:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:00 smithi033 ceph-mon[27845]: pgmap v19011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35160.197467 secs 2024-04-26T21:37:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:00 smithi096 ceph-mon[32753]: pgmap v19011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:02 smithi194 ceph-mon[33017]: pgmap v19012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:02.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:02 smithi033 ceph-mon[27845]: pgmap v19012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:02 smithi096 ceph-mon[32753]: pgmap v19012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:04 smithi033 ceph-mon[27845]: pgmap v19013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35162.239234 secs 2024-04-26T21:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:04 smithi096 ceph-mon[32753]: pgmap v19013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35162.239234 secs 2024-04-26T21:37:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:04 smithi194 ceph-mon[33017]: pgmap v19013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35162.239234 secs 2024-04-26T21:37:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35165.197597 secs 2024-04-26T21:37:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35165.197597 secs 2024-04-26T21:37:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35165.197597 secs 2024-04-26T21:37:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:06 smithi194 ceph-mon[33017]: pgmap v19014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:06 smithi033 ceph-mon[27845]: pgmap v19014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:06 smithi096 ceph-mon[32753]: pgmap v19014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:08 smithi194 ceph-mon[33017]: pgmap v19015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35167.239304 secs 2024-04-26T21:37:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:08 smithi033 ceph-mon[27845]: pgmap v19015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35167.239304 secs 2024-04-26T21:37:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:08 smithi096 ceph-mon[32753]: pgmap v19015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35167.239304 secs 2024-04-26T21:37:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35170.197738 secs 2024-04-26T21:37:09.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35170.197738 secs 2024-04-26T21:37:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35170.197738 secs 2024-04-26T21:37:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:10 smithi194 ceph-mon[33017]: pgmap v19016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:10 smithi033 ceph-mon[27845]: pgmap v19016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:10 smithi096 ceph-mon[32753]: pgmap v19016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:12 smithi194 ceph-mon[33017]: pgmap v19017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:12.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:12 smithi033 ceph-mon[27845]: pgmap v19017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:12 smithi096 ceph-mon[32753]: pgmap v19017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:13.653 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:13.653+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:13.653+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:13.653+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:37:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35172.239384 secs 2024-04-26T21:37:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35172.239384 secs 2024-04-26T21:37:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35172.239384 secs 2024-04-26T21:37:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:14 smithi194 ceph-mon[33017]: pgmap v19018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35175.197860 secs 2024-04-26T21:37:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:14 smithi033 ceph-mon[27845]: pgmap v19018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35175.197860 secs 2024-04-26T21:37:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:14 smithi096 ceph-mon[32753]: pgmap v19018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35175.197860 secs 2024-04-26T21:37:16.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:16 smithi194 ceph-mon[33017]: pgmap v19019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:16 smithi033 ceph-mon[27845]: pgmap v19019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:16 smithi096 ceph-mon[32753]: pgmap v19019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:18 smithi194 ceph-mon[33017]: pgmap v19020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35177.239467 secs 2024-04-26T21:37:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:18 smithi033 ceph-mon[27845]: pgmap v19020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35177.239467 secs 2024-04-26T21:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:18 smithi096 ceph-mon[32753]: pgmap v19020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35177.239467 secs 2024-04-26T21:37:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35180.197995 secs 2024-04-26T21:37:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35180.197995 secs 2024-04-26T21:37:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35180.197995 secs 2024-04-26T21:37:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:20 smithi194 ceph-mon[33017]: pgmap v19021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:20 smithi033 ceph-mon[27845]: pgmap v19021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:20 smithi096 ceph-mon[32753]: pgmap v19021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:22 smithi194 ceph-mon[33017]: pgmap v19022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:22 smithi033 ceph-mon[27845]: pgmap v19022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:22 smithi096 ceph-mon[32753]: pgmap v19022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35182.239546 secs 2024-04-26T21:37:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35182.239546 secs 2024-04-26T21:37:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35182.239546 secs 2024-04-26T21:37:24.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:24 smithi033 ceph-mon[27845]: pgmap v19023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:24.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35185.198143 secs 2024-04-26T21:37:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:24 smithi096 ceph-mon[32753]: pgmap v19023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:24.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35185.198143 secs 2024-04-26T21:37:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:24 smithi194 ceph-mon[33017]: pgmap v19023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35185.198143 secs 2024-04-26T21:37:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:26 smithi194 ceph-mon[33017]: pgmap v19024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:26.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:26 smithi033 ceph-mon[27845]: pgmap v19024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:26 smithi096 ceph-mon[32753]: pgmap v19024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:28.666+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:28.666+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:28.666+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:37:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:28 smithi194 ceph-mon[33017]: pgmap v19025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35187.239631 secs 2024-04-26T21:37:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:28 smithi033 ceph-mon[27845]: pgmap v19025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35187.239631 secs 2024-04-26T21:37:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:28 smithi096 ceph-mon[32753]: pgmap v19025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35187.239631 secs 2024-04-26T21:37:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35190.198177 secs 2024-04-26T21:37:29.904 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35190.198177 secs 2024-04-26T21:37:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35190.198177 secs 2024-04-26T21:37:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:30 smithi194 ceph-mon[33017]: pgmap v19026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:30 smithi033 ceph-mon[27845]: pgmap v19026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:30 smithi096 ceph-mon[32753]: pgmap v19026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:31 smithi194 ceph-mon[33017]: pgmap v19027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:31 smithi033 ceph-mon[27845]: pgmap v19027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:31 smithi096 ceph-mon[32753]: pgmap v19027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:34 smithi194 ceph-mon[33017]: pgmap v19028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35192.239710 secs 2024-04-26T21:37:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:34 smithi033 ceph-mon[27845]: pgmap v19028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35192.239710 secs 2024-04-26T21:37:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:34 smithi096 ceph-mon[32753]: pgmap v19028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35192.239710 secs 2024-04-26T21:37:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35195.198257 secs 2024-04-26T21:37:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35195.198257 secs 2024-04-26T21:37:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35195.198257 secs 2024-04-26T21:37:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:36 smithi194 ceph-mon[33017]: pgmap v19029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:36 smithi033 ceph-mon[27845]: pgmap v19029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:36 smithi096 ceph-mon[32753]: pgmap v19029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:38 smithi194 ceph-mon[33017]: pgmap v19030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:38 smithi033 ceph-mon[27845]: pgmap v19030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:38 smithi096 ceph-mon[32753]: pgmap v19030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35197.239794 secs 2024-04-26T21:37:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35197.239794 secs 2024-04-26T21:37:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35197.239794 secs 2024-04-26T21:37:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35200.198333 secs 2024-04-26T21:37:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:40 smithi194 ceph-mon[33017]: pgmap v19031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35200.198333 secs 2024-04-26T21:37:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:40 smithi033 ceph-mon[27845]: pgmap v19031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35200.198333 secs 2024-04-26T21:37:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:40 smithi096 ceph-mon[32753]: pgmap v19031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:37:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:37:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:37:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:42 smithi194 ceph-mon[33017]: pgmap v19032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:37:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:42 smithi194 ceph-mon[33017]: pgmap v19033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:42 smithi033 ceph-mon[27845]: pgmap v19032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:37:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:42 smithi033 ceph-mon[27845]: pgmap v19033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:42 smithi096 ceph-mon[32753]: pgmap v19032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:37:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:42 smithi096 ceph-mon[32753]: pgmap v19033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:43.674+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:43.674+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:37:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:43.674+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:44 smithi194 ceph-mon[33017]: pgmap v19034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35202.239874 secs 2024-04-26T21:37:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:44 smithi033 ceph-mon[27845]: pgmap v19034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35202.239874 secs 2024-04-26T21:37:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:44 smithi096 ceph-mon[32753]: pgmap v19034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35202.239874 secs 2024-04-26T21:37:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35205.198457 secs 2024-04-26T21:37:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35205.198457 secs 2024-04-26T21:37:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35205.198457 secs 2024-04-26T21:37:46.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:46 smithi033 ceph-mon[27845]: pgmap v19035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:46 smithi096 ceph-mon[32753]: pgmap v19035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:46 smithi194 ceph-mon[33017]: pgmap v19035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:48 smithi033 ceph-mon[27845]: pgmap v19036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:48.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:48 smithi096 ceph-mon[32753]: pgmap v19036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:48.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:48 smithi194 ceph-mon[33017]: pgmap v19036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35207.239957 secs 2024-04-26T21:37:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35207.239957 secs 2024-04-26T21:37:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35207.239957 secs 2024-04-26T21:37:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:50 smithi033 ceph-mon[27845]: pgmap v19037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35210.198571 secs 2024-04-26T21:37:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:50 smithi096 ceph-mon[32753]: pgmap v19037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35210.198571 secs 2024-04-26T21:37:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:50 smithi194 ceph-mon[33017]: pgmap v19037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:37:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35210.198571 secs 2024-04-26T21:37:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:52 smithi033 ceph-mon[27845]: pgmap v19038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:37:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:52 smithi096 ceph-mon[32753]: pgmap v19038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:37:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:52 smithi194 ceph-mon[33017]: pgmap v19038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:37:54.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:54 smithi194 ceph-mon[33017]: pgmap v19039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:54.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35212.240041 secs 2024-04-26T21:37:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:54 smithi033 ceph-mon[27845]: pgmap v19039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35212.240041 secs 2024-04-26T21:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:54 smithi096 ceph-mon[32753]: pgmap v19039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35212.240041 secs 2024-04-26T21:37:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35215.198633 secs 2024-04-26T21:37:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35215.198633 secs 2024-04-26T21:37:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35215.198633 secs 2024-04-26T21:37:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:56 smithi033 ceph-mon[27845]: pgmap v19040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:56 smithi096 ceph-mon[32753]: pgmap v19040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:56 smithi194 ceph-mon[33017]: pgmap v19040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:58.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:58 smithi033 ceph-mon[27845]: pgmap v19041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:58 smithi096 ceph-mon[32753]: pgmap v19041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:58.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:58 smithi194 ceph-mon[33017]: pgmap v19041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:37:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:58.675+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:58.675+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:37:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:37:58.675+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:37:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:37:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35217.240130 secs 2024-04-26T21:37:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:37:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35217.240130 secs 2024-04-26T21:37:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:37:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35217.240130 secs 2024-04-26T21:38:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:00 smithi194 ceph-mon[33017]: pgmap v19042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35220.198756 secs 2024-04-26T21:38:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:00 smithi033 ceph-mon[27845]: pgmap v19042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35220.198756 secs 2024-04-26T21:38:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:00 smithi096 ceph-mon[32753]: pgmap v19042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35220.198756 secs 2024-04-26T21:38:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:02 smithi194 ceph-mon[33017]: pgmap v19043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:02 smithi033 ceph-mon[27845]: pgmap v19043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:02 smithi096 ceph-mon[32753]: pgmap v19043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:04 smithi194 ceph-mon[33017]: pgmap v19044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35222.240215 secs 2024-04-26T21:38:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:04 smithi033 ceph-mon[27845]: pgmap v19044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35222.240215 secs 2024-04-26T21:38:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:04 smithi096 ceph-mon[32753]: pgmap v19044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35222.240215 secs 2024-04-26T21:38:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35225.198872 secs 2024-04-26T21:38:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35225.198872 secs 2024-04-26T21:38:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35225.198872 secs 2024-04-26T21:38:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:06 smithi194 ceph-mon[33017]: pgmap v19045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:06.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:06 smithi033 ceph-mon[27845]: pgmap v19045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:06 smithi096 ceph-mon[32753]: pgmap v19045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:08.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:08 smithi194 ceph-mon[33017]: pgmap v19046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:08 smithi033 ceph-mon[27845]: pgmap v19046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:08 smithi096 ceph-mon[32753]: pgmap v19046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35227.240274 secs 2024-04-26T21:38:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35227.240274 secs 2024-04-26T21:38:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35227.240274 secs 2024-04-26T21:38:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:10 smithi194 ceph-mon[33017]: pgmap v19047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35230.199005 secs 2024-04-26T21:38:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:10 smithi033 ceph-mon[27845]: pgmap v19047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35230.199005 secs 2024-04-26T21:38:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:10 smithi096 ceph-mon[32753]: pgmap v19047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35230.199005 secs 2024-04-26T21:38:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:12 smithi194 ceph-mon[33017]: pgmap v19048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:12 smithi033 ceph-mon[27845]: pgmap v19048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:12 smithi096 ceph-mon[32753]: pgmap v19048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:13.691+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:13.691+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:13.691+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:38:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:14 smithi194 ceph-mon[33017]: pgmap v19049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35232.240330 secs 2024-04-26T21:38:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:14 smithi033 ceph-mon[27845]: pgmap v19049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35232.240330 secs 2024-04-26T21:38:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:14 smithi096 ceph-mon[32753]: pgmap v19049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35232.240330 secs 2024-04-26T21:38:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35235.199180 secs 2024-04-26T21:38:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35235.199180 secs 2024-04-26T21:38:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35235.199180 secs 2024-04-26T21:38:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:16 smithi194 ceph-mon[33017]: pgmap v19050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:16 smithi033 ceph-mon[27845]: pgmap v19050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:16 smithi096 ceph-mon[32753]: pgmap v19050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:18 smithi194 ceph-mon[33017]: pgmap v19051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:18 smithi033 ceph-mon[27845]: pgmap v19051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:18 smithi096 ceph-mon[32753]: pgmap v19051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35237.240390 secs 2024-04-26T21:38:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35237.240390 secs 2024-04-26T21:38:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35237.240390 secs 2024-04-26T21:38:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:20 smithi194 ceph-mon[33017]: pgmap v19052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35240.199338 secs 2024-04-26T21:38:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:20 smithi033 ceph-mon[27845]: pgmap v19052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35240.199338 secs 2024-04-26T21:38:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:20 smithi096 ceph-mon[32753]: pgmap v19052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35240.199338 secs 2024-04-26T21:38:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:22 smithi194 ceph-mon[33017]: pgmap v19053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:22 smithi033 ceph-mon[27845]: pgmap v19053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:22 smithi096 ceph-mon[32753]: pgmap v19053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35242.240470 secs 2024-04-26T21:38:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35242.240470 secs 2024-04-26T21:38:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35242.240470 secs 2024-04-26T21:38:24.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:24 smithi033 ceph-mon[27845]: pgmap v19054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:24.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35245.199466 secs 2024-04-26T21:38:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:24 smithi096 ceph-mon[32753]: pgmap v19054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35245.199466 secs 2024-04-26T21:38:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:24 smithi194 ceph-mon[33017]: pgmap v19054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35245.199466 secs 2024-04-26T21:38:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:26 smithi194 ceph-mon[33017]: pgmap v19055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:26 smithi033 ceph-mon[27845]: pgmap v19055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:26 smithi096 ceph-mon[32753]: pgmap v19055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:27.797+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:38:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:27.797+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:38:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:27.797+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 276 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T21:38:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:27.797+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 276 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:38:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:28 smithi194 ceph-mon[33017]: pgmap v19056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:28.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35247.240552 secs 2024-04-26T21:38:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:28 smithi033 ceph-mon[27845]: pgmap v19056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:28.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35247.240552 secs 2024-04-26T21:38:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:28 smithi096 ceph-mon[32753]: pgmap v19056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35247.240552 secs 2024-04-26T21:38:28.701 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:28.700+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:28.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:28.700+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:28.702 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:28.700+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:38:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35250.199565 secs 2024-04-26T21:38:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35250.199565 secs 2024-04-26T21:38:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35250.199565 secs 2024-04-26T21:38:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:30 smithi194 ceph-mon[33017]: pgmap v19057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:30 smithi033 ceph-mon[27845]: pgmap v19057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:30 smithi096 ceph-mon[32753]: pgmap v19057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:32 smithi194 ceph-mon[33017]: pgmap v19058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:32 smithi033 ceph-mon[27845]: pgmap v19058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:32 smithi096 ceph-mon[32753]: pgmap v19058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35252.240635 secs 2024-04-26T21:38:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35252.240635 secs 2024-04-26T21:38:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35252.240635 secs 2024-04-26T21:38:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:34 smithi194 ceph-mon[33017]: pgmap v19059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35255.199688 secs 2024-04-26T21:38:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:34 smithi033 ceph-mon[27845]: pgmap v19059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35255.199688 secs 2024-04-26T21:38:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:34 smithi096 ceph-mon[32753]: pgmap v19059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35255.199688 secs 2024-04-26T21:38:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:36 smithi194 ceph-mon[33017]: pgmap v19060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:36.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:36 smithi033 ceph-mon[27845]: pgmap v19060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:36 smithi096 ceph-mon[32753]: pgmap v19060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:38 smithi194 ceph-mon[33017]: pgmap v19061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35257.240720 secs 2024-04-26T21:38:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:38 smithi033 ceph-mon[27845]: pgmap v19061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:38.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35257.240720 secs 2024-04-26T21:38:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:38 smithi096 ceph-mon[32753]: pgmap v19061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35257.240720 secs 2024-04-26T21:38:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35260.199833 secs 2024-04-26T21:38:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35260.199833 secs 2024-04-26T21:38:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35260.199833 secs 2024-04-26T21:38:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:40 smithi194 ceph-mon[33017]: pgmap v19062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:40.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:40 smithi033 ceph-mon[27845]: pgmap v19062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:40 smithi096 ceph-mon[32753]: pgmap v19062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:42 smithi033 ceph-mon[27845]: pgmap v19063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:42 smithi096 ceph-mon[32753]: pgmap v19063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:42 smithi194 ceph-mon[33017]: pgmap v19063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:43 smithi033 ceph-mon[27845]: pgmap v19064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:43 smithi033 ceph-mon[27845]: pgmap v19065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:38:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35262.240803 secs 2024-04-26T21:38:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:43 smithi096 ceph-mon[32753]: pgmap v19064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:43 smithi096 ceph-mon[32753]: pgmap v19065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:38:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35262.240803 secs 2024-04-26T21:38:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:43.712+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:43.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:43.712+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:43.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:43.712+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:38:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:43 smithi194 ceph-mon[33017]: pgmap v19064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:43 smithi194 ceph-mon[33017]: pgmap v19065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:38:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35262.240803 secs 2024-04-26T21:38:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35265.199980 secs 2024-04-26T21:38:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35265.199980 secs 2024-04-26T21:38:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35265.199980 secs 2024-04-26T21:38:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:45 smithi033 ceph-mon[27845]: pgmap v19066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:38:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:45 smithi096 ceph-mon[32753]: pgmap v19066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:38:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:45 smithi194 ceph-mon[33017]: pgmap v19066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:38:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:47 smithi194 ceph-mon[33017]: pgmap v19067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:47 smithi033 ceph-mon[27845]: pgmap v19067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:47 smithi096 ceph-mon[32753]: pgmap v19067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35267.240881 secs 2024-04-26T21:38:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35267.240881 secs 2024-04-26T21:38:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35267.240881 secs 2024-04-26T21:38:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:49 smithi194 ceph-mon[33017]: pgmap v19068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35270.200135 secs 2024-04-26T21:38:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:49 smithi033 ceph-mon[27845]: pgmap v19068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35270.200135 secs 2024-04-26T21:38:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:49 smithi096 ceph-mon[32753]: pgmap v19068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35270.200135 secs 2024-04-26T21:38:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:51 smithi194 ceph-mon[33017]: pgmap v19069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:51 smithi033 ceph-mon[27845]: pgmap v19069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:51 smithi096 ceph-mon[32753]: pgmap v19069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:38:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:53 smithi194 ceph-mon[33017]: pgmap v19070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:38:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35272.240987 secs 2024-04-26T21:38:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:53 smithi033 ceph-mon[27845]: pgmap v19070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:38:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35272.240987 secs 2024-04-26T21:38:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:53 smithi096 ceph-mon[32753]: pgmap v19070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:38:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35272.240987 secs 2024-04-26T21:38:54.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35275.200263 secs 2024-04-26T21:38:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35275.200263 secs 2024-04-26T21:38:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35275.200263 secs 2024-04-26T21:38:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:55 smithi194 ceph-mon[33017]: pgmap v19071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:55.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:55 smithi033 ceph-mon[27845]: pgmap v19071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:55 smithi096 ceph-mon[32753]: pgmap v19071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:57 smithi194 ceph-mon[33017]: pgmap v19072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:57.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:57 smithi033 ceph-mon[27845]: pgmap v19072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:57 smithi096 ceph-mon[32753]: pgmap v19072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:58.729+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:58.729+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:38:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:38:58.729+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:38:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35277.241006 secs 2024-04-26T21:38:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35277.241006 secs 2024-04-26T21:38:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35277.241006 secs 2024-04-26T21:38:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:59 smithi194 ceph-mon[33017]: pgmap v19073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:38:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35280.200390 secs 2024-04-26T21:38:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:59 smithi033 ceph-mon[27845]: pgmap v19073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:38:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35280.200390 secs 2024-04-26T21:38:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:59 smithi096 ceph-mon[32753]: pgmap v19073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:38:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:38:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35280.200390 secs 2024-04-26T21:39:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:01 smithi194 ceph-mon[33017]: pgmap v19074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:01.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:01 smithi033 ceph-mon[27845]: pgmap v19074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:01 smithi096 ceph-mon[32753]: pgmap v19074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:03 smithi194 ceph-mon[33017]: pgmap v19075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35282.241087 secs 2024-04-26T21:39:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:03 smithi033 ceph-mon[27845]: pgmap v19075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35282.241087 secs 2024-04-26T21:39:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:03 smithi096 ceph-mon[32753]: pgmap v19075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35282.241087 secs 2024-04-26T21:39:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35285.200514 secs 2024-04-26T21:39:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35285.200514 secs 2024-04-26T21:39:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35285.200514 secs 2024-04-26T21:39:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:05 smithi194 ceph-mon[33017]: pgmap v19076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:05 smithi033 ceph-mon[27845]: pgmap v19076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:05 smithi096 ceph-mon[32753]: pgmap v19076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:06 smithi194 ceph-mon[33017]: pgmap v19077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:06 smithi033 ceph-mon[27845]: pgmap v19077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:06 smithi096 ceph-mon[32753]: pgmap v19077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35287.241174 secs 2024-04-26T21:39:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35287.241174 secs 2024-04-26T21:39:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35287.241174 secs 2024-04-26T21:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:09 smithi194 ceph-mon[33017]: pgmap v19078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35290.200634 secs 2024-04-26T21:39:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:09 smithi033 ceph-mon[27845]: pgmap v19078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35290.200634 secs 2024-04-26T21:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:09 smithi096 ceph-mon[32753]: pgmap v19078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35290.200634 secs 2024-04-26T21:39:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:11 smithi194 ceph-mon[33017]: pgmap v19079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:11 smithi033 ceph-mon[27845]: pgmap v19079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:11 smithi096 ceph-mon[32753]: pgmap v19079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:13.744+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:13.744+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:13.744+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:39:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:13 smithi194 ceph-mon[33017]: pgmap v19080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35292.241259 secs 2024-04-26T21:39:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:13 smithi033 ceph-mon[27845]: pgmap v19080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35292.241259 secs 2024-04-26T21:39:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:13 smithi096 ceph-mon[32753]: pgmap v19080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35292.241259 secs 2024-04-26T21:39:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35295.200754 secs 2024-04-26T21:39:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35295.200754 secs 2024-04-26T21:39:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35295.200754 secs 2024-04-26T21:39:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:15 smithi194 ceph-mon[33017]: pgmap v19081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:15 smithi033 ceph-mon[27845]: pgmap v19081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:15 smithi096 ceph-mon[32753]: pgmap v19081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:17 smithi194 ceph-mon[33017]: pgmap v19082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:17 smithi033 ceph-mon[27845]: pgmap v19082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:17 smithi096 ceph-mon[32753]: pgmap v19082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35297.241344 secs 2024-04-26T21:39:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35297.241344 secs 2024-04-26T21:39:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35297.241344 secs 2024-04-26T21:39:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:19 smithi194 ceph-mon[33017]: pgmap v19083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35300.200905 secs 2024-04-26T21:39:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:19 smithi033 ceph-mon[27845]: pgmap v19083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35300.200905 secs 2024-04-26T21:39:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:19 smithi096 ceph-mon[32753]: pgmap v19083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35300.200905 secs 2024-04-26T21:39:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:21 smithi194 ceph-mon[33017]: pgmap v19084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:21 smithi033 ceph-mon[27845]: pgmap v19084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:21 smithi096 ceph-mon[32753]: pgmap v19084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:23 smithi194 ceph-mon[33017]: pgmap v19085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35302.241425 secs 2024-04-26T21:39:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:23 smithi033 ceph-mon[27845]: pgmap v19085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35302.241425 secs 2024-04-26T21:39:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:23 smithi096 ceph-mon[32753]: pgmap v19085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35302.241425 secs 2024-04-26T21:39:24.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35305.201059 secs 2024-04-26T21:39:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35305.201059 secs 2024-04-26T21:39:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35305.201059 secs 2024-04-26T21:39:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:25 smithi194 ceph-mon[33017]: pgmap v19086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:25 smithi033 ceph-mon[27845]: pgmap v19086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:25 smithi096 ceph-mon[32753]: pgmap v19086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:27 smithi194 ceph-mon[33017]: pgmap v19087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:27 smithi033 ceph-mon[27845]: pgmap v19087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:27 smithi096 ceph-mon[32753]: pgmap v19087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:28.753+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:28.753+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:39:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:28.753+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35307.241511 secs 2024-04-26T21:39:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35307.241511 secs 2024-04-26T21:39:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35307.241511 secs 2024-04-26T21:39:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:29 smithi194 ceph-mon[33017]: pgmap v19088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35310.201178 secs 2024-04-26T21:39:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:29 smithi033 ceph-mon[27845]: pgmap v19088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35310.201178 secs 2024-04-26T21:39:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:29 smithi096 ceph-mon[32753]: pgmap v19088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35310.201178 secs 2024-04-26T21:39:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:30 smithi194 ceph-mon[33017]: pgmap v19089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:30 smithi033 ceph-mon[27845]: pgmap v19089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:30 smithi096 ceph-mon[32753]: pgmap v19089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:33 smithi194 ceph-mon[33017]: pgmap v19090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35312.241595 secs 2024-04-26T21:39:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:33 smithi033 ceph-mon[27845]: pgmap v19090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35312.241595 secs 2024-04-26T21:39:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:33 smithi096 ceph-mon[32753]: pgmap v19090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35312.241595 secs 2024-04-26T21:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35315.201312 secs 2024-04-26T21:39:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35315.201312 secs 2024-04-26T21:39:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35315.201312 secs 2024-04-26T21:39:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:35 smithi194 ceph-mon[33017]: pgmap v19091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:35 smithi033 ceph-mon[27845]: pgmap v19091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:35 smithi096 ceph-mon[32753]: pgmap v19091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:37 smithi194 ceph-mon[33017]: pgmap v19092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:37 smithi033 ceph-mon[27845]: pgmap v19092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:37 smithi096 ceph-mon[32753]: pgmap v19092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35317.241654 secs 2024-04-26T21:39:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35317.241654 secs 2024-04-26T21:39:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35317.241654 secs 2024-04-26T21:39:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:39 smithi194 ceph-mon[33017]: pgmap v19093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35320.201427 secs 2024-04-26T21:39:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:39 smithi033 ceph-mon[27845]: pgmap v19093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35320.201427 secs 2024-04-26T21:39:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:39 smithi096 ceph-mon[32753]: pgmap v19093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35320.201427 secs 2024-04-26T21:39:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:41 smithi194 ceph-mon[33017]: pgmap v19094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:41 smithi033 ceph-mon[27845]: pgmap v19094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:41 smithi096 ceph-mon[32753]: pgmap v19094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:43.759+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:43.760+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:43.760+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:39:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:43 smithi194 ceph-mon[33017]: pgmap v19095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35322.241741 secs 2024-04-26T21:39:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:43 smithi033 ceph-mon[27845]: pgmap v19095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35322.241741 secs 2024-04-26T21:39:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:43 smithi096 ceph-mon[32753]: pgmap v19095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35322.241741 secs 2024-04-26T21:39:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35325.201532 secs 2024-04-26T21:39:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:44.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35325.201532 secs 2024-04-26T21:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35325.201532 secs 2024-04-26T21:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:39:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:45 smithi194 ceph-mon[33017]: pgmap v19096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:45 smithi194 ceph-mon[33017]: pgmap v19097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:39:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:45 smithi033 ceph-mon[27845]: pgmap v19096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:45 smithi033 ceph-mon[27845]: pgmap v19097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:39:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:45 smithi096 ceph-mon[32753]: pgmap v19096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:45 smithi096 ceph-mon[32753]: pgmap v19097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:39:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:47 smithi194 ceph-mon[33017]: pgmap v19098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:47 smithi033 ceph-mon[27845]: pgmap v19098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:47 smithi096 ceph-mon[32753]: pgmap v19098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35327.241827 secs 2024-04-26T21:39:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35327.241827 secs 2024-04-26T21:39:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35327.241827 secs 2024-04-26T21:39:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:49 smithi194 ceph-mon[33017]: pgmap v19099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35330.201658 secs 2024-04-26T21:39:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:49 smithi033 ceph-mon[27845]: pgmap v19099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35330.201658 secs 2024-04-26T21:39:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:49 smithi096 ceph-mon[32753]: pgmap v19099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35330.201658 secs 2024-04-26T21:39:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:50 smithi194 ceph-mon[33017]: pgmap v19100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:50 smithi033 ceph-mon[27845]: pgmap v19100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:50 smithi096 ceph-mon[32753]: pgmap v19100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:53 smithi033 ceph-mon[27845]: pgmap v19101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:53 smithi096 ceph-mon[32753]: pgmap v19101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:53 smithi194 ceph-mon[33017]: pgmap v19101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:39:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35332.241909 secs 2024-04-26T21:39:54.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35332.241909 secs 2024-04-26T21:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35332.241909 secs 2024-04-26T21:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35335.201765 secs 2024-04-26T21:39:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:55 smithi096 ceph-mon[32753]: pgmap v19102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:39:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35335.201765 secs 2024-04-26T21:39:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:55 smithi194 ceph-mon[33017]: pgmap v19102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:39:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35335.201765 secs 2024-04-26T21:39:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:55 smithi033 ceph-mon[27845]: pgmap v19102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:39:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:57 smithi194 ceph-mon[33017]: pgmap v19103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:57 smithi033 ceph-mon[27845]: pgmap v19103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:57 smithi096 ceph-mon[32753]: pgmap v19103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:58.776+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:58.776+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:39:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:39:58.776+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:39:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35337.242013 secs 2024-04-26T21:39:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:39:59 smithi194 ceph-mon[33017]: pgmap v19104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35337.242013 secs 2024-04-26T21:39:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:39:59 smithi033 ceph-mon[27845]: pgmap v19104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:39:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35337.242013 secs 2024-04-26T21:39:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:39:59 smithi096 ceph-mon[32753]: pgmap v19104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:00.447 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35340.201879 secs 2024-04-26T21:40:00.447 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35340.201879 secs 2024-04-26T21:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35340.201879 secs 2024-04-26T21:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:40:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:01 smithi194 ceph-mon[33017]: pgmap v19105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:01.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:01 smithi033 ceph-mon[27845]: pgmap v19105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:01 smithi096 ceph-mon[32753]: pgmap v19105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:03 smithi194 ceph-mon[33017]: pgmap v19106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:03 smithi033 ceph-mon[27845]: pgmap v19106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:03 smithi096 ceph-mon[32753]: pgmap v19106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35342.242101 secs 2024-04-26T21:40:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35342.242101 secs 2024-04-26T21:40:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35342.242101 secs 2024-04-26T21:40:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35345.202015 secs 2024-04-26T21:40:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:05 smithi194 ceph-mon[33017]: pgmap v19107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35345.202015 secs 2024-04-26T21:40:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:05 smithi033 ceph-mon[27845]: pgmap v19107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35345.202015 secs 2024-04-26T21:40:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:05 smithi096 ceph-mon[32753]: pgmap v19107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:07 smithi194 ceph-mon[33017]: pgmap v19108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:07 smithi033 ceph-mon[27845]: pgmap v19108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:07 smithi096 ceph-mon[32753]: pgmap v19108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35347.242163 secs 2024-04-26T21:40:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:09 smithi194 ceph-mon[33017]: pgmap v19109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35347.242163 secs 2024-04-26T21:40:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:09 smithi033 ceph-mon[27845]: pgmap v19109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35347.242163 secs 2024-04-26T21:40:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:09 smithi096 ceph-mon[32753]: pgmap v19109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35350.202140 secs 2024-04-26T21:40:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35350.202140 secs 2024-04-26T21:40:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35350.202140 secs 2024-04-26T21:40:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:11 smithi194 ceph-mon[33017]: pgmap v19110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:11 smithi033 ceph-mon[27845]: pgmap v19110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:11 smithi096 ceph-mon[32753]: pgmap v19110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:13 smithi194 ceph-mon[33017]: pgmap v19111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:13 smithi033 ceph-mon[27845]: pgmap v19111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:13 smithi096 ceph-mon[32753]: pgmap v19111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:13.790+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:13.790+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:40:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:13.790+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35352.242222 secs 2024-04-26T21:40:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35352.242222 secs 2024-04-26T21:40:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35352.242222 secs 2024-04-26T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35355.202250 secs 2024-04-26T21:40:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:15 smithi194 ceph-mon[33017]: pgmap v19112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35355.202250 secs 2024-04-26T21:40:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:15 smithi033 ceph-mon[27845]: pgmap v19112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35355.202250 secs 2024-04-26T21:40:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:15 smithi096 ceph-mon[32753]: pgmap v19112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:17 smithi194 ceph-mon[33017]: pgmap v19113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:17 smithi033 ceph-mon[27845]: pgmap v19113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:17 smithi096 ceph-mon[32753]: pgmap v19113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35357.242280 secs 2024-04-26T21:40:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35357.242280 secs 2024-04-26T21:40:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35357.242280 secs 2024-04-26T21:40:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:19 smithi194 ceph-mon[33017]: pgmap v19114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35360.202342 secs 2024-04-26T21:40:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:19 smithi033 ceph-mon[27845]: pgmap v19114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35360.202342 secs 2024-04-26T21:40:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:19 smithi096 ceph-mon[32753]: pgmap v19114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35360.202342 secs 2024-04-26T21:40:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:21 smithi194 ceph-mon[33017]: pgmap v19115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:21 smithi033 ceph-mon[27845]: pgmap v19115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:21 smithi096 ceph-mon[32753]: pgmap v19115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:23.342 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:23 smithi033 ceph-mon[27845]: pgmap v19116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:23.342 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35362.242337 secs 2024-04-26T21:40:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:23 smithi194 ceph-mon[33017]: pgmap v19116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35362.242337 secs 2024-04-26T21:40:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:23 smithi096 ceph-mon[32753]: pgmap v19116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35362.242337 secs 2024-04-26T21:40:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35365.202474 secs 2024-04-26T21:40:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35365.202474 secs 2024-04-26T21:40:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35365.202474 secs 2024-04-26T21:40:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:25 smithi194 ceph-mon[33017]: pgmap v19117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:25 smithi033 ceph-mon[27845]: pgmap v19117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:25 smithi096 ceph-mon[32753]: pgmap v19117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:27 smithi033 ceph-mon[27845]: pgmap v19118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:27 smithi096 ceph-mon[32753]: pgmap v19118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:27 smithi194 ceph-mon[33017]: pgmap v19118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35367.242419 secs 2024-04-26T21:40:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35367.242419 secs 2024-04-26T21:40:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:28.804+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:28.804+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:28.804+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:40:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35367.242419 secs 2024-04-26T21:40:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:29 smithi033 ceph-mon[27845]: pgmap v19119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35370.202597 secs 2024-04-26T21:40:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:29 smithi096 ceph-mon[32753]: pgmap v19119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35370.202597 secs 2024-04-26T21:40:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:29 smithi194 ceph-mon[33017]: pgmap v19119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:40:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35370.202597 secs 2024-04-26T21:40:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:31 smithi033 ceph-mon[27845]: pgmap v19120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:31 smithi096 ceph-mon[32753]: pgmap v19120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:31 smithi194 ceph-mon[33017]: pgmap v19120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:40:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:33 smithi033 ceph-mon[27845]: pgmap v19121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35372.242519 secs 2024-04-26T21:40:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:33 smithi096 ceph-mon[32753]: pgmap v19121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35372.242519 secs 2024-04-26T21:40:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:33 smithi194 ceph-mon[33017]: pgmap v19121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35372.242519 secs 2024-04-26T21:40:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35375.202702 secs 2024-04-26T21:40:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35375.202702 secs 2024-04-26T21:40:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35375.202702 secs 2024-04-26T21:40:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:35 smithi033 ceph-mon[27845]: pgmap v19122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:35 smithi096 ceph-mon[32753]: pgmap v19122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:35 smithi194 ceph-mon[33017]: pgmap v19122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:37 smithi033 ceph-mon[27845]: pgmap v19123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:37 smithi096 ceph-mon[32753]: pgmap v19123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:37 smithi194 ceph-mon[33017]: pgmap v19123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35377.242603 secs 2024-04-26T21:40:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35377.242603 secs 2024-04-26T21:40:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35377.242603 secs 2024-04-26T21:40:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:39 smithi033 ceph-mon[27845]: pgmap v19124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35380.202824 secs 2024-04-26T21:40:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:39 smithi096 ceph-mon[32753]: pgmap v19124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35380.202824 secs 2024-04-26T21:40:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:39 smithi194 ceph-mon[33017]: pgmap v19124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35380.202824 secs 2024-04-26T21:40:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:41 smithi033 ceph-mon[27845]: pgmap v19125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:41 smithi096 ceph-mon[32753]: pgmap v19125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:41 smithi194 ceph-mon[33017]: pgmap v19125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:43 smithi033 ceph-mon[27845]: pgmap v19126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:43.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35382.242661 secs 2024-04-26T21:40:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:43 smithi096 ceph-mon[32753]: pgmap v19126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35382.242661 secs 2024-04-26T21:40:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:43.813+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:43.813+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:43.813+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:40:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:43 smithi194 ceph-mon[33017]: pgmap v19126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35382.242661 secs 2024-04-26T21:40:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35385.202970 secs 2024-04-26T21:40:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35385.202970 secs 2024-04-26T21:40:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35385.202970 secs 2024-04-26T21:40:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:45 smithi194 ceph-mon[33017]: pgmap v19127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:45 smithi033 ceph-mon[27845]: pgmap v19127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:45 smithi096 ceph-mon[32753]: pgmap v19127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:40:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:40:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:40:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:47 smithi194 ceph-mon[33017]: pgmap v19128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:40:47.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:47 smithi194 ceph-mon[33017]: pgmap v19129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:47 smithi033 ceph-mon[27845]: pgmap v19128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:40:47.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:47 smithi033 ceph-mon[27845]: pgmap v19129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:47 smithi096 ceph-mon[32753]: pgmap v19128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:40:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:47 smithi096 ceph-mon[32753]: pgmap v19129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35387.242719 secs 2024-04-26T21:40:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35387.242719 secs 2024-04-26T21:40:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35387.242719 secs 2024-04-26T21:40:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:49 smithi194 ceph-mon[33017]: pgmap v19130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35390.203121 secs 2024-04-26T21:40:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:49 smithi033 ceph-mon[27845]: pgmap v19130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35390.203121 secs 2024-04-26T21:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:49 smithi096 ceph-mon[32753]: pgmap v19130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35390.203121 secs 2024-04-26T21:40:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:51 smithi194 ceph-mon[33017]: pgmap v19131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:51 smithi033 ceph-mon[27845]: pgmap v19131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:51 smithi096 ceph-mon[32753]: pgmap v19131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:53 smithi194 ceph-mon[33017]: pgmap v19132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35392.242777 secs 2024-04-26T21:40:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:53 smithi033 ceph-mon[27845]: pgmap v19132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35392.242777 secs 2024-04-26T21:40:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:53 smithi096 ceph-mon[32753]: pgmap v19132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35392.242777 secs 2024-04-26T21:40:54.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35395.203237 secs 2024-04-26T21:40:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35395.203237 secs 2024-04-26T21:40:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35395.203237 secs 2024-04-26T21:40:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:55 smithi194 ceph-mon[33017]: pgmap v19133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:55 smithi033 ceph-mon[27845]: pgmap v19133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:55 smithi096 ceph-mon[32753]: pgmap v19133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:40:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:57 smithi194 ceph-mon[33017]: pgmap v19134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:40:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:57 smithi033 ceph-mon[27845]: pgmap v19134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:40:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:57 smithi096 ceph-mon[32753]: pgmap v19134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:40:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:58.826+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:58.827+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:40:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:40:58.827+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:40:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35397.242837 secs 2024-04-26T21:40:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35397.242837 secs 2024-04-26T21:40:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35397.242837 secs 2024-04-26T21:40:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:59 smithi194 ceph-mon[33017]: pgmap v19135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:40:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35400.203380 secs 2024-04-26T21:40:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:59 smithi033 ceph-mon[27845]: pgmap v19135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:40:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35400.203380 secs 2024-04-26T21:40:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:59 smithi096 ceph-mon[32753]: pgmap v19135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:40:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:40:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35400.203380 secs 2024-04-26T21:41:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:01 smithi194 ceph-mon[33017]: pgmap v19136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:01 smithi033 ceph-mon[27845]: pgmap v19136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:01 smithi096 ceph-mon[32753]: pgmap v19136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:03 smithi194 ceph-mon[33017]: pgmap v19137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:03.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35402.242918 secs 2024-04-26T21:41:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:03 smithi033 ceph-mon[27845]: pgmap v19137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35402.242918 secs 2024-04-26T21:41:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:03 smithi096 ceph-mon[32753]: pgmap v19137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35402.242918 secs 2024-04-26T21:41:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35405.203513 secs 2024-04-26T21:41:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35405.203513 secs 2024-04-26T21:41:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35405.203513 secs 2024-04-26T21:41:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:05 smithi194 ceph-mon[33017]: pgmap v19138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:05 smithi033 ceph-mon[27845]: pgmap v19138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:05 smithi096 ceph-mon[32753]: pgmap v19138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:07 smithi194 ceph-mon[33017]: pgmap v19139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:07 smithi033 ceph-mon[27845]: pgmap v19139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:07 smithi096 ceph-mon[32753]: pgmap v19139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35407.243001 secs 2024-04-26T21:41:08.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35407.243001 secs 2024-04-26T21:41:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35407.243001 secs 2024-04-26T21:41:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:09 smithi194 ceph-mon[33017]: pgmap v19140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35410.203623 secs 2024-04-26T21:41:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:09 smithi033 ceph-mon[27845]: pgmap v19140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35410.203623 secs 2024-04-26T21:41:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:09 smithi096 ceph-mon[32753]: pgmap v19140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35410.203623 secs 2024-04-26T21:41:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:11 smithi194 ceph-mon[33017]: pgmap v19141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:11 smithi033 ceph-mon[27845]: pgmap v19141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:11 smithi096 ceph-mon[32753]: pgmap v19141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:13.830+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:13.831+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:13.831+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:41:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:13 smithi194 ceph-mon[33017]: pgmap v19142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35412.243088 secs 2024-04-26T21:41:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:13 smithi033 ceph-mon[27845]: pgmap v19142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35412.243088 secs 2024-04-26T21:41:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:13 smithi096 ceph-mon[32753]: pgmap v19142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35412.243088 secs 2024-04-26T21:41:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35415.203722 secs 2024-04-26T21:41:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35415.203722 secs 2024-04-26T21:41:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35415.203722 secs 2024-04-26T21:41:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:15 smithi194 ceph-mon[33017]: pgmap v19143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:15 smithi033 ceph-mon[27845]: pgmap v19143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:15 smithi096 ceph-mon[32753]: pgmap v19143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:17 smithi194 ceph-mon[33017]: pgmap v19144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:17 smithi033 ceph-mon[27845]: pgmap v19144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:17 smithi096 ceph-mon[32753]: pgmap v19144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35417.243178 secs 2024-04-26T21:41:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35417.243178 secs 2024-04-26T21:41:18.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35417.243178 secs 2024-04-26T21:41:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:19 smithi194 ceph-mon[33017]: pgmap v19145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35420.203852 secs 2024-04-26T21:41:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:19 smithi033 ceph-mon[27845]: pgmap v19145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35420.203852 secs 2024-04-26T21:41:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:19 smithi096 ceph-mon[32753]: pgmap v19145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35420.203852 secs 2024-04-26T21:41:21.816 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0012040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0012040 comp rx=0 tx=0).stop 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.816+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:21.817 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.817+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:21.819+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0012040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:21 smithi194 ceph-mon[33017]: pgmap v19146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:21 smithi033 ceph-mon[27845]: pgmap v19146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:21 smithi096 ceph-mon[32753]: pgmap v19146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:23 smithi194 ceph-mon[33017]: pgmap v19147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:23.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35422.243261 secs 2024-04-26T21:41:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:23 smithi033 ceph-mon[27845]: pgmap v19147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:23.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35422.243261 secs 2024-04-26T21:41:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:23 smithi096 ceph-mon[32753]: pgmap v19147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35422.243261 secs 2024-04-26T21:41:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35425.203991 secs 2024-04-26T21:41:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35425.203991 secs 2024-04-26T21:41:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35425.203991 secs 2024-04-26T21:41:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:25 smithi033 ceph-mon[27845]: pgmap v19148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:25 smithi096 ceph-mon[32753]: pgmap v19148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:25 smithi194 ceph-mon[33017]: pgmap v19148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:26 smithi033 ceph-mon[27845]: pgmap v19149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:26 smithi096 ceph-mon[32753]: pgmap v19149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:26 smithi194 ceph-mon[33017]: pgmap v19149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:27.829+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:41:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:27.829+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:41:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:27.829+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 277 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T21:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:27.829+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 277 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:41:28.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:28.832+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:28.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:28.832+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:28.832 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:28.832+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:41:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:28 smithi194 ceph-mon[33017]: pgmap v19150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35427.243345 secs 2024-04-26T21:41:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:28 smithi033 ceph-mon[27845]: pgmap v19150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:29.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35427.243345 secs 2024-04-26T21:41:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:28 smithi096 ceph-mon[32753]: pgmap v19150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35427.243345 secs 2024-04-26T21:41:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35430.204151 secs 2024-04-26T21:41:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35430.204151 secs 2024-04-26T21:41:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35430.204151 secs 2024-04-26T21:41:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:30 smithi194 ceph-mon[33017]: pgmap v19151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:30 smithi033 ceph-mon[27845]: pgmap v19151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:30 smithi096 ceph-mon[32753]: pgmap v19151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:32 smithi194 ceph-mon[33017]: pgmap v19152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:32 smithi033 ceph-mon[27845]: pgmap v19152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:32 smithi096 ceph-mon[32753]: pgmap v19152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35432.243405 secs 2024-04-26T21:41:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35432.243405 secs 2024-04-26T21:41:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35432.243405 secs 2024-04-26T21:41:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35435.204268 secs 2024-04-26T21:41:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:35 smithi194 ceph-mon[33017]: pgmap v19153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35435.204268 secs 2024-04-26T21:41:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:35 smithi033 ceph-mon[27845]: pgmap v19153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35435.204268 secs 2024-04-26T21:41:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:35 smithi096 ceph-mon[32753]: pgmap v19153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:37 smithi194 ceph-mon[33017]: pgmap v19154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:37 smithi033 ceph-mon[27845]: pgmap v19154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:37 smithi096 ceph-mon[32753]: pgmap v19154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:39 smithi194 ceph-mon[33017]: pgmap v19155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35437.243467 secs 2024-04-26T21:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:39 smithi033 ceph-mon[27845]: pgmap v19155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35437.243467 secs 2024-04-26T21:41:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:39 smithi096 ceph-mon[32753]: pgmap v19155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35437.243467 secs 2024-04-26T21:41:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35440.204388 secs 2024-04-26T21:41:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35440.204388 secs 2024-04-26T21:41:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35440.204388 secs 2024-04-26T21:41:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:41 smithi194 ceph-mon[33017]: pgmap v19156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:41 smithi033 ceph-mon[27845]: pgmap v19156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:41 smithi096 ceph-mon[32753]: pgmap v19156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:43 smithi194 ceph-mon[33017]: pgmap v19157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:43 smithi033 ceph-mon[27845]: pgmap v19157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:43 smithi096 ceph-mon[32753]: pgmap v19157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:43.837+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:43.837+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:43.837+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:41:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35442.243548 secs 2024-04-26T21:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35442.243548 secs 2024-04-26T21:41:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35442.243548 secs 2024-04-26T21:41:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35445.204509 secs 2024-04-26T21:41:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:45 smithi194 ceph-mon[33017]: pgmap v19158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35445.204509 secs 2024-04-26T21:41:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:45 smithi033 ceph-mon[27845]: pgmap v19158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35445.204509 secs 2024-04-26T21:41:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:45 smithi096 ceph-mon[32753]: pgmap v19158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:47 smithi194 ceph-mon[33017]: pgmap v19159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:47 smithi033 ceph-mon[27845]: pgmap v19159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:47 smithi096 ceph-mon[32753]: pgmap v19159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:41:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:48.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:41:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:41:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:49 smithi194 ceph-mon[33017]: pgmap v19160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:49 smithi194 ceph-mon[33017]: pgmap v19161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35447.243630 secs 2024-04-26T21:41:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:49 smithi033 ceph-mon[27845]: pgmap v19160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:49 smithi033 ceph-mon[27845]: pgmap v19161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35447.243630 secs 2024-04-26T21:41:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:49 smithi096 ceph-mon[32753]: pgmap v19160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:41:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:49 smithi096 ceph-mon[32753]: pgmap v19161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35447.243630 secs 2024-04-26T21:41:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35450.204605 secs 2024-04-26T21:41:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35450.204605 secs 2024-04-26T21:41:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35450.204605 secs 2024-04-26T21:41:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:51 smithi194 ceph-mon[33017]: pgmap v19162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:51 smithi033 ceph-mon[27845]: pgmap v19162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:51 smithi096 ceph-mon[32753]: pgmap v19162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:53 smithi194 ceph-mon[33017]: pgmap v19163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:53 smithi033 ceph-mon[27845]: pgmap v19163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:53 smithi096 ceph-mon[32753]: pgmap v19163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35452.243713 secs 2024-04-26T21:41:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35452.243713 secs 2024-04-26T21:41:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35452.243713 secs 2024-04-26T21:41:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:55 smithi194 ceph-mon[33017]: pgmap v19164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35455.204730 secs 2024-04-26T21:41:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:55 smithi033 ceph-mon[27845]: pgmap v19164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35455.204730 secs 2024-04-26T21:41:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:55 smithi096 ceph-mon[32753]: pgmap v19164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35455.204730 secs 2024-04-26T21:41:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:57 smithi033 ceph-mon[27845]: pgmap v19165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:57 smithi096 ceph-mon[32753]: pgmap v19165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:57 smithi194 ceph-mon[33017]: pgmap v19165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:41:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:58.850+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:58.850+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:41:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:41:58.850+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:41:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:59 smithi033 ceph-mon[27845]: pgmap v19166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:41:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:41:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35457.243797 secs 2024-04-26T21:41:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:59 smithi096 ceph-mon[32753]: pgmap v19166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:41:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:41:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35457.243797 secs 2024-04-26T21:41:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:59 smithi194 ceph-mon[33017]: pgmap v19166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:41:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:41:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35457.243797 secs 2024-04-26T21:42:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35460.204850 secs 2024-04-26T21:42:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35460.204850 secs 2024-04-26T21:42:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35460.204850 secs 2024-04-26T21:42:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:01 smithi033 ceph-mon[27845]: pgmap v19167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:01 smithi096 ceph-mon[32753]: pgmap v19167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:01 smithi194 ceph-mon[33017]: pgmap v19167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:03 smithi033 ceph-mon[27845]: pgmap v19168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:03 smithi096 ceph-mon[32753]: pgmap v19168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:03 smithi194 ceph-mon[33017]: pgmap v19168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35462.243878 secs 2024-04-26T21:42:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35462.243878 secs 2024-04-26T21:42:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35462.243878 secs 2024-04-26T21:42:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:05 smithi033 ceph-mon[27845]: pgmap v19169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35465.204988 secs 2024-04-26T21:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:05 smithi096 ceph-mon[32753]: pgmap v19169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35465.204988 secs 2024-04-26T21:42:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:05 smithi194 ceph-mon[33017]: pgmap v19169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35465.204988 secs 2024-04-26T21:42:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:07 smithi033 ceph-mon[27845]: pgmap v19170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:07 smithi096 ceph-mon[32753]: pgmap v19170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:07 smithi194 ceph-mon[33017]: pgmap v19170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:09 smithi033 ceph-mon[27845]: pgmap v19171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35467.243969 secs 2024-04-26T21:42:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:09 smithi096 ceph-mon[32753]: pgmap v19171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35467.243969 secs 2024-04-26T21:42:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:09 smithi194 ceph-mon[33017]: pgmap v19171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35467.243969 secs 2024-04-26T21:42:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35470.205129 secs 2024-04-26T21:42:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35470.205129 secs 2024-04-26T21:42:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35470.205129 secs 2024-04-26T21:42:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:11 smithi033 ceph-mon[27845]: pgmap v19172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:11 smithi096 ceph-mon[32753]: pgmap v19172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:11 smithi194 ceph-mon[33017]: pgmap v19172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:13 smithi033 ceph-mon[27845]: pgmap v19173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:13 smithi096 ceph-mon[32753]: pgmap v19173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:13 smithi194 ceph-mon[33017]: pgmap v19173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:13.852+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:13.852+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:13.852+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:42:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35472.244049 secs 2024-04-26T21:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35472.244049 secs 2024-04-26T21:42:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35472.244049 secs 2024-04-26T21:42:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:15 smithi194 ceph-mon[33017]: pgmap v19174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35475.205232 secs 2024-04-26T21:42:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:15 smithi033 ceph-mon[27845]: pgmap v19174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35475.205232 secs 2024-04-26T21:42:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:15 smithi096 ceph-mon[32753]: pgmap v19174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:42:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35475.205232 secs 2024-04-26T21:42:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:17 smithi194 ceph-mon[33017]: pgmap v19175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:17 smithi033 ceph-mon[27845]: pgmap v19175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:17 smithi096 ceph-mon[32753]: pgmap v19175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:42:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:19 smithi194 ceph-mon[33017]: pgmap v19176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35477.244180 secs 2024-04-26T21:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:19 smithi033 ceph-mon[27845]: pgmap v19176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35477.244180 secs 2024-04-26T21:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:19 smithi096 ceph-mon[32753]: pgmap v19176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35477.244180 secs 2024-04-26T21:42:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35480.205356 secs 2024-04-26T21:42:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35480.205356 secs 2024-04-26T21:42:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35480.205356 secs 2024-04-26T21:42:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:21 smithi194 ceph-mon[33017]: pgmap v19177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:21 smithi033 ceph-mon[27845]: pgmap v19177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:21 smithi096 ceph-mon[32753]: pgmap v19177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:23 smithi194 ceph-mon[33017]: pgmap v19178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:23 smithi033 ceph-mon[27845]: pgmap v19178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:23 smithi096 ceph-mon[32753]: pgmap v19178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:24.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35482.244260 secs 2024-04-26T21:42:24.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35482.244260 secs 2024-04-26T21:42:24.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35482.244260 secs 2024-04-26T21:42:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:25 smithi194 ceph-mon[33017]: pgmap v19179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35485.205481 secs 2024-04-26T21:42:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:25 smithi033 ceph-mon[27845]: pgmap v19179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35485.205481 secs 2024-04-26T21:42:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:25 smithi096 ceph-mon[32753]: pgmap v19179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35485.205481 secs 2024-04-26T21:42:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:27 smithi194 ceph-mon[33017]: pgmap v19180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:27 smithi033 ceph-mon[27845]: pgmap v19180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:27 smithi096 ceph-mon[32753]: pgmap v19180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:28.860+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:28.860+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:28.860+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:42:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:29 smithi194 ceph-mon[33017]: pgmap v19181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35487.244342 secs 2024-04-26T21:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:29 smithi033 ceph-mon[27845]: pgmap v19181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35487.244342 secs 2024-04-26T21:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:29 smithi096 ceph-mon[32753]: pgmap v19181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35487.244342 secs 2024-04-26T21:42:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35490.205602 secs 2024-04-26T21:42:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35490.205602 secs 2024-04-26T21:42:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35490.205602 secs 2024-04-26T21:42:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:31 smithi194 ceph-mon[33017]: pgmap v19182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:31 smithi033 ceph-mon[27845]: pgmap v19182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:31 smithi096 ceph-mon[32753]: pgmap v19182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:33 smithi194 ceph-mon[33017]: pgmap v19183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:33 smithi033 ceph-mon[27845]: pgmap v19183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:33 smithi096 ceph-mon[32753]: pgmap v19183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35492.244427 secs 2024-04-26T21:42:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35492.244427 secs 2024-04-26T21:42:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35492.244427 secs 2024-04-26T21:42:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:35 smithi194 ceph-mon[33017]: pgmap v19184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35495.205725 secs 2024-04-26T21:42:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:35 smithi033 ceph-mon[27845]: pgmap v19184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35495.205725 secs 2024-04-26T21:42:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:35 smithi096 ceph-mon[32753]: pgmap v19184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35495.205725 secs 2024-04-26T21:42:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:37 smithi194 ceph-mon[33017]: pgmap v19185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:37 smithi033 ceph-mon[27845]: pgmap v19185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:37 smithi096 ceph-mon[32753]: pgmap v19185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35497.244511 secs 2024-04-26T21:42:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35497.244511 secs 2024-04-26T21:42:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35497.244511 secs 2024-04-26T21:42:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:39 smithi194 ceph-mon[33017]: pgmap v19186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35500.205867 secs 2024-04-26T21:42:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:39 smithi033 ceph-mon[27845]: pgmap v19186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35500.205867 secs 2024-04-26T21:42:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:39 smithi096 ceph-mon[32753]: pgmap v19186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35500.205867 secs 2024-04-26T21:42:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:41 smithi033 ceph-mon[27845]: pgmap v19187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:41 smithi096 ceph-mon[32753]: pgmap v19187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:41 smithi194 ceph-mon[33017]: pgmap v19187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:43.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:43.874+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:43.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:43.874+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:43.875 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:43.874+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:42:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:43 smithi033 ceph-mon[27845]: pgmap v19188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35502.244593 secs 2024-04-26T21:42:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:43 smithi096 ceph-mon[32753]: pgmap v19188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35502.244593 secs 2024-04-26T21:42:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:43 smithi194 ceph-mon[33017]: pgmap v19188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35502.244593 secs 2024-04-26T21:42:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35505.205974 secs 2024-04-26T21:42:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35505.205974 secs 2024-04-26T21:42:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35505.205974 secs 2024-04-26T21:42:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:45 smithi033 ceph-mon[27845]: pgmap v19189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:45 smithi096 ceph-mon[32753]: pgmap v19189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:45 smithi194 ceph-mon[33017]: pgmap v19189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:46.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:46 smithi033 ceph-mon[27845]: pgmap v19190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:46 smithi096 ceph-mon[32753]: pgmap v19190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:46 smithi194 ceph-mon[33017]: pgmap v19190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:48 smithi194 ceph-mon[33017]: pgmap v19191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35507.244681 secs 2024-04-26T21:42:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:48 smithi033 ceph-mon[27845]: pgmap v19191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35507.244681 secs 2024-04-26T21:42:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:48 smithi096 ceph-mon[32753]: pgmap v19191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:42:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35507.244681 secs 2024-04-26T21:42:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35510.206036 secs 2024-04-26T21:42:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35510.206036 secs 2024-04-26T21:42:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35510.206036 secs 2024-04-26T21:42:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:50 smithi033 ceph-mon[27845]: pgmap v19192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:50 smithi033 ceph-mon[27845]: pgmap v19193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:42:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:50 smithi033 ceph-mon[27845]: pgmap v19194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:51.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:50 smithi096 ceph-mon[32753]: pgmap v19192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:50 smithi096 ceph-mon[32753]: pgmap v19193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:42:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:50 smithi096 ceph-mon[32753]: pgmap v19194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:51.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:50 smithi194 ceph-mon[33017]: pgmap v19192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:50 smithi194 ceph-mon[33017]: pgmap v19193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:42:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:50 smithi194 ceph-mon[33017]: pgmap v19194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:51.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:42:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:53 smithi194 ceph-mon[33017]: pgmap v19195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:53 smithi033 ceph-mon[27845]: pgmap v19195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:53 smithi096 ceph-mon[32753]: pgmap v19195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:54.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35512.244761 secs 2024-04-26T21:42:54.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35512.244761 secs 2024-04-26T21:42:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35512.244761 secs 2024-04-26T21:42:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35515.206194 secs 2024-04-26T21:42:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:55 smithi194 ceph-mon[33017]: pgmap v19196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:42:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35515.206194 secs 2024-04-26T21:42:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:55 smithi033 ceph-mon[27845]: pgmap v19196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35515.206194 secs 2024-04-26T21:42:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:55 smithi096 ceph-mon[32753]: pgmap v19196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:42:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:57 smithi194 ceph-mon[33017]: pgmap v19197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:57 smithi033 ceph-mon[27845]: pgmap v19197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:57 smithi096 ceph-mon[32753]: pgmap v19197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:58.884+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:58.885+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:42:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:42:58.885+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:42:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35517.244833 secs 2024-04-26T21:42:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35517.244833 secs 2024-04-26T21:42:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35517.244833 secs 2024-04-26T21:42:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:59 smithi033 ceph-mon[27845]: pgmap v19198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:42:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:42:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35520.206329 secs 2024-04-26T21:42:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:59 smithi096 ceph-mon[32753]: pgmap v19198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:42:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:42:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35520.206329 secs 2024-04-26T21:43:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:59 smithi194 ceph-mon[33017]: pgmap v19198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:43:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:42:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35520.206329 secs 2024-04-26T21:43:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:01 smithi033 ceph-mon[27845]: pgmap v19199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:43:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:01 smithi096 ceph-mon[32753]: pgmap v19199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:43:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:01 smithi194 ceph-mon[33017]: pgmap v19199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:43:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:03 smithi033 ceph-mon[27845]: pgmap v19200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35522.244914 secs 2024-04-26T21:43:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:03 smithi096 ceph-mon[32753]: pgmap v19200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35522.244914 secs 2024-04-26T21:43:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:03 smithi194 ceph-mon[33017]: pgmap v19200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35522.244914 secs 2024-04-26T21:43:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35525.206459 secs 2024-04-26T21:43:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35525.206459 secs 2024-04-26T21:43:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35525.206459 secs 2024-04-26T21:43:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:05 smithi033 ceph-mon[27845]: pgmap v19201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:05 smithi096 ceph-mon[32753]: pgmap v19201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:05 smithi194 ceph-mon[33017]: pgmap v19201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:06 smithi033 ceph-mon[27845]: pgmap v19202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:06 smithi096 ceph-mon[32753]: pgmap v19202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:06 smithi194 ceph-mon[33017]: pgmap v19202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35527.244984 secs 2024-04-26T21:43:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:09 smithi194 ceph-mon[33017]: pgmap v19203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35527.244984 secs 2024-04-26T21:43:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:09 smithi033 ceph-mon[27845]: pgmap v19203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35527.244984 secs 2024-04-26T21:43:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:09 smithi096 ceph-mon[32753]: pgmap v19203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35530.206585 secs 2024-04-26T21:43:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35530.206585 secs 2024-04-26T21:43:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35530.206585 secs 2024-04-26T21:43:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:11 smithi194 ceph-mon[33017]: pgmap v19204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:11 smithi033 ceph-mon[27845]: pgmap v19204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:11 smithi096 ceph-mon[32753]: pgmap v19204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:13 smithi194 ceph-mon[33017]: pgmap v19205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:13 smithi033 ceph-mon[27845]: pgmap v19205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:13 smithi096 ceph-mon[32753]: pgmap v19205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:13.900+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:13.900+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:13.900+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:43:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35532.245044 secs 2024-04-26T21:43:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35532.245044 secs 2024-04-26T21:43:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35532.245044 secs 2024-04-26T21:43:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35535.206698 secs 2024-04-26T21:43:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:15 smithi194 ceph-mon[33017]: pgmap v19206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35535.206698 secs 2024-04-26T21:43:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:15 smithi033 ceph-mon[27845]: pgmap v19206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35535.206698 secs 2024-04-26T21:43:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:15 smithi096 ceph-mon[32753]: pgmap v19206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:17 smithi194 ceph-mon[33017]: pgmap v19207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:17 smithi033 ceph-mon[27845]: pgmap v19207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:17 smithi096 ceph-mon[32753]: pgmap v19207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35537.245105 secs 2024-04-26T21:43:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35537.245105 secs 2024-04-26T21:43:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35537.245105 secs 2024-04-26T21:43:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:19 smithi194 ceph-mon[33017]: pgmap v19208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35540.206819 secs 2024-04-26T21:43:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:19 smithi033 ceph-mon[27845]: pgmap v19208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35540.206819 secs 2024-04-26T21:43:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:19 smithi096 ceph-mon[32753]: pgmap v19208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35540.206819 secs 2024-04-26T21:43:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:21 smithi194 ceph-mon[33017]: pgmap v19209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:21 smithi033 ceph-mon[27845]: pgmap v19209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:21 smithi096 ceph-mon[32753]: pgmap v19209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:23 smithi194 ceph-mon[33017]: pgmap v19210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:23.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35542.245188 secs 2024-04-26T21:43:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:23 smithi033 ceph-mon[27845]: pgmap v19210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35542.245188 secs 2024-04-26T21:43:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:23 smithi096 ceph-mon[32753]: pgmap v19210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35542.245188 secs 2024-04-26T21:43:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35545.206942 secs 2024-04-26T21:43:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35545.206942 secs 2024-04-26T21:43:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35545.206942 secs 2024-04-26T21:43:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:25 smithi194 ceph-mon[33017]: pgmap v19211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:25 smithi033 ceph-mon[27845]: pgmap v19211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:25 smithi096 ceph-mon[32753]: pgmap v19211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:27 smithi194 ceph-mon[33017]: pgmap v19212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:27 smithi033 ceph-mon[27845]: pgmap v19212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:27 smithi096 ceph-mon[32753]: pgmap v19212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35547.245281 secs 2024-04-26T21:43:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35547.245281 secs 2024-04-26T21:43:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35547.245281 secs 2024-04-26T21:43:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:28.916+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:28.916+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:43:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:28.916+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:29 smithi033 ceph-mon[27845]: pgmap v19213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35550.207059 secs 2024-04-26T21:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:29 smithi096 ceph-mon[32753]: pgmap v19213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35550.207059 secs 2024-04-26T21:43:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:29 smithi194 ceph-mon[33017]: pgmap v19213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35550.207059 secs 2024-04-26T21:43:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:31 smithi033 ceph-mon[27845]: pgmap v19214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:31 smithi096 ceph-mon[32753]: pgmap v19214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:31 smithi194 ceph-mon[33017]: pgmap v19214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:33 smithi033 ceph-mon[27845]: pgmap v19215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35552.245364 secs 2024-04-26T21:43:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:33 smithi096 ceph-mon[32753]: pgmap v19215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35552.245364 secs 2024-04-26T21:43:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:33 smithi194 ceph-mon[33017]: pgmap v19215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35552.245364 secs 2024-04-26T21:43:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35555.207175 secs 2024-04-26T21:43:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35555.207175 secs 2024-04-26T21:43:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35555.207175 secs 2024-04-26T21:43:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:35 smithi033 ceph-mon[27845]: pgmap v19216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:35 smithi096 ceph-mon[32753]: pgmap v19216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:35 smithi194 ceph-mon[33017]: pgmap v19216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:37 smithi033 ceph-mon[27845]: pgmap v19217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:37 smithi096 ceph-mon[32753]: pgmap v19217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:37 smithi194 ceph-mon[33017]: pgmap v19217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35557.245449 secs 2024-04-26T21:43:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35557.245449 secs 2024-04-26T21:43:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35557.245449 secs 2024-04-26T21:43:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:39 smithi033 ceph-mon[27845]: pgmap v19218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35560.207298 secs 2024-04-26T21:43:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:39 smithi096 ceph-mon[32753]: pgmap v19218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35560.207298 secs 2024-04-26T21:43:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:39 smithi194 ceph-mon[33017]: pgmap v19218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35560.207298 secs 2024-04-26T21:43:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:41 smithi033 ceph-mon[27845]: pgmap v19219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:41 smithi096 ceph-mon[32753]: pgmap v19219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:41 smithi194 ceph-mon[33017]: pgmap v19219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:42.850+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:43:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:42.850+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:43:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:42.851+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 278 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T21:43:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:42.851+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 278 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:43:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:43 smithi033 ceph-mon[27845]: pgmap v19220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35562.245532 secs 2024-04-26T21:43:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:43 smithi096 ceph-mon[32753]: pgmap v19220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35562.245532 secs 2024-04-26T21:43:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:43 smithi194 ceph-mon[33017]: pgmap v19220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35562.245532 secs 2024-04-26T21:43:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:43.929+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:43.929+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:43.929+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:43:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35565.207417 secs 2024-04-26T21:43:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35565.207417 secs 2024-04-26T21:43:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35565.207417 secs 2024-04-26T21:43:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:45 smithi033 ceph-mon[27845]: pgmap v19221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:45 smithi096 ceph-mon[32753]: pgmap v19221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:45 smithi194 ceph-mon[33017]: pgmap v19221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:47 smithi033 ceph-mon[27845]: pgmap v19222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:47 smithi096 ceph-mon[32753]: pgmap v19222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:47 smithi194 ceph-mon[33017]: pgmap v19222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:48.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35567.245611 secs 2024-04-26T21:43:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35567.245611 secs 2024-04-26T21:43:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35567.245611 secs 2024-04-26T21:43:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:49 smithi194 ceph-mon[33017]: pgmap v19223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35570.207524 secs 2024-04-26T21:43:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:49 smithi033 ceph-mon[27845]: pgmap v19223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35570.207524 secs 2024-04-26T21:43:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:49 smithi096 ceph-mon[32753]: pgmap v19223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35570.207524 secs 2024-04-26T21:43:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:51 smithi194 ceph-mon[33017]: pgmap v19224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:51 smithi033 ceph-mon[27845]: pgmap v19224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:51 smithi096 ceph-mon[32753]: pgmap v19224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:52.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:52 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:43:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:52 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:43:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:52 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:43:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:53 smithi194 ceph-mon[33017]: pgmap v19225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:53 smithi194 ceph-mon[33017]: pgmap v19226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35572.245697 secs 2024-04-26T21:43:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:53 smithi033 ceph-mon[27845]: pgmap v19225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:53 smithi033 ceph-mon[27845]: pgmap v19226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35572.245697 secs 2024-04-26T21:43:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:53 smithi096 ceph-mon[32753]: pgmap v19225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:43:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:53 smithi096 ceph-mon[32753]: pgmap v19226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35572.245697 secs 2024-04-26T21:43:54.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35575.207600 secs 2024-04-26T21:43:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35575.207600 secs 2024-04-26T21:43:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35575.207600 secs 2024-04-26T21:43:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:55 smithi194 ceph-mon[33017]: pgmap v19227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:55 smithi033 ceph-mon[27845]: pgmap v19227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:55 smithi096 ceph-mon[32753]: pgmap v19227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:57 smithi194 ceph-mon[33017]: pgmap v19228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:57 smithi033 ceph-mon[27845]: pgmap v19228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:57 smithi096 ceph-mon[32753]: pgmap v19228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35577.245783 secs 2024-04-26T21:43:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35577.245783 secs 2024-04-26T21:43:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35577.245783 secs 2024-04-26T21:43:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:58.943+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:58.943+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:43:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:43:58.943+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:43:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:59 smithi194 ceph-mon[33017]: pgmap v19229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:43:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35580.207742 secs 2024-04-26T21:43:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:59 smithi033 ceph-mon[27845]: pgmap v19229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:43:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35580.207742 secs 2024-04-26T21:43:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:59 smithi096 ceph-mon[32753]: pgmap v19229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:43:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:43:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35580.207742 secs 2024-04-26T21:44:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:01 smithi194 ceph-mon[33017]: pgmap v19230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:44:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:01 smithi033 ceph-mon[27845]: pgmap v19230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:44:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:01 smithi096 ceph-mon[32753]: pgmap v19230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:44:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:03 smithi194 ceph-mon[33017]: pgmap v19231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:44:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35582.245866 secs 2024-04-26T21:44:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:03 smithi033 ceph-mon[27845]: pgmap v19231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:44:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35582.245866 secs 2024-04-26T21:44:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:03 smithi096 ceph-mon[32753]: pgmap v19231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:44:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35582.245866 secs 2024-04-26T21:44:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35585.207879 secs 2024-04-26T21:44:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35585.207879 secs 2024-04-26T21:44:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35585.207879 secs 2024-04-26T21:44:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:05 smithi194 ceph-mon[33017]: pgmap v19232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:05 smithi033 ceph-mon[27845]: pgmap v19232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:05 smithi096 ceph-mon[32753]: pgmap v19232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:07 smithi194 ceph-mon[33017]: pgmap v19233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:07.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:07 smithi033 ceph-mon[27845]: pgmap v19233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:07 smithi096 ceph-mon[32753]: pgmap v19233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35587.245941 secs 2024-04-26T21:44:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35587.245941 secs 2024-04-26T21:44:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35587.245941 secs 2024-04-26T21:44:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:09 smithi194 ceph-mon[33017]: pgmap v19234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35590.208007 secs 2024-04-26T21:44:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:09 smithi033 ceph-mon[27845]: pgmap v19234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35590.208007 secs 2024-04-26T21:44:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:09 smithi096 ceph-mon[32753]: pgmap v19234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35590.208007 secs 2024-04-26T21:44:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:11 smithi194 ceph-mon[33017]: pgmap v19235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:11 smithi033 ceph-mon[27845]: pgmap v19235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:11 smithi096 ceph-mon[32753]: pgmap v19235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:13 smithi194 ceph-mon[33017]: pgmap v19236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35592.245972 secs 2024-04-26T21:44:13.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:13 smithi033 ceph-mon[27845]: pgmap v19236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:13.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35592.245972 secs 2024-04-26T21:44:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:13 smithi096 ceph-mon[32753]: pgmap v19236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35592.245972 secs 2024-04-26T21:44:13.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:13.950+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:13.950 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:13.950+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:13.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:13.950+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:44:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35595.208134 secs 2024-04-26T21:44:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35595.208134 secs 2024-04-26T21:44:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35595.208134 secs 2024-04-26T21:44:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:15 smithi194 ceph-mon[33017]: pgmap v19237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:15 smithi033 ceph-mon[27845]: pgmap v19237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:15.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:15 smithi096 ceph-mon[32753]: pgmap v19237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:17 smithi194 ceph-mon[33017]: pgmap v19238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:17 smithi033 ceph-mon[27845]: pgmap v19238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:17 smithi096 ceph-mon[32753]: pgmap v19238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35597.246035 secs 2024-04-26T21:44:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35597.246035 secs 2024-04-26T21:44:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35597.246035 secs 2024-04-26T21:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:19 smithi194 ceph-mon[33017]: pgmap v19239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35600.208259 secs 2024-04-26T21:44:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:19 smithi033 ceph-mon[27845]: pgmap v19239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35600.208259 secs 2024-04-26T21:44:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:19 smithi096 ceph-mon[32753]: pgmap v19239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35600.208259 secs 2024-04-26T21:44:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:21 smithi194 ceph-mon[33017]: pgmap v19240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:21 smithi033 ceph-mon[27845]: pgmap v19240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:21 smithi096 ceph-mon[32753]: pgmap v19240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:23 smithi194 ceph-mon[33017]: pgmap v19241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35602.246089 secs 2024-04-26T21:44:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:23 smithi033 ceph-mon[27845]: pgmap v19241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35602.246089 secs 2024-04-26T21:44:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:23 smithi096 ceph-mon[32753]: pgmap v19241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35602.246089 secs 2024-04-26T21:44:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35605.208409 secs 2024-04-26T21:44:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35605.208409 secs 2024-04-26T21:44:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35605.208409 secs 2024-04-26T21:44:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:25 smithi194 ceph-mon[33017]: pgmap v19242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:25 smithi033 ceph-mon[27845]: pgmap v19242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:25.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:25 smithi096 ceph-mon[32753]: pgmap v19242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:27.859+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:44:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:27.859+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:44:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:27.860+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 279 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T21:44:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:27.860+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 279 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:44:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:27 smithi033 ceph-mon[27845]: pgmap v19243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:27 smithi096 ceph-mon[32753]: pgmap v19243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:27 smithi194 ceph-mon[33017]: pgmap v19243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35607.246152 secs 2024-04-26T21:44:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35607.246152 secs 2024-04-26T21:44:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:28.963+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:28.963+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:44:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:28.963+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35607.246152 secs 2024-04-26T21:44:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:29 smithi033 ceph-mon[27845]: pgmap v19244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35610.208551 secs 2024-04-26T21:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:29 smithi096 ceph-mon[32753]: pgmap v19244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35610.208551 secs 2024-04-26T21:44:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:29 smithi194 ceph-mon[33017]: pgmap v19244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35610.208551 secs 2024-04-26T21:44:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:30 smithi033 ceph-mon[27845]: pgmap v19245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:30 smithi096 ceph-mon[32753]: pgmap v19245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:30 smithi194 ceph-mon[33017]: pgmap v19245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:33 smithi194 ceph-mon[33017]: pgmap v19246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:33 smithi033 ceph-mon[27845]: pgmap v19246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:33 smithi096 ceph-mon[32753]: pgmap v19246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35612.246214 secs 2024-04-26T21:44:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35612.246214 secs 2024-04-26T21:44:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35612.246214 secs 2024-04-26T21:44:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35615.208680 secs 2024-04-26T21:44:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:35 smithi194 ceph-mon[33017]: pgmap v19247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35615.208680 secs 2024-04-26T21:44:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:35 smithi033 ceph-mon[27845]: pgmap v19247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35615.208680 secs 2024-04-26T21:44:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:35 smithi096 ceph-mon[32753]: pgmap v19247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:37 smithi194 ceph-mon[33017]: pgmap v19248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:37 smithi033 ceph-mon[27845]: pgmap v19248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:37 smithi096 ceph-mon[32753]: pgmap v19248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:39 smithi194 ceph-mon[33017]: pgmap v19249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35617.246295 secs 2024-04-26T21:44:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:39 smithi033 ceph-mon[27845]: pgmap v19249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35617.246295 secs 2024-04-26T21:44:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:39 smithi096 ceph-mon[32753]: pgmap v19249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35617.246295 secs 2024-04-26T21:44:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35620.208800 secs 2024-04-26T21:44:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35620.208800 secs 2024-04-26T21:44:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35620.208800 secs 2024-04-26T21:44:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:41 smithi194 ceph-mon[33017]: pgmap v19250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:41 smithi033 ceph-mon[27845]: pgmap v19250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:41 smithi096 ceph-mon[32753]: pgmap v19250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:43 smithi194 ceph-mon[33017]: pgmap v19251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:43 smithi033 ceph-mon[27845]: pgmap v19251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:43 smithi096 ceph-mon[32753]: pgmap v19251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:43.978+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:43.978+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:43.978+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:44:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35622.246351 secs 2024-04-26T21:44:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35622.246351 secs 2024-04-26T21:44:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35622.246351 secs 2024-04-26T21:44:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35625.208941 secs 2024-04-26T21:44:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:45 smithi194 ceph-mon[33017]: pgmap v19252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35625.208941 secs 2024-04-26T21:44:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:45 smithi033 ceph-mon[27845]: pgmap v19252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35625.208941 secs 2024-04-26T21:44:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:45 smithi096 ceph-mon[32753]: pgmap v19252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:47 smithi194 ceph-mon[33017]: pgmap v19253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:47 smithi033 ceph-mon[27845]: pgmap v19253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:47 smithi096 ceph-mon[32753]: pgmap v19253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:49 smithi194 ceph-mon[33017]: pgmap v19254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35627.246436 secs 2024-04-26T21:44:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:49 smithi033 ceph-mon[27845]: pgmap v19254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35627.246436 secs 2024-04-26T21:44:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:49 smithi096 ceph-mon[32753]: pgmap v19254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35627.246436 secs 2024-04-26T21:44:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35630.209033 secs 2024-04-26T21:44:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35630.209033 secs 2024-04-26T21:44:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35630.209033 secs 2024-04-26T21:44:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:51 smithi194 ceph-mon[33017]: pgmap v19255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:51 smithi033 ceph-mon[27845]: pgmap v19255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:51 smithi096 ceph-mon[32753]: pgmap v19255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:53 smithi033 ceph-mon[27845]: pgmap v19256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:53 smithi096 ceph-mon[32753]: pgmap v19256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:53 smithi194 ceph-mon[33017]: pgmap v19256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:53.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35632.246503 secs 2024-04-26T21:44:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35632.246503 secs 2024-04-26T21:44:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35632.246503 secs 2024-04-26T21:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35635.209132 secs 2024-04-26T21:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:55 smithi033 ceph-mon[27845]: pgmap v19257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:55 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35635.209132 secs 2024-04-26T21:44:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:55 smithi096 ceph-mon[32753]: pgmap v19257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:55 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35635.209132 secs 2024-04-26T21:44:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:55 smithi194 ceph-mon[33017]: pgmap v19257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:55 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:56.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:57 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:44:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:57 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:44:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:57 smithi033 ceph-mon[27845]: pgmap v19258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:57.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:57 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:57 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:44:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:57 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:44:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:57 smithi096 ceph-mon[32753]: pgmap v19258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:57 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:57 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:44:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:57 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:44:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:57 smithi194 ceph-mon[33017]: pgmap v19258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:57.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:57 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:58.979+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:58.979+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:44:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:44:58.979+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:44:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: pgmap v19259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35637.246561 secs 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:59.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:59.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: pgmap v19260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:44:59.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: pgmap v19261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:44:59.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:44:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: pgmap v19259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35637.246561 secs 2024-04-26T21:44:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: pgmap v19260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: pgmap v19261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:44:59.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:44:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:44:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: pgmap v19259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35637.246561 secs 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: pgmap v19260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:44:59.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: pgmap v19261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:44:59.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:44:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:45:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35640.209278 secs 2024-04-26T21:45:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35640.209278 secs 2024-04-26T21:45:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35640.209278 secs 2024-04-26T21:45:01.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:01 smithi033 ceph-mon[27845]: pgmap v19262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:01 smithi096 ceph-mon[32753]: pgmap v19262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:01 smithi194 ceph-mon[33017]: pgmap v19262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:03 smithi033 ceph-mon[27845]: pgmap v19263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35642.246618 secs 2024-04-26T21:45:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:03 smithi096 ceph-mon[32753]: pgmap v19263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35642.246618 secs 2024-04-26T21:45:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:03 smithi194 ceph-mon[33017]: pgmap v19263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35642.246618 secs 2024-04-26T21:45:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35645.209409 secs 2024-04-26T21:45:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35645.209409 secs 2024-04-26T21:45:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35645.209409 secs 2024-04-26T21:45:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:05 smithi033 ceph-mon[27845]: pgmap v19264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:05 smithi096 ceph-mon[32753]: pgmap v19264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:05 smithi194 ceph-mon[33017]: pgmap v19264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:06 smithi033 ceph-mon[27845]: pgmap v19265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:06 smithi096 ceph-mon[32753]: pgmap v19265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:06 smithi194 ceph-mon[33017]: pgmap v19265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:45:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35647.246677 secs 2024-04-26T21:45:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35647.246677 secs 2024-04-26T21:45:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35647.246677 secs 2024-04-26T21:45:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:09 smithi033 ceph-mon[27845]: pgmap v19266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:45:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35650.209554 secs 2024-04-26T21:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:09 smithi096 ceph-mon[32753]: pgmap v19266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:45:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35650.209554 secs 2024-04-26T21:45:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:09 smithi194 ceph-mon[33017]: pgmap v19266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:45:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35650.209554 secs 2024-04-26T21:45:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:11 smithi033 ceph-mon[27845]: pgmap v19267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:11 smithi096 ceph-mon[32753]: pgmap v19267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:11 smithi194 ceph-mon[33017]: pgmap v19267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:13 smithi033 ceph-mon[27845]: pgmap v19268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35652.246739 secs 2024-04-26T21:45:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:13 smithi096 ceph-mon[32753]: pgmap v19268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35652.246739 secs 2024-04-26T21:45:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:13.994+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:13.994+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:13.994+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:45:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:13 smithi194 ceph-mon[33017]: pgmap v19268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35652.246739 secs 2024-04-26T21:45:14.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35655.209686 secs 2024-04-26T21:45:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35655.209686 secs 2024-04-26T21:45:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35655.209686 secs 2024-04-26T21:45:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:15 smithi033 ceph-mon[27845]: pgmap v19269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:15 smithi096 ceph-mon[32753]: pgmap v19269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:15 smithi194 ceph-mon[33017]: pgmap v19269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:16 smithi033 ceph-mon[27845]: pgmap v19270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:16 smithi096 ceph-mon[32753]: pgmap v19270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:16 smithi194 ceph-mon[33017]: pgmap v19270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35657.246798 secs 2024-04-26T21:45:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35657.246798 secs 2024-04-26T21:45:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35657.246798 secs 2024-04-26T21:45:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:19 smithi033 ceph-mon[27845]: pgmap v19271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35660.209821 secs 2024-04-26T21:45:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:19 smithi096 ceph-mon[32753]: pgmap v19271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35660.209821 secs 2024-04-26T21:45:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:19 smithi194 ceph-mon[33017]: pgmap v19271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35660.209821 secs 2024-04-26T21:45:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:21 smithi033 ceph-mon[27845]: pgmap v19272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:21 smithi096 ceph-mon[32753]: pgmap v19272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:21 smithi194 ceph-mon[33017]: pgmap v19272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:23 smithi033 ceph-mon[27845]: pgmap v19273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35662.246856 secs 2024-04-26T21:45:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:23 smithi096 ceph-mon[32753]: pgmap v19273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35662.246856 secs 2024-04-26T21:45:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:23 smithi194 ceph-mon[33017]: pgmap v19273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:24.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35662.246856 secs 2024-04-26T21:45:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35665.209945 secs 2024-04-26T21:45:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:24 smithi033 ceph-mon[27845]: pgmap v19274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35665.209945 secs 2024-04-26T21:45:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:24 smithi096 ceph-mon[32753]: pgmap v19274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:25.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35665.209945 secs 2024-04-26T21:45:25.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:24 smithi194 ceph-mon[33017]: pgmap v19274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:27 smithi033 ceph-mon[27845]: pgmap v19275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:27 smithi096 ceph-mon[32753]: pgmap v19275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:27 smithi194 ceph-mon[33017]: pgmap v19275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35667.246918 secs 2024-04-26T21:45:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:28 smithi033 ceph-mon[27845]: pgmap v19276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35667.246918 secs 2024-04-26T21:45:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:28 smithi096 ceph-mon[32753]: pgmap v19276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:28.998+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:28.998+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:28.999 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:28.998+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:45:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35667.246918 secs 2024-04-26T21:45:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:28 smithi194 ceph-mon[33017]: pgmap v19276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35670.210076 secs 2024-04-26T21:45:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35670.210076 secs 2024-04-26T21:45:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35670.210076 secs 2024-04-26T21:45:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:30 smithi096 ceph-mon[32753]: pgmap v19277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:30 smithi194 ceph-mon[33017]: pgmap v19277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:30 smithi033 ceph-mon[27845]: pgmap v19277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:33 smithi033 ceph-mon[27845]: pgmap v19278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:33.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35672.246978 secs 2024-04-26T21:45:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:33 smithi096 ceph-mon[32753]: pgmap v19278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35672.246978 secs 2024-04-26T21:45:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:33 smithi194 ceph-mon[33017]: pgmap v19278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:34.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35672.246978 secs 2024-04-26T21:45:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35675.210207 secs 2024-04-26T21:45:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35675.210207 secs 2024-04-26T21:45:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35675.210207 secs 2024-04-26T21:45:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:35 smithi033 ceph-mon[27845]: pgmap v19279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:35 smithi096 ceph-mon[32753]: pgmap v19279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:35 smithi194 ceph-mon[33017]: pgmap v19279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:36 smithi096 ceph-mon[32753]: pgmap v19280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:36 smithi194 ceph-mon[33017]: pgmap v19280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:36 smithi033 ceph-mon[27845]: pgmap v19280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35677.247037 secs 2024-04-26T21:45:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35677.247037 secs 2024-04-26T21:45:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35677.247037 secs 2024-04-26T21:45:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:39 smithi033 ceph-mon[27845]: pgmap v19281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35680.210332 secs 2024-04-26T21:45:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:39 smithi096 ceph-mon[32753]: pgmap v19281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35680.210332 secs 2024-04-26T21:45:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:39 smithi194 ceph-mon[33017]: pgmap v19281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:39.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35680.210332 secs 2024-04-26T21:45:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:41 smithi033 ceph-mon[27845]: pgmap v19282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:41 smithi096 ceph-mon[32753]: pgmap v19282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:41 smithi194 ceph-mon[33017]: pgmap v19282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:42 smithi033 ceph-mon[27845]: pgmap v19283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:42 smithi096 ceph-mon[32753]: pgmap v19283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:42 smithi194 ceph-mon[33017]: pgmap v19283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35682.247097 secs 2024-04-26T21:45:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35682.247097 secs 2024-04-26T21:45:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:43.999+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:43.999+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:43.999+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:45:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35682.247097 secs 2024-04-26T21:45:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35685.210468 secs 2024-04-26T21:45:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:44 smithi096 ceph-mon[32753]: pgmap v19284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35685.210468 secs 2024-04-26T21:45:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:44 smithi194 ceph-mon[33017]: pgmap v19284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35685.210468 secs 2024-04-26T21:45:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:44 smithi033 ceph-mon[27845]: pgmap v19284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:47 smithi033 ceph-mon[27845]: pgmap v19285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:47 smithi096 ceph-mon[32753]: pgmap v19285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:47 smithi194 ceph-mon[33017]: pgmap v19285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35687.247157 secs 2024-04-26T21:45:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:48 smithi033 ceph-mon[27845]: pgmap v19286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35687.247157 secs 2024-04-26T21:45:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:48 smithi096 ceph-mon[32753]: pgmap v19286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35687.247157 secs 2024-04-26T21:45:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:48 smithi194 ceph-mon[33017]: pgmap v19286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35690.210598 secs 2024-04-26T21:45:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35690.210598 secs 2024-04-26T21:45:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35690.210598 secs 2024-04-26T21:45:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:50 smithi096 ceph-mon[32753]: pgmap v19287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:50 smithi194 ceph-mon[33017]: pgmap v19287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:50 smithi033 ceph-mon[27845]: pgmap v19287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:53 smithi033 ceph-mon[27845]: pgmap v19288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35692.247217 secs 2024-04-26T21:45:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:53 smithi096 ceph-mon[32753]: pgmap v19288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35692.247217 secs 2024-04-26T21:45:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:53 smithi194 ceph-mon[33017]: pgmap v19288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35692.247217 secs 2024-04-26T21:45:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35695.210726 secs 2024-04-26T21:45:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:54 smithi033 ceph-mon[27845]: pgmap v19289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35695.210726 secs 2024-04-26T21:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:54 smithi096 ceph-mon[32753]: pgmap v19289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:55.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35695.210726 secs 2024-04-26T21:45:55.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:54 smithi194 ceph-mon[33017]: pgmap v19289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:56 smithi194 ceph-mon[33017]: pgmap v19290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:56 smithi033 ceph-mon[27845]: pgmap v19290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:56 smithi096 ceph-mon[32753]: pgmap v19290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35697.247277 secs 2024-04-26T21:45:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35697.247277 secs 2024-04-26T21:45:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35697.247277 secs 2024-04-26T21:45:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:59.014+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:59.014+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:45:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:45:59.014+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:59 smithi033 ceph-mon[27845]: pgmap v19291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35700.210848 secs 2024-04-26T21:45:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:45:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:59 smithi096 ceph-mon[32753]: pgmap v19291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35700.210848 secs 2024-04-26T21:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:45:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:59 smithi194 ceph-mon[33017]: pgmap v19291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:45:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35700.210848 secs 2024-04-26T21:45:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:45:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:46:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:46:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:46:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:46:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:46:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:46:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:46:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:46:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:01 smithi033 ceph-mon[27845]: pgmap v19292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:46:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:01 smithi096 ceph-mon[32753]: pgmap v19292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:46:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:01 smithi194 ceph-mon[33017]: pgmap v19292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:46:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:03 smithi033 ceph-mon[27845]: pgmap v19293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35702.247336 secs 2024-04-26T21:46:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:03 smithi096 ceph-mon[32753]: pgmap v19293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35702.247336 secs 2024-04-26T21:46:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:03 smithi194 ceph-mon[33017]: pgmap v19293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35702.247336 secs 2024-04-26T21:46:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35705.210964 secs 2024-04-26T21:46:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35705.210964 secs 2024-04-26T21:46:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35705.210964 secs 2024-04-26T21:46:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:05 smithi033 ceph-mon[27845]: pgmap v19294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:05 smithi096 ceph-mon[32753]: pgmap v19294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:05 smithi194 ceph-mon[33017]: pgmap v19294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:07 smithi033 ceph-mon[27845]: pgmap v19295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:07 smithi096 ceph-mon[32753]: pgmap v19295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:07 smithi194 ceph-mon[33017]: pgmap v19295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35707.247396 secs 2024-04-26T21:46:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35707.247396 secs 2024-04-26T21:46:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35707.247396 secs 2024-04-26T21:46:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:09 smithi033 ceph-mon[27845]: pgmap v19296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35710.211130 secs 2024-04-26T21:46:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:09 smithi096 ceph-mon[32753]: pgmap v19296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35710.211130 secs 2024-04-26T21:46:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:09 smithi194 ceph-mon[33017]: pgmap v19296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35710.211130 secs 2024-04-26T21:46:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:11 smithi033 ceph-mon[27845]: pgmap v19297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:46:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:11 smithi096 ceph-mon[32753]: pgmap v19297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:46:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:11 smithi194 ceph-mon[33017]: pgmap v19297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:46:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:13 smithi096 ceph-mon[32753]: pgmap v19298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35712.247455 secs 2024-04-26T21:46:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:13 smithi194 ceph-mon[33017]: pgmap v19298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35712.247455 secs 2024-04-26T21:46:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:13 smithi033 ceph-mon[27845]: pgmap v19298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35712.247455 secs 2024-04-26T21:46:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:14.025+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:14.026+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:14.026+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:46:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35715.211216 secs 2024-04-26T21:46:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35715.211216 secs 2024-04-26T21:46:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35715.211216 secs 2024-04-26T21:46:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:15 smithi194 ceph-mon[33017]: pgmap v19299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:15 smithi033 ceph-mon[27845]: pgmap v19299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:15 smithi096 ceph-mon[32753]: pgmap v19299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:17 smithi194 ceph-mon[33017]: pgmap v19300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:17 smithi033 ceph-mon[27845]: pgmap v19300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:17 smithi096 ceph-mon[32753]: pgmap v19300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35717.247514 secs 2024-04-26T21:46:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35717.247514 secs 2024-04-26T21:46:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35717.247514 secs 2024-04-26T21:46:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:19 smithi033 ceph-mon[27845]: pgmap v19301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35720.211332 secs 2024-04-26T21:46:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:19 smithi194 ceph-mon[33017]: pgmap v19301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35720.211332 secs 2024-04-26T21:46:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:19 smithi096 ceph-mon[32753]: pgmap v19301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35720.211332 secs 2024-04-26T21:46:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:21 smithi194 ceph-mon[33017]: pgmap v19302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:21 smithi033 ceph-mon[27845]: pgmap v19302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:21 smithi096 ceph-mon[32753]: pgmap v19302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:23 smithi194 ceph-mon[33017]: pgmap v19303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35722.247641 secs 2024-04-26T21:46:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:23 smithi033 ceph-mon[27845]: pgmap v19303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35722.247641 secs 2024-04-26T21:46:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:23 smithi096 ceph-mon[32753]: pgmap v19303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35722.247641 secs 2024-04-26T21:46:24.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35725.211462 secs 2024-04-26T21:46:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35725.211462 secs 2024-04-26T21:46:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35725.211462 secs 2024-04-26T21:46:25.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:25 smithi194 ceph-mon[33017]: pgmap v19304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:25.844 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:25 smithi096 ceph-mon[32753]: pgmap v19304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:25 smithi033 ceph-mon[27845]: pgmap v19304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:27 smithi194 ceph-mon[33017]: pgmap v19305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:27 smithi033 ceph-mon[27845]: pgmap v19305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:27 smithi096 ceph-mon[32753]: pgmap v19305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35727.247727 secs 2024-04-26T21:46:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35727.247727 secs 2024-04-26T21:46:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35727.247727 secs 2024-04-26T21:46:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:29.035+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:29.035+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:29.035+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:46:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:29 smithi194 ceph-mon[33017]: pgmap v19306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35730.211545 secs 2024-04-26T21:46:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:29 smithi033 ceph-mon[27845]: pgmap v19306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35730.211545 secs 2024-04-26T21:46:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:29 smithi096 ceph-mon[32753]: pgmap v19306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35730.211545 secs 2024-04-26T21:46:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:31 smithi194 ceph-mon[33017]: pgmap v19307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:31 smithi033 ceph-mon[27845]: pgmap v19307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:31 smithi096 ceph-mon[32753]: pgmap v19307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:33 smithi194 ceph-mon[33017]: pgmap v19308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35732.247810 secs 2024-04-26T21:46:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:33 smithi033 ceph-mon[27845]: pgmap v19308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35732.247810 secs 2024-04-26T21:46:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:33 smithi096 ceph-mon[32753]: pgmap v19308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35732.247810 secs 2024-04-26T21:46:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35735.211683 secs 2024-04-26T21:46:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35735.211683 secs 2024-04-26T21:46:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35735.211683 secs 2024-04-26T21:46:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:35 smithi194 ceph-mon[33017]: pgmap v19309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:35 smithi033 ceph-mon[27845]: pgmap v19309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:35 smithi096 ceph-mon[32753]: pgmap v19309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:37 smithi194 ceph-mon[33017]: pgmap v19310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:37 smithi033 ceph-mon[27845]: pgmap v19310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:37 smithi096 ceph-mon[32753]: pgmap v19310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35737.247897 secs 2024-04-26T21:46:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35737.247897 secs 2024-04-26T21:46:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35737.247897 secs 2024-04-26T21:46:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:39 smithi194 ceph-mon[33017]: pgmap v19311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35740.211834 secs 2024-04-26T21:46:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:39 smithi033 ceph-mon[27845]: pgmap v19311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35740.211834 secs 2024-04-26T21:46:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:39 smithi096 ceph-mon[32753]: pgmap v19311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35740.211834 secs 2024-04-26T21:46:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:41 smithi194 ceph-mon[33017]: pgmap v19312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:41 smithi033 ceph-mon[27845]: pgmap v19312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:41 smithi096 ceph-mon[32753]: pgmap v19312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:43 smithi194 ceph-mon[33017]: pgmap v19313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35742.247997 secs 2024-04-26T21:46:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:43 smithi033 ceph-mon[27845]: pgmap v19313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35742.247997 secs 2024-04-26T21:46:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:43 smithi096 ceph-mon[32753]: pgmap v19313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35742.247997 secs 2024-04-26T21:46:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:44.050+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:44.050+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:44.050+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:46:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35745.211975 secs 2024-04-26T21:46:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35745.211975 secs 2024-04-26T21:46:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35745.211975 secs 2024-04-26T21:46:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:45 smithi194 ceph-mon[33017]: pgmap v19314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:45 smithi033 ceph-mon[27845]: pgmap v19314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:45 smithi096 ceph-mon[32753]: pgmap v19314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:47 smithi194 ceph-mon[33017]: pgmap v19315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:47 smithi033 ceph-mon[27845]: pgmap v19315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:47 smithi096 ceph-mon[32753]: pgmap v19315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35747.248082 secs 2024-04-26T21:46:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35747.248082 secs 2024-04-26T21:46:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35747.248082 secs 2024-04-26T21:46:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:49 smithi194 ceph-mon[33017]: pgmap v19316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35750.212149 secs 2024-04-26T21:46:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:49 smithi033 ceph-mon[27845]: pgmap v19316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35750.212149 secs 2024-04-26T21:46:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:49 smithi096 ceph-mon[32753]: pgmap v19316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35750.212149 secs 2024-04-26T21:46:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:51 smithi194 ceph-mon[33017]: pgmap v19317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:51 smithi033 ceph-mon[27845]: pgmap v19317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:51 smithi096 ceph-mon[32753]: pgmap v19317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:53 smithi194 ceph-mon[33017]: pgmap v19318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35752.248170 secs 2024-04-26T21:46:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:53 smithi033 ceph-mon[27845]: pgmap v19318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35752.248170 secs 2024-04-26T21:46:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:53 smithi096 ceph-mon[32753]: pgmap v19318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35752.248170 secs 2024-04-26T21:46:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35755.212227 secs 2024-04-26T21:46:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35755.212227 secs 2024-04-26T21:46:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35755.212227 secs 2024-04-26T21:46:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:55 smithi033 ceph-mon[27845]: pgmap v19319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:55 smithi096 ceph-mon[32753]: pgmap v19319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:56.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:55 smithi194 ceph-mon[33017]: pgmap v19319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:46:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:57 smithi033 ceph-mon[27845]: pgmap v19320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:57 smithi096 ceph-mon[32753]: pgmap v19320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:57 smithi194 ceph-mon[33017]: pgmap v19320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:46:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35757.248279 secs 2024-04-26T21:46:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35757.248279 secs 2024-04-26T21:46:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:59.054+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:59.054+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:46:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:46:59.054+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:46:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35757.248279 secs 2024-04-26T21:46:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:59 smithi033 ceph-mon[27845]: pgmap v19321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:59.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:46:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35760.212350 secs 2024-04-26T21:46:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:59 smithi096 ceph-mon[32753]: pgmap v19321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:46:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:46:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35760.212350 secs 2024-04-26T21:47:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:59 smithi194 ceph-mon[33017]: pgmap v19321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:46:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35760.212350 secs 2024-04-26T21:47:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:47:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:47:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:47:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:01 smithi033 ceph-mon[27845]: pgmap v19322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:01 smithi096 ceph-mon[32753]: pgmap v19322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:01 smithi194 ceph-mon[33017]: pgmap v19322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:02.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:47:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:47:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:03.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:47:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:03 smithi033 ceph-mon[27845]: pgmap v19323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:03 smithi033 ceph-mon[27845]: pgmap v19324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35762.248365 secs 2024-04-26T21:47:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:03 smithi096 ceph-mon[32753]: pgmap v19323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:03 smithi096 ceph-mon[32753]: pgmap v19324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35762.248365 secs 2024-04-26T21:47:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:03 smithi194 ceph-mon[33017]: pgmap v19323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:03 smithi194 ceph-mon[33017]: pgmap v19324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35762.248365 secs 2024-04-26T21:47:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35765.212466 secs 2024-04-26T21:47:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35765.212466 secs 2024-04-26T21:47:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35765.212466 secs 2024-04-26T21:47:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:05 smithi033 ceph-mon[27845]: pgmap v19325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:47:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:05 smithi096 ceph-mon[32753]: pgmap v19325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:47:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:05 smithi194 ceph-mon[33017]: pgmap v19325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T21:47:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:07 smithi033 ceph-mon[27845]: pgmap v19326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:07 smithi096 ceph-mon[32753]: pgmap v19326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:07 smithi194 ceph-mon[33017]: pgmap v19326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35767.248450 secs 2024-04-26T21:47:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35767.248450 secs 2024-04-26T21:47:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35767.248450 secs 2024-04-26T21:47:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:09 smithi194 ceph-mon[33017]: pgmap v19327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35770.212578 secs 2024-04-26T21:47:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:09 smithi033 ceph-mon[27845]: pgmap v19327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35770.212578 secs 2024-04-26T21:47:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:09 smithi096 ceph-mon[32753]: pgmap v19327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35770.212578 secs 2024-04-26T21:47:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:10 smithi194 ceph-mon[33017]: pgmap v19328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:10 smithi033 ceph-mon[27845]: pgmap v19328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:10 smithi096 ceph-mon[32753]: pgmap v19328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:47:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:13 smithi194 ceph-mon[33017]: pgmap v19329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:47:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:13 smithi033 ceph-mon[27845]: pgmap v19329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:47:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:13 smithi096 ceph-mon[32753]: pgmap v19329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:47:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:14.070+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:14.070+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:47:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:14.070+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35772.248535 secs 2024-04-26T21:47:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35772.248535 secs 2024-04-26T21:47:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35772.248535 secs 2024-04-26T21:47:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35775.212693 secs 2024-04-26T21:47:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:15 smithi194 ceph-mon[33017]: pgmap v19330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35775.212693 secs 2024-04-26T21:47:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:15 smithi033 ceph-mon[27845]: pgmap v19330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35775.212693 secs 2024-04-26T21:47:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:15 smithi096 ceph-mon[32753]: pgmap v19330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:17 smithi194 ceph-mon[33017]: pgmap v19331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:17 smithi033 ceph-mon[27845]: pgmap v19331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:17 smithi096 ceph-mon[32753]: pgmap v19331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:19 smithi194 ceph-mon[33017]: pgmap v19332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35777.248621 secs 2024-04-26T21:47:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:19 smithi033 ceph-mon[27845]: pgmap v19332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35777.248621 secs 2024-04-26T21:47:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:19 smithi096 ceph-mon[32753]: pgmap v19332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35777.248621 secs 2024-04-26T21:47:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35780.212810 secs 2024-04-26T21:47:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35780.212810 secs 2024-04-26T21:47:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35780.212810 secs 2024-04-26T21:47:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:21 smithi194 ceph-mon[33017]: pgmap v19333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:21 smithi033 ceph-mon[27845]: pgmap v19333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:21 smithi096 ceph-mon[32753]: pgmap v19333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:23 smithi194 ceph-mon[33017]: pgmap v19334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:23 smithi033 ceph-mon[27845]: pgmap v19334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:23 smithi096 ceph-mon[32753]: pgmap v19334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35782.248707 secs 2024-04-26T21:47:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35782.248707 secs 2024-04-26T21:47:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35782.248707 secs 2024-04-26T21:47:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35785.212964 secs 2024-04-26T21:47:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:25 smithi194 ceph-mon[33017]: pgmap v19335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35785.212964 secs 2024-04-26T21:47:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:25 smithi033 ceph-mon[27845]: pgmap v19335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35785.212964 secs 2024-04-26T21:47:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:25 smithi096 ceph-mon[32753]: pgmap v19335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:27 smithi033 ceph-mon[27845]: pgmap v19336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:27 smithi096 ceph-mon[32753]: pgmap v19336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:27 smithi194 ceph-mon[33017]: pgmap v19336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:27.890+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:47:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:27.890+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:47:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:27.891+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 280 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T21:47:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:27.891+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 280 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x7fef00077780 2024-04-26T21:47:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:29.078+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:29.078+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:29.078+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:47:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:29 smithi033 ceph-mon[27845]: pgmap v19337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35787.248792 secs 2024-04-26T21:47:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:29 smithi096 ceph-mon[32753]: pgmap v19337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35787.248792 secs 2024-04-26T21:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:29 smithi194 ceph-mon[33017]: pgmap v19337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35787.248792 secs 2024-04-26T21:47:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35790.213130 secs 2024-04-26T21:47:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35790.213130 secs 2024-04-26T21:47:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35790.213130 secs 2024-04-26T21:47:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:31 smithi033 ceph-mon[27845]: pgmap v19338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:31 smithi096 ceph-mon[32753]: pgmap v19338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:31 smithi194 ceph-mon[33017]: pgmap v19338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:33 smithi033 ceph-mon[27845]: pgmap v19339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:33 smithi096 ceph-mon[32753]: pgmap v19339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:33 smithi194 ceph-mon[33017]: pgmap v19339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35792.248874 secs 2024-04-26T21:47:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35792.248874 secs 2024-04-26T21:47:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35792.248874 secs 2024-04-26T21:47:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35795.213211 secs 2024-04-26T21:47:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:35 smithi033 ceph-mon[27845]: pgmap v19340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35795.213211 secs 2024-04-26T21:47:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:35 smithi096 ceph-mon[32753]: pgmap v19340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35795.213211 secs 2024-04-26T21:47:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:35 smithi194 ceph-mon[33017]: pgmap v19340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:37 smithi033 ceph-mon[27845]: pgmap v19341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:37 smithi096 ceph-mon[32753]: pgmap v19341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:37 smithi194 ceph-mon[33017]: pgmap v19341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:39 smithi033 ceph-mon[27845]: pgmap v19342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35797.248957 secs 2024-04-26T21:47:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:39 smithi096 ceph-mon[32753]: pgmap v19342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35797.248957 secs 2024-04-26T21:47:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:39 smithi194 ceph-mon[33017]: pgmap v19342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35797.248957 secs 2024-04-26T21:47:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35800.213326 secs 2024-04-26T21:47:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35800.213326 secs 2024-04-26T21:47:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35800.213326 secs 2024-04-26T21:47:41.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:41 smithi033 ceph-mon[27845]: pgmap v19343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:41 smithi096 ceph-mon[32753]: pgmap v19343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:41 smithi194 ceph-mon[33017]: pgmap v19343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:43 smithi033 ceph-mon[27845]: pgmap v19344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:43 smithi096 ceph-mon[32753]: pgmap v19344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:43 smithi194 ceph-mon[33017]: pgmap v19344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:44.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:44.091+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:44.091+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:47:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:44.091+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35802.248979 secs 2024-04-26T21:47:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35802.248979 secs 2024-04-26T21:47:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35802.248979 secs 2024-04-26T21:47:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35805.213442 secs 2024-04-26T21:47:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:45 smithi194 ceph-mon[33017]: pgmap v19345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35805.213442 secs 2024-04-26T21:47:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:45 smithi033 ceph-mon[27845]: pgmap v19345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35805.213442 secs 2024-04-26T21:47:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:45 smithi096 ceph-mon[32753]: pgmap v19345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:47 smithi194 ceph-mon[33017]: pgmap v19346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:47 smithi033 ceph-mon[27845]: pgmap v19346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:47 smithi096 ceph-mon[32753]: pgmap v19346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:49 smithi194 ceph-mon[33017]: pgmap v19347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35807.249066 secs 2024-04-26T21:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:49 smithi033 ceph-mon[27845]: pgmap v19347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35807.249066 secs 2024-04-26T21:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:49 smithi096 ceph-mon[32753]: pgmap v19347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35807.249066 secs 2024-04-26T21:47:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35810.213556 secs 2024-04-26T21:47:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35810.213556 secs 2024-04-26T21:47:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35810.213556 secs 2024-04-26T21:47:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:51 smithi194 ceph-mon[33017]: pgmap v19348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:51 smithi033 ceph-mon[27845]: pgmap v19348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:51 smithi096 ceph-mon[32753]: pgmap v19348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:53 smithi033 ceph-mon[27845]: pgmap v19349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:53 smithi194 ceph-mon[33017]: pgmap v19349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:53 smithi096 ceph-mon[32753]: pgmap v19349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:54.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35812.249152 secs 2024-04-26T21:47:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35812.249152 secs 2024-04-26T21:47:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35812.249152 secs 2024-04-26T21:47:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35815.213679 secs 2024-04-26T21:47:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:55 smithi194 ceph-mon[33017]: pgmap v19350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35815.213679 secs 2024-04-26T21:47:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:55 smithi033 ceph-mon[27845]: pgmap v19350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35815.213679 secs 2024-04-26T21:47:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:55 smithi096 ceph-mon[32753]: pgmap v19350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:57 smithi194 ceph-mon[33017]: pgmap v19351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:57 smithi033 ceph-mon[27845]: pgmap v19351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:57 smithi096 ceph-mon[32753]: pgmap v19351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:59.095+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:59.095+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:47:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:47:59.095+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:47:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35817.249239 secs 2024-04-26T21:47:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:47:59 smithi194 ceph-mon[33017]: pgmap v19352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35817.249239 secs 2024-04-26T21:47:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:47:59 smithi033 ceph-mon[27845]: pgmap v19352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:47:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35817.249239 secs 2024-04-26T21:47:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:47:59 smithi096 ceph-mon[32753]: pgmap v19352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35820.213813 secs 2024-04-26T21:48:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35820.213813 secs 2024-04-26T21:48:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35820.213813 secs 2024-04-26T21:48:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:01 smithi194 ceph-mon[33017]: pgmap v19353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:01 smithi033 ceph-mon[27845]: pgmap v19353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:01 smithi096 ceph-mon[32753]: pgmap v19353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:03 smithi194 ceph-mon[33017]: pgmap v19354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:03 smithi033 ceph-mon[27845]: pgmap v19354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:03 smithi096 ceph-mon[32753]: pgmap v19354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35822.249315 secs 2024-04-26T21:48:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:04 smithi194 ceph-mon[33017]: pgmap v19355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:04 smithi194 ceph-mon[33017]: pgmap v19356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:48:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35822.249315 secs 2024-04-26T21:48:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:04 smithi033 ceph-mon[27845]: pgmap v19355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:04 smithi033 ceph-mon[27845]: pgmap v19356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:48:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35822.249315 secs 2024-04-26T21:48:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:04 smithi096 ceph-mon[32753]: pgmap v19355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:04 smithi096 ceph-mon[32753]: pgmap v19356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:48:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35825.213937 secs 2024-04-26T21:48:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35825.213937 secs 2024-04-26T21:48:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35825.213937 secs 2024-04-26T21:48:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:06 smithi194 ceph-mon[33017]: pgmap v19357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:06 smithi033 ceph-mon[27845]: pgmap v19357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:06 smithi096 ceph-mon[32753]: pgmap v19357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:07 smithi194 ceph-mon[33017]: pgmap v19358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:07 smithi033 ceph-mon[27845]: pgmap v19358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:07 smithi096 ceph-mon[32753]: pgmap v19358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35827.249401 secs 2024-04-26T21:48:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35827.249401 secs 2024-04-26T21:48:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35827.249401 secs 2024-04-26T21:48:10.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35830.214075 secs 2024-04-26T21:48:10.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:09 smithi194 ceph-mon[33017]: pgmap v19359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35830.214075 secs 2024-04-26T21:48:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:09 smithi033 ceph-mon[27845]: pgmap v19359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35830.214075 secs 2024-04-26T21:48:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:09 smithi096 ceph-mon[32753]: pgmap v19359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:11 smithi194 ceph-mon[33017]: pgmap v19360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:11 smithi033 ceph-mon[27845]: pgmap v19360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:11 smithi096 ceph-mon[32753]: pgmap v19360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:48:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35832.249448 secs 2024-04-26T21:48:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35832.249448 secs 2024-04-26T21:48:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35832.249448 secs 2024-04-26T21:48:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:14.104+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:14.104+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:14.104+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:48:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:14 smithi033 ceph-mon[27845]: pgmap v19361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:48:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35835.214173 secs 2024-04-26T21:48:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:14 smithi096 ceph-mon[32753]: pgmap v19361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:48:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35835.214173 secs 2024-04-26T21:48:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:14 smithi194 ceph-mon[33017]: pgmap v19361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:48:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35835.214173 secs 2024-04-26T21:48:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:15 smithi194 ceph-mon[33017]: pgmap v19362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:15 smithi033 ceph-mon[27845]: pgmap v19362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:15 smithi096 ceph-mon[32753]: pgmap v19362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:17 smithi194 ceph-mon[33017]: pgmap v19363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:17 smithi033 ceph-mon[27845]: pgmap v19363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:17 smithi096 ceph-mon[32753]: pgmap v19363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35837.249508 secs 2024-04-26T21:48:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35837.249508 secs 2024-04-26T21:48:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35837.249508 secs 2024-04-26T21:48:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35840.214303 secs 2024-04-26T21:48:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:19 smithi194 ceph-mon[33017]: pgmap v19364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35840.214303 secs 2024-04-26T21:48:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:19 smithi033 ceph-mon[27845]: pgmap v19364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35840.214303 secs 2024-04-26T21:48:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:19 smithi096 ceph-mon[32753]: pgmap v19364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:21 smithi194 ceph-mon[33017]: pgmap v19365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:21 smithi033 ceph-mon[27845]: pgmap v19365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:21 smithi096 ceph-mon[32753]: pgmap v19365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35842.249566 secs 2024-04-26T21:48:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35842.249566 secs 2024-04-26T21:48:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35842.249566 secs 2024-04-26T21:48:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:24 smithi033 ceph-mon[27845]: pgmap v19366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35845.214427 secs 2024-04-26T21:48:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:24 smithi096 ceph-mon[32753]: pgmap v19366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35845.214427 secs 2024-04-26T21:48:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:24 smithi194 ceph-mon[33017]: pgmap v19366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35845.214427 secs 2024-04-26T21:48:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:25 smithi194 ceph-mon[33017]: pgmap v19367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:25 smithi033 ceph-mon[27845]: pgmap v19367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:25 smithi096 ceph-mon[32753]: pgmap v19367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:27 smithi194 ceph-mon[33017]: pgmap v19368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:27 smithi033 ceph-mon[27845]: pgmap v19368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:27 smithi096 ceph-mon[32753]: pgmap v19368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35847.249620 secs 2024-04-26T21:48:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:29.120+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:29.120+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:29.120+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:48:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35847.249620 secs 2024-04-26T21:48:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35847.249620 secs 2024-04-26T21:48:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35850.214514 secs 2024-04-26T21:48:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:29 smithi194 ceph-mon[33017]: pgmap v19369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35850.214514 secs 2024-04-26T21:48:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:29 smithi033 ceph-mon[27845]: pgmap v19369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35850.214514 secs 2024-04-26T21:48:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:29 smithi096 ceph-mon[32753]: pgmap v19369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:31 smithi194 ceph-mon[33017]: pgmap v19370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:31 smithi033 ceph-mon[27845]: pgmap v19370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:31 smithi096 ceph-mon[32753]: pgmap v19370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35852.249702 secs 2024-04-26T21:48:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35852.249702 secs 2024-04-26T21:48:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35852.249702 secs 2024-04-26T21:48:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:34 smithi033 ceph-mon[27845]: pgmap v19371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35855.214641 secs 2024-04-26T21:48:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:34 smithi096 ceph-mon[32753]: pgmap v19371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35855.214641 secs 2024-04-26T21:48:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:34 smithi194 ceph-mon[33017]: pgmap v19371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35855.214641 secs 2024-04-26T21:48:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:35 smithi194 ceph-mon[33017]: pgmap v19372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:35 smithi033 ceph-mon[27845]: pgmap v19372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:35 smithi096 ceph-mon[32753]: pgmap v19372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:37 smithi194 ceph-mon[33017]: pgmap v19373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:37 smithi033 ceph-mon[27845]: pgmap v19373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:37 smithi096 ceph-mon[32753]: pgmap v19373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:39.087 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35857.249786 secs 2024-04-26T21:48:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35857.249786 secs 2024-04-26T21:48:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35857.249786 secs 2024-04-26T21:48:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35860.214725 secs 2024-04-26T21:48:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:39 smithi194 ceph-mon[33017]: pgmap v19374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35860.214725 secs 2024-04-26T21:48:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:39 smithi033 ceph-mon[27845]: pgmap v19374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35860.214725 secs 2024-04-26T21:48:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:39 smithi096 ceph-mon[32753]: pgmap v19374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:41 smithi194 ceph-mon[33017]: pgmap v19375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:41 smithi033 ceph-mon[27845]: pgmap v19375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:41 smithi096 ceph-mon[32753]: pgmap v19375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35862.249871 secs 2024-04-26T21:48:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35862.249871 secs 2024-04-26T21:48:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35862.249871 secs 2024-04-26T21:48:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:44.134+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:44.134+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:44.134+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:48:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:44 smithi033 ceph-mon[27845]: pgmap v19376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35865.214848 secs 2024-04-26T21:48:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:44 smithi096 ceph-mon[32753]: pgmap v19376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35865.214848 secs 2024-04-26T21:48:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:44 smithi194 ceph-mon[33017]: pgmap v19376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35865.214848 secs 2024-04-26T21:48:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:45 smithi194 ceph-mon[33017]: pgmap v19377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:45 smithi033 ceph-mon[27845]: pgmap v19377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:45 smithi096 ceph-mon[32753]: pgmap v19377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:47 smithi194 ceph-mon[33017]: pgmap v19378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:47 smithi033 ceph-mon[27845]: pgmap v19378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:47 smithi096 ceph-mon[32753]: pgmap v19378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35867.249935 secs 2024-04-26T21:48:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35867.249935 secs 2024-04-26T21:48:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35867.249935 secs 2024-04-26T21:48:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35870.214975 secs 2024-04-26T21:48:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:49 smithi194 ceph-mon[33017]: pgmap v19379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35870.214975 secs 2024-04-26T21:48:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:49 smithi033 ceph-mon[27845]: pgmap v19379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35870.214975 secs 2024-04-26T21:48:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:49 smithi096 ceph-mon[32753]: pgmap v19379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:51 smithi194 ceph-mon[33017]: pgmap v19380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:51 smithi033 ceph-mon[27845]: pgmap v19380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:51 smithi096 ceph-mon[32753]: pgmap v19380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35872.250026 secs 2024-04-26T21:48:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35872.250026 secs 2024-04-26T21:48:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35872.250026 secs 2024-04-26T21:48:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:54 smithi033 ceph-mon[27845]: pgmap v19381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35875.215098 secs 2024-04-26T21:48:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:54 smithi096 ceph-mon[32753]: pgmap v19381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35875.215098 secs 2024-04-26T21:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:54 smithi194 ceph-mon[33017]: pgmap v19381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35875.215098 secs 2024-04-26T21:48:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:55 smithi194 ceph-mon[33017]: pgmap v19382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:55 smithi033 ceph-mon[27845]: pgmap v19382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:55 smithi096 ceph-mon[32753]: pgmap v19382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:57 smithi194 ceph-mon[33017]: pgmap v19383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:57 smithi033 ceph-mon[27845]: pgmap v19383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:57 smithi096 ceph-mon[32753]: pgmap v19383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:48:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35877.250113 secs 2024-04-26T21:48:59.143 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:59.143+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:59.143+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:48:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:48:59.143+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:48:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35877.250113 secs 2024-04-26T21:48:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35877.250113 secs 2024-04-26T21:49:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35880.215224 secs 2024-04-26T21:49:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:48:59 smithi194 ceph-mon[33017]: pgmap v19384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35880.215224 secs 2024-04-26T21:49:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:48:59 smithi033 ceph-mon[27845]: pgmap v19384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35880.215224 secs 2024-04-26T21:49:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:48:59 smithi096 ceph-mon[32753]: pgmap v19384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:01 smithi194 ceph-mon[33017]: pgmap v19385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:01 smithi033 ceph-mon[27845]: pgmap v19385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:01 smithi096 ceph-mon[32753]: pgmap v19385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35882.250177 secs 2024-04-26T21:49:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35882.250177 secs 2024-04-26T21:49:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35882.250177 secs 2024-04-26T21:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:04 smithi033 ceph-mon[27845]: pgmap v19386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35885.215312 secs 2024-04-26T21:49:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:04 smithi096 ceph-mon[32753]: pgmap v19386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35885.215312 secs 2024-04-26T21:49:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:04 smithi194 ceph-mon[33017]: pgmap v19386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35885.215312 secs 2024-04-26T21:49:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:49:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:49:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:49:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:06 smithi194 ceph-mon[33017]: pgmap v19387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:49:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:06 smithi194 ceph-mon[33017]: pgmap v19388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:06 smithi033 ceph-mon[27845]: pgmap v19387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:49:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:06 smithi033 ceph-mon[27845]: pgmap v19388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:06 smithi096 ceph-mon[32753]: pgmap v19387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:49:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:06 smithi096 ceph-mon[32753]: pgmap v19388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:08 smithi194 ceph-mon[33017]: pgmap v19389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35887.250236 secs 2024-04-26T21:49:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:08 smithi033 ceph-mon[27845]: pgmap v19389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35887.250236 secs 2024-04-26T21:49:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:08 smithi096 ceph-mon[32753]: pgmap v19389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35887.250236 secs 2024-04-26T21:49:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35890.215446 secs 2024-04-26T21:49:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35890.215446 secs 2024-04-26T21:49:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35890.215446 secs 2024-04-26T21:49:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:10 smithi033 ceph-mon[27845]: pgmap v19390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:49:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:10 smithi194 ceph-mon[33017]: pgmap v19390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:49:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:10 smithi096 ceph-mon[32753]: pgmap v19390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:49:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:12 smithi194 ceph-mon[33017]: pgmap v19391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:12 smithi033 ceph-mon[27845]: pgmap v19391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:12 smithi096 ceph-mon[32753]: pgmap v19391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35892.250296 secs 2024-04-26T21:49:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35892.250296 secs 2024-04-26T21:49:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35892.250296 secs 2024-04-26T21:49:14.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:14.159+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:14.159 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:14.159+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:14.159+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:49:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:14 smithi194 ceph-mon[33017]: pgmap v19392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35895.215513 secs 2024-04-26T21:49:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:14 smithi033 ceph-mon[27845]: pgmap v19392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35895.215513 secs 2024-04-26T21:49:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:14 smithi096 ceph-mon[32753]: pgmap v19392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35895.215513 secs 2024-04-26T21:49:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:16 smithi194 ceph-mon[33017]: pgmap v19393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:16 smithi033 ceph-mon[27845]: pgmap v19393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:16 smithi096 ceph-mon[32753]: pgmap v19393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:18 smithi194 ceph-mon[33017]: pgmap v19394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35897.250356 secs 2024-04-26T21:49:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:18 smithi033 ceph-mon[27845]: pgmap v19394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35897.250356 secs 2024-04-26T21:49:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:18 smithi096 ceph-mon[32753]: pgmap v19394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35897.250356 secs 2024-04-26T21:49:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35900.215657 secs 2024-04-26T21:49:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35900.215657 secs 2024-04-26T21:49:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35900.215657 secs 2024-04-26T21:49:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:20 smithi194 ceph-mon[33017]: pgmap v19395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:20 smithi033 ceph-mon[27845]: pgmap v19395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:20 smithi096 ceph-mon[32753]: pgmap v19395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:22 smithi194 ceph-mon[33017]: pgmap v19396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:22 smithi033 ceph-mon[27845]: pgmap v19396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:22 smithi096 ceph-mon[32753]: pgmap v19396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35902.250417 secs 2024-04-26T21:49:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35902.250417 secs 2024-04-26T21:49:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35902.250417 secs 2024-04-26T21:49:24.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:24 smithi033 ceph-mon[27845]: pgmap v19397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:49:24.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35905.215782 secs 2024-04-26T21:49:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:24 smithi096 ceph-mon[32753]: pgmap v19397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:49:24.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35905.215782 secs 2024-04-26T21:49:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:24 smithi194 ceph-mon[33017]: pgmap v19397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:49:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35905.215782 secs 2024-04-26T21:49:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:26 smithi194 ceph-mon[33017]: pgmap v19398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:49:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:26 smithi033 ceph-mon[27845]: pgmap v19398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:49:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:26 smithi096 ceph-mon[32753]: pgmap v19398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:49:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:28 smithi194 ceph-mon[33017]: pgmap v19399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:49:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35907.250470 secs 2024-04-26T21:49:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:28 smithi033 ceph-mon[27845]: pgmap v19399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:49:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35907.250470 secs 2024-04-26T21:49:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:28 smithi096 ceph-mon[32753]: pgmap v19399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:49:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35907.250470 secs 2024-04-26T21:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:29.169+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:29.169+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:29.169+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:49:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35910.215918 secs 2024-04-26T21:49:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35910.215918 secs 2024-04-26T21:49:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35910.215918 secs 2024-04-26T21:49:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:30 smithi194 ceph-mon[33017]: pgmap v19400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:49:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:30 smithi033 ceph-mon[27845]: pgmap v19400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:49:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:30 smithi096 ceph-mon[32753]: pgmap v19400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:49:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:32 smithi194 ceph-mon[33017]: pgmap v19401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:32 smithi033 ceph-mon[27845]: pgmap v19401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:32 smithi096 ceph-mon[32753]: pgmap v19401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:49:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35912.250528 secs 2024-04-26T21:49:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35912.250528 secs 2024-04-26T21:49:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35912.250528 secs 2024-04-26T21:49:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:34 smithi194 ceph-mon[33017]: pgmap v19402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35915.216047 secs 2024-04-26T21:49:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:34 smithi033 ceph-mon[27845]: pgmap v19402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35915.216047 secs 2024-04-26T21:49:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:34 smithi096 ceph-mon[32753]: pgmap v19402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35915.216047 secs 2024-04-26T21:49:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:36 smithi194 ceph-mon[33017]: pgmap v19403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:36 smithi033 ceph-mon[27845]: pgmap v19403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:36 smithi096 ceph-mon[32753]: pgmap v19403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:38 smithi194 ceph-mon[33017]: pgmap v19404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35917.250589 secs 2024-04-26T21:49:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:38 smithi033 ceph-mon[27845]: pgmap v19404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35917.250589 secs 2024-04-26T21:49:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:38 smithi096 ceph-mon[32753]: pgmap v19404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35917.250589 secs 2024-04-26T21:49:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35920.216197 secs 2024-04-26T21:49:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35920.216197 secs 2024-04-26T21:49:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35920.216197 secs 2024-04-26T21:49:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:40 smithi033 ceph-mon[27845]: pgmap v19405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:40 smithi096 ceph-mon[32753]: pgmap v19405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:40 smithi194 ceph-mon[33017]: pgmap v19405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:42 smithi033 ceph-mon[27845]: pgmap v19406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:49:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:42 smithi096 ceph-mon[32753]: pgmap v19406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:49:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:42 smithi194 ceph-mon[33017]: pgmap v19406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:49:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35922.250649 secs 2024-04-26T21:49:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35922.250649 secs 2024-04-26T21:49:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35922.250649 secs 2024-04-26T21:49:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:44.185+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:44.185+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:49:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:44.185+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:44 smithi033 ceph-mon[27845]: pgmap v19407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35925.216323 secs 2024-04-26T21:49:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:44 smithi096 ceph-mon[32753]: pgmap v19407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35925.216323 secs 2024-04-26T21:49:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:44 smithi194 ceph-mon[33017]: pgmap v19407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35925.216323 secs 2024-04-26T21:49:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:46 smithi033 ceph-mon[27845]: pgmap v19408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:46 smithi096 ceph-mon[32753]: pgmap v19408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:46 smithi194 ceph-mon[33017]: pgmap v19408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:48 smithi033 ceph-mon[27845]: pgmap v19409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35927.250708 secs 2024-04-26T21:49:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:48 smithi096 ceph-mon[32753]: pgmap v19409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35927.250708 secs 2024-04-26T21:49:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:48 smithi194 ceph-mon[33017]: pgmap v19409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35927.250708 secs 2024-04-26T21:49:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35930.216408 secs 2024-04-26T21:49:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35930.216408 secs 2024-04-26T21:49:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35930.216408 secs 2024-04-26T21:49:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:50 smithi033 ceph-mon[27845]: pgmap v19410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:50 smithi096 ceph-mon[32753]: pgmap v19410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:50 smithi194 ceph-mon[33017]: pgmap v19410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:49:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:52 smithi033 ceph-mon[27845]: pgmap v19411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:49:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:52 smithi096 ceph-mon[32753]: pgmap v19411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:49:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:52 smithi194 ceph-mon[33017]: pgmap v19411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:49:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35932.250770 secs 2024-04-26T21:49:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35932.250770 secs 2024-04-26T21:49:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35932.250770 secs 2024-04-26T21:49:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:54 smithi033 ceph-mon[27845]: pgmap v19412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35935.216474 secs 2024-04-26T21:49:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:54 smithi096 ceph-mon[32753]: pgmap v19412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35935.216474 secs 2024-04-26T21:49:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:54 smithi194 ceph-mon[33017]: pgmap v19412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35935.216474 secs 2024-04-26T21:49:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:56 smithi033 ceph-mon[27845]: pgmap v19413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:56 smithi096 ceph-mon[32753]: pgmap v19413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:56 smithi194 ceph-mon[33017]: pgmap v19413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:58 smithi096 ceph-mon[32753]: pgmap v19414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35937.250852 secs 2024-04-26T21:49:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:58 smithi194 ceph-mon[33017]: pgmap v19414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35937.250852 secs 2024-04-26T21:49:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:58 smithi033 ceph-mon[27845]: pgmap v19414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:49:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35937.250852 secs 2024-04-26T21:49:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:59.186+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:59.186+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:49:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:49:59.186+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:50:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35940.216604 secs 2024-04-26T21:50:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35940.216604 secs 2024-04-26T21:50:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35940.216604 secs 2024-04-26T21:50:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:00 smithi194 ceph-mon[33017]: pgmap v19415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:50:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:00 smithi033 ceph-mon[27845]: pgmap v19415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:50:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:00 smithi096 ceph-mon[32753]: pgmap v19415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T21:50:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:02 smithi194 ceph-mon[33017]: pgmap v19416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:50:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:02 smithi033 ceph-mon[27845]: pgmap v19416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:50:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:02 smithi096 ceph-mon[32753]: pgmap v19416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:50:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35942.250938 secs 2024-04-26T21:50:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35942.250938 secs 2024-04-26T21:50:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35942.250938 secs 2024-04-26T21:50:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:04 smithi194 ceph-mon[33017]: pgmap v19417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35945.216749 secs 2024-04-26T21:50:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:04 smithi033 ceph-mon[27845]: pgmap v19417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35945.216749 secs 2024-04-26T21:50:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:04 smithi096 ceph-mon[32753]: pgmap v19417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35945.216749 secs 2024-04-26T21:50:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:05 smithi194 ceph-mon[33017]: pgmap v19418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:50:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:05 smithi033 ceph-mon[27845]: pgmap v19418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:50:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:05 smithi096 ceph-mon[32753]: pgmap v19418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:50:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:50:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:50:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:50:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:50:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:50:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:50:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:08 smithi194 ceph-mon[33017]: pgmap v19419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:08 smithi194 ceph-mon[33017]: pgmap v19420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:08 smithi033 ceph-mon[27845]: pgmap v19419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:08 smithi033 ceph-mon[27845]: pgmap v19420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:08 smithi096 ceph-mon[32753]: pgmap v19419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:08 smithi096 ceph-mon[32753]: pgmap v19420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35947.251036 secs 2024-04-26T21:50:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35947.251036 secs 2024-04-26T21:50:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35947.251036 secs 2024-04-26T21:50:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35950.216856 secs 2024-04-26T21:50:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:10 smithi194 ceph-mon[33017]: pgmap v19421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T21:50:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35950.216856 secs 2024-04-26T21:50:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:10 smithi033 ceph-mon[27845]: pgmap v19421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T21:50:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35950.216856 secs 2024-04-26T21:50:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:10 smithi096 ceph-mon[32753]: pgmap v19421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-26T21:50:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:12 smithi194 ceph-mon[33017]: pgmap v19422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:12 smithi033 ceph-mon[27845]: pgmap v19422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:12 smithi096 ceph-mon[32753]: pgmap v19422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:14.196+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:14.196+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:50:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:14.196+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35952.251121 secs 2024-04-26T21:50:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:14 smithi194 ceph-mon[33017]: pgmap v19423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35952.251121 secs 2024-04-26T21:50:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:14 smithi033 ceph-mon[27845]: pgmap v19423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35952.251121 secs 2024-04-26T21:50:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:14 smithi096 ceph-mon[32753]: pgmap v19423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35955.216978 secs 2024-04-26T21:50:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35955.216978 secs 2024-04-26T21:50:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35955.216978 secs 2024-04-26T21:50:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:16 smithi194 ceph-mon[33017]: pgmap v19424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:16 smithi033 ceph-mon[27845]: pgmap v19424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:16 smithi096 ceph-mon[32753]: pgmap v19424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:50:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:18 smithi194 ceph-mon[33017]: pgmap v19425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:50:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:18 smithi033 ceph-mon[27845]: pgmap v19425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:50:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:18 smithi096 ceph-mon[32753]: pgmap v19425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:50:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35957.251206 secs 2024-04-26T21:50:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35957.251206 secs 2024-04-26T21:50:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35957.251206 secs 2024-04-26T21:50:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35960.217081 secs 2024-04-26T21:50:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:20 smithi194 ceph-mon[33017]: pgmap v19426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35960.217081 secs 2024-04-26T21:50:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:20 smithi033 ceph-mon[27845]: pgmap v19426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35960.217081 secs 2024-04-26T21:50:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:20 smithi096 ceph-mon[32753]: pgmap v19426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:22 smithi194 ceph-mon[33017]: pgmap v19427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:22 smithi033 ceph-mon[27845]: pgmap v19427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:22 smithi096 ceph-mon[32753]: pgmap v19427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35962.251289 secs 2024-04-26T21:50:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:24 smithi194 ceph-mon[33017]: pgmap v19428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35962.251289 secs 2024-04-26T21:50:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:24 smithi033 ceph-mon[27845]: pgmap v19428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35962.251289 secs 2024-04-26T21:50:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:24 smithi096 ceph-mon[32753]: pgmap v19428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35965.217187 secs 2024-04-26T21:50:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35965.217187 secs 2024-04-26T21:50:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35965.217187 secs 2024-04-26T21:50:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:26 smithi194 ceph-mon[33017]: pgmap v19429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:26 smithi033 ceph-mon[27845]: pgmap v19429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:26 smithi096 ceph-mon[32753]: pgmap v19429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:27.921+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:50:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:27.921+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:50:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:27.922+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 281 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T21:50:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:27.922+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 281 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:50:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:28 smithi194 ceph-mon[33017]: pgmap v19430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:28 smithi033 ceph-mon[27845]: pgmap v19430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:28 smithi096 ceph-mon[32753]: pgmap v19430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:29.197+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:29.197+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:50:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:29.197+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35967.251350 secs 2024-04-26T21:50:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35967.251350 secs 2024-04-26T21:50:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35967.251350 secs 2024-04-26T21:50:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35970.217333 secs 2024-04-26T21:50:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:30 smithi194 ceph-mon[33017]: pgmap v19431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35970.217333 secs 2024-04-26T21:50:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:30 smithi033 ceph-mon[27845]: pgmap v19431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35970.217333 secs 2024-04-26T21:50:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:30 smithi096 ceph-mon[32753]: pgmap v19431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:32 smithi194 ceph-mon[33017]: pgmap v19432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:32 smithi033 ceph-mon[27845]: pgmap v19432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:32 smithi096 ceph-mon[32753]: pgmap v19432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35972.251410 secs 2024-04-26T21:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:34 smithi194 ceph-mon[33017]: pgmap v19433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35972.251410 secs 2024-04-26T21:50:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:34 smithi033 ceph-mon[27845]: pgmap v19433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35972.251410 secs 2024-04-26T21:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:34 smithi096 ceph-mon[32753]: pgmap v19433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35975.217473 secs 2024-04-26T21:50:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35975.217473 secs 2024-04-26T21:50:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35975.217473 secs 2024-04-26T21:50:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:36 smithi194 ceph-mon[33017]: pgmap v19434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:36 smithi033 ceph-mon[27845]: pgmap v19434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:36 smithi096 ceph-mon[32753]: pgmap v19434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:38 smithi194 ceph-mon[33017]: pgmap v19435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:38 smithi033 ceph-mon[27845]: pgmap v19435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:38 smithi096 ceph-mon[32753]: pgmap v19435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35977.251493 secs 2024-04-26T21:50:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35977.251493 secs 2024-04-26T21:50:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35977.251493 secs 2024-04-26T21:50:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35980.217559 secs 2024-04-26T21:50:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:40 smithi194 ceph-mon[33017]: pgmap v19436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35980.217559 secs 2024-04-26T21:50:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:40 smithi033 ceph-mon[27845]: pgmap v19436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35980.217559 secs 2024-04-26T21:50:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:40 smithi096 ceph-mon[32753]: pgmap v19436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:42 smithi194 ceph-mon[33017]: pgmap v19437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:42 smithi033 ceph-mon[27845]: pgmap v19437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:42 smithi096 ceph-mon[32753]: pgmap v19437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:44.212+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:44.212+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:44.212+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:50:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35982.251578 secs 2024-04-26T21:50:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:44 smithi194 ceph-mon[33017]: pgmap v19438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35982.251578 secs 2024-04-26T21:50:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:44 smithi033 ceph-mon[27845]: pgmap v19438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35982.251578 secs 2024-04-26T21:50:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:44 smithi096 ceph-mon[32753]: pgmap v19438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:45.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35985.217686 secs 2024-04-26T21:50:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35985.217686 secs 2024-04-26T21:50:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35985.217686 secs 2024-04-26T21:50:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:46 smithi033 ceph-mon[27845]: pgmap v19439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:46 smithi096 ceph-mon[32753]: pgmap v19439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:46 smithi194 ceph-mon[33017]: pgmap v19439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:48 smithi033 ceph-mon[27845]: pgmap v19440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35987.251663 secs 2024-04-26T21:50:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:48 smithi096 ceph-mon[32753]: pgmap v19440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35987.251663 secs 2024-04-26T21:50:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:48 smithi194 ceph-mon[33017]: pgmap v19440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35987.251663 secs 2024-04-26T21:50:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35990.217814 secs 2024-04-26T21:50:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:50 smithi033 ceph-mon[27845]: pgmap v19441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35990.217814 secs 2024-04-26T21:50:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:50 smithi096 ceph-mon[32753]: pgmap v19441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35990.217814 secs 2024-04-26T21:50:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:50 smithi194 ceph-mon[33017]: pgmap v19441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:51 smithi194 ceph-mon[33017]: pgmap v19442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:51 smithi033 ceph-mon[27845]: pgmap v19442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:51 smithi096 ceph-mon[32753]: pgmap v19442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35992.251744 secs 2024-04-26T21:50:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:54 smithi194 ceph-mon[33017]: pgmap v19443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35992.251744 secs 2024-04-26T21:50:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:54 smithi033 ceph-mon[27845]: pgmap v19443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35992.251744 secs 2024-04-26T21:50:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:54 smithi096 ceph-mon[32753]: pgmap v19443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 35995.217913 secs 2024-04-26T21:50:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 35995.217913 secs 2024-04-26T21:50:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 35995.217913 secs 2024-04-26T21:50:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:56 smithi194 ceph-mon[33017]: pgmap v19444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:56 smithi033 ceph-mon[27845]: pgmap v19444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:56 smithi096 ceph-mon[32753]: pgmap v19444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:50:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:58 smithi194 ceph-mon[33017]: pgmap v19445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:58 smithi033 ceph-mon[27845]: pgmap v19445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:58 smithi096 ceph-mon[32753]: pgmap v19445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:50:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:59.226+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:59.226+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:50:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:50:59.226+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:50:59.597 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:50:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 35997.251828 secs 2024-04-26T21:50:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:50:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 35997.251828 secs 2024-04-26T21:50:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:50:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 35997.251828 secs 2024-04-26T21:51:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36000.218024 secs 2024-04-26T21:51:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:00 smithi194 ceph-mon[33017]: pgmap v19446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36000.218024 secs 2024-04-26T21:51:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:00 smithi033 ceph-mon[27845]: pgmap v19446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36000.218024 secs 2024-04-26T21:51:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:00 smithi096 ceph-mon[32753]: pgmap v19446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:02.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:02 smithi194 ceph-mon[33017]: pgmap v19447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:02 smithi033 ceph-mon[27845]: pgmap v19447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:02 smithi096 ceph-mon[32753]: pgmap v19447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36002.251908 secs 2024-04-26T21:51:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:04 smithi194 ceph-mon[33017]: pgmap v19448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36002.251908 secs 2024-04-26T21:51:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:04 smithi033 ceph-mon[27845]: pgmap v19448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36002.251908 secs 2024-04-26T21:51:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:04 smithi096 ceph-mon[32753]: pgmap v19448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36005.218147 secs 2024-04-26T21:51:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36005.218147 secs 2024-04-26T21:51:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36005.218147 secs 2024-04-26T21:51:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:06 smithi194 ceph-mon[33017]: pgmap v19449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:06 smithi033 ceph-mon[27845]: pgmap v19449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:06 smithi096 ceph-mon[32753]: pgmap v19449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:08 smithi194 ceph-mon[33017]: pgmap v19450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:51:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:08 smithi033 ceph-mon[27845]: pgmap v19450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:51:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:08 smithi096 ceph-mon[32753]: pgmap v19450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:51:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36007.252001 secs 2024-04-26T21:51:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:51:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:51:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:51:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36007.252001 secs 2024-04-26T21:51:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:51:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:51:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:51:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36007.252001 secs 2024-04-26T21:51:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:51:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:51:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:51:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36010.218182 secs 2024-04-26T21:51:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:10 smithi194 ceph-mon[33017]: pgmap v19451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:10 smithi194 ceph-mon[33017]: pgmap v19452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:10.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36010.218182 secs 2024-04-26T21:51:10.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:10 smithi033 ceph-mon[27845]: pgmap v19451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:10.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:10 smithi033 ceph-mon[27845]: pgmap v19452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36010.218182 secs 2024-04-26T21:51:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:10 smithi096 ceph-mon[32753]: pgmap v19451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:10 smithi096 ceph-mon[32753]: pgmap v19452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:12.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:12 smithi194 ceph-mon[33017]: pgmap v19453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:12 smithi033 ceph-mon[27845]: pgmap v19453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:12 smithi096 ceph-mon[32753]: pgmap v19453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:14.243+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:14.243+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:14.243+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:51:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:14 smithi194 ceph-mon[33017]: pgmap v19454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36012.252080 secs 2024-04-26T21:51:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:14 smithi033 ceph-mon[27845]: pgmap v19454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36012.252080 secs 2024-04-26T21:51:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:14 smithi096 ceph-mon[32753]: pgmap v19454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36012.252080 secs 2024-04-26T21:51:15.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36015.218309 secs 2024-04-26T21:51:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36015.218309 secs 2024-04-26T21:51:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36015.218309 secs 2024-04-26T21:51:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:16 smithi194 ceph-mon[33017]: pgmap v19455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:16 smithi033 ceph-mon[27845]: pgmap v19455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:16 smithi096 ceph-mon[32753]: pgmap v19455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:18 smithi194 ceph-mon[33017]: pgmap v19456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:18 smithi033 ceph-mon[27845]: pgmap v19456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:18 smithi096 ceph-mon[32753]: pgmap v19456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:51:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36017.252163 secs 2024-04-26T21:51:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36017.252163 secs 2024-04-26T21:51:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36017.252163 secs 2024-04-26T21:51:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36020.218370 secs 2024-04-26T21:51:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:20 smithi194 ceph-mon[33017]: pgmap v19457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:51:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36020.218370 secs 2024-04-26T21:51:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:20 smithi033 ceph-mon[27845]: pgmap v19457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:51:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36020.218370 secs 2024-04-26T21:51:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:20 smithi096 ceph-mon[32753]: pgmap v19457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:51:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:22 smithi194 ceph-mon[33017]: pgmap v19458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:22 smithi033 ceph-mon[27845]: pgmap v19458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:22 smithi096 ceph-mon[32753]: pgmap v19458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:24 smithi194 ceph-mon[33017]: pgmap v19459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36022.252250 secs 2024-04-26T21:51:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:24 smithi033 ceph-mon[27845]: pgmap v19459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36022.252250 secs 2024-04-26T21:51:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:24 smithi096 ceph-mon[32753]: pgmap v19459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36022.252250 secs 2024-04-26T21:51:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36025.218517 secs 2024-04-26T21:51:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36025.218517 secs 2024-04-26T21:51:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36025.218517 secs 2024-04-26T21:51:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:26 smithi033 ceph-mon[27845]: pgmap v19460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:26 smithi096 ceph-mon[32753]: pgmap v19460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:26 smithi194 ceph-mon[33017]: pgmap v19460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:28.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:28 smithi194 ceph-mon[33017]: pgmap v19461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:28.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36027.252331 secs 2024-04-26T21:51:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:28 smithi033 ceph-mon[27845]: pgmap v19461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36027.252331 secs 2024-04-26T21:51:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:28 smithi096 ceph-mon[32753]: pgmap v19461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36027.252331 secs 2024-04-26T21:51:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:29.254+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:29.255+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:51:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:29.255+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36030.218582 secs 2024-04-26T21:51:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:30 smithi033 ceph-mon[27845]: pgmap v19462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36030.218582 secs 2024-04-26T21:51:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:30 smithi096 ceph-mon[32753]: pgmap v19462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36030.218582 secs 2024-04-26T21:51:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:30 smithi194 ceph-mon[33017]: pgmap v19462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:32 smithi033 ceph-mon[27845]: pgmap v19463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:32 smithi096 ceph-mon[32753]: pgmap v19463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:32 smithi194 ceph-mon[33017]: pgmap v19463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36032.252416 secs 2024-04-26T21:51:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36032.252416 secs 2024-04-26T21:51:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36032.252416 secs 2024-04-26T21:51:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:34 smithi033 ceph-mon[27845]: pgmap v19464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:34 smithi096 ceph-mon[32753]: pgmap v19464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:34 smithi194 ceph-mon[33017]: pgmap v19464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36035.218647 secs 2024-04-26T21:51:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36035.218647 secs 2024-04-26T21:51:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36035.218647 secs 2024-04-26T21:51:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:36 smithi033 ceph-mon[27845]: pgmap v19465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:36 smithi096 ceph-mon[32753]: pgmap v19465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:36 smithi194 ceph-mon[33017]: pgmap v19465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:38 smithi033 ceph-mon[27845]: pgmap v19466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36037.252498 secs 2024-04-26T21:51:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:38 smithi096 ceph-mon[32753]: pgmap v19466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36037.252498 secs 2024-04-26T21:51:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:38 smithi194 ceph-mon[33017]: pgmap v19466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36037.252498 secs 2024-04-26T21:51:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36040.218775 secs 2024-04-26T21:51:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36040.218775 secs 2024-04-26T21:51:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36040.218775 secs 2024-04-26T21:51:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:40 smithi033 ceph-mon[27845]: pgmap v19467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:40 smithi096 ceph-mon[32753]: pgmap v19467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:40 smithi194 ceph-mon[33017]: pgmap v19467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:42 smithi033 ceph-mon[27845]: pgmap v19468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:42 smithi096 ceph-mon[32753]: pgmap v19468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:42 smithi194 ceph-mon[33017]: pgmap v19468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36042.252580 secs 2024-04-26T21:51:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36042.252580 secs 2024-04-26T21:51:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36042.252580 secs 2024-04-26T21:51:44.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:44.267+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:44.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:44.267+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:51:44.268 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:44.267+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:44 smithi194 ceph-mon[33017]: pgmap v19469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36045.218922 secs 2024-04-26T21:51:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:44 smithi033 ceph-mon[27845]: pgmap v19469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36045.218922 secs 2024-04-26T21:51:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:44 smithi096 ceph-mon[32753]: pgmap v19469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36045.218922 secs 2024-04-26T21:51:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:46 smithi194 ceph-mon[33017]: pgmap v19470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:46 smithi033 ceph-mon[27845]: pgmap v19470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:46.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:46 smithi096 ceph-mon[32753]: pgmap v19470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:48 smithi194 ceph-mon[33017]: pgmap v19471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36047.252663 secs 2024-04-26T21:51:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:48 smithi033 ceph-mon[27845]: pgmap v19471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36047.252663 secs 2024-04-26T21:51:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:48 smithi096 ceph-mon[32753]: pgmap v19471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36047.252663 secs 2024-04-26T21:51:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36050.219019 secs 2024-04-26T21:51:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36050.219019 secs 2024-04-26T21:51:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36050.219019 secs 2024-04-26T21:51:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:50 smithi194 ceph-mon[33017]: pgmap v19472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:50 smithi033 ceph-mon[27845]: pgmap v19472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:50 smithi096 ceph-mon[32753]: pgmap v19472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:52 smithi194 ceph-mon[33017]: pgmap v19473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:52 smithi033 ceph-mon[27845]: pgmap v19473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:52 smithi096 ceph-mon[32753]: pgmap v19473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36052.252748 secs 2024-04-26T21:51:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36052.252748 secs 2024-04-26T21:51:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36052.252748 secs 2024-04-26T21:51:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:54 smithi033 ceph-mon[27845]: pgmap v19474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:54.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36055.219136 secs 2024-04-26T21:51:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:54 smithi096 ceph-mon[32753]: pgmap v19474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:54.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36055.219136 secs 2024-04-26T21:51:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:54 smithi194 ceph-mon[33017]: pgmap v19474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36055.219136 secs 2024-04-26T21:51:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:56 smithi194 ceph-mon[33017]: pgmap v19475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:56 smithi033 ceph-mon[27845]: pgmap v19475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:56 smithi096 ceph-mon[32753]: pgmap v19475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:51:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:57.935+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:51:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:57.935+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:51:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:57.935+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 282 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T21:51:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:57.936+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 282 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:51:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:58 smithi194 ceph-mon[33017]: pgmap v19476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36057.252833 secs 2024-04-26T21:51:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:58 smithi033 ceph-mon[27845]: pgmap v19476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36057.252833 secs 2024-04-26T21:51:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:58 smithi096 ceph-mon[32753]: pgmap v19476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:51:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36057.252833 secs 2024-04-26T21:51:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:59.277+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:59.277+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:51:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:51:59.277+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:51:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:51:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36060.219208 secs 2024-04-26T21:51:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:51:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36060.219208 secs 2024-04-26T21:51:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:51:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36060.219208 secs 2024-04-26T21:52:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:00 smithi194 ceph-mon[33017]: pgmap v19477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:00 smithi033 ceph-mon[27845]: pgmap v19477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:00 smithi096 ceph-mon[32753]: pgmap v19477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:02 smithi194 ceph-mon[33017]: pgmap v19478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:02 smithi033 ceph-mon[27845]: pgmap v19478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:02 smithi096 ceph-mon[32753]: pgmap v19478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36062.252892 secs 2024-04-26T21:52:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36062.252892 secs 2024-04-26T21:52:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36062.252892 secs 2024-04-26T21:52:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:04 smithi194 ceph-mon[33017]: pgmap v19479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36065.219272 secs 2024-04-26T21:52:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:04 smithi033 ceph-mon[27845]: pgmap v19479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36065.219272 secs 2024-04-26T21:52:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:04 smithi096 ceph-mon[32753]: pgmap v19479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36065.219272 secs 2024-04-26T21:52:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:06 smithi194 ceph-mon[33017]: pgmap v19480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:06 smithi033 ceph-mon[27845]: pgmap v19480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:06 smithi096 ceph-mon[32753]: pgmap v19480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:08 smithi194 ceph-mon[33017]: pgmap v19481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:08.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36067.252995 secs 2024-04-26T21:52:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:08 smithi033 ceph-mon[27845]: pgmap v19481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36067.252995 secs 2024-04-26T21:52:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:08 smithi096 ceph-mon[32753]: pgmap v19481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36067.252995 secs 2024-04-26T21:52:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36070.219424 secs 2024-04-26T21:52:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:52:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36070.219424 secs 2024-04-26T21:52:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:52:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36070.219424 secs 2024-04-26T21:52:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:52:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:10 smithi194 ceph-mon[33017]: pgmap v19482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:10 smithi033 ceph-mon[27845]: pgmap v19482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:10 smithi096 ceph-mon[32753]: pgmap v19482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:52:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:52:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:11 smithi194 ceph-mon[33017]: pgmap v19483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:52:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:11 smithi194 ceph-mon[33017]: pgmap v19484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:52:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:52:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:52:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:11 smithi033 ceph-mon[27845]: pgmap v19483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:52:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:11 smithi033 ceph-mon[27845]: pgmap v19484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:11.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:52:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:52:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:52:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:11 smithi096 ceph-mon[32753]: pgmap v19483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:52:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:11 smithi096 ceph-mon[32753]: pgmap v19484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:52:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:13 smithi194 ceph-mon[33017]: pgmap v19485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36072.253075 secs 2024-04-26T21:52:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:13 smithi033 ceph-mon[27845]: pgmap v19485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36072.253075 secs 2024-04-26T21:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:13 smithi096 ceph-mon[32753]: pgmap v19485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36072.253075 secs 2024-04-26T21:52:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:14.291+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:14.291+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:52:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:14.291+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36075.219493 secs 2024-04-26T21:52:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36075.219493 secs 2024-04-26T21:52:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36075.219493 secs 2024-04-26T21:52:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:15 smithi194 ceph-mon[33017]: pgmap v19486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:15 smithi033 ceph-mon[27845]: pgmap v19486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:15 smithi096 ceph-mon[32753]: pgmap v19486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:17 smithi194 ceph-mon[33017]: pgmap v19487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:17 smithi033 ceph-mon[27845]: pgmap v19487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:17 smithi096 ceph-mon[32753]: pgmap v19487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36077.253158 secs 2024-04-26T21:52:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36077.253158 secs 2024-04-26T21:52:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36077.253158 secs 2024-04-26T21:52:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:19 smithi194 ceph-mon[33017]: pgmap v19488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36080.219624 secs 2024-04-26T21:52:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:19 smithi033 ceph-mon[27845]: pgmap v19488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36080.219624 secs 2024-04-26T21:52:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:19 smithi096 ceph-mon[32753]: pgmap v19488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:52:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36080.219624 secs 2024-04-26T21:52:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:21 smithi194 ceph-mon[33017]: pgmap v19489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:52:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:21 smithi033 ceph-mon[27845]: pgmap v19489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:52:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:21 smithi096 ceph-mon[32753]: pgmap v19489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:52:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:23 smithi194 ceph-mon[33017]: pgmap v19490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36082.253260 secs 2024-04-26T21:52:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:23 smithi033 ceph-mon[27845]: pgmap v19490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36082.253260 secs 2024-04-26T21:52:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:23 smithi096 ceph-mon[32753]: pgmap v19490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36082.253260 secs 2024-04-26T21:52:25.076 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36085.219701 secs 2024-04-26T21:52:25.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36085.219701 secs 2024-04-26T21:52:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36085.219701 secs 2024-04-26T21:52:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:25 smithi194 ceph-mon[33017]: pgmap v19491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:25 smithi033 ceph-mon[27845]: pgmap v19491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:25 smithi096 ceph-mon[32753]: pgmap v19491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:26 smithi194 ceph-mon[33017]: pgmap v19492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:26 smithi033 ceph-mon[27845]: pgmap v19492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:26 smithi096 ceph-mon[32753]: pgmap v19492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36087.253326 secs 2024-04-26T21:52:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36087.253326 secs 2024-04-26T21:52:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36087.253326 secs 2024-04-26T21:52:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:29.293+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:29.294+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:52:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:29.294+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:29 smithi033 ceph-mon[27845]: pgmap v19493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36090.219816 secs 2024-04-26T21:52:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:29 smithi096 ceph-mon[32753]: pgmap v19493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36090.219816 secs 2024-04-26T21:52:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:29 smithi194 ceph-mon[33017]: pgmap v19493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36090.219816 secs 2024-04-26T21:52:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:31 smithi194 ceph-mon[33017]: pgmap v19494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:31 smithi033 ceph-mon[27845]: pgmap v19494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:31 smithi096 ceph-mon[32753]: pgmap v19494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:33 smithi194 ceph-mon[33017]: pgmap v19495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36092.253388 secs 2024-04-26T21:52:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:33 smithi033 ceph-mon[27845]: pgmap v19495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36092.253388 secs 2024-04-26T21:52:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:33 smithi096 ceph-mon[32753]: pgmap v19495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36092.253388 secs 2024-04-26T21:52:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36095.219961 secs 2024-04-26T21:52:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36095.219961 secs 2024-04-26T21:52:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36095.219961 secs 2024-04-26T21:52:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:35 smithi194 ceph-mon[33017]: pgmap v19496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:35 smithi033 ceph-mon[27845]: pgmap v19496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:35 smithi096 ceph-mon[32753]: pgmap v19496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:37 smithi194 ceph-mon[33017]: pgmap v19497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:37 smithi033 ceph-mon[27845]: pgmap v19497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:37 smithi096 ceph-mon[32753]: pgmap v19497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36097.253447 secs 2024-04-26T21:52:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36097.253447 secs 2024-04-26T21:52:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36097.253447 secs 2024-04-26T21:52:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:39 smithi194 ceph-mon[33017]: pgmap v19498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36100.220087 secs 2024-04-26T21:52:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:39 smithi033 ceph-mon[27845]: pgmap v19498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36100.220087 secs 2024-04-26T21:52:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:39 smithi096 ceph-mon[32753]: pgmap v19498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36100.220087 secs 2024-04-26T21:52:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:41 smithi194 ceph-mon[33017]: pgmap v19499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:41 smithi033 ceph-mon[27845]: pgmap v19499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:41 smithi096 ceph-mon[32753]: pgmap v19499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:43 smithi194 ceph-mon[33017]: pgmap v19500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36102.253530 secs 2024-04-26T21:52:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:43 smithi033 ceph-mon[27845]: pgmap v19500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36102.253530 secs 2024-04-26T21:52:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:43 smithi096 ceph-mon[32753]: pgmap v19500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36102.253530 secs 2024-04-26T21:52:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:44.309+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:44.309+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:52:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:44.309+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36105.220162 secs 2024-04-26T21:52:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36105.220162 secs 2024-04-26T21:52:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36105.220162 secs 2024-04-26T21:52:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:45 smithi194 ceph-mon[33017]: pgmap v19501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:45 smithi033 ceph-mon[27845]: pgmap v19501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:45 smithi096 ceph-mon[32753]: pgmap v19501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:46 smithi194 ceph-mon[33017]: pgmap v19502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:46 smithi033 ceph-mon[27845]: pgmap v19502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:46 smithi096 ceph-mon[32753]: pgmap v19502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36107.253611 secs 2024-04-26T21:52:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36107.253611 secs 2024-04-26T21:52:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36107.253611 secs 2024-04-26T21:52:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:49 smithi033 ceph-mon[27845]: pgmap v19503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36110.220290 secs 2024-04-26T21:52:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:49 smithi096 ceph-mon[32753]: pgmap v19503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36110.220290 secs 2024-04-26T21:52:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:49 smithi194 ceph-mon[33017]: pgmap v19503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:52:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36110.220290 secs 2024-04-26T21:52:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:51 smithi194 ceph-mon[33017]: pgmap v19504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:51 smithi033 ceph-mon[27845]: pgmap v19504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:51 smithi096 ceph-mon[32753]: pgmap v19504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:52:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:53 smithi194 ceph-mon[33017]: pgmap v19505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36112.253695 secs 2024-04-26T21:52:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:53 smithi033 ceph-mon[27845]: pgmap v19505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36112.253695 secs 2024-04-26T21:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:53 smithi096 ceph-mon[32753]: pgmap v19505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36112.253695 secs 2024-04-26T21:52:55.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36115.220412 secs 2024-04-26T21:52:55.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36115.220412 secs 2024-04-26T21:52:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36115.220412 secs 2024-04-26T21:52:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:55 smithi194 ceph-mon[33017]: pgmap v19506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:55 smithi033 ceph-mon[27845]: pgmap v19506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:55 smithi096 ceph-mon[32753]: pgmap v19506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:52:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:57 smithi194 ceph-mon[33017]: pgmap v19507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:57 smithi033 ceph-mon[27845]: pgmap v19507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:57 smithi096 ceph-mon[32753]: pgmap v19507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:52:59.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36117.253778 secs 2024-04-26T21:52:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36117.253778 secs 2024-04-26T21:52:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36117.253778 secs 2024-04-26T21:52:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:59.313+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:59.314+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:52:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:52:59.314+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:53:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:59 smithi194 ceph-mon[33017]: pgmap v19508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:52:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36120.220560 secs 2024-04-26T21:53:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:59 smithi033 ceph-mon[27845]: pgmap v19508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:52:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36120.220560 secs 2024-04-26T21:53:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:59 smithi096 ceph-mon[32753]: pgmap v19508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:52:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36120.220560 secs 2024-04-26T21:53:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:01 smithi194 ceph-mon[33017]: pgmap v19509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:01 smithi033 ceph-mon[27845]: pgmap v19509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:01 smithi096 ceph-mon[32753]: pgmap v19509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:03 smithi194 ceph-mon[33017]: pgmap v19510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36122.253862 secs 2024-04-26T21:53:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:03 smithi033 ceph-mon[27845]: pgmap v19510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36122.253862 secs 2024-04-26T21:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:03 smithi096 ceph-mon[32753]: pgmap v19510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36122.253862 secs 2024-04-26T21:53:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36125.220639 secs 2024-04-26T21:53:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:04 smithi194 ceph-mon[33017]: pgmap v19511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36125.220639 secs 2024-04-26T21:53:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:04 smithi033 ceph-mon[27845]: pgmap v19511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36125.220639 secs 2024-04-26T21:53:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:04 smithi096 ceph-mon[32753]: pgmap v19511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:07 smithi194 ceph-mon[33017]: pgmap v19512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:07 smithi033 ceph-mon[27845]: pgmap v19512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:07 smithi096 ceph-mon[32753]: pgmap v19512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36127.253944 secs 2024-04-26T21:53:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36127.253944 secs 2024-04-26T21:53:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36127.253944 secs 2024-04-26T21:53:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:09 smithi194 ceph-mon[33017]: pgmap v19513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36130.220766 secs 2024-04-26T21:53:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:09 smithi033 ceph-mon[27845]: pgmap v19513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36130.220766 secs 2024-04-26T21:53:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:09 smithi096 ceph-mon[32753]: pgmap v19513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36130.220766 secs 2024-04-26T21:53:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:11 smithi194 ceph-mon[33017]: pgmap v19514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:53:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:11 smithi033 ceph-mon[27845]: pgmap v19514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:11 smithi096 ceph-mon[32753]: pgmap v19514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:53:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:12 smithi033 ceph-mon[27845]: pgmap v19515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:12 smithi096 ceph-mon[32753]: pgmap v19515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:13.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:13.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:13.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:12 smithi194 ceph-mon[33017]: pgmap v19515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36132.254023 secs 2024-04-26T21:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:13 smithi033 ceph-mon[27845]: pgmap v19516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:13 smithi033 ceph-mon[27845]: pgmap v19517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:53:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36132.254023 secs 2024-04-26T21:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:13 smithi096 ceph-mon[32753]: pgmap v19516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:13 smithi096 ceph-mon[32753]: pgmap v19517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:53:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:14.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:14.315+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:14.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:14.315+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:53:14.316 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:14.315+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36132.254023 secs 2024-04-26T21:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:13 smithi194 ceph-mon[33017]: pgmap v19516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:13 smithi194 ceph-mon[33017]: pgmap v19517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:53:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:53:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36135.220892 secs 2024-04-26T21:53:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36135.220892 secs 2024-04-26T21:53:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36135.220892 secs 2024-04-26T21:53:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:15 smithi033 ceph-mon[27845]: pgmap v19518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:15 smithi096 ceph-mon[32753]: pgmap v19518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:15 smithi194 ceph-mon[33017]: pgmap v19518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:18 smithi194 ceph-mon[33017]: pgmap v19519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:18 smithi033 ceph-mon[27845]: pgmap v19519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:18 smithi096 ceph-mon[32753]: pgmap v19519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36137.254111 secs 2024-04-26T21:53:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36137.254111 secs 2024-04-26T21:53:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36137.254111 secs 2024-04-26T21:53:20.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36140.220956 secs 2024-04-26T21:53:20.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:20 smithi033 ceph-mon[27845]: pgmap v19520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:53:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36140.220956 secs 2024-04-26T21:53:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:20 smithi194 ceph-mon[33017]: pgmap v19520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:53:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36140.220956 secs 2024-04-26T21:53:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:20 smithi096 ceph-mon[32753]: pgmap v19520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:53:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:22 smithi194 ceph-mon[33017]: pgmap v19521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:22 smithi033 ceph-mon[27845]: pgmap v19521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:22 smithi096 ceph-mon[32753]: pgmap v19521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:53:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36142.254198 secs 2024-04-26T21:53:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:24 smithi194 ceph-mon[33017]: pgmap v19522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:53:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36142.254198 secs 2024-04-26T21:53:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:24 smithi033 ceph-mon[27845]: pgmap v19522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:53:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36142.254198 secs 2024-04-26T21:53:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:24 smithi096 ceph-mon[32753]: pgmap v19522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:53:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36145.221078 secs 2024-04-26T21:53:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36145.221078 secs 2024-04-26T21:53:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36145.221078 secs 2024-04-26T21:53:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:26 smithi194 ceph-mon[33017]: pgmap v19523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:26 smithi033 ceph-mon[27845]: pgmap v19523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:26 smithi096 ceph-mon[32753]: pgmap v19523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:27.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:27.951+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:53:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:27.951+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:53:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:27.952+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 283 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T21:53:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:27.952+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 283 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:53:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:28 smithi194 ceph-mon[33017]: pgmap v19524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:28 smithi033 ceph-mon[27845]: pgmap v19524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:28 smithi096 ceph-mon[32753]: pgmap v19524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:29.331+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:29.331+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:29.331+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:53:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36147.254284 secs 2024-04-26T21:53:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36147.254284 secs 2024-04-26T21:53:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36147.254284 secs 2024-04-26T21:53:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36150.221192 secs 2024-04-26T21:53:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:30 smithi033 ceph-mon[27845]: pgmap v19525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36150.221192 secs 2024-04-26T21:53:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:30 smithi096 ceph-mon[32753]: pgmap v19525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36150.221192 secs 2024-04-26T21:53:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:30 smithi194 ceph-mon[33017]: pgmap v19525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:32 smithi033 ceph-mon[27845]: pgmap v19526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:32 smithi096 ceph-mon[32753]: pgmap v19526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:32 smithi194 ceph-mon[33017]: pgmap v19526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36152.254367 secs 2024-04-26T21:53:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36152.254367 secs 2024-04-26T21:53:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36152.254367 secs 2024-04-26T21:53:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:34 smithi033 ceph-mon[27845]: pgmap v19527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:34 smithi096 ceph-mon[32753]: pgmap v19527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:34 smithi194 ceph-mon[33017]: pgmap v19527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36155.221270 secs 2024-04-26T21:53:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36155.221270 secs 2024-04-26T21:53:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36155.221270 secs 2024-04-26T21:53:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:36 smithi033 ceph-mon[27845]: pgmap v19528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:36 smithi096 ceph-mon[32753]: pgmap v19528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:36 smithi194 ceph-mon[33017]: pgmap v19528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:38 smithi033 ceph-mon[27845]: pgmap v19529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36157.254451 secs 2024-04-26T21:53:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:38 smithi096 ceph-mon[32753]: pgmap v19529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36157.254451 secs 2024-04-26T21:53:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:38 smithi194 ceph-mon[33017]: pgmap v19529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36157.254451 secs 2024-04-26T21:53:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36160.221380 secs 2024-04-26T21:53:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:40 smithi033 ceph-mon[27845]: pgmap v19530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36160.221380 secs 2024-04-26T21:53:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:40 smithi096 ceph-mon[32753]: pgmap v19530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36160.221380 secs 2024-04-26T21:53:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:40 smithi194 ceph-mon[33017]: pgmap v19530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:42 smithi033 ceph-mon[27845]: pgmap v19531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:42 smithi096 ceph-mon[32753]: pgmap v19531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:42 smithi194 ceph-mon[33017]: pgmap v19531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36162.254536 secs 2024-04-26T21:53:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36162.254536 secs 2024-04-26T21:53:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36162.254536 secs 2024-04-26T21:53:44.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:44.339+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:44.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:44.340+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:44.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:44.340+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:53:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:44 smithi033 ceph-mon[27845]: pgmap v19532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36165.221498 secs 2024-04-26T21:53:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:44 smithi096 ceph-mon[32753]: pgmap v19532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36165.221498 secs 2024-04-26T21:53:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:44 smithi194 ceph-mon[33017]: pgmap v19532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36165.221498 secs 2024-04-26T21:53:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:46 smithi096 ceph-mon[32753]: pgmap v19533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:46 smithi194 ceph-mon[33017]: pgmap v19533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:46 smithi033 ceph-mon[27845]: pgmap v19533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:48 smithi194 ceph-mon[33017]: pgmap v19534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36167.254632 secs 2024-04-26T21:53:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:48 smithi033 ceph-mon[27845]: pgmap v19534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36167.254632 secs 2024-04-26T21:53:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:48 smithi096 ceph-mon[32753]: pgmap v19534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36167.254632 secs 2024-04-26T21:53:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36170.221637 secs 2024-04-26T21:53:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36170.221637 secs 2024-04-26T21:53:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36170.221637 secs 2024-04-26T21:53:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:50 smithi194 ceph-mon[33017]: pgmap v19535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:50.895 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:50 smithi033 ceph-mon[27845]: pgmap v19535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:50 smithi096 ceph-mon[32753]: pgmap v19535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:52 smithi194 ceph-mon[33017]: pgmap v19536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:52 smithi033 ceph-mon[27845]: pgmap v19536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:52 smithi096 ceph-mon[32753]: pgmap v19536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36172.254718 secs 2024-04-26T21:53:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36172.254718 secs 2024-04-26T21:53:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36172.254718 secs 2024-04-26T21:53:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:54 smithi194 ceph-mon[33017]: pgmap v19537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36175.221774 secs 2024-04-26T21:53:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:54 smithi033 ceph-mon[27845]: pgmap v19537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36175.221774 secs 2024-04-26T21:53:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:54 smithi096 ceph-mon[32753]: pgmap v19537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36175.221774 secs 2024-04-26T21:53:56.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:56 smithi033 ceph-mon[27845]: pgmap v19538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:56.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:56 smithi096 ceph-mon[32753]: pgmap v19538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:56 smithi194 ceph-mon[33017]: pgmap v19538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:53:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:58 smithi194 ceph-mon[33017]: pgmap v19539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36177.254805 secs 2024-04-26T21:53:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:58 smithi033 ceph-mon[27845]: pgmap v19539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36177.254805 secs 2024-04-26T21:53:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:58 smithi096 ceph-mon[32753]: pgmap v19539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:53:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36177.254805 secs 2024-04-26T21:53:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:59.354+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:59.354+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:53:59.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:53:59.354+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:53:59.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36180.221889 secs 2024-04-26T21:53:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36180.221889 secs 2024-04-26T21:53:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36180.221889 secs 2024-04-26T21:54:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:00 smithi194 ceph-mon[33017]: pgmap v19540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:00 smithi033 ceph-mon[27845]: pgmap v19540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:00 smithi096 ceph-mon[32753]: pgmap v19540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:02 smithi194 ceph-mon[33017]: pgmap v19541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:02 smithi033 ceph-mon[27845]: pgmap v19541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:02 smithi096 ceph-mon[32753]: pgmap v19541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36182.254893 secs 2024-04-26T21:54:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36182.254893 secs 2024-04-26T21:54:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36182.254893 secs 2024-04-26T21:54:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:04 smithi194 ceph-mon[33017]: pgmap v19542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36185.222009 secs 2024-04-26T21:54:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:04 smithi033 ceph-mon[27845]: pgmap v19542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36185.222009 secs 2024-04-26T21:54:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:04 smithi096 ceph-mon[32753]: pgmap v19542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36185.222009 secs 2024-04-26T21:54:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:06 smithi194 ceph-mon[33017]: pgmap v19543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:06 smithi033 ceph-mon[27845]: pgmap v19543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:06 smithi096 ceph-mon[32753]: pgmap v19543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:08 smithi194 ceph-mon[33017]: pgmap v19544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36187.254996 secs 2024-04-26T21:54:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:08 smithi033 ceph-mon[27845]: pgmap v19544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36187.254996 secs 2024-04-26T21:54:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:08 smithi096 ceph-mon[32753]: pgmap v19544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36187.254996 secs 2024-04-26T21:54:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36190.222129 secs 2024-04-26T21:54:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36190.222129 secs 2024-04-26T21:54:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36190.222129 secs 2024-04-26T21:54:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:10 smithi194 ceph-mon[33017]: pgmap v19545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:10 smithi033 ceph-mon[27845]: pgmap v19545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:10 smithi096 ceph-mon[32753]: pgmap v19545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:12.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:12 smithi194 ceph-mon[33017]: pgmap v19546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:12 smithi033 ceph-mon[27845]: pgmap v19546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:12 smithi096 ceph-mon[32753]: pgmap v19546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36192.255086 secs 2024-04-26T21:54:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36192.255086 secs 2024-04-26T21:54:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36192.255086 secs 2024-04-26T21:54:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:14.370+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:14.370+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:14.370+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:54:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:14 smithi194 ceph-mon[33017]: pgmap v19547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:54:14.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36195.222204 secs 2024-04-26T21:54:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:14 smithi033 ceph-mon[27845]: pgmap v19547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:54:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36195.222204 secs 2024-04-26T21:54:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:14 smithi096 ceph-mon[32753]: pgmap v19547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:54:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36195.222204 secs 2024-04-26T21:54:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:54:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:54:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:54:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:54:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:54:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:54:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:54:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:54:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:54:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:16 smithi194 ceph-mon[33017]: pgmap v19548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:54:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:16 smithi194 ceph-mon[33017]: pgmap v19549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:16 smithi033 ceph-mon[27845]: pgmap v19548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:54:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:16 smithi033 ceph-mon[27845]: pgmap v19549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:16 smithi096 ceph-mon[32753]: pgmap v19548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:54:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:16 smithi096 ceph-mon[32753]: pgmap v19549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:18 smithi194 ceph-mon[33017]: pgmap v19550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36197.255186 secs 2024-04-26T21:54:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:18 smithi033 ceph-mon[27845]: pgmap v19550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36197.255186 secs 2024-04-26T21:54:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:18 smithi096 ceph-mon[32753]: pgmap v19550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36197.255186 secs 2024-04-26T21:54:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36200.222301 secs 2024-04-26T21:54:19.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36200.222301 secs 2024-04-26T21:54:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36200.222301 secs 2024-04-26T21:54:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:20 smithi194 ceph-mon[33017]: pgmap v19551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:20 smithi033 ceph-mon[27845]: pgmap v19551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:20 smithi096 ceph-mon[32753]: pgmap v19551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:22 smithi194 ceph-mon[33017]: pgmap v19552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:22 smithi033 ceph-mon[27845]: pgmap v19552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:22 smithi096 ceph-mon[32753]: pgmap v19552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36202.255273 secs 2024-04-26T21:54:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36202.255273 secs 2024-04-26T21:54:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36202.255273 secs 2024-04-26T21:54:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:24 smithi194 ceph-mon[33017]: pgmap v19553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36205.222418 secs 2024-04-26T21:54:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:24 smithi033 ceph-mon[27845]: pgmap v19553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36205.222418 secs 2024-04-26T21:54:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:24 smithi096 ceph-mon[32753]: pgmap v19553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:54:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36205.222418 secs 2024-04-26T21:54:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:26 smithi194 ceph-mon[33017]: pgmap v19554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:54:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:26 smithi033 ceph-mon[27845]: pgmap v19554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:54:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:26 smithi096 ceph-mon[32753]: pgmap v19554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:54:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:28 smithi033 ceph-mon[27845]: pgmap v19555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36207.255377 secs 2024-04-26T21:54:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:28 smithi096 ceph-mon[32753]: pgmap v19555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36207.255377 secs 2024-04-26T21:54:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:28 smithi194 ceph-mon[33017]: pgmap v19555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36207.255377 secs 2024-04-26T21:54:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:29.386+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:29.386+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:29.386+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:54:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36210.222486 secs 2024-04-26T21:54:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36210.222486 secs 2024-04-26T21:54:30.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36210.222486 secs 2024-04-26T21:54:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:30 smithi033 ceph-mon[27845]: pgmap v19556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:30 smithi096 ceph-mon[32753]: pgmap v19556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:30 smithi194 ceph-mon[33017]: pgmap v19556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:32 smithi033 ceph-mon[27845]: pgmap v19557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:32 smithi096 ceph-mon[32753]: pgmap v19557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:32 smithi194 ceph-mon[33017]: pgmap v19557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36212.255480 secs 2024-04-26T21:54:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36212.255480 secs 2024-04-26T21:54:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36212.255480 secs 2024-04-26T21:54:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:34 smithi033 ceph-mon[27845]: pgmap v19558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36215.222610 secs 2024-04-26T21:54:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:34 smithi096 ceph-mon[32753]: pgmap v19558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36215.222610 secs 2024-04-26T21:54:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:34 smithi194 ceph-mon[33017]: pgmap v19558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36215.222610 secs 2024-04-26T21:54:36.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:36 smithi033 ceph-mon[27845]: pgmap v19559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:36 smithi096 ceph-mon[32753]: pgmap v19559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:36 smithi194 ceph-mon[33017]: pgmap v19559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:38 smithi033 ceph-mon[27845]: pgmap v19560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36217.255584 secs 2024-04-26T21:54:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:38 smithi096 ceph-mon[32753]: pgmap v19560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36217.255584 secs 2024-04-26T21:54:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:38 smithi194 ceph-mon[33017]: pgmap v19560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36217.255584 secs 2024-04-26T21:54:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36220.222682 secs 2024-04-26T21:54:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36220.222682 secs 2024-04-26T21:54:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36220.222682 secs 2024-04-26T21:54:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:40 smithi033 ceph-mon[27845]: pgmap v19561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:40 smithi096 ceph-mon[32753]: pgmap v19561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:40 smithi194 ceph-mon[33017]: pgmap v19561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:42 smithi096 ceph-mon[32753]: pgmap v19562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:42 smithi194 ceph-mon[33017]: pgmap v19562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:42 smithi033 ceph-mon[27845]: pgmap v19562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36222.255686 secs 2024-04-26T21:54:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36222.255686 secs 2024-04-26T21:54:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36222.255686 secs 2024-04-26T21:54:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:44.397+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:44.397+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:54:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:44.397+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:44 smithi194 ceph-mon[33017]: pgmap v19563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36225.222817 secs 2024-04-26T21:54:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:44 smithi033 ceph-mon[27845]: pgmap v19563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36225.222817 secs 2024-04-26T21:54:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:44 smithi096 ceph-mon[32753]: pgmap v19563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36225.222817 secs 2024-04-26T21:54:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:46 smithi194 ceph-mon[33017]: pgmap v19564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:46 smithi033 ceph-mon[27845]: pgmap v19564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:46 smithi096 ceph-mon[32753]: pgmap v19564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:48 smithi194 ceph-mon[33017]: pgmap v19565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36227.255785 secs 2024-04-26T21:54:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:48 smithi033 ceph-mon[27845]: pgmap v19565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36227.255785 secs 2024-04-26T21:54:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:48 smithi096 ceph-mon[32753]: pgmap v19565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36227.255785 secs 2024-04-26T21:54:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36230.222968 secs 2024-04-26T21:54:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36230.222968 secs 2024-04-26T21:54:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36230.222968 secs 2024-04-26T21:54:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:50 smithi194 ceph-mon[33017]: pgmap v19566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:50 smithi033 ceph-mon[27845]: pgmap v19566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:50 smithi096 ceph-mon[32753]: pgmap v19566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:52.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:52 smithi033 ceph-mon[27845]: pgmap v19567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:52 smithi194 ceph-mon[33017]: pgmap v19567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:52 smithi096 ceph-mon[32753]: pgmap v19567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36232.255888 secs 2024-04-26T21:54:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36232.255888 secs 2024-04-26T21:54:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36232.255888 secs 2024-04-26T21:54:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:54 smithi194 ceph-mon[33017]: pgmap v19568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36235.223123 secs 2024-04-26T21:54:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:54 smithi033 ceph-mon[27845]: pgmap v19568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36235.223123 secs 2024-04-26T21:54:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:54 smithi096 ceph-mon[32753]: pgmap v19568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36235.223123 secs 2024-04-26T21:54:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:56 smithi194 ceph-mon[33017]: pgmap v19569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:56 smithi033 ceph-mon[27845]: pgmap v19569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:56 smithi096 ceph-mon[32753]: pgmap v19569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:54:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:58 smithi194 ceph-mon[33017]: pgmap v19570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36237.255993 secs 2024-04-26T21:54:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:58 smithi033 ceph-mon[27845]: pgmap v19570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36237.255993 secs 2024-04-26T21:54:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:58 smithi096 ceph-mon[32753]: pgmap v19570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:54:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36237.255993 secs 2024-04-26T21:54:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:59.401+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:59.401+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:54:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:54:59.401+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:55:00.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36240.223176 secs 2024-04-26T21:55:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36240.223176 secs 2024-04-26T21:55:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36240.223176 secs 2024-04-26T21:55:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:00 smithi194 ceph-mon[33017]: pgmap v19571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:00 smithi033 ceph-mon[27845]: pgmap v19571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:00 smithi096 ceph-mon[32753]: pgmap v19571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:02 smithi194 ceph-mon[33017]: pgmap v19572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:02 smithi033 ceph-mon[27845]: pgmap v19572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:02 smithi096 ceph-mon[32753]: pgmap v19572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36242.256096 secs 2024-04-26T21:55:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36242.256096 secs 2024-04-26T21:55:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36242.256096 secs 2024-04-26T21:55:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:04 smithi194 ceph-mon[33017]: pgmap v19573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36245.223304 secs 2024-04-26T21:55:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:04 smithi033 ceph-mon[27845]: pgmap v19573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36245.223304 secs 2024-04-26T21:55:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:04 smithi096 ceph-mon[32753]: pgmap v19573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36245.223304 secs 2024-04-26T21:55:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:05 smithi194 ceph-mon[33017]: pgmap v19574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:05 smithi033 ceph-mon[27845]: pgmap v19574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:05 smithi096 ceph-mon[32753]: pgmap v19574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:08 smithi194 ceph-mon[33017]: pgmap v19575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:08 smithi033 ceph-mon[27845]: pgmap v19575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:08 smithi096 ceph-mon[32753]: pgmap v19575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36247.256191 secs 2024-04-26T21:55:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36247.256191 secs 2024-04-26T21:55:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36247.256191 secs 2024-04-26T21:55:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36250.223393 secs 2024-04-26T21:55:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:10 smithi194 ceph-mon[33017]: pgmap v19576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36250.223393 secs 2024-04-26T21:55:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:10 smithi033 ceph-mon[27845]: pgmap v19576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36250.223393 secs 2024-04-26T21:55:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:10 smithi096 ceph-mon[32753]: pgmap v19576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:12 smithi194 ceph-mon[33017]: pgmap v19577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:12 smithi033 ceph-mon[27845]: pgmap v19577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:12 smithi096 ceph-mon[32753]: pgmap v19577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:14 smithi194 ceph-mon[33017]: pgmap v19578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36252.256292 secs 2024-04-26T21:55:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:14.412+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:55:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:14.412+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:55:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:14.412+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:55:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:14 smithi033 ceph-mon[27845]: pgmap v19578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36252.256292 secs 2024-04-26T21:55:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:14 smithi096 ceph-mon[32753]: pgmap v19578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36252.256292 secs 2024-04-26T21:55:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36255.223504 secs 2024-04-26T21:55:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36255.223504 secs 2024-04-26T21:55:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36255.223504 secs 2024-04-26T21:55:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:16 smithi194 ceph-mon[33017]: pgmap v19579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:55:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:16 smithi033 ceph-mon[27845]: pgmap v19579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:55:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:16 smithi096 ceph-mon[32753]: pgmap v19579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:55:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:18 smithi194 ceph-mon[33017]: pgmap v19580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:18 smithi033 ceph-mon[27845]: pgmap v19580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:18 smithi096 ceph-mon[32753]: pgmap v19580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36257.256357 secs 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36257.256357 secs 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:55:19.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36257.256357 secs 2024-04-26T21:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:19.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36260.223651 secs 2024-04-26T21:55:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:20 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:55:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:20 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:55:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:20 smithi194 ceph-mon[33017]: pgmap v19581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:20 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36260.223651 secs 2024-04-26T21:55:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:20 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:55:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:20 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:55:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:20 smithi033 ceph-mon[27845]: pgmap v19581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:20 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36260.223651 secs 2024-04-26T21:55:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:20 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T21:55:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:20 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T21:55:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:20 smithi096 ceph-mon[32753]: pgmap v19581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:20 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: pgmap v19582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: pgmap v19583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:55:22.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: pgmap v19584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:22.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: pgmap v19582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:55:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:55:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:55:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:55:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: pgmap v19583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:55:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: pgmap v19584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:22.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: pgmap v19582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: pgmap v19583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: pgmap v19584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:22.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:55:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36262.256439 secs 2024-04-26T21:55:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36262.256439 secs 2024-04-26T21:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36262.256439 secs 2024-04-26T21:55:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:24 smithi194 ceph-mon[33017]: pgmap v19585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:55:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36265.223779 secs 2024-04-26T21:55:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:24 smithi033 ceph-mon[27845]: pgmap v19585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:55:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36265.223779 secs 2024-04-26T21:55:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:24 smithi096 ceph-mon[32753]: pgmap v19585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T21:55:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36265.223779 secs 2024-04-26T21:55:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:26 smithi194 ceph-mon[33017]: pgmap v19586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:26 smithi033 ceph-mon[27845]: pgmap v19586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:26 smithi096 ceph-mon[32753]: pgmap v19586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:28 smithi194 ceph-mon[33017]: pgmap v19587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36267.256541 secs 2024-04-26T21:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:28 smithi033 ceph-mon[27845]: pgmap v19587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36267.256541 secs 2024-04-26T21:55:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:28 smithi096 ceph-mon[32753]: pgmap v19587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:28.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36267.256541 secs 2024-04-26T21:55:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:29.428+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:55:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:29.428+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:55:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:29.428+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36270.223906 secs 2024-04-26T21:55:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36270.223906 secs 2024-04-26T21:55:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36270.223906 secs 2024-04-26T21:55:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:30 smithi033 ceph-mon[27845]: pgmap v19588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:30 smithi096 ceph-mon[32753]: pgmap v19588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:31.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:30 smithi194 ceph-mon[33017]: pgmap v19588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:55:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:32 smithi033 ceph-mon[27845]: pgmap v19589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:55:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:32 smithi096 ceph-mon[32753]: pgmap v19589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:55:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:32 smithi194 ceph-mon[33017]: pgmap v19589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36272.256641 secs 2024-04-26T21:55:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36272.256641 secs 2024-04-26T21:55:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36272.256641 secs 2024-04-26T21:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:34 smithi033 ceph-mon[27845]: pgmap v19590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36275.224046 secs 2024-04-26T21:55:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:34 smithi096 ceph-mon[32753]: pgmap v19590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36275.224046 secs 2024-04-26T21:55:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:34 smithi194 ceph-mon[33017]: pgmap v19590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36275.224046 secs 2024-04-26T21:55:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:36 smithi033 ceph-mon[27845]: pgmap v19591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:36 smithi096 ceph-mon[32753]: pgmap v19591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:36 smithi194 ceph-mon[33017]: pgmap v19591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:38 smithi033 ceph-mon[27845]: pgmap v19592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36277.256741 secs 2024-04-26T21:55:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:38 smithi096 ceph-mon[32753]: pgmap v19592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36277.256741 secs 2024-04-26T21:55:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:38 smithi194 ceph-mon[33017]: pgmap v19592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36277.256741 secs 2024-04-26T21:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36280.224167 secs 2024-04-26T21:55:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36280.224167 secs 2024-04-26T21:55:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36280.224167 secs 2024-04-26T21:55:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:40 smithi033 ceph-mon[27845]: pgmap v19593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:40 smithi096 ceph-mon[32753]: pgmap v19593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:40 smithi194 ceph-mon[33017]: pgmap v19593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:42 smithi033 ceph-mon[27845]: pgmap v19594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:42 smithi096 ceph-mon[32753]: pgmap v19594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:42 smithi194 ceph-mon[33017]: pgmap v19594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36282.256843 secs 2024-04-26T21:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36282.256843 secs 2024-04-26T21:55:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36282.256843 secs 2024-04-26T21:55:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:44.442+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:55:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:44.442+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:55:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:44.442+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:44 smithi033 ceph-mon[27845]: pgmap v19595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36285.224258 secs 2024-04-26T21:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:44 smithi096 ceph-mon[32753]: pgmap v19595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36285.224258 secs 2024-04-26T21:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:44 smithi194 ceph-mon[33017]: pgmap v19595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36285.224258 secs 2024-04-26T21:55:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:46 smithi033 ceph-mon[27845]: pgmap v19596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:46 smithi096 ceph-mon[32753]: pgmap v19596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:46 smithi194 ceph-mon[33017]: pgmap v19596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:48 smithi096 ceph-mon[32753]: pgmap v19597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36287.256945 secs 2024-04-26T21:55:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:48 smithi194 ceph-mon[33017]: pgmap v19597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36287.256945 secs 2024-04-26T21:55:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:48 smithi033 ceph-mon[27845]: pgmap v19597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36287.256945 secs 2024-04-26T21:55:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36290.224381 secs 2024-04-26T21:55:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36290.224381 secs 2024-04-26T21:55:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36290.224381 secs 2024-04-26T21:55:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:50 smithi194 ceph-mon[33017]: pgmap v19598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:50 smithi033 ceph-mon[27845]: pgmap v19598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:50 smithi096 ceph-mon[32753]: pgmap v19598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:52 smithi194 ceph-mon[33017]: pgmap v19599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:52 smithi033 ceph-mon[27845]: pgmap v19599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:52 smithi096 ceph-mon[32753]: pgmap v19599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36292.257046 secs 2024-04-26T21:55:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36292.257046 secs 2024-04-26T21:55:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36292.257046 secs 2024-04-26T21:55:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:54 smithi194 ceph-mon[33017]: pgmap v19600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36295.224509 secs 2024-04-26T21:55:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:54 smithi033 ceph-mon[27845]: pgmap v19600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36295.224509 secs 2024-04-26T21:55:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:54 smithi096 ceph-mon[32753]: pgmap v19600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36295.224509 secs 2024-04-26T21:55:56.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:56 smithi096 ceph-mon[32753]: pgmap v19601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:56 smithi194 ceph-mon[33017]: pgmap v19601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:56 smithi033 ceph-mon[27845]: pgmap v19601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:55:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:58 smithi194 ceph-mon[33017]: pgmap v19602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36297.257150 secs 2024-04-26T21:55:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:58 smithi033 ceph-mon[27845]: pgmap v19602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36297.257150 secs 2024-04-26T21:55:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:58 smithi096 ceph-mon[32753]: pgmap v19602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:55:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36297.257150 secs 2024-04-26T21:55:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:59.447+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:55:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:59.447+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:55:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:55:59.447+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:55:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36300.224634 secs 2024-04-26T21:56:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:55:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36300.224634 secs 2024-04-26T21:56:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:55:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36300.224634 secs 2024-04-26T21:56:01.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:00 smithi194 ceph-mon[33017]: pgmap v19603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:00 smithi033 ceph-mon[27845]: pgmap v19603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:00 smithi096 ceph-mon[32753]: pgmap v19603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:02 smithi194 ceph-mon[33017]: pgmap v19604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:02 smithi033 ceph-mon[27845]: pgmap v19604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:02 smithi096 ceph-mon[32753]: pgmap v19604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36302.257239 secs 2024-04-26T21:56:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36302.257239 secs 2024-04-26T21:56:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36302.257239 secs 2024-04-26T21:56:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:04 smithi194 ceph-mon[33017]: pgmap v19605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36305.224787 secs 2024-04-26T21:56:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:04 smithi033 ceph-mon[27845]: pgmap v19605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36305.224787 secs 2024-04-26T21:56:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:04 smithi096 ceph-mon[32753]: pgmap v19605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36305.224787 secs 2024-04-26T21:56:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:06 smithi194 ceph-mon[33017]: pgmap v19606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:06 smithi033 ceph-mon[27845]: pgmap v19606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:06 smithi096 ceph-mon[32753]: pgmap v19606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:08 smithi194 ceph-mon[33017]: pgmap v19607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36307.257340 secs 2024-04-26T21:56:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:08 smithi033 ceph-mon[27845]: pgmap v19607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36307.257340 secs 2024-04-26T21:56:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:08 smithi096 ceph-mon[32753]: pgmap v19607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36307.257340 secs 2024-04-26T21:56:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36310.224909 secs 2024-04-26T21:56:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36310.224909 secs 2024-04-26T21:56:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36310.224909 secs 2024-04-26T21:56:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:10 smithi194 ceph-mon[33017]: pgmap v19608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:10 smithi033 ceph-mon[27845]: pgmap v19608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:10 smithi096 ceph-mon[32753]: pgmap v19608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:12 smithi194 ceph-mon[33017]: pgmap v19609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:12 smithi033 ceph-mon[27845]: pgmap v19609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:12 smithi096 ceph-mon[32753]: pgmap v19609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36312.257427 secs 2024-04-26T21:56:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36312.257427 secs 2024-04-26T21:56:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36312.257427 secs 2024-04-26T21:56:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:14.462+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:14.462+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:14.462+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:56:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:14 smithi194 ceph-mon[33017]: pgmap v19610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36315.225005 secs 2024-04-26T21:56:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:14 smithi033 ceph-mon[27845]: pgmap v19610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36315.225005 secs 2024-04-26T21:56:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:14 smithi096 ceph-mon[32753]: pgmap v19610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36315.225005 secs 2024-04-26T21:56:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:15 smithi194 ceph-mon[33017]: pgmap v19611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:15 smithi033 ceph-mon[27845]: pgmap v19611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:15 smithi096 ceph-mon[32753]: pgmap v19611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:18 smithi033 ceph-mon[27845]: pgmap v19612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36317.257512 secs 2024-04-26T21:56:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:18 smithi096 ceph-mon[32753]: pgmap v19612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36317.257512 secs 2024-04-26T21:56:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:18 smithi194 ceph-mon[33017]: pgmap v19612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36317.257512 secs 2024-04-26T21:56:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36320.225149 secs 2024-04-26T21:56:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36320.225149 secs 2024-04-26T21:56:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36320.225149 secs 2024-04-26T21:56:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:20 smithi033 ceph-mon[27845]: pgmap v19613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:20 smithi096 ceph-mon[32753]: pgmap v19613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:20 smithi194 ceph-mon[33017]: pgmap v19613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T21:56:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:56:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0012040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:56:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0012040 comp rx=0 tx=0).stop 2024-04-26T21:56:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T21:56:21.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T21:56:21.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:56:21.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.819+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:56:21.821 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.820+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:56:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:21.821+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0022040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:56:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:22 smithi033 ceph-mon[27845]: pgmap v19614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:56:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:22 smithi096 ceph-mon[32753]: pgmap v19614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:56:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:22 smithi194 ceph-mon[33017]: pgmap v19614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:56:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36322.257599 secs 2024-04-26T21:56:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:56:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:56:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:56:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36322.257599 secs 2024-04-26T21:56:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:56:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:56:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:56:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36322.257599 secs 2024-04-26T21:56:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:56:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:56:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:56:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:24 smithi033 ceph-mon[27845]: pgmap v19615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:24 smithi033 ceph-mon[27845]: pgmap v19616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36325.225219 secs 2024-04-26T21:56:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:24 smithi096 ceph-mon[32753]: pgmap v19615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:24 smithi096 ceph-mon[32753]: pgmap v19616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36325.225219 secs 2024-04-26T21:56:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:24 smithi194 ceph-mon[33017]: pgmap v19615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:24 smithi194 ceph-mon[33017]: pgmap v19616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36325.225219 secs 2024-04-26T21:56:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:26 smithi033 ceph-mon[27845]: pgmap v19617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:26 smithi096 ceph-mon[32753]: pgmap v19617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:26 smithi194 ceph-mon[33017]: pgmap v19617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:27.982+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:27.982+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:27.982+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 284 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T21:56:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:27.983+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 284 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T21:56:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:28 smithi033 ceph-mon[27845]: pgmap v19618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36327.257685 secs 2024-04-26T21:56:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:28 smithi096 ceph-mon[32753]: pgmap v19618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36327.257685 secs 2024-04-26T21:56:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:28 smithi194 ceph-mon[33017]: pgmap v19618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36327.257685 secs 2024-04-26T21:56:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:29.471+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:29.472+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:29.472+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:56:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36330.225284 secs 2024-04-26T21:56:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36330.225284 secs 2024-04-26T21:56:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36330.225284 secs 2024-04-26T21:56:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:30 smithi033 ceph-mon[27845]: pgmap v19619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:56:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:30 smithi096 ceph-mon[32753]: pgmap v19619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:56:31.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:30 smithi194 ceph-mon[33017]: pgmap v19619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T21:56:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:32 smithi096 ceph-mon[32753]: pgmap v19620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:32 smithi194 ceph-mon[33017]: pgmap v19620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:32 smithi033 ceph-mon[27845]: pgmap v19620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:56:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36332.257771 secs 2024-04-26T21:56:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36332.257771 secs 2024-04-26T21:56:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36332.257771 secs 2024-04-26T21:56:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:34 smithi194 ceph-mon[33017]: pgmap v19621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:56:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36335.225354 secs 2024-04-26T21:56:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:34 smithi033 ceph-mon[27845]: pgmap v19621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:56:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36335.225354 secs 2024-04-26T21:56:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:34 smithi096 ceph-mon[32753]: pgmap v19621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:56:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36335.225354 secs 2024-04-26T21:56:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:36 smithi194 ceph-mon[33017]: pgmap v19622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:36 smithi033 ceph-mon[27845]: pgmap v19622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:36 smithi096 ceph-mon[32753]: pgmap v19622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:38 smithi194 ceph-mon[33017]: pgmap v19623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36337.257861 secs 2024-04-26T21:56:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:38 smithi033 ceph-mon[27845]: pgmap v19623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36337.257861 secs 2024-04-26T21:56:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:38 smithi096 ceph-mon[32753]: pgmap v19623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36337.257861 secs 2024-04-26T21:56:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36340.225517 secs 2024-04-26T21:56:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36340.225517 secs 2024-04-26T21:56:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36340.225517 secs 2024-04-26T21:56:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:40 smithi194 ceph-mon[33017]: pgmap v19624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:40 smithi033 ceph-mon[27845]: pgmap v19624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:40 smithi096 ceph-mon[32753]: pgmap v19624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:42 smithi194 ceph-mon[33017]: pgmap v19625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:42 smithi033 ceph-mon[27845]: pgmap v19625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:42 smithi096 ceph-mon[32753]: pgmap v19625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36342.257947 secs 2024-04-26T21:56:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36342.257947 secs 2024-04-26T21:56:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36342.257947 secs 2024-04-26T21:56:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:44.486+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:44.486+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:44.486+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:56:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:44 smithi194 ceph-mon[33017]: pgmap v19626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36345.225606 secs 2024-04-26T21:56:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:44 smithi033 ceph-mon[27845]: pgmap v19626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36345.225606 secs 2024-04-26T21:56:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:44 smithi096 ceph-mon[32753]: pgmap v19626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36345.225606 secs 2024-04-26T21:56:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:46 smithi194 ceph-mon[33017]: pgmap v19627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:46 smithi033 ceph-mon[27845]: pgmap v19627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:46 smithi096 ceph-mon[32753]: pgmap v19627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:48 smithi194 ceph-mon[33017]: pgmap v19628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36347.258033 secs 2024-04-26T21:56:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:48 smithi033 ceph-mon[27845]: pgmap v19628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36347.258033 secs 2024-04-26T21:56:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:48 smithi096 ceph-mon[32753]: pgmap v19628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36347.258033 secs 2024-04-26T21:56:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36350.225735 secs 2024-04-26T21:56:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36350.225735 secs 2024-04-26T21:56:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36350.225735 secs 2024-04-26T21:56:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:50 smithi194 ceph-mon[33017]: pgmap v19629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:50 smithi033 ceph-mon[27845]: pgmap v19629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:50 smithi096 ceph-mon[32753]: pgmap v19629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:52 smithi194 ceph-mon[33017]: pgmap v19630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:52 smithi033 ceph-mon[27845]: pgmap v19630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:52 smithi096 ceph-mon[32753]: pgmap v19630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36352.258118 secs 2024-04-26T21:56:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36352.258118 secs 2024-04-26T21:56:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36352.258118 secs 2024-04-26T21:56:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:54 smithi194 ceph-mon[33017]: pgmap v19631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36355.225865 secs 2024-04-26T21:56:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:54 smithi033 ceph-mon[27845]: pgmap v19631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36355.225865 secs 2024-04-26T21:56:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:54 smithi096 ceph-mon[32753]: pgmap v19631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36355.225865 secs 2024-04-26T21:56:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:56 smithi194 ceph-mon[33017]: pgmap v19632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:56 smithi033 ceph-mon[27845]: pgmap v19632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:56 smithi096 ceph-mon[32753]: pgmap v19632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:56:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:58 smithi194 ceph-mon[33017]: pgmap v19633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36357.258179 secs 2024-04-26T21:56:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:58 smithi033 ceph-mon[27845]: pgmap v19633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36357.258179 secs 2024-04-26T21:56:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:58 smithi096 ceph-mon[32753]: pgmap v19633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:56:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36357.258179 secs 2024-04-26T21:56:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:59.495+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:59.495+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:56:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:56:59.495+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:57:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:56:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36360.226011 secs 2024-04-26T21:57:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:56:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36360.226011 secs 2024-04-26T21:57:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:56:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36360.226011 secs 2024-04-26T21:57:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:00 smithi194 ceph-mon[33017]: pgmap v19634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:00 smithi033 ceph-mon[27845]: pgmap v19634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:00 smithi096 ceph-mon[32753]: pgmap v19634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:02 smithi194 ceph-mon[33017]: pgmap v19635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:02 smithi033 ceph-mon[27845]: pgmap v19635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:02 smithi096 ceph-mon[32753]: pgmap v19635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36362.258239 secs 2024-04-26T21:57:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:03 smithi194 ceph-mon[33017]: pgmap v19636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36362.258239 secs 2024-04-26T21:57:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:03 smithi033 ceph-mon[27845]: pgmap v19636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36362.258239 secs 2024-04-26T21:57:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:03 smithi096 ceph-mon[32753]: pgmap v19636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36365.226167 secs 2024-04-26T21:57:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36365.226167 secs 2024-04-26T21:57:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36365.226167 secs 2024-04-26T21:57:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:05 smithi194 ceph-mon[33017]: pgmap v19637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:05 smithi033 ceph-mon[27845]: pgmap v19637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:05 smithi096 ceph-mon[32753]: pgmap v19637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:08 smithi194 ceph-mon[33017]: pgmap v19638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:08 smithi033 ceph-mon[27845]: pgmap v19638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:08 smithi096 ceph-mon[32753]: pgmap v19638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36367.258301 secs 2024-04-26T21:57:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36367.258301 secs 2024-04-26T21:57:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36367.258301 secs 2024-04-26T21:57:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36370.226287 secs 2024-04-26T21:57:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:10 smithi194 ceph-mon[33017]: pgmap v19639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36370.226287 secs 2024-04-26T21:57:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:10 smithi033 ceph-mon[27845]: pgmap v19639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36370.226287 secs 2024-04-26T21:57:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:10 smithi096 ceph-mon[32753]: pgmap v19639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:12 smithi194 ceph-mon[33017]: pgmap v19640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:12 smithi033 ceph-mon[27845]: pgmap v19640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:12 smithi096 ceph-mon[32753]: pgmap v19640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:14.501+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:14.501+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:14.501+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36372.258385 secs 2024-04-26T21:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:14 smithi194 ceph-mon[33017]: pgmap v19641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36372.258385 secs 2024-04-26T21:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:14 smithi033 ceph-mon[27845]: pgmap v19641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36372.258385 secs 2024-04-26T21:57:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:14 smithi096 ceph-mon[32753]: pgmap v19641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36375.226410 secs 2024-04-26T21:57:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36375.226410 secs 2024-04-26T21:57:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36375.226410 secs 2024-04-26T21:57:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:16 smithi033 ceph-mon[27845]: pgmap v19642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:16 smithi096 ceph-mon[32753]: pgmap v19642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:16 smithi194 ceph-mon[33017]: pgmap v19642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:18 smithi033 ceph-mon[27845]: pgmap v19643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:18 smithi096 ceph-mon[32753]: pgmap v19643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:18 smithi194 ceph-mon[33017]: pgmap v19643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36377.258469 secs 2024-04-26T21:57:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36377.258469 secs 2024-04-26T21:57:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36377.258469 secs 2024-04-26T21:57:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36380.226531 secs 2024-04-26T21:57:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:20 smithi033 ceph-mon[27845]: pgmap v19644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36380.226531 secs 2024-04-26T21:57:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:20 smithi096 ceph-mon[32753]: pgmap v19644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36380.226531 secs 2024-04-26T21:57:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:20 smithi194 ceph-mon[33017]: pgmap v19644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:22.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:22 smithi033 ceph-mon[27845]: pgmap v19645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:22 smithi096 ceph-mon[32753]: pgmap v19645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:22 smithi194 ceph-mon[33017]: pgmap v19645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36382.258553 secs 2024-04-26T21:57:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36382.258553 secs 2024-04-26T21:57:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36382.258553 secs 2024-04-26T21:57:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:24 smithi033 ceph-mon[27845]: pgmap v19646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:57:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:24 smithi096 ceph-mon[32753]: pgmap v19646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:57:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:24 smithi194 ceph-mon[33017]: pgmap v19646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:57:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36385.226626 secs 2024-04-26T21:57:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:57:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:57:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:57:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36385.226626 secs 2024-04-26T21:57:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:57:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:57:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:57:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36385.226626 secs 2024-04-26T21:57:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:57:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:57:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:57:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:26 smithi033 ceph-mon[27845]: pgmap v19647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:57:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:26 smithi033 ceph-mon[27845]: pgmap v19648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:26 smithi096 ceph-mon[32753]: pgmap v19647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:57:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:26 smithi096 ceph-mon[32753]: pgmap v19648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:26 smithi194 ceph-mon[33017]: pgmap v19647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T21:57:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:26 smithi194 ceph-mon[33017]: pgmap v19648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:28 smithi033 ceph-mon[27845]: pgmap v19649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36387.258638 secs 2024-04-26T21:57:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:28 smithi096 ceph-mon[32753]: pgmap v19649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36387.258638 secs 2024-04-26T21:57:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:28 smithi194 ceph-mon[33017]: pgmap v19649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36387.258638 secs 2024-04-26T21:57:29.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:29.512+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:29.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:29.512+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:57:29.513 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:29.512+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36390.226747 secs 2024-04-26T21:57:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:30 smithi033 ceph-mon[27845]: pgmap v19650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36390.226747 secs 2024-04-26T21:57:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:30 smithi096 ceph-mon[32753]: pgmap v19650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36390.226747 secs 2024-04-26T21:57:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:30 smithi194 ceph-mon[33017]: pgmap v19650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:32 smithi096 ceph-mon[32753]: pgmap v19651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:32 smithi194 ceph-mon[33017]: pgmap v19651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:32 smithi033 ceph-mon[27845]: pgmap v19651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36392.258715 secs 2024-04-26T21:57:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36392.258715 secs 2024-04-26T21:57:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36392.258715 secs 2024-04-26T21:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:34 smithi194 ceph-mon[33017]: pgmap v19652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36395.226877 secs 2024-04-26T21:57:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:34 smithi033 ceph-mon[27845]: pgmap v19652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36395.226877 secs 2024-04-26T21:57:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:34 smithi096 ceph-mon[32753]: pgmap v19652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:57:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36395.226877 secs 2024-04-26T21:57:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:36 smithi194 ceph-mon[33017]: pgmap v19653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:57:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:36 smithi033 ceph-mon[27845]: pgmap v19653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:57:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:36 smithi096 ceph-mon[32753]: pgmap v19653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:57:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:38 smithi194 ceph-mon[33017]: pgmap v19654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36397.258796 secs 2024-04-26T21:57:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:38 smithi033 ceph-mon[27845]: pgmap v19654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36397.258796 secs 2024-04-26T21:57:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:38 smithi096 ceph-mon[32753]: pgmap v19654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36397.258796 secs 2024-04-26T21:57:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36400.227000 secs 2024-04-26T21:57:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36400.227000 secs 2024-04-26T21:57:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36400.227000 secs 2024-04-26T21:57:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:40 smithi194 ceph-mon[33017]: pgmap v19655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:40 smithi033 ceph-mon[27845]: pgmap v19655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:40 smithi096 ceph-mon[32753]: pgmap v19655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:42 smithi194 ceph-mon[33017]: pgmap v19656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:43.245 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:42 smithi096 ceph-mon[32753]: pgmap v19656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:43.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:42 smithi033 ceph-mon[27845]: pgmap v19656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:43.483 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36402.258857 secs 2024-04-26T21:57:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36402.258857 secs 2024-04-26T21:57:43.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36402.258857 secs 2024-04-26T21:57:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:44.527+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:44.527+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:57:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:44.527+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:44 smithi194 ceph-mon[33017]: pgmap v19657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36405.227159 secs 2024-04-26T21:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:44 smithi033 ceph-mon[27845]: pgmap v19657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36405.227159 secs 2024-04-26T21:57:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:44 smithi096 ceph-mon[32753]: pgmap v19657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36405.227159 secs 2024-04-26T21:57:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:46 smithi194 ceph-mon[33017]: pgmap v19658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:46 smithi033 ceph-mon[27845]: pgmap v19658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:46 smithi096 ceph-mon[32753]: pgmap v19658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:48 smithi194 ceph-mon[33017]: pgmap v19659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36407.258943 secs 2024-04-26T21:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:48 smithi033 ceph-mon[27845]: pgmap v19659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36407.258943 secs 2024-04-26T21:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:48 smithi096 ceph-mon[32753]: pgmap v19659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36407.258943 secs 2024-04-26T21:57:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36410.227280 secs 2024-04-26T21:57:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36410.227280 secs 2024-04-26T21:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36410.227280 secs 2024-04-26T21:57:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:50 smithi194 ceph-mon[33017]: pgmap v19660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:50 smithi033 ceph-mon[27845]: pgmap v19660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:50 smithi096 ceph-mon[32753]: pgmap v19660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:52 smithi194 ceph-mon[33017]: pgmap v19661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:52 smithi033 ceph-mon[27845]: pgmap v19661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:52 smithi096 ceph-mon[32753]: pgmap v19661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36412.259028 secs 2024-04-26T21:57:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36412.259028 secs 2024-04-26T21:57:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36412.259028 secs 2024-04-26T21:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:54 smithi194 ceph-mon[33017]: pgmap v19662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36415.227363 secs 2024-04-26T21:57:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:54 smithi033 ceph-mon[27845]: pgmap v19662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36415.227363 secs 2024-04-26T21:57:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:54 smithi096 ceph-mon[32753]: pgmap v19662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36415.227363 secs 2024-04-26T21:57:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:56 smithi194 ceph-mon[33017]: pgmap v19663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:56 smithi033 ceph-mon[27845]: pgmap v19663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:56 smithi096 ceph-mon[32753]: pgmap v19663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:57:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:58 smithi194 ceph-mon[33017]: pgmap v19664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36417.259091 secs 2024-04-26T21:57:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:58 smithi033 ceph-mon[27845]: pgmap v19664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36417.259091 secs 2024-04-26T21:57:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:58 smithi096 ceph-mon[32753]: pgmap v19664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:57:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36417.259091 secs 2024-04-26T21:57:59.532 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:59.533+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:59.533+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:57:59.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:57:59.533+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:57:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36420.227482 secs 2024-04-26T21:57:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36420.227482 secs 2024-04-26T21:57:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36420.227482 secs 2024-04-26T21:58:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:00 smithi194 ceph-mon[33017]: pgmap v19665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:00 smithi033 ceph-mon[27845]: pgmap v19665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:00 smithi096 ceph-mon[32753]: pgmap v19665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:02 smithi194 ceph-mon[33017]: pgmap v19666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:02 smithi033 ceph-mon[27845]: pgmap v19666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:02 smithi096 ceph-mon[32753]: pgmap v19666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36422.259176 secs 2024-04-26T21:58:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36422.259176 secs 2024-04-26T21:58:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36422.259176 secs 2024-04-26T21:58:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:04 smithi194 ceph-mon[33017]: pgmap v19667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36425.227569 secs 2024-04-26T21:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:04 smithi033 ceph-mon[27845]: pgmap v19667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36425.227569 secs 2024-04-26T21:58:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:04 smithi096 ceph-mon[32753]: pgmap v19667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36425.227569 secs 2024-04-26T21:58:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:06 smithi194 ceph-mon[33017]: pgmap v19668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:06 smithi033 ceph-mon[27845]: pgmap v19668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:06 smithi096 ceph-mon[32753]: pgmap v19668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:08 smithi194 ceph-mon[33017]: pgmap v19669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36427.259238 secs 2024-04-26T21:58:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:08 smithi033 ceph-mon[27845]: pgmap v19669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36427.259238 secs 2024-04-26T21:58:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:08 smithi096 ceph-mon[32753]: pgmap v19669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36427.259238 secs 2024-04-26T21:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36430.227695 secs 2024-04-26T21:58:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36430.227695 secs 2024-04-26T21:58:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36430.227695 secs 2024-04-26T21:58:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:10 smithi033 ceph-mon[27845]: pgmap v19670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:10 smithi096 ceph-mon[32753]: pgmap v19670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:10 smithi194 ceph-mon[33017]: pgmap v19670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:12 smithi033 ceph-mon[27845]: pgmap v19671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:12 smithi096 ceph-mon[32753]: pgmap v19671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:12 smithi194 ceph-mon[33017]: pgmap v19671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36432.259299 secs 2024-04-26T21:58:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36432.259299 secs 2024-04-26T21:58:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36432.259299 secs 2024-04-26T21:58:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:14.536+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:14.536+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:14.536+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:58:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:14 smithi033 ceph-mon[27845]: pgmap v19672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36435.227820 secs 2024-04-26T21:58:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:14 smithi096 ceph-mon[32753]: pgmap v19672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36435.227820 secs 2024-04-26T21:58:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:14 smithi194 ceph-mon[33017]: pgmap v19672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36435.227820 secs 2024-04-26T21:58:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:16 smithi033 ceph-mon[27845]: pgmap v19673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:16 smithi096 ceph-mon[32753]: pgmap v19673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:16 smithi194 ceph-mon[33017]: pgmap v19673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:18 smithi033 ceph-mon[27845]: pgmap v19674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36437.259358 secs 2024-04-26T21:58:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:18 smithi096 ceph-mon[32753]: pgmap v19674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36437.259358 secs 2024-04-26T21:58:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:18 smithi194 ceph-mon[33017]: pgmap v19674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36437.259358 secs 2024-04-26T21:58:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36440.227960 secs 2024-04-26T21:58:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36440.227960 secs 2024-04-26T21:58:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36440.227960 secs 2024-04-26T21:58:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:20 smithi033 ceph-mon[27845]: pgmap v19675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:20 smithi096 ceph-mon[32753]: pgmap v19675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:20 smithi194 ceph-mon[33017]: pgmap v19675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:22 smithi033 ceph-mon[27845]: pgmap v19676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:22 smithi096 ceph-mon[32753]: pgmap v19676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:22 smithi194 ceph-mon[33017]: pgmap v19676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36442.259418 secs 2024-04-26T21:58:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36442.259418 secs 2024-04-26T21:58:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36442.259418 secs 2024-04-26T21:58:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:24 smithi033 ceph-mon[27845]: pgmap v19677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36445.228089 secs 2024-04-26T21:58:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:24 smithi096 ceph-mon[32753]: pgmap v19677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36445.228089 secs 2024-04-26T21:58:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:24 smithi194 ceph-mon[33017]: pgmap v19677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T21:58:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36445.228089 secs 2024-04-26T21:58:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:58:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:58:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:58:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:26 smithi096 ceph-mon[32753]: pgmap v19678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:58:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:58:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:58:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:26 smithi194 ceph-mon[33017]: pgmap v19678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:58:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:58:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:58:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:26 smithi033 ceph-mon[27845]: pgmap v19678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:58:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:58:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:58:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:58:28.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:27 smithi096 ceph-mon[32753]: pgmap v19679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:28.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:27 smithi096 ceph-mon[32753]: pgmap v19680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:27 smithi194 ceph-mon[33017]: pgmap v19679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:27 smithi194 ceph-mon[33017]: pgmap v19680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:27 smithi033 ceph-mon[27845]: pgmap v19679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:27 smithi033 ceph-mon[27845]: pgmap v19680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36447.259506 secs 2024-04-26T21:58:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36447.259506 secs 2024-04-26T21:58:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36447.259506 secs 2024-04-26T21:58:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:29.549+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:29.549+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:29.549+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:58:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:29 smithi194 ceph-mon[33017]: pgmap v19681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36450.228208 secs 2024-04-26T21:58:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:29 smithi033 ceph-mon[27845]: pgmap v19681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36450.228208 secs 2024-04-26T21:58:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:29 smithi096 ceph-mon[32753]: pgmap v19681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36450.228208 secs 2024-04-26T21:58:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:31 smithi194 ceph-mon[33017]: pgmap v19682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:31 smithi033 ceph-mon[27845]: pgmap v19682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:31 smithi096 ceph-mon[32753]: pgmap v19682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:33 smithi194 ceph-mon[33017]: pgmap v19683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36452.259596 secs 2024-04-26T21:58:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:33 smithi033 ceph-mon[27845]: pgmap v19683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36452.259596 secs 2024-04-26T21:58:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:33 smithi096 ceph-mon[32753]: pgmap v19683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36452.259596 secs 2024-04-26T21:58:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36455.228336 secs 2024-04-26T21:58:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36455.228336 secs 2024-04-26T21:58:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36455.228336 secs 2024-04-26T21:58:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:35 smithi194 ceph-mon[33017]: pgmap v19684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:35 smithi033 ceph-mon[27845]: pgmap v19684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:35 smithi096 ceph-mon[32753]: pgmap v19684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:58:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:37 smithi096 ceph-mon[32753]: pgmap v19685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:58:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:37 smithi194 ceph-mon[33017]: pgmap v19685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:58:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:37 smithi033 ceph-mon[27845]: pgmap v19685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:58:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36457.259682 secs 2024-04-26T21:58:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36457.259682 secs 2024-04-26T21:58:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36457.259682 secs 2024-04-26T21:58:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:39 smithi194 ceph-mon[33017]: pgmap v19686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36460.228454 secs 2024-04-26T21:58:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:39 smithi033 ceph-mon[27845]: pgmap v19686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36460.228454 secs 2024-04-26T21:58:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:39 smithi096 ceph-mon[32753]: pgmap v19686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36460.228454 secs 2024-04-26T21:58:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:41 smithi194 ceph-mon[33017]: pgmap v19687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:41 smithi033 ceph-mon[27845]: pgmap v19687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:41 smithi096 ceph-mon[32753]: pgmap v19687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:43 smithi194 ceph-mon[33017]: pgmap v19688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:44.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36462.259765 secs 2024-04-26T21:58:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:43 smithi033 ceph-mon[27845]: pgmap v19688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36462.259765 secs 2024-04-26T21:58:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:43 smithi096 ceph-mon[32753]: pgmap v19688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36462.259765 secs 2024-04-26T21:58:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:44.560+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:44.560+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:44.560+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:58:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36465.228524 secs 2024-04-26T21:58:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36465.228524 secs 2024-04-26T21:58:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36465.228524 secs 2024-04-26T21:58:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:45 smithi194 ceph-mon[33017]: pgmap v19689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:45 smithi033 ceph-mon[27845]: pgmap v19689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:45 smithi096 ceph-mon[32753]: pgmap v19689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:47 smithi194 ceph-mon[33017]: pgmap v19690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:47 smithi033 ceph-mon[27845]: pgmap v19690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:48.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:47 smithi096 ceph-mon[32753]: pgmap v19690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36467.259825 secs 2024-04-26T21:58:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36467.259825 secs 2024-04-26T21:58:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36467.259825 secs 2024-04-26T21:58:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:49 smithi194 ceph-mon[33017]: pgmap v19691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36470.228647 secs 2024-04-26T21:58:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:49 smithi033 ceph-mon[27845]: pgmap v19691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36470.228647 secs 2024-04-26T21:58:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:49 smithi096 ceph-mon[32753]: pgmap v19691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36470.228647 secs 2024-04-26T21:58:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:51 smithi194 ceph-mon[33017]: pgmap v19692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:51 smithi033 ceph-mon[27845]: pgmap v19692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:51 smithi096 ceph-mon[32753]: pgmap v19692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:53 smithi194 ceph-mon[33017]: pgmap v19693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36472.259938 secs 2024-04-26T21:58:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:53 smithi033 ceph-mon[27845]: pgmap v19693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36472.259938 secs 2024-04-26T21:58:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:53 smithi096 ceph-mon[32753]: pgmap v19693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36472.259938 secs 2024-04-26T21:58:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36475.228717 secs 2024-04-26T21:58:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36475.228717 secs 2024-04-26T21:58:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36475.228717 secs 2024-04-26T21:58:56.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:55 smithi033 ceph-mon[27845]: pgmap v19694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:56.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:55 smithi096 ceph-mon[32753]: pgmap v19694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:55 smithi194 ceph-mon[33017]: pgmap v19694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:58:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:57 smithi194 ceph-mon[33017]: pgmap v19695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:57 smithi033 ceph-mon[27845]: pgmap v19695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:57 smithi096 ceph-mon[32753]: pgmap v19695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:58:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36477.260007 secs 2024-04-26T21:58:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36477.260007 secs 2024-04-26T21:58:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36477.260007 secs 2024-04-26T21:58:59.573 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:59.573+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:59.573+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:58:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:58:59.573+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:59:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:59 smithi194 ceph-mon[33017]: pgmap v19696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36480.228845 secs 2024-04-26T21:59:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:59 smithi033 ceph-mon[27845]: pgmap v19696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36480.228845 secs 2024-04-26T21:59:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:59 smithi096 ceph-mon[32753]: pgmap v19696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36480.228845 secs 2024-04-26T21:59:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:00 smithi194 ceph-mon[33017]: pgmap v19697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:00 smithi033 ceph-mon[27845]: pgmap v19697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:00 smithi096 ceph-mon[32753]: pgmap v19697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:03 smithi194 ceph-mon[33017]: pgmap v19698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36482.260090 secs 2024-04-26T21:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:03 smithi033 ceph-mon[27845]: pgmap v19698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36482.260090 secs 2024-04-26T21:59:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:03 smithi096 ceph-mon[32753]: pgmap v19698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36482.260090 secs 2024-04-26T21:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36485.228922 secs 2024-04-26T21:59:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36485.228922 secs 2024-04-26T21:59:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36485.228922 secs 2024-04-26T21:59:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:05 smithi194 ceph-mon[33017]: pgmap v19699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:05 smithi033 ceph-mon[27845]: pgmap v19699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:05 smithi096 ceph-mon[32753]: pgmap v19699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:06 smithi194 ceph-mon[33017]: pgmap v19700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:06 smithi033 ceph-mon[27845]: pgmap v19700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:06 smithi096 ceph-mon[32753]: pgmap v19700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36487.260174 secs 2024-04-26T21:59:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36487.260174 secs 2024-04-26T21:59:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36487.260174 secs 2024-04-26T21:59:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:09 smithi194 ceph-mon[33017]: pgmap v19701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:09 smithi033 ceph-mon[27845]: pgmap v19701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:09 smithi096 ceph-mon[32753]: pgmap v19701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36490.229038 secs 2024-04-26T21:59:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36490.229038 secs 2024-04-26T21:59:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36490.229038 secs 2024-04-26T21:59:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:11 smithi194 ceph-mon[33017]: pgmap v19702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:11 smithi033 ceph-mon[27845]: pgmap v19702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:11 smithi096 ceph-mon[32753]: pgmap v19702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:13 smithi194 ceph-mon[33017]: pgmap v19703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36492.260255 secs 2024-04-26T21:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:13 smithi033 ceph-mon[27845]: pgmap v19703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36492.260255 secs 2024-04-26T21:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:13 smithi096 ceph-mon[32753]: pgmap v19703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36492.260255 secs 2024-04-26T21:59:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:14.580+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:14.580+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:14.580+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:59:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36495.229158 secs 2024-04-26T21:59:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36495.229158 secs 2024-04-26T21:59:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36495.229158 secs 2024-04-26T21:59:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:15 smithi194 ceph-mon[33017]: pgmap v19704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:15 smithi033 ceph-mon[27845]: pgmap v19704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:15 smithi096 ceph-mon[32753]: pgmap v19704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:18.063 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:17 smithi033 ceph-mon[27845]: pgmap v19705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:17 smithi194 ceph-mon[33017]: pgmap v19705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:17 smithi096 ceph-mon[32753]: pgmap v19705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36497.260341 secs 2024-04-26T21:59:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36497.260341 secs 2024-04-26T21:59:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36497.260341 secs 2024-04-26T21:59:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:19 smithi194 ceph-mon[33017]: pgmap v19706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36500.229277 secs 2024-04-26T21:59:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:19 smithi033 ceph-mon[27845]: pgmap v19706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36500.229277 secs 2024-04-26T21:59:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:19 smithi096 ceph-mon[32753]: pgmap v19706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36500.229277 secs 2024-04-26T21:59:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:21 smithi194 ceph-mon[33017]: pgmap v19707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:21 smithi033 ceph-mon[27845]: pgmap v19707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:21 smithi096 ceph-mon[32753]: pgmap v19707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:23 smithi194 ceph-mon[33017]: pgmap v19708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36502.260422 secs 2024-04-26T21:59:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:23 smithi033 ceph-mon[27845]: pgmap v19708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36502.260422 secs 2024-04-26T21:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:23 smithi096 ceph-mon[32753]: pgmap v19708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36502.260422 secs 2024-04-26T21:59:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36505.229400 secs 2024-04-26T21:59:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36505.229400 secs 2024-04-26T21:59:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36505.229400 secs 2024-04-26T21:59:26.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:25 smithi033 ceph-mon[27845]: pgmap v19709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:26.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:25 smithi096 ceph-mon[32753]: pgmap v19709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:26.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:25 smithi194 ceph-mon[33017]: pgmap v19709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:28.011+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:59:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:28.012+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T21:59:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:28.012+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 285 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T21:59:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:28.012+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 285 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T21:59:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:27 smithi194 ceph-mon[33017]: pgmap v19710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:27 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:59:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:27 smithi033 ceph-mon[27845]: pgmap v19710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:27 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:59:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:27 smithi096 ceph-mon[32753]: pgmap v19710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:27 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:59:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36507.260513 secs 2024-04-26T21:59:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36507.260513 secs 2024-04-26T21:59:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36507.260513 secs 2024-04-26T21:59:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:29.582+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:29.582+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:29.582+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:59:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:29 smithi033 ceph-mon[27845]: pgmap v19711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36510.229518 secs 2024-04-26T21:59:30.042 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:29 smithi096 ceph-mon[32753]: pgmap v19711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:30.043 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36510.229518 secs 2024-04-26T21:59:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:29 smithi194 ceph-mon[33017]: pgmap v19711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36510.229518 secs 2024-04-26T21:59:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:31 smithi194 ceph-mon[33017]: pgmap v19712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:31 smithi033 ceph-mon[27845]: pgmap v19712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:31 smithi096 ceph-mon[32753]: pgmap v19712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:34.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:33 smithi194 ceph-mon[33017]: pgmap v19713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:34.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36512.260572 secs 2024-04-26T21:59:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:33 smithi033 ceph-mon[27845]: pgmap v19713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36512.260572 secs 2024-04-26T21:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:33 smithi096 ceph-mon[32753]: pgmap v19713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36512.260572 secs 2024-04-26T21:59:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36515.229598 secs 2024-04-26T21:59:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:59:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:59:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: pgmap v19714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:59:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: pgmap v19715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:59:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36515.229598 secs 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: pgmap v19714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: pgmap v19715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:59:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36515.229598 secs 2024-04-26T21:59:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:59:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:59:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: pgmap v19714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:59:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: pgmap v19715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T21:59:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T21:59:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:37 smithi194 ceph-mon[33017]: pgmap v19716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:37 smithi033 ceph-mon[27845]: pgmap v19716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:37 smithi096 ceph-mon[32753]: pgmap v19716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36517.260675 secs 2024-04-26T21:59:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36517.260675 secs 2024-04-26T21:59:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36517.260675 secs 2024-04-26T21:59:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:39 smithi194 ceph-mon[33017]: pgmap v19717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36520.229693 secs 2024-04-26T21:59:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:39 smithi033 ceph-mon[27845]: pgmap v19717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36520.229693 secs 2024-04-26T21:59:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:39 smithi096 ceph-mon[32753]: pgmap v19717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36520.229693 secs 2024-04-26T21:59:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:41 smithi194 ceph-mon[33017]: pgmap v19718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:41 smithi033 ceph-mon[27845]: pgmap v19718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:41.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:41 smithi096 ceph-mon[32753]: pgmap v19718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:43 smithi194 ceph-mon[33017]: pgmap v19719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36522.260767 secs 2024-04-26T21:59:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:43 smithi033 ceph-mon[27845]: pgmap v19719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36522.260767 secs 2024-04-26T21:59:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:43 smithi096 ceph-mon[32753]: pgmap v19719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T21:59:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36522.260767 secs 2024-04-26T21:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:44.584+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:44.584+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:44.584+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:59:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36525.229815 secs 2024-04-26T21:59:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36525.229815 secs 2024-04-26T21:59:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36525.229815 secs 2024-04-26T21:59:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:45 smithi194 ceph-mon[33017]: pgmap v19720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:59:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:45 smithi033 ceph-mon[27845]: pgmap v19720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:59:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:45 smithi096 ceph-mon[32753]: pgmap v19720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T21:59:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:47 smithi033 ceph-mon[27845]: pgmap v19721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:59:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:47 smithi194 ceph-mon[33017]: pgmap v19721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:59:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:47 smithi096 ceph-mon[32753]: pgmap v19721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:59:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36527.260858 secs 2024-04-26T21:59:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36527.260858 secs 2024-04-26T21:59:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36527.260858 secs 2024-04-26T21:59:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:49 smithi194 ceph-mon[33017]: pgmap v19722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36530.229908 secs 2024-04-26T21:59:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:49 smithi033 ceph-mon[27845]: pgmap v19722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36530.229908 secs 2024-04-26T21:59:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:49 smithi096 ceph-mon[32753]: pgmap v19722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36530.229908 secs 2024-04-26T21:59:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:51 smithi194 ceph-mon[33017]: pgmap v19723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:51 smithi033 ceph-mon[27845]: pgmap v19723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:51 smithi096 ceph-mon[32753]: pgmap v19723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:53 smithi194 ceph-mon[33017]: pgmap v19724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36532.260940 secs 2024-04-26T21:59:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:53 smithi033 ceph-mon[27845]: pgmap v19724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36532.260940 secs 2024-04-26T21:59:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:53 smithi096 ceph-mon[32753]: pgmap v19724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36532.260940 secs 2024-04-26T21:59:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36535.230029 secs 2024-04-26T21:59:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36535.230029 secs 2024-04-26T21:59:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36535.230029 secs 2024-04-26T21:59:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:55 smithi194 ceph-mon[33017]: pgmap v19725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:59:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:55 smithi033 ceph-mon[27845]: pgmap v19725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:59:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:55 smithi096 ceph-mon[32753]: pgmap v19725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:59:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:57 smithi194 ceph-mon[33017]: pgmap v19726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:59:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:57 smithi033 ceph-mon[27845]: pgmap v19726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:59:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:57 smithi096 ceph-mon[32753]: pgmap v19726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T21:59:58.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36537.261008 secs 2024-04-26T21:59:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36537.261008 secs 2024-04-26T21:59:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36537.261008 secs 2024-04-26T21:59:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:59.599+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:59.599+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T21:59:59.599 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T21:59:59.599+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T21:59:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:59 smithi194 ceph-mon[33017]: pgmap v19727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 21:59:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36540.230149 secs 2024-04-26T21:59:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:59 smithi033 ceph-mon[27845]: pgmap v19727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 21:59:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36540.230149 secs 2024-04-26T21:59:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:59 smithi096 ceph-mon[32753]: pgmap v19727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T21:59:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 21:59:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36540.230149 secs 2024-04-26T22:00:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:00:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:00:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:00:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:01 smithi194 ceph-mon[33017]: pgmap v19728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:01 smithi033 ceph-mon[27845]: pgmap v19728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:01 smithi096 ceph-mon[32753]: pgmap v19728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:03 smithi033 ceph-mon[27845]: pgmap v19729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36542.261071 secs 2024-04-26T22:00:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:03 smithi096 ceph-mon[32753]: pgmap v19729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:03.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36542.261071 secs 2024-04-26T22:00:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:03 smithi194 ceph-mon[33017]: pgmap v19729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36542.261071 secs 2024-04-26T22:00:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36545.230299 secs 2024-04-26T22:00:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36545.230299 secs 2024-04-26T22:00:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36545.230299 secs 2024-04-26T22:00:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:05 smithi033 ceph-mon[27845]: pgmap v19730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:05 smithi096 ceph-mon[32753]: pgmap v19730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:05 smithi194 ceph-mon[33017]: pgmap v19730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:07 smithi033 ceph-mon[27845]: pgmap v19731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:07 smithi096 ceph-mon[32753]: pgmap v19731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:07 smithi194 ceph-mon[33017]: pgmap v19731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36547.261133 secs 2024-04-26T22:00:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36547.261133 secs 2024-04-26T22:00:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36547.261133 secs 2024-04-26T22:00:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:09 smithi033 ceph-mon[27845]: pgmap v19732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36550.230435 secs 2024-04-26T22:00:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:09 smithi096 ceph-mon[32753]: pgmap v19732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36550.230435 secs 2024-04-26T22:00:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:09 smithi194 ceph-mon[33017]: pgmap v19732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36550.230435 secs 2024-04-26T22:00:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:11 smithi033 ceph-mon[27845]: pgmap v19733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:11 smithi096 ceph-mon[32753]: pgmap v19733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:11 smithi194 ceph-mon[33017]: pgmap v19733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:13.020+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:00:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:13.020+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T22:00:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:13.020+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 286 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T22:00:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:13.020+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 286 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:13 smithi033 ceph-mon[27845]: pgmap v19734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36552.261220 secs 2024-04-26T22:00:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:13 smithi096 ceph-mon[32753]: pgmap v19734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36552.261220 secs 2024-04-26T22:00:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:13 smithi194 ceph-mon[33017]: pgmap v19734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36552.261220 secs 2024-04-26T22:00:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:14.608+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:14.608+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:14.608+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:00:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36555.230577 secs 2024-04-26T22:00:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36555.230577 secs 2024-04-26T22:00:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36555.230577 secs 2024-04-26T22:00:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:15 smithi033 ceph-mon[27845]: pgmap v19735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:15 smithi096 ceph-mon[32753]: pgmap v19735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:15 smithi194 ceph-mon[33017]: pgmap v19735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:17 smithi033 ceph-mon[27845]: pgmap v19736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:17 smithi096 ceph-mon[32753]: pgmap v19736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:17 smithi194 ceph-mon[33017]: pgmap v19736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36557.261285 secs 2024-04-26T22:00:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36557.261285 secs 2024-04-26T22:00:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36557.261285 secs 2024-04-26T22:00:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:19 smithi096 ceph-mon[32753]: pgmap v19737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36560.230706 secs 2024-04-26T22:00:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:19 smithi194 ceph-mon[33017]: pgmap v19737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36560.230706 secs 2024-04-26T22:00:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:19 smithi033 ceph-mon[27845]: pgmap v19737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36560.230706 secs 2024-04-26T22:00:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:21 smithi194 ceph-mon[33017]: pgmap v19738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:21 smithi033 ceph-mon[27845]: pgmap v19738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:21 smithi096 ceph-mon[32753]: pgmap v19738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:23 smithi194 ceph-mon[33017]: pgmap v19739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36562.261347 secs 2024-04-26T22:00:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:23 smithi033 ceph-mon[27845]: pgmap v19739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36562.261347 secs 2024-04-26T22:00:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:23 smithi096 ceph-mon[32753]: pgmap v19739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36562.261347 secs 2024-04-26T22:00:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36565.230829 secs 2024-04-26T22:00:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36565.230829 secs 2024-04-26T22:00:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36565.230829 secs 2024-04-26T22:00:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:25 smithi194 ceph-mon[33017]: pgmap v19740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:25 smithi033 ceph-mon[27845]: pgmap v19740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:25 smithi096 ceph-mon[32753]: pgmap v19740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:27.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:27 smithi033 ceph-mon[27845]: pgmap v19741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:27.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:27 smithi096 ceph-mon[32753]: pgmap v19741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:27 smithi194 ceph-mon[33017]: pgmap v19741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:00:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36567.261411 secs 2024-04-26T22:00:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36567.261411 secs 2024-04-26T22:00:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36567.261411 secs 2024-04-26T22:00:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:29.623+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:29.623+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:29.623+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:00:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:29 smithi194 ceph-mon[33017]: pgmap v19742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36570.230964 secs 2024-04-26T22:00:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:29 smithi033 ceph-mon[27845]: pgmap v19742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36570.230964 secs 2024-04-26T22:00:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:29 smithi096 ceph-mon[32753]: pgmap v19742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36570.230964 secs 2024-04-26T22:00:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:31 smithi194 ceph-mon[33017]: pgmap v19743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:31 smithi033 ceph-mon[27845]: pgmap v19743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:31 smithi096 ceph-mon[32753]: pgmap v19743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:33 smithi194 ceph-mon[33017]: pgmap v19744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36572.261500 secs 2024-04-26T22:00:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:33 smithi033 ceph-mon[27845]: pgmap v19744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36572.261500 secs 2024-04-26T22:00:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:33 smithi096 ceph-mon[32753]: pgmap v19744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36572.261500 secs 2024-04-26T22:00:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36575.231099 secs 2024-04-26T22:00:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36575.231099 secs 2024-04-26T22:00:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36575.231099 secs 2024-04-26T22:00:36.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:35 smithi033 ceph-mon[27845]: pgmap v19745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:36.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:00:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:35 smithi194 ceph-mon[33017]: pgmap v19745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:00:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:35 smithi096 ceph-mon[32753]: pgmap v19745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:00:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:00:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:00:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:36 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:00:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:00:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:00:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:36 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:00:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:00:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:00:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:36 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:00:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:37 smithi194 ceph-mon[33017]: pgmap v19746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:37 smithi194 ceph-mon[33017]: pgmap v19747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:37 smithi033 ceph-mon[27845]: pgmap v19746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:37 smithi033 ceph-mon[27845]: pgmap v19747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:37 smithi096 ceph-mon[32753]: pgmap v19746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:37 smithi096 ceph-mon[32753]: pgmap v19747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36577.261589 secs 2024-04-26T22:00:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36577.261589 secs 2024-04-26T22:00:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36577.261589 secs 2024-04-26T22:00:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:39 smithi194 ceph-mon[33017]: pgmap v19748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36580.231218 secs 2024-04-26T22:00:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:39 smithi033 ceph-mon[27845]: pgmap v19748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36580.231218 secs 2024-04-26T22:00:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:39 smithi096 ceph-mon[32753]: pgmap v19748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36580.231218 secs 2024-04-26T22:00:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:41 smithi194 ceph-mon[33017]: pgmap v19749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:41 smithi033 ceph-mon[27845]: pgmap v19749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:42.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:41 smithi096 ceph-mon[32753]: pgmap v19749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:43 smithi194 ceph-mon[33017]: pgmap v19750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36582.261675 secs 2024-04-26T22:00:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:43 smithi033 ceph-mon[27845]: pgmap v19750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36582.261675 secs 2024-04-26T22:00:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:43 smithi096 ceph-mon[32753]: pgmap v19750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:00:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36582.261675 secs 2024-04-26T22:00:44.639 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:44.640+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:44.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:44.640+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:44.640 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:44.640+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:00:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36585.231285 secs 2024-04-26T22:00:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36585.231285 secs 2024-04-26T22:00:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36585.231285 secs 2024-04-26T22:00:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:45 smithi194 ceph-mon[33017]: pgmap v19751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:00:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:45 smithi033 ceph-mon[27845]: pgmap v19751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:00:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:45 smithi096 ceph-mon[32753]: pgmap v19751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:00:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:47 smithi194 ceph-mon[33017]: pgmap v19752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:00:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:47 smithi033 ceph-mon[27845]: pgmap v19752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:00:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:47 smithi096 ceph-mon[32753]: pgmap v19752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:00:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36587.261768 secs 2024-04-26T22:00:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36587.261768 secs 2024-04-26T22:00:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36587.261768 secs 2024-04-26T22:00:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:49 smithi194 ceph-mon[33017]: pgmap v19753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36590.231363 secs 2024-04-26T22:00:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:49 smithi033 ceph-mon[27845]: pgmap v19753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:50.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36590.231363 secs 2024-04-26T22:00:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:49 smithi096 ceph-mon[32753]: pgmap v19753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36590.231363 secs 2024-04-26T22:00:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:51 smithi194 ceph-mon[33017]: pgmap v19754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:51 smithi033 ceph-mon[27845]: pgmap v19754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:51 smithi096 ceph-mon[32753]: pgmap v19754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:53 smithi194 ceph-mon[33017]: pgmap v19755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36592.261856 secs 2024-04-26T22:00:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:53 smithi033 ceph-mon[27845]: pgmap v19755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36592.261856 secs 2024-04-26T22:00:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:53 smithi096 ceph-mon[32753]: pgmap v19755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36592.261856 secs 2024-04-26T22:00:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36595.231443 secs 2024-04-26T22:00:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36595.231443 secs 2024-04-26T22:00:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36595.231443 secs 2024-04-26T22:00:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:55 smithi194 ceph-mon[33017]: pgmap v19756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:55 smithi033 ceph-mon[27845]: pgmap v19756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:55 smithi096 ceph-mon[32753]: pgmap v19756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:00:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:57 smithi194 ceph-mon[33017]: pgmap v19757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:57 smithi033 ceph-mon[27845]: pgmap v19757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:58.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:57 smithi096 ceph-mon[32753]: pgmap v19757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:00:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36597.261942 secs 2024-04-26T22:00:59.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36597.261942 secs 2024-04-26T22:00:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36597.261942 secs 2024-04-26T22:00:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:59.655+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:59.655+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:00:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:00:59.655+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:01:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:59 smithi033 ceph-mon[27845]: pgmap v19758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36600.231559 secs 2024-04-26T22:01:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:59 smithi096 ceph-mon[32753]: pgmap v19758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36600.231559 secs 2024-04-26T22:01:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:59 smithi194 ceph-mon[33017]: pgmap v19758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36600.231559 secs 2024-04-26T22:01:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:00 smithi033 ceph-mon[27845]: pgmap v19759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:00 smithi096 ceph-mon[32753]: pgmap v19759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:00 smithi194 ceph-mon[33017]: pgmap v19759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:03 smithi194 ceph-mon[33017]: pgmap v19760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:03 smithi033 ceph-mon[27845]: pgmap v19760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:03.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:03 smithi096 ceph-mon[32753]: pgmap v19760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36602.262004 secs 2024-04-26T22:01:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36602.262004 secs 2024-04-26T22:01:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36602.262004 secs 2024-04-26T22:01:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36605.231681 secs 2024-04-26T22:01:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:05 smithi194 ceph-mon[33017]: pgmap v19761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36605.231681 secs 2024-04-26T22:01:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:05 smithi033 ceph-mon[27845]: pgmap v19761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36605.231681 secs 2024-04-26T22:01:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:05 smithi096 ceph-mon[32753]: pgmap v19761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:07 smithi033 ceph-mon[27845]: pgmap v19762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:07 smithi194 ceph-mon[33017]: pgmap v19762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:07 smithi096 ceph-mon[32753]: pgmap v19762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36607.262067 secs 2024-04-26T22:01:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:09 smithi033 ceph-mon[27845]: pgmap v19763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36607.262067 secs 2024-04-26T22:01:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:09 smithi194 ceph-mon[33017]: pgmap v19763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36607.262067 secs 2024-04-26T22:01:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:09 smithi096 ceph-mon[32753]: pgmap v19763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36610.231815 secs 2024-04-26T22:01:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36610.231815 secs 2024-04-26T22:01:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36610.231815 secs 2024-04-26T22:01:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:11 smithi194 ceph-mon[33017]: pgmap v19764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:11 smithi033 ceph-mon[27845]: pgmap v19764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:11.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:11 smithi096 ceph-mon[32753]: pgmap v19764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:13 smithi194 ceph-mon[33017]: pgmap v19765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:13 smithi033 ceph-mon[27845]: pgmap v19765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:13 smithi096 ceph-mon[32753]: pgmap v19765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36612.262131 secs 2024-04-26T22:01:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:14.670+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:14.670+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:14.670+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:01:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36612.262131 secs 2024-04-26T22:01:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36612.262131 secs 2024-04-26T22:01:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36615.231964 secs 2024-04-26T22:01:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:15 smithi194 ceph-mon[33017]: pgmap v19766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36615.231964 secs 2024-04-26T22:01:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:15 smithi033 ceph-mon[27845]: pgmap v19766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36615.231964 secs 2024-04-26T22:01:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:15 smithi096 ceph-mon[32753]: pgmap v19766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:17 smithi194 ceph-mon[33017]: pgmap v19767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:17 smithi033 ceph-mon[27845]: pgmap v19767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:17 smithi096 ceph-mon[32753]: pgmap v19767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36617.262191 secs 2024-04-26T22:01:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:19 smithi194 ceph-mon[33017]: pgmap v19768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36617.262191 secs 2024-04-26T22:01:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:19 smithi033 ceph-mon[27845]: pgmap v19768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36617.262191 secs 2024-04-26T22:01:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:19 smithi096 ceph-mon[32753]: pgmap v19768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36620.232086 secs 2024-04-26T22:01:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36620.232086 secs 2024-04-26T22:01:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36620.232086 secs 2024-04-26T22:01:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:21 smithi033 ceph-mon[27845]: pgmap v19769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:21 smithi096 ceph-mon[32753]: pgmap v19769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:21 smithi194 ceph-mon[33017]: pgmap v19769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:23 smithi033 ceph-mon[27845]: pgmap v19770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:23 smithi096 ceph-mon[32753]: pgmap v19770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:23 smithi194 ceph-mon[33017]: pgmap v19770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36622.262252 secs 2024-04-26T22:01:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36622.262252 secs 2024-04-26T22:01:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36622.262252 secs 2024-04-26T22:01:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36625.232208 secs 2024-04-26T22:01:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:25 smithi033 ceph-mon[27845]: pgmap v19771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36625.232208 secs 2024-04-26T22:01:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:25 smithi096 ceph-mon[32753]: pgmap v19771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36625.232208 secs 2024-04-26T22:01:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:25 smithi194 ceph-mon[33017]: pgmap v19771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:27 smithi194 ceph-mon[33017]: pgmap v19772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:27.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:27 smithi033 ceph-mon[27845]: pgmap v19772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:27 smithi096 ceph-mon[32753]: pgmap v19772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36627.262338 secs 2024-04-26T22:01:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:29 smithi194 ceph-mon[33017]: pgmap v19773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36627.262338 secs 2024-04-26T22:01:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:29 smithi033 ceph-mon[27845]: pgmap v19773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36627.262338 secs 2024-04-26T22:01:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:29 smithi096 ceph-mon[32753]: pgmap v19773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:29.685+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:29.686+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:29.686+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:01:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36630.232324 secs 2024-04-26T22:01:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36630.232324 secs 2024-04-26T22:01:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36630.232324 secs 2024-04-26T22:01:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:31 smithi194 ceph-mon[33017]: pgmap v19774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:31 smithi033 ceph-mon[27845]: pgmap v19774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:31 smithi096 ceph-mon[32753]: pgmap v19774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:33.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:33 smithi194 ceph-mon[33017]: pgmap v19775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:33 smithi033 ceph-mon[27845]: pgmap v19775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:33 smithi096 ceph-mon[32753]: pgmap v19775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36632.262401 secs 2024-04-26T22:01:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36632.262401 secs 2024-04-26T22:01:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36632.262401 secs 2024-04-26T22:01:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36635.232424 secs 2024-04-26T22:01:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:35 smithi194 ceph-mon[33017]: pgmap v19776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36635.232424 secs 2024-04-26T22:01:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:35 smithi033 ceph-mon[27845]: pgmap v19776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36635.232424 secs 2024-04-26T22:01:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:35 smithi096 ceph-mon[32753]: pgmap v19776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:37 smithi194 ceph-mon[33017]: pgmap v19777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:01:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:37 smithi033 ceph-mon[27845]: pgmap v19777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:01:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:37 smithi096 ceph-mon[32753]: pgmap v19777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:01:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:01:38.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:01:38.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:01:38.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:01:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:01:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:01:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:01:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:01:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:01:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:01:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:39 smithi194 ceph-mon[33017]: pgmap v19778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:39 smithi194 ceph-mon[33017]: pgmap v19779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36637.262486 secs 2024-04-26T22:01:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:39 smithi033 ceph-mon[27845]: pgmap v19778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:39 smithi033 ceph-mon[27845]: pgmap v19779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36637.262486 secs 2024-04-26T22:01:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:39 smithi096 ceph-mon[32753]: pgmap v19778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:01:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:39 smithi096 ceph-mon[32753]: pgmap v19779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36637.262486 secs 2024-04-26T22:01:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36640.232497 secs 2024-04-26T22:01:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36640.232497 secs 2024-04-26T22:01:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36640.232497 secs 2024-04-26T22:01:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:41 smithi194 ceph-mon[33017]: pgmap v19780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:41 smithi033 ceph-mon[27845]: pgmap v19780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:41 smithi096 ceph-mon[32753]: pgmap v19780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:43 smithi194 ceph-mon[33017]: pgmap v19781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:43 smithi033 ceph-mon[27845]: pgmap v19781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:43.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:43 smithi096 ceph-mon[32753]: pgmap v19781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36642.262571 secs 2024-04-26T22:01:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36642.262571 secs 2024-04-26T22:01:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:44.699+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:44.699+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:44.699+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:01:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36642.262571 secs 2024-04-26T22:01:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36645.232635 secs 2024-04-26T22:01:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:45 smithi033 ceph-mon[27845]: pgmap v19782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36645.232635 secs 2024-04-26T22:01:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:45 smithi096 ceph-mon[32753]: pgmap v19782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36645.232635 secs 2024-04-26T22:01:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:45 smithi194 ceph-mon[33017]: pgmap v19782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:47 smithi033 ceph-mon[27845]: pgmap v19783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:47 smithi096 ceph-mon[32753]: pgmap v19783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:47 smithi194 ceph-mon[33017]: pgmap v19783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:01:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:49 smithi033 ceph-mon[27845]: pgmap v19784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:01:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36647.262653 secs 2024-04-26T22:01:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:49 smithi096 ceph-mon[32753]: pgmap v19784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:01:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36647.262653 secs 2024-04-26T22:01:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:49 smithi194 ceph-mon[33017]: pgmap v19784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:01:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36647.262653 secs 2024-04-26T22:01:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36650.232776 secs 2024-04-26T22:01:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36650.232776 secs 2024-04-26T22:01:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36650.232776 secs 2024-04-26T22:01:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:51 smithi033 ceph-mon[27845]: pgmap v19785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:01:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:51 smithi096 ceph-mon[32753]: pgmap v19785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:01:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:51 smithi194 ceph-mon[33017]: pgmap v19785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:01:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:53 smithi033 ceph-mon[27845]: pgmap v19786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:53 smithi096 ceph-mon[32753]: pgmap v19786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:53 smithi194 ceph-mon[33017]: pgmap v19786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36652.262733 secs 2024-04-26T22:01:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36652.262733 secs 2024-04-26T22:01:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36652.262733 secs 2024-04-26T22:01:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36655.232853 secs 2024-04-26T22:01:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:55 smithi033 ceph-mon[27845]: pgmap v19787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36655.232853 secs 2024-04-26T22:01:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:55 smithi096 ceph-mon[32753]: pgmap v19787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36655.232853 secs 2024-04-26T22:01:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:55 smithi194 ceph-mon[33017]: pgmap v19787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:57 smithi033 ceph-mon[27845]: pgmap v19788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:01:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:57 smithi096 ceph-mon[32753]: pgmap v19788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:01:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:57 smithi194 ceph-mon[33017]: pgmap v19788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:01:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:59 smithi033 ceph-mon[27845]: pgmap v19789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:01:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36657.262819 secs 2024-04-26T22:01:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:59 smithi096 ceph-mon[32753]: pgmap v19789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:01:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36657.262819 secs 2024-04-26T22:01:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:59.713+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:59.714+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:01:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:01:59.714+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:59 smithi194 ceph-mon[33017]: pgmap v19789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:01:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36657.262819 secs 2024-04-26T22:02:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36660.232924 secs 2024-04-26T22:02:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36660.232924 secs 2024-04-26T22:02:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36660.232924 secs 2024-04-26T22:02:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:01 smithi096 ceph-mon[32753]: pgmap v19790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:02:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:01 smithi194 ceph-mon[33017]: pgmap v19790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:02:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:01 smithi033 ceph-mon[27845]: pgmap v19790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:03 smithi194 ceph-mon[33017]: pgmap v19791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36662.262905 secs 2024-04-26T22:02:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:03 smithi033 ceph-mon[27845]: pgmap v19791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36662.262905 secs 2024-04-26T22:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:03 smithi096 ceph-mon[32753]: pgmap v19791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36662.262905 secs 2024-04-26T22:02:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36665.233041 secs 2024-04-26T22:02:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:05 smithi194 ceph-mon[33017]: pgmap v19792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36665.233041 secs 2024-04-26T22:02:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:05 smithi033 ceph-mon[27845]: pgmap v19792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36665.233041 secs 2024-04-26T22:02:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:05 smithi096 ceph-mon[32753]: pgmap v19792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:07 smithi194 ceph-mon[33017]: pgmap v19793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:07 smithi033 ceph-mon[27845]: pgmap v19793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:07 smithi096 ceph-mon[32753]: pgmap v19793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36667.263008 secs 2024-04-26T22:02:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36667.263008 secs 2024-04-26T22:02:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36667.263008 secs 2024-04-26T22:02:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:09 smithi194 ceph-mon[33017]: pgmap v19794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:09 smithi033 ceph-mon[27845]: pgmap v19794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:09 smithi096 ceph-mon[32753]: pgmap v19794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36670.233158 secs 2024-04-26T22:02:10.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36670.233158 secs 2024-04-26T22:02:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36670.233158 secs 2024-04-26T22:02:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:11 smithi194 ceph-mon[33017]: pgmap v19795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:11 smithi033 ceph-mon[27845]: pgmap v19795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:11 smithi096 ceph-mon[32753]: pgmap v19795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:13 smithi194 ceph-mon[33017]: pgmap v19796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36672.263070 secs 2024-04-26T22:02:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:13 smithi033 ceph-mon[27845]: pgmap v19796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36672.263070 secs 2024-04-26T22:02:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:13 smithi096 ceph-mon[32753]: pgmap v19796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36672.263070 secs 2024-04-26T22:02:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:14.728+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:14.728+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:02:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:14.728+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36675.233289 secs 2024-04-26T22:02:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:15 smithi194 ceph-mon[33017]: pgmap v19797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36675.233289 secs 2024-04-26T22:02:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:15 smithi033 ceph-mon[27845]: pgmap v19797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36675.233289 secs 2024-04-26T22:02:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:15 smithi096 ceph-mon[32753]: pgmap v19797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:17 smithi194 ceph-mon[33017]: pgmap v19798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:17 smithi033 ceph-mon[27845]: pgmap v19798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:17 smithi096 ceph-mon[32753]: pgmap v19798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36677.263133 secs 2024-04-26T22:02:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36677.263133 secs 2024-04-26T22:02:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36677.263133 secs 2024-04-26T22:02:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:19 smithi194 ceph-mon[33017]: pgmap v19799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36680.233411 secs 2024-04-26T22:02:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:19 smithi033 ceph-mon[27845]: pgmap v19799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36680.233411 secs 2024-04-26T22:02:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:19 smithi096 ceph-mon[32753]: pgmap v19799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36680.233411 secs 2024-04-26T22:02:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:21 smithi194 ceph-mon[33017]: pgmap v19800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:21 smithi033 ceph-mon[27845]: pgmap v19800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:21 smithi096 ceph-mon[32753]: pgmap v19800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:23 smithi194 ceph-mon[33017]: pgmap v19801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36682.263218 secs 2024-04-26T22:02:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:23 smithi033 ceph-mon[27845]: pgmap v19801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36682.263218 secs 2024-04-26T22:02:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:23 smithi096 ceph-mon[32753]: pgmap v19801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36682.263218 secs 2024-04-26T22:02:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36685.233533 secs 2024-04-26T22:02:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36685.233533 secs 2024-04-26T22:02:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36685.233533 secs 2024-04-26T22:02:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:25 smithi194 ceph-mon[33017]: pgmap v19802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:25 smithi033 ceph-mon[27845]: pgmap v19802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:25 smithi096 ceph-mon[32753]: pgmap v19802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:27 smithi194 ceph-mon[33017]: pgmap v19803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:27 smithi033 ceph-mon[27845]: pgmap v19803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:27 smithi096 ceph-mon[32753]: pgmap v19803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:28.043+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T22:02:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:28.043+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:02:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:28.043+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 287 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T22:02:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:28.043+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 287 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:02:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36687.263297 secs 2024-04-26T22:02:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36687.263297 secs 2024-04-26T22:02:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36687.263297 secs 2024-04-26T22:02:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:29.743+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:29.743+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:02:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:29.743+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:29 smithi194 ceph-mon[33017]: pgmap v19804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36690.233647 secs 2024-04-26T22:02:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:29 smithi033 ceph-mon[27845]: pgmap v19804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36690.233647 secs 2024-04-26T22:02:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:29 smithi096 ceph-mon[32753]: pgmap v19804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36690.233647 secs 2024-04-26T22:02:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:31 smithi194 ceph-mon[33017]: pgmap v19805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:31 smithi033 ceph-mon[27845]: pgmap v19805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:31 smithi096 ceph-mon[32753]: pgmap v19805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:33 smithi194 ceph-mon[33017]: pgmap v19806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36692.263360 secs 2024-04-26T22:02:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:33 smithi033 ceph-mon[27845]: pgmap v19806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36692.263360 secs 2024-04-26T22:02:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:33 smithi096 ceph-mon[32753]: pgmap v19806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36692.263360 secs 2024-04-26T22:02:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36695.233785 secs 2024-04-26T22:02:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36695.233785 secs 2024-04-26T22:02:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36695.233785 secs 2024-04-26T22:02:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:35 smithi194 ceph-mon[33017]: pgmap v19807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:35 smithi033 ceph-mon[27845]: pgmap v19807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:35 smithi096 ceph-mon[32753]: pgmap v19807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:37 smithi194 ceph-mon[33017]: pgmap v19808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:37 smithi033 ceph-mon[27845]: pgmap v19808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:37 smithi096 ceph-mon[32753]: pgmap v19808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36697.263443 secs 2024-04-26T22:02:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:02:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36697.263443 secs 2024-04-26T22:02:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:02:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36697.263443 secs 2024-04-26T22:02:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:02:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:39 smithi194 ceph-mon[33017]: pgmap v19809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36700.233876 secs 2024-04-26T22:02:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:39 smithi033 ceph-mon[27845]: pgmap v19809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36700.233876 secs 2024-04-26T22:02:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:39 smithi096 ceph-mon[32753]: pgmap v19809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36700.233876 secs 2024-04-26T22:02:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:02:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:02:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:40 smithi194 ceph-mon[33017]: pgmap v19810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:02:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:40 smithi194 ceph-mon[33017]: pgmap v19811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:40.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:02:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:02:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:02:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:40 smithi033 ceph-mon[27845]: pgmap v19810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:02:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:40 smithi033 ceph-mon[27845]: pgmap v19811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:40.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:02:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:02:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:02:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:40 smithi096 ceph-mon[32753]: pgmap v19810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:02:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:40 smithi096 ceph-mon[32753]: pgmap v19811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:02:43.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:42 smithi194 ceph-mon[33017]: pgmap v19812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:42 smithi033 ceph-mon[27845]: pgmap v19812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:42 smithi096 ceph-mon[32753]: pgmap v19812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36702.263525 secs 2024-04-26T22:02:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36702.263525 secs 2024-04-26T22:02:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36702.263525 secs 2024-04-26T22:02:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:44.752+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:44.752+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:44.752+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:02:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:44 smithi194 ceph-mon[33017]: pgmap v19813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36705.233997 secs 2024-04-26T22:02:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:44 smithi033 ceph-mon[27845]: pgmap v19813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36705.233997 secs 2024-04-26T22:02:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:44 smithi096 ceph-mon[32753]: pgmap v19813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36705.233997 secs 2024-04-26T22:02:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:46 smithi194 ceph-mon[33017]: pgmap v19814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:46 smithi033 ceph-mon[27845]: pgmap v19814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:46 smithi096 ceph-mon[32753]: pgmap v19814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:48 smithi194 ceph-mon[33017]: pgmap v19815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36707.263609 secs 2024-04-26T22:02:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:48 smithi033 ceph-mon[27845]: pgmap v19815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36707.263609 secs 2024-04-26T22:02:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:48 smithi096 ceph-mon[32753]: pgmap v19815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:02:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36707.263609 secs 2024-04-26T22:02:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36710.234151 secs 2024-04-26T22:02:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36710.234151 secs 2024-04-26T22:02:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36710.234151 secs 2024-04-26T22:02:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:50 smithi194 ceph-mon[33017]: pgmap v19816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:02:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:50 smithi033 ceph-mon[27845]: pgmap v19816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:02:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:50 smithi096 ceph-mon[32753]: pgmap v19816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:02:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:52 smithi194 ceph-mon[33017]: pgmap v19817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:52 smithi033 ceph-mon[27845]: pgmap v19817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:52 smithi096 ceph-mon[32753]: pgmap v19817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36712.263692 secs 2024-04-26T22:02:54.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36712.263692 secs 2024-04-26T22:02:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36712.263692 secs 2024-04-26T22:02:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:54 smithi194 ceph-mon[33017]: pgmap v19818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36715.234256 secs 2024-04-26T22:02:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:54 smithi033 ceph-mon[27845]: pgmap v19818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36715.234256 secs 2024-04-26T22:02:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:54 smithi096 ceph-mon[32753]: pgmap v19818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36715.234256 secs 2024-04-26T22:02:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:56 smithi194 ceph-mon[33017]: pgmap v19819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:56 smithi033 ceph-mon[27845]: pgmap v19819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:56 smithi096 ceph-mon[32753]: pgmap v19819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:02:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:58 smithi194 ceph-mon[33017]: pgmap v19820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36717.263774 secs 2024-04-26T22:02:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:58 smithi033 ceph-mon[27845]: pgmap v19820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36717.263774 secs 2024-04-26T22:02:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:58 smithi096 ceph-mon[32753]: pgmap v19820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:02:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36717.263774 secs 2024-04-26T22:02:59.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:59.760+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:59.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:59.760+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:02:59.760 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:02:59.760+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:03:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36720.234389 secs 2024-04-26T22:03:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36720.234389 secs 2024-04-26T22:03:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36720.234389 secs 2024-04-26T22:03:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:00 smithi194 ceph-mon[33017]: pgmap v19821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:00 smithi033 ceph-mon[27845]: pgmap v19821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:00 smithi096 ceph-mon[32753]: pgmap v19821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:02 smithi194 ceph-mon[33017]: pgmap v19822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:02 smithi033 ceph-mon[27845]: pgmap v19822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:02 smithi096 ceph-mon[32753]: pgmap v19822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36722.263833 secs 2024-04-26T22:03:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36722.263833 secs 2024-04-26T22:03:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36722.263833 secs 2024-04-26T22:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:04 smithi194 ceph-mon[33017]: pgmap v19823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36725.234467 secs 2024-04-26T22:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:04 smithi033 ceph-mon[27845]: pgmap v19823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36725.234467 secs 2024-04-26T22:03:05.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:04 smithi096 ceph-mon[32753]: pgmap v19823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36725.234467 secs 2024-04-26T22:03:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:06 smithi194 ceph-mon[33017]: pgmap v19824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:06 smithi033 ceph-mon[27845]: pgmap v19824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:06 smithi096 ceph-mon[32753]: pgmap v19824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:08 smithi194 ceph-mon[33017]: pgmap v19825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36727.263921 secs 2024-04-26T22:03:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:08 smithi033 ceph-mon[27845]: pgmap v19825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36727.263921 secs 2024-04-26T22:03:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:08 smithi096 ceph-mon[32753]: pgmap v19825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36727.263921 secs 2024-04-26T22:03:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36730.234609 secs 2024-04-26T22:03:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36730.234609 secs 2024-04-26T22:03:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36730.234609 secs 2024-04-26T22:03:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:10 smithi194 ceph-mon[33017]: pgmap v19826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:10 smithi033 ceph-mon[27845]: pgmap v19826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:10 smithi096 ceph-mon[32753]: pgmap v19826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:12 smithi194 ceph-mon[33017]: pgmap v19827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:12 smithi033 ceph-mon[27845]: pgmap v19827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:12 smithi096 ceph-mon[32753]: pgmap v19827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36732.264004 secs 2024-04-26T22:03:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36732.264004 secs 2024-04-26T22:03:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36732.264004 secs 2024-04-26T22:03:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:14.775+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:14.775+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:14.775+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:03:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:14 smithi194 ceph-mon[33017]: pgmap v19828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36735.234738 secs 2024-04-26T22:03:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:14 smithi033 ceph-mon[27845]: pgmap v19828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36735.234738 secs 2024-04-26T22:03:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:14 smithi096 ceph-mon[32753]: pgmap v19828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36735.234738 secs 2024-04-26T22:03:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:16 smithi194 ceph-mon[33017]: pgmap v19829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:16 smithi033 ceph-mon[27845]: pgmap v19829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:16 smithi096 ceph-mon[32753]: pgmap v19829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:18 smithi033 ceph-mon[27845]: pgmap v19830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36737.264064 secs 2024-04-26T22:03:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:18 smithi096 ceph-mon[32753]: pgmap v19830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36737.264064 secs 2024-04-26T22:03:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:18 smithi194 ceph-mon[33017]: pgmap v19830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36737.264064 secs 2024-04-26T22:03:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36740.234895 secs 2024-04-26T22:03:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36740.234895 secs 2024-04-26T22:03:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36740.234895 secs 2024-04-26T22:03:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:20 smithi033 ceph-mon[27845]: pgmap v19831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:20 smithi096 ceph-mon[32753]: pgmap v19831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:20 smithi194 ceph-mon[33017]: pgmap v19831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:23.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:22 smithi033 ceph-mon[27845]: pgmap v19832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:23.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:22 smithi096 ceph-mon[32753]: pgmap v19832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:23.370 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:22 smithi194 ceph-mon[33017]: pgmap v19832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36742.264149 secs 2024-04-26T22:03:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36742.264149 secs 2024-04-26T22:03:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36742.264149 secs 2024-04-26T22:03:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:24 smithi033 ceph-mon[27845]: pgmap v19833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36745.234995 secs 2024-04-26T22:03:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:24 smithi096 ceph-mon[32753]: pgmap v19833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36745.234995 secs 2024-04-26T22:03:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:24 smithi194 ceph-mon[33017]: pgmap v19833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36745.234995 secs 2024-04-26T22:03:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:25 smithi033 ceph-mon[27845]: pgmap v19834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:25 smithi096 ceph-mon[32753]: pgmap v19834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:25 smithi194 ceph-mon[33017]: pgmap v19834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:28 smithi194 ceph-mon[33017]: pgmap v19835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36747.264210 secs 2024-04-26T22:03:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:28 smithi033 ceph-mon[27845]: pgmap v19835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36747.264210 secs 2024-04-26T22:03:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:28 smithi096 ceph-mon[32753]: pgmap v19835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36747.264210 secs 2024-04-26T22:03:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:29.789+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:29.789+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:29.789+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:03:30.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36750.235150 secs 2024-04-26T22:03:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36750.235150 secs 2024-04-26T22:03:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36750.235150 secs 2024-04-26T22:03:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:30 smithi194 ceph-mon[33017]: pgmap v19836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:30 smithi033 ceph-mon[27845]: pgmap v19836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:30 smithi096 ceph-mon[32753]: pgmap v19836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:32 smithi194 ceph-mon[33017]: pgmap v19837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:32 smithi033 ceph-mon[27845]: pgmap v19837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:32 smithi096 ceph-mon[32753]: pgmap v19837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36752.264271 secs 2024-04-26T22:03:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36752.264271 secs 2024-04-26T22:03:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36752.264271 secs 2024-04-26T22:03:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:34 smithi194 ceph-mon[33017]: pgmap v19838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36755.235278 secs 2024-04-26T22:03:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:34 smithi033 ceph-mon[27845]: pgmap v19838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36755.235278 secs 2024-04-26T22:03:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:34 smithi096 ceph-mon[32753]: pgmap v19838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36755.235278 secs 2024-04-26T22:03:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:36 smithi194 ceph-mon[33017]: pgmap v19839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:36 smithi033 ceph-mon[27845]: pgmap v19839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:36 smithi096 ceph-mon[32753]: pgmap v19839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:38 smithi194 ceph-mon[33017]: pgmap v19840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36757.264354 secs 2024-04-26T22:03:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:38 smithi033 ceph-mon[27845]: pgmap v19840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36757.264354 secs 2024-04-26T22:03:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:38 smithi096 ceph-mon[32753]: pgmap v19840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36757.264354 secs 2024-04-26T22:03:40.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36760.235410 secs 2024-04-26T22:03:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36760.235410 secs 2024-04-26T22:03:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36760.235410 secs 2024-04-26T22:03:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:40 smithi194 ceph-mon[33017]: pgmap v19841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:03:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:40 smithi033 ceph-mon[27845]: pgmap v19841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:03:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:40 smithi096 ceph-mon[32753]: pgmap v19841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:03:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:41 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:41 smithi033 ceph-mon[27845]: pgmap v19842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:41 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:41 smithi096 ceph-mon[32753]: pgmap v19842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:41 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:41 smithi194 ceph-mon[33017]: pgmap v19842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:03:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:42 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:03:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:42 smithi194 ceph-mon[33017]: pgmap v19843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:03:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:42 smithi194 ceph-mon[33017]: pgmap v19844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:03:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:42 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:03:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:42 smithi033 ceph-mon[27845]: pgmap v19843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:03:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:42 smithi033 ceph-mon[27845]: pgmap v19844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:03:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:42 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:03:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:42 smithi096 ceph-mon[32753]: pgmap v19843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:03:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:42 smithi096 ceph-mon[32753]: pgmap v19844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:03:44.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36762.264441 secs 2024-04-26T22:03:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36762.264441 secs 2024-04-26T22:03:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36762.264441 secs 2024-04-26T22:03:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:44.792+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:44.792+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:03:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:44.792+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36765.235534 secs 2024-04-26T22:03:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:44 smithi194 ceph-mon[33017]: pgmap v19845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36765.235534 secs 2024-04-26T22:03:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:44 smithi033 ceph-mon[27845]: pgmap v19845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36765.235534 secs 2024-04-26T22:03:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:44 smithi096 ceph-mon[32753]: pgmap v19845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:47 smithi033 ceph-mon[27845]: pgmap v19846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:47 smithi096 ceph-mon[32753]: pgmap v19846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:47 smithi194 ceph-mon[33017]: pgmap v19846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36767.264525 secs 2024-04-26T22:03:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:49 smithi033 ceph-mon[27845]: pgmap v19847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:03:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36767.264525 secs 2024-04-26T22:03:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:49 smithi096 ceph-mon[32753]: pgmap v19847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:03:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36767.264525 secs 2024-04-26T22:03:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:49 smithi194 ceph-mon[33017]: pgmap v19847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:03:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36770.235618 secs 2024-04-26T22:03:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36770.235618 secs 2024-04-26T22:03:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36770.235618 secs 2024-04-26T22:03:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:51 smithi033 ceph-mon[27845]: pgmap v19848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:51 smithi096 ceph-mon[32753]: pgmap v19848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:51 smithi194 ceph-mon[33017]: pgmap v19848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:03:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:53 smithi033 ceph-mon[27845]: pgmap v19849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:03:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:53 smithi096 ceph-mon[32753]: pgmap v19849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:03:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:53 smithi194 ceph-mon[33017]: pgmap v19849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:03:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36772.264608 secs 2024-04-26T22:03:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36772.264608 secs 2024-04-26T22:03:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36772.264608 secs 2024-04-26T22:03:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36775.235742 secs 2024-04-26T22:03:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:55 smithi033 ceph-mon[27845]: pgmap v19850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36775.235742 secs 2024-04-26T22:03:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:55 smithi096 ceph-mon[32753]: pgmap v19850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36775.235742 secs 2024-04-26T22:03:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:55 smithi194 ceph-mon[33017]: pgmap v19850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:03:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:57 smithi096 ceph-mon[32753]: pgmap v19851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:57 smithi194 ceph-mon[33017]: pgmap v19851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:57 smithi033 ceph-mon[27845]: pgmap v19851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36777.264668 secs 2024-04-26T22:03:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36777.264668 secs 2024-04-26T22:03:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36777.264668 secs 2024-04-26T22:03:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:59.800+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:59.800+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:03:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:03:59.800+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:03:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:03:59 smithi194 ceph-mon[33017]: pgmap v19852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:03:59 smithi033 ceph-mon[27845]: pgmap v19852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:03:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:03:59 smithi096 ceph-mon[32753]: pgmap v19852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36780.235861 secs 2024-04-26T22:04:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36780.235861 secs 2024-04-26T22:04:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36780.235861 secs 2024-04-26T22:04:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:01 smithi194 ceph-mon[33017]: pgmap v19853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:01 smithi033 ceph-mon[27845]: pgmap v19853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:01 smithi096 ceph-mon[32753]: pgmap v19853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:03 smithi194 ceph-mon[33017]: pgmap v19854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36782.264753 secs 2024-04-26T22:04:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:03 smithi033 ceph-mon[27845]: pgmap v19854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36782.264753 secs 2024-04-26T22:04:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:03 smithi096 ceph-mon[32753]: pgmap v19854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36782.264753 secs 2024-04-26T22:04:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36785.235979 secs 2024-04-26T22:04:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:05 smithi194 ceph-mon[33017]: pgmap v19855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36785.235979 secs 2024-04-26T22:04:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:05 smithi033 ceph-mon[27845]: pgmap v19855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36785.235979 secs 2024-04-26T22:04:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:05 smithi096 ceph-mon[32753]: pgmap v19855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:07 smithi194 ceph-mon[33017]: pgmap v19856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:07 smithi033 ceph-mon[27845]: pgmap v19856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:07 smithi096 ceph-mon[32753]: pgmap v19856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36787.264855 secs 2024-04-26T22:04:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36787.264855 secs 2024-04-26T22:04:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36787.264855 secs 2024-04-26T22:04:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:09 smithi194 ceph-mon[33017]: pgmap v19857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:09 smithi033 ceph-mon[27845]: pgmap v19857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:09 smithi096 ceph-mon[32753]: pgmap v19857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36790.236101 secs 2024-04-26T22:04:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36790.236101 secs 2024-04-26T22:04:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36790.236101 secs 2024-04-26T22:04:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:11 smithi194 ceph-mon[33017]: pgmap v19858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:11 smithi033 ceph-mon[27845]: pgmap v19858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:11 smithi096 ceph-mon[32753]: pgmap v19858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:13 smithi194 ceph-mon[33017]: pgmap v19859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36792.264939 secs 2024-04-26T22:04:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:13 smithi033 ceph-mon[27845]: pgmap v19859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:13.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36792.264939 secs 2024-04-26T22:04:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:13 smithi096 ceph-mon[32753]: pgmap v19859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36792.264939 secs 2024-04-26T22:04:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:14.813+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:14.813+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:04:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:14.813+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36795.236226 secs 2024-04-26T22:04:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36795.236226 secs 2024-04-26T22:04:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36795.236226 secs 2024-04-26T22:04:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:15 smithi194 ceph-mon[33017]: pgmap v19860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:15 smithi033 ceph-mon[27845]: pgmap v19860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:15 smithi096 ceph-mon[32753]: pgmap v19860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:17 smithi194 ceph-mon[33017]: pgmap v19861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:17 smithi033 ceph-mon[27845]: pgmap v19861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:17 smithi096 ceph-mon[32753]: pgmap v19861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36797.265000 secs 2024-04-26T22:04:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36797.265000 secs 2024-04-26T22:04:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36797.265000 secs 2024-04-26T22:04:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:19 smithi194 ceph-mon[33017]: pgmap v19862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36800.236349 secs 2024-04-26T22:04:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:19 smithi033 ceph-mon[27845]: pgmap v19862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36800.236349 secs 2024-04-26T22:04:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:19 smithi096 ceph-mon[32753]: pgmap v19862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36800.236349 secs 2024-04-26T22:04:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:21 smithi194 ceph-mon[33017]: pgmap v19863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:21 smithi033 ceph-mon[27845]: pgmap v19863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:21 smithi096 ceph-mon[32753]: pgmap v19863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:23 smithi194 ceph-mon[33017]: pgmap v19864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36802.265091 secs 2024-04-26T22:04:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:23 smithi033 ceph-mon[27845]: pgmap v19864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36802.265091 secs 2024-04-26T22:04:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:23 smithi096 ceph-mon[32753]: pgmap v19864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36802.265091 secs 2024-04-26T22:04:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36805.236474 secs 2024-04-26T22:04:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36805.236474 secs 2024-04-26T22:04:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36805.236474 secs 2024-04-26T22:04:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:25 smithi194 ceph-mon[33017]: pgmap v19865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:25 smithi033 ceph-mon[27845]: pgmap v19865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:25 smithi096 ceph-mon[32753]: pgmap v19865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:27 smithi194 ceph-mon[33017]: pgmap v19866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:27 smithi033 ceph-mon[27845]: pgmap v19866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:27 smithi096 ceph-mon[32753]: pgmap v19866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:28.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36807.265178 secs 2024-04-26T22:04:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36807.265178 secs 2024-04-26T22:04:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36807.265178 secs 2024-04-26T22:04:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:29.826+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:29.826+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:29.826+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:04:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:29 smithi194 ceph-mon[33017]: pgmap v19867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36810.236571 secs 2024-04-26T22:04:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:29 smithi033 ceph-mon[27845]: pgmap v19867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36810.236571 secs 2024-04-26T22:04:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:29 smithi096 ceph-mon[32753]: pgmap v19867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36810.236571 secs 2024-04-26T22:04:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:31 smithi194 ceph-mon[33017]: pgmap v19868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:31 smithi033 ceph-mon[27845]: pgmap v19868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:31 smithi096 ceph-mon[32753]: pgmap v19868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:33 smithi194 ceph-mon[33017]: pgmap v19869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36812.265282 secs 2024-04-26T22:04:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:33 smithi033 ceph-mon[27845]: pgmap v19869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36812.265282 secs 2024-04-26T22:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:33 smithi096 ceph-mon[32753]: pgmap v19869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36812.265282 secs 2024-04-26T22:04:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36815.236694 secs 2024-04-26T22:04:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36815.236694 secs 2024-04-26T22:04:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36815.236694 secs 2024-04-26T22:04:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:35 smithi033 ceph-mon[27845]: pgmap v19870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:35 smithi096 ceph-mon[32753]: pgmap v19870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:36.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:35 smithi194 ceph-mon[33017]: pgmap v19870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:37 smithi033 ceph-mon[27845]: pgmap v19871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:37 smithi096 ceph-mon[32753]: pgmap v19871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:37 smithi194 ceph-mon[33017]: pgmap v19871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36817.265382 secs 2024-04-26T22:04:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36817.265382 secs 2024-04-26T22:04:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36817.265382 secs 2024-04-26T22:04:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:39 smithi033 ceph-mon[27845]: pgmap v19872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36820.236822 secs 2024-04-26T22:04:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:39 smithi096 ceph-mon[32753]: pgmap v19872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36820.236822 secs 2024-04-26T22:04:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:39 smithi194 ceph-mon[33017]: pgmap v19872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:04:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36820.236822 secs 2024-04-26T22:04:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:41 smithi033 ceph-mon[27845]: pgmap v19873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:41 smithi096 ceph-mon[32753]: pgmap v19873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:41 smithi194 ceph-mon[33017]: pgmap v19873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:04:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:43 smithi033 ceph-mon[27845]: pgmap v19874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:04:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36822.265449 secs 2024-04-26T22:04:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:43 smithi096 ceph-mon[32753]: pgmap v19874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:04:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36822.265449 secs 2024-04-26T22:04:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:43 smithi194 ceph-mon[33017]: pgmap v19874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:04:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36822.265449 secs 2024-04-26T22:04:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:44.839+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:44.839+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:04:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:44.839+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36825.236977 secs 2024-04-26T22:04:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:44 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36825.236977 secs 2024-04-26T22:04:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:44 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36825.236977 secs 2024-04-26T22:04:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:44 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:45 smithi033 ceph-mon[27845]: pgmap v19875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:04:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:04:46.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:45 smithi096 ceph-mon[32753]: pgmap v19875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:04:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:04:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:45 smithi194 ceph-mon[33017]: pgmap v19875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:04:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:04:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:04:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:04:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:46 smithi033 ceph-mon[27845]: pgmap v19876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T22:04:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:46 smithi033 ceph-mon[27845]: pgmap v19877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:46 smithi096 ceph-mon[32753]: pgmap v19876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T22:04:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:46 smithi096 ceph-mon[32753]: pgmap v19877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:46 smithi194 ceph-mon[33017]: pgmap v19876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T22:04:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:46 smithi194 ceph-mon[33017]: pgmap v19877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:48 smithi033 ceph-mon[27845]: pgmap v19878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36827.265548 secs 2024-04-26T22:04:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:48 smithi096 ceph-mon[32753]: pgmap v19878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36827.265548 secs 2024-04-26T22:04:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:48 smithi194 ceph-mon[33017]: pgmap v19878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36827.265548 secs 2024-04-26T22:04:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36830.237096 secs 2024-04-26T22:04:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:49 smithi033 ceph-mon[27845]: pgmap v19879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:04:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36830.237096 secs 2024-04-26T22:04:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:49 smithi096 ceph-mon[32753]: pgmap v19879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:04:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36830.237096 secs 2024-04-26T22:04:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:49 smithi194 ceph-mon[33017]: pgmap v19879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:04:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:51 smithi033 ceph-mon[27845]: pgmap v19880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:52.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:51 smithi096 ceph-mon[32753]: pgmap v19880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:51 smithi194 ceph-mon[33017]: pgmap v19880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:04:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:53 smithi033 ceph-mon[27845]: pgmap v19881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:04:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36832.265638 secs 2024-04-26T22:04:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:53 smithi096 ceph-mon[32753]: pgmap v19881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:04:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36832.265638 secs 2024-04-26T22:04:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:53 smithi194 ceph-mon[33017]: pgmap v19881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:04:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36832.265638 secs 2024-04-26T22:04:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36835.237235 secs 2024-04-26T22:04:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36835.237235 secs 2024-04-26T22:04:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36835.237235 secs 2024-04-26T22:04:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:55 smithi194 ceph-mon[33017]: pgmap v19882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:04:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:55 smithi033 ceph-mon[27845]: pgmap v19882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:04:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:55 smithi096 ceph-mon[32753]: pgmap v19882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:04:58.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:57 smithi096 ceph-mon[32753]: pgmap v19883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:57 smithi194 ceph-mon[33017]: pgmap v19883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:57 smithi033 ceph-mon[27845]: pgmap v19883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:04:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36837.265740 secs 2024-04-26T22:04:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36837.265740 secs 2024-04-26T22:04:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36837.265740 secs 2024-04-26T22:04:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:59.854+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:59.855+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:04:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:04:59.855+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:05:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36840.237362 secs 2024-04-26T22:05:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:04:59 smithi194 ceph-mon[33017]: pgmap v19884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36840.237362 secs 2024-04-26T22:05:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:04:59 smithi033 ceph-mon[27845]: pgmap v19884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36840.237362 secs 2024-04-26T22:05:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:04:59 smithi096 ceph-mon[32753]: pgmap v19884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:02.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:01 smithi033 ceph-mon[27845]: pgmap v19885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:01 smithi194 ceph-mon[33017]: pgmap v19885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:01 smithi096 ceph-mon[32753]: pgmap v19885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:03 smithi194 ceph-mon[33017]: pgmap v19886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36842.265840 secs 2024-04-26T22:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:03 smithi033 ceph-mon[27845]: pgmap v19886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36842.265840 secs 2024-04-26T22:05:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:03 smithi096 ceph-mon[32753]: pgmap v19886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36842.265840 secs 2024-04-26T22:05:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36845.237425 secs 2024-04-26T22:05:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36845.237425 secs 2024-04-26T22:05:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36845.237425 secs 2024-04-26T22:05:06.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:05 smithi194 ceph-mon[33017]: pgmap v19887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:05 smithi033 ceph-mon[27845]: pgmap v19887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:05 smithi096 ceph-mon[32753]: pgmap v19887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:07 smithi194 ceph-mon[33017]: pgmap v19888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:07 smithi033 ceph-mon[27845]: pgmap v19888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:07 smithi096 ceph-mon[32753]: pgmap v19888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36847.265942 secs 2024-04-26T22:05:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36847.265942 secs 2024-04-26T22:05:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36847.265942 secs 2024-04-26T22:05:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36850.237549 secs 2024-04-26T22:05:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:09 smithi194 ceph-mon[33017]: pgmap v19889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36850.237549 secs 2024-04-26T22:05:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:09 smithi033 ceph-mon[27845]: pgmap v19889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36850.237549 secs 2024-04-26T22:05:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:09 smithi096 ceph-mon[32753]: pgmap v19889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:12 smithi194 ceph-mon[33017]: pgmap v19890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:12 smithi033 ceph-mon[27845]: pgmap v19890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:12 smithi096 ceph-mon[32753]: pgmap v19890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:14 smithi194 ceph-mon[33017]: pgmap v19891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36852.266045 secs 2024-04-26T22:05:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:14 smithi033 ceph-mon[27845]: pgmap v19891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36852.266045 secs 2024-04-26T22:05:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:14 smithi096 ceph-mon[32753]: pgmap v19891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36852.266045 secs 2024-04-26T22:05:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:14.865+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:14.865+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:14.865+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:05:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36855.237631 secs 2024-04-26T22:05:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36855.237631 secs 2024-04-26T22:05:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36855.237631 secs 2024-04-26T22:05:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:16 smithi194 ceph-mon[33017]: pgmap v19892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:16 smithi033 ceph-mon[27845]: pgmap v19892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:16 smithi096 ceph-mon[32753]: pgmap v19892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:18 smithi194 ceph-mon[33017]: pgmap v19893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:18 smithi033 ceph-mon[27845]: pgmap v19893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:18 smithi096 ceph-mon[32753]: pgmap v19893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36857.266102 secs 2024-04-26T22:05:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36857.266102 secs 2024-04-26T22:05:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36857.266102 secs 2024-04-26T22:05:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36860.237743 secs 2024-04-26T22:05:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:20 smithi194 ceph-mon[33017]: pgmap v19894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36860.237743 secs 2024-04-26T22:05:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:20 smithi033 ceph-mon[27845]: pgmap v19894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36860.237743 secs 2024-04-26T22:05:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:20 smithi096 ceph-mon[32753]: pgmap v19894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:22 smithi194 ceph-mon[33017]: pgmap v19895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:22 smithi033 ceph-mon[27845]: pgmap v19895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:22 smithi096 ceph-mon[32753]: pgmap v19895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:24 smithi194 ceph-mon[33017]: pgmap v19896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36862.266192 secs 2024-04-26T22:05:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:24 smithi033 ceph-mon[27845]: pgmap v19896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36862.266192 secs 2024-04-26T22:05:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:24 smithi096 ceph-mon[32753]: pgmap v19896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36862.266192 secs 2024-04-26T22:05:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36865.237861 secs 2024-04-26T22:05:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36865.237861 secs 2024-04-26T22:05:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36865.237861 secs 2024-04-26T22:05:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:26 smithi194 ceph-mon[33017]: pgmap v19897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:26 smithi033 ceph-mon[27845]: pgmap v19897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:26 smithi096 ceph-mon[32753]: pgmap v19897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:28.073+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T22:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:28.073+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:28.074+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 288 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004910 con 0x564385821900 2024-04-26T22:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:28.074+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 288 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:05:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:28 smithi033 ceph-mon[27845]: pgmap v19898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:28 smithi096 ceph-mon[32753]: pgmap v19898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:28 smithi194 ceph-mon[33017]: pgmap v19898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36867.266254 secs 2024-04-26T22:05:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36867.266254 secs 2024-04-26T22:05:29.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36867.266254 secs 2024-04-26T22:05:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:29.880+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:29.880+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:29.880+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:05:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36870.237976 secs 2024-04-26T22:05:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:30 smithi033 ceph-mon[27845]: pgmap v19899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36870.237976 secs 2024-04-26T22:05:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:30 smithi096 ceph-mon[32753]: pgmap v19899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36870.237976 secs 2024-04-26T22:05:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:30 smithi194 ceph-mon[33017]: pgmap v19899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:32 smithi033 ceph-mon[27845]: pgmap v19900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:32 smithi096 ceph-mon[32753]: pgmap v19900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:32 smithi194 ceph-mon[33017]: pgmap v19900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:34 smithi033 ceph-mon[27845]: pgmap v19901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36872.266338 secs 2024-04-26T22:05:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:34 smithi096 ceph-mon[32753]: pgmap v19901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36872.266338 secs 2024-04-26T22:05:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:34 smithi194 ceph-mon[33017]: pgmap v19901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36872.266338 secs 2024-04-26T22:05:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36875.238127 secs 2024-04-26T22:05:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36875.238127 secs 2024-04-26T22:05:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36875.238127 secs 2024-04-26T22:05:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:36 smithi033 ceph-mon[27845]: pgmap v19902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:36 smithi096 ceph-mon[32753]: pgmap v19902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:36.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:36 smithi194 ceph-mon[33017]: pgmap v19902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:38 smithi033 ceph-mon[27845]: pgmap v19903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:38 smithi096 ceph-mon[32753]: pgmap v19903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:38 smithi194 ceph-mon[33017]: pgmap v19903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36877.266421 secs 2024-04-26T22:05:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36877.266421 secs 2024-04-26T22:05:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36877.266421 secs 2024-04-26T22:05:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36880.238239 secs 2024-04-26T22:05:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:40 smithi033 ceph-mon[27845]: pgmap v19904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36880.238239 secs 2024-04-26T22:05:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:40 smithi096 ceph-mon[32753]: pgmap v19904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36880.238239 secs 2024-04-26T22:05:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:40 smithi194 ceph-mon[33017]: pgmap v19904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:42 smithi033 ceph-mon[27845]: pgmap v19905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:42 smithi096 ceph-mon[32753]: pgmap v19905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:42 smithi194 ceph-mon[33017]: pgmap v19905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:44 smithi096 ceph-mon[32753]: pgmap v19906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36882.266505 secs 2024-04-26T22:05:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:44 smithi194 ceph-mon[33017]: pgmap v19906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36882.266505 secs 2024-04-26T22:05:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:44 smithi033 ceph-mon[27845]: pgmap v19906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36882.266505 secs 2024-04-26T22:05:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:44.896+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:44.896+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:44.896+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:05:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36885.238355 secs 2024-04-26T22:05:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36885.238355 secs 2024-04-26T22:05:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36885.238355 secs 2024-04-26T22:05:46.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:46 smithi033 ceph-mon[27845]: pgmap v19907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:46.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:46 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:05:46.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:46 smithi194 ceph-mon[33017]: pgmap v19907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:46 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:05:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:46 smithi096 ceph-mon[32753]: pgmap v19907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:46 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:05:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: pgmap v19908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36887.266593 secs 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: pgmap v19908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36887.266593 secs 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: pgmap v19908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36887.266593 secs 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:49.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:49 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:05:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:49 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:05:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36890.238462 secs 2024-04-26T22:05:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:49 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:05:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:49 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:05:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:49 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:05:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:49 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:05:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36890.238462 secs 2024-04-26T22:05:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:49 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:05:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:49 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:05:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:49 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:05:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:49 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:05:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36890.238462 secs 2024-04-26T22:05:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:49 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:05:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:49 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:05:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:50 smithi033 ceph-mon[27845]: pgmap v19909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:50 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:50 smithi096 ceph-mon[32753]: pgmap v19909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:50 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:50 smithi194 ceph-mon[33017]: pgmap v19909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:05:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:50 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:05:52.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:05:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:05:52.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:05:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:05:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:05:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:05:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:52 smithi033 ceph-mon[27845]: pgmap v19910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:52 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:05:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:52 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:05:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:52 smithi033 ceph-mon[27845]: pgmap v19911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:05:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:52 smithi033 ceph-mon[27845]: pgmap v19912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:52 smithi096 ceph-mon[32753]: pgmap v19910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:52 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:05:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:52 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:05:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:52 smithi096 ceph-mon[32753]: pgmap v19911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:05:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:52 smithi096 ceph-mon[32753]: pgmap v19912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:52 smithi194 ceph-mon[33017]: pgmap v19910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:05:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:52 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:05:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:52 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:05:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:52 smithi194 ceph-mon[33017]: pgmap v19911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:05:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:52 smithi194 ceph-mon[33017]: pgmap v19912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36892.266679 secs 2024-04-26T22:05:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36892.266679 secs 2024-04-26T22:05:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36892.266679 secs 2024-04-26T22:05:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:54 smithi033 ceph-mon[27845]: pgmap v19913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:05:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36895.238601 secs 2024-04-26T22:05:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:54 smithi096 ceph-mon[32753]: pgmap v19913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:05:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36895.238601 secs 2024-04-26T22:05:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:54 smithi194 ceph-mon[33017]: pgmap v19913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:05:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36895.238601 secs 2024-04-26T22:05:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:56 smithi033 ceph-mon[27845]: pgmap v19914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:56 smithi096 ceph-mon[32753]: pgmap v19914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:56 smithi194 ceph-mon[33017]: pgmap v19914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:58 smithi033 ceph-mon[27845]: pgmap v19915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36897.266740 secs 2024-04-26T22:05:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:58 smithi096 ceph-mon[32753]: pgmap v19915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36897.266740 secs 2024-04-26T22:05:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:58 smithi194 ceph-mon[33017]: pgmap v19915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:05:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36897.266740 secs 2024-04-26T22:05:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:59.912+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:59.912+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:05:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:05:59.912+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:06:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36900.238716 secs 2024-04-26T22:06:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:05:59 smithi096 ceph-mon[32753]: pgmap v19916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:06:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36900.238716 secs 2024-04-26T22:06:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:05:59 smithi194 ceph-mon[33017]: pgmap v19916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:06:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36900.238716 secs 2024-04-26T22:06:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:05:59 smithi033 ceph-mon[27845]: pgmap v19916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:06:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:02 smithi033 ceph-mon[27845]: pgmap v19917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:06:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:02 smithi096 ceph-mon[32753]: pgmap v19917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:06:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:02 smithi194 ceph-mon[33017]: pgmap v19917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:06:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36902.266825 secs 2024-04-26T22:06:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:04 smithi033 ceph-mon[27845]: pgmap v19918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36902.266825 secs 2024-04-26T22:06:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:04 smithi096 ceph-mon[32753]: pgmap v19918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36902.266825 secs 2024-04-26T22:06:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:04 smithi194 ceph-mon[33017]: pgmap v19918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36905.238833 secs 2024-04-26T22:06:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36905.238833 secs 2024-04-26T22:06:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36905.238833 secs 2024-04-26T22:06:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:06 smithi033 ceph-mon[27845]: pgmap v19919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:06 smithi096 ceph-mon[32753]: pgmap v19919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:06.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:06 smithi194 ceph-mon[33017]: pgmap v19919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:08 smithi096 ceph-mon[32753]: pgmap v19920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:08 smithi194 ceph-mon[33017]: pgmap v19920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:08 smithi033 ceph-mon[27845]: pgmap v19920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36907.266908 secs 2024-04-26T22:06:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36907.266908 secs 2024-04-26T22:06:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36907.266908 secs 2024-04-26T22:06:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36910.238917 secs 2024-04-26T22:06:10.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:10 smithi194 ceph-mon[33017]: pgmap v19921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36910.238917 secs 2024-04-26T22:06:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:10 smithi033 ceph-mon[27845]: pgmap v19921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36910.238917 secs 2024-04-26T22:06:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:10 smithi096 ceph-mon[32753]: pgmap v19921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:12 smithi194 ceph-mon[33017]: pgmap v19922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:12 smithi033 ceph-mon[27845]: pgmap v19922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:12 smithi096 ceph-mon[32753]: pgmap v19922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36912.266995 secs 2024-04-26T22:06:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36912.266995 secs 2024-04-26T22:06:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36912.266995 secs 2024-04-26T22:06:14.852 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:14 smithi033 ceph-mon[27845]: pgmap v19923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:14 smithi194 ceph-mon[33017]: pgmap v19923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:14.926+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:14.926+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:14.926+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:06:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:14 smithi096 ceph-mon[32753]: pgmap v19923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36915.238993 secs 2024-04-26T22:06:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36915.238993 secs 2024-04-26T22:06:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36915.238993 secs 2024-04-26T22:06:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:16 smithi194 ceph-mon[33017]: pgmap v19924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:16 smithi033 ceph-mon[27845]: pgmap v19924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:16 smithi096 ceph-mon[32753]: pgmap v19924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:18 smithi194 ceph-mon[33017]: pgmap v19925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36917.267078 secs 2024-04-26T22:06:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:18 smithi033 ceph-mon[27845]: pgmap v19925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36917.267078 secs 2024-04-26T22:06:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:18 smithi096 ceph-mon[32753]: pgmap v19925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36917.267078 secs 2024-04-26T22:06:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36920.239117 secs 2024-04-26T22:06:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:20 smithi194 ceph-mon[33017]: pgmap v19926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36920.239117 secs 2024-04-26T22:06:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:20 smithi033 ceph-mon[27845]: pgmap v19926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36920.239117 secs 2024-04-26T22:06:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:20 smithi096 ceph-mon[32753]: pgmap v19926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:22 smithi194 ceph-mon[33017]: pgmap v19927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:22 smithi033 ceph-mon[27845]: pgmap v19927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:22 smithi096 ceph-mon[32753]: pgmap v19927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36922.267163 secs 2024-04-26T22:06:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36922.267163 secs 2024-04-26T22:06:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36922.267163 secs 2024-04-26T22:06:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:24 smithi194 ceph-mon[33017]: pgmap v19928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:24 smithi033 ceph-mon[27845]: pgmap v19928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:24 smithi096 ceph-mon[32753]: pgmap v19928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36925.239267 secs 2024-04-26T22:06:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36925.239267 secs 2024-04-26T22:06:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36925.239267 secs 2024-04-26T22:06:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:26 smithi194 ceph-mon[33017]: pgmap v19929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:26 smithi033 ceph-mon[27845]: pgmap v19929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:26 smithi096 ceph-mon[32753]: pgmap v19929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:28 smithi194 ceph-mon[33017]: pgmap v19930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36927.267242 secs 2024-04-26T22:06:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:28 smithi033 ceph-mon[27845]: pgmap v19930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36927.267242 secs 2024-04-26T22:06:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:28 smithi096 ceph-mon[32753]: pgmap v19930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36927.267242 secs 2024-04-26T22:06:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36930.239373 secs 2024-04-26T22:06:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36930.239373 secs 2024-04-26T22:06:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36930.239373 secs 2024-04-26T22:06:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:29.937+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:29.937+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:29.937+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:06:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:30 smithi194 ceph-mon[33017]: pgmap v19931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:30 smithi033 ceph-mon[27845]: pgmap v19931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:30 smithi096 ceph-mon[32753]: pgmap v19931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:32 smithi194 ceph-mon[33017]: pgmap v19932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:32 smithi033 ceph-mon[27845]: pgmap v19932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:32 smithi096 ceph-mon[32753]: pgmap v19932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36932.267327 secs 2024-04-26T22:06:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36932.267327 secs 2024-04-26T22:06:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36932.267327 secs 2024-04-26T22:06:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:34 smithi194 ceph-mon[33017]: pgmap v19933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36935.239438 secs 2024-04-26T22:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:34 smithi033 ceph-mon[27845]: pgmap v19933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36935.239438 secs 2024-04-26T22:06:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:34 smithi096 ceph-mon[32753]: pgmap v19933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36935.239438 secs 2024-04-26T22:06:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:36 smithi194 ceph-mon[33017]: pgmap v19934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:36 smithi033 ceph-mon[27845]: pgmap v19934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:36 smithi096 ceph-mon[32753]: pgmap v19934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:38 smithi194 ceph-mon[33017]: pgmap v19935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36937.267413 secs 2024-04-26T22:06:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:38 smithi033 ceph-mon[27845]: pgmap v19935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36937.267413 secs 2024-04-26T22:06:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:38 smithi096 ceph-mon[32753]: pgmap v19935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36937.267413 secs 2024-04-26T22:06:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36940.239566 secs 2024-04-26T22:06:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36940.239566 secs 2024-04-26T22:06:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36940.239566 secs 2024-04-26T22:06:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:40 smithi194 ceph-mon[33017]: pgmap v19936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:40 smithi033 ceph-mon[27845]: pgmap v19936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:40 smithi096 ceph-mon[32753]: pgmap v19936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:42 smithi194 ceph-mon[33017]: pgmap v19937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:42 smithi033 ceph-mon[27845]: pgmap v19937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:42 smithi096 ceph-mon[32753]: pgmap v19937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36942.267495 secs 2024-04-26T22:06:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36942.267495 secs 2024-04-26T22:06:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36942.267495 secs 2024-04-26T22:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:44 smithi194 ceph-mon[33017]: pgmap v19938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36945.239702 secs 2024-04-26T22:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:44 smithi033 ceph-mon[27845]: pgmap v19938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36945.239702 secs 2024-04-26T22:06:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:44 smithi096 ceph-mon[32753]: pgmap v19938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36945.239702 secs 2024-04-26T22:06:44.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:44.953+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:44.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:44.953+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:44.953 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:44.953+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:06:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:46 smithi194 ceph-mon[33017]: pgmap v19939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:46 smithi033 ceph-mon[27845]: pgmap v19939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:46 smithi096 ceph-mon[32753]: pgmap v19939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:48 smithi033 ceph-mon[27845]: pgmap v19940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36947.267554 secs 2024-04-26T22:06:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:48 smithi096 ceph-mon[32753]: pgmap v19940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36947.267554 secs 2024-04-26T22:06:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:48 smithi194 ceph-mon[33017]: pgmap v19940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36947.267554 secs 2024-04-26T22:06:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36950.239834 secs 2024-04-26T22:06:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36950.239834 secs 2024-04-26T22:06:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36950.239834 secs 2024-04-26T22:06:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:50 smithi033 ceph-mon[27845]: pgmap v19941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:50 smithi096 ceph-mon[32753]: pgmap v19941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:50 smithi194 ceph-mon[33017]: pgmap v19941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:06:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:52 smithi033 ceph-mon[27845]: pgmap v19942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:06:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:52 smithi096 ceph-mon[32753]: pgmap v19942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:06:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:52 smithi194 ceph-mon[33017]: pgmap v19942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:06:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36952.267641 secs 2024-04-26T22:06:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:06:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:06:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:06:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36952.267641 secs 2024-04-26T22:06:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:06:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:06:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:06:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36952.267641 secs 2024-04-26T22:06:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:06:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:06:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:06:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:54 smithi033 ceph-mon[27845]: pgmap v19943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:54 smithi033 ceph-mon[27845]: pgmap v19944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36955.239970 secs 2024-04-26T22:06:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:54 smithi096 ceph-mon[32753]: pgmap v19943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:54 smithi096 ceph-mon[32753]: pgmap v19944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36955.239970 secs 2024-04-26T22:06:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:54 smithi194 ceph-mon[33017]: pgmap v19943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:06:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:54 smithi194 ceph-mon[33017]: pgmap v19944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36955.239970 secs 2024-04-26T22:06:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:56 smithi033 ceph-mon[27845]: pgmap v19945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:56 smithi096 ceph-mon[32753]: pgmap v19945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:56 smithi194 ceph-mon[33017]: pgmap v19945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:58 smithi033 ceph-mon[27845]: pgmap v19946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36957.267725 secs 2024-04-26T22:06:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:58 smithi096 ceph-mon[32753]: pgmap v19946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36957.267725 secs 2024-04-26T22:06:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:58 smithi194 ceph-mon[33017]: pgmap v19946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:06:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36957.267725 secs 2024-04-26T22:06:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:06:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36960.240102 secs 2024-04-26T22:06:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:06:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36960.240102 secs 2024-04-26T22:06:59.954 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:59.955+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:59.955+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:06:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:06:59.955+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:07:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:06:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36960.240102 secs 2024-04-26T22:07:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:00 smithi033 ceph-mon[27845]: pgmap v19947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:07:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:00 smithi096 ceph-mon[32753]: pgmap v19947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:07:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:00 smithi194 ceph-mon[33017]: pgmap v19947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:07:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:02 smithi033 ceph-mon[27845]: pgmap v19948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:02 smithi096 ceph-mon[32753]: pgmap v19948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:02 smithi194 ceph-mon[33017]: pgmap v19948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36962.267830 secs 2024-04-26T22:07:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36962.267830 secs 2024-04-26T22:07:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36962.267830 secs 2024-04-26T22:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:04 smithi096 ceph-mon[32753]: pgmap v19949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:07:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36965.240221 secs 2024-04-26T22:07:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:04 smithi194 ceph-mon[33017]: pgmap v19949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:07:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36965.240221 secs 2024-04-26T22:07:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:04 smithi033 ceph-mon[27845]: pgmap v19949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:07:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36965.240221 secs 2024-04-26T22:07:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:06 smithi194 ceph-mon[33017]: pgmap v19950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:06 smithi033 ceph-mon[27845]: pgmap v19950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:06 smithi096 ceph-mon[32753]: pgmap v19950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:08 smithi194 ceph-mon[33017]: pgmap v19951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36967.267888 secs 2024-04-26T22:07:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:08 smithi033 ceph-mon[27845]: pgmap v19951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36967.267888 secs 2024-04-26T22:07:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:08 smithi096 ceph-mon[32753]: pgmap v19951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36967.267888 secs 2024-04-26T22:07:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36970.240297 secs 2024-04-26T22:07:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36970.240297 secs 2024-04-26T22:07:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36970.240297 secs 2024-04-26T22:07:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:10 smithi194 ceph-mon[33017]: pgmap v19952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:10 smithi033 ceph-mon[27845]: pgmap v19952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:10 smithi096 ceph-mon[32753]: pgmap v19952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:12 smithi194 ceph-mon[33017]: pgmap v19953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:12 smithi033 ceph-mon[27845]: pgmap v19953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:12 smithi096 ceph-mon[32753]: pgmap v19953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36972.267949 secs 2024-04-26T22:07:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36972.267949 secs 2024-04-26T22:07:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36972.267949 secs 2024-04-26T22:07:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:14.970+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:07:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:14.970+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:07:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:14.970+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:07:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:14 smithi194 ceph-mon[33017]: pgmap v19954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36975.240415 secs 2024-04-26T22:07:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:14 smithi033 ceph-mon[27845]: pgmap v19954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36975.240415 secs 2024-04-26T22:07:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:14 smithi096 ceph-mon[32753]: pgmap v19954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36975.240415 secs 2024-04-26T22:07:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:16 smithi194 ceph-mon[33017]: pgmap v19955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:16 smithi033 ceph-mon[27845]: pgmap v19955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:16 smithi096 ceph-mon[32753]: pgmap v19955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:18 smithi194 ceph-mon[33017]: pgmap v19956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36977.268010 secs 2024-04-26T22:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:18 smithi033 ceph-mon[27845]: pgmap v19956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36977.268010 secs 2024-04-26T22:07:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:18 smithi096 ceph-mon[32753]: pgmap v19956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36977.268010 secs 2024-04-26T22:07:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36980.240536 secs 2024-04-26T22:07:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36980.240536 secs 2024-04-26T22:07:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36980.240536 secs 2024-04-26T22:07:21.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:20 smithi033 ceph-mon[27845]: pgmap v19957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:20 smithi194 ceph-mon[33017]: pgmap v19957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:20 smithi096 ceph-mon[32753]: pgmap v19957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:22 smithi194 ceph-mon[33017]: pgmap v19958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:22 smithi033 ceph-mon[27845]: pgmap v19958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:22 smithi096 ceph-mon[32753]: pgmap v19958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36982.268099 secs 2024-04-26T22:07:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36982.268099 secs 2024-04-26T22:07:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36982.268099 secs 2024-04-26T22:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:24 smithi194 ceph-mon[33017]: pgmap v19959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36985.240689 secs 2024-04-26T22:07:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:24 smithi033 ceph-mon[27845]: pgmap v19959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36985.240689 secs 2024-04-26T22:07:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:24 smithi096 ceph-mon[32753]: pgmap v19959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36985.240689 secs 2024-04-26T22:07:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:26 smithi194 ceph-mon[33017]: pgmap v19960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:26 smithi033 ceph-mon[27845]: pgmap v19960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:26 smithi096 ceph-mon[32753]: pgmap v19960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:28 smithi194 ceph-mon[33017]: pgmap v19961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36987.268185 secs 2024-04-26T22:07:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:28 smithi033 ceph-mon[27845]: pgmap v19961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36987.268185 secs 2024-04-26T22:07:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:28 smithi096 ceph-mon[32753]: pgmap v19961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36987.268185 secs 2024-04-26T22:07:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:29.985+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:07:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:29.985+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:07:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:29.985+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:07:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36990.240804 secs 2024-04-26T22:07:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36990.240804 secs 2024-04-26T22:07:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36990.240804 secs 2024-04-26T22:07:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:30 smithi194 ceph-mon[33017]: pgmap v19962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:30 smithi033 ceph-mon[27845]: pgmap v19962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:30 smithi096 ceph-mon[32753]: pgmap v19962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:32 smithi194 ceph-mon[33017]: pgmap v19963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:32 smithi033 ceph-mon[27845]: pgmap v19963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:32 smithi096 ceph-mon[32753]: pgmap v19963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36992.268270 secs 2024-04-26T22:07:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36992.268270 secs 2024-04-26T22:07:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36992.268270 secs 2024-04-26T22:07:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:34 smithi194 ceph-mon[33017]: pgmap v19964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 36995.240939 secs 2024-04-26T22:07:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:34 smithi033 ceph-mon[27845]: pgmap v19964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 36995.240939 secs 2024-04-26T22:07:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:34 smithi096 ceph-mon[32753]: pgmap v19964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 36995.240939 secs 2024-04-26T22:07:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:36 smithi194 ceph-mon[33017]: pgmap v19965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:36 smithi033 ceph-mon[27845]: pgmap v19965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:36 smithi096 ceph-mon[32753]: pgmap v19965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:38 smithi194 ceph-mon[33017]: pgmap v19966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 36997.268356 secs 2024-04-26T22:07:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:38 smithi033 ceph-mon[27845]: pgmap v19966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 36997.268356 secs 2024-04-26T22:07:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:38 smithi096 ceph-mon[32753]: pgmap v19966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 36997.268356 secs 2024-04-26T22:07:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37000.241022 secs 2024-04-26T22:07:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37000.241022 secs 2024-04-26T22:07:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37000.241022 secs 2024-04-26T22:07:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:40 smithi194 ceph-mon[33017]: pgmap v19967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:40 smithi033 ceph-mon[27845]: pgmap v19967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:40 smithi096 ceph-mon[32753]: pgmap v19967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:42 smithi194 ceph-mon[33017]: pgmap v19968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:42 smithi033 ceph-mon[27845]: pgmap v19968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:42 smithi096 ceph-mon[32753]: pgmap v19968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37002.268441 secs 2024-04-26T22:07:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37002.268441 secs 2024-04-26T22:07:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37002.268441 secs 2024-04-26T22:07:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:45.001+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:07:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:45.001+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:07:45.002 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:07:45.001+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:44 smithi033 ceph-mon[27845]: pgmap v19969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37005.241139 secs 2024-04-26T22:07:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:44 smithi096 ceph-mon[32753]: pgmap v19969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37005.241139 secs 2024-04-26T22:07:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:44 smithi194 ceph-mon[33017]: pgmap v19969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37005.241139 secs 2024-04-26T22:07:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:46 smithi033 ceph-mon[27845]: pgmap v19970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:46 smithi096 ceph-mon[32753]: pgmap v19970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:46 smithi194 ceph-mon[33017]: pgmap v19970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:48 smithi033 ceph-mon[27845]: pgmap v19971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37007.268527 secs 2024-04-26T22:07:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:48 smithi096 ceph-mon[32753]: pgmap v19971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37007.268527 secs 2024-04-26T22:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:48 smithi194 ceph-mon[33017]: pgmap v19971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37007.268527 secs 2024-04-26T22:07:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37010.241253 secs 2024-04-26T22:07:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37010.241253 secs 2024-04-26T22:07:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37010.241253 secs 2024-04-26T22:07:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:50 smithi033 ceph-mon[27845]: pgmap v19972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:50 smithi096 ceph-mon[32753]: pgmap v19972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:50 smithi194 ceph-mon[33017]: pgmap v19972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:52 smithi033 ceph-mon[27845]: pgmap v19973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:52 smithi096 ceph-mon[32753]: pgmap v19973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:52 smithi194 ceph-mon[33017]: pgmap v19973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:07:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37012.268613 secs 2024-04-26T22:07:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:07:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37012.268613 secs 2024-04-26T22:07:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37012.268613 secs 2024-04-26T22:07:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:07:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:54 smithi033 ceph-mon[27845]: pgmap v19974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37015.241315 secs 2024-04-26T22:07:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:07:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:07:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:07:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:54 smithi096 ceph-mon[32753]: pgmap v19974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37015.241315 secs 2024-04-26T22:07:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:07:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:07:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:54 smithi194 ceph-mon[33017]: pgmap v19974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37015.241315 secs 2024-04-26T22:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:07:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:07:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:55 smithi033 ceph-mon[27845]: pgmap v19975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:07:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:55 smithi033 ceph-mon[27845]: pgmap v19976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:55 smithi096 ceph-mon[32753]: pgmap v19975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:07:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:55 smithi096 ceph-mon[32753]: pgmap v19976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:55 smithi194 ceph-mon[33017]: pgmap v19975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:07:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:55 smithi194 ceph-mon[33017]: pgmap v19976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:57 smithi033 ceph-mon[27845]: pgmap v19977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:57 smithi096 ceph-mon[32753]: pgmap v19977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:57 smithi194 ceph-mon[33017]: pgmap v19977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:07:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37017.268700 secs 2024-04-26T22:07:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37017.268700 secs 2024-04-26T22:07:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37017.268700 secs 2024-04-26T22:08:00.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:00.013+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:00.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:00.013+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:00.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:00.013+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:08:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:59 smithi033 ceph-mon[27845]: pgmap v19978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:07:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37020.241429 secs 2024-04-26T22:08:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:59 smithi096 ceph-mon[32753]: pgmap v19978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:07:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37020.241429 secs 2024-04-26T22:08:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:59 smithi194 ceph-mon[33017]: pgmap v19978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:07:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37020.241429 secs 2024-04-26T22:08:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:01 smithi096 ceph-mon[32753]: pgmap v19979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:01 smithi194 ceph-mon[33017]: pgmap v19979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:01 smithi033 ceph-mon[27845]: pgmap v19979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:04.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:03 smithi033 ceph-mon[27845]: pgmap v19980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:04.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37022.268783 secs 2024-04-26T22:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:03 smithi194 ceph-mon[33017]: pgmap v19980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37022.268783 secs 2024-04-26T22:08:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:03 smithi096 ceph-mon[32753]: pgmap v19980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37022.268783 secs 2024-04-26T22:08:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37025.241593 secs 2024-04-26T22:08:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37025.241593 secs 2024-04-26T22:08:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37025.241593 secs 2024-04-26T22:08:06.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:05 smithi194 ceph-mon[33017]: pgmap v19981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:08:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:05 smithi033 ceph-mon[27845]: pgmap v19981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:08:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:05 smithi096 ceph-mon[32753]: pgmap v19981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:08:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:06 smithi194 ceph-mon[33017]: pgmap v19982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:06 smithi033 ceph-mon[27845]: pgmap v19982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:06 smithi096 ceph-mon[32753]: pgmap v19982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37027.268891 secs 2024-04-26T22:08:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37027.268891 secs 2024-04-26T22:08:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37027.268891 secs 2024-04-26T22:08:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:09 smithi194 ceph-mon[33017]: pgmap v19983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:09 smithi033 ceph-mon[27845]: pgmap v19983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:09 smithi096 ceph-mon[32753]: pgmap v19983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37030.241715 secs 2024-04-26T22:08:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37030.241715 secs 2024-04-26T22:08:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37030.241715 secs 2024-04-26T22:08:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:11 smithi194 ceph-mon[33017]: pgmap v19984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:11 smithi033 ceph-mon[27845]: pgmap v19984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:11 smithi096 ceph-mon[32753]: pgmap v19984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:13 smithi194 ceph-mon[33017]: pgmap v19985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37032.268993 secs 2024-04-26T22:08:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:13 smithi033 ceph-mon[27845]: pgmap v19985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37032.268993 secs 2024-04-26T22:08:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:13 smithi096 ceph-mon[32753]: pgmap v19985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37032.268993 secs 2024-04-26T22:08:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:15.025+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:15.025+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:08:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:15.025+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37035.241851 secs 2024-04-26T22:08:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37035.241851 secs 2024-04-26T22:08:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37035.241851 secs 2024-04-26T22:08:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:15 smithi033 ceph-mon[27845]: pgmap v19986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:15 smithi096 ceph-mon[32753]: pgmap v19986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:15 smithi194 ceph-mon[33017]: pgmap v19986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:17 smithi033 ceph-mon[27845]: pgmap v19987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:18.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:17 smithi096 ceph-mon[32753]: pgmap v19987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:17 smithi194 ceph-mon[33017]: pgmap v19987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37037.269080 secs 2024-04-26T22:08:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37037.269080 secs 2024-04-26T22:08:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37037.269080 secs 2024-04-26T22:08:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:19 smithi033 ceph-mon[27845]: pgmap v19988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37040.241946 secs 2024-04-26T22:08:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:19 smithi096 ceph-mon[32753]: pgmap v19988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37040.241946 secs 2024-04-26T22:08:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:19 smithi194 ceph-mon[33017]: pgmap v19988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37040.241946 secs 2024-04-26T22:08:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:21 smithi033 ceph-mon[27845]: pgmap v19989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:22.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:21 smithi096 ceph-mon[32753]: pgmap v19989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:21 smithi194 ceph-mon[33017]: pgmap v19989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:23 smithi033 ceph-mon[27845]: pgmap v19990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37042.269166 secs 2024-04-26T22:08:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:23 smithi096 ceph-mon[32753]: pgmap v19990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37042.269166 secs 2024-04-26T22:08:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:23 smithi194 ceph-mon[33017]: pgmap v19990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37042.269166 secs 2024-04-26T22:08:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37045.242067 secs 2024-04-26T22:08:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37045.242067 secs 2024-04-26T22:08:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37045.242067 secs 2024-04-26T22:08:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:25 smithi033 ceph-mon[27845]: pgmap v19991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:25 smithi096 ceph-mon[32753]: pgmap v19991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:25 smithi194 ceph-mon[33017]: pgmap v19991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.104+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:08:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.104+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004910 con 0x564385821900 2024-04-26T22:08:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.104+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 289 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T22:08:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.104+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T22:08:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.105+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 289 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:08:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.105+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x7fef00077780 2024-04-26T22:08:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.105+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 290 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:08:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:28.105+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 290 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc004740 con 0x7fef00077780 2024-04-26T22:08:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:27 smithi033 ceph-mon[27845]: pgmap v19992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:27 smithi096 ceph-mon[32753]: pgmap v19992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:27 smithi194 ceph-mon[33017]: pgmap v19992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37047.269252 secs 2024-04-26T22:08:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37047.269252 secs 2024-04-26T22:08:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37047.269252 secs 2024-04-26T22:08:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:30.033+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:30.033+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:30.033+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:08:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:29 smithi033 ceph-mon[27845]: pgmap v19993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37050.242192 secs 2024-04-26T22:08:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:29 smithi096 ceph-mon[32753]: pgmap v19993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37050.242192 secs 2024-04-26T22:08:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:29 smithi194 ceph-mon[33017]: pgmap v19993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37050.242192 secs 2024-04-26T22:08:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:31 smithi033 ceph-mon[27845]: pgmap v19994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:32.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:31 smithi096 ceph-mon[32753]: pgmap v19994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:31 smithi194 ceph-mon[33017]: pgmap v19994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:33 smithi096 ceph-mon[32753]: pgmap v19995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37052.269339 secs 2024-04-26T22:08:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:33 smithi194 ceph-mon[33017]: pgmap v19995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37052.269339 secs 2024-04-26T22:08:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:33 smithi033 ceph-mon[27845]: pgmap v19995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37052.269339 secs 2024-04-26T22:08:35.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37055.242307 secs 2024-04-26T22:08:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37055.242307 secs 2024-04-26T22:08:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37055.242307 secs 2024-04-26T22:08:36.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:35 smithi194 ceph-mon[33017]: pgmap v19996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:35 smithi033 ceph-mon[27845]: pgmap v19996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:35 smithi096 ceph-mon[32753]: pgmap v19996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:37 smithi194 ceph-mon[33017]: pgmap v19997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:37 smithi033 ceph-mon[27845]: pgmap v19997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:37 smithi096 ceph-mon[32753]: pgmap v19997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:39.215 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37057.269443 secs 2024-04-26T22:08:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37057.269443 secs 2024-04-26T22:08:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37057.269443 secs 2024-04-26T22:08:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:39 smithi194 ceph-mon[33017]: pgmap v19998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37060.242423 secs 2024-04-26T22:08:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:39 smithi033 ceph-mon[27845]: pgmap v19998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37060.242423 secs 2024-04-26T22:08:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:39 smithi096 ceph-mon[32753]: pgmap v19998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37060.242423 secs 2024-04-26T22:08:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:40 smithi194 ceph-mon[33017]: pgmap v19999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:40 smithi033 ceph-mon[27845]: pgmap v19999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:40 smithi096 ceph-mon[32753]: pgmap v19999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:43 smithi033 ceph-mon[27845]: pgmap v20000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:44.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37062.269530 secs 2024-04-26T22:08:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:43 smithi096 ceph-mon[32753]: pgmap v20000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37062.269530 secs 2024-04-26T22:08:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:43 smithi194 ceph-mon[33017]: pgmap v20000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37062.269530 secs 2024-04-26T22:08:45.049 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:45.048+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:45.048+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:08:45.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:08:45.048+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:08:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37065.242541 secs 2024-04-26T22:08:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37065.242541 secs 2024-04-26T22:08:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37065.242541 secs 2024-04-26T22:08:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:45 smithi033 ceph-mon[27845]: pgmap v20001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:45 smithi096 ceph-mon[32753]: pgmap v20001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:45 smithi194 ceph-mon[33017]: pgmap v20001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:47 smithi033 ceph-mon[27845]: pgmap v20002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:47 smithi096 ceph-mon[32753]: pgmap v20002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:47 smithi194 ceph-mon[33017]: pgmap v20002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37067.269612 secs 2024-04-26T22:08:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37067.269612 secs 2024-04-26T22:08:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37067.269612 secs 2024-04-26T22:08:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:49 smithi033 ceph-mon[27845]: pgmap v20003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37070.242608 secs 2024-04-26T22:08:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:49 smithi096 ceph-mon[32753]: pgmap v20003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37070.242608 secs 2024-04-26T22:08:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:49 smithi194 ceph-mon[33017]: pgmap v20003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37070.242608 secs 2024-04-26T22:08:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:51 smithi033 ceph-mon[27845]: pgmap v20004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:51 smithi096 ceph-mon[32753]: pgmap v20004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:51 smithi194 ceph-mon[33017]: pgmap v20004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:53 smithi033 ceph-mon[27845]: pgmap v20005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37072.269695 secs 2024-04-26T22:08:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:53 smithi096 ceph-mon[32753]: pgmap v20005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37072.269695 secs 2024-04-26T22:08:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:53 smithi194 ceph-mon[33017]: pgmap v20005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37072.269695 secs 2024-04-26T22:08:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37075.242746 secs 2024-04-26T22:08:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37075.242746 secs 2024-04-26T22:08:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37075.242746 secs 2024-04-26T22:08:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:55 smithi033 ceph-mon[27845]: pgmap v20006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:08:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:55 smithi096 ceph-mon[32753]: pgmap v20006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:08:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:55 smithi194 ceph-mon[33017]: pgmap v20006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:08:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:08:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:08:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:56 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:08:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:08:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:08:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:56 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:08:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:08:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:08:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:56 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:08:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:08:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:57 smithi033 ceph-mon[27845]: pgmap v20007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:57 smithi033 ceph-mon[27845]: pgmap v20008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:58.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:57 smithi096 ceph-mon[32753]: pgmap v20007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:57 smithi096 ceph-mon[32753]: pgmap v20008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:57 smithi194 ceph-mon[33017]: pgmap v20007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:08:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:57 smithi194 ceph-mon[33017]: pgmap v20008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:08:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37077.269781 secs 2024-04-26T22:08:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37077.269781 secs 2024-04-26T22:08:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37077.269781 secs 2024-04-26T22:09:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:00.054+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:00.054+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:09:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:00.054+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:59 smithi194 ceph-mon[33017]: pgmap v20009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:08:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37080.242859 secs 2024-04-26T22:09:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:59 smithi033 ceph-mon[27845]: pgmap v20009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:08:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37080.242859 secs 2024-04-26T22:09:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:59 smithi096 ceph-mon[32753]: pgmap v20009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:08:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37080.242859 secs 2024-04-26T22:09:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:01 smithi194 ceph-mon[33017]: pgmap v20010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:01 smithi033 ceph-mon[27845]: pgmap v20010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:01 smithi096 ceph-mon[32753]: pgmap v20010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:03 smithi194 ceph-mon[33017]: pgmap v20011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37082.269868 secs 2024-04-26T22:09:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:03 smithi033 ceph-mon[27845]: pgmap v20011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37082.269868 secs 2024-04-26T22:09:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:03 smithi096 ceph-mon[32753]: pgmap v20011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37082.269868 secs 2024-04-26T22:09:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37085.242995 secs 2024-04-26T22:09:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37085.242995 secs 2024-04-26T22:09:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37085.242995 secs 2024-04-26T22:09:06.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:05 smithi194 ceph-mon[33017]: pgmap v20012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:09:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:05 smithi033 ceph-mon[27845]: pgmap v20012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:09:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:05 smithi096 ceph-mon[32753]: pgmap v20012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:09:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:06 smithi194 ceph-mon[33017]: pgmap v20013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:09:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:06 smithi033 ceph-mon[27845]: pgmap v20013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:09:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:06 smithi096 ceph-mon[32753]: pgmap v20013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:09:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37087.269953 secs 2024-04-26T22:09:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37087.269953 secs 2024-04-26T22:09:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37087.269953 secs 2024-04-26T22:09:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:09 smithi194 ceph-mon[33017]: pgmap v20014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:09 smithi033 ceph-mon[27845]: pgmap v20014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:09 smithi096 ceph-mon[32753]: pgmap v20014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37090.243128 secs 2024-04-26T22:09:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37090.243128 secs 2024-04-26T22:09:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37090.243128 secs 2024-04-26T22:09:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:11 smithi194 ceph-mon[33017]: pgmap v20015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:11 smithi033 ceph-mon[27845]: pgmap v20015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:11 smithi096 ceph-mon[32753]: pgmap v20015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:13 smithi194 ceph-mon[33017]: pgmap v20016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37092.270004 secs 2024-04-26T22:09:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:13 smithi033 ceph-mon[27845]: pgmap v20016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37092.270004 secs 2024-04-26T22:09:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:13 smithi096 ceph-mon[32753]: pgmap v20016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37092.270004 secs 2024-04-26T22:09:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37095.243260 secs 2024-04-26T22:09:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37095.243260 secs 2024-04-26T22:09:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37095.243260 secs 2024-04-26T22:09:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:15.059+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:15.059+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:15.059+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:09:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:15 smithi194 ceph-mon[33017]: pgmap v20017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:15 smithi033 ceph-mon[27845]: pgmap v20017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:15 smithi096 ceph-mon[32753]: pgmap v20017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:17 smithi194 ceph-mon[33017]: pgmap v20018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:17 smithi033 ceph-mon[27845]: pgmap v20018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:17 smithi096 ceph-mon[32753]: pgmap v20018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37097.270091 secs 2024-04-26T22:09:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37097.270091 secs 2024-04-26T22:09:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37097.270091 secs 2024-04-26T22:09:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:19 smithi033 ceph-mon[27845]: pgmap v20019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37100.243385 secs 2024-04-26T22:09:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:19 smithi096 ceph-mon[32753]: pgmap v20019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37100.243385 secs 2024-04-26T22:09:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:19 smithi194 ceph-mon[33017]: pgmap v20019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37100.243385 secs 2024-04-26T22:09:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:21 smithi033 ceph-mon[27845]: pgmap v20020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:21 smithi096 ceph-mon[32753]: pgmap v20020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:21 smithi194 ceph-mon[33017]: pgmap v20020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:23 smithi033 ceph-mon[27845]: pgmap v20021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37102.270152 secs 2024-04-26T22:09:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:23 smithi096 ceph-mon[32753]: pgmap v20021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37102.270152 secs 2024-04-26T22:09:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:23 smithi194 ceph-mon[33017]: pgmap v20021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37102.270152 secs 2024-04-26T22:09:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37105.243461 secs 2024-04-26T22:09:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37105.243461 secs 2024-04-26T22:09:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37105.243461 secs 2024-04-26T22:09:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:25 smithi033 ceph-mon[27845]: pgmap v20022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:25 smithi096 ceph-mon[32753]: pgmap v20022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:25 smithi194 ceph-mon[33017]: pgmap v20022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:27 smithi194 ceph-mon[33017]: pgmap v20023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:27 smithi033 ceph-mon[27845]: pgmap v20023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:27 smithi096 ceph-mon[32753]: pgmap v20023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37107.270237 secs 2024-04-26T22:09:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37107.270237 secs 2024-04-26T22:09:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37107.270237 secs 2024-04-26T22:09:29.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:29 smithi096 ceph-mon[32753]: pgmap v20024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:29 smithi194 ceph-mon[33017]: pgmap v20024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:29 smithi033 ceph-mon[27845]: pgmap v20024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:30.073+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:30.073+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:30.073+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:09:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37110.243598 secs 2024-04-26T22:09:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37110.243598 secs 2024-04-26T22:09:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37110.243598 secs 2024-04-26T22:09:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:31 smithi194 ceph-mon[33017]: pgmap v20025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:31 smithi033 ceph-mon[27845]: pgmap v20025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:31 smithi096 ceph-mon[32753]: pgmap v20025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:33 smithi194 ceph-mon[33017]: pgmap v20026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37112.270319 secs 2024-04-26T22:09:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:33 smithi033 ceph-mon[27845]: pgmap v20026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37112.270319 secs 2024-04-26T22:09:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:33 smithi096 ceph-mon[32753]: pgmap v20026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37112.270319 secs 2024-04-26T22:09:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37115.243720 secs 2024-04-26T22:09:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37115.243720 secs 2024-04-26T22:09:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37115.243720 secs 2024-04-26T22:09:35.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:35 smithi033 ceph-mon[27845]: pgmap v20027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:35 smithi194 ceph-mon[33017]: pgmap v20027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:35 smithi096 ceph-mon[32753]: pgmap v20027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:37 smithi194 ceph-mon[33017]: pgmap v20028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:37 smithi033 ceph-mon[27845]: pgmap v20028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:37 smithi096 ceph-mon[32753]: pgmap v20028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37117.270378 secs 2024-04-26T22:09:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37117.270378 secs 2024-04-26T22:09:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37117.270378 secs 2024-04-26T22:09:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:39 smithi194 ceph-mon[33017]: pgmap v20029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37120.243842 secs 2024-04-26T22:09:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:39 smithi033 ceph-mon[27845]: pgmap v20029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37120.243842 secs 2024-04-26T22:09:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:39 smithi096 ceph-mon[32753]: pgmap v20029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37120.243842 secs 2024-04-26T22:09:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:41 smithi194 ceph-mon[33017]: pgmap v20030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:41 smithi033 ceph-mon[27845]: pgmap v20030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:41 smithi096 ceph-mon[32753]: pgmap v20030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:43 smithi194 ceph-mon[33017]: pgmap v20031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37122.270461 secs 2024-04-26T22:09:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:43 smithi033 ceph-mon[27845]: pgmap v20031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37122.270461 secs 2024-04-26T22:09:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:43 smithi096 ceph-mon[32753]: pgmap v20031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37122.270461 secs 2024-04-26T22:09:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37125.243959 secs 2024-04-26T22:09:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37125.243959 secs 2024-04-26T22:09:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37125.243959 secs 2024-04-26T22:09:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:45.089+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:45.089+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:09:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:09:45.089+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:09:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:45 smithi194 ceph-mon[33017]: pgmap v20032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:45 smithi033 ceph-mon[27845]: pgmap v20032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:45 smithi096 ceph-mon[32753]: pgmap v20032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:47 smithi033 ceph-mon[27845]: pgmap v20033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:47 smithi096 ceph-mon[32753]: pgmap v20033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:48.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:47 smithi194 ceph-mon[33017]: pgmap v20033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37127.270549 secs 2024-04-26T22:09:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37127.270549 secs 2024-04-26T22:09:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37127.270549 secs 2024-04-26T22:09:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:49 smithi033 ceph-mon[27845]: pgmap v20034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37130.244076 secs 2024-04-26T22:09:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:49 smithi096 ceph-mon[32753]: pgmap v20034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37130.244076 secs 2024-04-26T22:09:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:49 smithi194 ceph-mon[33017]: pgmap v20034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37130.244076 secs 2024-04-26T22:09:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:51 smithi033 ceph-mon[27845]: pgmap v20035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:51 smithi096 ceph-mon[32753]: pgmap v20035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:52.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:51 smithi194 ceph-mon[33017]: pgmap v20035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:53 smithi033 ceph-mon[27845]: pgmap v20036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37132.270633 secs 2024-04-26T22:09:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:53 smithi096 ceph-mon[32753]: pgmap v20036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37132.270633 secs 2024-04-26T22:09:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:53 smithi194 ceph-mon[33017]: pgmap v20036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37132.270633 secs 2024-04-26T22:09:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37135.244200 secs 2024-04-26T22:09:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37135.244200 secs 2024-04-26T22:09:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37135.244200 secs 2024-04-26T22:09:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:55 smithi033 ceph-mon[27845]: pgmap v20037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:55 smithi096 ceph-mon[32753]: pgmap v20037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:55 smithi194 ceph-mon[33017]: pgmap v20037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:09:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:57 smithi033 ceph-mon[27845]: pgmap v20038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:57 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:09:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:57 smithi096 ceph-mon[32753]: pgmap v20038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:57 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:09:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:57 smithi194 ceph-mon[33017]: pgmap v20038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:57 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:09:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:09:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:09:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37137.270724 secs 2024-04-26T22:09:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:09:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:09:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:09:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37137.270724 secs 2024-04-26T22:09:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:09:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:09:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:09:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37137.270724 secs 2024-04-26T22:09:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:59 smithi033 ceph-mon[27845]: pgmap v20039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:59 smithi033 ceph-mon[27845]: pgmap v20040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:09:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37140.244262 secs 2024-04-26T22:09:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:59 smithi096 ceph-mon[32753]: pgmap v20039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:09:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:59 smithi096 ceph-mon[32753]: pgmap v20040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:09:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:09:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37140.244262 secs 2024-04-26T22:10:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:00.101+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:00.101+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:00.101+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:10:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:59 smithi194 ceph-mon[33017]: pgmap v20039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:59 smithi194 ceph-mon[33017]: pgmap v20040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:09:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37140.244262 secs 2024-04-26T22:10:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:10:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:10:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:10:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:01 smithi096 ceph-mon[32753]: pgmap v20041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:01 smithi194 ceph-mon[33017]: pgmap v20041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:01 smithi033 ceph-mon[27845]: pgmap v20041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:03 smithi096 ceph-mon[32753]: pgmap v20042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37142.270814 secs 2024-04-26T22:10:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:03 smithi194 ceph-mon[33017]: pgmap v20042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37142.270814 secs 2024-04-26T22:10:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:03 smithi033 ceph-mon[27845]: pgmap v20042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37142.270814 secs 2024-04-26T22:10:05.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37145.244385 secs 2024-04-26T22:10:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37145.244385 secs 2024-04-26T22:10:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37145.244385 secs 2024-04-26T22:10:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:05 smithi194 ceph-mon[33017]: pgmap v20043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:10:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:05 smithi033 ceph-mon[27845]: pgmap v20043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:10:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:05 smithi096 ceph-mon[32753]: pgmap v20043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:10:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:07 smithi194 ceph-mon[33017]: pgmap v20044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:07 smithi033 ceph-mon[27845]: pgmap v20044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:07 smithi096 ceph-mon[32753]: pgmap v20044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:10:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37147.270901 secs 2024-04-26T22:10:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37147.270901 secs 2024-04-26T22:10:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37147.270901 secs 2024-04-26T22:10:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:09 smithi194 ceph-mon[33017]: pgmap v20045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:10:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37150.244514 secs 2024-04-26T22:10:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:09 smithi033 ceph-mon[27845]: pgmap v20045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:10:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37150.244514 secs 2024-04-26T22:10:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:09 smithi096 ceph-mon[32753]: pgmap v20045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:10:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37150.244514 secs 2024-04-26T22:10:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:11 smithi194 ceph-mon[33017]: pgmap v20046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:11 smithi033 ceph-mon[27845]: pgmap v20046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:11 smithi096 ceph-mon[32753]: pgmap v20046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:13 smithi194 ceph-mon[33017]: pgmap v20047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37152.271007 secs 2024-04-26T22:10:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:13 smithi033 ceph-mon[27845]: pgmap v20047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37152.271007 secs 2024-04-26T22:10:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:13 smithi096 ceph-mon[32753]: pgmap v20047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37152.271007 secs 2024-04-26T22:10:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37155.244599 secs 2024-04-26T22:10:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:15.116+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:15.116+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:15.116+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:10:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37155.244599 secs 2024-04-26T22:10:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37155.244599 secs 2024-04-26T22:10:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:15 smithi194 ceph-mon[33017]: pgmap v20048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:15 smithi033 ceph-mon[27845]: pgmap v20048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:15 smithi096 ceph-mon[32753]: pgmap v20048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:17 smithi194 ceph-mon[33017]: pgmap v20049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:17 smithi033 ceph-mon[27845]: pgmap v20049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:17 smithi096 ceph-mon[32753]: pgmap v20049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37157.271099 secs 2024-04-26T22:10:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37157.271099 secs 2024-04-26T22:10:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37157.271099 secs 2024-04-26T22:10:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:19 smithi194 ceph-mon[33017]: pgmap v20050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:20.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37160.244725 secs 2024-04-26T22:10:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:19 smithi033 ceph-mon[27845]: pgmap v20050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37160.244725 secs 2024-04-26T22:10:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:19 smithi096 ceph-mon[32753]: pgmap v20050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37160.244725 secs 2024-04-26T22:10:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:21 smithi194 ceph-mon[33017]: pgmap v20051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:21 smithi033 ceph-mon[27845]: pgmap v20051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:21 smithi096 ceph-mon[32753]: pgmap v20051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:23 smithi194 ceph-mon[33017]: pgmap v20052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37162.271162 secs 2024-04-26T22:10:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:23 smithi033 ceph-mon[27845]: pgmap v20052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37162.271162 secs 2024-04-26T22:10:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:23 smithi096 ceph-mon[32753]: pgmap v20052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37162.271162 secs 2024-04-26T22:10:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37165.244849 secs 2024-04-26T22:10:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37165.244849 secs 2024-04-26T22:10:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37165.244849 secs 2024-04-26T22:10:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:25 smithi194 ceph-mon[33017]: pgmap v20053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:25 smithi033 ceph-mon[27845]: pgmap v20053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:25 smithi096 ceph-mon[32753]: pgmap v20053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:28.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:27 smithi194 ceph-mon[33017]: pgmap v20054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:27 smithi033 ceph-mon[27845]: pgmap v20054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:28.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:27 smithi096 ceph-mon[32753]: pgmap v20054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:29.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37167.271250 secs 2024-04-26T22:10:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37167.271250 secs 2024-04-26T22:10:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37167.271250 secs 2024-04-26T22:10:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:29 smithi194 ceph-mon[33017]: pgmap v20055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37170.244992 secs 2024-04-26T22:10:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:30.116+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:30.116+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:30.116+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:10:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:29 smithi033 ceph-mon[27845]: pgmap v20055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37170.244992 secs 2024-04-26T22:10:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:29 smithi096 ceph-mon[32753]: pgmap v20055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37170.244992 secs 2024-04-26T22:10:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:31 smithi194 ceph-mon[33017]: pgmap v20056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:31 smithi033 ceph-mon[27845]: pgmap v20056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:31 smithi096 ceph-mon[32753]: pgmap v20056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:33 smithi194 ceph-mon[33017]: pgmap v20057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37172.271341 secs 2024-04-26T22:10:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:33 smithi033 ceph-mon[27845]: pgmap v20057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37172.271341 secs 2024-04-26T22:10:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:33 smithi096 ceph-mon[32753]: pgmap v20057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37172.271341 secs 2024-04-26T22:10:35.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37175.245052 secs 2024-04-26T22:10:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37175.245052 secs 2024-04-26T22:10:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37175.245052 secs 2024-04-26T22:10:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:35 smithi194 ceph-mon[33017]: pgmap v20058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:35 smithi033 ceph-mon[27845]: pgmap v20058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:35 smithi096 ceph-mon[32753]: pgmap v20058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:37 smithi194 ceph-mon[33017]: pgmap v20059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:37 smithi033 ceph-mon[27845]: pgmap v20059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:37 smithi096 ceph-mon[32753]: pgmap v20059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37177.271402 secs 2024-04-26T22:10:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37177.271402 secs 2024-04-26T22:10:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37177.271402 secs 2024-04-26T22:10:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:39 smithi194 ceph-mon[33017]: pgmap v20060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37180.245138 secs 2024-04-26T22:10:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:39 smithi033 ceph-mon[27845]: pgmap v20060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37180.245138 secs 2024-04-26T22:10:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:39 smithi096 ceph-mon[32753]: pgmap v20060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37180.245138 secs 2024-04-26T22:10:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:41 smithi194 ceph-mon[33017]: pgmap v20061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:41 smithi033 ceph-mon[27845]: pgmap v20061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:41 smithi096 ceph-mon[32753]: pgmap v20061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:44.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:43 smithi033 ceph-mon[27845]: pgmap v20062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:44.143 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37182.271483 secs 2024-04-26T22:10:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:43 smithi096 ceph-mon[32753]: pgmap v20062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37182.271483 secs 2024-04-26T22:10:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:43 smithi194 ceph-mon[33017]: pgmap v20062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37182.271483 secs 2024-04-26T22:10:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:45.131+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:45.131+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:10:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:10:45.131+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:10:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37185.245279 secs 2024-04-26T22:10:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37185.245279 secs 2024-04-26T22:10:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37185.245279 secs 2024-04-26T22:10:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:45 smithi033 ceph-mon[27845]: pgmap v20063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:45 smithi096 ceph-mon[32753]: pgmap v20063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:45 smithi194 ceph-mon[33017]: pgmap v20063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:47 smithi033 ceph-mon[27845]: pgmap v20064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:47 smithi096 ceph-mon[32753]: pgmap v20064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:47 smithi194 ceph-mon[33017]: pgmap v20064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37187.271567 secs 2024-04-26T22:10:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37187.271567 secs 2024-04-26T22:10:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37187.271567 secs 2024-04-26T22:10:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:49 smithi033 ceph-mon[27845]: pgmap v20065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37190.245402 secs 2024-04-26T22:10:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:49 smithi096 ceph-mon[32753]: pgmap v20065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37190.245402 secs 2024-04-26T22:10:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:49 smithi194 ceph-mon[33017]: pgmap v20065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37190.245402 secs 2024-04-26T22:10:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:51 smithi033 ceph-mon[27845]: pgmap v20066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:51 smithi096 ceph-mon[32753]: pgmap v20066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:51 smithi194 ceph-mon[33017]: pgmap v20066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:53 smithi033 ceph-mon[27845]: pgmap v20067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37192.271653 secs 2024-04-26T22:10:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:53 smithi096 ceph-mon[32753]: pgmap v20067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37192.271653 secs 2024-04-26T22:10:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:53 smithi194 ceph-mon[33017]: pgmap v20067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37192.271653 secs 2024-04-26T22:10:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37195.245520 secs 2024-04-26T22:10:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37195.245520 secs 2024-04-26T22:10:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37195.245520 secs 2024-04-26T22:10:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:55 smithi033 ceph-mon[27845]: pgmap v20068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:55 smithi096 ceph-mon[32753]: pgmap v20068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:55 smithi194 ceph-mon[33017]: pgmap v20068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:10:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:57 smithi096 ceph-mon[32753]: pgmap v20069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:57 smithi194 ceph-mon[33017]: pgmap v20069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:57 smithi033 ceph-mon[27845]: pgmap v20069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:10:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37197.271736 secs 2024-04-26T22:10:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:10:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37197.271736 secs 2024-04-26T22:10:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:10:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37197.271736 secs 2024-04-26T22:10:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:11:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:00.147+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:00.147+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:00.147+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:11:00.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:59 smithi033 ceph-mon[27845]: pgmap v20070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:00.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37200.245589 secs 2024-04-26T22:11:00.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:11:00.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:11:00.330 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:10:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:11:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:59 smithi194 ceph-mon[33017]: pgmap v20070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37200.245589 secs 2024-04-26T22:11:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:11:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:11:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:10:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:11:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:59 smithi096 ceph-mon[32753]: pgmap v20070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37200.245589 secs 2024-04-26T22:11:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:11:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:11:00.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:10:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:11:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:00 smithi194 ceph-mon[33017]: pgmap v20071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:11:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:00 smithi194 ceph-mon[33017]: pgmap v20072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:00 smithi033 ceph-mon[27845]: pgmap v20071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:11:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:00 smithi033 ceph-mon[27845]: pgmap v20072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:00 smithi096 ceph-mon[32753]: pgmap v20071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:11:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:00 smithi096 ceph-mon[32753]: pgmap v20072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:02 smithi194 ceph-mon[33017]: pgmap v20073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:02 smithi033 ceph-mon[27845]: pgmap v20073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:03.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:02 smithi096 ceph-mon[32753]: pgmap v20073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37202.271826 secs 2024-04-26T22:11:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37202.271826 secs 2024-04-26T22:11:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37202.271826 secs 2024-04-26T22:11:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:04 smithi033 ceph-mon[27845]: pgmap v20074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37205.245709 secs 2024-04-26T22:11:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:04 smithi194 ceph-mon[33017]: pgmap v20074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37205.245709 secs 2024-04-26T22:11:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:04 smithi096 ceph-mon[32753]: pgmap v20074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37205.245709 secs 2024-04-26T22:11:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:06 smithi194 ceph-mon[33017]: pgmap v20075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:06 smithi033 ceph-mon[27845]: pgmap v20075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:06 smithi096 ceph-mon[32753]: pgmap v20075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:08 smithi194 ceph-mon[33017]: pgmap v20076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37207.271912 secs 2024-04-26T22:11:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:08 smithi033 ceph-mon[27845]: pgmap v20076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37207.271912 secs 2024-04-26T22:11:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:08 smithi096 ceph-mon[32753]: pgmap v20076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:11:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37207.271912 secs 2024-04-26T22:11:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37210.245848 secs 2024-04-26T22:11:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37210.245848 secs 2024-04-26T22:11:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37210.245848 secs 2024-04-26T22:11:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:10 smithi194 ceph-mon[33017]: pgmap v20077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:11:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:10 smithi033 ceph-mon[27845]: pgmap v20077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:11:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:10 smithi096 ceph-mon[32753]: pgmap v20077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:11:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:12 smithi194 ceph-mon[33017]: pgmap v20078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:12 smithi033 ceph-mon[27845]: pgmap v20078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:12 smithi096 ceph-mon[32753]: pgmap v20078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37212.272015 secs 2024-04-26T22:11:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37212.272015 secs 2024-04-26T22:11:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37212.272015 secs 2024-04-26T22:11:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:14 smithi194 ceph-mon[33017]: pgmap v20079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:14.863 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:14 smithi033 ceph-mon[27845]: pgmap v20079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:14 smithi096 ceph-mon[32753]: pgmap v20079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:15.158+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:15.158+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:15.158+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:11:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37215.245969 secs 2024-04-26T22:11:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37215.245969 secs 2024-04-26T22:11:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37215.245969 secs 2024-04-26T22:11:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:16 smithi194 ceph-mon[33017]: pgmap v20080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:16 smithi033 ceph-mon[27845]: pgmap v20080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:16 smithi096 ceph-mon[32753]: pgmap v20080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:18 smithi194 ceph-mon[33017]: pgmap v20081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37217.272105 secs 2024-04-26T22:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:18 smithi033 ceph-mon[27845]: pgmap v20081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37217.272105 secs 2024-04-26T22:11:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:18 smithi096 ceph-mon[32753]: pgmap v20081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37217.272105 secs 2024-04-26T22:11:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37220.246088 secs 2024-04-26T22:11:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37220.246088 secs 2024-04-26T22:11:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37220.246088 secs 2024-04-26T22:11:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:20 smithi194 ceph-mon[33017]: pgmap v20082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:20 smithi033 ceph-mon[27845]: pgmap v20082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:20 smithi096 ceph-mon[32753]: pgmap v20082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T22:11:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T22:11:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0022040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T22:11:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0022040 comp rx=0 tx=0).stop 2024-04-26T22:11:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T22:11:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T22:11:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:11:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.822+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:11:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.823+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:11:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:21.824+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef006a070 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:11:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:22 smithi194 ceph-mon[33017]: pgmap v20083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:22 smithi033 ceph-mon[27845]: pgmap v20083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:22 smithi096 ceph-mon[32753]: pgmap v20083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37222.272194 secs 2024-04-26T22:11:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37222.272194 secs 2024-04-26T22:11:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37222.272194 secs 2024-04-26T22:11:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:24 smithi194 ceph-mon[33017]: pgmap v20084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37225.246235 secs 2024-04-26T22:11:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:24 smithi033 ceph-mon[27845]: pgmap v20084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37225.246235 secs 2024-04-26T22:11:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:24 smithi096 ceph-mon[32753]: pgmap v20084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37225.246235 secs 2024-04-26T22:11:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:26 smithi194 ceph-mon[33017]: pgmap v20085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:26 smithi033 ceph-mon[27845]: pgmap v20085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:26 smithi096 ceph-mon[32753]: pgmap v20085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:28.136+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:11:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:28.136+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:11:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:28.137+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 291 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T22:11:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:28.137+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 291 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T22:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:28 smithi194 ceph-mon[33017]: pgmap v20086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37227.272307 secs 2024-04-26T22:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:28 smithi033 ceph-mon[27845]: pgmap v20086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37227.272307 secs 2024-04-26T22:11:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:28 smithi096 ceph-mon[32753]: pgmap v20086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37227.272307 secs 2024-04-26T22:11:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37230.246297 secs 2024-04-26T22:11:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37230.246297 secs 2024-04-26T22:11:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:30.167+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:30.167+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:11:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:30.167+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37230.246297 secs 2024-04-26T22:11:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:30 smithi194 ceph-mon[33017]: pgmap v20087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:30 smithi033 ceph-mon[27845]: pgmap v20087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:30 smithi096 ceph-mon[32753]: pgmap v20087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:32 smithi194 ceph-mon[33017]: pgmap v20088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:32 smithi033 ceph-mon[27845]: pgmap v20088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:32 smithi096 ceph-mon[32753]: pgmap v20088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37232.272368 secs 2024-04-26T22:11:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37232.272368 secs 2024-04-26T22:11:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37232.272368 secs 2024-04-26T22:11:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:34 smithi194 ceph-mon[33017]: pgmap v20089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37235.246422 secs 2024-04-26T22:11:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:34 smithi033 ceph-mon[27845]: pgmap v20089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37235.246422 secs 2024-04-26T22:11:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:34 smithi096 ceph-mon[32753]: pgmap v20089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37235.246422 secs 2024-04-26T22:11:37.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:36 smithi194 ceph-mon[33017]: pgmap v20090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:36 smithi033 ceph-mon[27845]: pgmap v20090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:36 smithi096 ceph-mon[32753]: pgmap v20090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:38 smithi194 ceph-mon[33017]: pgmap v20091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37237.272451 secs 2024-04-26T22:11:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:38 smithi033 ceph-mon[27845]: pgmap v20091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37237.272451 secs 2024-04-26T22:11:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:38 smithi096 ceph-mon[32753]: pgmap v20091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37237.272451 secs 2024-04-26T22:11:40.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37240.246500 secs 2024-04-26T22:11:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37240.246500 secs 2024-04-26T22:11:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37240.246500 secs 2024-04-26T22:11:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:40 smithi033 ceph-mon[27845]: pgmap v20092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:40 smithi096 ceph-mon[32753]: pgmap v20092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:41.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:40 smithi194 ceph-mon[33017]: pgmap v20092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:42 smithi033 ceph-mon[27845]: pgmap v20093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:42 smithi096 ceph-mon[32753]: pgmap v20093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:42 smithi194 ceph-mon[33017]: pgmap v20093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37242.272542 secs 2024-04-26T22:11:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37242.272542 secs 2024-04-26T22:11:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37242.272542 secs 2024-04-26T22:11:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:44 smithi033 ceph-mon[27845]: pgmap v20094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37245.246621 secs 2024-04-26T22:11:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:45.177+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:45.177+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:11:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:11:45.177+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:11:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:44 smithi096 ceph-mon[32753]: pgmap v20094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37245.246621 secs 2024-04-26T22:11:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:44 smithi194 ceph-mon[33017]: pgmap v20094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37245.246621 secs 2024-04-26T22:11:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:46 smithi033 ceph-mon[27845]: pgmap v20095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:46 smithi096 ceph-mon[32753]: pgmap v20095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:46 smithi194 ceph-mon[33017]: pgmap v20095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:48 smithi033 ceph-mon[27845]: pgmap v20096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37247.272645 secs 2024-04-26T22:11:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:48 smithi096 ceph-mon[32753]: pgmap v20096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37247.272645 secs 2024-04-26T22:11:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:48 smithi194 ceph-mon[33017]: pgmap v20096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37247.272645 secs 2024-04-26T22:11:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37250.246737 secs 2024-04-26T22:11:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37250.246737 secs 2024-04-26T22:11:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37250.246737 secs 2024-04-26T22:11:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:50 smithi033 ceph-mon[27845]: pgmap v20097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:50 smithi096 ceph-mon[32753]: pgmap v20097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:50 smithi194 ceph-mon[33017]: pgmap v20097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:52 smithi033 ceph-mon[27845]: pgmap v20098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:52 smithi096 ceph-mon[32753]: pgmap v20098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:52 smithi194 ceph-mon[33017]: pgmap v20098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37252.272747 secs 2024-04-26T22:11:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37252.272747 secs 2024-04-26T22:11:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37252.272747 secs 2024-04-26T22:11:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:54 smithi033 ceph-mon[27845]: pgmap v20099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37255.246860 secs 2024-04-26T22:11:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:54 smithi096 ceph-mon[32753]: pgmap v20099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37255.246860 secs 2024-04-26T22:11:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:54 smithi194 ceph-mon[33017]: pgmap v20099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37255.246860 secs 2024-04-26T22:11:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:56 smithi096 ceph-mon[32753]: pgmap v20100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:56 smithi194 ceph-mon[33017]: pgmap v20100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:56 smithi033 ceph-mon[27845]: pgmap v20100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:11:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:58 smithi096 ceph-mon[32753]: pgmap v20101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37257.272851 secs 2024-04-26T22:11:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:58 smithi194 ceph-mon[33017]: pgmap v20101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37257.272851 secs 2024-04-26T22:11:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:58 smithi033 ceph-mon[27845]: pgmap v20101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:11:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37257.272851 secs 2024-04-26T22:12:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:00.193+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:00.193+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:00.193+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:12:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:11:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37260.246999 secs 2024-04-26T22:12:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:11:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37260.246999 secs 2024-04-26T22:12:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:11:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37260.246999 secs 2024-04-26T22:12:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:00 smithi194 ceph-mon[33017]: pgmap v20102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:12:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:00 smithi033 ceph-mon[27845]: pgmap v20102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:12:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:00 smithi096 ceph-mon[32753]: pgmap v20102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:12:02.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:12:02.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:01 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:12:02.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:01 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:12:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:12:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:01 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:12:02.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:01 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:12:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:12:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:01 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:12:02.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:01 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:12:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:02 smithi194 ceph-mon[33017]: pgmap v20103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:02 smithi194 ceph-mon[33017]: pgmap v20104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:02 smithi033 ceph-mon[27845]: pgmap v20103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:02 smithi033 ceph-mon[27845]: pgmap v20104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:02 smithi096 ceph-mon[32753]: pgmap v20103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:02 smithi096 ceph-mon[32753]: pgmap v20104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37262.272952 secs 2024-04-26T22:12:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37262.272952 secs 2024-04-26T22:12:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37262.272952 secs 2024-04-26T22:12:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:04 smithi033 ceph-mon[27845]: pgmap v20105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:05.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37265.247145 secs 2024-04-26T22:12:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:04 smithi194 ceph-mon[33017]: pgmap v20105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37265.247145 secs 2024-04-26T22:12:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:04 smithi096 ceph-mon[32753]: pgmap v20105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37265.247145 secs 2024-04-26T22:12:07.358 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:06 smithi194 ceph-mon[33017]: pgmap v20106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:06 smithi033 ceph-mon[27845]: pgmap v20106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:06 smithi096 ceph-mon[32753]: pgmap v20106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:08 smithi194 ceph-mon[33017]: pgmap v20107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37267.273061 secs 2024-04-26T22:12:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:08 smithi033 ceph-mon[27845]: pgmap v20107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37267.273061 secs 2024-04-26T22:12:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:08 smithi096 ceph-mon[32753]: pgmap v20107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37267.273061 secs 2024-04-26T22:12:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37270.247283 secs 2024-04-26T22:12:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37270.247283 secs 2024-04-26T22:12:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37270.247283 secs 2024-04-26T22:12:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:10 smithi194 ceph-mon[33017]: pgmap v20108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:10 smithi033 ceph-mon[27845]: pgmap v20108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:10 smithi096 ceph-mon[32753]: pgmap v20108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:12 smithi194 ceph-mon[33017]: pgmap v20109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:12:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:12 smithi033 ceph-mon[27845]: pgmap v20109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:12:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:12 smithi096 ceph-mon[32753]: pgmap v20109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:12:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37272.273124 secs 2024-04-26T22:12:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:14 smithi096 ceph-mon[32753]: pgmap v20110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37272.273124 secs 2024-04-26T22:12:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:14 smithi194 ceph-mon[33017]: pgmap v20110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37272.273124 secs 2024-04-26T22:12:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:14 smithi033 ceph-mon[27845]: pgmap v20110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:15.207+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:15.207+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:15.207+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:12:15.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37275.247405 secs 2024-04-26T22:12:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37275.247405 secs 2024-04-26T22:12:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37275.247405 secs 2024-04-26T22:12:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:16 smithi194 ceph-mon[33017]: pgmap v20111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:16 smithi033 ceph-mon[27845]: pgmap v20111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:16 smithi096 ceph-mon[32753]: pgmap v20111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:18 smithi194 ceph-mon[33017]: pgmap v20112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:18 smithi033 ceph-mon[27845]: pgmap v20112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:18 smithi096 ceph-mon[32753]: pgmap v20112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37277.273213 secs 2024-04-26T22:12:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37277.273213 secs 2024-04-26T22:12:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37277.273213 secs 2024-04-26T22:12:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37280.247483 secs 2024-04-26T22:12:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:20 smithi194 ceph-mon[33017]: pgmap v20113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37280.247483 secs 2024-04-26T22:12:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:20 smithi033 ceph-mon[27845]: pgmap v20113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37280.247483 secs 2024-04-26T22:12:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:20 smithi096 ceph-mon[32753]: pgmap v20113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:22 smithi194 ceph-mon[33017]: pgmap v20114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:22 smithi033 ceph-mon[27845]: pgmap v20114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:22 smithi096 ceph-mon[32753]: pgmap v20114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37282.273313 secs 2024-04-26T22:12:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:24 smithi194 ceph-mon[33017]: pgmap v20115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37282.273313 secs 2024-04-26T22:12:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:24 smithi033 ceph-mon[27845]: pgmap v20115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37282.273313 secs 2024-04-26T22:12:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:24 smithi096 ceph-mon[32753]: pgmap v20115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37285.247605 secs 2024-04-26T22:12:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37285.247605 secs 2024-04-26T22:12:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37285.247605 secs 2024-04-26T22:12:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:26 smithi194 ceph-mon[33017]: pgmap v20116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:26 smithi033 ceph-mon[27845]: pgmap v20116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:26 smithi096 ceph-mon[32753]: pgmap v20116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:28.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:28 smithi194 ceph-mon[33017]: pgmap v20117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:28 smithi033 ceph-mon[27845]: pgmap v20117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:28 smithi096 ceph-mon[32753]: pgmap v20117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37287.273415 secs 2024-04-26T22:12:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37287.273415 secs 2024-04-26T22:12:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37287.273415 secs 2024-04-26T22:12:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:30.223+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:30.223+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:30.223+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:12:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37290.247724 secs 2024-04-26T22:12:30.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:30 smithi096 ceph-mon[32753]: pgmap v20118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37290.247724 secs 2024-04-26T22:12:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:30 smithi194 ceph-mon[33017]: pgmap v20118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37290.247724 secs 2024-04-26T22:12:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:30 smithi033 ceph-mon[27845]: pgmap v20118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:32 smithi194 ceph-mon[33017]: pgmap v20119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:32 smithi033 ceph-mon[27845]: pgmap v20119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:32 smithi096 ceph-mon[32753]: pgmap v20119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37292.273515 secs 2024-04-26T22:12:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:34 smithi194 ceph-mon[33017]: pgmap v20120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37292.273515 secs 2024-04-26T22:12:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:34 smithi033 ceph-mon[27845]: pgmap v20120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37292.273515 secs 2024-04-26T22:12:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:34 smithi096 ceph-mon[32753]: pgmap v20120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:35.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37295.247869 secs 2024-04-26T22:12:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37295.247869 secs 2024-04-26T22:12:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37295.247869 secs 2024-04-26T22:12:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:36 smithi194 ceph-mon[33017]: pgmap v20121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:36 smithi033 ceph-mon[27845]: pgmap v20121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:36 smithi096 ceph-mon[32753]: pgmap v20121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:12:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:38 smithi194 ceph-mon[33017]: pgmap v20122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37297.273575 secs 2024-04-26T22:12:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:38 smithi033 ceph-mon[27845]: pgmap v20122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37297.273575 secs 2024-04-26T22:12:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:38 smithi096 ceph-mon[32753]: pgmap v20122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37297.273575 secs 2024-04-26T22:12:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37300.248007 secs 2024-04-26T22:12:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:40 smithi194 ceph-mon[33017]: pgmap v20123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37300.248007 secs 2024-04-26T22:12:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:40 smithi033 ceph-mon[27845]: pgmap v20123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37300.248007 secs 2024-04-26T22:12:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:40 smithi096 ceph-mon[32753]: pgmap v20123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:42 smithi194 ceph-mon[33017]: pgmap v20124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:42 smithi033 ceph-mon[27845]: pgmap v20124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:42 smithi096 ceph-mon[32753]: pgmap v20124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:12:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37302.273659 secs 2024-04-26T22:12:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37302.273659 secs 2024-04-26T22:12:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37302.273659 secs 2024-04-26T22:12:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:44 smithi194 ceph-mon[33017]: pgmap v20125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:44 smithi033 ceph-mon[27845]: pgmap v20125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:44 smithi096 ceph-mon[32753]: pgmap v20125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:45.237+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:45.237+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:12:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:12:45.237+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:12:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37305.248147 secs 2024-04-26T22:12:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37305.248147 secs 2024-04-26T22:12:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37305.248147 secs 2024-04-26T22:12:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:46 smithi194 ceph-mon[33017]: pgmap v20126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:12:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:46 smithi033 ceph-mon[27845]: pgmap v20126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:12:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:46 smithi096 ceph-mon[32753]: pgmap v20126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:12:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:48 smithi194 ceph-mon[33017]: pgmap v20127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37307.273745 secs 2024-04-26T22:12:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:48 smithi033 ceph-mon[27845]: pgmap v20127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37307.273745 secs 2024-04-26T22:12:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:48 smithi096 ceph-mon[32753]: pgmap v20127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37307.273745 secs 2024-04-26T22:12:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37310.248267 secs 2024-04-26T22:12:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:50 smithi194 ceph-mon[33017]: pgmap v20128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:12:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37310.248267 secs 2024-04-26T22:12:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:50 smithi033 ceph-mon[27845]: pgmap v20128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:12:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37310.248267 secs 2024-04-26T22:12:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:50 smithi096 ceph-mon[32753]: pgmap v20128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:12:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:52 smithi194 ceph-mon[33017]: pgmap v20129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:12:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:52 smithi033 ceph-mon[27845]: pgmap v20129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:12:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:52 smithi096 ceph-mon[32753]: pgmap v20129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:12:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37312.273830 secs 2024-04-26T22:12:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37312.273830 secs 2024-04-26T22:12:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37312.273830 secs 2024-04-26T22:12:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:54 smithi194 ceph-mon[33017]: pgmap v20130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:54 smithi033 ceph-mon[27845]: pgmap v20130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:54 smithi096 ceph-mon[32753]: pgmap v20130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:12:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37315.248405 secs 2024-04-26T22:12:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37315.248405 secs 2024-04-26T22:12:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37315.248405 secs 2024-04-26T22:12:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:56 smithi194 ceph-mon[33017]: pgmap v20131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:12:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:56 smithi033 ceph-mon[27845]: pgmap v20131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:12:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:56 smithi096 ceph-mon[32753]: pgmap v20131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:12:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:58 smithi194 ceph-mon[33017]: pgmap v20132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37317.273915 secs 2024-04-26T22:12:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:58 smithi033 ceph-mon[27845]: pgmap v20132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37317.273915 secs 2024-04-26T22:12:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:58 smithi096 ceph-mon[32753]: pgmap v20132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:12:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37317.273915 secs 2024-04-26T22:12:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:12:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37320.248527 secs 2024-04-26T22:12:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:12:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37320.248527 secs 2024-04-26T22:12:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:12:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37320.248527 secs 2024-04-26T22:13:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:00.254+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:00.254+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:13:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:00.254+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:00 smithi194 ceph-mon[33017]: pgmap v20133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:00 smithi033 ceph-mon[27845]: pgmap v20133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:00 smithi096 ceph-mon[32753]: pgmap v20133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:02 smithi194 ceph-mon[33017]: pgmap v20134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:13:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:02 smithi033 ceph-mon[27845]: pgmap v20134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:13:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:02 smithi096 ceph-mon[32753]: pgmap v20134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:13:03.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37322.274000 secs 2024-04-26T22:13:03.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:13:03.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:13:03.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37322.274000 secs 2024-04-26T22:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:13:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:13:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37322.274000 secs 2024-04-26T22:13:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:13:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:13:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:13:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:04 smithi033 ceph-mon[27845]: pgmap v20135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:04 smithi033 ceph-mon[27845]: pgmap v20136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37325.248603 secs 2024-04-26T22:13:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:04 smithi096 ceph-mon[32753]: pgmap v20135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:04 smithi096 ceph-mon[32753]: pgmap v20136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37325.248603 secs 2024-04-26T22:13:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:04 smithi194 ceph-mon[33017]: pgmap v20135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:04 smithi194 ceph-mon[33017]: pgmap v20136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37325.248603 secs 2024-04-26T22:13:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:06 smithi033 ceph-mon[27845]: pgmap v20137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:06 smithi096 ceph-mon[32753]: pgmap v20137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:06 smithi194 ceph-mon[33017]: pgmap v20137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:08 smithi033 ceph-mon[27845]: pgmap v20138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37327.274089 secs 2024-04-26T22:13:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:08 smithi096 ceph-mon[32753]: pgmap v20138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37327.274089 secs 2024-04-26T22:13:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:08 smithi194 ceph-mon[33017]: pgmap v20138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37327.274089 secs 2024-04-26T22:13:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37330.248714 secs 2024-04-26T22:13:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37330.248714 secs 2024-04-26T22:13:10.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37330.248714 secs 2024-04-26T22:13:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:10 smithi033 ceph-mon[27845]: pgmap v20139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:10 smithi096 ceph-mon[32753]: pgmap v20139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:10 smithi194 ceph-mon[33017]: pgmap v20139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:12 smithi033 ceph-mon[27845]: pgmap v20140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:12 smithi096 ceph-mon[32753]: pgmap v20140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:12 smithi194 ceph-mon[33017]: pgmap v20140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:13:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37332.274181 secs 2024-04-26T22:13:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37332.274181 secs 2024-04-26T22:13:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37332.274181 secs 2024-04-26T22:13:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:14 smithi033 ceph-mon[27845]: pgmap v20141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:13:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37335.248787 secs 2024-04-26T22:13:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:14 smithi096 ceph-mon[32753]: pgmap v20141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:13:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37335.248787 secs 2024-04-26T22:13:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:14 smithi194 ceph-mon[33017]: pgmap v20141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:13:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37335.248787 secs 2024-04-26T22:13:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:15.269+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:15.269+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:13:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:15.269+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:16 smithi033 ceph-mon[27845]: pgmap v20142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:16 smithi096 ceph-mon[32753]: pgmap v20142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:16 smithi194 ceph-mon[33017]: pgmap v20142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:18 smithi033 ceph-mon[27845]: pgmap v20143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37337.274266 secs 2024-04-26T22:13:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:18 smithi096 ceph-mon[32753]: pgmap v20143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37337.274266 secs 2024-04-26T22:13:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:18 smithi194 ceph-mon[33017]: pgmap v20143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37337.274266 secs 2024-04-26T22:13:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37340.248910 secs 2024-04-26T22:13:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37340.248910 secs 2024-04-26T22:13:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37340.248910 secs 2024-04-26T22:13:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:20 smithi096 ceph-mon[32753]: pgmap v20144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:20 smithi194 ceph-mon[33017]: pgmap v20144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:21.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:20 smithi033 ceph-mon[27845]: pgmap v20144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:22 smithi194 ceph-mon[33017]: pgmap v20145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:22 smithi033 ceph-mon[27845]: pgmap v20145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:22 smithi096 ceph-mon[32753]: pgmap v20145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37342.274355 secs 2024-04-26T22:13:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37342.274355 secs 2024-04-26T22:13:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37342.274355 secs 2024-04-26T22:13:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:24 smithi194 ceph-mon[33017]: pgmap v20146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37345.249029 secs 2024-04-26T22:13:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:24 smithi033 ceph-mon[27845]: pgmap v20146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37345.249029 secs 2024-04-26T22:13:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:24 smithi096 ceph-mon[32753]: pgmap v20146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37345.249029 secs 2024-04-26T22:13:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:26 smithi194 ceph-mon[33017]: pgmap v20147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:26 smithi033 ceph-mon[27845]: pgmap v20147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:26 smithi096 ceph-mon[32753]: pgmap v20147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:28 smithi194 ceph-mon[33017]: pgmap v20148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37347.274455 secs 2024-04-26T22:13:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:28 smithi033 ceph-mon[27845]: pgmap v20148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37347.274455 secs 2024-04-26T22:13:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:28 smithi096 ceph-mon[32753]: pgmap v20148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37347.274455 secs 2024-04-26T22:13:30.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37350.249171 secs 2024-04-26T22:13:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37350.249171 secs 2024-04-26T22:13:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37350.249171 secs 2024-04-26T22:13:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:30.285+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:30.285+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:30.285+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:13:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:30 smithi194 ceph-mon[33017]: pgmap v20149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:30 smithi033 ceph-mon[27845]: pgmap v20149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:30 smithi096 ceph-mon[32753]: pgmap v20149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:32 smithi194 ceph-mon[33017]: pgmap v20150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:32 smithi033 ceph-mon[27845]: pgmap v20150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:32 smithi096 ceph-mon[32753]: pgmap v20150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37352.274558 secs 2024-04-26T22:13:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37352.274558 secs 2024-04-26T22:13:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37352.274558 secs 2024-04-26T22:13:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:34 smithi194 ceph-mon[33017]: pgmap v20151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37355.249299 secs 2024-04-26T22:13:35.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:34 smithi033 ceph-mon[27845]: pgmap v20151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:35.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37355.249299 secs 2024-04-26T22:13:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:34 smithi096 ceph-mon[32753]: pgmap v20151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37355.249299 secs 2024-04-26T22:13:37.112 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:36 smithi194 ceph-mon[33017]: pgmap v20152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:37.185 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:36 smithi033 ceph-mon[27845]: pgmap v20152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:37.189 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:36 smithi096 ceph-mon[32753]: pgmap v20152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:38 smithi194 ceph-mon[33017]: pgmap v20153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37357.274662 secs 2024-04-26T22:13:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:38 smithi033 ceph-mon[27845]: pgmap v20153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37357.274662 secs 2024-04-26T22:13:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:38 smithi096 ceph-mon[32753]: pgmap v20153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37357.274662 secs 2024-04-26T22:13:40.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37360.249414 secs 2024-04-26T22:13:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37360.249414 secs 2024-04-26T22:13:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37360.249414 secs 2024-04-26T22:13:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:40 smithi194 ceph-mon[33017]: pgmap v20154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:40 smithi033 ceph-mon[27845]: pgmap v20154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:40 smithi096 ceph-mon[32753]: pgmap v20154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:13:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:42 smithi194 ceph-mon[33017]: pgmap v20155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:42 smithi033 ceph-mon[27845]: pgmap v20155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:42 smithi096 ceph-mon[32753]: pgmap v20155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:13:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37362.274752 secs 2024-04-26T22:13:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37362.274752 secs 2024-04-26T22:13:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37362.274752 secs 2024-04-26T22:13:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:44 smithi194 ceph-mon[33017]: pgmap v20156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37365.249499 secs 2024-04-26T22:13:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:44 smithi033 ceph-mon[27845]: pgmap v20156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37365.249499 secs 2024-04-26T22:13:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:44 smithi096 ceph-mon[32753]: pgmap v20156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37365.249499 secs 2024-04-26T22:13:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:45.299+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:45.299+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:13:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:13:45.299+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:13:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:46 smithi194 ceph-mon[33017]: pgmap v20157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:46 smithi033 ceph-mon[27845]: pgmap v20157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:46 smithi096 ceph-mon[32753]: pgmap v20157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:48 smithi194 ceph-mon[33017]: pgmap v20158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37367.274854 secs 2024-04-26T22:13:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:48 smithi033 ceph-mon[27845]: pgmap v20158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37367.274854 secs 2024-04-26T22:13:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:48 smithi096 ceph-mon[32753]: pgmap v20158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37367.274854 secs 2024-04-26T22:13:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37370.249633 secs 2024-04-26T22:13:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37370.249633 secs 2024-04-26T22:13:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37370.249633 secs 2024-04-26T22:13:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:50 smithi194 ceph-mon[33017]: pgmap v20159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:50 smithi033 ceph-mon[27845]: pgmap v20159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:50 smithi096 ceph-mon[32753]: pgmap v20159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:52 smithi194 ceph-mon[33017]: pgmap v20160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:53.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:52 smithi033 ceph-mon[27845]: pgmap v20160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:52 smithi096 ceph-mon[32753]: pgmap v20160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37372.274955 secs 2024-04-26T22:13:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37372.274955 secs 2024-04-26T22:13:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37372.274955 secs 2024-04-26T22:13:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:54 smithi194 ceph-mon[33017]: pgmap v20161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37375.249700 secs 2024-04-26T22:13:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:54 smithi033 ceph-mon[27845]: pgmap v20161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37375.249700 secs 2024-04-26T22:13:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:54 smithi096 ceph-mon[32753]: pgmap v20161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37375.249700 secs 2024-04-26T22:13:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:56 smithi194 ceph-mon[33017]: pgmap v20162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:56 smithi033 ceph-mon[27845]: pgmap v20162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:56 smithi096 ceph-mon[32753]: pgmap v20162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:13:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:58 smithi194 ceph-mon[33017]: pgmap v20163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37377.275042 secs 2024-04-26T22:13:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:58 smithi033 ceph-mon[27845]: pgmap v20163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37377.275042 secs 2024-04-26T22:13:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:58 smithi096 ceph-mon[32753]: pgmap v20163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:13:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37377.275042 secs 2024-04-26T22:14:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:13:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37380.249833 secs 2024-04-26T22:14:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:13:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37380.249833 secs 2024-04-26T22:14:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:13:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37380.249833 secs 2024-04-26T22:14:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:00.310+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:00.310+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:14:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:00.310+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:00 smithi194 ceph-mon[33017]: pgmap v20164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:00 smithi033 ceph-mon[27845]: pgmap v20164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:00 smithi096 ceph-mon[32753]: pgmap v20164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:02 smithi033 ceph-mon[27845]: pgmap v20165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:02 smithi096 ceph-mon[32753]: pgmap v20165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:02 smithi194 ceph-mon[33017]: pgmap v20165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37382.275103 secs 2024-04-26T22:14:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:14:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37382.275103 secs 2024-04-26T22:14:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37382.275103 secs 2024-04-26T22:14:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:14:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:04 smithi033 ceph-mon[27845]: pgmap v20166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37385.249921 secs 2024-04-26T22:14:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:04 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:04 smithi096 ceph-mon[32753]: pgmap v20166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37385.249921 secs 2024-04-26T22:14:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:04 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:04 smithi194 ceph-mon[33017]: pgmap v20166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37385.249921 secs 2024-04-26T22:14:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:04 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:14:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:14:06.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:14:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:14:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:14:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:14:06.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:14:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:06 smithi033 ceph-mon[27845]: pgmap v20167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:06 smithi033 ceph-mon[27845]: pgmap v20168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:14:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:06 smithi033 ceph-mon[27845]: pgmap v20169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:06 smithi096 ceph-mon[32753]: pgmap v20167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:06 smithi096 ceph-mon[32753]: pgmap v20168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:14:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:06 smithi096 ceph-mon[32753]: pgmap v20169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:06 smithi194 ceph-mon[33017]: pgmap v20167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:06 smithi194 ceph-mon[33017]: pgmap v20168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:14:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:06 smithi194 ceph-mon[33017]: pgmap v20169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:08 smithi033 ceph-mon[27845]: pgmap v20170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37387.275187 secs 2024-04-26T22:14:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:08 smithi096 ceph-mon[32753]: pgmap v20170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:09.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37387.275187 secs 2024-04-26T22:14:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:08 smithi194 ceph-mon[33017]: pgmap v20170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37387.275187 secs 2024-04-26T22:14:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37390.250035 secs 2024-04-26T22:14:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37390.250035 secs 2024-04-26T22:14:10.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37390.250035 secs 2024-04-26T22:14:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:10 smithi033 ceph-mon[27845]: pgmap v20171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:10 smithi096 ceph-mon[32753]: pgmap v20171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:10 smithi194 ceph-mon[33017]: pgmap v20171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:12 smithi033 ceph-mon[27845]: pgmap v20172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:12 smithi096 ceph-mon[32753]: pgmap v20172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:12 smithi194 ceph-mon[33017]: pgmap v20172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:14:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37392.275275 secs 2024-04-26T22:14:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37392.275275 secs 2024-04-26T22:14:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37392.275275 secs 2024-04-26T22:14:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:14 smithi033 ceph-mon[27845]: pgmap v20173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:14:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37395.250150 secs 2024-04-26T22:14:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:14 smithi096 ceph-mon[32753]: pgmap v20173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:14:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37395.250150 secs 2024-04-26T22:14:15.326 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:15.326+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:15.326 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:15.326+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:14:15.326 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:15.326+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:14 smithi194 ceph-mon[33017]: pgmap v20173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:14:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37395.250150 secs 2024-04-26T22:14:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:16 smithi033 ceph-mon[27845]: pgmap v20174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:14:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:16 smithi096 ceph-mon[32753]: pgmap v20174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:14:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:16 smithi194 ceph-mon[33017]: pgmap v20174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:18 smithi096 ceph-mon[32753]: pgmap v20175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37397.275364 secs 2024-04-26T22:14:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:18 smithi194 ceph-mon[33017]: pgmap v20175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37397.275364 secs 2024-04-26T22:14:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:18 smithi033 ceph-mon[27845]: pgmap v20175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37397.275364 secs 2024-04-26T22:14:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37400.250263 secs 2024-04-26T22:14:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37400.250263 secs 2024-04-26T22:14:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37400.250263 secs 2024-04-26T22:14:21.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:20 smithi033 ceph-mon[27845]: pgmap v20176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:20 smithi194 ceph-mon[33017]: pgmap v20176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:20 smithi096 ceph-mon[32753]: pgmap v20176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:22 smithi194 ceph-mon[33017]: pgmap v20177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:23.523 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:22 smithi096 ceph-mon[32753]: pgmap v20177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:23.523 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:22 smithi033 ceph-mon[27845]: pgmap v20177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37402.275452 secs 2024-04-26T22:14:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37402.275452 secs 2024-04-26T22:14:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37402.275452 secs 2024-04-26T22:14:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:24 smithi194 ceph-mon[33017]: pgmap v20178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37405.250379 secs 2024-04-26T22:14:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:24 smithi033 ceph-mon[27845]: pgmap v20178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37405.250379 secs 2024-04-26T22:14:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:24 smithi096 ceph-mon[32753]: pgmap v20178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37405.250379 secs 2024-04-26T22:14:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:26 smithi194 ceph-mon[33017]: pgmap v20179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:26 smithi033 ceph-mon[27845]: pgmap v20179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:26 smithi096 ceph-mon[32753]: pgmap v20179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:28.167+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:14:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:28.167+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:14:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:28.168+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 292 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:14:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:28.168+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 292 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:28 smithi194 ceph-mon[33017]: pgmap v20180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37407.275542 secs 2024-04-26T22:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:28 smithi033 ceph-mon[27845]: pgmap v20180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37407.275542 secs 2024-04-26T22:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:28 smithi096 ceph-mon[32753]: pgmap v20180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37407.275542 secs 2024-04-26T22:14:30.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37410.250496 secs 2024-04-26T22:14:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:30.340+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:30.340+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:14:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:30.340+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:30.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37410.250496 secs 2024-04-26T22:14:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37410.250496 secs 2024-04-26T22:14:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:30 smithi194 ceph-mon[33017]: pgmap v20181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:30 smithi033 ceph-mon[27845]: pgmap v20181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:30 smithi096 ceph-mon[32753]: pgmap v20181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:32 smithi194 ceph-mon[33017]: pgmap v20182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:32 smithi033 ceph-mon[27845]: pgmap v20182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:32 smithi096 ceph-mon[32753]: pgmap v20182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37412.275629 secs 2024-04-26T22:14:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37412.275629 secs 2024-04-26T22:14:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37412.275629 secs 2024-04-26T22:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:35 smithi194 ceph-mon[33017]: pgmap v20183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:35.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37415.250567 secs 2024-04-26T22:14:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:35 smithi033 ceph-mon[27845]: pgmap v20183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37415.250567 secs 2024-04-26T22:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:35 smithi096 ceph-mon[32753]: pgmap v20183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37415.250567 secs 2024-04-26T22:14:37.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:37 smithi194 ceph-mon[33017]: pgmap v20184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:37 smithi033 ceph-mon[27845]: pgmap v20184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:37 smithi096 ceph-mon[32753]: pgmap v20184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:39 smithi194 ceph-mon[33017]: pgmap v20185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37417.275714 secs 2024-04-26T22:14:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:39 smithi033 ceph-mon[27845]: pgmap v20185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37417.275714 secs 2024-04-26T22:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:39 smithi096 ceph-mon[32753]: pgmap v20185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37417.275714 secs 2024-04-26T22:14:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37420.250659 secs 2024-04-26T22:14:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37420.250659 secs 2024-04-26T22:14:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37420.250659 secs 2024-04-26T22:14:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:41 smithi194 ceph-mon[33017]: pgmap v20186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:41 smithi033 ceph-mon[27845]: pgmap v20186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:41 smithi096 ceph-mon[32753]: pgmap v20186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:43 smithi194 ceph-mon[33017]: pgmap v20187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:43 smithi033 ceph-mon[27845]: pgmap v20187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:43 smithi096 ceph-mon[32753]: pgmap v20187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37422.275776 secs 2024-04-26T22:14:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37422.275776 secs 2024-04-26T22:14:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37422.275776 secs 2024-04-26T22:14:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:45 smithi194 ceph-mon[33017]: pgmap v20188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37425.250793 secs 2024-04-26T22:14:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:45.355+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:45.355+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:14:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:14:45.355+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:14:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:45 smithi033 ceph-mon[27845]: pgmap v20188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37425.250793 secs 2024-04-26T22:14:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:45 smithi096 ceph-mon[32753]: pgmap v20188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37425.250793 secs 2024-04-26T22:14:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:46 smithi194 ceph-mon[33017]: pgmap v20189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:46 smithi033 ceph-mon[27845]: pgmap v20189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:46 smithi096 ceph-mon[32753]: pgmap v20189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:48 smithi033 ceph-mon[27845]: pgmap v20190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37427.275859 secs 2024-04-26T22:14:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:48 smithi096 ceph-mon[32753]: pgmap v20190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37427.275859 secs 2024-04-26T22:14:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:48 smithi194 ceph-mon[33017]: pgmap v20190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37427.275859 secs 2024-04-26T22:14:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37430.250944 secs 2024-04-26T22:14:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37430.250944 secs 2024-04-26T22:14:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37430.250944 secs 2024-04-26T22:14:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:50 smithi033 ceph-mon[27845]: pgmap v20191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:50 smithi096 ceph-mon[32753]: pgmap v20191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:50 smithi194 ceph-mon[33017]: pgmap v20191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:52 smithi033 ceph-mon[27845]: pgmap v20192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:52 smithi096 ceph-mon[32753]: pgmap v20192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:53.105 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:52 smithi194 ceph-mon[33017]: pgmap v20192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37432.275955 secs 2024-04-26T22:14:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37432.275955 secs 2024-04-26T22:14:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37432.275955 secs 2024-04-26T22:14:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:54 smithi033 ceph-mon[27845]: pgmap v20193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37435.251060 secs 2024-04-26T22:14:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:54 smithi096 ceph-mon[32753]: pgmap v20193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37435.251060 secs 2024-04-26T22:14:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:54 smithi194 ceph-mon[33017]: pgmap v20193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37435.251060 secs 2024-04-26T22:14:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:56 smithi033 ceph-mon[27845]: pgmap v20194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:56 smithi096 ceph-mon[32753]: pgmap v20194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:56 smithi194 ceph-mon[33017]: pgmap v20194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:14:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:58 smithi096 ceph-mon[32753]: pgmap v20195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37437.276053 secs 2024-04-26T22:14:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:58 smithi194 ceph-mon[33017]: pgmap v20195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37437.276053 secs 2024-04-26T22:14:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:58 smithi033 ceph-mon[27845]: pgmap v20195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:14:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37437.276053 secs 2024-04-26T22:15:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:14:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37440.251172 secs 2024-04-26T22:15:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:14:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37440.251172 secs 2024-04-26T22:15:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:14:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37440.251172 secs 2024-04-26T22:15:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:00.371+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:00.372+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:00.372+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:15:01.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:00 smithi096 ceph-mon[32753]: pgmap v20196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:00 smithi194 ceph-mon[33017]: pgmap v20196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:00 smithi033 ceph-mon[27845]: pgmap v20196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:02 smithi194 ceph-mon[33017]: pgmap v20197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:02 smithi033 ceph-mon[27845]: pgmap v20197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:02 smithi096 ceph-mon[32753]: pgmap v20197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37442.276142 secs 2024-04-26T22:15:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37442.276142 secs 2024-04-26T22:15:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37442.276142 secs 2024-04-26T22:15:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:04 smithi194 ceph-mon[33017]: pgmap v20198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37445.251298 secs 2024-04-26T22:15:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:04 smithi033 ceph-mon[27845]: pgmap v20198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37445.251298 secs 2024-04-26T22:15:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:04 smithi096 ceph-mon[32753]: pgmap v20198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37445.251298 secs 2024-04-26T22:15:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:06 smithi194 ceph-mon[33017]: pgmap v20199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:15:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:06 smithi033 ceph-mon[27845]: pgmap v20199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:15:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:06 smithi096 ceph-mon[32753]: pgmap v20199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:15:08.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:15:08.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:15:08.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:15:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:15:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:15:08.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:15:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:15:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:15:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:15:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:08 smithi194 ceph-mon[33017]: pgmap v20200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:08 smithi194 ceph-mon[33017]: pgmap v20201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37447.276204 secs 2024-04-26T22:15:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:08 smithi033 ceph-mon[27845]: pgmap v20200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:08 smithi033 ceph-mon[27845]: pgmap v20201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37447.276204 secs 2024-04-26T22:15:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:08 smithi096 ceph-mon[32753]: pgmap v20200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:08 smithi096 ceph-mon[32753]: pgmap v20201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37447.276204 secs 2024-04-26T22:15:10.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37450.251409 secs 2024-04-26T22:15:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37450.251409 secs 2024-04-26T22:15:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37450.251409 secs 2024-04-26T22:15:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:10 smithi194 ceph-mon[33017]: pgmap v20202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:15:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:10 smithi033 ceph-mon[27845]: pgmap v20202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:15:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:10 smithi096 ceph-mon[32753]: pgmap v20202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:15:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:12 smithi194 ceph-mon[33017]: pgmap v20203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:12 smithi033 ceph-mon[27845]: pgmap v20203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:12 smithi096 ceph-mon[32753]: pgmap v20203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37452.276293 secs 2024-04-26T22:15:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37452.276293 secs 2024-04-26T22:15:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37452.276293 secs 2024-04-26T22:15:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:14 smithi194 ceph-mon[33017]: pgmap v20204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37455.251531 secs 2024-04-26T22:15:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:14 smithi033 ceph-mon[27845]: pgmap v20204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37455.251531 secs 2024-04-26T22:15:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:14 smithi096 ceph-mon[32753]: pgmap v20204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37455.251531 secs 2024-04-26T22:15:15.385 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:15.385+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:15.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:15.385+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:15:15.386 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:15.385+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:16 smithi194 ceph-mon[33017]: pgmap v20205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:16 smithi033 ceph-mon[27845]: pgmap v20205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:16 smithi096 ceph-mon[32753]: pgmap v20205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:18 smithi194 ceph-mon[33017]: pgmap v20206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:15:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37457.276380 secs 2024-04-26T22:15:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:18 smithi033 ceph-mon[27845]: pgmap v20206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:15:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37457.276380 secs 2024-04-26T22:15:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:18 smithi096 ceph-mon[32753]: pgmap v20206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:15:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37457.276380 secs 2024-04-26T22:15:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37460.251646 secs 2024-04-26T22:15:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37460.251646 secs 2024-04-26T22:15:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37460.251646 secs 2024-04-26T22:15:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:20 smithi194 ceph-mon[33017]: pgmap v20207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:20 smithi033 ceph-mon[27845]: pgmap v20207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:20 smithi096 ceph-mon[32753]: pgmap v20207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:22 smithi194 ceph-mon[33017]: pgmap v20208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:22 smithi033 ceph-mon[27845]: pgmap v20208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:22 smithi096 ceph-mon[32753]: pgmap v20208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37462.276469 secs 2024-04-26T22:15:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37462.276469 secs 2024-04-26T22:15:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37462.276469 secs 2024-04-26T22:15:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:24 smithi194 ceph-mon[33017]: pgmap v20209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37465.251783 secs 2024-04-26T22:15:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:24 smithi033 ceph-mon[27845]: pgmap v20209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37465.251783 secs 2024-04-26T22:15:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:24 smithi096 ceph-mon[32753]: pgmap v20209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37465.251783 secs 2024-04-26T22:15:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:26 smithi194 ceph-mon[33017]: pgmap v20210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:26 smithi033 ceph-mon[27845]: pgmap v20210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:26 smithi096 ceph-mon[32753]: pgmap v20210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:28 smithi194 ceph-mon[33017]: pgmap v20211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37467.276569 secs 2024-04-26T22:15:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:28 smithi033 ceph-mon[27845]: pgmap v20211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37467.276569 secs 2024-04-26T22:15:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:28 smithi096 ceph-mon[32753]: pgmap v20211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37467.276569 secs 2024-04-26T22:15:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37470.251916 secs 2024-04-26T22:15:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37470.251916 secs 2024-04-26T22:15:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37470.251916 secs 2024-04-26T22:15:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:30.399+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:30.399+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:15:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:30.399+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:31.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:30 smithi096 ceph-mon[32753]: pgmap v20212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:30 smithi194 ceph-mon[33017]: pgmap v20212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:30 smithi033 ceph-mon[27845]: pgmap v20212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:32 smithi194 ceph-mon[33017]: pgmap v20213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:32 smithi033 ceph-mon[27845]: pgmap v20213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:32 smithi096 ceph-mon[32753]: pgmap v20213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37472.276655 secs 2024-04-26T22:15:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37472.276655 secs 2024-04-26T22:15:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37472.276655 secs 2024-04-26T22:15:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:34 smithi194 ceph-mon[33017]: pgmap v20214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37475.252032 secs 2024-04-26T22:15:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:34 smithi033 ceph-mon[27845]: pgmap v20214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37475.252032 secs 2024-04-26T22:15:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:34 smithi096 ceph-mon[32753]: pgmap v20214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37475.252032 secs 2024-04-26T22:15:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:36 smithi194 ceph-mon[33017]: pgmap v20215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:36 smithi033 ceph-mon[27845]: pgmap v20215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:36 smithi096 ceph-mon[32753]: pgmap v20215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:38 smithi194 ceph-mon[33017]: pgmap v20216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37477.276744 secs 2024-04-26T22:15:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:38 smithi033 ceph-mon[27845]: pgmap v20216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37477.276744 secs 2024-04-26T22:15:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:38 smithi096 ceph-mon[32753]: pgmap v20216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37477.276744 secs 2024-04-26T22:15:40.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37480.252148 secs 2024-04-26T22:15:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37480.252148 secs 2024-04-26T22:15:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37480.252148 secs 2024-04-26T22:15:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:40 smithi033 ceph-mon[27845]: pgmap v20217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:40 smithi096 ceph-mon[32753]: pgmap v20217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:40 smithi194 ceph-mon[33017]: pgmap v20217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:42 smithi033 ceph-mon[27845]: pgmap v20218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:42 smithi096 ceph-mon[32753]: pgmap v20218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:42 smithi194 ceph-mon[33017]: pgmap v20218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37482.276842 secs 2024-04-26T22:15:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37482.276842 secs 2024-04-26T22:15:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37482.276842 secs 2024-04-26T22:15:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:44 smithi033 ceph-mon[27845]: pgmap v20219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37485.252283 secs 2024-04-26T22:15:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:44 smithi096 ceph-mon[32753]: pgmap v20219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37485.252283 secs 2024-04-26T22:15:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:44 smithi194 ceph-mon[33017]: pgmap v20219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37485.252283 secs 2024-04-26T22:15:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:45.414+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:45.414+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:15:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:15:45.414+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:15:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:46 smithi033 ceph-mon[27845]: pgmap v20220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:46 smithi096 ceph-mon[32753]: pgmap v20220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:46 smithi194 ceph-mon[33017]: pgmap v20220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:48 smithi033 ceph-mon[27845]: pgmap v20221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37487.276944 secs 2024-04-26T22:15:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:48 smithi096 ceph-mon[32753]: pgmap v20221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37487.276944 secs 2024-04-26T22:15:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:48 smithi194 ceph-mon[33017]: pgmap v20221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37487.276944 secs 2024-04-26T22:15:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37490.252352 secs 2024-04-26T22:15:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37490.252352 secs 2024-04-26T22:15:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37490.252352 secs 2024-04-26T22:15:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:50 smithi033 ceph-mon[27845]: pgmap v20222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:50 smithi096 ceph-mon[32753]: pgmap v20222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:50 smithi194 ceph-mon[33017]: pgmap v20222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:52 smithi033 ceph-mon[27845]: pgmap v20223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:52 smithi096 ceph-mon[32753]: pgmap v20223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:52 smithi194 ceph-mon[33017]: pgmap v20223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:15:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37492.277047 secs 2024-04-26T22:15:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37492.277047 secs 2024-04-26T22:15:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37492.277047 secs 2024-04-26T22:15:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:54 smithi033 ceph-mon[27845]: pgmap v20224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:55.183 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37495.252449 secs 2024-04-26T22:15:55.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:54 smithi096 ceph-mon[32753]: pgmap v20224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:55.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37495.252449 secs 2024-04-26T22:15:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:54 smithi194 ceph-mon[33017]: pgmap v20224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37495.252449 secs 2024-04-26T22:15:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:56 smithi096 ceph-mon[32753]: pgmap v20225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:56 smithi194 ceph-mon[33017]: pgmap v20225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:56 smithi033 ceph-mon[27845]: pgmap v20225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:15:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:58 smithi096 ceph-mon[32753]: pgmap v20226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37497.277134 secs 2024-04-26T22:15:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:58 smithi194 ceph-mon[33017]: pgmap v20226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37497.277134 secs 2024-04-26T22:15:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:58 smithi033 ceph-mon[27845]: pgmap v20226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:15:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37497.277134 secs 2024-04-26T22:16:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:15:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37500.252509 secs 2024-04-26T22:16:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:15:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37500.252509 secs 2024-04-26T22:16:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:00.426+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:00.426+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:16:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:00.426+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:15:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37500.252509 secs 2024-04-26T22:16:01.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:00 smithi096 ceph-mon[32753]: pgmap v20227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:16:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:00 smithi194 ceph-mon[33017]: pgmap v20227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:16:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:00 smithi033 ceph-mon[27845]: pgmap v20227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:16:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:02 smithi194 ceph-mon[33017]: pgmap v20228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:16:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:02 smithi033 ceph-mon[27845]: pgmap v20228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:16:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:02 smithi096 ceph-mon[32753]: pgmap v20228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:16:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37502.277210 secs 2024-04-26T22:16:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37502.277210 secs 2024-04-26T22:16:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37502.277210 secs 2024-04-26T22:16:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:04 smithi194 ceph-mon[33017]: pgmap v20229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37505.252627 secs 2024-04-26T22:16:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:04 smithi033 ceph-mon[27845]: pgmap v20229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37505.252627 secs 2024-04-26T22:16:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:04 smithi096 ceph-mon[32753]: pgmap v20229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37505.252627 secs 2024-04-26T22:16:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:06 smithi194 ceph-mon[33017]: pgmap v20230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:06 smithi033 ceph-mon[27845]: pgmap v20230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:06 smithi096 ceph-mon[32753]: pgmap v20230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:08.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:16:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:16:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:16:09.280 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:08 smithi096 ceph-mon[32753]: pgmap v20231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:09.280 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37507.277294 secs 2024-04-26T22:16:09.280 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.280 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.280 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.280 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.290 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:08 smithi194 ceph-mon[33017]: pgmap v20231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:09.290 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37507.277294 secs 2024-04-26T22:16:09.290 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.290 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.290 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.290 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:08 smithi033 ceph-mon[27845]: pgmap v20231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37507.277294 secs 2024-04-26T22:16:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37510.252720 secs 2024-04-26T22:16:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37510.252720 secs 2024-04-26T22:16:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37510.252720 secs 2024-04-26T22:16:11.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:10 smithi033 ceph-mon[27845]: pgmap v20232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:16:11.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.311 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:10 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:10 smithi194 ceph-mon[33017]: pgmap v20232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:16:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:10 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:10 smithi096 ceph-mon[32753]: pgmap v20232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:16:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:10 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:12 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:12 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:12.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:12 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:13 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:13 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:13 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:13 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:13 smithi194 ceph-mon[33017]: pgmap v20233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:13.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:13 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:13 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:13 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:13 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:13 smithi033 ceph-mon[27845]: pgmap v20233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:13.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:13 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:13 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:13 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:13 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:13 smithi096 ceph-mon[32753]: pgmap v20233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37512.277379 secs 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:16:14.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:16:14.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:14 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37512.277379 secs 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:16:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:16:14.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:14 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37512.277379 secs 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:16:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:16:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:16:14.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:14 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:16:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:15 smithi194 ceph-mon[33017]: pgmap v20234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:15 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:16:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:15 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:16:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:15 smithi194 ceph-mon[33017]: pgmap v20235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:16:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:15 smithi194 ceph-mon[33017]: pgmap v20236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:16:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37515.252840 secs 2024-04-26T22:16:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:15 smithi033 ceph-mon[27845]: pgmap v20234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:15 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:16:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:15 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:16:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:15 smithi033 ceph-mon[27845]: pgmap v20235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:16:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:15 smithi033 ceph-mon[27845]: pgmap v20236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:16:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37515.252840 secs 2024-04-26T22:16:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:15 smithi096 ceph-mon[32753]: pgmap v20234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:15 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:16:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:15 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:16:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:15 smithi096 ceph-mon[32753]: pgmap v20235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:16:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:15 smithi096 ceph-mon[32753]: pgmap v20236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:16:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37515.252840 secs 2024-04-26T22:16:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:15.442+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:15.442+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:15.442+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:16:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:17 smithi194 ceph-mon[33017]: pgmap v20237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:17 smithi033 ceph-mon[27845]: pgmap v20237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:17 smithi096 ceph-mon[32753]: pgmap v20237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:19 smithi194 ceph-mon[33017]: pgmap v20238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T22:16:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37517.277469 secs 2024-04-26T22:16:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:19 smithi033 ceph-mon[27845]: pgmap v20238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T22:16:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37517.277469 secs 2024-04-26T22:16:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:19 smithi096 ceph-mon[32753]: pgmap v20238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-26T22:16:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37517.277469 secs 2024-04-26T22:16:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37520.252912 secs 2024-04-26T22:16:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37520.252912 secs 2024-04-26T22:16:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37520.252912 secs 2024-04-26T22:16:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:21 smithi194 ceph-mon[33017]: pgmap v20239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:21 smithi033 ceph-mon[27845]: pgmap v20239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:21 smithi096 ceph-mon[32753]: pgmap v20239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:23 smithi194 ceph-mon[33017]: pgmap v20240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:23 smithi033 ceph-mon[27845]: pgmap v20240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:23 smithi096 ceph-mon[32753]: pgmap v20240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:16:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37522.277529 secs 2024-04-26T22:16:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37522.277529 secs 2024-04-26T22:16:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37522.277529 secs 2024-04-26T22:16:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:25 smithi194 ceph-mon[33017]: pgmap v20241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:16:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37525.253033 secs 2024-04-26T22:16:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:25 smithi033 ceph-mon[27845]: pgmap v20241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:16:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37525.253033 secs 2024-04-26T22:16:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:25 smithi096 ceph-mon[32753]: pgmap v20241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:16:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37525.253033 secs 2024-04-26T22:16:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:26 smithi194 ceph-mon[33017]: pgmap v20242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:26 smithi033 ceph-mon[27845]: pgmap v20242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:26 smithi096 ceph-mon[32753]: pgmap v20242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:28 smithi033 ceph-mon[27845]: pgmap v20243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37527.277591 secs 2024-04-26T22:16:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:28 smithi096 ceph-mon[32753]: pgmap v20243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37527.277591 secs 2024-04-26T22:16:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:28 smithi194 ceph-mon[33017]: pgmap v20243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37527.277591 secs 2024-04-26T22:16:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37530.253148 secs 2024-04-26T22:16:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37530.253148 secs 2024-04-26T22:16:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37530.253148 secs 2024-04-26T22:16:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:30.449+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:30.449+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:16:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:30.449+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:30 smithi033 ceph-mon[27845]: pgmap v20244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:30 smithi096 ceph-mon[32753]: pgmap v20244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:30 smithi194 ceph-mon[33017]: pgmap v20244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:32 smithi033 ceph-mon[27845]: pgmap v20245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:32 smithi096 ceph-mon[32753]: pgmap v20245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:33.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:32 smithi194 ceph-mon[33017]: pgmap v20245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37532.277673 secs 2024-04-26T22:16:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37532.277673 secs 2024-04-26T22:16:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37532.277673 secs 2024-04-26T22:16:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:34 smithi033 ceph-mon[27845]: pgmap v20246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37535.253265 secs 2024-04-26T22:16:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:34 smithi096 ceph-mon[32753]: pgmap v20246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37535.253265 secs 2024-04-26T22:16:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:34 smithi194 ceph-mon[33017]: pgmap v20246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37535.253265 secs 2024-04-26T22:16:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:36 smithi033 ceph-mon[27845]: pgmap v20247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:36 smithi096 ceph-mon[32753]: pgmap v20247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:36 smithi194 ceph-mon[33017]: pgmap v20247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:38 smithi033 ceph-mon[27845]: pgmap v20248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37537.277757 secs 2024-04-26T22:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:38 smithi096 ceph-mon[32753]: pgmap v20248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37537.277757 secs 2024-04-26T22:16:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:38 smithi194 ceph-mon[33017]: pgmap v20248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37537.277757 secs 2024-04-26T22:16:40.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37540.253401 secs 2024-04-26T22:16:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37540.253401 secs 2024-04-26T22:16:40.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37540.253401 secs 2024-04-26T22:16:41.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:40 smithi033 ceph-mon[27845]: pgmap v20249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:40 smithi096 ceph-mon[32753]: pgmap v20249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:40 smithi194 ceph-mon[33017]: pgmap v20249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:42 smithi033 ceph-mon[27845]: pgmap v20250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:42 smithi096 ceph-mon[32753]: pgmap v20250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:43.188+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:16:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:43.188+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:16:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:43.189+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 293 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T22:16:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:43.189+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 293 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T22:16:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:42 smithi194 ceph-mon[33017]: pgmap v20250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37542.277844 secs 2024-04-26T22:16:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37542.277844 secs 2024-04-26T22:16:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37542.277844 secs 2024-04-26T22:16:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:44 smithi096 ceph-mon[32753]: pgmap v20251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37545.253524 secs 2024-04-26T22:16:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:44 smithi194 ceph-mon[33017]: pgmap v20251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37545.253524 secs 2024-04-26T22:16:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:44 smithi033 ceph-mon[27845]: pgmap v20251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37545.253524 secs 2024-04-26T22:16:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:45.463+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:45.463+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:16:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:16:45.463+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:16:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:46 smithi194 ceph-mon[33017]: pgmap v20252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:46 smithi033 ceph-mon[27845]: pgmap v20252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:46 smithi096 ceph-mon[32753]: pgmap v20252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:48 smithi194 ceph-mon[33017]: pgmap v20253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37547.277931 secs 2024-04-26T22:16:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:48 smithi033 ceph-mon[27845]: pgmap v20253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37547.277931 secs 2024-04-26T22:16:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:48 smithi096 ceph-mon[32753]: pgmap v20253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37547.277931 secs 2024-04-26T22:16:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37550.253666 secs 2024-04-26T22:16:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37550.253666 secs 2024-04-26T22:16:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37550.253666 secs 2024-04-26T22:16:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:50 smithi194 ceph-mon[33017]: pgmap v20254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:50 smithi033 ceph-mon[27845]: pgmap v20254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:50 smithi096 ceph-mon[32753]: pgmap v20254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:52 smithi194 ceph-mon[33017]: pgmap v20255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:52 smithi033 ceph-mon[27845]: pgmap v20255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:52 smithi096 ceph-mon[32753]: pgmap v20255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37552.278000 secs 2024-04-26T22:16:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37552.278000 secs 2024-04-26T22:16:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37552.278000 secs 2024-04-26T22:16:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:54 smithi194 ceph-mon[33017]: pgmap v20256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37555.253788 secs 2024-04-26T22:16:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:54 smithi033 ceph-mon[27845]: pgmap v20256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37555.253788 secs 2024-04-26T22:16:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:54 smithi096 ceph-mon[32753]: pgmap v20256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37555.253788 secs 2024-04-26T22:16:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:56 smithi194 ceph-mon[33017]: pgmap v20257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:56 smithi033 ceph-mon[27845]: pgmap v20257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:56 smithi096 ceph-mon[32753]: pgmap v20257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:16:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:58 smithi194 ceph-mon[33017]: pgmap v20258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37557.278084 secs 2024-04-26T22:16:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:58 smithi033 ceph-mon[27845]: pgmap v20258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37557.278084 secs 2024-04-26T22:16:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:58 smithi096 ceph-mon[32753]: pgmap v20258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:16:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37557.278084 secs 2024-04-26T22:17:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:16:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37560.253920 secs 2024-04-26T22:17:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:16:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37560.253920 secs 2024-04-26T22:17:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:16:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37560.253920 secs 2024-04-26T22:17:00.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:00.475+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:00.475 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:00.475+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:17:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:00.475+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:01.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:00 smithi096 ceph-mon[32753]: pgmap v20259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:00 smithi194 ceph-mon[33017]: pgmap v20259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:00 smithi033 ceph-mon[27845]: pgmap v20259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:02 smithi194 ceph-mon[33017]: pgmap v20260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:02 smithi033 ceph-mon[27845]: pgmap v20260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:02 smithi096 ceph-mon[32753]: pgmap v20260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37562.278166 secs 2024-04-26T22:17:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37562.278166 secs 2024-04-26T22:17:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37562.278166 secs 2024-04-26T22:17:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:05 smithi194 ceph-mon[33017]: pgmap v20261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37565.253984 secs 2024-04-26T22:17:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:05 smithi033 ceph-mon[27845]: pgmap v20261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37565.253984 secs 2024-04-26T22:17:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:05 smithi096 ceph-mon[32753]: pgmap v20261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37565.253984 secs 2024-04-26T22:17:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:07 smithi194 ceph-mon[33017]: pgmap v20262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:07 smithi033 ceph-mon[27845]: pgmap v20262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:07 smithi096 ceph-mon[32753]: pgmap v20262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:09 smithi194 ceph-mon[33017]: pgmap v20263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37567.278227 secs 2024-04-26T22:17:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:09 smithi033 ceph-mon[27845]: pgmap v20263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37567.278227 secs 2024-04-26T22:17:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:09 smithi096 ceph-mon[32753]: pgmap v20263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37567.278227 secs 2024-04-26T22:17:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37570.254099 secs 2024-04-26T22:17:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37570.254099 secs 2024-04-26T22:17:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37570.254099 secs 2024-04-26T22:17:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:11 smithi194 ceph-mon[33017]: pgmap v20264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:11 smithi033 ceph-mon[27845]: pgmap v20264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:11 smithi096 ceph-mon[32753]: pgmap v20264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:13 smithi194 ceph-mon[33017]: pgmap v20265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:13 smithi033 ceph-mon[27845]: pgmap v20265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:13 smithi096 ceph-mon[32753]: pgmap v20265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37572.278301 secs 2024-04-26T22:17:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37572.278301 secs 2024-04-26T22:17:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37572.278301 secs 2024-04-26T22:17:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:15 smithi194 ceph-mon[33017]: pgmap v20266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37575.254213 secs 2024-04-26T22:17:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:17:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:15 smithi033 ceph-mon[27845]: pgmap v20266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37575.254213 secs 2024-04-26T22:17:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:17:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:15 smithi096 ceph-mon[32753]: pgmap v20266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37575.254213 secs 2024-04-26T22:17:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:17:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:15.490+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:15.490+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:17:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:15.490+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:17:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:17:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:17:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:17:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:17:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:17:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:17:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:17:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:17:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:17 smithi194 ceph-mon[33017]: pgmap v20267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:17:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:17 smithi194 ceph-mon[33017]: pgmap v20268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:17 smithi033 ceph-mon[27845]: pgmap v20267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:17:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:17 smithi033 ceph-mon[27845]: pgmap v20268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:17 smithi096 ceph-mon[32753]: pgmap v20267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:17:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:17 smithi096 ceph-mon[32753]: pgmap v20268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:18 smithi194 ceph-mon[33017]: pgmap v20269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:18 smithi033 ceph-mon[27845]: pgmap v20269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:18 smithi096 ceph-mon[32753]: pgmap v20269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37577.278361 secs 2024-04-26T22:17:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37577.278361 secs 2024-04-26T22:17:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37577.278361 secs 2024-04-26T22:17:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37580.254329 secs 2024-04-26T22:17:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:20 smithi194 ceph-mon[33017]: pgmap v20270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37580.254329 secs 2024-04-26T22:17:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:20 smithi033 ceph-mon[27845]: pgmap v20270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37580.254329 secs 2024-04-26T22:17:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:20 smithi096 ceph-mon[32753]: pgmap v20270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:22 smithi194 ceph-mon[33017]: pgmap v20271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:22 smithi033 ceph-mon[27845]: pgmap v20271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:22 smithi096 ceph-mon[32753]: pgmap v20271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37582.278419 secs 2024-04-26T22:17:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37582.278419 secs 2024-04-26T22:17:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37582.278419 secs 2024-04-26T22:17:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:24 smithi033 ceph-mon[27845]: pgmap v20272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:24 smithi096 ceph-mon[32753]: pgmap v20272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:24 smithi194 ceph-mon[33017]: pgmap v20272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:17:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37585.254447 secs 2024-04-26T22:17:25.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37585.254447 secs 2024-04-26T22:17:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37585.254447 secs 2024-04-26T22:17:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:26 smithi033 ceph-mon[27845]: pgmap v20273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:17:26.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:26 smithi096 ceph-mon[32753]: pgmap v20273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:17:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:26 smithi194 ceph-mon[33017]: pgmap v20273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:17:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:28.197+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:17:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:28.197+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:17:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:28.198+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 294 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:17:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:28.198+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 294 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:17:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:28 smithi033 ceph-mon[27845]: pgmap v20274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:28.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37587.278501 secs 2024-04-26T22:17:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:28 smithi096 ceph-mon[32753]: pgmap v20274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37587.278501 secs 2024-04-26T22:17:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:28 smithi194 ceph-mon[33017]: pgmap v20274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37587.278501 secs 2024-04-26T22:17:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37590.254581 secs 2024-04-26T22:17:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37590.254581 secs 2024-04-26T22:17:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37590.254581 secs 2024-04-26T22:17:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:30.506+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:30.506+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:17:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:30.506+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:30 smithi033 ceph-mon[27845]: pgmap v20275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:30 smithi096 ceph-mon[32753]: pgmap v20275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:30 smithi194 ceph-mon[33017]: pgmap v20275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:32 smithi033 ceph-mon[27845]: pgmap v20276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:32 smithi096 ceph-mon[32753]: pgmap v20276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:32 smithi194 ceph-mon[33017]: pgmap v20276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37592.278585 secs 2024-04-26T22:17:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37592.278585 secs 2024-04-26T22:17:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37592.278585 secs 2024-04-26T22:17:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:34 smithi033 ceph-mon[27845]: pgmap v20277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37595.254697 secs 2024-04-26T22:17:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:34 smithi096 ceph-mon[32753]: pgmap v20277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37595.254697 secs 2024-04-26T22:17:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:34 smithi194 ceph-mon[33017]: pgmap v20277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37595.254697 secs 2024-04-26T22:17:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:36 smithi033 ceph-mon[27845]: pgmap v20278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:36 smithi096 ceph-mon[32753]: pgmap v20278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:36 smithi194 ceph-mon[33017]: pgmap v20278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:38 smithi033 ceph-mon[27845]: pgmap v20279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37597.278671 secs 2024-04-26T22:17:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:38 smithi096 ceph-mon[32753]: pgmap v20279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37597.278671 secs 2024-04-26T22:17:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:38 smithi194 ceph-mon[33017]: pgmap v20279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37597.278671 secs 2024-04-26T22:17:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37600.254810 secs 2024-04-26T22:17:40.006 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37600.254810 secs 2024-04-26T22:17:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37600.254810 secs 2024-04-26T22:17:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:40 smithi096 ceph-mon[32753]: pgmap v20280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:40 smithi194 ceph-mon[33017]: pgmap v20280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:40 smithi033 ceph-mon[27845]: pgmap v20280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:42 smithi194 ceph-mon[33017]: pgmap v20281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:42 smithi033 ceph-mon[27845]: pgmap v20281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:42 smithi096 ceph-mon[32753]: pgmap v20281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37602.278761 secs 2024-04-26T22:17:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37602.278761 secs 2024-04-26T22:17:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37602.278761 secs 2024-04-26T22:17:45.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:44 smithi033 ceph-mon[27845]: pgmap v20282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:45.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37605.254949 secs 2024-04-26T22:17:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:44 smithi194 ceph-mon[33017]: pgmap v20282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37605.254949 secs 2024-04-26T22:17:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:44 smithi096 ceph-mon[32753]: pgmap v20282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37605.254949 secs 2024-04-26T22:17:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:45.520+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:45.520+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:17:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:17:45.520+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:17:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:46 smithi194 ceph-mon[33017]: pgmap v20283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:47.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:46 smithi033 ceph-mon[27845]: pgmap v20283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:46 smithi096 ceph-mon[32753]: pgmap v20283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:48 smithi194 ceph-mon[33017]: pgmap v20284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37607.278846 secs 2024-04-26T22:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:48 smithi033 ceph-mon[27845]: pgmap v20284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37607.278846 secs 2024-04-26T22:17:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:48 smithi096 ceph-mon[32753]: pgmap v20284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37607.278846 secs 2024-04-26T22:17:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37610.255049 secs 2024-04-26T22:17:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37610.255049 secs 2024-04-26T22:17:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37610.255049 secs 2024-04-26T22:17:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:50 smithi194 ceph-mon[33017]: pgmap v20285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:50 smithi033 ceph-mon[27845]: pgmap v20285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:50 smithi096 ceph-mon[32753]: pgmap v20285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:52 smithi194 ceph-mon[33017]: pgmap v20286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:52 smithi033 ceph-mon[27845]: pgmap v20286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:52 smithi096 ceph-mon[32753]: pgmap v20286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37612.278933 secs 2024-04-26T22:17:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37612.278933 secs 2024-04-26T22:17:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37612.278933 secs 2024-04-26T22:17:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:54 smithi194 ceph-mon[33017]: pgmap v20287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37615.255195 secs 2024-04-26T22:17:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:54 smithi033 ceph-mon[27845]: pgmap v20287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37615.255195 secs 2024-04-26T22:17:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:54 smithi096 ceph-mon[32753]: pgmap v20287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37615.255195 secs 2024-04-26T22:17:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:56 smithi194 ceph-mon[33017]: pgmap v20288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:56 smithi033 ceph-mon[27845]: pgmap v20288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:56 smithi096 ceph-mon[32753]: pgmap v20288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:17:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:58 smithi194 ceph-mon[33017]: pgmap v20289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37617.279003 secs 2024-04-26T22:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:58 smithi033 ceph-mon[27845]: pgmap v20289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37617.279003 secs 2024-04-26T22:17:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:58 smithi096 ceph-mon[32753]: pgmap v20289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:17:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37617.279003 secs 2024-04-26T22:18:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:17:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37620.255260 secs 2024-04-26T22:18:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:17:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37620.255260 secs 2024-04-26T22:18:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:17:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37620.255260 secs 2024-04-26T22:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:00.536+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:00.536+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:00.536+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:00 smithi194 ceph-mon[33017]: pgmap v20290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:00 smithi033 ceph-mon[27845]: pgmap v20290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:00 smithi096 ceph-mon[32753]: pgmap v20290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:02 smithi194 ceph-mon[33017]: pgmap v20291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:02 smithi033 ceph-mon[27845]: pgmap v20291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:02 smithi096 ceph-mon[32753]: pgmap v20291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37622.279066 secs 2024-04-26T22:18:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37622.279066 secs 2024-04-26T22:18:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37622.279066 secs 2024-04-26T22:18:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:04 smithi194 ceph-mon[33017]: pgmap v20292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:05.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37625.255339 secs 2024-04-26T22:18:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:04 smithi033 ceph-mon[27845]: pgmap v20292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37625.255339 secs 2024-04-26T22:18:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:04 smithi096 ceph-mon[32753]: pgmap v20292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37625.255339 secs 2024-04-26T22:18:07.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:06 smithi194 ceph-mon[33017]: pgmap v20293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:06 smithi033 ceph-mon[27845]: pgmap v20293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:06 smithi096 ceph-mon[32753]: pgmap v20293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:08 smithi194 ceph-mon[33017]: pgmap v20294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37627.279129 secs 2024-04-26T22:18:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:08 smithi033 ceph-mon[27845]: pgmap v20294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37627.279129 secs 2024-04-26T22:18:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:08 smithi096 ceph-mon[32753]: pgmap v20294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37627.279129 secs 2024-04-26T22:18:10.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37630.255420 secs 2024-04-26T22:18:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37630.255420 secs 2024-04-26T22:18:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37630.255420 secs 2024-04-26T22:18:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:10 smithi194 ceph-mon[33017]: pgmap v20295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:10 smithi033 ceph-mon[27845]: pgmap v20295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:10 smithi096 ceph-mon[32753]: pgmap v20295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:12 smithi033 ceph-mon[27845]: pgmap v20296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:12 smithi096 ceph-mon[32753]: pgmap v20296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:12 smithi194 ceph-mon[33017]: pgmap v20296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37632.279190 secs 2024-04-26T22:18:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37632.279190 secs 2024-04-26T22:18:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37632.279190 secs 2024-04-26T22:18:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:14 smithi033 ceph-mon[27845]: pgmap v20297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:14 smithi096 ceph-mon[32753]: pgmap v20297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:14 smithi194 ceph-mon[33017]: pgmap v20297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:15.547+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:15.547+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:15.547+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:18:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37635.255537 secs 2024-04-26T22:18:15.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37635.255537 secs 2024-04-26T22:18:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37635.255537 secs 2024-04-26T22:18:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:16 smithi033 ceph-mon[27845]: pgmap v20298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:16.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:18:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:16 smithi096 ceph-mon[32753]: pgmap v20298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:18:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:16 smithi194 ceph-mon[33017]: pgmap v20298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:18:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:18:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:18:17.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:17 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:18:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:18:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:18:17.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:17 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:18:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:18:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:18:18.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:17 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:18:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:18 smithi033 ceph-mon[27845]: pgmap v20299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:18 smithi033 ceph-mon[27845]: pgmap v20300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:18.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37637.279249 secs 2024-04-26T22:18:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:18 smithi096 ceph-mon[32753]: pgmap v20299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:18 smithi096 ceph-mon[32753]: pgmap v20300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37637.279249 secs 2024-04-26T22:18:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:18 smithi194 ceph-mon[33017]: pgmap v20299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:18 smithi194 ceph-mon[33017]: pgmap v20300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37637.279249 secs 2024-04-26T22:18:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37640.255669 secs 2024-04-26T22:18:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37640.255669 secs 2024-04-26T22:18:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37640.255669 secs 2024-04-26T22:18:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:20 smithi033 ceph-mon[27845]: pgmap v20301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:18:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:20 smithi096 ceph-mon[32753]: pgmap v20301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:18:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:20 smithi194 ceph-mon[33017]: pgmap v20301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:18:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:22 smithi033 ceph-mon[27845]: pgmap v20302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:22 smithi096 ceph-mon[32753]: pgmap v20302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:22 smithi194 ceph-mon[33017]: pgmap v20302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:23.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37642.279300 secs 2024-04-26T22:18:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37642.279300 secs 2024-04-26T22:18:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37642.279300 secs 2024-04-26T22:18:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:24 smithi096 ceph-mon[32753]: pgmap v20303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37645.255788 secs 2024-04-26T22:18:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:24 smithi194 ceph-mon[33017]: pgmap v20303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37645.255788 secs 2024-04-26T22:18:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:24 smithi033 ceph-mon[27845]: pgmap v20303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37645.255788 secs 2024-04-26T22:18:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:26 smithi194 ceph-mon[33017]: pgmap v20304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:26 smithi033 ceph-mon[27845]: pgmap v20304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:26 smithi096 ceph-mon[32753]: pgmap v20304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:18:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:28 smithi194 ceph-mon[33017]: pgmap v20305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:18:29.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37647.279388 secs 2024-04-26T22:18:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:28 smithi033 ceph-mon[27845]: pgmap v20305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:18:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37647.279388 secs 2024-04-26T22:18:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:28 smithi096 ceph-mon[32753]: pgmap v20305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:18:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37647.279388 secs 2024-04-26T22:18:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37650.255921 secs 2024-04-26T22:18:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37650.255921 secs 2024-04-26T22:18:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37650.255921 secs 2024-04-26T22:18:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:30.562+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:30.562+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:30.563+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:18:31.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:30 smithi033 ceph-mon[27845]: pgmap v20306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:30 smithi194 ceph-mon[33017]: pgmap v20306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:30 smithi096 ceph-mon[32753]: pgmap v20306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:32.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:32 smithi096 ceph-mon[32753]: pgmap v20307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:32 smithi194 ceph-mon[33017]: pgmap v20307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:32 smithi033 ceph-mon[27845]: pgmap v20307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37652.279449 secs 2024-04-26T22:18:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37652.279449 secs 2024-04-26T22:18:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37652.279449 secs 2024-04-26T22:18:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:34 smithi194 ceph-mon[33017]: pgmap v20308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37655.256034 secs 2024-04-26T22:18:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:34 smithi033 ceph-mon[27845]: pgmap v20308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37655.256034 secs 2024-04-26T22:18:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:34 smithi096 ceph-mon[32753]: pgmap v20308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37655.256034 secs 2024-04-26T22:18:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:36 smithi194 ceph-mon[33017]: pgmap v20309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:36 smithi033 ceph-mon[27845]: pgmap v20309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:36 smithi096 ceph-mon[32753]: pgmap v20309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:38 smithi194 ceph-mon[33017]: pgmap v20310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37657.279536 secs 2024-04-26T22:18:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:38 smithi033 ceph-mon[27845]: pgmap v20310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37657.279536 secs 2024-04-26T22:18:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:38 smithi096 ceph-mon[32753]: pgmap v20310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37657.279536 secs 2024-04-26T22:18:40.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37660.256166 secs 2024-04-26T22:18:40.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37660.256166 secs 2024-04-26T22:18:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37660.256166 secs 2024-04-26T22:18:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:40 smithi194 ceph-mon[33017]: pgmap v20311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:40 smithi033 ceph-mon[27845]: pgmap v20311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:40 smithi096 ceph-mon[32753]: pgmap v20311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:42 smithi194 ceph-mon[33017]: pgmap v20312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:42 smithi033 ceph-mon[27845]: pgmap v20312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:42 smithi096 ceph-mon[32753]: pgmap v20312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37662.279622 secs 2024-04-26T22:18:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37662.279622 secs 2024-04-26T22:18:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37662.279622 secs 2024-04-26T22:18:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:44 smithi194 ceph-mon[33017]: pgmap v20313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37665.256290 secs 2024-04-26T22:18:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:44 smithi033 ceph-mon[27845]: pgmap v20313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37665.256290 secs 2024-04-26T22:18:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:44 smithi096 ceph-mon[32753]: pgmap v20313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37665.256290 secs 2024-04-26T22:18:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:45.576+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:45.576+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:18:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:18:45.576+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:18:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:46 smithi194 ceph-mon[33017]: pgmap v20314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:46 smithi033 ceph-mon[27845]: pgmap v20314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:46 smithi096 ceph-mon[32753]: pgmap v20314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:48 smithi194 ceph-mon[33017]: pgmap v20315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37667.279707 secs 2024-04-26T22:18:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:48 smithi033 ceph-mon[27845]: pgmap v20315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37667.279707 secs 2024-04-26T22:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:48 smithi096 ceph-mon[32753]: pgmap v20315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37667.279707 secs 2024-04-26T22:18:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37670.256411 secs 2024-04-26T22:18:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37670.256411 secs 2024-04-26T22:18:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37670.256411 secs 2024-04-26T22:18:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:50 smithi194 ceph-mon[33017]: pgmap v20316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:50 smithi033 ceph-mon[27845]: pgmap v20316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:50 smithi096 ceph-mon[32753]: pgmap v20316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:52 smithi194 ceph-mon[33017]: pgmap v20317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:52 smithi033 ceph-mon[27845]: pgmap v20317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:52 smithi096 ceph-mon[32753]: pgmap v20317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37672.279792 secs 2024-04-26T22:18:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37672.279792 secs 2024-04-26T22:18:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37672.279792 secs 2024-04-26T22:18:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:54 smithi194 ceph-mon[33017]: pgmap v20318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:55.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37675.256490 secs 2024-04-26T22:18:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:54 smithi033 ceph-mon[27845]: pgmap v20318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37675.256490 secs 2024-04-26T22:18:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:54 smithi096 ceph-mon[32753]: pgmap v20318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37675.256490 secs 2024-04-26T22:18:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:56 smithi194 ceph-mon[33017]: pgmap v20319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:56 smithi033 ceph-mon[27845]: pgmap v20319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:56 smithi096 ceph-mon[32753]: pgmap v20319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:18:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:58 smithi194 ceph-mon[33017]: pgmap v20320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37677.279877 secs 2024-04-26T22:18:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:58 smithi033 ceph-mon[27845]: pgmap v20320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37677.279877 secs 2024-04-26T22:18:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:58 smithi096 ceph-mon[32753]: pgmap v20320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:18:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37677.279877 secs 2024-04-26T22:19:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:18:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37680.256610 secs 2024-04-26T22:19:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:18:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37680.256610 secs 2024-04-26T22:19:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:18:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37680.256610 secs 2024-04-26T22:19:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:00.592+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:00.592+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:00.592+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:19:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:00 smithi194 ceph-mon[33017]: pgmap v20321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:00 smithi033 ceph-mon[27845]: pgmap v20321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:00 smithi096 ceph-mon[32753]: pgmap v20321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:02 smithi194 ceph-mon[33017]: pgmap v20322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:02 smithi033 ceph-mon[27845]: pgmap v20322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:02 smithi096 ceph-mon[32753]: pgmap v20322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37682.279936 secs 2024-04-26T22:19:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37682.279936 secs 2024-04-26T22:19:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37682.279936 secs 2024-04-26T22:19:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:04 smithi194 ceph-mon[33017]: pgmap v20323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37685.256765 secs 2024-04-26T22:19:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:04 smithi033 ceph-mon[27845]: pgmap v20323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:05.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37685.256765 secs 2024-04-26T22:19:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:04 smithi096 ceph-mon[32753]: pgmap v20323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37685.256765 secs 2024-04-26T22:19:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:06 smithi033 ceph-mon[27845]: pgmap v20324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:06 smithi096 ceph-mon[32753]: pgmap v20324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:06 smithi194 ceph-mon[33017]: pgmap v20324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:08 smithi033 ceph-mon[27845]: pgmap v20325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37687.280001 secs 2024-04-26T22:19:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:08 smithi096 ceph-mon[32753]: pgmap v20325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37687.280001 secs 2024-04-26T22:19:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:08 smithi194 ceph-mon[33017]: pgmap v20325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37687.280001 secs 2024-04-26T22:19:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37690.256877 secs 2024-04-26T22:19:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37690.256877 secs 2024-04-26T22:19:10.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37690.256877 secs 2024-04-26T22:19:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:10 smithi033 ceph-mon[27845]: pgmap v20326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:10 smithi096 ceph-mon[32753]: pgmap v20326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:10 smithi194 ceph-mon[33017]: pgmap v20326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:12 smithi033 ceph-mon[27845]: pgmap v20327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:12 smithi096 ceph-mon[32753]: pgmap v20327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:12 smithi194 ceph-mon[33017]: pgmap v20327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37692.280084 secs 2024-04-26T22:19:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37692.280084 secs 2024-04-26T22:19:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37692.280084 secs 2024-04-26T22:19:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:14 smithi033 ceph-mon[27845]: pgmap v20328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37695.256974 secs 2024-04-26T22:19:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:14 smithi096 ceph-mon[32753]: pgmap v20328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37695.256974 secs 2024-04-26T22:19:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:14 smithi194 ceph-mon[33017]: pgmap v20328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37695.256974 secs 2024-04-26T22:19:15.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:15.596+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:15.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:15.596+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:15.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:15.596+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:19:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:16 smithi033 ceph-mon[27845]: pgmap v20329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:16 smithi096 ceph-mon[32753]: pgmap v20329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:16 smithi194 ceph-mon[33017]: pgmap v20329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:19:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:19:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:19:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:18 smithi033 ceph-mon[27845]: pgmap v20330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37697.280156 secs 2024-04-26T22:19:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:19:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:19:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:18 smithi096 ceph-mon[32753]: pgmap v20330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37697.280156 secs 2024-04-26T22:19:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:19:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:19:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:18 smithi194 ceph-mon[33017]: pgmap v20330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37697.280156 secs 2024-04-26T22:19:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:19:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:19:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37700.257080 secs 2024-04-26T22:19:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:19 smithi033 ceph-mon[27845]: pgmap v20331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:19 smithi033 ceph-mon[27845]: pgmap v20332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:20.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:19 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:19:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37700.257080 secs 2024-04-26T22:19:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:19 smithi096 ceph-mon[32753]: pgmap v20331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:19 smithi096 ceph-mon[32753]: pgmap v20332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:19 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:19:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37700.257080 secs 2024-04-26T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:19 smithi194 ceph-mon[33017]: pgmap v20331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:19 smithi194 ceph-mon[33017]: pgmap v20332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:19 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:19:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:21 smithi033 ceph-mon[27845]: pgmap v20333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:21 smithi096 ceph-mon[32753]: pgmap v20333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:21 smithi194 ceph-mon[33017]: pgmap v20333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:23 smithi096 ceph-mon[32753]: pgmap v20334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37702.280211 secs 2024-04-26T22:19:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:23 smithi194 ceph-mon[33017]: pgmap v20334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37702.280211 secs 2024-04-26T22:19:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:23 smithi033 ceph-mon[27845]: pgmap v20334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37702.280211 secs 2024-04-26T22:19:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37705.257238 secs 2024-04-26T22:19:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37705.257238 secs 2024-04-26T22:19:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37705.257238 secs 2024-04-26T22:19:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:25 smithi194 ceph-mon[33017]: pgmap v20335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:25 smithi033 ceph-mon[27845]: pgmap v20335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:25 smithi096 ceph-mon[32753]: pgmap v20335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:27 smithi194 ceph-mon[33017]: pgmap v20336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:27 smithi033 ceph-mon[27845]: pgmap v20336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:27 smithi096 ceph-mon[32753]: pgmap v20336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:19:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37707.280293 secs 2024-04-26T22:19:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37707.280293 secs 2024-04-26T22:19:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37707.280293 secs 2024-04-26T22:19:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37710.257335 secs 2024-04-26T22:19:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:29 smithi194 ceph-mon[33017]: pgmap v20337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:19:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37710.257335 secs 2024-04-26T22:19:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:29 smithi033 ceph-mon[27845]: pgmap v20337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:19:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37710.257335 secs 2024-04-26T22:19:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:29 smithi096 ceph-mon[32753]: pgmap v20337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:19:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:30.611+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:30.612+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:19:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:30.612+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:32.234 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:31 smithi096 ceph-mon[32753]: pgmap v20338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:31 smithi194 ceph-mon[33017]: pgmap v20338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:31 smithi033 ceph-mon[27845]: pgmap v20338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:33 smithi194 ceph-mon[33017]: pgmap v20339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37712.280373 secs 2024-04-26T22:19:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:33 smithi033 ceph-mon[27845]: pgmap v20339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37712.280373 secs 2024-04-26T22:19:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:33 smithi096 ceph-mon[32753]: pgmap v20339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37712.280373 secs 2024-04-26T22:19:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37715.257446 secs 2024-04-26T22:19:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37715.257446 secs 2024-04-26T22:19:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37715.257446 secs 2024-04-26T22:19:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:35 smithi194 ceph-mon[33017]: pgmap v20340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:35 smithi033 ceph-mon[27845]: pgmap v20340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:35 smithi096 ceph-mon[32753]: pgmap v20340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:37 smithi194 ceph-mon[33017]: pgmap v20341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:37 smithi033 ceph-mon[27845]: pgmap v20341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:37 smithi096 ceph-mon[32753]: pgmap v20341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37717.280458 secs 2024-04-26T22:19:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37717.280458 secs 2024-04-26T22:19:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37717.280458 secs 2024-04-26T22:19:40.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37720.257582 secs 2024-04-26T22:19:40.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:40 smithi194 ceph-mon[33017]: pgmap v20342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:40.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37720.257582 secs 2024-04-26T22:19:40.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:40 smithi033 ceph-mon[27845]: pgmap v20342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37720.257582 secs 2024-04-26T22:19:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:40 smithi096 ceph-mon[32753]: pgmap v20342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:42 smithi194 ceph-mon[33017]: pgmap v20343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:42 smithi033 ceph-mon[27845]: pgmap v20343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:42 smithi096 ceph-mon[32753]: pgmap v20343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:44 smithi194 ceph-mon[33017]: pgmap v20344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37722.280517 secs 2024-04-26T22:19:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:44 smithi033 ceph-mon[27845]: pgmap v20344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37722.280517 secs 2024-04-26T22:19:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:44 smithi096 ceph-mon[32753]: pgmap v20344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37722.280517 secs 2024-04-26T22:19:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37725.257693 secs 2024-04-26T22:19:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37725.257693 secs 2024-04-26T22:19:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37725.257693 secs 2024-04-26T22:19:45.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:45.625+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:45.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:45.625+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:19:45.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:19:45.625+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:19:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:46 smithi194 ceph-mon[33017]: pgmap v20345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:46 smithi033 ceph-mon[27845]: pgmap v20345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:46 smithi096 ceph-mon[32753]: pgmap v20345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:48 smithi194 ceph-mon[33017]: pgmap v20346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:48 smithi033 ceph-mon[27845]: pgmap v20346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:48 smithi096 ceph-mon[32753]: pgmap v20346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37727.280579 secs 2024-04-26T22:19:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37727.280579 secs 2024-04-26T22:19:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37727.280579 secs 2024-04-26T22:19:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37730.257823 secs 2024-04-26T22:19:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:50 smithi194 ceph-mon[33017]: pgmap v20347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37730.257823 secs 2024-04-26T22:19:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:50 smithi033 ceph-mon[27845]: pgmap v20347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37730.257823 secs 2024-04-26T22:19:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:50 smithi096 ceph-mon[32753]: pgmap v20347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:52 smithi194 ceph-mon[33017]: pgmap v20348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:52 smithi033 ceph-mon[27845]: pgmap v20348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:52 smithi096 ceph-mon[32753]: pgmap v20348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:54 smithi194 ceph-mon[33017]: pgmap v20349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37732.280663 secs 2024-04-26T22:19:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:54 smithi033 ceph-mon[27845]: pgmap v20349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37732.280663 secs 2024-04-26T22:19:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:54 smithi096 ceph-mon[32753]: pgmap v20349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37732.280663 secs 2024-04-26T22:19:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37735.257957 secs 2024-04-26T22:19:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37735.257957 secs 2024-04-26T22:19:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37735.257957 secs 2024-04-26T22:19:56.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:56 smithi194 ceph-mon[33017]: pgmap v20350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:56 smithi033 ceph-mon[27845]: pgmap v20350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:56 smithi096 ceph-mon[32753]: pgmap v20350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:19:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:58 smithi194 ceph-mon[33017]: pgmap v20351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:58.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:58 smithi033 ceph-mon[27845]: pgmap v20351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:58 smithi096 ceph-mon[32753]: pgmap v20351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:19:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:19:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37737.280743 secs 2024-04-26T22:19:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:19:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37737.280743 secs 2024-04-26T22:19:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:19:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37737.280743 secs 2024-04-26T22:20:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37740.258086 secs 2024-04-26T22:20:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:00 smithi194 ceph-mon[33017]: pgmap v20352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:20:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37740.258086 secs 2024-04-26T22:20:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:00 smithi033 ceph-mon[27845]: pgmap v20352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:20:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37740.258086 secs 2024-04-26T22:20:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:00 smithi096 ceph-mon[32753]: pgmap v20352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:20:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:00.640+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:00.640+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:20:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:00.640+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:02 smithi194 ceph-mon[33017]: pgmap v20353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:02 smithi033 ceph-mon[27845]: pgmap v20353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:02 smithi096 ceph-mon[32753]: pgmap v20353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:04 smithi194 ceph-mon[33017]: pgmap v20354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:04.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37742.280827 secs 2024-04-26T22:20:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:04 smithi033 ceph-mon[27845]: pgmap v20354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37742.280827 secs 2024-04-26T22:20:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:04 smithi096 ceph-mon[32753]: pgmap v20354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37742.280827 secs 2024-04-26T22:20:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37745.258201 secs 2024-04-26T22:20:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37745.258201 secs 2024-04-26T22:20:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37745.258201 secs 2024-04-26T22:20:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:06 smithi033 ceph-mon[27845]: pgmap v20355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:06.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:06 smithi096 ceph-mon[32753]: pgmap v20355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:06 smithi194 ceph-mon[33017]: pgmap v20355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:07 smithi033 ceph-mon[27845]: pgmap v20356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:07 smithi096 ceph-mon[32753]: pgmap v20356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:07 smithi194 ceph-mon[33017]: pgmap v20356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37747.280912 secs 2024-04-26T22:20:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37747.280912 secs 2024-04-26T22:20:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37747.280912 secs 2024-04-26T22:20:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:09 smithi194 ceph-mon[33017]: pgmap v20357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:09 smithi033 ceph-mon[27845]: pgmap v20357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:09 smithi096 ceph-mon[32753]: pgmap v20357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37750.258298 secs 2024-04-26T22:20:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37750.258298 secs 2024-04-26T22:20:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37750.258298 secs 2024-04-26T22:20:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:11 smithi194 ceph-mon[33017]: pgmap v20358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:11 smithi033 ceph-mon[27845]: pgmap v20358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:11.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:11 smithi096 ceph-mon[32753]: pgmap v20358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:13 smithi194 ceph-mon[33017]: pgmap v20359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:14.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37752.280997 secs 2024-04-26T22:20:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:13 smithi033 ceph-mon[27845]: pgmap v20359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37752.280997 secs 2024-04-26T22:20:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:13 smithi096 ceph-mon[32753]: pgmap v20359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37752.280997 secs 2024-04-26T22:20:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37755.258417 secs 2024-04-26T22:20:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37755.258417 secs 2024-04-26T22:20:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37755.258417 secs 2024-04-26T22:20:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:15.656+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:15.656+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:15.656+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:20:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:15 smithi194 ceph-mon[33017]: pgmap v20360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:15 smithi033 ceph-mon[27845]: pgmap v20360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:15 smithi096 ceph-mon[32753]: pgmap v20360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:17 smithi194 ceph-mon[33017]: pgmap v20361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:17 smithi033 ceph-mon[27845]: pgmap v20361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:17 smithi096 ceph-mon[32753]: pgmap v20361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37757.281055 secs 2024-04-26T22:20:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37757.281055 secs 2024-04-26T22:20:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37757.281055 secs 2024-04-26T22:20:20.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37760.258530 secs 2024-04-26T22:20:20.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:20 smithi033 ceph-mon[27845]: pgmap v20362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:20.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:20 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:20:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37760.258530 secs 2024-04-26T22:20:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:20 smithi194 ceph-mon[33017]: pgmap v20362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:20 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:20:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37760.258530 secs 2024-04-26T22:20:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:20 smithi096 ceph-mon[32753]: pgmap v20362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:20 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:20:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:20:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:20:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:20:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:20:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:20:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:20:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:20:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:20:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:20:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:22 smithi194 ceph-mon[33017]: pgmap v20363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:20:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:22 smithi194 ceph-mon[33017]: pgmap v20364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:22 smithi033 ceph-mon[27845]: pgmap v20363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:20:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:22 smithi033 ceph-mon[27845]: pgmap v20364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:22 smithi096 ceph-mon[32753]: pgmap v20363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:20:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:22 smithi096 ceph-mon[32753]: pgmap v20364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:24 smithi194 ceph-mon[33017]: pgmap v20365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37762.281116 secs 2024-04-26T22:20:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:24 smithi033 ceph-mon[27845]: pgmap v20365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37762.281116 secs 2024-04-26T22:20:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:24 smithi096 ceph-mon[32753]: pgmap v20365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37762.281116 secs 2024-04-26T22:20:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37765.258625 secs 2024-04-26T22:20:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37765.258625 secs 2024-04-26T22:20:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37765.258625 secs 2024-04-26T22:20:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:26 smithi194 ceph-mon[33017]: pgmap v20366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:26 smithi033 ceph-mon[27845]: pgmap v20366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:26 smithi096 ceph-mon[32753]: pgmap v20366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:28.228+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:20:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:28.228+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:20:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:28.229+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 295 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:20:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:28.229+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 295 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:20:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:28 smithi194 ceph-mon[33017]: pgmap v20367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:28 smithi033 ceph-mon[27845]: pgmap v20367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:28 smithi096 ceph-mon[32753]: pgmap v20367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37767.281179 secs 2024-04-26T22:20:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37767.281179 secs 2024-04-26T22:20:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37767.281179 secs 2024-04-26T22:20:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:30 smithi194 ceph-mon[33017]: pgmap v20368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37770.258722 secs 2024-04-26T22:20:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:30 smithi033 ceph-mon[27845]: pgmap v20368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37770.258722 secs 2024-04-26T22:20:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:30 smithi096 ceph-mon[32753]: pgmap v20368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:20:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37770.258722 secs 2024-04-26T22:20:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:30.663+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:30.664+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:30.664+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:20:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:32 smithi194 ceph-mon[33017]: pgmap v20369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:20:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:32 smithi033 ceph-mon[27845]: pgmap v20369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:20:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:32 smithi096 ceph-mon[32753]: pgmap v20369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:20:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:34 smithi194 ceph-mon[33017]: pgmap v20370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37772.281239 secs 2024-04-26T22:20:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:34 smithi033 ceph-mon[27845]: pgmap v20370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37772.281239 secs 2024-04-26T22:20:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:34 smithi096 ceph-mon[32753]: pgmap v20370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37772.281239 secs 2024-04-26T22:20:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37775.258843 secs 2024-04-26T22:20:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37775.258843 secs 2024-04-26T22:20:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37775.258843 secs 2024-04-26T22:20:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:36 smithi194 ceph-mon[33017]: pgmap v20371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:36 smithi033 ceph-mon[27845]: pgmap v20371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:36 smithi096 ceph-mon[32753]: pgmap v20371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:38 smithi194 ceph-mon[33017]: pgmap v20372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:38 smithi033 ceph-mon[27845]: pgmap v20372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:38 smithi096 ceph-mon[32753]: pgmap v20372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37777.281300 secs 2024-04-26T22:20:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37777.281300 secs 2024-04-26T22:20:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37777.281300 secs 2024-04-26T22:20:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:40 smithi194 ceph-mon[33017]: pgmap v20373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:40.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37780.258964 secs 2024-04-26T22:20:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:40 smithi033 ceph-mon[27845]: pgmap v20373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37780.258964 secs 2024-04-26T22:20:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:40 smithi096 ceph-mon[32753]: pgmap v20373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37780.258964 secs 2024-04-26T22:20:42.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:42 smithi194 ceph-mon[33017]: pgmap v20374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:42 smithi033 ceph-mon[27845]: pgmap v20374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:42 smithi096 ceph-mon[32753]: pgmap v20374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:44 smithi033 ceph-mon[27845]: pgmap v20375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37782.281384 secs 2024-04-26T22:20:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:44 smithi096 ceph-mon[32753]: pgmap v20375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37782.281384 secs 2024-04-26T22:20:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:44 smithi194 ceph-mon[33017]: pgmap v20375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37782.281384 secs 2024-04-26T22:20:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37785.259138 secs 2024-04-26T22:20:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37785.259138 secs 2024-04-26T22:20:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37785.259138 secs 2024-04-26T22:20:45.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:45.667+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:45.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:45.667+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:20:45.667 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:20:45.667+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:20:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:46 smithi033 ceph-mon[27845]: pgmap v20376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:46 smithi096 ceph-mon[32753]: pgmap v20376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:46 smithi194 ceph-mon[33017]: pgmap v20376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:47 smithi033 ceph-mon[27845]: pgmap v20377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:47 smithi096 ceph-mon[32753]: pgmap v20377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:47 smithi194 ceph-mon[33017]: pgmap v20377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37787.281467 secs 2024-04-26T22:20:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37787.281467 secs 2024-04-26T22:20:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37787.281467 secs 2024-04-26T22:20:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:49 smithi194 ceph-mon[33017]: pgmap v20378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:49 smithi033 ceph-mon[27845]: pgmap v20378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:49 smithi096 ceph-mon[32753]: pgmap v20378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:50.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37790.259258 secs 2024-04-26T22:20:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37790.259258 secs 2024-04-26T22:20:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37790.259258 secs 2024-04-26T22:20:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:51 smithi033 ceph-mon[27845]: pgmap v20379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:51 smithi096 ceph-mon[32753]: pgmap v20379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:52.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:51 smithi194 ceph-mon[33017]: pgmap v20379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:53 smithi033 ceph-mon[27845]: pgmap v20380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37792.281552 secs 2024-04-26T22:20:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:53 smithi096 ceph-mon[32753]: pgmap v20380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37792.281552 secs 2024-04-26T22:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:53 smithi194 ceph-mon[33017]: pgmap v20380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37792.281552 secs 2024-04-26T22:20:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37795.259398 secs 2024-04-26T22:20:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:55 smithi033 ceph-mon[27845]: pgmap v20381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37795.259398 secs 2024-04-26T22:20:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:55 smithi096 ceph-mon[32753]: pgmap v20381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37795.259398 secs 2024-04-26T22:20:56.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:55 smithi194 ceph-mon[33017]: pgmap v20381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:20:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:57 smithi033 ceph-mon[27845]: pgmap v20382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:57.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:57 smithi096 ceph-mon[32753]: pgmap v20382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:58.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:57 smithi194 ceph-mon[33017]: pgmap v20382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37797.281636 secs 2024-04-26T22:20:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37797.281636 secs 2024-04-26T22:20:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37797.281636 secs 2024-04-26T22:20:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:20:59 smithi033 ceph-mon[27845]: pgmap v20383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:20:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:20:59 smithi096 ceph-mon[32753]: pgmap v20383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:20:59 smithi194 ceph-mon[33017]: pgmap v20383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:00.682+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:00.682+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:21:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:00.682+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37800.259516 secs 2024-04-26T22:21:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37800.259516 secs 2024-04-26T22:21:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37800.259516 secs 2024-04-26T22:21:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:01 smithi033 ceph-mon[27845]: pgmap v20384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:01 smithi096 ceph-mon[32753]: pgmap v20384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:01 smithi194 ceph-mon[33017]: pgmap v20384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:03 smithi033 ceph-mon[27845]: pgmap v20385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:03.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37802.281719 secs 2024-04-26T22:21:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:03 smithi096 ceph-mon[32753]: pgmap v20385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37802.281719 secs 2024-04-26T22:21:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:03 smithi194 ceph-mon[33017]: pgmap v20385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:04.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37802.281719 secs 2024-04-26T22:21:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37805.259635 secs 2024-04-26T22:21:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37805.259635 secs 2024-04-26T22:21:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37805.259635 secs 2024-04-26T22:21:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:05 smithi033 ceph-mon[27845]: pgmap v20386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:05.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:05 smithi096 ceph-mon[32753]: pgmap v20386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:05 smithi194 ceph-mon[33017]: pgmap v20386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:07.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:07 smithi096 ceph-mon[32753]: pgmap v20387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:08.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:07 smithi194 ceph-mon[33017]: pgmap v20387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:07 smithi033 ceph-mon[27845]: pgmap v20387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37807.281805 secs 2024-04-26T22:21:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37807.281805 secs 2024-04-26T22:21:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37807.281805 secs 2024-04-26T22:21:10.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:09 smithi194 ceph-mon[33017]: pgmap v20388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:10.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37810.259741 secs 2024-04-26T22:21:10.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:09 smithi033 ceph-mon[27845]: pgmap v20388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:10.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37810.259741 secs 2024-04-26T22:21:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:09 smithi096 ceph-mon[32753]: pgmap v20388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37810.259741 secs 2024-04-26T22:21:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:11 smithi194 ceph-mon[33017]: pgmap v20389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:11 smithi033 ceph-mon[27845]: pgmap v20389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:11 smithi096 ceph-mon[32753]: pgmap v20389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:13 smithi194 ceph-mon[33017]: pgmap v20390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37812.281891 secs 2024-04-26T22:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:13 smithi033 ceph-mon[27845]: pgmap v20390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37812.281891 secs 2024-04-26T22:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:13 smithi096 ceph-mon[32753]: pgmap v20390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37812.281891 secs 2024-04-26T22:21:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37815.259886 secs 2024-04-26T22:21:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37815.259886 secs 2024-04-26T22:21:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37815.259886 secs 2024-04-26T22:21:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:15.696+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:15.696+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:15.696+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:21:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:15 smithi194 ceph-mon[33017]: pgmap v20391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:16.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:15 smithi033 ceph-mon[27845]: pgmap v20391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:15 smithi096 ceph-mon[32753]: pgmap v20391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:17 smithi194 ceph-mon[33017]: pgmap v20392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:18.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:17 smithi033 ceph-mon[27845]: pgmap v20392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:17 smithi096 ceph-mon[32753]: pgmap v20392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37817.281976 secs 2024-04-26T22:21:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37817.281976 secs 2024-04-26T22:21:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37817.281976 secs 2024-04-26T22:21:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:19 smithi194 ceph-mon[33017]: pgmap v20393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37820.260002 secs 2024-04-26T22:21:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:19 smithi033 ceph-mon[27845]: pgmap v20393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37820.260002 secs 2024-04-26T22:21:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:19 smithi096 ceph-mon[32753]: pgmap v20393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37820.260002 secs 2024-04-26T22:21:22.097 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:21 smithi033 ceph-mon[27845]: pgmap v20394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:22.097 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:21:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:21 smithi194 ceph-mon[33017]: pgmap v20394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:21:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:21 smithi096 ceph-mon[32753]: pgmap v20394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:21:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:21:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:21:23.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:21:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:21:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:21:23.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:21:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:21:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:21:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:21:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:23 smithi194 ceph-mon[33017]: pgmap v20395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:23 smithi194 ceph-mon[33017]: pgmap v20396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37822.282063 secs 2024-04-26T22:21:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:23 smithi033 ceph-mon[27845]: pgmap v20395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:23 smithi033 ceph-mon[27845]: pgmap v20396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:24.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37822.282063 secs 2024-04-26T22:21:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:23 smithi096 ceph-mon[32753]: pgmap v20395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:23 smithi096 ceph-mon[32753]: pgmap v20396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37822.282063 secs 2024-04-26T22:21:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37825.260142 secs 2024-04-26T22:21:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37825.260142 secs 2024-04-26T22:21:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37825.260142 secs 2024-04-26T22:21:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:25 smithi194 ceph-mon[33017]: pgmap v20397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:21:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:25 smithi033 ceph-mon[27845]: pgmap v20397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:21:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:25 smithi096 ceph-mon[32753]: pgmap v20397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:21:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:27 smithi194 ceph-mon[33017]: pgmap v20398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:27 smithi033 ceph-mon[27845]: pgmap v20398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:27 smithi096 ceph-mon[32753]: pgmap v20398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37827.282140 secs 2024-04-26T22:21:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37827.282140 secs 2024-04-26T22:21:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37827.282140 secs 2024-04-26T22:21:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:29 smithi194 ceph-mon[33017]: pgmap v20399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:30.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37830.260258 secs 2024-04-26T22:21:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:29 smithi033 ceph-mon[27845]: pgmap v20399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37830.260258 secs 2024-04-26T22:21:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:29 smithi096 ceph-mon[32753]: pgmap v20399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37830.260258 secs 2024-04-26T22:21:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:30.708+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:30.708+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:21:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:30.708+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:31 smithi194 ceph-mon[33017]: pgmap v20400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:31 smithi033 ceph-mon[27845]: pgmap v20400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:31 smithi096 ceph-mon[32753]: pgmap v20400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:21:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:33 smithi194 ceph-mon[33017]: pgmap v20401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:21:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37832.282227 secs 2024-04-26T22:21:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:33 smithi033 ceph-mon[27845]: pgmap v20401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:21:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37832.282227 secs 2024-04-26T22:21:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:33 smithi096 ceph-mon[32753]: pgmap v20401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:21:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37832.282227 secs 2024-04-26T22:21:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37835.260375 secs 2024-04-26T22:21:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37835.260375 secs 2024-04-26T22:21:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37835.260375 secs 2024-04-26T22:21:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:35 smithi194 ceph-mon[33017]: pgmap v20402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:35 smithi033 ceph-mon[27845]: pgmap v20402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:35 smithi096 ceph-mon[32753]: pgmap v20402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:37 smithi194 ceph-mon[33017]: pgmap v20403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:37 smithi033 ceph-mon[27845]: pgmap v20403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:37 smithi096 ceph-mon[32753]: pgmap v20403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37837.282287 secs 2024-04-26T22:21:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37837.282287 secs 2024-04-26T22:21:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37837.282287 secs 2024-04-26T22:21:40.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:39 smithi033 ceph-mon[27845]: pgmap v20404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:40.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37840.260441 secs 2024-04-26T22:21:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:39 smithi194 ceph-mon[33017]: pgmap v20404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37840.260441 secs 2024-04-26T22:21:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:39 smithi096 ceph-mon[32753]: pgmap v20404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37840.260441 secs 2024-04-26T22:21:42.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:41 smithi194 ceph-mon[33017]: pgmap v20405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:41 smithi033 ceph-mon[27845]: pgmap v20405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:41 smithi096 ceph-mon[32753]: pgmap v20405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:43 smithi194 ceph-mon[33017]: pgmap v20406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37842.282371 secs 2024-04-26T22:21:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:43 smithi033 ceph-mon[27845]: pgmap v20406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37842.282371 secs 2024-04-26T22:21:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:43 smithi096 ceph-mon[32753]: pgmap v20406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37842.282371 secs 2024-04-26T22:21:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37845.260503 secs 2024-04-26T22:21:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37845.260503 secs 2024-04-26T22:21:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37845.260503 secs 2024-04-26T22:21:45.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:45.716+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:45.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:45.716+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:21:45.717 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:21:45.716+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:21:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:45 smithi194 ceph-mon[33017]: pgmap v20407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:45 smithi033 ceph-mon[27845]: pgmap v20407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:45 smithi096 ceph-mon[32753]: pgmap v20407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:48.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:47 smithi033 ceph-mon[27845]: pgmap v20408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:47 smithi096 ceph-mon[32753]: pgmap v20408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:47 smithi194 ceph-mon[33017]: pgmap v20408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37847.282456 secs 2024-04-26T22:21:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37847.282456 secs 2024-04-26T22:21:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37847.282456 secs 2024-04-26T22:21:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:49 smithi033 ceph-mon[27845]: pgmap v20409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37850.260618 secs 2024-04-26T22:21:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:49 smithi096 ceph-mon[32753]: pgmap v20409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37850.260618 secs 2024-04-26T22:21:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:49 smithi194 ceph-mon[33017]: pgmap v20409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37850.260618 secs 2024-04-26T22:21:52.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:51 smithi033 ceph-mon[27845]: pgmap v20410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:52.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:51 smithi096 ceph-mon[32753]: pgmap v20410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:51 smithi194 ceph-mon[33017]: pgmap v20410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:53 smithi033 ceph-mon[27845]: pgmap v20411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:54.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37852.282516 secs 2024-04-26T22:21:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:53 smithi096 ceph-mon[32753]: pgmap v20411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37852.282516 secs 2024-04-26T22:21:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:53 smithi194 ceph-mon[33017]: pgmap v20411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37852.282516 secs 2024-04-26T22:21:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37855.260714 secs 2024-04-26T22:21:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37855.260714 secs 2024-04-26T22:21:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37855.260714 secs 2024-04-26T22:21:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:55 smithi033 ceph-mon[27845]: pgmap v20412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:55 smithi096 ceph-mon[32753]: pgmap v20412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:55 smithi194 ceph-mon[33017]: pgmap v20412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:21:58.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:57 smithi033 ceph-mon[27845]: pgmap v20413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:58.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:57 smithi096 ceph-mon[32753]: pgmap v20413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:57 smithi194 ceph-mon[33017]: pgmap v20413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:21:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37857.282602 secs 2024-04-26T22:21:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37857.282602 secs 2024-04-26T22:21:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37857.282602 secs 2024-04-26T22:22:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:59 smithi033 ceph-mon[27845]: pgmap v20414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:21:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37860.260829 secs 2024-04-26T22:22:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:59 smithi096 ceph-mon[32753]: pgmap v20414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:21:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37860.260829 secs 2024-04-26T22:22:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:59 smithi194 ceph-mon[33017]: pgmap v20414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:21:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37860.260829 secs 2024-04-26T22:22:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:00.719+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:00.719+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:00.719+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:22:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:01 smithi033 ceph-mon[27845]: pgmap v20415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:02.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:01 smithi096 ceph-mon[32753]: pgmap v20415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:01 smithi194 ceph-mon[33017]: pgmap v20415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:03 smithi033 ceph-mon[27845]: pgmap v20416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:04.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37862.282703 secs 2024-04-26T22:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:03 smithi096 ceph-mon[32753]: pgmap v20416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37862.282703 secs 2024-04-26T22:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:03 smithi194 ceph-mon[33017]: pgmap v20416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37862.282703 secs 2024-04-26T22:22:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37865.260964 secs 2024-04-26T22:22:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37865.260964 secs 2024-04-26T22:22:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37865.260964 secs 2024-04-26T22:22:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:05 smithi194 ceph-mon[33017]: pgmap v20417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:05 smithi033 ceph-mon[27845]: pgmap v20417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:05 smithi096 ceph-mon[32753]: pgmap v20417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:07 smithi194 ceph-mon[33017]: pgmap v20418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:07 smithi033 ceph-mon[27845]: pgmap v20418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:07 smithi096 ceph-mon[32753]: pgmap v20418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:09.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37867.282787 secs 2024-04-26T22:22:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37867.282787 secs 2024-04-26T22:22:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37867.282787 secs 2024-04-26T22:22:10.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:09 smithi194 ceph-mon[33017]: pgmap v20419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:10.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37870.261098 secs 2024-04-26T22:22:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:09 smithi033 ceph-mon[27845]: pgmap v20419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37870.261098 secs 2024-04-26T22:22:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:09 smithi096 ceph-mon[32753]: pgmap v20419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37870.261098 secs 2024-04-26T22:22:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:11 smithi194 ceph-mon[33017]: pgmap v20420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:11 smithi033 ceph-mon[27845]: pgmap v20420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:11 smithi096 ceph-mon[32753]: pgmap v20420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:13 smithi194 ceph-mon[33017]: pgmap v20421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37872.282871 secs 2024-04-26T22:22:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:13 smithi033 ceph-mon[27845]: pgmap v20421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37872.282871 secs 2024-04-26T22:22:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:13 smithi096 ceph-mon[32753]: pgmap v20421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37872.282871 secs 2024-04-26T22:22:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37875.261188 secs 2024-04-26T22:22:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37875.261188 secs 2024-04-26T22:22:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37875.261188 secs 2024-04-26T22:22:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:15.728+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:15.728+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:15.728+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:22:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:15 smithi194 ceph-mon[33017]: pgmap v20422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:15 smithi033 ceph-mon[27845]: pgmap v20422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:15 smithi096 ceph-mon[32753]: pgmap v20422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:17 smithi194 ceph-mon[33017]: pgmap v20423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:18.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:17 smithi033 ceph-mon[27845]: pgmap v20423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:17 smithi096 ceph-mon[32753]: pgmap v20423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37877.282957 secs 2024-04-26T22:22:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37877.282957 secs 2024-04-26T22:22:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37877.282957 secs 2024-04-26T22:22:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:19 smithi194 ceph-mon[33017]: pgmap v20424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37880.261323 secs 2024-04-26T22:22:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:19 smithi033 ceph-mon[27845]: pgmap v20424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37880.261323 secs 2024-04-26T22:22:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:19 smithi096 ceph-mon[32753]: pgmap v20424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37880.261323 secs 2024-04-26T22:22:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:22 smithi194 ceph-mon[33017]: pgmap v20425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:22 smithi033 ceph-mon[27845]: pgmap v20425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:22 smithi096 ceph-mon[32753]: pgmap v20425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:22:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:22:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:22:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:24 smithi194 ceph-mon[33017]: pgmap v20426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37882.282983 secs 2024-04-26T22:22:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:22:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:24 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:22:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:24 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:22:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:24 smithi033 ceph-mon[27845]: pgmap v20426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37882.282983 secs 2024-04-26T22:22:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:22:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:24 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:22:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:24 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:24 smithi096 ceph-mon[32753]: pgmap v20426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37882.282983 secs 2024-04-26T22:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:24 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:22:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:24 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:22:25.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37885.261440 secs 2024-04-26T22:22:25.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:25 smithi033 ceph-mon[27845]: pgmap v20427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:25.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:25 smithi033 ceph-mon[27845]: pgmap v20428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37885.261440 secs 2024-04-26T22:22:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:25 smithi194 ceph-mon[33017]: pgmap v20427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:25 smithi194 ceph-mon[33017]: pgmap v20428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37885.261440 secs 2024-04-26T22:22:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:25 smithi096 ceph-mon[32753]: pgmap v20427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:25 smithi096 ceph-mon[32753]: pgmap v20428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:27 smithi194 ceph-mon[33017]: pgmap v20429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:27 smithi033 ceph-mon[27845]: pgmap v20429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:27 smithi096 ceph-mon[32753]: pgmap v20429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:29 smithi194 ceph-mon[33017]: pgmap v20430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37887.283074 secs 2024-04-26T22:22:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:29 smithi033 ceph-mon[27845]: pgmap v20430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37887.283074 secs 2024-04-26T22:22:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:29 smithi096 ceph-mon[32753]: pgmap v20430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37887.283074 secs 2024-04-26T22:22:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37890.261538 secs 2024-04-26T22:22:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37890.261538 secs 2024-04-26T22:22:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37890.261538 secs 2024-04-26T22:22:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:30.737+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:30.738+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:30.738+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:22:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:31 smithi194 ceph-mon[33017]: pgmap v20431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:31 smithi033 ceph-mon[27845]: pgmap v20431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:31 smithi096 ceph-mon[32753]: pgmap v20431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:33 smithi194 ceph-mon[33017]: pgmap v20432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:33 smithi033 ceph-mon[27845]: pgmap v20432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:33 smithi096 ceph-mon[32753]: pgmap v20432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:22:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37892.283137 secs 2024-04-26T22:22:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37892.283137 secs 2024-04-26T22:22:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37892.283137 secs 2024-04-26T22:22:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37895.261664 secs 2024-04-26T22:22:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:35 smithi194 ceph-mon[33017]: pgmap v20433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:22:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37895.261664 secs 2024-04-26T22:22:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:35 smithi033 ceph-mon[27845]: pgmap v20433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:22:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37895.261664 secs 2024-04-26T22:22:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:35 smithi096 ceph-mon[32753]: pgmap v20433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:22:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:37 smithi194 ceph-mon[33017]: pgmap v20434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:37 smithi033 ceph-mon[27845]: pgmap v20434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:37 smithi096 ceph-mon[32753]: pgmap v20434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:39 smithi194 ceph-mon[33017]: pgmap v20435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37897.283226 secs 2024-04-26T22:22:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:39 smithi033 ceph-mon[27845]: pgmap v20435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37897.283226 secs 2024-04-26T22:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:39 smithi096 ceph-mon[32753]: pgmap v20435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37897.283226 secs 2024-04-26T22:22:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37900.261777 secs 2024-04-26T22:22:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37900.261777 secs 2024-04-26T22:22:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37900.261777 secs 2024-04-26T22:22:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:41 smithi194 ceph-mon[33017]: pgmap v20436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:41 smithi033 ceph-mon[27845]: pgmap v20436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:41 smithi096 ceph-mon[32753]: pgmap v20436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:43 smithi194 ceph-mon[33017]: pgmap v20437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:43 smithi033 ceph-mon[27845]: pgmap v20437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:43 smithi096 ceph-mon[32753]: pgmap v20437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37902.283286 secs 2024-04-26T22:22:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37902.283286 secs 2024-04-26T22:22:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37902.283286 secs 2024-04-26T22:22:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37905.261910 secs 2024-04-26T22:22:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:45 smithi033 ceph-mon[27845]: pgmap v20438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37905.261910 secs 2024-04-26T22:22:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:45 smithi096 ceph-mon[32753]: pgmap v20438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37905.261910 secs 2024-04-26T22:22:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:45 smithi194 ceph-mon[33017]: pgmap v20438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:45.747 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:45.747+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:45.747+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:22:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:22:45.747+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:22:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:47 smithi033 ceph-mon[27845]: pgmap v20439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:47 smithi096 ceph-mon[32753]: pgmap v20439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:47 smithi194 ceph-mon[33017]: pgmap v20439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:49 smithi033 ceph-mon[27845]: pgmap v20440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37907.283363 secs 2024-04-26T22:22:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:49 smithi096 ceph-mon[32753]: pgmap v20440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37907.283363 secs 2024-04-26T22:22:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:49 smithi194 ceph-mon[33017]: pgmap v20440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:22:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37907.283363 secs 2024-04-26T22:22:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37910.262027 secs 2024-04-26T22:22:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37910.262027 secs 2024-04-26T22:22:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37910.262027 secs 2024-04-26T22:22:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:51 smithi033 ceph-mon[27845]: pgmap v20441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:51 smithi096 ceph-mon[32753]: pgmap v20441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:51 smithi194 ceph-mon[33017]: pgmap v20441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:22:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:53 smithi033 ceph-mon[27845]: pgmap v20442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:53 smithi096 ceph-mon[32753]: pgmap v20442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:53 smithi194 ceph-mon[33017]: pgmap v20442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37912.283449 secs 2024-04-26T22:22:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:54 smithi033 ceph-mon[27845]: pgmap v20443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37912.283449 secs 2024-04-26T22:22:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:54 smithi096 ceph-mon[32753]: pgmap v20443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37912.283449 secs 2024-04-26T22:22:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:54 smithi194 ceph-mon[33017]: pgmap v20443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37915.262142 secs 2024-04-26T22:22:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37915.262142 secs 2024-04-26T22:22:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37915.262142 secs 2024-04-26T22:22:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:56 smithi033 ceph-mon[27845]: pgmap v20444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:56 smithi096 ceph-mon[32753]: pgmap v20444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:56 smithi194 ceph-mon[33017]: pgmap v20444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:22:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:58 smithi194 ceph-mon[33017]: pgmap v20445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37917.283534 secs 2024-04-26T22:22:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:58 smithi033 ceph-mon[27845]: pgmap v20445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37917.283534 secs 2024-04-26T22:22:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:58 smithi096 ceph-mon[32753]: pgmap v20445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:22:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37917.283534 secs 2024-04-26T22:23:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:22:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37920.262257 secs 2024-04-26T22:23:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:22:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37920.262257 secs 2024-04-26T22:23:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:22:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37920.262257 secs 2024-04-26T22:23:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:00.762+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:00.762+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:00.762+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:23:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:01 smithi194 ceph-mon[33017]: pgmap v20446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:01 smithi033 ceph-mon[27845]: pgmap v20446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:01 smithi096 ceph-mon[32753]: pgmap v20446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:03 smithi194 ceph-mon[33017]: pgmap v20447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:03 smithi033 ceph-mon[27845]: pgmap v20447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:03 smithi096 ceph-mon[32753]: pgmap v20447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37922.283616 secs 2024-04-26T22:23:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37922.283616 secs 2024-04-26T22:23:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37922.283616 secs 2024-04-26T22:23:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37925.262373 secs 2024-04-26T22:23:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:05 smithi194 ceph-mon[33017]: pgmap v20448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37925.262373 secs 2024-04-26T22:23:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:05 smithi033 ceph-mon[27845]: pgmap v20448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37925.262373 secs 2024-04-26T22:23:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:05 smithi096 ceph-mon[32753]: pgmap v20448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:07 smithi194 ceph-mon[33017]: pgmap v20449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:07 smithi033 ceph-mon[27845]: pgmap v20449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:07 smithi096 ceph-mon[32753]: pgmap v20449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:09 smithi194 ceph-mon[33017]: pgmap v20450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37927.283702 secs 2024-04-26T22:23:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:09 smithi033 ceph-mon[27845]: pgmap v20450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37927.283702 secs 2024-04-26T22:23:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:09 smithi096 ceph-mon[32753]: pgmap v20450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37927.283702 secs 2024-04-26T22:23:10.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37930.262468 secs 2024-04-26T22:23:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37930.262468 secs 2024-04-26T22:23:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37930.262468 secs 2024-04-26T22:23:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:11 smithi194 ceph-mon[33017]: pgmap v20451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:11 smithi033 ceph-mon[27845]: pgmap v20451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:11 smithi096 ceph-mon[32753]: pgmap v20451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:13 smithi194 ceph-mon[33017]: pgmap v20452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:13 smithi033 ceph-mon[27845]: pgmap v20452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:13 smithi096 ceph-mon[32753]: pgmap v20452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37932.283786 secs 2024-04-26T22:23:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37932.283786 secs 2024-04-26T22:23:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37932.283786 secs 2024-04-26T22:23:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37935.262592 secs 2024-04-26T22:23:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:15 smithi194 ceph-mon[33017]: pgmap v20453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37935.262592 secs 2024-04-26T22:23:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:15 smithi033 ceph-mon[27845]: pgmap v20453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37935.262592 secs 2024-04-26T22:23:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:15 smithi096 ceph-mon[32753]: pgmap v20453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:15.778+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:15.778+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:15.778+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:23:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:17 smithi194 ceph-mon[33017]: pgmap v20454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:17.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:17 smithi033 ceph-mon[27845]: pgmap v20454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:17 smithi096 ceph-mon[32753]: pgmap v20454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:19 smithi194 ceph-mon[33017]: pgmap v20455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37937.283871 secs 2024-04-26T22:23:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:19 smithi033 ceph-mon[27845]: pgmap v20455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37937.283871 secs 2024-04-26T22:23:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:19 smithi096 ceph-mon[32753]: pgmap v20455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37937.283871 secs 2024-04-26T22:23:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37940.262684 secs 2024-04-26T22:23:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37940.262684 secs 2024-04-26T22:23:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37940.262684 secs 2024-04-26T22:23:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:21 smithi194 ceph-mon[33017]: pgmap v20456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:23:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:21 smithi033 ceph-mon[27845]: pgmap v20456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:23:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:21 smithi096 ceph-mon[32753]: pgmap v20456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:23:23.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:23 smithi194 ceph-mon[33017]: pgmap v20457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:23:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:23 smithi033 ceph-mon[27845]: pgmap v20457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:23:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:23 smithi096 ceph-mon[32753]: pgmap v20457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:23:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37942.283931 secs 2024-04-26T22:23:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37942.283931 secs 2024-04-26T22:23:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37942.283931 secs 2024-04-26T22:23:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37945.262851 secs 2024-04-26T22:23:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:25 smithi194 ceph-mon[33017]: pgmap v20458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:23:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:25 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:23:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37945.262851 secs 2024-04-26T22:23:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:25 smithi033 ceph-mon[27845]: pgmap v20458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:23:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:25 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:23:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37945.262851 secs 2024-04-26T22:23:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:25 smithi096 ceph-mon[32753]: pgmap v20458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:23:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:25 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:23:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:23:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:23:26.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:23:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:23:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:23:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:23:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:23:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:23:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:23:27.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:27 smithi033 ceph-mon[27845]: pgmap v20459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T22:23:27.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:27 smithi033 ceph-mon[27845]: pgmap v20460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:23:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:27 smithi096 ceph-mon[32753]: pgmap v20459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T22:23:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:27 smithi096 ceph-mon[32753]: pgmap v20460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:23:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:27 smithi194 ceph-mon[33017]: pgmap v20459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T22:23:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:27 smithi194 ceph-mon[33017]: pgmap v20460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:23:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:28.260+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:23:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:28.260+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:23:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:28.260+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 296 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T22:23:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:28.260+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 296 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T22:23:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:29 smithi033 ceph-mon[27845]: pgmap v20461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:23:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37947.284025 secs 2024-04-26T22:23:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:29 smithi096 ceph-mon[32753]: pgmap v20461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:23:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37947.284025 secs 2024-04-26T22:23:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:29 smithi194 ceph-mon[33017]: pgmap v20461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:23:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37947.284025 secs 2024-04-26T22:23:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37950.262965 secs 2024-04-26T22:23:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37950.262965 secs 2024-04-26T22:23:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37950.262965 secs 2024-04-26T22:23:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:30.793+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:30.794+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:30.794+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:23:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:31 smithi033 ceph-mon[27845]: pgmap v20462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:31 smithi096 ceph-mon[32753]: pgmap v20462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:31 smithi194 ceph-mon[33017]: pgmap v20462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:33 smithi033 ceph-mon[27845]: pgmap v20463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:33 smithi096 ceph-mon[32753]: pgmap v20463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:33 smithi194 ceph-mon[33017]: pgmap v20463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37952.284111 secs 2024-04-26T22:23:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37952.284111 secs 2024-04-26T22:23:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37952.284111 secs 2024-04-26T22:23:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:35 smithi033 ceph-mon[27845]: pgmap v20464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37955.263086 secs 2024-04-26T22:23:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:35 smithi096 ceph-mon[32753]: pgmap v20464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37955.263086 secs 2024-04-26T22:23:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:35 smithi194 ceph-mon[33017]: pgmap v20464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:23:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37955.263086 secs 2024-04-26T22:23:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:36 smithi033 ceph-mon[27845]: pgmap v20465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:23:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:36 smithi096 ceph-mon[32753]: pgmap v20465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:23:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:36 smithi194 ceph-mon[33017]: pgmap v20465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:23:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:38 smithi194 ceph-mon[33017]: pgmap v20466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37957.284199 secs 2024-04-26T22:23:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:38 smithi033 ceph-mon[27845]: pgmap v20466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37957.284199 secs 2024-04-26T22:23:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:38 smithi096 ceph-mon[32753]: pgmap v20466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37957.284199 secs 2024-04-26T22:23:40.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37960.263194 secs 2024-04-26T22:23:40.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37960.263194 secs 2024-04-26T22:23:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37960.263194 secs 2024-04-26T22:23:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:40 smithi194 ceph-mon[33017]: pgmap v20467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:40 smithi033 ceph-mon[27845]: pgmap v20467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:40 smithi096 ceph-mon[32753]: pgmap v20467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:42 smithi194 ceph-mon[33017]: pgmap v20468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:42 smithi033 ceph-mon[27845]: pgmap v20468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:42 smithi096 ceph-mon[32753]: pgmap v20468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37962.284304 secs 2024-04-26T22:23:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37962.284304 secs 2024-04-26T22:23:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37962.284304 secs 2024-04-26T22:23:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:44 smithi194 ceph-mon[33017]: pgmap v20469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:45.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37965.263303 secs 2024-04-26T22:23:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:44 smithi033 ceph-mon[27845]: pgmap v20469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37965.263303 secs 2024-04-26T22:23:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:44 smithi096 ceph-mon[32753]: pgmap v20469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37965.263303 secs 2024-04-26T22:23:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:45.796+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:45.796+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:23:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:23:45.796+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:23:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:46 smithi194 ceph-mon[33017]: pgmap v20470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:46 smithi033 ceph-mon[27845]: pgmap v20470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:46 smithi096 ceph-mon[32753]: pgmap v20470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:48 smithi194 ceph-mon[33017]: pgmap v20471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:49.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37967.284363 secs 2024-04-26T22:23:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:48 smithi033 ceph-mon[27845]: pgmap v20471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37967.284363 secs 2024-04-26T22:23:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:48 smithi096 ceph-mon[32753]: pgmap v20471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37967.284363 secs 2024-04-26T22:23:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37970.263416 secs 2024-04-26T22:23:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37970.263416 secs 2024-04-26T22:23:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37970.263416 secs 2024-04-26T22:23:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:50 smithi194 ceph-mon[33017]: pgmap v20472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:50 smithi033 ceph-mon[27845]: pgmap v20472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:51.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:50 smithi096 ceph-mon[32753]: pgmap v20472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:52 smithi194 ceph-mon[33017]: pgmap v20473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:52 smithi033 ceph-mon[27845]: pgmap v20473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:53.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:52 smithi096 ceph-mon[32753]: pgmap v20473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37972.284445 secs 2024-04-26T22:23:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37972.284445 secs 2024-04-26T22:23:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37972.284445 secs 2024-04-26T22:23:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:54 smithi194 ceph-mon[33017]: pgmap v20474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37975.263521 secs 2024-04-26T22:23:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:54 smithi033 ceph-mon[27845]: pgmap v20474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:55.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37975.263521 secs 2024-04-26T22:23:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:54 smithi096 ceph-mon[32753]: pgmap v20474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37975.263521 secs 2024-04-26T22:23:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:56 smithi194 ceph-mon[33017]: pgmap v20475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:56 smithi033 ceph-mon[27845]: pgmap v20475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:56 smithi096 ceph-mon[32753]: pgmap v20475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:23:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:58 smithi194 ceph-mon[33017]: pgmap v20476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37977.284527 secs 2024-04-26T22:23:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:58 smithi033 ceph-mon[27845]: pgmap v20476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37977.284527 secs 2024-04-26T22:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:58 smithi096 ceph-mon[32753]: pgmap v20476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:23:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37977.284527 secs 2024-04-26T22:24:00.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:23:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37980.263667 secs 2024-04-26T22:24:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:23:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37980.263667 secs 2024-04-26T22:24:00.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:23:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37980.263667 secs 2024-04-26T22:24:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:00.810+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:00.810+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:00.810+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:24:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:00 smithi194 ceph-mon[33017]: pgmap v20477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:00 smithi033 ceph-mon[27845]: pgmap v20477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:00 smithi096 ceph-mon[32753]: pgmap v20477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:02 smithi194 ceph-mon[33017]: pgmap v20478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:02 smithi033 ceph-mon[27845]: pgmap v20478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:02 smithi096 ceph-mon[32753]: pgmap v20478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:03.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37982.284613 secs 2024-04-26T22:24:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37982.284613 secs 2024-04-26T22:24:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37982.284613 secs 2024-04-26T22:24:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:04 smithi194 ceph-mon[33017]: pgmap v20479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:04 smithi033 ceph-mon[27845]: pgmap v20479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:04 smithi096 ceph-mon[32753]: pgmap v20479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37985.263802 secs 2024-04-26T22:24:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37985.263802 secs 2024-04-26T22:24:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37985.263802 secs 2024-04-26T22:24:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:06 smithi194 ceph-mon[33017]: pgmap v20480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:06 smithi033 ceph-mon[27845]: pgmap v20480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:06.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:06 smithi096 ceph-mon[32753]: pgmap v20480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:08 smithi194 ceph-mon[33017]: pgmap v20481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37987.284697 secs 2024-04-26T22:24:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:08 smithi033 ceph-mon[27845]: pgmap v20481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37987.284697 secs 2024-04-26T22:24:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:08 smithi096 ceph-mon[32753]: pgmap v20481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37987.284697 secs 2024-04-26T22:24:10.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37990.263916 secs 2024-04-26T22:24:10.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37990.263916 secs 2024-04-26T22:24:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37990.263916 secs 2024-04-26T22:24:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:10 smithi194 ceph-mon[33017]: pgmap v20482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:10 smithi033 ceph-mon[27845]: pgmap v20482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:10 smithi096 ceph-mon[32753]: pgmap v20482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:12 smithi194 ceph-mon[33017]: pgmap v20483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:13.104 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:12 smithi033 ceph-mon[27845]: pgmap v20483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:12 smithi096 ceph-mon[32753]: pgmap v20483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37992.284782 secs 2024-04-26T22:24:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37992.284782 secs 2024-04-26T22:24:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37992.284782 secs 2024-04-26T22:24:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:14 smithi194 ceph-mon[33017]: pgmap v20484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 37995.264051 secs 2024-04-26T22:24:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:14 smithi033 ceph-mon[27845]: pgmap v20484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 37995.264051 secs 2024-04-26T22:24:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:14 smithi096 ceph-mon[32753]: pgmap v20484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 37995.264051 secs 2024-04-26T22:24:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:15.825+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:15.825+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:24:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:15.825+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:16 smithi194 ceph-mon[33017]: pgmap v20485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:16 smithi033 ceph-mon[27845]: pgmap v20485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:16 smithi096 ceph-mon[32753]: pgmap v20485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:18 smithi194 ceph-mon[33017]: pgmap v20486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 37997.284869 secs 2024-04-26T22:24:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:18 smithi033 ceph-mon[27845]: pgmap v20486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 37997.284869 secs 2024-04-26T22:24:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:18 smithi096 ceph-mon[32753]: pgmap v20486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 37997.284869 secs 2024-04-26T22:24:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38000.264205 secs 2024-04-26T22:24:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38000.264205 secs 2024-04-26T22:24:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38000.264205 secs 2024-04-26T22:24:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:20 smithi194 ceph-mon[33017]: pgmap v20487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:20 smithi033 ceph-mon[27845]: pgmap v20487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:20 smithi096 ceph-mon[32753]: pgmap v20487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:22 smithi194 ceph-mon[33017]: pgmap v20488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:22 smithi033 ceph-mon[27845]: pgmap v20488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:22 smithi096 ceph-mon[32753]: pgmap v20488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38002.284956 secs 2024-04-26T22:24:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38002.284956 secs 2024-04-26T22:24:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38002.284956 secs 2024-04-26T22:24:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:24 smithi194 ceph-mon[33017]: pgmap v20489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38005.264325 secs 2024-04-26T22:24:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:24 smithi033 ceph-mon[27845]: pgmap v20489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38005.264325 secs 2024-04-26T22:24:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:24 smithi096 ceph-mon[32753]: pgmap v20489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38005.264325 secs 2024-04-26T22:24:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:26 smithi194 ceph-mon[33017]: pgmap v20490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:24:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:26 smithi033 ceph-mon[27845]: pgmap v20490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:24:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:26 smithi096 ceph-mon[32753]: pgmap v20490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:24:28.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:28.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:28 smithi033 ceph-mon[27845]: pgmap v20491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38007.285043 secs 2024-04-26T22:24:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:24:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:28 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:24:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:28 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:28 smithi096 ceph-mon[32753]: pgmap v20491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38007.285043 secs 2024-04-26T22:24:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:24:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:28 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:24:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:28 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:28 smithi194 ceph-mon[33017]: pgmap v20491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38007.285043 secs 2024-04-26T22:24:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:24:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:28 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:24:29.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:28 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:24:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:29 smithi033 ceph-mon[27845]: pgmap v20492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:24:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:29 smithi033 ceph-mon[27845]: pgmap v20493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:24:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38010.264440 secs 2024-04-26T22:24:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:29 smithi096 ceph-mon[32753]: pgmap v20492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:24:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:29 smithi096 ceph-mon[32753]: pgmap v20493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:24:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38010.264440 secs 2024-04-26T22:24:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:29 smithi194 ceph-mon[33017]: pgmap v20492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:24:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:29 smithi194 ceph-mon[33017]: pgmap v20493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:24:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38010.264440 secs 2024-04-26T22:24:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:30.841+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:30.841+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:24:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:30.841+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:31 smithi033 ceph-mon[27845]: pgmap v20494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:31 smithi096 ceph-mon[32753]: pgmap v20494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:31 smithi194 ceph-mon[33017]: pgmap v20494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:33 smithi033 ceph-mon[27845]: pgmap v20495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:34.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38012.285134 secs 2024-04-26T22:24:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:33 smithi096 ceph-mon[32753]: pgmap v20495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38012.285134 secs 2024-04-26T22:24:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:33 smithi194 ceph-mon[33017]: pgmap v20495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38012.285134 secs 2024-04-26T22:24:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38015.264556 secs 2024-04-26T22:24:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38015.264556 secs 2024-04-26T22:24:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38015.264556 secs 2024-04-26T22:24:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:35 smithi033 ceph-mon[27845]: pgmap v20496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:24:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:35 smithi096 ceph-mon[32753]: pgmap v20496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:24:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:35 smithi194 ceph-mon[33017]: pgmap v20496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:24:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:37 smithi033 ceph-mon[27845]: pgmap v20497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:37 smithi096 ceph-mon[32753]: pgmap v20497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:37 smithi194 ceph-mon[33017]: pgmap v20497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:24:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38017.285225 secs 2024-04-26T22:24:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38017.285225 secs 2024-04-26T22:24:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38017.285225 secs 2024-04-26T22:24:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:39 smithi033 ceph-mon[27845]: pgmap v20498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:24:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38020.264669 secs 2024-04-26T22:24:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:39 smithi096 ceph-mon[32753]: pgmap v20498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:24:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38020.264669 secs 2024-04-26T22:24:40.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:39 smithi194 ceph-mon[33017]: pgmap v20498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:24:40.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38020.264669 secs 2024-04-26T22:24:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:41 smithi033 ceph-mon[27845]: pgmap v20499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:41 smithi096 ceph-mon[32753]: pgmap v20499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:41 smithi194 ceph-mon[33017]: pgmap v20499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:43 smithi033 ceph-mon[27845]: pgmap v20500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38022.285311 secs 2024-04-26T22:24:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:43 smithi096 ceph-mon[32753]: pgmap v20500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38022.285311 secs 2024-04-26T22:24:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:43 smithi194 ceph-mon[33017]: pgmap v20500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38022.285311 secs 2024-04-26T22:24:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38025.264824 secs 2024-04-26T22:24:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38025.264824 secs 2024-04-26T22:24:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38025.264824 secs 2024-04-26T22:24:45.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:45.856+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:45.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:45.856+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:24:45.856 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:45.856+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:24:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:45 smithi194 ceph-mon[33017]: pgmap v20501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:45 smithi033 ceph-mon[27845]: pgmap v20501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:45 smithi096 ceph-mon[32753]: pgmap v20501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:48.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:47 smithi194 ceph-mon[33017]: pgmap v20502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:47 smithi033 ceph-mon[27845]: pgmap v20502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:47 smithi096 ceph-mon[32753]: pgmap v20502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38027.285394 secs 2024-04-26T22:24:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38027.285394 secs 2024-04-26T22:24:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38027.285394 secs 2024-04-26T22:24:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:49 smithi194 ceph-mon[33017]: pgmap v20503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38030.264940 secs 2024-04-26T22:24:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:49 smithi033 ceph-mon[27845]: pgmap v20503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38030.264940 secs 2024-04-26T22:24:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:49 smithi096 ceph-mon[32753]: pgmap v20503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38030.264940 secs 2024-04-26T22:24:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:51 smithi194 ceph-mon[33017]: pgmap v20504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:51 smithi033 ceph-mon[27845]: pgmap v20504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:51 smithi096 ceph-mon[32753]: pgmap v20504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:53 smithi194 ceph-mon[33017]: pgmap v20505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38032.285472 secs 2024-04-26T22:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:53 smithi033 ceph-mon[27845]: pgmap v20505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38032.285472 secs 2024-04-26T22:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:53 smithi096 ceph-mon[32753]: pgmap v20505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38032.285472 secs 2024-04-26T22:24:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38035.265077 secs 2024-04-26T22:24:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38035.265077 secs 2024-04-26T22:24:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38035.265077 secs 2024-04-26T22:24:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:55 smithi194 ceph-mon[33017]: pgmap v20506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:55 smithi033 ceph-mon[27845]: pgmap v20506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:55 smithi096 ceph-mon[32753]: pgmap v20506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:24:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:58.274+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:24:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:58.274+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:24:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:58.275+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 297 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:24:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:24:58.275+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 297 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:24:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:57 smithi194 ceph-mon[33017]: pgmap v20507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:57 smithi033 ceph-mon[27845]: pgmap v20507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:57 smithi096 ceph-mon[32753]: pgmap v20507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:24:59.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:24:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38037.285560 secs 2024-04-26T22:24:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:24:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38037.285560 secs 2024-04-26T22:24:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:24:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38037.285560 secs 2024-04-26T22:25:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:00 smithi194 ceph-mon[33017]: pgmap v20508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38040.265197 secs 2024-04-26T22:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:00 smithi033 ceph-mon[27845]: pgmap v20508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38040.265197 secs 2024-04-26T22:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:00 smithi096 ceph-mon[32753]: pgmap v20508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38040.265197 secs 2024-04-26T22:25:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:00.857+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:00.857+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:00.857+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:25:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:02 smithi194 ceph-mon[33017]: pgmap v20509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:02 smithi033 ceph-mon[27845]: pgmap v20509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:02 smithi096 ceph-mon[32753]: pgmap v20509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:04 smithi194 ceph-mon[33017]: pgmap v20510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38042.285646 secs 2024-04-26T22:25:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:04 smithi033 ceph-mon[27845]: pgmap v20510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38042.285646 secs 2024-04-26T22:25:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:04 smithi096 ceph-mon[32753]: pgmap v20510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38042.285646 secs 2024-04-26T22:25:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38045.265314 secs 2024-04-26T22:25:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38045.265314 secs 2024-04-26T22:25:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38045.265314 secs 2024-04-26T22:25:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:06 smithi194 ceph-mon[33017]: pgmap v20511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:06 smithi033 ceph-mon[27845]: pgmap v20511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:06 smithi096 ceph-mon[32753]: pgmap v20511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:08 smithi194 ceph-mon[33017]: pgmap v20512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:08 smithi033 ceph-mon[27845]: pgmap v20512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:08 smithi096 ceph-mon[32753]: pgmap v20512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38047.285735 secs 2024-04-26T22:25:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38047.285735 secs 2024-04-26T22:25:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38047.285735 secs 2024-04-26T22:25:10.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:10 smithi033 ceph-mon[27845]: pgmap v20513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:10.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38050.265435 secs 2024-04-26T22:25:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:10 smithi194 ceph-mon[33017]: pgmap v20513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:10.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38050.265435 secs 2024-04-26T22:25:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:10 smithi096 ceph-mon[32753]: pgmap v20513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38050.265435 secs 2024-04-26T22:25:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:12 smithi194 ceph-mon[33017]: pgmap v20514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:12 smithi033 ceph-mon[27845]: pgmap v20514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:12 smithi096 ceph-mon[32753]: pgmap v20514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:14 smithi194 ceph-mon[33017]: pgmap v20515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38052.285819 secs 2024-04-26T22:25:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:14 smithi033 ceph-mon[27845]: pgmap v20515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38052.285819 secs 2024-04-26T22:25:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:14 smithi096 ceph-mon[32753]: pgmap v20515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38052.285819 secs 2024-04-26T22:25:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38055.265528 secs 2024-04-26T22:25:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38055.265528 secs 2024-04-26T22:25:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38055.265528 secs 2024-04-26T22:25:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:15.873+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:15.873+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:25:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:15.873+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:16 smithi194 ceph-mon[33017]: pgmap v20516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:16 smithi033 ceph-mon[27845]: pgmap v20516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:16 smithi096 ceph-mon[32753]: pgmap v20516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:18 smithi194 ceph-mon[33017]: pgmap v20517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:18 smithi033 ceph-mon[27845]: pgmap v20517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:18 smithi096 ceph-mon[32753]: pgmap v20517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38057.285925 secs 2024-04-26T22:25:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38057.285925 secs 2024-04-26T22:25:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38057.285925 secs 2024-04-26T22:25:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:20 smithi194 ceph-mon[33017]: pgmap v20518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38060.265661 secs 2024-04-26T22:25:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:20 smithi033 ceph-mon[27845]: pgmap v20518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38060.265661 secs 2024-04-26T22:25:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:20 smithi096 ceph-mon[32753]: pgmap v20518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38060.265661 secs 2024-04-26T22:25:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:22 smithi194 ceph-mon[33017]: pgmap v20519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:22 smithi033 ceph-mon[27845]: pgmap v20519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:22 smithi096 ceph-mon[32753]: pgmap v20519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:24 smithi194 ceph-mon[33017]: pgmap v20520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38062.286017 secs 2024-04-26T22:25:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:24 smithi033 ceph-mon[27845]: pgmap v20520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:24.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38062.286017 secs 2024-04-26T22:25:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:24 smithi096 ceph-mon[32753]: pgmap v20520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38062.286017 secs 2024-04-26T22:25:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38065.265739 secs 2024-04-26T22:25:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38065.265739 secs 2024-04-26T22:25:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38065.265739 secs 2024-04-26T22:25:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:26 smithi033 ceph-mon[27845]: pgmap v20521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:26 smithi096 ceph-mon[32753]: pgmap v20521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:26 smithi194 ceph-mon[33017]: pgmap v20521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:28 smithi033 ceph-mon[27845]: pgmap v20522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:28 smithi096 ceph-mon[32753]: pgmap v20522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:28 smithi194 ceph-mon[33017]: pgmap v20522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38067.286077 secs 2024-04-26T22:25:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38067.286077 secs 2024-04-26T22:25:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:25:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38067.286077 secs 2024-04-26T22:25:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:25:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:30 smithi033 ceph-mon[27845]: pgmap v20523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38070.265825 secs 2024-04-26T22:25:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:25:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:25:30.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:25:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:30 smithi096 ceph-mon[32753]: pgmap v20523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38070.265825 secs 2024-04-26T22:25:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:25:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:25:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:25:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:30 smithi194 ceph-mon[33017]: pgmap v20523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38070.265825 secs 2024-04-26T22:25:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:25:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:25:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:25:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:30.888+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:30.888+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:25:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:30.888+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:31 smithi033 ceph-mon[27845]: pgmap v20524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:25:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:31 smithi033 ceph-mon[27845]: pgmap v20525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:31 smithi096 ceph-mon[32753]: pgmap v20524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:25:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:31 smithi096 ceph-mon[32753]: pgmap v20525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:31 smithi194 ceph-mon[33017]: pgmap v20524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:25:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:31 smithi194 ceph-mon[33017]: pgmap v20525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:33 smithi033 ceph-mon[27845]: pgmap v20526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:33 smithi096 ceph-mon[32753]: pgmap v20526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:33 smithi194 ceph-mon[33017]: pgmap v20526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38072.286177 secs 2024-04-26T22:25:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38072.286177 secs 2024-04-26T22:25:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38072.286177 secs 2024-04-26T22:25:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38075.265888 secs 2024-04-26T22:25:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:35 smithi033 ceph-mon[27845]: pgmap v20527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38075.265888 secs 2024-04-26T22:25:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:35 smithi096 ceph-mon[32753]: pgmap v20527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38075.265888 secs 2024-04-26T22:25:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:35 smithi194 ceph-mon[33017]: pgmap v20527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:37 smithi033 ceph-mon[27845]: pgmap v20528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:37 smithi096 ceph-mon[32753]: pgmap v20528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:37 smithi194 ceph-mon[33017]: pgmap v20528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:39 smithi033 ceph-mon[27845]: pgmap v20529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38077.286237 secs 2024-04-26T22:25:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:39 smithi096 ceph-mon[32753]: pgmap v20529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38077.286237 secs 2024-04-26T22:25:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:39 smithi194 ceph-mon[33017]: pgmap v20529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:25:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38077.286237 secs 2024-04-26T22:25:40.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38080.266010 secs 2024-04-26T22:25:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:40 smithi194 ceph-mon[33017]: pgmap v20530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:25:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38080.266010 secs 2024-04-26T22:25:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:40 smithi033 ceph-mon[27845]: pgmap v20530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:25:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38080.266010 secs 2024-04-26T22:25:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:40 smithi096 ceph-mon[32753]: pgmap v20530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:25:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:42 smithi033 ceph-mon[27845]: pgmap v20531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:42 smithi096 ceph-mon[32753]: pgmap v20531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:42 smithi194 ceph-mon[33017]: pgmap v20531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38082.286293 secs 2024-04-26T22:25:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38082.286293 secs 2024-04-26T22:25:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38082.286293 secs 2024-04-26T22:25:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38085.266076 secs 2024-04-26T22:25:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:44 smithi033 ceph-mon[27845]: pgmap v20532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38085.266076 secs 2024-04-26T22:25:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:44 smithi096 ceph-mon[32753]: pgmap v20532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38085.266076 secs 2024-04-26T22:25:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:44 smithi194 ceph-mon[33017]: pgmap v20532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:45.902+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:45.902+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:25:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:25:45.902+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:25:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:46 smithi033 ceph-mon[27845]: pgmap v20533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:46 smithi096 ceph-mon[32753]: pgmap v20533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:46 smithi194 ceph-mon[33017]: pgmap v20533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:48 smithi033 ceph-mon[27845]: pgmap v20534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:49.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38087.286378 secs 2024-04-26T22:25:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:48 smithi096 ceph-mon[32753]: pgmap v20534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38087.286378 secs 2024-04-26T22:25:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:48 smithi194 ceph-mon[33017]: pgmap v20534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38087.286378 secs 2024-04-26T22:25:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38090.266194 secs 2024-04-26T22:25:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38090.266194 secs 2024-04-26T22:25:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38090.266194 secs 2024-04-26T22:25:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:50 smithi194 ceph-mon[33017]: pgmap v20535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:50 smithi033 ceph-mon[27845]: pgmap v20535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:50 smithi096 ceph-mon[32753]: pgmap v20535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:52 smithi194 ceph-mon[33017]: pgmap v20536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:52 smithi033 ceph-mon[27845]: pgmap v20536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:52 smithi096 ceph-mon[32753]: pgmap v20536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38092.286462 secs 2024-04-26T22:25:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38092.286462 secs 2024-04-26T22:25:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38092.286462 secs 2024-04-26T22:25:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38095.266314 secs 2024-04-26T22:25:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:54 smithi194 ceph-mon[33017]: pgmap v20537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38095.266314 secs 2024-04-26T22:25:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:54 smithi033 ceph-mon[27845]: pgmap v20537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:55.440 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38095.266314 secs 2024-04-26T22:25:55.441 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:54 smithi096 ceph-mon[32753]: pgmap v20537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:56 smithi194 ceph-mon[33017]: pgmap v20538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:56 smithi033 ceph-mon[27845]: pgmap v20538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:56 smithi096 ceph-mon[32753]: pgmap v20538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:25:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:58 smithi194 ceph-mon[33017]: pgmap v20539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38097.286550 secs 2024-04-26T22:25:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:58 smithi033 ceph-mon[27845]: pgmap v20539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38097.286550 secs 2024-04-26T22:25:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:58 smithi096 ceph-mon[32753]: pgmap v20539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:25:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38097.286550 secs 2024-04-26T22:26:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:25:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38100.266429 secs 2024-04-26T22:26:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:25:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38100.266429 secs 2024-04-26T22:26:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:25:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38100.266429 secs 2024-04-26T22:26:00.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:00.902+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:00.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:00.902+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:00.904 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:00.903+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:26:01.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:00 smithi033 ceph-mon[27845]: pgmap v20540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:00 smithi194 ceph-mon[33017]: pgmap v20540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:00 smithi096 ceph-mon[32753]: pgmap v20540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:02 smithi194 ceph-mon[33017]: pgmap v20541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:02 smithi033 ceph-mon[27845]: pgmap v20541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:02 smithi096 ceph-mon[32753]: pgmap v20541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:04.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38102.286634 secs 2024-04-26T22:26:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38102.286634 secs 2024-04-26T22:26:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38102.286634 secs 2024-04-26T22:26:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38105.266508 secs 2024-04-26T22:26:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:04 smithi194 ceph-mon[33017]: pgmap v20542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38105.266508 secs 2024-04-26T22:26:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:04 smithi033 ceph-mon[27845]: pgmap v20542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38105.266508 secs 2024-04-26T22:26:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:04 smithi096 ceph-mon[32753]: pgmap v20542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:07 smithi194 ceph-mon[33017]: pgmap v20543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:07 smithi033 ceph-mon[27845]: pgmap v20543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:07 smithi096 ceph-mon[32753]: pgmap v20543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:09 smithi194 ceph-mon[33017]: pgmap v20544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38107.286719 secs 2024-04-26T22:26:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:09 smithi033 ceph-mon[27845]: pgmap v20544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38107.286719 secs 2024-04-26T22:26:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:09 smithi096 ceph-mon[32753]: pgmap v20544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38107.286719 secs 2024-04-26T22:26:10.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38110.266641 secs 2024-04-26T22:26:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38110.266641 secs 2024-04-26T22:26:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38110.266641 secs 2024-04-26T22:26:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:11 smithi194 ceph-mon[33017]: pgmap v20545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:11 smithi033 ceph-mon[27845]: pgmap v20545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:11 smithi096 ceph-mon[32753]: pgmap v20545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:13 smithi194 ceph-mon[33017]: pgmap v20546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:13 smithi033 ceph-mon[27845]: pgmap v20546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:13 smithi096 ceph-mon[32753]: pgmap v20546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38112.286778 secs 2024-04-26T22:26:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38112.286778 secs 2024-04-26T22:26:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38112.286778 secs 2024-04-26T22:26:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38115.266777 secs 2024-04-26T22:26:15.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:15 smithi194 ceph-mon[33017]: pgmap v20547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38115.266777 secs 2024-04-26T22:26:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:15 smithi033 ceph-mon[27845]: pgmap v20547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38115.266777 secs 2024-04-26T22:26:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:15 smithi096 ceph-mon[32753]: pgmap v20547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:15.919+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:15.919+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:15.919+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:26:17.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:17 smithi194 ceph-mon[33017]: pgmap v20548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:17 smithi033 ceph-mon[27845]: pgmap v20548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:17 smithi096 ceph-mon[32753]: pgmap v20548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:19 smithi194 ceph-mon[33017]: pgmap v20549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:19.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38117.286867 secs 2024-04-26T22:26:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:19 smithi033 ceph-mon[27845]: pgmap v20549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38117.286867 secs 2024-04-26T22:26:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:19 smithi096 ceph-mon[32753]: pgmap v20549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38117.286867 secs 2024-04-26T22:26:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38120.266864 secs 2024-04-26T22:26:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38120.266864 secs 2024-04-26T22:26:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38120.266864 secs 2024-04-26T22:26:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:21 smithi194 ceph-mon[33017]: pgmap v20550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:21 smithi033 ceph-mon[27845]: pgmap v20550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:21 smithi096 ceph-mon[32753]: pgmap v20550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef006a070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef006a070 comp rx=0 tx=0).stop 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:26:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.824+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:26:21.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.825+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:26:21.826 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:21.826+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0058000 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:26:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:23 smithi194 ceph-mon[33017]: pgmap v20551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:23 smithi033 ceph-mon[27845]: pgmap v20551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:23 smithi096 ceph-mon[32753]: pgmap v20551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38122.286952 secs 2024-04-26T22:26:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38122.286952 secs 2024-04-26T22:26:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38122.286952 secs 2024-04-26T22:26:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38125.266983 secs 2024-04-26T22:26:25.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:25 smithi194 ceph-mon[33017]: pgmap v20552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38125.266983 secs 2024-04-26T22:26:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:25 smithi033 ceph-mon[27845]: pgmap v20552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38125.266983 secs 2024-04-26T22:26:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:25 smithi096 ceph-mon[32753]: pgmap v20552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:27 smithi194 ceph-mon[33017]: pgmap v20553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:27 smithi033 ceph-mon[27845]: pgmap v20553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:27 smithi096 ceph-mon[32753]: pgmap v20553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:28.291+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:26:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:28.291+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:26:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:28.292+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 298 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:26:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:28.292+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 298 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:26:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:29 smithi194 ceph-mon[33017]: pgmap v20554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38127.286979 secs 2024-04-26T22:26:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:29 smithi033 ceph-mon[27845]: pgmap v20554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38127.286979 secs 2024-04-26T22:26:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:29 smithi096 ceph-mon[32753]: pgmap v20554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38127.286979 secs 2024-04-26T22:26:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38130.267044 secs 2024-04-26T22:26:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38130.267044 secs 2024-04-26T22:26:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38130.267044 secs 2024-04-26T22:26:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:30.928+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:30.928+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:30.928+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:26:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:31 smithi194 ceph-mon[33017]: pgmap v20555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:26:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:31 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:31 smithi033 ceph-mon[27845]: pgmap v20555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:26:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:31 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:31 smithi096 ceph-mon[32753]: pgmap v20555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:26:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:31 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:33 smithi033 ceph-mon[27845]: pgmap v20556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:33 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:33 smithi096 ceph-mon[32753]: pgmap v20556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:33 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:33 smithi194 ceph-mon[33017]: pgmap v20556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:33.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:33 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38132.287068 secs 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38132.287068 secs 2024-04-26T22:26:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38132.287068 secs 2024-04-26T22:26:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:34.606 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38135.267140 secs 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: pgmap v20557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:26:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:26:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:26:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:26:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38135.267140 secs 2024-04-26T22:26:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: pgmap v20557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:35.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38135.267140 secs 2024-04-26T22:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: pgmap v20557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:35.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: pgmap v20558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:26:37.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:26:37.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: pgmap v20558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:26:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:26:37.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: pgmap v20558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:26:37.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:26:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:26:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:26:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:38 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:26:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:38 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:26:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:38 smithi033 ceph-mon[27845]: pgmap v20559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:26:38.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:38 smithi033 ceph-mon[27845]: pgmap v20560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:38 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:26:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:38 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:26:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:38 smithi096 ceph-mon[32753]: pgmap v20559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:26:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:38 smithi096 ceph-mon[32753]: pgmap v20560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:38 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:26:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:38 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:26:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:38 smithi194 ceph-mon[33017]: pgmap v20559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:26:38.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:38 smithi194 ceph-mon[33017]: pgmap v20560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38137.287126 secs 2024-04-26T22:26:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38137.287126 secs 2024-04-26T22:26:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38137.287126 secs 2024-04-26T22:26:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:40 smithi033 ceph-mon[27845]: pgmap v20561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:26:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38140.267258 secs 2024-04-26T22:26:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:40 smithi096 ceph-mon[32753]: pgmap v20561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:26:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38140.267258 secs 2024-04-26T22:26:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:40 smithi194 ceph-mon[33017]: pgmap v20561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:26:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38140.267258 secs 2024-04-26T22:26:42.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:42 smithi033 ceph-mon[27845]: pgmap v20562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:42 smithi096 ceph-mon[32753]: pgmap v20562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:42 smithi194 ceph-mon[33017]: pgmap v20562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:44 smithi033 ceph-mon[27845]: pgmap v20563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38142.287180 secs 2024-04-26T22:26:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:44 smithi096 ceph-mon[32753]: pgmap v20563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38142.287180 secs 2024-04-26T22:26:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:44 smithi194 ceph-mon[33017]: pgmap v20563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:26:44.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38142.287180 secs 2024-04-26T22:26:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38145.267380 secs 2024-04-26T22:26:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38145.267380 secs 2024-04-26T22:26:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38145.267380 secs 2024-04-26T22:26:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:45.937+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:45.937+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:26:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:26:45.937+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:26:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:46 smithi033 ceph-mon[27845]: pgmap v20564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:26:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:46 smithi096 ceph-mon[32753]: pgmap v20564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:26:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:46 smithi194 ceph-mon[33017]: pgmap v20564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:26:48.434 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:48 smithi096 ceph-mon[32753]: pgmap v20565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:26:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:48 smithi194 ceph-mon[33017]: pgmap v20565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:26:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:48 smithi033 ceph-mon[27845]: pgmap v20565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:26:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38147.287266 secs 2024-04-26T22:26:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38147.287266 secs 2024-04-26T22:26:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38147.287266 secs 2024-04-26T22:26:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:50 smithi194 ceph-mon[33017]: pgmap v20566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38150.267486 secs 2024-04-26T22:26:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:50 smithi033 ceph-mon[27845]: pgmap v20566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38150.267486 secs 2024-04-26T22:26:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:50 smithi096 ceph-mon[32753]: pgmap v20566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38150.267486 secs 2024-04-26T22:26:52.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:52 smithi194 ceph-mon[33017]: pgmap v20567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:52 smithi033 ceph-mon[27845]: pgmap v20567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:52.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:52 smithi096 ceph-mon[32753]: pgmap v20567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:53 smithi033 ceph-mon[27845]: pgmap v20568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:53 smithi194 ceph-mon[33017]: pgmap v20568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:53 smithi096 ceph-mon[32753]: pgmap v20568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38152.287351 secs 2024-04-26T22:26:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38152.287351 secs 2024-04-26T22:26:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38152.287351 secs 2024-04-26T22:26:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38155.267559 secs 2024-04-26T22:26:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:55 smithi194 ceph-mon[33017]: pgmap v20569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38155.267559 secs 2024-04-26T22:26:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:55 smithi033 ceph-mon[27845]: pgmap v20569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38155.267559 secs 2024-04-26T22:26:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:55 smithi096 ceph-mon[32753]: pgmap v20569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:57 smithi194 ceph-mon[33017]: pgmap v20570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:57 smithi033 ceph-mon[27845]: pgmap v20570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:57 smithi096 ceph-mon[32753]: pgmap v20570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:26:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38157.287411 secs 2024-04-26T22:26:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:26:59 smithi194 ceph-mon[33017]: pgmap v20571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38157.287411 secs 2024-04-26T22:26:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:26:59 smithi033 ceph-mon[27845]: pgmap v20571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:26:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38157.287411 secs 2024-04-26T22:26:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:26:59 smithi096 ceph-mon[32753]: pgmap v20571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38160.267696 secs 2024-04-26T22:27:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38160.267696 secs 2024-04-26T22:27:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38160.267696 secs 2024-04-26T22:27:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:00.937+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:00.937+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:00.937+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:27:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:01 smithi194 ceph-mon[33017]: pgmap v20572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:01 smithi033 ceph-mon[27845]: pgmap v20572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:01 smithi096 ceph-mon[32753]: pgmap v20572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:03 smithi033 ceph-mon[27845]: pgmap v20573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:03 smithi194 ceph-mon[33017]: pgmap v20573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:03 smithi096 ceph-mon[32753]: pgmap v20573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38162.287496 secs 2024-04-26T22:27:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38162.287496 secs 2024-04-26T22:27:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38162.287496 secs 2024-04-26T22:27:05.599 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38165.267818 secs 2024-04-26T22:27:05.599 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:05 smithi096 ceph-mon[32753]: pgmap v20574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38165.267818 secs 2024-04-26T22:27:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:05 smithi194 ceph-mon[33017]: pgmap v20574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38165.267818 secs 2024-04-26T22:27:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:05 smithi033 ceph-mon[27845]: pgmap v20574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:07 smithi194 ceph-mon[33017]: pgmap v20575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:07 smithi033 ceph-mon[27845]: pgmap v20575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:07 smithi096 ceph-mon[32753]: pgmap v20575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38167.287580 secs 2024-04-26T22:27:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:09 smithi194 ceph-mon[33017]: pgmap v20576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38167.287580 secs 2024-04-26T22:27:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:09 smithi033 ceph-mon[27845]: pgmap v20576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38167.287580 secs 2024-04-26T22:27:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:09 smithi096 ceph-mon[32753]: pgmap v20576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38170.267929 secs 2024-04-26T22:27:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38170.267929 secs 2024-04-26T22:27:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38170.267929 secs 2024-04-26T22:27:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:11 smithi033 ceph-mon[27845]: pgmap v20577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:11 smithi096 ceph-mon[32753]: pgmap v20577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:11 smithi194 ceph-mon[33017]: pgmap v20577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:13 smithi033 ceph-mon[27845]: pgmap v20578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:13 smithi096 ceph-mon[32753]: pgmap v20578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:13 smithi194 ceph-mon[33017]: pgmap v20578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38172.287664 secs 2024-04-26T22:27:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38172.287664 secs 2024-04-26T22:27:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38172.287664 secs 2024-04-26T22:27:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38175.268045 secs 2024-04-26T22:27:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:15 smithi033 ceph-mon[27845]: pgmap v20579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38175.268045 secs 2024-04-26T22:27:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:15 smithi096 ceph-mon[32753]: pgmap v20579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38175.268045 secs 2024-04-26T22:27:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:15 smithi194 ceph-mon[33017]: pgmap v20579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:15.939+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:15.939+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:27:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:15.939+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:17 smithi033 ceph-mon[27845]: pgmap v20580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:17 smithi096 ceph-mon[32753]: pgmap v20580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:17 smithi194 ceph-mon[33017]: pgmap v20580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38177.287750 secs 2024-04-26T22:27:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:19 smithi033 ceph-mon[27845]: pgmap v20581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38177.287750 secs 2024-04-26T22:27:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:19 smithi096 ceph-mon[32753]: pgmap v20581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38177.287750 secs 2024-04-26T22:27:19.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:19 smithi194 ceph-mon[33017]: pgmap v20581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38180.268200 secs 2024-04-26T22:27:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38180.268200 secs 2024-04-26T22:27:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38180.268200 secs 2024-04-26T22:27:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:21 smithi033 ceph-mon[27845]: pgmap v20582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:21 smithi096 ceph-mon[32753]: pgmap v20582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:21 smithi194 ceph-mon[33017]: pgmap v20582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:23 smithi033 ceph-mon[27845]: pgmap v20583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:23.684 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:23 smithi096 ceph-mon[32753]: pgmap v20583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:23 smithi194 ceph-mon[33017]: pgmap v20583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38182.287834 secs 2024-04-26T22:27:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38182.287834 secs 2024-04-26T22:27:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38182.287834 secs 2024-04-26T22:27:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38185.268317 secs 2024-04-26T22:27:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:25 smithi096 ceph-mon[32753]: pgmap v20584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38185.268317 secs 2024-04-26T22:27:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:25 smithi194 ceph-mon[33017]: pgmap v20584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38185.268317 secs 2024-04-26T22:27:25.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:25 smithi033 ceph-mon[27845]: pgmap v20584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:27 smithi194 ceph-mon[33017]: pgmap v20585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:27 smithi033 ceph-mon[27845]: pgmap v20585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:27.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:27 smithi096 ceph-mon[32753]: pgmap v20585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38187.287921 secs 2024-04-26T22:27:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:29 smithi194 ceph-mon[33017]: pgmap v20586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38187.287921 secs 2024-04-26T22:27:29.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:29 smithi033 ceph-mon[27845]: pgmap v20586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38187.287921 secs 2024-04-26T22:27:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:29 smithi096 ceph-mon[32753]: pgmap v20586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38190.268449 secs 2024-04-26T22:27:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38190.268449 secs 2024-04-26T22:27:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38190.268449 secs 2024-04-26T22:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:30.951+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:30.951+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:30.951+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:27:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:31 smithi194 ceph-mon[33017]: pgmap v20587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:31 smithi033 ceph-mon[27845]: pgmap v20587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:31.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:31 smithi096 ceph-mon[32753]: pgmap v20587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:33 smithi194 ceph-mon[33017]: pgmap v20588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:33 smithi033 ceph-mon[27845]: pgmap v20588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:33.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:33 smithi096 ceph-mon[32753]: pgmap v20588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38192.288026 secs 2024-04-26T22:27:34.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38192.288026 secs 2024-04-26T22:27:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38192.288026 secs 2024-04-26T22:27:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38195.268566 secs 2024-04-26T22:27:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:35 smithi194 ceph-mon[33017]: pgmap v20589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38195.268566 secs 2024-04-26T22:27:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:35 smithi033 ceph-mon[27845]: pgmap v20589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38195.268566 secs 2024-04-26T22:27:35.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:35 smithi096 ceph-mon[32753]: pgmap v20589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:37 smithi194 ceph-mon[33017]: pgmap v20590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:27:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:37 smithi033 ceph-mon[27845]: pgmap v20590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:37.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:27:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:37 smithi096 ceph-mon[32753]: pgmap v20590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:37.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:27:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:27:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:38 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:27:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:38 smithi194 ceph-mon[33017]: pgmap v20591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:38 smithi194 ceph-mon[33017]: pgmap v20592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:38.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:38 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:27:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:27:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:38 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:27:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:38 smithi033 ceph-mon[27845]: pgmap v20591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:38 smithi033 ceph-mon[27845]: pgmap v20592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:38.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:38 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:27:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:27:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:38 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:27:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:38 smithi096 ceph-mon[32753]: pgmap v20591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:38 smithi096 ceph-mon[32753]: pgmap v20592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:38 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:27:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38197.288089 secs 2024-04-26T22:27:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38197.288089 secs 2024-04-26T22:27:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38197.288089 secs 2024-04-26T22:27:40.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38200.268682 secs 2024-04-26T22:27:40.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:40 smithi194 ceph-mon[33017]: pgmap v20593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:40.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38200.268682 secs 2024-04-26T22:27:40.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:40 smithi033 ceph-mon[27845]: pgmap v20593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38200.268682 secs 2024-04-26T22:27:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:40 smithi096 ceph-mon[32753]: pgmap v20593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:42 smithi194 ceph-mon[33017]: pgmap v20594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:42 smithi033 ceph-mon[27845]: pgmap v20594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:42 smithi096 ceph-mon[32753]: pgmap v20594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38202.288151 secs 2024-04-26T22:27:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38202.288151 secs 2024-04-26T22:27:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38202.288151 secs 2024-04-26T22:27:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38205.268803 secs 2024-04-26T22:27:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:44 smithi194 ceph-mon[33017]: pgmap v20595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38205.268803 secs 2024-04-26T22:27:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:44 smithi033 ceph-mon[27845]: pgmap v20595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38205.268803 secs 2024-04-26T22:27:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:44 smithi096 ceph-mon[32753]: pgmap v20595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:45.965+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:45.965+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:27:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:27:45.965+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:27:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:46 smithi194 ceph-mon[33017]: pgmap v20596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:47.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:46 smithi033 ceph-mon[27845]: pgmap v20596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:46 smithi096 ceph-mon[32753]: pgmap v20596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:27:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:48 smithi194 ceph-mon[33017]: pgmap v20597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:27:49.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38207.288242 secs 2024-04-26T22:27:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:48 smithi033 ceph-mon[27845]: pgmap v20597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:27:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38207.288242 secs 2024-04-26T22:27:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:48 smithi096 ceph-mon[32753]: pgmap v20597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:27:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38207.288242 secs 2024-04-26T22:27:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38210.268897 secs 2024-04-26T22:27:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38210.268897 secs 2024-04-26T22:27:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38210.268897 secs 2024-04-26T22:27:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:51 smithi194 ceph-mon[33017]: pgmap v20598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:51 smithi033 ceph-mon[27845]: pgmap v20598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:51 smithi096 ceph-mon[32753]: pgmap v20598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:53 smithi194 ceph-mon[33017]: pgmap v20599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:53 smithi033 ceph-mon[27845]: pgmap v20599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:53 smithi096 ceph-mon[32753]: pgmap v20599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38212.288328 secs 2024-04-26T22:27:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38212.288328 secs 2024-04-26T22:27:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38212.288328 secs 2024-04-26T22:27:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38215.269014 secs 2024-04-26T22:27:55.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:55 smithi194 ceph-mon[33017]: pgmap v20600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38215.269014 secs 2024-04-26T22:27:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:55 smithi033 ceph-mon[27845]: pgmap v20600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38215.269014 secs 2024-04-26T22:27:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:55 smithi096 ceph-mon[32753]: pgmap v20600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:57 smithi194 ceph-mon[33017]: pgmap v20601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:57 smithi033 ceph-mon[27845]: pgmap v20601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:57 smithi096 ceph-mon[32753]: pgmap v20601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:27:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:59 smithi194 ceph-mon[33017]: pgmap v20602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:59.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:27:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38217.288412 secs 2024-04-26T22:27:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:59 smithi033 ceph-mon[27845]: pgmap v20602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:27:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38217.288412 secs 2024-04-26T22:27:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:59 smithi096 ceph-mon[32753]: pgmap v20602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:27:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:27:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38217.288412 secs 2024-04-26T22:28:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38220.269144 secs 2024-04-26T22:28:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38220.269144 secs 2024-04-26T22:28:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38220.269144 secs 2024-04-26T22:28:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:00.966+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:00.966+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:00.966+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:28:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:01 smithi194 ceph-mon[33017]: pgmap v20603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:01 smithi033 ceph-mon[27845]: pgmap v20603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:01 smithi096 ceph-mon[32753]: pgmap v20603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:03 smithi194 ceph-mon[33017]: pgmap v20604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:03 smithi033 ceph-mon[27845]: pgmap v20604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:03 smithi096 ceph-mon[32753]: pgmap v20604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:04.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38222.288490 secs 2024-04-26T22:28:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38222.288490 secs 2024-04-26T22:28:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38222.288490 secs 2024-04-26T22:28:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38225.269262 secs 2024-04-26T22:28:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:05 smithi194 ceph-mon[33017]: pgmap v20605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38225.269262 secs 2024-04-26T22:28:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:05 smithi033 ceph-mon[27845]: pgmap v20605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38225.269262 secs 2024-04-26T22:28:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:05 smithi096 ceph-mon[32753]: pgmap v20605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:07 smithi194 ceph-mon[33017]: pgmap v20606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:07 smithi033 ceph-mon[27845]: pgmap v20606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:07 smithi096 ceph-mon[32753]: pgmap v20606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:09 smithi194 ceph-mon[33017]: pgmap v20607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38227.288576 secs 2024-04-26T22:28:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:09 smithi033 ceph-mon[27845]: pgmap v20607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38227.288576 secs 2024-04-26T22:28:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:09 smithi096 ceph-mon[32753]: pgmap v20607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38227.288576 secs 2024-04-26T22:28:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38230.269380 secs 2024-04-26T22:28:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38230.269380 secs 2024-04-26T22:28:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38230.269380 secs 2024-04-26T22:28:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:11 smithi194 ceph-mon[33017]: pgmap v20608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:11 smithi033 ceph-mon[27845]: pgmap v20608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:11 smithi096 ceph-mon[32753]: pgmap v20608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:13 smithi194 ceph-mon[33017]: pgmap v20609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:13 smithi033 ceph-mon[27845]: pgmap v20609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:13 smithi096 ceph-mon[32753]: pgmap v20609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38232.288637 secs 2024-04-26T22:28:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38232.288637 secs 2024-04-26T22:28:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38232.288637 secs 2024-04-26T22:28:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38235.269495 secs 2024-04-26T22:28:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:15 smithi033 ceph-mon[27845]: pgmap v20610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38235.269495 secs 2024-04-26T22:28:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:15 smithi096 ceph-mon[32753]: pgmap v20610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38235.269495 secs 2024-04-26T22:28:15.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:15 smithi194 ceph-mon[33017]: pgmap v20610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:15.980+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:15.980+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:15.980+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:28:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:17 smithi033 ceph-mon[27845]: pgmap v20611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:17 smithi096 ceph-mon[32753]: pgmap v20611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:17 smithi194 ceph-mon[33017]: pgmap v20611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:19 smithi033 ceph-mon[27845]: pgmap v20612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38237.288723 secs 2024-04-26T22:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:19 smithi096 ceph-mon[32753]: pgmap v20612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38237.288723 secs 2024-04-26T22:28:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:19 smithi194 ceph-mon[33017]: pgmap v20612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38237.288723 secs 2024-04-26T22:28:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38240.269614 secs 2024-04-26T22:28:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38240.269614 secs 2024-04-26T22:28:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38240.269614 secs 2024-04-26T22:28:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:21 smithi033 ceph-mon[27845]: pgmap v20613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:21 smithi096 ceph-mon[32753]: pgmap v20613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:21 smithi194 ceph-mon[33017]: pgmap v20613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:23 smithi033 ceph-mon[27845]: pgmap v20614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:23 smithi096 ceph-mon[32753]: pgmap v20614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:23 smithi194 ceph-mon[33017]: pgmap v20614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38242.288808 secs 2024-04-26T22:28:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38242.288808 secs 2024-04-26T22:28:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38242.288808 secs 2024-04-26T22:28:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38245.269753 secs 2024-04-26T22:28:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:25 smithi033 ceph-mon[27845]: pgmap v20615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38245.269753 secs 2024-04-26T22:28:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:25 smithi096 ceph-mon[32753]: pgmap v20615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38245.269753 secs 2024-04-26T22:28:25.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:25 smithi194 ceph-mon[33017]: pgmap v20615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:27 smithi033 ceph-mon[27845]: pgmap v20616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:27 smithi096 ceph-mon[32753]: pgmap v20616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:27 smithi194 ceph-mon[33017]: pgmap v20616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:29 smithi033 ceph-mon[27845]: pgmap v20617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38247.288896 secs 2024-04-26T22:28:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:29 smithi096 ceph-mon[32753]: pgmap v20617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38247.288896 secs 2024-04-26T22:28:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:29 smithi194 ceph-mon[33017]: pgmap v20617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38247.288896 secs 2024-04-26T22:28:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38250.269887 secs 2024-04-26T22:28:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38250.269887 secs 2024-04-26T22:28:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38250.269887 secs 2024-04-26T22:28:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:30.994+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:30.994+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:30.994+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:28:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:31 smithi033 ceph-mon[27845]: pgmap v20618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:31 smithi096 ceph-mon[32753]: pgmap v20618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:31 smithi194 ceph-mon[33017]: pgmap v20618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:33 smithi096 ceph-mon[32753]: pgmap v20619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:33 smithi194 ceph-mon[33017]: pgmap v20619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:33 smithi033 ceph-mon[27845]: pgmap v20619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38252.288982 secs 2024-04-26T22:28:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38252.288982 secs 2024-04-26T22:28:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38252.288982 secs 2024-04-26T22:28:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38255.269982 secs 2024-04-26T22:28:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:35 smithi194 ceph-mon[33017]: pgmap v20620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38255.269982 secs 2024-04-26T22:28:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:35 smithi033 ceph-mon[27845]: pgmap v20620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38255.269982 secs 2024-04-26T22:28:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:35 smithi096 ceph-mon[32753]: pgmap v20620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:37 smithi194 ceph-mon[33017]: pgmap v20621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:37 smithi033 ceph-mon[27845]: pgmap v20621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:37 smithi096 ceph-mon[32753]: pgmap v20621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:39 smithi194 ceph-mon[33017]: pgmap v20622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38257.289076 secs 2024-04-26T22:28:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:28:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:39 smithi033 ceph-mon[27845]: pgmap v20622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38257.289076 secs 2024-04-26T22:28:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:28:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:39 smithi096 ceph-mon[32753]: pgmap v20622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38257.289076 secs 2024-04-26T22:28:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:28:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38260.270150 secs 2024-04-26T22:28:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:28:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:28:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:28:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38260.270150 secs 2024-04-26T22:28:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:28:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:28:40.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:28:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38260.270150 secs 2024-04-26T22:28:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:28:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:28:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:28:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:41 smithi194 ceph-mon[33017]: pgmap v20623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:28:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:41 smithi194 ceph-mon[33017]: pgmap v20624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:41 smithi033 ceph-mon[27845]: pgmap v20623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:28:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:41 smithi033 ceph-mon[27845]: pgmap v20624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:41 smithi096 ceph-mon[32753]: pgmap v20623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:28:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:41 smithi096 ceph-mon[32753]: pgmap v20624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:42 smithi194 ceph-mon[33017]: pgmap v20625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:42 smithi033 ceph-mon[27845]: pgmap v20625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:42 smithi096 ceph-mon[32753]: pgmap v20625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38262.289158 secs 2024-04-26T22:28:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38262.289158 secs 2024-04-26T22:28:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38262.289158 secs 2024-04-26T22:28:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:44 smithi033 ceph-mon[27845]: pgmap v20626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:44 smithi096 ceph-mon[32753]: pgmap v20626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:44 smithi194 ceph-mon[33017]: pgmap v20626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38265.270289 secs 2024-04-26T22:28:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38265.270289 secs 2024-04-26T22:28:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:46.008+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:46.008+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:28:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:28:46.008+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:28:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38265.270289 secs 2024-04-26T22:28:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:46 smithi033 ceph-mon[27845]: pgmap v20627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:46.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:46 smithi096 ceph-mon[32753]: pgmap v20627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:46 smithi194 ceph-mon[33017]: pgmap v20627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:48 smithi033 ceph-mon[27845]: pgmap v20628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38267.289267 secs 2024-04-26T22:28:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:48 smithi096 ceph-mon[32753]: pgmap v20628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38267.289267 secs 2024-04-26T22:28:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:48 smithi194 ceph-mon[33017]: pgmap v20628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:28:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38267.289267 secs 2024-04-26T22:28:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38270.270408 secs 2024-04-26T22:28:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:50 smithi033 ceph-mon[27845]: pgmap v20629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:28:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38270.270408 secs 2024-04-26T22:28:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:50 smithi096 ceph-mon[32753]: pgmap v20629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:28:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38270.270408 secs 2024-04-26T22:28:51.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:50 smithi194 ceph-mon[33017]: pgmap v20629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:28:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:52 smithi033 ceph-mon[27845]: pgmap v20630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:52 smithi096 ceph-mon[32753]: pgmap v20630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:52 smithi194 ceph-mon[33017]: pgmap v20630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38272.289356 secs 2024-04-26T22:28:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38272.289356 secs 2024-04-26T22:28:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38272.289356 secs 2024-04-26T22:28:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:54 smithi096 ceph-mon[32753]: pgmap v20631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:54 smithi194 ceph-mon[33017]: pgmap v20631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:54 smithi033 ceph-mon[27845]: pgmap v20631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38275.270530 secs 2024-04-26T22:28:56.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38275.270530 secs 2024-04-26T22:28:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38275.270530 secs 2024-04-26T22:28:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:56 smithi194 ceph-mon[33017]: pgmap v20632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:56 smithi033 ceph-mon[27845]: pgmap v20632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:56 smithi096 ceph-mon[32753]: pgmap v20632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:28:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:58 smithi194 ceph-mon[33017]: pgmap v20633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:28:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38277.289444 secs 2024-04-26T22:28:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:58 smithi033 ceph-mon[27845]: pgmap v20633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:28:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38277.289444 secs 2024-04-26T22:28:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:58 smithi096 ceph-mon[32753]: pgmap v20633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:28:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:28:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38277.289444 secs 2024-04-26T22:29:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:01.020+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:01.020+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:29:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:01.020+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38280.270648 secs 2024-04-26T22:29:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:00 smithi194 ceph-mon[33017]: pgmap v20634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38280.270648 secs 2024-04-26T22:29:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:00 smithi033 ceph-mon[27845]: pgmap v20634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38280.270648 secs 2024-04-26T22:29:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:00 smithi096 ceph-mon[32753]: pgmap v20634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:02 smithi194 ceph-mon[33017]: pgmap v20635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:02 smithi033 ceph-mon[27845]: pgmap v20635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:02 smithi096 ceph-mon[32753]: pgmap v20635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38282.289532 secs 2024-04-26T22:29:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38282.289532 secs 2024-04-26T22:29:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38282.289532 secs 2024-04-26T22:29:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:04 smithi194 ceph-mon[33017]: pgmap v20636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38285.270767 secs 2024-04-26T22:29:05.177 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:04 smithi033 ceph-mon[27845]: pgmap v20636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:05.178 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38285.270767 secs 2024-04-26T22:29:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:04 smithi096 ceph-mon[32753]: pgmap v20636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38285.270767 secs 2024-04-26T22:29:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:06 smithi194 ceph-mon[33017]: pgmap v20637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:06 smithi033 ceph-mon[27845]: pgmap v20637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:06 smithi096 ceph-mon[32753]: pgmap v20637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:08 smithi194 ceph-mon[33017]: pgmap v20638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:09.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38287.289621 secs 2024-04-26T22:29:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:08 smithi033 ceph-mon[27845]: pgmap v20638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38287.289621 secs 2024-04-26T22:29:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:08 smithi096 ceph-mon[32753]: pgmap v20638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38287.289621 secs 2024-04-26T22:29:10.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38290.270879 secs 2024-04-26T22:29:10.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38290.270879 secs 2024-04-26T22:29:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38290.270879 secs 2024-04-26T22:29:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:10 smithi194 ceph-mon[33017]: pgmap v20639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:10 smithi033 ceph-mon[27845]: pgmap v20639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:10 smithi096 ceph-mon[32753]: pgmap v20639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:12 smithi194 ceph-mon[33017]: pgmap v20640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:12 smithi033 ceph-mon[27845]: pgmap v20640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:12 smithi096 ceph-mon[32753]: pgmap v20640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38292.289709 secs 2024-04-26T22:29:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38292.289709 secs 2024-04-26T22:29:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38292.289709 secs 2024-04-26T22:29:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:14 smithi194 ceph-mon[33017]: pgmap v20641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38295.271001 secs 2024-04-26T22:29:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:14 smithi033 ceph-mon[27845]: pgmap v20641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38295.271001 secs 2024-04-26T22:29:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:14 smithi096 ceph-mon[32753]: pgmap v20641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38295.271001 secs 2024-04-26T22:29:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:16.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:16.031+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:16.031+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:29:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:16 smithi194 ceph-mon[33017]: pgmap v20642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:16 smithi033 ceph-mon[27845]: pgmap v20642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:16 smithi096 ceph-mon[32753]: pgmap v20642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:18 smithi194 ceph-mon[33017]: pgmap v20643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38297.289799 secs 2024-04-26T22:29:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:18 smithi033 ceph-mon[27845]: pgmap v20643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:19.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38297.289799 secs 2024-04-26T22:29:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:18 smithi096 ceph-mon[32753]: pgmap v20643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38297.289799 secs 2024-04-26T22:29:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38300.271131 secs 2024-04-26T22:29:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38300.271131 secs 2024-04-26T22:29:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38300.271131 secs 2024-04-26T22:29:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:20 smithi194 ceph-mon[33017]: pgmap v20644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:20 smithi033 ceph-mon[27845]: pgmap v20644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:20 smithi096 ceph-mon[32753]: pgmap v20644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:22 smithi194 ceph-mon[33017]: pgmap v20645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:22 smithi033 ceph-mon[27845]: pgmap v20645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:22 smithi096 ceph-mon[32753]: pgmap v20645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38302.289888 secs 2024-04-26T22:29:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38302.289888 secs 2024-04-26T22:29:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38302.289888 secs 2024-04-26T22:29:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:24 smithi194 ceph-mon[33017]: pgmap v20646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38305.271223 secs 2024-04-26T22:29:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:24 smithi033 ceph-mon[27845]: pgmap v20646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38305.271223 secs 2024-04-26T22:29:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:24 smithi096 ceph-mon[32753]: pgmap v20646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38305.271223 secs 2024-04-26T22:29:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:26 smithi194 ceph-mon[33017]: pgmap v20647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:26 smithi033 ceph-mon[27845]: pgmap v20647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:26 smithi096 ceph-mon[32753]: pgmap v20647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:28.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:28.321+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:29:28.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:28.321+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:29:28.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:28.322+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 299 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T22:29:28.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:28.322+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 299 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T22:29:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:28 smithi194 ceph-mon[33017]: pgmap v20648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38307.290000 secs 2024-04-26T22:29:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:28 smithi033 ceph-mon[27845]: pgmap v20648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38307.290000 secs 2024-04-26T22:29:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:28 smithi096 ceph-mon[32753]: pgmap v20648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38307.290000 secs 2024-04-26T22:29:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38310.271348 secs 2024-04-26T22:29:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38310.271348 secs 2024-04-26T22:29:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38310.271348 secs 2024-04-26T22:29:31.047 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:31.047+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:31.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:31.047+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:31.048 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:31.047+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:29:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:30 smithi194 ceph-mon[33017]: pgmap v20649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:30 smithi033 ceph-mon[27845]: pgmap v20649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:30 smithi096 ceph-mon[32753]: pgmap v20649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:32 smithi194 ceph-mon[33017]: pgmap v20650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:32 smithi033 ceph-mon[27845]: pgmap v20650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:32 smithi096 ceph-mon[32753]: pgmap v20650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38312.290086 secs 2024-04-26T22:29:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38312.290086 secs 2024-04-26T22:29:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38312.290086 secs 2024-04-26T22:29:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:34 smithi194 ceph-mon[33017]: pgmap v20651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38315.271409 secs 2024-04-26T22:29:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:34 smithi033 ceph-mon[27845]: pgmap v20651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38315.271409 secs 2024-04-26T22:29:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:34 smithi096 ceph-mon[32753]: pgmap v20651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38315.271409 secs 2024-04-26T22:29:37.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:36 smithi033 ceph-mon[27845]: pgmap v20652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:36 smithi096 ceph-mon[32753]: pgmap v20652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:36 smithi194 ceph-mon[33017]: pgmap v20652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:38 smithi033 ceph-mon[27845]: pgmap v20653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38317.290171 secs 2024-04-26T22:29:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:38 smithi096 ceph-mon[32753]: pgmap v20653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38317.290171 secs 2024-04-26T22:29:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:38 smithi194 ceph-mon[33017]: pgmap v20653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38317.290171 secs 2024-04-26T22:29:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38320.271532 secs 2024-04-26T22:29:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38320.271532 secs 2024-04-26T22:29:40.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38320.271532 secs 2024-04-26T22:29:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:40 smithi033 ceph-mon[27845]: pgmap v20654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:29:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:40 smithi096 ceph-mon[32753]: pgmap v20654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:29:41.236 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:40 smithi194 ceph-mon[33017]: pgmap v20654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:41.236 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:29:43.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:42 smithi194 ceph-mon[33017]: pgmap v20655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:42 smithi033 ceph-mon[27845]: pgmap v20655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:42 smithi096 ceph-mon[32753]: pgmap v20655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:44.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38322.290274 secs 2024-04-26T22:29:44.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38322.290274 secs 2024-04-26T22:29:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38322.290274 secs 2024-04-26T22:29:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:45 smithi033 ceph-mon[27845]: pgmap v20656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38325.271597 secs 2024-04-26T22:29:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:45 smithi096 ceph-mon[32753]: pgmap v20656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38325.271597 secs 2024-04-26T22:29:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:45 smithi194 ceph-mon[33017]: pgmap v20656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38325.271597 secs 2024-04-26T22:29:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:46.063+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:46.063+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:29:46.063 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:29:46.063+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:29:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:47 smithi033 ceph-mon[27845]: pgmap v20657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:47.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:47 smithi096 ceph-mon[32753]: pgmap v20657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:47 smithi194 ceph-mon[33017]: pgmap v20657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:29:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:48 smithi194 ceph-mon[33017]: pgmap v20658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:48 smithi194 ceph-mon[33017]: pgmap v20659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:48.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:48 smithi033 ceph-mon[27845]: pgmap v20658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:48 smithi033 ceph-mon[27845]: pgmap v20659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:48.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:48 smithi096 ceph-mon[32753]: pgmap v20658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:48 smithi096 ceph-mon[32753]: pgmap v20659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:29:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38327.290367 secs 2024-04-26T22:29:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38327.290367 secs 2024-04-26T22:29:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38327.290367 secs 2024-04-26T22:29:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38330.271710 secs 2024-04-26T22:29:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:50 smithi033 ceph-mon[27845]: pgmap v20660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:29:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38330.271710 secs 2024-04-26T22:29:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:50 smithi096 ceph-mon[32753]: pgmap v20660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:29:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38330.271710 secs 2024-04-26T22:29:50.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:50 smithi194 ceph-mon[33017]: pgmap v20660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:29:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:52 smithi194 ceph-mon[33017]: pgmap v20661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:52 smithi033 ceph-mon[27845]: pgmap v20661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:52 smithi096 ceph-mon[32753]: pgmap v20661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38332.290426 secs 2024-04-26T22:29:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:54 smithi194 ceph-mon[33017]: pgmap v20662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38332.290426 secs 2024-04-26T22:29:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:54 smithi096 ceph-mon[32753]: pgmap v20662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38332.290426 secs 2024-04-26T22:29:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:54 smithi033 ceph-mon[27845]: pgmap v20662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:29:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38335.271827 secs 2024-04-26T22:29:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38335.271827 secs 2024-04-26T22:29:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38335.271827 secs 2024-04-26T22:29:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:56 smithi194 ceph-mon[33017]: pgmap v20663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:29:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:56 smithi033 ceph-mon[27845]: pgmap v20663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:29:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:56 smithi096 ceph-mon[32753]: pgmap v20663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:29:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:58 smithi194 ceph-mon[33017]: pgmap v20664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:29:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:29:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38337.290512 secs 2024-04-26T22:29:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:58 smithi033 ceph-mon[27845]: pgmap v20664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:29:59.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:29:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38337.290512 secs 2024-04-26T22:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:58 smithi096 ceph-mon[32753]: pgmap v20664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:29:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:29:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38337.290512 secs 2024-04-26T22:30:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:01.079+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:01.079+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:01.079+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:30:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38340.271944 secs 2024-04-26T22:30:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:00 smithi194 ceph-mon[33017]: pgmap v20665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38340.271944 secs 2024-04-26T22:30:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:00 smithi033 ceph-mon[27845]: pgmap v20665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:30:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38340.271944 secs 2024-04-26T22:30:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:00 smithi096 ceph-mon[32753]: pgmap v20665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:01.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:30:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:02 smithi194 ceph-mon[33017]: pgmap v20666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:02 smithi033 ceph-mon[27845]: pgmap v20666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:02 smithi096 ceph-mon[32753]: pgmap v20666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38342.290598 secs 2024-04-26T22:30:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38342.290598 secs 2024-04-26T22:30:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38342.290598 secs 2024-04-26T22:30:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:04 smithi194 ceph-mon[33017]: pgmap v20667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:04 smithi033 ceph-mon[27845]: pgmap v20667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:04 smithi096 ceph-mon[32753]: pgmap v20667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38345.272063 secs 2024-04-26T22:30:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38345.272063 secs 2024-04-26T22:30:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38345.272063 secs 2024-04-26T22:30:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:06 smithi194 ceph-mon[33017]: pgmap v20668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:06 smithi033 ceph-mon[27845]: pgmap v20668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:06 smithi096 ceph-mon[32753]: pgmap v20668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:08 smithi194 ceph-mon[33017]: pgmap v20669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38347.290685 secs 2024-04-26T22:30:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:08 smithi033 ceph-mon[27845]: pgmap v20669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38347.290685 secs 2024-04-26T22:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:08 smithi096 ceph-mon[32753]: pgmap v20669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38347.290685 secs 2024-04-26T22:30:10.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38350.272178 secs 2024-04-26T22:30:10.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38350.272178 secs 2024-04-26T22:30:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38350.272178 secs 2024-04-26T22:30:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:10 smithi194 ceph-mon[33017]: pgmap v20670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:11.116 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:10 smithi033 ceph-mon[27845]: pgmap v20670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:10 smithi096 ceph-mon[32753]: pgmap v20670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:13.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:12 smithi033 ceph-mon[27845]: pgmap v20671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:12 smithi194 ceph-mon[33017]: pgmap v20671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:12 smithi096 ceph-mon[32753]: pgmap v20671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:14.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38352.290768 secs 2024-04-26T22:30:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38352.290768 secs 2024-04-26T22:30:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38352.290768 secs 2024-04-26T22:30:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:14 smithi194 ceph-mon[33017]: pgmap v20672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38355.272279 secs 2024-04-26T22:30:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:14 smithi033 ceph-mon[27845]: pgmap v20672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38355.272279 secs 2024-04-26T22:30:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:14 smithi096 ceph-mon[32753]: pgmap v20672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38355.272279 secs 2024-04-26T22:30:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:16.094+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:16.094+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:30:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:16.094+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:16 smithi194 ceph-mon[33017]: pgmap v20673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:16 smithi033 ceph-mon[27845]: pgmap v20673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:16 smithi096 ceph-mon[32753]: pgmap v20673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:18 smithi194 ceph-mon[33017]: pgmap v20674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38357.290854 secs 2024-04-26T22:30:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:18 smithi033 ceph-mon[27845]: pgmap v20674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38357.290854 secs 2024-04-26T22:30:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:18 smithi096 ceph-mon[32753]: pgmap v20674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38357.290854 secs 2024-04-26T22:30:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38360.272344 secs 2024-04-26T22:30:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38360.272344 secs 2024-04-26T22:30:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38360.272344 secs 2024-04-26T22:30:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:20 smithi194 ceph-mon[33017]: pgmap v20675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:20 smithi033 ceph-mon[27845]: pgmap v20675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:20 smithi096 ceph-mon[32753]: pgmap v20675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:22 smithi194 ceph-mon[33017]: pgmap v20676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:22 smithi033 ceph-mon[27845]: pgmap v20676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:22 smithi096 ceph-mon[32753]: pgmap v20676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38362.290939 secs 2024-04-26T22:30:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38362.290939 secs 2024-04-26T22:30:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38362.290939 secs 2024-04-26T22:30:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:24 smithi194 ceph-mon[33017]: pgmap v20677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:25.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38365.272457 secs 2024-04-26T22:30:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:24 smithi033 ceph-mon[27845]: pgmap v20677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38365.272457 secs 2024-04-26T22:30:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:24 smithi096 ceph-mon[32753]: pgmap v20677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38365.272457 secs 2024-04-26T22:30:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:26 smithi194 ceph-mon[33017]: pgmap v20678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:26 smithi033 ceph-mon[27845]: pgmap v20678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:26 smithi096 ceph-mon[32753]: pgmap v20678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:28 smithi194 ceph-mon[33017]: pgmap v20679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:30:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38367.291028 secs 2024-04-26T22:30:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:28 smithi033 ceph-mon[27845]: pgmap v20679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:30:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38367.291028 secs 2024-04-26T22:30:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:28 smithi096 ceph-mon[32753]: pgmap v20679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:30:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38367.291028 secs 2024-04-26T22:30:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38370.272572 secs 2024-04-26T22:30:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38370.272572 secs 2024-04-26T22:30:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38370.272572 secs 2024-04-26T22:30:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:30 smithi194 ceph-mon[33017]: pgmap v20680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:31.108+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:31.108+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:31.108+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:30:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:30 smithi033 ceph-mon[27845]: pgmap v20680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:30 smithi096 ceph-mon[32753]: pgmap v20680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:32 smithi194 ceph-mon[33017]: pgmap v20681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:30:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:32 smithi033 ceph-mon[27845]: pgmap v20681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:30:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:32 smithi096 ceph-mon[32753]: pgmap v20681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:30:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38372.291115 secs 2024-04-26T22:30:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38372.291115 secs 2024-04-26T22:30:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38372.291115 secs 2024-04-26T22:30:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:34 smithi194 ceph-mon[33017]: pgmap v20682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:30:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38375.272633 secs 2024-04-26T22:30:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:34 smithi033 ceph-mon[27845]: pgmap v20682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:30:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38375.272633 secs 2024-04-26T22:30:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:34 smithi096 ceph-mon[32753]: pgmap v20682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:30:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38375.272633 secs 2024-04-26T22:30:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:36 smithi194 ceph-mon[33017]: pgmap v20683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:36 smithi033 ceph-mon[27845]: pgmap v20683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:36 smithi096 ceph-mon[32753]: pgmap v20683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:30:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:38 smithi194 ceph-mon[33017]: pgmap v20684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:39.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38377.291178 secs 2024-04-26T22:30:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:38 smithi033 ceph-mon[27845]: pgmap v20684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38377.291178 secs 2024-04-26T22:30:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:38 smithi096 ceph-mon[32753]: pgmap v20684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38377.291178 secs 2024-04-26T22:30:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38380.272764 secs 2024-04-26T22:30:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38380.272764 secs 2024-04-26T22:30:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38380.272764 secs 2024-04-26T22:30:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:40 smithi033 ceph-mon[27845]: pgmap v20685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:40 smithi096 ceph-mon[32753]: pgmap v20685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:40 smithi194 ceph-mon[33017]: pgmap v20685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:42 smithi033 ceph-mon[27845]: pgmap v20686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:42 smithi096 ceph-mon[32753]: pgmap v20686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:42 smithi194 ceph-mon[33017]: pgmap v20686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38382.291261 secs 2024-04-26T22:30:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38382.291261 secs 2024-04-26T22:30:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38382.291261 secs 2024-04-26T22:30:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:44 smithi033 ceph-mon[27845]: pgmap v20687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38385.272889 secs 2024-04-26T22:30:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:44 smithi096 ceph-mon[32753]: pgmap v20687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38385.272889 secs 2024-04-26T22:30:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:44 smithi194 ceph-mon[33017]: pgmap v20687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38385.272889 secs 2024-04-26T22:30:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:46.110+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:46.110+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:30:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:30:46.110+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:30:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:46 smithi033 ceph-mon[27845]: pgmap v20688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:46 smithi096 ceph-mon[32753]: pgmap v20688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:46 smithi194 ceph-mon[33017]: pgmap v20688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:30:48.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:47 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:30:48.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:47 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:30:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:47 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:30:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:48 smithi033 ceph-mon[27845]: pgmap v20689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38387.291359 secs 2024-04-26T22:30:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:48 smithi096 ceph-mon[32753]: pgmap v20689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38387.291359 secs 2024-04-26T22:30:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:48 smithi194 ceph-mon[33017]: pgmap v20689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38387.291359 secs 2024-04-26T22:30:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38390.273029 secs 2024-04-26T22:30:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:30:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:30:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:49 smithi033 ceph-mon[27845]: pgmap v20690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:49 smithi033 ceph-mon[27845]: pgmap v20691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:30:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38390.273029 secs 2024-04-26T22:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:49 smithi096 ceph-mon[32753]: pgmap v20690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:49 smithi096 ceph-mon[32753]: pgmap v20691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:30:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:30:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38390.273029 secs 2024-04-26T22:30:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:30:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:30:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:49 smithi194 ceph-mon[33017]: pgmap v20690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:30:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:49 smithi194 ceph-mon[33017]: pgmap v20691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:30:50.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:30:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:52 smithi033 ceph-mon[27845]: pgmap v20692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:52 smithi096 ceph-mon[32753]: pgmap v20692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:52 smithi194 ceph-mon[33017]: pgmap v20692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38392.291443 secs 2024-04-26T22:30:54.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:54 smithi033 ceph-mon[27845]: pgmap v20693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38392.291443 secs 2024-04-26T22:30:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:54 smithi096 ceph-mon[32753]: pgmap v20693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38392.291443 secs 2024-04-26T22:30:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:54 smithi194 ceph-mon[33017]: pgmap v20693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38395.273157 secs 2024-04-26T22:30:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38395.273157 secs 2024-04-26T22:30:55.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38395.273157 secs 2024-04-26T22:30:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:56 smithi033 ceph-mon[27845]: pgmap v20694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:56 smithi096 ceph-mon[32753]: pgmap v20694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:56 smithi194 ceph-mon[33017]: pgmap v20694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:58 smithi033 ceph-mon[27845]: pgmap v20695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:58 smithi096 ceph-mon[32753]: pgmap v20695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:58 smithi194 ceph-mon[33017]: pgmap v20695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:30:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:30:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38397.291529 secs 2024-04-26T22:30:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:30:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38397.291529 secs 2024-04-26T22:30:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:30:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38397.291529 secs 2024-04-26T22:31:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38400.273205 secs 2024-04-26T22:31:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:00 smithi194 ceph-mon[33017]: pgmap v20696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:31:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38400.273205 secs 2024-04-26T22:31:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:00 smithi033 ceph-mon[27845]: pgmap v20696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:31:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38400.273205 secs 2024-04-26T22:31:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:00 smithi096 ceph-mon[32753]: pgmap v20696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:31:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:01.122+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:01.122+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:31:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:01.122+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:02 smithi194 ceph-mon[33017]: pgmap v20697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:02 smithi033 ceph-mon[27845]: pgmap v20697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:02 smithi096 ceph-mon[32753]: pgmap v20697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:04.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38402.291615 secs 2024-04-26T22:31:04.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:04 smithi096 ceph-mon[32753]: pgmap v20698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38402.291615 secs 2024-04-26T22:31:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:04 smithi194 ceph-mon[33017]: pgmap v20698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38402.291615 secs 2024-04-26T22:31:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:04 smithi033 ceph-mon[27845]: pgmap v20698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38405.273295 secs 2024-04-26T22:31:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38405.273295 secs 2024-04-26T22:31:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38405.273295 secs 2024-04-26T22:31:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:06 smithi194 ceph-mon[33017]: pgmap v20699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:06 smithi033 ceph-mon[27845]: pgmap v20699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:06 smithi096 ceph-mon[32753]: pgmap v20699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:08 smithi194 ceph-mon[33017]: pgmap v20700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:08 smithi033 ceph-mon[27845]: pgmap v20700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:08 smithi096 ceph-mon[32753]: pgmap v20700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38407.291697 secs 2024-04-26T22:31:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38407.291697 secs 2024-04-26T22:31:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38407.291697 secs 2024-04-26T22:31:10.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38410.273362 secs 2024-04-26T22:31:10.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:10 smithi194 ceph-mon[33017]: pgmap v20701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:10.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38410.273362 secs 2024-04-26T22:31:10.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:10 smithi033 ceph-mon[27845]: pgmap v20701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38410.273362 secs 2024-04-26T22:31:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:10 smithi096 ceph-mon[32753]: pgmap v20701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:12.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:12 smithi194 ceph-mon[33017]: pgmap v20702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:12 smithi033 ceph-mon[27845]: pgmap v20702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:12 smithi096 ceph-mon[32753]: pgmap v20702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38412.291782 secs 2024-04-26T22:31:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:14 smithi194 ceph-mon[33017]: pgmap v20703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38412.291782 secs 2024-04-26T22:31:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:14 smithi033 ceph-mon[27845]: pgmap v20703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38412.291782 secs 2024-04-26T22:31:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:14 smithi096 ceph-mon[32753]: pgmap v20703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38415.273479 secs 2024-04-26T22:31:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38415.273479 secs 2024-04-26T22:31:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38415.273479 secs 2024-04-26T22:31:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:16.127+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:16.127+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:16.127+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:31:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:16 smithi194 ceph-mon[33017]: pgmap v20704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:16 smithi033 ceph-mon[27845]: pgmap v20704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:16 smithi096 ceph-mon[32753]: pgmap v20704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:17 smithi194 ceph-mon[33017]: pgmap v20705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:17 smithi033 ceph-mon[27845]: pgmap v20705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:17 smithi096 ceph-mon[32753]: pgmap v20705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38417.291867 secs 2024-04-26T22:31:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38417.291867 secs 2024-04-26T22:31:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38417.291867 secs 2024-04-26T22:31:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:19 smithi033 ceph-mon[27845]: pgmap v20706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:19 smithi096 ceph-mon[32753]: pgmap v20706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:19 smithi194 ceph-mon[33017]: pgmap v20706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38420.273604 secs 2024-04-26T22:31:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38420.273604 secs 2024-04-26T22:31:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38420.273604 secs 2024-04-26T22:31:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:21 smithi033 ceph-mon[27845]: pgmap v20707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:21.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:21 smithi096 ceph-mon[32753]: pgmap v20707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:22.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:21 smithi194 ceph-mon[33017]: pgmap v20707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38422.291949 secs 2024-04-26T22:31:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:24 smithi033 ceph-mon[27845]: pgmap v20708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38422.291949 secs 2024-04-26T22:31:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:24 smithi096 ceph-mon[32753]: pgmap v20708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38422.291949 secs 2024-04-26T22:31:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:24 smithi194 ceph-mon[33017]: pgmap v20708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38425.273725 secs 2024-04-26T22:31:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38425.273725 secs 2024-04-26T22:31:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38425.273725 secs 2024-04-26T22:31:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:26 smithi096 ceph-mon[32753]: pgmap v20709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:26 smithi194 ceph-mon[33017]: pgmap v20709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:26 smithi033 ceph-mon[27845]: pgmap v20709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:28 smithi194 ceph-mon[33017]: pgmap v20710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:28 smithi033 ceph-mon[27845]: pgmap v20710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:28 smithi096 ceph-mon[32753]: pgmap v20710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38427.292028 secs 2024-04-26T22:31:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38427.292028 secs 2024-04-26T22:31:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38427.292028 secs 2024-04-26T22:31:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38430.273785 secs 2024-04-26T22:31:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:30 smithi194 ceph-mon[33017]: pgmap v20711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38430.273785 secs 2024-04-26T22:31:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:30 smithi033 ceph-mon[27845]: pgmap v20711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38430.273785 secs 2024-04-26T22:31:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:30 smithi096 ceph-mon[32753]: pgmap v20711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:31.138+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:31.138+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:31.138+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:31:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:32 smithi194 ceph-mon[33017]: pgmap v20712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:32 smithi033 ceph-mon[27845]: pgmap v20712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:32 smithi096 ceph-mon[32753]: pgmap v20712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:34.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38432.292121 secs 2024-04-26T22:31:34.484 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:34 smithi096 ceph-mon[32753]: pgmap v20713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38432.292121 secs 2024-04-26T22:31:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:34 smithi194 ceph-mon[33017]: pgmap v20713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38432.292121 secs 2024-04-26T22:31:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:34 smithi033 ceph-mon[27845]: pgmap v20713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38435.273896 secs 2024-04-26T22:31:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38435.273896 secs 2024-04-26T22:31:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38435.273896 secs 2024-04-26T22:31:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:36 smithi194 ceph-mon[33017]: pgmap v20714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:36 smithi033 ceph-mon[27845]: pgmap v20714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:36 smithi096 ceph-mon[32753]: pgmap v20714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:38 smithi194 ceph-mon[33017]: pgmap v20715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:38 smithi033 ceph-mon[27845]: pgmap v20715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:38 smithi096 ceph-mon[32753]: pgmap v20715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38437.292207 secs 2024-04-26T22:31:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38437.292207 secs 2024-04-26T22:31:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38437.292207 secs 2024-04-26T22:31:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38440.274026 secs 2024-04-26T22:31:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:40 smithi194 ceph-mon[33017]: pgmap v20716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38440.274026 secs 2024-04-26T22:31:40.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:40 smithi033 ceph-mon[27845]: pgmap v20716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38440.274026 secs 2024-04-26T22:31:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:40 smithi096 ceph-mon[32753]: pgmap v20716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:42 smithi194 ceph-mon[33017]: pgmap v20717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:42.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:42 smithi033 ceph-mon[27845]: pgmap v20717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:42 smithi096 ceph-mon[32753]: pgmap v20717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:43 smithi194 ceph-mon[33017]: pgmap v20718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:43 smithi033 ceph-mon[27845]: pgmap v20718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:43 smithi096 ceph-mon[32753]: pgmap v20718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:44.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38442.292295 secs 2024-04-26T22:31:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38442.292295 secs 2024-04-26T22:31:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38442.292295 secs 2024-04-26T22:31:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38445.274163 secs 2024-04-26T22:31:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:45 smithi194 ceph-mon[33017]: pgmap v20719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38445.274163 secs 2024-04-26T22:31:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:45 smithi033 ceph-mon[27845]: pgmap v20719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38445.274163 secs 2024-04-26T22:31:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:45 smithi096 ceph-mon[32753]: pgmap v20719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:31:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:46.148+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:46.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:46.148+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:31:46.149 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:31:46.148+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:31:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:48 smithi096 ceph-mon[32753]: pgmap v20720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:48 smithi194 ceph-mon[33017]: pgmap v20720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:48 smithi033 ceph-mon[27845]: pgmap v20720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38447.292377 secs 2024-04-26T22:31:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38447.292377 secs 2024-04-26T22:31:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38447.292377 secs 2024-04-26T22:31:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38450.274200 secs 2024-04-26T22:31:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:50 smithi194 ceph-mon[33017]: pgmap v20721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:50.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:50 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:31:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38450.274200 secs 2024-04-26T22:31:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:50 smithi033 ceph-mon[27845]: pgmap v20721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:50 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:31:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38450.274200 secs 2024-04-26T22:31:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:50 smithi096 ceph-mon[32753]: pgmap v20721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:31:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:50 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:31:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:31:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:31:51.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:31:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:31:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:31:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:31:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:31:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:31:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:31:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:52 smithi194 ceph-mon[33017]: pgmap v20722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:31:52.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:52 smithi194 ceph-mon[33017]: pgmap v20723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:52 smithi033 ceph-mon[27845]: pgmap v20722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:31:52.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:52 smithi033 ceph-mon[27845]: pgmap v20723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:52 smithi096 ceph-mon[32753]: pgmap v20722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:31:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:52 smithi096 ceph-mon[32753]: pgmap v20723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:54 smithi194 ceph-mon[33017]: pgmap v20724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:54.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38452.292465 secs 2024-04-26T22:31:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:54 smithi033 ceph-mon[27845]: pgmap v20724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38452.292465 secs 2024-04-26T22:31:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:54 smithi096 ceph-mon[32753]: pgmap v20724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38452.292465 secs 2024-04-26T22:31:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38455.274291 secs 2024-04-26T22:31:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38455.274291 secs 2024-04-26T22:31:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38455.274291 secs 2024-04-26T22:31:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:56 smithi194 ceph-mon[33017]: pgmap v20725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:56 smithi033 ceph-mon[27845]: pgmap v20725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:56 smithi096 ceph-mon[32753]: pgmap v20725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:58 smithi194 ceph-mon[33017]: pgmap v20726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:58 smithi033 ceph-mon[27845]: pgmap v20726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:58 smithi096 ceph-mon[32753]: pgmap v20726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:31:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:31:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38457.292553 secs 2024-04-26T22:31:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:31:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38457.292553 secs 2024-04-26T22:31:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:31:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38457.292553 secs 2024-04-26T22:32:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:00 smithi033 ceph-mon[27845]: pgmap v20727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38460.274365 secs 2024-04-26T22:32:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:00 smithi194 ceph-mon[33017]: pgmap v20727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38460.274365 secs 2024-04-26T22:32:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:00 smithi096 ceph-mon[32753]: pgmap v20727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38460.274365 secs 2024-04-26T22:32:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:01.152+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:01.152+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:01.152+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:32:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:02 smithi194 ceph-mon[33017]: pgmap v20728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:32:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:02 smithi033 ceph-mon[27845]: pgmap v20728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:32:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:02 smithi096 ceph-mon[32753]: pgmap v20728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:32:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:04 smithi194 ceph-mon[33017]: pgmap v20729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38462.292642 secs 2024-04-26T22:32:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:04 smithi033 ceph-mon[27845]: pgmap v20729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38462.292642 secs 2024-04-26T22:32:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:04 smithi096 ceph-mon[32753]: pgmap v20729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38462.292642 secs 2024-04-26T22:32:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38465.274480 secs 2024-04-26T22:32:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:05 smithi194 ceph-mon[33017]: pgmap v20730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38465.274480 secs 2024-04-26T22:32:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:05 smithi033 ceph-mon[27845]: pgmap v20730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38465.274480 secs 2024-04-26T22:32:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:05 smithi096 ceph-mon[32753]: pgmap v20730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:08.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:07 smithi033 ceph-mon[27845]: pgmap v20731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:08.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:07 smithi096 ceph-mon[32753]: pgmap v20731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:07 smithi194 ceph-mon[33017]: pgmap v20731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38467.292731 secs 2024-04-26T22:32:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38467.292731 secs 2024-04-26T22:32:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38467.292731 secs 2024-04-26T22:32:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:09 smithi033 ceph-mon[27845]: pgmap v20732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38470.274798 secs 2024-04-26T22:32:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:09 smithi096 ceph-mon[32753]: pgmap v20732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38470.274798 secs 2024-04-26T22:32:10.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:09 smithi194 ceph-mon[33017]: pgmap v20732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:10.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38470.274798 secs 2024-04-26T22:32:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:11 smithi033 ceph-mon[27845]: pgmap v20733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:11 smithi096 ceph-mon[32753]: pgmap v20733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:11 smithi194 ceph-mon[33017]: pgmap v20733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:13 smithi033 ceph-mon[27845]: pgmap v20734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38472.292814 secs 2024-04-26T22:32:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:13 smithi096 ceph-mon[32753]: pgmap v20734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38472.292814 secs 2024-04-26T22:32:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:13 smithi194 ceph-mon[33017]: pgmap v20734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38472.292814 secs 2024-04-26T22:32:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38475.274939 secs 2024-04-26T22:32:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38475.274939 secs 2024-04-26T22:32:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38475.274939 secs 2024-04-26T22:32:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:16.167+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:16.167+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:32:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:16.167+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:15 smithi033 ceph-mon[27845]: pgmap v20735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:15 smithi096 ceph-mon[32753]: pgmap v20735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:15 smithi194 ceph-mon[33017]: pgmap v20735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:17 smithi096 ceph-mon[32753]: pgmap v20736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:17 smithi194 ceph-mon[33017]: pgmap v20736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:17 smithi033 ceph-mon[27845]: pgmap v20736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38477.292929 secs 2024-04-26T22:32:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38477.292929 secs 2024-04-26T22:32:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38477.292929 secs 2024-04-26T22:32:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:19 smithi194 ceph-mon[33017]: pgmap v20737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:20.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38480.275065 secs 2024-04-26T22:32:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:19 smithi033 ceph-mon[27845]: pgmap v20737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38480.275065 secs 2024-04-26T22:32:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:19 smithi096 ceph-mon[32753]: pgmap v20737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38480.275065 secs 2024-04-26T22:32:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:21 smithi194 ceph-mon[33017]: pgmap v20738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:21 smithi033 ceph-mon[27845]: pgmap v20738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:21 smithi096 ceph-mon[32753]: pgmap v20738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:23 smithi194 ceph-mon[33017]: pgmap v20739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38482.293018 secs 2024-04-26T22:32:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:23 smithi033 ceph-mon[27845]: pgmap v20739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38482.293018 secs 2024-04-26T22:32:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:23 smithi096 ceph-mon[32753]: pgmap v20739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38482.293018 secs 2024-04-26T22:32:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38485.275196 secs 2024-04-26T22:32:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38485.275196 secs 2024-04-26T22:32:25.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38485.275196 secs 2024-04-26T22:32:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:25 smithi194 ceph-mon[33017]: pgmap v20740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:25 smithi033 ceph-mon[27845]: pgmap v20740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:25 smithi096 ceph-mon[32753]: pgmap v20740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:28.351+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:32:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:28.351+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:32:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:28.351+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 300 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:32:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:28.351+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 300 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:32:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:27 smithi194 ceph-mon[33017]: pgmap v20741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:27 smithi033 ceph-mon[27845]: pgmap v20741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:27 smithi096 ceph-mon[32753]: pgmap v20741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38487.293084 secs 2024-04-26T22:32:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38487.293084 secs 2024-04-26T22:32:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38487.293084 secs 2024-04-26T22:32:30.256 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:29 smithi033 ceph-mon[27845]: pgmap v20742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:30.256 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38490.275348 secs 2024-04-26T22:32:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:29 smithi194 ceph-mon[33017]: pgmap v20742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38490.275348 secs 2024-04-26T22:32:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:29 smithi096 ceph-mon[32753]: pgmap v20742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:32:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38490.275348 secs 2024-04-26T22:32:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:31.171+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:31.171+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:31.171+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:32:32.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:31 smithi194 ceph-mon[33017]: pgmap v20743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:31 smithi033 ceph-mon[27845]: pgmap v20743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:31 smithi096 ceph-mon[32753]: pgmap v20743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:32:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:33 smithi194 ceph-mon[33017]: pgmap v20744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38492.293172 secs 2024-04-26T22:32:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:33 smithi033 ceph-mon[27845]: pgmap v20744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38492.293172 secs 2024-04-26T22:32:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:33 smithi096 ceph-mon[32753]: pgmap v20744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38492.293172 secs 2024-04-26T22:32:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38495.275477 secs 2024-04-26T22:32:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38495.275477 secs 2024-04-26T22:32:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38495.275477 secs 2024-04-26T22:32:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:36 smithi194 ceph-mon[33017]: pgmap v20745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:36 smithi033 ceph-mon[27845]: pgmap v20745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:36 smithi096 ceph-mon[32753]: pgmap v20745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:38.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:38 smithi194 ceph-mon[33017]: pgmap v20746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:38 smithi033 ceph-mon[27845]: pgmap v20746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:38 smithi096 ceph-mon[32753]: pgmap v20746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38497.293296 secs 2024-04-26T22:32:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38497.293296 secs 2024-04-26T22:32:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38497.293296 secs 2024-04-26T22:32:40.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:40 smithi033 ceph-mon[27845]: pgmap v20747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:40.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38500.275593 secs 2024-04-26T22:32:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:40 smithi194 ceph-mon[33017]: pgmap v20747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38500.275593 secs 2024-04-26T22:32:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:40 smithi096 ceph-mon[32753]: pgmap v20747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38500.275593 secs 2024-04-26T22:32:42.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:42 smithi194 ceph-mon[33017]: pgmap v20748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:42 smithi033 ceph-mon[27845]: pgmap v20748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:42.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:42 smithi096 ceph-mon[32753]: pgmap v20748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:44 smithi194 ceph-mon[33017]: pgmap v20749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38502.293365 secs 2024-04-26T22:32:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:44 smithi033 ceph-mon[27845]: pgmap v20749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38502.293365 secs 2024-04-26T22:32:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:44 smithi096 ceph-mon[32753]: pgmap v20749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38502.293365 secs 2024-04-26T22:32:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38505.275718 secs 2024-04-26T22:32:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38505.275718 secs 2024-04-26T22:32:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38505.275718 secs 2024-04-26T22:32:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:46.182+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:46.182+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:32:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:32:46.182+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:32:46.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:46 smithi033 ceph-mon[27845]: pgmap v20750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:46 smithi194 ceph-mon[33017]: pgmap v20750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:46 smithi096 ceph-mon[32753]: pgmap v20750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:48 smithi194 ceph-mon[33017]: pgmap v20751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:48 smithi033 ceph-mon[27845]: pgmap v20751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:48 smithi096 ceph-mon[32753]: pgmap v20751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38507.293427 secs 2024-04-26T22:32:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38507.293427 secs 2024-04-26T22:32:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38507.293427 secs 2024-04-26T22:32:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38510.275843 secs 2024-04-26T22:32:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:50 smithi194 ceph-mon[33017]: pgmap v20752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38510.275843 secs 2024-04-26T22:32:50.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:50 smithi033 ceph-mon[27845]: pgmap v20752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38510.275843 secs 2024-04-26T22:32:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:50 smithi096 ceph-mon[32753]: pgmap v20752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:52 smithi194 ceph-mon[33017]: pgmap v20753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:32:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:52 smithi033 ceph-mon[27845]: pgmap v20753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:52.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:32:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:52 smithi096 ceph-mon[32753]: pgmap v20753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:32:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:32:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:32:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:32:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:32:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:32:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:32:53.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:32:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:32:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:32:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:32:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:54 smithi194 ceph-mon[33017]: pgmap v20754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:54 smithi194 ceph-mon[33017]: pgmap v20755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38512.293514 secs 2024-04-26T22:32:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:54 smithi033 ceph-mon[27845]: pgmap v20754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:54 smithi033 ceph-mon[27845]: pgmap v20755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38512.293514 secs 2024-04-26T22:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:54 smithi096 ceph-mon[32753]: pgmap v20754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:54 smithi096 ceph-mon[32753]: pgmap v20755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38512.293514 secs 2024-04-26T22:32:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38515.275967 secs 2024-04-26T22:32:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38515.275967 secs 2024-04-26T22:32:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38515.275967 secs 2024-04-26T22:32:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:56 smithi194 ceph-mon[33017]: pgmap v20756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:56 smithi033 ceph-mon[27845]: pgmap v20756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:56 smithi096 ceph-mon[32753]: pgmap v20756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:58 smithi194 ceph-mon[33017]: pgmap v20757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:58 smithi033 ceph-mon[27845]: pgmap v20757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:58 smithi096 ceph-mon[32753]: pgmap v20757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:32:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:32:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38517.293598 secs 2024-04-26T22:32:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:32:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38517.293598 secs 2024-04-26T22:32:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:32:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38517.293598 secs 2024-04-26T22:33:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:00 smithi033 ceph-mon[27845]: pgmap v20758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38520.276087 secs 2024-04-26T22:33:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:00 smithi096 ceph-mon[32753]: pgmap v20758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38520.276087 secs 2024-04-26T22:33:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:00 smithi194 ceph-mon[33017]: pgmap v20758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38520.276087 secs 2024-04-26T22:33:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:01.188+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:01.188+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:01.188+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:33:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:02 smithi033 ceph-mon[27845]: pgmap v20759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:02 smithi096 ceph-mon[32753]: pgmap v20759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:02 smithi194 ceph-mon[33017]: pgmap v20759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:04 smithi033 ceph-mon[27845]: pgmap v20760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:33:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38522.293679 secs 2024-04-26T22:33:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:04 smithi096 ceph-mon[32753]: pgmap v20760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:33:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38522.293679 secs 2024-04-26T22:33:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:04 smithi194 ceph-mon[33017]: pgmap v20760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:33:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38522.293679 secs 2024-04-26T22:33:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38525.276209 secs 2024-04-26T22:33:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38525.276209 secs 2024-04-26T22:33:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38525.276209 secs 2024-04-26T22:33:06.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:06 smithi033 ceph-mon[27845]: pgmap v20761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:06 smithi096 ceph-mon[32753]: pgmap v20761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:06 smithi194 ceph-mon[33017]: pgmap v20761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:08 smithi033 ceph-mon[27845]: pgmap v20762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:08 smithi096 ceph-mon[32753]: pgmap v20762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:08 smithi194 ceph-mon[33017]: pgmap v20762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38527.293760 secs 2024-04-26T22:33:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38527.293760 secs 2024-04-26T22:33:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38527.293760 secs 2024-04-26T22:33:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:10 smithi033 ceph-mon[27845]: pgmap v20763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38530.276347 secs 2024-04-26T22:33:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:10 smithi096 ceph-mon[32753]: pgmap v20763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38530.276347 secs 2024-04-26T22:33:10.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:10 smithi194 ceph-mon[33017]: pgmap v20763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:10.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38530.276347 secs 2024-04-26T22:33:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:12 smithi033 ceph-mon[27845]: pgmap v20764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:12 smithi096 ceph-mon[32753]: pgmap v20764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:12 smithi194 ceph-mon[33017]: pgmap v20764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:13.358+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:33:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:13.358+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:33:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:13.358+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 301 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:33:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:13.359+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 301 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:33:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:14 smithi033 ceph-mon[27845]: pgmap v20765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38532.293840 secs 2024-04-26T22:33:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:14 smithi096 ceph-mon[32753]: pgmap v20765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38532.293840 secs 2024-04-26T22:33:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:14 smithi194 ceph-mon[33017]: pgmap v20765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38532.293840 secs 2024-04-26T22:33:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38535.276471 secs 2024-04-26T22:33:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38535.276471 secs 2024-04-26T22:33:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38535.276471 secs 2024-04-26T22:33:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:16.203+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:16.203+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:16.203+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:33:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:16 smithi096 ceph-mon[32753]: pgmap v20766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:16 smithi194 ceph-mon[33017]: pgmap v20766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:16 smithi033 ceph-mon[27845]: pgmap v20766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:18 smithi194 ceph-mon[33017]: pgmap v20767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:18 smithi033 ceph-mon[27845]: pgmap v20767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:18 smithi096 ceph-mon[32753]: pgmap v20767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38537.293922 secs 2024-04-26T22:33:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38537.293922 secs 2024-04-26T22:33:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38537.293922 secs 2024-04-26T22:33:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:20 smithi194 ceph-mon[33017]: pgmap v20768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38540.276539 secs 2024-04-26T22:33:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:20 smithi033 ceph-mon[27845]: pgmap v20768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38540.276539 secs 2024-04-26T22:33:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:20 smithi096 ceph-mon[32753]: pgmap v20768: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38540.276539 secs 2024-04-26T22:33:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:22 smithi194 ceph-mon[33017]: pgmap v20769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:22 smithi033 ceph-mon[27845]: pgmap v20769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:22 smithi096 ceph-mon[32753]: pgmap v20769: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:24 smithi194 ceph-mon[33017]: pgmap v20770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:24.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38542.294025 secs 2024-04-26T22:33:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:24 smithi033 ceph-mon[27845]: pgmap v20770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38542.294025 secs 2024-04-26T22:33:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:24 smithi096 ceph-mon[32753]: pgmap v20770: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38542.294025 secs 2024-04-26T22:33:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38545.276621 secs 2024-04-26T22:33:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38545.276621 secs 2024-04-26T22:33:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38545.276621 secs 2024-04-26T22:33:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:26 smithi194 ceph-mon[33017]: pgmap v20771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:26 smithi033 ceph-mon[27845]: pgmap v20771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:26 smithi096 ceph-mon[32753]: pgmap v20771: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:28 smithi194 ceph-mon[33017]: pgmap v20772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:28 smithi033 ceph-mon[27845]: pgmap v20772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:28 smithi096 ceph-mon[32753]: pgmap v20772: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38547.294111 secs 2024-04-26T22:33:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38547.294111 secs 2024-04-26T22:33:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38547.294111 secs 2024-04-26T22:33:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:30 smithi194 ceph-mon[33017]: pgmap v20773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38550.276751 secs 2024-04-26T22:33:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:30 smithi033 ceph-mon[27845]: pgmap v20773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38550.276751 secs 2024-04-26T22:33:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:30 smithi096 ceph-mon[32753]: pgmap v20773: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38550.276751 secs 2024-04-26T22:33:31.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:31.218+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:31.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:31.218+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:31.220 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:31.218+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:33:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:32 smithi194 ceph-mon[33017]: pgmap v20774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:32 smithi033 ceph-mon[27845]: pgmap v20774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:32 smithi096 ceph-mon[32753]: pgmap v20774: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:34 smithi194 ceph-mon[33017]: pgmap v20775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38552.294203 secs 2024-04-26T22:33:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:34 smithi033 ceph-mon[27845]: pgmap v20775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38552.294203 secs 2024-04-26T22:33:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:34 smithi096 ceph-mon[32753]: pgmap v20775: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38552.294203 secs 2024-04-26T22:33:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38555.276880 secs 2024-04-26T22:33:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38555.276880 secs 2024-04-26T22:33:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38555.276880 secs 2024-04-26T22:33:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:36 smithi194 ceph-mon[33017]: pgmap v20776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:36 smithi033 ceph-mon[27845]: pgmap v20776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:36 smithi096 ceph-mon[32753]: pgmap v20776: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:38 smithi194 ceph-mon[33017]: pgmap v20777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:38 smithi033 ceph-mon[27845]: pgmap v20777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:38 smithi096 ceph-mon[32753]: pgmap v20777: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38557.294291 secs 2024-04-26T22:33:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38557.294291 secs 2024-04-26T22:33:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38557.294291 secs 2024-04-26T22:33:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:40 smithi033 ceph-mon[27845]: pgmap v20778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38560.277017 secs 2024-04-26T22:33:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:40 smithi194 ceph-mon[33017]: pgmap v20778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38560.277017 secs 2024-04-26T22:33:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:40 smithi096 ceph-mon[32753]: pgmap v20778: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38560.277017 secs 2024-04-26T22:33:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:41 smithi194 ceph-mon[33017]: pgmap v20779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:41 smithi033 ceph-mon[27845]: pgmap v20779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:41 smithi096 ceph-mon[32753]: pgmap v20779: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:43 smithi194 ceph-mon[33017]: pgmap v20780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:43 smithi033 ceph-mon[27845]: pgmap v20780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:43 smithi096 ceph-mon[32753]: pgmap v20780: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38562.294379 secs 2024-04-26T22:33:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38562.294379 secs 2024-04-26T22:33:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38562.294379 secs 2024-04-26T22:33:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38565.277080 secs 2024-04-26T22:33:45.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:45 smithi194 ceph-mon[33017]: pgmap v20781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38565.277080 secs 2024-04-26T22:33:45.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:45 smithi033 ceph-mon[27845]: pgmap v20781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38565.277080 secs 2024-04-26T22:33:45.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:45 smithi096 ceph-mon[32753]: pgmap v20781: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:46.226+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:46.226+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:33:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:33:46.226+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:33:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:47 smithi194 ceph-mon[33017]: pgmap v20782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:47 smithi033 ceph-mon[27845]: pgmap v20782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:47.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:47 smithi096 ceph-mon[32753]: pgmap v20782: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38567.294439 secs 2024-04-26T22:33:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:49 smithi194 ceph-mon[33017]: pgmap v20783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38567.294439 secs 2024-04-26T22:33:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:49 smithi033 ceph-mon[27845]: pgmap v20783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38567.294439 secs 2024-04-26T22:33:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:49 smithi096 ceph-mon[32753]: pgmap v20783: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38570.277207 secs 2024-04-26T22:33:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38570.277207 secs 2024-04-26T22:33:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38570.277207 secs 2024-04-26T22:33:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:51 smithi194 ceph-mon[33017]: pgmap v20784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:51 smithi033 ceph-mon[27845]: pgmap v20784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:51 smithi096 ceph-mon[32753]: pgmap v20784: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:33:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:53 smithi194 ceph-mon[33017]: pgmap v20785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:53.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:33:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:53 smithi033 ceph-mon[27845]: pgmap v20785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:53.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:33:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:53 smithi096 ceph-mon[32753]: pgmap v20785: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:53.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:33:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38572.294536 secs 2024-04-26T22:33:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:33:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:54 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:33:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:54 smithi194 ceph-mon[33017]: pgmap v20786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:54 smithi194 ceph-mon[33017]: pgmap v20787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:33:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:54 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:33:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38572.294536 secs 2024-04-26T22:33:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:33:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:54 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:33:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:54 smithi033 ceph-mon[27845]: pgmap v20786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:54 smithi033 ceph-mon[27845]: pgmap v20787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:33:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:54 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:33:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38572.294536 secs 2024-04-26T22:33:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:33:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:54 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:33:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:54 smithi096 ceph-mon[32753]: pgmap v20786: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:33:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:54 smithi096 ceph-mon[32753]: pgmap v20787: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:33:54.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:54 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:33:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38575.277322 secs 2024-04-26T22:33:55.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38575.277322 secs 2024-04-26T22:33:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38575.277322 secs 2024-04-26T22:33:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:56 smithi033 ceph-mon[27845]: pgmap v20788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:56.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:56 smithi096 ceph-mon[32753]: pgmap v20788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:56 smithi194 ceph-mon[33017]: pgmap v20788: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38577.294620 secs 2024-04-26T22:33:59.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:33:59 smithi194 ceph-mon[33017]: pgmap v20789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38577.294620 secs 2024-04-26T22:33:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:33:59 smithi033 ceph-mon[27845]: pgmap v20789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:33:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38577.294620 secs 2024-04-26T22:33:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:33:59 smithi096 ceph-mon[32753]: pgmap v20789: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38580.277412 secs 2024-04-26T22:34:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38580.277412 secs 2024-04-26T22:34:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38580.277412 secs 2024-04-26T22:34:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:01.241+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:01.241+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:01.241+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:34:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:01 smithi194 ceph-mon[33017]: pgmap v20790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:01 smithi033 ceph-mon[27845]: pgmap v20790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:01 smithi096 ceph-mon[32753]: pgmap v20790: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:03 smithi194 ceph-mon[33017]: pgmap v20791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:03 smithi033 ceph-mon[27845]: pgmap v20791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:03 smithi096 ceph-mon[32753]: pgmap v20791: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38582.294706 secs 2024-04-26T22:34:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38582.294706 secs 2024-04-26T22:34:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38582.294706 secs 2024-04-26T22:34:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38585.277536 secs 2024-04-26T22:34:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:05 smithi194 ceph-mon[33017]: pgmap v20792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:34:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38585.277536 secs 2024-04-26T22:34:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:05 smithi033 ceph-mon[27845]: pgmap v20792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:34:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38585.277536 secs 2024-04-26T22:34:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:05 smithi096 ceph-mon[32753]: pgmap v20792: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:34:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:07 smithi194 ceph-mon[33017]: pgmap v20793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:07 smithi033 ceph-mon[27845]: pgmap v20793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:07 smithi096 ceph-mon[32753]: pgmap v20793: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38587.294793 secs 2024-04-26T22:34:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:09 smithi194 ceph-mon[33017]: pgmap v20794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38587.294793 secs 2024-04-26T22:34:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:09 smithi033 ceph-mon[27845]: pgmap v20794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38587.294793 secs 2024-04-26T22:34:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:09 smithi096 ceph-mon[32753]: pgmap v20794: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:10.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38590.277662 secs 2024-04-26T22:34:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38590.277662 secs 2024-04-26T22:34:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38590.277662 secs 2024-04-26T22:34:11.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:11 smithi194 ceph-mon[33017]: pgmap v20795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:11 smithi033 ceph-mon[27845]: pgmap v20795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:11 smithi096 ceph-mon[32753]: pgmap v20795: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:13 smithi194 ceph-mon[33017]: pgmap v20796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:13 smithi033 ceph-mon[27845]: pgmap v20796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:13 smithi096 ceph-mon[32753]: pgmap v20796: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38592.294878 secs 2024-04-26T22:34:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:14 smithi194 ceph-mon[33017]: pgmap v20797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38592.294878 secs 2024-04-26T22:34:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:14 smithi033 ceph-mon[27845]: pgmap v20797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38592.294878 secs 2024-04-26T22:34:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:14 smithi096 ceph-mon[32753]: pgmap v20797: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38595.277788 secs 2024-04-26T22:34:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38595.277788 secs 2024-04-26T22:34:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38595.277788 secs 2024-04-26T22:34:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:16.255+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:16.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:16.255+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:16.257 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:16.255+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:34:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:16 smithi194 ceph-mon[33017]: pgmap v20798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:16 smithi033 ceph-mon[27845]: pgmap v20798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:16 smithi096 ceph-mon[32753]: pgmap v20798: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38597.294971 secs 2024-04-26T22:34:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:19 smithi194 ceph-mon[33017]: pgmap v20799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38597.294971 secs 2024-04-26T22:34:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:19 smithi033 ceph-mon[27845]: pgmap v20799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38597.294971 secs 2024-04-26T22:34:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:19 smithi096 ceph-mon[32753]: pgmap v20799: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38600.277909 secs 2024-04-26T22:34:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38600.277909 secs 2024-04-26T22:34:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38600.277909 secs 2024-04-26T22:34:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:21 smithi194 ceph-mon[33017]: pgmap v20800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:21 smithi033 ceph-mon[27845]: pgmap v20800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:21 smithi096 ceph-mon[32753]: pgmap v20800: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:23 smithi194 ceph-mon[33017]: pgmap v20801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:23 smithi033 ceph-mon[27845]: pgmap v20801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:23 smithi096 ceph-mon[32753]: pgmap v20801: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38602.295051 secs 2024-04-26T22:34:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38602.295051 secs 2024-04-26T22:34:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38602.295051 secs 2024-04-26T22:34:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38605.278025 secs 2024-04-26T22:34:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:25 smithi194 ceph-mon[33017]: pgmap v20802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38605.278025 secs 2024-04-26T22:34:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:25 smithi033 ceph-mon[27845]: pgmap v20802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38605.278025 secs 2024-04-26T22:34:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:25 smithi096 ceph-mon[32753]: pgmap v20802: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:27 smithi194 ceph-mon[33017]: pgmap v20803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:27 smithi033 ceph-mon[27845]: pgmap v20803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:27 smithi096 ceph-mon[32753]: pgmap v20803: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38607.295143 secs 2024-04-26T22:34:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:29 smithi194 ceph-mon[33017]: pgmap v20804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38607.295143 secs 2024-04-26T22:34:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:29 smithi033 ceph-mon[27845]: pgmap v20804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38607.295143 secs 2024-04-26T22:34:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:29 smithi096 ceph-mon[32753]: pgmap v20804: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38610.278155 secs 2024-04-26T22:34:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38610.278155 secs 2024-04-26T22:34:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38610.278155 secs 2024-04-26T22:34:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:31.270+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:31.270+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:31.270+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:34:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:31 smithi194 ceph-mon[33017]: pgmap v20805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:31 smithi033 ceph-mon[27845]: pgmap v20805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:31 smithi096 ceph-mon[32753]: pgmap v20805: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:33 smithi194 ceph-mon[33017]: pgmap v20806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:33 smithi033 ceph-mon[27845]: pgmap v20806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:33 smithi096 ceph-mon[32753]: pgmap v20806: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:34.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38612.295203 secs 2024-04-26T22:34:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38612.295203 secs 2024-04-26T22:34:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38612.295203 secs 2024-04-26T22:34:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38615.278283 secs 2024-04-26T22:34:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:35 smithi194 ceph-mon[33017]: pgmap v20807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38615.278283 secs 2024-04-26T22:34:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:35 smithi033 ceph-mon[27845]: pgmap v20807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38615.278283 secs 2024-04-26T22:34:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:35 smithi096 ceph-mon[32753]: pgmap v20807: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:36 smithi194 ceph-mon[33017]: pgmap v20808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:36.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:36 smithi033 ceph-mon[27845]: pgmap v20808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:36 smithi096 ceph-mon[32753]: pgmap v20808: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38617.295288 secs 2024-04-26T22:34:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:39 smithi194 ceph-mon[33017]: pgmap v20809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38617.295288 secs 2024-04-26T22:34:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:39 smithi033 ceph-mon[27845]: pgmap v20809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38617.295288 secs 2024-04-26T22:34:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:39 smithi096 ceph-mon[32753]: pgmap v20809: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38620.278340 secs 2024-04-26T22:34:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38620.278340 secs 2024-04-26T22:34:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38620.278340 secs 2024-04-26T22:34:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:41 smithi194 ceph-mon[33017]: pgmap v20810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:41 smithi033 ceph-mon[27845]: pgmap v20810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:41 smithi096 ceph-mon[32753]: pgmap v20810: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:43 smithi194 ceph-mon[33017]: pgmap v20811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:43 smithi033 ceph-mon[27845]: pgmap v20811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:43 smithi096 ceph-mon[32753]: pgmap v20811: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38622.295365 secs 2024-04-26T22:34:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38622.295365 secs 2024-04-26T22:34:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38622.295365 secs 2024-04-26T22:34:45.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38625.278462 secs 2024-04-26T22:34:45.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:45 smithi194 ceph-mon[33017]: pgmap v20812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38625.278462 secs 2024-04-26T22:34:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:45 smithi033 ceph-mon[27845]: pgmap v20812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38625.278462 secs 2024-04-26T22:34:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:45 smithi096 ceph-mon[32753]: pgmap v20812: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:46.287+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:46.287+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:34:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:34:46.287+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:34:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:47 smithi194 ceph-mon[33017]: pgmap v20813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:47 smithi033 ceph-mon[27845]: pgmap v20813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:47 smithi096 ceph-mon[32753]: pgmap v20813: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38627.295452 secs 2024-04-26T22:34:49.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:49 smithi194 ceph-mon[33017]: pgmap v20814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38627.295452 secs 2024-04-26T22:34:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:49 smithi033 ceph-mon[27845]: pgmap v20814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38627.295452 secs 2024-04-26T22:34:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:49 smithi096 ceph-mon[32753]: pgmap v20814: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38630.278578 secs 2024-04-26T22:34:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38630.278578 secs 2024-04-26T22:34:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38630.278578 secs 2024-04-26T22:34:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:51 smithi033 ceph-mon[27845]: pgmap v20815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:51 smithi194 ceph-mon[33017]: pgmap v20815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:51 smithi096 ceph-mon[32753]: pgmap v20815: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:53 smithi194 ceph-mon[33017]: pgmap v20816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:53 smithi033 ceph-mon[27845]: pgmap v20816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:53 smithi096 ceph-mon[32753]: pgmap v20816: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38632.295536 secs 2024-04-26T22:34:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38632.295536 secs 2024-04-26T22:34:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38632.295536 secs 2024-04-26T22:34:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38635.278722 secs 2024-04-26T22:34:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:55 smithi194 ceph-mon[33017]: pgmap v20817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:55 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:34:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38635.278722 secs 2024-04-26T22:34:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:55 smithi033 ceph-mon[27845]: pgmap v20817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:55 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:34:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38635.278722 secs 2024-04-26T22:34:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:55 smithi096 ceph-mon[32753]: pgmap v20817: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:55 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:34:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:56 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:56 smithi033 ceph-mon[27845]: pgmap v20818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:56 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:56 smithi096 ceph-mon[32753]: pgmap v20818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:56 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:56.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:56 smithi194 ceph-mon[33017]: pgmap v20818: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: pgmap v20819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: pgmap v20820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:58.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:34:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38637.295636 secs 2024-04-26T22:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:34:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: pgmap v20819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: pgmap v20820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:58.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:34:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38637.295636 secs 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: pgmap v20819: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: pgmap v20820: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:34:59.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:34:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38637.295636 secs 2024-04-26T22:35:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38640.278839 secs 2024-04-26T22:35:01.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:00 smithi194 ceph-mon[33017]: pgmap v20821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38640.278839 secs 2024-04-26T22:35:01.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:00 smithi033 ceph-mon[27845]: pgmap v20821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:01.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38640.278839 secs 2024-04-26T22:35:01.184 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:00 smithi096 ceph-mon[32753]: pgmap v20821: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:01.294+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:01.294+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:35:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:01.294+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:02 smithi194 ceph-mon[33017]: pgmap v20822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:03.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:02 smithi033 ceph-mon[27845]: pgmap v20822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:03.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:02 smithi096 ceph-mon[32753]: pgmap v20822: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:04.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38642.295731 secs 2024-04-26T22:35:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38642.295731 secs 2024-04-26T22:35:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38642.295731 secs 2024-04-26T22:35:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:04 smithi194 ceph-mon[33017]: pgmap v20823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:04 smithi033 ceph-mon[27845]: pgmap v20823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:05.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:04 smithi096 ceph-mon[32753]: pgmap v20823: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38645.278966 secs 2024-04-26T22:35:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38645.278966 secs 2024-04-26T22:35:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38645.278966 secs 2024-04-26T22:35:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:06 smithi194 ceph-mon[33017]: pgmap v20824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:06 smithi033 ceph-mon[27845]: pgmap v20824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:07.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:06 smithi096 ceph-mon[32753]: pgmap v20824: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:35:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:08 smithi194 ceph-mon[33017]: pgmap v20825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:35:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38647.295826 secs 2024-04-26T22:35:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:08 smithi033 ceph-mon[27845]: pgmap v20825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:35:09.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38647.295826 secs 2024-04-26T22:35:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:08 smithi096 ceph-mon[32753]: pgmap v20825: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:35:09.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38647.295826 secs 2024-04-26T22:35:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38650.279084 secs 2024-04-26T22:35:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:10 smithi194 ceph-mon[33017]: pgmap v20826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38650.279084 secs 2024-04-26T22:35:11.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:10 smithi033 ceph-mon[27845]: pgmap v20826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38650.279084 secs 2024-04-26T22:35:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:10 smithi096 ceph-mon[32753]: pgmap v20826: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:12 smithi194 ceph-mon[33017]: pgmap v20827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:12 smithi033 ceph-mon[27845]: pgmap v20827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:13.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:12 smithi096 ceph-mon[32753]: pgmap v20827: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38652.295920 secs 2024-04-26T22:35:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38652.295920 secs 2024-04-26T22:35:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38652.295920 secs 2024-04-26T22:35:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:14 smithi194 ceph-mon[33017]: pgmap v20828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:15.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38655.279158 secs 2024-04-26T22:35:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:14 smithi033 ceph-mon[27845]: pgmap v20828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:15.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38655.279158 secs 2024-04-26T22:35:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:14 smithi096 ceph-mon[32753]: pgmap v20828: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38655.279158 secs 2024-04-26T22:35:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:16.305+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:16.305+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:16.305+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:35:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:16 smithi194 ceph-mon[33017]: pgmap v20829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:16 smithi033 ceph-mon[27845]: pgmap v20829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:16 smithi096 ceph-mon[32753]: pgmap v20829: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:18 smithi194 ceph-mon[33017]: pgmap v20830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:19.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38657.296025 secs 2024-04-26T22:35:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:18 smithi033 ceph-mon[27845]: pgmap v20830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38657.296025 secs 2024-04-26T22:35:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:18 smithi096 ceph-mon[32753]: pgmap v20830: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38657.296025 secs 2024-04-26T22:35:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38660.279250 secs 2024-04-26T22:35:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38660.279250 secs 2024-04-26T22:35:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38660.279250 secs 2024-04-26T22:35:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:20 smithi194 ceph-mon[33017]: pgmap v20831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:20 smithi033 ceph-mon[27845]: pgmap v20831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:20 smithi096 ceph-mon[32753]: pgmap v20831: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:22 smithi194 ceph-mon[33017]: pgmap v20832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:22 smithi033 ceph-mon[27845]: pgmap v20832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:23.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:22 smithi096 ceph-mon[32753]: pgmap v20832: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:24.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38662.296122 secs 2024-04-26T22:35:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38662.296122 secs 2024-04-26T22:35:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38662.296122 secs 2024-04-26T22:35:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:24 smithi194 ceph-mon[33017]: pgmap v20833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38665.279353 secs 2024-04-26T22:35:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:24 smithi033 ceph-mon[27845]: pgmap v20833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:25.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38665.279353 secs 2024-04-26T22:35:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:24 smithi096 ceph-mon[32753]: pgmap v20833: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38665.279353 secs 2024-04-26T22:35:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:26 smithi194 ceph-mon[33017]: pgmap v20834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:26 smithi033 ceph-mon[27845]: pgmap v20834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:26 smithi096 ceph-mon[32753]: pgmap v20834: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:28.381 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:28.380+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:35:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:28.380+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:35:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:28.381+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 302 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T22:35:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:28.381+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 302 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T22:35:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:28 smithi194 ceph-mon[33017]: pgmap v20835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38667.296208 secs 2024-04-26T22:35:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:28 smithi033 ceph-mon[27845]: pgmap v20835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38667.296208 secs 2024-04-26T22:35:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:28 smithi096 ceph-mon[32753]: pgmap v20835: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38667.296208 secs 2024-04-26T22:35:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38670.279436 secs 2024-04-26T22:35:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38670.279436 secs 2024-04-26T22:35:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38670.279436 secs 2024-04-26T22:35:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:30 smithi194 ceph-mon[33017]: pgmap v20836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:30 smithi033 ceph-mon[27845]: pgmap v20836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:30 smithi096 ceph-mon[32753]: pgmap v20836: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:31.320+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:31.320+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:31.320+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:35:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:32 smithi033 ceph-mon[27845]: pgmap v20837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:33.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:32 smithi096 ceph-mon[32753]: pgmap v20837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:32 smithi194 ceph-mon[33017]: pgmap v20837: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38672.296283 secs 2024-04-26T22:35:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38672.296283 secs 2024-04-26T22:35:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38672.296283 secs 2024-04-26T22:35:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:34 smithi033 ceph-mon[27845]: pgmap v20838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:35.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38675.279556 secs 2024-04-26T22:35:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:34 smithi096 ceph-mon[32753]: pgmap v20838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:35.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38675.279556 secs 2024-04-26T22:35:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:34 smithi194 ceph-mon[33017]: pgmap v20838: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38675.279556 secs 2024-04-26T22:35:37.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:36 smithi033 ceph-mon[27845]: pgmap v20839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:36 smithi096 ceph-mon[32753]: pgmap v20839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:36 smithi194 ceph-mon[33017]: pgmap v20839: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:38 smithi033 ceph-mon[27845]: pgmap v20840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38677.296366 secs 2024-04-26T22:35:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:38 smithi096 ceph-mon[32753]: pgmap v20840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38677.296366 secs 2024-04-26T22:35:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:38 smithi194 ceph-mon[33017]: pgmap v20840: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:39.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38677.296366 secs 2024-04-26T22:35:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38680.279710 secs 2024-04-26T22:35:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38680.279710 secs 2024-04-26T22:35:40.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38680.279710 secs 2024-04-26T22:35:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:40 smithi033 ceph-mon[27845]: pgmap v20841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:40 smithi096 ceph-mon[32753]: pgmap v20841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:40 smithi194 ceph-mon[33017]: pgmap v20841: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:42 smithi033 ceph-mon[27845]: pgmap v20842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:42 smithi096 ceph-mon[32753]: pgmap v20842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:42 smithi194 ceph-mon[33017]: pgmap v20842: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38682.296452 secs 2024-04-26T22:35:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38682.296452 secs 2024-04-26T22:35:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38682.296452 secs 2024-04-26T22:35:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:44 smithi033 ceph-mon[27845]: pgmap v20843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38685.279773 secs 2024-04-26T22:35:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:44 smithi096 ceph-mon[32753]: pgmap v20843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38685.279773 secs 2024-04-26T22:35:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:44 smithi194 ceph-mon[33017]: pgmap v20843: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:45.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38685.279773 secs 2024-04-26T22:35:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:46.336+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:46.336 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:46.336+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:35:46.337 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:35:46.336+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:35:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:46 smithi033 ceph-mon[27845]: pgmap v20844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:46 smithi096 ceph-mon[32753]: pgmap v20844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:46 smithi194 ceph-mon[33017]: pgmap v20844: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:48 smithi096 ceph-mon[32753]: pgmap v20845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:49.183 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38687.296538 secs 2024-04-26T22:35:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:48 smithi194 ceph-mon[33017]: pgmap v20845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38687.296538 secs 2024-04-26T22:35:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:48 smithi033 ceph-mon[27845]: pgmap v20845: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38687.296538 secs 2024-04-26T22:35:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38690.279834 secs 2024-04-26T22:35:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38690.279834 secs 2024-04-26T22:35:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38690.279834 secs 2024-04-26T22:35:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:50 smithi194 ceph-mon[33017]: pgmap v20846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:50 smithi033 ceph-mon[27845]: pgmap v20846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:50 smithi096 ceph-mon[32753]: pgmap v20846: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:53.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:52 smithi033 ceph-mon[27845]: pgmap v20847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:52 smithi194 ceph-mon[33017]: pgmap v20847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:52 smithi096 ceph-mon[32753]: pgmap v20847: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38692.296620 secs 2024-04-26T22:35:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38692.296620 secs 2024-04-26T22:35:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38692.296620 secs 2024-04-26T22:35:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:54 smithi194 ceph-mon[33017]: pgmap v20848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38695.279980 secs 2024-04-26T22:35:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:54 smithi033 ceph-mon[27845]: pgmap v20848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38695.279980 secs 2024-04-26T22:35:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:54 smithi096 ceph-mon[32753]: pgmap v20848: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38695.279980 secs 2024-04-26T22:35:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:56 smithi194 ceph-mon[33017]: pgmap v20849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:56 smithi033 ceph-mon[27845]: pgmap v20849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:56 smithi096 ceph-mon[32753]: pgmap v20849: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:58 smithi194 ceph-mon[33017]: pgmap v20850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:58 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:35:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38697.296691 secs 2024-04-26T22:35:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:58 smithi033 ceph-mon[27845]: pgmap v20850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:58 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:35:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38697.296691 secs 2024-04-26T22:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:58 smithi096 ceph-mon[32753]: pgmap v20850: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:58 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:35:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38697.296691 secs 2024-04-26T22:36:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38700.280145 secs 2024-04-26T22:36:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:36:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:59 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:36:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:59 smithi194 ceph-mon[33017]: pgmap v20851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:36:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:59 smithi194 ceph-mon[33017]: pgmap v20852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:36:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:35:59 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:36:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38700.280145 secs 2024-04-26T22:36:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:36:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:59 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:36:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:59 smithi033 ceph-mon[27845]: pgmap v20851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:36:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:59 smithi033 ceph-mon[27845]: pgmap v20852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:36:00.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:35:59 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38700.280145 secs 2024-04-26T22:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:59 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:59 smithi096 ceph-mon[32753]: pgmap v20851: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:59 smithi096 ceph-mon[32753]: pgmap v20852: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:36:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:35:59 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:36:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:01.352+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:01.352+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:01.352+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:36:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:01 smithi033 ceph-mon[27845]: pgmap v20853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:01 smithi096 ceph-mon[32753]: pgmap v20853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:01 smithi194 ceph-mon[33017]: pgmap v20853: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:03 smithi033 ceph-mon[27845]: pgmap v20854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:03 smithi096 ceph-mon[32753]: pgmap v20854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:03 smithi194 ceph-mon[33017]: pgmap v20854: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38702.296773 secs 2024-04-26T22:36:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38702.296773 secs 2024-04-26T22:36:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38702.296773 secs 2024-04-26T22:36:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38705.280288 secs 2024-04-26T22:36:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:05 smithi033 ceph-mon[27845]: pgmap v20855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38705.280288 secs 2024-04-26T22:36:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:05 smithi096 ceph-mon[32753]: pgmap v20855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38705.280288 secs 2024-04-26T22:36:05.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:05 smithi194 ceph-mon[33017]: pgmap v20855: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:07.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:07 smithi033 ceph-mon[27845]: pgmap v20856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:07 smithi096 ceph-mon[32753]: pgmap v20856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:07 smithi194 ceph-mon[33017]: pgmap v20856: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38707.296856 secs 2024-04-26T22:36:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38707.296856 secs 2024-04-26T22:36:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38707.296856 secs 2024-04-26T22:36:10.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:09 smithi194 ceph-mon[33017]: pgmap v20857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:36:10.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:09 smithi033 ceph-mon[27845]: pgmap v20857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:36:10.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:09 smithi096 ceph-mon[32753]: pgmap v20857: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:36:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38710.280442 secs 2024-04-26T22:36:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38710.280442 secs 2024-04-26T22:36:11.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38710.280442 secs 2024-04-26T22:36:12.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:11 smithi194 ceph-mon[33017]: pgmap v20858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:12.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:11 smithi033 ceph-mon[27845]: pgmap v20858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:12.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:11 smithi096 ceph-mon[32753]: pgmap v20858: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:13 smithi033 ceph-mon[27845]: pgmap v20859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:13 smithi096 ceph-mon[32753]: pgmap v20859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:13 smithi194 ceph-mon[33017]: pgmap v20859: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38712.296940 secs 2024-04-26T22:36:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38712.296940 secs 2024-04-26T22:36:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38712.296940 secs 2024-04-26T22:36:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38715.280506 secs 2024-04-26T22:36:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:15 smithi033 ceph-mon[27845]: pgmap v20860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38715.280506 secs 2024-04-26T22:36:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:15 smithi096 ceph-mon[32753]: pgmap v20860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38715.280506 secs 2024-04-26T22:36:15.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:15 smithi194 ceph-mon[33017]: pgmap v20860: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:16.357+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:16.357+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:16.357+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:36:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:17 smithi033 ceph-mon[27845]: pgmap v20861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:17 smithi096 ceph-mon[32753]: pgmap v20861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:17 smithi194 ceph-mon[33017]: pgmap v20861: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38717.297017 secs 2024-04-26T22:36:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38717.297017 secs 2024-04-26T22:36:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38717.297017 secs 2024-04-26T22:36:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:19 smithi194 ceph-mon[33017]: pgmap v20862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:19 smithi033 ceph-mon[27845]: pgmap v20862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:20.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:19 smithi096 ceph-mon[32753]: pgmap v20862: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38720.280627 secs 2024-04-26T22:36:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38720.280627 secs 2024-04-26T22:36:21.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38720.280627 secs 2024-04-26T22:36:22.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:21 smithi194 ceph-mon[33017]: pgmap v20863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:22.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:21 smithi033 ceph-mon[27845]: pgmap v20863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:22.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:21 smithi096 ceph-mon[32753]: pgmap v20863: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:23 smithi033 ceph-mon[27845]: pgmap v20864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:23.692 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:23 smithi096 ceph-mon[32753]: pgmap v20864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:23 smithi194 ceph-mon[33017]: pgmap v20864: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38722.297105 secs 2024-04-26T22:36:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38722.297105 secs 2024-04-26T22:36:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38722.297105 secs 2024-04-26T22:36:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38725.280749 secs 2024-04-26T22:36:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:25 smithi033 ceph-mon[27845]: pgmap v20865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38725.280749 secs 2024-04-26T22:36:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:25 smithi096 ceph-mon[32753]: pgmap v20865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38725.280749 secs 2024-04-26T22:36:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:25 smithi194 ceph-mon[33017]: pgmap v20865: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:27 smithi033 ceph-mon[27845]: pgmap v20866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:27 smithi096 ceph-mon[32753]: pgmap v20866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:27 smithi194 ceph-mon[33017]: pgmap v20866: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38727.297187 secs 2024-04-26T22:36:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38727.297187 secs 2024-04-26T22:36:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38727.297187 secs 2024-04-26T22:36:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:29 smithi194 ceph-mon[33017]: pgmap v20867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:30.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:29 smithi033 ceph-mon[27845]: pgmap v20867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:29 smithi096 ceph-mon[32753]: pgmap v20867: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38730.280878 secs 2024-04-26T22:36:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38730.280878 secs 2024-04-26T22:36:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38730.280878 secs 2024-04-26T22:36:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:31.360+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:31.360+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:36:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:31.360+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:31 smithi194 ceph-mon[33017]: pgmap v20868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:32.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:31 smithi033 ceph-mon[27845]: pgmap v20868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:32.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:31 smithi096 ceph-mon[32753]: pgmap v20868: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:33 smithi033 ceph-mon[27845]: pgmap v20869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:33 smithi096 ceph-mon[32753]: pgmap v20869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:33.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:33 smithi194 ceph-mon[33017]: pgmap v20869: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38732.297270 secs 2024-04-26T22:36:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38732.297270 secs 2024-04-26T22:36:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38732.297270 secs 2024-04-26T22:36:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38735.280964 secs 2024-04-26T22:36:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:35 smithi033 ceph-mon[27845]: pgmap v20870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38735.280964 secs 2024-04-26T22:36:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:35 smithi096 ceph-mon[32753]: pgmap v20870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38735.280964 secs 2024-04-26T22:36:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:35 smithi194 ceph-mon[33017]: pgmap v20870: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:37 smithi033 ceph-mon[27845]: pgmap v20871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:37 smithi096 ceph-mon[32753]: pgmap v20871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:37 smithi194 ceph-mon[33017]: pgmap v20871: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38737.297356 secs 2024-04-26T22:36:39.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38737.297356 secs 2024-04-26T22:36:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38737.297356 secs 2024-04-26T22:36:40.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:39 smithi194 ceph-mon[33017]: pgmap v20872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:39 smithi033 ceph-mon[27845]: pgmap v20872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:39 smithi096 ceph-mon[32753]: pgmap v20872: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38740.281091 secs 2024-04-26T22:36:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38740.281091 secs 2024-04-26T22:36:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38740.281091 secs 2024-04-26T22:36:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:41 smithi194 ceph-mon[33017]: pgmap v20873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:41 smithi033 ceph-mon[27845]: pgmap v20873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:41 smithi096 ceph-mon[32753]: pgmap v20873: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:43 smithi033 ceph-mon[27845]: pgmap v20874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:43.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:43 smithi096 ceph-mon[32753]: pgmap v20874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:43.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:43 smithi194 ceph-mon[33017]: pgmap v20874: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38742.297443 secs 2024-04-26T22:36:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38742.297443 secs 2024-04-26T22:36:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38742.297443 secs 2024-04-26T22:36:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38745.281214 secs 2024-04-26T22:36:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:45 smithi033 ceph-mon[27845]: pgmap v20875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38745.281214 secs 2024-04-26T22:36:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:45 smithi096 ceph-mon[32753]: pgmap v20875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38745.281214 secs 2024-04-26T22:36:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:45 smithi194 ceph-mon[33017]: pgmap v20875: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:46.364+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:46.364+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:36:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:36:46.364+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:36:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:48 smithi033 ceph-mon[27845]: pgmap v20876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:48 smithi096 ceph-mon[32753]: pgmap v20876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:48.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:48 smithi194 ceph-mon[33017]: pgmap v20876: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38747.297525 secs 2024-04-26T22:36:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38747.297525 secs 2024-04-26T22:36:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38747.297525 secs 2024-04-26T22:36:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38750.281334 secs 2024-04-26T22:36:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:50 smithi033 ceph-mon[27845]: pgmap v20877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38750.281334 secs 2024-04-26T22:36:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:50 smithi096 ceph-mon[32753]: pgmap v20877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38750.281334 secs 2024-04-26T22:36:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:50 smithi194 ceph-mon[33017]: pgmap v20877: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:52 smithi033 ceph-mon[27845]: pgmap v20878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:52 smithi096 ceph-mon[32753]: pgmap v20878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:52 smithi194 ceph-mon[33017]: pgmap v20878: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:36:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38752.297595 secs 2024-04-26T22:36:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:54 smithi033 ceph-mon[27845]: pgmap v20879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38752.297595 secs 2024-04-26T22:36:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:54 smithi096 ceph-mon[32753]: pgmap v20879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38752.297595 secs 2024-04-26T22:36:54.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:54 smithi194 ceph-mon[33017]: pgmap v20879: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38755.281448 secs 2024-04-26T22:36:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38755.281448 secs 2024-04-26T22:36:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38755.281448 secs 2024-04-26T22:36:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:56 smithi033 ceph-mon[27845]: pgmap v20880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:56 smithi096 ceph-mon[32753]: pgmap v20880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:56 smithi194 ceph-mon[33017]: pgmap v20880: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:36:58.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:58 smithi033 ceph-mon[27845]: pgmap v20881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:58 smithi096 ceph-mon[32753]: pgmap v20881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:58 smithi194 ceph-mon[33017]: pgmap v20881: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:36:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:36:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38757.297679 secs 2024-04-26T22:36:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:36:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38757.297679 secs 2024-04-26T22:36:59.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:36:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38757.297679 secs 2024-04-26T22:37:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38760.281530 secs 2024-04-26T22:37:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:00 smithi033 ceph-mon[27845]: pgmap v20882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:37:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:37:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38760.281530 secs 2024-04-26T22:37:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:00 smithi096 ceph-mon[32753]: pgmap v20882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:37:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:37:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38760.281530 secs 2024-04-26T22:37:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:00 smithi194 ceph-mon[33017]: pgmap v20882: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:37:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:37:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:00.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:01.380+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:01.380+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:37:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:01.380+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: pgmap v20883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: pgmap v20883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:37:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.433 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: pgmap v20883: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:03.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:04 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:04 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:04 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:04 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:37:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38762.297761 secs 2024-04-26T22:37:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:04 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:37:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:04 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:37:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:04 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:37:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:04 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:37:04.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38762.297761 secs 2024-04-26T22:37:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:04 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:37:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:04 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:37:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:04 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:37:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:04 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:37:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38762.297761 secs 2024-04-26T22:37:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:05 smithi096 ceph-mon[32753]: pgmap v20884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38765.281599 secs 2024-04-26T22:37:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:05 smithi194 ceph-mon[33017]: pgmap v20884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38765.281599 secs 2024-04-26T22:37:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:05.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:05 smithi033 ceph-mon[27845]: pgmap v20884: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38765.281599 secs 2024-04-26T22:37:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:37:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:06 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:37:06.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:37:06.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:06 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:37:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:37:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:37:06.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:06 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:37:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:07 smithi194 ceph-mon[33017]: pgmap v20885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:07 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:37:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:07 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:37:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:07 smithi194 ceph-mon[33017]: pgmap v20886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:37:07.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:07 smithi194 ceph-mon[33017]: pgmap v20887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:07 smithi033 ceph-mon[27845]: pgmap v20885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:07 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:37:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:07 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:37:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:07 smithi033 ceph-mon[27845]: pgmap v20886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:37:07.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:07 smithi033 ceph-mon[27845]: pgmap v20887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:07 smithi096 ceph-mon[32753]: pgmap v20885: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:07 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:37:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:07 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:37:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:07 smithi096 ceph-mon[32753]: pgmap v20886: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:37:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:07 smithi096 ceph-mon[32753]: pgmap v20887: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:09 smithi194 ceph-mon[33017]: pgmap v20888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T22:37:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38767.297842 secs 2024-04-26T22:37:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:09 smithi033 ceph-mon[27845]: pgmap v20888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T22:37:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38767.297842 secs 2024-04-26T22:37:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:09 smithi096 ceph-mon[32753]: pgmap v20888: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T22:37:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38767.297842 secs 2024-04-26T22:37:10.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38770.281718 secs 2024-04-26T22:37:10.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38770.281718 secs 2024-04-26T22:37:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38770.281718 secs 2024-04-26T22:37:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:11 smithi194 ceph-mon[33017]: pgmap v20889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:11 smithi033 ceph-mon[27845]: pgmap v20889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:11 smithi096 ceph-mon[32753]: pgmap v20889: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:13 smithi194 ceph-mon[33017]: pgmap v20890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:13 smithi033 ceph-mon[27845]: pgmap v20890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:13 smithi096 ceph-mon[32753]: pgmap v20890: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:37:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38772.297926 secs 2024-04-26T22:37:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38772.297926 secs 2024-04-26T22:37:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38772.297926 secs 2024-04-26T22:37:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:15 smithi194 ceph-mon[33017]: pgmap v20891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:37:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38775.281789 secs 2024-04-26T22:37:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:15 smithi033 ceph-mon[27845]: pgmap v20891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:37:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38775.281789 secs 2024-04-26T22:37:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:15 smithi096 ceph-mon[32753]: pgmap v20891: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:37:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38775.281789 secs 2024-04-26T22:37:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:16.389+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:16.389+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:16.389+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:37:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:17 smithi194 ceph-mon[33017]: pgmap v20892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:37:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:17 smithi033 ceph-mon[27845]: pgmap v20892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:37:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:17 smithi096 ceph-mon[32753]: pgmap v20892: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:37:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:19 smithi194 ceph-mon[33017]: pgmap v20893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38777.298026 secs 2024-04-26T22:37:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:19 smithi033 ceph-mon[27845]: pgmap v20893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38777.298026 secs 2024-04-26T22:37:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:19 smithi096 ceph-mon[32753]: pgmap v20893: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38777.298026 secs 2024-04-26T22:37:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38780.281911 secs 2024-04-26T22:37:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38780.281911 secs 2024-04-26T22:37:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38780.281911 secs 2024-04-26T22:37:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:21 smithi194 ceph-mon[33017]: pgmap v20894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:21 smithi033 ceph-mon[27845]: pgmap v20894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:21.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:21 smithi096 ceph-mon[32753]: pgmap v20894: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:23 smithi194 ceph-mon[33017]: pgmap v20895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:23 smithi033 ceph-mon[27845]: pgmap v20895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:23 smithi096 ceph-mon[32753]: pgmap v20895: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38782.298111 secs 2024-04-26T22:37:24.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38782.298111 secs 2024-04-26T22:37:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38782.298111 secs 2024-04-26T22:37:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:25 smithi194 ceph-mon[33017]: pgmap v20896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38785.282033 secs 2024-04-26T22:37:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:25 smithi033 ceph-mon[27845]: pgmap v20896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38785.282033 secs 2024-04-26T22:37:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:25 smithi096 ceph-mon[32753]: pgmap v20896: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38785.282033 secs 2024-04-26T22:37:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:27 smithi194 ceph-mon[33017]: pgmap v20897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:27 smithi033 ceph-mon[27845]: pgmap v20897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:27 smithi096 ceph-mon[32753]: pgmap v20897: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:29 smithi194 ceph-mon[33017]: pgmap v20898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38787.298203 secs 2024-04-26T22:37:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:29 smithi033 ceph-mon[27845]: pgmap v20898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38787.298203 secs 2024-04-26T22:37:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:29 smithi096 ceph-mon[32753]: pgmap v20898: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38787.298203 secs 2024-04-26T22:37:30.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38790.282178 secs 2024-04-26T22:37:30.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38790.282178 secs 2024-04-26T22:37:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38790.282178 secs 2024-04-26T22:37:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:31.394+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:31.394+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:31.394+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:37:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:31 smithi194 ceph-mon[33017]: pgmap v20899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:31 smithi033 ceph-mon[27845]: pgmap v20899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:31 smithi096 ceph-mon[32753]: pgmap v20899: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:33 smithi194 ceph-mon[33017]: pgmap v20900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:33 smithi033 ceph-mon[27845]: pgmap v20900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:33 smithi096 ceph-mon[32753]: pgmap v20900: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:34.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38792.298289 secs 2024-04-26T22:37:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38792.298289 secs 2024-04-26T22:37:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38792.298289 secs 2024-04-26T22:37:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:35 smithi194 ceph-mon[33017]: pgmap v20901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38795.282238 secs 2024-04-26T22:37:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:35 smithi033 ceph-mon[27845]: pgmap v20901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38795.282238 secs 2024-04-26T22:37:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:35 smithi096 ceph-mon[32753]: pgmap v20901: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38795.282238 secs 2024-04-26T22:37:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:36 smithi194 ceph-mon[33017]: pgmap v20902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:36 smithi033 ceph-mon[27845]: pgmap v20902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:36 smithi096 ceph-mon[32753]: pgmap v20902: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:38 smithi033 ceph-mon[27845]: pgmap v20903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:39.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38797.298371 secs 2024-04-26T22:37:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:38 smithi096 ceph-mon[32753]: pgmap v20903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38797.298371 secs 2024-04-26T22:37:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:38 smithi194 ceph-mon[33017]: pgmap v20903: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38797.298371 secs 2024-04-26T22:37:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38800.282368 secs 2024-04-26T22:37:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38800.282368 secs 2024-04-26T22:37:40.255 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38800.282368 secs 2024-04-26T22:37:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:40 smithi033 ceph-mon[27845]: pgmap v20904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:40 smithi096 ceph-mon[32753]: pgmap v20904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:40 smithi194 ceph-mon[33017]: pgmap v20904: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:42 smithi033 ceph-mon[27845]: pgmap v20905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:43.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:42 smithi096 ceph-mon[32753]: pgmap v20905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:42 smithi194 ceph-mon[33017]: pgmap v20905: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38802.298456 secs 2024-04-26T22:37:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38802.298456 secs 2024-04-26T22:37:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38802.298456 secs 2024-04-26T22:37:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:44 smithi033 ceph-mon[27845]: pgmap v20906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:45.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38805.282460 secs 2024-04-26T22:37:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:44 smithi096 ceph-mon[32753]: pgmap v20906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38805.282460 secs 2024-04-26T22:37:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:44 smithi194 ceph-mon[33017]: pgmap v20906: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38805.282460 secs 2024-04-26T22:37:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:46.404+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:46.404+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:37:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:37:46.404+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:37:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:46 smithi033 ceph-mon[27845]: pgmap v20907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:46 smithi096 ceph-mon[32753]: pgmap v20907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:46 smithi194 ceph-mon[33017]: pgmap v20907: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:48 smithi096 ceph-mon[32753]: pgmap v20908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38807.298541 secs 2024-04-26T22:37:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:48 smithi194 ceph-mon[33017]: pgmap v20908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38807.298541 secs 2024-04-26T22:37:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:48 smithi033 ceph-mon[27845]: pgmap v20908: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38807.298541 secs 2024-04-26T22:37:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38810.282556 secs 2024-04-26T22:37:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38810.282556 secs 2024-04-26T22:37:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38810.282556 secs 2024-04-26T22:37:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:50 smithi194 ceph-mon[33017]: pgmap v20909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:51.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:50 smithi033 ceph-mon[27845]: pgmap v20909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:51.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:50 smithi096 ceph-mon[32753]: pgmap v20909: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:52 smithi194 ceph-mon[33017]: pgmap v20910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:52 smithi033 ceph-mon[27845]: pgmap v20910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:52 smithi096 ceph-mon[32753]: pgmap v20910: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38812.298627 secs 2024-04-26T22:37:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38812.298627 secs 2024-04-26T22:37:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38812.298627 secs 2024-04-26T22:37:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:54 smithi194 ceph-mon[33017]: pgmap v20911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38815.282660 secs 2024-04-26T22:37:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:54 smithi033 ceph-mon[27845]: pgmap v20911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:55.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38815.282660 secs 2024-04-26T22:37:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:54 smithi096 ceph-mon[32753]: pgmap v20911: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38815.282660 secs 2024-04-26T22:37:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:56 smithi033 ceph-mon[27845]: pgmap v20912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:56 smithi194 ceph-mon[33017]: pgmap v20912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:57.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:56 smithi096 ceph-mon[32753]: pgmap v20912: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:37:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:58 smithi194 ceph-mon[33017]: pgmap v20913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38817.298714 secs 2024-04-26T22:37:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:58 smithi033 ceph-mon[27845]: pgmap v20913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38817.298714 secs 2024-04-26T22:37:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:58 smithi096 ceph-mon[32753]: pgmap v20913: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:37:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38817.298714 secs 2024-04-26T22:38:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:37:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38820.282801 secs 2024-04-26T22:38:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:37:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38820.282801 secs 2024-04-26T22:38:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:37:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38820.282801 secs 2024-04-26T22:38:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:00 smithi194 ceph-mon[33017]: pgmap v20914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:38:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:01.420+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:01.420+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:01.420+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:38:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:00 smithi033 ceph-mon[27845]: pgmap v20914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:38:01.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:00 smithi096 ceph-mon[32753]: pgmap v20914: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:38:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:02 smithi194 ceph-mon[33017]: pgmap v20915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:02 smithi033 ceph-mon[27845]: pgmap v20915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:02 smithi096 ceph-mon[32753]: pgmap v20915: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:04.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38822.298796 secs 2024-04-26T22:38:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38822.298796 secs 2024-04-26T22:38:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38822.298796 secs 2024-04-26T22:38:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:04 smithi194 ceph-mon[33017]: pgmap v20916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38825.282923 secs 2024-04-26T22:38:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:04 smithi033 ceph-mon[27845]: pgmap v20916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:05.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38825.282923 secs 2024-04-26T22:38:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:04 smithi096 ceph-mon[32753]: pgmap v20916: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38825.282923 secs 2024-04-26T22:38:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:07 smithi194 ceph-mon[33017]: pgmap v20917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:38:07.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:38:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:07 smithi033 ceph-mon[27845]: pgmap v20917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:38:07.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:38:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:07 smithi096 ceph-mon[32753]: pgmap v20917: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:38:07.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:38:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:38:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:38:08.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:08 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:38:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:38:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:38:08.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:08 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:38:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:38:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:38:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:08 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:38:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:09 smithi194 ceph-mon[33017]: pgmap v20918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:09 smithi194 ceph-mon[33017]: pgmap v20919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:38:09.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38827.298884 secs 2024-04-26T22:38:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:09 smithi033 ceph-mon[27845]: pgmap v20918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:09 smithi033 ceph-mon[27845]: pgmap v20919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:38:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38827.298884 secs 2024-04-26T22:38:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:09 smithi096 ceph-mon[32753]: pgmap v20918: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:38:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:09 smithi096 ceph-mon[32753]: pgmap v20919: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:38:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38827.298884 secs 2024-04-26T22:38:10.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38830.283066 secs 2024-04-26T22:38:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38830.283066 secs 2024-04-26T22:38:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38830.283066 secs 2024-04-26T22:38:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:11 smithi194 ceph-mon[33017]: pgmap v20920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:11 smithi033 ceph-mon[27845]: pgmap v20920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:11.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:11 smithi096 ceph-mon[32753]: pgmap v20920: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:13 smithi194 ceph-mon[33017]: pgmap v20921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:13 smithi033 ceph-mon[27845]: pgmap v20921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:13 smithi096 ceph-mon[32753]: pgmap v20921: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38832.298995 secs 2024-04-26T22:38:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38832.298995 secs 2024-04-26T22:38:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38832.298995 secs 2024-04-26T22:38:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:15 smithi194 ceph-mon[33017]: pgmap v20922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38835.283164 secs 2024-04-26T22:38:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:15 smithi033 ceph-mon[27845]: pgmap v20922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38835.283164 secs 2024-04-26T22:38:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:15 smithi096 ceph-mon[32753]: pgmap v20922: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38835.283164 secs 2024-04-26T22:38:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:16.423+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:16.423+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:16.423+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:38:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:17 smithi194 ceph-mon[33017]: pgmap v20923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:17 smithi033 ceph-mon[27845]: pgmap v20923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:17.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:17 smithi096 ceph-mon[32753]: pgmap v20923: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:38:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:19 smithi194 ceph-mon[33017]: pgmap v20924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:38:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38837.299078 secs 2024-04-26T22:38:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:19 smithi033 ceph-mon[27845]: pgmap v20924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:38:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38837.299078 secs 2024-04-26T22:38:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:19 smithi096 ceph-mon[32753]: pgmap v20924: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:38:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38837.299078 secs 2024-04-26T22:38:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38840.283249 secs 2024-04-26T22:38:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38840.283249 secs 2024-04-26T22:38:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38840.283249 secs 2024-04-26T22:38:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:21 smithi194 ceph-mon[33017]: pgmap v20925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:21 smithi033 ceph-mon[27845]: pgmap v20925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:21 smithi096 ceph-mon[32753]: pgmap v20925: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:23 smithi194 ceph-mon[33017]: pgmap v20926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:23 smithi033 ceph-mon[27845]: pgmap v20926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:23.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:23 smithi096 ceph-mon[32753]: pgmap v20926: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38842.299169 secs 2024-04-26T22:38:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38842.299169 secs 2024-04-26T22:38:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38842.299169 secs 2024-04-26T22:38:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:25 smithi194 ceph-mon[33017]: pgmap v20927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38845.283368 secs 2024-04-26T22:38:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:25 smithi033 ceph-mon[27845]: pgmap v20927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38845.283368 secs 2024-04-26T22:38:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:25 smithi096 ceph-mon[32753]: pgmap v20927: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38845.283368 secs 2024-04-26T22:38:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:27 smithi194 ceph-mon[33017]: pgmap v20928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:27 smithi033 ceph-mon[27845]: pgmap v20928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:27 smithi096 ceph-mon[32753]: pgmap v20928: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:28.410+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:38:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:28.410+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:38:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:28.411+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 303 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc006570 con 0x564385821900 2024-04-26T22:38:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:28.411+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 303 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:38:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:29 smithi033 ceph-mon[27845]: pgmap v20929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38847.299275 secs 2024-04-26T22:38:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:29 smithi096 ceph-mon[32753]: pgmap v20929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38847.299275 secs 2024-04-26T22:38:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:29 smithi194 ceph-mon[33017]: pgmap v20929: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38847.299275 secs 2024-04-26T22:38:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38850.283507 secs 2024-04-26T22:38:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38850.283507 secs 2024-04-26T22:38:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38850.283507 secs 2024-04-26T22:38:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:31 smithi033 ceph-mon[27845]: pgmap v20930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:31 smithi096 ceph-mon[32753]: pgmap v20930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:31.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:31.438+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:31.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:31.438+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:31.439 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:31.438+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:38:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:31 smithi194 ceph-mon[33017]: pgmap v20930: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:33 smithi033 ceph-mon[27845]: pgmap v20931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:33.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:33 smithi096 ceph-mon[32753]: pgmap v20931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:33 smithi194 ceph-mon[33017]: pgmap v20931: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38852.299361 secs 2024-04-26T22:38:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38852.299361 secs 2024-04-26T22:38:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38852.299361 secs 2024-04-26T22:38:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:35 smithi033 ceph-mon[27845]: pgmap v20932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:35.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38855.283567 secs 2024-04-26T22:38:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:35 smithi096 ceph-mon[32753]: pgmap v20932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38855.283567 secs 2024-04-26T22:38:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:35 smithi194 ceph-mon[33017]: pgmap v20932: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38855.283567 secs 2024-04-26T22:38:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:37 smithi033 ceph-mon[27845]: pgmap v20933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:37 smithi096 ceph-mon[32753]: pgmap v20933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:37 smithi194 ceph-mon[33017]: pgmap v20933: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:39 smithi033 ceph-mon[27845]: pgmap v20934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38857.299448 secs 2024-04-26T22:38:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:39 smithi096 ceph-mon[32753]: pgmap v20934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38857.299448 secs 2024-04-26T22:38:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:39 smithi194 ceph-mon[33017]: pgmap v20934: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:39.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38857.299448 secs 2024-04-26T22:38:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38860.283645 secs 2024-04-26T22:38:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38860.283645 secs 2024-04-26T22:38:40.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38860.283645 secs 2024-04-26T22:38:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:41 smithi033 ceph-mon[27845]: pgmap v20935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:41.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:41 smithi096 ceph-mon[32753]: pgmap v20935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:41 smithi194 ceph-mon[33017]: pgmap v20935: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:43.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:43 smithi033 ceph-mon[27845]: pgmap v20936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:43.436 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:43 smithi096 ceph-mon[32753]: pgmap v20936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:43.608 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:43 smithi194 ceph-mon[33017]: pgmap v20936: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38862.299535 secs 2024-04-26T22:38:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38862.299535 secs 2024-04-26T22:38:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38862.299535 secs 2024-04-26T22:38:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:45 smithi096 ceph-mon[32753]: pgmap v20937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38865.283778 secs 2024-04-26T22:38:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:45 smithi194 ceph-mon[33017]: pgmap v20937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38865.283778 secs 2024-04-26T22:38:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:45 smithi033 ceph-mon[27845]: pgmap v20937: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38865.283778 secs 2024-04-26T22:38:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:46.449+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:46.449+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:38:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:38:46.449+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:38:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:47 smithi194 ceph-mon[33017]: pgmap v20938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:47.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:47 smithi033 ceph-mon[27845]: pgmap v20938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:47 smithi096 ceph-mon[32753]: pgmap v20938: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:49 smithi194 ceph-mon[33017]: pgmap v20939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38867.299622 secs 2024-04-26T22:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:49 smithi033 ceph-mon[27845]: pgmap v20939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38867.299622 secs 2024-04-26T22:38:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:49 smithi096 ceph-mon[32753]: pgmap v20939: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38867.299622 secs 2024-04-26T22:38:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38870.283919 secs 2024-04-26T22:38:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38870.283919 secs 2024-04-26T22:38:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38870.283919 secs 2024-04-26T22:38:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:51 smithi194 ceph-mon[33017]: pgmap v20940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:51 smithi033 ceph-mon[27845]: pgmap v20940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:51 smithi096 ceph-mon[32753]: pgmap v20940: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:53 smithi194 ceph-mon[33017]: pgmap v20941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:53 smithi033 ceph-mon[27845]: pgmap v20941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:53 smithi096 ceph-mon[32753]: pgmap v20941: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38872.299707 secs 2024-04-26T22:38:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38872.299707 secs 2024-04-26T22:38:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38872.299707 secs 2024-04-26T22:38:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:55 smithi194 ceph-mon[33017]: pgmap v20942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38875.284053 secs 2024-04-26T22:38:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:55 smithi033 ceph-mon[27845]: pgmap v20942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38875.284053 secs 2024-04-26T22:38:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:55 smithi096 ceph-mon[32753]: pgmap v20942: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38875.284053 secs 2024-04-26T22:38:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:57 smithi194 ceph-mon[33017]: pgmap v20943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:57 smithi033 ceph-mon[27845]: pgmap v20943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:57 smithi096 ceph-mon[32753]: pgmap v20943: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:59 smithi194 ceph-mon[33017]: pgmap v20944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:38:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38877.299791 secs 2024-04-26T22:38:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:59 smithi033 ceph-mon[27845]: pgmap v20944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:38:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38877.299791 secs 2024-04-26T22:38:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:59 smithi096 ceph-mon[32753]: pgmap v20944: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:38:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:38:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38877.299791 secs 2024-04-26T22:39:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38880.284142 secs 2024-04-26T22:39:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38880.284142 secs 2024-04-26T22:39:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38880.284142 secs 2024-04-26T22:39:01.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:01.464+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:01.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:01.464+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:01.465 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:01.464+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:39:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:01 smithi194 ceph-mon[33017]: pgmap v20945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:01 smithi033 ceph-mon[27845]: pgmap v20945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:01.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:01 smithi096 ceph-mon[32753]: pgmap v20945: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:03 smithi194 ceph-mon[33017]: pgmap v20946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:03 smithi033 ceph-mon[27845]: pgmap v20946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:03.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:03 smithi096 ceph-mon[32753]: pgmap v20946: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:04.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38882.299877 secs 2024-04-26T22:39:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38882.299877 secs 2024-04-26T22:39:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38882.299877 secs 2024-04-26T22:39:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:05 smithi194 ceph-mon[33017]: pgmap v20947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38885.284267 secs 2024-04-26T22:39:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:05 smithi033 ceph-mon[27845]: pgmap v20947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38885.284267 secs 2024-04-26T22:39:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:05 smithi096 ceph-mon[32753]: pgmap v20947: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38885.284267 secs 2024-04-26T22:39:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:07 smithi194 ceph-mon[33017]: pgmap v20948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:07 smithi033 ceph-mon[27845]: pgmap v20948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:07.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:07 smithi096 ceph-mon[32753]: pgmap v20948: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:08 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:39:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:08 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:39:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:08 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:39:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:09 smithi194 ceph-mon[33017]: pgmap v20949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38887.299946 secs 2024-04-26T22:39:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:39:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:39:09.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:39:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:09 smithi033 ceph-mon[27845]: pgmap v20949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38887.299946 secs 2024-04-26T22:39:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:39:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:39:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:09 smithi096 ceph-mon[32753]: pgmap v20949: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38887.299946 secs 2024-04-26T22:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:39:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:39:10.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38890.284380 secs 2024-04-26T22:39:10.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:10 smithi033 ceph-mon[27845]: pgmap v20950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:10.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:10 smithi033 ceph-mon[27845]: pgmap v20951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:39:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38890.284380 secs 2024-04-26T22:39:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:10 smithi194 ceph-mon[33017]: pgmap v20950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:10.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:10 smithi194 ceph-mon[33017]: pgmap v20951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:39:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38890.284380 secs 2024-04-26T22:39:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:10 smithi096 ceph-mon[32753]: pgmap v20950: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:10 smithi096 ceph-mon[32753]: pgmap v20951: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:39:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:12 smithi194 ceph-mon[33017]: pgmap v20952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:12 smithi033 ceph-mon[27845]: pgmap v20952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:12 smithi096 ceph-mon[32753]: pgmap v20952: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38892.300003 secs 2024-04-26T22:39:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:14 smithi194 ceph-mon[33017]: pgmap v20953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38892.300003 secs 2024-04-26T22:39:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:14 smithi033 ceph-mon[27845]: pgmap v20953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38892.300003 secs 2024-04-26T22:39:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:14 smithi096 ceph-mon[32753]: pgmap v20953: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38895.284461 secs 2024-04-26T22:39:15.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38895.284461 secs 2024-04-26T22:39:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38895.284461 secs 2024-04-26T22:39:16.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:16.474+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:16.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:16.474+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:16.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:16.474+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:39:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:16 smithi194 ceph-mon[33017]: pgmap v20954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:16 smithi033 ceph-mon[27845]: pgmap v20954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:16 smithi096 ceph-mon[32753]: pgmap v20954: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:18 smithi194 ceph-mon[33017]: pgmap v20955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:18 smithi033 ceph-mon[27845]: pgmap v20955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:18 smithi096 ceph-mon[32753]: pgmap v20955: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:39:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38897.300086 secs 2024-04-26T22:39:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38897.300086 secs 2024-04-26T22:39:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38897.300086 secs 2024-04-26T22:39:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38900.284582 secs 2024-04-26T22:39:20.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:20 smithi194 ceph-mon[33017]: pgmap v20956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:39:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38900.284582 secs 2024-04-26T22:39:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:20 smithi033 ceph-mon[27845]: pgmap v20956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:39:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38900.284582 secs 2024-04-26T22:39:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:20 smithi096 ceph-mon[32753]: pgmap v20956: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:39:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:22 smithi194 ceph-mon[33017]: pgmap v20957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:22 smithi033 ceph-mon[27845]: pgmap v20957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:22 smithi096 ceph-mon[32753]: pgmap v20957: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38902.300147 secs 2024-04-26T22:39:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:24 smithi194 ceph-mon[33017]: pgmap v20958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38902.300147 secs 2024-04-26T22:39:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:24 smithi033 ceph-mon[27845]: pgmap v20958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38902.300147 secs 2024-04-26T22:39:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:24 smithi096 ceph-mon[32753]: pgmap v20958: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38905.284699 secs 2024-04-26T22:39:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:25 smithi033 ceph-mon[27845]: pgmap v20959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38905.284699 secs 2024-04-26T22:39:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:25 smithi096 ceph-mon[32753]: pgmap v20959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38905.284699 secs 2024-04-26T22:39:25.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:25 smithi194 ceph-mon[33017]: pgmap v20959: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:28.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:27 smithi033 ceph-mon[27845]: pgmap v20960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:28.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:27 smithi096 ceph-mon[32753]: pgmap v20960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:27 smithi194 ceph-mon[33017]: pgmap v20960: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38907.300230 secs 2024-04-26T22:39:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38907.300230 secs 2024-04-26T22:39:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38907.300230 secs 2024-04-26T22:39:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38910.284818 secs 2024-04-26T22:39:30.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:29 smithi096 ceph-mon[32753]: pgmap v20961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38910.284818 secs 2024-04-26T22:39:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:29 smithi194 ceph-mon[33017]: pgmap v20961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38910.284818 secs 2024-04-26T22:39:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:29 smithi033 ceph-mon[27845]: pgmap v20961: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:31.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:31.483+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:31.484 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:31.483+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:31.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:31.483+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:39:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:32 smithi194 ceph-mon[33017]: pgmap v20962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:32 smithi033 ceph-mon[27845]: pgmap v20962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:32 smithi096 ceph-mon[32753]: pgmap v20962: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38912.300314 secs 2024-04-26T22:39:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:34 smithi194 ceph-mon[33017]: pgmap v20963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38912.300314 secs 2024-04-26T22:39:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:34 smithi033 ceph-mon[27845]: pgmap v20963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38912.300314 secs 2024-04-26T22:39:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:34 smithi096 ceph-mon[32753]: pgmap v20963: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38915.284951 secs 2024-04-26T22:39:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38915.284951 secs 2024-04-26T22:39:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38915.284951 secs 2024-04-26T22:39:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:36 smithi194 ceph-mon[33017]: pgmap v20964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:36.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:36 smithi033 ceph-mon[27845]: pgmap v20964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:36.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:36 smithi096 ceph-mon[32753]: pgmap v20964: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:38 smithi194 ceph-mon[33017]: pgmap v20965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:38 smithi033 ceph-mon[27845]: pgmap v20965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:38 smithi096 ceph-mon[32753]: pgmap v20965: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38917.300372 secs 2024-04-26T22:39:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38917.300372 secs 2024-04-26T22:39:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38917.300372 secs 2024-04-26T22:39:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38920.285067 secs 2024-04-26T22:39:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:40 smithi033 ceph-mon[27845]: pgmap v20966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38920.285067 secs 2024-04-26T22:39:40.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:40 smithi194 ceph-mon[33017]: pgmap v20966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38920.285067 secs 2024-04-26T22:39:40.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:40 smithi096 ceph-mon[32753]: pgmap v20966: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:42 smithi194 ceph-mon[33017]: pgmap v20967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:42 smithi033 ceph-mon[27845]: pgmap v20967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:42.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:42 smithi096 ceph-mon[32753]: pgmap v20967: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38922.300430 secs 2024-04-26T22:39:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:44 smithi194 ceph-mon[33017]: pgmap v20968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38922.300430 secs 2024-04-26T22:39:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:44 smithi033 ceph-mon[27845]: pgmap v20968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38922.300430 secs 2024-04-26T22:39:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:44 smithi096 ceph-mon[32753]: pgmap v20968: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38925.285209 secs 2024-04-26T22:39:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38925.285209 secs 2024-04-26T22:39:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38925.285209 secs 2024-04-26T22:39:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:46.497+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:46.497+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:39:46.499 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:39:46.497+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:39:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:46 smithi194 ceph-mon[33017]: pgmap v20969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:46 smithi033 ceph-mon[27845]: pgmap v20969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:46.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:46 smithi096 ceph-mon[32753]: pgmap v20969: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:48 smithi194 ceph-mon[33017]: pgmap v20970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:48 smithi033 ceph-mon[27845]: pgmap v20970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:48.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:48 smithi096 ceph-mon[32753]: pgmap v20970: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38927.300514 secs 2024-04-26T22:39:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:49 smithi033 ceph-mon[27845]: pgmap v20971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38927.300514 secs 2024-04-26T22:39:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:49 smithi096 ceph-mon[32753]: pgmap v20971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38927.300514 secs 2024-04-26T22:39:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:49 smithi194 ceph-mon[33017]: pgmap v20971: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38930.285327 secs 2024-04-26T22:39:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38930.285327 secs 2024-04-26T22:39:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38930.285327 secs 2024-04-26T22:39:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:51 smithi033 ceph-mon[27845]: pgmap v20972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:51.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:51 smithi096 ceph-mon[32753]: pgmap v20972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:51 smithi194 ceph-mon[33017]: pgmap v20972: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38932.300596 secs 2024-04-26T22:39:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:54 smithi194 ceph-mon[33017]: pgmap v20973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38932.300596 secs 2024-04-26T22:39:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:54 smithi033 ceph-mon[27845]: pgmap v20973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38932.300596 secs 2024-04-26T22:39:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:54 smithi096 ceph-mon[32753]: pgmap v20973: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38935.285461 secs 2024-04-26T22:39:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38935.285461 secs 2024-04-26T22:39:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38935.285461 secs 2024-04-26T22:39:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:56 smithi194 ceph-mon[33017]: pgmap v20974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:56 smithi033 ceph-mon[27845]: pgmap v20974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:56 smithi096 ceph-mon[32753]: pgmap v20974: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:39:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:58 smithi194 ceph-mon[33017]: pgmap v20975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:58 smithi033 ceph-mon[27845]: pgmap v20975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:58 smithi096 ceph-mon[32753]: pgmap v20975: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:39:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:39:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38937.300679 secs 2024-04-26T22:39:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:39:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38937.300679 secs 2024-04-26T22:39:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:39:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38937.300679 secs 2024-04-26T22:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38940.285545 secs 2024-04-26T22:40:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:00 smithi194 ceph-mon[33017]: pgmap v20976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:00.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:40:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38940.285545 secs 2024-04-26T22:40:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:00 smithi033 ceph-mon[27845]: pgmap v20976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38940.285545 secs 2024-04-26T22:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:00 smithi096 ceph-mon[32753]: pgmap v20976: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:40:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:01.514+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:01.514+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:01.514+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:40:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:02 smithi194 ceph-mon[33017]: pgmap v20977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:02.611 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:02 smithi033 ceph-mon[27845]: pgmap v20977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:02 smithi096 ceph-mon[32753]: pgmap v20977: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38942.300764 secs 2024-04-26T22:40:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:04 smithi194 ceph-mon[33017]: pgmap v20978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38942.300764 secs 2024-04-26T22:40:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:04 smithi033 ceph-mon[27845]: pgmap v20978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38942.300764 secs 2024-04-26T22:40:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:04 smithi096 ceph-mon[32753]: pgmap v20978: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38945.285668 secs 2024-04-26T22:40:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38945.285668 secs 2024-04-26T22:40:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38945.285668 secs 2024-04-26T22:40:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:06 smithi194 ceph-mon[33017]: pgmap v20979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:06 smithi033 ceph-mon[27845]: pgmap v20979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:06 smithi096 ceph-mon[32753]: pgmap v20979: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:08 smithi033 ceph-mon[27845]: pgmap v20980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:08 smithi096 ceph-mon[32753]: pgmap v20980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:08 smithi194 ceph-mon[33017]: pgmap v20980: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38947.300848 secs 2024-04-26T22:40:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:09 smithi033 ceph-mon[27845]: pgmap v20981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38947.300848 secs 2024-04-26T22:40:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:09 smithi096 ceph-mon[32753]: pgmap v20981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38947.300848 secs 2024-04-26T22:40:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:09 smithi194 ceph-mon[33017]: pgmap v20981: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38950.285820 secs 2024-04-26T22:40:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:10 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38950.285820 secs 2024-04-26T22:40:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:10 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:10.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38950.285820 secs 2024-04-26T22:40:10.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:10 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:40:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:40:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:11 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:40:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:11 smithi033 ceph-mon[27845]: pgmap v20982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:40:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:11 smithi033 ceph-mon[27845]: pgmap v20983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:11.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:11 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:40:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:40:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:11 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:40:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:11 smithi096 ceph-mon[32753]: pgmap v20982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:40:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:11 smithi096 ceph-mon[32753]: pgmap v20983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:11.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:11 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:40:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:40:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:11 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:40:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:11 smithi194 ceph-mon[33017]: pgmap v20982: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:40:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:11 smithi194 ceph-mon[33017]: pgmap v20983: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:11.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:11 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:40:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:14 smithi194 ceph-mon[33017]: pgmap v20984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38952.300925 secs 2024-04-26T22:40:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:14 smithi033 ceph-mon[27845]: pgmap v20984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:14.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38952.300925 secs 2024-04-26T22:40:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:14 smithi096 ceph-mon[32753]: pgmap v20984: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38952.300925 secs 2024-04-26T22:40:15.318 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38955.285943 secs 2024-04-26T22:40:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38955.285943 secs 2024-04-26T22:40:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38955.285943 secs 2024-04-26T22:40:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:16 smithi194 ceph-mon[33017]: pgmap v20985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:16.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:16 smithi033 ceph-mon[27845]: pgmap v20985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:16.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:16 smithi096 ceph-mon[32753]: pgmap v20985: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:16.529+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:16.529+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:16.529+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:40:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:18 smithi194 ceph-mon[33017]: pgmap v20986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:18 smithi033 ceph-mon[27845]: pgmap v20986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:18.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:18 smithi096 ceph-mon[32753]: pgmap v20986: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38957.301026 secs 2024-04-26T22:40:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38957.301026 secs 2024-04-26T22:40:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38957.301026 secs 2024-04-26T22:40:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38960.286041 secs 2024-04-26T22:40:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:20 smithi194 ceph-mon[33017]: pgmap v20987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38960.286041 secs 2024-04-26T22:40:20.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:20 smithi033 ceph-mon[27845]: pgmap v20987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38960.286041 secs 2024-04-26T22:40:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:20 smithi096 ceph-mon[32753]: pgmap v20987: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:40:22.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:22 smithi194 ceph-mon[33017]: pgmap v20988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:40:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:22 smithi033 ceph-mon[27845]: pgmap v20988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:40:22.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:22 smithi096 ceph-mon[32753]: pgmap v20988: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:40:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:24 smithi194 ceph-mon[33017]: pgmap v20989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:24.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38962.301087 secs 2024-04-26T22:40:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:24 smithi033 ceph-mon[27845]: pgmap v20989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38962.301087 secs 2024-04-26T22:40:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:24 smithi096 ceph-mon[32753]: pgmap v20989: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38962.301087 secs 2024-04-26T22:40:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38965.286163 secs 2024-04-26T22:40:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38965.286163 secs 2024-04-26T22:40:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38965.286163 secs 2024-04-26T22:40:26.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:26 smithi194 ceph-mon[33017]: pgmap v20990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:26 smithi033 ceph-mon[27845]: pgmap v20990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:26.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:26 smithi096 ceph-mon[32753]: pgmap v20990: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:28 smithi194 ceph-mon[33017]: pgmap v20991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:28.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:28 smithi033 ceph-mon[27845]: pgmap v20991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:28.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:28 smithi096 ceph-mon[32753]: pgmap v20991: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38967.301174 secs 2024-04-26T22:40:29.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38967.301174 secs 2024-04-26T22:40:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38967.301174 secs 2024-04-26T22:40:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38970.286282 secs 2024-04-26T22:40:30.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:30 smithi194 ceph-mon[33017]: pgmap v20992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38970.286282 secs 2024-04-26T22:40:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:30 smithi033 ceph-mon[27845]: pgmap v20992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38970.286282 secs 2024-04-26T22:40:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:30 smithi096 ceph-mon[32753]: pgmap v20992: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:31.544+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:31.544+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:31.544+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:40:32.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:32 smithi194 ceph-mon[33017]: pgmap v20993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:32 smithi033 ceph-mon[27845]: pgmap v20993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:32.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:32 smithi096 ceph-mon[32753]: pgmap v20993: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:34 smithi033 ceph-mon[27845]: pgmap v20994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:34.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38972.301280 secs 2024-04-26T22:40:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:34 smithi096 ceph-mon[32753]: pgmap v20994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:34.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38972.301280 secs 2024-04-26T22:40:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:34 smithi194 ceph-mon[33017]: pgmap v20994: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:34.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38972.301280 secs 2024-04-26T22:40:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38975.286397 secs 2024-04-26T22:40:35.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38975.286397 secs 2024-04-26T22:40:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38975.286397 secs 2024-04-26T22:40:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:36 smithi033 ceph-mon[27845]: pgmap v20995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:36 smithi096 ceph-mon[32753]: pgmap v20995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:36.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:36 smithi194 ceph-mon[33017]: pgmap v20995: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:38 smithi033 ceph-mon[27845]: pgmap v20996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:38.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:38 smithi096 ceph-mon[32753]: pgmap v20996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:38 smithi194 ceph-mon[33017]: pgmap v20996: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38977.301343 secs 2024-04-26T22:40:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38977.301343 secs 2024-04-26T22:40:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38977.301343 secs 2024-04-26T22:40:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38980.286517 secs 2024-04-26T22:40:40.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:40 smithi033 ceph-mon[27845]: pgmap v20997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38980.286517 secs 2024-04-26T22:40:40.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:40 smithi096 ceph-mon[32753]: pgmap v20997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:40.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38980.286517 secs 2024-04-26T22:40:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:40 smithi194 ceph-mon[33017]: pgmap v20997: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:42 smithi033 ceph-mon[27845]: pgmap v20998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:42.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:42 smithi096 ceph-mon[32753]: pgmap v20998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:42.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:42 smithi194 ceph-mon[33017]: pgmap v20998: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:44 smithi033 ceph-mon[27845]: pgmap v20999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38982.301427 secs 2024-04-26T22:40:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:44 smithi096 ceph-mon[32753]: pgmap v20999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38982.301427 secs 2024-04-26T22:40:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:44 smithi194 ceph-mon[33017]: pgmap v20999: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38982.301427 secs 2024-04-26T22:40:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38985.286660 secs 2024-04-26T22:40:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38985.286660 secs 2024-04-26T22:40:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38985.286660 secs 2024-04-26T22:40:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:46 smithi033 ceph-mon[27845]: pgmap v21000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:46 smithi096 ceph-mon[32753]: pgmap v21000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:46.560+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:46.560+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:40:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:40:46.560+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:40:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:46 smithi194 ceph-mon[33017]: pgmap v21000: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:48 smithi033 ceph-mon[27845]: pgmap v21001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:48 smithi096 ceph-mon[32753]: pgmap v21001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:48 smithi194 ceph-mon[33017]: pgmap v21001: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38987.301484 secs 2024-04-26T22:40:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38987.301484 secs 2024-04-26T22:40:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38987.301484 secs 2024-04-26T22:40:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38990.286781 secs 2024-04-26T22:40:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:50 smithi194 ceph-mon[33017]: pgmap v21002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38990.286781 secs 2024-04-26T22:40:50.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:50 smithi033 ceph-mon[27845]: pgmap v21002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38990.286781 secs 2024-04-26T22:40:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:50 smithi096 ceph-mon[32753]: pgmap v21002: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:52 smithi194 ceph-mon[33017]: pgmap v21003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:52 smithi033 ceph-mon[27845]: pgmap v21003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:52 smithi096 ceph-mon[32753]: pgmap v21003: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:54.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:54 smithi194 ceph-mon[33017]: pgmap v21004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:54.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38992.301547 secs 2024-04-26T22:40:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:54 smithi033 ceph-mon[27845]: pgmap v21004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38992.301547 secs 2024-04-26T22:40:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:54 smithi096 ceph-mon[32753]: pgmap v21004: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38992.301547 secs 2024-04-26T22:40:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 38995.286939 secs 2024-04-26T22:40:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 38995.286939 secs 2024-04-26T22:40:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 38995.286939 secs 2024-04-26T22:40:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:56 smithi194 ceph-mon[33017]: pgmap v21005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:56 smithi033 ceph-mon[27845]: pgmap v21005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:56.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:56 smithi096 ceph-mon[32753]: pgmap v21005: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:40:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:58 smithi194 ceph-mon[33017]: pgmap v21006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:58 smithi033 ceph-mon[27845]: pgmap v21006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:58.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:58 smithi096 ceph-mon[32753]: pgmap v21006: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:40:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:40:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 38997.301606 secs 2024-04-26T22:40:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:40:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 38997.301606 secs 2024-04-26T22:40:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:40:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 38997.301606 secs 2024-04-26T22:41:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39000.286996 secs 2024-04-26T22:41:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:00 smithi194 ceph-mon[33017]: pgmap v21007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39000.286996 secs 2024-04-26T22:41:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:00 smithi033 ceph-mon[27845]: pgmap v21007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39000.286996 secs 2024-04-26T22:41:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:00 smithi096 ceph-mon[32753]: pgmap v21007: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:01.565+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:01.565+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:41:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:01.565+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:02 smithi194 ceph-mon[33017]: pgmap v21008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:02 smithi033 ceph-mon[27845]: pgmap v21008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:02.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:02 smithi096 ceph-mon[32753]: pgmap v21008: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:04 smithi194 ceph-mon[33017]: pgmap v21009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39002.301664 secs 2024-04-26T22:41:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:04 smithi033 ceph-mon[27845]: pgmap v21009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:04.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39002.301664 secs 2024-04-26T22:41:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:04 smithi096 ceph-mon[32753]: pgmap v21009: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39002.301664 secs 2024-04-26T22:41:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39005.287149 secs 2024-04-26T22:41:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39005.287149 secs 2024-04-26T22:41:05.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39005.287149 secs 2024-04-26T22:41:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:06 smithi194 ceph-mon[33017]: pgmap v21010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:06 smithi033 ceph-mon[27845]: pgmap v21010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:06.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:06 smithi096 ceph-mon[32753]: pgmap v21010: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:08 smithi194 ceph-mon[33017]: pgmap v21011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:08 smithi033 ceph-mon[27845]: pgmap v21011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:08.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:08 smithi096 ceph-mon[32753]: pgmap v21011: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39007.301723 secs 2024-04-26T22:41:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39007.301723 secs 2024-04-26T22:41:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39007.301723 secs 2024-04-26T22:41:10.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39010.287286 secs 2024-04-26T22:41:10.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:10 smithi033 ceph-mon[27845]: pgmap v21012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39010.287286 secs 2024-04-26T22:41:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:10 smithi194 ceph-mon[33017]: pgmap v21012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39010.287286 secs 2024-04-26T22:41:10.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:10 smithi096 ceph-mon[32753]: pgmap v21012: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:12 smithi194 ceph-mon[33017]: pgmap v21013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:12 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:12 smithi033 ceph-mon[27845]: pgmap v21013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:12.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:12 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:12 smithi096 ceph-mon[32753]: pgmap v21013: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:12.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:12 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:41:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:13 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:13 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:41:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:41:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:41:13.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:13 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:41:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:14 smithi194 ceph-mon[33017]: pgmap v21014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:14 smithi194 ceph-mon[33017]: pgmap v21015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:14.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39012.301783 secs 2024-04-26T22:41:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:14 smithi033 ceph-mon[27845]: pgmap v21014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:14 smithi033 ceph-mon[27845]: pgmap v21015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39012.301783 secs 2024-04-26T22:41:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:14 smithi096 ceph-mon[32753]: pgmap v21014: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:14 smithi096 ceph-mon[32753]: pgmap v21015: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39012.301783 secs 2024-04-26T22:41:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39015.287379 secs 2024-04-26T22:41:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39015.287379 secs 2024-04-26T22:41:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39015.287379 secs 2024-04-26T22:41:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:16.580+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:16.580+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:16.581+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:41:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:16 smithi194 ceph-mon[33017]: pgmap v21016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:16 smithi033 ceph-mon[27845]: pgmap v21016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:16 smithi096 ceph-mon[32753]: pgmap v21016: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:18 smithi194 ceph-mon[33017]: pgmap v21017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:18 smithi033 ceph-mon[27845]: pgmap v21017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:18 smithi096 ceph-mon[32753]: pgmap v21017: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39017.301838 secs 2024-04-26T22:41:19.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39017.301838 secs 2024-04-26T22:41:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39017.301838 secs 2024-04-26T22:41:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:20 smithi194 ceph-mon[33017]: pgmap v21018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39020.287503 secs 2024-04-26T22:41:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:20 smithi033 ceph-mon[27845]: pgmap v21018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:20.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39020.287503 secs 2024-04-26T22:41:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:20 smithi096 ceph-mon[32753]: pgmap v21018: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39020.287503 secs 2024-04-26T22:41:21.827 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0058000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7feef0015010 tx=0x7feef0058000 comp rx=0 tx=0).stop 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 msgr2=0x7feee40b3660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.827+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:41:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.828+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:41:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:21.829+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0022040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:41:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:22 smithi194 ceph-mon[33017]: pgmap v21019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:22 smithi033 ceph-mon[27845]: pgmap v21019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:22 smithi096 ceph-mon[32753]: pgmap v21019: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:41:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:24 smithi194 ceph-mon[33017]: pgmap v21020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:41:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39022.301898 secs 2024-04-26T22:41:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:24 smithi033 ceph-mon[27845]: pgmap v21020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:41:24.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39022.301898 secs 2024-04-26T22:41:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:24 smithi096 ceph-mon[32753]: pgmap v21020: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:41:24.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39022.301898 secs 2024-04-26T22:41:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39025.287646 secs 2024-04-26T22:41:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39025.287646 secs 2024-04-26T22:41:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39025.287646 secs 2024-04-26T22:41:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:26 smithi194 ceph-mon[33017]: pgmap v21021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:26 smithi033 ceph-mon[27845]: pgmap v21021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:26 smithi096 ceph-mon[32753]: pgmap v21021: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.442+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc006570 con 0x564385821900 2024-04-26T22:41:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.442+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed0003690 con 0x7fef00077780 2024-04-26T22:41:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.443+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 304 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7feed0003690 con 0x564385821900 2024-04-26T22:41:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.443+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x564385821900 2024-04-26T22:41:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.443+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 304 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7feecc006570 con 0x7fef00077780 2024-04-26T22:41:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.443+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:41:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.443+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 305 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x564385821900 2024-04-26T22:41:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:28.443+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 305 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:41:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:28 smithi194 ceph-mon[33017]: pgmap v21022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:28 smithi033 ceph-mon[27845]: pgmap v21022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:28.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:28 smithi096 ceph-mon[32753]: pgmap v21022: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39027.301959 secs 2024-04-26T22:41:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39027.301959 secs 2024-04-26T22:41:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39027.301959 secs 2024-04-26T22:41:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:30 smithi033 ceph-mon[27845]: pgmap v21023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39030.287784 secs 2024-04-26T22:41:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:30 smithi096 ceph-mon[32753]: pgmap v21023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:30.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39030.287784 secs 2024-04-26T22:41:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:30 smithi194 ceph-mon[33017]: pgmap v21023: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39030.287784 secs 2024-04-26T22:41:31.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:31.582+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:31.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:31.582+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:31.583 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:31.582+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:41:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:32 smithi033 ceph-mon[27845]: pgmap v21024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:32 smithi096 ceph-mon[32753]: pgmap v21024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:32 smithi194 ceph-mon[33017]: pgmap v21024: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:34 smithi033 ceph-mon[27845]: pgmap v21025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39032.302014 secs 2024-04-26T22:41:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:34 smithi096 ceph-mon[32753]: pgmap v21025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39032.302014 secs 2024-04-26T22:41:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:34 smithi194 ceph-mon[33017]: pgmap v21025: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39032.302014 secs 2024-04-26T22:41:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39035.287905 secs 2024-04-26T22:41:35.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:35 smithi033 ceph-mon[27845]: pgmap v21026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39035.287905 secs 2024-04-26T22:41:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:35 smithi096 ceph-mon[32753]: pgmap v21026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39035.287905 secs 2024-04-26T22:41:35.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:35 smithi194 ceph-mon[33017]: pgmap v21026: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:38.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:37 smithi194 ceph-mon[33017]: pgmap v21027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:38.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:37 smithi033 ceph-mon[27845]: pgmap v21027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:38.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:37 smithi096 ceph-mon[32753]: pgmap v21027: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39037.302076 secs 2024-04-26T22:41:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39037.302076 secs 2024-04-26T22:41:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39037.302076 secs 2024-04-26T22:41:40.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:39 smithi033 ceph-mon[27845]: pgmap v21028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:39 smithi194 ceph-mon[33017]: pgmap v21028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:40.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:39 smithi096 ceph-mon[32753]: pgmap v21028: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39040.288031 secs 2024-04-26T22:41:41.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39040.288031 secs 2024-04-26T22:41:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39040.288031 secs 2024-04-26T22:41:42.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:41 smithi194 ceph-mon[33017]: pgmap v21029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:41 smithi033 ceph-mon[27845]: pgmap v21029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:42.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:41 smithi096 ceph-mon[32753]: pgmap v21029: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:43 smithi194 ceph-mon[33017]: pgmap v21030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:44.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39042.302127 secs 2024-04-26T22:41:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:43 smithi033 ceph-mon[27845]: pgmap v21030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39042.302127 secs 2024-04-26T22:41:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:43 smithi096 ceph-mon[32753]: pgmap v21030: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39042.302127 secs 2024-04-26T22:41:45.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39045.288156 secs 2024-04-26T22:41:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39045.288156 secs 2024-04-26T22:41:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39045.288156 secs 2024-04-26T22:41:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:45 smithi194 ceph-mon[33017]: pgmap v21031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:45 smithi033 ceph-mon[27845]: pgmap v21031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:46.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:45 smithi096 ceph-mon[32753]: pgmap v21031: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:46.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:46.596+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:46.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:46.596+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:41:46.597 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:41:46.596+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:41:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:47 smithi194 ceph-mon[33017]: pgmap v21032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:47 smithi033 ceph-mon[27845]: pgmap v21032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:48.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:47 smithi096 ceph-mon[32753]: pgmap v21032: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39047.302188 secs 2024-04-26T22:41:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39047.302188 secs 2024-04-26T22:41:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39047.302188 secs 2024-04-26T22:41:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:50 smithi033 ceph-mon[27845]: pgmap v21033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:50.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39050.288272 secs 2024-04-26T22:41:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:50 smithi194 ceph-mon[33017]: pgmap v21033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39050.288272 secs 2024-04-26T22:41:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:50 smithi096 ceph-mon[32753]: pgmap v21033: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39050.288272 secs 2024-04-26T22:41:52.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:52 smithi194 ceph-mon[33017]: pgmap v21034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:52 smithi033 ceph-mon[27845]: pgmap v21034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:52.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:52 smithi096 ceph-mon[32753]: pgmap v21034: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:54 smithi194 ceph-mon[33017]: pgmap v21035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39052.302261 secs 2024-04-26T22:41:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:54 smithi033 ceph-mon[27845]: pgmap v21035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:54.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39052.302261 secs 2024-04-26T22:41:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:54 smithi096 ceph-mon[32753]: pgmap v21035: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39052.302261 secs 2024-04-26T22:41:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39055.288402 secs 2024-04-26T22:41:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39055.288402 secs 2024-04-26T22:41:55.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39055.288402 secs 2024-04-26T22:41:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:56 smithi194 ceph-mon[33017]: pgmap v21036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:56 smithi033 ceph-mon[27845]: pgmap v21036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:56.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:56 smithi096 ceph-mon[32753]: pgmap v21036: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:41:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:58 smithi194 ceph-mon[33017]: pgmap v21037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:58 smithi033 ceph-mon[27845]: pgmap v21037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:58.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:58 smithi096 ceph-mon[32753]: pgmap v21037: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:41:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:41:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39057.302345 secs 2024-04-26T22:41:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:41:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39057.302345 secs 2024-04-26T22:41:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:41:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39057.302345 secs 2024-04-26T22:42:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:00 smithi194 ceph-mon[33017]: pgmap v21038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:00.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39060.288520 secs 2024-04-26T22:42:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:00 smithi033 ceph-mon[27845]: pgmap v21038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:00.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39060.288520 secs 2024-04-26T22:42:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:00 smithi096 ceph-mon[32753]: pgmap v21038: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:00.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39060.288520 secs 2024-04-26T22:42:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:01.609+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:01.609+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:01.609+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:42:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:02 smithi194 ceph-mon[33017]: pgmap v21039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:02 smithi033 ceph-mon[27845]: pgmap v21039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:02.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:02 smithi096 ceph-mon[32753]: pgmap v21039: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:04 smithi194 ceph-mon[33017]: pgmap v21040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39062.302428 secs 2024-04-26T22:42:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:04 smithi033 ceph-mon[27845]: pgmap v21040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:04.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39062.302428 secs 2024-04-26T22:42:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:04 smithi096 ceph-mon[32753]: pgmap v21040: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39062.302428 secs 2024-04-26T22:42:05.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39065.288644 secs 2024-04-26T22:42:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39065.288644 secs 2024-04-26T22:42:05.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39065.288644 secs 2024-04-26T22:42:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:06 smithi194 ceph-mon[33017]: pgmap v21041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:06 smithi033 ceph-mon[27845]: pgmap v21041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:06.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:06 smithi096 ceph-mon[32753]: pgmap v21041: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:08 smithi194 ceph-mon[33017]: pgmap v21042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:08 smithi033 ceph-mon[27845]: pgmap v21042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:08.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:08 smithi096 ceph-mon[32753]: pgmap v21042: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39067.302513 secs 2024-04-26T22:42:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39067.302513 secs 2024-04-26T22:42:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39067.302513 secs 2024-04-26T22:42:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:10 smithi194 ceph-mon[33017]: pgmap v21043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39070.288778 secs 2024-04-26T22:42:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:10 smithi033 ceph-mon[27845]: pgmap v21043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:10.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39070.288778 secs 2024-04-26T22:42:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:10 smithi096 ceph-mon[32753]: pgmap v21043: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:10.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39070.288778 secs 2024-04-26T22:42:12.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:12 smithi194 ceph-mon[33017]: pgmap v21044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:12 smithi033 ceph-mon[27845]: pgmap v21044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:12.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:12 smithi096 ceph-mon[32753]: pgmap v21044: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:13 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:13 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:13.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:13 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:42:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:14 smithi033 ceph-mon[27845]: pgmap v21045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39072.302598 secs 2024-04-26T22:42:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:14 smithi096 ceph-mon[32753]: pgmap v21045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:14.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39072.302598 secs 2024-04-26T22:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:14 smithi194 ceph-mon[33017]: pgmap v21045: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39072.302598 secs 2024-04-26T22:42:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39075.288901 secs 2024-04-26T22:42:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:15 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:15 smithi033 ceph-mon[27845]: pgmap v21046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:15 smithi033 ceph-mon[27845]: pgmap v21047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:42:15.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:15 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:42:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39075.288901 secs 2024-04-26T22:42:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:15 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:15 smithi096 ceph-mon[32753]: pgmap v21046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:15 smithi096 ceph-mon[32753]: pgmap v21047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:42:15.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:15 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:42:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39075.288901 secs 2024-04-26T22:42:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:42:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:15 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:42:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:15 smithi194 ceph-mon[33017]: pgmap v21046: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:15 smithi194 ceph-mon[33017]: pgmap v21047: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:42:15.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:15 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:42:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:16.624+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:16.625+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:16.625+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:42:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:16 smithi194 ceph-mon[33017]: pgmap v21048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:16 smithi033 ceph-mon[27845]: pgmap v21048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:16.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:16 smithi096 ceph-mon[32753]: pgmap v21048: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:18 smithi194 ceph-mon[33017]: pgmap v21049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:18 smithi033 ceph-mon[27845]: pgmap v21049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:18.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:18 smithi096 ceph-mon[32753]: pgmap v21049: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39077.302678 secs 2024-04-26T22:42:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39077.302678 secs 2024-04-26T22:42:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39077.302678 secs 2024-04-26T22:42:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39080.289039 secs 2024-04-26T22:42:20.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:20 smithi194 ceph-mon[33017]: pgmap v21050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39080.289039 secs 2024-04-26T22:42:20.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:20 smithi033 ceph-mon[27845]: pgmap v21050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39080.289039 secs 2024-04-26T22:42:20.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:20 smithi096 ceph-mon[32753]: pgmap v21050: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:22 smithi194 ceph-mon[33017]: pgmap v21051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:22 smithi033 ceph-mon[27845]: pgmap v21051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:22 smithi096 ceph-mon[32753]: pgmap v21051: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39082.302759 secs 2024-04-26T22:42:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39082.302759 secs 2024-04-26T22:42:24.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39082.302759 secs 2024-04-26T22:42:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:24 smithi194 ceph-mon[33017]: pgmap v21052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:42:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:24 smithi033 ceph-mon[27845]: pgmap v21052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:42:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:24 smithi096 ceph-mon[32753]: pgmap v21052: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:42:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39085.289188 secs 2024-04-26T22:42:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39085.289188 secs 2024-04-26T22:42:26.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39085.289188 secs 2024-04-26T22:42:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:26 smithi194 ceph-mon[33017]: pgmap v21053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:26 smithi033 ceph-mon[27845]: pgmap v21053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:27.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:26 smithi096 ceph-mon[32753]: pgmap v21053: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:28 smithi194 ceph-mon[33017]: pgmap v21054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:28 smithi033 ceph-mon[27845]: pgmap v21054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:28.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:28 smithi096 ceph-mon[32753]: pgmap v21054: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:29.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39087.302841 secs 2024-04-26T22:42:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39087.302841 secs 2024-04-26T22:42:29.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39087.302841 secs 2024-04-26T22:42:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39090.289266 secs 2024-04-26T22:42:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:30 smithi194 ceph-mon[33017]: pgmap v21055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39090.289266 secs 2024-04-26T22:42:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:30 smithi033 ceph-mon[27845]: pgmap v21055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39090.289266 secs 2024-04-26T22:42:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:30 smithi096 ceph-mon[32753]: pgmap v21055: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:31.630+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:31.630+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:42:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:31.630+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:32 smithi194 ceph-mon[33017]: pgmap v21056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:32 smithi033 ceph-mon[27845]: pgmap v21056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:32.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:32 smithi096 ceph-mon[32753]: pgmap v21056: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39092.302925 secs 2024-04-26T22:42:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39092.302925 secs 2024-04-26T22:42:34.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39092.302925 secs 2024-04-26T22:42:35.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:34 smithi096 ceph-mon[32753]: pgmap v21057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:34 smithi194 ceph-mon[33017]: pgmap v21057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:34 smithi033 ceph-mon[27845]: pgmap v21057: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39095.289383 secs 2024-04-26T22:42:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39095.289383 secs 2024-04-26T22:42:36.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39095.289383 secs 2024-04-26T22:42:37.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:36 smithi033 ceph-mon[27845]: pgmap v21058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:36 smithi194 ceph-mon[33017]: pgmap v21058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:37.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:36 smithi096 ceph-mon[32753]: pgmap v21058: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:38 smithi194 ceph-mon[33017]: pgmap v21059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:38 smithi033 ceph-mon[27845]: pgmap v21059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:38.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:38 smithi096 ceph-mon[32753]: pgmap v21059: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39097.303025 secs 2024-04-26T22:42:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39097.303025 secs 2024-04-26T22:42:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39097.303025 secs 2024-04-26T22:42:40.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39100.289444 secs 2024-04-26T22:42:40.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:40 smithi194 ceph-mon[33017]: pgmap v21060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:40.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39100.289444 secs 2024-04-26T22:42:40.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:40 smithi033 ceph-mon[27845]: pgmap v21060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39100.289444 secs 2024-04-26T22:42:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:40 smithi096 ceph-mon[32753]: pgmap v21060: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:42 smithi194 ceph-mon[33017]: pgmap v21061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:42 smithi033 ceph-mon[27845]: pgmap v21061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:42.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:42 smithi096 ceph-mon[32753]: pgmap v21061: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39102.303112 secs 2024-04-26T22:42:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39102.303112 secs 2024-04-26T22:42:44.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39102.303112 secs 2024-04-26T22:42:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:44 smithi194 ceph-mon[33017]: pgmap v21062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:44 smithi033 ceph-mon[27845]: pgmap v21062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:44 smithi096 ceph-mon[32753]: pgmap v21062: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:46.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39105.289518 secs 2024-04-26T22:42:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39105.289518 secs 2024-04-26T22:42:46.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39105.289518 secs 2024-04-26T22:42:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:46.643+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:46.643+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:42:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:42:46.643+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:42:47.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:46 smithi194 ceph-mon[33017]: pgmap v21063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:47.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:46 smithi033 ceph-mon[27845]: pgmap v21063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:47.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:46 smithi096 ceph-mon[32753]: pgmap v21063: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:48 smithi194 ceph-mon[33017]: pgmap v21064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:48 smithi033 ceph-mon[27845]: pgmap v21064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:48 smithi096 ceph-mon[32753]: pgmap v21064: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39107.303200 secs 2024-04-26T22:42:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39107.303200 secs 2024-04-26T22:42:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39107.303200 secs 2024-04-26T22:42:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39110.289600 secs 2024-04-26T22:42:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:50 smithi194 ceph-mon[33017]: pgmap v21065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39110.289600 secs 2024-04-26T22:42:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:50 smithi033 ceph-mon[27845]: pgmap v21065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39110.289600 secs 2024-04-26T22:42:50.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:50 smithi096 ceph-mon[32753]: pgmap v21065: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:42:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:52 smithi194 ceph-mon[33017]: pgmap v21066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:52 smithi033 ceph-mon[27845]: pgmap v21066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:52 smithi096 ceph-mon[32753]: pgmap v21066: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39112.303301 secs 2024-04-26T22:42:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39112.303301 secs 2024-04-26T22:42:54.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39112.303301 secs 2024-04-26T22:42:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:54 smithi194 ceph-mon[33017]: pgmap v21067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:54 smithi033 ceph-mon[27845]: pgmap v21067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:54 smithi096 ceph-mon[32753]: pgmap v21067: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:56.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39115.289717 secs 2024-04-26T22:42:56.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39115.289717 secs 2024-04-26T22:42:56.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39115.289717 secs 2024-04-26T22:42:57.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:56 smithi194 ceph-mon[33017]: pgmap v21068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:56 smithi033 ceph-mon[27845]: pgmap v21068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:57.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:56 smithi096 ceph-mon[32753]: pgmap v21068: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:42:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:58 smithi194 ceph-mon[33017]: pgmap v21069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:58 smithi033 ceph-mon[27845]: pgmap v21069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:58.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:58 smithi096 ceph-mon[32753]: pgmap v21069: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:42:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:42:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39117.303385 secs 2024-04-26T22:42:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:42:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39117.303385 secs 2024-04-26T22:42:59.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:42:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39117.303385 secs 2024-04-26T22:43:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39120.289794 secs 2024-04-26T22:43:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:00 smithi194 ceph-mon[33017]: pgmap v21070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39120.289794 secs 2024-04-26T22:43:00.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:00 smithi033 ceph-mon[27845]: pgmap v21070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39120.289794 secs 2024-04-26T22:43:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:00 smithi096 ceph-mon[32753]: pgmap v21070: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:01.657+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:01.657+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:01.657+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:43:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:02 smithi194 ceph-mon[33017]: pgmap v21071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:02 smithi033 ceph-mon[27845]: pgmap v21071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:02.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:02 smithi096 ceph-mon[32753]: pgmap v21071: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39122.303469 secs 2024-04-26T22:43:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39122.303469 secs 2024-04-26T22:43:04.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39122.303469 secs 2024-04-26T22:43:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:04 smithi096 ceph-mon[32753]: pgmap v21072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:04 smithi194 ceph-mon[33017]: pgmap v21072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:04 smithi033 ceph-mon[27845]: pgmap v21072: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:06.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39125.289913 secs 2024-04-26T22:43:06.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39125.289913 secs 2024-04-26T22:43:06.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39125.289913 secs 2024-04-26T22:43:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:06 smithi194 ceph-mon[33017]: pgmap v21073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:06 smithi033 ceph-mon[27845]: pgmap v21073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:06 smithi096 ceph-mon[32753]: pgmap v21073: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:08 smithi194 ceph-mon[33017]: pgmap v21074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:43:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:08 smithi033 ceph-mon[27845]: pgmap v21074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:43:08.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:08 smithi096 ceph-mon[32753]: pgmap v21074: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:43:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39127.303551 secs 2024-04-26T22:43:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39127.303551 secs 2024-04-26T22:43:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39127.303551 secs 2024-04-26T22:43:10.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39130.290050 secs 2024-04-26T22:43:10.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:10 smithi194 ceph-mon[33017]: pgmap v21075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:43:10.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39130.290050 secs 2024-04-26T22:43:10.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:10 smithi033 ceph-mon[27845]: pgmap v21075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:43:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39130.290050 secs 2024-04-26T22:43:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:10 smithi096 ceph-mon[32753]: pgmap v21075: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:43:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:12 smithi194 ceph-mon[33017]: pgmap v21076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:43:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:12 smithi033 ceph-mon[27845]: pgmap v21076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:43:12.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:12 smithi096 ceph-mon[32753]: pgmap v21076: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:43:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39132.303637 secs 2024-04-26T22:43:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39132.303637 secs 2024-04-26T22:43:14.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39132.303637 secs 2024-04-26T22:43:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:14 smithi194 ceph-mon[33017]: pgmap v21077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:14 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:14 smithi033 ceph-mon[27845]: pgmap v21077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:14 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:14 smithi096 ceph-mon[32753]: pgmap v21077: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T22:43:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:14 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:43:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39135.290173 secs 2024-04-26T22:43:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39135.290173 secs 2024-04-26T22:43:16.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39135.290173 secs 2024-04-26T22:43:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:16.672+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:16.672+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:43:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:16.672+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:16 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:16 smithi194 ceph-mon[33017]: pgmap v21078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:16 smithi194 ceph-mon[33017]: pgmap v21079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:17.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:16 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:43:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:16 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:16 smithi033 ceph-mon[27845]: pgmap v21078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:16 smithi033 ceph-mon[27845]: pgmap v21079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:17.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:16 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:16 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:16 smithi096 ceph-mon[32753]: pgmap v21078: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:16 smithi096 ceph-mon[32753]: pgmap v21079: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:17.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:16 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:43:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39137.303720 secs 2024-04-26T22:43:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:19 smithi033 ceph-mon[27845]: pgmap v21080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39137.303720 secs 2024-04-26T22:43:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:19 smithi096 ceph-mon[32753]: pgmap v21080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39137.303720 secs 2024-04-26T22:43:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:19 smithi194 ceph-mon[33017]: pgmap v21080: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39140.290274 secs 2024-04-26T22:43:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39140.290274 secs 2024-04-26T22:43:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39140.290274 secs 2024-04-26T22:43:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:21 smithi033 ceph-mon[27845]: pgmap v21081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:21 smithi096 ceph-mon[32753]: pgmap v21081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:21 smithi194 ceph-mon[33017]: pgmap v21081: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:23.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:23 smithi033 ceph-mon[27845]: pgmap v21082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:23.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:23 smithi096 ceph-mon[32753]: pgmap v21082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:23 smithi194 ceph-mon[33017]: pgmap v21082: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:43:24.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39142.303804 secs 2024-04-26T22:43:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39142.303804 secs 2024-04-26T22:43:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39142.303804 secs 2024-04-26T22:43:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39145.290377 secs 2024-04-26T22:43:25.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:25 smithi033 ceph-mon[27845]: pgmap v21083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:43:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39145.290377 secs 2024-04-26T22:43:25.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:25 smithi096 ceph-mon[32753]: pgmap v21083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:43:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39145.290377 secs 2024-04-26T22:43:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:25 smithi194 ceph-mon[33017]: pgmap v21083: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:43:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:27 smithi033 ceph-mon[27845]: pgmap v21084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:43:27.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:27 smithi096 ceph-mon[32753]: pgmap v21084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:43:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:27 smithi194 ceph-mon[33017]: pgmap v21084: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:43:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39147.303891 secs 2024-04-26T22:43:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:29 smithi033 ceph-mon[27845]: pgmap v21085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39147.303891 secs 2024-04-26T22:43:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:29 smithi096 ceph-mon[32753]: pgmap v21085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39147.303891 secs 2024-04-26T22:43:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:29 smithi194 ceph-mon[33017]: pgmap v21085: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:30.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39150.290448 secs 2024-04-26T22:43:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39150.290448 secs 2024-04-26T22:43:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39150.290448 secs 2024-04-26T22:43:31.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:31 smithi096 ceph-mon[32753]: pgmap v21086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:31 smithi194 ceph-mon[33017]: pgmap v21086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:31 smithi033 ceph-mon[27845]: pgmap v21086: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:31.688+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:31.688+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:43:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:31.688+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:33.616 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:33 smithi194 ceph-mon[33017]: pgmap v21087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:33.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:33 smithi033 ceph-mon[27845]: pgmap v21087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:33 smithi096 ceph-mon[32753]: pgmap v21087: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39152.303996 secs 2024-04-26T22:43:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39152.303996 secs 2024-04-26T22:43:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39152.303996 secs 2024-04-26T22:43:35.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39155.290575 secs 2024-04-26T22:43:35.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:35 smithi096 ceph-mon[32753]: pgmap v21088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39155.290575 secs 2024-04-26T22:43:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:35 smithi194 ceph-mon[33017]: pgmap v21088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39155.290575 secs 2024-04-26T22:43:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:35 smithi033 ceph-mon[27845]: pgmap v21088: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:37 smithi194 ceph-mon[33017]: pgmap v21089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:37.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:37 smithi033 ceph-mon[27845]: pgmap v21089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:37 smithi096 ceph-mon[32753]: pgmap v21089: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39157.304081 secs 2024-04-26T22:43:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:39 smithi194 ceph-mon[33017]: pgmap v21090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39157.304081 secs 2024-04-26T22:43:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:39 smithi033 ceph-mon[27845]: pgmap v21090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39157.304081 secs 2024-04-26T22:43:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:39 smithi096 ceph-mon[32753]: pgmap v21090: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:40.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39160.290698 secs 2024-04-26T22:43:40.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39160.290698 secs 2024-04-26T22:43:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39160.290698 secs 2024-04-26T22:43:41.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:41 smithi194 ceph-mon[33017]: pgmap v21091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:41 smithi033 ceph-mon[27845]: pgmap v21091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:41.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:41 smithi096 ceph-mon[32753]: pgmap v21091: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:43 smithi194 ceph-mon[33017]: pgmap v21092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:43 smithi033 ceph-mon[27845]: pgmap v21092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:43 smithi096 ceph-mon[32753]: pgmap v21092: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39162.304166 secs 2024-04-26T22:43:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39162.304166 secs 2024-04-26T22:43:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39162.304166 secs 2024-04-26T22:43:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39165.290825 secs 2024-04-26T22:43:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:45 smithi194 ceph-mon[33017]: pgmap v21093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39165.290825 secs 2024-04-26T22:43:45.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:45 smithi033 ceph-mon[27845]: pgmap v21093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39165.290825 secs 2024-04-26T22:43:45.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:45 smithi096 ceph-mon[32753]: pgmap v21093: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:46.702+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:46.702+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:43:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:43:46.702+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:43:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:47 smithi194 ceph-mon[33017]: pgmap v21094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:47 smithi033 ceph-mon[27845]: pgmap v21094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:47.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:47 smithi096 ceph-mon[32753]: pgmap v21094: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39167.304250 secs 2024-04-26T22:43:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:49 smithi194 ceph-mon[33017]: pgmap v21095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39167.304250 secs 2024-04-26T22:43:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:49 smithi033 ceph-mon[27845]: pgmap v21095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39167.304250 secs 2024-04-26T22:43:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:49 smithi096 ceph-mon[32753]: pgmap v21095: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:50 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39170.290967 secs 2024-04-26T22:43:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:50 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39170.290967 secs 2024-04-26T22:43:50.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:50 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39170.290967 secs 2024-04-26T22:43:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:51 smithi194 ceph-mon[33017]: pgmap v21096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:51 smithi033 ceph-mon[27845]: pgmap v21096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:51 smithi096 ceph-mon[32753]: pgmap v21096: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:53 smithi194 ceph-mon[33017]: pgmap v21097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:53 smithi033 ceph-mon[27845]: pgmap v21097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:53 smithi096 ceph-mon[32753]: pgmap v21097: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39172.304337 secs 2024-04-26T22:43:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39172.304337 secs 2024-04-26T22:43:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39172.304337 secs 2024-04-26T22:43:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:55 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39175.291085 secs 2024-04-26T22:43:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:55 smithi194 ceph-mon[33017]: pgmap v21098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:55 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39175.291085 secs 2024-04-26T22:43:55.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:55 smithi033 ceph-mon[27845]: pgmap v21098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:55 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39175.291085 secs 2024-04-26T22:43:55.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:55 smithi096 ceph-mon[32753]: pgmap v21098: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:57 smithi194 ceph-mon[33017]: pgmap v21099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:57 smithi033 ceph-mon[27845]: pgmap v21099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:57 smithi096 ceph-mon[32753]: pgmap v21099: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:43:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39177.304419 secs 2024-04-26T22:43:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:43:59 smithi194 ceph-mon[33017]: pgmap v21100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39177.304419 secs 2024-04-26T22:43:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:43:59 smithi033 ceph-mon[27845]: pgmap v21100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:43:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39177.304419 secs 2024-04-26T22:43:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:43:59 smithi096 ceph-mon[32753]: pgmap v21100: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:00 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39180.291197 secs 2024-04-26T22:44:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:00 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39180.291197 secs 2024-04-26T22:44:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:00 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39180.291197 secs 2024-04-26T22:44:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:01 smithi194 ceph-mon[33017]: pgmap v21101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:01 smithi033 ceph-mon[27845]: pgmap v21101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:01 smithi096 ceph-mon[32753]: pgmap v21101: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:01.705+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:01.705+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:44:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:01.705+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:03 smithi194 ceph-mon[33017]: pgmap v21102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:03 smithi033 ceph-mon[27845]: pgmap v21102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:03 smithi096 ceph-mon[32753]: pgmap v21102: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39182.304478 secs 2024-04-26T22:44:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39182.304478 secs 2024-04-26T22:44:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39182.304478 secs 2024-04-26T22:44:05.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:05 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39185.291319 secs 2024-04-26T22:44:05.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:05 smithi096 ceph-mon[32753]: pgmap v21103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:05 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39185.291319 secs 2024-04-26T22:44:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:05 smithi194 ceph-mon[33017]: pgmap v21103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:05 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39185.291319 secs 2024-04-26T22:44:05.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:05 smithi033 ceph-mon[27845]: pgmap v21103: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:07 smithi194 ceph-mon[33017]: pgmap v21104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:07 smithi033 ceph-mon[27845]: pgmap v21104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:07 smithi096 ceph-mon[32753]: pgmap v21104: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39187.304563 secs 2024-04-26T22:44:09.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:09 smithi033 ceph-mon[27845]: pgmap v21105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39187.304563 secs 2024-04-26T22:44:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:09 smithi194 ceph-mon[33017]: pgmap v21105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39187.304563 secs 2024-04-26T22:44:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:09 smithi096 ceph-mon[32753]: pgmap v21105: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:10 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39190.291435 secs 2024-04-26T22:44:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:10 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39190.291435 secs 2024-04-26T22:44:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:10 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39190.291435 secs 2024-04-26T22:44:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:11 smithi194 ceph-mon[33017]: pgmap v21106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:11 smithi033 ceph-mon[27845]: pgmap v21106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:11 smithi096 ceph-mon[32753]: pgmap v21106: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:13 smithi033 ceph-mon[27845]: pgmap v21107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:13 smithi096 ceph-mon[32753]: pgmap v21107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:13 smithi194 ceph-mon[33017]: pgmap v21107: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39192.304651 secs 2024-04-26T22:44:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39192.304651 secs 2024-04-26T22:44:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39192.304651 secs 2024-04-26T22:44:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:15 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39195.291505 secs 2024-04-26T22:44:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:15 smithi033 ceph-mon[27845]: pgmap v21108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:15 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39195.291505 secs 2024-04-26T22:44:15.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:15 smithi096 ceph-mon[32753]: pgmap v21108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:15 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39195.291505 secs 2024-04-26T22:44:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:15 smithi194 ceph-mon[33017]: pgmap v21108: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:16.721+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:16.721+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:44:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:16.721+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:17 smithi033 ceph-mon[27845]: pgmap v21109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:17 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:17 smithi096 ceph-mon[32753]: pgmap v21109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:17 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:17 smithi194 ceph-mon[33017]: pgmap v21109: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:17 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:44:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:18.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:44:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:18.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:44:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:44:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:44:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:44:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:19 smithi033 ceph-mon[27845]: pgmap v21110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:19 smithi033 ceph-mon[27845]: pgmap v21111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39197.304733 secs 2024-04-26T22:44:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:19 smithi096 ceph-mon[32753]: pgmap v21110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:19 smithi096 ceph-mon[32753]: pgmap v21111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39197.304733 secs 2024-04-26T22:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:19 smithi194 ceph-mon[33017]: pgmap v21110: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:19 smithi194 ceph-mon[33017]: pgmap v21111: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39197.304733 secs 2024-04-26T22:44:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:20 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39200.291577 secs 2024-04-26T22:44:20.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39200.291577 secs 2024-04-26T22:44:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39200.291577 secs 2024-04-26T22:44:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:21 smithi033 ceph-mon[27845]: pgmap v21112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:21 smithi096 ceph-mon[32753]: pgmap v21112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:21 smithi194 ceph-mon[33017]: pgmap v21112: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:22 smithi194 ceph-mon[33017]: pgmap v21113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:22 smithi033 ceph-mon[27845]: pgmap v21113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:22 smithi096 ceph-mon[32753]: pgmap v21113: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39202.304821 secs 2024-04-26T22:44:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39202.304821 secs 2024-04-26T22:44:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39202.304821 secs 2024-04-26T22:44:25.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:24 smithi033 ceph-mon[27845]: pgmap v21114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:24 smithi194 ceph-mon[33017]: pgmap v21114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:24 smithi096 ceph-mon[32753]: pgmap v21114: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39205.291650 secs 2024-04-26T22:44:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39205.291650 secs 2024-04-26T22:44:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39205.291650 secs 2024-04-26T22:44:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:26 smithi194 ceph-mon[33017]: pgmap v21115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:26 smithi033 ceph-mon[27845]: pgmap v21115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:26 smithi096 ceph-mon[32753]: pgmap v21115: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:44:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:28.472+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:44:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:28.472+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T22:44:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:28.473+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 306 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T22:44:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:28.473+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 306 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:44:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:28 smithi194 ceph-mon[33017]: pgmap v21116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:44:29.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39207.304908 secs 2024-04-26T22:44:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:28 smithi033 ceph-mon[27845]: pgmap v21116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:44:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39207.304908 secs 2024-04-26T22:44:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:28 smithi096 ceph-mon[32753]: pgmap v21116: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:44:29.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39207.304908 secs 2024-04-26T22:44:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39210.291783 secs 2024-04-26T22:44:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:30 smithi194 ceph-mon[33017]: pgmap v21117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39210.291783 secs 2024-04-26T22:44:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:30 smithi033 ceph-mon[27845]: pgmap v21117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:31.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39210.291783 secs 2024-04-26T22:44:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:30 smithi096 ceph-mon[32753]: pgmap v21117: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:31.723+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:31.723+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:31.723+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:44:33.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:32 smithi194 ceph-mon[33017]: pgmap v21118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:32 smithi033 ceph-mon[27845]: pgmap v21118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:32 smithi096 ceph-mon[32753]: pgmap v21118: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39212.305004 secs 2024-04-26T22:44:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39212.305004 secs 2024-04-26T22:44:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39212.305004 secs 2024-04-26T22:44:35.086 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:34 smithi096 ceph-mon[32753]: pgmap v21119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:35.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:34 smithi194 ceph-mon[33017]: pgmap v21119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:34 smithi033 ceph-mon[27845]: pgmap v21119: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:35 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39215.291864 secs 2024-04-26T22:44:36.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:35 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39215.291864 secs 2024-04-26T22:44:36.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:35 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39215.291864 secs 2024-04-26T22:44:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:36 smithi194 ceph-mon[33017]: pgmap v21120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:37.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:36 smithi033 ceph-mon[27845]: pgmap v21120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:37.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:36 smithi096 ceph-mon[32753]: pgmap v21120: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:38 smithi194 ceph-mon[33017]: pgmap v21121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39217.305082 secs 2024-04-26T22:44:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:38 smithi033 ceph-mon[27845]: pgmap v21121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39217.305082 secs 2024-04-26T22:44:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:38 smithi096 ceph-mon[32753]: pgmap v21121: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:39.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39217.305082 secs 2024-04-26T22:44:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:40 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39220.291986 secs 2024-04-26T22:44:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:40 smithi194 ceph-mon[33017]: pgmap v21122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:40 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39220.291986 secs 2024-04-26T22:44:41.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:40 smithi033 ceph-mon[27845]: pgmap v21122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:41.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:40 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39220.291986 secs 2024-04-26T22:44:41.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:40 smithi096 ceph-mon[32753]: pgmap v21122: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:43.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:42 smithi194 ceph-mon[33017]: pgmap v21123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:43.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:42 smithi033 ceph-mon[27845]: pgmap v21123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:43.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:42 smithi096 ceph-mon[32753]: pgmap v21123: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39222.305144 secs 2024-04-26T22:44:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39222.305144 secs 2024-04-26T22:44:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39222.305144 secs 2024-04-26T22:44:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:44 smithi194 ceph-mon[33017]: pgmap v21124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:44 smithi033 ceph-mon[27845]: pgmap v21124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:45.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:44 smithi096 ceph-mon[32753]: pgmap v21124: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:46.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:45 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39225.292083 secs 2024-04-26T22:44:46.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:45 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39225.292083 secs 2024-04-26T22:44:46.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:45 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39225.292083 secs 2024-04-26T22:44:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:46.737+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:46.737+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:44:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:44:46.737+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:44:47.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:46 smithi033 ceph-mon[27845]: pgmap v21125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:47.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:46 smithi096 ceph-mon[32753]: pgmap v21125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:47.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:46 smithi194 ceph-mon[33017]: pgmap v21125: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:48 smithi033 ceph-mon[27845]: pgmap v21126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39227.305206 secs 2024-04-26T22:44:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:48 smithi096 ceph-mon[32753]: pgmap v21126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:49.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39227.305206 secs 2024-04-26T22:44:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:48 smithi194 ceph-mon[33017]: pgmap v21126: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39227.305206 secs 2024-04-26T22:44:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39230.292226 secs 2024-04-26T22:44:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39230.292226 secs 2024-04-26T22:44:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39230.292226 secs 2024-04-26T22:44:51.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:50 smithi033 ceph-mon[27845]: pgmap v21127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:51.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:50 smithi096 ceph-mon[32753]: pgmap v21127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:50 smithi194 ceph-mon[33017]: pgmap v21127: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:53.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:52 smithi033 ceph-mon[27845]: pgmap v21128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:53.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:52 smithi096 ceph-mon[32753]: pgmap v21128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:52 smithi194 ceph-mon[33017]: pgmap v21128: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39232.305266 secs 2024-04-26T22:44:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39232.305266 secs 2024-04-26T22:44:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39232.305266 secs 2024-04-26T22:44:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:54 smithi033 ceph-mon[27845]: pgmap v21129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39235.292291 secs 2024-04-26T22:44:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:54 smithi096 ceph-mon[32753]: pgmap v21129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:55.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39235.292291 secs 2024-04-26T22:44:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:54 smithi194 ceph-mon[33017]: pgmap v21129: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39235.292291 secs 2024-04-26T22:44:57.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:56 smithi033 ceph-mon[27845]: pgmap v21130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:57.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:56 smithi096 ceph-mon[32753]: pgmap v21130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:57.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:56 smithi194 ceph-mon[33017]: pgmap v21130: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:44:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:58 smithi033 ceph-mon[27845]: pgmap v21131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39237.305326 secs 2024-04-26T22:44:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:58 smithi096 ceph-mon[32753]: pgmap v21131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:59.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39237.305326 secs 2024-04-26T22:44:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:58 smithi194 ceph-mon[33017]: pgmap v21131: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:44:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39237.305326 secs 2024-04-26T22:45:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:44:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39240.292401 secs 2024-04-26T22:45:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:44:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39240.292401 secs 2024-04-26T22:45:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:44:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39240.292401 secs 2024-04-26T22:45:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:00 smithi033 ceph-mon[27845]: pgmap v21132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:00 smithi096 ceph-mon[32753]: pgmap v21132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:01.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:00 smithi194 ceph-mon[33017]: pgmap v21132: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:01.753+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:01.753+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:01.754 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:01.753+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:45:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:02 smithi033 ceph-mon[27845]: pgmap v21133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:02 smithi096 ceph-mon[32753]: pgmap v21133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:03.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:02 smithi194 ceph-mon[33017]: pgmap v21133: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39242.305411 secs 2024-04-26T22:45:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39242.305411 secs 2024-04-26T22:45:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39242.305411 secs 2024-04-26T22:45:05.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:04 smithi096 ceph-mon[32753]: pgmap v21134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:05.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39245.292512 secs 2024-04-26T22:45:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:04 smithi194 ceph-mon[33017]: pgmap v21134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39245.292512 secs 2024-04-26T22:45:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:04 smithi033 ceph-mon[27845]: pgmap v21134: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39245.292512 secs 2024-04-26T22:45:07.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:06 smithi194 ceph-mon[33017]: pgmap v21135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:07.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:06 smithi033 ceph-mon[27845]: pgmap v21135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:07.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:06 smithi096 ceph-mon[32753]: pgmap v21135: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:08 smithi194 ceph-mon[33017]: pgmap v21136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39247.305498 secs 2024-04-26T22:45:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:08 smithi033 ceph-mon[27845]: pgmap v21136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39247.305498 secs 2024-04-26T22:45:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:08 smithi096 ceph-mon[32753]: pgmap v21136: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:09.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39247.305498 secs 2024-04-26T22:45:10.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39250.292614 secs 2024-04-26T22:45:10.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39250.292614 secs 2024-04-26T22:45:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39250.292614 secs 2024-04-26T22:45:11.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:10 smithi194 ceph-mon[33017]: pgmap v21137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:11.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:10 smithi033 ceph-mon[27845]: pgmap v21137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:11.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:10 smithi096 ceph-mon[32753]: pgmap v21137: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:13.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:12 smithi194 ceph-mon[33017]: pgmap v21138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:13.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:12 smithi033 ceph-mon[27845]: pgmap v21138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:13.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:12 smithi096 ceph-mon[32753]: pgmap v21138: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39252.305581 secs 2024-04-26T22:45:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39252.305581 secs 2024-04-26T22:45:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39252.305581 secs 2024-04-26T22:45:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:14 smithi194 ceph-mon[33017]: pgmap v21139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39255.292680 secs 2024-04-26T22:45:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:14 smithi033 ceph-mon[27845]: pgmap v21139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39255.292680 secs 2024-04-26T22:45:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:14 smithi096 ceph-mon[32753]: pgmap v21139: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:15.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39255.292680 secs 2024-04-26T22:45:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:16.767+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:16.768+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:16.768+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:45:17.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:16 smithi194 ceph-mon[33017]: pgmap v21140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:17.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:16 smithi033 ceph-mon[27845]: pgmap v21140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:17.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:16 smithi096 ceph-mon[32753]: pgmap v21140: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:18 smithi194 ceph-mon[33017]: pgmap v21141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39257.305670 secs 2024-04-26T22:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:18 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:18 smithi033 ceph-mon[27845]: pgmap v21141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39257.305670 secs 2024-04-26T22:45:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:18 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:18 smithi096 ceph-mon[32753]: pgmap v21141: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:45:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39257.305670 secs 2024-04-26T22:45:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:19.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:18 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:20 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39260.292819 secs 2024-04-26T22:45:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39260.292819 secs 2024-04-26T22:45:20.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:20 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39260.292819 secs 2024-04-26T22:45:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:21 smithi194 ceph-mon[33017]: pgmap v21142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:21 smithi194 ceph-mon[33017]: pgmap v21143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:45:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:21 smithi194 ceph-mon[33017]: pgmap v21144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:21.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:21 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:21.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:21 smithi033 ceph-mon[27845]: pgmap v21142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:21 smithi033 ceph-mon[27845]: pgmap v21143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:45:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:21 smithi033 ceph-mon[27845]: pgmap v21144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:21.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:21 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:21 smithi096 ceph-mon[32753]: pgmap v21142: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:21.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:45:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:45:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:21 smithi096 ceph-mon[32753]: pgmap v21143: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:45:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:21 smithi096 ceph-mon[32753]: pgmap v21144: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:21.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:21 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:45:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:23 smithi033 ceph-mon[27845]: pgmap v21145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:23 smithi096 ceph-mon[32753]: pgmap v21145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:23 smithi194 ceph-mon[33017]: pgmap v21145: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39262.305753 secs 2024-04-26T22:45:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39262.305753 secs 2024-04-26T22:45:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39262.305753 secs 2024-04-26T22:45:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:25 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39265.292881 secs 2024-04-26T22:45:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:25 smithi033 ceph-mon[27845]: pgmap v21146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:25 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39265.292881 secs 2024-04-26T22:45:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:25 smithi096 ceph-mon[32753]: pgmap v21146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:25 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39265.292881 secs 2024-04-26T22:45:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:25 smithi194 ceph-mon[33017]: pgmap v21146: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:27 smithi033 ceph-mon[27845]: pgmap v21147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:27 smithi096 ceph-mon[32753]: pgmap v21147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:27 smithi194 ceph-mon[33017]: pgmap v21147: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:45:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39267.305837 secs 2024-04-26T22:45:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:29 smithi033 ceph-mon[27845]: pgmap v21148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:45:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39267.305837 secs 2024-04-26T22:45:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:29 smithi096 ceph-mon[32753]: pgmap v21148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:45:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39267.305837 secs 2024-04-26T22:45:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:29 smithi194 ceph-mon[33017]: pgmap v21148: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:45:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:30 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39270.293006 secs 2024-04-26T22:45:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:30 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39270.293006 secs 2024-04-26T22:45:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:30 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39270.293006 secs 2024-04-26T22:45:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:31 smithi033 ceph-mon[27845]: pgmap v21149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:45:31.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:31 smithi096 ceph-mon[32753]: pgmap v21149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:45:31.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:31.772+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:31.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:31.772+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:31.772 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:31.772+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:45:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:31 smithi194 ceph-mon[33017]: pgmap v21149: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:45:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:33 smithi033 ceph-mon[27845]: pgmap v21150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:33 smithi096 ceph-mon[32753]: pgmap v21150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:33 smithi194 ceph-mon[33017]: pgmap v21150: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39272.305922 secs 2024-04-26T22:45:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39275.293228 secs 2024-04-26T22:45:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39272.305922 secs 2024-04-26T22:45:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39275.293228 secs 2024-04-26T22:45:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39272.305922 secs 2024-04-26T22:45:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39275.293228 secs 2024-04-26T22:45:35.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:35 smithi096 ceph-mon[32753]: pgmap v21151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:35 smithi194 ceph-mon[33017]: pgmap v21151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:35 smithi033 ceph-mon[27845]: pgmap v21151: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:36 smithi194 ceph-mon[33017]: pgmap v21152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:36 smithi033 ceph-mon[27845]: pgmap v21152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:36.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:36 smithi096 ceph-mon[32753]: pgmap v21152: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39277.306005 secs 2024-04-26T22:45:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:39 smithi033 ceph-mon[27845]: pgmap v21153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39280.293243 secs 2024-04-26T22:45:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39277.306005 secs 2024-04-26T22:45:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:39 smithi096 ceph-mon[32753]: pgmap v21153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39280.293243 secs 2024-04-26T22:45:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39277.306005 secs 2024-04-26T22:45:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:39 smithi194 ceph-mon[33017]: pgmap v21153: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39280.293243 secs 2024-04-26T22:45:41.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:41 smithi194 ceph-mon[33017]: pgmap v21154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:41 smithi033 ceph-mon[27845]: pgmap v21154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:41 smithi096 ceph-mon[32753]: pgmap v21154: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:43 smithi033 ceph-mon[27845]: pgmap v21155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:43 smithi096 ceph-mon[32753]: pgmap v21155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:43 smithi194 ceph-mon[33017]: pgmap v21155: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39282.306065 secs 2024-04-26T22:45:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39285.293325 secs 2024-04-26T22:45:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39282.306065 secs 2024-04-26T22:45:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39285.293325 secs 2024-04-26T22:45:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39282.306065 secs 2024-04-26T22:45:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39285.293325 secs 2024-04-26T22:45:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:45 smithi033 ceph-mon[27845]: pgmap v21156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:45 smithi096 ceph-mon[32753]: pgmap v21156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:45 smithi194 ceph-mon[33017]: pgmap v21156: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:46.787+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:46.787+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:45:46.788 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:45:46.787+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:45:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:47 smithi033 ceph-mon[27845]: pgmap v21157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:47 smithi096 ceph-mon[32753]: pgmap v21157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:47 smithi194 ceph-mon[33017]: pgmap v21157: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39287.306129 secs 2024-04-26T22:45:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:49 smithi033 ceph-mon[27845]: pgmap v21158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:49.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39290.293423 secs 2024-04-26T22:45:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39287.306129 secs 2024-04-26T22:45:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:49 smithi096 ceph-mon[32753]: pgmap v21158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39290.293423 secs 2024-04-26T22:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39287.306129 secs 2024-04-26T22:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:49 smithi194 ceph-mon[33017]: pgmap v21158: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39290.293423 secs 2024-04-26T22:45:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:51 smithi033 ceph-mon[27845]: pgmap v21159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:51 smithi096 ceph-mon[32753]: pgmap v21159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:51 smithi194 ceph-mon[33017]: pgmap v21159: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:45:53.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:53 smithi096 ceph-mon[32753]: pgmap v21160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:53 smithi194 ceph-mon[33017]: pgmap v21160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:53 smithi033 ceph-mon[27845]: pgmap v21160: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39292.306212 secs 2024-04-26T22:45:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39295.293504 secs 2024-04-26T22:45:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:54 smithi194 ceph-mon[33017]: pgmap v21161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39292.306212 secs 2024-04-26T22:45:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39295.293504 secs 2024-04-26T22:45:54.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:54 smithi033 ceph-mon[27845]: pgmap v21161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39292.306212 secs 2024-04-26T22:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39295.293504 secs 2024-04-26T22:45:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:54 smithi096 ceph-mon[32753]: pgmap v21161: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:45:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:57 smithi033 ceph-mon[27845]: pgmap v21162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:57 smithi096 ceph-mon[32753]: pgmap v21162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:57 smithi194 ceph-mon[33017]: pgmap v21162: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39297.306321 secs 2024-04-26T22:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:59 smithi033 ceph-mon[27845]: pgmap v21163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:45:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39300.293618 secs 2024-04-26T22:45:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39297.306321 secs 2024-04-26T22:45:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:59 smithi096 ceph-mon[32753]: pgmap v21163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:45:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39300.293618 secs 2024-04-26T22:45:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39297.306321 secs 2024-04-26T22:45:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:59 smithi194 ceph-mon[33017]: pgmap v21163: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:45:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:45:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39300.293618 secs 2024-04-26T22:46:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:01 smithi033 ceph-mon[27845]: pgmap v21164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:01 smithi096 ceph-mon[32753]: pgmap v21164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:01.794+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:01.794+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:46:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:01.794+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:01 smithi194 ceph-mon[33017]: pgmap v21164: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:03 smithi033 ceph-mon[27845]: pgmap v21165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:03 smithi096 ceph-mon[32753]: pgmap v21165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:03 smithi194 ceph-mon[33017]: pgmap v21165: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39302.306418 secs 2024-04-26T22:46:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39305.293756 secs 2024-04-26T22:46:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39302.306418 secs 2024-04-26T22:46:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39305.293756 secs 2024-04-26T22:46:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39302.306418 secs 2024-04-26T22:46:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39305.293756 secs 2024-04-26T22:46:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:05 smithi033 ceph-mon[27845]: pgmap v21166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:46:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:05 smithi096 ceph-mon[32753]: pgmap v21166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:46:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:05 smithi194 ceph-mon[33017]: pgmap v21166: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:46:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:07 smithi033 ceph-mon[27845]: pgmap v21167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:07 smithi096 ceph-mon[32753]: pgmap v21167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:07.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:07 smithi194 ceph-mon[33017]: pgmap v21167: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39307.306522 secs 2024-04-26T22:46:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:09 smithi096 ceph-mon[32753]: pgmap v21168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39310.293869 secs 2024-04-26T22:46:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39307.306522 secs 2024-04-26T22:46:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:09 smithi194 ceph-mon[33017]: pgmap v21168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39310.293869 secs 2024-04-26T22:46:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39307.306522 secs 2024-04-26T22:46:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:09 smithi033 ceph-mon[27845]: pgmap v21168: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39310.293869 secs 2024-04-26T22:46:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:10 smithi194 ceph-mon[33017]: pgmap v21169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:10 smithi033 ceph-mon[27845]: pgmap v21169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:10 smithi096 ceph-mon[32753]: pgmap v21169: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:13 smithi033 ceph-mon[27845]: pgmap v21170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:13 smithi096 ceph-mon[32753]: pgmap v21170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:13 smithi194 ceph-mon[33017]: pgmap v21170: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39312.306626 secs 2024-04-26T22:46:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39315.294008 secs 2024-04-26T22:46:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39312.306626 secs 2024-04-26T22:46:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39315.294008 secs 2024-04-26T22:46:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39312.306626 secs 2024-04-26T22:46:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39315.294008 secs 2024-04-26T22:46:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:15 smithi033 ceph-mon[27845]: pgmap v21171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:46:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:15 smithi096 ceph-mon[32753]: pgmap v21171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:46:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:15 smithi194 ceph-mon[33017]: pgmap v21171: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:46:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:16.801+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:16.801+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:16.801+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:46:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:17 smithi033 ceph-mon[27845]: pgmap v21172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:17.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:17 smithi096 ceph-mon[32753]: pgmap v21172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:17 smithi194 ceph-mon[33017]: pgmap v21172: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39317.306727 secs 2024-04-26T22:46:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:19 smithi033 ceph-mon[27845]: pgmap v21173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:19.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39320.294141 secs 2024-04-26T22:46:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39317.306727 secs 2024-04-26T22:46:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:19 smithi096 ceph-mon[32753]: pgmap v21173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39320.294141 secs 2024-04-26T22:46:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39317.306727 secs 2024-04-26T22:46:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:19 smithi194 ceph-mon[33017]: pgmap v21173: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39320.294141 secs 2024-04-26T22:46:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:21 smithi033 ceph-mon[27845]: pgmap v21174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:21 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:21 smithi096 ceph-mon[32753]: pgmap v21174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:21 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:21 smithi194 ceph-mon[33017]: pgmap v21174: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:21 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:46:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:22 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:22 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:46:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:22 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:22 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:46:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:46:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:22 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:46:22.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:22 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:46:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:23 smithi194 ceph-mon[33017]: pgmap v21175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:23.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:23 smithi033 ceph-mon[27845]: pgmap v21175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:23.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:23 smithi096 ceph-mon[32753]: pgmap v21175: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39322.306822 secs 2024-04-26T22:46:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39325.294182 secs 2024-04-26T22:46:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:24 smithi194 ceph-mon[33017]: pgmap v21176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39322.306822 secs 2024-04-26T22:46:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39325.294182 secs 2024-04-26T22:46:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:24 smithi033 ceph-mon[27845]: pgmap v21176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39322.306822 secs 2024-04-26T22:46:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39325.294182 secs 2024-04-26T22:46:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:24 smithi096 ceph-mon[32753]: pgmap v21176: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:27 smithi194 ceph-mon[33017]: pgmap v21177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:27.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:27 smithi033 ceph-mon[27845]: pgmap v21177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:27.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:27 smithi096 ceph-mon[32753]: pgmap v21177: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:46:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:29 smithi033 ceph-mon[27845]: pgmap v21178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39327.306912 secs 2024-04-26T22:46:29.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39330.294296 secs 2024-04-26T22:46:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:29 smithi194 ceph-mon[33017]: pgmap v21178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39327.306912 secs 2024-04-26T22:46:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39330.294296 secs 2024-04-26T22:46:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:29 smithi096 ceph-mon[32753]: pgmap v21178: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39327.306912 secs 2024-04-26T22:46:29.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39330.294296 secs 2024-04-26T22:46:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:31 smithi194 ceph-mon[33017]: pgmap v21179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:46:31.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:31 smithi033 ceph-mon[27845]: pgmap v21179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:46:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:31 smithi096 ceph-mon[32753]: pgmap v21179: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:46:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:31.804+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:31.804+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:46:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:31.804+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:33 smithi194 ceph-mon[33017]: pgmap v21180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:33 smithi033 ceph-mon[27845]: pgmap v21180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:33 smithi096 ceph-mon[32753]: pgmap v21180: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39332.306995 secs 2024-04-26T22:46:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39335.294416 secs 2024-04-26T22:46:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39332.306995 secs 2024-04-26T22:46:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39335.294416 secs 2024-04-26T22:46:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39332.306995 secs 2024-04-26T22:46:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39335.294416 secs 2024-04-26T22:46:35.333 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:35 smithi096 ceph-mon[32753]: pgmap v21181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:35 smithi194 ceph-mon[33017]: pgmap v21181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:35 smithi033 ceph-mon[27845]: pgmap v21181: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:37.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:37 smithi194 ceph-mon[33017]: pgmap v21182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:37 smithi033 ceph-mon[27845]: pgmap v21182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:37.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:37 smithi096 ceph-mon[32753]: pgmap v21182: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:39 smithi194 ceph-mon[33017]: pgmap v21183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39337.307089 secs 2024-04-26T22:46:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39340.294533 secs 2024-04-26T22:46:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:39 smithi033 ceph-mon[27845]: pgmap v21183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39337.307089 secs 2024-04-26T22:46:39.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39340.294533 secs 2024-04-26T22:46:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:39 smithi096 ceph-mon[32753]: pgmap v21183: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39337.307089 secs 2024-04-26T22:46:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39340.294533 secs 2024-04-26T22:46:41.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:41 smithi194 ceph-mon[33017]: pgmap v21184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:41 smithi033 ceph-mon[27845]: pgmap v21184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:41 smithi096 ceph-mon[32753]: pgmap v21184: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:43.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:43 smithi194 ceph-mon[33017]: pgmap v21185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:43 smithi033 ceph-mon[27845]: pgmap v21185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:43.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:43 smithi096 ceph-mon[32753]: pgmap v21185: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39342.307165 secs 2024-04-26T22:46:44.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39345.294655 secs 2024-04-26T22:46:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39342.307165 secs 2024-04-26T22:46:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39345.294655 secs 2024-04-26T22:46:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39342.307165 secs 2024-04-26T22:46:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39345.294655 secs 2024-04-26T22:46:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:45 smithi033 ceph-mon[27845]: pgmap v21186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:45 smithi096 ceph-mon[32753]: pgmap v21186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:45 smithi194 ceph-mon[33017]: pgmap v21186: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:46.808+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:46.808+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:46:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:46:46.808+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:46:47.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:47 smithi033 ceph-mon[27845]: pgmap v21187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:47.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:47 smithi096 ceph-mon[32753]: pgmap v21187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:47 smithi194 ceph-mon[33017]: pgmap v21187: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:46:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:49 smithi033 ceph-mon[27845]: pgmap v21188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39347.307249 secs 2024-04-26T22:46:49.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39350.294770 secs 2024-04-26T22:46:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:49 smithi096 ceph-mon[32753]: pgmap v21188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39347.307249 secs 2024-04-26T22:46:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39350.294770 secs 2024-04-26T22:46:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:49 smithi194 ceph-mon[33017]: pgmap v21188: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:46:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39347.307249 secs 2024-04-26T22:46:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39350.294770 secs 2024-04-26T22:46:51.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:51 smithi033 ceph-mon[27845]: pgmap v21189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:46:51.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:51 smithi096 ceph-mon[32753]: pgmap v21189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:46:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:51 smithi194 ceph-mon[33017]: pgmap v21189: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:46:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:53 smithi033 ceph-mon[27845]: pgmap v21190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:53 smithi096 ceph-mon[32753]: pgmap v21190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:53 smithi194 ceph-mon[33017]: pgmap v21190: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39352.307330 secs 2024-04-26T22:46:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39355.294896 secs 2024-04-26T22:46:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39352.307330 secs 2024-04-26T22:46:54.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39355.294896 secs 2024-04-26T22:46:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39352.307330 secs 2024-04-26T22:46:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39355.294896 secs 2024-04-26T22:46:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:55 smithi033 ceph-mon[27845]: pgmap v21191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:55 smithi096 ceph-mon[32753]: pgmap v21191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:55.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:55 smithi194 ceph-mon[33017]: pgmap v21191: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:57.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:57 smithi033 ceph-mon[27845]: pgmap v21192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:46:57.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:57 smithi096 ceph-mon[32753]: pgmap v21192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:46:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:57 smithi194 ceph-mon[33017]: pgmap v21192: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:46:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:59 smithi033 ceph-mon[27845]: pgmap v21193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39357.307413 secs 2024-04-26T22:46:59.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:46:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39360.295011 secs 2024-04-26T22:46:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:59 smithi096 ceph-mon[32753]: pgmap v21193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39357.307413 secs 2024-04-26T22:46:59.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:46:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39360.295011 secs 2024-04-26T22:46:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:59 smithi194 ceph-mon[33017]: pgmap v21193: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:46:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39357.307413 secs 2024-04-26T22:46:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:46:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39360.295011 secs 2024-04-26T22:47:01.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:01 smithi033 ceph-mon[27845]: pgmap v21194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:01.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:01 smithi096 ceph-mon[32753]: pgmap v21194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:01 smithi194 ceph-mon[33017]: pgmap v21194: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:01.824+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:01.824+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:01.824+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:47:03.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:03 smithi033 ceph-mon[27845]: pgmap v21195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:03.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:03 smithi096 ceph-mon[32753]: pgmap v21195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:03 smithi194 ceph-mon[33017]: pgmap v21195: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39362.307499 secs 2024-04-26T22:47:04.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39365.295136 secs 2024-04-26T22:47:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39362.307499 secs 2024-04-26T22:47:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39365.295136 secs 2024-04-26T22:47:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39362.307499 secs 2024-04-26T22:47:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39365.295136 secs 2024-04-26T22:47:05.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:05 smithi096 ceph-mon[32753]: pgmap v21196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:05 smithi194 ceph-mon[33017]: pgmap v21196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:05 smithi033 ceph-mon[27845]: pgmap v21196: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:07 smithi194 ceph-mon[33017]: pgmap v21197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:07 smithi033 ceph-mon[27845]: pgmap v21197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:07 smithi096 ceph-mon[32753]: pgmap v21197: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:09 smithi194 ceph-mon[33017]: pgmap v21198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39367.307582 secs 2024-04-26T22:47:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39370.295249 secs 2024-04-26T22:47:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:09 smithi033 ceph-mon[27845]: pgmap v21198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39367.307582 secs 2024-04-26T22:47:09.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39370.295249 secs 2024-04-26T22:47:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:09 smithi096 ceph-mon[32753]: pgmap v21198: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39367.307582 secs 2024-04-26T22:47:09.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39370.295249 secs 2024-04-26T22:47:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:11 smithi194 ceph-mon[33017]: pgmap v21199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:11 smithi033 ceph-mon[27845]: pgmap v21199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:11 smithi096 ceph-mon[32753]: pgmap v21199: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:13.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:13 smithi033 ceph-mon[27845]: pgmap v21200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:13 smithi194 ceph-mon[33017]: pgmap v21200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:13 smithi096 ceph-mon[32753]: pgmap v21200: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39372.307696 secs 2024-04-26T22:47:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39375.295367 secs 2024-04-26T22:47:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39372.307696 secs 2024-04-26T22:47:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39375.295367 secs 2024-04-26T22:47:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39372.307696 secs 2024-04-26T22:47:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39375.295367 secs 2024-04-26T22:47:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:15 smithi194 ceph-mon[33017]: pgmap v21201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:15 smithi033 ceph-mon[27845]: pgmap v21201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:15 smithi096 ceph-mon[32753]: pgmap v21201: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:16.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:16.828+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:16.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:16.828+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:16.828 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:16.828+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:47:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:17 smithi194 ceph-mon[33017]: pgmap v21202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:17 smithi033 ceph-mon[27845]: pgmap v21202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:17 smithi096 ceph-mon[32753]: pgmap v21202: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:19 smithi194 ceph-mon[33017]: pgmap v21203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39377.307781 secs 2024-04-26T22:47:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39380.295444 secs 2024-04-26T22:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:19 smithi033 ceph-mon[27845]: pgmap v21203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39377.307781 secs 2024-04-26T22:47:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39380.295444 secs 2024-04-26T22:47:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:19 smithi096 ceph-mon[32753]: pgmap v21203: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39377.307781 secs 2024-04-26T22:47:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39380.295444 secs 2024-04-26T22:47:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:21 smithi194 ceph-mon[33017]: pgmap v21204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:21 smithi033 ceph-mon[27845]: pgmap v21204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:21 smithi096 ceph-mon[32753]: pgmap v21204: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:23 smithi194 ceph-mon[33017]: pgmap v21205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:23 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:23 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:23 smithi033 ceph-mon[27845]: pgmap v21205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:23 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:23 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:23 smithi096 ceph-mon[32753]: pgmap v21205: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:23 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:47:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:23.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:23 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39382.307877 secs 2024-04-26T22:47:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39385.295510 secs 2024-04-26T22:47:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39382.307877 secs 2024-04-26T22:47:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39385.295510 secs 2024-04-26T22:47:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39382.307877 secs 2024-04-26T22:47:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39385.295510 secs 2024-04-26T22:47:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:25 smithi194 ceph-mon[33017]: pgmap v21206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:25 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:25 smithi033 ceph-mon[27845]: pgmap v21206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:25 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:25 smithi096 ceph-mon[32753]: pgmap v21206: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:25.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:25 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:26 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:26 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:26.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:26 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:27 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:27 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:27 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:27 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:27 smithi194 ceph-mon[33017]: pgmap v21207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:27 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:27 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:47:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:27 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:47:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:27 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:47:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:27 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:47:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:27 smithi033 ceph-mon[27845]: pgmap v21207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:27.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:27 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:27 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:47:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:27 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:47:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:27 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:47:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:27 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:47:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:27 smithi096 ceph-mon[32753]: pgmap v21207: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:27.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:27 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:28.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:28.503+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T22:47:28.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:28.503+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:47:28.503 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:28.504+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 307 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T22:47:28.504 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:28.504+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 307 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feecc003690 con 0x7fef00077780 2024-04-26T22:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: pgmap v21208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39387.307961 secs 2024-04-26T22:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: pgmap v21209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: pgmap v21210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:29.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39390.295606 secs 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: pgmap v21208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39387.307961 secs 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: pgmap v21209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: pgmap v21210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:29.677 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39390.295606 secs 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: pgmap v21208: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39387.307961 secs 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:47:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:47:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: pgmap v21209: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T22:47:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: pgmap v21210: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:47:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:47:29.683 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39390.295606 secs 2024-04-26T22:47:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:30 smithi194 ceph-mon[33017]: pgmap v21211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:30.863 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:30 smithi033 ceph-mon[27845]: pgmap v21211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:30 smithi096 ceph-mon[32753]: pgmap v21211: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:31.836+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:31.836+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:31.836+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:47:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:32 smithi194 ceph-mon[33017]: pgmap v21212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:32 smithi033 ceph-mon[27845]: pgmap v21212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:32 smithi096 ceph-mon[32753]: pgmap v21212: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:34.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39392.308022 secs 2024-04-26T22:47:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39392.308022 secs 2024-04-26T22:47:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39392.308022 secs 2024-04-26T22:47:35.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39395.295727 secs 2024-04-26T22:47:35.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:34 smithi096 ceph-mon[32753]: pgmap v21213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T22:47:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39395.295727 secs 2024-04-26T22:47:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:34 smithi194 ceph-mon[33017]: pgmap v21213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T22:47:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39395.295727 secs 2024-04-26T22:47:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:34 smithi033 ceph-mon[27845]: pgmap v21213: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-26T22:47:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:36 smithi194 ceph-mon[33017]: pgmap v21214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:36 smithi033 ceph-mon[27845]: pgmap v21214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:36 smithi096 ceph-mon[32753]: pgmap v21214: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:47:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:38 smithi194 ceph-mon[33017]: pgmap v21215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:47:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:38 smithi033 ceph-mon[27845]: pgmap v21215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:47:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:38 smithi096 ceph-mon[32753]: pgmap v21215: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:47:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39397.308104 secs 2024-04-26T22:47:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39400.295860 secs 2024-04-26T22:47:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39397.308104 secs 2024-04-26T22:47:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39400.295860 secs 2024-04-26T22:47:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39397.308104 secs 2024-04-26T22:47:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39400.295860 secs 2024-04-26T22:47:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:40 smithi194 ceph-mon[33017]: pgmap v21216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:40 smithi033 ceph-mon[27845]: pgmap v21216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:40 smithi096 ceph-mon[32753]: pgmap v21216: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:42 smithi194 ceph-mon[33017]: pgmap v21217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:42 smithi033 ceph-mon[27845]: pgmap v21217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:42 smithi096 ceph-mon[32753]: pgmap v21217: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:44.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:43 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39402.308191 secs 2024-04-26T22:47:44.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:43 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39402.308191 secs 2024-04-26T22:47:44.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:43 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39402.308191 secs 2024-04-26T22:47:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39405.296003 secs 2024-04-26T22:47:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:44 smithi194 ceph-mon[33017]: pgmap v21218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:47:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39405.296003 secs 2024-04-26T22:47:45.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:44 smithi033 ceph-mon[27845]: pgmap v21218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:47:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39405.296003 secs 2024-04-26T22:47:45.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:44 smithi096 ceph-mon[32753]: pgmap v21218: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:47:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:46.845+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:46.845+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:47:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:47:46.845+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:47:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:46 smithi194 ceph-mon[33017]: pgmap v21219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:47:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:46 smithi033 ceph-mon[27845]: pgmap v21219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:47:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:46 smithi096 ceph-mon[32753]: pgmap v21219: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:47:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:48 smithi194 ceph-mon[33017]: pgmap v21220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:48 smithi033 ceph-mon[27845]: pgmap v21220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:48 smithi096 ceph-mon[32753]: pgmap v21220: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39407.308276 secs 2024-04-26T22:47:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39410.296146 secs 2024-04-26T22:47:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39407.308276 secs 2024-04-26T22:47:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39410.296146 secs 2024-04-26T22:47:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39407.308276 secs 2024-04-26T22:47:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39410.296146 secs 2024-04-26T22:47:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:50 smithi194 ceph-mon[33017]: pgmap v21221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:50.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:50 smithi033 ceph-mon[27845]: pgmap v21221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:50 smithi096 ceph-mon[32753]: pgmap v21221: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:47:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:52 smithi194 ceph-mon[33017]: pgmap v21222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:52 smithi033 ceph-mon[27845]: pgmap v21222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:52 smithi096 ceph-mon[32753]: pgmap v21222: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:54.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39412.308358 secs 2024-04-26T22:47:54.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39412.308358 secs 2024-04-26T22:47:54.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39412.308358 secs 2024-04-26T22:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39415.296215 secs 2024-04-26T22:47:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:54 smithi194 ceph-mon[33017]: pgmap v21223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:47:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39415.296215 secs 2024-04-26T22:47:55.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:54 smithi033 ceph-mon[27845]: pgmap v21223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:47:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39415.296215 secs 2024-04-26T22:47:55.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:54 smithi096 ceph-mon[32753]: pgmap v21223: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:47:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:56 smithi194 ceph-mon[33017]: pgmap v21224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:47:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:56 smithi033 ceph-mon[27845]: pgmap v21224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:47:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:56 smithi096 ceph-mon[32753]: pgmap v21224: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:47:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:58 smithi194 ceph-mon[33017]: pgmap v21225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:58 smithi033 ceph-mon[27845]: pgmap v21225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:58 smithi096 ceph-mon[32753]: pgmap v21225: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:47:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39417.308441 secs 2024-04-26T22:47:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:47:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39420.296292 secs 2024-04-26T22:47:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39417.308441 secs 2024-04-26T22:47:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:47:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39420.296292 secs 2024-04-26T22:47:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39417.308441 secs 2024-04-26T22:47:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:47:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39420.296292 secs 2024-04-26T22:48:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:00 smithi194 ceph-mon[33017]: pgmap v21226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:00 smithi033 ceph-mon[27845]: pgmap v21226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:00 smithi096 ceph-mon[32753]: pgmap v21226: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:01.856+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:01.856+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:01.856+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:48:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:02 smithi194 ceph-mon[33017]: pgmap v21227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:02 smithi033 ceph-mon[27845]: pgmap v21227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:02 smithi096 ceph-mon[32753]: pgmap v21227: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39422.308524 secs 2024-04-26T22:48:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39422.308524 secs 2024-04-26T22:48:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39422.308524 secs 2024-04-26T22:48:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39425.296414 secs 2024-04-26T22:48:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:04 smithi096 ceph-mon[32753]: pgmap v21228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39425.296414 secs 2024-04-26T22:48:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:04 smithi194 ceph-mon[33017]: pgmap v21228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39425.296414 secs 2024-04-26T22:48:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:04 smithi033 ceph-mon[27845]: pgmap v21228: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:06 smithi194 ceph-mon[33017]: pgmap v21229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:06 smithi033 ceph-mon[27845]: pgmap v21229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:06 smithi096 ceph-mon[32753]: pgmap v21229: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:08 smithi194 ceph-mon[33017]: pgmap v21230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:08 smithi033 ceph-mon[27845]: pgmap v21230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:08 smithi096 ceph-mon[32753]: pgmap v21230: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39427.308606 secs 2024-04-26T22:48:09.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39430.296486 secs 2024-04-26T22:48:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39427.308606 secs 2024-04-26T22:48:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39430.296486 secs 2024-04-26T22:48:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39427.308606 secs 2024-04-26T22:48:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39430.296486 secs 2024-04-26T22:48:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:10 smithi194 ceph-mon[33017]: pgmap v21231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:10 smithi033 ceph-mon[27845]: pgmap v21231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:10 smithi096 ceph-mon[32753]: pgmap v21231: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:12 smithi194 ceph-mon[33017]: pgmap v21232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:12 smithi033 ceph-mon[27845]: pgmap v21232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:12 smithi096 ceph-mon[32753]: pgmap v21232: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39432.308691 secs 2024-04-26T22:48:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39432.308691 secs 2024-04-26T22:48:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39432.308691 secs 2024-04-26T22:48:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39435.296556 secs 2024-04-26T22:48:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:14 smithi194 ceph-mon[33017]: pgmap v21233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39435.296556 secs 2024-04-26T22:48:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:14 smithi033 ceph-mon[27845]: pgmap v21233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39435.296556 secs 2024-04-26T22:48:15.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:14 smithi096 ceph-mon[32753]: pgmap v21233: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:16 smithi194 ceph-mon[33017]: pgmap v21234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:16.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:16.872+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:16.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:16.873+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:16.873 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:16.873+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:48:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:16 smithi033 ceph-mon[27845]: pgmap v21234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:16 smithi096 ceph-mon[32753]: pgmap v21234: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:18 smithi194 ceph-mon[33017]: pgmap v21235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:18 smithi033 ceph-mon[27845]: pgmap v21235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:18 smithi096 ceph-mon[32753]: pgmap v21235: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39437.308775 secs 2024-04-26T22:48:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39440.296617 secs 2024-04-26T22:48:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39437.308775 secs 2024-04-26T22:48:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39440.296617 secs 2024-04-26T22:48:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39437.308775 secs 2024-04-26T22:48:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39440.296617 secs 2024-04-26T22:48:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:20 smithi194 ceph-mon[33017]: pgmap v21236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:20 smithi033 ceph-mon[27845]: pgmap v21236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:20 smithi096 ceph-mon[32753]: pgmap v21236: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:22 smithi194 ceph-mon[33017]: pgmap v21237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:22 smithi033 ceph-mon[27845]: pgmap v21237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:22 smithi096 ceph-mon[32753]: pgmap v21237: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39442.308859 secs 2024-04-26T22:48:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39442.308859 secs 2024-04-26T22:48:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39442.308859 secs 2024-04-26T22:48:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39445.296744 secs 2024-04-26T22:48:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:24 smithi194 ceph-mon[33017]: pgmap v21238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39445.296744 secs 2024-04-26T22:48:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:24 smithi033 ceph-mon[27845]: pgmap v21238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39445.296744 secs 2024-04-26T22:48:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:24 smithi096 ceph-mon[32753]: pgmap v21238: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:26 smithi194 ceph-mon[33017]: pgmap v21239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:26 smithi033 ceph-mon[27845]: pgmap v21239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:26 smithi096 ceph-mon[32753]: pgmap v21239: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:48:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:28 smithi194 ceph-mon[33017]: pgmap v21240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:28 smithi033 ceph-mon[27845]: pgmap v21240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:28 smithi096 ceph-mon[32753]: pgmap v21240: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39447.308921 secs 2024-04-26T22:48:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:29 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:48:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39450.296829 secs 2024-04-26T22:48:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39447.308921 secs 2024-04-26T22:48:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:29 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:48:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39450.296829 secs 2024-04-26T22:48:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39447.308921 secs 2024-04-26T22:48:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:29 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:48:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39450.296829 secs 2024-04-26T22:48:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:48:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:30 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:48:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:30 smithi194 ceph-mon[33017]: pgmap v21241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:48:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:30 smithi194 ceph-mon[33017]: pgmap v21242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:30 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:48:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:48:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:30 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:48:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:30 smithi033 ceph-mon[27845]: pgmap v21241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:48:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:30 smithi033 ceph-mon[27845]: pgmap v21242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:30.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:30 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:48:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:48:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:30 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:48:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:30 smithi096 ceph-mon[32753]: pgmap v21241: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:48:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:30 smithi096 ceph-mon[32753]: pgmap v21242: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:30.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:30 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:48:31.884 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:31.884+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:31.884+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:31.884+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:48:33.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:33 smithi033 ceph-mon[27845]: pgmap v21243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:33.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:33 smithi096 ceph-mon[32753]: pgmap v21243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:33 smithi194 ceph-mon[33017]: pgmap v21243: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39452.309025 secs 2024-04-26T22:48:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39455.296950 secs 2024-04-26T22:48:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39452.309025 secs 2024-04-26T22:48:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39455.296950 secs 2024-04-26T22:48:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39452.309025 secs 2024-04-26T22:48:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39455.296950 secs 2024-04-26T22:48:35.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:35 smithi033 ceph-mon[27845]: pgmap v21244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T22:48:35.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:35 smithi096 ceph-mon[32753]: pgmap v21244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T22:48:35.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:35 smithi194 ceph-mon[33017]: pgmap v21244: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T22:48:37.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:37 smithi033 ceph-mon[27845]: pgmap v21245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:37.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:37 smithi096 ceph-mon[32753]: pgmap v21245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:37.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:37 smithi194 ceph-mon[33017]: pgmap v21245: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39457.309086 secs 2024-04-26T22:48:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:39 smithi033 ceph-mon[27845]: pgmap v21246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39460.297072 secs 2024-04-26T22:48:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39457.309086 secs 2024-04-26T22:48:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:39 smithi096 ceph-mon[32753]: pgmap v21246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:39.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39460.297072 secs 2024-04-26T22:48:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39457.309086 secs 2024-04-26T22:48:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:39 smithi194 ceph-mon[33017]: pgmap v21246: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:48:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39460.297072 secs 2024-04-26T22:48:41.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:41 smithi033 ceph-mon[27845]: pgmap v21247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:48:41.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:41 smithi096 ceph-mon[32753]: pgmap v21247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:48:41.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:41 smithi194 ceph-mon[33017]: pgmap v21247: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:48:43.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:43 smithi033 ceph-mon[27845]: pgmap v21248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:43.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:43 smithi096 ceph-mon[32753]: pgmap v21248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:43.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:43 smithi194 ceph-mon[33017]: pgmap v21248: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39462.309150 secs 2024-04-26T22:48:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39465.297134 secs 2024-04-26T22:48:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39462.309150 secs 2024-04-26T22:48:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39465.297134 secs 2024-04-26T22:48:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39462.309150 secs 2024-04-26T22:48:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39465.297134 secs 2024-04-26T22:48:45.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:45 smithi096 ceph-mon[32753]: pgmap v21249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:45 smithi194 ceph-mon[33017]: pgmap v21249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:45 smithi033 ceph-mon[27845]: pgmap v21249: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:46.888+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:46.888+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:48:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:48:46.888+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:48:47.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:47 smithi194 ceph-mon[33017]: pgmap v21250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:47 smithi033 ceph-mon[27845]: pgmap v21250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:47 smithi096 ceph-mon[32753]: pgmap v21250: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:49.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39467.309234 secs 2024-04-26T22:48:49.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:49 smithi033 ceph-mon[27845]: pgmap v21251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:49.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39470.297249 secs 2024-04-26T22:48:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39467.309234 secs 2024-04-26T22:48:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:49 smithi194 ceph-mon[33017]: pgmap v21251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39470.297249 secs 2024-04-26T22:48:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39467.309234 secs 2024-04-26T22:48:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:49 smithi096 ceph-mon[32753]: pgmap v21251: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:49.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39470.297249 secs 2024-04-26T22:48:51.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:51 smithi194 ceph-mon[33017]: pgmap v21252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:51 smithi033 ceph-mon[27845]: pgmap v21252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:51 smithi096 ceph-mon[32753]: pgmap v21252: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:53 smithi194 ceph-mon[33017]: pgmap v21253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:53 smithi033 ceph-mon[27845]: pgmap v21253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:53 smithi096 ceph-mon[32753]: pgmap v21253: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39472.309320 secs 2024-04-26T22:48:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39475.297348 secs 2024-04-26T22:48:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39472.309320 secs 2024-04-26T22:48:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39475.297348 secs 2024-04-26T22:48:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39472.309320 secs 2024-04-26T22:48:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39475.297348 secs 2024-04-26T22:48:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:55 smithi194 ceph-mon[33017]: pgmap v21254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:55 smithi033 ceph-mon[27845]: pgmap v21254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:55 smithi096 ceph-mon[32753]: pgmap v21254: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:57 smithi194 ceph-mon[33017]: pgmap v21255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:57 smithi033 ceph-mon[27845]: pgmap v21255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:57 smithi096 ceph-mon[32753]: pgmap v21255: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:48:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39477.309402 secs 2024-04-26T22:48:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:59 smithi194 ceph-mon[33017]: pgmap v21256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:48:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39480.297428 secs 2024-04-26T22:48:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39477.309402 secs 2024-04-26T22:48:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:59 smithi033 ceph-mon[27845]: pgmap v21256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:48:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39480.297428 secs 2024-04-26T22:48:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39477.309402 secs 2024-04-26T22:48:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:59 smithi096 ceph-mon[32753]: pgmap v21256: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:48:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:48:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39480.297428 secs 2024-04-26T22:49:01.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:01 smithi194 ceph-mon[33017]: pgmap v21257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:01 smithi033 ceph-mon[27845]: pgmap v21257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:01 smithi096 ceph-mon[32753]: pgmap v21257: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:01.904+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:01.904+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:01.904+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:49:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:03 smithi194 ceph-mon[33017]: pgmap v21258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:03 smithi033 ceph-mon[27845]: pgmap v21258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:03 smithi096 ceph-mon[32753]: pgmap v21258: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39482.309456 secs 2024-04-26T22:49:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39485.297547 secs 2024-04-26T22:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39482.309456 secs 2024-04-26T22:49:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39485.297547 secs 2024-04-26T22:49:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39482.309456 secs 2024-04-26T22:49:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39485.297547 secs 2024-04-26T22:49:05.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:05 smithi033 ceph-mon[27845]: pgmap v21259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:05.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:05 smithi096 ceph-mon[32753]: pgmap v21259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:05 smithi194 ceph-mon[33017]: pgmap v21259: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:07 smithi194 ceph-mon[33017]: pgmap v21260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:07 smithi033 ceph-mon[27845]: pgmap v21260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:07 smithi096 ceph-mon[32753]: pgmap v21260: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39487.309539 secs 2024-04-26T22:49:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:09 smithi194 ceph-mon[33017]: pgmap v21261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39490.297655 secs 2024-04-26T22:49:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39487.309539 secs 2024-04-26T22:49:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:09 smithi033 ceph-mon[27845]: pgmap v21261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39490.297655 secs 2024-04-26T22:49:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39487.309539 secs 2024-04-26T22:49:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:09 smithi096 ceph-mon[32753]: pgmap v21261: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39490.297655 secs 2024-04-26T22:49:11.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:11 smithi194 ceph-mon[33017]: pgmap v21262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:11 smithi033 ceph-mon[27845]: pgmap v21262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:11 smithi096 ceph-mon[32753]: pgmap v21262: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:13.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:13 smithi194 ceph-mon[33017]: pgmap v21263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:13 smithi033 ceph-mon[27845]: pgmap v21263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:13 smithi096 ceph-mon[32753]: pgmap v21263: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39492.309623 secs 2024-04-26T22:49:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39495.297776 secs 2024-04-26T22:49:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39492.309623 secs 2024-04-26T22:49:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39495.297776 secs 2024-04-26T22:49:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39492.309623 secs 2024-04-26T22:49:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39495.297776 secs 2024-04-26T22:49:15.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:15 smithi194 ceph-mon[33017]: pgmap v21264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:15 smithi033 ceph-mon[27845]: pgmap v21264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:15 smithi096 ceph-mon[32753]: pgmap v21264: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:16.916+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:16.916+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:16.916+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:49:17.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:17 smithi194 ceph-mon[33017]: pgmap v21265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:17 smithi033 ceph-mon[27845]: pgmap v21265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:17 smithi096 ceph-mon[32753]: pgmap v21265: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39497.309717 secs 2024-04-26T22:49:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:19 smithi194 ceph-mon[33017]: pgmap v21266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39500.297902 secs 2024-04-26T22:49:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39497.309717 secs 2024-04-26T22:49:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:19 smithi033 ceph-mon[27845]: pgmap v21266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39500.297902 secs 2024-04-26T22:49:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39497.309717 secs 2024-04-26T22:49:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:19 smithi096 ceph-mon[32753]: pgmap v21266: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39500.297902 secs 2024-04-26T22:49:21.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:21 smithi194 ceph-mon[33017]: pgmap v21267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:21 smithi033 ceph-mon[27845]: pgmap v21267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:21 smithi096 ceph-mon[32753]: pgmap v21267: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:23.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:23 smithi194 ceph-mon[33017]: pgmap v21268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:23 smithi033 ceph-mon[27845]: pgmap v21268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:23 smithi096 ceph-mon[32753]: pgmap v21268: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39502.309800 secs 2024-04-26T22:49:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39505.298019 secs 2024-04-26T22:49:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39502.309800 secs 2024-04-26T22:49:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39505.298019 secs 2024-04-26T22:49:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39502.309800 secs 2024-04-26T22:49:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39505.298019 secs 2024-04-26T22:49:25.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:25 smithi194 ceph-mon[33017]: pgmap v21269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:25 smithi033 ceph-mon[27845]: pgmap v21269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:25 smithi096 ceph-mon[32753]: pgmap v21269: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:27.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:27 smithi194 ceph-mon[33017]: pgmap v21270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:27 smithi033 ceph-mon[27845]: pgmap v21270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:27 smithi096 ceph-mon[32753]: pgmap v21270: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39507.309886 secs 2024-04-26T22:49:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:29 smithi194 ceph-mon[33017]: pgmap v21271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39510.298138 secs 2024-04-26T22:49:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39507.309886 secs 2024-04-26T22:49:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:29 smithi033 ceph-mon[27845]: pgmap v21271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39510.298138 secs 2024-04-26T22:49:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39507.309886 secs 2024-04-26T22:49:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:29 smithi096 ceph-mon[32753]: pgmap v21271: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39510.298138 secs 2024-04-26T22:49:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:31 smithi194 ceph-mon[33017]: pgmap v21272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:31 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:49:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:31 smithi033 ceph-mon[27845]: pgmap v21272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:31 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:49:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:31 smithi096 ceph-mon[32753]: pgmap v21272: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:31 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:49:31.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:31.927+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:31.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:31.927+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:31.927 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:31.927+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:49:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:49:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:49:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:32 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:49:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:49:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:49:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:32 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:49:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:49:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:49:32.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:32 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:49:33.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:33 smithi194 ceph-mon[33017]: pgmap v21273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:33.607 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:33 smithi194 ceph-mon[33017]: pgmap v21274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:33 smithi033 ceph-mon[27845]: pgmap v21273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:33 smithi033 ceph-mon[27845]: pgmap v21274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:33 smithi096 ceph-mon[32753]: pgmap v21273: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:33.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:33 smithi096 ceph-mon[32753]: pgmap v21274: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39512.309990 secs 2024-04-26T22:49:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39515.298277 secs 2024-04-26T22:49:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39512.309990 secs 2024-04-26T22:49:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39515.298277 secs 2024-04-26T22:49:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39512.309990 secs 2024-04-26T22:49:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39515.298277 secs 2024-04-26T22:49:35.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:35 smithi033 ceph-mon[27845]: pgmap v21275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:35.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:35 smithi096 ceph-mon[32753]: pgmap v21275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:35 smithi194 ceph-mon[33017]: pgmap v21275: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:37 smithi033 ceph-mon[27845]: pgmap v21276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:37 smithi096 ceph-mon[32753]: pgmap v21276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:37 smithi194 ceph-mon[33017]: pgmap v21276: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:39 smithi033 ceph-mon[27845]: pgmap v21277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39517.310082 secs 2024-04-26T22:49:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39520.298340 secs 2024-04-26T22:49:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:39 smithi096 ceph-mon[32753]: pgmap v21277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39517.310082 secs 2024-04-26T22:49:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39520.298340 secs 2024-04-26T22:49:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:39 smithi194 ceph-mon[33017]: pgmap v21277: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39517.310082 secs 2024-04-26T22:49:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39520.298340 secs 2024-04-26T22:49:41.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:41 smithi033 ceph-mon[27845]: pgmap v21278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:41.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:41 smithi096 ceph-mon[32753]: pgmap v21278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:41 smithi194 ceph-mon[33017]: pgmap v21278: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:49:43.525 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:43.526+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T22:49:43.525 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:43.526+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:49:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:43.526+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 308 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T22:49:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:43.526+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 308 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:49:43.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:43 smithi194 ceph-mon[33017]: pgmap v21279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:49:43.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:43 smithi033 ceph-mon[27845]: pgmap v21279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:49:43.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:43 smithi096 ceph-mon[32753]: pgmap v21279: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:49:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39522.310185 secs 2024-04-26T22:49:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39525.298461 secs 2024-04-26T22:49:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39522.310185 secs 2024-04-26T22:49:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39525.298461 secs 2024-04-26T22:49:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39522.310185 secs 2024-04-26T22:49:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39525.298461 secs 2024-04-26T22:49:45.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:45 smithi033 ceph-mon[27845]: pgmap v21280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:45.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:45 smithi096 ceph-mon[32753]: pgmap v21280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:45 smithi194 ceph-mon[33017]: pgmap v21280: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:46.940+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:46.940+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:49:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:49:46.940+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:49:47.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:47 smithi033 ceph-mon[27845]: pgmap v21281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:47.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:47 smithi096 ceph-mon[32753]: pgmap v21281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:47 smithi194 ceph-mon[33017]: pgmap v21281: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:49 smithi033 ceph-mon[27845]: pgmap v21282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39527.310243 secs 2024-04-26T22:49:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39530.298576 secs 2024-04-26T22:49:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:49 smithi096 ceph-mon[32753]: pgmap v21282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39527.310243 secs 2024-04-26T22:49:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39530.298576 secs 2024-04-26T22:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:49 smithi194 ceph-mon[33017]: pgmap v21282: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39527.310243 secs 2024-04-26T22:49:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39530.298576 secs 2024-04-26T22:49:51.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:51 smithi033 ceph-mon[27845]: pgmap v21283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:51.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:51 smithi096 ceph-mon[32753]: pgmap v21283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:51 smithi194 ceph-mon[33017]: pgmap v21283: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:53.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:53 smithi033 ceph-mon[27845]: pgmap v21284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:53 smithi096 ceph-mon[32753]: pgmap v21284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:53 smithi194 ceph-mon[33017]: pgmap v21284: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39532.310321 secs 2024-04-26T22:49:54.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39535.298656 secs 2024-04-26T22:49:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39532.310321 secs 2024-04-26T22:49:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39535.298656 secs 2024-04-26T22:49:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39532.310321 secs 2024-04-26T22:49:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39535.298656 secs 2024-04-26T22:49:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:55 smithi033 ceph-mon[27845]: pgmap v21285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:55 smithi096 ceph-mon[32753]: pgmap v21285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:55 smithi194 ceph-mon[33017]: pgmap v21285: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:57.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:57 smithi096 ceph-mon[32753]: pgmap v21286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:57 smithi194 ceph-mon[33017]: pgmap v21286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:57 smithi033 ceph-mon[27845]: pgmap v21286: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:59 smithi194 ceph-mon[33017]: pgmap v21287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39537.310408 secs 2024-04-26T22:49:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:49:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39540.298776 secs 2024-04-26T22:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:59 smithi033 ceph-mon[27845]: pgmap v21287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39537.310408 secs 2024-04-26T22:49:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:49:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39540.298776 secs 2024-04-26T22:49:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:59 smithi096 ceph-mon[32753]: pgmap v21287: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:49:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39537.310408 secs 2024-04-26T22:49:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:49:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39540.298776 secs 2024-04-26T22:50:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:50:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:50:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T22:50:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:01 smithi194 ceph-mon[33017]: pgmap v21288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:01 smithi033 ceph-mon[27845]: pgmap v21288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:01 smithi096 ceph-mon[32753]: pgmap v21288: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:01.956+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:01.956+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:01.956+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:50:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:03 smithi194 ceph-mon[33017]: pgmap v21289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:03 smithi033 ceph-mon[27845]: pgmap v21289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:03 smithi096 ceph-mon[32753]: pgmap v21289: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39542.310493 secs 2024-04-26T22:50:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39545.298913 secs 2024-04-26T22:50:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39542.310493 secs 2024-04-26T22:50:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39545.298913 secs 2024-04-26T22:50:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39542.310493 secs 2024-04-26T22:50:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39545.298913 secs 2024-04-26T22:50:05.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:05 smithi096 ceph-mon[32753]: pgmap v21290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:05 smithi194 ceph-mon[33017]: pgmap v21290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:05 smithi033 ceph-mon[27845]: pgmap v21290: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:07 smithi194 ceph-mon[33017]: pgmap v21291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:07 smithi033 ceph-mon[27845]: pgmap v21291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:07 smithi096 ceph-mon[32753]: pgmap v21291: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:09 smithi194 ceph-mon[33017]: pgmap v21292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39547.310578 secs 2024-04-26T22:50:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39550.299042 secs 2024-04-26T22:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:09 smithi033 ceph-mon[27845]: pgmap v21292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39547.310578 secs 2024-04-26T22:50:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39550.299042 secs 2024-04-26T22:50:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:09 smithi096 ceph-mon[32753]: pgmap v21292: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39547.310578 secs 2024-04-26T22:50:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39550.299042 secs 2024-04-26T22:50:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:11 smithi194 ceph-mon[33017]: pgmap v21293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:11 smithi033 ceph-mon[27845]: pgmap v21293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:11 smithi096 ceph-mon[32753]: pgmap v21293: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:12 smithi194 ceph-mon[33017]: pgmap v21294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:12 smithi033 ceph-mon[27845]: pgmap v21294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:12 smithi096 ceph-mon[32753]: pgmap v21294: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39552.310662 secs 2024-04-26T22:50:14.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39552.310662 secs 2024-04-26T22:50:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39552.310662 secs 2024-04-26T22:50:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:14 smithi194 ceph-mon[33017]: pgmap v21295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39555.299176 secs 2024-04-26T22:50:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:14 smithi033 ceph-mon[27845]: pgmap v21295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39555.299176 secs 2024-04-26T22:50:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:14 smithi096 ceph-mon[32753]: pgmap v21295: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39555.299176 secs 2024-04-26T22:50:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:16.972+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:16.972+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:16.972+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:50:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:16 smithi194 ceph-mon[33017]: pgmap v21296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:16 smithi033 ceph-mon[27845]: pgmap v21296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:16 smithi096 ceph-mon[32753]: pgmap v21296: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:18 smithi194 ceph-mon[33017]: pgmap v21297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39557.310751 secs 2024-04-26T22:50:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:18 smithi033 ceph-mon[27845]: pgmap v21297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39557.310751 secs 2024-04-26T22:50:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:18 smithi096 ceph-mon[32753]: pgmap v21297: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:19.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39557.310751 secs 2024-04-26T22:50:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39560.299238 secs 2024-04-26T22:50:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39560.299238 secs 2024-04-26T22:50:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39560.299238 secs 2024-04-26T22:50:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:20 smithi194 ceph-mon[33017]: pgmap v21298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:20 smithi033 ceph-mon[27845]: pgmap v21298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:20 smithi096 ceph-mon[32753]: pgmap v21298: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:22 smithi194 ceph-mon[33017]: pgmap v21299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:22 smithi033 ceph-mon[27845]: pgmap v21299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:22 smithi096 ceph-mon[32753]: pgmap v21299: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39562.310833 secs 2024-04-26T22:50:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39562.310833 secs 2024-04-26T22:50:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39562.310833 secs 2024-04-26T22:50:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:24 smithi194 ceph-mon[33017]: pgmap v21300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39565.299357 secs 2024-04-26T22:50:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:24 smithi033 ceph-mon[27845]: pgmap v21300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39565.299357 secs 2024-04-26T22:50:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:24 smithi096 ceph-mon[32753]: pgmap v21300: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39565.299357 secs 2024-04-26T22:50:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:26 smithi194 ceph-mon[33017]: pgmap v21301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:26 smithi033 ceph-mon[27845]: pgmap v21301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:26 smithi096 ceph-mon[32753]: pgmap v21301: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:28.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:28.533+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T22:50:28.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:28.533+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:50:28.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:28.534+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 309 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T22:50:28.534 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:28.534+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 309 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:50:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:29 smithi033 ceph-mon[27845]: pgmap v21302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39567.310917 secs 2024-04-26T22:50:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39570.299476 secs 2024-04-26T22:50:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:29 smithi096 ceph-mon[32753]: pgmap v21302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39567.310917 secs 2024-04-26T22:50:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39570.299476 secs 2024-04-26T22:50:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:29 smithi194 ceph-mon[33017]: pgmap v21302: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39567.310917 secs 2024-04-26T22:50:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39570.299476 secs 2024-04-26T22:50:31.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:31 smithi096 ceph-mon[32753]: pgmap v21303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:31.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:31 smithi194 ceph-mon[33017]: pgmap v21303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:31.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:31 smithi033 ceph-mon[27845]: pgmap v21303: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:31.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:31.975+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:31.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:31.975+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:31.975 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:31.975+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:50:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:32 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:50:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:32 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:50:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:32 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:50:33.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:33 smithi194 ceph-mon[33017]: pgmap v21304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:33.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:33 smithi033 ceph-mon[27845]: pgmap v21304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:33.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:33 smithi096 ceph-mon[32753]: pgmap v21304: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39572.311024 secs 2024-04-26T22:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:34 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:34 smithi194 ceph-mon[33017]: pgmap v21305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:34 smithi194 ceph-mon[33017]: pgmap v21306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:34 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:50:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39575.299557 secs 2024-04-26T22:50:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39572.311024 secs 2024-04-26T22:50:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:50:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:34 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:50:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:34 smithi033 ceph-mon[27845]: pgmap v21305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:34 smithi033 ceph-mon[27845]: pgmap v21306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:34 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:50:34.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39575.299557 secs 2024-04-26T22:50:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39572.311024 secs 2024-04-26T22:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:34 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:34 smithi096 ceph-mon[32753]: pgmap v21305: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:34 smithi096 ceph-mon[32753]: pgmap v21306: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:34 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:50:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39575.299557 secs 2024-04-26T22:50:35.844 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:35 smithi096 ceph-mon[32753]: pgmap v21307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:35 smithi194 ceph-mon[33017]: pgmap v21307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:35 smithi033 ceph-mon[27845]: pgmap v21307: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:37 smithi194 ceph-mon[33017]: pgmap v21308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:37 smithi033 ceph-mon[27845]: pgmap v21308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:37 smithi096 ceph-mon[32753]: pgmap v21308: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:39.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:38 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39577.311113 secs 2024-04-26T22:50:39.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:38 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39577.311113 secs 2024-04-26T22:50:39.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:38 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39577.311113 secs 2024-04-26T22:50:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39580.299630 secs 2024-04-26T22:50:40.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:39 smithi194 ceph-mon[33017]: pgmap v21309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:50:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39580.299630 secs 2024-04-26T22:50:40.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:39 smithi033 ceph-mon[27845]: pgmap v21309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:50:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39580.299630 secs 2024-04-26T22:50:40.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:39 smithi096 ceph-mon[32753]: pgmap v21309: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-26T22:50:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:41 smithi194 ceph-mon[33017]: pgmap v21310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:41 smithi033 ceph-mon[27845]: pgmap v21310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:41 smithi096 ceph-mon[32753]: pgmap v21310: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:43 smithi194 ceph-mon[33017]: pgmap v21311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:50:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:43 smithi033 ceph-mon[27845]: pgmap v21311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:50:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:43 smithi096 ceph-mon[32753]: pgmap v21311: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:50:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39582.311172 secs 2024-04-26T22:50:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39585.299765 secs 2024-04-26T22:50:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39582.311172 secs 2024-04-26T22:50:44.938 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39585.299765 secs 2024-04-26T22:50:44.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39582.311172 secs 2024-04-26T22:50:44.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39585.299765 secs 2024-04-26T22:50:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:45 smithi194 ceph-mon[33017]: pgmap v21312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:45 smithi033 ceph-mon[27845]: pgmap v21312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:45 smithi096 ceph-mon[32753]: pgmap v21312: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:46.976+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:46.976+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:50:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:50:46.976+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:50:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:47 smithi194 ceph-mon[33017]: pgmap v21313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:47 smithi033 ceph-mon[27845]: pgmap v21313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:47 smithi096 ceph-mon[32753]: pgmap v21313: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39587.311257 secs 2024-04-26T22:50:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39587.311257 secs 2024-04-26T22:50:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39587.311257 secs 2024-04-26T22:50:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39590.299845 secs 2024-04-26T22:50:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:49 smithi194 ceph-mon[33017]: pgmap v21314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39590.299845 secs 2024-04-26T22:50:50.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:49 smithi033 ceph-mon[27845]: pgmap v21314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39590.299845 secs 2024-04-26T22:50:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:49 smithi096 ceph-mon[32753]: pgmap v21314: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:50:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:51 smithi194 ceph-mon[33017]: pgmap v21315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:50:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:51 smithi033 ceph-mon[27845]: pgmap v21315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:50:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:51 smithi096 ceph-mon[32753]: pgmap v21315: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:50:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:53 smithi194 ceph-mon[33017]: pgmap v21316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:53 smithi033 ceph-mon[27845]: pgmap v21316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:53 smithi096 ceph-mon[32753]: pgmap v21316: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39592.311319 secs 2024-04-26T22:50:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39595.299978 secs 2024-04-26T22:50:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39592.311319 secs 2024-04-26T22:50:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39595.299978 secs 2024-04-26T22:50:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39592.311319 secs 2024-04-26T22:50:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39595.299978 secs 2024-04-26T22:50:55.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:55 smithi194 ceph-mon[33017]: pgmap v21317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:55 smithi033 ceph-mon[27845]: pgmap v21317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:55 smithi096 ceph-mon[32753]: pgmap v21317: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:50:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:57 smithi194 ceph-mon[33017]: pgmap v21318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:57 smithi033 ceph-mon[27845]: pgmap v21318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:57 smithi096 ceph-mon[32753]: pgmap v21318: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:50:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:58 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39597.311405 secs 2024-04-26T22:50:59.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:58 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39597.311405 secs 2024-04-26T22:50:59.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:58 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39597.311405 secs 2024-04-26T22:51:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:50:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39600.300012 secs 2024-04-26T22:51:00.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:50:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39600.300012 secs 2024-04-26T22:51:00.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:50:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39600.300012 secs 2024-04-26T22:51:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:00 smithi194 ceph-mon[33017]: pgmap v21319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:01.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:00 smithi033 ceph-mon[27845]: pgmap v21319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:01.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:00 smithi096 ceph-mon[32753]: pgmap v21319: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:01.989+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:01.989+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:51:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:01.989+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:03.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:02 smithi194 ceph-mon[33017]: pgmap v21320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:51:03.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:02 smithi033 ceph-mon[27845]: pgmap v21320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:51:03.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:02 smithi096 ceph-mon[32753]: pgmap v21320: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:51:04.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:03 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39602.311463 secs 2024-04-26T22:51:04.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:03 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39602.311463 secs 2024-04-26T22:51:04.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:03 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39602.311463 secs 2024-04-26T22:51:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:04 smithi096 ceph-mon[32753]: pgmap v21321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:05.083 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39605.300150 secs 2024-04-26T22:51:05.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:04 smithi194 ceph-mon[33017]: pgmap v21321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39605.300150 secs 2024-04-26T22:51:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:04 smithi033 ceph-mon[27845]: pgmap v21321: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:05.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39605.300150 secs 2024-04-26T22:51:07.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:06 smithi194 ceph-mon[33017]: pgmap v21322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:07.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:06 smithi033 ceph-mon[27845]: pgmap v21322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:07.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:06 smithi096 ceph-mon[32753]: pgmap v21322: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:08 smithi194 ceph-mon[33017]: pgmap v21323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:08 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39607.311545 secs 2024-04-26T22:51:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:08 smithi033 ceph-mon[27845]: pgmap v21323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:09.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:08 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39607.311545 secs 2024-04-26T22:51:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:08 smithi096 ceph-mon[32753]: pgmap v21323: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:09.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:08 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39607.311545 secs 2024-04-26T22:51:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39610.300289 secs 2024-04-26T22:51:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39610.300289 secs 2024-04-26T22:51:10.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39610.300289 secs 2024-04-26T22:51:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:10 smithi194 ceph-mon[33017]: pgmap v21324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:11.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:10 smithi033 ceph-mon[27845]: pgmap v21324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:11.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:10 smithi096 ceph-mon[32753]: pgmap v21324: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:12 smithi194 ceph-mon[33017]: pgmap v21325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:51:13.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:12 smithi033 ceph-mon[27845]: pgmap v21325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:51:13.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:12 smithi096 ceph-mon[32753]: pgmap v21325: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:51:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39612.311629 secs 2024-04-26T22:51:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39612.311629 secs 2024-04-26T22:51:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39612.311629 secs 2024-04-26T22:51:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:14 smithi194 ceph-mon[33017]: pgmap v21326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39615.300355 secs 2024-04-26T22:51:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:14 smithi033 ceph-mon[27845]: pgmap v21326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39615.300355 secs 2024-04-26T22:51:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:14 smithi096 ceph-mon[32753]: pgmap v21326: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39615.300355 secs 2024-04-26T22:51:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:17.006+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:17.006+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:17.006+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:51:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:16 smithi194 ceph-mon[33017]: pgmap v21327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:17.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:16 smithi033 ceph-mon[27845]: pgmap v21327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:17.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:16 smithi096 ceph-mon[32753]: pgmap v21327: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:18 smithi033 ceph-mon[27845]: pgmap v21328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:19.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39617.311715 secs 2024-04-26T22:51:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:18 smithi096 ceph-mon[32753]: pgmap v21328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:19.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39617.311715 secs 2024-04-26T22:51:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:18 smithi194 ceph-mon[33017]: pgmap v21328: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39617.311715 secs 2024-04-26T22:51:20.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39620.300437 secs 2024-04-26T22:51:20.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39620.300437 secs 2024-04-26T22:51:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39620.300437 secs 2024-04-26T22:51:21.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:20 smithi033 ceph-mon[27845]: pgmap v21329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:21.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:20 smithi096 ceph-mon[32753]: pgmap v21329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:21.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:20 smithi194 ceph-mon[33017]: pgmap v21329: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:22 smithi033 ceph-mon[27845]: pgmap v21330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:22 smithi096 ceph-mon[32753]: pgmap v21330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:23.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:22 smithi194 ceph-mon[33017]: pgmap v21330: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39622.311801 secs 2024-04-26T22:51:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39622.311801 secs 2024-04-26T22:51:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39622.311801 secs 2024-04-26T22:51:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:24 smithi033 ceph-mon[27845]: pgmap v21331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39625.300534 secs 2024-04-26T22:51:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:24 smithi096 ceph-mon[32753]: pgmap v21331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39625.300534 secs 2024-04-26T22:51:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:24 smithi194 ceph-mon[33017]: pgmap v21331: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39625.300534 secs 2024-04-26T22:51:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:26 smithi033 ceph-mon[27845]: pgmap v21332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:26 smithi096 ceph-mon[32753]: pgmap v21332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:26 smithi194 ceph-mon[33017]: pgmap v21332: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:28 smithi033 ceph-mon[27845]: pgmap v21333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:29.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39627.311887 secs 2024-04-26T22:51:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:28 smithi096 ceph-mon[32753]: pgmap v21333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:29.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39627.311887 secs 2024-04-26T22:51:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:28 smithi194 ceph-mon[33017]: pgmap v21333: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39627.311887 secs 2024-04-26T22:51:30.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39630.300661 secs 2024-04-26T22:51:30.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39630.300661 secs 2024-04-26T22:51:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39630.300661 secs 2024-04-26T22:51:31.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:30 smithi033 ceph-mon[27845]: pgmap v21334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:31.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:30 smithi096 ceph-mon[32753]: pgmap v21334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:31.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:30 smithi194 ceph-mon[33017]: pgmap v21334: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:32.007+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:32.007+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:32.007+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:51:33.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:32 smithi033 ceph-mon[27845]: pgmap v21335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:33.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:32 smithi096 ceph-mon[32753]: pgmap v21335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:32 smithi194 ceph-mon[33017]: pgmap v21335: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:33 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39632.311993 secs 2024-04-26T22:51:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:33 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:51:34.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:33 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39635.300746 secs 2024-04-26T22:51:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:33 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39632.311993 secs 2024-04-26T22:51:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:33 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:51:34.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:33 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39635.300746 secs 2024-04-26T22:51:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:33 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39632.311993 secs 2024-04-26T22:51:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:33 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:51:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:33 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39635.300746 secs 2024-04-26T22:51:35.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:34 smithi033 ceph-mon[27845]: pgmap v21336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:35.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:34 smithi096 ceph-mon[32753]: pgmap v21336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:35.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:34 smithi194 ceph-mon[33017]: pgmap v21336: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:51:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:35 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:51:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:35 smithi194 ceph-mon[33017]: pgmap v21337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:51:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:35 smithi194 ceph-mon[33017]: pgmap v21338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:35 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:51:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:51:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:35 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:51:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:35 smithi033 ceph-mon[27845]: pgmap v21337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:51:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:35 smithi033 ceph-mon[27845]: pgmap v21338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:36.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:35 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:51:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:51:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:35 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:51:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:35 smithi096 ceph-mon[32753]: pgmap v21337: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:51:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:35 smithi096 ceph-mon[32753]: pgmap v21338: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:36.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:35 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:51:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:38 smithi033 ceph-mon[27845]: pgmap v21339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:38 smithi096 ceph-mon[32753]: pgmap v21339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:38 smithi194 ceph-mon[33017]: pgmap v21339: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39637.312076 secs 2024-04-26T22:51:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39640.300873 secs 2024-04-26T22:51:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39637.312076 secs 2024-04-26T22:51:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39640.300873 secs 2024-04-26T22:51:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39637.312076 secs 2024-04-26T22:51:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39640.300873 secs 2024-04-26T22:51:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:40 smithi194 ceph-mon[33017]: pgmap v21340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:51:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:40 smithi033 ceph-mon[27845]: pgmap v21340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:51:40.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:40 smithi096 ceph-mon[32753]: pgmap v21340: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:51:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:42 smithi194 ceph-mon[33017]: pgmap v21341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:42 smithi033 ceph-mon[27845]: pgmap v21341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:42 smithi096 ceph-mon[32753]: pgmap v21341: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:44.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39642.312160 secs 2024-04-26T22:51:44.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:44 smithi033 ceph-mon[27845]: pgmap v21342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:44.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39645.300999 secs 2024-04-26T22:51:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39642.312160 secs 2024-04-26T22:51:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:44 smithi194 ceph-mon[33017]: pgmap v21342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39645.300999 secs 2024-04-26T22:51:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39642.312160 secs 2024-04-26T22:51:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:44 smithi096 ceph-mon[32753]: pgmap v21342: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:51:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39645.300999 secs 2024-04-26T22:51:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:46 smithi194 ceph-mon[33017]: pgmap v21343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:51:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:46 smithi033 ceph-mon[27845]: pgmap v21343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:51:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:46 smithi096 ceph-mon[32753]: pgmap v21343: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:51:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:47.008+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:47.008+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:51:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:51:47.008+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:51:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:48 smithi194 ceph-mon[33017]: pgmap v21344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:48 smithi033 ceph-mon[27845]: pgmap v21344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:48 smithi096 ceph-mon[32753]: pgmap v21344: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39647.312219 secs 2024-04-26T22:51:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39650.301145 secs 2024-04-26T22:51:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39647.312219 secs 2024-04-26T22:51:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39650.301145 secs 2024-04-26T22:51:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39647.312219 secs 2024-04-26T22:51:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39650.301145 secs 2024-04-26T22:51:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:50 smithi194 ceph-mon[33017]: pgmap v21345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:50 smithi033 ceph-mon[27845]: pgmap v21345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:50 smithi096 ceph-mon[32753]: pgmap v21345: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:52 smithi194 ceph-mon[33017]: pgmap v21346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:52 smithi033 ceph-mon[27845]: pgmap v21346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:52 smithi096 ceph-mon[32753]: pgmap v21346: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39652.312309 secs 2024-04-26T22:51:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:54 smithi194 ceph-mon[33017]: pgmap v21347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39655.301298 secs 2024-04-26T22:51:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39652.312309 secs 2024-04-26T22:51:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:54 smithi033 ceph-mon[27845]: pgmap v21347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39655.301298 secs 2024-04-26T22:51:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39652.312309 secs 2024-04-26T22:51:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:54 smithi096 ceph-mon[32753]: pgmap v21347: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39655.301298 secs 2024-04-26T22:51:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:56 smithi033 ceph-mon[27845]: pgmap v21348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:56 smithi194 ceph-mon[33017]: pgmap v21348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:56 smithi096 ceph-mon[32753]: pgmap v21348: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:51:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:58 smithi194 ceph-mon[33017]: pgmap v21349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:58 smithi033 ceph-mon[27845]: pgmap v21349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:58 smithi096 ceph-mon[32753]: pgmap v21349: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:51:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39657.312384 secs 2024-04-26T22:51:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:51:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39660.301425 secs 2024-04-26T22:51:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39657.312384 secs 2024-04-26T22:51:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:51:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39660.301425 secs 2024-04-26T22:51:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39657.312384 secs 2024-04-26T22:51:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:51:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39660.301425 secs 2024-04-26T22:52:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:00 smithi194 ceph-mon[33017]: pgmap v21350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:00 smithi033 ceph-mon[27845]: pgmap v21350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:00 smithi096 ceph-mon[32753]: pgmap v21350: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:02.011+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:02.011+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:02.012+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:52:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:02 smithi194 ceph-mon[33017]: pgmap v21351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:02 smithi033 ceph-mon[27845]: pgmap v21351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:02 smithi096 ceph-mon[32753]: pgmap v21351: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39662.312467 secs 2024-04-26T22:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:04 smithi194 ceph-mon[33017]: pgmap v21352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39665.301538 secs 2024-04-26T22:52:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39662.312467 secs 2024-04-26T22:52:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:04 smithi033 ceph-mon[27845]: pgmap v21352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39665.301538 secs 2024-04-26T22:52:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39662.312467 secs 2024-04-26T22:52:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:04 smithi096 ceph-mon[32753]: pgmap v21352: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39665.301538 secs 2024-04-26T22:52:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:06 smithi194 ceph-mon[33017]: pgmap v21353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:06 smithi033 ceph-mon[27845]: pgmap v21353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:06 smithi096 ceph-mon[32753]: pgmap v21353: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:08 smithi194 ceph-mon[33017]: pgmap v21354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:08 smithi033 ceph-mon[27845]: pgmap v21354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:08 smithi096 ceph-mon[32753]: pgmap v21354: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39667.312551 secs 2024-04-26T22:52:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39670.301666 secs 2024-04-26T22:52:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39667.312551 secs 2024-04-26T22:52:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39670.301666 secs 2024-04-26T22:52:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39667.312551 secs 2024-04-26T22:52:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39670.301666 secs 2024-04-26T22:52:10.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:10 smithi194 ceph-mon[33017]: pgmap v21355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:10 smithi033 ceph-mon[27845]: pgmap v21355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:10 smithi096 ceph-mon[32753]: pgmap v21355: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:12.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:12 smithi194 ceph-mon[33017]: pgmap v21356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:12 smithi033 ceph-mon[27845]: pgmap v21356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:12 smithi096 ceph-mon[32753]: pgmap v21356: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39672.312639 secs 2024-04-26T22:52:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:14 smithi194 ceph-mon[33017]: pgmap v21357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39675.301788 secs 2024-04-26T22:52:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39672.312639 secs 2024-04-26T22:52:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:14 smithi033 ceph-mon[27845]: pgmap v21357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39675.301788 secs 2024-04-26T22:52:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39672.312639 secs 2024-04-26T22:52:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:14 smithi096 ceph-mon[32753]: pgmap v21357: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39675.301788 secs 2024-04-26T22:52:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:16 smithi194 ceph-mon[33017]: pgmap v21358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:16 smithi033 ceph-mon[27845]: pgmap v21358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:16 smithi096 ceph-mon[32753]: pgmap v21358: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:17.016+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:17.016+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:17.016+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:52:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:18 smithi194 ceph-mon[33017]: pgmap v21359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:18 smithi033 ceph-mon[27845]: pgmap v21359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:18 smithi096 ceph-mon[32753]: pgmap v21359: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39677.312723 secs 2024-04-26T22:52:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39680.301925 secs 2024-04-26T22:52:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39677.312723 secs 2024-04-26T22:52:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39680.301925 secs 2024-04-26T22:52:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39677.312723 secs 2024-04-26T22:52:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39680.301925 secs 2024-04-26T22:52:20.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:20 smithi194 ceph-mon[33017]: pgmap v21360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:20 smithi033 ceph-mon[27845]: pgmap v21360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:20 smithi096 ceph-mon[32753]: pgmap v21360: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:22.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:22 smithi194 ceph-mon[33017]: pgmap v21361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:22 smithi033 ceph-mon[27845]: pgmap v21361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:22 smithi096 ceph-mon[32753]: pgmap v21361: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39682.312806 secs 2024-04-26T22:52:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:24 smithi194 ceph-mon[33017]: pgmap v21362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:24.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39685.302044 secs 2024-04-26T22:52:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39682.312806 secs 2024-04-26T22:52:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:24 smithi033 ceph-mon[27845]: pgmap v21362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39685.302044 secs 2024-04-26T22:52:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39682.312806 secs 2024-04-26T22:52:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:24 smithi096 ceph-mon[32753]: pgmap v21362: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39685.302044 secs 2024-04-26T22:52:26.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:26 smithi194 ceph-mon[33017]: pgmap v21363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:26 smithi033 ceph-mon[27845]: pgmap v21363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:26 smithi096 ceph-mon[32753]: pgmap v21363: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:28 smithi194 ceph-mon[33017]: pgmap v21364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:28 smithi033 ceph-mon[27845]: pgmap v21364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:28 smithi096 ceph-mon[32753]: pgmap v21364: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39687.312891 secs 2024-04-26T22:52:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39690.302164 secs 2024-04-26T22:52:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39687.312891 secs 2024-04-26T22:52:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39690.302164 secs 2024-04-26T22:52:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39687.312891 secs 2024-04-26T22:52:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39690.302164 secs 2024-04-26T22:52:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:30 smithi033 ceph-mon[27845]: pgmap v21365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:30 smithi096 ceph-mon[32753]: pgmap v21365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:30 smithi194 ceph-mon[33017]: pgmap v21365: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:32.031+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:32.031+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:52:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:32.031+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:32 smithi033 ceph-mon[27845]: pgmap v21366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:32 smithi096 ceph-mon[32753]: pgmap v21366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:32 smithi194 ceph-mon[33017]: pgmap v21366: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39692.312993 secs 2024-04-26T22:52:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:34 smithi033 ceph-mon[27845]: pgmap v21367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39695.302241 secs 2024-04-26T22:52:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39692.312993 secs 2024-04-26T22:52:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:34 smithi096 ceph-mon[32753]: pgmap v21367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:34.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39695.302241 secs 2024-04-26T22:52:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39692.312993 secs 2024-04-26T22:52:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:34 smithi194 ceph-mon[33017]: pgmap v21367: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39695.302241 secs 2024-04-26T22:52:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:36 smithi033 ceph-mon[27845]: pgmap v21368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:36 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:52:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:36 smithi096 ceph-mon[32753]: pgmap v21368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:36 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:36 smithi194 ceph-mon[33017]: pgmap v21368: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:36 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:52:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:52:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:52:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:37 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:52:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:52:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:52:37.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:37 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:52:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:52:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:52:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:37 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:52:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:38 smithi033 ceph-mon[27845]: pgmap v21369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:38 smithi033 ceph-mon[27845]: pgmap v21370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:38 smithi096 ceph-mon[32753]: pgmap v21369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:38 smithi096 ceph-mon[32753]: pgmap v21370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:38 smithi194 ceph-mon[33017]: pgmap v21369: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:38 smithi194 ceph-mon[33017]: pgmap v21370: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39697.313083 secs 2024-04-26T22:52:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39700.302354 secs 2024-04-26T22:52:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39697.313083 secs 2024-04-26T22:52:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39700.302354 secs 2024-04-26T22:52:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39697.313083 secs 2024-04-26T22:52:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39700.302354 secs 2024-04-26T22:52:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:40 smithi033 ceph-mon[27845]: pgmap v21371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:40 smithi096 ceph-mon[32753]: pgmap v21371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:40 smithi194 ceph-mon[33017]: pgmap v21371: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:42 smithi033 ceph-mon[27845]: pgmap v21372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:42 smithi096 ceph-mon[32753]: pgmap v21372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:42 smithi194 ceph-mon[33017]: pgmap v21372: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:44 smithi033 ceph-mon[27845]: pgmap v21373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39702.313175 secs 2024-04-26T22:52:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39705.302452 secs 2024-04-26T22:52:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:44 smithi096 ceph-mon[32753]: pgmap v21373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39702.313175 secs 2024-04-26T22:52:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39705.302452 secs 2024-04-26T22:52:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:44 smithi194 ceph-mon[33017]: pgmap v21373: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39702.313175 secs 2024-04-26T22:52:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39705.302452 secs 2024-04-26T22:52:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:46 smithi033 ceph-mon[27845]: pgmap v21374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:46 smithi096 ceph-mon[32753]: pgmap v21374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:46 smithi194 ceph-mon[33017]: pgmap v21374: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:52:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:47.035+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:47.035+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:52:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:52:47.035+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:52:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:48 smithi033 ceph-mon[27845]: pgmap v21375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:52:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:48 smithi096 ceph-mon[32753]: pgmap v21375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:52:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:48 smithi194 ceph-mon[33017]: pgmap v21375: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:52:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39707.313281 secs 2024-04-26T22:52:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39710.302542 secs 2024-04-26T22:52:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39707.313281 secs 2024-04-26T22:52:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39710.302542 secs 2024-04-26T22:52:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39707.313281 secs 2024-04-26T22:52:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39710.302542 secs 2024-04-26T22:52:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:50 smithi194 ceph-mon[33017]: pgmap v21376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:50 smithi033 ceph-mon[27845]: pgmap v21376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:50 smithi096 ceph-mon[32753]: pgmap v21376: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:52 smithi194 ceph-mon[33017]: pgmap v21377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:52 smithi033 ceph-mon[27845]: pgmap v21377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:52 smithi096 ceph-mon[32753]: pgmap v21377: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:54 smithi194 ceph-mon[33017]: pgmap v21378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39712.313339 secs 2024-04-26T22:52:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39715.302660 secs 2024-04-26T22:52:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:54 smithi033 ceph-mon[27845]: pgmap v21378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39712.313339 secs 2024-04-26T22:52:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39715.302660 secs 2024-04-26T22:52:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:54 smithi096 ceph-mon[32753]: pgmap v21378: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39712.313339 secs 2024-04-26T22:52:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39715.302660 secs 2024-04-26T22:52:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:56 smithi194 ceph-mon[33017]: pgmap v21379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:56 smithi033 ceph-mon[27845]: pgmap v21379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:56 smithi096 ceph-mon[32753]: pgmap v21379: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:52:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:58 smithi194 ceph-mon[33017]: pgmap v21380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:58.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:58 smithi033 ceph-mon[27845]: pgmap v21380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:58 smithi096 ceph-mon[32753]: pgmap v21380: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:52:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39717.313398 secs 2024-04-26T22:52:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:52:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39720.302781 secs 2024-04-26T22:52:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39717.313398 secs 2024-04-26T22:52:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:52:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39720.302781 secs 2024-04-26T22:52:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39717.313398 secs 2024-04-26T22:52:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:52:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39720.302781 secs 2024-04-26T22:53:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:00 smithi194 ceph-mon[33017]: pgmap v21381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:00 smithi033 ceph-mon[27845]: pgmap v21381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:00 smithi096 ceph-mon[32753]: pgmap v21381: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:02.050+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:02.050+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:02.050+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:53:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:02 smithi194 ceph-mon[33017]: pgmap v21382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:02 smithi033 ceph-mon[27845]: pgmap v21382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:02 smithi096 ceph-mon[32753]: pgmap v21382: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:04 smithi194 ceph-mon[33017]: pgmap v21383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39722.313483 secs 2024-04-26T22:53:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39725.302874 secs 2024-04-26T22:53:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:04 smithi033 ceph-mon[27845]: pgmap v21383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39722.313483 secs 2024-04-26T22:53:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39725.302874 secs 2024-04-26T22:53:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:04 smithi096 ceph-mon[32753]: pgmap v21383: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39722.313483 secs 2024-04-26T22:53:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39725.302874 secs 2024-04-26T22:53:06.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:06 smithi096 ceph-mon[32753]: pgmap v21384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:06 smithi194 ceph-mon[33017]: pgmap v21384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:06 smithi033 ceph-mon[27845]: pgmap v21384: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:08 smithi194 ceph-mon[33017]: pgmap v21385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:08 smithi033 ceph-mon[27845]: pgmap v21385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:08 smithi096 ceph-mon[32753]: pgmap v21385: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39727.313568 secs 2024-04-26T22:53:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39730.302995 secs 2024-04-26T22:53:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39727.313568 secs 2024-04-26T22:53:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39730.302995 secs 2024-04-26T22:53:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39727.313568 secs 2024-04-26T22:53:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39730.302995 secs 2024-04-26T22:53:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:10 smithi194 ceph-mon[33017]: pgmap v21386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:10 smithi033 ceph-mon[27845]: pgmap v21386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:10 smithi096 ceph-mon[32753]: pgmap v21386: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:12 smithi194 ceph-mon[33017]: pgmap v21387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:12 smithi033 ceph-mon[27845]: pgmap v21387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:12 smithi096 ceph-mon[32753]: pgmap v21387: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:14 smithi194 ceph-mon[33017]: pgmap v21388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39732.313649 secs 2024-04-26T22:53:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39735.303121 secs 2024-04-26T22:53:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:14 smithi033 ceph-mon[27845]: pgmap v21388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39732.313649 secs 2024-04-26T22:53:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39735.303121 secs 2024-04-26T22:53:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:14 smithi096 ceph-mon[32753]: pgmap v21388: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39732.313649 secs 2024-04-26T22:53:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39735.303121 secs 2024-04-26T22:53:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:15 smithi194 ceph-mon[33017]: pgmap v21389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:15 smithi033 ceph-mon[27845]: pgmap v21389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:15 smithi096 ceph-mon[32753]: pgmap v21389: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:17.052+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:17.052+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:53:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:17.052+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:18.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:17 smithi096 ceph-mon[32753]: pgmap v21390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:17 smithi194 ceph-mon[33017]: pgmap v21390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:17 smithi033 ceph-mon[27845]: pgmap v21390: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39737.313738 secs 2024-04-26T22:53:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39737.313738 secs 2024-04-26T22:53:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39737.313738 secs 2024-04-26T22:53:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39740.303194 secs 2024-04-26T22:53:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:19 smithi194 ceph-mon[33017]: pgmap v21391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39740.303194 secs 2024-04-26T22:53:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:19 smithi033 ceph-mon[27845]: pgmap v21391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39740.303194 secs 2024-04-26T22:53:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:19 smithi096 ceph-mon[32753]: pgmap v21391: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:21 smithi194 ceph-mon[33017]: pgmap v21392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:21 smithi033 ceph-mon[27845]: pgmap v21392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:21 smithi096 ceph-mon[32753]: pgmap v21392: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:23 smithi194 ceph-mon[33017]: pgmap v21393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39742.313825 secs 2024-04-26T22:53:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:23 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39745.303331 secs 2024-04-26T22:53:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:23 smithi033 ceph-mon[27845]: pgmap v21393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39742.313825 secs 2024-04-26T22:53:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:23 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39745.303331 secs 2024-04-26T22:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:23 smithi096 ceph-mon[32753]: pgmap v21393: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39742.313825 secs 2024-04-26T22:53:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:23 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39745.303331 secs 2024-04-26T22:53:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:25 smithi194 ceph-mon[33017]: pgmap v21394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:25 smithi033 ceph-mon[27845]: pgmap v21394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:25 smithi096 ceph-mon[32753]: pgmap v21394: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:28.564+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:53:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:28.564+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T22:53:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:28.565+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 310 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T22:53:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:28.565+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 310 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:53:28.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:28 smithi194 ceph-mon[33017]: pgmap v21395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:28 smithi033 ceph-mon[27845]: pgmap v21395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:28 smithi096 ceph-mon[32753]: pgmap v21395: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:29.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39747.313911 secs 2024-04-26T22:53:29.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39750.303410 secs 2024-04-26T22:53:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39747.313911 secs 2024-04-26T22:53:29.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39750.303410 secs 2024-04-26T22:53:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39747.313911 secs 2024-04-26T22:53:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39750.303410 secs 2024-04-26T22:53:30.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:30 smithi194 ceph-mon[33017]: pgmap v21396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:30 smithi033 ceph-mon[27845]: pgmap v21396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:30 smithi096 ceph-mon[32753]: pgmap v21396: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:32.064+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:32.064+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:32.064+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:53:32.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:32 smithi194 ceph-mon[33017]: pgmap v21397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:32 smithi033 ceph-mon[27845]: pgmap v21397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:32 smithi096 ceph-mon[32753]: pgmap v21397: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:34 smithi194 ceph-mon[33017]: pgmap v21398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39752.313997 secs 2024-04-26T22:53:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39755.303529 secs 2024-04-26T22:53:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:34 smithi033 ceph-mon[27845]: pgmap v21398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39752.313997 secs 2024-04-26T22:53:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39755.303529 secs 2024-04-26T22:53:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:34 smithi096 ceph-mon[32753]: pgmap v21398: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39752.313997 secs 2024-04-26T22:53:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39755.303529 secs 2024-04-26T22:53:36.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:36 smithi194 ceph-mon[33017]: pgmap v21399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:36 smithi033 ceph-mon[27845]: pgmap v21399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:36 smithi096 ceph-mon[32753]: pgmap v21399: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:37.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:37 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:53:37.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:37 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:53:37.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:37 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:53:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:38 smithi194 ceph-mon[33017]: pgmap v21400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:38 smithi033 ceph-mon[27845]: pgmap v21400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:38.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:38 smithi096 ceph-mon[32753]: pgmap v21400: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39757.314092 secs 2024-04-26T22:53:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:53:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:53:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:39 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:53:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39760.303672 secs 2024-04-26T22:53:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39757.314092 secs 2024-04-26T22:53:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:53:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:53:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:39 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:53:39.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39760.303672 secs 2024-04-26T22:53:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39757.314092 secs 2024-04-26T22:53:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:53:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:53:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:39 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:53:39.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39760.303672 secs 2024-04-26T22:53:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:40 smithi194 ceph-mon[33017]: pgmap v21401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:40 smithi194 ceph-mon[33017]: pgmap v21402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:40 smithi033 ceph-mon[27845]: pgmap v21401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:40 smithi033 ceph-mon[27845]: pgmap v21402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:40 smithi096 ceph-mon[32753]: pgmap v21401: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:40 smithi096 ceph-mon[32753]: pgmap v21402: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:42 smithi194 ceph-mon[33017]: pgmap v21403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:42 smithi033 ceph-mon[27845]: pgmap v21403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:42 smithi096 ceph-mon[32753]: pgmap v21403: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:44 smithi194 ceph-mon[33017]: pgmap v21404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39762.314175 secs 2024-04-26T22:53:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39765.303806 secs 2024-04-26T22:53:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:44 smithi033 ceph-mon[27845]: pgmap v21404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39762.314175 secs 2024-04-26T22:53:44.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39765.303806 secs 2024-04-26T22:53:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:44 smithi096 ceph-mon[32753]: pgmap v21404: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39762.314175 secs 2024-04-26T22:53:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39765.303806 secs 2024-04-26T22:53:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:46 smithi194 ceph-mon[33017]: pgmap v21405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:46 smithi033 ceph-mon[27845]: pgmap v21405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:46 smithi096 ceph-mon[32753]: pgmap v21405: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:47.076+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:47.076+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:53:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:53:47.076+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:53:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:48 smithi194 ceph-mon[33017]: pgmap v21406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:48 smithi033 ceph-mon[27845]: pgmap v21406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:48 smithi096 ceph-mon[32753]: pgmap v21406: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:53:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39767.314258 secs 2024-04-26T22:53:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39770.303943 secs 2024-04-26T22:53:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39767.314258 secs 2024-04-26T22:53:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39770.303943 secs 2024-04-26T22:53:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39767.314258 secs 2024-04-26T22:53:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39770.303943 secs 2024-04-26T22:53:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:50 smithi194 ceph-mon[33017]: pgmap v21407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:53:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:50 smithi033 ceph-mon[27845]: pgmap v21407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:53:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:50 smithi096 ceph-mon[32753]: pgmap v21407: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:53:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:52 smithi194 ceph-mon[33017]: pgmap v21408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:52 smithi033 ceph-mon[27845]: pgmap v21408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:52 smithi096 ceph-mon[32753]: pgmap v21408: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:53:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:54 smithi194 ceph-mon[33017]: pgmap v21409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39772.314345 secs 2024-04-26T22:53:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39775.304080 secs 2024-04-26T22:53:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:54 smithi033 ceph-mon[27845]: pgmap v21409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39772.314345 secs 2024-04-26T22:53:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39775.304080 secs 2024-04-26T22:53:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:54 smithi096 ceph-mon[32753]: pgmap v21409: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:53:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39772.314345 secs 2024-04-26T22:53:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39775.304080 secs 2024-04-26T22:53:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:56 smithi194 ceph-mon[33017]: pgmap v21410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:53:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:56 smithi033 ceph-mon[27845]: pgmap v21410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:53:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:56 smithi096 ceph-mon[32753]: pgmap v21410: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:53:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:58 smithi194 ceph-mon[33017]: pgmap v21411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:53:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:58 smithi033 ceph-mon[27845]: pgmap v21411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:53:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:58 smithi096 ceph-mon[32753]: pgmap v21411: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:53:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39777.314431 secs 2024-04-26T22:53:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:53:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39780.304201 secs 2024-04-26T22:53:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39777.314431 secs 2024-04-26T22:53:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:53:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39780.304201 secs 2024-04-26T22:53:59.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39777.314431 secs 2024-04-26T22:53:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:53:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39780.304201 secs 2024-04-26T22:54:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:00 smithi033 ceph-mon[27845]: pgmap v21412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:54:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:00 smithi096 ceph-mon[32753]: pgmap v21412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:54:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:00 smithi194 ceph-mon[33017]: pgmap v21412: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:54:02.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:02.077+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:02.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:02.077+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:02.078 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:02.077+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:54:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:02 smithi033 ceph-mon[27845]: pgmap v21413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:54:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:02 smithi096 ceph-mon[32753]: pgmap v21413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:54:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:02 smithi194 ceph-mon[33017]: pgmap v21413: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:54:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:04 smithi033 ceph-mon[27845]: pgmap v21414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:54:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39782.314519 secs 2024-04-26T22:54:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39785.304303 secs 2024-04-26T22:54:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:04 smithi096 ceph-mon[32753]: pgmap v21414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:54:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39782.314519 secs 2024-04-26T22:54:04.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39785.304303 secs 2024-04-26T22:54:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:04 smithi194 ceph-mon[33017]: pgmap v21414: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:54:04.865 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39782.314519 secs 2024-04-26T22:54:04.865 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39785.304303 secs 2024-04-26T22:54:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:06 smithi033 ceph-mon[27845]: pgmap v21415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:54:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:06 smithi096 ceph-mon[32753]: pgmap v21415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:54:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:06 smithi194 ceph-mon[33017]: pgmap v21415: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:54:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:08 smithi033 ceph-mon[27845]: pgmap v21416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:08 smithi096 ceph-mon[32753]: pgmap v21416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:08 smithi194 ceph-mon[33017]: pgmap v21416: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39787.314607 secs 2024-04-26T22:54:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39790.304442 secs 2024-04-26T22:54:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39787.314607 secs 2024-04-26T22:54:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39790.304442 secs 2024-04-26T22:54:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39787.314607 secs 2024-04-26T22:54:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39790.304442 secs 2024-04-26T22:54:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:10 smithi033 ceph-mon[27845]: pgmap v21417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:10 smithi096 ceph-mon[32753]: pgmap v21417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:10 smithi194 ceph-mon[33017]: pgmap v21417: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:12 smithi033 ceph-mon[27845]: pgmap v21418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:12 smithi096 ceph-mon[32753]: pgmap v21418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:12 smithi194 ceph-mon[33017]: pgmap v21418: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:14 smithi033 ceph-mon[27845]: pgmap v21419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39792.314691 secs 2024-04-26T22:54:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39795.304521 secs 2024-04-26T22:54:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:14 smithi096 ceph-mon[32753]: pgmap v21419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39792.314691 secs 2024-04-26T22:54:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39795.304521 secs 2024-04-26T22:54:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:14 smithi194 ceph-mon[33017]: pgmap v21419: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39792.314691 secs 2024-04-26T22:54:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39795.304521 secs 2024-04-26T22:54:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:16 smithi033 ceph-mon[27845]: pgmap v21420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:16.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:16 smithi096 ceph-mon[32753]: pgmap v21420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:16 smithi194 ceph-mon[33017]: pgmap v21420: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:17.091+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:17.091+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:17.091+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:54:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:18 smithi033 ceph-mon[27845]: pgmap v21421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:18 smithi096 ceph-mon[32753]: pgmap v21421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:18 smithi194 ceph-mon[33017]: pgmap v21421: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39797.314775 secs 2024-04-26T22:54:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39800.304586 secs 2024-04-26T22:54:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39797.314775 secs 2024-04-26T22:54:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39800.304586 secs 2024-04-26T22:54:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39797.314775 secs 2024-04-26T22:54:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39800.304586 secs 2024-04-26T22:54:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:20 smithi194 ceph-mon[33017]: pgmap v21422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:20 smithi033 ceph-mon[27845]: pgmap v21422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:20 smithi096 ceph-mon[32753]: pgmap v21422: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:22 smithi194 ceph-mon[33017]: pgmap v21423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:22 smithi033 ceph-mon[27845]: pgmap v21423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:22 smithi096 ceph-mon[32753]: pgmap v21423: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:24 smithi194 ceph-mon[33017]: pgmap v21424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39802.314861 secs 2024-04-26T22:54:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39805.304708 secs 2024-04-26T22:54:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:24 smithi033 ceph-mon[27845]: pgmap v21424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39802.314861 secs 2024-04-26T22:54:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39805.304708 secs 2024-04-26T22:54:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:24 smithi096 ceph-mon[32753]: pgmap v21424: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39802.314861 secs 2024-04-26T22:54:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39805.304708 secs 2024-04-26T22:54:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:26 smithi194 ceph-mon[33017]: pgmap v21425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:26 smithi033 ceph-mon[27845]: pgmap v21425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:26 smithi096 ceph-mon[32753]: pgmap v21425: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:28.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:28 smithi194 ceph-mon[33017]: pgmap v21426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:28 smithi033 ceph-mon[27845]: pgmap v21426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:28 smithi096 ceph-mon[32753]: pgmap v21426: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39807.314949 secs 2024-04-26T22:54:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39810.304845 secs 2024-04-26T22:54:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39807.314949 secs 2024-04-26T22:54:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39810.304845 secs 2024-04-26T22:54:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39807.314949 secs 2024-04-26T22:54:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39810.304845 secs 2024-04-26T22:54:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:30 smithi194 ceph-mon[33017]: pgmap v21427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:30 smithi033 ceph-mon[27845]: pgmap v21427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:30 smithi096 ceph-mon[32753]: pgmap v21427: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:32.098+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:32.098+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:32.098+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:54:32.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:32 smithi194 ceph-mon[33017]: pgmap v21428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:32 smithi033 ceph-mon[27845]: pgmap v21428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:32 smithi096 ceph-mon[32753]: pgmap v21428: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:34 smithi194 ceph-mon[33017]: pgmap v21429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39812.315035 secs 2024-04-26T22:54:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39815.304964 secs 2024-04-26T22:54:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:34 smithi033 ceph-mon[27845]: pgmap v21429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39812.315035 secs 2024-04-26T22:54:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39815.304964 secs 2024-04-26T22:54:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:34 smithi096 ceph-mon[32753]: pgmap v21429: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39812.315035 secs 2024-04-26T22:54:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39815.304964 secs 2024-04-26T22:54:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:36 smithi194 ceph-mon[33017]: pgmap v21430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:36 smithi033 ceph-mon[27845]: pgmap v21430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:36 smithi096 ceph-mon[32753]: pgmap v21430: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:38 smithi194 ceph-mon[33017]: pgmap v21431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:38 smithi033 ceph-mon[27845]: pgmap v21431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:38 smithi096 ceph-mon[32753]: pgmap v21431: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39817.315122 secs 2024-04-26T22:54:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:39 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:54:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39820.305034 secs 2024-04-26T22:54:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39817.315122 secs 2024-04-26T22:54:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:39 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:54:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39820.305034 secs 2024-04-26T22:54:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39817.315122 secs 2024-04-26T22:54:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:39 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:54:39.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39820.305034 secs 2024-04-26T22:54:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:40 smithi194 ceph-mon[33017]: pgmap v21432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:54:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:40 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:54:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:40 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:54:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:40 smithi033 ceph-mon[27845]: pgmap v21432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:54:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:40 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:54:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:40 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:54:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:40 smithi096 ceph-mon[32753]: pgmap v21432: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:54:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:40 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:54:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:40 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:54:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:41 smithi194 ceph-mon[33017]: pgmap v21433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:54:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:41 smithi194 ceph-mon[33017]: pgmap v21434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:41 smithi033 ceph-mon[27845]: pgmap v21433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:54:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:41 smithi033 ceph-mon[27845]: pgmap v21434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:41 smithi096 ceph-mon[32753]: pgmap v21433: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:54:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:41 smithi096 ceph-mon[32753]: pgmap v21434: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:43 smithi194 ceph-mon[33017]: pgmap v21435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:43 smithi033 ceph-mon[27845]: pgmap v21435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:43 smithi096 ceph-mon[32753]: pgmap v21435: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:44.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39822.315212 secs 2024-04-26T22:54:44.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39825.305174 secs 2024-04-26T22:54:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39822.315212 secs 2024-04-26T22:54:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39825.305174 secs 2024-04-26T22:54:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39822.315212 secs 2024-04-26T22:54:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39825.305174 secs 2024-04-26T22:54:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:45 smithi194 ceph-mon[33017]: pgmap v21436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:54:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:45 smithi033 ceph-mon[27845]: pgmap v21436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:54:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:45 smithi096 ceph-mon[32753]: pgmap v21436: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T22:54:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:47.109+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:47.109+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:54:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:54:47.109+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:54:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:47 smithi194 ceph-mon[33017]: pgmap v21437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:47 smithi033 ceph-mon[27845]: pgmap v21437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:47 smithi096 ceph-mon[32753]: pgmap v21437: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:48 smithi194 ceph-mon[33017]: pgmap v21438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:48 smithi033 ceph-mon[27845]: pgmap v21438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:48 smithi096 ceph-mon[32753]: pgmap v21438: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:54:49.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39827.315300 secs 2024-04-26T22:54:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39830.305286 secs 2024-04-26T22:54:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39827.315300 secs 2024-04-26T22:54:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39830.305286 secs 2024-04-26T22:54:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39827.315300 secs 2024-04-26T22:54:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39830.305286 secs 2024-04-26T22:54:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:50 smithi194 ceph-mon[33017]: pgmap v21439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:54:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:50 smithi033 ceph-mon[27845]: pgmap v21439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:54:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:50 smithi096 ceph-mon[32753]: pgmap v21439: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:54:53.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:53 smithi194 ceph-mon[33017]: pgmap v21440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:53.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:53 smithi033 ceph-mon[27845]: pgmap v21440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:53.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:53 smithi096 ceph-mon[32753]: pgmap v21440: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39832.315360 secs 2024-04-26T22:54:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39835.305379 secs 2024-04-26T22:54:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39832.315360 secs 2024-04-26T22:54:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39835.305379 secs 2024-04-26T22:54:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39832.315360 secs 2024-04-26T22:54:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39835.305379 secs 2024-04-26T22:54:55.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:55 smithi194 ceph-mon[33017]: pgmap v21441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:55.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:55 smithi033 ceph-mon[27845]: pgmap v21441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:55.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:55 smithi096 ceph-mon[32753]: pgmap v21441: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:57.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:57 smithi194 ceph-mon[33017]: pgmap v21442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:57.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:57 smithi033 ceph-mon[27845]: pgmap v21442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:57.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:57 smithi096 ceph-mon[32753]: pgmap v21442: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:54:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39837.315419 secs 2024-04-26T22:54:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:59 smithi194 ceph-mon[33017]: pgmap v21443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:54:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39840.305458 secs 2024-04-26T22:54:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39837.315419 secs 2024-04-26T22:54:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:59 smithi033 ceph-mon[27845]: pgmap v21443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:54:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39840.305458 secs 2024-04-26T22:54:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39837.315419 secs 2024-04-26T22:54:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:59 smithi096 ceph-mon[32753]: pgmap v21443: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:54:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:54:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39840.305458 secs 2024-04-26T22:55:01.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:01 smithi194 ceph-mon[33017]: pgmap v21444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:01 smithi033 ceph-mon[27845]: pgmap v21444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:01 smithi096 ceph-mon[32753]: pgmap v21444: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:02.122 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:02.121+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:02.122 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:02.121+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:02.122 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:02.121+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:55:03.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:03 smithi194 ceph-mon[33017]: pgmap v21445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:03 smithi033 ceph-mon[27845]: pgmap v21445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:03 smithi096 ceph-mon[32753]: pgmap v21445: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39842.315505 secs 2024-04-26T22:55:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39845.305577 secs 2024-04-26T22:55:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39842.315505 secs 2024-04-26T22:55:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39845.305577 secs 2024-04-26T22:55:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39842.315505 secs 2024-04-26T22:55:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39845.305577 secs 2024-04-26T22:55:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:05 smithi033 ceph-mon[27845]: pgmap v21446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:05 smithi096 ceph-mon[32753]: pgmap v21446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:05 smithi194 ceph-mon[33017]: pgmap v21446: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:07 smithi033 ceph-mon[27845]: pgmap v21447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:07 smithi096 ceph-mon[32753]: pgmap v21447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:07 smithi194 ceph-mon[33017]: pgmap v21447: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39847.315604 secs 2024-04-26T22:55:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:09 smithi033 ceph-mon[27845]: pgmap v21448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39850.305694 secs 2024-04-26T22:55:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39847.315604 secs 2024-04-26T22:55:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:09 smithi096 ceph-mon[32753]: pgmap v21448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39850.305694 secs 2024-04-26T22:55:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39847.315604 secs 2024-04-26T22:55:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:09 smithi194 ceph-mon[33017]: pgmap v21448: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39850.305694 secs 2024-04-26T22:55:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:11 smithi033 ceph-mon[27845]: pgmap v21449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:11 smithi096 ceph-mon[32753]: pgmap v21449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:11 smithi194 ceph-mon[33017]: pgmap v21449: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:13 smithi033 ceph-mon[27845]: pgmap v21450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:13 smithi096 ceph-mon[32753]: pgmap v21450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:13 smithi194 ceph-mon[33017]: pgmap v21450: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39852.315704 secs 2024-04-26T22:55:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39855.305827 secs 2024-04-26T22:55:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39852.315704 secs 2024-04-26T22:55:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39855.305827 secs 2024-04-26T22:55:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39852.315704 secs 2024-04-26T22:55:14.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39855.305827 secs 2024-04-26T22:55:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:15 smithi033 ceph-mon[27845]: pgmap v21451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:15 smithi096 ceph-mon[32753]: pgmap v21451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:15.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:15 smithi194 ceph-mon[33017]: pgmap v21451: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:17.125+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:17.125+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:17.125+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:55:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:17 smithi033 ceph-mon[27845]: pgmap v21452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:17 smithi096 ceph-mon[32753]: pgmap v21452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:17 smithi194 ceph-mon[33017]: pgmap v21452: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39857.315807 secs 2024-04-26T22:55:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:19 smithi033 ceph-mon[27845]: pgmap v21453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39860.305915 secs 2024-04-26T22:55:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39857.315807 secs 2024-04-26T22:55:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:19 smithi096 ceph-mon[32753]: pgmap v21453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:19.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39860.305915 secs 2024-04-26T22:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39857.315807 secs 2024-04-26T22:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:19 smithi194 ceph-mon[33017]: pgmap v21453: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39860.305915 secs 2024-04-26T22:55:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:21 smithi033 ceph-mon[27845]: pgmap v21454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:21 smithi096 ceph-mon[32753]: pgmap v21454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:21 smithi194 ceph-mon[33017]: pgmap v21454: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:23 smithi033 ceph-mon[27845]: pgmap v21455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:23 smithi096 ceph-mon[32753]: pgmap v21455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:23 smithi194 ceph-mon[33017]: pgmap v21455: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39862.315911 secs 2024-04-26T22:55:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39865.306053 secs 2024-04-26T22:55:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39862.315911 secs 2024-04-26T22:55:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39865.306053 secs 2024-04-26T22:55:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39862.315911 secs 2024-04-26T22:55:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39865.306053 secs 2024-04-26T22:55:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:25 smithi194 ceph-mon[33017]: pgmap v21456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:25.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:25 smithi033 ceph-mon[27845]: pgmap v21456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:25.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:25 smithi096 ceph-mon[32753]: pgmap v21456: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:27 smithi194 ceph-mon[33017]: pgmap v21457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:27 smithi033 ceph-mon[27845]: pgmap v21457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:27 smithi096 ceph-mon[32753]: pgmap v21457: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39867.316015 secs 2024-04-26T22:55:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:29 smithi194 ceph-mon[33017]: pgmap v21458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39870.306202 secs 2024-04-26T22:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39867.316015 secs 2024-04-26T22:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:29 smithi033 ceph-mon[27845]: pgmap v21458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39870.306202 secs 2024-04-26T22:55:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39867.316015 secs 2024-04-26T22:55:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:29 smithi096 ceph-mon[32753]: pgmap v21458: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39870.306202 secs 2024-04-26T22:55:31.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:31 smithi194 ceph-mon[33017]: pgmap v21459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:31 smithi033 ceph-mon[27845]: pgmap v21459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:31 smithi096 ceph-mon[32753]: pgmap v21459: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:32.141+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:32.141+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:32.141+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:55:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:33 smithi194 ceph-mon[33017]: pgmap v21460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:33 smithi033 ceph-mon[27845]: pgmap v21460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:33 smithi096 ceph-mon[32753]: pgmap v21460: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39872.316076 secs 2024-04-26T22:55:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39875.306323 secs 2024-04-26T22:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39872.316076 secs 2024-04-26T22:55:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39875.306323 secs 2024-04-26T22:55:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39872.316076 secs 2024-04-26T22:55:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39875.306323 secs 2024-04-26T22:55:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:35 smithi194 ceph-mon[33017]: pgmap v21461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:35 smithi033 ceph-mon[27845]: pgmap v21461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:35 smithi096 ceph-mon[32753]: pgmap v21461: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:37.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:37 smithi096 ceph-mon[32753]: pgmap v21462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:37 smithi194 ceph-mon[33017]: pgmap v21462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:37 smithi033 ceph-mon[27845]: pgmap v21462: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39877.316160 secs 2024-04-26T22:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:39 smithi194 ceph-mon[33017]: pgmap v21463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39880.306436 secs 2024-04-26T22:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39877.316160 secs 2024-04-26T22:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:39 smithi033 ceph-mon[27845]: pgmap v21463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39880.306436 secs 2024-04-26T22:55:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39877.316160 secs 2024-04-26T22:55:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:39 smithi096 ceph-mon[32753]: pgmap v21463: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39880.306436 secs 2024-04-26T22:55:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:41 smithi194 ceph-mon[33017]: pgmap v21464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:55:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:41 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:55:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:41 smithi033 ceph-mon[27845]: pgmap v21464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:55:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:41 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:55:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:41 smithi096 ceph-mon[32753]: pgmap v21464: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:55:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:41 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:55:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:42.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:42 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:42 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:42 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:43 smithi194 ceph-mon[33017]: pgmap v21465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:55:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:55:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:55:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:43 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:43 smithi033 ceph-mon[27845]: pgmap v21465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:55:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:43 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:43 smithi096 ceph-mon[32753]: pgmap v21465: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:55:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:55:43.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:43 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:55:44.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:44 smithi033 ceph-mon[27845]: pgmap v21466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T22:55:44.827 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:44 smithi033 ceph-mon[27845]: pgmap v21467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:44.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39882.316265 secs 2024-04-26T22:55:44.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39885.306555 secs 2024-04-26T22:55:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:44 smithi194 ceph-mon[33017]: pgmap v21466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T22:55:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:44 smithi194 ceph-mon[33017]: pgmap v21467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39882.316265 secs 2024-04-26T22:55:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39885.306555 secs 2024-04-26T22:55:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:44 smithi096 ceph-mon[32753]: pgmap v21466: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-26T22:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:44 smithi096 ceph-mon[32753]: pgmap v21467: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39882.316265 secs 2024-04-26T22:55:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39885.306555 secs 2024-04-26T22:55:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:46 smithi194 ceph-mon[33017]: pgmap v21468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:46 smithi033 ceph-mon[27845]: pgmap v21468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:46 smithi096 ceph-mon[32753]: pgmap v21468: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:47.155 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:47.155+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:47.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:47.155+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:55:47.156 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:55:47.155+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:55:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:47 smithi194 ceph-mon[33017]: pgmap v21469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:47 smithi033 ceph-mon[27845]: pgmap v21469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:47 smithi096 ceph-mon[32753]: pgmap v21469: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T22:55:49.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:48 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39887.316326 secs 2024-04-26T22:55:49.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:48 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39887.316326 secs 2024-04-26T22:55:49.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:48 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39887.316326 secs 2024-04-26T22:55:50.070 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39890.306691 secs 2024-04-26T22:55:50.070 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:49 smithi033 ceph-mon[27845]: pgmap v21470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:55:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39890.306691 secs 2024-04-26T22:55:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:49 smithi194 ceph-mon[33017]: pgmap v21470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:55:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39890.306691 secs 2024-04-26T22:55:50.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:49 smithi096 ceph-mon[32753]: pgmap v21470: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:55:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:51 smithi194 ceph-mon[33017]: pgmap v21471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:55:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:51 smithi033 ceph-mon[27845]: pgmap v21471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:55:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:51 smithi096 ceph-mon[32753]: pgmap v21471: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T22:55:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:53 smithi194 ceph-mon[33017]: pgmap v21472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:55:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:53 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39892.316405 secs 2024-04-26T22:55:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:53 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39895.306825 secs 2024-04-26T22:55:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:53 smithi033 ceph-mon[27845]: pgmap v21472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:55:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:53 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39892.316405 secs 2024-04-26T22:55:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:53 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39895.306825 secs 2024-04-26T22:55:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:53 smithi096 ceph-mon[32753]: pgmap v21472: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:55:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:53 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39892.316405 secs 2024-04-26T22:55:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:53 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39895.306825 secs 2024-04-26T22:55:56.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:55 smithi194 ceph-mon[33017]: pgmap v21473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:56.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:55 smithi033 ceph-mon[27845]: pgmap v21473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:56.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:55 smithi096 ceph-mon[32753]: pgmap v21473: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:55:58.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:57 smithi194 ceph-mon[33017]: pgmap v21474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:58.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:57 smithi033 ceph-mon[27845]: pgmap v21474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:58.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:57 smithi096 ceph-mon[32753]: pgmap v21474: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:55:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39897.316491 secs 2024-04-26T22:55:59.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:55:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39900.306944 secs 2024-04-26T22:55:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39897.316491 secs 2024-04-26T22:55:59.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:55:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39900.306944 secs 2024-04-26T22:55:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39897.316491 secs 2024-04-26T22:55:59.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:55:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39900.306944 secs 2024-04-26T22:56:00.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:00 smithi194 ceph-mon[33017]: pgmap v21475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:00.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:00 smithi033 ceph-mon[27845]: pgmap v21475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:00.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:00 smithi096 ceph-mon[32753]: pgmap v21475: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:02.156+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:02.156+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:56:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:02.156+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:02.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:02 smithi194 ceph-mon[33017]: pgmap v21476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:02.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:02 smithi033 ceph-mon[27845]: pgmap v21476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:02.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:02 smithi096 ceph-mon[32753]: pgmap v21476: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:04 smithi194 ceph-mon[33017]: pgmap v21477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39902.316568 secs 2024-04-26T22:56:04.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39905.307065 secs 2024-04-26T22:56:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:04 smithi033 ceph-mon[27845]: pgmap v21477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39902.316568 secs 2024-04-26T22:56:04.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39905.307065 secs 2024-04-26T22:56:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:04 smithi096 ceph-mon[32753]: pgmap v21477: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39902.316568 secs 2024-04-26T22:56:04.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39905.307065 secs 2024-04-26T22:56:06.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:06 smithi194 ceph-mon[33017]: pgmap v21478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:06.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:06 smithi033 ceph-mon[27845]: pgmap v21478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:06.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:06 smithi096 ceph-mon[32753]: pgmap v21478: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:08.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:08 smithi194 ceph-mon[33017]: pgmap v21479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:08.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:08 smithi033 ceph-mon[27845]: pgmap v21479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:08.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:08 smithi096 ceph-mon[32753]: pgmap v21479: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39907.316655 secs 2024-04-26T22:56:09.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39910.307203 secs 2024-04-26T22:56:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39907.316655 secs 2024-04-26T22:56:09.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39910.307203 secs 2024-04-26T22:56:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39907.316655 secs 2024-04-26T22:56:09.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39910.307203 secs 2024-04-26T22:56:10.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:10 smithi194 ceph-mon[33017]: pgmap v21480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:10.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:10 smithi033 ceph-mon[27845]: pgmap v21480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:10.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:10 smithi096 ceph-mon[32753]: pgmap v21480: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:12.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:12 smithi194 ceph-mon[33017]: pgmap v21481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:12.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:12 smithi033 ceph-mon[27845]: pgmap v21481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:12.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:12 smithi096 ceph-mon[32753]: pgmap v21481: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:14 smithi194 ceph-mon[33017]: pgmap v21482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39912.316739 secs 2024-04-26T22:56:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39915.307324 secs 2024-04-26T22:56:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:14 smithi033 ceph-mon[27845]: pgmap v21482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39912.316739 secs 2024-04-26T22:56:14.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39915.307324 secs 2024-04-26T22:56:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:14 smithi096 ceph-mon[32753]: pgmap v21482: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39912.316739 secs 2024-04-26T22:56:14.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39915.307324 secs 2024-04-26T22:56:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:16 smithi194 ceph-mon[33017]: pgmap v21483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:16 smithi033 ceph-mon[27845]: pgmap v21483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:16.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:16 smithi096 ceph-mon[32753]: pgmap v21483: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:17.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:17.165+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:17.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:17.165+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:56:17.165 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:17.165+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:18 smithi194 ceph-mon[33017]: pgmap v21484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:18 smithi033 ceph-mon[27845]: pgmap v21484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:18 smithi096 ceph-mon[32753]: pgmap v21484: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39917.316826 secs 2024-04-26T22:56:19.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39920.307442 secs 2024-04-26T22:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39917.316826 secs 2024-04-26T22:56:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39920.307442 secs 2024-04-26T22:56:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39917.316826 secs 2024-04-26T22:56:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39920.307442 secs 2024-04-26T22:56:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:20 smithi194 ceph-mon[33017]: pgmap v21485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:20 smithi033 ceph-mon[27845]: pgmap v21485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:20 smithi096 ceph-mon[32753]: pgmap v21485: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T22:56:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T22:56:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0022040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T22:56:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7feef0012390 tx=0x7feef0022040 comp rx=0 tx=0).stop 2024-04-26T22:56:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7fef05200640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.33:6800/516906866, fault. 2024-04-26T22:56:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 msgr2=0x7feee40b3120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T22:56:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b3ba0 0x7feee40b3120 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T22:56:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.829+0000 7feefe000640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T22:56:21.830 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.830+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T22:56:21.831 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:21.831+0000 7fef05200640 1 --2- 172.21.15.33:0/848178078 >> [v2:172.21.15.33:6800/516906866,v1:172.21.15.33:6801/516906866] conn(0x7feee40b2d40 0x7feee40b3660 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7feef0012870 tx=0x7feef006a040 comp rx=0 tx=0).ready entity=mgr.14931 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T22:56:22.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:22 smithi194 ceph-mon[33017]: pgmap v21486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:22 smithi033 ceph-mon[27845]: pgmap v21486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:22 smithi096 ceph-mon[32753]: pgmap v21486: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:24 smithi194 ceph-mon[33017]: pgmap v21487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39922.316912 secs 2024-04-26T22:56:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39925.307560 secs 2024-04-26T22:56:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:24 smithi033 ceph-mon[27845]: pgmap v21487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39922.316912 secs 2024-04-26T22:56:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39925.307560 secs 2024-04-26T22:56:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:24 smithi096 ceph-mon[32753]: pgmap v21487: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39922.316912 secs 2024-04-26T22:56:24.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39925.307560 secs 2024-04-26T22:56:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:26 smithi194 ceph-mon[33017]: pgmap v21488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:26 smithi033 ceph-mon[27845]: pgmap v21488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:26 smithi096 ceph-mon[32753]: pgmap v21488: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:28 smithi194 ceph-mon[33017]: pgmap v21489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:28 smithi033 ceph-mon[27845]: pgmap v21489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:28 smithi096 ceph-mon[32753]: pgmap v21489: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:28.595+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T22:56:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:28.595+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:56:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:28.596+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 311 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc004740 con 0x564385821900 2024-04-26T22:56:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:28.596+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 311 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:56:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39927.317014 secs 2024-04-26T22:56:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39930.307695 secs 2024-04-26T22:56:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39927.317014 secs 2024-04-26T22:56:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39930.307695 secs 2024-04-26T22:56:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39927.317014 secs 2024-04-26T22:56:29.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39930.307695 secs 2024-04-26T22:56:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:30 smithi033 ceph-mon[27845]: pgmap v21490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:30 smithi096 ceph-mon[32753]: pgmap v21490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:30.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:30 smithi194 ceph-mon[33017]: pgmap v21490: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:32.168+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:32.168+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:56:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:32.168+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:32 smithi033 ceph-mon[27845]: pgmap v21491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:32 smithi096 ceph-mon[32753]: pgmap v21491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:32.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:32 smithi194 ceph-mon[33017]: pgmap v21491: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:34 smithi033 ceph-mon[27845]: pgmap v21492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39932.317103 secs 2024-04-26T22:56:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39935.307813 secs 2024-04-26T22:56:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:34 smithi096 ceph-mon[32753]: pgmap v21492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39932.317103 secs 2024-04-26T22:56:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39935.307813 secs 2024-04-26T22:56:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:34 smithi194 ceph-mon[33017]: pgmap v21492: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39932.317103 secs 2024-04-26T22:56:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39935.307813 secs 2024-04-26T22:56:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:36 smithi033 ceph-mon[27845]: pgmap v21493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:36 smithi096 ceph-mon[32753]: pgmap v21493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:36 smithi194 ceph-mon[33017]: pgmap v21493: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:38 smithi033 ceph-mon[27845]: pgmap v21494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:38 smithi096 ceph-mon[32753]: pgmap v21494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:38 smithi194 ceph-mon[33017]: pgmap v21494: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39937.317167 secs 2024-04-26T22:56:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39940.307935 secs 2024-04-26T22:56:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39937.317167 secs 2024-04-26T22:56:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39940.307935 secs 2024-04-26T22:56:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39937.317167 secs 2024-04-26T22:56:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39940.307935 secs 2024-04-26T22:56:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:40 smithi033 ceph-mon[27845]: pgmap v21495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:40.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:40 smithi096 ceph-mon[32753]: pgmap v21495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:40 smithi194 ceph-mon[33017]: pgmap v21495: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:42 smithi033 ceph-mon[27845]: pgmap v21496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:42 smithi096 ceph-mon[32753]: pgmap v21496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:42 smithi194 ceph-mon[33017]: pgmap v21496: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:56:44.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:44 smithi194 ceph-mon[33017]: pgmap v21497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:44.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39942.317255 secs 2024-04-26T22:56:44.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:44 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:56:44.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39945.308057 secs 2024-04-26T22:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:44 smithi033 ceph-mon[27845]: pgmap v21497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39942.317255 secs 2024-04-26T22:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:44 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:56:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39945.308057 secs 2024-04-26T22:56:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:44 smithi096 ceph-mon[32753]: pgmap v21497: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39942.317255 secs 2024-04-26T22:56:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:44 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:56:44.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39945.308057 secs 2024-04-26T22:56:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:56:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:56:45.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:45 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:56:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:56:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:56:45.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:45 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:56:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:56:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:56:45.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:45 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:56:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:46 smithi033 ceph-mon[27845]: pgmap v21498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:46 smithi096 ceph-mon[32753]: pgmap v21498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:46.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:46 smithi194 ceph-mon[33017]: pgmap v21498: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:47.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:47.183+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:47.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:47.183+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:56:47.183 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:56:47.183+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:56:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:48 smithi033 ceph-mon[27845]: pgmap v21499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:48 smithi096 ceph-mon[32753]: pgmap v21499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:48 smithi194 ceph-mon[33017]: pgmap v21499: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39947.317359 secs 2024-04-26T22:56:49.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39950.308179 secs 2024-04-26T22:56:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39947.317359 secs 2024-04-26T22:56:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39950.308179 secs 2024-04-26T22:56:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39947.317359 secs 2024-04-26T22:56:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39950.308179 secs 2024-04-26T22:56:50.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:50 smithi096 ceph-mon[32753]: pgmap v21500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:50 smithi194 ceph-mon[33017]: pgmap v21500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:50 smithi033 ceph-mon[27845]: pgmap v21500: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:52 smithi033 ceph-mon[27845]: pgmap v21501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:52 smithi194 ceph-mon[33017]: pgmap v21501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:52.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:52 smithi096 ceph-mon[32753]: pgmap v21501: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:54 smithi194 ceph-mon[33017]: pgmap v21502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39952.317459 secs 2024-04-26T22:56:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39955.308245 secs 2024-04-26T22:56:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:54 smithi033 ceph-mon[27845]: pgmap v21502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39952.317459 secs 2024-04-26T22:56:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39955.308245 secs 2024-04-26T22:56:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:54 smithi096 ceph-mon[32753]: pgmap v21502: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39952.317459 secs 2024-04-26T22:56:54.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39955.308245 secs 2024-04-26T22:56:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:56 smithi194 ceph-mon[33017]: pgmap v21503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:56 smithi033 ceph-mon[27845]: pgmap v21503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:56 smithi096 ceph-mon[32753]: pgmap v21503: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T22:56:58.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:58 smithi194 ceph-mon[33017]: pgmap v21504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:58 smithi033 ceph-mon[27845]: pgmap v21504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:58 smithi096 ceph-mon[32753]: pgmap v21504: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:56:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39957.317562 secs 2024-04-26T22:56:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:56:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39960.308369 secs 2024-04-26T22:56:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39957.317562 secs 2024-04-26T22:56:59.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:56:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39960.308369 secs 2024-04-26T22:56:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39957.317562 secs 2024-04-26T22:56:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:56:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39960.308369 secs 2024-04-26T22:57:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:00 smithi194 ceph-mon[33017]: pgmap v21505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:00 smithi033 ceph-mon[27845]: pgmap v21505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:00 smithi096 ceph-mon[32753]: pgmap v21505: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:02.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:02.198+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:02.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:02.198+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:02.199 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:02.198+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:57:02.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:02 smithi194 ceph-mon[33017]: pgmap v21506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:02 smithi033 ceph-mon[27845]: pgmap v21506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:02 smithi096 ceph-mon[32753]: pgmap v21506: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:04 smithi194 ceph-mon[33017]: pgmap v21507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39962.317665 secs 2024-04-26T22:57:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39965.308493 secs 2024-04-26T22:57:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:04 smithi033 ceph-mon[27845]: pgmap v21507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39962.317665 secs 2024-04-26T22:57:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39965.308493 secs 2024-04-26T22:57:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:04 smithi096 ceph-mon[32753]: pgmap v21507: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39962.317665 secs 2024-04-26T22:57:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39965.308493 secs 2024-04-26T22:57:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:06 smithi194 ceph-mon[33017]: pgmap v21508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:06 smithi033 ceph-mon[27845]: pgmap v21508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:06 smithi096 ceph-mon[32753]: pgmap v21508: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:08.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:08 smithi194 ceph-mon[33017]: pgmap v21509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:08 smithi033 ceph-mon[27845]: pgmap v21509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:08 smithi096 ceph-mon[32753]: pgmap v21509: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39967.317760 secs 2024-04-26T22:57:09.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39970.308557 secs 2024-04-26T22:57:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39967.317760 secs 2024-04-26T22:57:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39970.308557 secs 2024-04-26T22:57:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39967.317760 secs 2024-04-26T22:57:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39970.308557 secs 2024-04-26T22:57:10.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:10 smithi194 ceph-mon[33017]: pgmap v21510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:10 smithi033 ceph-mon[27845]: pgmap v21510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:10 smithi096 ceph-mon[32753]: pgmap v21510: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:12.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:12 smithi194 ceph-mon[33017]: pgmap v21511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:12 smithi033 ceph-mon[27845]: pgmap v21511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:12 smithi096 ceph-mon[32753]: pgmap v21511: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:14 smithi194 ceph-mon[33017]: pgmap v21512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39972.317844 secs 2024-04-26T22:57:14.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39975.308656 secs 2024-04-26T22:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:14 smithi033 ceph-mon[27845]: pgmap v21512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39972.317844 secs 2024-04-26T22:57:14.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39975.308656 secs 2024-04-26T22:57:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:14 smithi096 ceph-mon[32753]: pgmap v21512: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39972.317844 secs 2024-04-26T22:57:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39975.308656 secs 2024-04-26T22:57:16.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:16 smithi194 ceph-mon[33017]: pgmap v21513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:16 smithi033 ceph-mon[27845]: pgmap v21513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:16 smithi096 ceph-mon[32753]: pgmap v21513: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:17.206+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:17.206+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:57:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:17.206+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:18.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:18 smithi194 ceph-mon[33017]: pgmap v21514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:18 smithi033 ceph-mon[27845]: pgmap v21514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:18 smithi096 ceph-mon[32753]: pgmap v21514: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39977.317948 secs 2024-04-26T22:57:19.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39980.308793 secs 2024-04-26T22:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39977.317948 secs 2024-04-26T22:57:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39980.308793 secs 2024-04-26T22:57:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39977.317948 secs 2024-04-26T22:57:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39980.308793 secs 2024-04-26T22:57:20.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:20 smithi194 ceph-mon[33017]: pgmap v21515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:20.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:20 smithi033 ceph-mon[27845]: pgmap v21515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:20 smithi096 ceph-mon[32753]: pgmap v21515: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:22.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:22 smithi194 ceph-mon[33017]: pgmap v21516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:22.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:22 smithi033 ceph-mon[27845]: pgmap v21516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:22.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:22 smithi096 ceph-mon[32753]: pgmap v21516: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:24 smithi033 ceph-mon[27845]: pgmap v21517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39982.318030 secs 2024-04-26T22:57:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39985.308914 secs 2024-04-26T22:57:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:24 smithi096 ceph-mon[32753]: pgmap v21517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39982.318030 secs 2024-04-26T22:57:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39985.308914 secs 2024-04-26T22:57:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:24 smithi194 ceph-mon[33017]: pgmap v21517: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39982.318030 secs 2024-04-26T22:57:24.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39985.308914 secs 2024-04-26T22:57:26.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:26 smithi033 ceph-mon[27845]: pgmap v21518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:26.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:26 smithi096 ceph-mon[32753]: pgmap v21518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:26 smithi194 ceph-mon[33017]: pgmap v21518: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:28.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:28 smithi033 ceph-mon[27845]: pgmap v21519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:28.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:28 smithi096 ceph-mon[32753]: pgmap v21519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:28 smithi194 ceph-mon[33017]: pgmap v21519: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39987.318096 secs 2024-04-26T22:57:29.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39990.309034 secs 2024-04-26T22:57:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39987.318096 secs 2024-04-26T22:57:29.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39990.309034 secs 2024-04-26T22:57:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39987.318096 secs 2024-04-26T22:57:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39990.309034 secs 2024-04-26T22:57:30.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:30 smithi033 ceph-mon[27845]: pgmap v21520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:30.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:30 smithi096 ceph-mon[32753]: pgmap v21520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:30 smithi194 ceph-mon[33017]: pgmap v21520: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:32.210+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:32.210+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:57:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:32.210+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:32.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:32 smithi033 ceph-mon[27845]: pgmap v21521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:32.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:32 smithi096 ceph-mon[32753]: pgmap v21521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:32 smithi194 ceph-mon[33017]: pgmap v21521: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:34 smithi033 ceph-mon[27845]: pgmap v21522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39992.318161 secs 2024-04-26T22:57:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 39995.309151 secs 2024-04-26T22:57:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:34 smithi096 ceph-mon[32753]: pgmap v21522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39992.318161 secs 2024-04-26T22:57:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 39995.309151 secs 2024-04-26T22:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:34 smithi194 ceph-mon[33017]: pgmap v21522: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39992.318161 secs 2024-04-26T22:57:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 39995.309151 secs 2024-04-26T22:57:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:36 smithi033 ceph-mon[27845]: pgmap v21523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:36 smithi096 ceph-mon[32753]: pgmap v21523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:36 smithi194 ceph-mon[33017]: pgmap v21523: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:38 smithi033 ceph-mon[27845]: pgmap v21524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:38 smithi096 ceph-mon[32753]: pgmap v21524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:38 smithi194 ceph-mon[33017]: pgmap v21524: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 39997.318249 secs 2024-04-26T22:57:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40000.309224 secs 2024-04-26T22:57:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 39997.318249 secs 2024-04-26T22:57:39.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40000.309224 secs 2024-04-26T22:57:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 39997.318249 secs 2024-04-26T22:57:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40000.309224 secs 2024-04-26T22:57:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:40 smithi033 ceph-mon[27845]: pgmap v21525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:40 smithi096 ceph-mon[32753]: pgmap v21525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:40 smithi194 ceph-mon[33017]: pgmap v21525: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:42 smithi033 ceph-mon[27845]: pgmap v21526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:42 smithi096 ceph-mon[32753]: pgmap v21526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:42 smithi194 ceph-mon[33017]: pgmap v21526: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:44 smithi096 ceph-mon[32753]: pgmap v21527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40002.318352 secs 2024-04-26T22:57:44.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40005.309337 secs 2024-04-26T22:57:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:44 smithi194 ceph-mon[33017]: pgmap v21527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40002.318352 secs 2024-04-26T22:57:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40005.309337 secs 2024-04-26T22:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:44 smithi033 ceph-mon[27845]: pgmap v21527: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40002.318352 secs 2024-04-26T22:57:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40005.309337 secs 2024-04-26T22:57:45.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:45 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:57:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:45 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:57:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:45 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:57:46.612 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:46 smithi096 ceph-mon[32753]: pgmap v21528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:46.612 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.612 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.612 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.612 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:46 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:46 smithi033 ceph-mon[27845]: pgmap v21528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:46 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:46 smithi194 ceph-mon[33017]: pgmap v21528: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:46.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:46.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:46 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:47.225+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:47.225+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:57:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:47.225+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: pgmap v21529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.856 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:48 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: pgmap v21529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:48 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: pgmap v21529: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:48.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:48 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40007.318443 secs 2024-04-26T22:57:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40010.309416 secs 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:49.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:49 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40007.318443 secs 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40010.309416 secs 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:49.927 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:49 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40007.318443 secs 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi194 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-26T22:57:49.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi096 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-26T22:57:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40010.309416 secs 2024-04-26T22:57:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:49.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:49 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:50 smithi194 ceph-mon[33017]: pgmap v21530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:50.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:50 smithi033 ceph-mon[27845]: pgmap v21530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:50 smithi096 ceph-mon[32753]: pgmap v21530: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:57:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:57:51.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:51 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:57:51.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:51 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:57:51.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:51 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:57:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:52 smithi194 ceph-mon[33017]: pgmap v21531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:52 smithi194 ceph-mon[33017]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:57:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:52 smithi194 ceph-mon[33017]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:57:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:52 smithi194 ceph-mon[33017]: pgmap v21532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:57:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:52 smithi194 ceph-mon[33017]: pgmap v21533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:52 smithi033 ceph-mon[27845]: pgmap v21531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:52 smithi033 ceph-mon[27845]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:57:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:52 smithi033 ceph-mon[27845]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:57:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:52 smithi033 ceph-mon[27845]: pgmap v21532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:57:52.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:52 smithi033 ceph-mon[27845]: pgmap v21533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:52 smithi096 ceph-mon[32753]: pgmap v21531: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:57:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:52 smithi096 ceph-mon[32753]: Adjusting osd_memory_target on smithi033 to 177.5M 2024-04-26T22:57:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:52 smithi096 ceph-mon[32753]: Unable to set osd_memory_target on smithi033 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T22:57:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:52 smithi096 ceph-mon[32753]: pgmap v21532: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:57:52.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:52 smithi096 ceph-mon[32753]: pgmap v21533: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:54 smithi194 ceph-mon[33017]: pgmap v21534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40012.318527 secs 2024-04-26T22:57:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40015.309546 secs 2024-04-26T22:57:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:54 smithi033 ceph-mon[27845]: pgmap v21534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:57:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40012.318527 secs 2024-04-26T22:57:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40015.309546 secs 2024-04-26T22:57:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:54 smithi096 ceph-mon[32753]: pgmap v21534: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:57:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40012.318527 secs 2024-04-26T22:57:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40015.309546 secs 2024-04-26T22:57:56.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:56 smithi194 ceph-mon[33017]: pgmap v21535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:56 smithi033 ceph-mon[27845]: pgmap v21535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:56 smithi096 ceph-mon[32753]: pgmap v21535: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:58.609+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:57:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:58.609+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7feecc004740 con 0x564385821900 2024-04-26T22:57:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:58.609+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 312 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T22:57:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:57:58.609+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 312 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:57:58.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:58 smithi194 ceph-mon[33017]: pgmap v21536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:58 smithi033 ceph-mon[27845]: pgmap v21536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:58 smithi096 ceph-mon[32753]: pgmap v21536: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T22:57:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40017.318622 secs 2024-04-26T22:57:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:57:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40020.309627 secs 2024-04-26T22:57:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40017.318622 secs 2024-04-26T22:57:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:57:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40020.309627 secs 2024-04-26T22:57:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40017.318622 secs 2024-04-26T22:57:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:57:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40020.309627 secs 2024-04-26T22:58:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:00 smithi194 ceph-mon[33017]: pgmap v21537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:58:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:00 smithi033 ceph-mon[27845]: pgmap v21537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:58:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:00 smithi096 ceph-mon[32753]: pgmap v21537: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T22:58:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:02.231+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:02.231+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:58:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:02.231+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:02 smithi194 ceph-mon[33017]: pgmap v21538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:58:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:02 smithi033 ceph-mon[27845]: pgmap v21538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:58:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:02 smithi096 ceph-mon[32753]: pgmap v21538: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:58:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:04 smithi194 ceph-mon[33017]: pgmap v21539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40022.318717 secs 2024-04-26T22:58:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40025.309745 secs 2024-04-26T22:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:04 smithi033 ceph-mon[27845]: pgmap v21539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40022.318717 secs 2024-04-26T22:58:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40025.309745 secs 2024-04-26T22:58:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:04 smithi096 ceph-mon[32753]: pgmap v21539: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40022.318717 secs 2024-04-26T22:58:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40025.309745 secs 2024-04-26T22:58:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:06 smithi194 ceph-mon[33017]: pgmap v21540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:06 smithi033 ceph-mon[27845]: pgmap v21540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:06 smithi096 ceph-mon[32753]: pgmap v21540: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:08 smithi194 ceph-mon[33017]: pgmap v21541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:08 smithi033 ceph-mon[27845]: pgmap v21541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:08 smithi096 ceph-mon[32753]: pgmap v21541: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40027.318814 secs 2024-04-26T22:58:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40030.309893 secs 2024-04-26T22:58:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40027.318814 secs 2024-04-26T22:58:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40030.309893 secs 2024-04-26T22:58:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40027.318814 secs 2024-04-26T22:58:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40030.309893 secs 2024-04-26T22:58:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:10 smithi194 ceph-mon[33017]: pgmap v21542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:10 smithi033 ceph-mon[27845]: pgmap v21542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:10 smithi096 ceph-mon[32753]: pgmap v21542: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:11 smithi194 ceph-mon[33017]: pgmap v21543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:11 smithi033 ceph-mon[27845]: pgmap v21543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:11 smithi096 ceph-mon[32753]: pgmap v21543: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:13 smithi033 ceph-mon[27845]: pgmap v21544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40032.318905 secs 2024-04-26T22:58:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:13 smithi096 ceph-mon[32753]: pgmap v21544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40032.318905 secs 2024-04-26T22:58:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:13 smithi194 ceph-mon[33017]: pgmap v21544: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:14.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40032.318905 secs 2024-04-26T22:58:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40035.310009 secs 2024-04-26T22:58:15.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40035.310009 secs 2024-04-26T22:58:15.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40035.310009 secs 2024-04-26T22:58:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:15 smithi096 ceph-mon[32753]: pgmap v21545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:16.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:15 smithi194 ceph-mon[33017]: pgmap v21545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:16.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:15 smithi033 ceph-mon[27845]: pgmap v21545: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:17.243+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:17.243+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:58:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:17.243+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:18.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:17 smithi194 ceph-mon[33017]: pgmap v21546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:18.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:17 smithi033 ceph-mon[27845]: pgmap v21546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:18.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:17 smithi096 ceph-mon[32753]: pgmap v21546: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:19.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:18 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40037.319003 secs 2024-04-26T22:58:19.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:18 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40037.319003 secs 2024-04-26T22:58:19.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:18 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40037.319003 secs 2024-04-26T22:58:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:19 smithi194 ceph-mon[33017]: pgmap v21547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:20.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40040.310149 secs 2024-04-26T22:58:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:19 smithi033 ceph-mon[27845]: pgmap v21547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:20.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40040.310149 secs 2024-04-26T22:58:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:19 smithi096 ceph-mon[32753]: pgmap v21547: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:20.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40040.310149 secs 2024-04-26T22:58:22.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:21 smithi194 ceph-mon[33017]: pgmap v21548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:22.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:21 smithi033 ceph-mon[27845]: pgmap v21548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:22.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:21 smithi096 ceph-mon[32753]: pgmap v21548: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:23 smithi194 ceph-mon[33017]: pgmap v21549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40042.319094 secs 2024-04-26T22:58:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:23 smithi033 ceph-mon[27845]: pgmap v21549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:24.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40042.319094 secs 2024-04-26T22:58:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:23 smithi096 ceph-mon[32753]: pgmap v21549: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:24.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40042.319094 secs 2024-04-26T22:58:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40045.310249 secs 2024-04-26T22:58:25.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40045.310249 secs 2024-04-26T22:58:25.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40045.310249 secs 2024-04-26T22:58:26.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:25 smithi194 ceph-mon[33017]: pgmap v21550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:26.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:25 smithi033 ceph-mon[27845]: pgmap v21550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:26.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:25 smithi096 ceph-mon[32753]: pgmap v21550: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:28.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:27 smithi194 ceph-mon[33017]: pgmap v21551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:28.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:27 smithi033 ceph-mon[27845]: pgmap v21551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:28.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:27 smithi096 ceph-mon[32753]: pgmap v21551: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:29.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:28 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40047.319158 secs 2024-04-26T22:58:29.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:28 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40047.319158 secs 2024-04-26T22:58:29.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:28 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40047.319158 secs 2024-04-26T22:58:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:29 smithi194 ceph-mon[33017]: pgmap v21552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:30.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40050.310342 secs 2024-04-26T22:58:30.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:29 smithi033 ceph-mon[27845]: pgmap v21552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:30.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40050.310342 secs 2024-04-26T22:58:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:29 smithi096 ceph-mon[32753]: pgmap v21552: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:30.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40050.310342 secs 2024-04-26T22:58:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:32.259+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:32.259+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:58:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:32.259+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:32.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:31 smithi194 ceph-mon[33017]: pgmap v21553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:32.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:31 smithi033 ceph-mon[27845]: pgmap v21553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:32.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:31 smithi096 ceph-mon[32753]: pgmap v21553: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:34 smithi194 ceph-mon[33017]: pgmap v21554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40052.319262 secs 2024-04-26T22:58:34.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40055.310463 secs 2024-04-26T22:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:34 smithi033 ceph-mon[27845]: pgmap v21554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40052.319262 secs 2024-04-26T22:58:34.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40055.310463 secs 2024-04-26T22:58:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:34 smithi096 ceph-mon[32753]: pgmap v21554: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40052.319262 secs 2024-04-26T22:58:34.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40055.310463 secs 2024-04-26T22:58:36.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:36 smithi194 ceph-mon[33017]: pgmap v21555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:36.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:36 smithi033 ceph-mon[27845]: pgmap v21555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:36.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:36 smithi096 ceph-mon[32753]: pgmap v21555: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:38.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:38 smithi194 ceph-mon[33017]: pgmap v21556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:38.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:38 smithi033 ceph-mon[27845]: pgmap v21556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:38.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:38 smithi096 ceph-mon[32753]: pgmap v21556: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40057.319362 secs 2024-04-26T22:58:39.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40060.310577 secs 2024-04-26T22:58:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40057.319362 secs 2024-04-26T22:58:39.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40060.310577 secs 2024-04-26T22:58:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40057.319362 secs 2024-04-26T22:58:39.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40060.310577 secs 2024-04-26T22:58:40.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:40 smithi194 ceph-mon[33017]: pgmap v21557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:40.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:40 smithi033 ceph-mon[27845]: pgmap v21557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:40.429 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:40 smithi096 ceph-mon[32753]: pgmap v21557: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:42.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:42 smithi194 ceph-mon[33017]: pgmap v21558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:42.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:42 smithi033 ceph-mon[27845]: pgmap v21558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:42.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:42 smithi096 ceph-mon[32753]: pgmap v21558: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:44 smithi194 ceph-mon[33017]: pgmap v21559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40062.319462 secs 2024-04-26T22:58:44.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40065.310694 secs 2024-04-26T22:58:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:44 smithi033 ceph-mon[27845]: pgmap v21559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40062.319462 secs 2024-04-26T22:58:44.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40065.310694 secs 2024-04-26T22:58:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:44 smithi096 ceph-mon[32753]: pgmap v21559: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40062.319462 secs 2024-04-26T22:58:44.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40065.310694 secs 2024-04-26T22:58:46.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:46 smithi194 ceph-mon[33017]: pgmap v21560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:46.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:46 smithi033 ceph-mon[27845]: pgmap v21560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:46.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:46 smithi096 ceph-mon[32753]: pgmap v21560: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:47.272+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:47.272+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:58:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:58:47.272+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:58:48.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:48 smithi194 ceph-mon[33017]: pgmap v21561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:48.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:48 smithi033 ceph-mon[27845]: pgmap v21561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:48.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:48 smithi096 ceph-mon[32753]: pgmap v21561: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40067.319566 secs 2024-04-26T22:58:49.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40070.310770 secs 2024-04-26T22:58:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40067.319566 secs 2024-04-26T22:58:49.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40070.310770 secs 2024-04-26T22:58:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40067.319566 secs 2024-04-26T22:58:49.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40070.310770 secs 2024-04-26T22:58:50.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:50 smithi096 ceph-mon[32753]: pgmap v21562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:50.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:50 smithi194 ceph-mon[33017]: pgmap v21562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:50.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:50 smithi033 ceph-mon[27845]: pgmap v21562: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:52 smithi194 ceph-mon[33017]: pgmap v21563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:52.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:52 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:58:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:52 smithi033 ceph-mon[27845]: pgmap v21563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:52.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:52 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:58:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:52 smithi096 ceph-mon[32753]: pgmap v21563: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:58:52.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:52 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:58:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:58:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:58:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:53 smithi194 ceph-mon[33017]: pgmap v21564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:53 smithi194 ceph-mon[33017]: pgmap v21565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:53.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:53 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:58:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:58:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:58:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:53 smithi033 ceph-mon[27845]: pgmap v21564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:53 smithi033 ceph-mon[27845]: pgmap v21565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:53.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:53 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:58:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T22:58:53.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T22:58:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:53 smithi096 ceph-mon[32753]: pgmap v21564: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:58:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:53 smithi096 ceph-mon[32753]: pgmap v21565: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:53.432 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:53 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:58:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40072.319669 secs 2024-04-26T22:58:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40075.310894 secs 2024-04-26T22:58:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40072.319669 secs 2024-04-26T22:58:54.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40075.310894 secs 2024-04-26T22:58:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40072.319669 secs 2024-04-26T22:58:54.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40075.310894 secs 2024-04-26T22:58:55.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:55 smithi194 ceph-mon[33017]: pgmap v21566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:58:55.425 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:55 smithi033 ceph-mon[27845]: pgmap v21566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:58:55.431 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:55 smithi096 ceph-mon[32753]: pgmap v21566: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T22:58:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:57 smithi194 ceph-mon[33017]: pgmap v21567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:57 smithi033 ceph-mon[27845]: pgmap v21567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:57 smithi096 ceph-mon[32753]: pgmap v21567: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40077.319771 secs 2024-04-26T22:58:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:59 smithi194 ceph-mon[33017]: pgmap v21568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:58:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40080.311035 secs 2024-04-26T22:58:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40077.319771 secs 2024-04-26T22:58:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:59 smithi033 ceph-mon[27845]: pgmap v21568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:58:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40080.311035 secs 2024-04-26T22:58:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40077.319771 secs 2024-04-26T22:58:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:59 smithi096 ceph-mon[32753]: pgmap v21568: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:58:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:58:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40080.311035 secs 2024-04-26T22:59:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:01 smithi194 ceph-mon[33017]: pgmap v21569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:01 smithi033 ceph-mon[27845]: pgmap v21569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:01 smithi096 ceph-mon[32753]: pgmap v21569: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:02.287+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:02.287+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:02.287+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:59:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:03 smithi194 ceph-mon[33017]: pgmap v21570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:59:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:03 smithi033 ceph-mon[27845]: pgmap v21570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:59:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:03 smithi096 ceph-mon[32753]: pgmap v21570: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T22:59:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40082.319828 secs 2024-04-26T22:59:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40085.311145 secs 2024-04-26T22:59:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:04 smithi033 ceph-mon[27845]: pgmap v21571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40082.319828 secs 2024-04-26T22:59:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40085.311145 secs 2024-04-26T22:59:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:04 smithi096 ceph-mon[32753]: pgmap v21571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40082.319828 secs 2024-04-26T22:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40085.311145 secs 2024-04-26T22:59:05.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:04 smithi194 ceph-mon[33017]: pgmap v21571: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:07 smithi194 ceph-mon[33017]: pgmap v21572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:07 smithi033 ceph-mon[27845]: pgmap v21572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:07 smithi096 ceph-mon[32753]: pgmap v21572: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40087.319929 secs 2024-04-26T22:59:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:09 smithi194 ceph-mon[33017]: pgmap v21573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40090.311264 secs 2024-04-26T22:59:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40087.319929 secs 2024-04-26T22:59:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:09 smithi033 ceph-mon[27845]: pgmap v21573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40090.311264 secs 2024-04-26T22:59:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40087.319929 secs 2024-04-26T22:59:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:09 smithi096 ceph-mon[32753]: pgmap v21573: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40090.311264 secs 2024-04-26T22:59:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:11 smithi194 ceph-mon[33017]: pgmap v21574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:11 smithi033 ceph-mon[27845]: pgmap v21574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:11 smithi096 ceph-mon[32753]: pgmap v21574: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:12 smithi033 ceph-mon[27845]: pgmap v21575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:12 smithi096 ceph-mon[32753]: pgmap v21575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:12 smithi194 ceph-mon[33017]: pgmap v21575: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:14.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:13 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40092.320003 secs 2024-04-26T22:59:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:13 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40092.320003 secs 2024-04-26T22:59:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:13 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40092.320003 secs 2024-04-26T22:59:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40095.311376 secs 2024-04-26T22:59:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:14 smithi194 ceph-mon[33017]: pgmap v21576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40095.311376 secs 2024-04-26T22:59:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:14 smithi033 ceph-mon[27845]: pgmap v21576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40095.311376 secs 2024-04-26T22:59:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:14 smithi096 ceph-mon[32753]: pgmap v21576: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:17.303+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:17.303+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:59:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:17.303+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:17.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:17 smithi194 ceph-mon[33017]: pgmap v21577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:17 smithi033 ceph-mon[27845]: pgmap v21577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:17 smithi096 ceph-mon[32753]: pgmap v21577: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40097.320088 secs 2024-04-26T22:59:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:19 smithi194 ceph-mon[33017]: pgmap v21578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40100.311494 secs 2024-04-26T22:59:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40097.320088 secs 2024-04-26T22:59:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:19 smithi033 ceph-mon[27845]: pgmap v21578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40100.311494 secs 2024-04-26T22:59:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40097.320088 secs 2024-04-26T22:59:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:19 smithi096 ceph-mon[32753]: pgmap v21578: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40100.311494 secs 2024-04-26T22:59:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:21 smithi194 ceph-mon[33017]: pgmap v21579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:21.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:21 smithi033 ceph-mon[27845]: pgmap v21579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:21.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:21 smithi096 ceph-mon[32753]: pgmap v21579: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:22 smithi033 ceph-mon[27845]: pgmap v21580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:22 smithi096 ceph-mon[32753]: pgmap v21580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:22 smithi194 ceph-mon[33017]: pgmap v21580: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:24.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40102.320186 secs 2024-04-26T22:59:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40102.320186 secs 2024-04-26T22:59:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40102.320186 secs 2024-04-26T22:59:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40105.311614 secs 2024-04-26T22:59:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:24 smithi194 ceph-mon[33017]: pgmap v21581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40105.311614 secs 2024-04-26T22:59:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:24 smithi033 ceph-mon[27845]: pgmap v21581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40105.311614 secs 2024-04-26T22:59:25.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:24 smithi096 ceph-mon[32753]: pgmap v21581: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:27 smithi194 ceph-mon[33017]: pgmap v21582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:27.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:27 smithi033 ceph-mon[27845]: pgmap v21582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:27.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:27 smithi096 ceph-mon[32753]: pgmap v21582: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:28.625+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T22:59:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:28.625+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T22:59:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:28.626+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 313 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feed00064e0 con 0x564385821900 2024-04-26T22:59:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:28.626+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 313 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0070 con 0x7fef00077780 2024-04-26T22:59:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40107.320286 secs 2024-04-26T22:59:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:29 smithi194 ceph-mon[33017]: pgmap v21583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40110.311732 secs 2024-04-26T22:59:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40107.320286 secs 2024-04-26T22:59:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:29 smithi033 ceph-mon[27845]: pgmap v21583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40110.311732 secs 2024-04-26T22:59:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40107.320286 secs 2024-04-26T22:59:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:29 smithi096 ceph-mon[32753]: pgmap v21583: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40110.311732 secs 2024-04-26T22:59:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:30 smithi033 ceph-mon[27845]: pgmap v21584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:30 smithi096 ceph-mon[32753]: pgmap v21584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:30 smithi194 ceph-mon[33017]: pgmap v21584: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:32.314+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:32.314+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:59:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:32.314+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:33 smithi194 ceph-mon[33017]: pgmap v21585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:33 smithi033 ceph-mon[27845]: pgmap v21585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:33 smithi096 ceph-mon[32753]: pgmap v21585: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40112.320347 secs 2024-04-26T22:59:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40115.311855 secs 2024-04-26T22:59:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40112.320347 secs 2024-04-26T22:59:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40115.311855 secs 2024-04-26T22:59:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40112.320347 secs 2024-04-26T22:59:35.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40115.311855 secs 2024-04-26T22:59:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:35 smithi033 ceph-mon[27845]: pgmap v21586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:35 smithi096 ceph-mon[32753]: pgmap v21586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:36.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:35 smithi194 ceph-mon[33017]: pgmap v21586: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:36 smithi033 ceph-mon[27845]: pgmap v21587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:36.934 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:36 smithi096 ceph-mon[32753]: pgmap v21587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:37.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:36 smithi194 ceph-mon[33017]: pgmap v21587: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40117.320443 secs 2024-04-26T22:59:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:39 smithi194 ceph-mon[33017]: pgmap v21588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40120.311997 secs 2024-04-26T22:59:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40117.320443 secs 2024-04-26T22:59:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:39 smithi033 ceph-mon[27845]: pgmap v21588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:39.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40120.311997 secs 2024-04-26T22:59:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40117.320443 secs 2024-04-26T22:59:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:39 smithi096 ceph-mon[32753]: pgmap v21588: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:39.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40120.311997 secs 2024-04-26T22:59:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:40 smithi033 ceph-mon[27845]: pgmap v21589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:40 smithi096 ceph-mon[32753]: pgmap v21589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:41.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:40 smithi194 ceph-mon[33017]: pgmap v21589: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:43.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:43 smithi194 ceph-mon[33017]: pgmap v21590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:43 smithi033 ceph-mon[27845]: pgmap v21590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:43 smithi096 ceph-mon[32753]: pgmap v21590: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40122.320513 secs 2024-04-26T22:59:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40125.312141 secs 2024-04-26T22:59:44.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:44 smithi033 ceph-mon[27845]: pgmap v21591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40122.320513 secs 2024-04-26T22:59:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40125.312141 secs 2024-04-26T22:59:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:44 smithi096 ceph-mon[32753]: pgmap v21591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40122.320513 secs 2024-04-26T22:59:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40125.312141 secs 2024-04-26T22:59:45.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:44 smithi194 ceph-mon[33017]: pgmap v21591: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:47.322+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:47.322+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T22:59:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T22:59:47.322+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T22:59:47.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:47 smithi194 ceph-mon[33017]: pgmap v21592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:47 smithi033 ceph-mon[27845]: pgmap v21592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:47 smithi096 ceph-mon[32753]: pgmap v21592: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:48 smithi033 ceph-mon[27845]: pgmap v21593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:48 smithi096 ceph-mon[32753]: pgmap v21593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:49.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:48 smithi194 ceph-mon[33017]: pgmap v21593: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40127.320614 secs 2024-04-26T22:59:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40130.312265 secs 2024-04-26T22:59:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40127.320614 secs 2024-04-26T22:59:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40130.312265 secs 2024-04-26T22:59:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40127.320614 secs 2024-04-26T22:59:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40130.312265 secs 2024-04-26T22:59:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:50 smithi033 ceph-mon[27845]: pgmap v21594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:50 smithi096 ceph-mon[32753]: pgmap v21594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:50 smithi194 ceph-mon[33017]: pgmap v21594: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T22:59:53.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:53 smithi194 ceph-mon[33017]: pgmap v21595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:53 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:59:53.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:53 smithi033 ceph-mon[27845]: pgmap v21595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:53 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:59:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:53 smithi096 ceph-mon[32753]: pgmap v21595: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:53 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T22:59:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40132.320707 secs 2024-04-26T22:59:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40135.312353 secs 2024-04-26T22:59:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:54 smithi194 ceph-mon[33017]: pgmap v21596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40132.320707 secs 2024-04-26T22:59:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40135.312353 secs 2024-04-26T22:59:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:54 smithi033 ceph-mon[27845]: pgmap v21596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40132.320707 secs 2024-04-26T22:59:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40135.312353 secs 2024-04-26T22:59:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:54 smithi096 ceph-mon[32753]: pgmap v21596: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T22:59:56.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:56 smithi194 ceph-mon[33017]: pgmap v21597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:56.878 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:56 smithi096 ceph-mon[32753]: pgmap v21597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:56 smithi033 ceph-mon[27845]: pgmap v21597: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-26T22:59:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:58 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:58 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:58 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T22:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40137.320810 secs 2024-04-26T22:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:59 smithi194 ceph-mon[33017]: pgmap v21598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 22:59:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40140.312469 secs 2024-04-26T22:59:59.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40137.320810 secs 2024-04-26T22:59:59.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:59 smithi033 ceph-mon[27845]: pgmap v21598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:59.639 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 22:59:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40140.312469 secs 2024-04-26T22:59:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40137.320810 secs 2024-04-26T22:59:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:59 smithi096 ceph-mon[32753]: pgmap v21598: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T22:59:59.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 22:59:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40140.312469 secs 2024-04-26T23:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:00 smithi194 ceph-mon[33017]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T23:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:00:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:00 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:00 smithi033 ceph-mon[27845]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T23:00:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:00:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:00:00.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:00 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:00 smithi096 ceph-mon[32753]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-04-26T23:00:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:00:00.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:00 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:00:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:01 smithi194 ceph-mon[33017]: pgmap v21599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T23:00:01.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:01 smithi194 ceph-mon[33017]: pgmap v21600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:01 smithi033 ceph-mon[27845]: pgmap v21599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T23:00:01.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:01 smithi033 ceph-mon[27845]: pgmap v21600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:01 smithi096 ceph-mon[32753]: pgmap v21599: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-26T23:00:01.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:01 smithi096 ceph-mon[32753]: pgmap v21600: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:02.334+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:02.334+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:02.334+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:00:03.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:03 smithi194 ceph-mon[33017]: pgmap v21601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:03 smithi033 ceph-mon[27845]: pgmap v21601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:03.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:03 smithi096 ceph-mon[32753]: pgmap v21601: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40142.320899 secs 2024-04-26T23:00:04.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40145.312597 secs 2024-04-26T23:00:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40142.320899 secs 2024-04-26T23:00:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40145.312597 secs 2024-04-26T23:00:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40142.320899 secs 2024-04-26T23:00:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40145.312597 secs 2024-04-26T23:00:05.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:05 smithi194 ceph-mon[33017]: pgmap v21602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T23:00:05.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:05 smithi033 ceph-mon[27845]: pgmap v21602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T23:00:05.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:05 smithi096 ceph-mon[32753]: pgmap v21602: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-26T23:00:07.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:07 smithi194 ceph-mon[33017]: pgmap v21603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:07 smithi033 ceph-mon[27845]: pgmap v21603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:07 smithi096 ceph-mon[32753]: pgmap v21603: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40147.321012 secs 2024-04-26T23:00:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:09 smithi033 ceph-mon[27845]: pgmap v21604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40150.312719 secs 2024-04-26T23:00:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40147.321012 secs 2024-04-26T23:00:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:09 smithi096 ceph-mon[32753]: pgmap v21604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:09.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40150.312719 secs 2024-04-26T23:00:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40147.321012 secs 2024-04-26T23:00:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:09 smithi194 ceph-mon[33017]: pgmap v21604: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:00:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40150.312719 secs 2024-04-26T23:00:11.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:11 smithi033 ceph-mon[27845]: pgmap v21605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:00:11.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:11 smithi096 ceph-mon[32753]: pgmap v21605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:00:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:11 smithi194 ceph-mon[33017]: pgmap v21605: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:00:13.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:13 smithi033 ceph-mon[27845]: pgmap v21606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:13.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:13 smithi096 ceph-mon[32753]: pgmap v21606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:13 smithi194 ceph-mon[33017]: pgmap v21606: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:14.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40152.321093 secs 2024-04-26T23:00:14.577 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40155.312837 secs 2024-04-26T23:00:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40152.321093 secs 2024-04-26T23:00:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40155.312837 secs 2024-04-26T23:00:14.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40152.321093 secs 2024-04-26T23:00:14.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40155.312837 secs 2024-04-26T23:00:15.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:15 smithi033 ceph-mon[27845]: pgmap v21607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:15.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:15 smithi096 ceph-mon[32753]: pgmap v21607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:15.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:15 smithi194 ceph-mon[33017]: pgmap v21607: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:17.348+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:17.348+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:17.348+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:00:17.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:17 smithi033 ceph-mon[27845]: pgmap v21608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:17.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:17 smithi096 ceph-mon[32753]: pgmap v21608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:17.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:17 smithi194 ceph-mon[33017]: pgmap v21608: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40157.321180 secs 2024-04-26T23:00:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:19 smithi033 ceph-mon[27845]: pgmap v21609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40160.312963 secs 2024-04-26T23:00:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40157.321180 secs 2024-04-26T23:00:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:19 smithi096 ceph-mon[32753]: pgmap v21609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40160.312963 secs 2024-04-26T23:00:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40157.321180 secs 2024-04-26T23:00:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:19 smithi194 ceph-mon[33017]: pgmap v21609: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40160.312963 secs 2024-04-26T23:00:21.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:21 smithi033 ceph-mon[27845]: pgmap v21610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:21.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:21 smithi096 ceph-mon[32753]: pgmap v21610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:21.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:21 smithi194 ceph-mon[33017]: pgmap v21610: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:23.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:23 smithi033 ceph-mon[27845]: pgmap v21611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:23.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:23 smithi096 ceph-mon[32753]: pgmap v21611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:23.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:23 smithi194 ceph-mon[33017]: pgmap v21611: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40162.321285 secs 2024-04-26T23:00:24.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40165.313098 secs 2024-04-26T23:00:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40162.321285 secs 2024-04-26T23:00:24.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40165.313098 secs 2024-04-26T23:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40162.321285 secs 2024-04-26T23:00:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40165.313098 secs 2024-04-26T23:00:25.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:25 smithi033 ceph-mon[27845]: pgmap v21612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:25.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:25 smithi096 ceph-mon[32753]: pgmap v21612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:25.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:25 smithi194 ceph-mon[33017]: pgmap v21612: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:27.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:27 smithi033 ceph-mon[27845]: pgmap v21613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:27.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:27 smithi096 ceph-mon[32753]: pgmap v21613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:27.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:27 smithi194 ceph-mon[33017]: pgmap v21613: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40167.321367 secs 2024-04-26T23:00:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:29 smithi096 ceph-mon[32753]: pgmap v21614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:29.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40170.313215 secs 2024-04-26T23:00:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40167.321367 secs 2024-04-26T23:00:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:29 smithi194 ceph-mon[33017]: pgmap v21614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40170.313215 secs 2024-04-26T23:00:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40167.321367 secs 2024-04-26T23:00:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:29 smithi033 ceph-mon[27845]: pgmap v21614: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40170.313215 secs 2024-04-26T23:00:31.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:31 smithi096 ceph-mon[32753]: pgmap v21615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:31 smithi194 ceph-mon[33017]: pgmap v21615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:31 smithi033 ceph-mon[27845]: pgmap v21615: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:32.364+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:32.364+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:32.365+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:00:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:33 smithi194 ceph-mon[33017]: pgmap v21616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:33 smithi033 ceph-mon[27845]: pgmap v21616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:33 smithi096 ceph-mon[32753]: pgmap v21616: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40172.321450 secs 2024-04-26T23:00:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40175.313284 secs 2024-04-26T23:00:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40172.321450 secs 2024-04-26T23:00:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40175.313284 secs 2024-04-26T23:00:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40172.321450 secs 2024-04-26T23:00:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40175.313284 secs 2024-04-26T23:00:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:35 smithi194 ceph-mon[33017]: pgmap v21617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:35 smithi033 ceph-mon[27845]: pgmap v21617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:35 smithi096 ceph-mon[32753]: pgmap v21617: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:37 smithi194 ceph-mon[33017]: pgmap v21618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:37 smithi033 ceph-mon[27845]: pgmap v21618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:37 smithi096 ceph-mon[32753]: pgmap v21618: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:39.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40177.321548 secs 2024-04-26T23:00:39.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:39 smithi096 ceph-mon[32753]: pgmap v21619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:39.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40180.313403 secs 2024-04-26T23:00:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40177.321548 secs 2024-04-26T23:00:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:39 smithi194 ceph-mon[33017]: pgmap v21619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40180.313403 secs 2024-04-26T23:00:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40177.321548 secs 2024-04-26T23:00:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:39 smithi033 ceph-mon[27845]: pgmap v21619: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40180.313403 secs 2024-04-26T23:00:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:41 smithi194 ceph-mon[33017]: pgmap v21620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:41 smithi033 ceph-mon[27845]: pgmap v21620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:41 smithi096 ceph-mon[32753]: pgmap v21620: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:43 smithi194 ceph-mon[33017]: pgmap v21621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:43 smithi033 ceph-mon[27845]: pgmap v21621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:43 smithi096 ceph-mon[32753]: pgmap v21621: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40182.321629 secs 2024-04-26T23:00:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40185.313533 secs 2024-04-26T23:00:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40182.321629 secs 2024-04-26T23:00:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40185.313533 secs 2024-04-26T23:00:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40182.321629 secs 2024-04-26T23:00:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40185.313533 secs 2024-04-26T23:00:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:45 smithi194 ceph-mon[33017]: pgmap v21622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:45 smithi033 ceph-mon[27845]: pgmap v21622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:45 smithi096 ceph-mon[32753]: pgmap v21622: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:47.368+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:47.368+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:00:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:00:47.368+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:00:47.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:47 smithi033 ceph-mon[27845]: pgmap v21623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:47 smithi194 ceph-mon[33017]: pgmap v21623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:47 smithi096 ceph-mon[32753]: pgmap v21623: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40187.321713 secs 2024-04-26T23:00:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:49 smithi194 ceph-mon[33017]: pgmap v21624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40190.313648 secs 2024-04-26T23:00:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40187.321713 secs 2024-04-26T23:00:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:49 smithi033 ceph-mon[27845]: pgmap v21624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40190.313648 secs 2024-04-26T23:00:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40187.321713 secs 2024-04-26T23:00:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:49 smithi096 ceph-mon[32753]: pgmap v21624: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40190.313648 secs 2024-04-26T23:00:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:51 smithi194 ceph-mon[33017]: pgmap v21625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:51 smithi033 ceph-mon[27845]: pgmap v21625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:51 smithi096 ceph-mon[32753]: pgmap v21625: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:53 smithi194 ceph-mon[33017]: pgmap v21626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:53 smithi033 ceph-mon[27845]: pgmap v21626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:53 smithi096 ceph-mon[32753]: pgmap v21626: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40192.321802 secs 2024-04-26T23:00:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40195.313766 secs 2024-04-26T23:00:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40192.321802 secs 2024-04-26T23:00:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40195.313766 secs 2024-04-26T23:00:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40192.321802 secs 2024-04-26T23:00:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40195.313766 secs 2024-04-26T23:00:55.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:55 smithi194 ceph-mon[33017]: pgmap v21627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:55 smithi033 ceph-mon[27845]: pgmap v21627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:55 smithi096 ceph-mon[32753]: pgmap v21627: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:57.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:57 smithi194 ceph-mon[33017]: pgmap v21628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:57 smithi033 ceph-mon[27845]: pgmap v21628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:57 smithi096 ceph-mon[32753]: pgmap v21628: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:00:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40197.321891 secs 2024-04-26T23:00:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:59 smithi194 ceph-mon[33017]: pgmap v21629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:00:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40200.313898 secs 2024-04-26T23:00:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40197.321891 secs 2024-04-26T23:00:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:59 smithi033 ceph-mon[27845]: pgmap v21629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:00:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40200.313898 secs 2024-04-26T23:00:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40197.321891 secs 2024-04-26T23:00:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:59 smithi096 ceph-mon[32753]: pgmap v21629: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:00:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:00:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40200.313898 secs 2024-04-26T23:01:00.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:00 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:01:00.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:00 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:01:00.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:00 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:01:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:01 smithi194 ceph-mon[33017]: pgmap v21630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:01 smithi033 ceph-mon[27845]: pgmap v21630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:01.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:01 smithi096 ceph-mon[32753]: pgmap v21630: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:02.372+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:02.373+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:02.373+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:01:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:01:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:01:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:02 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:01:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:01:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:01:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:02 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:01:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:01:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:01:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:02 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:01:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:03 smithi194 ceph-mon[33017]: pgmap v21631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:03 smithi194 ceph-mon[33017]: pgmap v21632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:03 smithi033 ceph-mon[27845]: pgmap v21631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:03.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:03 smithi033 ceph-mon[27845]: pgmap v21632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:03 smithi096 ceph-mon[32753]: pgmap v21631: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:03 smithi096 ceph-mon[32753]: pgmap v21632: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40202.321977 secs 2024-04-26T23:01:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40205.314023 secs 2024-04-26T23:01:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40202.321977 secs 2024-04-26T23:01:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40205.314023 secs 2024-04-26T23:01:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40202.321977 secs 2024-04-26T23:01:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40205.314023 secs 2024-04-26T23:01:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:05 smithi194 ceph-mon[33017]: pgmap v21633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:05 smithi033 ceph-mon[27845]: pgmap v21633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:05 smithi096 ceph-mon[32753]: pgmap v21633: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:07 smithi194 ceph-mon[33017]: pgmap v21634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:07 smithi033 ceph-mon[27845]: pgmap v21634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:07 smithi096 ceph-mon[32753]: pgmap v21634: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:09 smithi194 ceph-mon[33017]: pgmap v21635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40207.322066 secs 2024-04-26T23:01:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40210.314146 secs 2024-04-26T23:01:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:09 smithi033 ceph-mon[27845]: pgmap v21635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40207.322066 secs 2024-04-26T23:01:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40210.314146 secs 2024-04-26T23:01:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:09 smithi096 ceph-mon[32753]: pgmap v21635: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40207.322066 secs 2024-04-26T23:01:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40210.314146 secs 2024-04-26T23:01:11.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:11 smithi194 ceph-mon[33017]: pgmap v21636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:11.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:11 smithi033 ceph-mon[27845]: pgmap v21636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:11.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:11 smithi096 ceph-mon[32753]: pgmap v21636: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:01:13.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:13 smithi194 ceph-mon[33017]: pgmap v21637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:01:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:13 smithi033 ceph-mon[27845]: pgmap v21637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:01:13.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:13 smithi096 ceph-mon[32753]: pgmap v21637: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:01:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40212.322148 secs 2024-04-26T23:01:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40215.314244 secs 2024-04-26T23:01:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40212.322148 secs 2024-04-26T23:01:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40215.314244 secs 2024-04-26T23:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40212.322148 secs 2024-04-26T23:01:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40215.314244 secs 2024-04-26T23:01:15.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:15 smithi033 ceph-mon[27845]: pgmap v21638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:15.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:15 smithi096 ceph-mon[32753]: pgmap v21638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:15 smithi194 ceph-mon[33017]: pgmap v21638: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:17.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:17.387+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:17.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:17.387+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:17.387 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:17.387+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:01:17.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:17 smithi033 ceph-mon[27845]: pgmap v21639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:17.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:17 smithi096 ceph-mon[32753]: pgmap v21639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:18.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:17 smithi194 ceph-mon[33017]: pgmap v21639: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:18 smithi033 ceph-mon[27845]: pgmap v21640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:18 smithi096 ceph-mon[32753]: pgmap v21640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:18 smithi194 ceph-mon[33017]: pgmap v21640: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40217.322207 secs 2024-04-26T23:01:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40220.314309 secs 2024-04-26T23:01:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40217.322207 secs 2024-04-26T23:01:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40220.314309 secs 2024-04-26T23:01:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40217.322207 secs 2024-04-26T23:01:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40220.314309 secs 2024-04-26T23:01:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:20 smithi096 ceph-mon[32753]: pgmap v21641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:20 smithi033 ceph-mon[27845]: pgmap v21641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:20 smithi194 ceph-mon[33017]: pgmap v21641: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:23.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:22 smithi033 ceph-mon[27845]: pgmap v21642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:23.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:22 smithi096 ceph-mon[32753]: pgmap v21642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:23.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:22 smithi194 ceph-mon[33017]: pgmap v21642: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:24.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:23 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40222.322295 secs 2024-04-26T23:01:24.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:23 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40222.322295 secs 2024-04-26T23:01:24.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:23 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40222.322295 secs 2024-04-26T23:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:24 smithi033 ceph-mon[27845]: pgmap v21643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:25.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40225.314392 secs 2024-04-26T23:01:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:24 smithi096 ceph-mon[32753]: pgmap v21643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:25.182 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40225.314392 secs 2024-04-26T23:01:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:24 smithi194 ceph-mon[33017]: pgmap v21643: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:25.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40225.314392 secs 2024-04-26T23:01:27.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:26 smithi033 ceph-mon[27845]: pgmap v21644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:27.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:26 smithi096 ceph-mon[32753]: pgmap v21644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:27.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:26 smithi194 ceph-mon[33017]: pgmap v21644: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:29 smithi194 ceph-mon[33017]: pgmap v21645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40227.322354 secs 2024-04-26T23:01:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40230.314510 secs 2024-04-26T23:01:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:29 smithi033 ceph-mon[27845]: pgmap v21645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40227.322354 secs 2024-04-26T23:01:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40230.314510 secs 2024-04-26T23:01:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:29 smithi096 ceph-mon[32753]: pgmap v21645: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40227.322354 secs 2024-04-26T23:01:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40230.314510 secs 2024-04-26T23:01:31.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:31 smithi194 ceph-mon[33017]: pgmap v21646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:31.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:31 smithi033 ceph-mon[27845]: pgmap v21646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:31.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:31 smithi096 ceph-mon[32753]: pgmap v21646: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:32.390+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:32.390+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:32.390+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:01:33.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:33 smithi194 ceph-mon[33017]: pgmap v21647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:33.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:33 smithi033 ceph-mon[27845]: pgmap v21647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:33.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:33 smithi096 ceph-mon[32753]: pgmap v21647: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40232.322440 secs 2024-04-26T23:01:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40235.314600 secs 2024-04-26T23:01:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40232.322440 secs 2024-04-26T23:01:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40235.314600 secs 2024-04-26T23:01:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40232.322440 secs 2024-04-26T23:01:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40235.314600 secs 2024-04-26T23:01:35.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:35 smithi194 ceph-mon[33017]: pgmap v21648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:35.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:35 smithi033 ceph-mon[27845]: pgmap v21648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:35.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:35 smithi096 ceph-mon[32753]: pgmap v21648: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:37.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:37 smithi194 ceph-mon[33017]: pgmap v21649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:37.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:37 smithi033 ceph-mon[27845]: pgmap v21649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:37.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:37 smithi096 ceph-mon[32753]: pgmap v21649: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:39.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:39 smithi096 ceph-mon[32753]: pgmap v21650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:39.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40237.322528 secs 2024-04-26T23:01:39.734 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40240.314726 secs 2024-04-26T23:01:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:39 smithi194 ceph-mon[33017]: pgmap v21650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40237.322528 secs 2024-04-26T23:01:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40240.314726 secs 2024-04-26T23:01:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:39 smithi033 ceph-mon[27845]: pgmap v21650: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40237.322528 secs 2024-04-26T23:01:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40240.314726 secs 2024-04-26T23:01:41.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:41 smithi194 ceph-mon[33017]: pgmap v21651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:41.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:41 smithi033 ceph-mon[27845]: pgmap v21651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:41.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:41 smithi096 ceph-mon[32753]: pgmap v21651: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:43.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:43 smithi194 ceph-mon[33017]: pgmap v21652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:43 smithi033 ceph-mon[27845]: pgmap v21652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:43.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:43 smithi096 ceph-mon[32753]: pgmap v21652: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:44.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40242.322616 secs 2024-04-26T23:01:44.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40245.314811 secs 2024-04-26T23:01:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40242.322616 secs 2024-04-26T23:01:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40245.314811 secs 2024-04-26T23:01:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40242.322616 secs 2024-04-26T23:01:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40245.314811 secs 2024-04-26T23:01:45.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:45 smithi194 ceph-mon[33017]: pgmap v21653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:45.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:45 smithi033 ceph-mon[27845]: pgmap v21653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:45.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:45 smithi096 ceph-mon[32753]: pgmap v21653: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:47.404 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:47.404+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:47.405+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:01:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:01:47.405+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:01:47.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:47 smithi194 ceph-mon[33017]: pgmap v21654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:47.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:47 smithi033 ceph-mon[27845]: pgmap v21654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:47.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:47 smithi096 ceph-mon[32753]: pgmap v21654: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:49 smithi194 ceph-mon[33017]: pgmap v21655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40247.322701 secs 2024-04-26T23:01:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40250.314960 secs 2024-04-26T23:01:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:49 smithi033 ceph-mon[27845]: pgmap v21655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40247.322701 secs 2024-04-26T23:01:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40250.314960 secs 2024-04-26T23:01:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:49 smithi096 ceph-mon[32753]: pgmap v21655: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40247.322701 secs 2024-04-26T23:01:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40250.314960 secs 2024-04-26T23:01:51.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:51 smithi194 ceph-mon[33017]: pgmap v21656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:51.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:51 smithi033 ceph-mon[27845]: pgmap v21656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:51.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:51 smithi096 ceph-mon[32753]: pgmap v21656: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:53.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:53 smithi194 ceph-mon[33017]: pgmap v21657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:53.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:53 smithi033 ceph-mon[27845]: pgmap v21657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:53.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:53 smithi096 ceph-mon[32753]: pgmap v21657: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40252.322790 secs 2024-04-26T23:01:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40255.315101 secs 2024-04-26T23:01:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40252.322790 secs 2024-04-26T23:01:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40255.315101 secs 2024-04-26T23:01:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40252.322790 secs 2024-04-26T23:01:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40255.315101 secs 2024-04-26T23:01:55.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:55 smithi194 ceph-mon[33017]: pgmap v21658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:55.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:55 smithi033 ceph-mon[27845]: pgmap v21658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:55.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:55 smithi096 ceph-mon[32753]: pgmap v21658: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:57.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:57 smithi194 ceph-mon[33017]: pgmap v21659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:57.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:57 smithi033 ceph-mon[27845]: pgmap v21659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:57.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:57 smithi096 ceph-mon[32753]: pgmap v21659: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:59 smithi194 ceph-mon[33017]: pgmap v21660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40257.322872 secs 2024-04-26T23:01:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:01:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40260.315124 secs 2024-04-26T23:01:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:59 smithi033 ceph-mon[27845]: pgmap v21660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40257.322872 secs 2024-04-26T23:01:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:01:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40260.315124 secs 2024-04-26T23:01:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:59 smithi096 ceph-mon[32753]: pgmap v21660: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:01:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40257.322872 secs 2024-04-26T23:01:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:01:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40260.315124 secs 2024-04-26T23:02:01.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:01 smithi194 ceph-mon[33017]: pgmap v21661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:01.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:01 smithi033 ceph-mon[27845]: pgmap v21661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:01.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:01 smithi096 ceph-mon[32753]: pgmap v21661: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:02.419+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:02.419+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:02.419+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:02:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:02 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:02:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:02 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:02:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:02 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:02:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:03 smithi033 ceph-mon[27845]: pgmap v21662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:02:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:03 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:02:03.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:03 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:03 smithi194 ceph-mon[33017]: pgmap v21662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:03 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:02:03.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:03 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:02:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:03 smithi096 ceph-mon[32753]: pgmap v21662: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:03.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:03 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:02:03.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:03 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:02:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40262.322960 secs 2024-04-26T23:02:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:04 smithi194 ceph-mon[33017]: pgmap v21663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:04 smithi194 ceph-mon[33017]: pgmap v21664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:02:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40265.315231 secs 2024-04-26T23:02:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40262.322960 secs 2024-04-26T23:02:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:04 smithi033 ceph-mon[27845]: pgmap v21663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:04 smithi033 ceph-mon[27845]: pgmap v21664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:02:04.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40265.315231 secs 2024-04-26T23:02:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40262.322960 secs 2024-04-26T23:02:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:04 smithi096 ceph-mon[32753]: pgmap v21663: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:04 smithi096 ceph-mon[32753]: pgmap v21664: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:02:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40265.315231 secs 2024-04-26T23:02:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:06 smithi194 ceph-mon[33017]: pgmap v21665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:06 smithi033 ceph-mon[27845]: pgmap v21665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:06 smithi096 ceph-mon[32753]: pgmap v21665: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:08 smithi194 ceph-mon[33017]: pgmap v21666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:08 smithi033 ceph-mon[27845]: pgmap v21666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:08 smithi096 ceph-mon[32753]: pgmap v21666: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40267.323045 secs 2024-04-26T23:02:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40270.315349 secs 2024-04-26T23:02:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40267.323045 secs 2024-04-26T23:02:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40270.315349 secs 2024-04-26T23:02:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40267.323045 secs 2024-04-26T23:02:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40270.315349 secs 2024-04-26T23:02:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:10 smithi194 ceph-mon[33017]: pgmap v21667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:10 smithi033 ceph-mon[27845]: pgmap v21667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:10 smithi096 ceph-mon[32753]: pgmap v21667: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:12 smithi194 ceph-mon[33017]: pgmap v21668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:12 smithi033 ceph-mon[27845]: pgmap v21668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:12 smithi096 ceph-mon[32753]: pgmap v21668: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:02:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40272.323134 secs 2024-04-26T23:02:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:14 smithi194 ceph-mon[33017]: pgmap v21669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T23:02:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40275.315465 secs 2024-04-26T23:02:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40272.323134 secs 2024-04-26T23:02:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:14 smithi033 ceph-mon[27845]: pgmap v21669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T23:02:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40275.315465 secs 2024-04-26T23:02:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40272.323134 secs 2024-04-26T23:02:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:14 smithi096 ceph-mon[32753]: pgmap v21669: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-26T23:02:14.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40275.315465 secs 2024-04-26T23:02:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:16 smithi033 ceph-mon[27845]: pgmap v21670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:02:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:16 smithi096 ceph-mon[32753]: pgmap v21670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:02:17.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:16 smithi194 ceph-mon[33017]: pgmap v21670: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:02:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:17.426+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:17.426+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:17.426+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:02:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:18 smithi033 ceph-mon[27845]: pgmap v21671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:18 smithi096 ceph-mon[32753]: pgmap v21671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:19.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:18 smithi194 ceph-mon[33017]: pgmap v21671: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40277.323196 secs 2024-04-26T23:02:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40280.315581 secs 2024-04-26T23:02:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40277.323196 secs 2024-04-26T23:02:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40280.315581 secs 2024-04-26T23:02:20.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40277.323196 secs 2024-04-26T23:02:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40280.315581 secs 2024-04-26T23:02:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:20 smithi033 ceph-mon[27845]: pgmap v21672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:20 smithi096 ceph-mon[32753]: pgmap v21672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:21.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:20 smithi194 ceph-mon[33017]: pgmap v21672: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:22 smithi033 ceph-mon[27845]: pgmap v21673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:22 smithi096 ceph-mon[32753]: pgmap v21673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:22 smithi194 ceph-mon[33017]: pgmap v21673: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40282.323295 secs 2024-04-26T23:02:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:24 smithi033 ceph-mon[27845]: pgmap v21674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40285.315697 secs 2024-04-26T23:02:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40282.323295 secs 2024-04-26T23:02:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:24 smithi096 ceph-mon[32753]: pgmap v21674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40285.315697 secs 2024-04-26T23:02:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40282.323295 secs 2024-04-26T23:02:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:24 smithi194 ceph-mon[33017]: pgmap v21674: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40285.315697 secs 2024-04-26T23:02:26.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:25 smithi194 ceph-mon[33017]: pgmap v21675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:26.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:25 smithi033 ceph-mon[27845]: pgmap v21675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:26.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:25 smithi096 ceph-mon[32753]: pgmap v21675: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:28.656+0000 7feeed600640 1 -- 172.21.15.33:0/3958398144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feecc003690 con 0x564385821900 2024-04-26T23:02:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:28.656+0000 7feefcc00640 1 -- 172.21.15.33:0/848178078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7feed00064e0 con 0x7fef00077780 2024-04-26T23:02:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:28.656+0000 7feeeea00640 1 -- 172.21.15.33:0/3958398144 <== mon.2 v2:172.21.15.96:3300/0 314 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7feecc003690 con 0x564385821900 2024-04-26T23:02:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:28.657+0000 7feefe000640 1 -- 172.21.15.33:0/848178078 <== mon.1 v2:172.21.15.194:3300/0 314 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7feee80d0050 con 0x7fef00077780 2024-04-26T23:02:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:28 smithi194 ceph-mon[33017]: pgmap v21676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:28 smithi033 ceph-mon[27845]: pgmap v21676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:28 smithi096 ceph-mon[32753]: pgmap v21676: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40287.323379 secs 2024-04-26T23:02:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40290.315834 secs 2024-04-26T23:02:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40287.323379 secs 2024-04-26T23:02:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40290.315834 secs 2024-04-26T23:02:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40287.323379 secs 2024-04-26T23:02:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40290.315834 secs 2024-04-26T23:02:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:30 smithi194 ceph-mon[33017]: pgmap v21677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:30 smithi033 ceph-mon[27845]: pgmap v21677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:30 smithi096 ceph-mon[32753]: pgmap v21677: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:32.428+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:32.428+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:32.428+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:02:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:32 smithi194 ceph-mon[33017]: pgmap v21678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:32 smithi033 ceph-mon[27845]: pgmap v21678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:32 smithi096 ceph-mon[32753]: pgmap v21678: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40292.323466 secs 2024-04-26T23:02:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:34 smithi194 ceph-mon[33017]: pgmap v21679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40295.315958 secs 2024-04-26T23:02:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40292.323466 secs 2024-04-26T23:02:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:34 smithi033 ceph-mon[27845]: pgmap v21679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40295.315958 secs 2024-04-26T23:02:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40292.323466 secs 2024-04-26T23:02:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:34 smithi096 ceph-mon[32753]: pgmap v21679: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40295.315958 secs 2024-04-26T23:02:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:36 smithi194 ceph-mon[33017]: pgmap v21680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:36 smithi033 ceph-mon[27845]: pgmap v21680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:36.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:36 smithi096 ceph-mon[32753]: pgmap v21680: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:38 smithi194 ceph-mon[33017]: pgmap v21681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:38 smithi033 ceph-mon[27845]: pgmap v21681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:38 smithi096 ceph-mon[32753]: pgmap v21681: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:39.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40297.323553 secs 2024-04-26T23:02:39.832 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40300.316076 secs 2024-04-26T23:02:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40297.323553 secs 2024-04-26T23:02:39.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40300.316076 secs 2024-04-26T23:02:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40297.323553 secs 2024-04-26T23:02:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40300.316076 secs 2024-04-26T23:02:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:40 smithi194 ceph-mon[33017]: pgmap v21682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:40 smithi033 ceph-mon[27845]: pgmap v21682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:40.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:40 smithi096 ceph-mon[32753]: pgmap v21682: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:42.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:42 smithi194 ceph-mon[33017]: pgmap v21683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:42 smithi033 ceph-mon[27845]: pgmap v21683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:42 smithi096 ceph-mon[32753]: pgmap v21683: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40302.323640 secs 2024-04-26T23:02:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:44 smithi194 ceph-mon[33017]: pgmap v21684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40305.316196 secs 2024-04-26T23:02:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40302.323640 secs 2024-04-26T23:02:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:44 smithi033 ceph-mon[27845]: pgmap v21684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40305.316196 secs 2024-04-26T23:02:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40302.323640 secs 2024-04-26T23:02:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:44 smithi096 ceph-mon[32753]: pgmap v21684: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40305.316196 secs 2024-04-26T23:02:46.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:46 smithi194 ceph-mon[33017]: pgmap v21685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:46 smithi033 ceph-mon[27845]: pgmap v21685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:46 smithi096 ceph-mon[32753]: pgmap v21685: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:47.430+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:47.430+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:02:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:02:47.430+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:02:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:48 smithi194 ceph-mon[33017]: pgmap v21686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:48 smithi033 ceph-mon[27845]: pgmap v21686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:48 smithi096 ceph-mon[32753]: pgmap v21686: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:49.903 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40307.323727 secs 2024-04-26T23:02:49.904 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40310.316316 secs 2024-04-26T23:02:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40307.323727 secs 2024-04-26T23:02:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40310.316316 secs 2024-04-26T23:02:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40307.323727 secs 2024-04-26T23:02:50.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40310.316316 secs 2024-04-26T23:02:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:50 smithi033 ceph-mon[27845]: pgmap v21687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:50 smithi096 ceph-mon[32753]: pgmap v21687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:51.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:50 smithi194 ceph-mon[33017]: pgmap v21687: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:52 smithi033 ceph-mon[27845]: pgmap v21688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:52 smithi096 ceph-mon[32753]: pgmap v21688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:53.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:52 smithi194 ceph-mon[33017]: pgmap v21688: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40312.323815 secs 2024-04-26T23:02:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:54 smithi033 ceph-mon[27845]: pgmap v21689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:54.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40315.316442 secs 2024-04-26T23:02:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40312.323815 secs 2024-04-26T23:02:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:54 smithi096 ceph-mon[32753]: pgmap v21689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40315.316442 secs 2024-04-26T23:02:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40312.323815 secs 2024-04-26T23:02:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:54 smithi194 ceph-mon[33017]: pgmap v21689: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:55.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40315.316442 secs 2024-04-26T23:02:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:56 smithi033 ceph-mon[27845]: pgmap v21690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:56 smithi096 ceph-mon[32753]: pgmap v21690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:57.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:56 smithi194 ceph-mon[33017]: pgmap v21690: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:02:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:58 smithi033 ceph-mon[27845]: pgmap v21691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:58 smithi096 ceph-mon[32753]: pgmap v21691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:59.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:58 smithi194 ceph-mon[33017]: pgmap v21691: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:02:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40317.323901 secs 2024-04-26T23:02:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:02:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40320.316568 secs 2024-04-26T23:02:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40317.323901 secs 2024-04-26T23:02:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:02:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40320.316568 secs 2024-04-26T23:03:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40317.323901 secs 2024-04-26T23:03:00.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:02:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40320.316568 secs 2024-04-26T23:03:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:00 smithi033 ceph-mon[27845]: pgmap v21692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:03:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:00 smithi096 ceph-mon[32753]: pgmap v21692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:03:01.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:00 smithi194 ceph-mon[33017]: pgmap v21692: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:03:02.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:01 smithi194 ceph-mon[33017]: pgmap v21693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T23:03:02.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:01 smithi033 ceph-mon[27845]: pgmap v21693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T23:03:02.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:01 smithi096 ceph-mon[32753]: pgmap v21693: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T23:03:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:02.444+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:02.444+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:02.444+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:03:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40322.323987 secs 2024-04-26T23:03:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:04 smithi194 ceph-mon[33017]: pgmap v21694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T23:03:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:04 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:03:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40325.316636 secs 2024-04-26T23:03:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40322.323987 secs 2024-04-26T23:03:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:04 smithi033 ceph-mon[27845]: pgmap v21694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T23:03:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:04 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:03:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40325.316636 secs 2024-04-26T23:03:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40322.323987 secs 2024-04-26T23:03:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:04 smithi096 ceph-mon[32753]: pgmap v21694: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-26T23:03:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:04 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:03:04.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40325.316636 secs 2024-04-26T23:03:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:03:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:05 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:03:05.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:05 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:03:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:03:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:05 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:03:05.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:05 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:03:05.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:03:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:05 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:03:05.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:05 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:03:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:06 smithi194 ceph-mon[33017]: pgmap v21695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:03:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:06 smithi194 ceph-mon[33017]: pgmap v21696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:03:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:06 smithi033 ceph-mon[27845]: pgmap v21695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:03:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:06 smithi033 ceph-mon[27845]: pgmap v21696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:03:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:06 smithi096 ceph-mon[32753]: pgmap v21695: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:03:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:06 smithi096 ceph-mon[32753]: pgmap v21696: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:03:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:08 smithi194 ceph-mon[33017]: pgmap v21697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:03:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:08 smithi033 ceph-mon[27845]: pgmap v21697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:03:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:08 smithi096 ceph-mon[32753]: pgmap v21697: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-26T23:03:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40327.324070 secs 2024-04-26T23:03:09.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40330.316758 secs 2024-04-26T23:03:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40327.324070 secs 2024-04-26T23:03:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40330.316758 secs 2024-04-26T23:03:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40327.324070 secs 2024-04-26T23:03:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40330.316758 secs 2024-04-26T23:03:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:10 smithi194 ceph-mon[33017]: pgmap v21698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T23:03:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:10 smithi033 ceph-mon[27845]: pgmap v21698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T23:03:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:10 smithi096 ceph-mon[32753]: pgmap v21698: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-26T23:03:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:12 smithi194 ceph-mon[33017]: pgmap v21699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:03:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:12 smithi033 ceph-mon[27845]: pgmap v21699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:03:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:12 smithi096 ceph-mon[32753]: pgmap v21699: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:03:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40332.324151 secs 2024-04-26T23:03:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:14 smithi194 ceph-mon[33017]: pgmap v21700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:03:14.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40335.316834 secs 2024-04-26T23:03:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40332.324151 secs 2024-04-26T23:03:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:14 smithi033 ceph-mon[27845]: pgmap v21700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:03:14.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40335.316834 secs 2024-04-26T23:03:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40332.324151 secs 2024-04-26T23:03:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:14 smithi096 ceph-mon[32753]: pgmap v21700: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:03:14.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40335.316834 secs 2024-04-26T23:03:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:16 smithi194 ceph-mon[33017]: pgmap v21701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:03:16.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:16 smithi033 ceph-mon[27845]: pgmap v21701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:03:16.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:16 smithi096 ceph-mon[32753]: pgmap v21701: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:03:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:17.460+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:17.460+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:03:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:17.460+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:18.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:18 smithi194 ceph-mon[33017]: pgmap v21702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:18.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:18 smithi033 ceph-mon[27845]: pgmap v21702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:18 smithi096 ceph-mon[32753]: pgmap v21702: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40337.324235 secs 2024-04-26T23:03:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40340.316958 secs 2024-04-26T23:03:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40337.324235 secs 2024-04-26T23:03:19.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40340.316958 secs 2024-04-26T23:03:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40337.324235 secs 2024-04-26T23:03:20.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40340.316958 secs 2024-04-26T23:03:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:20 smithi033 ceph-mon[27845]: pgmap v21703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:20.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:20 smithi096 ceph-mon[32753]: pgmap v21703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:21.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:20 smithi194 ceph-mon[33017]: pgmap v21703: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:22 smithi033 ceph-mon[27845]: pgmap v21704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:22 smithi096 ceph-mon[32753]: pgmap v21704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:23.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:22 smithi194 ceph-mon[33017]: pgmap v21704: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40342.324318 secs 2024-04-26T23:03:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:24 smithi033 ceph-mon[27845]: pgmap v21705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40345.317079 secs 2024-04-26T23:03:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40342.324318 secs 2024-04-26T23:03:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:24 smithi096 ceph-mon[32753]: pgmap v21705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40345.317079 secs 2024-04-26T23:03:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40342.324318 secs 2024-04-26T23:03:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:24 smithi194 ceph-mon[33017]: pgmap v21705: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:25.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40345.317079 secs 2024-04-26T23:03:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:26 smithi033 ceph-mon[27845]: pgmap v21706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:26 smithi096 ceph-mon[32753]: pgmap v21706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:27.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:26 smithi194 ceph-mon[33017]: pgmap v21706: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:28 smithi033 ceph-mon[27845]: pgmap v21707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:28 smithi096 ceph-mon[32753]: pgmap v21707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:29.102 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:28 smithi194 ceph-mon[33017]: pgmap v21707: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40347.324379 secs 2024-04-26T23:03:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40350.317203 secs 2024-04-26T23:03:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40347.324379 secs 2024-04-26T23:03:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40350.317203 secs 2024-04-26T23:03:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40347.324379 secs 2024-04-26T23:03:30.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40350.317203 secs 2024-04-26T23:03:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:30 smithi033 ceph-mon[27845]: pgmap v21708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:30 smithi096 ceph-mon[32753]: pgmap v21708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:31.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:30 smithi194 ceph-mon[33017]: pgmap v21708: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:31.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:31 smithi033 ceph-mon[27845]: pgmap v21709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:31.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:31 smithi096 ceph-mon[32753]: pgmap v21709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:32.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:31 smithi194 ceph-mon[33017]: pgmap v21709: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:32.474+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:32.474+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:32.474+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:03:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40352.324465 secs 2024-04-26T23:03:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:34 smithi194 ceph-mon[33017]: pgmap v21710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:34.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40355.317358 secs 2024-04-26T23:03:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40352.324465 secs 2024-04-26T23:03:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:34 smithi033 ceph-mon[27845]: pgmap v21710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:34.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40355.317358 secs 2024-04-26T23:03:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40352.324465 secs 2024-04-26T23:03:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:34 smithi096 ceph-mon[32753]: pgmap v21710: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:34.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40355.317358 secs 2024-04-26T23:03:36.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:36 smithi194 ceph-mon[33017]: pgmap v21711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:36.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:36 smithi033 ceph-mon[27845]: pgmap v21711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:36 smithi096 ceph-mon[32753]: pgmap v21711: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:38.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:38 smithi194 ceph-mon[33017]: pgmap v21712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:38.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:38 smithi033 ceph-mon[27845]: pgmap v21712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:38.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:38 smithi096 ceph-mon[32753]: pgmap v21712: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:39.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40357.324552 secs 2024-04-26T23:03:39.583 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40360.317503 secs 2024-04-26T23:03:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40357.324552 secs 2024-04-26T23:03:39.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40360.317503 secs 2024-04-26T23:03:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40357.324552 secs 2024-04-26T23:03:39.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40360.317503 secs 2024-04-26T23:03:40.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:40 smithi194 ceph-mon[33017]: pgmap v21713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:40.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:40 smithi033 ceph-mon[27845]: pgmap v21713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:40.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:40 smithi096 ceph-mon[32753]: pgmap v21713: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:42.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:42 smithi194 ceph-mon[33017]: pgmap v21714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:42.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:42 smithi033 ceph-mon[27845]: pgmap v21714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:42.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:42 smithi096 ceph-mon[32753]: pgmap v21714: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40362.324640 secs 2024-04-26T23:03:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:44 smithi194 ceph-mon[33017]: pgmap v21715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:44.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40365.317650 secs 2024-04-26T23:03:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40362.324640 secs 2024-04-26T23:03:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:44 smithi033 ceph-mon[27845]: pgmap v21715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:44.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40365.317650 secs 2024-04-26T23:03:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40362.324640 secs 2024-04-26T23:03:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:44 smithi096 ceph-mon[32753]: pgmap v21715: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:44.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40365.317650 secs 2024-04-26T23:03:46.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:46 smithi194 ceph-mon[33017]: pgmap v21716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:46.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:46 smithi033 ceph-mon[27845]: pgmap v21716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:46.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:46 smithi096 ceph-mon[32753]: pgmap v21716: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:47.485+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:47.485+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:03:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:03:47.485+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:03:48.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:48 smithi194 ceph-mon[33017]: pgmap v21717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:48.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:48 smithi033 ceph-mon[27845]: pgmap v21717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:48.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:48 smithi096 ceph-mon[32753]: pgmap v21717: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40367.324728 secs 2024-04-26T23:03:49.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40370.317780 secs 2024-04-26T23:03:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40367.324728 secs 2024-04-26T23:03:49.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40370.317780 secs 2024-04-26T23:03:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40367.324728 secs 2024-04-26T23:03:49.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40370.317780 secs 2024-04-26T23:03:50.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:50 smithi194 ceph-mon[33017]: pgmap v21718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:50.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:50 smithi033 ceph-mon[27845]: pgmap v21718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:50.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:50 smithi096 ceph-mon[32753]: pgmap v21718: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:52.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:52 smithi194 ceph-mon[33017]: pgmap v21719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:52.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:52 smithi033 ceph-mon[27845]: pgmap v21719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:52.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:52 smithi096 ceph-mon[32753]: pgmap v21719: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40372.324815 secs 2024-04-26T23:03:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:54 smithi194 ceph-mon[33017]: pgmap v21720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:54.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40375.317857 secs 2024-04-26T23:03:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40372.324815 secs 2024-04-26T23:03:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:54 smithi033 ceph-mon[27845]: pgmap v21720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40375.317857 secs 2024-04-26T23:03:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40372.324815 secs 2024-04-26T23:03:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:54 smithi096 ceph-mon[32753]: pgmap v21720: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:54.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40375.317857 secs 2024-04-26T23:03:56.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:56 smithi194 ceph-mon[33017]: pgmap v21721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:56.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:56 smithi033 ceph-mon[27845]: pgmap v21721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:56.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:56 smithi096 ceph-mon[32753]: pgmap v21721: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:03:58.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:58 smithi194 ceph-mon[33017]: pgmap v21722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:58.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:58 smithi033 ceph-mon[27845]: pgmap v21722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:58.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:58 smithi096 ceph-mon[32753]: pgmap v21722: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:03:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40377.324921 secs 2024-04-26T23:03:59.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:03:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40380.317980 secs 2024-04-26T23:03:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40377.324921 secs 2024-04-26T23:03:59.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:03:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40380.317980 secs 2024-04-26T23:03:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40377.324921 secs 2024-04-26T23:03:59.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:03:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40380.317980 secs 2024-04-26T23:04:00.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:00 smithi194 ceph-mon[33017]: pgmap v21723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:00.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:00 smithi033 ceph-mon[27845]: pgmap v21723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:00.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:00 smithi096 ceph-mon[32753]: pgmap v21723: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:02.499+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:02.499+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:04:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:02.499+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:02.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:02 smithi033 ceph-mon[27845]: pgmap v21724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:02.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:02 smithi096 ceph-mon[32753]: pgmap v21724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:02 smithi194 ceph-mon[33017]: pgmap v21724: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40382.324994 secs 2024-04-26T23:04:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:04 smithi033 ceph-mon[27845]: pgmap v21725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:04.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40385.318137 secs 2024-04-26T23:04:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40382.324994 secs 2024-04-26T23:04:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:04 smithi096 ceph-mon[32753]: pgmap v21725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:04.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40385.318137 secs 2024-04-26T23:04:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40382.324994 secs 2024-04-26T23:04:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:04 smithi194 ceph-mon[33017]: pgmap v21725: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40385.318137 secs 2024-04-26T23:04:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:06 smithi033 ceph-mon[27845]: pgmap v21726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:06.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:06 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:04:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:06 smithi096 ceph-mon[32753]: pgmap v21726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:06.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:06 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:04:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:06 smithi194 ceph-mon[33017]: pgmap v21726: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:06 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:04:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:04:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:04:07.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:07 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:04:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:04:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:04:07.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:07 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:04:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:04:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:04:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:07 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:04:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:08 smithi033 ceph-mon[27845]: pgmap v21727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:08.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:08 smithi033 ceph-mon[27845]: pgmap v21728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:08 smithi096 ceph-mon[32753]: pgmap v21727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:08.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:08 smithi096 ceph-mon[32753]: pgmap v21728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:08 smithi194 ceph-mon[33017]: pgmap v21727: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:08.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:08 smithi194 ceph-mon[33017]: pgmap v21728: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40387.325054 secs 2024-04-26T23:04:09.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40390.318262 secs 2024-04-26T23:04:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40387.325054 secs 2024-04-26T23:04:09.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40390.318262 secs 2024-04-26T23:04:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40387.325054 secs 2024-04-26T23:04:09.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40390.318262 secs 2024-04-26T23:04:10.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:10 smithi033 ceph-mon[27845]: pgmap v21729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:10.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:10 smithi096 ceph-mon[32753]: pgmap v21729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:10.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:10 smithi194 ceph-mon[33017]: pgmap v21729: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:12.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:12 smithi033 ceph-mon[27845]: pgmap v21730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:12.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:12 smithi096 ceph-mon[32753]: pgmap v21730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:12.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:12 smithi194 ceph-mon[33017]: pgmap v21730: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:14.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:14 smithi194 ceph-mon[33017]: pgmap v21731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:14.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40392.325141 secs 2024-04-26T23:04:14.655 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40395.318385 secs 2024-04-26T23:04:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:14 smithi033 ceph-mon[27845]: pgmap v21731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40392.325141 secs 2024-04-26T23:04:14.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40395.318385 secs 2024-04-26T23:04:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:14 smithi096 ceph-mon[32753]: pgmap v21731: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:14.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40392.325141 secs 2024-04-26T23:04:14.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40395.318385 secs 2024-04-26T23:04:16.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:16 smithi033 ceph-mon[27845]: pgmap v21732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:16.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:16 smithi096 ceph-mon[32753]: pgmap v21732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:16.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:16 smithi194 ceph-mon[33017]: pgmap v21732: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T23:04:17.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:17.515+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:17.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:17.515+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:04:17.516 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:17.515+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:18.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:18 smithi033 ceph-mon[27845]: pgmap v21733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:04:18.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:18 smithi096 ceph-mon[32753]: pgmap v21733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:04:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:18 smithi194 ceph-mon[33017]: pgmap v21733: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:04:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40397.325202 secs 2024-04-26T23:04:19.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40400.318510 secs 2024-04-26T23:04:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40397.325202 secs 2024-04-26T23:04:19.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40400.318510 secs 2024-04-26T23:04:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40397.325202 secs 2024-04-26T23:04:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40400.318510 secs 2024-04-26T23:04:20.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:20 smithi096 ceph-mon[32753]: pgmap v21734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:20 smithi194 ceph-mon[33017]: pgmap v21734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:20 smithi033 ceph-mon[27845]: pgmap v21734: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:22.682 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:22 smithi096 ceph-mon[32753]: pgmap v21735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:22 smithi194 ceph-mon[33017]: pgmap v21735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:22.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:22 smithi033 ceph-mon[27845]: pgmap v21735: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:24 smithi194 ceph-mon[33017]: pgmap v21736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:24 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40402.325263 secs 2024-04-26T23:04:24.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:24 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40405.318631 secs 2024-04-26T23:04:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:24 smithi033 ceph-mon[27845]: pgmap v21736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:24.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:24 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40402.325263 secs 2024-04-26T23:04:24.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:24 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40405.318631 secs 2024-04-26T23:04:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:24 smithi096 ceph-mon[32753]: pgmap v21736: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:24.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:24 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40402.325263 secs 2024-04-26T23:04:24.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:24 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40405.318631 secs 2024-04-26T23:04:26.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:26 smithi194 ceph-mon[33017]: pgmap v21737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:26.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:26 smithi033 ceph-mon[27845]: pgmap v21737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:26.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:26 smithi096 ceph-mon[32753]: pgmap v21737: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:28.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:28 smithi194 ceph-mon[33017]: pgmap v21738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:28.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:28 smithi033 ceph-mon[27845]: pgmap v21738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:28.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:28 smithi096 ceph-mon[32753]: pgmap v21738: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:29 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40407.325321 secs 2024-04-26T23:04:29.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:29 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40410.318763 secs 2024-04-26T23:04:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:29 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40407.325321 secs 2024-04-26T23:04:29.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:29 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40410.318763 secs 2024-04-26T23:04:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:29 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40407.325321 secs 2024-04-26T23:04:29.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:29 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40410.318763 secs 2024-04-26T23:04:30.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:30 smithi194 ceph-mon[33017]: pgmap v21739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:30.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:30 smithi033 ceph-mon[27845]: pgmap v21739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:30.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:30 smithi096 ceph-mon[32753]: pgmap v21739: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:32.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:32.517+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:32.517 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:32.517+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:04:32.518 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:32.517+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:32.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:32 smithi194 ceph-mon[33017]: pgmap v21740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:32.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:32 smithi033 ceph-mon[27845]: pgmap v21740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:32.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:32 smithi096 ceph-mon[32753]: pgmap v21740: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:34 smithi194 ceph-mon[33017]: pgmap v21741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:34 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40412.325431 secs 2024-04-26T23:04:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:34 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40415.318905 secs 2024-04-26T23:04:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:34 smithi033 ceph-mon[27845]: pgmap v21741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:34.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:34 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40412.325431 secs 2024-04-26T23:04:34.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:34 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40415.318905 secs 2024-04-26T23:04:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:34 smithi096 ceph-mon[32753]: pgmap v21741: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:34.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:34 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40412.325431 secs 2024-04-26T23:04:34.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:34 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40415.318905 secs 2024-04-26T23:04:36.681 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:36 smithi096 ceph-mon[32753]: pgmap v21742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:36.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:36 smithi194 ceph-mon[33017]: pgmap v21742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:36.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:36 smithi033 ceph-mon[27845]: pgmap v21742: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:38.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:38 smithi194 ceph-mon[33017]: pgmap v21743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:38.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:38 smithi033 ceph-mon[27845]: pgmap v21743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:38.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:38 smithi096 ceph-mon[32753]: pgmap v21743: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:39.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:39 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40417.325519 secs 2024-04-26T23:04:39.833 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:39 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40420.319003 secs 2024-04-26T23:04:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:39 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40417.325519 secs 2024-04-26T23:04:39.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:39 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40420.319003 secs 2024-04-26T23:04:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:39 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40417.325519 secs 2024-04-26T23:04:39.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:39 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40420.319003 secs 2024-04-26T23:04:40.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:40 smithi194 ceph-mon[33017]: pgmap v21744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:40.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:40 smithi033 ceph-mon[27845]: pgmap v21744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:40.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:40 smithi096 ceph-mon[32753]: pgmap v21744: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:42.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:42 smithi194 ceph-mon[33017]: pgmap v21745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:42.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:42 smithi033 ceph-mon[27845]: pgmap v21745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:42.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:42 smithi096 ceph-mon[32753]: pgmap v21745: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:44 smithi194 ceph-mon[33017]: pgmap v21746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:44 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40422.325601 secs 2024-04-26T23:04:44.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:44 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40425.319165 secs 2024-04-26T23:04:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:44 smithi033 ceph-mon[27845]: pgmap v21746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:44 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40422.325601 secs 2024-04-26T23:04:44.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:44 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40425.319165 secs 2024-04-26T23:04:44.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:44 smithi096 ceph-mon[32753]: pgmap v21746: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:44 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40422.325601 secs 2024-04-26T23:04:44.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:44 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40425.319165 secs 2024-04-26T23:04:46.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:46 smithi194 ceph-mon[33017]: pgmap v21747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:46.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:46 smithi033 ceph-mon[27845]: pgmap v21747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:46.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:46 smithi096 ceph-mon[32753]: pgmap v21747: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:47.533+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:47.533+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:04:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:04:47.533+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:04:48.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:48 smithi194 ceph-mon[33017]: pgmap v21748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:48.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:48 smithi033 ceph-mon[27845]: pgmap v21748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:48.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:48 smithi096 ceph-mon[32753]: pgmap v21748: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:49 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40427.325688 secs 2024-04-26T23:04:49.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:49 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40430.319283 secs 2024-04-26T23:04:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:49 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40427.325688 secs 2024-04-26T23:04:49.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:49 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40430.319283 secs 2024-04-26T23:04:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:49 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40427.325688 secs 2024-04-26T23:04:49.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:49 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40430.319283 secs 2024-04-26T23:04:50.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:50 smithi194 ceph-mon[33017]: pgmap v21749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:50.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:50 smithi033 ceph-mon[27845]: pgmap v21749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:50.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:50 smithi096 ceph-mon[32753]: pgmap v21749: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:52.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:52 smithi194 ceph-mon[33017]: pgmap v21750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:52.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:52 smithi033 ceph-mon[27845]: pgmap v21750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:52.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:52 smithi096 ceph-mon[32753]: pgmap v21750: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:54 smithi033 ceph-mon[27845]: pgmap v21751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:54 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40432.325770 secs 2024-04-26T23:04:54.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:54 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40435.319399 secs 2024-04-26T23:04:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:54 smithi194 ceph-mon[33017]: pgmap v21751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:54 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40432.325770 secs 2024-04-26T23:04:54.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:54 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40435.319399 secs 2024-04-26T23:04:54.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:54 smithi096 ceph-mon[32753]: pgmap v21751: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:54 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40432.325770 secs 2024-04-26T23:04:54.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:54 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40435.319399 secs 2024-04-26T23:04:56.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:56 smithi194 ceph-mon[33017]: pgmap v21752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:56.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:56 smithi033 ceph-mon[27845]: pgmap v21752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:56.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:56 smithi096 ceph-mon[32753]: pgmap v21752: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:04:58.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:58 smithi194 ceph-mon[33017]: pgmap v21753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:58 smithi033 ceph-mon[27845]: pgmap v21753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:58.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:58 smithi096 ceph-mon[32753]: pgmap v21753: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:04:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:59 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40437.325853 secs 2024-04-26T23:04:59.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:04:59 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40440.319483 secs 2024-04-26T23:04:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:59 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40437.325853 secs 2024-04-26T23:04:59.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:04:59 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40440.319483 secs 2024-04-26T23:04:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:59 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40437.325853 secs 2024-04-26T23:04:59.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:04:59 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40440.319483 secs 2024-04-26T23:05:00.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:00 smithi194 ceph-mon[33017]: pgmap v21754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:00.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:00 smithi033 ceph-mon[27845]: pgmap v21754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:00.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:00 smithi096 ceph-mon[32753]: pgmap v21754: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:05:02.546+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:05:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:05:02.546+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:05:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:05:02.546+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:05:02.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:02 smithi194 ceph-mon[33017]: pgmap v21755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:05:02.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:02 smithi033 ceph-mon[27845]: pgmap v21755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:05:02.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:02 smithi096 ceph-mon[32753]: pgmap v21755: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:05:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:04 smithi194 ceph-mon[33017]: pgmap v21756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:04 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40442.325943 secs 2024-04-26T23:05:04.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:04 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40445.319548 secs 2024-04-26T23:05:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:04 smithi033 ceph-mon[27845]: pgmap v21756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:04 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40442.325943 secs 2024-04-26T23:05:04.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:04 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40445.319548 secs 2024-04-26T23:05:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:04 smithi096 ceph-mon[32753]: pgmap v21756: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:04 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40442.325943 secs 2024-04-26T23:05:04.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:04 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40445.319548 secs 2024-04-26T23:05:06.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:06 smithi194 ceph-mon[33017]: pgmap v21757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:05:06.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:06 smithi033 ceph-mon[27845]: pgmap v21757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:05:06.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:06 smithi096 ceph-mon[32753]: pgmap v21757: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:05:07.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:07 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:05:07.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:07 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:05:07.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:07 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:05:08.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:08 smithi033 ceph-mon[27845]: pgmap v21758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:08.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:08 smithi096 ceph-mon[32753]: pgmap v21758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:09.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:08 smithi194 ceph-mon[33017]: pgmap v21758: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40447.326004 secs 2024-04-26T23:05:09.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40450.319709 secs 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:05:09.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:09 smithi033 ceph-mon[27845]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40447.326004 secs 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40450.319709 secs 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:05:09.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:09 smithi096 ceph-mon[32753]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40447.326004 secs 2024-04-26T23:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40450.319709 secs 2024-04-26T23:05:10.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:05:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 172.21.15.33:0/1944267362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:05:10.104 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:09 smithi194 ceph-mon[33017]: from='mgr.14931 ' entity='mgr.x' 2024-04-26T23:05:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:10 smithi033 ceph-mon[27845]: pgmap v21759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:10 smithi033 ceph-mon[27845]: pgmap v21760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:05:10.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:10 smithi033 ceph-mon[27845]: pgmap v21761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:10.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:10 smithi096 ceph-mon[32753]: pgmap v21759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:10 smithi096 ceph-mon[32753]: pgmap v21760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:05:10.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:10 smithi096 ceph-mon[32753]: pgmap v21761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:10 smithi194 ceph-mon[33017]: pgmap v21759: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:10 smithi194 ceph-mon[33017]: pgmap v21760: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:05:11.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:10 smithi194 ceph-mon[33017]: pgmap v21761: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:12.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:12 smithi033 ceph-mon[27845]: pgmap v21762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:12.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:12 smithi096 ceph-mon[32753]: pgmap v21762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:13.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:12 smithi194 ceph-mon[33017]: pgmap v21762: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:14.004 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:13 smithi194 ceph-mon[33017]: pgmap v21763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T23:05:14.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:13 smithi033 ceph-mon[27845]: pgmap v21763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T23:05:14.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:13 smithi096 ceph-mon[32753]: pgmap v21763: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-26T23:05:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:14 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40452.326065 secs 2024-04-26T23:05:15.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:14 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40455.319833 secs 2024-04-26T23:05:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:14 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40452.326065 secs 2024-04-26T23:05:15.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:14 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40455.319833 secs 2024-04-26T23:05:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:14 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40452.326065 secs 2024-04-26T23:05:15.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:14 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40455.319833 secs 2024-04-26T23:05:16.103 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:15 smithi194 ceph-mon[33017]: pgmap v21764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:16.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:15 smithi033 ceph-mon[27845]: pgmap v21764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:16.181 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:15 smithi096 ceph-mon[32753]: pgmap v21764: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:05:17.550+0000 7fef05200640 1 -- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 msgr2=0x56438583e370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.96:6838/1966519759 2024-04-26T23:05:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:05:17.550+0000 7feeeea00640 1 client.26084 ms_handle_refused on v2:172.21.15.96:6838/1966519759 2024-04-26T23:05:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi033.front.sepia.ceph.com.smithi033.stderr:2024-04-26T23:05:17.550+0000 7fef05200640 1 --2- 172.21.15.33:0/3958398144 >> [v2:172.21.15.96:6838/1966519759,v1:172.21.15.96:6839/1966519759] conn(0x56438583bed0 0x56438583e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T23:05:18.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:18 smithi194 ceph-mon[33017]: pgmap v21765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:18.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:18 smithi033 ceph-mon[27845]: pgmap v21765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:18.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:18 smithi096 ceph-mon[32753]: pgmap v21765: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:05:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:19 smithi194 ceph-mon[33017]: 1 slow requests, 0 included below; oldest blocked for > 40457.326154 secs 2024-04-26T23:05:19.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:19 smithi194 ceph-mon[33017]: 2 slow requests, 0 included below; oldest blocked for > 40460.319970 secs 2024-04-26T23:05:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:19 smithi033 ceph-mon[27845]: 1 slow requests, 0 included below; oldest blocked for > 40457.326154 secs 2024-04-26T23:05:19.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:19 smithi033 ceph-mon[27845]: 2 slow requests, 0 included below; oldest blocked for > 40460.319970 secs 2024-04-26T23:05:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:19 smithi096 ceph-mon[32753]: 1 slow requests, 0 included below; oldest blocked for > 40457.326154 secs 2024-04-26T23:05:19.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:19 smithi096 ceph-mon[32753]: 2 slow requests, 0 included below; oldest blocked for > 40460.319970 secs 2024-04-26T23:05:20.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:20 smithi194 ceph-mon[33017]: pgmap v21766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:05:20.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:20 smithi033 ceph-mon[27845]: pgmap v21766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:05:20.935 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:20 smithi096 ceph-mon[32753]: pgmap v21766: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-26T23:05:22.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 26 23:05:22 smithi194 ceph-mon[33017]: pgmap v21767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:22.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Apr 26 23:05:22 smithi033 ceph-mon[27845]: pgmap v21767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:22.931 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 26 23:05:22 smithi096 ceph-mon[32753]: pgmap v21767: 97 pgs: 97 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-26T23:05:23.861 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-04-26T23:05:23.862 DEBUG:teuthology.task.console_log:Killing console logger for smithi033 2024-04-26T23:05:23.862 DEBUG:teuthology.task.console_log:Killing console logger for smithi096 2024-04-26T23:05:23.863 DEBUG:teuthology.task.console_log:Killing console logger for smithi194 2024-04-26T23:05:23.863 DEBUG:teuthology.exit:Finished running handlers